mirror of
https://github.com/PerpetualSoftware/pad.git
synced 2026-10-26 14:30:13 +00:00
dbbbefda75d4fc4aaeac7bdaecd65eab772f00a9
93 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 xarmian
|
7825f3afc5 |
fix(mcp): MCP's long-lived streams keep alive and take a stream admission slot (BUG-3835) (#2197)
* fix(mcp): MCP's long-lived streams keep alive and take a stream admission slot (BUG-3835) An idle MCP event stream made no write, so #2182's write deadline never fired on it: a client that stopped reading but held TCP open pinned a handler goroutine and its session until a notification happened to be written, and a signed-in client could open any number of them. - A ': keepalive' comment at the /events cadence on any response the transport commits as 200 text/event-stream, written through the same bounded writer. mcp-go writes each SSE event with one Write, so a mutex at Write boundaries keeps keepalives between events. No keepalive after a JSON-RPC response (the final event of a POST stream) or after the handler returns. mcp-go's own heartbeat is not used: it sends ping requests every client must answer. - The shared streamAdmission gate, keyed by streamPrincipal, taken at request entry for the two long-lived doors: GET /mcp and a subscriptions/listen POST. A refusal is the same plain 429 sse_limit_exceeded with Retry-After 5, sent before the transport runs. A tool call upgraded to SSE mid-call is not counted: a refusal there would land after its side effects. Listen is detected by decoding the whole body exactly as mcp-go does (encoding/json into a "method" field), not by a fixed-size peek that key order could evade. - Docs, config, the startup log and the metric help say the budget now covers MCP's long-lived streams. Credential expiry for /mcp streams is BUG-3837. Claude-Session: https://claude.ai/code/session_01Rj4CMyK2EqxerMihbGs3Kr * docs(mcp): the keepalive's shutdown window, the event-store premise and the chatgpt mount, stated precisely (BUG-3835) Codex r1 (Medium, proposed declined): a keepalive tick between the transport's return and end() can add one comment frame; it is still inside mcpBoundWrites, which joins the keepalive goroutine before returning, so net/http never sees a write after the handler. The comments now say so. Codex r1 (Low): GET /mcp/chatgpt is admitted too; the docs named only /mcp. Hunter r1: the one-Write premise holds only without an event store (writeSSEEventRaw writes twice); stated at the premise. Detection of a stalled idle client takes up to about three keepalive intervals; stated in docs/api.md. Claude-Session: https://claude.ai/code/session_01Rj4CMyK2EqxerMihbGs3Kr * fix(mcp): a panicking transport stops the keepalive; MCP streams are stated long-lived without a credential re-check (BUG-3835) Hunter r2 (Low-Medium): the keepalive shutdown ran after next.ServeHTTP without defer, so a panicking transport left it running past the handler, with chi's Recoverer swallowing the panic. Now deferred; pinned by TestMCP_BUG3835_PanickingTransportStopsTheKeepalive (red when not deferred). Hunter r2 (Medium): docs/api.md and two code comments said no MCP route is long-lived. The MCP transport's GET and subscriptions/listen streams are, and do not yet re-check their credential; that is BUG-3837. Claude-Session: https://claude.ai/code/session_01Rj4CMyK2EqxerMihbGs3Kr * docs(mcp): the keepalive catches dead peers; a live non-reader is bounded by the budget, not detected (BUG-3835) r3 hunter Medium: with default socket buffers a comment every 30s takes days to fill them, so the "caught within about three keepalive intervals" claim held only in the shrunk-buffer tests. api.md, the file header comment and the stalled-reader test comment now say what the keepalive catches and what the budget bounds. Codex r3 Low: deployment.md's stream gauge and checklist now include MCP's long-lived streams. The end()/ended comments state the one-frame window between the transport's return and end(). Claude-Session: https://claude.ai/code/session_01Rj4CMyK2EqxerMihbGs3Kr * fix(mcp): the admission body read is in the body census; dead-peer detection is stated as TCP gives it (BUG-3835) r4 blind hunter: - M1: mcpLongLivedRequest reads the request body and was not in the body reader census, so TestEveryRequestBodyReaderIsAccountedFor failed. It is now listed, with the reason and the second-buffer cost the lead accepted. - M3: a half-open peer accepts keepalive writes into the socket buffer until TCP gives up on it; "within one keepalive plus the write deadline" held only in the shrunk-buffer tests. api.md and the header comment now say so. - L3: the panicking-transport test waits for its first keepalive, read under the writer's mutex, instead of sleeping a fixed 10ms (M7 still red). Claude-Session: https://claude.ai/code/session_01Rj4CMyK2EqxerMihbGs3Kr * fix(mcp): admission reads a POST body only when its protocol header is modern; r6 fixes (BUG-3835) Pad serves the legacy MCP protocol versions only, and mcp-go serves a subscriptions/listen only on a modern-era POST whose Mcp-Protocol-Version header is modern and matches _meta. So a POST without a modern header is never a listen, and admission no longer reads its whole body a second time (r6 hunter M1, gate approved by the lead). The docs, the metrics help, the startup log and the census reason say listen is counted once the modern era is served. api.md states the keepalive outlives a proxy idle timeout until BUG-3837, the deployment note says the per-user bucket is shared with the browser's /events, and a test pins isFinalSSEEvent against the real transport's response frame. Claude-Session: https://claude.ai/code/session_01Rj4CMyK2EqxerMihbGs3Kr * fix(mcp): admission looks for a listen only on a protocol version the transport serves; r7 fixes (BUG-3835) The listen gate now also requires the header's version to be one the mount's transport serves, read through mcp-go's SupportsProtocolVersion (forwarded through the ChatGPT wrapper). A client probing an unserved modern version at a full budget reaches mcp-go and gets the 400 it falls back on, not a 429 (r7 ruling). Pad serves the legacy versions only, so no POST body is read for admission today, which the docs now say. New tests pin that mcp-go never serves a listen without a modern header (with a control that it serves a well-formed one), that an unserved modern version is not read or counted, and that both production mounts report their served versions. Claude-Session: https://claude.ai/code/session_01Rj4CMyK2EqxerMihbGs3Kr * test(mcp): the served-version gate is checked against the transports pad server mounts (BUG-3835) r8 L2 (lead ruling): internal/mcp's wiring test builds its own transports, so a shim around either production binding would silently send admission back to counting every modern version. Server.MCPMountServesVersion exposes the gate's read of the transport the server holds, and a cmd/pad test drives wireMCP and asserts both /mcp and /mcp/chatgpt report their versions. Shim mutants on each binding turn it red. Claude-Session: https://claude.ai/code/session_012QxKpjqXTpU84auY5zZ8FR * docs(mcp): the bus and mcpBoundWrites comments name the MCP stream slot and keepalive (BUG-3835) r5 hunter 3 and hunter 6's doc half, accepted by the lead (16:04Z): internal/events/bus.go said streamAdmission is acquired by "both endpoints", falsified once MCP's long-lived streams acquire it too, and mcpBoundWrites' doc comment did not mention the admission slot or the keepalive. Comment-only. Claude-Session: https://claude.ai/code/session_012QxKpjqXTpU84auY5zZ8FR * docs(api): the stream budget's 429 is stated once, and the MCP keepalive has its own paragraph (BUG-3835) r5 hunter 2, accepted by the lead (16:04Z): the admission paragraph stated the 429 refusal twice, the second time after the keepalive text it no longer fitted. Docs-only. Claude-Session: https://claude.ai/code/session_012QxKpjqXTpU84auY5zZ8FR |
||
|
xarmian
|
1fe9a15037 |
fix(deploy): Kubernetes keeps attachments on a PVC, and uploads get a writable /tmp (BUG-3714) (#2141)
* fix(deploy): the Kubernetes manifest keeps attachments on a PVC, and gives uploads a writable /tmp (BUG-3714)
/data holds every attachment's bytes in PostgreSQL mode too, but the
manifest mounted it as an emptyDir, so replacing the pod (an upgrade, a
drain, an eviction) lost them all while their rows stayed. It is now
a ReadWriteOnce PersistentVolumeClaim (deploy/k8s/pvc.yaml); the single
replica and Recreate strategy suit RWO.
On the way: with readOnlyRootFilesystem and no writable /tmp, every
upload failed with a 500 ("create upload temp: open /tmp/...:
read-only file system"). /tmp is now an emptyDir.
docs/deployment.md: pvc.yaml in the apply order, the StorageClass
prerequisite, back up the volume as well as the database, and a tar
block that carries attachments from an emptyDir install to the PVC.
Claude-Session: https://claude.ai/code/session_01Q9WssCjFuHjSfLABoW4tmS
* docs(deploy): the emptyDir-to-PVC migration block stops on a failed export and checks the restored count (BUG-3714, Codex r1)
Claude-Session: https://claude.ai/code/session_01Q9WssCjFuHjSfLABoW4tmS
* docs(deploy): the migration recovery line ends with the count check (BUG-3714, Codex r2)
Claude-Session: https://claude.ai/code/session_01Q9WssCjFuHjSfLABoW4tmS
|
||
|
xarmian
|
bc3e23d55a |
feat(collab): a patch to an item open in an editor applies as a narrowed replace (PLAN-3578 step 2) (#2056)
* feat(collab): a patch to an item open in an editor applies as a narrowed replace (PLAN-3578 step 2)
A content-anchored patch to an item a browser editor holds now goes to a
patch-capable tab (one that connected with ?patch_capable=1; an older tab
would read an edits-only request as an empty body). The tab applies it to
its live document as a replace of exactly the top-level blocks it changes,
aligned under the ruled equivalence (equal, or differing only in a link's
title or bold/italic on whitespace; a trailing empty paragraph ignored), so
it merges with typing and leaves cursors in place. It acks with the
canonical before/after; the server records the version with ExternalContent
and answers with the diff and content_state applied_pending_flush. A block
with no equivalent refuses patch_unstable_roundtrip; refusals are counted
in pad_patch_refusals_total{reason}. MCP 0.88 (behavior).
Claude-Session: https://claude.ai/code/session_01Q9WssCjFuHjSfLABoW4tmS
* test(collab): a room with only a non-capable tab is never sent a patch (PLAN-3578 step 2)
Claude-Session: https://claude.ai/code/session_01Q9WssCjFuHjSfLABoW4tmS
|
||
|
xarmian
|
bc329fb7f4 |
feat(items): agent body writes are version-checked by default, phase 1: count, warn, replace (PLAN-3555) (#2054)
Phase 1 of the ruling that agent content writes carry the version the agent
read. Nothing is refused yet; phase 2 (428 at the agent doors) follows a week
of these counts.
- server: `replace` request member; pad_content_writes_unguarded_total
{door,kind}; warnings.unguarded_content_write on an agent door's unguarded
content write; a seq update_conflict on a content write (applier route
included) carries content_write, changed_at, changed_by_kind and a
"do not resend your body" message.
- CLI: `pad item update --replace` (refused with a token or without a body);
a stderr warning on an unguarded body write; the merge line on conflict.
- MCP 0.86 (additive): pad_item.update `replace`; the content-conflict hint
on both transports; the update description leads with read, edit, write
with the seq.
- Docs and prompts: SKILL.md (both copies), instructions.md, api.md,
mcp.md, deployment.md, README, update help, plan and onboard playbooks.
Claude-Session: https://claude.ai/code/session_01Q9WssCjFuHjSfLABoW4tmS
|
||
|
xarmian
|
3e8bf36f3d |
feat(collab): count failed op-log appends (TASK-3574) (#2051)
pad_collab_append_failures_total counts each batch of sync frames whose op-log append failed. The frames are still broadcast, so if the originating tab closes before its catch-up, a later SyncStep2 can be the only stored copy of the edit, and the item reads clean. The lead ruled this counter the trigger for the SyncStep2 coverage check: any nonzero week where that case could have occurred means build it. Wired as an optional AppendFailureObserver on the TASK-3501 observer, like OverflowObserver; the Room calls onAppendFail, set by the manager. Claude-Session: https://claude.ai/code/session_01Q9WssCjFuHjSfLABoW4tmS |
||
|
xarmian
|
6fd59627bd |
fix(deploy): the Kubernetes manifests run one Pad replica with Recreate and no autoscaler, since collaboration needs a single writer (BUG-3573) (#2039)
deploy/k8s ran replicas: 2 with an HPA up to 10, but real-time collaboration keeps each item's room in the serving process (an in-process bus, op-log appends serialized per item): two pods hold independent rooms for one item, miss each other's edits and leave gaps in the op-log. A RollingUpdate overlapped the old and new writer too. deployment.yaml: replicas 1, strategy Recreate (the old pod stops first; a brief outage per upgrade). hpa.yaml removed. secret.yaml's encryption-key note no longer cites two replicas (the key must still be set: a pod's /data is not kept across restarts). docs/deployment.md says why, how to remove an HPA applied from an earlier version, and that scaling out needs a shared collab bus, which is not built. Claude-Session: https://claude.ai/code/session_01FAVXA5mJ5BNhKmv95jxQkQ |
||
|
xarmian
|
9a7b38ed12 |
build(deploy): docker compose runs the released, signed image; k8s pins a release (TASK-2034) (#2031)
* build(deploy): docker compose runs the released, signed image; k8s pins a release (TASK-2034)
docker-compose.yml built this checkout, so every self-hoster ran an unpinned
source build that reports its version as "dev (unknown)". It now runs
ghcr.io/perpetualsoftware/pad:${PAD_VERSION:-latest}: the multi-arch image
the release pipeline builds and cosign-signs. `latest` moves only on stable
releases (goreleaser's tag template), so an unset PAD_VERSION never pulls an
RC. .env.example documents PAD_VERSION and recommends pinning it.
Contributors keep a source build through a new override,
docker-compose.build.yml, which tags the result pad:local so it never
shadows the released image.
deploy/k8s/deployment.yaml pins 0.17.2 instead of `latest`. With several
replicas (hpa.yaml), `latest` lets pods restarted after a release run a
different version from the rest against one database.
README and docs/deployment.md describe the pin, the upgrade
(`docker compose pull && up -d`) and the contributor build.
Claude-Session: https://claude.ai/code/session_01Q9WssCjFuHjSfLABoW4tmS
* build(deploy): the source-build override never pulls pad:local (TASK-2034, codex P3)
Claude-Session: https://claude.ai/code/session_01Q9WssCjFuHjSfLABoW4tmS
|
||
|
xarmian
|
0b0398b2c0 |
feat(mcp): count admitted MCP requests by method and client (TASK-2307) (#2003)
* feat(mcp): count authenticated MCP requests by method and client (TASK-2307)
The go-sdk port would run its Streamable HTTP transport stateless, and a
stateless transport answers GET with 405, which ends the standalone SSE
stream. No unit test can say whether a real client opens that stream, so
before any engineering time goes into the port this measures it:
pad_mcp_http_requests_total{mount, method, client}, incremented on entry for
every authenticated request on /mcp and /mcp/chatgpt.
On entry, because pad_http_requests_total records a request when it ends,
and a GET holding an SSE stream open ends when the stream does. After
MCPBearerAuth, so refused requests stay with pad_mcp_preauth_denied_total.
The client label is a closed set read from the User-Agent (claude-code,
claude, codex, openai, cursor, windsurf, vscode, mcp-remote, mcp-inspector,
python, node, go, curl, okhttp, other, none), never the header itself.
* fix(mcp): read the User-Agent through requestUserAgent (TASK-2307)
The BUG-2803 call-site guard refuses a raw User-Agent read; the class is
computed from the sanitised value. The guard's exact count moves from 7
call sites to 8, re-pinned with the reason: like fromNativeShell, the counter
only substring-matches the header into a closed label and binds nothing.
* docs(mcp): name what the request counter admits, not 'every authenticated request' (TASK-2307)
Codex round 1: a caller refused by the per-token rate limit inside
MCPBearerAuth is authenticated and is not counted. That is the intended
boundary (a refused GET opens no stream, and the refusal is already counted
in pad_mcp_authz_denials_total{reason="rate_limited"}), so the comment, the
metric help and docs/deployment.md now say what is counted.
* test(mcp): a rate-limited MCP request is not counted (TASK-2307)
Codex round 2: the documented boundary (MCPBearerAuth's per-token 429 is
not an admitted request) had no test. Counted equals admitted; the mutant
moving the counter ahead of auth counts the 429 and fails.
|
||
|
xarmian
|
2898bff09e |
fix(collab): close a peer the op bus had to drop an op for (TASK-1273) (#1981)
* fix(collab): close a peer the op bus had to drop an op for (TASK-1273) When a subscriber's buffer was full, MemoryOpBus.Publish dropped the op with a warning and the peer stayed connected with a document missing it until it happened to reconnect. The bus doc promised a force-close that was never built. Now the first drop of an op a subscriber needs (anything but awareness) marks it overflowed: Publish delivers it nothing more, because an op after the gap would carry an op-log cursor past the dropped row and the reconnect would resume from there and never replay it. The bus signals the overflow (SubscribeWithOverflow) once, and the room manager closes the peer's socket, so its reconnect replays the gap from the op-log. Awareness drops stay silent. Counted in pad_collab_overflow_closes_total through an optional OverflowObserver. cmd/loadtest-collab gains -stalled N (clients that never read). Measured against a local instance: 5 editors at 200 ops/s of 4 KiB, 2 stalled: both stalled clients closed by the server, editors 0 errors; with the close disabled, both stayed open. Claude-Session: https://claude.ai/code/session_01Q9WssCjFuHjSfLABoW4tmS * docs(deployment): pad_collab_overflow_closes_total (TASK-1273) Claude-Session: https://claude.ai/code/session_01Q9WssCjFuHjSfLABoW4tmS * test(collab): a real overflow closes the peer once (TASK-1273, codex) Claude-Session: https://claude.ai/code/session_01Q9WssCjFuHjSfLABoW4tmS |
||
|
xarmian
|
4ed3b5eca2 |
fix(materialize): a job's deadline scales with its op-log (BUG-3521) (#1946)
* fix(materialize): a job's deadline scales with its op-log (BUG-3521) Op-log recovery could not finish a large document: the real op-log of an 86 KB plan (278 content-bearing rows, 470 KiB) took 1.36 s median in-process on an idle box and 1.77 s on one core, against a flat 2 s hard / 1.8 s soft deadline. It failed every attempt and exhausted its budget, leaving items.content stale for the CLI, MCP, search and exports until someone reopened the item. Each job's hard deadline is now the configured base (PAD_MATERIALIZE_TIMEOUT, 2 s) plus 20 ms per KiB of its rows, never below the base and at most MaxTimeout (60 s): ~5x the measured single-core rate, so that document gets ~11 s. The supervisor computes it from the job, so every caller gets it. The soft deadline stays at 90%, and recovery runs off the request path in its own worker. Claude-Session: https://claude.ai/code/session_01Q9WssCjFuHjSfLABoW4tmS * fix(materialize): the worker-start line states the deadline rule, not only its base (BUG-3521, codex r1) Claude-Session: https://claude.ai/code/session_01Q9WssCjFuHjSfLABoW4tmS |
||
|
xarmian
|
f9937c7653 |
fix(mcp): cloud refuses to start without an https origin for MCP OAuth; a blocked self-hosted MCP says so (TASK-1069) (#1920)
* fix(mcp): cloud refuses to start without an https origin for MCP OAuth; a blocked self-hosted MCP says so at startup and in health (TASK-1069) * fix(mcp): pin the cloud origin combinations; health skips the setting read when MCP cannot be blocked (TASK-1069 codex r1) * test(mcp): pad-cloud's production env (empty PAD_MCP_PUBLIC_URL, https PUBLIC_URL) starts, through the real env loading (TASK-1069) |
||
|
xarmian
|
9a99f06c87 |
feat(collab): count resumes and force_refresh on resume, to measure offline editing (TASK-3501) (#1908)
* feat(collab): count resumes and force_refresh on resume, by reason, to measure offline editing (TASK-3501) * fix(collab): resume reports run after itemLock is released (TASK-3501 codex r1) |
||
|
xarmian
|
cfc5391581 |
fix(events): a corrupted event sequence counter heals visibly into a new id space; the id is read back, not rendered through a double (BUG-2744) (#1906)
* fix(events): a corrupted event sequence counter heals visibly into a new id space, and the id is read back rather than rendered through a double (BUG-2744) * test(events): two fixtures that forced a script failure through the sequence key use routes the guard does not heal (BUG-2744 codex r1) |
||
|
xarmian
|
2bd0862af2 |
fix(auth): platform administration refuses an API token (BUG-3361) (#1742)
* fix(auth): platform administration refuses an API token (BUG-3361)
BREAKING for anyone driving /api/v1/admin/* or admin account creation with
a personal access token: those now answer 403 session_required. Use a
`pad auth login` CLI session, or on self-host set PAD_ADMIN_API_TOKENS=allow.
An admin's PAT carried full platform authority. codex reproduced the chain
during BUG-3336's review: register an account with a chosen password,
promote it to admin, sign in as it, and reset any account's password. A
leaked admin token therefore escalated to interactive sessions for any
account.
- refuseAPITokenForAdmin guards the whole /api/v1/admin route group, so an
admin route added later is refused by construction.
TestBUG3361_EveryAdminRouteRefusesAnAdminPAT walks the router to hold it.
- handleRegister refuses admin-create signup with an API token.
- PAD_ADMIN_API_TOKENS=allow is the self-host escape hatch. It is read at
startup, logs a WARN, and is never honoured in cloud mode. Documented in
docs/deployment.md, Security.
- A browser session and a CLI session (padsess_) are unaffected. So is the
Pad Cloud sidecar, which authenticates with the cloud secret, never a
PAT. /audit-log already refused a PAT (BUG-3334).
Claude-Session: https://claude.ai/code/session_01H63zaWhkb5pAafx2WGjrVC
* test: drive platform administration with the admin's session, not its API token (BUG-3361)
CI on the first push: 70 E2E specs and TestMCPCapability_UpgradeChangesNothingWhileOff
called /api/v1/admin/* or admin account creation with an admin PAT, exactly
the use this change refuses. Our own harness was the first operator script
the breaking change broke.
- E2E: global setup keeps the session token login already returns as
fixture.adminSessionToken (padsess_ is not User-Agent bound when sent as
a Bearer). The seven admin call sites use it: account creation in
permission-walk, workspace-tabs, workspace-access-live, no-commerce and
account-delete; the admin user search; and PUT /admin/mcp. Every other
call keeps the PAT, including the MCP client config in plan-2310.
- mcp_capability_test: the admin drives /admin with a CLI session and keeps
its PAT for /mcp, which takes one.
Claude-Session: https://claude.ai/code/session_01H63zaWhkb5pAafx2WGjrVC
* test(e2e): the admin console specs use the admin's session (BUG-3361)
CI on
|
||
|
xarmian
|
66b4a9d477 |
fix(auth): disabling an account ends every credential, and a disabled account gets no new one (BUG-3349) (#1741)
(b) Admin disable deleted sessions only, so the account's PATs and MCP grants kept working. DisableUserAndRevokeAccess does all of it in one transaction: - disables the account; - deletes its sessions and API tokens; - deactivates every OAuth access and refresh token (by subject), and every unexchanged authorization code (through its connection's request id); - deletes its PKCE and OAuth connection rows. EnableUser restores none of them. (c) A disabled account acts nowhere and gets no credential: - requireActiveUserTx is the one gate every credential insert passes. It reads the user row in the insert's own transaction, FOR SHARE on Postgres, and refuses a disabled or missing account with ErrUserDisabled. It covers sessions, API tokens, OAuth codes, access and refresh tokens, and OAuth connections. FOR SHARE conflicts with the disable's row lock, so a mint admitted before a disable cannot land after it. TestBUG3349_MintWaitsForAConcurrentDisable_Postgres proves the wait with two connections. TestEveryCredentialInsertRequiresAnActiveUser fails on any new credential insert that skips the gate. - RequireAuth checks disabled status before the /api/v1/auth/* exemption. Only POST /auth/logout and GET /auth/session stay open, and the session check answers account_disabled. - The MCP door, /oauth/token, /oauth/authorize and /authorize/decide, 2FA login-verify, and email verification (before its token is spent) refuse a disabled account. The login page shows "Your account has been disabled". - Share links the account created are suspended (lead's ruling): byte-identical to an unknown token while it is disabled, serving again once it is re-enabled. Webhooks are unchanged by design; they belong to the workspace. This is documented in docs/deployment.md, "Disabling an account". Part (a) shipped in BUG-3336. codex review: two rounds, eleven findings, all fixed. Claude-Session: https://claude.ai/code/session_01H63zaWhkb5pAafx2WGjrVC |
||
|
xarmian
|
6118f7065c |
feat(mcp): mount the ChatGPT catalog at /mcp/chatgpt, OAuth-only, off by default (TASK-3321 U2b) (#1719)
The ChatGPT projection (U1) gets its own transport at <MCP URL>/chatgpt, behind PAD_CHATGPT_MCP_ENABLED=true and served only while OAuth is available. It is its own OAuth resource (U2a's second canonical): its path-aware protected-resource document is /.well-known/oauth-protected-resource/mcp/chatgpt, its 401s name that document, and tokens do not cross between the two mounts. Personal access tokens are refused there with a 401 naming OAuth-only, ahead of the PAT branch that would accept them without the audience check. The consent screen names the ChatGPT surface when the requested resource is the ChatGPT one. The mount is wired only while the door versions content (ChatGPTDoorVersionsContent, held to the behaviour by a test) and every catalog tool has a response shape (ChatGPTCatalogReady); otherwise startup logs and skips it. /mcp is byte-identical. Claude-Session: https://claude.ai/code/session_01H63zaWhkb5pAafx2WGjrVC |
||
|
xarmian
|
5d420ef606 |
fix(materialize): the recovery sweep stops re-queuing an exhausted item every tick (BUG-3325) (#1704)
* fix(materialize): the recovery sweep stops re-queuing an exhausted item every tick (BUG-3325) An item whose failure budget was spent stayed a sweep candidate: the budget is in memory, so the candidate query cannot see it, and the sweep queued it on every tick for the worker to refuse. On Pad Cloud that logged "candidates=1 queued=1" every minute, forever. The candidate query now also returns MAX(id) (the same quantity as the worker's cursor), and the sweep holds back an item whose budget is spent for that op-log, through the predicate admit uses. It is reported as exhausted_skipped, written only when that count changes, and offered again once its op-log grows. A backing-off item is unchanged. Claude-Session: https://claude.ai/code/session_01H63zaWhkb5pAafx2WGjrVC * fix(materialize): count exhausted items over the budget map, not per page (BUG-3325 review) Codex round 1, P2: exhausted_skipped was counted per keyset page and compared across pages, so a stable population over several pages read as a change on every sweep. The line now carries exhausted_total, counted over the in-memory budget map, and is written when it changes. Claude-Session: https://claude.ai/code/session_01H63zaWhkb5pAafx2WGjrVC |
||
|
xarmian
|
fc1f4e6181 |
fix(server): resolve the client from the right-most untrusted X-Forwarded-For hop (BUG-3323) (#1700)
* fix(server): resolve the client from the right-most untrusted X-Forwarded-For hop (BUG-3323) From a trusted peer, TrustedProxyRealIP took X-Real-IP first and otherwise the LEFTMOST X-Forwarded-For entry. On an appending proxy chain both are client-controlled: the leftmost entry is whatever the client sent, and a proxy that only appends to X-Forwarded-For (Azure Application Gateway, many ingress controllers) passes a client's X-Real-IP through. Any client behind such a proxy chose the address pad used for per-IP rate limits, 2FA challenge binding, session IP checks and audit. Now X-Forwarded-For wins when present, walked right to left, skipping hops inside PAD_TRUSTED_PROXIES, taking the first address outside them (nginx real_ip_recursive). ip:port entries are read as their address; an unparseable entry stops the walk. X-Real-IP is used only when there is no X-Forwarded-For. Cloud is unaffected: its router appends the resolved client, which is the right-most hop. Claude-Session: https://claude.ai/code/session_01H63zaWhkb5pAafx2WGjrVC * fix(server): read every X-Forwarded-For field, and document the append requirement (BUG-3323 review) Codex round 1, P2: Header.Get read only the first X-Forwarded-For field, so a proxy appending its hop as a second field left the client in control. All fields are now joined in order before the walk. The other P2 (a proxy that sets X-Real-IP but passes a client XFF through unchanged) is the cost of preferring XFF; the deployment guide now states that every trusted proxy must append to X-Forwarded-For. Claude-Session: https://claude.ai/code/session_01H63zaWhkb5pAafx2WGjrVC |
||
|
xarmian
|
d44c4f6545 |
fix(web): one workspace-access stream per browser, so open tabs cannot starve a new page over HTTP/1.1 (BUG-3318) (#1696)
* test(e2e): open pad tabs must not starve a new page of HTTP/1.1 connections (BUG-3318, red) * fix(web): one workspace-access stream per browser, not per tab, so open tabs cannot starve a new page over HTTP/1.1 (BUG-3318) |
||
|
xarmian
|
15c872bfed |
fix(materialize): recovery never writes a blank body over a stored one (BUG-3316) (#1693)
The pre-rollout dry run of op-log recovery (TASK-2198 U4) over the live DB's 175 candidates found one that would have been wiped. A load-test item's op-log of 90 tiny rows, with no base document, replays to an EMPTY document, and recovery would have written "" over its 1,285-byte body. A tab never does this: its lazy seed (TASK-1261) sees the empty fragment and seeds FROM items.content. MaterializeFlush now refuses a blank recovered body over a non-blank stored one, decided in the precheck against the row locked for the write, as the new outcome empty_document. Nothing is written: no body, version, seq or watermark. Recovery treats it as terminal until the op-log changes (exhaust, which now takes its kind), so the sweep does not replay it every minute, and it logs the one exhaustion WARN with last_error_kind=empty_document. Stated divergence: a tab refuses only when the fragment is EMPTY, so a user who truly cleared an item's text and then crashed would have flushed "". Recovery cannot see the fragment, so it keeps the old body there too, which is the conservative side. No shrink threshold, by ruling: a tab would flush a shorter body, and the dry run found no such case. Gate: the dry run re-run on this branch, flushing into a disposable COPY of the same snapshot. All 175 items produce the same recovered body and class as the first run. TASK-1270 is refused (empty_document) with its body intact; 156 are stamped, 17 applied, and 1 (a deadline failure) is never flushed. Legs: store (blank and whitespace-only, nothing written) and recovery (refused, terminal, one WARN, reset when the op-log grows). Mutants: a guard that never fires (2 red), no exhaust (1), wrong kind (1). Claude-Session: https://claude.ai/code/session_01H63zaWhkb5pAafx2WGjrVC |
||
|
xarmian
|
cb99879d53 |
feat(materialize): recover unsaved editor sessions into items.content (TASK-2198 U4) (#1683)
* feat(materialize): recover unsaved editor sessions into items.content in the background (TASK-2198 U4) Wires the U3 supervisor in. A collab room reclaimed at the end of its grace TTL with content-bearing rows above the flush watermark (T3), or a dormant pending item found by a one-minute sweep (T1), is queued; one worker goroutine materializes the item's whole content-bearing op-log with the whole-workspace link index and lands it through store.MaterializeFlush, under the collab item lock with no room open. The write is a flush: cursor == MAX(op-log id), rows pending, no set-aside rows, checked in the write's own transaction; watermark to the cursor only; seq bumped; one version row attributed to system/recovery; the op-log is never pruned. PAD_MATERIALIZE=off disables it entirely. A per-item in-memory failure budget (3 failures, doubling backoff, reset by a new op-log row) keeps untrusted input from being retried in a loop. Claude-Session: https://claude.ai/code/session_01H63zaWhkb5pAafx2WGjrVC * test(store): recovery watermark is the cursor, not MAX at UPDATE time (TASK-2198 U4) Claude-Session: https://claude.ai/code/session_01H63zaWhkb5pAafx2WGjrVC * test(store): a second recovery still writes its own version row (TASK-2198 U4) Claude-Session: https://claude.ai/code/session_01H63zaWhkb5pAafx2WGjrVC * docs(mcp): op-log recovery bounds the stale-body window; tool surface v0.60 (TASK-2198 U4) content_state applied_pending_flush now clears on its own after the item's room closes, unless recovery fails or PAD_MATERIALIZE=off. Every sentence that called the window unbounded, or said nothing server-side moves op-log content into items.content, is corrected (CLAUDE.md, instructions.md, README, docs/deployment.md, docs/backup.md, code comments). BEHAVIOR bump to 0.60; goldens regenerated. Claude-Session: https://claude.ai/code/session_01H63zaWhkb5pAafx2WGjrVC * fix(materialize): count ErrNoMemoryCap silently against the per-item budget (TASK-2198 U4) U3 now refuses every job with ErrNoMemoryCap when no memory cap can be established, and warns once itself. The recovery worker counts it like any failure but logs it at debug, not per item. Its log lines go through a logger seam so the test can tell. Claude-Session: https://claude.ai/code/session_01H63zaWhkb5pAafx2WGjrVC * style: gofmt (TASK-2198 U4) Claude-Session: https://claude.ai/code/session_01H63zaWhkb5pAafx2WGjrVC * feat(materialize): one log line per budget exhaustion; count set-aside skips on the sweep line (TASK-2198 U4) An item that uses up its failure budget logs exactly one WARN naming the item, the failure count and the last error kind (DEBUG when the kind is no_memory_cap, which the supervisor already warned about once), and logs again only after op-log growth resets the budget and it is exhausted anew. The schema-version skip goes through the same line. Items skipped for set-aside rows are counted and reported on a new single sweep summary line, written only when it has news; never per item. Claude-Session: https://claude.ai/code/session_01H63zaWhkb5pAafx2WGjrVC * fix(materialize): keyset-page the recovery sweep so no candidate starves (TASK-2198 U4, codex r1) The sweep read the oldest 200 candidates from the start every tick, and the room check and the failure budget filtered them afterwards. Once 200 sat in backoff or exhausted, every sweep returned the same 200 and the 201st was never queued: filtering above a bound is starvation. Each tick now reads one page strictly after an in-memory keyset cursor (newest op-log created_at, item id) and wraps to the start after a short page, so every candidate is offered within ceil(N/limit)+1 ticks, whatever sorts ahead of it. Keyset over NOT IN: no parameter-count limit, and it also covers the room check, which an exclusion list would not. Claude-Session: https://claude.ai/code/session_01H63zaWhkb5pAafx2WGjrVC * test(materialize): the sweep cursor wraps; docs for the exhaustion line, the set-aside count and paging (TASK-2198 U4) Claude-Session: https://claude.ai/code/session_01H63zaWhkb5pAafx2WGjrVC * docs(deployment): reflow the recovery budget paragraph (TASK-2198 U4) Claude-Session: https://claude.ai/code/session_01H63zaWhkb5pAafx2WGjrVC * test(materialize): recovery strips the follows-title marker as a tab's save does (TASK-2198 U4, BUG-3315) Since #1688 a tab loads a link that follows its target's title with the follows-title marker in the link's title attribute, so the marker lives in Y.Docs that recovery replays. Three fixture cases (a follows-title link; the same link after a rename; a URL edited to an external address) are generated from documents that really hold the marker, and the existing 8 are unchanged. A new leg recovers each through the production path (whole-workspace index) and through the runner with a restricted and an empty index, and asserts no 'pad-follows-title' reaches the body. It goes red in all 7 ways when the bundle's marker strips are disabled. Claude-Session: https://claude.ai/code/session_01H63zaWhkb5pAafx2WGjrVC * test(store): a recovery version row carries #1691's history data (TASK-2198 U4) Rebased onto PLAN-2348 U2 (#1691), which adds user_id, line counts and a create marker to version rows. Recovery writes its row through the same UpdateItem insert, so it gets them automatically: no user_id (a system row), not a create, and line counts equal to diff.LineCounts of the stale and recovered bodies. The leg pins that. Claude-Session: https://claude.ai/code/session_01H63zaWhkb5pAafx2WGjrVC |
||
|
xarmian
|
d78ff9bdc5 |
feat(materialize): worker supervisor with per-OS memory cap (TASK-2198 U3) (#1682)
* feat(materialize): parent-side worker supervisor with per-OS memory cap (TASK-2198 U3) The Supervisor runs materializer jobs in ONE long-lived child process (`pad __materialize-worker`), started lazily on the first job and fed jobs serially. A job past PAD_MATERIALIZE_TIMEOUT (default 2s) kills the child (goja's Interrupt, sent as timeout_ms a tenth earlier, is only the soft layer); a dead child is respawned on the next job after an exponential backoff (1s doubling, capped at 5 min, reset by a child that lived a minute). Every stop is logged with its reason (deadline, memory, exit, protocol, start, canceled, closed); every start logs the effective timeout, memory limit and cap mechanism. The child's stderr goes to the log, pid-tagged and bounded, never near the protocol. A job whose child died is never retried here. Memory cap (PAD_MATERIALIZE_MEM_LIMIT, default 2GiB, bytes or KiB/MiB/GiB, clamped to 1536MiB-16GiB): - Linux: RLIMIT_AS set on the child by prlimit before its first byte of input; the child only runs its own startup uncapped. - Windows: Job Object with JOB_OBJECT_LIMIT_PROCESS_MEMORY; the child is created suspended, assigned, then resumed: no uncapped window. - macOS: parent-side RSS watchdog every 100ms (proc_info syscall, ps fallback). - Other: deadline kill only. Not wired into the server yet (U4). Claude-Session: https://claude.ai/code/session_01H63zaWhkb5pAafx2WGjrVC * ci(materialize): run the supervisor on the real macOS and Windows caps (TASK-2198 U3) native-smoke now builds the real materializer bundle (as the go job does) and runs the supervisor, memory-cap and deadline tests on macos-latest (proc_info RSS watchdog) and windows-latest (suspended start + Job Object). PAD_MATERIALIZE_TEST_NO_SKIP turns the cap tests' skips into failures there, so a cap that stops running cannot pass as skipped. Tests: a real Windows processGone (OpenProcess + GetExitCodeProcess), RSS samplers for darwin (the production proc_info path) and Windows (K32GetProcessMemoryInfo), and an allocation helper that writes noise into every byte so macOS memory compression cannot hide it from the watchdog. The worker-loop tests share one loaded Runner through serveWorker (RunWorker's loop, split out); TestWorkerRoundTrip still goes through RunWorker end to end. go test -race -count=3 on the package: 714s -> 413s. docs: the materializer section says the worker and its knobs are inert until U4 wires it in. Claude-Session: https://claude.ai/code/session_01H63zaWhkb5pAafx2WGjrVC * fix(materialize): a worker never outlives its supervising process (TASK-2198 U3, codex r1) If the server died mid-job (SIGKILL, OOM kill, crash) its worker ran on with no deadline kill and, on macOS, no memory watchdog. Two layers: - Linux: Pdeathsig = SIGKILL. The fork runs on a fresh goroutine (startUnlocked), never locked to its thread, so it cannot land on a thread the runtime later retires (golang/go#27505); Go's forkExec re-checks getppid if the parent is already gone. - Every Unix (the only layer on macOS): the worker command calls ExitWhenOrphaned before the bundle loads, polling its parent pid every 200ms against PAD_MATERIALIZE_PARENT_PID and exiting (status 70) when it changes. It writes nothing. - Windows was already covered by the Job Object's KILL_ON_JOB_CLOSE. TestSupervisorOrphanedWorkerExits kills a helper process that supervises a worker hung in a job and requires the worker gone within 1s: 0s on Linux, 105ms with the ppid watch alone; with both layers removed the worker outlives its parent by 5s. It matches native-smoke's TestSupervisor.* selection, so it runs on macOS and Windows too. Claude-Session: https://claude.ai/code/session_01H63zaWhkb5pAafx2WGjrVC * fix(materialize): relative memory cap, refuse when uncappable, GOMAXPROCS=2; nix-safe ps test (TASK-2198 U3) The absolute Linux RLIMIT_AS failed on CI: a 64 MiB allocation died under a 1600 MiB cap, because the runtime there reserved more address space than on the machine the floor was measured on. The cap is now RELATIVE on every OS: once the worker has loaded and before it is sent any job, its usage is read as a baseline and the cap is set to baseline + PAD_MATERIALIZE_MEM_LIMIT — address space on Linux (/proc/<pid>/stat vsize, prlimit, read back), committed memory on Windows (the Job Object's per-process limit, set after the suspended start and load), resident memory on macOS (the watchdog's threshold). Every start logs baseline, limit and effective cap. Lead ruling, REFUSE: if the baseline probe fails or is implausible (0, or above MaxPlausibleBaseline = 8 GiB, ~4.8x the largest measured 1704 MiB), onBaselineProbeFailure kills the worker without sending it the job and fails the job with ErrNoMemoryCap; one WARN until a probe succeeds again. An OS with no cap mechanism never starts a worker and refuses every job with ErrNoMemoryCap, one WARN naming runtime.GOOS. The worker runs with GOMAXPROCS=2: measured loaded-worker baseline 1357 MiB vs 1597 MiB at this host's default (8) and 1704 MiB at 64. MinMemLimit drops from 1536 MiB (an artefact of the absolute cap) to 256 MiB: the corpus grows the worker by at most +10 MiB of address space and +18 MiB resident, and a 4,000-deep nested document by +129..194 MiB. A cgo worker's "pthread_create failed" at the cap is now classified as a memory death. Nix: TestParsePSRSS parses fixed strings only; the live `ps` leg is TestPSRSSLive and skips when there is no `ps` (the Nix sandbox has none; ps is only macOS's fallback), unless PAD_MATERIALIZE_TEST_NO_SKIP is set. Claude-Session: https://claude.ai/code/session_01H63zaWhkb5pAafx2WGjrVC * fix(materialize): Windows kills on the job's memory notification; steady thread set before the baseline (TASK-2198 U3) Windows (CI on c1866957): the Job Object refused the 1 GiB commit (VirtualAlloc errno=1455), but the worker did not die and the job ran to its deadline. The cap no longer depends on the runtime choosing to crash: the job is associated with an I/O completion port, and JOB_OBJECT_MSG_PROCESS_MEMORY_LIMIT for the worker's pid kills it at once (ErrMemoryLimit). errno=1455 on stderr is a second signal. On every OS a cap mechanism that sees the limit hit calls onLimit, and the kill is reported as "reached its <mechanism> cap". Linux (CI's cgo test binary at GOMAXPROCS=64): threads created after the baseline ate the growth budget. The worker now brings its thread set to steady state before readiness (warmThreads: an allocating goroutine per P, then a GC). Measured on a cgo build: threads added after the baseline are 0 at GOMAXPROCS=2 with or without it; at 64, +3 threads and +272 MiB of address space during the corpus without it, +0 and +0 MiB with it. 256 MiB stays the floor (largest growth at GOMAXPROCS=2: +193 MiB, a 4,000-deep nested document; the corpus adds +0 MiB). The cap test's second leg no longer uses GOMAXPROCS=64 (which also changed thread creation) to stand in for a bigger host: the helper inflates its own baseline before readiness, with never-touched address space on Linux, committed memory on Windows and resident memory on macOS. Claude-Session: https://claude.ai/code/session_01H63zaWhkb5pAafx2WGjrVC * feat(materialize): the worker stops after an idle timeout; the next job restarts it (TASK-2198 U3) Dave's ruling on the worker process: GO, with an idle exit. A loaded worker holds ~200 MB resident, which an install that rarely materializes should not pay for all day. PAD_MATERIALIZE_IDLE_TIMEOUT: default 5m, clamped to 30s-24h, 0 = never. After each job the supervisor (re)arms a timer. When it fires, the stop runs only while holding the job slot, so it never takes a worker with a job in flight. It re-checks the idle time, forgets the worker WITHOUT recording a death (backoff is untouched), and closes the worker's stdin so its read loop ends with status 0, with a kill after a grace if it does not go. The stop is logged at INFO, reason=idle. A job that arrives meanwhile waits for the slot, finds no worker, and starts a new one with no backoff wait. Legs: the idle exit after the timeout (INFO, status 0, no WARN); the next job respawns and succeeds with no wait; an idle stop leaves the backoff streak alone (crash, idle, crash waits 2s, not 4s); a job arriving during the stop runs on a new worker; a timer that fires mid-job does not stop that worker; NeverIdle arms no timer; env parsing and clamps. Mutations, each red on its own: recording a death (3 legs), logging as a kill (1), not holding the slot (1), never arming (3). Docs: the knob beside the other two, and `idle` in the stop reasons. Claude-Session: https://claude.ai/code/session_01H63zaWhkb5pAafx2WGjrVC * fix(materialize): Close does not return while an idle-stopping worker still runs (TASK-2198 U3, codex r1) An idle stop forgets the worker before it has exited, so a Close in that window found no worker and returned with it still alive. The supervisor now tracks the worker an idle stop is ending, and Close kills and reaps it. A leg pins it, and goes red without the fix. Claude-Session: https://claude.ai/code/session_01H63zaWhkb5pAafx2WGjrVC * fix(materialize): an idle stop that Close ended says so (TASK-2198 U3, codex r2) The idle stop stored its kill reason unconditionally, so it overwrote the 'closed' of a Close that had already killed the worker, and its stop line said idle. It now claims the reason only if none is set, and the line reports reason=closed. The Close-during-idle-stop leg asserts it. Claude-Session: https://claude.ai/code/session_01H63zaWhkb5pAafx2WGjrVC |
||
|
xarmian
|
b510d67696 |
fix(oauth): sweep expired token, code and PKCE rows (BUG-3301) (#1671)
* fix(oauth): sweep expired token, code and PKCE rows (BUG-3301) Nothing deleted OAuth rows by age. Each refresh leaves the chain's previous access and refresh rows inactive and inserts a fresh pair, measured at +1 access and +1 refresh row per refresh (48 rows a day for one active client at the 1 h access lifetime), forever, on Pad Cloud and on any https self-host that turns MCP on. The token reaper now also runs SweepExpiredOAuthRows every hour. A row goes once its requested_at (its own issuance, measured) is older than its token's lifespan plus 24 h: - refresh and authorization codes: the refresh lifespan. A replayed rotated token or a reused code keeps revoking the family for as long as the token could have been valid, because fosite checks "inactive" before "expired". - access: the access lifespan. Nothing reads an expired access row. - PKCE: the code lifespan. Only abandoned authorizations leave one. The lifespans come from the running OAuth server (defaults when OAuth is not built). No row a live chain or a concurrent exchange could use is ever below a cutoff. Deletes run 500 rows per statement with a 20 ms pause between batches. Measured on SQLite with 50,000 rows and a concurrent token write every 5 ms: with no pause, one write waited 1.55 s; with the pause, the worst write took 52 ms. The per-tick cap is 200 batches per table. Connected Apps and the MCP resume count now drop a chain whose refresh token has been expired for the grace period, so their "upper bound" comments say how far. The deployment guide's "never swept" gap is gone. Tests: store cutoffs, zero cutoffs, and batches with the cap, on SQLite and Postgres. A server end-to-end test through /oauth/token: the live chain survives and still refreshes; an aged-out chain leaves Connected Apps and the resume count; a swept token is refused without revoking the family; a retained rotated token still revokes it. Mutants: inverted age comparison (store tests fail), refresh cutoff at now (e2e fails: the live chain's refresh answers invalid_grant). Claude-Session: https://claude.ai/code/session_01NidgPePY7UvHcEJCMkD2Qz * fix(oauth): code retention adds the code lifespan; Capped means "may remain" (BUG-3301, codex r1) - A code's requested_at is when it was authorized, but the first refresh token it yields starts at the exchange, up to a code lifespan later. The code cutoff now adds the code lifespan to the refresh lifespan, so reuse detection lasts as long as that refresh token could be valid. The comment no longer claims "the chain's lifetime": a chain renews forever. - Capped says the cap was used, so rows MAY remain; the last full batch may have been the end. Claude-Session: https://claude.ai/code/session_01NidgPePY7UvHcEJCMkD2Qz * fix(oauth): a non-expiring lifespan skips its tables (BUG-3301, codex r2) fosite reads a negative lifespan as "never expires"; the cutoff math turned -1 into "now minus the grace", which would delete a live non-expiring token after a day. A table whose retention depends on a negative lifespan now gets a zero cutoff, which the store skips (codes depend on both the code and refresh lifespans). Claude-Session: https://claude.ai/code/session_01NidgPePY7UvHcEJCMkD2Qz |
||
|
xarmian
|
0302977ebe |
fix(server): bound the per-key rate limiter maps and key IPv6 on the /64 (BUG-3308) (#1670)
* fix(server): bound the per-key rate limiter maps and key IPv6 on the /64 (BUG-3308) ipRateLimiter trimmed its map only after a key sat idle for the retention window (30 min, up to 2 h), so a caller with many source addresses grew any per-address bucket without bound. Measured on one limiter: 1,000,000 keys (one per /64 under a /32) held 1,000,000 entries and 182.9 MiB of heap. Two changes: - A size cap per limiter (rateLimitConfig.MaxEntries, default 65,536). At the cap, one sweep to 7/8 of it: first keys idle past retention and buckets back at burst (a full bucket is identical to a fresh one, so dropping it hands nobody anything), then the fullest buckets by whole-token level. A drained bucket goes last, so flooding the map cannot push an attacker's own exhausted bucket out and reset it. The cap is key-agnostic, which is what bounds AuthEmail: its keys are caller-typed email strings, and the plausible-email pre-filter never bounded them (its comment said it did; corrected). Same measurement after: 61,604 entries, 11.3 MiB. One sweep at the default cap costs about 17.5 ms, once per 8,192 new keys. - IPv6 clients are keyed on their /64 at every per-address bucket (rateLimitAddr): the 13 middleware doors, the /mcp pre-auth charge, the decision-provider charge and the share-password per-address bucket. IPv4 and IPv4-mapped addresses key on the address. clientIP is unchanged, so sessions, audit rows and the 2FA challenge binding still see the full address. Tests pin the key table, the cap at caps 1, 4 and 64, eviction order (a drained bucket survives a 1,000-key flood; full buckets go before spent ones; idle keys go), and the /64 binding through the router at every door. Claude-Session: https://claude.ai/code/session_016PXRzXhokr61CLbbu3Txj9 * fix(server): charge limiter buckets under the map lock; rank the sweep below a token (BUG-3308 codex r1) Codex round 1 findings, each verified in the code first: - A caller took the *rate.Limiter from getLimiter and charged it after the map lock dropped, so a request in flight across a sweep could spend a bucket that had been evicted and replaced, getting both the old bucket's tokens and the new one's burst. Every production charge (18 sites) now goes through ipRateLimiter.allow, which looks up and charges under one lock hold. getLimiter stays for tests only. - The sweep ranked buckets by whole tokens, so a drained bucket shared level 0 with buckets holding 0.9 tokens and the cut chose among them at random. It now ranks by the fraction of burst in 1,024 fixed levels. The fixed size also removes a histogram sized by Burst, which panicked for a negative burst. The eviction comment now states what the order buys as a cost: a bucket is evicted only once the low-water mark's worth of other buckets are drained at least as far, so each token handed back costs that many drained tokens elsewhere. Sweep at the default cap: 18.2-19.3 ms (3 benchmark runs). 1,000,000 keys: 61,968 entries, 11.4 MiB. Claude-Session: https://claude.ai/code/session_016PXRzXhokr61CLbbu3Txj9 * docs(server): state what the /64 key and the eviction order promise (BUG-3308 codex r2) Codex round 2 found no charge outside the lock and no missed key. Prose it found false, corrected: - The share-password per-address bucket said one caller could not lock out other viewers. Under the /64 key, viewers sharing a /64 share a bucket, as viewers behind one NATed IPv4 address always did. Both comments now say so. - The eviction comment claimed each token handed back costs a fixed amount of drained tokens inside a refill window. It now states the condition the code enforces: a bucket is evicted only when the low-water mark's worth of other buckets hold no more tokens at that moment, to within burst/1024. It also states the sweep's lock hold. Tests: a sibling address in the /64 is refused with the RIGHT share password too (the gate is charged before the compare), and an IPv4-mapped address shares the IPv4 bucket through the router. Claude-Session: https://claude.ai/code/session_016PXRzXhokr61CLbbu3Txj9 * test(server): say what the flood test does, not a sweep count (BUG-3308 codex r3) Claude-Session: https://claude.ai/code/session_016PXRzXhokr61CLbbu3Txj9 * docs(deployment): rate buckets key an IPv6 client on its /64 (BUG-3308) Claude-Session: https://claude.ai/code/session_016PXRzXhokr61CLbbu3Txj9 * docs(deployment): the MCP and OAuth limits key an IPv6 client on its /64 (BUG-3308) Claude-Session: https://claude.ai/code/session_016PXRzXhokr61CLbbu3Txj9 * test(server): freeze refill in the /64 binding tests (BUG-3308, CI run 36635279128) The door test exhausted a real bucket and then probed a sibling address. Under -race in CI, collab_dial (5/s) refilled past one token between the 429 and the probe, and the sibling got through. The tests now set every limiter refill to ~0 before the first request, so they measure keying, not timing. Control: with a 250 ms gap before the probe and refill live, collab_dial, api and auth_default_api fail; frozen, the same run passes. Every keying mutant is still killed. Claude-Session: https://claude.ai/code/session_016PXRzXhokr61CLbbu3Txj9 |
||
|
xarmian
|
754d85fa7c |
feat(web): the connect modal serves the token path (TASK-2321, PLAN-2310 U6) (#1668)
* feat(web): the connect modal serves the token path (TASK-2321, PLAN-2310 U6) PLAN-2310 DR-8. The session and setup payloads now carry mcp_public_url whenever MCP is available (it was cloud-only), plus mcp_auth: ["oauth","pat"] where OAuth is served, ["pat"] on an http self-host, and [] while MCP is off. sessionMCP reads the setting once, so the URL and the methods cannot disagree across a toggle. ConnectWorkspaceModal keys on both. With OAuth it keeps the sign-in path and offers a personal API token in a disclosure; with tokens only, the token steps are the path and the claim-code tab and Connected apps link are hidden, since both need OAuth. The token steps link to API Tokens in console settings (new #api-tokens anchor) and give per-client config with an Authorization: Bearer header for Claude Code, Cursor, Windsurf, VS Code and Claude Desktop (via mcp-remote). A server that predates mcp_auth gets the OAuth rendering it always had. Tests: a Go state table through the router (8 states, setup and session payloads); vitest for each mcp_auth shape and every client config; an e2e that copies the modal's URL and Cursor config, substitutes the suite's token, and gets tools/list from /mcp. Claude-Session: https://claude.ai/code/session_01NidgPePY7UvHcEJCMkD2Qz * fix(web): connect modal codex r1 (TASK-2321) - The e2e enables MCP inside its try, so a failed readiness assertion after the PUT still turns it back off. - The client selector is a labelled group of aria-pressed toggle buttons: it declared tablist/tab with no panels or arrow-key navigation. - Claude Desktop's config runs `npx -y`, since Desktop cannot answer the install prompt on first use. Claude-Session: https://claude.ai/code/session_01NidgPePY7UvHcEJCMkD2Qz * refactor(server): the four MCP session fields share one read (TASK-2321) After the rebase onto U5 (#1666), the session and setup payloads carried mcp_available and oauth_available from their own reads of the setting beside U6's mcp_public_url and mcp_auth from sessionMCP's. sessionMCP now returns all four from one read, so a concurrent toggle cannot produce oauth_available beside ["pat"], or a URL with mcp_available false. The state-table test asserts the four agree in every state. Claude-Session: https://claude.ai/code/session_01NidgPePY7UvHcEJCMkD2Qz * fix(server): /admin/stats reads the MCP setting once (TASK-2321, codex r4) Its mcp_available and oauth_available came from two reads, so a concurrent toggle could report OAuth available with MCP not. Both now come from one sessionMCP result, as the session payloads do. Claude-Session: https://claude.ai/code/session_01NidgPePY7UvHcEJCMkD2Qz * docs: the Connect dialog's MCP paths (TASK-2321, PLAN-2310 U6) The deployment guide (U7) now says where users find the MCP URL and the token configuration, which this PR adds to the connect modal. Claude-Session: https://claude.ai/code/session_01NidgPePY7UvHcEJCMkD2Qz |
||
|
xarmian
|
5d641b70d0 |
docs: MCP over HTTP on a self-hosted install (TASK-3304, PLAN-2310 U7) (#1669)
* docs: MCP over HTTP on a self-hosted install (TASK-3304, PLAN-2310 U7) docs/deployment.md gains "MCP for agents (self-hosted)": off by default; the origin (PAD_URL / toml url / --url, else PUBLIC_URL, with no host:port fallback), PAD_MCP_PUBLIC_URL, PAD_AUTH_SERVER_URL and PAD_MCP_ENABLED with their precedence and startup-time nature; the admin toggle (GET/PUT /api/v1/admin/mcp, the env lock, the readiness readout); https means OAuth plus tokens and http tokens only; the reverse-proxy Host requirement and PAD_TRUSTED_PROXIES; and the known gaps. .env.example, CLAUDE.md's API and MCP sections, and README follow. Claude-Session: https://claude.ai/code/session_01NidgPePY7UvHcEJCMkD2Qz * docs: .env.example says MCP's toggle and origin are self-host rules (TASK-3304, codex r1) Claude-Session: https://claude.ai/code/session_01NidgPePY7UvHcEJCMkD2Qz * docs: the MCP host check and rate limits as merged (TASK-3304, PLAN-2310 U7) Checked against U3 (#1665) and U4 (#1667) on main: - The 421: honours X-Forwarded-Host only from a PAD_TRUSTED_PROXIES peer; the body names the received host, its header and the configured origin; the first refusal per host is a rate-limited WARN; it runs before auth and after the availability gate. - The DR-9 limits as a table. Claim redemption is keyed per signed-in user, not per address, as the earlier draft said. - Pre-auth /mcp refusals are counted in pad_mcp_preauth_denied_total with a WARN per address, not audit rows. The once-per-process WARN when a proxied request resolves inside PAD_TRUSTED_PROXIES. Claude-Session: https://claude.ai/code/session_01NidgPePY7UvHcEJCMkD2Qz * docs: the console toggle and the pre-auth counter's scope (TASK-3304, codex r2) - The console now has the MCP section (#1666): Admin → Settings → MCP for agents. The "console does not show it yet" sentence is gone, and .env.example names both ways to toggle. - codex r2: pad_mcp_preauth_denied_total counts missing/invalid-token refusals and the pre-auth 429s, not the 421 host refusals, which happen before it. Claude-Session: https://claude.ai/code/session_01NidgPePY7UvHcEJCMkD2Qz * docs: OAuth follows the issuer URL; tokens keep their scopes (TASK-3304, codex r3) Claude-Session: https://claude.ai/code/session_01NidgPePY7UvHcEJCMkD2Qz |
||
|
xarmian
|
372f91f03b |
fix(events): a resume from exactly the discarded peak is refused after a counter-backwards reset, instead of being handed the new sequence as though it followed (BUG-3206) (#1530)
* fix(events): a resume from exactly the discarded peak is refused after a counter-backwards reset, instead of being handed the new sequence as though it followed (BUG-3206) Claude-Session: https://claude.ai/code/session_01A3rXV7GDLAhBe9mL4V6z2m * docs(BUG-3206): the floor does not refuse discarded+1, which is narrower than serving it (codex round 1) Claude-Session: https://claude.ai/code/session_01A3rXV7GDLAhBe9mL4V6z2m |
||
|
xarmian
|
67b25d1dc6 |
docs(BUG-2743): the ambiguity boundary refuses a new-space client on every reconnect until the space climbs past the old peak, not once (#1529)
Corrects the stated cost in the field comment, the counter-reset arm, and docs/deployment.md. The behaviour is unchanged, by lead ruling (a) on BUG-2743's trail. Claude-Session: https://claude.ai/code/session_01A3rXV7GDLAhBe9mL4V6z2m |
||
|
xarmian
|
70f7c0c15c |
fix(watchevents): a resume from a cursor inside an abandoned id space is refused on every arm, instead of being served the new space's ids or told it is caught up (BUG-2743, BUG-2728) (#1518)
* fix(watchevents): a resume from a cursor inside an abandoned id space is refused on every arm, instead of being served the new space's ids or told it is caught up (BUG-2743, BUG-2728) Every arm that detects a watch-stream id-space restart (the epoch arm and both counter-backward arms) now records the old space's high-water mark, and replaySince refuses any cursor at or below it. The boundary is never cleared: climbing past the old peak does not make a cursor below it any less ambiguous. Claude-Session: https://claude.ai/code/session_01A3rXV7GDLAhBe9mL4V6z2m * docs(BUG-2743): the counter-reset arm's prose states the boundary's cost and which id the resume check compares (codex round 1) Claude-Session: https://claude.ai/code/session_01A3rXV7GDLAhBe9mL4V6z2m * fix(watch): pad watch --stream prints a PAD (resync) line on sync_required, so the agent reading it learns it missed notifications instead of carrying on (BUG-2743 follow-through) Clearing the cursor was the process's whole response; the consumer of its stdout could not see a cursor and was never told. The line goes through the same consent gate as a notification, and repeats with no notification between them are folded into one. Claude-Session: https://claude.ai/code/session_01A3rXV7GDLAhBe9mL4V6z2m * docs(plugin): tell the agent what a PAD (resync) monitor line means and what to do with it (BUG-2743 follow-through) Claude-Session: https://claude.ai/code/session_01A3rXV7GDLAhBe9mL4V6z2m * docs(BUG-2743): the resync line says the stream MAY have missed notifications, since sync_required means the span is unvouched, not certainly lossy (codex round 3) Claude-Session: https://claude.ai/code/session_01A3rXV7GDLAhBe9mL4V6z2m |
||
|
xarmian
|
271174f2a5 |
fix(web): a server sync_required is answered after a per-tab random delay, so a fleet-wide coverage drop does not land every tab's resync on the database at once (BUG-2761) (#1517)
* fix(web): a server sync_required is answered after a per-tab random delay, so a fleet-wide coverage drop does not land every tab's resync on the database at once (BUG-2761) Item-change reconciles (items_bulk_updated, a collection change that rewrote items) and the first-connect sync stay immediate, per the lead's ruling on the item's trail. Claude-Session: https://claude.ai/code/session_01A3rXV7GDLAhBe9mL4V6z2m * test(BUG-2761): pin the fold at the first signal's delay, and read needsSync after clearing the connect's own flag Claude-Session: https://claude.ai/code/session_01A3rXV7GDLAhBe9mL4V6z2m * fix(BUG-2761): relay the spread as a flag on the existing envelope, so an older-bundle peer answers at once instead of dropping the signal (codex round 1) Claude-Session: https://claude.ai/code/session_01A3rXV7GDLAhBe9mL4V6z2m * docs(BUG-2761): onSyncRequired describes the mid-stream signal this client actually receives (codex round 2) Claude-Session: https://claude.ai/code/session_01A3rXV7GDLAhBe9mL4V6z2m |
||
|
xarmian
|
c50f0585ad |
fix(collab): WebSocket dials get their own per-user bucket and a per-user socket cap, and the editor jitters its reconnects (BUG-1308) (#1499)
* fix(collab): WebSocket dials get their own per-user bucket and a per-user socket cap; the web provider jitters its reconnects (BUG-1308) A collab dial drew on the general per-user API bucket (burst 60), so 100 dials from one user got 60 through, and a user's own dials and REST calls refused each other. Dials now draw on a CollabDial bucket (5/s, burst 50), and how many sockets a principal may hold is bounded separately by PAD_COLLAB_MAX_PER_USER (default 50), refused with 429 collab_limit_exceeded and Retry-After. Both defaults are sized from the reconnect-storm measurement on BUG-1308's trail (checkpoint 2). A browser cannot read a refused handshake's status, so the provider's backoff is what spaces a refused herd out: each step is now jittered to between half and all of itself. Claude-Session: https://claude.ai/code/session_01RYiqbqXmxQMoKHsGm8B77N * docs(collab): PAD_COLLAB_MAX_PER_USER and the collab dial bucket, in the deployment docs, samples and load-test usage (BUG-1308) Claude-Session: https://claude.ai/code/session_01RYiqbqXmxQMoKHsGm8B77N * fix(collab): only a WebSocket upgrade is a dial; per-user cap pinned within one workspace; dial-bucket keying stated correctly (BUG-1308, codex round 1) Claude-Session: https://claude.ai/code/session_01RYiqbqXmxQMoKHsGm8B77N * docs(collab): a PAD_COLLAB_MAX_PER_USER below 1 means unlimited, as the SSE knobs do (BUG-1308, codex round 2) Claude-Session: https://claude.ai/code/session_01RYiqbqXmxQMoKHsGm8B77N * docs(collab): the struct field agrees, 0 or less is unlimited (BUG-1308, codex round 3) Claude-Session: https://claude.ai/code/session_01RYiqbqXmxQMoKHsGm8B77N |
||
|
xarmian
|
7574c4b831 |
fix(events): Publish reports its outcome, and a failed activity publish is counted and logged (BUG-2732) (#1457)
* fix(events): Publish reports its outcome, and a failed activity publish is counted and logged (BUG-2732)
events.EventBus.Publish now returns an error under watchevents' three-way
contract: nil accepted, ErrBusClosed provably not published, anything else
unconfirmed. MemoryBus refuses after Close; RedisBus refuses once its root
context is cancelled and returns every Redis or marshal failure.
pad_eventbus_publish_total keeps its meaning (attempts). A sibling,
pad_eventbus_publish_failures_total{outcome="closed|unconfirmed"}, counts
failures; it is additive and pre-initialised at zero.
All 9 server producers publish through publishActivityEvent, which discards
the error (the write underneath committed) and logs outcome, event type and
workspace, never the payload, rate-bounded (burst 10, then 1/s, with a
suppressed count).
Pinned: an assigned-but-unpublished ID is NOT surfaced as a gap to a
subscriber or a resume, so the counter and the log are the only trace of a
lost event.
Claude-Session: https://claude.ai/code/session_01RYiqbqXmxQMoKHsGm8B77N
* docs(metrics): the failures counter moves during a Redis outage, not only at shutdown (BUG-2732, codex round 1)
Claude-Session: https://claude.ai/code/session_01RYiqbqXmxQMoKHsGm8B77N
* fix(events): the bus no longer logs each failed publish itself; the caller logs it, bounded (BUG-2732, lead review)
Claude-Session: https://claude.ai/code/session_01RYiqbqXmxQMoKHsGm8B77N
|
||
|
xarmian
|
033615ac94 |
fix(metrics): bound the MCP tool label and the HTTP method label, which callers could fill with invented values (BUG-2817) (#1456)
* fix(metrics): bound the MCP tool label and the HTTP method label, which callers could fill with invented values (BUG-2817) Two metric labels took caller text verbatim, and each new value minted a new Prometheus series. - pad_mcp_tool_calls_total and pad_mcp_tool_call_duration_seconds (tool): the value is params.name or the JSON-RPC method, recorded even for rejected and denied calls. It is now bounded by mcp.Server.IsKnownCallName, which asks the LIVE server for registered tools and adds the protocol methods. Anything else is recorded as "unknown". The predicate is injected (SetMCPMetricsCallNames) because internal/mcp imports internal/server. Unwired, every name is "unknown". The audit row keeps the full value. - pad_http_requests_total, _request_duration_seconds and _response_size_bytes (method): raw r.Method, recorded by middleware that runs AHEAD of auth, and Go accepts any token as a method, so an unauthenticated caller could mint series. Measured: 50 invented methods gave 50 series. A method outside the standard set is now "OTHER". The checkpoint-1 population on BUG-2817's trail enumerates all 14 labelled metrics and their 22 label slots; the other slots take constants or bounded values. Claude-Session: https://claude.ai/code/session_01Vf8uZmYdsrQqJtbAmjUFSX * fix(mcp): every mcp-go method constant is a known metrics call name (BUG-2817, codex round 2) The protocol set was a hand-picked subset, so a legitimate method such as notifications/tools/list_changed would have been counted as "unknown". It is now every method constant mcp-go v1.1.0 defines (34). Each is a library constant, so the label stays bounded. Claude-Session: https://claude.ai/code/session_01Vf8uZmYdsrQqJtbAmjUFSX * test(mcp): pin all 34 protocol methods as literals, so a dropped entry fails (BUG-2817, codex round 3) Claude-Session: https://claude.ai/code/session_01Vf8uZmYdsrQqJtbAmjUFSX * fix(server): the MCP metrics bound is a required SetMCPTransport argument, and the method test covers all nine standard methods (BUG-2817, codex round 4) A separate setter could be left uncalled with every test still green, so the predicate is now a parameter of the call that mounts the transport. nil fails closed. The HTTP test now keeps each of the nine standard methods as its own value, so mapping any one of them to OTHER fails it. Claude-Session: https://claude.ai/code/session_01Vf8uZmYdsrQqJtbAmjUFSX * test(server): the bounded-label test passes its predicate through SetMCPTransport, so a broken assignment there fails it (BUG-2817, codex round 5) Claude-Session: https://claude.ai/code/session_01Vf8uZmYdsrQqJtbAmjUFSX |
||
|
xarmian
|
d416a0bc1a |
docs(events, watchevents): a rejected automatic re-SUBSCRIBE is a documented limit, with the operator action (BUG-3155) (#1453)
Claude-Session: https://claude.ai/code/session_01RYiqbqXmxQMoKHsGm8B77N |
||
|
xarmian
|
ef6f57e4b0 |
fix(watchevents, server): the watch stream refuses a subscriber when the instance has no watch subscription, as the activity stream does (BUG-2800) (#1451)
* fix(watchevents, server): the watch stream refuses a subscriber when the instance has no watch subscription, as the activity stream does (BUG-2800) Claude-Session: https://claude.ai/code/session_01RYiqbqXmxQMoKHsGm8B77N * docs(deployment): both streams refuse a subscriber the instance cannot deliver to (BUG-2800, BUG-2799) Claude-Session: https://claude.ai/code/session_01RYiqbqXmxQMoKHsGm8B77N * docs(deployment): say why nothing revives a watch subscription that failed at startup on phase 1 (BUG-2800) Claude-Session: https://claude.ai/code/session_01RYiqbqXmxQMoKHsGm8B77N |
||
|
xarmian
|
1933041027 |
fix(events): surface the SUBSCRIBE error and refuse callers instead of admitting a dead stream (BUG-2764) (#1215)
* fix(events): surface the SUBSCRIBE error and refuse callers instead of admitting a dead stream (BUG-2764) * fix(watchevents): surface the SUBSCRIBE error at construction and on resubscribe (BUG-2764) * docs(events): the idle-cycle prose and metric Help now name the third install-nothing reason (BUG-2764) * fix(events): retry uncovered workspaces, wait on in-flight records, refuse after Close (BUG-2764 codex round 1) * fix(events): post-loop check waits on an in-flight record before trusting a live entry (BUG-2764 codex round 2) * test(watchevents): assert what the failed subscribe leaves, not how long it takes (BUG-2764 codex round 3) * docs(events): prose says what the code guarantees — delivery failure or shutdown, replacement refusals do not count as cycles (BUG-2764 codex round 4) * test(server): the 503 mapping is asserted on both subscribe branches; docs scope the refusal to the activity stream (BUG-2764 codex round 5, BUG-2800) * fix(events): uncovered-workspace retry logs are quiet once the bus is closing (BUG-2764 codex round 6) * fix(events): the uncovered-retry log promises a retry only while subscribers remain (BUG-2764 codex round 7) |
||
|
xarmian
|
c73584088f |
fix(watchevents): detect a half-open Redis connection with a bus heartbeat (BUG-2769) (#1199)
* fix(watchevents): detect a half-open Redis connection with a bus heartbeat (BUG-2769) internal/watchevents had the same defect as internal/events did, by the same mechanism: ChannelWithSubscriptions on a connection whose go-redis health check only writes. PubSub.Ping calls writeCmd and returns without reading a reply (v9.22.0), so a route that stops carrying traffic without closing is invisible — the instance blocks on a read forever while its replay buffer goes on looking complete. Named as a class sweep in BUG-2738's filing and deferred there. It became load-bearing when that unit shipped: docs/deployment.md told operators the gap was "closed on the activity stream and still open on the watch stream". This diff falsifies that, which is why the prose sweep is part of it. THE PORT IS SMALLER THAN THE ORIGINAL BY DESIGN. This bus holds ONE process-wide subscription created in its constructor, off any request path, so none of BUG-2747's establishment machinery exists to interact with: no per-workspace map, no establishment record, no single-establisher wall, no concurrency cap, no bounded-parallel recovery, and no per-workspace cycle scoping. Cost is flat too — one frame per instance per interval regardless of workspace count. NO COMPANION COUNTER, and that was CHECKED rather than inherited. internal/events needs pad_event_subscription_cycled_total because its dropWorkspaceCoverage returns early when a workspace has no buffer, so the reset reason under-reports the early-wedge case. dropCoverage here has no such branch: it replaces the buffer and reports unconditionally, so idle_timeout is a complete count on its own and a second metric would be a number needing to be explained against its neighbour for no signal. THE RECEIVE LOOP NOW OWNS ITS SUBSCRIPTION AND CONTEXT. A cycle replaces the subscription under a running bus, and the loop reading the old one must tell "I was replaced" from "the client died" — the second logs an ERROR and moves a counter documented to mean the instance has gone deaf. The cycle cancels that loop's own context before closing its PubSub, so it leaves by the quiet door. Its own test. I PORTED A FLAW ALONG WITH THE STRUCTURE, and the wiring test caught it: both maintenance halves shared one kick channel, so whichever goroutine was waiting consumed it and the other stayed on the stale cadence. internal/events' mutation matrix found exactly that (M11c) and fixed it; the fix did not come across. That is the contamination hazard this port's grounding warned about, in its literal form, caught by the CONVE-19 test rather than by review. Two more found by mutation, both missing tests rather than missing code: nothing asserted that ordinary traffic keeps the instance alive (removing the per-frame stamp survived, because every other test drives idleness through the clock), and nothing asked for a SECOND detection (a replacement inheriting stale stamps gives a detector that works exactly once, which is worse than one that never runs because it looks like it works). The second needed a direct assertion on the install stamps, because the behavioural route re-stamps the field it was meant to be testing. Trio in one commit as required: reason enumeration, the pad_watchevents_sequence_resets_total Help string, and docs/deployment.md — plus the two BUG-2738 sentences this falsifies and a new section explaining how the watch bus differs from the activity one. Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X * fix(watchevents): fence stragglers and re-validate before the drop (codex r1) Three findings, and two of them are BUG-2738 fixes I again failed to bring across with the structure. That is now three times in one port: the shared kick channel, the stale idle decision, and the missing generation. The mechanism is the same each time — I ported what the code DOES and not what its review history taught it, and each was caught by a test or a reviewer rather than by me reading the source I was copying. STALE IDLE DECISION. cycleIfIdle decided under one lock and tore down under another; a heartbeat or notification arriving between them left a demonstrably alive subscription being dropped and every client on the instance resynced for nothing. BUG-2738 fixed exactly this at its round 11. Re-validated immediately before the drop, with a positional seam so a test can land the recovery inside the window rather than racing it. NO GENERATION FENCE. Cancelling a receive loop and closing its PubSub does not JOIN the goroutine, and go-redis's channel is buffered, so a frame from a replaced subscription could still stamp the replacement's liveness, append to its buffer, or drop its coverage. On a wedged route that is the worst direction: the dead connection's buffered tail suppressing the detector for its successor. One check at the top of the frame handler covers all three, because the three must agree about whether a frame belongs to the live subscription. The probe stamp is fenced separately, since a slow publish can outlive the subscription it was sent for. A COPIED COST PARAGRAPH THAT CONTRADICTED ITS OWN SECTION. The activity bus's "each workspace has its own subscription, N frames per interval" text sat below the new watch-specific section saying the opposite. Retitled and moved above it. FOUR INSTRUMENT DEFECTS ON THE WAY, all found by mutation: - Nothing asserted ordinary traffic keeps the instance alive — every other test drives idleness through the clock, so removing the per-frame stamp survived. - Nothing asked for a SECOND detection, so a replacement inheriting stale stamps gave a detector that works exactly once — worse than one that never runs, because it looks like it works. Needed a direct assertion on the install stamps, since the behavioural route re-stamps the field under test. - The generation tests asserted the PREDICATE, not that the loop calls it. - And that wiring test could not discriminate on a frozen clock, where a stamp writes the value already there. It advances the clock first now. Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X * fix(watchevents): make the generation fence atomic with what it guards (r2) Two P1s, both mine, both the same shape: a check in one lock acquisition and the mutation it guards in another. THE FENCE WAS NOT ATOMIC WITH ITS MUTATIONS. One check at the top of the frame handler read well and guarded nothing reliably — a replacement between that check and stampLastSeen / fanOutFromRedis / dropCoverage let a straggler through to any of them. The generation now travels TO each mutation and is re-checked under the same lock that mutates. A stale notification entering the replacement's buffer is the worst of the three: it makes the instance vouch for a span it never received, which is the false coverage claim this whole family exists to remove. THE OLD GENERATION STAYED CURRENT ACROSS THE REPLACEMENT. subGen was incremented only after the new subscription was confirmed, leaving the cancel, the close, the dial and a round trip during which the OLD generation still passed every fence. Retired at teardown now, so during resubscribe NO generation is current and a late frame is ignored everywhere. That also makes the failure path honest: the "no notifications until restarted" log was false — no generation is current, so the next idle tick tries again. Revalidation and the drop are now ONE critical section rather than two, for the same reason at one level down: a frame arriving between them was silently discarded by a drop already decided on. Also: phase 1 no longer starts the maintenance goroutines, and the watch bus's phase is logged at startup — an operator cannot read an absence of idle_timeout without knowing whether the detector was running, and the two flags are independent. DOCS still described the workspace model in the section that claims to cover both buses: one heartbeat "per subscribed workspace", a phase table naming only PAD_EVENTS_HEARTBEAT, and coverage described as a workspace's. Generalised. Two more instrument gaps, both found by mutation: nothing asserted a straggler cannot enter the replacement's BUFFER (only the stamp was covered), and the phase-1 goroutine gate is untested by design — removing it changes no behaviour, only goroutine count, and the only assertion is a census that would be flaky here. Said out loud rather than left to look like coverage. Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X * test(watchevents): prove each generation fence on its own Round 3's fix put a generation check in each of the four places a frame from a replaced subscription can mutate shared state, rather than one check at the top of the receive path — a check in one lock acquisition and a write in another is a TOCTOU, which is what codex blocked. Four checks means four mutations, and the matrix found the first pass of tests could not tell them apart: removing the append's check, or the coverage drop's, left every test green. Not because the guards were redundant — because no test drove those paths with a stale generation. The straggler tests all enter through fanOutFromRedis, whose own guard returns early and hides the one below it, and nothing at all drove dropCoverageForGen with a straggler. So the fences are asserted one at a time, each through the entry point that actually reaches it: epoch bookkeeping fanOutFromRedis with a foreign epoch — the loudest of the four, since an accepted straggler would rewrite the id space and resync every client on the instance buffer append fanOutLocally directly, under the guard above it coverage drop dropCoverageForGen, previously undriven liveness stamp stampLastSeen, which would otherwise let a dead socket's traffic hold detection open Each fails against removal of the single check it names (M16/M17/M19 and the existing stamp mutation), and the four together still pass the end-to-end straggler tests unchanged. Refs BUG-2769 * test(metrics): prove the two new watch signals reach the registry Both were wired and neither was asserted at the metrics layer, which is where docs/deployment.md's claims about them actually live. A reason or a callback that never reaches the registry is a runbook pointing at a series that does not exist, and nothing in internal/watchevents can catch that — its observer is an interface, satisfied by a test double. pad_watchevents_heartbeat_publish_failures_total incremented six times, a count no other assertion in that test uses, so a callback wired to the wrong counter cannot land on the right number by coincidence. Fails when the increment is pointed at a neighbour. sequence_resets_total{reason="idle_timeout"} asserted with the literal label, alongside the four spellings already pinned there and for the same reason BUG-2739's rename left that test behind. Fails when the constant drifts. Also corrects the shared "what happens if you run them out of order" paragraph, which moved under a heading covering both buses while still describing only one: it said the frame travels on "the workspace's event channel" and that an un-upgraded instance resyncs "for every workspace", neither of which is the watch bus, where there is one channel and one buffer per instance. The blast radius differs in scale between the two and the paragraph now says so. Refs BUG-2769 * docs(watchevents): correct three counted claims that stopped being true All three said "three" where the code now has four, and each was accurate when written — the fourth fence (the epoch bookkeeping in fanOutFromRedis) was identified after them, in the pass that found the matrix could not tell the guards apart. That is the whole failure mode: a count is a claim, and a claim written before the last change is wrong afterwards with nothing to notice it. Two of the three sat inside a comment ABOUT how carefully the guards were enumerated, and one names them now instead of counting them, so the next site added has to appear in the list or contradict it visibly. Found by sweeping the branch diff for counted prose rather than by rereading, which is what had already missed them twice. Refs BUG-2769 * test(config): close the other half of the two-flag independence claim The flag tests asserted PAD_WATCH_HEARTBEAT does not move EventsHeartbeat and stopped there, while the comment above them and the deployment doc both claim the two buses roll INDEPENDENTLY. That is a biconditional and one leg does not establish it: a Load() that pointed PAD_EVENTS_HEARTBEAT at both fields passed everything. Now both directions are asserted, and the events leg checks its own premise first, so a fixture that stopped setting the flag fails as a fixture rather than as a pass. Also pins env-over-file precedence for the watch flag, in the direction that actually matters: PAD_WATCH_HEARTBEAT=false over watch_heartbeat=true in config.toml. That is the rollback for a bad phase-2 flip, and an operator reaching for it mid-incident cannot be editing a file on every host. Mutation matrix, each detected: the env var wired to the neighbouring field, the env var never read at all, and the toml tag dropped. Refs BUG-2769 * test(watchevents): fix five tests that passed for the wrong reason Codex round 4 went at test honesty rather than correctness and found no BLOCK, but it found five assertions that hold whether or not the thing they name works. Each is now driven through the path it claims, and each was mutation-checked against the specific defect it exists to catch. the malformed-frame contract only ever called isWatchHeartbeat. The predicate can be perfect while the receive loop routes every "hb|…" payload to the ignore arm without asking it, which is the defect, and the test's name promises coverage ends — a claim about the loop. Now published on the real channel, with a well-formed frame as the control so the assertion cannot be satisfied by a loop that finds everything undecodable. the receive-loop wiring test published, slept 300ms, and asserted nothing had changed. A loop that stalled or never started satisfies that perfectly. There is no natural signal to wait on instead, because a frame the fence refuses is by design invisible — hence a seam that fires after the loop handles a frame whichever arm it took. Bounded, so a stalled loop fails with a message rather than a package timeout, and followed by a control that the same loop still accepts a frame whose generation matches. the quiet-exit test asserted only that no loud exit was reported, which a replaced goroutine that never exits at all also satisfies — a leak, and the worse outcome. Now joins the loop first via a process-wide live-loop count, then checks the counter, so it is a statement about a goroutine that has finished. the maintenance-loop wiring test claimed both halves and observed a heartbeat, which a loop that started only the publisher passes. The idle half cannot be proved there at all: against a live miniredis this bus's own heartbeats come back and refresh liveness every cadence, so wedging it with the loop running is a race against the publisher — which is what my first fix for this turned out to be, flaky at 2 in 3. Renamed to what it proves, pointing at the blackhole end-to-end test, which drives the scanner for real and detects both mutations. the straggler test never delivered a straggler. It incremented subGen by hand, called isCurrentGen, and compared an unchanged timestamp without touching a mutation path — green with every fence removed. Deleted rather than repaired: the four-way per-fence test added earlier covers it properly, and isCurrentGen went with it. Plus two ordering changes in Close/resubscribe that ARE NOT fixes for an observed race, and say so in the test. Making b.pubsub reassignable made Close's unlocked read of it look wrong, and resubscribe's wg.Add outside the lock look like it could land after Close reached Wait. Both windows turn out to be shut already by resubscribe's b.closed check, which sits under the same acquisition as the count — reverting either fix leaves the new Close-during-cycle test green. Kept as defence because the invariant they lean on is three functions away, and documented so nobody later reads them as evidence of a bug that existed. Also corrects the metric help and two comments that said an idle cycle "replaced the connection" when it attempts a replacement that can fail; the deployment doc already said attempted. And the deployment doc's rollback, frame-validation, what-to-watch and startup-log paragraphs, all of which moved under a heading covering both buses while still describing only the activity one. Refs BUG-2769 * refactor(watchevents): drop an always-empty return and the branch reading it dropCoverageIfStillIdle returned (string, bool) where the string was never anything but empty — the reset it reports goes out through the pending/flush path inside the lock, so the caller's `if report != ""` was unreachable. A second reporting path that exists in the signature and never fires is a thing a later change wires up by accident. Refs BUG-2769 * fix(watchevents): a failed re-dial retries without re-dropping coverage Codex round 5, on behaviour across a full Redis outage. No BLOCK; this was its one P2 and it is real. The probe-failure suspension does not cover this case, and the reason is worth stating because the suspension looks like it should. Suspension asks "did our last probe get through", and that can be YES with the route already gone: the last successful publish stamps lastProbeOK, Redis dies before that frame comes back, and lastSeen stays behind it. From there both timestamps are frozen — the probe fails so nothing stamps lastProbeOK, nothing arrives so nothing stamps lastSeen — and the cycle's precondition stays true for the whole outage. Every pass then dropped coverage, announced to every subscriber, and re-dialled. Only the re-dial is owed. The second drop empties an already-empty buffer and re-announces a hole every subscriber has been told about, and it moves pad_watchevents_sequence_resets_total{reason="idle_timeout"} once per cadence — so a five-minute outage read as ten incidents on the series operators are told to alert on. cycleIfIdle now has a retry-only arm ahead of the decision, entered when there is no subscription at all, and the teardown clears b.pubsub / b.subCancel so that state is representable. Clearing them also stops Close closing an already-closed PubSub a second time. Two tests, discriminating in OPPOSITE directions, because the obvious fix for the noise is to suspend the pass and that would trade a noisy outage for one the instance never returns from — retrying the dial IS the recovery path: three passes with Redis away one reset, not three Redis returns after a failed pass the subscription is re-established and the counter does not move again Matrix: removing the retry arm, making it return without retrying, and leaving the torn-down subscription in place are each detected, the middle one only by the recovery test. internal/events has no equivalent defect. Its teardown deletes the workspace's subscription entry, so its next scan finds nothing live and abandons; recovery there runs off the request path. Refs BUG-2769 * fix(watchevents): only one caller may install a replacement subscription Codex round 6, verifying round 5's fix. No BLOCK; this was its P2. Both the cycle and its new retry arm dial with the lock RELEASED, which is deliberate — a Redis round trip under the bus's hot mutex would stall every fan-out on the instance — so two passes can each find no subscription and each dial one. Installing both is wrong twice over: two receive loops would run on the SAME generation, so both accept every frame and each notification is processed twice, and the loser's PubSub would be untracked, closed by nothing including Close. The install is what needs serialising, not the dial, so the loser discards its own connection under the lock rather than the two racing to overwrite b.pubsub. Only the idle scanner calls this today, so this guards an invariant rather than fixing an observed fault. Written down because the invariant lives in a different file from the code relying on it, and because the failure is silent duplication rather than a crash. The test races two resubscribes through the install seam. Two details it needed, both found by running it rather than reading it: the loop count is incremented INSIDE the goroutine, so sampling it right after the constructor returns reads zero — the first version did, and measured every later count against that wrong baseline. It waits for the loop now. the seam release is deferred, because without it the guard's mutation parks both callers in the callback, Close waits on receive loops that cannot start, and the detection arrives as a package-wide hang with no message. That is how the mutation first appeared to pass. Also completes the idle_timeout reason in three comment/help sites that still enumerated four reasons and said "the last two" — the same stale count corrected in the observer contract earlier on this branch, missed in its neighbours because I fixed the one the reviewer named instead of grepping for the claim. Refs BUG-2769 * test(watchevents): count installs instead of waiting for one that never comes Codex round 7 returned no BLOCK and no P2 on the production code, and two NITs on what round 6 added. Both are real. The concurrency test synchronised on a WaitGroup expecting BOTH callers to reach the install seam. Only the winner does — that is the property under test — so in the passing case the goroutine waiting on it blocks forever. A leak inside a test written to prove a leak does not happen is not a shape to leave standing. An atomic the abandoning caller never touches carries the same information and blocks nobody, and it removes the release channel and its deferred close along with it. The final assertion also moved off liveReceiveLoops and onto that count. A loop starts AFTER its install, so reading the loop count can catch a second caller's goroutine before it has begun and see the passing value on a failing run. Both callers have returned by the time the install count is read, so it is final. Detection over ten runs with the guard removed: 10/10, where the loop-count version was a race against a goroutine's first instruction. Also softens the retry arm's log line. It said the instance receives no notifications until an attempt succeeds, which is true for today's single scanner and stale the moment there are two: one caller's dial can fail while another has already installed. It now claims only what the failing call knows. Refs BUG-2769 * test(watchevents): hold both callers at the window, and say what that misses Codex round 8's P2, on the test the previous commit rewrote. Starting two goroutines from a start gate makes overlap likely and guarantees nothing: one can finish resubscribe before the other begins, so the window the install guard closes need never have been open. A seam at the dial/install boundary — connection dialled, lock not yet taken — lets both callers announce their arrival and wait for each other. Now the window is open by construction rather than by luck, and the test fails as a fixture if only one caller ever reaches it, instead of passing on evidence it never gathered. AND IT STILL DOES NOT DETECT EVERYTHING, which the test now says in place of leaving it implied. Measured: guard removed entirely 10 runs, 10 detected guard checked in its own acquisition, then 10 runs, 0 detected the lock retaken to install The second is the regression round 8 asked about, and catching it would mean landing the second caller inside a check-to-install gap that exists only in the mutant — there is nothing to yield on there, and no seam can be placed in code that is not written. So this test covers "a guard exists", not "the guard is in the right critical section". The latter is held by the comment at the guard and by review, and a test comment claiming otherwise would be worth less than the honest note. Refs BUG-2769 * fix(watchevents): make the frame seam and the cycle log tell the truth Codex round 9 was asked whether this should merge and said hold for a cleanup pass. Five findings, no correctness blocker, and every one of them a claim that had stopped matching the code. the frame seam did not fire for every arm, though its comment said so. The arms that decline to act — a heartbeat, an undecodable payload, an unsubscribe confirmation — were `continue` statements, which skipped everything after the switch. A test waiting on the seam for one of those frames would have HUNG rather than failed, which is the worst way to find this out. The switch is now its own method so every arm ends the frame by returning, and a test drives one frame per publisher-reachable arm and counts three. Detected against restoring the skip. the idle-cycle warning was emitted before the revalidation that can abandon the cycle, so it could announce coverage ending and resumes answering sync_required for a subscription that was then left alone — a log line with no counter behind it, and an on-call hunting a bug that is not there. internal/events learned this at its own round 6; the reason did not come across with the port. Moved after the decision is final, still saying "attempting" to replace because the resubscribe can fail. the quiet-exit test sampled liveReceiveLoops instead of waiting for it, so its "the replaced loop left" assertion could be satisfied by a loop that never ran. Same defect fixed in the sibling concurrency test a commit earlier and missed here, because I looked at the test the reviewer named rather than at the pattern. Latent rather than observed: sampling survives 10 runs, so this removes a possibility. the probe-failure log and metric help called an errored Publish a failure to publish. A returned error can also mean the reply was lost after Redis accepted the frame, so the honest claim is that the probe is UNCONFIRMED. It changes no behaviour — an unconfirmed probe is not evidence about the receive path either, so detection suspends the same way — but an operator reading the counter should not be told more than the instance knows. the deployment doc said the watch stream differs in "three things" and listed four, the fourth being the bullet I added last round. Third instance of that species on this branch; the count is gone rather than corrected. Refs BUG-2769 * docs(watchevents): stop one unconfirmed probe standing in for a broken path Codex round 10 confirmed four of round 9's five fixes and held the fifth as partial. It was right on all three residual sites. Renaming the condition to "could not confirm" did not fix the sentences downstream of it. The log still said silence cannot be read as a finding "when we could not ask" — but we may well have asked, and lost only the answer. And both the metric help and the observer contract said an instance in this state "is also failing to deliver its own notifications to every other instance", which is a conclusion about the outbound path drawn from a single call that did not come back. The inference is sound at a SUSTAINED rate and worthless at one increment, so both now say which is which. That distinction is the whole value of the counter to an on-call: a blip is a lost reply, a rate is a broken path, and the same wording for both makes the first look like the second. No behaviour change. An unconfirmed probe suspends detection exactly as a definite failure does, because it is not evidence about the receive path either way. Refs BUG-2769 * docs: sweep the BUG-2738 prose this change makes false BUG-2738 shipped documentation that describes the watch stream as still carrying the half-open defect. Merging this makes those sentences wrong, and I flagged the sweep as owed twice during the groundwork and then did not do it — the lead caught that the package said nothing about it. Five sites, each re-read after editing rather than grepped for, because grepping for a phrasing I chose is how I have twice verified a sweep that had not landed: the residual enumeration opened "One gap remains everywhere, and a second remains on the watch stream only", then described one gap and said it was open on both. The second WAS the half-open case. Now states one gap, on both streams, and says where the second went. the half-open paragraph already said "closed on both streams" — the one site I had fixed — but omitted that each half is behind its own phase-2 flag, so a reader takes it as closed on their deployment when it is closed only once they turn it on. "A third residual" counted the item it followed. With the second gone the ordinal was wrong; it does not need one. "these two gaps" in the closing sentence, same arithmetic. the pad_event_subscription_cycled_total row told an operator to read heartbeat_phase off the startup log. There are now two such fields on two lines under two flags, and only one bears on that counter. It names the line. No code change; suite 28/28 and lint 0 re-run because the branch is under review and a docs commit that skips them is a commit nobody checked. Refs BUG-2769 |
||
|
xarmian
|
effd0199cd |
fix(events): detect a half-open Redis connection with a bus heartbeat (BUG-2738) (#1195)
* fix(events): detect a half-open Redis connection with a bus heartbeat (BUG-2738)
A Redis connection can stop carrying traffic without closing -- no FIN, no
RST, just a route that stopped working. The instance blocks on a read that
never returns, receives nothing, and its replay buffer goes on looking
complete, so every resume is answered "caught up" from a coverage window that
ended when the route did.
go-redis cannot see it: PubSub.Ping writes the command and never reads a
reply (v9.22.0), so its health check reports healthy for as long as the socket
accepts writes. Measured on day-52 against a proxy that silently stopped
forwarding: no reconnect in 24 seconds.
Each subscription now records when it last received ANYTHING -- event,
heartbeat, or subscription acknowledgement -- and a background pass ends the
coverage of any workspace whose stamp goes stale past 3T, then REPLACES the
connection. Drop alone would not recover: the resync it demands is served from
the same dead socket, so the detector fires again on the next pass.
Dave's day-49 ruling dissolves the threshold rather than tuning it. The bus
publishes its own frame every T=30s and fires at 3T=90s, which turns "is this
workspace quiet or is the route dead?" -- unanswerable, deployment-dependent --
into "did our heartbeat arrive?".
TWO PHASES, ORDER NOT OPTIONAL. The frame must travel on the workspace's event
channel, because that connection is what needs proving. A pre-phase-1 binary
cannot classify it: the frame reaches the event decoder, fails, and since
BUG-2739 that is a hole in coverage -- so an early flip makes every un-upgraded
instance drop its buffer and resync all its clients, every 30s, per workspace,
for the length of a mixed deployment. Phase 1 recognises and ignores;
PAD_EVENTS_HEARTBEAT is phase 2, a constructor parameter with no default so
every call site states its phase.
The idle detector is a THIRD actor in a region whose invariants were designed
around request goroutines plus Close. Four rules, each commented at
cycleIdleSubscriptions and each with a test:
1. It refuses to cycle while pendingSubs holds a record, and MINTS the
record itself before tearing anything down -- subscribeAndReplay checks
pendingSubs before wsSubs, so a subscriber arriving mid-cycle joins the
replacement instead of being admitted into the doomed subscription.
2. lastSeen is stamped at INSTALL, not left at the zero value, which reads
as 1970 and would cycle hardest on an unconfirmed admission -- the
workspaces already having a bad time.
3. wsCounts is re-read under the lock that performs the teardown.
4. Re-establishment runs on b.ctx with a nil establisher; the bus has no
subscriber registration of its own to unwind.
Two decisions beyond the plan:
A NEW COUNTER, not just the reset reason. dropWorkspaceCoverage reports a
reset only when a buffer existed to drop, and the incidents this detector
exists for skew hard toward having none -- a route that wedged early on a
quiet workspace. Reading cycles off the reset label alone would under-report
exactly the case it was built to find, so pad_event_subscription_cycled_total
is the dependable count and idle_timeout is corroboration. Both comments say
which is which.
THE CADENCE IS A LIVE TUNABLE -- a timer re-read under b.mu each pass plus a
buffered kick, not a ticker constructed once. A ticker captures the interval
at goroutine start, which makes the field write-once while its comment calls
it a tunable and makes any later write a data race; it also leaves no
deterministic way to test the WIRING other than a test-only constructor.
decodePayload's signature grew a payloadKind. The classification belongs to
the decoder, not the call site, so no future caller can reintroduce the
coverage drop; and the prefix (rather than an exact payload) means a later
frame version needs no third roll.
Also swept, per the team's prose convention: receiveMessages' doc comment and
deployment.md both said this gap was open and needed a decision. Both now say
what closes it -- and deployment.md says the watch stream still has the same
defect by the same mechanism, which is its own unit.
Trio kept together: ResetReasonIdleTimeout, the metric Help strings, and
docs/deployment.md's rollout order with the mixed-fleet failure named.
Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X
* test(events): rebuild the instruments the BUG-2738 matrix showed were blind
The mutation matrix found a defect in the fix itself and three tests that
could not have caught what they were named for.
THE DEFECT: the idle scan skipped a subscription whose lastSeen was the zero
value. That reads as belt-and-braces beside the install-time stamp and is the
opposite -- it makes a subscription that has NEVER received anything
permanently uncyclable, which is the BUG-2747 unconfirmed admission: the one
population the plan singles out as mattering most, and the one where a wedged
route would then be undetectable forever. It was also masking rule 2: with the
skip present, removing the install stamp survived every test. Skip removed;
that mutation is now caught. Re-adding it is undetectable by construction and
the comment says so, because a guard that only acts once a real one has broken
converts a caught defect into a silent one.
THREE INSTRUMENTS THAT WERE NOT MEASURING:
- "Drop only, never cycle" passed because establishSubscription overwrites
wsSubs, so a generation check cannot see a replacement installed WITHOUT
tearing the old connection down -- a leaked PubSub, connection and receive
goroutine per cycle, forever, on exactly the wedged route where they never
die on their own. Now asserted on the receive loop exiting.
- The Close test was vacuous. Close drains wsSubs, so a loop that ignored
b.ctx entirely would find no workspaces and publish nothing: silence after
Close was evidence of nothing. maintenanceStopped makes the goroutine's exit
observable, which is the same reason Observer.ReceiveLoopExited exists.
- The joint test HUNG rather than failing under the drop-only mutation: the
seam never fires, so the joiner goroutine was never spawned and an unbounded
receive waited forever. The harness then aborted mid-run and LEFT THE
MUTATION APPLIED to the working tree, which a grep caught and a green test
run would not have. The wait is bounded and names the failure; the harness
bounds each run, reports a hang as its own status, and restores in a finally.
Added: a direct test that a straggler frame from a replaced generation cannot
refresh its successor's liveness -- on a wedged route, the dead connection's
buffered tail would otherwise suppress the detector for the replacement.
RULE 3 IS AN OPTIMISATION, NOT A CORRECTNESS GUARD, and the matrix says so
rather than an argument: removing the whole second read -- liveness, generation
and count terms together -- survives every test, because
establishSubscription's abandon path already refuses to install for an emptied
workspace and retires the record in the same critical section (BUG-2749). The
first read is redundant more sharply still: reaching zero takes the
subscription down with it, so this loop never sees such a workspace. Both are
kept, because neither DEPENDS on that coupling, and both comments now carry the
per-term reading instead of describing tested defence in depth. The generation
term is unreachable while the establishment record is held, by rule 1's own
mechanism.
Matrix: 16/22 detected, plus 4 follow-ups. Every survivor is documented at its
line with why it survives.
Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X
* fix(events): gate idle detection on heartbeat phase 2 (BUG-2738, codex r1)
Codex round 1 found a defect the first draft had shipped WITH A COMMENT
JUSTIFYING IT, plus two coupling hazards.
P2-as-filed, P1 in effect: idle detection ran on every instance from phase 1,
on the reasoning that it could "detect off whatever traffic the deployment
already carries". That holds only for a BUSY workspace. A QUIET one on phase 1
has no events and no heartbeat, so a perfectly healthy subscription crossed
the 90s threshold on every pass and was cycled: replay coverage dropped, every
live subscriber told to resync, indefinitely -- on the DEFAULT configuration
every deployment lands in before it flips anything. A resync storm shipped as
the default, by the feature whose stated purpose is to avoid exactly that load
inversion.
Publishing and detecting are now one switch, which is what they always were:
an instance detects off its OWN frames -- it publishes to the channels it
subscribes to and receives them back -- so it never depended on peers having
flipped, and there was never a reason for the two to be separable. Phase 1 is
"recognise the frame so a phase-2 peer costs you nothing", and nothing else.
Regression test plus its counterfactual, so "no cycles" cannot be satisfied by
a detector that has simply stopped working.
P1: the maintenance loop published heartbeats and scanned for idleness on one
goroutine. publishHeartbeats makes N synchronous Redis publishes, and against
the failure this feature exists to detect those are precisely the calls that
block -- bounded by go-redis's own Dial/Read/WriteTimeout, not by any context
we can pass. A stalled publisher could therefore delay detection for as long
as those timeouts take, on the very instance whose connections had wedged, and
for longer the more workspaces it carried. Two goroutines with their own kick
channels; a stalled publisher now just produces silence, which is what the
detector reads.
P3: the cycle held the workspace's establishment record across a synchronous
observer report, so an Observer callback that subscribed to that workspace
would wait on a record only the reporting goroutine could retire. Moved the
SubscriptionCycled report past establishment. The narrower half is older than
this code -- confirmSubscription's late-acknowledgement path already reported
from inside that window -- so it is documented on the Observer interface as a
contract rather than silently worked around: a callback may publish, read and
unsubscribe; it may not subscribe.
Prose swept for what the gate falsified, per the team convention: the
constructor comment that argued for the defect, config.EventsHeartbeat's
rollback paragraph, the config test's inverted-rationale comment,
ResetReasonIdleTimeout, both metric Help strings, and deployment.md's phase
table and rollback section. All of them now say that phase 1 detects nothing
and that the cycled counter is STRUCTURALLY zero there -- a zero on phase 1
says nothing about whether a route has wedged, which is the reading an
operator would otherwise get wrong.
Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X
* test(events): prove a resuming joiner is told sync_required across a cycle
Codex round 2 raised that a subscriber arriving DURING an idle cycle gets no
gap signal, because dropWorkspaceCoverage only signals subscribers present
when it runs. True, and for a RESUMING caller the gap signal is not what
protects it: the registration mark is. It registers while the workspace has no
buffer, so its mark cannot match whatever buffer exists by the time it reads,
and eventsSinceMarkLocked answers nil -- sync_required rather than a false
"caught up".
A FRESH caller is deliberately not signalled and the finding is DECLINED for
that case, with reasons recorded at the test: it holds no prior position, so
there is no span it could be missing; it is admitted only after the
replacement subscription is acknowledged, because it waits on the cycle's
establishment record which finishPending closes after the confirmation; and on
the unconfirmed-admission path it IS told to reconcile when the acknowledgement
lands. Signalling it anyway would demand a resync of a client with nothing to
reconcile -- the load inversion this unit already had to fix once.
THE FIRST TWO VERSIONS OF THIS TEST DID NOT DISCRIMINATE, which is the part
worth keeping. Version one asserted the empty case: the cycle leaves no buffer,
so eventsSinceMarkLocked returned nil from its `!ok` term and removing the mark
check entirely still passed. Version two published inside
afterSubscriptionConfirmed so a FRESH buffer exists before the joiner reads --
and deleting the `mark.buffer == nil` term still survived, because the keep
arithmetic in that function already reduces to zero for a nil mark. Only
replacing eventsSinceMarkLocked with the unmarked eventsSinceLocked fails the
test, handing the joiner the post-cycle event as though it followed its cursor.
That is the mutation the test is built against, and the redundancy inside
eventsSinceMarkLocked is recorded rather than mistaken for coverage.
Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X
* fix(events): only count a cycle that actually replaced the connection (codex r3)
Three findings from a fresh-angle round on shutdown, wire format and doc
accuracy. The wire-format angle came back clean -- events:<workspace> cannot
collide with watchevents under validated namespaces, and no valid activity
payload can be mistaken for an hb| frame.
P3, and the one that stings: config.EventsHeartbeat still said phase 1
"already runs idle detection off whatever traffic exists". That is the exact
sentence the previous commit's sweep existed to remove, in a file that sweep
edited. A grep for the phrasing I remembered writing missed the paraphrase
sitting four lines above the paragraph I did fix.
P3: SubscriptionCycled was reported unconditionally after establishSubscription
returned, but establishment has two reasons to install nothing -- the bus
closed, or the workspace emptied while we dialled. The counter's documented
meaning is "torn down AND replaced", and counting an aborted establishment is
wrong in the direction that matters: an operator reading a non-zero rate
concludes connections are being blackholed, so a shutdown would manufacture
that signal. Now reported only when a replacement is installed, verified by
generation. Both Help strings and deployment.md say "counts replacements, not
teardowns"; the teardown stays visible through the idle_timeout reset reason.
P2: Close does not join the maintenance goroutines. Kept that way and
documented on Close, because the publish half makes synchronous Redis calls
bounded by go-redis's own timeouts -- the calls that stall on exactly the
wedged route this feature detects -- so joining would let a dead network hold
shutdown open. What has to hold instead is that a cycle already past its ctx
check leaves nothing behind, which is now pinned by a test that closes the bus
from inside the cycle's establishment: no subscription installed, no
establishment record stranded, no counter moved.
liveGen moved from the test file into the package -- production needs it now.
Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X
* test(events): restore the coverage the phase gate silently removed
The mutation matrix, re-run against the post-codex code, showed M3 -- removing
the install-time lastSeen stamp -- going from DETECTED back to SURVIVED. The
cause was my own round-1 fix: gating idle detection on heartbeat phase 2 means
a phase-1 bus never scans, and TestAnUnconfirmedAdmissionIsNotCycledAsIdle
built its own phase-1 bus. It was the only test that could observe a zero
lastSeen, because the plain fresh-subscription case is stamped twice over --
at install, and again by the acknowledgement. Flipped to phase 2 and
re-verified: removing the stamp fails it again.
Worth naming the shape rather than just the fix. A behaviour change that
narrows when code runs silently narrows what the tests reach, and nothing in a
green suite says so -- the tests still pass, they just stopped asking. Only
re-running the matrix after the change surfaced it.
Two harness bugs fixed alongside, both of which had been reporting
non-results as if they were readings:
- A mutation that INSERTS keeps its own anchor, so the "did the edit land?"
check read every insertion as ANCHOR-ERROR. It compares the file now.
- The two rule-3 mutations left `sub`/`live` unused and came back BUILD-BREAK
rather than answering the question; they carry the same discard the
follow-up harness already used.
Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X
* test(events): close the wiring and barrier gaps codex round 4 found
Concurrency and lock discipline came back CLEAN -- the establishment record
and the generation checks cover two racing cycles, Unsubscribe, Publish and a
stale resubscription frame, with no lock-order deadlock. The four findings
were all about whether the tests measure what they claim.
P2, and it is the convention I had cited three commits earlier: the heartbeat
flip had no wiring test. internal/events proves a bus built with
publishHeartbeat=true emits frames and detects idleness, and every one of
those tests passes if newObservedEventBus hardcodes false -- the deployment
would simply never detect a wedged connection, which is indistinguishable from
a deployment that has none. Both directions asserted, because a helper that
ignored its config and hardcoded EITHER value passes a one-directional test.
Mutation-checked against exactly that edit.
P2: the metrics adapter test never touched SubscriptionCycled or the
idle_timeout reason, so an adapter that folded the counter into the reset
series -- destroying the very distinction those two are built to keep apart --
would have passed. Both added with counts that differ from their neighbours',
the pattern that file already uses so a label-dropping adapter cannot satisfy
the totals by coincidence.
P3: TestAHeartbeatConsumesNoEventID "waited" on a predicate that returned true
unconditionally. Not a slow wait -- no wait at all: the counter was read with
the publishes still in flight, so a heartbeat that DID consume an id could
land afterwards and the test would still pass. It now waits on the frames
arriving, and fails against a mutation that publishes an event alongside each
heartbeat.
P3: the maintenance goroutines started on phase 1, where both halves are
guaranteed no-ops -- two goroutines and two timers per process waking every
30s for the life of a deployment that asked for none of it, and phase 1 is the
DEFAULT. The flag is constructor-only so the decision is taken once. The
in-function gates stay: those are the correctness ones, and the tests reach
them directly without a loop.
Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X
* fix(events): validate the heartbeat frame and stop serialising recovery (r5)
Client-facing behaviour came back CLEAN: an idle cycle signals each local
subscriber, the SSE handler emits an in-band sync_required with an empty id
while holding the connection open, EventSource retires its cursor and the web
client runs the documented reconciliation. Two P2s on the other angles.
FRAME VALIDATION. Accepting any "hb|..." created a silently-ignored class on
the workspace event channel, where before this feature EVERY unreadable
payload ended coverage loudly and moved undecodable_message -- the counter
whose documented job is "suspect a namespace collision". A foreign or buggy
publisher whose bytes happened to start with the prefix slipped through that
signal without a trace. A frame is now hb|<version> plus optional short tokens
under a length cap; anything else wearing the prefix goes back to being a
coverage-ending decode failure, and the forward compatibility the prefix was
chosen for survives for a disciplined future frame.
What this deliberately does NOT try to fix, because it is not a hole: a forged
frame cannot fake liveness. Liveness means "this socket carried traffic", and a
frame that ARRIVES demonstrates exactly that whoever sent it -- which is why
stampLastSeen already fires for undecodable frames. There is no coverage claim
inside a heartbeat to forge.
CADENCE DRIFT, which was self-defeating rather than merely untidy. The timer
restarted after each pass, so the real period was T plus however long the pass
took. For the publisher that means an instance whose publishes are slow emits
heartbeats further apart, its own subscription sees them further apart, and it
can cross its own 3T threshold and cycle connections that were never wedged --
the slowness manufacturing the incident. Scheduling is deadline-based now, and
resets rather than bursting when a pass overruns badly.
SERIAL RECOVERY. One idle pass re-established every due workspace in sequence,
each re-dial bounded by go-redis's own timeouts, so recovery took N x that
timeout with the last workspaces reporting themselves uncovered throughout.
The failure that puts many workspaces on the due list at once is a Redis
failover, so the serial case was the common one. Bounded-parallel at 8 -- each
entry already owns its establishment record so they are independent by
construction, and an unbounded fan-out would answer a struggling Redis with one
dial per workspace at once. Test covers more workspaces than the cap, and
fails against a version that drops the overflow.
Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X
* docs(events): idle_timeout means coverage ended, not connection replaced (r6)
Codex round 6 came back clean on the non-Redis path (MemoryBus ignores the
Redis-only flag; EventBus and Close have not drifted), on the rollback
rehearsal (phase-2 to phase-1 and a mixed fleet are safe as documented,
including a bus mid-cycle -- Close cancels it, prevents installation and
retires its pending record), and on the operator surface
(PAD_EVENTS_HEARTBEAT is a server env/TOML setting; `pad configure` is client
connection config and needs no new surface).
The one finding is a contract drift I introduced two commits ago and then
wrote prose for in the same commit. Making SubscriptionCycled mean "replaced"
was right; what I missed is that the idle_timeout RESET REASON is emitted
earlier -- dropWorkspaceCoverage runs before the re-establishment -- so it can
fire when nothing is replaced, which is exactly the shutdown case the counter
was changed to exclude. Three doc sites and one log line said "replaced the
connection" anyway.
They now say what is true at the moment each fires: idle_timeout means
COVERAGE ENDED, only pad_event_subscription_cycled_total proves a replacement,
and the log says "attempting to replace" rather than "replacing". The log
wording matters on its own -- an operator correlating it with the counter
would otherwise find the log without the counter and go hunting a bug that
isn't there.
Third time this unit has produced prose the next change falsified, and each
time a different reviewer angle caught it rather than the sweep I ran at the
time. The pattern is that a behaviour change and the prose describing it land
in one commit, so there is no diff between them to notice.
Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X
* test(cmd): drive the heartbeat wiring test instead of sleeping at it (r7)
Codex round 7 found no leftovers across seven rounds of edits, and confirmed
the mass-cycle case does NOT produce a reconnect storm -- the SSE connections
stay open across a sync_required, so the admission limits are never consulted.
P3, and it is the failure I have been criticising in other people's tests: the
wiring test used a 300ms sleep as its ordering barrier. Under -race or on a
loaded CI box, a phase-1 bus that is correctly silent and a phase-2 goroutine
that merely has not been scheduled yet are indistinguishable, so the test could
pass or fail for reasons unrelated to the flip it exists to check. It now
drives one publish pass synchronously through a named test hook and uses an
ordinary event on the same channel as the barrier, which Redis delivers in
publish order. No timing left. Verified: still fails against the flag being
hardcoded false, and ten consecutive -race runs are green.
That replaces SetMaintenanceCadenceForTest with PublishHeartbeatsForTest rather
than adding to the exported test surface -- the loop's own wiring is covered
inside internal/events, where the unexported setter is available.
P2 is FILED, NOT FIXED, as BUG-2761: a mass coverage drop tells every connected
subscriber of every affected workspace to resync at once, and each browser tab
independently calls /changes with per-tab coalescing but no jitter and no
global budget. The fix is a web-client change plus possibly a wire-format hint,
which is independent of half-open detection and would materially expand this
diff. Worth filing rather than shrugging at because this unit makes the
simultaneous case MORE likely: it adds a third trigger of a class that already
existed (Redis failover, epoch change), and its natural cause is exactly a
network event that wedges many routes at once. deployment.md carries the
residual with the bug ref so an operator meets it before the incident does.
Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X
* test(events): make the tests prove what their comments claim (codex r8)
Round 8 was claim verification rather than bug hunting -- check the diff's
load-bearing assertions against the actual code -- and it was the highest-yield
round of the eight. The go-redis assertions (Ping writes without reading, the
channel path sets no read deadline, TLS dials ignore cancellation) and the four
claims about neighbouring functions all held. Seven other assertions did not.
TESTS THAT DID NOT PROVE THEIR OWN HEADLINE. This is the substance of the
round, and every one of these passed before and after:
- The JOINT TEST -- this unit's flagship -- claimed to discriminate the
two-subscriptions failure and did not. Fan-out is per subscriber, so a joiner
that opened its OWN second subscription still delivers the event to everyone
exactly as the test expected. Nothing separates one subscription from two
except counting them, which it now does at Redis, plus a duplicate-delivery
check for the second receive loop. Fails against the pending record not being
minted in the scan.
- The remedy test said "the old connection must also be gone" and waited for a
receive-loop exit. stopRedisSubscription does two things and the loop exits on
the first alone, so it passed against a version that cancelled the loop and
left the PubSub and its health check open. Counted at Redis now; fails against
exactly that mutation.
- The parallel-recovery test could not tell serial from parallel -- a serial
pass cycles all thirteen workspaces too. It now uses a rendezvous, asserts the
peak concurrency is above one AND within the cap, and fails against a serial
implementation.
- The prefixed-garbage test only exercised the classifier. Whether
receiveMessages ACTS on the error is a different claim, now driven through
the real Redis path.
- The metrics adapter test's comment said "every reason this bus can emit"
while subscription_unconfirmed was missing; its zero-assertion proved
non-leakage, not mapping. Emitted now with a count distinct from its
neighbour's, so a merging adapter cannot satisfy both.
PROSE THAT OUTLIVED THE CODE, again. The latency arithmetic still described the
single shared ticker that round 5 replaced with two independent loops; from
lastSeen [3T,4T) still holds, but from FAULT ONSET it is roughly [2T,4T)
because the publisher has its own phase. And a second "and replaces the
connection" in deployment.md that round 6's sweep missed.
Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X
* docs(events): correct three contract statements (codex r9)
Round 9 was cross-artifact conformance: every commitment the plan made was
checked against the code. All met -- wire classifier, lastSeen placement and
locking and install stamp and every-frame stamping, heartbeats bypassing
Publish and the shared counter, the drop-and-cycle remedy under the
single-establisher invariant, all four joint rules, the two-phase rollout with
its inverted-rationale test, and the reason/Help/deployment.md trio with the
rollout order. It also confirmed the three documented mutation survivors are
correctly dispositioned: both wsCounts checks are redundant-but-cheap under the
current invariant, and omitting the lastSeen.IsZero() skip is right because
adding it would mask a regression in the install stamp.
Three statements were wrong.
The env-var contract. My test comment said an unparseable PAD_EVENTS_HEARTBEAT
"must leave the flip off", which is true from a default config and false from a
config file that set it true -- there the value is left alone, as the
precedence test already asserts. The BEHAVIOUR is right and matches the epoch
flag: a typo must not move a migration in either direction, and silently
rolling an operator back to phase 1 would disable detection on a fleet that had
opted in with nothing saying so. Only the prose overclaimed, and it overclaimed
in the direction that invites someone to "fix" the ignore into a fail-closed
reset.
The constructor. NewRedisBusWithKeys documented publishEpoch and said nothing
about publishHeartbeat sitting next to it -- two adjacent booleans of the same
type belonging to two independent migrations, which is a shape that gets
swapped or dropped in a maintenance edit. Both now documented in order, with a
note that any combination is valid.
A stale count. EventSequenceResetsTotal's comment said "Five reasons" and there
are seven; it was already wrong by one before this unit added another. Replaced
with the count plus a pointer to the three artifacts that are authoritative and
move together, since the count itself is the part that goes stale first and is
read last.
Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X
* test(events): make the cadence arithmetic testable, and justify a guard pair
Matrix 5 (29 mutations, 21 detected) surfaced two things the previous run
could not, because both concern code the codex rounds added.
THE DRIFT FIX HAD NO TEST. Restoring the sleep-after-work form survived every
test in the package, and would have kept surviving: the only way to observe
drift through the loop is to time it, and a timing assertion is a flaky
assertion. Extracting nextTick makes the arithmetic checkable without a clock,
and the four cases now pin what the schedule is for -- a slow pass does not
push the next tick out, ten slow passes accumulate no drift, an overrun beyond
one interval resets instead of replaying the missed ticks, and an overrun
WITHIN one interval still catches up rather than re-phasing the schedule
permanently. Both directions mutation-checked.
The property is worth this much because breaking it is self-defeating rather
than merely untidy: an instance whose passes are slow emits heartbeats further
apart, its own subscription sees them further apart, and it crosses its own 3T
threshold and cycles connections that were never wedged.
A GUARD PAIR THAT ONLY DIES TOGETHER, which the team lesson says to treat as a
question rather than a clearance. The loop's ctx.Done select arm and its
post-wait ctx check each survive removal alone. Checked rather than assumed:
they cover disjoint moments and each is independently right -- the select arm
is the exit while WAITING, which is where the goroutine spends its life, and
the post-wait check stops a bus that closed DURING a pass from starting
another one against a cancelled context and a drained wsSubs. Removing BOTH is
detected. Reasoning recorded at the code, and the combined mutation added to
the matrix so the pair cannot quietly become a single point of failure.
Also fixed an ineffassign the lint gate caught in the new test.
Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X
* docs(events): state what the detector does not cover (codex r10)
Round 10 was adversarial: refute the unit's central claim rather than look for
defects in it. It partly succeeded, and the corrections are worth more than
most of the bug findings.
The claim was "a wedged connection is detected, coverage is ended, and the
connection is replaced so delivery resumes". Three parts of that were too
strong, and all three limits were checked against go-redis v9.22.0 rather than
argued:
IT IS A RECEIVE-SIDE DETECTOR, not a round-trip health check. It measures
whether frames ARRIVE. A subscription whose outbound direction is broken but
which still receives reads as healthy -- correctly, since nothing is lost, but
that is a narrower claim than "the connection is healthy".
IT CANNOT COVER THE PUBLISH PATH. PUBLISH travels on the client's connPool
while a subscription holds a connection from the separate pubSubPool
(redis.go:363, :1956) -- different sockets, different fates, and a reconnect of
one repairs nothing about the other. An instance whose publish path is wedged
loses its own events for every other instance and this feature will not say so.
That is a real gap in the family's coverage, now written down rather than
implied away.
REPLACEMENT IS ATTEMPTED, NOT GUARANTEED. If the path is still blackholed when
the cycle re-dials, the replacement cannot receive either. Coverage stays ended
so nothing is falsely claimed, but delivery resuming is a statement about the
network rather than about this code.
Filed BUG-2764 rather than folded in: establishSubscription's
`b.client.Subscribe(dialCtx, channel)` silently discards the SUBSCRIBE error,
because go-redis's own Client.Subscribe drops it (`_ = pubsub.Subscribe(...)`,
redis.go). A failed subscribe therefore installs a connection that looks live
and is subscribed to nothing. It is pre-existing, it lives in the establishment
path three bugs have already converged on, and changing how that function
issues its SUBSCRIBE does not belong in a diff about idle detection. Worth
knowing here because it is the one way the replacement can fail on a HEALTHY
network -- and because the detector now cycles it on the next pass, which is
why it self-heals on phase 2 and stays dead forever on phase 1.
Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X
* fix(events): do not cycle a workspace that recovered before its turn (r11 P1)
Codex round 11 attacked three claims. Phase-1 safety and rollback safety both
came back clean -- a phase-1 receiver stamps lastSeen and nothing else, touches
no buffer, metric, client, ID or epoch, and its maintenance loop is not started
at all, so that timestamp is inert; heartbeats leave no state in Redis or
across a process replacement, and a mid-cycle shutdown rechecks b.ctx before
installing. The third claim did not survive.
FALSE POSITIVES ON A HEALTHY SYSTEM, which is the property this design cares
about most: cycling a working subscription drops its coverage and resyncs every
one of its subscribers for nothing.
cycleIdleSubscriptions selects its victims under the lock and releases it; the
cycles run afterwards. Its re-checks asked about generation, subscriber count
and bus liveness -- and never re-asked the question the scan had asked. A
subscription that started receiving again in that window was cycled anyway.
The window is not theoretical, and this unit widened it itself: the 8-way
concurrency cap added in round 5 makes a workspace wait behind earlier batches
of slow replacement dials, and a GC or CPU pause leaves a backlog of heartbeats
undrained in the receive loop. Both are ordinary conditions on a loaded box.
cycleOne now validates, ends coverage and tears down WITHOUT RELEASING THE LOCK
in between, which needed dropWorkspaceCoverage split into a locked variant that
returns its reason for the caller to report after unlocking. That also removes
the ordering fragility the previous version documented rather than fixed: there
is no longer any window in which coverage is ended for a workspace this
function then decides to leave alone. The log moved after the decision for the
same reason -- it could previously describe a cycle that then abandoned.
The freshness term is load-bearing and says so, next to the three neighbouring
terms whose mutation survivals are recorded as redundant-but-cheap. Removing it
is detected, by a test that lands the recovery in the exact gap through a new
positional seam.
NTP steps were checked and are not a hazard: time.Time carries a monotonic
reading, so a wall-clock step cannot make a subscription look idle.
Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X
* perf(events): take logging and PubSub.Close off the global lock (codex r12)
Round 12 verified round 11's freshness fix: validation, coverage invalidation
and teardown are atomic under b.mu with no lock cycle,
dropWorkspaceCoverageLocked preserved the original semantics exactly including
the no-buffer branch that still signals subscribers, reset reporting happens
after unlocking, and the replacement metric still lands only when a new
generation does. Slow establishment stays outside b.mu, wg.Wait only delays the
next pass, and Close cancellation retires pending records.
Two P2s, both about what round 11 put UNDER that lock:
slog.Warn ran while b.mu was held. slog invokes the installed handler
synchronously, and b.mu is the lock every fan-out and every Subscribe on the
instance contends for -- a slow or custom handler stalls all of them, and one
that calls back into the bus deadlocks. Moved after the unlock; it still has to
come after the DECISION, for round 6's reason, so both constraints are now
stated together at the call.
PubSub.Close ran under b.mu too. It takes go-redis's own mutex, which the
health check can hold across reconnect work, so a network-bound wait sat inside
the instance's hottest lock. That was survivable when teardown only happened as
a workspace lost its last subscriber; the idle detector makes it happen on
every cycle, which is what turned a latent cost into a real one. Handed off to
a goroutine: nothing references the PubSub once the map entry is gone, and
cancel() -- which is what actually stops delivery -- still happens under the
lock.
Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X
* fix(events): do not read our own failed probe as a dead peer (codex r13)
Round 13 asked for a production-approval review. Four findings; the second is
the sharpest of the whole run because it is the mirror image of the failure
this feature exists to find.
A FAILED HEARTBEAT PUBLISH WAS READ AS A DEAD SUBSCRIPTION. The detector's
inference is "we published a frame and nothing came back, so the receive path
is dead" -- valid only if the publish actually happened. PUBLISH travels on the
client's connPool while the subscription holds a connection from the separate
pubSubPool, so a publish-side failure (pool exhaustion, a wedged outbound
route, Redis refusing writes) says nothing about whether that subscription can
receive. The detector was reading its own inability to probe as evidence about
the peer, and tearing down healthy connections on a schedule: a resync for
every subscriber of every workspace, every 90s, for as long as the outbound
path stayed broken. The third load inversion this unit has had to fix.
redisSub.lastProbeOK now records the last SUCCESSFUL publish, and detection is
suspended while it is stale -- checked in the scan and again in cycleOne, which
is a pair that only dies together and is therefore justified at the code:
the scan's keeps a workspace off the due list so no record is minted and no
joiner waits, cycleOne's covers the probe failing AFTER selection, a window the
concurrency cap makes real. Neither subsumes the other; removing both is
detected. New counter pad_event_heartbeat_publish_failures_total, documented as
DETECTION DEGRADED rather than as a peer being broken.
THE END-TO-END TEST THAT DID NOT EXIST. Every other test drives this through a
fake clock -- necessary, since the threshold is 90s by construction and
miniredis always answers, but it means they all ASSUME the wedge rather than
produce it. A TCP proxy that stops delivering server->client on the connections
already open, while writes keep succeeding and new connections stay healthy,
produces the real thing. The test asserts both halves of the claim: the wedge
is detected, and the replacement delivers. Both halves mutation-checked
(detector disabled; drop-only with no replacement).
The proxy's first version was vacuous -- a global flag consulted at read time
meant re-enabling delivery for future connections also revived the ones meant
to be dark. Per-connection now, and the comment says why.
Also: PubSub.Close taken off b.mu in Close() too (round 12 fixed only the cycle
path), and the replacement counter now takes an explicit installed result from
establishSubscription rather than inferring one from the live generation --
inference misattributed an unrelated caller's fresh subscription as this
cycle's replacement, and missed a real replacement that had lost its last
subscriber. Both mutation-checked.
Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X
* fix(events): bind the probe stamp to a generation; make the proxy test honest
Round 14 returned a BLOCK verdict on two P2s, both mine, both in the fix that
round 13 had just added.
lastProbeOK WAS NOT GENERATION-BOUND. publishHeartbeats snapshots the workspace
list, publishes off the lock -- for as long as go-redis's timeouts allow -- and
then stamped whatever subscription occupied that workspace by the time it
returned. A probe sent for generation A could credit generation B, which never
received one; if later probes then failed, B could be cycled while looking
recently probed. Exactly the hazard stampLastSeen already guards on the same
map, and I did not carry it across. The generation now travels with the
snapshot and is validated before stamping.
THE END-TO-END TEST COULD PASS WITHOUT EXERCISING WHAT IT CLAIMED. It darkened
the receive direction of every open connection, including the ordinary pooled
connection PUBLISH uses -- so the probe may have been failing too, and the run
would then have been exercising the cannot-probe path rather than a half-open
route, which is the very distinction round 13 added the premise check for. The
proxy now classifies connections as it forwards and darkens only one that has
carried a SUBSCRIBE, leaving the publish path healthy, and the test asserts
zero probe failures so a run that drifts back into the other case fails loudly
instead of passing quietly. Still fails against a disabled detector and against
drop-only.
Also covered the new counter's mapping in the metrics adapter test, with a
count distinct from both neighbours -- cycled, idle_timeout and
heartbeat-publish-failure say three different things and an operator acts on
the difference.
Verified by the same round: install-time stamping does not permanently suppress
detection, establishSubscription returns false only on abandon and true on all
three installed paths including the cancelled-establisher goroutine, and
Close's deferred PubSub.Close runs after the unlock.
Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X
* test(events): pin the probe-across-replacement race (closes r15's residual)
Round 15 returned CLEAN and approve-with-comments, naming one residual: the
generation binding on lastProbeOK had no deterministic test, only the argument
that it mirrors stampLastSeen. This closes it with a positional seam between
the publish and the stamp, which is the only place that interleave can be
forced.
TWO INSTRUMENT DEFECTS ON THE WAY, both caught by mutation rather than by
reading:
The first version compared the credited stamp against the PROBE's timestamp.
On a frozen clock the replacement's install stamp and a wrongly-credited probe
are the same value, so it could not tell them apart -- it failed on the install
stamp while claiming a credit had happened, and removing the generation binding
still passed. It now compares against what the replacement was INSTALLED with,
and the clock advances inside the seam so a buggy write lands strictly later.
The second version was FLAKY: 2 failures in 3 runs. The heartbeat that was just
published comes back through miniredis on another goroutine, and if it lands
between the forced-stale write and the scan it refreshes lastSeen, the
workspace is not due, and no replacement happens. Retried until the generation
actually moves. Now 5 of 5 green unmutated and 5 of 5 detected mutated -- which
is the bar, because a 2-in-3 detector reads as coverage while being noise.
Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X
* fix(events): on-call signals — log the cycle outcome, correct two claims (r16)
Round 16 read the diff as the person paged at 3am. Four findings.
THE CYCLE LOGGED ITS ATTEMPT AND NEVER ITS OUTCOME. The line says "attempting
to replace", which is correct and, on the one path where the replacement does
not happen, left an on-call with a warning, no counter movement, and no
explanation. Now there is a second line naming the reason.
pad_event_receive_loop_exits_total's documentation was falsified by this unit
and neither doc site said so: every idle cycle stops a receive loop while its
subscribers are still connected, and the comment still claimed exits happen
only at shutdown or when the last subscriber leaves. Both sites corrected, with
the expectation that it tracks the cycle counter during an incident.
A CLAIM I MADE AND THEN COULD NOT SUPPORT, recorded rather than quietly kept.
Round 16 argued the age-based premise check ("has a probe succeeded within the
threshold") failed to suspend detection where an ordering rule ("has a probe
succeeded since anything last arrived") would, and I rewrote the rule on that
argument and wrote a test named for the defect. The mutation matrix then
refused to confirm it: reverting to the age form leaves the test green, and so
does removing both copies of the check, and no case separates the two — on any
healthy path the two stamps advance together, because a probe whose frame
arrives sets both, and they diverge only on the wedge where both forms cycle.
The ordering rule is kept, because it states the intent exactly and is never
weaker. But the test and the comment now say what they actually establish —
that a probe which has started failing stops the detector concluding from
silence, which is the property both forms share and neither had before — rather
than claiming a fixed defect I cannot demonstrate.
The two remaining P2s are already-filed residuals: the cycled counter proves an
install rather than a working replacement (BUG-2764), and repeated cycling
amplifies /changes load with no jitter or global budget (BUG-2761). Both are
documented in deployment.md with their refs.
Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X
* docs(events): record what the final matrix actually says about four guards
Final matrix: 34 mutations, 22 detected, baseline restored green. Every
survivor is now documented at its line with why it survives, and two of them
turned out to be instrument defects rather than coverage gaps.
lastProbeOK's INSTALL STAMP IS REDUNDANT and the comment claimed otherwise. It
said a zero value "would permanently disqualify a subscription from ever being
cycled" -- true of the age-based premise it was written for, false under the
ordering rule that replaced it, because a zero value fails
`lastProbeOK.After(lastSeen)` exactly as an install stamp equal to lastSeen
does. Kept, for a reason it earns: it makes the field's invariant true by
construction, so a future rule reasoning about this value's AGE gets a real
timestamp rather than 1970 -- which is the trap the age-based rule fell into
one field over.
THE TWO cycleOne ABANDON GUARDS DIE ONLY TOGETHER AND ARE NOT REDUNDANT, which
took checking rather than assuming. They catch different shapes of the same
recovery: an arrival that has not been re-probed pushes lastSeen past
lastProbeOK so the premise case fires and the freshness case is unreachable --
that is the shape the test produces, and it is why removing either alone stays
green. But the publisher runs on its own goroutine at its own cadence and can
land a successful probe between the arrival and the decision, putting
lastProbeOK ahead again; there only the freshness case stops a healthy
subscription being torn down. Deleting it on the strength of the matrix would
remove the second shape's only guard.
Close's off-the-lock PubSub.Close is UNTESTED BY DESIGN, recorded rather than
papered over. It is a contention property, and the only assertion that
separates it is a timing one, which in this suite is a flaky one.
TWO HARNESS DEFECTS, both of which produced false survivors that would have
gone into the evidence package as findings. M11a inserted its mutation AFTER
the gate it was meant to disable -- unique anchor, wrong placement, so the
early return still fired and nothing changed; with a correct anchor it is
detected. M20 left variables unused and came back BUILD-BREAK rather than
answering; in compiling form it genuinely survives, consistent with
establishSubscription's abandon path already covering it.
The lesson worth keeping: when I rewrote all 34 anchors against current source
I verified each matched exactly ONCE, and uniqueness is not placement. An
anchor can be unique and still land somewhere that changes no behaviour.
Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X
* test(events): barrier the probe test on delivery — it was flaky, CI caught it
Go (PostgreSQL) failed on
|
||
|
xarmian
|
650d9df270 |
fix(events): bound the replay by append POSITION and buffer identity
Codex round 3. Both P1s real. The ceiling used lastAppendedID as if it were a time boundary. This bus's ids come from a counter shared across workspaces and a phase-1 publish assigns and publishes in two calls, so arrival order and numeric order genuinely disagree. Against an id-valued bound both directions break at once: a straggler arriving after registration is replayed although it also went to the caller's channel, and a pre-registration event carrying a higher id is filtered out and never replayed at all. replayBuffer now counts its appends, and the bound is a position — the entries to withhold are simply the final (appends - mark) of whatever since() returned, which may trim from the front but never the back. The mark also carries the BUFFER, not just a position in it. An ID-space reset during the wait replaces the buffer wholesale; a position in the old one describes nothing in the new one, and knownFrom may still accept an adjacent cursor, so the mismatch does not announce itself. Also corrected, all found by the same round and all mine: the Observer comment claimed this counter never reaches SequenceReset, which the late-confirmation path contradicts; the reason enumeration in metrics.go, its Help string and docs/deployment.md were never updated for the sixth reason; and both the metric and its comment said every increment is a client when it is one establishment however many subscribers were waiting. Accepted, not fixed: since() evaluates eviction over the whole buffer including post-registration appends, so a flood inside the wait can evict a cursor that missed nothing and force a sync_required. It costs a spurious resync, never silent loss, which is the direction this family chooses every time. Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X |
||
|
xarmian
|
0accca7a40 |
test(events,docs): pin the detection chain a colliding repair actually relies on (BUG-2740, codex round 5)
The repair seeds from wall-clock seconds, which is above any COUNTED history and is not a monotonicity guarantee. Corrupt the key twice inside one second and both repairs seed the same value, so two genuinely different id spaces carry the identical epoch — and an equal epoch means 'same space' by design, so neither epoch_change nor epoch_regressed fires. It is still not silent, and the reason is worth pinning because it is not the one the epoch mechanism suggests: a merge needs ids REUSED at a receiver, reuse needs the sequence to go BACKWARDS, and backwards is detected whatever the epoch says. counter_backward drops the affected buffers and refuses cursors below the discarded high-water mark. That was folklore until it was measured. The test drives the whole sequence — two repairs seeding the same value, a sequence reset between them — and asserts its own premise first (the two spaces really do share an epoch), that no epoch-based reason fires, that counter_backward does, and that the old cursor is refused rather than replayed the new space's events. Mutation-verified: stop reporting counter_backward and it fails. The docs carry the chain as a quoted rule, plus the two cases that look like it and are not — a counter set FORWARD is a jump inside one space with no reuse, and a receiver that never held the colliding range experiences a gap, which is BUG-2735's pre-existing class rather than anything this introduces. Lead re-ruled on the probed fact: residual ACCEPTED because it is detected, attribution corrected from epoch_regressed to counter_backward. Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X |
||
|
xarmian
|
383c6dc8d8 |
docs(events,metrics): finish the epoch_regressed claim and fix the heading level (BUG-2740, codex round 5)
Three of the round's four findings, all mine. THE SAME CLAIM IN THREE PLACES, TWO OF WHICH I HAD FIXED. Round 4 updated the struct comment above the counter and the new operator section; the EXPORTED Prometheus Help string and the deployment table row still told operators that epoch_regressed means Redis lost writes. The Help string is the one a person reads at the scrape endpoint, so it was the worst of the three to leave. Third time this run that a claim lived in more places than I enumerated before editing. 'CLIENTS RESYNC ONCE' was too strong. A repaired generation that lands BELOW one a receiver already holds is discarded as a straggler for that instance's 30-second window rather than adopted, so the same space can be disclaimed again when it is finally taken up. Bounded by the window, and distinguishable because it surfaces as epoch_regressed rather than epoch_change. The new section was a ### under a ## , which adopted every following #### section — including Event ID-space migration — as its children. It is a #### now, a sibling of the sections around it. Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X |
||
|
xarmian
|
8c05b02c7d |
docs(events,metrics): document the repair path an operator will actually meet (BUG-2740, codex round 4)
The fix changed an operator-visible outcome and nothing said so. A corrupted generation counter used to burn a sequence ID and fail every publish forever; it now repairs, publishes, and reseeds from wall-clock seconds — which the existing prose contradicted in two places, calling the generation 'monotonic' and telling operators that epoch_regressed means a failover to a replica that lost writes. Both corrected, and a new section states the two consequences: a repair can surface as epoch_change or, if a collision had pushed the counter higher, as epoch_regressed; and clients resync ONCE, not in a loop, because the repaired key is valid and the next rotation increments it normally. THE TELL IS THE VALUE, and saying so is the honest version of a claim I very nearly shipped instead. My first draft told operators to distinguish a repair from a failover 'by the neighbouring WARN log line naming the key'. There is no such line — the repair happens inside a Lua script, which cannot log through slog and does not change the counter's label. What actually distinguishes them is that a repaired generation LOOKS like a unix timestamp, ten digits around 1.7e9, rather than a small count of ID-space resets. That is a deliberate property of the seed, and it is now what the docs point at. The section also went in BETWEEN two rows of the metrics table on the first attempt, splitting it exactly as the previous unit's rollout note did. The check I wrote after that one looked for blank lines between rows and could not see a whole section inserted between them; the check is now 'is each table one contiguous run', which catches both. Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X |
||
|
xarmian
|
f4d1616078 |
docs(deployment): name the resume-boundary residual for operators too (BUG-2739)
The residual list covered the two that leave an OPEN stream stale (BUG-2735, BUG-2738) and not the one that affects RESUMES: a counter restart with the epoch intact leaves the two ID spaces overlapping, so a Last-Event-ID inside the overlap cannot be attributed to either, and a client holding an old-space cursor there can be handed new-space notifications as though they followed it. Filed as BUG-2743 during this branch's review. Named here because an operator deciding whether a counter reset is safe should see it alongside the other two, together with the thing that actually prevents it: rotating the epoch. Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X |
||
|
xarmian
|
e4167a9160 |
docs(deployment): five corrections from reading the doc as a document (BUG-2739, codex round 20)
- THE NAMESPACE GUIDANCE WAS WRONG, and this is the substantive one. It said a moving undecodable_message suggests two Pad installations sharing a Redis. It does not: two CURRENT installations publish the same wire format, so their messages decode fine and the damage is cross-feeding real notifications between installations while this counter stays flat — a worse and quieter failure, and the one PAD_REDIS_NAMESPACE actually prevents. The counter indicates genuinely unreadable input: a non-Pad publisher, a mixed-version wire format mid-upgrade, or corruption. The wording was inherited from internal/events without checking that it transferred. - THE FLOOD COSTS WERE OVERSTATED AS SELF-BOUNDING. Heap growth and the announcement are bounded; per-message CPU and allocation are not — a fresh replay buffer plus a pass over every subscriber, on the single goroutine that also delivers real notifications, so a sustained flood is receive-loop starvation as much as it is garbage collection. - THE CUTOVER SECTION described every reconnecting client running a /changes delta. True of the web activity client; pad watch --stream clears its cursor and keeps the connection open, refetching nothing. The doc contradicted its own watch-stream paragraph fifty lines later, which this branch added. - 'A reconnecting client is covered in both cases' was too absolute: the shared-counter check reads at one instant and cannot see a notification published after the read, which resumeOutrunsLocalView and cmd_watch.go both already document as an at-most-once residual. - The rollout note said a reason-specific alert on either surviving reason is unaffected. False for counter_backward, whose spelling changed — which is the entire reason that paragraph exists. Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X |
||
|
xarmian
|
bc87aee6b5 |
fix(docs,metrics): repair the metrics table I split, and bound the fan-out claim (BUG-2739, codex round 19)
Two real findings from an unconstrained fresh-eyes pass. THE TABLE WAS BROKEN. Round 11's rollout note was inserted BETWEEN two rows of the metrics table, so every row after it — eight of them, including all the pad_event_* counters and the presence failures — rendered as plain pipe-delimited text rather than a table. A documentation change that silently breaks the page it documents is worse than the omission it fixed, and no gate in this repo renders Markdown. The note now sits after the table, and a check across the whole file confirms no blank line or prose splits any of its eight tables. THE FAN-OUT CLAIM WAS STILL TOO STRONG, in both the metric help and the docs row: 'each moves pad_watchevents_midstream_resyncs_total once per such subscriber'. The gap signal is capacity-1 and coalescing, so a second cause firing before a client has acted on the first adds no announcement. It is AT MOST one per subscriber, and reading the fan-out off the two counters needs a reset observed in isolation against idle clients. Round 5 narrowed the aggregate version of this claim and left the per-event one standing. Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X |
||
|
xarmian
|
bc6d3541cb |
refactor(watchevents,docs): order the startup test, move operator prose out of the code (BUG-2739, codex round 16)
A future-maintainer round, three of whose four findings were fair. THE REAL DEFECT: TestNoCoverageIsDroppedAtStartup slept 500ms and hoped the receive goroutine had had its chance. No happens-before, so it could miss the regression or turn scheduler-sensitive. It now publishes and waits for DELIVERY instead: the pub/sub channel is FIFO, so a startup confirmation — if the constructor stopped consuming it — is queued AHEAD of that notification and has necessarily been processed by the time it comes out the other end. Deterministic, strictly stronger, and 0.00s instead of 0.50s. Re-verified against its mutation: removing the constructor's Receive still fails it 3/3. COMMENT ACCRETION: dropCoverage had 76 lines of commentary over 30 of code, including a threat model and a per-message cost breakdown that are operator decisions. Those moved to docs/deployment.md, where operators actually read, and the code keeps the invariants and the one design question a maintainer will ask (why not gate on the shared counter). 45 lines now, and nothing was deleted — only relocated to the artifact whose audience it was written for. THE TIME BOMB: the rollout note said 'this paragraph expires at the next tag' with nothing enforcing it. A claim about release state that goes stale silently is exactly what this branch has spent nine rounds removing, so it now carries the three commands to re-derive it instead of asking to be trusted. DECLINED: extracting fanOutLocally's switch into a coverage-state transition helper. The accretion is real and predates this branch — the switch, its four fields and their reset duplication are the existing design, to which this added one arm. A state-machine refactor of the receive path is its own change with its own review and its own mutation matrix; folding it into a bug fix at round 16 is how a fix's blast radius stops matching its claim. Worth filing if a third reviewer raises it. Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X |
||
|
xarmian
|
988bea658c |
docs(deployment): state the watch reset metric has no released contract (BUG-2739, codex round 11)
A mixed-version-fleet round raised two things — the label spelling, and the
metric widening from 'the ID space changed' to 'replay coverage was dropped'
with two new reasons — as a rollout hazard where old and new instances report
two shapes under one name.
Both collapse to one fact, verified rather than assumed: the entire metric was
introduced by
|
||
|
xarmian
|
720182a445 |
docs(watchevents): sweep the undecodable overclaim as a class, with its boundary (BUG-2739, codex round 9)
Third round running on the same claim, each time in a place the previous
instance fix had not looked — which is CONVE-18's failure mode in miniature,
committed by me twice after writing the sweep into two commit messages.
THE POPULATION, enumerated this time instead of chased: grep for
'undecodable' across the tree. Within internal/watchevents and the artifacts
this branch owns, three sites still said a notification was LOST — the test's
headline sentence (contradicted by its own closing paragraph eight lines
below), the receive-loop comment ('the id of what we missed is unknown BY
DEFINITION'), and the deployment table's trailing summary ('part of it was
simply missed'). All three now say what the instance actually knows:
something arrived that it could not read, it cannot tell whether that was
ours, and it stops vouching BECAUSE it cannot tell — a claim about our own
evidence rather than about the stream.
THE SEARCH BOUNDARY: internal/watchevents, internal/metrics and
docs/deployment.md — the artifacts this change owns. internal/events carries
the same wording for its own undecodable arm (observer.go, bus.go, the
pad_event_* Help string and its docs row) and is NOT touched here. Whether
that wording is equally overclaimed is an open question about a package this
PR otherwise leaves alone, not a cleared one.
Round 9 also read the branch's commits in order for cross-commit interaction
and found none.
Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X
|
||
|
xarmian
|
76aa111e53 |
docs(metrics,deployment): finish the undecodable wording and answer the label question in the artifact (BUG-2739, codex round 6)
Round 6's end-to-end failover trace matched the comments step for step, so both findings were prose again. The struct comment above WatchSequenceResetsTotal still grouped undecodable_message with subscription_resumed as 'we simply lost part of it'. Only the second is demonstrable: for the first, the instance knows an unreadable message arrived and cannot tell whether it was ours. Round 5 corrected the exported Help string and the docs but not this one — the same claim in a fourth place, which is what a class-wide sweep is supposed to prevent. The counter_backward rename has now been raised three rounds running, each time because the answer lives in a commit message and a reviewer reads artifacts. It is answered where an operator with a broken dashboard would look: no tagged release emitted the plural, so there is nothing to migrate. Re-derived at 40b0db06 rather than restated. Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X |
||
|
xarmian
|
2e9ace4194 |
docs: nine overclaims across code, metrics, docs and the CLI (BUG-2739, codex round 5)
A cross-artifact pass, which is the angle that keeps paying on this family. Every item below was a statement of mine that was false or unsupported; the code did not change. WRONG FACTS: - Watch epochs are opaque UUIDs, not numeric generations. I had copied internal/events' wording, where they ARE numeric — the distinction is the subject of internal/idspace's package comment. - undecodable_message was described as proof a notification was missed. The instance knows only that something it could not read arrived on its channel; it cannot tell whether that was ours. It stops vouching BECAUSE it cannot tell, which is a different and weaker claim. Corrected in four places. - The failover-cost paragraph said every SSE client on the instance reconciles. Wrong twice: a watch-bus resubscription ends the WATCH stream's coverage (activity coverage is per-workspace), and the one client that uses that stream today — pad watch --stream — answers sync_required by clearing its cursor and keeping the connection open, so it issues no request at all. Verified in cmd_watch.go rather than assumed. - The midstream/reset ratio is not fan-out in aggregate: the announcement counter also carries gaps and slow-subscriber drops and coalesces per connection. Only a reset observed in isolation reads that way. - 'The watch stream's only signal was a later non-contiguous notification' is true for a client HOLDING A STREAM OPEN. A reconnecting client was always covered, because a resume asks the shared counter instead of local state. Scoped in the doc and in the test header. - The dropped-confirmation fallback said coverage still ends. Usually, not necessarily: with no traffic during the outage nothing was lost, and if the drops continue through whatever would expose the hole and the stream goes quiet, nothing ever does — BUG-2727's boundary. Named both. - The new Observer Close warning was overbroad: reports run on the receive goroutine only on the RedisBus receive path, while a ResumeGap runs on the caller's and MemoryBus has no such goroutine. The rule stays unconditional, since a callback cannot tell which case it is in, but it now says why. STALE AFTER THIS BRANCH: - metrics.go's WatchSequenceResetsTotal comment listed two reset reasons. - observer_test.go said 'both reset reasons'. - The constructor comment named Channel() after the loop moved to ChannelWithSubscriptions, and did not say the Receive beneath it is load-bearing for that loop having no skip-the-first flag. It does now, and names the test that fails if it goes away. - cmd_watch.go's sync_required cause list predated BUG-2739 (and did not mention the mid-stream delivery BUG-2730 added). Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X |