* fix(collab): close a peer the op bus had to drop an op for (TASK-1273)
When a subscriber's buffer was full, MemoryOpBus.Publish dropped the op
with a warning and the peer stayed connected with a document missing it
until it happened to reconnect. The bus doc promised a force-close that
was never built.
Now the first drop of an op a subscriber needs (anything but awareness)
marks it overflowed: Publish delivers it nothing more, because an op
after the gap would carry an op-log cursor past the dropped row and the
reconnect would resume from there and never replay it. The bus signals
the overflow (SubscribeWithOverflow) once, and the room manager closes
the peer's socket, so its reconnect replays the gap from the op-log.
Awareness drops stay silent. Counted in pad_collab_overflow_closes_total
through an optional OverflowObserver.
cmd/loadtest-collab gains -stalled N (clients that never read). Measured
against a local instance: 5 editors at 200 ops/s of 4 KiB, 2 stalled:
both stalled clients closed by the server, editors 0 errors; with the
close disabled, both stayed open.
Claude-Session: https://claude.ai/code/session_01Q9WssCjFuHjSfLABoW4tmS
* docs(deployment): pad_collab_overflow_closes_total (TASK-1273)
Claude-Session: https://claude.ai/code/session_01Q9WssCjFuHjSfLABoW4tmS
* test(collab): a real overflow closes the peer once (TASK-1273, codex)
Claude-Session: https://claude.ai/code/session_01Q9WssCjFuHjSfLABoW4tmS
* fix(collab): WebSocket dials get their own per-user bucket and a per-user socket cap; the web provider jitters its reconnects (BUG-1308)
A collab dial drew on the general per-user API bucket (burst 60), so 100
dials from one user got 60 through, and a user's own dials and REST
calls refused each other. Dials now draw on a CollabDial bucket (5/s,
burst 50), and how many sockets a principal may hold is bounded
separately by PAD_COLLAB_MAX_PER_USER (default 50), refused with 429
collab_limit_exceeded and Retry-After. Both defaults are sized from the
reconnect-storm measurement on BUG-1308's trail (checkpoint 2).
A browser cannot read a refused handshake's status, so the provider's
backoff is what spaces a refused herd out: each step is now jittered to
between half and all of itself.
Claude-Session: https://claude.ai/code/session_01RYiqbqXmxQMoKHsGm8B77N
* docs(collab): PAD_COLLAB_MAX_PER_USER and the collab dial bucket, in the deployment docs, samples and load-test usage (BUG-1308)
Claude-Session: https://claude.ai/code/session_01RYiqbqXmxQMoKHsGm8B77N
* fix(collab): only a WebSocket upgrade is a dial; per-user cap pinned within one workspace; dial-bucket keying stated correctly (BUG-1308, codex round 1)
Claude-Session: https://claude.ai/code/session_01RYiqbqXmxQMoKHsGm8B77N
* docs(collab): a PAD_COLLAB_MAX_PER_USER below 1 means unlimited, as the SSE knobs do (BUG-1308, codex round 2)
Claude-Session: https://claude.ai/code/session_01RYiqbqXmxQMoKHsGm8B77N
* docs(collab): the struct field agrees, 0 or less is unlimited (BUG-1308, codex round 3)
Claude-Session: https://claude.ai/code/session_01RYiqbqXmxQMoKHsGm8B77N
PR #468 shipped without a codex review because codex was unavailable
that day. This is the catch-up; codex found 2 P2s and 5 NITs.
[P2] readSendTimestamp recorded latencies for prior-run replay
frames. The original guard only checked nonzero ts, not session
recency, so any stale op-log row inflated p95/p99. Fixed: runContext
captures startedAt; recv path filters frames whose embedded ts
predates this run. Verified: against an item with 90 stale rows the
test counts received frames (630) but only records latencies for
the 180 live ones.
[P2] Shutdown deadlock. The writer goroutine could be blocked in
conn.WriteMessage when rc.done closed; the only path to conn.Close
was that same goroutine's select-case, so wg.Wait() could hang
forever under server backpressure. Fixed: per-client watchdog
goroutine closes the conn from outside the writer when done fires,
plus a 5s SetWriteDeadline per send as defence-in-depth. A 5s
duration test now exits in exactly 5.008s.
NITs (also fixed):
- buildFrame docstring corrected (minimum is 16 metadata bytes,
buffer is frameBytes+1).
- buildFrame returns (bytes, error) instead of log.Fatalf-ing on
rand.Read; caller logs detail and increments errors counter.
- -cookie / -token flag help now states both can be set together.
- Watchdog-induced WriteMessage errors are no longer counted as
real errors (isClosedDone check).
- buildFrame error path now logs the actual error detail.
Two rounds of codex review: round 1 found the items above; round 2
returned CLEAN.
Synthetic Go load-test for the Yjs collab dumb-relay. Each
simulated client opens a WebSocket, sends tagged sync frames at
a configurable rate, consumes inbound frames, and computes
broadcast fanout latency.
Doesn't depend on a real Yjs port — the dumb-relay's first-byte
discriminator (yMessageSync=0) is enough to exercise the persist +
broadcast path with synthetic payloads. Each frame embeds a
unix-nano timestamp + client ID so receivers can compute round-
trip latency without out-of-band coordination.
Findings (in DOC-1307):
- N=5, N=25: clean, p95 < 10ms, fanout matches expected (N-1)x
- N=100: 38/100 dial failures (consistent), but the 62 successful
see p95=27ms — server rejects ~38% of simultaneous dials at this
level. Filed BUG-1308 to investigate the ceiling.
- Op-log grows unbounded without compaction; old runs replay
on reconnect causing latency blow-up. Filed TASK-1309 to wire
a periodic prune sweeper.
Self-reviewed only; codex was unresponsive today after multiple
hour-long retries.
xarmian