Files
pad/cmd/pad
xarmian 6afe683389 fix(events): do not arm reset detection where interleave is ordinary traffic (BUG-2736)
Codex round 9, from the 3am-operator angle. Six findings; one of them was a
regression this diff would have shipped in the DEFAULT configuration, and the
review framed it as a log-volume problem.

THE REGRESSION. Phase 1 publishes with a two-call INCR-then-PUBLISH, so on any
multi-instance deployment two publishers interleave routinely and a lower ID
arrives after a higher one as ordinary traffic. main has no counter-backwards
detection at all; this diff added it. Armed unconditionally, it would have
fired on that ordinary interleave, dropped EVERY workspace's replay buffer,
and resynced every client -- in phase 1, which is where every deployment sits
until an operator flips phase 2.

The check is now armed only once an epoch has been adopted. What that costs is
stated rather than hidden: a genuine counter reset on a never-flipped
deployment goes undetected, which is exactly the behaviour before this change
and precisely the case phase 2 exists to fix.

The new test asserts the gate, and also asserts what is NOT claimed -- the
interleaved workspace's own buffer still holds ids out of order, so a cursor
at the higher one reads as foreign. That is pre-existing, unchanged here, and
strictly less harmful than a global drop; it is asserted rather than described
so a future change to since() surfaces there.

THE REST ARE THE OPERATOR'S SIGNALS, which were unreadable:

- The effective phase was invisible. pad_event_sequence_resets_total cannot be
  interpreted without it -- a counter_backward rate is expected on phase 1 and
  an anomaly on phase 2 -- and the setting can arrive from an env var, a TOML
  file, or neither. It is now on the startup line as id_space_phase.
- An unparseable PAD_EVENTS_PUBLISH_EPOCH was silently ignored, so an operator
  who typed "yes" believed they had flipped. Ignoring it stays the right
  behaviour; being silent about it does not.
- Both publish-failure logs said only "failed to publish". They now say what
  the operator needs, which differs by phase: phase 1 may or may not have
  reached subscribers, and phase 2's script is atomic so it did not
  half-execute, but a lost reply means it may have published anyway -- do not
  re-publish by hand.
- Adopting an epoch with empty buffers is the moment the documented residual
  becomes possible on that replica, and it happened silently. It now logs at
  INFO -- not a reset count, deliberately, since counting it would give the
  reset metric a per-deploy baseline.

Declined with reasons: a cause label on the resume-gap counter and a publish
failure counter are both pre-existing shapes rather than anything this diff
changed, and the straggler log is already bounded by the recovery window.

Claude-Session: https://claude.ai/code/session_01JVDBKbgn3Xt7ndW1YoYd8X
2026-08-22 20:05:48 +00:00
..