diff --git a/docs/deployment.md b/docs/deployment.md index 7ee7f2c0..91994af4 100644 --- a/docs/deployment.md +++ b/docs/deployment.md @@ -302,9 +302,10 @@ Alert on these instead: | `pad_watchevents_sequence_resets_total` | The Redis counter or epoch changed; replay buffers dropped | | `pad_watchevents_receive_loop_exits_total` | Non-zero outside shutdown means an instance publishes but receives nothing | | `pad_event_resume_gaps_total` | The ACTIVITY stream's (`/api/v1/events`) twin of the watch resume counter above. **Expect a step around a deploy, with the RATE settling back to baseline** (the counter itself only ever increases) — each instance starts with no replay coverage, so an early resume against a workspace it has not seen yet is a warranted resync. It counts RESUMES, not clients: a deploy with no reconnects does not move it at all, and a client that reconnects several times is counted several times. A rate that does not settle is the thing to alert on | -| `pad_event_midstream_resyncs_total` | Activity-stream subscribers told MID-STREAM that they missed events, on a connection that stayed open. New in BUG-2730, and the counter to watch when judging whether that fix is costing more resyncs than it is worth. It counts ANNOUNCEMENTS, not causes and not distinct clients: one instance-wide coverage loss moves it once per live subscriber (and that ratio against `pad_event_sequence_resets_total` is the fan-out), while a burst of drops on ONE connection moves it once, because signals coalesce and are rate-limited per connection | +| `pad_event_midstream_resyncs_total` | Activity-stream subscribers told MID-STREAM that they missed events, on a connection that stayed open. New in BUG-2730, and the counter to watch when judging whether that fix is costing more resyncs than it is worth. It counts ANNOUNCEMENTS, not causes and not distinct clients: a reset that drops buffers moves it once per live subscriber (and that ratio against `pad_event_sequence_resets_total` is the fan-out); a burst of drops on ONE connection moves it once, because signals coalesce and are rate-limited per connection; and a coverage loss on a workspace with no buffer yet moves it while every cause counter stays flat, because there was no coverage to end but the subscribers still have a hole | +| `pad_watchevents_midstream_resyncs_total` (see also, listed above) | Same meaning for the watch stream. Its causes are a slow-subscriber drop and a received sequence gap or reset; a gap announces to EVERY subscriber on the instance, so it can exceed all of its cause counters | | `pad_event_sequence_resets_total` | Activity replay coverage dropped, by reason. `subscription_resumed` — a pub/sub connection dropped and resubscribed, dropping that workspace's buffer; expect it during a Redis failover and expect it to stop afterwards. `epoch_change` — the shared counter's ID space changed generation, dropping every buffer; expect a handful per cutover. `counter_backward` — an ID arrived at or below a buffer's high-water mark with no generation change; see *Event ID-space migration* for what to expect per phase. `epoch_regressed` — a LOWER generation was seen, so this instance stopped vouching for its buffers. One alongside an `epoch_change` is a message that was in flight when the generation rotated; a RUN of them means the counter itself went backwards, i.e. Redis lost writes. `undecodable_message` — a message on these channels could not be parsed, so that workspace's coverage ended; expect zero, and suspect a namespace collision | -| `pad_event_events_dropped_total` | Activity events not delivered to a live subscriber, by reason — today only `slow_subscriber` (that connection's 64-deep channel was full). Per-SUBSCRIBER: every subscriber that was keeping up received the event. Pairs with `pad_event_midstream_resyncs_total`, one client at a time. New in BUG-2730, along with the fix that stops the drop being silent, so a deploy that starts reporting these is not necessarily a regression — it may be the first time they were countable | +| `pad_event_events_dropped_total` | Activity events not delivered to a live subscriber, by reason — today only `slow_subscriber` (that connection's 64-deep channel was full). Per-SUBSCRIBER: every subscriber that was keeping up received the event. Pairs with `pad_event_midstream_resyncs_total`, though not one-for-one in either direction — see that row. New in BUG-2730, along with the fix that stops the drop being silent, so a deploy that starts reporting these is not necessarily a regression — it may be the first time they were countable | | `pad_event_receive_loop_exits_total` | A workspace's activity subscription loop stopped. Unlike the watch stream's twin this does **not** stay at zero — it is expected at shutdown and whenever a workspace's last local subscriber leaves. Read it as a rate against a stable subscriber count | | `pad_session_presence_failures_total` | Presence operations failing — **read the `op` label**, the risks differ and run in opposite directions: `register`/`renew` may under-report (a live session unlisted and untargetable), `deregister` may over-report (a dead session left listed, and a push aimed at it reaches nobody), `list` returns a 503, `prune` is benign. A failure means the operation reported an error — Redis can fail a pipeline after applying it, so the write may have landed anyway | diff --git a/internal/metrics/metrics.go b/internal/metrics/metrics.go index 522f9fa4..2d79f49b 100644 --- a/internal/metrics/metrics.go +++ b/internal/metrics/metrics.go @@ -134,12 +134,13 @@ type Metrics struct { // the drop honest — the affected subscriber is now told mid-stream. // // Read it alongside pad_event_midstream_resyncs_total, but NOT as a - // one-to-one correspondence: the gap channel coalesces and the handler - // rate-limits, so a burst of drops on one connection produces a single - // announcement. Expect drops >= announcements, and treat a large ratio as - // what it is — one client falling a long way behind, not many clients - // affected. It does NOT move pad_event_resume_gaps_total, which stayed - // resume-only so existing alerts keep their meaning. + // one-to-one correspondence, in either direction: the gap channel + // coalesces and the handler rate-limits, so a burst of drops on one + // connection produces a single announcement; and coverage losses produce + // announcements with no drop at all. A large drop-to-announcement ratio + // is one client falling a long way behind, not many clients affected. It + // does NOT move pad_event_resume_gaps_total, which stayed resume-only so + // existing alerts keep their meaning. EventEventsDroppedTotal *prometheus.CounterVec // EventMidstreamResyncsTotal and WatchMidstreamResyncsTotal count clients @@ -154,10 +155,14 @@ type Metrics struct { // Counts ANNOUNCEMENTS MADE, which is close to clients told but not // identical, and the difference matters when reading it: // - // - one instance-wide coverage loss moves this once per live - // subscriber, while pad_*_sequence_resets_total moves once. That - // ratio is the fan-out, and it is the number to look at when deciding - // whether a resync storm is underway. + // - a reset that drops buffers moves this once per live subscriber, + // while pad_*_sequence_resets_total moves once. That ratio is the + // fan-out, and it is the number to look at when deciding whether a + // resync storm is underway. + // - a coverage loss on a workspace with NO buffer yet announces while + // moving no cause counter at all, deliberately: there was no coverage + // to end, but the subscribers still have a hole. So this counter can + // move with every cause counter flat. // - a burst of drops on ONE connection moves this once, not once per // drop: the gap channel coalesces and the handler rate-limits. // - the same connection can be counted repeatedly over its life, once @@ -441,17 +446,17 @@ func New() *Metrics { eventMidstreamResyncsTotal := prometheus.NewCounter(prometheus.CounterOpts{ Name: "pad_event_midstream_resyncs_total", - Help: "Activity-stream subscribers told MID-STREAM that they missed events, on a connection that stayed open (BUG-2730). Counts clients told, not causes — one instance-wide coverage loss tells every subscriber and increments this once per subscriber, while pad_event_sequence_resets_total counts it once. New in BUG-2730; a fresh counter rather than folding into pad_event_resume_gaps_total, so alerts on that one keep their meaning.", + Help: "Activity-stream announcements made to a subscriber MID-STREAM, on a connection that stayed open (BUG-2730). Counts announcements, not causes and not distinct clients. A reset that drops buffers moves this once per live subscriber while pad_event_sequence_resets_total moves once, so that ratio is the fan-out; a coverage loss on a workspace with no buffer yet announces while moving NO cause counter, because there was no coverage to end; and a burst of drops on one connection announces once, because signals coalesce and are rate-limited. A fresh counter rather than folding into pad_event_resume_gaps_total, so alerts on that one keep their meaning.", }) watchMidstreamResyncsTotal := prometheus.NewCounter(prometheus.CounterOpts{ Name: "pad_watchevents_midstream_resyncs_total", - Help: "Watch-stream subscribers told MID-STREAM that they missed notifications, on a connection that stayed open (BUG-2730). Counts clients told, not causes. See pad_event_midstream_resyncs_total.", + Help: "Watch-stream announcements made to a subscriber MID-STREAM, on a connection that stayed open (BUG-2730). Counts announcements, not causes and not distinct clients. Its causes are a slow-subscriber drop (pad_watchevents_notifications_dropped_total) and a received sequence gap or reset (pad_watchevents_sequence_gaps_total, pad_watchevents_sequence_resets_total) — a gap announces to EVERY subscriber, so this can exceed all of them.", }) eventEventsDroppedTotal := prometheus.NewCounterVec(prometheus.CounterOpts{ Name: "pad_event_events_dropped_total", - Help: "Activity events not delivered to a live subscriber, by reason: slow_subscriber (that connection's 64-deep channel was full). Per-subscriber, not per-event. Since BUG-2730 a drop also tells that subscriber, so expect pad_event_midstream_resyncs_total to rise with it — but not one-for-one: signals coalesce and are rate-limited per connection, so drops >= announcements. It does NOT move pad_event_resume_gaps_total, which stayed resume-only.", + Help: "Activity events not delivered to a live subscriber, by reason: slow_subscriber (that connection's 64-deep channel was full). Per-subscriber, not per-event. Since BUG-2730 a drop also tells that subscriber, so expect pad_event_midstream_resyncs_total to rise with it — but not one-for-one, and not exclusively: signals coalesce and are rate-limited per connection, so drops exceed the announcements THIS cause produces, while coverage losses add announcements with no drop at all. It does NOT move pad_event_resume_gaps_total, which stayed resume-only.", }, []string{"reason"}) eventSequenceResetsTotal := prometheus.NewCounterVec(prometheus.CounterOpts{