docs: scope the metric correlations to the causes that produce them (BUG-2730, codex round 12)

A cross-artifact pass over every claim in the comments, help strings and
deployment doc found two, both mine and both the same shape — a
correlation stated as general when it holds for one cause:

The watch drop metric and the doc row above it pointed operators at
pad_event_midstream_resyncs_total, while watch announcements increment
pad_watchevents_midstream_resyncs_total. Following either reference led
to the wrong series.

"drops >= announcements" and "the reset ratio is the fan-out" are each
true of one cause and not of the others. A watch sequence gap announces
to every subscriber without moving the drop counter; and the no-buffer
coverage loss, which the previous round added deliberately, announces
while moving NO cause counter at all — there was no coverage to end, but
the subscribers still have a hole. That last one is the interesting case
to leave written down, because an operator seeing announcements with
every cause counter flat would otherwise reasonably conclude the metric
was broken.

Both counters' descriptions now say ANNOUNCEMENTS rather than clients
told, and enumerate which causes correlate how.

docs/deployment.md

@@ -302,9 +302,10 @@ Alert on these instead:
 | `pad_watchevents_sequence_resets_total` | The Redis counter or epoch changed; replay buffers dropped |
 | `pad_watchevents_receive_loop_exits_total` | Non-zero outside shutdown means an instance publishes but receives nothing |
 | `pad_event_resume_gaps_total` | The ACTIVITY stream's (`/api/v1/events`) twin of the watch resume counter above. **Expect a step around a deploy, with the RATE settling back to baseline** (the counter itself only ever increases) — each instance starts with no replay coverage, so an early resume against a workspace it has not seen yet is a warranted resync. It counts RESUMES, not clients: a deploy with no reconnects does not move it at all, and a client that reconnects several times is counted several times. A rate that does not settle is the thing to alert on |
-| `pad_event_midstream_resyncs_total` | Activity-stream subscribers told MID-STREAM that they missed events, on a connection that stayed open. New in BUG-2730, and the counter to watch when judging whether that fix is costing more resyncs than it is worth. It counts ANNOUNCEMENTS, not causes and not distinct clients: one instance-wide coverage loss moves it once per live subscriber (and that ratio against `pad_event_sequence_resets_total` is the fan-out), while a burst of drops on ONE connection moves it once, because signals coalesce and are rate-limited per connection |
+| `pad_event_midstream_resyncs_total` | Activity-stream subscribers told MID-STREAM that they missed events, on a connection that stayed open. New in BUG-2730, and the counter to watch when judging whether that fix is costing more resyncs than it is worth. It counts ANNOUNCEMENTS, not causes and not distinct clients: a reset that drops buffers moves it once per live subscriber (and that ratio against `pad_event_sequence_resets_total` is the fan-out); a burst of drops on ONE connection moves it once, because signals coalesce and are rate-limited per connection; and a coverage loss on a workspace with no buffer yet moves it while every cause counter stays flat, because there was no coverage to end but the subscribers still have a hole |
+| `pad_watchevents_midstream_resyncs_total` (see also, listed above) | Same meaning for the watch stream. Its causes are a slow-subscriber drop and a received sequence gap or reset; a gap announces to EVERY subscriber on the instance, so it can exceed all of its cause counters |
 | `pad_event_sequence_resets_total` | Activity replay coverage dropped, by reason. `subscription_resumed` — a pub/sub connection dropped and resubscribed, dropping that workspace's buffer; expect it during a Redis failover and expect it to stop afterwards. `epoch_change` — the shared counter's ID space changed generation, dropping every buffer; expect a handful per cutover. `counter_backward` — an ID arrived at or below a buffer's high-water mark with no generation change; see *Event ID-space migration* for what to expect per phase. `epoch_regressed` — a LOWER generation was seen, so this instance stopped vouching for its buffers. One alongside an `epoch_change` is a message that was in flight when the generation rotated; a RUN of them means the counter itself went backwards, i.e. Redis lost writes. `undecodable_message` — a message on these channels could not be parsed, so that workspace's coverage ended; expect zero, and suspect a namespace collision |
-| `pad_event_events_dropped_total` | Activity events not delivered to a live subscriber, by reason — today only `slow_subscriber` (that connection's 64-deep channel was full). Per-SUBSCRIBER: every subscriber that was keeping up received the event. Pairs with `pad_event_midstream_resyncs_total`, one client at a time. New in BUG-2730, along with the fix that stops the drop being silent, so a deploy that starts reporting these is not necessarily a regression — it may be the first time they were countable |
+| `pad_event_events_dropped_total` | Activity events not delivered to a live subscriber, by reason — today only `slow_subscriber` (that connection's 64-deep channel was full). Per-SUBSCRIBER: every subscriber that was keeping up received the event. Pairs with `pad_event_midstream_resyncs_total`, though not one-for-one in either direction — see that row. New in BUG-2730, along with the fix that stops the drop being silent, so a deploy that starts reporting these is not necessarily a regression — it may be the first time they were countable |
 | `pad_event_receive_loop_exits_total` | A workspace's activity subscription loop stopped. Unlike the watch stream's twin this does **not** stay at zero — it is expected at shutdown and whenever a workspace's last local subscriber leaves. Read it as a rate against a stable subscriber count |
 | `pad_session_presence_failures_total` | Presence operations failing — **read the `op` label**, the risks differ and run in opposite directions: `register`/`renew` may under-report (a live session unlisted and untargetable), `deregister` may over-report (a dead session left listed, and a push aimed at it reaches nobody), `list` returns a 503, `prune` is benign. A failure means the operation reported an error — Redis can fail a pipeline after applying it, so the write may have landed anyway |
 

internal/metrics/metrics.go

@@ -134,12 +134,13 @@ type Metrics struct {
 	// the drop honest — the affected subscriber is now told mid-stream.
 	//
 	// Read it alongside pad_event_midstream_resyncs_total, but NOT as a
-	// one-to-one correspondence: the gap channel coalesces and the handler
-	// rate-limits, so a burst of drops on one connection produces a single
-	// announcement. Expect drops >= announcements, and treat a large ratio as
-	// what it is — one client falling a long way behind, not many clients
-	// affected. It does NOT move pad_event_resume_gaps_total, which stayed
-	// resume-only so existing alerts keep their meaning.
+	// one-to-one correspondence, in either direction: the gap channel
+	// coalesces and the handler rate-limits, so a burst of drops on one
+	// connection produces a single announcement; and coverage losses produce
+	// announcements with no drop at all. A large drop-to-announcement ratio
+	// is one client falling a long way behind, not many clients affected. It
+	// does NOT move pad_event_resume_gaps_total, which stayed resume-only so
+	// existing alerts keep their meaning.
 	EventEventsDroppedTotal *prometheus.CounterVec
 
 	// EventMidstreamResyncsTotal and WatchMidstreamResyncsTotal count clients
@@ -154,10 +155,14 @@ type Metrics struct {
 	// Counts ANNOUNCEMENTS MADE, which is close to clients told but not
 	// identical, and the difference matters when reading it:
 	//
-	//   - one instance-wide coverage loss moves this once per live
-	//     subscriber, while pad_*_sequence_resets_total moves once. That
-	//     ratio is the fan-out, and it is the number to look at when deciding
-	//     whether a resync storm is underway.
+	//   - a reset that drops buffers moves this once per live subscriber,
+	//     while pad_*_sequence_resets_total moves once. That ratio is the
+	//     fan-out, and it is the number to look at when deciding whether a
+	//     resync storm is underway.
+	//   - a coverage loss on a workspace with NO buffer yet announces while
+	//     moving no cause counter at all, deliberately: there was no coverage
+	//     to end, but the subscribers still have a hole. So this counter can
+	//     move with every cause counter flat.
 	//   - a burst of drops on ONE connection moves this once, not once per
 	//     drop: the gap channel coalesces and the handler rate-limits.
 	//   - the same connection can be counted repeatedly over its life, once
@@ -441,17 +446,17 @@ func New() *Metrics {
 
 	eventMidstreamResyncsTotal := prometheus.NewCounter(prometheus.CounterOpts{
 		Name: "pad_event_midstream_resyncs_total",
-		Help: "Activity-stream subscribers told MID-STREAM that they missed events, on a connection that stayed open (BUG-2730). Counts clients told, not causes — one instance-wide coverage loss tells every subscriber and increments this once per subscriber, while pad_event_sequence_resets_total counts it once. New in BUG-2730; a fresh counter rather than folding into pad_event_resume_gaps_total, so alerts on that one keep their meaning.",
+		Help: "Activity-stream announcements made to a subscriber MID-STREAM, on a connection that stayed open (BUG-2730). Counts announcements, not causes and not distinct clients. A reset that drops buffers moves this once per live subscriber while pad_event_sequence_resets_total moves once, so that ratio is the fan-out; a coverage loss on a workspace with no buffer yet announces while moving NO cause counter, because there was no coverage to end; and a burst of drops on one connection announces once, because signals coalesce and are rate-limited. A fresh counter rather than folding into pad_event_resume_gaps_total, so alerts on that one keep their meaning.",
 	})
 
 	watchMidstreamResyncsTotal := prometheus.NewCounter(prometheus.CounterOpts{
 		Name: "pad_watchevents_midstream_resyncs_total",
-		Help: "Watch-stream subscribers told MID-STREAM that they missed notifications, on a connection that stayed open (BUG-2730). Counts clients told, not causes. See pad_event_midstream_resyncs_total.",
+		Help: "Watch-stream announcements made to a subscriber MID-STREAM, on a connection that stayed open (BUG-2730). Counts announcements, not causes and not distinct clients. Its causes are a slow-subscriber drop (pad_watchevents_notifications_dropped_total) and a received sequence gap or reset (pad_watchevents_sequence_gaps_total, pad_watchevents_sequence_resets_total) — a gap announces to EVERY subscriber, so this can exceed all of them.",
 	})
 
 	eventEventsDroppedTotal := prometheus.NewCounterVec(prometheus.CounterOpts{
 		Name: "pad_event_events_dropped_total",
-		Help: "Activity events not delivered to a live subscriber, by reason: slow_subscriber (that connection's 64-deep channel was full). Per-subscriber, not per-event. Since BUG-2730 a drop also tells that subscriber, so expect pad_event_midstream_resyncs_total to rise with it — but not one-for-one: signals coalesce and are rate-limited per connection, so drops >= announcements. It does NOT move pad_event_resume_gaps_total, which stayed resume-only.",
+		Help: "Activity events not delivered to a live subscriber, by reason: slow_subscriber (that connection's 64-deep channel was full). Per-subscriber, not per-event. Since BUG-2730 a drop also tells that subscriber, so expect pad_event_midstream_resyncs_total to rise with it — but not one-for-one, and not exclusively: signals coalesce and are rate-limited per connection, so drops exceed the announcements THIS cause produces, while coverage losses add announcements with no drop at all. It does NOT move pad_event_resume_gaps_total, which stayed resume-only.",
 	}, []string{"reason"})
 
 	eventSequenceResetsTotal := prometheus.NewCounterVec(prometheus.CounterOpts{