Files
pulse/docs/release-control
rcourtman 27e440339b Switch every running tenant monitor when mock mode flips
Mock mode is one process-wide flag, but SetMockMode ended the epoch,
cleared the alerts, forgot the fixture agents and reset the state of
only the monitor it was called on. The server runs one monitor per
organization, each with its own alert manager and its own mock alert
pass. A switch made through one of them (the mock-mode API uses the
requesting tenant's monitor, the demo-fixture licence sync the default
one) left every other running monitor holding what it raised in the mode
the process had left: fixture alerts and fixture-agent node links in
live mode, live alerts in mock mode, until the 24-hour stale sweep.

Monitors now join a process-wide set of running monitors when Start runs
and leave it when Start returns, under the same mutex that already
serialized switches. SetMockMode flips the flag once, then ends the
epoch of the monitor it was called on and of every running monitor: it
waits for their admitted calls, clears their alerts, forgets their
fixture agents and resets their state. Only then does it restart each
running monitor's sampler, polls and discovery. Start picks the mock
metrics sampler or discovery while it holds that mutex, so no switch can
land between reading the mode and starting the runtime for it, and a
switch leaves the runtime of a monitor that has not joined yet to its
own Start. A monitor records the mode and the switch count it was built
under. If it missed switches between New and Start, it leaves what it
holds before it joins, dropping live alerts New restored or fixture
alerts a read path raised meanwhile, even when a round trip brought the
flag back. A call for the mode the flag already holds still moves a
reached monitor whose alerts belong to the other mode, and leaves the
rest alone.

Release builds reach this only on an instance that runs DemoMode with
the demo_fixtures entitlement, since release ValidateEnablement refuses
mock fixtures otherwise, and that has more than one organization's
monitor running. The licence sync only switches the default
organization. So the fix stays inside the monitoring package. The
callers are unchanged and no API surface moves.

New tests start the default and one other tenant monitor through
MultiTenantMonitor, as the server does, and switch through one of them.
Against the per-monitor switch, the default monitor kept its fixture
alerts (69 to 104 across runs) and all 69 fixture agents after mock mode
was left through the other tenant. That tenant kept its live alert after
mock mode was entered through the default monitor, and its fixture node
stayed suppressed by a fixture agent's leftover node link. Four more
tests fail if Start joins without leaving the mode its monitor was built
in, if Start compares only the mode after a round trip, if a call for
the flag's current mode returns without moving such a monitor, or if a
switch starts the runtime of a monitor that has not joined.
2026-10-07 13:48:10 +01:00
..