Follow-up to #2656 (the per-monitor mock-mode epoch fence), which
documented this gap as not covered.
## Problem
Mock mode is one process-wide flag (`mock.SetEnabled`), but
`Monitor.SetMockMode` ended the fence epoch, cleared the alerts, forgot
the fixture agents and reset the state of only the monitor it was called
on. The server runs one `Monitor` per organization
(`MultiTenantMonitor.GetMonitor` → `New` + `Start`), and each runs its
own mock alert pass into its own alert manager. A switch made through
one monitor (the mock-mode API uses the requesting tenant's monitor; the
demo-fixture licence sync uses the default one) left every other running
monitor holding what it raised in the mode the process had left: fixture
alerts and fixture-agent node links in live mode, live alerts in mock
mode, until the 24-hour stale sweep.
## Fix
- `internal/monitoring/mock_mode_switch.go`: a process-wide set of
running monitors, guarded by the existing `mockModeSwitchMu`. `Start`
joins it and leaves when its loop returns. While joining it picks the
mock metrics sampler or discovery under the lock (now
`startModeRuntime`), so no switch lands between reading the mode and
starting the runtime for it.
- `SetMockMode` flips the flag once, then for the receiver and every
running monitor: ends the epoch, waits for admitted calls, clears
alerts, forgets fixture agents (when leaving mock mode) and resets state
(`endMockModeEpoch`). Only after that does it restart the sampler, polls
and discovery of each monitor in the set (`resumeAfterMockModeSwitch`).
A monitor that has not joined yet gets its runtime from its own `Start`.
- Each monitor records the mode its alerts and state belong to
(`mockModeAligned`, set in `New` from the same read that decides whether
persisted alerts are restored) and the number of switches made so far
(`mockModeSwitchesSeen`). If it missed switches between `New` and
`Start`, it leaves what it holds before joining, even after a round trip
that brought the flag back. A call for the mode the flag already holds
still moves any reached monitor that belongs to the other mode, and
leaves the rest alone.
Callers are unchanged and no API surface moves.
## Is this reachable in release builds?
Barely. In a `release` build `mockruntime.ValidateEnablement` refuses
mock fixtures unless `SetReleaseFixturesAuthorized(true)` was called.
Its only caller is `syncReleaseDemoFixtureRuntime`, which requires
`DemoMode` plus the `demo_fixtures` entitlement and runs only for the
default organization. Several tenant monitors in mock mode would also
need more than one organization's monitor running on that demo instance.
Demo mode's read-only middleware normally blocks creating organizations
and the mock-mode API's POST. It does forward any request carrying
`Upgrade: websocket` whatever its method; that bypass predates this
change and is being fixed separately. So in practice this is a
dev/mock-stack problem, and the fix stays inside the monitoring package.
## Tests
New tests in `internal/monitoring/monitor_mock_alerts_test.go`:
- `TestLeavingMockModeClearsEveryRunningTenantMonitor` and
`TestEnteringMockModeClearsEveryRunningTenantMonitor` start the default
and one other tenant monitor through `MultiTenantMonitor`, as the server
does, and wait until both have joined. They raise alerts in both, switch
through one, and check both.
- `TestMonitorStartedAfterAMockModeSwitchLeavesTheModeItWasBuiltIn`
covers a switch that lands between `New` and `Start`.
- `TestMonitorStartedAfterAMockModeRoundTripDropsWhatItRaisedMeanwhile`
covers two switches in that window, with fixture alerts raised in
between.
- `TestSetMockModeAlignsAMonitorWhoseModeTheFlagAlreadyHolds` covers a
call for the mode the flag already holds.
- `TestMockModeSwitchLeavesAnUnjoinedMonitorsRuntimeToItsStart` covers a
switch reaching a monitor that has not joined.
One more is in
`internal/monitoring/monitor_host_agent_removal_lifecycle_test.go`:
`TestLeavingMockModeReleasesFixtureAgentNodeLinksOnEveryRunningMonitor`.
It switches through the default monitor and checks that the other
tenant's fixture node is no longer suppressed by a fixture agent's
leftover node link.
Negative controls (`go test -overlay`):
| Control | Failing tests | What happened |
|---|---|---|
| Fence-PR runtime | Leaving, entering, node-link | Default monitor kept
69–104 fixture alerts (varies by run) and 69 fixture agents; org-b kept
its live alert; org-b's `mock-cluster-1-pve1` stayed suppressed |
| Set kept, switch reaches only the receiver | Same three | Same
failures |
| No realign at join |
`TestMonitorStartedAfterAMockModeSwitchLeavesTheModeItWasBuiltIn` | |
| Join compares only the mode, not the switch count |
`TestMonitorStartedAfterAMockModeRoundTripDropsWhatItRaisedMeanwhile` |
Started in live mode with the fixture node's memory alert |
| Plain early return when the flag already matches |
`TestSetMockModeAlignsAMonitorWhoseModeTheFlagAlreadyHolds` | |
| Resume ignores set membership |
`TestMockModeSwitchLeavesAnUnjoinedMonitorsRuntimeToItsStart` | |
Every control failed only the tests it targets.
Also run: the full `internal/monitoring` package; `-race -count=3` on
the new tests plus the existing `TestLeavingMockMode*` and
`TestMockModeFence*`; and the mock-mode tests in `internal/api` and
`internal/api/configapi`.
rcourtman
cfa026249f