Switch every running tenant monitor when mock mode flips

Mock mode is one process-wide flag, but SetMockMode ended the epoch,
cleared the alerts, forgot the fixture agents and reset the state of
only the monitor it was called on. The server runs one monitor per
organization, each with its own alert manager and its own mock alert
pass. A switch made through one of them (the mock-mode API uses the
requesting tenant's monitor, the demo-fixture licence sync the default
one) left every other running monitor holding what it raised in the mode
the process had left: fixture alerts and fixture-agent node links in
live mode, live alerts in mock mode, until the 24-hour stale sweep.

Monitors now join a process-wide set of running monitors when Start runs
and leave it when Start returns, under the same mutex that already
serialized switches. SetMockMode flips the flag once, then ends the
epoch of the monitor it was called on and of every running monitor: it
waits for their admitted calls, clears their alerts, forgets their
fixture agents and resets their state. Only then does it restart each
running monitor's sampler, polls and discovery. Start picks the mock
metrics sampler or discovery while it holds that mutex, so no switch can
land between reading the mode and starting the runtime for it, and a
switch leaves the runtime of a monitor that has not joined yet to its
own Start. A monitor records the mode and the switch count it was built
under. If it missed switches between New and Start, it leaves what it
holds before it joins, dropping live alerts New restored or fixture
alerts a read path raised meanwhile, even when a round trip brought the
flag back. A call for the mode the flag already holds still moves a
reached monitor whose alerts belong to the other mode, and leaves the
rest alone.

Release builds reach this only on an instance that runs DemoMode with
the demo_fixtures entitlement, since release ValidateEnablement refuses
mock fixtures otherwise, and that has more than one organization's
monitor running. The licence sync only switches the default
organization. So the fix stays inside the monitoring package. The
callers are unchanged and no API surface moves.

New tests start the default and one other tenant monitor through
MultiTenantMonitor, as the server does, and switch through one of them.
Against the per-monitor switch, the default monitor kept its fixture
alerts (69 to 104 across runs) and all 69 fixture agents after mock mode
was left through the other tenant. That tenant kept its live alert after
mock mode was entered through the default monitor, and its fixture node
stayed suppressed by a fixture agent's leftover node link. Four more
tests fail if Start joins without leaving the mode its monitor was built
in, if Start compares only the mode after a round trip, if a call for
the flag's current mode returns without moving such a monitor, or if a
switch starts the runtime of a monitor that has not joined.

docs/release-control/v6/internal/subsystems/agent-lifecycle.md

@@ -754,6 +754,15 @@ not fenced: a live report already past its mock-mode check when mock mode is
 switched on can still be evaluated after the clear, and its alerts can persist
 until mock mode is left.
 
+Every tenant monitor registers its own fixture agents, and their node links,
+in its own alert manager, so the toggle forgets the fixture agents of every
+running monitor, not only those of the monitor it was called through. A link
+left behind would keep owning its fixture node's usage alerts after mock mode
+ends. `TestLeavingMockModeReleasesFixtureAgentNodeLinksOnEveryRunningMonitor`
+in `internal/monitoring/monitor_host_agent_removal_lifecycle_test.go` switches
+through the default monitor and fails if the other tenant's node stays
+suppressed.
+
 Physical-disk evidence collected by a host agent must survive projection back
 into monitoring's models. Absent evidence has to carry its declared sentinel
 rather than a zero value that reads as a real measurement: an absent

docs/release-control/v6/internal/subsystems/monitoring.md

@@ -1855,11 +1855,33 @@ a live report or poll already past its mock-mode check when mock mode is
 switched on can still evaluate after that clear, and its alerts can persist
 until mock mode is left.
 
-The fence and the clear are per monitor while the mode is process-wide. A
-tenant monitor other than the one `SetMockMode` was called on neither drains
-nor clears, so fixture alerts it raised outlive the switch as they did before
-the fence; switching mock mode with several tenant monitors running is not
-covered.
+The mode is process-wide while each monitor keeps its own fence, alert manager,
+fixture agents and state, and the server runs one monitor per organization, so
+`SetMockMode` switches every running monitor, whichever tenant's monitor it is
+called on (the mock-mode API calls the requesting tenant's, the demo-fixture
+licence sync the default organization's). `Start` joins a process-wide set of
+running monitors under `mockModeSwitchMu`, and leaves it when its loop returns.
+A switch flips the mode once, then ends each monitor's epoch, waits for its
+admitted calls, clears its alerts, forgets its fixture agents when leaving mock
+mode and resets its state, and only then restarts the mode-dependent runtime of
+each monitor in the set. `Start` sets its runtime context before it joins and
+chooses the mock metrics sampler or discovery while it joins, under the same
+lock, so no switch lands between reading the mode and starting the runtime for
+it, and a switch that reaches a monitor not yet joined leaves the runtime to
+its `Start`. A monitor that missed switches between `New` and `Start` first
+leaves what it holds: `New` records the mode and the number of switches made so
+far, and `Start` compares both, so even a round trip that brought the flag back
+drops the fixture alerts a read path raised meanwhile, as well as the live
+alerts `New` restored outside mock mode. A monitor that is not running is
+switched only when the switch is made through it; a call for the mode the flag
+already holds still moves any reached monitor whose alerts belong to the other
+mode, and leaves the rest alone.
+
+Release builds reach several tenant monitors in mock mode only on an instance
+running `DemoMode` with the `demo_fixtures` entitlement (release
+`ValidateEnablement` refuses mock fixtures otherwise) and with more than one
+organization's monitor running; the licence sync switches only for the
+default organization.
 
 The proofs park a real evaluation at a fixed point, switch modes and release
 it. `TestLeavingMockModeRefusesTheRestOfAnInFlightPass` holds a mock pass at its
@@ -1889,7 +1911,25 @@ previous-state carry returns fixture nodes or guests. All eight are in
 `TestStoreRefreshTakesItsMockModeScopeBeforeReadingState` in
 `internal/monitoring/canonical_guardrails_test.go` rejects a call that reads
 state in an argument ahead of `m.mockModeFence.begin()` and pins the
-end-epoch-then-clear order in `SetMockMode`.
+end-epoch-then-clear order in `endMockModeEpoch`, which `SetMockMode` runs for
+every monitor it reaches.
+
+`TestLeavingMockModeClearsEveryRunningTenantMonitor` and
+`TestEnteringMockModeClearsEveryRunningTenantMonitor` start the default and one
+other tenant monitor through `MultiTenantMonitor`, wait until both have joined,
+raise alerts in both, switch through one of them and check both. Against a
+switch that reached only its own monitor, the default monitor kept its fixture
+alerts (69 to 104 across runs) and all 69 fixture agents after leaving mock
+mode through the other tenant, and that tenant kept its live alert after mock
+mode was entered through the default monitor. `TestMonitorStartedAfterAMockModeSwitchLeavesTheModeItWasBuiltIn`
+fails if `Start` joins without leaving the mode its monitor was built in, and
+`TestSetMockModeAlignsAMonitorWhoseModeTheFlagAlreadyHolds` fails if a call for
+the flag's current mode returns without moving such a monitor, and
+`TestMockModeSwitchLeavesAnUnjoinedMonitorsRuntimeToItsStart` fails if a switch
+starts the runtime of a monitor whose `Start` has not joined, and
+`TestMonitorStartedAfterAMockModeRoundTripDropsWhatItRaisedMeanwhile` fails if
+`Start` compares only the mode. All six are in
+`internal/monitoring/monitor_mock_alerts_test.go`.
 
 The disk evaluation runs on the mock alert tick, not the poller's disk
 interval (five minutes by default, `PhysicalDiskPollingMinutes` per instance).

internal/monitoring/canonical_guardrails_test.go

@@ -543,8 +543,12 @@ func TestStoreRefreshTakesItsMockModeScopeBeforeReadingState(t *testing.T) {
 	if err != nil {
 		t.Fatalf("failed to read monitor.go: %v", err)
 	}
-	if got := strings.Count(string(data), "m.mockModeFence.advance()\n\t\tm.alertManager.ClearActiveAlerts()"); got != 2 {
-		t.Fatalf("SetMockMode must end the mock-mode epoch immediately before clearing alerts in both directions, found %d", got)
+	source := string(data)
+	if got := strings.Count(source, "m.mockModeFence.advance()\n\tm.alertManager.ClearActiveAlerts()"); got != 1 {
+		t.Fatalf("endMockModeEpoch must end the mock-mode epoch immediately before clearing alerts, found %d", got)
+	}
+	if !strings.Contains(source, "for _, monitor := range monitors {\n\t\tmonitor.endMockModeEpoch(enable)\n\t}") {
+		t.Fatal("SetMockMode must end the epoch of every monitor the switch reaches")
 	}
 }
 
@@ -1664,16 +1668,11 @@ func TestMonitorSetMockModeAuthorizesBeforeResettingRuntimeState(t *testing.T) {
 	if err != nil {
 		t.Fatalf("failed to read monitor.go: %v", err)
 	}
-	source := string(data)
-	requiredSnippets := []string{
-		"if err := mock.SetEnabled(true); err != nil {",
-		"if err := mock.SetEnabled(false); err != nil {",
-		"return err",
-	}
-	for _, snippet := range requiredSnippets {
-		if !strings.Contains(source, snippet) {
-			t.Fatalf("monitor.go must contain %q", snippet)
-		}
+	source := strings.Join(strings.Fields(string(data)), " ")
+	authorize := strings.Index(source, "if err := mock.SetEnabled(enable); err != nil { return err }")
+	reset := strings.Index(source, "monitor.endMockModeEpoch(enable)")
+	if authorize < 0 || reset < 0 || authorize > reset {
+		t.Fatal("SetMockMode must return the mock.SetEnabled error before it ends any monitor's epoch or resets its state")
 	}
 }
 

internal/monitoring/mock_mode_switch.go

@@ -0,0 +1,75 @@
+package monitoring
+
+import (
+	"sync"
+	"sync/atomic"
+
+	"github.com/rcourtman/pulse-go-rewrite/internal/mock"
+)
+
+// Mock mode is process-wide (mock.SetEnabled flips one package flag), while
+// every Monitor keeps its own alert manager, fixture agents, state and
+// mock-mode fence, and the server runs one Monitor per organization. A switch
+// made through one monitor therefore has to reach every running monitor, or
+// the others keep what they raised in the mode the process left: fixture
+// alerts and fixture-agent node links in live mode, live alerts in mock mode,
+// until the 24-hour stale sweep.
+//
+// mockModeSwitchMu serializes switches and guards the set of running monitors
+// and each monitor's mockModeAligned and mockModeSwitchesSeen. A switch is
+// several steps per monitor (end the epoch, clear, reset), so two interleaved
+// switches could clear what the later one just admitted. Start joins the set
+// under the same lock, so a monitor joins either before a switch, which then
+// switches it with the others, or after it, and starts in the new mode.
+//
+// mockModeSwitchCount counts the flips SetMockMode made. A monitor that missed
+// flips while it was not running can hold the alerts and state of a mode the
+// process left even when a later flip brought the flag back to the mode it
+// records (a read path evaluated fixture data before Start), so Start
+// compares the count as well as the mode.
+var (
+	mockModeSwitchMu        sync.Mutex
+	mockModeRunningMonitors = make(map[*Monitor]struct{})
+	mockModeSwitchCount     atomic.Uint64
+)
+
+// mockModeSwitchTargetsLocked returns the monitors a switch made through m
+// reaches: m itself, which need not be running, and every running monitor.
+func mockModeSwitchTargetsLocked(m *Monitor) []*Monitor {
+	targets := make([]*Monitor, 0, len(mockModeRunningMonitors)+1)
+	targets = append(targets, m)
+	for monitor := range mockModeRunningMonitors {
+		if monitor != m {
+			targets = append(targets, monitor)
+		}
+	}
+	return targets
+}
+
+// joinMockModeSwitches adds m to the monitors every switch reaches, then runs
+// start with the current mode while no switch can run, so the mode-dependent
+// runtime start chooses (the mock metrics sampler or discovery) cannot be
+// overtaken by a switch. It returns the function that removes m again.
+//
+// A switch can land between New and Start, while m is not yet running. New
+// restores persisted alerts only outside mock mode, so m would then carry live
+// alerts into mock mode, or fixture alerts that a read path raised before
+// Start into live mode; m first leaves what it holds, as the switches it
+// missed would have done.
+func (m *Monitor) joinMockModeSwitches(start func(mockEnabled bool)) (leave func()) {
+	mockModeSwitchMu.Lock()
+	defer mockModeSwitchMu.Unlock()
+
+	enabled := mock.IsMockEnabled()
+	if m.mockModeAligned != enabled || m.mockModeSwitchesSeen != mockModeSwitchCount.Load() {
+		m.endMockModeEpoch(enabled)
+	}
+	mockModeRunningMonitors[m] = struct{}{}
+	start(enabled)
+
+	return func() {
+		mockModeSwitchMu.Lock()
+		defer mockModeSwitchMu.Unlock()
+		delete(mockModeRunningMonitors, m)
+	}
+}

internal/monitoring/monitor.go

@@ -13,6 +13,7 @@ import (
 	"os"
 	"path/filepath"
 	"reflect"
+	"slices"
 	"sort"
 	"strings"
 	"sync"
@@ -67,9 +68,9 @@ func keepRealPollingInMockMode() bool {
 // process that boots with mock mode already enabled never runs that path, so
 // alerts raised against real infrastructure are restored from disk and served
 // beside fixture data until something clears them.
-func alertManagerRestoreOptions() []alerts.ManagerOption {
+func alertManagerRestoreOptions(mockEnabled bool) []alerts.ManagerOption {
 	options := []alerts.ManagerOption{alerts.WithDurableAlertStore()}
-	if mock.IsMockEnabled() {
+	if mockEnabled {
 		options = append(options, alerts.WithoutPersistedAlertRestore())
 	}
 	return options
@@ -1283,6 +1284,8 @@ type Monitor struct {
 	mockHostAgentsMu          sync.Mutex
 	mockHostAgents            map[string]models.Host   // Fixture agents evaluated by the last mock alert pass
 	mockModeFence             mockModeFence            // Keeps mode-dependent alert evaluations inside the epoch they read in
+	mockModeAligned           bool                     // Mock mode the alerts and state belong to; guarded by mockModeSwitchMu
+	mockModeSwitchesSeen      uint64                   // mockModeSwitchCount when they were last aligned; guarded by mockModeSwitchMu
 	dockerChecker             DockerChecker            // Optional Docker checker for LXC containers
 	dockerCheckerConfiguredAt time.Time                // Last time the Docker checker was configured
 	dockerCheckAllowedVMIDs   map[int]struct{}         // Optional VMID allowlist gating the LXC Docker socket probe; empty means all guests
@@ -1771,6 +1774,12 @@ func New(cfg *config.Config) (*Monitor, error) {
 		DataDir: cfg.DataPath,
 	})
 
+	// One read decides the alert restore and records the mode it belongs to,
+	// so Start can tell whether a mock-mode switch landed in between. The
+	// count is read first: a switch flips the flag before it counts, so one
+	// that overlaps these reads leaves a stale count, never a stale mode.
+	switchesSeen := mockModeSwitchCount.Load()
+	builtInMockMode := mock.IsMockEnabled()
 	m := &Monitor{
 		config:                     cfg,
 		state:                      models.NewState(),
@@ -1804,7 +1813,9 @@ func New(cfg *config.Config) (*Monitor, error) {
 		rateTracker:                NewRateTracker(),
 		metricsHistory:             NewMetricsHistory(1000, 24*time.Hour), // Keep up to 1000 points (~8h @ 30s)
 		metricsStore:               metricsStore,                          // Persistent SQLite storage
-		alertManager:               alerts.NewManagerWithDataDir(cfg.DataPath, alertManagerRestoreOptions()...),
+		alertManager:               alerts.NewManagerWithDataDir(cfg.DataPath, alertManagerRestoreOptions(builtInMockMode)...),
+		mockModeAligned:            builtInMockMode,
+		mockModeSwitchesSeen:       switchesSeen,
 		incidentStore:              incidentStore,
 		notificationMgr:            notifications.NewNotificationManagerWithDeferredQueue(cfg.PublicURL, cfg.DataPath),
 		deadMan:                    newDeadManRuntime(config.ResolveRuntimeDataDir(cfg.DataPath)),
@@ -2086,33 +2097,13 @@ func (m *Monitor) Start(ctx context.Context, wsHub *websocket.Hub) {
 	// PVE-proxied PBS backup points to prevent duplicate recovery entries.
 	m.purgeStalePVEPBSBackupsBestEffort(ctx)
 
-	if mock.IsMockEnabled() {
-		m.startMockMetricsSampler(ctx)
-	}
-
-	// Initialize and start discovery service if enabled
-	if mock.IsMockEnabled() {
-		log.Info().Msg("mock mode enabled - skipping discovery service")
-		m.discoveryService = nil
-	} else if m.config.DiscoveryEnabled {
-		discoverySubnet := m.config.DiscoverySubnet
-		if discoverySubnet == "" {
-			discoverySubnet = "auto"
-		}
-		cfgProvider := func() config.DiscoveryConfig {
-			return m.discoveryConfigSnapshot()
-		}
-		m.discoveryService = discovery.NewService(wsHub, 5*time.Minute, discoverySubnet, cfgProvider)
-		if m.discoveryService != nil {
-			m.discoveryService.Start(ctx)
-			log.Info().Msg("discovery service initialized and started")
-		} else {
-			log.Error().Msg("failed to initialize discovery service")
-		}
-	} else {
-		log.Info().Msg("discovery service disabled by configuration")
-		m.discoveryService = nil
-	}
+	// From here every mock-mode switch reaches this monitor. The sampler or
+	// discovery is chosen while no switch can run, so a switch cannot land
+	// between reading the mode and starting the runtime for it.
+	leaveMockModeSwitches := m.joinMockModeSwitches(func(mockEnabled bool) {
+		m.startModeRuntime(ctx, wsHub, mockEnabled)
+	})
+	defer leaveMockModeSwitches()
 
 	// Set up alert callbacks. Projection replay is deliberately absent here:
 	// the canonical resource store is not attached yet, so a replay now could
@@ -2251,6 +2242,38 @@ func (m *Monitor) Start(ctx context.Context, wsHub *websocket.Hub) {
 	}
 }
 
+// startModeRuntime starts the mock metrics sampler in mock mode, or the
+// discovery service in live mode when it is enabled.
+func (m *Monitor) startModeRuntime(ctx context.Context, wsHub *websocket.Hub, mockEnabled bool) {
+	if mockEnabled {
+		m.startMockMetricsSampler(ctx)
+	}
+
+	// Initialize and start discovery service if enabled
+	if mockEnabled {
+		log.Info().Msg("mock mode enabled - skipping discovery service")
+		m.discoveryService = nil
+	} else if m.config.DiscoveryEnabled {
+		discoverySubnet := m.config.DiscoverySubnet
+		if discoverySubnet == "" {
+			discoverySubnet = "auto"
+		}
+		cfgProvider := func() config.DiscoveryConfig {
+			return m.discoveryConfigSnapshot()
+		}
+		m.discoveryService = discovery.NewService(wsHub, 5*time.Minute, discoverySubnet, cfgProvider)
+		if m.discoveryService != nil {
+			m.discoveryService.Start(ctx)
+			log.Info().Msg("discovery service initialized and started")
+		} else {
+			log.Error().Msg("failed to initialize discovery service")
+		}
+	} else {
+		log.Info().Msg("discovery service disabled by configuration")
+		m.discoveryService = nil
+	}
+}
+
 // wireExternalAlertCallbacks installs callbacks whose effects leave the alert
 // manager. It is safe to call again: these are single callback slots, so Start
 // replaces the constructor-time escalation closure with one carrying its hub.
@@ -4577,65 +4600,93 @@ func (m *Monitor) broadcastEscalatedAlert(hub *websocket.Hub, alert *alerts.Aler
 	hub.BroadcastAlertToTenant(m.GetOrgID(), alert)
 }
 
-// mockModeSwitchMu serializes mock-mode switches. The mode is process-wide,
-// and a switch is several steps (flip, end the epoch, clear, reset state), so
-// two interleaved switches could clear what the later one just admitted.
-var mockModeSwitchMu sync.Mutex
-
-// SetMockMode switches between mock data and real infrastructure data at runtime.
-// It must not be called from inside an alert evaluation, whose completion it
-// may wait for.
+// SetMockMode switches between mock data and real infrastructure data at
+// runtime. The mode is process-wide, so the switch reaches m and every running
+// monitor (see mockModeSwitchMu), whichever tenant's monitor it was made
+// through. It must not be called from inside an alert evaluation of any of
+// them, whose completion it may wait for.
 func (m *Monitor) SetMockMode(enable bool) error {
 	mockModeSwitchMu.Lock()
 	defer mockModeSwitchMu.Unlock()
 
-	current := mock.IsMockEnabled()
-	if current == enable {
-		log.Info().Bool("mockMode", enable).Msg("mock mode already in desired state")
-		return nil
+	monitors := mockModeSwitchTargetsLocked(m)
+	flip := mock.IsMockEnabled() != enable
+	if !flip {
+		// The flag already holds the mode, but a monitor that was not running
+		// during an earlier switch can still hold the other mode's alerts
+		// and state; only those leave it.
+		monitors = slices.DeleteFunc(monitors, func(monitor *Monitor) bool {
+			return monitor.mockModeAligned == enable
+		})
+		if len(monitors) == 0 {
+			log.Info().Bool("mockMode", enable).Msg("mock mode already in desired state")
+			return nil
+		}
 	}
 
-	// Every evaluation of mode-dependent data that started before the flip
-	// must finish or be refused before the clear, or it reopens alerts for
-	// the side the monitor just left (see mockModeFence).
+	for _, monitor := range monitors {
+		monitor.stopMockMetricsSampler()
+	}
+	if flip {
+		if err := mock.SetEnabled(enable); err != nil {
+			return err
+		}
+		mockModeSwitchCount.Add(1)
+	}
+	for _, monitor := range monitors {
+		monitor.endMockModeEpoch(enable)
+	}
+	for _, monitor := range monitors {
+		_, running := mockModeRunningMonitors[monitor]
+		monitor.resumeAfterMockModeSwitch(enable, running)
+	}
 	if enable {
-		m.stopMockMetricsSampler()
-		if err := mock.SetEnabled(true); err != nil {
-			return err
-		}
-		m.mockModeFence.advance()
-		m.alertManager.ClearActiveAlerts()
-		m.mu.Lock()
-		m.resetStateLocked()
-		m.metricsHistory.Reset()
-		m.mu.Unlock()
-		m.StopDiscoveryService()
-		m.mu.RLock()
-		ctx := m.runtimeCtx
-		m.mu.RUnlock()
-		if ctx != nil {
-			m.startMockMetricsSampler(ctx)
-		}
-		log.Info().Msg("switched monitor to mock mode")
+		log.Info().Int("monitors", len(monitors)).Msg("switched monitors to mock mode")
 	} else {
-		m.stopMockMetricsSampler()
-		if err := mock.SetEnabled(false); err != nil {
-			return err
-		}
-		m.mockModeFence.advance()
-		m.alertManager.ClearActiveAlerts()
-		m.forgetMockHostAgents()
-		m.mu.Lock()
-		m.resetStateLocked()
-		m.metricsHistory.Reset()
-		m.mu.Unlock()
-		log.Info().Msg("switched monitor to real data mode")
+		log.Info().Int("monitors", len(monitors)).Msg("switched monitors to real data mode")
 	}
+	return nil
+}
 
+// endMockModeEpoch leaves the mode the monitor's alerts and state belong to
+// once the process flag already holds the new one. Every evaluation of
+// mode-dependent data that started before the flip must finish or be refused
+// before the clear, or it reopens alerts for the side the monitor just left
+// (see mockModeFence). The caller holds mockModeSwitchMu.
+func (m *Monitor) endMockModeEpoch(enable bool) {
+	m.mockModeFence.advance()
+	m.alertManager.ClearActiveAlerts()
+	if !enable {
+		m.forgetMockHostAgents()
+	}
+	m.mu.Lock()
+	m.resetStateLocked()
+	m.metricsHistory.Reset()
+	m.mu.Unlock()
+	if enable {
+		m.StopDiscoveryService()
+	}
+	m.mockModeAligned = enable
+	m.mockModeSwitchesSeen = mockModeSwitchCount.Load()
+}
+
+// resumeAfterMockModeSwitch restarts the mode-dependent runtime of a monitor
+// whose Start loop has joined the switches, once every monitor has ended its
+// epoch. A monitor that has not joined yet starts that runtime itself when it
+// does (Start sets its runtime context first), and one whose loop has stopped
+// only broadcasts.
+func (m *Monitor) resumeAfterMockModeSwitch(enable, running bool) {
 	m.mu.RLock()
 	ctx := m.runtimeCtx
 	hub := m.wsHub
 	m.mu.RUnlock()
+	if !running || (ctx != nil && ctx.Err() != nil) {
+		ctx = nil
+	}
+
+	if enable && ctx != nil {
+		m.startMockMetricsSampler(ctx)
+	}
 
 	if hub != nil {
 		m.broadcastCurrentState(hub)
@@ -4653,8 +4704,6 @@ func (m *Monitor) SetMockMode(enable bool) error {
 			go m.StartDiscoveryService(ctx, hub, m.config.DiscoverySubnet)
 		}
 	}
-
-	return nil
 }
 
 func (m *Monitor) resetStateLocked() {

internal/monitoring/monitor_host_agent_removal_lifecycle_test.go

@@ -11,6 +11,7 @@ import (
 	"github.com/rcourtman/pulse-go-rewrite/internal/alerts"
 	"github.com/rcourtman/pulse-go-rewrite/internal/alerts/eventlog"
 	"github.com/rcourtman/pulse-go-rewrite/internal/config"
+	"github.com/rcourtman/pulse-go-rewrite/internal/mock"
 	"github.com/rcourtman/pulse-go-rewrite/internal/models"
 	"github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources"
 	agentshost "github.com/rcourtman/pulse-go-rewrite/pkg/agents/host"
@@ -857,3 +858,45 @@ func TestMockHostAgentLeavingFixtureUsesRemovalLifecycle(t *testing.T) {
 	}
 	t.Fatal("node pve9 raised no memory alert after its agent left the mock estate")
 }
+
+func TestLeavingMockModeReleasesFixtureAgentNodeLinksOnEveryRunningMonitor(t *testing.T) {
+	setMockSamplerTestEnv(t, time.Hour, 5*time.Minute)
+	pinDefaultMockEstate(t)
+	mustSetMockEnabled(t, true)
+	defaultMonitor, tenantMonitor := startTenantMonitors(t)
+
+	// org-b's start pass registers the fixture agents in org-b's own alert
+	// manager, each owning the usage alerts of the fixture node it runs on.
+	var linked models.Node
+	waitForCondition(t, 30*time.Second, func() bool {
+		tenantMonitor.mockHostAgentsMu.Lock()
+		defer tenantMonitor.mockHostAgentsMu.Unlock()
+		for _, host := range tenantMonitor.mockHostAgents {
+			if host.Status != "online" || host.LinkedNodeID == "" {
+				continue
+			}
+			for _, node := range mock.CurrentFixtureGraph().State.Nodes {
+				if node.ID == host.LinkedNodeID {
+					linked = node
+					return true
+				}
+			}
+		}
+		return false
+	}, "org-b registered no online fixture agent linked to a fixture node")
+
+	// The demo-fixture licence sync switches through the default monitor.
+	mustSetMonitorMockMode(t, defaultMonitor, false)
+
+	// org-b's fixture agents go through the removal lifecycle too, so the
+	// node an agent covered owns its metric alerts again.
+	linked.Status = "online"
+	linked.Memory = models.Memory{Total: 64 << 30, Used: 60 << 30, Free: 4 << 30, Usage: 93.75}
+	tenantMonitor.alertManager.CheckNode(linked)
+	for _, alert := range tenantMonitor.alertManager.GetActiveAlerts() {
+		if alert.ResourceID == linked.ID && alert.Type == "memory" {
+			return
+		}
+	}
+	t.Fatalf("org-b's node %s raised no memory alert: its fixture agent's node link outlived a switch made through the default monitor", linked.ID)
+}

internal/monitoring/monitor_mock_alerts_test.go

@@ -15,6 +15,7 @@ import (
 
 	"github.com/rcourtman/pulse-go-rewrite/internal/alerts"
 	"github.com/rcourtman/pulse-go-rewrite/internal/alerts/eventlog"
+	"github.com/rcourtman/pulse-go-rewrite/internal/config"
 	"github.com/rcourtman/pulse-go-rewrite/internal/mock"
 	"github.com/rcourtman/pulse-go-rewrite/internal/models"
 	"github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources"
@@ -809,6 +810,254 @@ func TestLeavingMockModeDiscardsAnInFlightConnectionCheck(t *testing.T) {
 	}
 }
 
+// startTenantMonitors starts the default organization's monitor and one
+// other tenant's through MultiTenantMonitor, as the server does, each with its
+// own alert manager, and waits until both monitoring loops run. An hour
+// between ticks leaves each loop the pass it runs on start, so the test drives
+// every later pass itself.
+func startTenantMonitors(t *testing.T) (defaultMonitor, tenantMonitor *Monitor) {
+	t.Helper()
+	mtp, _ := newTestTenantPersistence(t)
+	mtm := NewMultiTenantMonitor(&config.Config{DataPath: t.TempDir(), PVEPollingInterval: time.Hour}, mtp, nil)
+	t.Cleanup(mtm.Stop)
+	mtm.SetMonitorInitializer(func(monitor *Monitor) {
+		cfg := monitor.alertManager.GetConfig()
+		cfg.Enabled = true
+		cfg.TimeThresholds = map[string]int{}
+		cfg.MetricTimeThresholds = nil
+		cfg.BackupDefaults.Enabled = false
+		cfg.AgentDefaults.Memory = &alerts.HysteresisThreshold{Trigger: 1, Clear: 0.5}
+		cfg.NodeDefaults.Memory = &alerts.HysteresisThreshold{Trigger: 1, Clear: 0.5}
+		monitor.alertManager.UpdateConfig(cfg)
+	})
+	if err := mtp.SaveOrganization(&models.Organization{ID: "org-b", DisplayName: "Org B"}); err != nil {
+		t.Fatalf("SaveOrganization(org-b): %v", err)
+	}
+	monitors := make([]*Monitor, 0, 2)
+	for _, orgID := range []string{"default", "org-b"} {
+		monitor, err := mtm.GetMonitor(orgID)
+		if err != nil {
+			t.Fatalf("GetMonitor(%s): %v", orgID, err)
+		}
+		// Start sets its runtime context before it joins the monitors a
+		// switch reaches, so wait for the join itself.
+		waitForCondition(t, 30*time.Second, func() bool {
+			mockModeSwitchMu.Lock()
+			defer mockModeSwitchMu.Unlock()
+			_, running := mockModeRunningMonitors[monitor]
+			return running
+		}, orgID+" monitoring loop did not join mock-mode switches")
+		monitors = append(monitors, monitor)
+	}
+	return monitors[0], monitors[1]
+}
+
+func mockHostAgentCount(monitor *Monitor) int {
+	monitor.mockHostAgentsMu.Lock()
+	defer monitor.mockHostAgentsMu.Unlock()
+	return len(monitor.mockHostAgents)
+}
+
+func TestLeavingMockModeClearsEveryRunningTenantMonitor(t *testing.T) {
+	setMockSamplerTestEnv(t, time.Hour, 5*time.Minute)
+	pinDefaultMockEstate(t)
+	mustSetMockEnabled(t, true)
+	defaultMonitor, tenantMonitor := startTenantMonitors(t)
+
+	// Each loop runs a mock pass on start and raises the fixture's alerts in
+	// its own alert manager.
+	monitors := map[string]*Monitor{"default": defaultMonitor, "org-b": tenantMonitor}
+	for name, monitor := range monitors {
+		waitForCondition(t, 30*time.Second, func() bool {
+			return len(monitor.alertManager.GetActiveAlerts()) > 0 && mockHostAgentCount(monitor) > 0
+		}, name+" monitor raised no fixture alerts in mock mode")
+	}
+
+	// The mock-mode API switches through the requesting tenant's monitor.
+	mustSetMonitorMockMode(t, tenantMonitor, false)
+
+	for name, monitor := range monitors {
+		if active := monitor.alertManager.GetActiveAlerts(); len(active) != 0 {
+			t.Errorf("%s monitor kept %d fixture alerts after mock mode was left through org-b: %s",
+				name, len(active), describeAlerts(active))
+		}
+		if count := mockHostAgentCount(monitor); count != 0 {
+			t.Errorf("%s monitor still tracks %d fixture agents in live mode", name, count)
+		}
+	}
+}
+
+func TestEnteringMockModeClearsEveryRunningTenantMonitor(t *testing.T) {
+	setMockSamplerTestEnv(t, time.Hour, 5*time.Minute)
+	pinDefaultMockEstate(t)
+	defaultMonitor, tenantMonitor := startTenantMonitors(t)
+
+	monitors := map[string]*Monitor{"default": defaultMonitor, "org-b": tenantMonitor}
+	for name, monitor := range monitors {
+		monitor.alertManager.CheckNode(models.Node{
+			ID:       name + "-live-node",
+			Name:     "live-node",
+			Instance: "live",
+			Status:   "online",
+			Memory:   models.Memory{Total: 64 << 30, Used: 60 << 30, Free: 4 << 30, Usage: 93.75},
+		})
+		if len(monitor.alertManager.GetActiveAlerts()) == 0 {
+			t.Fatalf("live node raised no alert in %s", name)
+		}
+	}
+
+	// The demo-fixture licence sync switches through the default monitor.
+	mustSetMonitorMockMode(t, defaultMonitor, true)
+
+	for name, monitor := range monitors {
+		for _, alert := range monitor.alertManager.GetActiveAlerts() {
+			if alert.ResourceID == name+"-live-node" {
+				t.Errorf("%s kept its live %s alert after mock mode was entered through the default monitor", name, alert.Type)
+			}
+		}
+	}
+}
+
+func TestMonitorStartedAfterAMockModeSwitchLeavesTheModeItWasBuiltIn(t *testing.T) {
+	setMockSamplerTestEnv(t, time.Hour, 5*time.Minute)
+	pinDefaultMockEstate(t)
+	manager := newMockHostAlertTestManager(t)
+	monitor := &Monitor{
+		state:          models.NewState(),
+		alertManager:   manager,
+		metricsHistory: NewMetricsHistory(10, time.Hour),
+	}
+	// Built in live mode: New restored this live alert from disk.
+	manager.CheckNode(models.Node{
+		ID:       "live-node",
+		Name:     "live-node",
+		Instance: "live",
+		Status:   "online",
+		Memory:   models.Memory{Total: 64 << 30, Used: 60 << 30, Free: 4 << 30, Usage: 93.75},
+	})
+	if len(manager.GetActiveAlerts()) == 0 {
+		t.Fatal("live node raised no alert")
+	}
+
+	// Another monitor switched to mock mode before this one's Start joined.
+	mustSetMockEnabled(t, true)
+	started := false
+	leave := monitor.joinMockModeSwitches(func(mockEnabled bool) { started = mockEnabled })
+	defer leave()
+
+	if !started {
+		t.Fatal("Start was not handed mock mode")
+	}
+	if active := manager.GetActiveAlerts(); len(active) != 0 {
+		t.Fatalf("a monitor built in live mode started in mock mode with its live alerts: %s", describeAlerts(active))
+	}
+	mockModeSwitchMu.Lock()
+	_, running := mockModeRunningMonitors[monitor]
+	mockModeSwitchMu.Unlock()
+	if !running {
+		t.Fatal("joined monitor is not reached by later switches")
+	}
+}
+
+func TestMonitorStartedAfterAMockModeRoundTripDropsWhatItRaisedMeanwhile(t *testing.T) {
+	setMockSamplerTestEnv(t, time.Hour, 5*time.Minute)
+	pinDefaultMockEstate(t)
+	newMonitor := func() *Monitor {
+		return &Monitor{
+			state:          models.NewState(),
+			alertManager:   newMockHostAlertTestManager(t),
+			metricsHistory: NewMetricsHistory(10, time.Hour),
+		}
+	}
+	switcher, starting := newMonitor(), newMonitor()
+	// Built in live mode, as New records it, and not running yet.
+	mockModeSwitchMu.Lock()
+	starting.mockModeSwitchesSeen = mockModeSwitchCount.Load()
+	mockModeSwitchMu.Unlock()
+
+	mustSetMonitorMockMode(t, switcher, true)
+	// A read path evaluates fixture data on the monitor before its Start runs.
+	fixture := mock.CurrentFixtureGraph().State.Nodes[0]
+	fixture.Status = "online"
+	fixture.Memory = models.Memory{Total: 64 << 30, Used: 60 << 30, Free: 4 << 30, Usage: 93.75}
+	starting.alertManager.CheckNode(fixture)
+	if len(starting.alertManager.GetActiveAlerts()) == 0 {
+		t.Fatal("fixture node raised no alert")
+	}
+	mustSetMonitorMockMode(t, switcher, false)
+
+	leave := starting.joinMockModeSwitches(func(bool) {})
+	defer leave()
+	if active := starting.alertManager.GetActiveAlerts(); len(active) != 0 {
+		t.Fatalf("a monitor that missed a mock-mode round trip started in live mode with fixture alerts: %s", describeAlerts(active))
+	}
+}
+
+func TestSetMockModeAlignsAMonitorWhoseModeTheFlagAlreadyHolds(t *testing.T) {
+	setMockSamplerTestEnv(t, time.Hour, 5*time.Minute)
+	pinDefaultMockEstate(t)
+	manager := newMockHostAlertTestManager(t)
+	monitor := &Monitor{
+		state:          models.NewState(),
+		alertManager:   manager,
+		metricsHistory: NewMetricsHistory(10, time.Hour),
+	}
+	// Built in live mode, with a live alert, and not running yet.
+	manager.CheckNode(models.Node{
+		ID:       "live-node",
+		Name:     "live-node",
+		Instance: "live",
+		Status:   "online",
+		Memory:   models.Memory{Total: 64 << 30, Used: 60 << 30, Free: 4 << 30, Usage: 93.75},
+	})
+	if len(manager.GetActiveAlerts()) == 0 {
+		t.Fatal("live node raised no alert")
+	}
+
+	// Another monitor's switch already set the flag.
+	mustSetMockEnabled(t, true)
+	mustSetMonitorMockMode(t, monitor, true)
+
+	if active := manager.GetActiveAlerts(); len(active) != 0 {
+		t.Fatalf("SetMockMode(true) left a monitor built in live mode with its live alerts: %s", describeAlerts(active))
+	}
+}
+
+func TestMockModeSwitchLeavesAnUnjoinedMonitorsRuntimeToItsStart(t *testing.T) {
+	setMockSamplerTestEnv(t, time.Hour, 5*time.Minute)
+	pinDefaultMockEstate(t)
+	ctx, cancel := context.WithCancel(context.Background())
+	t.Cleanup(cancel)
+	// Start has set its runtime context but not joined the switches yet.
+	monitor := &Monitor{
+		config:         &config.Config{},
+		state:          models.NewState(),
+		alertManager:   newMockHostAlertTestManager(t),
+		metricsHistory: NewMetricsHistory(10, time.Hour),
+		runtimeCtx:     ctx,
+	}
+	t.Cleanup(monitor.stopMockMetricsSampler)
+
+	mustSetMonitorMockMode(t, monitor, true)
+	monitor.mu.RLock()
+	started := monitor.mockMetricsCancel != nil
+	monitor.mu.RUnlock()
+	if started {
+		t.Fatal("a switch started the runtime of a monitor whose Start has not joined; that runtime belongs to its Start")
+	}
+
+	leave := monitor.joinMockModeSwitches(func(mockEnabled bool) {
+		monitor.startModeRuntime(ctx, nil, mockEnabled)
+	})
+	defer leave()
+	monitor.mu.RLock()
+	started = monitor.mockMetricsCancel != nil
+	monitor.mu.RUnlock()
+	if !started {
+		t.Fatal("joining in mock mode did not start the mock sampler")
+	}
+}
+
 // diskAlertKeys lists the active disk-health and disk-wearout alerts as
 // "type instance/node device" so assertions name the disk, not the alert ID.
 func diskAlertKeys(manager *alerts.Manager) map[string]bool {