diff --git a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md b/docs/release-control/v6/internal/subsystems/agent-lifecycle.md index e7b4f59ab3..7582e7db33 100644 --- a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md +++ b/docs/release-control/v6/internal/subsystems/agent-lifecycle.md @@ -754,6 +754,15 @@ not fenced: a live report already past its mock-mode check when mock mode is switched on can still be evaluated after the clear, and its alerts can persist until mock mode is left. +Every tenant monitor registers its own fixture agents, and their node links, +in its own alert manager, so the toggle forgets the fixture agents of every +running monitor, not only those of the monitor it was called through. A link +left behind would keep owning its fixture node's usage alerts after mock mode +ends. `TestLeavingMockModeReleasesFixtureAgentNodeLinksOnEveryRunningMonitor` +in `internal/monitoring/monitor_host_agent_removal_lifecycle_test.go` switches +through the default monitor and fails if the other tenant's node stays +suppressed. + Physical-disk evidence collected by a host agent must survive projection back into monitoring's models. Absent evidence has to carry its declared sentinel rather than a zero value that reads as a real measurement: an absent diff --git a/docs/release-control/v6/internal/subsystems/monitoring.md b/docs/release-control/v6/internal/subsystems/monitoring.md index 60ff94e27d..7e54def128 100644 --- a/docs/release-control/v6/internal/subsystems/monitoring.md +++ b/docs/release-control/v6/internal/subsystems/monitoring.md @@ -1855,11 +1855,33 @@ a live report or poll already past its mock-mode check when mock mode is switched on can still evaluate after that clear, and its alerts can persist until mock mode is left. -The fence and the clear are per monitor while the mode is process-wide. A -tenant monitor other than the one `SetMockMode` was called on neither drains -nor clears, so fixture alerts it raised outlive the switch as they did before -the fence; switching mock mode with several tenant monitors running is not -covered. +The mode is process-wide while each monitor keeps its own fence, alert manager, +fixture agents and state, and the server runs one monitor per organization, so +`SetMockMode` switches every running monitor, whichever tenant's monitor it is +called on (the mock-mode API calls the requesting tenant's, the demo-fixture +licence sync the default organization's). `Start` joins a process-wide set of +running monitors under `mockModeSwitchMu`, and leaves it when its loop returns. +A switch flips the mode once, then ends each monitor's epoch, waits for its +admitted calls, clears its alerts, forgets its fixture agents when leaving mock +mode and resets its state, and only then restarts the mode-dependent runtime of +each monitor in the set. `Start` sets its runtime context before it joins and +chooses the mock metrics sampler or discovery while it joins, under the same +lock, so no switch lands between reading the mode and starting the runtime for +it, and a switch that reaches a monitor not yet joined leaves the runtime to +its `Start`. A monitor that missed switches between `New` and `Start` first +leaves what it holds: `New` records the mode and the number of switches made so +far, and `Start` compares both, so even a round trip that brought the flag back +drops the fixture alerts a read path raised meanwhile, as well as the live +alerts `New` restored outside mock mode. A monitor that is not running is +switched only when the switch is made through it; a call for the mode the flag +already holds still moves any reached monitor whose alerts belong to the other +mode, and leaves the rest alone. + +Release builds reach several tenant monitors in mock mode only on an instance +running `DemoMode` with the `demo_fixtures` entitlement (release +`ValidateEnablement` refuses mock fixtures otherwise) and with more than one +organization's monitor running; the licence sync switches only for the +default organization. The proofs park a real evaluation at a fixed point, switch modes and release it. `TestLeavingMockModeRefusesTheRestOfAnInFlightPass` holds a mock pass at its @@ -1889,7 +1911,25 @@ previous-state carry returns fixture nodes or guests. All eight are in `TestStoreRefreshTakesItsMockModeScopeBeforeReadingState` in `internal/monitoring/canonical_guardrails_test.go` rejects a call that reads state in an argument ahead of `m.mockModeFence.begin()` and pins the -end-epoch-then-clear order in `SetMockMode`. +end-epoch-then-clear order in `endMockModeEpoch`, which `SetMockMode` runs for +every monitor it reaches. + +`TestLeavingMockModeClearsEveryRunningTenantMonitor` and +`TestEnteringMockModeClearsEveryRunningTenantMonitor` start the default and one +other tenant monitor through `MultiTenantMonitor`, wait until both have joined, +raise alerts in both, switch through one of them and check both. Against a +switch that reached only its own monitor, the default monitor kept its fixture +alerts (69 to 104 across runs) and all 69 fixture agents after leaving mock +mode through the other tenant, and that tenant kept its live alert after mock +mode was entered through the default monitor. `TestMonitorStartedAfterAMockModeSwitchLeavesTheModeItWasBuiltIn` +fails if `Start` joins without leaving the mode its monitor was built in, and +`TestSetMockModeAlignsAMonitorWhoseModeTheFlagAlreadyHolds` fails if a call for +the flag's current mode returns without moving such a monitor, and +`TestMockModeSwitchLeavesAnUnjoinedMonitorsRuntimeToItsStart` fails if a switch +starts the runtime of a monitor whose `Start` has not joined, and +`TestMonitorStartedAfterAMockModeRoundTripDropsWhatItRaisedMeanwhile` fails if +`Start` compares only the mode. All six are in +`internal/monitoring/monitor_mock_alerts_test.go`. The disk evaluation runs on the mock alert tick, not the poller's disk interval (five minutes by default, `PhysicalDiskPollingMinutes` per instance). diff --git a/internal/monitoring/canonical_guardrails_test.go b/internal/monitoring/canonical_guardrails_test.go index fd27c2215b..8a13f8b631 100644 --- a/internal/monitoring/canonical_guardrails_test.go +++ b/internal/monitoring/canonical_guardrails_test.go @@ -543,8 +543,12 @@ func TestStoreRefreshTakesItsMockModeScopeBeforeReadingState(t *testing.T) { if err != nil { t.Fatalf("failed to read monitor.go: %v", err) } - if got := strings.Count(string(data), "m.mockModeFence.advance()\n\t\tm.alertManager.ClearActiveAlerts()"); got != 2 { - t.Fatalf("SetMockMode must end the mock-mode epoch immediately before clearing alerts in both directions, found %d", got) + source := string(data) + if got := strings.Count(source, "m.mockModeFence.advance()\n\tm.alertManager.ClearActiveAlerts()"); got != 1 { + t.Fatalf("endMockModeEpoch must end the mock-mode epoch immediately before clearing alerts, found %d", got) + } + if !strings.Contains(source, "for _, monitor := range monitors {\n\t\tmonitor.endMockModeEpoch(enable)\n\t}") { + t.Fatal("SetMockMode must end the epoch of every monitor the switch reaches") } } @@ -1664,16 +1668,11 @@ func TestMonitorSetMockModeAuthorizesBeforeResettingRuntimeState(t *testing.T) { if err != nil { t.Fatalf("failed to read monitor.go: %v", err) } - source := string(data) - requiredSnippets := []string{ - "if err := mock.SetEnabled(true); err != nil {", - "if err := mock.SetEnabled(false); err != nil {", - "return err", - } - for _, snippet := range requiredSnippets { - if !strings.Contains(source, snippet) { - t.Fatalf("monitor.go must contain %q", snippet) - } + source := strings.Join(strings.Fields(string(data)), " ") + authorize := strings.Index(source, "if err := mock.SetEnabled(enable); err != nil { return err }") + reset := strings.Index(source, "monitor.endMockModeEpoch(enable)") + if authorize < 0 || reset < 0 || authorize > reset { + t.Fatal("SetMockMode must return the mock.SetEnabled error before it ends any monitor's epoch or resets its state") } } diff --git a/internal/monitoring/mock_mode_switch.go b/internal/monitoring/mock_mode_switch.go new file mode 100644 index 0000000000..4419f77b88 --- /dev/null +++ b/internal/monitoring/mock_mode_switch.go @@ -0,0 +1,75 @@ +package monitoring + +import ( + "sync" + "sync/atomic" + + "github.com/rcourtman/pulse-go-rewrite/internal/mock" +) + +// Mock mode is process-wide (mock.SetEnabled flips one package flag), while +// every Monitor keeps its own alert manager, fixture agents, state and +// mock-mode fence, and the server runs one Monitor per organization. A switch +// made through one monitor therefore has to reach every running monitor, or +// the others keep what they raised in the mode the process left: fixture +// alerts and fixture-agent node links in live mode, live alerts in mock mode, +// until the 24-hour stale sweep. +// +// mockModeSwitchMu serializes switches and guards the set of running monitors +// and each monitor's mockModeAligned and mockModeSwitchesSeen. A switch is +// several steps per monitor (end the epoch, clear, reset), so two interleaved +// switches could clear what the later one just admitted. Start joins the set +// under the same lock, so a monitor joins either before a switch, which then +// switches it with the others, or after it, and starts in the new mode. +// +// mockModeSwitchCount counts the flips SetMockMode made. A monitor that missed +// flips while it was not running can hold the alerts and state of a mode the +// process left even when a later flip brought the flag back to the mode it +// records (a read path evaluated fixture data before Start), so Start +// compares the count as well as the mode. +var ( + mockModeSwitchMu sync.Mutex + mockModeRunningMonitors = make(map[*Monitor]struct{}) + mockModeSwitchCount atomic.Uint64 +) + +// mockModeSwitchTargetsLocked returns the monitors a switch made through m +// reaches: m itself, which need not be running, and every running monitor. +func mockModeSwitchTargetsLocked(m *Monitor) []*Monitor { + targets := make([]*Monitor, 0, len(mockModeRunningMonitors)+1) + targets = append(targets, m) + for monitor := range mockModeRunningMonitors { + if monitor != m { + targets = append(targets, monitor) + } + } + return targets +} + +// joinMockModeSwitches adds m to the monitors every switch reaches, then runs +// start with the current mode while no switch can run, so the mode-dependent +// runtime start chooses (the mock metrics sampler or discovery) cannot be +// overtaken by a switch. It returns the function that removes m again. +// +// A switch can land between New and Start, while m is not yet running. New +// restores persisted alerts only outside mock mode, so m would then carry live +// alerts into mock mode, or fixture alerts that a read path raised before +// Start into live mode; m first leaves what it holds, as the switches it +// missed would have done. +func (m *Monitor) joinMockModeSwitches(start func(mockEnabled bool)) (leave func()) { + mockModeSwitchMu.Lock() + defer mockModeSwitchMu.Unlock() + + enabled := mock.IsMockEnabled() + if m.mockModeAligned != enabled || m.mockModeSwitchesSeen != mockModeSwitchCount.Load() { + m.endMockModeEpoch(enabled) + } + mockModeRunningMonitors[m] = struct{}{} + start(enabled) + + return func() { + mockModeSwitchMu.Lock() + defer mockModeSwitchMu.Unlock() + delete(mockModeRunningMonitors, m) + } +} diff --git a/internal/monitoring/monitor.go b/internal/monitoring/monitor.go index 69c5c33f37..80fc4aa163 100644 --- a/internal/monitoring/monitor.go +++ b/internal/monitoring/monitor.go @@ -13,6 +13,7 @@ import ( "os" "path/filepath" "reflect" + "slices" "sort" "strings" "sync" @@ -67,9 +68,9 @@ func keepRealPollingInMockMode() bool { // process that boots with mock mode already enabled never runs that path, so // alerts raised against real infrastructure are restored from disk and served // beside fixture data until something clears them. -func alertManagerRestoreOptions() []alerts.ManagerOption { +func alertManagerRestoreOptions(mockEnabled bool) []alerts.ManagerOption { options := []alerts.ManagerOption{alerts.WithDurableAlertStore()} - if mock.IsMockEnabled() { + if mockEnabled { options = append(options, alerts.WithoutPersistedAlertRestore()) } return options @@ -1283,6 +1284,8 @@ type Monitor struct { mockHostAgentsMu sync.Mutex mockHostAgents map[string]models.Host // Fixture agents evaluated by the last mock alert pass mockModeFence mockModeFence // Keeps mode-dependent alert evaluations inside the epoch they read in + mockModeAligned bool // Mock mode the alerts and state belong to; guarded by mockModeSwitchMu + mockModeSwitchesSeen uint64 // mockModeSwitchCount when they were last aligned; guarded by mockModeSwitchMu dockerChecker DockerChecker // Optional Docker checker for LXC containers dockerCheckerConfiguredAt time.Time // Last time the Docker checker was configured dockerCheckAllowedVMIDs map[int]struct{} // Optional VMID allowlist gating the LXC Docker socket probe; empty means all guests @@ -1771,6 +1774,12 @@ func New(cfg *config.Config) (*Monitor, error) { DataDir: cfg.DataPath, }) + // One read decides the alert restore and records the mode it belongs to, + // so Start can tell whether a mock-mode switch landed in between. The + // count is read first: a switch flips the flag before it counts, so one + // that overlaps these reads leaves a stale count, never a stale mode. + switchesSeen := mockModeSwitchCount.Load() + builtInMockMode := mock.IsMockEnabled() m := &Monitor{ config: cfg, state: models.NewState(), @@ -1804,7 +1813,9 @@ func New(cfg *config.Config) (*Monitor, error) { rateTracker: NewRateTracker(), metricsHistory: NewMetricsHistory(1000, 24*time.Hour), // Keep up to 1000 points (~8h @ 30s) metricsStore: metricsStore, // Persistent SQLite storage - alertManager: alerts.NewManagerWithDataDir(cfg.DataPath, alertManagerRestoreOptions()...), + alertManager: alerts.NewManagerWithDataDir(cfg.DataPath, alertManagerRestoreOptions(builtInMockMode)...), + mockModeAligned: builtInMockMode, + mockModeSwitchesSeen: switchesSeen, incidentStore: incidentStore, notificationMgr: notifications.NewNotificationManagerWithDeferredQueue(cfg.PublicURL, cfg.DataPath), deadMan: newDeadManRuntime(config.ResolveRuntimeDataDir(cfg.DataPath)), @@ -2086,33 +2097,13 @@ func (m *Monitor) Start(ctx context.Context, wsHub *websocket.Hub) { // PVE-proxied PBS backup points to prevent duplicate recovery entries. m.purgeStalePVEPBSBackupsBestEffort(ctx) - if mock.IsMockEnabled() { - m.startMockMetricsSampler(ctx) - } - - // Initialize and start discovery service if enabled - if mock.IsMockEnabled() { - log.Info().Msg("mock mode enabled - skipping discovery service") - m.discoveryService = nil - } else if m.config.DiscoveryEnabled { - discoverySubnet := m.config.DiscoverySubnet - if discoverySubnet == "" { - discoverySubnet = "auto" - } - cfgProvider := func() config.DiscoveryConfig { - return m.discoveryConfigSnapshot() - } - m.discoveryService = discovery.NewService(wsHub, 5*time.Minute, discoverySubnet, cfgProvider) - if m.discoveryService != nil { - m.discoveryService.Start(ctx) - log.Info().Msg("discovery service initialized and started") - } else { - log.Error().Msg("failed to initialize discovery service") - } - } else { - log.Info().Msg("discovery service disabled by configuration") - m.discoveryService = nil - } + // From here every mock-mode switch reaches this monitor. The sampler or + // discovery is chosen while no switch can run, so a switch cannot land + // between reading the mode and starting the runtime for it. + leaveMockModeSwitches := m.joinMockModeSwitches(func(mockEnabled bool) { + m.startModeRuntime(ctx, wsHub, mockEnabled) + }) + defer leaveMockModeSwitches() // Set up alert callbacks. Projection replay is deliberately absent here: // the canonical resource store is not attached yet, so a replay now could @@ -2251,6 +2242,38 @@ func (m *Monitor) Start(ctx context.Context, wsHub *websocket.Hub) { } } +// startModeRuntime starts the mock metrics sampler in mock mode, or the +// discovery service in live mode when it is enabled. +func (m *Monitor) startModeRuntime(ctx context.Context, wsHub *websocket.Hub, mockEnabled bool) { + if mockEnabled { + m.startMockMetricsSampler(ctx) + } + + // Initialize and start discovery service if enabled + if mockEnabled { + log.Info().Msg("mock mode enabled - skipping discovery service") + m.discoveryService = nil + } else if m.config.DiscoveryEnabled { + discoverySubnet := m.config.DiscoverySubnet + if discoverySubnet == "" { + discoverySubnet = "auto" + } + cfgProvider := func() config.DiscoveryConfig { + return m.discoveryConfigSnapshot() + } + m.discoveryService = discovery.NewService(wsHub, 5*time.Minute, discoverySubnet, cfgProvider) + if m.discoveryService != nil { + m.discoveryService.Start(ctx) + log.Info().Msg("discovery service initialized and started") + } else { + log.Error().Msg("failed to initialize discovery service") + } + } else { + log.Info().Msg("discovery service disabled by configuration") + m.discoveryService = nil + } +} + // wireExternalAlertCallbacks installs callbacks whose effects leave the alert // manager. It is safe to call again: these are single callback slots, so Start // replaces the constructor-time escalation closure with one carrying its hub. @@ -4577,65 +4600,93 @@ func (m *Monitor) broadcastEscalatedAlert(hub *websocket.Hub, alert *alerts.Aler hub.BroadcastAlertToTenant(m.GetOrgID(), alert) } -// mockModeSwitchMu serializes mock-mode switches. The mode is process-wide, -// and a switch is several steps (flip, end the epoch, clear, reset state), so -// two interleaved switches could clear what the later one just admitted. -var mockModeSwitchMu sync.Mutex - -// SetMockMode switches between mock data and real infrastructure data at runtime. -// It must not be called from inside an alert evaluation, whose completion it -// may wait for. +// SetMockMode switches between mock data and real infrastructure data at +// runtime. The mode is process-wide, so the switch reaches m and every running +// monitor (see mockModeSwitchMu), whichever tenant's monitor it was made +// through. It must not be called from inside an alert evaluation of any of +// them, whose completion it may wait for. func (m *Monitor) SetMockMode(enable bool) error { mockModeSwitchMu.Lock() defer mockModeSwitchMu.Unlock() - current := mock.IsMockEnabled() - if current == enable { - log.Info().Bool("mockMode", enable).Msg("mock mode already in desired state") - return nil + monitors := mockModeSwitchTargetsLocked(m) + flip := mock.IsMockEnabled() != enable + if !flip { + // The flag already holds the mode, but a monitor that was not running + // during an earlier switch can still hold the other mode's alerts + // and state; only those leave it. + monitors = slices.DeleteFunc(monitors, func(monitor *Monitor) bool { + return monitor.mockModeAligned == enable + }) + if len(monitors) == 0 { + log.Info().Bool("mockMode", enable).Msg("mock mode already in desired state") + return nil + } } - // Every evaluation of mode-dependent data that started before the flip - // must finish or be refused before the clear, or it reopens alerts for - // the side the monitor just left (see mockModeFence). + for _, monitor := range monitors { + monitor.stopMockMetricsSampler() + } + if flip { + if err := mock.SetEnabled(enable); err != nil { + return err + } + mockModeSwitchCount.Add(1) + } + for _, monitor := range monitors { + monitor.endMockModeEpoch(enable) + } + for _, monitor := range monitors { + _, running := mockModeRunningMonitors[monitor] + monitor.resumeAfterMockModeSwitch(enable, running) + } if enable { - m.stopMockMetricsSampler() - if err := mock.SetEnabled(true); err != nil { - return err - } - m.mockModeFence.advance() - m.alertManager.ClearActiveAlerts() - m.mu.Lock() - m.resetStateLocked() - m.metricsHistory.Reset() - m.mu.Unlock() - m.StopDiscoveryService() - m.mu.RLock() - ctx := m.runtimeCtx - m.mu.RUnlock() - if ctx != nil { - m.startMockMetricsSampler(ctx) - } - log.Info().Msg("switched monitor to mock mode") + log.Info().Int("monitors", len(monitors)).Msg("switched monitors to mock mode") } else { - m.stopMockMetricsSampler() - if err := mock.SetEnabled(false); err != nil { - return err - } - m.mockModeFence.advance() - m.alertManager.ClearActiveAlerts() - m.forgetMockHostAgents() - m.mu.Lock() - m.resetStateLocked() - m.metricsHistory.Reset() - m.mu.Unlock() - log.Info().Msg("switched monitor to real data mode") + log.Info().Int("monitors", len(monitors)).Msg("switched monitors to real data mode") } + return nil +} +// endMockModeEpoch leaves the mode the monitor's alerts and state belong to +// once the process flag already holds the new one. Every evaluation of +// mode-dependent data that started before the flip must finish or be refused +// before the clear, or it reopens alerts for the side the monitor just left +// (see mockModeFence). The caller holds mockModeSwitchMu. +func (m *Monitor) endMockModeEpoch(enable bool) { + m.mockModeFence.advance() + m.alertManager.ClearActiveAlerts() + if !enable { + m.forgetMockHostAgents() + } + m.mu.Lock() + m.resetStateLocked() + m.metricsHistory.Reset() + m.mu.Unlock() + if enable { + m.StopDiscoveryService() + } + m.mockModeAligned = enable + m.mockModeSwitchesSeen = mockModeSwitchCount.Load() +} + +// resumeAfterMockModeSwitch restarts the mode-dependent runtime of a monitor +// whose Start loop has joined the switches, once every monitor has ended its +// epoch. A monitor that has not joined yet starts that runtime itself when it +// does (Start sets its runtime context first), and one whose loop has stopped +// only broadcasts. +func (m *Monitor) resumeAfterMockModeSwitch(enable, running bool) { m.mu.RLock() ctx := m.runtimeCtx hub := m.wsHub m.mu.RUnlock() + if !running || (ctx != nil && ctx.Err() != nil) { + ctx = nil + } + + if enable && ctx != nil { + m.startMockMetricsSampler(ctx) + } if hub != nil { m.broadcastCurrentState(hub) @@ -4653,8 +4704,6 @@ func (m *Monitor) SetMockMode(enable bool) error { go m.StartDiscoveryService(ctx, hub, m.config.DiscoverySubnet) } } - - return nil } func (m *Monitor) resetStateLocked() { diff --git a/internal/monitoring/monitor_host_agent_removal_lifecycle_test.go b/internal/monitoring/monitor_host_agent_removal_lifecycle_test.go index dca2a68b1c..e9d81cf562 100644 --- a/internal/monitoring/monitor_host_agent_removal_lifecycle_test.go +++ b/internal/monitoring/monitor_host_agent_removal_lifecycle_test.go @@ -11,6 +11,7 @@ import ( "github.com/rcourtman/pulse-go-rewrite/internal/alerts" "github.com/rcourtman/pulse-go-rewrite/internal/alerts/eventlog" "github.com/rcourtman/pulse-go-rewrite/internal/config" + "github.com/rcourtman/pulse-go-rewrite/internal/mock" "github.com/rcourtman/pulse-go-rewrite/internal/models" "github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources" agentshost "github.com/rcourtman/pulse-go-rewrite/pkg/agents/host" @@ -857,3 +858,45 @@ func TestMockHostAgentLeavingFixtureUsesRemovalLifecycle(t *testing.T) { } t.Fatal("node pve9 raised no memory alert after its agent left the mock estate") } + +func TestLeavingMockModeReleasesFixtureAgentNodeLinksOnEveryRunningMonitor(t *testing.T) { + setMockSamplerTestEnv(t, time.Hour, 5*time.Minute) + pinDefaultMockEstate(t) + mustSetMockEnabled(t, true) + defaultMonitor, tenantMonitor := startTenantMonitors(t) + + // org-b's start pass registers the fixture agents in org-b's own alert + // manager, each owning the usage alerts of the fixture node it runs on. + var linked models.Node + waitForCondition(t, 30*time.Second, func() bool { + tenantMonitor.mockHostAgentsMu.Lock() + defer tenantMonitor.mockHostAgentsMu.Unlock() + for _, host := range tenantMonitor.mockHostAgents { + if host.Status != "online" || host.LinkedNodeID == "" { + continue + } + for _, node := range mock.CurrentFixtureGraph().State.Nodes { + if node.ID == host.LinkedNodeID { + linked = node + return true + } + } + } + return false + }, "org-b registered no online fixture agent linked to a fixture node") + + // The demo-fixture licence sync switches through the default monitor. + mustSetMonitorMockMode(t, defaultMonitor, false) + + // org-b's fixture agents go through the removal lifecycle too, so the + // node an agent covered owns its metric alerts again. + linked.Status = "online" + linked.Memory = models.Memory{Total: 64 << 30, Used: 60 << 30, Free: 4 << 30, Usage: 93.75} + tenantMonitor.alertManager.CheckNode(linked) + for _, alert := range tenantMonitor.alertManager.GetActiveAlerts() { + if alert.ResourceID == linked.ID && alert.Type == "memory" { + return + } + } + t.Fatalf("org-b's node %s raised no memory alert: its fixture agent's node link outlived a switch made through the default monitor", linked.ID) +} diff --git a/internal/monitoring/monitor_mock_alerts_test.go b/internal/monitoring/monitor_mock_alerts_test.go index 3c02565ece..ce0b725e54 100644 --- a/internal/monitoring/monitor_mock_alerts_test.go +++ b/internal/monitoring/monitor_mock_alerts_test.go @@ -15,6 +15,7 @@ import ( "github.com/rcourtman/pulse-go-rewrite/internal/alerts" "github.com/rcourtman/pulse-go-rewrite/internal/alerts/eventlog" + "github.com/rcourtman/pulse-go-rewrite/internal/config" "github.com/rcourtman/pulse-go-rewrite/internal/mock" "github.com/rcourtman/pulse-go-rewrite/internal/models" "github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources" @@ -809,6 +810,254 @@ func TestLeavingMockModeDiscardsAnInFlightConnectionCheck(t *testing.T) { } } +// startTenantMonitors starts the default organization's monitor and one +// other tenant's through MultiTenantMonitor, as the server does, each with its +// own alert manager, and waits until both monitoring loops run. An hour +// between ticks leaves each loop the pass it runs on start, so the test drives +// every later pass itself. +func startTenantMonitors(t *testing.T) (defaultMonitor, tenantMonitor *Monitor) { + t.Helper() + mtp, _ := newTestTenantPersistence(t) + mtm := NewMultiTenantMonitor(&config.Config{DataPath: t.TempDir(), PVEPollingInterval: time.Hour}, mtp, nil) + t.Cleanup(mtm.Stop) + mtm.SetMonitorInitializer(func(monitor *Monitor) { + cfg := monitor.alertManager.GetConfig() + cfg.Enabled = true + cfg.TimeThresholds = map[string]int{} + cfg.MetricTimeThresholds = nil + cfg.BackupDefaults.Enabled = false + cfg.AgentDefaults.Memory = &alerts.HysteresisThreshold{Trigger: 1, Clear: 0.5} + cfg.NodeDefaults.Memory = &alerts.HysteresisThreshold{Trigger: 1, Clear: 0.5} + monitor.alertManager.UpdateConfig(cfg) + }) + if err := mtp.SaveOrganization(&models.Organization{ID: "org-b", DisplayName: "Org B"}); err != nil { + t.Fatalf("SaveOrganization(org-b): %v", err) + } + monitors := make([]*Monitor, 0, 2) + for _, orgID := range []string{"default", "org-b"} { + monitor, err := mtm.GetMonitor(orgID) + if err != nil { + t.Fatalf("GetMonitor(%s): %v", orgID, err) + } + // Start sets its runtime context before it joins the monitors a + // switch reaches, so wait for the join itself. + waitForCondition(t, 30*time.Second, func() bool { + mockModeSwitchMu.Lock() + defer mockModeSwitchMu.Unlock() + _, running := mockModeRunningMonitors[monitor] + return running + }, orgID+" monitoring loop did not join mock-mode switches") + monitors = append(monitors, monitor) + } + return monitors[0], monitors[1] +} + +func mockHostAgentCount(monitor *Monitor) int { + monitor.mockHostAgentsMu.Lock() + defer monitor.mockHostAgentsMu.Unlock() + return len(monitor.mockHostAgents) +} + +func TestLeavingMockModeClearsEveryRunningTenantMonitor(t *testing.T) { + setMockSamplerTestEnv(t, time.Hour, 5*time.Minute) + pinDefaultMockEstate(t) + mustSetMockEnabled(t, true) + defaultMonitor, tenantMonitor := startTenantMonitors(t) + + // Each loop runs a mock pass on start and raises the fixture's alerts in + // its own alert manager. + monitors := map[string]*Monitor{"default": defaultMonitor, "org-b": tenantMonitor} + for name, monitor := range monitors { + waitForCondition(t, 30*time.Second, func() bool { + return len(monitor.alertManager.GetActiveAlerts()) > 0 && mockHostAgentCount(monitor) > 0 + }, name+" monitor raised no fixture alerts in mock mode") + } + + // The mock-mode API switches through the requesting tenant's monitor. + mustSetMonitorMockMode(t, tenantMonitor, false) + + for name, monitor := range monitors { + if active := monitor.alertManager.GetActiveAlerts(); len(active) != 0 { + t.Errorf("%s monitor kept %d fixture alerts after mock mode was left through org-b: %s", + name, len(active), describeAlerts(active)) + } + if count := mockHostAgentCount(monitor); count != 0 { + t.Errorf("%s monitor still tracks %d fixture agents in live mode", name, count) + } + } +} + +func TestEnteringMockModeClearsEveryRunningTenantMonitor(t *testing.T) { + setMockSamplerTestEnv(t, time.Hour, 5*time.Minute) + pinDefaultMockEstate(t) + defaultMonitor, tenantMonitor := startTenantMonitors(t) + + monitors := map[string]*Monitor{"default": defaultMonitor, "org-b": tenantMonitor} + for name, monitor := range monitors { + monitor.alertManager.CheckNode(models.Node{ + ID: name + "-live-node", + Name: "live-node", + Instance: "live", + Status: "online", + Memory: models.Memory{Total: 64 << 30, Used: 60 << 30, Free: 4 << 30, Usage: 93.75}, + }) + if len(monitor.alertManager.GetActiveAlerts()) == 0 { + t.Fatalf("live node raised no alert in %s", name) + } + } + + // The demo-fixture licence sync switches through the default monitor. + mustSetMonitorMockMode(t, defaultMonitor, true) + + for name, monitor := range monitors { + for _, alert := range monitor.alertManager.GetActiveAlerts() { + if alert.ResourceID == name+"-live-node" { + t.Errorf("%s kept its live %s alert after mock mode was entered through the default monitor", name, alert.Type) + } + } + } +} + +func TestMonitorStartedAfterAMockModeSwitchLeavesTheModeItWasBuiltIn(t *testing.T) { + setMockSamplerTestEnv(t, time.Hour, 5*time.Minute) + pinDefaultMockEstate(t) + manager := newMockHostAlertTestManager(t) + monitor := &Monitor{ + state: models.NewState(), + alertManager: manager, + metricsHistory: NewMetricsHistory(10, time.Hour), + } + // Built in live mode: New restored this live alert from disk. + manager.CheckNode(models.Node{ + ID: "live-node", + Name: "live-node", + Instance: "live", + Status: "online", + Memory: models.Memory{Total: 64 << 30, Used: 60 << 30, Free: 4 << 30, Usage: 93.75}, + }) + if len(manager.GetActiveAlerts()) == 0 { + t.Fatal("live node raised no alert") + } + + // Another monitor switched to mock mode before this one's Start joined. + mustSetMockEnabled(t, true) + started := false + leave := monitor.joinMockModeSwitches(func(mockEnabled bool) { started = mockEnabled }) + defer leave() + + if !started { + t.Fatal("Start was not handed mock mode") + } + if active := manager.GetActiveAlerts(); len(active) != 0 { + t.Fatalf("a monitor built in live mode started in mock mode with its live alerts: %s", describeAlerts(active)) + } + mockModeSwitchMu.Lock() + _, running := mockModeRunningMonitors[monitor] + mockModeSwitchMu.Unlock() + if !running { + t.Fatal("joined monitor is not reached by later switches") + } +} + +func TestMonitorStartedAfterAMockModeRoundTripDropsWhatItRaisedMeanwhile(t *testing.T) { + setMockSamplerTestEnv(t, time.Hour, 5*time.Minute) + pinDefaultMockEstate(t) + newMonitor := func() *Monitor { + return &Monitor{ + state: models.NewState(), + alertManager: newMockHostAlertTestManager(t), + metricsHistory: NewMetricsHistory(10, time.Hour), + } + } + switcher, starting := newMonitor(), newMonitor() + // Built in live mode, as New records it, and not running yet. + mockModeSwitchMu.Lock() + starting.mockModeSwitchesSeen = mockModeSwitchCount.Load() + mockModeSwitchMu.Unlock() + + mustSetMonitorMockMode(t, switcher, true) + // A read path evaluates fixture data on the monitor before its Start runs. + fixture := mock.CurrentFixtureGraph().State.Nodes[0] + fixture.Status = "online" + fixture.Memory = models.Memory{Total: 64 << 30, Used: 60 << 30, Free: 4 << 30, Usage: 93.75} + starting.alertManager.CheckNode(fixture) + if len(starting.alertManager.GetActiveAlerts()) == 0 { + t.Fatal("fixture node raised no alert") + } + mustSetMonitorMockMode(t, switcher, false) + + leave := starting.joinMockModeSwitches(func(bool) {}) + defer leave() + if active := starting.alertManager.GetActiveAlerts(); len(active) != 0 { + t.Fatalf("a monitor that missed a mock-mode round trip started in live mode with fixture alerts: %s", describeAlerts(active)) + } +} + +func TestSetMockModeAlignsAMonitorWhoseModeTheFlagAlreadyHolds(t *testing.T) { + setMockSamplerTestEnv(t, time.Hour, 5*time.Minute) + pinDefaultMockEstate(t) + manager := newMockHostAlertTestManager(t) + monitor := &Monitor{ + state: models.NewState(), + alertManager: manager, + metricsHistory: NewMetricsHistory(10, time.Hour), + } + // Built in live mode, with a live alert, and not running yet. + manager.CheckNode(models.Node{ + ID: "live-node", + Name: "live-node", + Instance: "live", + Status: "online", + Memory: models.Memory{Total: 64 << 30, Used: 60 << 30, Free: 4 << 30, Usage: 93.75}, + }) + if len(manager.GetActiveAlerts()) == 0 { + t.Fatal("live node raised no alert") + } + + // Another monitor's switch already set the flag. + mustSetMockEnabled(t, true) + mustSetMonitorMockMode(t, monitor, true) + + if active := manager.GetActiveAlerts(); len(active) != 0 { + t.Fatalf("SetMockMode(true) left a monitor built in live mode with its live alerts: %s", describeAlerts(active)) + } +} + +func TestMockModeSwitchLeavesAnUnjoinedMonitorsRuntimeToItsStart(t *testing.T) { + setMockSamplerTestEnv(t, time.Hour, 5*time.Minute) + pinDefaultMockEstate(t) + ctx, cancel := context.WithCancel(context.Background()) + t.Cleanup(cancel) + // Start has set its runtime context but not joined the switches yet. + monitor := &Monitor{ + config: &config.Config{}, + state: models.NewState(), + alertManager: newMockHostAlertTestManager(t), + metricsHistory: NewMetricsHistory(10, time.Hour), + runtimeCtx: ctx, + } + t.Cleanup(monitor.stopMockMetricsSampler) + + mustSetMonitorMockMode(t, monitor, true) + monitor.mu.RLock() + started := monitor.mockMetricsCancel != nil + monitor.mu.RUnlock() + if started { + t.Fatal("a switch started the runtime of a monitor whose Start has not joined; that runtime belongs to its Start") + } + + leave := monitor.joinMockModeSwitches(func(mockEnabled bool) { + monitor.startModeRuntime(ctx, nil, mockEnabled) + }) + defer leave() + monitor.mu.RLock() + started = monitor.mockMetricsCancel != nil + monitor.mu.RUnlock() + if !started { + t.Fatal("joining in mock mode did not start the mock sampler") + } +} + // diskAlertKeys lists the active disk-health and disk-wearout alerts as // "type instance/node device" so assertions name the disk, not the alert ID. func diskAlertKeys(manager *alerts.Manager) map[string]bool {