mirror of
https://github.com/rcourtman/Pulse.git
synced 2026-10-27 14:07:31 +00:00
rcourtman
27e440339b
Mock mode is one process-wide flag, but SetMockMode ended the epoch, cleared the alerts, forgot the fixture agents and reset the state of only the monitor it was called on. The server runs one monitor per organization, each with its own alert manager and its own mock alert pass. A switch made through one of them (the mock-mode API uses the requesting tenant's monitor, the demo-fixture licence sync the default one) left every other running monitor holding what it raised in the mode the process had left: fixture alerts and fixture-agent node links in live mode, live alerts in mock mode, until the 24-hour stale sweep. Monitors now join a process-wide set of running monitors when Start runs and leave it when Start returns, under the same mutex that already serialized switches. SetMockMode flips the flag once, then ends the epoch of the monitor it was called on and of every running monitor: it waits for their admitted calls, clears their alerts, forgets their fixture agents and resets their state. Only then does it restart each running monitor's sampler, polls and discovery. Start picks the mock metrics sampler or discovery while it holds that mutex, so no switch can land between reading the mode and starting the runtime for it, and a switch leaves the runtime of a monitor that has not joined yet to its own Start. A monitor records the mode and the switch count it was built under. If it missed switches between New and Start, it leaves what it holds before it joins, dropping live alerts New restored or fixture alerts a read path raised meanwhile, even when a round trip brought the flag back. A call for the mode the flag already holds still moves a reached monitor whose alerts belong to the other mode, and leaves the rest alone. Release builds reach this only on an instance that runs DemoMode with the demo_fixtures entitlement, since release ValidateEnablement refuses mock fixtures otherwise, and that has more than one organization's monitor running. The licence sync only switches the default organization. So the fix stays inside the monitoring package. The callers are unchanged and no API surface moves. New tests start the default and one other tenant monitor through MultiTenantMonitor, as the server does, and switch through one of them. Against the per-monitor switch, the default monitor kept its fixture alerts (69 to 104 across runs) and all 69 fixture agents after mock mode was left through the other tenant. That tenant kept its live alert after mock mode was entered through the default monitor, and its fixture node stayed suppressed by a fixture agent's leftover node link. Four more tests fail if Start joins without leaving the mode its monitor was built in, if Start compares only the mode after a round trip, if a call for the flag's current mode returns without moving such a monitor, or if a switch starts the runtime of a monitor that has not joined.
76 lines
3.0 KiB
Go
76 lines
3.0 KiB
Go
package monitoring
|
|
|
|
import (
|
|
"sync"
|
|
"sync/atomic"
|
|
|
|
"github.com/rcourtman/pulse-go-rewrite/internal/mock"
|
|
)
|
|
|
|
// Mock mode is process-wide (mock.SetEnabled flips one package flag), while
|
|
// every Monitor keeps its own alert manager, fixture agents, state and
|
|
// mock-mode fence, and the server runs one Monitor per organization. A switch
|
|
// made through one monitor therefore has to reach every running monitor, or
|
|
// the others keep what they raised in the mode the process left: fixture
|
|
// alerts and fixture-agent node links in live mode, live alerts in mock mode,
|
|
// until the 24-hour stale sweep.
|
|
//
|
|
// mockModeSwitchMu serializes switches and guards the set of running monitors
|
|
// and each monitor's mockModeAligned and mockModeSwitchesSeen. A switch is
|
|
// several steps per monitor (end the epoch, clear, reset), so two interleaved
|
|
// switches could clear what the later one just admitted. Start joins the set
|
|
// under the same lock, so a monitor joins either before a switch, which then
|
|
// switches it with the others, or after it, and starts in the new mode.
|
|
//
|
|
// mockModeSwitchCount counts the flips SetMockMode made. A monitor that missed
|
|
// flips while it was not running can hold the alerts and state of a mode the
|
|
// process left even when a later flip brought the flag back to the mode it
|
|
// records (a read path evaluated fixture data before Start), so Start
|
|
// compares the count as well as the mode.
|
|
var (
|
|
mockModeSwitchMu sync.Mutex
|
|
mockModeRunningMonitors = make(map[*Monitor]struct{})
|
|
mockModeSwitchCount atomic.Uint64
|
|
)
|
|
|
|
// mockModeSwitchTargetsLocked returns the monitors a switch made through m
|
|
// reaches: m itself, which need not be running, and every running monitor.
|
|
func mockModeSwitchTargetsLocked(m *Monitor) []*Monitor {
|
|
targets := make([]*Monitor, 0, len(mockModeRunningMonitors)+1)
|
|
targets = append(targets, m)
|
|
for monitor := range mockModeRunningMonitors {
|
|
if monitor != m {
|
|
targets = append(targets, monitor)
|
|
}
|
|
}
|
|
return targets
|
|
}
|
|
|
|
// joinMockModeSwitches adds m to the monitors every switch reaches, then runs
|
|
// start with the current mode while no switch can run, so the mode-dependent
|
|
// runtime start chooses (the mock metrics sampler or discovery) cannot be
|
|
// overtaken by a switch. It returns the function that removes m again.
|
|
//
|
|
// A switch can land between New and Start, while m is not yet running. New
|
|
// restores persisted alerts only outside mock mode, so m would then carry live
|
|
// alerts into mock mode, or fixture alerts that a read path raised before
|
|
// Start into live mode; m first leaves what it holds, as the switches it
|
|
// missed would have done.
|
|
func (m *Monitor) joinMockModeSwitches(start func(mockEnabled bool)) (leave func()) {
|
|
mockModeSwitchMu.Lock()
|
|
defer mockModeSwitchMu.Unlock()
|
|
|
|
enabled := mock.IsMockEnabled()
|
|
if m.mockModeAligned != enabled || m.mockModeSwitchesSeen != mockModeSwitchCount.Load() {
|
|
m.endMockModeEpoch(enabled)
|
|
}
|
|
mockModeRunningMonitors[m] = struct{}{}
|
|
start(enabled)
|
|
|
|
return func() {
|
|
mockModeSwitchMu.Lock()
|
|
defer mockModeSwitchMu.Unlock()
|
|
delete(mockModeRunningMonitors, m)
|
|
}
|
|
}
|