mirror of
https://github.com/rcourtman/Pulse.git
synced 2026-10-24 04:33:17 +00:00
rcourtman
cfa026249f
Follow-up to #2656 (the per-monitor mock-mode epoch fence), which documented this gap as not covered. ## Problem Mock mode is one process-wide flag (`mock.SetEnabled`), but `Monitor.SetMockMode` ended the fence epoch, cleared the alerts, forgot the fixture agents and reset the state of only the monitor it was called on. The server runs one `Monitor` per organization (`MultiTenantMonitor.GetMonitor` → `New` + `Start`), and each runs its own mock alert pass into its own alert manager. A switch made through one monitor (the mock-mode API uses the requesting tenant's monitor; the demo-fixture licence sync uses the default one) left every other running monitor holding what it raised in the mode the process had left: fixture alerts and fixture-agent node links in live mode, live alerts in mock mode, until the 24-hour stale sweep. ## Fix - `internal/monitoring/mock_mode_switch.go`: a process-wide set of running monitors, guarded by the existing `mockModeSwitchMu`. `Start` joins it and leaves when its loop returns. While joining it picks the mock metrics sampler or discovery under the lock (now `startModeRuntime`), so no switch lands between reading the mode and starting the runtime for it. - `SetMockMode` flips the flag once, then for the receiver and every running monitor: ends the epoch, waits for admitted calls, clears alerts, forgets fixture agents (when leaving mock mode) and resets state (`endMockModeEpoch`). Only after that does it restart the sampler, polls and discovery of each monitor in the set (`resumeAfterMockModeSwitch`). A monitor that has not joined yet gets its runtime from its own `Start`. - Each monitor records the mode its alerts and state belong to (`mockModeAligned`, set in `New` from the same read that decides whether persisted alerts are restored) and the number of switches made so far (`mockModeSwitchesSeen`). If it missed switches between `New` and `Start`, it leaves what it holds before joining, even after a round trip that brought the flag back. A call for the mode the flag already holds still moves any reached monitor that belongs to the other mode, and leaves the rest alone. Callers are unchanged and no API surface moves. ## Is this reachable in release builds? Barely. In a `release` build `mockruntime.ValidateEnablement` refuses mock fixtures unless `SetReleaseFixturesAuthorized(true)` was called. Its only caller is `syncReleaseDemoFixtureRuntime`, which requires `DemoMode` plus the `demo_fixtures` entitlement and runs only for the default organization. Several tenant monitors in mock mode would also need more than one organization's monitor running on that demo instance. Demo mode's read-only middleware normally blocks creating organizations and the mock-mode API's POST. It does forward any request carrying `Upgrade: websocket` whatever its method; that bypass predates this change and is being fixed separately. So in practice this is a dev/mock-stack problem, and the fix stays inside the monitoring package. ## Tests New tests in `internal/monitoring/monitor_mock_alerts_test.go`: - `TestLeavingMockModeClearsEveryRunningTenantMonitor` and `TestEnteringMockModeClearsEveryRunningTenantMonitor` start the default and one other tenant monitor through `MultiTenantMonitor`, as the server does, and wait until both have joined. They raise alerts in both, switch through one, and check both. - `TestMonitorStartedAfterAMockModeSwitchLeavesTheModeItWasBuiltIn` covers a switch that lands between `New` and `Start`. - `TestMonitorStartedAfterAMockModeRoundTripDropsWhatItRaisedMeanwhile` covers two switches in that window, with fixture alerts raised in between. - `TestSetMockModeAlignsAMonitorWhoseModeTheFlagAlreadyHolds` covers a call for the mode the flag already holds. - `TestMockModeSwitchLeavesAnUnjoinedMonitorsRuntimeToItsStart` covers a switch reaching a monitor that has not joined. One more is in `internal/monitoring/monitor_host_agent_removal_lifecycle_test.go`: `TestLeavingMockModeReleasesFixtureAgentNodeLinksOnEveryRunningMonitor`. It switches through the default monitor and checks that the other tenant's fixture node is no longer suppressed by a fixture agent's leftover node link. Negative controls (`go test -overlay`): | Control | Failing tests | What happened | |---|---|---| | Fence-PR runtime | Leaving, entering, node-link | Default monitor kept 69–104 fixture alerts (varies by run) and 69 fixture agents; org-b kept its live alert; org-b's `mock-cluster-1-pve1` stayed suppressed | | Set kept, switch reaches only the receiver | Same three | Same failures | | No realign at join | `TestMonitorStartedAfterAMockModeSwitchLeavesTheModeItWasBuiltIn` | | | Join compares only the mode, not the switch count | `TestMonitorStartedAfterAMockModeRoundTripDropsWhatItRaisedMeanwhile` | Started in live mode with the fixture node's memory alert | | Plain early return when the flag already matches | `TestSetMockModeAlignsAMonitorWhoseModeTheFlagAlreadyHolds` | | | Resume ignores set membership | `TestMockModeSwitchLeavesAnUnjoinedMonitorsRuntimeToItsStart` | | Every control failed only the tests it targets. Also run: the full `internal/monitoring` package; `-race -count=3` on the new tests plus the existing `TestLeavingMockMode*` and `TestMockModeFence*`; and the mock-mode tests in `internal/api` and `internal/api/configapi`.
Pulse documentation
Start here for installation, platform setup, security, operations, and Pulse Intelligence. Commands, configuration keys, image names, API fields, and product identifiers remain untranslated in localized guides.
Start here
- Install Pulse — signed Proxmox/Linux installation, Docker, Docker Compose, Kubernetes, and first-run setup.
- Production deployment and security — least-privilege Proxmox setup, root-agent boundaries, discovery, supply-chain verification, plan limits, scale evidence, and a rollout checklist.
- Upgrade from Pulse v5 — migration prerequisites, rollback, agent continuity, and post-upgrade checks.
- Releases and update channels: choosing Stable or Preview, beta and RC expectations, and how releases are tested and promoted.
- Configure Pulse — authentication, notifications, discovery, retention, and system settings.
- Deployment models — data locations, lifecycle, and differences between supported deployment paths.
- Troubleshooting and FAQ — common failures, diagnostics, and operator questions.
Localized getting started guides: Deutsch · Español
Platforms and agents
- Proxmox Backup Server
- Proxmox Mail Gateway
- Docker and Podman
- Kubernetes and Helm
- TrueNAS SCALE and CORE
- Unified Agent
- Agent security
- VM disk monitoring
- ZFS monitoring
- Temperature monitoring
VMware vSphere support is early access. Current builds expose dedicated vSphere inventory and recovery context, but operators should validate the integration against their own vCenter before production use.
Monitoring and operations
- Metrics history
- Recovery data
- Webhooks
- Automatic updates
- Centralized agent management (Pro)
- Operational trust model
- Current product screenshots
Pulse Intelligence
- Assistant, Patrol, and external-agent overview
- Patrol modes and safety
- Assistant safety model
- External agent HTTP and MCP substrate
Patrol watch-only analysis is available on Community with a local model or the operator's own provider. Investigation and governed fixes require the relevant Pulse Pro capabilities.
Security, privacy, and access
- Production deployment and security
- Security review scope
- Authentication and credential storage review packet
- Security guide
- Privacy and telemetry disclosure
- OIDC and SSO
- Proxy authentication
- Role-based access control (Pro)
- Audit logging (Pro)
- Reverse proxy configuration
- Code-signing policy
Plans and managed access
- Community, Relay, and Pro capabilities
- Relay and Pulse Mobile handoff
- Multi-tenant organizations (Enterprise/custom)
- Provider-hosted MSP operations (request-assisted)
Pulse Cloud is not generally available. Ordinary self-hosted Pulse remains the primary installation path; MSP and Enterprise access are explicit commercial paths rather than defaults in self-hosted setup.
Development and reference
Detailed design notes and dated migration specifications may remain in this directory for maintainers, but they are not operator setup guides unless they are linked from the sections above.
Previous versions and migrations
- Upgrade from v4 to v5
- Retired unified-navigation migration — historical context only; current Pulse uses platform-shaped navigation.
- Move a Pulse installation
Found a bug? Use the issue forms. For setup questions, use GitHub Discussions.