SetMockMode(false) switched the mock package off and then cleared active
alerts, but every path that evaluates mode-dependent data checked the
mode once, up front. A mock alert pass (started from the ticker with go,
so passes overlap), a frontend or ingest refresh evaluating a GetState
snapshot, a backup-age check over fixture recovery rollups, or a
connection check over the mock ledger could already hold fixture data
when the switch happened. Its evaluations then landed after the clear,
in live mode, where nothing evaluates or resolves fixture resources, so
the alerts stayed until the 24-hour stale sweep.
A pass-level lock around checkMockAlerts was tried and reverted: it left
the other paths open and made the mock-mode API wait behind a whole pass
of synchronous event-log writes.
The monitor now owns a mock-mode epoch fence. The mock pass, backup-age
and snapshot checks, connection checks, Docker pruning and the
resource-store refresh take a scope before reading anything
mode-dependent and send each alert-manager call through it. SetMockMode,
now serialized, flips the mode, ends the epoch, waits only for calls
already admitted, and then clears. Unified metrics are evaluated one
resource per call, so the wait covers single calls, not passes.
Registry rebuilds are tracked rather than waited on. A refresh evaluates
the registry it read back only if no rebuild from an ended epoch
overlapped it, and the readers that open or remove alerts from the
inventory use current state until a clean rebuild in the new epoch
replaces the previous mode's registry. The mock supplemental fixture
sources now serve nothing outside mock mode: the router swaps them for
the real pollers only after the switch, refreshing between sources, so
a single switch to live mode used to raise fixture vSphere incidents.
The new tests park a real evaluation at a fixed point (a held lock or a
blocking supplier), switch modes and release it. Against the unfenced
code the mock pass left about 40 fixture alerts, the frontend refresh 5
vSphere incidents and the backup check 137 backup-age alerts in live
mode, and a mock-mode connection observation counted toward a live
connection's confirmation. The drain, republish and pruning tests fail
against a fence without its wait, overlap check or registry tracking.
Switching mock mode with several tenant monitors running remains
uncovered and is noted in the monitoring contract.
The metrics store retains second-resolution samples. Verify a changed observed memory value after recovery, rather than requiring a second row when polls share a bucket.
Change-source: pulse-maintainer
Persist measured PBS CPU and memory on the service metrics target when node status succeeds. Keep host-agent history separate and avoid false samples on unavailable status. Cover service reads, denial and recovery.
Change-source: pulse-maintainer
Keep topology capacity bands as resource risk rather than duplicate parent and datastore alerts. The existing live PBS capacity evaluator owns thresholds, aliases and hysteresis; retain datastore state failures and reconcile old duplicates normally.
Reproduce the 97.9% report with real registry projection and exercise threshold changes through synthetic PBS HTTP polling. Release adaptations require the live-poll evaluator, not this filter alone.
Change-source: pulse-maintainer
Cover disabled alerting, storage suppression and canonical datastore overrides before the polled capacity lifecycle. Removing the conversion alias makes the disabled-override regression case fail.
Change-source: pulse-maintainer
A reachable PBS can deny or fail its node-status endpoint while datastore collection succeeds. The resulting zero values previously resolved active CPU and memory alerts without a healthy measurement. Carry internal availability evidence from polling and skip metric evaluation for those samples, preserving independent connectivity and policy suppression. Add failing-before unit coverage and an HTTP polling lifecycle regression for dispatch and recent recovery history.
Change-source: pulse-maintainer
Connected systems showed a PBS machine twice (API row plus host-agent row)
whenever the PBS connection was configured with an address the agent never
reports, because PBS identity was limited to the configured name and host.
The PBS poll now captures the hostname the node reports about itself
(GET /nodes, the previously unused client GetNodeName) on
models.PBSInstance.NodeName, and /api/connections includes it in the PBS
connection's host aliases, so directPlatformHostAttachment merges the agent
running on that machine into the PBS row the same way PVE composes from
API-reported node identity. Node-name fetch failure stays partial data and
never fails the poll. Reported by Johannes Strasser (rc.5).
The PBS poller's version-failure exit at monitor_pbs_pmg.go did not
set pollErr before returning, so even after fixing the defer-arg
capture in bf6261adc the deferred recordTaskResult still saw nil and
recorded the poll as a success. PMG's analogous path already sets
pollErr correctly. Mirror that here so the per-instance pollStatusMap,
the connections aggregator, and the circuit breaker all see PBS auth
and version failures as failures.
Add assertions to the existing PBS and PMG auth-failure tests that
the per-instance pollStatusMap entry has a zero LastSuccess and a
non-zero ConsecutiveFailures. The original tests covered downstream
state but not the recorder, which is why two distinct cases of this
class of bug went unnoticed.
Introduces GET /api/connections and POST /api/connections/probe as the
backend half of the one-ledger / one-editor connection redesign.
- GET /api/connections aggregates PVE/PBS/PMG/VMware/TrueNAS/agent rows
into a unified Connection shape with derived state (active, paused,
unauthorized, unreachable, stale, pending) computed from in-memory
scheduler health plus agent Host.LastSeen. No new persisted state.
- POST /api/connections/probe fingerprints a host across the five
supported products in parallel (2s dial + 1s read, 3s total, max 5
concurrent). Admin-gated (RequireAdmin + ScopeSettingsWrite) to block
unauthenticated SSRF against internal hosts.
- Disabled bool on PVEInstance/PBSInstance/PMGInstance (zero-value =
enabled, preserves existing nodes.json); pollers skip disabled
instances at client init, reconnect, and per-node iteration.
- NodeConfigRequest/Response gain Enabled; write path translates
*bool -> Disabled so omitted field leaves state untouched.
- ConnectionsAPI frontend client (list/probe) typed off the Go shape.
Contracts updated: api-contracts, monitoring, agent-lifecycle,
performance-and-scalability, storage-recovery. Proofs added:
contract_test.go JSON snapshot for Connection and ProbeResponse,
monitoring guardrails for the Disabled-skip behavior, and a vitest
mock-client test for ConnectionsAPI.
Frontend editor / drawer / table rewrite lands in a separate block.
- Fix TestMonitor_PollGuestSnapshots_Coverage by correctly initializing State ID fields
- Improve PBS client to handle alternative datastore metric fields (total-space, etc.)
- Add comprehensive test coverage for PBS polling, auth failures, and datastore metrics
- Add various coverage tests for monitoring, alerts, and metadata handling
- Refactor Monitor to support better testing of client creation and auth handling
rcourtman