Follows #2643, which made the stale pass read each source's own verdict
but left availability checks counting by delivery.
## Problem
Once every source with a verdict had gone quiet, any current
availability sighting kept its target online. A node that
`preserveOrExpireNodes` expired stayed online while its HTTP check
failed, and a quiet failing check lifted that node's offline verdict to
warning. One sighting stands for every check on a target and carries
whichever check was projected last, so a failing check that had just run
also vouched for another check's old pass.
## Change
- **Registry verdict** (`aggregateStatus`,
`availabilityChecksProveOnline`): each check on a monitored resource is
judged by its own evidence window (`Evidence.ValidUntil`). An enabled,
passing check with current evidence proves the resource answers and
reads online once one of the resource's own sources has gone quiet and
none that is current has a verdict. Every other check abstains, current
or quiet. A failing probe never makes a guest count as stopped
(workloads summary, Proxmox page counts). A check's own row keeps the
ordinary rules.
- **Ownership**: only a target's own quiet sources hand the stale pass
its status. Projecting, re-running or retargeting a check re-applies the
stale pass's rule to targets it owns, because the resources API replays
checks after its stale pass. A manual link judges the merged resource by
the checks it carries.
- **Poller evidence** (`internal/monitoring`):
- A local location beside other locations ages out like a silent remote
path ("no recent local check").
- An available check's evidence lasts only as long as its newest
reachable path.
- After an entitlement lapse, an adopted remote result keeps the earlier
of its agent check time and the server receipt time.
- A multi-location aggregate is never adopted as a local observation;
the target reads as never checked until the local poller runs.
Contracts: `unified-resources.md` ("Status follows each source's own
verdict") replaces the residual paragraph, and `monitoring.md` covers
the multi-location rules.
## Verification
- New tests in `registry_test.go`, `registry_merge_policy_test.go` and
`availability_probe_agent_test.go`. Each fails against the base runtime
via `go test -overlay`, including both projection orders.
- Full `internal/unifiedresources` and `internal/monitoring` suites
pass, plus the availability, connection, workload and resource-state
tests in `internal/api`, `resourceapi` and `chartapi`.
Once every source with a verdict had gone quiet, the stale pass counted
any current availability sighting as online. A node that
preserveOrExpireNodes expired stayed online while its HTTP check failed,
and a quiet failing check lifted that node's offline verdict to warning.
One sighting stands for every check on a target and carries whichever
check was projected last, so a failing check that had just run also
vouched for another check's old pass.
Status aggregation now judges each check on a monitored resource by its
own evidence window. A passing check whose evidence is still valid
proves the resource answers, and reads online when no current source has
a verdict. Every other check abstains, current or quiet: a failing check
proves only that one port or service does not answer, so it never makes
a guest count as stopped in the workloads summary or the Proxmox page.
A check's own row keeps the ordinary rules, and a manual link judges the
merged resource by the checks it carries.
Projecting or retargeting a check now re-applies the stale pass's rule
to targets whose status that pass owns, since the resources API replays
checks after its stale pass and never runs another. Only a target's own
quiet sources give the pass that ownership, never its checks' sighting.
The poller now keeps each check's evidence tied to the observation that
passed. It ages out a local location that stopped running beside other
locations, as it already did for silent remote paths: the check took its
freshness from the newest location, so a failing remote report kept an
hour-old local pass counted as reachable with current evidence. An
available check's evidence lasts only as long as its newest reachable
path, and a remote result adopted as local after an entitlement lapse
keeps the server's receipt time rather than the agent's clock.
SetMockMode(false) switched the mock package off and then cleared active
alerts, but every path that evaluates mode-dependent data checked the
mode once, up front. A mock alert pass (started from the ticker with go,
so passes overlap), a frontend or ingest refresh evaluating a GetState
snapshot, a backup-age check over fixture recovery rollups, or a
connection check over the mock ledger could already hold fixture data
when the switch happened. Its evaluations then landed after the clear,
in live mode, where nothing evaluates or resolves fixture resources, so
the alerts stayed until the 24-hour stale sweep.
A pass-level lock around checkMockAlerts was tried and reverted: it left
the other paths open and made the mock-mode API wait behind a whole pass
of synchronous event-log writes.
The monitor now owns a mock-mode epoch fence. The mock pass, backup-age
and snapshot checks, connection checks, Docker pruning and the
resource-store refresh take a scope before reading anything
mode-dependent and send each alert-manager call through it. SetMockMode,
now serialized, flips the mode, ends the epoch, waits only for calls
already admitted, and then clears. Unified metrics are evaluated one
resource per call, so the wait covers single calls, not passes.
Registry rebuilds are tracked rather than waited on. A refresh evaluates
the registry it read back only if no rebuild from an ended epoch
overlapped it, and the readers that open or remove alerts from the
inventory use current state until a clean rebuild in the new epoch
replaces the previous mode's registry. The mock supplemental fixture
sources now serve nothing outside mock mode: the router swaps them for
the real pollers only after the switch, refreshing between sources, so
a single switch to live mode used to raise fixture vSphere incidents.
The new tests park a real evaluation at a fixed point (a held lock or a
blocking supplier), switch modes and release it. Against the unfenced
code the mock pass left about 40 fixture alerts, the frontend refresh 5
vSphere incidents and the backup check 137 backup-age alerts in live
mode, and a mock-mode connection observation counted toward a live
connection's confirmation. The drain, republish and pruning tests fail
against a fence without its wait, overlap check or registry tracking.
Switching mock mode with several tenant monitors running remains
uncovered and is noted in the monitoring contract.
The unified agent gains an availability module: probe assignments
arrive through the signed remote-config channel (missing key clears
the schedule), each enabled target runs on its own clamped interval
through the shared probe core, and results queue in a bounded
drop-oldest buffer. A result is offered to the primary server until
one delivery succeeds and never again after - buffered offline
reports are stripped of availability results so the disk buffer
cannot replay observations the queue still holds. ApplyHostReport
feeds accepted reports into the probe ingestion path, where the
ownership check and failure accounting live, and the probe agent id
is projected onto unified availability resources for source
attribution in the UI.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Availability targets gain an optional probe agent assignment. Setting
it requires the external_probe entitlement, enforced only at the
moment of assignment - local targets never consult the license path.
Assigned targets are delivered to their agent through the signed
agent-config channel, skipped by the local poller, and resume local
execution automatically if the entitlement lapses. Probe-reported
results are accepted only from the currently assigned agent, share
the local failure-threshold accounting, carry source attribution,
and derive to indeterminate at read time when reports go stale.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
rcourtman