Use server time for probe freshness

This commit is contained in:
courtmanr@gmail.com
2026-07-29 21:27:18 +01:00
parent d4e5326059
commit 0ff0a37f8d
13 changed files with 254 additions and 61 deletions
+7 -5
View File
@@ -509,11 +509,13 @@ its signed agent configuration, runs it on the configured interval, and
reports results back with its reports; results are only accepted from the
currently assigned agent. An assigned check is not also run locally. If no
report arrives for several intervals the check shows as indeterminate
("no recent report from probe agent"). After at least five minutes without a
report, Pulse raises one warning per disconnected probe through the normal
notification routes, even when that probe owns several checks. The incident is
keyed to the probe agent rather than an arbitrary check, so changing assignments
does not reopen it. If the agent heartbeat is also offline, the existing
("no recent report from probe agent"). Freshness uses the Pulse server's receipt
time, so clock drift on the probe cannot create or hide a disconnect. After at
least five minutes without a report, Pulse raises one warning per disconnected
probe through the normal notification routes, even when that probe owns several
checks. The incident is keyed to the probe agent rather than an arbitrary check,
so changing assignments does not reopen it. If the agent heartbeat is also
offline, the existing
host-offline alert owns the incident instead of producing a duplicate probe
warning. A paired Pulse Mobile client receives a privacy-safe
`external_probe_offline` Relay push linked to that canonical alert. If the whole
+8 -6
View File
@@ -106,12 +106,14 @@ in-memory queue; if the agent cannot deliver for several check intervals the
check shows as indeterminate in Pulse until reports resume. After the
five-minute minimum grace window, Pulse raises one
`availability_probe_unavailable` warning per disconnected probe, regardless of
how many checks it owns. That warning uses the normal email, webhook, Apprise,
and recovery-notification pipeline. When Pulse Mobile is paired through Relay,
Pulse also sends a privacy-safe `external_probe_offline` push linked to the
canonical mobile attention item without exposing target names or addresses.
The alert identity belongs to the probe agent, so adding or removing an
assigned check does not resolve and reopen it.
how many checks it owns. Pulse measures that reporting window from server receipt
time rather than the agent's clock, so clock skew cannot create or conceal the
disconnect. That warning uses the normal email, webhook, Apprise, and
recovery-notification pipeline. When Pulse Mobile is paired through Relay, Pulse
also sends a privacy-safe `external_probe_offline` push linked to the canonical
mobile attention item without exposing target names or addresses. The alert
identity belongs to the probe agent, so adding or removing an assigned check
does not resolve and reopen it.
When the host heartbeat itself is offline, Pulse keeps the existing
host-offline alert as the single canonical incident and suppresses the
@@ -60,6 +60,12 @@ and reassignment start a fresh reporting grace window; a result from the
previous agent must not satisfy the new lease. Once the host heartbeat itself
is unhealthy, the canonical host-offline lifecycle owns the outage so the
availability-probe lifecycle cannot create a duplicate whole-agent incident.
The adjacent `/api/connections` availability row uses monitoring's
server-receipt freshness reference for that lease while preserving the
agent-authored check time as observation metadata. That receipt timestamp is
probe-result liveness only: lifecycle consumers must not reinterpret it as a
host heartbeat, enrollment renewal, command-channel readiness, or fleet-update
observation.
## Canonical Files
@@ -1212,6 +1212,11 @@ payload shape change when the portal presents compact client rows.
`availability` must route management handoffs, pause, remove, and test
actions to those availability-target endpoints and must not reuse node,
SSH, platform API, or Pulse Agent setup payloads.
For probe-assigned rows, the Connections ledger consumes monitoring's
canonical server-receipt freshness reference and its already-derived stale
verdict. It must not independently age the agent-authored `LastSuccess`
timestamp, because agent clock skew could make Connections disagree with
the `network-endpoint` resource and canonical probe alert lifecycle.
Mock mode must expose authored availability targets through those same
list, saved-test, and connections-ledger payloads so demo endpoints exercise
the canonical API contract rather than a frontend-only fixture.
@@ -7846,6 +7851,11 @@ as a `network-endpoint`. Browser callers may test unsaved or saved targets, but
the persisted target list remains owned by `/api/availability-targets` and
must be managed from `/settings/monitoring/availability`, not reconstructed
from resource snapshots or monitored-system counts.
Remote probe rows preserve the agent-authored check time as observation
metadata, but API freshness and `lastSeenAt` projections use the server-authored
receipt time supplied by monitoring. API consumers must not substitute the
agent clock for disconnect detection or apply a second, shorter generic
Connections staleness window.
Mock availability fixtures must still behave like saved targets: `/api/connections`
reports them as availability rows, `/api/availability-targets` lists them with
probe status, and saved-test calls return the synthetic probe result instead of
@@ -324,13 +324,16 @@ yields no entry and must say so in the log rather than passing silently, and it
must never prevent the remaining configured hosts from being suppressed.
External availability-probe freshness is evaluated against the effective
target cadence with a five-minute minimum grace. Missing, stale, or
wrong-agent results are indeterminate monitoring evidence: their
`network-endpoint` resources degrade to warning without manufacturing a target
reachability incident. Monitoring aggregates stale targets by current agent,
updates the single canonical probe alert lifecycle, and treats a fresh result
from that same assignment as recovery. Assignment trackers are removed with
their targets and reset when agent identity changes.
target cadence with a five-minute minimum grace. The server-authored receipt
time is authoritative for reporting freshness; the agent-authored check time
remains observation metadata and must not create or suppress a disconnect when
clocks differ. Missing, stale, or wrong-agent results are indeterminate
monitoring evidence: their `network-endpoint` resources degrade to warning
without manufacturing a target reachability incident. Monitoring aggregates
stale targets by current agent, updates the single canonical probe alert
lifecycle, and treats a fresh result from that same assignment as recovery.
Assignment trackers are removed with their targets and reset when agent identity
changes.
## Canonical Files
@@ -999,9 +1002,12 @@ to resolve. Every completed probe also authors an operational-trust
`EvidenceEnvelope` with provider `availability`, collector
`availability-poller`, the saved target as its provider reference, the exact
observation/ingest times, and a validity window of twice the effective polling
interval. Before the first completed probe, evidence is explicitly partial and
unknown with reason `availability_not_observed`; monitoring must never encode
that state as a confirmed failure or a healthy observation. The registry owns
interval for local checks. Remote-probe evidence instead uses its canonical
server-receipt freshness window (three effective intervals with the five-minute
minimum), so resource evidence, Connections state, and the probe alert lifecycle
cannot disagree. Before the first completed probe, evidence is explicitly
partial and unknown with reason `availability_not_observed`; monitoring must
never encode that state as a confirmed failure or a healthy observation. The registry owns
binding the source envelope to the check resource and cloning a separately
bound envelope for any matched-resource facet projection after correlation.
Availability target kind is monitoring-owned runtime metadata, not a frontend
@@ -1029,10 +1035,12 @@ local and the normal poll provider resumes the target on its next planning
cycle, without a restart. Reported results are accepted only from the agent that
currently owns the target, and results for any other target or from any other
agent are dropped. Failure accounting, thresholds, and incident projection stay
server-side. When an assigned agent stops reporting, monitoring derives
indeterminate with a stale-report explanation at read time through the shared
probe-status snapshot rather than mutating stored state, so every availability
consumer sees the same staleness verdict.
server-side. The agent-authored observation time remains visible as the target's
last check, but staleness uses server receipt time so slow or fast agent clocks
cannot manufacture or conceal a disconnect. When an assigned agent stops
reporting, monitoring derives indeterminate with a stale-report explanation at
read time through the shared probe-status snapshot rather than mutating stored
state, so every availability consumer sees the same staleness verdict.
Mock-mode Discovery context follows the same fixture-graph rule. Demo service
details such as detected version, config/data/log paths, Docker bind mounts,
ports, and suggested web URLs may be authored in mock fixtures, but consumers
@@ -84,7 +84,11 @@ indeterminate reachability evidence. Even when its target is a NAS, PBS,
datastore, or other storage resource, the resulting warning and canonical
probe alert are not backup failure, protection drift, snapshot damage,
recovery-point loss, or restore-readiness evidence. Storage/recovery consumers
must wait for their domain-owned observations.
must wait for their domain-owned observations. The shared Connections row and
availability evidence use server receipt time for remote-probe freshness while
retaining agent check time as observation metadata; storage/recovery must not
reinterpret either timestamp as backup recency, datastore health, or recovery
freshness.
## Canonical Files
+7 -5
View File
@@ -509,11 +509,13 @@ its signed agent configuration, runs it on the configured interval, and
reports results back with its reports; results are only accepted from the
currently assigned agent. An assigned check is not also run locally. If no
report arrives for several intervals the check shows as indeterminate
("no recent report from probe agent"). After at least five minutes without a
report, Pulse raises one warning per disconnected probe through the normal
notification routes, even when that probe owns several checks. The incident is
keyed to the probe agent rather than an arbitrary check, so changing assignments
does not reopen it. If the agent heartbeat is also offline, the existing
("no recent report from probe agent"). Freshness uses the Pulse server's receipt
time, so clock drift on the probe cannot create or hide a disconnect. After at
least five minutes without a report, Pulse raises one warning per disconnected
probe through the normal notification routes, even when that probe owns several
checks. The incident is keyed to the probe agent rather than an arbitrary check,
so changing assignments does not reopen it. If the agent heartbeat is also
offline, the existing
host-offline alert owns the incident instead of producing a duplicate probe
warning. A paired Pulse Mobile client receives a privacy-safe
`external_probe_offline` Relay push linked to that canonical alert. If the whole
+8 -1
View File
@@ -1366,6 +1366,9 @@ func deriveAvailabilityConnectionState(target config.AvailabilityTarget, status
var lastSeen *time.Time
if !status.LastSuccess.IsZero() {
t := status.LastSuccess
if freshness := status.FreshnessTime(); strings.TrimSpace(status.ProbeAgentID) != "" && !freshness.IsZero() {
t = freshness
}
lastSeen = &t
}
@@ -1392,7 +1395,11 @@ func deriveAvailabilityConnectionState(target config.AvailabilityTarget, status
}
return ConnectionStateUnreachable, reason, lastSeen, lastError
}
if lastSeen != nil {
// Probe-assigned statuses have already passed through monitoring's
// canonical receipt-time staleness derivation. Do not apply this generic
// LastSuccess clock a second time or the Connections projection can
// disagree with the resource and alert lifecycle.
if lastSeen != nil && strings.TrimSpace(status.ProbeAgentID) == "" {
staleThreshold := time.Duration(target.EffectivePollIntervalSecs()*2) * time.Second
if staleThreshold < connectionStaleThreshold {
staleThreshold = connectionStaleThreshold
@@ -239,6 +239,40 @@ func TestBuildConnections_AvailabilityTargetStateFromProbeStatus(t *testing.T) {
}
}
func TestDeriveAvailabilityConnectionStateUsesCanonicalRemoteProbeFreshness(t *testing.T) {
receivedAt := time.Date(2026, 7, 29, 10, 0, 0, 0, time.UTC)
agentCheckedAt := receivedAt.Add(-24 * time.Hour)
target := config.NormalizeAvailabilityTarget(config.AvailabilityTarget{
ID: "remote-sensor",
Name: "Remote sensor",
Address: "sensor.example",
Protocol: config.AvailabilityProbeICMP,
Enabled: true,
PollIntervalSecs: 60,
ProbeAgentID: "probe-agent",
})
status := monitoring.AvailabilityProbeStatus{
TargetID: target.ID,
Available: true,
LastChecked: agentCheckedAt,
LastSuccess: agentCheckedAt,
ProbeAgentID: "probe-agent",
ProbeReportReceivedAt: receivedAt,
}
state, reason, lastSeen, lastError := deriveAvailabilityConnectionState(
target,
status,
receivedAt.Add(4*time.Minute),
)
if state != ConnectionStateActive || reason != "" || lastError != nil {
t.Fatalf("connection = state %q reason %q error %+v, want active", state, reason, lastError)
}
if lastSeen == nil || !lastSeen.Equal(receivedAt) {
t.Fatalf("last seen = %v, want server receipt %v", lastSeen, receivedAt)
}
}
func TestBuildConnections_AgentHostAliasesIncludeReportedIdentityHints(t *testing.T) {
now := time.Now()
in := aggregatorInputs{
+51 -8
View File
@@ -33,6 +33,21 @@ type AvailabilityProbeStatus struct {
LastError string `json:"lastError,omitempty"`
FailureThreshold int `json:"failureThreshold,omitempty"`
ProbeAgentID string `json:"probeAgentId,omitempty"`
// ProbeReportReceivedAt is server-authored freshness evidence for a remote
// observation. Keep it off the wire: LastChecked remains the agent's
// observation time, while disconnect detection must not trust agent clock
// skew in either direction.
ProbeReportReceivedAt time.Time `json:"-"`
}
// FreshnessTime returns the authoritative liveness reference for this status.
// Remote observations use server receipt time; local observations use their
// server-authored check time.
func (s AvailabilityProbeStatus) FreshnessTime() time.Time {
if strings.TrimSpace(s.ProbeAgentID) != "" && !s.ProbeReportReceivedAt.IsZero() {
return s.ProbeReportReceivedAt
}
return s.LastChecked
}
// AvailabilityProbeOutcome and its values are aliases for the shared probe
@@ -308,7 +323,7 @@ func (m *Monitor) pollAvailabilityTarget(ctx context.Context, target config.Avai
outcome, err := ProbeAvailabilityTargetResult(ctx, target)
latency := time.Since(start)
checkedAt := time.Now().UTC()
m.applyAvailabilityObservation(target, checkedAt, latency, outcome, err, "")
m.applyAvailabilityObservation(target, checkedAt, latency, outcome, err, "", time.Time{})
m.updateResourceStore(m.GetState())
}
@@ -322,8 +337,9 @@ func (m *Monitor) applyAvailabilityObservation(
outcome AvailabilityProbeOutcome,
probeErr error,
probeAgentID string,
probeReportReceivedAt time.Time,
) {
m.setAvailabilityStatus(target, checkedAt, latency, outcome, probeErr, probeAgentID)
m.setAvailabilityStatus(target, checkedAt, latency, outcome, probeErr, probeAgentID, probeReportReceivedAt)
if probeErr == nil {
if m.stalenessTracker != nil {
@@ -339,7 +355,15 @@ func (m *Monitor) applyAvailabilityObservation(
m.recordTaskResult(InstanceTypeAvailability, target.ID, nil)
}
func (m *Monitor) setAvailabilityStatus(target config.AvailabilityTarget, checkedAt time.Time, latency time.Duration, outcome AvailabilityProbeOutcome, probeErr error, probeAgentID string) {
func (m *Monitor) setAvailabilityStatus(
target config.AvailabilityTarget,
checkedAt time.Time,
latency time.Duration,
outcome AvailabilityProbeOutcome,
probeErr error,
probeAgentID string,
probeReportReceivedAt time.Time,
) {
if m == nil {
return
}
@@ -347,6 +371,9 @@ func (m *Monitor) setAvailabilityStatus(target config.AvailabilityTarget, checke
status.Outcome = string(outcome)
status.LastChecked = checkedAt
status.ProbeAgentID = strings.TrimSpace(probeAgentID)
if status.ProbeAgentID != "" {
status.ProbeReportReceivedAt = probeReportReceivedAt.UTC()
}
latencyMs := latency.Milliseconds()
if probeErr == nil && latencyMs == 0 {
latencyMs = 1
@@ -414,7 +441,7 @@ func availabilityStatusFromTarget(target config.AvailabilityTarget) Availability
}
func availabilityResourceFromTarget(target config.AvailabilityTarget, status AvailabilityProbeStatus, _ string, now time.Time) (unifiedresources.Resource, unifiedresources.ResourceIdentity) {
lastSeen := status.LastChecked
lastSeen := status.FreshnessTime()
if lastSeen.IsZero() {
lastSeen = now
}
@@ -442,7 +469,19 @@ func availabilityResourceFromTarget(target config.AvailabilityTarget, status Ava
PollIntervalSeconds: target.EffectivePollIntervalSecs(),
TimeoutMillis: target.EffectiveTimeoutMillis(),
}
data.Evidence = availabilityEvidenceEnvelope(target, status, lastSeen, now)
observedAt := status.LastChecked
if observedAt.IsZero() {
observedAt = lastSeen
}
if observedAt.After(lastSeen) {
// A fast remote clock cannot author evidence in the server's future.
observedAt = lastSeen
}
ingestedAt := now
if strings.TrimSpace(status.ProbeAgentID) != "" && !status.ProbeReportReceivedAt.IsZero() {
ingestedAt = status.ProbeReportReceivedAt
}
data.Evidence = availabilityEvidenceEnvelope(target, status, observedAt, ingestedAt)
resource := unifiedresources.Resource{
Type: unifiedresources.ResourceTypeNetworkEndpoint,
Technology: string(target.Protocol),
@@ -498,9 +537,13 @@ func availabilityEvidenceEnvelope(
return nil
}
validUntil := observedAt.Add(
time.Duration(target.EffectivePollIntervalSecs()*2) * time.Second,
)
freshnessAt := observedAt
validityWindow := time.Duration(target.EffectivePollIntervalSecs()*2) * time.Second
if strings.TrimSpace(status.ProbeAgentID) != "" && !status.ProbeReportReceivedAt.IsZero() {
freshnessAt = status.ProbeReportReceivedAt
validityWindow = availabilityProbeStaleWindow(target)
}
validUntil := freshnessAt.Add(validityWindow)
completeness := operationaltrust.EvidenceComplete
confidence := operationaltrust.EvidenceConfirmed
var reason *operationaltrust.EvidenceReason
@@ -83,6 +83,14 @@ func (m *Monitor) effectiveProbeAgentID(target config.AvailabilityTarget) string
// ApplyProbeAvailabilityResults ingests availability results reported by a host
// agent. Results are accepted only for targets currently assigned to that agent.
func (m *Monitor) ApplyProbeAvailabilityResults(hostID string, results []ProbeAvailabilityResult) {
m.applyProbeAvailabilityResultsAt(hostID, results, time.Now().UTC())
}
// applyProbeAvailabilityResultsAt keeps server receipt time explicit for the
// host-report path and deterministic tests. Agent-authored CheckedAt describes
// the observation; receivedAt is the authoritative signal that the probe is
// still reporting.
func (m *Monitor) applyProbeAvailabilityResultsAt(hostID string, results []ProbeAvailabilityResult, receivedAt time.Time) {
if m == nil {
return
}
@@ -90,6 +98,11 @@ func (m *Monitor) ApplyProbeAvailabilityResults(hostID string, results []ProbeAv
if hostID == "" || len(results) == 0 {
return
}
if receivedAt.IsZero() {
receivedAt = time.Now().UTC()
} else {
receivedAt = receivedAt.UTC()
}
applied := 0
for _, result := range results {
@@ -115,14 +128,14 @@ func (m *Monitor) ApplyProbeAvailabilityResults(hostID string, results []ProbeAv
checkedAt := result.CheckedAt
if checkedAt.IsZero() {
checkedAt = time.Now()
checkedAt = receivedAt
}
latency := time.Duration(result.LatencyMillis) * time.Millisecond
if latency < 0 {
latency = 0
}
outcome, probeErr := probeResultOutcome(result)
m.applyAvailabilityObservation(target, checkedAt.UTC(), latency, outcome, probeErr, hostID)
m.applyAvailabilityObservation(target, checkedAt.UTC(), latency, outcome, probeErr, hostID, receivedAt)
applied++
}
@@ -176,7 +189,7 @@ func (m *Monitor) deriveAvailabilityProbeStaleness(
status = availabilityStatusFromTarget(target)
}
status.ProbeAgentID = agentID
reference := status.LastChecked
reference := status.FreshnessTime()
if reference.IsZero() {
reference = m.availabilityProbeAssignmentReference(target.ID, agentID, now)
}
@@ -224,11 +237,15 @@ func availabilityProbeReportIsStale(target config.AvailabilityTarget, lastChecke
if lastChecked.IsZero() {
return true
}
return now.Sub(lastChecked) > availabilityProbeStaleWindow(target)
}
func availabilityProbeStaleWindow(target config.AvailabilityTarget) time.Duration {
window := time.Duration(target.EffectivePollIntervalSecs()) * 3 * time.Second
if window < availabilityProbeStaleFloor {
window = availabilityProbeStaleFloor
return availabilityProbeStaleFloor
}
return now.Sub(lastChecked) > window
return window
}
// availabilityProbeTargetsForAgent returns the probe payload for the targets the
@@ -337,9 +337,9 @@ func TestAvailabilityProbeStalenessDerivesIndeterminateAtReadTime(t *testing.T)
monitor.SetLicenseChecker(licenseWithExternalProbe(true))
checkedAt := time.Now().UTC()
monitor.ApplyProbeAvailabilityResults("agent-1", []ProbeAvailabilityResult{
monitor.applyProbeAvailabilityResultsAt("agent-1", []ProbeAvailabilityResult{
{TargetID: "remote", Outcome: AvailabilityProbeReachable, LatencyMillis: 9, CheckedAt: checkedAt},
})
}, checkedAt)
// Effective poll interval is 60s, so the floor of five minutes governs.
fresh := monitor.availabilityStatusSnapshotForTargets([]config.AvailabilityTarget{target}, checkedAt.Add(5*time.Minute))
@@ -382,13 +382,69 @@ func TestAvailabilityProbeStalenessDerivesIndeterminateAtReadTime(t *testing.T)
}
}
func TestAvailabilityProbeStalenessUsesServerReceiptTimeAcrossAgentClockSkew(t *testing.T) {
target := config.NormalizeAvailabilityTarget(probeAgentTarget("remote", "agent-1"))
monitor := newProbeAgentTestMonitor(t, target)
monitor.SetLicenseChecker(licenseWithExternalProbe(true))
receivedAt := time.Now().UTC()
for _, test := range []struct {
name string
checkedAt time.Time
}{
{name: "slow agent clock", checkedAt: receivedAt.Add(-24 * time.Hour)},
{name: "fast agent clock", checkedAt: receivedAt.Add(24 * time.Hour)},
} {
t.Run(test.name, func(t *testing.T) {
monitor.applyProbeAvailabilityResultsAt("agent-1", []ProbeAvailabilityResult{
{TargetID: "remote", Outcome: AvailabilityProbeReachable, CheckedAt: test.checkedAt},
}, receivedAt)
fresh := monitor.availabilityStatusSnapshotForTargets(
[]config.AvailabilityTarget{target},
receivedAt.Add(availabilityProbeStaleFloor),
)["remote"]
if !fresh.Available || availabilityProbeStatusIsStale(fresh) {
t.Fatalf("boundary status = %+v, server receipt must keep the report fresh", fresh)
}
if !fresh.LastChecked.Equal(test.checkedAt) {
t.Fatalf("last checked = %v, want agent observation time %v preserved", fresh.LastChecked, test.checkedAt)
}
resource, _ := availabilityResourceFromTarget(target, fresh, "", receivedAt)
if !resource.LastSeen.Equal(receivedAt) {
t.Fatalf("resource last seen = %v, want server receipt %v", resource.LastSeen, receivedAt)
}
evidence := resource.Availability.Evidence
if evidence == nil || !evidence.IngestedAt.Equal(receivedAt) {
t.Fatalf("evidence = %+v, want server receipt as ingest time", evidence)
}
wantValidUntil := receivedAt.Add(availabilityProbeStaleWindow(target))
if evidence.ValidUntil == nil || !evidence.ValidUntil.Equal(wantValidUntil) {
t.Fatalf("evidence valid until = %v, want %v", evidence.ValidUntil, wantValidUntil)
}
if evidence.ObservedAt.After(evidence.IngestedAt) {
t.Fatalf("evidence chronology = observed %v after ingested %v", evidence.ObservedAt, evidence.IngestedAt)
}
stale := monitor.availabilityStatusSnapshotForTargets(
[]config.AvailabilityTarget{target},
receivedAt.Add(availabilityProbeStaleFloor+time.Second),
)["remote"]
if !availabilityProbeStatusIsStale(stale) {
t.Fatalf("post-window status = %+v, want stale by server receipt time", stale)
}
})
}
}
func TestAvailabilitySupplementalRecordsPresentStaleProbeAsIndeterminate(t *testing.T) {
monitor := newProbeAgentTestMonitor(t, probeAgentTarget("remote", "agent-1"))
monitor.SetLicenseChecker(licenseWithExternalProbe(true))
monitor.ApplyProbeAvailabilityResults("agent-1", []ProbeAvailabilityResult{
{TargetID: "remote", Outcome: AvailabilityProbeReachable, LatencyMillis: 9, CheckedAt: time.Now().UTC().Add(-time.Hour)},
})
old := time.Now().UTC().Add(-time.Hour)
monitor.applyProbeAvailabilityResultsAt("agent-1", []ProbeAvailabilityResult{
{TargetID: "remote", Outcome: AvailabilityProbeReachable, LatencyMillis: 9, CheckedAt: old},
}, old)
records := availabilityPollProvider{}.SupplementalRecords(monitor, "org-a")
if len(records) != 1 {
@@ -487,13 +543,13 @@ func TestAvailabilityProbeStaleIncidentDeduplicatesPerAgent(t *testing.T) {
t.Cleanup(alertManager.Stop)
monitor.alertManager = alertManager
old := time.Now().UTC().Add(-time.Hour)
monitor.ApplyProbeAvailabilityResults("agent-1", []ProbeAvailabilityResult{
monitor.applyProbeAvailabilityResultsAt("agent-1", []ProbeAvailabilityResult{
{TargetID: "remote-a", Outcome: AvailabilityProbeReachable, CheckedAt: old},
{TargetID: "remote-b", Outcome: AvailabilityProbeReachable, CheckedAt: old},
})
monitor.ApplyProbeAvailabilityResults("agent-2", []ProbeAvailabilityResult{
}, old)
monitor.applyProbeAvailabilityResultsAt("agent-2", []ProbeAvailabilityResult{
{TargetID: "remote-c", Outcome: AvailabilityProbeReachable, CheckedAt: old},
})
}, old)
_ = (availabilityPollProvider{}).SupplementalRecords(monitor, "")
alertsByAgent := make(map[string]int)
@@ -515,9 +571,10 @@ func TestAvailabilityProbeFreshReportResolvesStaleIncident(t *testing.T) {
alertManager := alerts.NewManagerWithDataDir(t.TempDir())
t.Cleanup(alertManager.Stop)
monitor.alertManager = alertManager
monitor.ApplyProbeAvailabilityResults("agent-1", []ProbeAvailabilityResult{
{TargetID: "remote", Outcome: AvailabilityProbeReachable, CheckedAt: time.Now().UTC().Add(-time.Hour)},
})
old := time.Now().UTC().Add(-time.Hour)
monitor.applyProbeAvailabilityResultsAt("agent-1", []ProbeAvailabilityResult{
{TargetID: "remote", Outcome: AvailabilityProbeReachable, CheckedAt: old},
}, old)
_ = (availabilityPollProvider{}).SupplementalRecords(monitor, "")
if got := alertManager.GetActiveAlerts(); len(got) != 1 || got[0].Type != alerts.ExternalProbeUnavailableAlertType {
t.Fatalf("stale alerts = %+v, want one before recovery", got)
@@ -547,9 +604,10 @@ func TestAvailabilityProbeHostOfflineOwnsConnectivityLifecycle(t *testing.T) {
LastSeen: time.Now().UTC().Add(-time.Hour),
IntervalSeconds: 30,
})
monitor.ApplyProbeAvailabilityResults("agent-1", []ProbeAvailabilityResult{
{TargetID: "remote", Outcome: AvailabilityProbeReachable, CheckedAt: time.Now().UTC().Add(-time.Hour)},
})
old := time.Now().UTC().Add(-time.Hour)
monitor.applyProbeAvailabilityResultsAt("agent-1", []ProbeAvailabilityResult{
{TargetID: "remote", Outcome: AvailabilityProbeReachable, CheckedAt: old},
}, old)
_ = (availabilityPollProvider{}).SupplementalRecords(monitor, "")
for _, alert := range alertManager.GetActiveAlerts() {
+1 -1
View File
@@ -86,7 +86,7 @@ func TestAvailabilityUDPIndeterminateDoesNotClaimReachabilityOrAccumulateFailure
}}
checkedAt := time.Now().UTC()
monitor.setAvailabilityStatus(target, checkedAt, 250*time.Millisecond, AvailabilityProbeIndeterminate, nil, "")
monitor.setAvailabilityStatus(target, checkedAt, 250*time.Millisecond, AvailabilityProbeIndeterminate, nil, "", time.Time{})
status := monitor.AvailabilityStatusSnapshot()[target.ID]
if status.Available {
t.Fatal("indeterminate UDP result must not be reported as reachable")