Add multi-location service delivery evidence

This commit is contained in:
Pulse Test
2026-08-30 21:44:32 +01:00
parent e094a55b45
commit 16a1574a18
30 changed files with 1403 additions and 208 deletions
@@ -1,7 +1,7 @@
# Infrastructure-Aware Service Assurance Plan # Infrastructure-Aware Service Assurance Plan
Last updated: 2026-08-30 Last updated: 2026-08-30
Status: ACCEPTED — SLICES A, B, AND C IMPLEMENTED Status: ACCEPTED — SLICES A, B, C, AND D IMPLEMENTED
Governance surfaces: Governance surfaces:
- `status.json.coverage_gaps.infrastructure-aware-service-assurance` - `status.json.coverage_gaps.infrastructure-aware-service-assurance`
@@ -228,7 +228,7 @@ Implementation record (2026-08-30): Slice A is delivered through the
monitoring-owned categorical history store and rollups, server-authored monitoring-owned categorical history store and rollups, server-authored
configuration revisions and remote receipt timeline, the bounded configuration revisions and remote receipt timeline, the bounded
`/api/availability-history` batch contract, and the URL-owned Availability `/api/availability-history` batch contract, and the URL-owned Availability
fleet presentation. Its release proofs live in the owner contract. Slices D fleet presentation. Its release proofs live in the owner contract. Slices E
through H remain ordered future work; acceptance of this product lane does not through H remain ordered future work; acceptance of this product lane does not
imply that deferred breadth is already delivered. imply that deferred breadth is already delivered.
@@ -269,7 +269,7 @@ re-entry. Current status and unified-resource facets preserve transport
reachability separately from typed application correctness while the overall reachability separately from typed application correctness while the overall
result remains the alert and history outcome. Legacy targets without an result remains the alert and history outcome. Legacy targets without an
explicit contract retain their previous HEAD-with-bounded-GET-fallback explicit contract retain their previous HEAD-with-bounded-GET-fallback
semantics. Slices D through H remain ordered future work. semantics. Slices E through H remain ordered future work.
### Slice C: Discovery-Led Assurance Onboarding ### Slice C: Discovery-Led Assurance Onboarding
@@ -304,7 +304,7 @@ standalone endpoints, and creates one explicitly enabled availability target
with the drawer's canonical resource ID only after the operator chooses the with the drawer's canonical resource ID only after the operator chooses the
activation action. A machine-scoped queue supports bulk review and activation action. A machine-scoped queue supports bulk review and
evidence-bound dismiss/restore without bulk activation or guessed resource evidence-bound dismiss/restore without bulk activation or guessed resource
attachment. Slices D through H remain ordered future work. attachment. Slices E through H remain ordered future work.
### Slice D: Multi-Location Delivery Evidence ### Slice D: Multi-Location Delivery Evidence
@@ -332,6 +332,23 @@ Exit conditions:
- location names and customer-identifying network details obey the existing - location names and customer-identifying network details obey the existing
privacy and redaction policy. privacy and redaction policy.
Implementation record (2026-08-30): Slice D is delivered through a canonical
set of source-owned observation-location IDs on each saved target. The local
Pulse runtime and any selected eligible connected agents may observe the same
logical verification concurrently; monitoring retains their state, latency,
server-receipt freshness, and coverage separately, then derives one aggregate
state without duplicating the target resource. A reachable path plus a failed
or missing path is `degraded`, every current path must be unreachable before
the target is `unavailable`, and incomplete all-failed evidence remains
`unknown`. Stale agent paths become indeterminate evidence, retries remain
idempotent through the existing observation ID and configuration revision,
and only the assigned agent may author its location. Settings presents the
location set as a bounded multi-select, while the Availability fleet and
resource detail surfaces expose reporting coverage and path disagreement.
Compatibility with the former single `probeAgentId` contract is retained at
the API boundary without keeping single-agent execution as the canonical
model. Slices E through H remain ordered future work.
### Slice E: Infrastructure-Aware Incident Synthesis ### Slice E: Infrastructure-Aware Incident Synthesis
Deliver deterministic incident grouping and causal narrowing over canonical Deliver deterministic incident grouping and causal narrowing over canonical
@@ -6211,6 +6211,17 @@ remain ingestible through a deterministic server fallback, but they do not
gain authority to author revision boundaries, target identity, retention, or gain authority to author revision boundaries, target identity, retention, or
the service-assurance timeline. the service-assurance timeline.
The server may assign one target to several agent-backed observation locations,
but each agent receives and reports only the targets whose canonical
`agent:<agent-id>` location it owns. Location membership is server-authored
configuration, not agent enrollment identity: it does not grant control-plane
authority, create another agent resource, or let one reporter stand in for a
different site. Reassignment increments the target configuration revision, so
late results from the previous location set are rejected. Local Pulse execution
can coexist with assigned agent execution for the same logical target, while
license lapse removes every remote location from the effective assignment and
leaves the local path as the fail-safe execution boundary.
Observer configuration is explicit, versioned, and file-backed. It contains no Observer configuration is explicit, versioned, and file-backed. It contains no
raw token values and resolves each token from a separate private absolute-path raw token values and resolves each token from a separate private absolute-path
file. Proxmox registration is also destination-scoped: the primary retains its file. Proxmox registration is also destination-scoped: the primary retains its
@@ -8808,6 +8808,18 @@ metadata, but API freshness and `lastSeenAt` projections use the server-authored
receipt time supplied by monitoring. API consumers must not substitute the receipt time supplied by monitoring. API consumers must not substitute the
agent clock for disconnect detection or apply a second, shorter generic agent clock for disconnect detection or apply a second, shorter generic
Connections staleness window. Connections staleness window.
Availability target writes now use `observationLocationIds` as the canonical
bounded set. Values are source-owned IDs (`pulse:local` or
`agent:<agent-id>`), are normalized and deduplicated by the server, and every
remote member requires the external-probe entitlement plus an existing agent.
The compatibility `probeAgentId` field is accepted for older single-agent
clients and is emitted only as a compatibility hint when the selected set can
be represented by one remote path. Current status and unified-resource
transport add aggregate state, disagreement, expected/reporting counts, and a
per-location list containing only location ID, state, latency, observation and
receipt times, freshness, and a bounded failure code. Callers must consume that
server-authored aggregate rather than treating the latest reporting path as the
whole service result.
Mock availability fixtures must still behave like saved targets: `/api/connections` Mock availability fixtures must still behave like saved targets: `/api/connections`
reports them as availability rows, `/api/availability-targets` lists them with reports them as availability rows, `/api/availability-targets` lists them with
probe status, and saved-test calls return the synthetic probe result instead of probe status, and saved-test calls return the synthetic probe result instead of
@@ -1566,24 +1566,35 @@ the frontend-primitives-owned Standalone surface may read the same
`network-endpoint` projection to show current reachability, latency, check age, `network-endpoint` projection to show current reachability, latency, check age,
and failure state without creating another monitoring provider or top-level and failure state without creating another monitoring provider or top-level
availability route. availability route.
Availability targets may also be assigned to a remote host agent. Reachability Availability execution is location-owned. Each saved target has a normalized,
outcome and the optional certificate observation travel in the same bounded deduplicated set of observation-location IDs: `pulse:local` for this Pulse
report entry, and the server clones that observation before status and resource runtime and `agent:<agent-id>` for an eligible connected host agent. The legacy
projection so report buffers cannot alias live state. While the single `probeAgentId` field remains an input compatibility boundary only; the
`external_probe` entitlement is active, a probe-assigned target is executed location set owns scheduling, configuration revision, assignment, status, and
exclusively by its assigned agent: monitoring must not schedule or run it resource projection. While `external_probe` is entitled, Pulse may schedule
locally, so the check never executes twice. The assignment is effective only for the local path and every selected agent path for one logical target at the same
as long as the entitlement holds; on lapse the effective assignment collapses to time. On entitlement lapse the effective set collapses to the local path
local and the normal poll provider resumes the target on its next planning without a restart. An agent result is accepted only for its exact selected
cycle, without a restart. Reported results are accepted only from the agent that location and current target revision; another agent cannot author that path.
currently owns the target, and results for any other target or from any other
agent are dropped. Failure accounting, thresholds, and incident projection stay Reachability outcome and the optional certificate observation still travel in
server-side. The agent-authored observation time remains visible as the target's the same bounded report entry, and the server clones that observation before
last check, but staleness uses server receipt time so slow or fast agent clocks status and resource projection so report buffers cannot alias live state.
cannot manufacture or conceal a disconnect. When an assigned agent stops Monitoring retains one current status per target and location, using server
reporting, monitoring derives indeterminate with a stale-report explanation at receipt time for remote freshness while keeping the agent-authored observation
read time through the shared probe-status snapshot rather than mutating stored time as evidence metadata. A stale or disconnected agent path derives as
state, so every availability consumer sees the same staleness verdict. indeterminate at read time; slow or fast agent clocks cannot manufacture or
conceal a disconnect. Aggregation is conservative: all reachable paths are
healthy, mixed reachable and failed/unknown paths are degraded, all current
paths unreachable are unavailable, and no reachable path with incomplete or
indeterminate coverage is unknown. Only aggregate unavailability advances the
target failure threshold or authors an outage incident. A path-local failure
therefore cannot claim that the service is universally down. Single-location
targets preserve their prior status and history semantics. Multi-location
targets write only their aggregate service conclusion to the logical target's
deletable categorical history; current per-location evidence remains a status
and resource projection until a separately governed location-history schema
can preserve deletion, retention, and bounded-query semantics end to end.
Mock-mode Discovery context follows the same fixture-graph rule. Demo service Mock-mode Discovery context follows the same fixture-graph rule. Demo service
details such as detected version, config/data/log paths, Docker bind mounts, details such as detected version, config/data/log paths, Docker bind mounts,
ports, and suggested web URLs may be authored in mock fixtures, but consumers ports, and suggested web URLs may be authored in mock fixtures, but consumers
@@ -1331,6 +1331,14 @@ alerts. Those counts may describe scale and feature adoption, but they must not
include hostnames, resource IDs, infrastructure identifiers, credentials, include hostnames, resource IDs, infrastructure identifiers, credentials,
prompts, chat messages, command text, action output, token values, or personal prompts, chat messages, command text, action output, token values, or personal
information. information.
Multi-location availability evidence stays inside the same privacy floor.
Saved configuration and authenticated operator surfaces may use source-owned
location IDs and agent display names to distinguish selected paths, but
categorical history, evidence envelopes, telemetry, incidents, logs, and
Patrol context must not retain or export customer network names, addresses,
agent identity, or location labels. Durable categorical history remains keyed
only by the logical target and its aggregate conclusion; location identity is
not a new telemetry, history, or customer-identity dimension.
That same outbound usage telemetry floor now also permits content-free update That same outbound usage telemetry floor now also permits content-free update
funnel counters derived from local update history inside the same rotating funnel counters derived from local update history inside the same rotating
30-day telemetry window: update attempts, successful updates, failed or 30-day telemetry window: update attempts, successful updates, failed or
@@ -4670,6 +4670,18 @@ the shared resource drawer by canonical resource ID, while
already carried by that resource. History absence or failure cannot overwrite already carried by that resource. History absence or failure cannot overwrite
the unified resource's current status, correlation, source ownership, or the unified resource's current status, correlation, source ownership, or
navigation identity. navigation identity.
One availability resource may now carry several source-owned observation
locations without cloning the resource, relationship, or fleet row. Its
availability facet preserves the aggregate state, disagreement flag,
expected/reporting coverage, and the current state, latency, freshness, and
bounded failure reason for each path. Resource status remains active with
warning evidence when paths disagree, becomes stale/incomplete when coverage
is unknown, and becomes inactive with an outage incident only when every
current path is unreachable at the configured threshold. Frontend adapters,
REST, websocket, and mock projections must preserve this list rather than
flattening it to the most recent observer. The Availability table may summarize
the count and reporting coverage; the shared resource detail card owns the
expanded path comparison.
For HTTP/S checks with an explicit application response contract, the For HTTP/S checks with an explicit application response contract, the
availability facet preserves two source-owned facts: `transportOutcome` availability facet preserves two source-owned facts: `transportOutcome`
describes whether the endpoint answered, while `applicationOutcome`, describes whether the endpoint answered, while `applicationOutcome`,
+26 -20
View File
@@ -1,24 +1,29 @@
{ {
"version": 1, "version": 1,
"base_sha": "d9e9bf3679ab579be9f8c7350932008c287ad5ad", "base_sha": "e094a55b4500d2a10b6ab44b350c2fffcc55e73d",
"verified_at": "2026-08-30T20:24:19Z", "verified_at": "2026-08-30T20:37:10Z",
"result": "passed", "result": "passed",
"changed_paths": [ "changed_paths": [
"frontend-modern/src/components/Infrastructure/resourceDetailMappers.ts", "frontend-modern/src/api/availabilityTargets.ts",
"frontend-modern/src/features/docker/DockerHostDrawerOverview.tsx", "frontend-modern/src/components/Infrastructure/AvailabilityProbeStatusCard.tsx",
"frontend-modern/src/hooks/useWorkloads.ts", "frontend-modern/src/components/Settings/ConnectionEditor/CredentialSlots/AvailabilityTargetSlot.tsx",
"frontend-modern/src/types/api.ts", "frontend-modern/src/components/Settings/availabilitySettingsModel.ts",
"frontend-modern/src/types/resource.ts" "frontend-modern/src/features/standalone/AvailabilityChecksTable.tsx",
"frontend-modern/src/types/resource.ts",
"frontend-modern/src/utils/availabilityProbeAgents.ts"
], ],
"content_sha256": { "content_sha256": {
"frontend-modern/src/components/Infrastructure/resourceDetailMappers.ts": "8b1913ff24e6f5dbf074091f2bbaff1a8c3298126515ea288b055a02c0153997", "frontend-modern/src/api/availabilityTargets.ts": "0a20277a8c453b753e8c09f7250722670211a33bcae2addad7d19502f921f365",
"frontend-modern/src/features/docker/DockerHostDrawerOverview.tsx": "621af65f92c9d58312f019790ecf2eb6986fb260724e9ef53213972fd9b55579", "frontend-modern/src/components/Infrastructure/AvailabilityProbeStatusCard.tsx": "ec13f364ee9c55f6859c448797fed427255999723524ff0f51cb54cc9f864ea5",
"frontend-modern/src/hooks/useWorkloads.ts": "85719e8c93f503843cd09c1be41c66f9995288d1280ca2eedf7d47c3443bd9a1", "frontend-modern/src/components/Settings/ConnectionEditor/CredentialSlots/AvailabilityTargetSlot.tsx": "622e0c7ad72c4f313c8d971295662e6549d9f13c5fbd788d3cc8803c827af176",
"frontend-modern/src/types/api.ts": "be1b1889588085791880dbd6716955710cd66e3dc69339998f8b9ad30c0ac378", "frontend-modern/src/components/Settings/availabilitySettingsModel.ts": "cd03ed43bca379c8359c11c67f235d49a4a1c6b905c987a4ce404ce59443bbf6",
"frontend-modern/src/types/resource.ts": "74b26ae53a507f5608a3c191f7c386cffab636e0a95aeffa02028bd8199297b0" "frontend-modern/src/features/standalone/AvailabilityChecksTable.tsx": "3c775ad5b08023f9a2814a9dc939992db363a7e5113a351d4fb65ad0946f0d97",
"frontend-modern/src/types/resource.ts": "a4e87ae08ec69349122075ca422ced08a56e1f14294088488198c5ff8af25dba",
"frontend-modern/src/utils/availabilityProbeAgents.ts": "7f92c5fc5012bdbb322825b4a03f26fb5dc6d20d260cda8b2c2e86b66283b005"
}, },
"routes": [ "routes": [
"/browser-helper-summary-harness.html (temporary current-build Vite harness for the Docker host drawer)" "/standalone/availability",
"/settings/monitoring/availability?add=target&targetKind=service"
], ],
"viewports": [ "viewports": [
{ {
@@ -31,14 +36,15 @@
} }
], ],
"states": [ "states": [
"typed-helper summary mode with reduced-coverage warning and no container update management card", "multi-location target editor with the local Pulse path and connected-agent paths, including the Community-tier remote-path entitlement boundary",
"direct-runtime mode without the reduced-coverage warning and with container update management controls", "degraded service with one reachable and one unreachable observation path, preserved as Paths disagree with 2/2 reporting",
"return to typed-helper summary mode after direct-runtime mode" "expanded resource evidence showing per-path state, latency, checked time, freshness, aggregate degraded state, and no universal outage",
"desktop and narrow layouts with settled data, dialog bounds, responsive navigation, and no page or dialog horizontal overflow"
], ],
"interactions": [ "interactions": [
"loaded summary mode and confirmed warning visibility, update-control omission, and no desktop horizontal overflow", "opened the add service/device check flow and inspected the bounded observation-location selector without saving a mock target",
"switched to direct-runtime mode and confirmed the warning disappeared and update controls appeared", "opened the Frontend Swarm service resource details from Availability checks and expanded the multi-location evidence",
"switched back to summary mode and confirmed the warning returned and update controls were removed", "confirmed the local path reports 9 ms while edge-london reports Unreachable and the aggregate remains Degraded",
"inspected the final summary state at 390x844 and confirmed no horizontal overflow or clipped warning copy" "repeated the final route, state, clipping, and overflow checks at 1280x800 and 390x844 after rebasing onto the current main base"
] ]
} }
@@ -67,9 +67,33 @@ export interface AvailabilityProbeStatus {
* Absent (omitempty) when the local Pulse server ran the check. * Absent (omitempty) when the local Pulse server ran the check.
*/ */
probeAgentId?: string; probeAgentId?: string;
aggregateState?: 'healthy' | 'degraded' | 'unavailable' | 'unknown' | string;
disagreement?: boolean;
expectedLocations?: number;
reportingLocations?: number;
locations?: AvailabilityObservationLocationStatus[];
certificate?: ResourceCertificateObservation; certificate?: ResourceCertificateObservation;
} }
export interface AvailabilityObservationLocationStatus {
locationId: string;
kind: 'pulse' | 'agent' | string;
probeAgentId?: string;
outcome?: 'reachable' | 'unreachable' | 'indeterminate' | string;
transportOutcome?: string;
applicationOutcome?: string;
applicationStatusCode?: number;
applicationFailureCode?: string;
available: boolean;
lastChecked?: string;
lastSuccess?: string;
freshnessAt?: string;
latencyMillis?: number;
consecutiveFailures?: number;
lastError?: string;
stale?: boolean;
}
export interface AvailabilityTarget { export interface AvailabilityTarget {
id: string; id: string;
configRevision?: number; configRevision?: number;
@@ -96,6 +120,9 @@ export interface AvailabilityTarget {
* string because the server decodes updates onto the existing record. * string because the server decodes updates onto the existing record.
*/ */
probeAgentId?: string; probeAgentId?: string;
/** Canonical execution set. `pulse:local` names this Pulse server and
* `agent:<id>` names a connected Pulse Agent observation location. */
observationLocationIds?: string[];
http?: AvailabilityHTTPConfig; http?: AvailabilityHTTPConfig;
httpSecrets?: AvailabilityHTTPSecretState; httpSecrets?: AvailabilityHTTPSecretState;
status?: AvailabilityProbeStatus; status?: AvailabilityProbeStatus;
@@ -46,8 +46,13 @@ export function AvailabilityProbeStatusCards(props: AvailabilityProbeStatusCards
} }
export function AvailabilityProbeStatusCard(props: AvailabilityProbeStatusCardProps) { export function AvailabilityProbeStatusCard(props: AvailabilityProbeStatusCardProps) {
const isUp = () => props.availability.available === true; const aggregateState = () => props.availability.aggregateState;
const isDown = () => props.availability.available === false; const isUp = () =>
aggregateState() ? aggregateState() === 'healthy' : props.availability.available === true;
const isDegraded = () => aggregateState() === 'degraded';
const isUnknown = () => aggregateState() === 'unknown';
const isDown = () =>
aggregateState() ? aggregateState() === 'unavailable' : props.availability.available === false;
const latency = () => { const latency = () => {
const ms = props.availability.latencyMillis; const ms = props.availability.latencyMillis;
return typeof ms === 'number' && Number.isFinite(ms) && ms > 0 ? `${Math.round(ms)}ms` : null; return typeof ms === 'number' && Number.isFinite(ms) && ms > 0 ? `${Math.round(ms)}ms` : null;
@@ -129,6 +134,19 @@ export function AvailabilityProbeStatusCard(props: AvailabilityProbeStatusCardPr
if (days === 0) return `${date} (today)`; if (days === 0) return `${date} (today)`;
return `${date} (${days}d)`; return `${date} (${days}d)`;
}; };
const locationLabel = (location: NonNullable<ResourceAvailabilityMeta['locations']>[number]) =>
location.kind === 'pulse'
? 'This Pulse server'
: location.probeAgentId || location.locationId.replace(/^agent:/, '') || 'Pulse Agent';
const locationResult = (location: NonNullable<ResourceAvailabilityMeta['locations']>[number]) => {
if (location.stale) return 'No recent report';
if (!location.lastChecked) return 'Awaiting evidence';
if (location.outcome === 'reachable') {
return location.latencyMillis ? `${Math.round(location.latencyMillis)} ms` : 'Reachable';
}
if (location.outcome === 'unreachable') return 'Unreachable';
return 'Unknown';
};
return ( return (
<InfoCardFrame data-testid="availability-probe-status"> <InfoCardFrame data-testid="availability-probe-status">
@@ -145,11 +163,22 @@ export function AvailabilityProbeStatusCard(props: AvailabilityProbeStatusCardPr
'bg-emerald-100 text-emerald-700 dark:bg-emerald-900/40 dark:text-emerald-300': 'bg-emerald-100 text-emerald-700 dark:bg-emerald-900/40 dark:text-emerald-300':
isFreshUp(), isFreshUp(),
'bg-red-100 text-red-700 dark:bg-red-900/40 dark:text-red-300': isDown() && !isStale(), 'bg-red-100 text-red-700 dark:bg-red-900/40 dark:text-red-300': isDown() && !isStale(),
'bg-amber-100 text-amber-700 dark:bg-amber-900/40 dark:text-amber-300': isStale(), 'bg-amber-100 text-amber-700 dark:bg-amber-900/40 dark:text-amber-300':
'bg-base-200 text-muted': !isUp() && !isDown() && !isStale(), isStale() || isDegraded(),
'bg-base-200 text-muted': isUnknown() || (!isUp() && !isDown() && !isStale()),
}} }}
> >
{isStale() ? 'Stale' : isUp() ? 'Up' : isDown() ? 'Down' : 'Not checked'} {isDegraded()
? 'Paths disagree'
: isUnknown()
? 'Coverage incomplete'
: isStale()
? 'Stale'
: isUp()
? 'Up'
: isDown()
? 'Down'
: 'Not checked'}
</span> </span>
</div> </div>
<div class="space-y-1.5 text-[11px]"> <div class="space-y-1.5 text-[11px]">
@@ -171,6 +200,51 @@ export function AvailabilityProbeStatusCard(props: AvailabilityProbeStatusCardPr
</Show> </Show>
} }
/> />
<Show when={(props.availability.locations?.length ?? 0) > 0}>
<div
class="mt-2 space-y-1.5 border-t border-border pt-2"
data-observation-location-results
>
<div class="flex items-center justify-between gap-2 text-[10px] uppercase tracking-wide text-muted">
<span>Observation paths</span>
<span>
{props.availability.reportingLocations ?? 0}/
{props.availability.expectedLocations ?? props.availability.locations?.length ?? 0}{' '}
reporting
</span>
</div>
<For each={props.availability.locations}>
{(location) => (
<div class="flex items-start justify-between gap-3 rounded bg-base-200/60 px-2 py-1.5">
<div class="min-w-0">
<div
class="truncate font-medium text-base-content"
title={locationLabel(location)}
>
{locationLabel(location)}
</div>
<Show when={formatRelativeTime(location.lastChecked)}>
{(checked) => <div class="text-[10px] text-muted">checked {checked()}</div>}
</Show>
</div>
<span
class="shrink-0 font-medium"
classList={{
'text-emerald-600 dark:text-emerald-300':
location.outcome === 'reachable' && !location.stale,
'text-red-600 dark:text-red-300':
location.outcome === 'unreachable' && !location.stale,
'text-amber-600 dark:text-amber-300': Boolean(location.stale),
'text-muted': !location.outcome || location.outcome === 'indeterminate',
}}
>
{locationResult(location)}
</span>
</div>
)}
</For>
</div>
</Show>
<InfoCardKeyValueRow label="Method" value={method()} valueTitle={targetAddr()} /> <InfoCardKeyValueRow label="Method" value={method()} valueTitle={targetAddr()} />
<InfoCardKeyValueRow <InfoCardKeyValueRow
label="Target" label="Target"
@@ -0,0 +1,63 @@
import { render, screen, within } from '@solidjs/testing-library';
import { describe, expect, it } from 'vitest';
import { AvailabilityProbeStatusCard } from '../AvailabilityProbeStatusCard';
describe('AvailabilityProbeStatusCard', () => {
it('shows disagreement and keeps every observation path visible', () => {
render(() => (
<AvailabilityProbeStatusCard
availability={{
targetId: 'customer-api',
address: 'api.service.local',
protocol: 'https',
enabled: true,
available: true,
aggregateState: 'degraded',
disagreement: true,
expectedLocations: 3,
reportingLocations: 2,
locations: [
{
locationId: 'pulse:local',
kind: 'pulse',
outcome: 'reachable',
available: true,
latencyMillis: 8,
lastChecked: '2026-08-30T19:00:00Z',
},
{
locationId: 'agent:edge-a',
kind: 'agent',
probeAgentId: 'edge-a',
outcome: 'unreachable',
available: false,
lastChecked: '2026-08-30T19:00:00Z',
},
{
locationId: 'agent:edge-b',
kind: 'agent',
probeAgentId: 'edge-b',
outcome: 'indeterminate',
available: false,
stale: true,
lastChecked: '2026-08-30T18:00:00Z',
},
],
}}
/>
));
expect(screen.getByText('Paths disagree')).toBeInTheDocument();
const paths = screen.getByText('Observation paths').parentElement?.parentElement;
expect(paths).not.toBeNull();
const scope = within(paths!);
expect(scope.getByText('2/3 reporting')).toBeInTheDocument();
expect(scope.getByText('This Pulse server')).toBeInTheDocument();
expect(scope.getByText('8 ms')).toBeInTheDocument();
expect(scope.getByText('edge-a')).toBeInTheDocument();
expect(scope.getByText('Unreachable')).toBeInTheDocument();
expect(scope.getByText('edge-b')).toBeInTheDocument();
expect(scope.getByText('No recent report')).toBeInTheDocument();
});
});
@@ -33,13 +33,14 @@ import { getPreferredInfrastructureDisplayName } from '@/utils/resourceIdentity'
import { getResourceTypeLabel } from '@/utils/resourceTypePresentation'; import { getResourceTypeLabel } from '@/utils/resourceTypePresentation';
import { import {
EXTERNAL_PROBE_FEATURE, EXTERNAL_PROBE_FEATURE,
LOCAL_PROBE_AGENT_LABEL, AGENT_OBSERVATION_LOCATION_PREFIX,
buildProbeAgentOptions, LOCAL_OBSERVATION_LOCATION_ID,
agentIdFromObservationLocation,
buildObservationLocationOptions,
getExternalProbeGateBody, getExternalProbeGateBody,
getExternalProbeGateTitle, getExternalProbeGateTitle,
getExternalProbeLockedHelpText, getExternalProbeLockedHelpText,
isExternalProbeLicenseError, isExternalProbeLicenseError,
isProbeAgentMissing,
} from '@/utils/availabilityProbeAgents'; } from '@/utils/availabilityProbeAgents';
import { hasFeature, loadRuntimeCapabilities, runtimeCapabilitiesLoaded } from '@/stores/license'; import { hasFeature, loadRuntimeCapabilities, runtimeCapabilitiesLoaded } from '@/stores/license';
import { getUpgradeActionDestination } from '@/stores/licenseCommercial'; import { getUpgradeActionDestination } from '@/stores/licenseCommercial';
@@ -58,7 +59,7 @@ interface AvailabilityForm {
udpRequest: string; udpRequest: string;
udpExpectedResponse: string; udpExpectedResponse: string;
linkedResourceId: string; linkedResourceId: string;
probeAgentId: string; observationLocationIds: string[];
enabled: boolean; enabled: boolean;
pollIntervalSeconds: string; pollIntervalSeconds: string;
timeoutMillis: string; timeoutMillis: string;
@@ -118,7 +119,7 @@ const newAvailabilityForm = (
udpRequest: '', udpRequest: '',
udpExpectedResponse: '', udpExpectedResponse: '',
linkedResourceId: '', linkedResourceId: '',
probeAgentId: '', observationLocationIds: [LOCAL_OBSERVATION_LOCATION_ID],
enabled: true, enabled: true,
pollIntervalSeconds: '60', pollIntervalSeconds: '60',
timeoutMillis: '2000', timeoutMillis: '2000',
@@ -159,7 +160,11 @@ const formFromTarget = (target: AvailabilityTarget): AvailabilityForm => {
udpRequest: target.udpRequest ?? '', udpRequest: target.udpRequest ?? '',
udpExpectedResponse: target.udpExpectedResponse ?? '', udpExpectedResponse: target.udpExpectedResponse ?? '',
linkedResourceId: target.linkedResourceId ?? '', linkedResourceId: target.linkedResourceId ?? '',
probeAgentId: target.probeAgentId ?? '', observationLocationIds: target.observationLocationIds?.length
? [...target.observationLocationIds]
: target.probeAgentId
? [`${AGENT_OBSERVATION_LOCATION_PREFIX}${target.probeAgentId}`]
: [LOCAL_OBSERVATION_LOCATION_ID],
enabled: target.enabled ?? true, enabled: target.enabled ?? true,
pollIntervalSeconds: String(target.pollIntervalSeconds ?? 60), pollIntervalSeconds: String(target.pollIntervalSeconds ?? 60),
timeoutMillis: String(target.timeoutMillis ?? 2000), timeoutMillis: String(target.timeoutMillis ?? 2000),
@@ -247,7 +252,11 @@ const payloadFromForm = (form: AvailabilityForm): AvailabilityTarget => {
// Always serialized, never `undefined`: the server decodes updates onto the // Always serialized, never `undefined`: the server decodes updates onto the
// existing record, so an explicit empty string is what clears a probe // existing record, so an explicit empty string is what clears a probe
// assignment and moves the check back to the local Pulse server. // assignment and moves the check back to the local Pulse server.
probeAgentId: form.probeAgentId.trim(), probeAgentId:
form.observationLocationIds.length === 1
? agentIdFromObservationLocation(form.observationLocationIds[0])
: '',
observationLocationIds: [...form.observationLocationIds],
enabled: form.enabled, enabled: form.enabled,
pollIntervalSeconds: parsePositiveInt(form.pollIntervalSeconds), pollIntervalSeconds: parsePositiveInt(form.pollIntervalSeconds),
timeoutMillis: parsePositiveInt(form.timeoutMillis), timeoutMillis: parsePositiveInt(form.timeoutMillis),
@@ -348,10 +357,11 @@ export const AvailabilityTargetSlot: Component<AvailabilityTargetSlotProps> = (p
// stale cached capability set still lands on the upgrade gate. // stale cached capability set still lands on the upgrade gate.
const [probeLicenseRejected, setProbeLicenseRejected] = createSignal(false); const [probeLicenseRejected, setProbeLicenseRejected] = createSignal(false);
const probeAgentOptions = createMemo(() => buildProbeAgentOptions(resources())); const observationLocationOptions = createMemo(() => buildObservationLocationOptions(resources()));
const probeAgentMissing = createMemo(() => const missingObservationLocations = createMemo(() => {
isProbeAgentMissing(probeAgentOptions(), form().probeAgentId), const known = new Set(observationLocationOptions().map((option) => option.id));
); return form().observationLocationIds.filter((locationId) => !known.has(locationId));
});
const externalProbeLicensed = createMemo( const externalProbeLicensed = createMemo(
() => () =>
!probeLicenseRejected() && runtimeCapabilitiesLoaded() && hasFeature(EXTERNAL_PROBE_FEATURE), !probeLicenseRejected() && runtimeCapabilitiesLoaded() && hasFeature(EXTERNAL_PROBE_FEATURE),
@@ -378,6 +388,15 @@ export const AvailabilityTargetSlot: Component<AvailabilityTargetSlotProps> = (p
setTestResult(null); setTestResult(null);
}; };
const toggleObservationLocation = (locationId: string, selected: boolean) => {
const current = form().observationLocationIds;
const next = selected
? [...new Set([...current, locationId])]
: current.filter((candidate) => candidate !== locationId);
if (next.length === 0) return;
updateForm({ observationLocationIds: next });
};
const updateHTTPHeader = (index: number, patch: Partial<AvailabilityHTTPHeaderForm>) => { const updateHTTPHeader = (index: number, patch: Partial<AvailabilityHTTPHeaderForm>) => {
updateForm({ updateForm({
httpHeaders: form().httpHeaders.map((header, headerIndex) => httpHeaders: form().httpHeaders.map((header, headerIndex) =>
@@ -490,7 +509,12 @@ export const AvailabilityTargetSlot: Component<AvailabilityTargetSlotProps> = (p
} }
props.onSaved(); props.onSaved();
} catch (err) { } catch (err) {
if (payload.probeAgentId && isExternalProbeLicenseError(err)) { if (
payload.observationLocationIds?.some((locationId) =>
locationId.startsWith(AGENT_OBSERVATION_LOCATION_PREFIX),
) &&
isExternalProbeLicenseError(err)
) {
setProbeLicenseRejected(true); setProbeLicenseRejected(true);
setError(getExternalProbeGateBody()); setError(getExternalProbeGateBody());
return; return;
@@ -610,28 +634,67 @@ export const AvailabilityTargetSlot: Component<AvailabilityTargetSlotProps> = (p
)} )}
</For> </For>
</FormSelect> </FormSelect>
<div class="space-y-3 sm:col-span-2"> <fieldset class="space-y-3 sm:col-span-2">
<FormSelect <legend class={formLabel}>Observation locations</legend>
label="Run from" <p class={formHelpText}>
value={form().probeAgentId} One logical check can be observed from several networks. Pulse keeps each path separate
disabled={externalProbeLocked()} and shows disagreement instead of turning one path failure into a universal outage.
onChange={(event) => updateForm({ probeAgentId: event.currentTarget.value })} </p>
help={ <div class="grid gap-2 sm:grid-cols-2" data-observation-location-picker>
externalProbeLocked() <For each={observationLocationOptions()}>
? getExternalProbeLockedHelpText() {(option) => {
: 'Run this check from the Pulse server, or hand it to a connected Pulse Agent host so it is probed from that network.' const selected = () => form().observationLocationIds.includes(option.id);
} const locked = () => option.kind === 'agent' && externalProbeLocked();
> return (
<option value="">{LOCAL_PROBE_AGENT_LABEL}</option> <label class="flex items-start gap-3 rounded-md border border-border bg-surface px-3 py-3 text-sm">
<Show when={probeAgentMissing()}> <input
<option value={form().probeAgentId}> class={formCheckbox}
{form().probeAgentId} (not currently connected) type="checkbox"
</option> checked={selected()}
</Show> disabled={
<For each={probeAgentOptions()}> locked() || (selected() && form().observationLocationIds.length === 1)
{(option) => <option value={option.id}>{option.label}</option>} }
onChange={(event) =>
toggleObservationLocation(option.id, event.currentTarget.checked)
}
/>
<span class="min-w-0">
<span class="block font-medium text-foreground">{option.label}</span>
<span class="block text-xs text-muted">
{option.kind === 'pulse'
? 'Local Pulse path'
: 'Connected Pulse Agent path'}
</span>
</span>
</label>
);
}}
</For> </For>
</FormSelect> <For each={missingObservationLocations()}>
{(locationId) => (
<label class="flex items-start gap-3 rounded-md border border-warning/40 bg-warning/5 px-3 py-3 text-sm">
<input
class={formCheckbox}
type="checkbox"
checked
disabled={form().observationLocationIds.length === 1}
onChange={(event) =>
toggleObservationLocation(locationId, event.currentTarget.checked)
}
/>
<span class="min-w-0">
<span class="block font-medium text-foreground">
{agentIdFromObservationLocation(locationId) || locationId}
</span>
<span class="block text-xs text-warning">Not currently connected</span>
</span>
</label>
)}
</For>
</div>
<Show when={externalProbeLocked()}>
<p class={formHelpText}>{getExternalProbeLockedHelpText()}</p>
</Show>
<Show when={externalProbeLocked()}> <Show when={externalProbeLocked()}>
<div class="rounded-md border border-border bg-surface-alt p-4"> <div class="rounded-md border border-border bg-surface-alt p-4">
<FeatureGateSection <FeatureGateSection
@@ -642,7 +705,7 @@ export const AvailabilityTargetSlot: Component<AvailabilityTargetSlotProps> = (p
/> />
</div> </div>
</Show> </Show>
</div> </fieldset>
<Show when={form().protocol !== 'icmp'}> <Show when={form().protocol !== 'icmp'}>
<label class={formField}> <label class={formField}>
<span class={formLabel}>Port</span> <span class={formLabel}>Port</span>
@@ -424,22 +424,25 @@ describe('AvailabilityTargetSlot', () => {
resourceMocks.resources = [agentHostResource('host-edge-01', 'Edge 01')]; resourceMocks.resources = [agentHostResource('host-edge-01', 'Edge 01')];
render(() => <AvailabilityTargetSlot onCancel={vi.fn()} onSaved={vi.fn()} />); render(() => <AvailabilityTargetSlot onCancel={vi.fn()} onSaved={vi.fn()} />);
const runFrom = screen.getByLabelText('Run from') as HTMLSelectElement; const localLocation = screen.getByRole('checkbox', { name: /This Pulse server/ });
expect(runFrom).not.toBeDisabled(); const edgeLocation = screen.getByRole('checkbox', { name: /Edge 01/ });
expect(screen.getByRole('option', { name: 'This Pulse server' })).toBeInTheDocument(); expect(localLocation).toBeChecked();
expect(screen.getByRole('option', { name: 'Edge 01' })).toBeInTheDocument(); expect(edgeLocation).not.toBeDisabled();
expect(screen.queryByRole('link', { name: 'View plans' })).not.toBeInTheDocument(); expect(screen.queryByRole('link', { name: 'View plans' })).not.toBeInTheDocument();
fireEvent.input(screen.getByLabelText('Name'), { target: { value: 'Remote MQTT' } }); fireEvent.input(screen.getByLabelText('Name'), { target: { value: 'Remote MQTT' } });
fireEvent.input(screen.getByPlaceholderText('service.local'), { fireEvent.input(screen.getByPlaceholderText('service.local'), {
target: { value: 'mqtt.remote.local' }, target: { value: 'mqtt.remote.local' },
}); });
fireEvent.change(runFrom, { target: { value: 'host-edge-01' } }); fireEvent.click(edgeLocation);
fireEvent.click(screen.getByRole('button', { name: 'Add service/device check' })); fireEvent.click(screen.getByRole('button', { name: 'Add service/device check' }));
await waitFor(() => await waitFor(() =>
expect(mockedCreate).toHaveBeenCalledWith( expect(mockedCreate).toHaveBeenCalledWith(
expect.objectContaining({ probeAgentId: 'host-edge-01' }), expect.objectContaining({
probeAgentId: '',
observationLocationIds: ['pulse:local', 'agent:host-edge-01'],
}),
), ),
); );
}); });
@@ -450,9 +453,8 @@ describe('AvailabilityTargetSlot', () => {
render(() => <AvailabilityTargetSlot onCancel={vi.fn()} onSaved={vi.fn()} />); render(() => <AvailabilityTargetSlot onCancel={vi.fn()} onSaved={vi.fn()} />);
// Discoverability is the point of the Pro gate: the control stays visible. // Discoverability is the point of the Pro gate: the control stays visible.
const runFrom = screen.getByLabelText('Run from') as HTMLSelectElement; const edgeLocation = screen.getByRole('checkbox', { name: /Edge 01/ });
expect(runFrom).toBeDisabled(); expect(edgeLocation).toBeDisabled();
expect(screen.getByRole('option', { name: 'Edge 01' })).toBeInTheDocument();
expect(screen.getByRole('heading', { name: 'External Probes' })).toBeInTheDocument(); expect(screen.getByRole('heading', { name: 'External Probes' })).toBeInTheDocument();
expect(screen.getByRole('link', { name: 'View plans' })).toHaveAttribute( expect(screen.getByRole('link', { name: 'View plans' })).toHaveAttribute(
'href', 'href',
@@ -471,7 +473,12 @@ describe('AvailabilityTargetSlot', () => {
fireEvent.click(screen.getByRole('button', { name: 'Add service/device check' })); fireEvent.click(screen.getByRole('button', { name: 'Add service/device check' }));
await waitFor(() => await waitFor(() =>
expect(mockedCreate).toHaveBeenCalledWith(expect.objectContaining({ probeAgentId: '' })), expect(mockedCreate).toHaveBeenCalledWith(
expect.objectContaining({
probeAgentId: '',
observationLocationIds: ['pulse:local'],
}),
),
); );
expect(screen.queryByRole('alert')).not.toBeInTheDocument(); expect(screen.queryByRole('alert')).not.toBeInTheDocument();
}); });
@@ -501,14 +508,16 @@ describe('AvailabilityTargetSlot', () => {
<AvailabilityTargetSlot editingTargetId="target-1" onCancel={vi.fn()} onSaved={vi.fn()} /> <AvailabilityTargetSlot editingTargetId="target-1" onCancel={vi.fn()} onSaved={vi.fn()} />
)); ));
await waitFor(() => expect(screen.getByLabelText('Run from')).toHaveValue('host-edge-01')); await waitFor(() => expect(screen.getByRole('checkbox', { name: /Edge 01/ })).toBeChecked());
fireEvent.change(screen.getByLabelText('Run from'), { target: { value: '' } }); fireEvent.click(screen.getByRole('checkbox', { name: /This Pulse server/ }));
fireEvent.click(screen.getByRole('checkbox', { name: /Edge 01/ }));
fireEvent.click(screen.getByRole('button', { name: 'Save target' })); fireEvent.click(screen.getByRole('button', { name: 'Save target' }));
await waitFor(() => expect(mockedUpdate).toHaveBeenCalled()); await waitFor(() => expect(mockedUpdate).toHaveBeenCalled());
const [, payload] = mockedUpdate.mock.calls.at(-1)!; const [, payload] = mockedUpdate.mock.calls.at(-1)!;
expect(payload.probeAgentId).toBe(''); expect(payload.probeAgentId).toBe('');
expect(payload.observationLocationIds).toEqual(['pulse:local']);
expect(Object.prototype.hasOwnProperty.call(payload, 'probeAgentId')).toBe(true); expect(Object.prototype.hasOwnProperty.call(payload, 'probeAgentId')).toBe(true);
}); });
@@ -529,11 +538,9 @@ describe('AvailabilityTargetSlot', () => {
)); ));
await waitFor(() => await waitFor(() =>
expect( expect(screen.getByRole('checkbox', { name: /host-gone/ })).toBeChecked(),
screen.getByRole('option', { name: 'host-gone (not currently connected)' }),
).toBeInTheDocument(),
); );
expect(screen.getByLabelText('Run from')).toHaveValue('host-gone'); expect(screen.getByText('Not currently connected')).toBeInTheDocument();
}); });
it('falls back to the upgrade gate when the server answers 402 license_required', async () => { it('falls back to the upgrade gate when the server answers 402 license_required', async () => {
@@ -551,13 +558,13 @@ describe('AvailabilityTargetSlot', () => {
fireEvent.input(screen.getByPlaceholderText('service.local'), { fireEvent.input(screen.getByPlaceholderText('service.local'), {
target: { value: 'mqtt.remote.local' }, target: { value: 'mqtt.remote.local' },
}); });
fireEvent.change(screen.getByLabelText('Run from'), { target: { value: 'host-edge-01' } }); fireEvent.click(screen.getByRole('checkbox', { name: /Edge 01/ }));
fireEvent.click(screen.getByRole('button', { name: 'Add service/device check' })); fireEvent.click(screen.getByRole('button', { name: 'Add service/device check' }));
await waitFor(() => await waitFor(() =>
expect(screen.getByRole('heading', { name: 'External Probes' })).toBeInTheDocument(), expect(screen.getByRole('heading', { name: 'External Probes' })).toBeInTheDocument(),
); );
expect(screen.getByLabelText('Run from')).toBeDisabled(); expect(screen.getByRole('checkbox', { name: /Edge 01/ })).toBeDisabled();
expect(screen.getByRole('link', { name: 'View plans' })).toBeInTheDocument(); expect(screen.getByRole('link', { name: 'View plans' })).toBeInTheDocument();
}); });
}); });
@@ -108,6 +108,14 @@ export function getAvailabilityTargetStatusLabel(target: AvailabilityTarget): st
if (!target.enabled) return 'Paused'; if (!target.enabled) return 'Paused';
const status = target.status; const status = target.status;
if (!status) return 'Not checked yet'; if (!status) return 'Not checked yet';
if (status.aggregateState === 'degraded') return 'Observation paths disagree';
if (status.aggregateState === 'unknown') {
return `${status.reportingLocations ?? 0}/${status.expectedLocations ?? status.locations?.length ?? 0} locations reporting`;
}
if (status.aggregateState === 'unavailable') return 'Unavailable from all locations';
if (status.aggregateState === 'healthy' && (status.expectedLocations ?? 0) > 1) {
return `Available from all ${status.expectedLocations} locations`;
}
// A probe-assigned check whose agent stopped reporting derives to // A probe-assigned check whose agent stopped reporting derives to
// indeterminate at read time. It shares the warning treatment with the UDP // indeterminate at read time. It shares the warning treatment with the UDP
// open-or-filtered case but needs its own copy. // open-or-filtered case but needs its own copy.
@@ -124,6 +132,12 @@ export function getAvailabilityTargetStatusLabel(target: AvailabilityTarget): st
export function getAvailabilityTargetStatusClass(target: AvailabilityTarget): string { export function getAvailabilityTargetStatusClass(target: AvailabilityTarget): string {
if (!target.enabled) return 'bg-surface-alt text-muted'; if (!target.enabled) return 'bg-surface-alt text-muted';
if (!target.status) return 'bg-sky-100 text-sky-700 dark:bg-sky-900 dark:text-sky-300'; if (!target.status) return 'bg-sky-100 text-sky-700 dark:bg-sky-900 dark:text-sky-300';
if (target.status.aggregateState === 'degraded' || target.status.aggregateState === 'unknown') {
return 'bg-amber-100 text-amber-700 dark:bg-amber-900 dark:text-amber-300';
}
if (target.status.aggregateState === 'unavailable') {
return 'bg-rose-100 text-rose-700 dark:bg-rose-900 dark:text-rose-300';
}
if (target.status.outcome === 'indeterminate') { if (target.status.outcome === 'indeterminate') {
return 'bg-amber-100 text-amber-700 dark:bg-amber-900 dark:text-amber-300'; return 'bg-amber-100 text-amber-700 dark:bg-amber-900 dark:text-amber-300';
} }
@@ -141,6 +155,9 @@ export function getAvailabilityTargetProbeSourceLabel(
target: AvailabilityTarget, target: AvailabilityTarget,
probeAgentOptions: readonly ProbeAgentOption[], probeAgentOptions: readonly ProbeAgentOption[],
): string | null { ): string | null {
const locationCount =
target.status?.locations?.length ?? target.observationLocationIds?.length ?? 0;
if (locationCount > 1) return `${locationCount} observation locations`;
return getProbeSourceChipLabel(probeAgentOptions, target.status?.probeAgentId); return getProbeSourceChipLabel(probeAgentOptions, target.status?.probeAgentId);
} }
@@ -283,10 +283,12 @@ export const AvailabilityChecksTable: Component<{
const result = () => probe()?.resultLabel ?? indicator().label; const result = () => probe()?.resultLabel ?? indicator().label;
const target = () => formatTarget(check); const target = () => formatTarget(check);
const probeSource = () => const probeSource = () =>
getProbeSourceChipLabel( (availability()?.locations?.length ?? 0) > 1
props.probeAgentOptions ?? [], ? `${availability()?.locations?.length} locations · ${availability()?.reportingLocations ?? 0}/${availability()?.expectedLocations ?? availability()?.locations?.length ?? 0} reporting`
availability()?.probeAgentId, : getProbeSourceChipLabel(
); props.probeAgentOptions ?? [],
availability()?.probeAgentId,
);
const detailRowId = () => drawer.detailRowId(check); const detailRowId = () => drawer.detailRowId(check);
const isExpanded = () => drawer.isExpanded(check); const isExpanded = () => drawer.isExpanded(check);
+24
View File
@@ -1532,6 +1532,11 @@ export interface ResourceAvailabilityMeta {
* on the Pulse server itself. * on the Pulse server itself.
*/ */
probeAgentId?: string; probeAgentId?: string;
aggregateState?: 'healthy' | 'degraded' | 'unavailable' | 'unknown' | string;
disagreement?: boolean;
expectedLocations?: number;
reportingLocations?: number;
locations?: ResourceAvailabilityObservationLocation[];
correlationState?: 'attached' | 'standalone' | 'ambiguous' | 'unresolved'; correlationState?: 'attached' | 'standalone' | 'ambiguous' | 'unresolved';
correlationRule?: string; correlationRule?: string;
correlationReason?: string; correlationReason?: string;
@@ -1539,6 +1544,25 @@ export interface ResourceAvailabilityMeta {
evidence?: EvidenceEnvelope; evidence?: EvidenceEnvelope;
} }
export interface ResourceAvailabilityObservationLocation {
locationId: string;
kind: 'pulse' | 'agent' | string;
probeAgentId?: string;
outcome?: string;
transportOutcome?: string;
applicationOutcome?: string;
applicationStatusCode?: number;
applicationFailureCode?: string;
available?: boolean;
lastChecked?: string;
lastSuccess?: string;
freshnessAt?: string;
latencyMillis?: number;
consecutiveFailures?: number;
lastError?: string;
stale?: boolean;
}
export type ResourceCertificateTrustStatus = export type ResourceCertificateTrustStatus =
'trusted' | 'self-signed' | 'untrusted' | 'expired' | 'not-yet-valid'; 'trusted' | 'self-signed' | 'untrusted' | 'expired' | 'not-yet-valid';
@@ -23,6 +23,8 @@ export const EXTERNAL_PROBE_FEATURE = 'external_probe';
/** Wire value for "run this check on the Pulse server itself". */ /** Wire value for "run this check on the Pulse server itself". */
export const LOCAL_PROBE_AGENT_VALUE = ''; export const LOCAL_PROBE_AGENT_VALUE = '';
export const LOCAL_OBSERVATION_LOCATION_ID = 'pulse:local';
export const AGENT_OBSERVATION_LOCATION_PREFIX = 'agent:';
/** Default option label for the local Pulse server. */ /** Default option label for the local Pulse server. */
export const LOCAL_PROBE_AGENT_LABEL = 'This Pulse server'; export const LOCAL_PROBE_AGENT_LABEL = 'This Pulse server';
@@ -40,6 +42,23 @@ export interface ProbeAgentOption {
label: string; label: string;
} }
export interface ObservationLocationOption {
id: string;
label: string;
kind: 'pulse' | 'agent';
agentId?: string;
}
export const observationLocationIdForAgent = (agentId: string): string =>
`${AGENT_OBSERVATION_LOCATION_PREFIX}${agentId.trim()}`;
export const agentIdFromObservationLocation = (locationId?: string | null): string => {
const normalized = (locationId ?? '').trim();
return normalized.startsWith(AGENT_OBSERVATION_LOCATION_PREFIX)
? normalized.slice(AGENT_OBSERVATION_LOCATION_PREFIX.length).trim()
: '';
};
/** /**
* Build the selectable probe agent hosts from unified resources. * Build the selectable probe agent hosts from unified resources.
* *
@@ -62,6 +81,32 @@ export function buildProbeAgentOptions(resources: readonly Resource[]): ProbeAge
return [...byId.values()].sort((left, right) => left.label.localeCompare(right.label)); return [...byId.values()].sort((left, right) => left.label.localeCompare(right.label));
} }
export function buildObservationLocationOptions(
resources: readonly Resource[],
): ObservationLocationOption[] {
return [
{ id: LOCAL_OBSERVATION_LOCATION_ID, label: LOCAL_PROBE_AGENT_LABEL, kind: 'pulse' },
...buildProbeAgentOptions(resources).map((option) => ({
id: observationLocationIdForAgent(option.id),
label: option.label,
kind: 'agent' as const,
agentId: option.id,
})),
];
}
export function getObservationLocationLabel(
options: readonly ObservationLocationOption[],
locationId?: string | null,
): string {
const normalized = (locationId ?? '').trim();
if (!normalized || normalized === LOCAL_OBSERVATION_LOCATION_ID) return LOCAL_PROBE_AGENT_LABEL;
return (
options.find((option) => option.id === normalized)?.label ??
(agentIdFromObservationLocation(normalized) || normalized)
);
}
/** /**
* Resolve a probe agent id to its display name, falling back to the raw id when * Resolve a probe agent id to its display name, falling back to the raw id when
* the host is not currently in the list. * the host is not currently in the list.
+32 -6
View File
@@ -3,6 +3,7 @@ package api
import ( import (
"context" "context"
"encoding/json" "encoding/json"
"io"
"net/http" "net/http"
"strings" "strings"
"time" "time"
@@ -78,7 +79,8 @@ func (f availabilityFeatureResolverFunc) FeatureService(ctx context.Context) lic
// Unassigned targets are the community behaviour and must never consult the // Unassigned targets are the community behaviour and must never consult the
// license path. // license path.
func (h *AvailabilityHandlers) requireProbeAssignment(w http.ResponseWriter, r *http.Request, target config.AvailabilityTarget) bool { func (h *AvailabilityHandlers) requireProbeAssignment(w http.ResponseWriter, r *http.Request, target config.AvailabilityTarget) bool {
if strings.TrimSpace(target.ProbeAgentID) == "" { agentIDs := target.AssignedProbeAgentIDs()
if len(agentIDs) == 0 {
return true return true
} }
if h == nil || h.licenseResolver == nil { if h == nil || h.licenseResolver == nil {
@@ -94,10 +96,12 @@ func (h *AvailabilityHandlers) requireProbeAssignment(w http.ResponseWriter, r *
WriteLicenseRequired(w, featureExternalProbeValue, err.Error()) WriteLicenseRequired(w, featureExternalProbeValue, err.Error())
return false return false
} }
if !h.probeAgentExists(r.Context(), target.ProbeAgentID) { for _, agentID := range agentIDs {
writeErrorResponse(w, http.StatusBadRequest, "unknown_probe_agent", if !h.probeAgentExists(r.Context(), agentID) {
"Probe agent "+target.ProbeAgentID+" is not a registered host agent", nil) writeErrorResponse(w, http.StatusBadRequest, "unknown_probe_agent",
return false "Observation location agent "+agentID+" is not a registered host agent", nil)
return false
}
} }
return true return true
} }
@@ -384,10 +388,31 @@ func decodeAvailabilityTargetRequest(w http.ResponseWriter, r *http.Request, bas
r.Body = http.MaxBytesReader(w, r.Body, 16*1024) r.Body = http.MaxBytesReader(w, r.Body, 16*1024)
defer r.Body.Close() defer r.Body.Close()
target := cloneAvailabilityTarget(base) target := cloneAvailabilityTarget(base)
if err := json.NewDecoder(r.Body).Decode(&target); err != nil { body, err := io.ReadAll(r.Body)
if err != nil || json.Unmarshal(body, &target) != nil {
writeErrorResponse(w, http.StatusBadRequest, "invalid_request", "Invalid JSON body", nil) writeErrorResponse(w, http.StatusBadRequest, "invalid_request", "Invalid JSON body", nil)
return config.AvailabilityTarget{}, false return config.AvailabilityTarget{}, false
} }
var compatibility struct {
ProbeAgentID *string `json:"probeAgentId"`
ObservationLocationIDs *[]string `json:"observationLocationIds"`
}
if json.Unmarshal(body, &compatibility) == nil && compatibility.ObservationLocationIDs == nil && compatibility.ProbeAgentID != nil {
agentID := strings.TrimSpace(*compatibility.ProbeAgentID)
if agentID == "" {
target.ObservationLocationIDs = []string{config.AvailabilityObservationLocationLocal}
} else {
target.ObservationLocationIDs = []string{config.AvailabilityAgentObservationLocationID(agentID)}
}
} else if compatibility.ProbeAgentID != nil && compatibility.ObservationLocationIDs != nil {
agentID := strings.TrimSpace(*compatibility.ProbeAgentID)
if agentID != "" && len(*compatibility.ObservationLocationIDs) == 1 {
// Full-object pre-location clients may echo the canonical field added by
// the server while editing only probeAgentId. Honor that single-source
// edit; multi-location clients send a set the legacy field cannot express.
target.ObservationLocationIDs = []string{config.AvailabilityAgentObservationLocationID(agentID)}
}
}
return target, true return target, true
} }
@@ -487,6 +512,7 @@ func sameAvailabilityHTTPOrigin(previous, next config.AvailabilityTarget) bool {
func cloneAvailabilityTarget(target config.AvailabilityTarget) config.AvailabilityTarget { func cloneAvailabilityTarget(target config.AvailabilityTarget) config.AvailabilityTarget {
clone := target clone := target
clone.ObservationLocationIDs = append([]string(nil), target.ObservationLocationIDs...)
if target.HTTP == nil { if target.HTTP == nil {
return clone return clone
} }
@@ -154,6 +154,42 @@ func TestAvailabilityHandlersAcceptProbeAssignmentWithLicense(t *testing.T) {
} }
} }
func TestAvailabilityHandlersAcceptSeveralObservationLocationsOnOneTarget(t *testing.T) {
persistence := config.NewConfigPersistence(t.TempDir())
monitor := monitorWithHostAgent(t, "edge-a")
if _, err := monitor.ApplyHostReport(agentshost.Report{
Agent: agentshost.AgentInfo{ID: "edge-b-agent", Version: "6.1.1", IntervalSeconds: 30},
Host: agentshost.HostInfo{ID: "edge-b", MachineID: "edge-b", Hostname: "edge-b", Platform: "linux"},
Timestamp: time.Now().UTC(),
}, &config.APITokenRecord{ID: "token-edge-b", Name: "edge-b"}); err != nil {
t.Fatalf("ApplyHostReport() error = %v", err)
}
handler := NewAvailabilityHandlers(
func(context.Context) *config.ConfigPersistence { return persistence },
func(context.Context) *monitoring.Monitor { return monitor },
stubFeatureResolver(featureExternalProbeValue),
)
target := config.AvailabilityTarget{
Name: "Customer API",
Address: "api.service.local",
Protocol: config.AvailabilityProbeHTTPS,
Enabled: true,
ObservationLocationIDs: []string{config.AvailabilityObservationLocationLocal, "agent:edge-a", "agent:edge-b"},
}
rec := httptest.NewRecorder()
handler.HandleAdd(rec, httptest.NewRequest(http.MethodPost, "/api/availability-targets", availabilityRequestBody(t, target)))
if rec.Code != http.StatusCreated {
t.Fatalf("HandleAdd status = %d, want 201; body=%s", rec.Code, rec.Body.String())
}
var created config.AvailabilityTarget
if err := json.NewDecoder(rec.Body).Decode(&created); err != nil {
t.Fatalf("decode created target: %v", err)
}
if len(created.ObservationLocationIDs) != 3 || created.ProbeAgentID != "" {
t.Fatalf("created target = %+v, want one logical target with three locations", created)
}
}
func TestAvailabilityHandlersRejectUnknownProbeAgent(t *testing.T) { func TestAvailabilityHandlersRejectUnknownProbeAgent(t *testing.T) {
persistence := config.NewConfigPersistence(t.TempDir()) persistence := config.NewConfigPersistence(t.TempDir())
monitor := monitorWithHostAgent(t, "probe-host") monitor := monitorWithHostAgent(t, "probe-host")
+6
View File
@@ -1497,6 +1497,12 @@ func deriveAvailabilityConnectionState(target config.AvailabilityTarget, status
if !target.Enabled { if !target.Enabled {
return ConnectionStatePaused, "paused by user", lastSeen, lastError return ConnectionStatePaused, "paused by user", lastSeen, lastError
} }
if status.AggregateState == monitoring.AvailabilityAggregateUnknown {
return ConnectionStateStale, "observation coverage is incomplete", lastSeen, lastError
}
if status.AggregateState == monitoring.AvailabilityAggregateDegraded {
return ConnectionStateActive, "observation locations disagree", lastSeen, lastError
}
if status.LastChecked.IsZero() { if status.LastChecked.IsZero() {
return ConnectionStatePending, "awaiting first probe", nil, nil return ConnectionStatePending, "awaiting first probe", nil, nil
} }
+40 -3
View File
@@ -274,7 +274,7 @@ func mockAvailabilityTestResponse(targetID string) (availabilityTestResponse, bo
} }
func mockAvailabilityConfigTarget(target mock.AvailabilityTargetFixture) config.AvailabilityTarget { func mockAvailabilityConfigTarget(target mock.AvailabilityTargetFixture) config.AvailabilityTarget {
return config.NormalizeAvailabilityTarget(config.AvailabilityTarget{ configTarget := config.AvailabilityTarget{
ID: target.ID, ID: target.ID,
Name: target.Name, Name: target.Name,
TargetKind: config.AvailabilityTargetKind(target.TargetKind), TargetKind: config.AvailabilityTargetKind(target.TargetKind),
@@ -286,11 +286,18 @@ func mockAvailabilityConfigTarget(target mock.AvailabilityTargetFixture) config.
PollIntervalSecs: target.PollIntervalSecs, PollIntervalSecs: target.PollIntervalSecs,
TimeoutMillis: target.TimeoutMillis, TimeoutMillis: target.TimeoutMillis,
FailureThreshold: target.FailureThreshold, FailureThreshold: target.FailureThreshold,
}) }
if target.ID == "mock-availability-docker-frontend-service" {
configTarget.ObservationLocationIDs = []string{
config.AvailabilityObservationLocationLocal,
config.AvailabilityAgentObservationLocationID("edge-london"),
}
}
return config.NormalizeAvailabilityTarget(configTarget)
} }
func mockAvailabilityProbeStatus(fixture mock.AvailabilityFixture, target config.AvailabilityTarget) monitoring.AvailabilityProbeStatus { func mockAvailabilityProbeStatus(fixture mock.AvailabilityFixture, target config.AvailabilityTarget) monitoring.AvailabilityProbeStatus {
return monitoring.AvailabilityProbeStatus{ status := monitoring.AvailabilityProbeStatus{
TargetID: target.ID, TargetID: target.ID,
Name: target.DisplayName(), Name: target.DisplayName(),
TargetKind: string(target.TargetKind), TargetKind: string(target.TargetKind),
@@ -305,6 +312,36 @@ func mockAvailabilityProbeStatus(fixture mock.AvailabilityFixture, target config
LastError: fixture.LastError, LastError: fixture.LastError,
FailureThreshold: target.EffectiveFailureThreshold(), FailureThreshold: target.EffectiveFailureThreshold(),
} }
if target.ID == "mock-availability-docker-frontend-service" {
status.AggregateState = monitoring.AvailabilityAggregateDegraded
status.Disagreement = true
status.ExpectedLocations = 2
status.ReportingLocations = 2
status.Locations = []monitoring.AvailabilityObservationLocationStatus{
{
LocationID: config.AvailabilityObservationLocationLocal,
Kind: "pulse",
Outcome: string(monitoring.AvailabilityProbeReachable),
Available: true,
LastChecked: fixture.LastChecked,
LastSuccess: fixture.LastSuccess,
FreshnessAt: fixture.LastChecked,
LatencyMillis: fixture.LatencyMillis,
},
{
LocationID: config.AvailabilityAgentObservationLocationID("edge-london"),
Kind: "agent",
ProbeAgentID: "edge-london",
Outcome: string(monitoring.AvailabilityProbeUnreachable),
Available: false,
LastChecked: fixture.LastChecked,
FreshnessAt: fixture.LastChecked,
ConsecutiveFailures: 2,
LastError: "connection timed out",
},
}
}
return status
} }
func mockPlatformTimePointer(value time.Time) *time.Time { func mockPlatformTimePointer(value time.Time) *time.Time {
+146 -9
View File
@@ -5,6 +5,7 @@ import (
"net" "net"
"net/url" "net/url"
"reflect" "reflect"
"sort"
"strconv" "strconv"
"strings" "strings"
@@ -12,12 +13,14 @@ import (
) )
const ( const (
DefaultAvailabilityPollIntervalSecs = 60 DefaultAvailabilityPollIntervalSecs = 60
DefaultAvailabilityTimeoutMillis = 2000 DefaultAvailabilityTimeoutMillis = 2000
DefaultAvailabilityFailureThreshold = 2 DefaultAvailabilityFailureThreshold = 2
DefaultCertificateExpiryWarningDays = 30 DefaultCertificateExpiryWarningDays = 30
MaxAvailabilityHTTPBodyBytes = 8192 MaxAvailabilityHTTPBodyBytes = 8192
MaxAvailabilityHTTPResponseBytes = 65536 MaxAvailabilityHTTPResponseBytes = 65536
AvailabilityObservationLocationLocal = "pulse:local"
AvailabilityObservationLocationAgentPrefix = "agent:"
) )
type AvailabilityHTTPMethod string type AvailabilityHTTPMethod string
@@ -125,9 +128,14 @@ type AvailabilityTarget struct {
CertificateMonitoringDisabled bool `json:"certificateMonitoringDisabled,omitempty"` CertificateMonitoringDisabled bool `json:"certificateMonitoringDisabled,omitempty"`
CertificateExpiryWarningDays int `json:"certificateExpiryWarningDays,omitempty"` CertificateExpiryWarningDays int `json:"certificateExpiryWarningDays,omitempty"`
// ProbeAgentID assigns execution to a remote host agent. Empty means the // ProbeAgentID assigns execution to a remote host agent. Empty means the
// check runs from the local Pulse instance. Agent existence is validated at // check runs from the local Pulse instance. It remains a compatibility field
// the API layer, not here, because config has no view of monitor state. // for pre-location clients; ObservationLocationIDs is canonical.
ProbeAgentID string `json:"probeAgentId,omitempty"` ProbeAgentID string `json:"probeAgentId,omitempty"`
// ObservationLocationIDs selects every source-owned path that observes this
// one logical verification. pulse:local is the Pulse server and agent:<id>
// names an eligible connected host agent. Names and network details stay out
// of persisted target configuration and are resolved at presentation time.
ObservationLocationIDs []string `json:"observationLocationIds,omitempty"`
// HTTP is absent for legacy targets. Its request body, credentials, and // HTTP is absent for legacy targets. Its request body, credentials, and
// header values are encrypted with the rest of availability target storage // header values are encrypted with the rest of availability target storage
// and are never returned by the API. // and are never returned by the API.
@@ -227,6 +235,7 @@ func (t AvailabilityTarget) EffectiveFailureThreshold() int {
// executed or where it executes. Display, correlation, alert-threshold, and // executed or where it executes. Display, correlation, alert-threshold, and
// certificate-presentation edits intentionally stay within the same revision. // certificate-presentation edits intentionally stay within the same revision.
func AvailabilityExecutionConfigChanged(previous, next AvailabilityTarget) bool { func AvailabilityExecutionConfigChanged(previous, next AvailabilityTarget) bool {
legacyProbeAssignmentChanged := strings.TrimSpace(previous.ProbeAgentID) != strings.TrimSpace(next.ProbeAgentID)
previous = NormalizeAvailabilityTarget(previous) previous = NormalizeAvailabilityTarget(previous)
next = NormalizeAvailabilityTarget(next) next = NormalizeAvailabilityTarget(next)
return previous.Address != next.Address || return previous.Address != next.Address ||
@@ -239,7 +248,67 @@ func AvailabilityExecutionConfigChanged(previous, next AvailabilityTarget) bool
!reflect.DeepEqual(previous.HTTP, next.HTTP) || !reflect.DeepEqual(previous.HTTP, next.HTTP) ||
previous.EffectiveTimeoutMillis() != next.EffectiveTimeoutMillis() || previous.EffectiveTimeoutMillis() != next.EffectiveTimeoutMillis() ||
previous.EffectivePollIntervalSecs() != next.EffectivePollIntervalSecs() || previous.EffectivePollIntervalSecs() != next.EffectivePollIntervalSecs() ||
previous.ProbeAgentID != next.ProbeAgentID legacyProbeAssignmentChanged ||
!reflect.DeepEqual(previous.EffectiveObservationLocationIDs(), next.EffectiveObservationLocationIDs())
}
func AvailabilityAgentObservationLocationID(agentID string) string {
agentID = strings.TrimSpace(agentID)
if agentID == "" {
return ""
}
return AvailabilityObservationLocationAgentPrefix + agentID
}
func AvailabilityObservationLocationAgentID(locationID string) string {
locationID = strings.TrimSpace(locationID)
if !strings.HasPrefix(locationID, AvailabilityObservationLocationAgentPrefix) {
return ""
}
return strings.TrimSpace(strings.TrimPrefix(locationID, AvailabilityObservationLocationAgentPrefix))
}
func (t AvailabilityTarget) EffectiveObservationLocationIDs() []string {
locations := normalizeAvailabilityObservationLocationIDs(t.ObservationLocationIDs)
if len(locations) > 0 {
return locations
}
if agentID := strings.TrimSpace(t.ProbeAgentID); agentID != "" {
return []string{AvailabilityAgentObservationLocationID(agentID)}
}
return []string{AvailabilityObservationLocationLocal}
}
func (t AvailabilityTarget) IncludesLocalObservation() bool {
for _, locationID := range t.EffectiveObservationLocationIDs() {
if locationID == AvailabilityObservationLocationLocal {
return true
}
}
return false
}
func (t AvailabilityTarget) AssignedProbeAgentIDs() []string {
agents := make([]string, 0, len(t.ObservationLocationIDs))
for _, locationID := range t.EffectiveObservationLocationIDs() {
if agentID := AvailabilityObservationLocationAgentID(locationID); agentID != "" {
agents = append(agents, agentID)
}
}
return agents
}
func (t AvailabilityTarget) IsAssignedToProbeAgent(agentID string) bool {
agentID = strings.TrimSpace(agentID)
if agentID == "" {
return false
}
for _, assignedID := range t.AssignedProbeAgentIDs() {
if assignedID == agentID {
return true
}
}
return false
} }
func (t AvailabilityTarget) CertificateMonitoringEnabled() bool { func (t AvailabilityTarget) CertificateMonitoringEnabled() bool {
@@ -331,6 +400,21 @@ func (t AvailabilityTarget) Validate() error {
if t.FailureThreshold > 0 && t.FailureThreshold > 10 { if t.FailureThreshold > 0 && t.FailureThreshold > 10 {
return fmt.Errorf("availability failure threshold must be 10 or less") return fmt.Errorf("availability failure threshold must be 10 or less")
} }
locations := t.EffectiveObservationLocationIDs()
if len(locations) == 0 {
return fmt.Errorf("availability target requires at least one observation location")
}
if len(locations) > 16 {
return fmt.Errorf("availability target supports at most 16 observation locations")
}
for _, locationID := range locations {
if locationID == AvailabilityObservationLocationLocal {
continue
}
if AvailabilityObservationLocationAgentID(locationID) == "" {
return fmt.Errorf("unsupported availability observation location %q", locationID)
}
}
if protocol == AvailabilityProbeHTTP || protocol == AvailabilityProbeHTTPS { if protocol == AvailabilityProbeHTTP || protocol == AvailabilityProbeHTTPS {
if _, err := t.HTTPURL(); err != nil { if _, err := t.HTTPURL(); err != nil {
return err return err
@@ -403,6 +487,24 @@ func NormalizeAvailabilityTarget(target AvailabilityTarget) AvailabilityTarget {
} }
target.LinkedResourceID = strings.TrimSpace(target.LinkedResourceID) target.LinkedResourceID = strings.TrimSpace(target.LinkedResourceID)
target.ProbeAgentID = strings.TrimSpace(target.ProbeAgentID) target.ProbeAgentID = strings.TrimSpace(target.ProbeAgentID)
target.ObservationLocationIDs = normalizeAvailabilityObservationLocationIDs(target.ObservationLocationIDs)
if target.ProbeAgentID != "" && len(target.ObservationLocationIDs) == 1 && target.ObservationLocationIDs[0] == AvailabilityObservationLocationLocal {
// A pre-location client edits ProbeAgentID on a normalized local target.
target.ObservationLocationIDs = []string{AvailabilityAgentObservationLocationID(target.ProbeAgentID)}
}
if len(target.ObservationLocationIDs) == 0 {
if target.ProbeAgentID != "" {
target.ObservationLocationIDs = []string{AvailabilityAgentObservationLocationID(target.ProbeAgentID)}
} else {
target.ObservationLocationIDs = []string{AvailabilityObservationLocationLocal}
}
}
// Keep the legacy field truthful only for the shape it can represent.
if len(target.ObservationLocationIDs) == 1 {
target.ProbeAgentID = AvailabilityObservationLocationAgentID(target.ObservationLocationIDs[0])
} else {
target.ProbeAgentID = ""
}
if target.HTTP != nil { if target.HTTP != nil {
target.HTTP.Method = AvailabilityHTTPMethod(strings.ToUpper(strings.TrimSpace(string(target.HTTP.Method)))) target.HTTP.Method = AvailabilityHTTPMethod(strings.ToUpper(strings.TrimSpace(string(target.HTTP.Method))))
target.HTTP.Authentication.Type = AvailabilityHTTPAuthType(strings.ToLower(strings.TrimSpace(string(target.HTTP.Authentication.Type)))) target.HTTP.Authentication.Type = AvailabilityHTTPAuthType(strings.ToLower(strings.TrimSpace(string(target.HTTP.Authentication.Type))))
@@ -432,6 +534,41 @@ func NormalizeAvailabilityTarget(target AvailabilityTarget) AvailabilityTarget {
return target return target
} }
func normalizeAvailabilityObservationLocationIDs(locationIDs []string) []string {
if len(locationIDs) == 0 {
return nil
}
seen := make(map[string]struct{}, len(locationIDs))
locations := make([]string, 0, len(locationIDs))
for _, raw := range locationIDs {
locationID := strings.TrimSpace(raw)
if locationID == "local" {
locationID = AvailabilityObservationLocationLocal
}
if agentID := AvailabilityObservationLocationAgentID(locationID); agentID != "" {
locationID = AvailabilityAgentObservationLocationID(agentID)
}
if locationID == "" {
continue
}
if _, ok := seen[locationID]; ok {
continue
}
seen[locationID] = struct{}{}
locations = append(locations, locationID)
}
sort.SliceStable(locations, func(i, j int) bool {
if locations[i] == AvailabilityObservationLocationLocal {
return true
}
if locations[j] == AvailabilityObservationLocationLocal {
return false
}
return locations[i] < locations[j]
})
return locations
}
func validateAvailabilityHTTPConfig(contract *AvailabilityHTTPConfig) error { func validateAvailabilityHTTPConfig(contract *AvailabilityHTTPConfig) error {
if contract == nil { if contract == nil {
return nil return nil
+39
View File
@@ -2,6 +2,7 @@ package config
import ( import (
"os" "os"
"reflect"
"strings" "strings"
"testing" "testing"
) )
@@ -312,6 +313,44 @@ func TestNormalizeAvailabilityTargetTrimsProbeAgentID(t *testing.T) {
} }
} }
func TestNormalizeAvailabilityTargetBuildsCanonicalObservationLocationSet(t *testing.T) {
target := NormalizeAvailabilityTarget(AvailabilityTarget{
Address: "service.local",
Protocol: AvailabilityProbeICMP,
Enabled: true,
ObservationLocationIDs: []string{" agent:edge-b ", "local", "agent:edge-a", "agent:edge-a"},
})
want := []string{AvailabilityObservationLocationLocal, "agent:edge-a", "agent:edge-b"}
if !reflect.DeepEqual(target.ObservationLocationIDs, want) {
t.Fatalf("ObservationLocationIDs = %#v, want %#v", target.ObservationLocationIDs, want)
}
if target.ProbeAgentID != "" {
t.Fatalf("ProbeAgentID = %q, multi-location targets must not claim one owner", target.ProbeAgentID)
}
if !target.IncludesLocalObservation() || !reflect.DeepEqual(target.AssignedProbeAgentIDs(), []string{"edge-a", "edge-b"}) {
t.Fatalf("normalized locations = local %v agents %#v", target.IncludesLocalObservation(), target.AssignedProbeAgentIDs())
}
}
func TestAvailabilityExecutionConfigChangedTracksObservationLocationSet(t *testing.T) {
base := NormalizeAvailabilityTarget(AvailabilityTarget{
Address: "service.local",
Protocol: AvailabilityProbeICMP,
Enabled: true,
ObservationLocationIDs: []string{AvailabilityObservationLocationLocal, "agent:edge-a"},
})
reordered := base
reordered.ObservationLocationIDs = []string{"agent:edge-a", AvailabilityObservationLocationLocal}
if AvailabilityExecutionConfigChanged(base, reordered) {
t.Fatal("location ordering changed the execution revision")
}
expanded := base
expanded.ObservationLocationIDs = append(expanded.ObservationLocationIDs, "agent:edge-b")
if !AvailabilityExecutionConfigChanged(base, expanded) {
t.Fatal("adding an observation location did not change the execution revision")
}
}
func TestAvailabilityTargetProbeAgentIDRoundTripsThroughPersistence(t *testing.T) { func TestAvailabilityTargetProbeAgentIDRoundTripsThroughPersistence(t *testing.T) {
persistence := NewConfigPersistence(t.TempDir()) persistence := NewConfigPersistence(t.TempDir())
targets := []AvailabilityTarget{ targets := []AvailabilityTarget{
+32
View File
@@ -349,6 +349,35 @@ func availabilityFixtureRecord(fixture AvailabilityFixture, now time.Time) (unif
PollIntervalSeconds: target.effectivePollIntervalSecs(), PollIntervalSeconds: target.effectivePollIntervalSecs(),
TimeoutMillis: target.effectiveTimeoutMillis(), TimeoutMillis: target.effectiveTimeoutMillis(),
} }
if target.ID == "mock-availability-docker-frontend-service" {
data.AggregateState = "degraded"
data.Disagreement = true
data.ExpectedLocations = 2
data.ReportingLocations = 2
data.Locations = []unifiedresources.AvailabilityObservationLocation{
{
LocationID: "pulse:local",
Kind: "pulse",
Outcome: "reachable",
Available: true,
LastChecked: availabilityFixtureTimePointer(fixture.LastChecked),
LastSuccess: availabilityFixtureTimePointer(fixture.LastSuccess),
FreshnessAt: availabilityFixtureTimePointer(fixture.LastChecked),
LatencyMillis: fixture.LatencyMillis,
},
{
LocationID: "agent:edge-london",
Kind: "agent",
ProbeAgentID: "edge-london",
Outcome: "unreachable",
Available: false,
LastChecked: availabilityFixtureTimePointer(fixture.LastChecked),
FreshnessAt: availabilityFixtureTimePointer(fixture.LastChecked),
ConsecutiveFailures: 2,
LastError: "connection timed out",
},
}
}
data.Evidence = mockAvailabilityEvidence(target, fixture, lastSeen, now) data.Evidence = mockAvailabilityEvidence(target, fixture, lastSeen, now)
if target.Protocol == mockAvailabilityProbeHTTPS { if target.Protocol == mockAvailabilityProbeHTTPS {
data.CertificateMonitoring = true data.CertificateMonitoring = true
@@ -377,6 +406,9 @@ func availabilityFixtureRecord(fixture AvailabilityFixture, now time.Time) (unif
Tags: availabilityFixtureTags(target), Tags: availabilityFixtureTags(target),
Availability: data, Availability: data,
} }
if data.AggregateState == "degraded" {
resource.Status = unifiedresources.StatusWarning
}
if incident := availabilityFixtureIncident(target, fixture, lastSeen); incident != nil { if incident := availabilityFixtureIncident(target, fixture, lastSeen); incident != nil {
resource.Incidents = []unifiedresources.ResourceIncident{*incident} resource.Incidents = []unifiedresources.ResourceIncident{*incident}
} }
+377 -48
View File
@@ -25,27 +25,32 @@ type tlsCert = tlsutil.CertificateObservation
// AvailabilityProbeStatus captures the last observed state of an agentless // AvailabilityProbeStatus captures the last observed state of an agentless
// endpoint probe. // endpoint probe.
type AvailabilityProbeStatus struct { type AvailabilityProbeStatus struct {
TargetID string `json:"targetId"` TargetID string `json:"targetId"`
Name string `json:"name"` Name string `json:"name"`
TargetKind string `json:"targetKind,omitempty"` TargetKind string `json:"targetKind,omitempty"`
Address string `json:"address"` Address string `json:"address"`
Protocol string `json:"protocol"` Protocol string `json:"protocol"`
Outcome string `json:"outcome,omitempty"` Outcome string `json:"outcome,omitempty"`
TransportOutcome string `json:"transportOutcome,omitempty"` TransportOutcome string `json:"transportOutcome,omitempty"`
ApplicationOutcome string `json:"applicationOutcome,omitempty"` ApplicationOutcome string `json:"applicationOutcome,omitempty"`
ApplicationStatusCode int `json:"applicationStatusCode,omitempty"` ApplicationStatusCode int `json:"applicationStatusCode,omitempty"`
ApplicationFailureCode string `json:"applicationFailureCode,omitempty"` ApplicationFailureCode string `json:"applicationFailureCode,omitempty"`
Enabled bool `json:"enabled"` Enabled bool `json:"enabled"`
Available bool `json:"available"` Available bool `json:"available"`
LastChecked time.Time `json:"lastChecked,omitempty"` LastChecked time.Time `json:"lastChecked,omitempty"`
LastSuccess time.Time `json:"lastSuccess,omitempty"` LastSuccess time.Time `json:"lastSuccess,omitempty"`
LatencyMillis int64 `json:"latencyMillis,omitempty"` LatencyMillis int64 `json:"latencyMillis,omitempty"`
ConsecutiveFailures int `json:"consecutiveFailures,omitempty"` ConsecutiveFailures int `json:"consecutiveFailures,omitempty"`
LastError string `json:"lastError,omitempty"` LastError string `json:"lastError,omitempty"`
FailureThreshold int `json:"failureThreshold,omitempty"` FailureThreshold int `json:"failureThreshold,omitempty"`
ProbeAgentID string `json:"probeAgentId,omitempty"` ProbeAgentID string `json:"probeAgentId,omitempty"`
Certificate *tlsCert `json:"certificate,omitempty"` AggregateState string `json:"aggregateState,omitempty"`
CertificateCurrent bool `json:"-"` Disagreement bool `json:"disagreement,omitempty"`
ExpectedLocations int `json:"expectedLocations,omitempty"`
ReportingLocations int `json:"reportingLocations,omitempty"`
Locations []AvailabilityObservationLocationStatus `json:"locations,omitempty"`
Certificate *tlsCert `json:"certificate,omitempty"`
CertificateCurrent bool `json:"-"`
// ProbeReportReceivedAt is server-authored freshness evidence for a remote // ProbeReportReceivedAt is server-authored freshness evidence for a remote
// observation. Keep it off the wire: LastChecked remains the agent's // observation. Keep it off the wire: LastChecked remains the agent's
// observation time, while disconnect detection must not trust agent clock // observation time, while disconnect detection must not trust agent clock
@@ -53,11 +58,41 @@ type AvailabilityProbeStatus struct {
ProbeReportReceivedAt time.Time `json:"-"` ProbeReportReceivedAt time.Time `json:"-"`
} }
const (
AvailabilityAggregateHealthy = "healthy"
AvailabilityAggregateDegraded = "degraded"
AvailabilityAggregateUnavailable = "unavailable"
AvailabilityAggregateUnknown = "unknown"
)
// AvailabilityObservationLocationStatus is one source-owned view of a logical
// verification. LocationID is stable configuration identity; Label is resolved
// by the authenticated client so host names and network details are not copied
// into target or history persistence.
type AvailabilityObservationLocationStatus struct {
LocationID string `json:"locationId"`
Kind string `json:"kind"`
ProbeAgentID string `json:"probeAgentId,omitempty"`
Outcome string `json:"outcome,omitempty"`
TransportOutcome string `json:"transportOutcome,omitempty"`
ApplicationOutcome string `json:"applicationOutcome,omitempty"`
ApplicationStatusCode int `json:"applicationStatusCode,omitempty"`
ApplicationFailureCode string `json:"applicationFailureCode,omitempty"`
Available bool `json:"available"`
LastChecked time.Time `json:"lastChecked,omitempty"`
LastSuccess time.Time `json:"lastSuccess,omitempty"`
FreshnessAt time.Time `json:"freshnessAt,omitempty"`
LatencyMillis int64 `json:"latencyMillis,omitempty"`
ConsecutiveFailures int `json:"consecutiveFailures,omitempty"`
LastError string `json:"lastError,omitempty"`
Stale bool `json:"stale,omitempty"`
}
// FreshnessTime returns the authoritative liveness reference for this status. // FreshnessTime returns the authoritative liveness reference for this status.
// Remote observations use server receipt time; local observations use their // Remote observations use server receipt time; local observations use their
// server-authored check time. // server-authored check time.
func (s AvailabilityProbeStatus) FreshnessTime() time.Time { func (s AvailabilityProbeStatus) FreshnessTime() time.Time {
if strings.TrimSpace(s.ProbeAgentID) != "" && !s.ProbeReportReceivedAt.IsZero() { if (strings.TrimSpace(s.ProbeAgentID) != "" || s.ExpectedLocations > 1) && !s.ProbeReportReceivedAt.IsZero() {
return s.ProbeReportReceivedAt return s.ProbeReportReceivedAt
} }
return s.LastChecked return s.LastChecked
@@ -93,7 +128,7 @@ func (availabilityPollProvider) ListInstances(m *Monitor) []string {
// A licensed probe assignment moves execution to the agent. When the // A licensed probe assignment moves execution to the agent. When the
// entitlement lapses the effective assignment collapses to local, and // entitlement lapses the effective assignment collapses to local, and
// the next polling cycle re-plans this instance without a restart. // the next polling cycle re-plans this instance without a restart.
if m.effectiveProbeAgentID(target) != "" { if !m.targetUsesLocalObservation(target) {
continue continue
} }
names = append(names, target.ID) names = append(names, target.ID)
@@ -132,8 +167,8 @@ func (availabilityPollProvider) BuildPollTask(m *Monitor, instanceName string) (
} }
// Refuse the local run even for a queue entry scheduled before the target // Refuse the local run even for a queue entry scheduled before the target
// was assigned, so a probe-assigned target never executes twice. // was assigned, so a probe-assigned target never executes twice.
if agentID := m.effectiveProbeAgentID(target); agentID != "" { if !m.targetUsesLocalObservation(target) {
return PollTask{}, fmt.Errorf("availability target %q is assigned to probe agent %q", instanceName, agentID) return PollTask{}, fmt.Errorf("availability target %q has no local observation location", instanceName)
} }
return PollTask{ return PollTask{
InstanceName: target.ID, InstanceName: target.ID,
@@ -262,17 +297,61 @@ func (m *Monitor) availabilityStatusSnapshotForTargets(targets []config.Availabi
for id, status := range m.availabilityStatuses { for id, status := range m.availabilityStatuses {
out[id] = status out[id] = status
} }
locationSnapshots := make(map[string]map[string]AvailabilityProbeStatus, len(m.availabilityByLocation))
for targetID, locations := range m.availabilityByLocation {
locationSnapshots[targetID] = make(map[string]AvailabilityProbeStatus, len(locations))
for locationID, status := range locations {
locationSnapshots[targetID][locationID] = status
}
}
m.mu.RUnlock() m.mu.RUnlock()
for _, target := range targets { for _, target := range targets {
status, ok := out[target.ID] expected := m.effectiveObservationLocationIDs(target)
if !ok { locations := locationSnapshots[target.ID]
if m.effectiveProbeAgentID(target) == "" { if locations == nil {
continue locations = make(map[string]AvailabilityProbeStatus)
}
status = availabilityStatusFromTarget(target)
} }
out[target.ID] = m.deriveAvailabilityProbeStaleness(target, status, now) // Adopt pre-location in-memory state into its one effective location.
if len(locations) == 0 {
if legacy, ok := out[target.ID]; ok && len(expected) == 1 {
legacyLocationID := config.AvailabilityObservationLocationLocal
if legacy.ProbeAgentID != "" {
legacyLocationID = config.AvailabilityAgentObservationLocationID(legacy.ProbeAgentID)
}
if legacyLocationID == expected[0] || expected[0] == config.AvailabilityObservationLocationLocal {
if expected[0] == config.AvailabilityObservationLocationLocal {
legacy.ProbeAgentID = ""
legacy.ProbeReportReceivedAt = time.Time{}
}
locations[expected[0]] = legacy
}
}
}
if len(expected) == 1 && expected[0] == config.AvailabilityObservationLocationLocal {
if _, ok := locations[expected[0]]; !ok {
if legacy, exists := out[target.ID]; exists {
legacy.ProbeAgentID = ""
legacy.ProbeReportReceivedAt = time.Time{}
locations[expected[0]] = legacy
} else {
delete(out, target.ID)
continue
}
}
}
for _, locationID := range expected {
status, ok := locations[locationID]
if !ok {
status = availabilityStatusFromTarget(target)
status.ProbeAgentID = config.AvailabilityObservationLocationAgentID(locationID)
}
if status.ProbeAgentID != "" {
status = m.deriveAvailabilityProbeStaleness(target, status, now)
}
locations[locationID] = status
}
out[target.ID] = aggregateAvailabilityLocationStatuses(target, expected, locations)
} }
return out return out
} }
@@ -287,9 +366,13 @@ func (m *Monitor) RefreshAvailabilityTargets() {
now := time.Now() now := time.Now()
for _, target := range targets { for _, target := range targets {
activeIDs[target.ID] = struct{}{} activeIDs[target.ID] = struct{}{}
if agentID := m.effectiveProbeAgentID(target); agentID != "" { for _, locationID := range m.effectiveObservationLocationIDs(target) {
activeProbeIDs[target.ID] = struct{}{} if config.AvailabilityObservationLocationAgentID(locationID) == "" {
m.availabilityProbeAssignmentReference(target.ID, agentID, now) continue
}
trackerID := target.ID + "\x00" + locationID
activeProbeIDs[trackerID] = struct{}{}
m.availabilityProbeAssignmentReference(target.ID, locationID, now)
} }
if m.taskQueue == nil { if m.taskQueue == nil {
continue continue
@@ -300,7 +383,7 @@ func (m *Monitor) RefreshAvailabilityTargets() {
NextRun: now, NextRun: now,
Interval: clampInterval(time.Duration(target.EffectivePollIntervalSecs())*time.Second, 10*time.Second, time.Hour), Interval: clampInterval(time.Duration(target.EffectivePollIntervalSecs())*time.Second, 10*time.Second, time.Hour),
} }
if target.Enabled && m.effectiveProbeAgentID(target) == "" { if target.Enabled && m.targetUsesLocalObservation(target) {
m.taskQueue.Upsert(task) m.taskQueue.Upsert(task)
} else { } else {
m.taskQueue.Remove(InstanceTypeAvailability, target.ID) m.taskQueue.Remove(InstanceTypeAvailability, target.ID)
@@ -316,6 +399,11 @@ func (m *Monitor) RefreshAvailabilityTargets() {
removedIDs = append(removedIDs, id) removedIDs = append(removedIDs, id)
} }
} }
for id := range m.availabilityByLocation {
if _, ok := activeIDs[id]; !ok {
delete(m.availabilityByLocation, id)
}
}
for id := range m.availabilityProbeTrackers { for id := range m.availabilityProbeTrackers {
if _, ok := activeProbeIDs[id]; !ok { if _, ok := activeProbeIDs[id]; !ok {
delete(m.availabilityProbeTrackers, id) delete(m.availabilityProbeTrackers, id)
@@ -375,15 +463,21 @@ func (m *Monitor) applyAvailabilityObservationDetailed(
} }
m.setAvailabilityStatusWithDetails(target, checkedAt, latency, outcome, transportOutcome, application, probeErr, certificate, probeAgentID, probeReportReceivedAt) m.setAvailabilityStatusWithDetails(target, checkedAt, latency, outcome, transportOutcome, application, probeErr, certificate, probeAgentID, probeReportReceivedAt)
m.recordAvailabilityHistory(target, observationID, checkedAt, latency, outcome, probeErr, probeAgentID, probeReportReceivedAt) m.recordAvailabilityHistory(target, observationID, checkedAt, latency, outcome, probeErr, probeAgentID, probeReportReceivedAt)
current := m.AvailabilityStatusSnapshot()[target.ID]
aggregateHealthy := current.AggregateState == AvailabilityAggregateHealthy || current.AggregateState == AvailabilityAggregateDegraded
if probeErr == nil { if aggregateHealthy {
if m.stalenessTracker != nil { if m.stalenessTracker != nil {
m.stalenessTracker.UpdateSuccess(InstanceTypeAvailability, target.ID, nil) m.stalenessTracker.UpdateSuccess(InstanceTypeAvailability, target.ID, nil)
} }
m.setProviderConnectionHealth(InstanceTypeAvailability, target.ID, true) m.setProviderConnectionHealth(InstanceTypeAvailability, target.ID, true)
} else { } else {
if m.stalenessTracker != nil { if m.stalenessTracker != nil {
m.stalenessTracker.UpdateSuccess(InstanceTypeAvailability, target.ID, []byte(probeErr.Error())) reason := strings.TrimSpace(current.LastError)
if reason == "" {
reason = "availability evidence is incomplete"
}
m.stalenessTracker.UpdateSuccess(InstanceTypeAvailability, target.ID, []byte(reason))
} }
m.setProviderConnectionHealth(InstanceTypeAvailability, target.ID, false) m.setProviderConnectionHealth(InstanceTypeAvailability, target.ID, false)
} }
@@ -421,6 +515,23 @@ func (m *Monitor) recordAvailabilityHistory(
} }
latencyMillis = &value latencyMillis = &value
} }
expected := m.effectiveObservationLocationIDs(target)
if len(expected) > 1 {
aggregate := m.AvailabilityStatusSnapshot()[target.ID]
outcome = AvailabilityProbeIndeterminate
latencyMillis = nil
switch aggregate.AggregateState {
case AvailabilityAggregateHealthy, AvailabilityAggregateDegraded:
outcome = AvailabilityProbeReachable
if aggregate.LatencyMillis > 0 {
value := aggregate.LatencyMillis
latencyMillis = &value
}
case AvailabilityAggregateUnavailable:
outcome = AvailabilityProbeUnreachable
}
observationID += "|aggregate"
}
if err := m.metricsStore.WriteAvailabilityObservationBounded(pkgmetrics.AvailabilityObservation{ if err := m.metricsStore.WriteAvailabilityObservationBounded(pkgmetrics.AvailabilityObservation{
ObservationID: observationID, ObservationID: observationID,
TargetID: target.ID, TargetID: target.ID,
@@ -477,6 +588,11 @@ func (m *Monitor) setAvailabilityStatusWithDetails(
if m == nil { if m == nil {
return return
} }
locationID := config.AvailabilityObservationLocationLocal
if agentID := strings.TrimSpace(probeAgentID); agentID != "" {
locationID = config.AvailabilityAgentObservationLocationID(agentID)
}
expectedLocations := m.effectiveObservationLocationIDs(target)
status := availabilityStatusFromTarget(target) status := availabilityStatusFromTarget(target)
status.Outcome = string(outcome) status.Outcome = string(outcome)
status.TransportOutcome = string(transportOutcome) status.TransportOutcome = string(transportOutcome)
@@ -514,7 +630,13 @@ func (m *Monitor) setAvailabilityStatusWithDetails(
if m.availabilityStatuses == nil { if m.availabilityStatuses == nil {
m.availabilityStatuses = make(map[string]AvailabilityProbeStatus) m.availabilityStatuses = make(map[string]AvailabilityProbeStatus)
} }
if previous, ok := m.availabilityStatuses[target.ID]; ok { if m.availabilityByLocation == nil {
m.availabilityByLocation = make(map[string]map[string]AvailabilityProbeStatus)
}
if m.availabilityByLocation[target.ID] == nil {
m.availabilityByLocation[target.ID] = make(map[string]AvailabilityProbeStatus)
}
if previous, ok := m.availabilityByLocation[target.ID][locationID]; ok {
status.LastSuccess = previous.LastSuccess status.LastSuccess = previous.LastSuccess
if status.Certificate == nil { if status.Certificate == nil {
status.Certificate = previous.Certificate.Clone() status.Certificate = previous.Certificate.Clone()
@@ -531,7 +653,12 @@ func (m *Monitor) setAvailabilityStatusWithDetails(
} else if probeErr != nil { } else if probeErr != nil {
status.ConsecutiveFailures = 1 status.ConsecutiveFailures = 1
} }
m.availabilityStatuses[target.ID] = status m.availabilityByLocation[target.ID][locationID] = status
m.availabilityStatuses[target.ID] = aggregateAvailabilityLocationStatuses(
target,
expectedLocations,
m.availabilityByLocation[target.ID],
)
m.mu.Unlock() m.mu.Unlock()
} }
@@ -567,6 +694,168 @@ func availabilityStatusFromTarget(target config.AvailabilityTarget) Availability
} }
} }
func aggregateAvailabilityLocationStatuses(
target config.AvailabilityTarget,
expected []string,
statuses map[string]AvailabilityProbeStatus,
) AvailabilityProbeStatus {
aggregate := availabilityStatusFromTarget(target)
aggregate.ExpectedLocations = len(expected)
if len(expected) == 0 {
aggregate.AggregateState = AvailabilityAggregateUnknown
return aggregate
}
reachable, unreachable, indeterminate := 0, 0, 0
minFailures := 0
latestFreshness := time.Time{}
latestChecked := time.Time{}
latestSuccess := time.Time{}
var representative *AvailabilityProbeStatus
var latestReachable *AvailabilityProbeStatus
for _, locationID := range expected {
status, ok := statuses[locationID]
if !ok {
status = availabilityStatusFromTarget(target)
status.ProbeAgentID = config.AvailabilityObservationLocationAgentID(locationID)
}
effectiveOutcome := status.Outcome
if effectiveOutcome == "" && !status.LastChecked.IsZero() {
if status.Available {
effectiveOutcome = string(AvailabilityProbeReachable)
} else if status.ConsecutiveFailures > 0 || strings.TrimSpace(status.LastError) != "" {
effectiveOutcome = string(AvailabilityProbeUnreachable)
} else {
effectiveOutcome = string(AvailabilityProbeIndeterminate)
}
status.Outcome = effectiveOutcome
}
freshness := status.FreshnessTime()
location := AvailabilityObservationLocationStatus{
LocationID: locationID,
Kind: "pulse",
ProbeAgentID: status.ProbeAgentID,
Outcome: status.Outcome,
TransportOutcome: status.TransportOutcome,
ApplicationOutcome: status.ApplicationOutcome,
ApplicationStatusCode: status.ApplicationStatusCode,
ApplicationFailureCode: status.ApplicationFailureCode,
Available: status.Available,
LastChecked: status.LastChecked,
LastSuccess: status.LastSuccess,
FreshnessAt: freshness,
LatencyMillis: status.LatencyMillis,
ConsecutiveFailures: status.ConsecutiveFailures,
LastError: status.LastError,
Stale: availabilityProbeStatusIsStale(status),
}
if status.ProbeAgentID != "" {
location.Kind = "agent"
}
aggregate.Locations = append(aggregate.Locations, location)
if status.LastChecked.IsZero() {
continue
}
if !availabilityProbeStatusIsStale(status) {
aggregate.ReportingLocations++
}
if freshness.After(latestFreshness) {
latestFreshness = freshness
}
if status.LastChecked.After(latestChecked) {
latestChecked = status.LastChecked
copy := status
representative = &copy
}
if status.LastSuccess.After(latestSuccess) {
latestSuccess = status.LastSuccess
}
switch effectiveOutcome {
case string(AvailabilityProbeReachable):
reachable++
if latestReachable == nil || status.LastChecked.After(latestReachable.LastChecked) {
copy := status
latestReachable = &copy
}
case string(AvailabilityProbeUnreachable):
unreachable++
if minFailures == 0 || status.ConsecutiveFailures < minFailures {
minFailures = status.ConsecutiveFailures
}
case string(AvailabilityProbeIndeterminate):
indeterminate++
}
}
if representative != nil {
aggregate.Outcome = representative.Outcome
aggregate.TransportOutcome = representative.TransportOutcome
aggregate.ApplicationOutcome = representative.ApplicationOutcome
aggregate.ApplicationStatusCode = representative.ApplicationStatusCode
aggregate.ApplicationFailureCode = representative.ApplicationFailureCode
aggregate.LatencyMillis = representative.LatencyMillis
aggregate.Certificate = representative.Certificate.Clone()
aggregate.CertificateCurrent = representative.CertificateCurrent
}
if latestReachable != nil {
aggregate.LatencyMillis = latestReachable.LatencyMillis
} else if len(expected) > 1 {
aggregate.LatencyMillis = 0
}
aggregate.LastChecked = latestChecked
aggregate.LastSuccess = latestSuccess
aggregate.ProbeReportReceivedAt = latestFreshness
aggregate.Disagreement = reachable > 0 && unreachable > 0
unknown := len(expected) - reachable - unreachable - indeterminate
switch {
case reachable == len(expected):
aggregate.AggregateState = AvailabilityAggregateHealthy
aggregate.Available = true
aggregate.Outcome = string(AvailabilityProbeReachable)
case reachable > 0:
aggregate.AggregateState = AvailabilityAggregateDegraded
aggregate.Available = true
aggregate.Outcome = string(AvailabilityProbeReachable)
case unreachable == len(expected):
aggregate.AggregateState = AvailabilityAggregateUnavailable
aggregate.Available = false
aggregate.Outcome = string(AvailabilityProbeUnreachable)
aggregate.ConsecutiveFailures = minFailures
aggregate.LastError = "all observation locations report the target unreachable"
default:
aggregate.AggregateState = AvailabilityAggregateUnknown
aggregate.Available = false
aggregate.Outcome = string(AvailabilityProbeIndeterminate)
if indeterminate > 0 || unknown > 0 {
aggregate.LastError = "observation coverage is incomplete"
}
}
if len(expected) == 1 && representative != nil {
// Preserve the established one-location response while adding the location
// envelope, so older consumers do not lose source attribution or errors.
locations := aggregate.Locations
state := aggregate.AggregateState
reporting := aggregate.ReportingLocations
aggregate = *representative
aggregate.AggregateState = state
aggregate.ExpectedLocations = 1
aggregate.ReportingLocations = reporting
aggregate.Locations = locations
}
if len(expected) == 1 && representative == nil {
status := statuses[expected[0]]
locations := aggregate.Locations
state := aggregate.AggregateState
reporting := aggregate.ReportingLocations
aggregate = status
aggregate.AggregateState = state
aggregate.ExpectedLocations = 1
aggregate.ReportingLocations = reporting
aggregate.Locations = locations
}
return aggregate
}
func availabilityResourceFromTarget(target config.AvailabilityTarget, status AvailabilityProbeStatus, _ string, now time.Time) (unifiedresources.Resource, unifiedresources.ResourceIdentity) { func availabilityResourceFromTarget(target config.AvailabilityTarget, status AvailabilityProbeStatus, _ string, now time.Time) (unifiedresources.Resource, unifiedresources.ResourceIdentity) {
lastSeen := status.FreshnessTime() lastSeen := status.FreshnessTime()
if lastSeen.IsZero() { if lastSeen.IsZero() {
@@ -586,6 +875,10 @@ func availabilityResourceFromTarget(target config.AvailabilityTarget, status Ava
ApplicationStatusCode: status.ApplicationStatusCode, ApplicationStatusCode: status.ApplicationStatusCode,
ApplicationFailureCode: status.ApplicationFailureCode, ApplicationFailureCode: status.ApplicationFailureCode,
ProbeAgentID: status.ProbeAgentID, ProbeAgentID: status.ProbeAgentID,
AggregateState: status.AggregateState,
Disagreement: status.Disagreement,
ExpectedLocations: status.ExpectedLocations,
ReportingLocations: status.ReportingLocations,
UDPMode: string(target.UDPMode), UDPMode: string(target.UDPMode),
Port: target.Port, Port: target.Port,
Path: target.Path, Path: target.Path,
@@ -600,6 +893,26 @@ func availabilityResourceFromTarget(target config.AvailabilityTarget, status Ava
PollIntervalSeconds: target.EffectivePollIntervalSecs(), PollIntervalSeconds: target.EffectivePollIntervalSecs(),
TimeoutMillis: target.EffectiveTimeoutMillis(), TimeoutMillis: target.EffectiveTimeoutMillis(),
} }
for _, location := range status.Locations {
data.Locations = append(data.Locations, unifiedresources.AvailabilityObservationLocation{
LocationID: location.LocationID,
Kind: location.Kind,
ProbeAgentID: location.ProbeAgentID,
Outcome: location.Outcome,
TransportOutcome: location.TransportOutcome,
ApplicationOutcome: location.ApplicationOutcome,
ApplicationStatusCode: location.ApplicationStatusCode,
ApplicationFailureCode: location.ApplicationFailureCode,
Available: location.Available,
LastChecked: timePointerIfSet(location.LastChecked),
LastSuccess: timePointerIfSet(location.LastSuccess),
FreshnessAt: timePointerIfSet(location.FreshnessAt),
LatencyMillis: location.LatencyMillis,
ConsecutiveFailures: location.ConsecutiveFailures,
LastError: location.LastError,
Stale: location.Stale,
})
}
data.CertificateMonitoring = target.CertificateMonitoringEnabled() data.CertificateMonitoring = target.CertificateMonitoringEnabled()
data.CertificateExpiryWarningDays = target.EffectiveCertificateExpiryWarningDays() data.CertificateExpiryWarningDays = target.EffectiveCertificateExpiryWarningDays()
data.Certificate = status.Certificate.Clone() data.Certificate = status.Certificate.Clone()
@@ -612,7 +925,7 @@ func availabilityResourceFromTarget(target config.AvailabilityTarget, status Ava
observedAt = lastSeen observedAt = lastSeen
} }
ingestedAt := now ingestedAt := now
if strings.TrimSpace(status.ProbeAgentID) != "" && !status.ProbeReportReceivedAt.IsZero() { if !status.ProbeReportReceivedAt.IsZero() {
ingestedAt = status.ProbeReportReceivedAt ingestedAt = status.ProbeReportReceivedAt
} }
data.Evidence = availabilityEvidenceEnvelope(target, status, observedAt, ingestedAt) data.Evidence = availabilityEvidenceEnvelope(target, status, observedAt, ingestedAt)
@@ -672,22 +985,24 @@ func availabilityEvidenceEnvelope(
return nil return nil
} }
freshnessAt := observedAt freshnessAt := status.FreshnessTime()
if freshnessAt.IsZero() {
freshnessAt = observedAt
}
validityWindow := time.Duration(target.EffectivePollIntervalSecs()*2) * time.Second validityWindow := time.Duration(target.EffectivePollIntervalSecs()*2) * time.Second
if strings.TrimSpace(status.ProbeAgentID) != "" && !status.ProbeReportReceivedAt.IsZero() { if strings.TrimSpace(status.ProbeAgentID) != "" || status.ExpectedLocations > 1 {
freshnessAt = status.ProbeReportReceivedAt
validityWindow = availabilityProbeStaleWindow(target) validityWindow = availabilityProbeStaleWindow(target)
} }
validUntil := freshnessAt.Add(validityWindow) validUntil := freshnessAt.Add(validityWindow)
completeness := operationaltrust.EvidenceComplete completeness := operationaltrust.EvidenceComplete
confidence := operationaltrust.EvidenceConfirmed confidence := operationaltrust.EvidenceConfirmed
var reason *operationaltrust.EvidenceReason var reason *operationaltrust.EvidenceReason
if status.LastChecked.IsZero() { if status.LastChecked.IsZero() || status.ReportingLocations < status.ExpectedLocations {
completeness = operationaltrust.EvidencePartial completeness = operationaltrust.EvidencePartial
confidence = operationaltrust.EvidenceUnknown confidence = operationaltrust.EvidenceUnknown
reason = &operationaltrust.EvidenceReason{ reason = &operationaltrust.EvidenceReason{
Code: "availability_not_observed", Code: "availability_coverage_incomplete",
Message: "The availability target has not completed its first probe.", Message: "One or more observation locations have no current evidence.",
} }
} }
@@ -728,6 +1043,17 @@ func availabilityResourceStatus(target config.AvailabilityTarget, status Availab
if availabilityProbeStatusIsStale(status) { if availabilityProbeStatusIsStale(status) {
return unifiedresources.StatusWarning return unifiedresources.StatusWarning
} }
switch status.AggregateState {
case AvailabilityAggregateHealthy:
return unifiedresources.StatusOnline
case AvailabilityAggregateDegraded, AvailabilityAggregateUnknown:
return unifiedresources.StatusWarning
case AvailabilityAggregateUnavailable:
if status.ConsecutiveFailures >= target.EffectiveFailureThreshold() {
return unifiedresources.StatusOffline
}
return unifiedresources.StatusWarning
}
if status.LastChecked.IsZero() { if status.LastChecked.IsZero() {
return unifiedresources.StatusUnknown return unifiedresources.StatusUnknown
} }
@@ -747,13 +1073,16 @@ func availabilityIncident(target config.AvailabilityTarget, status AvailabilityP
if availabilityProbeStatusIsStale(status) { if availabilityProbeStatusIsStale(status) {
return nil return nil
} }
if status.AggregateState != "" && status.AggregateState != AvailabilityAggregateUnavailable {
return nil
}
if status.LastChecked.IsZero() { if status.LastChecked.IsZero() {
return nil return nil
} }
if status.ConsecutiveFailures < target.EffectiveFailureThreshold() { if status.ConsecutiveFailures < target.EffectiveFailureThreshold() {
return nil return nil
} }
summary := fmt.Sprintf("%s is unreachable by %s probe", target.DisplayName(), strings.ToUpper(string(target.Protocol))) summary := fmt.Sprintf("%s is unreachable from every current observation location by %s probe", target.DisplayName(), strings.ToUpper(string(target.Protocol)))
if status.LastError != "" { if status.LastError != "" {
summary = summary + ": " + status.LastError summary = summary + ": " + status.LastError
} }
+50 -22
View File
@@ -42,8 +42,8 @@ type ProbeAvailabilityResult struct {
// assigned target before its first report. AgentID is retained so reassigning // assigned target before its first report. AgentID is retained so reassigning
// the same target to a different probe starts a fresh grace window. // the same target to a different probe starts a fresh grace window.
type availabilityProbeAssignmentTracker struct { type availabilityProbeAssignmentTracker struct {
AgentID string LocationID string
Since time.Time Since time.Time
} }
// probeAvailabilityResultsFromReport converts the wire results carried by a // probeAvailabilityResultsFromReport converts the wire results carried by a
@@ -89,14 +89,45 @@ func probeAvailabilityResultsFromReport(reported []agentshost.AvailabilityProbeR
// entitlement is absent, so a license lapse resumes local polling instead of // entitlement is absent, so a license lapse resumes local polling instead of
// stranding the check on an agent that is no longer allowed to run it. // stranding the check on an agent that is no longer allowed to run it.
func (m *Monitor) effectiveProbeAgentID(target config.AvailabilityTarget) string { func (m *Monitor) effectiveProbeAgentID(target config.AvailabilityTarget) string {
assigned := strings.TrimSpace(target.ProbeAgentID) locations := m.effectiveObservationLocationIDs(target)
if assigned == "" { if len(locations) != 1 {
return "" return ""
} }
if !m.hasLicensedFeature(pkglicensing.FeatureExternalProbe) { return config.AvailabilityObservationLocationAgentID(locations[0])
return "" }
func (m *Monitor) effectiveObservationLocationIDs(target config.AvailabilityTarget) []string {
locations := target.EffectiveObservationLocationIDs()
if m.hasLicensedFeature(pkglicensing.FeatureExternalProbe) {
return locations
} }
return assigned for _, locationID := range locations {
if locationID == config.AvailabilityObservationLocationLocal {
return []string{config.AvailabilityObservationLocationLocal}
}
}
// Preserve the historical entitlement-lapse behavior: a remote-only check
// resumes locally instead of becoming unobserved.
return []string{config.AvailabilityObservationLocationLocal}
}
func (m *Monitor) targetUsesLocalObservation(target config.AvailabilityTarget) bool {
for _, locationID := range m.effectiveObservationLocationIDs(target) {
if locationID == config.AvailabilityObservationLocationLocal {
return true
}
}
return false
}
func (m *Monitor) targetAssignedToProbeAgent(target config.AvailabilityTarget, agentID string) bool {
locationID := config.AvailabilityAgentObservationLocationID(agentID)
for _, assignedLocationID := range m.effectiveObservationLocationIDs(target) {
if assignedLocationID == locationID {
return true
}
}
return false
} }
// ApplyProbeAvailabilityResults ingests availability results reported by a host // ApplyProbeAvailabilityResults ingests availability results reported by a host
@@ -137,7 +168,7 @@ func (m *Monitor) applyProbeAvailabilityResultsAt(hostID string, results []Probe
Msg("Rejecting probe availability result for unknown target") Msg("Rejecting probe availability result for unknown target")
continue continue
} }
if m.effectiveProbeAgentID(target) != hostID { if !m.targetAssignedToProbeAgent(target, hostID) {
log.Debug(). log.Debug().
Str("hostID", hostID). Str("hostID", hostID).
Str("targetID", targetID). Str("targetID", targetID).
@@ -240,18 +271,14 @@ func (m *Monitor) deriveAvailabilityProbeStaleness(
status AvailabilityProbeStatus, status AvailabilityProbeStatus,
now time.Time, now time.Time,
) AvailabilityProbeStatus { ) AvailabilityProbeStatus {
agentID := m.effectiveProbeAgentID(target) agentID := strings.TrimSpace(status.ProbeAgentID)
if agentID == "" { if agentID == "" {
return status return status
} }
reportingAgentID := strings.TrimSpace(status.ProbeAgentID)
if reportingAgentID != agentID {
status = availabilityStatusFromTarget(target)
}
status.ProbeAgentID = agentID status.ProbeAgentID = agentID
reference := status.FreshnessTime() reference := status.FreshnessTime()
if reference.IsZero() { if reference.IsZero() {
reference = m.availabilityProbeAssignmentReference(target.ID, agentID, now) reference = m.availabilityProbeAssignmentReference(target.ID, config.AvailabilityAgentObservationLocationID(agentID), now)
} }
if !availabilityProbeReportIsStale(target, reference, now) { if !availabilityProbeReportIsStale(target, reference, now) {
return status return status
@@ -263,27 +290,28 @@ func (m *Monitor) deriveAvailabilityProbeStaleness(
return status return status
} }
func (m *Monitor) availabilityProbeAssignmentReference(targetID, agentID string, now time.Time) time.Time { func (m *Monitor) availabilityProbeAssignmentReference(targetID, locationID string, now time.Time) time.Time {
if m == nil { if m == nil {
return now return now
} }
targetID = strings.TrimSpace(targetID) targetID = strings.TrimSpace(targetID)
agentID = strings.TrimSpace(agentID) locationID = strings.TrimSpace(locationID)
if targetID == "" || agentID == "" { if targetID == "" || locationID == "" {
return now return now
} }
trackerID := targetID + "\x00" + locationID
m.mu.Lock() m.mu.Lock()
defer m.mu.Unlock() defer m.mu.Unlock()
if m.availabilityProbeTrackers == nil { if m.availabilityProbeTrackers == nil {
m.availabilityProbeTrackers = make(map[string]availabilityProbeAssignmentTracker) m.availabilityProbeTrackers = make(map[string]availabilityProbeAssignmentTracker)
} }
if tracker, ok := m.availabilityProbeTrackers[targetID]; ok && tracker.AgentID == agentID && !tracker.Since.IsZero() { if tracker, ok := m.availabilityProbeTrackers[trackerID]; ok && tracker.LocationID == locationID && !tracker.Since.IsZero() {
return tracker.Since return tracker.Since
} }
m.availabilityProbeTrackers[targetID] = availabilityProbeAssignmentTracker{ m.availabilityProbeTrackers[trackerID] = availabilityProbeAssignmentTracker{
AgentID: agentID, LocationID: locationID,
Since: now, Since: now,
} }
return now return now
} }
@@ -317,7 +345,7 @@ func (m *Monitor) availabilityProbeTargetsForAgent(hostID string) []map[string]i
} }
var assigned []map[string]interface{} var assigned []map[string]interface{}
for _, target := range m.availabilityTargets() { for _, target := range m.availabilityTargets() {
if m.effectiveProbeAgentID(target) != hostID { if !m.targetAssignedToProbeAgent(target, hostID) {
continue continue
} }
assigned = append(assigned, availabilityProbeAgentTargetPayload(target)) assigned = append(assigned, availabilityProbeAgentTargetPayload(target))
@@ -69,14 +69,99 @@ func newProbeAgentTestMonitor(t *testing.T, targets ...config.AvailabilityTarget
t.Fatalf("SaveAvailabilityTargets() error = %v", err) t.Fatalf("SaveAvailabilityTargets() error = %v", err)
} }
return &Monitor{ return &Monitor{
state: models.NewState(), state: models.NewState(),
configPersist: persistence, configPersist: persistence,
availabilityStatuses: make(map[string]AvailabilityProbeStatus), availabilityStatuses: make(map[string]AvailabilityProbeStatus),
pollStatusMap: make(map[string]*pollStatus), availabilityByLocation: make(map[string]map[string]AvailabilityProbeStatus),
failureCounts: make(map[string]int), pollStatusMap: make(map[string]*pollStatus),
lastOutcome: make(map[string]taskOutcome), failureCounts: make(map[string]int),
circuitBreakers: make(map[string]*circuitBreaker), lastOutcome: make(map[string]taskOutcome),
taskQueue: NewTaskQueue(), circuitBreakers: make(map[string]*circuitBreaker),
taskQueue: NewTaskQueue(),
}
}
func TestMultiLocationAvailabilityPreservesDisagreementUntilEveryPathFails(t *testing.T) {
target := config.NormalizeAvailabilityTarget(config.AvailabilityTarget{
ID: "service",
Name: "Customer API",
Address: "api.service.local",
Protocol: config.AvailabilityProbeHTTPS,
Enabled: true,
PollIntervalSecs: 60,
FailureThreshold: 2,
ObservationLocationIDs: []string{config.AvailabilityObservationLocationLocal, config.AvailabilityAgentObservationLocationID("edge-1")},
})
monitor := newProbeAgentTestMonitor(t, target)
monitor.SetLicenseChecker(licenseWithExternalProbe(true))
now := time.Now().UTC()
monitor.applyAvailabilityObservation(target, "local-ok", now, 8*time.Millisecond, AvailabilityProbeReachable, nil, nil, "", time.Time{})
for attempt := 0; attempt < 2; attempt++ {
monitor.applyAvailabilityObservation(target, "edge-fail-"+string(rune('a'+attempt)), now.Add(time.Duration(attempt)*time.Second), 30*time.Millisecond, AvailabilityProbeUnreachable, context.DeadlineExceeded, nil, "edge-1", now.Add(time.Duration(attempt)*time.Second))
}
status := monitor.AvailabilityStatusSnapshot()[target.ID]
if status.AggregateState != AvailabilityAggregateDegraded || !status.Disagreement || !status.Available {
t.Fatalf("aggregate status = %+v, want available disagreement", status)
}
if status.ExpectedLocations != 2 || status.ReportingLocations != 2 || len(status.Locations) != 2 {
t.Fatalf("location coverage = %+v, want 2/2 locations", status)
}
if want := now.Add(time.Second); !status.FreshnessTime().Equal(want) {
t.Fatalf("aggregate freshness = %s, want latest server-authored path freshness %s", status.FreshnessTime(), want)
}
if status.LatencyMillis != 8 {
t.Fatalf("aggregate latency = %dms, want the current reachable path latency", status.LatencyMillis)
}
resource, _ := availabilityResourceFromTarget(target, status, "", now)
if resource.Status != unifiedresources.StatusWarning || len(resource.Incidents) != 0 {
t.Fatalf("resource = status %q incidents %+v, one path failure must not author an outage", resource.Status, resource.Incidents)
}
for attempt := 0; attempt < 2; attempt++ {
monitor.applyAvailabilityObservation(target, "local-fail-"+string(rune('a'+attempt)), now.Add(time.Duration(attempt+2)*time.Second), 20*time.Millisecond, AvailabilityProbeUnreachable, context.DeadlineExceeded, nil, "", time.Time{})
}
status = monitor.AvailabilityStatusSnapshot()[target.ID]
if status.AggregateState != AvailabilityAggregateUnavailable || status.Available || status.ConsecutiveFailures != 2 {
t.Fatalf("aggregate status = %+v, want thresholded all-location outage", status)
}
if status.LatencyMillis != 0 {
t.Fatalf("unavailable aggregate latency = %dms, want no reachable latency", status.LatencyMillis)
}
resource, _ = availabilityResourceFromTarget(target, status, "", now.Add(4*time.Second))
if resource.Status != unifiedresources.StatusOffline || len(resource.Incidents) != 1 {
t.Fatalf("resource = status %q incidents %+v, want one universal outage", resource.Status, resource.Incidents)
}
}
func TestMultiLocationAvailabilityTreatsLapsedPathAsUnknownCoverage(t *testing.T) {
target := config.NormalizeAvailabilityTarget(config.AvailabilityTarget{
ID: "service",
Name: "Customer API",
Address: "api.service.local",
Protocol: config.AvailabilityProbeHTTPS,
Enabled: true,
PollIntervalSecs: 60,
FailureThreshold: 2,
ObservationLocationIDs: []string{config.AvailabilityObservationLocationLocal, config.AvailabilityAgentObservationLocationID("edge-1")},
})
monitor := newProbeAgentTestMonitor(t, target)
monitor.SetLicenseChecker(licenseWithExternalProbe(true))
now := time.Now().UTC()
old := time.Now().UTC().Add(-time.Hour)
monitor.applyAvailabilityObservation(target, "local-ok", now, 8*time.Millisecond, AvailabilityProbeReachable, nil, nil, "", time.Time{})
monitor.applyAvailabilityObservation(target, "edge-ok", old, 18*time.Millisecond, AvailabilityProbeReachable, nil, nil, "edge-1", old)
status := monitor.availabilityStatusSnapshotForTargets([]config.AvailabilityTarget{target}, now)[target.ID]
if status.AggregateState != AvailabilityAggregateDegraded || status.Disagreement {
t.Fatalf("aggregate status = %+v, want healthy path plus unknown coverage", status)
}
if status.ReportingLocations != 1 || status.ExpectedLocations != 2 {
t.Fatalf("coverage = %d/%d, want the lapsed path excluded", status.ReportingLocations, status.ExpectedLocations)
}
if !status.Locations[1].Stale || status.Locations[1].Outcome != string(AvailabilityProbeIndeterminate) {
t.Fatalf("remote location = %+v, want stale unknown evidence", status.Locations[1])
} }
} }
@@ -673,8 +673,14 @@ func TestAvailabilityProviderStaysOnCanonicalMonitoringPath(t *testing.T) {
`InstanceTypeAvailability InstanceType = "availability"`, `InstanceTypeAvailability InstanceType = "availability"`,
}, },
"monitor.go": { "monitor.go": {
"availabilityStatuses map[string]AvailabilityProbeStatus", "availabilityStatuses",
"availabilityStatuses: make(map[string]AvailabilityProbeStatus)", "map[string]AvailabilityProbeStatus",
"availabilityByLocation",
"map[string]map[string]AvailabilityProbeStatus",
"availabilityStatuses:",
"make(map[string]AvailabilityProbeStatus)",
"availabilityByLocation:",
"make(map[string]map[string]AvailabilityProbeStatus)",
}, },
"poll_providers.go": { "poll_providers.go": {
"_ = m.RegisterPollProvider(newAvailabilityPollProvider())", "_ = m.RegisterPollProvider(newAvailabilityPollProvider())",
@@ -713,9 +719,11 @@ func TestMonitoredSystemUsageReadinessGuardrailsRemainCanonical(t *testing.T) {
"monitor.go": { "monitor.go": {
"type MonitorSupplementalInventoryReadinessProvider interface {", "type MonitorSupplementalInventoryReadinessProvider interface {",
"SupplementalInventoryReadyAt(m *Monitor, orgID string) (time.Time, bool)", "SupplementalInventoryReadyAt(m *Monitor, orgID string) (time.Time, bool)",
"hostContinuityStore *config.HostContinuityStore", "hostContinuityStore",
"*config.HostContinuityStore",
"hostContinuityStore := config.NewHostContinuityStore(cfg.DataPath, nil)", "hostContinuityStore := config.NewHostContinuityStore(cfg.DataPath, nil)",
"hostContinuityStore: hostContinuityStore,", "hostContinuityStore:",
"hostContinuityStore,",
"func (m *Monitor) HostsSnapshot() []models.Host {", "func (m *Monitor) HostsSnapshot() []models.Host {",
"readState = m.readStateWithStandaloneHostContinuity(readState)", "readState = m.readStateWithStandaloneHostContinuity(readState)",
"func (m *Monitor) unifiedStateViewWithStandaloneHostContinuity(view monitorUnifiedStateView) monitorUnifiedStateView {", "func (m *Monitor) unifiedStateViewWithStandaloneHostContinuity(view monitorUnifiedStateView) monitorUnifiedStateView {",
@@ -905,7 +913,8 @@ func TestProxmoxGuestMemoryFallbackUsesInstanceScopedCachesAndAgentMeminfo(t *te
`memorySource = "unavailable"`, `memorySource = "unavailable"`,
}, },
"monitor.go": { "monitor.go": {
"vmAgentMemCache map[string]agentMemCacheEntry", "vmAgentMemCache",
"map[string]agentMemCacheEntry",
}, },
"monitor_agents.go": { "monitor_agents.go": {
"for key, entry := range m.vmAgentMemCache {", "for key, entry := range m.vmAgentMemCache {",
@@ -1238,8 +1247,10 @@ func TestProxmoxGuestDockerLXCProbingRequiresExplicitOptIn(t *testing.T) {
func TestBackupOrphanDetectionUsesCanonicalInventoryReadinessScope(t *testing.T) { func TestBackupOrphanDetectionUsesCanonicalInventoryReadinessScope(t *testing.T) {
requiredSnippets := map[string][]string{ requiredSnippets := map[string][]string{
"monitor.go": { "monitor.go": {
"pveBackupInventoryReady map[string]map[string]bool", "pveBackupInventoryReady",
"pveBackupTemplateSubjects map[string]map[string]struct{}", "map[string]map[string]bool",
"pveBackupTemplateSubjects",
"map[string]map[string]struct{}",
}, },
"monitor_backups.go": { "monitor_backups.go": {
"func (m *Monitor) updatePVEBackupTemplateSubjectsForType(instanceName, guestType string, subjects map[string]struct{}) {", "func (m *Monitor) updatePVEBackupTemplateSubjectsForType(instanceName, guestType string, subjects map[string]struct{}) {",
+2
View File
@@ -1078,6 +1078,7 @@ type Monitor struct {
pbsClients map[string]*pbs.Client pbsClients map[string]*pbs.Client
pmgClients map[string]*pmg.Client pmgClients map[string]*pmg.Client
availabilityStatuses map[string]AvailabilityProbeStatus availabilityStatuses map[string]AvailabilityProbeStatus
availabilityByLocation map[string]map[string]AvailabilityProbeStatus
availabilityProbeTrackers map[string]availabilityProbeAssignmentTracker availabilityProbeTrackers map[string]availabilityProbeAssignmentTracker
pollProviders map[InstanceType]PollProvider pollProviders map[InstanceType]PollProvider
pollMetrics *PollMetrics pollMetrics *PollMetrics
@@ -1668,6 +1669,7 @@ func New(cfg *config.Config) (*Monitor, error) {
pbsClients: make(map[string]*pbs.Client), pbsClients: make(map[string]*pbs.Client),
pmgClients: make(map[string]*pmg.Client), pmgClients: make(map[string]*pmg.Client),
availabilityStatuses: make(map[string]AvailabilityProbeStatus), availabilityStatuses: make(map[string]AvailabilityProbeStatus),
availabilityByLocation: make(map[string]map[string]AvailabilityProbeStatus),
availabilityProbeTrackers: make(map[string]availabilityProbeAssignmentTracker), availabilityProbeTrackers: make(map[string]availabilityProbeAssignmentTracker),
pollProviders: make(map[InstanceType]PollProvider), pollProviders: make(map[InstanceType]PollProvider),
pollMetrics: getPollMetrics(), pollMetrics: getPollMetrics(),
@@ -740,7 +740,9 @@ func TestAgentlessAvailabilityTargetKindStaysCanonical(t *testing.T) {
"LinkedResourceID string `json:\"linkedResourceId,omitempty\"`", "LinkedResourceID string `json:\"linkedResourceId,omitempty\"`",
}, },
filepath.Join("..", "monitoring", "availability_poller.go"): { filepath.Join("..", "monitoring", "availability_poller.go"): {
"TargetKind string `json:\"targetKind,omitempty\"`", "TargetKind",
"string",
"`json:\"targetKind,omitempty\"`",
"TargetKind: string(target.TargetKind),", "TargetKind: string(target.TargetKind),",
"TargetKind: string(target.TargetKind),", "TargetKind: string(target.TargetKind),",
"tags = append(tags, string(target.TargetKind))", "tags = append(tags, string(target.TargetKind))",
@@ -748,7 +750,9 @@ func TestAgentlessAvailabilityTargetKindStaysCanonical(t *testing.T) {
}, },
"types.go": { "types.go": {
"Availability *AvailabilityData `json:\"availability,omitempty\"`", "Availability *AvailabilityData `json:\"availability,omitempty\"`",
"TargetKind string `json:\"targetKind,omitempty\"`", "TargetKind",
"string",
"`json:\"targetKind,omitempty\"`",
"LinkedResourceID string `json:\"linkedResourceId,omitempty\"`", "LinkedResourceID string `json:\"linkedResourceId,omitempty\"`",
"LastChecked *time.Time `json:\"lastChecked,omitempty\"`", "LastChecked *time.Time `json:\"lastChecked,omitempty\"`",
"LastSuccess *time.Time `json:\"lastSuccess,omitempty\"`", "LastSuccess *time.Time `json:\"lastSuccess,omitempty\"`",
+24
View File
@@ -1753,6 +1753,11 @@ type AvailabilityData struct {
ApplicationStatusCode int `json:"applicationStatusCode,omitempty"` ApplicationStatusCode int `json:"applicationStatusCode,omitempty"`
ApplicationFailureCode string `json:"applicationFailureCode,omitempty"` ApplicationFailureCode string `json:"applicationFailureCode,omitempty"`
ProbeAgentID string `json:"probeAgentId,omitempty"` ProbeAgentID string `json:"probeAgentId,omitempty"`
AggregateState string `json:"aggregateState,omitempty"`
Disagreement bool `json:"disagreement,omitempty"`
ExpectedLocations int `json:"expectedLocations,omitempty"`
ReportingLocations int `json:"reportingLocations,omitempty"`
Locations []AvailabilityObservationLocation `json:"locations,omitempty"`
UDPMode string `json:"udpMode,omitempty"` UDPMode string `json:"udpMode,omitempty"`
Port int `json:"port,omitempty"` Port int `json:"port,omitempty"`
Path string `json:"path,omitempty"` Path string `json:"path,omitempty"`
@@ -1777,6 +1782,25 @@ type AvailabilityData struct {
Certificate *tlsutil.CertificateObservation `json:"certificate,omitempty"` Certificate *tlsutil.CertificateObservation `json:"certificate,omitempty"`
} }
type AvailabilityObservationLocation struct {
LocationID string `json:"locationId"`
Kind string `json:"kind"`
ProbeAgentID string `json:"probeAgentId,omitempty"`
Outcome string `json:"outcome,omitempty"`
TransportOutcome string `json:"transportOutcome,omitempty"`
ApplicationOutcome string `json:"applicationOutcome,omitempty"`
ApplicationStatusCode int `json:"applicationStatusCode,omitempty"`
ApplicationFailureCode string `json:"applicationFailureCode,omitempty"`
Available bool `json:"available"`
LastChecked *time.Time `json:"lastChecked,omitempty"`
LastSuccess *time.Time `json:"lastSuccess,omitempty"`
FreshnessAt *time.Time `json:"freshnessAt,omitempty"`
LatencyMillis int64 `json:"latencyMillis,omitempty"`
ConsecutiveFailures int `json:"consecutiveFailures,omitempty"`
LastError string `json:"lastError,omitempty"`
Stale bool `json:"stale,omitempty"`
}
// K8sMetricCapabilities describes which Kubernetes metric families are available // K8sMetricCapabilities describes which Kubernetes metric families are available
// for this cluster right now based on active collection paths. // for this cluster right now based on active collection paths.
type K8sMetricCapabilities struct { type K8sMetricCapabilities struct {