Add multi-location service delivery evidence

This commit is contained in:
Pulse Test
2026-08-30 21:44:32 +01:00
parent e094a55b45
commit 16a1574a18
30 changed files with 1403 additions and 208 deletions
@@ -1,7 +1,7 @@
# Infrastructure-Aware Service Assurance Plan
Last updated: 2026-08-30
Status: ACCEPTED — SLICES A, B, AND C IMPLEMENTED
Status: ACCEPTED — SLICES A, B, C, AND D IMPLEMENTED
Governance surfaces:
- `status.json.coverage_gaps.infrastructure-aware-service-assurance`
@@ -228,7 +228,7 @@ Implementation record (2026-08-30): Slice A is delivered through the
monitoring-owned categorical history store and rollups, server-authored
configuration revisions and remote receipt timeline, the bounded
`/api/availability-history` batch contract, and the URL-owned Availability
fleet presentation. Its release proofs live in the owner contract. Slices D
fleet presentation. Its release proofs live in the owner contract. Slices E
through H remain ordered future work; acceptance of this product lane does not
imply that deferred breadth is already delivered.
@@ -269,7 +269,7 @@ re-entry. Current status and unified-resource facets preserve transport
reachability separately from typed application correctness while the overall
result remains the alert and history outcome. Legacy targets without an
explicit contract retain their previous HEAD-with-bounded-GET-fallback
semantics. Slices D through H remain ordered future work.
semantics. Slices E through H remain ordered future work.
### Slice C: Discovery-Led Assurance Onboarding
@@ -304,7 +304,7 @@ standalone endpoints, and creates one explicitly enabled availability target
with the drawer's canonical resource ID only after the operator chooses the
activation action. A machine-scoped queue supports bulk review and
evidence-bound dismiss/restore without bulk activation or guessed resource
attachment. Slices D through H remain ordered future work.
attachment. Slices E through H remain ordered future work.
### Slice D: Multi-Location Delivery Evidence
@@ -332,6 +332,23 @@ Exit conditions:
- location names and customer-identifying network details obey the existing
privacy and redaction policy.
Implementation record (2026-08-30): Slice D is delivered through a canonical
set of source-owned observation-location IDs on each saved target. The local
Pulse runtime and any selected eligible connected agents may observe the same
logical verification concurrently; monitoring retains their state, latency,
server-receipt freshness, and coverage separately, then derives one aggregate
state without duplicating the target resource. A reachable path plus a failed
or missing path is `degraded`, every current path must be unreachable before
the target is `unavailable`, and incomplete all-failed evidence remains
`unknown`. Stale agent paths become indeterminate evidence, retries remain
idempotent through the existing observation ID and configuration revision,
and only the assigned agent may author its location. Settings presents the
location set as a bounded multi-select, while the Availability fleet and
resource detail surfaces expose reporting coverage and path disagreement.
Compatibility with the former single `probeAgentId` contract is retained at
the API boundary without keeping single-agent execution as the canonical
model. Slices E through H remain ordered future work.
### Slice E: Infrastructure-Aware Incident Synthesis
Deliver deterministic incident grouping and causal narrowing over canonical
@@ -6211,6 +6211,17 @@ remain ingestible through a deterministic server fallback, but they do not
gain authority to author revision boundaries, target identity, retention, or
the service-assurance timeline.
The server may assign one target to several agent-backed observation locations,
but each agent receives and reports only the targets whose canonical
`agent:<agent-id>` location it owns. Location membership is server-authored
configuration, not agent enrollment identity: it does not grant control-plane
authority, create another agent resource, or let one reporter stand in for a
different site. Reassignment increments the target configuration revision, so
late results from the previous location set are rejected. Local Pulse execution
can coexist with assigned agent execution for the same logical target, while
license lapse removes every remote location from the effective assignment and
leaves the local path as the fail-safe execution boundary.
Observer configuration is explicit, versioned, and file-backed. It contains no
raw token values and resolves each token from a separate private absolute-path
file. Proxmox registration is also destination-scoped: the primary retains its
@@ -8808,6 +8808,18 @@ metadata, but API freshness and `lastSeenAt` projections use the server-authored
receipt time supplied by monitoring. API consumers must not substitute the
agent clock for disconnect detection or apply a second, shorter generic
Connections staleness window.
Availability target writes now use `observationLocationIds` as the canonical
bounded set. Values are source-owned IDs (`pulse:local` or
`agent:<agent-id>`), are normalized and deduplicated by the server, and every
remote member requires the external-probe entitlement plus an existing agent.
The compatibility `probeAgentId` field is accepted for older single-agent
clients and is emitted only as a compatibility hint when the selected set can
be represented by one remote path. Current status and unified-resource
transport add aggregate state, disagreement, expected/reporting counts, and a
per-location list containing only location ID, state, latency, observation and
receipt times, freshness, and a bounded failure code. Callers must consume that
server-authored aggregate rather than treating the latest reporting path as the
whole service result.
Mock availability fixtures must still behave like saved targets: `/api/connections`
reports them as availability rows, `/api/availability-targets` lists them with
probe status, and saved-test calls return the synthetic probe result instead of
@@ -1566,24 +1566,35 @@ the frontend-primitives-owned Standalone surface may read the same
`network-endpoint` projection to show current reachability, latency, check age,
and failure state without creating another monitoring provider or top-level
availability route.
Availability targets may also be assigned to a remote host agent. Reachability
outcome and the optional certificate observation travel in the same bounded
report entry, and the server clones that observation before status and resource
projection so report buffers cannot alias live state. While the
`external_probe` entitlement is active, a probe-assigned target is executed
exclusively by its assigned agent: monitoring must not schedule or run it
locally, so the check never executes twice. The assignment is effective only for
as long as the entitlement holds; on lapse the effective assignment collapses to
local and the normal poll provider resumes the target on its next planning
cycle, without a restart. Reported results are accepted only from the agent that
currently owns the target, and results for any other target or from any other
agent are dropped. Failure accounting, thresholds, and incident projection stay
server-side. The agent-authored observation time remains visible as the target's
last check, but staleness uses server receipt time so slow or fast agent clocks
cannot manufacture or conceal a disconnect. When an assigned agent stops
reporting, monitoring derives indeterminate with a stale-report explanation at
read time through the shared probe-status snapshot rather than mutating stored
state, so every availability consumer sees the same staleness verdict.
Availability execution is location-owned. Each saved target has a normalized,
deduplicated set of observation-location IDs: `pulse:local` for this Pulse
runtime and `agent:<agent-id>` for an eligible connected host agent. The legacy
single `probeAgentId` field remains an input compatibility boundary only; the
location set owns scheduling, configuration revision, assignment, status, and
resource projection. While `external_probe` is entitled, Pulse may schedule
the local path and every selected agent path for one logical target at the same
time. On entitlement lapse the effective set collapses to the local path
without a restart. An agent result is accepted only for its exact selected
location and current target revision; another agent cannot author that path.
Reachability outcome and the optional certificate observation still travel in
the same bounded report entry, and the server clones that observation before
status and resource projection so report buffers cannot alias live state.
Monitoring retains one current status per target and location, using server
receipt time for remote freshness while keeping the agent-authored observation
time as evidence metadata. A stale or disconnected agent path derives as
indeterminate at read time; slow or fast agent clocks cannot manufacture or
conceal a disconnect. Aggregation is conservative: all reachable paths are
healthy, mixed reachable and failed/unknown paths are degraded, all current
paths unreachable are unavailable, and no reachable path with incomplete or
indeterminate coverage is unknown. Only aggregate unavailability advances the
target failure threshold or authors an outage incident. A path-local failure
therefore cannot claim that the service is universally down. Single-location
targets preserve their prior status and history semantics. Multi-location
targets write only their aggregate service conclusion to the logical target's
deletable categorical history; current per-location evidence remains a status
and resource projection until a separately governed location-history schema
can preserve deletion, retention, and bounded-query semantics end to end.
Mock-mode Discovery context follows the same fixture-graph rule. Demo service
details such as detected version, config/data/log paths, Docker bind mounts,
ports, and suggested web URLs may be authored in mock fixtures, but consumers
@@ -1331,6 +1331,14 @@ alerts. Those counts may describe scale and feature adoption, but they must not
include hostnames, resource IDs, infrastructure identifiers, credentials,
prompts, chat messages, command text, action output, token values, or personal
information.
Multi-location availability evidence stays inside the same privacy floor.
Saved configuration and authenticated operator surfaces may use source-owned
location IDs and agent display names to distinguish selected paths, but
categorical history, evidence envelopes, telemetry, incidents, logs, and
Patrol context must not retain or export customer network names, addresses,
agent identity, or location labels. Durable categorical history remains keyed
only by the logical target and its aggregate conclusion; location identity is
not a new telemetry, history, or customer-identity dimension.
That same outbound usage telemetry floor now also permits content-free update
funnel counters derived from local update history inside the same rotating
30-day telemetry window: update attempts, successful updates, failed or
@@ -4670,6 +4670,18 @@ the shared resource drawer by canonical resource ID, while
already carried by that resource. History absence or failure cannot overwrite
the unified resource's current status, correlation, source ownership, or
navigation identity.
One availability resource may now carry several source-owned observation
locations without cloning the resource, relationship, or fleet row. Its
availability facet preserves the aggregate state, disagreement flag,
expected/reporting coverage, and the current state, latency, freshness, and
bounded failure reason for each path. Resource status remains active with
warning evidence when paths disagree, becomes stale/incomplete when coverage
is unknown, and becomes inactive with an outage incident only when every
current path is unreachable at the configured threshold. Frontend adapters,
REST, websocket, and mock projections must preserve this list rather than
flattening it to the most recent observer. The Availability table may summarize
the count and reporting coverage; the shared resource detail card owns the
expanded path comparison.
For HTTP/S checks with an explicit application response contract, the
availability facet preserves two source-owned facts: `transportOutcome`
describes whether the endpoint answered, while `applicationOutcome`,
+26 -20
View File
@@ -1,24 +1,29 @@
{
"version": 1,
"base_sha": "d9e9bf3679ab579be9f8c7350932008c287ad5ad",
"verified_at": "2026-08-30T20:24:19Z",
"base_sha": "e094a55b4500d2a10b6ab44b350c2fffcc55e73d",
"verified_at": "2026-08-30T20:37:10Z",
"result": "passed",
"changed_paths": [
"frontend-modern/src/components/Infrastructure/resourceDetailMappers.ts",
"frontend-modern/src/features/docker/DockerHostDrawerOverview.tsx",
"frontend-modern/src/hooks/useWorkloads.ts",
"frontend-modern/src/types/api.ts",
"frontend-modern/src/types/resource.ts"
"frontend-modern/src/api/availabilityTargets.ts",
"frontend-modern/src/components/Infrastructure/AvailabilityProbeStatusCard.tsx",
"frontend-modern/src/components/Settings/ConnectionEditor/CredentialSlots/AvailabilityTargetSlot.tsx",
"frontend-modern/src/components/Settings/availabilitySettingsModel.ts",
"frontend-modern/src/features/standalone/AvailabilityChecksTable.tsx",
"frontend-modern/src/types/resource.ts",
"frontend-modern/src/utils/availabilityProbeAgents.ts"
],
"content_sha256": {
"frontend-modern/src/components/Infrastructure/resourceDetailMappers.ts": "8b1913ff24e6f5dbf074091f2bbaff1a8c3298126515ea288b055a02c0153997",
"frontend-modern/src/features/docker/DockerHostDrawerOverview.tsx": "621af65f92c9d58312f019790ecf2eb6986fb260724e9ef53213972fd9b55579",
"frontend-modern/src/hooks/useWorkloads.ts": "85719e8c93f503843cd09c1be41c66f9995288d1280ca2eedf7d47c3443bd9a1",
"frontend-modern/src/types/api.ts": "be1b1889588085791880dbd6716955710cd66e3dc69339998f8b9ad30c0ac378",
"frontend-modern/src/types/resource.ts": "74b26ae53a507f5608a3c191f7c386cffab636e0a95aeffa02028bd8199297b0"
"frontend-modern/src/api/availabilityTargets.ts": "0a20277a8c453b753e8c09f7250722670211a33bcae2addad7d19502f921f365",
"frontend-modern/src/components/Infrastructure/AvailabilityProbeStatusCard.tsx": "ec13f364ee9c55f6859c448797fed427255999723524ff0f51cb54cc9f864ea5",
"frontend-modern/src/components/Settings/ConnectionEditor/CredentialSlots/AvailabilityTargetSlot.tsx": "622e0c7ad72c4f313c8d971295662e6549d9f13c5fbd788d3cc8803c827af176",
"frontend-modern/src/components/Settings/availabilitySettingsModel.ts": "cd03ed43bca379c8359c11c67f235d49a4a1c6b905c987a4ce404ce59443bbf6",
"frontend-modern/src/features/standalone/AvailabilityChecksTable.tsx": "3c775ad5b08023f9a2814a9dc939992db363a7e5113a351d4fb65ad0946f0d97",
"frontend-modern/src/types/resource.ts": "a4e87ae08ec69349122075ca422ced08a56e1f14294088488198c5ff8af25dba",
"frontend-modern/src/utils/availabilityProbeAgents.ts": "7f92c5fc5012bdbb322825b4a03f26fb5dc6d20d260cda8b2c2e86b66283b005"
},
"routes": [
"/browser-helper-summary-harness.html (temporary current-build Vite harness for the Docker host drawer)"
"/standalone/availability",
"/settings/monitoring/availability?add=target&targetKind=service"
],
"viewports": [
{
@@ -31,14 +36,15 @@
}
],
"states": [
"typed-helper summary mode with reduced-coverage warning and no container update management card",
"direct-runtime mode without the reduced-coverage warning and with container update management controls",
"return to typed-helper summary mode after direct-runtime mode"
"multi-location target editor with the local Pulse path and connected-agent paths, including the Community-tier remote-path entitlement boundary",
"degraded service with one reachable and one unreachable observation path, preserved as Paths disagree with 2/2 reporting",
"expanded resource evidence showing per-path state, latency, checked time, freshness, aggregate degraded state, and no universal outage",
"desktop and narrow layouts with settled data, dialog bounds, responsive navigation, and no page or dialog horizontal overflow"
],
"interactions": [
"loaded summary mode and confirmed warning visibility, update-control omission, and no desktop horizontal overflow",
"switched to direct-runtime mode and confirmed the warning disappeared and update controls appeared",
"switched back to summary mode and confirmed the warning returned and update controls were removed",
"inspected the final summary state at 390x844 and confirmed no horizontal overflow or clipped warning copy"
"opened the add service/device check flow and inspected the bounded observation-location selector without saving a mock target",
"opened the Frontend Swarm service resource details from Availability checks and expanded the multi-location evidence",
"confirmed the local path reports 9 ms while edge-london reports Unreachable and the aggregate remains Degraded",
"repeated the final route, state, clipping, and overflow checks at 1280x800 and 390x844 after rebasing onto the current main base"
]
}
@@ -67,9 +67,33 @@ export interface AvailabilityProbeStatus {
* Absent (omitempty) when the local Pulse server ran the check.
*/
probeAgentId?: string;
aggregateState?: 'healthy' | 'degraded' | 'unavailable' | 'unknown' | string;
disagreement?: boolean;
expectedLocations?: number;
reportingLocations?: number;
locations?: AvailabilityObservationLocationStatus[];
certificate?: ResourceCertificateObservation;
}
export interface AvailabilityObservationLocationStatus {
locationId: string;
kind: 'pulse' | 'agent' | string;
probeAgentId?: string;
outcome?: 'reachable' | 'unreachable' | 'indeterminate' | string;
transportOutcome?: string;
applicationOutcome?: string;
applicationStatusCode?: number;
applicationFailureCode?: string;
available: boolean;
lastChecked?: string;
lastSuccess?: string;
freshnessAt?: string;
latencyMillis?: number;
consecutiveFailures?: number;
lastError?: string;
stale?: boolean;
}
export interface AvailabilityTarget {
id: string;
configRevision?: number;
@@ -96,6 +120,9 @@ export interface AvailabilityTarget {
* string because the server decodes updates onto the existing record.
*/
probeAgentId?: string;
/** Canonical execution set. `pulse:local` names this Pulse server and
* `agent:<id>` names a connected Pulse Agent observation location. */
observationLocationIds?: string[];
http?: AvailabilityHTTPConfig;
httpSecrets?: AvailabilityHTTPSecretState;
status?: AvailabilityProbeStatus;
@@ -46,8 +46,13 @@ export function AvailabilityProbeStatusCards(props: AvailabilityProbeStatusCards
}
export function AvailabilityProbeStatusCard(props: AvailabilityProbeStatusCardProps) {
const isUp = () => props.availability.available === true;
const isDown = () => props.availability.available === false;
const aggregateState = () => props.availability.aggregateState;
const isUp = () =>
aggregateState() ? aggregateState() === 'healthy' : props.availability.available === true;
const isDegraded = () => aggregateState() === 'degraded';
const isUnknown = () => aggregateState() === 'unknown';
const isDown = () =>
aggregateState() ? aggregateState() === 'unavailable' : props.availability.available === false;
const latency = () => {
const ms = props.availability.latencyMillis;
return typeof ms === 'number' && Number.isFinite(ms) && ms > 0 ? `${Math.round(ms)}ms` : null;
@@ -129,6 +134,19 @@ export function AvailabilityProbeStatusCard(props: AvailabilityProbeStatusCardPr
if (days === 0) return `${date} (today)`;
return `${date} (${days}d)`;
};
const locationLabel = (location: NonNullable<ResourceAvailabilityMeta['locations']>[number]) =>
location.kind === 'pulse'
? 'This Pulse server'
: location.probeAgentId || location.locationId.replace(/^agent:/, '') || 'Pulse Agent';
const locationResult = (location: NonNullable<ResourceAvailabilityMeta['locations']>[number]) => {
if (location.stale) return 'No recent report';
if (!location.lastChecked) return 'Awaiting evidence';
if (location.outcome === 'reachable') {
return location.latencyMillis ? `${Math.round(location.latencyMillis)} ms` : 'Reachable';
}
if (location.outcome === 'unreachable') return 'Unreachable';
return 'Unknown';
};
return (
<InfoCardFrame data-testid="availability-probe-status">
@@ -145,11 +163,22 @@ export function AvailabilityProbeStatusCard(props: AvailabilityProbeStatusCardPr
'bg-emerald-100 text-emerald-700 dark:bg-emerald-900/40 dark:text-emerald-300':
isFreshUp(),
'bg-red-100 text-red-700 dark:bg-red-900/40 dark:text-red-300': isDown() && !isStale(),
'bg-amber-100 text-amber-700 dark:bg-amber-900/40 dark:text-amber-300': isStale(),
'bg-base-200 text-muted': !isUp() && !isDown() && !isStale(),
'bg-amber-100 text-amber-700 dark:bg-amber-900/40 dark:text-amber-300':
isStale() || isDegraded(),
'bg-base-200 text-muted': isUnknown() || (!isUp() && !isDown() && !isStale()),
}}
>
{isStale() ? 'Stale' : isUp() ? 'Up' : isDown() ? 'Down' : 'Not checked'}
{isDegraded()
? 'Paths disagree'
: isUnknown()
? 'Coverage incomplete'
: isStale()
? 'Stale'
: isUp()
? 'Up'
: isDown()
? 'Down'
: 'Not checked'}
</span>
</div>
<div class="space-y-1.5 text-[11px]">
@@ -171,6 +200,51 @@ export function AvailabilityProbeStatusCard(props: AvailabilityProbeStatusCardPr
</Show>
}
/>
<Show when={(props.availability.locations?.length ?? 0) > 0}>
<div
class="mt-2 space-y-1.5 border-t border-border pt-2"
data-observation-location-results
>
<div class="flex items-center justify-between gap-2 text-[10px] uppercase tracking-wide text-muted">
<span>Observation paths</span>
<span>
{props.availability.reportingLocations ?? 0}/
{props.availability.expectedLocations ?? props.availability.locations?.length ?? 0}{' '}
reporting
</span>
</div>
<For each={props.availability.locations}>
{(location) => (
<div class="flex items-start justify-between gap-3 rounded bg-base-200/60 px-2 py-1.5">
<div class="min-w-0">
<div
class="truncate font-medium text-base-content"
title={locationLabel(location)}
>
{locationLabel(location)}
</div>
<Show when={formatRelativeTime(location.lastChecked)}>
{(checked) => <div class="text-[10px] text-muted">checked {checked()}</div>}
</Show>
</div>
<span
class="shrink-0 font-medium"
classList={{
'text-emerald-600 dark:text-emerald-300':
location.outcome === 'reachable' && !location.stale,
'text-red-600 dark:text-red-300':
location.outcome === 'unreachable' && !location.stale,
'text-amber-600 dark:text-amber-300': Boolean(location.stale),
'text-muted': !location.outcome || location.outcome === 'indeterminate',
}}
>
{locationResult(location)}
</span>
</div>
)}
</For>
</div>
</Show>
<InfoCardKeyValueRow label="Method" value={method()} valueTitle={targetAddr()} />
<InfoCardKeyValueRow
label="Target"
@@ -0,0 +1,63 @@
import { render, screen, within } from '@solidjs/testing-library';
import { describe, expect, it } from 'vitest';
import { AvailabilityProbeStatusCard } from '../AvailabilityProbeStatusCard';
describe('AvailabilityProbeStatusCard', () => {
it('shows disagreement and keeps every observation path visible', () => {
render(() => (
<AvailabilityProbeStatusCard
availability={{
targetId: 'customer-api',
address: 'api.service.local',
protocol: 'https',
enabled: true,
available: true,
aggregateState: 'degraded',
disagreement: true,
expectedLocations: 3,
reportingLocations: 2,
locations: [
{
locationId: 'pulse:local',
kind: 'pulse',
outcome: 'reachable',
available: true,
latencyMillis: 8,
lastChecked: '2026-08-30T19:00:00Z',
},
{
locationId: 'agent:edge-a',
kind: 'agent',
probeAgentId: 'edge-a',
outcome: 'unreachable',
available: false,
lastChecked: '2026-08-30T19:00:00Z',
},
{
locationId: 'agent:edge-b',
kind: 'agent',
probeAgentId: 'edge-b',
outcome: 'indeterminate',
available: false,
stale: true,
lastChecked: '2026-08-30T18:00:00Z',
},
],
}}
/>
));
expect(screen.getByText('Paths disagree')).toBeInTheDocument();
const paths = screen.getByText('Observation paths').parentElement?.parentElement;
expect(paths).not.toBeNull();
const scope = within(paths!);
expect(scope.getByText('2/3 reporting')).toBeInTheDocument();
expect(scope.getByText('This Pulse server')).toBeInTheDocument();
expect(scope.getByText('8 ms')).toBeInTheDocument();
expect(scope.getByText('edge-a')).toBeInTheDocument();
expect(scope.getByText('Unreachable')).toBeInTheDocument();
expect(scope.getByText('edge-b')).toBeInTheDocument();
expect(scope.getByText('No recent report')).toBeInTheDocument();
});
});
@@ -33,13 +33,14 @@ import { getPreferredInfrastructureDisplayName } from '@/utils/resourceIdentity'
import { getResourceTypeLabel } from '@/utils/resourceTypePresentation';
import {
EXTERNAL_PROBE_FEATURE,
LOCAL_PROBE_AGENT_LABEL,
buildProbeAgentOptions,
AGENT_OBSERVATION_LOCATION_PREFIX,
LOCAL_OBSERVATION_LOCATION_ID,
agentIdFromObservationLocation,
buildObservationLocationOptions,
getExternalProbeGateBody,
getExternalProbeGateTitle,
getExternalProbeLockedHelpText,
isExternalProbeLicenseError,
isProbeAgentMissing,
} from '@/utils/availabilityProbeAgents';
import { hasFeature, loadRuntimeCapabilities, runtimeCapabilitiesLoaded } from '@/stores/license';
import { getUpgradeActionDestination } from '@/stores/licenseCommercial';
@@ -58,7 +59,7 @@ interface AvailabilityForm {
udpRequest: string;
udpExpectedResponse: string;
linkedResourceId: string;
probeAgentId: string;
observationLocationIds: string[];
enabled: boolean;
pollIntervalSeconds: string;
timeoutMillis: string;
@@ -118,7 +119,7 @@ const newAvailabilityForm = (
udpRequest: '',
udpExpectedResponse: '',
linkedResourceId: '',
probeAgentId: '',
observationLocationIds: [LOCAL_OBSERVATION_LOCATION_ID],
enabled: true,
pollIntervalSeconds: '60',
timeoutMillis: '2000',
@@ -159,7 +160,11 @@ const formFromTarget = (target: AvailabilityTarget): AvailabilityForm => {
udpRequest: target.udpRequest ?? '',
udpExpectedResponse: target.udpExpectedResponse ?? '',
linkedResourceId: target.linkedResourceId ?? '',
probeAgentId: target.probeAgentId ?? '',
observationLocationIds: target.observationLocationIds?.length
? [...target.observationLocationIds]
: target.probeAgentId
? [`${AGENT_OBSERVATION_LOCATION_PREFIX}${target.probeAgentId}`]
: [LOCAL_OBSERVATION_LOCATION_ID],
enabled: target.enabled ?? true,
pollIntervalSeconds: String(target.pollIntervalSeconds ?? 60),
timeoutMillis: String(target.timeoutMillis ?? 2000),
@@ -247,7 +252,11 @@ const payloadFromForm = (form: AvailabilityForm): AvailabilityTarget => {
// Always serialized, never `undefined`: the server decodes updates onto the
// existing record, so an explicit empty string is what clears a probe
// assignment and moves the check back to the local Pulse server.
probeAgentId: form.probeAgentId.trim(),
probeAgentId:
form.observationLocationIds.length === 1
? agentIdFromObservationLocation(form.observationLocationIds[0])
: '',
observationLocationIds: [...form.observationLocationIds],
enabled: form.enabled,
pollIntervalSeconds: parsePositiveInt(form.pollIntervalSeconds),
timeoutMillis: parsePositiveInt(form.timeoutMillis),
@@ -348,10 +357,11 @@ export const AvailabilityTargetSlot: Component<AvailabilityTargetSlotProps> = (p
// stale cached capability set still lands on the upgrade gate.
const [probeLicenseRejected, setProbeLicenseRejected] = createSignal(false);
const probeAgentOptions = createMemo(() => buildProbeAgentOptions(resources()));
const probeAgentMissing = createMemo(() =>
isProbeAgentMissing(probeAgentOptions(), form().probeAgentId),
);
const observationLocationOptions = createMemo(() => buildObservationLocationOptions(resources()));
const missingObservationLocations = createMemo(() => {
const known = new Set(observationLocationOptions().map((option) => option.id));
return form().observationLocationIds.filter((locationId) => !known.has(locationId));
});
const externalProbeLicensed = createMemo(
() =>
!probeLicenseRejected() && runtimeCapabilitiesLoaded() && hasFeature(EXTERNAL_PROBE_FEATURE),
@@ -378,6 +388,15 @@ export const AvailabilityTargetSlot: Component<AvailabilityTargetSlotProps> = (p
setTestResult(null);
};
const toggleObservationLocation = (locationId: string, selected: boolean) => {
const current = form().observationLocationIds;
const next = selected
? [...new Set([...current, locationId])]
: current.filter((candidate) => candidate !== locationId);
if (next.length === 0) return;
updateForm({ observationLocationIds: next });
};
const updateHTTPHeader = (index: number, patch: Partial<AvailabilityHTTPHeaderForm>) => {
updateForm({
httpHeaders: form().httpHeaders.map((header, headerIndex) =>
@@ -490,7 +509,12 @@ export const AvailabilityTargetSlot: Component<AvailabilityTargetSlotProps> = (p
}
props.onSaved();
} catch (err) {
if (payload.probeAgentId && isExternalProbeLicenseError(err)) {
if (
payload.observationLocationIds?.some((locationId) =>
locationId.startsWith(AGENT_OBSERVATION_LOCATION_PREFIX),
) &&
isExternalProbeLicenseError(err)
) {
setProbeLicenseRejected(true);
setError(getExternalProbeGateBody());
return;
@@ -610,28 +634,67 @@ export const AvailabilityTargetSlot: Component<AvailabilityTargetSlotProps> = (p
)}
</For>
</FormSelect>
<div class="space-y-3 sm:col-span-2">
<FormSelect
label="Run from"
value={form().probeAgentId}
disabled={externalProbeLocked()}
onChange={(event) => updateForm({ probeAgentId: event.currentTarget.value })}
help={
externalProbeLocked()
? getExternalProbeLockedHelpText()
: 'Run this check from the Pulse server, or hand it to a connected Pulse Agent host so it is probed from that network.'
}
>
<option value="">{LOCAL_PROBE_AGENT_LABEL}</option>
<Show when={probeAgentMissing()}>
<option value={form().probeAgentId}>
{form().probeAgentId} (not currently connected)
</option>
</Show>
<For each={probeAgentOptions()}>
{(option) => <option value={option.id}>{option.label}</option>}
<fieldset class="space-y-3 sm:col-span-2">
<legend class={formLabel}>Observation locations</legend>
<p class={formHelpText}>
One logical check can be observed from several networks. Pulse keeps each path separate
and shows disagreement instead of turning one path failure into a universal outage.
</p>
<div class="grid gap-2 sm:grid-cols-2" data-observation-location-picker>
<For each={observationLocationOptions()}>
{(option) => {
const selected = () => form().observationLocationIds.includes(option.id);
const locked = () => option.kind === 'agent' && externalProbeLocked();
return (
<label class="flex items-start gap-3 rounded-md border border-border bg-surface px-3 py-3 text-sm">
<input
class={formCheckbox}
type="checkbox"
checked={selected()}
disabled={
locked() || (selected() && form().observationLocationIds.length === 1)
}
onChange={(event) =>
toggleObservationLocation(option.id, event.currentTarget.checked)
}
/>
<span class="min-w-0">
<span class="block font-medium text-foreground">{option.label}</span>
<span class="block text-xs text-muted">
{option.kind === 'pulse'
? 'Local Pulse path'
: 'Connected Pulse Agent path'}
</span>
</span>
</label>
);
}}
</For>
</FormSelect>
<For each={missingObservationLocations()}>
{(locationId) => (
<label class="flex items-start gap-3 rounded-md border border-warning/40 bg-warning/5 px-3 py-3 text-sm">
<input
class={formCheckbox}
type="checkbox"
checked
disabled={form().observationLocationIds.length === 1}
onChange={(event) =>
toggleObservationLocation(locationId, event.currentTarget.checked)
}
/>
<span class="min-w-0">
<span class="block font-medium text-foreground">
{agentIdFromObservationLocation(locationId) || locationId}
</span>
<span class="block text-xs text-warning">Not currently connected</span>
</span>
</label>
)}
</For>
</div>
<Show when={externalProbeLocked()}>
<p class={formHelpText}>{getExternalProbeLockedHelpText()}</p>
</Show>
<Show when={externalProbeLocked()}>
<div class="rounded-md border border-border bg-surface-alt p-4">
<FeatureGateSection
@@ -642,7 +705,7 @@ export const AvailabilityTargetSlot: Component<AvailabilityTargetSlotProps> = (p
/>
</div>
</Show>
</div>
</fieldset>
<Show when={form().protocol !== 'icmp'}>
<label class={formField}>
<span class={formLabel}>Port</span>
@@ -424,22 +424,25 @@ describe('AvailabilityTargetSlot', () => {
resourceMocks.resources = [agentHostResource('host-edge-01', 'Edge 01')];
render(() => <AvailabilityTargetSlot onCancel={vi.fn()} onSaved={vi.fn()} />);
const runFrom = screen.getByLabelText('Run from') as HTMLSelectElement;
expect(runFrom).not.toBeDisabled();
expect(screen.getByRole('option', { name: 'This Pulse server' })).toBeInTheDocument();
expect(screen.getByRole('option', { name: 'Edge 01' })).toBeInTheDocument();
const localLocation = screen.getByRole('checkbox', { name: /This Pulse server/ });
const edgeLocation = screen.getByRole('checkbox', { name: /Edge 01/ });
expect(localLocation).toBeChecked();
expect(edgeLocation).not.toBeDisabled();
expect(screen.queryByRole('link', { name: 'View plans' })).not.toBeInTheDocument();
fireEvent.input(screen.getByLabelText('Name'), { target: { value: 'Remote MQTT' } });
fireEvent.input(screen.getByPlaceholderText('service.local'), {
target: { value: 'mqtt.remote.local' },
});
fireEvent.change(runFrom, { target: { value: 'host-edge-01' } });
fireEvent.click(edgeLocation);
fireEvent.click(screen.getByRole('button', { name: 'Add service/device check' }));
await waitFor(() =>
expect(mockedCreate).toHaveBeenCalledWith(
expect.objectContaining({ probeAgentId: 'host-edge-01' }),
expect.objectContaining({
probeAgentId: '',
observationLocationIds: ['pulse:local', 'agent:host-edge-01'],
}),
),
);
});
@@ -450,9 +453,8 @@ describe('AvailabilityTargetSlot', () => {
render(() => <AvailabilityTargetSlot onCancel={vi.fn()} onSaved={vi.fn()} />);
// Discoverability is the point of the Pro gate: the control stays visible.
const runFrom = screen.getByLabelText('Run from') as HTMLSelectElement;
expect(runFrom).toBeDisabled();
expect(screen.getByRole('option', { name: 'Edge 01' })).toBeInTheDocument();
const edgeLocation = screen.getByRole('checkbox', { name: /Edge 01/ });
expect(edgeLocation).toBeDisabled();
expect(screen.getByRole('heading', { name: 'External Probes' })).toBeInTheDocument();
expect(screen.getByRole('link', { name: 'View plans' })).toHaveAttribute(
'href',
@@ -471,7 +473,12 @@ describe('AvailabilityTargetSlot', () => {
fireEvent.click(screen.getByRole('button', { name: 'Add service/device check' }));
await waitFor(() =>
expect(mockedCreate).toHaveBeenCalledWith(expect.objectContaining({ probeAgentId: '' })),
expect(mockedCreate).toHaveBeenCalledWith(
expect.objectContaining({
probeAgentId: '',
observationLocationIds: ['pulse:local'],
}),
),
);
expect(screen.queryByRole('alert')).not.toBeInTheDocument();
});
@@ -501,14 +508,16 @@ describe('AvailabilityTargetSlot', () => {
<AvailabilityTargetSlot editingTargetId="target-1" onCancel={vi.fn()} onSaved={vi.fn()} />
));
await waitFor(() => expect(screen.getByLabelText('Run from')).toHaveValue('host-edge-01'));
await waitFor(() => expect(screen.getByRole('checkbox', { name: /Edge 01/ })).toBeChecked());
fireEvent.change(screen.getByLabelText('Run from'), { target: { value: '' } });
fireEvent.click(screen.getByRole('checkbox', { name: /This Pulse server/ }));
fireEvent.click(screen.getByRole('checkbox', { name: /Edge 01/ }));
fireEvent.click(screen.getByRole('button', { name: 'Save target' }));
await waitFor(() => expect(mockedUpdate).toHaveBeenCalled());
const [, payload] = mockedUpdate.mock.calls.at(-1)!;
expect(payload.probeAgentId).toBe('');
expect(payload.observationLocationIds).toEqual(['pulse:local']);
expect(Object.prototype.hasOwnProperty.call(payload, 'probeAgentId')).toBe(true);
});
@@ -529,11 +538,9 @@ describe('AvailabilityTargetSlot', () => {
));
await waitFor(() =>
expect(
screen.getByRole('option', { name: 'host-gone (not currently connected)' }),
).toBeInTheDocument(),
expect(screen.getByRole('checkbox', { name: /host-gone/ })).toBeChecked(),
);
expect(screen.getByLabelText('Run from')).toHaveValue('host-gone');
expect(screen.getByText('Not currently connected')).toBeInTheDocument();
});
it('falls back to the upgrade gate when the server answers 402 license_required', async () => {
@@ -551,13 +558,13 @@ describe('AvailabilityTargetSlot', () => {
fireEvent.input(screen.getByPlaceholderText('service.local'), {
target: { value: 'mqtt.remote.local' },
});
fireEvent.change(screen.getByLabelText('Run from'), { target: { value: 'host-edge-01' } });
fireEvent.click(screen.getByRole('checkbox', { name: /Edge 01/ }));
fireEvent.click(screen.getByRole('button', { name: 'Add service/device check' }));
await waitFor(() =>
expect(screen.getByRole('heading', { name: 'External Probes' })).toBeInTheDocument(),
);
expect(screen.getByLabelText('Run from')).toBeDisabled();
expect(screen.getByRole('checkbox', { name: /Edge 01/ })).toBeDisabled();
expect(screen.getByRole('link', { name: 'View plans' })).toBeInTheDocument();
});
});
@@ -108,6 +108,14 @@ export function getAvailabilityTargetStatusLabel(target: AvailabilityTarget): st
if (!target.enabled) return 'Paused';
const status = target.status;
if (!status) return 'Not checked yet';
if (status.aggregateState === 'degraded') return 'Observation paths disagree';
if (status.aggregateState === 'unknown') {
return `${status.reportingLocations ?? 0}/${status.expectedLocations ?? status.locations?.length ?? 0} locations reporting`;
}
if (status.aggregateState === 'unavailable') return 'Unavailable from all locations';
if (status.aggregateState === 'healthy' && (status.expectedLocations ?? 0) > 1) {
return `Available from all ${status.expectedLocations} locations`;
}
// A probe-assigned check whose agent stopped reporting derives to
// indeterminate at read time. It shares the warning treatment with the UDP
// open-or-filtered case but needs its own copy.
@@ -124,6 +132,12 @@ export function getAvailabilityTargetStatusLabel(target: AvailabilityTarget): st
export function getAvailabilityTargetStatusClass(target: AvailabilityTarget): string {
if (!target.enabled) return 'bg-surface-alt text-muted';
if (!target.status) return 'bg-sky-100 text-sky-700 dark:bg-sky-900 dark:text-sky-300';
if (target.status.aggregateState === 'degraded' || target.status.aggregateState === 'unknown') {
return 'bg-amber-100 text-amber-700 dark:bg-amber-900 dark:text-amber-300';
}
if (target.status.aggregateState === 'unavailable') {
return 'bg-rose-100 text-rose-700 dark:bg-rose-900 dark:text-rose-300';
}
if (target.status.outcome === 'indeterminate') {
return 'bg-amber-100 text-amber-700 dark:bg-amber-900 dark:text-amber-300';
}
@@ -141,6 +155,9 @@ export function getAvailabilityTargetProbeSourceLabel(
target: AvailabilityTarget,
probeAgentOptions: readonly ProbeAgentOption[],
): string | null {
const locationCount =
target.status?.locations?.length ?? target.observationLocationIds?.length ?? 0;
if (locationCount > 1) return `${locationCount} observation locations`;
return getProbeSourceChipLabel(probeAgentOptions, target.status?.probeAgentId);
}
@@ -283,10 +283,12 @@ export const AvailabilityChecksTable: Component<{
const result = () => probe()?.resultLabel ?? indicator().label;
const target = () => formatTarget(check);
const probeSource = () =>
getProbeSourceChipLabel(
props.probeAgentOptions ?? [],
availability()?.probeAgentId,
);
(availability()?.locations?.length ?? 0) > 1
? `${availability()?.locations?.length} locations · ${availability()?.reportingLocations ?? 0}/${availability()?.expectedLocations ?? availability()?.locations?.length ?? 0} reporting`
: getProbeSourceChipLabel(
props.probeAgentOptions ?? [],
availability()?.probeAgentId,
);
const detailRowId = () => drawer.detailRowId(check);
const isExpanded = () => drawer.isExpanded(check);
+24
View File
@@ -1532,6 +1532,11 @@ export interface ResourceAvailabilityMeta {
* on the Pulse server itself.
*/
probeAgentId?: string;
aggregateState?: 'healthy' | 'degraded' | 'unavailable' | 'unknown' | string;
disagreement?: boolean;
expectedLocations?: number;
reportingLocations?: number;
locations?: ResourceAvailabilityObservationLocation[];
correlationState?: 'attached' | 'standalone' | 'ambiguous' | 'unresolved';
correlationRule?: string;
correlationReason?: string;
@@ -1539,6 +1544,25 @@ export interface ResourceAvailabilityMeta {
evidence?: EvidenceEnvelope;
}
export interface ResourceAvailabilityObservationLocation {
locationId: string;
kind: 'pulse' | 'agent' | string;
probeAgentId?: string;
outcome?: string;
transportOutcome?: string;
applicationOutcome?: string;
applicationStatusCode?: number;
applicationFailureCode?: string;
available?: boolean;
lastChecked?: string;
lastSuccess?: string;
freshnessAt?: string;
latencyMillis?: number;
consecutiveFailures?: number;
lastError?: string;
stale?: boolean;
}
export type ResourceCertificateTrustStatus =
'trusted' | 'self-signed' | 'untrusted' | 'expired' | 'not-yet-valid';
@@ -23,6 +23,8 @@ export const EXTERNAL_PROBE_FEATURE = 'external_probe';
/** Wire value for "run this check on the Pulse server itself". */
export const LOCAL_PROBE_AGENT_VALUE = '';
export const LOCAL_OBSERVATION_LOCATION_ID = 'pulse:local';
export const AGENT_OBSERVATION_LOCATION_PREFIX = 'agent:';
/** Default option label for the local Pulse server. */
export const LOCAL_PROBE_AGENT_LABEL = 'This Pulse server';
@@ -40,6 +42,23 @@ export interface ProbeAgentOption {
label: string;
}
export interface ObservationLocationOption {
id: string;
label: string;
kind: 'pulse' | 'agent';
agentId?: string;
}
export const observationLocationIdForAgent = (agentId: string): string =>
`${AGENT_OBSERVATION_LOCATION_PREFIX}${agentId.trim()}`;
export const agentIdFromObservationLocation = (locationId?: string | null): string => {
const normalized = (locationId ?? '').trim();
return normalized.startsWith(AGENT_OBSERVATION_LOCATION_PREFIX)
? normalized.slice(AGENT_OBSERVATION_LOCATION_PREFIX.length).trim()
: '';
};
/**
* Build the selectable probe agent hosts from unified resources.
*
@@ -62,6 +81,32 @@ export function buildProbeAgentOptions(resources: readonly Resource[]): ProbeAge
return [...byId.values()].sort((left, right) => left.label.localeCompare(right.label));
}
export function buildObservationLocationOptions(
resources: readonly Resource[],
): ObservationLocationOption[] {
return [
{ id: LOCAL_OBSERVATION_LOCATION_ID, label: LOCAL_PROBE_AGENT_LABEL, kind: 'pulse' },
...buildProbeAgentOptions(resources).map((option) => ({
id: observationLocationIdForAgent(option.id),
label: option.label,
kind: 'agent' as const,
agentId: option.id,
})),
];
}
export function getObservationLocationLabel(
options: readonly ObservationLocationOption[],
locationId?: string | null,
): string {
const normalized = (locationId ?? '').trim();
if (!normalized || normalized === LOCAL_OBSERVATION_LOCATION_ID) return LOCAL_PROBE_AGENT_LABEL;
return (
options.find((option) => option.id === normalized)?.label ??
(agentIdFromObservationLocation(normalized) || normalized)
);
}
/**
* Resolve a probe agent id to its display name, falling back to the raw id when
* the host is not currently in the list.
+32 -6
View File
@@ -3,6 +3,7 @@ package api
import (
"context"
"encoding/json"
"io"
"net/http"
"strings"
"time"
@@ -78,7 +79,8 @@ func (f availabilityFeatureResolverFunc) FeatureService(ctx context.Context) lic
// Unassigned targets are the community behaviour and must never consult the
// license path.
func (h *AvailabilityHandlers) requireProbeAssignment(w http.ResponseWriter, r *http.Request, target config.AvailabilityTarget) bool {
if strings.TrimSpace(target.ProbeAgentID) == "" {
agentIDs := target.AssignedProbeAgentIDs()
if len(agentIDs) == 0 {
return true
}
if h == nil || h.licenseResolver == nil {
@@ -94,10 +96,12 @@ func (h *AvailabilityHandlers) requireProbeAssignment(w http.ResponseWriter, r *
WriteLicenseRequired(w, featureExternalProbeValue, err.Error())
return false
}
if !h.probeAgentExists(r.Context(), target.ProbeAgentID) {
writeErrorResponse(w, http.StatusBadRequest, "unknown_probe_agent",
"Probe agent "+target.ProbeAgentID+" is not a registered host agent", nil)
return false
for _, agentID := range agentIDs {
if !h.probeAgentExists(r.Context(), agentID) {
writeErrorResponse(w, http.StatusBadRequest, "unknown_probe_agent",
"Observation location agent "+agentID+" is not a registered host agent", nil)
return false
}
}
return true
}
@@ -384,10 +388,31 @@ func decodeAvailabilityTargetRequest(w http.ResponseWriter, r *http.Request, bas
r.Body = http.MaxBytesReader(w, r.Body, 16*1024)
defer r.Body.Close()
target := cloneAvailabilityTarget(base)
if err := json.NewDecoder(r.Body).Decode(&target); err != nil {
body, err := io.ReadAll(r.Body)
if err != nil || json.Unmarshal(body, &target) != nil {
writeErrorResponse(w, http.StatusBadRequest, "invalid_request", "Invalid JSON body", nil)
return config.AvailabilityTarget{}, false
}
var compatibility struct {
ProbeAgentID *string `json:"probeAgentId"`
ObservationLocationIDs *[]string `json:"observationLocationIds"`
}
if json.Unmarshal(body, &compatibility) == nil && compatibility.ObservationLocationIDs == nil && compatibility.ProbeAgentID != nil {
agentID := strings.TrimSpace(*compatibility.ProbeAgentID)
if agentID == "" {
target.ObservationLocationIDs = []string{config.AvailabilityObservationLocationLocal}
} else {
target.ObservationLocationIDs = []string{config.AvailabilityAgentObservationLocationID(agentID)}
}
} else if compatibility.ProbeAgentID != nil && compatibility.ObservationLocationIDs != nil {
agentID := strings.TrimSpace(*compatibility.ProbeAgentID)
if agentID != "" && len(*compatibility.ObservationLocationIDs) == 1 {
// Full-object pre-location clients may echo the canonical field added by
// the server while editing only probeAgentId. Honor that single-source
// edit; multi-location clients send a set the legacy field cannot express.
target.ObservationLocationIDs = []string{config.AvailabilityAgentObservationLocationID(agentID)}
}
}
return target, true
}
@@ -487,6 +512,7 @@ func sameAvailabilityHTTPOrigin(previous, next config.AvailabilityTarget) bool {
func cloneAvailabilityTarget(target config.AvailabilityTarget) config.AvailabilityTarget {
clone := target
clone.ObservationLocationIDs = append([]string(nil), target.ObservationLocationIDs...)
if target.HTTP == nil {
return clone
}
@@ -154,6 +154,42 @@ func TestAvailabilityHandlersAcceptProbeAssignmentWithLicense(t *testing.T) {
}
}
func TestAvailabilityHandlersAcceptSeveralObservationLocationsOnOneTarget(t *testing.T) {
persistence := config.NewConfigPersistence(t.TempDir())
monitor := monitorWithHostAgent(t, "edge-a")
if _, err := monitor.ApplyHostReport(agentshost.Report{
Agent: agentshost.AgentInfo{ID: "edge-b-agent", Version: "6.1.1", IntervalSeconds: 30},
Host: agentshost.HostInfo{ID: "edge-b", MachineID: "edge-b", Hostname: "edge-b", Platform: "linux"},
Timestamp: time.Now().UTC(),
}, &config.APITokenRecord{ID: "token-edge-b", Name: "edge-b"}); err != nil {
t.Fatalf("ApplyHostReport() error = %v", err)
}
handler := NewAvailabilityHandlers(
func(context.Context) *config.ConfigPersistence { return persistence },
func(context.Context) *monitoring.Monitor { return monitor },
stubFeatureResolver(featureExternalProbeValue),
)
target := config.AvailabilityTarget{
Name: "Customer API",
Address: "api.service.local",
Protocol: config.AvailabilityProbeHTTPS,
Enabled: true,
ObservationLocationIDs: []string{config.AvailabilityObservationLocationLocal, "agent:edge-a", "agent:edge-b"},
}
rec := httptest.NewRecorder()
handler.HandleAdd(rec, httptest.NewRequest(http.MethodPost, "/api/availability-targets", availabilityRequestBody(t, target)))
if rec.Code != http.StatusCreated {
t.Fatalf("HandleAdd status = %d, want 201; body=%s", rec.Code, rec.Body.String())
}
var created config.AvailabilityTarget
if err := json.NewDecoder(rec.Body).Decode(&created); err != nil {
t.Fatalf("decode created target: %v", err)
}
if len(created.ObservationLocationIDs) != 3 || created.ProbeAgentID != "" {
t.Fatalf("created target = %+v, want one logical target with three locations", created)
}
}
func TestAvailabilityHandlersRejectUnknownProbeAgent(t *testing.T) {
persistence := config.NewConfigPersistence(t.TempDir())
monitor := monitorWithHostAgent(t, "probe-host")
+6
View File
@@ -1497,6 +1497,12 @@ func deriveAvailabilityConnectionState(target config.AvailabilityTarget, status
if !target.Enabled {
return ConnectionStatePaused, "paused by user", lastSeen, lastError
}
if status.AggregateState == monitoring.AvailabilityAggregateUnknown {
return ConnectionStateStale, "observation coverage is incomplete", lastSeen, lastError
}
if status.AggregateState == monitoring.AvailabilityAggregateDegraded {
return ConnectionStateActive, "observation locations disagree", lastSeen, lastError
}
if status.LastChecked.IsZero() {
return ConnectionStatePending, "awaiting first probe", nil, nil
}
+40 -3
View File
@@ -274,7 +274,7 @@ func mockAvailabilityTestResponse(targetID string) (availabilityTestResponse, bo
}
func mockAvailabilityConfigTarget(target mock.AvailabilityTargetFixture) config.AvailabilityTarget {
return config.NormalizeAvailabilityTarget(config.AvailabilityTarget{
configTarget := config.AvailabilityTarget{
ID: target.ID,
Name: target.Name,
TargetKind: config.AvailabilityTargetKind(target.TargetKind),
@@ -286,11 +286,18 @@ func mockAvailabilityConfigTarget(target mock.AvailabilityTargetFixture) config.
PollIntervalSecs: target.PollIntervalSecs,
TimeoutMillis: target.TimeoutMillis,
FailureThreshold: target.FailureThreshold,
})
}
if target.ID == "mock-availability-docker-frontend-service" {
configTarget.ObservationLocationIDs = []string{
config.AvailabilityObservationLocationLocal,
config.AvailabilityAgentObservationLocationID("edge-london"),
}
}
return config.NormalizeAvailabilityTarget(configTarget)
}
func mockAvailabilityProbeStatus(fixture mock.AvailabilityFixture, target config.AvailabilityTarget) monitoring.AvailabilityProbeStatus {
return monitoring.AvailabilityProbeStatus{
status := monitoring.AvailabilityProbeStatus{
TargetID: target.ID,
Name: target.DisplayName(),
TargetKind: string(target.TargetKind),
@@ -305,6 +312,36 @@ func mockAvailabilityProbeStatus(fixture mock.AvailabilityFixture, target config
LastError: fixture.LastError,
FailureThreshold: target.EffectiveFailureThreshold(),
}
if target.ID == "mock-availability-docker-frontend-service" {
status.AggregateState = monitoring.AvailabilityAggregateDegraded
status.Disagreement = true
status.ExpectedLocations = 2
status.ReportingLocations = 2
status.Locations = []monitoring.AvailabilityObservationLocationStatus{
{
LocationID: config.AvailabilityObservationLocationLocal,
Kind: "pulse",
Outcome: string(monitoring.AvailabilityProbeReachable),
Available: true,
LastChecked: fixture.LastChecked,
LastSuccess: fixture.LastSuccess,
FreshnessAt: fixture.LastChecked,
LatencyMillis: fixture.LatencyMillis,
},
{
LocationID: config.AvailabilityAgentObservationLocationID("edge-london"),
Kind: "agent",
ProbeAgentID: "edge-london",
Outcome: string(monitoring.AvailabilityProbeUnreachable),
Available: false,
LastChecked: fixture.LastChecked,
FreshnessAt: fixture.LastChecked,
ConsecutiveFailures: 2,
LastError: "connection timed out",
},
}
}
return status
}
func mockPlatformTimePointer(value time.Time) *time.Time {
+146 -9
View File
@@ -5,6 +5,7 @@ import (
"net"
"net/url"
"reflect"
"sort"
"strconv"
"strings"
@@ -12,12 +13,14 @@ import (
)
const (
DefaultAvailabilityPollIntervalSecs = 60
DefaultAvailabilityTimeoutMillis = 2000
DefaultAvailabilityFailureThreshold = 2
DefaultCertificateExpiryWarningDays = 30
MaxAvailabilityHTTPBodyBytes = 8192
MaxAvailabilityHTTPResponseBytes = 65536
DefaultAvailabilityPollIntervalSecs = 60
DefaultAvailabilityTimeoutMillis = 2000
DefaultAvailabilityFailureThreshold = 2
DefaultCertificateExpiryWarningDays = 30
MaxAvailabilityHTTPBodyBytes = 8192
MaxAvailabilityHTTPResponseBytes = 65536
AvailabilityObservationLocationLocal = "pulse:local"
AvailabilityObservationLocationAgentPrefix = "agent:"
)
type AvailabilityHTTPMethod string
@@ -125,9 +128,14 @@ type AvailabilityTarget struct {
CertificateMonitoringDisabled bool `json:"certificateMonitoringDisabled,omitempty"`
CertificateExpiryWarningDays int `json:"certificateExpiryWarningDays,omitempty"`
// ProbeAgentID assigns execution to a remote host agent. Empty means the
// check runs from the local Pulse instance. Agent existence is validated at
// the API layer, not here, because config has no view of monitor state.
// check runs from the local Pulse instance. It remains a compatibility field
// for pre-location clients; ObservationLocationIDs is canonical.
ProbeAgentID string `json:"probeAgentId,omitempty"`
// ObservationLocationIDs selects every source-owned path that observes this
// one logical verification. pulse:local is the Pulse server and agent:<id>
// names an eligible connected host agent. Names and network details stay out
// of persisted target configuration and are resolved at presentation time.
ObservationLocationIDs []string `json:"observationLocationIds,omitempty"`
// HTTP is absent for legacy targets. Its request body, credentials, and
// header values are encrypted with the rest of availability target storage
// and are never returned by the API.
@@ -227,6 +235,7 @@ func (t AvailabilityTarget) EffectiveFailureThreshold() int {
// executed or where it executes. Display, correlation, alert-threshold, and
// certificate-presentation edits intentionally stay within the same revision.
func AvailabilityExecutionConfigChanged(previous, next AvailabilityTarget) bool {
legacyProbeAssignmentChanged := strings.TrimSpace(previous.ProbeAgentID) != strings.TrimSpace(next.ProbeAgentID)
previous = NormalizeAvailabilityTarget(previous)
next = NormalizeAvailabilityTarget(next)
return previous.Address != next.Address ||
@@ -239,7 +248,67 @@ func AvailabilityExecutionConfigChanged(previous, next AvailabilityTarget) bool
!reflect.DeepEqual(previous.HTTP, next.HTTP) ||
previous.EffectiveTimeoutMillis() != next.EffectiveTimeoutMillis() ||
previous.EffectivePollIntervalSecs() != next.EffectivePollIntervalSecs() ||
previous.ProbeAgentID != next.ProbeAgentID
legacyProbeAssignmentChanged ||
!reflect.DeepEqual(previous.EffectiveObservationLocationIDs(), next.EffectiveObservationLocationIDs())
}
func AvailabilityAgentObservationLocationID(agentID string) string {
agentID = strings.TrimSpace(agentID)
if agentID == "" {
return ""
}
return AvailabilityObservationLocationAgentPrefix + agentID
}
func AvailabilityObservationLocationAgentID(locationID string) string {
locationID = strings.TrimSpace(locationID)
if !strings.HasPrefix(locationID, AvailabilityObservationLocationAgentPrefix) {
return ""
}
return strings.TrimSpace(strings.TrimPrefix(locationID, AvailabilityObservationLocationAgentPrefix))
}
func (t AvailabilityTarget) EffectiveObservationLocationIDs() []string {
locations := normalizeAvailabilityObservationLocationIDs(t.ObservationLocationIDs)
if len(locations) > 0 {
return locations
}
if agentID := strings.TrimSpace(t.ProbeAgentID); agentID != "" {
return []string{AvailabilityAgentObservationLocationID(agentID)}
}
return []string{AvailabilityObservationLocationLocal}
}
func (t AvailabilityTarget) IncludesLocalObservation() bool {
for _, locationID := range t.EffectiveObservationLocationIDs() {
if locationID == AvailabilityObservationLocationLocal {
return true
}
}
return false
}
func (t AvailabilityTarget) AssignedProbeAgentIDs() []string {
agents := make([]string, 0, len(t.ObservationLocationIDs))
for _, locationID := range t.EffectiveObservationLocationIDs() {
if agentID := AvailabilityObservationLocationAgentID(locationID); agentID != "" {
agents = append(agents, agentID)
}
}
return agents
}
func (t AvailabilityTarget) IsAssignedToProbeAgent(agentID string) bool {
agentID = strings.TrimSpace(agentID)
if agentID == "" {
return false
}
for _, assignedID := range t.AssignedProbeAgentIDs() {
if assignedID == agentID {
return true
}
}
return false
}
func (t AvailabilityTarget) CertificateMonitoringEnabled() bool {
@@ -331,6 +400,21 @@ func (t AvailabilityTarget) Validate() error {
if t.FailureThreshold > 0 && t.FailureThreshold > 10 {
return fmt.Errorf("availability failure threshold must be 10 or less")
}
locations := t.EffectiveObservationLocationIDs()
if len(locations) == 0 {
return fmt.Errorf("availability target requires at least one observation location")
}
if len(locations) > 16 {
return fmt.Errorf("availability target supports at most 16 observation locations")
}
for _, locationID := range locations {
if locationID == AvailabilityObservationLocationLocal {
continue
}
if AvailabilityObservationLocationAgentID(locationID) == "" {
return fmt.Errorf("unsupported availability observation location %q", locationID)
}
}
if protocol == AvailabilityProbeHTTP || protocol == AvailabilityProbeHTTPS {
if _, err := t.HTTPURL(); err != nil {
return err
@@ -403,6 +487,24 @@ func NormalizeAvailabilityTarget(target AvailabilityTarget) AvailabilityTarget {
}
target.LinkedResourceID = strings.TrimSpace(target.LinkedResourceID)
target.ProbeAgentID = strings.TrimSpace(target.ProbeAgentID)
target.ObservationLocationIDs = normalizeAvailabilityObservationLocationIDs(target.ObservationLocationIDs)
if target.ProbeAgentID != "" && len(target.ObservationLocationIDs) == 1 && target.ObservationLocationIDs[0] == AvailabilityObservationLocationLocal {
// A pre-location client edits ProbeAgentID on a normalized local target.
target.ObservationLocationIDs = []string{AvailabilityAgentObservationLocationID(target.ProbeAgentID)}
}
if len(target.ObservationLocationIDs) == 0 {
if target.ProbeAgentID != "" {
target.ObservationLocationIDs = []string{AvailabilityAgentObservationLocationID(target.ProbeAgentID)}
} else {
target.ObservationLocationIDs = []string{AvailabilityObservationLocationLocal}
}
}
// Keep the legacy field truthful only for the shape it can represent.
if len(target.ObservationLocationIDs) == 1 {
target.ProbeAgentID = AvailabilityObservationLocationAgentID(target.ObservationLocationIDs[0])
} else {
target.ProbeAgentID = ""
}
if target.HTTP != nil {
target.HTTP.Method = AvailabilityHTTPMethod(strings.ToUpper(strings.TrimSpace(string(target.HTTP.Method))))
target.HTTP.Authentication.Type = AvailabilityHTTPAuthType(strings.ToLower(strings.TrimSpace(string(target.HTTP.Authentication.Type))))
@@ -432,6 +534,41 @@ func NormalizeAvailabilityTarget(target AvailabilityTarget) AvailabilityTarget {
return target
}
func normalizeAvailabilityObservationLocationIDs(locationIDs []string) []string {
if len(locationIDs) == 0 {
return nil
}
seen := make(map[string]struct{}, len(locationIDs))
locations := make([]string, 0, len(locationIDs))
for _, raw := range locationIDs {
locationID := strings.TrimSpace(raw)
if locationID == "local" {
locationID = AvailabilityObservationLocationLocal
}
if agentID := AvailabilityObservationLocationAgentID(locationID); agentID != "" {
locationID = AvailabilityAgentObservationLocationID(agentID)
}
if locationID == "" {
continue
}
if _, ok := seen[locationID]; ok {
continue
}
seen[locationID] = struct{}{}
locations = append(locations, locationID)
}
sort.SliceStable(locations, func(i, j int) bool {
if locations[i] == AvailabilityObservationLocationLocal {
return true
}
if locations[j] == AvailabilityObservationLocationLocal {
return false
}
return locations[i] < locations[j]
})
return locations
}
func validateAvailabilityHTTPConfig(contract *AvailabilityHTTPConfig) error {
if contract == nil {
return nil
+39
View File
@@ -2,6 +2,7 @@ package config
import (
"os"
"reflect"
"strings"
"testing"
)
@@ -312,6 +313,44 @@ func TestNormalizeAvailabilityTargetTrimsProbeAgentID(t *testing.T) {
}
}
func TestNormalizeAvailabilityTargetBuildsCanonicalObservationLocationSet(t *testing.T) {
target := NormalizeAvailabilityTarget(AvailabilityTarget{
Address: "service.local",
Protocol: AvailabilityProbeICMP,
Enabled: true,
ObservationLocationIDs: []string{" agent:edge-b ", "local", "agent:edge-a", "agent:edge-a"},
})
want := []string{AvailabilityObservationLocationLocal, "agent:edge-a", "agent:edge-b"}
if !reflect.DeepEqual(target.ObservationLocationIDs, want) {
t.Fatalf("ObservationLocationIDs = %#v, want %#v", target.ObservationLocationIDs, want)
}
if target.ProbeAgentID != "" {
t.Fatalf("ProbeAgentID = %q, multi-location targets must not claim one owner", target.ProbeAgentID)
}
if !target.IncludesLocalObservation() || !reflect.DeepEqual(target.AssignedProbeAgentIDs(), []string{"edge-a", "edge-b"}) {
t.Fatalf("normalized locations = local %v agents %#v", target.IncludesLocalObservation(), target.AssignedProbeAgentIDs())
}
}
func TestAvailabilityExecutionConfigChangedTracksObservationLocationSet(t *testing.T) {
base := NormalizeAvailabilityTarget(AvailabilityTarget{
Address: "service.local",
Protocol: AvailabilityProbeICMP,
Enabled: true,
ObservationLocationIDs: []string{AvailabilityObservationLocationLocal, "agent:edge-a"},
})
reordered := base
reordered.ObservationLocationIDs = []string{"agent:edge-a", AvailabilityObservationLocationLocal}
if AvailabilityExecutionConfigChanged(base, reordered) {
t.Fatal("location ordering changed the execution revision")
}
expanded := base
expanded.ObservationLocationIDs = append(expanded.ObservationLocationIDs, "agent:edge-b")
if !AvailabilityExecutionConfigChanged(base, expanded) {
t.Fatal("adding an observation location did not change the execution revision")
}
}
func TestAvailabilityTargetProbeAgentIDRoundTripsThroughPersistence(t *testing.T) {
persistence := NewConfigPersistence(t.TempDir())
targets := []AvailabilityTarget{
+32
View File
@@ -349,6 +349,35 @@ func availabilityFixtureRecord(fixture AvailabilityFixture, now time.Time) (unif
PollIntervalSeconds: target.effectivePollIntervalSecs(),
TimeoutMillis: target.effectiveTimeoutMillis(),
}
if target.ID == "mock-availability-docker-frontend-service" {
data.AggregateState = "degraded"
data.Disagreement = true
data.ExpectedLocations = 2
data.ReportingLocations = 2
data.Locations = []unifiedresources.AvailabilityObservationLocation{
{
LocationID: "pulse:local",
Kind: "pulse",
Outcome: "reachable",
Available: true,
LastChecked: availabilityFixtureTimePointer(fixture.LastChecked),
LastSuccess: availabilityFixtureTimePointer(fixture.LastSuccess),
FreshnessAt: availabilityFixtureTimePointer(fixture.LastChecked),
LatencyMillis: fixture.LatencyMillis,
},
{
LocationID: "agent:edge-london",
Kind: "agent",
ProbeAgentID: "edge-london",
Outcome: "unreachable",
Available: false,
LastChecked: availabilityFixtureTimePointer(fixture.LastChecked),
FreshnessAt: availabilityFixtureTimePointer(fixture.LastChecked),
ConsecutiveFailures: 2,
LastError: "connection timed out",
},
}
}
data.Evidence = mockAvailabilityEvidence(target, fixture, lastSeen, now)
if target.Protocol == mockAvailabilityProbeHTTPS {
data.CertificateMonitoring = true
@@ -377,6 +406,9 @@ func availabilityFixtureRecord(fixture AvailabilityFixture, now time.Time) (unif
Tags: availabilityFixtureTags(target),
Availability: data,
}
if data.AggregateState == "degraded" {
resource.Status = unifiedresources.StatusWarning
}
if incident := availabilityFixtureIncident(target, fixture, lastSeen); incident != nil {
resource.Incidents = []unifiedresources.ResourceIncident{*incident}
}
+377 -48
View File
@@ -25,27 +25,32 @@ type tlsCert = tlsutil.CertificateObservation
// AvailabilityProbeStatus captures the last observed state of an agentless
// endpoint probe.
type AvailabilityProbeStatus struct {
TargetID string `json:"targetId"`
Name string `json:"name"`
TargetKind string `json:"targetKind,omitempty"`
Address string `json:"address"`
Protocol string `json:"protocol"`
Outcome string `json:"outcome,omitempty"`
TransportOutcome string `json:"transportOutcome,omitempty"`
ApplicationOutcome string `json:"applicationOutcome,omitempty"`
ApplicationStatusCode int `json:"applicationStatusCode,omitempty"`
ApplicationFailureCode string `json:"applicationFailureCode,omitempty"`
Enabled bool `json:"enabled"`
Available bool `json:"available"`
LastChecked time.Time `json:"lastChecked,omitempty"`
LastSuccess time.Time `json:"lastSuccess,omitempty"`
LatencyMillis int64 `json:"latencyMillis,omitempty"`
ConsecutiveFailures int `json:"consecutiveFailures,omitempty"`
LastError string `json:"lastError,omitempty"`
FailureThreshold int `json:"failureThreshold,omitempty"`
ProbeAgentID string `json:"probeAgentId,omitempty"`
Certificate *tlsCert `json:"certificate,omitempty"`
CertificateCurrent bool `json:"-"`
TargetID string `json:"targetId"`
Name string `json:"name"`
TargetKind string `json:"targetKind,omitempty"`
Address string `json:"address"`
Protocol string `json:"protocol"`
Outcome string `json:"outcome,omitempty"`
TransportOutcome string `json:"transportOutcome,omitempty"`
ApplicationOutcome string `json:"applicationOutcome,omitempty"`
ApplicationStatusCode int `json:"applicationStatusCode,omitempty"`
ApplicationFailureCode string `json:"applicationFailureCode,omitempty"`
Enabled bool `json:"enabled"`
Available bool `json:"available"`
LastChecked time.Time `json:"lastChecked,omitempty"`
LastSuccess time.Time `json:"lastSuccess,omitempty"`
LatencyMillis int64 `json:"latencyMillis,omitempty"`
ConsecutiveFailures int `json:"consecutiveFailures,omitempty"`
LastError string `json:"lastError,omitempty"`
FailureThreshold int `json:"failureThreshold,omitempty"`
ProbeAgentID string `json:"probeAgentId,omitempty"`
AggregateState string `json:"aggregateState,omitempty"`
Disagreement bool `json:"disagreement,omitempty"`
ExpectedLocations int `json:"expectedLocations,omitempty"`
ReportingLocations int `json:"reportingLocations,omitempty"`
Locations []AvailabilityObservationLocationStatus `json:"locations,omitempty"`
Certificate *tlsCert `json:"certificate,omitempty"`
CertificateCurrent bool `json:"-"`
// ProbeReportReceivedAt is server-authored freshness evidence for a remote
// observation. Keep it off the wire: LastChecked remains the agent's
// observation time, while disconnect detection must not trust agent clock
@@ -53,11 +58,41 @@ type AvailabilityProbeStatus struct {
ProbeReportReceivedAt time.Time `json:"-"`
}
const (
AvailabilityAggregateHealthy = "healthy"
AvailabilityAggregateDegraded = "degraded"
AvailabilityAggregateUnavailable = "unavailable"
AvailabilityAggregateUnknown = "unknown"
)
// AvailabilityObservationLocationStatus is one source-owned view of a logical
// verification. LocationID is stable configuration identity; Label is resolved
// by the authenticated client so host names and network details are not copied
// into target or history persistence.
type AvailabilityObservationLocationStatus struct {
LocationID string `json:"locationId"`
Kind string `json:"kind"`
ProbeAgentID string `json:"probeAgentId,omitempty"`
Outcome string `json:"outcome,omitempty"`
TransportOutcome string `json:"transportOutcome,omitempty"`
ApplicationOutcome string `json:"applicationOutcome,omitempty"`
ApplicationStatusCode int `json:"applicationStatusCode,omitempty"`
ApplicationFailureCode string `json:"applicationFailureCode,omitempty"`
Available bool `json:"available"`
LastChecked time.Time `json:"lastChecked,omitempty"`
LastSuccess time.Time `json:"lastSuccess,omitempty"`
FreshnessAt time.Time `json:"freshnessAt,omitempty"`
LatencyMillis int64 `json:"latencyMillis,omitempty"`
ConsecutiveFailures int `json:"consecutiveFailures,omitempty"`
LastError string `json:"lastError,omitempty"`
Stale bool `json:"stale,omitempty"`
}
// FreshnessTime returns the authoritative liveness reference for this status.
// Remote observations use server receipt time; local observations use their
// server-authored check time.
func (s AvailabilityProbeStatus) FreshnessTime() time.Time {
if strings.TrimSpace(s.ProbeAgentID) != "" && !s.ProbeReportReceivedAt.IsZero() {
if (strings.TrimSpace(s.ProbeAgentID) != "" || s.ExpectedLocations > 1) && !s.ProbeReportReceivedAt.IsZero() {
return s.ProbeReportReceivedAt
}
return s.LastChecked
@@ -93,7 +128,7 @@ func (availabilityPollProvider) ListInstances(m *Monitor) []string {
// A licensed probe assignment moves execution to the agent. When the
// entitlement lapses the effective assignment collapses to local, and
// the next polling cycle re-plans this instance without a restart.
if m.effectiveProbeAgentID(target) != "" {
if !m.targetUsesLocalObservation(target) {
continue
}
names = append(names, target.ID)
@@ -132,8 +167,8 @@ func (availabilityPollProvider) BuildPollTask(m *Monitor, instanceName string) (
}
// Refuse the local run even for a queue entry scheduled before the target
// was assigned, so a probe-assigned target never executes twice.
if agentID := m.effectiveProbeAgentID(target); agentID != "" {
return PollTask{}, fmt.Errorf("availability target %q is assigned to probe agent %q", instanceName, agentID)
if !m.targetUsesLocalObservation(target) {
return PollTask{}, fmt.Errorf("availability target %q has no local observation location", instanceName)
}
return PollTask{
InstanceName: target.ID,
@@ -262,17 +297,61 @@ func (m *Monitor) availabilityStatusSnapshotForTargets(targets []config.Availabi
for id, status := range m.availabilityStatuses {
out[id] = status
}
locationSnapshots := make(map[string]map[string]AvailabilityProbeStatus, len(m.availabilityByLocation))
for targetID, locations := range m.availabilityByLocation {
locationSnapshots[targetID] = make(map[string]AvailabilityProbeStatus, len(locations))
for locationID, status := range locations {
locationSnapshots[targetID][locationID] = status
}
}
m.mu.RUnlock()
for _, target := range targets {
status, ok := out[target.ID]
if !ok {
if m.effectiveProbeAgentID(target) == "" {
continue
}
status = availabilityStatusFromTarget(target)
expected := m.effectiveObservationLocationIDs(target)
locations := locationSnapshots[target.ID]
if locations == nil {
locations = make(map[string]AvailabilityProbeStatus)
}
out[target.ID] = m.deriveAvailabilityProbeStaleness(target, status, now)
// Adopt pre-location in-memory state into its one effective location.
if len(locations) == 0 {
if legacy, ok := out[target.ID]; ok && len(expected) == 1 {
legacyLocationID := config.AvailabilityObservationLocationLocal
if legacy.ProbeAgentID != "" {
legacyLocationID = config.AvailabilityAgentObservationLocationID(legacy.ProbeAgentID)
}
if legacyLocationID == expected[0] || expected[0] == config.AvailabilityObservationLocationLocal {
if expected[0] == config.AvailabilityObservationLocationLocal {
legacy.ProbeAgentID = ""
legacy.ProbeReportReceivedAt = time.Time{}
}
locations[expected[0]] = legacy
}
}
}
if len(expected) == 1 && expected[0] == config.AvailabilityObservationLocationLocal {
if _, ok := locations[expected[0]]; !ok {
if legacy, exists := out[target.ID]; exists {
legacy.ProbeAgentID = ""
legacy.ProbeReportReceivedAt = time.Time{}
locations[expected[0]] = legacy
} else {
delete(out, target.ID)
continue
}
}
}
for _, locationID := range expected {
status, ok := locations[locationID]
if !ok {
status = availabilityStatusFromTarget(target)
status.ProbeAgentID = config.AvailabilityObservationLocationAgentID(locationID)
}
if status.ProbeAgentID != "" {
status = m.deriveAvailabilityProbeStaleness(target, status, now)
}
locations[locationID] = status
}
out[target.ID] = aggregateAvailabilityLocationStatuses(target, expected, locations)
}
return out
}
@@ -287,9 +366,13 @@ func (m *Monitor) RefreshAvailabilityTargets() {
now := time.Now()
for _, target := range targets {
activeIDs[target.ID] = struct{}{}
if agentID := m.effectiveProbeAgentID(target); agentID != "" {
activeProbeIDs[target.ID] = struct{}{}
m.availabilityProbeAssignmentReference(target.ID, agentID, now)
for _, locationID := range m.effectiveObservationLocationIDs(target) {
if config.AvailabilityObservationLocationAgentID(locationID) == "" {
continue
}
trackerID := target.ID + "\x00" + locationID
activeProbeIDs[trackerID] = struct{}{}
m.availabilityProbeAssignmentReference(target.ID, locationID, now)
}
if m.taskQueue == nil {
continue
@@ -300,7 +383,7 @@ func (m *Monitor) RefreshAvailabilityTargets() {
NextRun: now,
Interval: clampInterval(time.Duration(target.EffectivePollIntervalSecs())*time.Second, 10*time.Second, time.Hour),
}
if target.Enabled && m.effectiveProbeAgentID(target) == "" {
if target.Enabled && m.targetUsesLocalObservation(target) {
m.taskQueue.Upsert(task)
} else {
m.taskQueue.Remove(InstanceTypeAvailability, target.ID)
@@ -316,6 +399,11 @@ func (m *Monitor) RefreshAvailabilityTargets() {
removedIDs = append(removedIDs, id)
}
}
for id := range m.availabilityByLocation {
if _, ok := activeIDs[id]; !ok {
delete(m.availabilityByLocation, id)
}
}
for id := range m.availabilityProbeTrackers {
if _, ok := activeProbeIDs[id]; !ok {
delete(m.availabilityProbeTrackers, id)
@@ -375,15 +463,21 @@ func (m *Monitor) applyAvailabilityObservationDetailed(
}
m.setAvailabilityStatusWithDetails(target, checkedAt, latency, outcome, transportOutcome, application, probeErr, certificate, probeAgentID, probeReportReceivedAt)
m.recordAvailabilityHistory(target, observationID, checkedAt, latency, outcome, probeErr, probeAgentID, probeReportReceivedAt)
current := m.AvailabilityStatusSnapshot()[target.ID]
aggregateHealthy := current.AggregateState == AvailabilityAggregateHealthy || current.AggregateState == AvailabilityAggregateDegraded
if probeErr == nil {
if aggregateHealthy {
if m.stalenessTracker != nil {
m.stalenessTracker.UpdateSuccess(InstanceTypeAvailability, target.ID, nil)
}
m.setProviderConnectionHealth(InstanceTypeAvailability, target.ID, true)
} else {
if m.stalenessTracker != nil {
m.stalenessTracker.UpdateSuccess(InstanceTypeAvailability, target.ID, []byte(probeErr.Error()))
reason := strings.TrimSpace(current.LastError)
if reason == "" {
reason = "availability evidence is incomplete"
}
m.stalenessTracker.UpdateSuccess(InstanceTypeAvailability, target.ID, []byte(reason))
}
m.setProviderConnectionHealth(InstanceTypeAvailability, target.ID, false)
}
@@ -421,6 +515,23 @@ func (m *Monitor) recordAvailabilityHistory(
}
latencyMillis = &value
}
expected := m.effectiveObservationLocationIDs(target)
if len(expected) > 1 {
aggregate := m.AvailabilityStatusSnapshot()[target.ID]
outcome = AvailabilityProbeIndeterminate
latencyMillis = nil
switch aggregate.AggregateState {
case AvailabilityAggregateHealthy, AvailabilityAggregateDegraded:
outcome = AvailabilityProbeReachable
if aggregate.LatencyMillis > 0 {
value := aggregate.LatencyMillis
latencyMillis = &value
}
case AvailabilityAggregateUnavailable:
outcome = AvailabilityProbeUnreachable
}
observationID += "|aggregate"
}
if err := m.metricsStore.WriteAvailabilityObservationBounded(pkgmetrics.AvailabilityObservation{
ObservationID: observationID,
TargetID: target.ID,
@@ -477,6 +588,11 @@ func (m *Monitor) setAvailabilityStatusWithDetails(
if m == nil {
return
}
locationID := config.AvailabilityObservationLocationLocal
if agentID := strings.TrimSpace(probeAgentID); agentID != "" {
locationID = config.AvailabilityAgentObservationLocationID(agentID)
}
expectedLocations := m.effectiveObservationLocationIDs(target)
status := availabilityStatusFromTarget(target)
status.Outcome = string(outcome)
status.TransportOutcome = string(transportOutcome)
@@ -514,7 +630,13 @@ func (m *Monitor) setAvailabilityStatusWithDetails(
if m.availabilityStatuses == nil {
m.availabilityStatuses = make(map[string]AvailabilityProbeStatus)
}
if previous, ok := m.availabilityStatuses[target.ID]; ok {
if m.availabilityByLocation == nil {
m.availabilityByLocation = make(map[string]map[string]AvailabilityProbeStatus)
}
if m.availabilityByLocation[target.ID] == nil {
m.availabilityByLocation[target.ID] = make(map[string]AvailabilityProbeStatus)
}
if previous, ok := m.availabilityByLocation[target.ID][locationID]; ok {
status.LastSuccess = previous.LastSuccess
if status.Certificate == nil {
status.Certificate = previous.Certificate.Clone()
@@ -531,7 +653,12 @@ func (m *Monitor) setAvailabilityStatusWithDetails(
} else if probeErr != nil {
status.ConsecutiveFailures = 1
}
m.availabilityStatuses[target.ID] = status
m.availabilityByLocation[target.ID][locationID] = status
m.availabilityStatuses[target.ID] = aggregateAvailabilityLocationStatuses(
target,
expectedLocations,
m.availabilityByLocation[target.ID],
)
m.mu.Unlock()
}
@@ -567,6 +694,168 @@ func availabilityStatusFromTarget(target config.AvailabilityTarget) Availability
}
}
func aggregateAvailabilityLocationStatuses(
target config.AvailabilityTarget,
expected []string,
statuses map[string]AvailabilityProbeStatus,
) AvailabilityProbeStatus {
aggregate := availabilityStatusFromTarget(target)
aggregate.ExpectedLocations = len(expected)
if len(expected) == 0 {
aggregate.AggregateState = AvailabilityAggregateUnknown
return aggregate
}
reachable, unreachable, indeterminate := 0, 0, 0
minFailures := 0
latestFreshness := time.Time{}
latestChecked := time.Time{}
latestSuccess := time.Time{}
var representative *AvailabilityProbeStatus
var latestReachable *AvailabilityProbeStatus
for _, locationID := range expected {
status, ok := statuses[locationID]
if !ok {
status = availabilityStatusFromTarget(target)
status.ProbeAgentID = config.AvailabilityObservationLocationAgentID(locationID)
}
effectiveOutcome := status.Outcome
if effectiveOutcome == "" && !status.LastChecked.IsZero() {
if status.Available {
effectiveOutcome = string(AvailabilityProbeReachable)
} else if status.ConsecutiveFailures > 0 || strings.TrimSpace(status.LastError) != "" {
effectiveOutcome = string(AvailabilityProbeUnreachable)
} else {
effectiveOutcome = string(AvailabilityProbeIndeterminate)
}
status.Outcome = effectiveOutcome
}
freshness := status.FreshnessTime()
location := AvailabilityObservationLocationStatus{
LocationID: locationID,
Kind: "pulse",
ProbeAgentID: status.ProbeAgentID,
Outcome: status.Outcome,
TransportOutcome: status.TransportOutcome,
ApplicationOutcome: status.ApplicationOutcome,
ApplicationStatusCode: status.ApplicationStatusCode,
ApplicationFailureCode: status.ApplicationFailureCode,
Available: status.Available,
LastChecked: status.LastChecked,
LastSuccess: status.LastSuccess,
FreshnessAt: freshness,
LatencyMillis: status.LatencyMillis,
ConsecutiveFailures: status.ConsecutiveFailures,
LastError: status.LastError,
Stale: availabilityProbeStatusIsStale(status),
}
if status.ProbeAgentID != "" {
location.Kind = "agent"
}
aggregate.Locations = append(aggregate.Locations, location)
if status.LastChecked.IsZero() {
continue
}
if !availabilityProbeStatusIsStale(status) {
aggregate.ReportingLocations++
}
if freshness.After(latestFreshness) {
latestFreshness = freshness
}
if status.LastChecked.After(latestChecked) {
latestChecked = status.LastChecked
copy := status
representative = &copy
}
if status.LastSuccess.After(latestSuccess) {
latestSuccess = status.LastSuccess
}
switch effectiveOutcome {
case string(AvailabilityProbeReachable):
reachable++
if latestReachable == nil || status.LastChecked.After(latestReachable.LastChecked) {
copy := status
latestReachable = &copy
}
case string(AvailabilityProbeUnreachable):
unreachable++
if minFailures == 0 || status.ConsecutiveFailures < minFailures {
minFailures = status.ConsecutiveFailures
}
case string(AvailabilityProbeIndeterminate):
indeterminate++
}
}
if representative != nil {
aggregate.Outcome = representative.Outcome
aggregate.TransportOutcome = representative.TransportOutcome
aggregate.ApplicationOutcome = representative.ApplicationOutcome
aggregate.ApplicationStatusCode = representative.ApplicationStatusCode
aggregate.ApplicationFailureCode = representative.ApplicationFailureCode
aggregate.LatencyMillis = representative.LatencyMillis
aggregate.Certificate = representative.Certificate.Clone()
aggregate.CertificateCurrent = representative.CertificateCurrent
}
if latestReachable != nil {
aggregate.LatencyMillis = latestReachable.LatencyMillis
} else if len(expected) > 1 {
aggregate.LatencyMillis = 0
}
aggregate.LastChecked = latestChecked
aggregate.LastSuccess = latestSuccess
aggregate.ProbeReportReceivedAt = latestFreshness
aggregate.Disagreement = reachable > 0 && unreachable > 0
unknown := len(expected) - reachable - unreachable - indeterminate
switch {
case reachable == len(expected):
aggregate.AggregateState = AvailabilityAggregateHealthy
aggregate.Available = true
aggregate.Outcome = string(AvailabilityProbeReachable)
case reachable > 0:
aggregate.AggregateState = AvailabilityAggregateDegraded
aggregate.Available = true
aggregate.Outcome = string(AvailabilityProbeReachable)
case unreachable == len(expected):
aggregate.AggregateState = AvailabilityAggregateUnavailable
aggregate.Available = false
aggregate.Outcome = string(AvailabilityProbeUnreachable)
aggregate.ConsecutiveFailures = minFailures
aggregate.LastError = "all observation locations report the target unreachable"
default:
aggregate.AggregateState = AvailabilityAggregateUnknown
aggregate.Available = false
aggregate.Outcome = string(AvailabilityProbeIndeterminate)
if indeterminate > 0 || unknown > 0 {
aggregate.LastError = "observation coverage is incomplete"
}
}
if len(expected) == 1 && representative != nil {
// Preserve the established one-location response while adding the location
// envelope, so older consumers do not lose source attribution or errors.
locations := aggregate.Locations
state := aggregate.AggregateState
reporting := aggregate.ReportingLocations
aggregate = *representative
aggregate.AggregateState = state
aggregate.ExpectedLocations = 1
aggregate.ReportingLocations = reporting
aggregate.Locations = locations
}
if len(expected) == 1 && representative == nil {
status := statuses[expected[0]]
locations := aggregate.Locations
state := aggregate.AggregateState
reporting := aggregate.ReportingLocations
aggregate = status
aggregate.AggregateState = state
aggregate.ExpectedLocations = 1
aggregate.ReportingLocations = reporting
aggregate.Locations = locations
}
return aggregate
}
func availabilityResourceFromTarget(target config.AvailabilityTarget, status AvailabilityProbeStatus, _ string, now time.Time) (unifiedresources.Resource, unifiedresources.ResourceIdentity) {
lastSeen := status.FreshnessTime()
if lastSeen.IsZero() {
@@ -586,6 +875,10 @@ func availabilityResourceFromTarget(target config.AvailabilityTarget, status Ava
ApplicationStatusCode: status.ApplicationStatusCode,
ApplicationFailureCode: status.ApplicationFailureCode,
ProbeAgentID: status.ProbeAgentID,
AggregateState: status.AggregateState,
Disagreement: status.Disagreement,
ExpectedLocations: status.ExpectedLocations,
ReportingLocations: status.ReportingLocations,
UDPMode: string(target.UDPMode),
Port: target.Port,
Path: target.Path,
@@ -600,6 +893,26 @@ func availabilityResourceFromTarget(target config.AvailabilityTarget, status Ava
PollIntervalSeconds: target.EffectivePollIntervalSecs(),
TimeoutMillis: target.EffectiveTimeoutMillis(),
}
for _, location := range status.Locations {
data.Locations = append(data.Locations, unifiedresources.AvailabilityObservationLocation{
LocationID: location.LocationID,
Kind: location.Kind,
ProbeAgentID: location.ProbeAgentID,
Outcome: location.Outcome,
TransportOutcome: location.TransportOutcome,
ApplicationOutcome: location.ApplicationOutcome,
ApplicationStatusCode: location.ApplicationStatusCode,
ApplicationFailureCode: location.ApplicationFailureCode,
Available: location.Available,
LastChecked: timePointerIfSet(location.LastChecked),
LastSuccess: timePointerIfSet(location.LastSuccess),
FreshnessAt: timePointerIfSet(location.FreshnessAt),
LatencyMillis: location.LatencyMillis,
ConsecutiveFailures: location.ConsecutiveFailures,
LastError: location.LastError,
Stale: location.Stale,
})
}
data.CertificateMonitoring = target.CertificateMonitoringEnabled()
data.CertificateExpiryWarningDays = target.EffectiveCertificateExpiryWarningDays()
data.Certificate = status.Certificate.Clone()
@@ -612,7 +925,7 @@ func availabilityResourceFromTarget(target config.AvailabilityTarget, status Ava
observedAt = lastSeen
}
ingestedAt := now
if strings.TrimSpace(status.ProbeAgentID) != "" && !status.ProbeReportReceivedAt.IsZero() {
if !status.ProbeReportReceivedAt.IsZero() {
ingestedAt = status.ProbeReportReceivedAt
}
data.Evidence = availabilityEvidenceEnvelope(target, status, observedAt, ingestedAt)
@@ -672,22 +985,24 @@ func availabilityEvidenceEnvelope(
return nil
}
freshnessAt := observedAt
freshnessAt := status.FreshnessTime()
if freshnessAt.IsZero() {
freshnessAt = observedAt
}
validityWindow := time.Duration(target.EffectivePollIntervalSecs()*2) * time.Second
if strings.TrimSpace(status.ProbeAgentID) != "" && !status.ProbeReportReceivedAt.IsZero() {
freshnessAt = status.ProbeReportReceivedAt
if strings.TrimSpace(status.ProbeAgentID) != "" || status.ExpectedLocations > 1 {
validityWindow = availabilityProbeStaleWindow(target)
}
validUntil := freshnessAt.Add(validityWindow)
completeness := operationaltrust.EvidenceComplete
confidence := operationaltrust.EvidenceConfirmed
var reason *operationaltrust.EvidenceReason
if status.LastChecked.IsZero() {
if status.LastChecked.IsZero() || status.ReportingLocations < status.ExpectedLocations {
completeness = operationaltrust.EvidencePartial
confidence = operationaltrust.EvidenceUnknown
reason = &operationaltrust.EvidenceReason{
Code: "availability_not_observed",
Message: "The availability target has not completed its first probe.",
Code: "availability_coverage_incomplete",
Message: "One or more observation locations have no current evidence.",
}
}
@@ -728,6 +1043,17 @@ func availabilityResourceStatus(target config.AvailabilityTarget, status Availab
if availabilityProbeStatusIsStale(status) {
return unifiedresources.StatusWarning
}
switch status.AggregateState {
case AvailabilityAggregateHealthy:
return unifiedresources.StatusOnline
case AvailabilityAggregateDegraded, AvailabilityAggregateUnknown:
return unifiedresources.StatusWarning
case AvailabilityAggregateUnavailable:
if status.ConsecutiveFailures >= target.EffectiveFailureThreshold() {
return unifiedresources.StatusOffline
}
return unifiedresources.StatusWarning
}
if status.LastChecked.IsZero() {
return unifiedresources.StatusUnknown
}
@@ -747,13 +1073,16 @@ func availabilityIncident(target config.AvailabilityTarget, status AvailabilityP
if availabilityProbeStatusIsStale(status) {
return nil
}
if status.AggregateState != "" && status.AggregateState != AvailabilityAggregateUnavailable {
return nil
}
if status.LastChecked.IsZero() {
return nil
}
if status.ConsecutiveFailures < target.EffectiveFailureThreshold() {
return nil
}
summary := fmt.Sprintf("%s is unreachable by %s probe", target.DisplayName(), strings.ToUpper(string(target.Protocol)))
summary := fmt.Sprintf("%s is unreachable from every current observation location by %s probe", target.DisplayName(), strings.ToUpper(string(target.Protocol)))
if status.LastError != "" {
summary = summary + ": " + status.LastError
}
+50 -22
View File
@@ -42,8 +42,8 @@ type ProbeAvailabilityResult struct {
// assigned target before its first report. AgentID is retained so reassigning
// the same target to a different probe starts a fresh grace window.
type availabilityProbeAssignmentTracker struct {
AgentID string
Since time.Time
LocationID string
Since time.Time
}
// probeAvailabilityResultsFromReport converts the wire results carried by a
@@ -89,14 +89,45 @@ func probeAvailabilityResultsFromReport(reported []agentshost.AvailabilityProbeR
// entitlement is absent, so a license lapse resumes local polling instead of
// stranding the check on an agent that is no longer allowed to run it.
func (m *Monitor) effectiveProbeAgentID(target config.AvailabilityTarget) string {
assigned := strings.TrimSpace(target.ProbeAgentID)
if assigned == "" {
locations := m.effectiveObservationLocationIDs(target)
if len(locations) != 1 {
return ""
}
if !m.hasLicensedFeature(pkglicensing.FeatureExternalProbe) {
return ""
return config.AvailabilityObservationLocationAgentID(locations[0])
}
func (m *Monitor) effectiveObservationLocationIDs(target config.AvailabilityTarget) []string {
locations := target.EffectiveObservationLocationIDs()
if m.hasLicensedFeature(pkglicensing.FeatureExternalProbe) {
return locations
}
return assigned
for _, locationID := range locations {
if locationID == config.AvailabilityObservationLocationLocal {
return []string{config.AvailabilityObservationLocationLocal}
}
}
// Preserve the historical entitlement-lapse behavior: a remote-only check
// resumes locally instead of becoming unobserved.
return []string{config.AvailabilityObservationLocationLocal}
}
func (m *Monitor) targetUsesLocalObservation(target config.AvailabilityTarget) bool {
for _, locationID := range m.effectiveObservationLocationIDs(target) {
if locationID == config.AvailabilityObservationLocationLocal {
return true
}
}
return false
}
func (m *Monitor) targetAssignedToProbeAgent(target config.AvailabilityTarget, agentID string) bool {
locationID := config.AvailabilityAgentObservationLocationID(agentID)
for _, assignedLocationID := range m.effectiveObservationLocationIDs(target) {
if assignedLocationID == locationID {
return true
}
}
return false
}
// ApplyProbeAvailabilityResults ingests availability results reported by a host
@@ -137,7 +168,7 @@ func (m *Monitor) applyProbeAvailabilityResultsAt(hostID string, results []Probe
Msg("Rejecting probe availability result for unknown target")
continue
}
if m.effectiveProbeAgentID(target) != hostID {
if !m.targetAssignedToProbeAgent(target, hostID) {
log.Debug().
Str("hostID", hostID).
Str("targetID", targetID).
@@ -240,18 +271,14 @@ func (m *Monitor) deriveAvailabilityProbeStaleness(
status AvailabilityProbeStatus,
now time.Time,
) AvailabilityProbeStatus {
agentID := m.effectiveProbeAgentID(target)
agentID := strings.TrimSpace(status.ProbeAgentID)
if agentID == "" {
return status
}
reportingAgentID := strings.TrimSpace(status.ProbeAgentID)
if reportingAgentID != agentID {
status = availabilityStatusFromTarget(target)
}
status.ProbeAgentID = agentID
reference := status.FreshnessTime()
if reference.IsZero() {
reference = m.availabilityProbeAssignmentReference(target.ID, agentID, now)
reference = m.availabilityProbeAssignmentReference(target.ID, config.AvailabilityAgentObservationLocationID(agentID), now)
}
if !availabilityProbeReportIsStale(target, reference, now) {
return status
@@ -263,27 +290,28 @@ func (m *Monitor) deriveAvailabilityProbeStaleness(
return status
}
func (m *Monitor) availabilityProbeAssignmentReference(targetID, agentID string, now time.Time) time.Time {
func (m *Monitor) availabilityProbeAssignmentReference(targetID, locationID string, now time.Time) time.Time {
if m == nil {
return now
}
targetID = strings.TrimSpace(targetID)
agentID = strings.TrimSpace(agentID)
if targetID == "" || agentID == "" {
locationID = strings.TrimSpace(locationID)
if targetID == "" || locationID == "" {
return now
}
trackerID := targetID + "\x00" + locationID
m.mu.Lock()
defer m.mu.Unlock()
if m.availabilityProbeTrackers == nil {
m.availabilityProbeTrackers = make(map[string]availabilityProbeAssignmentTracker)
}
if tracker, ok := m.availabilityProbeTrackers[targetID]; ok && tracker.AgentID == agentID && !tracker.Since.IsZero() {
if tracker, ok := m.availabilityProbeTrackers[trackerID]; ok && tracker.LocationID == locationID && !tracker.Since.IsZero() {
return tracker.Since
}
m.availabilityProbeTrackers[targetID] = availabilityProbeAssignmentTracker{
AgentID: agentID,
Since: now,
m.availabilityProbeTrackers[trackerID] = availabilityProbeAssignmentTracker{
LocationID: locationID,
Since: now,
}
return now
}
@@ -317,7 +345,7 @@ func (m *Monitor) availabilityProbeTargetsForAgent(hostID string) []map[string]i
}
var assigned []map[string]interface{}
for _, target := range m.availabilityTargets() {
if m.effectiveProbeAgentID(target) != hostID {
if !m.targetAssignedToProbeAgent(target, hostID) {
continue
}
assigned = append(assigned, availabilityProbeAgentTargetPayload(target))
@@ -69,14 +69,99 @@ func newProbeAgentTestMonitor(t *testing.T, targets ...config.AvailabilityTarget
t.Fatalf("SaveAvailabilityTargets() error = %v", err)
}
return &Monitor{
state: models.NewState(),
configPersist: persistence,
availabilityStatuses: make(map[string]AvailabilityProbeStatus),
pollStatusMap: make(map[string]*pollStatus),
failureCounts: make(map[string]int),
lastOutcome: make(map[string]taskOutcome),
circuitBreakers: make(map[string]*circuitBreaker),
taskQueue: NewTaskQueue(),
state: models.NewState(),
configPersist: persistence,
availabilityStatuses: make(map[string]AvailabilityProbeStatus),
availabilityByLocation: make(map[string]map[string]AvailabilityProbeStatus),
pollStatusMap: make(map[string]*pollStatus),
failureCounts: make(map[string]int),
lastOutcome: make(map[string]taskOutcome),
circuitBreakers: make(map[string]*circuitBreaker),
taskQueue: NewTaskQueue(),
}
}
func TestMultiLocationAvailabilityPreservesDisagreementUntilEveryPathFails(t *testing.T) {
target := config.NormalizeAvailabilityTarget(config.AvailabilityTarget{
ID: "service",
Name: "Customer API",
Address: "api.service.local",
Protocol: config.AvailabilityProbeHTTPS,
Enabled: true,
PollIntervalSecs: 60,
FailureThreshold: 2,
ObservationLocationIDs: []string{config.AvailabilityObservationLocationLocal, config.AvailabilityAgentObservationLocationID("edge-1")},
})
monitor := newProbeAgentTestMonitor(t, target)
monitor.SetLicenseChecker(licenseWithExternalProbe(true))
now := time.Now().UTC()
monitor.applyAvailabilityObservation(target, "local-ok", now, 8*time.Millisecond, AvailabilityProbeReachable, nil, nil, "", time.Time{})
for attempt := 0; attempt < 2; attempt++ {
monitor.applyAvailabilityObservation(target, "edge-fail-"+string(rune('a'+attempt)), now.Add(time.Duration(attempt)*time.Second), 30*time.Millisecond, AvailabilityProbeUnreachable, context.DeadlineExceeded, nil, "edge-1", now.Add(time.Duration(attempt)*time.Second))
}
status := monitor.AvailabilityStatusSnapshot()[target.ID]
if status.AggregateState != AvailabilityAggregateDegraded || !status.Disagreement || !status.Available {
t.Fatalf("aggregate status = %+v, want available disagreement", status)
}
if status.ExpectedLocations != 2 || status.ReportingLocations != 2 || len(status.Locations) != 2 {
t.Fatalf("location coverage = %+v, want 2/2 locations", status)
}
if want := now.Add(time.Second); !status.FreshnessTime().Equal(want) {
t.Fatalf("aggregate freshness = %s, want latest server-authored path freshness %s", status.FreshnessTime(), want)
}
if status.LatencyMillis != 8 {
t.Fatalf("aggregate latency = %dms, want the current reachable path latency", status.LatencyMillis)
}
resource, _ := availabilityResourceFromTarget(target, status, "", now)
if resource.Status != unifiedresources.StatusWarning || len(resource.Incidents) != 0 {
t.Fatalf("resource = status %q incidents %+v, one path failure must not author an outage", resource.Status, resource.Incidents)
}
for attempt := 0; attempt < 2; attempt++ {
monitor.applyAvailabilityObservation(target, "local-fail-"+string(rune('a'+attempt)), now.Add(time.Duration(attempt+2)*time.Second), 20*time.Millisecond, AvailabilityProbeUnreachable, context.DeadlineExceeded, nil, "", time.Time{})
}
status = monitor.AvailabilityStatusSnapshot()[target.ID]
if status.AggregateState != AvailabilityAggregateUnavailable || status.Available || status.ConsecutiveFailures != 2 {
t.Fatalf("aggregate status = %+v, want thresholded all-location outage", status)
}
if status.LatencyMillis != 0 {
t.Fatalf("unavailable aggregate latency = %dms, want no reachable latency", status.LatencyMillis)
}
resource, _ = availabilityResourceFromTarget(target, status, "", now.Add(4*time.Second))
if resource.Status != unifiedresources.StatusOffline || len(resource.Incidents) != 1 {
t.Fatalf("resource = status %q incidents %+v, want one universal outage", resource.Status, resource.Incidents)
}
}
func TestMultiLocationAvailabilityTreatsLapsedPathAsUnknownCoverage(t *testing.T) {
target := config.NormalizeAvailabilityTarget(config.AvailabilityTarget{
ID: "service",
Name: "Customer API",
Address: "api.service.local",
Protocol: config.AvailabilityProbeHTTPS,
Enabled: true,
PollIntervalSecs: 60,
FailureThreshold: 2,
ObservationLocationIDs: []string{config.AvailabilityObservationLocationLocal, config.AvailabilityAgentObservationLocationID("edge-1")},
})
monitor := newProbeAgentTestMonitor(t, target)
monitor.SetLicenseChecker(licenseWithExternalProbe(true))
now := time.Now().UTC()
old := time.Now().UTC().Add(-time.Hour)
monitor.applyAvailabilityObservation(target, "local-ok", now, 8*time.Millisecond, AvailabilityProbeReachable, nil, nil, "", time.Time{})
monitor.applyAvailabilityObservation(target, "edge-ok", old, 18*time.Millisecond, AvailabilityProbeReachable, nil, nil, "edge-1", old)
status := monitor.availabilityStatusSnapshotForTargets([]config.AvailabilityTarget{target}, now)[target.ID]
if status.AggregateState != AvailabilityAggregateDegraded || status.Disagreement {
t.Fatalf("aggregate status = %+v, want healthy path plus unknown coverage", status)
}
if status.ReportingLocations != 1 || status.ExpectedLocations != 2 {
t.Fatalf("coverage = %d/%d, want the lapsed path excluded", status.ReportingLocations, status.ExpectedLocations)
}
if !status.Locations[1].Stale || status.Locations[1].Outcome != string(AvailabilityProbeIndeterminate) {
t.Fatalf("remote location = %+v, want stale unknown evidence", status.Locations[1])
}
}
@@ -673,8 +673,14 @@ func TestAvailabilityProviderStaysOnCanonicalMonitoringPath(t *testing.T) {
`InstanceTypeAvailability InstanceType = "availability"`,
},
"monitor.go": {
"availabilityStatuses map[string]AvailabilityProbeStatus",
"availabilityStatuses: make(map[string]AvailabilityProbeStatus)",
"availabilityStatuses",
"map[string]AvailabilityProbeStatus",
"availabilityByLocation",
"map[string]map[string]AvailabilityProbeStatus",
"availabilityStatuses:",
"make(map[string]AvailabilityProbeStatus)",
"availabilityByLocation:",
"make(map[string]map[string]AvailabilityProbeStatus)",
},
"poll_providers.go": {
"_ = m.RegisterPollProvider(newAvailabilityPollProvider())",
@@ -713,9 +719,11 @@ func TestMonitoredSystemUsageReadinessGuardrailsRemainCanonical(t *testing.T) {
"monitor.go": {
"type MonitorSupplementalInventoryReadinessProvider interface {",
"SupplementalInventoryReadyAt(m *Monitor, orgID string) (time.Time, bool)",
"hostContinuityStore *config.HostContinuityStore",
"hostContinuityStore",
"*config.HostContinuityStore",
"hostContinuityStore := config.NewHostContinuityStore(cfg.DataPath, nil)",
"hostContinuityStore: hostContinuityStore,",
"hostContinuityStore:",
"hostContinuityStore,",
"func (m *Monitor) HostsSnapshot() []models.Host {",
"readState = m.readStateWithStandaloneHostContinuity(readState)",
"func (m *Monitor) unifiedStateViewWithStandaloneHostContinuity(view monitorUnifiedStateView) monitorUnifiedStateView {",
@@ -905,7 +913,8 @@ func TestProxmoxGuestMemoryFallbackUsesInstanceScopedCachesAndAgentMeminfo(t *te
`memorySource = "unavailable"`,
},
"monitor.go": {
"vmAgentMemCache map[string]agentMemCacheEntry",
"vmAgentMemCache",
"map[string]agentMemCacheEntry",
},
"monitor_agents.go": {
"for key, entry := range m.vmAgentMemCache {",
@@ -1238,8 +1247,10 @@ func TestProxmoxGuestDockerLXCProbingRequiresExplicitOptIn(t *testing.T) {
func TestBackupOrphanDetectionUsesCanonicalInventoryReadinessScope(t *testing.T) {
requiredSnippets := map[string][]string{
"monitor.go": {
"pveBackupInventoryReady map[string]map[string]bool",
"pveBackupTemplateSubjects map[string]map[string]struct{}",
"pveBackupInventoryReady",
"map[string]map[string]bool",
"pveBackupTemplateSubjects",
"map[string]map[string]struct{}",
},
"monitor_backups.go": {
"func (m *Monitor) updatePVEBackupTemplateSubjectsForType(instanceName, guestType string, subjects map[string]struct{}) {",
+2
View File
@@ -1078,6 +1078,7 @@ type Monitor struct {
pbsClients map[string]*pbs.Client
pmgClients map[string]*pmg.Client
availabilityStatuses map[string]AvailabilityProbeStatus
availabilityByLocation map[string]map[string]AvailabilityProbeStatus
availabilityProbeTrackers map[string]availabilityProbeAssignmentTracker
pollProviders map[InstanceType]PollProvider
pollMetrics *PollMetrics
@@ -1668,6 +1669,7 @@ func New(cfg *config.Config) (*Monitor, error) {
pbsClients: make(map[string]*pbs.Client),
pmgClients: make(map[string]*pmg.Client),
availabilityStatuses: make(map[string]AvailabilityProbeStatus),
availabilityByLocation: make(map[string]map[string]AvailabilityProbeStatus),
availabilityProbeTrackers: make(map[string]availabilityProbeAssignmentTracker),
pollProviders: make(map[InstanceType]PollProvider),
pollMetrics: getPollMetrics(),
@@ -740,7 +740,9 @@ func TestAgentlessAvailabilityTargetKindStaysCanonical(t *testing.T) {
"LinkedResourceID string `json:\"linkedResourceId,omitempty\"`",
},
filepath.Join("..", "monitoring", "availability_poller.go"): {
"TargetKind string `json:\"targetKind,omitempty\"`",
"TargetKind",
"string",
"`json:\"targetKind,omitempty\"`",
"TargetKind: string(target.TargetKind),",
"TargetKind: string(target.TargetKind),",
"tags = append(tags, string(target.TargetKind))",
@@ -748,7 +750,9 @@ func TestAgentlessAvailabilityTargetKindStaysCanonical(t *testing.T) {
},
"types.go": {
"Availability *AvailabilityData `json:\"availability,omitempty\"`",
"TargetKind string `json:\"targetKind,omitempty\"`",
"TargetKind",
"string",
"`json:\"targetKind,omitempty\"`",
"LinkedResourceID string `json:\"linkedResourceId,omitempty\"`",
"LastChecked *time.Time `json:\"lastChecked,omitempty\"`",
"LastSuccess *time.Time `json:\"lastSuccess,omitempty\"`",
+24
View File
@@ -1753,6 +1753,11 @@ type AvailabilityData struct {
ApplicationStatusCode int `json:"applicationStatusCode,omitempty"`
ApplicationFailureCode string `json:"applicationFailureCode,omitempty"`
ProbeAgentID string `json:"probeAgentId,omitempty"`
AggregateState string `json:"aggregateState,omitempty"`
Disagreement bool `json:"disagreement,omitempty"`
ExpectedLocations int `json:"expectedLocations,omitempty"`
ReportingLocations int `json:"reportingLocations,omitempty"`
Locations []AvailabilityObservationLocation `json:"locations,omitempty"`
UDPMode string `json:"udpMode,omitempty"`
Port int `json:"port,omitempty"`
Path string `json:"path,omitempty"`
@@ -1777,6 +1782,25 @@ type AvailabilityData struct {
Certificate *tlsutil.CertificateObservation `json:"certificate,omitempty"`
}
type AvailabilityObservationLocation struct {
LocationID string `json:"locationId"`
Kind string `json:"kind"`
ProbeAgentID string `json:"probeAgentId,omitempty"`
Outcome string `json:"outcome,omitempty"`
TransportOutcome string `json:"transportOutcome,omitempty"`
ApplicationOutcome string `json:"applicationOutcome,omitempty"`
ApplicationStatusCode int `json:"applicationStatusCode,omitempty"`
ApplicationFailureCode string `json:"applicationFailureCode,omitempty"`
Available bool `json:"available"`
LastChecked *time.Time `json:"lastChecked,omitempty"`
LastSuccess *time.Time `json:"lastSuccess,omitempty"`
FreshnessAt *time.Time `json:"freshnessAt,omitempty"`
LatencyMillis int64 `json:"latencyMillis,omitempty"`
ConsecutiveFailures int `json:"consecutiveFailures,omitempty"`
LastError string `json:"lastError,omitempty"`
Stale bool `json:"stale,omitempty"`
}
// K8sMetricCapabilities describes which Kubernetes metric families are available
// for this cluster right now based on active collection paths.
type K8sMetricCapabilities struct {