mirror of
https://github.com/rcourtman/Pulse.git
synced 2026-09-10 02:25:56 +00:00
Add multi-location service delivery evidence
This commit is contained in:
@@ -1,7 +1,7 @@
|
|||||||
# Infrastructure-Aware Service Assurance Plan
|
# Infrastructure-Aware Service Assurance Plan
|
||||||
|
|
||||||
Last updated: 2026-08-30
|
Last updated: 2026-08-30
|
||||||
Status: ACCEPTED — SLICES A, B, AND C IMPLEMENTED
|
Status: ACCEPTED — SLICES A, B, C, AND D IMPLEMENTED
|
||||||
Governance surfaces:
|
Governance surfaces:
|
||||||
|
|
||||||
- `status.json.coverage_gaps.infrastructure-aware-service-assurance`
|
- `status.json.coverage_gaps.infrastructure-aware-service-assurance`
|
||||||
@@ -228,7 +228,7 @@ Implementation record (2026-08-30): Slice A is delivered through the
|
|||||||
monitoring-owned categorical history store and rollups, server-authored
|
monitoring-owned categorical history store and rollups, server-authored
|
||||||
configuration revisions and remote receipt timeline, the bounded
|
configuration revisions and remote receipt timeline, the bounded
|
||||||
`/api/availability-history` batch contract, and the URL-owned Availability
|
`/api/availability-history` batch contract, and the URL-owned Availability
|
||||||
fleet presentation. Its release proofs live in the owner contract. Slices D
|
fleet presentation. Its release proofs live in the owner contract. Slices E
|
||||||
through H remain ordered future work; acceptance of this product lane does not
|
through H remain ordered future work; acceptance of this product lane does not
|
||||||
imply that deferred breadth is already delivered.
|
imply that deferred breadth is already delivered.
|
||||||
|
|
||||||
@@ -269,7 +269,7 @@ re-entry. Current status and unified-resource facets preserve transport
|
|||||||
reachability separately from typed application correctness while the overall
|
reachability separately from typed application correctness while the overall
|
||||||
result remains the alert and history outcome. Legacy targets without an
|
result remains the alert and history outcome. Legacy targets without an
|
||||||
explicit contract retain their previous HEAD-with-bounded-GET-fallback
|
explicit contract retain their previous HEAD-with-bounded-GET-fallback
|
||||||
semantics. Slices D through H remain ordered future work.
|
semantics. Slices E through H remain ordered future work.
|
||||||
|
|
||||||
### Slice C: Discovery-Led Assurance Onboarding
|
### Slice C: Discovery-Led Assurance Onboarding
|
||||||
|
|
||||||
@@ -304,7 +304,7 @@ standalone endpoints, and creates one explicitly enabled availability target
|
|||||||
with the drawer's canonical resource ID only after the operator chooses the
|
with the drawer's canonical resource ID only after the operator chooses the
|
||||||
activation action. A machine-scoped queue supports bulk review and
|
activation action. A machine-scoped queue supports bulk review and
|
||||||
evidence-bound dismiss/restore without bulk activation or guessed resource
|
evidence-bound dismiss/restore without bulk activation or guessed resource
|
||||||
attachment. Slices D through H remain ordered future work.
|
attachment. Slices E through H remain ordered future work.
|
||||||
|
|
||||||
### Slice D: Multi-Location Delivery Evidence
|
### Slice D: Multi-Location Delivery Evidence
|
||||||
|
|
||||||
@@ -332,6 +332,23 @@ Exit conditions:
|
|||||||
- location names and customer-identifying network details obey the existing
|
- location names and customer-identifying network details obey the existing
|
||||||
privacy and redaction policy.
|
privacy and redaction policy.
|
||||||
|
|
||||||
|
Implementation record (2026-08-30): Slice D is delivered through a canonical
|
||||||
|
set of source-owned observation-location IDs on each saved target. The local
|
||||||
|
Pulse runtime and any selected eligible connected agents may observe the same
|
||||||
|
logical verification concurrently; monitoring retains their state, latency,
|
||||||
|
server-receipt freshness, and coverage separately, then derives one aggregate
|
||||||
|
state without duplicating the target resource. A reachable path plus a failed
|
||||||
|
or missing path is `degraded`, every current path must be unreachable before
|
||||||
|
the target is `unavailable`, and incomplete all-failed evidence remains
|
||||||
|
`unknown`. Stale agent paths become indeterminate evidence, retries remain
|
||||||
|
idempotent through the existing observation ID and configuration revision,
|
||||||
|
and only the assigned agent may author its location. Settings presents the
|
||||||
|
location set as a bounded multi-select, while the Availability fleet and
|
||||||
|
resource detail surfaces expose reporting coverage and path disagreement.
|
||||||
|
Compatibility with the former single `probeAgentId` contract is retained at
|
||||||
|
the API boundary without keeping single-agent execution as the canonical
|
||||||
|
model. Slices E through H remain ordered future work.
|
||||||
|
|
||||||
### Slice E: Infrastructure-Aware Incident Synthesis
|
### Slice E: Infrastructure-Aware Incident Synthesis
|
||||||
|
|
||||||
Deliver deterministic incident grouping and causal narrowing over canonical
|
Deliver deterministic incident grouping and causal narrowing over canonical
|
||||||
|
|||||||
@@ -6211,6 +6211,17 @@ remain ingestible through a deterministic server fallback, but they do not
|
|||||||
gain authority to author revision boundaries, target identity, retention, or
|
gain authority to author revision boundaries, target identity, retention, or
|
||||||
the service-assurance timeline.
|
the service-assurance timeline.
|
||||||
|
|
||||||
|
The server may assign one target to several agent-backed observation locations,
|
||||||
|
but each agent receives and reports only the targets whose canonical
|
||||||
|
`agent:<agent-id>` location it owns. Location membership is server-authored
|
||||||
|
configuration, not agent enrollment identity: it does not grant control-plane
|
||||||
|
authority, create another agent resource, or let one reporter stand in for a
|
||||||
|
different site. Reassignment increments the target configuration revision, so
|
||||||
|
late results from the previous location set are rejected. Local Pulse execution
|
||||||
|
can coexist with assigned agent execution for the same logical target, while
|
||||||
|
license lapse removes every remote location from the effective assignment and
|
||||||
|
leaves the local path as the fail-safe execution boundary.
|
||||||
|
|
||||||
Observer configuration is explicit, versioned, and file-backed. It contains no
|
Observer configuration is explicit, versioned, and file-backed. It contains no
|
||||||
raw token values and resolves each token from a separate private absolute-path
|
raw token values and resolves each token from a separate private absolute-path
|
||||||
file. Proxmox registration is also destination-scoped: the primary retains its
|
file. Proxmox registration is also destination-scoped: the primary retains its
|
||||||
|
|||||||
@@ -8808,6 +8808,18 @@ metadata, but API freshness and `lastSeenAt` projections use the server-authored
|
|||||||
receipt time supplied by monitoring. API consumers must not substitute the
|
receipt time supplied by monitoring. API consumers must not substitute the
|
||||||
agent clock for disconnect detection or apply a second, shorter generic
|
agent clock for disconnect detection or apply a second, shorter generic
|
||||||
Connections staleness window.
|
Connections staleness window.
|
||||||
|
Availability target writes now use `observationLocationIds` as the canonical
|
||||||
|
bounded set. Values are source-owned IDs (`pulse:local` or
|
||||||
|
`agent:<agent-id>`), are normalized and deduplicated by the server, and every
|
||||||
|
remote member requires the external-probe entitlement plus an existing agent.
|
||||||
|
The compatibility `probeAgentId` field is accepted for older single-agent
|
||||||
|
clients and is emitted only as a compatibility hint when the selected set can
|
||||||
|
be represented by one remote path. Current status and unified-resource
|
||||||
|
transport add aggregate state, disagreement, expected/reporting counts, and a
|
||||||
|
per-location list containing only location ID, state, latency, observation and
|
||||||
|
receipt times, freshness, and a bounded failure code. Callers must consume that
|
||||||
|
server-authored aggregate rather than treating the latest reporting path as the
|
||||||
|
whole service result.
|
||||||
Mock availability fixtures must still behave like saved targets: `/api/connections`
|
Mock availability fixtures must still behave like saved targets: `/api/connections`
|
||||||
reports them as availability rows, `/api/availability-targets` lists them with
|
reports them as availability rows, `/api/availability-targets` lists them with
|
||||||
probe status, and saved-test calls return the synthetic probe result instead of
|
probe status, and saved-test calls return the synthetic probe result instead of
|
||||||
|
|||||||
@@ -1566,24 +1566,35 @@ the frontend-primitives-owned Standalone surface may read the same
|
|||||||
`network-endpoint` projection to show current reachability, latency, check age,
|
`network-endpoint` projection to show current reachability, latency, check age,
|
||||||
and failure state without creating another monitoring provider or top-level
|
and failure state without creating another monitoring provider or top-level
|
||||||
availability route.
|
availability route.
|
||||||
Availability targets may also be assigned to a remote host agent. Reachability
|
Availability execution is location-owned. Each saved target has a normalized,
|
||||||
outcome and the optional certificate observation travel in the same bounded
|
deduplicated set of observation-location IDs: `pulse:local` for this Pulse
|
||||||
report entry, and the server clones that observation before status and resource
|
runtime and `agent:<agent-id>` for an eligible connected host agent. The legacy
|
||||||
projection so report buffers cannot alias live state. While the
|
single `probeAgentId` field remains an input compatibility boundary only; the
|
||||||
`external_probe` entitlement is active, a probe-assigned target is executed
|
location set owns scheduling, configuration revision, assignment, status, and
|
||||||
exclusively by its assigned agent: monitoring must not schedule or run it
|
resource projection. While `external_probe` is entitled, Pulse may schedule
|
||||||
locally, so the check never executes twice. The assignment is effective only for
|
the local path and every selected agent path for one logical target at the same
|
||||||
as long as the entitlement holds; on lapse the effective assignment collapses to
|
time. On entitlement lapse the effective set collapses to the local path
|
||||||
local and the normal poll provider resumes the target on its next planning
|
without a restart. An agent result is accepted only for its exact selected
|
||||||
cycle, without a restart. Reported results are accepted only from the agent that
|
location and current target revision; another agent cannot author that path.
|
||||||
currently owns the target, and results for any other target or from any other
|
|
||||||
agent are dropped. Failure accounting, thresholds, and incident projection stay
|
Reachability outcome and the optional certificate observation still travel in
|
||||||
server-side. The agent-authored observation time remains visible as the target's
|
the same bounded report entry, and the server clones that observation before
|
||||||
last check, but staleness uses server receipt time so slow or fast agent clocks
|
status and resource projection so report buffers cannot alias live state.
|
||||||
cannot manufacture or conceal a disconnect. When an assigned agent stops
|
Monitoring retains one current status per target and location, using server
|
||||||
reporting, monitoring derives indeterminate with a stale-report explanation at
|
receipt time for remote freshness while keeping the agent-authored observation
|
||||||
read time through the shared probe-status snapshot rather than mutating stored
|
time as evidence metadata. A stale or disconnected agent path derives as
|
||||||
state, so every availability consumer sees the same staleness verdict.
|
indeterminate at read time; slow or fast agent clocks cannot manufacture or
|
||||||
|
conceal a disconnect. Aggregation is conservative: all reachable paths are
|
||||||
|
healthy, mixed reachable and failed/unknown paths are degraded, all current
|
||||||
|
paths unreachable are unavailable, and no reachable path with incomplete or
|
||||||
|
indeterminate coverage is unknown. Only aggregate unavailability advances the
|
||||||
|
target failure threshold or authors an outage incident. A path-local failure
|
||||||
|
therefore cannot claim that the service is universally down. Single-location
|
||||||
|
targets preserve their prior status and history semantics. Multi-location
|
||||||
|
targets write only their aggregate service conclusion to the logical target's
|
||||||
|
deletable categorical history; current per-location evidence remains a status
|
||||||
|
and resource projection until a separately governed location-history schema
|
||||||
|
can preserve deletion, retention, and bounded-query semantics end to end.
|
||||||
Mock-mode Discovery context follows the same fixture-graph rule. Demo service
|
Mock-mode Discovery context follows the same fixture-graph rule. Demo service
|
||||||
details such as detected version, config/data/log paths, Docker bind mounts,
|
details such as detected version, config/data/log paths, Docker bind mounts,
|
||||||
ports, and suggested web URLs may be authored in mock fixtures, but consumers
|
ports, and suggested web URLs may be authored in mock fixtures, but consumers
|
||||||
|
|||||||
@@ -1331,6 +1331,14 @@ alerts. Those counts may describe scale and feature adoption, but they must not
|
|||||||
include hostnames, resource IDs, infrastructure identifiers, credentials,
|
include hostnames, resource IDs, infrastructure identifiers, credentials,
|
||||||
prompts, chat messages, command text, action output, token values, or personal
|
prompts, chat messages, command text, action output, token values, or personal
|
||||||
information.
|
information.
|
||||||
|
Multi-location availability evidence stays inside the same privacy floor.
|
||||||
|
Saved configuration and authenticated operator surfaces may use source-owned
|
||||||
|
location IDs and agent display names to distinguish selected paths, but
|
||||||
|
categorical history, evidence envelopes, telemetry, incidents, logs, and
|
||||||
|
Patrol context must not retain or export customer network names, addresses,
|
||||||
|
agent identity, or location labels. Durable categorical history remains keyed
|
||||||
|
only by the logical target and its aggregate conclusion; location identity is
|
||||||
|
not a new telemetry, history, or customer-identity dimension.
|
||||||
That same outbound usage telemetry floor now also permits content-free update
|
That same outbound usage telemetry floor now also permits content-free update
|
||||||
funnel counters derived from local update history inside the same rotating
|
funnel counters derived from local update history inside the same rotating
|
||||||
30-day telemetry window: update attempts, successful updates, failed or
|
30-day telemetry window: update attempts, successful updates, failed or
|
||||||
|
|||||||
@@ -4670,6 +4670,18 @@ the shared resource drawer by canonical resource ID, while
|
|||||||
already carried by that resource. History absence or failure cannot overwrite
|
already carried by that resource. History absence or failure cannot overwrite
|
||||||
the unified resource's current status, correlation, source ownership, or
|
the unified resource's current status, correlation, source ownership, or
|
||||||
navigation identity.
|
navigation identity.
|
||||||
|
One availability resource may now carry several source-owned observation
|
||||||
|
locations without cloning the resource, relationship, or fleet row. Its
|
||||||
|
availability facet preserves the aggregate state, disagreement flag,
|
||||||
|
expected/reporting coverage, and the current state, latency, freshness, and
|
||||||
|
bounded failure reason for each path. Resource status remains active with
|
||||||
|
warning evidence when paths disagree, becomes stale/incomplete when coverage
|
||||||
|
is unknown, and becomes inactive with an outage incident only when every
|
||||||
|
current path is unreachable at the configured threshold. Frontend adapters,
|
||||||
|
REST, websocket, and mock projections must preserve this list rather than
|
||||||
|
flattening it to the most recent observer. The Availability table may summarize
|
||||||
|
the count and reporting coverage; the shared resource detail card owns the
|
||||||
|
expanded path comparison.
|
||||||
For HTTP/S checks with an explicit application response contract, the
|
For HTTP/S checks with an explicit application response contract, the
|
||||||
availability facet preserves two source-owned facts: `transportOutcome`
|
availability facet preserves two source-owned facts: `transportOutcome`
|
||||||
describes whether the endpoint answered, while `applicationOutcome`,
|
describes whether the endpoint answered, while `applicationOutcome`,
|
||||||
|
|||||||
@@ -1,24 +1,29 @@
|
|||||||
{
|
{
|
||||||
"version": 1,
|
"version": 1,
|
||||||
"base_sha": "d9e9bf3679ab579be9f8c7350932008c287ad5ad",
|
"base_sha": "e094a55b4500d2a10b6ab44b350c2fffcc55e73d",
|
||||||
"verified_at": "2026-08-30T20:24:19Z",
|
"verified_at": "2026-08-30T20:37:10Z",
|
||||||
"result": "passed",
|
"result": "passed",
|
||||||
"changed_paths": [
|
"changed_paths": [
|
||||||
"frontend-modern/src/components/Infrastructure/resourceDetailMappers.ts",
|
"frontend-modern/src/api/availabilityTargets.ts",
|
||||||
"frontend-modern/src/features/docker/DockerHostDrawerOverview.tsx",
|
"frontend-modern/src/components/Infrastructure/AvailabilityProbeStatusCard.tsx",
|
||||||
"frontend-modern/src/hooks/useWorkloads.ts",
|
"frontend-modern/src/components/Settings/ConnectionEditor/CredentialSlots/AvailabilityTargetSlot.tsx",
|
||||||
"frontend-modern/src/types/api.ts",
|
"frontend-modern/src/components/Settings/availabilitySettingsModel.ts",
|
||||||
"frontend-modern/src/types/resource.ts"
|
"frontend-modern/src/features/standalone/AvailabilityChecksTable.tsx",
|
||||||
|
"frontend-modern/src/types/resource.ts",
|
||||||
|
"frontend-modern/src/utils/availabilityProbeAgents.ts"
|
||||||
],
|
],
|
||||||
"content_sha256": {
|
"content_sha256": {
|
||||||
"frontend-modern/src/components/Infrastructure/resourceDetailMappers.ts": "8b1913ff24e6f5dbf074091f2bbaff1a8c3298126515ea288b055a02c0153997",
|
"frontend-modern/src/api/availabilityTargets.ts": "0a20277a8c453b753e8c09f7250722670211a33bcae2addad7d19502f921f365",
|
||||||
"frontend-modern/src/features/docker/DockerHostDrawerOverview.tsx": "621af65f92c9d58312f019790ecf2eb6986fb260724e9ef53213972fd9b55579",
|
"frontend-modern/src/components/Infrastructure/AvailabilityProbeStatusCard.tsx": "ec13f364ee9c55f6859c448797fed427255999723524ff0f51cb54cc9f864ea5",
|
||||||
"frontend-modern/src/hooks/useWorkloads.ts": "85719e8c93f503843cd09c1be41c66f9995288d1280ca2eedf7d47c3443bd9a1",
|
"frontend-modern/src/components/Settings/ConnectionEditor/CredentialSlots/AvailabilityTargetSlot.tsx": "622e0c7ad72c4f313c8d971295662e6549d9f13c5fbd788d3cc8803c827af176",
|
||||||
"frontend-modern/src/types/api.ts": "be1b1889588085791880dbd6716955710cd66e3dc69339998f8b9ad30c0ac378",
|
"frontend-modern/src/components/Settings/availabilitySettingsModel.ts": "cd03ed43bca379c8359c11c67f235d49a4a1c6b905c987a4ce404ce59443bbf6",
|
||||||
"frontend-modern/src/types/resource.ts": "74b26ae53a507f5608a3c191f7c386cffab636e0a95aeffa02028bd8199297b0"
|
"frontend-modern/src/features/standalone/AvailabilityChecksTable.tsx": "3c775ad5b08023f9a2814a9dc939992db363a7e5113a351d4fb65ad0946f0d97",
|
||||||
|
"frontend-modern/src/types/resource.ts": "a4e87ae08ec69349122075ca422ced08a56e1f14294088488198c5ff8af25dba",
|
||||||
|
"frontend-modern/src/utils/availabilityProbeAgents.ts": "7f92c5fc5012bdbb322825b4a03f26fb5dc6d20d260cda8b2c2e86b66283b005"
|
||||||
},
|
},
|
||||||
"routes": [
|
"routes": [
|
||||||
"/browser-helper-summary-harness.html (temporary current-build Vite harness for the Docker host drawer)"
|
"/standalone/availability",
|
||||||
|
"/settings/monitoring/availability?add=target&targetKind=service"
|
||||||
],
|
],
|
||||||
"viewports": [
|
"viewports": [
|
||||||
{
|
{
|
||||||
@@ -31,14 +36,15 @@
|
|||||||
}
|
}
|
||||||
],
|
],
|
||||||
"states": [
|
"states": [
|
||||||
"typed-helper summary mode with reduced-coverage warning and no container update management card",
|
"multi-location target editor with the local Pulse path and connected-agent paths, including the Community-tier remote-path entitlement boundary",
|
||||||
"direct-runtime mode without the reduced-coverage warning and with container update management controls",
|
"degraded service with one reachable and one unreachable observation path, preserved as Paths disagree with 2/2 reporting",
|
||||||
"return to typed-helper summary mode after direct-runtime mode"
|
"expanded resource evidence showing per-path state, latency, checked time, freshness, aggregate degraded state, and no universal outage",
|
||||||
|
"desktop and narrow layouts with settled data, dialog bounds, responsive navigation, and no page or dialog horizontal overflow"
|
||||||
],
|
],
|
||||||
"interactions": [
|
"interactions": [
|
||||||
"loaded summary mode and confirmed warning visibility, update-control omission, and no desktop horizontal overflow",
|
"opened the add service/device check flow and inspected the bounded observation-location selector without saving a mock target",
|
||||||
"switched to direct-runtime mode and confirmed the warning disappeared and update controls appeared",
|
"opened the Frontend Swarm service resource details from Availability checks and expanded the multi-location evidence",
|
||||||
"switched back to summary mode and confirmed the warning returned and update controls were removed",
|
"confirmed the local path reports 9 ms while edge-london reports Unreachable and the aggregate remains Degraded",
|
||||||
"inspected the final summary state at 390x844 and confirmed no horizontal overflow or clipped warning copy"
|
"repeated the final route, state, clipping, and overflow checks at 1280x800 and 390x844 after rebasing onto the current main base"
|
||||||
]
|
]
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -67,9 +67,33 @@ export interface AvailabilityProbeStatus {
|
|||||||
* Absent (omitempty) when the local Pulse server ran the check.
|
* Absent (omitempty) when the local Pulse server ran the check.
|
||||||
*/
|
*/
|
||||||
probeAgentId?: string;
|
probeAgentId?: string;
|
||||||
|
aggregateState?: 'healthy' | 'degraded' | 'unavailable' | 'unknown' | string;
|
||||||
|
disagreement?: boolean;
|
||||||
|
expectedLocations?: number;
|
||||||
|
reportingLocations?: number;
|
||||||
|
locations?: AvailabilityObservationLocationStatus[];
|
||||||
certificate?: ResourceCertificateObservation;
|
certificate?: ResourceCertificateObservation;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
export interface AvailabilityObservationLocationStatus {
|
||||||
|
locationId: string;
|
||||||
|
kind: 'pulse' | 'agent' | string;
|
||||||
|
probeAgentId?: string;
|
||||||
|
outcome?: 'reachable' | 'unreachable' | 'indeterminate' | string;
|
||||||
|
transportOutcome?: string;
|
||||||
|
applicationOutcome?: string;
|
||||||
|
applicationStatusCode?: number;
|
||||||
|
applicationFailureCode?: string;
|
||||||
|
available: boolean;
|
||||||
|
lastChecked?: string;
|
||||||
|
lastSuccess?: string;
|
||||||
|
freshnessAt?: string;
|
||||||
|
latencyMillis?: number;
|
||||||
|
consecutiveFailures?: number;
|
||||||
|
lastError?: string;
|
||||||
|
stale?: boolean;
|
||||||
|
}
|
||||||
|
|
||||||
export interface AvailabilityTarget {
|
export interface AvailabilityTarget {
|
||||||
id: string;
|
id: string;
|
||||||
configRevision?: number;
|
configRevision?: number;
|
||||||
@@ -96,6 +120,9 @@ export interface AvailabilityTarget {
|
|||||||
* string because the server decodes updates onto the existing record.
|
* string because the server decodes updates onto the existing record.
|
||||||
*/
|
*/
|
||||||
probeAgentId?: string;
|
probeAgentId?: string;
|
||||||
|
/** Canonical execution set. `pulse:local` names this Pulse server and
|
||||||
|
* `agent:<id>` names a connected Pulse Agent observation location. */
|
||||||
|
observationLocationIds?: string[];
|
||||||
http?: AvailabilityHTTPConfig;
|
http?: AvailabilityHTTPConfig;
|
||||||
httpSecrets?: AvailabilityHTTPSecretState;
|
httpSecrets?: AvailabilityHTTPSecretState;
|
||||||
status?: AvailabilityProbeStatus;
|
status?: AvailabilityProbeStatus;
|
||||||
|
|||||||
@@ -46,8 +46,13 @@ export function AvailabilityProbeStatusCards(props: AvailabilityProbeStatusCards
|
|||||||
}
|
}
|
||||||
|
|
||||||
export function AvailabilityProbeStatusCard(props: AvailabilityProbeStatusCardProps) {
|
export function AvailabilityProbeStatusCard(props: AvailabilityProbeStatusCardProps) {
|
||||||
const isUp = () => props.availability.available === true;
|
const aggregateState = () => props.availability.aggregateState;
|
||||||
const isDown = () => props.availability.available === false;
|
const isUp = () =>
|
||||||
|
aggregateState() ? aggregateState() === 'healthy' : props.availability.available === true;
|
||||||
|
const isDegraded = () => aggregateState() === 'degraded';
|
||||||
|
const isUnknown = () => aggregateState() === 'unknown';
|
||||||
|
const isDown = () =>
|
||||||
|
aggregateState() ? aggregateState() === 'unavailable' : props.availability.available === false;
|
||||||
const latency = () => {
|
const latency = () => {
|
||||||
const ms = props.availability.latencyMillis;
|
const ms = props.availability.latencyMillis;
|
||||||
return typeof ms === 'number' && Number.isFinite(ms) && ms > 0 ? `${Math.round(ms)}ms` : null;
|
return typeof ms === 'number' && Number.isFinite(ms) && ms > 0 ? `${Math.round(ms)}ms` : null;
|
||||||
@@ -129,6 +134,19 @@ export function AvailabilityProbeStatusCard(props: AvailabilityProbeStatusCardPr
|
|||||||
if (days === 0) return `${date} (today)`;
|
if (days === 0) return `${date} (today)`;
|
||||||
return `${date} (${days}d)`;
|
return `${date} (${days}d)`;
|
||||||
};
|
};
|
||||||
|
const locationLabel = (location: NonNullable<ResourceAvailabilityMeta['locations']>[number]) =>
|
||||||
|
location.kind === 'pulse'
|
||||||
|
? 'This Pulse server'
|
||||||
|
: location.probeAgentId || location.locationId.replace(/^agent:/, '') || 'Pulse Agent';
|
||||||
|
const locationResult = (location: NonNullable<ResourceAvailabilityMeta['locations']>[number]) => {
|
||||||
|
if (location.stale) return 'No recent report';
|
||||||
|
if (!location.lastChecked) return 'Awaiting evidence';
|
||||||
|
if (location.outcome === 'reachable') {
|
||||||
|
return location.latencyMillis ? `${Math.round(location.latencyMillis)} ms` : 'Reachable';
|
||||||
|
}
|
||||||
|
if (location.outcome === 'unreachable') return 'Unreachable';
|
||||||
|
return 'Unknown';
|
||||||
|
};
|
||||||
|
|
||||||
return (
|
return (
|
||||||
<InfoCardFrame data-testid="availability-probe-status">
|
<InfoCardFrame data-testid="availability-probe-status">
|
||||||
@@ -145,11 +163,22 @@ export function AvailabilityProbeStatusCard(props: AvailabilityProbeStatusCardPr
|
|||||||
'bg-emerald-100 text-emerald-700 dark:bg-emerald-900/40 dark:text-emerald-300':
|
'bg-emerald-100 text-emerald-700 dark:bg-emerald-900/40 dark:text-emerald-300':
|
||||||
isFreshUp(),
|
isFreshUp(),
|
||||||
'bg-red-100 text-red-700 dark:bg-red-900/40 dark:text-red-300': isDown() && !isStale(),
|
'bg-red-100 text-red-700 dark:bg-red-900/40 dark:text-red-300': isDown() && !isStale(),
|
||||||
'bg-amber-100 text-amber-700 dark:bg-amber-900/40 dark:text-amber-300': isStale(),
|
'bg-amber-100 text-amber-700 dark:bg-amber-900/40 dark:text-amber-300':
|
||||||
'bg-base-200 text-muted': !isUp() && !isDown() && !isStale(),
|
isStale() || isDegraded(),
|
||||||
|
'bg-base-200 text-muted': isUnknown() || (!isUp() && !isDown() && !isStale()),
|
||||||
}}
|
}}
|
||||||
>
|
>
|
||||||
{isStale() ? 'Stale' : isUp() ? 'Up' : isDown() ? 'Down' : 'Not checked'}
|
{isDegraded()
|
||||||
|
? 'Paths disagree'
|
||||||
|
: isUnknown()
|
||||||
|
? 'Coverage incomplete'
|
||||||
|
: isStale()
|
||||||
|
? 'Stale'
|
||||||
|
: isUp()
|
||||||
|
? 'Up'
|
||||||
|
: isDown()
|
||||||
|
? 'Down'
|
||||||
|
: 'Not checked'}
|
||||||
</span>
|
</span>
|
||||||
</div>
|
</div>
|
||||||
<div class="space-y-1.5 text-[11px]">
|
<div class="space-y-1.5 text-[11px]">
|
||||||
@@ -171,6 +200,51 @@ export function AvailabilityProbeStatusCard(props: AvailabilityProbeStatusCardPr
|
|||||||
</Show>
|
</Show>
|
||||||
}
|
}
|
||||||
/>
|
/>
|
||||||
|
<Show when={(props.availability.locations?.length ?? 0) > 0}>
|
||||||
|
<div
|
||||||
|
class="mt-2 space-y-1.5 border-t border-border pt-2"
|
||||||
|
data-observation-location-results
|
||||||
|
>
|
||||||
|
<div class="flex items-center justify-between gap-2 text-[10px] uppercase tracking-wide text-muted">
|
||||||
|
<span>Observation paths</span>
|
||||||
|
<span>
|
||||||
|
{props.availability.reportingLocations ?? 0}/
|
||||||
|
{props.availability.expectedLocations ?? props.availability.locations?.length ?? 0}{' '}
|
||||||
|
reporting
|
||||||
|
</span>
|
||||||
|
</div>
|
||||||
|
<For each={props.availability.locations}>
|
||||||
|
{(location) => (
|
||||||
|
<div class="flex items-start justify-between gap-3 rounded bg-base-200/60 px-2 py-1.5">
|
||||||
|
<div class="min-w-0">
|
||||||
|
<div
|
||||||
|
class="truncate font-medium text-base-content"
|
||||||
|
title={locationLabel(location)}
|
||||||
|
>
|
||||||
|
{locationLabel(location)}
|
||||||
|
</div>
|
||||||
|
<Show when={formatRelativeTime(location.lastChecked)}>
|
||||||
|
{(checked) => <div class="text-[10px] text-muted">checked {checked()}</div>}
|
||||||
|
</Show>
|
||||||
|
</div>
|
||||||
|
<span
|
||||||
|
class="shrink-0 font-medium"
|
||||||
|
classList={{
|
||||||
|
'text-emerald-600 dark:text-emerald-300':
|
||||||
|
location.outcome === 'reachable' && !location.stale,
|
||||||
|
'text-red-600 dark:text-red-300':
|
||||||
|
location.outcome === 'unreachable' && !location.stale,
|
||||||
|
'text-amber-600 dark:text-amber-300': Boolean(location.stale),
|
||||||
|
'text-muted': !location.outcome || location.outcome === 'indeterminate',
|
||||||
|
}}
|
||||||
|
>
|
||||||
|
{locationResult(location)}
|
||||||
|
</span>
|
||||||
|
</div>
|
||||||
|
)}
|
||||||
|
</For>
|
||||||
|
</div>
|
||||||
|
</Show>
|
||||||
<InfoCardKeyValueRow label="Method" value={method()} valueTitle={targetAddr()} />
|
<InfoCardKeyValueRow label="Method" value={method()} valueTitle={targetAddr()} />
|
||||||
<InfoCardKeyValueRow
|
<InfoCardKeyValueRow
|
||||||
label="Target"
|
label="Target"
|
||||||
|
|||||||
+63
@@ -0,0 +1,63 @@
|
|||||||
|
import { render, screen, within } from '@solidjs/testing-library';
|
||||||
|
import { describe, expect, it } from 'vitest';
|
||||||
|
|
||||||
|
import { AvailabilityProbeStatusCard } from '../AvailabilityProbeStatusCard';
|
||||||
|
|
||||||
|
describe('AvailabilityProbeStatusCard', () => {
|
||||||
|
it('shows disagreement and keeps every observation path visible', () => {
|
||||||
|
render(() => (
|
||||||
|
<AvailabilityProbeStatusCard
|
||||||
|
availability={{
|
||||||
|
targetId: 'customer-api',
|
||||||
|
address: 'api.service.local',
|
||||||
|
protocol: 'https',
|
||||||
|
enabled: true,
|
||||||
|
available: true,
|
||||||
|
aggregateState: 'degraded',
|
||||||
|
disagreement: true,
|
||||||
|
expectedLocations: 3,
|
||||||
|
reportingLocations: 2,
|
||||||
|
locations: [
|
||||||
|
{
|
||||||
|
locationId: 'pulse:local',
|
||||||
|
kind: 'pulse',
|
||||||
|
outcome: 'reachable',
|
||||||
|
available: true,
|
||||||
|
latencyMillis: 8,
|
||||||
|
lastChecked: '2026-08-30T19:00:00Z',
|
||||||
|
},
|
||||||
|
{
|
||||||
|
locationId: 'agent:edge-a',
|
||||||
|
kind: 'agent',
|
||||||
|
probeAgentId: 'edge-a',
|
||||||
|
outcome: 'unreachable',
|
||||||
|
available: false,
|
||||||
|
lastChecked: '2026-08-30T19:00:00Z',
|
||||||
|
},
|
||||||
|
{
|
||||||
|
locationId: 'agent:edge-b',
|
||||||
|
kind: 'agent',
|
||||||
|
probeAgentId: 'edge-b',
|
||||||
|
outcome: 'indeterminate',
|
||||||
|
available: false,
|
||||||
|
stale: true,
|
||||||
|
lastChecked: '2026-08-30T18:00:00Z',
|
||||||
|
},
|
||||||
|
],
|
||||||
|
}}
|
||||||
|
/>
|
||||||
|
));
|
||||||
|
|
||||||
|
expect(screen.getByText('Paths disagree')).toBeInTheDocument();
|
||||||
|
const paths = screen.getByText('Observation paths').parentElement?.parentElement;
|
||||||
|
expect(paths).not.toBeNull();
|
||||||
|
const scope = within(paths!);
|
||||||
|
expect(scope.getByText('2/3 reporting')).toBeInTheDocument();
|
||||||
|
expect(scope.getByText('This Pulse server')).toBeInTheDocument();
|
||||||
|
expect(scope.getByText('8 ms')).toBeInTheDocument();
|
||||||
|
expect(scope.getByText('edge-a')).toBeInTheDocument();
|
||||||
|
expect(scope.getByText('Unreachable')).toBeInTheDocument();
|
||||||
|
expect(scope.getByText('edge-b')).toBeInTheDocument();
|
||||||
|
expect(scope.getByText('No recent report')).toBeInTheDocument();
|
||||||
|
});
|
||||||
|
});
|
||||||
+97
-34
@@ -33,13 +33,14 @@ import { getPreferredInfrastructureDisplayName } from '@/utils/resourceIdentity'
|
|||||||
import { getResourceTypeLabel } from '@/utils/resourceTypePresentation';
|
import { getResourceTypeLabel } from '@/utils/resourceTypePresentation';
|
||||||
import {
|
import {
|
||||||
EXTERNAL_PROBE_FEATURE,
|
EXTERNAL_PROBE_FEATURE,
|
||||||
LOCAL_PROBE_AGENT_LABEL,
|
AGENT_OBSERVATION_LOCATION_PREFIX,
|
||||||
buildProbeAgentOptions,
|
LOCAL_OBSERVATION_LOCATION_ID,
|
||||||
|
agentIdFromObservationLocation,
|
||||||
|
buildObservationLocationOptions,
|
||||||
getExternalProbeGateBody,
|
getExternalProbeGateBody,
|
||||||
getExternalProbeGateTitle,
|
getExternalProbeGateTitle,
|
||||||
getExternalProbeLockedHelpText,
|
getExternalProbeLockedHelpText,
|
||||||
isExternalProbeLicenseError,
|
isExternalProbeLicenseError,
|
||||||
isProbeAgentMissing,
|
|
||||||
} from '@/utils/availabilityProbeAgents';
|
} from '@/utils/availabilityProbeAgents';
|
||||||
import { hasFeature, loadRuntimeCapabilities, runtimeCapabilitiesLoaded } from '@/stores/license';
|
import { hasFeature, loadRuntimeCapabilities, runtimeCapabilitiesLoaded } from '@/stores/license';
|
||||||
import { getUpgradeActionDestination } from '@/stores/licenseCommercial';
|
import { getUpgradeActionDestination } from '@/stores/licenseCommercial';
|
||||||
@@ -58,7 +59,7 @@ interface AvailabilityForm {
|
|||||||
udpRequest: string;
|
udpRequest: string;
|
||||||
udpExpectedResponse: string;
|
udpExpectedResponse: string;
|
||||||
linkedResourceId: string;
|
linkedResourceId: string;
|
||||||
probeAgentId: string;
|
observationLocationIds: string[];
|
||||||
enabled: boolean;
|
enabled: boolean;
|
||||||
pollIntervalSeconds: string;
|
pollIntervalSeconds: string;
|
||||||
timeoutMillis: string;
|
timeoutMillis: string;
|
||||||
@@ -118,7 +119,7 @@ const newAvailabilityForm = (
|
|||||||
udpRequest: '',
|
udpRequest: '',
|
||||||
udpExpectedResponse: '',
|
udpExpectedResponse: '',
|
||||||
linkedResourceId: '',
|
linkedResourceId: '',
|
||||||
probeAgentId: '',
|
observationLocationIds: [LOCAL_OBSERVATION_LOCATION_ID],
|
||||||
enabled: true,
|
enabled: true,
|
||||||
pollIntervalSeconds: '60',
|
pollIntervalSeconds: '60',
|
||||||
timeoutMillis: '2000',
|
timeoutMillis: '2000',
|
||||||
@@ -159,7 +160,11 @@ const formFromTarget = (target: AvailabilityTarget): AvailabilityForm => {
|
|||||||
udpRequest: target.udpRequest ?? '',
|
udpRequest: target.udpRequest ?? '',
|
||||||
udpExpectedResponse: target.udpExpectedResponse ?? '',
|
udpExpectedResponse: target.udpExpectedResponse ?? '',
|
||||||
linkedResourceId: target.linkedResourceId ?? '',
|
linkedResourceId: target.linkedResourceId ?? '',
|
||||||
probeAgentId: target.probeAgentId ?? '',
|
observationLocationIds: target.observationLocationIds?.length
|
||||||
|
? [...target.observationLocationIds]
|
||||||
|
: target.probeAgentId
|
||||||
|
? [`${AGENT_OBSERVATION_LOCATION_PREFIX}${target.probeAgentId}`]
|
||||||
|
: [LOCAL_OBSERVATION_LOCATION_ID],
|
||||||
enabled: target.enabled ?? true,
|
enabled: target.enabled ?? true,
|
||||||
pollIntervalSeconds: String(target.pollIntervalSeconds ?? 60),
|
pollIntervalSeconds: String(target.pollIntervalSeconds ?? 60),
|
||||||
timeoutMillis: String(target.timeoutMillis ?? 2000),
|
timeoutMillis: String(target.timeoutMillis ?? 2000),
|
||||||
@@ -247,7 +252,11 @@ const payloadFromForm = (form: AvailabilityForm): AvailabilityTarget => {
|
|||||||
// Always serialized, never `undefined`: the server decodes updates onto the
|
// Always serialized, never `undefined`: the server decodes updates onto the
|
||||||
// existing record, so an explicit empty string is what clears a probe
|
// existing record, so an explicit empty string is what clears a probe
|
||||||
// assignment and moves the check back to the local Pulse server.
|
// assignment and moves the check back to the local Pulse server.
|
||||||
probeAgentId: form.probeAgentId.trim(),
|
probeAgentId:
|
||||||
|
form.observationLocationIds.length === 1
|
||||||
|
? agentIdFromObservationLocation(form.observationLocationIds[0])
|
||||||
|
: '',
|
||||||
|
observationLocationIds: [...form.observationLocationIds],
|
||||||
enabled: form.enabled,
|
enabled: form.enabled,
|
||||||
pollIntervalSeconds: parsePositiveInt(form.pollIntervalSeconds),
|
pollIntervalSeconds: parsePositiveInt(form.pollIntervalSeconds),
|
||||||
timeoutMillis: parsePositiveInt(form.timeoutMillis),
|
timeoutMillis: parsePositiveInt(form.timeoutMillis),
|
||||||
@@ -348,10 +357,11 @@ export const AvailabilityTargetSlot: Component<AvailabilityTargetSlotProps> = (p
|
|||||||
// stale cached capability set still lands on the upgrade gate.
|
// stale cached capability set still lands on the upgrade gate.
|
||||||
const [probeLicenseRejected, setProbeLicenseRejected] = createSignal(false);
|
const [probeLicenseRejected, setProbeLicenseRejected] = createSignal(false);
|
||||||
|
|
||||||
const probeAgentOptions = createMemo(() => buildProbeAgentOptions(resources()));
|
const observationLocationOptions = createMemo(() => buildObservationLocationOptions(resources()));
|
||||||
const probeAgentMissing = createMemo(() =>
|
const missingObservationLocations = createMemo(() => {
|
||||||
isProbeAgentMissing(probeAgentOptions(), form().probeAgentId),
|
const known = new Set(observationLocationOptions().map((option) => option.id));
|
||||||
);
|
return form().observationLocationIds.filter((locationId) => !known.has(locationId));
|
||||||
|
});
|
||||||
const externalProbeLicensed = createMemo(
|
const externalProbeLicensed = createMemo(
|
||||||
() =>
|
() =>
|
||||||
!probeLicenseRejected() && runtimeCapabilitiesLoaded() && hasFeature(EXTERNAL_PROBE_FEATURE),
|
!probeLicenseRejected() && runtimeCapabilitiesLoaded() && hasFeature(EXTERNAL_PROBE_FEATURE),
|
||||||
@@ -378,6 +388,15 @@ export const AvailabilityTargetSlot: Component<AvailabilityTargetSlotProps> = (p
|
|||||||
setTestResult(null);
|
setTestResult(null);
|
||||||
};
|
};
|
||||||
|
|
||||||
|
const toggleObservationLocation = (locationId: string, selected: boolean) => {
|
||||||
|
const current = form().observationLocationIds;
|
||||||
|
const next = selected
|
||||||
|
? [...new Set([...current, locationId])]
|
||||||
|
: current.filter((candidate) => candidate !== locationId);
|
||||||
|
if (next.length === 0) return;
|
||||||
|
updateForm({ observationLocationIds: next });
|
||||||
|
};
|
||||||
|
|
||||||
const updateHTTPHeader = (index: number, patch: Partial<AvailabilityHTTPHeaderForm>) => {
|
const updateHTTPHeader = (index: number, patch: Partial<AvailabilityHTTPHeaderForm>) => {
|
||||||
updateForm({
|
updateForm({
|
||||||
httpHeaders: form().httpHeaders.map((header, headerIndex) =>
|
httpHeaders: form().httpHeaders.map((header, headerIndex) =>
|
||||||
@@ -490,7 +509,12 @@ export const AvailabilityTargetSlot: Component<AvailabilityTargetSlotProps> = (p
|
|||||||
}
|
}
|
||||||
props.onSaved();
|
props.onSaved();
|
||||||
} catch (err) {
|
} catch (err) {
|
||||||
if (payload.probeAgentId && isExternalProbeLicenseError(err)) {
|
if (
|
||||||
|
payload.observationLocationIds?.some((locationId) =>
|
||||||
|
locationId.startsWith(AGENT_OBSERVATION_LOCATION_PREFIX),
|
||||||
|
) &&
|
||||||
|
isExternalProbeLicenseError(err)
|
||||||
|
) {
|
||||||
setProbeLicenseRejected(true);
|
setProbeLicenseRejected(true);
|
||||||
setError(getExternalProbeGateBody());
|
setError(getExternalProbeGateBody());
|
||||||
return;
|
return;
|
||||||
@@ -610,28 +634,67 @@ export const AvailabilityTargetSlot: Component<AvailabilityTargetSlotProps> = (p
|
|||||||
)}
|
)}
|
||||||
</For>
|
</For>
|
||||||
</FormSelect>
|
</FormSelect>
|
||||||
<div class="space-y-3 sm:col-span-2">
|
<fieldset class="space-y-3 sm:col-span-2">
|
||||||
<FormSelect
|
<legend class={formLabel}>Observation locations</legend>
|
||||||
label="Run from"
|
<p class={formHelpText}>
|
||||||
value={form().probeAgentId}
|
One logical check can be observed from several networks. Pulse keeps each path separate
|
||||||
disabled={externalProbeLocked()}
|
and shows disagreement instead of turning one path failure into a universal outage.
|
||||||
onChange={(event) => updateForm({ probeAgentId: event.currentTarget.value })}
|
</p>
|
||||||
help={
|
<div class="grid gap-2 sm:grid-cols-2" data-observation-location-picker>
|
||||||
externalProbeLocked()
|
<For each={observationLocationOptions()}>
|
||||||
? getExternalProbeLockedHelpText()
|
{(option) => {
|
||||||
: 'Run this check from the Pulse server, or hand it to a connected Pulse Agent host so it is probed from that network.'
|
const selected = () => form().observationLocationIds.includes(option.id);
|
||||||
}
|
const locked = () => option.kind === 'agent' && externalProbeLocked();
|
||||||
>
|
return (
|
||||||
<option value="">{LOCAL_PROBE_AGENT_LABEL}</option>
|
<label class="flex items-start gap-3 rounded-md border border-border bg-surface px-3 py-3 text-sm">
|
||||||
<Show when={probeAgentMissing()}>
|
<input
|
||||||
<option value={form().probeAgentId}>
|
class={formCheckbox}
|
||||||
{form().probeAgentId} (not currently connected)
|
type="checkbox"
|
||||||
</option>
|
checked={selected()}
|
||||||
</Show>
|
disabled={
|
||||||
<For each={probeAgentOptions()}>
|
locked() || (selected() && form().observationLocationIds.length === 1)
|
||||||
{(option) => <option value={option.id}>{option.label}</option>}
|
}
|
||||||
|
onChange={(event) =>
|
||||||
|
toggleObservationLocation(option.id, event.currentTarget.checked)
|
||||||
|
}
|
||||||
|
/>
|
||||||
|
<span class="min-w-0">
|
||||||
|
<span class="block font-medium text-foreground">{option.label}</span>
|
||||||
|
<span class="block text-xs text-muted">
|
||||||
|
{option.kind === 'pulse'
|
||||||
|
? 'Local Pulse path'
|
||||||
|
: 'Connected Pulse Agent path'}
|
||||||
|
</span>
|
||||||
|
</span>
|
||||||
|
</label>
|
||||||
|
);
|
||||||
|
}}
|
||||||
</For>
|
</For>
|
||||||
</FormSelect>
|
<For each={missingObservationLocations()}>
|
||||||
|
{(locationId) => (
|
||||||
|
<label class="flex items-start gap-3 rounded-md border border-warning/40 bg-warning/5 px-3 py-3 text-sm">
|
||||||
|
<input
|
||||||
|
class={formCheckbox}
|
||||||
|
type="checkbox"
|
||||||
|
checked
|
||||||
|
disabled={form().observationLocationIds.length === 1}
|
||||||
|
onChange={(event) =>
|
||||||
|
toggleObservationLocation(locationId, event.currentTarget.checked)
|
||||||
|
}
|
||||||
|
/>
|
||||||
|
<span class="min-w-0">
|
||||||
|
<span class="block font-medium text-foreground">
|
||||||
|
{agentIdFromObservationLocation(locationId) || locationId}
|
||||||
|
</span>
|
||||||
|
<span class="block text-xs text-warning">Not currently connected</span>
|
||||||
|
</span>
|
||||||
|
</label>
|
||||||
|
)}
|
||||||
|
</For>
|
||||||
|
</div>
|
||||||
|
<Show when={externalProbeLocked()}>
|
||||||
|
<p class={formHelpText}>{getExternalProbeLockedHelpText()}</p>
|
||||||
|
</Show>
|
||||||
<Show when={externalProbeLocked()}>
|
<Show when={externalProbeLocked()}>
|
||||||
<div class="rounded-md border border-border bg-surface-alt p-4">
|
<div class="rounded-md border border-border bg-surface-alt p-4">
|
||||||
<FeatureGateSection
|
<FeatureGateSection
|
||||||
@@ -642,7 +705,7 @@ export const AvailabilityTargetSlot: Component<AvailabilityTargetSlotProps> = (p
|
|||||||
/>
|
/>
|
||||||
</div>
|
</div>
|
||||||
</Show>
|
</Show>
|
||||||
</div>
|
</fieldset>
|
||||||
<Show when={form().protocol !== 'icmp'}>
|
<Show when={form().protocol !== 'icmp'}>
|
||||||
<label class={formField}>
|
<label class={formField}>
|
||||||
<span class={formLabel}>Port</span>
|
<span class={formLabel}>Port</span>
|
||||||
|
|||||||
+25
-18
@@ -424,22 +424,25 @@ describe('AvailabilityTargetSlot', () => {
|
|||||||
resourceMocks.resources = [agentHostResource('host-edge-01', 'Edge 01')];
|
resourceMocks.resources = [agentHostResource('host-edge-01', 'Edge 01')];
|
||||||
render(() => <AvailabilityTargetSlot onCancel={vi.fn()} onSaved={vi.fn()} />);
|
render(() => <AvailabilityTargetSlot onCancel={vi.fn()} onSaved={vi.fn()} />);
|
||||||
|
|
||||||
const runFrom = screen.getByLabelText('Run from') as HTMLSelectElement;
|
const localLocation = screen.getByRole('checkbox', { name: /This Pulse server/ });
|
||||||
expect(runFrom).not.toBeDisabled();
|
const edgeLocation = screen.getByRole('checkbox', { name: /Edge 01/ });
|
||||||
expect(screen.getByRole('option', { name: 'This Pulse server' })).toBeInTheDocument();
|
expect(localLocation).toBeChecked();
|
||||||
expect(screen.getByRole('option', { name: 'Edge 01' })).toBeInTheDocument();
|
expect(edgeLocation).not.toBeDisabled();
|
||||||
expect(screen.queryByRole('link', { name: 'View plans' })).not.toBeInTheDocument();
|
expect(screen.queryByRole('link', { name: 'View plans' })).not.toBeInTheDocument();
|
||||||
|
|
||||||
fireEvent.input(screen.getByLabelText('Name'), { target: { value: 'Remote MQTT' } });
|
fireEvent.input(screen.getByLabelText('Name'), { target: { value: 'Remote MQTT' } });
|
||||||
fireEvent.input(screen.getByPlaceholderText('service.local'), {
|
fireEvent.input(screen.getByPlaceholderText('service.local'), {
|
||||||
target: { value: 'mqtt.remote.local' },
|
target: { value: 'mqtt.remote.local' },
|
||||||
});
|
});
|
||||||
fireEvent.change(runFrom, { target: { value: 'host-edge-01' } });
|
fireEvent.click(edgeLocation);
|
||||||
fireEvent.click(screen.getByRole('button', { name: 'Add service/device check' }));
|
fireEvent.click(screen.getByRole('button', { name: 'Add service/device check' }));
|
||||||
|
|
||||||
await waitFor(() =>
|
await waitFor(() =>
|
||||||
expect(mockedCreate).toHaveBeenCalledWith(
|
expect(mockedCreate).toHaveBeenCalledWith(
|
||||||
expect.objectContaining({ probeAgentId: 'host-edge-01' }),
|
expect.objectContaining({
|
||||||
|
probeAgentId: '',
|
||||||
|
observationLocationIds: ['pulse:local', 'agent:host-edge-01'],
|
||||||
|
}),
|
||||||
),
|
),
|
||||||
);
|
);
|
||||||
});
|
});
|
||||||
@@ -450,9 +453,8 @@ describe('AvailabilityTargetSlot', () => {
|
|||||||
render(() => <AvailabilityTargetSlot onCancel={vi.fn()} onSaved={vi.fn()} />);
|
render(() => <AvailabilityTargetSlot onCancel={vi.fn()} onSaved={vi.fn()} />);
|
||||||
|
|
||||||
// Discoverability is the point of the Pro gate: the control stays visible.
|
// Discoverability is the point of the Pro gate: the control stays visible.
|
||||||
const runFrom = screen.getByLabelText('Run from') as HTMLSelectElement;
|
const edgeLocation = screen.getByRole('checkbox', { name: /Edge 01/ });
|
||||||
expect(runFrom).toBeDisabled();
|
expect(edgeLocation).toBeDisabled();
|
||||||
expect(screen.getByRole('option', { name: 'Edge 01' })).toBeInTheDocument();
|
|
||||||
expect(screen.getByRole('heading', { name: 'External Probes' })).toBeInTheDocument();
|
expect(screen.getByRole('heading', { name: 'External Probes' })).toBeInTheDocument();
|
||||||
expect(screen.getByRole('link', { name: 'View plans' })).toHaveAttribute(
|
expect(screen.getByRole('link', { name: 'View plans' })).toHaveAttribute(
|
||||||
'href',
|
'href',
|
||||||
@@ -471,7 +473,12 @@ describe('AvailabilityTargetSlot', () => {
|
|||||||
fireEvent.click(screen.getByRole('button', { name: 'Add service/device check' }));
|
fireEvent.click(screen.getByRole('button', { name: 'Add service/device check' }));
|
||||||
|
|
||||||
await waitFor(() =>
|
await waitFor(() =>
|
||||||
expect(mockedCreate).toHaveBeenCalledWith(expect.objectContaining({ probeAgentId: '' })),
|
expect(mockedCreate).toHaveBeenCalledWith(
|
||||||
|
expect.objectContaining({
|
||||||
|
probeAgentId: '',
|
||||||
|
observationLocationIds: ['pulse:local'],
|
||||||
|
}),
|
||||||
|
),
|
||||||
);
|
);
|
||||||
expect(screen.queryByRole('alert')).not.toBeInTheDocument();
|
expect(screen.queryByRole('alert')).not.toBeInTheDocument();
|
||||||
});
|
});
|
||||||
@@ -501,14 +508,16 @@ describe('AvailabilityTargetSlot', () => {
|
|||||||
<AvailabilityTargetSlot editingTargetId="target-1" onCancel={vi.fn()} onSaved={vi.fn()} />
|
<AvailabilityTargetSlot editingTargetId="target-1" onCancel={vi.fn()} onSaved={vi.fn()} />
|
||||||
));
|
));
|
||||||
|
|
||||||
await waitFor(() => expect(screen.getByLabelText('Run from')).toHaveValue('host-edge-01'));
|
await waitFor(() => expect(screen.getByRole('checkbox', { name: /Edge 01/ })).toBeChecked());
|
||||||
|
|
||||||
fireEvent.change(screen.getByLabelText('Run from'), { target: { value: '' } });
|
fireEvent.click(screen.getByRole('checkbox', { name: /This Pulse server/ }));
|
||||||
|
fireEvent.click(screen.getByRole('checkbox', { name: /Edge 01/ }));
|
||||||
fireEvent.click(screen.getByRole('button', { name: 'Save target' }));
|
fireEvent.click(screen.getByRole('button', { name: 'Save target' }));
|
||||||
|
|
||||||
await waitFor(() => expect(mockedUpdate).toHaveBeenCalled());
|
await waitFor(() => expect(mockedUpdate).toHaveBeenCalled());
|
||||||
const [, payload] = mockedUpdate.mock.calls.at(-1)!;
|
const [, payload] = mockedUpdate.mock.calls.at(-1)!;
|
||||||
expect(payload.probeAgentId).toBe('');
|
expect(payload.probeAgentId).toBe('');
|
||||||
|
expect(payload.observationLocationIds).toEqual(['pulse:local']);
|
||||||
expect(Object.prototype.hasOwnProperty.call(payload, 'probeAgentId')).toBe(true);
|
expect(Object.prototype.hasOwnProperty.call(payload, 'probeAgentId')).toBe(true);
|
||||||
});
|
});
|
||||||
|
|
||||||
@@ -529,11 +538,9 @@ describe('AvailabilityTargetSlot', () => {
|
|||||||
));
|
));
|
||||||
|
|
||||||
await waitFor(() =>
|
await waitFor(() =>
|
||||||
expect(
|
expect(screen.getByRole('checkbox', { name: /host-gone/ })).toBeChecked(),
|
||||||
screen.getByRole('option', { name: 'host-gone (not currently connected)' }),
|
|
||||||
).toBeInTheDocument(),
|
|
||||||
);
|
);
|
||||||
expect(screen.getByLabelText('Run from')).toHaveValue('host-gone');
|
expect(screen.getByText('Not currently connected')).toBeInTheDocument();
|
||||||
});
|
});
|
||||||
|
|
||||||
it('falls back to the upgrade gate when the server answers 402 license_required', async () => {
|
it('falls back to the upgrade gate when the server answers 402 license_required', async () => {
|
||||||
@@ -551,13 +558,13 @@ describe('AvailabilityTargetSlot', () => {
|
|||||||
fireEvent.input(screen.getByPlaceholderText('service.local'), {
|
fireEvent.input(screen.getByPlaceholderText('service.local'), {
|
||||||
target: { value: 'mqtt.remote.local' },
|
target: { value: 'mqtt.remote.local' },
|
||||||
});
|
});
|
||||||
fireEvent.change(screen.getByLabelText('Run from'), { target: { value: 'host-edge-01' } });
|
fireEvent.click(screen.getByRole('checkbox', { name: /Edge 01/ }));
|
||||||
fireEvent.click(screen.getByRole('button', { name: 'Add service/device check' }));
|
fireEvent.click(screen.getByRole('button', { name: 'Add service/device check' }));
|
||||||
|
|
||||||
await waitFor(() =>
|
await waitFor(() =>
|
||||||
expect(screen.getByRole('heading', { name: 'External Probes' })).toBeInTheDocument(),
|
expect(screen.getByRole('heading', { name: 'External Probes' })).toBeInTheDocument(),
|
||||||
);
|
);
|
||||||
expect(screen.getByLabelText('Run from')).toBeDisabled();
|
expect(screen.getByRole('checkbox', { name: /Edge 01/ })).toBeDisabled();
|
||||||
expect(screen.getByRole('link', { name: 'View plans' })).toBeInTheDocument();
|
expect(screen.getByRole('link', { name: 'View plans' })).toBeInTheDocument();
|
||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|||||||
@@ -108,6 +108,14 @@ export function getAvailabilityTargetStatusLabel(target: AvailabilityTarget): st
|
|||||||
if (!target.enabled) return 'Paused';
|
if (!target.enabled) return 'Paused';
|
||||||
const status = target.status;
|
const status = target.status;
|
||||||
if (!status) return 'Not checked yet';
|
if (!status) return 'Not checked yet';
|
||||||
|
if (status.aggregateState === 'degraded') return 'Observation paths disagree';
|
||||||
|
if (status.aggregateState === 'unknown') {
|
||||||
|
return `${status.reportingLocations ?? 0}/${status.expectedLocations ?? status.locations?.length ?? 0} locations reporting`;
|
||||||
|
}
|
||||||
|
if (status.aggregateState === 'unavailable') return 'Unavailable from all locations';
|
||||||
|
if (status.aggregateState === 'healthy' && (status.expectedLocations ?? 0) > 1) {
|
||||||
|
return `Available from all ${status.expectedLocations} locations`;
|
||||||
|
}
|
||||||
// A probe-assigned check whose agent stopped reporting derives to
|
// A probe-assigned check whose agent stopped reporting derives to
|
||||||
// indeterminate at read time. It shares the warning treatment with the UDP
|
// indeterminate at read time. It shares the warning treatment with the UDP
|
||||||
// open-or-filtered case but needs its own copy.
|
// open-or-filtered case but needs its own copy.
|
||||||
@@ -124,6 +132,12 @@ export function getAvailabilityTargetStatusLabel(target: AvailabilityTarget): st
|
|||||||
export function getAvailabilityTargetStatusClass(target: AvailabilityTarget): string {
|
export function getAvailabilityTargetStatusClass(target: AvailabilityTarget): string {
|
||||||
if (!target.enabled) return 'bg-surface-alt text-muted';
|
if (!target.enabled) return 'bg-surface-alt text-muted';
|
||||||
if (!target.status) return 'bg-sky-100 text-sky-700 dark:bg-sky-900 dark:text-sky-300';
|
if (!target.status) return 'bg-sky-100 text-sky-700 dark:bg-sky-900 dark:text-sky-300';
|
||||||
|
if (target.status.aggregateState === 'degraded' || target.status.aggregateState === 'unknown') {
|
||||||
|
return 'bg-amber-100 text-amber-700 dark:bg-amber-900 dark:text-amber-300';
|
||||||
|
}
|
||||||
|
if (target.status.aggregateState === 'unavailable') {
|
||||||
|
return 'bg-rose-100 text-rose-700 dark:bg-rose-900 dark:text-rose-300';
|
||||||
|
}
|
||||||
if (target.status.outcome === 'indeterminate') {
|
if (target.status.outcome === 'indeterminate') {
|
||||||
return 'bg-amber-100 text-amber-700 dark:bg-amber-900 dark:text-amber-300';
|
return 'bg-amber-100 text-amber-700 dark:bg-amber-900 dark:text-amber-300';
|
||||||
}
|
}
|
||||||
@@ -141,6 +155,9 @@ export function getAvailabilityTargetProbeSourceLabel(
|
|||||||
target: AvailabilityTarget,
|
target: AvailabilityTarget,
|
||||||
probeAgentOptions: readonly ProbeAgentOption[],
|
probeAgentOptions: readonly ProbeAgentOption[],
|
||||||
): string | null {
|
): string | null {
|
||||||
|
const locationCount =
|
||||||
|
target.status?.locations?.length ?? target.observationLocationIds?.length ?? 0;
|
||||||
|
if (locationCount > 1) return `${locationCount} observation locations`;
|
||||||
return getProbeSourceChipLabel(probeAgentOptions, target.status?.probeAgentId);
|
return getProbeSourceChipLabel(probeAgentOptions, target.status?.probeAgentId);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -283,10 +283,12 @@ export const AvailabilityChecksTable: Component<{
|
|||||||
const result = () => probe()?.resultLabel ?? indicator().label;
|
const result = () => probe()?.resultLabel ?? indicator().label;
|
||||||
const target = () => formatTarget(check);
|
const target = () => formatTarget(check);
|
||||||
const probeSource = () =>
|
const probeSource = () =>
|
||||||
getProbeSourceChipLabel(
|
(availability()?.locations?.length ?? 0) > 1
|
||||||
props.probeAgentOptions ?? [],
|
? `${availability()?.locations?.length} locations · ${availability()?.reportingLocations ?? 0}/${availability()?.expectedLocations ?? availability()?.locations?.length ?? 0} reporting`
|
||||||
availability()?.probeAgentId,
|
: getProbeSourceChipLabel(
|
||||||
);
|
props.probeAgentOptions ?? [],
|
||||||
|
availability()?.probeAgentId,
|
||||||
|
);
|
||||||
const detailRowId = () => drawer.detailRowId(check);
|
const detailRowId = () => drawer.detailRowId(check);
|
||||||
const isExpanded = () => drawer.isExpanded(check);
|
const isExpanded = () => drawer.isExpanded(check);
|
||||||
|
|
||||||
|
|||||||
@@ -1532,6 +1532,11 @@ export interface ResourceAvailabilityMeta {
|
|||||||
* on the Pulse server itself.
|
* on the Pulse server itself.
|
||||||
*/
|
*/
|
||||||
probeAgentId?: string;
|
probeAgentId?: string;
|
||||||
|
aggregateState?: 'healthy' | 'degraded' | 'unavailable' | 'unknown' | string;
|
||||||
|
disagreement?: boolean;
|
||||||
|
expectedLocations?: number;
|
||||||
|
reportingLocations?: number;
|
||||||
|
locations?: ResourceAvailabilityObservationLocation[];
|
||||||
correlationState?: 'attached' | 'standalone' | 'ambiguous' | 'unresolved';
|
correlationState?: 'attached' | 'standalone' | 'ambiguous' | 'unresolved';
|
||||||
correlationRule?: string;
|
correlationRule?: string;
|
||||||
correlationReason?: string;
|
correlationReason?: string;
|
||||||
@@ -1539,6 +1544,25 @@ export interface ResourceAvailabilityMeta {
|
|||||||
evidence?: EvidenceEnvelope;
|
evidence?: EvidenceEnvelope;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
export interface ResourceAvailabilityObservationLocation {
|
||||||
|
locationId: string;
|
||||||
|
kind: 'pulse' | 'agent' | string;
|
||||||
|
probeAgentId?: string;
|
||||||
|
outcome?: string;
|
||||||
|
transportOutcome?: string;
|
||||||
|
applicationOutcome?: string;
|
||||||
|
applicationStatusCode?: number;
|
||||||
|
applicationFailureCode?: string;
|
||||||
|
available?: boolean;
|
||||||
|
lastChecked?: string;
|
||||||
|
lastSuccess?: string;
|
||||||
|
freshnessAt?: string;
|
||||||
|
latencyMillis?: number;
|
||||||
|
consecutiveFailures?: number;
|
||||||
|
lastError?: string;
|
||||||
|
stale?: boolean;
|
||||||
|
}
|
||||||
|
|
||||||
export type ResourceCertificateTrustStatus =
|
export type ResourceCertificateTrustStatus =
|
||||||
'trusted' | 'self-signed' | 'untrusted' | 'expired' | 'not-yet-valid';
|
'trusted' | 'self-signed' | 'untrusted' | 'expired' | 'not-yet-valid';
|
||||||
|
|
||||||
|
|||||||
@@ -23,6 +23,8 @@ export const EXTERNAL_PROBE_FEATURE = 'external_probe';
|
|||||||
|
|
||||||
/** Wire value for "run this check on the Pulse server itself". */
|
/** Wire value for "run this check on the Pulse server itself". */
|
||||||
export const LOCAL_PROBE_AGENT_VALUE = '';
|
export const LOCAL_PROBE_AGENT_VALUE = '';
|
||||||
|
export const LOCAL_OBSERVATION_LOCATION_ID = 'pulse:local';
|
||||||
|
export const AGENT_OBSERVATION_LOCATION_PREFIX = 'agent:';
|
||||||
|
|
||||||
/** Default option label for the local Pulse server. */
|
/** Default option label for the local Pulse server. */
|
||||||
export const LOCAL_PROBE_AGENT_LABEL = 'This Pulse server';
|
export const LOCAL_PROBE_AGENT_LABEL = 'This Pulse server';
|
||||||
@@ -40,6 +42,23 @@ export interface ProbeAgentOption {
|
|||||||
label: string;
|
label: string;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
export interface ObservationLocationOption {
|
||||||
|
id: string;
|
||||||
|
label: string;
|
||||||
|
kind: 'pulse' | 'agent';
|
||||||
|
agentId?: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
export const observationLocationIdForAgent = (agentId: string): string =>
|
||||||
|
`${AGENT_OBSERVATION_LOCATION_PREFIX}${agentId.trim()}`;
|
||||||
|
|
||||||
|
export const agentIdFromObservationLocation = (locationId?: string | null): string => {
|
||||||
|
const normalized = (locationId ?? '').trim();
|
||||||
|
return normalized.startsWith(AGENT_OBSERVATION_LOCATION_PREFIX)
|
||||||
|
? normalized.slice(AGENT_OBSERVATION_LOCATION_PREFIX.length).trim()
|
||||||
|
: '';
|
||||||
|
};
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Build the selectable probe agent hosts from unified resources.
|
* Build the selectable probe agent hosts from unified resources.
|
||||||
*
|
*
|
||||||
@@ -62,6 +81,32 @@ export function buildProbeAgentOptions(resources: readonly Resource[]): ProbeAge
|
|||||||
return [...byId.values()].sort((left, right) => left.label.localeCompare(right.label));
|
return [...byId.values()].sort((left, right) => left.label.localeCompare(right.label));
|
||||||
}
|
}
|
||||||
|
|
||||||
|
export function buildObservationLocationOptions(
|
||||||
|
resources: readonly Resource[],
|
||||||
|
): ObservationLocationOption[] {
|
||||||
|
return [
|
||||||
|
{ id: LOCAL_OBSERVATION_LOCATION_ID, label: LOCAL_PROBE_AGENT_LABEL, kind: 'pulse' },
|
||||||
|
...buildProbeAgentOptions(resources).map((option) => ({
|
||||||
|
id: observationLocationIdForAgent(option.id),
|
||||||
|
label: option.label,
|
||||||
|
kind: 'agent' as const,
|
||||||
|
agentId: option.id,
|
||||||
|
})),
|
||||||
|
];
|
||||||
|
}
|
||||||
|
|
||||||
|
export function getObservationLocationLabel(
|
||||||
|
options: readonly ObservationLocationOption[],
|
||||||
|
locationId?: string | null,
|
||||||
|
): string {
|
||||||
|
const normalized = (locationId ?? '').trim();
|
||||||
|
if (!normalized || normalized === LOCAL_OBSERVATION_LOCATION_ID) return LOCAL_PROBE_AGENT_LABEL;
|
||||||
|
return (
|
||||||
|
options.find((option) => option.id === normalized)?.label ??
|
||||||
|
(agentIdFromObservationLocation(normalized) || normalized)
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Resolve a probe agent id to its display name, falling back to the raw id when
|
* Resolve a probe agent id to its display name, falling back to the raw id when
|
||||||
* the host is not currently in the list.
|
* the host is not currently in the list.
|
||||||
|
|||||||
@@ -3,6 +3,7 @@ package api
|
|||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
"encoding/json"
|
"encoding/json"
|
||||||
|
"io"
|
||||||
"net/http"
|
"net/http"
|
||||||
"strings"
|
"strings"
|
||||||
"time"
|
"time"
|
||||||
@@ -78,7 +79,8 @@ func (f availabilityFeatureResolverFunc) FeatureService(ctx context.Context) lic
|
|||||||
// Unassigned targets are the community behaviour and must never consult the
|
// Unassigned targets are the community behaviour and must never consult the
|
||||||
// license path.
|
// license path.
|
||||||
func (h *AvailabilityHandlers) requireProbeAssignment(w http.ResponseWriter, r *http.Request, target config.AvailabilityTarget) bool {
|
func (h *AvailabilityHandlers) requireProbeAssignment(w http.ResponseWriter, r *http.Request, target config.AvailabilityTarget) bool {
|
||||||
if strings.TrimSpace(target.ProbeAgentID) == "" {
|
agentIDs := target.AssignedProbeAgentIDs()
|
||||||
|
if len(agentIDs) == 0 {
|
||||||
return true
|
return true
|
||||||
}
|
}
|
||||||
if h == nil || h.licenseResolver == nil {
|
if h == nil || h.licenseResolver == nil {
|
||||||
@@ -94,10 +96,12 @@ func (h *AvailabilityHandlers) requireProbeAssignment(w http.ResponseWriter, r *
|
|||||||
WriteLicenseRequired(w, featureExternalProbeValue, err.Error())
|
WriteLicenseRequired(w, featureExternalProbeValue, err.Error())
|
||||||
return false
|
return false
|
||||||
}
|
}
|
||||||
if !h.probeAgentExists(r.Context(), target.ProbeAgentID) {
|
for _, agentID := range agentIDs {
|
||||||
writeErrorResponse(w, http.StatusBadRequest, "unknown_probe_agent",
|
if !h.probeAgentExists(r.Context(), agentID) {
|
||||||
"Probe agent "+target.ProbeAgentID+" is not a registered host agent", nil)
|
writeErrorResponse(w, http.StatusBadRequest, "unknown_probe_agent",
|
||||||
return false
|
"Observation location agent "+agentID+" is not a registered host agent", nil)
|
||||||
|
return false
|
||||||
|
}
|
||||||
}
|
}
|
||||||
return true
|
return true
|
||||||
}
|
}
|
||||||
@@ -384,10 +388,31 @@ func decodeAvailabilityTargetRequest(w http.ResponseWriter, r *http.Request, bas
|
|||||||
r.Body = http.MaxBytesReader(w, r.Body, 16*1024)
|
r.Body = http.MaxBytesReader(w, r.Body, 16*1024)
|
||||||
defer r.Body.Close()
|
defer r.Body.Close()
|
||||||
target := cloneAvailabilityTarget(base)
|
target := cloneAvailabilityTarget(base)
|
||||||
if err := json.NewDecoder(r.Body).Decode(&target); err != nil {
|
body, err := io.ReadAll(r.Body)
|
||||||
|
if err != nil || json.Unmarshal(body, &target) != nil {
|
||||||
writeErrorResponse(w, http.StatusBadRequest, "invalid_request", "Invalid JSON body", nil)
|
writeErrorResponse(w, http.StatusBadRequest, "invalid_request", "Invalid JSON body", nil)
|
||||||
return config.AvailabilityTarget{}, false
|
return config.AvailabilityTarget{}, false
|
||||||
}
|
}
|
||||||
|
var compatibility struct {
|
||||||
|
ProbeAgentID *string `json:"probeAgentId"`
|
||||||
|
ObservationLocationIDs *[]string `json:"observationLocationIds"`
|
||||||
|
}
|
||||||
|
if json.Unmarshal(body, &compatibility) == nil && compatibility.ObservationLocationIDs == nil && compatibility.ProbeAgentID != nil {
|
||||||
|
agentID := strings.TrimSpace(*compatibility.ProbeAgentID)
|
||||||
|
if agentID == "" {
|
||||||
|
target.ObservationLocationIDs = []string{config.AvailabilityObservationLocationLocal}
|
||||||
|
} else {
|
||||||
|
target.ObservationLocationIDs = []string{config.AvailabilityAgentObservationLocationID(agentID)}
|
||||||
|
}
|
||||||
|
} else if compatibility.ProbeAgentID != nil && compatibility.ObservationLocationIDs != nil {
|
||||||
|
agentID := strings.TrimSpace(*compatibility.ProbeAgentID)
|
||||||
|
if agentID != "" && len(*compatibility.ObservationLocationIDs) == 1 {
|
||||||
|
// Full-object pre-location clients may echo the canonical field added by
|
||||||
|
// the server while editing only probeAgentId. Honor that single-source
|
||||||
|
// edit; multi-location clients send a set the legacy field cannot express.
|
||||||
|
target.ObservationLocationIDs = []string{config.AvailabilityAgentObservationLocationID(agentID)}
|
||||||
|
}
|
||||||
|
}
|
||||||
return target, true
|
return target, true
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -487,6 +512,7 @@ func sameAvailabilityHTTPOrigin(previous, next config.AvailabilityTarget) bool {
|
|||||||
|
|
||||||
func cloneAvailabilityTarget(target config.AvailabilityTarget) config.AvailabilityTarget {
|
func cloneAvailabilityTarget(target config.AvailabilityTarget) config.AvailabilityTarget {
|
||||||
clone := target
|
clone := target
|
||||||
|
clone.ObservationLocationIDs = append([]string(nil), target.ObservationLocationIDs...)
|
||||||
if target.HTTP == nil {
|
if target.HTTP == nil {
|
||||||
return clone
|
return clone
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -154,6 +154,42 @@ func TestAvailabilityHandlersAcceptProbeAssignmentWithLicense(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestAvailabilityHandlersAcceptSeveralObservationLocationsOnOneTarget(t *testing.T) {
|
||||||
|
persistence := config.NewConfigPersistence(t.TempDir())
|
||||||
|
monitor := monitorWithHostAgent(t, "edge-a")
|
||||||
|
if _, err := monitor.ApplyHostReport(agentshost.Report{
|
||||||
|
Agent: agentshost.AgentInfo{ID: "edge-b-agent", Version: "6.1.1", IntervalSeconds: 30},
|
||||||
|
Host: agentshost.HostInfo{ID: "edge-b", MachineID: "edge-b", Hostname: "edge-b", Platform: "linux"},
|
||||||
|
Timestamp: time.Now().UTC(),
|
||||||
|
}, &config.APITokenRecord{ID: "token-edge-b", Name: "edge-b"}); err != nil {
|
||||||
|
t.Fatalf("ApplyHostReport() error = %v", err)
|
||||||
|
}
|
||||||
|
handler := NewAvailabilityHandlers(
|
||||||
|
func(context.Context) *config.ConfigPersistence { return persistence },
|
||||||
|
func(context.Context) *monitoring.Monitor { return monitor },
|
||||||
|
stubFeatureResolver(featureExternalProbeValue),
|
||||||
|
)
|
||||||
|
target := config.AvailabilityTarget{
|
||||||
|
Name: "Customer API",
|
||||||
|
Address: "api.service.local",
|
||||||
|
Protocol: config.AvailabilityProbeHTTPS,
|
||||||
|
Enabled: true,
|
||||||
|
ObservationLocationIDs: []string{config.AvailabilityObservationLocationLocal, "agent:edge-a", "agent:edge-b"},
|
||||||
|
}
|
||||||
|
rec := httptest.NewRecorder()
|
||||||
|
handler.HandleAdd(rec, httptest.NewRequest(http.MethodPost, "/api/availability-targets", availabilityRequestBody(t, target)))
|
||||||
|
if rec.Code != http.StatusCreated {
|
||||||
|
t.Fatalf("HandleAdd status = %d, want 201; body=%s", rec.Code, rec.Body.String())
|
||||||
|
}
|
||||||
|
var created config.AvailabilityTarget
|
||||||
|
if err := json.NewDecoder(rec.Body).Decode(&created); err != nil {
|
||||||
|
t.Fatalf("decode created target: %v", err)
|
||||||
|
}
|
||||||
|
if len(created.ObservationLocationIDs) != 3 || created.ProbeAgentID != "" {
|
||||||
|
t.Fatalf("created target = %+v, want one logical target with three locations", created)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestAvailabilityHandlersRejectUnknownProbeAgent(t *testing.T) {
|
func TestAvailabilityHandlersRejectUnknownProbeAgent(t *testing.T) {
|
||||||
persistence := config.NewConfigPersistence(t.TempDir())
|
persistence := config.NewConfigPersistence(t.TempDir())
|
||||||
monitor := monitorWithHostAgent(t, "probe-host")
|
monitor := monitorWithHostAgent(t, "probe-host")
|
||||||
|
|||||||
@@ -1497,6 +1497,12 @@ func deriveAvailabilityConnectionState(target config.AvailabilityTarget, status
|
|||||||
if !target.Enabled {
|
if !target.Enabled {
|
||||||
return ConnectionStatePaused, "paused by user", lastSeen, lastError
|
return ConnectionStatePaused, "paused by user", lastSeen, lastError
|
||||||
}
|
}
|
||||||
|
if status.AggregateState == monitoring.AvailabilityAggregateUnknown {
|
||||||
|
return ConnectionStateStale, "observation coverage is incomplete", lastSeen, lastError
|
||||||
|
}
|
||||||
|
if status.AggregateState == monitoring.AvailabilityAggregateDegraded {
|
||||||
|
return ConnectionStateActive, "observation locations disagree", lastSeen, lastError
|
||||||
|
}
|
||||||
if status.LastChecked.IsZero() {
|
if status.LastChecked.IsZero() {
|
||||||
return ConnectionStatePending, "awaiting first probe", nil, nil
|
return ConnectionStatePending, "awaiting first probe", nil, nil
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -274,7 +274,7 @@ func mockAvailabilityTestResponse(targetID string) (availabilityTestResponse, bo
|
|||||||
}
|
}
|
||||||
|
|
||||||
func mockAvailabilityConfigTarget(target mock.AvailabilityTargetFixture) config.AvailabilityTarget {
|
func mockAvailabilityConfigTarget(target mock.AvailabilityTargetFixture) config.AvailabilityTarget {
|
||||||
return config.NormalizeAvailabilityTarget(config.AvailabilityTarget{
|
configTarget := config.AvailabilityTarget{
|
||||||
ID: target.ID,
|
ID: target.ID,
|
||||||
Name: target.Name,
|
Name: target.Name,
|
||||||
TargetKind: config.AvailabilityTargetKind(target.TargetKind),
|
TargetKind: config.AvailabilityTargetKind(target.TargetKind),
|
||||||
@@ -286,11 +286,18 @@ func mockAvailabilityConfigTarget(target mock.AvailabilityTargetFixture) config.
|
|||||||
PollIntervalSecs: target.PollIntervalSecs,
|
PollIntervalSecs: target.PollIntervalSecs,
|
||||||
TimeoutMillis: target.TimeoutMillis,
|
TimeoutMillis: target.TimeoutMillis,
|
||||||
FailureThreshold: target.FailureThreshold,
|
FailureThreshold: target.FailureThreshold,
|
||||||
})
|
}
|
||||||
|
if target.ID == "mock-availability-docker-frontend-service" {
|
||||||
|
configTarget.ObservationLocationIDs = []string{
|
||||||
|
config.AvailabilityObservationLocationLocal,
|
||||||
|
config.AvailabilityAgentObservationLocationID("edge-london"),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return config.NormalizeAvailabilityTarget(configTarget)
|
||||||
}
|
}
|
||||||
|
|
||||||
func mockAvailabilityProbeStatus(fixture mock.AvailabilityFixture, target config.AvailabilityTarget) monitoring.AvailabilityProbeStatus {
|
func mockAvailabilityProbeStatus(fixture mock.AvailabilityFixture, target config.AvailabilityTarget) monitoring.AvailabilityProbeStatus {
|
||||||
return monitoring.AvailabilityProbeStatus{
|
status := monitoring.AvailabilityProbeStatus{
|
||||||
TargetID: target.ID,
|
TargetID: target.ID,
|
||||||
Name: target.DisplayName(),
|
Name: target.DisplayName(),
|
||||||
TargetKind: string(target.TargetKind),
|
TargetKind: string(target.TargetKind),
|
||||||
@@ -305,6 +312,36 @@ func mockAvailabilityProbeStatus(fixture mock.AvailabilityFixture, target config
|
|||||||
LastError: fixture.LastError,
|
LastError: fixture.LastError,
|
||||||
FailureThreshold: target.EffectiveFailureThreshold(),
|
FailureThreshold: target.EffectiveFailureThreshold(),
|
||||||
}
|
}
|
||||||
|
if target.ID == "mock-availability-docker-frontend-service" {
|
||||||
|
status.AggregateState = monitoring.AvailabilityAggregateDegraded
|
||||||
|
status.Disagreement = true
|
||||||
|
status.ExpectedLocations = 2
|
||||||
|
status.ReportingLocations = 2
|
||||||
|
status.Locations = []monitoring.AvailabilityObservationLocationStatus{
|
||||||
|
{
|
||||||
|
LocationID: config.AvailabilityObservationLocationLocal,
|
||||||
|
Kind: "pulse",
|
||||||
|
Outcome: string(monitoring.AvailabilityProbeReachable),
|
||||||
|
Available: true,
|
||||||
|
LastChecked: fixture.LastChecked,
|
||||||
|
LastSuccess: fixture.LastSuccess,
|
||||||
|
FreshnessAt: fixture.LastChecked,
|
||||||
|
LatencyMillis: fixture.LatencyMillis,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
LocationID: config.AvailabilityAgentObservationLocationID("edge-london"),
|
||||||
|
Kind: "agent",
|
||||||
|
ProbeAgentID: "edge-london",
|
||||||
|
Outcome: string(monitoring.AvailabilityProbeUnreachable),
|
||||||
|
Available: false,
|
||||||
|
LastChecked: fixture.LastChecked,
|
||||||
|
FreshnessAt: fixture.LastChecked,
|
||||||
|
ConsecutiveFailures: 2,
|
||||||
|
LastError: "connection timed out",
|
||||||
|
},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return status
|
||||||
}
|
}
|
||||||
|
|
||||||
func mockPlatformTimePointer(value time.Time) *time.Time {
|
func mockPlatformTimePointer(value time.Time) *time.Time {
|
||||||
|
|||||||
@@ -5,6 +5,7 @@ import (
|
|||||||
"net"
|
"net"
|
||||||
"net/url"
|
"net/url"
|
||||||
"reflect"
|
"reflect"
|
||||||
|
"sort"
|
||||||
"strconv"
|
"strconv"
|
||||||
"strings"
|
"strings"
|
||||||
|
|
||||||
@@ -12,12 +13,14 @@ import (
|
|||||||
)
|
)
|
||||||
|
|
||||||
const (
|
const (
|
||||||
DefaultAvailabilityPollIntervalSecs = 60
|
DefaultAvailabilityPollIntervalSecs = 60
|
||||||
DefaultAvailabilityTimeoutMillis = 2000
|
DefaultAvailabilityTimeoutMillis = 2000
|
||||||
DefaultAvailabilityFailureThreshold = 2
|
DefaultAvailabilityFailureThreshold = 2
|
||||||
DefaultCertificateExpiryWarningDays = 30
|
DefaultCertificateExpiryWarningDays = 30
|
||||||
MaxAvailabilityHTTPBodyBytes = 8192
|
MaxAvailabilityHTTPBodyBytes = 8192
|
||||||
MaxAvailabilityHTTPResponseBytes = 65536
|
MaxAvailabilityHTTPResponseBytes = 65536
|
||||||
|
AvailabilityObservationLocationLocal = "pulse:local"
|
||||||
|
AvailabilityObservationLocationAgentPrefix = "agent:"
|
||||||
)
|
)
|
||||||
|
|
||||||
type AvailabilityHTTPMethod string
|
type AvailabilityHTTPMethod string
|
||||||
@@ -125,9 +128,14 @@ type AvailabilityTarget struct {
|
|||||||
CertificateMonitoringDisabled bool `json:"certificateMonitoringDisabled,omitempty"`
|
CertificateMonitoringDisabled bool `json:"certificateMonitoringDisabled,omitempty"`
|
||||||
CertificateExpiryWarningDays int `json:"certificateExpiryWarningDays,omitempty"`
|
CertificateExpiryWarningDays int `json:"certificateExpiryWarningDays,omitempty"`
|
||||||
// ProbeAgentID assigns execution to a remote host agent. Empty means the
|
// ProbeAgentID assigns execution to a remote host agent. Empty means the
|
||||||
// check runs from the local Pulse instance. Agent existence is validated at
|
// check runs from the local Pulse instance. It remains a compatibility field
|
||||||
// the API layer, not here, because config has no view of monitor state.
|
// for pre-location clients; ObservationLocationIDs is canonical.
|
||||||
ProbeAgentID string `json:"probeAgentId,omitempty"`
|
ProbeAgentID string `json:"probeAgentId,omitempty"`
|
||||||
|
// ObservationLocationIDs selects every source-owned path that observes this
|
||||||
|
// one logical verification. pulse:local is the Pulse server and agent:<id>
|
||||||
|
// names an eligible connected host agent. Names and network details stay out
|
||||||
|
// of persisted target configuration and are resolved at presentation time.
|
||||||
|
ObservationLocationIDs []string `json:"observationLocationIds,omitempty"`
|
||||||
// HTTP is absent for legacy targets. Its request body, credentials, and
|
// HTTP is absent for legacy targets. Its request body, credentials, and
|
||||||
// header values are encrypted with the rest of availability target storage
|
// header values are encrypted with the rest of availability target storage
|
||||||
// and are never returned by the API.
|
// and are never returned by the API.
|
||||||
@@ -227,6 +235,7 @@ func (t AvailabilityTarget) EffectiveFailureThreshold() int {
|
|||||||
// executed or where it executes. Display, correlation, alert-threshold, and
|
// executed or where it executes. Display, correlation, alert-threshold, and
|
||||||
// certificate-presentation edits intentionally stay within the same revision.
|
// certificate-presentation edits intentionally stay within the same revision.
|
||||||
func AvailabilityExecutionConfigChanged(previous, next AvailabilityTarget) bool {
|
func AvailabilityExecutionConfigChanged(previous, next AvailabilityTarget) bool {
|
||||||
|
legacyProbeAssignmentChanged := strings.TrimSpace(previous.ProbeAgentID) != strings.TrimSpace(next.ProbeAgentID)
|
||||||
previous = NormalizeAvailabilityTarget(previous)
|
previous = NormalizeAvailabilityTarget(previous)
|
||||||
next = NormalizeAvailabilityTarget(next)
|
next = NormalizeAvailabilityTarget(next)
|
||||||
return previous.Address != next.Address ||
|
return previous.Address != next.Address ||
|
||||||
@@ -239,7 +248,67 @@ func AvailabilityExecutionConfigChanged(previous, next AvailabilityTarget) bool
|
|||||||
!reflect.DeepEqual(previous.HTTP, next.HTTP) ||
|
!reflect.DeepEqual(previous.HTTP, next.HTTP) ||
|
||||||
previous.EffectiveTimeoutMillis() != next.EffectiveTimeoutMillis() ||
|
previous.EffectiveTimeoutMillis() != next.EffectiveTimeoutMillis() ||
|
||||||
previous.EffectivePollIntervalSecs() != next.EffectivePollIntervalSecs() ||
|
previous.EffectivePollIntervalSecs() != next.EffectivePollIntervalSecs() ||
|
||||||
previous.ProbeAgentID != next.ProbeAgentID
|
legacyProbeAssignmentChanged ||
|
||||||
|
!reflect.DeepEqual(previous.EffectiveObservationLocationIDs(), next.EffectiveObservationLocationIDs())
|
||||||
|
}
|
||||||
|
|
||||||
|
func AvailabilityAgentObservationLocationID(agentID string) string {
|
||||||
|
agentID = strings.TrimSpace(agentID)
|
||||||
|
if agentID == "" {
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
return AvailabilityObservationLocationAgentPrefix + agentID
|
||||||
|
}
|
||||||
|
|
||||||
|
func AvailabilityObservationLocationAgentID(locationID string) string {
|
||||||
|
locationID = strings.TrimSpace(locationID)
|
||||||
|
if !strings.HasPrefix(locationID, AvailabilityObservationLocationAgentPrefix) {
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
return strings.TrimSpace(strings.TrimPrefix(locationID, AvailabilityObservationLocationAgentPrefix))
|
||||||
|
}
|
||||||
|
|
||||||
|
func (t AvailabilityTarget) EffectiveObservationLocationIDs() []string {
|
||||||
|
locations := normalizeAvailabilityObservationLocationIDs(t.ObservationLocationIDs)
|
||||||
|
if len(locations) > 0 {
|
||||||
|
return locations
|
||||||
|
}
|
||||||
|
if agentID := strings.TrimSpace(t.ProbeAgentID); agentID != "" {
|
||||||
|
return []string{AvailabilityAgentObservationLocationID(agentID)}
|
||||||
|
}
|
||||||
|
return []string{AvailabilityObservationLocationLocal}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (t AvailabilityTarget) IncludesLocalObservation() bool {
|
||||||
|
for _, locationID := range t.EffectiveObservationLocationIDs() {
|
||||||
|
if locationID == AvailabilityObservationLocationLocal {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
|
||||||
|
func (t AvailabilityTarget) AssignedProbeAgentIDs() []string {
|
||||||
|
agents := make([]string, 0, len(t.ObservationLocationIDs))
|
||||||
|
for _, locationID := range t.EffectiveObservationLocationIDs() {
|
||||||
|
if agentID := AvailabilityObservationLocationAgentID(locationID); agentID != "" {
|
||||||
|
agents = append(agents, agentID)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return agents
|
||||||
|
}
|
||||||
|
|
||||||
|
func (t AvailabilityTarget) IsAssignedToProbeAgent(agentID string) bool {
|
||||||
|
agentID = strings.TrimSpace(agentID)
|
||||||
|
if agentID == "" {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
for _, assignedID := range t.AssignedProbeAgentIDs() {
|
||||||
|
if assignedID == agentID {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return false
|
||||||
}
|
}
|
||||||
|
|
||||||
func (t AvailabilityTarget) CertificateMonitoringEnabled() bool {
|
func (t AvailabilityTarget) CertificateMonitoringEnabled() bool {
|
||||||
@@ -331,6 +400,21 @@ func (t AvailabilityTarget) Validate() error {
|
|||||||
if t.FailureThreshold > 0 && t.FailureThreshold > 10 {
|
if t.FailureThreshold > 0 && t.FailureThreshold > 10 {
|
||||||
return fmt.Errorf("availability failure threshold must be 10 or less")
|
return fmt.Errorf("availability failure threshold must be 10 or less")
|
||||||
}
|
}
|
||||||
|
locations := t.EffectiveObservationLocationIDs()
|
||||||
|
if len(locations) == 0 {
|
||||||
|
return fmt.Errorf("availability target requires at least one observation location")
|
||||||
|
}
|
||||||
|
if len(locations) > 16 {
|
||||||
|
return fmt.Errorf("availability target supports at most 16 observation locations")
|
||||||
|
}
|
||||||
|
for _, locationID := range locations {
|
||||||
|
if locationID == AvailabilityObservationLocationLocal {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if AvailabilityObservationLocationAgentID(locationID) == "" {
|
||||||
|
return fmt.Errorf("unsupported availability observation location %q", locationID)
|
||||||
|
}
|
||||||
|
}
|
||||||
if protocol == AvailabilityProbeHTTP || protocol == AvailabilityProbeHTTPS {
|
if protocol == AvailabilityProbeHTTP || protocol == AvailabilityProbeHTTPS {
|
||||||
if _, err := t.HTTPURL(); err != nil {
|
if _, err := t.HTTPURL(); err != nil {
|
||||||
return err
|
return err
|
||||||
@@ -403,6 +487,24 @@ func NormalizeAvailabilityTarget(target AvailabilityTarget) AvailabilityTarget {
|
|||||||
}
|
}
|
||||||
target.LinkedResourceID = strings.TrimSpace(target.LinkedResourceID)
|
target.LinkedResourceID = strings.TrimSpace(target.LinkedResourceID)
|
||||||
target.ProbeAgentID = strings.TrimSpace(target.ProbeAgentID)
|
target.ProbeAgentID = strings.TrimSpace(target.ProbeAgentID)
|
||||||
|
target.ObservationLocationIDs = normalizeAvailabilityObservationLocationIDs(target.ObservationLocationIDs)
|
||||||
|
if target.ProbeAgentID != "" && len(target.ObservationLocationIDs) == 1 && target.ObservationLocationIDs[0] == AvailabilityObservationLocationLocal {
|
||||||
|
// A pre-location client edits ProbeAgentID on a normalized local target.
|
||||||
|
target.ObservationLocationIDs = []string{AvailabilityAgentObservationLocationID(target.ProbeAgentID)}
|
||||||
|
}
|
||||||
|
if len(target.ObservationLocationIDs) == 0 {
|
||||||
|
if target.ProbeAgentID != "" {
|
||||||
|
target.ObservationLocationIDs = []string{AvailabilityAgentObservationLocationID(target.ProbeAgentID)}
|
||||||
|
} else {
|
||||||
|
target.ObservationLocationIDs = []string{AvailabilityObservationLocationLocal}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
// Keep the legacy field truthful only for the shape it can represent.
|
||||||
|
if len(target.ObservationLocationIDs) == 1 {
|
||||||
|
target.ProbeAgentID = AvailabilityObservationLocationAgentID(target.ObservationLocationIDs[0])
|
||||||
|
} else {
|
||||||
|
target.ProbeAgentID = ""
|
||||||
|
}
|
||||||
if target.HTTP != nil {
|
if target.HTTP != nil {
|
||||||
target.HTTP.Method = AvailabilityHTTPMethod(strings.ToUpper(strings.TrimSpace(string(target.HTTP.Method))))
|
target.HTTP.Method = AvailabilityHTTPMethod(strings.ToUpper(strings.TrimSpace(string(target.HTTP.Method))))
|
||||||
target.HTTP.Authentication.Type = AvailabilityHTTPAuthType(strings.ToLower(strings.TrimSpace(string(target.HTTP.Authentication.Type))))
|
target.HTTP.Authentication.Type = AvailabilityHTTPAuthType(strings.ToLower(strings.TrimSpace(string(target.HTTP.Authentication.Type))))
|
||||||
@@ -432,6 +534,41 @@ func NormalizeAvailabilityTarget(target AvailabilityTarget) AvailabilityTarget {
|
|||||||
return target
|
return target
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func normalizeAvailabilityObservationLocationIDs(locationIDs []string) []string {
|
||||||
|
if len(locationIDs) == 0 {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
seen := make(map[string]struct{}, len(locationIDs))
|
||||||
|
locations := make([]string, 0, len(locationIDs))
|
||||||
|
for _, raw := range locationIDs {
|
||||||
|
locationID := strings.TrimSpace(raw)
|
||||||
|
if locationID == "local" {
|
||||||
|
locationID = AvailabilityObservationLocationLocal
|
||||||
|
}
|
||||||
|
if agentID := AvailabilityObservationLocationAgentID(locationID); agentID != "" {
|
||||||
|
locationID = AvailabilityAgentObservationLocationID(agentID)
|
||||||
|
}
|
||||||
|
if locationID == "" {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if _, ok := seen[locationID]; ok {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
seen[locationID] = struct{}{}
|
||||||
|
locations = append(locations, locationID)
|
||||||
|
}
|
||||||
|
sort.SliceStable(locations, func(i, j int) bool {
|
||||||
|
if locations[i] == AvailabilityObservationLocationLocal {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
if locations[j] == AvailabilityObservationLocationLocal {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
return locations[i] < locations[j]
|
||||||
|
})
|
||||||
|
return locations
|
||||||
|
}
|
||||||
|
|
||||||
func validateAvailabilityHTTPConfig(contract *AvailabilityHTTPConfig) error {
|
func validateAvailabilityHTTPConfig(contract *AvailabilityHTTPConfig) error {
|
||||||
if contract == nil {
|
if contract == nil {
|
||||||
return nil
|
return nil
|
||||||
|
|||||||
@@ -2,6 +2,7 @@ package config
|
|||||||
|
|
||||||
import (
|
import (
|
||||||
"os"
|
"os"
|
||||||
|
"reflect"
|
||||||
"strings"
|
"strings"
|
||||||
"testing"
|
"testing"
|
||||||
)
|
)
|
||||||
@@ -312,6 +313,44 @@ func TestNormalizeAvailabilityTargetTrimsProbeAgentID(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestNormalizeAvailabilityTargetBuildsCanonicalObservationLocationSet(t *testing.T) {
|
||||||
|
target := NormalizeAvailabilityTarget(AvailabilityTarget{
|
||||||
|
Address: "service.local",
|
||||||
|
Protocol: AvailabilityProbeICMP,
|
||||||
|
Enabled: true,
|
||||||
|
ObservationLocationIDs: []string{" agent:edge-b ", "local", "agent:edge-a", "agent:edge-a"},
|
||||||
|
})
|
||||||
|
want := []string{AvailabilityObservationLocationLocal, "agent:edge-a", "agent:edge-b"}
|
||||||
|
if !reflect.DeepEqual(target.ObservationLocationIDs, want) {
|
||||||
|
t.Fatalf("ObservationLocationIDs = %#v, want %#v", target.ObservationLocationIDs, want)
|
||||||
|
}
|
||||||
|
if target.ProbeAgentID != "" {
|
||||||
|
t.Fatalf("ProbeAgentID = %q, multi-location targets must not claim one owner", target.ProbeAgentID)
|
||||||
|
}
|
||||||
|
if !target.IncludesLocalObservation() || !reflect.DeepEqual(target.AssignedProbeAgentIDs(), []string{"edge-a", "edge-b"}) {
|
||||||
|
t.Fatalf("normalized locations = local %v agents %#v", target.IncludesLocalObservation(), target.AssignedProbeAgentIDs())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestAvailabilityExecutionConfigChangedTracksObservationLocationSet(t *testing.T) {
|
||||||
|
base := NormalizeAvailabilityTarget(AvailabilityTarget{
|
||||||
|
Address: "service.local",
|
||||||
|
Protocol: AvailabilityProbeICMP,
|
||||||
|
Enabled: true,
|
||||||
|
ObservationLocationIDs: []string{AvailabilityObservationLocationLocal, "agent:edge-a"},
|
||||||
|
})
|
||||||
|
reordered := base
|
||||||
|
reordered.ObservationLocationIDs = []string{"agent:edge-a", AvailabilityObservationLocationLocal}
|
||||||
|
if AvailabilityExecutionConfigChanged(base, reordered) {
|
||||||
|
t.Fatal("location ordering changed the execution revision")
|
||||||
|
}
|
||||||
|
expanded := base
|
||||||
|
expanded.ObservationLocationIDs = append(expanded.ObservationLocationIDs, "agent:edge-b")
|
||||||
|
if !AvailabilityExecutionConfigChanged(base, expanded) {
|
||||||
|
t.Fatal("adding an observation location did not change the execution revision")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestAvailabilityTargetProbeAgentIDRoundTripsThroughPersistence(t *testing.T) {
|
func TestAvailabilityTargetProbeAgentIDRoundTripsThroughPersistence(t *testing.T) {
|
||||||
persistence := NewConfigPersistence(t.TempDir())
|
persistence := NewConfigPersistence(t.TempDir())
|
||||||
targets := []AvailabilityTarget{
|
targets := []AvailabilityTarget{
|
||||||
|
|||||||
@@ -349,6 +349,35 @@ func availabilityFixtureRecord(fixture AvailabilityFixture, now time.Time) (unif
|
|||||||
PollIntervalSeconds: target.effectivePollIntervalSecs(),
|
PollIntervalSeconds: target.effectivePollIntervalSecs(),
|
||||||
TimeoutMillis: target.effectiveTimeoutMillis(),
|
TimeoutMillis: target.effectiveTimeoutMillis(),
|
||||||
}
|
}
|
||||||
|
if target.ID == "mock-availability-docker-frontend-service" {
|
||||||
|
data.AggregateState = "degraded"
|
||||||
|
data.Disagreement = true
|
||||||
|
data.ExpectedLocations = 2
|
||||||
|
data.ReportingLocations = 2
|
||||||
|
data.Locations = []unifiedresources.AvailabilityObservationLocation{
|
||||||
|
{
|
||||||
|
LocationID: "pulse:local",
|
||||||
|
Kind: "pulse",
|
||||||
|
Outcome: "reachable",
|
||||||
|
Available: true,
|
||||||
|
LastChecked: availabilityFixtureTimePointer(fixture.LastChecked),
|
||||||
|
LastSuccess: availabilityFixtureTimePointer(fixture.LastSuccess),
|
||||||
|
FreshnessAt: availabilityFixtureTimePointer(fixture.LastChecked),
|
||||||
|
LatencyMillis: fixture.LatencyMillis,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
LocationID: "agent:edge-london",
|
||||||
|
Kind: "agent",
|
||||||
|
ProbeAgentID: "edge-london",
|
||||||
|
Outcome: "unreachable",
|
||||||
|
Available: false,
|
||||||
|
LastChecked: availabilityFixtureTimePointer(fixture.LastChecked),
|
||||||
|
FreshnessAt: availabilityFixtureTimePointer(fixture.LastChecked),
|
||||||
|
ConsecutiveFailures: 2,
|
||||||
|
LastError: "connection timed out",
|
||||||
|
},
|
||||||
|
}
|
||||||
|
}
|
||||||
data.Evidence = mockAvailabilityEvidence(target, fixture, lastSeen, now)
|
data.Evidence = mockAvailabilityEvidence(target, fixture, lastSeen, now)
|
||||||
if target.Protocol == mockAvailabilityProbeHTTPS {
|
if target.Protocol == mockAvailabilityProbeHTTPS {
|
||||||
data.CertificateMonitoring = true
|
data.CertificateMonitoring = true
|
||||||
@@ -377,6 +406,9 @@ func availabilityFixtureRecord(fixture AvailabilityFixture, now time.Time) (unif
|
|||||||
Tags: availabilityFixtureTags(target),
|
Tags: availabilityFixtureTags(target),
|
||||||
Availability: data,
|
Availability: data,
|
||||||
}
|
}
|
||||||
|
if data.AggregateState == "degraded" {
|
||||||
|
resource.Status = unifiedresources.StatusWarning
|
||||||
|
}
|
||||||
if incident := availabilityFixtureIncident(target, fixture, lastSeen); incident != nil {
|
if incident := availabilityFixtureIncident(target, fixture, lastSeen); incident != nil {
|
||||||
resource.Incidents = []unifiedresources.ResourceIncident{*incident}
|
resource.Incidents = []unifiedresources.ResourceIncident{*incident}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -25,27 +25,32 @@ type tlsCert = tlsutil.CertificateObservation
|
|||||||
// AvailabilityProbeStatus captures the last observed state of an agentless
|
// AvailabilityProbeStatus captures the last observed state of an agentless
|
||||||
// endpoint probe.
|
// endpoint probe.
|
||||||
type AvailabilityProbeStatus struct {
|
type AvailabilityProbeStatus struct {
|
||||||
TargetID string `json:"targetId"`
|
TargetID string `json:"targetId"`
|
||||||
Name string `json:"name"`
|
Name string `json:"name"`
|
||||||
TargetKind string `json:"targetKind,omitempty"`
|
TargetKind string `json:"targetKind,omitempty"`
|
||||||
Address string `json:"address"`
|
Address string `json:"address"`
|
||||||
Protocol string `json:"protocol"`
|
Protocol string `json:"protocol"`
|
||||||
Outcome string `json:"outcome,omitempty"`
|
Outcome string `json:"outcome,omitempty"`
|
||||||
TransportOutcome string `json:"transportOutcome,omitempty"`
|
TransportOutcome string `json:"transportOutcome,omitempty"`
|
||||||
ApplicationOutcome string `json:"applicationOutcome,omitempty"`
|
ApplicationOutcome string `json:"applicationOutcome,omitempty"`
|
||||||
ApplicationStatusCode int `json:"applicationStatusCode,omitempty"`
|
ApplicationStatusCode int `json:"applicationStatusCode,omitempty"`
|
||||||
ApplicationFailureCode string `json:"applicationFailureCode,omitempty"`
|
ApplicationFailureCode string `json:"applicationFailureCode,omitempty"`
|
||||||
Enabled bool `json:"enabled"`
|
Enabled bool `json:"enabled"`
|
||||||
Available bool `json:"available"`
|
Available bool `json:"available"`
|
||||||
LastChecked time.Time `json:"lastChecked,omitempty"`
|
LastChecked time.Time `json:"lastChecked,omitempty"`
|
||||||
LastSuccess time.Time `json:"lastSuccess,omitempty"`
|
LastSuccess time.Time `json:"lastSuccess,omitempty"`
|
||||||
LatencyMillis int64 `json:"latencyMillis,omitempty"`
|
LatencyMillis int64 `json:"latencyMillis,omitempty"`
|
||||||
ConsecutiveFailures int `json:"consecutiveFailures,omitempty"`
|
ConsecutiveFailures int `json:"consecutiveFailures,omitempty"`
|
||||||
LastError string `json:"lastError,omitempty"`
|
LastError string `json:"lastError,omitempty"`
|
||||||
FailureThreshold int `json:"failureThreshold,omitempty"`
|
FailureThreshold int `json:"failureThreshold,omitempty"`
|
||||||
ProbeAgentID string `json:"probeAgentId,omitempty"`
|
ProbeAgentID string `json:"probeAgentId,omitempty"`
|
||||||
Certificate *tlsCert `json:"certificate,omitempty"`
|
AggregateState string `json:"aggregateState,omitempty"`
|
||||||
CertificateCurrent bool `json:"-"`
|
Disagreement bool `json:"disagreement,omitempty"`
|
||||||
|
ExpectedLocations int `json:"expectedLocations,omitempty"`
|
||||||
|
ReportingLocations int `json:"reportingLocations,omitempty"`
|
||||||
|
Locations []AvailabilityObservationLocationStatus `json:"locations,omitempty"`
|
||||||
|
Certificate *tlsCert `json:"certificate,omitempty"`
|
||||||
|
CertificateCurrent bool `json:"-"`
|
||||||
// ProbeReportReceivedAt is server-authored freshness evidence for a remote
|
// ProbeReportReceivedAt is server-authored freshness evidence for a remote
|
||||||
// observation. Keep it off the wire: LastChecked remains the agent's
|
// observation. Keep it off the wire: LastChecked remains the agent's
|
||||||
// observation time, while disconnect detection must not trust agent clock
|
// observation time, while disconnect detection must not trust agent clock
|
||||||
@@ -53,11 +58,41 @@ type AvailabilityProbeStatus struct {
|
|||||||
ProbeReportReceivedAt time.Time `json:"-"`
|
ProbeReportReceivedAt time.Time `json:"-"`
|
||||||
}
|
}
|
||||||
|
|
||||||
|
const (
|
||||||
|
AvailabilityAggregateHealthy = "healthy"
|
||||||
|
AvailabilityAggregateDegraded = "degraded"
|
||||||
|
AvailabilityAggregateUnavailable = "unavailable"
|
||||||
|
AvailabilityAggregateUnknown = "unknown"
|
||||||
|
)
|
||||||
|
|
||||||
|
// AvailabilityObservationLocationStatus is one source-owned view of a logical
|
||||||
|
// verification. LocationID is stable configuration identity; Label is resolved
|
||||||
|
// by the authenticated client so host names and network details are not copied
|
||||||
|
// into target or history persistence.
|
||||||
|
type AvailabilityObservationLocationStatus struct {
|
||||||
|
LocationID string `json:"locationId"`
|
||||||
|
Kind string `json:"kind"`
|
||||||
|
ProbeAgentID string `json:"probeAgentId,omitempty"`
|
||||||
|
Outcome string `json:"outcome,omitempty"`
|
||||||
|
TransportOutcome string `json:"transportOutcome,omitempty"`
|
||||||
|
ApplicationOutcome string `json:"applicationOutcome,omitempty"`
|
||||||
|
ApplicationStatusCode int `json:"applicationStatusCode,omitempty"`
|
||||||
|
ApplicationFailureCode string `json:"applicationFailureCode,omitempty"`
|
||||||
|
Available bool `json:"available"`
|
||||||
|
LastChecked time.Time `json:"lastChecked,omitempty"`
|
||||||
|
LastSuccess time.Time `json:"lastSuccess,omitempty"`
|
||||||
|
FreshnessAt time.Time `json:"freshnessAt,omitempty"`
|
||||||
|
LatencyMillis int64 `json:"latencyMillis,omitempty"`
|
||||||
|
ConsecutiveFailures int `json:"consecutiveFailures,omitempty"`
|
||||||
|
LastError string `json:"lastError,omitempty"`
|
||||||
|
Stale bool `json:"stale,omitempty"`
|
||||||
|
}
|
||||||
|
|
||||||
// FreshnessTime returns the authoritative liveness reference for this status.
|
// FreshnessTime returns the authoritative liveness reference for this status.
|
||||||
// Remote observations use server receipt time; local observations use their
|
// Remote observations use server receipt time; local observations use their
|
||||||
// server-authored check time.
|
// server-authored check time.
|
||||||
func (s AvailabilityProbeStatus) FreshnessTime() time.Time {
|
func (s AvailabilityProbeStatus) FreshnessTime() time.Time {
|
||||||
if strings.TrimSpace(s.ProbeAgentID) != "" && !s.ProbeReportReceivedAt.IsZero() {
|
if (strings.TrimSpace(s.ProbeAgentID) != "" || s.ExpectedLocations > 1) && !s.ProbeReportReceivedAt.IsZero() {
|
||||||
return s.ProbeReportReceivedAt
|
return s.ProbeReportReceivedAt
|
||||||
}
|
}
|
||||||
return s.LastChecked
|
return s.LastChecked
|
||||||
@@ -93,7 +128,7 @@ func (availabilityPollProvider) ListInstances(m *Monitor) []string {
|
|||||||
// A licensed probe assignment moves execution to the agent. When the
|
// A licensed probe assignment moves execution to the agent. When the
|
||||||
// entitlement lapses the effective assignment collapses to local, and
|
// entitlement lapses the effective assignment collapses to local, and
|
||||||
// the next polling cycle re-plans this instance without a restart.
|
// the next polling cycle re-plans this instance without a restart.
|
||||||
if m.effectiveProbeAgentID(target) != "" {
|
if !m.targetUsesLocalObservation(target) {
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
names = append(names, target.ID)
|
names = append(names, target.ID)
|
||||||
@@ -132,8 +167,8 @@ func (availabilityPollProvider) BuildPollTask(m *Monitor, instanceName string) (
|
|||||||
}
|
}
|
||||||
// Refuse the local run even for a queue entry scheduled before the target
|
// Refuse the local run even for a queue entry scheduled before the target
|
||||||
// was assigned, so a probe-assigned target never executes twice.
|
// was assigned, so a probe-assigned target never executes twice.
|
||||||
if agentID := m.effectiveProbeAgentID(target); agentID != "" {
|
if !m.targetUsesLocalObservation(target) {
|
||||||
return PollTask{}, fmt.Errorf("availability target %q is assigned to probe agent %q", instanceName, agentID)
|
return PollTask{}, fmt.Errorf("availability target %q has no local observation location", instanceName)
|
||||||
}
|
}
|
||||||
return PollTask{
|
return PollTask{
|
||||||
InstanceName: target.ID,
|
InstanceName: target.ID,
|
||||||
@@ -262,17 +297,61 @@ func (m *Monitor) availabilityStatusSnapshotForTargets(targets []config.Availabi
|
|||||||
for id, status := range m.availabilityStatuses {
|
for id, status := range m.availabilityStatuses {
|
||||||
out[id] = status
|
out[id] = status
|
||||||
}
|
}
|
||||||
|
locationSnapshots := make(map[string]map[string]AvailabilityProbeStatus, len(m.availabilityByLocation))
|
||||||
|
for targetID, locations := range m.availabilityByLocation {
|
||||||
|
locationSnapshots[targetID] = make(map[string]AvailabilityProbeStatus, len(locations))
|
||||||
|
for locationID, status := range locations {
|
||||||
|
locationSnapshots[targetID][locationID] = status
|
||||||
|
}
|
||||||
|
}
|
||||||
m.mu.RUnlock()
|
m.mu.RUnlock()
|
||||||
|
|
||||||
for _, target := range targets {
|
for _, target := range targets {
|
||||||
status, ok := out[target.ID]
|
expected := m.effectiveObservationLocationIDs(target)
|
||||||
if !ok {
|
locations := locationSnapshots[target.ID]
|
||||||
if m.effectiveProbeAgentID(target) == "" {
|
if locations == nil {
|
||||||
continue
|
locations = make(map[string]AvailabilityProbeStatus)
|
||||||
}
|
|
||||||
status = availabilityStatusFromTarget(target)
|
|
||||||
}
|
}
|
||||||
out[target.ID] = m.deriveAvailabilityProbeStaleness(target, status, now)
|
// Adopt pre-location in-memory state into its one effective location.
|
||||||
|
if len(locations) == 0 {
|
||||||
|
if legacy, ok := out[target.ID]; ok && len(expected) == 1 {
|
||||||
|
legacyLocationID := config.AvailabilityObservationLocationLocal
|
||||||
|
if legacy.ProbeAgentID != "" {
|
||||||
|
legacyLocationID = config.AvailabilityAgentObservationLocationID(legacy.ProbeAgentID)
|
||||||
|
}
|
||||||
|
if legacyLocationID == expected[0] || expected[0] == config.AvailabilityObservationLocationLocal {
|
||||||
|
if expected[0] == config.AvailabilityObservationLocationLocal {
|
||||||
|
legacy.ProbeAgentID = ""
|
||||||
|
legacy.ProbeReportReceivedAt = time.Time{}
|
||||||
|
}
|
||||||
|
locations[expected[0]] = legacy
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if len(expected) == 1 && expected[0] == config.AvailabilityObservationLocationLocal {
|
||||||
|
if _, ok := locations[expected[0]]; !ok {
|
||||||
|
if legacy, exists := out[target.ID]; exists {
|
||||||
|
legacy.ProbeAgentID = ""
|
||||||
|
legacy.ProbeReportReceivedAt = time.Time{}
|
||||||
|
locations[expected[0]] = legacy
|
||||||
|
} else {
|
||||||
|
delete(out, target.ID)
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
for _, locationID := range expected {
|
||||||
|
status, ok := locations[locationID]
|
||||||
|
if !ok {
|
||||||
|
status = availabilityStatusFromTarget(target)
|
||||||
|
status.ProbeAgentID = config.AvailabilityObservationLocationAgentID(locationID)
|
||||||
|
}
|
||||||
|
if status.ProbeAgentID != "" {
|
||||||
|
status = m.deriveAvailabilityProbeStaleness(target, status, now)
|
||||||
|
}
|
||||||
|
locations[locationID] = status
|
||||||
|
}
|
||||||
|
out[target.ID] = aggregateAvailabilityLocationStatuses(target, expected, locations)
|
||||||
}
|
}
|
||||||
return out
|
return out
|
||||||
}
|
}
|
||||||
@@ -287,9 +366,13 @@ func (m *Monitor) RefreshAvailabilityTargets() {
|
|||||||
now := time.Now()
|
now := time.Now()
|
||||||
for _, target := range targets {
|
for _, target := range targets {
|
||||||
activeIDs[target.ID] = struct{}{}
|
activeIDs[target.ID] = struct{}{}
|
||||||
if agentID := m.effectiveProbeAgentID(target); agentID != "" {
|
for _, locationID := range m.effectiveObservationLocationIDs(target) {
|
||||||
activeProbeIDs[target.ID] = struct{}{}
|
if config.AvailabilityObservationLocationAgentID(locationID) == "" {
|
||||||
m.availabilityProbeAssignmentReference(target.ID, agentID, now)
|
continue
|
||||||
|
}
|
||||||
|
trackerID := target.ID + "\x00" + locationID
|
||||||
|
activeProbeIDs[trackerID] = struct{}{}
|
||||||
|
m.availabilityProbeAssignmentReference(target.ID, locationID, now)
|
||||||
}
|
}
|
||||||
if m.taskQueue == nil {
|
if m.taskQueue == nil {
|
||||||
continue
|
continue
|
||||||
@@ -300,7 +383,7 @@ func (m *Monitor) RefreshAvailabilityTargets() {
|
|||||||
NextRun: now,
|
NextRun: now,
|
||||||
Interval: clampInterval(time.Duration(target.EffectivePollIntervalSecs())*time.Second, 10*time.Second, time.Hour),
|
Interval: clampInterval(time.Duration(target.EffectivePollIntervalSecs())*time.Second, 10*time.Second, time.Hour),
|
||||||
}
|
}
|
||||||
if target.Enabled && m.effectiveProbeAgentID(target) == "" {
|
if target.Enabled && m.targetUsesLocalObservation(target) {
|
||||||
m.taskQueue.Upsert(task)
|
m.taskQueue.Upsert(task)
|
||||||
} else {
|
} else {
|
||||||
m.taskQueue.Remove(InstanceTypeAvailability, target.ID)
|
m.taskQueue.Remove(InstanceTypeAvailability, target.ID)
|
||||||
@@ -316,6 +399,11 @@ func (m *Monitor) RefreshAvailabilityTargets() {
|
|||||||
removedIDs = append(removedIDs, id)
|
removedIDs = append(removedIDs, id)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
for id := range m.availabilityByLocation {
|
||||||
|
if _, ok := activeIDs[id]; !ok {
|
||||||
|
delete(m.availabilityByLocation, id)
|
||||||
|
}
|
||||||
|
}
|
||||||
for id := range m.availabilityProbeTrackers {
|
for id := range m.availabilityProbeTrackers {
|
||||||
if _, ok := activeProbeIDs[id]; !ok {
|
if _, ok := activeProbeIDs[id]; !ok {
|
||||||
delete(m.availabilityProbeTrackers, id)
|
delete(m.availabilityProbeTrackers, id)
|
||||||
@@ -375,15 +463,21 @@ func (m *Monitor) applyAvailabilityObservationDetailed(
|
|||||||
}
|
}
|
||||||
m.setAvailabilityStatusWithDetails(target, checkedAt, latency, outcome, transportOutcome, application, probeErr, certificate, probeAgentID, probeReportReceivedAt)
|
m.setAvailabilityStatusWithDetails(target, checkedAt, latency, outcome, transportOutcome, application, probeErr, certificate, probeAgentID, probeReportReceivedAt)
|
||||||
m.recordAvailabilityHistory(target, observationID, checkedAt, latency, outcome, probeErr, probeAgentID, probeReportReceivedAt)
|
m.recordAvailabilityHistory(target, observationID, checkedAt, latency, outcome, probeErr, probeAgentID, probeReportReceivedAt)
|
||||||
|
current := m.AvailabilityStatusSnapshot()[target.ID]
|
||||||
|
aggregateHealthy := current.AggregateState == AvailabilityAggregateHealthy || current.AggregateState == AvailabilityAggregateDegraded
|
||||||
|
|
||||||
if probeErr == nil {
|
if aggregateHealthy {
|
||||||
if m.stalenessTracker != nil {
|
if m.stalenessTracker != nil {
|
||||||
m.stalenessTracker.UpdateSuccess(InstanceTypeAvailability, target.ID, nil)
|
m.stalenessTracker.UpdateSuccess(InstanceTypeAvailability, target.ID, nil)
|
||||||
}
|
}
|
||||||
m.setProviderConnectionHealth(InstanceTypeAvailability, target.ID, true)
|
m.setProviderConnectionHealth(InstanceTypeAvailability, target.ID, true)
|
||||||
} else {
|
} else {
|
||||||
if m.stalenessTracker != nil {
|
if m.stalenessTracker != nil {
|
||||||
m.stalenessTracker.UpdateSuccess(InstanceTypeAvailability, target.ID, []byte(probeErr.Error()))
|
reason := strings.TrimSpace(current.LastError)
|
||||||
|
if reason == "" {
|
||||||
|
reason = "availability evidence is incomplete"
|
||||||
|
}
|
||||||
|
m.stalenessTracker.UpdateSuccess(InstanceTypeAvailability, target.ID, []byte(reason))
|
||||||
}
|
}
|
||||||
m.setProviderConnectionHealth(InstanceTypeAvailability, target.ID, false)
|
m.setProviderConnectionHealth(InstanceTypeAvailability, target.ID, false)
|
||||||
}
|
}
|
||||||
@@ -421,6 +515,23 @@ func (m *Monitor) recordAvailabilityHistory(
|
|||||||
}
|
}
|
||||||
latencyMillis = &value
|
latencyMillis = &value
|
||||||
}
|
}
|
||||||
|
expected := m.effectiveObservationLocationIDs(target)
|
||||||
|
if len(expected) > 1 {
|
||||||
|
aggregate := m.AvailabilityStatusSnapshot()[target.ID]
|
||||||
|
outcome = AvailabilityProbeIndeterminate
|
||||||
|
latencyMillis = nil
|
||||||
|
switch aggregate.AggregateState {
|
||||||
|
case AvailabilityAggregateHealthy, AvailabilityAggregateDegraded:
|
||||||
|
outcome = AvailabilityProbeReachable
|
||||||
|
if aggregate.LatencyMillis > 0 {
|
||||||
|
value := aggregate.LatencyMillis
|
||||||
|
latencyMillis = &value
|
||||||
|
}
|
||||||
|
case AvailabilityAggregateUnavailable:
|
||||||
|
outcome = AvailabilityProbeUnreachable
|
||||||
|
}
|
||||||
|
observationID += "|aggregate"
|
||||||
|
}
|
||||||
if err := m.metricsStore.WriteAvailabilityObservationBounded(pkgmetrics.AvailabilityObservation{
|
if err := m.metricsStore.WriteAvailabilityObservationBounded(pkgmetrics.AvailabilityObservation{
|
||||||
ObservationID: observationID,
|
ObservationID: observationID,
|
||||||
TargetID: target.ID,
|
TargetID: target.ID,
|
||||||
@@ -477,6 +588,11 @@ func (m *Monitor) setAvailabilityStatusWithDetails(
|
|||||||
if m == nil {
|
if m == nil {
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
locationID := config.AvailabilityObservationLocationLocal
|
||||||
|
if agentID := strings.TrimSpace(probeAgentID); agentID != "" {
|
||||||
|
locationID = config.AvailabilityAgentObservationLocationID(agentID)
|
||||||
|
}
|
||||||
|
expectedLocations := m.effectiveObservationLocationIDs(target)
|
||||||
status := availabilityStatusFromTarget(target)
|
status := availabilityStatusFromTarget(target)
|
||||||
status.Outcome = string(outcome)
|
status.Outcome = string(outcome)
|
||||||
status.TransportOutcome = string(transportOutcome)
|
status.TransportOutcome = string(transportOutcome)
|
||||||
@@ -514,7 +630,13 @@ func (m *Monitor) setAvailabilityStatusWithDetails(
|
|||||||
if m.availabilityStatuses == nil {
|
if m.availabilityStatuses == nil {
|
||||||
m.availabilityStatuses = make(map[string]AvailabilityProbeStatus)
|
m.availabilityStatuses = make(map[string]AvailabilityProbeStatus)
|
||||||
}
|
}
|
||||||
if previous, ok := m.availabilityStatuses[target.ID]; ok {
|
if m.availabilityByLocation == nil {
|
||||||
|
m.availabilityByLocation = make(map[string]map[string]AvailabilityProbeStatus)
|
||||||
|
}
|
||||||
|
if m.availabilityByLocation[target.ID] == nil {
|
||||||
|
m.availabilityByLocation[target.ID] = make(map[string]AvailabilityProbeStatus)
|
||||||
|
}
|
||||||
|
if previous, ok := m.availabilityByLocation[target.ID][locationID]; ok {
|
||||||
status.LastSuccess = previous.LastSuccess
|
status.LastSuccess = previous.LastSuccess
|
||||||
if status.Certificate == nil {
|
if status.Certificate == nil {
|
||||||
status.Certificate = previous.Certificate.Clone()
|
status.Certificate = previous.Certificate.Clone()
|
||||||
@@ -531,7 +653,12 @@ func (m *Monitor) setAvailabilityStatusWithDetails(
|
|||||||
} else if probeErr != nil {
|
} else if probeErr != nil {
|
||||||
status.ConsecutiveFailures = 1
|
status.ConsecutiveFailures = 1
|
||||||
}
|
}
|
||||||
m.availabilityStatuses[target.ID] = status
|
m.availabilityByLocation[target.ID][locationID] = status
|
||||||
|
m.availabilityStatuses[target.ID] = aggregateAvailabilityLocationStatuses(
|
||||||
|
target,
|
||||||
|
expectedLocations,
|
||||||
|
m.availabilityByLocation[target.ID],
|
||||||
|
)
|
||||||
m.mu.Unlock()
|
m.mu.Unlock()
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -567,6 +694,168 @@ func availabilityStatusFromTarget(target config.AvailabilityTarget) Availability
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func aggregateAvailabilityLocationStatuses(
|
||||||
|
target config.AvailabilityTarget,
|
||||||
|
expected []string,
|
||||||
|
statuses map[string]AvailabilityProbeStatus,
|
||||||
|
) AvailabilityProbeStatus {
|
||||||
|
aggregate := availabilityStatusFromTarget(target)
|
||||||
|
aggregate.ExpectedLocations = len(expected)
|
||||||
|
if len(expected) == 0 {
|
||||||
|
aggregate.AggregateState = AvailabilityAggregateUnknown
|
||||||
|
return aggregate
|
||||||
|
}
|
||||||
|
|
||||||
|
reachable, unreachable, indeterminate := 0, 0, 0
|
||||||
|
minFailures := 0
|
||||||
|
latestFreshness := time.Time{}
|
||||||
|
latestChecked := time.Time{}
|
||||||
|
latestSuccess := time.Time{}
|
||||||
|
var representative *AvailabilityProbeStatus
|
||||||
|
var latestReachable *AvailabilityProbeStatus
|
||||||
|
for _, locationID := range expected {
|
||||||
|
status, ok := statuses[locationID]
|
||||||
|
if !ok {
|
||||||
|
status = availabilityStatusFromTarget(target)
|
||||||
|
status.ProbeAgentID = config.AvailabilityObservationLocationAgentID(locationID)
|
||||||
|
}
|
||||||
|
effectiveOutcome := status.Outcome
|
||||||
|
if effectiveOutcome == "" && !status.LastChecked.IsZero() {
|
||||||
|
if status.Available {
|
||||||
|
effectiveOutcome = string(AvailabilityProbeReachable)
|
||||||
|
} else if status.ConsecutiveFailures > 0 || strings.TrimSpace(status.LastError) != "" {
|
||||||
|
effectiveOutcome = string(AvailabilityProbeUnreachable)
|
||||||
|
} else {
|
||||||
|
effectiveOutcome = string(AvailabilityProbeIndeterminate)
|
||||||
|
}
|
||||||
|
status.Outcome = effectiveOutcome
|
||||||
|
}
|
||||||
|
freshness := status.FreshnessTime()
|
||||||
|
location := AvailabilityObservationLocationStatus{
|
||||||
|
LocationID: locationID,
|
||||||
|
Kind: "pulse",
|
||||||
|
ProbeAgentID: status.ProbeAgentID,
|
||||||
|
Outcome: status.Outcome,
|
||||||
|
TransportOutcome: status.TransportOutcome,
|
||||||
|
ApplicationOutcome: status.ApplicationOutcome,
|
||||||
|
ApplicationStatusCode: status.ApplicationStatusCode,
|
||||||
|
ApplicationFailureCode: status.ApplicationFailureCode,
|
||||||
|
Available: status.Available,
|
||||||
|
LastChecked: status.LastChecked,
|
||||||
|
LastSuccess: status.LastSuccess,
|
||||||
|
FreshnessAt: freshness,
|
||||||
|
LatencyMillis: status.LatencyMillis,
|
||||||
|
ConsecutiveFailures: status.ConsecutiveFailures,
|
||||||
|
LastError: status.LastError,
|
||||||
|
Stale: availabilityProbeStatusIsStale(status),
|
||||||
|
}
|
||||||
|
if status.ProbeAgentID != "" {
|
||||||
|
location.Kind = "agent"
|
||||||
|
}
|
||||||
|
aggregate.Locations = append(aggregate.Locations, location)
|
||||||
|
if status.LastChecked.IsZero() {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if !availabilityProbeStatusIsStale(status) {
|
||||||
|
aggregate.ReportingLocations++
|
||||||
|
}
|
||||||
|
if freshness.After(latestFreshness) {
|
||||||
|
latestFreshness = freshness
|
||||||
|
}
|
||||||
|
if status.LastChecked.After(latestChecked) {
|
||||||
|
latestChecked = status.LastChecked
|
||||||
|
copy := status
|
||||||
|
representative = ©
|
||||||
|
}
|
||||||
|
if status.LastSuccess.After(latestSuccess) {
|
||||||
|
latestSuccess = status.LastSuccess
|
||||||
|
}
|
||||||
|
switch effectiveOutcome {
|
||||||
|
case string(AvailabilityProbeReachable):
|
||||||
|
reachable++
|
||||||
|
if latestReachable == nil || status.LastChecked.After(latestReachable.LastChecked) {
|
||||||
|
copy := status
|
||||||
|
latestReachable = ©
|
||||||
|
}
|
||||||
|
case string(AvailabilityProbeUnreachable):
|
||||||
|
unreachable++
|
||||||
|
if minFailures == 0 || status.ConsecutiveFailures < minFailures {
|
||||||
|
minFailures = status.ConsecutiveFailures
|
||||||
|
}
|
||||||
|
case string(AvailabilityProbeIndeterminate):
|
||||||
|
indeterminate++
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if representative != nil {
|
||||||
|
aggregate.Outcome = representative.Outcome
|
||||||
|
aggregate.TransportOutcome = representative.TransportOutcome
|
||||||
|
aggregate.ApplicationOutcome = representative.ApplicationOutcome
|
||||||
|
aggregate.ApplicationStatusCode = representative.ApplicationStatusCode
|
||||||
|
aggregate.ApplicationFailureCode = representative.ApplicationFailureCode
|
||||||
|
aggregate.LatencyMillis = representative.LatencyMillis
|
||||||
|
aggregate.Certificate = representative.Certificate.Clone()
|
||||||
|
aggregate.CertificateCurrent = representative.CertificateCurrent
|
||||||
|
}
|
||||||
|
if latestReachable != nil {
|
||||||
|
aggregate.LatencyMillis = latestReachable.LatencyMillis
|
||||||
|
} else if len(expected) > 1 {
|
||||||
|
aggregate.LatencyMillis = 0
|
||||||
|
}
|
||||||
|
aggregate.LastChecked = latestChecked
|
||||||
|
aggregate.LastSuccess = latestSuccess
|
||||||
|
aggregate.ProbeReportReceivedAt = latestFreshness
|
||||||
|
aggregate.Disagreement = reachable > 0 && unreachable > 0
|
||||||
|
unknown := len(expected) - reachable - unreachable - indeterminate
|
||||||
|
switch {
|
||||||
|
case reachable == len(expected):
|
||||||
|
aggregate.AggregateState = AvailabilityAggregateHealthy
|
||||||
|
aggregate.Available = true
|
||||||
|
aggregate.Outcome = string(AvailabilityProbeReachable)
|
||||||
|
case reachable > 0:
|
||||||
|
aggregate.AggregateState = AvailabilityAggregateDegraded
|
||||||
|
aggregate.Available = true
|
||||||
|
aggregate.Outcome = string(AvailabilityProbeReachable)
|
||||||
|
case unreachable == len(expected):
|
||||||
|
aggregate.AggregateState = AvailabilityAggregateUnavailable
|
||||||
|
aggregate.Available = false
|
||||||
|
aggregate.Outcome = string(AvailabilityProbeUnreachable)
|
||||||
|
aggregate.ConsecutiveFailures = minFailures
|
||||||
|
aggregate.LastError = "all observation locations report the target unreachable"
|
||||||
|
default:
|
||||||
|
aggregate.AggregateState = AvailabilityAggregateUnknown
|
||||||
|
aggregate.Available = false
|
||||||
|
aggregate.Outcome = string(AvailabilityProbeIndeterminate)
|
||||||
|
if indeterminate > 0 || unknown > 0 {
|
||||||
|
aggregate.LastError = "observation coverage is incomplete"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if len(expected) == 1 && representative != nil {
|
||||||
|
// Preserve the established one-location response while adding the location
|
||||||
|
// envelope, so older consumers do not lose source attribution or errors.
|
||||||
|
locations := aggregate.Locations
|
||||||
|
state := aggregate.AggregateState
|
||||||
|
reporting := aggregate.ReportingLocations
|
||||||
|
aggregate = *representative
|
||||||
|
aggregate.AggregateState = state
|
||||||
|
aggregate.ExpectedLocations = 1
|
||||||
|
aggregate.ReportingLocations = reporting
|
||||||
|
aggregate.Locations = locations
|
||||||
|
}
|
||||||
|
if len(expected) == 1 && representative == nil {
|
||||||
|
status := statuses[expected[0]]
|
||||||
|
locations := aggregate.Locations
|
||||||
|
state := aggregate.AggregateState
|
||||||
|
reporting := aggregate.ReportingLocations
|
||||||
|
aggregate = status
|
||||||
|
aggregate.AggregateState = state
|
||||||
|
aggregate.ExpectedLocations = 1
|
||||||
|
aggregate.ReportingLocations = reporting
|
||||||
|
aggregate.Locations = locations
|
||||||
|
}
|
||||||
|
return aggregate
|
||||||
|
}
|
||||||
|
|
||||||
func availabilityResourceFromTarget(target config.AvailabilityTarget, status AvailabilityProbeStatus, _ string, now time.Time) (unifiedresources.Resource, unifiedresources.ResourceIdentity) {
|
func availabilityResourceFromTarget(target config.AvailabilityTarget, status AvailabilityProbeStatus, _ string, now time.Time) (unifiedresources.Resource, unifiedresources.ResourceIdentity) {
|
||||||
lastSeen := status.FreshnessTime()
|
lastSeen := status.FreshnessTime()
|
||||||
if lastSeen.IsZero() {
|
if lastSeen.IsZero() {
|
||||||
@@ -586,6 +875,10 @@ func availabilityResourceFromTarget(target config.AvailabilityTarget, status Ava
|
|||||||
ApplicationStatusCode: status.ApplicationStatusCode,
|
ApplicationStatusCode: status.ApplicationStatusCode,
|
||||||
ApplicationFailureCode: status.ApplicationFailureCode,
|
ApplicationFailureCode: status.ApplicationFailureCode,
|
||||||
ProbeAgentID: status.ProbeAgentID,
|
ProbeAgentID: status.ProbeAgentID,
|
||||||
|
AggregateState: status.AggregateState,
|
||||||
|
Disagreement: status.Disagreement,
|
||||||
|
ExpectedLocations: status.ExpectedLocations,
|
||||||
|
ReportingLocations: status.ReportingLocations,
|
||||||
UDPMode: string(target.UDPMode),
|
UDPMode: string(target.UDPMode),
|
||||||
Port: target.Port,
|
Port: target.Port,
|
||||||
Path: target.Path,
|
Path: target.Path,
|
||||||
@@ -600,6 +893,26 @@ func availabilityResourceFromTarget(target config.AvailabilityTarget, status Ava
|
|||||||
PollIntervalSeconds: target.EffectivePollIntervalSecs(),
|
PollIntervalSeconds: target.EffectivePollIntervalSecs(),
|
||||||
TimeoutMillis: target.EffectiveTimeoutMillis(),
|
TimeoutMillis: target.EffectiveTimeoutMillis(),
|
||||||
}
|
}
|
||||||
|
for _, location := range status.Locations {
|
||||||
|
data.Locations = append(data.Locations, unifiedresources.AvailabilityObservationLocation{
|
||||||
|
LocationID: location.LocationID,
|
||||||
|
Kind: location.Kind,
|
||||||
|
ProbeAgentID: location.ProbeAgentID,
|
||||||
|
Outcome: location.Outcome,
|
||||||
|
TransportOutcome: location.TransportOutcome,
|
||||||
|
ApplicationOutcome: location.ApplicationOutcome,
|
||||||
|
ApplicationStatusCode: location.ApplicationStatusCode,
|
||||||
|
ApplicationFailureCode: location.ApplicationFailureCode,
|
||||||
|
Available: location.Available,
|
||||||
|
LastChecked: timePointerIfSet(location.LastChecked),
|
||||||
|
LastSuccess: timePointerIfSet(location.LastSuccess),
|
||||||
|
FreshnessAt: timePointerIfSet(location.FreshnessAt),
|
||||||
|
LatencyMillis: location.LatencyMillis,
|
||||||
|
ConsecutiveFailures: location.ConsecutiveFailures,
|
||||||
|
LastError: location.LastError,
|
||||||
|
Stale: location.Stale,
|
||||||
|
})
|
||||||
|
}
|
||||||
data.CertificateMonitoring = target.CertificateMonitoringEnabled()
|
data.CertificateMonitoring = target.CertificateMonitoringEnabled()
|
||||||
data.CertificateExpiryWarningDays = target.EffectiveCertificateExpiryWarningDays()
|
data.CertificateExpiryWarningDays = target.EffectiveCertificateExpiryWarningDays()
|
||||||
data.Certificate = status.Certificate.Clone()
|
data.Certificate = status.Certificate.Clone()
|
||||||
@@ -612,7 +925,7 @@ func availabilityResourceFromTarget(target config.AvailabilityTarget, status Ava
|
|||||||
observedAt = lastSeen
|
observedAt = lastSeen
|
||||||
}
|
}
|
||||||
ingestedAt := now
|
ingestedAt := now
|
||||||
if strings.TrimSpace(status.ProbeAgentID) != "" && !status.ProbeReportReceivedAt.IsZero() {
|
if !status.ProbeReportReceivedAt.IsZero() {
|
||||||
ingestedAt = status.ProbeReportReceivedAt
|
ingestedAt = status.ProbeReportReceivedAt
|
||||||
}
|
}
|
||||||
data.Evidence = availabilityEvidenceEnvelope(target, status, observedAt, ingestedAt)
|
data.Evidence = availabilityEvidenceEnvelope(target, status, observedAt, ingestedAt)
|
||||||
@@ -672,22 +985,24 @@ func availabilityEvidenceEnvelope(
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
freshnessAt := observedAt
|
freshnessAt := status.FreshnessTime()
|
||||||
|
if freshnessAt.IsZero() {
|
||||||
|
freshnessAt = observedAt
|
||||||
|
}
|
||||||
validityWindow := time.Duration(target.EffectivePollIntervalSecs()*2) * time.Second
|
validityWindow := time.Duration(target.EffectivePollIntervalSecs()*2) * time.Second
|
||||||
if strings.TrimSpace(status.ProbeAgentID) != "" && !status.ProbeReportReceivedAt.IsZero() {
|
if strings.TrimSpace(status.ProbeAgentID) != "" || status.ExpectedLocations > 1 {
|
||||||
freshnessAt = status.ProbeReportReceivedAt
|
|
||||||
validityWindow = availabilityProbeStaleWindow(target)
|
validityWindow = availabilityProbeStaleWindow(target)
|
||||||
}
|
}
|
||||||
validUntil := freshnessAt.Add(validityWindow)
|
validUntil := freshnessAt.Add(validityWindow)
|
||||||
completeness := operationaltrust.EvidenceComplete
|
completeness := operationaltrust.EvidenceComplete
|
||||||
confidence := operationaltrust.EvidenceConfirmed
|
confidence := operationaltrust.EvidenceConfirmed
|
||||||
var reason *operationaltrust.EvidenceReason
|
var reason *operationaltrust.EvidenceReason
|
||||||
if status.LastChecked.IsZero() {
|
if status.LastChecked.IsZero() || status.ReportingLocations < status.ExpectedLocations {
|
||||||
completeness = operationaltrust.EvidencePartial
|
completeness = operationaltrust.EvidencePartial
|
||||||
confidence = operationaltrust.EvidenceUnknown
|
confidence = operationaltrust.EvidenceUnknown
|
||||||
reason = &operationaltrust.EvidenceReason{
|
reason = &operationaltrust.EvidenceReason{
|
||||||
Code: "availability_not_observed",
|
Code: "availability_coverage_incomplete",
|
||||||
Message: "The availability target has not completed its first probe.",
|
Message: "One or more observation locations have no current evidence.",
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -728,6 +1043,17 @@ func availabilityResourceStatus(target config.AvailabilityTarget, status Availab
|
|||||||
if availabilityProbeStatusIsStale(status) {
|
if availabilityProbeStatusIsStale(status) {
|
||||||
return unifiedresources.StatusWarning
|
return unifiedresources.StatusWarning
|
||||||
}
|
}
|
||||||
|
switch status.AggregateState {
|
||||||
|
case AvailabilityAggregateHealthy:
|
||||||
|
return unifiedresources.StatusOnline
|
||||||
|
case AvailabilityAggregateDegraded, AvailabilityAggregateUnknown:
|
||||||
|
return unifiedresources.StatusWarning
|
||||||
|
case AvailabilityAggregateUnavailable:
|
||||||
|
if status.ConsecutiveFailures >= target.EffectiveFailureThreshold() {
|
||||||
|
return unifiedresources.StatusOffline
|
||||||
|
}
|
||||||
|
return unifiedresources.StatusWarning
|
||||||
|
}
|
||||||
if status.LastChecked.IsZero() {
|
if status.LastChecked.IsZero() {
|
||||||
return unifiedresources.StatusUnknown
|
return unifiedresources.StatusUnknown
|
||||||
}
|
}
|
||||||
@@ -747,13 +1073,16 @@ func availabilityIncident(target config.AvailabilityTarget, status AvailabilityP
|
|||||||
if availabilityProbeStatusIsStale(status) {
|
if availabilityProbeStatusIsStale(status) {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
if status.AggregateState != "" && status.AggregateState != AvailabilityAggregateUnavailable {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
if status.LastChecked.IsZero() {
|
if status.LastChecked.IsZero() {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
if status.ConsecutiveFailures < target.EffectiveFailureThreshold() {
|
if status.ConsecutiveFailures < target.EffectiveFailureThreshold() {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
summary := fmt.Sprintf("%s is unreachable by %s probe", target.DisplayName(), strings.ToUpper(string(target.Protocol)))
|
summary := fmt.Sprintf("%s is unreachable from every current observation location by %s probe", target.DisplayName(), strings.ToUpper(string(target.Protocol)))
|
||||||
if status.LastError != "" {
|
if status.LastError != "" {
|
||||||
summary = summary + ": " + status.LastError
|
summary = summary + ": " + status.LastError
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -42,8 +42,8 @@ type ProbeAvailabilityResult struct {
|
|||||||
// assigned target before its first report. AgentID is retained so reassigning
|
// assigned target before its first report. AgentID is retained so reassigning
|
||||||
// the same target to a different probe starts a fresh grace window.
|
// the same target to a different probe starts a fresh grace window.
|
||||||
type availabilityProbeAssignmentTracker struct {
|
type availabilityProbeAssignmentTracker struct {
|
||||||
AgentID string
|
LocationID string
|
||||||
Since time.Time
|
Since time.Time
|
||||||
}
|
}
|
||||||
|
|
||||||
// probeAvailabilityResultsFromReport converts the wire results carried by a
|
// probeAvailabilityResultsFromReport converts the wire results carried by a
|
||||||
@@ -89,14 +89,45 @@ func probeAvailabilityResultsFromReport(reported []agentshost.AvailabilityProbeR
|
|||||||
// entitlement is absent, so a license lapse resumes local polling instead of
|
// entitlement is absent, so a license lapse resumes local polling instead of
|
||||||
// stranding the check on an agent that is no longer allowed to run it.
|
// stranding the check on an agent that is no longer allowed to run it.
|
||||||
func (m *Monitor) effectiveProbeAgentID(target config.AvailabilityTarget) string {
|
func (m *Monitor) effectiveProbeAgentID(target config.AvailabilityTarget) string {
|
||||||
assigned := strings.TrimSpace(target.ProbeAgentID)
|
locations := m.effectiveObservationLocationIDs(target)
|
||||||
if assigned == "" {
|
if len(locations) != 1 {
|
||||||
return ""
|
return ""
|
||||||
}
|
}
|
||||||
if !m.hasLicensedFeature(pkglicensing.FeatureExternalProbe) {
|
return config.AvailabilityObservationLocationAgentID(locations[0])
|
||||||
return ""
|
}
|
||||||
|
|
||||||
|
func (m *Monitor) effectiveObservationLocationIDs(target config.AvailabilityTarget) []string {
|
||||||
|
locations := target.EffectiveObservationLocationIDs()
|
||||||
|
if m.hasLicensedFeature(pkglicensing.FeatureExternalProbe) {
|
||||||
|
return locations
|
||||||
}
|
}
|
||||||
return assigned
|
for _, locationID := range locations {
|
||||||
|
if locationID == config.AvailabilityObservationLocationLocal {
|
||||||
|
return []string{config.AvailabilityObservationLocationLocal}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
// Preserve the historical entitlement-lapse behavior: a remote-only check
|
||||||
|
// resumes locally instead of becoming unobserved.
|
||||||
|
return []string{config.AvailabilityObservationLocationLocal}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (m *Monitor) targetUsesLocalObservation(target config.AvailabilityTarget) bool {
|
||||||
|
for _, locationID := range m.effectiveObservationLocationIDs(target) {
|
||||||
|
if locationID == config.AvailabilityObservationLocationLocal {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
|
||||||
|
func (m *Monitor) targetAssignedToProbeAgent(target config.AvailabilityTarget, agentID string) bool {
|
||||||
|
locationID := config.AvailabilityAgentObservationLocationID(agentID)
|
||||||
|
for _, assignedLocationID := range m.effectiveObservationLocationIDs(target) {
|
||||||
|
if assignedLocationID == locationID {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return false
|
||||||
}
|
}
|
||||||
|
|
||||||
// ApplyProbeAvailabilityResults ingests availability results reported by a host
|
// ApplyProbeAvailabilityResults ingests availability results reported by a host
|
||||||
@@ -137,7 +168,7 @@ func (m *Monitor) applyProbeAvailabilityResultsAt(hostID string, results []Probe
|
|||||||
Msg("Rejecting probe availability result for unknown target")
|
Msg("Rejecting probe availability result for unknown target")
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
if m.effectiveProbeAgentID(target) != hostID {
|
if !m.targetAssignedToProbeAgent(target, hostID) {
|
||||||
log.Debug().
|
log.Debug().
|
||||||
Str("hostID", hostID).
|
Str("hostID", hostID).
|
||||||
Str("targetID", targetID).
|
Str("targetID", targetID).
|
||||||
@@ -240,18 +271,14 @@ func (m *Monitor) deriveAvailabilityProbeStaleness(
|
|||||||
status AvailabilityProbeStatus,
|
status AvailabilityProbeStatus,
|
||||||
now time.Time,
|
now time.Time,
|
||||||
) AvailabilityProbeStatus {
|
) AvailabilityProbeStatus {
|
||||||
agentID := m.effectiveProbeAgentID(target)
|
agentID := strings.TrimSpace(status.ProbeAgentID)
|
||||||
if agentID == "" {
|
if agentID == "" {
|
||||||
return status
|
return status
|
||||||
}
|
}
|
||||||
reportingAgentID := strings.TrimSpace(status.ProbeAgentID)
|
|
||||||
if reportingAgentID != agentID {
|
|
||||||
status = availabilityStatusFromTarget(target)
|
|
||||||
}
|
|
||||||
status.ProbeAgentID = agentID
|
status.ProbeAgentID = agentID
|
||||||
reference := status.FreshnessTime()
|
reference := status.FreshnessTime()
|
||||||
if reference.IsZero() {
|
if reference.IsZero() {
|
||||||
reference = m.availabilityProbeAssignmentReference(target.ID, agentID, now)
|
reference = m.availabilityProbeAssignmentReference(target.ID, config.AvailabilityAgentObservationLocationID(agentID), now)
|
||||||
}
|
}
|
||||||
if !availabilityProbeReportIsStale(target, reference, now) {
|
if !availabilityProbeReportIsStale(target, reference, now) {
|
||||||
return status
|
return status
|
||||||
@@ -263,27 +290,28 @@ func (m *Monitor) deriveAvailabilityProbeStaleness(
|
|||||||
return status
|
return status
|
||||||
}
|
}
|
||||||
|
|
||||||
func (m *Monitor) availabilityProbeAssignmentReference(targetID, agentID string, now time.Time) time.Time {
|
func (m *Monitor) availabilityProbeAssignmentReference(targetID, locationID string, now time.Time) time.Time {
|
||||||
if m == nil {
|
if m == nil {
|
||||||
return now
|
return now
|
||||||
}
|
}
|
||||||
targetID = strings.TrimSpace(targetID)
|
targetID = strings.TrimSpace(targetID)
|
||||||
agentID = strings.TrimSpace(agentID)
|
locationID = strings.TrimSpace(locationID)
|
||||||
if targetID == "" || agentID == "" {
|
if targetID == "" || locationID == "" {
|
||||||
return now
|
return now
|
||||||
}
|
}
|
||||||
|
trackerID := targetID + "\x00" + locationID
|
||||||
|
|
||||||
m.mu.Lock()
|
m.mu.Lock()
|
||||||
defer m.mu.Unlock()
|
defer m.mu.Unlock()
|
||||||
if m.availabilityProbeTrackers == nil {
|
if m.availabilityProbeTrackers == nil {
|
||||||
m.availabilityProbeTrackers = make(map[string]availabilityProbeAssignmentTracker)
|
m.availabilityProbeTrackers = make(map[string]availabilityProbeAssignmentTracker)
|
||||||
}
|
}
|
||||||
if tracker, ok := m.availabilityProbeTrackers[targetID]; ok && tracker.AgentID == agentID && !tracker.Since.IsZero() {
|
if tracker, ok := m.availabilityProbeTrackers[trackerID]; ok && tracker.LocationID == locationID && !tracker.Since.IsZero() {
|
||||||
return tracker.Since
|
return tracker.Since
|
||||||
}
|
}
|
||||||
m.availabilityProbeTrackers[targetID] = availabilityProbeAssignmentTracker{
|
m.availabilityProbeTrackers[trackerID] = availabilityProbeAssignmentTracker{
|
||||||
AgentID: agentID,
|
LocationID: locationID,
|
||||||
Since: now,
|
Since: now,
|
||||||
}
|
}
|
||||||
return now
|
return now
|
||||||
}
|
}
|
||||||
@@ -317,7 +345,7 @@ func (m *Monitor) availabilityProbeTargetsForAgent(hostID string) []map[string]i
|
|||||||
}
|
}
|
||||||
var assigned []map[string]interface{}
|
var assigned []map[string]interface{}
|
||||||
for _, target := range m.availabilityTargets() {
|
for _, target := range m.availabilityTargets() {
|
||||||
if m.effectiveProbeAgentID(target) != hostID {
|
if !m.targetAssignedToProbeAgent(target, hostID) {
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
assigned = append(assigned, availabilityProbeAgentTargetPayload(target))
|
assigned = append(assigned, availabilityProbeAgentTargetPayload(target))
|
||||||
|
|||||||
@@ -69,14 +69,99 @@ func newProbeAgentTestMonitor(t *testing.T, targets ...config.AvailabilityTarget
|
|||||||
t.Fatalf("SaveAvailabilityTargets() error = %v", err)
|
t.Fatalf("SaveAvailabilityTargets() error = %v", err)
|
||||||
}
|
}
|
||||||
return &Monitor{
|
return &Monitor{
|
||||||
state: models.NewState(),
|
state: models.NewState(),
|
||||||
configPersist: persistence,
|
configPersist: persistence,
|
||||||
availabilityStatuses: make(map[string]AvailabilityProbeStatus),
|
availabilityStatuses: make(map[string]AvailabilityProbeStatus),
|
||||||
pollStatusMap: make(map[string]*pollStatus),
|
availabilityByLocation: make(map[string]map[string]AvailabilityProbeStatus),
|
||||||
failureCounts: make(map[string]int),
|
pollStatusMap: make(map[string]*pollStatus),
|
||||||
lastOutcome: make(map[string]taskOutcome),
|
failureCounts: make(map[string]int),
|
||||||
circuitBreakers: make(map[string]*circuitBreaker),
|
lastOutcome: make(map[string]taskOutcome),
|
||||||
taskQueue: NewTaskQueue(),
|
circuitBreakers: make(map[string]*circuitBreaker),
|
||||||
|
taskQueue: NewTaskQueue(),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestMultiLocationAvailabilityPreservesDisagreementUntilEveryPathFails(t *testing.T) {
|
||||||
|
target := config.NormalizeAvailabilityTarget(config.AvailabilityTarget{
|
||||||
|
ID: "service",
|
||||||
|
Name: "Customer API",
|
||||||
|
Address: "api.service.local",
|
||||||
|
Protocol: config.AvailabilityProbeHTTPS,
|
||||||
|
Enabled: true,
|
||||||
|
PollIntervalSecs: 60,
|
||||||
|
FailureThreshold: 2,
|
||||||
|
ObservationLocationIDs: []string{config.AvailabilityObservationLocationLocal, config.AvailabilityAgentObservationLocationID("edge-1")},
|
||||||
|
})
|
||||||
|
monitor := newProbeAgentTestMonitor(t, target)
|
||||||
|
monitor.SetLicenseChecker(licenseWithExternalProbe(true))
|
||||||
|
now := time.Now().UTC()
|
||||||
|
|
||||||
|
monitor.applyAvailabilityObservation(target, "local-ok", now, 8*time.Millisecond, AvailabilityProbeReachable, nil, nil, "", time.Time{})
|
||||||
|
for attempt := 0; attempt < 2; attempt++ {
|
||||||
|
monitor.applyAvailabilityObservation(target, "edge-fail-"+string(rune('a'+attempt)), now.Add(time.Duration(attempt)*time.Second), 30*time.Millisecond, AvailabilityProbeUnreachable, context.DeadlineExceeded, nil, "edge-1", now.Add(time.Duration(attempt)*time.Second))
|
||||||
|
}
|
||||||
|
|
||||||
|
status := monitor.AvailabilityStatusSnapshot()[target.ID]
|
||||||
|
if status.AggregateState != AvailabilityAggregateDegraded || !status.Disagreement || !status.Available {
|
||||||
|
t.Fatalf("aggregate status = %+v, want available disagreement", status)
|
||||||
|
}
|
||||||
|
if status.ExpectedLocations != 2 || status.ReportingLocations != 2 || len(status.Locations) != 2 {
|
||||||
|
t.Fatalf("location coverage = %+v, want 2/2 locations", status)
|
||||||
|
}
|
||||||
|
if want := now.Add(time.Second); !status.FreshnessTime().Equal(want) {
|
||||||
|
t.Fatalf("aggregate freshness = %s, want latest server-authored path freshness %s", status.FreshnessTime(), want)
|
||||||
|
}
|
||||||
|
if status.LatencyMillis != 8 {
|
||||||
|
t.Fatalf("aggregate latency = %dms, want the current reachable path latency", status.LatencyMillis)
|
||||||
|
}
|
||||||
|
resource, _ := availabilityResourceFromTarget(target, status, "", now)
|
||||||
|
if resource.Status != unifiedresources.StatusWarning || len(resource.Incidents) != 0 {
|
||||||
|
t.Fatalf("resource = status %q incidents %+v, one path failure must not author an outage", resource.Status, resource.Incidents)
|
||||||
|
}
|
||||||
|
|
||||||
|
for attempt := 0; attempt < 2; attempt++ {
|
||||||
|
monitor.applyAvailabilityObservation(target, "local-fail-"+string(rune('a'+attempt)), now.Add(time.Duration(attempt+2)*time.Second), 20*time.Millisecond, AvailabilityProbeUnreachable, context.DeadlineExceeded, nil, "", time.Time{})
|
||||||
|
}
|
||||||
|
status = monitor.AvailabilityStatusSnapshot()[target.ID]
|
||||||
|
if status.AggregateState != AvailabilityAggregateUnavailable || status.Available || status.ConsecutiveFailures != 2 {
|
||||||
|
t.Fatalf("aggregate status = %+v, want thresholded all-location outage", status)
|
||||||
|
}
|
||||||
|
if status.LatencyMillis != 0 {
|
||||||
|
t.Fatalf("unavailable aggregate latency = %dms, want no reachable latency", status.LatencyMillis)
|
||||||
|
}
|
||||||
|
resource, _ = availabilityResourceFromTarget(target, status, "", now.Add(4*time.Second))
|
||||||
|
if resource.Status != unifiedresources.StatusOffline || len(resource.Incidents) != 1 {
|
||||||
|
t.Fatalf("resource = status %q incidents %+v, want one universal outage", resource.Status, resource.Incidents)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestMultiLocationAvailabilityTreatsLapsedPathAsUnknownCoverage(t *testing.T) {
|
||||||
|
target := config.NormalizeAvailabilityTarget(config.AvailabilityTarget{
|
||||||
|
ID: "service",
|
||||||
|
Name: "Customer API",
|
||||||
|
Address: "api.service.local",
|
||||||
|
Protocol: config.AvailabilityProbeHTTPS,
|
||||||
|
Enabled: true,
|
||||||
|
PollIntervalSecs: 60,
|
||||||
|
FailureThreshold: 2,
|
||||||
|
ObservationLocationIDs: []string{config.AvailabilityObservationLocationLocal, config.AvailabilityAgentObservationLocationID("edge-1")},
|
||||||
|
})
|
||||||
|
monitor := newProbeAgentTestMonitor(t, target)
|
||||||
|
monitor.SetLicenseChecker(licenseWithExternalProbe(true))
|
||||||
|
now := time.Now().UTC()
|
||||||
|
old := time.Now().UTC().Add(-time.Hour)
|
||||||
|
monitor.applyAvailabilityObservation(target, "local-ok", now, 8*time.Millisecond, AvailabilityProbeReachable, nil, nil, "", time.Time{})
|
||||||
|
monitor.applyAvailabilityObservation(target, "edge-ok", old, 18*time.Millisecond, AvailabilityProbeReachable, nil, nil, "edge-1", old)
|
||||||
|
|
||||||
|
status := monitor.availabilityStatusSnapshotForTargets([]config.AvailabilityTarget{target}, now)[target.ID]
|
||||||
|
if status.AggregateState != AvailabilityAggregateDegraded || status.Disagreement {
|
||||||
|
t.Fatalf("aggregate status = %+v, want healthy path plus unknown coverage", status)
|
||||||
|
}
|
||||||
|
if status.ReportingLocations != 1 || status.ExpectedLocations != 2 {
|
||||||
|
t.Fatalf("coverage = %d/%d, want the lapsed path excluded", status.ReportingLocations, status.ExpectedLocations)
|
||||||
|
}
|
||||||
|
if !status.Locations[1].Stale || status.Locations[1].Outcome != string(AvailabilityProbeIndeterminate) {
|
||||||
|
t.Fatalf("remote location = %+v, want stale unknown evidence", status.Locations[1])
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -673,8 +673,14 @@ func TestAvailabilityProviderStaysOnCanonicalMonitoringPath(t *testing.T) {
|
|||||||
`InstanceTypeAvailability InstanceType = "availability"`,
|
`InstanceTypeAvailability InstanceType = "availability"`,
|
||||||
},
|
},
|
||||||
"monitor.go": {
|
"monitor.go": {
|
||||||
"availabilityStatuses map[string]AvailabilityProbeStatus",
|
"availabilityStatuses",
|
||||||
"availabilityStatuses: make(map[string]AvailabilityProbeStatus)",
|
"map[string]AvailabilityProbeStatus",
|
||||||
|
"availabilityByLocation",
|
||||||
|
"map[string]map[string]AvailabilityProbeStatus",
|
||||||
|
"availabilityStatuses:",
|
||||||
|
"make(map[string]AvailabilityProbeStatus)",
|
||||||
|
"availabilityByLocation:",
|
||||||
|
"make(map[string]map[string]AvailabilityProbeStatus)",
|
||||||
},
|
},
|
||||||
"poll_providers.go": {
|
"poll_providers.go": {
|
||||||
"_ = m.RegisterPollProvider(newAvailabilityPollProvider())",
|
"_ = m.RegisterPollProvider(newAvailabilityPollProvider())",
|
||||||
@@ -713,9 +719,11 @@ func TestMonitoredSystemUsageReadinessGuardrailsRemainCanonical(t *testing.T) {
|
|||||||
"monitor.go": {
|
"monitor.go": {
|
||||||
"type MonitorSupplementalInventoryReadinessProvider interface {",
|
"type MonitorSupplementalInventoryReadinessProvider interface {",
|
||||||
"SupplementalInventoryReadyAt(m *Monitor, orgID string) (time.Time, bool)",
|
"SupplementalInventoryReadyAt(m *Monitor, orgID string) (time.Time, bool)",
|
||||||
"hostContinuityStore *config.HostContinuityStore",
|
"hostContinuityStore",
|
||||||
|
"*config.HostContinuityStore",
|
||||||
"hostContinuityStore := config.NewHostContinuityStore(cfg.DataPath, nil)",
|
"hostContinuityStore := config.NewHostContinuityStore(cfg.DataPath, nil)",
|
||||||
"hostContinuityStore: hostContinuityStore,",
|
"hostContinuityStore:",
|
||||||
|
"hostContinuityStore,",
|
||||||
"func (m *Monitor) HostsSnapshot() []models.Host {",
|
"func (m *Monitor) HostsSnapshot() []models.Host {",
|
||||||
"readState = m.readStateWithStandaloneHostContinuity(readState)",
|
"readState = m.readStateWithStandaloneHostContinuity(readState)",
|
||||||
"func (m *Monitor) unifiedStateViewWithStandaloneHostContinuity(view monitorUnifiedStateView) monitorUnifiedStateView {",
|
"func (m *Monitor) unifiedStateViewWithStandaloneHostContinuity(view monitorUnifiedStateView) monitorUnifiedStateView {",
|
||||||
@@ -905,7 +913,8 @@ func TestProxmoxGuestMemoryFallbackUsesInstanceScopedCachesAndAgentMeminfo(t *te
|
|||||||
`memorySource = "unavailable"`,
|
`memorySource = "unavailable"`,
|
||||||
},
|
},
|
||||||
"monitor.go": {
|
"monitor.go": {
|
||||||
"vmAgentMemCache map[string]agentMemCacheEntry",
|
"vmAgentMemCache",
|
||||||
|
"map[string]agentMemCacheEntry",
|
||||||
},
|
},
|
||||||
"monitor_agents.go": {
|
"monitor_agents.go": {
|
||||||
"for key, entry := range m.vmAgentMemCache {",
|
"for key, entry := range m.vmAgentMemCache {",
|
||||||
@@ -1238,8 +1247,10 @@ func TestProxmoxGuestDockerLXCProbingRequiresExplicitOptIn(t *testing.T) {
|
|||||||
func TestBackupOrphanDetectionUsesCanonicalInventoryReadinessScope(t *testing.T) {
|
func TestBackupOrphanDetectionUsesCanonicalInventoryReadinessScope(t *testing.T) {
|
||||||
requiredSnippets := map[string][]string{
|
requiredSnippets := map[string][]string{
|
||||||
"monitor.go": {
|
"monitor.go": {
|
||||||
"pveBackupInventoryReady map[string]map[string]bool",
|
"pveBackupInventoryReady",
|
||||||
"pveBackupTemplateSubjects map[string]map[string]struct{}",
|
"map[string]map[string]bool",
|
||||||
|
"pveBackupTemplateSubjects",
|
||||||
|
"map[string]map[string]struct{}",
|
||||||
},
|
},
|
||||||
"monitor_backups.go": {
|
"monitor_backups.go": {
|
||||||
"func (m *Monitor) updatePVEBackupTemplateSubjectsForType(instanceName, guestType string, subjects map[string]struct{}) {",
|
"func (m *Monitor) updatePVEBackupTemplateSubjectsForType(instanceName, guestType string, subjects map[string]struct{}) {",
|
||||||
|
|||||||
@@ -1078,6 +1078,7 @@ type Monitor struct {
|
|||||||
pbsClients map[string]*pbs.Client
|
pbsClients map[string]*pbs.Client
|
||||||
pmgClients map[string]*pmg.Client
|
pmgClients map[string]*pmg.Client
|
||||||
availabilityStatuses map[string]AvailabilityProbeStatus
|
availabilityStatuses map[string]AvailabilityProbeStatus
|
||||||
|
availabilityByLocation map[string]map[string]AvailabilityProbeStatus
|
||||||
availabilityProbeTrackers map[string]availabilityProbeAssignmentTracker
|
availabilityProbeTrackers map[string]availabilityProbeAssignmentTracker
|
||||||
pollProviders map[InstanceType]PollProvider
|
pollProviders map[InstanceType]PollProvider
|
||||||
pollMetrics *PollMetrics
|
pollMetrics *PollMetrics
|
||||||
@@ -1668,6 +1669,7 @@ func New(cfg *config.Config) (*Monitor, error) {
|
|||||||
pbsClients: make(map[string]*pbs.Client),
|
pbsClients: make(map[string]*pbs.Client),
|
||||||
pmgClients: make(map[string]*pmg.Client),
|
pmgClients: make(map[string]*pmg.Client),
|
||||||
availabilityStatuses: make(map[string]AvailabilityProbeStatus),
|
availabilityStatuses: make(map[string]AvailabilityProbeStatus),
|
||||||
|
availabilityByLocation: make(map[string]map[string]AvailabilityProbeStatus),
|
||||||
availabilityProbeTrackers: make(map[string]availabilityProbeAssignmentTracker),
|
availabilityProbeTrackers: make(map[string]availabilityProbeAssignmentTracker),
|
||||||
pollProviders: make(map[InstanceType]PollProvider),
|
pollProviders: make(map[InstanceType]PollProvider),
|
||||||
pollMetrics: getPollMetrics(),
|
pollMetrics: getPollMetrics(),
|
||||||
|
|||||||
@@ -740,7 +740,9 @@ func TestAgentlessAvailabilityTargetKindStaysCanonical(t *testing.T) {
|
|||||||
"LinkedResourceID string `json:\"linkedResourceId,omitempty\"`",
|
"LinkedResourceID string `json:\"linkedResourceId,omitempty\"`",
|
||||||
},
|
},
|
||||||
filepath.Join("..", "monitoring", "availability_poller.go"): {
|
filepath.Join("..", "monitoring", "availability_poller.go"): {
|
||||||
"TargetKind string `json:\"targetKind,omitempty\"`",
|
"TargetKind",
|
||||||
|
"string",
|
||||||
|
"`json:\"targetKind,omitempty\"`",
|
||||||
"TargetKind: string(target.TargetKind),",
|
"TargetKind: string(target.TargetKind),",
|
||||||
"TargetKind: string(target.TargetKind),",
|
"TargetKind: string(target.TargetKind),",
|
||||||
"tags = append(tags, string(target.TargetKind))",
|
"tags = append(tags, string(target.TargetKind))",
|
||||||
@@ -748,7 +750,9 @@ func TestAgentlessAvailabilityTargetKindStaysCanonical(t *testing.T) {
|
|||||||
},
|
},
|
||||||
"types.go": {
|
"types.go": {
|
||||||
"Availability *AvailabilityData `json:\"availability,omitempty\"`",
|
"Availability *AvailabilityData `json:\"availability,omitempty\"`",
|
||||||
"TargetKind string `json:\"targetKind,omitempty\"`",
|
"TargetKind",
|
||||||
|
"string",
|
||||||
|
"`json:\"targetKind,omitempty\"`",
|
||||||
"LinkedResourceID string `json:\"linkedResourceId,omitempty\"`",
|
"LinkedResourceID string `json:\"linkedResourceId,omitempty\"`",
|
||||||
"LastChecked *time.Time `json:\"lastChecked,omitempty\"`",
|
"LastChecked *time.Time `json:\"lastChecked,omitempty\"`",
|
||||||
"LastSuccess *time.Time `json:\"lastSuccess,omitempty\"`",
|
"LastSuccess *time.Time `json:\"lastSuccess,omitempty\"`",
|
||||||
|
|||||||
@@ -1753,6 +1753,11 @@ type AvailabilityData struct {
|
|||||||
ApplicationStatusCode int `json:"applicationStatusCode,omitempty"`
|
ApplicationStatusCode int `json:"applicationStatusCode,omitempty"`
|
||||||
ApplicationFailureCode string `json:"applicationFailureCode,omitempty"`
|
ApplicationFailureCode string `json:"applicationFailureCode,omitempty"`
|
||||||
ProbeAgentID string `json:"probeAgentId,omitempty"`
|
ProbeAgentID string `json:"probeAgentId,omitempty"`
|
||||||
|
AggregateState string `json:"aggregateState,omitempty"`
|
||||||
|
Disagreement bool `json:"disagreement,omitempty"`
|
||||||
|
ExpectedLocations int `json:"expectedLocations,omitempty"`
|
||||||
|
ReportingLocations int `json:"reportingLocations,omitempty"`
|
||||||
|
Locations []AvailabilityObservationLocation `json:"locations,omitempty"`
|
||||||
UDPMode string `json:"udpMode,omitempty"`
|
UDPMode string `json:"udpMode,omitempty"`
|
||||||
Port int `json:"port,omitempty"`
|
Port int `json:"port,omitempty"`
|
||||||
Path string `json:"path,omitempty"`
|
Path string `json:"path,omitempty"`
|
||||||
@@ -1777,6 +1782,25 @@ type AvailabilityData struct {
|
|||||||
Certificate *tlsutil.CertificateObservation `json:"certificate,omitempty"`
|
Certificate *tlsutil.CertificateObservation `json:"certificate,omitempty"`
|
||||||
}
|
}
|
||||||
|
|
||||||
|
type AvailabilityObservationLocation struct {
|
||||||
|
LocationID string `json:"locationId"`
|
||||||
|
Kind string `json:"kind"`
|
||||||
|
ProbeAgentID string `json:"probeAgentId,omitempty"`
|
||||||
|
Outcome string `json:"outcome,omitempty"`
|
||||||
|
TransportOutcome string `json:"transportOutcome,omitempty"`
|
||||||
|
ApplicationOutcome string `json:"applicationOutcome,omitempty"`
|
||||||
|
ApplicationStatusCode int `json:"applicationStatusCode,omitempty"`
|
||||||
|
ApplicationFailureCode string `json:"applicationFailureCode,omitempty"`
|
||||||
|
Available bool `json:"available"`
|
||||||
|
LastChecked *time.Time `json:"lastChecked,omitempty"`
|
||||||
|
LastSuccess *time.Time `json:"lastSuccess,omitempty"`
|
||||||
|
FreshnessAt *time.Time `json:"freshnessAt,omitempty"`
|
||||||
|
LatencyMillis int64 `json:"latencyMillis,omitempty"`
|
||||||
|
ConsecutiveFailures int `json:"consecutiveFailures,omitempty"`
|
||||||
|
LastError string `json:"lastError,omitempty"`
|
||||||
|
Stale bool `json:"stale,omitempty"`
|
||||||
|
}
|
||||||
|
|
||||||
// K8sMetricCapabilities describes which Kubernetes metric families are available
|
// K8sMetricCapabilities describes which Kubernetes metric families are available
|
||||||
// for this cluster right now based on active collection paths.
|
// for this cluster right now based on active collection paths.
|
||||||
type K8sMetricCapabilities struct {
|
type K8sMetricCapabilities struct {
|
||||||
|
|||||||
Reference in New Issue
Block a user