mirror of
https://github.com/rcourtman/Pulse.git
synced 2026-10-24 04:33:17 +00:00
Show a host whose agent stopped reporting as offline
A host agent past its reporting lease is offline: the monitor marks it so and raises "Host is offline". The registry's stale pass ranked the agent's stale sighting above offline, though, so the machine reached every consumer as a warning. The Machines table showed it amber and rendered its last CPU, memory, disk and temperature as current readings, while its own drawer led with the critical offline alert. A Docker host was offline only until its sighting crossed the 120-second stale threshold, then flipped to warning, and Kubernetes clusters likewise. Each sighting now records when a push reporter (host agent, Docker, Kubernetes) delivered its machine, Docker host or cluster as offline, and status aggregation counts that as offline whatever the sighting's age. The sighting itself still reads stale, and a live source still carries a merged row, so a Proxmox node with a dead agent stays online. A copy that went through JSON re-derives the marker from its stored status, and a manual link joining two resources from one source keeps the fresher sighting, so a retired agent cannot pull a live one down. Offline rows then take the existing danger gate. The Machines Uptime and Temp cells and the Docker hosts Temp cell had no gate and now follow it. The drawer drops a silent agent's uptime and marks its non-disk Thermals rows "(last known)", as retained disk rows already are.
This commit is contained in:
@@ -5591,6 +5591,14 @@ Standalone, TrueNAS, and vSphere platform tables and their table-model helpers
|
||||
must compose those helpers instead of declaring local `metricFallback` /
|
||||
`finiteMetric` helpers or inlining centered muted dash fallback markup in
|
||||
metric cells.
|
||||
A row whose status indicator is `danger` (offline) blanks every reading from
|
||||
its last report, not only the metric bars. Proxmox node rows already gate
|
||||
uptime and temperature on online. The Machines table gates its Uptime and
|
||||
Temperature cells, and the Docker hosts table its temperature cell, on the same
|
||||
check. A machine another source keeps up stays rendered, since its cell may
|
||||
carry that source's current reading. Numeric cells keep their own
|
||||
right-aligned empty dash; the centred `PlatformTableMetricFallback` marker
|
||||
stays specific to metric-bar cells.
|
||||
Platform table metric severity coloring is alert-backed, not hardcoded. The
|
||||
Docker host and container, Proxmox node, Kubernetes cluster and node, TrueNAS
|
||||
system and app, and vSphere host tables must resolve display thresholds
|
||||
|
||||
@@ -2051,6 +2051,11 @@ in `frontend-modern/src/features/storageBackups/diskTemperaturePresentation.ts`.
|
||||
That module stays separate from `diskPresentation.ts` so the drawer and
|
||||
Machines chunks take the decision without the Storage presenter, and the rows
|
||||
add no reads, polling or aggregation work.
|
||||
A silent host agent's other Thermals rows (pressure, limits, GPU, temperatures,
|
||||
additional sensors, fans, power) take the same "(last known)" suffix and reason
|
||||
title. `buildTemperatureRows` gets them through its `lastKnownReason` option,
|
||||
which the drawer sets from the resource's own `agent.stale`, so the drawer
|
||||
still reads nothing extra.
|
||||
The Proxmox node drawer overview should follow the existing guest drawer
|
||||
compact detail-section pattern and expose node-specific context such as platform,
|
||||
kernel, hardware, raw capacity, telemetry, and thermal facts rather than
|
||||
|
||||
@@ -3169,6 +3169,16 @@ pointer isolation are
|
||||
pinned by `TestResourceFromHostPreservesCustomSensorMeta` and
|
||||
`TestCloneResourceIsolatesCustomSensorValues`.
|
||||
|
||||
### Expired reporting leases change status only
|
||||
|
||||
`SourceStatus` now carries an unexported marker that a push reporter (host
|
||||
agent, Docker, Kubernetes) delivered its machine, Docker host or cluster as
|
||||
offline at lease expiry. Status aggregation reads it, so such a row is
|
||||
`offline` instead of `warning`. The marker is not serialized, persisted or
|
||||
exposed on the wire, and the public `status`, `lastSeen` and cadence fields
|
||||
are unchanged. No storage path, recovery read or protection judgement depends
|
||||
on it. `TestCloneResource_MutateSourceStatusMap` pins that clones keep it.
|
||||
|
||||
### Shared system-settings boundary dropped dead auto-update schedule fields
|
||||
|
||||
The shared `internal/api` system-settings surface this subsystem consumes
|
||||
|
||||
@@ -2270,10 +2270,12 @@ and temperature facts must also compose the frontend-primitives
|
||||
desktop rows use the shared fixed label track so labels and values remain
|
||||
visually adjacent instead of spanning the full drawer width.
|
||||
A standalone host agent past its reporting lease reaches the Machines table as
|
||||
`warning` with `agent.stale`, not `offline`: by the time the lease expires the
|
||||
registry has marked its only source stale, which outranks offline. The offline
|
||||
metric fallback therefore does not blank its cells, and the temperature cell
|
||||
has no such gate. For SMART disk temperatures the provenance travels on
|
||||
`offline`, as described in "A push reporter past its lease is offline". The
|
||||
offline metric fallback blanks its CPU, memory, disk, network, disk I/O, uptime
|
||||
and temperature cells. The machine drawer drops a silent agent's uptime and
|
||||
marks its non-disk Thermals rows "(last known)" with the reason as title,
|
||||
keyed on `agent.stale` because those rows are that agent's own sensors. For SMART disk
|
||||
temperatures on rows that still render, the provenance travels on
|
||||
`agent.sensors.smart[].collection`: the Machines temperature cell
|
||||
(`AgentsMachinesTable.tsx`, `agentMachineTableModel.ts`), its tooltip and the
|
||||
drawer's Thermals rows (`resourceDetailMappers.ts`) read it through
|
||||
@@ -6030,6 +6032,34 @@ to that boolean. Identity succession continues to rekey the same state row.
|
||||
round-trip persistence, attention suppression, and the retired remediation
|
||||
lock.
|
||||
|
||||
### A push reporter past its lease is offline
|
||||
|
||||
The host agent, Docker and Kubernetes collectors push their own reports, and
|
||||
the monitor marks the machine, Docker host or cluster offline only once its
|
||||
reporting lease runs out. Ingest records that verdict on the source's
|
||||
sighting, and `aggregateStatus` counts such a sighting as `offline` whatever
|
||||
its age. The stale pass used to rank the stale sighting above `offline`, so a
|
||||
silent standalone agent reached every consumer as `warning`. The Machines
|
||||
table then showed it amber with its last report rendered as current readings,
|
||||
while its own drawer led with a critical "Host is offline" alert. A Docker host
|
||||
was offline only until its sighting crossed the 120-second stale threshold,
|
||||
then flipped to `warning`. With this rule, such rows take the existing
|
||||
offline treatment: frontend danger gates, the health verdict's `offline`
|
||||
reason, offline filters and counts.
|
||||
|
||||
`SourceStatus.Status` keeps describing delivery freshness, so the sighting
|
||||
still reads `stale`. Health keeps its `telemetry_stale` reason, and
|
||||
monitored-system reasons are unchanged. A live source still carries a merged
|
||||
row: a Proxmox node whose linked agent stopped reporting stays `online` through
|
||||
the PVE poll. Pull sources (Proxmox, PBS, PMG, TrueNAS, vSphere) and the
|
||||
resources push collectors report about (guests, containers, pods, disks) keep
|
||||
the stale-to-warning rule. A JSON copy re-derives the unexported marker from
|
||||
its stored status in `IngestResources`, and a manual link that joins two
|
||||
resources reported by one source keeps the fresher sighting. The tests in
|
||||
`registry_merge_policy_test.go` pin the agent, Docker, Kubernetes,
|
||||
mixed-source, round-trip and manual-link cases, plus the boundaries that stay
|
||||
unchanged.
|
||||
|
||||
### Canonical object drawer hierarchy
|
||||
|
||||
The unified resource drawer is the shared operator-facing detail contract for
|
||||
|
||||
@@ -0,0 +1,49 @@
|
||||
{
|
||||
"version": 1,
|
||||
"base_sha": "723f3165a1292268634b6054ab93d3015805d990",
|
||||
"verified_at": "2026-10-06T23:14:41Z",
|
||||
"result": "passed",
|
||||
"changed_paths": [
|
||||
"frontend-modern/src/components/Infrastructure/ResourceDetailSummary.tsx",
|
||||
"frontend-modern/src/components/Infrastructure/resourceDetailMappers.ts",
|
||||
"frontend-modern/src/components/Infrastructure/useResourceDetailDrawerDerivedState.ts",
|
||||
"frontend-modern/src/features/docker/DockerHostsTable.tsx",
|
||||
"frontend-modern/src/features/standalone/AgentsMachinesTable.tsx"
|
||||
],
|
||||
"content_sha256": {
|
||||
"frontend-modern/src/components/Infrastructure/ResourceDetailSummary.tsx": "f141220216a968f5953b7646035108f015a9f7de3c5ae5a9e3251ec6e7046fa5",
|
||||
"frontend-modern/src/components/Infrastructure/resourceDetailMappers.ts": "07b26eafecbd1ec93fb1dd90d4af6629778cef59a2d9e7450426cb88c90bdfdb",
|
||||
"frontend-modern/src/components/Infrastructure/useResourceDetailDrawerDerivedState.ts": "70955e370c5b2611557a5de8ba4df44b8f6dce47c1b08c611a154d232c786501",
|
||||
"frontend-modern/src/features/docker/DockerHostsTable.tsx": "450cce7a4723ce7fe7b01ffe9d10c7d6439c34a1a6dda4fc0af6689ceb4741b5",
|
||||
"frontend-modern/src/features/standalone/AgentsMachinesTable.tsx": "8d63a04f6a475f39de90cb9813b8bd4315525bf63318cfaf958ac195ae07950f"
|
||||
},
|
||||
"routes": [
|
||||
"/standalone/machines",
|
||||
"/docker"
|
||||
],
|
||||
"viewports": [
|
||||
{
|
||||
"width": 1440,
|
||||
"height": 900
|
||||
},
|
||||
{
|
||||
"width": 390,
|
||||
"height": 844
|
||||
}
|
||||
],
|
||||
"states": [
|
||||
"Two private mock stacks (8-node estate) built from this tree: one with this commit's registry.go/types.go, one with origin/main's, both with a local-only go build -overlay of internal/mock/generator.go that holds standalone Linux agent host-linux-1 (Apollo-114) and Docker host edge-proxy-01 past their reporting leases the way evaluateHostAgents/evaluateDockerAgents leave them: status offline, LastSeen frozen 14m/9m before the first tick, last readings kept. Mock mode skips the host health sweep, hence the overlay.",
|
||||
"Before (origin/main backend and frontend): /api/resources served Apollo-114 as status warning with agent.stale true and edge-proxy-01 as warning with health stale. The Machines row showed an amber dot, CPU 32%, Memory 74%, Disk 80%, Net and Disk I/O rates, Up 29d and Temp 45\u00b0C in threshold green beside 'Seen 14m ago', while the drawer led with the critical 'Host Apollo-114 is offline' alert and read 'Observed state Degraded, Uptime 29d 12h'.",
|
||||
"After (this commit): both resources serve status offline, the agent/docker sightings still read stale, and health reads critical with reasons offline then telemetry_stale. The Apollo-114 row shows a red dot and '\u2014' in CPU, Memory, Disk, Net I/O, Disk I/O, Up and Temp at 1440 and 390; the status filter counts it under Offline. Spectrum-709 (reporting) keeps every value, Temp 47\u00b0C included.",
|
||||
"Machine drawer at 1440 and 390: Runtime context reads 'Observed state Offline' and 'Last seen N mins ago' with no Uptime row; Platform details > Machine 'Show details' drops the System card's Uptime and the Thermals card reads 'Cpu.package 45\u00b0C (last known)' with title 'Last known reading, not current: host agent stopped reporting'.",
|
||||
"Docker page at 1440 and 390: edge-proxy-01 shows a red dot, '\u2014' for CPU, Memory, Disk and Temp (the hold seeds a retained 51\u00b0C), and the existing 'No report for 16m' marker in Uptime; Edge Apps 01 keeps its values, Temp 38.5\u00b0C included.",
|
||||
"Bundle check: WorkloadsSurface unchanged (70.20 kB), Standalone +0.04 kB, Docker +0.01 kB, InfrastructureWorkspace flat versus this tree without the frontend diff; all chunks within budget.",
|
||||
"Not covered: a real agent going silent on hardware, light/dark differences (no new colour classes), and the drawer Disks card, which still threshold-colours a silent agent's retained filesystem usage (left for a follow-up)."
|
||||
],
|
||||
"interactions": [
|
||||
"Enabled the Network, Disk I/O, Uptime and Temp columns through the Machines column preferences (localStorage init) and read every cell of the offline and reporting rows plus the status dot class at 1440 and 390.",
|
||||
"Opened the Apollo-114 drawer by row click at both widths, read the Runtime context rows, expanded Platform details and the Machine 'Show details', and read the System and Thermals cards with their value titles.",
|
||||
"Opened /docker at both widths and read every cell of edge-proxy-01 and Edge Apps 01.",
|
||||
"Queried /api/resources on both stacks for status, per-source sighting status and health reasons of the held agent and Docker host."
|
||||
]
|
||||
}
|
||||
@@ -217,9 +217,13 @@ export const InlineResourceSummaryTables: Component<ResourceSummaryPresentationP
|
||||
const now = useRelativeTimeNow();
|
||||
const sections = (): DetailSection[] => {
|
||||
const docker = dockerContainerMeta(props.resource);
|
||||
// An offline resource has no current uptime: the retained value is from
|
||||
// its last report, and the Last seen row already says when that was.
|
||||
const uptimeSeconds = docker
|
||||
? getDockerContainerUptimeSeconds(props.resource)
|
||||
: props.resource.uptime;
|
||||
: props.resource.status === 'offline'
|
||||
? undefined
|
||||
: props.resource.uptime;
|
||||
const identityRows = compactDetailRows([
|
||||
...props.drawer.primaryIdentityRows().map((row) => makeDetailRow(row.label, row.value)),
|
||||
props.showPlatformId ? makeDetailRow('Platform ID', props.resource.platformId) : null,
|
||||
|
||||
+64
@@ -0,0 +1,64 @@
|
||||
import { cleanup, render, screen, within } from '@solidjs/testing-library';
|
||||
import { afterEach, describe, expect, it } from 'vitest';
|
||||
|
||||
import type { Resource } from '@/types/resource';
|
||||
import { InlineResourceSummaryTables } from '../ResourceDetailSummary';
|
||||
import type { UseResourceDetailDrawerStateResult } from '../useResourceDetailDrawerState';
|
||||
|
||||
const drawerState = {
|
||||
primaryIdentityRows: () => [],
|
||||
identityIpValues: () => [],
|
||||
identityAliasValues: () => [],
|
||||
aliasPreviewValues: () => [],
|
||||
hasAliasOverflow: () => false,
|
||||
identityCardHasRichData: () => true,
|
||||
sourceSummary: () => null,
|
||||
lastSeen: () => '14 mins ago',
|
||||
lastSeenAbsolute: () => '',
|
||||
} as unknown as UseResourceDetailDrawerStateResult;
|
||||
|
||||
const machine = (status: Resource['status']): Resource => ({
|
||||
id: 'agent-apollo',
|
||||
name: 'Apollo-114',
|
||||
displayName: 'Apollo-114',
|
||||
platformId: 'host-linux-1',
|
||||
platformType: 'agent',
|
||||
sourceType: 'agent',
|
||||
type: 'agent',
|
||||
status,
|
||||
uptime: 2_500_000,
|
||||
lastSeen: Date.parse('2026-10-06T22:00:00Z'),
|
||||
});
|
||||
|
||||
afterEach(cleanup);
|
||||
|
||||
describe('resource drawer runtime context for an offline resource', () => {
|
||||
it('drops the uptime of an offline machine and keeps when it was last seen', () => {
|
||||
render(() => (
|
||||
<InlineResourceSummaryTables
|
||||
resource={machine('offline')}
|
||||
drawer={drawerState}
|
||||
showPlatformId={false}
|
||||
/>
|
||||
));
|
||||
|
||||
const section = screen.getByTestId('resource-runtime-context-section');
|
||||
expect(within(section).getByText('offline')).toBeInTheDocument();
|
||||
expect(within(section).queryByText('Uptime')).toBeNull();
|
||||
expect(within(section).getByText('Last seen')).toBeInTheDocument();
|
||||
expect(within(section).getByText('14 mins ago')).toBeInTheDocument();
|
||||
});
|
||||
|
||||
it('keeps the uptime of a machine that is up', () => {
|
||||
render(() => (
|
||||
<InlineResourceSummaryTables
|
||||
resource={machine('online')}
|
||||
drawer={drawerState}
|
||||
showPlatformId={false}
|
||||
/>
|
||||
));
|
||||
|
||||
const section = screen.getByTestId('resource-runtime-context-section');
|
||||
expect(within(section).getByText('Uptime')).toBeInTheDocument();
|
||||
});
|
||||
});
|
||||
@@ -2,6 +2,7 @@ import { describe, expect, it } from 'vitest';
|
||||
import {
|
||||
buildCustomSensorRows,
|
||||
buildTemperatureRows,
|
||||
HOST_AGENT_STOPPED_REPORTING_REASON,
|
||||
formatInteger,
|
||||
formatSensorName,
|
||||
formatSourceType,
|
||||
@@ -248,6 +249,37 @@ describe('resourceDetailMappers', () => {
|
||||
]);
|
||||
});
|
||||
|
||||
it("marks a silent agent's sensor readings last known and leaves disks to their own state", () => {
|
||||
const rows = buildTemperatureRows(
|
||||
{
|
||||
thermalState: { pressure: 'serious' },
|
||||
temperatureCelsius: { 'cpu.package': 45 },
|
||||
fanRpm: { chassis_fan: 1200 },
|
||||
smart: [
|
||||
{
|
||||
device: 'sda',
|
||||
temperature: 38,
|
||||
collection: { temperature: { state: 'available', source: 'smartctl' } },
|
||||
},
|
||||
],
|
||||
},
|
||||
{ lastKnownReason: HOST_AGENT_STOPPED_REPORTING_REASON },
|
||||
);
|
||||
|
||||
const lastKnown = 'Last known reading, not current: host agent stopped reporting';
|
||||
expect(rows).toEqual([
|
||||
{ label: 'Thermal pressure', value: 'Serious (last known)', valueTitle: lastKnown },
|
||||
{ label: 'Cpu.package', value: '45°C (last known)', valueTitle: lastKnown },
|
||||
{ label: 'Chassis Fan', value: '1,200 RPM (last known)', valueTitle: lastKnown },
|
||||
// A disk row follows its own collection state, which the backend
|
||||
// withdraws when the lease expires.
|
||||
{ label: 'Disk sda', value: '38°C', valueTitle: '38.0°C' },
|
||||
]);
|
||||
expect(buildTemperatureRows({ temperatureCelsius: { 'cpu.package': 45 } })).toEqual([
|
||||
{ label: 'Cpu.package', value: '45°C', valueTitle: '45.0°C' },
|
||||
]);
|
||||
});
|
||||
|
||||
it('lists a SMART row only for a positive reading in any state, never a standby disk', () => {
|
||||
const states: (PhysicalDiskCollectionStatus | undefined)[] = [
|
||||
undefined,
|
||||
@@ -442,6 +474,15 @@ describe('resourceDetailMappers', () => {
|
||||
expect(agent?.id).not.toBe('resource:host:hash-1');
|
||||
});
|
||||
|
||||
it('reports no uptime for a silent agent, whose last report is not how long it has been up', () => {
|
||||
const base = createHybridHostResource();
|
||||
const reporting = { ...base, uptime: 2_500_000 } as Resource;
|
||||
expect(toAgentFromResource(reporting)?.uptimeSeconds).toBe(2_500_000);
|
||||
|
||||
const silent = { ...reporting, agent: { stale: true } } as Resource;
|
||||
expect(toAgentFromResource(silent)?.uptimeSeconds).toBe(0);
|
||||
});
|
||||
|
||||
it('preserves the local infrastructure display name for governed resources', () => {
|
||||
const agent = toAgentFromResource({
|
||||
...createHybridHostResource(),
|
||||
|
||||
@@ -317,7 +317,9 @@ export const toAgentFromResource = (
|
||||
sensors: agent.sensors,
|
||||
raid: agent.raid,
|
||||
status: resource.status,
|
||||
uptimeSeconds: agent.uptimeSeconds ?? resource.uptime ?? 0,
|
||||
// A silent agent's uptime is from its last report, not how long the
|
||||
// machine has been up.
|
||||
uptimeSeconds: resource.agent?.stale ? 0 : (agent.uptimeSeconds ?? resource.uptime ?? 0),
|
||||
lastSeen: resource.lastSeen,
|
||||
agentVersion: agent.agentVersion,
|
||||
commandsEnabled: agent.commandsEnabled,
|
||||
@@ -400,7 +402,23 @@ const buildTypedGPUTemperatureKeys = (gpus?: HostSensorSummary['gpu']) => {
|
||||
return keys;
|
||||
};
|
||||
|
||||
export const buildTemperatureRows = (sensors?: HostSensorSummary) => {
|
||||
// Matches the backend's reason for readings a host agent past its reporting
|
||||
// lease no longer collects (models.HostAgentStoppedReportingReason).
|
||||
export const HOST_AGENT_STOPPED_REPORTING_REASON = 'host agent stopped reporting';
|
||||
|
||||
export type TemperatureRowsOptions = {
|
||||
/**
|
||||
* Why the machine's own sensor readings are retained rather than current,
|
||||
* such as a host agent past its reporting lease. Disk rows keep their own
|
||||
* collection state instead.
|
||||
*/
|
||||
lastKnownReason?: string;
|
||||
};
|
||||
|
||||
export const buildTemperatureRows = (
|
||||
sensors?: HostSensorSummary,
|
||||
options: TemperatureRowsOptions = {},
|
||||
) => {
|
||||
const rows: { label: string; value: string; valueTitle?: string }[] = [];
|
||||
const thermalState = sensors?.thermalState;
|
||||
const typedGPUTemperatureKeys = buildTypedGPUTemperatureKeys(sensors?.gpu);
|
||||
@@ -500,6 +518,17 @@ export const buildTemperatureRows = (sensors?: HostSensorSummary) => {
|
||||
});
|
||||
}
|
||||
|
||||
// This card has no other cue that a value is not current, so a retained
|
||||
// sensor reading says so in its value, as a retained disk reading does.
|
||||
const lastKnownReason = options.lastKnownReason?.trim();
|
||||
if (lastKnownReason) {
|
||||
const title = `Last known reading, not current: ${lastKnownReason}`;
|
||||
for (const row of rows) {
|
||||
row.value = `${row.value} (last known)`;
|
||||
row.valueTitle = title;
|
||||
}
|
||||
}
|
||||
|
||||
const smart = sensors?.smart;
|
||||
if (smart) {
|
||||
smart
|
||||
|
||||
+8
-1
@@ -35,6 +35,7 @@ import type { ResourceIntelligence } from '@/types/aiIntelligence';
|
||||
import {
|
||||
buildCustomSensorRows,
|
||||
buildTemperatureRows,
|
||||
HOST_AGENT_STOPPED_REPORTING_REASON,
|
||||
toAgentFromResource,
|
||||
toNodeFromProxmox,
|
||||
type AgentPlatformData,
|
||||
@@ -201,7 +202,13 @@ export const useResourceDetailDrawerDerivedState = (
|
||||
[],
|
||||
);
|
||||
const agentInfo = createMemo(() => toAgentFromResource(resource(), agentMeta()));
|
||||
const temperatureRows = createMemo(() => buildTemperatureRows(agentInfo()?.sensors));
|
||||
// A silent agent's sensors are its last report, even on a row another
|
||||
// source keeps online.
|
||||
const temperatureRows = createMemo(() =>
|
||||
buildTemperatureRows(agentInfo()?.sensors, {
|
||||
lastKnownReason: resource().agent?.stale ? HOST_AGENT_STOPPED_REPORTING_REASON : undefined,
|
||||
}),
|
||||
);
|
||||
const customSensorRows = createMemo(() => buildCustomSensorRows(agentInfo()?.sensors));
|
||||
|
||||
const dockerHostData = createMemo(() => platformData()?.docker as DockerPlatformData | undefined);
|
||||
|
||||
@@ -517,7 +517,11 @@ export const DockerHostsTable: Component<{
|
||||
class={`${getPlatformTableCellClassForKind('numeric-value')} platform-table-narrow-hidden hidden text-base-content min-[360px]:table-cell md:table-cell`}
|
||||
>
|
||||
<PlatformTableTemperatureValue
|
||||
value={host.temperature ?? docker()?.temperature}
|
||||
value={
|
||||
canRenderMetrics()
|
||||
? (host.temperature ?? docker()?.temperature)
|
||||
: undefined
|
||||
}
|
||||
/>
|
||||
</TableCell>
|
||||
<Show when={showSwarmColumn()}>
|
||||
|
||||
@@ -217,6 +217,58 @@ describe('DockerHostsTable', () => {
|
||||
);
|
||||
});
|
||||
|
||||
it("drops an offline host's retained temperature with its other readings", () => {
|
||||
// A Docker host past its reporting lease is offline, and its last
|
||||
// temperature is no more current than its last CPU, memory and disk.
|
||||
const { container } = render(() => (
|
||||
<DockerHostsTable
|
||||
resources={[
|
||||
makeDockerHost({
|
||||
status: 'offline',
|
||||
temperature: 51,
|
||||
health: {
|
||||
verdict: 'critical',
|
||||
reasons: [
|
||||
{ code: 'offline', detail: '' },
|
||||
{ code: 'telemetry_stale', detail: '9m' },
|
||||
],
|
||||
},
|
||||
}),
|
||||
makeDockerHost({
|
||||
id: 'agent:docker-02',
|
||||
name: 'docker-02',
|
||||
displayName: 'docker-02',
|
||||
temperature: 51,
|
||||
}),
|
||||
]}
|
||||
emptyIcon={<span />}
|
||||
emptyTitle="No Docker hosts"
|
||||
emptyDescription="No hosts"
|
||||
showToolbar={false}
|
||||
/>
|
||||
));
|
||||
|
||||
const headers = Array.from(container.querySelectorAll('thead th')).map((th) =>
|
||||
(th.textContent ?? '').trim(),
|
||||
);
|
||||
const tempIndex = headers.findIndex((header) => header.startsWith('Temp'));
|
||||
expect(tempIndex).toBeGreaterThan(-1);
|
||||
const temp = (id: string) =>
|
||||
(
|
||||
container.querySelectorAll(`[data-docker-host-row="${id}"] td`)[tempIndex]?.textContent ??
|
||||
''
|
||||
).trim();
|
||||
expect(temp('agent:docker-01')).toBe('—');
|
||||
expect(temp('agent:docker-02')).toBe('51.0°C');
|
||||
const offlineRow = container.querySelector('[data-docker-host-row="agent:docker-01"]')!;
|
||||
expect(within(offlineRow as HTMLElement).queryAllByTestId('stacked-memory-bar')).toHaveLength(
|
||||
0,
|
||||
);
|
||||
expect(offlineRow.querySelector('[data-docker-host-stale] .sr-only')).toHaveTextContent(
|
||||
'No report for 9m',
|
||||
);
|
||||
});
|
||||
|
||||
it('tints a host row for the Docker alerts its drawer lists, not by display name', () => {
|
||||
// Docker alerts are keyed on the Docker host id and carry the hostname
|
||||
// as their node; neither is the row's id or its display name.
|
||||
|
||||
@@ -1806,7 +1806,9 @@ export const AgentsMachinesTable: Component<{
|
||||
class={`${getPlatformTableCellClassForKind('numeric-value')} ${machineColumnWidthClass('uptime')} text-base-content`}
|
||||
>
|
||||
{formatPlatformTableUptimeValue(
|
||||
machine.uptime ?? machine.agent?.uptimeSeconds,
|
||||
canRenderMetrics()
|
||||
? (machine.uptime ?? machine.agent?.uptimeSeconds)
|
||||
: undefined,
|
||||
)}
|
||||
</TableCell>
|
||||
</Show>
|
||||
@@ -1814,15 +1816,20 @@ export const AgentsMachinesTable: Component<{
|
||||
<TableCell
|
||||
class={`${getPlatformTableCellClassForKind('numeric-value')} ${machineColumnWidthClass('temp')} text-base-content`}
|
||||
>
|
||||
<AgentMachineTemperatureCell
|
||||
celsius={temperature()}
|
||||
lastKnown={temperatureLastKnown()}
|
||||
metric={temperatureMetric()}
|
||||
sections={temperatureSections()}
|
||||
thresholds={temperatureThresholds()}
|
||||
title={temperatureTitle()}
|
||||
thermalPressure={thermalPressure()}
|
||||
/>
|
||||
<Show
|
||||
when={canRenderMetrics()}
|
||||
fallback={<span class="text-muted">—</span>}
|
||||
>
|
||||
<AgentMachineTemperatureCell
|
||||
celsius={temperature()}
|
||||
lastKnown={temperatureLastKnown()}
|
||||
metric={temperatureMetric()}
|
||||
sections={temperatureSections()}
|
||||
thresholds={temperatureThresholds()}
|
||||
title={temperatureTitle()}
|
||||
thermalPressure={thermalPressure()}
|
||||
/>
|
||||
</Show>
|
||||
</TableCell>
|
||||
</Show>
|
||||
<Show when={columnVisibility.isColumnVisible('lastSeen')}>
|
||||
|
||||
@@ -1160,14 +1160,15 @@ describe('AgentsMachinesTable', () => {
|
||||
expect(screen.getByText('1400 RPM')).toBeInTheDocument();
|
||||
});
|
||||
|
||||
it('shows a silent agent disk temperature as last known, never as a current reading', async () => {
|
||||
// A host agent past its reporting lease is stale and keeps its last SMART
|
||||
// temperatures with a non-available collection state.
|
||||
it('shows a retained disk temperature as last known, never as a current reading', async () => {
|
||||
// A reporting agent whose SMART probe returned no temperature keeps the
|
||||
// last reading with a non-available collection state. (A host agent past
|
||||
// its lease is offline, and its row shows no temperature at all.)
|
||||
const stoppedReporting = {
|
||||
temperature: {
|
||||
state: 'unavailable' as const,
|
||||
source: 'smartctl',
|
||||
reason: 'host agent stopped reporting',
|
||||
reason: 'SMART probe returned no usable temperature data',
|
||||
},
|
||||
};
|
||||
const { container } = render(() => (
|
||||
@@ -1176,9 +1177,7 @@ describe('AgentsMachinesTable', () => {
|
||||
resource({
|
||||
id: 'silent-nas',
|
||||
name: 'Silent NAS',
|
||||
status: 'warning',
|
||||
agent: {
|
||||
stale: true,
|
||||
sensors: {
|
||||
smart: [
|
||||
{ device: '/dev/sda', model: 'Archive HDD', temperature: 71 },
|
||||
@@ -1249,6 +1248,92 @@ describe('AgentsMachinesTable', () => {
|
||||
expect(within(mixed).queryByText('69°C')).toBeNull();
|
||||
});
|
||||
|
||||
it("blanks an offline machine's retained readings, temperature and uptime included", async () => {
|
||||
// A host agent past its reporting lease is offline: CPU, memory, disk and
|
||||
// the sensors, uptime and disks it last reported are not current.
|
||||
const stoppedReporting = {
|
||||
temperature: {
|
||||
state: 'unavailable' as const,
|
||||
source: 'smartctl',
|
||||
reason: 'host agent stopped reporting',
|
||||
},
|
||||
};
|
||||
const lastReport: Partial<Resource> = {
|
||||
cpu: { current: 32 },
|
||||
memory: { total: 100, used: 73, free: 27, current: 73 },
|
||||
disk: { total: 100, used: 80, free: 20, current: 80 },
|
||||
};
|
||||
const sensors = {
|
||||
temperatureCelsius: { 'cpu.package': 45 },
|
||||
thermalState: { pressure: 'serious' },
|
||||
smart: [
|
||||
{
|
||||
device: '/dev/sda',
|
||||
model: 'Archive HDD',
|
||||
temperature: 41,
|
||||
collection: stoppedReporting,
|
||||
},
|
||||
],
|
||||
};
|
||||
const { container } = render(() => (
|
||||
<AgentsMachinesTable
|
||||
resources={[
|
||||
resource({
|
||||
id: 'silent-host',
|
||||
name: 'Silent Host',
|
||||
status: 'offline',
|
||||
uptime: 2_500_000,
|
||||
...lastReport,
|
||||
agent: { stale: true, uptimeSeconds: 2_500_000, sensors },
|
||||
}),
|
||||
resource({
|
||||
id: 'live-host',
|
||||
name: 'Live Host',
|
||||
uptime: 2_500_000,
|
||||
...lastReport,
|
||||
agent: {
|
||||
uptimeSeconds: 2_500_000,
|
||||
sensors: { temperatureCelsius: { 'cpu.package': 45 } },
|
||||
},
|
||||
}),
|
||||
]}
|
||||
emptyIcon={emptyIcon}
|
||||
emptyTitle="No machines"
|
||||
emptyDescription="Install Pulse Agent."
|
||||
/>
|
||||
));
|
||||
|
||||
await openMachineColumnPicker();
|
||||
await fireEvent.click(screen.getByLabelText('Temp'));
|
||||
await fireEvent.click(screen.getByLabelText('Uptime'));
|
||||
|
||||
const cells = (id: string) => {
|
||||
const row = container.querySelector(`[data-agents-machine-row="${id}"]`) as HTMLElement;
|
||||
const headers = Array.from(container.querySelectorAll('thead th')).map((th) =>
|
||||
(th.textContent ?? '').replace(/[▲▼]/g, '').trim(),
|
||||
);
|
||||
const values = Array.from(row.querySelectorAll('td')).map((td) =>
|
||||
(td.textContent ?? '').trim(),
|
||||
);
|
||||
return (label: string) => values[headers.indexOf(label)];
|
||||
};
|
||||
|
||||
const silent = cells('silent-host');
|
||||
expect(silent('CPU')).toBe('—');
|
||||
expect(silent('Up')).toBe('—');
|
||||
expect(silent('Temp')).toBe('—');
|
||||
expect(
|
||||
container.querySelector(
|
||||
'[data-agents-machine-row="silent-host"] [data-agent-machine-temperature-trigger="true"]',
|
||||
),
|
||||
).toBeNull();
|
||||
|
||||
const live = cells('live-host');
|
||||
expect(live('CPU')).not.toBe('—');
|
||||
expect(live('Up')).toBe('28d');
|
||||
expect(live('Temp')).toBe('45°C');
|
||||
});
|
||||
|
||||
it('moves a machine temperature between current and last known as its disk reports change', async () => {
|
||||
// Fresh objects per update, like parsed API payloads: the store reconciles
|
||||
// into the objects it was given, so a reused fixture would be rewritten.
|
||||
|
||||
@@ -221,12 +221,18 @@ func TestCloneResource_MutateSourceStatusMap(t *testing.T) {
|
||||
ID: "r-1",
|
||||
SourceStatus: map[DataSource]SourceStatus{
|
||||
SourceProxmox: {Status: "online", ExpectedUpdateIntervalSeconds: 300},
|
||||
SourceAgent: {Status: "stale", leaseExpired: true},
|
||||
},
|
||||
}
|
||||
cloned := cloneResource(original)
|
||||
if cloned.SourceStatus[SourceProxmox].ExpectedUpdateIntervalSeconds != 300 {
|
||||
t.Fatal("clone lost collector cadence")
|
||||
}
|
||||
// A clone that dropped the lease verdict would aggregate a silent agent
|
||||
// back to warning.
|
||||
if !cloned.SourceStatus[SourceAgent].leaseExpired {
|
||||
t.Fatal("clone lost the agent's expired reporting lease")
|
||||
}
|
||||
|
||||
cloned.SourceStatus[SourceDocker] = SourceStatus{Status: "online"}
|
||||
if _, exists := original.SourceStatus[SourceDocker]; exists {
|
||||
|
||||
@@ -809,6 +809,14 @@ func (rr *ResourceRegistry) ingestResources(resources []Resource, thresholds map
|
||||
}
|
||||
}
|
||||
}
|
||||
// The lease marker is unexported, so a persisted or serialized copy
|
||||
// arrives without it. Its stored status still carries the verdict.
|
||||
for source, status := range resource.SourceStatus {
|
||||
if !status.leaseExpired && reportingLeaseExpired(source, *resource) {
|
||||
status.leaseExpired = true
|
||||
resource.SourceStatus[source] = status
|
||||
}
|
||||
}
|
||||
|
||||
rr.mu.Lock()
|
||||
rr.canonicalIdentityIndex = nil
|
||||
@@ -1756,6 +1764,29 @@ func sourceSightingStatus(lastSeen time.Time) string {
|
||||
return "online"
|
||||
}
|
||||
|
||||
// reportingLeaseExpired reports whether a source delivered the machine,
|
||||
// Docker host or Kubernetes cluster it reports for as offline. Those
|
||||
// collectors push their own reports, and Pulse marks the reporter offline
|
||||
// only once its reporting lease runs out (evaluateHostAgents,
|
||||
// evaluateDockerAgents, evaluateKubernetesAgents). The registry's stale
|
||||
// threshold is a different clock: it can mark the sighting stale before
|
||||
// the lease ends, or leave it fresh after, so it must not decide the status.
|
||||
// Resources those sources describe (guests, containers, pods, disks) are
|
||||
// not lease holders and keep the stale-to-warning rule.
|
||||
func reportingLeaseExpired(source DataSource, resource Resource) bool {
|
||||
if resource.Status != StatusOffline {
|
||||
return false
|
||||
}
|
||||
switch source {
|
||||
case SourceAgent, SourceDocker:
|
||||
return resource.Type == ResourceTypeAgent
|
||||
case SourceK8s:
|
||||
return resource.Type == ResourceTypeK8sCluster
|
||||
default:
|
||||
return false
|
||||
}
|
||||
}
|
||||
|
||||
func (rr *ResourceRegistry) markStaleLocked(now time.Time, thresholds map[DataSource]time.Duration) {
|
||||
thresholds = effectiveStaleThresholds(thresholds)
|
||||
|
||||
@@ -2848,6 +2879,7 @@ func (rr *ResourceRegistry) ingestRecord(source DataSource, sourceID string, res
|
||||
sighting := resource.SourceStatus[source]
|
||||
sighting.Status = sourceSightingStatus(resource.LastSeen)
|
||||
sighting.LastSeen = resource.LastSeen
|
||||
sighting.leaseExpired = reportingLeaseExpired(source, resource)
|
||||
resource.SourceStatus = map[DataSource]SourceStatus{source: sighting}
|
||||
resource.parentBySource = make(map[DataSource]string)
|
||||
rr.setSourceParent(&resource, source, resource.ParentID)
|
||||
@@ -3787,6 +3819,7 @@ func (rr *ResourceRegistry) mergeInto(existing *Resource, incoming Resource, sou
|
||||
sighting := incoming.SourceStatus[source]
|
||||
sighting.Status = sourceSightingStatus(incoming.LastSeen)
|
||||
sighting.LastSeen = incoming.LastSeen
|
||||
sighting.leaseExpired = reportingLeaseExpired(source, incoming)
|
||||
existing.SourceStatus[source] = sighting
|
||||
|
||||
if incoming.LastSeen.After(existing.LastSeen) {
|
||||
@@ -4425,6 +4458,12 @@ func (rr *ResourceRegistry) mergeResourceData(primary *Resource, other *Resource
|
||||
primary.SourceStatus = make(map[DataSource]SourceStatus)
|
||||
}
|
||||
for source, status := range other.SourceStatus {
|
||||
// Linked resources can share a source, such as an agent reinstalled
|
||||
// under a new id. Keep the fresher sighting so a stale or expired one
|
||||
// cannot override a reporter that is still live.
|
||||
if current, ok := primary.SourceStatus[source]; ok && current.LastSeen.After(status.LastSeen) {
|
||||
continue
|
||||
}
|
||||
primary.SourceStatus[source] = status
|
||||
}
|
||||
if other.LastSeen.After(primary.LastSeen) {
|
||||
@@ -5793,14 +5832,22 @@ func aggregateStatus(resource *Resource) ResourceStatus {
|
||||
best := StatusUnknown
|
||||
bestScore := 0
|
||||
for _, status := range resource.SourceStatus {
|
||||
score := statusPriority[strings.ToLower(status.Status)]
|
||||
state := strings.ToLower(strings.TrimSpace(status.Status))
|
||||
// A reporter past its lease is offline, not merely late. Ranking its
|
||||
// stale sighting above offline would show a silent machine as a
|
||||
// warning while its retained readings render as current.
|
||||
if status.leaseExpired {
|
||||
state = "offline"
|
||||
}
|
||||
score := statusPriority[state]
|
||||
if score > bestScore {
|
||||
bestScore = score
|
||||
if status.Status == "online" {
|
||||
switch state {
|
||||
case "online":
|
||||
best = StatusOnline
|
||||
} else if status.Status == "stale" {
|
||||
case "stale":
|
||||
best = StatusWarning
|
||||
} else if status.Status == "offline" {
|
||||
case "offline":
|
||||
best = StatusOffline
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
package unifiedresources
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
@@ -131,6 +132,353 @@ func TestHostnameIPDoesNotAutoMerge(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func onlyResourceOfType(t *testing.T, rr *ResourceRegistry, resourceType ResourceType) Resource {
|
||||
t.Helper()
|
||||
resources := rr.ListByType(resourceType)
|
||||
if len(resources) != 1 {
|
||||
t.Fatalf("expected one %s resource, got %d: %+v", resourceType, len(resources), resources)
|
||||
}
|
||||
return resources[0]
|
||||
}
|
||||
|
||||
// A host agent past its reporting lease is offline. Its source sighting is
|
||||
// stale too, and the stale pass used to rank that above offline, so the
|
||||
// machine reached the frontend as a warning with its last report rendered
|
||||
// as current readings.
|
||||
func TestLeaseExpiredHostAgentStaysOffline(t *testing.T) {
|
||||
lastReport := time.Now().UTC().Add(-14 * time.Minute)
|
||||
rr := NewRegistry(nil)
|
||||
rr.IngestSnapshot(models.StateSnapshot{
|
||||
Hosts: []models.Host{{
|
||||
ID: "host-silent",
|
||||
MachineID: "machine-silent",
|
||||
Hostname: "silent",
|
||||
Status: "offline",
|
||||
LastSeen: lastReport,
|
||||
IntervalSeconds: 30,
|
||||
CPUUsage: 32,
|
||||
}},
|
||||
})
|
||||
|
||||
resource := onlyResourceOfType(t, rr, ResourceTypeAgent)
|
||||
if resource.Status != StatusOffline {
|
||||
t.Fatalf("status = %q, want offline for an agent past its lease", resource.Status)
|
||||
}
|
||||
if resource.Agent == nil || !resource.Agent.Stale {
|
||||
t.Fatalf("expected the agent facet to stay flagged stale, got %+v", resource.Agent)
|
||||
}
|
||||
// Status describes the resource; the sighting keeps describing delivery
|
||||
// freshness, which health and monitored-system reasons read.
|
||||
if got := resource.SourceStatus[SourceAgent].Status; got != "stale" {
|
||||
t.Fatalf("agent sighting = %q, want stale", got)
|
||||
}
|
||||
|
||||
// The monitor adapter runs the stale pass again after record sources.
|
||||
rr.MarkStale(time.Now().UTC(), nil)
|
||||
if got, _ := rr.Get(resource.ID); got.Status != StatusOffline {
|
||||
t.Fatalf("second stale pass changed status to %q, want offline", got.Status)
|
||||
}
|
||||
|
||||
health := EvaluateResourceHealth(resource, nil, time.Now().UTC())
|
||||
if health.Verdict != HealthCritical || len(health.Reasons) < 2 ||
|
||||
health.Reasons[0].Code != "offline" || health.Reasons[1].Code != "telemetry_stale" {
|
||||
t.Fatalf("health = %+v, want critical offline with the stale reason after it", health)
|
||||
}
|
||||
}
|
||||
|
||||
// An agent that is late but still inside its lease keeps the warning: the
|
||||
// monitor has not decided it is gone, so its last readings are only stale.
|
||||
func TestLateHostAgentInsideLeaseStaysWarning(t *testing.T) {
|
||||
rr := NewRegistry(nil)
|
||||
rr.IngestSnapshot(models.StateSnapshot{
|
||||
Hosts: []models.Host{{
|
||||
ID: "host-late",
|
||||
MachineID: "machine-late",
|
||||
Hostname: "late",
|
||||
Status: "online",
|
||||
LastSeen: time.Now().UTC().Add(-90 * time.Second),
|
||||
IntervalSeconds: 30,
|
||||
}},
|
||||
})
|
||||
|
||||
resource := onlyResourceOfType(t, rr, ResourceTypeAgent)
|
||||
if resource.Status != StatusWarning {
|
||||
t.Fatalf("status = %q, want warning for a late agent inside its lease", resource.Status)
|
||||
}
|
||||
}
|
||||
|
||||
// A Proxmox node whose linked agent stopped reporting stays online through
|
||||
// the PVE poll (issue #1515). Only the agent sighting carries the lease.
|
||||
func TestLeaseExpiredAgentOnLiveProxmoxNodeStaysOnline(t *testing.T) {
|
||||
now := time.Now().UTC()
|
||||
const machineID = "machine-pve1"
|
||||
rr := NewRegistry(nil)
|
||||
rr.IngestSnapshot(models.StateSnapshot{
|
||||
Nodes: []models.Node{{
|
||||
ID: "homelab-pve1",
|
||||
Name: "pve1",
|
||||
Instance: "homelab",
|
||||
ClusterName: "homelab",
|
||||
Status: "online",
|
||||
LastSeen: now,
|
||||
}},
|
||||
Hosts: []models.Host{{
|
||||
ID: machineID,
|
||||
MachineID: machineID,
|
||||
Hostname: "pve1",
|
||||
LinkedNodeID: "homelab-pve1",
|
||||
Status: "offline",
|
||||
LastSeen: now.Add(-14 * time.Minute),
|
||||
IntervalSeconds: 30,
|
||||
}},
|
||||
})
|
||||
|
||||
resource := onlyResourceOfType(t, rr, ResourceTypeAgent)
|
||||
if _, ok := resource.SourceStatus[SourceProxmox]; !ok {
|
||||
t.Fatalf("expected the node and its agent to share one resource, got sources %+v", resource.SourceStatus)
|
||||
}
|
||||
if resource.Status != StatusOnline {
|
||||
t.Fatalf("status = %q, want online through the live PVE poll", resource.Status)
|
||||
}
|
||||
if resource.Agent == nil || !resource.Agent.Stale {
|
||||
t.Fatalf("expected the dead agent to stay flagged stale, got %+v", resource.Agent)
|
||||
}
|
||||
}
|
||||
|
||||
// Docker hosts hold a shorter lease than the registry's Docker stale
|
||||
// threshold, so a silent Docker host was offline for a while and then
|
||||
// flipped to warning once its sighting went stale.
|
||||
func TestLeaseExpiredDockerHostStaysOfflineAfterSightingGoesStale(t *testing.T) {
|
||||
lastReport := time.Now().UTC().Add(-5 * time.Minute)
|
||||
rr := NewRegistry(nil)
|
||||
rr.IngestSnapshot(models.StateSnapshot{
|
||||
DockerHosts: []models.DockerHost{{
|
||||
ID: "docker-silent",
|
||||
AgentID: "docker-agent-silent",
|
||||
Hostname: "docker-silent",
|
||||
MachineID: "machine-docker-silent",
|
||||
Status: "offline",
|
||||
LastSeen: lastReport,
|
||||
IntervalSeconds: 30,
|
||||
Containers: []models.DockerContainer{{
|
||||
ID: "container-web",
|
||||
Name: "web",
|
||||
State: "running",
|
||||
}},
|
||||
}},
|
||||
})
|
||||
|
||||
host := onlyResourceOfType(t, rr, ResourceTypeAgent)
|
||||
if host.Status != StatusOffline {
|
||||
t.Fatalf("docker host status = %q, want offline after its sighting went stale", host.Status)
|
||||
}
|
||||
if got := host.SourceStatus[SourceDocker].Status; got != "stale" {
|
||||
t.Fatalf("docker sighting = %q, want stale", got)
|
||||
}
|
||||
|
||||
// Containers are what the host reports about, not lease holders: a
|
||||
// running container on a silent host is stale, not known to be down.
|
||||
container := onlyResourceOfType(t, rr, ResourceTypeAppContainer)
|
||||
if container.Status != StatusWarning {
|
||||
t.Fatalf("container status = %q, want warning on a silent host", container.Status)
|
||||
}
|
||||
}
|
||||
|
||||
// A machine reporting through both the host agent and Docker is offline once
|
||||
// both leases expire. The Docker lease ends first, so its sighting can still
|
||||
// be fresh when the agent's goes stale.
|
||||
func TestMachineWithExpiredAgentAndDockerLeasesIsOffline(t *testing.T) {
|
||||
now := time.Now().UTC()
|
||||
const machineID = "machine-both"
|
||||
rr := NewRegistry(nil)
|
||||
rr.IngestSnapshot(models.StateSnapshot{
|
||||
Hosts: []models.Host{{
|
||||
ID: "host-both",
|
||||
MachineID: machineID,
|
||||
Hostname: "both",
|
||||
Status: "offline",
|
||||
LastSeen: now.Add(-3 * time.Minute),
|
||||
IntervalSeconds: 30,
|
||||
}},
|
||||
DockerHosts: []models.DockerHost{{
|
||||
ID: "docker-both",
|
||||
AgentID: "host-both",
|
||||
Hostname: "both",
|
||||
MachineID: machineID,
|
||||
Status: "offline",
|
||||
LastSeen: now.Add(-90 * time.Second),
|
||||
IntervalSeconds: 10,
|
||||
}},
|
||||
})
|
||||
|
||||
resource := onlyResourceOfType(t, rr, ResourceTypeAgent)
|
||||
if _, ok := resource.SourceStatus[SourceDocker]; !ok {
|
||||
t.Fatalf("expected agent and Docker to share one resource, got sources %+v", resource.SourceStatus)
|
||||
}
|
||||
if got := resource.SourceStatus[SourceDocker].Status; got != "online" {
|
||||
t.Fatalf("docker sighting = %q, want it still inside the stale threshold", got)
|
||||
}
|
||||
if resource.Status != StatusOffline {
|
||||
t.Fatalf("status = %q, want offline once both leases expired", resource.Status)
|
||||
}
|
||||
}
|
||||
|
||||
// A live reporter still carries the machine when the other lease expired.
|
||||
func TestMachineWithLiveDockerReporterStaysOnline(t *testing.T) {
|
||||
now := time.Now().UTC()
|
||||
const machineID = "machine-split"
|
||||
rr := NewRegistry(nil)
|
||||
rr.IngestSnapshot(models.StateSnapshot{
|
||||
Hosts: []models.Host{{
|
||||
ID: "host-split",
|
||||
MachineID: machineID,
|
||||
Hostname: "split",
|
||||
Status: "offline",
|
||||
LastSeen: now.Add(-14 * time.Minute),
|
||||
IntervalSeconds: 30,
|
||||
}},
|
||||
DockerHosts: []models.DockerHost{{
|
||||
ID: "docker-split",
|
||||
AgentID: "docker-agent-split",
|
||||
Hostname: "split",
|
||||
MachineID: machineID,
|
||||
Status: "online",
|
||||
LastSeen: now,
|
||||
IntervalSeconds: 10,
|
||||
}},
|
||||
})
|
||||
|
||||
resource := onlyResourceOfType(t, rr, ResourceTypeAgent)
|
||||
if resource.Status != StatusOnline {
|
||||
t.Fatalf("status = %q, want online through the live Docker reporter", resource.Status)
|
||||
}
|
||||
}
|
||||
|
||||
// The lease marker is unexported, so a resource that went through JSON (a
|
||||
// persisted or remote copy) must recover it from its stored status.
|
||||
func TestLeaseExpiredHostAgentStaysOfflineAfterJSONRoundTrip(t *testing.T) {
|
||||
source := NewRegistry(nil)
|
||||
source.IngestSnapshot(models.StateSnapshot{
|
||||
Hosts: []models.Host{{
|
||||
ID: "host-silent",
|
||||
MachineID: "machine-silent",
|
||||
Hostname: "silent",
|
||||
Status: "offline",
|
||||
LastSeen: time.Now().UTC().Add(-14 * time.Minute),
|
||||
IntervalSeconds: 30,
|
||||
}},
|
||||
})
|
||||
payload, err := json.Marshal(source.List())
|
||||
if err != nil {
|
||||
t.Fatalf("marshal resources: %v", err)
|
||||
}
|
||||
var copied []Resource
|
||||
if err := json.Unmarshal(payload, &copied); err != nil {
|
||||
t.Fatalf("unmarshal resources: %v", err)
|
||||
}
|
||||
|
||||
rr := NewRegistry(nil)
|
||||
rr.IngestResources(copied)
|
||||
|
||||
resource := onlyResourceOfType(t, rr, ResourceTypeAgent)
|
||||
if resource.Status != StatusOffline {
|
||||
t.Fatalf("status = %q, want offline after a JSON round trip", resource.Status)
|
||||
}
|
||||
}
|
||||
|
||||
// A manual link can join two resources reported by the same source, such as
|
||||
// an agent reinstalled under a new id. The expired sighting of the old one
|
||||
// must not replace the live sighting of the new one.
|
||||
func TestManualLinkKeepsTheLiveSightingOfASharedSource(t *testing.T) {
|
||||
now := time.Now().UTC()
|
||||
store := NewMemoryStore()
|
||||
if err := store.AddLink(ResourceLink{
|
||||
ResourceA: "agent-reinstalled",
|
||||
ResourceB: "agent-retired",
|
||||
PrimaryID: "agent-reinstalled",
|
||||
}); err != nil {
|
||||
t.Fatalf("add link: %v", err)
|
||||
}
|
||||
rr := NewRegistry(store)
|
||||
rr.IngestResources([]Resource{
|
||||
{
|
||||
ID: "agent-reinstalled",
|
||||
Type: ResourceTypeAgent,
|
||||
Name: "tower",
|
||||
Status: StatusOnline,
|
||||
LastSeen: now,
|
||||
Sources: []DataSource{SourceAgent},
|
||||
SourceStatus: map[DataSource]SourceStatus{
|
||||
SourceAgent: {Status: "online", LastSeen: now},
|
||||
},
|
||||
Agent: &AgentData{AgentID: "host-new", Hostname: "tower"},
|
||||
},
|
||||
{
|
||||
ID: "agent-retired",
|
||||
Type: ResourceTypeAgent,
|
||||
Name: "tower",
|
||||
Status: StatusOffline,
|
||||
LastSeen: now.Add(-14 * time.Minute),
|
||||
Sources: []DataSource{SourceAgent},
|
||||
SourceStatus: map[DataSource]SourceStatus{
|
||||
SourceAgent: {Status: "stale", LastSeen: now.Add(-14 * time.Minute)},
|
||||
},
|
||||
Agent: &AgentData{AgentID: "host-old", Hostname: "tower", Stale: true},
|
||||
},
|
||||
})
|
||||
|
||||
resource, ok := rr.Get("agent-reinstalled")
|
||||
if !ok {
|
||||
t.Fatal("linked resource missing")
|
||||
}
|
||||
if resource.Status != StatusOnline {
|
||||
t.Fatalf("status = %q, want online through the live reinstalled agent", resource.Status)
|
||||
}
|
||||
if got := resource.SourceStatus[SourceAgent]; !got.LastSeen.Equal(now) {
|
||||
t.Fatalf("agent sighting = %+v, want the live sighting", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLeaseExpiredKubernetesClusterStaysOffline(t *testing.T) {
|
||||
rr := NewRegistry(nil)
|
||||
rr.IngestSnapshot(models.StateSnapshot{
|
||||
KubernetesClusters: []models.KubernetesCluster{{
|
||||
ID: "cluster-silent",
|
||||
AgentID: "k8s-agent-silent",
|
||||
Name: "silent",
|
||||
Status: "offline",
|
||||
LastSeen: time.Now().UTC().Add(-5 * time.Minute),
|
||||
IntervalSeconds: 30,
|
||||
}},
|
||||
})
|
||||
|
||||
cluster := onlyResourceOfType(t, rr, ResourceTypeK8sCluster)
|
||||
if cluster.Status != StatusOffline {
|
||||
t.Fatalf("cluster status = %q, want offline after its sighting went stale", cluster.Status)
|
||||
}
|
||||
}
|
||||
|
||||
// Pull sources keep the stale-to-warning rule: a stale PVE sighting means
|
||||
// Pulse lost its poller, not that the node reported itself gone.
|
||||
func TestStaleProxmoxNodeStaysWarning(t *testing.T) {
|
||||
rr := NewRegistry(nil)
|
||||
rr.IngestSnapshot(models.StateSnapshot{
|
||||
Nodes: []models.Node{{
|
||||
ID: "homelab-pve9",
|
||||
Name: "pve9",
|
||||
Instance: "homelab",
|
||||
ClusterName: "homelab",
|
||||
Status: "online",
|
||||
LastSeen: time.Now().UTC().Add(-5 * time.Minute),
|
||||
}},
|
||||
})
|
||||
|
||||
node := onlyResourceOfType(t, rr, ResourceTypeAgent)
|
||||
if node.Status != StatusWarning {
|
||||
t.Fatalf("node status = %q, want warning for a stale PVE sighting", node.Status)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCloneDockerDataPreservesContainerRuntimeMetadata(t *testing.T) {
|
||||
startedAt := time.Date(2026, 6, 11, 13, 15, 30, 0, time.UTC)
|
||||
finishedAt := startedAt.Add(45 * time.Minute)
|
||||
|
||||
@@ -295,6 +295,12 @@ type SourceStatus struct {
|
||||
LastSeen time.Time `json:"lastSeen"`
|
||||
Error string `json:"error,omitempty"`
|
||||
ExpectedUpdateIntervalSeconds int64 `json:"expectedUpdateIntervalSeconds,omitempty"` // Collector-authored cadence, zero uses source default
|
||||
|
||||
// leaseExpired records that this source delivered the machine, Docker
|
||||
// host or cluster it reports for as offline because its reporting lease
|
||||
// ran out. Status aggregation counts that as offline whatever the
|
||||
// sighting's age, while Status keeps describing delivery freshness.
|
||||
leaseExpired bool
|
||||
}
|
||||
|
||||
// ResourceIdentity holds identifiers used for matching.
|
||||
|
||||
Reference in New Issue
Block a user