Show a host whose agent stopped reporting as offline

A host agent past its reporting lease is offline: the monitor marks it
so and raises "Host is offline". The registry's stale pass ranked the
agent's stale sighting above offline, though, so the machine reached
every consumer as a warning. The Machines table showed it amber and
rendered its last CPU, memory, disk and temperature as current readings,
while its own drawer led with the critical offline alert. A Docker host
was offline only until its sighting crossed the 120-second stale
threshold, then flipped to warning, and Kubernetes clusters likewise.

Each sighting now records when a push reporter (host agent, Docker,
Kubernetes) delivered its machine, Docker host or cluster as offline,
and status aggregation counts that as offline whatever the sighting's
age. The sighting itself still reads stale, and a live source still
carries a merged row, so a Proxmox node with a dead agent stays online.

A copy that went through JSON re-derives the marker from its stored
status, and a manual link joining two resources from one source keeps
the fresher sighting, so a retired agent cannot pull a live one down.

Offline rows then take the existing danger gate. The Machines Uptime
and Temp cells and the Docker hosts Temp cell had no gate and now
follow it. The drawer drops a silent agent's uptime and marks its
non-disk Thermals rows "(last known)", as retained disk rows already
are.
This commit is contained in:
rcourtman
2026-10-07 00:14:49 +01:00
parent 723f3165a1
commit c06af69fcb
18 changed files with 831 additions and 29 deletions
@@ -5591,6 +5591,14 @@ Standalone, TrueNAS, and vSphere platform tables and their table-model helpers
must compose those helpers instead of declaring local `metricFallback` /
`finiteMetric` helpers or inlining centered muted dash fallback markup in
metric cells.
A row whose status indicator is `danger` (offline) blanks every reading from
its last report, not only the metric bars. Proxmox node rows already gate
uptime and temperature on online. The Machines table gates its Uptime and
Temperature cells, and the Docker hosts table its temperature cell, on the same
check. A machine another source keeps up stays rendered, since its cell may
carry that source's current reading. Numeric cells keep their own
right-aligned empty dash; the centred `PlatformTableMetricFallback` marker
stays specific to metric-bar cells.
Platform table metric severity coloring is alert-backed, not hardcoded. The
Docker host and container, Proxmox node, Kubernetes cluster and node, TrueNAS
system and app, and vSphere host tables must resolve display thresholds
@@ -2051,6 +2051,11 @@ in `frontend-modern/src/features/storageBackups/diskTemperaturePresentation.ts`.
That module stays separate from `diskPresentation.ts` so the drawer and
Machines chunks take the decision without the Storage presenter, and the rows
add no reads, polling or aggregation work.
A silent host agent's other Thermals rows (pressure, limits, GPU, temperatures,
additional sensors, fans, power) take the same "(last known)" suffix and reason
title. `buildTemperatureRows` gets them through its `lastKnownReason` option,
which the drawer sets from the resource's own `agent.stale`, so the drawer
still reads nothing extra.
The Proxmox node drawer overview should follow the existing guest drawer
compact detail-section pattern and expose node-specific context such as platform,
kernel, hardware, raw capacity, telemetry, and thermal facts rather than
@@ -3169,6 +3169,16 @@ pointer isolation are
pinned by `TestResourceFromHostPreservesCustomSensorMeta` and
`TestCloneResourceIsolatesCustomSensorValues`.
### Expired reporting leases change status only
`SourceStatus` now carries an unexported marker that a push reporter (host
agent, Docker, Kubernetes) delivered its machine, Docker host or cluster as
offline at lease expiry. Status aggregation reads it, so such a row is
`offline` instead of `warning`. The marker is not serialized, persisted or
exposed on the wire, and the public `status`, `lastSeen` and cadence fields
are unchanged. No storage path, recovery read or protection judgement depends
on it. `TestCloneResource_MutateSourceStatusMap` pins that clones keep it.
### Shared system-settings boundary dropped dead auto-update schedule fields
The shared `internal/api` system-settings surface this subsystem consumes
@@ -2270,10 +2270,12 @@ and temperature facts must also compose the frontend-primitives
desktop rows use the shared fixed label track so labels and values remain
visually adjacent instead of spanning the full drawer width.
A standalone host agent past its reporting lease reaches the Machines table as
`warning` with `agent.stale`, not `offline`: by the time the lease expires the
registry has marked its only source stale, which outranks offline. The offline
metric fallback therefore does not blank its cells, and the temperature cell
has no such gate. For SMART disk temperatures the provenance travels on
`offline`, as described in "A push reporter past its lease is offline". The
offline metric fallback blanks its CPU, memory, disk, network, disk I/O, uptime
and temperature cells. The machine drawer drops a silent agent's uptime and
marks its non-disk Thermals rows "(last known)" with the reason as title,
keyed on `agent.stale` because those rows are that agent's own sensors. For SMART disk
temperatures on rows that still render, the provenance travels on
`agent.sensors.smart[].collection`: the Machines temperature cell
(`AgentsMachinesTable.tsx`, `agentMachineTableModel.ts`), its tooltip and the
drawer's Thermals rows (`resourceDetailMappers.ts`) read it through
@@ -6030,6 +6032,34 @@ to that boolean. Identity succession continues to rekey the same state row.
round-trip persistence, attention suppression, and the retired remediation
lock.
### A push reporter past its lease is offline
The host agent, Docker and Kubernetes collectors push their own reports, and
the monitor marks the machine, Docker host or cluster offline only once its
reporting lease runs out. Ingest records that verdict on the source's
sighting, and `aggregateStatus` counts such a sighting as `offline` whatever
its age. The stale pass used to rank the stale sighting above `offline`, so a
silent standalone agent reached every consumer as `warning`. The Machines
table then showed it amber with its last report rendered as current readings,
while its own drawer led with a critical "Host is offline" alert. A Docker host
was offline only until its sighting crossed the 120-second stale threshold,
then flipped to `warning`. With this rule, such rows take the existing
offline treatment: frontend danger gates, the health verdict's `offline`
reason, offline filters and counts.
`SourceStatus.Status` keeps describing delivery freshness, so the sighting
still reads `stale`. Health keeps its `telemetry_stale` reason, and
monitored-system reasons are unchanged. A live source still carries a merged
row: a Proxmox node whose linked agent stopped reporting stays `online` through
the PVE poll. Pull sources (Proxmox, PBS, PMG, TrueNAS, vSphere) and the
resources push collectors report about (guests, containers, pods, disks) keep
the stale-to-warning rule. A JSON copy re-derives the unexported marker from
its stored status in `IngestResources`, and a manual link that joins two
resources reported by one source keeps the fresher sighting. The tests in
`registry_merge_policy_test.go` pin the agent, Docker, Kubernetes,
mixed-source, round-trip and manual-link cases, plus the boundaries that stay
unchanged.
### Canonical object drawer hierarchy
The unified resource drawer is the shared operator-facing detail contract for
@@ -0,0 +1,49 @@
{
"version": 1,
"base_sha": "723f3165a1292268634b6054ab93d3015805d990",
"verified_at": "2026-10-06T23:14:41Z",
"result": "passed",
"changed_paths": [
"frontend-modern/src/components/Infrastructure/ResourceDetailSummary.tsx",
"frontend-modern/src/components/Infrastructure/resourceDetailMappers.ts",
"frontend-modern/src/components/Infrastructure/useResourceDetailDrawerDerivedState.ts",
"frontend-modern/src/features/docker/DockerHostsTable.tsx",
"frontend-modern/src/features/standalone/AgentsMachinesTable.tsx"
],
"content_sha256": {
"frontend-modern/src/components/Infrastructure/ResourceDetailSummary.tsx": "f141220216a968f5953b7646035108f015a9f7de3c5ae5a9e3251ec6e7046fa5",
"frontend-modern/src/components/Infrastructure/resourceDetailMappers.ts": "07b26eafecbd1ec93fb1dd90d4af6629778cef59a2d9e7450426cb88c90bdfdb",
"frontend-modern/src/components/Infrastructure/useResourceDetailDrawerDerivedState.ts": "70955e370c5b2611557a5de8ba4df44b8f6dce47c1b08c611a154d232c786501",
"frontend-modern/src/features/docker/DockerHostsTable.tsx": "450cce7a4723ce7fe7b01ffe9d10c7d6439c34a1a6dda4fc0af6689ceb4741b5",
"frontend-modern/src/features/standalone/AgentsMachinesTable.tsx": "8d63a04f6a475f39de90cb9813b8bd4315525bf63318cfaf958ac195ae07950f"
},
"routes": [
"/standalone/machines",
"/docker"
],
"viewports": [
{
"width": 1440,
"height": 900
},
{
"width": 390,
"height": 844
}
],
"states": [
"Two private mock stacks (8-node estate) built from this tree: one with this commit's registry.go/types.go, one with origin/main's, both with a local-only go build -overlay of internal/mock/generator.go that holds standalone Linux agent host-linux-1 (Apollo-114) and Docker host edge-proxy-01 past their reporting leases the way evaluateHostAgents/evaluateDockerAgents leave them: status offline, LastSeen frozen 14m/9m before the first tick, last readings kept. Mock mode skips the host health sweep, hence the overlay.",
"Before (origin/main backend and frontend): /api/resources served Apollo-114 as status warning with agent.stale true and edge-proxy-01 as warning with health stale. The Machines row showed an amber dot, CPU 32%, Memory 74%, Disk 80%, Net and Disk I/O rates, Up 29d and Temp 45\u00b0C in threshold green beside 'Seen 14m ago', while the drawer led with the critical 'Host Apollo-114 is offline' alert and read 'Observed state Degraded, Uptime 29d 12h'.",
"After (this commit): both resources serve status offline, the agent/docker sightings still read stale, and health reads critical with reasons offline then telemetry_stale. The Apollo-114 row shows a red dot and '\u2014' in CPU, Memory, Disk, Net I/O, Disk I/O, Up and Temp at 1440 and 390; the status filter counts it under Offline. Spectrum-709 (reporting) keeps every value, Temp 47\u00b0C included.",
"Machine drawer at 1440 and 390: Runtime context reads 'Observed state Offline' and 'Last seen N mins ago' with no Uptime row; Platform details > Machine 'Show details' drops the System card's Uptime and the Thermals card reads 'Cpu.package 45\u00b0C (last known)' with title 'Last known reading, not current: host agent stopped reporting'.",
"Docker page at 1440 and 390: edge-proxy-01 shows a red dot, '\u2014' for CPU, Memory, Disk and Temp (the hold seeds a retained 51\u00b0C), and the existing 'No report for 16m' marker in Uptime; Edge Apps 01 keeps its values, Temp 38.5\u00b0C included.",
"Bundle check: WorkloadsSurface unchanged (70.20 kB), Standalone +0.04 kB, Docker +0.01 kB, InfrastructureWorkspace flat versus this tree without the frontend diff; all chunks within budget.",
"Not covered: a real agent going silent on hardware, light/dark differences (no new colour classes), and the drawer Disks card, which still threshold-colours a silent agent's retained filesystem usage (left for a follow-up)."
],
"interactions": [
"Enabled the Network, Disk I/O, Uptime and Temp columns through the Machines column preferences (localStorage init) and read every cell of the offline and reporting rows plus the status dot class at 1440 and 390.",
"Opened the Apollo-114 drawer by row click at both widths, read the Runtime context rows, expanded Platform details and the Machine 'Show details', and read the System and Thermals cards with their value titles.",
"Opened /docker at both widths and read every cell of edge-proxy-01 and Edge Apps 01.",
"Queried /api/resources on both stacks for status, per-source sighting status and health reasons of the held agent and Docker host."
]
}
@@ -217,9 +217,13 @@ export const InlineResourceSummaryTables: Component<ResourceSummaryPresentationP
const now = useRelativeTimeNow();
const sections = (): DetailSection[] => {
const docker = dockerContainerMeta(props.resource);
// An offline resource has no current uptime: the retained value is from
// its last report, and the Last seen row already says when that was.
const uptimeSeconds = docker
? getDockerContainerUptimeSeconds(props.resource)
: props.resource.uptime;
: props.resource.status === 'offline'
? undefined
: props.resource.uptime;
const identityRows = compactDetailRows([
...props.drawer.primaryIdentityRows().map((row) => makeDetailRow(row.label, row.value)),
props.showPlatformId ? makeDetailRow('Platform ID', props.resource.platformId) : null,
@@ -0,0 +1,64 @@
import { cleanup, render, screen, within } from '@solidjs/testing-library';
import { afterEach, describe, expect, it } from 'vitest';
import type { Resource } from '@/types/resource';
import { InlineResourceSummaryTables } from '../ResourceDetailSummary';
import type { UseResourceDetailDrawerStateResult } from '../useResourceDetailDrawerState';
const drawerState = {
primaryIdentityRows: () => [],
identityIpValues: () => [],
identityAliasValues: () => [],
aliasPreviewValues: () => [],
hasAliasOverflow: () => false,
identityCardHasRichData: () => true,
sourceSummary: () => null,
lastSeen: () => '14 mins ago',
lastSeenAbsolute: () => '',
} as unknown as UseResourceDetailDrawerStateResult;
const machine = (status: Resource['status']): Resource => ({
id: 'agent-apollo',
name: 'Apollo-114',
displayName: 'Apollo-114',
platformId: 'host-linux-1',
platformType: 'agent',
sourceType: 'agent',
type: 'agent',
status,
uptime: 2_500_000,
lastSeen: Date.parse('2026-10-06T22:00:00Z'),
});
afterEach(cleanup);
describe('resource drawer runtime context for an offline resource', () => {
it('drops the uptime of an offline machine and keeps when it was last seen', () => {
render(() => (
<InlineResourceSummaryTables
resource={machine('offline')}
drawer={drawerState}
showPlatformId={false}
/>
));
const section = screen.getByTestId('resource-runtime-context-section');
expect(within(section).getByText('offline')).toBeInTheDocument();
expect(within(section).queryByText('Uptime')).toBeNull();
expect(within(section).getByText('Last seen')).toBeInTheDocument();
expect(within(section).getByText('14 mins ago')).toBeInTheDocument();
});
it('keeps the uptime of a machine that is up', () => {
render(() => (
<InlineResourceSummaryTables
resource={machine('online')}
drawer={drawerState}
showPlatformId={false}
/>
));
const section = screen.getByTestId('resource-runtime-context-section');
expect(within(section).getByText('Uptime')).toBeInTheDocument();
});
});
@@ -2,6 +2,7 @@ import { describe, expect, it } from 'vitest';
import {
buildCustomSensorRows,
buildTemperatureRows,
HOST_AGENT_STOPPED_REPORTING_REASON,
formatInteger,
formatSensorName,
formatSourceType,
@@ -248,6 +249,37 @@ describe('resourceDetailMappers', () => {
]);
});
it("marks a silent agent's sensor readings last known and leaves disks to their own state", () => {
const rows = buildTemperatureRows(
{
thermalState: { pressure: 'serious' },
temperatureCelsius: { 'cpu.package': 45 },
fanRpm: { chassis_fan: 1200 },
smart: [
{
device: 'sda',
temperature: 38,
collection: { temperature: { state: 'available', source: 'smartctl' } },
},
],
},
{ lastKnownReason: HOST_AGENT_STOPPED_REPORTING_REASON },
);
const lastKnown = 'Last known reading, not current: host agent stopped reporting';
expect(rows).toEqual([
{ label: 'Thermal pressure', value: 'Serious (last known)', valueTitle: lastKnown },
{ label: 'Cpu.package', value: '45°C (last known)', valueTitle: lastKnown },
{ label: 'Chassis Fan', value: '1,200 RPM (last known)', valueTitle: lastKnown },
// A disk row follows its own collection state, which the backend
// withdraws when the lease expires.
{ label: 'Disk sda', value: '38°C', valueTitle: '38.0°C' },
]);
expect(buildTemperatureRows({ temperatureCelsius: { 'cpu.package': 45 } })).toEqual([
{ label: 'Cpu.package', value: '45°C', valueTitle: '45.0°C' },
]);
});
it('lists a SMART row only for a positive reading in any state, never a standby disk', () => {
const states: (PhysicalDiskCollectionStatus | undefined)[] = [
undefined,
@@ -442,6 +474,15 @@ describe('resourceDetailMappers', () => {
expect(agent?.id).not.toBe('resource:host:hash-1');
});
it('reports no uptime for a silent agent, whose last report is not how long it has been up', () => {
const base = createHybridHostResource();
const reporting = { ...base, uptime: 2_500_000 } as Resource;
expect(toAgentFromResource(reporting)?.uptimeSeconds).toBe(2_500_000);
const silent = { ...reporting, agent: { stale: true } } as Resource;
expect(toAgentFromResource(silent)?.uptimeSeconds).toBe(0);
});
it('preserves the local infrastructure display name for governed resources', () => {
const agent = toAgentFromResource({
...createHybridHostResource(),
@@ -317,7 +317,9 @@ export const toAgentFromResource = (
sensors: agent.sensors,
raid: agent.raid,
status: resource.status,
uptimeSeconds: agent.uptimeSeconds ?? resource.uptime ?? 0,
// A silent agent's uptime is from its last report, not how long the
// machine has been up.
uptimeSeconds: resource.agent?.stale ? 0 : (agent.uptimeSeconds ?? resource.uptime ?? 0),
lastSeen: resource.lastSeen,
agentVersion: agent.agentVersion,
commandsEnabled: agent.commandsEnabled,
@@ -400,7 +402,23 @@ const buildTypedGPUTemperatureKeys = (gpus?: HostSensorSummary['gpu']) => {
return keys;
};
export const buildTemperatureRows = (sensors?: HostSensorSummary) => {
// Matches the backend's reason for readings a host agent past its reporting
// lease no longer collects (models.HostAgentStoppedReportingReason).
export const HOST_AGENT_STOPPED_REPORTING_REASON = 'host agent stopped reporting';
export type TemperatureRowsOptions = {
/**
* Why the machine's own sensor readings are retained rather than current,
* such as a host agent past its reporting lease. Disk rows keep their own
* collection state instead.
*/
lastKnownReason?: string;
};
export const buildTemperatureRows = (
sensors?: HostSensorSummary,
options: TemperatureRowsOptions = {},
) => {
const rows: { label: string; value: string; valueTitle?: string }[] = [];
const thermalState = sensors?.thermalState;
const typedGPUTemperatureKeys = buildTypedGPUTemperatureKeys(sensors?.gpu);
@@ -500,6 +518,17 @@ export const buildTemperatureRows = (sensors?: HostSensorSummary) => {
});
}
// This card has no other cue that a value is not current, so a retained
// sensor reading says so in its value, as a retained disk reading does.
const lastKnownReason = options.lastKnownReason?.trim();
if (lastKnownReason) {
const title = `Last known reading, not current: ${lastKnownReason}`;
for (const row of rows) {
row.value = `${row.value} (last known)`;
row.valueTitle = title;
}
}
const smart = sensors?.smart;
if (smart) {
smart
@@ -35,6 +35,7 @@ import type { ResourceIntelligence } from '@/types/aiIntelligence';
import {
buildCustomSensorRows,
buildTemperatureRows,
HOST_AGENT_STOPPED_REPORTING_REASON,
toAgentFromResource,
toNodeFromProxmox,
type AgentPlatformData,
@@ -201,7 +202,13 @@ export const useResourceDetailDrawerDerivedState = (
[],
);
const agentInfo = createMemo(() => toAgentFromResource(resource(), agentMeta()));
const temperatureRows = createMemo(() => buildTemperatureRows(agentInfo()?.sensors));
// A silent agent's sensors are its last report, even on a row another
// source keeps online.
const temperatureRows = createMemo(() =>
buildTemperatureRows(agentInfo()?.sensors, {
lastKnownReason: resource().agent?.stale ? HOST_AGENT_STOPPED_REPORTING_REASON : undefined,
}),
);
const customSensorRows = createMemo(() => buildCustomSensorRows(agentInfo()?.sensors));
const dockerHostData = createMemo(() => platformData()?.docker as DockerPlatformData | undefined);
@@ -517,7 +517,11 @@ export const DockerHostsTable: Component<{
class={`${getPlatformTableCellClassForKind('numeric-value')} platform-table-narrow-hidden hidden text-base-content min-[360px]:table-cell md:table-cell`}
>
<PlatformTableTemperatureValue
value={host.temperature ?? docker()?.temperature}
value={
canRenderMetrics()
? (host.temperature ?? docker()?.temperature)
: undefined
}
/>
</TableCell>
<Show when={showSwarmColumn()}>
@@ -217,6 +217,58 @@ describe('DockerHostsTable', () => {
);
});
it("drops an offline host's retained temperature with its other readings", () => {
// A Docker host past its reporting lease is offline, and its last
// temperature is no more current than its last CPU, memory and disk.
const { container } = render(() => (
<DockerHostsTable
resources={[
makeDockerHost({
status: 'offline',
temperature: 51,
health: {
verdict: 'critical',
reasons: [
{ code: 'offline', detail: '' },
{ code: 'telemetry_stale', detail: '9m' },
],
},
}),
makeDockerHost({
id: 'agent:docker-02',
name: 'docker-02',
displayName: 'docker-02',
temperature: 51,
}),
]}
emptyIcon={<span />}
emptyTitle="No Docker hosts"
emptyDescription="No hosts"
showToolbar={false}
/>
));
const headers = Array.from(container.querySelectorAll('thead th')).map((th) =>
(th.textContent ?? '').trim(),
);
const tempIndex = headers.findIndex((header) => header.startsWith('Temp'));
expect(tempIndex).toBeGreaterThan(-1);
const temp = (id: string) =>
(
container.querySelectorAll(`[data-docker-host-row="${id}"] td`)[tempIndex]?.textContent ??
''
).trim();
expect(temp('agent:docker-01')).toBe('—');
expect(temp('agent:docker-02')).toBe('51.0°C');
const offlineRow = container.querySelector('[data-docker-host-row="agent:docker-01"]')!;
expect(within(offlineRow as HTMLElement).queryAllByTestId('stacked-memory-bar')).toHaveLength(
0,
);
expect(offlineRow.querySelector('[data-docker-host-stale] .sr-only')).toHaveTextContent(
'No report for 9m',
);
});
it('tints a host row for the Docker alerts its drawer lists, not by display name', () => {
// Docker alerts are keyed on the Docker host id and carry the hostname
// as their node; neither is the row's id or its display name.
@@ -1806,7 +1806,9 @@ export const AgentsMachinesTable: Component<{
class={`${getPlatformTableCellClassForKind('numeric-value')} ${machineColumnWidthClass('uptime')} text-base-content`}
>
{formatPlatformTableUptimeValue(
machine.uptime ?? machine.agent?.uptimeSeconds,
canRenderMetrics()
? (machine.uptime ?? machine.agent?.uptimeSeconds)
: undefined,
)}
</TableCell>
</Show>
@@ -1814,15 +1816,20 @@ export const AgentsMachinesTable: Component<{
<TableCell
class={`${getPlatformTableCellClassForKind('numeric-value')} ${machineColumnWidthClass('temp')} text-base-content`}
>
<AgentMachineTemperatureCell
celsius={temperature()}
lastKnown={temperatureLastKnown()}
metric={temperatureMetric()}
sections={temperatureSections()}
thresholds={temperatureThresholds()}
title={temperatureTitle()}
thermalPressure={thermalPressure()}
/>
<Show
when={canRenderMetrics()}
fallback={<span class="text-muted">—</span>}
>
<AgentMachineTemperatureCell
celsius={temperature()}
lastKnown={temperatureLastKnown()}
metric={temperatureMetric()}
sections={temperatureSections()}
thresholds={temperatureThresholds()}
title={temperatureTitle()}
thermalPressure={thermalPressure()}
/>
</Show>
</TableCell>
</Show>
<Show when={columnVisibility.isColumnVisible('lastSeen')}>
@@ -1160,14 +1160,15 @@ describe('AgentsMachinesTable', () => {
expect(screen.getByText('1400 RPM')).toBeInTheDocument();
});
it('shows a silent agent disk temperature as last known, never as a current reading', async () => {
// A host agent past its reporting lease is stale and keeps its last SMART
// temperatures with a non-available collection state.
it('shows a retained disk temperature as last known, never as a current reading', async () => {
// A reporting agent whose SMART probe returned no temperature keeps the
// last reading with a non-available collection state. (A host agent past
// its lease is offline, and its row shows no temperature at all.)
const stoppedReporting = {
temperature: {
state: 'unavailable' as const,
source: 'smartctl',
reason: 'host agent stopped reporting',
reason: 'SMART probe returned no usable temperature data',
},
};
const { container } = render(() => (
@@ -1176,9 +1177,7 @@ describe('AgentsMachinesTable', () => {
resource({
id: 'silent-nas',
name: 'Silent NAS',
status: 'warning',
agent: {
stale: true,
sensors: {
smart: [
{ device: '/dev/sda', model: 'Archive HDD', temperature: 71 },
@@ -1249,6 +1248,92 @@ describe('AgentsMachinesTable', () => {
expect(within(mixed).queryByText('69°C')).toBeNull();
});
it("blanks an offline machine's retained readings, temperature and uptime included", async () => {
// A host agent past its reporting lease is offline: CPU, memory, disk and
// the sensors, uptime and disks it last reported are not current.
const stoppedReporting = {
temperature: {
state: 'unavailable' as const,
source: 'smartctl',
reason: 'host agent stopped reporting',
},
};
const lastReport: Partial<Resource> = {
cpu: { current: 32 },
memory: { total: 100, used: 73, free: 27, current: 73 },
disk: { total: 100, used: 80, free: 20, current: 80 },
};
const sensors = {
temperatureCelsius: { 'cpu.package': 45 },
thermalState: { pressure: 'serious' },
smart: [
{
device: '/dev/sda',
model: 'Archive HDD',
temperature: 41,
collection: stoppedReporting,
},
],
};
const { container } = render(() => (
<AgentsMachinesTable
resources={[
resource({
id: 'silent-host',
name: 'Silent Host',
status: 'offline',
uptime: 2_500_000,
...lastReport,
agent: { stale: true, uptimeSeconds: 2_500_000, sensors },
}),
resource({
id: 'live-host',
name: 'Live Host',
uptime: 2_500_000,
...lastReport,
agent: {
uptimeSeconds: 2_500_000,
sensors: { temperatureCelsius: { 'cpu.package': 45 } },
},
}),
]}
emptyIcon={emptyIcon}
emptyTitle="No machines"
emptyDescription="Install Pulse Agent."
/>
));
await openMachineColumnPicker();
await fireEvent.click(screen.getByLabelText('Temp'));
await fireEvent.click(screen.getByLabelText('Uptime'));
const cells = (id: string) => {
const row = container.querySelector(`[data-agents-machine-row="${id}"]`) as HTMLElement;
const headers = Array.from(container.querySelectorAll('thead th')).map((th) =>
(th.textContent ?? '').replace(/[▲▼]/g, '').trim(),
);
const values = Array.from(row.querySelectorAll('td')).map((td) =>
(td.textContent ?? '').trim(),
);
return (label: string) => values[headers.indexOf(label)];
};
const silent = cells('silent-host');
expect(silent('CPU')).toBe('—');
expect(silent('Up')).toBe('—');
expect(silent('Temp')).toBe('—');
expect(
container.querySelector(
'[data-agents-machine-row="silent-host"] [data-agent-machine-temperature-trigger="true"]',
),
).toBeNull();
const live = cells('live-host');
expect(live('CPU')).not.toBe('—');
expect(live('Up')).toBe('28d');
expect(live('Temp')).toBe('45°C');
});
it('moves a machine temperature between current and last known as its disk reports change', async () => {
// Fresh objects per update, like parsed API payloads: the store reconciles
// into the objects it was given, so a reused fixture would be rewritten.
+6
View File
@@ -221,12 +221,18 @@ func TestCloneResource_MutateSourceStatusMap(t *testing.T) {
ID: "r-1",
SourceStatus: map[DataSource]SourceStatus{
SourceProxmox: {Status: "online", ExpectedUpdateIntervalSeconds: 300},
SourceAgent: {Status: "stale", leaseExpired: true},
},
}
cloned := cloneResource(original)
if cloned.SourceStatus[SourceProxmox].ExpectedUpdateIntervalSeconds != 300 {
t.Fatal("clone lost collector cadence")
}
// A clone that dropped the lease verdict would aggregate a silent agent
// back to warning.
if !cloned.SourceStatus[SourceAgent].leaseExpired {
t.Fatal("clone lost the agent's expired reporting lease")
}
cloned.SourceStatus[SourceDocker] = SourceStatus{Status: "online"}
if _, exists := original.SourceStatus[SourceDocker]; exists {
+51 -4
View File
@@ -809,6 +809,14 @@ func (rr *ResourceRegistry) ingestResources(resources []Resource, thresholds map
}
}
}
// The lease marker is unexported, so a persisted or serialized copy
// arrives without it. Its stored status still carries the verdict.
for source, status := range resource.SourceStatus {
if !status.leaseExpired && reportingLeaseExpired(source, *resource) {
status.leaseExpired = true
resource.SourceStatus[source] = status
}
}
rr.mu.Lock()
rr.canonicalIdentityIndex = nil
@@ -1756,6 +1764,29 @@ func sourceSightingStatus(lastSeen time.Time) string {
return "online"
}
// reportingLeaseExpired reports whether a source delivered the machine,
// Docker host or Kubernetes cluster it reports for as offline. Those
// collectors push their own reports, and Pulse marks the reporter offline
// only once its reporting lease runs out (evaluateHostAgents,
// evaluateDockerAgents, evaluateKubernetesAgents). The registry's stale
// threshold is a different clock: it can mark the sighting stale before
// the lease ends, or leave it fresh after, so it must not decide the status.
// Resources those sources describe (guests, containers, pods, disks) are
// not lease holders and keep the stale-to-warning rule.
func reportingLeaseExpired(source DataSource, resource Resource) bool {
if resource.Status != StatusOffline {
return false
}
switch source {
case SourceAgent, SourceDocker:
return resource.Type == ResourceTypeAgent
case SourceK8s:
return resource.Type == ResourceTypeK8sCluster
default:
return false
}
}
func (rr *ResourceRegistry) markStaleLocked(now time.Time, thresholds map[DataSource]time.Duration) {
thresholds = effectiveStaleThresholds(thresholds)
@@ -2848,6 +2879,7 @@ func (rr *ResourceRegistry) ingestRecord(source DataSource, sourceID string, res
sighting := resource.SourceStatus[source]
sighting.Status = sourceSightingStatus(resource.LastSeen)
sighting.LastSeen = resource.LastSeen
sighting.leaseExpired = reportingLeaseExpired(source, resource)
resource.SourceStatus = map[DataSource]SourceStatus{source: sighting}
resource.parentBySource = make(map[DataSource]string)
rr.setSourceParent(&resource, source, resource.ParentID)
@@ -3787,6 +3819,7 @@ func (rr *ResourceRegistry) mergeInto(existing *Resource, incoming Resource, sou
sighting := incoming.SourceStatus[source]
sighting.Status = sourceSightingStatus(incoming.LastSeen)
sighting.LastSeen = incoming.LastSeen
sighting.leaseExpired = reportingLeaseExpired(source, incoming)
existing.SourceStatus[source] = sighting
if incoming.LastSeen.After(existing.LastSeen) {
@@ -4425,6 +4458,12 @@ func (rr *ResourceRegistry) mergeResourceData(primary *Resource, other *Resource
primary.SourceStatus = make(map[DataSource]SourceStatus)
}
for source, status := range other.SourceStatus {
// Linked resources can share a source, such as an agent reinstalled
// under a new id. Keep the fresher sighting so a stale or expired one
// cannot override a reporter that is still live.
if current, ok := primary.SourceStatus[source]; ok && current.LastSeen.After(status.LastSeen) {
continue
}
primary.SourceStatus[source] = status
}
if other.LastSeen.After(primary.LastSeen) {
@@ -5793,14 +5832,22 @@ func aggregateStatus(resource *Resource) ResourceStatus {
best := StatusUnknown
bestScore := 0
for _, status := range resource.SourceStatus {
score := statusPriority[strings.ToLower(status.Status)]
state := strings.ToLower(strings.TrimSpace(status.Status))
// A reporter past its lease is offline, not merely late. Ranking its
// stale sighting above offline would show a silent machine as a
// warning while its retained readings render as current.
if status.leaseExpired {
state = "offline"
}
score := statusPriority[state]
if score > bestScore {
bestScore = score
if status.Status == "online" {
switch state {
case "online":
best = StatusOnline
} else if status.Status == "stale" {
case "stale":
best = StatusWarning
} else if status.Status == "offline" {
case "offline":
best = StatusOffline
}
}
@@ -1,6 +1,7 @@
package unifiedresources
import (
"encoding/json"
"strings"
"testing"
"time"
@@ -131,6 +132,353 @@ func TestHostnameIPDoesNotAutoMerge(t *testing.T) {
}
}
func onlyResourceOfType(t *testing.T, rr *ResourceRegistry, resourceType ResourceType) Resource {
t.Helper()
resources := rr.ListByType(resourceType)
if len(resources) != 1 {
t.Fatalf("expected one %s resource, got %d: %+v", resourceType, len(resources), resources)
}
return resources[0]
}
// A host agent past its reporting lease is offline. Its source sighting is
// stale too, and the stale pass used to rank that above offline, so the
// machine reached the frontend as a warning with its last report rendered
// as current readings.
func TestLeaseExpiredHostAgentStaysOffline(t *testing.T) {
lastReport := time.Now().UTC().Add(-14 * time.Minute)
rr := NewRegistry(nil)
rr.IngestSnapshot(models.StateSnapshot{
Hosts: []models.Host{{
ID: "host-silent",
MachineID: "machine-silent",
Hostname: "silent",
Status: "offline",
LastSeen: lastReport,
IntervalSeconds: 30,
CPUUsage: 32,
}},
})
resource := onlyResourceOfType(t, rr, ResourceTypeAgent)
if resource.Status != StatusOffline {
t.Fatalf("status = %q, want offline for an agent past its lease", resource.Status)
}
if resource.Agent == nil || !resource.Agent.Stale {
t.Fatalf("expected the agent facet to stay flagged stale, got %+v", resource.Agent)
}
// Status describes the resource; the sighting keeps describing delivery
// freshness, which health and monitored-system reasons read.
if got := resource.SourceStatus[SourceAgent].Status; got != "stale" {
t.Fatalf("agent sighting = %q, want stale", got)
}
// The monitor adapter runs the stale pass again after record sources.
rr.MarkStale(time.Now().UTC(), nil)
if got, _ := rr.Get(resource.ID); got.Status != StatusOffline {
t.Fatalf("second stale pass changed status to %q, want offline", got.Status)
}
health := EvaluateResourceHealth(resource, nil, time.Now().UTC())
if health.Verdict != HealthCritical || len(health.Reasons) < 2 ||
health.Reasons[0].Code != "offline" || health.Reasons[1].Code != "telemetry_stale" {
t.Fatalf("health = %+v, want critical offline with the stale reason after it", health)
}
}
// An agent that is late but still inside its lease keeps the warning: the
// monitor has not decided it is gone, so its last readings are only stale.
func TestLateHostAgentInsideLeaseStaysWarning(t *testing.T) {
rr := NewRegistry(nil)
rr.IngestSnapshot(models.StateSnapshot{
Hosts: []models.Host{{
ID: "host-late",
MachineID: "machine-late",
Hostname: "late",
Status: "online",
LastSeen: time.Now().UTC().Add(-90 * time.Second),
IntervalSeconds: 30,
}},
})
resource := onlyResourceOfType(t, rr, ResourceTypeAgent)
if resource.Status != StatusWarning {
t.Fatalf("status = %q, want warning for a late agent inside its lease", resource.Status)
}
}
// A Proxmox node whose linked agent stopped reporting stays online through
// the PVE poll (issue #1515). Only the agent sighting carries the lease.
func TestLeaseExpiredAgentOnLiveProxmoxNodeStaysOnline(t *testing.T) {
now := time.Now().UTC()
const machineID = "machine-pve1"
rr := NewRegistry(nil)
rr.IngestSnapshot(models.StateSnapshot{
Nodes: []models.Node{{
ID: "homelab-pve1",
Name: "pve1",
Instance: "homelab",
ClusterName: "homelab",
Status: "online",
LastSeen: now,
}},
Hosts: []models.Host{{
ID: machineID,
MachineID: machineID,
Hostname: "pve1",
LinkedNodeID: "homelab-pve1",
Status: "offline",
LastSeen: now.Add(-14 * time.Minute),
IntervalSeconds: 30,
}},
})
resource := onlyResourceOfType(t, rr, ResourceTypeAgent)
if _, ok := resource.SourceStatus[SourceProxmox]; !ok {
t.Fatalf("expected the node and its agent to share one resource, got sources %+v", resource.SourceStatus)
}
if resource.Status != StatusOnline {
t.Fatalf("status = %q, want online through the live PVE poll", resource.Status)
}
if resource.Agent == nil || !resource.Agent.Stale {
t.Fatalf("expected the dead agent to stay flagged stale, got %+v", resource.Agent)
}
}
// Docker hosts hold a shorter lease than the registry's Docker stale
// threshold, so a silent Docker host was offline for a while and then
// flipped to warning once its sighting went stale.
func TestLeaseExpiredDockerHostStaysOfflineAfterSightingGoesStale(t *testing.T) {
lastReport := time.Now().UTC().Add(-5 * time.Minute)
rr := NewRegistry(nil)
rr.IngestSnapshot(models.StateSnapshot{
DockerHosts: []models.DockerHost{{
ID: "docker-silent",
AgentID: "docker-agent-silent",
Hostname: "docker-silent",
MachineID: "machine-docker-silent",
Status: "offline",
LastSeen: lastReport,
IntervalSeconds: 30,
Containers: []models.DockerContainer{{
ID: "container-web",
Name: "web",
State: "running",
}},
}},
})
host := onlyResourceOfType(t, rr, ResourceTypeAgent)
if host.Status != StatusOffline {
t.Fatalf("docker host status = %q, want offline after its sighting went stale", host.Status)
}
if got := host.SourceStatus[SourceDocker].Status; got != "stale" {
t.Fatalf("docker sighting = %q, want stale", got)
}
// Containers are what the host reports about, not lease holders: a
// running container on a silent host is stale, not known to be down.
container := onlyResourceOfType(t, rr, ResourceTypeAppContainer)
if container.Status != StatusWarning {
t.Fatalf("container status = %q, want warning on a silent host", container.Status)
}
}
// A machine reporting through both the host agent and Docker is offline once
// both leases expire. The Docker lease ends first, so its sighting can still
// be fresh when the agent's goes stale.
func TestMachineWithExpiredAgentAndDockerLeasesIsOffline(t *testing.T) {
now := time.Now().UTC()
const machineID = "machine-both"
rr := NewRegistry(nil)
rr.IngestSnapshot(models.StateSnapshot{
Hosts: []models.Host{{
ID: "host-both",
MachineID: machineID,
Hostname: "both",
Status: "offline",
LastSeen: now.Add(-3 * time.Minute),
IntervalSeconds: 30,
}},
DockerHosts: []models.DockerHost{{
ID: "docker-both",
AgentID: "host-both",
Hostname: "both",
MachineID: machineID,
Status: "offline",
LastSeen: now.Add(-90 * time.Second),
IntervalSeconds: 10,
}},
})
resource := onlyResourceOfType(t, rr, ResourceTypeAgent)
if _, ok := resource.SourceStatus[SourceDocker]; !ok {
t.Fatalf("expected agent and Docker to share one resource, got sources %+v", resource.SourceStatus)
}
if got := resource.SourceStatus[SourceDocker].Status; got != "online" {
t.Fatalf("docker sighting = %q, want it still inside the stale threshold", got)
}
if resource.Status != StatusOffline {
t.Fatalf("status = %q, want offline once both leases expired", resource.Status)
}
}
// A live reporter still carries the machine when the other lease expired.
func TestMachineWithLiveDockerReporterStaysOnline(t *testing.T) {
now := time.Now().UTC()
const machineID = "machine-split"
rr := NewRegistry(nil)
rr.IngestSnapshot(models.StateSnapshot{
Hosts: []models.Host{{
ID: "host-split",
MachineID: machineID,
Hostname: "split",
Status: "offline",
LastSeen: now.Add(-14 * time.Minute),
IntervalSeconds: 30,
}},
DockerHosts: []models.DockerHost{{
ID: "docker-split",
AgentID: "docker-agent-split",
Hostname: "split",
MachineID: machineID,
Status: "online",
LastSeen: now,
IntervalSeconds: 10,
}},
})
resource := onlyResourceOfType(t, rr, ResourceTypeAgent)
if resource.Status != StatusOnline {
t.Fatalf("status = %q, want online through the live Docker reporter", resource.Status)
}
}
// The lease marker is unexported, so a resource that went through JSON (a
// persisted or remote copy) must recover it from its stored status.
func TestLeaseExpiredHostAgentStaysOfflineAfterJSONRoundTrip(t *testing.T) {
source := NewRegistry(nil)
source.IngestSnapshot(models.StateSnapshot{
Hosts: []models.Host{{
ID: "host-silent",
MachineID: "machine-silent",
Hostname: "silent",
Status: "offline",
LastSeen: time.Now().UTC().Add(-14 * time.Minute),
IntervalSeconds: 30,
}},
})
payload, err := json.Marshal(source.List())
if err != nil {
t.Fatalf("marshal resources: %v", err)
}
var copied []Resource
if err := json.Unmarshal(payload, &copied); err != nil {
t.Fatalf("unmarshal resources: %v", err)
}
rr := NewRegistry(nil)
rr.IngestResources(copied)
resource := onlyResourceOfType(t, rr, ResourceTypeAgent)
if resource.Status != StatusOffline {
t.Fatalf("status = %q, want offline after a JSON round trip", resource.Status)
}
}
// A manual link can join two resources reported by the same source, such as
// an agent reinstalled under a new id. The expired sighting of the old one
// must not replace the live sighting of the new one.
func TestManualLinkKeepsTheLiveSightingOfASharedSource(t *testing.T) {
now := time.Now().UTC()
store := NewMemoryStore()
if err := store.AddLink(ResourceLink{
ResourceA: "agent-reinstalled",
ResourceB: "agent-retired",
PrimaryID: "agent-reinstalled",
}); err != nil {
t.Fatalf("add link: %v", err)
}
rr := NewRegistry(store)
rr.IngestResources([]Resource{
{
ID: "agent-reinstalled",
Type: ResourceTypeAgent,
Name: "tower",
Status: StatusOnline,
LastSeen: now,
Sources: []DataSource{SourceAgent},
SourceStatus: map[DataSource]SourceStatus{
SourceAgent: {Status: "online", LastSeen: now},
},
Agent: &AgentData{AgentID: "host-new", Hostname: "tower"},
},
{
ID: "agent-retired",
Type: ResourceTypeAgent,
Name: "tower",
Status: StatusOffline,
LastSeen: now.Add(-14 * time.Minute),
Sources: []DataSource{SourceAgent},
SourceStatus: map[DataSource]SourceStatus{
SourceAgent: {Status: "stale", LastSeen: now.Add(-14 * time.Minute)},
},
Agent: &AgentData{AgentID: "host-old", Hostname: "tower", Stale: true},
},
})
resource, ok := rr.Get("agent-reinstalled")
if !ok {
t.Fatal("linked resource missing")
}
if resource.Status != StatusOnline {
t.Fatalf("status = %q, want online through the live reinstalled agent", resource.Status)
}
if got := resource.SourceStatus[SourceAgent]; !got.LastSeen.Equal(now) {
t.Fatalf("agent sighting = %+v, want the live sighting", got)
}
}
func TestLeaseExpiredKubernetesClusterStaysOffline(t *testing.T) {
rr := NewRegistry(nil)
rr.IngestSnapshot(models.StateSnapshot{
KubernetesClusters: []models.KubernetesCluster{{
ID: "cluster-silent",
AgentID: "k8s-agent-silent",
Name: "silent",
Status: "offline",
LastSeen: time.Now().UTC().Add(-5 * time.Minute),
IntervalSeconds: 30,
}},
})
cluster := onlyResourceOfType(t, rr, ResourceTypeK8sCluster)
if cluster.Status != StatusOffline {
t.Fatalf("cluster status = %q, want offline after its sighting went stale", cluster.Status)
}
}
// Pull sources keep the stale-to-warning rule: a stale PVE sighting means
// Pulse lost its poller, not that the node reported itself gone.
func TestStaleProxmoxNodeStaysWarning(t *testing.T) {
rr := NewRegistry(nil)
rr.IngestSnapshot(models.StateSnapshot{
Nodes: []models.Node{{
ID: "homelab-pve9",
Name: "pve9",
Instance: "homelab",
ClusterName: "homelab",
Status: "online",
LastSeen: time.Now().UTC().Add(-5 * time.Minute),
}},
})
node := onlyResourceOfType(t, rr, ResourceTypeAgent)
if node.Status != StatusWarning {
t.Fatalf("node status = %q, want warning for a stale PVE sighting", node.Status)
}
}
func TestCloneDockerDataPreservesContainerRuntimeMetadata(t *testing.T) {
startedAt := time.Date(2026, 6, 11, 13, 15, 30, 0, time.UTC)
finishedAt := startedAt.Add(45 * time.Minute)
+6
View File
@@ -295,6 +295,12 @@ type SourceStatus struct {
LastSeen time.Time `json:"lastSeen"`
Error string `json:"error,omitempty"`
ExpectedUpdateIntervalSeconds int64 `json:"expectedUpdateIntervalSeconds,omitempty"` // Collector-authored cadence, zero uses source default
// leaseExpired records that this source delivered the machine, Docker
// host or cluster it reports for as offline because its reporting lease
// ran out. Status aggregation counts that as offline whatever the
// sighting's age, while Status keeps describing delivery freshness.
leaseExpired bool
}
// ResourceIdentity holds identifiers used for matching.