mirror of
https://github.com/rcourtman/Pulse.git
synced 2026-10-24 12:42:49 +00:00
396bb5d517f14466795a2ccd2ff438fdba0ab077
6006 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 rcourtman
|
396bb5d517 |
Keep a disk heat finding open without a current reading
Patrol verifies a disk-high finding as recovered once the disk's temperature cools to its alert clear value. A disk in standby, or one whose host agent stopped reporting, has no current reading, and a missing reading counted as cooled. A finding raised for a SATA disk at 56C would close as soon as the disk spun down, while its temperature alert stayed open. When a disk has no current reading and its last known reading is still at or above the clear value, verification now reports the recovery as unknown instead of proven. A disk last seen cool still recovers. |
||
|
rcourtman
|
35bf55af23 |
Colour report disk temperatures by the alert thresholds
The PDF performance report's disk table coloured a temperature amber from 50C and red from 60C for every disk type, the last fixed disk heat line after disk heat moved to the alert disk temperature policy. An NVMe at 63C, well under its 70C alert trigger, printed red. A user's raised trigger changed nothing. Each reported disk now carries the tenant's alert thresholds for its type (the clear value and the trigger, from alerts.Manager.DiskTemperatureThreshold). The PDF colours the reading amber from the clear value and red from the trigger, as the Physical Disks Temp column does, and leaves it plain when disk temperature alerting is off. |
||
|
rcourtman
|
3fd5411ecc |
Judge disk heat by the alert disk temperature thresholds
Pulse decided whether a physical disk was too hot in three places with three thresholds. Disk risk added temperature_high at a flat 60/70C for every disk type and drove the disk status and the Physical Disks Health verdict. Patrol flagged anything over 55C and the AI chat context listed disks from 50C. Alerts and the Temp column used the per-type thresholds users tune in Alerts (NVMe 70, SAS 65, SATA 55 by default). An NVMe at 63C read Running Hot beside a green Temp cell. A SATA disk at 56C read Healthy beside a red one. A user who raised the NVMe trigger to 75 still got a hot verdict at 70. The alert disk temperature policy now decides disk heat. alerts.Manager.DiskTemperatureThreshold exposes the per-type resolution CheckHost already applies. - Disk risk no longer judges temperature. Every registry that rebuilds risk lacks the alert configuration, so no rule there could honour it. A hot disk keeps an online status. - The Health verdict reads Running Hot when a current reading reaches the disk type's trigger. It uses the thresholds object that colours the Temp cell, so the two cannot disagree. The amber band below the trigger stays a cell colour, not a verdict. - The TrueNAS storage table lost the heat it used to get from risk. It now judges heat the same way: an Attention row with the reason, and a warning dot for phones. - Patrol triage, its disk issue filter, finding recovery and the chat context's attention list judge the collected reading against the same trigger, including scoped runs and a provider set before Patrol starts. - A switched-off agent Disk Temp default resolves to no thresholds, which the displays that read them now treat as off. Per-host Disk Temp overrides do not reach these surfaces yet. TrueNAS disk alerts, the TrueNAS disk drawer and PDF reports keep their own fixed temperatures for now. |
||
|
rcourtman
|
1b6a9b45d7 |
Keep retained disk temperatures out of AI, Patrol and reports (#2625)
Normalization keeps a disk's last-known temperature when the current observation did not collect one (a disk in standby, a host agent past its reporting lease) and marks it non-available. These readers still took `PhysicalDisk.Temperature` at face value; each now goes through `diskinventory.TemperatureCollected`: - **AI chat context**: "Physical Disks Needing Attention" no longer flags a disk for heat from a retained reading. A disk listed for another reason shows it as `last known 52C (host agent stopped reporting)`. - **Patrol**: the disk health-issue gate, triage flags and `disk-high` finding verification judge only a collected reading. Both seed tables label a retained one as last known. - **AI tools**: `pulse_query` / `pulse_metrics` physical disks and `pulse_storage disk_health` put a retained value in `last_known_temperature` + `last_known_temperature_reason` instead of `temperature`. - **Reports**: the performance report's disk table and the reporting runtime snapshot leave a retained value out (no reading). - **Disk temperature charts**: a short series is padded out to now only with a collected reading. Otherwise stored samples are returned as they are. - `monitor.go`'s private copy of the predicate is replaced by the shared helper. Each new test reproduces against the pre-change code: retained `unavailable` temperatures (standby / silent agent) beside a collected one. |
||
|
rcourtman
|
684c0dc598 |
Show a host whose agent stopped reporting as offline (#2634)
A host agent that stops reporting is never shown as offline in the Machines table. Its row renders its last report as if it were current. ## Why The monitor marks a host agent offline once its reporting lease expires, and raises "Host is offline". The registry's stale pass marks the agent sighting stale after 60 seconds, and `aggregateStatus` ranked stale above offline. So the machine reached every consumer as `warning`. On the Machines table that meant an amber dot. CPU, memory, disk, network, disk I/O, uptime and temperature all rendered as live, with threshold colours. Meanwhile the row's own drawer led with the critical offline alert and read "Observed state: Degraded". Docker hosts and Kubernetes clusters hold the same kind of lease, and theirs is shorter than the 120-second stale threshold. A silent Docker host therefore read offline at first, then flipped to warning once its sighting went stale. ## What changes - **Backend.** Each sighting records when a push reporter delivered its machine, Docker host or cluster as offline. The reporters are the host agent, Docker and Kubernetes. Status aggregation counts that sighting as offline whatever its age. - The sighting itself still reads `stale`, so health keeps its `telemetry_stale` reason and monitored-system reasons are unchanged. - A live source still carries a merged row. A Proxmox node whose agent died stays online through the PVE poll. - Pull sources, and the guests, containers, pods and disks that push collectors report about, keep the stale-to-warning rule. - A JSON copy re-derives the marker from its stored status. - A manual link joining two resources from one source keeps the fresher sighting. - **Frontend.** Offline rows already blank CPU, memory, disk, network and disk I/O through the danger gate. - The Machines Uptime and Temp cells and the Docker hosts Temp cell had no gate, and now use it. - The drawer drops an offline resource's uptime and a silent agent's uptime. - The drawer marks a silent agent's non-disk Thermals rows "(last known)" with the reason as title, as retained disk rows already are. ## Verification - Go tests cover the agent, Docker, Kubernetes, mixed-source, Proxmox-node, JSON round-trip and manual-link cases, plus the unchanged late-agent and stale-PVE boundaries. Each fix's test fails without it. The `unifiedresources` package passes. - The `api`, `monitoring`, `ai`, `alerts` and `agentcontext` suites ran on pulse-dev. One `internal/api` test failed there, `TestServerInfoEndpointReportsDevelopment`, and it fails identically on the pre-fix build in that scratch tree. - Frontend tests cover the Machines offline row, the Docker offline Temp cell, the drawer thermals marking and the drawer uptime rows. The adjacent suites pass. - Live check on a private mock stack, holding one standalone Linux agent and one Docker host past their leases, at 1440 and 390 wide. Before: amber row with live-looking readings. After: red row with every reading blanked, the drawer reading "Offline", and thermals marked last known. - The browser receipt is included. Bundle sizes are within budget: WorkloadsSurface is unchanged, Standalone +0.04 kB, Docker +0.01 kB. Builds on rcourtman/Pulse#2630 and updates its contract sentence, which described the lapsed agent as reaching the table as `warning`. Known follow-up: the drawer's Disks card still threshold-colours a silent agent's retained filesystem usage. |
||
|
rcourtman
|
fb55d47517 |
Raise disk health alerts for mock physical disks (#2633)
`checkMockAlerts` evaluated guests, agents, nodes, storage, PBS, PMG and Docker against the mock estate but never physical disks. The mock generator deliberately keeps a FAILED health cohort and worn SSDs stable across restarts, so mock and demo stacks showed failing and "Replace Now" disks with no `disk-health` or `disk-wearout` alert, and the disk alert lifecycle could not be exercised without real hardware. ## Change - `checkMockAlerts` now evaluates fixture disks through `checkMockPhysicalDiskAlerts`, applying the physical disk poller's boundary: only disks on online nodes (matched by instance and node name), with the linked agent's `--disk-exclude` patterns forcing a matched device healthy. - The poller's per-disk rule (exclusion forces healthy, else `CheckDiskHealth`) moves into `checkPhysicalDiskAlerts`, shared by `maybePollPhysicalDisksAsync` and the mock pass. The poller's behaviour is unchanged; the source guardrail `TestProxmoxDiskAlertsRunOnMergedDiskState` now pins the helper call. - The poller's host-agent link settle deferral (#1674) is not applied in mock mode: fixture links and exclusions are complete from the first pass. ## Demo-facing effect Measured on private stacks with the public demo's exact estate (`/etc/pulse/.env`: 8 nodes, `DEMO_MODE`), baseline vs patched: the demo estate has 12 Proxmox disks and none FAILED. The only new disk alert is one warning-level `disk-wearout` ("SSD has less than 10% life remaining (7% wearout)" on West Production E), whose resource ID is one of the disk resource's canonical aliases. No `disk-health` alert opens on that estate. The default test estate (50 nodes) raises five `disk-health` and one `disk-wearout` alert. ## Known gaps, left out of scope - Like the other loops in `checkMockAlerts`, the disk loop relies on the pass's single up-front mode check, so a pass in flight when mock mode is switched off can reopen alerts after `ClearActiveAlerts`. That race predates this change and also involves the frontend refresh and backup rollup paths, so it needs a monitor-wide fix rather than a lock around this pass alone. - `CheckDiskHealth` logs at error/warn on every evaluation of a FAILED or worn disk, so in mock mode such a disk logs once per alert tick. - The wearout message calls 7% life remaining "7% wearout". ## Verification - `TestCheckMockAlertsEvaluatesPhysicalDisks` (pinned default estate) and `TestCheckMockPhysicalDiskAlertsFollowsPollerBoundary` (offline node, same node name on another instance, exclusion resolving existing alerts, node departure cleanup). Mutations that drop the pass, ignore offline nodes, ignore exclusions, or key by node name only each fail a test. - Full `go test ./internal/monitoring/` and the mock tests under `-race` pass. |
||
|
rcourtman
|
efd3b29ac8 |
Poll only the history each metric table renders (#2632)
`useWorkloadTableMetricHistory` ran two polled estate-wide reads per instance in Trends mode: the workloads history (`/api/charts/workloads`) and the infrastructure summary (`/api/charts/infrastructure`). Since #2622 removed the host metric cells from grouped workload rows, the Workloads surface reads only guest series, and the Proxmox nodes table never read anything but node series. Two instances never shared reads either: each forwards its own abort signal, which skips the summary caches' in-flight dedupe, and retained query values only seed matching mounts and source changes without sharing requests. So every reader paid for both routes every 30 seconds. The hook now takes a required `series: 'guests' | 'nodes'` option. The Workloads state passes `'guests'` and the Proxmox nodes table passes `'nodes'` (and drops the `selectedNode` it never needed), so each reader polls one route. ### Measured Private demo-mode mock stack, Trends on, 95 s per page (first read + three 30 s polls), reads per poll tick: | Page | Viewport | Before (infra + workloads) | After (infra + workloads) | |---|---|---|---| | /vmware | desktop 1440x900 | 1 + 1 | 0 + 1 | | /vmware | phone 390x844 | 1 + 1 | 0 + 1 | | /proxmox/overview | desktop 1440x900 | 2 + 2 | 1 + 1 | | /proxmox/overview | phone 390x844 | 2 + 2 | 1 + 1 | Node table sparklines on /proxmox/overview still draw for every row (36 on desktop, 18 on phone), and the guest table sparklines are unchanged. ### Tests - `useWorkloadTableMetricHistory.test.tsx`: a `'guests'` reader polls only the workloads route and a `'nodes'` reader only the infrastructure route, across two polls. Both cases fail with the gating reverted. - `ProxmoxNodesTable.test.tsx`: the table asks for node history only, with no node scope, and enables it only in Trends. - `WorkloadsSurface.performance.contract.test.tsx` pins both call sites and the gating. The performance contract claimed the two readers deduped their fetches; it now says they do not and that each reader polls only what it renders. The adjacent Bars-mode prose (performance and frontend-primitives contracts) described one active-row request that leaving the row cancels; it now describes the bounded warming window the hook has actually run since the history lens shipped. Stacked on #2622, whose auto-merge is already on: this branch contains its commits and merges cleanly with main either way round. |
||
|
rcourtman
|
4c4e6e7c57 |
Call a disk that is only hot Running Hot, not Replace Now (#2623)
On Physical Disks, a SMART-passed disk with good life running at 72°C
read **Replace Now** ("Disk temperature is 72C"), and **Replace** on
phones. Disk risk grades temperature (`temperature_high`, ≥70°C
critical, ≥60°C warning), and the Health verdict mapped any critical
risk to Replace Now. That tells a homelab user to buy a disk when the
fix is airflow. Alerts already keep heat separate: the SMART health and
wearout alerts skip `temperature_high`, and disk heat is raised as a
Temperature metric.
## Change
- `diskPresentation.ts` keeps each risk reason's code and severity, and
splits the reasons into heat and everything else.
- FAILED health, any other critical reason, or a critical level that no
reason explains → **Replace Now** (unchanged).
- Heat alone → **Running Hot** (phone **Hot**). Red at the critical
tier, amber at warning. The temperature reason is still the cell title
and the drawer summary.
- Other warning evidence, SMART counters or low life → **Needs
Attention** (unchanged).
- The more severe class names the verdict, and failure evidence wins a
tie. Merged risks that list heat first (`mergePhysicalDiskRisk` does not
re-sort) no longer hide pending sectors behind a temperature summary.
- Sort priority and the health filter derive from the same verdict.
Order is Replace Now, then red Running Hot, then Needs Attention, then
amber Running Hot. Hot disks stay in **Needs attention** and land in
Critical or Warning by their tier.
- Data with no reason codes keeps the old behaviour.
## Not changed
Thresholds. The tier still comes from the server's flat 60/70°C rule,
not the per-type alert thresholds that colour the Temp cell. An NVMe at
60–64°C therefore reads amber Running Hot next to a green Temp cell.
That disk read Needs Attention before this change, so the judgment is
the same and only the word is clearer. Reconciling the thresholds is
being done separately.
## Verification
- New tests in `diskPresentation.test.ts` cover heat at both tiers,
failure evidence listed after heat, heat plus wear, an unexplained
critical level, sort order and the health filter. Five of them fail on
the old code, including the sort test, whose device names make name
order alone disagree. `DiskList.test.tsx` checks the full and compact
words, tone, title and row order.
- Targeted suites (storageBackups, Storage, GuestPhysicalDisks,
frontendResourceTypeBoundaries): 89 files, 1513 tests pass. tsc and
eslint are clean, and the canonical platform audit is clean.
- Browser check on a private mock stack (local, uncommitted generator
overlay pinning pve3 disks at 72°C, 63°C, and 72°C with 2 pending
sectors), at 1440×900 and 390×844. The rows read Replace Now / Running
Hot / Needs Attention / Running Hot (phone: Replace / Hot / Attention /
Hot) with the expected tones and titles. The drawer reads "Running Hot ·
Disk temperature is 72C". The `status=attention|critical|warning`
filters list the expected rows. A same-session control with the old file
showed the 72°C disk as Replace Now. The receipt is
`frontend-modern/browser-verification/259ae9ced29e073f.json`.
|
||
|
rcourtman
|
c06af69fcb |
Show a host whose agent stopped reporting as offline
A host agent past its reporting lease is offline: the monitor marks it so and raises "Host is offline". The registry's stale pass ranked the agent's stale sighting above offline, though, so the machine reached every consumer as a warning. The Machines table showed it amber and rendered its last CPU, memory, disk and temperature as current readings, while its own drawer led with the critical offline alert. A Docker host was offline only until its sighting crossed the 120-second stale threshold, then flipped to warning, and Kubernetes clusters likewise. Each sighting now records when a push reporter (host agent, Docker, Kubernetes) delivered its machine, Docker host or cluster as offline, and status aggregation counts that as offline whatever the sighting's age. The sighting itself still reads stale, and a live source still carries a merged row, so a Proxmox node with a dead agent stays online. A copy that went through JSON re-derives the marker from its stored status, and a manual link joining two resources from one source keeps the fresher sighting, so a retired agent cannot pull a live one down. Offline rows then take the existing danger gate. The Machines Uptime and Temp cells and the Docker hosts Temp cell had no gate and now follow it. The drawer drops a silent agent's uptime and marks its non-disk Thermals rows "(last known)", as retained disk rows already are. |
||
|
rcourtman
|
3c1464e42f | Merge remote-tracking branch 'origin/main' into fix/disk-temp-consumers | ||
|
rcourtman
|
344f1f8531 |
Raise disk health alerts for mock physical disks
checkMockAlerts evaluated guests, agents, nodes, storage, PBS, PMG and Docker against the mock estate but never physical disks. The fixture keeps a FAILED cohort and worn SSDs stable on purpose, so mock and demo stacks showed failing and "Replace Now" disks with no disk-health or disk-wearout alert, and the disk alert lifecycle could not be exercised without real hardware. The pass applies the physical disk poller's boundary: only disks on online nodes, matched by instance and node name, with the linked agent's --disk-exclude patterns forcing a matched device healthy. The per-disk evaluation moves into checkPhysicalDiskAlerts so the poller and the mock pass share one rule. The poller's agent-link settle deferral is not applied, because fixture links and exclusions are complete from the first pass. Like the other loops in checkMockAlerts, the disk loop runs after the pass's single mode check, so the existing race with leaving mock mode now covers disk alerts too. That race also involves other fixture evaluation paths and is left for a monitor-wide fix. On the public demo's estate (8 nodes) this adds one warning-level disk-wearout alert and no disk-health alert: that estate has no FAILED Proxmox disk. |
||
|
rcourtman
|
8732bc1923 |
Remove the unreachable host metric cells from grouped workload rows (#2622)
Both production mounts of the Workloads surface, the Proxmox and vSphere overviews, set `compactGroupHeaders` unconditionally. The non-compact branch in `WorkloadPanel` (per-column node CPU, memory, disk and I/O bars or sparklines, uptime, version, temperature) and the `NodeGroupHeader` column mode and inline host facts that only it reached never rendered. They were left over from the retired standalone Workloads page. This removes the option, the dead renderers and the `NodeGroupHeader` props that only served them. Grouped host rows keep the shape both pages already render: one spanning cell with status, linked name, cluster and agent badges. Host stats stay in each page's own hosts table. Verification - No other production consumer: `WorkloadsSurface` → `WorkloadsTable` → `WorkloadPanel` is entered only from `ProxmoxPageSurface` and `VmwarePageSurface`; no references in pulse-pro, pulse-enterprise or pulse-mobile. - Browser pass on a production build embedded in a private mock backend, `/proxmox/overview` and `/vmware` at 1440x900 and 390x844, Bars and Trends modes: every grouped host row is one cell spanning the guest table's visible columns, with no metric cells or host facts; group-row clicks open nothing; guest drawers still open; no page errors. - The rewritten `NodeGroupHeader` test fails against the old component (its default facts render `PVE 9.1.9`). - `tsc`, full `npm run lint`, vitest over Workloads, shared, features, hooks, utils, Storage and app tests (only the Mac-only `agentInstallCommand` failures, unrelated), bundle budgets, and all release-control guards pass. |
||
|
rcourtman
|
8e93360d10 |
Poll only the history each metric table renders
useWorkloadTableMetricHistory ran two polled estate-wide reads per instance in Trends mode: the workloads history and the infrastructure summary. Since grouped workload rows lost their host metric cells, the Workloads surface reads only guest series, and the Proxmox nodes table never read anything but node series. Each reader passes its own abort signal, which skips the summary caches' in-flight dedupe, so every instance paid for both routes every 30 seconds. A required series option now names the rows a caller draws. The Workloads state passes 'guests' and the Proxmox nodes table passes 'nodes', so each polls one route. On the demo estate in Trends this takes the vSphere overview from one infrastructure and one workloads read per poll to one workloads read, and the Proxmox overview from two of each to one of each. Node table sparklines are unchanged. The performance contract claimed the two readers deduped their reads; it now states that they do not and that each reader polls only what it renders. The adjacent Bars-mode prose in the performance and frontend-primitives contracts described one active-row request that leaving the row cancels; it now describes the bounded warming window the hook has run since the history lens shipped. |
||
|
rcourtman
|
723f3165a1 |
Show a silent agent's retained disk temperatures as last known
When a host agent passes its reporting lease, its collected SMART temperatures keep their values with collection.temperature unavailable (#2593), and #2603 taught the Storage disk surfaces to show such a value as last known. The frontend HostDiskSMART type never carried `collection`, so the agent's own surfaces kept presenting the retained readings as current: - The Machines temperature cell fell back to the hottest SMART reading and coloured it against the disk thresholds. On a live mock stack a silent, SMART-only agent read 52°C in threshold green. - Its tooltip, the machine drawer's Thermals rows and the guest drawer's Physical Disks card listed the retained disks as plain readings. The row does not make this obvious. By the time the lease expires the registry has marked a standalone agent's source stale, which outranks offline, so a silent agent reaches the table as `warning` with `agent.stale`, not `offline`. The offline metric fallback never blanks its cells, and the temperature cell has no such gate. Now `HostDiskSMART.collection` is typed and every reader decides through `isPhysicalDiskTemperatureCurrent`: - A retained disk reading never stands for the machine while another disk is current. When no non-standby disk is current and the machine has no positive direct or temperatureCelsius reading, the cell shows the hottest retained non-standby value muted with a dotted underline and screen-reader "last known", without threshold colour. - Tooltip and Thermals rows read "(last known)", the Thermals row with the collection reason as its title. A disk with a zero or negative reading no longer shows a temperature row there. - The guest Physical Disks card uses the Storage table's last-known treatment. The temperature helpers move to diskTemperaturePresentation.ts, which diskPresentation.ts re-exports. On the earlier base, importing diskPresentation from the Machines page and the drawer mapper folded the whole Storage presenter into the shared chunk (+4.9 kB gzip) and renamed its tracked budget entry, failing the bundle check. With the small module, Standalone grows 0.22 kB, the shared chunk 0.20 kB, and Workloads is flat against a build of main. The guest card still loads diskPresentation for extraction and health. Direct and temperatureCelsius readings carry no collection state and still lead the cell. A stale agent's CPU, memory, disk and sensor values still render as current. That is a row-wide decision, left for a separate change. |
||
|
rcourtman
|
8424fd6505 |
Show real CPU low and record in the Proxmox node drawer (#2594)
The Proxmox node drawer's **CPU low** and **CPU record** rows repeated the current temperature for every node without a host agent. `nodeFromResource` never read `proxmox.temperatureDetails`, which is where the resource API publishes the poller's full reading, including its low and record values. For those nodes it fell back to the scalar `proxmox.temperature` and copied that one value into both fields. On the dev stack, `delly` and `minipc` showed CPU low = CPU record = current. - The adapter now takes the first source with a usable reading. `temperatureDetails` comes after the platform record and the agent facet, so agent-linked nodes render exactly as before. - The bare-reading fallback leaves low and record unset, so the drawer omits those rows rather than inventing them. The poller also reset those values to the current reading every poll. That is a separate backend fix to the previous-node projection, landing in its own PR. Until it lands, PVE-only nodes will still show low = record = current, now because that is what the backend reports. ### Verification - Unit tests pin the API's real payload shape (scalar `proxmox.temperature` plus `proxmox.temperatureDetails`), the no-record fallback, and agent-facet precedence. The new test fails against main's adapter, which returns `cpuMin` 58 = current. - Browser pass on a private mock stack at `/proxmox`, 1440x900 and 390x844, with REST and websocket payloads rewritten to make one node PVE-only: - main's adapter showed CPU low 82°C / CPU record 82°C; - this change shows no rows without a record, and 39°C / 91°C with one; - agent-linked nodes and the table TEMP column are unchanged. - `vitest` over `src/utils`, `src/components/Workloads`, `src/features/proxmox` and `src/features/platformPage` passes, apart from `agentInstallCommand.test.ts`, which fails on macOS regardless of this change. |
||
|
rcourtman
|
7b362b3532 |
Load guest physical disks with the drawer to keep Workloads in budget
GuestDrawerOverview statically imported GuestPhysicalDisks, which pulled all of diskPresentation.ts into the WorkloadsSurface table chunk. With the Running Hot verdict that chunk measured 70.64 kB gzip against its 70.23 kB cap, so the Frontend bundle check would fail. The card renders only inside an open drawer for a guest with a Pulse agent, so the overview now lazy-loads it under a local Suspense, the same pattern as the bulk web-link editor. WorkloadsSurface drops to 64.31 kB and the disk presentation builds as its own chunk. The contract test pins the dynamic import. |
||
|
rcourtman
|
fb7dd955ed |
Keep retained disk temperatures out of AI, Patrol and reports
Normalization keeps a disk's last-known temperature when the current observation did not collect one (a disk in standby, a host agent past its reporting lease) and marks it non-available. Several readers still took PhysicalDisk.Temperature at face value. The AI chat context listed a standby disk as needing attention for heat. Patrol raised health issues and triage flags from the retained value. The AI query and disk-health tools handed it to the model as the disk's temperature. Performance reports and the reporting runtime snapshot tabulated it as measured, and the disk temperature charts padded a series out to now with it. Each of them now goes through diskinventory.TemperatureCollected. AI surfaces keep a retained value as context, labelled last known with its reason. Reports and the charts leave it out. monitor.go's private copy of the predicate is replaced by the shared helper. |
||
|
rcourtman
|
d9ffb39ce8 |
Keep a merged disk's temperature under its own row's state (#2618)
Stacked on #2609; until it merges this branch also carries its commit, and the change here is the last commit. ## Why The registry merges a physical disk's rows (Proxmox, host agent SMART, Unraid inventory), choosing the temperature by source preference and merging collection states separately. When two rows carry different temperatures, the shown value could sit under the other row's `available` state. A real case: a node whose disk temperature Proxmox collects now (`proxmox_node_smart`, 40) and whose linked host agent has gone silent (retained 72, `unavailable`). In either ingest order the merged disk showed **72 labelled available**, so risk, history and the UI treated the frozen reading as current. ## What - `pairPhysicalDiskTemperatureState` runs at the end of each merge step. When the merged state says collected but the row the shown value came from says it was not, the merged state becomes that row's own. - It never changes a value and never grants availability, so it cannot revive a withdrawn reading or override a resumed agent's fresh one. - Rows with the same value keep the merged state, so the agent's own withdrawal still supersedes the Proxmox copy (#2593). - With that guarantee the Unraid-native row no longer needs to withhold `available` (#2609 restricted it), so it carries the same derived state as a SMART row borrowing its reading. ## Known limit This is exact for the two rows of one merge step. The merged disk keeps one state per field, so with three or more rows for one disk, a withdrawal that an earlier step replaced is invisible to a later step. Sol found two such counterexamples, both needing three rows (one also needs coincidentally equal values, the other a legacy agent plus two Proxmox rows). Closing that needs each reading's own state tracked across merges, and that redesign is filed separately. Preferring another source's current value instead of the retained one was tried and dropped: it needs an ordering between same-source states that the rows do not record, and each version reopened a withdrawn reading somewhere. ## Tests - `TestPhysicalDiskMergePairsTemperatureWithItsCollectionState`, in both ingest orders: - silent agent against a collected Proxmox reading, and against no Proxmox reading - reporting agent - silent agent against a Proxmox copy of its earlier reading, with and without SMART attributes - resumed agent against a Proxmox copy of its withdrawal - standby agent against a Proxmox copy of its pre-sleep reading - `TestPhysicalDiskMergeNeverPresentsAWithdrawnTemperatureAsCollected`: three distinct-valued withdrawn or stale rows in all six orders. - `TestUnraidInventoryRowDoesNotMaskTheSMARTRowState`: the Unraid inventory's `available` no longer lends itself to a SMART row's uncollected reading. - Without the rule, the peer-reported case, the Unraid masking case and one three-way order fail. |
||
|
rcourtman
|
dcb69cfb02 |
Record Docker host, service and sub-resource alert history (#2620)
Follows #2608, which made `MonitorAdapter.RecordChange` resolve Proxmox and agent alert references to canonical resources. Three kinds of alert reference still journaled under strings no resource answers to: - Docker host alerts (`docker:<host ID>`) - Swarm service alerts (`docker:<host ID>/service/<ID>`) - sub-resource alerts such as ZFS pool and device health (`<storage ID>/zfs-pool:<pool>[/device:<dev>]`) and host filesystem, SMART, RAID and sensor alerts (`agent:<host ID>/disk:…` etc.) Resource facets, timelines, resource intelligence and Assistant context read history by canonical ID, so these alerts never appeared there. ## What changes - **Docker host refs** resolve through the registry's Docker source ID. - **Swarm service refs** resolve through the host's Swarm cluster to the service with that exact ID, from any manager that reports it. A service with no ID is referenced by its name in the same slot, so while a cluster has such a service, every service ref there is a conflict and binds nothing. - **Never bind:** container names, shortened container IDs and hostless `docker-service:<name>` refs never bind, and are no longer retried against later inventory. - **Sub-resource refs** bind to the owner whose durable ID they carry: - ZFS pool and device alerts go to the storage. - The Unraid array alert goes to its array storage. - Filesystem, SMART, disk temperature, RAID and custom-sensor alerts go to the host. - The owner must resolve to exactly one resource of the expected type. An ambiguous or incompatible owner is a conflict. - Mount and kernel device labels can name a different disk after a reboot, so these alerts never bind a physical disk. - **Retries:** a reference that hits a conflict at write time now enters the backfill retry set, the same as one found at startup. ## Known residuals (producer-side, follow-up proposed) - A container with no ID alerts under its host's reference (`DockerResourceID(host, "")`), so its events join that host's history (never another container's). - An ID-less service whose name equals another service's ID shares that service's reference. Rows it journaled can join the other service's history once a later generation or event binds the reference. Swarm always reports service IDs, so this needs a malformed report. Both are documented in the contract. Separating them needs a producer change in `internal/alerts`. ## Evidence Private `PULSE_MOCK_MODE` stack, fresh data dir. The binary at #2608's tip ran first, then this change on the same dir. Alert rows were classified by source reference (from `alert_identifier`) and checked against each owner's `/api/resources/{id}/timeline`: | ref shape | rows readable via owner timeline, before | after | |---|---|---| | Docker host | 0 / 1 | 3 / 3 | | Swarm service | 0 / 5 | 19 / 19 | | ZFS pool | 0 / 9 | 26 / 26 | | ZFS device | 0 / 9 | 27 / 27 | | agent filesystem | 0 / 1 | 3 / 3 | | Docker container (mock short IDs) | 0 / 11 | 0 / 30, no container alias written | Tests: - `TestHistoryIdentityMonitorAdapterResolvesDockerHostAndServiceReferences` covers multi-manager services, a Docker host merged into its agent machine, legacy-row backfill, recovery after removal, ID-less service conflicts, and names and short IDs staying unbound. - `TestHistoryIdentityMonitorAdapterResolvesSubResourceReferences` covers every owner shape, wrong-type and ambiguous owners, retained bindings, write-time conflict retry, and physical disks staying unbound. - `TestOwnerAlertTimelinesUseCanonicalHistoryIdentity` drives the real alert manager for Docker host offline, Swarm service gap, ZFS pool/device and host filesystem alerts. It fails on #2608's tip. UI reach: the Docker, storage and platform host-row drawers don't request per-resource history. This fix reaches the facets/timeline/intelligence APIs and Assistant resource context, not a drawer. |
||
|
rcourtman
|
1fe9f86d55 | Merge remote-tracking branch 'origin/main' into refactor/drop-noncompact-group-node-metrics | ||
|
rcourtman
|
94b7870855 |
Call a disk that is only hot Running Hot, not Replace Now
The Physical Disks Health verdict mapped any critical disk risk to "Replace Now". Disk risk also grades temperature (temperature_high at >=70C critical, >=60C warning), so a SMART-passed disk with good life running at 72C told the user to buy a new disk when the fix is airflow. Alerts already keep heat apart: the SMART health and wearout alerts skip temperature_high and raise disk heat as a Temperature metric. The verdict now splits risk reasons into heat and everything else. Heat alone reads "Running Hot" (phone "Hot"), red at the critical tier and amber at warning, with the temperature reason still as its title and drawer summary. FAILED health, any other critical reason, and a critical level no reason explains keep "Replace Now". The more severe class names the verdict and failure evidence wins a tie, so merged reasons that list heat first no longer hide pending sectors. Sorting and the health filter follow the same verdict, so hot disks stay in "Needs attention" at their temperature tier. A reason's code and severity count even when it has no display text. A critical level that no reason explains shows the generic summary rather than a weaker reason's text. Thresholds are unchanged. The tier still comes from the server's flat 60/70C rule, not the per-type alert thresholds that colour the Temp cell. Reconciling those is separate work. |
||
|
rcourtman
|
5becf59536 |
Remove the unreachable host metric cells from grouped workload rows
Both production mounts of the Workloads surface, the Proxmox and vSphere overviews, set compactGroupHeaders unconditionally. WorkloadPanel's per-column node cells (CPU, memory, disk and I/O bars or sparklines, uptime, version, temperature) and NodeGroupHeader's column mode and inline host facts therefore never rendered. They were left over from the retired standalone Workloads page, whose group rows had no hosts table beside them. Drop the option, the dead renderers and the NodeGroupHeader props that only served them. Grouped host rows keep exactly the shape both pages already render: one spanning cell with status, linked name, cluster and agent badges, while host stats stay in each page's own hosts table. Tests that only pinned the option are removed. The NodeGroupHeader test now covers the spanning-cell shape and fails against the old default facts, and the PVE version formatter guardrail moves to proxmoxPageModel, the remaining version renderer. |
||
|
rcourtman
|
0fb5ccb62b |
Keep node temperatures off same-named nodes at other sites (#2617)
Two Proxmox connections can each have a node with the same name, which
is common for MSPs running one `px1` per site. With one host agent
linked to site A's `px1`, `getHostAgentTemperatureByID("siteB-px1",
"px1")` returned site A's temperature. The hostname fallback took the
first agent with that hostname, even one linked to another node, and
even when the name was ambiguous enough that the agent linker had
refused to link it. The cluster sensor cache leaked the same way: SSH
readings an agent collects for its cluster siblings were keyed by bare
node name. Site B's `px1` then showed, alerted on and recorded history
for another machine's sensors.
## Change
- The poller hands the lookup the polled node
(`getHostAgentTemperatureForNode`,
`carriedTemperatureOutlivesAgentLease`), and the lookup works on that
node's **slot**: its own source ID, plus, when the read state does not
hold that ID, the one same-named node proven to be the same machine.
- The state folds the views of one machine reached through two
connections (a cluster added twice, or a multi-homed host added by each
address) into a single node kept under one view's ID, and an automatic
agent link follows it. A narrowed fallback on its own would leave those
views without a temperature permanently; that was reproduced with real
`UpdateNodesForInstance` folds.
- Same-machine proof comes from the state's own evidence: new
`models.NodeObservationsSameMachine` and
`models.HostAgentBridgesNodeViews`
(`internal/models/node_machine_identity.go`). `UpdateNodesForInstance`'s
agent-corroboration check now shares its rule with them. Two or more
candidates are ambiguous and none is taken, as the state declines an
ambiguous alias.
- An agent linked to the slot is the node's agent. The hostname fallback
only takes an agent with no node, VM or container link, and only when no
node outside the slot and no second unlinked agent has the name.
- Cluster sensor cache entries record their reporting agent. A reading
serves a node only when that agent's linked node shares one of the
slot's connections, or belongs to a connection whose configured view of
the node has a matching TLS fingerprint (the other connection of a
cluster added twice).
`LinkedNodeID` holds the Proxmox source node ID for every link source
(automatic `findLinkedProxmoxEntityWithHints` → `NodeView.SourceID()`,
manual `LinkHostAgentToNode` validating against `models.Node.ID`,
reconciliation, continuity restore), so the link match needs no
namespace normalization. A manual link stays pinned to the ID the
operator chose (the state never transfers intent to a replacement
provider identity), so a node held under the other view's ID has no
linked agent. The physical-disk SMART merge already matches by
`node.LinkedAgentID` within one instance and is unchanged.
## Tests
- `TestNodeTemperatureStaysWithinSameNamedNodesSite` (registered
monitoring proof, `monitor_additional_test.go`), through
`collectNodeTemperatureData` against registry-backed and state-folded
read states: agent linked to the other site's node; ambiguous unlinked
agents; unambiguous unlinked agent (still served); cluster added through
two connections (alternating real state folds); multi-homed host via two
standalone connections; ambiguous address alias; a namesake agent with a
contradicting fingerprint; sibling readings across both connections of
one cluster; sibling reading from the other site. The leaking cases fail
on main, and each folded-view guard was checked with an overlay negative
control.
- `TestApplyHostReportScopesClusterSensorsToReportersConnection`
(`monitor_host_agents_test.go`): real `ApplyHostReport` ingest of
cluster sensors.
- `TestNodeObservationsSameMachineMatchesStateFold` and
`TestHostAgentBridgesNodeViewsMatchesStateFold` pin the predicates
against `UpdateNodesForInstance`'s actual fold.
- Full `internal/monitoring`, `internal/models` and
`internal/unifiedresources` pass on pulse-dev.
|
||
|
rcourtman
|
3baab9bc59 |
Say a reviewed Patrol issue stays reviewed when suppression ends (#2615)
Ending a suppression puts the alert in Acknowledged when it is
acknowledged and in Open otherwise (`unsuppressOperationalAlert` in
internal/alerts/active_lifecycle.go). Acknowledged is not active
attention (`attentionStateCountsAsActive`).
The Patrol attention detail still promised otherwise for an issue that
was marked reviewed and then suppressed:
- its early-end button read "Return to active attention"
- the handled-view notice read "Returned to decision inbox"
- the suppression form offered "Return it to active attention after"
In each case the issue actually went back to Reviewed.
| | Reviewed issue | Unreviewed issue (unchanged) |
|---|---|---|
| Suppression duration | End suppression after | Return it to active
attention after |
| Early-end button | End suppression | Return to active attention |
| Handled-view notice | Suppression ended. It stays reviewed. | Returned
to decision inbox |
The duration label and the button follow the detail's
`operationalRecord.acknowledgement`. The notice instead reads the item's
state from the reload that follows the change. If the review was removed
elsewhere after the detail loaded (another tab), it still reports what
actually happened. `getAttentionSuppressionEndPresentation` owns the
copy.
The countdown line under the suppression ("Returns to active attention
in 1d.") is fixed separately by fix/future-relative-time, using the same
wording.
Verification:
- Three new tests in `PatrolAttentionWorkbench.test.tsx`:
- The reviewed case fails on main.
- The stale-detail case fails on a version that predicts the notice from
the detail.
- The unreviewed case pins the copy that stays the same.
- `src/features/patrol` and the two other test files that import the
workbench pass (492 tests).
- Run live on a private mock stack at 1440px, 800px and 390px:
- Reviewed case: mark an issue reviewed, suppress it, end the
suppression early from the handled view. The button read End
suppression, the notice read "Suppression ended. It stays reviewed. N
handled issues remain.", and the attention API reported the item
`acknowledged` with the active count unchanged.
- Unreviewed case: the item came back to the inbox as `open`.
- Stale-detail case: the review was removed through the API while the
detail stayed open. The notice read "Returned to decision inbox",
matching the API state `open`.
|
||
|
rcourtman
|
c3adf0c7b9 |
Chart Net and Disk I/O in the row history lens (#2614)
Hovering a guest row in bars mode swaps CPU, memory and disk for history
sparklines, but Net I/O and Disk I/O stayed as static numbers. They are
the jumpiest readings in the row, so they gain the most from history
("short burst or sustained?"). The hover request already returns the I/O
series for that guest, so this adds no reads.
Changes:
- The row history lens now swaps both I/O cells too, sharing the
synchronized scrub cursor.
- I/O sparklines (lens and Trends) keep a compact current pair beside
the chart (`↓3.3M ↑2.2M`, `R1.4M W826K`) instead of hiding the values in
a tooltip. The glyphs use the series colours, so the pair doubles as the
legend. The full rates stay in the chart's accessible label.
- Rate sparklines get a 100 KiB/s scale floor. Before, a guest moving
between 2 and 6 KB/s drew the same full-height spike as one at 500 MB/s.
- `GuestRow` builds every sparkline from accessors. Each live snapshot
used to remount the lens and Trends charts (dropping an open scrub
tooltip). An open tooltip now also re-reads the current series instead
of holding the values from the last mouse move.
- `formatCompactSpeed` for the compact pair, the "History" hover option
tooltip, and the frontend-primitives and performance contract clauses.
Verified on a private mock stack at 1920, 1600, 1440 and 768 (dark and
light), including scrub sync, an intercepted idle-I/O history, a live
websocket update under an open tooltip, and Trends mode (0 clipped
cells). Receipt included.
Two pre-existing issues found while verifying are left out of scope. At
about 1440px the first-use history hint's dismissal shifts the table
36px, so the first preview opens on the next row. And `WorkloadPanel`'s
non-compact node header metric cells never render in production.
|
||
|
rcourtman
|
8259a74c9d |
Show the live reading of threshold alerts held below trigger (#2567)
A threshold alert stays open after its reading drops below the trigger. It holds until the reading reaches the clear level, then must stay there for the recovery delay. While it held, the evaluator never touched the alert, so every surface kept showing the last breach. On minipc the drawer said "Node temperature at 80.0°C" next to a green 72°C in the node table, with nothing explaining why the warning was still there. #2068 reports the same thing for a disk alert. **What changes** - Open threshold alerts carry a volatile `metricStatus`: the value Pulse is comparing now (the average for rolling-average rules, with the latest sample beside it), the phase (`breaching`, `latched` or `recovering`), the trigger and clear levels, and recovery progress from the reducer's monotonic run. `value`, `message` and `lastSeen` keep meaning the last breach, and the hold branch still never notifies. - The status is stripped from the active-state checkpoint, the recovery mirror and the degraded recovery projection, so holding adds no durable writes. Resolved records carry none. - Stale-alert cleanup and the retention TTLs use the latest observation. Previously an alert held below its trigger for more than a day was auto-resolved as "resource no longer being monitored". - Node, guest and Mail Gateway drawer rows lead with the reading now and say what clears the alert, with the last breach on hover. The Proxmox TEMP cell keeps its own open temperature alert's tone and explains it on hover. Storage rows stop saying "Over N%" once usage is back under the limit, and the Assistant handoff no longer labels the breach value as current. **Verified** - Unit tests drive a node temperature alert through 85 → 78 → 72 → 75 → 77 → 72 → resolved, and fail against the old evaluator. - Private mock stack with pve2 stepped 88 → 78 → 72°C at desktop and phone widths; see the browser receipt in this PR. Refs #2068 |
||
|
rcourtman
|
38c53d047f |
Remove the never-mounted unified resource table (#2612)
`PlatformResourceTable` was the only renderer of `UnifiedResourceTable`, and nothing imports it. Every platform page renders its own native tables, and `/infrastructure` only shows the opening-workspace placeholder. This removes the never-mounted cluster and the governance that described it. ## Evidence it was dead - Import graph from `src/index.tsx` (static, dynamic, `vi.mock`/`importActual`): the cluster was reachable only through `sharedPlatformPage.tsx`'s import for the unused `PlatformResourceTable` export. No barrel, namespace import, lazy route or re-export. - Production build of main: none of the cluster's literals ship (`infrastructure-table-surface`, `ready for Pulse`, `Agent Infrastructure`), while a live control string (`workloads-table-surface`) does. ## What changes - Deleted: `PlatformResourceTable` and its `PLATFORM_STATUS_FILTER_OPTIONS`; `UnifiedResourceTable` with its host, service, PBS and PMG cards, source badge cell, state hook, viewport sync and models; `ClusterDeployBanner` (only the host card rendered it; the live cluster deploy flow is the Settings `ClusterAgentDeployDialog`); and their seven test files. - Kept coverage for live code those tests also exercised: the detail-mapper no-fetch boundary and the online boolean sensor case move to `resourceDetailMappers.test.ts`; the `ResourceFacetSummary` render cases move to `ResourceFacetSummary.test.tsx`. Both are now registered proofs on the policies that govern their subjects. - Guardrail tests, `shared-template-registry.json`, `registry.json`, `status.json` evidence (RA1, L10, L13; L13 now cites `useUnifiedResources.ts`) and the performance-and-scalability, unified-resources, frontend-primitives and cloud-paid contracts drop the deleted paths and the rules that only governed them. - `.bundlesize.json`: Rollup used to name the large shared chunk `sharedPlatformPage`; it now names it `MetricBar` (118.05 kB gzip), so that entry is re-keyed. Total JS gzip moves +1.96 kB from re-chunking and total CSS drops 0.5 kB. ## Verification - Browser pass on a production build embedded in a private mock backend: all 32 platform sub-tabs at 1440x900 and 390x844 render populated tables with no horizontal overflow; shared toolbar search, the no-match empty state and inline row detail work on Machines and Docker images. The row detail raises eight `style-src-attr` CSP violations, identical on the main build (pre-existing static `overflow-anchor` styles in `ResourceDetailDrawer.tsx`). - `tsc`, eslint, vitest over Infrastructure, platformPage, shared, standalone and app tests; shared-template audit; bundle-size check; contract, registry and status audits; the release-control unittest suites; the full pre-commit battery. |
||
|
rcourtman
|
0f5f054ecb |
Settle slow polled reads and keep replication links honest (#2610)
Follow-up to #2578 from a second-opinion review. It closes three gaps #2578 left, plus the issues found while fixing them. **Polled reads that never settle.** The shared poll in `createNonSuspendingQuery` replaced any read still in flight on every tick. The API answers 429 with `Retry-After: 60` and `apiFetch` waits it out, so under rate limiting a 30-second poll discarded every read and the query never settled. Replication reads also ignored the abort signal, so each replaced request kept running. - A poll now skips its tick while a read that an earlier poll started is still in flight. - A refresh, a source change or a remount revalidation stays replaceable at the next tick, as before. The behaviour pinned by `744160fa0e` and the History tests is unchanged. - A polled read still in flight at or after 150 seconds is presumed stalled. It is replaced and settled as a failed read, so nothing spins forever when every read hangs. - `fetchReplicationJobs` forwards the signal to `apiFetch`. - It throws `apiErrorFromResponse` errors that keep the HTTP status, so a 401 or 403 withdraws the jobs. **Replication direct links.** A failed first read made the job count zero, so `/proxmox/replication` fell to Overview with no error or Retry. - Replication presence is now unknown until a read succeeds. - After a read succeeds, a failed poll keeps the last answer instead of reopening a route that an empty read closed. - An access denial withdraws the tab, and the link holds on the denial message. **The shared clock trailed by a tick.** `useRelativeTimeNow` returned its last tick, so a cell mounted between ticks measured from up to 30 seconds ago, and the desktop backup age could disagree with its band. - Every read now returns the wall clock, and the tick only schedules re-reads. - This retires the backup `Math.max` workaround. - The #2578 age tests now jump the clock and fire one tick. They used to loop hundreds of re-renders, and one timed out under load. Each new test fails without its fix. Live check: a private mock stack driven by Playwright, with replication answers shaped per scenario, at 1600px and 390px. - A 500 on the first read keeps the error and Refresh, and Refresh loads all 8 jobs. - A 403 after loading withdraws the tab and shows the server's denial. - Empty, then failed, then empty stays on Overview. - With a polled read held open, two poll ticks start nothing and abort nothing. - Leaving the page aborts the read. - Backup ages and bands render consistently with no Unknown cells. The same session with main's runtime files reproduced each failure. The browser receipt is in the commit. |
||
|
rcourtman
|
38a5334b90 |
Judge disk risk only on a temperature collected now (#2611)
Since #2593, a disk whose host agent stops reporting keeps its last SMART temperature as a last-known value marked `unavailable`, and a disk in standby keeps its pre-sleep reading the same way. The shared disk risk assessment still fed that value into its temperature rule (>=70C critical, >=60C warning). So a silent agent's frozen 72°C made the Physical Disks row read **Replace Now** ("Disk temperature is 72C") with a `warning` resource status, while #2603 makes the temperature cell beside it say the reading is not current. ## Change - `storagehealth.CollectedTemperature(temperature, collection)` passes a temperature through only when its `collection.temperature` state is `available`, or when the source predates collection state (nil or empty state). Otherwise it returns 0 before risk judges it. - Three places use it: - `AssessPhysicalDisk`, for Proxmox inventory disks, through the unified adapter; - `AssessHostSMARTDisk(WithThresholds)`, for a host agent's own SMART rows; - `physicalDiskAssessmentFromMeta`, the registry's recompute after it merges sources. - The retained value stays on the resource as last-known history. Health, wearout and SMART counters are unchanged: they are durable evidence and keep judging as before. Alerts are unaffected: - SMART disk-health alerts already discard `temperature_high` (`internal/alerts/host.go`). - Disk temperature alerts evaluate the agent's own fresh report. - No fresh collector emits a positive temperature with a non-available state. ## Evidence On a private mock stack, with a local generator overlay stamping real collector states: - Before: the silent-agent disk (72°C retained) read **Replace Now** and the API reported `status: warning`, `risk: critical`. - After: it reads **Healthy**, the API reports `online` with no risk, and its temperature shows as last known. - A genuinely hot disk (72°C `available`) still reads Replace Now. Whether "Replace Now" is the right word for a heat-only verdict is a separate question. ## Tests - `internal/storagehealth/risk_test.go`: `TestCollectedTemperatureKeepsOnlyCurrentReadings` and `TestDiskAssessmentsIgnoreRetainedTemperature`. - `internal/unifiedresources/registry_test.go`: `TestPhysicalDiskRiskIgnoresRetainedTemperature`. It covers Proxmox disks in every collection state through the monitor adapter, plus a real agent+Proxmox merge of the same disk, collected and retained. - Mutation-checked: removing any of the three gates fails a case, and the merged case fails without the registry-recompute gate. - Full `internal/storagehealth`, `internal/unifiedresources`, `internal/api/resourceapi` and `internal/truenas` pass, plus the SMART/disk/host subsets of `internal/alerts` and `internal/monitoring`. ## Related - The Unraid inventory temperature path in `adapters.go` (`assessUnraidPhysicalDisk`) gets the same gate in the Unraid change that stamps its collection state. - A registry merge can still pair a retained agent temperature with a Proxmox row's `available` state (`registry.go`, Proxmox-over-agent carry). That is being fixed separately. |
||
|
rcourtman
|
4f9bf4c8c4 |
Show retained disk temperatures as last known, not live (#2603)
Normalization keeps a disk's last known temperature when the current observation is not available: a disk in standby, or a linked host agent past its reporting lease (#2593). The monitoring contract says consumers must not present that retained value as freshly collected, but three storage surfaces still painted any positive temperature with live threshold colour: - the Physical Disks table's Temp cell - the pool drawer's linked-disk list - the disk drawer's temperature card Only the drawer's collection message reflected the state, so it read "Temperature is temporarily unavailable: host agent stopped reporting" directly above a red "Temperature 72°C" card. ## Change - One decision in `features/storageBackups/diskPresentation.ts` (`isPhysicalDiskTemperatureCurrent`, `getPhysicalDiskTemperaturePresentation`): - a temperature is current only when its state is `available`, or the source predates collection state (absent or empty state); - `unavailable`, `unsupported` and `missing` make it a retained reading. - A retained reading renders muted, with no hot/warm/healthy colour. It gets a dotted underline (the "this has a note" affordance), the collection reason as its title (for example "Last known reading, not current: disk is in standby"), and screen-reader ", last known" text. - The drawer card is labelled "Last known temperature" and its value is muted. - The pool drawer now formats through the shared temperature formatter, so it follows the Fahrenheit setting like the table. A visible word does not fit the 41px phone Temp column. There, the grey dotted value is the cue, and a tap opens the drawer, which says why. ## Evidence - Searched issues for disk-temperature confusion: #1540, where a user saw a disk temperature and alarm higher than reality that "cleared over night", and #1471, which set the expectation that spun-down disks stay blank rather than show a pre-sleep reading. - Live pass on a private mock stack. The generator was overlaid locally to stamp real collector states, and origin/main's files were swapped in under the same Vite for a control: - Before: the silent-agent disk's retained 72°C was red and indistinguishable from a genuinely hot disk beside it, and the standby disk showed a green 36°C. - After: both read muted with reason titles, and the live 72°C stays red. - Checked at desktop 1440 light and phone 390 dark, with no clipping and no page errors. Receipt in `frontend-modern/browser-verification/`. ## Tests - `diskPresentation.test.ts`: state matrix, including empty state and no reading. - `diskDetailPresentation.test.ts`: last-known card and its tone. - `storagePoolDetailPresentation.test.ts`: an Unraid spun-down member. - `DiskList.test.tsx`: one row moving current → retained → legacy. - `DiskDetail.test.tsx`: the existing collection-message case now asserts the muted last-known card. - `src/components/Storage`, `src/features/storageBackups` and `src/features/platformPage` all pass (107 files). ## Not in this change - The backend risk verdict still judges a retained temperature, so the Health column can read "Replace Now: Disk temperature is 72C" for a disk whose temperature is not current. That is fixed separately in `internal/storagehealth`. - `GuestPhysicalDisks` already renders temperatures without threshold colour. - The Machines table and drawer read host-agent SMART rows whose frontend type does not project `collection` yet. |
||
|
rcourtman
|
cbedbc7edf |
Stop serving retained disk temperatures as current readings (#2609)
Follow-up to #2593 (silent agent SMART readings), which made retained disk temperatures carry a non-available collection state and kept them out of history. ## Why Normalization may keep a disk temperature it did not collect (a disk in standby, a host agent past its reporting lease), provided the collection state says so. #2593 stopped the history writers recording those values. Two consumers still presented them as current: - **History live point.** When a disk range has no samples, `/api/metrics-store/history` answered with the disk's temperature as a `source: "live"` point at now, whatever its state. Now that retained temperatures never reach history, a short-range chart for a silent agent's disk or a sleeping disk showed the retained value as a fresh reading. Mock mode also seeded synthetic series from it, and padded sparse stored series out to now. - **Unraid inventory temperatures.** Unraid array rows carry no per-field provenance, so a temperature taken from one (the Unraid-native disk row, or a SMART row falling back to the inventory reading) could not be told apart from a collected one once the agent went silent. `ExpireHostTelemetry` only marks SMART rows. ## What - `diskinventory.TemperatureCollected` is the shared form of the rule the history writers apply (no state, or `available`). The router's disk live point, the mock synthetic seed and the mock padding all require it. - The unified-resources adapter derives the state of an Unraid inventory temperature from the host lease that `ExpireHostTelemetry` already records atomically, and from the disk's spin state: `unavailable` from `unraid` ("host agent stopped reporting", or "disk is reported spun down"), otherwise `available`. The value stays as last-known context. - A SMART row that borrows the inventory reading takes that state, including `available`. - The Unraid-native row records only the withdrawing states. The registry picks a merged disk's temperature and its state separately, so an `available` claim from the provenance-less row would mask the SMART row's own state for the same disk. - A withdrawn inventory temperature no longer counts as heat in the disk's risk assessment. - Storing per-row state on `models.HostUnraidDisk` instead would have added a model field, a read-state round trip and a frontend type change just to duplicate `host.Status`. ## Tests - `TestContract_MetricsHistoryDiskLivePointOnlyForCollectedTemperature`: a silent agent's disk (through the real `ExpireHostTelemetry` and registry path) and a standby disk get no live point in single-metric or all-metrics responses, and no value-derived mock series. Sparse stored samples are not padded to now for a retained, absent or unknown disk. A collected disk still gets its live point and mock series. Each router gate was reverted individually and the test failed each time. - `TestSilentUnraidAgentDiskTemperatureIsRetainedButNotCollected`: three Unraid disks (inventory only, SMART row without a temperature, SMART row whose temperature is unsupported) go collected, then retained but not collected after `evaluateHostAgents` expires the lease, then collected again. - `TestUnraidDiskTemperatureFollowsTheHostLease`: reporting, silent and spun-down hosts across the native row, the borrowed reading and a SMART row with its own reading, including risk. - `TestUnraidInventoryRowDoesNotMaskTheSMARTRowState`: the merged-registry case, which fails if the native row claims `available`. - `TestTemperatureCollectedOnlyForAReadingTakenByThisObservation`: the helper. |
||
|
pulse-triage[bot]
|
30202e235f | Merge branch 'main' into fix/node-drawer-temperature-records | ||
|
rcourtman
|
f4c1046215 |
Record Docker host, service and sub-resource alert history
Alert lifecycle events resolve to canonical resources since Proxmox and
agent references gained durable resolution, but Docker host alerts
("docker:<host ID>"), Swarm service alerts ("docker:<host ID>/service/<ID>")
and sub-resource alerts such as ZFS pool and device health
("<storage ID>/zfs-pool:<pool>") or host filesystem usage
("agent:<host ID>/disk:var") were still journaled under references no
resource answers to. Resource facets, timelines, resource intelligence and
Assistant context read history by canonical ID, so they never saw them.
Docker host and service references now resolve through the registry's own
Docker source identities. A service reference must carry the service's real
ID in the host's Swarm cluster, from any manager that reports it; services
without an ID are referenced by name and never bind. Container names and
shortened IDs still bind nothing and are no longer retried against later
inventory.
Sub-resource references bind to the owner whose durable ID they carry: the
storage for ZFS pools and devices, the Unraid array storage, and the host
for filesystem, SMART disk, disk temperature, RAID and custom-sensor
alerts. Mount and kernel device labels can name a different disk after a
reboot, so they never bind a physical disk. Any other shape stays unbound.
On a mock stack none of the Docker host, Swarm service, ZFS and agent disk
alert rows were readable through their owner's timeline API before this
change; all of them are after it, including rows journaled earlier. The
mock's shortened container ID rows stay unbound.
|
||
|
rcourtman
|
1b081c6475 |
Record Proxmox and agent alert history on the canonical resource (#2608)
Alert lifecycle changes (fired, acknowledged, snoozed, resolved) are
journaled with the alert's source-native resource ID.
`MonitorAdapter.RecordChange` only mapped full Docker container IDs, so
Proxmox node, guest and storage alerts and host-agent alerts landed
under IDs such as `instance-node`, `instance:node:vmid` or
`agent:<host>`. Resource facets, resource intelligence
(`recent_changes`) and Assistant resource context read history by
canonical ID, so they never showed those alerts. Incident timelines were
unaffected because they read by alert identifier.
**Change**
- `RecordChange` resolves non-Docker references to the canonical
resource and records a history binding, so the reference keeps reaching
the same history after a guest migration or inventory removal.
- Only durable identities bind: canonical ID, retired era, source ID,
node-scoped guest reference, `agent:<host ID>`. Names and hostnames
never bind (a VM named `pulse-system` cannot capture system alerts). A
reference that two resources answer to binds to neither. Docker keeps
its full-ID rule.
- Rows already journaled under a source reference join canonical history
without being rewritten. Each registry generation binds the unbound
alert references it can resolve, both those found at startup and those
written before inventory named the resource, retrying each for 30
minutes.
**Evidence** (private mock stack, same data dir, origin/main first, then
this branch)
- main: 51 of 52 alert references were source-shaped and there were zero
bindings. `GET /api/resources/{id}/facets` for affected nodes and guests
returned only `state_transition`.
- this branch: the first registry generation logged `bound 17 alert
history references`. Every non-Docker whole-resource reference is now
bound: node metric and connectivity alerts, guests, Proxmox storage,
`agent:host-linux-1` and `pmg-main`. Facets and
`/api/ai/intelligence?resource_id=` now include
`alert_fired`/`alert_resolved` for the VM, node and storage checked.
- Still unbound, as before: ZFS pool/device and agent-disk sub-resource
references (no resource of their own) and Docker host/service/short-ID
references.
**Where it shows**
In the UI, only the full-presentation `ResourceDetailDrawer` requests
this history. Proxmox node and Agents rows open the `table-row` drawer,
and guest and storage rows open their own drawers; none of these fetch
it (confirmed live: the node History tab makes no facets or intelligence
request). So the user-visible gain is in Assistant and Patrol resource
context and the facets and intelligence APIs, not in a Proxmox drawer.
**Tests**
`TestHistoryIdentityMonitorAdapterResolvesProxmoxAlertReferences` and
`TestHistoryIdentityBindsLegacyAlertRowsFromRegistryGenerations`
(registered `history_identity_test.go`), plus
`TestProxmoxAlertTimelineUsesCanonicalHistoryIdentity` through the real
alert-manager callback. The monitor test fails on main. Mutation checks
confirm the new tests catch alias-based resolution, a missing post-scan
retry and ignored ambiguity.
|
||
|
rcourtman
|
41b94b29a8 |
Keep a merged disk's temperature under its own row's state
The registry merges a physical disk's rows (Proxmox, host agent SMART, Unraid inventory), choosing the temperature by source preference and merging field states separately. When two rows carry different temperatures the shown value could sit under the other row's "available" state: a silent agent's retained reading carried over a Proxmox row that collected its own reading now, in either ingest order. Risk, history and the UI then treated a frozen reading as current. pairPhysicalDiskTemperatureState runs at the end of each merge step: when the merged state says collected but the row the shown value came from says it was not, the merged state becomes that row's own. It never changes a value and never grants availability, so it cannot revive a withdrawn reading or override a resumed agent's fresh one; rows with the same value keep the merged state. This is exact for the two rows of a step. The merged disk keeps one state per field, so with three or more rows a withdrawal an earlier step replaced is invisible to a later one; closing that needs each reading's own state tracked across merges. Preferring another source's current value instead would need an ordering between rows that the merge does not record. With this in place the Unraid-native row no longer needs to withhold an "available" state to avoid masking a SMART row, so it now carries the same derived state as a SMART row borrowing its reading. |
||
|
rcourtman
|
8c9000ac63 |
Keep node temperatures off same-named nodes at other sites
Two Proxmox connections can each have a node with the same name, which is common for MSPs running one px1 per site. The node temperature lookup fell back to the first host agent whose hostname matched the node name, even when that agent was linked to the other site's node, and even when the name was ambiguous enough that the agent linker had refused to link it. Site B's px1 then showed, alerted on and recorded history for site A's sensors. The cluster sensor cache leaked the same way: an agent's SSH readings for its cluster siblings were keyed by bare node name, so a site A sibling reading fed site B's node. The poller now hands the lookup the polled node, and the lookup works on that node's slot. The state folds the views of one machine reached through two connections (a cluster added twice, or a multi-homed host added by each address) into one node held under one view's ID, and the agent link follows it. Those views used to find their agent only through the loose hostname match, so a polled view the read state does not hold also takes the one same-named node proven to be the same machine. models.NodeObservationsSameMachine and models.HostAgentBridgesNodeViews give that proof from the evidence UpdateNodesForInstance folds on, and the state's agent corroboration check now shares its rule with them. Ambiguous candidates are declined, as the state declines an ambiguous alias. An agent linked to the slot is the node's agent. The hostname fallback only takes an agent with no link when no node outside the slot and no second unlinked agent has the name. Cluster sensor cache entries record their reporting agent and serve a node only when that agent's linked node shares one of the slot's connections, or belongs to a connection whose configured view of the node has a matching TLS fingerprint. LinkedNodeID holds the Proxmox source node ID for automatic, manual and continuity-restored links, so the link match needs no normalization. The physical-disk SMART merge already matches agents by node.LinkedAgentID within one instance and is unchanged. |
||
|
rcourtman
|
3ac5664d95 |
Key previous Proxmox nodes by source ID so poll carry-overs match (#2604)
The Proxmox poller reads its previous poll's nodes back from the unified read state, but `previousNodeFromView` returned the registry's unified resource ID (e.g. `agent-ab60dcc1ebca813b`) instead of the poller's source ID (`pve1-node1`), and dropped the temperature. Registry-backed probes confirmed every lookup keyed on the source ID missed: - `nodeLastOnline` in the offline grace policy; - the temperature carry, plus the CPU low/record tracking, which reset to the current reading every poll. The dev stack shows it live: `delly` and `minipc` report `cpuMin = cpuMaxRecord = cpuPackage`; - the node memory carry after a failed status read. When a whole instance poll failed, `markPVEInstanceNodesUnreachable` wrote the projected nodes back, so every failed poll re-ingested the node under a new unified ID (`agent-ab60…` → `agent-aec93…` → `agent-e798…`). The projection also dropped the cluster name the canonical ID is derived from, so cluster members changed ID for the length of an outage. ### Change - The projection returns the source ID, the stored temperature details, the cluster name and the linked agent ID. `previousNodesForInstance` restores the config-derived identity stamps (provider-scoped flag, native aliases, TLS fingerprint) the same way `pollPVENode` writes them. Reviving those paths needed bounds: - **Memory carry** uses the poller's own validated `NodeMemorySnapshot`. The read state holds the agent's reading on a node merged with a host agent. used + cache + free still sum to the total. - **Agent lease:** a carried temperature stamped no later than a lapsed host agent's last report is not carried. Otherwise the 5-minute carry window would hold a silent 30s agent's reading in alerts and history about 2 minutes past the lease that #2582 enforces. Readings record no source, so an SSH reading of that age is dropped too; the bound errs toward a gap, never a stale value. - **No disk readings in a carry:** a carried reading keeps CPU/GPU only. SMART/NVMe rows also stamp physical disks, so a disk that went to standby would otherwise be stamped and recorded with its pre-sleep value. - **Preserved nodes:** outage preservation and authoritative membership reconciliation follow the same rules, and drop the temperature when the node goes offline. ### Tests - New registry-backed tests in `monitor_additional_test.go` cover: CPU low/record tracking across polls, outage writeback keeping the unified ID (including provider-scoped clusters), membership writeback, the lease rule, disk rows excluded from carries, and the memory carry on an agent-merged node. Each fails against the old projection or without its bound (checked with `go test -overlay`). - `TestPollPVEInstanceMarksStaleNodesOfflineWhenGetNodesReturnsEmpty` relied on the dead `nodeLastOnline` lookup; it now ages that sighting too. - The silent-agent test from #2582 now rebuilds the store via `PopulateFromSnapshot` and feeds the polled node back, as production does. Re-ingesting into one registry keeps the first `TemperatureDetails`, so the old setup never exercised the carry. Follow-ups found on the way: same-named nodes on different sites leaking agent temperatures (separate PR stacked on this one), and the node drawer never reading `temperatureDetails` (#2594). |
||
|
rcourtman
|
844e1dbb97 |
Say a reviewed Patrol issue stays reviewed when suppression ends
Ending a suppression puts the alert in Acknowledged when it is acknowledged and in Open otherwise (unsuppressOperationalAlert), and Acknowledged is not active attention. The attention detail still promised otherwise for an issue marked reviewed and then suppressed: its early-end button read "Return to active attention", the handled-view notice read "Returned to decision inbox", and the suppression form offered "Return it to active attention after", while the issue went back to Reviewed. The duration label and the early-end button now follow the detail's operationalRecord.acknowledgement, which the backend derives from alert.Acknowledged: "End suppression after" and "End suppression" for a reviewed issue, the existing copy for an unreviewed one. The notice reads the item's state from the reload that follows the change instead of that snapshot, so it reports "Suppression ended. It stays reviewed." only when the item really is Acknowledged, even if the review was removed elsewhere after the detail loaded. Contract-Neutral: No api-contracts or frontend-primitives delta: Patrol lifecycle copy only, no API payload or shared primitive change; patrol-intelligence.md carries the note |
||
|
rcourtman
|
2265d28242 |
Remove the never-mounted unified resource table
PlatformResourceTable was the only renderer of UnifiedResourceTable, and nothing imports it. Every platform page (Proxmox, Docker, Kubernetes, TrueNAS, vSphere, Machines) renders its own native tables, and /infrastructure only shows the opening-workspace placeholder. A production build of main already tree-shook the whole cluster (none of its strings ship), yet it still carried 11 source files, 7 test files, registry ownership in two subsystems and contract rules for a table nobody sees. Removed: PlatformResourceTable and its PLATFORM_STATUS_FILTER_OPTIONS; UnifiedResourceTable with its host, service, PBS and PMG cards, source badge cell, state hook, viewport sync and models; and ClusterDeployBanner, which only the host card rendered (the live cluster deploy flow is the Settings ClusterAgentDeployDialog). Their tests go with them. Coverage for live code those tests also exercised is kept: the detail mapper no-fetch boundary and the online boolean sensor case move to resourceDetailMappers.test.ts, the ResourceFacetSummary render cases move to their own test file, and both files become registered proofs for the policies that govern their subjects. Guardrail tests, the shared template registry, registry.json, status.json evidence and the performance, unified-resources, frontend-primitives and cloud-paid contracts drop the deleted paths and the rules that only governed them. Rollup used to name the large shared chunk after sharedPlatformPage; it now names it MetricBar, so the bundle baseline entry is re-keyed at its measured size. |
||
|
rcourtman
|
cdae811bb7 |
Chart Net and Disk I/O in the row history lens
Hovering a guest row swapped CPU, memory and disk for history sparklines but left Net I/O and Disk I/O as static numbers, although they are the jumpiest readings in the row and the hover request already returns their series. The lens now charts all five cells on one synchronized cursor. I/O charts, on hover and in Trends, keep a compact current pair beside the chart (↓3.3M ↑2.2M, R1.4M W826K) instead of hiding the rates in a tooltip; the glyphs use the series colours so the pair is also the legend. Rate sparklines get a 100 KiB/s scale floor so an idle guest's few-KB/s chatter no longer draws the same full-height spike as a busy one. GuestRow now builds every sparkline from accessors. Each live snapshot used to remount the lens and Trends charts, dropping an open scrub tooltip; an open tooltip also re-reads the current series instead of holding the values from the last mouse move. |
||
|
rcourtman
|
28d5f44257 |
Judge disk risk only on a temperature collected now
Since #2593, a disk whose host agent stops reporting keeps its last SMART temperature as a last-known value marked unavailable, and a disk in standby keeps its pre-sleep reading the same way. The shared disk risk assessment still fed that value into its temperature rule, so a silent agent's frozen 72C made the disk read "Replace Now: Disk temperature is 72C" with a warning status, while the temperature cell beside it now says the reading is not current. storagehealth.CollectedTemperature passes a temperature through only when its collection state is available, or when the source predates collection state. AssessPhysicalDisk, AssessHostSMARTDisk and the registry's recompute from merged metadata use it, so a retained reading stays on the resource as history but adds no temperature_high reason and leaves the disk online. Health, wearout and SMART counters are unchanged: they are durable evidence and keep judging as before. |
||
|
rcourtman
|
e25472feab | Merge remote-tracking branch 'origin/main' into fix/metric-alert-live-status | ||
|
rcourtman
|
95b29286dd |
Stop serving retained disk temperatures as current readings
Normalization may keep a disk temperature it did not collect (a disk in standby, a host agent past its reporting lease) as long as its collection state says so. Two consumers still presented that value as current. The metrics-history handler answered an empty disk range with the disk's temperature as a "live" point at now, regardless of its state. Since the history writers stopped recording retained temperatures, a short-range chart for a silent agent's disk or a sleeping disk showed the retained value as a fresh reading. Mock mode also seeded synthetic series from it and padded sparse stored series out to now. All of these now require a temperature collected now (diskinventory.TemperatureCollected, the shared form of the rule the history writers apply). Unraid array-inventory rows carry no per-field provenance, so a temperature taken from one looked collected after the agent went silent. The adapter now derives its state from the host lease that ExpireHostTelemetry already records, and from the disk's spin state: unavailable once the host is offline or the disk is spun down. A SMART row that borrows the inventory reading takes that state, including available. The native row records only the withdrawing states, because the registry merge picks temperature and state separately and an available claim from it would mask the SMART row's own state. A withdrawn inventory temperature no longer counts as heat in disk risk. |
||
|
rcourtman
|
ba26b3581e |
Settle slow polled reads and keep replication links honest
A review of #2578 found three gaps it left. The shared poll in createNonSuspendingQuery replaced any read still in flight on every tick. The API answers 429 with Retry-After: 60 and apiFetch waits it out, so under rate limiting a polled query slower than its 30-second interval never settled. Replication reads also ignored the abort signal, so each replaced read kept running. A poll now skips its tick while a read an earlier poll started is still in flight; a refresh, source change or remount revalidation stays replaceable at the next tick as before. A polled read in flight for 150 seconds is presumed stalled and replaced, which also clears any spinner it held. Replication reads forward the signal to apiFetch. A failed first replication read made the job count zero, so a direct /proxmox/replication link fell to Overview without its error or Retry. Replication presence is now unknown until a read succeeds, and a failed poll after that keeps the last answer instead of reopening the route. The shared clock returned its last tick, so a cell mounted between ticks measured from up to 30 seconds ago and the desktop backup age could disagree with its band. Every read now returns the wall clock and the tick only schedules re-reads, which retires the backup Math.max workaround. The #2578 age tests now jump the clock and fire one tick instead of looping hundreds of re-renders, which timed out under load. |
||
|
rcourtman
|
474dbc4f09 |
Record Proxmox and agent alert history on the canonical resource
Alert lifecycle changes carry the alert's source-native resource ID, and MonitorAdapter.RecordChange only mapped full Docker container IDs. Proxmox node, guest and storage alerts and host-agent alerts were journaled under IDs such as "instance-node", "instance:node:vmid" or "agent:<host>", while resource facets, resource intelligence and Assistant resource context read history by canonical ID, so those surfaces never saw the alerts. Resolve such references to the canonical resource and record a history binding, so the reference keeps reaching the same history after a guest migration or inventory removal. Only durable identities bind: the canonical ID, a retired era, a source ID, a node-scoped guest reference or an agent host ID. Names and hostnames never bind, so a VM named "pulse-system" cannot capture system alerts, and a reference two resources answer to binds to neither. Docker keeps its full-ID rule. Rows already journaled under a source reference join canonical history without being rewritten: each registry generation binds the unbound alert references it can resolve, those found at startup and those written before inventory named the resource, retrying each for 30 minutes. |
||
|
rcourtman
|
9aaac4d3e5 |
Read a linked agent's own heartbeat in host snapshots (#2601)
A host agent linked to a Proxmox node merges into one unified host row. When the agent stops reporting but PVE polling continues, that row stays online and its `LastSeen` follows the PVE sighting. `hostFromReadStateView` copied both into `models.Host`, so a silent agent looked current to every consumer of the projection: - `linkedHostForNode` accepted it, and `resolveNodeDisk` kept using the agent's last filesystem summary as the node's disk, feeding disk alerts and history with a frozen reading. Its retained ZFS datasets kept merging into the node's pools. - Agent connections built from `HostsSnapshot()` (the Connections list and the update-readiness agent check) aged the agent from the PVE poll, so a dead agent stayed Active. The projection now takes `LastSeen` from the agent source's own sighting (`HostView.SourceStatus(SourceAgent)`, added in #2582) and reports the agent offline once that sighting is past its reporting lease (`hostAgentReportCurrent`, the same `hostAgentHealthWindow` that `evaluateHostAgents` applies). Rows without an agent sighting, such as vSphere and TrueNAS hosts, keep the row's values. Verification: - New `TestSilentLinkedAgentStopsFeedingNodeDisk` (registry-backed merged agent + node row, driven through `evaluateHostAgents` and the store refresh): node disk uses the agent summary while it reports, falls back to the Proxmox rootfs past the lease, and returns when the agent reports again; the projected agent reads offline with its own last report. It fails against the previous `monitor.go` (checked with `go test -overlay`). - `go test ./internal/monitoring/` and the connections, readiness, update and agent-lifecycle tests in `./internal/api/` pass. |
||
|
rcourtman
|
f28f73b28f |
Stop presenting a silent agent's disk readings as collected (#2593)
A host agent that stops reporting keeps its last SMART rows in state.
Its frozen disk readings then kept being treated as current:
- `mergeHostAgentSMARTIntoDisks` copied the silent agent's temperature
and I/O counters into the linked Proxmox physical disk, along with its
`available` collection state.
- The registry merged the agent's retained SMART rows into the canonical
disk the same way. `MergeStatus` lets any `available` state win, so the
API kept reporting the readings as collected.
- Every full disk poll (default every 5 minutes) wrote the frozen
temperature to SMART history as a new sample.
## Change
- **Lease expiry is the cut-off.** `State.ExpireHostTelemetry` (run by
the offline sweep once `hostAgentHealthWindow` passes) keeps SMART
temperature and I/O as last-known values but marks them `unavailable`
("host agent stopped reporting"). This follows the physical-disk
retained-evidence clause in the monitoring contract. The agent's next
accepted report replaces them.
- **A source can withdraw its own evidence.** New
`diskinventory.MergeReportedStatus` lets a source's later non-available
statement supersede an `available` state that another row still carries
from the same source.
- Used by the monitoring merge, where skipped disk polls re-merge the
previous state.
- Used in both registry ingest orders. This covers a PVE host that is
down entirely: it is never disk-polled again, so its Proxmox disk row
stays frozen.
- **History records only collected temperatures.** All SMART temperature
history writers (`writeSMARTMetrics`, `smartMetricStoreWrites`,
`writeHostSMARTMetrics`, and the unified disk sync via its collection
passthrough) skip a temperature whose current state is not collected.
This also stops a standby disk's retained pre-sleep reading from being
recorded.
- **Pre-6.2 agents keep their history.** These agents send no collection
provenance, so the merge now records their temperature as available from
`host_agent` instead of inheriting Proxmox's `unsupported`.
- **Expiry is compare-and-set** on the `LastSeen` the sweep judged
stale, so a report accepted in between is never expired. The sweep's
alert-side offline handling for that race is unchanged; that race
already existed.
Deliberately unchanged:
- SMART attributes, wearout and health stay last-known. They are durable
health evidence, and dropping them would resolve disk-health and
low-life alerts whenever an agent goes quiet (cf. #2112).
- Disk temperature alerts are evaluated in `CheckHost` from the agent's
own report, only when a report is ingested, so they never read this
merged value.
## Related
- The node-temperature path (agent sensors → `node.Temperature.SMART`,
which the disk merge labels `proxmox_node_smart`) is gated on the
agent's own sighting in #2582. A carried node reading can still stamp a
disk as collected; that path belongs to the node poller.
- Filed separately:
- the chart API's `liveMetricPoints` returns a retained disk temperature
as a single "live" point when a range has no samples;
- Unraid-native disk temperatures carry no collection state;
- the disk table renders non-collected temperatures with live threshold
colouring;
- a merged agent+node row reads `online` after its agent dies, so
`linkedHostForNode` keeps using the silent agent's filesystem summary.
## Tests
- `TestSilentLinkedAgentSMARTIsRetainedButNotPresentedAsCollected`
(registered monitoring proof `physical_disk_roundtrip_test.go`). Uses a
registry-backed monitor with a real disk poll and a metrics store. A
reporting agent's reading is collected and recorded. After lease expiry,
the canonical disk reads as not collected before any disk poll runs, and
again after a skipped poll and a full poll. The value is retained and no
new history sample is written. A resumed agent is collected and recorded
again. Fails on origin/main. Mutation-checked: removing the expiry
marking, the monitoring supersede, the registry supersede, or the
history gate each makes it fail.
- `TestMergeHostAgentSMARTIntoDisks_LegacyAgentTemperatureFollowsLease`
- `TestPhysicalDiskMergeFollowsAgentWithdrawalOfItsOwnReadings` (both
ingest orders)
- `TestMergeReportedStatusLetsTheSourceWithdrawItsOwnAvailability`
- `TestExpireHostTelemetryMarksSMARTReadingsNoLongerCollected`
- `TestExpireHostTelemetryLeavesHostThatReportedSinceJudgement`
- `TestWriteHostSMARTMetricsRecordsOnlyCollectedTemperatures`
Ran the full `internal/monitoring`, `internal/unifiedresources`,
`internal/models` and `pkg/diskinventory` packages locally; all pass.
|
||
|
rcourtman
|
0c130a4855 |
Key previous Proxmox nodes by source ID so poll carry-overs match
The poller reads the previous poll's nodes back from the unified read state, but previousNodeFromView returned the registry's unified resource ID and no temperature. Every lookup keyed on the Proxmox source ID missed: nodeLastOnline in the offline grace policy, the temperature carry, the CPU low/record tracking (reset to the current reading every poll) and the node memory carry. When a whole instance poll failed, markPVEInstanceNodesUnreachable wrote the projected nodes back, so each failed poll re-ingested the node under a new unified ID. Cluster members also lost the cluster identity their canonical ID derives from, so they changed ID for the length of an outage. The projection now returns the source ID, the stored temperature details, the cluster name and the linked agent ID, and previousNodesForInstance restores the config-derived identity stamps (provider-scoped flag, native aliases, TLS fingerprint) the way pollPVENode writes them. Reviving those paths needed four bounds: - The node memory carry uses the poller's own validated snapshot rather than the read state, which holds the agent's reading on a node merged with a host agent, and keeps used + cache + free equal to the total. - A carried temperature keeps a lapsed host agent's lease. A reading stamped no later than the agent's last report is not carried, so the five-minute carry window cannot hold a silent agent's reading in alerts and history past its lease, which #2582 guarantees. - A carried reading keeps the node's CPU and GPU temperatures only. Its SMART and NVMe rows also stamp the node's physical disks, so a disk that went to standby would be stamped and recorded with its pre-sleep value. - Outage preservation and authoritative membership reconciliation keep a node's temperature only under those rules and drop it when the node goes offline. TestPollPVEInstanceMarksStaleNodesOfflineWhenGetNodesReturnsEmpty relied on the dead nodeLastOnline lookup and now ages that sighting too. The silent-agent test from #2582 now rebuilds the store as production does and feeds the polled node back, which is what exercises the carry. |
||
|
rcourtman
|
d26a483731 |
Show a moved alert as moved in Patrol, history and assistant (#2591)
Follow-up to rcourtman/Pulse#2580. A node alert handed to its Pulse agent closes with an `AlertResolution` (`moved_to_agent`), and notifications already say it moved. Every in-app reader still treated that close as a recovery: - Patrol's attention detail showed a green **Resolved** badge, and its timeline read "Open to Resolved · Recovery evidence" directly above "…This is not a recovery…". - The Alerts history listed the row as **resolved**, and its incident timeline ended "Alert resolved: Node memory at 95.0%". - `GetRecentlyResolved` rebuilt `models.Alert` without the resolution. So the state snapshot, websocket, assistant context, Patrol's seed prompt and the `pulse_alerts` `resolved` action all saw a plain resolve. **Operational trust.** New `TransitionCause` `ownership_transferred`. It may only enter `resolved`, and only with evidence. `markOperationalResolved` uses it whenever the alert carries a `moved_to_agent` resolution; every other close stays on `recovery_evidence`. Alert-quality telemetry keys "resolved while snoozed" on `recovery_evidence`, so a handover during a snooze no longer counts there. No notification code reads the cause: links key on the transition ID and state. **State and AI.** `models.Alert.Resolution` holds the reason, successor and the engine's `Summary()`. `cloneAlert` copies it, and `GetRecentlyResolved` fills it. - Patrol's seed writes `closed 3m ago. Alert moved to pve1 (Host Agent). This is not a recovery: …`. - The assistant's context writes `closed … ago) <summary>`. - The resolved-alerts tool returns `resolution` and `successor_resource_id`. **Timelines.** The `alert_resolved` resource change carries the summary as its reason and the code as `alert_resolution` metadata. Incident memory titles the close with it, both when projecting from the canonical timeline and on its own write path. **UI.** - The history row reads **moved to agent**, even if it was acknowledged. The full account is in the badge title and is printed under the message on the phone card. - Patrol's detail shows a **Moved** badge (info tone), with the handover sentence directly under the summary rather than only inside Evidence and history. Its timeline says "Ownership transferred". **Landing order.** rcourtman/Pulse#2586 is the only producer of a handover, and it has no auto-merge. It should land after this PR (or merge main once this is in). Any handover recorded before this lands would otherwise keep a recovery-labelled transition and timeline. **Not covered here:** - Performance reports still list a handover from the last 5 minutes as resolved. - The resource drawer labels the change "Alert resolved" before its reason. - The incident header in the history expansion still shows its open/resolved status word. Both of the first two are filed as a follow-up and noted in ai-runtime.md and unified-resources.md. **Tests:** - `TestOwnershipTransferOnlyClosesWithEvidence`. - The cause assertions in `TestReleasedMetricAlertCarriesResolutionInsteadOfRecovery`. - `TestRecentlyResolvedCarriesHandoverResolution`. - `TestAlertResolutionWireContract`, which fails without the `cloneAlert` copy. - `TestMonitor_HandleAlertLifecycle_HandoverCloseIsNotARecovery`, which fails on the old code with "Alert resolved: Memory usage at 95%". - `TestIncidentStore_HandoverCloseIsNotARecovery`. - `TestSeedHealthAndAlerts_HandoverIsNotARecovery`. - `TestService_buildAlertContext_HandoverIsNotARecovery`. - `TestExecuteListResolvedAlertsReportsHandover`. - Frontend history, presentation and Patrol workbench tests. The operationaltrust, alerts, models, ai, ai/tools, ai/memory, unifiedresources, monitoring and notifications suites pass. `tsc`, eslint and the vitest sweep pass, apart from the Mac-only `agentInstallCommand` PTY test. **Browser pass.** A private non-mock stack ran on an alert history written by the real `releaseCanonicalMetricAlert` handover path. I checked Alerts history (row, tooltip, expanded timeline) and Patrol's deep-linked detail at 1280 and 375: no overflow, no console errors. |
||
|
rcourtman
|
779b3d4ff8 |
Tint Kubernetes node rows for their drawer's alerts (#2596)
A Kubernetes node that runs a Pulse agent is an agent row, and the agent's alerts are keyed `agent:<id>` and labelled with the hostname. The nodes table matched only the row id and display name, so on a mock estate "Prod Euw1 K8s 02" sat at 100% memory with a critical alert, its drawer listed "Agent memory at 100.0%", and its row was untinted. The overview's "Nodes needing attention" looked only at readiness and cordons, so it named K8s 03 (Degraded) and left K8s 02 out. - Kubernetes node rows tint from `getUnifiedResourceAlertStyles` (#2579), the set the drawer reads, and drop the tint while expanded like the Docker host and Machines rows. - `getKubernetesNodeAttentionRank` ranks a node by the stronger of its state and its unacknowledged open alerts: NotReady and critical alerts first, then cordoned or degraded nodes and other alerts. The overview list and the nodes table's default order both use it, so a critical node no longer sorts below a degraded one. Acknowledging the alert drops the node out of both, as it already drops the tint. - vSphere host and TrueNAS system rows are deliberately unchanged. Checked against the producers: their VMs', pools' and apps' alerts carry the host name as their node (VM `ParentName` is the runtime host; registry children take the parent's name), the TrueNAS drawer states the same pool problems, and `agent:` alert ids only come from Pulse agent reports (`internal/alerts/host.go`, `external_probe.go`). A Pulse agent on a TrueNAS box stays a separate resource (`TestUnifiedDiskWriteMetricsKeepSameHostnameHostAgentAndTrueNASTargetsSeparate`). Switching them would drop the pool rollup that tints the TrueNAS row for a degraded pool, so a test now pins it. Verified in a browser on a private mock stack (receipt `frontend-modern/browser-verification/83848f2d5ef2a326.json`): Nodes and Overview at 1440px, 390px and in dark mode, the drawer, and acknowledge then unacknowledge of the K8s 02 alert. The new Kubernetes tests fail against the previous runtime and pass with this change. |