5993 Commits

Author	SHA1	Message	Date
rcourtman	4c4e6e7c57	Call a disk that is only hot Running Hot, not Replace Now (#2623) ... On Physical Disks, a SMART-passed disk with good life running at 72°C read **Replace Now** ("Disk temperature is 72C"), and **Replace** on phones. Disk risk grades temperature (`temperature_high`, ≥70°C critical, ≥60°C warning), and the Health verdict mapped any critical risk to Replace Now. That tells a homelab user to buy a disk when the fix is airflow. Alerts already keep heat separate: the SMART health and wearout alerts skip `temperature_high`, and disk heat is raised as a Temperature metric. ## Change - `diskPresentation.ts` keeps each risk reason's code and severity, and splits the reasons into heat and everything else. - FAILED health, any other critical reason, or a critical level that no reason explains → **Replace Now** (unchanged). - Heat alone → **Running Hot** (phone **Hot**). Red at the critical tier, amber at warning. The temperature reason is still the cell title and the drawer summary. - Other warning evidence, SMART counters or low life → **Needs Attention** (unchanged). - The more severe class names the verdict, and failure evidence wins a tie. Merged risks that list heat first (`mergePhysicalDiskRisk` does not re-sort) no longer hide pending sectors behind a temperature summary. - Sort priority and the health filter derive from the same verdict. Order is Replace Now, then red Running Hot, then Needs Attention, then amber Running Hot. Hot disks stay in **Needs attention** and land in Critical or Warning by their tier. - Data with no reason codes keeps the old behaviour. ## Not changed Thresholds. The tier still comes from the server's flat 60/70°C rule, not the per-type alert thresholds that colour the Temp cell. An NVMe at 60–64°C therefore reads amber Running Hot next to a green Temp cell. That disk read Needs Attention before this change, so the judgment is the same and only the word is clearer. Reconciling the thresholds is being done separately. ## Verification - New tests in `diskPresentation.test.ts` cover heat at both tiers, failure evidence listed after heat, heat plus wear, an unexplained critical level, sort order and the health filter. Five of them fail on the old code, including the sort test, whose device names make name order alone disagree. `DiskList.test.tsx` checks the full and compact words, tone, title and row order. - Targeted suites (storageBackups, Storage, GuestPhysicalDisks, frontendResourceTypeBoundaries): 89 files, 1513 tests pass. tsc and eslint are clean, and the canonical platform audit is clean. - Browser check on a private mock stack (local, uncommitted generator overlay pinning pve3 disks at 72°C, 63°C, and 72°C with 2 pending sectors), at 1440×900 and 390×844. The rows read Replace Now / Running Hot / Needs Attention / Running Hot (phone: Replace / Hot / Attention / Hot) with the expected tones and titles. The drawer reads "Running Hot · Disk temperature is 72C". The `status=attention|critical|warning` filters list the expected rows. A same-session control with the old file showed the 72°C disk as Replace Now. The receipt is `frontend-modern/browser-verification/259ae9ced29e073f.json`.	2026-10-07 00:47:36 +01:00
rcourtman	8732bc1923	Remove the unreachable host metric cells from grouped workload rows (#2622) ... Both production mounts of the Workloads surface, the Proxmox and vSphere overviews, set `compactGroupHeaders` unconditionally. The non-compact branch in `WorkloadPanel` (per-column node CPU, memory, disk and I/O bars or sparklines, uptime, version, temperature) and the `NodeGroupHeader` column mode and inline host facts that only it reached never rendered. They were left over from the retired standalone Workloads page. This removes the option, the dead renderers and the `NodeGroupHeader` props that only served them. Grouped host rows keep the shape both pages already render: one spanning cell with status, linked name, cluster and agent badges. Host stats stay in each page's own hosts table. Verification - No other production consumer: `WorkloadsSurface` → `WorkloadsTable` → `WorkloadPanel` is entered only from `ProxmoxPageSurface` and `VmwarePageSurface`; no references in pulse-pro, pulse-enterprise or pulse-mobile. - Browser pass on a production build embedded in a private mock backend, `/proxmox/overview` and `/vmware` at 1440x900 and 390x844, Bars and Trends modes: every grouped host row is one cell spanning the guest table's visible columns, with no metric cells or host facts; group-row clicks open nothing; guest drawers still open; no page errors. - The rewritten `NodeGroupHeader` test fails against the old component (its default facts render `PVE 9.1.9`). - `tsc`, full `npm run lint`, vitest over Workloads, shared, features, hooks, utils, Storage and app tests (only the Mac-only `agentInstallCommand` failures, unrelated), bundle budgets, and all release-control guards pass.	2026-10-06 23:42:36 +01:00
rcourtman	8424fd6505	Show real CPU low and record in the Proxmox node drawer (#2594) ... The Proxmox node drawer's **CPU low** and **CPU record** rows repeated the current temperature for every node without a host agent. `nodeFromResource` never read `proxmox.temperatureDetails`, which is where the resource API publishes the poller's full reading, including its low and record values. For those nodes it fell back to the scalar `proxmox.temperature` and copied that one value into both fields. On the dev stack, `delly` and `minipc` showed CPU low = CPU record = current. - The adapter now takes the first source with a usable reading. `temperatureDetails` comes after the platform record and the agent facet, so agent-linked nodes render exactly as before. - The bare-reading fallback leaves low and record unset, so the drawer omits those rows rather than inventing them. The poller also reset those values to the current reading every poll. That is a separate backend fix to the previous-node projection, landing in its own PR. Until it lands, PVE-only nodes will still show low = record = current, now because that is what the backend reports. ### Verification - Unit tests pin the API's real payload shape (scalar `proxmox.temperature` plus `proxmox.temperatureDetails`), the no-record fallback, and agent-facet precedence. The new test fails against main's adapter, which returns `cpuMin` 58 = current. - Browser pass on a private mock stack at `/proxmox`, 1440x900 and 390x844, with REST and websocket payloads rewritten to make one node PVE-only: - main's adapter showed CPU low 82°C / CPU record 82°C; - this change shows no rows without a record, and 39°C / 91°C with one; - agent-linked nodes and the table TEMP column are unchanged. - `vitest` over `src/utils`, `src/components/Workloads`, `src/features/proxmox` and `src/features/platformPage` passes, apart from `agentInstallCommand.test.ts`, which fails on macOS regardless of this change.	2026-10-06 23:21:38 +01:00
rcourtman	7b362b3532	Load guest physical disks with the drawer to keep Workloads in budget ... GuestDrawerOverview statically imported GuestPhysicalDisks, which pulled all of diskPresentation.ts into the WorkloadsSurface table chunk. With the Running Hot verdict that chunk measured 70.64 kB gzip against its 70.23 kB cap, so the Frontend bundle check would fail. The card renders only inside an open drawer for a guest with a Pulse agent, so the overview now lazy-loads it under a local Suspense, the same pattern as the bulk web-link editor. WorkloadsSurface drops to 64.31 kB and the disk presentation builds as its own chunk. The contract test pins the dynamic import.	2026-10-06 23:16:04 +01:00
rcourtman	d9ffb39ce8	Keep a merged disk's temperature under its own row's state (#2618) ... Stacked on #2609; until it merges this branch also carries its commit, and the change here is the last commit. ## Why The registry merges a physical disk's rows (Proxmox, host agent SMART, Unraid inventory), choosing the temperature by source preference and merging collection states separately. When two rows carry different temperatures, the shown value could sit under the other row's `available` state. A real case: a node whose disk temperature Proxmox collects now (`proxmox_node_smart`, 40) and whose linked host agent has gone silent (retained 72, `unavailable`). In either ingest order the merged disk showed **72 labelled available**, so risk, history and the UI treated the frozen reading as current. ## What - `pairPhysicalDiskTemperatureState` runs at the end of each merge step. When the merged state says collected but the row the shown value came from says it was not, the merged state becomes that row's own. - It never changes a value and never grants availability, so it cannot revive a withdrawn reading or override a resumed agent's fresh one. - Rows with the same value keep the merged state, so the agent's own withdrawal still supersedes the Proxmox copy (#2593). - With that guarantee the Unraid-native row no longer needs to withhold `available` (#2609 restricted it), so it carries the same derived state as a SMART row borrowing its reading. ## Known limit This is exact for the two rows of one merge step. The merged disk keeps one state per field, so with three or more rows for one disk, a withdrawal that an earlier step replaced is invisible to a later step. Sol found two such counterexamples, both needing three rows (one also needs coincidentally equal values, the other a legacy agent plus two Proxmox rows). Closing that needs each reading's own state tracked across merges, and that redesign is filed separately. Preferring another source's current value instead of the retained one was tried and dropped: it needs an ordering between same-source states that the rows do not record, and each version reopened a withdrawn reading somewhere. ## Tests - `TestPhysicalDiskMergePairsTemperatureWithItsCollectionState`, in both ingest orders: - silent agent against a collected Proxmox reading, and against no Proxmox reading - reporting agent - silent agent against a Proxmox copy of its earlier reading, with and without SMART attributes - resumed agent against a Proxmox copy of its withdrawal - standby agent against a Proxmox copy of its pre-sleep reading - `TestPhysicalDiskMergeNeverPresentsAWithdrawnTemperatureAsCollected`: three distinct-valued withdrawn or stale rows in all six orders. - `TestUnraidInventoryRowDoesNotMaskTheSMARTRowState`: the Unraid inventory's `available` no longer lends itself to a SMART row's uncollected reading. - Without the rule, the peer-reported case, the Unraid masking case and one three-way order fail.	2026-10-06 23:05:53 +01:00
rcourtman	dcb69cfb02	Record Docker host, service and sub-resource alert history (#2620) ... Follows #2608, which made `MonitorAdapter.RecordChange` resolve Proxmox and agent alert references to canonical resources. Three kinds of alert reference still journaled under strings no resource answers to: - Docker host alerts (`docker:<host ID>`) - Swarm service alerts (`docker:<host ID>/service/<ID>`) - sub-resource alerts such as ZFS pool and device health (`<storage ID>/zfs-pool:<pool>[/device:<dev>]`) and host filesystem, SMART, RAID and sensor alerts (`agent:<host ID>/disk:…` etc.) Resource facets, timelines, resource intelligence and Assistant context read history by canonical ID, so these alerts never appeared there. ## What changes - **Docker host refs** resolve through the registry's Docker source ID. - **Swarm service refs** resolve through the host's Swarm cluster to the service with that exact ID, from any manager that reports it. A service with no ID is referenced by its name in the same slot, so while a cluster has such a service, every service ref there is a conflict and binds nothing. - **Never bind:** container names, shortened container IDs and hostless `docker-service:<name>` refs never bind, and are no longer retried against later inventory. - **Sub-resource refs** bind to the owner whose durable ID they carry: - ZFS pool and device alerts go to the storage. - The Unraid array alert goes to its array storage. - Filesystem, SMART, disk temperature, RAID and custom-sensor alerts go to the host. - The owner must resolve to exactly one resource of the expected type. An ambiguous or incompatible owner is a conflict. - Mount and kernel device labels can name a different disk after a reboot, so these alerts never bind a physical disk. - **Retries:** a reference that hits a conflict at write time now enters the backfill retry set, the same as one found at startup. ## Known residuals (producer-side, follow-up proposed) - A container with no ID alerts under its host's reference (`DockerResourceID(host, "")`), so its events join that host's history (never another container's). - An ID-less service whose name equals another service's ID shares that service's reference. Rows it journaled can join the other service's history once a later generation or event binds the reference. Swarm always reports service IDs, so this needs a malformed report. Both are documented in the contract. Separating them needs a producer change in `internal/alerts`. ## Evidence Private `PULSE_MOCK_MODE` stack, fresh data dir. The binary at #2608's tip ran first, then this change on the same dir. Alert rows were classified by source reference (from `alert_identifier`) and checked against each owner's `/api/resources/{id}/timeline`: | ref shape | rows readable via owner timeline, before | after | |---|---|---| | Docker host | 0 / 1 | 3 / 3 | | Swarm service | 0 / 5 | 19 / 19 | | ZFS pool | 0 / 9 | 26 / 26 | | ZFS device | 0 / 9 | 27 / 27 | | agent filesystem | 0 / 1 | 3 / 3 | | Docker container (mock short IDs) | 0 / 11 | 0 / 30, no container alias written | Tests: - `TestHistoryIdentityMonitorAdapterResolvesDockerHostAndServiceReferences` covers multi-manager services, a Docker host merged into its agent machine, legacy-row backfill, recovery after removal, ID-less service conflicts, and names and short IDs staying unbound. - `TestHistoryIdentityMonitorAdapterResolvesSubResourceReferences` covers every owner shape, wrong-type and ambiguous owners, retained bindings, write-time conflict retry, and physical disks staying unbound. - `TestOwnerAlertTimelinesUseCanonicalHistoryIdentity` drives the real alert manager for Docker host offline, Swarm service gap, ZFS pool/device and host filesystem alerts. It fails on #2608's tip. UI reach: the Docker, storage and platform host-row drawers don't request per-resource history. This fix reaches the facets/timeline/intelligence APIs and Assistant resource context, not a drawer.	2026-10-06 22:57:41 +01:00
rcourtman	1fe9f86d55	Merge remote-tracking branch 'origin/main' into refactor/drop-noncompact-group-node-metrics	2026-10-06 22:49:45 +01:00
rcourtman	94b7870855	Call a disk that is only hot Running Hot, not Replace Now ... The Physical Disks Health verdict mapped any critical disk risk to "Replace Now". Disk risk also grades temperature (temperature_high at >=70C critical, >=60C warning), so a SMART-passed disk with good life running at 72C told the user to buy a new disk when the fix is airflow. Alerts already keep heat apart: the SMART health and wearout alerts skip temperature_high and raise disk heat as a Temperature metric. The verdict now splits risk reasons into heat and everything else. Heat alone reads "Running Hot" (phone "Hot"), red at the critical tier and amber at warning, with the temperature reason still as its title and drawer summary. FAILED health, any other critical reason, and a critical level no reason explains keep "Replace Now". The more severe class names the verdict and failure evidence wins a tie, so merged reasons that list heat first no longer hide pending sectors. Sorting and the health filter follow the same verdict, so hot disks stay in "Needs attention" at their temperature tier. A reason's code and severity count even when it has no display text. A critical level that no reason explains shows the generic summary rather than a weaker reason's text. Thresholds are unchanged. The tier still comes from the server's flat 60/70C rule, not the per-type alert thresholds that colour the Temp cell. Reconciling those is separate work.	2026-10-06 22:47:48 +01:00
rcourtman	5becf59536	Remove the unreachable host metric cells from grouped workload rows ... Both production mounts of the Workloads surface, the Proxmox and vSphere overviews, set compactGroupHeaders unconditionally. WorkloadPanel's per-column node cells (CPU, memory, disk and I/O bars or sparklines, uptime, version, temperature) and NodeGroupHeader's column mode and inline host facts therefore never rendered. They were left over from the retired standalone Workloads page, whose group rows had no hosts table beside them. Drop the option, the dead renderers and the NodeGroupHeader props that only served them. Grouped host rows keep exactly the shape both pages already render: one spanning cell with status, linked name, cluster and agent badges, while host stats stay in each page's own hosts table. Tests that only pinned the option are removed. The NodeGroupHeader test now covers the spanning-cell shape and fails against the old default facts, and the PVE version formatter guardrail moves to proxmoxPageModel, the remaining version renderer.	2026-10-06 22:46:05 +01:00
rcourtman	0fb5ccb62b	Keep node temperatures off same-named nodes at other sites (#2617) ... Two Proxmox connections can each have a node with the same name, which is common for MSPs running one `px1` per site. With one host agent linked to site A's `px1`, `getHostAgentTemperatureByID("siteB-px1", "px1")` returned site A's temperature. The hostname fallback took the first agent with that hostname, even one linked to another node, and even when the name was ambiguous enough that the agent linker had refused to link it. The cluster sensor cache leaked the same way: SSH readings an agent collects for its cluster siblings were keyed by bare node name. Site B's `px1` then showed, alerted on and recorded history for another machine's sensors. ## Change - The poller hands the lookup the polled node (`getHostAgentTemperatureForNode`, `carriedTemperatureOutlivesAgentLease`), and the lookup works on that node's **slot**: its own source ID, plus, when the read state does not hold that ID, the one same-named node proven to be the same machine. - The state folds the views of one machine reached through two connections (a cluster added twice, or a multi-homed host added by each address) into a single node kept under one view's ID, and an automatic agent link follows it. A narrowed fallback on its own would leave those views without a temperature permanently; that was reproduced with real `UpdateNodesForInstance` folds. - Same-machine proof comes from the state's own evidence: new `models.NodeObservationsSameMachine` and `models.HostAgentBridgesNodeViews` (`internal/models/node_machine_identity.go`). `UpdateNodesForInstance`'s agent-corroboration check now shares its rule with them. Two or more candidates are ambiguous and none is taken, as the state declines an ambiguous alias. - An agent linked to the slot is the node's agent. The hostname fallback only takes an agent with no node, VM or container link, and only when no node outside the slot and no second unlinked agent has the name. - Cluster sensor cache entries record their reporting agent. A reading serves a node only when that agent's linked node shares one of the slot's connections, or belongs to a connection whose configured view of the node has a matching TLS fingerprint (the other connection of a cluster added twice). `LinkedNodeID` holds the Proxmox source node ID for every link source (automatic `findLinkedProxmoxEntityWithHints` → `NodeView.SourceID()`, manual `LinkHostAgentToNode` validating against `models.Node.ID`, reconciliation, continuity restore), so the link match needs no namespace normalization. A manual link stays pinned to the ID the operator chose (the state never transfers intent to a replacement provider identity), so a node held under the other view's ID has no linked agent. The physical-disk SMART merge already matches by `node.LinkedAgentID` within one instance and is unchanged. ## Tests - `TestNodeTemperatureStaysWithinSameNamedNodesSite` (registered monitoring proof, `monitor_additional_test.go`), through `collectNodeTemperatureData` against registry-backed and state-folded read states: agent linked to the other site's node; ambiguous unlinked agents; unambiguous unlinked agent (still served); cluster added through two connections (alternating real state folds); multi-homed host via two standalone connections; ambiguous address alias; a namesake agent with a contradicting fingerprint; sibling readings across both connections of one cluster; sibling reading from the other site. The leaking cases fail on main, and each folded-view guard was checked with an overlay negative control. - `TestApplyHostReportScopesClusterSensorsToReportersConnection` (`monitor_host_agents_test.go`): real `ApplyHostReport` ingest of cluster sensors. - `TestNodeObservationsSameMachineMatchesStateFold` and `TestHostAgentBridgesNodeViewsMatchesStateFold` pin the predicates against `UpdateNodesForInstance`'s actual fold. - Full `internal/monitoring`, `internal/models` and `internal/unifiedresources` pass on pulse-dev.	2026-10-06 22:40:51 +01:00
rcourtman	3baab9bc59	Say a reviewed Patrol issue stays reviewed when suppression ends (#2615) ... Ending a suppression puts the alert in Acknowledged when it is acknowledged and in Open otherwise (`unsuppressOperationalAlert` in internal/alerts/active_lifecycle.go). Acknowledged is not active attention (`attentionStateCountsAsActive`). The Patrol attention detail still promised otherwise for an issue that was marked reviewed and then suppressed: - its early-end button read "Return to active attention" - the handled-view notice read "Returned to decision inbox" - the suppression form offered "Return it to active attention after" In each case the issue actually went back to Reviewed. | | Reviewed issue | Unreviewed issue (unchanged) | |---|---|---| | Suppression duration | End suppression after | Return it to active attention after | | Early-end button | End suppression | Return to active attention | | Handled-view notice | Suppression ended. It stays reviewed. | Returned to decision inbox | The duration label and the button follow the detail's `operationalRecord.acknowledgement`. The notice instead reads the item's state from the reload that follows the change. If the review was removed elsewhere after the detail loaded (another tab), it still reports what actually happened. `getAttentionSuppressionEndPresentation` owns the copy. The countdown line under the suppression ("Returns to active attention in 1d.") is fixed separately by fix/future-relative-time, using the same wording. Verification: - Three new tests in `PatrolAttentionWorkbench.test.tsx`: - The reviewed case fails on main. - The stale-detail case fails on a version that predicts the notice from the detail. - The unreviewed case pins the copy that stays the same. - `src/features/patrol` and the two other test files that import the workbench pass (492 tests). - Run live on a private mock stack at 1440px, 800px and 390px: - Reviewed case: mark an issue reviewed, suppress it, end the suppression early from the handled view. The button read End suppression, the notice read "Suppression ended. It stays reviewed. N handled issues remain.", and the attention API reported the item `acknowledged` with the active count unchanged. - Unreviewed case: the item came back to the inbox as `open`. - Stale-detail case: the review was removed through the API while the detail stayed open. The notice read "Returned to decision inbox", matching the API state `open`.	2026-10-06 22:37:10 +01:00
rcourtman	c3adf0c7b9	Chart Net and Disk I/O in the row history lens (#2614) ... Hovering a guest row in bars mode swaps CPU, memory and disk for history sparklines, but Net I/O and Disk I/O stayed as static numbers. They are the jumpiest readings in the row, so they gain the most from history ("short burst or sustained?"). The hover request already returns the I/O series for that guest, so this adds no reads. Changes: - The row history lens now swaps both I/O cells too, sharing the synchronized scrub cursor. - I/O sparklines (lens and Trends) keep a compact current pair beside the chart (`↓3.3M ↑2.2M`, `R1.4M W826K`) instead of hiding the values in a tooltip. The glyphs use the series colours, so the pair doubles as the legend. The full rates stay in the chart's accessible label. - Rate sparklines get a 100 KiB/s scale floor. Before, a guest moving between 2 and 6 KB/s drew the same full-height spike as one at 500 MB/s. - `GuestRow` builds every sparkline from accessors. Each live snapshot used to remount the lens and Trends charts (dropping an open scrub tooltip). An open tooltip now also re-reads the current series instead of holding the values from the last mouse move. - `formatCompactSpeed` for the compact pair, the "History" hover option tooltip, and the frontend-primitives and performance contract clauses. Verified on a private mock stack at 1920, 1600, 1440 and 768 (dark and light), including scrub sync, an intercepted idle-I/O history, a live websocket update under an open tooltip, and Trends mode (0 clipped cells). Receipt included. Two pre-existing issues found while verifying are left out of scope. At about 1440px the first-use history hint's dismissal shifts the table 36px, so the first preview opens on the next row. And `WorkloadPanel`'s non-compact node header metric cells never render in production.	2026-10-06 22:34:55 +01:00
rcourtman	8259a74c9d	Show the live reading of threshold alerts held below trigger (#2567) ... A threshold alert stays open after its reading drops below the trigger. It holds until the reading reaches the clear level, then must stay there for the recovery delay. While it held, the evaluator never touched the alert, so every surface kept showing the last breach. On minipc the drawer said "Node temperature at 80.0°C" next to a green 72°C in the node table, with nothing explaining why the warning was still there. #2068 reports the same thing for a disk alert. **What changes** - Open threshold alerts carry a volatile `metricStatus`: the value Pulse is comparing now (the average for rolling-average rules, with the latest sample beside it), the phase (`breaching`, `latched` or `recovering`), the trigger and clear levels, and recovery progress from the reducer's monotonic run. `value`, `message` and `lastSeen` keep meaning the last breach, and the hold branch still never notifies. - The status is stripped from the active-state checkpoint, the recovery mirror and the degraded recovery projection, so holding adds no durable writes. Resolved records carry none. - Stale-alert cleanup and the retention TTLs use the latest observation. Previously an alert held below its trigger for more than a day was auto-resolved as "resource no longer being monitored". - Node, guest and Mail Gateway drawer rows lead with the reading now and say what clears the alert, with the last breach on hover. The Proxmox TEMP cell keeps its own open temperature alert's tone and explains it on hover. Storage rows stop saying "Over N%" once usage is back under the limit, and the Assistant handoff no longer labels the breach value as current. **Verified** - Unit tests drive a node temperature alert through 85 → 78 → 72 → 75 → 77 → 72 → resolved, and fail against the old evaluator. - Private mock stack with pve2 stepped 88 → 78 → 72°C at desktop and phone widths; see the browser receipt in this PR. Refs #2068	2026-10-06 22:23:05 +01:00
rcourtman	38c53d047f	Remove the never-mounted unified resource table (#2612) ... `PlatformResourceTable` was the only renderer of `UnifiedResourceTable`, and nothing imports it. Every platform page renders its own native tables, and `/infrastructure` only shows the opening-workspace placeholder. This removes the never-mounted cluster and the governance that described it. ## Evidence it was dead - Import graph from `src/index.tsx` (static, dynamic, `vi.mock`/`importActual`): the cluster was reachable only through `sharedPlatformPage.tsx`'s import for the unused `PlatformResourceTable` export. No barrel, namespace import, lazy route or re-export. - Production build of main: none of the cluster's literals ship (`infrastructure-table-surface`, `ready for Pulse`, `Agent Infrastructure`), while a live control string (`workloads-table-surface`) does. ## What changes - Deleted: `PlatformResourceTable` and its `PLATFORM_STATUS_FILTER_OPTIONS`; `UnifiedResourceTable` with its host, service, PBS and PMG cards, source badge cell, state hook, viewport sync and models; `ClusterDeployBanner` (only the host card rendered it; the live cluster deploy flow is the Settings `ClusterAgentDeployDialog`); and their seven test files. - Kept coverage for live code those tests also exercised: the detail-mapper no-fetch boundary and the online boolean sensor case move to `resourceDetailMappers.test.ts`; the `ResourceFacetSummary` render cases move to `ResourceFacetSummary.test.tsx`. Both are now registered proofs on the policies that govern their subjects. - Guardrail tests, `shared-template-registry.json`, `registry.json`, `status.json` evidence (RA1, L10, L13; L13 now cites `useUnifiedResources.ts`) and the performance-and-scalability, unified-resources, frontend-primitives and cloud-paid contracts drop the deleted paths and the rules that only governed them. - `.bundlesize.json`: Rollup used to name the large shared chunk `sharedPlatformPage`; it now names it `MetricBar` (118.05 kB gzip), so that entry is re-keyed. Total JS gzip moves +1.96 kB from re-chunking and total CSS drops 0.5 kB. ## Verification - Browser pass on a production build embedded in a private mock backend: all 32 platform sub-tabs at 1440x900 and 390x844 render populated tables with no horizontal overflow; shared toolbar search, the no-match empty state and inline row detail work on Machines and Docker images. The row detail raises eight `style-src-attr` CSP violations, identical on the main build (pre-existing static `overflow-anchor` styles in `ResourceDetailDrawer.tsx`). - `tsc`, eslint, vitest over Infrastructure, platformPage, shared, standalone and app tests; shared-template audit; bundle-size check; contract, registry and status audits; the release-control unittest suites; the full pre-commit battery.	2026-10-06 22:21:02 +01:00
rcourtman	0f5f054ecb	Settle slow polled reads and keep replication links honest (#2610) ... Follow-up to #2578 from a second-opinion review. It closes three gaps #2578 left, plus the issues found while fixing them. **Polled reads that never settle.** The shared poll in `createNonSuspendingQuery` replaced any read still in flight on every tick. The API answers 429 with `Retry-After: 60` and `apiFetch` waits it out, so under rate limiting a 30-second poll discarded every read and the query never settled. Replication reads also ignored the abort signal, so each replaced request kept running. - A poll now skips its tick while a read that an earlier poll started is still in flight. - A refresh, a source change or a remount revalidation stays replaceable at the next tick, as before. The behaviour pinned by `744160fa0e` and the History tests is unchanged. - A polled read still in flight at or after 150 seconds is presumed stalled. It is replaced and settled as a failed read, so nothing spins forever when every read hangs. - `fetchReplicationJobs` forwards the signal to `apiFetch`. - It throws `apiErrorFromResponse` errors that keep the HTTP status, so a 401 or 403 withdraws the jobs. **Replication direct links.** A failed first read made the job count zero, so `/proxmox/replication` fell to Overview with no error or Retry. - Replication presence is now unknown until a read succeeds. - After a read succeeds, a failed poll keeps the last answer instead of reopening a route that an empty read closed. - An access denial withdraws the tab, and the link holds on the denial message. **The shared clock trailed by a tick.** `useRelativeTimeNow` returned its last tick, so a cell mounted between ticks measured from up to 30 seconds ago, and the desktop backup age could disagree with its band. - Every read now returns the wall clock, and the tick only schedules re-reads. - This retires the backup `Math.max` workaround. - The #2578 age tests now jump the clock and fire one tick. They used to loop hundreds of re-renders, and one timed out under load. Each new test fails without its fix. Live check: a private mock stack driven by Playwright, with replication answers shaped per scenario, at 1600px and 390px. - A 500 on the first read keeps the error and Refresh, and Refresh loads all 8 jobs. - A 403 after loading withdraws the tab and shows the server's denial. - Empty, then failed, then empty stays on Overview. - With a polled read held open, two poll ticks start nothing and abort nothing. - Leaving the page aborts the read. - Backup ages and bands render consistently with no Unknown cells. The same session with main's runtime files reproduced each failure. The browser receipt is in the commit.	2026-10-06 22:18:42 +01:00
rcourtman	38a5334b90	Judge disk risk only on a temperature collected now (#2611) ... Since #2593, a disk whose host agent stops reporting keeps its last SMART temperature as a last-known value marked `unavailable`, and a disk in standby keeps its pre-sleep reading the same way. The shared disk risk assessment still fed that value into its temperature rule (>=70C critical, >=60C warning). So a silent agent's frozen 72°C made the Physical Disks row read **Replace Now** ("Disk temperature is 72C") with a `warning` resource status, while #2603 makes the temperature cell beside it say the reading is not current. ## Change - `storagehealth.CollectedTemperature(temperature, collection)` passes a temperature through only when its `collection.temperature` state is `available`, or when the source predates collection state (nil or empty state). Otherwise it returns 0 before risk judges it. - Three places use it: - `AssessPhysicalDisk`, for Proxmox inventory disks, through the unified adapter; - `AssessHostSMARTDisk(WithThresholds)`, for a host agent's own SMART rows; - `physicalDiskAssessmentFromMeta`, the registry's recompute after it merges sources. - The retained value stays on the resource as last-known history. Health, wearout and SMART counters are unchanged: they are durable evidence and keep judging as before. Alerts are unaffected: - SMART disk-health alerts already discard `temperature_high` (`internal/alerts/host.go`). - Disk temperature alerts evaluate the agent's own fresh report. - No fresh collector emits a positive temperature with a non-available state. ## Evidence On a private mock stack, with a local generator overlay stamping real collector states: - Before: the silent-agent disk (72°C retained) read **Replace Now** and the API reported `status: warning`, `risk: critical`. - After: it reads **Healthy**, the API reports `online` with no risk, and its temperature shows as last known. - A genuinely hot disk (72°C `available`) still reads Replace Now. Whether "Replace Now" is the right word for a heat-only verdict is a separate question. ## Tests - `internal/storagehealth/risk_test.go`: `TestCollectedTemperatureKeepsOnlyCurrentReadings` and `TestDiskAssessmentsIgnoreRetainedTemperature`. - `internal/unifiedresources/registry_test.go`: `TestPhysicalDiskRiskIgnoresRetainedTemperature`. It covers Proxmox disks in every collection state through the monitor adapter, plus a real agent+Proxmox merge of the same disk, collected and retained. - Mutation-checked: removing any of the three gates fails a case, and the merged case fails without the registry-recompute gate. - Full `internal/storagehealth`, `internal/unifiedresources`, `internal/api/resourceapi` and `internal/truenas` pass, plus the SMART/disk/host subsets of `internal/alerts` and `internal/monitoring`. ## Related - The Unraid inventory temperature path in `adapters.go` (`assessUnraidPhysicalDisk`) gets the same gate in the Unraid change that stamps its collection state. - A registry merge can still pair a retained agent temperature with a Proxmox row's `available` state (`registry.go`, Proxmox-over-agent carry). That is being fixed separately.	2026-10-06 22:10:42 +01:00
rcourtman	4f9bf4c8c4	Show retained disk temperatures as last known, not live (#2603) ... Normalization keeps a disk's last known temperature when the current observation is not available: a disk in standby, or a linked host agent past its reporting lease (#2593). The monitoring contract says consumers must not present that retained value as freshly collected, but three storage surfaces still painted any positive temperature with live threshold colour: - the Physical Disks table's Temp cell - the pool drawer's linked-disk list - the disk drawer's temperature card Only the drawer's collection message reflected the state, so it read "Temperature is temporarily unavailable: host agent stopped reporting" directly above a red "Temperature 72°C" card. ## Change - One decision in `features/storageBackups/diskPresentation.ts` (`isPhysicalDiskTemperatureCurrent`, `getPhysicalDiskTemperaturePresentation`): - a temperature is current only when its state is `available`, or the source predates collection state (absent or empty state); - `unavailable`, `unsupported` and `missing` make it a retained reading. - A retained reading renders muted, with no hot/warm/healthy colour. It gets a dotted underline (the "this has a note" affordance), the collection reason as its title (for example "Last known reading, not current: disk is in standby"), and screen-reader ", last known" text. - The drawer card is labelled "Last known temperature" and its value is muted. - The pool drawer now formats through the shared temperature formatter, so it follows the Fahrenheit setting like the table. A visible word does not fit the 41px phone Temp column. There, the grey dotted value is the cue, and a tap opens the drawer, which says why. ## Evidence - Searched issues for disk-temperature confusion: #1540, where a user saw a disk temperature and alarm higher than reality that "cleared over night", and #1471, which set the expectation that spun-down disks stay blank rather than show a pre-sleep reading. - Live pass on a private mock stack. The generator was overlaid locally to stamp real collector states, and origin/main's files were swapped in under the same Vite for a control: - Before: the silent-agent disk's retained 72°C was red and indistinguishable from a genuinely hot disk beside it, and the standby disk showed a green 36°C. - After: both read muted with reason titles, and the live 72°C stays red. - Checked at desktop 1440 light and phone 390 dark, with no clipping and no page errors. Receipt in `frontend-modern/browser-verification/`. ## Tests - `diskPresentation.test.ts`: state matrix, including empty state and no reading. - `diskDetailPresentation.test.ts`: last-known card and its tone. - `storagePoolDetailPresentation.test.ts`: an Unraid spun-down member. - `DiskList.test.tsx`: one row moving current → retained → legacy. - `DiskDetail.test.tsx`: the existing collection-message case now asserts the muted last-known card. - `src/components/Storage`, `src/features/storageBackups` and `src/features/platformPage` all pass (107 files). ## Not in this change - The backend risk verdict still judges a retained temperature, so the Health column can read "Replace Now: Disk temperature is 72C" for a disk whose temperature is not current. That is fixed separately in `internal/storagehealth`. - `GuestPhysicalDisks` already renders temperatures without threshold colour. - The Machines table and drawer read host-agent SMART rows whose frontend type does not project `collection` yet.	2026-10-06 22:06:45 +01:00
rcourtman	cbedbc7edf	Stop serving retained disk temperatures as current readings (#2609) ... Follow-up to #2593 (silent agent SMART readings), which made retained disk temperatures carry a non-available collection state and kept them out of history. ## Why Normalization may keep a disk temperature it did not collect (a disk in standby, a host agent past its reporting lease), provided the collection state says so. #2593 stopped the history writers recording those values. Two consumers still presented them as current: - **History live point.** When a disk range has no samples, `/api/metrics-store/history` answered with the disk's temperature as a `source: "live"` point at now, whatever its state. Now that retained temperatures never reach history, a short-range chart for a silent agent's disk or a sleeping disk showed the retained value as a fresh reading. Mock mode also seeded synthetic series from it, and padded sparse stored series out to now. - **Unraid inventory temperatures.** Unraid array rows carry no per-field provenance, so a temperature taken from one (the Unraid-native disk row, or a SMART row falling back to the inventory reading) could not be told apart from a collected one once the agent went silent. `ExpireHostTelemetry` only marks SMART rows. ## What - `diskinventory.TemperatureCollected` is the shared form of the rule the history writers apply (no state, or `available`). The router's disk live point, the mock synthetic seed and the mock padding all require it. - The unified-resources adapter derives the state of an Unraid inventory temperature from the host lease that `ExpireHostTelemetry` already records atomically, and from the disk's spin state: `unavailable` from `unraid` ("host agent stopped reporting", or "disk is reported spun down"), otherwise `available`. The value stays as last-known context. - A SMART row that borrows the inventory reading takes that state, including `available`. - The Unraid-native row records only the withdrawing states. The registry picks a merged disk's temperature and its state separately, so an `available` claim from the provenance-less row would mask the SMART row's own state for the same disk. - A withdrawn inventory temperature no longer counts as heat in the disk's risk assessment. - Storing per-row state on `models.HostUnraidDisk` instead would have added a model field, a read-state round trip and a frontend type change just to duplicate `host.Status`. ## Tests - `TestContract_MetricsHistoryDiskLivePointOnlyForCollectedTemperature`: a silent agent's disk (through the real `ExpireHostTelemetry` and registry path) and a standby disk get no live point in single-metric or all-metrics responses, and no value-derived mock series. Sparse stored samples are not padded to now for a retained, absent or unknown disk. A collected disk still gets its live point and mock series. Each router gate was reverted individually and the test failed each time. - `TestSilentUnraidAgentDiskTemperatureIsRetainedButNotCollected`: three Unraid disks (inventory only, SMART row without a temperature, SMART row whose temperature is unsupported) go collected, then retained but not collected after `evaluateHostAgents` expires the lease, then collected again. - `TestUnraidDiskTemperatureFollowsTheHostLease`: reporting, silent and spun-down hosts across the native row, the borrowed reading and a SMART row with its own reading, including risk. - `TestUnraidInventoryRowDoesNotMaskTheSMARTRowState`: the merged-registry case, which fails if the native row claims `available`. - `TestTemperatureCollectedOnlyForAReadingTakenByThisObservation`: the helper.	2026-10-06 21:34:16 +01:00
pulse-triage[bot]	30202e235f	Merge branch 'main' into fix/node-drawer-temperature-records	2026-10-06 21:34:08 +01:00
rcourtman	f4c1046215	Record Docker host, service and sub-resource alert history ... Alert lifecycle events resolve to canonical resources since Proxmox and agent references gained durable resolution, but Docker host alerts ("docker:<host ID>"), Swarm service alerts ("docker:<host ID>/service/<ID>") and sub-resource alerts such as ZFS pool and device health ("<storage ID>/zfs-pool:<pool>") or host filesystem usage ("agent:<host ID>/disk:var") were still journaled under references no resource answers to. Resource facets, timelines, resource intelligence and Assistant context read history by canonical ID, so they never saw them. Docker host and service references now resolve through the registry's own Docker source identities. A service reference must carry the service's real ID in the host's Swarm cluster, from any manager that reports it; services without an ID are referenced by name and never bind. Container names and shortened IDs still bind nothing and are no longer retried against later inventory. Sub-resource references bind to the owner whose durable ID they carry: the storage for ZFS pools and devices, the Unraid array storage, and the host for filesystem, SMART disk, disk temperature, RAID and custom-sensor alerts. Mount and kernel device labels can name a different disk after a reboot, so they never bind a physical disk. Any other shape stays unbound. On a mock stack none of the Docker host, Swarm service, ZFS and agent disk alert rows were readable through their owner's timeline API before this change; all of them are after it, including rows journaled earlier. The mock's shortened container ID rows stay unbound.	2026-10-06 21:23:26 +01:00
rcourtman	1b081c6475	Record Proxmox and agent alert history on the canonical resource (#2608) ... Alert lifecycle changes (fired, acknowledged, snoozed, resolved) are journaled with the alert's source-native resource ID. `MonitorAdapter.RecordChange` only mapped full Docker container IDs, so Proxmox node, guest and storage alerts and host-agent alerts landed under IDs such as `instance-node`, `instance:node:vmid` or `agent:<host>`. Resource facets, resource intelligence (`recent_changes`) and Assistant resource context read history by canonical ID, so they never showed those alerts. Incident timelines were unaffected because they read by alert identifier. **Change** - `RecordChange` resolves non-Docker references to the canonical resource and records a history binding, so the reference keeps reaching the same history after a guest migration or inventory removal. - Only durable identities bind: canonical ID, retired era, source ID, node-scoped guest reference, `agent:<host ID>`. Names and hostnames never bind (a VM named `pulse-system` cannot capture system alerts). A reference that two resources answer to binds to neither. Docker keeps its full-ID rule. - Rows already journaled under a source reference join canonical history without being rewritten. Each registry generation binds the unbound alert references it can resolve, both those found at startup and those written before inventory named the resource, retrying each for 30 minutes. **Evidence** (private mock stack, same data dir, origin/main first, then this branch) - main: 51 of 52 alert references were source-shaped and there were zero bindings. `GET /api/resources/{id}/facets` for affected nodes and guests returned only `state_transition`. - this branch: the first registry generation logged `bound 17 alert history references`. Every non-Docker whole-resource reference is now bound: node metric and connectivity alerts, guests, Proxmox storage, `agent:host-linux-1` and `pmg-main`. Facets and `/api/ai/intelligence?resource_id=` now include `alert_fired`/`alert_resolved` for the VM, node and storage checked. - Still unbound, as before: ZFS pool/device and agent-disk sub-resource references (no resource of their own) and Docker host/service/short-ID references. **Where it shows** In the UI, only the full-presentation `ResourceDetailDrawer` requests this history. Proxmox node and Agents rows open the `table-row` drawer, and guest and storage rows open their own drawers; none of these fetch it (confirmed live: the node History tab makes no facets or intelligence request). So the user-visible gain is in Assistant and Patrol resource context and the facets and intelligence APIs, not in a Proxmox drawer. **Tests** `TestHistoryIdentityMonitorAdapterResolvesProxmoxAlertReferences` and `TestHistoryIdentityBindsLegacyAlertRowsFromRegistryGenerations` (registered `history_identity_test.go`), plus `TestProxmoxAlertTimelineUsesCanonicalHistoryIdentity` through the real alert-manager callback. The monitor test fails on main. Mutation checks confirm the new tests catch alias-based resolution, a missing post-scan retry and ignored ambiguity.	2026-10-06 21:22:46 +01:00
rcourtman	41b94b29a8	Keep a merged disk's temperature under its own row's state ... The registry merges a physical disk's rows (Proxmox, host agent SMART, Unraid inventory), choosing the temperature by source preference and merging field states separately. When two rows carry different temperatures the shown value could sit under the other row's "available" state: a silent agent's retained reading carried over a Proxmox row that collected its own reading now, in either ingest order. Risk, history and the UI then treated a frozen reading as current. pairPhysicalDiskTemperatureState runs at the end of each merge step: when the merged state says collected but the row the shown value came from says it was not, the merged state becomes that row's own. It never changes a value and never grants availability, so it cannot revive a withdrawn reading or override a resumed agent's fresh one; rows with the same value keep the merged state. This is exact for the two rows of a step. The merged disk keeps one state per field, so with three or more rows a withdrawal an earlier step replaced is invisible to a later one; closing that needs each reading's own state tracked across merges. Preferring another source's current value instead would need an ordering between rows that the merge does not record. With this in place the Unraid-native row no longer needs to withhold an "available" state to avoid masking a SMART row, so it now carries the same derived state as a SMART row borrowing its reading.	2026-10-06 20:55:59 +01:00
rcourtman	8c9000ac63	Keep node temperatures off same-named nodes at other sites ... Two Proxmox connections can each have a node with the same name, which is common for MSPs running one px1 per site. The node temperature lookup fell back to the first host agent whose hostname matched the node name, even when that agent was linked to the other site's node, and even when the name was ambiguous enough that the agent linker had refused to link it. Site B's px1 then showed, alerted on and recorded history for site A's sensors. The cluster sensor cache leaked the same way: an agent's SSH readings for its cluster siblings were keyed by bare node name, so a site A sibling reading fed site B's node. The poller now hands the lookup the polled node, and the lookup works on that node's slot. The state folds the views of one machine reached through two connections (a cluster added twice, or a multi-homed host added by each address) into one node held under one view's ID, and the agent link follows it. Those views used to find their agent only through the loose hostname match, so a polled view the read state does not hold also takes the one same-named node proven to be the same machine. models.NodeObservationsSameMachine and models.HostAgentBridgesNodeViews give that proof from the evidence UpdateNodesForInstance folds on, and the state's agent corroboration check now shares its rule with them. Ambiguous candidates are declined, as the state declines an ambiguous alias. An agent linked to the slot is the node's agent. The hostname fallback only takes an agent with no link when no node outside the slot and no second unlinked agent has the name. Cluster sensor cache entries record their reporting agent and serve a node only when that agent's linked node shares one of the slot's connections, or belongs to a connection whose configured view of the node has a matching TLS fingerprint. LinkedNodeID holds the Proxmox source node ID for automatic, manual and continuity-restored links, so the link match needs no normalization. The physical-disk SMART merge already matches agents by node.LinkedAgentID within one instance and is unchanged.	2026-10-06 20:50:21 +01:00
rcourtman	3ac5664d95	Key previous Proxmox nodes by source ID so poll carry-overs match (#2604) ... The Proxmox poller reads its previous poll's nodes back from the unified read state, but `previousNodeFromView` returned the registry's unified resource ID (e.g. `agent-ab60dcc1ebca813b`) instead of the poller's source ID (`pve1-node1`), and dropped the temperature. Registry-backed probes confirmed every lookup keyed on the source ID missed: - `nodeLastOnline` in the offline grace policy; - the temperature carry, plus the CPU low/record tracking, which reset to the current reading every poll. The dev stack shows it live: `delly` and `minipc` report `cpuMin = cpuMaxRecord = cpuPackage`; - the node memory carry after a failed status read. When a whole instance poll failed, `markPVEInstanceNodesUnreachable` wrote the projected nodes back, so every failed poll re-ingested the node under a new unified ID (`agent-ab60…` → `agent-aec93…` → `agent-e798…`). The projection also dropped the cluster name the canonical ID is derived from, so cluster members changed ID for the length of an outage. ### Change - The projection returns the source ID, the stored temperature details, the cluster name and the linked agent ID. `previousNodesForInstance` restores the config-derived identity stamps (provider-scoped flag, native aliases, TLS fingerprint) the same way `pollPVENode` writes them. Reviving those paths needed bounds: - **Memory carry** uses the poller's own validated `NodeMemorySnapshot`. The read state holds the agent's reading on a node merged with a host agent. used + cache + free still sum to the total. - **Agent lease:** a carried temperature stamped no later than a lapsed host agent's last report is not carried. Otherwise the 5-minute carry window would hold a silent 30s agent's reading in alerts and history about 2 minutes past the lease that #2582 enforces. Readings record no source, so an SSH reading of that age is dropped too; the bound errs toward a gap, never a stale value. - **No disk readings in a carry:** a carried reading keeps CPU/GPU only. SMART/NVMe rows also stamp physical disks, so a disk that went to standby would otherwise be stamped and recorded with its pre-sleep value. - **Preserved nodes:** outage preservation and authoritative membership reconciliation follow the same rules, and drop the temperature when the node goes offline. ### Tests - New registry-backed tests in `monitor_additional_test.go` cover: CPU low/record tracking across polls, outage writeback keeping the unified ID (including provider-scoped clusters), membership writeback, the lease rule, disk rows excluded from carries, and the memory carry on an agent-merged node. Each fails against the old projection or without its bound (checked with `go test -overlay`). - `TestPollPVEInstanceMarksStaleNodesOfflineWhenGetNodesReturnsEmpty` relied on the dead `nodeLastOnline` lookup; it now ages that sighting too. - The silent-agent test from #2582 now rebuilds the store via `PopulateFromSnapshot` and feeds the polled node back, as production does. Re-ingesting into one registry keeps the first `TemperatureDetails`, so the old setup never exercised the carry. Follow-ups found on the way: same-named nodes on different sites leaking agent temperatures (separate PR stacked on this one), and the node drawer never reading `temperatureDetails` (#2594).	2026-10-06 20:49:02 +01:00
rcourtman	844e1dbb97	Say a reviewed Patrol issue stays reviewed when suppression ends ... Ending a suppression puts the alert in Acknowledged when it is acknowledged and in Open otherwise (unsuppressOperationalAlert), and Acknowledged is not active attention. The attention detail still promised otherwise for an issue marked reviewed and then suppressed: its early-end button read "Return to active attention", the handled-view notice read "Returned to decision inbox", and the suppression form offered "Return it to active attention after", while the issue went back to Reviewed. The duration label and the early-end button now follow the detail's operationalRecord.acknowledgement, which the backend derives from alert.Acknowledged: "End suppression after" and "End suppression" for a reviewed issue, the existing copy for an unreviewed one. The notice reads the item's state from the reload that follows the change instead of that snapshot, so it reports "Suppression ended. It stays reviewed." only when the item really is Acknowledged, even if the review was removed elsewhere after the detail loaded. Contract-Neutral: No api-contracts or frontend-primitives delta: Patrol lifecycle copy only, no API payload or shared primitive change; patrol-intelligence.md carries the note	2026-10-06 20:36:49 +01:00
rcourtman	2265d28242	Remove the never-mounted unified resource table ... PlatformResourceTable was the only renderer of UnifiedResourceTable, and nothing imports it. Every platform page (Proxmox, Docker, Kubernetes, TrueNAS, vSphere, Machines) renders its own native tables, and /infrastructure only shows the opening-workspace placeholder. A production build of main already tree-shook the whole cluster (none of its strings ship), yet it still carried 11 source files, 7 test files, registry ownership in two subsystems and contract rules for a table nobody sees. Removed: PlatformResourceTable and its PLATFORM_STATUS_FILTER_OPTIONS; UnifiedResourceTable with its host, service, PBS and PMG cards, source badge cell, state hook, viewport sync and models; and ClusterDeployBanner, which only the host card rendered (the live cluster deploy flow is the Settings ClusterAgentDeployDialog). Their tests go with them. Coverage for live code those tests also exercised is kept: the detail mapper no-fetch boundary and the online boolean sensor case move to resourceDetailMappers.test.ts, the ResourceFacetSummary render cases move to their own test file, and both files become registered proofs for the policies that govern their subjects. Guardrail tests, the shared template registry, registry.json, status.json evidence and the performance, unified-resources, frontend-primitives and cloud-paid contracts drop the deleted paths and the rules that only governed them. Rollup used to name the large shared chunk after sharedPlatformPage; it now names it MetricBar, so the bundle baseline entry is re-keyed at its measured size.	2026-10-06 20:34:35 +01:00
rcourtman	cdae811bb7	Chart Net and Disk I/O in the row history lens ... Hovering a guest row swapped CPU, memory and disk for history sparklines but left Net I/O and Disk I/O as static numbers, although they are the jumpiest readings in the row and the hover request already returns their series. The lens now charts all five cells on one synchronized cursor. I/O charts, on hover and in Trends, keep a compact current pair beside the chart (↓3.3M ↑2.2M, R1.4M W826K) instead of hiding the rates in a tooltip; the glyphs use the series colours so the pair is also the legend. Rate sparklines get a 100 KiB/s scale floor so an idle guest's few-KB/s chatter no longer draws the same full-height spike as a busy one. GuestRow now builds every sparkline from accessors. Each live snapshot used to remount the lens and Trends charts, dropping an open scrub tooltip; an open tooltip also re-reads the current series instead of holding the values from the last mouse move.	2026-10-06 20:33:10 +01:00
rcourtman	28d5f44257	Judge disk risk only on a temperature collected now ... Since #2593, a disk whose host agent stops reporting keeps its last SMART temperature as a last-known value marked unavailable, and a disk in standby keeps its pre-sleep reading the same way. The shared disk risk assessment still fed that value into its temperature rule, so a silent agent's frozen 72C made the disk read "Replace Now: Disk temperature is 72C" with a warning status, while the temperature cell beside it now says the reading is not current. storagehealth.CollectedTemperature passes a temperature through only when its collection state is available, or when the source predates collection state. AssessPhysicalDisk, AssessHostSMARTDisk and the registry's recompute from merged metadata use it, so a retained reading stays on the resource as history but adds no temperature_high reason and leaves the disk online. Health, wearout and SMART counters are unchanged: they are durable evidence and keep judging as before.	2026-10-06 20:32:00 +01:00
rcourtman	e25472feab	Merge remote-tracking branch 'origin/main' into fix/metric-alert-live-status	2026-10-06 20:23:38 +01:00
rcourtman	95b29286dd	Stop serving retained disk temperatures as current readings ... Normalization may keep a disk temperature it did not collect (a disk in standby, a host agent past its reporting lease) as long as its collection state says so. Two consumers still presented that value as current. The metrics-history handler answered an empty disk range with the disk's temperature as a "live" point at now, regardless of its state. Since the history writers stopped recording retained temperatures, a short-range chart for a silent agent's disk or a sleeping disk showed the retained value as a fresh reading. Mock mode also seeded synthetic series from it and padded sparse stored series out to now. All of these now require a temperature collected now (diskinventory.TemperatureCollected, the shared form of the rule the history writers apply). Unraid array-inventory rows carry no per-field provenance, so a temperature taken from one looked collected after the agent went silent. The adapter now derives its state from the host lease that ExpireHostTelemetry already records, and from the disk's spin state: unavailable once the host is offline or the disk is spun down. A SMART row that borrows the inventory reading takes that state, including available. The native row records only the withdrawing states, because the registry merge picks temperature and state separately and an available claim from it would mask the SMART row's own state. A withdrawn inventory temperature no longer counts as heat in disk risk.	2026-10-06 20:16:38 +01:00
rcourtman	ba26b3581e	Settle slow polled reads and keep replication links honest ... A review of #2578 found three gaps it left. The shared poll in createNonSuspendingQuery replaced any read still in flight on every tick. The API answers 429 with Retry-After: 60 and apiFetch waits it out, so under rate limiting a polled query slower than its 30-second interval never settled. Replication reads also ignored the abort signal, so each replaced read kept running. A poll now skips its tick while a read an earlier poll started is still in flight; a refresh, source change or remount revalidation stays replaceable at the next tick as before. A polled read in flight for 150 seconds is presumed stalled and replaced, which also clears any spinner it held. Replication reads forward the signal to apiFetch. A failed first replication read made the job count zero, so a direct /proxmox/replication link fell to Overview without its error or Retry. Replication presence is now unknown until a read succeeds, and a failed poll after that keeps the last answer instead of reopening the route. The shared clock returned its last tick, so a cell mounted between ticks measured from up to 30 seconds ago and the desktop backup age could disagree with its band. Every read now returns the wall clock and the tick only schedules re-reads, which retires the backup Math.max workaround. The #2578 age tests now jump the clock and fire one tick instead of looping hundreds of re-renders, which timed out under load.	2026-10-06 20:15:12 +01:00
rcourtman	474dbc4f09	Record Proxmox and agent alert history on the canonical resource ... Alert lifecycle changes carry the alert's source-native resource ID, and MonitorAdapter.RecordChange only mapped full Docker container IDs. Proxmox node, guest and storage alerts and host-agent alerts were journaled under IDs such as "instance-node", "instance:node:vmid" or "agent:<host>", while resource facets, resource intelligence and Assistant resource context read history by canonical ID, so those surfaces never saw the alerts. Resolve such references to the canonical resource and record a history binding, so the reference keeps reaching the same history after a guest migration or inventory removal. Only durable identities bind: the canonical ID, a retired era, a source ID, a node-scoped guest reference or an agent host ID. Names and hostnames never bind, so a VM named "pulse-system" cannot capture system alerts, and a reference two resources answer to binds to neither. Docker keeps its full-ID rule. Rows already journaled under a source reference join canonical history without being rewritten: each registry generation binds the unbound alert references it can resolve, those found at startup and those written before inventory named the resource, retrying each for 30 minutes.	2026-10-06 20:14:49 +01:00
rcourtman	9aaac4d3e5	Read a linked agent's own heartbeat in host snapshots (#2601) ... A host agent linked to a Proxmox node merges into one unified host row. When the agent stops reporting but PVE polling continues, that row stays online and its `LastSeen` follows the PVE sighting. `hostFromReadStateView` copied both into `models.Host`, so a silent agent looked current to every consumer of the projection: - `linkedHostForNode` accepted it, and `resolveNodeDisk` kept using the agent's last filesystem summary as the node's disk, feeding disk alerts and history with a frozen reading. Its retained ZFS datasets kept merging into the node's pools. - Agent connections built from `HostsSnapshot()` (the Connections list and the update-readiness agent check) aged the agent from the PVE poll, so a dead agent stayed Active. The projection now takes `LastSeen` from the agent source's own sighting (`HostView.SourceStatus(SourceAgent)`, added in #2582) and reports the agent offline once that sighting is past its reporting lease (`hostAgentReportCurrent`, the same `hostAgentHealthWindow` that `evaluateHostAgents` applies). Rows without an agent sighting, such as vSphere and TrueNAS hosts, keep the row's values. Verification: - New `TestSilentLinkedAgentStopsFeedingNodeDisk` (registry-backed merged agent + node row, driven through `evaluateHostAgents` and the store refresh): node disk uses the agent summary while it reports, falls back to the Proxmox rootfs past the lease, and returns when the agent reports again; the projected agent reads offline with its own last report. It fails against the previous `monitor.go` (checked with `go test -overlay`). - `go test ./internal/monitoring/` and the connections, readiness, update and agent-lifecycle tests in `./internal/api/` pass.	2026-10-06 20:02:09 +01:00
rcourtman	f28f73b28f	Stop presenting a silent agent's disk readings as collected (#2593) ... A host agent that stops reporting keeps its last SMART rows in state. Its frozen disk readings then kept being treated as current: - `mergeHostAgentSMARTIntoDisks` copied the silent agent's temperature and I/O counters into the linked Proxmox physical disk, along with its `available` collection state. - The registry merged the agent's retained SMART rows into the canonical disk the same way. `MergeStatus` lets any `available` state win, so the API kept reporting the readings as collected. - Every full disk poll (default every 5 minutes) wrote the frozen temperature to SMART history as a new sample. ## Change - **Lease expiry is the cut-off.** `State.ExpireHostTelemetry` (run by the offline sweep once `hostAgentHealthWindow` passes) keeps SMART temperature and I/O as last-known values but marks them `unavailable` ("host agent stopped reporting"). This follows the physical-disk retained-evidence clause in the monitoring contract. The agent's next accepted report replaces them. - **A source can withdraw its own evidence.** New `diskinventory.MergeReportedStatus` lets a source's later non-available statement supersede an `available` state that another row still carries from the same source. - Used by the monitoring merge, where skipped disk polls re-merge the previous state. - Used in both registry ingest orders. This covers a PVE host that is down entirely: it is never disk-polled again, so its Proxmox disk row stays frozen. - **History records only collected temperatures.** All SMART temperature history writers (`writeSMARTMetrics`, `smartMetricStoreWrites`, `writeHostSMARTMetrics`, and the unified disk sync via its collection passthrough) skip a temperature whose current state is not collected. This also stops a standby disk's retained pre-sleep reading from being recorded. - **Pre-6.2 agents keep their history.** These agents send no collection provenance, so the merge now records their temperature as available from `host_agent` instead of inheriting Proxmox's `unsupported`. - **Expiry is compare-and-set** on the `LastSeen` the sweep judged stale, so a report accepted in between is never expired. The sweep's alert-side offline handling for that race is unchanged; that race already existed. Deliberately unchanged: - SMART attributes, wearout and health stay last-known. They are durable health evidence, and dropping them would resolve disk-health and low-life alerts whenever an agent goes quiet (cf. #2112). - Disk temperature alerts are evaluated in `CheckHost` from the agent's own report, only when a report is ingested, so they never read this merged value. ## Related - The node-temperature path (agent sensors → `node.Temperature.SMART`, which the disk merge labels `proxmox_node_smart`) is gated on the agent's own sighting in #2582. A carried node reading can still stamp a disk as collected; that path belongs to the node poller. - Filed separately: - the chart API's `liveMetricPoints` returns a retained disk temperature as a single "live" point when a range has no samples; - Unraid-native disk temperatures carry no collection state; - the disk table renders non-collected temperatures with live threshold colouring; - a merged agent+node row reads `online` after its agent dies, so `linkedHostForNode` keeps using the silent agent's filesystem summary. ## Tests - `TestSilentLinkedAgentSMARTIsRetainedButNotPresentedAsCollected` (registered monitoring proof `physical_disk_roundtrip_test.go`). Uses a registry-backed monitor with a real disk poll and a metrics store. A reporting agent's reading is collected and recorded. After lease expiry, the canonical disk reads as not collected before any disk poll runs, and again after a skipped poll and a full poll. The value is retained and no new history sample is written. A resumed agent is collected and recorded again. Fails on origin/main. Mutation-checked: removing the expiry marking, the monitoring supersede, the registry supersede, or the history gate each makes it fail. - `TestMergeHostAgentSMARTIntoDisks_LegacyAgentTemperatureFollowsLease` - `TestPhysicalDiskMergeFollowsAgentWithdrawalOfItsOwnReadings` (both ingest orders) - `TestMergeReportedStatusLetsTheSourceWithdrawItsOwnAvailability` - `TestExpireHostTelemetryMarksSMARTReadingsNoLongerCollected` - `TestExpireHostTelemetryLeavesHostThatReportedSinceJudgement` - `TestWriteHostSMARTMetricsRecordsOnlyCollectedTemperatures` Ran the full `internal/monitoring`, `internal/unifiedresources`, `internal/models` and `pkg/diskinventory` packages locally; all pass.	2026-10-06 19:53:05 +01:00
rcourtman	0c130a4855	Key previous Proxmox nodes by source ID so poll carry-overs match ... The poller reads the previous poll's nodes back from the unified read state, but previousNodeFromView returned the registry's unified resource ID and no temperature. Every lookup keyed on the Proxmox source ID missed: nodeLastOnline in the offline grace policy, the temperature carry, the CPU low/record tracking (reset to the current reading every poll) and the node memory carry. When a whole instance poll failed, markPVEInstanceNodesUnreachable wrote the projected nodes back, so each failed poll re-ingested the node under a new unified ID. Cluster members also lost the cluster identity their canonical ID derives from, so they changed ID for the length of an outage. The projection now returns the source ID, the stored temperature details, the cluster name and the linked agent ID, and previousNodesForInstance restores the config-derived identity stamps (provider-scoped flag, native aliases, TLS fingerprint) the way pollPVENode writes them. Reviving those paths needed four bounds: - The node memory carry uses the poller's own validated snapshot rather than the read state, which holds the agent's reading on a node merged with a host agent, and keeps used + cache + free equal to the total. - A carried temperature keeps a lapsed host agent's lease. A reading stamped no later than the agent's last report is not carried, so the five-minute carry window cannot hold a silent agent's reading in alerts and history past its lease, which #2582 guarantees. - A carried reading keeps the node's CPU and GPU temperatures only. Its SMART and NVMe rows also stamp the node's physical disks, so a disk that went to standby would be stamped and recorded with its pre-sleep value. - Outage preservation and authoritative membership reconciliation keep a node's temperature only under those rules and drop it when the node goes offline. TestPollPVEInstanceMarksStaleNodesOfflineWhenGetNodesReturnsEmpty relied on the dead nodeLastOnline lookup and now ages that sighting too. The silent-agent test from #2582 now rebuilds the store as production does and feeds the polled node back, which is what exercises the carry.	2026-10-06 19:37:14 +01:00
rcourtman	d26a483731	Show a moved alert as moved in Patrol, history and assistant (#2591) ... Follow-up to rcourtman/Pulse#2580. A node alert handed to its Pulse agent closes with an `AlertResolution` (`moved_to_agent`), and notifications already say it moved. Every in-app reader still treated that close as a recovery: - Patrol's attention detail showed a green **Resolved** badge, and its timeline read "Open to Resolved · Recovery evidence" directly above "…This is not a recovery…". - The Alerts history listed the row as **resolved**, and its incident timeline ended "Alert resolved: Node memory at 95.0%". - `GetRecentlyResolved` rebuilt `models.Alert` without the resolution. So the state snapshot, websocket, assistant context, Patrol's seed prompt and the `pulse_alerts` `resolved` action all saw a plain resolve. **Operational trust.** New `TransitionCause` `ownership_transferred`. It may only enter `resolved`, and only with evidence. `markOperationalResolved` uses it whenever the alert carries a `moved_to_agent` resolution; every other close stays on `recovery_evidence`. Alert-quality telemetry keys "resolved while snoozed" on `recovery_evidence`, so a handover during a snooze no longer counts there. No notification code reads the cause: links key on the transition ID and state. **State and AI.** `models.Alert.Resolution` holds the reason, successor and the engine's `Summary()`. `cloneAlert` copies it, and `GetRecentlyResolved` fills it. - Patrol's seed writes `closed 3m ago. Alert moved to pve1 (Host Agent). This is not a recovery: …`. - The assistant's context writes `closed … ago) <summary>`. - The resolved-alerts tool returns `resolution` and `successor_resource_id`. **Timelines.** The `alert_resolved` resource change carries the summary as its reason and the code as `alert_resolution` metadata. Incident memory titles the close with it, both when projecting from the canonical timeline and on its own write path. **UI.** - The history row reads **moved to agent**, even if it was acknowledged. The full account is in the badge title and is printed under the message on the phone card. - Patrol's detail shows a **Moved** badge (info tone), with the handover sentence directly under the summary rather than only inside Evidence and history. Its timeline says "Ownership transferred". **Landing order.** rcourtman/Pulse#2586 is the only producer of a handover, and it has no auto-merge. It should land after this PR (or merge main once this is in). Any handover recorded before this lands would otherwise keep a recovery-labelled transition and timeline. **Not covered here:** - Performance reports still list a handover from the last 5 minutes as resolved. - The resource drawer labels the change "Alert resolved" before its reason. - The incident header in the history expansion still shows its open/resolved status word. Both of the first two are filed as a follow-up and noted in ai-runtime.md and unified-resources.md. **Tests:** - `TestOwnershipTransferOnlyClosesWithEvidence`. - The cause assertions in `TestReleasedMetricAlertCarriesResolutionInsteadOfRecovery`. - `TestRecentlyResolvedCarriesHandoverResolution`. - `TestAlertResolutionWireContract`, which fails without the `cloneAlert` copy. - `TestMonitor_HandleAlertLifecycle_HandoverCloseIsNotARecovery`, which fails on the old code with "Alert resolved: Memory usage at 95%". - `TestIncidentStore_HandoverCloseIsNotARecovery`. - `TestSeedHealthAndAlerts_HandoverIsNotARecovery`. - `TestService_buildAlertContext_HandoverIsNotARecovery`. - `TestExecuteListResolvedAlertsReportsHandover`. - Frontend history, presentation and Patrol workbench tests. The operationaltrust, alerts, models, ai, ai/tools, ai/memory, unifiedresources, monitoring and notifications suites pass. `tsc`, eslint and the vitest sweep pass, apart from the Mac-only `agentInstallCommand` PTY test. **Browser pass.** A private non-mock stack ran on an alert history written by the real `releaseCanonicalMetricAlert` handover path. I checked Alerts history (row, tooltip, expanded timeline) and Patrol's deep-linked detail at 1280 and 375: no overflow, no console errors.	2026-10-06 19:24:36 +01:00
rcourtman	779b3d4ff8	Tint Kubernetes node rows for their drawer's alerts (#2596) ... A Kubernetes node that runs a Pulse agent is an agent row, and the agent's alerts are keyed `agent:<id>` and labelled with the hostname. The nodes table matched only the row id and display name, so on a mock estate "Prod Euw1 K8s 02" sat at 100% memory with a critical alert, its drawer listed "Agent memory at 100.0%", and its row was untinted. The overview's "Nodes needing attention" looked only at readiness and cordons, so it named K8s 03 (Degraded) and left K8s 02 out. - Kubernetes node rows tint from `getUnifiedResourceAlertStyles` (#2579), the set the drawer reads, and drop the tint while expanded like the Docker host and Machines rows. - `getKubernetesNodeAttentionRank` ranks a node by the stronger of its state and its unacknowledged open alerts: NotReady and critical alerts first, then cordoned or degraded nodes and other alerts. The overview list and the nodes table's default order both use it, so a critical node no longer sorts below a degraded one. Acknowledging the alert drops the node out of both, as it already drops the tint. - vSphere host and TrueNAS system rows are deliberately unchanged. Checked against the producers: their VMs', pools' and apps' alerts carry the host name as their node (VM `ParentName` is the runtime host; registry children take the parent's name), the TrueNAS drawer states the same pool problems, and `agent:` alert ids only come from Pulse agent reports (`internal/alerts/host.go`, `external_probe.go`). A Pulse agent on a TrueNAS box stays a separate resource (`TestUnifiedDiskWriteMetricsKeepSameHostnameHostAgentAndTrueNASTargetsSeparate`). Switching them would drop the pool rollup that tints the TrueNAS row for a degraded pool, so a test now pins it. Verified in a browser on a private mock stack (receipt `frontend-modern/browser-verification/83848f2d5ef2a326.json`): Nodes and Overview at 1440px, 390px and in dark mode, the drawer, and acknowledge then unacknowledge of the K8s 02 alert. The new Kubernetes tests fail against the previous runtime and pass with this change.	2026-10-06 19:22:22 +01:00
rcourtman	40ebce161a	Clear agent disk temperature alerts when the disk leaves (#2595) ... Stacked on #2589 (the threshold-off fix). Until it merges this PR also shows its two commits; the change here is the last commit, 7c8e9099a1. A host-agent SMART disk temperature alert (`diskTemperature`, resource `agent:<host>/disk_temp:<device>`) stayed active for about a day after its disk left the agent's report (removed, replaced, or a renamed device path). `CheckHost` only evaluates disks it still lists, and `shouldPreserveAlertOutsideNodeCleanup` exempts `agent:` alerts from node cleanup, so only the 24-hour stale-alert sweep resolved it. What changes: - `CheckHost` records every disk a non-empty SMART list reports. A disk holding an alert or a pending threshold run that is missing from three consecutive non-empty reports has its alert cleared and its pending run dropped (`cleanupHostDiskTemperatureAlerts`). The count restarts when the disk is reported again, because the Windows, FreeBSD and controller-multiplexed Linux collectors drop a disk whose probe fails. - An empty SMART list holds every alert. The agent omits the list when SMART collection fails or is unsupported (`collectSMARTData`). - A listed disk in standby, or with no temperature (a failed Linux probe lists the disk identity-only), holds its alert. Neither reading shows the disk cooled, and SMART health alerts already hold through standby. - Turning the threshold off, or removing the agent, now also drops the host's pending disk temperature runs and absence counts. `TestCheckHostClearsDiskTemperatureAlertWhenDiskLeavesSMARTReport` covers removal, rename, the three holds, the restarted count, and the pending-run and count cleanup. Each case was checked against a mutation that disables its mechanism.	2026-10-06 19:18:41 +01:00
rcourtman	4bd567b41a	Clear agent disk temperature alerts when their threshold is off (#2589) ... Turning off the agent disk temperature threshold left every open disk temperature alert active indefinitely. **Why nothing cleared them** - `CheckHost`'s branch for a disabled `DiskTemperature` threshold was an empty placeholder comment. - The host-wide clears (all agent alerts disabled, a disabled host override, agent removal, confirmed agent offline) matched only `agent:<host>/disk:` and missed `agent:<host>/disk_temp:`. - A config save skipped these alerts because `diskTemperature` was not classed as a threshold metric in `metric_runtime.go`. **Changes** - `host.go`: `CheckHost` clears a host's disk temperature alerts when the effective threshold is off, and `clearHostDiskAlerts` plus the confirmed-offline sweep include the `disk_temp` prefix. The per-type `DiskTempByType` refinement moves into `hostDiskTemperatureThresholdNoLock`, which `CheckHost` and `resolveHostAlertThresholdsNoLock` share. A config save therefore judges an alert against the threshold for its disk type, not the agent default. - `host.go`: an alert persisted before `CheckHost` recorded `diskType` (May 2026) has no disk type until its next firing evaluation. A save judges it against the lowest enabled trigger any of the host's disks can use, so a save cannot resolve an alert its disk type would still fire. - `metric_runtime.go`: `diskTemperature` is a threshold metric, so a save resolves these alerts once the threshold is off or the reading is below it. - `config_runtime.go`: when there is no recovery band, the save path now matches the evaluator. `buildCanonicalMetricSpec` drops a clear level at or above the trigger, and the reducer keeps firing at the trigger itself. Before this, a save resolved those readings and the next report raised them again. This applies to every metric alert that reaches the threshold comparison (docker-host alerts return before it); with a valid band, behaviour is unchanged. **Tests** (`internal/alerts/host_unraid_lifecycle_test.go`) - `TestCheckHostClearsDiskTemperatureAlertsWhenThresholdTurnsOff`: agent default off, host override off, host override disabled, all agents disabled. - `TestHostDiskTemperatureAlertsClearWhenAgentLeaves`: agent removed, agent confirmed offline. - `TestConfigSaveResolvesDiskTemperatureAlertAgainstItsDiskTypeThreshold`: an unrelated save keeps an alert above its nvme trigger; turning the threshold off resolves it. - `TestConfigSaveKeepsDiskTemperatureAlertWithoutDiskType`: an alert with no `diskType`, at 72C under a 75C agent default and a 70C nvme entry, survives an unrelated save. - `TestConfigSaveKeepsAlertTheEvaluatorKeepsWithoutRecoveryBand`: a 70/75 per-type entry at 72C and at 70C. Every case fails against origin/main's code (checked with a build overlay) and passes here. `go test ./internal/alerts/...` passes.	2026-10-06 18:59:53 +01:00
rcourtman	e9069aed8c	Carry a typed reason when an alert closes without recovering (#2580) ... When a Pulse agent is linked to a Proxmox node, the node releases its CPU, memory and disk alerts to the agent, and any node alert still open closes. The close must reach integrations, so a PagerDuty or webhook incident opened from the node alert does not stay open forever. But it went out as a recovery: "pve1 on pve1 is now healthy" for a node still at 95% memory, backed by synthesized "recovery projection" evidence. #1682 is the same complaint from another trigger. This adds a typed `AlertResolution` (first reason: `moved_to_agent`, with the agent's alert resource ID and name) and carries it from the release path to every resolved consumer: - `releaseCanonicalMetricAlert` is the disabled-threshold release pulled out of `evaluateCanonicalMetricAlert`. It takes a resolution, and `clearAlertWithResolution` stamps a copy on the alert before it resolves. - The resolution rides the Alert JSON into the resolved callback, the delivery queue, the history row and the event log (reason code plus summary). - The synthesized closing evidence and the resolve transition say what happened instead of claiming a recovery projection. The cause stays `recovery_evidence`, the only cause operational trust lets enter `resolved`. - Notifications still send the close. Messages say "Memory alert moved to pve1 (Host Agent). This is not a recovery: check the agent for the current reading." in webhooks, `MOVED:` on a single ntfy push, and "Pulse alert moved" on email and Apprise. When any close in a delivery was not a recovery, `WebhookPayloadData.NotRecovered` drops recovery visuals: Discord's green embed turns neutral blue, both Teams cards lose recovery green, Mattermost's check emoji becomes an arrow, and ntfy uses the arrow tag for a single alert or a batch. Titles still say Resolved. PagerDuty still gets `resolve`. Every caller passes a nil resolution today, so ordinary recoveries and the disabled-threshold path are unchanged. The node release adopts the reason in a follow-up, once the host-agent node link records which metrics the agent evaluates (#2577 lane). Tests: `TestReleasedMetricAlertCarriesResolutionInsteadOfRecovery` and `TestResolvedHistoryRowKeepsResolutionWithoutEventLog` (operational_contract_test.go); `resolved_handover_test.go` drives the firing, its receipt and the queued close through the normal ntfy entry points, and renders Discord, both Teams cards, Mattermost, PagerDuty, a grouped Slack list, a grouped ntfy batch and email/Apprise content, with isolated managers. The full alerts, notifications and monitoring suites pass locally.	2026-10-06 18:57:09 +01:00
rcourtman	50b3fd8226	Merge remote-tracking branch 'origin/main' into fix/metric-alert-live-status	2026-10-06 18:55:48 +01:00
rcourtman	69979370e9	Show each identifier once in the drawer's Identity card (#2590) ... From a live user-lens pass over Machines. A Machines agent drawer's Identity card listed `agent:host-linux-1` as its Primary ID, its Discovery target and its Metrics Target. Its Aliases then repeated it again, along with the hostname and the Platform ID the card had just shown: eleven values, four of them the same ID. Every platform's drawer shares this card. A vSphere datastore repeated its ID the same way. **What changes** - **Discovery and Metrics Target** rows appear only when they name something not already on screen: the header's name (`getPreferredInfrastructureDisplayName`, which is what the header renders), the rows above, or the Platform ID row. - **Aliases** leave out those same values. - **Hidden names don't count.** Candidate names the drawer never shows don't count as shown, so a distinct target still appears. - **No false empty state.** A shown Platform ID counts as identity data, so filtering can't leave a "No identity metadata yet" message beside it. - **Display only.** Search and alias lookups keep the full lists. **Before and after (mock stack, same drawers)** - **Machines (Apollo-114)** - Before: Hostname · Primary ID · Discovery · Metrics Target (the last three identical) · Platform ID · IPs · Aliases: `agent:host-linux-1`, `host-linux-1`, `apollo-114`, `agent-acdf…` - After: Hostname · Primary ID · Platform ID · IPs · Aliases: `agent-acdf…` - **vSphere datastore (archive-tier)** - Before: Hostname · Primary ID · Metrics Target (same as Primary ID) · Platform ID · Tags · Aliases: Primary ID again, two short IDs, the hostname again, +1 - After: Hostname · Primary ID · Platform ID · Tags · Aliases: the two short IDs Checked live at 1440 and 390px. **Tests:** 1,278 tests pass. Three existing tests pinned the repetition and are updated. New tests cover: - an identity shown once; - a distinct target kept even when it matches a name the drawer never shows; - aliases filtered against the header's actual name; - a shown Platform ID counting as data. **Review:** a second reviewer found that my first version compared against names the drawer never shows, used a different header-name fallback, and could produce a false empty state. All three are fixed, and the reviewer confirmed.	2026-10-06 18:22:12 +01:00
rcourtman	1f627a7ff3	Keep the availability result readable and give names room (#2588) ... From a live user-lens pass over Machines → Availability checks. **The result clipped.** A check observed from several locations (Frontend Swarm service in the mock) showed "failed…" in its Result cell, with a "2 locations · 2/2 reporting" chip spilling past the column. The chip was given `flex` next to the badge's own `inline-flex`, which left it inline beside the result, so the result clipped at every width up to 1440px. The result and chip now share one line: - **The result** keeps its full width at the cell's right edge, where every row's result aligns. - **The chip** truncates to its left with its full text on hover. It now reads "2/2 locations reporting", the wording Settings uses for the same fact. - **Rows stay one line** because the table windows them at a single measured height. A second line would desynchronize scrolling on large estates. **Names were cut on phones.** Method and Target took the room, so check names read "Workshop do…" and "Frontend Sw…". Both now hide on phones, since the row drawer still shows them. The Method cell also truncates with an ellipsis instead of clipping. **Verification** - **Live, private mock stack**, at 390, 600, 768, 1024, 1440 and 1920px: - The result never clips. - Rows measure 32–33px at every width. - Phone check names read in full, except one row that is 3px short. - **Tests:** 3,977 tests pass across the standalone, Settings and shared suites, including new tests for the one-line result with a multi-location source and for the phone-hidden columns. - **Review:** a second reviewer flagged that my first version (chip on a second line) broke the table's uniform-height windowing. This one-line layout fixes it, and the reviewer confirmed.	2026-10-06 18:04:19 +01:00
rcourtman	66c8c63e79	Hand a node only the usage metrics its agent evaluates (#2581) ... This stacks on #2577. Once #2577 merges, the diff here reduces to the last commit. ## Problem `CheckHost` registered a host agent's node link before its disabled returns. With agent alerts switched off, or a host override disabling the agent, the node still released its CPU, memory and disk alerts to an agent that evaluated nothing, so nothing alerted on that machine's usage. The same happened per metric: an agent CPU threshold set to off with the node CPU threshold on left the machine with no CPU alert. The gap predates #2569; the old hostname dedup registered in the same place. ## Fix - The link now records which usage metrics the agent actually evaluated on its last report: a live CPU threshold, a live memory threshold with a known reading, and at least one filesystem with a live disk threshold. - `CheckHost` registers the link after evaluation and removes it on the disabled returns. - `CheckNode` releases only the metrics the agent owns and evaluates the rest itself. Deduplication can only remove a duplicate, never the last alert for a machine. - The link also carries the agent's display name and a stable lookup (`linkedHostAgentForNode`, lowest agent ID wins). The handover notification work builds on these. ## Tests `TestCheckNodeKeepsUsageMetricsTheAgentDoesNotEvaluate` covers two cases: agent alerts disabled (the node alerts on CPU and memory), and the agent CPU threshold off (the node keeps CPU while the agent keeps memory, so the machine still has exactly one memory alert). Both cases fail with the old registration order. The dedup tests use the new link struct. `go test ./internal/alerts/...` passes, and so do the monitoring alert, temperature, host-agent and node tests. ## Review follow-up (second commit) A review of the first commit found three more ways the node and its linked agent could disagree about who alerts: - **Memory ownership flipped with each report.** Ownership needed a known memory reading on every report. When the reading went missing while the agent's alert was open, the agent kept its alert but stopped owning memory, so the node raised a second alert. That alert resolved as soon as the reading came back. Ownership now follows the agent's configured thresholds, and a missing reading is treated as missing. - **Only one linked agent counted.** With several agents linked to one node, only the lowest agent ID's coverage was used. A metric is now owned when any linked agent evaluates it. - **Disk ownership covered the wrong filesystem.** A linked node's disk metric is the agent's summary filesystem: root when reported (`models.SummaryDisk`). Any evaluated mount was enough to take ownership, so with root disabled by a per-disk override and `/data` still alerting, a full root filesystem raised nothing. The agent now owns the node's disk metric only while it evaluates that same filesystem. `TestCheckNodeUsageOwnershipFollowsWhatAgentsEvaluate` covers all three cases; each fails without its fix.	2026-10-06 17:50:49 +01:00
rcourtman	f4c3040373	Read a linked agent's own heartbeat in host snapshots ... A host agent linked to a Proxmox node merges into one unified host row. When the agent stops reporting but PVE polling continues, that row stays online and its LastSeen follows the PVE sighting. hostFromReadStateView copied both into models.Host, so a silent agent looked current to every consumer of the projection: - linkedHostForNode accepted it, and resolveNodeDisk kept using the agent's last filesystem summary as the node's disk, feeding disk alerts and history with a frozen reading. Its retained ZFS datasets kept merging into the node's pools too. - Agent connections built from HostsSnapshot (the Connections list and the update-readiness agent check) aged the agent from the PVE poll, so a dead agent stayed Active. The projection now takes LastSeen from the agent source's own sighting and reports the agent offline once that sighting is past its reporting lease, the same hostAgentHealthWindow evaluateHostAgents applies. Rows without an agent sighting, such as vSphere and TrueNAS hosts, keep the row's values. The node falls back to the Proxmox rootfs reading while the agent is silent and returns to the agent summary when it reports.	2026-10-06 17:50:02 +01:00
rcourtman	267ffcc3af	Keep node temperature selection when details lack a CPU reading ... The first version of this change walked every temperature source and took the first usable one, which changed two cases that worked before. Details holding only NVMe or GPU readings displaced a valid scalar CPU reading, so the node lost its CPU temperature. An unusable earlier record no longer fell back to the scalar; the walk went on to a later record that was marked unavailable and suppressed the display. The record selection is back to exactly what it was. Only when it yields nothing usable does the adapter use proxmox.temperatureDetails, and only if that holds a current CPU reading; otherwise it falls back to the scalar as before, still without inventing a low or record. The contract notes now say the adapter forwards the poller's low and record values and that keeping them across polls is the poller's job.	2026-10-06 17:45:10 +01:00
rcourtman	bbd44455c8	Show retained disk temperatures as last known, not live ... Normalization keeps a disk's last known temperature when the current observation is not available: a disk in standby, or a linked host agent past its reporting lease. The Physical Disks table, the pool drawer's disk list and the disk drawer card still painted that value with the live threshold colour, so a silent agent's frozen 72C looked exactly like a disk running hot right now, and a sleeping disk looked freshly read. The drawer even printed "Temperature is temporarily unavailable" directly above a red Temperature card. One helper in diskPresentation.ts now decides: only an available temperature state, or a source that predates collection state, is a current reading. Anything else renders muted with no threshold colour, a dotted underline, the collection reason as its title and screen-reader "last known" text; the drawer card reads "Last known temperature". The pool drawer also formats through the shared temperature formatter, so it follows the Fahrenheit setting.	2026-10-06 17:41:25 +01:00
rcourtman	f419149458	Evaluate host agents in mock mode like live agents (#2585) ... ## Why `checkMockAlerts` evaluated mock nodes, guests, storage, PBS, PMG and Docker hosts but never host agents. Live agents are evaluated by `CheckHost` as each report lands; mock mode discards reports and the host health sweep never sees fixture agents, so no agent CPU, memory, disk, temperature or offline alert could open against mock data. A mock agent at 94% memory sat under the 85% default trigger with no alert, and any alert work on agents needed a private overlay build to reproduce. ## What Mock passes evaluate every fixture agent (`evaluateMockHostAgents`) with the live lifecycle boundaries: - Offline fixtures go through `HandleHostOfflineWithCorrelation`, because `CheckHost` treats its input as a fresh report and marks the host online. - Agents run before nodes, so a node with an online linked agent hands its CPU/memory/disk alerts to the agent on the first tick (host-agent hostname deduplication). - An agent that leaves the estate on a runtime mock config change goes through `HandleHostRemoved`, like a deleted live agent. - Leaving mock mode routes the fixture agents through `HandleHostRemoved`, releasing hostname registrations that `ClearActiveAlerts` leaves behind (they would otherwise suppress a real node named `pve1`). A pass already in flight finds mock mode off under the same lock and evaluates nothing. Evaluating agents exposed a mock-only reading: each Kubernetes pod took a whole single-pod footprint (10-83%) of its node's allocatable memory, so any node running four or more pods, and its linked agent, read a constant 100%. On the public demo's estate that became two permanent critical agent memory alerts. Pods now share 0.7 footprints per node (at most 58% in steady state), and a 60% per-node ceiling clips the transient while rescheduled pods' readings settle, so k8s nodes stay under 82%. Contract notes: `monitoring.md`, `performance-and-scalability.md`, `agent-lifecycle.md` (the `monitor.go` field touches only the alert-side removal boundary, with no tombstone, token or continuity change). ## Demo impact Measured on private stacks with the demo host's exact `PULSE_MOCK_*` estate (8 nodes, 2 standalone agents, 3 Kubernetes node agents): | | before | after | |---|---|---| | Kubernetes node agents' memory | 100%, 100% | 19-50% (up to ~76% briefly after boot) | | Agent alerts at default thresholds | none, ever | per-boot readings decide: one boot opened none, another opened Apollo-114 memory 91.3% (warning) | ## Verification - New tests, each shown to fail on the code it guards (via `go build -overlay`): `TestCheckMockAlertsEvaluatesHostAgentsBeforeLinkedNodes` (also fails with the agent loop after the node loop), `TestCheckMockHostAlertsUsesHostLifecycleForOfflineFixtures`, `TestLeavingMockModeReleasesFixtureAgentHostnames`, `TestMockHostAgentPassAfterLeavingMockModeRegistersNothing`, `TestMockHostAgentLeavingFixtureUsesRemovalLifecycle`, `TestMockKubernetesDenseNodesStayBelowAllocatableMemory`, `TestMockKubernetesRescheduledPodsDoNotSaturateReceivingNode`, `TestMockKubernetesLoneHeavyPodStaysUnderNodeCeiling`. - `go test ./internal/mock/ ./internal/monitoring/` pass. - Live, private demo-config stack: with the agent memory trigger at 60%, exactly the agents above it opened `agent:` memory alerts and their linked Proxmox nodes raised none; the Alerts page renders them. Shrinking the estate from 8 to 4 nodes through `/api/system/mock-mode` cleared the departed agents' alerts on the next pass. - Three rounds of independent review; the first two found the stale-registration, removal, relocation and steady-state ceiling issues fixed here, the third found none.	2026-10-06 17:38:10 +01:00
rcourtman	0a55939c72	Keep open row drawers spanning the table when columns change (#2571) ... An open inline row drawer measured its colspan from the summary row's visible cells only on mount and on resize. Hiding or showing a column with the drawer open, from View > Columns or through a data-driven column such as the vCenter column added in #2566, left the drawer with its old span. Reproduced on a mock stack before the change: - **Machines (`/standalone`), 1440px**: hiding Uptime with a drawer open left colspan 9 against 8 headers. The header and rows shrank to 939px of the 1375px table, beside a blank 436px column. Showing System left colspan 7 against 8, and the drawer stopped 79px short of the actions column. - **Proxmox guests (`/proxmox`), 1440/1024/768px**: hiding Type, Uptime or Backup left colspan one above the header count. The fixed layout collapses the phantom column to about 1px there. Fix, in the shared `InlineDetailTableRow`: - A MutationObserver on the summary row re-measures when cells are added, removed or hidden (class, style, hidden or colspan changes on a cell). Changes inside a cell's content are ignored, so live metric updates do not re-measure. - The span also re-syncs when the caller's requested colspan changes, which covers rows with no summary row to measure. - When the span grows, Chromium keeps a fixed-layout table's old column widths for the widened cell (minimal repro: 400px instead of 600px after a column is added and laid out). The row sets the cell's own width for one forced layout and restores it. Focus and inner scroll positions survive. After the change, on both tables at 1440 and 768px, the drawer colspan equals the visible header count after every toggle. The drawer, the last header cell and the last summary cell all end at the table edge, and the drawer row stays mounted. The vSphere Activity drawer still spans 6 of 6 columns. `InlineDetailTableRow.test.tsx` gains three tests that fail on main: removing and restoring a summary cell, hiding one in place, and changing the requested span with no summary row. It also becomes a registered frontend-primitives proof. The contract note sits beside the existing `InlineDetailTableRow` clause in `frontend-primitives.md`. Follow-up commits on this branch: - `InlineDetailTableRow.test.tsx` spies on `offsetWidth`. It requires exactly one forced layout read of the detail cell, at width 0px with the new colspan, when the requested span and the summary cells grow in the same update, and none when they shrink. A further test checks that a summary cell's colspan change re-measures while class and node churn inside a cell does not. Removing the forced read, doubling the workaround, removing it, or widening the observer filter each fails the matching test. - `tests/integration/tests/98-inline-drawer-column-span.spec.ts` is a real-browser check on the Machines table. It opens a drawer, hides and restores System through View > Columns, and asserts the following after each step: the drawer spans the visible headers, the drawer, last header and table share a right edge, and the drawer row, focus and any inner scroll survive. It fails on main before the fix (colspan 8 against 7) and with only the width workaround removed (drawer 79px short). It passed five runs in a row with the fix.	2026-10-06 17:37:34 +01:00
rcourtman	ee3ebd800e	Stop presenting a silent agent's node temperature as live (#2582) ... A host agent linked to a Proxmox node merges with it into one read-state row, and every PVE poll refreshes that row's `LastSeen`. The agent's last sensors stay in state after it stops reporting. The node temperature lookup (`getHostAgentTemperatureByID`) checked neither, so each poll merged the silent agent's last reading into the node as a fresh measurement, stamped with the row's fresh `LastSeen`: - an open node temperature alert was re-observed every poll, so its `LastSeen` kept moving and the 24h stale-alert cleanup never reached it; a stale hot reading could keep activating - node history recorded the frozen value as new samples, and the UI and reports showed it as current - SSH collection was skipped as if a recent agent payload existed Since #2569 evaluates node temperature for agent-backed nodes, this holds their temperature alerts open. ## Change - `HostView.SourceStatus(source)` exposes the per-source freshness the registry already records (same accessor the VM, container and physical-disk views have). - The linked-agent lookup reads the agent source's own sighting, uses the agent's sensors only inside the reporting lease that keeps the agent online (`hostAgentHealthWindow`, as `evaluateHostAgents` uses), and stamps the reading with the agent's report time. Past the lease the node has no agent reading (the cluster sensor cache is still consulted), and alert evaluation treats that as a missing reading, not as evidence the node cooled down. - The poller's carry-over of a previous reading after a failed collection is bounded to max(5 min, 2 × PVE polling interval) and keeps its original timestamp. This branch does not engage today: the previous-node projection carries the unified resource id and no temperature, so it never matches the polled node. That is filed separately; the bound keeps the carry honest once it's repaired. The owner is the poller rather than `checkNodeTemperature`: the same stale value also reaches history, reports and the UI. An alerts-side `LastUpdate` check would also have missed this case, because the reading was stamped with the merged row's fresh `LastSeen`. Mock fixtures are untouched. ## Tests - `TestSilentLinkedAgentStopsRefreshingNodeTemperatureAlert` (registered monitoring proof): registry-backed merged agent+node row through the production previous-node projection and `CheckNode`. A reporting agent opens the alert. A silent agent leaves the alert open but stops refreshing `LastSeen`. A resumed agent is evaluated again. It fails on origin/main. - `TestView_HostViewSourceStatusSeparatesAgentFromMergedRow`: a merged row's `LastSeen` follows PVE, while `SourceStatus(SourceAgent)` keeps the agent's own report time. - `TestCollectNodeTemperatureCarryIsBoundedByCarryWindow`, `TestHostAgentReportCurrent`, `TestGetHostAgentTemperatureByID_IgnoresAgentPastReportingLease`. Out of scope, filed separately: `mergeHostAgentSMARTIntoDisks` copies silent agents' retained SMART temperatures into physical disks.	2026-10-06 17:31:31 +01:00