mirror of
https://github.com/rcourtman/Pulse.git
synced 2026-10-24 12:42:49 +00:00
ff67e38968c02a7cec2bf3dd72d9afe2febc8227
6030 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Richard Courtman
|
ff67e38968 |
Pair a merged disk's temperature with its rows' own states
The registry merges a physical disk's rows one step at a time, choosing the temperature by source preference and merging collection states separately. The pairing rule from #2618 was exact for two rows only: with three or more, an earlier step's single merged state stood in for its rows. A silent agent, an offline Unraid inventory and a stale Proxmox copy could leave a withdrawn 41 labelled collected, and a legacy agent's stateless reading could pick up another row's "unsupported". The merged disk now keeps each row's current reading and state (merge-time only, never serialized), and the shown value takes its state from the readings of that value: a collected reading first, then the agent's own report over a copy. A copy of an agent source's state gives way to the agent's own withdrawal, including a legacy agent's sourceless one, and a value source preference keeps after its row reported again keeps its presented state under that row's later withdrawal. Value selection is unchanged. |
||
|
rcourtman
|
8b36dec0a2 |
Keep a source's own offline verdict through the stale pass (#2643)
## Problem The unified registry's stale pass recomputed a resource's status from its source sightings, and read a sighting that had *delivered recently* as *online*. A sighting only says the source delivered, not what it reported: - A Proxmox node the cluster reports offline on a live poll came back `online` once its linked host agent went silent, which is exactly what a powered-off machine's agent does (the #1441 / discussion #1135 symptom for users with an agent on the node). The Proxmox nodes table showed an amber dot next to its own "Offline" label while the drawer led with a critical "Node is offline" alert. - A node `preserveOrExpireNodes` expired after failed polls read `warning` ("Degraded"), and the stopped guests on it were counted under "Attention". Node alerts were not affected: `CheckNode` reads the poller's own `models.Node`. Everything that reads the projection was (`/api/resources`, websocket state, `NodesSnapshot`, Patrol and AI tools, connections grouping, reporting, charts). ## Fix Each sighting records the status its source reported (an unexported verdict that replaces the narrower lease marker from #2634), and `aggregateStatus` reads it: - While any source with a verdict is current, only the current sources decide, in the priority order the merge uses (agent, then Proxmox or vSphere, then Docker), so a live agent still keeps a node online when the Proxmox API path to it fails. - Once every source is quiet, an `offline` verdict survives and anything else reads `warning`, as before. - Sightings still report delivery, so health keeps its `telemetry_stale` reason. Availability checks projected onto a target keep counting by delivery, exactly as on main. Giving them a verdict needs each check's own freshness, and that is left to a follow-up (documented in unified-resources.md). ## Verification - Registry tests in `registry_merge_policy_test.go`: live poll reports offline with a silent agent (inside and past its lease), poller-expired node alone and with its agent, stopped and running guests on a quiet poll, JSON round trip, in-memory copy keeping a facet without a verdict, and an aggregation table. The node, poller-expired, guest and round-trip tests fail on main with the bug's exact symptoms; all of #2634's lease tests still pass. - Private mock stack (demo estate, one node held offline on a fresh poll with its agent silent): before, both offline nodes showed amber ("Offline"/"Degraded" pills) and 44 stopped guests counted as Attention; after, both read Offline and the guests moved to Stopped. A full status diff of ~1,780 resources showed no other systematic change. - `go test` for unifiedresources, monitoring, ai, alerts, mock, api, api/resourceapi and api/chartapi passes. |
||
|
rcourtman
|
8ad0cbbe26 |
Move static inline styles to classes the production CSP allows (#2642)
## Why
The production server sends `style-src 'self' 'nonce-...'` with no
`'unsafe-inline'` (`internal/api/security.go`). Solid compiles static
style values, such as `style="..."` or a literal entry of a `style={{
... }}` object, into template HTML. The browser reports each one as a
`style-src-attr` violation. Opening a row's inline detail on Machines or
Docker Images logged eight of them. The HelpIcon popover, the Assistant
typing dots and the alert threshold slider logged their own.
Chromium still applied the styles after Solid cloned the template, so
the visible effect there was console noise. Whether a stricter browser
applies them is not guaranteed. The Vite dev server allows inline
styles, so none of this showed up in development.
## What changed
- 24 drawer tab-panel `overflow-anchor` styles (ResourceDetailDrawer,
GuestDrawer, NodeDrawer, DockerHostDrawer, StoragePoolDetail,
DiskDetail) became the `[overflow-anchor:none]` class.
- The Assistant typing-dot delays became `[animation-delay:...]`
classes. `animate-bounce` already sets the 1s duration.
- ThresholdSlider uses `touch-none`.
- The HelpIcon popover keeps its dynamic position in `style` and takes
`min-w-[200px]` as a class.
- The boot-failure fallback in `index.tsx` is built from DOM nodes and
classes, which also stops the error text being parsed as markup.
- A new guard in `SharedPrimitives.guardrails.test.ts` compiles every
runtime `.tsx` that mentions `style` with the client-build Solid preset.
It fails on any `style` attribute or `<style>` element in the resulting
templates, nested `<template>` content included. The regex scans I used
first missed a mixed style object in HelpIcon; this check caught it.
- A note in `frontend-primitives.md` records the rule.
## Verification
- Production build of this branch embedded in a mock-mode backend that
serves the real nonce CSP, compared with an origin/main build on the
same setup. Checked at 1440x900 and 390x844, plus 1920/2560 for the
desktop thresholds table.
- Before: 8 violations per drawer open on `/standalone/machines` and
`/docker/images`, 3 on Docker host and Proxmox guest drawers, 1 on the
HelpIcon popover, 5 on the typing dots, 1 on the threshold slider.
- After: 0 violations on all of those, plus the Proxmox node drawer,
storage pool detail and disk detail.
- Computed styles match the old ones: `overflow-anchor: none` on every
visible panel, dot delays of 0/0.12/0.24s and 0/0.15/0.3s,
`touch-action: none`, popover `min-width: 200px`.
- Rows opened below the fold (drawer 215 to 351px past it) did not move.
Other drift readings were identical in both builds.
- Vitest: 933 files pass across the affected directories. The only
failure is `agentInstallCommand.test.ts`, which fails on macOS
regardless of this change.
- A new MessageItem test fails on origin/main and passes here.
|
||
|
rcourtman
|
ea1a07a721 |
Keep a source's own offline verdict through the stale pass
The registry's stale pass recomputed a resource's status from its source
sightings, and read a sighting that had delivered recently as "online".
A sighting only says the source delivered, not what it reported, so:
- A Proxmox node the cluster reports offline on a live poll came back
online once its linked host agent went silent, which is what a
powered-off machine's agent does. The Proxmox nodes table showed an
amber dot beside its own "Offline" label while the drawer led with a
critical "Node is offline" alert.
- A node the poller expired after failed polls read as a warning
("Degraded"), and the stopped guests on it counted as needing
attention.
Each sighting now records the status its source reported, and the
aggregation reads that verdict. Current sources decide in the same
priority order the merge uses, so a live agent still keeps a node online
when the Proxmox API path to it fails. A source that went quiet drops
out while another is current. Once every source is quiet, an offline
verdict survives and anything else reads as warning, as before. This
generalises the lease marker that kept a silent agent offline, so a
stopped guest keeps its offline verdict too when its only source goes
quiet. A running guest whose only source goes quiet still reads as
warning.
Sightings still report delivery, so health keeps its telemetry_stale
reason. An in-memory copy keeps its verdicts; only a copy that lost
them all, a serialized one, takes its stored status as each source's
verdict.
Availability checks projected onto a target still count by delivery, as
before. Giving them a verdict needs each check's own freshness, so it is
left to a follow-up.
|
||
|
rcourtman
|
8c1c467190 | Merge remote-tracking branch 'origin/main' into fix/drawer-inline-style-csp | ||
|
rcourtman
|
579ff5ca0f |
Move static inline styles to classes the production CSP allows
The production server sends style-src 'self' 'nonce-...' with no
'unsafe-inline'. Solid compiles static style values (style="..." or a
literal entry of a style={{ ... }} object) into template HTML, so every
such style is a style attribute the browser reports as a style-src-attr
violation. Opening a row's inline detail on Machines or Docker Images
logged eight of these, and the HelpIcon popover, the Assistant typing
dots and the threshold slider logged their own. Chromium still applied
the copied styles, so the visible effect there was console noise;
whether a stricter browser applies them is not guaranteed. The Vite dev
server allows inline styles, so none of this showed in development.
The 24 drawer tab-panel overflow-anchor styles become the
[overflow-anchor:none] class, the typing-dot delays become arbitrary
animation-delay classes (animate-bounce already sets the 1s duration),
the slider uses touch-none, the HelpIcon popover keeps its dynamic
position in style and takes min-w-[200px], and the boot-failure
fallback is built with DOM nodes, which also stops the error text being
parsed as markup.
SharedPrimitives.guardrails.test.ts now compiles every runtime .tsx
that mentions style with the client-build Solid preset and fails on any
style attribute or <style> element in the resulting templates, nested
<template> content included. The regex scans first used to find these
missed a mixed style object in HelpIcon; the compiler-backed check did
not.
Contract-Neutral: Static inline styles become equivalent classes; only frontend-primitives gains a rule, recorded in its contract
|
||
|
rcourtman
|
79b6a0974e |
Name disk wearout as life remaining and log disk alerts once
The Proxmox disk wearout alert read "SSD has less than 10% life remaining (7% wearout)". The number is percent life remaining, so "7% wearout" reads as 7% worn and contradicts the first clause, and the Alerts overview shows that text verbatim. It now reads "SSD life remaining is 7%", the wording host-agent SMART wearout alerts already use. The threshold stays in the alert's Threshold field, which the card and notifications show beside the message; spelling it into the sentence as "below 10%" would be false at exactly 10%, which alerts. CheckDiskHealth also logged a failed disk at error and a worn disk at warn on every evaluation, not when the alert opened. The physical disk poll runs every 5 minutes by default, so one failed disk logged an error 288 times a day for as long as it stayed failed, and mock mode repeated it on every alert tick. Both now log at error or warn only on the evaluation that opens the alert and at debug afterwards. The new log test captures through a hook installed in TestMain: assigning log.Logger inside a test would race with managers that earlier tests leave running. The alerts contract also said wearout recovery needs readings of at least 10%; a 10% reading still matches, so recovery needs readings above it. |
||
|
rcourtman
|
a21e0ef8f8 |
Stop reports and resource history calling a moved alert recovered (#2607)
Follows #2591. A node alert that moves to its linked Pulse agent closes with a `moved_to_agent` resolution (#2580). Two readers still presented that close as a recovery. **Performance reports.** The node, VM and container enrichers copied recently resolved alerts into `reporting.AlertInfo` without the resolution, so a report generated in the five-minute window after a handover read HEALTHY, "All systems operating normally", counted the alert as "triggered and resolved", and sent the AI narrator `resolved: true`. - Report rows now carry the alert's resource and resolution (`reportActiveAlertInfo` / `reportResolvedAlertInfo`). - `pkg/reporting` has one rule (`unresolvedAlerts`): an alert that closed without recovering stays live unless the report also shows its successor's reading for that metric, meaning an open alert or a recovery since the handover. That way the condition is counted once, and fleets match handovers across all their resources. - The single-resource PDF card reads WARNING (or CRITICAL at the alert's level) with "Alert moved to <agent> - check the agent for the current reading". The active list names the agent, the alerts table shows "Moved" with the engine's summary as a note, and recommendations say to check the agent. - Fleets count the alert as active and name the agent. Their all-clear pattern and "No fleet-wide action required" no longer appear while any alert is active. - The narrator payload carries `resolved: false`, the summary and `successor_alert_listed`, with prompt rules for both. **Resource history.** `alert_resolved` changes were labelled by kind alone ("Alert resolved", green) with that label prefixed to the headline. `getResourceChangePresentation` now reads the `alert_resolution` metadata: the change reads "Alert moved" in the neutral blue of the history badge, and its summary is the whole headline in the drawer, `ResourceChangeSummary` and Patrol's assessment handoff. **Verification** - A private mock stack running this branch, with a local-only overlay that closes a real pve1 memory alert through `clearAlertWithResolution`. Within the window, `/api/admin/reports/generate?resourceType=node` rendered CRITICAL with the handover message and a "Moved" row, while the ordinary CPU recovery stayed green. - In the browser, a full drawer (Availability checks, Fleet view) showed the producer-made rows as "Alert moved" next to "Alert resolved" at 1280 and 375. A Proxmox node's own drawer uses the `table-row` presentation, which fetches no change history (see below). - Tests in registered proof files: `pkg/reporting/alert_state_test.go` (verdict, successor timing, fleet pool, PDF text and width), `internal/api/contract_test.go`, `internal/ai/report_narrator_test.go`, `resourceChangePresentation.test.ts`, `ResourceDetailDrawer.history.test.tsx` and `patrolInvestigationContextModel.test.ts`. Each failed with the fix reverted. - Second opinion: two read-only review rounds. The temporal match, fleet-wide pool, narrator dedupe signal, fleet all-clear fallbacks and contract wording came from them. **Not fixed here (separate changes)** - Reports requested as `agent` carry no alert rows at all. That is how the Reports page addresses every Proxmox node in v6, so such a report reads HEALTHY even with active critical alerts. This change makes the shared rule correct for when those reports attach the machine's alerts. - Node and guest alert lifecycle changes are recorded under legacy alert resource IDs, which canonical history queries never match. Facet kind counts still group moved closes under "Alert resolved". |
||
|
rcourtman
|
a570d9f335 |
Say the alert card's 16rem floor covers its icon column
The frontend-primitives note said the card's text column keeps a 16rem floor, but the floor sits on the left column that holds the status icon and the text, so the text itself gets about 230px before wrapping. |
||
|
rcourtman
|
097858bc41 | Merge remote-tracking branch 'origin/main' into fix/alert-card-live-reading | ||
|
rcourtman
|
bea0cec8b9 |
Keep PVE disk alert references out of resource history (#2627)
PVE disk health and wearout alerts reference the disk by instance, node and device path (`<instance>:<node>:disk:<device key>`). Operator mutes resolve that reference to the disk at the path (#2112), but resource history leaves it unbound, so a physical disk's history (facets, timeline and intelligence APIs, Assistant resource context) does not show its own SMART and wearout alerts. This records why that stays the decision, and pins it with a test. - A device path is not hardware identity. A replacement disk in the same slot, or a reboot that reorders devices, takes over the path, and disk alerts are evaluated before their poll's disks reach the registry, so an event can resolve against a generation that still places another disk there. - A history binding belongs to the reference string and moves to its latest target, carrying every row journaled under the reference and every read of it. Bound, the path would put a replaced disk's alerts into its successor's history, and the Alerts history Resource action (which reads incidents by the alert's own reference) would show the successor's incidents for the predecessor's alert. - A binding guarded by the serial each disk alert records was built and measured on a mock estate (mock mode needed a build overlay to evaluate PVE disk alerts at all). It bound correctly while disks kept their paths, but review found rejected events still joining the currently bound disk, a removed retained target accepting a successor's event, no WWN in the alert metadata, a non-atomic check and write, and retry expiry abandoning rows. That attempt is not in this PR. Binding these alerts needs ownership per row (each event under the disk its recorded hardware identity names, rows without usable identity left under the reference, and a decided read for the Resource action), not a per-reference binding. The contract paragraph in `unified-resources.md` says so. Test: `TestHistoryIdentityLeavesProxmoxDiskAlertReferencesUnbound` (live event and backfilled legacy row stay under the reference and out of the disk's history, while `ResolveReferenceID` still resolves the reference for mutes). It fails against the reverted binding implementation. |
||
|
rcourtman
|
458ee510c1 |
Replace stored alerts with each payload instead of merging (#2631)
The websocket store wrote each incoming active alert with `setActiveAlerts(id, alert)`. Solid's path setter shallow-merges into the existing object, so any field the server stops sending kept its old value in the browser: - After a backend restart, threshold alerts are restored without `metricStatus` until their next evaluation (and indefinitely while the reading is missing). An open tab kept the pre-restart live reading, so the node drawer's "Needs attention" row and the node table temperature tooltip went on saying "Temperature 78°C now" for a reading the server no longer reports. - An unacknowledge from another session left `ackTime`/`ackUser` behind, as did a dropped `nodeDisplayName` or `metadata`. Changes in `frontend-modern/src/stores/websocket.ts`: - Active and recently resolved alerts are written with `reconcile`, so omitted keys are removed on the snapshot, keyed-delta and REST recovery commits alike. `reconcile` diffs nested values in place, so an unchanged alert's live status no longer re-notifies keyed-store readers on every frame (the shallow merge swapped in a fresh nested object each time). - `updateAlert` stores a copy of each local update, because in-place reconciliation would otherwise rewrite the record a snooze keeps for rollback (its optimistic record shares nested objects with it). - The pending acknowledgement hold is unchanged. Tests (`websocket-unified.test.ts`): omitted `metricStatus`, `nodeDisplayName` and `metadata` clear via snapshot, keyed delta (`null` patch) and REST recovery; a server-side unacknowledge clears `ackTime`/`ackUser`; local acknowledge/unacknowledge still wait for confirmation; the rollback record stays intact; an unchanged alert's nested reader is not re-notified; recently resolved alerts clear omitted fields. The omitted-field, unacknowledge, reactivity and resolved tests fail on the previous store. The rollback test fails if the `updateAlert` copy is removed. The acknowledgement-hold test passes on both and guards the existing behaviour. Browser verification on a private mock stack at 1280×900 and 375×812, with a local-only overlay that pins a standalone node's evaluated temperature and lets mock mode restore alerts across restarts. Latch at 78°C (breach 95), restart the backend with the reading missing (`/api/alerts/active` returns the alert without `metricStatus`), then restore the reading: - This change: the open tab's drawer and tooltip drop to "Node temperature at 95.0°C" within seconds of reconnecting, then show "78°C now" again on the next delta. - Previous store, same run: the open tab kept "Temperature 78°C now" for the full two-minute window. - Snooze, resume and acknowledge on the Alerts card work with no page errors. A remote acknowledge/unacknowledge round trip updates the open tab within 2s. Seen during verification and not changed here (it reproduces on main): after acknowledging from the Alerts card, an unacknowledge from another session does not bring the card back in that tab until a reload. The cause is the acknowledgement override in `useAlertAcknowledgementState`. It is tracked as a separate fix. |
||
|
rcourtman
|
f4ddeeb0e5 |
Give Docker containers and services without IDs their own alert refs (#2629)
Follow-up to #2620, which documented both conflations below as residuals. ## Problem Two Docker alert producers wrote names into identity slots: - A container reported without an ID alerted under its Docker host's own reference, `docker:<host>`. Its alerts showed on the host, joined the host's resource history (which #2620 binds to the host), were never cleared when the container left the report (cleanup matches `docker:<host>/`), and Patrol registered `docker:<host>` on both the host and the container, so the host's scope alias was ambiguous. - A Swarm service without an ID put its normalized name in the `/service/<ID>` slot, so a name equal to another service's Swarm ID shared that service's alerts and history. #2620's resolver had to treat every service in such a cluster as a conflict. Pulse's own agent always sends Docker and Swarm IDs, so this needs a malformed or third-party report. The registry still shows those containers and services. ## Change - `internal/alerts`: `DockerHostResourceID`, `DockerContainerResourceID(host, id, name)` and `DockerServiceResourceID` are the only builders. An ID-less container alerts under `docker:<host>/name:<name>` and an ID-less service under `docker:<host>/service/name:<normalized name>`. Docker and Swarm IDs never contain a colon, so neither can equal a host or ID reference. One with neither an ID nor a name raises no alert. - The container and service policy switches classify by the first path segment after the host, so a container name containing `/service/` stays a container. - Report cleanup, host removal and the host's offline alert also clear a pre-upgrade container alert left on exactly `docker:<host>` (recorded host must match, container ID empty). A pending image update recorded there keeps its age when the first report still shows it pending. - Canonical state IDs were split at the first `::`, but colliding Docker hosts get IDs like `<base>::<suffix>`. Every container alert of such a host was rewritten to `docker:<base>`, the other host's reference: it joined that host's history and missed its own host's cleanup. `splitCanonicalStateID` now splits before the alert's recorded spec ID, else after its resource ID, so alerts saved the old way also recover on restore. - Patrol registers the same builders' output as scope aliases, with the container's name. - History: a service slot containing `:` is a name reference (never binds, never retried), and the cluster-wide conflict for ID-less services is gone. A real service ID reference now binds even when its cluster also has an ID-less service. - Contract notes in alerts, ai-runtime, monitoring and unified-resources. Remaining legacy boundary (in the unified-resources contract): rows journaled before this change keep their old reference. An ID-less container's old rows stay in its host's history, and an ID-less service's old rows bind only to a service whose ID equals its normalized name. ## Proof - `TestDockerAlertsWithoutIDsUseTheirOwnReferences` (real `CheckDockerHost`: fired and active references, removal cleanup, pre-upgrade leftover on report and offline, config-save container/service switches). - `TestDockerHostIDWithSeparatorKeepsItsAlertReferences`, `TestDockerAlertsRestoredWithCutShortReferencesRecover` (writes `active-alerts.json`, restores through `LoadActiveAlerts`) and `TestDockerContainerWithoutIDKeepsPendingUpdateAgeAcrossUpgrade`. - `TestOwnerAlertTimelinesUseCanonicalHistoryIdentity`: real producers into the SQLite history. The ID-less container and the service named like `web`'s Swarm ID stay out of the host's and `web`'s histories. - `TestResolvePatrolScopeResolvesDockerAlertResourceIDs`: the host's alias stays unambiguous, and the ID-less container's name reference resolves to the container in the unified read state. - `TestHistoryIdentityMonitorAdapterResolvesDockerHostAndServiceReferences`: the ID reference binds before and after the service leaves inventory, and name references never bind or retry. - Mutation check: putting back each old behavior fails at least one of these tests. Covered: host fallback, unmarked service name, no skip for unidentifiable containers or services, each legacy clause and its host check, the update-age carry-over, first-`::` splitting with and without the spec rule, substring classification, the Patrol alias without the name, and the resolver accepting a `name:` slot. |
||
|
pulse-triage[bot]
|
44569ddbf0 | Merge branch 'main' into fix/active-alert-store-replace | ||
|
pulse-triage[bot]
|
809629954d | Merge branch 'main' into fix/docker-alert-unique-refs | ||
|
pulse-triage[bot]
|
d4ab4a2324 | Merge branch 'main' into fix/pve-disk-alert-history | ||
|
pulse-triage[bot]
|
c48289160d | Merge branch 'main' into fix/workload-hover-hint-reflow | ||
|
rcourtman
|
e3e743b076 |
List disk heat beside TrueNAS alert text
The TrueNAS storage table's Health reasons put native TrueNAS alert text first and used Pulse's derived summaries only when TrueNAS raised no alert. Disk heat landed in that fallback, so a hot disk with any incident, even an informational one, showed only the incident's text under Attention, and nothing said the disk was hot. Heat is Pulse's own judgement against the alert disk temperature thresholds, not a restatement of TrueNAS state, so it is now always listed after any native alert text. |
||
|
rcourtman
|
396bb5d517 |
Keep a disk heat finding open without a current reading
Patrol verifies a disk-high finding as recovered once the disk's temperature cools to its alert clear value. A disk in standby, or one whose host agent stopped reporting, has no current reading, and a missing reading counted as cooled. A finding raised for a SATA disk at 56C would close as soon as the disk spun down, while its temperature alert stayed open. When a disk has no current reading and its last known reading is still at or above the clear value, verification now reports the recovery as unknown instead of proven. A disk last seen cool still recovers. |
||
|
rcourtman
|
35bf55af23 |
Colour report disk temperatures by the alert thresholds
The PDF performance report's disk table coloured a temperature amber from 50C and red from 60C for every disk type, the last fixed disk heat line after disk heat moved to the alert disk temperature policy. An NVMe at 63C, well under its 70C alert trigger, printed red. A user's raised trigger changed nothing. Each reported disk now carries the tenant's alert thresholds for its type (the clear value and the trigger, from alerts.Manager.DiskTemperatureThreshold). The PDF colours the reading amber from the clear value and red from the trigger, as the Physical Disks Temp column does, and leaves it plain when disk temperature alerting is off. |
||
|
rcourtman
|
3fd5411ecc |
Judge disk heat by the alert disk temperature thresholds
Pulse decided whether a physical disk was too hot in three places with three thresholds. Disk risk added temperature_high at a flat 60/70C for every disk type and drove the disk status and the Physical Disks Health verdict. Patrol flagged anything over 55C and the AI chat context listed disks from 50C. Alerts and the Temp column used the per-type thresholds users tune in Alerts (NVMe 70, SAS 65, SATA 55 by default). An NVMe at 63C read Running Hot beside a green Temp cell. A SATA disk at 56C read Healthy beside a red one. A user who raised the NVMe trigger to 75 still got a hot verdict at 70. The alert disk temperature policy now decides disk heat. alerts.Manager.DiskTemperatureThreshold exposes the per-type resolution CheckHost already applies. - Disk risk no longer judges temperature. Every registry that rebuilds risk lacks the alert configuration, so no rule there could honour it. A hot disk keeps an online status. - The Health verdict reads Running Hot when a current reading reaches the disk type's trigger. It uses the thresholds object that colours the Temp cell, so the two cannot disagree. The amber band below the trigger stays a cell colour, not a verdict. - The TrueNAS storage table lost the heat it used to get from risk. It now judges heat the same way: an Attention row with the reason, and a warning dot for phones. - Patrol triage, its disk issue filter, finding recovery and the chat context's attention list judge the collected reading against the same trigger, including scoped runs and a provider set before Patrol starts. - A switched-off agent Disk Temp default resolves to no thresholds, which the displays that read them now treat as off. Per-host Disk Temp overrides do not reach these surfaces yet. TrueNAS disk alerts, the TrueNAS disk drawer and PDF reports keep their own fixed temperatures for now. |
||
|
rcourtman
|
1b6a9b45d7 |
Keep retained disk temperatures out of AI, Patrol and reports (#2625)
Normalization keeps a disk's last-known temperature when the current observation did not collect one (a disk in standby, a host agent past its reporting lease) and marks it non-available. These readers still took `PhysicalDisk.Temperature` at face value; each now goes through `diskinventory.TemperatureCollected`: - **AI chat context**: "Physical Disks Needing Attention" no longer flags a disk for heat from a retained reading. A disk listed for another reason shows it as `last known 52C (host agent stopped reporting)`. - **Patrol**: the disk health-issue gate, triage flags and `disk-high` finding verification judge only a collected reading. Both seed tables label a retained one as last known. - **AI tools**: `pulse_query` / `pulse_metrics` physical disks and `pulse_storage disk_health` put a retained value in `last_known_temperature` + `last_known_temperature_reason` instead of `temperature`. - **Reports**: the performance report's disk table and the reporting runtime snapshot leave a retained value out (no reading). - **Disk temperature charts**: a short series is padded out to now only with a collected reading. Otherwise stored samples are returned as they are. - `monitor.go`'s private copy of the predicate is replaced by the shared helper. Each new test reproduces against the pre-change code: retained `unavailable` temperatures (standby / silent agent) beside a collected one. |
||
|
rcourtman
|
684c0dc598 |
Show a host whose agent stopped reporting as offline (#2634)
A host agent that stops reporting is never shown as offline in the Machines table. Its row renders its last report as if it were current. ## Why The monitor marks a host agent offline once its reporting lease expires, and raises "Host is offline". The registry's stale pass marks the agent sighting stale after 60 seconds, and `aggregateStatus` ranked stale above offline. So the machine reached every consumer as `warning`. On the Machines table that meant an amber dot. CPU, memory, disk, network, disk I/O, uptime and temperature all rendered as live, with threshold colours. Meanwhile the row's own drawer led with the critical offline alert and read "Observed state: Degraded". Docker hosts and Kubernetes clusters hold the same kind of lease, and theirs is shorter than the 120-second stale threshold. A silent Docker host therefore read offline at first, then flipped to warning once its sighting went stale. ## What changes - **Backend.** Each sighting records when a push reporter delivered its machine, Docker host or cluster as offline. The reporters are the host agent, Docker and Kubernetes. Status aggregation counts that sighting as offline whatever its age. - The sighting itself still reads `stale`, so health keeps its `telemetry_stale` reason and monitored-system reasons are unchanged. - A live source still carries a merged row. A Proxmox node whose agent died stays online through the PVE poll. - Pull sources, and the guests, containers, pods and disks that push collectors report about, keep the stale-to-warning rule. - A JSON copy re-derives the marker from its stored status. - A manual link joining two resources from one source keeps the fresher sighting. - **Frontend.** Offline rows already blank CPU, memory, disk, network and disk I/O through the danger gate. - The Machines Uptime and Temp cells and the Docker hosts Temp cell had no gate, and now use it. - The drawer drops an offline resource's uptime and a silent agent's uptime. - The drawer marks a silent agent's non-disk Thermals rows "(last known)" with the reason as title, as retained disk rows already are. ## Verification - Go tests cover the agent, Docker, Kubernetes, mixed-source, Proxmox-node, JSON round-trip and manual-link cases, plus the unchanged late-agent and stale-PVE boundaries. Each fix's test fails without it. The `unifiedresources` package passes. - The `api`, `monitoring`, `ai`, `alerts` and `agentcontext` suites ran on pulse-dev. One `internal/api` test failed there, `TestServerInfoEndpointReportsDevelopment`, and it fails identically on the pre-fix build in that scratch tree. - Frontend tests cover the Machines offline row, the Docker offline Temp cell, the drawer thermals marking and the drawer uptime rows. The adjacent suites pass. - Live check on a private mock stack, holding one standalone Linux agent and one Docker host past their leases, at 1440 and 390 wide. Before: amber row with live-looking readings. After: red row with every reading blanked, the drawer reading "Offline", and thermals marked last known. - The browser receipt is included. Bundle sizes are within budget: WorkloadsSurface is unchanged, Standalone +0.04 kB, Docker +0.01 kB. Builds on rcourtman/Pulse#2630 and updates its contract sentence, which described the lapsed agent as reaching the table as `warning`. Known follow-up: the drawer's Disks card still threshold-colours a silent agent's retained filesystem usage. |
||
|
rcourtman
|
fb55d47517 |
Raise disk health alerts for mock physical disks (#2633)
`checkMockAlerts` evaluated guests, agents, nodes, storage, PBS, PMG and Docker against the mock estate but never physical disks. The mock generator deliberately keeps a FAILED health cohort and worn SSDs stable across restarts, so mock and demo stacks showed failing and "Replace Now" disks with no `disk-health` or `disk-wearout` alert, and the disk alert lifecycle could not be exercised without real hardware. ## Change - `checkMockAlerts` now evaluates fixture disks through `checkMockPhysicalDiskAlerts`, applying the physical disk poller's boundary: only disks on online nodes (matched by instance and node name), with the linked agent's `--disk-exclude` patterns forcing a matched device healthy. - The poller's per-disk rule (exclusion forces healthy, else `CheckDiskHealth`) moves into `checkPhysicalDiskAlerts`, shared by `maybePollPhysicalDisksAsync` and the mock pass. The poller's behaviour is unchanged; the source guardrail `TestProxmoxDiskAlertsRunOnMergedDiskState` now pins the helper call. - The poller's host-agent link settle deferral (#1674) is not applied in mock mode: fixture links and exclusions are complete from the first pass. ## Demo-facing effect Measured on private stacks with the public demo's exact estate (`/etc/pulse/.env`: 8 nodes, `DEMO_MODE`), baseline vs patched: the demo estate has 12 Proxmox disks and none FAILED. The only new disk alert is one warning-level `disk-wearout` ("SSD has less than 10% life remaining (7% wearout)" on West Production E), whose resource ID is one of the disk resource's canonical aliases. No `disk-health` alert opens on that estate. The default test estate (50 nodes) raises five `disk-health` and one `disk-wearout` alert. ## Known gaps, left out of scope - Like the other loops in `checkMockAlerts`, the disk loop relies on the pass's single up-front mode check, so a pass in flight when mock mode is switched off can reopen alerts after `ClearActiveAlerts`. That race predates this change and also involves the frontend refresh and backup rollup paths, so it needs a monitor-wide fix rather than a lock around this pass alone. - `CheckDiskHealth` logs at error/warn on every evaluation of a FAILED or worn disk, so in mock mode such a disk logs once per alert tick. - The wearout message calls 7% life remaining "7% wearout". ## Verification - `TestCheckMockAlertsEvaluatesPhysicalDisks` (pinned default estate) and `TestCheckMockPhysicalDiskAlertsFollowsPollerBoundary` (offline node, same node name on another instance, exclusion resolving existing alerts, node departure cleanup). Mutations that drop the pass, ignore offline nodes, ignore exclusions, or key by node name only each fail a test. - Full `go test ./internal/monitoring/` and the mock tests under `-race` pass. |
||
|
rcourtman
|
efd3b29ac8 |
Poll only the history each metric table renders (#2632)
`useWorkloadTableMetricHistory` ran two polled estate-wide reads per instance in Trends mode: the workloads history (`/api/charts/workloads`) and the infrastructure summary (`/api/charts/infrastructure`). Since #2622 removed the host metric cells from grouped workload rows, the Workloads surface reads only guest series, and the Proxmox nodes table never read anything but node series. Two instances never shared reads either: each forwards its own abort signal, which skips the summary caches' in-flight dedupe, and retained query values only seed matching mounts and source changes without sharing requests. So every reader paid for both routes every 30 seconds. The hook now takes a required `series: 'guests' | 'nodes'` option. The Workloads state passes `'guests'` and the Proxmox nodes table passes `'nodes'` (and drops the `selectedNode` it never needed), so each reader polls one route. ### Measured Private demo-mode mock stack, Trends on, 95 s per page (first read + three 30 s polls), reads per poll tick: | Page | Viewport | Before (infra + workloads) | After (infra + workloads) | |---|---|---|---| | /vmware | desktop 1440x900 | 1 + 1 | 0 + 1 | | /vmware | phone 390x844 | 1 + 1 | 0 + 1 | | /proxmox/overview | desktop 1440x900 | 2 + 2 | 1 + 1 | | /proxmox/overview | phone 390x844 | 2 + 2 | 1 + 1 | Node table sparklines on /proxmox/overview still draw for every row (36 on desktop, 18 on phone), and the guest table sparklines are unchanged. ### Tests - `useWorkloadTableMetricHistory.test.tsx`: a `'guests'` reader polls only the workloads route and a `'nodes'` reader only the infrastructure route, across two polls. Both cases fail with the gating reverted. - `ProxmoxNodesTable.test.tsx`: the table asks for node history only, with no node scope, and enables it only in Trends. - `WorkloadsSurface.performance.contract.test.tsx` pins both call sites and the gating. The performance contract claimed the two readers deduped their fetches; it now says they do not and that each reader polls only what it renders. The adjacent Bars-mode prose (performance and frontend-primitives contracts) described one active-row request that leaving the row cancels; it now describes the bounded warming window the hook has actually run since the history lens shipped. Stacked on #2622, whose auto-merge is already on: this branch contains its commits and merges cleanly with main either way round. |
||
|
rcourtman
|
4c4e6e7c57 |
Call a disk that is only hot Running Hot, not Replace Now (#2623)
On Physical Disks, a SMART-passed disk with good life running at 72°C
read **Replace Now** ("Disk temperature is 72C"), and **Replace** on
phones. Disk risk grades temperature (`temperature_high`, ≥70°C
critical, ≥60°C warning), and the Health verdict mapped any critical
risk to Replace Now. That tells a homelab user to buy a disk when the
fix is airflow. Alerts already keep heat separate: the SMART health and
wearout alerts skip `temperature_high`, and disk heat is raised as a
Temperature metric.
## Change
- `diskPresentation.ts` keeps each risk reason's code and severity, and
splits the reasons into heat and everything else.
- FAILED health, any other critical reason, or a critical level that no
reason explains → **Replace Now** (unchanged).
- Heat alone → **Running Hot** (phone **Hot**). Red at the critical
tier, amber at warning. The temperature reason is still the cell title
and the drawer summary.
- Other warning evidence, SMART counters or low life → **Needs
Attention** (unchanged).
- The more severe class names the verdict, and failure evidence wins a
tie. Merged risks that list heat first (`mergePhysicalDiskRisk` does not
re-sort) no longer hide pending sectors behind a temperature summary.
- Sort priority and the health filter derive from the same verdict.
Order is Replace Now, then red Running Hot, then Needs Attention, then
amber Running Hot. Hot disks stay in **Needs attention** and land in
Critical or Warning by their tier.
- Data with no reason codes keeps the old behaviour.
## Not changed
Thresholds. The tier still comes from the server's flat 60/70°C rule,
not the per-type alert thresholds that colour the Temp cell. An NVMe at
60–64°C therefore reads amber Running Hot next to a green Temp cell.
That disk read Needs Attention before this change, so the judgment is
the same and only the word is clearer. Reconciling the thresholds is
being done separately.
## Verification
- New tests in `diskPresentation.test.ts` cover heat at both tiers,
failure evidence listed after heat, heat plus wear, an unexplained
critical level, sort order and the health filter. Five of them fail on
the old code, including the sort test, whose device names make name
order alone disagree. `DiskList.test.tsx` checks the full and compact
words, tone, title and row order.
- Targeted suites (storageBackups, Storage, GuestPhysicalDisks,
frontendResourceTypeBoundaries): 89 files, 1513 tests pass. tsc and
eslint are clean, and the canonical platform audit is clean.
- Browser check on a private mock stack (local, uncommitted generator
overlay pinning pve3 disks at 72°C, 63°C, and 72°C with 2 pending
sectors), at 1440×900 and 390×844. The rows read Replace Now / Running
Hot / Needs Attention / Running Hot (phone: Replace / Hot / Attention /
Hot) with the expected tones and titles. The drawer reads "Running Hot ·
Disk temperature is 72C". The `status=attention|critical|warning`
filters list the expected rows. A same-session control with the old file
showed the 72°C disk as Replace Now. The receipt is
`frontend-modern/browser-verification/259ae9ced29e073f.json`.
|
||
|
rcourtman
|
c06af69fcb |
Show a host whose agent stopped reporting as offline
A host agent past its reporting lease is offline: the monitor marks it so and raises "Host is offline". The registry's stale pass ranked the agent's stale sighting above offline, though, so the machine reached every consumer as a warning. The Machines table showed it amber and rendered its last CPU, memory, disk and temperature as current readings, while its own drawer led with the critical offline alert. A Docker host was offline only until its sighting crossed the 120-second stale threshold, then flipped to warning, and Kubernetes clusters likewise. Each sighting now records when a push reporter (host agent, Docker, Kubernetes) delivered its machine, Docker host or cluster as offline, and status aggregation counts that as offline whatever the sighting's age. The sighting itself still reads stale, and a live source still carries a merged row, so a Proxmox node with a dead agent stays online. A copy that went through JSON re-derives the marker from its stored status, and a manual link joining two resources from one source keeps the fresher sighting, so a retired agent cannot pull a live one down. Offline rows then take the existing danger gate. The Machines Uptime and Temp cells and the Docker hosts Temp cell had no gate and now follow it. The drawer drops a silent agent's uptime and marks its non-disk Thermals rows "(last known)", as retained disk rows already are. |
||
|
rcourtman
|
3c1464e42f | Merge remote-tracking branch 'origin/main' into fix/disk-temp-consumers | ||
|
rcourtman
|
344f1f8531 |
Raise disk health alerts for mock physical disks
checkMockAlerts evaluated guests, agents, nodes, storage, PBS, PMG and Docker against the mock estate but never physical disks. The fixture keeps a FAILED cohort and worn SSDs stable on purpose, so mock and demo stacks showed failing and "Replace Now" disks with no disk-health or disk-wearout alert, and the disk alert lifecycle could not be exercised without real hardware. The pass applies the physical disk poller's boundary: only disks on online nodes, matched by instance and node name, with the linked agent's --disk-exclude patterns forcing a matched device healthy. The per-disk evaluation moves into checkPhysicalDiskAlerts so the poller and the mock pass share one rule. The poller's agent-link settle deferral is not applied, because fixture links and exclusions are complete from the first pass. Like the other loops in checkMockAlerts, the disk loop runs after the pass's single mode check, so the existing race with leaving mock mode now covers disk alerts too. That race also involves other fixture evaluation paths and is left for a monitor-wide fix. On the public demo's estate (8 nodes) this adds one warning-level disk-wearout alert and no disk-health alert: that estate has no FAILED Proxmox disk. |
||
|
rcourtman
|
8732bc1923 |
Remove the unreachable host metric cells from grouped workload rows (#2622)
Both production mounts of the Workloads surface, the Proxmox and vSphere overviews, set `compactGroupHeaders` unconditionally. The non-compact branch in `WorkloadPanel` (per-column node CPU, memory, disk and I/O bars or sparklines, uptime, version, temperature) and the `NodeGroupHeader` column mode and inline host facts that only it reached never rendered. They were left over from the retired standalone Workloads page. This removes the option, the dead renderers and the `NodeGroupHeader` props that only served them. Grouped host rows keep the shape both pages already render: one spanning cell with status, linked name, cluster and agent badges. Host stats stay in each page's own hosts table. Verification - No other production consumer: `WorkloadsSurface` → `WorkloadsTable` → `WorkloadPanel` is entered only from `ProxmoxPageSurface` and `VmwarePageSurface`; no references in pulse-pro, pulse-enterprise or pulse-mobile. - Browser pass on a production build embedded in a private mock backend, `/proxmox/overview` and `/vmware` at 1440x900 and 390x844, Bars and Trends modes: every grouped host row is one cell spanning the guest table's visible columns, with no metric cells or host facts; group-row clicks open nothing; guest drawers still open; no page errors. - The rewritten `NodeGroupHeader` test fails against the old component (its default facts render `PVE 9.1.9`). - `tsc`, full `npm run lint`, vitest over Workloads, shared, features, hooks, utils, Storage and app tests (only the Mac-only `agentInstallCommand` failures, unrelated), bundle budgets, and all release-control guards pass. |
||
|
rcourtman
|
8e93360d10 |
Poll only the history each metric table renders
useWorkloadTableMetricHistory ran two polled estate-wide reads per instance in Trends mode: the workloads history and the infrastructure summary. Since grouped workload rows lost their host metric cells, the Workloads surface reads only guest series, and the Proxmox nodes table never read anything but node series. Each reader passes its own abort signal, which skips the summary caches' in-flight dedupe, so every instance paid for both routes every 30 seconds. A required series option now names the rows a caller draws. The Workloads state passes 'guests' and the Proxmox nodes table passes 'nodes', so each polls one route. On the demo estate in Trends this takes the vSphere overview from one infrastructure and one workloads read per poll to one workloads read, and the Proxmox overview from two of each to one of each. Node table sparklines are unchanged. The performance contract claimed the two readers deduped their reads; it now states that they do not and that each reader polls only what it renders. The adjacent Bars-mode prose in the performance and frontend-primitives contracts described one active-row request that leaving the row cancels; it now describes the bounded warming window the hook has run since the history lens shipped. |
||
|
rcourtman
|
1c101b024d |
Replace stored alerts with each payload instead of merging
The websocket store wrote every incoming active alert with setActiveAlerts(id, alert). Solid's path setter shallow-merges into an existing object, so a field the server stops sending kept its old value in the browser. After a backend restart, threshold alerts are restored without metricStatus until their next evaluation, but an open tab kept the pre-restart live reading and the node drawer and node table went on saying "78°C now" for a reading the server no longer reports. An unacknowledge from another session left ackTime and ackUser behind the same way, as did a dropped nodeDisplayName or metadata. Active and recently resolved alerts are now written with reconcile, so omitted keys are removed on the snapshot, keyed-delta and REST recovery commits alike. reconcile diffs nested values in place, so an unchanged alert's live status no longer re-notifies its readers on every frame, which the shallow merge did by swapping in a fresh object each time. The pending acknowledgement hold is unchanged: it still skips payloads that contradict an unconfirmed local acknowledge or unacknowledge. updateAlert now stores a copy of each local update. Because payloads reconcile the stored alert in place, a snooze's optimistic record, which shares nested objects with the record the caller keeps for rollback, would otherwise have that rollback record rewritten by the next payload. |
||
|
rcourtman
|
38954b0b71 |
Give Docker containers and services without IDs their own alert refs
A container reported without an ID alerted under its Docker host's own reference (docker:<host>). Its alerts showed as host alerts, joined the host's resource history, were never cleared when the container left the report, and made the host's Patrol scope alias ambiguous. A Swarm service without an ID put its normalized name in the service-ID slot, so a name equal to another service's ID shared that service's alerts and history, and the history resolver had to treat the whole cluster as a conflict. Both now alert under their name, marked in the ID position (docker:<host>/name:<name>, docker:<host>/service/name:<name>). Docker and Swarm IDs never contain a colon, so these cannot equal a host or ID reference, and one with neither an ID nor a name raises no alert. The policy switches, cleanup, Patrol aliases and the history resolver use the same builders and parse the same shapes, so the resolver no longer needs the cluster-wide conflict. A pre-upgrade container alert left on the host's reference is cleared on the host's next report, and a pending image update recorded there keeps its age. Canonical state IDs were split at the first "::", but Docker hosts that collide are disambiguated as "<base>::<suffix>". Every container alert of such a host was rewritten to the reference of the host named <base>, so it joined that host's history and skipped its own host's cleanup. State IDs now split before the alert's recorded spec ID, or after its own resource ID, so alerts saved that way also recover on restore. |
||
|
rcourtman
|
723f3165a1 |
Show a silent agent's retained disk temperatures as last known
When a host agent passes its reporting lease, its collected SMART temperatures keep their values with collection.temperature unavailable (#2593), and #2603 taught the Storage disk surfaces to show such a value as last known. The frontend HostDiskSMART type never carried `collection`, so the agent's own surfaces kept presenting the retained readings as current: - The Machines temperature cell fell back to the hottest SMART reading and coloured it against the disk thresholds. On a live mock stack a silent, SMART-only agent read 52°C in threshold green. - Its tooltip, the machine drawer's Thermals rows and the guest drawer's Physical Disks card listed the retained disks as plain readings. The row does not make this obvious. By the time the lease expires the registry has marked a standalone agent's source stale, which outranks offline, so a silent agent reaches the table as `warning` with `agent.stale`, not `offline`. The offline metric fallback never blanks its cells, and the temperature cell has no such gate. Now `HostDiskSMART.collection` is typed and every reader decides through `isPhysicalDiskTemperatureCurrent`: - A retained disk reading never stands for the machine while another disk is current. When no non-standby disk is current and the machine has no positive direct or temperatureCelsius reading, the cell shows the hottest retained non-standby value muted with a dotted underline and screen-reader "last known", without threshold colour. - Tooltip and Thermals rows read "(last known)", the Thermals row with the collection reason as its title. A disk with a zero or negative reading no longer shows a temperature row there. - The guest Physical Disks card uses the Storage table's last-known treatment. The temperature helpers move to diskTemperaturePresentation.ts, which diskPresentation.ts re-exports. On the earlier base, importing diskPresentation from the Machines page and the drawer mapper folded the whole Storage presenter into the shared chunk (+4.9 kB gzip) and renamed its tracked budget entry, failing the bundle check. With the small module, Standalone grows 0.22 kB, the shared chunk 0.20 kB, and Workloads is flat against a build of main. The guest card still loads diskPresentation for extraction and health. Direct and temperatureCelsius readings carry no collection state and still lead the cell. A stale agent's CPU, memory, disk and sensor values still render as current. That is a row-wide decision, left for a separate change. |
||
|
rcourtman
|
8424fd6505 |
Show real CPU low and record in the Proxmox node drawer (#2594)
The Proxmox node drawer's **CPU low** and **CPU record** rows repeated the current temperature for every node without a host agent. `nodeFromResource` never read `proxmox.temperatureDetails`, which is where the resource API publishes the poller's full reading, including its low and record values. For those nodes it fell back to the scalar `proxmox.temperature` and copied that one value into both fields. On the dev stack, `delly` and `minipc` showed CPU low = CPU record = current. - The adapter now takes the first source with a usable reading. `temperatureDetails` comes after the platform record and the agent facet, so agent-linked nodes render exactly as before. - The bare-reading fallback leaves low and record unset, so the drawer omits those rows rather than inventing them. The poller also reset those values to the current reading every poll. That is a separate backend fix to the previous-node projection, landing in its own PR. Until it lands, PVE-only nodes will still show low = record = current, now because that is what the backend reports. ### Verification - Unit tests pin the API's real payload shape (scalar `proxmox.temperature` plus `proxmox.temperatureDetails`), the no-record fallback, and agent-facet precedence. The new test fails against main's adapter, which returns `cpuMin` 58 = current. - Browser pass on a private mock stack at `/proxmox`, 1440x900 and 390x844, with REST and websocket payloads rewritten to make one node PVE-only: - main's adapter showed CPU low 82°C / CPU record 82°C; - this change shows no rows without a record, and 39°C / 91°C with one; - agent-linked nodes and the table TEMP column are unchanged. - `vitest` over `src/utils`, `src/components/Workloads`, `src/features/proxmox` and `src/features/platformPage` passes, apart from `agentInstallCommand.test.ts`, which fails on macOS regardless of this change. |
||
|
rcourtman
|
18f0fbd826 |
Keep PVE disk alert references out of resource history
PVE disk health and wearout alerts reference the disk by instance, node and device path. Operator mutes resolve that reference to the disk at the path (#2112), but resource history must not bind it: a replacement disk in the same slot, or a reboot that reorders devices, takes over the path, and disk alerts are evaluated before their poll's disks reach the registry. A history binding belongs to the reference string, so it would carry a replaced disk's journaled alerts into its successor's history and make the Alerts history Resource action show the successor's incidents for its predecessor's alert. A binding guarded by each alert's recorded disk serial was built and measured on a mock estate. It bound correctly while the disk stayed at its path, but rejected events still joined whichever disk the reference was bound to, and the serial check cannot be atomic with journal writes. Record the decision in the unified-resources contract and pin it with a test, so binding these alerts later goes through ownership per row rather than a per-reference binding. |
||
|
rcourtman
|
7b362b3532 |
Load guest physical disks with the drawer to keep Workloads in budget
GuestDrawerOverview statically imported GuestPhysicalDisks, which pulled all of diskPresentation.ts into the WorkloadsSurface table chunk. With the Running Hot verdict that chunk measured 70.64 kB gzip against its 70.23 kB cap, so the Frontend bundle check would fail. The card renders only inside an open drawer for a guest with a Pulse agent, so the overview now lazy-loads it under a local Suspense, the same pattern as the bulk web-link editor. WorkloadsSurface drops to 64.31 kB and the disk presentation builds as its own chunk. The contract test pins the dynamic import. |
||
|
rcourtman
|
fb7dd955ed |
Keep retained disk temperatures out of AI, Patrol and reports
Normalization keeps a disk's last-known temperature when the current observation did not collect one (a disk in standby, a host agent past its reporting lease) and marks it non-available. Several readers still took PhysicalDisk.Temperature at face value. The AI chat context listed a standby disk as needing attention for heat. Patrol raised health issues and triage flags from the retained value. The AI query and disk-health tools handed it to the model as the disk's temperature. Performance reports and the reporting runtime snapshot tabulated it as measured, and the disk temperature charts padded a series out to now with it. Each of them now goes through diskinventory.TemperatureCollected. AI surfaces keep a retained value as context, labelled last known with its reason. Reports and the charts leave it out. monitor.go's private copy of the predicate is replaced by the shared helper. |
||
|
rcourtman
|
d9ffb39ce8 |
Keep a merged disk's temperature under its own row's state (#2618)
Stacked on #2609; until it merges this branch also carries its commit, and the change here is the last commit. ## Why The registry merges a physical disk's rows (Proxmox, host agent SMART, Unraid inventory), choosing the temperature by source preference and merging collection states separately. When two rows carry different temperatures, the shown value could sit under the other row's `available` state. A real case: a node whose disk temperature Proxmox collects now (`proxmox_node_smart`, 40) and whose linked host agent has gone silent (retained 72, `unavailable`). In either ingest order the merged disk showed **72 labelled available**, so risk, history and the UI treated the frozen reading as current. ## What - `pairPhysicalDiskTemperatureState` runs at the end of each merge step. When the merged state says collected but the row the shown value came from says it was not, the merged state becomes that row's own. - It never changes a value and never grants availability, so it cannot revive a withdrawn reading or override a resumed agent's fresh one. - Rows with the same value keep the merged state, so the agent's own withdrawal still supersedes the Proxmox copy (#2593). - With that guarantee the Unraid-native row no longer needs to withhold `available` (#2609 restricted it), so it carries the same derived state as a SMART row borrowing its reading. ## Known limit This is exact for the two rows of one merge step. The merged disk keeps one state per field, so with three or more rows for one disk, a withdrawal that an earlier step replaced is invisible to a later step. Sol found two such counterexamples, both needing three rows (one also needs coincidentally equal values, the other a legacy agent plus two Proxmox rows). Closing that needs each reading's own state tracked across merges, and that redesign is filed separately. Preferring another source's current value instead of the retained one was tried and dropped: it needs an ordering between same-source states that the rows do not record, and each version reopened a withdrawn reading somewhere. ## Tests - `TestPhysicalDiskMergePairsTemperatureWithItsCollectionState`, in both ingest orders: - silent agent against a collected Proxmox reading, and against no Proxmox reading - reporting agent - silent agent against a Proxmox copy of its earlier reading, with and without SMART attributes - resumed agent against a Proxmox copy of its withdrawal - standby agent against a Proxmox copy of its pre-sleep reading - `TestPhysicalDiskMergeNeverPresentsAWithdrawnTemperatureAsCollected`: three distinct-valued withdrawn or stale rows in all six orders. - `TestUnraidInventoryRowDoesNotMaskTheSMARTRowState`: the Unraid inventory's `available` no longer lends itself to a SMART row's uncollected reading. - Without the rule, the peer-reported case, the Unraid masking case and one three-way order fail. |
||
|
rcourtman
|
dcb69cfb02 |
Record Docker host, service and sub-resource alert history (#2620)
Follows #2608, which made `MonitorAdapter.RecordChange` resolve Proxmox and agent alert references to canonical resources. Three kinds of alert reference still journaled under strings no resource answers to: - Docker host alerts (`docker:<host ID>`) - Swarm service alerts (`docker:<host ID>/service/<ID>`) - sub-resource alerts such as ZFS pool and device health (`<storage ID>/zfs-pool:<pool>[/device:<dev>]`) and host filesystem, SMART, RAID and sensor alerts (`agent:<host ID>/disk:…` etc.) Resource facets, timelines, resource intelligence and Assistant context read history by canonical ID, so these alerts never appeared there. ## What changes - **Docker host refs** resolve through the registry's Docker source ID. - **Swarm service refs** resolve through the host's Swarm cluster to the service with that exact ID, from any manager that reports it. A service with no ID is referenced by its name in the same slot, so while a cluster has such a service, every service ref there is a conflict and binds nothing. - **Never bind:** container names, shortened container IDs and hostless `docker-service:<name>` refs never bind, and are no longer retried against later inventory. - **Sub-resource refs** bind to the owner whose durable ID they carry: - ZFS pool and device alerts go to the storage. - The Unraid array alert goes to its array storage. - Filesystem, SMART, disk temperature, RAID and custom-sensor alerts go to the host. - The owner must resolve to exactly one resource of the expected type. An ambiguous or incompatible owner is a conflict. - Mount and kernel device labels can name a different disk after a reboot, so these alerts never bind a physical disk. - **Retries:** a reference that hits a conflict at write time now enters the backfill retry set, the same as one found at startup. ## Known residuals (producer-side, follow-up proposed) - A container with no ID alerts under its host's reference (`DockerResourceID(host, "")`), so its events join that host's history (never another container's). - An ID-less service whose name equals another service's ID shares that service's reference. Rows it journaled can join the other service's history once a later generation or event binds the reference. Swarm always reports service IDs, so this needs a malformed report. Both are documented in the contract. Separating them needs a producer change in `internal/alerts`. ## Evidence Private `PULSE_MOCK_MODE` stack, fresh data dir. The binary at #2608's tip ran first, then this change on the same dir. Alert rows were classified by source reference (from `alert_identifier`) and checked against each owner's `/api/resources/{id}/timeline`: | ref shape | rows readable via owner timeline, before | after | |---|---|---| | Docker host | 0 / 1 | 3 / 3 | | Swarm service | 0 / 5 | 19 / 19 | | ZFS pool | 0 / 9 | 26 / 26 | | ZFS device | 0 / 9 | 27 / 27 | | agent filesystem | 0 / 1 | 3 / 3 | | Docker container (mock short IDs) | 0 / 11 | 0 / 30, no container alias written | Tests: - `TestHistoryIdentityMonitorAdapterResolvesDockerHostAndServiceReferences` covers multi-manager services, a Docker host merged into its agent machine, legacy-row backfill, recovery after removal, ID-less service conflicts, and names and short IDs staying unbound. - `TestHistoryIdentityMonitorAdapterResolvesSubResourceReferences` covers every owner shape, wrong-type and ambiguous owners, retained bindings, write-time conflict retry, and physical disks staying unbound. - `TestOwnerAlertTimelinesUseCanonicalHistoryIdentity` drives the real alert manager for Docker host offline, Swarm service gap, ZFS pool/device and host filesystem alerts. It fails on #2608's tip. UI reach: the Docker, storage and platform host-row drawers don't request per-resource history. This fix reaches the facets/timeline/intelligence APIs and Assistant resource context, not a drawer. |
||
|
rcourtman
|
23fdfbc493 |
Keep the cleared history hint box so hovered guests stay put
The first-use "Hover a guest to preview history" hint clears on the first populated row preview, while the pointer is over a guest row. It used to unmount at that moment. On widths where the hint wraps the filter action row onto its own line (1024-1440px with a typical estate), the toolbar then shrank by 36px, the table slid up under the pointer, and the history lens opened on the next guest instead of the one being hovered. Laptop-width users hit this on their very first use of the feature. WorkloadsFilter now keeps the cleared hint mounted but invisible while row hover stays in bars + history mode, so clearing it never changes toolbar geometry at any width. Leaving that mode or remounting the filter releases the box, and the next visit gets the compact one-line toolbar. |
||
|
rcourtman
|
1fe9f86d55 | Merge remote-tracking branch 'origin/main' into refactor/drop-noncompact-group-node-metrics | ||
|
rcourtman
|
94b7870855 |
Call a disk that is only hot Running Hot, not Replace Now
The Physical Disks Health verdict mapped any critical disk risk to "Replace Now". Disk risk also grades temperature (temperature_high at >=70C critical, >=60C warning), so a SMART-passed disk with good life running at 72C told the user to buy a new disk when the fix is airflow. Alerts already keep heat apart: the SMART health and wearout alerts skip temperature_high and raise disk heat as a Temperature metric. The verdict now splits risk reasons into heat and everything else. Heat alone reads "Running Hot" (phone "Hot"), red at the critical tier and amber at warning, with the temperature reason still as its title and drawer summary. FAILED health, any other critical reason, and a critical level no reason explains keep "Replace Now". The more severe class names the verdict and failure evidence wins a tie, so merged reasons that list heat first no longer hide pending sectors. Sorting and the health filter follow the same verdict, so hot disks stay in "Needs attention" at their temperature tier. A reason's code and severity count even when it has no display text. A critical level that no reason explains shows the generic summary rather than a weaker reason's text. Thresholds are unchanged. The tier still comes from the server's flat 60/70C rule, not the per-type alert thresholds that colour the Temp cell. Reconciling those is separate work. |
||
|
rcourtman
|
5becf59536 |
Remove the unreachable host metric cells from grouped workload rows
Both production mounts of the Workloads surface, the Proxmox and vSphere overviews, set compactGroupHeaders unconditionally. WorkloadPanel's per-column node cells (CPU, memory, disk and I/O bars or sparklines, uptime, version, temperature) and NodeGroupHeader's column mode and inline host facts therefore never rendered. They were left over from the retired standalone Workloads page, whose group rows had no hosts table beside them. Drop the option, the dead renderers and the NodeGroupHeader props that only served them. Grouped host rows keep exactly the shape both pages already render: one spanning cell with status, linked name, cluster and agent badges, while host stats stay in each page's own hosts table. Tests that only pinned the option are removed. The NodeGroupHeader test now covers the spanning-cell shape and fails against the old default facts, and the PVE version formatter guardrail moves to proxmoxPageModel, the remaining version renderer. |
||
|
rcourtman
|
0fb5ccb62b |
Keep node temperatures off same-named nodes at other sites (#2617)
Two Proxmox connections can each have a node with the same name, which
is common for MSPs running one `px1` per site. With one host agent
linked to site A's `px1`, `getHostAgentTemperatureByID("siteB-px1",
"px1")` returned site A's temperature. The hostname fallback took the
first agent with that hostname, even one linked to another node, and
even when the name was ambiguous enough that the agent linker had
refused to link it. The cluster sensor cache leaked the same way: SSH
readings an agent collects for its cluster siblings were keyed by bare
node name. Site B's `px1` then showed, alerted on and recorded history
for another machine's sensors.
## Change
- The poller hands the lookup the polled node
(`getHostAgentTemperatureForNode`,
`carriedTemperatureOutlivesAgentLease`), and the lookup works on that
node's **slot**: its own source ID, plus, when the read state does not
hold that ID, the one same-named node proven to be the same machine.
- The state folds the views of one machine reached through two
connections (a cluster added twice, or a multi-homed host added by each
address) into a single node kept under one view's ID, and an automatic
agent link follows it. A narrowed fallback on its own would leave those
views without a temperature permanently; that was reproduced with real
`UpdateNodesForInstance` folds.
- Same-machine proof comes from the state's own evidence: new
`models.NodeObservationsSameMachine` and
`models.HostAgentBridgesNodeViews`
(`internal/models/node_machine_identity.go`). `UpdateNodesForInstance`'s
agent-corroboration check now shares its rule with them. Two or more
candidates are ambiguous and none is taken, as the state declines an
ambiguous alias.
- An agent linked to the slot is the node's agent. The hostname fallback
only takes an agent with no node, VM or container link, and only when no
node outside the slot and no second unlinked agent has the name.
- Cluster sensor cache entries record their reporting agent. A reading
serves a node only when that agent's linked node shares one of the
slot's connections, or belongs to a connection whose configured view of
the node has a matching TLS fingerprint (the other connection of a
cluster added twice).
`LinkedNodeID` holds the Proxmox source node ID for every link source
(automatic `findLinkedProxmoxEntityWithHints` → `NodeView.SourceID()`,
manual `LinkHostAgentToNode` validating against `models.Node.ID`,
reconciliation, continuity restore), so the link match needs no
namespace normalization. A manual link stays pinned to the ID the
operator chose (the state never transfers intent to a replacement
provider identity), so a node held under the other view's ID has no
linked agent. The physical-disk SMART merge already matches by
`node.LinkedAgentID` within one instance and is unchanged.
## Tests
- `TestNodeTemperatureStaysWithinSameNamedNodesSite` (registered
monitoring proof, `monitor_additional_test.go`), through
`collectNodeTemperatureData` against registry-backed and state-folded
read states: agent linked to the other site's node; ambiguous unlinked
agents; unambiguous unlinked agent (still served); cluster added through
two connections (alternating real state folds); multi-homed host via two
standalone connections; ambiguous address alias; a namesake agent with a
contradicting fingerprint; sibling readings across both connections of
one cluster; sibling reading from the other site. The leaking cases fail
on main, and each folded-view guard was checked with an overlay negative
control.
- `TestApplyHostReportScopesClusterSensorsToReportersConnection`
(`monitor_host_agents_test.go`): real `ApplyHostReport` ingest of
cluster sensors.
- `TestNodeObservationsSameMachineMatchesStateFold` and
`TestHostAgentBridgesNodeViewsMatchesStateFold` pin the predicates
against `UpdateNodesForInstance`'s actual fold.
- Full `internal/monitoring`, `internal/models` and
`internal/unifiedresources` pass on pulse-dev.
|
||
|
rcourtman
|
3baab9bc59 |
Say a reviewed Patrol issue stays reviewed when suppression ends (#2615)
Ending a suppression puts the alert in Acknowledged when it is
acknowledged and in Open otherwise (`unsuppressOperationalAlert` in
internal/alerts/active_lifecycle.go). Acknowledged is not active
attention (`attentionStateCountsAsActive`).
The Patrol attention detail still promised otherwise for an issue that
was marked reviewed and then suppressed:
- its early-end button read "Return to active attention"
- the handled-view notice read "Returned to decision inbox"
- the suppression form offered "Return it to active attention after"
In each case the issue actually went back to Reviewed.
| | Reviewed issue | Unreviewed issue (unchanged) |
|---|---|---|
| Suppression duration | End suppression after | Return it to active
attention after |
| Early-end button | End suppression | Return to active attention |
| Handled-view notice | Suppression ended. It stays reviewed. | Returned
to decision inbox |
The duration label and the button follow the detail's
`operationalRecord.acknowledgement`. The notice instead reads the item's
state from the reload that follows the change. If the review was removed
elsewhere after the detail loaded (another tab), it still reports what
actually happened. `getAttentionSuppressionEndPresentation` owns the
copy.
The countdown line under the suppression ("Returns to active attention
in 1d.") is fixed separately by fix/future-relative-time, using the same
wording.
Verification:
- Three new tests in `PatrolAttentionWorkbench.test.tsx`:
- The reviewed case fails on main.
- The stale-detail case fails on a version that predicts the notice from
the detail.
- The unreviewed case pins the copy that stays the same.
- `src/features/patrol` and the two other test files that import the
workbench pass (492 tests).
- Run live on a private mock stack at 1440px, 800px and 390px:
- Reviewed case: mark an issue reviewed, suppress it, end the
suppression early from the handled view. The button read End
suppression, the notice read "Suppression ended. It stays reviewed. N
handled issues remain.", and the attention API reported the item
`acknowledged` with the active count unchanged.
- Unreviewed case: the item came back to the inbox as `open`.
- Stale-detail case: the review was removed through the API while the
detail stayed open. The notice read "Returned to decision inbox",
matching the API state `open`.
|
||
|
rcourtman
|
c3adf0c7b9 |
Chart Net and Disk I/O in the row history lens (#2614)
Hovering a guest row in bars mode swaps CPU, memory and disk for history
sparklines, but Net I/O and Disk I/O stayed as static numbers. They are
the jumpiest readings in the row, so they gain the most from history
("short burst or sustained?"). The hover request already returns the I/O
series for that guest, so this adds no reads.
Changes:
- The row history lens now swaps both I/O cells too, sharing the
synchronized scrub cursor.
- I/O sparklines (lens and Trends) keep a compact current pair beside
the chart (`↓3.3M ↑2.2M`, `R1.4M W826K`) instead of hiding the values in
a tooltip. The glyphs use the series colours, so the pair doubles as the
legend. The full rates stay in the chart's accessible label.
- Rate sparklines get a 100 KiB/s scale floor. Before, a guest moving
between 2 and 6 KB/s drew the same full-height spike as one at 500 MB/s.
- `GuestRow` builds every sparkline from accessors. Each live snapshot
used to remount the lens and Trends charts (dropping an open scrub
tooltip). An open tooltip now also re-reads the current series instead
of holding the values from the last mouse move.
- `formatCompactSpeed` for the compact pair, the "History" hover option
tooltip, and the frontend-primitives and performance contract clauses.
Verified on a private mock stack at 1920, 1600, 1440 and 768 (dark and
light), including scrub sync, an intercepted idle-I/O history, a live
websocket update under an open tooltip, and Trends mode (0 clipped
cells). Receipt included.
Two pre-existing issues found while verifying are left out of scope. At
about 1440px the first-use history hint's dismissal shifts the table
36px, so the first preview opens on the next row. And `WorkloadPanel`'s
non-compact node header metric cells never render in production.
|
||
|
rcourtman
|
8259a74c9d |
Show the live reading of threshold alerts held below trigger (#2567)
A threshold alert stays open after its reading drops below the trigger. It holds until the reading reaches the clear level, then must stay there for the recovery delay. While it held, the evaluator never touched the alert, so every surface kept showing the last breach. On minipc the drawer said "Node temperature at 80.0°C" next to a green 72°C in the node table, with nothing explaining why the warning was still there. #2068 reports the same thing for a disk alert. **What changes** - Open threshold alerts carry a volatile `metricStatus`: the value Pulse is comparing now (the average for rolling-average rules, with the latest sample beside it), the phase (`breaching`, `latched` or `recovering`), the trigger and clear levels, and recovery progress from the reducer's monotonic run. `value`, `message` and `lastSeen` keep meaning the last breach, and the hold branch still never notifies. - The status is stripped from the active-state checkpoint, the recovery mirror and the degraded recovery projection, so holding adds no durable writes. Resolved records carry none. - Stale-alert cleanup and the retention TTLs use the latest observation. Previously an alert held below its trigger for more than a day was auto-resolved as "resource no longer being monitored". - Node, guest and Mail Gateway drawer rows lead with the reading now and say what clears the alert, with the last breach on hover. The Proxmox TEMP cell keeps its own open temperature alert's tone and explains it on hover. Storage rows stop saying "Over N%" once usage is back under the limit, and the Assistant handoff no longer labels the breach value as current. **Verified** - Unit tests drive a node temperature alert through 85 → 78 → 72 → 75 → 77 → 72 → resolved, and fail against the old evaluator. - Private mock stack with pve2 stepped 88 → 78 → 72°C at desktop and phone widths; see the browser receipt in this PR. Refs #2068 |
||
|
rcourtman
|
38c53d047f |
Remove the never-mounted unified resource table (#2612)
`PlatformResourceTable` was the only renderer of `UnifiedResourceTable`, and nothing imports it. Every platform page renders its own native tables, and `/infrastructure` only shows the opening-workspace placeholder. This removes the never-mounted cluster and the governance that described it. ## Evidence it was dead - Import graph from `src/index.tsx` (static, dynamic, `vi.mock`/`importActual`): the cluster was reachable only through `sharedPlatformPage.tsx`'s import for the unused `PlatformResourceTable` export. No barrel, namespace import, lazy route or re-export. - Production build of main: none of the cluster's literals ship (`infrastructure-table-surface`, `ready for Pulse`, `Agent Infrastructure`), while a live control string (`workloads-table-surface`) does. ## What changes - Deleted: `PlatformResourceTable` and its `PLATFORM_STATUS_FILTER_OPTIONS`; `UnifiedResourceTable` with its host, service, PBS and PMG cards, source badge cell, state hook, viewport sync and models; `ClusterDeployBanner` (only the host card rendered it; the live cluster deploy flow is the Settings `ClusterAgentDeployDialog`); and their seven test files. - Kept coverage for live code those tests also exercised: the detail-mapper no-fetch boundary and the online boolean sensor case move to `resourceDetailMappers.test.ts`; the `ResourceFacetSummary` render cases move to `ResourceFacetSummary.test.tsx`. Both are now registered proofs on the policies that govern their subjects. - Guardrail tests, `shared-template-registry.json`, `registry.json`, `status.json` evidence (RA1, L10, L13; L13 now cites `useUnifiedResources.ts`) and the performance-and-scalability, unified-resources, frontend-primitives and cloud-paid contracts drop the deleted paths and the rules that only governed them. - `.bundlesize.json`: Rollup used to name the large shared chunk `sharedPlatformPage`; it now names it `MetricBar` (118.05 kB gzip), so that entry is re-keyed. Total JS gzip moves +1.96 kB from re-chunking and total CSS drops 0.5 kB. ## Verification - Browser pass on a production build embedded in a private mock backend: all 32 platform sub-tabs at 1440x900 and 390x844 render populated tables with no horizontal overflow; shared toolbar search, the no-match empty state and inline row detail work on Machines and Docker images. The row detail raises eight `style-src-attr` CSP violations, identical on the main build (pre-existing static `overflow-anchor` styles in `ResourceDetailDrawer.tsx`). - `tsc`, eslint, vitest over Infrastructure, platformPage, shared, standalone and app tests; shared-template audit; bundle-size check; contract, registry and status audits; the release-control unittest suites; the full pre-commit battery. |
||
|
rcourtman
|
0f5f054ecb |
Settle slow polled reads and keep replication links honest (#2610)
Follow-up to #2578 from a second-opinion review. It closes three gaps #2578 left, plus the issues found while fixing them. **Polled reads that never settle.** The shared poll in `createNonSuspendingQuery` replaced any read still in flight on every tick. The API answers 429 with `Retry-After: 60` and `apiFetch` waits it out, so under rate limiting a 30-second poll discarded every read and the query never settled. Replication reads also ignored the abort signal, so each replaced request kept running. - A poll now skips its tick while a read that an earlier poll started is still in flight. - A refresh, a source change or a remount revalidation stays replaceable at the next tick, as before. The behaviour pinned by `744160fa0e` and the History tests is unchanged. - A polled read still in flight at or after 150 seconds is presumed stalled. It is replaced and settled as a failed read, so nothing spins forever when every read hangs. - `fetchReplicationJobs` forwards the signal to `apiFetch`. - It throws `apiErrorFromResponse` errors that keep the HTTP status, so a 401 or 403 withdraws the jobs. **Replication direct links.** A failed first read made the job count zero, so `/proxmox/replication` fell to Overview with no error or Retry. - Replication presence is now unknown until a read succeeds. - After a read succeeds, a failed poll keeps the last answer instead of reopening a route that an empty read closed. - An access denial withdraws the tab, and the link holds on the denial message. **The shared clock trailed by a tick.** `useRelativeTimeNow` returned its last tick, so a cell mounted between ticks measured from up to 30 seconds ago, and the desktop backup age could disagree with its band. - Every read now returns the wall clock, and the tick only schedules re-reads. - This retires the backup `Math.max` workaround. - The #2578 age tests now jump the clock and fire one tick. They used to loop hundreds of re-renders, and one timed out under load. Each new test fails without its fix. Live check: a private mock stack driven by Playwright, with replication answers shaped per scenario, at 1600px and 390px. - A 500 on the first read keeps the error and Refresh, and Refresh loads all 8 jobs. - A 403 after loading withdraws the tab and shows the server's denial. - Empty, then failed, then empty stays on Overview. - With a polled read held open, two poll ticks start nothing and abort nothing. - Leaving the page aborts the read. - Backup ages and bands render consistently with no Unknown cells. The same session with main's runtime files reproduced each failure. The browser receipt is in the commit. |