mirror of
https://github.com/rcourtman/Pulse.git
synced 2026-10-24 04:33:17 +00:00
112ce030d7ae4287a35e32d7bf9e722b971afdb1
12831 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 rcourtman
|
112ce030d7 |
Stop reports and resource history calling a moved alert recovered
A node alert that moves to its linked Pulse agent closes with a moved_to_agent resolution (#2580), and #2591 taught Patrol, alert history and the assistant to say so. Two readers still showed the close as a recovery. Performance reports dropped the resolution when copying recently resolved alerts into reporting.AlertInfo. A node report generated in the five-minute window after a handover read HEALTHY, "All systems operating normally", counted the alert as triggered and resolved, and told the AI narrator resolved: true, although nothing had seen the condition clear. Report rows now carry the resolution and the alert's resource. pkg/reporting treats an alert that closed without recovering as still live unless the report also shows its successor's reading for that metric (an open alert, or a recovery since the handover), so a handover never reads as healthy and one condition is counted once; fleet reports match handovers across all their resources. The PDF card, Quick Stats, active list and heuristic narrative say where the alert went and to check the agent, the alerts table shows "Moved" with the engine's summary, fleets count it as active and name the agent, and the narrator gets resolved: false, the summary and whether the successor's alert is listed, with prompt rules for both. Resource change readers labelled every alert_resolved change "Alert resolved" in green and prefixed the headline with it, so a moved close read "Alert resolved: Alert moved to pve1 (Host Agent). This is not a recovery...". getResourceChangePresentation now reads the alert_resolution metadata: such a change is "Alert moved" in the neutral blue of the history badge, and its summary is the whole headline in the drawer, ResourceChangeSummary and Patrol's assessment handoff. Reports requested as agent, which is how the Reports page addresses every Proxmox node, still carry no alert rows at all, and node alert history is stored under legacy node IDs that the drawer never queries; both are separate fixes. |
||
|
rcourtman
|
d26a483731 |
Show a moved alert as moved in Patrol, history and assistant (#2591)
Follow-up to rcourtman/Pulse#2580. A node alert handed to its Pulse agent closes with an `AlertResolution` (`moved_to_agent`), and notifications already say it moved. Every in-app reader still treated that close as a recovery: - Patrol's attention detail showed a green **Resolved** badge, and its timeline read "Open to Resolved · Recovery evidence" directly above "…This is not a recovery…". - The Alerts history listed the row as **resolved**, and its incident timeline ended "Alert resolved: Node memory at 95.0%". - `GetRecentlyResolved` rebuilt `models.Alert` without the resolution. So the state snapshot, websocket, assistant context, Patrol's seed prompt and the `pulse_alerts` `resolved` action all saw a plain resolve. **Operational trust.** New `TransitionCause` `ownership_transferred`. It may only enter `resolved`, and only with evidence. `markOperationalResolved` uses it whenever the alert carries a `moved_to_agent` resolution; every other close stays on `recovery_evidence`. Alert-quality telemetry keys "resolved while snoozed" on `recovery_evidence`, so a handover during a snooze no longer counts there. No notification code reads the cause: links key on the transition ID and state. **State and AI.** `models.Alert.Resolution` holds the reason, successor and the engine's `Summary()`. `cloneAlert` copies it, and `GetRecentlyResolved` fills it. - Patrol's seed writes `closed 3m ago. Alert moved to pve1 (Host Agent). This is not a recovery: …`. - The assistant's context writes `closed … ago) <summary>`. - The resolved-alerts tool returns `resolution` and `successor_resource_id`. **Timelines.** The `alert_resolved` resource change carries the summary as its reason and the code as `alert_resolution` metadata. Incident memory titles the close with it, both when projecting from the canonical timeline and on its own write path. **UI.** - The history row reads **moved to agent**, even if it was acknowledged. The full account is in the badge title and is printed under the message on the phone card. - Patrol's detail shows a **Moved** badge (info tone), with the handover sentence directly under the summary rather than only inside Evidence and history. Its timeline says "Ownership transferred". **Landing order.** rcourtman/Pulse#2586 is the only producer of a handover, and it has no auto-merge. It should land after this PR (or merge main once this is in). Any handover recorded before this lands would otherwise keep a recovery-labelled transition and timeline. **Not covered here:** - Performance reports still list a handover from the last 5 minutes as resolved. - The resource drawer labels the change "Alert resolved" before its reason. - The incident header in the history expansion still shows its open/resolved status word. Both of the first two are filed as a follow-up and noted in ai-runtime.md and unified-resources.md. **Tests:** - `TestOwnershipTransferOnlyClosesWithEvidence`. - The cause assertions in `TestReleasedMetricAlertCarriesResolutionInsteadOfRecovery`. - `TestRecentlyResolvedCarriesHandoverResolution`. - `TestAlertResolutionWireContract`, which fails without the `cloneAlert` copy. - `TestMonitor_HandleAlertLifecycle_HandoverCloseIsNotARecovery`, which fails on the old code with "Alert resolved: Memory usage at 95%". - `TestIncidentStore_HandoverCloseIsNotARecovery`. - `TestSeedHealthAndAlerts_HandoverIsNotARecovery`. - `TestService_buildAlertContext_HandoverIsNotARecovery`. - `TestExecuteListResolvedAlertsReportsHandover`. - Frontend history, presentation and Patrol workbench tests. The operationaltrust, alerts, models, ai, ai/tools, ai/memory, unifiedresources, monitoring and notifications suites pass. `tsc`, eslint and the vitest sweep pass, apart from the Mac-only `agentInstallCommand` PTY test. **Browser pass.** A private non-mock stack ran on an alert history written by the real `releaseCanonicalMetricAlert` handover path. I checked Alerts history (row, tooltip, expanded timeline) and Patrol's deep-linked detail at 1280 and 375: no overflow, no console errors. |
||
|
rcourtman
|
779b3d4ff8 |
Tint Kubernetes node rows for their drawer's alerts (#2596)
A Kubernetes node that runs a Pulse agent is an agent row, and the agent's alerts are keyed `agent:<id>` and labelled with the hostname. The nodes table matched only the row id and display name, so on a mock estate "Prod Euw1 K8s 02" sat at 100% memory with a critical alert, its drawer listed "Agent memory at 100.0%", and its row was untinted. The overview's "Nodes needing attention" looked only at readiness and cordons, so it named K8s 03 (Degraded) and left K8s 02 out. - Kubernetes node rows tint from `getUnifiedResourceAlertStyles` (#2579), the set the drawer reads, and drop the tint while expanded like the Docker host and Machines rows. - `getKubernetesNodeAttentionRank` ranks a node by the stronger of its state and its unacknowledged open alerts: NotReady and critical alerts first, then cordoned or degraded nodes and other alerts. The overview list and the nodes table's default order both use it, so a critical node no longer sorts below a degraded one. Acknowledging the alert drops the node out of both, as it already drops the tint. - vSphere host and TrueNAS system rows are deliberately unchanged. Checked against the producers: their VMs', pools' and apps' alerts carry the host name as their node (VM `ParentName` is the runtime host; registry children take the parent's name), the TrueNAS drawer states the same pool problems, and `agent:` alert ids only come from Pulse agent reports (`internal/alerts/host.go`, `external_probe.go`). A Pulse agent on a TrueNAS box stays a separate resource (`TestUnifiedDiskWriteMetricsKeepSameHostnameHostAgentAndTrueNASTargetsSeparate`). Switching them would drop the pool rollup that tints the TrueNAS row for a degraded pool, so a test now pins it. Verified in a browser on a private mock stack (receipt `frontend-modern/browser-verification/83848f2d5ef2a326.json`): Nodes and Overview at 1440px, 390px and in dark mode, the drawer, and acknowledge then unacknowledge of the K8s 02 alert. The new Kubernetes tests fail against the previous runtime and pass with this change. |
||
|
rcourtman
|
40ebce161a |
Clear agent disk temperature alerts when the disk leaves (#2595)
Stacked on #2589 (the threshold-off fix). Until it merges this PR also
shows its two commits; the change here is the last commit,
|
||
|
rcourtman
|
4bd567b41a |
Clear agent disk temperature alerts when their threshold is off (#2589)
Turning off the agent disk temperature threshold left every open disk temperature alert active indefinitely. **Why nothing cleared them** - `CheckHost`'s branch for a disabled `DiskTemperature` threshold was an empty placeholder comment. - The host-wide clears (all agent alerts disabled, a disabled host override, agent removal, confirmed agent offline) matched only `agent:<host>/disk:` and missed `agent:<host>/disk_temp:`. - A config save skipped these alerts because `diskTemperature` was not classed as a threshold metric in `metric_runtime.go`. **Changes** - `host.go`: `CheckHost` clears a host's disk temperature alerts when the effective threshold is off, and `clearHostDiskAlerts` plus the confirmed-offline sweep include the `disk_temp` prefix. The per-type `DiskTempByType` refinement moves into `hostDiskTemperatureThresholdNoLock`, which `CheckHost` and `resolveHostAlertThresholdsNoLock` share. A config save therefore judges an alert against the threshold for its disk type, not the agent default. - `host.go`: an alert persisted before `CheckHost` recorded `diskType` (May 2026) has no disk type until its next firing evaluation. A save judges it against the lowest enabled trigger any of the host's disks can use, so a save cannot resolve an alert its disk type would still fire. - `metric_runtime.go`: `diskTemperature` is a threshold metric, so a save resolves these alerts once the threshold is off or the reading is below it. - `config_runtime.go`: when there is no recovery band, the save path now matches the evaluator. `buildCanonicalMetricSpec` drops a clear level at or above the trigger, and the reducer keeps firing at the trigger itself. Before this, a save resolved those readings and the next report raised them again. This applies to every metric alert that reaches the threshold comparison (docker-host alerts return before it); with a valid band, behaviour is unchanged. **Tests** (`internal/alerts/host_unraid_lifecycle_test.go`) - `TestCheckHostClearsDiskTemperatureAlertsWhenThresholdTurnsOff`: agent default off, host override off, host override disabled, all agents disabled. - `TestHostDiskTemperatureAlertsClearWhenAgentLeaves`: agent removed, agent confirmed offline. - `TestConfigSaveResolvesDiskTemperatureAlertAgainstItsDiskTypeThreshold`: an unrelated save keeps an alert above its nvme trigger; turning the threshold off resolves it. - `TestConfigSaveKeepsDiskTemperatureAlertWithoutDiskType`: an alert with no `diskType`, at 72C under a 75C agent default and a 70C nvme entry, survives an unrelated save. - `TestConfigSaveKeepsAlertTheEvaluatorKeepsWithoutRecoveryBand`: a 70/75 per-type entry at 72C and at 70C. Every case fails against origin/main's code (checked with a build overlay) and passes here. `go test ./internal/alerts/...` passes. |
||
|
rcourtman
|
e9069aed8c |
Carry a typed reason when an alert closes without recovering (#2580)
When a Pulse agent is linked to a Proxmox node, the node releases its CPU, memory and disk alerts to the agent, and any node alert still open closes. The close must reach integrations, so a PagerDuty or webhook incident opened from the node alert does not stay open forever. But it went out as a recovery: "pve1 on pve1 is now healthy" for a node still at 95% memory, backed by synthesized "recovery projection" evidence. #1682 is the same complaint from another trigger. This adds a typed `AlertResolution` (first reason: `moved_to_agent`, with the agent's alert resource ID and name) and carries it from the release path to every resolved consumer: - `releaseCanonicalMetricAlert` is the disabled-threshold release pulled out of `evaluateCanonicalMetricAlert`. It takes a resolution, and `clearAlertWithResolution` stamps a copy on the alert before it resolves. - The resolution rides the Alert JSON into the resolved callback, the delivery queue, the history row and the event log (reason code plus summary). - The synthesized closing evidence and the resolve transition say what happened instead of claiming a recovery projection. The cause stays `recovery_evidence`, the only cause operational trust lets enter `resolved`. - Notifications still send the close. Messages say "Memory alert moved to pve1 (Host Agent). This is not a recovery: check the agent for the current reading." in webhooks, `MOVED:` on a single ntfy push, and "Pulse alert moved" on email and Apprise. When any close in a delivery was not a recovery, `WebhookPayloadData.NotRecovered` drops recovery visuals: Discord's green embed turns neutral blue, both Teams cards lose recovery green, Mattermost's check emoji becomes an arrow, and ntfy uses the arrow tag for a single alert or a batch. Titles still say Resolved. PagerDuty still gets `resolve`. Every caller passes a nil resolution today, so ordinary recoveries and the disabled-threshold path are unchanged. The node release adopts the reason in a follow-up, once the host-agent node link records which metrics the agent evaluates (#2577 lane). Tests: `TestReleasedMetricAlertCarriesResolutionInsteadOfRecovery` and `TestResolvedHistoryRowKeepsResolutionWithoutEventLog` (operational_contract_test.go); `resolved_handover_test.go` drives the firing, its receipt and the queued close through the normal ntfy entry points, and renders Discord, both Teams cards, Mattermost, PagerDuty, a grouped Slack list, a grouped ntfy batch and email/Apprise content, with isolated managers. The full alerts, notifications and monitoring suites pass locally. |
||
|
rcourtman
|
69979370e9 |
Show each identifier once in the drawer's Identity card (#2590)
From a live user-lens pass over Machines. A Machines agent drawer's Identity card listed `agent:host-linux-1` as its Primary ID, its Discovery target and its Metrics Target. Its Aliases then repeated it again, along with the hostname and the Platform ID the card had just shown: eleven values, four of them the same ID. Every platform's drawer shares this card. A vSphere datastore repeated its ID the same way. **What changes** - **Discovery and Metrics Target** rows appear only when they name something not already on screen: the header's name (`getPreferredInfrastructureDisplayName`, which is what the header renders), the rows above, or the Platform ID row. - **Aliases** leave out those same values. - **Hidden names don't count.** Candidate names the drawer never shows don't count as shown, so a distinct target still appears. - **No false empty state.** A shown Platform ID counts as identity data, so filtering can't leave a "No identity metadata yet" message beside it. - **Display only.** Search and alias lookups keep the full lists. **Before and after (mock stack, same drawers)** - **Machines (Apollo-114)** - Before: Hostname · Primary ID · Discovery · Metrics Target (the last three identical) · Platform ID · IPs · Aliases: `agent:host-linux-1`, `host-linux-1`, `apollo-114`, `agent-acdf…` - After: Hostname · Primary ID · Platform ID · IPs · Aliases: `agent-acdf…` - **vSphere datastore (archive-tier)** - Before: Hostname · Primary ID · Metrics Target (same as Primary ID) · Platform ID · Tags · Aliases: Primary ID again, two short IDs, the hostname again, +1 - After: Hostname · Primary ID · Platform ID · Tags · Aliases: the two short IDs Checked live at 1440 and 390px. **Tests:** 1,278 tests pass. Three existing tests pinned the repetition and are updated. New tests cover: - an identity shown once; - a distinct target kept even when it matches a name the drawer never shows; - aliases filtered against the header's actual name; - a shown Platform ID counting as data. **Review:** a second reviewer found that my first version compared against names the drawer never shows, used a different header-name fallback, and could produce a false empty state. All three are fixed, and the reviewer confirmed. |
||
|
rcourtman
|
1f627a7ff3 |
Keep the availability result readable and give names room (#2588)
From a live user-lens pass over Machines → Availability checks. **The result clipped.** A check observed from several locations (Frontend Swarm service in the mock) showed "failed…" in its Result cell, with a "2 locations · 2/2 reporting" chip spilling past the column. The chip was given `flex` next to the badge's own `inline-flex`, which left it inline beside the result, so the result clipped at every width up to 1440px. The result and chip now share one line: - **The result** keeps its full width at the cell's right edge, where every row's result aligns. - **The chip** truncates to its left with its full text on hover. It now reads "2/2 locations reporting", the wording Settings uses for the same fact. - **Rows stay one line** because the table windows them at a single measured height. A second line would desynchronize scrolling on large estates. **Names were cut on phones.** Method and Target took the room, so check names read "Workshop do…" and "Frontend Sw…". Both now hide on phones, since the row drawer still shows them. The Method cell also truncates with an ellipsis instead of clipping. **Verification** - **Live, private mock stack**, at 390, 600, 768, 1024, 1440 and 1920px: - The result never clips. - Rows measure 32–33px at every width. - Phone check names read in full, except one row that is 3px short. - **Tests:** 3,977 tests pass across the standalone, Settings and shared suites, including new tests for the one-line result with a multi-location source and for the phone-hidden columns. - **Review:** a second reviewer flagged that my first version (chip on a second line) broke the table's uniform-height windowing. This one-line layout fixes it, and the reviewer confirmed. |
||
|
rcourtman
|
66c8c63e79 |
Hand a node only the usage metrics its agent evaluates (#2581)
This stacks on #2577. Once #2577 merges, the diff here reduces to the last commit. ## Problem `CheckHost` registered a host agent's node link before its disabled returns. With agent alerts switched off, or a host override disabling the agent, the node still released its CPU, memory and disk alerts to an agent that evaluated nothing, so nothing alerted on that machine's usage. The same happened per metric: an agent CPU threshold set to off with the node CPU threshold on left the machine with no CPU alert. The gap predates #2569; the old hostname dedup registered in the same place. ## Fix - The link now records which usage metrics the agent actually evaluated on its last report: a live CPU threshold, a live memory threshold with a known reading, and at least one filesystem with a live disk threshold. - `CheckHost` registers the link after evaluation and removes it on the disabled returns. - `CheckNode` releases only the metrics the agent owns and evaluates the rest itself. Deduplication can only remove a duplicate, never the last alert for a machine. - The link also carries the agent's display name and a stable lookup (`linkedHostAgentForNode`, lowest agent ID wins). The handover notification work builds on these. ## Tests `TestCheckNodeKeepsUsageMetricsTheAgentDoesNotEvaluate` covers two cases: agent alerts disabled (the node alerts on CPU and memory), and the agent CPU threshold off (the node keeps CPU while the agent keeps memory, so the machine still has exactly one memory alert). Both cases fail with the old registration order. The dedup tests use the new link struct. `go test ./internal/alerts/...` passes, and so do the monitoring alert, temperature, host-agent and node tests. ## Review follow-up (second commit) A review of the first commit found three more ways the node and its linked agent could disagree about who alerts: - **Memory ownership flipped with each report.** Ownership needed a known memory reading on every report. When the reading went missing while the agent's alert was open, the agent kept its alert but stopped owning memory, so the node raised a second alert. That alert resolved as soon as the reading came back. Ownership now follows the agent's configured thresholds, and a missing reading is treated as missing. - **Only one linked agent counted.** With several agents linked to one node, only the lowest agent ID's coverage was used. A metric is now owned when any linked agent evaluates it. - **Disk ownership covered the wrong filesystem.** A linked node's disk metric is the agent's summary filesystem: root when reported (`models.SummaryDisk`). Any evaluated mount was enough to take ownership, so with root disabled by a per-disk override and `/data` still alerting, a full root filesystem raised nothing. The agent now owns the node's disk metric only while it evaluates that same filesystem. `TestCheckNodeUsageOwnershipFollowsWhatAgentsEvaluate` covers all three cases; each fails without its fix. |
||
|
rcourtman
|
f419149458 |
Evaluate host agents in mock mode like live agents (#2585)
## Why `checkMockAlerts` evaluated mock nodes, guests, storage, PBS, PMG and Docker hosts but never host agents. Live agents are evaluated by `CheckHost` as each report lands; mock mode discards reports and the host health sweep never sees fixture agents, so no agent CPU, memory, disk, temperature or offline alert could open against mock data. A mock agent at 94% memory sat under the 85% default trigger with no alert, and any alert work on agents needed a private overlay build to reproduce. ## What Mock passes evaluate every fixture agent (`evaluateMockHostAgents`) with the live lifecycle boundaries: - Offline fixtures go through `HandleHostOfflineWithCorrelation`, because `CheckHost` treats its input as a fresh report and marks the host online. - Agents run before nodes, so a node with an online linked agent hands its CPU/memory/disk alerts to the agent on the first tick (host-agent hostname deduplication). - An agent that leaves the estate on a runtime mock config change goes through `HandleHostRemoved`, like a deleted live agent. - Leaving mock mode routes the fixture agents through `HandleHostRemoved`, releasing hostname registrations that `ClearActiveAlerts` leaves behind (they would otherwise suppress a real node named `pve1`). A pass already in flight finds mock mode off under the same lock and evaluates nothing. Evaluating agents exposed a mock-only reading: each Kubernetes pod took a whole single-pod footprint (10-83%) of its node's allocatable memory, so any node running four or more pods, and its linked agent, read a constant 100%. On the public demo's estate that became two permanent critical agent memory alerts. Pods now share 0.7 footprints per node (at most 58% in steady state), and a 60% per-node ceiling clips the transient while rescheduled pods' readings settle, so k8s nodes stay under 82%. Contract notes: `monitoring.md`, `performance-and-scalability.md`, `agent-lifecycle.md` (the `monitor.go` field touches only the alert-side removal boundary, with no tombstone, token or continuity change). ## Demo impact Measured on private stacks with the demo host's exact `PULSE_MOCK_*` estate (8 nodes, 2 standalone agents, 3 Kubernetes node agents): | | before | after | |---|---|---| | Kubernetes node agents' memory | 100%, 100% | 19-50% (up to ~76% briefly after boot) | | Agent alerts at default thresholds | none, ever | per-boot readings decide: one boot opened none, another opened Apollo-114 memory 91.3% (warning) | ## Verification - New tests, each shown to fail on the code it guards (via `go build -overlay`): `TestCheckMockAlertsEvaluatesHostAgentsBeforeLinkedNodes` (also fails with the agent loop after the node loop), `TestCheckMockHostAlertsUsesHostLifecycleForOfflineFixtures`, `TestLeavingMockModeReleasesFixtureAgentHostnames`, `TestMockHostAgentPassAfterLeavingMockModeRegistersNothing`, `TestMockHostAgentLeavingFixtureUsesRemovalLifecycle`, `TestMockKubernetesDenseNodesStayBelowAllocatableMemory`, `TestMockKubernetesRescheduledPodsDoNotSaturateReceivingNode`, `TestMockKubernetesLoneHeavyPodStaysUnderNodeCeiling`. - `go test ./internal/mock/ ./internal/monitoring/` pass. - Live, private demo-config stack: with the agent memory trigger at 60%, exactly the agents above it opened `agent:` memory alerts and their linked Proxmox nodes raised none; the Alerts page renders them. Shrinking the estate from 8 to 4 nodes through `/api/system/mock-mode` cleared the departed agents' alerts on the next pass. - Three rounds of independent review; the first two found the stale-registration, removal, relocation and steady-state ceiling issues fixed here, the third found none. |
||
|
rcourtman
|
0a55939c72 |
Keep open row drawers spanning the table when columns change (#2571)
An open inline row drawer measured its colspan from the summary row's visible cells only on mount and on resize. Hiding or showing a column with the drawer open, from View > Columns or through a data-driven column such as the vCenter column added in #2566, left the drawer with its old span. Reproduced on a mock stack before the change: - **Machines (`/standalone`), 1440px**: hiding Uptime with a drawer open left colspan 9 against 8 headers. The header and rows shrank to 939px of the 1375px table, beside a blank 436px column. Showing System left colspan 7 against 8, and the drawer stopped 79px short of the actions column. - **Proxmox guests (`/proxmox`), 1440/1024/768px**: hiding Type, Uptime or Backup left colspan one above the header count. The fixed layout collapses the phantom column to about 1px there. Fix, in the shared `InlineDetailTableRow`: - A MutationObserver on the summary row re-measures when cells are added, removed or hidden (class, style, hidden or colspan changes on a cell). Changes inside a cell's content are ignored, so live metric updates do not re-measure. - The span also re-syncs when the caller's requested colspan changes, which covers rows with no summary row to measure. - When the span grows, Chromium keeps a fixed-layout table's old column widths for the widened cell (minimal repro: 400px instead of 600px after a column is added and laid out). The row sets the cell's own width for one forced layout and restores it. Focus and inner scroll positions survive. After the change, on both tables at 1440 and 768px, the drawer colspan equals the visible header count after every toggle. The drawer, the last header cell and the last summary cell all end at the table edge, and the drawer row stays mounted. The vSphere Activity drawer still spans 6 of 6 columns. `InlineDetailTableRow.test.tsx` gains three tests that fail on main: removing and restoring a summary cell, hiding one in place, and changing the requested span with no summary row. It also becomes a registered frontend-primitives proof. The contract note sits beside the existing `InlineDetailTableRow` clause in `frontend-primitives.md`. Follow-up commits on this branch: - `InlineDetailTableRow.test.tsx` spies on `offsetWidth`. It requires exactly one forced layout read of the detail cell, at width 0px with the new colspan, when the requested span and the summary cells grow in the same update, and none when they shrink. A further test checks that a summary cell's colspan change re-measures while class and node churn inside a cell does not. Removing the forced read, doubling the workaround, removing it, or widening the observer filter each fails the matching test. - `tests/integration/tests/98-inline-drawer-column-span.spec.ts` is a real-browser check on the Machines table. It opens a drawer, hides and restores System through View > Columns, and asserts the following after each step: the drawer spans the visible headers, the drawer, last header and table share a right edge, and the drawer row, focus and any inner scroll survive. It fails on main before the fix (colspan 8 against 7) and with only the width workaround removed (drawer 79px short). It passed five runs in a row with the fix. |
||
|
rcourtman
|
ee3ebd800e |
Stop presenting a silent agent's node temperature as live (#2582)
A host agent linked to a Proxmox node merges with it into one read-state row, and every PVE poll refreshes that row's `LastSeen`. The agent's last sensors stay in state after it stops reporting. The node temperature lookup (`getHostAgentTemperatureByID`) checked neither, so each poll merged the silent agent's last reading into the node as a fresh measurement, stamped with the row's fresh `LastSeen`: - an open node temperature alert was re-observed every poll, so its `LastSeen` kept moving and the 24h stale-alert cleanup never reached it; a stale hot reading could keep activating - node history recorded the frozen value as new samples, and the UI and reports showed it as current - SSH collection was skipped as if a recent agent payload existed Since #2569 evaluates node temperature for agent-backed nodes, this holds their temperature alerts open. ## Change - `HostView.SourceStatus(source)` exposes the per-source freshness the registry already records (same accessor the VM, container and physical-disk views have). - The linked-agent lookup reads the agent source's own sighting, uses the agent's sensors only inside the reporting lease that keeps the agent online (`hostAgentHealthWindow`, as `evaluateHostAgents` uses), and stamps the reading with the agent's report time. Past the lease the node has no agent reading (the cluster sensor cache is still consulted), and alert evaluation treats that as a missing reading, not as evidence the node cooled down. - The poller's carry-over of a previous reading after a failed collection is bounded to max(5 min, 2 × PVE polling interval) and keeps its original timestamp. This branch does not engage today: the previous-node projection carries the unified resource id and no temperature, so it never matches the polled node. That is filed separately; the bound keeps the carry honest once it's repaired. The owner is the poller rather than `checkNodeTemperature`: the same stale value also reaches history, reports and the UI. An alerts-side `LastUpdate` check would also have missed this case, because the reading was stamped with the merged row's fresh `LastSeen`. Mock fixtures are untouched. ## Tests - `TestSilentLinkedAgentStopsRefreshingNodeTemperatureAlert` (registered monitoring proof): registry-backed merged agent+node row through the production previous-node projection and `CheckNode`. A reporting agent opens the alert. A silent agent leaves the alert open but stops refreshing `LastSeen`. A resumed agent is evaluated again. It fails on origin/main. - `TestView_HostViewSourceStatusSeparatesAgentFromMergedRow`: a merged row's `LastSeen` follows PVE, while `SourceStatus(SourceAgent)` keeps the agent's own report time. - `TestCollectNodeTemperatureCarryIsBoundedByCarryWindow`, `TestHostAgentReportCurrent`, `TestGetHostAgentTemperatureByID_IgnoresAgentPastReportingLease`. Out of scope, filed separately: `mergeHostAgentSMARTIntoDisks` copies silent agents' retained SMART temperatures into physical disks. |
||
|
rcourtman
|
c4bef05354 |
Give agent disk, array and sensor alerts their own resource type (#2583)
## Why Alerts a host agent raises about something other than its own machine (filesystems, disk temperatures, SMART disks, RAID and Unraid arrays, custom sensors) carried the host's `resourceType: agent`. The alert card's Monitoring menu writes policy to the alert's child resource id (for example `agent:<host>/disk:sda`), yet it named the Pulse agent as the inventory owner, which is the copy meant for the agent's own machine. Follow-up to #2574, whose card logic gives agent-platform children the neutral copy once their producers say what they are. ## What - `internal/alerts/host.go`: every host alert stamps `platformType: agent`. Child alerts carry `agent-disk` (filesystem usage, disk temperature, SMART health and wearout), `agent-storage` (RAID, Unraid array) or `agent-sensor` (custom sensors). The machine's own CPU, memory and offline alerts keep `agent`. - `internal/alerts/config/identity.go`: `agent-storage` and `agent-sensor` keep `agent` among their policy keys, as `agent-disk` already did, so agent thresholds and switches still govern them. - `internal/alerts/config_runtime.go`: configuration re-evaluation stops at the agent branch for host-agent alerts. Before, a SMART, RAID or sensor alert fell through to the guest switch, so disabling guest alerts resolved it on save and the next agent report raised and notified it again. - `internal/alerts/canonical_metric.go`: a metric alert held between its clear and trigger thresholds now refreshes `resourceType` and `platformType` from its producer, so an alert restored from a checkpoint written before these keys existed stops linking to the wrong page. Value, message and last-seen time still stay at the last breach. Webhook payloads report the new `resource_type` values for these alerts. ## Verification - `TestHostChildAlertsNameTheirOwnResourceType`, `TestHostChildAlertsIgnoreOtherPlatformSwitchesOnConfigSave` and `TestAlertCharacterizationHeldMetricAlertRefreshesItsClassification` fail on the previous code and pass now. `TestHostChildAlertsSurviveProxmoxNodeCleanup` guards the Proxmox node sweep, which keeps these alerts by their `agent:` resource id. `go test ./internal/alerts/...` passes. - Private mock stack, with before and after binaries on one data directory. A local-only overlay gave mock agents SMART, RAID, Unraid and custom-sensor readings and ran `CheckHost`. Before: every child alert's Monitoring menu read "Pulse agent remains the inventory owner". After: the host's memory alert still does, while the filesystem, SMART, RAID, sensor and Unraid alerts read "Source system remains the inventory owner", and all still link to `/standalone/machines`. Muting from the SMART card wrote operator state for `agent:host-linux-1/disk:sda` and left `agent:host-linux-1` unconfigured. - Restore path: a filesystem alert fired at 60%, held at 40%, and was saved by the old binary as `agent` with no platform. The new binary restored it, and after one evaluation it read `agent-disk` / `agent`, with its value, `lastSeen` and `diskUsagePercent` unchanged. |
||
|
rcourtman
|
f9ce06123a |
Say what an agent update brings and what the CPU count is (#2584)
Two small user-lens fixes from a live pass over Machines. **Outdated-agent notice wording** Machines told users to update "for the latest agent command support and agent-managed platform detail". Proxmox, TrueNAS and vSphere used similar internal phrasing: "agent-contributed Proxmox node detail and command support", "agent-contributed TrueNAS system detail and command support" and "in-guest telemetry and command support". All four now read "Update them to v4.26.0 for the latest fixes and machine details", or node, system or VM details. The notice does not promise Patrol actions or command support. It selects hosts by agent version alone, and command execution is a separate setting that the upgrade command preserves, so an update neither grants nor removes it. **CPU count** A machine's CPU cell reads "22% (18)", and nothing explained the 18: the hover panel listed only Usage and Load (1m). The shared CPU bar's hover panel now leads with "CPUs 18". The number is logical CPUs on a host and configured vCPUs on a guest, so "Cores" would be wrong on hyperthreaded machines. **Verification** - **Live, private mock stack:** - The Machines and Proxmox notices read the new wording. TrueNAS and vSphere have no outdated agents in the mock, so the updated page contract tests cover them. - Hovering a machine's CPU bar shows "CPUs 18" above Usage and Load. - **Tests:** 1,236 tests pass across the touched surfaces, including two new hover tests (row present, row omitted when the count is unknown). - **Review:** a second reviewer caught that "Patrol actions" overclaimed what an update brings, and that "Cores" mislabels logical CPUs and vCPUs. Both are fixed. |
||
|
rcourtman
|
bffd21f470 |
Keep row, drawer and replication ages moving on the shared clock (#2578)
Follow-up to #2573, which added the shared relative-time clock (`useRelativeTimeNow`) and put `PlatformTableRelativeTimeValue` cells and the platform alert tables on it. Several other ages were still computed once at render. Rows and open drawers reconcile in place, so those ages froze whenever the timestamp stopped changing. The frozen cases are the ones that matter: a silent agent, a completed Job, or a stalled replication scheduler. Now on the shared clock: - **Kubernetes controllers**: Job and CronJob Detail ages, plus the drawer's controller section annotation (`buildKubernetesControllerSection` takes `now`). - **Resource drawer**: container Created, Started and Finished rows, Last seen, Docker update-check age and Mail Gateway updated age. - **Proxmox backups**: full and compact age text, and the current/aging/stale band. The band never measures from earlier than the wall clock, so a backup that finished inside the clock's last tick can't band as an unknown, future age. - **Proxmox replication**: compact Last sync and Next sync, the expanded Last sync, and the Next sync countdown, which now turns overdue on its own. - **Machines**: identity subtitle (`seen ...` while the Last seen column is hidden) and the stale-agent Last report tooltip. Replication jobs were fetched once per Proxmox page mount. With moving ages, that snapshot would have aged a job that keeps syncing and counted it overdue. `ProxmoxPageSurface` now reads `/api/replication/jobs` through `createNonSuspendingQuery` with a 30-second background poll. A `createResource` refetch would have flashed the app-level Suspense fallback. A failed read keeps the last jobs and the tab, and a direct replication link holds until the first read resolves. Tests: one fake-timer test per surface shows the age advancing with no data change. Each of the 8 behaviour tests fails with the runtime files reverted. A ninth test covers the clock-trailing case for backup bands, and a surface test covers the replication background poll. Live check: a private mock stack with Playwright `page.clock`, `/ws` held after first render and REST reads replayed, then the clock jumped 3 hours (7 days for backups). Results: - Kubernetes: "Completed 48m ago" → "3h ago" - Drawer: "(48m ago)" → "(3h ago)", Last seen "1s ago" → "3 hours ago" - Replication: "in 8m" → "2h overdue"; phone "21m"/"8m" → "3h"/"-2h" - Backups: "1d" green → "8d" amber - Machines: "seen just now" → "seen 3h ago" The same run with the runtime files reverted stayed frozen. The browser receipt is in the commit. |
||
|
rcourtman
|
a649dd9b45 |
Tint Kubernetes node rows for their drawer's alerts
A Kubernetes node that runs a Pulse agent is an agent row, and the agent's alerts are keyed "agent:<id>" and labelled with the hostname. The nodes table matched only the row id and display name, so a node at 100% memory with a critical alert stayed untinted while its drawer listed the alert. The rows now use getUnifiedResourceAlertStyles, the set the drawer reads, and drop the tint while expanded like the Docker host and Machines rows. The overview's "Nodes needing attention" looked only at readiness and cordons, so that node was missing from the list too. It now ranks each node by the stronger of its state and its unacknowledged open alerts, NotReady and critical alerts first, and the nodes table's default order uses the same rank so a critical node no longer sorts below a degraded one. An acknowledged alert drops out of both, as it does from the tint. vSphere host and TrueNAS system rows keep the id-plus-host-name match. Their VMs', pools' and apps' alerts carry the host name as their node, the TrueNAS drawer states the same pool problems, and no producer keys alerts on those rows' "agent:" aliases (a Pulse agent on a TrueNAS box stays a separate row). A test now pins the TrueNAS pool rollup. |
||
|
rcourtman
|
7c8e9099a1 |
Clear agent disk temperature alerts when the disk leaves
A host-agent SMART disk temperature alert stayed active for a day once its disk left the agent's report (removed, replaced, or a renamed device path). CheckHost only evaluated disks still listed and node cleanup exempts agent: alerts, so only the 24-hour stale sweep resolved the alert or dropped a pending threshold run. CheckHost now tracks which disks a non-empty SMART list reports. Once a disk with an alert or a pending threshold run has been missing from three consecutive non-empty reports, the alert clears and the pending run is dropped. The Windows, FreeBSD and controller-multiplexed Linux collectors drop a disk whose probe fails, so a single omission does not count as the disk leaving, and the count restarts when the disk is reported again. An empty SMART list holds every alert, because the agent omits the list when collection fails or is unsupported. A listed disk in standby or without a temperature also holds its alert. Neither shows the disk cooled, and SMART health alerts already hold through standby. Turning the threshold off or removing the agent now drops pending runs and absence counts as well as alerts. Cleanup matches the disk_temp resource prefix alone, which also catches alerts raised under the original disk_temperature type. |
||
|
rcourtman
|
3b290640a5 |
Tint Docker host and Machines rows for their drawer's alerts (#2579)
Follow-up to #2575. The Docker host and Machines drawers list a resource's open alerts through `getAlertsForUnifiedResource`, but their table rows matched alerts differently, so a row could look healthy while its drawer said "Needs attention". - **Docker hosts:** rows called `getAlertStyles(host.id, …, host.name)`. Docker alerts are keyed `docker:<host source id>[/…]` and carry the hostname as their node, so a host whose display name differs from its hostname was never highlighted. Example on the mock estate: "Ops Services 01" (ops-services-01, source orion-2-mock) had three open service-health alerts and no highlight. - **Machines:** rows had no alert highlighting, so a machine with an open `agent:<id>` memory alert looked normal until its drawer was opened. `getUnifiedResourceAlertStyles` computes the existing row styles from the same set the drawer lists. Both tables now use it with the red/yellow tint the other platform host rows use: unacknowledged alerts only, and not on an expanded row. `getAlertStyles` keeps its id-plus-node matching for the tables whose alerts are keyed that way. **Verification** - Unit tests: the helper with Docker, agent, acknowledged-only and detection-off cases. Docker host rows are checked against their own drawer. Machines rows follow the live alert store through acknowledge, escalation and resolution. Both table tests fail on the previous table code. - Private mock stack (embedded build of this tree): every Docker host and machine row tinted exactly when its drawer lists an unacknowledged alert, at 1440px and 390px in light and dark. Acknowledging Apollo-114's memory alert from the Alerts page clears its row tint while the drawer keeps it as Acknowledged. Receipt: `frontend-modern/browser-verification/18ad64507908105c.json`. |
||
|
rcourtman
|
6413cda171 |
Judge legacy disk temperature alerts by the lowest trigger on save
A disk temperature alert persisted before CheckHost recorded its disk type has no diskType metadata until its next firing evaluation. A config save in that window judged it against the agent threshold. With a 75C agent default and a 70C nvme entry, a 72C alert was resolved on save and raised again by the next report. Without a known disk type, a save now judges the alert against the lowest enabled trigger the host's disks can use: the agent threshold or any DiskTempByType entry. A save therefore cannot resolve an alert its disk type would still fire, and turning the threshold off still resolves it. |
||
|
rcourtman
|
91575a285e |
Merge remote-tracking branch 'origin/fix/alert-handover-resolution' into fix/moved-alert-in-app
# Conflicts: # docs/release-control/v6/internal/subsystems/notifications.md |
||
|
rcourtman
|
47e6927819 | Merge remote-tracking branch 'origin/main' into fix/alert-handover-resolution | ||
|
rcourtman
|
a89c08a59e |
Drop recovery colours and check marks from moved alert closes
A second review found the moved-alert text was honest but the visuals still said recovered: Discord kept its green embed, both Teams cards kept recovery green, Mattermost led with a check emoji, and a grouped ntfy batch holding a moved alert kept the green check tag. A check mark on a phone is read before the sentence under it. Resolved webhook data now carries NotRecovered when any close in the delivery was not a recovery. The built-in templates switch to neutral colours and an arrow when it is set, and ntfy uses the arrow tag for a single alert or a batch. Titles still say Resolved, which stays true for the closed alert, and ordinary recoveries keep their green. The render tests now use an isolated data directory, stop their managers, and hand the captured request over a channel, so they no longer touch a shared queue or race on the captured body. |
||
|
rcourtman
|
23f603da9f |
Show a moved alert as moved in Patrol, history and assistant
A node alert handed to its Pulse agent closes with an AlertResolution (#2580), and notifications already say it moved. Every in-app reader still treated the close as a recovery: Patrol's detail wore a green Resolved badge over a timeline labelled "Recovery evidence", the alert history listed the row as resolved and its incident timeline said "Alert resolved: Node memory at 95%", and the assistant, Patrol's seed prompt and the pulse_alerts tool saw a plain resolve because GetRecentlyResolved dropped the resolution. Operational trust gains a non-recovery cause, ownership_transferred, which may only enter resolved and only with evidence. The resolve transition uses it whenever the alert carries a moved_to_agent resolution, so a handover during a snooze also stops counting as resolved while snoozed. models.Alert gains Resolution (reason, successor and the engine's summary), copied by cloneAlert and filled by GetRecentlyResolved, so the state snapshot, websocket, assistant context, Patrol seed and the resolved-alerts tool all report where the alert went. The resolved resource change carries the summary as its reason with alert_resolution metadata, and incident memory titles the close with it. In the UI, the history row reads "moved to agent" with the account in its title and on the phone card, and Patrol's detail shows a Moved badge with the handover sentence under the summary. |
||
|
rcourtman
|
982a6f6148 | Merge remote-tracking branch 'origin/main' into fix/moved-alert-in-app | ||
|
rcourtman
|
36b2e5fa0a |
Clear agent disk temperature alerts when their threshold is off
Turning off the agent disk temperature threshold left every open disk temperature alert active indefinitely. CheckHost's disabled branch was an empty placeholder, the host-wide clear paths (all agent alerts disabled, a disabled host override, agent removal, confirmed offline) matched only the /disk: prefix and missed /disk_temp:, and a config save skipped these alerts because diskTemperature was not classed as a threshold metric. CheckHost now clears a host's disk temperature alerts when the effective threshold is off, and the host-wide clears include them. A config save resolves them against the same threshold CheckHost evaluates, including the per-type DiskTempByType refinement. That way an unrelated save does not resolve an alert above its disk type's trigger only for the next report to raise it again. The save path now also matches the evaluator when there is no recovery band. buildCanonicalMetricSpec drops a clear level at or above the trigger, and the reducer keeps firing at the trigger itself, but reevaluateActiveAlertsLocked resolved any reading at or below that clear level. A per-type entry such as 70/75 (normalization keeps it) resolved a 72C alert on save, and the next report raised it again. With no recovery band, a reading exactly at the trigger was resolved the same way. |
||
|
rcourtman
|
4884052bc5 |
Show each identifier once in the drawer's Identity card
A Machines agent drawer listed agent:host-linux-1 as its Primary ID, its Discovery target and its Metrics Target, then again in Aliases next to the hostname and Platform ID it had just shown. Every platform's drawer shares this card, and a vSphere datastore repeated its ID the same way. Discovery and Metrics Target rows now appear only when they name something not already on screen: the header's name, the rows above, or the Platform ID row. Aliases leave out those same values. Names the drawer never shows do not count, so a distinct target still appears. A shown Platform ID counts as identity data, so filtering cannot leave a false "No identity metadata yet". Search and alias lookups keep the full lists; only the card's display changes. |
||
|
rcourtman
|
27051deb27 |
Dedup node alerts by agent link and keep node thresholds on save (#2577)
A review of #2569 (node temperature alerts with a host agent) found three more ways node alerts went wrong. Each was reproduced with a test that fails on `main`. ## Changes - **Dedup follows the agent's node link, not its hostname.** Node deduplication matched an agent's hostname against node names across every PVE instance. With two instances that each have a node called `pve` and an agent on only one, both nodes released their CPU, memory and disk alerts, so the second machine went unmonitored. An agent reporting an FQDN never matched its node, so that machine alerted twice. An operator unlinking an agent didn't hand the alerts back either. The manager now tracks each reporting agent's `LinkedNodeID`, which the monitor already sets automatically or from an operator link. On pulse-prod the delly2 agent is linked to `homelab-delly2`, the same ID as the node alert it should dedup. - **A config save keeps node thresholds.** Saving the alert config resolved node temperature alerts that were still over their trigger. Reevaluation only treated an alert as a node alert when its `Instance` equalled its node name. Current node alerts keep the PVE instance name there (`homelab`), so they fell through to guest thresholds, and guests have no temperature threshold. Metadata `resourceType: node` now decides, and the `Instance` heuristic only covers legacy alerts that have no resource type. - **Missing readings interrupt timing runs.** Since #2569, a missing temperature reading no longer resolves an open alert, but a pending run still spanned the gap. One hot sample after ten minutes without readings fired at once, and a recovery run could complete the same way. The reducer gains `InterruptMetricRun`, which drops a pending run or restarts a recovery run without resolving anything. The node temperature check calls it, along with the explicit-intent pending clear, when the reading is missing. ## Not in this PR Two other review findings are filed separately: - When collection fails, the poller re-presents a stale temperature as current with no time limit, and alerting treats it as live. The fix touches the poller, mock timestamps and what the UI shows. - At handover, the released node alert closes with "now healthy" wording. That needs a typed resolution reason in notifications. ## Tests - `TestHostAgentDeduplicationFollowsNodeLink`: same-named nodes in two instances, an FQDN hostname, an unlinked agent, and an operator unlink. - `TestConfigSaveKeepsNodeTemperatureAlertOverTrigger`: fails on `main`. - `TestCheckNodeMissingTemperatureInterruptsTimingRuns`: fails on `main` (one sample back after the gap fires at once). - The existing dedup tests now use link semantics. `go test ./internal/alerts/...` passes, and so do the monitoring alert, temperature, host-agent and node tests. |
||
|
rcourtman
|
4775fcbf38 |
Base node usage ownership on agent config, all links, summary disk
A review of the previous commit found three ways the node and its linked agent could still disagree about who alerts. Memory ownership required a known reading on every report. When an agent's memory reading went missing while its alert was open, the agent kept that alert (a missing reading is not a recovery) but stopped owning memory, so the node raised a second alert. It resolved again as soon as the reading returned, with history and notifications for each flip. Ownership now follows the agent's configured thresholds, not one report's data. With several agents linked to one node, only the lowest agent ID's coverage counted. If that agent had CPU off and another linked agent evaluated CPU, the node alerted on CPU as well. A metric is now owned when any linked agent evaluates it. Disk ownership needed any evaluated filesystem, but a linked node's disk metric is the agent's summary filesystem (root when reported, see models.SummaryDisk). With root disabled by a per-disk override and /data still alerting, the node gave up its root disk alert and a full root filesystem raised nothing. The agent now owns the node's disk metric only while it evaluates that same filesystem. |
||
|
rcourtman
|
d4e94a7e34 |
Keep the availability result readable and give names room
A check observed from several locations showed "failed…" in its Result cell, with a "2 locations · 2/2 reporting" chip spilling past the column. The chip was given flex next to the badge's own inline-flex, which left it inline beside the result, so the result clipped at every width up to 1440px. The result and the chip now share one line. The result keeps its full width at the cell's right edge, and the chip, now "2/2 locations reporting" as Settings words it, truncates to its left with the full text on hover. Rows stay one line because the table windows them at a single measured height. On phones, Method and Target took the room and cut check names to "Workshop do…". They now hide there, as the row drawer still shows them, and the Method cell truncates with an ellipsis instead of clipping. |
||
|
rcourtman
|
c4db7a3fba |
Revalidate restored backup alerts before startup notifications (#2572)
After restart, critical backup-age alerts restored from JSON or SQLite could notify before a delayed first backup check removed obsolete rollup identities. Hold their firing and escalation until valid backup evidence clears the incident or confirms it is still overdue. Confirmed overdue incidents resume normal delivery while retaining the original occurrence and acknowledgement, snooze, cooldown and quiet-hours policy. Verified the original supplied-data reproducer and final startup regression cases with Go 1.26.8. The full `go test -race -p 2 ./internal/alerts/... -count=1 -timeout=8m` passed on Linux, along with staged commit checks and status, registry and contract audits. The commit uses the documented contract-neutral exception for notification ordering without changing alert truth, identity, restart authority or the delivery API. |
||
|
rcourtman
|
a6535ec0d1 | Merge remote-tracking branch 'origin/main' into fix/alert-handover-resolution | ||
|
rcourtman
|
602e2d04cd |
Move the handover tests off the end of the contract test file
main appended platform-stamping tests to the end of operational_contract_test.go after this branch appended the handover tests there, so the two insertions conflicted. Placing the handover tests beside the resolved-history test lets main merge in cleanly. No test changes. |
||
|
rcourtman
|
e3a42d86c3 |
Evaluate host agents in mock mode like live agents
checkMockAlerts evaluated mock nodes, guests, storage, PBS, PMG and Docker hosts but never host agents. Live agents are evaluated by CheckHost as each report lands; mock mode discards reports and the host health sweep never sees fixture agents, so no agent CPU, memory, disk, temperature or offline alert could open against mock data. A mock agent at 94% memory sat under an 85% trigger with no alert, and alert work on agents needed a private build to reproduce anything. Mock passes now evaluate every fixture agent with the live lifecycle boundaries: - offline fixtures go through HandleHostOfflineWithCorrelation, since CheckHost treats its input as a fresh report and marks it online; - agents run before nodes, so a node with an online linked agent hands its metric alerts to the agent on the first tick; - an agent that leaves the estate on a runtime mock config change goes through HandleHostRemoved, like a deleted live agent; - leaving mock mode routes the fixture agents through HandleHostRemoved, releasing hostname registrations that ClearActiveAlerts leaves behind and that would otherwise suppress a real node named pve1; a pass already in flight finds mock mode off and evaluates nothing. Evaluating agents exposed a mock-only reading: every Kubernetes pod took a single-pod share (10-80%) of its node's allocatable memory, so any node running four or more pods, and its linked agent, read a constant 100%. On the public demo's estate that became two permanent critical memory alerts no real cluster would raise. Pods now share 0.7 single-pod footprints per node (at most 58% of allocatable in steady state), and a 60% per-node ceiling clips the transient while rescheduled pods' readings settle, so those nodes stay under 82%. |
||
|
rcourtman
|
56e88d5561 |
Say what an agent update brings and what the CPU count is
The outdated-agent notice told Machines users to update "for the latest
agent command support and agent-managed platform detail". Proxmox,
TrueNAS and vSphere used similar internal phrasing ("agent-contributed
node detail", "in-guest telemetry"). It now reads "for the latest fixes
and machine details", or node, system or VM details. It does not
promise Patrol actions or command support: the notice selects hosts by
version alone, and command execution is a separate setting that the
update keeps as it is.
A machine's CPU cell reads "22% (18)", and nothing explained the 18.
The shared CPU bar's hover panel now names it "CPUs". That is logical
CPUs on a host and configured vCPUs on a guest, so "Cores" would be
wrong on hyperthreaded machines.
|
||
|
rcourtman
|
019963b60e |
Give agent disk, array and sensor alerts their own resource type
Host-agent alerts about something other than the machine itself (filesystems, disk temperatures, SMART disks, RAID and Unraid arrays, custom sensors) cloned the host's alert metadata and carried resourceType "agent". The alert card's Monitoring menu writes policy to their child resource id, yet it named the Pulse agent as the inventory owner, the copy meant for retiring the agent's own machine. CheckHost now stamps platformType "agent" on every host alert and gives child alerts agent-disk, agent-storage or agent-sensor, so the card still links to Machines and uses the neutral source-system copy. Webhook payloads carry the same resourceType values. Each child type keeps "agent" among its CanonicalResourceTypeKeys, and configuration re-evaluation now treats agent policy as final for agent-keyed alerts. Before, a SMART, RAID or sensor alert fell through to the guest switch, so disabling guest alerts resolved it on save and the next agent report raised and notified it again. A metric alert held between its clear and trigger thresholds returned before merging metadata, so one restored from a checkpoint written before platformType or these child types existed kept linking to the wrong page until it re-fired or cleared. The hold now refreshes resourceType and platformType only; value, message and last-seen time stay at the last breach. |
||
|
rcourtman
|
0dd4d1690d |
Stop presenting a silent agent's node temperature as live
A host agent linked to a Proxmox node merges with it into one read-state row, and every PVE poll refreshes that row's LastSeen. The agent's last sensors stay in state after it stops reporting. The node temperature lookup checked neither: it returned the retained sensors stamped with the row's LastSeen, so each poll presented the silent agent's last reading as a fresh measurement. An open node temperature alert was re-observed every poll (LastSeen kept moving, so stale-alert cleanup never reached it), a stale hot reading could keep activating, node history recorded it as new samples, the UI showed it as current, and SSH collection was skipped as if a recent agent payload existed. Since #2569 evaluates node temperature for agent-backed nodes, this holds their temperature alerts open. The lookup now reads the agent source's own sighting through a new HostView.SourceStatus (matching the VM, container and disk views), uses the agent's sensors only inside the reporting lease that keeps the agent online, and stamps the reading with the agent's report time. Past the lease the node has no agent reading, which alert evaluation treats as missing rather than as evidence the node cooled down. The poller's carry-over of a previous reading when collection fails is also bounded to max(5m, 2 x PVE polling interval) and keeps its original timestamp. That branch does not engage today, because the previous-node projection carries the unified resource id and no temperature, so it never matches the polled node; the bound keeps it from presenting a frozen value as current once that projection is repaired. |
||
|
rcourtman
|
7f83d73cfd |
Tighten the drawer span tests
The unit test only showed that a width write happened, so a doubled nudge or a missing forced layout read would still pass. It now spies on offsetWidth and requires exactly one forced read of the detail cell, at width 0px with the new colspan, when the requested span and the summary cells grow in the same update, and none when they shrink. The browser spec read every `thead th` under the table, which would also match headers of tables inside the drawer. It now reads only the outer table's own header row, and uses the frontend Prettier settings (single quotes, 100 columns) instead of Prettier's defaults. |
||
|
rcourtman
|
b19c6aaf70 |
Stamp alert platforms so cards stop guessing the policy owner (#2574)
Follow-up to #2560 and #2564. A review of both found alert cards that still linked to the wrong platform page and named the wrong inventory owner in the Monitoring policy menu. **Metric and connection alerts carry their platform.** Metric alerts put a display label in `metadata.resourceType` (`vSphere VM`, `Kubernetes Pod`, `TrueNAS Pool`) and carried no platform, so every Kubernetes, TrueNAS and vSphere metric alert fell through to Proxmox ("Proxmox remains the inventory owner" on a vSphere VM CPU alert). The unified metric and capacity evaluators now stamp the canonical `platformType` beside the label, and connection-degraded alerts stamp it from the connection type (a TrueNAS connection alert also linked to Proxmox). The card already reads `platformType` first. **Names no longer outrank types.** For alerts without platform metadata, the card's fallback now routes by resource type first and lets the `agent:` id prefix, the `docker` id substring and the "vmware" message decide only when the alert has no type. Proxmox guest ids embed the user's cluster and node names, so a VM on node `docker-01` went to Docker (a regression from #2560), one in a cluster named `agent` went to Machines, and a guest named `vmware-test` went to vSphere. **Agent copy only for the agent's machine.** Libvirt VMs and Unraid arrays in resource drawers got "Retiring this machine … Agent removal remains available from Machines"; they now get the neutral source-system copy. The menu also capitalises the owner label at sentence start. Not in this change: agent disk-usage, SMART disk and Unraid array *alerts* still carry their host's `agent` metadata (producer fix filed separately), and alerts restored from before this change gain `platformType` at their next full evaluation. Verified live on a private mock stack at 1440 and 390, against main's frontend and alert metadata in the same session. Receipt: `frontend-modern/browser-verification/e80596c28c0384f3.json`. |
||
|
rcourtman
|
123821b02a |
Carry a typed reason when an alert closes without recovering
When a Pulse agent is linked to a Proxmox node, the node releases its CPU, memory and disk alerts to the agent, and any node alert still open closes. The close has to reach integrations so a PagerDuty or webhook incident opened from the node alert does not stay open forever, but it went out as a recovery: "pve1 on pve1 is now healthy" for a node still at 95% memory, with synthesized "recovery projection" evidence behind it. #1682 is the same complaint from another trigger: a close announced as "now healthy" when nothing recovered. A resolved alert can now carry an AlertResolution saying why it closed. The first reason is moved_to_agent, with the agent's alert resource ID and name. releaseCanonicalMetricAlert, the disabled-threshold release pulled out of evaluateCanonicalMetricAlert, takes the resolution, and clearAlertWithResolution stamps a copy on the alert before it resolves. It rides the Alert JSON into the resolved callback, the delivery queue, the history row and the event log. The synthesized closing evidence and the resolve transition say what happened instead of claiming a recovery projection. Notifications still send the close and change only the words: "Memory alert moved to pve1 (Host Agent). This is not a recovery: check the agent for the current reading." for webhook messages, a MOVED title without the green check on ntfy, and "Pulse alert moved" on email and Apprise. A nil resolution, which every caller passes today, leaves ordinary recoveries and the disabled-threshold path unchanged. The node release adopts the reason once the host-agent node link records which metrics the agent evaluates. |
||
|
rcourtman
|
15dc868c7e |
Tint Docker host and Machines rows for their drawer's alerts
The Docker host and Machines drawers list a resource's open alerts through getAlertsForUnifiedResource, but their table rows matched alerts differently, so a row could look healthy while its drawer said "Needs attention". Docker host rows called getAlertStyles with the row id and display name. Docker alerts are keyed docker:<host source id>[/...] and carry the hostname as their node, so a host whose display name differs from its hostname (mock "Ops Services 01" / ops-services-01, three open service-health alerts) was never highlighted. Machines rows had no alert highlighting at all, so a machine with an open agent:<id> memory alert looked normal until its drawer was opened. getUnifiedResourceAlertStyles computes the existing row styles from the same set the drawer lists. Both tables use it with the red/yellow tint the other platform host rows use: unacknowledged alerts only, and not on an expanded row. getAlertStyles keeps its id-plus-node matching for the tables whose alerts are keyed that way. |
||
|
rcourtman
|
9206486d56 |
Hand a node only the usage metrics its agent evaluates
CheckHost registered a host agent's node link before its disabled returns. With agent alerts switched off, or a host override disabling the agent, the node still released its CPU, memory and disk alerts to an agent that evaluated nothing, so nothing alerted on that machine's usage. The same happened per metric: an agent CPU threshold set to off with the node CPU threshold on left the machine with no CPU alert. The link now records which usage metrics the agent actually evaluated on its last report: a live CPU threshold, a live memory threshold with a known reading, and at least one filesystem with a live disk threshold. CheckHost registers it after evaluation and removes it on the disabled returns. CheckNode releases only the metrics the agent owns and evaluates the rest itself, so deduplication can only remove a duplicate, never the last alert for a machine. The link also carries the agent's display name, which the handover notification work builds on. |
||
|
rcourtman
|
362ab2f235 |
Show open alerts in the Machines and Docker host drawers (#2575)
The Machines/Infrastructure drawer and the Docker host drawer built "Needs attention" from the alerts embedded in each resource. Nothing in the backend ever filled that list, so every resource carried `alerts: []` and both drawers stayed silent while an alert was open (a mock stack had 0 of 142 resources with embedded alerts while 14 alerts were active; a real dev Pulse showed `alerts: []` on a host with an open temperature alert). **What changes** - Both drawers read the websocket `activeAlerts` map, the source the Proxmox node, guest and Mail Gateway drawers already use, through a new `getAlertsForUnifiedResource` in `frontend-modern/src/utils/alerts.ts`. It matches the ids the backend writes for a resource (unified id, canonical primary/superseded ids, Proxmox source id, metrics target) and, for agent and Docker hosts, `agent:<id>` and `docker:<host source id>` plus the disk, RAID, container and service alerts nested under them. Rows show resource and metric like the other drawers, most severe first. - The dead embedded path is removed: frontend `ResourceAlert` and the unused `hasAlerts` filter, the Proxmox summary `alertCount` nothing rendered, and backend `ResourceFrontend.alerts` / `ResourceAlertInput` (which also drops `"alerts":[]` from every resource in each broadcast). - With incident alerts now reaching the infrastructure drawer, a health reason counts as stated when it is a whole sentence of an alert's text, so a storage incident alert that joins its summary and impact does not repeat beside either reason. - `utils/__tests__/alerts.test.ts` and `internal/models/converters_additional_test.go` are registered as proof files for the policies that own `utils/alerts.ts` and the resource converters. **Verification** - Unit tests for the matcher across agent, Proxmox-linked agent, Docker host, PBS/PMG and pod identity shapes copied from a live `/api/state`; drawer tests for a host plus a nested disk alert, a Docker host with nested service alerts, and the incident-sentence dedupe (each fails on the previous code). - Private mock stack on this tree at 1440 and 390 px: Apollo-114's backend-raised memory alert appears in its Machines drawer, a machine with no alert shows no section, Ops Services 01 lists its three service alerts, and the vSphere `archive-tier` and `Edge Stateful` drawers show one row each. Receipt: `frontend-modern/browser-verification/89dcce7f1458e2ba.json`. |
||
|
rcourtman
|
75e4ed44a6 |
Keep row, drawer and replication ages moving on the shared clock
The shared relative-time clock from the alert Started change reached only PlatformTableRelativeTimeValue cells and the platform alert tables. Other ages were still computed once at render, so they froze while a row or an open drawer stayed mounted and its timestamp did not change: Kubernetes Job and CronJob Detail ages and the drawer's controller section, the drawer's container Created, Started and Finished rows and its Last seen, the Machines identity subtitle and stale-agent tooltip, Proxmox backup ages and their current, aging or stale band, and the Proxmox replication compact ages, expanded Last sync and Next sync countdown. The frozen cases are the ones that matter: a silent agent read "seen just now" for hours, and a stalled replication scheduler's "in 3m" never turned overdue. Each owning component now reads useRelativeTimeNow and passes it as `now`. The backup band measures from the later of the clock and the wall clock, because the clock can trail by one tick and a backup that finished inside it must not band as a future, unknown age. Replication jobs were read once per page mount, so a moving Last sync would have aged a job that keeps syncing and counted it overdue. The Proxmox surface now reads them through createNonSuspendingQuery with a 30-second background poll. A failed read keeps the last jobs and the tab, and a direct replication link holds until the first read resolves. |
||
|
rcourtman
|
7aa537abe1 |
Read platform alert Started as an age that keeps moving (#2573)
The Docker, Kubernetes, TrueNAS and vSphere alert tables showed Started as "Oct 6, 10:34 AM". It clipped on two or three rows at 390 and 768px (TrueNAS, vSphere), cutting off the time of day. **What changes** - **Started reads as an age.** An open alert's Started cell answers how long it has been going on, so it now reads "2h ago", as alert cards do. The cell's hover title is the exact time, and the drawer's Started row keeps it. Unset or invalid times stay "-". - **Docker alignment.** Docker's Started column now uses the `numeric-value` alignment kind, like the other three tables. - **A shared clock, which fixes a wider bug.** Platform tables keep their rows mounted across data refreshes, so an age computed at render froze whenever its timestamp did not change: a row could read "2h ago" three hours later. That already applied to every `PlatformTableRelativeTimeValue` cell, including Kubernetes events, Docker volumes and vSphere activity. - `useRelativeTimeNow` (`frontend-modern/src/utils/relativeTimeClock.ts`) ticks every 30 seconds while any cell reads it, and stops when none do. - `formatRelativeTime` takes an optional `now`. - The shared relative-time cell and the four alert tables pass the clock in, and the shared-primitive guardrails pin that. **Verification** - **Private mock stack, TrueNAS and vSphere:** Started no longer clips at any width from 390 to 1920px. The hover shows the exact time. - **Docker and Kubernetes:** neither has provider incidents in the mock, so I injected one, started two hours earlier, into the API responses with the websocket held. Both read "2h ago" with the exact time on hover, right-aligned and unclipped at 390, 768 and 1440. - **The clock, live:** I installed Playwright's page clock and held the websocket so no data changed. - vSphere Started ages went from "9m ago" to "3h ago" after a three-hour fast-forward. - vSphere Activity went from "1d ago" to "2d ago" after 24 hours. - A unit test proves the same and fails without the clock. - **Tests:** 20,903 frontend tests pass, along with the shared-template audit and the completion guard. Six `agentInstallCommand` shell tests also fail on unmodified main on macOS, so they're environmental. - **Review:** a second reviewer found that ages froze on mounted rows, which led to the shared clock. Its re-review found no defects. It also listed older spots that compute ages once outside these cells: Kubernetes Job details, the Kubernetes drawer annotation, Proxmox backup and replication ages, and machine subtitles. Those move onto the clock in a separate change. |
||
|
rcourtman
|
0bd9064513 |
Add a browser check for drawer span after column toggles
The unit tests cannot see layout, so a regression in the Chromium width workaround would only show in a real browser. The new spec opens a Machines drawer, hides and restores System through View > Columns, and checks after each step that the drawer spans the visible headers, that the drawer, last header and table share a right edge, and that the drawer row, focus and any inner scroll position survive. It fails on main before the fix (colspan 8 against 7 after hiding) and with only the width workaround removed (drawer 79px short of the table edge after showing), and passed five runs in a row with the fix. |
||
|
rcourtman
|
7ee8c0d1e2 |
Dedup node alerts by agent link and keep node thresholds on save
A review of the node temperature fix found three more ways node alerts went wrong. Node deduplication matched a host agent's hostname against node names across every PVE instance. With two instances that each have a node called pve and an agent on only one of them, both nodes released their CPU, memory and disk alerts, so the second machine went unmonitored. An agent reporting an FQDN never matched its node, so that machine alerted twice, and an operator unlinking an agent did not hand the alerts back. Dedup now keys on the agent's LinkedNodeID, which the monitor already sets automatically or from an operator link. On pulse-prod the delly2 agent is linked to homelab-delly2, the node alert ID it should dedup. Saving the alert config resolved node temperature alerts that were still over their trigger. Reevaluation only treated an alert as a node alert when its Instance equalled its node name, but current node alerts keep the PVE instance name there, so they fell through to guest thresholds, which have no temperature threshold. Metadata resourceType "node" now decides, and the Instance heuristic only covers legacy alerts. A missing temperature reading no longer resolves an open alert, but a pending run still spanned the gap: one hot sample after ten minutes without readings fired at once, and a recovery run could complete the same way. The reducer gains InterruptMetricRun, which drops a pending run or restarts a recovery run without resolving anything, and the node temperature check calls it when the reading is missing. |
||
|
rcourtman
|
18b46a4d14 |
Guard the drawer span nudge and cell-churn filter in tests
The width assertion in the first commit would still pass if the Chromium relayout nudge were deleted, and nothing checked that churn inside a summary cell is ignored. jsdom has no layout, so the new test records the detail cell's style writes: one width change when the span grows, none when it shrinks. A second test checks that a summary cell's colspan change re-measures while class and node churn inside a cell does not. Each fails when the nudge is removed or the observer filter is widened to every mutation. |
||
|
rcourtman
|
da3fcf69f7 |
Show open alerts in the Machines and Docker host drawers
The Machines/Infrastructure drawer and the Docker host drawer built their "Needs attention" list from the alerts embedded in each resource. No backend code ever filled that list, so every resource carried `alerts: []` and both drawers stayed silent while an alert was open: a mock stack had 0 of 142 resources with embedded alerts while 14 alerts were active, and a real dev Pulse showed `alerts: []` on a host whose temperature alert was open. Both drawers now read the websocket activeAlerts map, the source the Proxmox node, guest and Mail Gateway drawers already use, through one matcher, getAlertsForUnifiedResource. It keys a resource on the ids the backend writes for it (unified id, canonical primary and superseded ids, Proxmox source id, metrics target) and, for agent and Docker hosts, agent:<id> and docker:<host source id> plus the disk, RAID, container and service alerts nested under them. Rows show the affected resource and metric like the other drawers, most severe first. The dead embedded path is gone: the frontend ResourceAlert type and its unused hasAlerts filter, the Proxmox summary alertCount nobody rendered, and ResourceFrontend.alerts / ResourceAlertInput on the backend, which also drops "alerts":[] from every resource in each broadcast. Now that incident alerts reach the infrastructure drawer, a health reason counts as stated when it is a whole sentence of an alert's text, so a storage incident alert that joins its summary and impact does not repeat beside either reason. |
||
|
rcourtman
|
942f6b257a |
Open an issue when the weekly signing preflight fails (#2570)
## Why #2568 added a weekly schedule to `release-signing-preflight.yml`. For a failed scheduled run, GitHub only emails whoever last edited the cron line, and only if their notification settings allow it. If a bot edits the file later, the alerts go to the bot. A credential problem needs a signal that stays visible until it is fixed. ## What changes - A `report-scheduled-result` job runs only for scheduled runs, with only `issues: write`. - On failure it opens an issue titled "Release signing preflight is failing", or comments on the open one. The next passing scheduled run closes it. - The open issue is found by a server-side title search followed by an exact title match, however many newer issues are open. - Cancelled or skipped runs and manual dispatches report nothing. - The issue links to the run and contains no credential details. - `.github/workflows/README.md` describes the behaviour. ## Checks - `scripts/check_workflow_trust.py` passes. - The reporting script was exercised against a stubbed `gh` for a first failure, a repeat failure, recovery, a pass with no issue, and a cancelled run. |
||
|
rcourtman
|
911296fa9d |
Stamp alert platforms so cards stop guessing the policy owner
Two reviews of #2560 found alert cards that still linked to the wrong platform page and named the wrong inventory owner in the Monitoring policy menu, because the card guessed the platform. Metric alerts put a display label in metadata.resourceType ("vSphere VM", "Kubernetes Pod", "TrueNAS Pool") and carried no platform, so every Kubernetes, TrueNAS and vSphere metric alert fell through to Proxmox. A vSphere VM CPU alert said "Proxmox remains the inventory owner". The unified metric and capacity evaluators now stamp the canonical platformType beside the label, and connection-degraded alerts stamp it from their connection type (a TrueNAS connection alert also linked to Proxmox). The card already reads platformType ahead of every fallback. For alerts with no platform metadata, ids and messages carry names the user chose: a Proxmox guest id embeds its cluster and node names and a powered-off message names the guest. A VM on node docker-01 went to Docker and named the container runtime (a regression from #2560's precedence change), a VM in a cluster named agent went to Machines, and a guest named vmware-test went to vSphere. The card's fallback now routes by resource type first and lets the id prefix, id substring and message decide only when the alert has no type. Libvirt VMs and Unraid arrays arrive through the agent, so their drawers got "Retiring this machine ... Agent removal remains available from Machines", which points at the host's agent. The agent copy is now kept to the agent's own machine; everything else an agent reports gets the neutral source-system copy. Agent disk-usage, SMART disk and Unraid array alerts still carry their host's agent metadata and are left to a separate change. Alerts restored from before this change gain platformType at their next full evaluation. The menu also opened sentences with a lowercase owner label ("container runtime remains the inventory owner"); it now capitalises the label at sentence start. |