mirror of
https://github.com/rcourtman/Pulse.git
synced 2026-10-24 04:33:17 +00:00
396bb5d517f14466795a2ccd2ff438fdba0ab077
5122 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 rcourtman
|
396bb5d517 |
Keep a disk heat finding open without a current reading
Patrol verifies a disk-high finding as recovered once the disk's temperature cools to its alert clear value. A disk in standby, or one whose host agent stopped reporting, has no current reading, and a missing reading counted as cooled. A finding raised for a SATA disk at 56C would close as soon as the disk spun down, while its temperature alert stayed open. When a disk has no current reading and its last known reading is still at or above the clear value, verification now reports the recovery as unknown instead of proven. A disk last seen cool still recovers. |
||
|
rcourtman
|
35bf55af23 |
Colour report disk temperatures by the alert thresholds
The PDF performance report's disk table coloured a temperature amber from 50C and red from 60C for every disk type, the last fixed disk heat line after disk heat moved to the alert disk temperature policy. An NVMe at 63C, well under its 70C alert trigger, printed red. A user's raised trigger changed nothing. Each reported disk now carries the tenant's alert thresholds for its type (the clear value and the trigger, from alerts.Manager.DiskTemperatureThreshold). The PDF colours the reading amber from the clear value and red from the trigger, as the Physical Disks Temp column does, and leaves it plain when disk temperature alerting is off. |
||
|
rcourtman
|
3fd5411ecc |
Judge disk heat by the alert disk temperature thresholds
Pulse decided whether a physical disk was too hot in three places with three thresholds. Disk risk added temperature_high at a flat 60/70C for every disk type and drove the disk status and the Physical Disks Health verdict. Patrol flagged anything over 55C and the AI chat context listed disks from 50C. Alerts and the Temp column used the per-type thresholds users tune in Alerts (NVMe 70, SAS 65, SATA 55 by default). An NVMe at 63C read Running Hot beside a green Temp cell. A SATA disk at 56C read Healthy beside a red one. A user who raised the NVMe trigger to 75 still got a hot verdict at 70. The alert disk temperature policy now decides disk heat. alerts.Manager.DiskTemperatureThreshold exposes the per-type resolution CheckHost already applies. - Disk risk no longer judges temperature. Every registry that rebuilds risk lacks the alert configuration, so no rule there could honour it. A hot disk keeps an online status. - The Health verdict reads Running Hot when a current reading reaches the disk type's trigger. It uses the thresholds object that colours the Temp cell, so the two cannot disagree. The amber band below the trigger stays a cell colour, not a verdict. - The TrueNAS storage table lost the heat it used to get from risk. It now judges heat the same way: an Attention row with the reason, and a warning dot for phones. - Patrol triage, its disk issue filter, finding recovery and the chat context's attention list judge the collected reading against the same trigger, including scoped runs and a provider set before Patrol starts. - A switched-off agent Disk Temp default resolves to no thresholds, which the displays that read them now treat as off. Per-host Disk Temp overrides do not reach these surfaces yet. TrueNAS disk alerts, the TrueNAS disk drawer and PDF reports keep their own fixed temperatures for now. |
||
|
rcourtman
|
1b6a9b45d7 |
Keep retained disk temperatures out of AI, Patrol and reports (#2625)
Normalization keeps a disk's last-known temperature when the current observation did not collect one (a disk in standby, a host agent past its reporting lease) and marks it non-available. These readers still took `PhysicalDisk.Temperature` at face value; each now goes through `diskinventory.TemperatureCollected`: - **AI chat context**: "Physical Disks Needing Attention" no longer flags a disk for heat from a retained reading. A disk listed for another reason shows it as `last known 52C (host agent stopped reporting)`. - **Patrol**: the disk health-issue gate, triage flags and `disk-high` finding verification judge only a collected reading. Both seed tables label a retained one as last known. - **AI tools**: `pulse_query` / `pulse_metrics` physical disks and `pulse_storage disk_health` put a retained value in `last_known_temperature` + `last_known_temperature_reason` instead of `temperature`. - **Reports**: the performance report's disk table and the reporting runtime snapshot leave a retained value out (no reading). - **Disk temperature charts**: a short series is padded out to now only with a collected reading. Otherwise stored samples are returned as they are. - `monitor.go`'s private copy of the predicate is replaced by the shared helper. Each new test reproduces against the pre-change code: retained `unavailable` temperatures (standby / silent agent) beside a collected one. |
||
|
rcourtman
|
684c0dc598 |
Show a host whose agent stopped reporting as offline (#2634)
A host agent that stops reporting is never shown as offline in the Machines table. Its row renders its last report as if it were current. ## Why The monitor marks a host agent offline once its reporting lease expires, and raises "Host is offline". The registry's stale pass marks the agent sighting stale after 60 seconds, and `aggregateStatus` ranked stale above offline. So the machine reached every consumer as `warning`. On the Machines table that meant an amber dot. CPU, memory, disk, network, disk I/O, uptime and temperature all rendered as live, with threshold colours. Meanwhile the row's own drawer led with the critical offline alert and read "Observed state: Degraded". Docker hosts and Kubernetes clusters hold the same kind of lease, and theirs is shorter than the 120-second stale threshold. A silent Docker host therefore read offline at first, then flipped to warning once its sighting went stale. ## What changes - **Backend.** Each sighting records when a push reporter delivered its machine, Docker host or cluster as offline. The reporters are the host agent, Docker and Kubernetes. Status aggregation counts that sighting as offline whatever its age. - The sighting itself still reads `stale`, so health keeps its `telemetry_stale` reason and monitored-system reasons are unchanged. - A live source still carries a merged row. A Proxmox node whose agent died stays online through the PVE poll. - Pull sources, and the guests, containers, pods and disks that push collectors report about, keep the stale-to-warning rule. - A JSON copy re-derives the marker from its stored status. - A manual link joining two resources from one source keeps the fresher sighting. - **Frontend.** Offline rows already blank CPU, memory, disk, network and disk I/O through the danger gate. - The Machines Uptime and Temp cells and the Docker hosts Temp cell had no gate, and now use it. - The drawer drops an offline resource's uptime and a silent agent's uptime. - The drawer marks a silent agent's non-disk Thermals rows "(last known)" with the reason as title, as retained disk rows already are. ## Verification - Go tests cover the agent, Docker, Kubernetes, mixed-source, Proxmox-node, JSON round-trip and manual-link cases, plus the unchanged late-agent and stale-PVE boundaries. Each fix's test fails without it. The `unifiedresources` package passes. - The `api`, `monitoring`, `ai`, `alerts` and `agentcontext` suites ran on pulse-dev. One `internal/api` test failed there, `TestServerInfoEndpointReportsDevelopment`, and it fails identically on the pre-fix build in that scratch tree. - Frontend tests cover the Machines offline row, the Docker offline Temp cell, the drawer thermals marking and the drawer uptime rows. The adjacent suites pass. - Live check on a private mock stack, holding one standalone Linux agent and one Docker host past their leases, at 1440 and 390 wide. Before: amber row with live-looking readings. After: red row with every reading blanked, the drawer reading "Offline", and thermals marked last known. - The browser receipt is included. Bundle sizes are within budget: WorkloadsSurface is unchanged, Standalone +0.04 kB, Docker +0.01 kB. Builds on rcourtman/Pulse#2630 and updates its contract sentence, which described the lapsed agent as reaching the table as `warning`. Known follow-up: the drawer's Disks card still threshold-colours a silent agent's retained filesystem usage. |
||
|
rcourtman
|
c06af69fcb |
Show a host whose agent stopped reporting as offline
A host agent past its reporting lease is offline: the monitor marks it so and raises "Host is offline". The registry's stale pass ranked the agent's stale sighting above offline, though, so the machine reached every consumer as a warning. The Machines table showed it amber and rendered its last CPU, memory, disk and temperature as current readings, while its own drawer led with the critical offline alert. A Docker host was offline only until its sighting crossed the 120-second stale threshold, then flipped to warning, and Kubernetes clusters likewise. Each sighting now records when a push reporter (host agent, Docker, Kubernetes) delivered its machine, Docker host or cluster as offline, and status aggregation counts that as offline whatever the sighting's age. The sighting itself still reads stale, and a live source still carries a merged row, so a Proxmox node with a dead agent stays online. A copy that went through JSON re-derives the marker from its stored status, and a manual link joining two resources from one source keeps the fresher sighting, so a retired agent cannot pull a live one down. Offline rows then take the existing danger gate. The Machines Uptime and Temp cells and the Docker hosts Temp cell had no gate and now follow it. The drawer drops a silent agent's uptime and marks its non-disk Thermals rows "(last known)", as retained disk rows already are. |
||
|
rcourtman
|
344f1f8531 |
Raise disk health alerts for mock physical disks
checkMockAlerts evaluated guests, agents, nodes, storage, PBS, PMG and Docker against the mock estate but never physical disks. The fixture keeps a FAILED cohort and worn SSDs stable on purpose, so mock and demo stacks showed failing and "Replace Now" disks with no disk-health or disk-wearout alert, and the disk alert lifecycle could not be exercised without real hardware. The pass applies the physical disk poller's boundary: only disks on online nodes, matched by instance and node name, with the linked agent's --disk-exclude patterns forcing a matched device healthy. The per-disk evaluation moves into checkPhysicalDiskAlerts so the poller and the mock pass share one rule. The poller's agent-link settle deferral is not applied, because fixture links and exclusions are complete from the first pass. Like the other loops in checkMockAlerts, the disk loop runs after the pass's single mode check, so the existing race with leaving mock mode now covers disk alerts too. That race also involves other fixture evaluation paths and is left for a monitor-wide fix. On the public demo's estate (8 nodes) this adds one warning-level disk-wearout alert and no disk-health alert: that estate has no FAILED Proxmox disk. |
||
|
rcourtman
|
fb7dd955ed |
Keep retained disk temperatures out of AI, Patrol and reports
Normalization keeps a disk's last-known temperature when the current observation did not collect one (a disk in standby, a host agent past its reporting lease) and marks it non-available. Several readers still took PhysicalDisk.Temperature at face value. The AI chat context listed a standby disk as needing attention for heat. Patrol raised health issues and triage flags from the retained value. The AI query and disk-health tools handed it to the model as the disk's temperature. Performance reports and the reporting runtime snapshot tabulated it as measured, and the disk temperature charts padded a series out to now with it. Each of them now goes through diskinventory.TemperatureCollected. AI surfaces keep a retained value as context, labelled last known with its reason. Reports and the charts leave it out. monitor.go's private copy of the predicate is replaced by the shared helper. |
||
|
rcourtman
|
d9ffb39ce8 |
Keep a merged disk's temperature under its own row's state (#2618)
Stacked on #2609; until it merges this branch also carries its commit, and the change here is the last commit. ## Why The registry merges a physical disk's rows (Proxmox, host agent SMART, Unraid inventory), choosing the temperature by source preference and merging collection states separately. When two rows carry different temperatures, the shown value could sit under the other row's `available` state. A real case: a node whose disk temperature Proxmox collects now (`proxmox_node_smart`, 40) and whose linked host agent has gone silent (retained 72, `unavailable`). In either ingest order the merged disk showed **72 labelled available**, so risk, history and the UI treated the frozen reading as current. ## What - `pairPhysicalDiskTemperatureState` runs at the end of each merge step. When the merged state says collected but the row the shown value came from says it was not, the merged state becomes that row's own. - It never changes a value and never grants availability, so it cannot revive a withdrawn reading or override a resumed agent's fresh one. - Rows with the same value keep the merged state, so the agent's own withdrawal still supersedes the Proxmox copy (#2593). - With that guarantee the Unraid-native row no longer needs to withhold `available` (#2609 restricted it), so it carries the same derived state as a SMART row borrowing its reading. ## Known limit This is exact for the two rows of one merge step. The merged disk keeps one state per field, so with three or more rows for one disk, a withdrawal that an earlier step replaced is invisible to a later step. Sol found two such counterexamples, both needing three rows (one also needs coincidentally equal values, the other a legacy agent plus two Proxmox rows). Closing that needs each reading's own state tracked across merges, and that redesign is filed separately. Preferring another source's current value instead of the retained one was tried and dropped: it needs an ordering between same-source states that the rows do not record, and each version reopened a withdrawn reading somewhere. ## Tests - `TestPhysicalDiskMergePairsTemperatureWithItsCollectionState`, in both ingest orders: - silent agent against a collected Proxmox reading, and against no Proxmox reading - reporting agent - silent agent against a Proxmox copy of its earlier reading, with and without SMART attributes - resumed agent against a Proxmox copy of its withdrawal - standby agent against a Proxmox copy of its pre-sleep reading - `TestPhysicalDiskMergeNeverPresentsAWithdrawnTemperatureAsCollected`: three distinct-valued withdrawn or stale rows in all six orders. - `TestUnraidInventoryRowDoesNotMaskTheSMARTRowState`: the Unraid inventory's `available` no longer lends itself to a SMART row's uncollected reading. - Without the rule, the peer-reported case, the Unraid masking case and one three-way order fail. |
||
|
rcourtman
|
dcb69cfb02 |
Record Docker host, service and sub-resource alert history (#2620)
Follows #2608, which made `MonitorAdapter.RecordChange` resolve Proxmox and agent alert references to canonical resources. Three kinds of alert reference still journaled under strings no resource answers to: - Docker host alerts (`docker:<host ID>`) - Swarm service alerts (`docker:<host ID>/service/<ID>`) - sub-resource alerts such as ZFS pool and device health (`<storage ID>/zfs-pool:<pool>[/device:<dev>]`) and host filesystem, SMART, RAID and sensor alerts (`agent:<host ID>/disk:…` etc.) Resource facets, timelines, resource intelligence and Assistant context read history by canonical ID, so these alerts never appeared there. ## What changes - **Docker host refs** resolve through the registry's Docker source ID. - **Swarm service refs** resolve through the host's Swarm cluster to the service with that exact ID, from any manager that reports it. A service with no ID is referenced by its name in the same slot, so while a cluster has such a service, every service ref there is a conflict and binds nothing. - **Never bind:** container names, shortened container IDs and hostless `docker-service:<name>` refs never bind, and are no longer retried against later inventory. - **Sub-resource refs** bind to the owner whose durable ID they carry: - ZFS pool and device alerts go to the storage. - The Unraid array alert goes to its array storage. - Filesystem, SMART, disk temperature, RAID and custom-sensor alerts go to the host. - The owner must resolve to exactly one resource of the expected type. An ambiguous or incompatible owner is a conflict. - Mount and kernel device labels can name a different disk after a reboot, so these alerts never bind a physical disk. - **Retries:** a reference that hits a conflict at write time now enters the backfill retry set, the same as one found at startup. ## Known residuals (producer-side, follow-up proposed) - A container with no ID alerts under its host's reference (`DockerResourceID(host, "")`), so its events join that host's history (never another container's). - An ID-less service whose name equals another service's ID shares that service's reference. Rows it journaled can join the other service's history once a later generation or event binds the reference. Swarm always reports service IDs, so this needs a malformed report. Both are documented in the contract. Separating them needs a producer change in `internal/alerts`. ## Evidence Private `PULSE_MOCK_MODE` stack, fresh data dir. The binary at #2608's tip ran first, then this change on the same dir. Alert rows were classified by source reference (from `alert_identifier`) and checked against each owner's `/api/resources/{id}/timeline`: | ref shape | rows readable via owner timeline, before | after | |---|---|---| | Docker host | 0 / 1 | 3 / 3 | | Swarm service | 0 / 5 | 19 / 19 | | ZFS pool | 0 / 9 | 26 / 26 | | ZFS device | 0 / 9 | 27 / 27 | | agent filesystem | 0 / 1 | 3 / 3 | | Docker container (mock short IDs) | 0 / 11 | 0 / 30, no container alias written | Tests: - `TestHistoryIdentityMonitorAdapterResolvesDockerHostAndServiceReferences` covers multi-manager services, a Docker host merged into its agent machine, legacy-row backfill, recovery after removal, ID-less service conflicts, and names and short IDs staying unbound. - `TestHistoryIdentityMonitorAdapterResolvesSubResourceReferences` covers every owner shape, wrong-type and ambiguous owners, retained bindings, write-time conflict retry, and physical disks staying unbound. - `TestOwnerAlertTimelinesUseCanonicalHistoryIdentity` drives the real alert manager for Docker host offline, Swarm service gap, ZFS pool/device and host filesystem alerts. It fails on #2608's tip. UI reach: the Docker, storage and platform host-row drawers don't request per-resource history. This fix reaches the facets/timeline/intelligence APIs and Assistant resource context, not a drawer. |
||
|
rcourtman
|
0fb5ccb62b |
Keep node temperatures off same-named nodes at other sites (#2617)
Two Proxmox connections can each have a node with the same name, which
is common for MSPs running one `px1` per site. With one host agent
linked to site A's `px1`, `getHostAgentTemperatureByID("siteB-px1",
"px1")` returned site A's temperature. The hostname fallback took the
first agent with that hostname, even one linked to another node, and
even when the name was ambiguous enough that the agent linker had
refused to link it. The cluster sensor cache leaked the same way: SSH
readings an agent collects for its cluster siblings were keyed by bare
node name. Site B's `px1` then showed, alerted on and recorded history
for another machine's sensors.
## Change
- The poller hands the lookup the polled node
(`getHostAgentTemperatureForNode`,
`carriedTemperatureOutlivesAgentLease`), and the lookup works on that
node's **slot**: its own source ID, plus, when the read state does not
hold that ID, the one same-named node proven to be the same machine.
- The state folds the views of one machine reached through two
connections (a cluster added twice, or a multi-homed host added by each
address) into a single node kept under one view's ID, and an automatic
agent link follows it. A narrowed fallback on its own would leave those
views without a temperature permanently; that was reproduced with real
`UpdateNodesForInstance` folds.
- Same-machine proof comes from the state's own evidence: new
`models.NodeObservationsSameMachine` and
`models.HostAgentBridgesNodeViews`
(`internal/models/node_machine_identity.go`). `UpdateNodesForInstance`'s
agent-corroboration check now shares its rule with them. Two or more
candidates are ambiguous and none is taken, as the state declines an
ambiguous alias.
- An agent linked to the slot is the node's agent. The hostname fallback
only takes an agent with no node, VM or container link, and only when no
node outside the slot and no second unlinked agent has the name.
- Cluster sensor cache entries record their reporting agent. A reading
serves a node only when that agent's linked node shares one of the
slot's connections, or belongs to a connection whose configured view of
the node has a matching TLS fingerprint (the other connection of a
cluster added twice).
`LinkedNodeID` holds the Proxmox source node ID for every link source
(automatic `findLinkedProxmoxEntityWithHints` → `NodeView.SourceID()`,
manual `LinkHostAgentToNode` validating against `models.Node.ID`,
reconciliation, continuity restore), so the link match needs no
namespace normalization. A manual link stays pinned to the ID the
operator chose (the state never transfers intent to a replacement
provider identity), so a node held under the other view's ID has no
linked agent. The physical-disk SMART merge already matches by
`node.LinkedAgentID` within one instance and is unchanged.
## Tests
- `TestNodeTemperatureStaysWithinSameNamedNodesSite` (registered
monitoring proof, `monitor_additional_test.go`), through
`collectNodeTemperatureData` against registry-backed and state-folded
read states: agent linked to the other site's node; ambiguous unlinked
agents; unambiguous unlinked agent (still served); cluster added through
two connections (alternating real state folds); multi-homed host via two
standalone connections; ambiguous address alias; a namesake agent with a
contradicting fingerprint; sibling readings across both connections of
one cluster; sibling reading from the other site. The leaking cases fail
on main, and each folded-view guard was checked with an overlay negative
control.
- `TestApplyHostReportScopesClusterSensorsToReportersConnection`
(`monitor_host_agents_test.go`): real `ApplyHostReport` ingest of
cluster sensors.
- `TestNodeObservationsSameMachineMatchesStateFold` and
`TestHostAgentBridgesNodeViewsMatchesStateFold` pin the predicates
against `UpdateNodesForInstance`'s actual fold.
- Full `internal/monitoring`, `internal/models` and
`internal/unifiedresources` pass on pulse-dev.
|
||
|
rcourtman
|
8259a74c9d |
Show the live reading of threshold alerts held below trigger (#2567)
A threshold alert stays open after its reading drops below the trigger. It holds until the reading reaches the clear level, then must stay there for the recovery delay. While it held, the evaluator never touched the alert, so every surface kept showing the last breach. On minipc the drawer said "Node temperature at 80.0°C" next to a green 72°C in the node table, with nothing explaining why the warning was still there. #2068 reports the same thing for a disk alert. **What changes** - Open threshold alerts carry a volatile `metricStatus`: the value Pulse is comparing now (the average for rolling-average rules, with the latest sample beside it), the phase (`breaching`, `latched` or `recovering`), the trigger and clear levels, and recovery progress from the reducer's monotonic run. `value`, `message` and `lastSeen` keep meaning the last breach, and the hold branch still never notifies. - The status is stripped from the active-state checkpoint, the recovery mirror and the degraded recovery projection, so holding adds no durable writes. Resolved records carry none. - Stale-alert cleanup and the retention TTLs use the latest observation. Previously an alert held below its trigger for more than a day was auto-resolved as "resource no longer being monitored". - Node, guest and Mail Gateway drawer rows lead with the reading now and say what clears the alert, with the last breach on hover. The Proxmox TEMP cell keeps its own open temperature alert's tone and explains it on hover. Storage rows stop saying "Over N%" once usage is back under the limit, and the Assistant handoff no longer labels the breach value as current. **Verified** - Unit tests drive a node temperature alert through 85 → 78 → 72 → 75 → 77 → 72 → resolved, and fail against the old evaluator. - Private mock stack with pve2 stepped 88 → 78 → 72°C at desktop and phone widths; see the browser receipt in this PR. Refs #2068 |
||
|
rcourtman
|
38a5334b90 |
Judge disk risk only on a temperature collected now (#2611)
Since #2593, a disk whose host agent stops reporting keeps its last SMART temperature as a last-known value marked `unavailable`, and a disk in standby keeps its pre-sleep reading the same way. The shared disk risk assessment still fed that value into its temperature rule (>=70C critical, >=60C warning). So a silent agent's frozen 72°C made the Physical Disks row read **Replace Now** ("Disk temperature is 72C") with a `warning` resource status, while #2603 makes the temperature cell beside it say the reading is not current. ## Change - `storagehealth.CollectedTemperature(temperature, collection)` passes a temperature through only when its `collection.temperature` state is `available`, or when the source predates collection state (nil or empty state). Otherwise it returns 0 before risk judges it. - Three places use it: - `AssessPhysicalDisk`, for Proxmox inventory disks, through the unified adapter; - `AssessHostSMARTDisk(WithThresholds)`, for a host agent's own SMART rows; - `physicalDiskAssessmentFromMeta`, the registry's recompute after it merges sources. - The retained value stays on the resource as last-known history. Health, wearout and SMART counters are unchanged: they are durable evidence and keep judging as before. Alerts are unaffected: - SMART disk-health alerts already discard `temperature_high` (`internal/alerts/host.go`). - Disk temperature alerts evaluate the agent's own fresh report. - No fresh collector emits a positive temperature with a non-available state. ## Evidence On a private mock stack, with a local generator overlay stamping real collector states: - Before: the silent-agent disk (72°C retained) read **Replace Now** and the API reported `status: warning`, `risk: critical`. - After: it reads **Healthy**, the API reports `online` with no risk, and its temperature shows as last known. - A genuinely hot disk (72°C `available`) still reads Replace Now. Whether "Replace Now" is the right word for a heat-only verdict is a separate question. ## Tests - `internal/storagehealth/risk_test.go`: `TestCollectedTemperatureKeepsOnlyCurrentReadings` and `TestDiskAssessmentsIgnoreRetainedTemperature`. - `internal/unifiedresources/registry_test.go`: `TestPhysicalDiskRiskIgnoresRetainedTemperature`. It covers Proxmox disks in every collection state through the monitor adapter, plus a real agent+Proxmox merge of the same disk, collected and retained. - Mutation-checked: removing any of the three gates fails a case, and the merged case fails without the registry-recompute gate. - Full `internal/storagehealth`, `internal/unifiedresources`, `internal/api/resourceapi` and `internal/truenas` pass, plus the SMART/disk/host subsets of `internal/alerts` and `internal/monitoring`. ## Related - The Unraid inventory temperature path in `adapters.go` (`assessUnraidPhysicalDisk`) gets the same gate in the Unraid change that stamps its collection state. - A registry merge can still pair a retained agent temperature with a Proxmox row's `available` state (`registry.go`, Proxmox-over-agent carry). That is being fixed separately. |
||
|
rcourtman
|
cbedbc7edf |
Stop serving retained disk temperatures as current readings (#2609)
Follow-up to #2593 (silent agent SMART readings), which made retained disk temperatures carry a non-available collection state and kept them out of history. ## Why Normalization may keep a disk temperature it did not collect (a disk in standby, a host agent past its reporting lease), provided the collection state says so. #2593 stopped the history writers recording those values. Two consumers still presented them as current: - **History live point.** When a disk range has no samples, `/api/metrics-store/history` answered with the disk's temperature as a `source: "live"` point at now, whatever its state. Now that retained temperatures never reach history, a short-range chart for a silent agent's disk or a sleeping disk showed the retained value as a fresh reading. Mock mode also seeded synthetic series from it, and padded sparse stored series out to now. - **Unraid inventory temperatures.** Unraid array rows carry no per-field provenance, so a temperature taken from one (the Unraid-native disk row, or a SMART row falling back to the inventory reading) could not be told apart from a collected one once the agent went silent. `ExpireHostTelemetry` only marks SMART rows. ## What - `diskinventory.TemperatureCollected` is the shared form of the rule the history writers apply (no state, or `available`). The router's disk live point, the mock synthetic seed and the mock padding all require it. - The unified-resources adapter derives the state of an Unraid inventory temperature from the host lease that `ExpireHostTelemetry` already records atomically, and from the disk's spin state: `unavailable` from `unraid` ("host agent stopped reporting", or "disk is reported spun down"), otherwise `available`. The value stays as last-known context. - A SMART row that borrows the inventory reading takes that state, including `available`. - The Unraid-native row records only the withdrawing states. The registry picks a merged disk's temperature and its state separately, so an `available` claim from the provenance-less row would mask the SMART row's own state for the same disk. - A withdrawn inventory temperature no longer counts as heat in the disk's risk assessment. - Storing per-row state on `models.HostUnraidDisk` instead would have added a model field, a read-state round trip and a frontend type change just to duplicate `host.Status`. ## Tests - `TestContract_MetricsHistoryDiskLivePointOnlyForCollectedTemperature`: a silent agent's disk (through the real `ExpireHostTelemetry` and registry path) and a standby disk get no live point in single-metric or all-metrics responses, and no value-derived mock series. Sparse stored samples are not padded to now for a retained, absent or unknown disk. A collected disk still gets its live point and mock series. Each router gate was reverted individually and the test failed each time. - `TestSilentUnraidAgentDiskTemperatureIsRetainedButNotCollected`: three Unraid disks (inventory only, SMART row without a temperature, SMART row whose temperature is unsupported) go collected, then retained but not collected after `evaluateHostAgents` expires the lease, then collected again. - `TestUnraidDiskTemperatureFollowsTheHostLease`: reporting, silent and spun-down hosts across the native row, the borrowed reading and a SMART row with its own reading, including risk. - `TestUnraidInventoryRowDoesNotMaskTheSMARTRowState`: the merged-registry case, which fails if the native row claims `available`. - `TestTemperatureCollectedOnlyForAReadingTakenByThisObservation`: the helper. |
||
|
rcourtman
|
f4c1046215 |
Record Docker host, service and sub-resource alert history
Alert lifecycle events resolve to canonical resources since Proxmox and
agent references gained durable resolution, but Docker host alerts
("docker:<host ID>"), Swarm service alerts ("docker:<host ID>/service/<ID>")
and sub-resource alerts such as ZFS pool and device health
("<storage ID>/zfs-pool:<pool>") or host filesystem usage
("agent:<host ID>/disk:var") were still journaled under references no
resource answers to. Resource facets, timelines, resource intelligence and
Assistant context read history by canonical ID, so they never saw them.
Docker host and service references now resolve through the registry's own
Docker source identities. A service reference must carry the service's real
ID in the host's Swarm cluster, from any manager that reports it; services
without an ID are referenced by name and never bind. Container names and
shortened IDs still bind nothing and are no longer retried against later
inventory.
Sub-resource references bind to the owner whose durable ID they carry: the
storage for ZFS pools and devices, the Unraid array storage, and the host
for filesystem, SMART disk, disk temperature, RAID and custom-sensor
alerts. Mount and kernel device labels can name a different disk after a
reboot, so they never bind a physical disk. Any other shape stays unbound.
On a mock stack none of the Docker host, Swarm service, ZFS and agent disk
alert rows were readable through their owner's timeline API before this
change; all of them are after it, including rows journaled earlier. The
mock's shortened container ID rows stay unbound.
|
||
|
rcourtman
|
1b081c6475 |
Record Proxmox and agent alert history on the canonical resource (#2608)
Alert lifecycle changes (fired, acknowledged, snoozed, resolved) are
journaled with the alert's source-native resource ID.
`MonitorAdapter.RecordChange` only mapped full Docker container IDs, so
Proxmox node, guest and storage alerts and host-agent alerts landed
under IDs such as `instance-node`, `instance:node:vmid` or
`agent:<host>`. Resource facets, resource intelligence
(`recent_changes`) and Assistant resource context read history by
canonical ID, so they never showed those alerts. Incident timelines were
unaffected because they read by alert identifier.
**Change**
- `RecordChange` resolves non-Docker references to the canonical
resource and records a history binding, so the reference keeps reaching
the same history after a guest migration or inventory removal.
- Only durable identities bind: canonical ID, retired era, source ID,
node-scoped guest reference, `agent:<host ID>`. Names and hostnames
never bind (a VM named `pulse-system` cannot capture system alerts). A
reference that two resources answer to binds to neither. Docker keeps
its full-ID rule.
- Rows already journaled under a source reference join canonical history
without being rewritten. Each registry generation binds the unbound
alert references it can resolve, both those found at startup and those
written before inventory named the resource, retrying each for 30
minutes.
**Evidence** (private mock stack, same data dir, origin/main first, then
this branch)
- main: 51 of 52 alert references were source-shaped and there were zero
bindings. `GET /api/resources/{id}/facets` for affected nodes and guests
returned only `state_transition`.
- this branch: the first registry generation logged `bound 17 alert
history references`. Every non-Docker whole-resource reference is now
bound: node metric and connectivity alerts, guests, Proxmox storage,
`agent:host-linux-1` and `pmg-main`. Facets and
`/api/ai/intelligence?resource_id=` now include
`alert_fired`/`alert_resolved` for the VM, node and storage checked.
- Still unbound, as before: ZFS pool/device and agent-disk sub-resource
references (no resource of their own) and Docker host/service/short-ID
references.
**Where it shows**
In the UI, only the full-presentation `ResourceDetailDrawer` requests
this history. Proxmox node and Agents rows open the `table-row` drawer,
and guest and storage rows open their own drawers; none of these fetch
it (confirmed live: the node History tab makes no facets or intelligence
request). So the user-visible gain is in Assistant and Patrol resource
context and the facets and intelligence APIs, not in a Proxmox drawer.
**Tests**
`TestHistoryIdentityMonitorAdapterResolvesProxmoxAlertReferences` and
`TestHistoryIdentityBindsLegacyAlertRowsFromRegistryGenerations`
(registered `history_identity_test.go`), plus
`TestProxmoxAlertTimelineUsesCanonicalHistoryIdentity` through the real
alert-manager callback. The monitor test fails on main. Mutation checks
confirm the new tests catch alias-based resolution, a missing post-scan
retry and ignored ambiguity.
|
||
|
rcourtman
|
41b94b29a8 |
Keep a merged disk's temperature under its own row's state
The registry merges a physical disk's rows (Proxmox, host agent SMART, Unraid inventory), choosing the temperature by source preference and merging field states separately. When two rows carry different temperatures the shown value could sit under the other row's "available" state: a silent agent's retained reading carried over a Proxmox row that collected its own reading now, in either ingest order. Risk, history and the UI then treated a frozen reading as current. pairPhysicalDiskTemperatureState runs at the end of each merge step: when the merged state says collected but the row the shown value came from says it was not, the merged state becomes that row's own. It never changes a value and never grants availability, so it cannot revive a withdrawn reading or override a resumed agent's fresh one; rows with the same value keep the merged state. This is exact for the two rows of a step. The merged disk keeps one state per field, so with three or more rows a withdrawal an earlier step replaced is invisible to a later one; closing that needs each reading's own state tracked across merges. Preferring another source's current value instead would need an ordering between rows that the merge does not record. With this in place the Unraid-native row no longer needs to withhold an "available" state to avoid masking a SMART row, so it now carries the same derived state as a SMART row borrowing its reading. |
||
|
rcourtman
|
8c9000ac63 |
Keep node temperatures off same-named nodes at other sites
Two Proxmox connections can each have a node with the same name, which is common for MSPs running one px1 per site. The node temperature lookup fell back to the first host agent whose hostname matched the node name, even when that agent was linked to the other site's node, and even when the name was ambiguous enough that the agent linker had refused to link it. Site B's px1 then showed, alerted on and recorded history for site A's sensors. The cluster sensor cache leaked the same way: an agent's SSH readings for its cluster siblings were keyed by bare node name, so a site A sibling reading fed site B's node. The poller now hands the lookup the polled node, and the lookup works on that node's slot. The state folds the views of one machine reached through two connections (a cluster added twice, or a multi-homed host added by each address) into one node held under one view's ID, and the agent link follows it. Those views used to find their agent only through the loose hostname match, so a polled view the read state does not hold also takes the one same-named node proven to be the same machine. models.NodeObservationsSameMachine and models.HostAgentBridgesNodeViews give that proof from the evidence UpdateNodesForInstance folds on, and the state's agent corroboration check now shares its rule with them. Ambiguous candidates are declined, as the state declines an ambiguous alias. An agent linked to the slot is the node's agent. The hostname fallback only takes an agent with no link when no node outside the slot and no second unlinked agent has the name. Cluster sensor cache entries record their reporting agent and serve a node only when that agent's linked node shares one of the slot's connections, or belongs to a connection whose configured view of the node has a matching TLS fingerprint. LinkedNodeID holds the Proxmox source node ID for automatic, manual and continuity-restored links, so the link match needs no normalization. The physical-disk SMART merge already matches agents by node.LinkedAgentID within one instance and is unchanged. |
||
|
rcourtman
|
3ac5664d95 |
Key previous Proxmox nodes by source ID so poll carry-overs match (#2604)
The Proxmox poller reads its previous poll's nodes back from the unified read state, but `previousNodeFromView` returned the registry's unified resource ID (e.g. `agent-ab60dcc1ebca813b`) instead of the poller's source ID (`pve1-node1`), and dropped the temperature. Registry-backed probes confirmed every lookup keyed on the source ID missed: - `nodeLastOnline` in the offline grace policy; - the temperature carry, plus the CPU low/record tracking, which reset to the current reading every poll. The dev stack shows it live: `delly` and `minipc` report `cpuMin = cpuMaxRecord = cpuPackage`; - the node memory carry after a failed status read. When a whole instance poll failed, `markPVEInstanceNodesUnreachable` wrote the projected nodes back, so every failed poll re-ingested the node under a new unified ID (`agent-ab60…` → `agent-aec93…` → `agent-e798…`). The projection also dropped the cluster name the canonical ID is derived from, so cluster members changed ID for the length of an outage. ### Change - The projection returns the source ID, the stored temperature details, the cluster name and the linked agent ID. `previousNodesForInstance` restores the config-derived identity stamps (provider-scoped flag, native aliases, TLS fingerprint) the same way `pollPVENode` writes them. Reviving those paths needed bounds: - **Memory carry** uses the poller's own validated `NodeMemorySnapshot`. The read state holds the agent's reading on a node merged with a host agent. used + cache + free still sum to the total. - **Agent lease:** a carried temperature stamped no later than a lapsed host agent's last report is not carried. Otherwise the 5-minute carry window would hold a silent 30s agent's reading in alerts and history about 2 minutes past the lease that #2582 enforces. Readings record no source, so an SSH reading of that age is dropped too; the bound errs toward a gap, never a stale value. - **No disk readings in a carry:** a carried reading keeps CPU/GPU only. SMART/NVMe rows also stamp physical disks, so a disk that went to standby would otherwise be stamped and recorded with its pre-sleep value. - **Preserved nodes:** outage preservation and authoritative membership reconciliation follow the same rules, and drop the temperature when the node goes offline. ### Tests - New registry-backed tests in `monitor_additional_test.go` cover: CPU low/record tracking across polls, outage writeback keeping the unified ID (including provider-scoped clusters), membership writeback, the lease rule, disk rows excluded from carries, and the memory carry on an agent-merged node. Each fails against the old projection or without its bound (checked with `go test -overlay`). - `TestPollPVEInstanceMarksStaleNodesOfflineWhenGetNodesReturnsEmpty` relied on the dead `nodeLastOnline` lookup; it now ages that sighting too. - The silent-agent test from #2582 now rebuilds the store via `PopulateFromSnapshot` and feeds the polled node back, as production does. Re-ingesting into one registry keeps the first `TemperatureDetails`, so the old setup never exercised the carry. Follow-ups found on the way: same-named nodes on different sites leaking agent temperatures (separate PR stacked on this one), and the node drawer never reading `temperatureDetails` (#2594). |
||
|
rcourtman
|
28d5f44257 |
Judge disk risk only on a temperature collected now
Since #2593, a disk whose host agent stops reporting keeps its last SMART temperature as a last-known value marked unavailable, and a disk in standby keeps its pre-sleep reading the same way. The shared disk risk assessment still fed that value into its temperature rule, so a silent agent's frozen 72C made the disk read "Replace Now: Disk temperature is 72C" with a warning status, while the temperature cell beside it now says the reading is not current. storagehealth.CollectedTemperature passes a temperature through only when its collection state is available, or when the source predates collection state. AssessPhysicalDisk, AssessHostSMARTDisk and the registry's recompute from merged metadata use it, so a retained reading stays on the resource as history but adds no temperature_high reason and leaves the disk online. Health, wearout and SMART counters are unchanged: they are durable evidence and keep judging as before. |
||
|
rcourtman
|
e25472feab | Merge remote-tracking branch 'origin/main' into fix/metric-alert-live-status | ||
|
rcourtman
|
8b61559f5a |
Move the metric status declarations clear of main's alert resolution
Main added Alert.resolution and its AlertResolution type at the end of the Alert struct in models.go and of the Alert interface in types/api.ts, exactly where this branch added metricStatus and its types, so the two edits conflict. Move this branch's field up beside metadata and its types below ResolvedAlert. Nothing else changes: the built frontend is byte-identical, and the Go types are only reordered. Contract-Neutral: declaration order only; no field, type or behaviour change |
||
|
rcourtman
|
95b29286dd |
Stop serving retained disk temperatures as current readings
Normalization may keep a disk temperature it did not collect (a disk in standby, a host agent past its reporting lease) as long as its collection state says so. Two consumers still presented that value as current. The metrics-history handler answered an empty disk range with the disk's temperature as a "live" point at now, regardless of its state. Since the history writers stopped recording retained temperatures, a short-range chart for a silent agent's disk or a sleeping disk showed the retained value as a fresh reading. Mock mode also seeded synthetic series from it and padded sparse stored series out to now. All of these now require a temperature collected now (diskinventory.TemperatureCollected, the shared form of the rule the history writers apply). Unraid array-inventory rows carry no per-field provenance, so a temperature taken from one looked collected after the agent went silent. The adapter now derives its state from the host lease that ExpireHostTelemetry already records, and from the disk's spin state: unavailable once the host is offline or the disk is spun down. A SMART row that borrows the inventory reading takes that state, including available. The native row records only the withdrawing states, because the registry merge picks temperature and state separately and an available claim from it would mask the SMART row's own state. A withdrawn inventory temperature no longer counts as heat in disk risk. |
||
|
rcourtman
|
474dbc4f09 |
Record Proxmox and agent alert history on the canonical resource
Alert lifecycle changes carry the alert's source-native resource ID, and MonitorAdapter.RecordChange only mapped full Docker container IDs. Proxmox node, guest and storage alerts and host-agent alerts were journaled under IDs such as "instance-node", "instance:node:vmid" or "agent:<host>", while resource facets, resource intelligence and Assistant resource context read history by canonical ID, so those surfaces never saw the alerts. Resolve such references to the canonical resource and record a history binding, so the reference keeps reaching the same history after a guest migration or inventory removal. Only durable identities bind: the canonical ID, a retired era, a source ID, a node-scoped guest reference or an agent host ID. Names and hostnames never bind, so a VM named "pulse-system" cannot capture system alerts, and a reference two resources answer to binds to neither. Docker keeps its full-ID rule. Rows already journaled under a source reference join canonical history without being rewritten: each registry generation binds the unbound alert references it can resolve, those found at startup and those written before inventory named the resource, retrying each for 30 minutes. |
||
|
rcourtman
|
9aaac4d3e5 |
Read a linked agent's own heartbeat in host snapshots (#2601)
A host agent linked to a Proxmox node merges into one unified host row. When the agent stops reporting but PVE polling continues, that row stays online and its `LastSeen` follows the PVE sighting. `hostFromReadStateView` copied both into `models.Host`, so a silent agent looked current to every consumer of the projection: - `linkedHostForNode` accepted it, and `resolveNodeDisk` kept using the agent's last filesystem summary as the node's disk, feeding disk alerts and history with a frozen reading. Its retained ZFS datasets kept merging into the node's pools. - Agent connections built from `HostsSnapshot()` (the Connections list and the update-readiness agent check) aged the agent from the PVE poll, so a dead agent stayed Active. The projection now takes `LastSeen` from the agent source's own sighting (`HostView.SourceStatus(SourceAgent)`, added in #2582) and reports the agent offline once that sighting is past its reporting lease (`hostAgentReportCurrent`, the same `hostAgentHealthWindow` that `evaluateHostAgents` applies). Rows without an agent sighting, such as vSphere and TrueNAS hosts, keep the row's values. Verification: - New `TestSilentLinkedAgentStopsFeedingNodeDisk` (registry-backed merged agent + node row, driven through `evaluateHostAgents` and the store refresh): node disk uses the agent summary while it reports, falls back to the Proxmox rootfs past the lease, and returns when the agent reports again; the projected agent reads offline with its own last report. It fails against the previous `monitor.go` (checked with `go test -overlay`). - `go test ./internal/monitoring/` and the connections, readiness, update and agent-lifecycle tests in `./internal/api/` pass. |
||
|
rcourtman
|
f28f73b28f |
Stop presenting a silent agent's disk readings as collected (#2593)
A host agent that stops reporting keeps its last SMART rows in state.
Its frozen disk readings then kept being treated as current:
- `mergeHostAgentSMARTIntoDisks` copied the silent agent's temperature
and I/O counters into the linked Proxmox physical disk, along with its
`available` collection state.
- The registry merged the agent's retained SMART rows into the canonical
disk the same way. `MergeStatus` lets any `available` state win, so the
API kept reporting the readings as collected.
- Every full disk poll (default every 5 minutes) wrote the frozen
temperature to SMART history as a new sample.
## Change
- **Lease expiry is the cut-off.** `State.ExpireHostTelemetry` (run by
the offline sweep once `hostAgentHealthWindow` passes) keeps SMART
temperature and I/O as last-known values but marks them `unavailable`
("host agent stopped reporting"). This follows the physical-disk
retained-evidence clause in the monitoring contract. The agent's next
accepted report replaces them.
- **A source can withdraw its own evidence.** New
`diskinventory.MergeReportedStatus` lets a source's later non-available
statement supersede an `available` state that another row still carries
from the same source.
- Used by the monitoring merge, where skipped disk polls re-merge the
previous state.
- Used in both registry ingest orders. This covers a PVE host that is
down entirely: it is never disk-polled again, so its Proxmox disk row
stays frozen.
- **History records only collected temperatures.** All SMART temperature
history writers (`writeSMARTMetrics`, `smartMetricStoreWrites`,
`writeHostSMARTMetrics`, and the unified disk sync via its collection
passthrough) skip a temperature whose current state is not collected.
This also stops a standby disk's retained pre-sleep reading from being
recorded.
- **Pre-6.2 agents keep their history.** These agents send no collection
provenance, so the merge now records their temperature as available from
`host_agent` instead of inheriting Proxmox's `unsupported`.
- **Expiry is compare-and-set** on the `LastSeen` the sweep judged
stale, so a report accepted in between is never expired. The sweep's
alert-side offline handling for that race is unchanged; that race
already existed.
Deliberately unchanged:
- SMART attributes, wearout and health stay last-known. They are durable
health evidence, and dropping them would resolve disk-health and
low-life alerts whenever an agent goes quiet (cf. #2112).
- Disk temperature alerts are evaluated in `CheckHost` from the agent's
own report, only when a report is ingested, so they never read this
merged value.
## Related
- The node-temperature path (agent sensors → `node.Temperature.SMART`,
which the disk merge labels `proxmox_node_smart`) is gated on the
agent's own sighting in #2582. A carried node reading can still stamp a
disk as collected; that path belongs to the node poller.
- Filed separately:
- the chart API's `liveMetricPoints` returns a retained disk temperature
as a single "live" point when a range has no samples;
- Unraid-native disk temperatures carry no collection state;
- the disk table renders non-collected temperatures with live threshold
colouring;
- a merged agent+node row reads `online` after its agent dies, so
`linkedHostForNode` keeps using the silent agent's filesystem summary.
## Tests
- `TestSilentLinkedAgentSMARTIsRetainedButNotPresentedAsCollected`
(registered monitoring proof `physical_disk_roundtrip_test.go`). Uses a
registry-backed monitor with a real disk poll and a metrics store. A
reporting agent's reading is collected and recorded. After lease expiry,
the canonical disk reads as not collected before any disk poll runs, and
again after a skipped poll and a full poll. The value is retained and no
new history sample is written. A resumed agent is collected and recorded
again. Fails on origin/main. Mutation-checked: removing the expiry
marking, the monitoring supersede, the registry supersede, or the
history gate each makes it fail.
- `TestMergeHostAgentSMARTIntoDisks_LegacyAgentTemperatureFollowsLease`
- `TestPhysicalDiskMergeFollowsAgentWithdrawalOfItsOwnReadings` (both
ingest orders)
- `TestMergeReportedStatusLetsTheSourceWithdrawItsOwnAvailability`
- `TestExpireHostTelemetryMarksSMARTReadingsNoLongerCollected`
- `TestExpireHostTelemetryLeavesHostThatReportedSinceJudgement`
- `TestWriteHostSMARTMetricsRecordsOnlyCollectedTemperatures`
Ran the full `internal/monitoring`, `internal/unifiedresources`,
`internal/models` and `pkg/diskinventory` packages locally; all pass.
|
||
|
rcourtman
|
0c130a4855 |
Key previous Proxmox nodes by source ID so poll carry-overs match
The poller reads the previous poll's nodes back from the unified read state, but previousNodeFromView returned the registry's unified resource ID and no temperature. Every lookup keyed on the Proxmox source ID missed: nodeLastOnline in the offline grace policy, the temperature carry, the CPU low/record tracking (reset to the current reading every poll) and the node memory carry. When a whole instance poll failed, markPVEInstanceNodesUnreachable wrote the projected nodes back, so each failed poll re-ingested the node under a new unified ID. Cluster members also lost the cluster identity their canonical ID derives from, so they changed ID for the length of an outage. The projection now returns the source ID, the stored temperature details, the cluster name and the linked agent ID, and previousNodesForInstance restores the config-derived identity stamps (provider-scoped flag, native aliases, TLS fingerprint) the way pollPVENode writes them. Reviving those paths needed four bounds: - The node memory carry uses the poller's own validated snapshot rather than the read state, which holds the agent's reading on a node merged with a host agent, and keeps used + cache + free equal to the total. - A carried temperature keeps a lapsed host agent's lease. A reading stamped no later than the agent's last report is not carried, so the five-minute carry window cannot hold a silent agent's reading in alerts and history past its lease, which #2582 guarantees. - A carried reading keeps the node's CPU and GPU temperatures only. Its SMART and NVMe rows also stamp the node's physical disks, so a disk that went to standby would be stamped and recorded with its pre-sleep value. - Outage preservation and authoritative membership reconciliation keep a node's temperature only under those rules and drop it when the node goes offline. TestPollPVEInstanceMarksStaleNodesOfflineWhenGetNodesReturnsEmpty relied on the dead nodeLastOnline lookup and now ages that sighting too. The silent-agent test from #2582 now rebuilds the store as production does and feeds the polled node back, which is what exercises the carry. |
||
|
rcourtman
|
d26a483731 |
Show a moved alert as moved in Patrol, history and assistant (#2591)
Follow-up to rcourtman/Pulse#2580. A node alert handed to its Pulse agent closes with an `AlertResolution` (`moved_to_agent`), and notifications already say it moved. Every in-app reader still treated that close as a recovery: - Patrol's attention detail showed a green **Resolved** badge, and its timeline read "Open to Resolved · Recovery evidence" directly above "…This is not a recovery…". - The Alerts history listed the row as **resolved**, and its incident timeline ended "Alert resolved: Node memory at 95.0%". - `GetRecentlyResolved` rebuilt `models.Alert` without the resolution. So the state snapshot, websocket, assistant context, Patrol's seed prompt and the `pulse_alerts` `resolved` action all saw a plain resolve. **Operational trust.** New `TransitionCause` `ownership_transferred`. It may only enter `resolved`, and only with evidence. `markOperationalResolved` uses it whenever the alert carries a `moved_to_agent` resolution; every other close stays on `recovery_evidence`. Alert-quality telemetry keys "resolved while snoozed" on `recovery_evidence`, so a handover during a snooze no longer counts there. No notification code reads the cause: links key on the transition ID and state. **State and AI.** `models.Alert.Resolution` holds the reason, successor and the engine's `Summary()`. `cloneAlert` copies it, and `GetRecentlyResolved` fills it. - Patrol's seed writes `closed 3m ago. Alert moved to pve1 (Host Agent). This is not a recovery: …`. - The assistant's context writes `closed … ago) <summary>`. - The resolved-alerts tool returns `resolution` and `successor_resource_id`. **Timelines.** The `alert_resolved` resource change carries the summary as its reason and the code as `alert_resolution` metadata. Incident memory titles the close with it, both when projecting from the canonical timeline and on its own write path. **UI.** - The history row reads **moved to agent**, even if it was acknowledged. The full account is in the badge title and is printed under the message on the phone card. - Patrol's detail shows a **Moved** badge (info tone), with the handover sentence directly under the summary rather than only inside Evidence and history. Its timeline says "Ownership transferred". **Landing order.** rcourtman/Pulse#2586 is the only producer of a handover, and it has no auto-merge. It should land after this PR (or merge main once this is in). Any handover recorded before this lands would otherwise keep a recovery-labelled transition and timeline. **Not covered here:** - Performance reports still list a handover from the last 5 minutes as resolved. - The resource drawer labels the change "Alert resolved" before its reason. - The incident header in the history expansion still shows its open/resolved status word. Both of the first two are filed as a follow-up and noted in ai-runtime.md and unified-resources.md. **Tests:** - `TestOwnershipTransferOnlyClosesWithEvidence`. - The cause assertions in `TestReleasedMetricAlertCarriesResolutionInsteadOfRecovery`. - `TestRecentlyResolvedCarriesHandoverResolution`. - `TestAlertResolutionWireContract`, which fails without the `cloneAlert` copy. - `TestMonitor_HandleAlertLifecycle_HandoverCloseIsNotARecovery`, which fails on the old code with "Alert resolved: Memory usage at 95%". - `TestIncidentStore_HandoverCloseIsNotARecovery`. - `TestSeedHealthAndAlerts_HandoverIsNotARecovery`. - `TestService_buildAlertContext_HandoverIsNotARecovery`. - `TestExecuteListResolvedAlertsReportsHandover`. - Frontend history, presentation and Patrol workbench tests. The operationaltrust, alerts, models, ai, ai/tools, ai/memory, unifiedresources, monitoring and notifications suites pass. `tsc`, eslint and the vitest sweep pass, apart from the Mac-only `agentInstallCommand` PTY test. **Browser pass.** A private non-mock stack ran on an alert history written by the real `releaseCanonicalMetricAlert` handover path. I checked Alerts history (row, tooltip, expanded timeline) and Patrol's deep-linked detail at 1280 and 375: no overflow, no console errors. |
||
|
rcourtman
|
40ebce161a |
Clear agent disk temperature alerts when the disk leaves (#2595)
Stacked on #2589 (the threshold-off fix). Until it merges this PR also
shows its two commits; the change here is the last commit,
|
||
|
rcourtman
|
4bd567b41a |
Clear agent disk temperature alerts when their threshold is off (#2589)
Turning off the agent disk temperature threshold left every open disk temperature alert active indefinitely. **Why nothing cleared them** - `CheckHost`'s branch for a disabled `DiskTemperature` threshold was an empty placeholder comment. - The host-wide clears (all agent alerts disabled, a disabled host override, agent removal, confirmed agent offline) matched only `agent:<host>/disk:` and missed `agent:<host>/disk_temp:`. - A config save skipped these alerts because `diskTemperature` was not classed as a threshold metric in `metric_runtime.go`. **Changes** - `host.go`: `CheckHost` clears a host's disk temperature alerts when the effective threshold is off, and `clearHostDiskAlerts` plus the confirmed-offline sweep include the `disk_temp` prefix. The per-type `DiskTempByType` refinement moves into `hostDiskTemperatureThresholdNoLock`, which `CheckHost` and `resolveHostAlertThresholdsNoLock` share. A config save therefore judges an alert against the threshold for its disk type, not the agent default. - `host.go`: an alert persisted before `CheckHost` recorded `diskType` (May 2026) has no disk type until its next firing evaluation. A save judges it against the lowest enabled trigger any of the host's disks can use, so a save cannot resolve an alert its disk type would still fire. - `metric_runtime.go`: `diskTemperature` is a threshold metric, so a save resolves these alerts once the threshold is off or the reading is below it. - `config_runtime.go`: when there is no recovery band, the save path now matches the evaluator. `buildCanonicalMetricSpec` drops a clear level at or above the trigger, and the reducer keeps firing at the trigger itself. Before this, a save resolved those readings and the next report raised them again. This applies to every metric alert that reaches the threshold comparison (docker-host alerts return before it); with a valid band, behaviour is unchanged. **Tests** (`internal/alerts/host_unraid_lifecycle_test.go`) - `TestCheckHostClearsDiskTemperatureAlertsWhenThresholdTurnsOff`: agent default off, host override off, host override disabled, all agents disabled. - `TestHostDiskTemperatureAlertsClearWhenAgentLeaves`: agent removed, agent confirmed offline. - `TestConfigSaveResolvesDiskTemperatureAlertAgainstItsDiskTypeThreshold`: an unrelated save keeps an alert above its nvme trigger; turning the threshold off resolves it. - `TestConfigSaveKeepsDiskTemperatureAlertWithoutDiskType`: an alert with no `diskType`, at 72C under a 75C agent default and a 70C nvme entry, survives an unrelated save. - `TestConfigSaveKeepsAlertTheEvaluatorKeepsWithoutRecoveryBand`: a 70/75 per-type entry at 72C and at 70C. Every case fails against origin/main's code (checked with a build overlay) and passes here. `go test ./internal/alerts/...` passes. |
||
|
rcourtman
|
e9069aed8c |
Carry a typed reason when an alert closes without recovering (#2580)
When a Pulse agent is linked to a Proxmox node, the node releases its CPU, memory and disk alerts to the agent, and any node alert still open closes. The close must reach integrations, so a PagerDuty or webhook incident opened from the node alert does not stay open forever. But it went out as a recovery: "pve1 on pve1 is now healthy" for a node still at 95% memory, backed by synthesized "recovery projection" evidence. #1682 is the same complaint from another trigger. This adds a typed `AlertResolution` (first reason: `moved_to_agent`, with the agent's alert resource ID and name) and carries it from the release path to every resolved consumer: - `releaseCanonicalMetricAlert` is the disabled-threshold release pulled out of `evaluateCanonicalMetricAlert`. It takes a resolution, and `clearAlertWithResolution` stamps a copy on the alert before it resolves. - The resolution rides the Alert JSON into the resolved callback, the delivery queue, the history row and the event log (reason code plus summary). - The synthesized closing evidence and the resolve transition say what happened instead of claiming a recovery projection. The cause stays `recovery_evidence`, the only cause operational trust lets enter `resolved`. - Notifications still send the close. Messages say "Memory alert moved to pve1 (Host Agent). This is not a recovery: check the agent for the current reading." in webhooks, `MOVED:` on a single ntfy push, and "Pulse alert moved" on email and Apprise. When any close in a delivery was not a recovery, `WebhookPayloadData.NotRecovered` drops recovery visuals: Discord's green embed turns neutral blue, both Teams cards lose recovery green, Mattermost's check emoji becomes an arrow, and ntfy uses the arrow tag for a single alert or a batch. Titles still say Resolved. PagerDuty still gets `resolve`. Every caller passes a nil resolution today, so ordinary recoveries and the disabled-threshold path are unchanged. The node release adopts the reason in a follow-up, once the host-agent node link records which metrics the agent evaluates (#2577 lane). Tests: `TestReleasedMetricAlertCarriesResolutionInsteadOfRecovery` and `TestResolvedHistoryRowKeepsResolutionWithoutEventLog` (operational_contract_test.go); `resolved_handover_test.go` drives the firing, its receipt and the queued close through the normal ntfy entry points, and renders Discord, both Teams cards, Mattermost, PagerDuty, a grouped Slack list, a grouped ntfy batch and email/Apprise content, with isolated managers. The full alerts, notifications and monitoring suites pass locally. |
||
|
rcourtman
|
50b3fd8226 | Merge remote-tracking branch 'origin/main' into fix/metric-alert-live-status | ||
|
rcourtman
|
3ca8b9647a |
Move the live metric status projection test clear of main
Main appended TestSilentLinkedAgentStopsRefreshingNodeTemperatureAlert at the end of monitor_alert_handling_test.go, where this branch had appended its own projection test, so the two edits conflict. Move the test unchanged in substance to monitor_alert_intent_test.go, which main has not touched, so the merge with main is clean. |
||
|
rcourtman
|
66c8c63e79 |
Hand a node only the usage metrics its agent evaluates (#2581)
This stacks on #2577. Once #2577 merges, the diff here reduces to the last commit. ## Problem `CheckHost` registered a host agent's node link before its disabled returns. With agent alerts switched off, or a host override disabling the agent, the node still released its CPU, memory and disk alerts to an agent that evaluated nothing, so nothing alerted on that machine's usage. The same happened per metric: an agent CPU threshold set to off with the node CPU threshold on left the machine with no CPU alert. The gap predates #2569; the old hostname dedup registered in the same place. ## Fix - The link now records which usage metrics the agent actually evaluated on its last report: a live CPU threshold, a live memory threshold with a known reading, and at least one filesystem with a live disk threshold. - `CheckHost` registers the link after evaluation and removes it on the disabled returns. - `CheckNode` releases only the metrics the agent owns and evaluates the rest itself. Deduplication can only remove a duplicate, never the last alert for a machine. - The link also carries the agent's display name and a stable lookup (`linkedHostAgentForNode`, lowest agent ID wins). The handover notification work builds on these. ## Tests `TestCheckNodeKeepsUsageMetricsTheAgentDoesNotEvaluate` covers two cases: agent alerts disabled (the node alerts on CPU and memory), and the agent CPU threshold off (the node keeps CPU while the agent keeps memory, so the machine still has exactly one memory alert). Both cases fail with the old registration order. The dedup tests use the new link struct. `go test ./internal/alerts/...` passes, and so do the monitoring alert, temperature, host-agent and node tests. ## Review follow-up (second commit) A review of the first commit found three more ways the node and its linked agent could disagree about who alerts: - **Memory ownership flipped with each report.** Ownership needed a known memory reading on every report. When the reading went missing while the agent's alert was open, the agent kept its alert but stopped owning memory, so the node raised a second alert. That alert resolved as soon as the reading came back. Ownership now follows the agent's configured thresholds, and a missing reading is treated as missing. - **Only one linked agent counted.** With several agents linked to one node, only the lowest agent ID's coverage was used. A metric is now owned when any linked agent evaluates it. - **Disk ownership covered the wrong filesystem.** A linked node's disk metric is the agent's summary filesystem: root when reported (`models.SummaryDisk`). Any evaluated mount was enough to take ownership, so with root disabled by a per-disk override and `/data` still alerting, a full root filesystem raised nothing. The agent now owns the node's disk metric only while it evaluates that same filesystem. `TestCheckNodeUsageOwnershipFollowsWhatAgentsEvaluate` covers all three cases; each fails without its fix. |
||
|
rcourtman
|
f4c3040373 |
Read a linked agent's own heartbeat in host snapshots
A host agent linked to a Proxmox node merges into one unified host row. When the agent stops reporting but PVE polling continues, that row stays online and its LastSeen follows the PVE sighting. hostFromReadStateView copied both into models.Host, so a silent agent looked current to every consumer of the projection: - linkedHostForNode accepted it, and resolveNodeDisk kept using the agent's last filesystem summary as the node's disk, feeding disk alerts and history with a frozen reading. Its retained ZFS datasets kept merging into the node's pools too. - Agent connections built from HostsSnapshot (the Connections list and the update-readiness agent check) aged the agent from the PVE poll, so a dead agent stayed Active. The projection now takes LastSeen from the agent source's own sighting and reports the agent offline once that sighting is past its reporting lease, the same hostAgentHealthWindow evaluateHostAgents applies. Rows without an agent sighting, such as vSphere and TrueNAS hosts, keep the row's values. The node falls back to the Proxmox rootfs reading while the agent is silent and returns to the agent summary when it reports. |
||
|
rcourtman
|
f419149458 |
Evaluate host agents in mock mode like live agents (#2585)
## Why `checkMockAlerts` evaluated mock nodes, guests, storage, PBS, PMG and Docker hosts but never host agents. Live agents are evaluated by `CheckHost` as each report lands; mock mode discards reports and the host health sweep never sees fixture agents, so no agent CPU, memory, disk, temperature or offline alert could open against mock data. A mock agent at 94% memory sat under the 85% default trigger with no alert, and any alert work on agents needed a private overlay build to reproduce. ## What Mock passes evaluate every fixture agent (`evaluateMockHostAgents`) with the live lifecycle boundaries: - Offline fixtures go through `HandleHostOfflineWithCorrelation`, because `CheckHost` treats its input as a fresh report and marks the host online. - Agents run before nodes, so a node with an online linked agent hands its CPU/memory/disk alerts to the agent on the first tick (host-agent hostname deduplication). - An agent that leaves the estate on a runtime mock config change goes through `HandleHostRemoved`, like a deleted live agent. - Leaving mock mode routes the fixture agents through `HandleHostRemoved`, releasing hostname registrations that `ClearActiveAlerts` leaves behind (they would otherwise suppress a real node named `pve1`). A pass already in flight finds mock mode off under the same lock and evaluates nothing. Evaluating agents exposed a mock-only reading: each Kubernetes pod took a whole single-pod footprint (10-83%) of its node's allocatable memory, so any node running four or more pods, and its linked agent, read a constant 100%. On the public demo's estate that became two permanent critical agent memory alerts. Pods now share 0.7 footprints per node (at most 58% in steady state), and a 60% per-node ceiling clips the transient while rescheduled pods' readings settle, so k8s nodes stay under 82%. Contract notes: `monitoring.md`, `performance-and-scalability.md`, `agent-lifecycle.md` (the `monitor.go` field touches only the alert-side removal boundary, with no tombstone, token or continuity change). ## Demo impact Measured on private stacks with the demo host's exact `PULSE_MOCK_*` estate (8 nodes, 2 standalone agents, 3 Kubernetes node agents): | | before | after | |---|---|---| | Kubernetes node agents' memory | 100%, 100% | 19-50% (up to ~76% briefly after boot) | | Agent alerts at default thresholds | none, ever | per-boot readings decide: one boot opened none, another opened Apollo-114 memory 91.3% (warning) | ## Verification - New tests, each shown to fail on the code it guards (via `go build -overlay`): `TestCheckMockAlertsEvaluatesHostAgentsBeforeLinkedNodes` (also fails with the agent loop after the node loop), `TestCheckMockHostAlertsUsesHostLifecycleForOfflineFixtures`, `TestLeavingMockModeReleasesFixtureAgentHostnames`, `TestMockHostAgentPassAfterLeavingMockModeRegistersNothing`, `TestMockHostAgentLeavingFixtureUsesRemovalLifecycle`, `TestMockKubernetesDenseNodesStayBelowAllocatableMemory`, `TestMockKubernetesRescheduledPodsDoNotSaturateReceivingNode`, `TestMockKubernetesLoneHeavyPodStaysUnderNodeCeiling`. - `go test ./internal/mock/ ./internal/monitoring/` pass. - Live, private demo-config stack: with the agent memory trigger at 60%, exactly the agents above it opened `agent:` memory alerts and their linked Proxmox nodes raised none; the Alerts page renders them. Shrinking the estate from 8 to 4 nodes through `/api/system/mock-mode` cleared the departed agents' alerts on the next pass. - Three rounds of independent review; the first two found the stale-registration, removal, relocation and steady-state ceiling issues fixed here, the third found none. |
||
|
rcourtman
|
ee3ebd800e |
Stop presenting a silent agent's node temperature as live (#2582)
A host agent linked to a Proxmox node merges with it into one read-state row, and every PVE poll refreshes that row's `LastSeen`. The agent's last sensors stay in state after it stops reporting. The node temperature lookup (`getHostAgentTemperatureByID`) checked neither, so each poll merged the silent agent's last reading into the node as a fresh measurement, stamped with the row's fresh `LastSeen`: - an open node temperature alert was re-observed every poll, so its `LastSeen` kept moving and the 24h stale-alert cleanup never reached it; a stale hot reading could keep activating - node history recorded the frozen value as new samples, and the UI and reports showed it as current - SSH collection was skipped as if a recent agent payload existed Since #2569 evaluates node temperature for agent-backed nodes, this holds their temperature alerts open. ## Change - `HostView.SourceStatus(source)` exposes the per-source freshness the registry already records (same accessor the VM, container and physical-disk views have). - The linked-agent lookup reads the agent source's own sighting, uses the agent's sensors only inside the reporting lease that keeps the agent online (`hostAgentHealthWindow`, as `evaluateHostAgents` uses), and stamps the reading with the agent's report time. Past the lease the node has no agent reading (the cluster sensor cache is still consulted), and alert evaluation treats that as a missing reading, not as evidence the node cooled down. - The poller's carry-over of a previous reading after a failed collection is bounded to max(5 min, 2 × PVE polling interval) and keeps its original timestamp. This branch does not engage today: the previous-node projection carries the unified resource id and no temperature, so it never matches the polled node. That is filed separately; the bound keeps the carry honest once it's repaired. The owner is the poller rather than `checkNodeTemperature`: the same stale value also reaches history, reports and the UI. An alerts-side `LastUpdate` check would also have missed this case, because the reading was stamped with the merged row's fresh `LastSeen`. Mock fixtures are untouched. ## Tests - `TestSilentLinkedAgentStopsRefreshingNodeTemperatureAlert` (registered monitoring proof): registry-backed merged agent+node row through the production previous-node projection and `CheckNode`. A reporting agent opens the alert. A silent agent leaves the alert open but stops refreshing `LastSeen`. A resumed agent is evaluated again. It fails on origin/main. - `TestView_HostViewSourceStatusSeparatesAgentFromMergedRow`: a merged row's `LastSeen` follows PVE, while `SourceStatus(SourceAgent)` keeps the agent's own report time. - `TestCollectNodeTemperatureCarryIsBoundedByCarryWindow`, `TestHostAgentReportCurrent`, `TestGetHostAgentTemperatureByID_IgnoresAgentPastReportingLease`. Out of scope, filed separately: `mergeHostAgentSMARTIntoDisks` copies silent agents' retained SMART temperatures into physical disks. |
||
|
rcourtman
|
c4bef05354 |
Give agent disk, array and sensor alerts their own resource type (#2583)
## Why Alerts a host agent raises about something other than its own machine (filesystems, disk temperatures, SMART disks, RAID and Unraid arrays, custom sensors) carried the host's `resourceType: agent`. The alert card's Monitoring menu writes policy to the alert's child resource id (for example `agent:<host>/disk:sda`), yet it named the Pulse agent as the inventory owner, which is the copy meant for the agent's own machine. Follow-up to #2574, whose card logic gives agent-platform children the neutral copy once their producers say what they are. ## What - `internal/alerts/host.go`: every host alert stamps `platformType: agent`. Child alerts carry `agent-disk` (filesystem usage, disk temperature, SMART health and wearout), `agent-storage` (RAID, Unraid array) or `agent-sensor` (custom sensors). The machine's own CPU, memory and offline alerts keep `agent`. - `internal/alerts/config/identity.go`: `agent-storage` and `agent-sensor` keep `agent` among their policy keys, as `agent-disk` already did, so agent thresholds and switches still govern them. - `internal/alerts/config_runtime.go`: configuration re-evaluation stops at the agent branch for host-agent alerts. Before, a SMART, RAID or sensor alert fell through to the guest switch, so disabling guest alerts resolved it on save and the next agent report raised and notified it again. - `internal/alerts/canonical_metric.go`: a metric alert held between its clear and trigger thresholds now refreshes `resourceType` and `platformType` from its producer, so an alert restored from a checkpoint written before these keys existed stops linking to the wrong page. Value, message and last-seen time still stay at the last breach. Webhook payloads report the new `resource_type` values for these alerts. ## Verification - `TestHostChildAlertsNameTheirOwnResourceType`, `TestHostChildAlertsIgnoreOtherPlatformSwitchesOnConfigSave` and `TestAlertCharacterizationHeldMetricAlertRefreshesItsClassification` fail on the previous code and pass now. `TestHostChildAlertsSurviveProxmoxNodeCleanup` guards the Proxmox node sweep, which keeps these alerts by their `agent:` resource id. `go test ./internal/alerts/...` passes. - Private mock stack, with before and after binaries on one data directory. A local-only overlay gave mock agents SMART, RAID, Unraid and custom-sensor readings and ran `CheckHost`. Before: every child alert's Monitoring menu read "Pulse agent remains the inventory owner". After: the host's memory alert still does, while the filesystem, SMART, RAID, sensor and Unraid alerts read "Source system remains the inventory owner", and all still link to `/standalone/machines`. Muting from the SMART card wrote operator state for `agent:host-linux-1/disk:sda` and left `agent:host-linux-1` unconfigured. - Restore path: a filesystem alert fired at 60%, held at 40%, and was saved by the old binary as `agent` with no platform. The new binary restored it, and after one evaluation it read `agent-disk` / `agent`, with its value, `lastSeen` and `diskUsagePercent` unchanged. |
||
|
rcourtman
|
7c8e9099a1 |
Clear agent disk temperature alerts when the disk leaves
A host-agent SMART disk temperature alert stayed active for a day once its disk left the agent's report (removed, replaced, or a renamed device path). CheckHost only evaluated disks still listed and node cleanup exempts agent: alerts, so only the 24-hour stale sweep resolved the alert or dropped a pending threshold run. CheckHost now tracks which disks a non-empty SMART list reports. Once a disk with an alert or a pending threshold run has been missing from three consecutive non-empty reports, the alert clears and the pending run is dropped. The Windows, FreeBSD and controller-multiplexed Linux collectors drop a disk whose probe fails, so a single omission does not count as the disk leaving, and the count restarts when the disk is reported again. An empty SMART list holds every alert, because the agent omits the list when collection fails or is unsupported. A listed disk in standby or without a temperature also holds its alert. Neither shows the disk cooled, and SMART health alerts already hold through standby. Turning the threshold off or removing the agent now drops pending runs and absence counts as well as alerts. Cleanup matches the disk_temp resource prefix alone, which also catches alerts raised under the original disk_temperature type. |
||
|
rcourtman
|
aa252362a0 |
Prove a standby disk's retained temperature stays out of history
A disk that goes into standby keeps its pre-sleep temperature through preserveUnavailablePhysicalDiskEvidence, and before the collection-state gate every full disk poll wrote that value to SMART history as a new sample. The silent-agent test covers lease expiry only; this pins the standby path the same gate fixes, starting from a disk that already has a recorded temperature. |
||
|
rcourtman
|
6413cda171 |
Judge legacy disk temperature alerts by the lowest trigger on save
A disk temperature alert persisted before CheckHost recorded its disk type has no diskType metadata until its next firing evaluation. A config save in that window judged it against the agent threshold. With a 75C agent default and a 70C nvme entry, a 72C alert was resolved on save and raised again by the next report. Without a known disk type, a save now judges the alert against the lowest enabled trigger the host's disks can use: the agent threshold or any DiskTempByType entry. A save therefore cannot resolve an alert its disk type would still fire, and turning the threshold off still resolves it. |
||
|
rcourtman
|
549161bf6f |
Stop presenting a silent agent's disk readings as collected
A host agent that stops reporting keeps its last SMART rows in state.
The Proxmox physical-disk merge copied that agent's temperature and I/O
counters into the disk together with its "available" collection state,
the registry merged the agent's retained rows into the canonical disk the
same way, and every full disk poll wrote the frozen temperature to SMART
history as a new sample. A dead agent's disk temperatures therefore stayed
"collected" in the API and kept extending the history chart indefinitely.
Expiring the agent's reporting lease is now the point where those
readings stop being current. ExpireHostTelemetry keeps the SMART
temperature and I/O values as last-known evidence but marks them
unavailable ("host agent stopped reporting"), as the physical-disk
contract asks for retained evidence. Because MergeStatus lets any
available state win, a Proxmox disk row still holding the earlier copy
would mask that, both on skipped disk polls and when the PVE host is down
and never disk-polled again. diskinventory.MergeReportedStatus lets a
source's later word on its own field supersede the availability it
supplied earlier; the monitoring merge and both registry ingest orders
use it. Every SMART temperature history writer now records a
temperature only when its current state is collected, so a retained
reading (an agent past its lease, or a disk in standby) is not written as
a new sample.
Agents before 6.2 send no collection provenance, so their merged
temperature is recorded as collected from "host_agent" instead of
inheriting Proxmox's "unsupported", keeping their history intact.
Expiry is now compare-and-set on the report time the evaluation judged
stale, so a report accepted in between is not expired.
SMART attributes, wearout and health stay last-known on purpose: they
are durable health evidence, and dropping them would resolve disk-health
and low-life alerts when an agent goes quiet. Disk temperature alerts
are evaluated from the agent's own report, so they never read this
merged value. The node-temperature path (agent sensors feeding
node.Temperature.SMART, which the disk merge labels proxmox_node_smart)
is gated by the agent's sighting in #2582; a carried node reading can
still stamp a disk as collected and belongs to that path.
|
||
|
rcourtman
|
91575a285e |
Merge remote-tracking branch 'origin/fix/alert-handover-resolution' into fix/moved-alert-in-app
# Conflicts: # docs/release-control/v6/internal/subsystems/notifications.md |
||
|
rcourtman
|
47e6927819 | Merge remote-tracking branch 'origin/main' into fix/alert-handover-resolution | ||
|
rcourtman
|
a89c08a59e |
Drop recovery colours and check marks from moved alert closes
A second review found the moved-alert text was honest but the visuals still said recovered: Discord kept its green embed, both Teams cards kept recovery green, Mattermost led with a check emoji, and a grouped ntfy batch holding a moved alert kept the green check tag. A check mark on a phone is read before the sentence under it. Resolved webhook data now carries NotRecovered when any close in the delivery was not a recovery. The built-in templates switch to neutral colours and an arrow when it is set, and ntfy uses the arrow tag for a single alert or a batch. Titles still say Resolved, which stays true for the closed alert, and ordinary recoveries keep their green. The render tests now use an isolated data directory, stop their managers, and hand the captured request over a channel, so they no longer touch a shared queue or race on the captured body. |
||
|
rcourtman
|
23f603da9f |
Show a moved alert as moved in Patrol, history and assistant
A node alert handed to its Pulse agent closes with an AlertResolution (#2580), and notifications already say it moved. Every in-app reader still treated the close as a recovery: Patrol's detail wore a green Resolved badge over a timeline labelled "Recovery evidence", the alert history listed the row as resolved and its incident timeline said "Alert resolved: Node memory at 95%", and the assistant, Patrol's seed prompt and the pulse_alerts tool saw a plain resolve because GetRecentlyResolved dropped the resolution. Operational trust gains a non-recovery cause, ownership_transferred, which may only enter resolved and only with evidence. The resolve transition uses it whenever the alert carries a moved_to_agent resolution, so a handover during a snooze also stops counting as resolved while snoozed. models.Alert gains Resolution (reason, successor and the engine's summary), copied by cloneAlert and filled by GetRecentlyResolved, so the state snapshot, websocket, assistant context, Patrol seed and the resolved-alerts tool all report where the alert went. The resolved resource change carries the summary as its reason with alert_resolution metadata, and incident memory titles the close with it. In the UI, the history row reads "moved to agent" with the account in its title and on the phone card, and Patrol's detail shows a Moved badge with the handover sentence under the summary. |
||
|
rcourtman
|
982a6f6148 | Merge remote-tracking branch 'origin/main' into fix/moved-alert-in-app | ||
|
rcourtman
|
36b2e5fa0a |
Clear agent disk temperature alerts when their threshold is off
Turning off the agent disk temperature threshold left every open disk temperature alert active indefinitely. CheckHost's disabled branch was an empty placeholder, the host-wide clear paths (all agent alerts disabled, a disabled host override, agent removal, confirmed offline) matched only the /disk: prefix and missed /disk_temp:, and a config save skipped these alerts because diskTemperature was not classed as a threshold metric. CheckHost now clears a host's disk temperature alerts when the effective threshold is off, and the host-wide clears include them. A config save resolves them against the same threshold CheckHost evaluates, including the per-type DiskTempByType refinement. That way an unrelated save does not resolve an alert above its disk type's trigger only for the next report to raise it again. The save path now also matches the evaluator when there is no recovery band. buildCanonicalMetricSpec drops a clear level at or above the trigger, and the reducer keeps firing at the trigger itself, but reevaluateActiveAlertsLocked resolved any reading at or below that clear level. A per-type entry such as 70/75 (normalization keeps it) resolved a 72C alert on save, and the next report raised it again. With no recovery band, a reading exactly at the trigger was resolved the same way. |
||
|
rcourtman
|
27051deb27 |
Dedup node alerts by agent link and keep node thresholds on save (#2577)
A review of #2569 (node temperature alerts with a host agent) found three more ways node alerts went wrong. Each was reproduced with a test that fails on `main`. ## Changes - **Dedup follows the agent's node link, not its hostname.** Node deduplication matched an agent's hostname against node names across every PVE instance. With two instances that each have a node called `pve` and an agent on only one, both nodes released their CPU, memory and disk alerts, so the second machine went unmonitored. An agent reporting an FQDN never matched its node, so that machine alerted twice. An operator unlinking an agent didn't hand the alerts back either. The manager now tracks each reporting agent's `LinkedNodeID`, which the monitor already sets automatically or from an operator link. On pulse-prod the delly2 agent is linked to `homelab-delly2`, the same ID as the node alert it should dedup. - **A config save keeps node thresholds.** Saving the alert config resolved node temperature alerts that were still over their trigger. Reevaluation only treated an alert as a node alert when its `Instance` equalled its node name. Current node alerts keep the PVE instance name there (`homelab`), so they fell through to guest thresholds, and guests have no temperature threshold. Metadata `resourceType: node` now decides, and the `Instance` heuristic only covers legacy alerts that have no resource type. - **Missing readings interrupt timing runs.** Since #2569, a missing temperature reading no longer resolves an open alert, but a pending run still spanned the gap. One hot sample after ten minutes without readings fired at once, and a recovery run could complete the same way. The reducer gains `InterruptMetricRun`, which drops a pending run or restarts a recovery run without resolving anything. The node temperature check calls it, along with the explicit-intent pending clear, when the reading is missing. ## Not in this PR Two other review findings are filed separately: - When collection fails, the poller re-presents a stale temperature as current with no time limit, and alerting treats it as live. The fix touches the poller, mock timestamps and what the UI shows. - At handover, the released node alert closes with "now healthy" wording. That needs a typed resolution reason in notifications. ## Tests - `TestHostAgentDeduplicationFollowsNodeLink`: same-named nodes in two instances, an FQDN hostname, an unlinked agent, and an operator unlink. - `TestConfigSaveKeepsNodeTemperatureAlertOverTrigger`: fails on `main`. - `TestCheckNodeMissingTemperatureInterruptsTimingRuns`: fails on `main` (one sample back after the gap fires at once). - The existing dedup tests now use link semantics. `go test ./internal/alerts/...` passes, and so do the monitoring alert, temperature, host-agent and node tests. |
||
|
rcourtman
|
4775fcbf38 |
Base node usage ownership on agent config, all links, summary disk
A review of the previous commit found three ways the node and its linked agent could still disagree about who alerts. Memory ownership required a known reading on every report. When an agent's memory reading went missing while its alert was open, the agent kept that alert (a missing reading is not a recovery) but stopped owning memory, so the node raised a second alert. It resolved again as soon as the reading returned, with history and notifications for each flip. Ownership now follows the agent's configured thresholds, not one report's data. With several agents linked to one node, only the lowest agent ID's coverage counted. If that agent had CPU off and another linked agent evaluated CPU, the node alerted on CPU as well. A metric is now owned when any linked agent evaluates it. Disk ownership needed any evaluated filesystem, but a linked node's disk metric is the agent's summary filesystem (root when reported, see models.SummaryDisk). With root disabled by a per-disk override and /data still alerting, the node gave up its root disk alert and a full root filesystem raised nothing. The agent now owns the node's disk metric only while it evaluates that same filesystem. |