gsadmin/pulse
1
0
Fork 0
mirror of https://github.com/rcourtman/Pulse.git synced 2026-10-24 04:33:17 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge remote-tracking branch 'origin/main' into fix/metric-alert-live-status

Browse Source
This commit is contained in:
rcourtman
2026-10-06 20:23:38 +01:00
parent 8b61559f5a 9aaac4d3e5
commit e25472feab
78 changed files with 3519 additions and 120 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/release-control/v6/internal/subsystems/agent-lifecycle.md
+22
View File
@@ -736,6 +736,12 @@ into monitoring's models. Absent evidence has to carry its declared sentinel
rather than a zero value that reads as a real measurement: an absent
physical-disk view projects `Wearout` as `unifiedresources.WearoutUnreported`,
never `0`, which would announce a spent disk the agent never reported.
When the host heartbeat lease expires, `State.ExpireHostTelemetry` keeps that
agent's SMART temperature and I/O counters as last-known values but marks them
`unavailable` ("host agent stopped reporting"); the agent's next accepted
report replaces them. Expiry is compare-and-set on the report time the offline
sweep judged stale, so a report admitted between that judgement and the expiry
is never expired.
An enabled availability target assigned to a host agent creates an
agent-lifecycle lease for that exact target/agent pairing. First assignment
@@ -1270,6 +1276,10 @@ update, profile rollout, command reachability, or fleet-control authority.
lifecycle presentation evidence. It keeps mock active-alert and history
payloads aligned with the canonical alert contract, but it cannot enroll,
identify, link, command, remove, or otherwise grant authority to an agent.
The optional `models.Alert.Resolution` field is the same kind of
evidence: it says a resolved node alert now belongs to a linked agent's
own alert and names that agent as display text, but the link itself stays
the agent lifecycle's, and the field cannot create, change or authorize it.
27. `internal/monitoring/monitor.go` shared with `monitoring`: monitor construction owns both monitoring runtime initialization and fail-closed agent lifecycle journal hydration before report admission.
28. `internal/monitoring/monitor_agents.go` shared with `monitoring`: server-side Unified Agent report, removal, token binding, tombstone expiry, and re-enrollment semantics are jointly owned by agent lifecycle authority and monitoring ingest.
29. `pkg/agents/host/report.go` shared with `monitoring`: the Unified Agent host report is both an agent lifecycle authored-state contract and a monitoring ingest contract for host maintenance posture.
@@ -8805,3 +8815,15 @@ no update status, instead of reporting "no update available" (#2353). Agent
registration, enrolment, install, update, removal and report identity are
unchanged; the per-image memo lives in the registry checker and is pruned
each collection cycle to the images in use.
### Host snapshots report a linked agent's own heartbeat
`internal/monitoring/monitor.go` changed only so a host produced from the read
state carries the agent source's own last report and reads offline past the
agent's reporting lease, even when the agent's row is merged with a Proxmox
node that PVE polling keeps fresh. The agent connection on the Connections
list and the update-readiness agent-continuity check therefore age a silent
agent from its last report instead of from the PVE poll, and mark it stale at
their own heartbeat cutoff (`fleethealth.AgentStaleThreshold`, which is
separate from the monitoring reporting lease). Agent registration, enrolment,
install, update, removal and report identity are unchanged.
docs/release-control/v6/internal/subsystems/ai-runtime.md
+16 -1
View File
@@ -4638,7 +4638,22 @@ summaries became the alarm name still carry "<entity> <moref> has VMware alarm
<name> (<colour>)", so the title strips that wording to the same alarm name.
Patrol's seed context lists active and recently resolved alerts by resource:
each line names the alert's resource unless the message already does, because
provider incidents such as vCenter alarms say only what was flagged. The projection also collapses open/resolved churn:
provider incidents such as vCenter alarms say only what was flagged.
A recently resolved alert that closed without recovering, such as a node metric
handed to its Pulse agent, must not read as a recovery to Patrol, the
assistant or the alert tool: Patrol's seed writes `closed <ago>. <summary>` in place of `resolved <ago>`, the
assistant's alert context writes `closed <ago> ago) <summary>`, the
`pulse_alerts` `resolved` action returns the summary as `resolution` with
`successor_resource_id`, and incident memory titles the close with the summary
(`alertResolvedEventSummary`, or the `alert_resolution` resource-change
metadata when it projects from the canonical timeline).
`TestSeedHealthAndAlerts_HandoverIsNotARecovery`,
`TestService_buildAlertContext_HandoverIsNotARecovery`,
`TestExecuteListResolvedAlertsReportsHandover` and
`TestIncidentStore_HandoverCloseIsNotARecovery` pin it. Performance reports
(`pkg/reporting`, the report narrator) do not carry the resolution yet: a
report generated within the five-minute recently-resolved window after a
handover still lists that alert as resolved. The projection also collapses open/resolved churn:
when a record's timeline holds at least four open-to-resolved or
resolved-to-open transitions inside the last 24 hours, the item carries a
`flapping` summary (transition count, window, first and latest transition) and
docs/release-control/v6/internal/subsystems/alerts.md
+104 -1
View File
@@ -797,6 +797,39 @@ must migrate through `internal/alerts/operational_contract.go` and name their
limited provenance honestly rather than inventing confirmed provider evidence.
Acknowledgement remains distinct from resolution, and every resolution
transition references recovery evidence separate from its trigger evidence.
A resolved alert may carry a typed `AlertResolution`
(`internal/alerts/resolution.go`) when it closed without its condition being
observed to clear. The one reason so far, `moved_to_agent`, describes a metric
alert whose resource's linked Pulse agent now owns the metric.
`releaseCanonicalMetricAlert` is the disabled-threshold release extracted from
`evaluateCanonicalMetricAlert`: a caller releasing a metric for any other
reason passes a resolution, and `clearAlertWithResolution` stamps a copy on the
removed alert before it is resolved. The resolved callback snapshot, the
history row, the resolved event-log entry (the reason code, with `Summary()` as
its message) and the Alert JSON then carry it. Resolved consumers still receive
the close, because integrations close their incident by alert ID. The
synthesized closing evidence carries the resolution reason and summary instead
of `legacy_alert_recovery_projection`, and the resolve transition carries the
summary as its `reason`. Its cause is `ownership_transferred`, an
`operationaltrust.TransitionCause` that may only enter `resolved` and only with
evidence (`resolvedTransitionCause` in `operational_contract.go`), so Patrol's
timeline never labels a handover "Recovery evidence" and alert-quality
telemetry does not count a handover during a snooze as resolved while snoozed.
A nil resolution leaves the disabled-threshold path and every recovery on
`recovery_evidence`.
`TestReleasedMetricAlertCarriesResolutionInsteadOfRecovery` and
`TestResolvedHistoryRowKeepsResolutionWithoutEventLog` in
`internal/alerts/operational_contract_test.go` pin this.
The in-app readers carry it too. `GetRecentlyResolved` projects it into
`models.Alert.Resolution` (reason, successor and the `Summary()` text) for the
state snapshot, the websocket and the assistant and Patrol prompts, pinned by
`TestRecentlyResolvedCarriesHandoverResolution` in `history_test.go`. The
alerts history lists such a row as `moved to agent` instead of `resolved`, even
when it was acknowledged, with the account in the badge title and on the phone
card (`buildAlertHistoryItems`, `getAlertResolutionDetail`). The resolved
lifecycle change written to the resource timeline carries the summary as its
reason and the code as `alert_resolution` metadata, so the row's incident
timeline says the alert moved instead of "Alert resolved: <breach message>".
## Extension Points
@@ -1609,6 +1642,75 @@ per-alert rate limiting; future notification-gating changes should extend that
policy owner rather than burying new checks inside metric or resource-specific
evaluators.
### Agent disk temperature alerts clear when their threshold is off
Host-agent SMART disk temperature alerts (`diskTemperature`, resource
`agent:<host>/disk_temp:<device>`) follow one effective threshold: the
resolved agent `DiskTemperature`, refined by the disk type's `DiskTempByType`
entry unless an explicit host or linked-resource override set it
(`hostDiskTemperatureThresholdNoLock` in `internal/alerts/host.go`). When the
host's resolved `DiskTemperature` is off, `CheckHost` clears every disk
temperature alert on the host. A disabled `DiskTempByType` entry clears only
the alerts of disks of that type, through the disabled metric spec. The
all-agents switch, a disabled host override, agent removal and confirmed
agent offline clear them with the host's other disk alerts. Before this,
nothing cleared them once the agent-level threshold was off.
A config save judges an open disk temperature alert against the same
threshold (`resolveHostAlertThresholdsNoLock` refines it by the alert's
`diskType` metadata, and `metric_runtime.go` classifies `diskTemperature` as a
threshold metric), so turning the host's `DiskTemperature` off resolves the
alert immediately. An alert persisted before `CheckHost` recorded `diskType` (May
2026) carries no disk type until its next firing evaluation, so a save judges
it against the lowest enabled trigger any of the host's disks can use: the
agent threshold or a `DiskTempByType` entry
(`lowestHostDiskTemperatureThresholdNoLock`). A save therefore cannot resolve
an alert its disk type would still fire.
For a metric alert that reaches the threshold comparison in
`reevaluateActiveAlertsLocked` (docker-host alerts return before it), a config
save resolves an alert whose reading is
below the new trigger. That includes readings inside a valid recovery band,
which the evaluator would hold. A clear level at or above the trigger is no
recovery band (`buildCanonicalMetricSpec` drops it), and a reading at the
trigger stays firing, as it does in the reducer, so a save no longer resolves
an alert the next evaluation raises again.
`TestCheckHostClearsDiskTemperatureAlertsWhenThresholdTurnsOff`,
`TestHostDiskTemperatureAlertsClearWhenAgentLeaves`,
`TestConfigSaveResolvesDiskTemperatureAlertAgainstItsDiskTypeThreshold`,
`TestConfigSaveKeepsDiskTemperatureAlertWithoutDiskType` and
`TestConfigSaveKeepsAlertTheEvaluatorKeepsWithoutRecoveryBand` in
`internal/alerts/host_unraid_lifecycle_test.go` pin these paths.
### Agent disk temperature alerts clear when their disk leaves the report
`CheckHost` also clears a disk temperature alert, and drops its pending
threshold run, once its device is missing from three consecutive non-empty
SMART lists (`cleanupHostDiskTemperatureAlerts`, a seen map keyed by resource
ID like `cleanupHostCustomSensorAlerts`). A removed, replaced or renamed disk
never sends the reading that would resolve its alert, and
`shouldPreserveAlertOutsideNodeCleanup` exempts `agent:` alerts from node
cleanup, so before this only the 24-hour stale-alert sweep in
`cleanupStaleMaps` resolved it. One omission is not proof the disk left: the
Windows, FreeBSD and controller-multiplexed Linux collectors drop a disk whose
probe fails, so the count restarts whenever the disk is reported again. Three
reports in a row of failed probes on such a disk still clear its alert.
An empty SMART list skips this cleanup and does not count toward the three,
because the agent omits the list when SMART collection fails or is
unsupported. A listed disk in standby or without a temperature also holds its
alert, since neither shows the disk cooled: standby reports avoid waking the
disk, and most failed Linux probes still list the disk with identity only.
SMART health alerts hold through standby the same way. A held alert resolves
through the normal recovery path, including any configured recovery delay,
once the disk reports cool readings again; with no evaluation for 24 hours,
the stale-alert sweep resolves it. Turning the threshold off, or removing the
agent, drops the host's pending disk temperature runs and absence counts as
well as its alerts.
`TestCheckHostClearsDiskTemperatureAlertWhenDiskLeavesSMARTReport` in
`internal/alerts/host_unraid_lifecycle_test.go` pins the clear, the holds, the
restarted count and the pending-run and count cleanup.
### Configured flapping thresholds remain reachable
Every accepted positive `FlappingThreshold`, including values above ten, must
@@ -2748,7 +2850,8 @@ legacy alert timestamp.
Suppression is bounded and reasoned, leaves the default active queue, and
remains inspectable. Expiry or explicit unsuppression returns the record to its
detector-owned state; it never resolves it. Only fresh sufficient recovery
evidence may enter resolving, and only detector recovery may resolve.
evidence may enter resolving, and only detector recovery, or a handover
recorded as `ownership_transferred` with its closing evidence, may resolve.
Per-alert snooze is the customer-facing bounded suppression contract, not a
page-local timer. `SnoozeAlert` writes the same canonical operational record,
docs/release-control/v6/internal/subsystems/api-contracts.md
+7
View File
@@ -674,6 +674,13 @@ transitions. Older payloads without these optional fields remain readable;
frontend consumers use `frontend-modern/src/types/operationalTrust.ts` as the
single TypeScript projection rather than recreating lifecycle or evidence
enums locally.
`TransitionCause` includes `ownership_transferred` for a close whose condition
another record now owns, such as a node metric handed to its Pulse agent; it
may only enter `resolved`. Alert payloads (history, state snapshot and
websocket) may also carry an optional `resolution` (`reason`,
`successorResourceId`, `successorName`, and on state payloads the alert
engine's `summary`), typed once as `AlertResolution` in
`frontend-modern/src/types/api.ts`. Its absence means an ordinary recovery.
## Shared Boundaries
docs/release-control/v6/internal/subsystems/frontend-primitives.md
+9 -1
View File
@@ -5003,7 +5003,8 @@ name, so it keeps the shared single-line rhythm. Drawers for unified resources
websocket `activeAlerts` map through `getAlertsForUnifiedResource`
(`frontend-modern/src/utils/alerts.ts`), most severe first, never from a
resource-embedded list, and show each alert's resource name and metric beside
its text like the other drawers. Their table rows take the platform host rows'
its text like the other drawers. Their table rows (Docker hosts, Machines,
Kubernetes nodes) take the platform host rows'
red (critical) or yellow tint from the same set through
`getUnifiedResourceAlertStyles`, only for unacknowledged alerts and never on
an expanded row, so a row and its drawer agree. The remaining Overview rows are additive
@@ -7120,6 +7121,13 @@ should extend the model, and section rendering should extend those owners
rather than rebuilding any of those concerns in the tab shell.
The resource-resolution handoff into the resource-incident panel now belongs to
the history state rather than the tab shell.
The history model also owns the close outcome: an alert that closed with a
typed `resolution` is status `moved` (badge `moved to agent`, its account in
`HistoryItem.closeDetail`) rather than `resolved` or `acknowledged`. The table
row carries the account in the badge title and the phone card renders it under
the message, while `getAlertHistoryStatusPresentation` and
`getAlertResolutionDetail` in `utils/alertIncidentPresentation.ts` own the
wording.
`frontend-modern/src/features/alerts/useAlertHistoryState.ts` re-exposes the
`getResource` resolver it is already given, and
`frontend-modern/src/features/alerts/AlertResourceIncidentsPanel.tsx` reads it
docs/release-control/v6/internal/subsystems/monitoring.md
+43 -1
View File
@@ -2646,6 +2646,25 @@ online/running/healthy vocabulary. The diagnostics subject set is unchanged:
snapshot (real agents), which now matches what agent-only surfaces show once
integration-backed ledger rows are excluded.
A host produced by `hostFromReadStateView` describes the agent, so its
`Status` and `LastSeen` are the agent's own. A host agent linked to a Proxmox
node merges into one row whose `LastSeen` follows its freshest source and
whose status stays online while PVE polling continues, so the row's values
would present a silent agent as online and current. The projection reads the
agent source's sighting from `HostView.SourceStatus(SourceAgent)`: the host
carries that report time and reads `offline` once it is past the reporting
lease (`hostAgentReportCurrent`, the `hostAgentHealthWindow` that
`evaluateHostAgents` applies to the host in state). Rows without an agent
sighting, such as vSphere and TrueNAS integration hosts, keep the row's
values. `linkedHostForNode` therefore stops handing a silent agent's retained
filesystem summary and ZFS datasets to its node, whose disk falls back to the
Proxmox rootfs or `/nodes` reading before disk alerts and history read it, and
agent connections built from `HostsSnapshot()` (the Connections list and the
update-readiness agent-continuity check) age the agent from its last report
rather than the PVE poll time, applying their own heartbeat cutoff.
`TestSilentLinkedAgentStopsFeedingNodeDisk` drives the disk and projection
path through `evaluateHostAgents` and the registry-backed store refresh.
Unified Agent host reports now make module readiness and updater/config
lifecycle evidence monitoring-owned observed state. Monitoring preserves the
last successful one-shot update transition across subsequent reports, forwards
@@ -3647,6 +3666,15 @@ truth. Monitoring APIs that still serve `StateSnapshot` must project
instead of trusting the cached snapshot fields, so externally served alert
counts and recently resolved incidents do not lag behind acknowledgement,
resolve, or clear operations between explicit sync points.
A recently resolved alert that closed without recovering keeps that on the
projection: `models.Alert.Resolution` (reason, successor resource ID and name,
and the alert engine's summary) is set only for such a close, `cloneAlert`
copies it into every snapshot, and `recordAlertTimelineChange` passes it into
the `alert_resolved` resource change so the incident timeline says the alert
moved. `TestAlertResolutionWireContract` in
`internal/models/metrics_types_test.go` and
`TestMonitor_HandleAlertLifecycle_HandoverCloseIsNotARecovery` in
`internal/monitoring/monitor_alert_handling_test.go` pin it.
The container entrypoint in `docker-entrypoint.sh` now also lives under this
boundary. Hosted or managed tenant bootstrap changes must preserve safe startup
when immutable read-only mounts are layered into `/etc/pulse`; the entrypoint
@@ -3744,7 +3772,21 @@ transiently `unavailable`, provider/controller `unsupported`, or unexpectedly
`missing`. Normalization may retain the last known value when the current
observation is not available, but it must preserve the current state and
reason so API and UI consumers do not present retained evidence as freshly
collected. Unified-resource physical-disk round trips must retain named
collected. A host agent's SMART temperature and I/O counters follow its
reporting lease: once `State.ExpireHostTelemetry` expires the agent, they stay
as last-known values marked `unavailable` ("host agent stopped reporting"),
and the linked Proxmox disk merge and the registry let the agent's own later
state supersede the availability it supplied earlier
(`diskinventory.MergeReportedStatus`), so neither a skipped disk poll nor a
host that is never disk-polled again can carry it forward. Expiry is
compare-and-set on the report time the evaluation judged stale, so a report
accepted in between is never expired. Every SMART temperature history writer
records a temperature only when its current collection state is available (or
predates collection state), so a retained reading has to keep its non-available
state to stay out of history; a path that relabels a carried reading as
available, such as a node-temperature carry stamped `proxmox_node_smart`, is
not covered by this rule.
Unified-resource physical-disk round trips must retain named
`StorageGroup` membership rather than degrading it to the generic `Used`
filesystem label.
That same host-agent temperature boundary must prefer a recent linked host-agent
docs/release-control/v6/internal/subsystems/notifications.md
+38
View File
@@ -1216,3 +1216,41 @@ The live monitor does not interpret pending-only cancellation counts as
recipient acceptance. Per-occurrence/per-destination receipts remain the sole
recovery admission proof, including partial delivery and lost RAM markers after
restart; an unannounced destination receives no recovery.
### A close that was not a recovery says what happened
A resolved alert can carry an `alerts.AlertResolution` when it closed without
recovering. The one reason so far, `moved_to_agent`, means the metric now
belongs to the resource's linked Pulse agent. Delivery is unchanged: the close
is still queued, gated on the firing receipt and sent, and PagerDuty still
receives `resolve` for the alert's dedup key. Only the wording changes.
`resolvedAlertMessage` gives the resolution's summary with the alert named
("Memory alert moved to pve1 (Host Agent). This is not a recovery: check the
agent for the current reading.") in place of "<resource> on <node> is now
healthy" for every templated webhook `.Message`. Grouped lists append the
short outcome ("moved to pve1 (Host Agent)") to the moved entry. ntfy titles a
single moved alert `MOVED: <resource>`. Email and Apprise title it `Pulse alert
moved: <resource>` and add the summary after the breach message. Recovery
visuals follow the same rule: when any close in a resolved delivery was not a
recovery, `WebhookPayloadData.NotRecovered` is set and the built-in templates
drop them (Discord's green embed becomes the neutral blue, both Teams cards
lose recovery green, Mattermost's check emoji becomes `:arrow_right:`), and
ntfy swaps its green check tag for `arrow_right`, for a single alert or a batch.
Built-in titles still say Resolved, which is accurate for the closed alert;
custom templates can read `.NotRecovered`, and one that hard-codes recovery
text keeps it. The resolution rides the Alert JSON, so it survives the delivery
queue round trip and appears in the default JSON payload's `alerts` array.
Issue #1682 is the precedent: a recipient called "is now healthy" a false
factual claim when an alert closed without recovering.
The resolve transition such a close links to has cause `ownership_transferred`
in `internal/operationaltrust/contracts.go`, not `recovery_evidence`. A
notification link records the transition ID and lifecycle state, never the
cause, so delivery, receipts and recovery admission treat it exactly like a
recovery's resolve transition. `TestOwnershipTransferOnlyClosesWithEvidence`
pins that the cause may only enter `resolved` and only with evidence.
Verification: `resolved_handover_test.go` drives the firing, its receipt and
the queued close through the normal entry points for ntfy, and renders Discord,
both Teams cards, Mattermost, PagerDuty, a grouped Slack list, a grouped ntfy
batch and email/Apprise content, plus the unchanged ordinary recovery wording
and visuals.
docs/release-control/v6/internal/subsystems/patrol-intelligence.md
+8
View File
@@ -1380,6 +1380,14 @@ to traverse the list again. At widths where the detail replaces the queue's
reading position, its visible close affordance is an explicit `Back to list`
control; wide layouts retain the compact close icon. Both controls clear the
deep link and restore focus to the originating queue row.
A resolved item whose latest close is an `ownership_transferred` transition did
not recover: its condition moved to another record, such as the linked Pulse
agent's own alert. The detail's state badge reads `Moved` in the info tone
instead of the green `Resolved`, and the transition's reason (where the
condition went, and that it is not a recovery) sits directly under the summary
rather than only inside Evidence and history (`getAttentionHandoverClose` in
`patrolHomePresentation.ts`). The timeline names that cause `Ownership
transferred`, never `Recovery evidence`.
The active Patrol queue now uses compact severity-accented rows for
prioritisation and a single focused review panel for the selected issue. The
docs/release-control/v6/internal/subsystems/unified-resources.md
+26 -1
View File
@@ -2265,7 +2265,17 @@ may repeat beside it. The Docker hosts (`DockerHostsTable.tsx`) and Machines
`getUnifiedResourceAlertStyles`, so a collapsed row never looks healthy while
its drawer lists an unacknowledged alert (an expanded row drops the tint while
its drawer shows the alerts); acknowledged alerts stay in the drawer without
tinting the row.
tinting the row. Kubernetes node rows (`KubernetesNodesTable.tsx`) tint the
same way: a node that runs a Pulse agent is an agent row whose alerts are keyed
`agent:<id>` and carry the hostname, so the earlier id-and-display-name match
left it untinted while its drawer listed a critical alert. The Kubernetes
overview's "Nodes needing attention" and the nodes table's default order rank
a node by the stronger of its readiness state and those unacknowledged alerts
(`getKubernetesNodeAttentionRank`), so a Ready node at 100% memory is named
beside the NotReady ones. vSphere host and TrueNAS system rows keep
`getAlertStyles` with the host name: their VMs', pools' and apps' alerts carry
that name as their node, and no producer keys alerts on those rows' `agent:`
aliases (a Pulse agent on a TrueNAS box stays a separate resource).
Machine and host overview cards that render compact system, hardware, disk,
and temperature facts must also compose the frontend-primitives
`InfoCardKeyValueRow`. Mobile rows retain their condensed endpoint layout;
@@ -3567,6 +3577,13 @@ the five field statuses for serial, temperature, I/O, controller, and pool.
`unavailable`, `unsupported`, or `missing` state may retain a prior value for
continuity, but the state and reason must survive so the consumer cannot claim
fresh evidence or synthesize controller-level activity for one member.
The one exception is a source withdrawing its own evidence: when the agent
row reports a field as no longer available, that supersedes an `available`
state the Proxmox row still carries from the same source
(`diskinventory.MergeReportedStatus`), in either ingest order. A host agent
past its reporting lease must not stay "collected" through the Proxmox row's
copy of the agent's own state, including when its host is down and no disk
poll refreshes that copy.
Cross-source correlation compares normalized serial and WWN values across
fields without truncation, allowing a PVE bare-hex array-volume serial to join
@@ -4222,6 +4239,14 @@ such as `alert_fired`, `alert_acknowledged`, `alert_unacknowledged`,
AI-local annotations. Snooze and resume projections must preserve the actor
and exact suppression expiry when present; they pause delivery and escalation
without acknowledging, resolving, or replacing the underlying incident.
An `alert_resolved` change for a close that was not a recovery carries the
alert engine's summary as its reason and the reason code as `alert_resolution`
metadata (`AlertTimelineChange.ResolutionReason` and `ResolutionSummary`), so
incident projection titles the close with where the alert went instead of
"Alert resolved: <breach message>". The resource drawer still labels the change
by kind (`Alert resolved`) and prefixes its headline with that label; only the
reason after it says the alert moved. Ordinary recoveries omit the key, and
every other lifecycle kind ignores the fields.
Alert-scoped
incident memory may still project those events for one investigation thread,
but the durable source of truth for resource-affecting alert lifecycle and
frontend-modern/browser-verification/0434175b9a7f1326.json
+38
View File
@@ -0,0 +1,38 @@
{
"version": 1,
"base_sha": "bffd21f47026ca2408ee1d0b5bb071c02e60c623",
"verified_at": "2026-10-06T15:48:24Z",
"result": "passed",
"changed_paths": [
"frontend-modern/src/features/standalone/AgentsMachinesTable.tsx"
],
"content_sha256": {
"frontend-modern/src/features/standalone/AgentsMachinesTable.tsx": "db24691402c341a2ceec3135d6425c03c9706b6cb57630f9c0efc2e956179264"
},
"routes": [
"/standalone/machines"
],
"viewports": [
{
"width": 1600,
"height": 1000
},
{
"width": 390,
"height": 844
}
],
"states": [
"Combined AgentsMachinesTable.tsx on main after #2579 (row alert tint) and #2578 (shared relative-time clock) merged; private mock stack built from bffd21f470",
"A critical memory alert for Apollo-114 (resourceId agent-acdfdee2953587fb) injected into the websocket activeAlerts and /api/alerts/active; Spectrum-709 left without alerts",
"1600px light: Apollo-114 row bg-red-50 (rgb 254,242,242), Spectrum-709 untinted; with Last seen hidden the identity title read \"| seen just now\"",
"Playwright page.clock fast-forwarded 3h with /ws held and REST replayed: Apollo-114 title \"| seen 3h ago\" and the red tint unchanged, Spectrum-709 still untinted",
"390px dark: Apollo-114 row dark:bg-red-950/25, Seen cell \"just now\" -> \"3h ago\" after the same 3h jump with the tint unchanged",
"Apollo-114 drawer lists the injected \"Memory usage is 96%\" alert at both widths; no page errors"
],
"interactions": [
"Opened View > Choose which columns to display and unchecked Last seen on Machines (1600px)",
"page.clock.fastForward 3h with data held",
"Clicked the Apollo-114 row to open its drawer"
]
}
frontend-modern/browser-verification/83848f2d5ef2a326.json
+51
View File
@@ -0,0 +1,51 @@
{
"version": 1,
"base_sha": "3b290640a51ef0f488f9cd06b4aa644adbdcc6b3",
"verified_at": "2026-10-06T15:43:08Z",
"result": "passed",
"changed_paths": [
"frontend-modern/src/features/kubernetes/KubernetesNodesTable.tsx",
"frontend-modern/src/features/kubernetes/KubernetesPageSurface.tsx",
"frontend-modern/src/features/kubernetes/kubernetesPageModel.ts"
],
"content_sha256": {
"frontend-modern/src/features/kubernetes/KubernetesNodesTable.tsx": "8861f6f170dd9ebab4eebe949e3a311af46916aa10d8d2d2768a33e3d64b2d1a",
"frontend-modern/src/features/kubernetes/KubernetesPageSurface.tsx": "9a2096f3ff967cec93024466d15c1b5117f08d89985c1955965d336a38d03bc0",
"frontend-modern/src/features/kubernetes/kubernetesPageModel.ts": "0c2f0be19aa05f8e07f1167273b26b17b0643c73855df8246fcf5f591d3ae5b8"
},
"routes": [
"/kubernetes/overview",
"/kubernetes/nodes"
],
"viewports": [
{
"width": 1440,
"height": 1000
},
{
"width": 1280,
"height": 900
},
{
"width": 390,
"height": 844
}
],
"states": [
"Private mock stack (backend built from base 3b290640a5 on its own port, Vite serving this tree); mock mode skips CheckHost, so the private binary added that call through a go build overlay of internal/monitoring/monitor_alerts.go that is not part of this commit",
"Kubernetes nodes: Prod Euw1 K8s 02 (agent row agent-c8db98b4db98ed5c) has a backend-raised critical alert keyed agent:host-k8s-k8s-production-1-k8s-production-1-node-2-955a86fe (node prod-euw1-k8s-02, Agent memory at 100.0%); before the change its row was untinted while its drawer listed the alert, now the row is red-50 and its drawer lists the same alert",
"Kubernetes nodes default order: K8s 02 (Ready, critical alert) first, then K8s 03 (Degraded), then K8s 01 (Ready); K8s 01 and K8s 03 untinted with no drawer alerts",
"Kubernetes overview: Nodes needing attention lists K8s 02 (red) then K8s 03 (Degraded); before the change it listed only K8s 03",
"After acknowledging the K8s 02 alert through the private stack API: its row is untinted, it drops out of Nodes needing attention, the nodes table falls back to K8s 03, K8s 01, K8s 02, and its drawer still lists the alert as Acknowledged; unacknowledged again afterwards",
"Expanded K8s 02 row drops the tint (surface hover background) while its drawer shows the alert",
"Light theme at 1440px and 390px and dark theme at 1280px: red-50 / red-950/25 on K8s 02 only, no horizontal overflow at 390px",
"TrueNAS and vSphere tables unchanged: truenas-main stays amber for its pool alerts (node truenas-main) and its drawer lists the pool problems; esxi-07 amber for its own alert",
"No page errors or console errors"
],
"interactions": [
"Open Kubernetes overview and Nodes, read each node row's tint and order",
"Expand and collapse the K8s 02 row drawer and read its Needs attention list",
"Acknowledge then unacknowledge the K8s 02 memory alert and re-read the overview list, row tint, order and drawer",
"Repeat the overview and nodes checks at the 390px phone viewport (row tap to expand) and in dark mode"
]
}
frontend-modern/browser-verification/8b6df6398a29e2e3.json
+54
View File
@@ -0,0 +1,54 @@
{
"version": 1,
"base_sha": "982a6f6148cf9d39883e4443191b4d40b59cfd7c",
"verified_at": "2026-10-06T15:03:00Z",
"result": "passed",
"changed_paths": [
"frontend-modern/src/features/alerts/AlertHistoryMobileList.tsx",
"frontend-modern/src/features/alerts/AlertHistoryTableAlertRow.tsx",
"frontend-modern/src/features/alerts/alertHistoryModel.ts",
"frontend-modern/src/features/patrol/PatrolAttentionWorkbench.tsx",
"frontend-modern/src/features/patrol/patrolHomePresentation.ts",
"frontend-modern/src/types/api.ts",
"frontend-modern/src/types/operationalTrust.ts",
"frontend-modern/src/utils/alertIncidentPresentation.ts"
],
"content_sha256": {
"frontend-modern/src/features/alerts/AlertHistoryMobileList.tsx": "94bb73163dd86c1f045a0b5559b42486198ab824ef2abccfc3c8c1dcc0ff2784",
"frontend-modern/src/features/alerts/AlertHistoryTableAlertRow.tsx": "aacc7232596a47d64c7e6ec3b29dfaf51be6e1691d9be906b30bfce49f4daf6f",
"frontend-modern/src/features/alerts/alertHistoryModel.ts": "30548565d29ca16fe1c1192687c881defc064ba1b267fdcd40fc28922a51fed4",
"frontend-modern/src/features/patrol/PatrolAttentionWorkbench.tsx": "b3c87b3ead9bde22a091b2c080b9eaca18face9ba153085bb1a1dbb9b6ae9f54",
"frontend-modern/src/features/patrol/patrolHomePresentation.ts": "04338cdce5b54bc041267b325a6985a9a8f84ef9e5a8dba8040c7d9fc5ab1811",
"frontend-modern/src/types/api.ts": "17a948b6ba78780ef2b95c1f397d2d9a04eebd2726c568a8316e25e7c8f9e6b3",
"frontend-modern/src/types/operationalTrust.ts": "9a50a869b1f6ef2885f30a354fb70b36b64fbb1a3f00ba4476d592389f9ba954",
"frontend-modern/src/utils/alertIncidentPresentation.ts": "f77c42b48f22bddcccb43a3da43b3771bb03d25c6d23b7efa200a34dae1da0e9"
},
"routes": [
"/alerts/history",
"/patrol?attention=pve1::metric-threshold:memory"
],
"viewports": [
{
"width": 1280,
"height": 900
},
{
"width": 375,
"height": 812
}
],
"states": [
"Private non-mock stack (backend :7691, Vite :5291) on a data dir whose alert history was written by the real alerts manager: a pve1 node memory alert closed through releaseCanonicalMetricAlert with a moved_to_agent resolution (successor pve1 (Host Agent)) and a pve1 CPU alert closed as an ordinary release; the handover record's resolve transition carries cause ownership_transferred",
"Alerts history at 1280: the CPU row's status badge reads resolved, the memory row's reads moved to agent in blue, and its title is 'Moved to pve1 (Host Agent). This is not a recovery: check the agent for the current reading.'",
"Memory row Timeline expanded: the incident timeline lists 'Node memory at 95.0%' then 'Alert moved to pve1 (Host Agent). This is not a recovery: check the agent for the current reading.' where it said 'Alert resolved' before; the incident header still says resolved (incident status vocabulary, unchanged)",
"Alerts history at 375: the memory card shows the moved to agent badge and the account as a muted line under the breach message; the CPU card is unchanged",
"Patrol deep link to the moved record at 1280 and 375: the detail badge reads Moved (info tone) instead of green Resolved, the handover sentence sits directly under 'Node memory at 95.0%', and Evidence and history shows 'Open to Resolved, Ownership transferred' with the same reason; no Recovery evidence label anywhere; no horizontal scroll at 375 and no console errors"
],
"interactions": [
"Loaded /alerts/history at 1280x900 and read both rows' status badges and titles",
"Clicked the memory row's Timeline button and read the expanded incident timeline",
"Reloaded /alerts/history at 375x812 and read both history cards",
"Opened /patrol?attention=pve1::metric-threshold:memory at 1280x900 and 375x812, opened Evidence and history, and read the badge, summary, handover line and timeline",
"Checked document width against the viewport at 375 and read the console for errors"
]
}
frontend-modern/src/features/alerts/AlertHistoryMobileList.tsx
+3
View File
@@ -229,6 +229,9 @@ export function AlertHistoryMobileList(props: AlertHistoryMobileListProps) {
</div>
<p class="mt-2 text-xs leading-relaxed text-base-content">{alert.description}</p>
<Show when={alert.closeDetail}>
{(detail) => <p class="mt-1 text-xs leading-relaxed text-muted">{detail()}</p>}
</Show>
<dl class="mt-3 grid grid-cols-3 gap-2 text-[10px] text-muted">
<div>
frontend-modern/src/features/alerts/AlertHistoryTableAlertRow.tsx
+1 -1
View File
@@ -74,7 +74,7 @@ export function AlertHistoryTableAlertRow(props: AlertHistoryTableAlertRowProps)
</TableCell>
<TableCell class={getPlatformTableCellClassForKind('badge')}>
<span class={historyStatusPresentation().className}>
<span class={historyStatusPresentation().className} title={props.alert.closeDetail}>
{historyStatusPresentation().label}
</span>
</TableCell>
frontend-modern/src/features/alerts/alertHistoryModel.ts
+14 -1
View File
@@ -1,5 +1,9 @@
import type { Alert } from '@/types/api';
import type { Resource } from '@/types/resource';
import {
ALERT_HISTORY_MOVED_STATUS,
getAlertResolutionDetail,
} from '@/utils/alertIncidentPresentation';
import { isPulseSystemAlert } from '@/utils/alertScope';
import { alertTypeDisplayLabel, unifiedTypeToAlertDisplayType } from './helpers';
@@ -25,6 +29,8 @@ export interface HistoryItem {
title: string;
rawAlertType?: string;
description?: string;
// Set when the alert closed without recovering: what happened instead.
closeDetail?: string;
acknowledged?: boolean;
systemAlert?: boolean;
}
@@ -206,11 +212,17 @@ export function buildAlertHistoryItems({
alertHistory.forEach((alert) => {
if (activeAlertIds.has(alert.id)) return;
const systemScoped = isPulseSystemAlert(alert);
// A close that was not a recovery must not read as "resolved".
const closeDetail = getAlertResolutionDetail(alert.resolution);
items.push({
id: alert.id,
source: 'alert',
status: alert.acknowledged ? 'acknowledged' : 'resolved',
status: closeDetail
? ALERT_HISTORY_MOVED_STATUS
: alert.acknowledged
? 'acknowledged'
: 'resolved',
startTime: alert.startTime,
endTime: alert.lastSeen,
duration: formatAlertHistoryDuration(alert.startTime, alert.lastSeen, now),
@@ -231,6 +243,7 @@ export function buildAlertHistoryItems({
title: alertTypeDisplayLabel(alert.type),
rawAlertType: alert.type,
description: alert.message,
...(closeDetail ? { closeDetail } : {}),
acknowledged: alert.acknowledged,
});
});
frontend-modern/src/features/kubernetes/KubernetesNodesTable.tsx
+25 -10
View File
@@ -6,7 +6,7 @@ import { StackedMemoryBar } from '@/components/Workloads/StackedMemoryBar';
import { TableCell, TableRow } from '@/components/shared/Table';
import { ResourceNameWithWebInterfaceLink } from '@/components/shared/WebInterfaceLink';
import { asTrimmedString } from '@/utils/stringUtils';
import { getAlertStyles } from '@/utils/alerts';
import { getUnifiedResourceAlertStyles } from '@/utils/alerts';
import { useWebSocket } from '@/contexts/appRuntime';
import { useAlertsActivation } from '@/stores/alertsActivation';
import { unifiedPlatformOverrideIdCandidates } from '@/features/alerts/alertOverridesModel';
@@ -41,6 +41,7 @@ import type { Resource } from '@/types/resource';
import {
compareKubernetesNodes,
filterKubernetesResources,
getKubernetesNodeAttentionRank,
kubernetesClusterLabel,
mapKubernetesNodeStatus,
type KubernetesResourceStatusFilter,
@@ -145,19 +146,27 @@ export const KubernetesNodesTable: Component<{
const drawer = createPlatformResourceDetailState({ idPrefix: 'kubernetes-node-drawer' });
const resolveResourceLabel = createPlatformResourceLabelResolver(() => props.resources);
// User-controlled sorting layered over the attention-first default: rows
// are pre-sorted by the status compare, so a user sort keeps that order
// for ties and the table falls straight back to it when the sort clears.
// are pre-sorted by attention (status and open alerts), then the status
// compare, so a user sort keeps that order for ties and the table falls
// straight back to it when the sort clears.
const sort = createPlatformTableSortState({
storageKey: props.sortStorageKey ?? 'kubernetesNodes',
sortKeys: KUBERNETES_NODE_SORT_KEYS,
descendingFirst: ['cpu', 'memory', 'uptime', 'capacity'],
});
const sortedRows = createMemo(() =>
sort.sortRows(
[...tableState.filtered()].sort(compareKubernetesNodes),
const sortedRows = createMemo(() => {
const rows = tableState.filtered();
const enabled = alertsEnabled();
const ranks = new Map(
rows.map((node) => [node.id, getKubernetesNodeAttentionRank(node, activeAlerts, enabled)]),
);
return sort.sortRows(
[...rows].sort(
(a, b) => (ranks.get(a.id) ?? 2) - (ranks.get(b.id) ?? 2) || compareKubernetesNodes(a, b),
),
getKubernetesNodeSortValue,
),
);
);
});
return (
<Show
@@ -367,8 +376,12 @@ export const KubernetesNodesTable: Component<{
const canRenderMetrics = () => indicator().variant !== 'muted';
const detailRowId = () => drawer.detailRowId(node);
const isExpanded = () => drawer.isExpanded(node);
// The same open alerts the node's drawer lists: a node that
// runs a Pulse agent has its alerts keyed on the agent and
// labelled with its hostname, so matching the row's id or
// display name finds none of them.
const nodeAlertStyles = createMemo(() =>
getAlertStyles(node.id, activeAlerts, alertsEnabled(), name()),
getUnifiedResourceAlertStyles(node, activeAlerts, alertsEnabled()),
);
const nodeAlertBg = () => {
const s = nodeAlertStyles();
@@ -380,7 +393,9 @@ export const KubernetesNodesTable: Component<{
return (
<>
<TableRow
class={`${getPlatformResourceDetailRowClass(isExpanded())} text-[11px] sm:text-xs ${nodeAlertBg()}`}
class={`${getPlatformResourceDetailRowClass(isExpanded())} text-[11px] sm:text-xs ${
isExpanded() ? '' : nodeAlertBg()
}`}
data-kubernetes-node-row={node.id}
onClick={() => drawer.toggle(node)}
>
frontend-modern/src/features/kubernetes/KubernetesPageSurface.tsx
+14 -4
View File
@@ -9,8 +9,10 @@ import {
collectOutdatedAgentHosts,
formatAgentVersionDisplay,
} from '@/features/platformPage/agentVersion';
import { useWebSocket } from '@/contexts/appRuntime';
import { useUnifiedResources, type UnifiedResourceFacets } from '@/hooks/useUnifiedResources';
import { buildKubernetesPath, KUBERNETES_QUERY_PARAMS } from '@/routing/resourceLinks';
import { useAlertsActivation } from '@/stores/alertsActivation';
import { updateStore } from '@/stores/updates';
import {
PLATFORM_HEALTH_FILTER_OPTIONS,
@@ -710,14 +712,22 @@ function KubernetesOverview(props: KubernetesOverviewProps) {
return posture.podAttention + posture.deploymentAttention;
});
const nodesNeedingAttention = createMemo(() => getKubernetesNodesNeedingAttention(props.model()));
const { activeAlerts } = useWebSocket();
const alertsActivation = useAlertsActivation();
const nodesNeedingAttention = createMemo(() =>
getKubernetesNodesNeedingAttention(
props.model(),
activeAlerts,
alertsActivation.detectionEnabled(),
),
);
return (
<div class="space-y-4">
{/*
Problems lead, inventory follows: active health signals, then the
nodes that are not Ready, before the cluster summary and workloads.
Filtering stays on the Nodes and Events tabs.
nodes that are not Ready or have an open alert, before the cluster
summary and workloads. Filtering stays on the Nodes and Events tabs.
*/}
<Show when={props.model().incidents.length > 0}>
<div id="kubernetes-health-signals">
@@ -738,7 +748,7 @@ function KubernetesOverview(props: KubernetesOverviewProps) {
sortStorageKey="kubernetesNodesNeedingAttention"
emptyIcon={k8sIcon()}
emptyTitle="All nodes are Ready"
emptyDescription="Nodes appear here when they are not Ready, cordoned or degraded."
emptyDescription="Nodes appear here when they are not Ready, cordoned, degraded or alerting."
showToolbar={false}
/>
</div>
frontend-modern/src/features/kubernetes/__tests__/KubernetesNodesTable.test.tsx
+85 -1
View File
@@ -1,6 +1,8 @@
import { cleanup, fireEvent, render, screen } from '@solidjs/testing-library';
import { createStore, reconcile } from 'solid-js/store';
import { afterEach, describe, expect, it, vi } from 'vitest';
import type { Alert } from '@/types/api';
import type { Resource } from '@/types/resource';
import { KubernetesNodesTable } from '../KubernetesNodesTable';
@@ -12,8 +14,11 @@ vi.mock('@/components/Workloads/StackedMemoryBar', () => ({
StackedMemoryBar: () => <div data-testid="stacked-memory-bar" />,
}));
// The websocket's activeAlerts is a Solid store, so rows must follow it live.
const activeAlertsRef = vi.hoisted(() => ({ current: {} as Record<string, Alert> }));
vi.mock('@/contexts/appRuntime', () => ({
useWebSocket: () => ({ activeAlerts: [] }),
useWebSocket: () => ({ activeAlerts: activeAlertsRef.current }),
}));
vi.mock('@/stores/alertsActivation', () => ({
@@ -51,8 +56,12 @@ const makeNodeResource = (overrides: Partial<Resource> = {}): Resource => ({
...overrides,
});
const [activeAlerts, setActiveAlerts] = createStore<Record<string, Alert>>({});
activeAlertsRef.current = activeAlerts;
afterEach(() => {
cleanup();
setActiveAlerts(reconcile({}));
vi.clearAllMocks();
});
@@ -146,4 +155,79 @@ describe('KubernetesNodesTable', () => {
'false',
);
});
it('tints a node row for the open alerts its drawer lists and leads with it', () => {
const agentAlert = (id: string, resourceId: string, overrides: Partial<Alert> = {}): Alert => ({
id,
type: 'memory',
level: 'critical',
resourceId,
resourceName: 'Prod Euw1 K8s 02',
node: 'prod-euw1-k8s-02',
instance: '',
message: 'Agent memory at 100.0%',
value: 100,
threshold: 90,
startTime: '2026-10-06T10:00:00Z',
acknowledged: false,
...overrides,
});
// A node that runs a Pulse agent is an agent row; its alerts are keyed
// "agent:<agentId>" and carry the hostname, never the row id or name.
const agentNode = (id: string, name: string, agentId: string, ready = true) =>
makeNodeResource({
id,
name,
displayName: name,
type: 'agent',
agent: { agentId },
kubernetes: { ...makeNodeResource().kubernetes, nodeName: id, ready },
} as Partial<Resource>);
setActiveAlerts({
memory: agentAlert('memory', 'agent:host-k8s-node-2'),
});
const { container } = render(() => (
<KubernetesNodesTable
resources={[
agentNode('node-1', 'Prod Euw1 K8s 01', 'host-k8s-node-1'),
agentNode('node-2', 'Prod Euw1 K8s 02', 'host-k8s-node-2'),
makeNodeResource({
id: 'node-3',
name: 'Prod Euw1 K8s 03',
kubernetes: { ...makeNodeResource().kubernetes, unschedulable: true },
}),
]}
emptyIcon={<span />}
emptyTitle="No nodes"
emptyDescription="No nodes"
showToolbar={false}
/>
));
const row = (id: string) =>
container.querySelector(`[data-kubernetes-node-row="${id}"]`) as HTMLElement;
const order = () =>
[...container.querySelectorAll('[data-kubernetes-node-row]')].map((element) =>
element.getAttribute('data-kubernetes-node-row'),
);
expect(row('node-2')).toHaveClass('bg-red-50');
expect(row('node-1')).not.toHaveClass('bg-red-50');
// A Ready node with a critical alert leads, ahead of a cordoned node.
expect(order()).toEqual(['node-2', 'node-3', 'node-1']);
// The tint and the order follow the live alert store: acknowledging
// clears both, and a new warning tints the row amber.
setActiveAlerts('memory', 'acknowledged', true);
expect(row('node-2')).not.toHaveClass('bg-red-50');
expect(order()).toEqual(['node-3', 'node-1', 'node-2']);
setActiveAlerts('memory', { acknowledged: false, level: 'warning' });
expect(row('node-2')).toHaveClass('bg-yellow-50');
// An expanded row drops the tint while its drawer lists the alert.
fireEvent.click(row('node-2'));
expect(row('node-2')).not.toHaveClass('bg-yellow-50');
fireEvent.click(row('node-2'));
expect(row('node-2')).toHaveClass('bg-yellow-50');
setActiveAlerts(reconcile({}));
expect(row('node-2')).not.toHaveClass('bg-yellow-50');
});
});
frontend-modern/src/features/kubernetes/__tests__/KubernetesPageSurface.contract.test.tsx
+68
View File
@@ -1,12 +1,15 @@
import { Route, Router } from '@solidjs/router';
import { cleanup, render, screen } from '@solidjs/testing-library';
import { createStore, reconcile } from 'solid-js/store';
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
import type { Alert } from '@/types/api';
import type { Resource } from '@/types/resource';
import { KubernetesPageSurface } from '../KubernetesPageSurface';
const mockUseUnifiedResources = vi.fn();
const mockPathname = vi.hoisted(() => vi.fn(() => '/'));
const mockVersionInfo = vi.hoisted(() => vi.fn());
const activeAlertsRef = vi.hoisted(() => ({ current: {} as Record<string, Alert> }));
const makeResource = (resource: Partial<Resource> & Pick<Resource, 'id' | 'type'>): Resource => ({
name: resource.id,
@@ -40,6 +43,14 @@ vi.mock('@/hooks/useUnifiedResources', () => ({
useUnifiedResources: (...args: unknown[]) => mockUseUnifiedResources(...args),
}));
vi.mock('@/contexts/appRuntime', () => ({
useWebSocket: () => ({ activeAlerts: activeAlertsRef.current }),
}));
vi.mock('@/stores/alertsActivation', () => ({
useAlertsActivation: () => ({ detectionEnabled: () => true }),
}));
vi.mock('@/stores/updates', () => ({
updateStore: {
versionInfo: mockVersionInfo,
@@ -129,6 +140,7 @@ vi.mock('../KubernetesNodesTable', () => ({
<div
data-testid="nodes-table"
data-rows={props.resources.length}
data-ids={props.resources.map((resource) => resource.id).join(',')}
data-title={props.title}
data-sort-key={props.sortStorageKey}
/>
@@ -159,6 +171,9 @@ vi.mock('../KubernetesStorageTable', () => ({
),
}));
const [activeAlerts, setActiveAlerts] = createStore<Record<string, Alert>>({});
activeAlertsRef.current = activeAlerts;
describe('KubernetesPageSurface contract', () => {
beforeEach(() => {
mockPathname.mockReturnValue('/');
@@ -167,6 +182,7 @@ describe('KubernetesPageSurface contract', () => {
afterEach(() => {
cleanup();
setActiveAlerts(reconcile({}));
vi.clearAllMocks();
});
@@ -207,6 +223,58 @@ describe('KubernetesPageSurface contract', () => {
).toBeTruthy();
});
it('names a Ready node with an open alert among the nodes needing attention', () => {
setResources([
makeResource({ id: 'cluster-1', type: 'k8s-cluster' }),
makeResource({
id: 'node-memory',
type: 'agent',
sources: ['agent', 'kubernetes'],
agent: { agentId: 'host-k8s-02' },
kubernetes: { ready: true } as Resource['kubernetes'],
} as Partial<Resource> & Pick<Resource, 'id' | 'type'>),
makeResource({
id: 'node-ok',
type: 'k8s-node',
kubernetes: { ready: true } as Resource['kubernetes'],
}),
]);
renderSurface();
// Every node is Ready, so nothing needs attention until an alert opens.
expect(screen.queryByTestId('nodes-table')).toBeNull();
// The node's agent raises it: keyed on the agent, labelled with the hostname.
setActiveAlerts({
memory: {
id: 'memory',
type: 'memory',
level: 'critical',
resourceId: 'agent:host-k8s-02',
resourceName: 'Prod Euw1 K8s 02',
node: 'prod-euw1-k8s-02',
instance: '',
message: 'Agent memory at 100.0%',
value: 100,
threshold: 90,
startTime: '2026-10-06T10:00:00Z',
acknowledged: false,
},
});
const attention = screen.getByTestId('nodes-table');
expect(attention).toHaveAttribute('data-title', 'Nodes needing attention');
expect(attention).toHaveAttribute('data-ids', 'node-memory');
// Acknowledged: someone owns it, and the row is no longer tinted either.
setActiveAlerts('memory', 'acknowledged', true);
expect(screen.queryByTestId('nodes-table')).toBeNull();
setActiveAlerts('memory', 'acknowledged', false);
expect(screen.getByTestId('nodes-table')).toHaveAttribute('data-ids', 'node-memory');
// Resolved: the alert leaves the store and the node leaves the list.
setActiveAlerts(reconcile({}));
expect(screen.queryByTestId('nodes-table')).toBeNull();
});
it('shows no attention table when every node is Ready', () => {
setResources([
makeResource({ id: 'cluster-1', type: 'k8s-cluster' }),
frontend-modern/src/features/kubernetes/__tests__/kubernetesPageModel.test.ts
+89
View File
@@ -1,6 +1,8 @@
import { describe, expect, it } from 'vitest';
import type { Alert } from '@/types/api';
import type { Resource } from '@/types/resource';
import {
getKubernetesNodeAttentionRank,
getKubernetesNodesNeedingAttention,
splitKubernetesNameTail,
isSingleKubernetesCluster,
@@ -1178,4 +1180,91 @@ describe('getKubernetesNodesNeedingAttention', () => {
expect(result.map((entry: { id: string }) => entry.id)).toEqual(['a-down', 'c-cordoned']);
});
// A node running a Pulse agent is an agent row whose alerts are keyed
// "agent:<agentId>" and labelled with its hostname.
const agentNode = (id: string, agentId: string, kubernetes: Record<string, unknown>) =>
({
id,
name: id,
type: 'agent',
status: 'online',
agent: { agentId },
kubernetes,
}) as never;
const alert = (id: string, resourceId: string, overrides: Partial<Alert> = {}): Alert => ({
id,
type: 'memory',
level: 'critical',
resourceId,
resourceName: id,
node: id,
instance: '',
message: id,
value: 100,
threshold: 90,
startTime: '2026-10-06T10:00:00Z',
acknowledged: false,
...overrides,
});
it('names Ready nodes with an unacknowledged open alert, critical ones with NotReady', () => {
const nodes = [
agentNode('a-ready', 'host-a', { ready: true }),
agentNode('b-memory', 'host-b', { ready: true }),
agentNode('c-cordoned', 'host-c', { ready: true, unschedulable: true }),
agentNode('d-down', 'host-d', { ready: false }),
agentNode('e-disk', 'host-e', { ready: true }),
agentNode('f-acked', 'host-f', { ready: true }),
];
const activeAlerts = {
memory: alert('memory', 'agent:host-b'),
// A component alert the drawer lists counts for its node too.
disk: alert('disk', 'agent:host-e/disk:root', { type: 'disk', level: 'warning' }),
acked: alert('acked', 'agent:host-f', { acknowledged: true }),
};
const result = getKubernetesNodesNeedingAttention({ nodes } as never, activeAlerts, true);
// Within a rank, node state orders: NotReady ahead of a Ready node.
expect(result.map((entry: { id: string }) => entry.id)).toEqual([
'd-down',
'b-memory',
'c-cordoned',
'e-disk',
]);
// With alert detection off the list falls back to node state alone.
expect(
getKubernetesNodesNeedingAttention({ nodes } as never, activeAlerts, false).map(
(entry: { id: string }) => entry.id,
),
).toEqual(['d-down', 'c-cordoned']);
});
it('counts alerts keyed on a node without an agent by its own id', () => {
// Unified Kubernetes metric and incident alerts carry the resource id and
// the cluster as their node.
const nodes = [node('k8s-node-a', { ready: true }), node('k8s-node-b', { ready: true })];
const activeAlerts = {
cpu: alert('cpu', 'k8s-node-b', { type: 'cpu', level: 'warning', node: 'prod-west' }),
};
expect(
getKubernetesNodesNeedingAttention({ nodes } as never, activeAlerts, true).map(
(entry: { id: string }) => entry.id,
),
).toEqual(['k8s-node-b']);
});
it('ranks a node by the stronger of its state and its open alerts', () => {
const ready = agentNode('ready', 'host-ready', { ready: true });
const cordoned = agentNode('cordoned', 'host-cordoned', { ready: true, unschedulable: true });
const warning = { w: alert('w', 'agent:host-ready', { level: 'warning' }) };
const critical = { c: alert('c', 'agent:host-cordoned') };
expect(getKubernetesNodeAttentionRank(ready, {}, true)).toBe(2);
expect(getKubernetesNodeAttentionRank(ready, warning, true)).toBe(1);
expect(getKubernetesNodeAttentionRank(cordoned, {}, true)).toBe(1);
expect(getKubernetesNodeAttentionRank(cordoned, critical, true)).toBe(0);
});
});
frontend-modern/src/features/kubernetes/kubernetesPageModel.ts
+39 -10
View File
@@ -4,7 +4,9 @@ import type {
ResourceKubernetesPodContainerStatus,
ResourceType,
} from '@/types/resource';
import type { Alert } from '@/types/api';
import type { StatusIndicator, StatusIndicatorVariant } from '@/utils/status';
import { getUnifiedResourceAlertStyles } from '@/utils/alerts';
import { resolveResourcePlatformType } from '@/utils/sourcePlatforms';
import { matchesSearchTermSplit, splitSearchExclusions } from '@/utils/searchQuery';
@@ -801,17 +803,44 @@ const countKubernetesDanger = (
mapper: (resource: Resource) => StatusIndicator,
): number => resources.filter((resource) => mapper(resource).variant === 'danger').length;
// Nodes the overview should name before any inventory: NotReady first, then
// cordoned or degraded ones. "2/3 nodes" on the cluster row says something is
// wrong but not which node; this list answers that.
export function getKubernetesNodesNeedingAttention(model: KubernetesPageModel): Resource[] {
const rank = (node: Resource) => {
const variant = mapKubernetesNodeStatus(node).variant;
return variant === 'danger' ? 0 : variant === 'warning' ? 1 : 2;
};
// Attention order shared by the overview's "Nodes needing attention" list and
// the nodes table's default sort: 0 for a NotReady node or one with an
// unacknowledged critical alert, 1 for a cordoned or degraded node or one with
// another unacknowledged alert, 2 otherwise. The alerts are the ones the
// node's row tint and drawer use, so a Ready node at 100% memory ranks with
// the NotReady ones instead of below every degraded node.
export function getKubernetesNodeAttentionRank(
node: Resource,
activeAlerts: Record<string, Alert>,
alertsEnabled: boolean,
): 0 | 1 | 2 {
const variant = mapKubernetesNodeStatus(node).variant;
const statusRank = variant === 'danger' ? 0 : variant === 'warning' ? 1 : 2;
const alertStyles = getUnifiedResourceAlertStyles(node, activeAlerts, alertsEnabled);
const alertRank = !alertStyles.hasUnacknowledgedAlert
? 2
: alertStyles.severity === 'critical'
? 0
: 1;
return Math.min(statusRank, alertRank) as 0 | 1 | 2;
}
// Nodes the overview should name before any inventory, in attention order.
// "2/3 nodes" on the cluster row says something is wrong but not which node;
// this list answers that.
export function getKubernetesNodesNeedingAttention(
model: KubernetesPageModel,
activeAlerts: Record<string, Alert> = {},
alertsEnabled = false,
): Resource[] {
return model.nodes
.filter((node) => rank(node) < 2)
.sort((a, b) => rank(a) - rank(b) || a.name.localeCompare(b.name));
.map((node) => ({
node,
rank: getKubernetesNodeAttentionRank(node, activeAlerts, alertsEnabled),
}))
.filter((entry) => entry.rank < 2)
.sort((a, b) => a.rank - b.rank || compareKubernetesNodes(a.node, b.node))
.map((entry) => entry.node);
}
export function buildKubernetesOverviewPosture(
frontend-modern/src/features/patrol/PatrolAttentionWorkbench.tsx
+18 -4
View File
@@ -68,6 +68,7 @@ import { copyToClipboard } from '@/utils/clipboard';
import type { PatrolAutonomyLevel } from '@/api/patrol';
import {
getAttentionFlappingPresentation,
getAttentionHandoverClose,
getLinkedPatrolFindings,
getPatrolRememberedDecisionPresentation,
partitionPatrolAttention,
@@ -971,7 +972,10 @@ function AttentionDetail(props: {
<div class="space-y-5 px-4 py-4 sm:px-5">
<section>
<div class="flex flex-wrap items-center gap-x-2 gap-y-1 text-xs text-muted">
<StateBadge item={loaded().item} />
<StateBadge
item={loaded().item}
movedClose={Boolean(getAttentionHandoverClose(loaded()))}
/>
<span aria-hidden="true">·</span>
<EvidenceLabel item={loaded().item} detail />
<span aria-hidden="true">·</span>
@@ -984,6 +988,9 @@ function AttentionDetail(props: {
<p class="mt-3 text-sm leading-6 text-base-content">
{loaded().item.plainLanguageSummary}
</p>
<Show when={getAttentionHandoverClose(loaded())}>
{(handover) => <p class="mt-1 text-xs leading-5 text-muted">{handover()}</p>}
</Show>
<Show when={getAttentionFlappingPresentation(loaded().item.flapping)}>
{(flapping) => (
<p class="mt-2 flex items-start gap-2 rounded-md border border-amber-200 bg-amber-50/70 px-3 py-2 text-xs leading-5 text-amber-900 dark:border-amber-900/25 dark:bg-amber-950/30 dark:text-amber-200">
@@ -1793,10 +1800,17 @@ function severityTone(item: AttentionItem): MetadataBadgeTone {
}
}
function StateBadge(props: { item: AttentionItem }) {
// movedClose marks a resolved item whose condition moved to another owner: it
// did not recover, so it must not wear the green "Resolved" badge.
function StateBadge(props: { item: AttentionItem; movedClose?: boolean }) {
return (
<MetadataBadge tone={stateTone(props.item)} size="xs" shape="rounded" appearance="outline">
{formatLabel(props.item.state)}
<MetadataBadge
tone={props.movedClose ? 'info' : stateTone(props.item)}
size="xs"
shape="rounded"
appearance="outline"
>
{props.movedClose ? 'Moved' : formatLabel(props.item.state)}
</MetadataBadge>
);
}
frontend-modern/src/features/patrol/__tests__/PatrolAttentionWorkbench.test.tsx
+50
View File
@@ -988,6 +988,56 @@ describe('PatrolAttentionWorkbench', () => {
expect(screen.getByText('Show all 11 transitions')).toBeInTheDocument();
});
it('shows an alert handed to its Pulse agent as moved, not recovered', async () => {
const reason =
'Alert moved to pve1 (Host Agent). This is not a recovery: check the agent for the current reading.';
const moved = item({
id: 'record-moved',
operationalRecordId: 'record-moved',
subjectResourceId: 'pve1',
subjectResourceName: 'pve1',
subjectResourceType: 'node',
kind: 'memory',
title: 'Memory on pve1',
plainLanguageSummary: 'Memory usage at 95%',
severity: 'warning',
state: 'resolved',
protectionPosture: undefined,
});
const movedDetail = detail(moved);
movedDetail.timeline = [
...movedDetail.timeline,
{
id: 'transition-moved',
operationalRecordId: 'record-moved',
from: 'open',
to: 'resolved',
at: '2026-07-19T07:45:00Z',
cause: 'ownership_transferred',
causeKey: 'memory:pve1',
evidenceIds: ['evidence-handover'],
reason,
},
];
window.history.replaceState({}, '', '/patrol?attention=record-moved');
apiMocks.getList.mockResolvedValue(listResponse([], summary()));
apiMocks.getDetail.mockResolvedValue(movedDetail);
renderWorkbench();
const panel = await screen.findByRole('complementary', { name: 'pve1 · Memory' });
expect(await within(panel).findByText('Moved')).toBeInTheDocument();
expect(within(panel).queryByText('Resolved')).not.toBeInTheDocument();
// The account sits under the summary, outside the collapsed history.
const summaryLine = within(panel).getByText('Memory usage at 95%');
expect(summaryLine.nextElementSibling).toHaveTextContent(reason);
expect(summaryLine.nextElementSibling?.closest('details')).toBeNull();
fireEvent.click(within(panel).getByText('Evidence and history'));
expect(within(panel).getByText('Open to Resolved')).toBeInTheDocument();
expect(within(panel).getByText(/Ownership transferred/)).toBeInTheDocument();
expect(within(panel).queryByText(/Recovery evidence/)).not.toBeInTheDocument();
});
it('opens the canonical governed action review from an eligible attention item', async () => {
const actionOffer = {
targetResourceId: 'docker:host-1/container-1',
frontend-modern/src/features/patrol/__tests__/patrolHomePresentation.test.ts
+61 -1
View File
@@ -1,7 +1,8 @@
import { describe, expect, it } from 'vitest';
import type { PatrolObjective } from '@/api/patrol';
import type { AttentionItem } from '@/api/patrolAttention';
import type { AttentionItem, AttentionItemDetail } from '@/api/patrolAttention';
import {
getAttentionHandoverClose,
getPatrolAttentionDecisionReason,
getPatrolObjectiveProtectionSummary,
partitionPatrolAttention,
@@ -145,3 +146,62 @@ describe('patrol home presentation', () => {
});
});
});
describe('getAttentionHandoverClose', () => {
const transition = (
overrides: Partial<AttentionItemDetail['timeline'][number]>,
): AttentionItemDetail['timeline'][number] => ({
id: 'transition-1',
operationalRecordId: 'record-1',
from: 'open',
to: 'resolved',
at: '2026-08-14T08:00:00Z',
cause: 'recovery_evidence',
causeKey: 'memory:pve1',
evidenceIds: ['evidence-1'],
...overrides,
});
const reason =
'Alert moved to pve1 (Host Agent). This is not a recovery: check the agent for the current reading.';
it('returns the handover account only for a record closed by an ownership transfer', () => {
const resolved = attentionItem({ state: 'resolved' });
expect(
getAttentionHandoverClose({
item: resolved,
timeline: [transition({ cause: 'ownership_transferred', reason })],
}),
).toBe(reason);
expect(
getAttentionHandoverClose({
item: resolved,
timeline: [transition({ cause: 'ownership_transferred' })],
}),
).toBe('This alert moved to another owner when it closed. It did not recover.');
expect(getAttentionHandoverClose({ item: resolved, timeline: [transition({})] })).toBeNull();
});
it('follows the latest close, so a later recovery or reopen wins', () => {
const handover = transition({ cause: 'ownership_transferred', reason });
const reopened = transition({
id: 'transition-2',
from: 'resolved',
to: 'open',
at: '2026-08-14T09:00:00Z',
cause: 'detector_decision',
});
const recovered = transition({ id: 'transition-3', at: '2026-08-14T10:00:00Z' });
expect(
getAttentionHandoverClose({
item: attentionItem({ state: 'resolved' }),
timeline: [handover, reopened, recovered],
}),
).toBeNull();
expect(
getAttentionHandoverClose({
item: attentionItem({ state: 'open' }),
timeline: [handover, reopened],
}),
).toBeNull();
});
});
frontend-modern/src/features/patrol/patrolHomePresentation.ts
+21 -1
View File
@@ -1,5 +1,5 @@
import type { PatrolAutonomyLevel, PatrolObjective } from '@/api/patrol';
import type { AttentionFlapping, AttentionItem } from '@/api/patrolAttention';
import type { AttentionFlapping, AttentionItem, AttentionItemDetail } from '@/api/patrolAttention';
import type { UnifiedFinding } from '@/stores/aiIntelligence';
export interface PatrolAutonomyExperience {
@@ -207,6 +207,26 @@ export function getAttentionFlappingPresentation(
return getFlappingPresentation(flapping.transitionCount, flapping.windowHours);
}
/**
* A resolved item whose close handed the condition to another record, such as
* a node metric its Pulse agent took over, did not recover. Returns the
* account to show instead of a healthy "Resolved", or null for a recovery.
*/
export function getAttentionHandoverClose(
detail: Pick<AttentionItemDetail, 'item' | 'timeline'> | null | undefined,
): string | null {
if (!detail || detail.item.state !== 'resolved') return null;
let close: AttentionItemDetail['timeline'][number] | undefined;
for (const transition of detail.timeline ?? []) {
if (transition.to !== 'resolved') continue;
if (!close || Date.parse(transition.at) >= Date.parse(close.at)) close = transition;
}
if (close?.cause !== 'ownership_transferred') return null;
return (
close.reason?.trim() || 'This alert moved to another owner when it closed. It did not recover.'
);
}
const normalizeResourceKey = (value: string | undefined): string =>
(value ?? '').trim().toLowerCase();
frontend-modern/src/features/truenas/__tests__/TrueNASSystemsTable.test.tsx
+42 -1
View File
@@ -9,8 +9,10 @@ vi.mock('@/components/shared/responsive', () => ({
ResponsiveMetricCell: () => <div data-testid="responsive-metric-cell" />,
}));
const activeAlertsRef = vi.hoisted(() => ({ current: {} as Record<string, unknown> }));
vi.mock('@/contexts/appRuntime', () => ({
useWebSocket: () => ({ activeAlerts: {} as Record<string, never> }),
useWebSocket: () => ({ activeAlerts: activeAlertsRef.current }),
}));
vi.mock('@/stores/alertsActivation', () => ({
useAlertsActivation: () => ({
@@ -43,6 +45,7 @@ const makeSystem = (overrides: Partial<Resource> & Pick<Resource, 'id'>): Resour
afterEach(() => {
cleanup();
activeAlertsRef.current = {};
});
describe('TrueNASSystemsTable', () => {
@@ -107,4 +110,42 @@ describe('TrueNASSystemsTable', () => {
expect(container.querySelector('[data-truenas-system-row="truenas-healthy"]')).toBeNull();
expect(screen.getByText('1 of 2 systems')).toBeInTheDocument();
});
it("tints a system row for its pools' alerts, which carry the system hostname", () => {
// TrueNAS pool, dataset, disk and app alerts are keyed on the child
// resource and name the system in "node". The system's drawer states the
// same pool problem through its health issue, so the row matches by
// hostname as well as id; no producer keys a TrueNAS system's alerts on
// its "agent:" alias.
activeAlertsRef.current = {
pool: {
id: 'pool',
type: 'truenas-pool-health',
level: 'warning',
resourceId: 'storage-archive',
resourceName: 'archive',
node: 'truenas-main',
instance: 'TrueNAS',
message: 'Pool archive is DEGRADED',
value: 0,
threshold: 0,
startTime: '2026-10-06T10:00:00Z',
acknowledged: false,
},
};
const { container } = render(() => (
<TrueNASSystemsTable
systems={[makeSystem({ id: 'truenas-main' }), makeSystem({ id: 'truenas-backup' })]}
scope={[makeSystem({ id: 'truenas-main' }), makeSystem({ id: 'truenas-backup' })]}
emptyIcon={<span />}
emptyTitle="No systems"
emptyDescription="No systems"
showToolbar={false}
/>
));
const row = (id: string) => container.querySelector(`[data-truenas-system-row="${id}"]`);
expect(row('truenas-main')).toHaveClass('bg-yellow-50');
expect(row('truenas-backup')).not.toHaveClass('bg-yellow-50');
});
});
frontend-modern/src/pages/__tests__/Alerts.helpers.test.ts
+57
View File
@@ -100,6 +100,7 @@ import {
buildProjectedOverrides,
normalizeRawOverridesConfig,
} from '@/features/alerts/alertOverridesModel';
import { buildAlertHistoryItems } from '@/features/alerts/alertHistoryModel';
import {
getAlertIncidentAcknowledgedBadgeClass,
getAlertIncidentEventFilterActionButtonClass,
@@ -1320,3 +1321,59 @@ describe('Unified selector parity', () => {
);
});
});
describe('alert history close outcome', () => {
it('lists a node alert handed to its Pulse agent as moved, not resolved', () => {
const base = {
type: 'memory',
level: 'warning' as const,
resourceName: 'pve1',
node: 'pve1',
instance: 'pve1',
value: 95,
threshold: 85,
startTime: '2026-10-06T09:00:00Z',
lastSeen: '2026-10-06T11:00:00Z',
acknowledged: true,
};
const items = buildAlertHistoryItems({
activeAlerts: {},
alertHistory: [
{
...base,
id: 'pve1-memory',
resourceId: 'pve1',
message: 'Memory usage at 95%',
resolution: {
reason: 'moved_to_agent',
successorResourceId: 'agent-pve1',
successorName: 'pve1 (Host Agent)',
},
},
{
...base,
id: 'pve1-cpu',
resourceId: 'pve1',
message: 'CPU usage at 90%',
acknowledged: false,
},
],
getResource: () => undefined,
allResources: [],
now: Date.parse('2026-10-06T12:00:00Z'),
});
const moved = items.find((item) => item.id === 'pve1-memory');
// Acknowledgement does not hide the outcome: the condition went to the agent.
expect(moved?.status).toBe('moved');
expect(moved?.description).toBe('Memory usage at 95%');
expect(moved?.closeDetail).toBe(
'Moved to pve1 (Host Agent). This is not a recovery: check the agent for the current reading.',
);
const recovered = items.find((item) => item.id === 'pve1-cpu');
expect(recovered?.status).toBe('resolved');
expect(recovered).not.toHaveProperty('closeDetail');
expect(alertHistoryTableAlertRowSource).toContain('title={props.alert.closeDetail}');
});
});
frontend-modern/src/types/api.ts
+12
View File
@@ -1386,6 +1386,18 @@ export interface Alert {
latestTransition?: LifecycleTransition;
transitions?: LifecycleTransition[];
evidence?: EvidenceEnvelope[];
// Set only on a closed alert whose condition did not clear, such as a node
// metric handed to its Pulse agent. Absent for an ordinary recovery.
resolution?: AlertResolution;
}
// Why an alert closed without recovering (alerts.AlertResolution).
export interface AlertResolution {
reason: 'moved_to_agent' | (string & {});
successorResourceId?: string;
successorName?: string;
// The alert engine's one-line account; carried on state snapshots only.
summary?: string;
}
export interface ResolvedAlert extends Alert {
frontend-modern/src/types/operationalTrust.ts
+1
View File
@@ -97,6 +97,7 @@ export type TransitionCause =
| 'suppression'
| 'suppression_expired'
| 'recovery_evidence'
| 'ownership_transferred'
| 'collection_stale'
| 'collection_unknown';
frontend-modern/src/utils/__tests__/alertIncidentPresentation.test.ts
+25
View File
@@ -10,8 +10,10 @@ import {
ALERT_RESOURCE_INCIDENT_REFRESH_LABEL,
ALERT_RESOURCE_INCIDENT_SAVE_NOTE_LABEL,
ALERT_RESOURCE_INCIDENT_SAVING_NOTE_LABEL,
ALERT_HISTORY_MOVED_STATUS,
getAlertHistoryStatusPresentation,
getAlertIncidentLevelBadgeClass,
getAlertResolutionDetail,
getAlertIncidentTimelineDialogTitle,
getAlertHistoryInvestigationDialogTitle,
getAlertResourceIncidentAcknowledgedByLabel,
@@ -103,6 +105,29 @@ describe('alertIncidentPresentation', () => {
className: 'text-xs px-2 py-0.5 rounded-sm bg-surface-hover text-base-content',
rowClassName: '',
});
expect(getAlertHistoryStatusPresentation(ALERT_HISTORY_MOVED_STATUS)).toEqual({
label: 'moved to agent',
className:
'text-xs px-2 py-0.5 rounded-sm bg-blue-100 dark:bg-blue-900/25 text-blue-700 dark:text-blue-300',
rowClassName: '',
});
});
it('says a handed-over alert did not recover, and leaves recoveries alone', () => {
expect(getAlertResolutionDetail(undefined)).toBeUndefined();
expect(
getAlertResolutionDetail({ reason: 'moved_to_agent', successorName: 'pve1 (Host Agent)' }),
).toBe(
'Moved to pve1 (Host Agent). This is not a recovery: check the agent for the current reading.',
);
expect(getAlertResolutionDetail({ reason: 'moved_to_agent' })).toBe(
'Moved to its Pulse agent. This is not a recovery: check the agent for the current reading.',
);
// A future reason the UI has no copy for still shows the engine's account.
expect(
getAlertResolutionDetail({ reason: 'replaced', summary: 'Alert replaced by a new check.' }),
).toBe('Alert replaced by a new check.');
});
it('returns canonical resource incident loading and empty-state copy', () => {
frontend-modern/src/utils/alertIncidentPresentation.ts
+30
View File
@@ -1,3 +1,4 @@
import type { AlertResolution } from '@/types/api';
import { getAlertSeverityBadgeClass } from '@/utils/alertSeverityPresentation';
export type AlertIncidentLevel = 'warning' | 'critical' | string | null | undefined;
@@ -76,6 +77,24 @@ export function getAlertIncidentLevelBadgeClass(level: AlertIncidentLevel): stri
return getAlertSeverityBadgeClass(String(level ?? 'warning'));
}
// History status for an alert that closed because its Pulse agent took the
// metric over. The condition did not clear; the agent's own alert owns it.
export const ALERT_HISTORY_MOVED_STATUS = 'moved';
/**
* The account of a close that was not a recovery, for history rows and their
* tooltips. Returns undefined for an ordinary recovery so callers keep their
* "resolved" wording.
*/
export function getAlertResolutionDetail(resolution?: AlertResolution | null): string | undefined {
if (!resolution) return undefined;
if (resolution.reason === 'moved_to_agent') {
const successor = resolution.successorName?.trim() || 'its Pulse agent';
return `Moved to ${successor}. This is not a recovery: check the agent for the current reading.`;
}
return resolution.summary?.trim() || undefined;
}
export function getAlertHistoryStatusPresentation(
status?: string | null,
): AlertHistoryStatusPresentation {
@@ -99,6 +118,17 @@ export function getAlertHistoryStatusPresentation(
};
}
// Closed without recovering: neither a live problem on this row nor a
// healthy outcome, so it gets its own neutral-blue badge.
if (normalized === ALERT_HISTORY_MOVED_STATUS) {
return {
label: 'moved to agent',
className:
'text-xs px-2 py-0.5 rounded-sm bg-blue-100 dark:bg-blue-900/25 text-blue-700 dark:text-blue-300',
rowClassName: '',
};
}
return {
label: normalized || 'resolved',
className: 'text-xs px-2 py-0.5 rounded-sm bg-surface-hover text-base-content',
internal/ai/alert_adapter.go
+12
View File
@@ -2,6 +2,7 @@ package ai
import (
"fmt"
"strings"
"time"
"github.com/rcourtman/pulse-go-rewrite/internal/alerts"
@@ -56,6 +57,7 @@ func (a *AlertManagerAdapter) GetRecentlyResolved(minutes int) []ResolvedAlertIn
AlertInfo: convertAlertFromModels(&resolved.Alert),
ResolvedTime: resolved.ResolvedTime,
Duration: formatDuration(resolved.ResolvedTime.Sub(resolved.StartTime)),
Resolution: resolvedAlertResolution(resolved.Resolution),
}
result = append(result, info)
}
@@ -98,6 +100,7 @@ func (a *AlertManagerAdapter) GetAlertHistory(resourceID string, limit int) []Re
AlertInfo: convertAlertFromModels(&resolved.Alert),
ResolvedTime: resolved.ResolvedTime,
Duration: formatDuration(resolved.ResolvedTime.Sub(resolved.StartTime)),
Resolution: resolvedAlertResolution(resolved.Resolution),
}
result = append(result, info)
if len(result) >= limit {
@@ -245,3 +248,12 @@ func formatDuration(d time.Duration) string {
}
return fmt.Sprintf("%d days", days)
}
// resolvedAlertResolution is the one-line account of a close that was not a
// recovery, empty for an ordinary resolve.
func resolvedAlertResolution(resolution *models.AlertResolution) string {
if resolution == nil {
return ""
}
return strings.TrimSpace(resolution.Summary)
}
internal/ai/alert_provider.go
+9
View File
@@ -31,6 +31,9 @@ type ResolvedAlertInfo struct {
AlertInfo
ResolvedTime time.Time `json:"resolved_time"`
Duration string `json:"total_duration"` // How long the alert lasted
// Resolution is set only when the alert closed without recovering, such
// as a node metric handed to its Pulse agent.
Resolution string `json:"resolution,omitempty"`
}
// AlertProvider provides access to the current alert state
@@ -131,6 +134,12 @@ func (s *Service) buildAlertContext() string {
}
for i := 0; i < limit; i++ {
a := recentlyResolved[i]
if resolution := strings.TrimSpace(a.Resolution); resolution != "" {
sections = append(sections, fmt.Sprintf("- **%s** on %s: %s (lasted %s, closed %s ago) %s",
a.Type, a.ResourceName, a.Message, a.Duration,
formatTimeAgo(a.ResolvedTime), resolution))
continue
}
sections = append(sections, fmt.Sprintf("- **%s** on %s: %s (lasted %s, resolved %s ago)",
a.Type, a.ResourceName, a.Message, a.Duration,
formatTimeAgo(a.ResolvedTime)))
internal/ai/alert_provider_test.go
+57
View File
@@ -5,6 +5,8 @@ import (
"strings"
"testing"
"time"
"github.com/rcourtman/pulse-go-rewrite/internal/models"
)
type stubAlertProvider struct {
@@ -151,6 +153,61 @@ func TestService_buildAlertContext_NoActiveWithResolved(t *testing.T) {
}
}
// The assistant reads recently resolved alerts through the adapter, so a node
// alert handed to its Pulse agent must reach the prompt as a handover and not
// as "resolved", which the model would read as the node being healthy.
func TestService_buildAlertContext_HandoverIsNotARecovery(t *testing.T) {
now := time.Now()
summary := "Alert moved to pve1 (Host Agent). This is not a recovery: check the agent for the current reading."
adapter := NewAlertManagerAdapter(&stubAlertManager{resolved: []models.ResolvedAlert{
{
Alert: models.Alert{
ID: "pve1-memory",
Type: "memory",
Level: "warning",
ResourceID: "pve1",
ResourceName: "pve1",
Message: "Memory usage at 95%",
StartTime: now.Add(-time.Hour),
Resolution: &models.AlertResolution{Reason: "moved_to_agent", Summary: summary},
},
ResolvedTime: now.Add(-time.Minute),
},
{
Alert: models.Alert{
ID: "pve2-cpu",
Type: "cpu",
Level: "warning",
ResourceID: "pve2",
ResourceName: "pve2",
Message: "CPU usage at 90%",
StartTime: now.Add(-time.Hour),
},
ResolvedTime: now.Add(-time.Minute),
},
}})
resolved := adapter.GetRecentlyResolved(30)
if len(resolved) != 2 || resolved[0].Resolution != summary || resolved[1].Resolution != "" {
t.Fatalf("GetRecentlyResolved = %+v, want only the handover to carry a resolution", resolved)
}
if history := adapter.GetAlertHistory("pve1", 5); len(history) != 1 || history[0].Resolution != summary {
t.Fatalf("GetAlertHistory = %+v, want the handover resolution", history)
}
s := &Service{}
s.SetAlertProvider(adapter)
ctx := s.buildAlertContext()
if !strings.Contains(ctx, "- **memory** on pve1: Memory usage at 95% (lasted ") ||
!strings.Contains(ctx, ", closed 1 minute ago) "+summary) {
t.Fatalf("expected the handover line, got: %s", ctx)
}
if !strings.Contains(ctx, "- **cpu** on pve2: CPU usage at 90% (lasted ") ||
!strings.Contains(ctx, ", resolved 1 minute ago)") {
t.Fatalf("expected the ordinary recovery line, got: %s", ctx)
}
}
func TestService_buildTargetAlertContext(t *testing.T) {
s := &Service{}
s.SetAlertProvider(&stubAlertProvider{
internal/ai/memory/incidents.go
+20 -2
View File
@@ -474,6 +474,17 @@ func (s *IncidentStore) RecordAlertUnacknowledged(alert *alerts.Alert, user stri
s.saveAsync()
}
// alertResolvedEventSummary is the timeline line for an alert's close: the
// handover account when it closed without recovering, else "Alert resolved".
func alertResolvedEventSummary(alert *alerts.Alert) string {
if alert != nil {
if summary := alert.Resolution.Summary(); summary != "" {
return summary
}
}
return "Alert resolved"
}
// RecordAlertResolved records a resolved event and closes the incident.
func (s *IncidentStore) RecordAlertResolved(alert *alerts.Alert, resolvedAt time.Time) {
if alert == nil {
@@ -507,7 +518,7 @@ func (s *IncidentStore) RecordAlertResolved(alert *alerts.Alert, resolvedAt time
shell.OccurrenceClosedAt = cloneTime(resolvedAt)
if !s.projectsFromCanonicalLocked() {
s.addEventAtLocked(shell, IncidentEventAlertResolved, resolvedAt, "Alert resolved", map[string]interface{}{
s.addEventAtLocked(shell, IncidentEventAlertResolved, resolvedAt, alertResolvedEventSummary(alert), map[string]interface{}{
"resolved_at": resolvedAt.Format(time.RFC3339),
})
}
@@ -563,7 +574,7 @@ func (s *IncidentStore) EnsureAlertOccurrence(alert *alerts.Alert, resolvedAt *t
if resolvedAt != nil && !resolvedAt.IsZero() && (shell.OccurrenceRefiredAt == nil || !resolvedAt.Before(*shell.OccurrenceRefiredAt)) {
shell.OccurrenceClosedAt = cloneTime(*resolvedAt)
if !hasIncidentEventType(shell.Events, IncidentEventAlertResolved) {
s.addEventAtLocked(shell, IncidentEventAlertResolved, *resolvedAt, "Alert resolved", map[string]interface{}{
s.addEventAtLocked(shell, IncidentEventAlertResolved, *resolvedAt, alertResolvedEventSummary(alert), map[string]interface{}{
"resolved_at": resolvedAt.Format(time.RFC3339),
"projection_source": incidentSnapshotSource,
})
@@ -1113,6 +1124,13 @@ func incidentEventSummaryFromChange(change unifiedresources.ResourceChange, even
case IncidentEventAlertUnsnoozed:
return "Alert notifications resumed"
case IncidentEventAlertResolved:
// A handover closes the alert without recovering; its change reason
// says so and where the condition went.
if _, ok := stringMetadata(change.Metadata, unifiedresources.MetadataAlertResolution); ok {
if reason := strings.TrimSpace(change.Reason); reason != "" {
return reason
}
}
return "Alert resolved"
default:
if summary := strings.TrimSpace(change.Reason); summary != "" {
internal/ai/memory/incidents_test.go
+33
View File
@@ -239,6 +239,39 @@ func TestIncidentStore_EqualTimestampLifecycleKeepsResolvedState(t *testing.T) {
}
}
// Without a canonical resource timeline the store writes its own close event,
// which must say the alert moved rather than resolved.
func TestIncidentStore_HandoverCloseIsNotARecovery(t *testing.T) {
store := NewIncidentStore(IncidentStoreConfig{MaxIncidents: 10, MaxEventsPerIncident: 10, MaxAgeDays: 30})
startedAt := time.Now().UTC().Truncate(time.Second).Add(-time.Hour)
movedAt := startedAt.Add(30 * time.Minute)
alert := &alerts.Alert{
ID: "pve1-memory",
Type: "memory",
ResourceID: "pve1",
ResourceName: "pve1",
Message: "Memory usage at 95%",
StartTime: startedAt,
LastSeen: startedAt,
Resolution: &alerts.AlertResolution{Reason: alerts.AlertResolutionMovedToAgent, SuccessorName: "pve1 (Host Agent)"},
}
summary := "Alert moved to pve1 (Host Agent). This is not a recovery: check the agent for the current reading."
store.RecordAlertFired(alert)
store.RecordAlertResolved(alert, movedAt)
timeline := store.GetTimelineByAlertAt(alert.ID, startedAt)
if timeline == nil || len(timeline.Events) != 2 || timeline.Events[1].Summary != summary {
t.Fatalf("recorded timeline = %#v, want the handover close", timeline)
}
snapshot := alert.Clone()
snapshot.ID = "pve1-memory-imported"
repaired := store.EnsureAlertOccurrence(snapshot, &movedAt)
if repaired == nil || len(repaired.Events) != 2 || repaired.Events[1].Summary != summary {
t.Fatalf("repaired timeline = %#v, want the handover close", repaired)
}
}
func TestIncidentStore_ProjectsCanonicalTimelineWhenAttached(t *testing.T) {
store := NewIncidentStore(IncidentStoreConfig{
MaxIncidents: 10,
internal/ai/patrol_ai.go
+6
View File
@@ -3499,6 +3499,12 @@ func (p *PatrolService) seedHealthAndAlertsState(snap patrolRuntimeState, scoped
sb.WriteString("# Recently Resolved Alerts\n")
for _, r := range alerts {
ago := seedFormatTimeAgo(now, r.ResolvedTime)
// A handover closes the alert without the condition clearing, so
// it must not read as the resource having recovered.
if r.Resolution != nil && strings.TrimSpace(r.Resolution.Summary) != "" {
sb.WriteString(fmt.Sprintf("- %s — closed %s. %s\n", seedAlertSubject(r.Alert), ago, strings.TrimSpace(r.Resolution.Summary)))
continue
}
sb.WriteString(fmt.Sprintf("- %s — resolved %s\n", seedAlertSubject(r.Alert), ago))
}
sb.WriteString("\n")
internal/ai/patrol_ai_more_test.go
+36
View File
@@ -922,6 +922,42 @@ func TestSeedHealthAndAlerts_NoIssues(t *testing.T) {
}
}
// Patrol's seed must not tell the model a node recovered when its alert only
// moved to the node's Pulse agent.
func TestSeedHealthAndAlerts_HandoverIsNotARecovery(t *testing.T) {
ps := NewPatrolService(nil, nil)
cfg := DefaultPatrolConfig()
now := time.Now()
summary := "Alert moved to pve1 (Host Agent). This is not a recovery: check the agent for the current reading."
state := models.StateSnapshot{
RecentlyResolved: []models.ResolvedAlert{
{
Alert: models.Alert{
ResourceName: "pve1",
Message: "Memory usage at 95%",
Resolution: &models.AlertResolution{Reason: "moved_to_agent", Summary: summary},
},
ResolvedTime: now.Add(-3 * time.Minute),
},
{
Alert: models.Alert{ResourceName: "pve2", Message: "CPU usage at 90%"},
ResolvedTime: now.Add(-5 * time.Minute),
},
},
}
out := ps.seedHealthAndAlertsState(patrolRuntimeStateForTest(ps, state), nil, cfg, now)
if !strings.Contains(out, "- pve1: Memory usage at 95% — closed 3m ago. "+summary+"\n") {
t.Fatalf("expected the handover line, got: %s", out)
}
if strings.Contains(out, "pve1: Memory usage at 95% — resolved") {
t.Fatalf("handover reported as a recovery: %s", out)
}
if !strings.Contains(out, "- pve2: CPU usage at 90% — resolved 5m ago\n") {
t.Fatalf("expected the ordinary recovery line, got: %s", out)
}
}
func TestSeedHealthAndAlerts_UnknownDiskHealthDoesNotClaimSMARTPassed(t *testing.T) {
ps := NewPatrolService(nil, nil)
cfg := DefaultPatrolConfig()
internal/ai/tools/data_types.go
+5
View File
@@ -2364,4 +2364,9 @@ type ResolvedAlertSummary struct {
Threshold float64 `json:"threshold,omitempty"`
StartTime time.Time `json:"start_time"`
ResolvedTime time.Time `json:"resolved_time"`
// Resolution is set only when the alert closed without recovering, such
// as a node metric handed to its Pulse agent; the successor resource
// owns the condition from then on.
Resolution string `json:"resolution,omitempty"`
SuccessorResourceID string `json:"successor_resource_id,omitempty"`
}
internal/ai/tools/tools_alerts.go
+8 -3
View File
@@ -19,7 +19,7 @@ func (e *PulseToolExecutor) registerAlertsTools() {
Actions:
- list: List active threshold alerts (CPU > 80%, disk full, etc.)
- findings: List AI patrol findings (detected issues)
- resolved: List recently resolved alerts
- resolved: List recently resolved alerts; one with a resolution closed without recovering
- resolve: Mark a finding as resolved
- dismiss: Dismiss a finding as not an issue
@@ -417,7 +417,7 @@ func (e *PulseToolExecutor) executeListResolvedAlerts(_ context.Context, args ma
break
}
alerts = append(alerts, ResolvedAlertSummary{
summary := ResolvedAlertSummary{
ID: alert.ID,
Type: alert.Type,
Level: alert.Level,
@@ -430,7 +430,12 @@ func (e *PulseToolExecutor) executeListResolvedAlerts(_ context.Context, args ma
Threshold: alert.Threshold,
StartTime: alert.StartTime,
ResolvedTime: alert.ResolvedTime,
})
}
if resolution := alert.Resolution; resolution != nil {
summary.Resolution = resolution.Summary
summary.SuccessorResourceID = resolution.SuccessorResourceID
}
alerts = append(alerts, summary)
}
if alerts == nil {
internal/ai/tools/tools_metrics_alerts_test.go
+43
View File
@@ -215,6 +215,49 @@ func TestExecuteListResolvedAlerts(t *testing.T) {
if len(resp.Alerts) != 1 || resp.Alerts[0].ID != "a1" {
t.Fatalf("unexpected resolved alerts: %+v", resp)
}
if resp.Alerts[0].Resolution != "" || resp.Alerts[0].SuccessorResourceID != "" {
t.Fatalf("a recovery must not carry a resolution: %+v", resp.Alerts[0])
}
}
// A node alert handed to its Pulse agent closes without recovering, so the
// assistant must see that, and where the condition went, instead of a plain
// resolve it would read as the node being healthy again.
func TestExecuteListResolvedAlertsReportsHandover(t *testing.T) {
alertProv := &mockAlertProvider{}
executor := NewPulseToolExecutor(ExecutorConfig{})
executor.alertProvider = alertProv
now := time.Now()
summary := "Alert moved to pve1 (Host Agent). This is not a recovery: check the agent for the current reading."
alertProv.On("GetRecentlyResolved", 24*60).Return([]models.ResolvedAlert{{
Alert: models.Alert{
ID: "pve1-memory",
Type: "memory",
Level: "warning",
ResourceID: "pve1",
ResourceName: "pve1",
Message: "Memory usage at 95%",
StartTime: now.Add(-time.Hour),
Resolution: &models.AlertResolution{
Reason: "moved_to_agent",
SuccessorResourceID: "agent-pve1",
SuccessorName: "pve1 (Host Agent)",
Summary: summary,
},
},
ResolvedTime: now,
}})
result, _ := executor.executeListResolvedAlerts(context.Background(), map[string]interface{}{})
var resp ResolvedAlertsResponse
if err := json.Unmarshal([]byte(result.Content[0].Text), &resp); err != nil {
t.Fatalf("decode resolved alerts: %v", err)
}
if len(resp.Alerts) != 1 ||
resp.Alerts[0].Resolution != summary ||
resp.Alerts[0].SuccessorResourceID != "agent-pve1" {
t.Fatalf("resolved alerts = %+v, want the handover and its successor", resp.Alerts)
}
}
func TestExecuteListAlertsAndFindings(t *testing.T) {
internal/alerts/active_lifecycle.go
+7
View File
@@ -98,6 +98,12 @@ func (m *Manager) addRecentlyResolvedWithPrimaryLock(resolved *ResolvedAlert) {
// clearAlert removes an alert if it exists.
func (m *Manager) clearAlert(alertID string) {
m.clearAlertWithResolution(alertID, nil)
}
// clearAlertWithResolution removes an alert if it exists. A non-nil
// resolution marks the close as something other than a recovery.
func (m *Manager) clearAlertWithResolution(alertID string, resolution *AlertResolution) {
m.mu.Lock()
alert, exists := m.getActiveAlertNoLock(alertID)
if exists {
@@ -111,6 +117,7 @@ func (m *Manager) clearAlert(alertID string) {
}
publicID := effectiveAlertID(alert, alertID)
alert.Resolution = resolution.Clone()
resolvedAlert := m.newResolvedAlert(alert, time.Now(), nil)
m.addRecentlyResolvedUnlocked(resolvedAlert)
internal/alerts/callbacks.go
+7 -1
View File
@@ -430,7 +430,13 @@ func (m *Manager) safeCallResolvedAlertCallback(resolved *ResolvedAlert, fallbac
}
// Record the resolution before the callback guard: this funnel is the
// resolve seam for every lifecycle path, with or without subscribers.
m.recordAlertEvent(eventlog.TypeResolved, alert, fallbackID, "", "Alert resolved.", nil)
eventReason, eventMessage := "", "Alert resolved."
if alert != nil {
if summary := alert.Resolution.Summary(); summary != "" {
eventReason, eventMessage = string(alert.Resolution.Reason), summary
}
}
m.recordAlertEvent(eventlog.TypeResolved, alert, fallbackID, eventReason, eventMessage, nil)
callbacks := m.getResolvedCallbacks()
resolvedCallback := m.callbacks.resolvedAlertCallback()
internal/alerts/canonical_metric.go
+25 -13
View File
@@ -76,6 +76,30 @@ func formatMetricValue(value float64, suffix string) string {
return fmt.Sprintf("%.1f%s", value, suffix)
}
// releaseCanonicalMetricAlert stops evaluating a metric spec the way a
// disabled threshold does: any pending run is dropped and any open alert is
// resolved. A non-nil resolution says why the close was not a recovery, and
// travels with the resolved alert to every consumer.
func (m *Manager) releaseCanonicalMetricAlert(spec alertspecs.ResourceAlertSpec, resourceName, node, instance, resourceType string, value float64, resolution *AlertResolution) {
if spec.MetricThreshold == nil {
return
}
storageKey := canonicalTrackingKeyForSpec(spec, spec.ID)
m.mu.Lock()
m.core.ApplyMetric(reducer.MetricSignal{
ResourceID: spec.ResourceID,
Key: spec.ID,
Metric: spec.MetricThreshold.Metric,
Value: value,
ObservedAt: time.Now(),
}, reducer.MetricRule{})
m.mu.Unlock()
// A guest that moved nodes may hold this alert under its old node-scoped
// identity; re-home it first so the clear below can resolve it.
m.rehomeStrandedGuestAlert(storageKey, spec.ID, string(spec.Kind), spec.ResourceID, resourceName, node, instance, resourceType)
m.clearAlertWithResolution(storageKey, resolution)
}
func (m *Manager) evaluateCanonicalMetricAlert(spec alertspecs.ResourceAlertSpec, resourceName, node, instance, resourceType string, value float64, threshold *HysteresisThreshold, opts *metricOptions) {
if spec.MetricThreshold == nil {
return
@@ -86,19 +110,7 @@ func (m *Manager) evaluateCanonicalMetricAlert(spec alertspecs.ResourceAlertSpec
trackingKey := storageKey
metricType := spec.MetricThreshold.Metric
if spec.Disabled || spec.MetricThreshold.Trigger <= 0 {
m.mu.Lock()
m.core.ApplyMetric(reducer.MetricSignal{
ResourceID: spec.ResourceID,
Key: spec.ID,
Metric: metricType,
Value: value,
ObservedAt: time.Now(),
}, reducer.MetricRule{})
m.mu.Unlock()
// A guest that moved nodes may hold this alert under its old node-scoped
// identity; re-home it first so the clear below can resolve it.
m.rehomeStrandedGuestAlert(storageKey, spec.ID, string(spec.Kind), spec.ResourceID, resourceName, node, instance, resourceType)
m.clearAlert(storageKey)
m.releaseCanonicalMetricAlert(spec, resourceName, node, instance, resourceType, value, nil)
return
}
internal/alerts/config_runtime.go
+6 -4
View File
@@ -589,12 +589,14 @@ func (m *Manager) reevaluateActiveAlertsLocked() {
continue
}
// buildCanonicalMetricSpec keeps a clear level only below the trigger,
// and without one the evaluator keeps firing at the trigger itself.
// A clear level at or above the trigger must not resolve a reading
// the next evaluation raises again.
clearThreshold := threshold.Clear
if clearThreshold <= 0 {
clearThreshold = threshold.Trigger
}
hasRecoveryBand := clearThreshold > 0 && clearThreshold < threshold.Trigger
if alert.Value <= clearThreshold {
if hasRecoveryBand && alert.Value <= clearThreshold {
alertsToResolve = append(alertsToResolve, alertID)
log.Info().
Str("alertID", alertID).
internal/alerts/history.go
+1
View File
@@ -394,6 +394,7 @@ func (hm *HistoryManager) UpdateAlertOperationalContractForAlert(alert *Alert) {
hm.history[i].Alert.LatestTransition = snapshot.LatestTransition
hm.history[i].Alert.Transitions = snapshot.Transitions
hm.history[i].Alert.Evidence = snapshot.Evidence
hm.history[i].Alert.Resolution = snapshot.Resolution
// The live alert is authoritative for acknowledgement; a row
// that keeps its fire-time ack state shows an acknowledged
// alert as unacknowledged for its whole active life.
internal/alerts/history_test.go
+61
View File
@@ -7,7 +7,9 @@ import (
"testing"
"time"
"github.com/rcourtman/pulse-go-rewrite/internal/models"
"github.com/rcourtman/pulse-go-rewrite/internal/operationaltrust"
"github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources"
)
// newTestHistoryManager creates a HistoryManager using a temp directory
@@ -1058,3 +1060,62 @@ func TestSaveHistoryWithRetry_RestoresBackupOnFailure(t *testing.T) {
_ = hm
}
// The state snapshot, the websocket and the assistant read recently resolved
// alerts through GetRecentlyResolved, so a handover must reach them as one
// and not as a recovery.
func TestRecentlyResolvedCarriesHandoverResolution(t *testing.T) {
manager := newUnifiedEvalParityManager(t)
manager.UpdateConfig(AlertConfig{
Enabled: true,
NodeDefaults: ThresholdConfig{
CPU: &HysteresisThreshold{Trigger: 80, Clear: 75},
Memory: &HysteresisThreshold{Trigger: 85, Clear: 80},
},
})
disableTestTimeThresholds(manager)
node := models.Node{
ID: "pve-1",
Name: "pve-1",
Instance: "test",
Status: "online",
CPU: 0.95,
Memory: models.Memory{Total: 8 << 30, Used: 7 << 30, Usage: 95},
}
manager.CheckNode(node)
release := func(metric string, resolution *AlertResolution) {
t.Helper()
spec, err := buildCanonicalMetricSpec(node.ID, node.Name, unifiedresources.ResourceType("node"), metric, nil)
if err != nil {
t.Fatalf("build %s spec: %v", metric, err)
}
manager.releaseCanonicalMetricAlert(spec, node.Name, node.Name, node.Instance, "node", 0, resolution)
}
release("memory", &AlertResolution{
Reason: AlertResolutionMovedToAgent,
SuccessorResourceID: "agent-pve-1",
SuccessorName: "pve-1 (Host Agent)",
})
release("cpu", nil)
byType := map[string]models.ResolvedAlert{}
for _, resolved := range manager.GetRecentlyResolved() {
byType[resolved.Type] = resolved
}
moved, ok := byType["memory"]
if !ok {
t.Fatalf("recently resolved = %+v, want the moved memory alert", byType)
}
want := models.AlertResolution{
Reason: "moved_to_agent",
SuccessorResourceID: "agent-pve-1",
SuccessorName: "pve-1 (Host Agent)",
Summary: "Alert moved to pve-1 (Host Agent). This is not a recovery: check the agent for the current reading.",
}
if moved.Resolution == nil || *moved.Resolution != want {
t.Fatalf("moved resolution = %+v, want %+v", moved.Resolution, want)
}
if recovered, ok := byType["cpu"]; !ok || recovered.Resolution != nil {
t.Fatalf("recovered cpu alert = %+v, want an ordinary resolve with no resolution", recovered)
}
}
internal/alerts/host.go
+165 -16
View File
@@ -157,7 +157,22 @@ func (m *Manager) resolveHostAlertThresholdsNoLock(alert *Alert, resourceID stri
linkedContainerID = metadataStringValue(alert.Metadata, "linkedContainerId")
}
return m.resolveHostThresholdsNoLock(hostID, linkedNodeID, linkedVMID, linkedContainerID)
thresholds := m.resolveHostThresholdsNoLock(hostID, linkedNodeID, linkedVMID, linkedContainerID)
// A disk temperature alert is judged against the per-type threshold
// CheckHost evaluates for that disk, so a config save does not resolve an
// alert the next report would raise again.
if alert != nil && alert.Type == "diskTemperature" {
override, exists := m.hostThresholdOverrideNoLock(hostID, linkedNodeID, linkedVMID, linkedContainerID)
overridden := exists && override.DiskTemperature != nil
if diskType, known := alert.Metadata["diskType"].(string); known || overridden {
thresholds.DiskTemperature = m.hostDiskTemperatureThresholdNoLock(thresholds.DiskTemperature, overridden, diskType)
} else {
// Alerts persisted before CheckHost recorded diskType carry no
// disk type until their next firing evaluation.
thresholds.DiskTemperature = m.lowestHostDiskTemperatureThresholdNoLock(thresholds.DiskTemperature)
}
}
return thresholds
}
func sanitizeHostComponent(value string) string {
@@ -363,21 +378,21 @@ func (m *Manager) CheckHost(host models.Host) {
}
if thresholds.DiskTemperature != nil && thresholds.DiskTemperature.Trigger > 0 {
// An empty SMART list means collection failed or is unsupported for
// this report, not that every disk left, so existing alerts are held.
if len(host.Sensors.SMART) > 0 {
seenDiskTemps := make(map[string]struct{}, len(host.Sensors.SMART))
for _, disk := range host.Sensors.SMART {
// A listed disk in standby, or without a temperature after a
// failed probe, is still present. Its alert holds until a fresh
// reading resolves it instead of clearing and re-raising.
tempResourceID := hostDiskTemperatureResourceID(host.ID, disk.Device)
seenDiskTemps[tempResourceID] = struct{}{}
if disk.Temperature > 0 && !disk.Standby {
effectiveTempThreshold := thresholds.DiskTemperature
if diskType := strings.ToLower(strings.TrimSpace(disk.Type)); diskType != "" && !diskTempOverridden {
m.mu.RLock()
if th, ok := m.config.DiskTempByType[diskType]; ok {
t := th
effectiveTempThreshold = &t
}
m.mu.RUnlock()
}
m.mu.RLock()
effectiveTempThreshold := m.hostDiskTemperatureThresholdNoLock(thresholds.DiskTemperature, diskTempOverridden, disk.Type)
m.mu.RUnlock()
// Use specific resource ID for the disk: hostID/disk-temp:device
tempResourceID := fmt.Sprintf("%s/disk_temp:%s", hostResourceID(host.ID), sanitizeHostComponent(disk.Device))
tempResourceName := fmt.Sprintf("%s (%s Temp)", hostDisplayName(host), disk.Device)
diskTempMetadata := hostChildAlertMetadata(baseMetadata, hostDiskAlertResourceType)
@@ -400,12 +415,14 @@ func (m *Manager) CheckHost(host models.Host) {
m.checkMetricWithCanonicalSpec(spec, tempResourceName, nodeName, disk.Device, "agent", float64(disk.Temperature), effectiveTempThreshold, &metricOptions{Metadata: diskTempMetadata})
}
}
// A disk missing from consecutive non-empty reports was removed,
// replaced or renamed, so no later reading will resolve its alert.
m.cleanupHostDiskTemperatureAlerts(host.ID, seenDiskTemps)
}
} else {
// We can't easily clear all disk temp alerts without tracking them,
// but checkMetric logic handles auto-resolution if value drops.
// If feature is disabled, ideally we should clear existing alerts.
// For now simple implementation.
// Disk temperature alerting is off for this host, so no later reading
// will resolve an alert it raised while it was on.
m.clearHostDiskTemperatureAlerts(host.ID)
}
seenDisks := make(map[string]struct{}, len(host.Disks))
@@ -671,6 +688,9 @@ func (m *Manager) HandleHostRemoved(host models.Host) {
m.clearHostRAIDAlerts(host.ID)
m.clearHostUnraidAlerts(host.ID)
m.clearHostCustomSensorAlerts(host.ID)
// No later report will close the host's pending disk temperature runs or
// reset its disk absence counts.
m.clearHostDiskTemperatureAlerts(host.ID)
}
// HandleHostTelemetryExpired re-evaluates transient storage-operation evidence
@@ -875,6 +895,7 @@ func (m *Manager) HandleHostOfflineWithCorrelation(host models.Host, correlation
diskResourcePrefixes := []string{
fmt.Sprintf("%s/disk:", resourceKey),
hostDiskTemperatureResourcePrefix(host.ID),
}
raidAlertPrefix := fmt.Sprintf("host-%s-raid-", host.ID)
var alertsToClear []string
@@ -931,6 +952,7 @@ func (m *Manager) clearHostDiskAlerts(hostID string) {
prefixes := []string{
fmt.Sprintf("%s/disk:", hostResourceID(hostID)),
hostDiskTemperatureResourcePrefix(hostID),
}
m.mu.Lock()
@@ -955,6 +977,133 @@ func (m *Manager) clearHostDiskAlerts(hostID string) {
}
}
// hostDiskTemperatureResourcePrefix is the resource ID prefix of every SMART
// disk temperature alert CheckHost raises for a host.
func hostDiskTemperatureResourcePrefix(hostID string) string {
return hostResourceID(hostID) + "/disk_temp:"
}
// hostDiskTemperatureResourceID is the resource ID of the SMART disk
// temperature alert for one device on a host.
func hostDiskTemperatureResourceID(hostID, device string) string {
return hostDiskTemperatureResourcePrefix(hostID) + sanitizeHostComponent(device)
}
// hostDiskTemperatureThresholdNoLock returns the threshold for one SMART disk
// given the host's resolved disk temperature threshold. An enabled threshold
// gives way to the disk type's DiskTempByType entry unless an explicit host or
// linked-resource override set it. Callers must hold m.mu.
func (m *Manager) hostDiskTemperatureThresholdNoLock(hostThreshold *HysteresisThreshold, overridden bool, diskType string) *HysteresisThreshold {
if hostThreshold == nil || hostThreshold.Trigger <= 0 || overridden {
return hostThreshold
}
if diskType = strings.ToLower(strings.TrimSpace(diskType)); diskType != "" {
if th, ok := m.config.DiskTempByType[diskType]; ok {
return &th
}
}
return hostThreshold
}
// lowestHostDiskTemperatureThresholdNoLock returns the lowest enabled
// threshold any SMART disk of the host can be evaluated against. An alert
// whose disk type is unknown is judged against it, so a config save never
// resolves an alert its disk type would still fire. Callers must hold m.mu.
func (m *Manager) lowestHostDiskTemperatureThresholdNoLock(hostThreshold *HysteresisThreshold) *HysteresisThreshold {
if hostThreshold == nil || hostThreshold.Trigger <= 0 {
return hostThreshold
}
lowest := hostThreshold
for _, th := range m.config.DiskTempByType {
if th.Trigger > 0 && th.Trigger < lowest.Trigger {
t := th
lowest = &t
}
}
return lowest
}
// clearHostDiskTemperatureAlerts resolves every SMART disk temperature alert
// for a host.
func (m *Manager) clearHostDiskTemperatureAlerts(hostID string) {
m.cleanupHostDiskTemperatureAlerts(hostID, nil)
}
// hostDiskTemperatureAbsenceConfirmations is how many consecutive non-empty
// SMART reports must omit a disk before its temperature alert clears. The
// Windows, FreeBSD and controller-multiplexed Linux collectors drop a disk
// whose probe fails, so one omission is not evidence the disk left.
const hostDiskTemperatureAbsenceConfirmations = 3
// cleanupHostDiskTemperatureAlerts resolves a host's SMART disk temperature
// alerts, and drops their pending threshold runs, once their resource ID has
// been missing from seen for hostDiskTemperatureAbsenceConfirmations calls in
// a row. A nil seen resolves all of them at once.
func (m *Manager) cleanupHostDiskTemperatureAlerts(hostID string, seen map[string]struct{}) {
if hostID == "" {
return
}
resourcePrefix := hostDiskTemperatureResourcePrefix(hostID)
m.mu.Lock()
defer m.mu.Unlock()
// Absent resources map to the storage keys of their active alerts; a
// resource with only a pending run maps to none.
absent := make(map[string][]string)
for storageKey, alert := range m.activeAlerts {
// Only CheckHost raises alerts under this prefix. Matching on it
// alone also catches alerts raised before the type became
// diskTemperature ("disk_temperature", January 2026).
if alert == nil || !strings.HasPrefix(alert.ResourceID, resourcePrefix) {
continue
}
if _, exists := seen[alert.ResourceID]; exists {
continue
}
absent[alert.ResourceID] = append(absent[alert.ResourceID], storageKey)
}
for _, resourceID := range m.core.PendingResourceIDs() {
if !strings.HasPrefix(resourceID, resourcePrefix) {
continue
}
if _, exists := seen[resourceID]; exists {
continue
}
if _, exists := absent[resourceID]; !exists {
absent[resourceID] = nil
}
}
if m.hostDiskTempAbsences == nil {
m.hostDiskTempAbsences = make(map[string]int)
}
// A disk seen again, or whose state is gone, restarts its count.
for resourceID := range m.hostDiskTempAbsences {
if !strings.HasPrefix(resourceID, resourcePrefix) {
continue
}
if _, exists := absent[resourceID]; !exists {
delete(m.hostDiskTempAbsences, resourceID)
}
}
for resourceID, storageKeys := range absent {
if seen != nil {
m.hostDiskTempAbsences[resourceID]++
if m.hostDiskTempAbsences[resourceID] < hostDiskTemperatureAbsenceConfirmations {
continue
}
}
delete(m.hostDiskTempAbsences, resourceID)
// A departed disk never sends the reading that would close its run.
m.core.DropPendingForResource(resourceID)
for _, storageKey := range storageKeys {
m.clearAlertNoLock(storageKey)
}
}
}
var customSensorAssessmentCodes = []string{
"custom_sensor_warning",
"custom_sensor_critical",
internal/alerts/host_unraid_lifecycle_test.go
+347
View File
@@ -1,6 +1,7 @@
package alerts
import (
"fmt"
"strings"
"testing"
"time"
@@ -199,6 +200,352 @@ func TestHandleHostTelemetryExpiredPreservesStaticStorageRisk(t *testing.T) {
})
}
func TestCheckHostClearsDiskTemperatureAlertsWhenThresholdTurnsOff(t *testing.T) {
cases := []struct {
name string
turnOff func(m *Manager, hostID string)
}{
{
name: "agent default disabled",
turnOff: func(m *Manager, hostID string) {
m.config.AgentDefaults.DiskTemperature = &HysteresisThreshold{Trigger: 0, Clear: 0}
},
},
{
name: "host override disables disk temperature",
turnOff: func(m *Manager, hostID string) {
m.config.Overrides[hostID] = ThresholdConfig{DiskTemperature: &HysteresisThreshold{Trigger: 0, Clear: 0}}
},
},
{
name: "host override disables the agent",
turnOff: func(m *Manager, hostID string) {
m.config.Overrides[hostID] = ThresholdConfig{Disabled: true}
},
},
{
name: "all agent alerts disabled",
turnOff: func(m *Manager, hostID string) {
m.config.DisableAllAgents = true
},
},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
m := configureDiskTempTypeHostManager(t)
host := hostWithSMARTDiskTemp("host-temp-off", "sata", 60)
alertID := hostDiskTempAlertID(host)
m.CheckHost(host)
alert, exists := testLookupActiveAlert(t, m, alertID)
if !exists {
t.Fatalf("expected disk temperature alert at 60C (sata trigger 55), active: %v", alertKeys(m))
}
if alert.Type != "diskTemperature" || !strings.HasPrefix(alert.ResourceID, hostDiskTemperatureResourcePrefix(host.ID)) {
t.Fatalf("disk temperature alert type/resource = %q/%q", alert.Type, alert.ResourceID)
}
// Change config directly so only CheckHost can resolve the alert.
m.mu.Lock()
tc.turnOff(m, host.ID)
m.mu.Unlock()
m.CheckHost(host)
if _, exists := testLookupActiveAlert(t, m, alertID); exists {
t.Fatalf("disk temperature alert stayed active after its threshold was turned off, active: %v", alertKeys(m))
}
})
}
}
func TestHostDiskTemperatureAlertsClearWhenAgentLeaves(t *testing.T) {
t.Run("agent removed", func(t *testing.T) {
m := configureDiskTempTypeHostManager(t)
host := hostWithSMARTDiskTemp("host-temp-removed", "sata", 60)
m.CheckHost(host)
if _, exists := testLookupActiveAlert(t, m, hostDiskTempAlertID(host)); !exists {
t.Fatalf("expected disk temperature alert, active: %v", alertKeys(m))
}
m.HandleHostRemoved(host)
if _, exists := testLookupActiveAlert(t, m, hostDiskTempAlertID(host)); exists {
t.Fatalf("disk temperature alert outlived its removed agent, active: %v", alertKeys(m))
}
})
t.Run("agent confirmed offline", func(t *testing.T) {
m := configureDiskTempTypeHostManager(t)
host := hostWithSMARTDiskTemp("host-temp-offline", "sata", 60)
m.CheckHost(host)
if _, exists := testLookupActiveAlert(t, m, hostDiskTempAlertID(host)); !exists {
t.Fatalf("expected disk temperature alert, active: %v", alertKeys(m))
}
for i := 0; i < 3; i++ {
m.HandleHostOffline(host)
}
if _, exists := testLookupActiveAlert(t, m, canonicalConnectivityStateID(hostResourceID(host.ID))); !exists {
t.Fatalf("expected confirmed offline alert, active: %v", alertKeys(m))
}
if _, exists := testLookupActiveAlert(t, m, hostDiskTempAlertID(host)); exists {
t.Fatalf("disk temperature alert stayed beside the offline alert, active: %v", alertKeys(m))
}
})
}
func TestCheckHostClearsDiskTemperatureAlertWhenDiskLeavesSMARTReport(t *testing.T) {
const hostID = "host-temp-absent"
smartDisk := func(device string, temperature int) models.HostDiskSMART {
return models.HostDiskSMART{Device: device, Model: "test-disk", Type: "sata", Temperature: temperature}
}
alertIDFor := func(device string) string {
return canonicalMetricStateID(hostDiskTemperatureResourceID(hostID, device), "diskTemperature")
}
sdaActive := func(m *Manager) bool {
_, exists := testLookupActiveAlert(t, m, alertIDFor("/dev/sda"))
return exists
}
// sda runs hot (sata trigger 55, clear 50); sdb stays cool.
hotHost := func(t *testing.T) (*Manager, models.Host) {
m := configureDiskTempTypeHostManager(t)
host := hostWithSMARTDiskTemp(hostID, "sata", 60)
host.Sensors.SMART = []models.HostDiskSMART{smartDisk("/dev/sda", 60), smartDisk("/dev/sdb", 40)}
m.CheckHost(host)
if !sdaActive(m) {
t.Fatalf("expected sda disk temperature alert at 60C, active: %v", alertKeys(m))
}
return m, host
}
cases := []struct {
name string
next []models.HostDiskSMART
wantHeld bool
wantRaised string
}{
{name: "disk removed", next: []models.HostDiskSMART{smartDisk("/dev/sdb", 40)}},
{name: "device renamed", next: []models.HostDiskSMART{smartDisk("/dev/sdc", 60), smartDisk("/dev/sdb", 40)}, wantRaised: "/dev/sdc"},
// No fresh reading says the disk cooled or left, so the alert holds.
{name: "SMART collection empty", next: nil, wantHeld: true},
{name: "disk in standby", next: []models.HostDiskSMART{{Device: "/dev/sda", Type: "sata", Temperature: 45, Standby: true}, smartDisk("/dev/sdb", 40)}, wantHeld: true},
{name: "probe returned no temperature", next: []models.HostDiskSMART{{Device: "/dev/sda", Type: "sata"}, smartDisk("/dev/sdb", 40)}, wantHeld: true},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
m, host := hotHost(t)
host.Sensors.SMART = tc.next
for report := 1; report <= hostDiskTemperatureAbsenceConfirmations; report++ {
m.CheckHost(host)
wantActive := tc.wantHeld || report < hostDiskTemperatureAbsenceConfirmations
if active := sdaActive(m); active != wantActive {
t.Fatalf("report %d: sda disk temperature alert active = %v, want %v; active: %v", report, active, wantActive, alertKeys(m))
}
}
if tc.wantRaised != "" {
if _, exists := testLookupActiveAlert(t, m, alertIDFor(tc.wantRaised)); !exists {
t.Fatalf("expected %s disk temperature alert, active: %v", tc.wantRaised, alertKeys(m))
}
}
if !tc.wantHeld {
return
}
// A held alert still resolves on the disk's next cool reading.
host.Sensors.SMART = []models.HostDiskSMART{smartDisk("/dev/sda", 45), smartDisk("/dev/sdb", 40)}
m.CheckHost(host)
if sdaActive(m) {
t.Fatalf("held sda disk temperature alert outlived a 45C reading, active: %v", alertKeys(m))
}
})
}
t.Run("intermittently dropped disk keeps its alert", func(t *testing.T) {
m, host := hotHost(t)
present := host.Sensors.SMART
dropped := []models.HostDiskSMART{smartDisk("/dev/sdb", 40)}
// The run of omissions restarts whenever the disk is reported again.
for _, smart := range [][]models.HostDiskSMART{dropped, dropped, present, dropped, dropped} {
host.Sensors.SMART = smart
m.CheckHost(host)
if !sdaActive(m) {
t.Fatalf("sda disk temperature alert cleared by an omission run shorter than %d reports, active: %v", hostDiskTemperatureAbsenceConfirmations, alertKeys(m))
}
}
})
// pendingHost leaves a hot sda inside a 300s alert delay, so sda holds a
// pending run and no alert.
pendingHost := func(t *testing.T) (*Manager, models.Host, func() bool) {
m := configureDiskTempTypeHostManager(t)
m.mu.Lock()
m.config.TimeThresholds = map[string]int{"agent": 300}
m.mu.Unlock()
sdaResourceID := hostDiskTemperatureResourceID(hostID, "/dev/sda")
sdaPending := func() bool {
m.mu.RLock()
defer m.mu.RUnlock()
for _, resourceID := range m.core.PendingResourceIDs() {
if resourceID == sdaResourceID {
return true
}
}
return false
}
host := hostWithSMARTDiskTemp(hostID, "sata", 60)
host.Sensors.SMART = []models.HostDiskSMART{smartDisk("/dev/sda", 60), smartDisk("/dev/sdb", 40)}
m.CheckHost(host)
if sdaActive(m) || !sdaPending() {
t.Fatalf("expected a pending sda run inside the 300s delay, active: %v", alertKeys(m))
}
return m, host, sdaPending
}
t.Run("departed disk drops its pending run", func(t *testing.T) {
m, host, sdaPending := pendingHost(t)
host.Sensors.SMART = []models.HostDiskSMART{smartDisk("/dev/sdb", 40)}
for report := 1; report <= hostDiskTemperatureAbsenceConfirmations; report++ {
m.CheckHost(host)
wantPending := report < hostDiskTemperatureAbsenceConfirmations
if pending := sdaPending(); pending != wantPending {
t.Fatalf("report %d: pending sda run = %v, want %v", report, pending, wantPending)
}
}
})
t.Run("threshold off drops pending runs", func(t *testing.T) {
m, host, sdaPending := pendingHost(t)
m.mu.Lock()
m.config.AgentDefaults.DiskTemperature = &HysteresisThreshold{Trigger: 0, Clear: 0}
m.mu.Unlock()
m.CheckHost(host)
if sdaPending() {
t.Fatal("pending sda run survived its threshold being turned off; re-enabling would inherit its old start time")
}
})
t.Run("removed host leaves no pending run or absence count", func(t *testing.T) {
m, host, sdaPending := pendingHost(t)
host.Sensors.SMART = []models.HostDiskSMART{smartDisk("/dev/sdb", 40)}
m.CheckHost(host)
m.HandleHostRemoved(host)
if sdaPending() {
t.Fatal("pending sda run outlived its removed host")
}
m.mu.RLock()
defer m.mu.RUnlock()
if len(m.hostDiskTempAbsences) != 0 {
t.Fatalf("absence counts outlived their removed host: %v", m.hostDiskTempAbsences)
}
})
}
func TestConfigSaveResolvesDiskTemperatureAlertAgainstItsDiskTypeThreshold(t *testing.T) {
m := configureDiskTempTypeHostManager(t)
m.mu.Lock()
m.config.AgentDefaults.DiskTemperature = &HysteresisThreshold{Trigger: 75, Clear: 70}
m.mu.Unlock()
// 72C is above the nvme trigger (70) but below the agent default (75).
host := hostWithSMARTDiskTemp("host-temp-save", "nvme", 72)
alertID := hostDiskTempAlertID(host)
m.CheckHost(host)
if _, exists := testLookupActiveAlert(t, m, alertID); !exists {
t.Fatalf("expected nvme disk temperature alert at 72C, active: %v", alertKeys(m))
}
// An unrelated save keeps the alert CheckHost would raise again.
cfg := m.GetConfig()
cfg.AgentDefaults.CPU = &HysteresisThreshold{Trigger: 85, Clear: 80}
m.UpdateConfig(cfg)
if _, exists := testLookupActiveAlert(t, m, alertID); !exists {
t.Fatalf("unrelated config save resolved a disk temperature alert above its nvme trigger, active: %v", alertKeys(m))
}
// Turning disk temperature off resolves it on save, before any report.
cfg = m.GetConfig()
cfg.AgentDefaults.DiskTemperature = &HysteresisThreshold{Trigger: 0, Clear: 0}
m.UpdateConfig(cfg)
if _, exists := testLookupActiveAlert(t, m, alertID); exists {
t.Fatalf("disk temperature alert stayed active after the threshold was turned off, active: %v", alertKeys(m))
}
}
func TestConfigSaveKeepsDiskTemperatureAlertWithoutDiskType(t *testing.T) {
m := configureDiskTempTypeHostManager(t)
m.mu.Lock()
m.config.AgentDefaults.DiskTemperature = &HysteresisThreshold{Trigger: 75, Clear: 70}
m.mu.Unlock()
host := hostWithSMARTDiskTemp("host-temp-legacy", "nvme", 72)
alertID := hostDiskTempAlertID(host)
m.CheckHost(host)
alert, exists := testLookupActiveAlert(t, m, alertID)
if !exists {
t.Fatalf("expected nvme disk temperature alert at 72C, active: %v", alertKeys(m))
}
// Restored from a release that did not record the disk type.
m.mu.Lock()
delete(alert.Metadata, "diskType")
m.mu.Unlock()
// 72C is above the nvme trigger (70) but below the agent default (75).
cfg := m.GetConfig()
cfg.AgentDefaults.CPU = &HysteresisThreshold{Trigger: 85, Clear: 80}
m.UpdateConfig(cfg)
if _, exists := testLookupActiveAlert(t, m, alertID); !exists {
t.Fatalf("config save resolved a disk temperature alert of unknown disk type that its nvme trigger still fires, active: %v", alertKeys(m))
}
cfg = m.GetConfig()
cfg.AgentDefaults.DiskTemperature = &HysteresisThreshold{Trigger: 0, Clear: 0}
m.UpdateConfig(cfg)
if _, exists := testLookupActiveAlert(t, m, alertID); exists {
t.Fatalf("disk temperature alert of unknown disk type stayed active after the threshold was turned off, active: %v", alertKeys(m))
}
}
func TestConfigSaveKeepsAlertTheEvaluatorKeepsWithoutRecoveryBand(t *testing.T) {
// Normalization keeps a 70/75 entry, and the evaluator treats a clear
// level at or above the trigger as no recovery band, firing from 70C.
for _, reading := range []int{72, 70} {
t.Run(fmt.Sprintf("%dC", reading), func(t *testing.T) {
m := configureDiskTempTypeHostManager(t)
m.mu.Lock()
m.config.DiskTempByType["nvme"] = HysteresisThreshold{Trigger: 70, Clear: 75}
m.mu.Unlock()
host := hostWithSMARTDiskTemp("host-temp-no-band", "nvme", reading)
alertID := hostDiskTempAlertID(host)
m.CheckHost(host)
if _, exists := testLookupActiveAlert(t, m, alertID); !exists {
t.Fatalf("expected nvme disk temperature alert at %dC, active: %v", reading, alertKeys(m))
}
cfg := m.GetConfig()
cfg.AgentDefaults.CPU = &HysteresisThreshold{Trigger: 85, Clear: 80}
m.UpdateConfig(cfg)
if _, exists := testLookupActiveAlert(t, m, alertID); !exists {
t.Fatalf("config save resolved an alert the evaluator keeps firing at %dC, active: %v", reading, alertKeys(m))
}
m.CheckHost(host)
if _, exists := testLookupActiveAlert(t, m, alertID); !exists {
t.Fatalf("expected the next report to keep the alert, active: %v", alertKeys(m))
}
})
}
}
func hasAlertType(alerts []Alert, alertType string) bool {
for _, alert := range alerts {
if alert.Type == alertType {
internal/alerts/manager.go
+2
View File
@@ -78,6 +78,7 @@ type Manager struct {
unifiedIncidentConfirmations map[string]int // Track consecutive provider-incident observations before activation
unifiedIncidentFirstSeen map[string]time.Time // Preserve the first confirmed observation as lifecycle start
unifiedIncidentRecoveries map[string]int // Track consecutive healthy observations before provider-incident recovery
hostDiskTempAbsences map[string]int // Track consecutive SMART reports missing a disk that holds temperature alert state
dockerRestartTracking map[string]*dockerRestartRecord // Track restart counts and times for restart loop detection
dockerUpdateFirstSeen map[string]time.Time // Track when image updates were first detected for alert delay
// Stable identity tracking prevents update-delay resets when host IDs churn.
@@ -233,6 +234,7 @@ func NewManagerWithDataDir(dataDir string, options ...ManagerOption) *Manager {
unifiedIncidentConfirmations: make(map[string]int),
unifiedIncidentFirstSeen: make(map[string]time.Time),
unifiedIncidentRecoveries: make(map[string]int),
hostDiskTempAbsences: make(map[string]int),
dockerRestartTracking: make(map[string]*dockerRestartRecord),
dockerUpdateFirstSeen: make(map[string]time.Time),
dockerUpdateFirstSeenByIdentity: make(map[string]time.Time),
internal/alerts/metric_runtime.go
+3 -1
View File
@@ -51,7 +51,7 @@ func metricStabilityFor(metricType string, delaySeconds int) metricStabilityPoli
func isMetricThresholdAlertType(metricType string) bool {
switch metricType {
case "cpu", "memory", "disk", "diskRead", "diskWrite", "networkIn", "networkOut", "temperature", "usage":
case "cpu", "memory", "disk", "diskRead", "diskWrite", "networkIn", "networkOut", "temperature", "diskTemperature", "usage":
return true
default:
return false
@@ -77,6 +77,8 @@ func getThresholdForMetric(config ThresholdConfig, metricType string) *Hysteresi
return config.NetworkOut
case "temperature":
return config.Temperature
case "diskTemperature":
return config.DiskTemperature
case "usage":
return config.Usage
default:
internal/alerts/model.go
+3
View File
@@ -137,6 +137,7 @@ type Alert struct {
AckTime *time.Time `json:"ackTime,omitempty"`
AckUser string `json:"ackUser,omitempty"`
Correlation *AlertCorrelation `json:"correlation,omitempty"`
Resolution *AlertResolution `json:"resolution,omitempty"` // set only when a close was not a recovery
Metadata map[string]interface{} `json:"metadata,omitempty"`
LastNotified *time.Time `json:"lastNotified,omitempty"`
LastEscalation int `json:"lastEscalation,omitempty"`
@@ -186,6 +187,8 @@ func (a *Alert) Clone() *Alert {
clone.Correlation = cloneAlertCorrelation(a.Correlation)
}
clone.Resolution = a.Resolution.Clone()
if len(a.EscalationTimes) > 0 {
clone.EscalationTimes = append([]time.Time(nil), a.EscalationTimes...)
}
internal/alerts/operational_contract.go
+25 -5
View File
@@ -285,6 +285,7 @@ func markOperationalResolved(
}
alert.Evidence = appendOperationalEvidence(alert.Evidence, recoveryEvidence.Clone())
recoveryEvidenceIDs := []string{recoveryEvidence.ID}
cause := resolvedTransitionCause(alert.Resolution)
from := record.State
record.State = operationaltrust.OperationalResolved
record.StateChangedAt = resolvedAt
@@ -295,7 +296,7 @@ func markOperationalResolved(
from,
operationaltrust.OperationalResolved,
resolvedAt,
operationaltrust.TransitionRecoveryEvidence,
cause,
record.CauseKey,
recoveryEvidenceIDs,
)
@@ -308,9 +309,10 @@ func markOperationalResolved(
From: from,
To: operationaltrust.OperationalResolved,
At: resolvedAt,
Cause: operationaltrust.TransitionRecoveryEvidence,
Cause: cause,
CauseKey: record.CauseKey,
EvidenceIDs: recoveryEvidenceIDs,
Reason: alert.Resolution.Summary(),
}
alert.Transitions = appendOperationalTransition(
alert.Transitions,
@@ -318,6 +320,16 @@ func markOperationalResolved(
)
}
// resolvedTransitionCause names why a record entered resolved. Only a close
// with no typed resolution is a recovery; a handover keeps its own cause so
// timelines and alert-quality counts never report it as one.
func resolvedTransitionCause(resolution *AlertResolution) operationaltrust.TransitionCause {
if resolution != nil && resolution.Reason == AlertResolutionMovedToAgent {
return operationaltrust.TransitionOwnershipTransferred
}
return operationaltrust.TransitionRecoveryEvidence
}
func (m *Manager) newResolvedAlert(
alert *Alert,
resolvedAt time.Time,
@@ -497,6 +509,16 @@ func legacyAlertRecoveryEvidenceEnvelope(
if err != nil {
return operationaltrust.EvidenceEnvelope{}, false
}
reason := &operationaltrust.EvidenceReason{
Code: legacyAlertRecoveryEvidenceReason,
}
// A close that was not a recovery must not be recorded as one.
if summary := alert.Resolution.Summary(); summary != "" {
reason = &operationaltrust.EvidenceReason{
Code: string(alert.Resolution.Reason),
Message: summary,
}
}
envelope := operationaltrust.EvidenceEnvelope{
ID: id,
Source: source,
@@ -506,9 +528,7 @@ func legacyAlertRecoveryEvidenceEnvelope(
Completeness: operationaltrust.EvidencePartial,
Confidence: operationaltrust.EvidenceUnknown,
Permissions: operationaltrust.EvidencePermissionsUnknown,
Reason: &operationaltrust.EvidenceReason{
Code: legacyAlertRecoveryEvidenceReason,
},
Reason: reason,
}
return envelope, envelope.Validate() == nil
}
internal/alerts/operational_contract_test.go
+170
View File
@@ -5,6 +5,7 @@ import (
"testing"
"time"
"github.com/rcourtman/pulse-go-rewrite/internal/alerts/eventlog"
alertspecs "github.com/rcourtman/pulse-go-rewrite/internal/alerts/specs"
"github.com/rcourtman/pulse-go-rewrite/internal/models"
"github.com/rcourtman/pulse-go-rewrite/internal/operationaltrust"
@@ -412,6 +413,175 @@ func TestResolvedOperationalContractUpdatesHistorySnapshot(t *testing.T) {
}
}
func TestReleasedMetricAlertCarriesResolutionInsteadOfRecovery(t *testing.T) {
manager := newUnifiedEvalParityManager(t)
store, err := eventlog.OpenInMemory()
if err != nil {
t.Fatalf("open in-memory event log: %v", err)
}
manager.SetEventLog(store)
manager.UpdateConfig(AlertConfig{
Enabled: true,
NodeDefaults: ThresholdConfig{
CPU: &HysteresisThreshold{Trigger: 80, Clear: 75},
Memory: &HysteresisThreshold{Trigger: 85, Clear: 80},
},
})
disableTestTimeThresholds(manager)
node := models.Node{
ID: "pve-1",
Name: "pve-1",
Instance: "test",
Status: "online",
CPU: 0.95,
Memory: models.Memory{Total: 8 << 30, Used: 7 << 30, Usage: 95},
}
manager.CheckNode(node)
if active := manager.GetActiveAlerts(); len(active) != 2 {
t.Fatalf("active alerts = %#v, want cpu and memory", active)
}
resolvedByType := map[string]*ResolvedAlert{}
manager.SetResolvedAlertCallback(func(resolved *ResolvedAlert) {
resolvedByType[resolved.Alert.Type] = resolved
})
release := func(metric string, resolution *AlertResolution) {
t.Helper()
spec, err := buildCanonicalMetricSpec(node.ID, node.Name, unifiedresources.ResourceType("node"), metric, nil)
if err != nil {
t.Fatalf("build %s spec: %v", metric, err)
}
manager.releaseCanonicalMetricAlert(spec, node.Name, node.Name, node.Instance, "node", 0, resolution)
}
handover := &AlertResolution{
Reason: AlertResolutionMovedToAgent,
SuccessorResourceID: "agent-pve-1",
SuccessorName: "pve-1 (Host Agent)",
}
release("memory", handover)
// The close is recorded as a copy: the caller's value cannot rewrite it.
handover.SuccessorName = "changed"
release("cpu", nil)
if active := manager.GetActiveAlerts(); len(active) != 0 {
t.Fatalf("active alerts after release = %#v, want none: the close must still happen", active)
}
moved := resolvedByType["memory"]
if moved == nil {
t.Fatal("memory release did not reach the resolved consumer")
}
resolution := moved.Alert.Resolution
if resolution == nil ||
resolution.Reason != AlertResolutionMovedToAgent ||
resolution.SuccessorResourceID != "agent-pve-1" ||
resolution.SuccessorName != "pve-1 (Host Agent)" {
t.Fatalf("resolution = %+v, want moved to pve-1 (Host Agent)", resolution)
}
wantSummary := "Alert moved to pve-1 (Host Agent). This is not a recovery: check the agent for the current reading."
if got := resolution.Summary(); got != wantSummary {
t.Fatalf("Summary() = %q, want %q", got, wantSummary)
}
if got := resolution.Outcome(); got != "moved to pve-1 (Host Agent)" {
t.Fatalf("Outcome() = %q", got)
}
if got := resolution.Describe("Memory alert"); got != "Memory alert moved to pve-1 (Host Agent). This is not a recovery: check the agent for the current reading." {
t.Fatalf("Describe() = %q", got)
}
record := moved.Alert.OperationalRecord
if record == nil || record.State != operationaltrust.OperationalResolved {
t.Fatalf("operational record = %+v, want resolved", record)
}
if err := record.Validate(); err != nil {
t.Fatalf("operational record Validate() error = %v", err)
}
transition := moved.Alert.LatestTransition
if transition == nil || transition.Reason != wantSummary {
t.Fatalf("resolve transition = %+v, want reason %q", transition, wantSummary)
}
if err := transition.Validate(); err != nil {
t.Fatalf("resolve transition Validate() error = %v", err)
}
// Patrol's timeline prints the cause above the reason, so a handover
// must not be labelled as recovery evidence.
if transition.Cause != operationaltrust.TransitionOwnershipTransferred {
t.Fatalf("resolve transition cause = %q, want %q", transition.Cause, operationaltrust.TransitionOwnershipTransferred)
}
if recorded := moved.Alert.Transitions[len(moved.Alert.Transitions)-1]; recorded.ID != transition.ID ||
recorded.Cause != operationaltrust.TransitionOwnershipTransferred {
t.Fatalf("recorded timeline transition = %+v, want the ownership transfer", recorded)
}
closing := moved.Alert.Evidence[len(moved.Alert.Evidence)-1]
if closing.ID != transition.EvidenceIDs[0] ||
closing.Reason == nil ||
closing.Reason.Code != string(AlertResolutionMovedToAgent) ||
closing.Reason.Message != wantSummary {
t.Fatalf("closing evidence = %+v, want the handover, not a recovery projection", closing)
}
recovered := resolvedByType["cpu"]
if recovered == nil || recovered.Alert.Resolution != nil {
t.Fatalf("plain disabled release = %+v, want an ordinary resolve with no resolution", recovered)
}
if last := recovered.Alert.Evidence[len(recovered.Alert.Evidence)-1]; last.Reason == nil ||
last.Reason.Code != legacyAlertRecoveryEvidenceReason {
t.Fatalf("plain release evidence = %+v, want legacy recovery projection", last.Reason)
}
if recovered.Alert.LatestTransition == nil ||
recovered.Alert.LatestTransition.Cause != operationaltrust.TransitionRecoveryEvidence {
t.Fatalf("plain release transition = %+v, want recovery_evidence", recovered.Alert.LatestTransition)
}
var historyResolution *AlertResolution
for _, entry := range manager.GetAlertHistory(10) {
if entry.Type == "memory" {
historyResolution = entry.Resolution
}
}
if historyResolution == nil || historyResolution.Reason != AlertResolutionMovedToAgent {
t.Fatalf("history resolution = %+v, want moved_to_agent", historyResolution)
}
events, err := manager.AlertEvents(eventlog.Filter{AlertID: moved.Alert.ID, Types: []string{eventlog.TypeResolved}})
if err != nil {
t.Fatalf("AlertEvents: %v", err)
}
if len(events) != 1 ||
events[0].Reason != string(AlertResolutionMovedToAgent) ||
events[0].Message != wantSummary {
t.Fatalf("resolved events = %+v, want one handover event", events)
}
}
func TestResolvedHistoryRowKeepsResolutionWithoutEventLog(t *testing.T) {
firstObservedAt := time.Date(2026, 10, 6, 9, 0, 0, 0, time.UTC)
alert := &Alert{
ID: "pve-1-memory",
Type: "memory",
Level: AlertLevelWarning,
ResourceID: "pve-1",
CanonicalSpecID: "metric-threshold:memory",
CanonicalState: "pve-1::metric-threshold:memory",
StartTime: firstObservedAt,
LastSeen: firstObservedAt,
}
ensureOperationalContract(alert, firstObservedAt)
history := newTestHistoryManager(t)
history.AddAlert(*alert)
manager := &Manager{historyManager: history}
alert.Resolution = &AlertResolution{Reason: AlertResolutionMovedToAgent, SuccessorName: "pve-1 (Host Agent)"}
manager.newResolvedAlert(alert, firstObservedAt.Add(time.Minute), nil)
entries := history.GetAllHistory(1)
if len(entries) != 1 || entries[0].Resolution == nil ||
entries[0].Resolution.Reason != AlertResolutionMovedToAgent {
t.Fatalf("history entries = %+v, want the row to keep the moved_to_agent resolution", entries)
}
}
func TestMergeOperationalRecurrenceReopensStableCause(t *testing.T) {
firstObservedAt := time.Date(2026, 7, 18, 20, 0, 0, 0, time.UTC)
resolvedAt := firstObservedAt.Add(5 * time.Minute)
internal/alerts/read_model.go
+16 -1
View File
@@ -301,7 +301,8 @@ func (m *Manager) GetRecentlyResolved() []models.ResolvedAlert {
StartTime: exported.StartTime,
Acknowledged: exported.Acknowledged,
// exported is a deep clone, so the map is already private.
Metadata: exported.Metadata,
Metadata: exported.Metadata,
Resolution: modelAlertResolution(exported.Resolution),
},
ResolvedTime: alert.ResolvedTime,
})
@@ -309,6 +310,20 @@ func (m *Manager) GetRecentlyResolved() []models.ResolvedAlert {
return resolved
}
// modelAlertResolution projects a non-recovery close into the state model, so
// snapshot and assistant readers can tell a handover from a recovery.
func modelAlertResolution(resolution *AlertResolution) *models.AlertResolution {
if resolution == nil {
return nil
}
return &models.AlertResolution{
Reason: string(resolution.Reason),
SuccessorResourceID: resolution.SuccessorResourceID,
SuccessorName: resolution.SuccessorName,
Summary: resolution.Summary(),
}
}
// GetResolvedAlert returns a copy of a recently resolved alert by ID.
func (m *Manager) GetResolvedAlert(alertID string) *ResolvedAlert {
// Write lock: getResolvedAlertNoLock backfills the resolvedAlias map on a
internal/alerts/resolution.go
+79
View File
@@ -0,0 +1,79 @@
package alerts
import (
"fmt"
"strings"
)
// AlertResolutionReason says why an alert closed when the close was not a
// recovery. The close still reaches every resolved consumer, so an incident
// opened from the alert does not stay open on the receiving side, but no
// surface may report the resource as healthy.
type AlertResolutionReason string
const (
// AlertResolutionMovedToAgent means a Pulse agent linked to the alert's
// resource now owns the same metric, so the agent's own alert settings
// decide whether the condition is still alerted on.
AlertResolutionMovedToAgent AlertResolutionReason = "moved_to_agent"
)
// AlertResolution records why an alert closed without its condition being
// observed to clear. A nil resolution is an ordinary recovery.
type AlertResolution struct {
Reason AlertResolutionReason `json:"reason"`
// SuccessorResourceID and SuccessorName identify the resource that owns
// the alert from now on, named the way that resource's own alerts are.
SuccessorResourceID string `json:"successorResourceId,omitempty"`
SuccessorName string `json:"successorName,omitempty"`
}
// Clone returns a copy of the resolution.
func (r *AlertResolution) Clone() *AlertResolution {
if r == nil {
return nil
}
clone := *r
return &clone
}
// Outcome is the short form of a close that was not a recovery, for lists
// such as "moved to pve1 (Host Agent)". It is empty for an ordinary recovery.
func (r *AlertResolution) Outcome() string {
if r == nil {
return ""
}
switch r.Reason {
case AlertResolutionMovedToAgent:
successor := strings.TrimSpace(r.SuccessorName)
if successor == "" {
successor = "its Pulse agent"
}
return "moved to " + successor
default:
return ""
}
}
// Summary is the one-line account of a close that was not a recovery. It is
// empty for an ordinary recovery.
func (r *AlertResolution) Summary() string {
return r.Describe("Alert")
}
// Describe is Summary with the alert named by the caller, such as
// "Memory alert", so a message about one of several closes says which one.
func (r *AlertResolution) Describe(subject string) string {
if r == nil {
return ""
}
if subject = strings.TrimSpace(subject); subject == "" {
subject = "Alert"
}
switch r.Reason {
case AlertResolutionMovedToAgent:
return fmt.Sprintf("%s %s. This is not a recovery: check the agent for the current reading.", subject, r.Outcome())
default:
return ""
}
}
internal/models/deepcopy.go
+4
View File
@@ -1337,6 +1337,10 @@ func cloneAlert(src Alert) Alert {
dest.Metadata[k] = v
}
}
if src.Resolution != nil {
resolution := *src.Resolution
dest.Resolution = &resolution
}
return dest
}
internal/models/host_transient_expiry_test.go
+96 -2
View File
@@ -1,6 +1,11 @@
package models
import "testing"
import (
"testing"
"time"
"github.com/rcourtman/pulse-go-rewrite/pkg/diskinventory"
)
func TestExpireHostTelemetryClearsTransientStorageOperationsOnly(t *testing.T) {
state := NewState()
@@ -24,7 +29,7 @@ func TestExpireHostTelemetryClearsTransientStorageOperationsOnly(t *testing.T) {
},
})
host, changed := state.ExpireHostTelemetry("storage-host")
host, changed := state.ExpireHostTelemetry("storage-host", time.Time{})
if !changed {
t.Fatal("expected host telemetry to change")
}
@@ -47,3 +52,92 @@ func TestExpireHostTelemetryClearsTransientStorageOperationsOnly(t *testing.T) {
t.Fatalf("RAID last-known health was lost: %+v", host.RAID[0])
}
}
func TestExpireHostTelemetryMarksSMARTReadingsNoLongerCollected(t *testing.T) {
state := NewState()
judged := time.Now().Add(-time.Hour)
state.UpsertHost(Host{
ID: "agent",
Status: "online",
LastSeen: judged,
Sensors: HostSensorSummary{SMART: []HostDiskSMART{
{
Device: "sda", Serial: "CURRENT", Temperature: 41, Health: "PASSED",
IO: &DiskIO{Device: "sda", ReadBytes: 1000},
Collection: &diskinventory.CollectionStatus{
Serial: diskinventory.Available("smartctl"),
Temperature: diskinventory.Available("smartctl"),
IO: diskinventory.Available("kernel_diskstats"),
},
},
// Agents before collection provenance report only what they collected.
{Device: "sdb", Serial: "LEGACY", Temperature: 38, IO: &DiskIO{Device: "sdb"}},
{
Device: "sdc", Serial: "ASLEEP", Standby: true,
Collection: &diskinventory.CollectionStatus{
Temperature: diskinventory.Unavailable("smartctl", "disk is in standby"),
IO: diskinventory.Unsupported("controller", "per-member counters unavailable"),
},
},
}},
})
host, changed := state.ExpireHostTelemetry("agent", judged)
if !changed {
t.Fatal("expected host telemetry to change")
}
stopped := diskinventory.Unavailable("smartctl", "host agent stopped reporting")
current := host.Sensors.SMART[0]
if current.Temperature != 41 || current.IO == nil || current.IO.ReadBytes != 1000 {
t.Fatalf("last-known SMART readings were dropped: %+v", current)
}
if current.Collection.Temperature != stopped ||
current.Collection.IO != diskinventory.Unavailable("kernel_diskstats", "host agent stopped reporting") {
t.Fatalf("expired readings still claim to be collected: %+v", current.Collection)
}
if current.Collection.Serial != diskinventory.Available("smartctl") {
t.Fatalf("disk identity evidence must stay last-known: %+v", current.Collection.Serial)
}
legacy := host.Sensors.SMART[1]
if legacy.Temperature != 38 || legacy.Collection == nil ||
legacy.Collection.Temperature.State != diskinventory.FieldUnavailable ||
legacy.Collection.IO.State != diskinventory.FieldUnavailable {
t.Fatalf("provenance-less readings not expired: %+v", legacy)
}
asleep := host.Sensors.SMART[2].Collection
if asleep.Temperature != diskinventory.Unavailable("smartctl", "disk is in standby") ||
asleep.IO.State != diskinventory.FieldUnsupported {
t.Fatalf("readings that were already not collected must keep their state: %+v", asleep)
}
if _, changed := state.ExpireHostTelemetry("agent", judged); changed {
t.Fatal("re-evaluating an expired host must not report a change")
}
stored, _ := state.GetHost("agent")
if stored.Sensors.SMART[0].Collection.Temperature != stopped {
t.Fatalf("expiry was not kept in state: %+v", stored.Sensors.SMART[0].Collection)
}
}
// Expiry is judged on a host snapshot. A report accepted before the expiry
// takes the state lock must not be expired on the strength of that judgement.
func TestExpireHostTelemetryLeavesHostThatReportedSinceJudgement(t *testing.T) {
state := NewState()
judged := time.Now().Add(-time.Hour)
fresh := Host{
ID: "agent", Status: "online", LastSeen: time.Now(),
Sensors: HostSensorSummary{SMART: []HostDiskSMART{{
Device: "sda", Temperature: 41,
Collection: &diskinventory.CollectionStatus{Temperature: diskinventory.Available("smartctl")},
}}},
}
state.UpsertHost(fresh)
if host, changed := state.ExpireHostTelemetry("agent", judged); changed || host.ID != "" {
t.Fatalf("a host that reported after the stale judgement was expired: %+v changed=%v", host, changed)
}
stored, _ := state.GetHost("agent")
if stored.Status != "online" || stored.Sensors.SMART[0].Collection.Temperature.State != diskinventory.FieldAvailable {
t.Fatalf("fresh report was altered: %+v", stored)
}
}
internal/models/metrics_types_test.go
+37
View File
@@ -96,6 +96,43 @@ func TestAlertLastSeenWireContract(t *testing.T) {
}
}
// A resolved alert that closed without recovering says so on the state and
// websocket payloads; an ordinary recovery carries no resolution at all.
func TestAlertResolutionWireContract(t *testing.T) {
recovered, err := json.Marshal(ResolvedAlert{Alert: Alert{ID: "alert-1"}})
if err != nil {
t.Fatalf("marshal recovered alert: %v", err)
}
if strings.Contains(string(recovered), "resolution") {
t.Fatalf("payload = %s, a recovery must not carry a resolution", recovered)
}
moved := ResolvedAlert{Alert: Alert{ID: "alert-1", Resolution: &AlertResolution{
Reason: "moved_to_agent",
SuccessorResourceID: "agent-pve-1",
SuccessorName: "pve-1 (Host Agent)",
Summary: "Alert moved to pve-1 (Host Agent). This is not a recovery: check the agent for the current reading.",
}}}
payload, err := json.Marshal(moved)
if err != nil {
t.Fatalf("marshal moved alert: %v", err)
}
want := `"resolution":{"reason":"moved_to_agent","successorResourceId":"agent-pve-1","successorName":"pve-1 (Host Agent)","summary":"Alert moved to pve-1 (Host Agent). This is not a recovery: check the agent for the current reading."}`
if !strings.Contains(string(payload), want) {
t.Fatalf("payload = %s, want %s", payload, want)
}
// The state store keeps its own copy of each resolution.
state := NewState()
state.UpdateRecentlyResolved([]ResolvedAlert{moved})
moved.Resolution.SuccessorName = "changed"
snapshot := state.GetSnapshot().RecentlyResolved
if len(snapshot) != 1 || snapshot[0].Resolution == nil ||
snapshot[0].Resolution.SuccessorName != "pve-1 (Host Agent)" {
t.Fatalf("snapshot = %+v, want the resolution as it was recorded", snapshot)
}
}
func TestDockerHostCollectionModeWireContract(t *testing.T) {
payload, err := json.Marshal(DockerHost{
ID: "docker-host-1",
internal/models/models.go
+63 -1
View File
@@ -112,6 +112,20 @@ type Alert struct {
// Metadata carries alert-engine annotations (notably resourceType) so the
// frontend can classify an alert without re-deriving resource identity.
Metadata map[string]interface{} `json:"metadata,omitempty"`
// Resolution is set only on a resolved alert whose close was not a
// recovery, so readers never report its resource as healthy.
Resolution *AlertResolution `json:"resolution,omitempty"`
}
// AlertResolution is the state projection of the alert engine's
// AlertResolution: why an alert closed without its condition clearing.
type AlertResolution struct {
Reason string `json:"reason"`
SuccessorResourceID string `json:"successorResourceId,omitempty"`
SuccessorName string `json:"successorName,omitempty"`
// Summary is the alert engine's one-line account, such as "Alert moved
// to pve1 (Host Agent). This is not a recovery: ...".
Summary string `json:"summary,omitempty"`
}
// ResolvedAlert represents a recently resolved alert
@@ -5686,7 +5700,13 @@ func (s *State) SetHostStatus(hostID, status string) bool {
// ExpireHostTelemetry marks a host offline and clears transient operation
// claims that cannot remain authoritative after the reporting lease expires.
// Static topology and health counters remain available as last-known context.
func (s *State) ExpireHostTelemetry(hostID string) (Host, bool) {
// SMART temperature and I/O readings stay as last-known values, but their
// collection state says they are no longer collected.
//
// lastSeen is the report time the caller judged stale. A host that has
// reported since then is left untouched and nothing is returned, so a report
// accepted between that judgement and this call is never expired.
func (s *State) ExpireHostTelemetry(hostID string, lastSeen time.Time) (Host, bool) {
s.mu.Lock()
defer s.mu.Unlock()
@@ -5694,6 +5714,9 @@ func (s *State) ExpireHostTelemetry(hostID string) (Host, bool) {
if host.ID != hostID {
continue
}
if !host.LastSeen.Equal(lastSeen) {
return Host{}, false
}
changed := host.Status != "offline"
host.Status = "offline"
for idx := range host.RAID {
@@ -5710,6 +5733,11 @@ func (s *State) ExpireHostTelemetry(hostID string) (Host, bool) {
host.Unraid.SyncProgress = 0
changed = true
}
for idx := range host.Sensors.SMART {
if expireHostSMARTReadings(&host.Sensors.SMART[idx]) {
changed = true
}
}
s.Hosts[i] = host
if changed {
s.LastUpdate = time.Now()
@@ -5719,6 +5747,40 @@ func (s *State) ExpireHostTelemetry(hostID string) (Host, bool) {
return Host{}, false
}
const hostAgentStoppedReportingReason = "host agent stopped reporting"
// expireHostSMARTReadings marks the temperature and I/O counters of one SMART
// row as no longer collected, keeping their values as last-known evidence.
// Reports from agents that predate collection provenance carry no state, so a
// present reading is treated as collected. It reports whether anything changed.
func expireHostSMARTReadings(disk *HostDiskSMART) bool {
collection := diskinventory.CloneStatus(disk.Collection)
if collection == nil {
collection = &diskinventory.CollectionStatus{}
}
temperature := expireCollectedReading(&collection.Temperature, disk.Temperature > 0)
io := expireCollectedReading(&collection.IO, disk.IO != nil)
if !temperature && !io {
return false
}
disk.Collection = collection
return true
}
func expireCollectedReading(status *diskinventory.FieldStatus, hasValue bool) bool {
switch status.State {
case diskinventory.FieldAvailable:
case "":
if !hasValue {
return false
}
default:
return false
}
*status = diskinventory.Unavailable(status.Source, hostAgentStoppedReportingReason)
return true
}
// TouchHost updates the last seen timestamp for a host.
func (s *State) TouchHost(hostID string, ts time.Time) bool {
s.mu.Lock()
internal/monitoring/issue1595_collection_trust_test.go
+44
View File
@@ -10,6 +10,7 @@ import (
"github.com/rcourtman/pulse-go-rewrite/internal/models"
"github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources"
"github.com/rcourtman/pulse-go-rewrite/pkg/diskinventory"
"github.com/rcourtman/pulse-go-rewrite/pkg/metrics"
)
type issue1595TopologyFixture struct {
@@ -316,3 +317,46 @@ func hasIssue1595Source(sources []unifiedresources.DataSource, want unifiedresou
}
return false
}
// An agent report's SMART row may carry a temperature its collector did not
// collect this time (retained across a standby or failed probe). Only a
// collected temperature becomes a history sample.
func TestWriteHostSMARTMetricsRecordsOnlyCollectedTemperatures(t *testing.T) {
storeConfig := metrics.DefaultConfig(t.TempDir())
storeConfig.FlushInterval = time.Hour
store, err := metrics.NewStore(storeConfig)
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { _ = store.Close() })
m := &Monitor{metricsStore: store}
host := models.Host{ID: "agent-1", Hostname: "node1", Sensors: models.HostSensorSummary{SMART: []models.HostDiskSMART{
{Device: "sda", Serial: "COLLECTED1", Temperature: 41,
Collection: &diskinventory.CollectionStatus{Temperature: diskinventory.Available("smartctl")}},
{Device: "sdb", Serial: "LEGACY1", Temperature: 38},
{Device: "sdc", Serial: "RETAINED1", Temperature: 36,
Collection: &diskinventory.CollectionStatus{Temperature: diskinventory.Unavailable("smartctl", "disk is in standby")}},
}}}
now := time.Now()
m.writeHostSMARTMetrics(host, now)
store.Flush()
for _, tc := range []struct {
disk models.HostDiskSMART
want int
}{
{host.Sensors.SMART[0], 1},
{host.Sensors.SMART[1], 1},
{host.Sensors.SMART[2], 0},
} {
id := unifiedresources.HostSMARTDiskSourceID(host, tc.disk)
points, err := store.Query("disk", id, "smart_temp", now.Add(-time.Minute), now.Add(time.Minute), 0)
if err != nil {
t.Fatal(err)
}
if len(points) != tc.want {
t.Fatalf("%s: temperature samples = %d, want %d", tc.disk.Serial, len(points), tc.want)
}
}
}
internal/monitoring/monitor.go
+67 -7
View File
@@ -648,7 +648,10 @@ func mergeHostAgentSMARTIntoDisks(disks []models.PhysicalDisk, nodes []models.No
ioCopy := *matched.IO
updated[i].IO = &ioCopy
}
updated[i].Collection = diskinventory.MergeStatus(updated[i].Collection, matched.Collection)
// The agent's later word on a field it supplied (its lease expired, or
// the disk went into standby) must win over the copy a skipped disk
// poll carries forward from the previous merge.
updated[i].Collection = diskinventory.MergeReportedStatus(updated[i].Collection, hostAgentSMARTCollection(*matched))
if shouldUseHostAgentPhysicalDiskHealth(updated[i].Health, matched.Health) {
updated[i].Health = matched.Health
@@ -658,6 +661,28 @@ func mergeHostAgentSMARTIntoDisks(disks []models.PhysicalDisk, nodes []models.No
return updated
}
// hostAgentLegacySource is the provenance recorded for a temperature from an
// agent that predates collection provenance (before 6.2). Those agents only
// send readings they collected.
const hostAgentLegacySource = "host_agent"
// hostAgentSMARTCollection returns the agent's collection state for one SMART
// row, giving a provenance-less temperature the legacy agent source so that
// history and later lease expiry treat it like any other agent reading.
func hostAgentSMARTCollection(smart models.HostDiskSMART) *diskinventory.CollectionStatus {
collection := diskinventory.CloneStatus(smart.Collection)
if collection == nil {
collection = &diskinventory.CollectionStatus{}
}
switch {
case collection.Temperature.State == "" && smart.Temperature > 0 && !smart.Standby:
collection.Temperature = diskinventory.Available(hostAgentLegacySource)
case collection.Temperature.State != "" && strings.TrimSpace(collection.Temperature.Source) == "":
collection.Temperature.Source = hostAgentLegacySource
}
return collection
}
func shouldUseHostAgentPhysicalDiskHealth(existing, incoming string) bool {
incoming = strings.ToUpper(strings.TrimSpace(incoming))
if incoming == "" || incoming == "UNKNOWN" {
@@ -914,8 +939,7 @@ func (m *Monitor) writeSMARTMetrics(disk models.PhysicalDisk, now time.Time) {
return
}
// Temperature (always write if > 0)
if disk.Temperature > 0 && m.metricsHistory != nil {
if diskTemperatureCollected(disk.Temperature, disk.Collection) && m.metricsHistory != nil {
m.metricsHistory.AddDiskMetric(resourceID, "smart_temp", float64(disk.Temperature), now)
}
@@ -929,6 +953,22 @@ func (m *Monitor) writeSMARTMetrics(disk models.PhysicalDisk, now time.Time) {
}
}
// diskTemperatureCollected reports whether a disk temperature was collected by
// its current observation. Normalization may carry a last-known temperature
// when it was not (standby, an agent past its reporting lease); that value must
// not be recorded as a new history sample. A temperature without collection
// state predates the contract and keeps the old behavior.
func diskTemperatureCollected(temperature int, collection *diskinventory.CollectionStatus) bool {
if temperature <= 0 {
return false
}
if collection == nil {
return true
}
state := collection.Temperature.State
return state == "" || state == diskinventory.FieldAvailable
}
// smartMetricStoreWrites builds the persisted SMART writes for one physical
// disk observation. The caller owns batching so the unified sync can anchor the
// observation time and drop exact replays (#1966).
@@ -949,8 +989,7 @@ func (m *Monitor) smartMetricStoreWrites(disk models.PhysicalDisk, resourceID st
})
}
// Temperature (always write if > 0)
if disk.Temperature > 0 {
if diskTemperatureCollected(disk.Temperature, disk.Collection) {
appendWrite("smart_temp", float64(disk.Temperature))
}
@@ -3577,6 +3616,7 @@ func hostFromReadStateView(view *unifiedresources.HostView) models.Host {
if trimmed := strings.TrimSpace(view.Name()); trimmed != "" && trimmed != view.Hostname() {
displayName = trimmed
}
status, lastSeen := hostAgentLivenessFromReadStateView(view, time.Now())
return models.Host{
ID: firstNonEmptyString(view.AgentID(), view.ID()),
@@ -3598,10 +3638,10 @@ func hostFromReadStateView(view *unifiedresources.HostView) models.Host {
RAID: hostRAIDFromReadStateView(view.RAID()),
Unraid: hostUnraidFromReadStateView(view.Unraid()),
Ceph: hostCephFromReadStateView(view.Ceph()),
Status: string(view.Status()),
Status: status,
UptimeSeconds: view.UptimeSeconds(),
IntervalSeconds: view.IntervalSeconds(),
LastSeen: view.LastSeen(),
LastSeen: lastSeen,
AgentVersion: view.AgentVersion(),
IntegrationSource: view.IntegrationSource(),
MachineID: view.MachineID(),
@@ -3624,6 +3664,25 @@ func hostFromReadStateView(view *unifiedresources.HostView) models.Host {
}
}
// hostAgentLivenessFromReadStateView returns the status and last report of the
// agent behind a read-state host row. A row merged with a Proxmox node takes
// LastSeen from its freshest source and stays online while PVE polling
// continues, so a silent agent would look current: its retained disk summary
// and datasets kept feeding the node, and its connection read as active. A
// models.Host describes the agent, so read the agent source's own sighting
// and lease, as evaluateHostAgents does for the host in state.
func hostAgentLivenessFromReadStateView(view *unifiedresources.HostView, now time.Time) (string, time.Time) {
status, lastSeen := string(view.Status()), view.LastSeen()
agentSighting, ok := view.SourceStatus(unifiedresources.SourceAgent)
if !ok || agentSighting.LastSeen.IsZero() {
return status, lastSeen
}
if !hostAgentReportCurrent(agentSighting.LastSeen, view.IntervalSeconds(), now) {
status = "offline"
}
return status, agentSighting.LastSeen
}
func vmFromReadStateView(view *unifiedresources.VMView) models.VM {
if view == nil {
return models.VM{}
@@ -5987,6 +6046,7 @@ func (m *Monitor) syncUnifiedPhysicalDiskMetrics(store ResourceStoreInterface) {
RPM: resource.PhysicalDisk.RPM,
Used: resource.PhysicalDisk.Used,
SmartAttributes: smartAttributesFromUnifiedMeta(resource.PhysicalDisk.SMART),
Collection: diskinventory.CloneStatus(resource.PhysicalDisk.Collection),
LastChecked: resource.LastSeen,
}
if disk.Serial == "" {
internal/monitoring/monitor_agents.go
+2 -2
View File
@@ -4086,7 +4086,7 @@ func (m *Monitor) writeHostSMARTMetrics(host models.Host, now time.Time) {
continue
}
if disk.Temperature > 0 {
if diskTemperatureCollected(disk.Temperature, disk.Collection) {
m.metricsStore.Write("disk", resourceID, "smart_temp", float64(disk.Temperature), now)
}
@@ -5090,7 +5090,7 @@ func (m *Monitor) evaluateHostAgents(now time.Time) {
Bool("lastSeenZero", host.LastSeen.IsZero()).
Msg("Host agent appears offline")
}
if expiredHost, changed := m.state.ExpireHostTelemetry(host.ID); expiredHost.ID != "" {
if expiredHost, changed := m.state.ExpireHostTelemetry(host.ID, host.LastSeen); expiredHost.ID != "" {
hostCopy = expiredHost
resourceRefreshNeeded = resourceRefreshNeeded || changed
} else {
internal/monitoring/monitor_alert_handling_test.go
+62
View File
@@ -241,6 +241,68 @@ func TestMonitor_HandleAlertLifecycle_WritesCanonicalChanges(t *testing.T) {
}
}
// A node alert handed to its Pulse agent closes without recovering. The
// resource history and the alert's incident timeline (the Alerts history row
// expansion) must say where it went, not "Alert resolved: Memory usage at 95%".
func TestMonitor_HandleAlertLifecycle_HandoverCloseIsNotARecovery(t *testing.T) {
resourceStore := unifiedresources.NewMemoryStore()
incidentStore := memory.NewIncidentStore(memory.IncidentStoreConfig{})
m := &Monitor{
incidentStore: incidentStore,
resourceStore: unifiedresources.NewMonitorAdapter(unifiedresources.NewRegistry(resourceStore)),
}
incidentStore.SetResourceTimelineStore(m.resourceStore.(memory.IncidentTimelineStore))
startedAt := time.Date(2026, 10, 6, 9, 0, 0, 0, time.UTC)
movedAt := startedAt.Add(2 * time.Hour)
alert := &alerts.Alert{
ID: "pve1-memory",
Type: "memory",
Level: alerts.AlertLevelWarning,
ResourceID: "pve1",
Message: "Memory usage at 95%",
Value: 95,
Threshold: 85,
StartTime: startedAt,
}
m.handleAlertLifecycleEvent(alerts.LifecycleEvent{Type: eventlog.TypeFired, OccurredAt: startedAt, Alert: alert})
closed := alert.Clone()
closed.Resolution = &alerts.AlertResolution{
Reason: alerts.AlertResolutionMovedToAgent,
SuccessorResourceID: "agent-pve1",
SuccessorName: "pve1 (Host Agent)",
}
m.handleAlertLifecycleEvent(alerts.LifecycleEvent{Type: eventlog.TypeResolved, OccurredAt: movedAt, Alert: closed})
summary := "Alert moved to pve1 (Host Agent). This is not a recovery: check the agent for the current reading."
changes, err := resourceStore.GetRecentChanges("pve1", time.Time{}, 10)
require.NoError(t, err)
require.Len(t, changes, 2)
require.Equal(t, unifiedresources.ChangeAlertResolved, changes[0].Kind)
require.Equal(t, summary, changes[0].Reason)
require.Equal(t, "moved_to_agent", changes[0].Metadata[unifiedresources.MetadataAlertResolution])
require.Equal(t, unifiedresources.ChangeAlertFired, changes[1].Kind)
require.NotContains(t, changes[1].Metadata, unifiedresources.MetadataAlertResolution)
timeline := incidentStore.GetTimelineByAlertAt(alert.ID, startedAt)
require.NotNil(t, timeline)
require.Len(t, timeline.Events, 2)
require.Equal(t, memory.IncidentEventAlertResolved, timeline.Events[1].Type)
require.Equal(t, summary, timeline.Events[1].Summary)
// An ordinary recovery keeps its existing wording.
recovered := alert.Clone()
recovered.ID = "pve1-cpu"
recovered.Type = "cpu"
recovered.Message = "CPU usage at 90%"
m.handleAlertLifecycleEvent(alerts.LifecycleEvent{Type: eventlog.TypeFired, OccurredAt: startedAt, Alert: recovered})
m.handleAlertLifecycleEvent(alerts.LifecycleEvent{Type: eventlog.TypeResolved, OccurredAt: movedAt.Add(time.Minute), Alert: recovered})
recoveredTimeline := incidentStore.GetTimelineByAlertAt(recovered.ID, startedAt)
require.NotNil(t, recoveredTimeline)
require.Len(t, recoveredTimeline.Events, 2)
require.Equal(t, "Alert resolved", recoveredTimeline.Events[1].Summary)
}
func TestPausedDeliveryStillBuildsTimelineThroughRealAlertLifecycle(t *testing.T) {
manager := alerts.NewManagerWithDataDir(t.TempDir())
t.Cleanup(manager.Stop)
internal/monitoring/monitor_alerts.go
+7 -2
View File
@@ -576,7 +576,7 @@ func (m *Monitor) recordAlertTimelineChange(alert *alerts.Alert, kind unifiedres
return
}
change := unifiedresources.BuildAlertTimelineChange(alert.ResourceID, kind, occurredAt, actor, unifiedresources.AlertTimelineChange{
timelineChange := unifiedresources.AlertTimelineChange{
AlertIdentifier: alert.ID,
AlertStartedAt: alert.StartTime,
AlertType: alert.Type,
@@ -585,7 +585,12 @@ func (m *Monitor) recordAlertTimelineChange(alert *alerts.Alert, kind unifiedres
AlertValue: alert.Value,
AlertThreshold: alert.Threshold,
AlertMetadata: alert.Metadata,
})
}
if summary := alert.Resolution.Summary(); summary != "" {
timelineChange.ResolutionReason = string(alert.Resolution.Reason)
timelineChange.ResolutionSummary = summary
}
change := unifiedresources.BuildAlertTimelineChange(alert.ResourceID, kind, occurredAt, actor, timelineChange)
if change == nil {
return
}
internal/monitoring/monitor_host_agents_test.go
+69
View File
@@ -24,6 +24,7 @@ import (
agentsdocker "github.com/rcourtman/pulse-go-rewrite/pkg/agents/docker"
agentshost "github.com/rcourtman/pulse-go-rewrite/pkg/agents/host"
"github.com/rcourtman/pulse-go-rewrite/pkg/metrics"
"github.com/rcourtman/pulse-go-rewrite/pkg/proxmox"
)
func TestNewMonitorRoutesStartupCustomSensorWarningBeforeStart(t *testing.T) {
@@ -6843,3 +6844,71 @@ func TestBroadcastProjectionListsRegistryOnceAndKeepsLiveChanges(t *testing.T) {
t.Fatal("removed host lifecycle surface disappeared")
}
}
// A host agent linked to a Proxmox node merges into one read-state row that
// every PVE poll keeps online and fresh. Once the agent stops reporting, its
// retained filesystem summary must stop standing in for the node's disk (and
// through it disk alerts and history), and the projected agent must read as
// offline with its own last report, so its connection shows the missed
// heartbeat instead of the PVE poll time.
func TestSilentLinkedAgentStopsFeedingNodeDisk(t *testing.T) {
state := models.NewState()
adapter := unifiedresources.NewMonitorAdapter(unifiedresources.NewRegistry(nil))
m := &Monitor{config: &config.Config{}, state: state, resourceStore: adapter}
node := models.Node{
ID: "pve1-node1", Name: "node1", Instance: "pve1", Status: "online",
ConnectionHealth: "healthy", LinkedAgentID: "agent-1",
}
rootFS := &proxmox.NodeStatus{RootFS: &proxmox.RootFS{Total: 200, Used: 50, Free: 150}}
// poll records a PVE poll that just saw the node, with the agent as of its
// last report, and resolves the node's disk the way pollPVENode does.
poll := func(agentLastReport time.Time) (models.Disk, string, models.Host) {
t.Helper()
polledNode := node
polledNode.LastSeen = time.Now()
state.UpdateNodes([]models.Node{polledNode})
state.UpsertHost(models.Host{
ID: "agent-1", Hostname: "node1", Status: "online", LinkedNodeID: node.ID,
IntervalSeconds: 30, LastSeen: agentLastReport,
Disks: []models.Disk{{Mountpoint: "/", Type: "ext4", Total: 1000, Used: 900, Free: 100, Usage: 90}},
})
m.evaluateHostAgents(time.Now())
m.refreshUnifiedResourceStoreAfterAgentStateChange()
disk, source := m.resolveNodeDisk("pve1", node.ID, node.Name, proxmox.Node{Node: node.Name}, rootFS)
for _, host := range m.HostsSnapshot() {
if host.ID == "agent-1" {
return disk, source, host
}
}
t.Fatal("HostsSnapshot does not list agent-1")
return disk, source, models.Host{}
}
agentReport := time.Now().Add(-10 * time.Second)
disk, source, host := poll(agentReport)
if source != "agent" || disk.Used != 900 {
t.Fatalf("reporting agent: node disk = %+v from %q, want the agent's summary", disk, source)
}
if host.Status != "online" || !host.LastSeen.Equal(agentReport) {
t.Fatalf("reporting agent projected as %q last seen %s, want online at its report %s", host.Status, host.LastSeen, agentReport)
}
silentSince := time.Now().Add(-hostAgentHealthWindow(30) - time.Minute)
disk, source, host = poll(silentSince)
if source != "node-status-rootfs" || disk.Used != 50 {
t.Fatalf("silent agent: node disk = %+v from %q, want Proxmox rootfs", disk, source)
}
if linked := m.linkedHostForNode("pve1", node.ID, node.Name); linked != nil {
t.Fatalf("silent agent still linked to the node for disk and dataset data: %+v", linked)
}
if host.Status != "offline" || !host.LastSeen.Equal(silentSince) {
t.Fatalf("silent agent projected as %q last seen %s, want offline at its last report %s", host.Status, host.LastSeen, silentSince)
}
disk, source, host = poll(time.Now())
if source != "agent" || disk.Used != 900 || host.Status != "online" {
t.Fatalf("agent reporting again: node disk = %+v from %q, agent %q; want the agent's summary and online", disk, source, host.Status)
}
}
internal/monitoring/physical_disk_roundtrip_test.go
+307
View File
@@ -11,6 +11,9 @@ import (
"github.com/rcourtman/pulse-go-rewrite/internal/models"
"github.com/rcourtman/pulse-go-rewrite/internal/notifications"
"github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources"
"github.com/rcourtman/pulse-go-rewrite/pkg/diskinventory"
"github.com/rcourtman/pulse-go-rewrite/pkg/metrics"
"github.com/rcourtman/pulse-go-rewrite/pkg/proxmox"
)
// Monitor construction restores notification destinations before admitting
@@ -326,3 +329,307 @@ func TestMergeHostAgentSMARTIntoDisks_AgentWearoutFillsUnreportedPVELife(t *test
t.Fatalf("agent endurance did not fill unreported PVE life: wearout=%d", result[0].Wearout)
}
}
type silentAgentDiskPVEClient struct {
fakeStorageClient
}
// The Proxmox inventory carries no temperature or I/O counters; for this disk
// both come only from the node's linked host agent.
func (*silentAgentDiskPVEClient) GetDisks(ctx context.Context, node string) ([]proxmox.Disk, error) {
return []proxmox.Disk{{
DevPath: "/dev/sdb",
Model: "WDC WD40EFRX",
Serial: "WD-SILENT1",
Type: "hdd",
Health: "PASSED",
Wearout: 100,
Size: 4000787030016,
}}, nil
}
// A host agent that stops reporting keeps its last SMART rows in state. Once
// its reporting lease expires, the linked Proxmox disk may retain the agent's
// last temperature and I/O counters, but the disk state, the canonical
// resource and temperature history must stop treating them as collected.
func TestSilentLinkedAgentSMARTIsRetainedButNotPresentedAsCollected(t *testing.T) {
t.Setenv("PULSE_DATA_DIR", t.TempDir())
now := time.Now()
state := models.NewState()
state.UpdateNodesForInstance("pve1", []models.Node{{
ID: "pve1-node1", Name: "node1", Instance: "pve1", Status: "online",
LastSeen: now, LinkedAgentID: "agent-1",
}})
report := func(seen time.Time, temperature int, readBytes uint64) {
state.UpsertHost(models.Host{
ID: "agent-1", Hostname: "node1", LinkedNodeID: "pve1-node1",
Status: "online", IntervalSeconds: 30, LastSeen: seen,
Sensors: models.HostSensorSummary{SMART: []models.HostDiskSMART{{
Device: "/dev/sdb", Serial: "WD-SILENT1", Type: "sata", Health: "PASSED",
Temperature: temperature,
IO: &models.DiskIO{Device: "sdb", ReadBytes: readBytes},
Collection: &diskinventory.CollectionStatus{
Serial: diskinventory.Available("smartctl"),
Temperature: diskinventory.Available("smartctl"),
IO: diskinventory.Available("kernel_diskstats"),
},
}}},
})
}
alertManager := alerts.NewManager()
t.Cleanup(alertManager.Stop)
storeConfig := metrics.DefaultConfig(t.TempDir())
storeConfig.FlushInterval = time.Hour
store, err := metrics.NewStore(storeConfig)
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { _ = store.Close() })
adapter := unifiedresources.NewMonitorAdapter(unifiedresources.NewRegistry(nil))
history := NewMetricsHistory(100, time.Hour)
m := &Monitor{
state: state, resourceStore: adapter, metricsHistory: history, metricsStore: store,
alertManager: alertManager, startTime: now.Add(-time.Hour),
lastPhysicalDiskPoll: make(map[string]time.Time),
}
pveNodes := []proxmox.Node{{Node: "node1", Status: "online"}}
nodeStatus := map[string]string{"node1": "online"}
disk := func() models.PhysicalDisk {
t.Helper()
disks := state.GetSnapshot().PhysicalDisks
if len(disks) != 1 {
t.Fatalf("physical disks = %+v, want one", disks)
}
return disks[0]
}
fullPoll := func() models.PhysicalDisk {
t.Helper()
adapter.PopulateFromSnapshot(state.GetSnapshot())
started := time.Now()
delete(m.lastPhysicalDiskPoll, "pve1")
m.maybePollPhysicalDisksAsync(context.Background(), "pve1", &config.PVEInstance{}, &silentAgentDiskPVEClient{}, pveNodes, nodeStatus, nil)
deadline := time.Now().Add(3 * time.Second)
for {
if disks := state.GetSnapshot().PhysicalDisks; len(disks) == 1 && !disks[0].LastChecked.Before(started) {
return disks[0]
}
if time.Now().After(deadline) {
t.Fatal("full physical disk poll did not land in state")
}
time.Sleep(10 * time.Millisecond)
}
}
skippedPoll := func() models.PhysicalDisk {
t.Helper()
adapter.PopulateFromSnapshot(state.GetSnapshot())
m.lastPhysicalDiskPoll["pve1"] = time.Now()
m.maybePollPhysicalDisksAsync(context.Background(), "pve1", &config.PVEInstance{}, nil, nil, nil, nil)
return disk()
}
// Chart samples, and the persisted values. The store keys samples by
// second, so polls run back to back in this test can share one row.
temperatureSamples := func(d models.PhysicalDisk) (int, []float64) {
t.Helper()
id := unifiedresources.PhysicalDiskMetricID(d)
chart := len(history.GetDiskMetrics(id, "smart_temp", time.Hour))
store.Flush()
points, err := store.Query("disk", id, "smart_temp", now.Add(-time.Hour), time.Now().Add(time.Hour), 0)
if err != nil {
t.Fatal(err)
}
stored := make([]float64, 0, len(points))
for _, point := range points {
stored = append(stored, point.Value)
}
return chart, stored
}
canonicalCollection := func() *diskinventory.CollectionStatus {
t.Helper()
adapter.PopulateFromSnapshot(state.GetSnapshot())
views := adapter.PhysicalDisks()
if len(views) != 1 {
t.Fatalf("canonical physical disks = %d, want one", len(views))
}
if views[0].Temperature() != 41 {
t.Fatalf("canonical disk temperature = %d, want the retained 41", views[0].Temperature())
}
return views[0].Collection()
}
assertRetainedNotCollected := func(step string, got models.PhysicalDisk) {
t.Helper()
if got.Temperature != 41 || got.IO == nil || got.IO.ReadBytes != 1000 {
t.Fatalf("%s: last-known temperature and I/O were not retained: temp=%d io=%+v", step, got.Temperature, got.IO)
}
if got.Collection == nil ||
got.Collection.Temperature.State != diskinventory.FieldUnavailable ||
got.Collection.IO.State != diskinventory.FieldUnavailable {
t.Fatalf("%s: silent agent's readings presented as collected: %+v", step, got.Collection)
}
}
report(now, 41, 1000)
got := fullPoll()
if got.Temperature != 41 || got.Collection == nil || got.Collection.Temperature.State != diskinventory.FieldAvailable {
t.Fatalf("reporting agent temperature not merged: temp=%d collection=%+v", got.Temperature, got.Collection)
}
if chart, stored := temperatureSamples(got); chart != 1 || len(stored) != 1 || stored[0] != 41 {
t.Fatalf("reporting agent temperature history: chart=%d stored=%v, want one 41 sample", chart, stored)
}
// The agent stops reporting; its lease expires on the next evaluation.
state.TouchHost("agent-1", now.Add(-hostAgentHealthWindow(30)-time.Minute))
m.evaluateHostAgents(now)
// No disk poll runs when the whole host is down (the PVE poll fails before
// reaching it), so the canonical disk must reflect the expiry on its own
// even though the Proxmox disk row still carries the earlier merge.
if collection := canonicalCollection(); collection == nil ||
collection.Temperature.State != diskinventory.FieldUnavailable ||
collection.IO.State != diskinventory.FieldUnavailable {
t.Fatalf("canonical disk presents the silent agent's readings as collected before any disk poll: %+v", collection)
}
assertRetainedNotCollected("skipped poll", skippedPoll())
got = fullPoll()
assertRetainedNotCollected("full poll", got)
if chart, stored := temperatureSamples(got); chart != 1 || len(stored) != 1 {
t.Fatalf("silent agent's retained temperature was recorded as new history: chart=%d stored=%v", chart, stored)
}
if collection := canonicalCollection(); collection == nil ||
collection.Temperature.State != diskinventory.FieldUnavailable ||
collection.IO.State != diskinventory.FieldUnavailable {
t.Fatalf("canonical disk presents the silent agent's readings as collected: %+v", collection)
}
// A resumed agent is collected again.
report(time.Now(), 39, 2000)
m.evaluateHostAgents(time.Now())
got = fullPoll()
if got.Temperature != 39 || got.Collection.Temperature.State != diskinventory.FieldAvailable ||
got.IO == nil || got.IO.ReadBytes != 2000 || got.Collection.IO.State != diskinventory.FieldAvailable {
t.Fatalf("resumed agent readings not collected: temp=%d io=%+v collection=%+v", got.Temperature, got.IO, got.Collection)
}
if chart, stored := temperatureSamples(got); chart != 2 || len(stored) == 0 || stored[len(stored)-1] != 39 {
t.Fatalf("resumed agent temperature history: chart=%d stored=%v, want a new 39 sample", chart, stored)
}
}
// Agents before 6.2 send SMART rows without collection provenance. Their
// temperature is still a collected reading (history records it), and once the
// lease expires the disk must stop presenting it as collected even when a
// skipped disk poll carries the previous merge forward.
func TestMergeHostAgentSMARTIntoDisks_LegacyAgentTemperatureFollowsLease(t *testing.T) {
proxmoxDisk := models.PhysicalDisk{
ID: "pve1-node1-sdb", Node: "node1", Instance: "pve1", DevPath: "/dev/sdb", Serial: "LEGACY1",
Collection: &diskinventory.CollectionStatus{
Temperature: diskinventory.Unsupported("proxmox_disks", "Proxmox disk inventory does not expose temperature"),
},
}
nodes := []models.Node{{Name: "node1", LinkedAgentID: "agent-1"}}
state := models.NewState()
state.UpsertHost(models.Host{ID: "agent-1", Sensors: models.HostSensorSummary{SMART: []models.HostDiskSMART{{
Device: "/dev/sdb", Serial: "LEGACY1", Temperature: 38,
}}}})
reporting := mergeHostAgentSMARTIntoDisks([]models.PhysicalDisk{proxmoxDisk}, nodes, state.GetHosts())[0]
if reporting.Temperature != 38 || reporting.Collection.Temperature != diskinventory.Available(hostAgentLegacySource) {
t.Fatalf("legacy agent temperature not recorded as collected: temp=%d collection=%+v", reporting.Temperature, reporting.Collection)
}
if !diskTemperatureCollected(reporting.Temperature, reporting.Collection) {
t.Fatal("a reporting legacy agent's temperature must reach history")
}
state.ExpireHostTelemetry("agent-1", time.Time{})
for name, disk := range map[string]models.PhysicalDisk{
"full poll": proxmoxDisk,
"skipped poll": reporting,
} {
got := mergeHostAgentSMARTIntoDisks([]models.PhysicalDisk{disk}, nodes, state.GetHosts())[0]
if got.Temperature != 38 || got.Collection.Temperature.State != diskinventory.FieldUnavailable {
t.Fatalf("%s: silent legacy agent temperature presented as collected: temp=%d collection=%+v", name, got.Temperature, got.Collection)
}
if diskTemperatureCollected(got.Temperature, got.Collection) {
t.Fatalf("%s: a silent legacy agent's retained temperature must not reach history", name)
}
}
}
// A disk that goes into standby keeps its pre-sleep temperature as retained
// evidence (preserveUnavailablePhysicalDiskEvidence), with the agent's standby
// state. Full disk polls during standby must not record that value again.
func TestStandbyDiskRetainedTemperatureIsNotRecordedAsHistory(t *testing.T) {
t.Setenv("PULSE_DATA_DIR", t.TempDir())
state := models.NewState()
state.UpdateNodesForInstance("pve1", []models.Node{{
ID: "pve1-node1", Name: "node1", Instance: "pve1", Status: "online",
LastSeen: time.Now(), LinkedAgentID: "agent-1",
}})
report := func(row models.HostDiskSMART) {
row.Device, row.Serial, row.Type, row.Health = "/dev/sdb", "WD-SILENT1", "sata", "PASSED"
state.UpsertHost(models.Host{
ID: "agent-1", Hostname: "node1", LinkedNodeID: "pve1-node1", Status: "online",
IntervalSeconds: 30, LastSeen: time.Now(),
Sensors: models.HostSensorSummary{SMART: []models.HostDiskSMART{row}},
})
}
alertManager := alerts.NewManager()
t.Cleanup(alertManager.Stop)
storeConfig := metrics.DefaultConfig(t.TempDir())
storeConfig.FlushInterval = time.Hour
store, err := metrics.NewStore(storeConfig)
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { _ = store.Close() })
adapter := unifiedresources.NewMonitorAdapter(unifiedresources.NewRegistry(nil))
history := NewMetricsHistory(100, time.Hour)
m := &Monitor{
state: state, resourceStore: adapter, metricsHistory: history, metricsStore: store,
alertManager: alertManager, startTime: time.Now().Add(-time.Hour),
lastPhysicalDiskPoll: make(map[string]time.Time),
}
fullPoll := func() models.PhysicalDisk {
t.Helper()
adapter.PopulateFromSnapshot(state.GetSnapshot())
started := time.Now()
delete(m.lastPhysicalDiskPoll, "pve1")
m.maybePollPhysicalDisksAsync(context.Background(), "pve1", &config.PVEInstance{}, &silentAgentDiskPVEClient{},
[]proxmox.Node{{Node: "node1", Status: "online"}}, map[string]string{"node1": "online"}, nil)
deadline := time.Now().Add(3 * time.Second)
for {
if disks := state.GetSnapshot().PhysicalDisks; len(disks) == 1 && !disks[0].LastChecked.Before(started) {
return disks[0]
}
if time.Now().After(deadline) {
t.Fatal("full physical disk poll did not land in state")
}
time.Sleep(10 * time.Millisecond)
}
}
samples := func(d models.PhysicalDisk) int {
return len(history.GetDiskMetrics(unifiedresources.PhysicalDiskMetricID(d), "smart_temp", time.Hour))
}
report(models.HostDiskSMART{Temperature: 38, Collection: &diskinventory.CollectionStatus{
Temperature: diskinventory.Available("smartctl"),
}})
if got := fullPoll(); got.Temperature != 38 || samples(got) != 1 {
t.Fatalf("awake disk temperature not recorded: temp=%d samples=%d", got.Temperature, samples(got))
}
// The agent keeps reporting, but the disk is now asleep.
report(models.HostDiskSMART{Standby: true, Collection: &diskinventory.CollectionStatus{
Temperature: diskinventory.Unavailable("smartctl", "disk is in standby"),
}})
for poll := 0; poll < 2; poll++ {
got := fullPoll()
if got.Temperature != 38 || got.Collection == nil || got.Collection.Temperature.State != diskinventory.FieldUnavailable {
t.Fatalf("poll %d: standby disk lost its retained temperature or claims it was collected: temp=%d collection=%+v",
poll, got.Temperature, got.Collection)
}
if n := samples(got); n != 1 {
t.Fatalf("poll %d: standby disk's pre-sleep temperature recorded as new history: samples=%d", poll, n)
}
}
}
internal/notifications/notifications.go
+69 -14
View File
@@ -432,6 +432,47 @@ func annotateResolvedMetadata(alert *alerts.Alert, resolvedAt time.Time) {
alert.Metadata[metadataResolvedAt] = resolvedAt.Format(time.RFC3339)
}
// resolvedAlertMessage is the one-line account of a resolved alert. A close
// that was not a recovery, such as a node alert that moved to the Pulse agent
// now monitoring the machine, says what happened instead of "now healthy".
func resolvedAlertMessage(alert *alerts.Alert) string {
if summary := resolvedAlertNotRecoveredSummary(alert); summary != "" {
return summary
}
return fmt.Sprintf("%s on %s is now healthy", alert.ResourceName, alert.Node)
}
// resolvedAlertNotRecoveredSummary names the alert in its resolution summary,
// so one of several closes for a machine says which alert moved. It is empty
// for an ordinary recovery.
func resolvedAlertNotRecoveredSummary(alert *alerts.Alert) string {
label := alertTypeDisplay(alert.Type)
if label != "" {
label += " alert"
}
return alert.Resolution.Describe(label)
}
// resolvedAlertsIncludeNotRecovered reports whether any close in a resolved
// notification was not a recovery, so recovery visuals must not be used.
func resolvedAlertsIncludeNotRecovered(alertList []*alerts.Alert) bool {
for _, alert := range alertList {
if alert != nil && alert.Resolution.Outcome() != "" {
return true
}
}
return false
}
// resolvedAlertListEntry names one alert in a grouped recovery list.
func resolvedAlertListEntry(alert *alerts.Alert) string {
entry := fmt.Sprintf("%s on %s", alert.ResourceName, alert.Node)
if outcome := alert.Resolution.Outcome(); outcome != "" {
entry += ", " + outcome
}
return entry
}
func quietHoursReplayAtForAlert(alert *alerts.Alert, now time.Time) *time.Time {
if alert == nil || alert.Metadata == nil {
return nil
@@ -2267,6 +2308,9 @@ func buildResolvedNotificationContent(alertList []*alerts.Alert, resolvedAt time
resolvedLabel := resolvedAt.Format(time.RFC3339)
title := fmt.Sprintf("Pulse alert resolved: %s", primary.ResourceName)
if primary.Resolution.Outcome() != "" {
title = fmt.Sprintf("Pulse alert moved: %s", primary.ResourceName)
}
if len(validAlerts) > 1 {
title = fmt.Sprintf("Pulse alerts resolved (%d)", len(validAlerts))
}
@@ -2282,6 +2326,10 @@ func buildResolvedNotificationContent(alertList []*alerts.Alert, resolvedAt time
bodyBuilder.WriteString(alert.Message)
bodyBuilder.WriteString("\n")
}
if summary := resolvedAlertNotRecoveredSummary(alert); summary != "" {
bodyBuilder.WriteString(summary)
bodyBuilder.WriteString("\n")
}
if !alert.StartTime.IsZero() {
bodyBuilder.WriteString("Started: ")
bodyBuilder.WriteString(alert.StartTime.Format(time.RFC3339))
@@ -2980,14 +3028,15 @@ func (n *NotificationManager) sendResolvedWebhook(webhook WebhookConfig, alertLi
data.ResolvedAt = resolvedAt.Format(time.RFC3339)
data.ResolvedAtISO = resolvedAt.Format(time.RFC3339)
data.Duration = formatWebhookDuration(resolvedAt.Sub(alert.StartTime))
data.Message = fmt.Sprintf("%s on %s is now healthy", alert.ResourceName, alert.Node)
data.Message = resolvedAlertMessage(alert)
data.NotRecovered = resolvedAlertsIncludeNotRecovered(alertList)
data.AlertCount = len(alertList)
data.Alerts = alertList
if len(alertList) > 1 {
names := make([]string, 0, len(alertList))
for _, a := range alertList {
if a != nil {
names = append(names, fmt.Sprintf("%s on %s", a.ResourceName, a.Node))
names = append(names, resolvedAlertListEntry(a))
}
}
data.Message = fmt.Sprintf("%d alerts resolved: %s", len(alertList), strings.Join(names, "; "))
@@ -3036,28 +3085,34 @@ func (n *NotificationManager) sendResolvedWebhookNtfy(webhook WebhookConfig, ale
return fmt.Errorf("rate limit exceeded for webhook %s", webhook.Name)
}
// Build plain-text body
// Build plain-text body and title. A single alert that moved rather than
// recovered says so in both, and no batch holding such a close carries
// the green check.
var body strings.Builder
title := "RESOLVED"
tags := "white_check_mark,pulse,resolved"
if resolvedAlertsIncludeNotRecovered(alertList) {
tags = "arrow_right,pulse,resolved"
}
if len(alertList) == 1 && alertList[0] != nil {
a := alertList[0]
fmt.Fprintf(&body, "Resolved: %s on %s is now healthy", a.ResourceName, a.Node)
title = fmt.Sprintf("RESOLVED: %s", a.ResourceName)
if summary := resolvedAlertNotRecoveredSummary(a); summary != "" {
title = fmt.Sprintf("MOVED: %s", a.ResourceName)
body.WriteString(summary)
} else {
fmt.Fprintf(&body, "Resolved: %s", resolvedAlertMessage(a))
}
} else {
title = fmt.Sprintf("RESOLVED: %d alerts", len(alertList))
fmt.Fprintf(&body, "%d alerts resolved at %s:\n", len(alertList), resolvedAt.Format(time.RFC822))
for _, a := range alertList {
if a != nil {
fmt.Fprintf(&body, "- %s on %s\n", a.ResourceName, a.Node)
fmt.Fprintf(&body, "- %s\n", resolvedAlertListEntry(a))
}
}
}
// Build title
title := "RESOLVED"
if len(alertList) == 1 && alertList[0] != nil {
title = fmt.Sprintf("RESOLVED: %s", alertList[0].ResourceName)
} else {
title = fmt.Sprintf("RESOLVED: %d alerts", len(alertList))
}
method := webhook.Method
if method == "" {
method = "POST"
@@ -3071,7 +3126,7 @@ func (n *NotificationManager) sendResolvedWebhookNtfy(webhook WebhookConfig, ale
req.Header.Set("Content-Type", "text/plain")
req.Header.Set("Title", title)
req.Header.Set("Priority", "default")
req.Header.Set("Tags", "white_check_mark,pulse,resolved")
req.Header.Set("Tags", tags)
req.Header.Set("User-Agent", "Pulse-Monitoring/2.0")
// Apply any custom headers from webhook config
internal/notifications/resolved_handover_test.go
+239
View File
@@ -0,0 +1,239 @@
package notifications
import (
"encoding/json"
"io"
"net/http"
"strings"
"testing"
"time"
"github.com/rcourtman/pulse-go-rewrite/internal/alerts"
)
const movedSummary = "Memory alert moved to pve1 (Host Agent). This is not a recovery: check the agent for the current reading."
// movedNodeMemoryAlert is a node memory alert that closed because the Pulse
// agent linked to the node took over its usage alerts, not because memory
// came back down.
func movedNodeMemoryAlert() *alerts.Alert {
return &alerts.Alert{
ID: "pve1-memory",
Type: "memory",
Level: alerts.AlertLevelWarning,
ResourceName: "pve1",
ResourceID: "pve1",
Node: "pve1",
Instance: "https://pve.local:8006",
Message: "Node memory at 95.0%",
Value: 95,
Threshold: 90,
StartTime: time.Now().Add(-10 * time.Minute),
Resolution: &alerts.AlertResolution{
Reason: alerts.AlertResolutionMovedToAgent,
SuccessorResourceID: "agent-pve1",
SuccessorName: "pve1 (Host Agent)",
},
}
}
// Exercise the normal entry points: the firing goes out, its receipt gates the
// close, and the close is persisted in the delivery queue and read back before
// it is rendered.
func TestMovedAlertCloseIsDeliveredAsMovedNotRecovered(t *testing.T) {
type message struct{ title, body, tags string }
received := make(chan message, 4)
server := newIPv4HTTPServer(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
body, err := io.ReadAll(r.Body)
if err != nil {
t.Errorf("read notification: %v", err)
}
received <- message{r.Header.Get("Title"), string(body), r.Header.Get("Tags")}
w.WriteHeader(http.StatusAccepted)
}))
defer server.Close()
webhook := WebhookConfig{ID: "ops", Name: "ops", URL: server.URL + "/topic", Enabled: true, Service: "ntfy"}
m := NewNotificationManagerWithDataDir("", t.TempDir())
t.Cleanup(m.Stop)
m.webhookClient = server.Client()
if err := m.UpdateAllowedPrivateCIDRs("127.0.0.1/32"); err != nil {
t.Fatal(err)
}
m.SetGroupingConfig(false, 0, false, false)
m.SetNotifyOnResolve(true)
m.AddWebhook(webhook)
next := func() message {
t.Helper()
select {
case msg := <-received:
return msg
case <-time.After(10 * time.Second):
t.Fatal("notification not received")
return message{}
}
}
alert := movedNodeMemoryAlert()
firing := alert.Clone()
firing.Resolution = nil
m.SendAlert(firing)
next()
recoveryJob := notificationDeliveryJob{Type: "webhook", Event: eventResolved, Alerts: []*alerts.Alert{alert}, WebhookConfig: &webhook}
deadline := time.Now().Add(10 * time.Second)
for len(m.filterResolvedJobsByDeliveryReceipt([]notificationDeliveryJob{recoveryJob})) != 1 {
if time.Now().After(deadline) {
t.Fatal("firing receipt never recorded")
}
time.Sleep(time.Millisecond)
}
m.SendResolvedAlert(&alerts.ResolvedAlert{Alert: alert, ResolvedTime: time.Now()})
closing := next()
if closing.title != "MOVED: pve1" || closing.body != movedSummary {
t.Fatalf("close = %+v, want MOVED title and the handover summary", closing)
}
if strings.Contains(closing.body, "healthy") || strings.Contains(closing.tags, "white_check_mark") {
t.Fatalf("close reads as a recovery: %+v", closing)
}
}
// captureResolvedWebhook renders one resolved delivery through an isolated
// manager and returns what the destination received.
func captureResolvedWebhook(t *testing.T, service string, alertList []*alerts.Alert) ([]byte, http.Header) {
t.Helper()
type request struct {
body []byte
header http.Header
}
received := make(chan request, 1)
server := newIPv4HTTPServer(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
body, _ := io.ReadAll(r.Body)
received <- request{body: body, header: r.Header.Clone()}
w.WriteHeader(http.StatusOK)
}))
defer server.Close()
nm := NewNotificationManagerWithDataDir("https://pulse.local", t.TempDir())
t.Cleanup(nm.Stop)
if err := nm.UpdateAllowedPrivateCIDRs("127.0.0.1"); err != nil {
t.Fatal(err)
}
webhook := WebhookConfig{Name: service, URL: server.URL + "/hook", Enabled: true, Service: service,
CustomFields: map[string]string{"routing_key": "rk"}}
if err := nm.sendResolvedWebhook(webhook, alertList, time.Now()); err != nil {
t.Fatalf("sendResolvedWebhook %s: %v", service, err)
}
select {
case got := <-received:
return got.body, got.header
case <-time.After(5 * time.Second):
t.Fatalf("%s destination received nothing", service)
return nil, nil
}
}
func decodeResolvedPayload(t *testing.T, service string, body []byte) map[string]any {
t.Helper()
var payload map[string]any
if err := json.Unmarshal(body, &payload); err != nil {
t.Fatalf("unmarshal %s payload: %v\n%s", service, err, body)
}
return payload
}
func TestMovedAlertCloseRendersHonestlyOnEveryChannel(t *testing.T) {
recovered := &alerts.Alert{ID: "vm-cpu", Type: "cpu", ResourceName: "web-vm-01", Node: "pve1", StartTime: time.Now().Add(-time.Minute)}
t.Run("discord says moved without the recovery colour", func(t *testing.T) {
body, _ := captureResolvedWebhook(t, "discord", []*alerts.Alert{movedNodeMemoryAlert()})
embed := decodeResolvedPayload(t, "discord", body)["embeds"].([]any)[0].(map[string]any)
description, _ := embed["description"].(string)
if !strings.Contains(description, movedSummary) || strings.Contains(description, "healthy") {
t.Fatalf("description = %q, want the handover summary", description)
}
if embed["color"] != float64(3447003) {
t.Fatalf("embed color = %v, want the neutral 3447003, not recovery green", embed["color"])
}
})
t.Run("teams cards drop recovery green", func(t *testing.T) {
body, _ := captureResolvedWebhook(t, "teams", []*alerts.Alert{movedNodeMemoryAlert()})
if got := decodeResolvedPayload(t, "teams", body)["themeColor"]; got != "0076D7" {
t.Fatalf("teams themeColor = %v, want 0076D7", got)
}
body, _ = captureResolvedWebhook(t, "teams-adaptive", []*alerts.Alert{movedNodeMemoryAlert()})
if !strings.Contains(string(body), `"color": "Accent"`) || strings.Contains(string(body), `"color": "Good"`) {
t.Fatalf("teams adaptive card = %s, want the Accent title colour", body)
}
})
t.Run("mattermost drops the check mark", func(t *testing.T) {
body, _ := captureResolvedWebhook(t, "mattermost", []*alerts.Alert{movedNodeMemoryAlert()})
text, _ := decodeResolvedPayload(t, "mattermost", body)["text"].(string)
if strings.Contains(text, ":white_check_mark:") || !strings.Contains(text, ":arrow_right: **RESOLVED**") || !strings.Contains(text, movedSummary) {
t.Fatalf("mattermost text = %q", text)
}
})
t.Run("pagerduty still closes the incident", func(t *testing.T) {
body, _ := captureResolvedWebhook(t, "pagerduty", []*alerts.Alert{movedNodeMemoryAlert()})
payload := decodeResolvedPayload(t, "pagerduty", body)
if payload["event_action"] != "resolve" || payload["dedup_key"] != "pve1-memory" {
t.Fatalf("payload = %v, want a resolve for the node alert's incident", payload)
}
})
t.Run("grouped list names the move", func(t *testing.T) {
body, _ := captureResolvedWebhook(t, "slack", []*alerts.Alert{movedNodeMemoryAlert(), recovered})
if !strings.Contains(string(body), "pve1 on pve1, moved to pve1 (Host Agent)") || !strings.Contains(string(body), "web-vm-01 on pve1") {
t.Fatalf("grouped body = %s, want the moved alert marked and the recovery listed", body)
}
})
t.Run("grouped ntfy batch drops the check mark", func(t *testing.T) {
body, header := captureResolvedWebhook(t, "ntfy", []*alerts.Alert{movedNodeMemoryAlert(), recovered})
if tags := header.Get("Tags"); strings.Contains(tags, "white_check_mark") || !strings.Contains(tags, "arrow_right") {
t.Fatalf("ntfy tags = %q, want no check mark on a batch holding a move", tags)
}
if !strings.Contains(string(body), "- pve1 on pve1, moved to pve1 (Host Agent)\n") {
t.Fatalf("ntfy body = %q", body)
}
})
t.Run("email and apprise say moved", func(t *testing.T) {
title, htmlBody, textBody := buildResolvedNotificationContent([]*alerts.Alert{movedNodeMemoryAlert()}, time.Now(), "")
if title != "Pulse alert moved: pve1" {
t.Fatalf("title = %q", title)
}
if !strings.Contains(textBody, "Node memory at 95.0%\n"+movedSummary+"\n") {
t.Fatalf("text body = %q, want the breach followed by the handover summary", textBody)
}
if !strings.Contains(htmlBody, "This is not a recovery: check the agent for the current reading.") {
t.Fatalf("html body = %q", htmlBody)
}
})
t.Run("ordinary recovery keeps its wording and visuals", func(t *testing.T) {
plain := movedNodeMemoryAlert()
plain.Resolution = nil
if got := resolvedAlertMessage(plain); got != "pve1 on pve1 is now healthy" {
t.Fatalf("resolvedAlertMessage() = %q", got)
}
if title, _, _ := buildResolvedNotificationContent([]*alerts.Alert{plain}, time.Now(), ""); title != "Pulse alert resolved: pve1" {
t.Fatalf("title = %q", title)
}
body, _ := captureResolvedWebhook(t, "discord", []*alerts.Alert{plain})
if embed := decodeResolvedPayload(t, "discord", body)["embeds"].([]any)[0].(map[string]any); embed["color"] != float64(3066993) {
t.Fatalf("ordinary recovery embed color = %v, want green 3066993", embed["color"])
}
body, _ = captureResolvedWebhook(t, "mattermost", []*alerts.Alert{plain})
if text, _ := decodeResolvedPayload(t, "mattermost", body)["text"].(string); !strings.Contains(text, ":white_check_mark: **RESOLVED**") {
t.Fatalf("ordinary mattermost text = %q", text)
}
_, header := captureResolvedWebhook(t, "ntfy", []*alerts.Alert{plain})
if tags := header.Get("Tags"); tags != "white_check_mark,pulse,resolved" {
t.Fatalf("ordinary ntfy tags = %q", tags)
}
})
}
internal/notifications/webhook_enhanced.go
+4
View File
@@ -65,6 +65,10 @@ type WebhookPayloadData struct {
Event string // "alert" or "resolved"
ResolvedAt string // RFC3339 timestamp when alert was resolved
ResolvedAtISO string // Same as ResolvedAt (alias for template consistency)
// NotRecovered marks a resolved notification in which at least one close
// was not a recovery (an alert that moved to a linked Pulse agent), so
// templates can drop recovery colours and check marks.
NotRecovered bool
// Tenant identity (multi-tenant orgs and MSP client runtimes).
// Empty on plain single-tenant installs.
internal/notifications/webhook_templates.go
+4 -4
View File
@@ -56,7 +56,7 @@ func GetWebhookTemplates() []WebhookTemplate {
"embeds": [{
"title": "Resolved: {{.ResourceName | jsonString}}",
"description": "{{.Message | jsonString}}",
"color": 3066993,
"color": {{if .NotRecovered}}3447003{{else}}3066993{{end}},
"fields": [
{"name": "Resource", "value": "{{.ResourceName | jsonString}}", "inline": true},
{"name": "Node", "value": "{{.Node | jsonString}}", "inline": true},
@@ -241,7 +241,7 @@ func GetWebhookTemplates() []WebhookTemplate {
ResolvedPayloadTemplate: `{
"@type": "MessageCard",
"@context": "http://schema.org/extensions",
"themeColor": "2DC72D",
"themeColor": "{{if .NotRecovered}}0076D7{{else}}2DC72D{{end}}",
"summary": "Resolved: {{.ResourceName | jsonString}}",
{{if .Mention}}"text": "{{.Mention | jsonString}}",{{end}}
"sections": [{
@@ -372,7 +372,7 @@ func GetWebhookTemplates() []WebhookTemplate {
"text": "Resolved: {{.ResourceName | jsonString}}",
"weight": "Bolder",
"size": "Large",
"color": "Good"
"color": "{{if .NotRecovered}}Accent{{else}}Good{{end}}"
},
{
"type": "TextBlock",
@@ -521,7 +521,7 @@ View in Pulse: {{.Instance}}`,
ResolvedPayloadTemplate: `{
"username": "Pulse Monitoring",
"icon_url": "https://raw.githubusercontent.com/rcourtman/Pulse/main/frontend-modern/public/android-chrome-192x192.png",
"text": "{{if .Mention}}{{.Mention | jsonString}}\n\n{{end}}:white_check_mark: **RESOLVED**\n\n**{{.ResourceName | jsonString}}** on **{{.Node | jsonString}}**\n\n{{.Message | jsonString}}\n\n| Detail | Value |\n|:-------|:------|\n| Resource | {{.ResourceName | jsonString}} |\n| Node | {{.Node | jsonString}} |\n| Type | {{.Type | title | jsonString}} |\n| Duration | {{.Duration | jsonString}} |\n| Resolved At | {{.ResolvedAt | jsonString}} |\n| Alert Identifier | {{.ID | jsonString}} |"
"text": "{{if .Mention}}{{.Mention | jsonString}}\n\n{{end}}{{if .NotRecovered}}:arrow_right:{{else}}:white_check_mark:{{end}} **RESOLVED**\n\n**{{.ResourceName | jsonString}}** on **{{.Node | jsonString}}**\n\n{{.Message | jsonString}}\n\n| Detail | Value |\n|:-------|:------|\n| Resource | {{.ResourceName | jsonString}} |\n| Node | {{.Node | jsonString}} |\n| Type | {{.Type | title | jsonString}} |\n| Duration | {{.Duration | jsonString}} |\n| Resolved At | {{.ResolvedAt | jsonString}} |\n| Alert Identifier | {{.ID | jsonString}} |"
}`,
Instructions: "1. In Mattermost, go to Integrations > Incoming Webhooks\n2. Create a new webhook and select the channel\n3. Copy the webhook URL and paste it here\n\nNote: This template uses Markdown formatting which is fully supported by Mattermost.",
},
internal/operationaltrust/contracts.go
+14 -2
View File
@@ -493,8 +493,12 @@ const (
TransitionSuppression TransitionCause = "suppression"
TransitionSuppressionExpired TransitionCause = "suppression_expired"
TransitionRecoveryEvidence TransitionCause = "recovery_evidence"
TransitionCollectionStale TransitionCause = "collection_stale"
TransitionCollectionUnknown TransitionCause = "collection_unknown"
// TransitionOwnershipTransferred closes a record whose condition another
// record now owns, such as a node metric a linked Pulse agent took over.
// The condition was not observed to clear, so it is not a recovery.
TransitionOwnershipTransferred TransitionCause = "ownership_transferred"
TransitionCollectionStale TransitionCause = "collection_stale"
TransitionCollectionUnknown TransitionCause = "collection_unknown"
)
func (cause TransitionCause) valid() bool {
@@ -505,6 +509,7 @@ func (cause TransitionCause) valid() bool {
TransitionSuppression,
TransitionSuppressionExpired,
TransitionRecoveryEvidence,
TransitionOwnershipTransferred,
TransitionCollectionStale,
TransitionCollectionUnknown:
return true
@@ -617,6 +622,13 @@ func (transition LifecycleTransition) validateWithoutID() error {
if len(canonicalIDs(transition.EvidenceIDs)) == 0 {
return errors.New("recovery transition requires evidence")
}
case TransitionOwnershipTransferred:
if transition.To != OperationalResolved {
return errors.New("ownership transfer must enter resolved state")
}
if len(canonicalIDs(transition.EvidenceIDs)) == 0 {
return errors.New("ownership transfer requires evidence")
}
case TransitionCollectionStale:
if transition.To != OperationalStale {
return errors.New("collection-stale transition must enter stale state")
internal/operationaltrust/contracts_test.go
+42
View File
@@ -235,6 +235,48 @@ func TestLifecycleTransitionEnforcesOperationalSemantics(t *testing.T) {
}
}
// A close that hands the condition to another record is not a recovery, so it
// has its own cause. It may only close the record, and only with the evidence
// that recorded the handover.
func TestOwnershipTransferOnlyClosesWithEvidence(t *testing.T) {
at := time.Date(2026, 10, 6, 15, 0, 0, 0, time.UTC)
if _, err := NewTransitionID(
"record-1",
OperationalOpen,
OperationalResolved,
at,
TransitionOwnershipTransferred,
"memory-high",
[]string{"evidence-handover"},
); err != nil {
t.Fatalf("open-to-resolved ownership transfer error = %v", err)
}
if _, err := NewTransitionID(
"record-1",
OperationalAcknowledged,
OperationalResolved,
at,
TransitionOwnershipTransferred,
"memory-high",
nil,
); err == nil || !strings.Contains(err.Error(), "requires evidence") {
t.Fatalf("evidenceless ownership transfer error = %v", err)
}
for _, to := range []OperationalState{OperationalResolving, OperationalOpen, OperationalStale} {
if _, err := NewTransitionID(
"record-1",
OperationalAcknowledged,
to,
at,
TransitionOwnershipTransferred,
"memory-high",
[]string{"evidence-handover"},
); err == nil || !strings.Contains(err.Error(), "must enter resolved") {
t.Fatalf("ownership transfer into %q error = %v", to, err)
}
}
}
func TestOperationalRecordRequiresResolutionEvidence(t *testing.T) {
firstObserved := time.Date(2026, 7, 18, 20, 0, 0, 0, time.UTC)
resolvedAt := firstObserved.Add(5 * time.Minute)
internal/unifiedresources/incident_changes.go
+16
View File
@@ -16,6 +16,9 @@ const (
MetadataAlertMessage = "alert_message"
MetadataAlertValue = "alert_value"
MetadataAlertThreshold = "alert_threshold"
// MetadataAlertResolution is the reason code of a resolve that was not a
// recovery, such as "moved_to_agent". Ordinary recoveries omit it.
MetadataAlertResolution = "alert_resolution"
MetadataCommand = "command"
MetadataSuccess = "success"
MetadataOutputExcerpt = "output_excerpt"
@@ -38,6 +41,11 @@ type AlertTimelineChange struct {
AlertValue float64
AlertThreshold float64
AlertMetadata map[string]any
// ResolutionReason and ResolutionSummary describe a resolve that was not
// a recovery. They are empty for an ordinary recovery and ignored on
// every other lifecycle kind.
ResolutionReason string
ResolutionSummary string
}
// BuildAlertTimelineChange constructs a canonical resource change for alert
@@ -74,6 +82,9 @@ func BuildAlertTimelineChange(resourceID string, kind ChangeKind, occurredAt tim
if !alert.AlertStartedAt.IsZero() {
change.Metadata[MetadataAlertStartedAt] = alert.AlertStartedAt.UTC().Format(time.RFC3339Nano)
}
if reason := strings.TrimSpace(alert.ResolutionReason); kind == ChangeAlertResolved && reason != "" {
change.Metadata[MetadataAlertResolution] = reason
}
for key, value := range cloneChangeMetadata(alert.AlertMetadata) {
if _, exists := change.Metadata[key]; exists {
continue
@@ -178,6 +189,11 @@ func alertChangeReason(kind ChangeKind, alert AlertTimelineChange) string {
case ChangeAlertUnsnoozed:
return "Alert notifications resumed"
case ChangeAlertResolved:
// A close that was not a recovery must not read as one: the alert's
// own message describes the condition that is still unresolved.
if summary := strings.TrimSpace(alert.ResolutionSummary); summary != "" {
return summary
}
if message != "" {
return fmt.Sprintf("Alert resolved: %s", message)
}
internal/unifiedresources/registry.go
+9 -1
View File
@@ -3719,11 +3719,19 @@ func (rr *ResourceRegistry) mergeInto(existing *Resource, incoming Resource, sou
if previous.IO != nil {
existing.PhysicalDisk.IO = clonePhysicalDiskIOMeta(previous.IO)
}
existing.PhysicalDisk.Collection = diskinventory.MergeStatus(
// The agent's own report is the later word on fields it supplied;
// the Proxmox row may still carry an earlier copy of them.
existing.PhysicalDisk.Collection = diskinventory.MergeReportedStatus(
incoming.PhysicalDisk.Collection,
previous.Collection,
)
}
if source == SourceAgent && previous != nil {
existing.PhysicalDisk.Collection = diskinventory.MergeReportedStatus(
previous.Collection,
incoming.PhysicalDisk.Collection,
)
}
}
if existing.PhysicalDisk != nil && (mergedPhysicalDisk || len(incoming.Incidents) > 0) {
existing.PhysicalDisk.Risk = physicalDiskRiskFromMeta(existing.PhysicalDisk, existing.Incidents)
internal/unifiedresources/registry_merge_policy_test.go
+77
View File
@@ -6,6 +6,7 @@ import (
"time"
"github.com/rcourtman/pulse-go-rewrite/internal/models"
"github.com/rcourtman/pulse-go-rewrite/pkg/diskinventory"
)
func TestLinkedMergeAllowsOneSidedNodeHostLinkWhenHostnameCorroborates(t *testing.T) {
@@ -557,3 +558,79 @@ func TestProxmoxOneWayLinkRejectsHostLocalEndpointCorroboration(t *testing.T) {
t.Fatal("explicit private report-IP corroboration lost")
}
}
// The Proxmox disk row carries a copy of the agent's collection state from the
// last disk poll. When the agent's own row later withdraws a reading (its
// reporting lease expired), the canonical disk must follow the agent in either
// ingest order, even if no disk poll has refreshed the Proxmox copy (a host that
// is down entirely is never disk-polled).
func TestPhysicalDiskMergeFollowsAgentWithdrawalOfItsOwnReadings(t *testing.T) {
proxmoxDisk := func(agent diskinventory.FieldStatus) Resource {
return Resource{
Type: ResourceTypePhysicalDisk, Name: "WDC", Status: StatusOnline,
PhysicalDisk: &PhysicalDiskMeta{
DevPath: "/dev/sdb", Serial: "WD-SILENT1", Temperature: 41,
Collection: &diskinventory.CollectionStatus{
Serial: diskinventory.Available("proxmox_disks"),
Temperature: agent,
Pool: diskinventory.Available("proxmox_zfs"),
},
},
}
}
agentDisk := func(temperature diskinventory.FieldStatus) Resource {
return Resource{
Type: ResourceTypePhysicalDisk, Name: "WDC", Status: StatusOnline,
PhysicalDisk: &PhysicalDiskMeta{
DevPath: "/dev/sdb", Serial: "WD-SILENT1", Temperature: 41,
Collection: &diskinventory.CollectionStatus{
Serial: diskinventory.Available("smartctl"),
Temperature: temperature,
},
},
}
}
identity := ResourceIdentity{MachineID: "WD-SILENT1", Hostnames: []string{"node1"}}
collected := diskinventory.Available("smartctl")
expired := diskinventory.Unavailable("smartctl", "host agent stopped reporting")
for _, tc := range []struct {
name string
proxmoxFirst bool
agent diskinventory.FieldStatus
want diskinventory.FieldStatus
}{
{"proxmox then expired agent", true, expired, expired},
{"expired agent then proxmox", false, expired, expired},
{"proxmox then reporting agent", true, collected, collected},
{"reporting agent then proxmox", false, collected, collected},
} {
t.Run(tc.name, func(t *testing.T) {
registry := NewRegistry(nil)
// The Proxmox row still holds the earlier "collected" copy.
ingestProxmox := func() { registry.ingest(SourceProxmox, "pve1-node1-sdb", proxmoxDisk(collected), identity) }
ingestAgent := func() { registry.ingest(SourceAgent, "agent-1-sdb", agentDisk(tc.agent), identity) }
if tc.proxmoxFirst {
ingestProxmox()
ingestAgent()
} else {
ingestAgent()
ingestProxmox()
}
disks := registry.ListByType(ResourceTypePhysicalDisk)
if len(disks) != 1 || disks[0].PhysicalDisk == nil || disks[0].PhysicalDisk.Collection == nil {
t.Fatalf("expected one merged disk, got %+v", disks)
}
collection := disks[0].PhysicalDisk.Collection
if collection.Temperature != tc.want {
t.Fatalf("temperature collection = %+v, want %+v", collection.Temperature, tc.want)
}
if disks[0].PhysicalDisk.Temperature != 41 {
t.Fatalf("last-known temperature was not retained: %d", disks[0].PhysicalDisk.Temperature)
}
if collection.Pool != diskinventory.Available("proxmox_zfs") {
t.Fatalf("Proxmox-owned evidence changed: %+v", collection.Pool)
}
})
}
}
pkg/diskinventory/status.go
+29
View File
@@ -89,6 +89,35 @@ func MergeStatus(existing, incoming *CollectionStatus) *CollectionStatus {
return &merged
}
// MergeReportedStatus merges a source's own report into existing state that
// may still hold an earlier copy of that report. It behaves like MergeStatus,
// except that a field the report no longer marks available supersedes an
// available state that existing carries from the same source: the source's
// later word on its own field wins, so a stale copy cannot keep claiming the
// field was collected.
func MergeReportedStatus(existing, reported *CollectionStatus) *CollectionStatus {
merged := MergeStatus(existing, reported)
if existing == nil || reported == nil {
return merged
}
merged.Serial = supersedeFieldStatus(merged.Serial, existing.Serial, reported.Serial)
merged.Temperature = supersedeFieldStatus(merged.Temperature, existing.Temperature, reported.Temperature)
merged.IO = supersedeFieldStatus(merged.IO, existing.IO, reported.IO)
merged.Controller = supersedeFieldStatus(merged.Controller, existing.Controller, reported.Controller)
merged.Pool = supersedeFieldStatus(merged.Pool, existing.Pool, reported.Pool)
return merged
}
func supersedeFieldStatus(merged, existing, reported FieldStatus) FieldStatus {
source := strings.TrimSpace(existing.Source)
if existing.State == FieldAvailable &&
reported.State != "" && reported.State != FieldAvailable &&
source != "" && strings.EqualFold(source, strings.TrimSpace(reported.Source)) {
return reported
}
return merged
}
func mergeFieldStatus(existing, incoming FieldStatus) FieldStatus {
if incoming.State == "" {
return existing
pkg/diskinventory/status_test.go
+41
View File
@@ -24,3 +24,44 @@ func TestMergeStatusPrefersAvailableEvidencePerField(t *testing.T) {
t.Fatalf("missing I/O state was not retained: %+v", got.IO)
}
}
func TestMergeReportedStatusLetsTheSourceWithdrawItsOwnAvailability(t *testing.T) {
existing := &CollectionStatus{
Serial: Available("smartctl"),
Temperature: Available("smartctl"),
IO: Available("kernel_diskstats"),
Pool: Available("proxmox_zfs"),
}
reported := &CollectionStatus{
Serial: Available("smartctl"),
Temperature: Unavailable("smartctl", "host agent stopped reporting"),
IO: Unavailable("kernel_diskstats", "host agent stopped reporting"),
Pool: Unavailable("zpool", "command failed"),
}
got := MergeReportedStatus(existing, reported)
if got.Temperature != reported.Temperature || got.IO != reported.IO {
t.Fatalf("the source's later word on its own fields was ignored: %+v", got)
}
if got.Serial != Available("smartctl") {
t.Fatalf("still-available field changed: %+v", got.Serial)
}
if got.Pool != Available("proxmox_zfs") {
t.Fatalf("another source's available evidence was withdrawn: %+v", got.Pool)
}
// A copy that is already weaker never overrides a current report, and the
// rule is plain MergeStatus when either side is absent.
if got := MergeReportedStatus(reported, existing); got.Temperature != Available("smartctl") {
t.Fatalf("current report lost to a weaker copy: %+v", got.Temperature)
}
if got := MergeReportedStatus(nil, reported); got.Temperature != reported.Temperature {
t.Fatalf("nil existing: %+v", got)
}
if got := MergeReportedStatus(existing, nil); got.Temperature != existing.Temperature {
t.Fatalf("nil report: %+v", got)
}
if existing.Temperature != Available("smartctl") {
t.Fatal("MergeReportedStatus mutated its input")
}
}