mirror of
https://github.com/rcourtman/Pulse.git
synced 2026-10-24 04:33:17 +00:00
Show a moved alert as moved in Patrol, history and assistant
A node alert handed to its Pulse agent closes with an AlertResolution (#2580), and notifications already say it moved. Every in-app reader still treated the close as a recovery: Patrol's detail wore a green Resolved badge over a timeline labelled "Recovery evidence", the alert history listed the row as resolved and its incident timeline said "Alert resolved: Node memory at 95%", and the assistant, Patrol's seed prompt and the pulse_alerts tool saw a plain resolve because GetRecentlyResolved dropped the resolution. Operational trust gains a non-recovery cause, ownership_transferred, which may only enter resolved and only with evidence. The resolve transition uses it whenever the alert carries a moved_to_agent resolution, so a handover during a snooze also stops counting as resolved while snoozed. models.Alert gains Resolution (reason, successor and the engine's summary), copied by cloneAlert and filled by GetRecentlyResolved, so the state snapshot, websocket, assistant context, Patrol seed and the resolved-alerts tool all report where the alert went. The resolved resource change carries the summary as its reason with alert_resolution metadata, and incident memory titles the close with it. In the UI, the history row reads "moved to agent" with the account in its title and on the phone card, and Patrol's detail shows a Moved badge with the handover sentence under the summary.
This commit is contained in:
@@ -1262,6 +1262,10 @@ update, profile rollout, command reachability, or fleet-control authority.
|
||||
lifecycle presentation evidence. It keeps mock active-alert and history
|
||||
payloads aligned with the canonical alert contract, but it cannot enroll,
|
||||
identify, link, command, remove, or otherwise grant authority to an agent.
|
||||
The optional `models.Alert.Resolution` field is the same kind of
|
||||
evidence: it says a resolved node alert now belongs to a linked agent's
|
||||
own alert and names that agent as display text, but the link itself stays
|
||||
the agent lifecycle's, and the field cannot create, change or authorize it.
|
||||
27. `internal/monitoring/monitor.go` shared with `monitoring`: monitor construction owns both monitoring runtime initialization and fail-closed agent lifecycle journal hydration before report admission.
|
||||
28. `internal/monitoring/monitor_agents.go` shared with `monitoring`: server-side Unified Agent report, removal, token binding, tombstone expiry, and re-enrollment semantics are jointly owned by agent lifecycle authority and monitoring ingest.
|
||||
29. `pkg/agents/host/report.go` shared with `monitoring`: the Unified Agent host report is both an agent lifecycle authored-state contract and a monitoring ingest contract for host maintenance posture.
|
||||
|
||||
@@ -4638,7 +4638,22 @@ summaries became the alarm name still carry "<entity> <moref> has VMware alarm
|
||||
<name> (<colour>)", so the title strips that wording to the same alarm name.
|
||||
Patrol's seed context lists active and recently resolved alerts by resource:
|
||||
each line names the alert's resource unless the message already does, because
|
||||
provider incidents such as vCenter alarms say only what was flagged. The projection also collapses open/resolved churn:
|
||||
provider incidents such as vCenter alarms say only what was flagged.
|
||||
A recently resolved alert that closed without recovering, such as a node metric
|
||||
handed to its Pulse agent, must not read as a recovery to Patrol, the
|
||||
assistant or the alert tool: Patrol's seed writes `closed <ago>. <summary>` in place of `resolved <ago>`, the
|
||||
assistant's alert context writes `closed <ago> ago) <summary>`, the
|
||||
`pulse_alerts` `resolved` action returns the summary as `resolution` with
|
||||
`successor_resource_id`, and incident memory titles the close with the summary
|
||||
(`alertResolvedEventSummary`, or the `alert_resolution` resource-change
|
||||
metadata when it projects from the canonical timeline).
|
||||
`TestSeedHealthAndAlerts_HandoverIsNotARecovery`,
|
||||
`TestService_buildAlertContext_HandoverIsNotARecovery`,
|
||||
`TestExecuteListResolvedAlertsReportsHandover` and
|
||||
`TestIncidentStore_HandoverCloseIsNotARecovery` pin it. Performance reports
|
||||
(`pkg/reporting`, the report narrator) do not carry the resolution yet: a
|
||||
report generated within the five-minute recently-resolved window after a
|
||||
handover still lists that alert as resolved. The projection also collapses open/resolved churn:
|
||||
when a record's timeline holds at least four open-to-resolved or
|
||||
resolved-to-open transitions inside the last 24 hours, the item carries a
|
||||
`flapping` summary (transition count, window, first and latest transition) and
|
||||
|
||||
@@ -793,12 +793,26 @@ its message) and the Alert JSON then carry it. Resolved consumers still receive
|
||||
the close, because integrations close their incident by alert ID. The
|
||||
synthesized closing evidence carries the resolution reason and summary instead
|
||||
of `legacy_alert_recovery_projection`, and the resolve transition carries the
|
||||
summary as its `reason`. The cause stays `recovery_evidence`, the only cause
|
||||
operational trust lets enter `resolved`. A nil resolution leaves the
|
||||
disabled-threshold path and every recovery unchanged.
|
||||
summary as its `reason`. Its cause is `ownership_transferred`, an
|
||||
`operationaltrust.TransitionCause` that may only enter `resolved` and only with
|
||||
evidence (`resolvedTransitionCause` in `operational_contract.go`), so Patrol's
|
||||
timeline never labels a handover "Recovery evidence" and alert-quality
|
||||
telemetry does not count a handover during a snooze as resolved while snoozed.
|
||||
A nil resolution leaves the disabled-threshold path and every recovery on
|
||||
`recovery_evidence`.
|
||||
`TestReleasedMetricAlertCarriesResolutionInsteadOfRecovery` and
|
||||
`TestResolvedHistoryRowKeepsResolutionWithoutEventLog` in
|
||||
`internal/alerts/operational_contract_test.go` pin this.
|
||||
The in-app readers carry it too. `GetRecentlyResolved` projects it into
|
||||
`models.Alert.Resolution` (reason, successor and the `Summary()` text) for the
|
||||
state snapshot, the websocket and the assistant and Patrol prompts, pinned by
|
||||
`TestRecentlyResolvedCarriesHandoverResolution` in `history_test.go`. The
|
||||
alerts history lists such a row as `moved to agent` instead of `resolved`, even
|
||||
when it was acknowledged, with the account in the badge title and on the phone
|
||||
card (`buildAlertHistoryItems`, `getAlertResolutionDetail`). The resolved
|
||||
lifecycle change written to the resource timeline carries the summary as its
|
||||
reason and the code as `alert_resolution` metadata, so the row's incident
|
||||
timeline says the alert moved instead of "Alert resolved: <breach message>".
|
||||
|
||||
## Extension Points
|
||||
|
||||
@@ -2728,7 +2742,8 @@ legacy alert timestamp.
|
||||
Suppression is bounded and reasoned, leaves the default active queue, and
|
||||
remains inspectable. Expiry or explicit unsuppression returns the record to its
|
||||
detector-owned state; it never resolves it. Only fresh sufficient recovery
|
||||
evidence may enter resolving, and only detector recovery may resolve.
|
||||
evidence may enter resolving, and only detector recovery, or a handover
|
||||
recorded as `ownership_transferred` with its closing evidence, may resolve.
|
||||
|
||||
Per-alert snooze is the customer-facing bounded suppression contract, not a
|
||||
page-local timer. `SnoozeAlert` writes the same canonical operational record,
|
||||
|
||||
@@ -661,6 +661,13 @@ transitions. Older payloads without these optional fields remain readable;
|
||||
frontend consumers use `frontend-modern/src/types/operationalTrust.ts` as the
|
||||
single TypeScript projection rather than recreating lifecycle or evidence
|
||||
enums locally.
|
||||
`TransitionCause` includes `ownership_transferred` for a close whose condition
|
||||
another record now owns, such as a node metric handed to its Pulse agent; it
|
||||
may only enter `resolved`. Alert payloads (history, state snapshot and
|
||||
websocket) may also carry an optional `resolution` (`reason`,
|
||||
`successorResourceId`, `successorName`, and on state payloads the alert
|
||||
engine's `summary`), typed once as `AlertResolution` in
|
||||
`frontend-modern/src/types/api.ts`. Its absence means an ordinary recovery.
|
||||
|
||||
## Shared Boundaries
|
||||
|
||||
|
||||
@@ -7088,6 +7088,13 @@ should extend the model, and section rendering should extend those owners
|
||||
rather than rebuilding any of those concerns in the tab shell.
|
||||
The resource-resolution handoff into the resource-incident panel now belongs to
|
||||
the history state rather than the tab shell.
|
||||
The history model also owns the close outcome: an alert that closed with a
|
||||
typed `resolution` is status `moved` (badge `moved to agent`, its account in
|
||||
`HistoryItem.closeDetail`) rather than `resolved` or `acknowledged`. The table
|
||||
row carries the account in the badge title and the phone card renders it under
|
||||
the message, while `getAlertHistoryStatusPresentation` and
|
||||
`getAlertResolutionDetail` in `utils/alertIncidentPresentation.ts` own the
|
||||
wording.
|
||||
`frontend-modern/src/features/alerts/useAlertHistoryState.ts` re-exposes the
|
||||
`getResource` resolver it is already given, and
|
||||
`frontend-modern/src/features/alerts/AlertResourceIncidentsPanel.tsx` reads it
|
||||
|
||||
@@ -3587,6 +3587,15 @@ truth. Monitoring APIs that still serve `StateSnapshot` must project
|
||||
instead of trusting the cached snapshot fields, so externally served alert
|
||||
counts and recently resolved incidents do not lag behind acknowledgement,
|
||||
resolve, or clear operations between explicit sync points.
|
||||
A recently resolved alert that closed without recovering keeps that on the
|
||||
projection: `models.Alert.Resolution` (reason, successor resource ID and name,
|
||||
and the alert engine's summary) is set only for such a close, `cloneAlert`
|
||||
copies it into every snapshot, and `recordAlertTimelineChange` passes it into
|
||||
the `alert_resolved` resource change so the incident timeline says the alert
|
||||
moved. `TestAlertResolutionWireContract` in
|
||||
`internal/models/metrics_types_test.go` and
|
||||
`TestMonitor_HandleAlertLifecycle_HandoverCloseIsNotARecovery` in
|
||||
`internal/monitoring/monitor_alert_handling_test.go` pin it.
|
||||
The container entrypoint in `docker-entrypoint.sh` now also lives under this
|
||||
boundary. Hosted or managed tenant bootstrap changes must preserve safe startup
|
||||
when immutable read-only mounts are layered into `/etc/pulse`; the entrypoint
|
||||
|
||||
@@ -1237,6 +1237,12 @@ payload's `alerts` array. Built-in template titles still say Resolved, which is
|
||||
accurate for the closed alert, and a custom template that hard-codes recovery
|
||||
text keeps it. Issue #1682 is the precedent: a recipient called "is now
|
||||
healthy" a false factual claim when an alert closed without recovering.
|
||||
The resolve transition such a close links to has cause `ownership_transferred`
|
||||
in `internal/operationaltrust/contracts.go`, not `recovery_evidence`. A
|
||||
notification link records the transition ID and lifecycle state, never the
|
||||
cause, so delivery, receipts and recovery admission treat it exactly like a
|
||||
recovery's resolve transition. `TestOwnershipTransferOnlyClosesWithEvidence`
|
||||
pins that the cause may only enter `resolved` and only with evidence.
|
||||
|
||||
Verification: `resolved_handover_test.go` drives the firing, its receipt and
|
||||
the queued close through the normal entry points for ntfy, and renders Discord,
|
||||
|
||||
@@ -1380,6 +1380,14 @@ to traverse the list again. At widths where the detail replaces the queue's
|
||||
reading position, its visible close affordance is an explicit `Back to list`
|
||||
control; wide layouts retain the compact close icon. Both controls clear the
|
||||
deep link and restore focus to the originating queue row.
|
||||
A resolved item whose latest close is an `ownership_transferred` transition did
|
||||
not recover: its condition moved to another record, such as the linked Pulse
|
||||
agent's own alert. The detail's state badge reads `Moved` in the info tone
|
||||
instead of the green `Resolved`, and the transition's reason (where the
|
||||
condition went, and that it is not a recovery) sits directly under the summary
|
||||
rather than only inside Evidence and history (`getAttentionHandoverClose` in
|
||||
`patrolHomePresentation.ts`). The timeline names that cause `Ownership
|
||||
transferred`, never `Recovery evidence`.
|
||||
|
||||
The active Patrol queue now uses compact severity-accented rows for
|
||||
prioritisation and a single focused review panel for the selected issue. The
|
||||
|
||||
@@ -4171,6 +4171,14 @@ such as `alert_fired`, `alert_acknowledged`, `alert_unacknowledged`,
|
||||
AI-local annotations. Snooze and resume projections must preserve the actor
|
||||
and exact suppression expiry when present; they pause delivery and escalation
|
||||
without acknowledging, resolving, or replacing the underlying incident.
|
||||
An `alert_resolved` change for a close that was not a recovery carries the
|
||||
alert engine's summary as its reason and the reason code as `alert_resolution`
|
||||
metadata (`AlertTimelineChange.ResolutionReason` and `ResolutionSummary`), so
|
||||
incident projection titles the close with where the alert went instead of
|
||||
"Alert resolved: <breach message>". The resource drawer still labels the change
|
||||
by kind (`Alert resolved`) and prefixes its headline with that label; only the
|
||||
reason after it says the alert moved. Ordinary recoveries omit the key, and
|
||||
every other lifecycle kind ignores the fields.
|
||||
Alert-scoped
|
||||
incident memory may still project those events for one investigation thread,
|
||||
but the durable source of truth for resource-affecting alert lifecycle and
|
||||
|
||||
@@ -0,0 +1,54 @@
|
||||
{
|
||||
"version": 1,
|
||||
"base_sha": "982a6f6148cf9d39883e4443191b4d40b59cfd7c",
|
||||
"verified_at": "2026-10-06T15:03:00Z",
|
||||
"result": "passed",
|
||||
"changed_paths": [
|
||||
"frontend-modern/src/features/alerts/AlertHistoryMobileList.tsx",
|
||||
"frontend-modern/src/features/alerts/AlertHistoryTableAlertRow.tsx",
|
||||
"frontend-modern/src/features/alerts/alertHistoryModel.ts",
|
||||
"frontend-modern/src/features/patrol/PatrolAttentionWorkbench.tsx",
|
||||
"frontend-modern/src/features/patrol/patrolHomePresentation.ts",
|
||||
"frontend-modern/src/types/api.ts",
|
||||
"frontend-modern/src/types/operationalTrust.ts",
|
||||
"frontend-modern/src/utils/alertIncidentPresentation.ts"
|
||||
],
|
||||
"content_sha256": {
|
||||
"frontend-modern/src/features/alerts/AlertHistoryMobileList.tsx": "94bb73163dd86c1f045a0b5559b42486198ab824ef2abccfc3c8c1dcc0ff2784",
|
||||
"frontend-modern/src/features/alerts/AlertHistoryTableAlertRow.tsx": "aacc7232596a47d64c7e6ec3b29dfaf51be6e1691d9be906b30bfce49f4daf6f",
|
||||
"frontend-modern/src/features/alerts/alertHistoryModel.ts": "30548565d29ca16fe1c1192687c881defc064ba1b267fdcd40fc28922a51fed4",
|
||||
"frontend-modern/src/features/patrol/PatrolAttentionWorkbench.tsx": "b3c87b3ead9bde22a091b2c080b9eaca18face9ba153085bb1a1dbb9b6ae9f54",
|
||||
"frontend-modern/src/features/patrol/patrolHomePresentation.ts": "04338cdce5b54bc041267b325a6985a9a8f84ef9e5a8dba8040c7d9fc5ab1811",
|
||||
"frontend-modern/src/types/api.ts": "17a948b6ba78780ef2b95c1f397d2d9a04eebd2726c568a8316e25e7c8f9e6b3",
|
||||
"frontend-modern/src/types/operationalTrust.ts": "9a50a869b1f6ef2885f30a354fb70b36b64fbb1a3f00ba4476d592389f9ba954",
|
||||
"frontend-modern/src/utils/alertIncidentPresentation.ts": "f77c42b48f22bddcccb43a3da43b3771bb03d25c6d23b7efa200a34dae1da0e9"
|
||||
},
|
||||
"routes": [
|
||||
"/alerts/history",
|
||||
"/patrol?attention=pve1::metric-threshold:memory"
|
||||
],
|
||||
"viewports": [
|
||||
{
|
||||
"width": 1280,
|
||||
"height": 900
|
||||
},
|
||||
{
|
||||
"width": 375,
|
||||
"height": 812
|
||||
}
|
||||
],
|
||||
"states": [
|
||||
"Private non-mock stack (backend :7691, Vite :5291) on a data dir whose alert history was written by the real alerts manager: a pve1 node memory alert closed through releaseCanonicalMetricAlert with a moved_to_agent resolution (successor pve1 (Host Agent)) and a pve1 CPU alert closed as an ordinary release; the handover record's resolve transition carries cause ownership_transferred",
|
||||
"Alerts history at 1280: the CPU row's status badge reads resolved, the memory row's reads moved to agent in blue, and its title is 'Moved to pve1 (Host Agent). This is not a recovery: check the agent for the current reading.'",
|
||||
"Memory row Timeline expanded: the incident timeline lists 'Node memory at 95.0%' then 'Alert moved to pve1 (Host Agent). This is not a recovery: check the agent for the current reading.' where it said 'Alert resolved' before; the incident header still says resolved (incident status vocabulary, unchanged)",
|
||||
"Alerts history at 375: the memory card shows the moved to agent badge and the account as a muted line under the breach message; the CPU card is unchanged",
|
||||
"Patrol deep link to the moved record at 1280 and 375: the detail badge reads Moved (info tone) instead of green Resolved, the handover sentence sits directly under 'Node memory at 95.0%', and Evidence and history shows 'Open to Resolved, Ownership transferred' with the same reason; no Recovery evidence label anywhere; no horizontal scroll at 375 and no console errors"
|
||||
],
|
||||
"interactions": [
|
||||
"Loaded /alerts/history at 1280x900 and read both rows' status badges and titles",
|
||||
"Clicked the memory row's Timeline button and read the expanded incident timeline",
|
||||
"Reloaded /alerts/history at 375x812 and read both history cards",
|
||||
"Opened /patrol?attention=pve1::metric-threshold:memory at 1280x900 and 375x812, opened Evidence and history, and read the badge, summary, handover line and timeline",
|
||||
"Checked document width against the viewport at 375 and read the console for errors"
|
||||
]
|
||||
}
|
||||
@@ -229,6 +229,9 @@ export function AlertHistoryMobileList(props: AlertHistoryMobileListProps) {
|
||||
</div>
|
||||
|
||||
<p class="mt-2 text-xs leading-relaxed text-base-content">{alert.description}</p>
|
||||
<Show when={alert.closeDetail}>
|
||||
{(detail) => <p class="mt-1 text-xs leading-relaxed text-muted">{detail()}</p>}
|
||||
</Show>
|
||||
|
||||
<dl class="mt-3 grid grid-cols-3 gap-2 text-[10px] text-muted">
|
||||
<div>
|
||||
|
||||
@@ -74,7 +74,7 @@ export function AlertHistoryTableAlertRow(props: AlertHistoryTableAlertRowProps)
|
||||
</TableCell>
|
||||
|
||||
<TableCell class={getPlatformTableCellClassForKind('badge')}>
|
||||
<span class={historyStatusPresentation().className}>
|
||||
<span class={historyStatusPresentation().className} title={props.alert.closeDetail}>
|
||||
{historyStatusPresentation().label}
|
||||
</span>
|
||||
</TableCell>
|
||||
|
||||
@@ -1,5 +1,9 @@
|
||||
import type { Alert } from '@/types/api';
|
||||
import type { Resource } from '@/types/resource';
|
||||
import {
|
||||
ALERT_HISTORY_MOVED_STATUS,
|
||||
getAlertResolutionDetail,
|
||||
} from '@/utils/alertIncidentPresentation';
|
||||
import { isPulseSystemAlert } from '@/utils/alertScope';
|
||||
|
||||
import { alertTypeDisplayLabel, unifiedTypeToAlertDisplayType } from './helpers';
|
||||
@@ -25,6 +29,8 @@ export interface HistoryItem {
|
||||
title: string;
|
||||
rawAlertType?: string;
|
||||
description?: string;
|
||||
// Set when the alert closed without recovering: what happened instead.
|
||||
closeDetail?: string;
|
||||
acknowledged?: boolean;
|
||||
systemAlert?: boolean;
|
||||
}
|
||||
@@ -206,11 +212,17 @@ export function buildAlertHistoryItems({
|
||||
alertHistory.forEach((alert) => {
|
||||
if (activeAlertIds.has(alert.id)) return;
|
||||
const systemScoped = isPulseSystemAlert(alert);
|
||||
// A close that was not a recovery must not read as "resolved".
|
||||
const closeDetail = getAlertResolutionDetail(alert.resolution);
|
||||
|
||||
items.push({
|
||||
id: alert.id,
|
||||
source: 'alert',
|
||||
status: alert.acknowledged ? 'acknowledged' : 'resolved',
|
||||
status: closeDetail
|
||||
? ALERT_HISTORY_MOVED_STATUS
|
||||
: alert.acknowledged
|
||||
? 'acknowledged'
|
||||
: 'resolved',
|
||||
startTime: alert.startTime,
|
||||
endTime: alert.lastSeen,
|
||||
duration: formatAlertHistoryDuration(alert.startTime, alert.lastSeen, now),
|
||||
@@ -231,6 +243,7 @@ export function buildAlertHistoryItems({
|
||||
title: alertTypeDisplayLabel(alert.type),
|
||||
rawAlertType: alert.type,
|
||||
description: alert.message,
|
||||
...(closeDetail ? { closeDetail } : {}),
|
||||
acknowledged: alert.acknowledged,
|
||||
});
|
||||
});
|
||||
|
||||
@@ -68,6 +68,7 @@ import { copyToClipboard } from '@/utils/clipboard';
|
||||
import type { PatrolAutonomyLevel } from '@/api/patrol';
|
||||
import {
|
||||
getAttentionFlappingPresentation,
|
||||
getAttentionHandoverClose,
|
||||
getLinkedPatrolFindings,
|
||||
getPatrolRememberedDecisionPresentation,
|
||||
partitionPatrolAttention,
|
||||
@@ -971,7 +972,10 @@ function AttentionDetail(props: {
|
||||
<div class="space-y-5 px-4 py-4 sm:px-5">
|
||||
<section>
|
||||
<div class="flex flex-wrap items-center gap-x-2 gap-y-1 text-xs text-muted">
|
||||
<StateBadge item={loaded().item} />
|
||||
<StateBadge
|
||||
item={loaded().item}
|
||||
movedClose={Boolean(getAttentionHandoverClose(loaded()))}
|
||||
/>
|
||||
<span aria-hidden="true">·</span>
|
||||
<EvidenceLabel item={loaded().item} detail />
|
||||
<span aria-hidden="true">·</span>
|
||||
@@ -984,6 +988,9 @@ function AttentionDetail(props: {
|
||||
<p class="mt-3 text-sm leading-6 text-base-content">
|
||||
{loaded().item.plainLanguageSummary}
|
||||
</p>
|
||||
<Show when={getAttentionHandoverClose(loaded())}>
|
||||
{(handover) => <p class="mt-1 text-xs leading-5 text-muted">{handover()}</p>}
|
||||
</Show>
|
||||
<Show when={getAttentionFlappingPresentation(loaded().item.flapping)}>
|
||||
{(flapping) => (
|
||||
<p class="mt-2 flex items-start gap-2 rounded-md border border-amber-200 bg-amber-50/70 px-3 py-2 text-xs leading-5 text-amber-900 dark:border-amber-900/25 dark:bg-amber-950/30 dark:text-amber-200">
|
||||
@@ -1793,10 +1800,17 @@ function severityTone(item: AttentionItem): MetadataBadgeTone {
|
||||
}
|
||||
}
|
||||
|
||||
function StateBadge(props: { item: AttentionItem }) {
|
||||
// movedClose marks a resolved item whose condition moved to another owner: it
|
||||
// did not recover, so it must not wear the green "Resolved" badge.
|
||||
function StateBadge(props: { item: AttentionItem; movedClose?: boolean }) {
|
||||
return (
|
||||
<MetadataBadge tone={stateTone(props.item)} size="xs" shape="rounded" appearance="outline">
|
||||
{formatLabel(props.item.state)}
|
||||
<MetadataBadge
|
||||
tone={props.movedClose ? 'info' : stateTone(props.item)}
|
||||
size="xs"
|
||||
shape="rounded"
|
||||
appearance="outline"
|
||||
>
|
||||
{props.movedClose ? 'Moved' : formatLabel(props.item.state)}
|
||||
</MetadataBadge>
|
||||
);
|
||||
}
|
||||
|
||||
@@ -988,6 +988,56 @@ describe('PatrolAttentionWorkbench', () => {
|
||||
expect(screen.getByText('Show all 11 transitions')).toBeInTheDocument();
|
||||
});
|
||||
|
||||
it('shows an alert handed to its Pulse agent as moved, not recovered', async () => {
|
||||
const reason =
|
||||
'Alert moved to pve1 (Host Agent). This is not a recovery: check the agent for the current reading.';
|
||||
const moved = item({
|
||||
id: 'record-moved',
|
||||
operationalRecordId: 'record-moved',
|
||||
subjectResourceId: 'pve1',
|
||||
subjectResourceName: 'pve1',
|
||||
subjectResourceType: 'node',
|
||||
kind: 'memory',
|
||||
title: 'Memory on pve1',
|
||||
plainLanguageSummary: 'Memory usage at 95%',
|
||||
severity: 'warning',
|
||||
state: 'resolved',
|
||||
protectionPosture: undefined,
|
||||
});
|
||||
const movedDetail = detail(moved);
|
||||
movedDetail.timeline = [
|
||||
...movedDetail.timeline,
|
||||
{
|
||||
id: 'transition-moved',
|
||||
operationalRecordId: 'record-moved',
|
||||
from: 'open',
|
||||
to: 'resolved',
|
||||
at: '2026-07-19T07:45:00Z',
|
||||
cause: 'ownership_transferred',
|
||||
causeKey: 'memory:pve1',
|
||||
evidenceIds: ['evidence-handover'],
|
||||
reason,
|
||||
},
|
||||
];
|
||||
window.history.replaceState({}, '', '/patrol?attention=record-moved');
|
||||
apiMocks.getList.mockResolvedValue(listResponse([], summary()));
|
||||
apiMocks.getDetail.mockResolvedValue(movedDetail);
|
||||
renderWorkbench();
|
||||
|
||||
const panel = await screen.findByRole('complementary', { name: 'pve1 · Memory' });
|
||||
expect(await within(panel).findByText('Moved')).toBeInTheDocument();
|
||||
expect(within(panel).queryByText('Resolved')).not.toBeInTheDocument();
|
||||
// The account sits under the summary, outside the collapsed history.
|
||||
const summaryLine = within(panel).getByText('Memory usage at 95%');
|
||||
expect(summaryLine.nextElementSibling).toHaveTextContent(reason);
|
||||
expect(summaryLine.nextElementSibling?.closest('details')).toBeNull();
|
||||
|
||||
fireEvent.click(within(panel).getByText('Evidence and history'));
|
||||
expect(within(panel).getByText('Open to Resolved')).toBeInTheDocument();
|
||||
expect(within(panel).getByText(/Ownership transferred/)).toBeInTheDocument();
|
||||
expect(within(panel).queryByText(/Recovery evidence/)).not.toBeInTheDocument();
|
||||
});
|
||||
|
||||
it('opens the canonical governed action review from an eligible attention item', async () => {
|
||||
const actionOffer = {
|
||||
targetResourceId: 'docker:host-1/container-1',
|
||||
|
||||
@@ -1,7 +1,8 @@
|
||||
import { describe, expect, it } from 'vitest';
|
||||
import type { PatrolObjective } from '@/api/patrol';
|
||||
import type { AttentionItem } from '@/api/patrolAttention';
|
||||
import type { AttentionItem, AttentionItemDetail } from '@/api/patrolAttention';
|
||||
import {
|
||||
getAttentionHandoverClose,
|
||||
getPatrolAttentionDecisionReason,
|
||||
getPatrolObjectiveProtectionSummary,
|
||||
partitionPatrolAttention,
|
||||
@@ -145,3 +146,62 @@ describe('patrol home presentation', () => {
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
describe('getAttentionHandoverClose', () => {
|
||||
const transition = (
|
||||
overrides: Partial<AttentionItemDetail['timeline'][number]>,
|
||||
): AttentionItemDetail['timeline'][number] => ({
|
||||
id: 'transition-1',
|
||||
operationalRecordId: 'record-1',
|
||||
from: 'open',
|
||||
to: 'resolved',
|
||||
at: '2026-08-14T08:00:00Z',
|
||||
cause: 'recovery_evidence',
|
||||
causeKey: 'memory:pve1',
|
||||
evidenceIds: ['evidence-1'],
|
||||
...overrides,
|
||||
});
|
||||
const reason =
|
||||
'Alert moved to pve1 (Host Agent). This is not a recovery: check the agent for the current reading.';
|
||||
|
||||
it('returns the handover account only for a record closed by an ownership transfer', () => {
|
||||
const resolved = attentionItem({ state: 'resolved' });
|
||||
expect(
|
||||
getAttentionHandoverClose({
|
||||
item: resolved,
|
||||
timeline: [transition({ cause: 'ownership_transferred', reason })],
|
||||
}),
|
||||
).toBe(reason);
|
||||
expect(
|
||||
getAttentionHandoverClose({
|
||||
item: resolved,
|
||||
timeline: [transition({ cause: 'ownership_transferred' })],
|
||||
}),
|
||||
).toBe('This alert moved to another owner when it closed. It did not recover.');
|
||||
expect(getAttentionHandoverClose({ item: resolved, timeline: [transition({})] })).toBeNull();
|
||||
});
|
||||
|
||||
it('follows the latest close, so a later recovery or reopen wins', () => {
|
||||
const handover = transition({ cause: 'ownership_transferred', reason });
|
||||
const reopened = transition({
|
||||
id: 'transition-2',
|
||||
from: 'resolved',
|
||||
to: 'open',
|
||||
at: '2026-08-14T09:00:00Z',
|
||||
cause: 'detector_decision',
|
||||
});
|
||||
const recovered = transition({ id: 'transition-3', at: '2026-08-14T10:00:00Z' });
|
||||
expect(
|
||||
getAttentionHandoverClose({
|
||||
item: attentionItem({ state: 'resolved' }),
|
||||
timeline: [handover, reopened, recovered],
|
||||
}),
|
||||
).toBeNull();
|
||||
expect(
|
||||
getAttentionHandoverClose({
|
||||
item: attentionItem({ state: 'open' }),
|
||||
timeline: [handover, reopened],
|
||||
}),
|
||||
).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
import type { PatrolAutonomyLevel, PatrolObjective } from '@/api/patrol';
|
||||
import type { AttentionFlapping, AttentionItem } from '@/api/patrolAttention';
|
||||
import type { AttentionFlapping, AttentionItem, AttentionItemDetail } from '@/api/patrolAttention';
|
||||
import type { UnifiedFinding } from '@/stores/aiIntelligence';
|
||||
|
||||
export interface PatrolAutonomyExperience {
|
||||
@@ -207,6 +207,26 @@ export function getAttentionFlappingPresentation(
|
||||
return getFlappingPresentation(flapping.transitionCount, flapping.windowHours);
|
||||
}
|
||||
|
||||
/**
|
||||
* A resolved item whose close handed the condition to another record, such as
|
||||
* a node metric its Pulse agent took over, did not recover. Returns the
|
||||
* account to show instead of a healthy "Resolved", or null for a recovery.
|
||||
*/
|
||||
export function getAttentionHandoverClose(
|
||||
detail: Pick<AttentionItemDetail, 'item' | 'timeline'> | null | undefined,
|
||||
): string | null {
|
||||
if (!detail || detail.item.state !== 'resolved') return null;
|
||||
let close: AttentionItemDetail['timeline'][number] | undefined;
|
||||
for (const transition of detail.timeline ?? []) {
|
||||
if (transition.to !== 'resolved') continue;
|
||||
if (!close || Date.parse(transition.at) >= Date.parse(close.at)) close = transition;
|
||||
}
|
||||
if (close?.cause !== 'ownership_transferred') return null;
|
||||
return (
|
||||
close.reason?.trim() || 'This alert moved to another owner when it closed. It did not recover.'
|
||||
);
|
||||
}
|
||||
|
||||
const normalizeResourceKey = (value: string | undefined): string =>
|
||||
(value ?? '').trim().toLowerCase();
|
||||
|
||||
|
||||
@@ -100,6 +100,7 @@ import {
|
||||
buildProjectedOverrides,
|
||||
normalizeRawOverridesConfig,
|
||||
} from '@/features/alerts/alertOverridesModel';
|
||||
import { buildAlertHistoryItems } from '@/features/alerts/alertHistoryModel';
|
||||
import {
|
||||
getAlertIncidentAcknowledgedBadgeClass,
|
||||
getAlertIncidentEventFilterActionButtonClass,
|
||||
@@ -1320,3 +1321,59 @@ describe('Unified selector parity', () => {
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
describe('alert history close outcome', () => {
|
||||
it('lists a node alert handed to its Pulse agent as moved, not resolved', () => {
|
||||
const base = {
|
||||
type: 'memory',
|
||||
level: 'warning' as const,
|
||||
resourceName: 'pve1',
|
||||
node: 'pve1',
|
||||
instance: 'pve1',
|
||||
value: 95,
|
||||
threshold: 85,
|
||||
startTime: '2026-10-06T09:00:00Z',
|
||||
lastSeen: '2026-10-06T11:00:00Z',
|
||||
acknowledged: true,
|
||||
};
|
||||
const items = buildAlertHistoryItems({
|
||||
activeAlerts: {},
|
||||
alertHistory: [
|
||||
{
|
||||
...base,
|
||||
id: 'pve1-memory',
|
||||
resourceId: 'pve1',
|
||||
message: 'Memory usage at 95%',
|
||||
resolution: {
|
||||
reason: 'moved_to_agent',
|
||||
successorResourceId: 'agent-pve1',
|
||||
successorName: 'pve1 (Host Agent)',
|
||||
},
|
||||
},
|
||||
{
|
||||
...base,
|
||||
id: 'pve1-cpu',
|
||||
resourceId: 'pve1',
|
||||
message: 'CPU usage at 90%',
|
||||
acknowledged: false,
|
||||
},
|
||||
],
|
||||
getResource: () => undefined,
|
||||
allResources: [],
|
||||
now: Date.parse('2026-10-06T12:00:00Z'),
|
||||
});
|
||||
|
||||
const moved = items.find((item) => item.id === 'pve1-memory');
|
||||
// Acknowledgement does not hide the outcome: the condition went to the agent.
|
||||
expect(moved?.status).toBe('moved');
|
||||
expect(moved?.description).toBe('Memory usage at 95%');
|
||||
expect(moved?.closeDetail).toBe(
|
||||
'Moved to pve1 (Host Agent). This is not a recovery: check the agent for the current reading.',
|
||||
);
|
||||
const recovered = items.find((item) => item.id === 'pve1-cpu');
|
||||
expect(recovered?.status).toBe('resolved');
|
||||
expect(recovered).not.toHaveProperty('closeDetail');
|
||||
|
||||
expect(alertHistoryTableAlertRowSource).toContain('title={props.alert.closeDetail}');
|
||||
});
|
||||
});
|
||||
|
||||
@@ -1380,6 +1380,18 @@ export interface Alert {
|
||||
latestTransition?: LifecycleTransition;
|
||||
transitions?: LifecycleTransition[];
|
||||
evidence?: EvidenceEnvelope[];
|
||||
// Set only on a closed alert whose condition did not clear, such as a node
|
||||
// metric handed to its Pulse agent. Absent for an ordinary recovery.
|
||||
resolution?: AlertResolution;
|
||||
}
|
||||
|
||||
// Why an alert closed without recovering (alerts.AlertResolution).
|
||||
export interface AlertResolution {
|
||||
reason: 'moved_to_agent' | (string & {});
|
||||
successorResourceId?: string;
|
||||
successorName?: string;
|
||||
// The alert engine's one-line account; carried on state snapshots only.
|
||||
summary?: string;
|
||||
}
|
||||
|
||||
export interface ResolvedAlert extends Alert {
|
||||
|
||||
@@ -97,6 +97,7 @@ export type TransitionCause =
|
||||
| 'suppression'
|
||||
| 'suppression_expired'
|
||||
| 'recovery_evidence'
|
||||
| 'ownership_transferred'
|
||||
| 'collection_stale'
|
||||
| 'collection_unknown';
|
||||
|
||||
|
||||
@@ -10,8 +10,10 @@ import {
|
||||
ALERT_RESOURCE_INCIDENT_REFRESH_LABEL,
|
||||
ALERT_RESOURCE_INCIDENT_SAVE_NOTE_LABEL,
|
||||
ALERT_RESOURCE_INCIDENT_SAVING_NOTE_LABEL,
|
||||
ALERT_HISTORY_MOVED_STATUS,
|
||||
getAlertHistoryStatusPresentation,
|
||||
getAlertIncidentLevelBadgeClass,
|
||||
getAlertResolutionDetail,
|
||||
getAlertIncidentTimelineDialogTitle,
|
||||
getAlertHistoryInvestigationDialogTitle,
|
||||
getAlertResourceIncidentAcknowledgedByLabel,
|
||||
@@ -103,6 +105,29 @@ describe('alertIncidentPresentation', () => {
|
||||
className: 'text-xs px-2 py-0.5 rounded-sm bg-surface-hover text-base-content',
|
||||
rowClassName: '',
|
||||
});
|
||||
|
||||
expect(getAlertHistoryStatusPresentation(ALERT_HISTORY_MOVED_STATUS)).toEqual({
|
||||
label: 'moved to agent',
|
||||
className:
|
||||
'text-xs px-2 py-0.5 rounded-sm bg-blue-100 dark:bg-blue-900/25 text-blue-700 dark:text-blue-300',
|
||||
rowClassName: '',
|
||||
});
|
||||
});
|
||||
|
||||
it('says a handed-over alert did not recover, and leaves recoveries alone', () => {
|
||||
expect(getAlertResolutionDetail(undefined)).toBeUndefined();
|
||||
expect(
|
||||
getAlertResolutionDetail({ reason: 'moved_to_agent', successorName: 'pve1 (Host Agent)' }),
|
||||
).toBe(
|
||||
'Moved to pve1 (Host Agent). This is not a recovery: check the agent for the current reading.',
|
||||
);
|
||||
expect(getAlertResolutionDetail({ reason: 'moved_to_agent' })).toBe(
|
||||
'Moved to its Pulse agent. This is not a recovery: check the agent for the current reading.',
|
||||
);
|
||||
// A future reason the UI has no copy for still shows the engine's account.
|
||||
expect(
|
||||
getAlertResolutionDetail({ reason: 'replaced', summary: 'Alert replaced by a new check.' }),
|
||||
).toBe('Alert replaced by a new check.');
|
||||
});
|
||||
|
||||
it('returns canonical resource incident loading and empty-state copy', () => {
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import type { AlertResolution } from '@/types/api';
|
||||
import { getAlertSeverityBadgeClass } from '@/utils/alertSeverityPresentation';
|
||||
|
||||
export type AlertIncidentLevel = 'warning' | 'critical' | string | null | undefined;
|
||||
@@ -76,6 +77,24 @@ export function getAlertIncidentLevelBadgeClass(level: AlertIncidentLevel): stri
|
||||
return getAlertSeverityBadgeClass(String(level ?? 'warning'));
|
||||
}
|
||||
|
||||
// History status for an alert that closed because its Pulse agent took the
|
||||
// metric over. The condition did not clear; the agent's own alert owns it.
|
||||
export const ALERT_HISTORY_MOVED_STATUS = 'moved';
|
||||
|
||||
/**
|
||||
* The account of a close that was not a recovery, for history rows and their
|
||||
* tooltips. Returns undefined for an ordinary recovery so callers keep their
|
||||
* "resolved" wording.
|
||||
*/
|
||||
export function getAlertResolutionDetail(resolution?: AlertResolution | null): string | undefined {
|
||||
if (!resolution) return undefined;
|
||||
if (resolution.reason === 'moved_to_agent') {
|
||||
const successor = resolution.successorName?.trim() || 'its Pulse agent';
|
||||
return `Moved to ${successor}. This is not a recovery: check the agent for the current reading.`;
|
||||
}
|
||||
return resolution.summary?.trim() || undefined;
|
||||
}
|
||||
|
||||
export function getAlertHistoryStatusPresentation(
|
||||
status?: string | null,
|
||||
): AlertHistoryStatusPresentation {
|
||||
@@ -99,6 +118,17 @@ export function getAlertHistoryStatusPresentation(
|
||||
};
|
||||
}
|
||||
|
||||
// Closed without recovering: neither a live problem on this row nor a
|
||||
// healthy outcome, so it gets its own neutral-blue badge.
|
||||
if (normalized === ALERT_HISTORY_MOVED_STATUS) {
|
||||
return {
|
||||
label: 'moved to agent',
|
||||
className:
|
||||
'text-xs px-2 py-0.5 rounded-sm bg-blue-100 dark:bg-blue-900/25 text-blue-700 dark:text-blue-300',
|
||||
rowClassName: '',
|
||||
};
|
||||
}
|
||||
|
||||
return {
|
||||
label: normalized || 'resolved',
|
||||
className: 'text-xs px-2 py-0.5 rounded-sm bg-surface-hover text-base-content',
|
||||
|
||||
@@ -2,6 +2,7 @@ package ai
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/alerts"
|
||||
@@ -56,6 +57,7 @@ func (a *AlertManagerAdapter) GetRecentlyResolved(minutes int) []ResolvedAlertIn
|
||||
AlertInfo: convertAlertFromModels(&resolved.Alert),
|
||||
ResolvedTime: resolved.ResolvedTime,
|
||||
Duration: formatDuration(resolved.ResolvedTime.Sub(resolved.StartTime)),
|
||||
Resolution: resolvedAlertResolution(resolved.Resolution),
|
||||
}
|
||||
result = append(result, info)
|
||||
}
|
||||
@@ -98,6 +100,7 @@ func (a *AlertManagerAdapter) GetAlertHistory(resourceID string, limit int) []Re
|
||||
AlertInfo: convertAlertFromModels(&resolved.Alert),
|
||||
ResolvedTime: resolved.ResolvedTime,
|
||||
Duration: formatDuration(resolved.ResolvedTime.Sub(resolved.StartTime)),
|
||||
Resolution: resolvedAlertResolution(resolved.Resolution),
|
||||
}
|
||||
result = append(result, info)
|
||||
if len(result) >= limit {
|
||||
@@ -245,3 +248,12 @@ func formatDuration(d time.Duration) string {
|
||||
}
|
||||
return fmt.Sprintf("%d days", days)
|
||||
}
|
||||
|
||||
// resolvedAlertResolution is the one-line account of a close that was not a
|
||||
// recovery, empty for an ordinary resolve.
|
||||
func resolvedAlertResolution(resolution *models.AlertResolution) string {
|
||||
if resolution == nil {
|
||||
return ""
|
||||
}
|
||||
return strings.TrimSpace(resolution.Summary)
|
||||
}
|
||||
|
||||
@@ -31,6 +31,9 @@ type ResolvedAlertInfo struct {
|
||||
AlertInfo
|
||||
ResolvedTime time.Time `json:"resolved_time"`
|
||||
Duration string `json:"total_duration"` // How long the alert lasted
|
||||
// Resolution is set only when the alert closed without recovering, such
|
||||
// as a node metric handed to its Pulse agent.
|
||||
Resolution string `json:"resolution,omitempty"`
|
||||
}
|
||||
|
||||
// AlertProvider provides access to the current alert state
|
||||
@@ -131,6 +134,12 @@ func (s *Service) buildAlertContext() string {
|
||||
}
|
||||
for i := 0; i < limit; i++ {
|
||||
a := recentlyResolved[i]
|
||||
if resolution := strings.TrimSpace(a.Resolution); resolution != "" {
|
||||
sections = append(sections, fmt.Sprintf("- **%s** on %s: %s (lasted %s, closed %s ago) %s",
|
||||
a.Type, a.ResourceName, a.Message, a.Duration,
|
||||
formatTimeAgo(a.ResolvedTime), resolution))
|
||||
continue
|
||||
}
|
||||
sections = append(sections, fmt.Sprintf("- **%s** on %s: %s (lasted %s, resolved %s ago)",
|
||||
a.Type, a.ResourceName, a.Message, a.Duration,
|
||||
formatTimeAgo(a.ResolvedTime)))
|
||||
|
||||
@@ -5,6 +5,8 @@ import (
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/models"
|
||||
)
|
||||
|
||||
type stubAlertProvider struct {
|
||||
@@ -151,6 +153,61 @@ func TestService_buildAlertContext_NoActiveWithResolved(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// The assistant reads recently resolved alerts through the adapter, so a node
|
||||
// alert handed to its Pulse agent must reach the prompt as a handover and not
|
||||
// as "resolved", which the model would read as the node being healthy.
|
||||
func TestService_buildAlertContext_HandoverIsNotARecovery(t *testing.T) {
|
||||
now := time.Now()
|
||||
summary := "Alert moved to pve1 (Host Agent). This is not a recovery: check the agent for the current reading."
|
||||
adapter := NewAlertManagerAdapter(&stubAlertManager{resolved: []models.ResolvedAlert{
|
||||
{
|
||||
Alert: models.Alert{
|
||||
ID: "pve1-memory",
|
||||
Type: "memory",
|
||||
Level: "warning",
|
||||
ResourceID: "pve1",
|
||||
ResourceName: "pve1",
|
||||
Message: "Memory usage at 95%",
|
||||
StartTime: now.Add(-time.Hour),
|
||||
Resolution: &models.AlertResolution{Reason: "moved_to_agent", Summary: summary},
|
||||
},
|
||||
ResolvedTime: now.Add(-time.Minute),
|
||||
},
|
||||
{
|
||||
Alert: models.Alert{
|
||||
ID: "pve2-cpu",
|
||||
Type: "cpu",
|
||||
Level: "warning",
|
||||
ResourceID: "pve2",
|
||||
ResourceName: "pve2",
|
||||
Message: "CPU usage at 90%",
|
||||
StartTime: now.Add(-time.Hour),
|
||||
},
|
||||
ResolvedTime: now.Add(-time.Minute),
|
||||
},
|
||||
}})
|
||||
|
||||
resolved := adapter.GetRecentlyResolved(30)
|
||||
if len(resolved) != 2 || resolved[0].Resolution != summary || resolved[1].Resolution != "" {
|
||||
t.Fatalf("GetRecentlyResolved = %+v, want only the handover to carry a resolution", resolved)
|
||||
}
|
||||
if history := adapter.GetAlertHistory("pve1", 5); len(history) != 1 || history[0].Resolution != summary {
|
||||
t.Fatalf("GetAlertHistory = %+v, want the handover resolution", history)
|
||||
}
|
||||
|
||||
s := &Service{}
|
||||
s.SetAlertProvider(adapter)
|
||||
ctx := s.buildAlertContext()
|
||||
if !strings.Contains(ctx, "- **memory** on pve1: Memory usage at 95% (lasted ") ||
|
||||
!strings.Contains(ctx, ", closed 1 minute ago) "+summary) {
|
||||
t.Fatalf("expected the handover line, got: %s", ctx)
|
||||
}
|
||||
if !strings.Contains(ctx, "- **cpu** on pve2: CPU usage at 90% (lasted ") ||
|
||||
!strings.Contains(ctx, ", resolved 1 minute ago)") {
|
||||
t.Fatalf("expected the ordinary recovery line, got: %s", ctx)
|
||||
}
|
||||
}
|
||||
|
||||
func TestService_buildTargetAlertContext(t *testing.T) {
|
||||
s := &Service{}
|
||||
s.SetAlertProvider(&stubAlertProvider{
|
||||
|
||||
@@ -474,6 +474,17 @@ func (s *IncidentStore) RecordAlertUnacknowledged(alert *alerts.Alert, user stri
|
||||
s.saveAsync()
|
||||
}
|
||||
|
||||
// alertResolvedEventSummary is the timeline line for an alert's close: the
|
||||
// handover account when it closed without recovering, else "Alert resolved".
|
||||
func alertResolvedEventSummary(alert *alerts.Alert) string {
|
||||
if alert != nil {
|
||||
if summary := alert.Resolution.Summary(); summary != "" {
|
||||
return summary
|
||||
}
|
||||
}
|
||||
return "Alert resolved"
|
||||
}
|
||||
|
||||
// RecordAlertResolved records a resolved event and closes the incident.
|
||||
func (s *IncidentStore) RecordAlertResolved(alert *alerts.Alert, resolvedAt time.Time) {
|
||||
if alert == nil {
|
||||
@@ -507,7 +518,7 @@ func (s *IncidentStore) RecordAlertResolved(alert *alerts.Alert, resolvedAt time
|
||||
shell.OccurrenceClosedAt = cloneTime(resolvedAt)
|
||||
|
||||
if !s.projectsFromCanonicalLocked() {
|
||||
s.addEventAtLocked(shell, IncidentEventAlertResolved, resolvedAt, "Alert resolved", map[string]interface{}{
|
||||
s.addEventAtLocked(shell, IncidentEventAlertResolved, resolvedAt, alertResolvedEventSummary(alert), map[string]interface{}{
|
||||
"resolved_at": resolvedAt.Format(time.RFC3339),
|
||||
})
|
||||
}
|
||||
@@ -563,7 +574,7 @@ func (s *IncidentStore) EnsureAlertOccurrence(alert *alerts.Alert, resolvedAt *t
|
||||
if resolvedAt != nil && !resolvedAt.IsZero() && (shell.OccurrenceRefiredAt == nil || !resolvedAt.Before(*shell.OccurrenceRefiredAt)) {
|
||||
shell.OccurrenceClosedAt = cloneTime(*resolvedAt)
|
||||
if !hasIncidentEventType(shell.Events, IncidentEventAlertResolved) {
|
||||
s.addEventAtLocked(shell, IncidentEventAlertResolved, *resolvedAt, "Alert resolved", map[string]interface{}{
|
||||
s.addEventAtLocked(shell, IncidentEventAlertResolved, *resolvedAt, alertResolvedEventSummary(alert), map[string]interface{}{
|
||||
"resolved_at": resolvedAt.Format(time.RFC3339),
|
||||
"projection_source": incidentSnapshotSource,
|
||||
})
|
||||
@@ -1113,6 +1124,13 @@ func incidentEventSummaryFromChange(change unifiedresources.ResourceChange, even
|
||||
case IncidentEventAlertUnsnoozed:
|
||||
return "Alert notifications resumed"
|
||||
case IncidentEventAlertResolved:
|
||||
// A handover closes the alert without recovering; its change reason
|
||||
// says so and where the condition went.
|
||||
if _, ok := stringMetadata(change.Metadata, unifiedresources.MetadataAlertResolution); ok {
|
||||
if reason := strings.TrimSpace(change.Reason); reason != "" {
|
||||
return reason
|
||||
}
|
||||
}
|
||||
return "Alert resolved"
|
||||
default:
|
||||
if summary := strings.TrimSpace(change.Reason); summary != "" {
|
||||
|
||||
@@ -239,6 +239,39 @@ func TestIncidentStore_EqualTimestampLifecycleKeepsResolvedState(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// Without a canonical resource timeline the store writes its own close event,
|
||||
// which must say the alert moved rather than resolved.
|
||||
func TestIncidentStore_HandoverCloseIsNotARecovery(t *testing.T) {
|
||||
store := NewIncidentStore(IncidentStoreConfig{MaxIncidents: 10, MaxEventsPerIncident: 10, MaxAgeDays: 30})
|
||||
startedAt := time.Now().UTC().Truncate(time.Second).Add(-time.Hour)
|
||||
movedAt := startedAt.Add(30 * time.Minute)
|
||||
alert := &alerts.Alert{
|
||||
ID: "pve1-memory",
|
||||
Type: "memory",
|
||||
ResourceID: "pve1",
|
||||
ResourceName: "pve1",
|
||||
Message: "Memory usage at 95%",
|
||||
StartTime: startedAt,
|
||||
LastSeen: startedAt,
|
||||
Resolution: &alerts.AlertResolution{Reason: alerts.AlertResolutionMovedToAgent, SuccessorName: "pve1 (Host Agent)"},
|
||||
}
|
||||
summary := "Alert moved to pve1 (Host Agent). This is not a recovery: check the agent for the current reading."
|
||||
|
||||
store.RecordAlertFired(alert)
|
||||
store.RecordAlertResolved(alert, movedAt)
|
||||
timeline := store.GetTimelineByAlertAt(alert.ID, startedAt)
|
||||
if timeline == nil || len(timeline.Events) != 2 || timeline.Events[1].Summary != summary {
|
||||
t.Fatalf("recorded timeline = %#v, want the handover close", timeline)
|
||||
}
|
||||
|
||||
snapshot := alert.Clone()
|
||||
snapshot.ID = "pve1-memory-imported"
|
||||
repaired := store.EnsureAlertOccurrence(snapshot, &movedAt)
|
||||
if repaired == nil || len(repaired.Events) != 2 || repaired.Events[1].Summary != summary {
|
||||
t.Fatalf("repaired timeline = %#v, want the handover close", repaired)
|
||||
}
|
||||
}
|
||||
|
||||
func TestIncidentStore_ProjectsCanonicalTimelineWhenAttached(t *testing.T) {
|
||||
store := NewIncidentStore(IncidentStoreConfig{
|
||||
MaxIncidents: 10,
|
||||
|
||||
@@ -3499,6 +3499,12 @@ func (p *PatrolService) seedHealthAndAlertsState(snap patrolRuntimeState, scoped
|
||||
sb.WriteString("# Recently Resolved Alerts\n")
|
||||
for _, r := range alerts {
|
||||
ago := seedFormatTimeAgo(now, r.ResolvedTime)
|
||||
// A handover closes the alert without the condition clearing, so
|
||||
// it must not read as the resource having recovered.
|
||||
if r.Resolution != nil && strings.TrimSpace(r.Resolution.Summary) != "" {
|
||||
sb.WriteString(fmt.Sprintf("- %s — closed %s. %s\n", seedAlertSubject(r.Alert), ago, strings.TrimSpace(r.Resolution.Summary)))
|
||||
continue
|
||||
}
|
||||
sb.WriteString(fmt.Sprintf("- %s — resolved %s\n", seedAlertSubject(r.Alert), ago))
|
||||
}
|
||||
sb.WriteString("\n")
|
||||
|
||||
@@ -922,6 +922,42 @@ func TestSeedHealthAndAlerts_NoIssues(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// Patrol's seed must not tell the model a node recovered when its alert only
|
||||
// moved to the node's Pulse agent.
|
||||
func TestSeedHealthAndAlerts_HandoverIsNotARecovery(t *testing.T) {
|
||||
ps := NewPatrolService(nil, nil)
|
||||
cfg := DefaultPatrolConfig()
|
||||
now := time.Now()
|
||||
summary := "Alert moved to pve1 (Host Agent). This is not a recovery: check the agent for the current reading."
|
||||
state := models.StateSnapshot{
|
||||
RecentlyResolved: []models.ResolvedAlert{
|
||||
{
|
||||
Alert: models.Alert{
|
||||
ResourceName: "pve1",
|
||||
Message: "Memory usage at 95%",
|
||||
Resolution: &models.AlertResolution{Reason: "moved_to_agent", Summary: summary},
|
||||
},
|
||||
ResolvedTime: now.Add(-3 * time.Minute),
|
||||
},
|
||||
{
|
||||
Alert: models.Alert{ResourceName: "pve2", Message: "CPU usage at 90%"},
|
||||
ResolvedTime: now.Add(-5 * time.Minute),
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
out := ps.seedHealthAndAlertsState(patrolRuntimeStateForTest(ps, state), nil, cfg, now)
|
||||
if !strings.Contains(out, "- pve1: Memory usage at 95% — closed 3m ago. "+summary+"\n") {
|
||||
t.Fatalf("expected the handover line, got: %s", out)
|
||||
}
|
||||
if strings.Contains(out, "pve1: Memory usage at 95% — resolved") {
|
||||
t.Fatalf("handover reported as a recovery: %s", out)
|
||||
}
|
||||
if !strings.Contains(out, "- pve2: CPU usage at 90% — resolved 5m ago\n") {
|
||||
t.Fatalf("expected the ordinary recovery line, got: %s", out)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSeedHealthAndAlerts_UnknownDiskHealthDoesNotClaimSMARTPassed(t *testing.T) {
|
||||
ps := NewPatrolService(nil, nil)
|
||||
cfg := DefaultPatrolConfig()
|
||||
|
||||
@@ -2364,4 +2364,9 @@ type ResolvedAlertSummary struct {
|
||||
Threshold float64 `json:"threshold,omitempty"`
|
||||
StartTime time.Time `json:"start_time"`
|
||||
ResolvedTime time.Time `json:"resolved_time"`
|
||||
// Resolution is set only when the alert closed without recovering, such
|
||||
// as a node metric handed to its Pulse agent; the successor resource
|
||||
// owns the condition from then on.
|
||||
Resolution string `json:"resolution,omitempty"`
|
||||
SuccessorResourceID string `json:"successor_resource_id,omitempty"`
|
||||
}
|
||||
|
||||
@@ -19,7 +19,7 @@ func (e *PulseToolExecutor) registerAlertsTools() {
|
||||
Actions:
|
||||
- list: List active threshold alerts (CPU > 80%, disk full, etc.)
|
||||
- findings: List AI patrol findings (detected issues)
|
||||
- resolved: List recently resolved alerts
|
||||
- resolved: List recently resolved alerts; one with a resolution closed without recovering
|
||||
- resolve: Mark a finding as resolved
|
||||
- dismiss: Dismiss a finding as not an issue
|
||||
|
||||
@@ -417,7 +417,7 @@ func (e *PulseToolExecutor) executeListResolvedAlerts(_ context.Context, args ma
|
||||
break
|
||||
}
|
||||
|
||||
alerts = append(alerts, ResolvedAlertSummary{
|
||||
summary := ResolvedAlertSummary{
|
||||
ID: alert.ID,
|
||||
Type: alert.Type,
|
||||
Level: alert.Level,
|
||||
@@ -430,7 +430,12 @@ func (e *PulseToolExecutor) executeListResolvedAlerts(_ context.Context, args ma
|
||||
Threshold: alert.Threshold,
|
||||
StartTime: alert.StartTime,
|
||||
ResolvedTime: alert.ResolvedTime,
|
||||
})
|
||||
}
|
||||
if resolution := alert.Resolution; resolution != nil {
|
||||
summary.Resolution = resolution.Summary
|
||||
summary.SuccessorResourceID = resolution.SuccessorResourceID
|
||||
}
|
||||
alerts = append(alerts, summary)
|
||||
}
|
||||
|
||||
if alerts == nil {
|
||||
|
||||
@@ -215,6 +215,49 @@ func TestExecuteListResolvedAlerts(t *testing.T) {
|
||||
if len(resp.Alerts) != 1 || resp.Alerts[0].ID != "a1" {
|
||||
t.Fatalf("unexpected resolved alerts: %+v", resp)
|
||||
}
|
||||
if resp.Alerts[0].Resolution != "" || resp.Alerts[0].SuccessorResourceID != "" {
|
||||
t.Fatalf("a recovery must not carry a resolution: %+v", resp.Alerts[0])
|
||||
}
|
||||
}
|
||||
|
||||
// A node alert handed to its Pulse agent closes without recovering, so the
|
||||
// assistant must see that, and where the condition went, instead of a plain
|
||||
// resolve it would read as the node being healthy again.
|
||||
func TestExecuteListResolvedAlertsReportsHandover(t *testing.T) {
|
||||
alertProv := &mockAlertProvider{}
|
||||
executor := NewPulseToolExecutor(ExecutorConfig{})
|
||||
executor.alertProvider = alertProv
|
||||
now := time.Now()
|
||||
summary := "Alert moved to pve1 (Host Agent). This is not a recovery: check the agent for the current reading."
|
||||
alertProv.On("GetRecentlyResolved", 24*60).Return([]models.ResolvedAlert{{
|
||||
Alert: models.Alert{
|
||||
ID: "pve1-memory",
|
||||
Type: "memory",
|
||||
Level: "warning",
|
||||
ResourceID: "pve1",
|
||||
ResourceName: "pve1",
|
||||
Message: "Memory usage at 95%",
|
||||
StartTime: now.Add(-time.Hour),
|
||||
Resolution: &models.AlertResolution{
|
||||
Reason: "moved_to_agent",
|
||||
SuccessorResourceID: "agent-pve1",
|
||||
SuccessorName: "pve1 (Host Agent)",
|
||||
Summary: summary,
|
||||
},
|
||||
},
|
||||
ResolvedTime: now,
|
||||
}})
|
||||
|
||||
result, _ := executor.executeListResolvedAlerts(context.Background(), map[string]interface{}{})
|
||||
var resp ResolvedAlertsResponse
|
||||
if err := json.Unmarshal([]byte(result.Content[0].Text), &resp); err != nil {
|
||||
t.Fatalf("decode resolved alerts: %v", err)
|
||||
}
|
||||
if len(resp.Alerts) != 1 ||
|
||||
resp.Alerts[0].Resolution != summary ||
|
||||
resp.Alerts[0].SuccessorResourceID != "agent-pve1" {
|
||||
t.Fatalf("resolved alerts = %+v, want the handover and its successor", resp.Alerts)
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteListAlertsAndFindings(t *testing.T) {
|
||||
|
||||
@@ -7,7 +7,9 @@ import (
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/models"
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/operationaltrust"
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources"
|
||||
)
|
||||
|
||||
// newTestHistoryManager creates a HistoryManager using a temp directory
|
||||
@@ -1058,3 +1060,62 @@ func TestSaveHistoryWithRetry_RestoresBackupOnFailure(t *testing.T) {
|
||||
|
||||
_ = hm
|
||||
}
|
||||
|
||||
// The state snapshot, the websocket and the assistant read recently resolved
|
||||
// alerts through GetRecentlyResolved, so a handover must reach them as one
|
||||
// and not as a recovery.
|
||||
func TestRecentlyResolvedCarriesHandoverResolution(t *testing.T) {
|
||||
manager := newUnifiedEvalParityManager(t)
|
||||
manager.UpdateConfig(AlertConfig{
|
||||
Enabled: true,
|
||||
NodeDefaults: ThresholdConfig{
|
||||
CPU: &HysteresisThreshold{Trigger: 80, Clear: 75},
|
||||
Memory: &HysteresisThreshold{Trigger: 85, Clear: 80},
|
||||
},
|
||||
})
|
||||
disableTestTimeThresholds(manager)
|
||||
node := models.Node{
|
||||
ID: "pve-1",
|
||||
Name: "pve-1",
|
||||
Instance: "test",
|
||||
Status: "online",
|
||||
CPU: 0.95,
|
||||
Memory: models.Memory{Total: 8 << 30, Used: 7 << 30, Usage: 95},
|
||||
}
|
||||
manager.CheckNode(node)
|
||||
release := func(metric string, resolution *AlertResolution) {
|
||||
t.Helper()
|
||||
spec, err := buildCanonicalMetricSpec(node.ID, node.Name, unifiedresources.ResourceType("node"), metric, nil)
|
||||
if err != nil {
|
||||
t.Fatalf("build %s spec: %v", metric, err)
|
||||
}
|
||||
manager.releaseCanonicalMetricAlert(spec, node.Name, node.Name, node.Instance, "node", 0, resolution)
|
||||
}
|
||||
release("memory", &AlertResolution{
|
||||
Reason: AlertResolutionMovedToAgent,
|
||||
SuccessorResourceID: "agent-pve-1",
|
||||
SuccessorName: "pve-1 (Host Agent)",
|
||||
})
|
||||
release("cpu", nil)
|
||||
|
||||
byType := map[string]models.ResolvedAlert{}
|
||||
for _, resolved := range manager.GetRecentlyResolved() {
|
||||
byType[resolved.Type] = resolved
|
||||
}
|
||||
moved, ok := byType["memory"]
|
||||
if !ok {
|
||||
t.Fatalf("recently resolved = %+v, want the moved memory alert", byType)
|
||||
}
|
||||
want := models.AlertResolution{
|
||||
Reason: "moved_to_agent",
|
||||
SuccessorResourceID: "agent-pve-1",
|
||||
SuccessorName: "pve-1 (Host Agent)",
|
||||
Summary: "Alert moved to pve-1 (Host Agent). This is not a recovery: check the agent for the current reading.",
|
||||
}
|
||||
if moved.Resolution == nil || *moved.Resolution != want {
|
||||
t.Fatalf("moved resolution = %+v, want %+v", moved.Resolution, want)
|
||||
}
|
||||
if recovered, ok := byType["cpu"]; !ok || recovered.Resolution != nil {
|
||||
t.Fatalf("recovered cpu alert = %+v, want an ordinary resolve with no resolution", recovered)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -285,6 +285,7 @@ func markOperationalResolved(
|
||||
}
|
||||
alert.Evidence = appendOperationalEvidence(alert.Evidence, recoveryEvidence.Clone())
|
||||
recoveryEvidenceIDs := []string{recoveryEvidence.ID}
|
||||
cause := resolvedTransitionCause(alert.Resolution)
|
||||
from := record.State
|
||||
record.State = operationaltrust.OperationalResolved
|
||||
record.StateChangedAt = resolvedAt
|
||||
@@ -295,7 +296,7 @@ func markOperationalResolved(
|
||||
from,
|
||||
operationaltrust.OperationalResolved,
|
||||
resolvedAt,
|
||||
operationaltrust.TransitionRecoveryEvidence,
|
||||
cause,
|
||||
record.CauseKey,
|
||||
recoveryEvidenceIDs,
|
||||
)
|
||||
@@ -308,7 +309,7 @@ func markOperationalResolved(
|
||||
From: from,
|
||||
To: operationaltrust.OperationalResolved,
|
||||
At: resolvedAt,
|
||||
Cause: operationaltrust.TransitionRecoveryEvidence,
|
||||
Cause: cause,
|
||||
CauseKey: record.CauseKey,
|
||||
EvidenceIDs: recoveryEvidenceIDs,
|
||||
Reason: alert.Resolution.Summary(),
|
||||
@@ -319,6 +320,16 @@ func markOperationalResolved(
|
||||
)
|
||||
}
|
||||
|
||||
// resolvedTransitionCause names why a record entered resolved. Only a close
|
||||
// with no typed resolution is a recovery; a handover keeps its own cause so
|
||||
// timelines and alert-quality counts never report it as one.
|
||||
func resolvedTransitionCause(resolution *AlertResolution) operationaltrust.TransitionCause {
|
||||
if resolution != nil && resolution.Reason == AlertResolutionMovedToAgent {
|
||||
return operationaltrust.TransitionOwnershipTransferred
|
||||
}
|
||||
return operationaltrust.TransitionRecoveryEvidence
|
||||
}
|
||||
|
||||
func (m *Manager) newResolvedAlert(
|
||||
alert *Alert,
|
||||
resolvedAt time.Time,
|
||||
|
||||
@@ -504,6 +504,15 @@ func TestReleasedMetricAlertCarriesResolutionInsteadOfRecovery(t *testing.T) {
|
||||
if err := transition.Validate(); err != nil {
|
||||
t.Fatalf("resolve transition Validate() error = %v", err)
|
||||
}
|
||||
// Patrol's timeline prints the cause above the reason, so a handover
|
||||
// must not be labelled as recovery evidence.
|
||||
if transition.Cause != operationaltrust.TransitionOwnershipTransferred {
|
||||
t.Fatalf("resolve transition cause = %q, want %q", transition.Cause, operationaltrust.TransitionOwnershipTransferred)
|
||||
}
|
||||
if recorded := moved.Alert.Transitions[len(moved.Alert.Transitions)-1]; recorded.ID != transition.ID ||
|
||||
recorded.Cause != operationaltrust.TransitionOwnershipTransferred {
|
||||
t.Fatalf("recorded timeline transition = %+v, want the ownership transfer", recorded)
|
||||
}
|
||||
closing := moved.Alert.Evidence[len(moved.Alert.Evidence)-1]
|
||||
if closing.ID != transition.EvidenceIDs[0] ||
|
||||
closing.Reason == nil ||
|
||||
@@ -520,6 +529,10 @@ func TestReleasedMetricAlertCarriesResolutionInsteadOfRecovery(t *testing.T) {
|
||||
last.Reason.Code != legacyAlertRecoveryEvidenceReason {
|
||||
t.Fatalf("plain release evidence = %+v, want legacy recovery projection", last.Reason)
|
||||
}
|
||||
if recovered.Alert.LatestTransition == nil ||
|
||||
recovered.Alert.LatestTransition.Cause != operationaltrust.TransitionRecoveryEvidence {
|
||||
t.Fatalf("plain release transition = %+v, want recovery_evidence", recovered.Alert.LatestTransition)
|
||||
}
|
||||
|
||||
var historyResolution *AlertResolution
|
||||
for _, entry := range manager.GetAlertHistory(10) {
|
||||
|
||||
@@ -301,7 +301,8 @@ func (m *Manager) GetRecentlyResolved() []models.ResolvedAlert {
|
||||
StartTime: exported.StartTime,
|
||||
Acknowledged: exported.Acknowledged,
|
||||
// exported is a deep clone, so the map is already private.
|
||||
Metadata: exported.Metadata,
|
||||
Metadata: exported.Metadata,
|
||||
Resolution: modelAlertResolution(exported.Resolution),
|
||||
},
|
||||
ResolvedTime: alert.ResolvedTime,
|
||||
})
|
||||
@@ -309,6 +310,20 @@ func (m *Manager) GetRecentlyResolved() []models.ResolvedAlert {
|
||||
return resolved
|
||||
}
|
||||
|
||||
// modelAlertResolution projects a non-recovery close into the state model, so
|
||||
// snapshot and assistant readers can tell a handover from a recovery.
|
||||
func modelAlertResolution(resolution *AlertResolution) *models.AlertResolution {
|
||||
if resolution == nil {
|
||||
return nil
|
||||
}
|
||||
return &models.AlertResolution{
|
||||
Reason: string(resolution.Reason),
|
||||
SuccessorResourceID: resolution.SuccessorResourceID,
|
||||
SuccessorName: resolution.SuccessorName,
|
||||
Summary: resolution.Summary(),
|
||||
}
|
||||
}
|
||||
|
||||
// GetResolvedAlert returns a copy of a recently resolved alert by ID.
|
||||
func (m *Manager) GetResolvedAlert(alertID string) *ResolvedAlert {
|
||||
// Write lock: getResolvedAlertNoLock backfills the resolvedAlias map on a
|
||||
|
||||
@@ -1337,6 +1337,10 @@ func cloneAlert(src Alert) Alert {
|
||||
dest.Metadata[k] = v
|
||||
}
|
||||
}
|
||||
if src.Resolution != nil {
|
||||
resolution := *src.Resolution
|
||||
dest.Resolution = &resolution
|
||||
}
|
||||
return dest
|
||||
}
|
||||
|
||||
|
||||
@@ -96,6 +96,43 @@ func TestAlertLastSeenWireContract(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// A resolved alert that closed without recovering says so on the state and
|
||||
// websocket payloads; an ordinary recovery carries no resolution at all.
|
||||
func TestAlertResolutionWireContract(t *testing.T) {
|
||||
recovered, err := json.Marshal(ResolvedAlert{Alert: Alert{ID: "alert-1"}})
|
||||
if err != nil {
|
||||
t.Fatalf("marshal recovered alert: %v", err)
|
||||
}
|
||||
if strings.Contains(string(recovered), "resolution") {
|
||||
t.Fatalf("payload = %s, a recovery must not carry a resolution", recovered)
|
||||
}
|
||||
|
||||
moved := ResolvedAlert{Alert: Alert{ID: "alert-1", Resolution: &AlertResolution{
|
||||
Reason: "moved_to_agent",
|
||||
SuccessorResourceID: "agent-pve-1",
|
||||
SuccessorName: "pve-1 (Host Agent)",
|
||||
Summary: "Alert moved to pve-1 (Host Agent). This is not a recovery: check the agent for the current reading.",
|
||||
}}}
|
||||
payload, err := json.Marshal(moved)
|
||||
if err != nil {
|
||||
t.Fatalf("marshal moved alert: %v", err)
|
||||
}
|
||||
want := `"resolution":{"reason":"moved_to_agent","successorResourceId":"agent-pve-1","successorName":"pve-1 (Host Agent)","summary":"Alert moved to pve-1 (Host Agent). This is not a recovery: check the agent for the current reading."}`
|
||||
if !strings.Contains(string(payload), want) {
|
||||
t.Fatalf("payload = %s, want %s", payload, want)
|
||||
}
|
||||
|
||||
// The state store keeps its own copy of each resolution.
|
||||
state := NewState()
|
||||
state.UpdateRecentlyResolved([]ResolvedAlert{moved})
|
||||
moved.Resolution.SuccessorName = "changed"
|
||||
snapshot := state.GetSnapshot().RecentlyResolved
|
||||
if len(snapshot) != 1 || snapshot[0].Resolution == nil ||
|
||||
snapshot[0].Resolution.SuccessorName != "pve-1 (Host Agent)" {
|
||||
t.Fatalf("snapshot = %+v, want the resolution as it was recorded", snapshot)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDockerHostCollectionModeWireContract(t *testing.T) {
|
||||
payload, err := json.Marshal(DockerHost{
|
||||
ID: "docker-host-1",
|
||||
|
||||
@@ -110,6 +110,20 @@ type Alert struct {
|
||||
// Metadata carries alert-engine annotations (notably resourceType) so the
|
||||
// frontend can classify an alert without re-deriving resource identity.
|
||||
Metadata map[string]interface{} `json:"metadata,omitempty"`
|
||||
// Resolution is set only on a resolved alert whose close was not a
|
||||
// recovery, so readers never report its resource as healthy.
|
||||
Resolution *AlertResolution `json:"resolution,omitempty"`
|
||||
}
|
||||
|
||||
// AlertResolution is the state projection of the alert engine's
|
||||
// AlertResolution: why an alert closed without its condition clearing.
|
||||
type AlertResolution struct {
|
||||
Reason string `json:"reason"`
|
||||
SuccessorResourceID string `json:"successorResourceId,omitempty"`
|
||||
SuccessorName string `json:"successorName,omitempty"`
|
||||
// Summary is the alert engine's one-line account, such as "Alert moved
|
||||
// to pve1 (Host Agent). This is not a recovery: ...".
|
||||
Summary string `json:"summary,omitempty"`
|
||||
}
|
||||
|
||||
// ResolvedAlert represents a recently resolved alert
|
||||
|
||||
@@ -238,6 +238,68 @@ func TestMonitor_HandleAlertLifecycle_WritesCanonicalChanges(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// A node alert handed to its Pulse agent closes without recovering. The
|
||||
// resource history and the alert's incident timeline (the Alerts history row
|
||||
// expansion) must say where it went, not "Alert resolved: Memory usage at 95%".
|
||||
func TestMonitor_HandleAlertLifecycle_HandoverCloseIsNotARecovery(t *testing.T) {
|
||||
resourceStore := unifiedresources.NewMemoryStore()
|
||||
incidentStore := memory.NewIncidentStore(memory.IncidentStoreConfig{})
|
||||
m := &Monitor{
|
||||
incidentStore: incidentStore,
|
||||
resourceStore: unifiedresources.NewMonitorAdapter(unifiedresources.NewRegistry(resourceStore)),
|
||||
}
|
||||
incidentStore.SetResourceTimelineStore(m.resourceStore.(memory.IncidentTimelineStore))
|
||||
|
||||
startedAt := time.Date(2026, 10, 6, 9, 0, 0, 0, time.UTC)
|
||||
movedAt := startedAt.Add(2 * time.Hour)
|
||||
alert := &alerts.Alert{
|
||||
ID: "pve1-memory",
|
||||
Type: "memory",
|
||||
Level: alerts.AlertLevelWarning,
|
||||
ResourceID: "pve1",
|
||||
Message: "Memory usage at 95%",
|
||||
Value: 95,
|
||||
Threshold: 85,
|
||||
StartTime: startedAt,
|
||||
}
|
||||
m.handleAlertLifecycleEvent(alerts.LifecycleEvent{Type: eventlog.TypeFired, OccurredAt: startedAt, Alert: alert})
|
||||
closed := alert.Clone()
|
||||
closed.Resolution = &alerts.AlertResolution{
|
||||
Reason: alerts.AlertResolutionMovedToAgent,
|
||||
SuccessorResourceID: "agent-pve1",
|
||||
SuccessorName: "pve1 (Host Agent)",
|
||||
}
|
||||
m.handleAlertLifecycleEvent(alerts.LifecycleEvent{Type: eventlog.TypeResolved, OccurredAt: movedAt, Alert: closed})
|
||||
|
||||
summary := "Alert moved to pve1 (Host Agent). This is not a recovery: check the agent for the current reading."
|
||||
changes, err := resourceStore.GetRecentChanges("pve1", time.Time{}, 10)
|
||||
require.NoError(t, err)
|
||||
require.Len(t, changes, 2)
|
||||
require.Equal(t, unifiedresources.ChangeAlertResolved, changes[0].Kind)
|
||||
require.Equal(t, summary, changes[0].Reason)
|
||||
require.Equal(t, "moved_to_agent", changes[0].Metadata[unifiedresources.MetadataAlertResolution])
|
||||
require.Equal(t, unifiedresources.ChangeAlertFired, changes[1].Kind)
|
||||
require.NotContains(t, changes[1].Metadata, unifiedresources.MetadataAlertResolution)
|
||||
|
||||
timeline := incidentStore.GetTimelineByAlertAt(alert.ID, startedAt)
|
||||
require.NotNil(t, timeline)
|
||||
require.Len(t, timeline.Events, 2)
|
||||
require.Equal(t, memory.IncidentEventAlertResolved, timeline.Events[1].Type)
|
||||
require.Equal(t, summary, timeline.Events[1].Summary)
|
||||
|
||||
// An ordinary recovery keeps its existing wording.
|
||||
recovered := alert.Clone()
|
||||
recovered.ID = "pve1-cpu"
|
||||
recovered.Type = "cpu"
|
||||
recovered.Message = "CPU usage at 90%"
|
||||
m.handleAlertLifecycleEvent(alerts.LifecycleEvent{Type: eventlog.TypeFired, OccurredAt: startedAt, Alert: recovered})
|
||||
m.handleAlertLifecycleEvent(alerts.LifecycleEvent{Type: eventlog.TypeResolved, OccurredAt: movedAt.Add(time.Minute), Alert: recovered})
|
||||
recoveredTimeline := incidentStore.GetTimelineByAlertAt(recovered.ID, startedAt)
|
||||
require.NotNil(t, recoveredTimeline)
|
||||
require.Len(t, recoveredTimeline.Events, 2)
|
||||
require.Equal(t, "Alert resolved", recoveredTimeline.Events[1].Summary)
|
||||
}
|
||||
|
||||
func TestPausedDeliveryStillBuildsTimelineThroughRealAlertLifecycle(t *testing.T) {
|
||||
manager := alerts.NewManagerWithDataDir(t.TempDir())
|
||||
t.Cleanup(manager.Stop)
|
||||
|
||||
@@ -576,7 +576,7 @@ func (m *Monitor) recordAlertTimelineChange(alert *alerts.Alert, kind unifiedres
|
||||
return
|
||||
}
|
||||
|
||||
change := unifiedresources.BuildAlertTimelineChange(alert.ResourceID, kind, occurredAt, actor, unifiedresources.AlertTimelineChange{
|
||||
timelineChange := unifiedresources.AlertTimelineChange{
|
||||
AlertIdentifier: alert.ID,
|
||||
AlertStartedAt: alert.StartTime,
|
||||
AlertType: alert.Type,
|
||||
@@ -585,7 +585,12 @@ func (m *Monitor) recordAlertTimelineChange(alert *alerts.Alert, kind unifiedres
|
||||
AlertValue: alert.Value,
|
||||
AlertThreshold: alert.Threshold,
|
||||
AlertMetadata: alert.Metadata,
|
||||
})
|
||||
}
|
||||
if summary := alert.Resolution.Summary(); summary != "" {
|
||||
timelineChange.ResolutionReason = string(alert.Resolution.Reason)
|
||||
timelineChange.ResolutionSummary = summary
|
||||
}
|
||||
change := unifiedresources.BuildAlertTimelineChange(alert.ResourceID, kind, occurredAt, actor, timelineChange)
|
||||
if change == nil {
|
||||
return
|
||||
}
|
||||
|
||||
@@ -493,8 +493,12 @@ const (
|
||||
TransitionSuppression TransitionCause = "suppression"
|
||||
TransitionSuppressionExpired TransitionCause = "suppression_expired"
|
||||
TransitionRecoveryEvidence TransitionCause = "recovery_evidence"
|
||||
TransitionCollectionStale TransitionCause = "collection_stale"
|
||||
TransitionCollectionUnknown TransitionCause = "collection_unknown"
|
||||
// TransitionOwnershipTransferred closes a record whose condition another
|
||||
// record now owns, such as a node metric a linked Pulse agent took over.
|
||||
// The condition was not observed to clear, so it is not a recovery.
|
||||
TransitionOwnershipTransferred TransitionCause = "ownership_transferred"
|
||||
TransitionCollectionStale TransitionCause = "collection_stale"
|
||||
TransitionCollectionUnknown TransitionCause = "collection_unknown"
|
||||
)
|
||||
|
||||
func (cause TransitionCause) valid() bool {
|
||||
@@ -505,6 +509,7 @@ func (cause TransitionCause) valid() bool {
|
||||
TransitionSuppression,
|
||||
TransitionSuppressionExpired,
|
||||
TransitionRecoveryEvidence,
|
||||
TransitionOwnershipTransferred,
|
||||
TransitionCollectionStale,
|
||||
TransitionCollectionUnknown:
|
||||
return true
|
||||
@@ -617,6 +622,13 @@ func (transition LifecycleTransition) validateWithoutID() error {
|
||||
if len(canonicalIDs(transition.EvidenceIDs)) == 0 {
|
||||
return errors.New("recovery transition requires evidence")
|
||||
}
|
||||
case TransitionOwnershipTransferred:
|
||||
if transition.To != OperationalResolved {
|
||||
return errors.New("ownership transfer must enter resolved state")
|
||||
}
|
||||
if len(canonicalIDs(transition.EvidenceIDs)) == 0 {
|
||||
return errors.New("ownership transfer requires evidence")
|
||||
}
|
||||
case TransitionCollectionStale:
|
||||
if transition.To != OperationalStale {
|
||||
return errors.New("collection-stale transition must enter stale state")
|
||||
|
||||
@@ -235,6 +235,48 @@ func TestLifecycleTransitionEnforcesOperationalSemantics(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// A close that hands the condition to another record is not a recovery, so it
|
||||
// has its own cause. It may only close the record, and only with the evidence
|
||||
// that recorded the handover.
|
||||
func TestOwnershipTransferOnlyClosesWithEvidence(t *testing.T) {
|
||||
at := time.Date(2026, 10, 6, 15, 0, 0, 0, time.UTC)
|
||||
if _, err := NewTransitionID(
|
||||
"record-1",
|
||||
OperationalOpen,
|
||||
OperationalResolved,
|
||||
at,
|
||||
TransitionOwnershipTransferred,
|
||||
"memory-high",
|
||||
[]string{"evidence-handover"},
|
||||
); err != nil {
|
||||
t.Fatalf("open-to-resolved ownership transfer error = %v", err)
|
||||
}
|
||||
if _, err := NewTransitionID(
|
||||
"record-1",
|
||||
OperationalAcknowledged,
|
||||
OperationalResolved,
|
||||
at,
|
||||
TransitionOwnershipTransferred,
|
||||
"memory-high",
|
||||
nil,
|
||||
); err == nil || !strings.Contains(err.Error(), "requires evidence") {
|
||||
t.Fatalf("evidenceless ownership transfer error = %v", err)
|
||||
}
|
||||
for _, to := range []OperationalState{OperationalResolving, OperationalOpen, OperationalStale} {
|
||||
if _, err := NewTransitionID(
|
||||
"record-1",
|
||||
OperationalAcknowledged,
|
||||
to,
|
||||
at,
|
||||
TransitionOwnershipTransferred,
|
||||
"memory-high",
|
||||
[]string{"evidence-handover"},
|
||||
); err == nil || !strings.Contains(err.Error(), "must enter resolved") {
|
||||
t.Fatalf("ownership transfer into %q error = %v", to, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestOperationalRecordRequiresResolutionEvidence(t *testing.T) {
|
||||
firstObserved := time.Date(2026, 7, 18, 20, 0, 0, 0, time.UTC)
|
||||
resolvedAt := firstObserved.Add(5 * time.Minute)
|
||||
|
||||
@@ -16,6 +16,9 @@ const (
|
||||
MetadataAlertMessage = "alert_message"
|
||||
MetadataAlertValue = "alert_value"
|
||||
MetadataAlertThreshold = "alert_threshold"
|
||||
// MetadataAlertResolution is the reason code of a resolve that was not a
|
||||
// recovery, such as "moved_to_agent". Ordinary recoveries omit it.
|
||||
MetadataAlertResolution = "alert_resolution"
|
||||
MetadataCommand = "command"
|
||||
MetadataSuccess = "success"
|
||||
MetadataOutputExcerpt = "output_excerpt"
|
||||
@@ -38,6 +41,11 @@ type AlertTimelineChange struct {
|
||||
AlertValue float64
|
||||
AlertThreshold float64
|
||||
AlertMetadata map[string]any
|
||||
// ResolutionReason and ResolutionSummary describe a resolve that was not
|
||||
// a recovery. They are empty for an ordinary recovery and ignored on
|
||||
// every other lifecycle kind.
|
||||
ResolutionReason string
|
||||
ResolutionSummary string
|
||||
}
|
||||
|
||||
// BuildAlertTimelineChange constructs a canonical resource change for alert
|
||||
@@ -74,6 +82,9 @@ func BuildAlertTimelineChange(resourceID string, kind ChangeKind, occurredAt tim
|
||||
if !alert.AlertStartedAt.IsZero() {
|
||||
change.Metadata[MetadataAlertStartedAt] = alert.AlertStartedAt.UTC().Format(time.RFC3339Nano)
|
||||
}
|
||||
if reason := strings.TrimSpace(alert.ResolutionReason); kind == ChangeAlertResolved && reason != "" {
|
||||
change.Metadata[MetadataAlertResolution] = reason
|
||||
}
|
||||
for key, value := range cloneChangeMetadata(alert.AlertMetadata) {
|
||||
if _, exists := change.Metadata[key]; exists {
|
||||
continue
|
||||
@@ -178,6 +189,11 @@ func alertChangeReason(kind ChangeKind, alert AlertTimelineChange) string {
|
||||
case ChangeAlertUnsnoozed:
|
||||
return "Alert notifications resumed"
|
||||
case ChangeAlertResolved:
|
||||
// A close that was not a recovery must not read as one: the alert's
|
||||
// own message describes the condition that is still unresolved.
|
||||
if summary := strings.TrimSpace(alert.ResolutionSummary); summary != "" {
|
||||
return summary
|
||||
}
|
||||
if message != "" {
|
||||
return fmt.Sprintf("Alert resolved: %s", message)
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user