Show a moved alert as moved in Patrol, history and assistant

A node alert handed to its Pulse agent closes with an AlertResolution
(#2580), and notifications already say it moved. Every in-app reader
still treated the close as a recovery: Patrol's detail wore a green
Resolved badge over a timeline labelled "Recovery evidence", the alert
history listed the row as resolved and its incident timeline said
"Alert resolved: Node memory at 95%", and the assistant, Patrol's seed
prompt and the pulse_alerts tool saw a plain resolve because
GetRecentlyResolved dropped the resolution.

Operational trust gains a non-recovery cause, ownership_transferred,
which may only enter resolved and only with evidence. The resolve
transition uses it whenever the alert carries a moved_to_agent
resolution, so a handover during a snooze also stops counting as
resolved while snoozed.

models.Alert gains Resolution (reason, successor and the engine's
summary), copied by cloneAlert and filled by GetRecentlyResolved, so
the state snapshot, websocket, assistant context, Patrol seed and the
resolved-alerts tool all report where the alert went. The resolved
resource change carries the summary as its reason with alert_resolution
metadata, and incident memory titles the close with it.

In the UI, the history row reads "moved to agent" with the account in
its title and on the phone card, and Patrol's detail shows a Moved
badge with the handover sentence under the summary.
This commit is contained in:
rcourtman
2026-10-06 16:08:15 +01:00
parent 982a6f6148
commit 23f603da9f
44 changed files with 959 additions and 25 deletions
@@ -1262,6 +1262,10 @@ update, profile rollout, command reachability, or fleet-control authority.
lifecycle presentation evidence. It keeps mock active-alert and history
payloads aligned with the canonical alert contract, but it cannot enroll,
identify, link, command, remove, or otherwise grant authority to an agent.
The optional `models.Alert.Resolution` field is the same kind of
evidence: it says a resolved node alert now belongs to a linked agent's
own alert and names that agent as display text, but the link itself stays
the agent lifecycle's, and the field cannot create, change or authorize it.
27. `internal/monitoring/monitor.go` shared with `monitoring`: monitor construction owns both monitoring runtime initialization and fail-closed agent lifecycle journal hydration before report admission.
28. `internal/monitoring/monitor_agents.go` shared with `monitoring`: server-side Unified Agent report, removal, token binding, tombstone expiry, and re-enrollment semantics are jointly owned by agent lifecycle authority and monitoring ingest.
29. `pkg/agents/host/report.go` shared with `monitoring`: the Unified Agent host report is both an agent lifecycle authored-state contract and a monitoring ingest contract for host maintenance posture.
@@ -4638,7 +4638,22 @@ summaries became the alarm name still carry "<entity> <moref> has VMware alarm
<name> (<colour>)", so the title strips that wording to the same alarm name.
Patrol's seed context lists active and recently resolved alerts by resource:
each line names the alert's resource unless the message already does, because
provider incidents such as vCenter alarms say only what was flagged. The projection also collapses open/resolved churn:
provider incidents such as vCenter alarms say only what was flagged.
A recently resolved alert that closed without recovering, such as a node metric
handed to its Pulse agent, must not read as a recovery to Patrol, the
assistant or the alert tool: Patrol's seed writes `closed <ago>. <summary>` in place of `resolved <ago>`, the
assistant's alert context writes `closed <ago> ago) <summary>`, the
`pulse_alerts` `resolved` action returns the summary as `resolution` with
`successor_resource_id`, and incident memory titles the close with the summary
(`alertResolvedEventSummary`, or the `alert_resolution` resource-change
metadata when it projects from the canonical timeline).
`TestSeedHealthAndAlerts_HandoverIsNotARecovery`,
`TestService_buildAlertContext_HandoverIsNotARecovery`,
`TestExecuteListResolvedAlertsReportsHandover` and
`TestIncidentStore_HandoverCloseIsNotARecovery` pin it. Performance reports
(`pkg/reporting`, the report narrator) do not carry the resolution yet: a
report generated within the five-minute recently-resolved window after a
handover still lists that alert as resolved. The projection also collapses open/resolved churn:
when a record's timeline holds at least four open-to-resolved or
resolved-to-open transitions inside the last 24 hours, the item carries a
`flapping` summary (transition count, window, first and latest transition) and
@@ -793,12 +793,26 @@ its message) and the Alert JSON then carry it. Resolved consumers still receive
the close, because integrations close their incident by alert ID. The
synthesized closing evidence carries the resolution reason and summary instead
of `legacy_alert_recovery_projection`, and the resolve transition carries the
summary as its `reason`. The cause stays `recovery_evidence`, the only cause
operational trust lets enter `resolved`. A nil resolution leaves the
disabled-threshold path and every recovery unchanged.
summary as its `reason`. Its cause is `ownership_transferred`, an
`operationaltrust.TransitionCause` that may only enter `resolved` and only with
evidence (`resolvedTransitionCause` in `operational_contract.go`), so Patrol's
timeline never labels a handover "Recovery evidence" and alert-quality
telemetry does not count a handover during a snooze as resolved while snoozed.
A nil resolution leaves the disabled-threshold path and every recovery on
`recovery_evidence`.
`TestReleasedMetricAlertCarriesResolutionInsteadOfRecovery` and
`TestResolvedHistoryRowKeepsResolutionWithoutEventLog` in
`internal/alerts/operational_contract_test.go` pin this.
The in-app readers carry it too. `GetRecentlyResolved` projects it into
`models.Alert.Resolution` (reason, successor and the `Summary()` text) for the
state snapshot, the websocket and the assistant and Patrol prompts, pinned by
`TestRecentlyResolvedCarriesHandoverResolution` in `history_test.go`. The
alerts history lists such a row as `moved to agent` instead of `resolved`, even
when it was acknowledged, with the account in the badge title and on the phone
card (`buildAlertHistoryItems`, `getAlertResolutionDetail`). The resolved
lifecycle change written to the resource timeline carries the summary as its
reason and the code as `alert_resolution` metadata, so the row's incident
timeline says the alert moved instead of "Alert resolved: <breach message>".
## Extension Points
@@ -2728,7 +2742,8 @@ legacy alert timestamp.
Suppression is bounded and reasoned, leaves the default active queue, and
remains inspectable. Expiry or explicit unsuppression returns the record to its
detector-owned state; it never resolves it. Only fresh sufficient recovery
evidence may enter resolving, and only detector recovery may resolve.
evidence may enter resolving, and only detector recovery, or a handover
recorded as `ownership_transferred` with its closing evidence, may resolve.
Per-alert snooze is the customer-facing bounded suppression contract, not a
page-local timer. `SnoozeAlert` writes the same canonical operational record,
@@ -661,6 +661,13 @@ transitions. Older payloads without these optional fields remain readable;
frontend consumers use `frontend-modern/src/types/operationalTrust.ts` as the
single TypeScript projection rather than recreating lifecycle or evidence
enums locally.
`TransitionCause` includes `ownership_transferred` for a close whose condition
another record now owns, such as a node metric handed to its Pulse agent; it
may only enter `resolved`. Alert payloads (history, state snapshot and
websocket) may also carry an optional `resolution` (`reason`,
`successorResourceId`, `successorName`, and on state payloads the alert
engine's `summary`), typed once as `AlertResolution` in
`frontend-modern/src/types/api.ts`. Its absence means an ordinary recovery.
## Shared Boundaries
@@ -7088,6 +7088,13 @@ should extend the model, and section rendering should extend those owners
rather than rebuilding any of those concerns in the tab shell.
The resource-resolution handoff into the resource-incident panel now belongs to
the history state rather than the tab shell.
The history model also owns the close outcome: an alert that closed with a
typed `resolution` is status `moved` (badge `moved to agent`, its account in
`HistoryItem.closeDetail`) rather than `resolved` or `acknowledged`. The table
row carries the account in the badge title and the phone card renders it under
the message, while `getAlertHistoryStatusPresentation` and
`getAlertResolutionDetail` in `utils/alertIncidentPresentation.ts` own the
wording.
`frontend-modern/src/features/alerts/useAlertHistoryState.ts` re-exposes the
`getResource` resolver it is already given, and
`frontend-modern/src/features/alerts/AlertResourceIncidentsPanel.tsx` reads it
@@ -3587,6 +3587,15 @@ truth. Monitoring APIs that still serve `StateSnapshot` must project
instead of trusting the cached snapshot fields, so externally served alert
counts and recently resolved incidents do not lag behind acknowledgement,
resolve, or clear operations between explicit sync points.
A recently resolved alert that closed without recovering keeps that on the
projection: `models.Alert.Resolution` (reason, successor resource ID and name,
and the alert engine's summary) is set only for such a close, `cloneAlert`
copies it into every snapshot, and `recordAlertTimelineChange` passes it into
the `alert_resolved` resource change so the incident timeline says the alert
moved. `TestAlertResolutionWireContract` in
`internal/models/metrics_types_test.go` and
`TestMonitor_HandleAlertLifecycle_HandoverCloseIsNotARecovery` in
`internal/monitoring/monitor_alert_handling_test.go` pin it.
The container entrypoint in `docker-entrypoint.sh` now also lives under this
boundary. Hosted or managed tenant bootstrap changes must preserve safe startup
when immutable read-only mounts are layered into `/etc/pulse`; the entrypoint
@@ -1237,6 +1237,12 @@ payload's `alerts` array. Built-in template titles still say Resolved, which is
accurate for the closed alert, and a custom template that hard-codes recovery
text keeps it. Issue #1682 is the precedent: a recipient called "is now
healthy" a false factual claim when an alert closed without recovering.
The resolve transition such a close links to has cause `ownership_transferred`
in `internal/operationaltrust/contracts.go`, not `recovery_evidence`. A
notification link records the transition ID and lifecycle state, never the
cause, so delivery, receipts and recovery admission treat it exactly like a
recovery's resolve transition. `TestOwnershipTransferOnlyClosesWithEvidence`
pins that the cause may only enter `resolved` and only with evidence.
Verification: `resolved_handover_test.go` drives the firing, its receipt and
the queued close through the normal entry points for ntfy, and renders Discord,
@@ -1380,6 +1380,14 @@ to traverse the list again. At widths where the detail replaces the queue's
reading position, its visible close affordance is an explicit `Back to list`
control; wide layouts retain the compact close icon. Both controls clear the
deep link and restore focus to the originating queue row.
A resolved item whose latest close is an `ownership_transferred` transition did
not recover: its condition moved to another record, such as the linked Pulse
agent's own alert. The detail's state badge reads `Moved` in the info tone
instead of the green `Resolved`, and the transition's reason (where the
condition went, and that it is not a recovery) sits directly under the summary
rather than only inside Evidence and history (`getAttentionHandoverClose` in
`patrolHomePresentation.ts`). The timeline names that cause `Ownership
transferred`, never `Recovery evidence`.
The active Patrol queue now uses compact severity-accented rows for
prioritisation and a single focused review panel for the selected issue. The
@@ -4171,6 +4171,14 @@ such as `alert_fired`, `alert_acknowledged`, `alert_unacknowledged`,
AI-local annotations. Snooze and resume projections must preserve the actor
and exact suppression expiry when present; they pause delivery and escalation
without acknowledging, resolving, or replacing the underlying incident.
An `alert_resolved` change for a close that was not a recovery carries the
alert engine's summary as its reason and the reason code as `alert_resolution`
metadata (`AlertTimelineChange.ResolutionReason` and `ResolutionSummary`), so
incident projection titles the close with where the alert went instead of
"Alert resolved: <breach message>". The resource drawer still labels the change
by kind (`Alert resolved`) and prefixes its headline with that label; only the
reason after it says the alert moved. Ordinary recoveries omit the key, and
every other lifecycle kind ignores the fields.
Alert-scoped
incident memory may still project those events for one investigation thread,
but the durable source of truth for resource-affecting alert lifecycle and
@@ -0,0 +1,54 @@
{
"version": 1,
"base_sha": "982a6f6148cf9d39883e4443191b4d40b59cfd7c",
"verified_at": "2026-10-06T15:03:00Z",
"result": "passed",
"changed_paths": [
"frontend-modern/src/features/alerts/AlertHistoryMobileList.tsx",
"frontend-modern/src/features/alerts/AlertHistoryTableAlertRow.tsx",
"frontend-modern/src/features/alerts/alertHistoryModel.ts",
"frontend-modern/src/features/patrol/PatrolAttentionWorkbench.tsx",
"frontend-modern/src/features/patrol/patrolHomePresentation.ts",
"frontend-modern/src/types/api.ts",
"frontend-modern/src/types/operationalTrust.ts",
"frontend-modern/src/utils/alertIncidentPresentation.ts"
],
"content_sha256": {
"frontend-modern/src/features/alerts/AlertHistoryMobileList.tsx": "94bb73163dd86c1f045a0b5559b42486198ab824ef2abccfc3c8c1dcc0ff2784",
"frontend-modern/src/features/alerts/AlertHistoryTableAlertRow.tsx": "aacc7232596a47d64c7e6ec3b29dfaf51be6e1691d9be906b30bfce49f4daf6f",
"frontend-modern/src/features/alerts/alertHistoryModel.ts": "30548565d29ca16fe1c1192687c881defc064ba1b267fdcd40fc28922a51fed4",
"frontend-modern/src/features/patrol/PatrolAttentionWorkbench.tsx": "b3c87b3ead9bde22a091b2c080b9eaca18face9ba153085bb1a1dbb9b6ae9f54",
"frontend-modern/src/features/patrol/patrolHomePresentation.ts": "04338cdce5b54bc041267b325a6985a9a8f84ef9e5a8dba8040c7d9fc5ab1811",
"frontend-modern/src/types/api.ts": "17a948b6ba78780ef2b95c1f397d2d9a04eebd2726c568a8316e25e7c8f9e6b3",
"frontend-modern/src/types/operationalTrust.ts": "9a50a869b1f6ef2885f30a354fb70b36b64fbb1a3f00ba4476d592389f9ba954",
"frontend-modern/src/utils/alertIncidentPresentation.ts": "f77c42b48f22bddcccb43a3da43b3771bb03d25c6d23b7efa200a34dae1da0e9"
},
"routes": [
"/alerts/history",
"/patrol?attention=pve1::metric-threshold:memory"
],
"viewports": [
{
"width": 1280,
"height": 900
},
{
"width": 375,
"height": 812
}
],
"states": [
"Private non-mock stack (backend :7691, Vite :5291) on a data dir whose alert history was written by the real alerts manager: a pve1 node memory alert closed through releaseCanonicalMetricAlert with a moved_to_agent resolution (successor pve1 (Host Agent)) and a pve1 CPU alert closed as an ordinary release; the handover record's resolve transition carries cause ownership_transferred",
"Alerts history at 1280: the CPU row's status badge reads resolved, the memory row's reads moved to agent in blue, and its title is 'Moved to pve1 (Host Agent). This is not a recovery: check the agent for the current reading.'",
"Memory row Timeline expanded: the incident timeline lists 'Node memory at 95.0%' then 'Alert moved to pve1 (Host Agent). This is not a recovery: check the agent for the current reading.' where it said 'Alert resolved' before; the incident header still says resolved (incident status vocabulary, unchanged)",
"Alerts history at 375: the memory card shows the moved to agent badge and the account as a muted line under the breach message; the CPU card is unchanged",
"Patrol deep link to the moved record at 1280 and 375: the detail badge reads Moved (info tone) instead of green Resolved, the handover sentence sits directly under 'Node memory at 95.0%', and Evidence and history shows 'Open to Resolved, Ownership transferred' with the same reason; no Recovery evidence label anywhere; no horizontal scroll at 375 and no console errors"
],
"interactions": [
"Loaded /alerts/history at 1280x900 and read both rows' status badges and titles",
"Clicked the memory row's Timeline button and read the expanded incident timeline",
"Reloaded /alerts/history at 375x812 and read both history cards",
"Opened /patrol?attention=pve1::metric-threshold:memory at 1280x900 and 375x812, opened Evidence and history, and read the badge, summary, handover line and timeline",
"Checked document width against the viewport at 375 and read the console for errors"
]
}
@@ -229,6 +229,9 @@ export function AlertHistoryMobileList(props: AlertHistoryMobileListProps) {
</div>
<p class="mt-2 text-xs leading-relaxed text-base-content">{alert.description}</p>
<Show when={alert.closeDetail}>
{(detail) => <p class="mt-1 text-xs leading-relaxed text-muted">{detail()}</p>}
</Show>
<dl class="mt-3 grid grid-cols-3 gap-2 text-[10px] text-muted">
<div>
@@ -74,7 +74,7 @@ export function AlertHistoryTableAlertRow(props: AlertHistoryTableAlertRowProps)
</TableCell>
<TableCell class={getPlatformTableCellClassForKind('badge')}>
<span class={historyStatusPresentation().className}>
<span class={historyStatusPresentation().className} title={props.alert.closeDetail}>
{historyStatusPresentation().label}
</span>
</TableCell>
@@ -1,5 +1,9 @@
import type { Alert } from '@/types/api';
import type { Resource } from '@/types/resource';
import {
ALERT_HISTORY_MOVED_STATUS,
getAlertResolutionDetail,
} from '@/utils/alertIncidentPresentation';
import { isPulseSystemAlert } from '@/utils/alertScope';
import { alertTypeDisplayLabel, unifiedTypeToAlertDisplayType } from './helpers';
@@ -25,6 +29,8 @@ export interface HistoryItem {
title: string;
rawAlertType?: string;
description?: string;
// Set when the alert closed without recovering: what happened instead.
closeDetail?: string;
acknowledged?: boolean;
systemAlert?: boolean;
}
@@ -206,11 +212,17 @@ export function buildAlertHistoryItems({
alertHistory.forEach((alert) => {
if (activeAlertIds.has(alert.id)) return;
const systemScoped = isPulseSystemAlert(alert);
// A close that was not a recovery must not read as "resolved".
const closeDetail = getAlertResolutionDetail(alert.resolution);
items.push({
id: alert.id,
source: 'alert',
status: alert.acknowledged ? 'acknowledged' : 'resolved',
status: closeDetail
? ALERT_HISTORY_MOVED_STATUS
: alert.acknowledged
? 'acknowledged'
: 'resolved',
startTime: alert.startTime,
endTime: alert.lastSeen,
duration: formatAlertHistoryDuration(alert.startTime, alert.lastSeen, now),
@@ -231,6 +243,7 @@ export function buildAlertHistoryItems({
title: alertTypeDisplayLabel(alert.type),
rawAlertType: alert.type,
description: alert.message,
...(closeDetail ? { closeDetail } : {}),
acknowledged: alert.acknowledged,
});
});
@@ -68,6 +68,7 @@ import { copyToClipboard } from '@/utils/clipboard';
import type { PatrolAutonomyLevel } from '@/api/patrol';
import {
getAttentionFlappingPresentation,
getAttentionHandoverClose,
getLinkedPatrolFindings,
getPatrolRememberedDecisionPresentation,
partitionPatrolAttention,
@@ -971,7 +972,10 @@ function AttentionDetail(props: {
<div class="space-y-5 px-4 py-4 sm:px-5">
<section>
<div class="flex flex-wrap items-center gap-x-2 gap-y-1 text-xs text-muted">
<StateBadge item={loaded().item} />
<StateBadge
item={loaded().item}
movedClose={Boolean(getAttentionHandoverClose(loaded()))}
/>
<span aria-hidden="true">·</span>
<EvidenceLabel item={loaded().item} detail />
<span aria-hidden="true">·</span>
@@ -984,6 +988,9 @@ function AttentionDetail(props: {
<p class="mt-3 text-sm leading-6 text-base-content">
{loaded().item.plainLanguageSummary}
</p>
<Show when={getAttentionHandoverClose(loaded())}>
{(handover) => <p class="mt-1 text-xs leading-5 text-muted">{handover()}</p>}
</Show>
<Show when={getAttentionFlappingPresentation(loaded().item.flapping)}>
{(flapping) => (
<p class="mt-2 flex items-start gap-2 rounded-md border border-amber-200 bg-amber-50/70 px-3 py-2 text-xs leading-5 text-amber-900 dark:border-amber-900/25 dark:bg-amber-950/30 dark:text-amber-200">
@@ -1793,10 +1800,17 @@ function severityTone(item: AttentionItem): MetadataBadgeTone {
}
}
function StateBadge(props: { item: AttentionItem }) {
// movedClose marks a resolved item whose condition moved to another owner: it
// did not recover, so it must not wear the green "Resolved" badge.
function StateBadge(props: { item: AttentionItem; movedClose?: boolean }) {
return (
<MetadataBadge tone={stateTone(props.item)} size="xs" shape="rounded" appearance="outline">
{formatLabel(props.item.state)}
<MetadataBadge
tone={props.movedClose ? 'info' : stateTone(props.item)}
size="xs"
shape="rounded"
appearance="outline"
>
{props.movedClose ? 'Moved' : formatLabel(props.item.state)}
</MetadataBadge>
);
}
@@ -988,6 +988,56 @@ describe('PatrolAttentionWorkbench', () => {
expect(screen.getByText('Show all 11 transitions')).toBeInTheDocument();
});
it('shows an alert handed to its Pulse agent as moved, not recovered', async () => {
const reason =
'Alert moved to pve1 (Host Agent). This is not a recovery: check the agent for the current reading.';
const moved = item({
id: 'record-moved',
operationalRecordId: 'record-moved',
subjectResourceId: 'pve1',
subjectResourceName: 'pve1',
subjectResourceType: 'node',
kind: 'memory',
title: 'Memory on pve1',
plainLanguageSummary: 'Memory usage at 95%',
severity: 'warning',
state: 'resolved',
protectionPosture: undefined,
});
const movedDetail = detail(moved);
movedDetail.timeline = [
...movedDetail.timeline,
{
id: 'transition-moved',
operationalRecordId: 'record-moved',
from: 'open',
to: 'resolved',
at: '2026-07-19T07:45:00Z',
cause: 'ownership_transferred',
causeKey: 'memory:pve1',
evidenceIds: ['evidence-handover'],
reason,
},
];
window.history.replaceState({}, '', '/patrol?attention=record-moved');
apiMocks.getList.mockResolvedValue(listResponse([], summary()));
apiMocks.getDetail.mockResolvedValue(movedDetail);
renderWorkbench();
const panel = await screen.findByRole('complementary', { name: 'pve1 · Memory' });
expect(await within(panel).findByText('Moved')).toBeInTheDocument();
expect(within(panel).queryByText('Resolved')).not.toBeInTheDocument();
// The account sits under the summary, outside the collapsed history.
const summaryLine = within(panel).getByText('Memory usage at 95%');
expect(summaryLine.nextElementSibling).toHaveTextContent(reason);
expect(summaryLine.nextElementSibling?.closest('details')).toBeNull();
fireEvent.click(within(panel).getByText('Evidence and history'));
expect(within(panel).getByText('Open to Resolved')).toBeInTheDocument();
expect(within(panel).getByText(/Ownership transferred/)).toBeInTheDocument();
expect(within(panel).queryByText(/Recovery evidence/)).not.toBeInTheDocument();
});
it('opens the canonical governed action review from an eligible attention item', async () => {
const actionOffer = {
targetResourceId: 'docker:host-1/container-1',
@@ -1,7 +1,8 @@
import { describe, expect, it } from 'vitest';
import type { PatrolObjective } from '@/api/patrol';
import type { AttentionItem } from '@/api/patrolAttention';
import type { AttentionItem, AttentionItemDetail } from '@/api/patrolAttention';
import {
getAttentionHandoverClose,
getPatrolAttentionDecisionReason,
getPatrolObjectiveProtectionSummary,
partitionPatrolAttention,
@@ -145,3 +146,62 @@ describe('patrol home presentation', () => {
});
});
});
describe('getAttentionHandoverClose', () => {
const transition = (
overrides: Partial<AttentionItemDetail['timeline'][number]>,
): AttentionItemDetail['timeline'][number] => ({
id: 'transition-1',
operationalRecordId: 'record-1',
from: 'open',
to: 'resolved',
at: '2026-08-14T08:00:00Z',
cause: 'recovery_evidence',
causeKey: 'memory:pve1',
evidenceIds: ['evidence-1'],
...overrides,
});
const reason =
'Alert moved to pve1 (Host Agent). This is not a recovery: check the agent for the current reading.';
it('returns the handover account only for a record closed by an ownership transfer', () => {
const resolved = attentionItem({ state: 'resolved' });
expect(
getAttentionHandoverClose({
item: resolved,
timeline: [transition({ cause: 'ownership_transferred', reason })],
}),
).toBe(reason);
expect(
getAttentionHandoverClose({
item: resolved,
timeline: [transition({ cause: 'ownership_transferred' })],
}),
).toBe('This alert moved to another owner when it closed. It did not recover.');
expect(getAttentionHandoverClose({ item: resolved, timeline: [transition({})] })).toBeNull();
});
it('follows the latest close, so a later recovery or reopen wins', () => {
const handover = transition({ cause: 'ownership_transferred', reason });
const reopened = transition({
id: 'transition-2',
from: 'resolved',
to: 'open',
at: '2026-08-14T09:00:00Z',
cause: 'detector_decision',
});
const recovered = transition({ id: 'transition-3', at: '2026-08-14T10:00:00Z' });
expect(
getAttentionHandoverClose({
item: attentionItem({ state: 'resolved' }),
timeline: [handover, reopened, recovered],
}),
).toBeNull();
expect(
getAttentionHandoverClose({
item: attentionItem({ state: 'open' }),
timeline: [handover, reopened],
}),
).toBeNull();
});
});
@@ -1,5 +1,5 @@
import type { PatrolAutonomyLevel, PatrolObjective } from '@/api/patrol';
import type { AttentionFlapping, AttentionItem } from '@/api/patrolAttention';
import type { AttentionFlapping, AttentionItem, AttentionItemDetail } from '@/api/patrolAttention';
import type { UnifiedFinding } from '@/stores/aiIntelligence';
export interface PatrolAutonomyExperience {
@@ -207,6 +207,26 @@ export function getAttentionFlappingPresentation(
return getFlappingPresentation(flapping.transitionCount, flapping.windowHours);
}
/**
* A resolved item whose close handed the condition to another record, such as
* a node metric its Pulse agent took over, did not recover. Returns the
* account to show instead of a healthy "Resolved", or null for a recovery.
*/
export function getAttentionHandoverClose(
detail: Pick<AttentionItemDetail, 'item' | 'timeline'> | null | undefined,
): string | null {
if (!detail || detail.item.state !== 'resolved') return null;
let close: AttentionItemDetail['timeline'][number] | undefined;
for (const transition of detail.timeline ?? []) {
if (transition.to !== 'resolved') continue;
if (!close || Date.parse(transition.at) >= Date.parse(close.at)) close = transition;
}
if (close?.cause !== 'ownership_transferred') return null;
return (
close.reason?.trim() || 'This alert moved to another owner when it closed. It did not recover.'
);
}
const normalizeResourceKey = (value: string | undefined): string =>
(value ?? '').trim().toLowerCase();
@@ -100,6 +100,7 @@ import {
buildProjectedOverrides,
normalizeRawOverridesConfig,
} from '@/features/alerts/alertOverridesModel';
import { buildAlertHistoryItems } from '@/features/alerts/alertHistoryModel';
import {
getAlertIncidentAcknowledgedBadgeClass,
getAlertIncidentEventFilterActionButtonClass,
@@ -1320,3 +1321,59 @@ describe('Unified selector parity', () => {
);
});
});
describe('alert history close outcome', () => {
it('lists a node alert handed to its Pulse agent as moved, not resolved', () => {
const base = {
type: 'memory',
level: 'warning' as const,
resourceName: 'pve1',
node: 'pve1',
instance: 'pve1',
value: 95,
threshold: 85,
startTime: '2026-10-06T09:00:00Z',
lastSeen: '2026-10-06T11:00:00Z',
acknowledged: true,
};
const items = buildAlertHistoryItems({
activeAlerts: {},
alertHistory: [
{
...base,
id: 'pve1-memory',
resourceId: 'pve1',
message: 'Memory usage at 95%',
resolution: {
reason: 'moved_to_agent',
successorResourceId: 'agent-pve1',
successorName: 'pve1 (Host Agent)',
},
},
{
...base,
id: 'pve1-cpu',
resourceId: 'pve1',
message: 'CPU usage at 90%',
acknowledged: false,
},
],
getResource: () => undefined,
allResources: [],
now: Date.parse('2026-10-06T12:00:00Z'),
});
const moved = items.find((item) => item.id === 'pve1-memory');
// Acknowledgement does not hide the outcome: the condition went to the agent.
expect(moved?.status).toBe('moved');
expect(moved?.description).toBe('Memory usage at 95%');
expect(moved?.closeDetail).toBe(
'Moved to pve1 (Host Agent). This is not a recovery: check the agent for the current reading.',
);
const recovered = items.find((item) => item.id === 'pve1-cpu');
expect(recovered?.status).toBe('resolved');
expect(recovered).not.toHaveProperty('closeDetail');
expect(alertHistoryTableAlertRowSource).toContain('title={props.alert.closeDetail}');
});
});
+12
View File
@@ -1380,6 +1380,18 @@ export interface Alert {
latestTransition?: LifecycleTransition;
transitions?: LifecycleTransition[];
evidence?: EvidenceEnvelope[];
// Set only on a closed alert whose condition did not clear, such as a node
// metric handed to its Pulse agent. Absent for an ordinary recovery.
resolution?: AlertResolution;
}
// Why an alert closed without recovering (alerts.AlertResolution).
export interface AlertResolution {
reason: 'moved_to_agent' | (string & {});
successorResourceId?: string;
successorName?: string;
// The alert engine's one-line account; carried on state snapshots only.
summary?: string;
}
export interface ResolvedAlert extends Alert {
@@ -97,6 +97,7 @@ export type TransitionCause =
| 'suppression'
| 'suppression_expired'
| 'recovery_evidence'
| 'ownership_transferred'
| 'collection_stale'
| 'collection_unknown';
@@ -10,8 +10,10 @@ import {
ALERT_RESOURCE_INCIDENT_REFRESH_LABEL,
ALERT_RESOURCE_INCIDENT_SAVE_NOTE_LABEL,
ALERT_RESOURCE_INCIDENT_SAVING_NOTE_LABEL,
ALERT_HISTORY_MOVED_STATUS,
getAlertHistoryStatusPresentation,
getAlertIncidentLevelBadgeClass,
getAlertResolutionDetail,
getAlertIncidentTimelineDialogTitle,
getAlertHistoryInvestigationDialogTitle,
getAlertResourceIncidentAcknowledgedByLabel,
@@ -103,6 +105,29 @@ describe('alertIncidentPresentation', () => {
className: 'text-xs px-2 py-0.5 rounded-sm bg-surface-hover text-base-content',
rowClassName: '',
});
expect(getAlertHistoryStatusPresentation(ALERT_HISTORY_MOVED_STATUS)).toEqual({
label: 'moved to agent',
className:
'text-xs px-2 py-0.5 rounded-sm bg-blue-100 dark:bg-blue-900/25 text-blue-700 dark:text-blue-300',
rowClassName: '',
});
});
it('says a handed-over alert did not recover, and leaves recoveries alone', () => {
expect(getAlertResolutionDetail(undefined)).toBeUndefined();
expect(
getAlertResolutionDetail({ reason: 'moved_to_agent', successorName: 'pve1 (Host Agent)' }),
).toBe(
'Moved to pve1 (Host Agent). This is not a recovery: check the agent for the current reading.',
);
expect(getAlertResolutionDetail({ reason: 'moved_to_agent' })).toBe(
'Moved to its Pulse agent. This is not a recovery: check the agent for the current reading.',
);
// A future reason the UI has no copy for still shows the engine's account.
expect(
getAlertResolutionDetail({ reason: 'replaced', summary: 'Alert replaced by a new check.' }),
).toBe('Alert replaced by a new check.');
});
it('returns canonical resource incident loading and empty-state copy', () => {
@@ -1,3 +1,4 @@
import type { AlertResolution } from '@/types/api';
import { getAlertSeverityBadgeClass } from '@/utils/alertSeverityPresentation';
export type AlertIncidentLevel = 'warning' | 'critical' | string | null | undefined;
@@ -76,6 +77,24 @@ export function getAlertIncidentLevelBadgeClass(level: AlertIncidentLevel): stri
return getAlertSeverityBadgeClass(String(level ?? 'warning'));
}
// History status for an alert that closed because its Pulse agent took the
// metric over. The condition did not clear; the agent's own alert owns it.
export const ALERT_HISTORY_MOVED_STATUS = 'moved';
/**
* The account of a close that was not a recovery, for history rows and their
* tooltips. Returns undefined for an ordinary recovery so callers keep their
* "resolved" wording.
*/
export function getAlertResolutionDetail(resolution?: AlertResolution | null): string | undefined {
if (!resolution) return undefined;
if (resolution.reason === 'moved_to_agent') {
const successor = resolution.successorName?.trim() || 'its Pulse agent';
return `Moved to ${successor}. This is not a recovery: check the agent for the current reading.`;
}
return resolution.summary?.trim() || undefined;
}
export function getAlertHistoryStatusPresentation(
status?: string | null,
): AlertHistoryStatusPresentation {
@@ -99,6 +118,17 @@ export function getAlertHistoryStatusPresentation(
};
}
// Closed without recovering: neither a live problem on this row nor a
// healthy outcome, so it gets its own neutral-blue badge.
if (normalized === ALERT_HISTORY_MOVED_STATUS) {
return {
label: 'moved to agent',
className:
'text-xs px-2 py-0.5 rounded-sm bg-blue-100 dark:bg-blue-900/25 text-blue-700 dark:text-blue-300',
rowClassName: '',
};
}
return {
label: normalized || 'resolved',
className: 'text-xs px-2 py-0.5 rounded-sm bg-surface-hover text-base-content',
+12
View File
@@ -2,6 +2,7 @@ package ai
import (
"fmt"
"strings"
"time"
"github.com/rcourtman/pulse-go-rewrite/internal/alerts"
@@ -56,6 +57,7 @@ func (a *AlertManagerAdapter) GetRecentlyResolved(minutes int) []ResolvedAlertIn
AlertInfo: convertAlertFromModels(&resolved.Alert),
ResolvedTime: resolved.ResolvedTime,
Duration: formatDuration(resolved.ResolvedTime.Sub(resolved.StartTime)),
Resolution: resolvedAlertResolution(resolved.Resolution),
}
result = append(result, info)
}
@@ -98,6 +100,7 @@ func (a *AlertManagerAdapter) GetAlertHistory(resourceID string, limit int) []Re
AlertInfo: convertAlertFromModels(&resolved.Alert),
ResolvedTime: resolved.ResolvedTime,
Duration: formatDuration(resolved.ResolvedTime.Sub(resolved.StartTime)),
Resolution: resolvedAlertResolution(resolved.Resolution),
}
result = append(result, info)
if len(result) >= limit {
@@ -245,3 +248,12 @@ func formatDuration(d time.Duration) string {
}
return fmt.Sprintf("%d days", days)
}
// resolvedAlertResolution is the one-line account of a close that was not a
// recovery, empty for an ordinary resolve.
func resolvedAlertResolution(resolution *models.AlertResolution) string {
if resolution == nil {
return ""
}
return strings.TrimSpace(resolution.Summary)
}
+9
View File
@@ -31,6 +31,9 @@ type ResolvedAlertInfo struct {
AlertInfo
ResolvedTime time.Time `json:"resolved_time"`
Duration string `json:"total_duration"` // How long the alert lasted
// Resolution is set only when the alert closed without recovering, such
// as a node metric handed to its Pulse agent.
Resolution string `json:"resolution,omitempty"`
}
// AlertProvider provides access to the current alert state
@@ -131,6 +134,12 @@ func (s *Service) buildAlertContext() string {
}
for i := 0; i < limit; i++ {
a := recentlyResolved[i]
if resolution := strings.TrimSpace(a.Resolution); resolution != "" {
sections = append(sections, fmt.Sprintf("- **%s** on %s: %s (lasted %s, closed %s ago) %s",
a.Type, a.ResourceName, a.Message, a.Duration,
formatTimeAgo(a.ResolvedTime), resolution))
continue
}
sections = append(sections, fmt.Sprintf("- **%s** on %s: %s (lasted %s, resolved %s ago)",
a.Type, a.ResourceName, a.Message, a.Duration,
formatTimeAgo(a.ResolvedTime)))
+57
View File
@@ -5,6 +5,8 @@ import (
"strings"
"testing"
"time"
"github.com/rcourtman/pulse-go-rewrite/internal/models"
)
type stubAlertProvider struct {
@@ -151,6 +153,61 @@ func TestService_buildAlertContext_NoActiveWithResolved(t *testing.T) {
}
}
// The assistant reads recently resolved alerts through the adapter, so a node
// alert handed to its Pulse agent must reach the prompt as a handover and not
// as "resolved", which the model would read as the node being healthy.
func TestService_buildAlertContext_HandoverIsNotARecovery(t *testing.T) {
now := time.Now()
summary := "Alert moved to pve1 (Host Agent). This is not a recovery: check the agent for the current reading."
adapter := NewAlertManagerAdapter(&stubAlertManager{resolved: []models.ResolvedAlert{
{
Alert: models.Alert{
ID: "pve1-memory",
Type: "memory",
Level: "warning",
ResourceID: "pve1",
ResourceName: "pve1",
Message: "Memory usage at 95%",
StartTime: now.Add(-time.Hour),
Resolution: &models.AlertResolution{Reason: "moved_to_agent", Summary: summary},
},
ResolvedTime: now.Add(-time.Minute),
},
{
Alert: models.Alert{
ID: "pve2-cpu",
Type: "cpu",
Level: "warning",
ResourceID: "pve2",
ResourceName: "pve2",
Message: "CPU usage at 90%",
StartTime: now.Add(-time.Hour),
},
ResolvedTime: now.Add(-time.Minute),
},
}})
resolved := adapter.GetRecentlyResolved(30)
if len(resolved) != 2 || resolved[0].Resolution != summary || resolved[1].Resolution != "" {
t.Fatalf("GetRecentlyResolved = %+v, want only the handover to carry a resolution", resolved)
}
if history := adapter.GetAlertHistory("pve1", 5); len(history) != 1 || history[0].Resolution != summary {
t.Fatalf("GetAlertHistory = %+v, want the handover resolution", history)
}
s := &Service{}
s.SetAlertProvider(adapter)
ctx := s.buildAlertContext()
if !strings.Contains(ctx, "- **memory** on pve1: Memory usage at 95% (lasted ") ||
!strings.Contains(ctx, ", closed 1 minute ago) "+summary) {
t.Fatalf("expected the handover line, got: %s", ctx)
}
if !strings.Contains(ctx, "- **cpu** on pve2: CPU usage at 90% (lasted ") ||
!strings.Contains(ctx, ", resolved 1 minute ago)") {
t.Fatalf("expected the ordinary recovery line, got: %s", ctx)
}
}
func TestService_buildTargetAlertContext(t *testing.T) {
s := &Service{}
s.SetAlertProvider(&stubAlertProvider{
+20 -2
View File
@@ -474,6 +474,17 @@ func (s *IncidentStore) RecordAlertUnacknowledged(alert *alerts.Alert, user stri
s.saveAsync()
}
// alertResolvedEventSummary is the timeline line for an alert's close: the
// handover account when it closed without recovering, else "Alert resolved".
func alertResolvedEventSummary(alert *alerts.Alert) string {
if alert != nil {
if summary := alert.Resolution.Summary(); summary != "" {
return summary
}
}
return "Alert resolved"
}
// RecordAlertResolved records a resolved event and closes the incident.
func (s *IncidentStore) RecordAlertResolved(alert *alerts.Alert, resolvedAt time.Time) {
if alert == nil {
@@ -507,7 +518,7 @@ func (s *IncidentStore) RecordAlertResolved(alert *alerts.Alert, resolvedAt time
shell.OccurrenceClosedAt = cloneTime(resolvedAt)
if !s.projectsFromCanonicalLocked() {
s.addEventAtLocked(shell, IncidentEventAlertResolved, resolvedAt, "Alert resolved", map[string]interface{}{
s.addEventAtLocked(shell, IncidentEventAlertResolved, resolvedAt, alertResolvedEventSummary(alert), map[string]interface{}{
"resolved_at": resolvedAt.Format(time.RFC3339),
})
}
@@ -563,7 +574,7 @@ func (s *IncidentStore) EnsureAlertOccurrence(alert *alerts.Alert, resolvedAt *t
if resolvedAt != nil && !resolvedAt.IsZero() && (shell.OccurrenceRefiredAt == nil || !resolvedAt.Before(*shell.OccurrenceRefiredAt)) {
shell.OccurrenceClosedAt = cloneTime(*resolvedAt)
if !hasIncidentEventType(shell.Events, IncidentEventAlertResolved) {
s.addEventAtLocked(shell, IncidentEventAlertResolved, *resolvedAt, "Alert resolved", map[string]interface{}{
s.addEventAtLocked(shell, IncidentEventAlertResolved, *resolvedAt, alertResolvedEventSummary(alert), map[string]interface{}{
"resolved_at": resolvedAt.Format(time.RFC3339),
"projection_source": incidentSnapshotSource,
})
@@ -1113,6 +1124,13 @@ func incidentEventSummaryFromChange(change unifiedresources.ResourceChange, even
case IncidentEventAlertUnsnoozed:
return "Alert notifications resumed"
case IncidentEventAlertResolved:
// A handover closes the alert without recovering; its change reason
// says so and where the condition went.
if _, ok := stringMetadata(change.Metadata, unifiedresources.MetadataAlertResolution); ok {
if reason := strings.TrimSpace(change.Reason); reason != "" {
return reason
}
}
return "Alert resolved"
default:
if summary := strings.TrimSpace(change.Reason); summary != "" {
+33
View File
@@ -239,6 +239,39 @@ func TestIncidentStore_EqualTimestampLifecycleKeepsResolvedState(t *testing.T) {
}
}
// Without a canonical resource timeline the store writes its own close event,
// which must say the alert moved rather than resolved.
func TestIncidentStore_HandoverCloseIsNotARecovery(t *testing.T) {
store := NewIncidentStore(IncidentStoreConfig{MaxIncidents: 10, MaxEventsPerIncident: 10, MaxAgeDays: 30})
startedAt := time.Now().UTC().Truncate(time.Second).Add(-time.Hour)
movedAt := startedAt.Add(30 * time.Minute)
alert := &alerts.Alert{
ID: "pve1-memory",
Type: "memory",
ResourceID: "pve1",
ResourceName: "pve1",
Message: "Memory usage at 95%",
StartTime: startedAt,
LastSeen: startedAt,
Resolution: &alerts.AlertResolution{Reason: alerts.AlertResolutionMovedToAgent, SuccessorName: "pve1 (Host Agent)"},
}
summary := "Alert moved to pve1 (Host Agent). This is not a recovery: check the agent for the current reading."
store.RecordAlertFired(alert)
store.RecordAlertResolved(alert, movedAt)
timeline := store.GetTimelineByAlertAt(alert.ID, startedAt)
if timeline == nil || len(timeline.Events) != 2 || timeline.Events[1].Summary != summary {
t.Fatalf("recorded timeline = %#v, want the handover close", timeline)
}
snapshot := alert.Clone()
snapshot.ID = "pve1-memory-imported"
repaired := store.EnsureAlertOccurrence(snapshot, &movedAt)
if repaired == nil || len(repaired.Events) != 2 || repaired.Events[1].Summary != summary {
t.Fatalf("repaired timeline = %#v, want the handover close", repaired)
}
}
func TestIncidentStore_ProjectsCanonicalTimelineWhenAttached(t *testing.T) {
store := NewIncidentStore(IncidentStoreConfig{
MaxIncidents: 10,
+6
View File
@@ -3499,6 +3499,12 @@ func (p *PatrolService) seedHealthAndAlertsState(snap patrolRuntimeState, scoped
sb.WriteString("# Recently Resolved Alerts\n")
for _, r := range alerts {
ago := seedFormatTimeAgo(now, r.ResolvedTime)
// A handover closes the alert without the condition clearing, so
// it must not read as the resource having recovered.
if r.Resolution != nil && strings.TrimSpace(r.Resolution.Summary) != "" {
sb.WriteString(fmt.Sprintf("- %s — closed %s. %s\n", seedAlertSubject(r.Alert), ago, strings.TrimSpace(r.Resolution.Summary)))
continue
}
sb.WriteString(fmt.Sprintf("- %s — resolved %s\n", seedAlertSubject(r.Alert), ago))
}
sb.WriteString("\n")
+36
View File
@@ -922,6 +922,42 @@ func TestSeedHealthAndAlerts_NoIssues(t *testing.T) {
}
}
// Patrol's seed must not tell the model a node recovered when its alert only
// moved to the node's Pulse agent.
func TestSeedHealthAndAlerts_HandoverIsNotARecovery(t *testing.T) {
ps := NewPatrolService(nil, nil)
cfg := DefaultPatrolConfig()
now := time.Now()
summary := "Alert moved to pve1 (Host Agent). This is not a recovery: check the agent for the current reading."
state := models.StateSnapshot{
RecentlyResolved: []models.ResolvedAlert{
{
Alert: models.Alert{
ResourceName: "pve1",
Message: "Memory usage at 95%",
Resolution: &models.AlertResolution{Reason: "moved_to_agent", Summary: summary},
},
ResolvedTime: now.Add(-3 * time.Minute),
},
{
Alert: models.Alert{ResourceName: "pve2", Message: "CPU usage at 90%"},
ResolvedTime: now.Add(-5 * time.Minute),
},
},
}
out := ps.seedHealthAndAlertsState(patrolRuntimeStateForTest(ps, state), nil, cfg, now)
if !strings.Contains(out, "- pve1: Memory usage at 95% — closed 3m ago. "+summary+"\n") {
t.Fatalf("expected the handover line, got: %s", out)
}
if strings.Contains(out, "pve1: Memory usage at 95% — resolved") {
t.Fatalf("handover reported as a recovery: %s", out)
}
if !strings.Contains(out, "- pve2: CPU usage at 90% — resolved 5m ago\n") {
t.Fatalf("expected the ordinary recovery line, got: %s", out)
}
}
func TestSeedHealthAndAlerts_UnknownDiskHealthDoesNotClaimSMARTPassed(t *testing.T) {
ps := NewPatrolService(nil, nil)
cfg := DefaultPatrolConfig()
+5
View File
@@ -2364,4 +2364,9 @@ type ResolvedAlertSummary struct {
Threshold float64 `json:"threshold,omitempty"`
StartTime time.Time `json:"start_time"`
ResolvedTime time.Time `json:"resolved_time"`
// Resolution is set only when the alert closed without recovering, such
// as a node metric handed to its Pulse agent; the successor resource
// owns the condition from then on.
Resolution string `json:"resolution,omitempty"`
SuccessorResourceID string `json:"successor_resource_id,omitempty"`
}
+8 -3
View File
@@ -19,7 +19,7 @@ func (e *PulseToolExecutor) registerAlertsTools() {
Actions:
- list: List active threshold alerts (CPU > 80%, disk full, etc.)
- findings: List AI patrol findings (detected issues)
- resolved: List recently resolved alerts
- resolved: List recently resolved alerts; one with a resolution closed without recovering
- resolve: Mark a finding as resolved
- dismiss: Dismiss a finding as not an issue
@@ -417,7 +417,7 @@ func (e *PulseToolExecutor) executeListResolvedAlerts(_ context.Context, args ma
break
}
alerts = append(alerts, ResolvedAlertSummary{
summary := ResolvedAlertSummary{
ID: alert.ID,
Type: alert.Type,
Level: alert.Level,
@@ -430,7 +430,12 @@ func (e *PulseToolExecutor) executeListResolvedAlerts(_ context.Context, args ma
Threshold: alert.Threshold,
StartTime: alert.StartTime,
ResolvedTime: alert.ResolvedTime,
})
}
if resolution := alert.Resolution; resolution != nil {
summary.Resolution = resolution.Summary
summary.SuccessorResourceID = resolution.SuccessorResourceID
}
alerts = append(alerts, summary)
}
if alerts == nil {
@@ -215,6 +215,49 @@ func TestExecuteListResolvedAlerts(t *testing.T) {
if len(resp.Alerts) != 1 || resp.Alerts[0].ID != "a1" {
t.Fatalf("unexpected resolved alerts: %+v", resp)
}
if resp.Alerts[0].Resolution != "" || resp.Alerts[0].SuccessorResourceID != "" {
t.Fatalf("a recovery must not carry a resolution: %+v", resp.Alerts[0])
}
}
// A node alert handed to its Pulse agent closes without recovering, so the
// assistant must see that, and where the condition went, instead of a plain
// resolve it would read as the node being healthy again.
func TestExecuteListResolvedAlertsReportsHandover(t *testing.T) {
alertProv := &mockAlertProvider{}
executor := NewPulseToolExecutor(ExecutorConfig{})
executor.alertProvider = alertProv
now := time.Now()
summary := "Alert moved to pve1 (Host Agent). This is not a recovery: check the agent for the current reading."
alertProv.On("GetRecentlyResolved", 24*60).Return([]models.ResolvedAlert{{
Alert: models.Alert{
ID: "pve1-memory",
Type: "memory",
Level: "warning",
ResourceID: "pve1",
ResourceName: "pve1",
Message: "Memory usage at 95%",
StartTime: now.Add(-time.Hour),
Resolution: &models.AlertResolution{
Reason: "moved_to_agent",
SuccessorResourceID: "agent-pve1",
SuccessorName: "pve1 (Host Agent)",
Summary: summary,
},
},
ResolvedTime: now,
}})
result, _ := executor.executeListResolvedAlerts(context.Background(), map[string]interface{}{})
var resp ResolvedAlertsResponse
if err := json.Unmarshal([]byte(result.Content[0].Text), &resp); err != nil {
t.Fatalf("decode resolved alerts: %v", err)
}
if len(resp.Alerts) != 1 ||
resp.Alerts[0].Resolution != summary ||
resp.Alerts[0].SuccessorResourceID != "agent-pve1" {
t.Fatalf("resolved alerts = %+v, want the handover and its successor", resp.Alerts)
}
}
func TestExecuteListAlertsAndFindings(t *testing.T) {
+61
View File
@@ -7,7 +7,9 @@ import (
"testing"
"time"
"github.com/rcourtman/pulse-go-rewrite/internal/models"
"github.com/rcourtman/pulse-go-rewrite/internal/operationaltrust"
"github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources"
)
// newTestHistoryManager creates a HistoryManager using a temp directory
@@ -1058,3 +1060,62 @@ func TestSaveHistoryWithRetry_RestoresBackupOnFailure(t *testing.T) {
_ = hm
}
// The state snapshot, the websocket and the assistant read recently resolved
// alerts through GetRecentlyResolved, so a handover must reach them as one
// and not as a recovery.
func TestRecentlyResolvedCarriesHandoverResolution(t *testing.T) {
manager := newUnifiedEvalParityManager(t)
manager.UpdateConfig(AlertConfig{
Enabled: true,
NodeDefaults: ThresholdConfig{
CPU: &HysteresisThreshold{Trigger: 80, Clear: 75},
Memory: &HysteresisThreshold{Trigger: 85, Clear: 80},
},
})
disableTestTimeThresholds(manager)
node := models.Node{
ID: "pve-1",
Name: "pve-1",
Instance: "test",
Status: "online",
CPU: 0.95,
Memory: models.Memory{Total: 8 << 30, Used: 7 << 30, Usage: 95},
}
manager.CheckNode(node)
release := func(metric string, resolution *AlertResolution) {
t.Helper()
spec, err := buildCanonicalMetricSpec(node.ID, node.Name, unifiedresources.ResourceType("node"), metric, nil)
if err != nil {
t.Fatalf("build %s spec: %v", metric, err)
}
manager.releaseCanonicalMetricAlert(spec, node.Name, node.Name, node.Instance, "node", 0, resolution)
}
release("memory", &AlertResolution{
Reason: AlertResolutionMovedToAgent,
SuccessorResourceID: "agent-pve-1",
SuccessorName: "pve-1 (Host Agent)",
})
release("cpu", nil)
byType := map[string]models.ResolvedAlert{}
for _, resolved := range manager.GetRecentlyResolved() {
byType[resolved.Type] = resolved
}
moved, ok := byType["memory"]
if !ok {
t.Fatalf("recently resolved = %+v, want the moved memory alert", byType)
}
want := models.AlertResolution{
Reason: "moved_to_agent",
SuccessorResourceID: "agent-pve-1",
SuccessorName: "pve-1 (Host Agent)",
Summary: "Alert moved to pve-1 (Host Agent). This is not a recovery: check the agent for the current reading.",
}
if moved.Resolution == nil || *moved.Resolution != want {
t.Fatalf("moved resolution = %+v, want %+v", moved.Resolution, want)
}
if recovered, ok := byType["cpu"]; !ok || recovered.Resolution != nil {
t.Fatalf("recovered cpu alert = %+v, want an ordinary resolve with no resolution", recovered)
}
}
+13 -2
View File
@@ -285,6 +285,7 @@ func markOperationalResolved(
}
alert.Evidence = appendOperationalEvidence(alert.Evidence, recoveryEvidence.Clone())
recoveryEvidenceIDs := []string{recoveryEvidence.ID}
cause := resolvedTransitionCause(alert.Resolution)
from := record.State
record.State = operationaltrust.OperationalResolved
record.StateChangedAt = resolvedAt
@@ -295,7 +296,7 @@ func markOperationalResolved(
from,
operationaltrust.OperationalResolved,
resolvedAt,
operationaltrust.TransitionRecoveryEvidence,
cause,
record.CauseKey,
recoveryEvidenceIDs,
)
@@ -308,7 +309,7 @@ func markOperationalResolved(
From: from,
To: operationaltrust.OperationalResolved,
At: resolvedAt,
Cause: operationaltrust.TransitionRecoveryEvidence,
Cause: cause,
CauseKey: record.CauseKey,
EvidenceIDs: recoveryEvidenceIDs,
Reason: alert.Resolution.Summary(),
@@ -319,6 +320,16 @@ func markOperationalResolved(
)
}
// resolvedTransitionCause names why a record entered resolved. Only a close
// with no typed resolution is a recovery; a handover keeps its own cause so
// timelines and alert-quality counts never report it as one.
func resolvedTransitionCause(resolution *AlertResolution) operationaltrust.TransitionCause {
if resolution != nil && resolution.Reason == AlertResolutionMovedToAgent {
return operationaltrust.TransitionOwnershipTransferred
}
return operationaltrust.TransitionRecoveryEvidence
}
func (m *Manager) newResolvedAlert(
alert *Alert,
resolvedAt time.Time,
@@ -504,6 +504,15 @@ func TestReleasedMetricAlertCarriesResolutionInsteadOfRecovery(t *testing.T) {
if err := transition.Validate(); err != nil {
t.Fatalf("resolve transition Validate() error = %v", err)
}
// Patrol's timeline prints the cause above the reason, so a handover
// must not be labelled as recovery evidence.
if transition.Cause != operationaltrust.TransitionOwnershipTransferred {
t.Fatalf("resolve transition cause = %q, want %q", transition.Cause, operationaltrust.TransitionOwnershipTransferred)
}
if recorded := moved.Alert.Transitions[len(moved.Alert.Transitions)-1]; recorded.ID != transition.ID ||
recorded.Cause != operationaltrust.TransitionOwnershipTransferred {
t.Fatalf("recorded timeline transition = %+v, want the ownership transfer", recorded)
}
closing := moved.Alert.Evidence[len(moved.Alert.Evidence)-1]
if closing.ID != transition.EvidenceIDs[0] ||
closing.Reason == nil ||
@@ -520,6 +529,10 @@ func TestReleasedMetricAlertCarriesResolutionInsteadOfRecovery(t *testing.T) {
last.Reason.Code != legacyAlertRecoveryEvidenceReason {
t.Fatalf("plain release evidence = %+v, want legacy recovery projection", last.Reason)
}
if recovered.Alert.LatestTransition == nil ||
recovered.Alert.LatestTransition.Cause != operationaltrust.TransitionRecoveryEvidence {
t.Fatalf("plain release transition = %+v, want recovery_evidence", recovered.Alert.LatestTransition)
}
var historyResolution *AlertResolution
for _, entry := range manager.GetAlertHistory(10) {
+16 -1
View File
@@ -301,7 +301,8 @@ func (m *Manager) GetRecentlyResolved() []models.ResolvedAlert {
StartTime: exported.StartTime,
Acknowledged: exported.Acknowledged,
// exported is a deep clone, so the map is already private.
Metadata: exported.Metadata,
Metadata: exported.Metadata,
Resolution: modelAlertResolution(exported.Resolution),
},
ResolvedTime: alert.ResolvedTime,
})
@@ -309,6 +310,20 @@ func (m *Manager) GetRecentlyResolved() []models.ResolvedAlert {
return resolved
}
// modelAlertResolution projects a non-recovery close into the state model, so
// snapshot and assistant readers can tell a handover from a recovery.
func modelAlertResolution(resolution *AlertResolution) *models.AlertResolution {
if resolution == nil {
return nil
}
return &models.AlertResolution{
Reason: string(resolution.Reason),
SuccessorResourceID: resolution.SuccessorResourceID,
SuccessorName: resolution.SuccessorName,
Summary: resolution.Summary(),
}
}
// GetResolvedAlert returns a copy of a recently resolved alert by ID.
func (m *Manager) GetResolvedAlert(alertID string) *ResolvedAlert {
// Write lock: getResolvedAlertNoLock backfills the resolvedAlias map on a
+4
View File
@@ -1337,6 +1337,10 @@ func cloneAlert(src Alert) Alert {
dest.Metadata[k] = v
}
}
if src.Resolution != nil {
resolution := *src.Resolution
dest.Resolution = &resolution
}
return dest
}
+37
View File
@@ -96,6 +96,43 @@ func TestAlertLastSeenWireContract(t *testing.T) {
}
}
// A resolved alert that closed without recovering says so on the state and
// websocket payloads; an ordinary recovery carries no resolution at all.
func TestAlertResolutionWireContract(t *testing.T) {
recovered, err := json.Marshal(ResolvedAlert{Alert: Alert{ID: "alert-1"}})
if err != nil {
t.Fatalf("marshal recovered alert: %v", err)
}
if strings.Contains(string(recovered), "resolution") {
t.Fatalf("payload = %s, a recovery must not carry a resolution", recovered)
}
moved := ResolvedAlert{Alert: Alert{ID: "alert-1", Resolution: &AlertResolution{
Reason: "moved_to_agent",
SuccessorResourceID: "agent-pve-1",
SuccessorName: "pve-1 (Host Agent)",
Summary: "Alert moved to pve-1 (Host Agent). This is not a recovery: check the agent for the current reading.",
}}}
payload, err := json.Marshal(moved)
if err != nil {
t.Fatalf("marshal moved alert: %v", err)
}
want := `"resolution":{"reason":"moved_to_agent","successorResourceId":"agent-pve-1","successorName":"pve-1 (Host Agent)","summary":"Alert moved to pve-1 (Host Agent). This is not a recovery: check the agent for the current reading."}`
if !strings.Contains(string(payload), want) {
t.Fatalf("payload = %s, want %s", payload, want)
}
// The state store keeps its own copy of each resolution.
state := NewState()
state.UpdateRecentlyResolved([]ResolvedAlert{moved})
moved.Resolution.SuccessorName = "changed"
snapshot := state.GetSnapshot().RecentlyResolved
if len(snapshot) != 1 || snapshot[0].Resolution == nil ||
snapshot[0].Resolution.SuccessorName != "pve-1 (Host Agent)" {
t.Fatalf("snapshot = %+v, want the resolution as it was recorded", snapshot)
}
}
func TestDockerHostCollectionModeWireContract(t *testing.T) {
payload, err := json.Marshal(DockerHost{
ID: "docker-host-1",
+14
View File
@@ -110,6 +110,20 @@ type Alert struct {
// Metadata carries alert-engine annotations (notably resourceType) so the
// frontend can classify an alert without re-deriving resource identity.
Metadata map[string]interface{} `json:"metadata,omitempty"`
// Resolution is set only on a resolved alert whose close was not a
// recovery, so readers never report its resource as healthy.
Resolution *AlertResolution `json:"resolution,omitempty"`
}
// AlertResolution is the state projection of the alert engine's
// AlertResolution: why an alert closed without its condition clearing.
type AlertResolution struct {
Reason string `json:"reason"`
SuccessorResourceID string `json:"successorResourceId,omitempty"`
SuccessorName string `json:"successorName,omitempty"`
// Summary is the alert engine's one-line account, such as "Alert moved
// to pve1 (Host Agent). This is not a recovery: ...".
Summary string `json:"summary,omitempty"`
}
// ResolvedAlert represents a recently resolved alert
@@ -238,6 +238,68 @@ func TestMonitor_HandleAlertLifecycle_WritesCanonicalChanges(t *testing.T) {
}
}
// A node alert handed to its Pulse agent closes without recovering. The
// resource history and the alert's incident timeline (the Alerts history row
// expansion) must say where it went, not "Alert resolved: Memory usage at 95%".
func TestMonitor_HandleAlertLifecycle_HandoverCloseIsNotARecovery(t *testing.T) {
resourceStore := unifiedresources.NewMemoryStore()
incidentStore := memory.NewIncidentStore(memory.IncidentStoreConfig{})
m := &Monitor{
incidentStore: incidentStore,
resourceStore: unifiedresources.NewMonitorAdapter(unifiedresources.NewRegistry(resourceStore)),
}
incidentStore.SetResourceTimelineStore(m.resourceStore.(memory.IncidentTimelineStore))
startedAt := time.Date(2026, 10, 6, 9, 0, 0, 0, time.UTC)
movedAt := startedAt.Add(2 * time.Hour)
alert := &alerts.Alert{
ID: "pve1-memory",
Type: "memory",
Level: alerts.AlertLevelWarning,
ResourceID: "pve1",
Message: "Memory usage at 95%",
Value: 95,
Threshold: 85,
StartTime: startedAt,
}
m.handleAlertLifecycleEvent(alerts.LifecycleEvent{Type: eventlog.TypeFired, OccurredAt: startedAt, Alert: alert})
closed := alert.Clone()
closed.Resolution = &alerts.AlertResolution{
Reason: alerts.AlertResolutionMovedToAgent,
SuccessorResourceID: "agent-pve1",
SuccessorName: "pve1 (Host Agent)",
}
m.handleAlertLifecycleEvent(alerts.LifecycleEvent{Type: eventlog.TypeResolved, OccurredAt: movedAt, Alert: closed})
summary := "Alert moved to pve1 (Host Agent). This is not a recovery: check the agent for the current reading."
changes, err := resourceStore.GetRecentChanges("pve1", time.Time{}, 10)
require.NoError(t, err)
require.Len(t, changes, 2)
require.Equal(t, unifiedresources.ChangeAlertResolved, changes[0].Kind)
require.Equal(t, summary, changes[0].Reason)
require.Equal(t, "moved_to_agent", changes[0].Metadata[unifiedresources.MetadataAlertResolution])
require.Equal(t, unifiedresources.ChangeAlertFired, changes[1].Kind)
require.NotContains(t, changes[1].Metadata, unifiedresources.MetadataAlertResolution)
timeline := incidentStore.GetTimelineByAlertAt(alert.ID, startedAt)
require.NotNil(t, timeline)
require.Len(t, timeline.Events, 2)
require.Equal(t, memory.IncidentEventAlertResolved, timeline.Events[1].Type)
require.Equal(t, summary, timeline.Events[1].Summary)
// An ordinary recovery keeps its existing wording.
recovered := alert.Clone()
recovered.ID = "pve1-cpu"
recovered.Type = "cpu"
recovered.Message = "CPU usage at 90%"
m.handleAlertLifecycleEvent(alerts.LifecycleEvent{Type: eventlog.TypeFired, OccurredAt: startedAt, Alert: recovered})
m.handleAlertLifecycleEvent(alerts.LifecycleEvent{Type: eventlog.TypeResolved, OccurredAt: movedAt.Add(time.Minute), Alert: recovered})
recoveredTimeline := incidentStore.GetTimelineByAlertAt(recovered.ID, startedAt)
require.NotNil(t, recoveredTimeline)
require.Len(t, recoveredTimeline.Events, 2)
require.Equal(t, "Alert resolved", recoveredTimeline.Events[1].Summary)
}
func TestPausedDeliveryStillBuildsTimelineThroughRealAlertLifecycle(t *testing.T) {
manager := alerts.NewManagerWithDataDir(t.TempDir())
t.Cleanup(manager.Stop)
+7 -2
View File
@@ -576,7 +576,7 @@ func (m *Monitor) recordAlertTimelineChange(alert *alerts.Alert, kind unifiedres
return
}
change := unifiedresources.BuildAlertTimelineChange(alert.ResourceID, kind, occurredAt, actor, unifiedresources.AlertTimelineChange{
timelineChange := unifiedresources.AlertTimelineChange{
AlertIdentifier: alert.ID,
AlertStartedAt: alert.StartTime,
AlertType: alert.Type,
@@ -585,7 +585,12 @@ func (m *Monitor) recordAlertTimelineChange(alert *alerts.Alert, kind unifiedres
AlertValue: alert.Value,
AlertThreshold: alert.Threshold,
AlertMetadata: alert.Metadata,
})
}
if summary := alert.Resolution.Summary(); summary != "" {
timelineChange.ResolutionReason = string(alert.Resolution.Reason)
timelineChange.ResolutionSummary = summary
}
change := unifiedresources.BuildAlertTimelineChange(alert.ResourceID, kind, occurredAt, actor, timelineChange)
if change == nil {
return
}
+14 -2
View File
@@ -493,8 +493,12 @@ const (
TransitionSuppression TransitionCause = "suppression"
TransitionSuppressionExpired TransitionCause = "suppression_expired"
TransitionRecoveryEvidence TransitionCause = "recovery_evidence"
TransitionCollectionStale TransitionCause = "collection_stale"
TransitionCollectionUnknown TransitionCause = "collection_unknown"
// TransitionOwnershipTransferred closes a record whose condition another
// record now owns, such as a node metric a linked Pulse agent took over.
// The condition was not observed to clear, so it is not a recovery.
TransitionOwnershipTransferred TransitionCause = "ownership_transferred"
TransitionCollectionStale TransitionCause = "collection_stale"
TransitionCollectionUnknown TransitionCause = "collection_unknown"
)
func (cause TransitionCause) valid() bool {
@@ -505,6 +509,7 @@ func (cause TransitionCause) valid() bool {
TransitionSuppression,
TransitionSuppressionExpired,
TransitionRecoveryEvidence,
TransitionOwnershipTransferred,
TransitionCollectionStale,
TransitionCollectionUnknown:
return true
@@ -617,6 +622,13 @@ func (transition LifecycleTransition) validateWithoutID() error {
if len(canonicalIDs(transition.EvidenceIDs)) == 0 {
return errors.New("recovery transition requires evidence")
}
case TransitionOwnershipTransferred:
if transition.To != OperationalResolved {
return errors.New("ownership transfer must enter resolved state")
}
if len(canonicalIDs(transition.EvidenceIDs)) == 0 {
return errors.New("ownership transfer requires evidence")
}
case TransitionCollectionStale:
if transition.To != OperationalStale {
return errors.New("collection-stale transition must enter stale state")
@@ -235,6 +235,48 @@ func TestLifecycleTransitionEnforcesOperationalSemantics(t *testing.T) {
}
}
// A close that hands the condition to another record is not a recovery, so it
// has its own cause. It may only close the record, and only with the evidence
// that recorded the handover.
func TestOwnershipTransferOnlyClosesWithEvidence(t *testing.T) {
at := time.Date(2026, 10, 6, 15, 0, 0, 0, time.UTC)
if _, err := NewTransitionID(
"record-1",
OperationalOpen,
OperationalResolved,
at,
TransitionOwnershipTransferred,
"memory-high",
[]string{"evidence-handover"},
); err != nil {
t.Fatalf("open-to-resolved ownership transfer error = %v", err)
}
if _, err := NewTransitionID(
"record-1",
OperationalAcknowledged,
OperationalResolved,
at,
TransitionOwnershipTransferred,
"memory-high",
nil,
); err == nil || !strings.Contains(err.Error(), "requires evidence") {
t.Fatalf("evidenceless ownership transfer error = %v", err)
}
for _, to := range []OperationalState{OperationalResolving, OperationalOpen, OperationalStale} {
if _, err := NewTransitionID(
"record-1",
OperationalAcknowledged,
to,
at,
TransitionOwnershipTransferred,
"memory-high",
[]string{"evidence-handover"},
); err == nil || !strings.Contains(err.Error(), "must enter resolved") {
t.Fatalf("ownership transfer into %q error = %v", to, err)
}
}
}
func TestOperationalRecordRequiresResolutionEvidence(t *testing.T) {
firstObserved := time.Date(2026, 7, 18, 20, 0, 0, 0, time.UTC)
resolvedAt := firstObserved.Add(5 * time.Minute)
@@ -16,6 +16,9 @@ const (
MetadataAlertMessage = "alert_message"
MetadataAlertValue = "alert_value"
MetadataAlertThreshold = "alert_threshold"
// MetadataAlertResolution is the reason code of a resolve that was not a
// recovery, such as "moved_to_agent". Ordinary recoveries omit it.
MetadataAlertResolution = "alert_resolution"
MetadataCommand = "command"
MetadataSuccess = "success"
MetadataOutputExcerpt = "output_excerpt"
@@ -38,6 +41,11 @@ type AlertTimelineChange struct {
AlertValue float64
AlertThreshold float64
AlertMetadata map[string]any
// ResolutionReason and ResolutionSummary describe a resolve that was not
// a recovery. They are empty for an ordinary recovery and ignored on
// every other lifecycle kind.
ResolutionReason string
ResolutionSummary string
}
// BuildAlertTimelineChange constructs a canonical resource change for alert
@@ -74,6 +82,9 @@ func BuildAlertTimelineChange(resourceID string, kind ChangeKind, occurredAt tim
if !alert.AlertStartedAt.IsZero() {
change.Metadata[MetadataAlertStartedAt] = alert.AlertStartedAt.UTC().Format(time.RFC3339Nano)
}
if reason := strings.TrimSpace(alert.ResolutionReason); kind == ChangeAlertResolved && reason != "" {
change.Metadata[MetadataAlertResolution] = reason
}
for key, value := range cloneChangeMetadata(alert.AlertMetadata) {
if _, exists := change.Metadata[key]; exists {
continue
@@ -178,6 +189,11 @@ func alertChangeReason(kind ChangeKind, alert AlertTimelineChange) string {
case ChangeAlertUnsnoozed:
return "Alert notifications resumed"
case ChangeAlertResolved:
// A close that was not a recovery must not read as one: the alert's
// own message describes the condition that is still unresolved.
if summary := strings.TrimSpace(alert.ResolutionSummary); summary != "" {
return summary
}
if message != "" {
return fmt.Sprintf("Alert resolved: %s", message)
}