gsadmin/pulse
1
0
Fork 0
mirror of https://github.com/rcourtman/Pulse.git synced 2026-09-10 18:45:53 +00:00
Code Issues Packages Projects Releases Wiki Activity

fix(alerts): make incident timelines lifecycle-driven

Browse Source 
Contract-Neutral: monitor startup wiring changes only alert lifecycle projection behavior; agent-lifecycle deletion and reenrollment contracts are unchanged
This commit is contained in:
rcourtman
2026-08-27 14:45:36 +01:00
parent 1e75423116
commit b6babd0b4b
26 changed files with 919 additions and 205 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/release-control/v6/internal/subsystems/alerts.md
+31 -7
View File
@@ -991,14 +991,21 @@ projection once per refresh and may render its current delivery outcome on an
unacknowledged active-alert card. Missing or failed diagnosis reads degrade to
no delivery line; they must not hide alert truth or trigger per-card requests.
`internal/alerts/eventlog/` and `internal/alerts/event_emission.go` own the
additive append-only alert event record. Persistent managers enable a
canonical append-only alert event record. Persistent managers enable a
SQLite-backed store under the alerts data directory; ephemeral managers record
nothing unless a store is installed explicitly. Resolution, acknowledgement,
unacknowledgement, escalation, flapping detection, dispatch, quiet-hours
deferral, and suppression append immutable events without changing lifecycle
or delivery behavior. The write path is non-blocking and fail-open for alert
evaluation: a full buffer counts and drops its event, while an unavailable
store disables recording. Events retain for 90 days and prune hourly. Fired and
or delivery behavior. Snapshot-bearing lifecycle transitions commit
synchronously before downstream lifecycle projections run; they never share
the droppable diagnostic buffer because alert history is reconstructed from
them. High-volume notification decisions remain non-blocking and fail-open for
alert evaluation: a full diagnostic buffer counts and drops that delivery
evidence. Failed asynchronous batches do not advance the successful-write
counter, and `Flush` must report the failure or timeout rather than falsely
claiming the batch landed. An unavailable or failed lifecycle store degrades
history reads to the recovery model without hiding live active-alert truth.
Events retain for 90 days and prune hourly. Fired and
refired lifecycle events come only from the reducer core's explicit activation
events: canonical lifecycle reactivation maps `EventRefired` separately, while
shared metric activation records one fired event when its pending incident
@@ -1020,8 +1027,11 @@ Lifecycle events now carry a full alert snapshot and
`internal/alerts/history_projection.go` can fold those snapshots into one
history row per occurrence. Existing pre-snapshot SQLite stores upgrade in
place, and rows without snapshots remain readable but cannot contribute to the
projection. With an event log enabled, it is the alert-history read authority;
managers without one retain the in-memory JSON-history model as their fallback.
projection. The event log becomes the alert-history read authority only after
legacy history is absent or its complete import and retirement succeeds.
Managers without an authoritative event store, and stores that report a
lifecycle write failure, retain the in-memory JSON-history model as their
fail-soft read fallback.
`internal/alerts/history_projection_parity_test.go` characterizes the event-log
projection against that fallback for active, resolved, acknowledged,
multi-resource, and repeated occurrences. Live active-alert state overlays the
@@ -1038,12 +1048,26 @@ failure leaves JSON history authoritative for the next attempt. An unreadable
or malformed source, or any entry that cannot produce an identified complete
snapshot, blocks both retirement and further JSON writes so startup cannot
silently replace recoverable user history with an empty file. Successful
retirement stops further JSON writes. Clearing history appends a
retirement stops further JSON writes. Retired `.imported` leaves remain bounded
recovery sources: startup loads and idempotently replays them if `events.db` is
recreated, so loss or quarantine of the database cannot turn a successful
migration into silent empty history. Clearing history appends a
`history_cleared` tombstone rather than deleting log rows; the projection
ignores earlier lifecycle events but overlays still-active alerts as current
state. Unified-incident and system-alert activation must emit snapshot-bearing
fired events so their projected history does not begin at acknowledgement or
resolution.
`SubscribeLifecycleCallback` is the delivery-independent projection seam for
that canonical stream. It runs for lifecycle transitions regardless of
activation, quiet hours, grouping, rate limits, or destination state; delivery
callbacks remain policy-controlled consumers. Monitoring uses this seam to
materialize canonical resource-history breadcrumbs and incident shells. On
startup it replays durable lifecycle transitions oldest first, then
idempotently reconciles restored active alerts that predate the event store.
This repairs resolved as well as active incident timelines without replaying
any delivery side effect, and ensures an alert already visible in Overview
cannot keep returning an unavailable timeline merely because its notification
was held.
The same dispatch policy owns firing-notification evidence on active alerts:
any alert that passes notification suppression and enters the fired callback
fan-out must carry `LastNotified` before the callback clone is emitted. Resolved
docs/release-control/v6/internal/subsystems/frontend-primitives.md
+7
View File
@@ -5687,6 +5687,13 @@ while `frontend-modern/src/features/alerts/OverviewTab.tsx` and
surface composition. Future incident timeline fetch, note-save, or expansion
control flow should extend that feature hook rather than forking back into
either tab surface.
Every surface invoking that owner must pass the canonical alert identifier and
the occurrence start time as distinct arguments; a row key is local UI state,
not a substitute for either API identity field.
The shared `IncidentTimelinePanel` consumes loading, error, timeline, draft,
and save state as accessors so an asynchronous result remains reactive across
Overview, desktop History, and mobile History instead of freezing the values
present when the panel first expands.
Overview alert runtime now follows that same shell-versus-runtime split. The
shell stays in `frontend-modern/src/features/alerts/OverviewTab.tsx`, while
`frontend-modern/src/features/alerts/useAlertOverviewState.ts` owns derived
docs/release-control/v6/internal/subsystems/monitoring.md
+17 -8
View File
@@ -2071,14 +2071,23 @@ rebuilds and supplemental ingest into the unified-resource timeline. That means
monitoring no longer just materializes state snapshots for consumers; it also
emits durable `ResourceChange` history through the shared resource store so
live monitoring updates and historical inspection stay aligned.
That same ownership now includes alert-lifecycle facts emitted by monitoring.
When an alert is fired, acknowledged, unacknowledged, or resolved for a
canonical resource, the monitoring runtime must write the corresponding durable
resource-history event into the unified-resource change store instead of
leaving that lifecycle only inside alert-scoped incident memory. Incident
timelines may still project those breadcrumbs for operator flow, but the
durable backend truth for alert lifecycle now lives on the canonical resource
timeline.
That same ownership now includes the resource-history projection of canonical
alert-lifecycle facts. The alerts-owned SQLite event log is the lifecycle source
of truth; monitoring consumes its delivery-independent lifecycle seam and
materializes fired, acknowledged, unacknowledged, and resolved breadcrumbs in
the unified-resource change store. Projection IDs derive deterministically from
alert identity, canonical resource, transition kind, and occurrence time, so
restart repair and duplicate consumer delivery are idempotent. Notification
activation, quiet hours, grouping, throttling, and destination health may never
gate this projection. Incident timelines project those breadcrumbs for
operator flow, while the resource timeline remains the durable resource-scoped
index rather than a second alert lifecycle authority.
Monitoring must install the lifecycle consumer, replay durable lifecycle
events oldest first, and then reconcile restored active alerts that predate
the event store. Replay and reconciliation create only missing projections,
including a stable `pulse-system` timeline identity for system alerts whose
public alert payload intentionally has no monitored-resource link; neither path
may duplicate a resource change or invoke notification delivery.
The monitor-owned incident store wiring must therefore attach the canonical
resource timeline reader whenever the unified monitor adapter is present, so
operator alert timelines and AI incident context project those lifecycle events
frontend-modern/browser-verification.json
+24 -25
View File
@@ -1,27 +1,25 @@
{
"version": 1,
"base_sha": "3d91709ebad0332858e4615d72daf8e424ab42a8",
"verified_at": "2026-08-27T12:19:53Z",
"base_sha": "1e75423116984e67ca92a8904f913e5e6a194324",
"verified_at": "2026-08-27T13:41:54Z",
"result": "passed",
"changed_paths": [
"frontend-modern/src/components/Alerts/ThresholdsTableAgentsTab.tsx",
"frontend-modern/src/components/Alerts/ThresholdsTableSMARTDefaultsCard.tsx",
"frontend-modern/src/features/alerts/alertsConfigurationModel.ts",
"frontend-modern/src/types/alerts.ts",
"frontend-modern/src/utils/alertThresholdDefaults.ts"
"frontend-modern/src/components/Alerts/IncidentTimelinePanel.tsx",
"frontend-modern/src/features/alerts/AlertHistoryMobileList.tsx",
"frontend-modern/src/features/alerts/AlertHistoryTableAlertRow.tsx",
"frontend-modern/src/features/alerts/AlertOverviewAlertCard.tsx"
],
"content_sha256": {
"frontend-modern/src/components/Alerts/ThresholdsTableAgentsTab.tsx": "89646984deb982699d3359953ab85a2ceae15830465ccf20b5603f5a426e3193",
"frontend-modern/src/components/Alerts/ThresholdsTableSMARTDefaultsCard.tsx": "c1bd1947dbcbab5806e11fa905271039131cebb0a736642e73fd3652f63e6b66",
"frontend-modern/src/features/alerts/alertsConfigurationModel.ts": "2100da1f350cbbea386f1ae669a655d213ea710f1636eb94fe80c6d47d29313d",
"frontend-modern/src/types/alerts.ts": "cc8b88ec5d8de87dbb69116114e9cd889347972b59dd81bec081096ab791c693",
"frontend-modern/src/utils/alertThresholdDefaults.ts": "6ea4a3f99c8bd27a4fcca98f67c866d06a0f5d3e3211550a58447688cb61352a"
"frontend-modern/src/components/Alerts/IncidentTimelinePanel.tsx": "2f32d34e7bfedd2e052cf20075a4eb711d114ee5715c13a108f6b3666f29ad91",
"frontend-modern/src/features/alerts/AlertHistoryMobileList.tsx": "d5179ae59605d318f522483b13e5925fd7a6890a066b129eaf769d42b334d2d4",
"frontend-modern/src/features/alerts/AlertHistoryTableAlertRow.tsx": "cc763cdadc2f48e47437cfddf61ebbd5723e74a6a89f3640da013b63f6598cdb",
"frontend-modern/src/features/alerts/AlertOverviewAlertCard.tsx": "fc2822b803ab1c250811e8a374c7cba94ae1f4b08130acf48ba6b98cea3f0a5c"
},
"routes": ["/alerts/thresholds/systems"],
"routes": ["/alerts/overview", "/alerts/history"],
"viewports": [
{
"width": 1440,
"height": 1000
"width": 1280,
"height": 720
},
{
"width": 390,
@@ -29,17 +27,18 @@
}
],
"states": [
"Machines thresholds loaded with the backward-compatible SMART factory rules",
"failed-health disabled with pending sectors set to 4, life warning set to 15 percent, and NVMe spare critical disabled at 0",
"edited SMART rules persisted after Save Changes and a full route reload",
"factory SMART values restored and saved from the narrow layout",
"complete single-column SMART card inspected through the app scroll shell at 390x844"
"restored active alert visible while notification delivery is pending review",
"canonical incident API returns the occurrence-specific fired event for the active alert",
"expanded Overview timeline renders incident status, fired event, filters, timestamp, and note editor",
"legacy JSON history imported once into SQLite and retained across backend restart",
"history clear tombstone removes retired history while preserving the active alert overlay"
],
"interactions": [
"opened Alerts, selected Thresholds, and selected the Machines platform tab at 1440x1000",
"toggled failed-health, edited counter and percentage rules including an explicit 0 disable value, and saved through the shared dirty-state bar",
"reloaded /alerts/thresholds/systems and confirmed every edited value persisted",
"scrolled the complete SMART rules card at 390x844, confirmed scroll width matched client width, restored factory values, and saved from the narrow layout",
"confirmed labels, help text, controls, percentage suffixes, focusable save actions, and the browser console remained free of errors"
"opened Alerts Overview at 1280x720 and expanded the active alert timeline",
"confirmed the Timeline control requested the canonical alert identifier and occurrence start time exactly once",
"confirmed the rendered fired event replaced the unavailable-timeline state while delivery remained paused",
"reloaded Alerts Overview at 390x844, expanded the same timeline, and confirmed no horizontal overflow",
"restarted the managed real backend twice and confirmed history import, clear tombstone, and active-overlay continuity",
"confirmed the five-test alert operator and real-backend qualification suite passed with no product errors"
]
}
frontend-modern/src/components/Alerts/IncidentTimelinePanel.tsx
+13 -13
View File
@@ -30,31 +30,31 @@ import {
} from '@/utils/alertIncidentPresentation';
export interface IncidentTimelinePanelProps {
timeline?: Incident | null;
loading: boolean;
error: boolean;
timeline: Accessor<Incident | null | undefined>;
loading: Accessor<boolean>;
error: Accessor<boolean>;
filters: Accessor<Set<string>>;
setFilters: (next: Set<string>) => void;
filterVariant: AlertIncidentEventFilterVariant;
eventCardVariant: 'surface' | 'alt';
noteDraft: string;
noteDraft: Accessor<string>;
onNoteDraftChange: (value: string) => void;
noteSaving: boolean;
noteSaving: Accessor<boolean>;
onSaveNote: () => void;
onRetry: () => void;
}
export function IncidentTimelinePanel(props: IncidentTimelinePanelProps) {
const timeline = () => props.timeline;
const timeline = props.timeline;
const events = createMemo(() => timeline()?.events || []);
const filteredEvents = createMemo(() => filterIncidentEvents(events(), props.filters()));
return (
<>
<Show when={props.loading}>
<Show when={props.loading()}>
<p class="text-xs text-muted">{getAlertTimelineLoadingState().text}</p>
</Show>
<Show when={!props.loading && timeline()}>
<Show when={!props.loading() && timeline()}>
{(loadedTimeline) => (
<div class="space-y-3">
<div class="flex flex-col gap-2 sm:flex-row sm:items-start sm:justify-between">
@@ -121,25 +121,25 @@ export function IncidentTimelinePanel(props: IncidentTimelinePanelProps) {
textareaBaseClass={getAlertIncidentNoteTextareaClass()}
rows={2}
placeholder={getAlertTimelineNotePlaceholder()}
value={props.noteDraft}
value={props.noteDraft()}
onInput={(event) => props.onNoteDraftChange(event.currentTarget.value)}
/>
<div class="flex justify-end">
<button
class={getAlertIncidentNoteSaveButtonClass()}
disabled={props.noteSaving || !props.noteDraft.trim()}
disabled={props.noteSaving() || !props.noteDraft().trim()}
onClick={() => props.onSaveNote()}
>
{getAlertTimelineSaveNoteLabel(props.noteSaving)}
{getAlertTimelineSaveNoteLabel(props.noteSaving())}
</button>
</div>
</div>
</div>
)}
</Show>
<Show when={!props.loading && !timeline()}>
<Show when={!props.loading() && !timeline()}>
<Show
when={props.error}
when={props.error()}
fallback={<p class="text-xs text-muted">{getAlertTimelineUnavailableState().text}</p>}
>
<div class="flex items-center gap-2">
frontend-modern/src/components/Alerts/__tests__/IncidentTimelinePanel.test.tsx
+15 -15
View File
@@ -55,16 +55,16 @@ describe('IncidentTimelinePanel', () => {
render(() => (
<IncidentTimelinePanel
loading={state().loading}
error={state().error}
timeline={state().timeline}
loading={() => state().loading}
error={() => state().error}
timeline={() => state().timeline}
filters={filters}
setFilters={setFilters}
filterVariant="panel"
eventCardVariant="alt"
noteDraft=""
noteDraft={() => ''}
onNoteDraftChange={vi.fn()}
noteSaving={false}
noteSaving={() => false}
onSaveNote={vi.fn()}
onRetry={vi.fn()}
/>
@@ -88,16 +88,16 @@ describe('IncidentTimelinePanel', () => {
render(() => (
<IncidentTimelinePanel
loading={false}
error={false}
timeline={makeTimeline()}
loading={() => false}
error={() => false}
timeline={() => makeTimeline()}
filters={filters}
setFilters={setFilters}
filterVariant="compact"
eventCardVariant="surface"
noteDraft="operator note"
noteDraft={() => 'operator note'}
onNoteDraftChange={handleNoteDraftChange}
noteSaving={false}
noteSaving={() => false}
onSaveNote={handleSave}
onRetry={vi.fn()}
/>
@@ -127,16 +127,16 @@ describe('IncidentTimelinePanel', () => {
render(() => (
<IncidentTimelinePanel
loading={false}
error={false}
timeline={makeTimeline()}
loading={() => false}
error={() => false}
timeline={() => makeTimeline()}
filters={filters}
setFilters={setFilters}
filterVariant="compact"
eventCardVariant="surface"
noteDraft=""
noteDraft={() => ''}
onNoteDraftChange={vi.fn()}
noteSaving={false}
noteSaving={() => false}
onSaveNote={vi.fn()}
onRetry={vi.fn()}
/>
frontend-modern/src/features/alerts/AlertHistoryMobileList.tsx
+5 -5
View File
@@ -125,16 +125,16 @@ export function AlertHistoryMobileList(props: AlertHistoryMobileListProps) {
>
<div class="mt-3 border-t border-border pt-3">
<IncidentTimelinePanel
loading={props.state.incidentLoading()[rowKey()]}
error={props.state.incidentErrors()[rowKey()]}
timeline={props.state.incidentTimelines()[rowKey()]}
loading={() => props.state.incidentLoading()[rowKey()]}
error={() => props.state.incidentErrors()[rowKey()]}
timeline={() => props.state.incidentTimelines()[rowKey()]}
filters={props.state.historyIncidentEventFilters}
setFilters={props.state.setHistoryIncidentEventFilters}
filterVariant="compact"
eventCardVariant="surface"
noteDraft={props.state.incidentNoteDrafts()[rowKey()] || ''}
noteDraft={() => props.state.incidentNoteDrafts()[rowKey()] || ''}
onNoteDraftChange={(value) => props.state.setIncidentNoteDraft(rowKey(), value)}
noteSaving={props.state.incidentNoteSaving().has(rowKey())}
noteSaving={() => props.state.incidentNoteSaving().has(rowKey())}
onSaveNote={() => {
void props.state.saveIncidentNote(rowKey(), alert.id, alert.startTime);
}}
frontend-modern/src/features/alerts/AlertHistoryTableAlertRow.tsx
+5 -5
View File
@@ -96,16 +96,16 @@ export function AlertHistoryTableAlertRow(props: AlertHistoryTableAlertRowProps)
<TableRow class="border-b border-border bg-surface-alt">
<TableCell colspan={9} class="p-3">
<IncidentTimelinePanel
loading={props.state.incidentLoading()[rowKey()]}
error={props.state.incidentErrors()[rowKey()]}
timeline={props.state.incidentTimelines()[rowKey()]}
loading={() => props.state.incidentLoading()[rowKey()]}
error={() => props.state.incidentErrors()[rowKey()]}
timeline={() => props.state.incidentTimelines()[rowKey()]}
filters={props.state.historyIncidentEventFilters}
setFilters={props.state.setHistoryIncidentEventFilters}
filterVariant="compact"
eventCardVariant="surface"
noteDraft={props.state.incidentNoteDrafts()[rowKey()] || ''}
noteDraft={() => props.state.incidentNoteDrafts()[rowKey()] || ''}
onNoteDraftChange={(value) => props.state.setIncidentNoteDraft(rowKey(), value)}
noteSaving={props.state.incidentNoteSaving().has(rowKey())}
noteSaving={() => props.state.incidentNoteSaving().has(rowKey())}
onSaveNote={() => {
void props.state.saveIncidentNote(rowKey(), props.alert.id, props.alert.startTime);
}}
frontend-modern/src/features/alerts/AlertOverviewAlertCard.tsx
+20 -8
View File
@@ -191,7 +191,11 @@ export function AlertOverviewAlertCard(props: AlertOverviewAlertCardProps) {
<button
class={getAlertOverviewSecondaryActionClass()}
onClick={() => {
void props.timelineState.toggleIncidentTimeline(alertKey(), props.alert.startTime);
void props.timelineState.toggleIncidentTimeline(
alertKey(),
alertKey(),
props.alert.startTime,
);
}}
>
{getAlertOverviewTimelineActionLabel(
@@ -222,23 +226,31 @@ export function AlertOverviewAlertCard(props: AlertOverviewAlertCardProps) {
<Show when={props.timelineState.expandedIncidents().has(alertKey())}>
<div class="mt-3 border-t border-border pt-3">
<IncidentTimelinePanel
loading={props.timelineState.incidentLoading()[alertKey()]}
error={props.timelineState.incidentErrors()[alertKey()]}
timeline={props.timelineState.incidentTimelines()[alertKey()]}
loading={() => props.timelineState.incidentLoading()[alertKey()]}
error={() => props.timelineState.incidentErrors()[alertKey()]}
timeline={() => props.timelineState.incidentTimelines()[alertKey()]}
filters={props.timelineState.eventFilters}
setFilters={props.timelineState.setEventFilters}
filterVariant="panel"
eventCardVariant="alt"
noteDraft={props.timelineState.incidentNoteDrafts()[alertKey()] || ''}
noteDraft={() => props.timelineState.incidentNoteDrafts()[alertKey()] || ''}
onNoteDraftChange={(value) =>
props.timelineState.setIncidentNoteDraft(alertKey(), value)
}
noteSaving={props.timelineState.incidentNoteSaving().has(alertKey())}
noteSaving={() => props.timelineState.incidentNoteSaving().has(alertKey())}
onSaveNote={() => {
void props.timelineState.saveIncidentNote(alertKey(), props.alert.startTime);
void props.timelineState.saveIncidentNote(
alertKey(),
alertKey(),
props.alert.startTime,
);
}}
onRetry={() => {
void props.timelineState.loadIncidentTimeline(alertKey(), props.alert.startTime);
void props.timelineState.loadIncidentTimeline(
alertKey(),
alertKey(),
props.alert.startTime,
);
}}
/>
</div>
frontend-modern/src/features/alerts/__tests__/OverviewTab.timelineerror.test.tsx
+1
View File
@@ -167,6 +167,7 @@ describe('OverviewTab incident timeline error state', () => {
await waitFor(() => {
expect(screen.getByText('Command executed')).toBeInTheDocument();
});
expect(mockGetIncidentTimeline).toHaveBeenCalledWith('alert-1', '2026-01-01T00:00:00Z');
expect(screen.getByText('checked service health')).toBeInTheDocument();
expect(screen.getByText('systemctl status pulse')).toBeInTheDocument();
expect(screen.getByText('Active: active (running)')).toBeInTheDocument();
internal/alerts/callbacks.go
+58
View File
@@ -3,11 +3,23 @@ package alerts
import (
"strconv"
"sync"
"time"
"github.com/rcourtman/pulse-go-rewrite/internal/alerts/eventlog"
"github.com/rs/zerolog/log"
)
// LifecycleEvent is an alert-state transition independent of notification
// delivery. Consumers use this stream for durable incident history; delivery
// callbacks remain governed by activation, quiet-hours, and suppression policy.
type LifecycleEvent struct {
Type string
OccurredAt time.Time
Alert *Alert
Details map[string]string
Persisted bool
}
type callbackBus struct {
mu sync.RWMutex
@@ -17,6 +29,7 @@ type callbackBus struct {
alertForAISubs map[int]func(alert *Alert)
onResolved func(alertID string)
resolvedSubs map[int]func(alertID string)
lifecycleSubs map[int]func(event LifecycleEvent)
onAcknowledged func(alert *Alert, user string)
onUnacknowledged func(alert *Alert, user string)
@@ -32,6 +45,27 @@ func newCallbackBus() callbackBus {
alertSubs: make(map[int]func(*Alert)),
alertForAISubs: make(map[int]func(*Alert)),
resolvedSubs: make(map[int]func(string)),
lifecycleSubs: make(map[int]func(LifecycleEvent)),
}
}
func (b *callbackBus) subscribeLifecycleCallback(cb func(event LifecycleEvent)) func() {
if cb == nil {
return func() {}
}
b.mu.Lock()
if b.lifecycleSubs == nil {
b.lifecycleSubs = make(map[int]func(LifecycleEvent))
}
id := b.nextIDLocked()
b.lifecycleSubs[id] = cb
b.mu.Unlock()
return func() {
b.mu.Lock()
delete(b.lifecycleSubs, id)
b.mu.Unlock()
}
}
@@ -216,6 +250,19 @@ func (b *callbackBus) resolvedCallbacks() []func(alertID string) {
return callbacks
}
func (b *callbackBus) lifecycleCallbacks() []func(event LifecycleEvent) {
b.mu.RLock()
defer b.mu.RUnlock()
callbacks := make([]func(LifecycleEvent), 0, len(b.lifecycleSubs))
for _, cb := range b.lifecycleSubs {
if cb != nil {
callbacks = append(callbacks, cb)
}
}
return callbacks
}
func (b *callbackBus) acknowledgedCallback() func(alert *Alert, user string) {
b.mu.RLock()
cb := b.onAcknowledged
@@ -284,6 +331,13 @@ func (m *Manager) SubscribeResolvedCallback(cb func(alertID string)) func() {
return m.callbacks.subscribeResolvedCallback(cb)
}
// SubscribeLifecycleCallback registers a delivery-independent lifecycle
// consumer. It is invoked synchronously at the canonical event-emission seam;
// callbacks must not re-enter the alert manager.
func (m *Manager) SubscribeLifecycleCallback(cb func(event LifecycleEvent)) func() {
return m.callbacks.subscribeLifecycleCallback(cb)
}
// SetAcknowledgedCallback sets the callback for acknowledged alerts.
func (m *Manager) SetAcknowledgedCallback(cb func(alert *Alert, user string)) {
m.callbacks.setAcknowledgedCallback(cb)
@@ -329,6 +383,10 @@ func (m *Manager) getResolvedCallbacks() []func(alertID string) {
return m.callbacks.resolvedCallbacks()
}
func (m *Manager) getLifecycleCallbacks() []func(event LifecycleEvent) {
return m.callbacks.lifecycleCallbacks()
}
func (m *Manager) getAcknowledgedCallback() func(alert *Alert, user string) {
return m.callbacks.acknowledgedCallback()
}
internal/alerts/event_emission.go
+103 -8
View File
@@ -31,7 +31,7 @@ func (m *Manager) EnableEventLog() {
return
}
m.SetEventLog(store)
m.importLegacyHistoryIntoEventLog(store)
m.eventHistoryAuthoritative.Store(m.importLegacyHistoryIntoEventLog(store))
}
// SetEventLog installs an event log store. Passing nil disables recording.
@@ -41,6 +41,11 @@ func (m *Manager) SetEventLog(store *eventlog.Store) {
return
}
previous := m.eventLog.Swap(store)
authoritative := store != nil && m.historyManager != nil &&
!m.historyManager.StorageFileExists() &&
!m.historyManager.ImportedStorageFileExists() &&
m.historyManager.StorageLoadError() == nil
m.eventHistoryAuthoritative.Store(authoritative)
if previous != nil && previous != store {
previous.Close()
}
@@ -61,19 +66,58 @@ func (m *Manager) AlertEvents(filter eventlog.Filter) ([]eventlog.Event, error)
if store == nil {
return nil, nil
}
store.Flush()
if err := store.Flush(); err != nil {
return nil, err
}
return store.Query(filter)
}
// recordAlertEvent appends one event for an alert. alertID is the identity
// fallback for callers whose *Alert may be nil (some resolve paths); when the
// alert is present its exported ID wins. Never blocks; safe under m.mu.
func (m *Manager) recordAlertEvent(eventType string, alert *Alert, alertID, reason, message string, details map[string]string) {
// ReplayLifecycleEvents visits every durable lifecycle transition oldest
// first. It is the projection-repair seam for consumers such as incident and
// resource timelines: delivery callbacks are deliberately not involved.
func (m *Manager) ReplayLifecycleEvents(visit func(LifecycleEvent) error) error {
if m == nil || visit == nil {
return nil
}
store := m.eventLogStore()
if store == nil {
return
return nil
}
return store.WalkOldest(eventlog.Filter{Types: []string{
eventlog.TypeFired,
eventlog.TypeRefired,
eventlog.TypeResolved,
eventlog.TypeAcknowledged,
eventlog.TypeUnacknowledged,
}}, func(event eventlog.Event) error {
if len(event.Snapshot) == 0 {
return nil
}
var snapshot Alert
if err := json.Unmarshal(event.Snapshot, &snapshot); err != nil {
log.Warn().Err(err).
Int64("eventID", event.ID).
Str("alertID", event.AlertID).
Msg("skipping invalid alert lifecycle snapshot during projection replay")
return nil
}
return visit(LifecycleEvent{
Type: event.Type,
OccurredAt: event.OccurredAt,
Alert: &snapshot,
Details: cloneStringMap(event.Details),
Persisted: true,
})
})
}
// recordAlertEvent emits one canonical lifecycle transition and appends it to
// the durable event log. alertID is the identity fallback for callers whose
// *Alert may be nil (some resolve paths); when the alert is present its
// exported ID wins. Lifecycle subscribers run even when the diagnostic event
// log is unavailable, so incident history never depends on notification or
// event-log availability.
func (m *Manager) recordAlertEvent(eventType string, alert *Alert, alertID, reason, message string, details map[string]string) {
event := eventlog.Event{
OccurredAt: m.policyNow(),
Type: eventType,
@@ -99,7 +143,58 @@ func (m *Manager) recordAlertEvent(eventType string, alert *Alert, alertID, reas
if event.AlertID == "" {
return
}
store.Append(event)
persisted := false
if store := m.eventLogStore(); store != nil {
if eventCarriesAlertSnapshot(eventType) {
if err := store.AppendDurable(event); err != nil {
m.eventHistoryAuthoritative.Store(false)
log.Error().Err(err).
Str("alertID", event.AlertID).
Str("eventType", eventType).
Msg("durable alert lifecycle event write failed")
} else {
persisted = true
}
} else {
store.Append(event)
}
}
if eventCarriesAlertSnapshot(eventType) && alert != nil {
lifecycleEvent := LifecycleEvent{
Type: eventType,
OccurredAt: event.OccurredAt,
Alert: cloneAlertForOutput(alert),
Details: cloneStringMap(details),
Persisted: persisted,
}
for _, callback := range m.getLifecycleCallbacks() {
func(cb func(LifecycleEvent)) {
defer func() {
if recovered := recover(); recovered != nil {
log.Error().
Interface("panic", recovered).
Str("alertID", event.AlertID).
Str("eventType", eventType).
Msg("panic in alert lifecycle callback")
}
}()
cb(lifecycleEvent)
}(callback)
}
}
}
func cloneStringMap(input map[string]string) map[string]string {
if len(input) == 0 {
return nil
}
output := make(map[string]string, len(input))
for key, value := range input {
output[key] = value
}
return output
}
// eventCarriesAlertSnapshot reports whether an event type records the full
internal/alerts/eventlog/eventlog.go
+65 -8
View File
@@ -4,9 +4,9 @@
// "why didn't I get notified?" is answerable from durable data instead of
// being reconstructed from logs (docs/ALERT_ENGINE_EVOLUTION.md, Phase 0).
//
// The store is additive to the live alert manager: appends never block the
// evaluation path (a full buffer drops the event and counts the drop), and a
// store that fails to open degrades to a nil store whose methods are no-ops.
// Delivery-diagnostic appends are additive to the live alert manager: they do
// not block evaluation and may be dropped under sustained pressure. Lifecycle
// events use AppendDurable because alert history is projected from this store.
package eventlog
import (
@@ -105,6 +105,9 @@ type Store struct {
dropped atomic.Int64
appended atomic.Int64
written atomic.Int64
failed atomic.Int64
failureMu sync.RWMutex
lastError error
retention time.Duration
}
@@ -247,6 +250,23 @@ func (s *Store) Append(event Event) {
}
}
// AppendDurable commits one lifecycle event before returning. Lifecycle
// volume is low, and this explicit durability boundary prevents alert history
// from silently losing state when the diagnostic append buffer is saturated.
func (s *Store) AppendDurable(event Event) error {
if s == nil {
return fmt.Errorf("event log is not enabled")
}
if event.OccurredAt.IsZero() {
event.OccurredAt = time.Now()
}
if err := s.insertBatch([]Event{event}); err != nil {
s.recordWriteFailure(err)
return err
}
return nil
}
// ImportEvents writes events synchronously, bypassing the droppable append
// buffer. It exists for the one-time legacy-history migration, where losing
// an entry to a full buffer would silently lose user data.
@@ -291,7 +311,9 @@ func (s *Store) writeLoop() {
}
}
if err := s.insertBatch(batch); err != nil {
s.recordWriteFailure(err)
log.Error().Err(err).Int("events", len(batch)).Msg("alert event log write failed")
continue
}
s.written.Add(int64(len(batch)))
case <-pruneTicker.C:
@@ -302,7 +324,9 @@ func (s *Store) writeLoop() {
select {
case event := <-s.events:
if err := s.insertBatch([]Event{event}); err != nil {
s.recordWriteFailure(err)
log.Error().Err(err).Msg("alert event log final drain write failed")
continue
}
s.written.Add(1)
default:
@@ -313,6 +337,29 @@ func (s *Store) writeLoop() {
}
}
func (s *Store) recordWriteFailure(err error) {
if s == nil || err == nil {
return
}
s.failureMu.Lock()
s.lastError = err
s.failureMu.Unlock()
s.failed.Add(1)
}
func (s *Store) writeError() error {
if s == nil || s.failed.Load() == 0 {
return nil
}
s.failureMu.RLock()
err := s.lastError
s.failureMu.RUnlock()
if err == nil {
return fmt.Errorf("alert event log write failed")
}
return fmt.Errorf("alert event log write failed: %w", err)
}
func (s *Store) insertBatch(batch []Event) error {
tx, err := s.db.Begin()
if err != nil {
@@ -401,18 +448,28 @@ func (s *Store) pruneOld() {
}
}
// Flush blocks until every event appended before the call has been written
// (or a short deadline passes). It exists for tests and API reads that must
// observe just-appended events.
func (s *Store) Flush() {
// Flush blocks until every diagnostic event appended before the call has been
// written. It reports failed writes and timeouts rather than claiming a lost
// batch was successfully flushed.
func (s *Store) Flush() error {
if s == nil {
return
return nil
}
target := s.appended.Load()
deadline := time.Now().Add(5 * time.Second)
for s.written.Load() < target && time.Now().Before(deadline) {
if err := s.writeError(); err != nil {
return err
}
time.Sleep(time.Millisecond)
}
if err := s.writeError(); err != nil {
return err
}
if written := s.written.Load(); written < target {
return fmt.Errorf("alert event log flush timed out: wrote %d of %d events", written, target)
}
return nil
}
// Query returns matching events, newest first.
internal/alerts/eventlog/eventlog_test.go
+16
View File
@@ -240,3 +240,19 @@ func TestOpenRequiresExistingDirectory(t *testing.T) {
}
store.Close()
}
func TestDurableAppendAndFlushReportDatabaseFailure(t *testing.T) {
store := newTestStore(t)
if err := store.db.Close(); err != nil {
t.Fatalf("close database: %v", err)
}
if err := store.AppendDurable(Event{Type: TypeFired, AlertID: "durable-failure"}); err == nil {
t.Fatal("AppendDurable reported success after the database closed")
}
store.Append(Event{Type: TypeNotificationSuppressed, AlertID: "async-failure"})
if err := store.Flush(); err == nil {
t.Fatal("Flush reported success after the async batch failed")
}
}
internal/alerts/history.go
+39 -7
View File
@@ -295,6 +295,18 @@ func (hm *HistoryManager) StorageFileExists() bool {
return false
}
// ImportedStorageFileExists reports whether a retired JSON source remains as
// a recovery copy. It is replayed idempotently if events.db is recreated and
// remains the read fallback when the event store is unavailable.
func (hm *HistoryManager) ImportedStorageFileExists() bool {
for _, path := range []string{hm.historyFile + ".imported", hm.backupFile + ".imported"} {
if _, err := os.Stat(path); err == nil {
return true
}
}
return false
}
// StorageLoadError reports why a present legacy source could not be loaded.
// Migration and periodic persistence must leave that source untouched.
func (hm *HistoryManager) StorageLoadError() error {
@@ -493,26 +505,46 @@ func (hm *HistoryManager) loadHistory() error {
// Try backup file
var backupErr error
loadedRetired := false
data, backupErr = readLimitedRegularFile(hm.backupFile, maxAlertHistoryFileSizeBytes)
if backupErr != nil {
if os.IsNotExist(backupErr) && os.IsNotExist(mainErr) {
// Both files don't exist - this is normal on first startup.
log.Debug().Msg("No alert history files found, starting fresh")
return nil
// Retired JSON remains a recovery source if events.db is lost or
// unavailable. Prefer the primary retired leaf, then its backup.
var importedErr error
data, importedErr = readLimitedRegularFile(hm.historyFile+".imported", maxAlertHistoryFileSizeBytes)
if importedErr != nil {
data, importedErr = readLimitedRegularFile(hm.backupFile+".imported", maxAlertHistoryFileSizeBytes)
}
if importedErr != nil {
if os.IsNotExist(importedErr) {
log.Debug().Msg("No alert history files found, starting fresh")
return nil
}
return fmt.Errorf("failed to read retired alert history recovery source: %w", importedErr)
}
hm.storageRetired = true
loadedRetired = true
log.Info().Msg("loaded retired alert history recovery source")
}
if os.IsPermission(backupErr) {
if !loadedRetired && os.IsPermission(backupErr) {
log.Warn().
Err(backupErr).
Str("file", hm.backupFile).
Msg("Permission denied reading backup history file - check file ownership")
}
if !os.IsNotExist(mainErr) {
if loadedRetired {
backupErr = nil
} else if !os.IsNotExist(mainErr) {
return fmt.Errorf("failed to read history file %q (%v); failed to read history backup file %q: %w",
hm.historyFile, mainErr, hm.backupFile, backupErr)
} else {
return fmt.Errorf("failed to read history backup file %q: %w", hm.backupFile, backupErr)
}
return fmt.Errorf("failed to read history backup file %q: %w", hm.backupFile, backupErr)
}
log.Info().Msg("loaded alert history from backup file")
if !loadedRetired {
log.Info().Msg("loaded alert history from backup file")
}
}
var history []HistoryEntry
internal/alerts/history_migration.go
+19 -12
View File
@@ -21,17 +21,21 @@ import (
// history reads fall back to the in-memory entries either way. A retirement
// failure also retries safely because history_imported events are inserted by
// immutable event identity.
func (m *Manager) importLegacyHistoryIntoEventLog(store *eventlog.Store) {
if m == nil || store == nil || m.historyManager == nil {
return
func (m *Manager) importLegacyHistoryIntoEventLog(store *eventlog.Store) (authoritative bool) {
if m != nil {
defer func() { m.eventHistoryAuthoritative.Store(authoritative) }()
}
if !m.historyManager.StorageFileExists() {
return
if m == nil || store == nil || m.historyManager == nil {
return false
}
hasLegacySource := m.historyManager.StorageFileExists()
if !hasLegacySource && !m.historyManager.ImportedStorageFileExists() {
return true
}
if err := m.historyManager.StorageLoadError(); err != nil {
log.Error().Err(err).
Msg("legacy alert history import deferred; source could not be loaded and remains untouched")
return
return false
}
entries := m.historyManager.SnapshotEntries()
@@ -42,13 +46,13 @@ func (m *Manager) importLegacyHistoryIntoEventLog(store *eventlog.Store) {
if exported == nil || exported.ID == "" {
log.Error().Int("entry", i).
Msg("legacy alert history import deferred; entry has no alert identity")
return
return false
}
snapshot, err := json.Marshal(exported)
if err != nil {
log.Error().Err(err).Int("entry", i).
Msg("legacy alert history import deferred; entry snapshot could not be encoded")
return
return false
}
occurredAt := entry.Timestamp
if entry.Alert.LastSeen.After(occurredAt) {
@@ -72,14 +76,17 @@ func (m *Manager) importLegacyHistoryIntoEventLog(store *eventlog.Store) {
log.Error().Err(err).
Int("entries", len(events)).
Msg("legacy alert history import failed; JSON history stays authoritative until the next attempt")
return
return false
}
}
if err := m.historyManager.RetireStorage(); err != nil {
log.Error().Err(err).Msg("legacy alert history files could not be retired after import")
return
if hasLegacySource {
if err := m.historyManager.RetireStorage(); err != nil {
log.Error().Err(err).Msg("legacy alert history files could not be retired after import")
return false
}
}
log.Info().
Int("entries", len(events)).
Msg("legacy alert history imported into the event log; JSON history files retired")
return true
}
internal/alerts/history_migration_test.go
+35
View File
@@ -82,6 +82,41 @@ func TestLegacyHistoryImportRetiresJSONAndServesEntries(t *testing.T) {
}
}
func TestRetiredLegacyHistoryRecoversRecreatedEventDatabase(t *testing.T) {
dataDir := t.TempDir()
seed := NewManagerWithDataDir(dataDir)
past := time.Now().Add(-2 * time.Hour).UTC()
seed.historyManager.AddAlert(Alert{
ID: "legacy-recovery::metric-threshold:cpu",
Type: "cpu",
ResourceID: "legacy-recovery",
StartTime: past,
LastSeen: past.Add(time.Minute),
})
if err := seed.historyManager.saveHistory(); err != nil {
t.Fatalf("persist legacy history: %v", err)
}
seed.EnableEventLog()
seed.SetEventLog(nil)
seed.Stop()
eventsPath := filepath.Join(dataDir, "alerts", "events.db")
if err := os.Remove(eventsPath); err != nil {
t.Fatalf("remove event database: %v", err)
}
for _, suffix := range []string{"-shm", "-wal"} {
_ = os.Remove(eventsPath + suffix)
}
restarted := NewManagerWithDataDir(dataDir)
t.Cleanup(restarted.Stop)
restarted.EnableEventLog()
entries := restarted.GetAlertHistory(0)
if len(entries) != 1 || entries[0].ResourceID != "legacy-recovery" {
t.Fatalf("recreated event database did not recover retired history: %#v", entries)
}
}
func TestLegacyHistoryImportRetryAfterRetirementFailureIsIdempotent(t *testing.T) {
dataDir := t.TempDir()
m := NewManagerWithDataDir(dataDir)
internal/alerts/history_projection.go
+1 -2
View File
@@ -34,10 +34,9 @@ func historyOccurrenceKey(alertID string, snapshot *Alert) string {
// fall back to the JSON-backed history manager.
func (m *Manager) AlertHistoryFromEvents(since time.Time, limit int) ([]Alert, bool) {
store := m.eventLogStore()
if store == nil {
if store == nil || !m.eventHistoryAuthoritative.Load() {
return nil, false
}
store.Flush()
// Fold oldest to newest through the store's bounded-page walker. The
// ordinary Query API intentionally caps responses at 1,000 rows, which is
internal/alerts/manager.go
+4
View File
@@ -108,6 +108,10 @@ type Manager struct {
// Append-only alert event log (transitions + notification decisions).
// Nil until EnableEventLog/SetEventLog; recording is then a no-op.
eventLog atomic.Pointer[eventlog.Store]
// eventHistoryAuthoritative becomes true only after legacy JSON history is
// absent or has been durably imported. Reads keep using JSON while migration
// is incomplete or the event store reports a write failure.
eventHistoryAuthoritative atomic.Bool
// Shadow-mode reducer feed (Phase 1 capstone). Nil until
// EnableShadowFeed; all access is under m.mu.
internal/alerts/notification_activation_boundary_test.go
+13
View File
@@ -13,9 +13,13 @@ func TestNotificationActivationDoesNotSuppressDetectionOrActiveReadModel(t *test
t.Run(string(activationState), func(t *testing.T) {
m := newTestManager(t)
delivered := make(chan *Alert, 1)
lifecycle := make(chan LifecycleEvent, 1)
m.SetAlertCallback(func(alert *Alert) {
delivered <- alert
})
m.SubscribeLifecycleCallback(func(event LifecycleEvent) {
lifecycle <- event
})
m.mu.Lock()
m.config.Enabled = true
@@ -53,6 +57,15 @@ func TestNotificationActivationDoesNotSuppressDetectionOrActiveReadModel(t *test
)
}
select {
case event := <-lifecycle:
if event.Type != "fired" || event.Alert == nil || event.Alert.ID != active[0].ID {
t.Fatalf("activation state %q emitted unexpected lifecycle event: %#v", activationState, event)
}
default:
t.Fatalf("activation state %q suppressed the alert lifecycle event", activationState)
}
select {
case alert := <-delivered:
t.Fatalf(
internal/monitoring/canonical_guardrails_test.go
+6 -1
View File
@@ -11,6 +11,7 @@ import (
"github.com/rcourtman/pulse-go-rewrite/internal/ai/memory"
"github.com/rcourtman/pulse-go-rewrite/internal/alerts"
"github.com/rcourtman/pulse-go-rewrite/internal/alerts/eventlog"
"github.com/rcourtman/pulse-go-rewrite/internal/config"
"github.com/rcourtman/pulse-go-rewrite/internal/mock"
"github.com/rcourtman/pulse-go-rewrite/internal/models"
@@ -2344,7 +2345,11 @@ func TestAlertLifecycleCanonicalChangesRemainWritable(t *testing.T) {
StartTime: startedAt,
}
monitor.handleAlertFired(alert)
monitor.handleAlertLifecycleEvent(alerts.LifecycleEvent{
Type: eventlog.TypeFired,
OccurredAt: startedAt,
Alert: alert,
})
changes, err := store.GetRecentChanges("vm-guardrail", time.Time{}, 10)
if err != nil {
internal/monitoring/monitor.go
+3 -6
View File
@@ -1977,6 +1977,7 @@ func (m *Monitor) Start(ctx context.Context, wsHub *websocket.Hub) {
}
// Set up alert callbacks
m.alertManager.SubscribeLifecycleCallback(m.handleAlertLifecycleEvent)
m.alertManager.SetAlertCallback(func(alert *alerts.Alert) {
m.handleAlertFired(alert)
})
@@ -1993,15 +1994,11 @@ func (m *Monitor) Start(ctx context.Context, wsHub *websocket.Hub) {
// Don't broadcast full state here - it causes a cascade with many guests.
// The frontend will get the updated alerts through the regular broadcast ticker.
})
m.alertManager.SetAcknowledgedCallback(func(alert *alerts.Alert, user string) {
m.handleAlertAcknowledged(alert, user)
})
m.alertManager.SetUnacknowledgedCallback(func(alert *alerts.Alert, user string) {
m.handleAlertUnacknowledged(alert, user)
})
m.alertManager.SetEscalateCallback(func(alert *alerts.Alert, level int) {
m.handleAlertEscalated(wsHub, alert, level)
})
m.replayAlertLifecycleProjections()
m.reconcileActiveAlertTimelines()
// Create separate tickers for polling and broadcasting using the configured cadence
internal/monitoring/monitor_alert_handling_test.go
+187 -3
View File
@@ -9,7 +9,9 @@ import (
"testing"
"time"
"github.com/rcourtman/pulse-go-rewrite/internal/ai/memory"
"github.com/rcourtman/pulse-go-rewrite/internal/alerts"
"github.com/rcourtman/pulse-go-rewrite/internal/alerts/eventlog"
"github.com/rcourtman/pulse-go-rewrite/internal/models"
"github.com/rcourtman/pulse-go-rewrite/internal/notifications"
unifiedresources "github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources"
@@ -67,6 +69,20 @@ func TestMonitor_HandleAlertFired_RecoversFromPushCallbackPanic(t *testing.T) {
m.handleAlertFired(&alerts.Alert{ID: "alert-push-panic"})
}
func TestDeliveryCallbackDoesNotDuplicateCanonicalAITrigger(t *testing.T) {
called := make(chan struct{}, 1)
monitor := &Monitor{
alertTriggeredAICallback: func(*alerts.Alert) { called <- struct{}{} },
}
monitor.handleAlertFired(&alerts.Alert{ID: "single-ai-trigger"})
select {
case <-called:
t.Fatal("delivery callback invoked AI analysis; the manager-owned unconditional AI callback is the sole trigger")
case <-time.After(100 * time.Millisecond):
}
}
func TestMonitor_HandleAlertLifecycle_WritesCanonicalChanges(t *testing.T) {
store := unifiedresources.NewMemoryStore()
m := &Monitor{
@@ -91,9 +107,19 @@ func TestMonitor_HandleAlertLifecycle_WritesCanonicalChanges(t *testing.T) {
},
}
m.handleAlertFired(alert)
m.handleAlertAcknowledged(alert, "admin")
m.handleAlertUnacknowledged(alert, "admin")
m.handleAlertLifecycleEvent(alerts.LifecycleEvent{Type: eventlog.TypeFired, OccurredAt: startedAt, Alert: alert})
m.handleAlertLifecycleEvent(alerts.LifecycleEvent{
Type: eventlog.TypeAcknowledged,
OccurredAt: ackAt,
Alert: alert,
Details: map[string]string{"user": "admin"},
})
m.handleAlertLifecycleEvent(alerts.LifecycleEvent{
Type: eventlog.TypeUnacknowledged,
OccurredAt: ackAt.Add(time.Minute),
Alert: alert,
Details: map[string]string{"user": "admin"},
})
changes, err := store.GetRecentChanges("vm-1", time.Time{}, 10)
if err != nil {
@@ -123,6 +149,164 @@ func TestMonitor_HandleAlertLifecycle_WritesCanonicalChanges(t *testing.T) {
}
}
func TestPausedDeliveryStillBuildsTimelineThroughRealAlertLifecycle(t *testing.T) {
manager := alerts.NewManagerWithDataDir(t.TempDir())
t.Cleanup(manager.Stop)
config := manager.GetConfig()
config.Enabled = true
config.ActivationState = alerts.ActivationPending
config.TimeThresholds = map[string]int{}
config.SuppressionWindow = 0
manager.UpdateConfig(config)
resourceStore := unifiedresources.NewMemoryStore()
incidentStore := memory.NewIncidentStore(memory.IncidentStoreConfig{})
monitor := &Monitor{
alertManager: manager,
incidentStore: incidentStore,
resourceStore: unifiedresources.NewMonitorAdapter(unifiedresources.NewRegistry(resourceStore)),
}
incidentStore.SetResourceTimelineStore(monitor.resourceStore.(memory.IncidentTimelineStore))
manager.SubscribeLifecycleCallback(monitor.handleAlertLifecycleEvent)
delivered := make(chan *alerts.Alert, 1)
manager.SetAlertCallback(func(alert *alerts.Alert) { delivered <- alert })
vm := models.VM{ID: "paused-vm", Name: "Paused VM", Node: "node-1", Instance: "pve-1", Status: "stopped"}
manager.CheckGuest(vm, vm.Instance)
manager.CheckGuest(vm, vm.Instance)
active := manager.GetActiveAlerts()
if len(active) != 1 {
t.Fatalf("active alerts = %d, want 1", len(active))
}
timeline := incidentStore.GetTimelineByAlertAt(active[0].ID, active[0].StartTime)
if timeline == nil || len(timeline.Events) == 0 || timeline.Events[0].Type != memory.IncidentEventAlertFired {
t.Fatalf("paused-delivery lifecycle did not produce an incident timeline: %#v", timeline)
}
select {
case alert := <-delivered:
t.Fatalf("pending-review alert reached delivery callback: %s", alert.ID)
default:
}
}
func TestActiveTimelineReconciliationIsIdempotent(t *testing.T) {
manager := alerts.NewManagerWithDataDir(t.TempDir())
t.Cleanup(manager.Stop)
config := manager.GetConfig()
config.Enabled = true
config.ActivationState = alerts.ActivationPending
config.TimeThresholds = map[string]int{}
manager.UpdateConfig(config)
vm := models.VM{ID: "restored-vm", Name: "Restored VM", Node: "node-1", Instance: "pve-1", Status: "stopped"}
manager.CheckGuest(vm, vm.Instance)
manager.CheckGuest(vm, vm.Instance)
resourceStore := unifiedresources.NewMemoryStore()
incidentStore := memory.NewIncidentStore(memory.IncidentStoreConfig{})
monitor := &Monitor{
alertManager: manager,
incidentStore: incidentStore,
resourceStore: unifiedresources.NewMonitorAdapter(unifiedresources.NewRegistry(resourceStore)),
}
incidentStore.SetResourceTimelineStore(monitor.resourceStore.(memory.IncidentTimelineStore))
monitor.reconcileActiveAlertTimelines()
monitor.reconcileActiveAlertTimelines()
active := manager.GetActiveAlerts()
if len(active) != 1 {
t.Fatalf("active alerts = %d, want 1", len(active))
}
timeline := incidentStore.GetTimelineByAlertAt(active[0].ID, active[0].StartTime)
if timeline == nil || len(timeline.Events) != 1 || timeline.Events[0].Type != memory.IncidentEventAlertFired {
t.Fatalf("reconciled timeline = %#v, want one fired event", timeline)
}
changes, err := resourceStore.GetRecentChanges(active[0].ResourceID, time.Time{}, 10)
if err != nil {
t.Fatalf("GetRecentChanges: %v", err)
}
if len(changes) != 1 {
t.Fatalf("reconciliation wrote %d canonical changes, want 1", len(changes))
}
}
func TestLifecycleReplayRepairsResolvedIncidentTimeline(t *testing.T) {
manager := alerts.NewManagerWithDataDir(t.TempDir(), alerts.WithoutPersistedAlertRestore())
t.Cleanup(manager.Stop)
manager.EnableEventLog()
config := manager.GetConfig()
config.Enabled = true
config.ActivationState = alerts.ActivationPending
config.TimeThresholds = map[string]int{}
config.SuppressionWindow = 0
manager.UpdateConfig(config)
vm := models.VM{ID: "historical-vm", Name: "Historical VM", Node: "node-1", Instance: "pve-1", Status: "stopped"}
manager.CheckGuest(vm, vm.Instance)
manager.CheckGuest(vm, vm.Instance)
active := manager.GetActiveAlerts()
if len(active) != 1 {
t.Fatalf("active alerts = %d, want 1", len(active))
}
alertID, startedAt := active[0].ID, active[0].StartTime
vm.Status = "running"
manager.CheckGuest(vm, vm.Instance)
if active := manager.GetActiveAlerts(); len(active) != 0 {
t.Fatalf("active alerts after recovery = %d, want 0", len(active))
}
resourceStore := unifiedresources.NewMemoryStore()
incidentStore := memory.NewIncidentStore(memory.IncidentStoreConfig{})
monitor := &Monitor{
alertManager: manager,
incidentStore: incidentStore,
resourceStore: unifiedresources.NewMonitorAdapter(unifiedresources.NewRegistry(resourceStore)),
}
incidentStore.SetResourceTimelineStore(monitor.resourceStore.(memory.IncidentTimelineStore))
monitor.replayAlertLifecycleProjections()
monitor.replayAlertLifecycleProjections()
timeline := incidentStore.GetTimelineByAlertAt(alertID, startedAt)
if timeline == nil || len(timeline.Events) != 2 {
t.Fatalf("replayed timeline = %#v, want fired and resolved events", timeline)
}
if timeline.Events[0].Type != memory.IncidentEventAlertFired || timeline.Events[1].Type != memory.IncidentEventAlertResolved {
t.Fatalf("replayed event types = %#v, want fired then resolved", timeline.Events)
}
changes, err := resourceStore.GetRecentChanges(vm.ID, time.Time{}, 10)
if err != nil {
t.Fatalf("GetRecentChanges: %v", err)
}
if len(changes) != 2 {
t.Fatalf("replay wrote %d canonical changes, want 2", len(changes))
}
}
func TestSystemAlertTimelineUsesCanonicalPulseResource(t *testing.T) {
resourceStore := unifiedresources.NewMemoryStore()
incidentStore := memory.NewIncidentStore(memory.IncidentStoreConfig{})
monitor := &Monitor{
incidentStore: incidentStore,
resourceStore: unifiedresources.NewMonitorAdapter(unifiedresources.NewRegistry(resourceStore)),
}
incidentStore.SetResourceTimelineStore(monitor.resourceStore.(memory.IncidentTimelineStore))
alert := &alerts.Alert{
ID: alerts.SystemAlertID("event-store-health"),
Type: "event-store-health",
Level: alerts.AlertLevelCritical,
ResourceName: alerts.SystemAlertResourceName,
Message: "Alert history storage is unavailable",
StartTime: time.Now().UTC(),
}
monitor.handleAlertLifecycleEvent(alerts.LifecycleEvent{Type: eventlog.TypeFired, OccurredAt: alert.StartTime, Alert: alert})
timeline := incidentStore.GetTimelineByAlertAt(alert.ID, alert.StartTime)
if timeline == nil || timeline.ResourceID != "pulse-system" || len(timeline.Events) != 1 {
t.Fatalf("system alert timeline = %#v", timeline)
}
}
func TestMonitor_HandleAlertResolved_Detailed_Extra(t *testing.T) {
// 1. With Hub and NotificationMgr and Resolve Notify ON
hub := websocket.NewHub(nil)
internal/monitoring/monitor_alerts.go
+88 -52
View File
@@ -5,8 +5,10 @@ import (
"strings"
"time"
"github.com/google/uuid"
"github.com/rcourtman/pulse-go-rewrite/internal/ai/memory"
"github.com/rcourtman/pulse-go-rewrite/internal/alerts"
"github.com/rcourtman/pulse-go-rewrite/internal/alerts/eventlog"
"github.com/rcourtman/pulse-go-rewrite/internal/mock"
"github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources"
"github.com/rcourtman/pulse-go-rewrite/internal/websocket"
@@ -122,23 +124,6 @@ func (m *Monitor) handleAlertFired(alert *alerts.Alert) {
}()
}
if m.incidentStore != nil {
m.incidentStore.RecordAlertFired(alert)
}
m.recordAlertTimelineChange(alert, unifiedresources.ChangeAlertFired, alert.StartTime, "")
// Trigger AI analysis if callback is configured
if m.alertTriggeredAICallback != nil {
// Run in goroutine to avoid blocking the monitor loop
go func() {
defer func() {
if r := recover(); r != nil {
log.Error().Interface("panic", r).Msg("panic in AI alert callback")
}
}()
m.alertTriggeredAICallback(alert)
}()
}
}
func (m *Monitor) handleAlertResolved(alertID string) {
@@ -148,21 +133,6 @@ func (m *Monitor) handleAlertResolved(alertID string) {
m.wsHub.BroadcastAlertResolvedToTenant(m.GetOrgID(), alertID)
}
// Always record incident timeline, regardless of notification suppression.
// This ensures we have a complete history even during quiet hours.
if m.incidentStore != nil {
resolvedAlert = m.alertManager.GetResolvedAlert(alertID)
if resolvedAlert != nil && resolvedAlert.Alert != nil {
m.incidentStore.RecordAlertResolved(resolvedAlert.Alert, resolvedAlert.ResolvedTime)
}
}
if resolvedAlert == nil && m.alertManager != nil {
resolvedAlert = m.alertManager.GetResolvedAlert(alertID)
}
if resolvedAlert != nil && resolvedAlert.Alert != nil {
m.recordAlertTimelineChange(resolvedAlert.Alert, unifiedresources.ChangeAlertResolved, resolvedAlert.ResolvedTime, "")
}
// Always trigger AI callback, regardless of notification suppression.
if m.alertResolvedAICallback != nil {
if resolvedAlert == nil {
@@ -248,29 +218,88 @@ func (m *Monitor) handleAlertEscalated(hub *websocket.Hub, alert *alerts.Alert,
m.broadcastEscalatedAlert(hub, alert)
}
func (m *Monitor) handleAlertAcknowledged(alert *alerts.Alert, user string) {
if m.incidentStore == nil || alert == nil {
if alert == nil {
return
}
} else {
m.incidentStore.RecordAlertAcknowledged(alert, user)
}
occurredAt := time.Now()
if alert.AckTime != nil {
occurredAt = *alert.AckTime
}
m.recordAlertTimelineChange(alert, unifiedresources.ChangeAlertAcknowledged, occurredAt, user)
}
func (m *Monitor) handleAlertUnacknowledged(alert *alerts.Alert, user string) {
if alert == nil {
func (m *Monitor) handleAlertLifecycleEvent(event alerts.LifecycleEvent) {
alert := event.Alert
if m == nil || alert == nil {
return
}
if m.incidentStore != nil {
m.incidentStore.RecordAlertUnacknowledged(alert, user)
actor := event.Details["user"]
timelineAlert := alert
if alerts.IsSystemAlert(alert) && strings.TrimSpace(alert.ResourceID) == "" {
timelineAlert = alert.Clone()
timelineAlert.ResourceID = "pulse-system"
}
switch event.Type {
case eventlog.TypeFired, eventlog.TypeRefired:
if m.incidentStore != nil {
m.incidentStore.RecordAlertFired(timelineAlert)
}
occurredAt := event.OccurredAt
if event.Type == eventlog.TypeFired && !alert.StartTime.IsZero() {
occurredAt = alert.StartTime
}
m.recordAlertTimelineChange(timelineAlert, unifiedresources.ChangeAlertFired, occurredAt, "")
case eventlog.TypeAcknowledged:
if m.incidentStore != nil {
m.incidentStore.RecordAlertAcknowledged(timelineAlert, actor)
}
occurredAt := event.OccurredAt
if alert.AckTime != nil && !alert.AckTime.IsZero() {
occurredAt = *alert.AckTime
}
m.recordAlertTimelineChange(timelineAlert, unifiedresources.ChangeAlertAcknowledged, occurredAt, actor)
case eventlog.TypeUnacknowledged:
if m.incidentStore != nil {
m.incidentStore.RecordAlertUnacknowledged(timelineAlert, actor)
}
m.recordAlertTimelineChange(timelineAlert, unifiedresources.ChangeAlertUnacknowledged, event.OccurredAt, actor)
case eventlog.TypeResolved:
if m.incidentStore != nil {
m.incidentStore.RecordAlertResolved(timelineAlert, event.OccurredAt)
}
m.recordAlertTimelineChange(timelineAlert, unifiedresources.ChangeAlertResolved, event.OccurredAt, "")
}
}
func (m *Monitor) replayAlertLifecycleProjections() {
if m == nil || m.alertManager == nil {
return
}
if err := m.alertManager.ReplayLifecycleEvents(func(event alerts.LifecycleEvent) error {
m.handleAlertLifecycleEvent(event)
return nil
}); err != nil {
log.Error().Err(err).Msg("failed to replay canonical alert lifecycle projections")
}
}
func (m *Monitor) reconcileActiveAlertTimelines() {
if m == nil || m.alertManager == nil || m.incidentStore == nil {
return
}
activeAlerts := m.alertManager.GetActiveAlerts()
for i := range activeAlerts {
alert := &activeAlerts[i]
timeline := m.incidentStore.GetTimelineByAlertAt(alert.ID, alert.StartTime)
if timeline != nil {
hasFired := false
for _, event := range timeline.Events {
if event.Type == memory.IncidentEventAlertFired {
hasFired = true
break
}
}
if hasFired {
continue
}
}
m.handleAlertLifecycleEvent(alerts.LifecycleEvent{
Type: eventlog.TypeFired,
OccurredAt: alert.StartTime,
Alert: alert,
})
}
m.recordAlertTimelineChange(alert, unifiedresources.ChangeAlertUnacknowledged, time.Now(), user)
}
func (m *Monitor) recordAlertTimelineChange(alert *alerts.Alert, kind unifiedresources.ChangeKind, occurredAt time.Time, actor string) {
@@ -294,6 +323,13 @@ func (m *Monitor) recordAlertTimelineChange(alert *alerts.Alert, kind unifiedres
if change == nil {
return
}
change.ID = uuid.NewSHA1(uuid.NameSpaceOID, []byte(strings.Join([]string{
"pulse-alert-lifecycle-v1",
strings.TrimSpace(alert.ID),
strings.TrimSpace(alert.ResourceID),
string(kind),
occurredAt.UTC().Format(time.RFC3339Nano),
}, "\x00"))).String()
if err := recorder.RecordChange(*change); err != nil {
log.Warn().
Err(err).
internal/monitoring/monitor_full_coverage_test.go
-2
View File
@@ -402,8 +402,6 @@ func TestMonitor_AlertCallbacks(t *testing.T) {
// No panic = pass
m.handleAlertResolved("test-alert")
m.handleAlertAcknowledged(alert, "user")
m.handleAlertUnacknowledged(alert, "user")
}
type mockResourceStore struct{}
tests/integration/tests/94-alert-history-real-backend.spec.ts
+144 -18
View File
@@ -6,7 +6,11 @@ import { expect, test as base } from '@playwright/test';
import { restartManagedLocalBackend } from '../scripts/managed-local-backend.mjs';
import { readRuntimeState } from '../scripts/runtime-state.mjs';
import { apiRequest, createAuthenticatedStorageState, ensureAuthenticated } from './helpers';
import {
apiRequest,
createAuthenticatedStorageState,
ensureAuthenticated,
} from './helpers';
const __dirname = path.dirname(fileURLToPath(import.meta.url));
@@ -15,7 +19,8 @@ type WorkerFixtures = {
};
const test = base.extend<{}, WorkerFixtures>({
storageState: async ({ authStorageStatePath }, use) => use(authStorageStatePath),
storageState: async ({ authStorageStatePath }, use) =>
use(authStorageStatePath),
authStorageStatePath: [
async ({ browser }, use, workerInfo) => {
const storageStatePath = path.resolve(
@@ -58,20 +63,31 @@ type HistoryAlert = {
lastSeen: string;
};
async function readHistory(page: import('@playwright/test').Page): Promise<HistoryAlert[]> {
async function readHistory(
page: import('@playwright/test').Page,
): Promise<HistoryAlert[]> {
const response = await apiRequest(page, '/api/alerts/history?limit=0');
expect(response.ok(), `history API returned ${response.status()}`).toBeTruthy();
expect(
response.ok(),
`history API returned ${response.status()}`,
).toBeTruthy();
return (await response.json()) as HistoryAlert[];
}
test.describe('Real backend alert history qualification', () => {
test.skip(!qualificationEnabled, 'Run through npm run test:alerts:qualification');
test.skip(
!qualificationEnabled,
'Run through npm run test:alerts:qualification',
);
test.setTimeout(240_000);
test('imports once, renders exact fields, survives restart, and preserves clear tombstones', async ({
page,
}, testInfo) => {
test.skip(testInfo.project.name !== 'chromium', 'The isolated qualification runner owns Chromium');
test.skip(
testInfo.project.name !== 'chromium',
'The isolated qualification runner owns Chromium',
);
const runtimeState = await readRuntimeState();
expect(runtimeState?.managedLocalBackend).toBe(true);
@@ -81,26 +97,46 @@ test.describe('Real backend alert history qualification', () => {
const importedPath = path.join(alertsDir, 'alert-history.json.imported');
const sourcePath = path.join(alertsDir, 'alert-history.json');
expect(fs.existsSync(importedPath), 'legacy history source must retire after import').toBe(true);
expect(fs.existsSync(sourcePath), 'retired legacy history source must leave the load path').toBe(false);
expect(
fs.existsSync(importedPath),
'legacy history source must retire after import',
).toBe(true);
expect(
fs.existsSync(sourcePath),
'retired legacy history source must leave the load path',
).toBe(false);
expect(fs.readFileSync(runtimeState!.logPath, 'utf8')).toContain(
'legacy alert history imported into the event log; JSON history files retired',
);
await page.goto('/alerts/history', { waitUntil: 'domcontentloaded' });
await expect(page.getByRole('heading', { name: 'Alert History' })).toBeVisible();
const legacyRow = page.locator('tr').filter({ hasText: 'Legacy Qualification VM' }).first();
await expect(legacyRow).toContainText('Legacy history import rendered from SQLite');
await expect(
page.getByRole('heading', { name: 'Alert History' }),
).toBeVisible();
const legacyRow = page
.locator('tr')
.filter({ hasText: 'Legacy Qualification VM' })
.first();
await expect(legacyRow).toContainText(
'Legacy history import rendered from SQLite',
);
await expect(legacyRow).toContainText('CPU');
await expect(legacyRow).toContainText('Qualification Node');
await expect(legacyRow).toContainText('30m');
await expect(
page.locator('table.alert-history-responsive-table').getByText('Active Overlay Node').first(),
page
.locator('table.alert-history-responsive-table')
.getByText('Active Overlay Node')
.first(),
).toBeVisible();
let history = await readHistory(page);
expect(history.filter((alert) => alert.id === LEGACY_ALERT_ID)).toHaveLength(1);
expect(history.filter((alert) => alert.id === ACTIVE_ALERT_ID)).toHaveLength(1);
expect(
history.filter((alert) => alert.id === LEGACY_ALERT_ID),
).toHaveLength(1);
expect(
history.filter((alert) => alert.id === ACTIVE_ALERT_ID),
).toHaveLength(1);
const importedAlert = history.find((alert) => alert.id === LEGACY_ALERT_ID);
expect(importedAlert).toMatchObject({
resourceName: 'Legacy Qualification VM',
@@ -118,11 +154,93 @@ test.describe('Real backend alert history qualification', () => {
'Seed timestamps may be constructed a few milliseconds apart',
).toBeLessThanOrEqual(100);
const activeAlert = history.find((alert) => alert.id === ACTIVE_ALERT_ID);
expect(activeAlert).toBeTruthy();
const timelineResponse = await apiRequest(
page,
`/api/alerts/incidents?alertIdentifier=${encodeURIComponent(ACTIVE_ALERT_ID)}&started_at=${encodeURIComponent(activeAlert!.startTime)}`,
);
expect(
timelineResponse.ok(),
`timeline API returned ${timelineResponse.status()}`,
).toBeTruthy();
const timeline = (await timelineResponse.json()) as {
alertIdentifier: string;
events: Array<{ type: string; summary: string }>;
} | null;
expect(
timeline,
'restored active alert must have a reconciled timeline',
).not.toBeNull();
expect(timeline?.alertIdentifier).toBe(ACTIVE_ALERT_ID);
expect(timeline?.events.some((event) => event.type === 'alert_fired')).toBe(
true,
);
await page.goto('/alerts/overview', { waitUntil: 'domcontentloaded' });
await expect(page.getByText('Active Overlay Node').first()).toBeVisible();
const overviewTimelineRequests: string[] = [];
const recordOverviewTimelineRequest = (
request: import('@playwright/test').Request,
) => {
if (new URL(request.url()).pathname === '/api/alerts/incidents') {
overviewTimelineRequests.push(request.url());
}
};
page.on('request', recordOverviewTimelineRequest);
const overviewTimelineResponsePromise = page.waitForResponse((response) => {
const url = new URL(response.url());
return (
url.pathname === '/api/alerts/incidents' &&
url.searchParams.has('alertIdentifier')
);
});
await page.getByRole('button', { name: 'Timeline' }).click();
const overviewTimelineResponse = await overviewTimelineResponsePromise;
const overviewTimelineURL = new URL(overviewTimelineResponse.url());
expect(overviewTimelineURL.searchParams.get('alertIdentifier')).toBe(
ACTIVE_ALERT_ID,
);
expect(overviewTimelineURL.searchParams.get('started_at')).toBe(
activeAlert!.startTime,
);
const overviewTimeline = (await overviewTimelineResponse.json()) as {
events?: Array<{ type: string }>;
} | null;
expect(overviewTimeline).not.toBeNull();
expect(
overviewTimeline?.events?.some((event) => event.type === 'alert_fired'),
).toBe(true);
await page.waitForTimeout(250);
page.off('request', recordOverviewTimelineRequest);
expect(overviewTimelineRequests).toHaveLength(1);
await expect(page.getByText(/Alert triggered: memory/)).toBeVisible();
await expect(page.getByText('No incident timeline available.')).toHaveCount(
0,
);
await page.setViewportSize({ width: 390, height: 844 });
await page.reload({ waitUntil: 'domcontentloaded' });
await expect(page.getByText('Active Overlay Node').first()).toBeVisible();
await page.getByRole('button', { name: 'Timeline' }).click();
await expect(page.getByText(/Alert triggered: memory/)).toBeVisible();
const narrowOverflow = await page.evaluate(
() =>
document.documentElement.scrollWidth -
document.documentElement.clientWidth,
);
expect(narrowOverflow).toBeLessThanOrEqual(1);
await page.setViewportSize({ width: 1280, height: 720 });
await page.goto('/alerts/history', { waitUntil: 'domcontentloaded' });
await restartManagedLocalBackend();
await ensureAuthenticated(page);
await page.goto('/alerts/history', { waitUntil: 'domcontentloaded' });
await expect(
page.locator('table.alert-history-responsive-table').getByText('Legacy Qualification VM'),
page
.locator('table.alert-history-responsive-table')
.getByText('Legacy Qualification VM'),
).toBeVisible();
history = await readHistory(page);
expect(
@@ -134,7 +252,10 @@ test.describe('Real backend alert history qualification', () => {
await page.getByRole('button', { name: 'Clear All History' }).click();
await expect(page.getByText('Legacy Qualification VM')).toHaveCount(0);
await expect(
page.locator('table.alert-history-responsive-table').getByText('Active Overlay Node').first(),
page
.locator('table.alert-history-responsive-table')
.getByText('Active Overlay Node')
.first(),
).toBeVisible();
history = await readHistory(page);
expect(history.map((alert) => alert.id)).not.toContain(LEGACY_ALERT_ID);
@@ -145,11 +266,16 @@ test.describe('Real backend alert history qualification', () => {
await page.goto('/alerts/history', { waitUntil: 'domcontentloaded' });
await expect(page.getByText('Legacy Qualification VM')).toHaveCount(0);
await expect(
page.locator('table.alert-history-responsive-table').getByText('Active Overlay Node').first(),
page
.locator('table.alert-history-responsive-table')
.getByText('Active Overlay Node')
.first(),
).toBeVisible();
history = await readHistory(page);
expect(history.map((alert) => alert.id)).not.toContain(LEGACY_ALERT_ID);
expect(history.filter((alert) => alert.id === ACTIVE_ALERT_ID)).toHaveLength(1);
expect(
history.filter((alert) => alert.id === ACTIVE_ALERT_ID),
).toHaveLength(1);
await testInfo.attach('real-backend-alert-history-final.json', {
body: Buffer.from(JSON.stringify(history, null, 2)),