diff --git a/docs/release-control/v6/internal/subsystems/alerts.md b/docs/release-control/v6/internal/subsystems/alerts.md
index 63bab6578..73a1e11a9 100644
--- a/docs/release-control/v6/internal/subsystems/alerts.md
+++ b/docs/release-control/v6/internal/subsystems/alerts.md
@@ -991,14 +991,21 @@ projection once per refresh and may render its current delivery outcome on an
unacknowledged active-alert card. Missing or failed diagnosis reads degrade to
no delivery line; they must not hide alert truth or trigger per-card requests.
`internal/alerts/eventlog/` and `internal/alerts/event_emission.go` own the
-additive append-only alert event record. Persistent managers enable a
+canonical append-only alert event record. Persistent managers enable a
SQLite-backed store under the alerts data directory; ephemeral managers record
nothing unless a store is installed explicitly. Resolution, acknowledgement,
unacknowledgement, escalation, flapping detection, dispatch, quiet-hours
deferral, and suppression append immutable events without changing lifecycle
-or delivery behavior. The write path is non-blocking and fail-open for alert
-evaluation: a full buffer counts and drops its event, while an unavailable
-store disables recording. Events retain for 90 days and prune hourly. Fired and
+or delivery behavior. Snapshot-bearing lifecycle transitions commit
+synchronously before downstream lifecycle projections run; they never share
+the droppable diagnostic buffer because alert history is reconstructed from
+them. High-volume notification decisions remain non-blocking and fail-open for
+alert evaluation: a full diagnostic buffer counts and drops that delivery
+evidence. Failed asynchronous batches do not advance the successful-write
+counter, and `Flush` must report the failure or timeout rather than falsely
+claiming the batch landed. An unavailable or failed lifecycle store degrades
+history reads to the recovery model without hiding live active-alert truth.
+Events retain for 90 days and prune hourly. Fired and
refired lifecycle events come only from the reducer core's explicit activation
events: canonical lifecycle reactivation maps `EventRefired` separately, while
shared metric activation records one fired event when its pending incident
@@ -1020,8 +1027,11 @@ Lifecycle events now carry a full alert snapshot and
`internal/alerts/history_projection.go` can fold those snapshots into one
history row per occurrence. Existing pre-snapshot SQLite stores upgrade in
place, and rows without snapshots remain readable but cannot contribute to the
-projection. With an event log enabled, it is the alert-history read authority;
-managers without one retain the in-memory JSON-history model as their fallback.
+projection. The event log becomes the alert-history read authority only after
+legacy history is absent or its complete import and retirement succeeds.
+Managers without an authoritative event store, and stores that report a
+lifecycle write failure, retain the in-memory JSON-history model as their
+fail-soft read fallback.
`internal/alerts/history_projection_parity_test.go` characterizes the event-log
projection against that fallback for active, resolved, acknowledged,
multi-resource, and repeated occurrences. Live active-alert state overlays the
@@ -1038,12 +1048,26 @@ failure leaves JSON history authoritative for the next attempt. An unreadable
or malformed source, or any entry that cannot produce an identified complete
snapshot, blocks both retirement and further JSON writes so startup cannot
silently replace recoverable user history with an empty file. Successful
-retirement stops further JSON writes. Clearing history appends a
+retirement stops further JSON writes. Retired `.imported` leaves remain bounded
+recovery sources: startup loads and idempotently replays them if `events.db` is
+recreated, so loss or quarantine of the database cannot turn a successful
+migration into silent empty history. Clearing history appends a
`history_cleared` tombstone rather than deleting log rows; the projection
ignores earlier lifecycle events but overlays still-active alerts as current
state. Unified-incident and system-alert activation must emit snapshot-bearing
fired events so their projected history does not begin at acknowledgement or
resolution.
+`SubscribeLifecycleCallback` is the delivery-independent projection seam for
+that canonical stream. It runs for lifecycle transitions regardless of
+activation, quiet hours, grouping, rate limits, or destination state; delivery
+callbacks remain policy-controlled consumers. Monitoring uses this seam to
+materialize canonical resource-history breadcrumbs and incident shells. On
+startup it replays durable lifecycle transitions oldest first, then
+idempotently reconciles restored active alerts that predate the event store.
+This repairs resolved as well as active incident timelines without replaying
+any delivery side effect, and ensures an alert already visible in Overview
+cannot keep returning an unavailable timeline merely because its notification
+was held.
The same dispatch policy owns firing-notification evidence on active alerts:
any alert that passes notification suppression and enters the fired callback
fan-out must carry `LastNotified` before the callback clone is emitted. Resolved
diff --git a/docs/release-control/v6/internal/subsystems/frontend-primitives.md b/docs/release-control/v6/internal/subsystems/frontend-primitives.md
index 83934539d..b989c0a8d 100644
--- a/docs/release-control/v6/internal/subsystems/frontend-primitives.md
+++ b/docs/release-control/v6/internal/subsystems/frontend-primitives.md
@@ -5687,6 +5687,13 @@ while `frontend-modern/src/features/alerts/OverviewTab.tsx` and
surface composition. Future incident timeline fetch, note-save, or expansion
control flow should extend that feature hook rather than forking back into
either tab surface.
+Every surface invoking that owner must pass the canonical alert identifier and
+the occurrence start time as distinct arguments; a row key is local UI state,
+not a substitute for either API identity field.
+The shared `IncidentTimelinePanel` consumes loading, error, timeline, draft,
+and save state as accessors so an asynchronous result remains reactive across
+Overview, desktop History, and mobile History instead of freezing the values
+present when the panel first expands.
Overview alert runtime now follows that same shell-versus-runtime split. The
shell stays in `frontend-modern/src/features/alerts/OverviewTab.tsx`, while
`frontend-modern/src/features/alerts/useAlertOverviewState.ts` owns derived
diff --git a/docs/release-control/v6/internal/subsystems/monitoring.md b/docs/release-control/v6/internal/subsystems/monitoring.md
index 333941e7f..2c60cbc19 100644
--- a/docs/release-control/v6/internal/subsystems/monitoring.md
+++ b/docs/release-control/v6/internal/subsystems/monitoring.md
@@ -2071,14 +2071,23 @@ rebuilds and supplemental ingest into the unified-resource timeline. That means
monitoring no longer just materializes state snapshots for consumers; it also
emits durable `ResourceChange` history through the shared resource store so
live monitoring updates and historical inspection stay aligned.
-That same ownership now includes alert-lifecycle facts emitted by monitoring.
-When an alert is fired, acknowledged, unacknowledged, or resolved for a
-canonical resource, the monitoring runtime must write the corresponding durable
-resource-history event into the unified-resource change store instead of
-leaving that lifecycle only inside alert-scoped incident memory. Incident
-timelines may still project those breadcrumbs for operator flow, but the
-durable backend truth for alert lifecycle now lives on the canonical resource
-timeline.
+That same ownership now includes the resource-history projection of canonical
+alert-lifecycle facts. The alerts-owned SQLite event log is the lifecycle source
+of truth; monitoring consumes its delivery-independent lifecycle seam and
+materializes fired, acknowledged, unacknowledged, and resolved breadcrumbs in
+the unified-resource change store. Projection IDs derive deterministically from
+alert identity, canonical resource, transition kind, and occurrence time, so
+restart repair and duplicate consumer delivery are idempotent. Notification
+activation, quiet hours, grouping, throttling, and destination health may never
+gate this projection. Incident timelines project those breadcrumbs for
+operator flow, while the resource timeline remains the durable resource-scoped
+index rather than a second alert lifecycle authority.
+Monitoring must install the lifecycle consumer, replay durable lifecycle
+events oldest first, and then reconcile restored active alerts that predate
+the event store. Replay and reconciliation create only missing projections,
+including a stable `pulse-system` timeline identity for system alerts whose
+public alert payload intentionally has no monitored-resource link; neither path
+may duplicate a resource change or invoke notification delivery.
The monitor-owned incident store wiring must therefore attach the canonical
resource timeline reader whenever the unified monitor adapter is present, so
operator alert timelines and AI incident context project those lifecycle events
diff --git a/frontend-modern/browser-verification.json b/frontend-modern/browser-verification.json
index c271341c8..56d7cd838 100644
--- a/frontend-modern/browser-verification.json
+++ b/frontend-modern/browser-verification.json
@@ -1,27 +1,25 @@
{
"version": 1,
- "base_sha": "3d91709ebad0332858e4615d72daf8e424ab42a8",
- "verified_at": "2026-08-27T12:19:53Z",
+ "base_sha": "1e75423116984e67ca92a8904f913e5e6a194324",
+ "verified_at": "2026-08-27T13:41:54Z",
"result": "passed",
"changed_paths": [
- "frontend-modern/src/components/Alerts/ThresholdsTableAgentsTab.tsx",
- "frontend-modern/src/components/Alerts/ThresholdsTableSMARTDefaultsCard.tsx",
- "frontend-modern/src/features/alerts/alertsConfigurationModel.ts",
- "frontend-modern/src/types/alerts.ts",
- "frontend-modern/src/utils/alertThresholdDefaults.ts"
+ "frontend-modern/src/components/Alerts/IncidentTimelinePanel.tsx",
+ "frontend-modern/src/features/alerts/AlertHistoryMobileList.tsx",
+ "frontend-modern/src/features/alerts/AlertHistoryTableAlertRow.tsx",
+ "frontend-modern/src/features/alerts/AlertOverviewAlertCard.tsx"
],
"content_sha256": {
- "frontend-modern/src/components/Alerts/ThresholdsTableAgentsTab.tsx": "89646984deb982699d3359953ab85a2ceae15830465ccf20b5603f5a426e3193",
- "frontend-modern/src/components/Alerts/ThresholdsTableSMARTDefaultsCard.tsx": "c1bd1947dbcbab5806e11fa905271039131cebb0a736642e73fd3652f63e6b66",
- "frontend-modern/src/features/alerts/alertsConfigurationModel.ts": "2100da1f350cbbea386f1ae669a655d213ea710f1636eb94fe80c6d47d29313d",
- "frontend-modern/src/types/alerts.ts": "cc8b88ec5d8de87dbb69116114e9cd889347972b59dd81bec081096ab791c693",
- "frontend-modern/src/utils/alertThresholdDefaults.ts": "6ea4a3f99c8bd27a4fcca98f67c866d06a0f5d3e3211550a58447688cb61352a"
+ "frontend-modern/src/components/Alerts/IncidentTimelinePanel.tsx": "2f32d34e7bfedd2e052cf20075a4eb711d114ee5715c13a108f6b3666f29ad91",
+ "frontend-modern/src/features/alerts/AlertHistoryMobileList.tsx": "d5179ae59605d318f522483b13e5925fd7a6890a066b129eaf769d42b334d2d4",
+ "frontend-modern/src/features/alerts/AlertHistoryTableAlertRow.tsx": "cc763cdadc2f48e47437cfddf61ebbd5723e74a6a89f3640da013b63f6598cdb",
+ "frontend-modern/src/features/alerts/AlertOverviewAlertCard.tsx": "fc2822b803ab1c250811e8a374c7cba94ae1f4b08130acf48ba6b98cea3f0a5c"
},
- "routes": ["/alerts/thresholds/systems"],
+ "routes": ["/alerts/overview", "/alerts/history"],
"viewports": [
{
- "width": 1440,
- "height": 1000
+ "width": 1280,
+ "height": 720
},
{
"width": 390,
@@ -29,17 +27,18 @@
}
],
"states": [
- "Machines thresholds loaded with the backward-compatible SMART factory rules",
- "failed-health disabled with pending sectors set to 4, life warning set to 15 percent, and NVMe spare critical disabled at 0",
- "edited SMART rules persisted after Save Changes and a full route reload",
- "factory SMART values restored and saved from the narrow layout",
- "complete single-column SMART card inspected through the app scroll shell at 390x844"
+ "restored active alert visible while notification delivery is pending review",
+ "canonical incident API returns the occurrence-specific fired event for the active alert",
+ "expanded Overview timeline renders incident status, fired event, filters, timestamp, and note editor",
+ "legacy JSON history imported once into SQLite and retained across backend restart",
+ "history clear tombstone removes retired history while preserving the active alert overlay"
],
"interactions": [
- "opened Alerts, selected Thresholds, and selected the Machines platform tab at 1440x1000",
- "toggled failed-health, edited counter and percentage rules including an explicit 0 disable value, and saved through the shared dirty-state bar",
- "reloaded /alerts/thresholds/systems and confirmed every edited value persisted",
- "scrolled the complete SMART rules card at 390x844, confirmed scroll width matched client width, restored factory values, and saved from the narrow layout",
- "confirmed labels, help text, controls, percentage suffixes, focusable save actions, and the browser console remained free of errors"
+ "opened Alerts Overview at 1280x720 and expanded the active alert timeline",
+ "confirmed the Timeline control requested the canonical alert identifier and occurrence start time exactly once",
+ "confirmed the rendered fired event replaced the unavailable-timeline state while delivery remained paused",
+ "reloaded Alerts Overview at 390x844, expanded the same timeline, and confirmed no horizontal overflow",
+ "restarted the managed real backend twice and confirmed history import, clear tombstone, and active-overlay continuity",
+ "confirmed the five-test alert operator and real-backend qualification suite passed with no product errors"
]
}
diff --git a/frontend-modern/src/components/Alerts/IncidentTimelinePanel.tsx b/frontend-modern/src/components/Alerts/IncidentTimelinePanel.tsx
index a4c853c5f..28b16f810 100644
--- a/frontend-modern/src/components/Alerts/IncidentTimelinePanel.tsx
+++ b/frontend-modern/src/components/Alerts/IncidentTimelinePanel.tsx
@@ -30,31 +30,31 @@ import {
} from '@/utils/alertIncidentPresentation';
export interface IncidentTimelinePanelProps {
- timeline?: Incident | null;
- loading: boolean;
- error: boolean;
+ timeline: Accessor;
+ loading: Accessor;
+ error: Accessor;
filters: Accessor>;
setFilters: (next: Set) => void;
filterVariant: AlertIncidentEventFilterVariant;
eventCardVariant: 'surface' | 'alt';
- noteDraft: string;
+ noteDraft: Accessor;
onNoteDraftChange: (value: string) => void;
- noteSaving: boolean;
+ noteSaving: Accessor;
onSaveNote: () => void;
onRetry: () => void;
}
export function IncidentTimelinePanel(props: IncidentTimelinePanelProps) {
- const timeline = () => props.timeline;
+ const timeline = props.timeline;
const events = createMemo(() => timeline()?.events || []);
const filteredEvents = createMemo(() => filterIncidentEvents(events(), props.filters()));
return (
<>
-
+
{getAlertTimelineLoadingState().text}
-
+
{(loadedTimeline) => (
@@ -121,25 +121,25 @@ export function IncidentTimelinePanel(props: IncidentTimelinePanelProps) {
textareaBaseClass={getAlertIncidentNoteTextareaClass()}
rows={2}
placeholder={getAlertTimelineNotePlaceholder()}
- value={props.noteDraft}
+ value={props.noteDraft()}
onInput={(event) => props.onNoteDraftChange(event.currentTarget.value)}
/>
props.onSaveNote()}
>
- {getAlertTimelineSaveNoteLabel(props.noteSaving)}
+ {getAlertTimelineSaveNoteLabel(props.noteSaving())}
)}
-
+
{getAlertTimelineUnavailableState().text}
}
>
diff --git a/frontend-modern/src/components/Alerts/__tests__/IncidentTimelinePanel.test.tsx b/frontend-modern/src/components/Alerts/__tests__/IncidentTimelinePanel.test.tsx
index 5689b79ab..e5ee72f5a 100644
--- a/frontend-modern/src/components/Alerts/__tests__/IncidentTimelinePanel.test.tsx
+++ b/frontend-modern/src/components/Alerts/__tests__/IncidentTimelinePanel.test.tsx
@@ -55,16 +55,16 @@ describe('IncidentTimelinePanel', () => {
render(() => (
state().loading}
+ error={() => state().error}
+ timeline={() => state().timeline}
filters={filters}
setFilters={setFilters}
filterVariant="panel"
eventCardVariant="alt"
- noteDraft=""
+ noteDraft={() => ''}
onNoteDraftChange={vi.fn()}
- noteSaving={false}
+ noteSaving={() => false}
onSaveNote={vi.fn()}
onRetry={vi.fn()}
/>
@@ -88,16 +88,16 @@ describe('IncidentTimelinePanel', () => {
render(() => (
false}
+ error={() => false}
+ timeline={() => makeTimeline()}
filters={filters}
setFilters={setFilters}
filterVariant="compact"
eventCardVariant="surface"
- noteDraft="operator note"
+ noteDraft={() => 'operator note'}
onNoteDraftChange={handleNoteDraftChange}
- noteSaving={false}
+ noteSaving={() => false}
onSaveNote={handleSave}
onRetry={vi.fn()}
/>
@@ -127,16 +127,16 @@ describe('IncidentTimelinePanel', () => {
render(() => (
false}
+ error={() => false}
+ timeline={() => makeTimeline()}
filters={filters}
setFilters={setFilters}
filterVariant="compact"
eventCardVariant="surface"
- noteDraft=""
+ noteDraft={() => ''}
onNoteDraftChange={vi.fn()}
- noteSaving={false}
+ noteSaving={() => false}
onSaveNote={vi.fn()}
onRetry={vi.fn()}
/>
diff --git a/frontend-modern/src/features/alerts/AlertHistoryMobileList.tsx b/frontend-modern/src/features/alerts/AlertHistoryMobileList.tsx
index bc6bed864..3bccbf129 100644
--- a/frontend-modern/src/features/alerts/AlertHistoryMobileList.tsx
+++ b/frontend-modern/src/features/alerts/AlertHistoryMobileList.tsx
@@ -125,16 +125,16 @@ export function AlertHistoryMobileList(props: AlertHistoryMobileListProps) {
>
props.state.incidentLoading()[rowKey()]}
+ error={() => props.state.incidentErrors()[rowKey()]}
+ timeline={() => props.state.incidentTimelines()[rowKey()]}
filters={props.state.historyIncidentEventFilters}
setFilters={props.state.setHistoryIncidentEventFilters}
filterVariant="compact"
eventCardVariant="surface"
- noteDraft={props.state.incidentNoteDrafts()[rowKey()] || ''}
+ noteDraft={() => props.state.incidentNoteDrafts()[rowKey()] || ''}
onNoteDraftChange={(value) => props.state.setIncidentNoteDraft(rowKey(), value)}
- noteSaving={props.state.incidentNoteSaving().has(rowKey())}
+ noteSaving={() => props.state.incidentNoteSaving().has(rowKey())}
onSaveNote={() => {
void props.state.saveIncidentNote(rowKey(), alert.id, alert.startTime);
}}
diff --git a/frontend-modern/src/features/alerts/AlertHistoryTableAlertRow.tsx b/frontend-modern/src/features/alerts/AlertHistoryTableAlertRow.tsx
index 2d9ec8df7..9187526f5 100644
--- a/frontend-modern/src/features/alerts/AlertHistoryTableAlertRow.tsx
+++ b/frontend-modern/src/features/alerts/AlertHistoryTableAlertRow.tsx
@@ -96,16 +96,16 @@ export function AlertHistoryTableAlertRow(props: AlertHistoryTableAlertRowProps)
props.state.incidentLoading()[rowKey()]}
+ error={() => props.state.incidentErrors()[rowKey()]}
+ timeline={() => props.state.incidentTimelines()[rowKey()]}
filters={props.state.historyIncidentEventFilters}
setFilters={props.state.setHistoryIncidentEventFilters}
filterVariant="compact"
eventCardVariant="surface"
- noteDraft={props.state.incidentNoteDrafts()[rowKey()] || ''}
+ noteDraft={() => props.state.incidentNoteDrafts()[rowKey()] || ''}
onNoteDraftChange={(value) => props.state.setIncidentNoteDraft(rowKey(), value)}
- noteSaving={props.state.incidentNoteSaving().has(rowKey())}
+ noteSaving={() => props.state.incidentNoteSaving().has(rowKey())}
onSaveNote={() => {
void props.state.saveIncidentNote(rowKey(), props.alert.id, props.alert.startTime);
}}
diff --git a/frontend-modern/src/features/alerts/AlertOverviewAlertCard.tsx b/frontend-modern/src/features/alerts/AlertOverviewAlertCard.tsx
index 3d7575c5e..f6e4a2d26 100644
--- a/frontend-modern/src/features/alerts/AlertOverviewAlertCard.tsx
+++ b/frontend-modern/src/features/alerts/AlertOverviewAlertCard.tsx
@@ -191,7 +191,11 @@ export function AlertOverviewAlertCard(props: AlertOverviewAlertCardProps) {
{
- void props.timelineState.toggleIncidentTimeline(alertKey(), props.alert.startTime);
+ void props.timelineState.toggleIncidentTimeline(
+ alertKey(),
+ alertKey(),
+ props.alert.startTime,
+ );
}}
>
{getAlertOverviewTimelineActionLabel(
@@ -222,23 +226,31 @@ export function AlertOverviewAlertCard(props: AlertOverviewAlertCardProps) {
props.timelineState.incidentLoading()[alertKey()]}
+ error={() => props.timelineState.incidentErrors()[alertKey()]}
+ timeline={() => props.timelineState.incidentTimelines()[alertKey()]}
filters={props.timelineState.eventFilters}
setFilters={props.timelineState.setEventFilters}
filterVariant="panel"
eventCardVariant="alt"
- noteDraft={props.timelineState.incidentNoteDrafts()[alertKey()] || ''}
+ noteDraft={() => props.timelineState.incidentNoteDrafts()[alertKey()] || ''}
onNoteDraftChange={(value) =>
props.timelineState.setIncidentNoteDraft(alertKey(), value)
}
- noteSaving={props.timelineState.incidentNoteSaving().has(alertKey())}
+ noteSaving={() => props.timelineState.incidentNoteSaving().has(alertKey())}
onSaveNote={() => {
- void props.timelineState.saveIncidentNote(alertKey(), props.alert.startTime);
+ void props.timelineState.saveIncidentNote(
+ alertKey(),
+ alertKey(),
+ props.alert.startTime,
+ );
}}
onRetry={() => {
- void props.timelineState.loadIncidentTimeline(alertKey(), props.alert.startTime);
+ void props.timelineState.loadIncidentTimeline(
+ alertKey(),
+ alertKey(),
+ props.alert.startTime,
+ );
}}
/>
diff --git a/frontend-modern/src/features/alerts/__tests__/OverviewTab.timelineerror.test.tsx b/frontend-modern/src/features/alerts/__tests__/OverviewTab.timelineerror.test.tsx
index 92090a4f6..68612a127 100644
--- a/frontend-modern/src/features/alerts/__tests__/OverviewTab.timelineerror.test.tsx
+++ b/frontend-modern/src/features/alerts/__tests__/OverviewTab.timelineerror.test.tsx
@@ -167,6 +167,7 @@ describe('OverviewTab incident timeline error state', () => {
await waitFor(() => {
expect(screen.getByText('Command executed')).toBeInTheDocument();
});
+ expect(mockGetIncidentTimeline).toHaveBeenCalledWith('alert-1', '2026-01-01T00:00:00Z');
expect(screen.getByText('checked service health')).toBeInTheDocument();
expect(screen.getByText('systemctl status pulse')).toBeInTheDocument();
expect(screen.getByText('Active: active (running)')).toBeInTheDocument();
diff --git a/internal/alerts/callbacks.go b/internal/alerts/callbacks.go
index 5f78b6afa..f88356ce6 100644
--- a/internal/alerts/callbacks.go
+++ b/internal/alerts/callbacks.go
@@ -3,11 +3,23 @@ package alerts
import (
"strconv"
"sync"
+ "time"
"github.com/rcourtman/pulse-go-rewrite/internal/alerts/eventlog"
"github.com/rs/zerolog/log"
)
+// LifecycleEvent is an alert-state transition independent of notification
+// delivery. Consumers use this stream for durable incident history; delivery
+// callbacks remain governed by activation, quiet-hours, and suppression policy.
+type LifecycleEvent struct {
+ Type string
+ OccurredAt time.Time
+ Alert *Alert
+ Details map[string]string
+ Persisted bool
+}
+
type callbackBus struct {
mu sync.RWMutex
@@ -17,6 +29,7 @@ type callbackBus struct {
alertForAISubs map[int]func(alert *Alert)
onResolved func(alertID string)
resolvedSubs map[int]func(alertID string)
+ lifecycleSubs map[int]func(event LifecycleEvent)
onAcknowledged func(alert *Alert, user string)
onUnacknowledged func(alert *Alert, user string)
@@ -32,6 +45,27 @@ func newCallbackBus() callbackBus {
alertSubs: make(map[int]func(*Alert)),
alertForAISubs: make(map[int]func(*Alert)),
resolvedSubs: make(map[int]func(string)),
+ lifecycleSubs: make(map[int]func(LifecycleEvent)),
+ }
+}
+
+func (b *callbackBus) subscribeLifecycleCallback(cb func(event LifecycleEvent)) func() {
+ if cb == nil {
+ return func() {}
+ }
+
+ b.mu.Lock()
+ if b.lifecycleSubs == nil {
+ b.lifecycleSubs = make(map[int]func(LifecycleEvent))
+ }
+ id := b.nextIDLocked()
+ b.lifecycleSubs[id] = cb
+ b.mu.Unlock()
+
+ return func() {
+ b.mu.Lock()
+ delete(b.lifecycleSubs, id)
+ b.mu.Unlock()
}
}
@@ -216,6 +250,19 @@ func (b *callbackBus) resolvedCallbacks() []func(alertID string) {
return callbacks
}
+func (b *callbackBus) lifecycleCallbacks() []func(event LifecycleEvent) {
+ b.mu.RLock()
+ defer b.mu.RUnlock()
+
+ callbacks := make([]func(LifecycleEvent), 0, len(b.lifecycleSubs))
+ for _, cb := range b.lifecycleSubs {
+ if cb != nil {
+ callbacks = append(callbacks, cb)
+ }
+ }
+ return callbacks
+}
+
func (b *callbackBus) acknowledgedCallback() func(alert *Alert, user string) {
b.mu.RLock()
cb := b.onAcknowledged
@@ -284,6 +331,13 @@ func (m *Manager) SubscribeResolvedCallback(cb func(alertID string)) func() {
return m.callbacks.subscribeResolvedCallback(cb)
}
+// SubscribeLifecycleCallback registers a delivery-independent lifecycle
+// consumer. It is invoked synchronously at the canonical event-emission seam;
+// callbacks must not re-enter the alert manager.
+func (m *Manager) SubscribeLifecycleCallback(cb func(event LifecycleEvent)) func() {
+ return m.callbacks.subscribeLifecycleCallback(cb)
+}
+
// SetAcknowledgedCallback sets the callback for acknowledged alerts.
func (m *Manager) SetAcknowledgedCallback(cb func(alert *Alert, user string)) {
m.callbacks.setAcknowledgedCallback(cb)
@@ -329,6 +383,10 @@ func (m *Manager) getResolvedCallbacks() []func(alertID string) {
return m.callbacks.resolvedCallbacks()
}
+func (m *Manager) getLifecycleCallbacks() []func(event LifecycleEvent) {
+ return m.callbacks.lifecycleCallbacks()
+}
+
func (m *Manager) getAcknowledgedCallback() func(alert *Alert, user string) {
return m.callbacks.acknowledgedCallback()
}
diff --git a/internal/alerts/event_emission.go b/internal/alerts/event_emission.go
index 9901866f7..aef6a2d16 100644
--- a/internal/alerts/event_emission.go
+++ b/internal/alerts/event_emission.go
@@ -31,7 +31,7 @@ func (m *Manager) EnableEventLog() {
return
}
m.SetEventLog(store)
- m.importLegacyHistoryIntoEventLog(store)
+ m.eventHistoryAuthoritative.Store(m.importLegacyHistoryIntoEventLog(store))
}
// SetEventLog installs an event log store. Passing nil disables recording.
@@ -41,6 +41,11 @@ func (m *Manager) SetEventLog(store *eventlog.Store) {
return
}
previous := m.eventLog.Swap(store)
+ authoritative := store != nil && m.historyManager != nil &&
+ !m.historyManager.StorageFileExists() &&
+ !m.historyManager.ImportedStorageFileExists() &&
+ m.historyManager.StorageLoadError() == nil
+ m.eventHistoryAuthoritative.Store(authoritative)
if previous != nil && previous != store {
previous.Close()
}
@@ -61,19 +66,58 @@ func (m *Manager) AlertEvents(filter eventlog.Filter) ([]eventlog.Event, error)
if store == nil {
return nil, nil
}
- store.Flush()
+ if err := store.Flush(); err != nil {
+ return nil, err
+ }
return store.Query(filter)
}
-// recordAlertEvent appends one event for an alert. alertID is the identity
-// fallback for callers whose *Alert may be nil (some resolve paths); when the
-// alert is present its exported ID wins. Never blocks; safe under m.mu.
-func (m *Manager) recordAlertEvent(eventType string, alert *Alert, alertID, reason, message string, details map[string]string) {
+// ReplayLifecycleEvents visits every durable lifecycle transition oldest
+// first. It is the projection-repair seam for consumers such as incident and
+// resource timelines: delivery callbacks are deliberately not involved.
+func (m *Manager) ReplayLifecycleEvents(visit func(LifecycleEvent) error) error {
+ if m == nil || visit == nil {
+ return nil
+ }
store := m.eventLogStore()
if store == nil {
- return
+ return nil
}
+ return store.WalkOldest(eventlog.Filter{Types: []string{
+ eventlog.TypeFired,
+ eventlog.TypeRefired,
+ eventlog.TypeResolved,
+ eventlog.TypeAcknowledged,
+ eventlog.TypeUnacknowledged,
+ }}, func(event eventlog.Event) error {
+ if len(event.Snapshot) == 0 {
+ return nil
+ }
+ var snapshot Alert
+ if err := json.Unmarshal(event.Snapshot, &snapshot); err != nil {
+ log.Warn().Err(err).
+ Int64("eventID", event.ID).
+ Str("alertID", event.AlertID).
+ Msg("skipping invalid alert lifecycle snapshot during projection replay")
+ return nil
+ }
+ return visit(LifecycleEvent{
+ Type: event.Type,
+ OccurredAt: event.OccurredAt,
+ Alert: &snapshot,
+ Details: cloneStringMap(event.Details),
+ Persisted: true,
+ })
+ })
+}
+// recordAlertEvent emits one canonical lifecycle transition and appends it to
+// the durable event log. alertID is the identity fallback for callers whose
+// *Alert may be nil (some resolve paths); when the alert is present its
+// exported ID wins. Lifecycle subscribers run even when the diagnostic event
+// log is unavailable, so incident history never depends on notification or
+// event-log availability.
+func (m *Manager) recordAlertEvent(eventType string, alert *Alert, alertID, reason, message string, details map[string]string) {
event := eventlog.Event{
OccurredAt: m.policyNow(),
Type: eventType,
@@ -99,7 +143,58 @@ func (m *Manager) recordAlertEvent(eventType string, alert *Alert, alertID, reas
if event.AlertID == "" {
return
}
- store.Append(event)
+
+ persisted := false
+ if store := m.eventLogStore(); store != nil {
+ if eventCarriesAlertSnapshot(eventType) {
+ if err := store.AppendDurable(event); err != nil {
+ m.eventHistoryAuthoritative.Store(false)
+ log.Error().Err(err).
+ Str("alertID", event.AlertID).
+ Str("eventType", eventType).
+ Msg("durable alert lifecycle event write failed")
+ } else {
+ persisted = true
+ }
+ } else {
+ store.Append(event)
+ }
+ }
+
+ if eventCarriesAlertSnapshot(eventType) && alert != nil {
+ lifecycleEvent := LifecycleEvent{
+ Type: eventType,
+ OccurredAt: event.OccurredAt,
+ Alert: cloneAlertForOutput(alert),
+ Details: cloneStringMap(details),
+ Persisted: persisted,
+ }
+ for _, callback := range m.getLifecycleCallbacks() {
+ func(cb func(LifecycleEvent)) {
+ defer func() {
+ if recovered := recover(); recovered != nil {
+ log.Error().
+ Interface("panic", recovered).
+ Str("alertID", event.AlertID).
+ Str("eventType", eventType).
+ Msg("panic in alert lifecycle callback")
+ }
+ }()
+ cb(lifecycleEvent)
+ }(callback)
+ }
+ }
+}
+
+func cloneStringMap(input map[string]string) map[string]string {
+ if len(input) == 0 {
+ return nil
+ }
+ output := make(map[string]string, len(input))
+ for key, value := range input {
+ output[key] = value
+ }
+ return output
}
// eventCarriesAlertSnapshot reports whether an event type records the full
diff --git a/internal/alerts/eventlog/eventlog.go b/internal/alerts/eventlog/eventlog.go
index eb3cb3e8e..fe5bec420 100644
--- a/internal/alerts/eventlog/eventlog.go
+++ b/internal/alerts/eventlog/eventlog.go
@@ -4,9 +4,9 @@
// "why didn't I get notified?" is answerable from durable data instead of
// being reconstructed from logs (docs/ALERT_ENGINE_EVOLUTION.md, Phase 0).
//
-// The store is additive to the live alert manager: appends never block the
-// evaluation path (a full buffer drops the event and counts the drop), and a
-// store that fails to open degrades to a nil store whose methods are no-ops.
+// Delivery-diagnostic appends are additive to the live alert manager: they do
+// not block evaluation and may be dropped under sustained pressure. Lifecycle
+// events use AppendDurable because alert history is projected from this store.
package eventlog
import (
@@ -105,6 +105,9 @@ type Store struct {
dropped atomic.Int64
appended atomic.Int64
written atomic.Int64
+ failed atomic.Int64
+ failureMu sync.RWMutex
+ lastError error
retention time.Duration
}
@@ -247,6 +250,23 @@ func (s *Store) Append(event Event) {
}
}
+// AppendDurable commits one lifecycle event before returning. Lifecycle
+// volume is low, and this explicit durability boundary prevents alert history
+// from silently losing state when the diagnostic append buffer is saturated.
+func (s *Store) AppendDurable(event Event) error {
+ if s == nil {
+ return fmt.Errorf("event log is not enabled")
+ }
+ if event.OccurredAt.IsZero() {
+ event.OccurredAt = time.Now()
+ }
+ if err := s.insertBatch([]Event{event}); err != nil {
+ s.recordWriteFailure(err)
+ return err
+ }
+ return nil
+}
+
// ImportEvents writes events synchronously, bypassing the droppable append
// buffer. It exists for the one-time legacy-history migration, where losing
// an entry to a full buffer would silently lose user data.
@@ -291,7 +311,9 @@ func (s *Store) writeLoop() {
}
}
if err := s.insertBatch(batch); err != nil {
+ s.recordWriteFailure(err)
log.Error().Err(err).Int("events", len(batch)).Msg("alert event log write failed")
+ continue
}
s.written.Add(int64(len(batch)))
case <-pruneTicker.C:
@@ -302,7 +324,9 @@ func (s *Store) writeLoop() {
select {
case event := <-s.events:
if err := s.insertBatch([]Event{event}); err != nil {
+ s.recordWriteFailure(err)
log.Error().Err(err).Msg("alert event log final drain write failed")
+ continue
}
s.written.Add(1)
default:
@@ -313,6 +337,29 @@ func (s *Store) writeLoop() {
}
}
+func (s *Store) recordWriteFailure(err error) {
+ if s == nil || err == nil {
+ return
+ }
+ s.failureMu.Lock()
+ s.lastError = err
+ s.failureMu.Unlock()
+ s.failed.Add(1)
+}
+
+func (s *Store) writeError() error {
+ if s == nil || s.failed.Load() == 0 {
+ return nil
+ }
+ s.failureMu.RLock()
+ err := s.lastError
+ s.failureMu.RUnlock()
+ if err == nil {
+ return fmt.Errorf("alert event log write failed")
+ }
+ return fmt.Errorf("alert event log write failed: %w", err)
+}
+
func (s *Store) insertBatch(batch []Event) error {
tx, err := s.db.Begin()
if err != nil {
@@ -401,18 +448,28 @@ func (s *Store) pruneOld() {
}
}
-// Flush blocks until every event appended before the call has been written
-// (or a short deadline passes). It exists for tests and API reads that must
-// observe just-appended events.
-func (s *Store) Flush() {
+// Flush blocks until every diagnostic event appended before the call has been
+// written. It reports failed writes and timeouts rather than claiming a lost
+// batch was successfully flushed.
+func (s *Store) Flush() error {
if s == nil {
- return
+ return nil
}
target := s.appended.Load()
deadline := time.Now().Add(5 * time.Second)
for s.written.Load() < target && time.Now().Before(deadline) {
+ if err := s.writeError(); err != nil {
+ return err
+ }
time.Sleep(time.Millisecond)
}
+ if err := s.writeError(); err != nil {
+ return err
+ }
+ if written := s.written.Load(); written < target {
+ return fmt.Errorf("alert event log flush timed out: wrote %d of %d events", written, target)
+ }
+ return nil
}
// Query returns matching events, newest first.
diff --git a/internal/alerts/eventlog/eventlog_test.go b/internal/alerts/eventlog/eventlog_test.go
index 119e009e2..756043196 100644
--- a/internal/alerts/eventlog/eventlog_test.go
+++ b/internal/alerts/eventlog/eventlog_test.go
@@ -240,3 +240,19 @@ func TestOpenRequiresExistingDirectory(t *testing.T) {
}
store.Close()
}
+
+func TestDurableAppendAndFlushReportDatabaseFailure(t *testing.T) {
+ store := newTestStore(t)
+ if err := store.db.Close(); err != nil {
+ t.Fatalf("close database: %v", err)
+ }
+
+ if err := store.AppendDurable(Event{Type: TypeFired, AlertID: "durable-failure"}); err == nil {
+ t.Fatal("AppendDurable reported success after the database closed")
+ }
+
+ store.Append(Event{Type: TypeNotificationSuppressed, AlertID: "async-failure"})
+ if err := store.Flush(); err == nil {
+ t.Fatal("Flush reported success after the async batch failed")
+ }
+}
diff --git a/internal/alerts/history.go b/internal/alerts/history.go
index 120a24e2e..10d38cd24 100644
--- a/internal/alerts/history.go
+++ b/internal/alerts/history.go
@@ -295,6 +295,18 @@ func (hm *HistoryManager) StorageFileExists() bool {
return false
}
+// ImportedStorageFileExists reports whether a retired JSON source remains as
+// a recovery copy. It is replayed idempotently if events.db is recreated and
+// remains the read fallback when the event store is unavailable.
+func (hm *HistoryManager) ImportedStorageFileExists() bool {
+ for _, path := range []string{hm.historyFile + ".imported", hm.backupFile + ".imported"} {
+ if _, err := os.Stat(path); err == nil {
+ return true
+ }
+ }
+ return false
+}
+
// StorageLoadError reports why a present legacy source could not be loaded.
// Migration and periodic persistence must leave that source untouched.
func (hm *HistoryManager) StorageLoadError() error {
@@ -493,26 +505,46 @@ func (hm *HistoryManager) loadHistory() error {
// Try backup file
var backupErr error
+ loadedRetired := false
data, backupErr = readLimitedRegularFile(hm.backupFile, maxAlertHistoryFileSizeBytes)
if backupErr != nil {
if os.IsNotExist(backupErr) && os.IsNotExist(mainErr) {
- // Both files don't exist - this is normal on first startup.
- log.Debug().Msg("No alert history files found, starting fresh")
- return nil
+ // Retired JSON remains a recovery source if events.db is lost or
+ // unavailable. Prefer the primary retired leaf, then its backup.
+ var importedErr error
+ data, importedErr = readLimitedRegularFile(hm.historyFile+".imported", maxAlertHistoryFileSizeBytes)
+ if importedErr != nil {
+ data, importedErr = readLimitedRegularFile(hm.backupFile+".imported", maxAlertHistoryFileSizeBytes)
+ }
+ if importedErr != nil {
+ if os.IsNotExist(importedErr) {
+ log.Debug().Msg("No alert history files found, starting fresh")
+ return nil
+ }
+ return fmt.Errorf("failed to read retired alert history recovery source: %w", importedErr)
+ }
+ hm.storageRetired = true
+ loadedRetired = true
+ log.Info().Msg("loaded retired alert history recovery source")
}
- if os.IsPermission(backupErr) {
+ if !loadedRetired && os.IsPermission(backupErr) {
log.Warn().
Err(backupErr).
Str("file", hm.backupFile).
Msg("Permission denied reading backup history file - check file ownership")
}
- if !os.IsNotExist(mainErr) {
+ if loadedRetired {
+ backupErr = nil
+ } else if !os.IsNotExist(mainErr) {
return fmt.Errorf("failed to read history file %q (%v); failed to read history backup file %q: %w",
hm.historyFile, mainErr, hm.backupFile, backupErr)
+ } else {
+ return fmt.Errorf("failed to read history backup file %q: %w", hm.backupFile, backupErr)
}
- return fmt.Errorf("failed to read history backup file %q: %w", hm.backupFile, backupErr)
}
- log.Info().Msg("loaded alert history from backup file")
+ if !loadedRetired {
+ log.Info().Msg("loaded alert history from backup file")
+ }
}
var history []HistoryEntry
diff --git a/internal/alerts/history_migration.go b/internal/alerts/history_migration.go
index e2c713b7b..b4e7654f2 100644
--- a/internal/alerts/history_migration.go
+++ b/internal/alerts/history_migration.go
@@ -21,17 +21,21 @@ import (
// history reads fall back to the in-memory entries either way. A retirement
// failure also retries safely because history_imported events are inserted by
// immutable event identity.
-func (m *Manager) importLegacyHistoryIntoEventLog(store *eventlog.Store) {
- if m == nil || store == nil || m.historyManager == nil {
- return
+func (m *Manager) importLegacyHistoryIntoEventLog(store *eventlog.Store) (authoritative bool) {
+ if m != nil {
+ defer func() { m.eventHistoryAuthoritative.Store(authoritative) }()
}
- if !m.historyManager.StorageFileExists() {
- return
+ if m == nil || store == nil || m.historyManager == nil {
+ return false
+ }
+ hasLegacySource := m.historyManager.StorageFileExists()
+ if !hasLegacySource && !m.historyManager.ImportedStorageFileExists() {
+ return true
}
if err := m.historyManager.StorageLoadError(); err != nil {
log.Error().Err(err).
Msg("legacy alert history import deferred; source could not be loaded and remains untouched")
- return
+ return false
}
entries := m.historyManager.SnapshotEntries()
@@ -42,13 +46,13 @@ func (m *Manager) importLegacyHistoryIntoEventLog(store *eventlog.Store) {
if exported == nil || exported.ID == "" {
log.Error().Int("entry", i).
Msg("legacy alert history import deferred; entry has no alert identity")
- return
+ return false
}
snapshot, err := json.Marshal(exported)
if err != nil {
log.Error().Err(err).Int("entry", i).
Msg("legacy alert history import deferred; entry snapshot could not be encoded")
- return
+ return false
}
occurredAt := entry.Timestamp
if entry.Alert.LastSeen.After(occurredAt) {
@@ -72,14 +76,17 @@ func (m *Manager) importLegacyHistoryIntoEventLog(store *eventlog.Store) {
log.Error().Err(err).
Int("entries", len(events)).
Msg("legacy alert history import failed; JSON history stays authoritative until the next attempt")
- return
+ return false
}
}
- if err := m.historyManager.RetireStorage(); err != nil {
- log.Error().Err(err).Msg("legacy alert history files could not be retired after import")
- return
+ if hasLegacySource {
+ if err := m.historyManager.RetireStorage(); err != nil {
+ log.Error().Err(err).Msg("legacy alert history files could not be retired after import")
+ return false
+ }
}
log.Info().
Int("entries", len(events)).
Msg("legacy alert history imported into the event log; JSON history files retired")
+ return true
}
diff --git a/internal/alerts/history_migration_test.go b/internal/alerts/history_migration_test.go
index 67d1e3673..15c31b51a 100644
--- a/internal/alerts/history_migration_test.go
+++ b/internal/alerts/history_migration_test.go
@@ -82,6 +82,41 @@ func TestLegacyHistoryImportRetiresJSONAndServesEntries(t *testing.T) {
}
}
+func TestRetiredLegacyHistoryRecoversRecreatedEventDatabase(t *testing.T) {
+ dataDir := t.TempDir()
+ seed := NewManagerWithDataDir(dataDir)
+ past := time.Now().Add(-2 * time.Hour).UTC()
+ seed.historyManager.AddAlert(Alert{
+ ID: "legacy-recovery::metric-threshold:cpu",
+ Type: "cpu",
+ ResourceID: "legacy-recovery",
+ StartTime: past,
+ LastSeen: past.Add(time.Minute),
+ })
+ if err := seed.historyManager.saveHistory(); err != nil {
+ t.Fatalf("persist legacy history: %v", err)
+ }
+ seed.EnableEventLog()
+ seed.SetEventLog(nil)
+ seed.Stop()
+
+ eventsPath := filepath.Join(dataDir, "alerts", "events.db")
+ if err := os.Remove(eventsPath); err != nil {
+ t.Fatalf("remove event database: %v", err)
+ }
+ for _, suffix := range []string{"-shm", "-wal"} {
+ _ = os.Remove(eventsPath + suffix)
+ }
+
+ restarted := NewManagerWithDataDir(dataDir)
+ t.Cleanup(restarted.Stop)
+ restarted.EnableEventLog()
+ entries := restarted.GetAlertHistory(0)
+ if len(entries) != 1 || entries[0].ResourceID != "legacy-recovery" {
+ t.Fatalf("recreated event database did not recover retired history: %#v", entries)
+ }
+}
+
func TestLegacyHistoryImportRetryAfterRetirementFailureIsIdempotent(t *testing.T) {
dataDir := t.TempDir()
m := NewManagerWithDataDir(dataDir)
diff --git a/internal/alerts/history_projection.go b/internal/alerts/history_projection.go
index 1e93f15e7..fca798373 100644
--- a/internal/alerts/history_projection.go
+++ b/internal/alerts/history_projection.go
@@ -34,10 +34,9 @@ func historyOccurrenceKey(alertID string, snapshot *Alert) string {
// fall back to the JSON-backed history manager.
func (m *Manager) AlertHistoryFromEvents(since time.Time, limit int) ([]Alert, bool) {
store := m.eventLogStore()
- if store == nil {
+ if store == nil || !m.eventHistoryAuthoritative.Load() {
return nil, false
}
- store.Flush()
// Fold oldest to newest through the store's bounded-page walker. The
// ordinary Query API intentionally caps responses at 1,000 rows, which is
diff --git a/internal/alerts/manager.go b/internal/alerts/manager.go
index bf7cdb1fb..ab2db1578 100644
--- a/internal/alerts/manager.go
+++ b/internal/alerts/manager.go
@@ -108,6 +108,10 @@ type Manager struct {
// Append-only alert event log (transitions + notification decisions).
// Nil until EnableEventLog/SetEventLog; recording is then a no-op.
eventLog atomic.Pointer[eventlog.Store]
+ // eventHistoryAuthoritative becomes true only after legacy JSON history is
+ // absent or has been durably imported. Reads keep using JSON while migration
+ // is incomplete or the event store reports a write failure.
+ eventHistoryAuthoritative atomic.Bool
// Shadow-mode reducer feed (Phase 1 capstone). Nil until
// EnableShadowFeed; all access is under m.mu.
diff --git a/internal/alerts/notification_activation_boundary_test.go b/internal/alerts/notification_activation_boundary_test.go
index 65223ae7b..8106c3c6c 100644
--- a/internal/alerts/notification_activation_boundary_test.go
+++ b/internal/alerts/notification_activation_boundary_test.go
@@ -13,9 +13,13 @@ func TestNotificationActivationDoesNotSuppressDetectionOrActiveReadModel(t *test
t.Run(string(activationState), func(t *testing.T) {
m := newTestManager(t)
delivered := make(chan *Alert, 1)
+ lifecycle := make(chan LifecycleEvent, 1)
m.SetAlertCallback(func(alert *Alert) {
delivered <- alert
})
+ m.SubscribeLifecycleCallback(func(event LifecycleEvent) {
+ lifecycle <- event
+ })
m.mu.Lock()
m.config.Enabled = true
@@ -53,6 +57,15 @@ func TestNotificationActivationDoesNotSuppressDetectionOrActiveReadModel(t *test
)
}
+ select {
+ case event := <-lifecycle:
+ if event.Type != "fired" || event.Alert == nil || event.Alert.ID != active[0].ID {
+ t.Fatalf("activation state %q emitted unexpected lifecycle event: %#v", activationState, event)
+ }
+ default:
+ t.Fatalf("activation state %q suppressed the alert lifecycle event", activationState)
+ }
+
select {
case alert := <-delivered:
t.Fatalf(
diff --git a/internal/monitoring/canonical_guardrails_test.go b/internal/monitoring/canonical_guardrails_test.go
index 2f232572d..118a5ab9a 100644
--- a/internal/monitoring/canonical_guardrails_test.go
+++ b/internal/monitoring/canonical_guardrails_test.go
@@ -11,6 +11,7 @@ import (
"github.com/rcourtman/pulse-go-rewrite/internal/ai/memory"
"github.com/rcourtman/pulse-go-rewrite/internal/alerts"
+ "github.com/rcourtman/pulse-go-rewrite/internal/alerts/eventlog"
"github.com/rcourtman/pulse-go-rewrite/internal/config"
"github.com/rcourtman/pulse-go-rewrite/internal/mock"
"github.com/rcourtman/pulse-go-rewrite/internal/models"
@@ -2344,7 +2345,11 @@ func TestAlertLifecycleCanonicalChangesRemainWritable(t *testing.T) {
StartTime: startedAt,
}
- monitor.handleAlertFired(alert)
+ monitor.handleAlertLifecycleEvent(alerts.LifecycleEvent{
+ Type: eventlog.TypeFired,
+ OccurredAt: startedAt,
+ Alert: alert,
+ })
changes, err := store.GetRecentChanges("vm-guardrail", time.Time{}, 10)
if err != nil {
diff --git a/internal/monitoring/monitor.go b/internal/monitoring/monitor.go
index 5b278478d..53cd14690 100644
--- a/internal/monitoring/monitor.go
+++ b/internal/monitoring/monitor.go
@@ -1977,6 +1977,7 @@ func (m *Monitor) Start(ctx context.Context, wsHub *websocket.Hub) {
}
// Set up alert callbacks
+ m.alertManager.SubscribeLifecycleCallback(m.handleAlertLifecycleEvent)
m.alertManager.SetAlertCallback(func(alert *alerts.Alert) {
m.handleAlertFired(alert)
})
@@ -1993,15 +1994,11 @@ func (m *Monitor) Start(ctx context.Context, wsHub *websocket.Hub) {
// Don't broadcast full state here - it causes a cascade with many guests.
// The frontend will get the updated alerts through the regular broadcast ticker.
})
- m.alertManager.SetAcknowledgedCallback(func(alert *alerts.Alert, user string) {
- m.handleAlertAcknowledged(alert, user)
- })
- m.alertManager.SetUnacknowledgedCallback(func(alert *alerts.Alert, user string) {
- m.handleAlertUnacknowledged(alert, user)
- })
m.alertManager.SetEscalateCallback(func(alert *alerts.Alert, level int) {
m.handleAlertEscalated(wsHub, alert, level)
})
+ m.replayAlertLifecycleProjections()
+ m.reconcileActiveAlertTimelines()
// Create separate tickers for polling and broadcasting using the configured cadence
diff --git a/internal/monitoring/monitor_alert_handling_test.go b/internal/monitoring/monitor_alert_handling_test.go
index ce93717b6..8c645e20c 100644
--- a/internal/monitoring/monitor_alert_handling_test.go
+++ b/internal/monitoring/monitor_alert_handling_test.go
@@ -9,7 +9,9 @@ import (
"testing"
"time"
+ "github.com/rcourtman/pulse-go-rewrite/internal/ai/memory"
"github.com/rcourtman/pulse-go-rewrite/internal/alerts"
+ "github.com/rcourtman/pulse-go-rewrite/internal/alerts/eventlog"
"github.com/rcourtman/pulse-go-rewrite/internal/models"
"github.com/rcourtman/pulse-go-rewrite/internal/notifications"
unifiedresources "github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources"
@@ -67,6 +69,20 @@ func TestMonitor_HandleAlertFired_RecoversFromPushCallbackPanic(t *testing.T) {
m.handleAlertFired(&alerts.Alert{ID: "alert-push-panic"})
}
+func TestDeliveryCallbackDoesNotDuplicateCanonicalAITrigger(t *testing.T) {
+ called := make(chan struct{}, 1)
+ monitor := &Monitor{
+ alertTriggeredAICallback: func(*alerts.Alert) { called <- struct{}{} },
+ }
+
+ monitor.handleAlertFired(&alerts.Alert{ID: "single-ai-trigger"})
+ select {
+ case <-called:
+ t.Fatal("delivery callback invoked AI analysis; the manager-owned unconditional AI callback is the sole trigger")
+ case <-time.After(100 * time.Millisecond):
+ }
+}
+
func TestMonitor_HandleAlertLifecycle_WritesCanonicalChanges(t *testing.T) {
store := unifiedresources.NewMemoryStore()
m := &Monitor{
@@ -91,9 +107,19 @@ func TestMonitor_HandleAlertLifecycle_WritesCanonicalChanges(t *testing.T) {
},
}
- m.handleAlertFired(alert)
- m.handleAlertAcknowledged(alert, "admin")
- m.handleAlertUnacknowledged(alert, "admin")
+ m.handleAlertLifecycleEvent(alerts.LifecycleEvent{Type: eventlog.TypeFired, OccurredAt: startedAt, Alert: alert})
+ m.handleAlertLifecycleEvent(alerts.LifecycleEvent{
+ Type: eventlog.TypeAcknowledged,
+ OccurredAt: ackAt,
+ Alert: alert,
+ Details: map[string]string{"user": "admin"},
+ })
+ m.handleAlertLifecycleEvent(alerts.LifecycleEvent{
+ Type: eventlog.TypeUnacknowledged,
+ OccurredAt: ackAt.Add(time.Minute),
+ Alert: alert,
+ Details: map[string]string{"user": "admin"},
+ })
changes, err := store.GetRecentChanges("vm-1", time.Time{}, 10)
if err != nil {
@@ -123,6 +149,164 @@ func TestMonitor_HandleAlertLifecycle_WritesCanonicalChanges(t *testing.T) {
}
}
+func TestPausedDeliveryStillBuildsTimelineThroughRealAlertLifecycle(t *testing.T) {
+ manager := alerts.NewManagerWithDataDir(t.TempDir())
+ t.Cleanup(manager.Stop)
+ config := manager.GetConfig()
+ config.Enabled = true
+ config.ActivationState = alerts.ActivationPending
+ config.TimeThresholds = map[string]int{}
+ config.SuppressionWindow = 0
+ manager.UpdateConfig(config)
+
+ resourceStore := unifiedresources.NewMemoryStore()
+ incidentStore := memory.NewIncidentStore(memory.IncidentStoreConfig{})
+ monitor := &Monitor{
+ alertManager: manager,
+ incidentStore: incidentStore,
+ resourceStore: unifiedresources.NewMonitorAdapter(unifiedresources.NewRegistry(resourceStore)),
+ }
+ incidentStore.SetResourceTimelineStore(monitor.resourceStore.(memory.IncidentTimelineStore))
+ manager.SubscribeLifecycleCallback(monitor.handleAlertLifecycleEvent)
+ delivered := make(chan *alerts.Alert, 1)
+ manager.SetAlertCallback(func(alert *alerts.Alert) { delivered <- alert })
+
+ vm := models.VM{ID: "paused-vm", Name: "Paused VM", Node: "node-1", Instance: "pve-1", Status: "stopped"}
+ manager.CheckGuest(vm, vm.Instance)
+ manager.CheckGuest(vm, vm.Instance)
+
+ active := manager.GetActiveAlerts()
+ if len(active) != 1 {
+ t.Fatalf("active alerts = %d, want 1", len(active))
+ }
+ timeline := incidentStore.GetTimelineByAlertAt(active[0].ID, active[0].StartTime)
+ if timeline == nil || len(timeline.Events) == 0 || timeline.Events[0].Type != memory.IncidentEventAlertFired {
+ t.Fatalf("paused-delivery lifecycle did not produce an incident timeline: %#v", timeline)
+ }
+ select {
+ case alert := <-delivered:
+ t.Fatalf("pending-review alert reached delivery callback: %s", alert.ID)
+ default:
+ }
+}
+
+func TestActiveTimelineReconciliationIsIdempotent(t *testing.T) {
+ manager := alerts.NewManagerWithDataDir(t.TempDir())
+ t.Cleanup(manager.Stop)
+ config := manager.GetConfig()
+ config.Enabled = true
+ config.ActivationState = alerts.ActivationPending
+ config.TimeThresholds = map[string]int{}
+ manager.UpdateConfig(config)
+
+ vm := models.VM{ID: "restored-vm", Name: "Restored VM", Node: "node-1", Instance: "pve-1", Status: "stopped"}
+ manager.CheckGuest(vm, vm.Instance)
+ manager.CheckGuest(vm, vm.Instance)
+
+ resourceStore := unifiedresources.NewMemoryStore()
+ incidentStore := memory.NewIncidentStore(memory.IncidentStoreConfig{})
+ monitor := &Monitor{
+ alertManager: manager,
+ incidentStore: incidentStore,
+ resourceStore: unifiedresources.NewMonitorAdapter(unifiedresources.NewRegistry(resourceStore)),
+ }
+ incidentStore.SetResourceTimelineStore(monitor.resourceStore.(memory.IncidentTimelineStore))
+ monitor.reconcileActiveAlertTimelines()
+ monitor.reconcileActiveAlertTimelines()
+
+ active := manager.GetActiveAlerts()
+ if len(active) != 1 {
+ t.Fatalf("active alerts = %d, want 1", len(active))
+ }
+ timeline := incidentStore.GetTimelineByAlertAt(active[0].ID, active[0].StartTime)
+ if timeline == nil || len(timeline.Events) != 1 || timeline.Events[0].Type != memory.IncidentEventAlertFired {
+ t.Fatalf("reconciled timeline = %#v, want one fired event", timeline)
+ }
+ changes, err := resourceStore.GetRecentChanges(active[0].ResourceID, time.Time{}, 10)
+ if err != nil {
+ t.Fatalf("GetRecentChanges: %v", err)
+ }
+ if len(changes) != 1 {
+ t.Fatalf("reconciliation wrote %d canonical changes, want 1", len(changes))
+ }
+}
+
+func TestLifecycleReplayRepairsResolvedIncidentTimeline(t *testing.T) {
+ manager := alerts.NewManagerWithDataDir(t.TempDir(), alerts.WithoutPersistedAlertRestore())
+ t.Cleanup(manager.Stop)
+ manager.EnableEventLog()
+ config := manager.GetConfig()
+ config.Enabled = true
+ config.ActivationState = alerts.ActivationPending
+ config.TimeThresholds = map[string]int{}
+ config.SuppressionWindow = 0
+ manager.UpdateConfig(config)
+
+ vm := models.VM{ID: "historical-vm", Name: "Historical VM", Node: "node-1", Instance: "pve-1", Status: "stopped"}
+ manager.CheckGuest(vm, vm.Instance)
+ manager.CheckGuest(vm, vm.Instance)
+ active := manager.GetActiveAlerts()
+ if len(active) != 1 {
+ t.Fatalf("active alerts = %d, want 1", len(active))
+ }
+ alertID, startedAt := active[0].ID, active[0].StartTime
+
+ vm.Status = "running"
+ manager.CheckGuest(vm, vm.Instance)
+ if active := manager.GetActiveAlerts(); len(active) != 0 {
+ t.Fatalf("active alerts after recovery = %d, want 0", len(active))
+ }
+
+ resourceStore := unifiedresources.NewMemoryStore()
+ incidentStore := memory.NewIncidentStore(memory.IncidentStoreConfig{})
+ monitor := &Monitor{
+ alertManager: manager,
+ incidentStore: incidentStore,
+ resourceStore: unifiedresources.NewMonitorAdapter(unifiedresources.NewRegistry(resourceStore)),
+ }
+ incidentStore.SetResourceTimelineStore(monitor.resourceStore.(memory.IncidentTimelineStore))
+ monitor.replayAlertLifecycleProjections()
+ monitor.replayAlertLifecycleProjections()
+
+ timeline := incidentStore.GetTimelineByAlertAt(alertID, startedAt)
+ if timeline == nil || len(timeline.Events) != 2 {
+ t.Fatalf("replayed timeline = %#v, want fired and resolved events", timeline)
+ }
+ if timeline.Events[0].Type != memory.IncidentEventAlertFired || timeline.Events[1].Type != memory.IncidentEventAlertResolved {
+ t.Fatalf("replayed event types = %#v, want fired then resolved", timeline.Events)
+ }
+ changes, err := resourceStore.GetRecentChanges(vm.ID, time.Time{}, 10)
+ if err != nil {
+ t.Fatalf("GetRecentChanges: %v", err)
+ }
+ if len(changes) != 2 {
+ t.Fatalf("replay wrote %d canonical changes, want 2", len(changes))
+ }
+}
+
+func TestSystemAlertTimelineUsesCanonicalPulseResource(t *testing.T) {
+ resourceStore := unifiedresources.NewMemoryStore()
+ incidentStore := memory.NewIncidentStore(memory.IncidentStoreConfig{})
+ monitor := &Monitor{
+ incidentStore: incidentStore,
+ resourceStore: unifiedresources.NewMonitorAdapter(unifiedresources.NewRegistry(resourceStore)),
+ }
+ incidentStore.SetResourceTimelineStore(monitor.resourceStore.(memory.IncidentTimelineStore))
+ alert := &alerts.Alert{
+ ID: alerts.SystemAlertID("event-store-health"),
+ Type: "event-store-health",
+ Level: alerts.AlertLevelCritical,
+ ResourceName: alerts.SystemAlertResourceName,
+ Message: "Alert history storage is unavailable",
+ StartTime: time.Now().UTC(),
+ }
+ monitor.handleAlertLifecycleEvent(alerts.LifecycleEvent{Type: eventlog.TypeFired, OccurredAt: alert.StartTime, Alert: alert})
+ timeline := incidentStore.GetTimelineByAlertAt(alert.ID, alert.StartTime)
+ if timeline == nil || timeline.ResourceID != "pulse-system" || len(timeline.Events) != 1 {
+ t.Fatalf("system alert timeline = %#v", timeline)
+ }
+}
+
func TestMonitor_HandleAlertResolved_Detailed_Extra(t *testing.T) {
// 1. With Hub and NotificationMgr and Resolve Notify ON
hub := websocket.NewHub(nil)
diff --git a/internal/monitoring/monitor_alerts.go b/internal/monitoring/monitor_alerts.go
index d7a8d06c5..2a9e3ac57 100644
--- a/internal/monitoring/monitor_alerts.go
+++ b/internal/monitoring/monitor_alerts.go
@@ -5,8 +5,10 @@ import (
"strings"
"time"
+ "github.com/google/uuid"
"github.com/rcourtman/pulse-go-rewrite/internal/ai/memory"
"github.com/rcourtman/pulse-go-rewrite/internal/alerts"
+ "github.com/rcourtman/pulse-go-rewrite/internal/alerts/eventlog"
"github.com/rcourtman/pulse-go-rewrite/internal/mock"
"github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources"
"github.com/rcourtman/pulse-go-rewrite/internal/websocket"
@@ -122,23 +124,6 @@ func (m *Monitor) handleAlertFired(alert *alerts.Alert) {
}()
}
- if m.incidentStore != nil {
- m.incidentStore.RecordAlertFired(alert)
- }
- m.recordAlertTimelineChange(alert, unifiedresources.ChangeAlertFired, alert.StartTime, "")
-
- // Trigger AI analysis if callback is configured
- if m.alertTriggeredAICallback != nil {
- // Run in goroutine to avoid blocking the monitor loop
- go func() {
- defer func() {
- if r := recover(); r != nil {
- log.Error().Interface("panic", r).Msg("panic in AI alert callback")
- }
- }()
- m.alertTriggeredAICallback(alert)
- }()
- }
}
func (m *Monitor) handleAlertResolved(alertID string) {
@@ -148,21 +133,6 @@ func (m *Monitor) handleAlertResolved(alertID string) {
m.wsHub.BroadcastAlertResolvedToTenant(m.GetOrgID(), alertID)
}
- // Always record incident timeline, regardless of notification suppression.
- // This ensures we have a complete history even during quiet hours.
- if m.incidentStore != nil {
- resolvedAlert = m.alertManager.GetResolvedAlert(alertID)
- if resolvedAlert != nil && resolvedAlert.Alert != nil {
- m.incidentStore.RecordAlertResolved(resolvedAlert.Alert, resolvedAlert.ResolvedTime)
- }
- }
- if resolvedAlert == nil && m.alertManager != nil {
- resolvedAlert = m.alertManager.GetResolvedAlert(alertID)
- }
- if resolvedAlert != nil && resolvedAlert.Alert != nil {
- m.recordAlertTimelineChange(resolvedAlert.Alert, unifiedresources.ChangeAlertResolved, resolvedAlert.ResolvedTime, "")
- }
-
// Always trigger AI callback, regardless of notification suppression.
if m.alertResolvedAICallback != nil {
if resolvedAlert == nil {
@@ -248,29 +218,88 @@ func (m *Monitor) handleAlertEscalated(hub *websocket.Hub, alert *alerts.Alert,
m.broadcastEscalatedAlert(hub, alert)
}
-func (m *Monitor) handleAlertAcknowledged(alert *alerts.Alert, user string) {
- if m.incidentStore == nil || alert == nil {
- if alert == nil {
- return
- }
- } else {
- m.incidentStore.RecordAlertAcknowledged(alert, user)
- }
- occurredAt := time.Now()
- if alert.AckTime != nil {
- occurredAt = *alert.AckTime
- }
- m.recordAlertTimelineChange(alert, unifiedresources.ChangeAlertAcknowledged, occurredAt, user)
-}
-
-func (m *Monitor) handleAlertUnacknowledged(alert *alerts.Alert, user string) {
- if alert == nil {
+func (m *Monitor) handleAlertLifecycleEvent(event alerts.LifecycleEvent) {
+ alert := event.Alert
+ if m == nil || alert == nil {
return
}
- if m.incidentStore != nil {
- m.incidentStore.RecordAlertUnacknowledged(alert, user)
+
+ actor := event.Details["user"]
+ timelineAlert := alert
+ if alerts.IsSystemAlert(alert) && strings.TrimSpace(alert.ResourceID) == "" {
+ timelineAlert = alert.Clone()
+ timelineAlert.ResourceID = "pulse-system"
+ }
+ switch event.Type {
+ case eventlog.TypeFired, eventlog.TypeRefired:
+ if m.incidentStore != nil {
+ m.incidentStore.RecordAlertFired(timelineAlert)
+ }
+ occurredAt := event.OccurredAt
+ if event.Type == eventlog.TypeFired && !alert.StartTime.IsZero() {
+ occurredAt = alert.StartTime
+ }
+ m.recordAlertTimelineChange(timelineAlert, unifiedresources.ChangeAlertFired, occurredAt, "")
+ case eventlog.TypeAcknowledged:
+ if m.incidentStore != nil {
+ m.incidentStore.RecordAlertAcknowledged(timelineAlert, actor)
+ }
+ occurredAt := event.OccurredAt
+ if alert.AckTime != nil && !alert.AckTime.IsZero() {
+ occurredAt = *alert.AckTime
+ }
+ m.recordAlertTimelineChange(timelineAlert, unifiedresources.ChangeAlertAcknowledged, occurredAt, actor)
+ case eventlog.TypeUnacknowledged:
+ if m.incidentStore != nil {
+ m.incidentStore.RecordAlertUnacknowledged(timelineAlert, actor)
+ }
+ m.recordAlertTimelineChange(timelineAlert, unifiedresources.ChangeAlertUnacknowledged, event.OccurredAt, actor)
+ case eventlog.TypeResolved:
+ if m.incidentStore != nil {
+ m.incidentStore.RecordAlertResolved(timelineAlert, event.OccurredAt)
+ }
+ m.recordAlertTimelineChange(timelineAlert, unifiedresources.ChangeAlertResolved, event.OccurredAt, "")
+ }
+}
+
+func (m *Monitor) replayAlertLifecycleProjections() {
+ if m == nil || m.alertManager == nil {
+ return
+ }
+ if err := m.alertManager.ReplayLifecycleEvents(func(event alerts.LifecycleEvent) error {
+ m.handleAlertLifecycleEvent(event)
+ return nil
+ }); err != nil {
+ log.Error().Err(err).Msg("failed to replay canonical alert lifecycle projections")
+ }
+}
+
+func (m *Monitor) reconcileActiveAlertTimelines() {
+ if m == nil || m.alertManager == nil || m.incidentStore == nil {
+ return
+ }
+ activeAlerts := m.alertManager.GetActiveAlerts()
+ for i := range activeAlerts {
+ alert := &activeAlerts[i]
+ timeline := m.incidentStore.GetTimelineByAlertAt(alert.ID, alert.StartTime)
+ if timeline != nil {
+ hasFired := false
+ for _, event := range timeline.Events {
+ if event.Type == memory.IncidentEventAlertFired {
+ hasFired = true
+ break
+ }
+ }
+ if hasFired {
+ continue
+ }
+ }
+ m.handleAlertLifecycleEvent(alerts.LifecycleEvent{
+ Type: eventlog.TypeFired,
+ OccurredAt: alert.StartTime,
+ Alert: alert,
+ })
}
- m.recordAlertTimelineChange(alert, unifiedresources.ChangeAlertUnacknowledged, time.Now(), user)
}
func (m *Monitor) recordAlertTimelineChange(alert *alerts.Alert, kind unifiedresources.ChangeKind, occurredAt time.Time, actor string) {
@@ -294,6 +323,13 @@ func (m *Monitor) recordAlertTimelineChange(alert *alerts.Alert, kind unifiedres
if change == nil {
return
}
+ change.ID = uuid.NewSHA1(uuid.NameSpaceOID, []byte(strings.Join([]string{
+ "pulse-alert-lifecycle-v1",
+ strings.TrimSpace(alert.ID),
+ strings.TrimSpace(alert.ResourceID),
+ string(kind),
+ occurredAt.UTC().Format(time.RFC3339Nano),
+ }, "\x00"))).String()
if err := recorder.RecordChange(*change); err != nil {
log.Warn().
Err(err).
diff --git a/internal/monitoring/monitor_full_coverage_test.go b/internal/monitoring/monitor_full_coverage_test.go
index 25b28c9ac..b079a3ff1 100644
--- a/internal/monitoring/monitor_full_coverage_test.go
+++ b/internal/monitoring/monitor_full_coverage_test.go
@@ -402,8 +402,6 @@ func TestMonitor_AlertCallbacks(t *testing.T) {
// No panic = pass
m.handleAlertResolved("test-alert")
- m.handleAlertAcknowledged(alert, "user")
- m.handleAlertUnacknowledged(alert, "user")
}
type mockResourceStore struct{}
diff --git a/tests/integration/tests/94-alert-history-real-backend.spec.ts b/tests/integration/tests/94-alert-history-real-backend.spec.ts
index 35c6b32da..25bddb6ab 100644
--- a/tests/integration/tests/94-alert-history-real-backend.spec.ts
+++ b/tests/integration/tests/94-alert-history-real-backend.spec.ts
@@ -6,7 +6,11 @@ import { expect, test as base } from '@playwright/test';
import { restartManagedLocalBackend } from '../scripts/managed-local-backend.mjs';
import { readRuntimeState } from '../scripts/runtime-state.mjs';
-import { apiRequest, createAuthenticatedStorageState, ensureAuthenticated } from './helpers';
+import {
+ apiRequest,
+ createAuthenticatedStorageState,
+ ensureAuthenticated,
+} from './helpers';
const __dirname = path.dirname(fileURLToPath(import.meta.url));
@@ -15,7 +19,8 @@ type WorkerFixtures = {
};
const test = base.extend<{}, WorkerFixtures>({
- storageState: async ({ authStorageStatePath }, use) => use(authStorageStatePath),
+ storageState: async ({ authStorageStatePath }, use) =>
+ use(authStorageStatePath),
authStorageStatePath: [
async ({ browser }, use, workerInfo) => {
const storageStatePath = path.resolve(
@@ -58,20 +63,31 @@ type HistoryAlert = {
lastSeen: string;
};
-async function readHistory(page: import('@playwright/test').Page): Promise {
+async function readHistory(
+ page: import('@playwright/test').Page,
+): Promise {
const response = await apiRequest(page, '/api/alerts/history?limit=0');
- expect(response.ok(), `history API returned ${response.status()}`).toBeTruthy();
+ expect(
+ response.ok(),
+ `history API returned ${response.status()}`,
+ ).toBeTruthy();
return (await response.json()) as HistoryAlert[];
}
test.describe('Real backend alert history qualification', () => {
- test.skip(!qualificationEnabled, 'Run through npm run test:alerts:qualification');
+ test.skip(
+ !qualificationEnabled,
+ 'Run through npm run test:alerts:qualification',
+ );
test.setTimeout(240_000);
test('imports once, renders exact fields, survives restart, and preserves clear tombstones', async ({
page,
}, testInfo) => {
- test.skip(testInfo.project.name !== 'chromium', 'The isolated qualification runner owns Chromium');
+ test.skip(
+ testInfo.project.name !== 'chromium',
+ 'The isolated qualification runner owns Chromium',
+ );
const runtimeState = await readRuntimeState();
expect(runtimeState?.managedLocalBackend).toBe(true);
@@ -81,26 +97,46 @@ test.describe('Real backend alert history qualification', () => {
const importedPath = path.join(alertsDir, 'alert-history.json.imported');
const sourcePath = path.join(alertsDir, 'alert-history.json');
- expect(fs.existsSync(importedPath), 'legacy history source must retire after import').toBe(true);
- expect(fs.existsSync(sourcePath), 'retired legacy history source must leave the load path').toBe(false);
+ expect(
+ fs.existsSync(importedPath),
+ 'legacy history source must retire after import',
+ ).toBe(true);
+ expect(
+ fs.existsSync(sourcePath),
+ 'retired legacy history source must leave the load path',
+ ).toBe(false);
expect(fs.readFileSync(runtimeState!.logPath, 'utf8')).toContain(
'legacy alert history imported into the event log; JSON history files retired',
);
await page.goto('/alerts/history', { waitUntil: 'domcontentloaded' });
- await expect(page.getByRole('heading', { name: 'Alert History' })).toBeVisible();
- const legacyRow = page.locator('tr').filter({ hasText: 'Legacy Qualification VM' }).first();
- await expect(legacyRow).toContainText('Legacy history import rendered from SQLite');
+ await expect(
+ page.getByRole('heading', { name: 'Alert History' }),
+ ).toBeVisible();
+ const legacyRow = page
+ .locator('tr')
+ .filter({ hasText: 'Legacy Qualification VM' })
+ .first();
+ await expect(legacyRow).toContainText(
+ 'Legacy history import rendered from SQLite',
+ );
await expect(legacyRow).toContainText('CPU');
await expect(legacyRow).toContainText('Qualification Node');
await expect(legacyRow).toContainText('30m');
await expect(
- page.locator('table.alert-history-responsive-table').getByText('Active Overlay Node').first(),
+ page
+ .locator('table.alert-history-responsive-table')
+ .getByText('Active Overlay Node')
+ .first(),
).toBeVisible();
let history = await readHistory(page);
- expect(history.filter((alert) => alert.id === LEGACY_ALERT_ID)).toHaveLength(1);
- expect(history.filter((alert) => alert.id === ACTIVE_ALERT_ID)).toHaveLength(1);
+ expect(
+ history.filter((alert) => alert.id === LEGACY_ALERT_ID),
+ ).toHaveLength(1);
+ expect(
+ history.filter((alert) => alert.id === ACTIVE_ALERT_ID),
+ ).toHaveLength(1);
const importedAlert = history.find((alert) => alert.id === LEGACY_ALERT_ID);
expect(importedAlert).toMatchObject({
resourceName: 'Legacy Qualification VM',
@@ -118,11 +154,93 @@ test.describe('Real backend alert history qualification', () => {
'Seed timestamps may be constructed a few milliseconds apart',
).toBeLessThanOrEqual(100);
+ const activeAlert = history.find((alert) => alert.id === ACTIVE_ALERT_ID);
+ expect(activeAlert).toBeTruthy();
+ const timelineResponse = await apiRequest(
+ page,
+ `/api/alerts/incidents?alertIdentifier=${encodeURIComponent(ACTIVE_ALERT_ID)}&started_at=${encodeURIComponent(activeAlert!.startTime)}`,
+ );
+ expect(
+ timelineResponse.ok(),
+ `timeline API returned ${timelineResponse.status()}`,
+ ).toBeTruthy();
+ const timeline = (await timelineResponse.json()) as {
+ alertIdentifier: string;
+ events: Array<{ type: string; summary: string }>;
+ } | null;
+ expect(
+ timeline,
+ 'restored active alert must have a reconciled timeline',
+ ).not.toBeNull();
+ expect(timeline?.alertIdentifier).toBe(ACTIVE_ALERT_ID);
+ expect(timeline?.events.some((event) => event.type === 'alert_fired')).toBe(
+ true,
+ );
+
+ await page.goto('/alerts/overview', { waitUntil: 'domcontentloaded' });
+ await expect(page.getByText('Active Overlay Node').first()).toBeVisible();
+ const overviewTimelineRequests: string[] = [];
+ const recordOverviewTimelineRequest = (
+ request: import('@playwright/test').Request,
+ ) => {
+ if (new URL(request.url()).pathname === '/api/alerts/incidents') {
+ overviewTimelineRequests.push(request.url());
+ }
+ };
+ page.on('request', recordOverviewTimelineRequest);
+ const overviewTimelineResponsePromise = page.waitForResponse((response) => {
+ const url = new URL(response.url());
+ return (
+ url.pathname === '/api/alerts/incidents' &&
+ url.searchParams.has('alertIdentifier')
+ );
+ });
+ await page.getByRole('button', { name: 'Timeline' }).click();
+ const overviewTimelineResponse = await overviewTimelineResponsePromise;
+ const overviewTimelineURL = new URL(overviewTimelineResponse.url());
+ expect(overviewTimelineURL.searchParams.get('alertIdentifier')).toBe(
+ ACTIVE_ALERT_ID,
+ );
+ expect(overviewTimelineURL.searchParams.get('started_at')).toBe(
+ activeAlert!.startTime,
+ );
+ const overviewTimeline = (await overviewTimelineResponse.json()) as {
+ events?: Array<{ type: string }>;
+ } | null;
+ expect(overviewTimeline).not.toBeNull();
+ expect(
+ overviewTimeline?.events?.some((event) => event.type === 'alert_fired'),
+ ).toBe(true);
+ await page.waitForTimeout(250);
+ page.off('request', recordOverviewTimelineRequest);
+ expect(overviewTimelineRequests).toHaveLength(1);
+ await expect(page.getByText(/Alert triggered: memory/)).toBeVisible();
+ await expect(page.getByText('No incident timeline available.')).toHaveCount(
+ 0,
+ );
+
+ await page.setViewportSize({ width: 390, height: 844 });
+ await page.reload({ waitUntil: 'domcontentloaded' });
+ await expect(page.getByText('Active Overlay Node').first()).toBeVisible();
+ await page.getByRole('button', { name: 'Timeline' }).click();
+ await expect(page.getByText(/Alert triggered: memory/)).toBeVisible();
+ const narrowOverflow = await page.evaluate(
+ () =>
+ document.documentElement.scrollWidth -
+ document.documentElement.clientWidth,
+ );
+ expect(narrowOverflow).toBeLessThanOrEqual(1);
+
+ await page.setViewportSize({ width: 1280, height: 720 });
+ await page.goto('/alerts/history', { waitUntil: 'domcontentloaded' });
+
await restartManagedLocalBackend();
await ensureAuthenticated(page);
await page.goto('/alerts/history', { waitUntil: 'domcontentloaded' });
await expect(
- page.locator('table.alert-history-responsive-table').getByText('Legacy Qualification VM'),
+ page
+ .locator('table.alert-history-responsive-table')
+ .getByText('Legacy Qualification VM'),
).toBeVisible();
history = await readHistory(page);
expect(
@@ -134,7 +252,10 @@ test.describe('Real backend alert history qualification', () => {
await page.getByRole('button', { name: 'Clear All History' }).click();
await expect(page.getByText('Legacy Qualification VM')).toHaveCount(0);
await expect(
- page.locator('table.alert-history-responsive-table').getByText('Active Overlay Node').first(),
+ page
+ .locator('table.alert-history-responsive-table')
+ .getByText('Active Overlay Node')
+ .first(),
).toBeVisible();
history = await readHistory(page);
expect(history.map((alert) => alert.id)).not.toContain(LEGACY_ALERT_ID);
@@ -145,11 +266,16 @@ test.describe('Real backend alert history qualification', () => {
await page.goto('/alerts/history', { waitUntil: 'domcontentloaded' });
await expect(page.getByText('Legacy Qualification VM')).toHaveCount(0);
await expect(
- page.locator('table.alert-history-responsive-table').getByText('Active Overlay Node').first(),
+ page
+ .locator('table.alert-history-responsive-table')
+ .getByText('Active Overlay Node')
+ .first(),
).toBeVisible();
history = await readHistory(page);
expect(history.map((alert) => alert.id)).not.toContain(LEGACY_ALERT_ID);
- expect(history.filter((alert) => alert.id === ACTIVE_ALERT_ID)).toHaveLength(1);
+ expect(
+ history.filter((alert) => alert.id === ACTIVE_ALERT_ID),
+ ).toHaveLength(1);
await testInfo.attach('real-backend-alert-history-final.json', {
body: Buffer.from(JSON.stringify(history, null, 2)),