diff --git a/docs/release-control/v6/internal/subsystems/ai-runtime.md b/docs/release-control/v6/internal/subsystems/ai-runtime.md index c89347758..5decceecf 100644 --- a/docs/release-control/v6/internal/subsystems/ai-runtime.md +++ b/docs/release-control/v6/internal/subsystems/ai-runtime.md @@ -25,6 +25,36 @@ that same result. Successful reads retain their content and execution provenance ## Purpose +### Incident lifecycle occurrence identity (7 September 2026) + +Fired, resolved and acknowledgement snapshots with a nonzero start time select +an exact (alert identifier, occurrence start) pair, not whichever incident is +open or latest. Replaying a retained closed occurrence must neither allocate a +new ID nor close a newer recurrence. A distinct start remains a distinct +occurrence even within the legacy timeline read tolerance. Zero-start legacy +callers retain best-effort matching. `TestIncidentStore_LifecycleReplayIdentity` +covers unchanged active evaluations, repeated fired/resolved transitions, +JSON checkpoint reload, canonical-backed shells and delayed historical events. +`TestIncidentStore_LifecycleRapidRecurrence` covers distinct subsecond starts. + +This does not migrate existing duplicate records, change bounded retention, +make acknowledgement event replay idempotent, or prove aggregate write-byte +reductions or recipient delivery. An evicted occurrence can still be recreated +by replay; exact identity is guaranteed only while its shell is retained. + +Canonical timeline reads for retained incident shells select events from the +shell's exact opening time up to (but not including) the next retained opening +for the same alert/resource. A newer firing must not reopen an older resolved +incident, and a subsecond recurrence must not inherit its predecessor's +resolution or acknowledgement. Local analysis annotations remain attached to +their shell. This is a read-projection boundary, not event deletion or a change +to notification delivery. Canonical-only fallback when no shell is retained, +missing recurrence boundaries after retention, and already-duplicated shells +remain outside this guarantee. Legacy shell lookup retains its time tolerance; +events preceding the selected shell's exact opening are not projected into it. +`TestIncidentStore_CanonicalProjectionOccurrenceBounds` verifies both boundaries, +subsecond starts, unordered successors and unrelated alert/resource isolation. + ### Unchanged incident JSON checkpoints Incident-memory checkpoints compare the serialized snapshot with bounded bytes diff --git a/docs/release-control/v6/internal/subsystems/performance-and-scalability.md b/docs/release-control/v6/internal/subsystems/performance-and-scalability.md index 71eaeced7..0404a4e2b 100644 --- a/docs/release-control/v6/internal/subsystems/performance-and-scalability.md +++ b/docs/release-control/v6/internal/subsystems/performance-and-scalability.md @@ -15,6 +15,8 @@ ## Purpose +The resource adapter fast delta path must remain content-equivalent to the full merge for explicit Proxmox memory withdrawal. An absent canonical metric plus incoming Proxmox usageUnavailable clears the old display value; store patch operations must emit that clear even when only the raw facet key changed. This bounded per-changed-row check must not introduce an estate-wide scan or defeat untouched-row identity preservation. Adapter tests cover both delta paths and store writes, including trusted-zero recovery and ordinary partial omission. + The PR #1935 log-level parser benchmark remains an unresolved environment-bound observation. Two CI comparisons on unchanged parser source report +10.04 and +10.23 percent for the empty-string case, with stable base/candidate binaries. diff --git a/docs/release-control/v6/internal/subsystems/unified-resources.md b/docs/release-control/v6/internal/subsystems/unified-resources.md index f66a85ba8..86604f092 100644 --- a/docs/release-control/v6/internal/subsystems/unified-resources.md +++ b/docs/release-control/v6/internal/subsystems/unified-resources.md @@ -23,6 +23,8 @@ and sort the complete canonical change table while startup and ingestion wait. ## Purpose +Canonical frontend memory withdrawal is explicit: when a resource snapshot omits the canonical memory metric and its Proxmox memory facet marks usageUnavailable, the display merge must clear any previous metric. Plain partial omission remains compatible with richer REST state, and an incoming canonical metric (including measured zero) takes precedence over unavailable raw evidence. The adapter transition tests pin withdrawal and recovery; the hybrid-memory Chromium fixture exercises the rendered table and drawer at 1280px and 390px. Workload details remain canonical-only: withdrawal shows N/A in the table and removes the Memory section rather than manufacturing a raw-facet total; measured-zero recovery restores Total and Free, with screenshot positioning above fixed navigation. This does not change agent-only or arbitrary field-deletion semantics. + **VM-linked agent memory read — issue #1962 (7 September 2026)** `VMView.LinkedAgentMemory` exposes a value-copy of the agent's own memory sample diff --git a/frontend-modern/browser-verification.json b/frontend-modern/browser-verification.json index bb5b912ba..c211fb630 100644 --- a/frontend-modern/browser-verification.json +++ b/frontend-modern/browser-verification.json @@ -1,55 +1,35 @@ { "version": 1, - "base_sha": "7e34b00d4face3ffa33a36946bb527dbfbdfd74c", - "verified_at": "2026-09-07T16:12:24.652755Z", + "base_sha": "c857802df6b00e61e2229eea411d6675deff2d8a", + "verified_at": "2026-09-07T22:03:54.189527Z", "result": "passed", "changed_paths": [ - "frontend-modern/src/components/AI/Chat/ChatMessages.tsx", - "frontend-modern/src/components/patrol/ApprovalSection.tsx", - "frontend-modern/src/features/patrol/patrolInvestigationContextModel.ts" + "frontend-modern/src/utils/resourceStateAdapters.ts" ], "content_sha256": { - "frontend-modern/src/components/AI/Chat/ChatMessages.tsx": "6f2bea6530d383abe81e13054ef9b7a90f4324c244166fb0a8afaac354439b48", - "frontend-modern/src/components/patrol/ApprovalSection.tsx": "5e7ac87d87674b4c05fe80c3933d4ab542366054e2abbfa3f96afbfe90e993a4", - "frontend-modern/src/features/patrol/patrolInvestigationContextModel.ts": "781c8730a7f566831562160a31ba437b2aabd8840330f59b521894198f79b0f1" + "frontend-modern/src/utils/resourceStateAdapters.ts": "ce1f8f2aef4b04b64dd2533c09458349811ba7cec9ec2e4bd8634b8e9c3c848f" }, "routes": [ - "/patrol", - "/actions?action=act_fd2d0bdab1930e2d3a75faf080b8ba7a", - "/actions?action=act_369c466aea179f93fc69aa04a710689a", - "/actions?action=act_d2f137411342c995d840d529a70fbf95", - "/docs/ASSISTANT_ARCHITECTURE", - "/docs/ASSISTANT_SAFETY" + "/proxmox" ], "viewports": [ { - "width": 1440, - "height": 1000 - }, - { - "width": 900, - "height": 1000 + "width": 1280, + "height": 900 }, { "width": 390, - "height": 1000 + "height": 844 } ], "states": [ - "Approved and independently verified outcome", - "Rejected and expired unexecuted plans", - "Expanded filesystem evidence and risk", - "Attached finding context, explicit new session, retained saved context", - "Fresh streamed explanation and saved missing-access, VM, storage and rejection outcomes" + "Canonical 100% to 35%; stale/offline/unavailable agent evidence retains trusted 100%.", + "Explicit Proxmox withdrawal with omitted canonical memory: table N/A, Memory section absent.", + "Measured-zero recovery: table 0%, Memory Total and Free both 100 MB; second cluster stays 80%." ], "interactions": [ - "Open review, close, focus return and Enter reopen", - "Expand every investigation tool card, deep evidence scroll and actual pixel inspection", - "Discuss with Assistant, new-session clearing and saved-session resumption", - "Exact action link, terminal authority controls, reload and Escape", - "Stream at bottom, wheel overscroll containment, scroll away, Latest, resize while open and reachable header/composer", - "Session-history picker and exact persisted message read", - "Documentation linked navigation, reload and end-of-document scroll" + "Open guest drawer; deliver synthetic canonical socket snapshots while open.", + "Resize to desktop and narrow Chromium; scroll recovered Memory into view above fixed navigation; inspect all five final screenshots." ], - "notes": "Final runtime r34 SHA256 e1ae053e656b28af9ba0e7541a47bdeb56303d1d7912edc57b2dac2f94620bd1 matches 4,853 source inputs. Named r28 fault matrix and subsequent affected writer/streaming proofs are detailed in docs/qualification/PATROL_ASSISTANT_CUSTOMER_JOURNEY.md. Fresh r34 approved explanation passed, with saved final-runtime missing-access/VM/rejected/storage explanations and deep Patrol history. Receipts: workspace tmp/patrol-planning-continuation/patrol-browser and docs-browser-r34. Initial history-locator and network-idle failures are retained separately. Only login, selected authorized API provider readiness and one explicit fresh explanation were submitted during these checks. No lab action was repeated. Independent customer environments and broader readiness remain unqualified." + "notes": "Synthetic Desktop Chrome acceptance only, not touch/mobile-device, installed poller, FreeBSD reporter or alert-delivery proof. Base reproduced stale 100% after withdrawal; adapter repair passed final Chromium test without retries and pageerror assertion. Earlier fixture wrongly expected raw-facet total after withdrawal; useWorkloads maps canonical metrics only. Two selector/absent-section failures and first passing but bottom-nav-obscured screenshot run are retained, not counted as final visual acceptance. Final four transition screenshots show withdrawn section absent and recovered Total/Free in frame at both widths; initial 35% screenshot is table-only. Evidence: /var/lib/pulse-maintainer/queue/staging/20260907T215015Z-web-product/ (moves to completed): browser-base.log, browser-repair.log, browser-repair-selector.log, browser-canonical.log, browser-final.log, final-report/data/*.png. Browser command: pulse-heavy-run -- bash wrapper starting Vite 127.0.0.1:5187 then PLAYWRIGHT_BASE_URL=http://127.0.0.1:5187 playwright test tests/86-hybrid-memory-browser.spec.ts --project=chromium --retries=0; Vite stopped by wrapper trap." } diff --git a/frontend-modern/src/utils/__tests__/resourceStateAdapters.test.ts b/frontend-modern/src/utils/__tests__/resourceStateAdapters.test.ts index 797ee03f2..da7e9dbfb 100644 --- a/frontend-modern/src/utils/__tests__/resourceStateAdapters.test.ts +++ b/frontend-modern/src/utils/__tests__/resourceStateAdapters.test.ts @@ -1181,6 +1181,58 @@ describe('resourceStateAdapters unavailable memory contract', () => { }, } as unknown as Resource['proxmox']; + it('clears a previous metric when a snapshot explicitly reports unavailable memory', () => { + const previous = createNodeResource({}); + const incoming = { ...previous, proxmox: unavailableProxmoxMemory }; + delete incoming.memory; // JSON omitempty, not an explicit undefined property. + const [merged] = mergeCanonicalResourceSnapshot([incoming], [previous]); + expect(nodeFromResource(merged)?.memory.usageUnavailable).toBe(true); + expect(merged.memory).toBeUndefined(); + }); + + it('clears withdrawn memory through both delta paths and emits the store clear', () => { + const previous = { + ...createNodeResource({}), + type: 'vm', + proxmox: unavailableProxmoxMemory, + } as Resource; + const incoming = { ...previous }; + delete incoming.memory; + for (const keys of [undefined, new Map([[previous.id, ['memory']]])]) { + const [merged] = mergeCanonicalResourceDeltaSnapshot( + [incoming], + [previous], + new Set([previous.id]), + keys, + ); + expect(merged.memory).toBeUndefined(); + expect(buildFastResourceStorePatchOps(merged, ['memory'])).toEqual([ + { key: 'memory', value: undefined, mode: 'set' }, + ]); + expect(buildFastResourceStorePatchOps(merged, ['proxmox'])).toContainEqual({ + key: 'memory', + value: undefined, + mode: 'set', + }); + } + }); + + it('retains richer memory on partial omission without unavailable evidence', () => { + const previous = createNodeResource({}); + const incoming = { ...previous }; + delete incoming.memory; + const [merged] = mergeCanonicalResourceSnapshot([incoming], [previous]); + expect(merged.memory).toEqual(previous.memory); + }); + + it('accepts a newly trusted metric after an unavailable snapshot', () => { + const previous = { ...createNodeResource({}), proxmox: unavailableProxmoxMemory }; + delete previous.memory; + const incoming = { ...previous, memory: { current: 0, total: 8192, used: 0 } }; + const [merged] = mergeCanonicalResourceSnapshot([incoming], [previous]); + expect(nodeFromResource(merged)?.memory).toMatchObject({ usage: 0, usageUnavailable: false }); + }); + it('preserves explicit unavailable usage when no trusted metric exists', () => { const node = nodeFromResource({ ...createNodeResource({}), diff --git a/frontend-modern/src/utils/resourceStateAdapters.ts b/frontend-modern/src/utils/resourceStateAdapters.ts index c9617893d..8315e0e87 100644 --- a/frontend-modern/src/utils/resourceStateAdapters.ts +++ b/frontend-modern/src/utils/resourceStateAdapters.ts @@ -867,6 +867,13 @@ const mergeCanonicalSourceFacet = ( ? mergeRecord(incomingFacet, existingFacet) : incomingFacet; +// A missing field alone can be a partial snapshot. Explicit unavailable raw +// evidence plus no canonical metric means the producer has withdrawn memory; +// retaining the previous display metric would hide that state. +const hasWithdrawnProxmoxMemory = (incoming: Resource): boolean => + incoming.memory == null && + asBoolean(asRecord(asRecord(incoming.proxmox)?.memory)?.usageUnavailable) === true; + export const mergeCanonicalResource = (incoming: Resource, existing?: Resource): Resource => { if (!existing) { return canonicalizeRealtimeResource(incoming); @@ -876,6 +883,7 @@ export const mergeCanonicalResource = (incoming: Resource, existing?: Resource): return { ...existingCanonical, ...incoming, + ...(hasWithdrawnProxmoxMemory(incoming) ? { memory: undefined } : {}), clusterId: incoming.clusterId ?? existingCanonical.clusterId, platformScopes: normalizeSourcePlatformScopes( incoming.platformScopes ?? existingCanonical.platformScopes, @@ -1105,8 +1113,8 @@ const applyFastResourceMergePatch = ( (platformDataLeaves ??= []).push(key.slice(FAST_MERGE_PLATFORM_DATA_PREFIX.length)); continue; } - // A merge-patch deletion removed the key from the raw row; the full merge's - // `...incoming` spread would keep the existing display value, so keep it. + // A merge-patch deletion normally preserves the richer display value. + // Explicitly withdrawn Proxmox memory is cleared below, as in the full merge. if (!(key in rawIncoming)) continue; const value = rawIncoming[key]; if (key === 'proxmox') { @@ -1137,6 +1145,7 @@ const applyFastResourceMergePatch = ( } next.platformData = nextPlatformData; } + if (hasWithdrawnProxmoxMemory(incoming)) next.memory = undefined; return next as unknown as Resource; }; @@ -1159,6 +1168,11 @@ export const buildFastResourceStorePatchOps = ( ): FastResourceStorePatchOp[] => { const record = rawStoreValue(row as unknown as JsonRecord); const ops: FastResourceStorePatchOp[] = []; + // Unavailability can also arrive with a raw Proxmox facet change rather + // than a memory key (the canonical metric may already be absent on wire). + if (!keys.includes('memory') && hasWithdrawnProxmoxMemory(row)) { + ops.push({ key: 'memory', value: undefined, mode: 'set' }); + } const platformData = asRecord(record.platformData); for (const key of keys) { if (key.startsWith(FAST_MERGE_PLATFORM_DATA_PREFIX)) { diff --git a/internal/ai/memory/incidents.go b/internal/ai/memory/incidents.go index fa02ff75e..2ca35edb6 100644 --- a/internal/ai/memory/incidents.go +++ b/internal/ai/memory/incidents.go @@ -361,7 +361,10 @@ func (s *IncidentStore) RecordAlertFired(alert *alerts.Alert) { s.mu.Lock() defer s.mu.Unlock() - shell := s.findOpenIncidentByAlertIdentifierLocked(alert.ID) + shell := s.findLifecycleOccurrenceLocked(alert) + if alert.StartTime.IsZero() { + shell = s.findOpenIncidentByAlertIdentifierLocked(alert.ID) + } if shell == nil { shell = newIncidentShellFromAlert(alert) s.incidents = append(s.incidents, shell) @@ -373,6 +376,9 @@ func (s *IncidentStore) RecordAlertFired(alert *alerts.Alert) { "threshold": alert.Threshold, }) } + } else if incidentOccurrenceClosedAt(shell) != nil { + // A replayed fired event must not reopen a completed occurrence. + return } else { updateIncidentShellFromAlert(shell, alert) } @@ -434,12 +440,20 @@ func (s *IncidentStore) RecordAlertResolved(alert *alerts.Alert, resolvedAt time s.mu.Lock() defer s.mu.Unlock() - shell := s.findOpenIncidentByAlertIdentifierLocked(alert.ID) + shell := s.findLifecycleOccurrenceLocked(alert) + if alert.StartTime.IsZero() { + shell = s.findOpenIncidentByAlertIdentifierLocked(alert.ID) + } if shell == nil { shell = newIncidentShellFromAlert(alert) s.incidents = append(s.incidents, shell) } + // Resolution replay is idempotent, including after a checkpoint reload. + if incidentOccurrenceClosedAt(shell) != nil { + return + } + if resolvedAt.IsZero() { now := time.Now() resolvedAt = now @@ -999,6 +1013,20 @@ func (s *IncidentStore) loadProjectedIncidentEvents(incident *Incident, timeline since = since.Add(-incidentStartMatchTolerance) } + // Retained shells identify exact occurrences. Canonical history is keyed by + // alert identifier, so without an upper bound an old incident absorbs every + // later recurrence and can appear open again. Do not use the legacy read + // tolerance to include events from a preceding subsecond occurrence either. + var nextStart time.Time + s.mu.RLock() + for _, shell := range s.incidents { + if shell != nil && shell.AlertIdentifier == alertIdentifier && shell.ResourceID == resourceID && + shell.OpenedAt.After(incident.OpenedAt) && (nextStart.IsZero() || shell.OpenedAt.Before(nextStart)) { + nextStart = shell.OpenedAt + } + } + s.mu.RUnlock() + changes, err := timelineStore.GetRecentChanges(resourceID, since, projectedIncidentChangeLimit) if err != nil || len(changes) == 0 { return nil @@ -1013,6 +1041,10 @@ func (s *IncidentStore) loadProjectedIncidentEvents(incident *Incident, timeline if !ok { continue } + if (!incident.OpenedAt.IsZero() && event.Timestamp.Before(incident.OpenedAt)) || + (!nextStart.IsZero() && !event.Timestamp.Before(nextStart)) { + continue + } events = append(events, event) hydrateIncidentFromCanonicalChange(incident, change) } @@ -1320,8 +1352,28 @@ func updateIncidentShellFromAlert(shell *incidentShell, alert *alerts.Alert) { shell.Message = alert.Message } +// Lifecycle snapshots carry the exact occurrence start. Unlike legacy read +// repair, they must not use a time tolerance or select an unrelated open/latest +// incident: delayed transitions and replay can arrive after a genuine recurrence. +func (s *IncidentStore) findLifecycleOccurrenceLocked(alert *alerts.Alert) *incidentShell { + if alert.ID == "" { + return nil + } + if alert.StartTime.IsZero() { + // Legacy callers without an occurrence key retain best-effort matching. + return s.findLatestIncidentByAlertIdentifierLocked(alert.ID) + } + for i := len(s.incidents) - 1; i >= 0; i-- { + shell := s.incidents[i] + if shell != nil && shell.AlertIdentifier == alert.ID && shell.OpenedAt.Equal(alert.StartTime) { + return shell + } + } + return nil +} + func (s *IncidentStore) ensureIncidentForAlertLocked(alert *alerts.Alert) *incidentShell { - shell := s.findLatestIncidentByAlertIdentifierLocked(alert.ID) + shell := s.findLifecycleOccurrenceLocked(alert) if shell == nil { shell = newIncidentShellFromAlert(alert) s.incidents = append(s.incidents, shell) diff --git a/internal/ai/memory/incidents_test.go b/internal/ai/memory/incidents_test.go index e271d2133..3540d5f53 100644 --- a/internal/ai/memory/incidents_test.go +++ b/internal/ai/memory/incidents_test.go @@ -1059,3 +1059,152 @@ func TestIncidentStore_RecordNote_NonexistentIncident(t *testing.T) { t.Error("expected false for non-existent incident") } } + +// Lifecycle replay may revisit a closed occurrence after checkpoint restart, +// including while a newer occurrence of the same alert is already active. +func TestIncidentStore_LifecycleReplayIdentity(t *testing.T) { + for _, tc := range []struct { + name string + restart, canonical bool + }{ + {name: "memory"}, + {name: "checkpoint-restart", restart: true}, + {name: "canonical", canonical: true}, + {name: "canonical-checkpoint-restart", restart: true, canonical: true}, + } { + t.Run(tc.name, func(t *testing.T) { + store := NewIncidentStore(IncidentStoreConfig{}) + wantClosedEvents, wantOpenEvents := 2, 1 + if tc.canonical { + store.SetResourceTimelineStore(unifiedresources.NewMemoryStore()) + wantClosedEvents, wantOpenEvents = 0, 0 + } + start := time.Now().Add(-time.Hour).UTC() + old := &alerts.Alert{ID: "pbs-connectivity", ResourceID: "pbs", StartTime: start} + store.RecordAlertFired(old) + for i := 0; i < 10; i++ { + store.RecordAlertFired(old) + } + if len(store.incidents) != 1 { + t.Fatal("unchanged active evaluations duplicated incident identity") + } + oldID := store.incidents[0].ID + end := start.Add(time.Minute) + store.RecordAlertResolved(old, end) + if tc.restart { + // Synchronous JSON checkpoint/reload isolates application lifecycle + // identity from asynchronous scheduling and external event stores. + store.dataDir = t.TempDir() + store.filePath = store.dataDir + "/ai_incidents.json" + if err := store.saveToDisk(); err != nil { + t.Fatal(err) + } + reloaded := NewIncidentStore(IncidentStoreConfig{}) + reloaded.filePath = store.filePath + if err := reloaded.loadFromDisk(); err != nil { + t.Fatal(err) + } + store = reloaded + if tc.canonical { + store.SetResourceTimelineStore(unifiedresources.NewMemoryStore()) + } + } + for i := 0; i < 10; i++ { + store.RecordAlertFired(old) + store.RecordAlertResolved(old, end) + } + if len(store.incidents) != 1 { + t.Fatalf("replayed one occurrence produced %d distinct incident shells", len(store.incidents)) + } + if store.incidents[0].ID != oldID || len(store.incidents[0].Events) != wantClosedEvents { + t.Fatal("replay changed incident identity or duplicated lifecycle events") + } + newAlert := old.Clone() + newAlert.StartTime = end.Add(time.Second) + store.RecordAlertFired(newAlert) + if len(store.incidents) != 2 { + t.Fatal("genuine recurrence did not open a separate incident") + } + newID := store.incidents[1].ID + store.RecordAlertResolved(old, end) + store.RecordAlertAcknowledged(old, "historical-actor") + if len(store.incidents) != 2 || store.incidents[1].ID != newID || store.incidents[1].OccurrenceClosedAt != nil || len(store.incidents[1].Events) != wantOpenEvents { + t.Fatal("late historical lifecycle event mutated the active recurrence") + } + }) + } +} + +func TestIncidentStore_LifecycleRapidRecurrence(t *testing.T) { + store := NewIncidentStore(IncidentStoreConfig{}) + first := &alerts.Alert{ID: "same-alert", StartTime: time.Now().Add(-time.Minute)} + store.RecordAlertFired(first) + // A missed resolution must not merge distinct starts, even within the + // tolerance used by legacy timeline reads. + second := first.Clone() + second.StartTime = first.StartTime.Add(500 * time.Millisecond) + store.RecordAlertFired(second) + if len(store.incidents) != 2 { + t.Fatalf("distinct occurrence starts merged: got %d incident shells", len(store.incidents)) + } + store.RecordAlertResolved(first, first.StartTime.Add(100*time.Millisecond)) + if store.incidents[1].OccurrenceClosedAt != nil { + t.Fatal("old resolution closed rapid recurrence") + } +} + +func TestIncidentStore_CanonicalProjectionOccurrenceBounds(t *testing.T) { + for _, gap := range []time.Duration{2 * time.Minute, 500 * time.Millisecond} { + t.Run(gap.String(), func(t *testing.T) { + store := NewIncidentStore(IncidentStoreConfig{}) + canonical := unifiedresources.NewMemoryStore() + store.SetResourceTimelineStore(canonical) + start := time.Now().UTC().Add(-time.Hour) + old := &alerts.Alert{ID: "alert-bounds", ResourceID: "pbs", StartTime: start} + store.RecordAlertFired(old) + // Deliberately insert successors out of time order. The earliest + // matching next start, not insertion order, is the upper boundary. + for _, offset := range []time.Duration{3 * gap, gap} { + next := old.Clone() + next.StartTime = start.Add(offset) + store.RecordAlertFired(next) + } + // Neither another alert nor another resource may shorten the window. + other := old.Clone() + other.ID = "unrelated-alert" + other.StartTime = start.Add(gap / 4) + store.RecordAlertFired(other) + other.ID = old.ID + other.ResourceID = "other-resource" + other.StartTime = start.Add(gap / 3) + store.RecordAlertFired(other) + + end := start.Add(gap - time.Nanosecond) + for _, event := range []struct { + at time.Time + kind unifiedresources.ChangeKind + }{ + {start.Add(-time.Nanosecond), unifiedresources.ChangeAlertResolved}, + {start, unifiedresources.ChangeAlertFired}, + {end, unifiedresources.ChangeAlertResolved}, + {start.Add(gap), unifiedresources.ChangeAlertFired}, + {start.Add(2 * gap), unifiedresources.ChangeAlertAcknowledged}, + } { + change := unifiedresources.BuildAlertTimelineChange(old.ResourceID, event.kind, event.at, "", unifiedresources.AlertTimelineChange{AlertIdentifier: old.ID}) + if err := canonical.RecordChange(*change); err != nil { + t.Fatal(err) + } + } + projected := store.GetTimelineByAlertAt(old.ID, start) + if projected == nil || len(projected.Events) != 2 { + t.Fatalf("expected only this occurrence's fired/resolved events, got %+v", projected) + } + if !projected.Events[0].Timestamp.Equal(start) || !projected.Events[1].Timestamp.Equal(end) { + t.Fatalf("wrong inclusive lower/exclusive upper boundary: %+v", projected.Events) + } + if projected.Status != IncidentStatusResolved || projected.ClosedAt == nil || !projected.ClosedAt.Equal(end) || projected.Acknowledged { + t.Fatalf("another occurrence changed historical state: %+v", projected) + } + }) + } +} diff --git a/internal/monitoring/monitor_alert_handling_test.go b/internal/monitoring/monitor_alert_handling_test.go index 945a5c3ea..f60cd108c 100644 --- a/internal/monitoring/monitor_alert_handling_test.go +++ b/internal/monitoring/monitor_alert_handling_test.go @@ -1095,3 +1095,76 @@ func TestMonitor_HandleAlertEscalated_BypassesDeliveryCooldown(t *testing.T) { t.Fatal("expected escalated notification delivery despite active cooldown") } } + +// Exercise the dispatcher used by lifecycle replay with both projections +// attached. Store-only tests cannot detect cross-occurrence canonical history. +func TestMonitorLifecycleReplayPreservesOccurrenceTimelines(t *testing.T) { + for _, gap := range []time.Duration{2 * time.Minute, 500 * time.Millisecond} { + t.Run(gap.String(), func(t *testing.T) { + store := unifiedresources.NewMemoryStore() + incidents := memory.NewIncidentStore(memory.IncidentStoreConfig{}) + m := &Monitor{ + incidentStore: incidents, + resourceStore: unifiedresources.NewMonitorAdapter(unifiedresources.NewRegistry(store)), + } + incidents.SetResourceTimelineStore(m.resourceStore.(memory.IncidentTimelineStore)) + start := time.Now().UTC().Add(-time.Hour).Truncate(time.Second) + old := &alerts.Alert{ID: "pbs-connectivity", ResourceID: "pbs", StartTime: start} + end := start.Add(gap / 2) + fired := alerts.LifecycleEvent{Type: eventlog.TypeFired, OccurredAt: start, Alert: old} + resolved := alerts.LifecycleEvent{Type: eventlog.TypeResolved, OccurredAt: end, Alert: old} + m.handleAlertLifecycleEvent(fired) + m.handleAlertLifecycleEvent(resolved) + first := incidents.GetTimelineByAlertAt(old.ID, start) + require.NotNil(t, first) + for i := 0; i < 10; i++ { + m.handleAlertLifecycleEvent(fired) + m.handleAlertLifecycleEvent(resolved) + } + require.Len(t, incidents.ListIncidentsByResource(old.ResourceID, 0), 1) + replayed := incidents.GetTimelineByAlertAt(old.ID, start) + require.Equal(t, first.ID, replayed.ID) + require.Equal(t, memory.IncidentStatusResolved, replayed.Status) + require.Len(t, replayed.Events, 2) + + next := old.Clone() + next.StartTime = start.Add(gap) + m.handleAlertLifecycleEvent(alerts.LifecycleEvent{Type: eventlog.TypeFired, OccurredAt: next.StartTime, Alert: next}) + // A historical resolution arriving after recurrence must not close it; + // conversely, the recurrence must not reopen the historical timeline. + m.handleAlertLifecycleEvent(resolved) + require.Len(t, incidents.ListIncidentsByResource(old.ResourceID, 0), 2) + historical := incidents.GetTimelineByAlertAt(old.ID, start) + current := incidents.GetTimelineByAlertAt(next.ID, next.StartTime) + require.NotNil(t, historical) + require.NotNil(t, current) + require.Equal(t, first.ID, historical.ID) + require.NotEqual(t, historical.ID, current.ID) + require.Equal(t, memory.IncidentStatusResolved, historical.Status) + require.Equal(t, &end, historical.ClosedAt) + require.Len(t, historical.Events, 2) + require.Equal(t, memory.IncidentStatusOpen, current.Status) + require.Nil(t, current.ClosedAt) + require.Len(t, current.Events, 1) + changes, err := store.GetRecentChanges(old.ResourceID, time.Time{}, 100) + require.NoError(t, err) + require.Len(t, changes, 3) + + ackAt := start.Add(gap / 4) + old.AckTime = &ackAt + m.handleAlertLifecycleEvent(alerts.LifecycleEvent{ + Type: eventlog.TypeAcknowledged, OccurredAt: ackAt, Alert: old, + Details: map[string]string{"user": "historical-operator"}, + }) + historical = incidents.GetTimelineByAlertAt(old.ID, start) + current = incidents.GetTimelineByAlertAt(next.ID, next.StartTime) + require.Equal(t, memory.IncidentStatusResolved, historical.Status) + require.True(t, historical.Acknowledged) + require.Equal(t, "historical-operator", historical.AckUser) + require.Len(t, historical.Events, 3) + require.False(t, current.Acknowledged) + require.Equal(t, memory.IncidentStatusOpen, current.Status) + require.Len(t, current.Events, 1) + }) + } +} diff --git a/tests/integration/tests/86-hybrid-memory-browser.spec.ts b/tests/integration/tests/86-hybrid-memory-browser.spec.ts index 7dda194ce..c9b26c664 100644 --- a/tests/integration/tests/86-hybrid-memory-browser.spec.ts +++ b/tests/integration/tests/86-hybrid-memory-browser.spec.ts @@ -163,17 +163,73 @@ test("hybrid guest memory follows canonical snapshots in the browser", async ({ page.locator(".workload-row").filter({ hasText: "cluster-b-vm" }), ).toContainText("80%"); } - resources = [vm("cluster-a", 0), vm("cluster-b", 80)]; - send(); - await expect(memoryCell).toContainText("0%"); - await expect(memoryCell).not.toContainText("N/A"); - await expect(drawer).toContainText("100 MB"); - await expect( - page.locator(".workload-row").filter({ hasText: "cluster-b-vm" }), - ).toContainText("80%"); - expect(errors).toEqual([]); - await testInfo.attach("hybrid-memory-measured-zero", { - body: await page.screenshot(), - contentType: "image/png", + // Workload details use canonical metrics only: withdrawal removes the Memory + // section; recovery restores it. Do not invent Usage UI or raw-facet totals. + const memoryDetails = drawer.locator("tbody").filter({ + has: page.locator("th").filter({ hasText: /^Memory$/ }), }); + for (const viewport of [ + { width: 1280, height: 900 }, + { width: 390, height: 844 }, + ]) { + await page.setViewportSize(viewport); + resources = [ + { + ...vm("cluster-a"), + proxmox: { + ...vm("cluster-a").proxmox, + memory: { + total: 100 * 1024 * 1024, + used: 0, + free: 0, + usage: 0, + usageUnavailable: true, + }, + }, + } as ReturnType, + vm("cluster-b", 80), + ]; + send(); + await expect(memoryCell).toContainText("N/A"); + await expect(memoryCell).not.toContainText("100%"); + await expect(memoryDetails).toHaveCount(0); + await expect(drawer.getByText("Free", { exact: true })).toHaveCount(0); + await drawer.scrollIntoViewIfNeeded(); + await expect(drawer).toBeInViewport(); + await expect( + page.locator(".workload-row").filter({ hasText: "cluster-b-vm" }), + ).toContainText("80%"); + await testInfo.attach(`withdrawn-memory-${viewport.width}`, { + body: await page.screenshot(), + contentType: "image/png", + }); + + resources = [vm("cluster-a", 0), vm("cluster-b", 80)]; + send(); + await expect(memoryCell).toContainText("0%"); + await expect(memoryCell).not.toContainText("N/A"); + // Centre the recovered details above the fixed narrow-screen navigation. + await memoryDetails.evaluate((element) => + element.scrollIntoView({ block: "center", inline: "nearest" }), + ); + await expect.poll(async () => { + const box = await memoryDetails.boundingBox(); + return box ? box.y + box.height : Infinity; + }).toBeLessThan(viewport.height - 64); + await expect(memoryDetails).toBeInViewport({ ratio: 1 }); + await expect(memoryDetails).toContainText("100 MB"); + await expect( + memoryDetails.locator("tr").filter({ + has: page.getByText("Free", { exact: true }), + }), + ).toContainText("100 MB"); + await expect( + page.locator(".workload-row").filter({ hasText: "cluster-b-vm" }), + ).toContainText("80%"); + await testInfo.attach(`recovered-memory-${viewport.width}`, { + body: await page.screenshot(), + contentType: "image/png", + }); + } + expect(errors).toEqual([]); });