alerts: keep only the newest evidence envelope in history entries (#3096)

## Why
After #3093, the remaining weight of the alert history fold is evidence
nothing reads. On the long-running dogfood install, a resolved snapshot
averages 21 KB, and 83% of that is `evidence`: a trail of up to 64
envelopes (35 on average), across 1,895 occurrences kept for 90 days
(#1803).

Nothing in Pulse reads `Alert.evidence` from history. The incident
panel's evidence comes from the incident timeline endpoint, the AI's
alert history uses recently resolved alerts, and the attention summary
and quality telemetry never touch it.

## What changed
- The fold trims each decoded snapshot to its newest envelope, by
`observedAt` and then position. An updated envelope keeps its slot, so
the last one isn't always the newest. Merging takes the incoming
snapshot's evidence, so the fold never holds a full trail.
- History output applies the same rule to live overlays and the JSON
fallback, so every `GET /api/alerts/history` entry carries at most one
envelope.
- `operationalRecord.evidenceIds` and `transitions[].evidenceIds` keep
their full lists. The event log keeps every snapshot whole, and `GET
/api/alerts/active` still returns an open alert's full evidence.
- The api-contracts history clause, `docs/API.md` (and its shipped
mirror) and the alerts contract say so.

This is an intentional change to a public API response, approved by the
maintainer. A client that read full evidence from history should read
the active alert, the incident timeline or the event log.

## Measured
On a log shaped like the dogfood install's (1,900 occurrences, 35
envelopes each), the history fold retains 2.1 MB of heap, down from 27.2
MB.

## Checks (pulse-dev)
- New tests:
  - newest by time and tie-breaking;
  - the fold holds one envelope while the record keeps all 64 IDs;
  - the event log stays whole;
  - live overlay and JSON fallback;
  - direct output shaping, with its input untouched;
  - `docs/API.md` states the rule.
- Negative controls: keeping full evidence in the fold, in the output,
or choosing by position each fail their test.
- `internal/alerts/...`, `internal/api/alerting/...` and
`docsLinks.test.ts` pass. Prettier, vet, the completion, contract,
docs-mirror and browser-verification guards all pass.

Related to #1803

docs/API.md

@@ -1070,7 +1070,7 @@ it, use the snooze and unsnooze endpoints below.
 - `GET /api/alerts/active`
 - `GET /api/alerts/delivery-diagnosis?alertIdentifier=<alert-id>` (omit `alertIdentifier` to get the diagnosis array for every active alert)
 - `GET /api/alerts/events?alertIdentifier=<alert-id>&type=<event-type,...>&since=<RFC3339>&limit=<n>` — append-only alert event log: lifecycle transitions and notification decisions, including suppressions with reasons; all parameters optional, newest first
-- `GET /api/alerts/history`
+- `GET /api/alerts/history` — each entry carries at most one `evidence` envelope, the newest; `operationalRecord.evidenceIds` keeps the full list, and `GET /api/alerts/active` returns an open alert's full evidence
 - `DELETE /api/alerts/history`
 - `GET /api/alerts/incidents`
 - `POST /api/alerts/incidents/note`

docs/release-control/v6/internal/subsystems/alerts.md

@@ -1041,6 +1041,13 @@ independent walk of only the window, and
 `TestWindowedHistoryReusesTheFullHistoryFold` and
 `TestHistoryReadResultsDoNotAliasTheSharedFold` pin the copy, reuse and
 isolation rules.
+The fold keeps only each snapshot's newest evidence envelope
+(`newestHistoryEvidence`, by `observedAt` then position), and history output
+applies the same rule to live overlays and the JSON fallback. A resolved
+alert's trail of up to 64 envelopes was about 83% of its snapshot. The event log
+still stores every snapshot whole. `TestHistoryEntriesKeepOnlyTheNewestEvidence`,
+`TestHistoryOverlayKeepsOnlyTheNewestEvidence` and
+`TestHistoryFallbackKeepsOnlyTheNewestEvidence` pin this.
 
 
 PBS node-status availability is distinct from connectivity. While a connected

docs/release-control/v6/internal/subsystems/api-contracts.md

@@ -645,7 +645,14 @@ grouping-key fields while silently ignoring `enabled`.
 `warning`, and `critical` (plus omitted/`all` for no severity filter) and
 returns only matching rows after the time and resource filters. Alert payloads
 use the same closed three-level vocabulary; legacy `error` data is normalized
-to `critical` before it crosses the API boundary. Notification configuration
+to `critical` before it crosses the API boundary.
+Each history entry carries at most one `evidence` envelope, the newest by
+`observedAt`, whether it comes from the event-log projection, a live overlay or
+the JSON fallback (#1803). `operationalRecord.evidenceIds` and
+`transitions[].evidenceIds` keep their full lists, so some identifiers name
+envelopes the entry omits. The alert event log keeps every snapshot whole, and
+`GET /api/alerts/active` returns an open alert's full evidence.
+Notification configuration
 round-trips destination minimum severity as `all`, `warning`, or `critical`.
 The Relay adapter has a deliberately narrower persisted floor of `all` or
 `critical`, but an all-alert mobile payload still preserves canonical `info`

frontend-modern/public/docs/API.md

@@ -1070,7 +1070,7 @@ it, use the snooze and unsnooze endpoints below.
 - `GET /api/alerts/active`
 - `GET /api/alerts/delivery-diagnosis?alertIdentifier=<alert-id>` (omit `alertIdentifier` to get the diagnosis array for every active alert)
 - `GET /api/alerts/events?alertIdentifier=<alert-id>&type=<event-type,...>&since=<RFC3339>&limit=<n>` — append-only alert event log: lifecycle transitions and notification decisions, including suppressions with reasons; all parameters optional, newest first
-- `GET /api/alerts/history`
+- `GET /api/alerts/history` — each entry carries at most one `evidence` envelope, the newest; `operationalRecord.evidenceIds` keeps the full list, and `GET /api/alerts/active` returns an open alert's full evidence
 - `DELETE /api/alerts/history`
 - `GET /api/alerts/incidents`
 - `POST /api/alerts/incidents/note`

frontend-modern/src/utils/__tests__/docsLinks.test.ts

@@ -220,6 +220,19 @@ describe('docsLinks', () => {
     expect(handler).toContain('agentID != "" && len(*compatibility.ObservationLocationIDs) == 1');
   });
 
+  it('documents alert history entries carrying only the newest evidence envelope', () => {
+    const api = readFileSync(path.join(repoRoot, 'docs', 'API.md'), 'utf8');
+    expect(api).toContain(
+      '`GET /api/alerts/history` — each entry carries at most one `evidence` envelope, the newest',
+    );
+    expect(api).toContain('`operationalRecord.evidenceIds` keeps the full list');
+    const projection = readFileSync(
+      path.join(repoRoot, 'internal', 'alerts', 'history_projection.go'),
+      'utf8',
+    );
+    expect(projection).toContain('snapshot.Evidence = newestHistoryEvidence(snapshot.Evidence)');
+  });
+
   it('documents alert configuration saves as replacing only the keys they send', () => {
     const api = readFileSync(path.join(repoRoot, 'docs', 'API.md'), 'utf8');
     expect(api).toContain('`PUT /api/alerts/config` — replaces only the top-level keys the body');

internal/alerts/canonical_identity.go

@@ -407,6 +407,9 @@ func canonicalizeAlertHistoryForOutput(history []Alert) []Alert {
 		if exportedAlert == nil {
 			continue
 		}
+		// Live overlays and the JSON fallback follow the projection's rule:
+		// a history entry carries only its newest evidence envelope.
+		exportedAlert.Evidence = newestHistoryEvidence(exportedAlert.Evidence)
 		exported = append(exported, *exportedAlert)
 	}
 	return exported

internal/alerts/history_projection.go

@@ -15,8 +15,28 @@ import (
 	"time"
 
 	"github.com/rcourtman/pulse-go-rewrite/internal/alerts/eventlog"
+	"github.com/rcourtman/pulse-go-rewrite/internal/operationaltrust"
 )
 
+// newestHistoryEvidence keeps the newest evidence envelope, by ObservedAt and
+// then position, in a fresh slice. A history entry carries at most this one:
+// a resolved alert's full trail of up to maxOperationalEvidencePerAlert
+// envelopes was most of each entry's size, and nothing reads it from history
+// (#1803). The event log keeps every snapshot whole, and operationalRecord and
+// transitions keep their full evidence ID lists.
+func newestHistoryEvidence(evidence []operationaltrust.EvidenceEnvelope) []operationaltrust.EvidenceEnvelope {
+	if len(evidence) == 0 {
+		return nil
+	}
+	newest := 0
+	for i := range evidence {
+		if !evidence[i].ObservedAt.Before(evidence[newest].ObservedAt) {
+			newest = i
+		}
+	}
+	return []operationaltrust.EvidenceEnvelope{evidence[newest].Clone()}
+}
+
 // historyEventMark is what one folded lifecycle event contributed to its
 // occurrence. The marks let a Since window be answered from the full fold
 // instead of decoding every snapshot in the window again (#1803).
@@ -182,6 +202,9 @@ func (p *historyProjection) fold(throughID int64) error {
 		if err := json.Unmarshal(event.Snapshot, &snapshot); err != nil {
 			return nil
 		}
+		// History keeps only the newest evidence envelope. Merging takes the
+		// incoming snapshot's evidence, so trimming here is all the fold holds.
+		snapshot.Evidence = newestHistoryEvidence(snapshot.Evidence)
 		key := historyOccurrenceKey(event.AlertID, &snapshot)
 		occ, exists := p.occurrences[key]
 		if !exists {

internal/alerts/history_projection_parity_test.go

@@ -19,6 +19,7 @@ import (
 	"time"
 
 	"github.com/rcourtman/pulse-go-rewrite/internal/alerts/eventlog"
+	"github.com/rcourtman/pulse-go-rewrite/internal/operationaltrust"
 )
 
 func TestHistoryProjectionIncrementalReadersPreserveChronologyAndIsolation(t *testing.T) {
@@ -554,3 +555,130 @@ func TestLimitedHistoryReadCopiesOnlyReturnedRows(t *testing.T) {
 		t.Fatalf("a 10-row read allocated %d bytes against %d for all 2000 rows; it must not copy the whole fold", ten, everything)
 	}
 }
+
+// historyEvidence builds n envelopes whose ObservedAt order differs from their
+// position, so "newest" must be chosen by time, not by slice order.
+func historyEvidence(base time.Time, n int, newestIndex int) []operationaltrust.EvidenceEnvelope {
+	evidence := make([]operationaltrust.EvidenceEnvelope, 0, n)
+	for i := 0; i < n; i++ {
+		observed := base.Add(time.Duration(n-i) * time.Second)
+		if i == newestIndex {
+			observed = base.Add(time.Hour)
+		}
+		evidence = append(evidence, operationaltrust.EvidenceEnvelope{ID: fmt.Sprintf("ev-%02d", i), ObservedAt: observed})
+	}
+	return evidence
+}
+
+func TestNewestHistoryEvidenceChoosesByObservationTime(t *testing.T) {
+	base := time.Date(2026, 9, 1, 0, 0, 0, 0, time.UTC)
+	if got := newestHistoryEvidence(nil); got != nil {
+		t.Fatalf("no evidence must stay nil, got %+v", got)
+	}
+	got := newestHistoryEvidence(historyEvidence(base, 64, 17))
+	if len(got) != 1 || got[0].ID != "ev-17" {
+		t.Fatalf("newest by ObservedAt = %+v, want ev-17", got)
+	}
+	tied := []operationaltrust.EvidenceEnvelope{{ID: "first", ObservedAt: base}, {ID: "second", ObservedAt: base}}
+	if got := newestHistoryEvidence(tied); len(got) != 1 || got[0].ID != "second" {
+		t.Fatalf("a tie must keep the later envelope, got %+v", got)
+	}
+}
+
+// A history entry carries only its newest evidence envelope, in the shared
+// fold as well as the response; the event log and the record keep everything.
+func TestHistoryEntriesKeepOnlyTheNewestEvidence(t *testing.T) {
+	m := newHistoryParityManager(t)
+	store := m.eventLogStore()
+	start := time.Date(2026, 9, 1, 0, 0, 0, 0, time.UTC)
+	ids := make([]string, 0, 64)
+	for i := 0; i < 64; i++ {
+		ids = append(ids, fmt.Sprintf("ev-%02d", i))
+	}
+	write := func(kind string, occurred time.Time, evidence []operationaltrust.EvidenceEnvelope) {
+		t.Helper()
+		snapshot, err := json.Marshal(Alert{ID: "disk-1", ResourceID: "vm-1", StartTime: start, LastSeen: occurred,
+			Evidence: evidence, OperationalRecord: &operationaltrust.OperationalRecord{ID: "record-1", EvidenceIDs: ids}})
+		if err != nil {
+			t.Fatal(err)
+		}
+		if err := store.AppendDurable(eventlog.Event{Type: kind, AlertID: "disk-1", OccurredAt: occurred, Snapshot: snapshot}); err != nil {
+			t.Fatal(err)
+		}
+	}
+	write(eventlog.TypeFired, start, historyEvidence(start, 5, 2))
+	write(eventlog.TypeResolved, start.Add(time.Minute), historyEvidence(start, 64, 40))
+
+	for _, since := range []time.Time{{}, start.Add(30 * time.Second)} {
+		history, ok := m.AlertHistoryFromEvents(since, 0)
+		if !ok || len(history) != 1 {
+			t.Fatalf("since=%s: history = %d entries, ok=%v", since, len(history), ok)
+		}
+		if got := history[0].Evidence; len(got) != 1 || got[0].ID != "ev-40" {
+			t.Fatalf("since=%s: evidence = %+v, want only the newest envelope ev-40", since, got)
+		}
+		if got := history[0].OperationalRecord; got == nil || len(got.EvidenceIDs) != 64 {
+			t.Fatalf("since=%s: the record must keep its full evidence ID list, got %+v", since, got)
+		}
+	}
+	m.historyProjectionMu.Lock()
+	for _, occ := range m.historyProjection.occurrences {
+		if len(occ.alert.Evidence) != 1 {
+			t.Errorf("the shared fold holds %d evidence envelopes; it must hold only the newest", len(occ.alert.Evidence))
+		}
+	}
+	m.historyProjectionMu.Unlock()
+
+	var logged []int
+	if err := store.WalkOldest(eventlog.Filter{AlertID: "disk-1"}, func(event eventlog.Event) error {
+		var snapshot Alert
+		if err := json.Unmarshal(event.Snapshot, &snapshot); err != nil {
+			return err
+		}
+		logged = append(logged, len(snapshot.Evidence))
+		return nil
+	}); err != nil {
+		t.Fatal(err)
+	}
+	if !reflect.DeepEqual(logged, []int{5, 64}) {
+		t.Fatalf("the event log must keep every snapshot whole, evidence lengths = %v", logged)
+	}
+}
+
+func TestHistoryOverlayKeepsOnlyTheNewestEvidence(t *testing.T) {
+	start := time.Date(2026, 9, 1, 0, 0, 0, 0, time.UTC)
+	m := newHistoryParityManager(t)
+	m.mu.Lock()
+	m.activeAlerts["live"] = &Alert{ID: "cpu-1", ResourceID: "vm-2", StartTime: start, LastSeen: start,
+		Evidence: historyEvidence(start, 12, 3)}
+	m.mu.Unlock()
+	history, ok := m.AlertHistoryFromEvents(time.Time{}, 0)
+	if !ok || len(history) != 1 {
+		t.Fatalf("overlay history = %d entries, ok=%v", len(history), ok)
+	}
+	if got := history[0].Evidence; len(got) != 1 || got[0].ID != "ev-03" {
+		t.Fatalf("a live overlay entry carries %+v, want only ev-03", got)
+	}
+	m.mu.RLock()
+	live := len(m.activeAlerts["live"].Evidence)
+	m.mu.RUnlock()
+	if live != 12 {
+		t.Fatalf("trimming history output changed the live alert's evidence to %d envelopes", live)
+	}
+}
+
+// A separate test: two newTestManager calls on one t deadlock.
+func TestHistoryFallbackKeepsOnlyTheNewestEvidence(t *testing.T) {
+	start := time.Date(2026, 9, 1, 0, 0, 0, 0, time.UTC)
+	fallback := newTestManager(t)
+	fallback.SetEventLog(nil) // read the JSON-backed history, not a projection
+	fallback.historyManager.AddAlert(Alert{ID: "mem-1", ResourceID: "vm-3", StartTime: start, LastSeen: start,
+		Evidence: historyEvidence(start, 9, 6)})
+	entries := fallback.GetAlertHistory(0)
+	if len(entries) != 1 {
+		t.Fatalf("fallback history = %d entries", len(entries))
+	}
+	if got := entries[0].Evidence; len(got) != 1 || got[0].ID != "ev-06" {
+		t.Fatalf("a JSON-history entry carries %+v, want only ev-06", got)
+	}
+}

internal/alerts/operational_contract_test.go

@@ -980,3 +980,39 @@ func TestCapacityForecastAlertNamesItsPlatform(t *testing.T) {
 		t.Fatalf("platformType metadata = %v, want vmware-vsphere", got)
 	}
 }
+
+// History output keeps an alert's operational record and transitions whole but
+// only its newest evidence envelope, and never changes the alert it copies
+// (#1803).
+func TestHistoryOutputKeepsTheRecordAndOnlyTheNewestEvidence(t *testing.T) {
+	observed := time.Date(2026, 9, 1, 12, 0, 0, 0, time.UTC)
+	alert := Alert{
+		ID:         "disk-1",
+		ResourceID: "vm-1",
+		Level:      AlertLevelWarning,
+		StartTime:  observed,
+		LastSeen:   observed,
+		Evidence: []operationaltrust.EvidenceEnvelope{
+			{ID: "ev-new", ObservedAt: observed.Add(time.Hour)},
+			{ID: "ev-old", ObservedAt: observed},
+		},
+		OperationalRecord: &operationaltrust.OperationalRecord{ID: "record-1", EvidenceIDs: []string{"ev-old", "ev-new"}},
+		Transitions:       []operationaltrust.LifecycleTransition{{ID: "transition-1", EvidenceIDs: []string{"ev-old"}}},
+	}
+	output := canonicalizeAlertHistoryForOutput([]Alert{alert})
+	if len(output) != 1 {
+		t.Fatalf("output = %d entries", len(output))
+	}
+	if got := output[0].Evidence; len(got) != 1 || got[0].ID != "ev-new" {
+		t.Fatalf("history evidence = %+v, want only ev-new", got)
+	}
+	if got := output[0].OperationalRecord; got == nil || len(got.EvidenceIDs) != 2 {
+		t.Fatalf("the operational record must keep both evidence IDs, got %+v", got)
+	}
+	if got := output[0].Transitions; len(got) != 1 || len(got[0].EvidenceIDs) != 1 {
+		t.Fatalf("transitions must be kept whole, got %+v", got)
+	}
+	if len(alert.Evidence) != 2 {
+		t.Fatalf("shaping history output changed its input alert to %d envelopes", len(alert.Evidence))
+	}
+}