gsadmin/pulse
1
0
Fork 0
mirror of https://github.com/rcourtman/Pulse.git synced 2026-09-09 18:15:50 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge pull request #1973 from rcourtman/fix/canonical-incident-history

Browse Source 
Preserve canonical incident history and Assistant handoffs
This commit is contained in:
rcourtman
2026-09-08 03:11:45 +01:00
committed by GitHub
parent 9b57f7cacd e2b6fe3b16
commit 783571bb35
61 changed files with 3026 additions and 586 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/qualification/PATROL_ASSISTANT_CUSTOMER_JOURNEY.md
+623 -2
View File
@@ -4,9 +4,12 @@ The customer job is: "Tell me what needs my attention, explain why, and help me
deal with it without creating more work." Patrol owns the issue and investigation.
Assistant explains that same issue and uses existing governed action contracts.
The local redesign and named qualification matrix are delivered. See
The earlier r34 local redesign and named qualification matrix landed. See
[verified delivery and remaining gate](#verified-local-delivery-and-remaining-release-gate)
for the exact source and CI results. Production-wide readiness remains open.
for that source and CI evidence. The subsequent
[shared incident-history continuation](#continued-shared-incident-history-modernization-2026-09-07)
is unlanded and its required qualification remains incomplete. Production-wide
readiness also remains open.
## Active redesign plan, 2026-09-05
@@ -3615,3 +3618,621 @@ operator access and consent. Do not manufacture it from adoption telemetry or
repeat maintainer-lab runs as a substitute. No production rollout or release was
performed. The explicit subscription-provider refusal remains preserved without
retry or bypass.
## Continued shared incident-history modernization, 2026-09-07
The maintainer requested continuation after the named local redesign matrix
landed. The independent-environment question remains open. This slice addresses
the recorded incident-memory query residual and does not satisfy the wider
readiness gate by extending the same homelab evidence.
The reviewed continuation baseline selected legacy memory shells before consulting
canonical history. This omits canonical-only alert occurrences and aliases whose
shell identifiers differ from the query. Projection readers also discard store
errors, scan an infrastructure-wide 256-event window before selecting an alert,
and can combine events from repeated occurrences of the same alert. The owning
fix belongs in the shared incident query/projection and canonical history query,
not in a single HTTP handler or model-written summary.
Executable order and acceptance:
1. Reproduce canonical-only, alias, projection-error, bounded-history and repeated
occurrence cases with actual memory/SQLite history stores. Record provenance
and observation/occurrence time expectations before changing projection.
2. Read canonical history first through its shared query contract, apply query
filters before limits, retain explicit bounds and unavailable evidence, and
preserve legacy notes as attributed history. Keep separate alert occurrences
separate. Do not create a diagnosis, recovery or current-health verdict from
missing records or a completed query.
3. Route resource listings, alert timeline reads and model context through that
shared result. Propagate read failures to API callers and disclose unavailable
context to the model. Preserve canonical event provenance and existing exact
alert/resource authority. No new diagnosis orchestration or provider retry
policy is authorized by this change.
4. Run targeted regressions and affected complete race suites on the worker.
Inspect the affected live Alerts timeline and Assistant journey with
Playwright after the final build at 1440x1000, 900x1000 and 390x1000. Cover
loading, empty, failed reads, repeated occurrences, expanded event details,
filters, dismissal/focus return, reload and deep scrolling. Inspect pixels as
well as DOM state. If model-visible context changes, qualify the affected
real-model history explanation through an authorized provider route and
preserve any unavailable-provider limit explicitly.
5. Update the owning contracts and this record with exact source, proofs and
remaining limits, then land the verified scope through the repository
workflow. Local proof and independent customer qualification stay distinct.
Current state: the incident-history continuation has final r10 source acceptance
on integrated main `977afdd9559c0e9d5859f4c79bcc48e889381bba` plus the scoped
changes. Affected regressions and final browser interaction/pixel checks pass.
Three funded Astra explanations passed, with both earlier Gemini failures
retained. The actual note survives reload and its text reaches Assistant.
Repository commit checks and scoped landing remain pending. Both the old
candidate and matching unchanged base timed out in the full API race suite,
so no passing whole-suite receipt is claimed. The subscription-provider refusal
is preserved without retry or bypass, and wider readiness remains open.
### Incident-history candidate r1: implementation and proof in progress
The shared `IncidentStore.QueryIncidents` now reads canonical evidence before
selecting incident rows. Alert metadata filters and observation bounds are
applied before limits in SQLite and memory, with matching aggregate counts.
Both stores order by observation time and event ID. The earlier memory behavior
used reverse insertion order, which disagreed with SQLite for late observations.
The relationship-aware store and monitoring replay fixtures now assert the
canonical observation order while retaining every expected event.
Explicit firing times partition repeated occurrences. The alert identifier and
firing time determine canonical-only occurrence identity. Repeated observations
of the same firing do not rename it, and two firings 500 milliseconds apart stay
separate. Exact-start selection chooses the closest occurrence within the
existing one-second shell matching tolerance. A capped query does not attach
ambiguous later events to an earlier saved occurrence. Missing starts stay
unknown. Source event IDs, observed/occurred times, actor, adapter, confidence,
metadata and related resource identities remain on the returned evidence.
Canonical-only incidents can retain an operator note without copying canonical
lifecycle events into a second durable history. Saved notes and snapshots carry
their own source attribution. History aliases are read selectors only and do not
change action authority. API reads return service-unavailable on canonical read
failure, and monitoring skips reconciliation instead of fabricating a fallback
when that read fails. Assistant's incident summaries disclose unavailable or
truncated history and do not claim current health from historical lifecycle.
User job: “What happened last time, and is this the same problem?” The live r34
baseline was opened at `/alerts`, History, then a timeline at 1440 × 1000. This is
three navigation/control actions. The expanded timeline has twelve event filter
controls plus the Assistant handoff before its event content. The current slice
keeps the established timeline interaction and places new forensic provenance
behind one “Evidence details” disclosure. Missing timestamps are shown as
unavailable, and a truncated query receives an explicit notice. The inspected
issue #1782 supports current-evidence grounding and preservation of the governed
action path. It does not establish customer demand for extra timeline decoration.
No issue comment was sent.
Current proof receipts under the worker's `patrol-incident-history` directory:
- New filter regression first failed compilation because the filter fields did
not exist. It then passed against both real stores. The full store race suite
initially exposed the reverse-insertion expectation described above.
- Focused memory and incident HTTP race tests passed. The full affected suites
passed: memory 2.217 s, unified resources 72.877 s, alerting 3.691 s,
monitoring 212.854 s and runtime 24.119 s. An earlier monitoring run failed
its reverse-insertion expectation, before that expectation was corrected.
- Frontend type checking passed. Nine tests across the timeline panel and event
card passed, followed by the current frontend build. An initial dependency
symlink attempt failed Vite module resolution before running tests. A fresh
task-local `npm ci` resolved that environment failure.
- Cross-build verified 5,985 core and 60 enterprise source hashes with no
mismatches. Candidate `0.0.0-dev-pro+incident-history-r1` has SHA-256
`1cd63e63b767a7a02334a89ead3b8f34941b1371fdd764646ae6f689df7455c0`.
This is a candidate under qualification, not a completed continuation. The full
API race suite, final-build Playwright matrix, affected real-model explanation,
final review and repository landing are still pending at this checkpoint. The
previously authorized Gemini key had a US$5 limit and expiry on 2026-09-07.
No provider call has been made in this continuation, and the Claude subscription
refusal remains untouched. Independent customer-environment consent/evidence is
still unavailable, so the wider rollout gate remains open independently of this
local implementation work.
### Incident-history r1 live failures and r2 corrections
The r1 live timeline of disposable run `q-20260907-150920-a22e598e` failed
qualification. The alert-history row was resolved, but its timeline was open.
The canonical resource timeline contained both records: firing
`85ddd9f5-0368-5740-a041-eb68d2f131c6` at 15:10:01.025055Z and resolution
`f3c13163-33dd-5b10-9b37-ef8c8a1bc37b` at 15:10:30.96527Z. Two legacy shells
had the same alert identifier and explicit firing time. Projection assigned
firing to one and resolution to the other. The shared query now merges those
shells before projecting canonical events, retains a stable existing ID and
all local notes, and accepts either saved ID as a read selector. Regression
covers both aliases and the combined lifecycle. API snapshot fallback was not
used to mask this defect.
A separate live backup timeline displayed `warning 0.0 >= 0.0`. The shared event
summary had invented an inequality from numeric fields without retaining the
source condition. Source messages now own fired-event descriptions. Where no
message exists, the fallback names only the alert type and level. Regression
covers source conditions, including a below-threshold comparison, and neutral
fallbacks for CPU and backup incidents.
Read-failure inspection also exposed an error-visibility gap. A timeline refresh
could retain cached evidence while hiding its error, and resource history could
show an empty-state claim after its error toast disappeared. Both views now
retain visible read failure and a retry control. Resource-history state preserves
cached evidence separately from loading and failure, and clears failure only
when a new read succeeds. Tests exercise failed initial read, successful retry,
failed cached refresh and successful empty read.
The r1 Playwright script also had a harness error: Escape cleared its search,
so its single-row expectation saw 138 timeline buttons. Removing that unintended
search reset repaired the script. This is separate from the reproduced product
failures above. No final r2 browser or real-model pass is claimed here.
The saved non-secret provider-limit receipt confirms the authorized key expired
at `2026-09-07T17:23:20.631Z`. This was verified after that timestamp, without
calling the provider. Its initial US$5 balance is not a current spend balance.
The affected r2 real-model history explanation requires a valid funded route
and remains unperformed. The subscription-provider refusal is not an alternative
route and has not been retried.
Remaining affected model qualification is a read-only Assistant turn from the
repaired historical lab incident. Ask: “Explain what happened in this occurrence,
what evidence records its resolution, and what we can and cannot conclude about
its current health. Do not change anything.” Verify the response against the
independent canonical firing/resolution records above, its saved note and exact
occurrence ID. The answer must distinguish observation from occurrence time,
recorded resolution from present health, and historical evidence from an action
outcome. No action proposal, execution or provider substitution is needed. A
second turn with unavailable history must say that the evidence could not be
read, without converting failure into no incidents or a healthy result. Retain
session IDs, provider/model, exact runtime hash, latency and actual spend. A
scripted fixture response does not satisfy this real-model check.
The maintainer supplied a replacement OpenRouter key in this continuation and
authorized its use. The key metadata endpoint reports a valid paid key with
zero initial usage and no provider-side limit or expiry. The prior US$5
qualification ceiling is retained as a task budget, not claimed as a provider
enforcement boundary. Only the two read-only history turns above are planned.
Authenticated local settings confirmed Patrol disabled, control level read-only,
and `openrouter:google/gemini-3.8-flash` selected for chat and Patrol before the
credential update. The update succeeded through the ordinary settings endpoint.
An initial helper request omitted its CSRF header and was rejected before the
update. The corrected request supplied the normal session CSRF token. No
subscription request or model call was made during configuration.
Narrow exploration reproduced an additional handoff defect at 390 × 1000.
The incident drawer remained above Assistant after Discuss, hiding the
continuation. The shared incident handoff now exposes the same explicit callback
pattern used by finding handoffs. Mobile timeline and resource-history drawers
close after the incident context has been handed to Assistant. Desktop inline
panels remain governed by their existing controls. Mounted tests cover both
mobile source views and preserve the same incident ID, status and read-only
context. Final browser qualification must repeat the drawer transition and
inspect the reachable Assistant composer, dismissal and focus.
The first callback attempt still failed because the app's shared blocking-dialog
guard correctly closed Assistant while the source drawer was mounted. The final
transition captures the incident context, closes the source drawer, then opens
Assistant in the next microtask, matching the existing command-palette handoff.
The mounted regression checks that the dialog stack is no longer blocking when
Assistant opens. Narrow exploration then passed without a model request. This
exploration used the r1 backend and does not qualify its known split lifecycle.
### Candidate r2 qualification checkpoint
Final r2 source passed memory and unified-resource race suites in 2.225 s and
72.169 s, plus focused incident alerting/API/monitoring race checks in 1.203 s,
2.576 s and 1.770 s. Frontend type checking and 62 tests in eight files passed,
followed by the frontend build and verification of 5,987 core and 60 enterprise
source hashes. Binary SHA-256:
`7c82377ca4cc58f0f3ab3e3d64bd2edac7692309596416ce0f70e836d731726a`.
The local built runtime confirmed the previously split lab occurrence as
resolved with both canonical lifecycle records.
The broad r1 API race run failed. It reported
`TestServerInfoEndpointReportsDevelopment` expecting development mode, then hit
its one-hour deadline in `TestAuthenticatedEndpointsRequireToken`, which had
spent 39m6s in the running test. The timeout stack is in mock unified-resource
fixture expansion during per-resource metric-window evaluation and router
construction. This is not a passing broad API receipt. The worker source export
has no Git directory. A fresh exact-base Git checkout passed the same two tests
in isolation in 62.298 s after its required frontend embed build. The initial
baseline attempt lacked generated embed assets and failed setup. Candidate
comparison in that Git checkout remains pending at this checkpoint. Neither
isolated result can establish that the full suite is free of shared-state or
performance problems.
R2 browser qualification also found that Assistant's composer registration and
focus happened only at mount. Reopening from a mobile drawer left focus on the
underlying alert search, so Escape cleared the search rather than dismissing
Assistant. Candidate r3 moves registration and focus into Assistant's shared
open lifecycle and clears the registration on close. A mounted regression
starts closed and checks two separate composer instances across reopening.
Final build, complete browser repetition and both funded model explanations
remain pending. No full-task completion is claimed.
### Candidate r3 browser failure and r4 correction
The isolated candidate API comparison passed the same two tests in 63.937 s.
The complete Assistant component file passed 211 tests. These receipts do not
replace the failed broad API race run above.
R3 was built and deployed locally with SHA-256
`f4ec960416cff6fa5c779119689d04f2842e38aee6494e23a4f1649d4f7e1d01`.
The repeated live matrix still failed narrow Escape dismissal. Browser inspection
identified a second focus owner in the mobile alert list. Its normal drawer-close
callback restores focus twice through animation frames, overriding the shared
Assistant open lifecycle. R4 gives the source drawer an explicit handoff callback
that closes the investigation without scheduling return focus. Ordinary drawer
dismissal retains its existing focus restoration. The regression now mounts the
parent mobile list and waits through both animation frames before checking the
destination focus. Its first run exposed an incomplete test fixture, which omitted
the required alert type. That fixture was corrected before repeating qualification.
Opening Assistant also sends ordinary provider-readiness requests. The browser
receipt's chat/session submission counter excludes those probes and must not be
interpreted as a count of all provider traffic. No history explanation has been
submitted at this checkpoint. The subscription refusal remains unchanged.
R4 passed 218 frontend tests across the full Assistant file and both mobile
investigation files, type checking and the frontend build. Runtime SHA-256:
`bad20d2dee980abf826809755689538ae8c1a5ea113656619e7d5d92741af366`.
The complete live `/alerts` history matrix passed at 1440, 900 and 390 × 1000,
including both Assistant handoffs, Escape, evidence expansion, cached read
failure/retry and reload. Controlled-state scripts initially selected notification
toasts or hidden desktop copies of mobile elements. Those locator failures were
corrected without changing product source.
The first funded history explanation was a failed qualification, despite correctly
distinguishing alert closure from current health and verified remediation.
Session `efdc8896-747c-4516-8b43-d1e697495c00`, Gemini 3.8 Flash through OpenRouter,
HTTP 200, 15.806 s, 28,070 input and 1,622 output tokens, recorded session cost
US$0.027135. The model read inventory and canonical history, then asserted no governed action record existed without reading actions. The
initial network-removal criticism did not account for automatically attached
related-resource history and is corrected in the r6 review below. R5 corrects the shared history tool's existing
read-scope description: resolution records closure rather than workload recovery,
related-resource IDs identify relationships rather than additional event targets,
and this read does not query action records. These are factual source boundaries,
not deterministic diagnosis or response-scoring rules. The failed transcript is
retained locally. A fresh-session rerun is required before the unavailable-history
turn. The additional turn remains within the existing US$5 task ceiling.
Provider-readiness inspection confirms that this OpenRouter route performs an
authenticated key-metadata GET, not a model completion
(`internal/ai/providers/openai.go`, `TestConnection` / `testOpenRouterKey`).
The first funded session increased the local one-day estimated usage total by
US$0.02794875, including US$0.00081375 for its automatic session title. These are
Pulse pricing estimates. Actual provider billing and remaining balance have not
been independently reconciled after that turn.
### R5 outcome and R6 qualification plan
R5 tool regression passed in 1.640 s and the final source hashes matched.
Runtime SHA-256:
`f43652b8c939238b5dcb7848e68c99d72def8ade316234010d6767784e99447f`.
The live three-viewport history matrix and controlled desktop/mobile loading,
initial failure, retry, empty, partial and unknown-time cases passed again.
The controlled cases substitute HTTP responses, not a real database outage.
The second funded Gemini history explanation also failed factual qualification.
Session `c2f40f27-6d7c-40f6-8e3a-549ac5232071`, HTTP 200, 18.406 s,
40,235 input and 1,630 output tokens, estimated session cost US$0.03628875.
Including its automatic title, the local estimated usage increase was
US$0.037116. It stopped claiming the related network was removed, but still
asserted no action plan or verified execution existed without reading action
records. Both failed attempts remain in the denominator. Their total estimated
increment including titles is US$0.06506475. Neither is a passing history
explanation or evidence of population-wide effectiveness.
The request inspection exposed a separate shared projection defect: saved shells
retained a legacy resource ID and stale risk fields despite canonical alert
evidence. R6 gives explicit canonical alert event kinds ownership of resource
identity, type, severity and message, while retaining the saved incident ID and
notes. A related command's execution host cannot retarget the incident. The new
single regression passed locally in 0.495 s after correcting its fixture's
collection type. Full memory regression and the rebuilt runtime remain required.
The full candidate API race run in the disposable Git checkout failed at its
30-minute deadline, with `TestAuthenticatedEndpointsRequireToken` running 6m40s.
The package elapsed time was 1802.152 s. Its stack traverses mock fixture cloning,
unified snapshot construction, per-resource metric-window evaluation and router
construction. Those stack functions are not modified by this incident query
slice. This is an unresolved broad performance/qualification limit, not a pass
and not proof that the entire failure is unrelated to all candidate effects.
No further broad rerun was queued ahead of the release rehearsal.
The next two history checks use the exact funded route
`openrouter:openai/gpt-6-astra`, in fresh sessions with Patrol disabled and control
read-only. The provider's public Models API lists function calling and the route's
prices. [Official model documentation](https://developers.openai.com/api/docs/models/gpt-6-astra)
identifies it as a model for complex reasoning and end-to-end work. The route was
selected as a stronger-model qualification, not to erase Gemini's failures or
claim that its earlier seven-case lab matrix transfers to this model.
The exact OpenRouter price row was reviewed on 2026-09-07: US$10 input / US$50
output per million tokens below its `min_prompt_tokens=272000` override, and
US$20 / US$75 at that override. Cache discounts are omitted for conservative
budget estimation. Unreviewed variants remain unknown. A targeted boundary and
alias regression passed locally in 0.364 s. These are estimated list-price costs,
not actual billing. The aggregate qualification ceiling remains US$5. The
subscription-provider refusal is unchanged. No GPT-6 Astra explanation has been
submitted at this checkpoint.
The local 30-day estimated budget was temporarily set to US$5 before any Astra
explanation, with a pre-existing estimated total of US$1.45530525. This is stricter
than a separate US$5 allowance for this continuation. The existing budget check
runs between model turns and cannot cap an already in-flight provider charge.
The prior budget was zero and the prior chat route was Gemini 3.8 Flash. Restore
those temporary qualification settings after the two checks, retaining the
user-authorized replacement credential in encrypted runtime configuration.
A matching full API race baseline comparison is queued after the candidate r6
proof/build, behind the release rehearsal. It uses unchanged base
`1a822d164b3119724b01615da217618edf7e37e5`, a disposable Git checkout, the same
Go 1.26.8 toolchain, non-root execution, GOMAXPROCS=4 and 30-minute timeout.
This is a pending diagnostic comparison, not a passing receipt or a relaxation
of the candidate failure.
### R6 final source qualification in progress
The release rehearsal released the shared worker at 23:28:46 UTC. The normally
queued r6 job completed with exit 0. The full memory race suite passed in
2.189 s and the full cost race suite in 1.148 s. Focused alerting and monitoring
race checks passed in 1.222 s and 1.496 s. The first filtered memory invocation
reported no tests, so it is not counted as proof. The subsequent full memory
invocation is the passing receipt. All 5,987 core and 60 enterprise source
hashes matched. The built r6 runtime SHA-256 is
`3c875fcfe1826c4ef50fe6f116d49269c1111dc569b09a0e80175d13f9421f92`.
It was hash-checked and installed on the local development stack. Final browser,
funded-model and saved-note qualification remains pending at this checkpoint.
The r6 live matrix passed at 1440, 900 and 390 × 1000 with the canonical resource
ID and type in the incident and Assistant handoff. Controlled desktop/mobile
loading, initial error, retry to empty, partial-history and unknown-time cases
also passed. Pixels confirmed visible evidence and errors, reachable handoffs
and the mobile composer. This proof preceded the resize defect described below.
Both funded Astra turns passed factual review. History session
`1b881e3f-57b1-4fcd-be64-ee8e46885394` completed HTTP 200 in 18.507 s,
with reported 8,054 input and 456 output tokens, estimated session cost
US$0.10334 and total usage increase US$0.11216 including its title. It explained
the recorded closure, kept present health and remediation unknown, and said an
action record was not supplied rather than asserting none existed. Its network
removal statement is supported by separate canonical record
`8539f198-82b0-4720-81c1-abdf871e60b1`, targeted at
`docker-network-7f19c579093f3cd0`. The shared server handoff automatically reads
related resource history with a five-record limit. This record was within that
input. Tool calls alone are not the complete model context. The earlier r4
network criticism is therefore withdrawn as an unsupported qualification
finding. Both Gemini attempts still fail on their independent unsupported
claims about action-record absence.
Unavailable-history session `8365f1d5-e20b-4317-b297-3200f37ad67c` completed
HTTP 200 in 8.957 s, reported 6,843 input and 100 output tokens, estimated
session cost US$0.07343 and total increase US$0.07883 including its title. It
stated only that canonical evidence could not be read and left incidents,
health, actions, cause and transience unknown. Neither Astra turn called tools
or changed infrastructure. The unavailable case uses the actual formatter's
error text in controlled handoff context, not a live database outage. Total
estimated spend across all four turns and titles is US$0.25605475. Actual
provider billing remains unreconciled. Two failed explanations remain in the
four-attempt denominator. This is not a population success-rate estimate.
Some earlier saved SSE text had mojibake because the browser receipt helper used
response.text decoding. Actual pixels rendered punctuation correctly. Subsequent
capture explicitly decodes response.body as UTF-8. No output-rewriting product
patch was added. After both turns, the temporary chat model was restored to
Gemini 3.8 Flash and budget to zero, with Patrol still disabled and read-only.
Zero budget is omitted by the settings JSON, which initially confused the
restoration assertion. A subsequent read verified the restored settings.
### R7 responsive investigation correction in progress
One actual note was saved to the owned historical lab occurrence and survived
read-error retry and desktop reload. Shrinking that open desktop timeline to
390 pixels exposed a product defect: its card said Hide, but the timeline was
hidden and no mobile drawer existed. Two read-only attempts confirmed it without
writing another note. The local mobile drawer selection had diverged from the
shared expansion state. The mobile layout now projects existing shared resource
or timeline selection when entering the phone layout and leaves that state
intact when returning to desktop. Its action label reflects the actual drawer,
including when other desktop rows remain expanded. Both timeline and resource
resize regressions pass in the seven-test mobile-list suite. R7 build and full
affected browser repetition remain required. The responsive correction alone left the model input unchanged. A subsequent
review found that the handoff discarded operator note text, retaining only its
"Note added" summary. R7 now preserves the attributed note in shared incident
model formatting and the Assistant handoff, while excluding unrelated details
and raw command output. The new memory regression passed in 0.437 s and the
combined handoff/mobile-list checks passed 11 tests in 1.58 s. One additional
funded history-and-note explanation is required after the r7 build, within the
existing US$5 ceiling. The two r6 outcomes remain recorded without replacing
that final changed-context check.
### Integration with updated main, 2026-09-07
Main advanced from `1a822d164b3119724b01615da217618edf7e37e5` to
`203b50a46ece5e911d6f5f6ea46bbeef218aeab5` during qualification. The candidate
was saved to a task-local archive and explicit-path Git stash, main was
fast-forwarded, and the work was reapplied. Upstream lifecycle replay identity,
unchanged-checkpoint suppression and their regressions are retained. The
superseded legacy projection helper remains removed in favour of QueryIncidents.
Owning contract additions from both changes are preserved.
The upstream cross-resource occurrence regression failed on the initial merge
in both the two-minute and 500-millisecond cases. An unrelated resource's shell
was shortening the selected alert occurrence. The shared query now resolves
resource history identities, separates saved-shell grouping by resource, adopts
the exact canonical firing target before assigning subsequent evidence, and
excludes unrelated-resource boundaries for alert events. The five focused
integrated regression groups passed in 0.491 s. Full integrated source proof and
r7 qualification remain required. The old-base API comparison only diagnoses the
earlier full-suite timeout and cannot qualify the integrated candidate.
The matching clean old-base full API race comparison also failed its 30-minute
bound. The checkout was exactly `1a822d164b3119724b01615da217618edf7e37e5`
and stayed clean. It ran as the normal worker user through the allocator with
Go 1.26.8 and GOMAXPROCS=4. The package failed after 1802.195 s, with
`TestRecoveryEndpointRequiresDirectLoopback` running for 7 m 46 s at timeout.
The earlier candidate failed after 1802.152 s with a different active test,
`TestAuthenticatedEndpointsRequireToken`. The worker log is
`/opt/pulse-release-worker/patrol-incident-history/base-api-proof.log`.
This establishes that the unchanged old base also cannot finish that broad
race command within the same bound. It does not attribute every candidate
effect to the base, and neither run is a passing whole-API-suite receipt.
The integrated candidate still requires its scoped proof and repository checks.
### R7 integrated source qualification, 2026-09-08
The normal worker r7 command completed with exit 0. All 5,990 core and 60
enterprise source hashes matched. Complete affected race suites passed:
incident memory 2.252 s, unified resources 72.880 s, alerting 3.598 s and
monitoring 206.311 s. Frontend type checking and all 51 tests in the four
handoff/mobile investigation files passed. The frontend build completed in
27.50 s. The resulting development runtime is
`0.0.0-dev-pro+incident-history-r7`, with core base
`203b50a46ece5e911d6f5f6ea46bbeef218aeab5` plus the scoped changes and enterprise
`b9fa43dcf0ee743652b20d1a866da8ca9c82cdbd`. Its SHA-256 is
`341b9d2e3c685d0280a321453546a5a02321d8ba9ce923f99a51a961da9c8340`.
The same binary and frontend assets were installed on the local development
stack. Final browser, changed-context model and repository checks remain
required at this checkpoint.
The remaining r7 scoped race proof also passed: complete shared AI 23.613 s,
tools 62.645 s and cost 1.148 s, then the selected incident API handlers in
2.015 s. It ran from the exact integrated Git checkout under the normal worker
allocator. These passes do not replace the failed broad API runs.
R7 live history and controlled-state Playwright scripts passed at their planned
widths, including composer focus after both handoffs. Pixel inspection then
found the saved operator note clipped on the 900-pixel desktop timeline because
event text inherited the table's no-wrap style. R8 corrects the shared event
card's wrapping and preserves note line breaks. This is a real browser failure,
not a passing visual receipt. The affected matrix must repeat on r8. The initial
resize command also failed in its invocation wrapper before the script ran,
while node's syntax check of the actual script passed. That invocation provides
no product evidence. No additional operator note was written.
The direct resize invocation then reached the real r7 UI and failed the new
saved-note overflow assertion, independently reproducing the clipped text.
R8 type checking, 51 frontend tests and the 26.50-second build passed. Its
runtime hash is `90d912e8cea77ffd108be0046430cec1e90d4403900861fcd8c6aa1ae8265152`.
Live and controlled-state scripts passed. The resize matrix retained the note,
draft and expanded events across 390/900/767/768-pixel transitions, but closing
the resource drawer with Escape cleared the background search and lost focus.
The first assertion reported 22 Resource buttons. A second read-only capture
confirmed the search had become empty and focus moved to body, so this is a
product defect rather than merely a selector ambiguity. The shared type-to-search
registry treated an inert modal background as eligible for keyboard shortcuts.
Two new regressions failed before the fix. R9 excludes inert inputs from both
visible and prepared search targets using the shared dialog's existing DOM
ownership boundary. No new dialog lifecycle or lane-local keyboard handler is
added. The final browser and funded model check remain pending.
The corrected shared search, Dialog and SearchInput suites passed 36 tests in
1.54 s locally. The normal worker repeated those 36 tests, type checking and
the frontend build (28.14 s), then built r9 with exit 0 and matching 5,990/60
source hashes. The installed runtime SHA-256 is
`7bad38f8da70a89b56b6693f41af5fa3176c4375bf10b5b1f4958aaa8fc929ba`.
The shared keyboard hook now has explicit frontend-primitives registry ownership.
Three existing presentation expectation files were updated after this build
and passed 118 tests in 1.36 s. Those later edits are test-only and do not change
the built runtime source. The integrated r7 backend proofs remain applicable
because r8 and r9 changed only frontend presentation and keyboard ownership.
All three r9 browser scripts passed, including note wrapping, retained draft,
saved-note reload, modal typing, retained search, Escape and focus return at
1440/900/390 and the 767/768 breakpoint. The subsequent funded history-and-note
turn passed factual review. Session `81c92125-cdc3-4611-931a-687e6548bf65`
completed HTTP 200 in 18.668 s with reported 8,130 input and 456 output tokens.
Its estimated session cost was US$0.1041, plus US$0.0088 for the title, for
US$0.1129 total. Five continuation turns and titles now total US$0.36895475
estimated, with three passing explanations and both Gemini failures retained.
Actual provider billing is not reconciled, and this is not a population rate.
The answer attributed the historical closure, retained current-health and
action-outcome uncertainty, and correctly described the operator note as a
display/persistence check with no infrastructure change or verified recovery.
It called no tools and performed no infrastructure action. Temporary model and
budget settings were restored to Gemini 3.8 Flash and zero, with Patrol disabled
and read-only. The subscription-provider refusal remains untouched.
The same model run exposed a further responsive defect: shrinking a desktop
conversation reopened its source history drawer above Assistant. The earlier
matrix had resized the source before handoff, not the open destination. Two
regressions reproduced this for timeline and resource history. R10 keeps the
existing Assistant destination active when projecting history state on a layout
change. It adds no model-visible context and needs no additional funded turn.
The affected browser matrix, including resizing the open conversation and
reading the saved response, must pass before source-bound acceptance and landing.
### Final incident-history source acceptance, r10
R10 passed 43 affected frontend tests and the 27.07-second worker build with
matching 5,990 core and 60 enterprise source hashes. Its installed runtime hash
is `2e1958d0856c812d22870144cf228509f2a4a3696258a533d34802442f500edb`.
All four final functional browser scripts passed, followed by actual pixel
inspection. `/alerts` was exercised at 1440, 900 and 390 × 1000, with source
and destination resize transitions also covering 767 and 768 pixels. The saved
funded response was resumed by exact session ID and read at every width without
another model submission. Assistant remains visible above the retained source,
the composer is usable, and the operator-note explanation is readable.
The final interaction matrix includes canonical occurrence selection, native
Evidence details with keyboard focus and activation, All/None filters, note
draft enable/clear, actual saved-note reload, source/destination resizing,
expanded resource events, cached and initial read errors, Retry/Refresh, empty
and partial history, unknown timestamps, modal typing, Escape and return focus.
Long notes wrap at the intermediate desktop width. Error toasts are not the only
failure indication. Controlled HTTP variants qualify rendering, not a real
database outage. Final acceptance is bound to exact frontend bytes in
`frontend-modern/browser-verification.json`. Earlier visual and factual failures
remain recorded above. The final helpers are executable at
`/Volumes/Development/pulse/tmp/patrol-incident-history/verify-live.mjs`,
`verify-states.mjs`, `verify-resize.mjs` and `verify-saved-explanation.mjs`.
Their receipt directories are `browser`, `browser-states`, `browser-resize` and
`browser-saved-explanation` under that task directory.
Repository commit checks and scoped landing are the remaining local delivery
steps. The broader customer-outcome gap stays open. These receipts do not
qualify independent customer environments, unattended autonomy, backup restore,
population false-alarm/miss rates or latency SLOs. The failed broad API race
runs remain an explicit test limit, despite passing affected package and handler
proof. The final history explanation qualifies the funded Astra route and does
not erase the two Gemini failures or transfer the earlier named lab matrix to
another model.
Commit preflight initially rejected missing dependent agent-lifecycle and
storage-recovery contract updates and missing explicit proof mappings. Those
contracts now state the historical-evidence boundary, and the registry maps the
actual incident query and lifecycle regression files to their owning paths.
Additional adapter assertions verify filtered-query forwarding, alias identities
and unavailable reads. The targeted query tests passed in 0.573 s. These are
test/governance changes only, leaving the r10 runtime and browser hashes intact.
The staged canonical completion and registry guards then passed without a
contract-neutral bypass.
The final upstream integration advances the base to
`977afdd9559c0e9d5859f4c79bcc48e889381bba`. It adds checkpoint and guest-memory
tests plus contract text, with no runtime change. Comparison against the 5,990
source manifest found only three test files and the browser receipt different.
All 14 accepted frontend content hashes remain identical. The newly integrated
checkpoint assertion initially compared fresh query timestamps as durable
incident state and failed. It now compares stable projection identity/events
separately from refreshed query bounds, while preserving coverage semantics and
the no-checkpoint-replacement assertion. The targeted occurrence test passed
in 0.450 s after this test-only integration.
The first full worker hook stopped at missing private repository evidence roots.
The existing filesystem-proof copies of pulse-pro and pulse-mobile and the
current task enterprise source export supply those roots through the supported
repository-root overrides. Status audit then reported no errors or warnings.
This repairs the proof environment and does not suppress the audit.
Registry audit also requires sibling Git checkouts. Initial sibling links made
that audit pass, but the cross-repository absolute-path helper correctly rejected
paths resolved outside its workspace. Real disposable clones of the existing
private filesystem-proof checkouts replace those links, with no audit bypass.
Its audit and contract audit pass. The full hook then caught one expected-file
fixture missing the newly registered monitoring regression. Updating that fixture
to include the actual proof file preserves the guard's exact mapping assertion.
All 130 completion-helper tests passed in 2.645 s. This is test-only scope.
docs/release-control/v6/internal/status.json
+9 -9
View File
@@ -10201,7 +10201,7 @@
},
{
"id": "patrol-assistant-customer-outcome-qualification",
"summary": "The wider readiness gate remains open. The executable contract, honest baseline, exact runs and residuals are in docs/qualification/PATROL_ASSISTANT_CUSTOMER_JOURNEY.md. Earlier evidence/history/risk and typed-runner changes landed through core PR1928/1929/1934/1935/1951/1955 and enterprise PR22. PR1957 auto-merged as a66b8e11d7ca9ed5660ffd8725a1461661ca2fdf despite its recorded benchmark failure, which is not counted as a pass. The current slice makes canonical planning acceptance/refusal available inside the model turn, retains accepted actions across provider failure, enforces persisted actor/request identity, preserves complete approval/risk/outcome context and removes proposal/tool-count diagnosis proxies. Shared disk unknowns, streamed whitespace and replayed resolution timestamps are corrected. Affected package race suites and final frontend checks pass. Source-bound r28 Gemini qualification performed healthy, unhealthy, dependency, missing-access, storage-capacity, approved and rejected cases, including independently observed Docker recovery and VM110 start/stop. Earlier semantic failures remain recorded. Current r34 runtime passes final linked-history and Assistant Playwright at 1440, 900 and 390 widths, including fresh approved explanation, saved missing-access/VM/rejected/storage continuation, deep evidence, exact action states and scrolling. The local named implementation matrix is performed, with source-equivalence and exact limits recorded. Verified local delivery landed through core PR1960 (a42e3800d9a1b5ea185823469e891f44bc698c2c) and enterprise PR23 (b9fa43dcf0ee743652b20d1a866da8ca9c82cdbd). Their final CI and source-bound local proofs pass. Production-wide readiness remains a separate release_gate. Temporary collectors, runners, tokens and fixtures were cleaned and production processes preserved. Storage capacity does not qualify backup/restore. Legacy incident-memory/compatibility residuals, all-filter coverage, unattended autonomy and latency SLOs remain unqualified. Adoption of 127 paid installations, 71 Patrol-enabled installations and 23 Assistant users does not measure effectiveness. Fourteen verified resolutions from one installation do not establish population useful-diagnosis, false-alarm, missed-problem or latency rates, which remain unknown. Independent volunteered customer environments are the separate wider rollout gate. Preserve the explicit subscription-provider refusal without retry or bypass. Do not close this gap or candidate while wider readiness evidence is missing.",
"summary": "The wider readiness gate remains open. The executable contract, honest baseline, exact runs and residuals are in docs/qualification/PATROL_ASSISTANT_CUSTOMER_JOURNEY.md. Earlier evidence/history/risk and typed-runner changes landed through core PR1928/1929/1934/1935/1951/1955 and enterprise PR22. PR1957 auto-merged as a66b8e11d7ca9ed5660ffd8725a1461661ca2fdf despite its recorded benchmark failure, which is not counted as a pass. The current slice makes canonical planning acceptance/refusal available inside the model turn, retains accepted actions across provider failure, enforces persisted actor/request identity, preserves complete approval/risk/outcome context and removes proposal/tool-count diagnosis proxies. Shared disk unknowns, streamed whitespace and replayed resolution timestamps are corrected. Affected package race suites and final frontend checks pass. Source-bound r28 Gemini qualification performed healthy, unhealthy, dependency, missing-access, storage-capacity, approved and rejected cases, including independently observed Docker recovery and VM110 start/stop. Earlier semantic failures remain recorded. Current r34 runtime passes final linked-history and Assistant Playwright at 1440, 900 and 390 widths, including fresh approved explanation, saved missing-access/VM/rejected/storage continuation, deep evidence, exact action states and scrolling. The local named implementation matrix is performed, with source-equivalence and exact limits recorded. Verified local delivery landed through core PR1960 (a42e3800d9a1b5ea185823469e891f44bc698c2c) and enterprise PR23 (b9fa43dcf0ee743652b20d1a866da8ca9c82cdbd). Their final CI and source-bound local proofs pass. Production-wide readiness remains a separate release_gate. Temporary collectors, runners, tokens and fixtures were cleaned and production processes preserved. Storage capacity does not qualify backup/restore. Legacy incident-memory/compatibility residuals, all-filter coverage, unattended autonomy and latency SLOs remain unqualified. Adoption of 127 paid installations, 71 Patrol-enabled installations and 23 Assistant users does not measure effectiveness. Fourteen verified resolutions from one installation do not establish population useful-diagnosis, false-alarm, missed-problem or latency rates, which remain unknown. Independent volunteered customer environments are the separate wider rollout gate. Preserve the explicit subscription-provider refusal without retry or bypass. Do not close this gap or candidate while wider readiness evidence is missing. The incident-history continuation implements canonical filtered occurrence queries, provenance, bounded history and explicit read errors, then repairs duplicate saved shells and mobile Assistant focus ownership found in live inspection. R4 browser checks pass, but its funded history explanation made unsupported action-record claims. Its initial related-resource criticism was withdrawn after auditing automatically attached history. R5 narrowed the shared history tool read scope, but its second Gemini explanation still asserted absent action records without reading actions. R6 repairs canonical alert target/risk ownership over legacy shells and adds reviewed cost estimates for funded GPT-6 Astra history qualification. Both funded Astra history and unavailable-read explanations passed factual review, with both Gemini failures retained and exact costs and context limits recorded. The actual note survived desktop reload, but resizing exposed a hidden mobile timeline. R7 projects shared expansion state into the phone drawer, and seven mobile-list tests pass. R7 also retains operator note text in the shared model context. Integrated main preserves upstream replay and checkpoint fixes, with a cross-resource occurrence-boundary regression corrected in the shared query. Final r10 build and source-bound browser acceptance pass, including saved notes, long-text wrapping, modal search ownership and keeping Assistant visible when resizing the destination. The additional funded history-and-note explanation passed. Five turns and titles cost US$0.36895475 estimated, with both earlier Gemini failures retained. Runtime defaults were restored and the subscription refusal preserved. Repository commit checks and scoped landing remain pending. Both the earlier candidate and its exact unchanged base timed out in the full API race suite at 30 minutes, with different active tests. Neither is a passing whole-suite receipt, and the comparison does not qualify the integrated candidate. These continuation requirements remain open.",
"owner": "project-owner",
"status": "planned",
"recorded_at": "2026-09-05",
@@ -10371,7 +10371,7 @@
{
"id": "patrol-assistant-customer-outcomes",
"name": "Patrol and Assistant Customer Outcomes",
"summary": "Simplify Patrol around model-owned investigation and one issue-to-verified-outcome journey shared with Assistant. Preserve canonical evidence and remove proposal-as-proof and proxy-driven diagnostic policy while retaining deterministic authority and independent verification. Execute the ordered redesign plan in docs/qualification/PATROL_ASSISTANT_CUSTOMER_JOURNEY.md, qualify unhealthy service diagnosis, backup or capacity risk and supported VM/LXC actions with negative controls, and record real model quality, latency and cost limits. Independent volunteered Pro qualification remains required for wider readiness. Local implementation and the named qualification matrix are delivered through core PR1960 and enterprise PR23. The candidate remains open for the independent-environment release gate.",
"summary": "Simplify Patrol around model-owned investigation and one issue-to-verified-outcome journey shared with Assistant. Preserve canonical evidence and remove proposal-as-proof and proxy-driven diagnostic policy while retaining deterministic authority and independent verification. Execute the ordered redesign plan in docs/qualification/PATROL_ASSISTANT_CUSTOMER_JOURNEY.md, qualify unhealthy service diagnosis, backup or capacity risk and supported VM/LXC actions with negative controls, and record real model quality, latency and cost limits. Independent volunteered Pro qualification remains required for wider readiness. Local implementation and the named qualification matrix are delivered through core PR1960 and enterprise PR23. The candidate remains open for the independent-environment release gate. The subsequent canonical incident-history continuation is still in qualification, including a recorded failed model explanation. Do not treat that continuation as delivered until its final evidence and landing are recorded.",
"status": "proposed",
"recorded_at": "2026-09-05",
"target_id": "v6-product-lane-expansion",
@@ -10403,16 +10403,16 @@
],
"work_claims": [
{
"id": "release-snapshot-identity-coverage-gap-release-train-exact-candidate-promotion",
"agent_id": "release-snapshot-identity",
"summary": "Implement immutable release snapshots in release workflow, validation and promotion policy",
"id": "patrol-incident-history-continuation-coverage-gap-patrol-assistant-customer-outcome-qualification",
"agent_id": "patrol-incident-history-continuation",
"summary": "Canonical incident history and final linked Assistant qualification",
"target_id": "v6-product-lane-expansion",
"claimed_at": "2026-09-07T17:58:01Z",
"heartbeat_at": "2026-09-07T17:58:01Z",
"expires_at": "2026-09-07T19:58:01Z",
"claimed_at": "2026-09-07T20:00:54Z",
"heartbeat_at": "2026-09-08T00:40:59Z",
"expires_at": "2026-09-08T02:40:59Z",
"work_item": {
"kind": "coverage-gap",
"id": "release-train-exact-candidate-promotion"
"id": "patrol-assistant-customer-outcome-qualification"
}
}
],
docs/release-control/v6/internal/subsystems/agent-lifecycle.md
+5
View File
@@ -15,6 +15,11 @@
## Purpose
Historical incident reads preserve canonical event targets and treat resource
aliases as read selectors. They do not grant command authority to an alias or
infer present agent liveness from an old alert closure. A failed canonical read
is exposed as unavailable, rather than replaced with a synthetic healthy state.
Delivery-log diagnostic text is a notification-attempt projection, not an agent admission or liveness verdict. Masking embedded destination credentials preserves failure context without changing agent identities, credentials, session replacement or removal policy. Consumers must use the retained failureClass and outcome as delivery evidence only; a transport error does not establish that the monitored agent is offline.
### Canonical Patrol and Assistant continuation, 2026-09-07
docs/release-control/v6/internal/subsystems/ai-runtime.md
+52 -23
View File
@@ -42,22 +42,20 @@ make acknowledgement event replay idempotent, or prove aggregate write-byte
reductions or recipient delivery. An evicted occurrence can still be recreated
by replay; exact identity is guaranteed only while its shell is retained.
Canonical timeline reads for retained incident shells select events from the
shell's exact opening time up to (but not including) the next retained opening
for the same alert/resource. A newer firing must not reopen an older resolved
incident, and a subsecond recurrence must not inherit its predecessor's
resolution or acknowledgement. Local analysis annotations remain attached to
their shell. This is a read-projection boundary, not event deletion or a change
to notification delivery. Canonical-only fallback when no shell is retained,
missing recurrence boundaries after retention, and already-duplicated shells
remain outside this guarantee. Legacy shell lookup retains its time tolerance;
events preceding the selected shell's exact opening are not projected into it.
`TestIncidentStore_CanonicalProjectionOccurrenceBounds` verifies both boundaries,
subsecond starts, unordered successors and unrelated alert/resource isolation.
It also checkpoints all retained shells to JSON, reconstructs the incident store
and replays a firing: projected identity/state stays identical and the unchanged
checkpoint is not replaced. The canonical timeline stays in memory; this is not
a durable event-store restart or installed-process write-rate measurement.
The shared canonical incident query supersedes the earlier retained-shell-only
projection. Explicit firing boundaries, resource history identities and retained
shell boundaries isolate repeated occurrences. An unrelated resource's shell
cannot shorten the selected occurrence merely because its alert identifier
matches. Canonical-only occurrences and duplicate saved-shell notes use the same
query. Query bounds and read failures remain explicit. The upstream
`TestIncidentStore_CanonicalProjectionOccurrenceBounds` regression is preserved
alongside the new canonical-only, alias and duplicate-note cases.
The same regression checkpoints retained shells to JSON, reconstructs the store
and replays a firing. Stable projected identity and events remain identical,
while query observation bounds describe each fresh read. The unchanged JSON
checkpoint is not replaced. The canonical timeline stays in memory, so this is
not a durable event-store restart or installed-process write-rate measurement.
### Unchanged incident JSON checkpoints
@@ -70,6 +68,23 @@ does it eliminate snapshot serialization. Proof:
`internal/ai/memory/incidents_unchanged_test.go` covers unchanged evaluations,
metadata, restart, resolution/recurrence, file loss and failed-write retry.
Assistant owns composer registration and focus on every open, rather than only
on component mount. Closing clears the registered input so later keyboard
commands cannot target a detached composer. A handoff must leave Escape and
keyboard input in Assistant, not the underlying alert search.
### Canonical incident-history queries
Incident context reads `IncidentStore.QueryIncidents`, whose page contains
canonical evidence coverage and explicit truncation. Read failures become
unavailable context, not a healthy or empty-history claim. Explicit firing times
separate occurrences, and missing starts stay unknown. Historical lifecycle
status does not establish current resource health. Assistant handoff includes
the latest bounded events, their source provenance and query coverage. It
preserves the existing exclusion of raw command arguments/output. An absent
closure does not acquire a duration measured up to the current time.
### Canonical Patrol and Assistant continuation, 2026-09-07
Canonical investigation planning runs inside the model's tool call. The model
@@ -8275,10 +8290,24 @@ distinguishes unavailable archives, failed reads and absent exact resource/windo
pairs. Proof lives in `internal/metrics/incident_archive_test.go` and the
registered-tool cases in `internal/ai/tools/incident_history_test.go`.
The legacy incidents listing still exposes incident memory and is not a complete
canonical incident query. Its old sampler-derived `active_count` is now null with
`active_count_status=not_measured`. Canonical-only events, alias-aware listing,
query bounds and projection-read errors remain an explicit modernization gap in
`patrol-assistant-customer-outcome-qualification`. The shared resource timeline
remains the evidence owner. Do not invent another incident lifecycle to repair
this listing.
The incidents listing now uses the shared canonical incident query, including
canonical-only events, alias-aware selection, explicit query bounds and surfaced
read errors. Saved incident shells preserve occurrence identity and local notes.
They do not own a second lifecycle. The old sampler-derived `active_count` stays
null with `active_count_status=not_measured`. Implementation and qualification
receipts are recorded in `docs/qualification/PATROL_ASSISTANT_CUSTOMER_JOURNEY.md`.
The wider customer-outcome qualification gate remains open.
The history tool describes its actual read scope: alert resolution records
closure rather than workload recovery, related-resource IDs identify relationships
rather than additional event targets, and the read does not query action records.
The model owns interpretation of those records. A history-only read cannot prove
that a governed action exists or does not exist.
Canonical alert lifecycle evidence owns the projected incident's resource ID,
resource type, alert type, level and message when present. Saved shells keep
stable incident IDs and local notes, but cannot override that canonical context.
Related command targets do not retarget an alert. The funded OpenRouter
`openai/gpt-6-astra` route has explicitly reviewed, tiered cost estimates dated
2026-09-07. Unreviewed aliases stay unknown. Its history-explanation qualification
is distinct from the earlier Gemini lab matrix and from wider rollout evidence.
docs/release-control/v6/internal/subsystems/alerts.md
+18
View File
@@ -15,6 +15,24 @@
## Purpose
Mobile incident drawers transfer their exact context to Assistant and close
through the shared explicit handoff callback. Keeping the source drawer above
Assistant, or dropping its occurrence identity to make navigation work, fails
the linked investigation journey. Both timeline and resource handoffs require
mounted regression and final-build narrow browser proof.
### Incident history preserves unavailable reads and provenance
Timeline and resource-history views use the shared incident projection. A failed
canonical read stays visibly unavailable, including after an error toast expires
or when cached evidence remains. Empty-history copy is shown only after a
successful empty read. Retry or Refresh repeats the read without discarding
saved notes. The event card keeps canonical observation/occurrence timestamps,
source, actor and record ID behind the same accessible evidence disclosure.
Unknown timestamps remain unavailable. Truncation is explicit. The Assistant
handoff preserves occurrence identity and the latest bounded event window, and
labels history as insufficient to establish current health.
### Retained-queue recovery feedback has no reading deadline
Retry and Dismiss failures have a view-local untimed equivalent beside the
docs/release-control/v6/internal/subsystems/api-contracts.md
+15 -2
View File
@@ -22,6 +22,18 @@
The delivery-log response redacts embedded webhook URLs in errorMessage rather than treating the entire diagnostic as a URL. Non-secret operation and failure context survive alongside notificationId, destinationId, outcome, failureClass and attempts; malformed URL diagnostics fail closed. The HTTP payload proof is TestContract_DeliveryDiagnosticPayload; notification handler tests pin plain, credential-bearing and malformed errors. No response keys or route permissions change.
### Canonical incident-history query contract
Incident APIs discover canonical-only occurrences and retain exact-start
selection. `/api/ai/incidents` returns structured incidents even without a
resource filter, plus history coverage and the existing unmeasured active-count
contract. Resource-detail summaries describe the same page, not a second read.
Canonical read failures return HTTP 503 before snapshot fallback. Timeline
exports preserve source event identity, observed/occurred timestamps, actor,
source, confidence, metadata and related identities. Notes can attach to a
canonical-only occurrence without storing a second canonical lifecycle.
### Canonical Patrol and Assistant continuation, 2026-09-07
Canonical planning owns persisted actor/request idempotency. The same trusted
@@ -10754,8 +10766,9 @@ anomalies and marks historical recording status as historical.
`active_count_status=not_measured` in every response. Incident memory, an empty
result and unavailable services cannot establish a current count. The old
coordinator never received production alert callbacks, so its zero was not a
measurement. The legacy listing's broader canonical query and read-error
modernization remains open under the customer-outcome qualification gap.
measurement. The shared QueryIncidents path now owns structured listing and
read-error propagation. Its qualification and delivery receipts remain in the
customer-outcome qualification record, separate from wider readiness.
### Explicit credentials during development qualification
docs/release-control/v6/internal/subsystems/frontend-primitives.md
+38
View File
@@ -20,6 +20,28 @@
## Purpose
Assistant owns composer registration and focus on every open, rather than only
on component mount. Closing clears the registered input so later keyboard
commands cannot target a detached composer. A handoff must leave Escape and
keyboard input in Assistant, not the underlying alert search.
Mobile incident drawers transfer their exact context to Assistant and close
through the shared explicit handoff callback. Keeping the source drawer above
Assistant, or dropping its occurrence identity to make navigation work, fails
the linked investigation journey. Both timeline and resource handoffs require
mounted regression and final-build narrow browser proof.
### Shared incident evidence disclosure
Alerts timeline and resource-history events share IncidentTimelineEventCard.
Its native details/summary disclosure preserves keyboard activation and keeps
forensic provenance out of the default event summary. The same timestamp
formatter rejects missing or invalid evidence times. Timeline and resource
history compose persistent failure copy with their existing retry controls,
independently of transient notification toasts. The affected interaction and
viewport qualification is recorded in
`docs/qualification/PATROL_ASSISTANT_CUSTOMER_JOURNEY.md`.
### Canonical Patrol and Assistant continuation, 2026-09-07
Patrol's Assistant context preserves unknown destructive risk and distinguishes
@@ -7222,3 +7244,19 @@ widths despite the longer labels. The presentation and Overview delivery-status
tests cover the evidence boundary; `scripts/check-alert-dispatch-copy.mjs`
qualifies the real Overview with scripted API data in Chromium, not installed
notification delivery.
The incident-history continuation also projects retained desktop expansion into
the mobile drawer at the shared CSS breakpoint and keeps expansion state on
return to desktop. The mobile action label describes the visible drawer.
Operator note text and attribution survive the Assistant handoff while raw
command output remains excluded. Shared event cards override inherited table
no-wrap styling, and notes preserve line breaks. Long notes must remain readable
in both inline desktop history and the mobile drawer. Final source-bound
browser/model qualification is recorded in the customer-journey document.
The shared type-to-search registry excludes inputs in inert modal backgrounds,
including prepared shortcut targets. Escape belongs to the active dialog and
must not clear a background history filter or invalidate its return-focus target.
Ordinary search shortcuts resume when the background becomes interactive again.
When Assistant is already open, a desktop-to-phone transition must retain that
destination rather than reopening the underlying history drawer above it.
docs/release-control/v6/internal/subsystems/monitoring.md
+9
View File
@@ -47,6 +47,15 @@ snapshots and actual memory-threshold evaluation, plus stale/offline/unavailable
samples and other-instance lookup. It is synthetic runtime proof, not a live
FreeBSD installation, automatic identity-match or browser receipt.
### Incident-history reconciliation boundary
Active-alert timeline reconciliation uses the shared error-bearing incident
query. If canonical history is unavailable, reconciliation stops for that alert
and records the read failure. It must not treat a failed read as proof that a
firing event is absent. Historical source messages own alert condition wording.
Projection does not manufacture comparison direction from numeric placeholders.
Container filesystem observations are native, resource-scoped reads. The shared
`pkg/agents/filesystem` contract keeps measurement time, mountpoint, source,
filesystem type and optional usage together. Unavailable reads have an error
docs/release-control/v6/internal/subsystems/registry.json
+19
View File
@@ -3219,6 +3219,7 @@
"frontend-modern/src/api/__tests__/alertIntentPolicies.test.ts",
"internal/api/alerting/alerts_test.go",
"internal/api/alerting/deadman_handlers_test.go",
"internal/api/alerting/incident_query_test.go",
"internal/api/alerts_endpoints_test.go"
]
},
@@ -5204,6 +5205,7 @@
"frontend-modern/src/features/platformPage/sharedPlatformPage.tsx",
"frontend-modern/src/features/platformPage/usePlatformWindowedItems.ts",
"frontend-modern/src/hooks/createNonSuspendingQuery.ts",
"frontend-modern/src/hooks/useTypeToSearch.ts",
"frontend-modern/src/i18n/index.ts",
"frontend-modern/src/i18n/messages.de.ts",
"frontend-modern/src/i18n/messages.es.ts",
@@ -5246,6 +5248,21 @@
],
"require_explicit_path_policy_coverage": true,
"path_policies": [
{
"id": "type-to-search-keyboard-ownership",
"label": "shared search keyboard and modal ownership proof",
"match_prefixes": [],
"match_files": [
"frontend-modern/src/hooks/useTypeToSearch.ts"
],
"allow_same_subsystem_tests": false,
"test_prefixes": [],
"exact_files": [
"frontend-modern/src/components/shared/__tests__/Dialog.test.tsx",
"frontend-modern/src/components/shared/__tests__/SearchInput.test.tsx",
"frontend-modern/src/hooks/__tests__/useTypeToSearch.test.ts"
]
},
{
"id": "tracked-planning-document-status-audit",
"label": "tracked planning-document status audit proof",
@@ -6396,6 +6413,7 @@
"internal/monitoring/issue1638_dns_cache_test.go",
"internal/monitoring/metric_window_provider_test.go",
"internal/monitoring/monitor_additional_test.go",
"internal/monitoring/monitor_alert_handling_test.go",
"internal/monitoring/monitor_alert_intent_test.go",
"internal/monitoring/monitor_alert_override_migration_test.go",
"internal/monitoring/monitor_backups_readstate_test.go",
@@ -8151,6 +8169,7 @@
"internal/unifiedresources/ceph_pool_health_contract_test.go",
"internal/unifiedresources/history_identity_test.go",
"internal/unifiedresources/host_storage_cleanup_test.go",
"internal/unifiedresources/incident_history_query_test.go",
"internal/unifiedresources/monitor_adapter_read_state_test.go",
"internal/unifiedresources/views_test.go"
]
docs/release-control/v6/internal/subsystems/storage-recovery.md
+6
View File
@@ -21,6 +21,12 @@
## Purpose
Incident history retains canonical storage-related risk, source identity and
separate observation/occurrence timestamps. A recorded alert closure or operator
note is not a restore result. Canonical read failure remains unavailable and
bounded or partial history cannot establish current storage health or backup
coverage. Recovery judgment still requires independent current evidence.
Delivery-log URL masking occurs in the HTTP diagnostic projection and does not rewrite retained attempt history or destination configuration. Preserved failure context and attempt identifiers describe notification delivery only, not a restore result or storage freshness. Malformed URL diagnostics fail closed in the response; this does not delete the retained failure, change retention windows, or grant retry, storage or recovery authority.
### Canonical Patrol and Assistant continuation, 2026-09-07
docs/release-control/v6/internal/subsystems/unified-resources.md
+16
View File
@@ -37,6 +37,22 @@ policy remains authoritative; this accessor introduces no second timeout.
`TestCorrelatedGuestMemoryNextPoll` verifies the manually merged row through the
next-poll consumers. Existing host-list membership and metric priority are unchanged.
### Bounded incident-history selection
Canonical history queries filter exact alert identifiers and observation windows
before applying limits. The observation window is inclusive at `since` and
exclusive at `ObservedBefore`. All aggregate counts use the same filter
predicate. Both stores order by observation time descending, then event ID
descending, including late-arriving records. SQLite indexes the same guarded
alert-identity expression used by the query. Invalid legacy JSON and non-string
metadata cannot establish an alert identity.
`ResourceHistoryIDs` exposes the existing alias/succession expansion for
read-only incident memory association. It does not rewrite event resource IDs,
resource operator state, approvals, action request identity or execution
capabilities. An unavailable canonical read remains an error.
### Canonical Patrol and Assistant continuation, 2026-09-07
Persisted action request identity is scoped to the trusted actor and exact
frontend-modern/browser-verification.json
+56 -14
View File
@@ -1,35 +1,77 @@
{
"version": 1,
"base_sha": "c857802df6b00e61e2229eea411d6675deff2d8a",
"verified_at": "2026-09-07T22:03:54.189527Z",
"base_sha": "977afdd9559c0e9d5859f4c79bcc48e889381bba",
"verified_at": "2026-09-08T00:40:37.659048Z",
"result": "passed",
"changed_paths": [
"frontend-modern/src/utils/resourceStateAdapters.ts"
"frontend-modern/src/components/AI/Chat/index.tsx",
"frontend-modern/src/components/Alerts/IncidentAssistantHandoffButton.tsx",
"frontend-modern/src/components/Alerts/IncidentTimelineEventCard.tsx",
"frontend-modern/src/components/Alerts/IncidentTimelinePanel.tsx",
"frontend-modern/src/components/Alerts/incidentAssistantHandoffModel.ts",
"frontend-modern/src/features/alerts/AlertHistoryItemActions.tsx",
"frontend-modern/src/features/alerts/AlertHistoryMobileList.tsx",
"frontend-modern/src/features/alerts/AlertResourceIncidentsPanel.tsx",
"frontend-modern/src/features/alerts/MobileAlertHistoryInvestigationDialog.tsx",
"frontend-modern/src/features/alerts/useAlertHistoryState.ts",
"frontend-modern/src/features/alerts/useAlertResourceIncidentsState.ts",
"frontend-modern/src/hooks/useTypeToSearch.ts",
"frontend-modern/src/types/api.ts",
"frontend-modern/src/utils/alertIncidentPresentation.ts"
],
"content_sha256": {
"frontend-modern/src/utils/resourceStateAdapters.ts": "ce1f8f2aef4b04b64dd2533c09458349811ba7cec9ec2e4bd8634b8e9c3c848f"
"frontend-modern/src/components/AI/Chat/index.tsx": "ca9e630294db426df591c3386d4fec161f34ade0bb6a8c4ea769130545a12341",
"frontend-modern/src/components/Alerts/IncidentAssistantHandoffButton.tsx": "b7a6ea6676ed9834d8254bb4ddab1301ec3b0d2368a96fa3e91e9ab5d2cefb4f",
"frontend-modern/src/components/Alerts/IncidentTimelineEventCard.tsx": "21e90602de00aa4765b1c40b5295a618a237501389d50ca9dbe36454abd11f32",
"frontend-modern/src/components/Alerts/IncidentTimelinePanel.tsx": "7298d4184c243325dbd671777257c7efd66c86091eb4b83663315e2bdc131fcb",
"frontend-modern/src/components/Alerts/incidentAssistantHandoffModel.ts": "9e777eee1be46f33ac146770fac6193cf6cd1c8b33ab0d0cd72fe57b98907a37",
"frontend-modern/src/features/alerts/AlertHistoryItemActions.tsx": "06e75d16ccad383b7c89d109baaee71852482f49b41ef3d0f5cc99768dd35101",
"frontend-modern/src/features/alerts/AlertHistoryMobileList.tsx": "851a2c1dc11ccbc827a6a50dfe06088b13a7307ce489b0767804daf6f33d023c",
"frontend-modern/src/features/alerts/AlertResourceIncidentsPanel.tsx": "c6f89c4991c9cfc2b1c8c82f7fed35af93b3b0b29b16b2c4bdc61a51005bec4c",
"frontend-modern/src/features/alerts/MobileAlertHistoryInvestigationDialog.tsx": "55af36a7194c2dbe9d944ca08f52e794f2b5e2ccbe79427bc55abe66aaa8a522",
"frontend-modern/src/features/alerts/useAlertHistoryState.ts": "652a09b02321ad1cfd087aacc7937a6afb5cfb7da0e0d6ca997b30d5c9ac53bd",
"frontend-modern/src/features/alerts/useAlertResourceIncidentsState.ts": "f9d44d5bd2199afb274b329877e19e891942106a63cecbeb26796fa4c7b3b507",
"frontend-modern/src/hooks/useTypeToSearch.ts": "b3f5244fd951baedc33fc8efee32c28077d94c07e5fce958eb6611bf50992a20",
"frontend-modern/src/types/api.ts": "82177cc67fc449a28d2d3cc226605ae091bd1f83fbdb001d15b3b430e5cd4100",
"frontend-modern/src/utils/alertIncidentPresentation.ts": "2e098b24b04c047a1b024a93608c5133720660cafc54c40408972413786a0a07"
},
"routes": [
"/proxmox"
"/alerts"
],
"viewports": [
{
"width": 1280,
"height": 900
"width": 1440,
"height": 1000
},
{
"width": 900,
"height": 1000
},
{
"width": 768,
"height": 1000
},
{
"width": 767,
"height": 1000
},
{
"width": 390,
"height": 844
"height": 1000
}
],
"states": [
"Canonical 100% to 35%; stale/offline/unavailable agent evidence retains trusted 100%.",
"Explicit Proxmox withdrawal with omitted canonical memory: table N/A, Memory section absent.",
"Measured-zero recovery: table 0%, Memory Total and Free both 100 MB; second cluster stays 80%."
"Canonical merged occurrence and source evidence, resolved alert with independently unknown current health.",
"Loading and disabled Refresh, initial/cached read failures, Retry/Refresh, successful empty, partial history and unknown times.",
"Saved operator note and unsaved draft across desktop/phone layout changes, long-note wrapping and persisted reload.",
"Timeline/resource handoff into Assistant, reopened composer focus, saved funded history-and-note response across widths."
],
"interactions": [
"Open guest drawer; deliver synthetic canonical socket snapshots while open.",
"Resize to desktop and narrow Chromium; scroll recovered Memory into view above fixed navigation; inspect all five final screenshots."
"Expand/collapse native Evidence details by keyboard, All/None event filters and note draft enable/clear.",
"Expand resource Events, inject read errors, retry and reload.",
"Resize source and open Assistant destination across 390/900/767/768/1440 widths.",
"Type in a modal without clearing background search, Escape and return focus to the selected card.",
"Resume exact saved session 81c92125-cdc3-4611-931a-687e6548bf65, scroll through its note explanation and inspect actual pixels."
],
"notes": "Synthetic Desktop Chrome acceptance only, not touch/mobile-device, installed poller, FreeBSD reporter or alert-delivery proof. Base reproduced stale 100% after withdrawal; adapter repair passed final Chromium test without retries and pageerror assertion. Earlier fixture wrongly expected raw-facet total after withdrawal; useWorkloads maps canonical metrics only. Two selector/absent-section failures and first passing but bottom-nav-obscured screenshot run are retained, not counted as final visual acceptance. Final four transition screenshots show withdrawn section absent and recovered Total/Free in frame at both widths; initial 35% screenshot is table-only. Evidence: /var/lib/pulse-maintainer/queue/staging/20260907T215015Z-web-product/ (moves to completed): browser-base.log, browser-repair.log, browser-repair-selector.log, browser-canonical.log, browser-final.log, final-report/data/*.png. Browser command: pulse-heavy-run -- bash wrapper starting Vite 127.0.0.1:5187 then PLAYWRIGHT_BASE_URL=http://127.0.0.1:5187 playwright test tests/86-hybrid-memory-browser.spec.ts --project=chromium --retries=0; Vite stopped by wrapper trap."
"notes": "Final local Chromium functional and visual acceptance of runtime 0.0.0-dev-pro+incident-history-r10, binary SHA256 2e1958d0856c812d22870144cf228509f2a4a3696258a533d34802442f500edb. Commands and retained failed attempts are in docs/qualification/PATROL_ASSISTANT_CUSTOMER_JOURNEY.md. Receipt directories: /Volumes/Development/pulse/tmp/patrol-incident-history/browser, browser-states, browser-resize and browser-saved-explanation. No new model calls in final browser repetition. Funded changed-context r9 response remains applicable because r10 changes only responsive destination ownership. Controlled HTTP variants are not a database outage. Not touch-device, production rollout, backup/restore or population reliability proof. Upstream 977afdd adds tests and contract documentation only. All 14 accepted frontend content hashes remain identical."
}
frontend-modern/src/components/AI/Chat/__tests__/AIChat.test.tsx
+19
View File
@@ -1078,6 +1078,25 @@ describe('AIChat', () => {
});
});
it('focuses and registers the current composer after each reopen', async () => {
const [open, setOpen] = createSignal(false);
mockAiChatStore.isOpenSignal.mockImplementation(open);
renderChat();
expect(screen.queryByPlaceholderText('Ask about your infrastructure...')).toBeNull();
setOpen(true);
const first = await screen.findByPlaceholderText('Ask about your infrastructure...');
await waitFor(() => expect(document.activeElement).toBe(first));
setOpen(false);
expect(mockAiChatStore.registerInput).toHaveBeenLastCalledWith(null);
setOpen(true);
const second = await screen.findByPlaceholderText('Ask about your infrastructure...');
expect(second).not.toBe(first);
await waitFor(() => {
expect(document.activeElement).toBe(second);
expect(mockAiChatStore.registerInput).toHaveBeenLastCalledWith(second);
});
});
it('renders the ChatMessages child component', () => {
renderChat();
expect(screen.getByTestId('chat-messages')).toBeInTheDocument();
frontend-modern/src/components/AI/Chat/index.tsx
+6 -2
View File
@@ -2819,8 +2819,14 @@ export const AIChat: Component<AIChatProps> = (props) => {
createEffect(() => {
if (!isOpen()) {
setShowCommandHelp(false);
aiChatStore.registerInput?.(null);
return;
}
queueMicrotask(() => {
if (!isOpen()) return;
aiChatStore.registerInput?.(textareaRef ?? null);
textareaRef?.focus();
});
openInitialization = initializeWhenOpen();
});
@@ -2915,9 +2921,7 @@ export const AIChat: Component<AIChatProps> = (props) => {
// Click outside handler to close all dropdowns
onMount(() => {
setPromptHistory(loadPromptHistory());
aiChatStore.registerInput?.(textareaRef ?? null);
restoreStashedComposerDraft();
focusComposer();
const handleClickOutside = (e: MouseEvent) => {
const target = e.target as HTMLElement;
frontend-modern/src/components/Alerts/IncidentAssistantHandoffButton.tsx
+10 -1
View File
@@ -7,6 +7,7 @@ interface IncidentAssistantHandoffButtonProps {
incident: Incident;
label?: string;
class?: string;
onAssistantHandoff?: () => void;
}
export function IncidentAssistantHandoffButton(props: IncidentAssistantHandoffButtonProps) {
@@ -19,7 +20,15 @@ export function IncidentAssistantHandoffButton(props: IncidentAssistantHandoffBu
event.stopPropagation();
const handoff = buildAlertIncidentAssistantHandoff({ incident: props.incident });
aiChatStore.open(handoff.context);
const openAssistant = () => aiChatStore.open(handoff.context);
if (props.onAssistantHandoff) {
props.onAssistantHandoff();
// Let the source dialog release its blocking layer before opening the
// destination, as with command-palette handoffs.
queueMicrotask(openAssistant);
} else {
openAssistant();
}
};
return (
frontend-modern/src/components/Alerts/IncidentTimelineEventCard.tsx
+33 -1
View File
@@ -1,6 +1,9 @@
import { Show } from 'solid-js';
import type { IncidentEvent } from '@/types/api';
import {
formatIncidentEvidenceTime,
INCIDENT_EVIDENCE_DETAILS,
INCIDENT_TIME_UNAVAILABLE,
getAlertIncidentTimelineCommandClass,
getAlertIncidentTimelineDetailClass,
getAlertIncidentTimelineEventCardClass,
@@ -33,8 +36,37 @@ export function IncidentTimelineEventCard(props: IncidentTimelineEventCardProps)
<div class={getAlertIncidentTimelineEventCardClass(props.variant)}>
<div class={getAlertIncidentTimelineMetaRowClass()}>
<span class={getAlertIncidentTimelineHeadingClass()}>{props.event.summary}</span>
<span>{new Date(props.event.timestamp).toLocaleString()}</span>
<span>
{formatIncidentEvidenceTime(props.event.timestamp) ?? INCIDENT_TIME_UNAVAILABLE}
</span>
</div>
<Show when={props.event.evidence}>
{(evidence) => (
<details class="text-xs text-muted mt-2">
<summary class="cursor-pointer rounded focus-visible:outline focus-visible:outline-2 focus-visible:outline-offset-2">
{INCIDENT_EVIDENCE_DETAILS}
</summary>
<dl class="mt-2 grid grid-cols-[auto_minmax(0,1fr)] gap-x-3 gap-y-1 break-words">
<dt>Observed</dt>
<dd>
{formatIncidentEvidenceTime(evidence().observedAt) ?? INCIDENT_TIME_UNAVAILABLE}
</dd>
<dt>Occurred</dt>
<dd>
{formatIncidentEvidenceTime(evidence().occurredAt) ?? INCIDENT_TIME_UNAVAILABLE}
</dd>
<dt>Source</dt>
<dd>{evidence().sourceAdapter || evidence().sourceType || 'Unknown'}</dd>
<Show when={evidence().actor}>
<dt>Recorded actor</dt>
<dd>{evidence().actor}</dd>
</Show>
<dt>Record</dt>
<dd>{evidence().id}</dd>
</dl>
</details>
)}
</Show>
<Show when={note()}>
<p class={getAlertIncidentTimelineDetailClass()}>{note()}</p>
</Show>
frontend-modern/src/components/Alerts/IncidentTimelinePanel.tsx
+37 -26
View File
@@ -23,6 +23,9 @@ import {
} from '@/utils/alertOverviewPresentation';
import {
type AlertIncidentEventFilterVariant,
formatIncidentEvidenceTime,
INCIDENT_HISTORY_PARTIAL,
INCIDENT_HISTORY_LEGACY,
getAlertIncidentAcknowledgedBadgeClass,
getAlertIncidentNoteSaveButtonClass,
getAlertIncidentNoteTextareaClass,
@@ -43,6 +46,7 @@ export interface IncidentTimelinePanelProps {
noteSaving: Accessor<boolean>;
onSaveNote: () => void;
onRetry: () => void;
onAssistantHandoff?: () => void;
}
export function IncidentTimelinePanel(props: IncidentTimelinePanelProps) {
@@ -57,6 +61,21 @@ export function IncidentTimelinePanel(props: IncidentTimelinePanelProps) {
{getAlertTimelineLoadingState().text}
</p>
</Show>
<Show when={!props.loading() && props.error()}>
<div class="flex items-center gap-2">
<p class="text-xs text-error" role="alert">
{getAlertTimelineFailureState().text}
</p>
<Button
variant="ghost"
size="sm"
class="text-primary hover:underline"
onClick={() => props.onRetry()}
>
{getAlertTimelineFailureState().actionLabel}
</Button>
</div>
</Show>
<Show when={!props.loading() && timeline()}>
{(loadedTimeline) => (
<div class="space-y-3">
@@ -71,10 +90,10 @@ export function IncidentTimelinePanel(props: IncidentTimelinePanelProps) {
{getAlertTimelineAcknowledgedLabel()}
</span>
</Show>
<Show when={loadedTimeline().openedAt}>
<Show when={formatIncidentEvidenceTime(loadedTimeline().openedAt)}>
<span>
{getAlertTimelineOpenedAtLabel(
new Date(loadedTimeline().openedAt).toLocaleString(),
formatIncidentEvidenceTime(loadedTimeline().openedAt)!,
)}
</span>
</Show>
@@ -86,8 +105,19 @@ export function IncidentTimelinePanel(props: IncidentTimelinePanelProps) {
</span>
</Show>
</div>
<IncidentAssistantHandoffButton incident={loadedTimeline()} />
<IncidentAssistantHandoffButton
incident={loadedTimeline()}
onAssistantHandoff={props.onAssistantHandoff}
/>
</div>
<Show when={loadedTimeline().history?.hasMoreChanges}>
<p class="text-xs text-muted" role="status">
{INCIDENT_HISTORY_PARTIAL}
</p>
</Show>
<Show when={loadedTimeline().history?.source === 'legacy_incident_memory'}>
<p class="text-xs text-muted">{INCIDENT_HISTORY_LEGACY}</p>
</Show>
<Show when={events().length > 0}>
<IncidentEventFilters
filters={props.filters}
@@ -141,29 +171,10 @@ export function IncidentTimelinePanel(props: IncidentTimelinePanelProps) {
</div>
)}
</Show>
<Show when={!props.loading() && !timeline()}>
<Show
when={props.error()}
fallback={
<p class="text-xs text-muted" role="status">
{getAlertTimelineUnavailableState().text}
</p>
}
>
<div class="flex items-center gap-2">
<p class="text-xs text-error" role="alert">
{getAlertTimelineFailureState().text}
</p>
<Button
variant="ghost"
size="sm"
class="text-primary hover:underline"
onClick={() => props.onRetry()}
>
{getAlertTimelineFailureState().actionLabel}
</Button>
</div>
</Show>
<Show when={!props.loading() && !timeline() && !props.error()}>
<p class="text-xs text-muted" role="status">
{getAlertTimelineUnavailableState().text}
</p>
</Show>
</>
);
frontend-modern/src/components/Alerts/__tests__/IncidentTimelineEventCard.test.tsx
+38
View File
@@ -49,4 +49,42 @@ describe('IncidentTimelineEventCard', () => {
expect(screen.queryByText('systemctl restart alert')).not.toBeInTheDocument();
expect(screen.queryByText('stdout: ok')).not.toBeInTheDocument();
});
it('preserves observation and occurrence provenance behind a disclosure', () => {
const { container } = render(() => (
<IncidentTimelineEventCard
variant="surface"
event={makeEvent({
evidence: {
id: 'canonical-record',
resourceId: 'resource-a',
kind: 'alert_fired',
observedAt: '2026-03-18T12:10:00Z',
occurredAt: '2026-03-18T12:00:00Z',
sourceType: 'platform_event',
sourceAdapter: 'proxmox_adapter',
confidence: 'high',
actor: 'operator',
},
})}
/>
));
const details = container.querySelector('details');
expect(details).not.toHaveAttribute('open');
expect(screen.getByText('Evidence details')).toBeInTheDocument();
expect(screen.getByText('Observed')).toBeInTheDocument();
expect(screen.getByText('Occurred')).toBeInTheDocument();
expect(screen.getByText('canonical-record')).toBeInTheDocument();
expect(screen.getByText('operator')).toBeInTheDocument();
});
it('does not fabricate a date for missing evidence time', () => {
render(() => (
<IncidentTimelineEventCard
variant="surface"
event={makeEvent({ timestamp: '0001-01-01T00:00:00Z' })}
/>
));
expect(screen.getByText('Time unavailable')).toBeInTheDocument();
expect(screen.queryByText(/1\/1\/1/)).not.toBeInTheDocument();
});
});
frontend-modern/src/components/Alerts/__tests__/IncidentTimelinePanel.test.tsx
+59
View File
@@ -191,4 +191,63 @@ describe('IncidentTimelinePanel', () => {
});
expect(JSON.stringify(context)).not.toContain('systemctl status pulse');
});
it('discloses a capped history without inventing its opening time', () => {
const [filters, setFilters] = createSignal(new Set(['command']));
render(() => (
<IncidentTimelinePanel
timeline={() =>
makeTimeline({
status: 'unknown',
openedAt: '0001-01-01T00:00:00Z',
history: {
source: 'canonical_resource_history',
observedSince: '2026-03-01T00:00:00Z',
observedBefore: '2026-03-20T12:00:00Z',
changeLimit: 1,
hasMoreChanges: true,
hasMoreIncidents: false,
},
})
}
loading={() => false}
error={() => false}
filters={filters}
setFilters={setFilters}
filterVariant="compact"
eventCardVariant="surface"
noteDraft={() => ''}
onNoteDraftChange={() => undefined}
noteSaving={() => false}
onSaveNote={() => undefined}
onRetry={() => undefined}
/>
));
expect(screen.getByRole('status')).toHaveTextContent(
'Earlier occurrences or events may be missing',
);
expect(screen.getByText('unknown')).toBeInTheDocument();
expect(screen.queryByText(/^opened /)).not.toBeInTheDocument();
});
it('keeps refresh failure and retry visible alongside saved evidence', () => {
const [filters, setFilters] = createSignal(new Set(['command']));
render(() => (
<IncidentTimelinePanel
timeline={() => makeTimeline()}
loading={() => false}
error={() => true}
filters={filters}
setFilters={setFilters}
filterVariant="compact"
eventCardVariant="surface"
noteDraft={() => ''}
onNoteDraftChange={() => undefined}
noteSaving={() => false}
onSaveNote={() => undefined}
onRetry={() => undefined}
/>
));
expect(screen.getByRole('alert')).toHaveTextContent('Failed to load timeline.');
expect(screen.getByRole('button', { name: 'Retry' })).toBeInTheDocument();
expect(screen.getByText('Command executed')).toBeInTheDocument();
});
});
frontend-modern/src/components/Alerts/__tests__/incidentAssistantHandoffModel.branchcov2.test.ts
+9 -46
View File
@@ -55,10 +55,8 @@ interface StrictHandoff {
context: StrictHandoffContext;
}
const DEFAULT_NOW = new Date('2026-03-20T10:05:00Z');
function buildHandoff(incident: Incident, now: Date = DEFAULT_NOW): StrictHandoff {
return buildAlertIncidentAssistantHandoff({ incident, now }) as unknown as StrictHandoff;
function buildHandoff(incident: Incident): StrictHandoff {
return buildAlertIncidentAssistantHandoff({ incident }) as unknown as StrictHandoff;
}
function makeIncident(overrides: Partial<Incident> = {}): Incident {
@@ -114,7 +112,6 @@ describe('formatIncidentDuration (exercised via buildAlertIncidentAssistantHando
openedAt: '2026-03-20T10:00:00Z',
closedAt: '2026-03-20T10:01:00Z',
}),
new Date('2026-03-20T11:00:00Z'),
);
expect(handoff.context.briefing.statusLabel).toBe(`${STATUS_PREFIX}1 min`);
expect(handoff.context.handoffContext).toContain('Duration: 1 min');
@@ -126,7 +123,6 @@ describe('formatIncidentDuration (exercised via buildAlertIncidentAssistantHando
openedAt: '2026-03-20T10:00:00Z',
closedAt: '2026-03-20T10:05:00Z',
}),
new Date('2026-03-20T11:00:00Z'),
);
expect(handoff.context.briefing.statusLabel).toBe(`${STATUS_PREFIX}5 mins`);
});
@@ -137,20 +133,18 @@ describe('formatIncidentDuration (exercised via buildAlertIncidentAssistantHando
openedAt: '2026-03-20T10:00:00Z',
closedAt: '2026-03-20T10:00:00Z',
}),
new Date('2026-03-20T11:00:00Z'),
);
expect(handoff.context.briefing.statusLabel).toBe(`${STATUS_PREFIX}0 mins`);
});
it('clamps a negative (closedAt before openedAt) delta to "0 mins" via Math.max(0, ...)', () => {
it('keeps an inconsistent closure time unknown', () => {
const handoff = buildHandoff(
makeIncident({
openedAt: '2026-03-20T10:05:00Z',
closedAt: '2026-03-20T10:00:00Z',
}),
new Date('2026-03-20T11:00:00Z'),
);
expect(handoff.context.briefing.statusLabel).toBe(`${STATUS_PREFIX}0 mins`);
expect(handoff.context.briefing.statusLabel).toBe(`${STATUS_PREFIX}unknown duration`);
});
it('formats a sub-day >= 60min delta as "Xh Ym" (durationMins >= 60, durationHours < 24 arm)', () => {
@@ -159,7 +153,6 @@ describe('formatIncidentDuration (exercised via buildAlertIncidentAssistantHando
openedAt: '2026-03-20T10:00:00Z',
closedAt: '2026-03-20T11:05:00Z',
}),
new Date('2026-03-20T12:00:00Z'),
);
expect(handoff.context.briefing.statusLabel).toBe(`${STATUS_PREFIX}1h 5m`);
});
@@ -170,7 +163,6 @@ describe('formatIncidentDuration (exercised via buildAlertIncidentAssistantHando
openedAt: '2026-03-20T10:00:00Z',
closedAt: '2026-03-21T14:00:00Z',
}),
new Date('2026-03-22T10:00:00Z'),
);
// 28h elapsed -> floor(28/24)=1 day, 28%24=4 hours
expect(handoff.context.briefing.statusLabel).toBe(`${STATUS_PREFIX}1d 4h`);
@@ -182,7 +174,6 @@ describe('formatIncidentDuration (exercised via buildAlertIncidentAssistantHando
openedAt: 'not-a-valid-date',
closedAt: '2026-03-20T11:00:00Z',
}),
new Date('2026-03-20T12:00:00Z'),
);
expect(handoff.context.briefing.statusLabel).toBe(`${STATUS_PREFIX}unknown duration`);
expect(handoff.context.handoffContext).toContain('Duration: unknown duration');
@@ -194,37 +185,13 @@ describe('formatIncidentDuration (exercised via buildAlertIncidentAssistantHando
openedAt: '2026-03-20T10:00:00Z',
closedAt: 'not-a-valid-date',
}),
new Date('2026-03-20T12:00:00Z'),
);
expect(handoff.context.briefing.statusLabel).toBe(`${STATUS_PREFIX}unknown duration`);
});
it('falls back to `now` when closedAt is undefined (closedAt-falsy ternary arm, nonzero delta)', () => {
const handoff = buildHandoff(
makeIncident({ closedAt: undefined }),
new Date('2026-03-20T10:05:00Z'),
);
expect(handoff.context.briefing.statusLabel).toBe(`${STATUS_PREFIX}5 mins`);
});
it('falls back to `now` yielding zero when closedAt is undefined and now === openedAt', () => {
const handoff = buildHandoff(
makeIncident({ closedAt: undefined }),
new Date('2026-03-20T10:00:00Z'),
);
expect(handoff.context.briefing.statusLabel).toBe(`${STATUS_PREFIX}0 mins`);
});
it('uses closedAt (not now) when closedAt is present (closedAt-truthy ternary arm)', () => {
// closedAt gives 5 mins; now is 1 hour after openedAt and would give 60 mins -> "1h 0m".
const handoff = buildHandoff(
makeIncident({
openedAt: '2026-03-20T10:00:00Z',
closedAt: '2026-03-20T10:05:00Z',
}),
new Date('2026-03-20T11:00:00Z'),
);
expect(handoff.context.briefing.statusLabel).toBe(`${STATUS_PREFIX}5 mins`);
it('keeps an absent closure unrecorded instead of measuring through now', () => {
const handoff = buildHandoff(makeIncident({ closedAt: undefined }));
expect(handoff.context.briefing.statusLabel).toBe(`${STATUS_PREFIX}closure not recorded`);
});
});
@@ -374,17 +341,13 @@ describe('formatContextLine (exercised via buildAlertIncidentAssistantHandoff)',
});
it('renders the "Closed At:" context line when closedAt is a present string', () => {
const handoff = buildHandoff(
makeIncident({ closedAt: '2026-03-20T10:05:00Z' }),
new Date('2026-03-20T11:00:00Z'),
);
const handoff = buildHandoff(makeIncident({ closedAt: '2026-03-20T10:05:00Z' }));
expect(handoff.context.handoffContext).toContain('Closed At: 2026-03-20T10:05:00Z');
});
it('trims a whitespace-padded closedAt value to its inner text', () => {
const handoff = buildHandoff(
makeIncident({ closedAt: ' 2026-03-20T10:05:00Z ' as unknown as string }),
new Date('2026-03-20T11:00:00Z'),
);
expect(handoff.context.handoffContext).toContain('Closed At: 2026-03-20T10:05:00Z');
expect(handoff.context.handoffContext).not.toContain('Closed At: 2026-03-20T10:05:00Z');
@@ -393,7 +356,7 @@ describe('formatContextLine (exercised via buildAlertIncidentAssistantHandoff)',
describe('buildAlertIncidentAssistantHandoff (resource-label, event-count & capping branches)', () => {
it('uses resourceName first (|| chain 1st arm) and builds the full concrete handoff', () => {
const handoff = buildHandoff(makeIncident(), new Date('2026-03-20T10:05:00Z'));
const handoff = buildHandoff(makeIncident());
// 1st arm of the resource-label || chain.
expect(handoff.context.briefing.subject).toBe(
'Critical docker-container-health on checkout-api',
frontend-modern/src/components/Alerts/__tests__/incidentAssistantHandoffModel.test.ts
+70 -2
View File
@@ -44,7 +44,6 @@ describe('incidentAssistantHandoffModel', () => {
it('builds an approval-required incident timeline handoff without raw command payloads', () => {
const handoff = buildAlertIncidentAssistantHandoff({
incident: makeIncident(),
now: new Date('2026-03-20T10:05:00Z'),
});
expect(handoff).not.toHaveProperty('prompt');
@@ -64,7 +63,7 @@ describe('incidentAssistantHandoffModel', () => {
sourceLabel: 'Pulse Alerts',
title: 'Incident timeline attached',
subject: 'Critical docker-container-health on checkout-api',
statusLabel: 'Critical incident · Open · 5 mins',
statusLabel: 'Critical incident · Open · closure not recorded',
detailLines: [
'2 timeline events',
'Node: edge-1',
@@ -110,4 +109,73 @@ describe('incidentAssistantHandoffModel', () => {
expect(JSON.stringify(handoff)).not.toContain('systemctl');
expect(JSON.stringify(handoff)).not.toContain('secret-value');
});
it('includes latest evidence and query bounds without exposing command payloads', () => {
const events = Array.from({ length: 12 }, (_, index) => ({
id: `event-${index}`,
type: index === 11 ? 'alert_resolved' : 'command',
timestamp: '2026-03-20T10:05:00Z',
summary: index === 11 ? 'Independently resolved' : 'sensitive command',
evidence: {
id: `canonical-${index}`,
resourceId: 'resource-a',
kind: 'alert_resolved' as const,
observedAt: '2026-03-20T10:06:00Z',
occurredAt: '2026-03-20T10:05:00Z',
sourceType: 'platform_event' as const,
confidence: 'high' as const,
metadata: { command: 'must not appear' },
},
}));
const incident = makeIncident({
events,
history: {
source: 'canonical_resource_history',
observedSince: '2026-03-01T00:00:00Z',
observedBefore: '2026-03-20T11:00:00Z',
changeLimit: 12,
hasMoreChanges: true,
hasMoreIncidents: false,
},
});
const handoff = buildAlertIncidentAssistantHandoff({ incident });
expect(handoff.context.handoffContext).toContain('Latest 8 of 12 returned events');
expect(handoff.context.handoffContext).toContain('Independently resolved');
expect(handoff.context.handoffContext).toContain('canonical-11');
expect(handoff.context.handoffContext).not.toContain('canonical-0"');
expect(handoff.context.handoffContext).toContain('"hasMoreChanges":true');
expect(handoff.context.handoffContext).toContain('2026-03-20T10:06:00Z');
expect(handoff.context.handoffContext).not.toContain('must not appear');
expect(handoff.context.handoffContext).not.toContain('sensitive command');
});
it('keeps an absent occurrence start unknown in Assistant context', () => {
const handoff = buildAlertIncidentAssistantHandoff({
incident: makeIncident({ openedAt: '0001-01-01T00:00:00Z', status: 'unknown' }),
});
expect(handoff.context.handoffContext).toContain('Opened At: unknown');
expect(handoff.context.handoffContext).toContain('unknown duration');
expect(handoff.context.handoffContext).not.toContain('0001-01-01');
});
});
it('retains operator note content and attribution without forwarding command output', () => {
const incident = makeIncident();
incident.events!.push({
id: 'operator-note',
type: 'note',
timestamp: '2026-03-20T10:04:00Z',
summary: 'Note added by operator',
source: 'operator_note',
details: {
note: 'Keep the old pool until its replacement is verified',
output_excerpt: 'unrelated private output',
},
});
const { context } = buildAlertIncidentAssistantHandoff({ incident });
expect(context.handoffContext).toContain(
'Note added by operator: Keep the old pool until its replacement is verified',
);
expect(context.handoffContext).toContain('source=operator_note');
expect(context.handoffContext).not.toContain('unrelated private output');
expect(context.handoffContext).not.toContain('token=secret-value');
});
frontend-modern/src/components/Alerts/incidentAssistantHandoffModel.ts
+60 -12
View File
@@ -1,10 +1,10 @@
import type { AIChatContext } from '@/stores/aiChat';
import type { Incident, IncidentEvent } from '@/types/api';
import { formatIncidentEvidenceTime } from '@/utils/alertIncidentPresentation';
import { resolveAlertTargetType } from '@/utils/alertTargetTypes';
interface BuildAlertIncidentAssistantHandoffInput {
incident: Incident;
now?: Date;
}
interface AlertIncidentAssistantHandoff {
@@ -12,6 +12,19 @@ interface AlertIncidentAssistantHandoff {
}
interface SanitizedIncidentEvent {
source?: string;
evidence?: Pick<
NonNullable<IncidentEvent['evidence']>,
| 'id'
| 'resourceId'
| 'kind'
| 'observedAt'
| 'occurredAt'
| 'sourceType'
| 'sourceAdapter'
| 'actor'
| 'confidence'
>;
id: string;
type: string;
timestamp: string;
@@ -30,7 +43,6 @@ const LABEL_INITIALISMS: Record<string, string> = {
export function buildAlertIncidentAssistantHandoff({
incident,
now = new Date(),
}: BuildAlertIncidentAssistantHandoffInput): AlertIncidentAssistantHandoff {
const resourceLabel = incident.resourceName || incident.resourceId || 'unknown resource';
const targetType = resolveAlertTargetType({
@@ -40,7 +52,7 @@ export function buildAlertIncidentAssistantHandoff({
});
const levelLabel = formatIncidentLabel(incident.level);
const statusLabel = formatIncidentLabel(incident.status);
const durationText = formatIncidentDuration(incident.openedAt, incident.closedAt, now);
const durationText = formatIncidentDuration(incident.openedAt, incident.closedAt);
const events = sanitizeIncidentEvents(incident.events ?? []);
const eventCount = events.length;
const eventCountLabel = `${eventCount} timeline event${eventCount === 1 ? '' : 's'}`;
@@ -79,7 +91,7 @@ export function buildAlertIncidentAssistantHandoff({
incident.message ? `Message: ${incident.message}` : undefined,
].filter((line): line is string => Boolean(line)),
evidence: events
.slice(0, MAX_BRIEFING_EVENTS)
.slice(-MAX_BRIEFING_EVENTS)
.map((event) => `${formatIncidentLabel(event.type)}: ${event.summary}`),
actionLabel: `Discuss incident ${incident.id}`,
safetyNote: 'Diagnostics and remediation require operator approval.',
@@ -95,11 +107,13 @@ export function buildAlertIncidentAssistantHandoff({
resourceType: incident.resourceType,
node: incident.node,
instance: incident.instance,
openedAt: incident.openedAt,
openedAt: formatIncidentEvidenceTime(incident.openedAt) ? incident.openedAt : undefined,
closedAt: incident.closedAt,
acknowledged: incident.acknowledged,
eventCount,
eventSummaries: events.slice(0, MAX_CONTEXT_EVENTS),
includedEventCount: Math.min(eventCount, MAX_CONTEXT_EVENTS),
history: incident.history ? { ...incident.history } : undefined,
eventSummaries: events.slice(-MAX_CONTEXT_EVENTS),
},
},
};
@@ -123,11 +137,11 @@ function buildIncidentAssistantModelContext({
eventCountLabel: string;
}): string {
const eventLines = events
.slice(0, MAX_CONTEXT_EVENTS)
.slice(-MAX_CONTEXT_EVENTS)
.map((event, index) =>
formatContextLine(
`Timeline Event ${index + 1}`,
`${event.timestamp} | ${formatIncidentLabel(event.type)} | ${event.summary}`,
`${event.timestamp} | ${formatIncidentLabel(event.type)} | ${event.summary}${event.evidence ? ` | evidence=${JSON.stringify(event.evidence)}` : event.source ? ` | source=${event.source}` : ''}`,
),
);
@@ -144,10 +158,22 @@ function buildIncidentAssistantModelContext({
formatContextLine('Resource Type', incident.resourceType),
formatContextLine('Node', incident.node),
formatContextLine('Instance', incident.instance),
formatContextLine('Opened At', incident.openedAt),
formatContextLine(
'Opened At',
formatIncidentEvidenceTime(incident.openedAt) ? incident.openedAt : 'unknown',
),
formatContextLine('Closed At', incident.closedAt),
formatContextLine('Duration', durationText),
formatContextLine('Timeline Summary', eventCountLabel),
formatContextLine(
'Included Events',
`Latest ${Math.min(events.length, MAX_CONTEXT_EVENTS)} of ${events.length} returned events`,
),
formatContextLine(
'History Coverage',
incident.history ? JSON.stringify(incident.history) : 'not recorded',
),
'History Boundary: Retained historical evidence does not establish current resource health.',
formatContextLine('Message', incident.message),
...eventLines,
'Timeline Boundary: Command events are summarized only. Raw command details and output stay in the incident or governed approval surface.',
@@ -163,6 +189,22 @@ function sanitizeIncidentEvents(events: IncidentEvent[]): SanitizedIncidentEvent
type: event.type,
timestamp: event.timestamp,
summary: sanitizeIncidentEventSummary(event),
...(event.source ? { source: event.source } : {}),
...(event.evidence
? {
evidence: {
id: event.evidence.id,
resourceId: event.evidence.resourceId,
kind: event.evidence.kind,
observedAt: event.evidence.observedAt,
occurredAt: event.evidence.occurredAt,
sourceType: event.evidence.sourceType,
sourceAdapter: event.evidence.sourceAdapter,
actor: event.evidence.actor,
confidence: event.evidence.confidence,
},
}
: {}),
}));
}
@@ -173,6 +215,10 @@ function sanitizeIncidentEventSummary(event: IncidentEvent): string {
}
const summary = event.summary.trim();
if (normalizedType === 'note' && typeof event.details?.note === 'string') {
const note = event.details.note.trim();
if (note) return `${summary || 'Operator note'}: ${note}`;
}
return summary.length > 0 ? summary : 'Timeline event recorded';
}
@@ -185,10 +231,12 @@ function formatContextLine(
return text ? `${label}: ${text}` : undefined;
}
function formatIncidentDuration(openedAt: string, closedAt: string | undefined, now: Date): string {
function formatIncidentDuration(openedAt: string, closedAt: string | undefined): string {
if (!formatIncidentEvidenceTime(openedAt)) return 'unknown duration';
const openedMs = new Date(openedAt).getTime();
const closedMs = closedAt ? new Date(closedAt).getTime() : now.getTime();
if (!Number.isFinite(openedMs) || !Number.isFinite(closedMs)) {
if (!closedAt) return 'closure not recorded';
const closedMs = new Date(closedAt).getTime();
if (!Number.isFinite(openedMs) || !Number.isFinite(closedMs) || closedMs < openedMs) {
return 'unknown duration';
}
frontend-modern/src/features/alerts/AlertHistoryItemActions.tsx
+4 -1
View File
@@ -14,6 +14,7 @@ interface AlertHistoryItemActionsProps {
touchSized?: boolean;
onTimelineClick?: () => void;
onResourceClick?: () => void;
timelineOpen?: boolean;
}
export function AlertHistoryItemActions(props: AlertHistoryItemActionsProps) {
@@ -40,7 +41,9 @@ export function AlertHistoryItemActions(props: AlertHistoryItemActionsProps) {
);
}}
>
{props.state.expandedIncidents().has(rowKey()) ? 'Hide' : 'Timeline'}
{(props.timelineOpen ?? props.state.expandedIncidents().has(rowKey()))
? 'Hide'
: 'Timeline'}
</button>
</Show>
<Show when={props.alert.source === 'alert' && props.alert.resourceId}>
frontend-modern/src/features/alerts/AlertHistoryMobileList.tsx
+49 -7
View File
@@ -1,4 +1,5 @@
import { Show, createMemo, createSignal } from 'solid-js';
import { Show, createMemo, createSignal, onCleanup, onMount } from 'solid-js';
import { aiChatStore } from '@/stores/aiChat';
import { getAlertHistoryResourceTypeBadgeClass } from '@/utils/alertHistoryPresentation';
import {
@@ -41,6 +42,41 @@ export function AlertHistoryMobileList(props: AlertHistoryMobileListProps) {
return items;
});
// The shared expansion state survives layout changes. Project it into the
// phone's single drawer instead of leaving a hidden desktop expansion open.
onMount(() => {
if (typeof window.matchMedia !== 'function') return;
const viewport = window.matchMedia('(min-width: 768px)');
const synchronizeLayout = () => {
if (viewport.matches || aiChatStore.isOpen) {
setInvestigation(null);
return;
}
const alerts = props.state.groupedAlerts().flatMap((group) => group.alerts);
const resourcePanel = props.state.resourceIncidentPanel();
const resourceAlert =
resourcePanel &&
alerts.find((alert) => props.state.getIncidentRowKey(alert) === resourcePanel.rowKey);
if (resourcePanel && resourceAlert) {
setInvestigation({ kind: 'resource', alert: resourceAlert, rowKey: resourcePanel.rowKey });
return;
}
for (const rowKey of [...props.state.expandedIncidents()].reverse()) {
const alert = alerts.find(
(candidate) => props.state.getIncidentRowKey(candidate) === rowKey,
);
if (alert) {
setInvestigation({ kind: 'timeline', alert, rowKey });
return;
}
}
setInvestigation(null);
};
synchronizeLayout();
viewport.addEventListener('change', synchronizeLayout);
onCleanup(() => viewport.removeEventListener('change', synchronizeLayout));
});
const getInvestigationReturnFocusTarget = (
current: MobileAlertHistoryInvestigation,
): HTMLElement | null => {
@@ -54,18 +90,20 @@ export function AlertHistoryMobileList(props: AlertHistoryMobileListProps) {
return action ?? list;
};
const closeInvestigation = () => {
const closeInvestigation = (restoreFocus = true) => {
const current = investigation();
if (!current) return;
const currentFocusTarget = getInvestigationReturnFocusTarget(current);
setInvestigation(null);
currentFocusTarget?.focus({ preventScroll: true });
if (restoreFocus) {
currentFocusTarget?.focus({ preventScroll: true });
requestAnimationFrame(() => {
requestAnimationFrame(() => {
getInvestigationReturnFocusTarget(current)?.focus({ preventScroll: true });
requestAnimationFrame(() => {
getInvestigationReturnFocusTarget(current)?.focus({ preventScroll: true });
});
});
});
}
if (current.kind === 'timeline') {
if (props.state.expandedIncidents().has(current.rowKey)) {
@@ -222,6 +260,9 @@ export function AlertHistoryMobileList(props: AlertHistoryMobileListProps) {
state={props.state}
class="mt-3"
touchSized
timelineOpen={
investigation()?.kind === 'timeline' && investigation()?.rowKey === rowKey()
}
onTimelineClick={() => openTimelineInvestigation(alert, rowKey())}
onResourceClick={() => openResourceInvestigation(alert, rowKey())}
/>
@@ -236,7 +277,8 @@ export function AlertHistoryMobileList(props: AlertHistoryMobileListProps) {
<MobileAlertHistoryInvestigationDialog
investigation={selectedInvestigation()}
state={props.state}
onClose={closeInvestigation}
onClose={() => closeInvestigation()}
onAssistantHandoff={() => closeInvestigation(false)}
/>
)}
</Show>
frontend-modern/src/features/alerts/AlertResourceIncidentsPanel.tsx
+27 -3
View File
@@ -7,6 +7,8 @@ import { Card } from '@/components/shared/Card';
import type { Resource } from '@/types/resource';
import { getPreferredInfrastructureDisplayName } from '@/utils/resourceIdentity';
import {
formatIncidentEvidenceTime,
INCIDENT_HISTORY_PARTIAL,
getAlertIncidentLevelBadgeClass,
getAlertIncidentStatusPresentation,
getAlertIncidentTimelineHeadingClass,
@@ -20,6 +22,7 @@ import {
getAlertResourceIncidentEmptyState,
getAlertResourceIncidentFilteredEventsEmptyState,
getAlertResourceIncidentLoadingState,
getAlertResourceIncidentLoadFailure,
getAlertResourceIncidentPanelTitle,
getAlertResourceIncidentRefreshLabel,
getAlertResourceIncidentSummaryRowClass,
@@ -35,6 +38,7 @@ interface AlertResourceIncidentsPanelProps {
state: AlertHistoryState;
getResource?: (resourceId: string) => Resource | undefined;
onClose?: () => void;
onAssistantHandoff?: () => void;
showCloseAction?: boolean;
showTitle?: boolean;
}
@@ -46,6 +50,7 @@ export function AlertResourceIncidentsPanel(props: AlertResourceIncidentsPanelPr
const resourceId = selection().resourceId;
const incidents = () => props.state.resourceIncidents()[resourceId] || [];
const isLoading = () => props.state.resourceIncidentLoading()[resourceId];
const hasError = () => props.state.resourceIncidentError()[resourceId];
const lookupResource = () => props.getResource ?? props.state.getResource;
const resource = () => lookupResource()?.(resourceId);
const resourceDisplayName = () => {
@@ -99,6 +104,11 @@ export function AlertResourceIncidentsPanel(props: AlertResourceIncidentsPanelPr
<Show when={isLoading()}>
<p class="mt-2 text-xs text-muted">{getAlertResourceIncidentLoadingState().text}</p>
</Show>
<Show when={!isLoading() && hasError()}>
<p class="mt-2 text-xs text-error" role="alert">
{getAlertResourceIncidentLoadFailure()}. Use Refresh to try again.
</p>
</Show>
<Show when={!isLoading()}>
<Show when={incidents().length > 0}>
<div class="mt-2">
@@ -113,7 +123,11 @@ export function AlertResourceIncidentsPanel(props: AlertResourceIncidentsPanelPr
<Show
when={incidents().length > 0}
fallback={
<p class="mt-2 text-xs text-muted">{getAlertResourceIncidentEmptyState().text}</p>
<Show when={!hasError()}>
<p class="mt-2 text-xs text-muted">
{getAlertResourceIncidentEmptyState().text}
</p>
</Show>
}
>
<div class="mt-3 space-y-3">
@@ -159,15 +173,25 @@ export function AlertResourceIncidentsPanel(props: AlertResourceIncidentsPanelPr
<span class={statusPresentation.className}>
{statusPresentation.label}
</span>
<span>opened {new Date(incident.openedAt).toLocaleString()}</span>
<Show when={formatIncidentEvidenceTime(incident.openedAt)}>
<span>opened {formatIncidentEvidenceTime(incident.openedAt)}</span>
</Show>
<Show when={incident.closedAt}>
<span>
closed {new Date(incident.closedAt as string).toLocaleString()}
</span>
</Show>
</div>
<IncidentAssistantHandoffButton incident={incident} />
<IncidentAssistantHandoffButton
incident={incident}
onAssistantHandoff={props.onAssistantHandoff}
/>
</div>
<Show when={incident.history?.hasMoreChanges}>
<p class="text-xs text-muted" role="status">
{INCIDENT_HISTORY_PARTIAL}
</p>
</Show>
<Show when={incident.message}>
<p class={getAlertIncidentTimelineOutputClass()}>{incident.message}</p>
</Show>
frontend-modern/src/features/alerts/MobileAlertHistoryInvestigationDialog.tsx
+3
View File
@@ -21,6 +21,7 @@ interface MobileAlertHistoryInvestigationDialogProps {
investigation: MobileAlertHistoryInvestigation;
state: AlertHistoryState;
onClose: () => void;
onAssistantHandoff?: () => void;
}
export function MobileAlertHistoryInvestigationDialog(
@@ -66,6 +67,7 @@ export function MobileAlertHistoryInvestigationDialog(
<Switch>
<Match when={props.investigation.kind === 'timeline'}>
<IncidentTimelinePanel
onAssistantHandoff={props.onAssistantHandoff ?? props.onClose}
loading={() => props.state.incidentLoading()[rowKey()]}
error={() => props.state.incidentErrors()[rowKey()]}
timeline={() => props.state.incidentTimelines()[rowKey()]}
@@ -88,6 +90,7 @@ export function MobileAlertHistoryInvestigationDialog(
<AlertResourceIncidentsPanel
state={props.state}
onClose={props.onClose}
onAssistantHandoff={props.onAssistantHandoff ?? props.onClose}
showCloseAction={false}
showTitle={false}
/>
frontend-modern/src/features/alerts/__tests__/AlertHistoryMobileList.test.tsx
+112
View File
@@ -2,6 +2,7 @@ import { fireEvent, render, screen } from '@solidjs/testing-library';
import { createSignal } from 'solid-js';
import { describe, expect, it, vi } from 'vitest';
import { aiChatStore } from '@/stores/aiChat';
import { AlertHistoryMobileList } from '../AlertHistoryMobileList';
import type { AlertHistoryState } from '../useAlertHistoryState';
@@ -70,6 +71,7 @@ function createState() {
resourceIncidentPanel,
resourceIncidents: () => ({ 'node-1': [] }),
resourceIncidentLoading: () => ({}),
resourceIncidentError: () => ({}),
resourceIncidentEventFilters: () => new Set<string>(),
setResourceIncidentEventFilters: vi.fn(),
refreshResourceIncidentPanel: vi.fn(),
@@ -176,6 +178,74 @@ describe('AlertHistoryMobileList', () => {
});
describe('AlertHistoryMobileList changing history', () => {
it.each(['timeline', 'resource'] as const)(
'preserves an open %s across desktop and phone layouts',
async (kind) => {
const { state } = createState();
let resize = () => {};
const viewport = {
matches: true,
addEventListener: vi.fn((_event: string, listener: () => void) => {
resize = listener;
}),
removeEventListener: vi.fn(),
};
vi.stubGlobal('matchMedia', () => viewport);
if (kind === 'timeline') await state.toggleIncidentTimeline('alert-1-row', 'alert-1');
else await state.openResourceIncidentPanel('node-1', 'pve-production-01', 'alert-1-row');
const mounted = render(() => <AlertHistoryMobileList state={state} />);
const assistantOpen = vi.spyOn(aiChatStore, 'isOpen', 'get').mockReturnValue(false);
try {
expect(screen.queryByRole('dialog')).toBeNull();
expect(screen.getByRole('button', { name: 'Timeline' })).toBeVisible();
assistantOpen.mockReturnValue(true);
viewport.matches = false;
resize();
expect(screen.queryByRole('dialog')).toBeNull();
viewport.matches = true;
resize();
assistantOpen.mockReturnValue(false);
viewport.matches = false;
resize();
expect(
screen.getByRole('dialog', {
name:
kind === 'timeline'
? 'Incident timeline for pve-production-01'
: 'Resource incidents for pve-production-01',
}),
).toBeVisible();
viewport.matches = true;
resize();
expect(screen.queryByRole('dialog')).toBeNull();
expect(
kind === 'timeline'
? state.expandedIncidents().has('alert-1-row')
: state.resourceIncidentPanel()?.rowKey === 'alert-1-row',
).toBe(true);
viewport.matches = false;
resize();
expect(screen.getByRole('dialog')).toBeVisible();
await fireEvent.click(
screen.getByRole('button', {
name: kind === 'timeline' ? 'Close incident timeline' : 'Close resource incidents',
}),
);
expect(screen.queryByRole('dialog')).toBeNull();
expect(
kind === 'timeline'
? state.expandedIncidents().has('alert-1-row')
: state.resourceIncidentPanel() !== null,
).toBe(false);
} finally {
assistantOpen.mockRestore();
mounted.unmount();
vi.unstubAllGlobals();
}
expect(viewport.removeEventListener).toHaveBeenCalledWith('change', resize);
},
);
it('returns focus to the list when the investigated row disappears', async () => {
const { closeResourceIncidentPanel, setGroupedAlerts, state } = createState();
render(() => <AlertHistoryMobileList state={state} />);
@@ -202,3 +272,45 @@ describe('AlertHistoryMobileList changing history', () => {
expect(screen.getByTestId('alert-history-mobile-list')).toHaveFocus();
});
});
describe('history handoff focus ownership', () => {
it('leaves focus with Assistant after the source drawer closes', async () => {
const { state } = createState();
state.incidentTimelines = () =>
({
'alert-1-row': {
id: 'incident-focus',
alertType: 'backup',
level: 'warning',
openedAt: '2026-09-07T12:00:00Z',
resourceId: 'node-1',
resourceName: 'Backup',
status: 'resolved',
events: [],
},
}) as any;
aiChatStore.setEnabled(true);
const composer = document.createElement('textarea');
document.body.append(composer);
const open = vi.spyOn(aiChatStore, 'open').mockImplementation(() => composer.focus());
try {
render(() => <AlertHistoryMobileList state={state} />);
await fireEvent.click(screen.getByRole('button', { name: 'Timeline' }));
await fireEvent.click(
screen.getByRole('button', {
name: 'Discuss incident incident-focus with Pulse Assistant',
}),
);
await new Promise<void>((resolve) =>
requestAnimationFrame(() => requestAnimationFrame(() => resolve())),
);
expect(screen.queryByRole('dialog')).toBeNull();
expect(open).toHaveBeenCalledOnce();
expect(composer).toHaveFocus();
} finally {
open.mockRestore();
composer.remove();
aiChatStore.setEnabled(false);
}
});
});
frontend-modern/src/features/alerts/__tests__/AlertResourceIncidentsPanel.test.tsx
+24
View File
@@ -22,6 +22,27 @@ describe('AlertResourceIncidentsPanel', () => {
vi.restoreAllMocks();
});
it('shows persistent read failure without claiming an empty history', () => {
const [failed, setFailed] = createSignal(true);
render(() => (
<AlertResourceIncidentsPanel
state={
{
resourceIncidentPanel: () => ({ resourceId: 'resource-1', resourceName: 'Resource' }),
resourceIncidents: () => ({}),
resourceIncidentLoading: () => ({}),
resourceIncidentError: () => ({ 'resource-1': failed() }),
refreshResourceIncidentPanel: vi.fn(),
} as any
}
/>
));
expect(screen.getByRole('alert').textContent).toContain('Use Refresh');
expect(screen.queryByText('No incidents recorded for this resource yet.')).toBeNull();
setFailed(false);
expect(screen.queryByRole('alert')).toBeNull();
});
it('does not surface broad cross-jump links from incident details', () => {
// Surface link chips into /infrastructure and broad aggregate workspaces
// were retired with the platform-first migration.
@@ -47,6 +68,7 @@ describe('AlertResourceIncidentsPanel', () => {
},
],
}),
resourceIncidentError: () => ({}),
resourceIncidentLoading: () => ({ 'truenas-main': false }),
expandedResourceIncidentIds: () => new Set<string>(),
resourceIncidentEventFilters: () => new Set<string>(['opened']),
@@ -130,6 +152,7 @@ describe('AlertResourceIncidentsPanel', () => {
},
],
}),
resourceIncidentError: () => ({}),
resourceIncidentLoading: () => ({ 'truenas-main': false }),
expandedResourceIncidentIds: () => new Set<string>(),
resourceIncidentEventFilters: () => new Set<string>(['command']),
@@ -213,6 +236,7 @@ describe('AlertResourceIncidentsPanel', () => {
},
],
}),
resourceIncidentError: () => ({}),
resourceIncidentLoading: () => ({ 'node-1': false }),
expandedResourceIncidentIds: expandedIncidentIds,
resourceIncidentEventFilters: eventFilters,
frontend-modern/src/features/alerts/__tests__/MobileAlertHistoryInvestigationDialog.test.tsx
+85
View File
@@ -0,0 +1,85 @@
import { cleanup, fireEvent, render, screen, waitFor } from '@solidjs/testing-library';
import { Show, createSignal } from 'solid-js';
import { afterEach, describe, expect, it, vi } from 'vitest';
import { dialogStackHasBlockingDialog } from '@/components/shared/useDialogState';
import { aiChatStore } from '@/stores/aiChat';
import { MobileAlertHistoryInvestigationDialog } from '../MobileAlertHistoryInvestigationDialog';
const incident = {
id: 'incident-1',
alertIdentifier: 'alert-1',
alertType: 'backup',
level: 'warning',
resourceId: 'resource-1',
resourceName: 'Backup',
resourceType: 'storage',
status: 'resolved',
openedAt: '2026-09-07T12:00:00Z',
closedAt: '2026-09-07T12:10:00Z',
events: [],
};
describe('mobile incident Assistant transition', () => {
afterEach(() => {
cleanup();
aiChatStore.close();
aiChatStore.clearAllContext();
aiChatStore.setEnabled(false);
vi.restoreAllMocks();
});
for (const kind of ['timeline', 'resource'] as const) {
it(`dismisses the ${kind} drawer after preserving its incident context`, async () => {
aiChatStore.setEnabled(true);
const nativeOpen = aiChatStore.open;
const openAssistant = vi.spyOn(aiChatStore, 'open').mockImplementation((context) => {
expect(dialogStackHasBlockingDialog()).toBe(false);
nativeOpen(context);
});
const [open, setOpen] = createSignal(true);
const state = {
incidentLoading: () => ({}),
incidentErrors: () => ({}),
incidentTimelines: () => ({ row: incident }),
historyIncidentEventFilters: () => new Set(),
incidentNoteDrafts: () => ({}),
incidentNoteSaving: () => new Set(),
resourceIncidentPanel: () => ({
resourceId: 'resource-1',
resourceName: 'Backup',
rowKey: 'row',
}),
resourceIncidents: () => ({ 'resource-1': [incident] }),
resourceIncidentLoading: () => ({}),
resourceIncidentError: () => ({}),
expandedResourceIncidentIds: () => new Set(),
resourceIncidentEventFilters: () => new Set(),
} as any;
render(() => (
<Show when={open()}>
<MobileAlertHistoryInvestigationDialog
investigation={{ kind, rowKey: 'row', alert: { resourceName: 'Backup' } as any }}
state={state}
onClose={() => setOpen(false)}
/>
</Show>
));
expect(screen.getByRole('dialog')).toBeInTheDocument();
fireEvent.click(
screen.getByRole('button', { name: 'Discuss incident incident-1 with Pulse Assistant' }),
);
expect(screen.queryByRole('dialog')).toBeNull();
await waitFor(() => expect(openAssistant).toHaveBeenCalledTimes(1));
expect(aiChatStore.isOpen).toBe(true);
expect(openAssistant).toHaveBeenCalledWith(
expect.objectContaining({
targetId: 'resource-1',
autonomousMode: false,
context: expect.objectContaining({
alertIncidentId: 'incident-1',
alertStatus: 'resolved',
}),
}),
);
});
}
});
frontend-modern/src/features/alerts/__tests__/useAlertResourceIncidentsState.test.ts
+48
View File
@@ -0,0 +1,48 @@
import { createRoot } from 'solid-js';
import { afterEach, describe, expect, it, vi } from 'vitest';
import { AlertsAPI } from '@/api/alerts';
import type { Incident } from '@/types/api';
import { useAlertResourceIncidentsState } from '../useAlertResourceIncidentsState';
vi.mock('@/stores/notifications', () => ({ notificationStore: { error: vi.fn() } }));
vi.mock('@/utils/logger', () => ({ logger: { error: vi.fn() } }));
describe('resource incident read failures', () => {
afterEach(() => vi.restoreAllMocks());
it('keeps failed reads distinct from empty evidence, including a cached refresh', async () => {
let dispose!: () => void;
const state = createRoot((cleanup) => {
dispose = cleanup;
return useAlertResourceIncidentsState();
});
const incident = { id: 'retained-occurrence', events: [] } as unknown as Incident;
const read = vi.spyOn(AlertsAPI, 'getIncidentsForResource');
try {
read.mockRejectedValueOnce(new Error('canonical history unavailable'));
await state.openResourceIncidentPanel('resource-1', 'Resource', 'row-1');
expect(state.resourceIncidentError()['resource-1']).toBe(true);
expect(state.resourceIncidents()['resource-1']).toBeUndefined();
expect(state.resourceIncidentLoading()['resource-1']).toBe(false);
read.mockResolvedValueOnce([incident]);
await state.refreshResourceIncidentPanel();
expect(state.resourceIncidentError()['resource-1']).toBe(false);
expect(state.resourceIncidents()['resource-1']).toEqual([incident]);
read.mockRejectedValueOnce(new Error('canonical history unavailable'));
await state.refreshResourceIncidentPanel();
expect(state.resourceIncidentError()['resource-1']).toBe(true);
expect(state.resourceIncidents()['resource-1']).toEqual([incident]);
read.mockResolvedValueOnce([]);
await state.refreshResourceIncidentPanel();
expect(state.resourceIncidentError()['resource-1']).toBe(false);
expect(state.resourceIncidents()['resource-1']).toEqual([]);
state.resetResourceIncidentsState();
expect(state.resourceIncidentError()).toEqual({});
} finally {
dispose();
}
});
});
frontend-modern/src/features/alerts/useAlertHistoryState.ts
+1
View File
@@ -336,6 +336,7 @@ export function useAlertHistoryState(props: UseAlertHistoryStateProps) {
setResourceIncidentPanel: resourceIncidentsState.setResourceIncidentPanel,
resourceIncidents: resourceIncidentsState.resourceIncidents,
resourceIncidentLoading: resourceIncidentsState.resourceIncidentLoading,
resourceIncidentError: resourceIncidentsState.resourceIncidentError,
expandedResourceIncidentIds: resourceIncidentsState.expandedResourceIncidentIds,
resourceIncidentEventFilters: resourceIncidentsState.resourceIncidentEventFilters,
setResourceIncidentEventFilters: resourceIncidentsState.setResourceIncidentEventFilters,
frontend-modern/src/features/alerts/useAlertResourceIncidentsState.ts
+7
View File
@@ -21,6 +21,9 @@ export function useAlertResourceIncidentsState() {
const [resourceIncidentLoading, setResourceIncidentLoading] = createSignal<
Record<string, boolean>
>({});
const [resourceIncidentError, setResourceIncidentError] = createSignal<Record<string, boolean>>(
{},
);
const [expandedResourceIncidentIds, setExpandedResourceIncidentIds] = createSignal<Set<string>>(
new Set(),
);
@@ -32,10 +35,12 @@ export function useAlertResourceIncidentsState() {
if (!resourceId) return;
setResourceIncidentLoading((prev) => ({ ...prev, [resourceId]: true }));
setResourceIncidentError((prev) => ({ ...prev, [resourceId]: false }));
try {
const incidents = await AlertsAPI.getIncidentsForResource(resourceId, limit);
setResourceIncidents((prev) => ({ ...prev, [resourceId]: incidents }));
} catch (error) {
setResourceIncidentError((prev) => ({ ...prev, [resourceId]: true }));
logger.error(getAlertResourceIncidentLoadFailure(), error);
notificationStore.error(getAlertResourceIncidentLoadFailure());
} finally {
@@ -87,6 +92,7 @@ export function useAlertResourceIncidentsState() {
setResourceIncidentPanel(null);
setResourceIncidents({});
setResourceIncidentLoading({});
setResourceIncidentError({});
setExpandedResourceIncidentIds(new Set<string>());
setResourceIncidentEventFilters(new Set(INCIDENT_EVENT_TYPES));
};
@@ -96,6 +102,7 @@ export function useAlertResourceIncidentsState() {
setResourceIncidentPanel,
resourceIncidents,
resourceIncidentLoading,
resourceIncidentError,
expandedResourceIncidentIds,
resourceIncidentEventFilters,
setResourceIncidentEventFilters,
frontend-modern/src/hooks/__tests__/useTypeToSearch.test.ts
+56
View File
@@ -53,4 +53,60 @@ describe('type-to-search keyboard ownership', () => {
}
expect(input.value).toBe('a ');
});
it.each([false, true])(
'leaves modal keys alone when background is inert (prepared=%s)',
async (prepared) => {
const background = document.createElement('div');
background.setAttribute('inert', '');
const input = document.createElement('input');
input.value = 'selected occurrence';
background.append(input);
const modalControl = document.createElement('button');
document.body.append(background, modalControl);
let preparations = 0;
createRoot((cleanup) => {
dispose = cleanup;
useTypeToSearch({
getInput: () => input,
prepareInput: prepared
? () => {
preparations++;
}
: undefined,
clearOnEscape: true,
focusOnShortcut: true,
captureBackspace: true,
getValue: () => input.value,
onClear: () => {
input.value = '';
},
});
});
modalControl.focus();
for (const init of [
{ key: 'Escape' },
{ key: 'f', ctrlKey: true },
{ key: 'Backspace' },
{ key: 'x' },
]) {
const event = new KeyboardEvent('keydown', { ...init, bubbles: true, cancelable: true });
modalControl.dispatchEvent(event);
await Promise.resolve();
expect(event.defaultPrevented).toBe(false);
expect(input.value).toBe('selected occurrence');
expect(document.activeElement).toBe(modalControl);
}
expect(preparations).toBe(0);
background.removeAttribute('inert');
const escape = new KeyboardEvent('keydown', {
key: 'Escape',
bubbles: true,
cancelable: true,
});
modalControl.dispatchEvent(escape);
expect(escape.defaultPrevented).toBe(true);
expect(input.value).toBe('');
},
);
});
frontend-modern/src/hooks/useTypeToSearch.ts
+3
View File
@@ -52,6 +52,7 @@ const deleteBackward = (input: HTMLInputElement) => {
const isVisibleInput = (input: HTMLInputElement | undefined): input is HTMLInputElement => {
if (!input || !input.isConnected || input.disabled) return false;
if (input.hidden) return false;
if (input.closest('[inert]')) return false;
if (typeof window !== 'undefined') {
const style = window.getComputedStyle(input);
if (style.display === 'none' || style.visibility === 'hidden') {
@@ -110,6 +111,8 @@ const getActiveEntry = (
const entry = registry[index];
if (!readEnabled(entry.enabled)) continue;
const input = entry.getInput();
// Modal background ownership also excludes prepared shortcut targets.
if (input?.closest('[inert]')) continue;
const visibleInput = isVisibleInput(input) ? input : undefined;
if (!visibleInput && !(options?.allowPrepared && entry.prepareInput)) continue;
if (!predicate(entry, visibleInput)) continue;
frontend-modern/src/pages/__tests__/Alerts.helpers.test.ts
+4 -2
View File
@@ -906,7 +906,7 @@ describe('incident timeline presentation helpers', () => {
it('returns the surface event-card presentation', () => {
expect(getAlertIncidentTimelineEventCardClass('surface')).toBe(
'rounded border border-border bg-surface p-2',
'rounded border border-border bg-surface p-2 whitespace-normal break-words',
);
});
@@ -924,7 +924,9 @@ describe('incident timeline presentation helpers', () => {
'flex flex-wrap items-center gap-2 text-xs text-muted',
);
expect(getAlertIncidentTimelineHeadingClass()).toBe('font-medium text-base-content');
expect(getAlertIncidentTimelineDetailClass()).toBe('mt-1 text-xs text-base-content');
expect(getAlertIncidentTimelineDetailClass()).toBe(
'mt-1 text-xs text-base-content whitespace-pre-wrap',
);
expect(getAlertIncidentTimelineCommandClass()).toBe('mt-1 font-mono text-xs text-base-content');
expect(getAlertIncidentTimelineOutputClass()).toBe('mt-1 text-xs text-muted');
});
frontend-modern/src/types/api.ts
+13 -1
View File
@@ -1,6 +1,6 @@
// Properly typed TypeScript interfaces for Pulse API
import type { Resource, ResourceCapability, ResourcePolicy } from './resource';
import type { Resource, ResourceCapability, ResourcePolicy, ResourceChange } from './resource';
import type { EvidenceEnvelope, LifecycleTransition, OperationalRecord } from './operationalTrust';
export interface ResourceListFacetsPayload {
@@ -1423,6 +1423,8 @@ export interface AlertDeliveryDiagnosis {
}
export interface IncidentEvent {
source?: string;
evidence?: ResourceChange;
id: string;
type: string;
timestamp: string;
@@ -1430,7 +1432,17 @@ export interface IncidentEvent {
details?: Record<string, unknown>;
}
export interface IncidentHistoryCoverage {
source: string;
observedSince: string;
observedBefore: string;
changeLimit: number;
hasMoreChanges: boolean;
hasMoreIncidents: boolean;
}
export interface Incident {
history?: IncidentHistoryCoverage;
id: string;
alertIdentifier: string;
alertType: string;
frontend-modern/src/utils/__tests__/alertIncidentPresentation.branchcov0717.test.ts
+5 -3
View File
@@ -370,13 +370,13 @@ describe('event-filter class helpers — variant + selected branch coverage', ()
describe('timeline + card class helpers — branch coverage', () => {
it('getAlertIncidentTimelineEventCardClass uses bg-surface-alt for variant "alt"', () => {
expect(getAlertIncidentTimelineEventCardClass('alt')).toBe(
'rounded border border-border bg-surface-alt p-2',
'rounded border border-border bg-surface-alt p-2 whitespace-normal break-words',
);
});
it('getAlertIncidentTimelineEventCardClass uses bg-surface for variant "surface"', () => {
expect(getAlertIncidentTimelineEventCardClass('surface')).toBe(
'rounded border border-border bg-surface p-2',
'rounded border border-border bg-surface p-2 whitespace-normal break-words',
);
});
@@ -394,7 +394,9 @@ describe('timeline + card class helpers — branch coverage', () => {
'flex flex-wrap items-center gap-2 text-xs text-muted',
);
expect(getAlertIncidentTimelineHeadingClass()).toBe('font-medium text-base-content');
expect(getAlertIncidentTimelineDetailClass()).toBe('mt-1 text-xs text-base-content');
expect(getAlertIncidentTimelineDetailClass()).toBe(
'mt-1 text-xs text-base-content whitespace-pre-wrap',
);
expect(getAlertIncidentTimelineCommandClass()).toBe('mt-1 font-mono text-xs text-base-content');
expect(getAlertIncidentTimelineOutputClass()).toBe('mt-1 text-xs text-muted');
expect(getAlertResourceIncidentCardClass()).toBe('rounded border border-border bg-surface p-3');
frontend-modern/src/utils/__tests__/alertIncidentPresentation.test.ts
+5 -3
View File
@@ -177,10 +177,10 @@ describe('alertIncidentPresentation', () => {
'px-2 py-0.5 rounded bg-emerald-100 text-emerald-700 dark:bg-emerald-900 dark:text-emerald-300',
);
expect(getAlertIncidentTimelineEventCardClass('alt')).toBe(
'rounded border border-border bg-surface-alt p-2',
'rounded border border-border bg-surface-alt p-2 whitespace-normal break-words',
);
expect(getAlertIncidentTimelineEventCardClass('surface')).toBe(
'rounded border border-border bg-surface p-2',
'rounded border border-border bg-surface p-2 whitespace-normal break-words',
);
expect(getAlertIncidentNoteTextareaClass()).toBe(
'w-full rounded border border-border bg-surface p-2 text-xs text-base-content',
@@ -192,7 +192,9 @@ describe('alertIncidentPresentation', () => {
'flex flex-wrap items-center gap-2 text-xs text-muted',
);
expect(getAlertIncidentTimelineHeadingClass()).toBe('font-medium text-base-content');
expect(getAlertIncidentTimelineDetailClass()).toBe('mt-1 text-xs text-base-content');
expect(getAlertIncidentTimelineDetailClass()).toBe(
'mt-1 text-xs text-base-content whitespace-pre-wrap',
);
expect(getAlertIncidentTimelineCommandClass()).toBe('mt-1 font-mono text-xs text-base-content');
expect(getAlertIncidentTimelineOutputClass()).toBe('mt-1 text-xs text-muted');
});
frontend-modern/src/utils/alertIncidentPresentation.ts
+15 -2
View File
@@ -234,7 +234,7 @@ export function getAlertIncidentAcknowledgedBadgeClass(): string {
}
export function getAlertIncidentTimelineEventCardClass(variant: 'surface' | 'alt'): string {
return `rounded border border-border ${variant === 'alt' ? 'bg-surface-alt' : 'bg-surface'} p-2`;
return `rounded border border-border ${variant === 'alt' ? 'bg-surface-alt' : 'bg-surface'} p-2 whitespace-normal break-words`;
}
export function getAlertIncidentNoteTextareaClass(): string {
@@ -254,7 +254,7 @@ export function getAlertIncidentTimelineHeadingClass(): string {
}
export function getAlertIncidentTimelineDetailClass(): string {
return 'mt-1 text-xs text-base-content';
return 'mt-1 text-xs text-base-content whitespace-pre-wrap';
}
export function getAlertIncidentTimelineCommandClass(): string {
@@ -297,3 +297,16 @@ export function getAlertResourceIncidentTruncatedEventsLabel(
}
return `Showing last ${count} events`;
}
// A zero Go timestamp means the retained evidence does not establish a time.
export function formatIncidentEvidenceTime(value?: string | null): string | null {
if (!value || value.startsWith('0001-')) return null;
const date = new Date(value);
return Number.isNaN(date.getTime()) ? null : date.toLocaleString();
}
export const INCIDENT_HISTORY_PARTIAL =
'Only part of this history was returned. Earlier occurrences or events may be missing.';
export const INCIDENT_HISTORY_LEGACY = 'This timeline uses saved incident notes and snapshots.';
export const INCIDENT_EVIDENCE_DETAILS = 'Evidence details';
export const INCIDENT_TIME_UNAVAILABLE = 'Time unavailable';
internal/ai/cost/coverage_test.go
+19
View File
@@ -2,6 +2,7 @@ package cost
import (
"errors"
"math"
"testing"
"time"
)
@@ -444,3 +445,21 @@ func TestSummarizeTargets_SortTiebreakers(t *testing.T) {
t.Fatalf("expected target ID ordering, got %+v", rollup)
}
}
func TestReviewedAstraRouteBudgetBoundary(t *testing.T) {
for _, tc := range []struct {
input int64
want float64
}{
{271999, 2.76999},
{272000, 5.515},
} {
usd, known, price := EstimateUSD("openrouter", "openai/gpt-6-astra", tc.input, 1000)
if !known || math.Abs(usd-tc.want) > 0.000001 || price.AsOf != "2026-09-07" {
t.Fatalf("input=%d: usd=%f known=%v price=%+v", tc.input, usd, known, price)
}
}
if _, known, _ := EstimateUSD("openrouter", "openai/gpt-6-astra:free", 1000, 1000); known {
t.Fatal("unreviewed alias must not inherit funded route prices")
}
}
internal/ai/cost/pricing.go
+7
View File
@@ -86,6 +86,13 @@ var providerPrices = map[string][]modelPrice{
// published standard price changes on 2027-01-01. Batch/alias routes
// are deliberately not covered by this exact model ID.
flatPriceAsOf("google/gemini-3.8-flash", 0.75, 3.75, "2026-09-06"),
// Exact funded route reviewed from the Models API on 2026-09-07.
// Its override starts at min_prompt_tokens=272000. Cache discounts
// are omitted so budget estimates remain conservative.
{Pattern: "openai/gpt-6-astra", AsOf: "2026-09-07", Tiers: []priceTier{
{MaxInputTokens: 271999, InputUSDPerMTok: 10, OutputUSDPerMTok: 50},
{InputUSDPerMTok: 20, OutputUSDPerMTok: 75},
}},
flatPriceAsOf("nvidia/nemotron-3.5-lightning:free", 0, 0, "2026-08-14"),
flatPriceAsOf("nvidia/nemotron-3-super-120b-a12b:free", 0, 0, "2026-08-14"),
flatPriceAsOf("nvidia/nemotron-3-ultra-550b-a55b:free", 0, 0, "2026-08-15"),
internal/ai/intelligence.go
+24 -16
View File
@@ -57,21 +57,23 @@ type HealthScore struct {
// ResourceIntelligence aggregates all AI knowledge about a single resource
type ResourceIntelligence struct {
ResourceID string `json:"resource_id"`
ResourceName string `json:"resource_name,omitempty"`
ResourceType string `json:"resource_type,omitempty"`
Health HealthScore `json:"health"`
ActiveFindings []*Finding `json:"active_findings,omitempty"`
Predictions []patterns.FailurePrediction `json:"predictions,omitempty"`
Dependencies []string `json:"dependencies,omitempty"` // Resources this depends on
Dependents []string `json:"dependents,omitempty"` // Resources that depend on this
Correlations []*correlation.Correlation `json:"correlations,omitempty"`
Baselines map[string]*baseline.FlatBaseline `json:"baselines,omitempty"`
Anomalies []AnomalyReport `json:"anomalies,omitempty"`
RecentIncidents []*memory.Incident `json:"recent_incidents,omitempty"`
RecentChanges []unifiedresources.ResourceChange `json:"recent_changes,omitempty"`
Knowledge *knowledge.GuestKnowledge `json:"knowledge,omitempty"`
NoteCount int `json:"note_count"`
IncidentHistory *memory.IncidentHistoryCoverage `json:"incident_history,omitempty"`
IncidentHistoryError string `json:"incident_history_error,omitempty"`
ResourceID string `json:"resource_id"`
ResourceName string `json:"resource_name,omitempty"`
ResourceType string `json:"resource_type,omitempty"`
Health HealthScore `json:"health"`
ActiveFindings []*Finding `json:"active_findings,omitempty"`
Predictions []patterns.FailurePrediction `json:"predictions,omitempty"`
Dependencies []string `json:"dependencies,omitempty"` // Resources this depends on
Dependents []string `json:"dependents,omitempty"` // Resources that depend on this
Correlations []*correlation.Correlation `json:"correlations,omitempty"`
Baselines map[string]*baseline.FlatBaseline `json:"baselines,omitempty"`
Anomalies []AnomalyReport `json:"anomalies,omitempty"`
RecentIncidents []*memory.Incident `json:"recent_incidents,omitempty"`
RecentChanges []unifiedresources.ResourceChange `json:"recent_changes,omitempty"`
Knowledge *knowledge.GuestKnowledge `json:"knowledge,omitempty"`
NoteCount int `json:"note_count"`
}
// AnomalyReport describes a metric that's deviating from baseline
@@ -347,7 +349,13 @@ func (i *Intelligence) GetResourceIntelligence(resourceID string) *ResourceIntel
// Recent incidents
if i.incidents != nil {
intel.RecentIncidents = i.incidents.ListIncidentsByResource(resourceID, 5)
page, err := i.incidents.QueryIncidents(memory.IncidentQuery{ResourceID: resourceID, Limit: 5})
if err != nil {
intel.IncidentHistoryError = "Canonical incident history unavailable"
} else {
intel.RecentIncidents = page.Incidents
intel.IncidentHistory = &page.History
}
}
// Recent changes
internal/ai/memory/incident_query.go
+388
View File
@@ -0,0 +1,388 @@
package memory
import (
"crypto/sha256"
"fmt"
"sort"
"strings"
"time"
"github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources"
)
// IncidentQuery selects historical evidence, not current resource health.
// Limits bound returned evidence rather than deciding diagnostic quality.
type IncidentQuery struct {
IncidentID string
ResourceID string
AlertIdentifier string
StartedAt time.Time
Limit int
ChangeLimit int
}
type IncidentHistoryCoverage struct {
Source string `json:"source"`
ObservedSince time.Time `json:"observedSince"`
ObservedBefore time.Time `json:"observedBefore"`
ChangeLimit int `json:"changeLimit"`
HasMoreChanges bool `json:"hasMoreChanges"`
HasMoreIncidents bool `json:"hasMoreIncidents"`
}
type IncidentPage struct {
Incidents []*Incident `json:"incidents"`
History IncidentHistoryCoverage `json:"history"`
}
// QueryIncidents reads canonical evidence before selecting incident rows. Shells
// contribute occurrence identity and attributed local notes, never a second
// authoritative lifecycle. A failed canonical read is not an empty history.
func (s *IncidentStore) QueryIncidents(query IncidentQuery) (IncidentPage, error) {
query.ResourceID = strings.TrimSpace(query.ResourceID)
query.AlertIdentifier = strings.TrimSpace(query.AlertIdentifier)
if query.Limit <= 0 {
query.Limit = 20
}
if query.Limit > 100 {
return IncidentPage{}, fmt.Errorf("incident limit exceeds 100")
}
if query.ChangeLimit <= 0 {
query.ChangeLimit = 4096
}
if query.ChangeLimit > 8192 {
return IncidentPage{}, fmt.Errorf("incident change limit exceeds 8192")
}
s.mu.RLock()
store, maxAge := s.resourceTimelineStore, s.maxAge
shells := make([]*incidentShell, 0, len(s.incidents))
for _, shell := range s.incidents {
if shell != nil {
shells = append(shells, cloneIncidentShell(shell))
}
}
s.mu.RUnlock()
before := time.Now().UTC()
page := IncidentPage{Incidents: []*Incident{}, History: IncidentHistoryCoverage{Source: "legacy_incident_memory", ChangeLimit: query.ChangeLimit}}
var changes []unifiedresources.ResourceChange
resourceIDs := map[string]bool{query.ResourceID: true}
if store != nil {
page.History.Source = "canonical_resource_history"
page.History.ObservedSince = before.Add(-maxAge)
page.History.ObservedBefore = before
filters := unifiedresources.ResourceChangeFilters{ObservedBefore: &before, Kinds: []unifiedresources.ChangeKind{
unifiedresources.ChangeAlertFired, unifiedresources.ChangeAlertResolved,
unifiedresources.ChangeAlertAcknowledged, unifiedresources.ChangeAlertUnacknowledged,
unifiedresources.ChangeAlertSnoozed, unifiedresources.ChangeAlertUnsnoozed,
unifiedresources.ChangeCommandExecuted, unifiedresources.ChangeRunbookExecuted,
}}
if query.AlertIdentifier != "" {
filters.AlertIdentifiers = []string{query.AlertIdentifier}
}
var err error
changes, err = store.GetRecentChangesFiltered(query.ResourceID, page.History.ObservedSince, query.ChangeLimit+1, filters)
if err != nil {
return IncidentPage{}, fmt.Errorf("read canonical incident history: %w", err)
}
if len(changes) > query.ChangeLimit {
page.History.HasMoreChanges = true
changes = changes[:query.ChangeLimit]
}
if query.ResourceID != "" {
ids, err := store.ResourceHistoryIDs(query.ResourceID)
if err != nil {
return IncidentPage{}, fmt.Errorf("read incident history identities: %w", err)
}
for _, id := range ids {
resourceIDs[id] = true
}
}
}
// Resource aliases are read identities. A shell for another resource must
// not become an occurrence boundary merely because its alert ID matches.
resourceKeys := make(map[string]string)
identityIDs := make([]string, 0, len(shells)+len(changes))
for _, shell := range shells {
identityIDs = append(identityIDs, shell.ResourceID)
}
for _, change := range changes {
identityIDs = append(identityIDs, change.ResourceID)
}
for _, id := range identityIDs {
if _, known := resourceKeys[id]; known {
continue
}
ids := []string{id}
if store != nil && id != "" {
aliases, err := store.ResourceHistoryIDs(id)
if err != nil {
return IncidentPage{}, fmt.Errorf("read incident occurrence identities: %w", err)
}
ids = append(ids, aliases...)
}
sort.Strings(ids)
for _, alias := range ids {
resourceKeys[alias] = ids[0]
}
}
sameResource := func(a, b string) bool { return resourceKeys[a] == resourceKeys[b] }
byAlert := make(map[string][]*Incident)
byOccurrence := make(map[string]*Incident)
occurrenceIDs := make(map[*Incident]map[string]bool)
// Historical imports can leave several shells for the same explicit firing.
// Keep a stable existing ID and all local notes, not competing lifecycles.
sort.Slice(shells, func(i, j int) bool { return shells[i].ID < shells[j].ID })
for _, shell := range shells {
if query.ResourceID != "" && !resourceIDs[shell.ResourceID] {
continue
}
if query.AlertIdentifier != "" && shell.AlertIdentifier != query.AlertIdentifier {
continue
}
incident := incidentFromShell(shell)
for i := range incident.Events {
if incident.Events[i].Source == "" {
incident.Events[i].Source = "legacy_incident_memory"
}
}
key := incident.AlertIdentifier + "\x00" + resourceKeys[incident.ResourceID] + "\x00" + incident.OpenedAt.UTC().Format(time.RFC3339Nano)
if incident.OpenedAt.IsZero() {
key += "\x00" + incident.ID
}
if existing := byOccurrence[key]; existing != nil {
occurrenceIDs[existing][incident.ID] = true
for _, fields := range [][2]*string{{&existing.ResourceID, &incident.ResourceID}, {&existing.ResourceName, &incident.ResourceName}, {&existing.ResourceType, &incident.ResourceType}, {&existing.AlertType, &incident.AlertType}, {&existing.Level, &incident.Level}, {&existing.Message, &incident.Message}, {&existing.Node, &incident.Node}, {&existing.Instance, &incident.Instance}} {
if *fields[0] == "" {
*fields[0] = *fields[1]
}
}
seen := make(map[string]bool, len(existing.Events))
for _, event := range existing.Events {
seen[event.ID] = true
}
for _, event := range incident.Events {
if !seen[event.ID] {
existing.Events = append(existing.Events, event)
seen[event.ID] = true
}
}
if existing.ClosedAt == nil && incident.ClosedAt != nil {
existing.ClosedAt = incident.ClosedAt
existing.Status = incident.Status
}
continue
}
byOccurrence[key] = incident
occurrenceIDs[incident] = map[string]bool{incident.ID: true}
byAlert[incident.AlertIdentifier] = append(byAlert[incident.AlertIdentifier], incident)
}
// Explicit fired events establish occurrence boundaries even if no shell was
// saved. Sort by occurrence time, retaining the distinct observation time.
sort.Slice(changes, func(i, j int) bool {
a, b := incidentEventTimestamp(changes[i]), incidentEventTimestamp(changes[j])
if a.Equal(b) {
if changes[i].Kind == unifiedresources.ChangeAlertFired && changes[j].Kind != unifiedresources.ChangeAlertFired {
return true
}
if changes[j].Kind == unifiedresources.ChangeAlertFired && changes[i].Kind != unifiedresources.ChangeAlertFired {
return false
}
return changes[i].ID < changes[j].ID
}
return a.Before(b)
})
boundaries := make(map[*Incident]time.Time)
firedOwners := make(map[string]*Incident)
canonicalBoundary := make(map[*Incident]bool)
for _, occurrences := range byAlert {
for _, occurrence := range occurrences {
boundaries[occurrence] = occurrence.OpenedAt
}
}
for _, change := range changes {
identifier := projectedAlertIdentifier(change)
if identifier == "" || change.Kind != unifiedresources.ChangeAlertFired {
continue
}
started := incidentEventTimestamp(change)
var occurrence *Incident
for _, candidate := range byAlert[identifier] {
if canonicalBoundary[candidate] {
if sameResource(candidate.ResourceID, change.ResourceID) && boundaries[candidate].Equal(started) {
occurrence = candidate
break
}
continue
}
if incidentStartsMatch(candidate.OpenedAt, started) && (occurrence == nil || incidentStartDelta(candidate.OpenedAt, started) < incidentStartDelta(occurrence.OpenedAt, started)) {
occurrence = candidate
}
}
if occurrence == nil {
occurrence = &Incident{ID: canonicalIncidentID(identifier, started), AlertIdentifier: identifier, OpenedAt: started, Status: IncidentStatusUnknown}
byAlert[identifier] = append(byAlert[identifier], occurrence)
}
hydrateIncidentFromCanonicalChange(occurrence, change)
boundaries[occurrence] = started
firedOwners[change.ID] = occurrence
canonicalBoundary[occurrence] = true
}
for _, occurrences := range byAlert {
sort.Slice(occurrences, func(i, j int) bool { return boundaries[occurrences[i]].Before(boundaries[occurrences[j]]) })
}
canonicalEvents := make(map[*Incident][]IncidentEvent)
for _, change := range changes {
identifier := projectedAlertIdentifier(change)
if identifier == "" {
continue
}
event, ok := incidentEventFromResourceChange(change)
if !ok {
continue
}
occurrence := firedOwners[change.ID]
if occurrence == nil {
for _, candidate := range byAlert[identifier] {
if change.Kind != unifiedresources.ChangeCommandExecuted && change.Kind != unifiedresources.ChangeRunbookExecuted && !sameResource(candidate.ResourceID, change.ResourceID) {
continue
}
// A capped read may have omitted a later firing after a saved shell.
// Never attach those ambiguous events to the earlier occurrence.
if page.History.HasMoreChanges && !canonicalBoundary[candidate] && !candidate.OpenedAt.IsZero() {
continue
}
if boundaries[candidate].After(event.Timestamp) {
break
}
occurrence = candidate
}
}
if occurrence == nil {
// An event without a retained start cannot establish when the alert
// opened. Preserve it as incomplete evidence with an unknown lifecycle.
for _, candidate := range byAlert[identifier] {
if candidate.OpenedAt.IsZero() && sameResource(candidate.ResourceID, change.ResourceID) {
occurrence = candidate
break
}
}
if occurrence == nil {
occurrence = &Incident{ID: "projected-partial-" + change.ID, AlertIdentifier: identifier, Status: IncidentStatusUnknown}
byAlert[identifier] = append([]*Incident{occurrence}, byAlert[identifier]...)
}
}
openedAt := occurrence.OpenedAt
hydrateIncidentFromCanonicalChange(occurrence, change)
occurrence.OpenedAt = openedAt
canonicalEvents[occurrence] = append(canonicalEvents[occurrence], event)
}
for _, occurrences := range byAlert {
for _, occurrence := range occurrences {
events := canonicalEvents[occurrence]
if len(events) > 0 {
kept := make([]IncidentEvent, 0, len(occurrence.Events)+len(events))
for _, event := range occurrence.Events {
if isCanonicalProjectedIncidentEventType(event.Type) && (!isSnapshotProjectionEvent(event) || hasIncidentEventType(events, event.Type)) {
continue
}
kept = append(kept, event)
}
occurrence.Events = append(kept, events...)
sortIncidentEvents(occurrence.Events)
openedAt := occurrence.OpenedAt
resetDerivedIncidentState(occurrence)
occurrence.Status = IncidentStatusUnknown
applyProjectedIncidentState(occurrence, occurrence.Events)
occurrence.OpenedAt = openedAt
}
if query.IncidentID != "" && occurrence.ID != query.IncidentID && !occurrenceIDs[occurrence][query.IncidentID] {
continue
}
if !query.StartedAt.IsZero() && !incidentStartsMatch(occurrence.OpenedAt, query.StartedAt) {
continue
}
coverage := page.History
if len(events) == 0 {
coverage.Source = "legacy_incident_memory"
}
occurrence.History = &coverage
page.Incidents = append(page.Incidents, occurrence)
}
}
if !query.StartedAt.IsZero() && len(page.Incidents) > 1 {
closest := page.Incidents[0]
for _, incident := range page.Incidents[1:] {
if incidentStartDelta(incident.OpenedAt, query.StartedAt) < incidentStartDelta(closest.OpenedAt, query.StartedAt) {
closest = incident
}
}
page.Incidents = []*Incident{closest}
}
sort.Slice(page.Incidents, func(i, j int) bool {
if page.Incidents[i].OpenedAt.Equal(page.Incidents[j].OpenedAt) {
return page.Incidents[i].ID > page.Incidents[j].ID
}
return page.Incidents[i].OpenedAt.After(page.Incidents[j].OpenedAt)
})
if len(page.Incidents) > query.Limit {
page.History.HasMoreIncidents = true
page.Incidents = page.Incidents[:query.Limit]
}
for _, incident := range page.Incidents {
incident.History.HasMoreIncidents = page.History.HasMoreIncidents
}
return page, nil
}
func incidentStartsMatch(a, b time.Time) bool {
if a.IsZero() || b.IsZero() {
return false
}
delta := a.Sub(b)
return delta >= -incidentStartMatchTolerance && delta <= incidentStartMatchTolerance
}
func formatIncidentCoverage(page IncidentPage) string {
if len(page.Incidents) == 0 && page.History.Source == "legacy_incident_memory" {
return ""
}
if page.History.Source == "legacy_incident_memory" {
return "Saved incident memory only. Canonical history coverage was not read. This does not establish current health.\n"
}
text := fmt.Sprintf("Historical incident evidence: %s, observations [%s, %s). Retained history only. This does not establish current health.\n", page.History.Source, page.History.ObservedSince.Format(time.RFC3339), page.History.ObservedBefore.Format(time.RFC3339))
if page.History.HasMoreChanges || page.History.HasMoreIncidents {
text += "The query is truncated. Earlier occurrences or events may be absent.\n"
}
if len(page.Incidents) == 0 {
text += "No incident evidence was returned in this window.\n"
}
return text
}
func formatIncidentTime(value time.Time) string {
if value.IsZero() {
return "unknown"
}
return value.Format(time.RFC3339)
}
func formatIncidentOccurredTime(value *time.Time) string {
if value == nil {
return "unknown"
}
return formatIncidentTime(*value)
}
// The alert identifier and explicit firing time name the occurrence. Event
// arrival order, duplicated observations and resource aliases do not rename it.
func canonicalIncidentID(alertIdentifier string, startedAt time.Time) string {
return fmt.Sprintf("projected-%x", sha256.Sum256([]byte(alertIdentifier+"\x00"+startedAt.UTC().Format(time.RFC3339Nano))))
}
func incidentStartDelta(a, b time.Time) time.Duration {
d := a.Sub(b)
if d < 0 {
return -d
}
return d
}
internal/ai/memory/incident_query_test.go
+234
View File
@@ -0,0 +1,234 @@
package memory
import (
"errors"
"testing"
"time"
"github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources"
"github.com/stretchr/testify/require"
)
func TestIncidentQueryCanonicalOccurrencesAndNotes(t *testing.T) {
for _, backend := range []string{"memory", "sqlite"} {
t.Run(backend, func(t *testing.T) {
var canonical unifiedresources.ResourceStore
if backend == "memory" {
canonical = unifiedresources.NewMemoryStore()
} else {
store, err := unifiedresources.NewSQLiteResourceStore(t.TempDir(), "default")
require.NoError(t, err)
t.Cleanup(func() { require.NoError(t, store.Close()) })
canonical = store
}
config := IncidentStoreConfig{DataDir: t.TempDir()}
memory := NewIncidentStore(config)
memory.SetResourceTimelineStore(canonical)
start := time.Now().UTC().Add(-time.Hour).Truncate(time.Second)
second := start.Add(2 * time.Minute)
record := func(id string, kind unifiedresources.ChangeKind, at time.Time) {
observed := at.Add(10 * time.Second)
require.NoError(t, canonical.RecordChange(unifiedresources.ResourceChange{ID: id, ResourceID: "resource-canonical", Kind: kind, ObservedAt: observed, OccurredAt: &at, SourceType: unifiedresources.SourcePlatformEvent, SourceAdapter: unifiedresources.AdapterProxmox, Actor: "operator", Confidence: unifiedresources.ConfidenceHigh, Metadata: map[string]any{unifiedresources.MetadataAlertIdentifier: "repeat-alert", "nested": map[string]any{"risk": "storage"}}}))
}
record("first-fired", unifiedresources.ChangeAlertFired, start)
record("first-resolved", unifiedresources.ChangeAlertResolved, start.Add(time.Minute))
record("second-fired", unifiedresources.ChangeAlertFired, second)
record("second-ack", unifiedresources.ChangeAlertAcknowledged, second.Add(time.Minute))
page, err := memory.QueryIncidents(IncidentQuery{ResourceID: "resource-canonical", Limit: 10})
require.NoError(t, err)
require.Len(t, page.Incidents, 2)
require.NotEmpty(t, page.Incidents[0].ID)
require.Equal(t, IncidentStatusOpen, page.Incidents[0].Status)
require.True(t, page.Incidents[0].Acknowledged)
first := page.Incidents[1]
require.NotEqual(t, page.Incidents[0].ID, first.ID)
require.Equal(t, IncidentStatusResolved, first.Status)
require.False(t, first.Acknowledged)
require.Len(t, first.Events, 2)
require.Equal(t, start.Add(time.Minute), *first.ClosedAt)
require.Equal(t, start, first.OpenedAt)
require.Equal(t, "canonical_resource_history", first.Events[0].Source)
require.Equal(t, start.Add(10*time.Second), first.Events[0].Evidence.ObservedAt)
require.Equal(t, start, *first.Events[0].Evidence.OccurredAt)
require.Equal(t, "operator", first.Events[0].Evidence.Actor)
first.Events[0].Evidence.Metadata["nested"].(map[string]any)["risk"] = "mutated"
again, err := memory.QueryIncidents(IncidentQuery{AlertIdentifier: "repeat-alert", StartedAt: start, Limit: 1})
require.NoError(t, err)
require.Len(t, again.Incidents, 1)
require.Equal(t, "storage", again.Incidents[0].Events[0].Evidence.Metadata["nested"].(map[string]any)["risk"])
require.True(t, memory.RecordNote("repeat-alert", first.ID, "Check the old pool before replacing it", "operator"))
noted, err := memory.QueryIncidents(IncidentQuery{AlertIdentifier: "repeat-alert", StartedAt: start, Limit: 1})
require.NoError(t, err)
require.Len(t, noted.Incidents, 1)
require.Len(t, noted.Incidents[0].Events, 3)
require.Equal(t, IncidentStatusResolved, noted.Incidents[0].Status)
require.Equal(t, IncidentEventNote, noted.Incidents[0].Events[2].Type)
require.Equal(t, "operator_note", noted.Incidents[0].Events[2].Source)
require.NoError(t, memory.saveToDisk())
restored := NewIncidentStore(config)
restored.SetResourceTimelineStore(canonical)
persisted, err := restored.QueryIncidents(IncidentQuery{AlertIdentifier: "repeat-alert", StartedAt: start, Limit: 1})
require.NoError(t, err)
require.Len(t, persisted.Incidents, 1)
require.Equal(t, first.ID, persisted.Incidents[0].ID)
require.Equal(t, "Check the old pool before replacing it", persisted.Incidents[0].Events[2].Details["note"])
latest, err := memory.QueryIncidents(IncidentQuery{AlertIdentifier: "repeat-alert", Limit: 1})
require.NoError(t, err)
require.True(t, latest.History.HasMoreIncidents)
require.Len(t, latest.Incidents[0].Events, 2) // old note remains attached to old occurrence
formatted := memory.FormatForAlert("repeat-alert", 1)
require.Contains(t, formatted, "Showing latest 1 of 2 recorded events")
require.Contains(t, formatted, "Occurrence evidence source: canonical_resource_history")
bounded, err := memory.QueryIncidents(IncidentQuery{AlertIdentifier: "repeat-alert", ChangeLimit: 1})
require.NoError(t, err)
require.True(t, bounded.History.HasMoreChanges)
// The saved first occurrence remains separate from incomplete later evidence.
require.Contains(t, formatIncidentCoverage(bounded), "truncated")
})
}
}
type failingIncidentHistory struct{ *unifiedresources.MemoryStore }
func (s failingIncidentHistory) GetRecentChangesFiltered(string, time.Time, int, unifiedresources.ResourceChangeFilters) ([]unifiedresources.ResourceChange, error) {
return nil, errors.New("history read denied")
}
func TestIncidentQueryUnavailableAndMissingStart(t *testing.T) {
memory := NewIncidentStore(IncidentStoreConfig{})
memory.SetResourceTimelineStore(failingIncidentHistory{unifiedresources.NewMemoryStore()})
_, err := memory.QueryIncidents(IncidentQuery{ResourceID: "resource-a"})
require.ErrorContains(t, err, "history read denied")
require.Contains(t, memory.FormatForResource("resource-a", 5), "unavailable")
require.Contains(t, memory.FormatForAlert("alert", 5), "unavailable")
require.Contains(t, memory.FormatForPatrol(5), "unavailable")
canonical := unifiedresources.NewMemoryStore()
memory.SetResourceTimelineStore(canonical)
now := time.Now().UTC().Add(-time.Minute)
require.NoError(t, canonical.RecordChange(unifiedresources.ResourceChange{ID: "orphan-ack", ResourceID: "resource-a", ObservedAt: now, Kind: unifiedresources.ChangeAlertAcknowledged, Metadata: map[string]any{unifiedresources.MetadataAlertIdentifier: "alert"}}))
page, err := memory.QueryIncidents(IncidentQuery{ResourceID: "resource-a"})
require.NoError(t, err)
require.Len(t, page.Incidents, 1)
require.Equal(t, IncidentStatus("unknown"), page.Incidents[0].Status)
require.True(t, page.Incidents[0].OpenedAt.IsZero())
require.True(t, incidentEventTimestamp(unifiedresources.ResourceChange{}).IsZero())
}
func TestIncidentQueryDoesNotMergeRapidOccurrences(t *testing.T) {
canonical := unifiedresources.NewMemoryStore()
store := NewIncidentStore(IncidentStoreConfig{})
store.SetResourceTimelineStore(canonical)
first := time.Now().UTC().Add(-time.Hour).Truncate(time.Second)
for _, at := range []time.Time{first, first.Add(500 * time.Millisecond)} {
change := unifiedresources.BuildAlertTimelineChange("resource-a", unifiedresources.ChangeAlertFired, at, "", unifiedresources.AlertTimelineChange{AlertIdentifier: "rapid"})
require.NoError(t, canonical.RecordChange(*change))
}
page, err := store.QueryIncidents(IncidentQuery{AlertIdentifier: "rapid"})
require.NoError(t, err)
require.Len(t, page.Incidents, 2)
exact, err := store.QueryIncidents(IncidentQuery{AlertIdentifier: "rapid", StartedAt: first, Limit: 1})
require.NoError(t, err)
require.Len(t, exact.Incidents, 1)
require.Equal(t, first, exact.Incidents[0].OpenedAt)
stableID := exact.Incidents[0].ID
// An independently recorded copy of the same firing is still one occurrence.
duplicate := unifiedresources.BuildAlertTimelineChange("resource-a", unifiedresources.ChangeAlertFired, first, "", unifiedresources.AlertTimelineChange{AlertIdentifier: "rapid"})
require.NoError(t, canonical.RecordChange(*duplicate))
exact, err = store.QueryIncidents(IncidentQuery{AlertIdentifier: "rapid", StartedAt: first, Limit: 1})
require.NoError(t, err)
require.Len(t, exact.Incidents, 1)
require.Equal(t, stableID, exact.Incidents[0].ID)
require.Len(t, exact.Incidents[0].Events, 2)
}
func TestIncidentQueryMergesDuplicateSavedOccurrencesWithoutLosingNotes(t *testing.T) {
canonical := unifiedresources.NewMemoryStore()
store := NewIncidentStore(IncidentStoreConfig{})
store.SetResourceTimelineStore(canonical)
start := time.Now().UTC().Add(-time.Hour)
closed := start.Add(time.Minute)
// Reproduce independently imported legacy shells carrying the same firing.
store.incidents = []*incidentShell{
{ID: "original", AlertIdentifier: "duplicated", ResourceID: "resource-a", OpenedAt: start, Events: []IncidentEvent{{ID: "note-a", Type: IncidentEventNote, Timestamp: start, Details: map[string]any{"note": "first note"}}}},
{ID: "imported", AlertIdentifier: "duplicated", ResourceID: "resource-a", OpenedAt: start, OccurrenceClosedAt: &closed, Events: []IncidentEvent{{ID: "note-b", Type: IncidentEventNote, Timestamp: closed, Details: map[string]any{"note": "second note"}}}},
}
for _, kind := range []unifiedresources.ChangeKind{unifiedresources.ChangeAlertFired, unifiedresources.ChangeAlertResolved} {
at := start
if kind == unifiedresources.ChangeAlertResolved {
at = closed
}
change := unifiedresources.BuildAlertTimelineChange("resource-a", kind, at, "", unifiedresources.AlertTimelineChange{AlertIdentifier: "duplicated"})
require.NoError(t, canonical.RecordChange(*change))
}
page, err := store.QueryIncidents(IncidentQuery{AlertIdentifier: "duplicated", StartedAt: start})
require.NoError(t, err)
require.Len(t, page.Incidents, 1)
require.Equal(t, IncidentStatusResolved, page.Incidents[0].Status)
require.Len(t, page.Incidents[0].Events, 4)
for _, id := range []string{"original", "imported"} {
require.True(t, store.RecordNote("duplicated", id, "after merge "+id, "operator"))
aliased, err := store.QueryIncidents(IncidentQuery{IncidentID: id, AlertIdentifier: "duplicated"})
require.NoError(t, err)
require.Len(t, aliased.Incidents, 1)
require.Equal(t, page.Incidents[0].ID, aliased.Incidents[0].ID)
}
final, err := store.QueryIncidents(IncidentQuery{AlertIdentifier: "duplicated"})
require.NoError(t, err)
require.Len(t, final.Incidents[0].Events, 6)
}
func TestIncidentHistorySummaryPreservesLegacyAttribution(t *testing.T) {
page := IncidentPage{History: IncidentHistoryCoverage{Source: "canonical_resource_history"}, Incidents: []*Incident{{ID: "legacy", AlertType: "backup", Status: IncidentStatusUnknown, History: &IncidentHistoryCoverage{Source: "legacy_incident_memory"}}}}
require.Contains(t, FormatIncidentPageForResource(page), "[source=legacy_incident_memory]")
}
func TestIncidentQueryCanonicalAlertContextOwnsLegacyShell(t *testing.T) {
canonical := unifiedresources.NewMemoryStore()
store := NewIncidentStore(IncidentStoreConfig{})
store.SetResourceTimelineStore(canonical)
start := time.Now().UTC().Add(-time.Hour)
store.incidents = []*incidentShell{{
ID: "saved-id", AlertIdentifier: "alert-context", ResourceID: "docker:legacy-id",
ResourceType: "host", AlertType: "old-type", Level: "warning", Message: "old message", OpenedAt: start,
}}
require.NoError(t, canonical.RecordChange(unifiedresources.ResourceChange{
ID: "canonical-fired", ResourceID: "app-container-current", Kind: unifiedresources.ChangeAlertFired,
ObservedAt: start, OccurredAt: &start, Metadata: map[string]any{
unifiedresources.MetadataAlertIdentifier: "alert-context", unifiedresources.MetadataAlertType: "docker-container-health",
unifiedresources.MetadataAlertLevel: "critical", unifiedresources.MetadataAlertMessage: "Container health is unhealthy",
"resourceType": "app-container",
},
}))
require.NoError(t, canonical.RecordChange(unifiedresources.ResourceChange{
ID: "related-command", ResourceID: "node-executor", Kind: unifiedresources.ChangeCommandExecuted,
ObservedAt: start.Add(time.Minute), Metadata: map[string]any{unifiedresources.MetadataAlertIdentifier: "alert-context"},
}))
page, err := store.QueryIncidents(IncidentQuery{AlertIdentifier: "alert-context"})
require.NoError(t, err)
require.Len(t, page.Incidents, 1)
incident := page.Incidents[0]
require.Equal(t, "saved-id", incident.ID)
require.Equal(t, "app-container-current", incident.ResourceID)
require.Equal(t, "app-container", incident.ResourceType)
require.Equal(t, "docker-container-health", incident.AlertType)
require.Equal(t, "critical", incident.Level)
require.Equal(t, "Container health is unhealthy", incident.Message)
require.Equal(t, "docker:legacy-id", store.incidents[0].ResourceID)
}
func TestIncidentAlertContextRetainsOperatorNote(t *testing.T) {
canonical := unifiedresources.NewMemoryStore()
store := NewIncidentStore(IncidentStoreConfig{DataDir: t.TempDir()})
store.SetResourceTimelineStore(canonical)
started := time.Now().UTC().Add(-time.Hour)
change := unifiedresources.BuildAlertTimelineChange("resource-a", unifiedresources.ChangeAlertFired, started, "", unifiedresources.AlertTimelineChange{AlertIdentifier: "noted-alert"})
require.NoError(t, canonical.RecordChange(*change))
page, err := store.QueryIncidents(IncidentQuery{AlertIdentifier: "noted-alert"})
require.NoError(t, err)
require.Len(t, page.Incidents, 1)
require.True(t, store.RecordNote("noted-alert", page.Incidents[0].ID, "Keep the old pool until its replacement is verified", "operator"))
context := store.FormatForAlert("noted-alert", 10)
require.Contains(t, context, "Note added by operator: Keep the old pool until its replacement is verified")
require.Contains(t, context, "source=operator_note")
}
internal/ai/memory/incidents.go
+225 -313
View File
@@ -22,6 +22,7 @@ type IncidentStatus string
const (
IncidentStatusOpen IncidentStatus = "open"
IncidentStatusUnknown IncidentStatus = "unknown"
IncidentStatusResolved IncidentStatus = "resolved"
)
@@ -43,34 +44,37 @@ const (
// IncidentEvent represents a single timeline entry for an incident.
type IncidentEvent struct {
ID string `json:"id"`
Type IncidentEventType `json:"type"`
Timestamp time.Time `json:"timestamp"`
Summary string `json:"summary"`
Details map[string]interface{} `json:"details,omitempty"`
Source string `json:"source,omitempty"`
Evidence *unifiedresources.ResourceChange `json:"evidence,omitempty"`
ID string `json:"id"`
Type IncidentEventType `json:"type"`
Timestamp time.Time `json:"timestamp"`
Summary string `json:"summary"`
Details map[string]interface{} `json:"details,omitempty"`
}
// Incident captures an alert occurrence and its investigation timeline.
// It is an alert-scoped memory/projection for investigation support rather than
// the canonical durable resource-change history.
type Incident struct {
ID string `json:"id"`
AlertIdentifier string `json:"alertIdentifier"`
AlertType string `json:"alertType"`
Level string `json:"level"`
ResourceID string `json:"resourceId"`
ResourceName string `json:"resourceName"`
ResourceType string `json:"resourceType,omitempty"`
Node string `json:"node,omitempty"`
Instance string `json:"instance,omitempty"`
Message string `json:"message,omitempty"`
Status IncidentStatus `json:"status"`
OpenedAt time.Time `json:"openedAt"`
ClosedAt *time.Time `json:"closedAt,omitempty"`
Acknowledged bool `json:"acknowledged"`
AckUser string `json:"ackUser,omitempty"`
AckTime *time.Time `json:"ackTime,omitempty"`
Events []IncidentEvent `json:"events,omitempty"`
History *IncidentHistoryCoverage `json:"history,omitempty"`
ID string `json:"id"`
AlertIdentifier string `json:"alertIdentifier"`
AlertType string `json:"alertType"`
Level string `json:"level"`
ResourceID string `json:"resourceId"`
ResourceName string `json:"resourceName"`
ResourceType string `json:"resourceType,omitempty"`
Node string `json:"node,omitempty"`
Instance string `json:"instance,omitempty"`
Message string `json:"message,omitempty"`
Status IncidentStatus `json:"status"`
OpenedAt time.Time `json:"openedAt"`
ClosedAt *time.Time `json:"closedAt,omitempty"`
Acknowledged bool `json:"acknowledged"`
AckUser string `json:"ackUser,omitempty"`
AckTime *time.Time `json:"ackTime,omitempty"`
Events []IncidentEvent `json:"events,omitempty"`
}
type incidentShell struct {
@@ -90,23 +94,24 @@ type incidentShell struct {
}
type incidentJSON struct {
ID string `json:"id"`
AlertIdentifier string `json:"alertIdentifier"`
AlertType string `json:"alertType"`
Level string `json:"level"`
ResourceID string `json:"resourceId"`
ResourceName string `json:"resourceName"`
ResourceType string `json:"resourceType,omitempty"`
Node string `json:"node,omitempty"`
Instance string `json:"instance,omitempty"`
Message string `json:"message,omitempty"`
Status IncidentStatus `json:"status"`
OpenedAt time.Time `json:"openedAt"`
ClosedAt *time.Time `json:"closedAt,omitempty"`
Acknowledged bool `json:"acknowledged"`
AckUser string `json:"ackUser,omitempty"`
AckTime *time.Time `json:"ackTime,omitempty"`
Events []IncidentEvent `json:"events,omitempty"`
History *IncidentHistoryCoverage `json:"history,omitempty"`
ID string `json:"id"`
AlertIdentifier string `json:"alertIdentifier"`
AlertType string `json:"alertType"`
Level string `json:"level"`
ResourceID string `json:"resourceId"`
ResourceName string `json:"resourceName"`
ResourceType string `json:"resourceType,omitempty"`
Node string `json:"node,omitempty"`
Instance string `json:"instance,omitempty"`
Message string `json:"message,omitempty"`
Status IncidentStatus `json:"status"`
OpenedAt time.Time `json:"openedAt"`
ClosedAt *time.Time `json:"closedAt,omitempty"`
Acknowledged bool `json:"acknowledged"`
AckUser string `json:"ackUser,omitempty"`
AckTime *time.Time `json:"ackTime,omitempty"`
Events []IncidentEvent `json:"events,omitempty"`
}
type incidentShellJSON struct {
@@ -133,6 +138,7 @@ type incidentShellJSON struct {
func (i Incident) MarshalJSON() ([]byte, error) {
alertIdentifier := strings.TrimSpace(i.AlertIdentifier)
return json.Marshal(incidentJSON{
History: i.History,
ID: i.ID,
AlertIdentifier: alertIdentifier,
AlertType: i.AlertType,
@@ -162,6 +168,7 @@ func (i *Incident) UnmarshalJSON(data []byte) error {
return err
}
*i = Incident{
History: payload.History,
ID: payload.ID,
AlertIdentifier: strings.TrimSpace(payload.AlertIdentifier),
AlertType: payload.AlertType,
@@ -287,18 +294,19 @@ type IncidentStore struct {
}
const (
defaultIncidentMaxIncidents = 500
defaultIncidentMaxEvents = 120
defaultIncidentMaxAgeDays = 90
maxIncidentFileSize = 20 * 1024 * 1024 // 20MB
incidentStartMatchTolerance = time.Second
projectedIncidentChangeLimit = 256
incidentSnapshotSource = "alert_history_snapshot"
defaultIncidentMaxIncidents = 500
defaultIncidentMaxEvents = 120
defaultIncidentMaxAgeDays = 90
maxIncidentFileSize = 20 * 1024 * 1024 // 20MB
incidentStartMatchTolerance = time.Second
incidentSnapshotSource = "alert_history_snapshot"
)
// IncidentTimelineStore exposes the canonical resource timeline used to derive
// incident lifecycle and remediation history.
type IncidentTimelineStore interface {
GetRecentChangesFiltered(string, time.Time, int, unifiedresources.ResourceChangeFilters) ([]unifiedresources.ResourceChange, error)
ResourceHistoryIDs(string) ([]string, error)
GetRecentChanges(canonicalID string, since time.Time, limit int) ([]unifiedresources.ResourceChange, error)
}
@@ -473,7 +481,7 @@ func (s *IncidentStore) RecordAlertResolved(alert *alerts.Alert, resolvedAt time
// EnsureAlertOccurrence materializes the minimum honest timeline carried by
// an alert snapshot. It is the read-repair boundary for active alerts and
// legacy history entries that predate the canonical resource timeline. When
// canonical events exist, projectIncident replaces these snapshot-derived
// canonical events exist, QueryIncidents replaces these snapshot-derived
// lifecycle breadcrumbs with the durable projection.
func (s *IncidentStore) EnsureAlertOccurrence(alert *alerts.Alert, resolvedAt *time.Time) *Incident {
if s == nil || alert == nil || strings.TrimSpace(alert.ID) == "" {
@@ -651,6 +659,15 @@ func (s *IncidentStore) RecordNote(alertIdentifier, incidentID, note, user strin
return false
}
var projected *Incident
if strings.HasPrefix(incidentID, "projected-") {
page, err := s.QueryIncidents(IncidentQuery{AlertIdentifier: alertIdentifier, IncidentID: incidentID, Limit: 1})
if err != nil || len(page.Incidents) != 1 {
return false
}
projected = page.Incidents[0]
}
s.mu.Lock()
defer s.mu.Unlock()
@@ -660,6 +677,14 @@ func (s *IncidentStore) RecordNote(alertIdentifier, incidentID, note, user strin
} else if alertIdentifier != "" {
shell = s.findLatestIncidentByAlertIdentifierLocked(alertIdentifier)
}
if shell == nil && projected != nil {
shell = &incidentShell{ID: projected.ID, AlertIdentifier: projected.AlertIdentifier, AlertType: projected.AlertType, Level: projected.Level, ResourceID: projected.ResourceID, ResourceName: projected.ResourceName, ResourceType: projected.ResourceType, Node: projected.Node, Instance: projected.Instance, Message: projected.Message, OpenedAt: projected.OpenedAt}
if projected.ClosedAt != nil {
closed := *projected.ClosedAt
shell.OccurrenceClosedAt = &closed
}
s.incidents = append(s.incidents, shell)
}
if shell == nil {
return false
}
@@ -681,42 +706,24 @@ func (s *IncidentStore) RecordNote(alertIdentifier, incidentID, note, user strin
// GetTimelineByAlertIdentifier returns the most recent incident for the alert.
func (s *IncidentStore) GetTimelineByAlertIdentifier(alertIdentifier string) *Incident {
if alertIdentifier == "" {
return nil
}
s.mu.RLock()
shell := cloneIncidentShell(s.findLatestIncidentByAlertIdentifierLocked(alertIdentifier))
timelineStore := s.resourceTimelineStore
maxAge := s.maxAge
s.mu.RUnlock()
if shell == nil {
return s.projectIncidentFromCanonical(alertIdentifier, time.Time{}, timelineStore, maxAge)
}
return s.projectIncident(incidentFromShell(shell), timelineStore)
return s.GetTimelineByAlertAt(alertIdentifier, time.Time{})
}
// GetTimelineByAlertAt returns the incident closest to the provided start time for an alert.
// GetTimelineByAlertAt is retained for local writer callers. Runtime readers
// use QueryIncidents to distinguish an unavailable store from absent history.
func (s *IncidentStore) GetTimelineByAlertAt(alertIdentifier string, startedAt time.Time) *Incident {
if alertIdentifier == "" {
if strings.TrimSpace(alertIdentifier) == "" {
return nil
}
if startedAt.IsZero() {
return s.GetTimelineByAlertIdentifier(alertIdentifier)
page, err := s.QueryIncidents(IncidentQuery{AlertIdentifier: alertIdentifier, StartedAt: startedAt, Limit: 1})
if err != nil {
log.Warn().Err(err).Msg("incident history unavailable")
return nil
}
s.mu.RLock()
best, bestDelta := s.findClosestIncidentByAlertAtLocked(alertIdentifier, startedAt)
best = cloneIncidentShell(best)
timelineStore := s.resourceTimelineStore
maxAge := s.maxAge
s.mu.RUnlock()
if best == nil || bestDelta > incidentStartMatchTolerance {
return s.projectIncidentFromCanonical(alertIdentifier, startedAt, timelineStore, maxAge)
if len(page.Incidents) == 0 {
return nil
}
return s.projectIncident(incidentFromShell(best), timelineStore)
return page.Incidents[0]
}
func (s *IncidentStore) findIncidentByAlertAtLocked(alertIdentifier string, startedAt time.Time) *incidentShell {
@@ -751,56 +758,58 @@ func (s *IncidentStore) findClosestIncidentByAlertAtLocked(alertIdentifier strin
// ListIncidentsByResource returns recent incidents for a resource.
func (s *IncidentStore) ListIncidentsByResource(resourceID string, limit int) []*Incident {
if resourceID == "" {
if strings.TrimSpace(resourceID) == "" {
return nil
}
s.mu.RLock()
var matches []*Incident
for i := len(s.incidents) - 1; i >= 0; i-- {
shell := s.incidents[i]
if shell != nil && shell.ResourceID == resourceID {
matches = append(matches, incidentFromShell(cloneIncidentShell(shell)))
if limit > 0 && len(matches) >= limit {
break
}
}
page, err := s.QueryIncidents(IncidentQuery{ResourceID: resourceID, Limit: limit})
if err != nil {
log.Warn().Err(err).Msg("incident history unavailable")
return nil
}
timelineStore := s.resourceTimelineStore
s.mu.RUnlock()
if timelineStore == nil {
return matches
}
projected := make([]*Incident, 0, len(matches))
for _, incident := range matches {
projected = append(projected, s.projectIncident(incident, timelineStore))
}
return projected
return page.Incidents
}
// FormatForAlert returns a condensed incident timeline for prompt injection.
func (s *IncidentStore) FormatForAlert(alertIdentifier string, maxEvents int) string {
incident := s.GetTimelineByAlertIdentifier(alertIdentifier)
if incident == nil {
return ""
page, err := s.QueryIncidents(IncidentQuery{AlertIdentifier: alertIdentifier, Limit: 1})
if err != nil {
return "\n\nIncident history unavailable: canonical evidence could not be read.\n"
}
if len(page.Incidents) == 0 {
return formatIncidentCoverage(page)
}
incident := page.Incidents[0]
var b strings.Builder
b.WriteString("\n\n## Incident Memory\n")
b.WriteString(formatIncidentCoverage(page))
b.WriteString(fmt.Sprintf("Alert incident for %s (%s, %s)\n",
incident.ResourceName, incident.AlertType, incident.Level))
b.WriteString(fmt.Sprintf("Status: %s\n", incident.Status))
if incident.History != nil {
b.WriteString("Occurrence evidence source: " + incident.History.Source + "\n")
}
events := incident.Events
if maxEvents > 0 && len(events) > maxEvents {
events = events[len(events)-maxEvents:]
b.WriteString(fmt.Sprintf("Showing latest %d of %d recorded events.\n", len(events), len(incident.Events)))
}
for _, evt := range events {
b.WriteString("- ")
b.WriteString(evt.Timestamp.Format(time.RFC3339))
b.WriteString(formatIncidentTime(evt.Timestamp))
b.WriteString(": ")
b.WriteString(evt.Summary)
if evt.Type == IncidentEventNote {
if note, ok := stringMetadata(evt.Details, "note"); ok {
b.WriteString(": " + note)
}
}
if evt.Evidence != nil {
b.WriteString(fmt.Sprintf(" [record=%s observed=%s occurred=%s source=%s actor=%s]", evt.Evidence.ID, formatIncidentTime(evt.Evidence.ObservedAt), formatIncidentOccurredTime(evt.Evidence.OccurredAt), evt.Evidence.SourceAdapter, evt.Evidence.Actor))
} else if evt.Source != "" {
b.WriteString(" [source=" + evt.Source + "]")
}
b.WriteString("\n")
}
return b.String()
@@ -808,21 +817,32 @@ func (s *IncidentStore) FormatForAlert(alertIdentifier string, maxEvents int) st
// FormatForResource returns a condensed incident summary for a resource.
func (s *IncidentStore) FormatForResource(resourceID string, limit int) string {
incidents := s.ListIncidentsByResource(resourceID, limit)
page, err := s.QueryIncidents(IncidentQuery{ResourceID: resourceID, Limit: limit})
if err != nil {
return "\n\nIncident history unavailable: canonical evidence could not be read.\n"
}
return FormatIncidentPageForResource(page)
}
// FormatIncidentPageForResource formats the same snapshot returned to an API
// caller, so a second read cannot silently describe a different history.
func FormatIncidentPageForResource(page IncidentPage) string {
incidents := page.Incidents
if len(incidents) == 0 {
return ""
return formatIncidentCoverage(page)
}
var b strings.Builder
b.WriteString("\n\n## Incident Memory\n")
b.WriteString("Recent incidents for this resource:\n")
b.WriteString(formatIncidentCoverage(page))
b.WriteString("Recent incidents:\n")
for _, incident := range incidents {
status := string(incident.Status)
if incident.Acknowledged && incident.Status == IncidentStatusOpen {
status = "acknowledged"
}
b.WriteString("- ")
b.WriteString(incident.OpenedAt.Format(time.RFC3339))
b.WriteString(formatIncidentTime(incident.OpenedAt))
b.WriteString(": ")
b.WriteString(incident.AlertType)
if incident.Level != "" {
@@ -832,6 +852,9 @@ func (s *IncidentStore) FormatForResource(resourceID string, limit int) string {
}
b.WriteString(" - ")
b.WriteString(status)
if incident.History != nil {
b.WriteString(" [source=" + incident.History.Source + "]")
}
b.WriteString("\n")
}
return b.String()
@@ -843,28 +866,18 @@ func (s *IncidentStore) FormatForPatrol(limit int) string {
limit = 8
}
s.mu.RLock()
snapshot := make([]*Incident, 0, len(s.incidents))
for i := len(s.incidents) - 1; i >= 0 && len(snapshot) < limit; i-- {
if shell := s.incidents[i]; shell != nil {
snapshot = append(snapshot, incidentFromShell(cloneIncidentShell(shell)))
}
page, err := s.QueryIncidents(IncidentQuery{Limit: limit})
if err != nil {
return "\n\nIncident history unavailable: canonical evidence could not be read.\n"
}
timelineStore := s.resourceTimelineStore
s.mu.RUnlock()
snapshot := page.Incidents
if len(snapshot) == 0 {
return ""
}
if timelineStore != nil {
for i := range snapshot {
snapshot[i] = s.projectIncident(snapshot[i], timelineStore)
}
return formatIncidentCoverage(page)
}
var b strings.Builder
b.WriteString("\n\n## Incident Memory\n")
b.WriteString(formatIncidentCoverage(page))
b.WriteString("Recent incidents across infrastructure:\n")
for _, incident := range snapshot {
@@ -879,7 +892,7 @@ func (s *IncidentStore) FormatForPatrol(limit int) string {
}
b.WriteString("- ")
b.WriteString(incident.OpenedAt.Format(time.RFC3339))
b.WriteString(formatIncidentTime(incident.OpenedAt))
b.WriteString(": ")
if incident.ResourceName != "" {
b.WriteString(incident.ResourceName)
@@ -895,6 +908,9 @@ func (s *IncidentStore) FormatForPatrol(limit int) string {
}
b.WriteString(" - ")
b.WriteString(status)
if incident.History != nil {
b.WriteString(" [source=" + incident.History.Source + "]")
}
if lastSummary != "" {
b.WriteString(" - last: ")
b.WriteString(truncateOutput(lastSummary, 80))
@@ -912,162 +928,34 @@ func (s *IncidentStore) projectsFromCanonicalLocked() bool {
return s.resourceTimelineStore != nil
}
func (s *IncidentStore) projectIncident(incident *Incident, timelineStore IncidentTimelineStore) *Incident {
if incident == nil || timelineStore == nil {
return incident
}
projectedEvents := s.loadProjectedIncidentEvents(incident, timelineStore)
if len(projectedEvents) == 0 {
return incident
}
projected := cloneIncident(incident)
resetDerivedIncidentState(projected)
filtered := make([]IncidentEvent, 0, len(projected.Events)+len(projectedEvents))
for _, event := range projected.Events {
if isCanonicalProjectedIncidentEventType(event.Type) {
if !isSnapshotProjectionEvent(event) || hasIncidentEventType(projectedEvents, event.Type) {
continue
}
}
filtered = append(filtered, cloneIncidentEvent(event))
}
filtered = append(filtered, projectedEvents...)
sortIncidentEvents(filtered)
projected.Events = filtered
applyProjectedIncidentState(projected, filtered)
return projected
}
func (s *IncidentStore) projectIncidentFromCanonical(alertIdentifier string, startedAt time.Time, timelineStore IncidentTimelineStore, maxAge time.Duration) *Incident {
if timelineStore == nil || strings.TrimSpace(alertIdentifier) == "" {
return nil
}
since := time.Now().Add(-defaultIncidentMaxAgeDays * 24 * time.Hour)
if maxAge > 0 {
since = time.Now().Add(-maxAge)
}
changes, err := timelineStore.GetRecentChanges("", since, projectedIncidentChangeLimit)
if err != nil || len(changes) == 0 {
return nil
}
events := make([]IncidentEvent, 0, len(changes))
projected := &Incident{
ID: "projected-" + strings.TrimSpace(alertIdentifier),
AlertIdentifier: strings.TrimSpace(alertIdentifier),
Status: IncidentStatusOpen,
}
for _, change := range changes {
if projectedAlertIdentifier(change) != projected.AlertIdentifier {
continue
}
event, ok := incidentEventFromResourceChange(change)
if !ok {
continue
}
events = append(events, event)
hydrateIncidentFromCanonicalChange(projected, change)
}
if len(events) == 0 {
return nil
}
sortIncidentEvents(events)
projected.Events = events
if !startedAt.IsZero() {
openAt := projected.OpenedAt
if openAt.IsZero() {
openAt = events[0].Timestamp
}
delta := openAt.Sub(startedAt)
if delta < 0 {
delta = -delta
}
if delta > incidentStartMatchTolerance {
return nil
}
}
applyProjectedIncidentState(projected, events)
return projected
}
func (s *IncidentStore) loadProjectedIncidentEvents(incident *Incident, timelineStore IncidentTimelineStore) []IncidentEvent {
if incident == nil || timelineStore == nil {
return nil
}
resourceID := strings.TrimSpace(incident.ResourceID)
alertIdentifier := strings.TrimSpace(incident.AlertIdentifier)
if resourceID == "" || alertIdentifier == "" {
return nil
}
since := incident.OpenedAt
if since.IsZero() {
since = time.Now().Add(-defaultIncidentMaxAgeDays * 24 * time.Hour)
} else {
since = since.Add(-incidentStartMatchTolerance)
}
// Retained shells identify exact occurrences. Canonical history is keyed by
// alert identifier, so without an upper bound an old incident absorbs every
// later recurrence and can appear open again. Do not use the legacy read
// tolerance to include events from a preceding subsecond occurrence either.
var nextStart time.Time
s.mu.RLock()
for _, shell := range s.incidents {
if shell != nil && shell.AlertIdentifier == alertIdentifier && shell.ResourceID == resourceID &&
shell.OpenedAt.After(incident.OpenedAt) && (nextStart.IsZero() || shell.OpenedAt.Before(nextStart)) {
nextStart = shell.OpenedAt
}
}
s.mu.RUnlock()
changes, err := timelineStore.GetRecentChanges(resourceID, since, projectedIncidentChangeLimit)
if err != nil || len(changes) == 0 {
return nil
}
events := make([]IncidentEvent, 0, len(changes))
for _, change := range changes {
if projectedAlertIdentifier(change) != alertIdentifier {
continue
}
event, ok := incidentEventFromResourceChange(change)
if !ok {
continue
}
if (!incident.OpenedAt.IsZero() && event.Timestamp.Before(incident.OpenedAt)) ||
(!nextStart.IsZero() && !event.Timestamp.Before(nextStart)) {
continue
}
events = append(events, event)
hydrateIncidentFromCanonicalChange(incident, change)
}
sortIncidentEvents(events)
return events
}
func hydrateIncidentFromCanonicalChange(incident *Incident, change unifiedresources.ResourceChange) {
if incident == nil {
return
}
if resourceID := strings.TrimSpace(change.ResourceID); resourceID != "" && incident.ResourceID == "" {
// Alert evidence owns its target and risk context. A command may execute on
// a related host, so its target must not rename the alert's resource.
alertEvidence := false
switch change.Kind {
case unifiedresources.ChangeAlertFired, unifiedresources.ChangeAlertResolved,
unifiedresources.ChangeAlertAcknowledged, unifiedresources.ChangeAlertUnacknowledged,
unifiedresources.ChangeAlertSnoozed, unifiedresources.ChangeAlertUnsnoozed:
alertEvidence = true
}
if resourceID := strings.TrimSpace(change.ResourceID); resourceID != "" && (incident.ResourceID == "" || alertEvidence) {
incident.ResourceID = resourceID
}
if alertType, ok := stringMetadata(change.Metadata, unifiedresources.MetadataAlertType); ok && incident.AlertType == "" {
if alertType, ok := stringMetadata(change.Metadata, unifiedresources.MetadataAlertType); ok && (incident.AlertType == "" || alertEvidence) {
incident.AlertType = alertType
}
if level, ok := stringMetadata(change.Metadata, unifiedresources.MetadataAlertLevel); ok && incident.Level == "" {
if level, ok := stringMetadata(change.Metadata, unifiedresources.MetadataAlertLevel); ok && (incident.Level == "" || alertEvidence) {
incident.Level = level
}
if message, ok := stringMetadata(change.Metadata, unifiedresources.MetadataAlertMessage); ok && incident.Message == "" {
if message, ok := stringMetadata(change.Metadata, unifiedresources.MetadataAlertMessage); ok && (incident.Message == "" || alertEvidence) {
incident.Message = message
}
if resourceType, ok := stringMetadata(change.Metadata, "resourceType"); ok && (incident.ResourceType == "" || alertEvidence) {
incident.ResourceType = resourceType
}
if incident.OpenedAt.IsZero() {
incident.OpenedAt = incidentEventTimestamp(change)
}
@@ -1119,7 +1007,16 @@ func incidentEventFromResourceChange(change unifiedresources.ResourceChange) (In
}
}
evidence := change
evidence.Metadata = cloneIncidentEventDetails(change.Metadata)
evidence.RelatedResources = append([]string(nil), change.RelatedResources...)
if change.OccurredAt != nil {
occurred := *change.OccurredAt
evidence.OccurredAt = &occurred
}
return IncidentEvent{
Source: "canonical_resource_history",
Evidence: &evidence,
ID: strings.TrimSpace(change.ID),
Type: eventType,
Timestamp: incidentEventTimestamp(change),
@@ -1154,17 +1051,19 @@ func incidentEventTypeFromChangeKind(kind unifiedresources.ChangeKind) (Incident
func incidentEventSummaryFromChange(change unifiedresources.ResourceChange, eventType IncidentEventType) string {
switch eventType {
case IncidentEventAlertFired:
// The source owns the alert condition and comparison direction. Numeric
// placeholders cannot establish either, including for legacy records.
if message, ok := stringMetadata(change.Metadata, unifiedresources.MetadataAlertMessage); ok {
return message
}
if reason := strings.TrimSpace(change.Reason); reason != "" {
return reason
}
if alertType, ok := stringMetadata(change.Metadata, unifiedresources.MetadataAlertType); ok {
level, _ := stringMetadata(change.Metadata, unifiedresources.MetadataAlertLevel)
value, hasValue := floatMetadata(change.Metadata, unifiedresources.MetadataAlertValue)
threshold, hasThreshold := floatMetadata(change.Metadata, unifiedresources.MetadataAlertThreshold)
// Resource incidents carry numeric zero placeholders, not threshold evidence.
if alertType != "resource-incident" && (hasValue || hasThreshold) {
return fmt.Sprintf("Alert triggered: %s (%s %.1f >= %.1f)", alertType, level, value, threshold)
}
if level != "" {
if level, ok := stringMetadata(change.Metadata, unifiedresources.MetadataAlertLevel); ok {
return fmt.Sprintf("Alert triggered: %s (%s)", alertType, level)
}
return "Alert triggered: " + alertType
}
return "Alert triggered"
case IncidentEventAlertAcknowledged:
@@ -1192,7 +1091,7 @@ func incidentEventTimestamp(change unifiedresources.ResourceChange) time.Time {
if !change.ObservedAt.IsZero() {
return change.ObservedAt.UTC()
}
return time.Now().UTC()
return time.Time{}
}
func projectedAlertIdentifier(change unifiedresources.ResourceChange) string {
@@ -1219,28 +1118,6 @@ func stringMetadata(metadata map[string]any, key string) (string, bool) {
return str, true
}
func floatMetadata(metadata map[string]any, key string) (float64, bool) {
if len(metadata) == 0 {
return 0, false
}
value, ok := metadata[key]
if !ok {
return 0, false
}
switch typed := value.(type) {
case float64:
return typed, true
case float32:
return float64(typed), true
case int:
return float64(typed), true
case int64:
return float64(typed), true
default:
return 0, false
}
}
func eventActor(event IncidentEvent) string {
if event.Details == nil {
return ""
@@ -1304,6 +1181,16 @@ func incidentEventTimestampRank(eventType IncidentEventType) int {
func cloneIncidentEvent(event IncidentEvent) IncidentEvent {
cloned := event
cloned.Details = cloneIncidentEventDetails(event.Details)
if event.Evidence != nil {
evidence := *event.Evidence
evidence.Metadata = cloneIncidentEventDetails(evidence.Metadata)
evidence.RelatedResources = append([]string(nil), evidence.RelatedResources...)
if evidence.OccurredAt != nil {
occurred := *evidence.OccurredAt
evidence.OccurredAt = &occurred
}
cloned.Evidence = &evidence
}
return cloned
}
@@ -1313,11 +1200,28 @@ func cloneIncidentEventDetails(details map[string]any) map[string]interface{} {
}
cloned := make(map[string]interface{}, len(details))
for key, value := range details {
cloned[key] = value
cloned[key] = cloneIncidentDetailValue(value)
}
return cloned
}
func cloneIncidentDetailValue(value any) any {
switch v := value.(type) {
case map[string]any:
return cloneIncidentEventDetails(v)
case []any:
out := make([]any, len(v))
for i, item := range v {
out[i] = cloneIncidentDetailValue(item)
}
return out
case []string:
return append([]string(nil), v...)
default:
return value
}
}
func newIncidentShellFromAlert(alert *alerts.Alert) *incidentShell {
openedAt := alert.StartTime
if openedAt.IsZero() {
@@ -1415,6 +1319,13 @@ func (s *IncidentStore) addEventAtLocked(shell *incidentShell, eventType Inciden
Summary: summary,
Details: details,
}
if eventType == IncidentEventNote {
event.Source = "operator_note"
}
if isSnapshotProjectionEvent(event) {
event.Source = incidentSnapshotSource
}
shell.Events = append(shell.Events, event)
if s.maxEvents > 0 && len(shell.Events) > s.maxEvents {
shell.Events = shell.Events[len(shell.Events)-s.maxEvents:]
@@ -1660,7 +1571,16 @@ func incidentFromShell(shell *incidentShell) *Incident {
OpenedAt: shell.OpenedAt,
Events: cloneIncidentEvents(shell.Events),
}
if shell.OpenedAt.IsZero() {
incident.Status = IncidentStatusUnknown
}
if shell.OccurrenceClosedAt != nil {
closed := *shell.OccurrenceClosedAt
incident.ClosedAt = &closed
incident.Status = IncidentStatusResolved
}
applyProjectedIncidentState(incident, incident.Events)
incident.OpenedAt = shell.OpenedAt
return incident
}
@@ -1677,20 +1597,12 @@ func cloneIncident(src *Incident) *Incident {
t := *src.ClosedAt
clone.ClosedAt = &t
}
if len(src.Events) > 0 {
clone.Events = make([]IncidentEvent, len(src.Events))
for i, event := range src.Events {
cloneEvent := event
if event.Details != nil {
detailsCopy := make(map[string]interface{}, len(event.Details))
for key, value := range event.Details {
detailsCopy[key] = value
}
cloneEvent.Details = detailsCopy
}
clone.Events[i] = cloneEvent
}
clone.Events = cloneIncidentEvents(src.Events)
if src.History != nil {
history := *src.History
clone.History = &history
}
return &clone
}
internal/ai/memory/incidents_test.go
+19 -3
View File
@@ -838,7 +838,7 @@ func TestIncidentStore_FormatForResource(t *testing.T) {
if !strings.Contains(result, "## Incident Memory") {
t.Error("expected '## Incident Memory' header")
}
if !strings.Contains(result, "Recent incidents for this resource") {
if !strings.Contains(result, "Recent incidents:") {
t.Error("expected resource incidents header")
}
if !strings.Contains(result, "disk") {
@@ -1209,7 +1209,9 @@ func TestIncidentStore_CanonicalProjectionOccurrenceBounds(t *testing.T) {
// Checkpoint all retained boundaries, not just an isolated closed
// shell. The canonical timeline remains in memory: this exercises
// JSON recovery and projection, not a durable event-store restart.
before, err := json.Marshal(projected)
beforeProjection := *projected
beforeProjection.History = nil // Query coverage is refreshed on each read.
before, err := json.Marshal(beforeProjection)
if err != nil {
t.Fatal(err)
}
@@ -1230,7 +1232,21 @@ func TestIncidentStore_CanonicalProjectionOccurrenceBounds(t *testing.T) {
if len(reloaded.incidents) != len(store.incidents) {
t.Fatal("checkpoint replay duplicated a retained occurrence")
}
after, err := json.Marshal(reloaded.GetTimelineByAlertAt(old.ID, start))
afterProjection := reloaded.GetTimelineByAlertAt(old.ID, start)
if afterProjection == nil || projected.History == nil || afterProjection.History == nil {
t.Fatal("checkpoint replay lost the projection or its query coverage")
}
if afterProjection.History.ObservedBefore.Before(projected.History.ObservedBefore) || afterProjection.History.ObservedSince.Before(projected.History.ObservedSince) {
t.Fatal("fresh query coverage moved backwards")
}
beforeCoverage, afterCoverage := *projected.History, *afterProjection.History
afterCoverage.ObservedSince = beforeCoverage.ObservedSince
afterCoverage.ObservedBefore = beforeCoverage.ObservedBefore
if beforeCoverage != afterCoverage {
t.Fatalf("checkpoint replay changed coverage semantics: before %+v, after %+v", beforeCoverage, afterCoverage)
}
afterProjection.History = nil
after, err := json.Marshal(afterProjection)
if err != nil {
t.Fatal(err)
}
internal/ai/memory/provider_incident_summary_test.go
+16 -5
View File
@@ -6,19 +6,30 @@ import (
)
func TestProviderIncidentSummaryDoesNotInventThreshold(t *testing.T) {
for _, typ := range []string{"resource-incident", "cpu"} {
for _, typ := range []string{"resource-incident", "backup-storage-incident", "cpu"} {
change := unifiedresources.ResourceChange{Metadata: map[string]interface{}{
unifiedresources.MetadataAlertType: typ,
unifiedresources.MetadataAlertLevel: "warning",
unifiedresources.MetadataAlertValue: float64(0),
unifiedresources.MetadataAlertThreshold: float64(0),
}}
want := "Alert triggered: resource-incident (warning)"
if typ == "cpu" {
want = "Alert triggered: cpu (warning 0.0 >= 0.0)"
}
want := "Alert triggered: " + typ + " (warning)"
if got := incidentEventSummaryFromChange(change, IncidentEventAlertFired); got != want {
t.Fatalf("%s: got %q, want %q", typ, got, want)
}
}
}
func TestIncidentSummaryUsesSourceConditionWithoutInventingComparison(t *testing.T) {
for _, message := range []string{"Backup datastore is 90% full and protects seven workloads", "Free space fell below the configured minimum"} {
change := unifiedresources.ResourceChange{Reason: message, Metadata: map[string]any{
unifiedresources.MetadataAlertType: "backup-storage-incident",
unifiedresources.MetadataAlertMessage: message,
unifiedresources.MetadataAlertValue: 0.0,
unifiedresources.MetadataAlertThreshold: 0.0,
}}
if got := incidentEventSummaryFromChange(change, IncidentEventAlertFired); got != message {
t.Fatalf("got %q, want recorded condition %q", got, message)
}
}
}
internal/ai/service.go
+7 -3
View File
@@ -4999,11 +4999,15 @@ func (s *Service) lookupIncidentResourceIdentifier(alertIdentifier string) strin
if store == nil {
return ""
}
timeline := store.GetTimelineByAlertIdentifier(alertIdentifier)
if timeline == nil {
page, err := store.QueryIncidents(memory.IncidentQuery{AlertIdentifier: alertIdentifier, Limit: 1})
if err != nil {
log.Warn().Err(err).Msg("Canonical incident resource lookup unavailable")
return ""
}
return strings.TrimSpace(timeline.ResourceID)
if len(page.Incidents) == 0 {
return ""
}
return strings.TrimSpace(page.Incidents[0].ResourceID)
}
func (s *Service) recordCanonicalResourceChange(change *unifiedresources.ResourceChange) {
internal/ai/tools/tools_knowledge.go
+1 -1
View File
@@ -219,7 +219,7 @@ func (e *PulseToolExecutor) executeGetIncidentWindow(_ context.Context, args map
"limit": limit,
"has_more": hasMore,
"coverage": "retained_records_only",
"evidence_limit": "These are retained observations, not continuous coverage. Empty history does not establish health or absence of incidents. ObservedAt is when Pulse observed a change, while OccurredAt is present only when its occurrence time is known. An alert resolving establishes that alert's recovery, not its cause or a verified action outcome.",
"evidence_limit": "These are retained observations, not continuous coverage. Empty history does not establish health or absence of incidents. ObservedAt is when Pulse observed a change, while OccurredAt is present only when its occurrence time is known. A resolution records alert closure, not its cause, workload recovery or a verified action outcome. RelatedResources identifies relationships, not additional targets of this event. This read does not query action records.",
}), nil
}
internal/api/ai_intelligence_handlers.go
+16 -21
View File
@@ -9,6 +9,7 @@ import (
"github.com/rcourtman/pulse-go-rewrite/internal/ai"
"github.com/rcourtman/pulse-go-rewrite/internal/ai/baseline"
"github.com/rcourtman/pulse-go-rewrite/internal/ai/memory"
"github.com/rcourtman/pulse-go-rewrite/internal/ai/unified"
"github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources"
"github.com/rcourtman/pulse-go-rewrite/internal/utils"
@@ -1464,31 +1465,21 @@ func (h *AISettingsHandler) HandleGetRecentIncidents(w http.ResponseWriter, r *h
// Get the resource ID filter if provided
resourceID := r.URL.Query().Get("resource_id")
var incidents interface{}
if resourceID != "" {
incidents = incidentStore.ListIncidentsByResource(resourceID, limit)
} else {
// No direct method to list all incidents, use FormatForPatrol for now
// This is a limitation - we may want to add ListRecentIncidents to the store
incidentSummary := incidentStore.FormatForPatrol(limit)
if err := utils.WriteJSONResponse(w, map[string]interface{}{
"incidents": []interface{}{},
"incident_summary": incidentSummary,
"active_count": nil,
"active_count_status": "not_measured",
}); err != nil {
log.Error().Err(err).Msg("Failed to write incidents response")
}
page, err := incidentStore.QueryIncidents(memory.IncidentQuery{ResourceID: resourceID, Limit: limit})
if err != nil {
http.Error(w, "Incident history unavailable", http.StatusServiceUnavailable)
return
}
if err := utils.WriteJSONResponse(w, map[string]interface{}{
"incidents": incidents,
"incidents": page.Incidents,
"history": page.History,
"incident_summary": memory.FormatIncidentPageForResource(page),
"active_count": nil,
"active_count_status": "not_measured",
}); err != nil {
log.Error().Err(err).Msg("Failed to write incidents response")
}
}
// HandleGetIncidentData returns incident data for a specific resource (GET /api/ai/incidents/{resourceID})
@@ -1544,15 +1535,19 @@ func (h *AISettingsHandler) HandleGetIncidentData(w http.ResponseWriter, r *http
return
}
incidents := incidentStore.ListIncidentsByResource(resourceID, limit)
// Also get formatted context for AI
formattedContext := incidentStore.FormatForResource(resourceID, limit)
page, err := incidentStore.QueryIncidents(memory.IncidentQuery{ResourceID: resourceID, Limit: limit})
if err != nil {
http.Error(w, "Incident history unavailable", http.StatusServiceUnavailable)
return
}
incidents := page.Incidents
formattedContext := memory.FormatIncidentPageForResource(page)
if err := utils.WriteJSONResponse(w, map[string]interface{}{
"resource_id": resourceID,
"incidents": incidents,
"formatted_context": formattedContext,
"history": page.History,
}); err != nil {
log.Error().Err(err).Msg("Failed to write incident data response")
}
internal/api/ai_intelligence_handlers_remediation_additional_test.go
+36
View File
@@ -13,6 +13,7 @@ import (
"github.com/rcourtman/pulse-go-rewrite/internal/ai/memory"
"github.com/rcourtman/pulse-go-rewrite/internal/alerts"
"github.com/rcourtman/pulse-go-rewrite/internal/config"
"github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources"
)
func TestHandleGetCircuitBreakerStatus(t *testing.T) {
@@ -194,3 +195,38 @@ func TestHandleGetRecentIncidentsCountIsNotMeasured(t *testing.T) {
})
}
}
func TestHandleGetRecentIncidentsCanonicalOnlyAndReadFailure(t *testing.T) {
handler, incidents := setupIncidentHandler(t)
canonical, err := unifiedresources.NewSQLiteResourceStore(t.TempDir(), "default")
if err != nil {
t.Fatal(err)
}
defer canonical.Close()
incidents.SetResourceTimelineStore(canonical)
change := unifiedresources.BuildAlertTimelineChange("canonical-only-resource", unifiedresources.ChangeAlertFired, time.Now().Add(-time.Minute), "", unifiedresources.AlertTimelineChange{AlertIdentifier: "canonical-only-alert", AlertType: "disk"})
if err := canonical.RecordChange(*change); err != nil {
t.Fatal(err)
}
request := httptest.NewRequest(http.MethodGet, "/api/ai/incidents?resource_id=canonical-only-resource", nil)
response := httptest.NewRecorder()
handler.HandleGetRecentIncidents(response, request)
if response.Code != http.StatusOK {
t.Fatal(response.Code, response.Body.String())
}
var page memory.IncidentPage
if err := json.Unmarshal(response.Body.Bytes(), &page); err != nil {
t.Fatal(err)
}
if len(page.Incidents) != 1 || page.Incidents[0].AlertIdentifier != "canonical-only-alert" || page.History.Source != "canonical_resource_history" {
t.Fatalf("canonical evidence missing: %#v", page)
}
if err := canonical.Close(); err != nil {
t.Fatal(err)
}
response = httptest.NewRecorder()
handler.HandleGetRecentIncidents(response, request)
if response.Code != http.StatusServiceUnavailable {
t.Fatal(response.Code, response.Body.String())
}
}
internal/api/alerting/alerts.go
+43 -29
View File
@@ -22,6 +22,7 @@ import (
"github.com/rcourtman/pulse-go-rewrite/internal/models"
"github.com/rcourtman/pulse-go-rewrite/internal/monitoring"
"github.com/rcourtman/pulse-go-rewrite/internal/notifications"
"github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources"
"github.com/rcourtman/pulse-go-rewrite/internal/utils"
"github.com/rcourtman/pulse-go-rewrite/internal/websocket"
"github.com/rs/zerolog/log"
@@ -834,12 +835,16 @@ func (h *AlertHandlers) GetAlertIncidentTimeline(w http.ResponseWriter, r *http.
return
}
var incident *memory.Incident
if !startedAt.IsZero() {
incident = store.GetTimelineByAlertAt(alertID, startedAt)
} else {
incident = store.GetTimelineByAlertIdentifier(alertID)
page, err := store.QueryIncidents(memory.IncidentQuery{AlertIdentifier: alertID, StartedAt: startedAt, Limit: 1})
if err != nil {
http.Error(w, "Incident history unavailable", http.StatusServiceUnavailable)
return
}
var incident *memory.Incident
if len(page.Incidents) > 0 {
incident = page.Incidents[0]
}
if !startedAt.IsZero() && (incident == nil || len(incident.Events) == 0) {
if alert, resolvedAt := findAlertOccurrenceForTimeline(h.getMonitor(r.Context()).GetAlertManager(), alertID, startedAt); alert != nil {
incident = store.EnsureAlertOccurrence(alert, resolvedAt)
@@ -868,8 +873,12 @@ func (h *AlertHandlers) GetAlertIncidentTimeline(w http.ResponseWriter, r *http.
http.Error(w, "Incident store unavailable", http.StatusServiceUnavailable)
return
}
incidents := store.ListIncidentsByResource(resourceID, limit)
if err := utils.WriteJSONResponse(w, exportIncidents(incidents)); err != nil {
page, err := store.QueryIncidents(memory.IncidentQuery{ResourceID: resourceID, Limit: limit})
if err != nil {
http.Error(w, "Incident history unavailable", http.StatusServiceUnavailable)
return
}
if err := utils.WriteJSONResponse(w, exportIncidents(page.Incidents)); err != nil {
log.Error().Err(err).Msg("Failed to write incident list response")
}
return
@@ -987,11 +996,13 @@ func (h *AlertHandlers) SaveAlertIncidentNote(w http.ResponseWriter, r *http.Req
}
type incidentEventView struct {
ID string `json:"id"`
Type memory.IncidentEventType `json:"type"`
Timestamp time.Time `json:"timestamp"`
Summary string `json:"summary"`
Details map[string]interface{} `json:"details"`
Source string `json:"source,omitempty"`
Evidence *unifiedresources.ResourceChange `json:"evidence,omitempty"`
ID string `json:"id"`
Type memory.IncidentEventType `json:"type"`
Timestamp time.Time `json:"timestamp"`
Summary string `json:"summary"`
Details map[string]interface{} `json:"details"`
}
func (v incidentEventView) NormalizeCollections() incidentEventView {
@@ -1002,23 +1013,24 @@ func (v incidentEventView) NormalizeCollections() incidentEventView {
}
type incidentView struct {
ID string `json:"id"`
AlertIdentifier string `json:"alertIdentifier"`
AlertType string `json:"alertType"`
Level string `json:"level"`
ResourceID string `json:"resourceId"`
ResourceName string `json:"resourceName"`
ResourceType string `json:"resourceType,omitempty"`
Node string `json:"node,omitempty"`
Instance string `json:"instance,omitempty"`
Message string `json:"message,omitempty"`
Status memory.IncidentStatus `json:"status"`
OpenedAt time.Time `json:"openedAt"`
ClosedAt *time.Time `json:"closedAt,omitempty"`
Acknowledged bool `json:"acknowledged"`
AckUser string `json:"ackUser,omitempty"`
AckTime *time.Time `json:"ackTime,omitempty"`
Events []incidentEventView `json:"events"`
History *memory.IncidentHistoryCoverage `json:"history,omitempty"`
ID string `json:"id"`
AlertIdentifier string `json:"alertIdentifier"`
AlertType string `json:"alertType"`
Level string `json:"level"`
ResourceID string `json:"resourceId"`
ResourceName string `json:"resourceName"`
ResourceType string `json:"resourceType,omitempty"`
Node string `json:"node,omitempty"`
Instance string `json:"instance,omitempty"`
Message string `json:"message,omitempty"`
Status memory.IncidentStatus `json:"status"`
OpenedAt time.Time `json:"openedAt"`
ClosedAt *time.Time `json:"closedAt,omitempty"`
Acknowledged bool `json:"acknowledged"`
AckUser string `json:"ackUser,omitempty"`
AckTime *time.Time `json:"ackTime,omitempty"`
Events []incidentEventView `json:"events"`
}
func emptyIncidentView() incidentView {
@@ -1042,6 +1054,7 @@ func exportIncident(incident *memory.Incident) *incidentView {
events := make([]incidentEventView, 0, len(incident.Events))
for _, event := range incident.Events {
events = append(events, incidentEventView{
Source: event.Source, Evidence: event.Evidence,
ID: event.ID,
Type: event.Type,
Timestamp: event.Timestamp,
@@ -1052,6 +1065,7 @@ func exportIncident(incident *memory.Incident) *incidentView {
alertIdentifier := strings.TrimSpace(incident.AlertIdentifier)
view := emptyIncidentView()
view.ID = incident.ID
view.History = incident.History
view.AlertIdentifier = alertIdentifier
view.AlertType = incident.AlertType
view.Level = incident.Level
internal/api/alerting/incident_query_test.go
+49
View File
@@ -0,0 +1,49 @@
package alerting
import (
"encoding/json"
"net/http"
"net/http/httptest"
"net/url"
"testing"
"time"
"github.com/rcourtman/pulse-go-rewrite/internal/ai/memory"
"github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources"
"github.com/stretchr/testify/require"
)
func TestIncidentQueryHTTPPreservesOccurrenceAndEvidence(t *testing.T) {
canonical, err := unifiedresources.NewSQLiteResourceStore(t.TempDir(), "default")
require.NoError(t, err)
defer canonical.Close()
incidents := memory.NewIncidentStore(memory.IncidentStoreConfig{})
incidents.SetResourceTimelineStore(canonical)
monitor := new(MockAlertMonitor)
monitor.On("GetIncidentStore").Return(incidents)
handler := NewAlertHandlers(nil, monitor, nil)
first := time.Now().UTC().Add(-time.Hour).Truncate(time.Second)
for index, kind := range []unifiedresources.ChangeKind{unifiedresources.ChangeAlertFired, unifiedresources.ChangeAlertResolved, unifiedresources.ChangeAlertFired} {
at := first.Add(time.Duration(index) * time.Minute)
change := unifiedresources.BuildAlertTimelineChange("resource-a", kind, at, "operator", unifiedresources.AlertTimelineChange{AlertIdentifier: "repeated", AlertType: "disk", AlertLevel: "warning"})
require.NoError(t, canonical.RecordChange(*change))
}
request := httptest.NewRequest(http.MethodGet, "/api/alerts/incidents?alertIdentifier=repeated&started_at="+url.QueryEscape(first.Format(time.RFC3339)), nil)
response := httptest.NewRecorder()
handler.GetAlertIncidentTimeline(response, request)
require.Equal(t, http.StatusOK, response.Code, response.Body.String())
var timeline memory.Incident
require.NoError(t, json.Unmarshal(response.Body.Bytes(), &timeline))
require.Equal(t, memory.IncidentStatusResolved, timeline.Status)
require.Len(t, timeline.Events, 2)
require.NotNil(t, timeline.History)
require.NotNil(t, timeline.Events[0].Evidence)
require.Equal(t, "operator", timeline.Events[0].Evidence.Actor)
require.Equal(t, "canonical_resource_history", timeline.Events[0].Source)
// A failed read must not enter snapshot-repair fallback or return null/200.
require.NoError(t, canonical.Close())
response = httptest.NewRecorder()
handler.GetAlertIncidentTimeline(response, request)
require.Equal(t, http.StatusServiceUnavailable, response.Code, response.Body.String())
monitor.AssertNotCalled(t, "GetAlertManager")
}
internal/monitoring/monitor_alert_handling_test.go
+2 -1
View File
@@ -214,10 +214,11 @@ func TestMonitor_HandleAlertLifecycle_WritesCanonicalChanges(t *testing.T) {
if len(changes) != 5 {
t.Fatalf("expected 5 canonical changes, got %d", len(changes))
}
// Arrival order can differ from observation order during lifecycle replay.
wantKinds := []unifiedresources.ChangeKind{
unifiedresources.ChangeAlertUnacknowledged,
unifiedresources.ChangeAlertUnsnoozed,
unifiedresources.ChangeAlertSnoozed,
unifiedresources.ChangeAlertUnacknowledged,
unifiedresources.ChangeAlertAcknowledged,
unifiedresources.ChangeAlertFired,
}
internal/monitoring/monitor_alerts.go
+9 -1
View File
@@ -521,7 +521,15 @@ func (m *Monitor) reconcileActiveAlertTimelines() {
activeAlerts := m.alertManager.GetActiveAlerts()
for i := range activeAlerts {
alert := &activeAlerts[i]
timeline := m.incidentStore.GetTimelineByAlertAt(alert.ID, alert.StartTime)
page, err := m.incidentStore.QueryIncidents(memory.IncidentQuery{AlertIdentifier: alert.ID, StartedAt: alert.StartTime, Limit: 1})
if err != nil {
log.Warn().Err(err).Msg("Skipping alert timeline reconciliation because canonical history is unavailable")
continue
}
var timeline *memory.Incident
if len(page.Incidents) > 0 {
timeline = page.Incidents[0]
}
if timeline != nil {
hasFired := false
for _, event := range timeline.Events {
internal/unifiedresources/changes.go
+25
View File
@@ -59,6 +59,12 @@ const (
// ResourceChangeFilters narrows the resource timeline to specific change kinds
// and source origins while preserving the canonical change record shape.
type ResourceChangeFilters struct {
// AlertIdentifiers matches exact canonical alert metadata before any limit.
// It is a history selector, never an action or resource authority binding.
AlertIdentifiers []string `json:"alertIdentifiers,omitempty"`
// ObservedBefore is an exclusive upper observation bound. Together with
// since it selects [since, before), without changing occurrence timestamps.
ObservedBefore *time.Time `json:"observedBefore,omitempty"`
Kinds []ChangeKind `json:"kinds,omitempty"`
SourceTypes []ChangeSourceType `json:"sourceTypes,omitempty"`
SourceAdapters []ChangeSourceAdapter `json:"sourceAdapters,omitempty"`
@@ -66,6 +72,25 @@ type ResourceChangeFilters struct {
}
func (filters ResourceChangeFilters) matches(change ResourceChange) bool {
if filters.ObservedBefore != nil && !change.ObservedAt.Before(*filters.ObservedBefore) {
return false
}
if len(filters.AlertIdentifiers) > 0 {
identifier, ok := change.Metadata[MetadataAlertIdentifier].(string)
if !ok {
return false
}
found := false
for _, wanted := range filters.AlertIdentifiers {
if identifier == wanted {
found = true
break
}
}
if !found {
return false
}
}
if len(filters.Kinds) > 0 {
match := false
for _, kind := range filters.Kinds {
internal/unifiedresources/incident_history_query_test.go
+147
View File
@@ -0,0 +1,147 @@
package unifiedresources
import (
"fmt"
"strings"
"testing"
"time"
"github.com/stretchr/testify/require"
)
// These queries must not lose a relevant incident behind a fleet-wide limit.
// Use both durable and in-memory stores, including late observations and aliases.
func TestIncidentHistoryQueryFiltersBeforeLimit(t *testing.T) {
for _, backend := range []string{"memory", "sqlite"} {
t.Run(backend, func(t *testing.T) {
var store ResourceStore
if backend == "memory" {
store = NewMemoryStore()
} else {
sqlite, err := NewSQLiteResourceStore(t.TempDir(), "default")
require.NoError(t, err)
t.Cleanup(func() { require.NoError(t, sqlite.Close()) })
store = sqlite
}
now := time.Now().UTC().Truncate(time.Second)
older := now.Add(-time.Hour)
record := func(id, alert string, observed time.Time) {
require.NoError(t, store.RecordChange(ResourceChange{ID: id, ResourceID: "resource-a", Kind: ChangeAlertFired, ObservedAt: observed, OccurredAt: &older, SourceType: SourceHeuristic, SourceAdapter: AdapterProxmox, Metadata: map[string]any{MetadataAlertIdentifier: alert}}))
}
record("target-a", "wanted", now)
record("target-b", "wanted", now) // equal timestamps have stable ID order
for i := 0; i < 300; i++ {
record(fmt.Sprintf("noise-%03d", i), "unrelated", now.Add(time.Second))
}
record("late-insertion", "wanted", older)
record("upper-bound", "wanted", now.Add(2*time.Second))
before := now.Add(2 * time.Second)
filters := ResourceChangeFilters{AlertIdentifiers: []string{"wanted"}, ObservedBefore: &before, Kinds: []ChangeKind{ChangeAlertFired}, SourceTypes: []ChangeSourceType{SourceHeuristic}, SourceAdapters: []ChangeSourceAdapter{AdapterProxmox}}
got, err := store.GetRecentChangesFiltered("", older, 2, filters)
require.NoError(t, err)
require.Len(t, got, 2)
require.Equal(t, "target-b", got[0].ID)
require.Equal(t, "target-a", got[1].ID)
adapter := NewMonitorAdapter(NewRegistry(store))
forwarded, err := adapter.GetRecentChangesFiltered("", older, 2, filters)
require.NoError(t, err)
require.Equal(t, got, forwarded)
require.Equal(t, older, *got[0].OccurredAt) // observation bounds do not rewrite occurrence time
count, err := store.CountRecentChangesFiltered("", older, filters)
require.NoError(t, err)
require.Equal(t, 3, count)
kinds, err := store.CountRecentChangesByKindFiltered("", older, filters)
require.NoError(t, err)
require.Equal(t, 3, kinds[ChangeAlertFired])
sources, err := store.CountRecentChangesBySourceTypeFiltered("", older, filters)
require.NoError(t, err)
require.Equal(t, 3, sources[SourceHeuristic])
adapters, err := store.CountRecentChangesBySourceAdapterFiltered("", older, filters)
require.NoError(t, err)
require.Equal(t, 3, adapters[AdapterProxmox])
// A history alias selects the original evidence without rewriting its
// resource identity, or promoting the alias into action authority.
writer := store.(interface {
RecordChangeWithSourceIdentity(ResourceChange, string) error
})
require.NoError(t, writer.RecordChangeWithSourceIdentity(ResourceChange{ID: "alias-event", ResourceID: "resource-canonical", Kind: ChangeAlertResolved, ObservedAt: now, Metadata: map[string]any{MetadataAlertIdentifier: "alias-alert"}}, "resource-legacy"))
aliasFilters := ResourceChangeFilters{AlertIdentifiers: []string{"alias-alert"}, ObservedBefore: &before}
for _, resource := range []string{"resource-legacy", "resource-canonical"} {
identities, err := adapter.ResourceHistoryIDs(resource)
require.NoError(t, err)
require.Contains(t, identities, "resource-legacy")
require.Contains(t, identities, "resource-canonical")
aliased, err := store.GetRecentChangesFiltered(resource, older, 1, aliasFilters)
require.NoError(t, err)
require.Len(t, aliased, 1)
require.Equal(t, "resource-canonical", aliased[0].ResourceID)
count, err := store.CountRecentChangesFiltered(resource, older, aliasFilters)
require.NoError(t, err)
require.Equal(t, 1, count)
}
// JSON numbers must not compare equal to textual alert identifiers.
require.NoError(t, store.RecordChange(ResourceChange{ID: "numeric-identity", ResourceID: "resource-a", Kind: ChangeAlertFired, ObservedAt: now, Metadata: map[string]any{MetadataAlertIdentifier: 123}}))
malformed, err := store.GetRecentChangesFiltered("", older, 10, ResourceChangeFilters{AlertIdentifiers: []string{"123"}})
require.NoError(t, err)
require.Empty(t, malformed)
filters.AlertIdentifiers = []string{"wanted' OR 1=1 --"}
got, err = store.GetRecentChangesFiltered("", older, 2, filters)
require.NoError(t, err)
require.Empty(t, got)
})
}
}
func TestIncidentHistoryQueryReadFailureIsNotEmptyHistory(t *testing.T) {
store, err := NewSQLiteResourceStore(t.TempDir(), "default")
require.NoError(t, err)
require.NoError(t, store.Close())
filters := ResourceChangeFilters{AlertIdentifiers: []string{"wanted"}}
_, err = store.GetRecentChangesFiltered("", time.Time{}, 1, filters)
require.Error(t, err)
_, err = store.CountRecentChangesFiltered("", time.Time{}, filters)
require.Error(t, err)
adapter := NewMonitorAdapter(NewRegistry(store))
_, err = adapter.GetRecentChangesFiltered("", time.Time{}, 1, filters)
require.Error(t, err)
_, err = (&MonitorAdapter{}).GetRecentChangesFiltered("", time.Time{}, 1, filters)
require.ErrorContains(t, err, "unavailable")
_, err = (&MonitorAdapter{}).ResourceHistoryIDs("resource-a")
require.ErrorContains(t, err, "unavailable")
}
func TestIncidentHistoryQueryInvalidLegacyMetadata(t *testing.T) {
store, err := NewSQLiteResourceStore(t.TempDir(), "default")
require.NoError(t, err)
defer store.Close()
require.NoError(t, store.RecordChange(ResourceChange{ID: "bad-json", ResourceID: "resource-a", Kind: ChangeAlertFired, ObservedAt: time.Now()}))
_, err = store.db.Exec(`UPDATE resource_changes SET metadata_json = '{invalid' WHERE id = 'bad-json'`)
require.NoError(t, err)
filters := ResourceChangeFilters{AlertIdentifiers: []string{"wanted"}}
got, err := store.GetRecentChangesFiltered("", time.Time{}, 1, filters)
require.NoError(t, err)
require.Empty(t, got)
count, err := store.CountRecentChangesFiltered("", time.Time{}, filters)
require.NoError(t, err)
require.Zero(t, count)
}
func TestIncidentHistoryQueryUsesAlertIdentityIndex(t *testing.T) {
store, err := NewSQLiteResourceStore(t.TempDir(), "default")
require.NoError(t, err)
defer store.Close()
query, args := buildRecentChangeCountQuery(nil, time.Time{}, ResourceChangeFilters{AlertIdentifiers: []string{"wanted"}, Kinds: []ChangeKind{ChangeAlertFired, ChangeAlertResolved}}, "EXPLAIN QUERY PLAN SELECT COUNT(*) FROM resource_changes", store.resourceChangesObservedAtExpr(), store.resourceChangesSourceTypeExpr(), store.resourceChangesSourceAdapterExpr())
rows, err := store.db.Query(query, args...)
require.NoError(t, err)
defer rows.Close()
var plans []string
for rows.Next() {
var id, parent, unused int
var detail string
require.NoError(t, rows.Scan(&id, &parent, &unused, &detail))
plans = append(plans, detail)
}
require.NoError(t, rows.Err())
require.Contains(t, strings.Join(plans, "\n"), "idx_resource_changes_alert_time")
}
internal/unifiedresources/monitor_adapter.go
+25
View File
@@ -1,6 +1,7 @@
package unifiedresources
import (
"fmt"
"sort"
"strings"
"sync"
@@ -151,6 +152,30 @@ func (a *MonitorAdapter) GetRecentChanges(canonicalID string, since time.Time, l
return registry.store.GetRecentChanges(canonicalID, since, limit)
}
// GetRecentChangesFiltered preserves the canonical query predicates used by
// incident projections, including alert identity before the evidence limit.
func (a *MonitorAdapter) GetRecentChangesFiltered(resourceID string, since time.Time, limit int, filters ResourceChangeFilters) ([]ResourceChange, error) {
registry := a.currentRegistry()
if registry == nil || registry.store == nil {
return nil, fmt.Errorf("canonical resource history unavailable")
}
return registry.store.GetRecentChangesFiltered(resourceID, since, limit, filters)
}
func (a *MonitorAdapter) ResourceHistoryIDs(resourceID string) ([]string, error) {
registry := a.currentRegistry()
if registry == nil || registry.store == nil {
return nil, fmt.Errorf("canonical resource history unavailable")
}
reader, ok := registry.store.(interface {
ResourceHistoryIDs(string) ([]string, error)
})
if !ok {
return nil, fmt.Errorf("canonical history identities unavailable")
}
return reader.ResourceHistoryIDs(resourceID)
}
// GetResourceOperatorState exposes the durable operator-intent record through
// the monitor adapter without widening monitoring's core resource-store
// interface. Alert policy resolution discovers this capability optionally.
internal/unifiedresources/store.go
+63 -29
View File
@@ -31,6 +31,7 @@ type ResourceStore interface {
UpsertResourceIdentityPins(pins []ResourceIdentityPin) error
ListResourceIdentityPins() ([]ResourceIdentityPin, error)
RecordChange(change ResourceChange) error
ResourceHistoryIDs(resourceID string) ([]string, error)
GetRecentChanges(canonicalID string, since time.Time, limit int) ([]ResourceChange, error)
GetRecentChangesFiltered(canonicalID string, since time.Time, limit int, filters ResourceChangeFilters) ([]ResourceChange, error)
CountRecentChanges(canonicalID string, since time.Time) (int, error)
@@ -1033,6 +1034,7 @@ func (s *SQLiteResourceStore) migrateResourceChangesSchema() error {
func (s *SQLiteResourceStore) ensureResourceChangesIndexes() error {
indexes := []string{
`CREATE INDEX IF NOT EXISTS idx_resource_changes_alert_time ON resource_changes(` + resourceChangesAlertIdentifierExpr() + `, observed_at DESC, id DESC)`,
`CREATE INDEX IF NOT EXISTS idx_resource_changes_time ON resource_changes(observed_at DESC)`,
`CREATE INDEX IF NOT EXISTS idx_resource_changes_canonical_time ON resource_changes(canonical_id, observed_at DESC)`,
`CREATE INDEX IF NOT EXISTS idx_resource_changes_kind_time ON resource_changes(kind, observed_at DESC)`,
@@ -1165,6 +1167,12 @@ func resourceChangesRelatedResourcesExpr() string {
return "COALESCE(NULLIF(TRIM(related_resources), ''), '[]')"
}
// The same guarded expression owns both the alert selector and its index.
// Invalid legacy metadata cannot establish an alert identity.
func resourceChangesAlertIdentifierExpr() string {
return "json_extract(CASE WHEN json_valid(metadata_json) THEN metadata_json ELSE '{}' END, '$.alert_identifier')"
}
func resourceChangesMetadataJSONExpr() string {
return "COALESCE(NULLIF(TRIM(metadata_json), ''), '{}')"
}
@@ -1754,6 +1762,24 @@ func recordChangeSQL(execer sqlExecutor, change ResourceChange, includeTimestamp
return nil
}
// ResourceHistoryIDs expands read-only history identities. These aliases do
// not authorize actions or change the resource identity recorded on evidence.
func (s *SQLiteResourceStore) ResourceHistoryIDs(resourceID string) ([]string, error) {
return s.resourceChangeIDSet(resourceID)
}
func (m *MemoryStore) ResourceHistoryIDs(resourceID string) ([]string, error) {
m.mu.RLock()
defer m.mu.RUnlock()
ids := m.resourceChangeIDSetLocked(CanonicalResourceID(resourceID))
result := make([]string, 0, len(ids))
for _, id := range ids {
result = append(result, id)
}
sort.Strings(result)
return result, nil
}
func (s *SQLiteResourceStore) GetRecentChanges(canonicalID string, since time.Time, limit int) ([]ResourceChange, error) {
return s.GetRecentChangesFiltered(canonicalID, since, limit, ResourceChangeFilters{})
}
@@ -1783,35 +1809,12 @@ func (s *SQLiteResourceStore) GetRecentChangesFiltered(canonicalID string, since
conditions = append(conditions, observedAtExpr+" >= ?")
args = append(args, since)
}
if len(filters.Kinds) > 0 {
placeholders := make([]string, 0, len(filters.Kinds))
for _, kind := range filters.Kinds {
placeholders = append(placeholders, "?")
args = append(args, string(kind))
}
conditions = append(conditions, "kind IN ("+strings.Join(placeholders, ", ")+")")
}
if len(filters.SourceTypes) > 0 {
placeholders := make([]string, 0, len(filters.SourceTypes))
for _, sourceType := range filters.SourceTypes {
placeholders = append(placeholders, "?")
args = append(args, string(sourceType))
}
conditions = append(conditions, sourceTypeExpr+" IN ("+strings.Join(placeholders, ", ")+")")
}
if len(filters.SourceAdapters) > 0 {
placeholders := make([]string, 0, len(filters.SourceAdapters))
for _, sourceAdapter := range filters.SourceAdapters {
placeholders = append(placeholders, "?")
args = append(args, string(sourceAdapter))
}
conditions = append(conditions, sourceAdapterExpr+" IN ("+strings.Join(placeholders, ", ")+")")
}
conditions, args = appendRecentChangeFilterConditions(conditions, args, filters, observedAtExpr, sourceTypeExpr, sourceAdapterExpr)
if len(conditions) > 0 {
query += "\n\t\tWHERE " + strings.Join(conditions, " AND ")
}
query += `
ORDER BY ` + observedAtExpr + ` DESC`
ORDER BY ` + observedAtExpr + ` DESC, id DESC`
if limit > 0 {
query += ` LIMIT ?`
args = append(args, limit)
@@ -3552,9 +3555,18 @@ func (m *MemoryStore) GetRecentChangesFiltered(canonicalID string, since time.Ti
continue
}
out = append(out, change)
if limit > 0 && len(out) >= limit {
break
}
// Late-arriving events need not be inserted in observation order.
// Apply the same stable order as SQLite before selecting the bounded page.
sort.Slice(out, func(i, j int) bool {
if out[i].ObservedAt.Equal(out[j].ObservedAt) {
return out[i].ID > out[j].ID
}
return out[i].ObservedAt.After(out[j].ObservedAt)
})
if limit > 0 && len(out) > limit {
out = out[:limit]
}
return out, nil
}
@@ -3676,6 +3688,29 @@ func buildRecentChangeCountQuery(canonicalIDs []string, since time.Time, filters
if len(canonicalIDs) > 0 {
conditions, args = appendRecentChangeResourceCondition(conditions, args, canonicalIDs, filters.IncludeRelated)
}
conditions, args = appendRecentChangeFilterConditions(conditions, args, filters, observedAtExpr, sourceTypeExpr, sourceAdapterExpr)
query += ` WHERE ` + strings.Join(conditions, " AND ")
return query, args
}
// appendRecentChangeFilterConditions keeps bounded reads and all aggregate
// counts on the same predicate. Metadata values are bound parameters, not SQL.
func appendRecentChangeFilterConditions(conditions []string, args []any, filters ResourceChangeFilters, observedAtExpr, sourceTypeExpr, sourceAdapterExpr string) ([]string, []any) {
if filters.ObservedBefore != nil {
conditions = append(conditions, observedAtExpr+" < ?")
args = append(args, *filters.ObservedBefore)
}
if len(filters.AlertIdentifiers) > 0 {
placeholders := make([]string, 0, len(filters.AlertIdentifiers))
for _, identifier := range filters.AlertIdentifiers {
placeholders = append(placeholders, "?")
args = append(args, identifier)
}
// Invalid legacy JSON cannot establish a matching alert identity. Reject
// non-string JSON values rather than coercing them into an identifier.
metadata := "CASE WHEN json_valid(metadata_json) THEN metadata_json ELSE '{}' END"
conditions = append(conditions, "(json_type("+metadata+", '$.alert_identifier') = 'text' AND "+resourceChangesAlertIdentifierExpr()+" IN ("+strings.Join(placeholders, ", ")+"))")
}
if len(filters.Kinds) > 0 {
placeholders := make([]string, 0, len(filters.Kinds))
for _, kind := range filters.Kinds {
@@ -3700,8 +3735,7 @@ func buildRecentChangeCountQuery(canonicalIDs []string, since time.Time, filters
}
conditions = append(conditions, sourceAdapterExpr+" IN ("+strings.Join(placeholders, ", ")+")")
}
query += ` WHERE ` + strings.Join(conditions, " AND ")
return query, args
return conditions, args
}
func appendRecentChangeResourceCondition(conditions []string, args []any, canonicalIDs []string, includeRelated bool) ([]string, []any) {
internal/unifiedresources/store_test.go
+2 -2
View File
@@ -1318,8 +1318,8 @@ func TestMemoryStoreResourceChangeFiltersIncludeRelatedResources(t *testing.T) {
if err != nil {
t.Fatalf("GetRecentChangesFiltered include related: %v", err)
}
if len(timeline) != 2 || timeline[0].ID != "mem-direct" || timeline[1].ID != "mem-related" {
t.Fatalf("relationship-aware memory timeline = %#v, want reverse insertion order direct plus related", timeline)
if len(timeline) != 2 || timeline[0].ID != "mem-related" || timeline[1].ID != "mem-direct" {
t.Fatalf("relationship-aware memory timeline = %#v, want newest observation first, as in the durable store", timeline)
}
count, err := store.CountRecentChangesFiltered("node:1", now.Add(-time.Hour), ResourceChangeFilters{IncludeRelated: true})
scripts/release_control/canonical_completion_guard_test.py
+1
View File
@@ -311,6 +311,7 @@ class CanonicalCompletionGuardTest(unittest.TestCase):
"internal/monitoring/issue1638_dns_cache_test.go",
"internal/monitoring/metric_window_provider_test.go",
"internal/monitoring/monitor_additional_test.go",
"internal/monitoring/monitor_alert_handling_test.go",
"internal/monitoring/monitor_alert_intent_test.go",
"internal/monitoring/monitor_alert_override_migration_test.go",
"internal/monitoring/monitor_backups_readstate_test.go",