diff --git a/docs/qualification/PATROL_ASSISTANT_CUSTOMER_JOURNEY.md b/docs/qualification/PATROL_ASSISTANT_CUSTOMER_JOURNEY.md index 9bd4f4c0c..d74ced14e 100644 --- a/docs/qualification/PATROL_ASSISTANT_CUSTOMER_JOURNEY.md +++ b/docs/qualification/PATROL_ASSISTANT_CUSTOMER_JOURNEY.md @@ -4,9 +4,12 @@ The customer job is: "Tell me what needs my attention, explain why, and help me deal with it without creating more work." Patrol owns the issue and investigation. Assistant explains that same issue and uses existing governed action contracts. -The local redesign and named qualification matrix are delivered. See +The earlier r34 local redesign and named qualification matrix landed. See [verified delivery and remaining gate](#verified-local-delivery-and-remaining-release-gate) -for the exact source and CI results. Production-wide readiness remains open. +for that source and CI evidence. The subsequent +[shared incident-history continuation](#continued-shared-incident-history-modernization-2026-09-07) +is unlanded and its required qualification remains incomplete. Production-wide +readiness also remains open. ## Active redesign plan, 2026-09-05 @@ -3615,3 +3618,621 @@ operator access and consent. Do not manufacture it from adoption telemetry or repeat maintainer-lab runs as a substitute. No production rollout or release was performed. The explicit subscription-provider refusal remains preserved without retry or bypass. + +## Continued shared incident-history modernization, 2026-09-07 + +The maintainer requested continuation after the named local redesign matrix +landed. The independent-environment question remains open. This slice addresses +the recorded incident-memory query residual and does not satisfy the wider +readiness gate by extending the same homelab evidence. + +The reviewed continuation baseline selected legacy memory shells before consulting +canonical history. This omits canonical-only alert occurrences and aliases whose +shell identifiers differ from the query. Projection readers also discard store +errors, scan an infrastructure-wide 256-event window before selecting an alert, +and can combine events from repeated occurrences of the same alert. The owning +fix belongs in the shared incident query/projection and canonical history query, +not in a single HTTP handler or model-written summary. + +Executable order and acceptance: + +1. Reproduce canonical-only, alias, projection-error, bounded-history and repeated + occurrence cases with actual memory/SQLite history stores. Record provenance + and observation/occurrence time expectations before changing projection. +2. Read canonical history first through its shared query contract, apply query + filters before limits, retain explicit bounds and unavailable evidence, and + preserve legacy notes as attributed history. Keep separate alert occurrences + separate. Do not create a diagnosis, recovery or current-health verdict from + missing records or a completed query. +3. Route resource listings, alert timeline reads and model context through that + shared result. Propagate read failures to API callers and disclose unavailable + context to the model. Preserve canonical event provenance and existing exact + alert/resource authority. No new diagnosis orchestration or provider retry + policy is authorized by this change. +4. Run targeted regressions and affected complete race suites on the worker. + Inspect the affected live Alerts timeline and Assistant journey with + Playwright after the final build at 1440x1000, 900x1000 and 390x1000. Cover + loading, empty, failed reads, repeated occurrences, expanded event details, + filters, dismissal/focus return, reload and deep scrolling. Inspect pixels as + well as DOM state. If model-visible context changes, qualify the affected + real-model history explanation through an authorized provider route and + preserve any unavailable-provider limit explicitly. +5. Update the owning contracts and this record with exact source, proofs and + remaining limits, then land the verified scope through the repository + workflow. Local proof and independent customer qualification stay distinct. + +Current state: the incident-history continuation has final r10 source acceptance +on integrated main `977afdd9559c0e9d5859f4c79bcc48e889381bba` plus the scoped +changes. Affected regressions and final browser interaction/pixel checks pass. +Three funded Astra explanations passed, with both earlier Gemini failures +retained. The actual note survives reload and its text reaches Assistant. +Repository commit checks and scoped landing remain pending. Both the old +candidate and matching unchanged base timed out in the full API race suite, +so no passing whole-suite receipt is claimed. The subscription-provider refusal +is preserved without retry or bypass, and wider readiness remains open. + +### Incident-history candidate r1: implementation and proof in progress + +The shared `IncidentStore.QueryIncidents` now reads canonical evidence before +selecting incident rows. Alert metadata filters and observation bounds are +applied before limits in SQLite and memory, with matching aggregate counts. +Both stores order by observation time and event ID. The earlier memory behavior +used reverse insertion order, which disagreed with SQLite for late observations. +The relationship-aware store and monitoring replay fixtures now assert the +canonical observation order while retaining every expected event. + +Explicit firing times partition repeated occurrences. The alert identifier and +firing time determine canonical-only occurrence identity. Repeated observations +of the same firing do not rename it, and two firings 500 milliseconds apart stay +separate. Exact-start selection chooses the closest occurrence within the +existing one-second shell matching tolerance. A capped query does not attach +ambiguous later events to an earlier saved occurrence. Missing starts stay +unknown. Source event IDs, observed/occurred times, actor, adapter, confidence, +metadata and related resource identities remain on the returned evidence. + +Canonical-only incidents can retain an operator note without copying canonical +lifecycle events into a second durable history. Saved notes and snapshots carry +their own source attribution. History aliases are read selectors only and do not +change action authority. API reads return service-unavailable on canonical read +failure, and monitoring skips reconciliation instead of fabricating a fallback +when that read fails. Assistant's incident summaries disclose unavailable or +truncated history and do not claim current health from historical lifecycle. + +User job: “What happened last time, and is this the same problem?” The live r34 +baseline was opened at `/alerts`, History, then a timeline at 1440 × 1000. This is +three navigation/control actions. The expanded timeline has twelve event filter +controls plus the Assistant handoff before its event content. The current slice +keeps the established timeline interaction and places new forensic provenance +behind one “Evidence details” disclosure. Missing timestamps are shown as +unavailable, and a truncated query receives an explicit notice. The inspected +issue #1782 supports current-evidence grounding and preservation of the governed +action path. It does not establish customer demand for extra timeline decoration. +No issue comment was sent. + +Current proof receipts under the worker's `patrol-incident-history` directory: + +- New filter regression first failed compilation because the filter fields did + not exist. It then passed against both real stores. The full store race suite + initially exposed the reverse-insertion expectation described above. +- Focused memory and incident HTTP race tests passed. The full affected suites + passed: memory 2.217 s, unified resources 72.877 s, alerting 3.691 s, + monitoring 212.854 s and runtime 24.119 s. An earlier monitoring run failed + its reverse-insertion expectation, before that expectation was corrected. +- Frontend type checking passed. Nine tests across the timeline panel and event + card passed, followed by the current frontend build. An initial dependency + symlink attempt failed Vite module resolution before running tests. A fresh + task-local `npm ci` resolved that environment failure. +- Cross-build verified 5,985 core and 60 enterprise source hashes with no + mismatches. Candidate `0.0.0-dev-pro+incident-history-r1` has SHA-256 + `1cd63e63b767a7a02334a89ead3b8f34941b1371fdd764646ae6f689df7455c0`. + +This is a candidate under qualification, not a completed continuation. The full +API race suite, final-build Playwright matrix, affected real-model explanation, +final review and repository landing are still pending at this checkpoint. The +previously authorized Gemini key had a US$5 limit and expiry on 2026-09-07. +No provider call has been made in this continuation, and the Claude subscription +refusal remains untouched. Independent customer-environment consent/evidence is +still unavailable, so the wider rollout gate remains open independently of this +local implementation work. + +### Incident-history r1 live failures and r2 corrections + +The r1 live timeline of disposable run `q-20260907-150920-a22e598e` failed +qualification. The alert-history row was resolved, but its timeline was open. +The canonical resource timeline contained both records: firing +`85ddd9f5-0368-5740-a041-eb68d2f131c6` at 15:10:01.025055Z and resolution +`f3c13163-33dd-5b10-9b37-ef8c8a1bc37b` at 15:10:30.96527Z. Two legacy shells +had the same alert identifier and explicit firing time. Projection assigned +firing to one and resolution to the other. The shared query now merges those +shells before projecting canonical events, retains a stable existing ID and +all local notes, and accepts either saved ID as a read selector. Regression +covers both aliases and the combined lifecycle. API snapshot fallback was not +used to mask this defect. + +A separate live backup timeline displayed `warning 0.0 >= 0.0`. The shared event +summary had invented an inequality from numeric fields without retaining the +source condition. Source messages now own fired-event descriptions. Where no +message exists, the fallback names only the alert type and level. Regression +covers source conditions, including a below-threshold comparison, and neutral +fallbacks for CPU and backup incidents. + +Read-failure inspection also exposed an error-visibility gap. A timeline refresh +could retain cached evidence while hiding its error, and resource history could +show an empty-state claim after its error toast disappeared. Both views now +retain visible read failure and a retry control. Resource-history state preserves +cached evidence separately from loading and failure, and clears failure only +when a new read succeeds. Tests exercise failed initial read, successful retry, +failed cached refresh and successful empty read. + +The r1 Playwright script also had a harness error: Escape cleared its search, +so its single-row expectation saw 138 timeline buttons. Removing that unintended +search reset repaired the script. This is separate from the reproduced product +failures above. No final r2 browser or real-model pass is claimed here. + +The saved non-secret provider-limit receipt confirms the authorized key expired +at `2026-09-07T17:23:20.631Z`. This was verified after that timestamp, without +calling the provider. Its initial US$5 balance is not a current spend balance. +The affected r2 real-model history explanation requires a valid funded route +and remains unperformed. The subscription-provider refusal is not an alternative +route and has not been retried. + +Remaining affected model qualification is a read-only Assistant turn from the +repaired historical lab incident. Ask: “Explain what happened in this occurrence, +what evidence records its resolution, and what we can and cannot conclude about +its current health. Do not change anything.” Verify the response against the +independent canonical firing/resolution records above, its saved note and exact +occurrence ID. The answer must distinguish observation from occurrence time, +recorded resolution from present health, and historical evidence from an action +outcome. No action proposal, execution or provider substitution is needed. A +second turn with unavailable history must say that the evidence could not be +read, without converting failure into no incidents or a healthy result. Retain +session IDs, provider/model, exact runtime hash, latency and actual spend. A +scripted fixture response does not satisfy this real-model check. + +The maintainer supplied a replacement OpenRouter key in this continuation and +authorized its use. The key metadata endpoint reports a valid paid key with +zero initial usage and no provider-side limit or expiry. The prior US$5 +qualification ceiling is retained as a task budget, not claimed as a provider +enforcement boundary. Only the two read-only history turns above are planned. +Authenticated local settings confirmed Patrol disabled, control level read-only, +and `openrouter:google/gemini-3.8-flash` selected for chat and Patrol before the +credential update. The update succeeded through the ordinary settings endpoint. +An initial helper request omitted its CSRF header and was rejected before the +update. The corrected request supplied the normal session CSRF token. No +subscription request or model call was made during configuration. + +Narrow exploration reproduced an additional handoff defect at 390 × 1000. +The incident drawer remained above Assistant after Discuss, hiding the +continuation. The shared incident handoff now exposes the same explicit callback +pattern used by finding handoffs. Mobile timeline and resource-history drawers +close after the incident context has been handed to Assistant. Desktop inline +panels remain governed by their existing controls. Mounted tests cover both +mobile source views and preserve the same incident ID, status and read-only +context. Final browser qualification must repeat the drawer transition and +inspect the reachable Assistant composer, dismissal and focus. + +The first callback attempt still failed because the app's shared blocking-dialog +guard correctly closed Assistant while the source drawer was mounted. The final +transition captures the incident context, closes the source drawer, then opens +Assistant in the next microtask, matching the existing command-palette handoff. +The mounted regression checks that the dialog stack is no longer blocking when +Assistant opens. Narrow exploration then passed without a model request. This +exploration used the r1 backend and does not qualify its known split lifecycle. + +### Candidate r2 qualification checkpoint + +Final r2 source passed memory and unified-resource race suites in 2.225 s and +72.169 s, plus focused incident alerting/API/monitoring race checks in 1.203 s, +2.576 s and 1.770 s. Frontend type checking and 62 tests in eight files passed, +followed by the frontend build and verification of 5,987 core and 60 enterprise +source hashes. Binary SHA-256: +`7c82377ca4cc58f0f3ab3e3d64bd2edac7692309596416ce0f70e836d731726a`. +The local built runtime confirmed the previously split lab occurrence as +resolved with both canonical lifecycle records. + +The broad r1 API race run failed. It reported +`TestServerInfoEndpointReportsDevelopment` expecting development mode, then hit +its one-hour deadline in `TestAuthenticatedEndpointsRequireToken`, which had +spent 39m6s in the running test. The timeout stack is in mock unified-resource +fixture expansion during per-resource metric-window evaluation and router +construction. This is not a passing broad API receipt. The worker source export +has no Git directory. A fresh exact-base Git checkout passed the same two tests +in isolation in 62.298 s after its required frontend embed build. The initial +baseline attempt lacked generated embed assets and failed setup. Candidate +comparison in that Git checkout remains pending at this checkpoint. Neither +isolated result can establish that the full suite is free of shared-state or +performance problems. + +R2 browser qualification also found that Assistant's composer registration and +focus happened only at mount. Reopening from a mobile drawer left focus on the +underlying alert search, so Escape cleared the search rather than dismissing +Assistant. Candidate r3 moves registration and focus into Assistant's shared +open lifecycle and clears the registration on close. A mounted regression +starts closed and checks two separate composer instances across reopening. +Final build, complete browser repetition and both funded model explanations +remain pending. No full-task completion is claimed. + +### Candidate r3 browser failure and r4 correction + +The isolated candidate API comparison passed the same two tests in 63.937 s. +The complete Assistant component file passed 211 tests. These receipts do not +replace the failed broad API race run above. + +R3 was built and deployed locally with SHA-256 +`f4ec960416cff6fa5c779119689d04f2842e38aee6494e23a4f1649d4f7e1d01`. +The repeated live matrix still failed narrow Escape dismissal. Browser inspection +identified a second focus owner in the mobile alert list. Its normal drawer-close +callback restores focus twice through animation frames, overriding the shared +Assistant open lifecycle. R4 gives the source drawer an explicit handoff callback +that closes the investigation without scheduling return focus. Ordinary drawer +dismissal retains its existing focus restoration. The regression now mounts the +parent mobile list and waits through both animation frames before checking the +destination focus. Its first run exposed an incomplete test fixture, which omitted +the required alert type. That fixture was corrected before repeating qualification. + +Opening Assistant also sends ordinary provider-readiness requests. The browser +receipt's chat/session submission counter excludes those probes and must not be +interpreted as a count of all provider traffic. No history explanation has been +submitted at this checkpoint. The subscription refusal remains unchanged. + +R4 passed 218 frontend tests across the full Assistant file and both mobile +investigation files, type checking and the frontend build. Runtime SHA-256: +`bad20d2dee980abf826809755689538ae8c1a5ea113656619e7d5d92741af366`. +The complete live `/alerts` history matrix passed at 1440, 900 and 390 × 1000, +including both Assistant handoffs, Escape, evidence expansion, cached read +failure/retry and reload. Controlled-state scripts initially selected notification +toasts or hidden desktop copies of mobile elements. Those locator failures were +corrected without changing product source. + +The first funded history explanation was a failed qualification, despite correctly +distinguishing alert closure from current health and verified remediation. +Session `efdc8896-747c-4516-8b43-d1e697495c00`, Gemini 3.8 Flash through OpenRouter, +HTTP 200, 15.806 s, 28,070 input and 1,622 output tokens, recorded session cost +US$0.027135. The model read inventory and canonical history, then asserted no governed action record existed without reading actions. The +initial network-removal criticism did not account for automatically attached +related-resource history and is corrected in the r6 review below. R5 corrects the shared history tool's existing +read-scope description: resolution records closure rather than workload recovery, +related-resource IDs identify relationships rather than additional event targets, +and this read does not query action records. These are factual source boundaries, +not deterministic diagnosis or response-scoring rules. The failed transcript is +retained locally. A fresh-session rerun is required before the unavailable-history +turn. The additional turn remains within the existing US$5 task ceiling. + +Provider-readiness inspection confirms that this OpenRouter route performs an +authenticated key-metadata GET, not a model completion +(`internal/ai/providers/openai.go`, `TestConnection` / `testOpenRouterKey`). +The first funded session increased the local one-day estimated usage total by +US$0.02794875, including US$0.00081375 for its automatic session title. These are +Pulse pricing estimates. Actual provider billing and remaining balance have not +been independently reconciled after that turn. + +### R5 outcome and R6 qualification plan + +R5 tool regression passed in 1.640 s and the final source hashes matched. +Runtime SHA-256: +`f43652b8c939238b5dcb7848e68c99d72def8ade316234010d6767784e99447f`. +The live three-viewport history matrix and controlled desktop/mobile loading, +initial failure, retry, empty, partial and unknown-time cases passed again. +The controlled cases substitute HTTP responses, not a real database outage. + +The second funded Gemini history explanation also failed factual qualification. +Session `c2f40f27-6d7c-40f6-8e3a-549ac5232071`, HTTP 200, 18.406 s, +40,235 input and 1,630 output tokens, estimated session cost US$0.03628875. +Including its automatic title, the local estimated usage increase was +US$0.037116. It stopped claiming the related network was removed, but still +asserted no action plan or verified execution existed without reading action +records. Both failed attempts remain in the denominator. Their total estimated +increment including titles is US$0.06506475. Neither is a passing history +explanation or evidence of population-wide effectiveness. + +The request inspection exposed a separate shared projection defect: saved shells +retained a legacy resource ID and stale risk fields despite canonical alert +evidence. R6 gives explicit canonical alert event kinds ownership of resource +identity, type, severity and message, while retaining the saved incident ID and +notes. A related command's execution host cannot retarget the incident. The new +single regression passed locally in 0.495 s after correcting its fixture's +collection type. Full memory regression and the rebuilt runtime remain required. + +The full candidate API race run in the disposable Git checkout failed at its +30-minute deadline, with `TestAuthenticatedEndpointsRequireToken` running 6m40s. +The package elapsed time was 1802.152 s. Its stack traverses mock fixture cloning, +unified snapshot construction, per-resource metric-window evaluation and router +construction. Those stack functions are not modified by this incident query +slice. This is an unresolved broad performance/qualification limit, not a pass +and not proof that the entire failure is unrelated to all candidate effects. +No further broad rerun was queued ahead of the release rehearsal. + +The next two history checks use the exact funded route +`openrouter:openai/gpt-6-astra`, in fresh sessions with Patrol disabled and control +read-only. The provider's public Models API lists function calling and the route's +prices. [Official model documentation](https://developers.openai.com/api/docs/models/gpt-6-astra) +identifies it as a model for complex reasoning and end-to-end work. The route was +selected as a stronger-model qualification, not to erase Gemini's failures or +claim that its earlier seven-case lab matrix transfers to this model. + +The exact OpenRouter price row was reviewed on 2026-09-07: US$10 input / US$50 +output per million tokens below its `min_prompt_tokens=272000` override, and +US$20 / US$75 at that override. Cache discounts are omitted for conservative +budget estimation. Unreviewed variants remain unknown. A targeted boundary and +alias regression passed locally in 0.364 s. These are estimated list-price costs, +not actual billing. The aggregate qualification ceiling remains US$5. The +subscription-provider refusal is unchanged. No GPT-6 Astra explanation has been +submitted at this checkpoint. + +The local 30-day estimated budget was temporarily set to US$5 before any Astra +explanation, with a pre-existing estimated total of US$1.45530525. This is stricter +than a separate US$5 allowance for this continuation. The existing budget check +runs between model turns and cannot cap an already in-flight provider charge. +The prior budget was zero and the prior chat route was Gemini 3.8 Flash. Restore +those temporary qualification settings after the two checks, retaining the +user-authorized replacement credential in encrypted runtime configuration. + +A matching full API race baseline comparison is queued after the candidate r6 +proof/build, behind the release rehearsal. It uses unchanged base +`1a822d164b3119724b01615da217618edf7e37e5`, a disposable Git checkout, the same +Go 1.26.8 toolchain, non-root execution, GOMAXPROCS=4 and 30-minute timeout. +This is a pending diagnostic comparison, not a passing receipt or a relaxation +of the candidate failure. + +### R6 final source qualification in progress + +The release rehearsal released the shared worker at 23:28:46 UTC. The normally +queued r6 job completed with exit 0. The full memory race suite passed in +2.189 s and the full cost race suite in 1.148 s. Focused alerting and monitoring +race checks passed in 1.222 s and 1.496 s. The first filtered memory invocation +reported no tests, so it is not counted as proof. The subsequent full memory +invocation is the passing receipt. All 5,987 core and 60 enterprise source +hashes matched. The built r6 runtime SHA-256 is +`3c875fcfe1826c4ef50fe6f116d49269c1111dc569b09a0e80175d13f9421f92`. +It was hash-checked and installed on the local development stack. Final browser, +funded-model and saved-note qualification remains pending at this checkpoint. + +The r6 live matrix passed at 1440, 900 and 390 × 1000 with the canonical resource +ID and type in the incident and Assistant handoff. Controlled desktop/mobile +loading, initial error, retry to empty, partial-history and unknown-time cases +also passed. Pixels confirmed visible evidence and errors, reachable handoffs +and the mobile composer. This proof preceded the resize defect described below. + +Both funded Astra turns passed factual review. History session +`1b881e3f-57b1-4fcd-be64-ee8e46885394` completed HTTP 200 in 18.507 s, +with reported 8,054 input and 456 output tokens, estimated session cost +US$0.10334 and total usage increase US$0.11216 including its title. It explained +the recorded closure, kept present health and remediation unknown, and said an +action record was not supplied rather than asserting none existed. Its network +removal statement is supported by separate canonical record +`8539f198-82b0-4720-81c1-abdf871e60b1`, targeted at +`docker-network-7f19c579093f3cd0`. The shared server handoff automatically reads +related resource history with a five-record limit. This record was within that +input. Tool calls alone are not the complete model context. The earlier r4 +network criticism is therefore withdrawn as an unsupported qualification +finding. Both Gemini attempts still fail on their independent unsupported +claims about action-record absence. + +Unavailable-history session `8365f1d5-e20b-4317-b297-3200f37ad67c` completed +HTTP 200 in 8.957 s, reported 6,843 input and 100 output tokens, estimated +session cost US$0.07343 and total increase US$0.07883 including its title. It +stated only that canonical evidence could not be read and left incidents, +health, actions, cause and transience unknown. Neither Astra turn called tools +or changed infrastructure. The unavailable case uses the actual formatter's +error text in controlled handoff context, not a live database outage. Total +estimated spend across all four turns and titles is US$0.25605475. Actual +provider billing remains unreconciled. Two failed explanations remain in the +four-attempt denominator. This is not a population success-rate estimate. + +Some earlier saved SSE text had mojibake because the browser receipt helper used +response.text decoding. Actual pixels rendered punctuation correctly. Subsequent +capture explicitly decodes response.body as UTF-8. No output-rewriting product +patch was added. After both turns, the temporary chat model was restored to +Gemini 3.8 Flash and budget to zero, with Patrol still disabled and read-only. +Zero budget is omitted by the settings JSON, which initially confused the +restoration assertion. A subsequent read verified the restored settings. + +### R7 responsive investigation correction in progress + +One actual note was saved to the owned historical lab occurrence and survived +read-error retry and desktop reload. Shrinking that open desktop timeline to +390 pixels exposed a product defect: its card said Hide, but the timeline was +hidden and no mobile drawer existed. Two read-only attempts confirmed it without +writing another note. The local mobile drawer selection had diverged from the +shared expansion state. The mobile layout now projects existing shared resource +or timeline selection when entering the phone layout and leaves that state +intact when returning to desktop. Its action label reflects the actual drawer, +including when other desktop rows remain expanded. Both timeline and resource +resize regressions pass in the seven-test mobile-list suite. R7 build and full +affected browser repetition remain required. The responsive correction alone left the model input unchanged. A subsequent +review found that the handoff discarded operator note text, retaining only its +"Note added" summary. R7 now preserves the attributed note in shared incident +model formatting and the Assistant handoff, while excluding unrelated details +and raw command output. The new memory regression passed in 0.437 s and the +combined handoff/mobile-list checks passed 11 tests in 1.58 s. One additional +funded history-and-note explanation is required after the r7 build, within the +existing US$5 ceiling. The two r6 outcomes remain recorded without replacing +that final changed-context check. + +### Integration with updated main, 2026-09-07 + +Main advanced from `1a822d164b3119724b01615da217618edf7e37e5` to +`203b50a46ece5e911d6f5f6ea46bbeef218aeab5` during qualification. The candidate +was saved to a task-local archive and explicit-path Git stash, main was +fast-forwarded, and the work was reapplied. Upstream lifecycle replay identity, +unchanged-checkpoint suppression and their regressions are retained. The +superseded legacy projection helper remains removed in favour of QueryIncidents. +Owning contract additions from both changes are preserved. + +The upstream cross-resource occurrence regression failed on the initial merge +in both the two-minute and 500-millisecond cases. An unrelated resource's shell +was shortening the selected alert occurrence. The shared query now resolves +resource history identities, separates saved-shell grouping by resource, adopts +the exact canonical firing target before assigning subsequent evidence, and +excludes unrelated-resource boundaries for alert events. The five focused +integrated regression groups passed in 0.491 s. Full integrated source proof and +r7 qualification remain required. The old-base API comparison only diagnoses the +earlier full-suite timeout and cannot qualify the integrated candidate. + +The matching clean old-base full API race comparison also failed its 30-minute +bound. The checkout was exactly `1a822d164b3119724b01615da217618edf7e37e5` +and stayed clean. It ran as the normal worker user through the allocator with +Go 1.26.8 and GOMAXPROCS=4. The package failed after 1802.195 s, with +`TestRecoveryEndpointRequiresDirectLoopback` running for 7 m 46 s at timeout. +The earlier candidate failed after 1802.152 s with a different active test, +`TestAuthenticatedEndpointsRequireToken`. The worker log is +`/opt/pulse-release-worker/patrol-incident-history/base-api-proof.log`. +This establishes that the unchanged old base also cannot finish that broad +race command within the same bound. It does not attribute every candidate +effect to the base, and neither run is a passing whole-API-suite receipt. +The integrated candidate still requires its scoped proof and repository checks. + +### R7 integrated source qualification, 2026-09-08 + +The normal worker r7 command completed with exit 0. All 5,990 core and 60 +enterprise source hashes matched. Complete affected race suites passed: +incident memory 2.252 s, unified resources 72.880 s, alerting 3.598 s and +monitoring 206.311 s. Frontend type checking and all 51 tests in the four +handoff/mobile investigation files passed. The frontend build completed in +27.50 s. The resulting development runtime is +`0.0.0-dev-pro+incident-history-r7`, with core base +`203b50a46ece5e911d6f5f6ea46bbeef218aeab5` plus the scoped changes and enterprise +`b9fa43dcf0ee743652b20d1a866da8ca9c82cdbd`. Its SHA-256 is +`341b9d2e3c685d0280a321453546a5a02321d8ba9ce923f99a51a961da9c8340`. +The same binary and frontend assets were installed on the local development +stack. Final browser, changed-context model and repository checks remain +required at this checkpoint. + +The remaining r7 scoped race proof also passed: complete shared AI 23.613 s, +tools 62.645 s and cost 1.148 s, then the selected incident API handlers in +2.015 s. It ran from the exact integrated Git checkout under the normal worker +allocator. These passes do not replace the failed broad API runs. + +R7 live history and controlled-state Playwright scripts passed at their planned +widths, including composer focus after both handoffs. Pixel inspection then +found the saved operator note clipped on the 900-pixel desktop timeline because +event text inherited the table's no-wrap style. R8 corrects the shared event +card's wrapping and preserves note line breaks. This is a real browser failure, +not a passing visual receipt. The affected matrix must repeat on r8. The initial +resize command also failed in its invocation wrapper before the script ran, +while node's syntax check of the actual script passed. That invocation provides +no product evidence. No additional operator note was written. +The direct resize invocation then reached the real r7 UI and failed the new +saved-note overflow assertion, independently reproducing the clipped text. + +R8 type checking, 51 frontend tests and the 26.50-second build passed. Its +runtime hash is `90d912e8cea77ffd108be0046430cec1e90d4403900861fcd8c6aa1ae8265152`. +Live and controlled-state scripts passed. The resize matrix retained the note, +draft and expanded events across 390/900/767/768-pixel transitions, but closing +the resource drawer with Escape cleared the background search and lost focus. +The first assertion reported 22 Resource buttons. A second read-only capture +confirmed the search had become empty and focus moved to body, so this is a +product defect rather than merely a selector ambiguity. The shared type-to-search +registry treated an inert modal background as eligible for keyboard shortcuts. +Two new regressions failed before the fix. R9 excludes inert inputs from both +visible and prepared search targets using the shared dialog's existing DOM +ownership boundary. No new dialog lifecycle or lane-local keyboard handler is +added. The final browser and funded model check remain pending. + +The corrected shared search, Dialog and SearchInput suites passed 36 tests in +1.54 s locally. The normal worker repeated those 36 tests, type checking and +the frontend build (28.14 s), then built r9 with exit 0 and matching 5,990/60 +source hashes. The installed runtime SHA-256 is +`7bad38f8da70a89b56b6693f41af5fa3176c4375bf10b5b1f4958aaa8fc929ba`. +The shared keyboard hook now has explicit frontend-primitives registry ownership. +Three existing presentation expectation files were updated after this build +and passed 118 tests in 1.36 s. Those later edits are test-only and do not change +the built runtime source. The integrated r7 backend proofs remain applicable +because r8 and r9 changed only frontend presentation and keyboard ownership. + +All three r9 browser scripts passed, including note wrapping, retained draft, +saved-note reload, modal typing, retained search, Escape and focus return at +1440/900/390 and the 767/768 breakpoint. The subsequent funded history-and-note +turn passed factual review. Session `81c92125-cdc3-4611-931a-687e6548bf65` +completed HTTP 200 in 18.668 s with reported 8,130 input and 456 output tokens. +Its estimated session cost was US$0.1041, plus US$0.0088 for the title, for +US$0.1129 total. Five continuation turns and titles now total US$0.36895475 +estimated, with three passing explanations and both Gemini failures retained. +Actual provider billing is not reconciled, and this is not a population rate. +The answer attributed the historical closure, retained current-health and +action-outcome uncertainty, and correctly described the operator note as a +display/persistence check with no infrastructure change or verified recovery. +It called no tools and performed no infrastructure action. Temporary model and +budget settings were restored to Gemini 3.8 Flash and zero, with Patrol disabled +and read-only. The subscription-provider refusal remains untouched. + +The same model run exposed a further responsive defect: shrinking a desktop +conversation reopened its source history drawer above Assistant. The earlier +matrix had resized the source before handoff, not the open destination. Two +regressions reproduced this for timeline and resource history. R10 keeps the +existing Assistant destination active when projecting history state on a layout +change. It adds no model-visible context and needs no additional funded turn. +The affected browser matrix, including resizing the open conversation and +reading the saved response, must pass before source-bound acceptance and landing. + +### Final incident-history source acceptance, r10 + +R10 passed 43 affected frontend tests and the 27.07-second worker build with +matching 5,990 core and 60 enterprise source hashes. Its installed runtime hash +is `2e1958d0856c812d22870144cf228509f2a4a3696258a533d34802442f500edb`. +All four final functional browser scripts passed, followed by actual pixel +inspection. `/alerts` was exercised at 1440, 900 and 390 × 1000, with source +and destination resize transitions also covering 767 and 768 pixels. The saved +funded response was resumed by exact session ID and read at every width without +another model submission. Assistant remains visible above the retained source, +the composer is usable, and the operator-note explanation is readable. + +The final interaction matrix includes canonical occurrence selection, native +Evidence details with keyboard focus and activation, All/None filters, note +draft enable/clear, actual saved-note reload, source/destination resizing, +expanded resource events, cached and initial read errors, Retry/Refresh, empty +and partial history, unknown timestamps, modal typing, Escape and return focus. +Long notes wrap at the intermediate desktop width. Error toasts are not the only +failure indication. Controlled HTTP variants qualify rendering, not a real +database outage. Final acceptance is bound to exact frontend bytes in +`frontend-modern/browser-verification.json`. Earlier visual and factual failures +remain recorded above. The final helpers are executable at +`/Volumes/Development/pulse/tmp/patrol-incident-history/verify-live.mjs`, +`verify-states.mjs`, `verify-resize.mjs` and `verify-saved-explanation.mjs`. +Their receipt directories are `browser`, `browser-states`, `browser-resize` and +`browser-saved-explanation` under that task directory. + +Repository commit checks and scoped landing are the remaining local delivery +steps. The broader customer-outcome gap stays open. These receipts do not +qualify independent customer environments, unattended autonomy, backup restore, +population false-alarm/miss rates or latency SLOs. The failed broad API race +runs remain an explicit test limit, despite passing affected package and handler +proof. The final history explanation qualifies the funded Astra route and does +not erase the two Gemini failures or transfer the earlier named lab matrix to +another model. + +Commit preflight initially rejected missing dependent agent-lifecycle and +storage-recovery contract updates and missing explicit proof mappings. Those +contracts now state the historical-evidence boundary, and the registry maps the +actual incident query and lifecycle regression files to their owning paths. +Additional adapter assertions verify filtered-query forwarding, alias identities +and unavailable reads. The targeted query tests passed in 0.573 s. These are +test/governance changes only, leaving the r10 runtime and browser hashes intact. +The staged canonical completion and registry guards then passed without a +contract-neutral bypass. + +The final upstream integration advances the base to +`977afdd9559c0e9d5859f4c79bcc48e889381bba`. It adds checkpoint and guest-memory +tests plus contract text, with no runtime change. Comparison against the 5,990 +source manifest found only three test files and the browser receipt different. +All 14 accepted frontend content hashes remain identical. The newly integrated +checkpoint assertion initially compared fresh query timestamps as durable +incident state and failed. It now compares stable projection identity/events +separately from refreshed query bounds, while preserving coverage semantics and +the no-checkpoint-replacement assertion. The targeted occurrence test passed +in 0.450 s after this test-only integration. + +The first full worker hook stopped at missing private repository evidence roots. +The existing filesystem-proof copies of pulse-pro and pulse-mobile and the +current task enterprise source export supply those roots through the supported +repository-root overrides. Status audit then reported no errors or warnings. +This repairs the proof environment and does not suppress the audit. + +Registry audit also requires sibling Git checkouts. Initial sibling links made +that audit pass, but the cross-repository absolute-path helper correctly rejected +paths resolved outside its workspace. Real disposable clones of the existing +private filesystem-proof checkouts replace those links, with no audit bypass. +Its audit and contract audit pass. The full hook then caught one expected-file +fixture missing the newly registered monitoring regression. Updating that fixture +to include the actual proof file preserves the guard's exact mapping assertion. +All 130 completion-helper tests passed in 2.645 s. This is test-only scope. diff --git a/docs/release-control/v6/internal/status.json b/docs/release-control/v6/internal/status.json index bed162da1..f21c648f0 100644 --- a/docs/release-control/v6/internal/status.json +++ b/docs/release-control/v6/internal/status.json @@ -10201,7 +10201,7 @@ }, { "id": "patrol-assistant-customer-outcome-qualification", - "summary": "The wider readiness gate remains open. The executable contract, honest baseline, exact runs and residuals are in docs/qualification/PATROL_ASSISTANT_CUSTOMER_JOURNEY.md. Earlier evidence/history/risk and typed-runner changes landed through core PR1928/1929/1934/1935/1951/1955 and enterprise PR22. PR1957 auto-merged as a66b8e11d7ca9ed5660ffd8725a1461661ca2fdf despite its recorded benchmark failure, which is not counted as a pass. The current slice makes canonical planning acceptance/refusal available inside the model turn, retains accepted actions across provider failure, enforces persisted actor/request identity, preserves complete approval/risk/outcome context and removes proposal/tool-count diagnosis proxies. Shared disk unknowns, streamed whitespace and replayed resolution timestamps are corrected. Affected package race suites and final frontend checks pass. Source-bound r28 Gemini qualification performed healthy, unhealthy, dependency, missing-access, storage-capacity, approved and rejected cases, including independently observed Docker recovery and VM110 start/stop. Earlier semantic failures remain recorded. Current r34 runtime passes final linked-history and Assistant Playwright at 1440, 900 and 390 widths, including fresh approved explanation, saved missing-access/VM/rejected/storage continuation, deep evidence, exact action states and scrolling. The local named implementation matrix is performed, with source-equivalence and exact limits recorded. Verified local delivery landed through core PR1960 (a42e3800d9a1b5ea185823469e891f44bc698c2c) and enterprise PR23 (b9fa43dcf0ee743652b20d1a866da8ca9c82cdbd). Their final CI and source-bound local proofs pass. Production-wide readiness remains a separate release_gate. Temporary collectors, runners, tokens and fixtures were cleaned and production processes preserved. Storage capacity does not qualify backup/restore. Legacy incident-memory/compatibility residuals, all-filter coverage, unattended autonomy and latency SLOs remain unqualified. Adoption of 127 paid installations, 71 Patrol-enabled installations and 23 Assistant users does not measure effectiveness. Fourteen verified resolutions from one installation do not establish population useful-diagnosis, false-alarm, missed-problem or latency rates, which remain unknown. Independent volunteered customer environments are the separate wider rollout gate. Preserve the explicit subscription-provider refusal without retry or bypass. Do not close this gap or candidate while wider readiness evidence is missing.", + "summary": "The wider readiness gate remains open. The executable contract, honest baseline, exact runs and residuals are in docs/qualification/PATROL_ASSISTANT_CUSTOMER_JOURNEY.md. Earlier evidence/history/risk and typed-runner changes landed through core PR1928/1929/1934/1935/1951/1955 and enterprise PR22. PR1957 auto-merged as a66b8e11d7ca9ed5660ffd8725a1461661ca2fdf despite its recorded benchmark failure, which is not counted as a pass. The current slice makes canonical planning acceptance/refusal available inside the model turn, retains accepted actions across provider failure, enforces persisted actor/request identity, preserves complete approval/risk/outcome context and removes proposal/tool-count diagnosis proxies. Shared disk unknowns, streamed whitespace and replayed resolution timestamps are corrected. Affected package race suites and final frontend checks pass. Source-bound r28 Gemini qualification performed healthy, unhealthy, dependency, missing-access, storage-capacity, approved and rejected cases, including independently observed Docker recovery and VM110 start/stop. Earlier semantic failures remain recorded. Current r34 runtime passes final linked-history and Assistant Playwright at 1440, 900 and 390 widths, including fresh approved explanation, saved missing-access/VM/rejected/storage continuation, deep evidence, exact action states and scrolling. The local named implementation matrix is performed, with source-equivalence and exact limits recorded. Verified local delivery landed through core PR1960 (a42e3800d9a1b5ea185823469e891f44bc698c2c) and enterprise PR23 (b9fa43dcf0ee743652b20d1a866da8ca9c82cdbd). Their final CI and source-bound local proofs pass. Production-wide readiness remains a separate release_gate. Temporary collectors, runners, tokens and fixtures were cleaned and production processes preserved. Storage capacity does not qualify backup/restore. Legacy incident-memory/compatibility residuals, all-filter coverage, unattended autonomy and latency SLOs remain unqualified. Adoption of 127 paid installations, 71 Patrol-enabled installations and 23 Assistant users does not measure effectiveness. Fourteen verified resolutions from one installation do not establish population useful-diagnosis, false-alarm, missed-problem or latency rates, which remain unknown. Independent volunteered customer environments are the separate wider rollout gate. Preserve the explicit subscription-provider refusal without retry or bypass. Do not close this gap or candidate while wider readiness evidence is missing. The incident-history continuation implements canonical filtered occurrence queries, provenance, bounded history and explicit read errors, then repairs duplicate saved shells and mobile Assistant focus ownership found in live inspection. R4 browser checks pass, but its funded history explanation made unsupported action-record claims. Its initial related-resource criticism was withdrawn after auditing automatically attached history. R5 narrowed the shared history tool read scope, but its second Gemini explanation still asserted absent action records without reading actions. R6 repairs canonical alert target/risk ownership over legacy shells and adds reviewed cost estimates for funded GPT-6 Astra history qualification. Both funded Astra history and unavailable-read explanations passed factual review, with both Gemini failures retained and exact costs and context limits recorded. The actual note survived desktop reload, but resizing exposed a hidden mobile timeline. R7 projects shared expansion state into the phone drawer, and seven mobile-list tests pass. R7 also retains operator note text in the shared model context. Integrated main preserves upstream replay and checkpoint fixes, with a cross-resource occurrence-boundary regression corrected in the shared query. Final r10 build and source-bound browser acceptance pass, including saved notes, long-text wrapping, modal search ownership and keeping Assistant visible when resizing the destination. The additional funded history-and-note explanation passed. Five turns and titles cost US$0.36895475 estimated, with both earlier Gemini failures retained. Runtime defaults were restored and the subscription refusal preserved. Repository commit checks and scoped landing remain pending. Both the earlier candidate and its exact unchanged base timed out in the full API race suite at 30 minutes, with different active tests. Neither is a passing whole-suite receipt, and the comparison does not qualify the integrated candidate. These continuation requirements remain open.", "owner": "project-owner", "status": "planned", "recorded_at": "2026-09-05", @@ -10371,7 +10371,7 @@ { "id": "patrol-assistant-customer-outcomes", "name": "Patrol and Assistant Customer Outcomes", - "summary": "Simplify Patrol around model-owned investigation and one issue-to-verified-outcome journey shared with Assistant. Preserve canonical evidence and remove proposal-as-proof and proxy-driven diagnostic policy while retaining deterministic authority and independent verification. Execute the ordered redesign plan in docs/qualification/PATROL_ASSISTANT_CUSTOMER_JOURNEY.md, qualify unhealthy service diagnosis, backup or capacity risk and supported VM/LXC actions with negative controls, and record real model quality, latency and cost limits. Independent volunteered Pro qualification remains required for wider readiness. Local implementation and the named qualification matrix are delivered through core PR1960 and enterprise PR23. The candidate remains open for the independent-environment release gate.", + "summary": "Simplify Patrol around model-owned investigation and one issue-to-verified-outcome journey shared with Assistant. Preserve canonical evidence and remove proposal-as-proof and proxy-driven diagnostic policy while retaining deterministic authority and independent verification. Execute the ordered redesign plan in docs/qualification/PATROL_ASSISTANT_CUSTOMER_JOURNEY.md, qualify unhealthy service diagnosis, backup or capacity risk and supported VM/LXC actions with negative controls, and record real model quality, latency and cost limits. Independent volunteered Pro qualification remains required for wider readiness. Local implementation and the named qualification matrix are delivered through core PR1960 and enterprise PR23. The candidate remains open for the independent-environment release gate. The subsequent canonical incident-history continuation is still in qualification, including a recorded failed model explanation. Do not treat that continuation as delivered until its final evidence and landing are recorded.", "status": "proposed", "recorded_at": "2026-09-05", "target_id": "v6-product-lane-expansion", @@ -10403,16 +10403,16 @@ ], "work_claims": [ { - "id": "release-snapshot-identity-coverage-gap-release-train-exact-candidate-promotion", - "agent_id": "release-snapshot-identity", - "summary": "Implement immutable release snapshots in release workflow, validation and promotion policy", + "id": "patrol-incident-history-continuation-coverage-gap-patrol-assistant-customer-outcome-qualification", + "agent_id": "patrol-incident-history-continuation", + "summary": "Canonical incident history and final linked Assistant qualification", "target_id": "v6-product-lane-expansion", - "claimed_at": "2026-09-07T17:58:01Z", - "heartbeat_at": "2026-09-07T17:58:01Z", - "expires_at": "2026-09-07T19:58:01Z", + "claimed_at": "2026-09-07T20:00:54Z", + "heartbeat_at": "2026-09-08T00:40:59Z", + "expires_at": "2026-09-08T02:40:59Z", "work_item": { "kind": "coverage-gap", - "id": "release-train-exact-candidate-promotion" + "id": "patrol-assistant-customer-outcome-qualification" } } ], diff --git a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md b/docs/release-control/v6/internal/subsystems/agent-lifecycle.md index bf292515c..a8ce06e35 100644 --- a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md +++ b/docs/release-control/v6/internal/subsystems/agent-lifecycle.md @@ -15,6 +15,11 @@ ## Purpose +Historical incident reads preserve canonical event targets and treat resource +aliases as read selectors. They do not grant command authority to an alias or +infer present agent liveness from an old alert closure. A failed canonical read +is exposed as unavailable, rather than replaced with a synthetic healthy state. + Delivery-log diagnostic text is a notification-attempt projection, not an agent admission or liveness verdict. Masking embedded destination credentials preserves failure context without changing agent identities, credentials, session replacement or removal policy. Consumers must use the retained failureClass and outcome as delivery evidence only; a transport error does not establish that the monitored agent is offline. ### Canonical Patrol and Assistant continuation, 2026-09-07 diff --git a/docs/release-control/v6/internal/subsystems/ai-runtime.md b/docs/release-control/v6/internal/subsystems/ai-runtime.md index 00c7e0f8d..60ac40224 100644 --- a/docs/release-control/v6/internal/subsystems/ai-runtime.md +++ b/docs/release-control/v6/internal/subsystems/ai-runtime.md @@ -42,22 +42,20 @@ make acknowledgement event replay idempotent, or prove aggregate write-byte reductions or recipient delivery. An evicted occurrence can still be recreated by replay; exact identity is guaranteed only while its shell is retained. -Canonical timeline reads for retained incident shells select events from the -shell's exact opening time up to (but not including) the next retained opening -for the same alert/resource. A newer firing must not reopen an older resolved -incident, and a subsecond recurrence must not inherit its predecessor's -resolution or acknowledgement. Local analysis annotations remain attached to -their shell. This is a read-projection boundary, not event deletion or a change -to notification delivery. Canonical-only fallback when no shell is retained, -missing recurrence boundaries after retention, and already-duplicated shells -remain outside this guarantee. Legacy shell lookup retains its time tolerance; -events preceding the selected shell's exact opening are not projected into it. -`TestIncidentStore_CanonicalProjectionOccurrenceBounds` verifies both boundaries, -subsecond starts, unordered successors and unrelated alert/resource isolation. -It also checkpoints all retained shells to JSON, reconstructs the incident store -and replays a firing: projected identity/state stays identical and the unchanged -checkpoint is not replaced. The canonical timeline stays in memory; this is not -a durable event-store restart or installed-process write-rate measurement. +The shared canonical incident query supersedes the earlier retained-shell-only +projection. Explicit firing boundaries, resource history identities and retained +shell boundaries isolate repeated occurrences. An unrelated resource's shell +cannot shorten the selected occurrence merely because its alert identifier +matches. Canonical-only occurrences and duplicate saved-shell notes use the same +query. Query bounds and read failures remain explicit. The upstream +`TestIncidentStore_CanonicalProjectionOccurrenceBounds` regression is preserved +alongside the new canonical-only, alias and duplicate-note cases. +The same regression checkpoints retained shells to JSON, reconstructs the store +and replays a firing. Stable projected identity and events remain identical, +while query observation bounds describe each fresh read. The unchanged JSON +checkpoint is not replaced. The canonical timeline stays in memory, so this is +not a durable event-store restart or installed-process write-rate measurement. + ### Unchanged incident JSON checkpoints @@ -70,6 +68,23 @@ does it eliminate snapshot serialization. Proof: `internal/ai/memory/incidents_unchanged_test.go` covers unchanged evaluations, metadata, restart, resolution/recurrence, file loss and failed-write retry. +Assistant owns composer registration and focus on every open, rather than only +on component mount. Closing clears the registered input so later keyboard +commands cannot target a detached composer. A handoff must leave Escape and +keyboard input in Assistant, not the underlying alert search. + +### Canonical incident-history queries + +Incident context reads `IncidentStore.QueryIncidents`, whose page contains +canonical evidence coverage and explicit truncation. Read failures become +unavailable context, not a healthy or empty-history claim. Explicit firing times +separate occurrences, and missing starts stay unknown. Historical lifecycle +status does not establish current resource health. Assistant handoff includes +the latest bounded events, their source provenance and query coverage. It +preserves the existing exclusion of raw command arguments/output. An absent +closure does not acquire a duration measured up to the current time. + + ### Canonical Patrol and Assistant continuation, 2026-09-07 Canonical investigation planning runs inside the model's tool call. The model @@ -8275,10 +8290,24 @@ distinguishes unavailable archives, failed reads and absent exact resource/windo pairs. Proof lives in `internal/metrics/incident_archive_test.go` and the registered-tool cases in `internal/ai/tools/incident_history_test.go`. -The legacy incidents listing still exposes incident memory and is not a complete -canonical incident query. Its old sampler-derived `active_count` is now null with -`active_count_status=not_measured`. Canonical-only events, alias-aware listing, -query bounds and projection-read errors remain an explicit modernization gap in -`patrol-assistant-customer-outcome-qualification`. The shared resource timeline -remains the evidence owner. Do not invent another incident lifecycle to repair -this listing. +The incidents listing now uses the shared canonical incident query, including +canonical-only events, alias-aware selection, explicit query bounds and surfaced +read errors. Saved incident shells preserve occurrence identity and local notes. +They do not own a second lifecycle. The old sampler-derived `active_count` stays +null with `active_count_status=not_measured`. Implementation and qualification +receipts are recorded in `docs/qualification/PATROL_ASSISTANT_CUSTOMER_JOURNEY.md`. +The wider customer-outcome qualification gate remains open. + +The history tool describes its actual read scope: alert resolution records +closure rather than workload recovery, related-resource IDs identify relationships +rather than additional event targets, and the read does not query action records. +The model owns interpretation of those records. A history-only read cannot prove +that a governed action exists or does not exist. + +Canonical alert lifecycle evidence owns the projected incident's resource ID, +resource type, alert type, level and message when present. Saved shells keep +stable incident IDs and local notes, but cannot override that canonical context. +Related command targets do not retarget an alert. The funded OpenRouter +`openai/gpt-6-astra` route has explicitly reviewed, tiered cost estimates dated +2026-09-07. Unreviewed aliases stay unknown. Its history-explanation qualification +is distinct from the earlier Gemini lab matrix and from wider rollout evidence. diff --git a/docs/release-control/v6/internal/subsystems/alerts.md b/docs/release-control/v6/internal/subsystems/alerts.md index 27fd5c156..6a28d1d9f 100644 --- a/docs/release-control/v6/internal/subsystems/alerts.md +++ b/docs/release-control/v6/internal/subsystems/alerts.md @@ -15,6 +15,24 @@ ## Purpose +Mobile incident drawers transfer their exact context to Assistant and close +through the shared explicit handoff callback. Keeping the source drawer above +Assistant, or dropping its occurrence identity to make navigation work, fails +the linked investigation journey. Both timeline and resource handoffs require +mounted regression and final-build narrow browser proof. + +### Incident history preserves unavailable reads and provenance + +Timeline and resource-history views use the shared incident projection. A failed +canonical read stays visibly unavailable, including after an error toast expires +or when cached evidence remains. Empty-history copy is shown only after a +successful empty read. Retry or Refresh repeats the read without discarding +saved notes. The event card keeps canonical observation/occurrence timestamps, +source, actor and record ID behind the same accessible evidence disclosure. +Unknown timestamps remain unavailable. Truncation is explicit. The Assistant +handoff preserves occurrence identity and the latest bounded event window, and +labels history as insufficient to establish current health. + ### Retained-queue recovery feedback has no reading deadline Retry and Dismiss failures have a view-local untimed equivalent beside the diff --git a/docs/release-control/v6/internal/subsystems/api-contracts.md b/docs/release-control/v6/internal/subsystems/api-contracts.md index 32e6fac3c..05da18545 100644 --- a/docs/release-control/v6/internal/subsystems/api-contracts.md +++ b/docs/release-control/v6/internal/subsystems/api-contracts.md @@ -22,6 +22,18 @@ The delivery-log response redacts embedded webhook URLs in errorMessage rather than treating the entire diagnostic as a URL. Non-secret operation and failure context survive alongside notificationId, destinationId, outcome, failureClass and attempts; malformed URL diagnostics fail closed. The HTTP payload proof is TestContract_DeliveryDiagnosticPayload; notification handler tests pin plain, credential-bearing and malformed errors. No response keys or route permissions change. +### Canonical incident-history query contract + +Incident APIs discover canonical-only occurrences and retain exact-start +selection. `/api/ai/incidents` returns structured incidents even without a +resource filter, plus history coverage and the existing unmeasured active-count +contract. Resource-detail summaries describe the same page, not a second read. +Canonical read failures return HTTP 503 before snapshot fallback. Timeline +exports preserve source event identity, observed/occurred timestamps, actor, +source, confidence, metadata and related identities. Notes can attach to a +canonical-only occurrence without storing a second canonical lifecycle. + + ### Canonical Patrol and Assistant continuation, 2026-09-07 Canonical planning owns persisted actor/request idempotency. The same trusted @@ -10754,8 +10766,9 @@ anomalies and marks historical recording status as historical. `active_count_status=not_measured` in every response. Incident memory, an empty result and unavailable services cannot establish a current count. The old coordinator never received production alert callbacks, so its zero was not a -measurement. The legacy listing's broader canonical query and read-error -modernization remains open under the customer-outcome qualification gap. +measurement. The shared QueryIncidents path now owns structured listing and +read-error propagation. Its qualification and delivery receipts remain in the +customer-outcome qualification record, separate from wider readiness. ### Explicit credentials during development qualification diff --git a/docs/release-control/v6/internal/subsystems/frontend-primitives.md b/docs/release-control/v6/internal/subsystems/frontend-primitives.md index e909142fc..cba226780 100644 --- a/docs/release-control/v6/internal/subsystems/frontend-primitives.md +++ b/docs/release-control/v6/internal/subsystems/frontend-primitives.md @@ -20,6 +20,28 @@ ## Purpose +Assistant owns composer registration and focus on every open, rather than only +on component mount. Closing clears the registered input so later keyboard +commands cannot target a detached composer. A handoff must leave Escape and +keyboard input in Assistant, not the underlying alert search. + +Mobile incident drawers transfer their exact context to Assistant and close +through the shared explicit handoff callback. Keeping the source drawer above +Assistant, or dropping its occurrence identity to make navigation work, fails +the linked investigation journey. Both timeline and resource handoffs require +mounted regression and final-build narrow browser proof. + +### Shared incident evidence disclosure + +Alerts timeline and resource-history events share IncidentTimelineEventCard. +Its native details/summary disclosure preserves keyboard activation and keeps +forensic provenance out of the default event summary. The same timestamp +formatter rejects missing or invalid evidence times. Timeline and resource +history compose persistent failure copy with their existing retry controls, +independently of transient notification toasts. The affected interaction and +viewport qualification is recorded in +`docs/qualification/PATROL_ASSISTANT_CUSTOMER_JOURNEY.md`. + ### Canonical Patrol and Assistant continuation, 2026-09-07 Patrol's Assistant context preserves unknown destructive risk and distinguishes @@ -7222,3 +7244,19 @@ widths despite the longer labels. The presentation and Overview delivery-status tests cover the evidence boundary; `scripts/check-alert-dispatch-copy.mjs` qualifies the real Overview with scripted API data in Chromium, not installed notification delivery. + +The incident-history continuation also projects retained desktop expansion into +the mobile drawer at the shared CSS breakpoint and keeps expansion state on +return to desktop. The mobile action label describes the visible drawer. +Operator note text and attribution survive the Assistant handoff while raw +command output remains excluded. Shared event cards override inherited table +no-wrap styling, and notes preserve line breaks. Long notes must remain readable +in both inline desktop history and the mobile drawer. Final source-bound +browser/model qualification is recorded in the customer-journey document. + +The shared type-to-search registry excludes inputs in inert modal backgrounds, +including prepared shortcut targets. Escape belongs to the active dialog and +must not clear a background history filter or invalidate its return-focus target. +Ordinary search shortcuts resume when the background becomes interactive again. +When Assistant is already open, a desktop-to-phone transition must retain that +destination rather than reopening the underlying history drawer above it. diff --git a/docs/release-control/v6/internal/subsystems/monitoring.md b/docs/release-control/v6/internal/subsystems/monitoring.md index 71a991037..e44e52ace 100644 --- a/docs/release-control/v6/internal/subsystems/monitoring.md +++ b/docs/release-control/v6/internal/subsystems/monitoring.md @@ -47,6 +47,15 @@ snapshots and actual memory-threshold evaluation, plus stale/offline/unavailable samples and other-instance lookup. It is synthetic runtime proof, not a live FreeBSD installation, automatic identity-match or browser receipt. +### Incident-history reconciliation boundary + +Active-alert timeline reconciliation uses the shared error-bearing incident +query. If canonical history is unavailable, reconciliation stops for that alert +and records the read failure. It must not treat a failed read as proof that a +firing event is absent. Historical source messages own alert condition wording. +Projection does not manufacture comparison direction from numeric placeholders. + + Container filesystem observations are native, resource-scoped reads. The shared `pkg/agents/filesystem` contract keeps measurement time, mountpoint, source, filesystem type and optional usage together. Unavailable reads have an error diff --git a/docs/release-control/v6/internal/subsystems/registry.json b/docs/release-control/v6/internal/subsystems/registry.json index 59ad6aaba..ff68e83b5 100644 --- a/docs/release-control/v6/internal/subsystems/registry.json +++ b/docs/release-control/v6/internal/subsystems/registry.json @@ -3219,6 +3219,7 @@ "frontend-modern/src/api/__tests__/alertIntentPolicies.test.ts", "internal/api/alerting/alerts_test.go", "internal/api/alerting/deadman_handlers_test.go", + "internal/api/alerting/incident_query_test.go", "internal/api/alerts_endpoints_test.go" ] }, @@ -5204,6 +5205,7 @@ "frontend-modern/src/features/platformPage/sharedPlatformPage.tsx", "frontend-modern/src/features/platformPage/usePlatformWindowedItems.ts", "frontend-modern/src/hooks/createNonSuspendingQuery.ts", + "frontend-modern/src/hooks/useTypeToSearch.ts", "frontend-modern/src/i18n/index.ts", "frontend-modern/src/i18n/messages.de.ts", "frontend-modern/src/i18n/messages.es.ts", @@ -5246,6 +5248,21 @@ ], "require_explicit_path_policy_coverage": true, "path_policies": [ + { + "id": "type-to-search-keyboard-ownership", + "label": "shared search keyboard and modal ownership proof", + "match_prefixes": [], + "match_files": [ + "frontend-modern/src/hooks/useTypeToSearch.ts" + ], + "allow_same_subsystem_tests": false, + "test_prefixes": [], + "exact_files": [ + "frontend-modern/src/components/shared/__tests__/Dialog.test.tsx", + "frontend-modern/src/components/shared/__tests__/SearchInput.test.tsx", + "frontend-modern/src/hooks/__tests__/useTypeToSearch.test.ts" + ] + }, { "id": "tracked-planning-document-status-audit", "label": "tracked planning-document status audit proof", @@ -6396,6 +6413,7 @@ "internal/monitoring/issue1638_dns_cache_test.go", "internal/monitoring/metric_window_provider_test.go", "internal/monitoring/monitor_additional_test.go", + "internal/monitoring/monitor_alert_handling_test.go", "internal/monitoring/monitor_alert_intent_test.go", "internal/monitoring/monitor_alert_override_migration_test.go", "internal/monitoring/monitor_backups_readstate_test.go", @@ -8151,6 +8169,7 @@ "internal/unifiedresources/ceph_pool_health_contract_test.go", "internal/unifiedresources/history_identity_test.go", "internal/unifiedresources/host_storage_cleanup_test.go", + "internal/unifiedresources/incident_history_query_test.go", "internal/unifiedresources/monitor_adapter_read_state_test.go", "internal/unifiedresources/views_test.go" ] diff --git a/docs/release-control/v6/internal/subsystems/storage-recovery.md b/docs/release-control/v6/internal/subsystems/storage-recovery.md index e3eca6e02..31438722a 100644 --- a/docs/release-control/v6/internal/subsystems/storage-recovery.md +++ b/docs/release-control/v6/internal/subsystems/storage-recovery.md @@ -21,6 +21,12 @@ ## Purpose +Incident history retains canonical storage-related risk, source identity and +separate observation/occurrence timestamps. A recorded alert closure or operator +note is not a restore result. Canonical read failure remains unavailable and +bounded or partial history cannot establish current storage health or backup +coverage. Recovery judgment still requires independent current evidence. + Delivery-log URL masking occurs in the HTTP diagnostic projection and does not rewrite retained attempt history or destination configuration. Preserved failure context and attempt identifiers describe notification delivery only, not a restore result or storage freshness. Malformed URL diagnostics fail closed in the response; this does not delete the retained failure, change retention windows, or grant retry, storage or recovery authority. ### Canonical Patrol and Assistant continuation, 2026-09-07 diff --git a/docs/release-control/v6/internal/subsystems/unified-resources.md b/docs/release-control/v6/internal/subsystems/unified-resources.md index 86604f092..0ab64f428 100644 --- a/docs/release-control/v6/internal/subsystems/unified-resources.md +++ b/docs/release-control/v6/internal/subsystems/unified-resources.md @@ -37,6 +37,22 @@ policy remains authoritative; this accessor introduces no second timeout. `TestCorrelatedGuestMemoryNextPoll` verifies the manually merged row through the next-poll consumers. Existing host-list membership and metric priority are unchanged. +### Bounded incident-history selection + +Canonical history queries filter exact alert identifiers and observation windows +before applying limits. The observation window is inclusive at `since` and +exclusive at `ObservedBefore`. All aggregate counts use the same filter +predicate. Both stores order by observation time descending, then event ID +descending, including late-arriving records. SQLite indexes the same guarded +alert-identity expression used by the query. Invalid legacy JSON and non-string +metadata cannot establish an alert identity. + +`ResourceHistoryIDs` exposes the existing alias/succession expansion for +read-only incident memory association. It does not rewrite event resource IDs, +resource operator state, approvals, action request identity or execution +capabilities. An unavailable canonical read remains an error. + + ### Canonical Patrol and Assistant continuation, 2026-09-07 Persisted action request identity is scoped to the trusted actor and exact diff --git a/frontend-modern/browser-verification.json b/frontend-modern/browser-verification.json index c211fb630..264d64068 100644 --- a/frontend-modern/browser-verification.json +++ b/frontend-modern/browser-verification.json @@ -1,35 +1,77 @@ { "version": 1, - "base_sha": "c857802df6b00e61e2229eea411d6675deff2d8a", - "verified_at": "2026-09-07T22:03:54.189527Z", + "base_sha": "977afdd9559c0e9d5859f4c79bcc48e889381bba", + "verified_at": "2026-09-08T00:40:37.659048Z", "result": "passed", "changed_paths": [ - "frontend-modern/src/utils/resourceStateAdapters.ts" + "frontend-modern/src/components/AI/Chat/index.tsx", + "frontend-modern/src/components/Alerts/IncidentAssistantHandoffButton.tsx", + "frontend-modern/src/components/Alerts/IncidentTimelineEventCard.tsx", + "frontend-modern/src/components/Alerts/IncidentTimelinePanel.tsx", + "frontend-modern/src/components/Alerts/incidentAssistantHandoffModel.ts", + "frontend-modern/src/features/alerts/AlertHistoryItemActions.tsx", + "frontend-modern/src/features/alerts/AlertHistoryMobileList.tsx", + "frontend-modern/src/features/alerts/AlertResourceIncidentsPanel.tsx", + "frontend-modern/src/features/alerts/MobileAlertHistoryInvestigationDialog.tsx", + "frontend-modern/src/features/alerts/useAlertHistoryState.ts", + "frontend-modern/src/features/alerts/useAlertResourceIncidentsState.ts", + "frontend-modern/src/hooks/useTypeToSearch.ts", + "frontend-modern/src/types/api.ts", + "frontend-modern/src/utils/alertIncidentPresentation.ts" ], "content_sha256": { - "frontend-modern/src/utils/resourceStateAdapters.ts": "ce1f8f2aef4b04b64dd2533c09458349811ba7cec9ec2e4bd8634b8e9c3c848f" + "frontend-modern/src/components/AI/Chat/index.tsx": "ca9e630294db426df591c3386d4fec161f34ade0bb6a8c4ea769130545a12341", + "frontend-modern/src/components/Alerts/IncidentAssistantHandoffButton.tsx": "b7a6ea6676ed9834d8254bb4ddab1301ec3b0d2368a96fa3e91e9ab5d2cefb4f", + "frontend-modern/src/components/Alerts/IncidentTimelineEventCard.tsx": "21e90602de00aa4765b1c40b5295a618a237501389d50ca9dbe36454abd11f32", + "frontend-modern/src/components/Alerts/IncidentTimelinePanel.tsx": "7298d4184c243325dbd671777257c7efd66c86091eb4b83663315e2bdc131fcb", + "frontend-modern/src/components/Alerts/incidentAssistantHandoffModel.ts": "9e777eee1be46f33ac146770fac6193cf6cd1c8b33ab0d0cd72fe57b98907a37", + "frontend-modern/src/features/alerts/AlertHistoryItemActions.tsx": "06e75d16ccad383b7c89d109baaee71852482f49b41ef3d0f5cc99768dd35101", + "frontend-modern/src/features/alerts/AlertHistoryMobileList.tsx": "851a2c1dc11ccbc827a6a50dfe06088b13a7307ce489b0767804daf6f33d023c", + "frontend-modern/src/features/alerts/AlertResourceIncidentsPanel.tsx": "c6f89c4991c9cfc2b1c8c82f7fed35af93b3b0b29b16b2c4bdc61a51005bec4c", + "frontend-modern/src/features/alerts/MobileAlertHistoryInvestigationDialog.tsx": "55af36a7194c2dbe9d944ca08f52e794f2b5e2ccbe79427bc55abe66aaa8a522", + "frontend-modern/src/features/alerts/useAlertHistoryState.ts": "652a09b02321ad1cfd087aacc7937a6afb5cfb7da0e0d6ca997b30d5c9ac53bd", + "frontend-modern/src/features/alerts/useAlertResourceIncidentsState.ts": "f9d44d5bd2199afb274b329877e19e891942106a63cecbeb26796fa4c7b3b507", + "frontend-modern/src/hooks/useTypeToSearch.ts": "b3f5244fd951baedc33fc8efee32c28077d94c07e5fce958eb6611bf50992a20", + "frontend-modern/src/types/api.ts": "82177cc67fc449a28d2d3cc226605ae091bd1f83fbdb001d15b3b430e5cd4100", + "frontend-modern/src/utils/alertIncidentPresentation.ts": "2e098b24b04c047a1b024a93608c5133720660cafc54c40408972413786a0a07" }, "routes": [ - "/proxmox" + "/alerts" ], "viewports": [ { - "width": 1280, - "height": 900 + "width": 1440, + "height": 1000 + }, + { + "width": 900, + "height": 1000 + }, + { + "width": 768, + "height": 1000 + }, + { + "width": 767, + "height": 1000 }, { "width": 390, - "height": 844 + "height": 1000 } ], "states": [ - "Canonical 100% to 35%; stale/offline/unavailable agent evidence retains trusted 100%.", - "Explicit Proxmox withdrawal with omitted canonical memory: table N/A, Memory section absent.", - "Measured-zero recovery: table 0%, Memory Total and Free both 100 MB; second cluster stays 80%." + "Canonical merged occurrence and source evidence, resolved alert with independently unknown current health.", + "Loading and disabled Refresh, initial/cached read failures, Retry/Refresh, successful empty, partial history and unknown times.", + "Saved operator note and unsaved draft across desktop/phone layout changes, long-note wrapping and persisted reload.", + "Timeline/resource handoff into Assistant, reopened composer focus, saved funded history-and-note response across widths." ], "interactions": [ - "Open guest drawer; deliver synthetic canonical socket snapshots while open.", - "Resize to desktop and narrow Chromium; scroll recovered Memory into view above fixed navigation; inspect all five final screenshots." + "Expand/collapse native Evidence details by keyboard, All/None event filters and note draft enable/clear.", + "Expand resource Events, inject read errors, retry and reload.", + "Resize source and open Assistant destination across 390/900/767/768/1440 widths.", + "Type in a modal without clearing background search, Escape and return focus to the selected card.", + "Resume exact saved session 81c92125-cdc3-4611-931a-687e6548bf65, scroll through its note explanation and inspect actual pixels." ], - "notes": "Synthetic Desktop Chrome acceptance only, not touch/mobile-device, installed poller, FreeBSD reporter or alert-delivery proof. Base reproduced stale 100% after withdrawal; adapter repair passed final Chromium test without retries and pageerror assertion. Earlier fixture wrongly expected raw-facet total after withdrawal; useWorkloads maps canonical metrics only. Two selector/absent-section failures and first passing but bottom-nav-obscured screenshot run are retained, not counted as final visual acceptance. Final four transition screenshots show withdrawn section absent and recovered Total/Free in frame at both widths; initial 35% screenshot is table-only. Evidence: /var/lib/pulse-maintainer/queue/staging/20260907T215015Z-web-product/ (moves to completed): browser-base.log, browser-repair.log, browser-repair-selector.log, browser-canonical.log, browser-final.log, final-report/data/*.png. Browser command: pulse-heavy-run -- bash wrapper starting Vite 127.0.0.1:5187 then PLAYWRIGHT_BASE_URL=http://127.0.0.1:5187 playwright test tests/86-hybrid-memory-browser.spec.ts --project=chromium --retries=0; Vite stopped by wrapper trap." + "notes": "Final local Chromium functional and visual acceptance of runtime 0.0.0-dev-pro+incident-history-r10, binary SHA256 2e1958d0856c812d22870144cf228509f2a4a3696258a533d34802442f500edb. Commands and retained failed attempts are in docs/qualification/PATROL_ASSISTANT_CUSTOMER_JOURNEY.md. Receipt directories: /Volumes/Development/pulse/tmp/patrol-incident-history/browser, browser-states, browser-resize and browser-saved-explanation. No new model calls in final browser repetition. Funded changed-context r9 response remains applicable because r10 changes only responsive destination ownership. Controlled HTTP variants are not a database outage. Not touch-device, production rollout, backup/restore or population reliability proof. Upstream 977afdd adds tests and contract documentation only. All 14 accepted frontend content hashes remain identical." } diff --git a/frontend-modern/src/components/AI/Chat/__tests__/AIChat.test.tsx b/frontend-modern/src/components/AI/Chat/__tests__/AIChat.test.tsx index 6928e7768..5d6c26fd3 100644 --- a/frontend-modern/src/components/AI/Chat/__tests__/AIChat.test.tsx +++ b/frontend-modern/src/components/AI/Chat/__tests__/AIChat.test.tsx @@ -1078,6 +1078,25 @@ describe('AIChat', () => { }); }); + it('focuses and registers the current composer after each reopen', async () => { + const [open, setOpen] = createSignal(false); + mockAiChatStore.isOpenSignal.mockImplementation(open); + renderChat(); + expect(screen.queryByPlaceholderText('Ask about your infrastructure...')).toBeNull(); + setOpen(true); + const first = await screen.findByPlaceholderText('Ask about your infrastructure...'); + await waitFor(() => expect(document.activeElement).toBe(first)); + setOpen(false); + expect(mockAiChatStore.registerInput).toHaveBeenLastCalledWith(null); + setOpen(true); + const second = await screen.findByPlaceholderText('Ask about your infrastructure...'); + expect(second).not.toBe(first); + await waitFor(() => { + expect(document.activeElement).toBe(second); + expect(mockAiChatStore.registerInput).toHaveBeenLastCalledWith(second); + }); + }); + it('renders the ChatMessages child component', () => { renderChat(); expect(screen.getByTestId('chat-messages')).toBeInTheDocument(); diff --git a/frontend-modern/src/components/AI/Chat/index.tsx b/frontend-modern/src/components/AI/Chat/index.tsx index b3a45eb8c..9be55a529 100644 --- a/frontend-modern/src/components/AI/Chat/index.tsx +++ b/frontend-modern/src/components/AI/Chat/index.tsx @@ -2819,8 +2819,14 @@ export const AIChat: Component = (props) => { createEffect(() => { if (!isOpen()) { setShowCommandHelp(false); + aiChatStore.registerInput?.(null); return; } + queueMicrotask(() => { + if (!isOpen()) return; + aiChatStore.registerInput?.(textareaRef ?? null); + textareaRef?.focus(); + }); openInitialization = initializeWhenOpen(); }); @@ -2915,9 +2921,7 @@ export const AIChat: Component = (props) => { // Click outside handler to close all dropdowns onMount(() => { setPromptHistory(loadPromptHistory()); - aiChatStore.registerInput?.(textareaRef ?? null); restoreStashedComposerDraft(); - focusComposer(); const handleClickOutside = (e: MouseEvent) => { const target = e.target as HTMLElement; diff --git a/frontend-modern/src/components/Alerts/IncidentAssistantHandoffButton.tsx b/frontend-modern/src/components/Alerts/IncidentAssistantHandoffButton.tsx index fc5f7e6ad..07699af34 100644 --- a/frontend-modern/src/components/Alerts/IncidentAssistantHandoffButton.tsx +++ b/frontend-modern/src/components/Alerts/IncidentAssistantHandoffButton.tsx @@ -7,6 +7,7 @@ interface IncidentAssistantHandoffButtonProps { incident: Incident; label?: string; class?: string; + onAssistantHandoff?: () => void; } export function IncidentAssistantHandoffButton(props: IncidentAssistantHandoffButtonProps) { @@ -19,7 +20,15 @@ export function IncidentAssistantHandoffButton(props: IncidentAssistantHandoffBu event.stopPropagation(); const handoff = buildAlertIncidentAssistantHandoff({ incident: props.incident }); - aiChatStore.open(handoff.context); + const openAssistant = () => aiChatStore.open(handoff.context); + if (props.onAssistantHandoff) { + props.onAssistantHandoff(); + // Let the source dialog release its blocking layer before opening the + // destination, as with command-palette handoffs. + queueMicrotask(openAssistant); + } else { + openAssistant(); + } }; return ( diff --git a/frontend-modern/src/components/Alerts/IncidentTimelineEventCard.tsx b/frontend-modern/src/components/Alerts/IncidentTimelineEventCard.tsx index 55664ef2b..ff40b5125 100644 --- a/frontend-modern/src/components/Alerts/IncidentTimelineEventCard.tsx +++ b/frontend-modern/src/components/Alerts/IncidentTimelineEventCard.tsx @@ -1,6 +1,9 @@ import { Show } from 'solid-js'; import type { IncidentEvent } from '@/types/api'; import { + formatIncidentEvidenceTime, + INCIDENT_EVIDENCE_DETAILS, + INCIDENT_TIME_UNAVAILABLE, getAlertIncidentTimelineCommandClass, getAlertIncidentTimelineDetailClass, getAlertIncidentTimelineEventCardClass, @@ -33,8 +36,37 @@ export function IncidentTimelineEventCard(props: IncidentTimelineEventCardProps)
{props.event.summary} - {new Date(props.event.timestamp).toLocaleString()} + + {formatIncidentEvidenceTime(props.event.timestamp) ?? INCIDENT_TIME_UNAVAILABLE} +
+ + {(evidence) => ( +
+ + {INCIDENT_EVIDENCE_DETAILS} + +
+
Observed
+
+ {formatIncidentEvidenceTime(evidence().observedAt) ?? INCIDENT_TIME_UNAVAILABLE} +
+
Occurred
+
+ {formatIncidentEvidenceTime(evidence().occurredAt) ?? INCIDENT_TIME_UNAVAILABLE} +
+
Source
+
{evidence().sourceAdapter || evidence().sourceType || 'Unknown'}
+ +
Recorded actor
+
{evidence().actor}
+ +
Record
+
{evidence().id}
+
+
+ )} +

{note()}

 diff --git a/frontend-modern/src/components/Alerts/IncidentTimelinePanel.tsx b/frontend-modern/src/components/Alerts/IncidentTimelinePanel.tsx index f0079e357..db3fb5a92 100644 --- a/frontend-modern/src/components/Alerts/IncidentTimelinePanel.tsx +++ b/frontend-modern/src/components/Alerts/IncidentTimelinePanel.tsx @@ -23,6 +23,9 @@ import { } from '@/utils/alertOverviewPresentation'; import { type AlertIncidentEventFilterVariant, + formatIncidentEvidenceTime, + INCIDENT_HISTORY_PARTIAL, + INCIDENT_HISTORY_LEGACY, getAlertIncidentAcknowledgedBadgeClass, getAlertIncidentNoteSaveButtonClass, getAlertIncidentNoteTextareaClass, @@ -43,6 +46,7 @@ export interface IncidentTimelinePanelProps { noteSaving: Accessor; onSaveNote: () => void; onRetry: () => void; + onAssistantHandoff?: () => void; } export function IncidentTimelinePanel(props: IncidentTimelinePanelProps) { @@ -57,6 +61,21 @@ export function IncidentTimelinePanel(props: IncidentTimelinePanelProps) { {getAlertTimelineLoadingState().text}

+ +
+ + +
+ {(loadedTimeline) => (
@@ -71,10 +90,10 @@ export function IncidentTimelinePanel(props: IncidentTimelinePanelProps) { {getAlertTimelineAcknowledgedLabel()} - + {getAlertTimelineOpenedAtLabel( - new Date(loadedTimeline().openedAt).toLocaleString(), + formatIncidentEvidenceTime(loadedTimeline().openedAt)!, )} @@ -86,8 +105,19 @@ export function IncidentTimelinePanel(props: IncidentTimelinePanelProps) {
- +
+ +

+ {INCIDENT_HISTORY_PARTIAL} +

+ + +

{INCIDENT_HISTORY_LEGACY}

+ 0}> )} - - - {getAlertTimelineUnavailableState().text} -

- } - > -
- - -
- + +

+ {getAlertTimelineUnavailableState().text} +

); diff --git a/frontend-modern/src/components/Alerts/__tests__/IncidentTimelineEventCard.test.tsx b/frontend-modern/src/components/Alerts/__tests__/IncidentTimelineEventCard.test.tsx index 03ab6f121..20c247b4d 100644 --- a/frontend-modern/src/components/Alerts/__tests__/IncidentTimelineEventCard.test.tsx +++ b/frontend-modern/src/components/Alerts/__tests__/IncidentTimelineEventCard.test.tsx @@ -49,4 +49,42 @@ describe('IncidentTimelineEventCard', () => { expect(screen.queryByText('systemctl restart alert')).not.toBeInTheDocument(); expect(screen.queryByText('stdout: ok')).not.toBeInTheDocument(); }); + it('preserves observation and occurrence provenance behind a disclosure', () => { + const { container } = render(() => ( + + )); + const details = container.querySelector('details'); + expect(details).not.toHaveAttribute('open'); + expect(screen.getByText('Evidence details')).toBeInTheDocument(); + expect(screen.getByText('Observed')).toBeInTheDocument(); + expect(screen.getByText('Occurred')).toBeInTheDocument(); + expect(screen.getByText('canonical-record')).toBeInTheDocument(); + expect(screen.getByText('operator')).toBeInTheDocument(); + }); + + it('does not fabricate a date for missing evidence time', () => { + render(() => ( + + )); + expect(screen.getByText('Time unavailable')).toBeInTheDocument(); + expect(screen.queryByText(/1\/1\/1/)).not.toBeInTheDocument(); + }); }); diff --git a/frontend-modern/src/components/Alerts/__tests__/IncidentTimelinePanel.test.tsx b/frontend-modern/src/components/Alerts/__tests__/IncidentTimelinePanel.test.tsx index 178bb120c..b48995410 100644 --- a/frontend-modern/src/components/Alerts/__tests__/IncidentTimelinePanel.test.tsx +++ b/frontend-modern/src/components/Alerts/__tests__/IncidentTimelinePanel.test.tsx @@ -191,4 +191,63 @@ describe('IncidentTimelinePanel', () => { }); expect(JSON.stringify(context)).not.toContain('systemctl status pulse'); }); + it('discloses a capped history without inventing its opening time', () => { + const [filters, setFilters] = createSignal(new Set(['command'])); + render(() => ( + + makeTimeline({ + status: 'unknown', + openedAt: '0001-01-01T00:00:00Z', + history: { + source: 'canonical_resource_history', + observedSince: '2026-03-01T00:00:00Z', + observedBefore: '2026-03-20T12:00:00Z', + changeLimit: 1, + hasMoreChanges: true, + hasMoreIncidents: false, + }, + }) + } + loading={() => false} + error={() => false} + filters={filters} + setFilters={setFilters} + filterVariant="compact" + eventCardVariant="surface" + noteDraft={() => ''} + onNoteDraftChange={() => undefined} + noteSaving={() => false} + onSaveNote={() => undefined} + onRetry={() => undefined} + /> + )); + expect(screen.getByRole('status')).toHaveTextContent( + 'Earlier occurrences or events may be missing', + ); + expect(screen.getByText('unknown')).toBeInTheDocument(); + expect(screen.queryByText(/^opened /)).not.toBeInTheDocument(); + }); + it('keeps refresh failure and retry visible alongside saved evidence', () => { + const [filters, setFilters] = createSignal(new Set(['command'])); + render(() => ( + makeTimeline()} + loading={() => false} + error={() => true} + filters={filters} + setFilters={setFilters} + filterVariant="compact" + eventCardVariant="surface" + noteDraft={() => ''} + onNoteDraftChange={() => undefined} + noteSaving={() => false} + onSaveNote={() => undefined} + onRetry={() => undefined} + /> + )); + expect(screen.getByRole('alert')).toHaveTextContent('Failed to load timeline.'); + expect(screen.getByRole('button', { name: 'Retry' })).toBeInTheDocument(); + expect(screen.getByText('Command executed')).toBeInTheDocument(); + }); }); diff --git a/frontend-modern/src/components/Alerts/__tests__/incidentAssistantHandoffModel.branchcov2.test.ts b/frontend-modern/src/components/Alerts/__tests__/incidentAssistantHandoffModel.branchcov2.test.ts index e3f1b0e70..58da93f66 100644 --- a/frontend-modern/src/components/Alerts/__tests__/incidentAssistantHandoffModel.branchcov2.test.ts +++ b/frontend-modern/src/components/Alerts/__tests__/incidentAssistantHandoffModel.branchcov2.test.ts @@ -55,10 +55,8 @@ interface StrictHandoff { context: StrictHandoffContext; } -const DEFAULT_NOW = new Date('2026-03-20T10:05:00Z'); - -function buildHandoff(incident: Incident, now: Date = DEFAULT_NOW): StrictHandoff { - return buildAlertIncidentAssistantHandoff({ incident, now }) as unknown as StrictHandoff; +function buildHandoff(incident: Incident): StrictHandoff { + return buildAlertIncidentAssistantHandoff({ incident }) as unknown as StrictHandoff; } function makeIncident(overrides: Partial = {}): Incident { @@ -114,7 +112,6 @@ describe('formatIncidentDuration (exercised via buildAlertIncidentAssistantHando openedAt: '2026-03-20T10:00:00Z', closedAt: '2026-03-20T10:01:00Z', }), - new Date('2026-03-20T11:00:00Z'), ); expect(handoff.context.briefing.statusLabel).toBe(`${STATUS_PREFIX}1 min`); expect(handoff.context.handoffContext).toContain('Duration: 1 min'); @@ -126,7 +123,6 @@ describe('formatIncidentDuration (exercised via buildAlertIncidentAssistantHando openedAt: '2026-03-20T10:00:00Z', closedAt: '2026-03-20T10:05:00Z', }), - new Date('2026-03-20T11:00:00Z'), ); expect(handoff.context.briefing.statusLabel).toBe(`${STATUS_PREFIX}5 mins`); }); @@ -137,20 +133,18 @@ describe('formatIncidentDuration (exercised via buildAlertIncidentAssistantHando openedAt: '2026-03-20T10:00:00Z', closedAt: '2026-03-20T10:00:00Z', }), - new Date('2026-03-20T11:00:00Z'), ); expect(handoff.context.briefing.statusLabel).toBe(`${STATUS_PREFIX}0 mins`); }); - it('clamps a negative (closedAt before openedAt) delta to "0 mins" via Math.max(0, ...)', () => { + it('keeps an inconsistent closure time unknown', () => { const handoff = buildHandoff( makeIncident({ openedAt: '2026-03-20T10:05:00Z', closedAt: '2026-03-20T10:00:00Z', }), - new Date('2026-03-20T11:00:00Z'), ); - expect(handoff.context.briefing.statusLabel).toBe(`${STATUS_PREFIX}0 mins`); + expect(handoff.context.briefing.statusLabel).toBe(`${STATUS_PREFIX}unknown duration`); }); it('formats a sub-day >= 60min delta as "Xh Ym" (durationMins >= 60, durationHours < 24 arm)', () => { @@ -159,7 +153,6 @@ describe('formatIncidentDuration (exercised via buildAlertIncidentAssistantHando openedAt: '2026-03-20T10:00:00Z', closedAt: '2026-03-20T11:05:00Z', }), - new Date('2026-03-20T12:00:00Z'), ); expect(handoff.context.briefing.statusLabel).toBe(`${STATUS_PREFIX}1h 5m`); }); @@ -170,7 +163,6 @@ describe('formatIncidentDuration (exercised via buildAlertIncidentAssistantHando openedAt: '2026-03-20T10:00:00Z', closedAt: '2026-03-21T14:00:00Z', }), - new Date('2026-03-22T10:00:00Z'), ); // 28h elapsed -> floor(28/24)=1 day, 28%24=4 hours expect(handoff.context.briefing.statusLabel).toBe(`${STATUS_PREFIX}1d 4h`); @@ -182,7 +174,6 @@ describe('formatIncidentDuration (exercised via buildAlertIncidentAssistantHando openedAt: 'not-a-valid-date', closedAt: '2026-03-20T11:00:00Z', }), - new Date('2026-03-20T12:00:00Z'), ); expect(handoff.context.briefing.statusLabel).toBe(`${STATUS_PREFIX}unknown duration`); expect(handoff.context.handoffContext).toContain('Duration: unknown duration'); @@ -194,37 +185,13 @@ describe('formatIncidentDuration (exercised via buildAlertIncidentAssistantHando openedAt: '2026-03-20T10:00:00Z', closedAt: 'not-a-valid-date', }), - new Date('2026-03-20T12:00:00Z'), ); expect(handoff.context.briefing.statusLabel).toBe(`${STATUS_PREFIX}unknown duration`); }); - it('falls back to `now` when closedAt is undefined (closedAt-falsy ternary arm, nonzero delta)', () => { - const handoff = buildHandoff( - makeIncident({ closedAt: undefined }), - new Date('2026-03-20T10:05:00Z'), - ); - expect(handoff.context.briefing.statusLabel).toBe(`${STATUS_PREFIX}5 mins`); - }); - - it('falls back to `now` yielding zero when closedAt is undefined and now === openedAt', () => { - const handoff = buildHandoff( - makeIncident({ closedAt: undefined }), - new Date('2026-03-20T10:00:00Z'), - ); - expect(handoff.context.briefing.statusLabel).toBe(`${STATUS_PREFIX}0 mins`); - }); - - it('uses closedAt (not now) when closedAt is present (closedAt-truthy ternary arm)', () => { - // closedAt gives 5 mins; now is 1 hour after openedAt and would give 60 mins -> "1h 0m". - const handoff = buildHandoff( - makeIncident({ - openedAt: '2026-03-20T10:00:00Z', - closedAt: '2026-03-20T10:05:00Z', - }), - new Date('2026-03-20T11:00:00Z'), - ); - expect(handoff.context.briefing.statusLabel).toBe(`${STATUS_PREFIX}5 mins`); + it('keeps an absent closure unrecorded instead of measuring through now', () => { + const handoff = buildHandoff(makeIncident({ closedAt: undefined })); + expect(handoff.context.briefing.statusLabel).toBe(`${STATUS_PREFIX}closure not recorded`); }); }); @@ -374,17 +341,13 @@ describe('formatContextLine (exercised via buildAlertIncidentAssistantHandoff)', }); it('renders the "Closed At:" context line when closedAt is a present string', () => { - const handoff = buildHandoff( - makeIncident({ closedAt: '2026-03-20T10:05:00Z' }), - new Date('2026-03-20T11:00:00Z'), - ); + const handoff = buildHandoff(makeIncident({ closedAt: '2026-03-20T10:05:00Z' })); expect(handoff.context.handoffContext).toContain('Closed At: 2026-03-20T10:05:00Z'); }); it('trims a whitespace-padded closedAt value to its inner text', () => { const handoff = buildHandoff( makeIncident({ closedAt: ' 2026-03-20T10:05:00Z ' as unknown as string }), - new Date('2026-03-20T11:00:00Z'), ); expect(handoff.context.handoffContext).toContain('Closed At: 2026-03-20T10:05:00Z'); expect(handoff.context.handoffContext).not.toContain('Closed At: 2026-03-20T10:05:00Z'); @@ -393,7 +356,7 @@ describe('formatContextLine (exercised via buildAlertIncidentAssistantHandoff)', describe('buildAlertIncidentAssistantHandoff (resource-label, event-count & capping branches)', () => { it('uses resourceName first (|| chain 1st arm) and builds the full concrete handoff', () => { - const handoff = buildHandoff(makeIncident(), new Date('2026-03-20T10:05:00Z')); + const handoff = buildHandoff(makeIncident()); // 1st arm of the resource-label || chain. expect(handoff.context.briefing.subject).toBe( 'Critical docker-container-health on checkout-api', diff --git a/frontend-modern/src/components/Alerts/__tests__/incidentAssistantHandoffModel.test.ts b/frontend-modern/src/components/Alerts/__tests__/incidentAssistantHandoffModel.test.ts index ada64019d..da1776636 100644 --- a/frontend-modern/src/components/Alerts/__tests__/incidentAssistantHandoffModel.test.ts +++ b/frontend-modern/src/components/Alerts/__tests__/incidentAssistantHandoffModel.test.ts @@ -44,7 +44,6 @@ describe('incidentAssistantHandoffModel', () => { it('builds an approval-required incident timeline handoff without raw command payloads', () => { const handoff = buildAlertIncidentAssistantHandoff({ incident: makeIncident(), - now: new Date('2026-03-20T10:05:00Z'), }); expect(handoff).not.toHaveProperty('prompt'); @@ -64,7 +63,7 @@ describe('incidentAssistantHandoffModel', () => { sourceLabel: 'Pulse Alerts', title: 'Incident timeline attached', subject: 'Critical docker-container-health on checkout-api', - statusLabel: 'Critical incident · Open · 5 mins', + statusLabel: 'Critical incident · Open · closure not recorded', detailLines: [ '2 timeline events', 'Node: edge-1', @@ -110,4 +109,73 @@ describe('incidentAssistantHandoffModel', () => { expect(JSON.stringify(handoff)).not.toContain('systemctl'); expect(JSON.stringify(handoff)).not.toContain('secret-value'); }); + it('includes latest evidence and query bounds without exposing command payloads', () => { + const events = Array.from({ length: 12 }, (_, index) => ({ + id: `event-${index}`, + type: index === 11 ? 'alert_resolved' : 'command', + timestamp: '2026-03-20T10:05:00Z', + summary: index === 11 ? 'Independently resolved' : 'sensitive command', + evidence: { + id: `canonical-${index}`, + resourceId: 'resource-a', + kind: 'alert_resolved' as const, + observedAt: '2026-03-20T10:06:00Z', + occurredAt: '2026-03-20T10:05:00Z', + sourceType: 'platform_event' as const, + confidence: 'high' as const, + metadata: { command: 'must not appear' }, + }, + })); + const incident = makeIncident({ + events, + history: { + source: 'canonical_resource_history', + observedSince: '2026-03-01T00:00:00Z', + observedBefore: '2026-03-20T11:00:00Z', + changeLimit: 12, + hasMoreChanges: true, + hasMoreIncidents: false, + }, + }); + const handoff = buildAlertIncidentAssistantHandoff({ incident }); + expect(handoff.context.handoffContext).toContain('Latest 8 of 12 returned events'); + expect(handoff.context.handoffContext).toContain('Independently resolved'); + expect(handoff.context.handoffContext).toContain('canonical-11'); + expect(handoff.context.handoffContext).not.toContain('canonical-0"'); + expect(handoff.context.handoffContext).toContain('"hasMoreChanges":true'); + expect(handoff.context.handoffContext).toContain('2026-03-20T10:06:00Z'); + expect(handoff.context.handoffContext).not.toContain('must not appear'); + expect(handoff.context.handoffContext).not.toContain('sensitive command'); + }); + + it('keeps an absent occurrence start unknown in Assistant context', () => { + const handoff = buildAlertIncidentAssistantHandoff({ + incident: makeIncident({ openedAt: '0001-01-01T00:00:00Z', status: 'unknown' }), + }); + expect(handoff.context.handoffContext).toContain('Opened At: unknown'); + expect(handoff.context.handoffContext).toContain('unknown duration'); + expect(handoff.context.handoffContext).not.toContain('0001-01-01'); + }); +}); + +it('retains operator note content and attribution without forwarding command output', () => { + const incident = makeIncident(); + incident.events!.push({ + id: 'operator-note', + type: 'note', + timestamp: '2026-03-20T10:04:00Z', + summary: 'Note added by operator', + source: 'operator_note', + details: { + note: 'Keep the old pool until its replacement is verified', + output_excerpt: 'unrelated private output', + }, + }); + const { context } = buildAlertIncidentAssistantHandoff({ incident }); + expect(context.handoffContext).toContain( + 'Note added by operator: Keep the old pool until its replacement is verified', + ); + expect(context.handoffContext).toContain('source=operator_note'); + expect(context.handoffContext).not.toContain('unrelated private output'); + expect(context.handoffContext).not.toContain('token=secret-value'); }); diff --git a/frontend-modern/src/components/Alerts/incidentAssistantHandoffModel.ts b/frontend-modern/src/components/Alerts/incidentAssistantHandoffModel.ts index 40387e999..7c671b15e 100644 --- a/frontend-modern/src/components/Alerts/incidentAssistantHandoffModel.ts +++ b/frontend-modern/src/components/Alerts/incidentAssistantHandoffModel.ts @@ -1,10 +1,10 @@ import type { AIChatContext } from '@/stores/aiChat'; import type { Incident, IncidentEvent } from '@/types/api'; +import { formatIncidentEvidenceTime } from '@/utils/alertIncidentPresentation'; import { resolveAlertTargetType } from '@/utils/alertTargetTypes'; interface BuildAlertIncidentAssistantHandoffInput { incident: Incident; - now?: Date; } interface AlertIncidentAssistantHandoff { @@ -12,6 +12,19 @@ interface AlertIncidentAssistantHandoff { } interface SanitizedIncidentEvent { + source?: string; + evidence?: Pick< + NonNullable, + | 'id' + | 'resourceId' + | 'kind' + | 'observedAt' + | 'occurredAt' + | 'sourceType' + | 'sourceAdapter' + | 'actor' + | 'confidence' + >; id: string; type: string; timestamp: string; @@ -30,7 +43,6 @@ const LABEL_INITIALISMS: Record = { export function buildAlertIncidentAssistantHandoff({ incident, - now = new Date(), }: BuildAlertIncidentAssistantHandoffInput): AlertIncidentAssistantHandoff { const resourceLabel = incident.resourceName || incident.resourceId || 'unknown resource'; const targetType = resolveAlertTargetType({ @@ -40,7 +52,7 @@ export function buildAlertIncidentAssistantHandoff({ }); const levelLabel = formatIncidentLabel(incident.level); const statusLabel = formatIncidentLabel(incident.status); - const durationText = formatIncidentDuration(incident.openedAt, incident.closedAt, now); + const durationText = formatIncidentDuration(incident.openedAt, incident.closedAt); const events = sanitizeIncidentEvents(incident.events ?? []); const eventCount = events.length; const eventCountLabel = `${eventCount} timeline event${eventCount === 1 ? '' : 's'}`; @@ -79,7 +91,7 @@ export function buildAlertIncidentAssistantHandoff({ incident.message ? `Message: ${incident.message}` : undefined, ].filter((line): line is string => Boolean(line)), evidence: events - .slice(0, MAX_BRIEFING_EVENTS) + .slice(-MAX_BRIEFING_EVENTS) .map((event) => `${formatIncidentLabel(event.type)}: ${event.summary}`), actionLabel: `Discuss incident ${incident.id}`, safetyNote: 'Diagnostics and remediation require operator approval.', @@ -95,11 +107,13 @@ export function buildAlertIncidentAssistantHandoff({ resourceType: incident.resourceType, node: incident.node, instance: incident.instance, - openedAt: incident.openedAt, + openedAt: formatIncidentEvidenceTime(incident.openedAt) ? incident.openedAt : undefined, closedAt: incident.closedAt, acknowledged: incident.acknowledged, eventCount, - eventSummaries: events.slice(0, MAX_CONTEXT_EVENTS), + includedEventCount: Math.min(eventCount, MAX_CONTEXT_EVENTS), + history: incident.history ? { ...incident.history } : undefined, + eventSummaries: events.slice(-MAX_CONTEXT_EVENTS), }, }, }; @@ -123,11 +137,11 @@ function buildIncidentAssistantModelContext({ eventCountLabel: string; }): string { const eventLines = events - .slice(0, MAX_CONTEXT_EVENTS) + .slice(-MAX_CONTEXT_EVENTS) .map((event, index) => formatContextLine( `Timeline Event ${index + 1}`, - `${event.timestamp} | ${formatIncidentLabel(event.type)} | ${event.summary}`, + `${event.timestamp} | ${formatIncidentLabel(event.type)} | ${event.summary}${event.evidence ? ` | evidence=${JSON.stringify(event.evidence)}` : event.source ? ` | source=${event.source}` : ''}`, ), ); @@ -144,10 +158,22 @@ function buildIncidentAssistantModelContext({ formatContextLine('Resource Type', incident.resourceType), formatContextLine('Node', incident.node), formatContextLine('Instance', incident.instance), - formatContextLine('Opened At', incident.openedAt), + formatContextLine( + 'Opened At', + formatIncidentEvidenceTime(incident.openedAt) ? incident.openedAt : 'unknown', + ), formatContextLine('Closed At', incident.closedAt), formatContextLine('Duration', durationText), formatContextLine('Timeline Summary', eventCountLabel), + formatContextLine( + 'Included Events', + `Latest ${Math.min(events.length, MAX_CONTEXT_EVENTS)} of ${events.length} returned events`, + ), + formatContextLine( + 'History Coverage', + incident.history ? JSON.stringify(incident.history) : 'not recorded', + ), + 'History Boundary: Retained historical evidence does not establish current resource health.', formatContextLine('Message', incident.message), ...eventLines, 'Timeline Boundary: Command events are summarized only. Raw command details and output stay in the incident or governed approval surface.', @@ -163,6 +189,22 @@ function sanitizeIncidentEvents(events: IncidentEvent[]): SanitizedIncidentEvent type: event.type, timestamp: event.timestamp, summary: sanitizeIncidentEventSummary(event), + ...(event.source ? { source: event.source } : {}), + ...(event.evidence + ? { + evidence: { + id: event.evidence.id, + resourceId: event.evidence.resourceId, + kind: event.evidence.kind, + observedAt: event.evidence.observedAt, + occurredAt: event.evidence.occurredAt, + sourceType: event.evidence.sourceType, + sourceAdapter: event.evidence.sourceAdapter, + actor: event.evidence.actor, + confidence: event.evidence.confidence, + }, + } + : {}), })); } @@ -173,6 +215,10 @@ function sanitizeIncidentEventSummary(event: IncidentEvent): string { } const summary = event.summary.trim(); + if (normalizedType === 'note' && typeof event.details?.note === 'string') { + const note = event.details.note.trim(); + if (note) return `${summary || 'Operator note'}: ${note}`; + } return summary.length > 0 ? summary : 'Timeline event recorded'; } @@ -185,10 +231,12 @@ function formatContextLine( return text ? `${label}: ${text}` : undefined; } -function formatIncidentDuration(openedAt: string, closedAt: string | undefined, now: Date): string { +function formatIncidentDuration(openedAt: string, closedAt: string | undefined): string { + if (!formatIncidentEvidenceTime(openedAt)) return 'unknown duration'; const openedMs = new Date(openedAt).getTime(); - const closedMs = closedAt ? new Date(closedAt).getTime() : now.getTime(); - if (!Number.isFinite(openedMs) || !Number.isFinite(closedMs)) { + if (!closedAt) return 'closure not recorded'; + const closedMs = new Date(closedAt).getTime(); + if (!Number.isFinite(openedMs) || !Number.isFinite(closedMs) || closedMs < openedMs) { return 'unknown duration'; } diff --git a/frontend-modern/src/features/alerts/AlertHistoryItemActions.tsx b/frontend-modern/src/features/alerts/AlertHistoryItemActions.tsx index 5a6ba06b8..4532b6229 100644 --- a/frontend-modern/src/features/alerts/AlertHistoryItemActions.tsx +++ b/frontend-modern/src/features/alerts/AlertHistoryItemActions.tsx @@ -14,6 +14,7 @@ interface AlertHistoryItemActionsProps { touchSized?: boolean; onTimelineClick?: () => void; onResourceClick?: () => void; + timelineOpen?: boolean; } export function AlertHistoryItemActions(props: AlertHistoryItemActionsProps) { @@ -40,7 +41,9 @@ export function AlertHistoryItemActions(props: AlertHistoryItemActionsProps) { ); }} > - {props.state.expandedIncidents().has(rowKey()) ? 'Hide' : 'Timeline'} + {(props.timelineOpen ?? props.state.expandedIncidents().has(rowKey())) + ? 'Hide' + : 'Timeline'} diff --git a/frontend-modern/src/features/alerts/AlertHistoryMobileList.tsx b/frontend-modern/src/features/alerts/AlertHistoryMobileList.tsx index 3d1c85680..0b4df0f6a 100644 --- a/frontend-modern/src/features/alerts/AlertHistoryMobileList.tsx +++ b/frontend-modern/src/features/alerts/AlertHistoryMobileList.tsx @@ -1,4 +1,5 @@ -import { Show, createMemo, createSignal } from 'solid-js'; +import { Show, createMemo, createSignal, onCleanup, onMount } from 'solid-js'; +import { aiChatStore } from '@/stores/aiChat'; import { getAlertHistoryResourceTypeBadgeClass } from '@/utils/alertHistoryPresentation'; import { @@ -41,6 +42,41 @@ export function AlertHistoryMobileList(props: AlertHistoryMobileListProps) { return items; }); + // The shared expansion state survives layout changes. Project it into the + // phone's single drawer instead of leaving a hidden desktop expansion open. + onMount(() => { + if (typeof window.matchMedia !== 'function') return; + const viewport = window.matchMedia('(min-width: 768px)'); + const synchronizeLayout = () => { + if (viewport.matches || aiChatStore.isOpen) { + setInvestigation(null); + return; + } + const alerts = props.state.groupedAlerts().flatMap((group) => group.alerts); + const resourcePanel = props.state.resourceIncidentPanel(); + const resourceAlert = + resourcePanel && + alerts.find((alert) => props.state.getIncidentRowKey(alert) === resourcePanel.rowKey); + if (resourcePanel && resourceAlert) { + setInvestigation({ kind: 'resource', alert: resourceAlert, rowKey: resourcePanel.rowKey }); + return; + } + for (const rowKey of [...props.state.expandedIncidents()].reverse()) { + const alert = alerts.find( + (candidate) => props.state.getIncidentRowKey(candidate) === rowKey, + ); + if (alert) { + setInvestigation({ kind: 'timeline', alert, rowKey }); + return; + } + } + setInvestigation(null); + }; + synchronizeLayout(); + viewport.addEventListener('change', synchronizeLayout); + onCleanup(() => viewport.removeEventListener('change', synchronizeLayout)); + }); + const getInvestigationReturnFocusTarget = ( current: MobileAlertHistoryInvestigation, ): HTMLElement | null => { @@ -54,18 +90,20 @@ export function AlertHistoryMobileList(props: AlertHistoryMobileListProps) { return action ?? list; }; - const closeInvestigation = () => { + const closeInvestigation = (restoreFocus = true) => { const current = investigation(); if (!current) return; const currentFocusTarget = getInvestigationReturnFocusTarget(current); setInvestigation(null); - currentFocusTarget?.focus({ preventScroll: true }); + if (restoreFocus) { + currentFocusTarget?.focus({ preventScroll: true }); - requestAnimationFrame(() => { requestAnimationFrame(() => { - getInvestigationReturnFocusTarget(current)?.focus({ preventScroll: true }); + requestAnimationFrame(() => { + getInvestigationReturnFocusTarget(current)?.focus({ preventScroll: true }); + }); }); - }); + } if (current.kind === 'timeline') { if (props.state.expandedIncidents().has(current.rowKey)) { @@ -222,6 +260,9 @@ export function AlertHistoryMobileList(props: AlertHistoryMobileListProps) { state={props.state} class="mt-3" touchSized + timelineOpen={ + investigation()?.kind === 'timeline' && investigation()?.rowKey === rowKey() + } onTimelineClick={() => openTimelineInvestigation(alert, rowKey())} onResourceClick={() => openResourceInvestigation(alert, rowKey())} /> @@ -236,7 +277,8 @@ export function AlertHistoryMobileList(props: AlertHistoryMobileListProps) { closeInvestigation()} + onAssistantHandoff={() => closeInvestigation(false)} /> )} diff --git a/frontend-modern/src/features/alerts/AlertResourceIncidentsPanel.tsx b/frontend-modern/src/features/alerts/AlertResourceIncidentsPanel.tsx index 00af74fec..13c198d3b 100644 --- a/frontend-modern/src/features/alerts/AlertResourceIncidentsPanel.tsx +++ b/frontend-modern/src/features/alerts/AlertResourceIncidentsPanel.tsx @@ -7,6 +7,8 @@ import { Card } from '@/components/shared/Card'; import type { Resource } from '@/types/resource'; import { getPreferredInfrastructureDisplayName } from '@/utils/resourceIdentity'; import { + formatIncidentEvidenceTime, + INCIDENT_HISTORY_PARTIAL, getAlertIncidentLevelBadgeClass, getAlertIncidentStatusPresentation, getAlertIncidentTimelineHeadingClass, @@ -20,6 +22,7 @@ import { getAlertResourceIncidentEmptyState, getAlertResourceIncidentFilteredEventsEmptyState, getAlertResourceIncidentLoadingState, + getAlertResourceIncidentLoadFailure, getAlertResourceIncidentPanelTitle, getAlertResourceIncidentRefreshLabel, getAlertResourceIncidentSummaryRowClass, @@ -35,6 +38,7 @@ interface AlertResourceIncidentsPanelProps { state: AlertHistoryState; getResource?: (resourceId: string) => Resource | undefined; onClose?: () => void; + onAssistantHandoff?: () => void; showCloseAction?: boolean; showTitle?: boolean; } @@ -46,6 +50,7 @@ export function AlertResourceIncidentsPanel(props: AlertResourceIncidentsPanelPr const resourceId = selection().resourceId; const incidents = () => props.state.resourceIncidents()[resourceId] || []; const isLoading = () => props.state.resourceIncidentLoading()[resourceId]; + const hasError = () => props.state.resourceIncidentError()[resourceId]; const lookupResource = () => props.getResource ?? props.state.getResource; const resource = () => lookupResource()?.(resourceId); const resourceDisplayName = () => { @@ -99,6 +104,11 @@ export function AlertResourceIncidentsPanel(props: AlertResourceIncidentsPanelPr

{getAlertResourceIncidentLoadingState().text}

 + + + 0}>
@@ -113,7 +123,11 @@ export function AlertResourceIncidentsPanel(props: AlertResourceIncidentsPanelPr 0} fallback={ -

{getAlertResourceIncidentEmptyState().text}

+ +

+ {getAlertResourceIncidentEmptyState().text} +

+ } >
@@ -159,15 +173,25 @@ export function AlertResourceIncidentsPanel(props: AlertResourceIncidentsPanelPr {statusPresentation.label} - opened {new Date(incident.openedAt).toLocaleString()} + + opened {formatIncidentEvidenceTime(incident.openedAt)} + closed {new Date(incident.closedAt as string).toLocaleString()}
- +
+ +

+ {INCIDENT_HISTORY_PARTIAL} +

+

{incident.message}

 diff --git a/frontend-modern/src/features/alerts/MobileAlertHistoryInvestigationDialog.tsx b/frontend-modern/src/features/alerts/MobileAlertHistoryInvestigationDialog.tsx index f8c8d7a2e..e15eab49b 100644 --- a/frontend-modern/src/features/alerts/MobileAlertHistoryInvestigationDialog.tsx +++ b/frontend-modern/src/features/alerts/MobileAlertHistoryInvestigationDialog.tsx @@ -21,6 +21,7 @@ interface MobileAlertHistoryInvestigationDialogProps { investigation: MobileAlertHistoryInvestigation; state: AlertHistoryState; onClose: () => void; + onAssistantHandoff?: () => void; } export function MobileAlertHistoryInvestigationDialog( @@ -66,6 +67,7 @@ export function MobileAlertHistoryInvestigationDialog( props.state.incidentLoading()[rowKey()]} error={() => props.state.incidentErrors()[rowKey()]} timeline={() => props.state.incidentTimelines()[rowKey()]} @@ -88,6 +90,7 @@ export function MobileAlertHistoryInvestigationDialog( diff --git a/frontend-modern/src/features/alerts/__tests__/AlertHistoryMobileList.test.tsx b/frontend-modern/src/features/alerts/__tests__/AlertHistoryMobileList.test.tsx index 70141394d..38dbf0cae 100644 --- a/frontend-modern/src/features/alerts/__tests__/AlertHistoryMobileList.test.tsx +++ b/frontend-modern/src/features/alerts/__tests__/AlertHistoryMobileList.test.tsx @@ -2,6 +2,7 @@ import { fireEvent, render, screen } from '@solidjs/testing-library'; import { createSignal } from 'solid-js'; import { describe, expect, it, vi } from 'vitest'; +import { aiChatStore } from '@/stores/aiChat'; import { AlertHistoryMobileList } from '../AlertHistoryMobileList'; import type { AlertHistoryState } from '../useAlertHistoryState'; @@ -70,6 +71,7 @@ function createState() { resourceIncidentPanel, resourceIncidents: () => ({ 'node-1': [] }), resourceIncidentLoading: () => ({}), + resourceIncidentError: () => ({}), resourceIncidentEventFilters: () => new Set(), setResourceIncidentEventFilters: vi.fn(), refreshResourceIncidentPanel: vi.fn(), @@ -176,6 +178,74 @@ describe('AlertHistoryMobileList', () => { }); describe('AlertHistoryMobileList changing history', () => { + it.each(['timeline', 'resource'] as const)( + 'preserves an open %s across desktop and phone layouts', + async (kind) => { + const { state } = createState(); + let resize = () => {}; + const viewport = { + matches: true, + addEventListener: vi.fn((_event: string, listener: () => void) => { + resize = listener; + }), + removeEventListener: vi.fn(), + }; + vi.stubGlobal('matchMedia', () => viewport); + if (kind === 'timeline') await state.toggleIncidentTimeline('alert-1-row', 'alert-1'); + else await state.openResourceIncidentPanel('node-1', 'pve-production-01', 'alert-1-row'); + const mounted = render(() => ); + const assistantOpen = vi.spyOn(aiChatStore, 'isOpen', 'get').mockReturnValue(false); + try { + expect(screen.queryByRole('dialog')).toBeNull(); + expect(screen.getByRole('button', { name: 'Timeline' })).toBeVisible(); + assistantOpen.mockReturnValue(true); + viewport.matches = false; + resize(); + expect(screen.queryByRole('dialog')).toBeNull(); + viewport.matches = true; + resize(); + assistantOpen.mockReturnValue(false); + viewport.matches = false; + resize(); + expect( + screen.getByRole('dialog', { + name: + kind === 'timeline' + ? 'Incident timeline for pve-production-01' + : 'Resource incidents for pve-production-01', + }), + ).toBeVisible(); + viewport.matches = true; + resize(); + expect(screen.queryByRole('dialog')).toBeNull(); + expect( + kind === 'timeline' + ? state.expandedIncidents().has('alert-1-row') + : state.resourceIncidentPanel()?.rowKey === 'alert-1-row', + ).toBe(true); + viewport.matches = false; + resize(); + expect(screen.getByRole('dialog')).toBeVisible(); + await fireEvent.click( + screen.getByRole('button', { + name: kind === 'timeline' ? 'Close incident timeline' : 'Close resource incidents', + }), + ); + expect(screen.queryByRole('dialog')).toBeNull(); + expect( + kind === 'timeline' + ? state.expandedIncidents().has('alert-1-row') + : state.resourceIncidentPanel() !== null, + ).toBe(false); + } finally { + assistantOpen.mockRestore(); + mounted.unmount(); + vi.unstubAllGlobals(); + } + expect(viewport.removeEventListener).toHaveBeenCalledWith('change', resize); + }, + ); + it('returns focus to the list when the investigated row disappears', async () => { const { closeResourceIncidentPanel, setGroupedAlerts, state } = createState(); render(() => ); @@ -202,3 +272,45 @@ describe('AlertHistoryMobileList changing history', () => { expect(screen.getByTestId('alert-history-mobile-list')).toHaveFocus(); }); }); + +describe('history handoff focus ownership', () => { + it('leaves focus with Assistant after the source drawer closes', async () => { + const { state } = createState(); + state.incidentTimelines = () => + ({ + 'alert-1-row': { + id: 'incident-focus', + alertType: 'backup', + level: 'warning', + openedAt: '2026-09-07T12:00:00Z', + resourceId: 'node-1', + resourceName: 'Backup', + status: 'resolved', + events: [], + }, + }) as any; + aiChatStore.setEnabled(true); + const composer = document.createElement('textarea'); + document.body.append(composer); + const open = vi.spyOn(aiChatStore, 'open').mockImplementation(() => composer.focus()); + try { + render(() => ); + await fireEvent.click(screen.getByRole('button', { name: 'Timeline' })); + await fireEvent.click( + screen.getByRole('button', { + name: 'Discuss incident incident-focus with Pulse Assistant', + }), + ); + await new Promise((resolve) => + requestAnimationFrame(() => requestAnimationFrame(() => resolve())), + ); + expect(screen.queryByRole('dialog')).toBeNull(); + expect(open).toHaveBeenCalledOnce(); + expect(composer).toHaveFocus(); + } finally { + open.mockRestore(); + composer.remove(); + aiChatStore.setEnabled(false); + } + }); +}); diff --git a/frontend-modern/src/features/alerts/__tests__/AlertResourceIncidentsPanel.test.tsx b/frontend-modern/src/features/alerts/__tests__/AlertResourceIncidentsPanel.test.tsx index f86025cf9..390e7ebd1 100644 --- a/frontend-modern/src/features/alerts/__tests__/AlertResourceIncidentsPanel.test.tsx +++ b/frontend-modern/src/features/alerts/__tests__/AlertResourceIncidentsPanel.test.tsx @@ -22,6 +22,27 @@ describe('AlertResourceIncidentsPanel', () => { vi.restoreAllMocks(); }); + it('shows persistent read failure without claiming an empty history', () => { + const [failed, setFailed] = createSignal(true); + render(() => ( + ({ resourceId: 'resource-1', resourceName: 'Resource' }), + resourceIncidents: () => ({}), + resourceIncidentLoading: () => ({}), + resourceIncidentError: () => ({ 'resource-1': failed() }), + refreshResourceIncidentPanel: vi.fn(), + } as any + } + /> + )); + expect(screen.getByRole('alert').textContent).toContain('Use Refresh'); + expect(screen.queryByText('No incidents recorded for this resource yet.')).toBeNull(); + setFailed(false); + expect(screen.queryByRole('alert')).toBeNull(); + }); + it('does not surface broad cross-jump links from incident details', () => { // Surface link chips into /infrastructure and broad aggregate workspaces // were retired with the platform-first migration. @@ -47,6 +68,7 @@ describe('AlertResourceIncidentsPanel', () => { }, ], }), + resourceIncidentError: () => ({}), resourceIncidentLoading: () => ({ 'truenas-main': false }), expandedResourceIncidentIds: () => new Set(), resourceIncidentEventFilters: () => new Set(['opened']), @@ -130,6 +152,7 @@ describe('AlertResourceIncidentsPanel', () => { }, ], }), + resourceIncidentError: () => ({}), resourceIncidentLoading: () => ({ 'truenas-main': false }), expandedResourceIncidentIds: () => new Set(), resourceIncidentEventFilters: () => new Set(['command']), @@ -213,6 +236,7 @@ describe('AlertResourceIncidentsPanel', () => { }, ], }), + resourceIncidentError: () => ({}), resourceIncidentLoading: () => ({ 'node-1': false }), expandedResourceIncidentIds: expandedIncidentIds, resourceIncidentEventFilters: eventFilters, diff --git a/frontend-modern/src/features/alerts/__tests__/MobileAlertHistoryInvestigationDialog.test.tsx b/frontend-modern/src/features/alerts/__tests__/MobileAlertHistoryInvestigationDialog.test.tsx new file mode 100644 index 000000000..ccabee5c7 --- /dev/null +++ b/frontend-modern/src/features/alerts/__tests__/MobileAlertHistoryInvestigationDialog.test.tsx @@ -0,0 +1,85 @@ +import { cleanup, fireEvent, render, screen, waitFor } from '@solidjs/testing-library'; +import { Show, createSignal } from 'solid-js'; +import { afterEach, describe, expect, it, vi } from 'vitest'; +import { dialogStackHasBlockingDialog } from '@/components/shared/useDialogState'; +import { aiChatStore } from '@/stores/aiChat'; +import { MobileAlertHistoryInvestigationDialog } from '../MobileAlertHistoryInvestigationDialog'; + +const incident = { + id: 'incident-1', + alertIdentifier: 'alert-1', + alertType: 'backup', + level: 'warning', + resourceId: 'resource-1', + resourceName: 'Backup', + resourceType: 'storage', + status: 'resolved', + openedAt: '2026-09-07T12:00:00Z', + closedAt: '2026-09-07T12:10:00Z', + events: [], +}; + +describe('mobile incident Assistant transition', () => { + afterEach(() => { + cleanup(); + aiChatStore.close(); + aiChatStore.clearAllContext(); + aiChatStore.setEnabled(false); + vi.restoreAllMocks(); + }); + for (const kind of ['timeline', 'resource'] as const) { + it(`dismisses the ${kind} drawer after preserving its incident context`, async () => { + aiChatStore.setEnabled(true); + const nativeOpen = aiChatStore.open; + const openAssistant = vi.spyOn(aiChatStore, 'open').mockImplementation((context) => { + expect(dialogStackHasBlockingDialog()).toBe(false); + nativeOpen(context); + }); + const [open, setOpen] = createSignal(true); + const state = { + incidentLoading: () => ({}), + incidentErrors: () => ({}), + incidentTimelines: () => ({ row: incident }), + historyIncidentEventFilters: () => new Set(), + incidentNoteDrafts: () => ({}), + incidentNoteSaving: () => new Set(), + resourceIncidentPanel: () => ({ + resourceId: 'resource-1', + resourceName: 'Backup', + rowKey: 'row', + }), + resourceIncidents: () => ({ 'resource-1': [incident] }), + resourceIncidentLoading: () => ({}), + resourceIncidentError: () => ({}), + expandedResourceIncidentIds: () => new Set(), + resourceIncidentEventFilters: () => new Set(), + } as any; + render(() => ( + + setOpen(false)} + /> + + )); + expect(screen.getByRole('dialog')).toBeInTheDocument(); + fireEvent.click( + screen.getByRole('button', { name: 'Discuss incident incident-1 with Pulse Assistant' }), + ); + expect(screen.queryByRole('dialog')).toBeNull(); + await waitFor(() => expect(openAssistant).toHaveBeenCalledTimes(1)); + expect(aiChatStore.isOpen).toBe(true); + expect(openAssistant).toHaveBeenCalledWith( + expect.objectContaining({ + targetId: 'resource-1', + autonomousMode: false, + context: expect.objectContaining({ + alertIncidentId: 'incident-1', + alertStatus: 'resolved', + }), + }), + ); + }); + } +}); diff --git a/frontend-modern/src/features/alerts/__tests__/useAlertResourceIncidentsState.test.ts b/frontend-modern/src/features/alerts/__tests__/useAlertResourceIncidentsState.test.ts new file mode 100644 index 000000000..a79a3b504 --- /dev/null +++ b/frontend-modern/src/features/alerts/__tests__/useAlertResourceIncidentsState.test.ts @@ -0,0 +1,48 @@ +import { createRoot } from 'solid-js'; +import { afterEach, describe, expect, it, vi } from 'vitest'; +import { AlertsAPI } from '@/api/alerts'; +import type { Incident } from '@/types/api'; +import { useAlertResourceIncidentsState } from '../useAlertResourceIncidentsState'; + +vi.mock('@/stores/notifications', () => ({ notificationStore: { error: vi.fn() } })); +vi.mock('@/utils/logger', () => ({ logger: { error: vi.fn() } })); + +describe('resource incident read failures', () => { + afterEach(() => vi.restoreAllMocks()); + + it('keeps failed reads distinct from empty evidence, including a cached refresh', async () => { + let dispose!: () => void; + const state = createRoot((cleanup) => { + dispose = cleanup; + return useAlertResourceIncidentsState(); + }); + const incident = { id: 'retained-occurrence', events: [] } as unknown as Incident; + const read = vi.spyOn(AlertsAPI, 'getIncidentsForResource'); + try { + read.mockRejectedValueOnce(new Error('canonical history unavailable')); + await state.openResourceIncidentPanel('resource-1', 'Resource', 'row-1'); + expect(state.resourceIncidentError()['resource-1']).toBe(true); + expect(state.resourceIncidents()['resource-1']).toBeUndefined(); + expect(state.resourceIncidentLoading()['resource-1']).toBe(false); + + read.mockResolvedValueOnce([incident]); + await state.refreshResourceIncidentPanel(); + expect(state.resourceIncidentError()['resource-1']).toBe(false); + expect(state.resourceIncidents()['resource-1']).toEqual([incident]); + + read.mockRejectedValueOnce(new Error('canonical history unavailable')); + await state.refreshResourceIncidentPanel(); + expect(state.resourceIncidentError()['resource-1']).toBe(true); + expect(state.resourceIncidents()['resource-1']).toEqual([incident]); + + read.mockResolvedValueOnce([]); + await state.refreshResourceIncidentPanel(); + expect(state.resourceIncidentError()['resource-1']).toBe(false); + expect(state.resourceIncidents()['resource-1']).toEqual([]); + state.resetResourceIncidentsState(); + expect(state.resourceIncidentError()).toEqual({}); + } finally { + dispose(); + } + }); +}); diff --git a/frontend-modern/src/features/alerts/useAlertHistoryState.ts b/frontend-modern/src/features/alerts/useAlertHistoryState.ts index 76e3e021c..9b6ded81c 100644 --- a/frontend-modern/src/features/alerts/useAlertHistoryState.ts +++ b/frontend-modern/src/features/alerts/useAlertHistoryState.ts @@ -336,6 +336,7 @@ export function useAlertHistoryState(props: UseAlertHistoryStateProps) { setResourceIncidentPanel: resourceIncidentsState.setResourceIncidentPanel, resourceIncidents: resourceIncidentsState.resourceIncidents, resourceIncidentLoading: resourceIncidentsState.resourceIncidentLoading, + resourceIncidentError: resourceIncidentsState.resourceIncidentError, expandedResourceIncidentIds: resourceIncidentsState.expandedResourceIncidentIds, resourceIncidentEventFilters: resourceIncidentsState.resourceIncidentEventFilters, setResourceIncidentEventFilters: resourceIncidentsState.setResourceIncidentEventFilters, diff --git a/frontend-modern/src/features/alerts/useAlertResourceIncidentsState.ts b/frontend-modern/src/features/alerts/useAlertResourceIncidentsState.ts index 6bd7ec971..8f6d549e7 100644 --- a/frontend-modern/src/features/alerts/useAlertResourceIncidentsState.ts +++ b/frontend-modern/src/features/alerts/useAlertResourceIncidentsState.ts @@ -21,6 +21,9 @@ export function useAlertResourceIncidentsState() { const [resourceIncidentLoading, setResourceIncidentLoading] = createSignal< Record >({}); + const [resourceIncidentError, setResourceIncidentError] = createSignal>( + {}, + ); const [expandedResourceIncidentIds, setExpandedResourceIncidentIds] = createSignal>( new Set(), ); @@ -32,10 +35,12 @@ export function useAlertResourceIncidentsState() { if (!resourceId) return; setResourceIncidentLoading((prev) => ({ ...prev, [resourceId]: true })); + setResourceIncidentError((prev) => ({ ...prev, [resourceId]: false })); try { const incidents = await AlertsAPI.getIncidentsForResource(resourceId, limit); setResourceIncidents((prev) => ({ ...prev, [resourceId]: incidents })); } catch (error) { + setResourceIncidentError((prev) => ({ ...prev, [resourceId]: true })); logger.error(getAlertResourceIncidentLoadFailure(), error); notificationStore.error(getAlertResourceIncidentLoadFailure()); } finally { @@ -87,6 +92,7 @@ export function useAlertResourceIncidentsState() { setResourceIncidentPanel(null); setResourceIncidents({}); setResourceIncidentLoading({}); + setResourceIncidentError({}); setExpandedResourceIncidentIds(new Set()); setResourceIncidentEventFilters(new Set(INCIDENT_EVENT_TYPES)); }; @@ -96,6 +102,7 @@ export function useAlertResourceIncidentsState() { setResourceIncidentPanel, resourceIncidents, resourceIncidentLoading, + resourceIncidentError, expandedResourceIncidentIds, resourceIncidentEventFilters, setResourceIncidentEventFilters, diff --git a/frontend-modern/src/hooks/__tests__/useTypeToSearch.test.ts b/frontend-modern/src/hooks/__tests__/useTypeToSearch.test.ts index f5609b5c4..090440c18 100644 --- a/frontend-modern/src/hooks/__tests__/useTypeToSearch.test.ts +++ b/frontend-modern/src/hooks/__tests__/useTypeToSearch.test.ts @@ -53,4 +53,60 @@ describe('type-to-search keyboard ownership', () => { } expect(input.value).toBe('a '); }); + + it.each([false, true])( + 'leaves modal keys alone when background is inert (prepared=%s)', + async (prepared) => { + const background = document.createElement('div'); + background.setAttribute('inert', ''); + const input = document.createElement('input'); + input.value = 'selected occurrence'; + background.append(input); + const modalControl = document.createElement('button'); + document.body.append(background, modalControl); + let preparations = 0; + createRoot((cleanup) => { + dispose = cleanup; + useTypeToSearch({ + getInput: () => input, + prepareInput: prepared + ? () => { + preparations++; + } + : undefined, + clearOnEscape: true, + focusOnShortcut: true, + captureBackspace: true, + getValue: () => input.value, + onClear: () => { + input.value = ''; + }, + }); + }); + modalControl.focus(); + for (const init of [ + { key: 'Escape' }, + { key: 'f', ctrlKey: true }, + { key: 'Backspace' }, + { key: 'x' }, + ]) { + const event = new KeyboardEvent('keydown', { ...init, bubbles: true, cancelable: true }); + modalControl.dispatchEvent(event); + await Promise.resolve(); + expect(event.defaultPrevented).toBe(false); + expect(input.value).toBe('selected occurrence'); + expect(document.activeElement).toBe(modalControl); + } + expect(preparations).toBe(0); + background.removeAttribute('inert'); + const escape = new KeyboardEvent('keydown', { + key: 'Escape', + bubbles: true, + cancelable: true, + }); + modalControl.dispatchEvent(escape); + expect(escape.defaultPrevented).toBe(true); + expect(input.value).toBe(''); + }, + ); }); diff --git a/frontend-modern/src/hooks/useTypeToSearch.ts b/frontend-modern/src/hooks/useTypeToSearch.ts index 4e74bffc1..df9b02a22 100644 --- a/frontend-modern/src/hooks/useTypeToSearch.ts +++ b/frontend-modern/src/hooks/useTypeToSearch.ts @@ -52,6 +52,7 @@ const deleteBackward = (input: HTMLInputElement) => { const isVisibleInput = (input: HTMLInputElement | undefined): input is HTMLInputElement => { if (!input || !input.isConnected || input.disabled) return false; if (input.hidden) return false; + if (input.closest('[inert]')) return false; if (typeof window !== 'undefined') { const style = window.getComputedStyle(input); if (style.display === 'none' || style.visibility === 'hidden') { @@ -110,6 +111,8 @@ const getActiveEntry = ( const entry = registry[index]; if (!readEnabled(entry.enabled)) continue; const input = entry.getInput(); + // Modal background ownership also excludes prepared shortcut targets. + if (input?.closest('[inert]')) continue; const visibleInput = isVisibleInput(input) ? input : undefined; if (!visibleInput && !(options?.allowPrepared && entry.prepareInput)) continue; if (!predicate(entry, visibleInput)) continue; diff --git a/frontend-modern/src/pages/__tests__/Alerts.helpers.test.ts b/frontend-modern/src/pages/__tests__/Alerts.helpers.test.ts index c5a5ce866..0a5428ac7 100644 --- a/frontend-modern/src/pages/__tests__/Alerts.helpers.test.ts +++ b/frontend-modern/src/pages/__tests__/Alerts.helpers.test.ts @@ -906,7 +906,7 @@ describe('incident timeline presentation helpers', () => { it('returns the surface event-card presentation', () => { expect(getAlertIncidentTimelineEventCardClass('surface')).toBe( - 'rounded border border-border bg-surface p-2', + 'rounded border border-border bg-surface p-2 whitespace-normal break-words', ); }); @@ -924,7 +924,9 @@ describe('incident timeline presentation helpers', () => { 'flex flex-wrap items-center gap-2 text-xs text-muted', ); expect(getAlertIncidentTimelineHeadingClass()).toBe('font-medium text-base-content'); - expect(getAlertIncidentTimelineDetailClass()).toBe('mt-1 text-xs text-base-content'); + expect(getAlertIncidentTimelineDetailClass()).toBe( + 'mt-1 text-xs text-base-content whitespace-pre-wrap', + ); expect(getAlertIncidentTimelineCommandClass()).toBe('mt-1 font-mono text-xs text-base-content'); expect(getAlertIncidentTimelineOutputClass()).toBe('mt-1 text-xs text-muted'); }); diff --git a/frontend-modern/src/types/api.ts b/frontend-modern/src/types/api.ts index fceb0202c..29e6a0ee9 100644 --- a/frontend-modern/src/types/api.ts +++ b/frontend-modern/src/types/api.ts @@ -1,6 +1,6 @@ // Properly typed TypeScript interfaces for Pulse API -import type { Resource, ResourceCapability, ResourcePolicy } from './resource'; +import type { Resource, ResourceCapability, ResourcePolicy, ResourceChange } from './resource'; import type { EvidenceEnvelope, LifecycleTransition, OperationalRecord } from './operationalTrust'; export interface ResourceListFacetsPayload { @@ -1423,6 +1423,8 @@ export interface AlertDeliveryDiagnosis { } export interface IncidentEvent { + source?: string; + evidence?: ResourceChange; id: string; type: string; timestamp: string; @@ -1430,7 +1432,17 @@ export interface IncidentEvent { details?: Record; } +export interface IncidentHistoryCoverage { + source: string; + observedSince: string; + observedBefore: string; + changeLimit: number; + hasMoreChanges: boolean; + hasMoreIncidents: boolean; +} + export interface Incident { + history?: IncidentHistoryCoverage; id: string; alertIdentifier: string; alertType: string; diff --git a/frontend-modern/src/utils/__tests__/alertIncidentPresentation.branchcov0717.test.ts b/frontend-modern/src/utils/__tests__/alertIncidentPresentation.branchcov0717.test.ts index 484c02975..4d4ddda3a 100644 --- a/frontend-modern/src/utils/__tests__/alertIncidentPresentation.branchcov0717.test.ts +++ b/frontend-modern/src/utils/__tests__/alertIncidentPresentation.branchcov0717.test.ts @@ -370,13 +370,13 @@ describe('event-filter class helpers — variant + selected branch coverage', () describe('timeline + card class helpers — branch coverage', () => { it('getAlertIncidentTimelineEventCardClass uses bg-surface-alt for variant "alt"', () => { expect(getAlertIncidentTimelineEventCardClass('alt')).toBe( - 'rounded border border-border bg-surface-alt p-2', + 'rounded border border-border bg-surface-alt p-2 whitespace-normal break-words', ); }); it('getAlertIncidentTimelineEventCardClass uses bg-surface for variant "surface"', () => { expect(getAlertIncidentTimelineEventCardClass('surface')).toBe( - 'rounded border border-border bg-surface p-2', + 'rounded border border-border bg-surface p-2 whitespace-normal break-words', ); }); @@ -394,7 +394,9 @@ describe('timeline + card class helpers — branch coverage', () => { 'flex flex-wrap items-center gap-2 text-xs text-muted', ); expect(getAlertIncidentTimelineHeadingClass()).toBe('font-medium text-base-content'); - expect(getAlertIncidentTimelineDetailClass()).toBe('mt-1 text-xs text-base-content'); + expect(getAlertIncidentTimelineDetailClass()).toBe( + 'mt-1 text-xs text-base-content whitespace-pre-wrap', + ); expect(getAlertIncidentTimelineCommandClass()).toBe('mt-1 font-mono text-xs text-base-content'); expect(getAlertIncidentTimelineOutputClass()).toBe('mt-1 text-xs text-muted'); expect(getAlertResourceIncidentCardClass()).toBe('rounded border border-border bg-surface p-3'); diff --git a/frontend-modern/src/utils/__tests__/alertIncidentPresentation.test.ts b/frontend-modern/src/utils/__tests__/alertIncidentPresentation.test.ts index 51b6383e9..21f083db5 100644 --- a/frontend-modern/src/utils/__tests__/alertIncidentPresentation.test.ts +++ b/frontend-modern/src/utils/__tests__/alertIncidentPresentation.test.ts @@ -177,10 +177,10 @@ describe('alertIncidentPresentation', () => { 'px-2 py-0.5 rounded bg-emerald-100 text-emerald-700 dark:bg-emerald-900 dark:text-emerald-300', ); expect(getAlertIncidentTimelineEventCardClass('alt')).toBe( - 'rounded border border-border bg-surface-alt p-2', + 'rounded border border-border bg-surface-alt p-2 whitespace-normal break-words', ); expect(getAlertIncidentTimelineEventCardClass('surface')).toBe( - 'rounded border border-border bg-surface p-2', + 'rounded border border-border bg-surface p-2 whitespace-normal break-words', ); expect(getAlertIncidentNoteTextareaClass()).toBe( 'w-full rounded border border-border bg-surface p-2 text-xs text-base-content', @@ -192,7 +192,9 @@ describe('alertIncidentPresentation', () => { 'flex flex-wrap items-center gap-2 text-xs text-muted', ); expect(getAlertIncidentTimelineHeadingClass()).toBe('font-medium text-base-content'); - expect(getAlertIncidentTimelineDetailClass()).toBe('mt-1 text-xs text-base-content'); + expect(getAlertIncidentTimelineDetailClass()).toBe( + 'mt-1 text-xs text-base-content whitespace-pre-wrap', + ); expect(getAlertIncidentTimelineCommandClass()).toBe('mt-1 font-mono text-xs text-base-content'); expect(getAlertIncidentTimelineOutputClass()).toBe('mt-1 text-xs text-muted'); }); diff --git a/frontend-modern/src/utils/alertIncidentPresentation.ts b/frontend-modern/src/utils/alertIncidentPresentation.ts index a87402e7b..69057299a 100644 --- a/frontend-modern/src/utils/alertIncidentPresentation.ts +++ b/frontend-modern/src/utils/alertIncidentPresentation.ts @@ -234,7 +234,7 @@ export function getAlertIncidentAcknowledgedBadgeClass(): string { } export function getAlertIncidentTimelineEventCardClass(variant: 'surface' | 'alt'): string { - return `rounded border border-border ${variant === 'alt' ? 'bg-surface-alt' : 'bg-surface'} p-2`; + return `rounded border border-border ${variant === 'alt' ? 'bg-surface-alt' : 'bg-surface'} p-2 whitespace-normal break-words`; } export function getAlertIncidentNoteTextareaClass(): string { @@ -254,7 +254,7 @@ export function getAlertIncidentTimelineHeadingClass(): string { } export function getAlertIncidentTimelineDetailClass(): string { - return 'mt-1 text-xs text-base-content'; + return 'mt-1 text-xs text-base-content whitespace-pre-wrap'; } export function getAlertIncidentTimelineCommandClass(): string { @@ -297,3 +297,16 @@ export function getAlertResourceIncidentTruncatedEventsLabel( } return `Showing last ${count} events`; } + +// A zero Go timestamp means the retained evidence does not establish a time. +export function formatIncidentEvidenceTime(value?: string | null): string | null { + if (!value || value.startsWith('0001-')) return null; + const date = new Date(value); + return Number.isNaN(date.getTime()) ? null : date.toLocaleString(); +} + +export const INCIDENT_HISTORY_PARTIAL = + 'Only part of this history was returned. Earlier occurrences or events may be missing.'; +export const INCIDENT_HISTORY_LEGACY = 'This timeline uses saved incident notes and snapshots.'; +export const INCIDENT_EVIDENCE_DETAILS = 'Evidence details'; +export const INCIDENT_TIME_UNAVAILABLE = 'Time unavailable'; diff --git a/internal/ai/cost/coverage_test.go b/internal/ai/cost/coverage_test.go index 6fa19493a..5046e4d82 100644 --- a/internal/ai/cost/coverage_test.go +++ b/internal/ai/cost/coverage_test.go @@ -2,6 +2,7 @@ package cost import ( "errors" + "math" "testing" "time" ) @@ -444,3 +445,21 @@ func TestSummarizeTargets_SortTiebreakers(t *testing.T) { t.Fatalf("expected target ID ordering, got %+v", rollup) } } + +func TestReviewedAstraRouteBudgetBoundary(t *testing.T) { + for _, tc := range []struct { + input int64 + want float64 + }{ + {271999, 2.76999}, + {272000, 5.515}, + } { + usd, known, price := EstimateUSD("openrouter", "openai/gpt-6-astra", tc.input, 1000) + if !known || math.Abs(usd-tc.want) > 0.000001 || price.AsOf != "2026-09-07" { + t.Fatalf("input=%d: usd=%f known=%v price=%+v", tc.input, usd, known, price) + } + } + if _, known, _ := EstimateUSD("openrouter", "openai/gpt-6-astra:free", 1000, 1000); known { + t.Fatal("unreviewed alias must not inherit funded route prices") + } +} diff --git a/internal/ai/cost/pricing.go b/internal/ai/cost/pricing.go index 2f43d6b05..c2717f18c 100644 --- a/internal/ai/cost/pricing.go +++ b/internal/ai/cost/pricing.go @@ -86,6 +86,13 @@ var providerPrices = map[string][]modelPrice{ // published standard price changes on 2027-01-01. Batch/alias routes // are deliberately not covered by this exact model ID. flatPriceAsOf("google/gemini-3.8-flash", 0.75, 3.75, "2026-09-06"), + // Exact funded route reviewed from the Models API on 2026-09-07. + // Its override starts at min_prompt_tokens=272000. Cache discounts + // are omitted so budget estimates remain conservative. + {Pattern: "openai/gpt-6-astra", AsOf: "2026-09-07", Tiers: []priceTier{ + {MaxInputTokens: 271999, InputUSDPerMTok: 10, OutputUSDPerMTok: 50}, + {InputUSDPerMTok: 20, OutputUSDPerMTok: 75}, + }}, flatPriceAsOf("nvidia/nemotron-3.5-lightning:free", 0, 0, "2026-08-14"), flatPriceAsOf("nvidia/nemotron-3-super-120b-a12b:free", 0, 0, "2026-08-14"), flatPriceAsOf("nvidia/nemotron-3-ultra-550b-a55b:free", 0, 0, "2026-08-15"), diff --git a/internal/ai/intelligence.go b/internal/ai/intelligence.go index a6adf6b00..8cfc8482c 100644 --- a/internal/ai/intelligence.go +++ b/internal/ai/intelligence.go @@ -57,21 +57,23 @@ type HealthScore struct { // ResourceIntelligence aggregates all AI knowledge about a single resource type ResourceIntelligence struct { - ResourceID string `json:"resource_id"` - ResourceName string `json:"resource_name,omitempty"` - ResourceType string `json:"resource_type,omitempty"` - Health HealthScore `json:"health"` - ActiveFindings []*Finding `json:"active_findings,omitempty"` - Predictions []patterns.FailurePrediction `json:"predictions,omitempty"` - Dependencies []string `json:"dependencies,omitempty"` // Resources this depends on - Dependents []string `json:"dependents,omitempty"` // Resources that depend on this - Correlations []*correlation.Correlation `json:"correlations,omitempty"` - Baselines map[string]*baseline.FlatBaseline `json:"baselines,omitempty"` - Anomalies []AnomalyReport `json:"anomalies,omitempty"` - RecentIncidents []*memory.Incident `json:"recent_incidents,omitempty"` - RecentChanges []unifiedresources.ResourceChange `json:"recent_changes,omitempty"` - Knowledge *knowledge.GuestKnowledge `json:"knowledge,omitempty"` - NoteCount int `json:"note_count"` + IncidentHistory *memory.IncidentHistoryCoverage `json:"incident_history,omitempty"` + IncidentHistoryError string `json:"incident_history_error,omitempty"` + ResourceID string `json:"resource_id"` + ResourceName string `json:"resource_name,omitempty"` + ResourceType string `json:"resource_type,omitempty"` + Health HealthScore `json:"health"` + ActiveFindings []*Finding `json:"active_findings,omitempty"` + Predictions []patterns.FailurePrediction `json:"predictions,omitempty"` + Dependencies []string `json:"dependencies,omitempty"` // Resources this depends on + Dependents []string `json:"dependents,omitempty"` // Resources that depend on this + Correlations []*correlation.Correlation `json:"correlations,omitempty"` + Baselines map[string]*baseline.FlatBaseline `json:"baselines,omitempty"` + Anomalies []AnomalyReport `json:"anomalies,omitempty"` + RecentIncidents []*memory.Incident `json:"recent_incidents,omitempty"` + RecentChanges []unifiedresources.ResourceChange `json:"recent_changes,omitempty"` + Knowledge *knowledge.GuestKnowledge `json:"knowledge,omitempty"` + NoteCount int `json:"note_count"` } // AnomalyReport describes a metric that's deviating from baseline @@ -347,7 +349,13 @@ func (i *Intelligence) GetResourceIntelligence(resourceID string) *ResourceIntel // Recent incidents if i.incidents != nil { - intel.RecentIncidents = i.incidents.ListIncidentsByResource(resourceID, 5) + page, err := i.incidents.QueryIncidents(memory.IncidentQuery{ResourceID: resourceID, Limit: 5}) + if err != nil { + intel.IncidentHistoryError = "Canonical incident history unavailable" + } else { + intel.RecentIncidents = page.Incidents + intel.IncidentHistory = &page.History + } } // Recent changes diff --git a/internal/ai/memory/incident_query.go b/internal/ai/memory/incident_query.go new file mode 100644 index 000000000..0802529d0 --- /dev/null +++ b/internal/ai/memory/incident_query.go @@ -0,0 +1,388 @@ +package memory + +import ( + "crypto/sha256" + "fmt" + "sort" + "strings" + "time" + + "github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources" +) + +// IncidentQuery selects historical evidence, not current resource health. +// Limits bound returned evidence rather than deciding diagnostic quality. +type IncidentQuery struct { + IncidentID string + ResourceID string + AlertIdentifier string + StartedAt time.Time + Limit int + ChangeLimit int +} + +type IncidentHistoryCoverage struct { + Source string `json:"source"` + ObservedSince time.Time `json:"observedSince"` + ObservedBefore time.Time `json:"observedBefore"` + ChangeLimit int `json:"changeLimit"` + HasMoreChanges bool `json:"hasMoreChanges"` + HasMoreIncidents bool `json:"hasMoreIncidents"` +} + +type IncidentPage struct { + Incidents []*Incident `json:"incidents"` + History IncidentHistoryCoverage `json:"history"` +} + +// QueryIncidents reads canonical evidence before selecting incident rows. Shells +// contribute occurrence identity and attributed local notes, never a second +// authoritative lifecycle. A failed canonical read is not an empty history. +func (s *IncidentStore) QueryIncidents(query IncidentQuery) (IncidentPage, error) { + query.ResourceID = strings.TrimSpace(query.ResourceID) + query.AlertIdentifier = strings.TrimSpace(query.AlertIdentifier) + if query.Limit <= 0 { + query.Limit = 20 + } + if query.Limit > 100 { + return IncidentPage{}, fmt.Errorf("incident limit exceeds 100") + } + if query.ChangeLimit <= 0 { + query.ChangeLimit = 4096 + } + if query.ChangeLimit > 8192 { + return IncidentPage{}, fmt.Errorf("incident change limit exceeds 8192") + } + s.mu.RLock() + store, maxAge := s.resourceTimelineStore, s.maxAge + shells := make([]*incidentShell, 0, len(s.incidents)) + for _, shell := range s.incidents { + if shell != nil { + shells = append(shells, cloneIncidentShell(shell)) + } + } + s.mu.RUnlock() + before := time.Now().UTC() + page := IncidentPage{Incidents: []*Incident{}, History: IncidentHistoryCoverage{Source: "legacy_incident_memory", ChangeLimit: query.ChangeLimit}} + var changes []unifiedresources.ResourceChange + resourceIDs := map[string]bool{query.ResourceID: true} + if store != nil { + page.History.Source = "canonical_resource_history" + page.History.ObservedSince = before.Add(-maxAge) + page.History.ObservedBefore = before + filters := unifiedresources.ResourceChangeFilters{ObservedBefore: &before, Kinds: []unifiedresources.ChangeKind{ + unifiedresources.ChangeAlertFired, unifiedresources.ChangeAlertResolved, + unifiedresources.ChangeAlertAcknowledged, unifiedresources.ChangeAlertUnacknowledged, + unifiedresources.ChangeAlertSnoozed, unifiedresources.ChangeAlertUnsnoozed, + unifiedresources.ChangeCommandExecuted, unifiedresources.ChangeRunbookExecuted, + }} + if query.AlertIdentifier != "" { + filters.AlertIdentifiers = []string{query.AlertIdentifier} + } + var err error + changes, err = store.GetRecentChangesFiltered(query.ResourceID, page.History.ObservedSince, query.ChangeLimit+1, filters) + if err != nil { + return IncidentPage{}, fmt.Errorf("read canonical incident history: %w", err) + } + if len(changes) > query.ChangeLimit { + page.History.HasMoreChanges = true + changes = changes[:query.ChangeLimit] + } + if query.ResourceID != "" { + ids, err := store.ResourceHistoryIDs(query.ResourceID) + if err != nil { + return IncidentPage{}, fmt.Errorf("read incident history identities: %w", err) + } + for _, id := range ids { + resourceIDs[id] = true + } + } + } + // Resource aliases are read identities. A shell for another resource must + // not become an occurrence boundary merely because its alert ID matches. + resourceKeys := make(map[string]string) + identityIDs := make([]string, 0, len(shells)+len(changes)) + for _, shell := range shells { + identityIDs = append(identityIDs, shell.ResourceID) + } + for _, change := range changes { + identityIDs = append(identityIDs, change.ResourceID) + } + for _, id := range identityIDs { + if _, known := resourceKeys[id]; known { + continue + } + ids := []string{id} + if store != nil && id != "" { + aliases, err := store.ResourceHistoryIDs(id) + if err != nil { + return IncidentPage{}, fmt.Errorf("read incident occurrence identities: %w", err) + } + ids = append(ids, aliases...) + } + sort.Strings(ids) + for _, alias := range ids { + resourceKeys[alias] = ids[0] + } + } + sameResource := func(a, b string) bool { return resourceKeys[a] == resourceKeys[b] } + byAlert := make(map[string][]*Incident) + byOccurrence := make(map[string]*Incident) + occurrenceIDs := make(map[*Incident]map[string]bool) + // Historical imports can leave several shells for the same explicit firing. + // Keep a stable existing ID and all local notes, not competing lifecycles. + sort.Slice(shells, func(i, j int) bool { return shells[i].ID < shells[j].ID }) + for _, shell := range shells { + if query.ResourceID != "" && !resourceIDs[shell.ResourceID] { + continue + } + if query.AlertIdentifier != "" && shell.AlertIdentifier != query.AlertIdentifier { + continue + } + incident := incidentFromShell(shell) + for i := range incident.Events { + if incident.Events[i].Source == "" { + incident.Events[i].Source = "legacy_incident_memory" + } + } + key := incident.AlertIdentifier + "\x00" + resourceKeys[incident.ResourceID] + "\x00" + incident.OpenedAt.UTC().Format(time.RFC3339Nano) + if incident.OpenedAt.IsZero() { + key += "\x00" + incident.ID + } + if existing := byOccurrence[key]; existing != nil { + occurrenceIDs[existing][incident.ID] = true + for _, fields := range [][2]*string{{&existing.ResourceID, &incident.ResourceID}, {&existing.ResourceName, &incident.ResourceName}, {&existing.ResourceType, &incident.ResourceType}, {&existing.AlertType, &incident.AlertType}, {&existing.Level, &incident.Level}, {&existing.Message, &incident.Message}, {&existing.Node, &incident.Node}, {&existing.Instance, &incident.Instance}} { + if *fields[0] == "" { + *fields[0] = *fields[1] + } + } + seen := make(map[string]bool, len(existing.Events)) + for _, event := range existing.Events { + seen[event.ID] = true + } + for _, event := range incident.Events { + if !seen[event.ID] { + existing.Events = append(existing.Events, event) + seen[event.ID] = true + } + } + if existing.ClosedAt == nil && incident.ClosedAt != nil { + existing.ClosedAt = incident.ClosedAt + existing.Status = incident.Status + } + continue + } + byOccurrence[key] = incident + occurrenceIDs[incident] = map[string]bool{incident.ID: true} + byAlert[incident.AlertIdentifier] = append(byAlert[incident.AlertIdentifier], incident) + } + // Explicit fired events establish occurrence boundaries even if no shell was + // saved. Sort by occurrence time, retaining the distinct observation time. + sort.Slice(changes, func(i, j int) bool { + a, b := incidentEventTimestamp(changes[i]), incidentEventTimestamp(changes[j]) + if a.Equal(b) { + if changes[i].Kind == unifiedresources.ChangeAlertFired && changes[j].Kind != unifiedresources.ChangeAlertFired { + return true + } + if changes[j].Kind == unifiedresources.ChangeAlertFired && changes[i].Kind != unifiedresources.ChangeAlertFired { + return false + } + return changes[i].ID < changes[j].ID + } + return a.Before(b) + }) + boundaries := make(map[*Incident]time.Time) + firedOwners := make(map[string]*Incident) + canonicalBoundary := make(map[*Incident]bool) + for _, occurrences := range byAlert { + for _, occurrence := range occurrences { + boundaries[occurrence] = occurrence.OpenedAt + } + } + for _, change := range changes { + identifier := projectedAlertIdentifier(change) + if identifier == "" || change.Kind != unifiedresources.ChangeAlertFired { + continue + } + started := incidentEventTimestamp(change) + var occurrence *Incident + for _, candidate := range byAlert[identifier] { + if canonicalBoundary[candidate] { + if sameResource(candidate.ResourceID, change.ResourceID) && boundaries[candidate].Equal(started) { + occurrence = candidate + break + } + continue + } + if incidentStartsMatch(candidate.OpenedAt, started) && (occurrence == nil || incidentStartDelta(candidate.OpenedAt, started) < incidentStartDelta(occurrence.OpenedAt, started)) { + occurrence = candidate + } + } + if occurrence == nil { + occurrence = &Incident{ID: canonicalIncidentID(identifier, started), AlertIdentifier: identifier, OpenedAt: started, Status: IncidentStatusUnknown} + byAlert[identifier] = append(byAlert[identifier], occurrence) + } + hydrateIncidentFromCanonicalChange(occurrence, change) + boundaries[occurrence] = started + firedOwners[change.ID] = occurrence + canonicalBoundary[occurrence] = true + } + for _, occurrences := range byAlert { + sort.Slice(occurrences, func(i, j int) bool { return boundaries[occurrences[i]].Before(boundaries[occurrences[j]]) }) + } + canonicalEvents := make(map[*Incident][]IncidentEvent) + for _, change := range changes { + identifier := projectedAlertIdentifier(change) + if identifier == "" { + continue + } + event, ok := incidentEventFromResourceChange(change) + if !ok { + continue + } + occurrence := firedOwners[change.ID] + if occurrence == nil { + for _, candidate := range byAlert[identifier] { + if change.Kind != unifiedresources.ChangeCommandExecuted && change.Kind != unifiedresources.ChangeRunbookExecuted && !sameResource(candidate.ResourceID, change.ResourceID) { + continue + } + // A capped read may have omitted a later firing after a saved shell. + // Never attach those ambiguous events to the earlier occurrence. + if page.History.HasMoreChanges && !canonicalBoundary[candidate] && !candidate.OpenedAt.IsZero() { + continue + } + if boundaries[candidate].After(event.Timestamp) { + break + } + occurrence = candidate + } + } + if occurrence == nil { + // An event without a retained start cannot establish when the alert + // opened. Preserve it as incomplete evidence with an unknown lifecycle. + for _, candidate := range byAlert[identifier] { + if candidate.OpenedAt.IsZero() && sameResource(candidate.ResourceID, change.ResourceID) { + occurrence = candidate + break + } + } + if occurrence == nil { + occurrence = &Incident{ID: "projected-partial-" + change.ID, AlertIdentifier: identifier, Status: IncidentStatusUnknown} + byAlert[identifier] = append([]*Incident{occurrence}, byAlert[identifier]...) + } + } + openedAt := occurrence.OpenedAt + hydrateIncidentFromCanonicalChange(occurrence, change) + occurrence.OpenedAt = openedAt + canonicalEvents[occurrence] = append(canonicalEvents[occurrence], event) + } + for _, occurrences := range byAlert { + for _, occurrence := range occurrences { + events := canonicalEvents[occurrence] + if len(events) > 0 { + kept := make([]IncidentEvent, 0, len(occurrence.Events)+len(events)) + for _, event := range occurrence.Events { + if isCanonicalProjectedIncidentEventType(event.Type) && (!isSnapshotProjectionEvent(event) || hasIncidentEventType(events, event.Type)) { + continue + } + kept = append(kept, event) + } + occurrence.Events = append(kept, events...) + sortIncidentEvents(occurrence.Events) + openedAt := occurrence.OpenedAt + resetDerivedIncidentState(occurrence) + occurrence.Status = IncidentStatusUnknown + applyProjectedIncidentState(occurrence, occurrence.Events) + occurrence.OpenedAt = openedAt + } + if query.IncidentID != "" && occurrence.ID != query.IncidentID && !occurrenceIDs[occurrence][query.IncidentID] { + continue + } + if !query.StartedAt.IsZero() && !incidentStartsMatch(occurrence.OpenedAt, query.StartedAt) { + continue + } + coverage := page.History + if len(events) == 0 { + coverage.Source = "legacy_incident_memory" + } + occurrence.History = &coverage + page.Incidents = append(page.Incidents, occurrence) + } + } + if !query.StartedAt.IsZero() && len(page.Incidents) > 1 { + closest := page.Incidents[0] + + for _, incident := range page.Incidents[1:] { + if incidentStartDelta(incident.OpenedAt, query.StartedAt) < incidentStartDelta(closest.OpenedAt, query.StartedAt) { + closest = incident + } + } + page.Incidents = []*Incident{closest} + } + sort.Slice(page.Incidents, func(i, j int) bool { + if page.Incidents[i].OpenedAt.Equal(page.Incidents[j].OpenedAt) { + return page.Incidents[i].ID > page.Incidents[j].ID + } + return page.Incidents[i].OpenedAt.After(page.Incidents[j].OpenedAt) + }) + if len(page.Incidents) > query.Limit { + page.History.HasMoreIncidents = true + page.Incidents = page.Incidents[:query.Limit] + } + for _, incident := range page.Incidents { + incident.History.HasMoreIncidents = page.History.HasMoreIncidents + } + return page, nil +} + +func incidentStartsMatch(a, b time.Time) bool { + if a.IsZero() || b.IsZero() { + return false + } + delta := a.Sub(b) + return delta >= -incidentStartMatchTolerance && delta <= incidentStartMatchTolerance +} + +func formatIncidentCoverage(page IncidentPage) string { + if len(page.Incidents) == 0 && page.History.Source == "legacy_incident_memory" { + return "" + } + if page.History.Source == "legacy_incident_memory" { + return "Saved incident memory only. Canonical history coverage was not read. This does not establish current health.\n" + } + text := fmt.Sprintf("Historical incident evidence: %s, observations [%s, %s). Retained history only. This does not establish current health.\n", page.History.Source, page.History.ObservedSince.Format(time.RFC3339), page.History.ObservedBefore.Format(time.RFC3339)) + if page.History.HasMoreChanges || page.History.HasMoreIncidents { + text += "The query is truncated. Earlier occurrences or events may be absent.\n" + } + if len(page.Incidents) == 0 { + text += "No incident evidence was returned in this window.\n" + } + return text +} + +func formatIncidentTime(value time.Time) string { + if value.IsZero() { + return "unknown" + } + return value.Format(time.RFC3339) +} +func formatIncidentOccurredTime(value *time.Time) string { + if value == nil { + return "unknown" + } + return formatIncidentTime(*value) +} + +// The alert identifier and explicit firing time name the occurrence. Event +// arrival order, duplicated observations and resource aliases do not rename it. +func canonicalIncidentID(alertIdentifier string, startedAt time.Time) string { + return fmt.Sprintf("projected-%x", sha256.Sum256([]byte(alertIdentifier+"\x00"+startedAt.UTC().Format(time.RFC3339Nano)))) +} + +func incidentStartDelta(a, b time.Time) time.Duration { + d := a.Sub(b) + if d < 0 { + return -d + } + return d +} diff --git a/internal/ai/memory/incident_query_test.go b/internal/ai/memory/incident_query_test.go new file mode 100644 index 000000000..ecf2ccbee --- /dev/null +++ b/internal/ai/memory/incident_query_test.go @@ -0,0 +1,234 @@ +package memory + +import ( + "errors" + "testing" + "time" + + "github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources" + "github.com/stretchr/testify/require" +) + +func TestIncidentQueryCanonicalOccurrencesAndNotes(t *testing.T) { + for _, backend := range []string{"memory", "sqlite"} { + t.Run(backend, func(t *testing.T) { + var canonical unifiedresources.ResourceStore + if backend == "memory" { + canonical = unifiedresources.NewMemoryStore() + } else { + store, err := unifiedresources.NewSQLiteResourceStore(t.TempDir(), "default") + require.NoError(t, err) + t.Cleanup(func() { require.NoError(t, store.Close()) }) + canonical = store + } + config := IncidentStoreConfig{DataDir: t.TempDir()} + memory := NewIncidentStore(config) + memory.SetResourceTimelineStore(canonical) + start := time.Now().UTC().Add(-time.Hour).Truncate(time.Second) + second := start.Add(2 * time.Minute) + record := func(id string, kind unifiedresources.ChangeKind, at time.Time) { + observed := at.Add(10 * time.Second) + require.NoError(t, canonical.RecordChange(unifiedresources.ResourceChange{ID: id, ResourceID: "resource-canonical", Kind: kind, ObservedAt: observed, OccurredAt: &at, SourceType: unifiedresources.SourcePlatformEvent, SourceAdapter: unifiedresources.AdapterProxmox, Actor: "operator", Confidence: unifiedresources.ConfidenceHigh, Metadata: map[string]any{unifiedresources.MetadataAlertIdentifier: "repeat-alert", "nested": map[string]any{"risk": "storage"}}})) + } + record("first-fired", unifiedresources.ChangeAlertFired, start) + record("first-resolved", unifiedresources.ChangeAlertResolved, start.Add(time.Minute)) + record("second-fired", unifiedresources.ChangeAlertFired, second) + record("second-ack", unifiedresources.ChangeAlertAcknowledged, second.Add(time.Minute)) + page, err := memory.QueryIncidents(IncidentQuery{ResourceID: "resource-canonical", Limit: 10}) + require.NoError(t, err) + require.Len(t, page.Incidents, 2) + require.NotEmpty(t, page.Incidents[0].ID) + require.Equal(t, IncidentStatusOpen, page.Incidents[0].Status) + require.True(t, page.Incidents[0].Acknowledged) + first := page.Incidents[1] + require.NotEqual(t, page.Incidents[0].ID, first.ID) + require.Equal(t, IncidentStatusResolved, first.Status) + require.False(t, first.Acknowledged) + require.Len(t, first.Events, 2) + require.Equal(t, start.Add(time.Minute), *first.ClosedAt) + require.Equal(t, start, first.OpenedAt) + require.Equal(t, "canonical_resource_history", first.Events[0].Source) + require.Equal(t, start.Add(10*time.Second), first.Events[0].Evidence.ObservedAt) + require.Equal(t, start, *first.Events[0].Evidence.OccurredAt) + require.Equal(t, "operator", first.Events[0].Evidence.Actor) + first.Events[0].Evidence.Metadata["nested"].(map[string]any)["risk"] = "mutated" + again, err := memory.QueryIncidents(IncidentQuery{AlertIdentifier: "repeat-alert", StartedAt: start, Limit: 1}) + require.NoError(t, err) + require.Len(t, again.Incidents, 1) + require.Equal(t, "storage", again.Incidents[0].Events[0].Evidence.Metadata["nested"].(map[string]any)["risk"]) + require.True(t, memory.RecordNote("repeat-alert", first.ID, "Check the old pool before replacing it", "operator")) + noted, err := memory.QueryIncidents(IncidentQuery{AlertIdentifier: "repeat-alert", StartedAt: start, Limit: 1}) + require.NoError(t, err) + require.Len(t, noted.Incidents, 1) + require.Len(t, noted.Incidents[0].Events, 3) + require.Equal(t, IncidentStatusResolved, noted.Incidents[0].Status) + require.Equal(t, IncidentEventNote, noted.Incidents[0].Events[2].Type) + require.Equal(t, "operator_note", noted.Incidents[0].Events[2].Source) + require.NoError(t, memory.saveToDisk()) + restored := NewIncidentStore(config) + restored.SetResourceTimelineStore(canonical) + persisted, err := restored.QueryIncidents(IncidentQuery{AlertIdentifier: "repeat-alert", StartedAt: start, Limit: 1}) + require.NoError(t, err) + require.Len(t, persisted.Incidents, 1) + require.Equal(t, first.ID, persisted.Incidents[0].ID) + require.Equal(t, "Check the old pool before replacing it", persisted.Incidents[0].Events[2].Details["note"]) + + latest, err := memory.QueryIncidents(IncidentQuery{AlertIdentifier: "repeat-alert", Limit: 1}) + require.NoError(t, err) + require.True(t, latest.History.HasMoreIncidents) + require.Len(t, latest.Incidents[0].Events, 2) // old note remains attached to old occurrence + formatted := memory.FormatForAlert("repeat-alert", 1) + require.Contains(t, formatted, "Showing latest 1 of 2 recorded events") + require.Contains(t, formatted, "Occurrence evidence source: canonical_resource_history") + bounded, err := memory.QueryIncidents(IncidentQuery{AlertIdentifier: "repeat-alert", ChangeLimit: 1}) + require.NoError(t, err) + require.True(t, bounded.History.HasMoreChanges) + // The saved first occurrence remains separate from incomplete later evidence. + require.Contains(t, formatIncidentCoverage(bounded), "truncated") + }) + } +} + +type failingIncidentHistory struct{ *unifiedresources.MemoryStore } + +func (s failingIncidentHistory) GetRecentChangesFiltered(string, time.Time, int, unifiedresources.ResourceChangeFilters) ([]unifiedresources.ResourceChange, error) { + return nil, errors.New("history read denied") +} + +func TestIncidentQueryUnavailableAndMissingStart(t *testing.T) { + memory := NewIncidentStore(IncidentStoreConfig{}) + memory.SetResourceTimelineStore(failingIncidentHistory{unifiedresources.NewMemoryStore()}) + _, err := memory.QueryIncidents(IncidentQuery{ResourceID: "resource-a"}) + require.ErrorContains(t, err, "history read denied") + require.Contains(t, memory.FormatForResource("resource-a", 5), "unavailable") + require.Contains(t, memory.FormatForAlert("alert", 5), "unavailable") + require.Contains(t, memory.FormatForPatrol(5), "unavailable") + canonical := unifiedresources.NewMemoryStore() + memory.SetResourceTimelineStore(canonical) + now := time.Now().UTC().Add(-time.Minute) + require.NoError(t, canonical.RecordChange(unifiedresources.ResourceChange{ID: "orphan-ack", ResourceID: "resource-a", ObservedAt: now, Kind: unifiedresources.ChangeAlertAcknowledged, Metadata: map[string]any{unifiedresources.MetadataAlertIdentifier: "alert"}})) + page, err := memory.QueryIncidents(IncidentQuery{ResourceID: "resource-a"}) + require.NoError(t, err) + require.Len(t, page.Incidents, 1) + require.Equal(t, IncidentStatus("unknown"), page.Incidents[0].Status) + require.True(t, page.Incidents[0].OpenedAt.IsZero()) + require.True(t, incidentEventTimestamp(unifiedresources.ResourceChange{}).IsZero()) +} + +func TestIncidentQueryDoesNotMergeRapidOccurrences(t *testing.T) { + canonical := unifiedresources.NewMemoryStore() + store := NewIncidentStore(IncidentStoreConfig{}) + store.SetResourceTimelineStore(canonical) + first := time.Now().UTC().Add(-time.Hour).Truncate(time.Second) + for _, at := range []time.Time{first, first.Add(500 * time.Millisecond)} { + change := unifiedresources.BuildAlertTimelineChange("resource-a", unifiedresources.ChangeAlertFired, at, "", unifiedresources.AlertTimelineChange{AlertIdentifier: "rapid"}) + require.NoError(t, canonical.RecordChange(*change)) + } + page, err := store.QueryIncidents(IncidentQuery{AlertIdentifier: "rapid"}) + require.NoError(t, err) + require.Len(t, page.Incidents, 2) + exact, err := store.QueryIncidents(IncidentQuery{AlertIdentifier: "rapid", StartedAt: first, Limit: 1}) + require.NoError(t, err) + require.Len(t, exact.Incidents, 1) + require.Equal(t, first, exact.Incidents[0].OpenedAt) + stableID := exact.Incidents[0].ID + // An independently recorded copy of the same firing is still one occurrence. + duplicate := unifiedresources.BuildAlertTimelineChange("resource-a", unifiedresources.ChangeAlertFired, first, "", unifiedresources.AlertTimelineChange{AlertIdentifier: "rapid"}) + require.NoError(t, canonical.RecordChange(*duplicate)) + exact, err = store.QueryIncidents(IncidentQuery{AlertIdentifier: "rapid", StartedAt: first, Limit: 1}) + require.NoError(t, err) + require.Len(t, exact.Incidents, 1) + require.Equal(t, stableID, exact.Incidents[0].ID) + require.Len(t, exact.Incidents[0].Events, 2) +} + +func TestIncidentQueryMergesDuplicateSavedOccurrencesWithoutLosingNotes(t *testing.T) { + canonical := unifiedresources.NewMemoryStore() + store := NewIncidentStore(IncidentStoreConfig{}) + store.SetResourceTimelineStore(canonical) + start := time.Now().UTC().Add(-time.Hour) + closed := start.Add(time.Minute) + // Reproduce independently imported legacy shells carrying the same firing. + store.incidents = []*incidentShell{ + {ID: "original", AlertIdentifier: "duplicated", ResourceID: "resource-a", OpenedAt: start, Events: []IncidentEvent{{ID: "note-a", Type: IncidentEventNote, Timestamp: start, Details: map[string]any{"note": "first note"}}}}, + {ID: "imported", AlertIdentifier: "duplicated", ResourceID: "resource-a", OpenedAt: start, OccurrenceClosedAt: &closed, Events: []IncidentEvent{{ID: "note-b", Type: IncidentEventNote, Timestamp: closed, Details: map[string]any{"note": "second note"}}}}, + } + for _, kind := range []unifiedresources.ChangeKind{unifiedresources.ChangeAlertFired, unifiedresources.ChangeAlertResolved} { + at := start + if kind == unifiedresources.ChangeAlertResolved { + at = closed + } + change := unifiedresources.BuildAlertTimelineChange("resource-a", kind, at, "", unifiedresources.AlertTimelineChange{AlertIdentifier: "duplicated"}) + require.NoError(t, canonical.RecordChange(*change)) + } + page, err := store.QueryIncidents(IncidentQuery{AlertIdentifier: "duplicated", StartedAt: start}) + require.NoError(t, err) + require.Len(t, page.Incidents, 1) + require.Equal(t, IncidentStatusResolved, page.Incidents[0].Status) + require.Len(t, page.Incidents[0].Events, 4) + for _, id := range []string{"original", "imported"} { + require.True(t, store.RecordNote("duplicated", id, "after merge "+id, "operator")) + aliased, err := store.QueryIncidents(IncidentQuery{IncidentID: id, AlertIdentifier: "duplicated"}) + require.NoError(t, err) + require.Len(t, aliased.Incidents, 1) + require.Equal(t, page.Incidents[0].ID, aliased.Incidents[0].ID) + } + final, err := store.QueryIncidents(IncidentQuery{AlertIdentifier: "duplicated"}) + require.NoError(t, err) + require.Len(t, final.Incidents[0].Events, 6) +} + +func TestIncidentHistorySummaryPreservesLegacyAttribution(t *testing.T) { + page := IncidentPage{History: IncidentHistoryCoverage{Source: "canonical_resource_history"}, Incidents: []*Incident{{ID: "legacy", AlertType: "backup", Status: IncidentStatusUnknown, History: &IncidentHistoryCoverage{Source: "legacy_incident_memory"}}}} + require.Contains(t, FormatIncidentPageForResource(page), "[source=legacy_incident_memory]") +} + +func TestIncidentQueryCanonicalAlertContextOwnsLegacyShell(t *testing.T) { + canonical := unifiedresources.NewMemoryStore() + store := NewIncidentStore(IncidentStoreConfig{}) + store.SetResourceTimelineStore(canonical) + start := time.Now().UTC().Add(-time.Hour) + store.incidents = []*incidentShell{{ + ID: "saved-id", AlertIdentifier: "alert-context", ResourceID: "docker:legacy-id", + ResourceType: "host", AlertType: "old-type", Level: "warning", Message: "old message", OpenedAt: start, + }} + require.NoError(t, canonical.RecordChange(unifiedresources.ResourceChange{ + ID: "canonical-fired", ResourceID: "app-container-current", Kind: unifiedresources.ChangeAlertFired, + ObservedAt: start, OccurredAt: &start, Metadata: map[string]any{ + unifiedresources.MetadataAlertIdentifier: "alert-context", unifiedresources.MetadataAlertType: "docker-container-health", + unifiedresources.MetadataAlertLevel: "critical", unifiedresources.MetadataAlertMessage: "Container health is unhealthy", + "resourceType": "app-container", + }, + })) + require.NoError(t, canonical.RecordChange(unifiedresources.ResourceChange{ + ID: "related-command", ResourceID: "node-executor", Kind: unifiedresources.ChangeCommandExecuted, + ObservedAt: start.Add(time.Minute), Metadata: map[string]any{unifiedresources.MetadataAlertIdentifier: "alert-context"}, + })) + page, err := store.QueryIncidents(IncidentQuery{AlertIdentifier: "alert-context"}) + require.NoError(t, err) + require.Len(t, page.Incidents, 1) + incident := page.Incidents[0] + require.Equal(t, "saved-id", incident.ID) + require.Equal(t, "app-container-current", incident.ResourceID) + require.Equal(t, "app-container", incident.ResourceType) + require.Equal(t, "docker-container-health", incident.AlertType) + require.Equal(t, "critical", incident.Level) + require.Equal(t, "Container health is unhealthy", incident.Message) + require.Equal(t, "docker:legacy-id", store.incidents[0].ResourceID) +} + +func TestIncidentAlertContextRetainsOperatorNote(t *testing.T) { + canonical := unifiedresources.NewMemoryStore() + store := NewIncidentStore(IncidentStoreConfig{DataDir: t.TempDir()}) + store.SetResourceTimelineStore(canonical) + started := time.Now().UTC().Add(-time.Hour) + change := unifiedresources.BuildAlertTimelineChange("resource-a", unifiedresources.ChangeAlertFired, started, "", unifiedresources.AlertTimelineChange{AlertIdentifier: "noted-alert"}) + require.NoError(t, canonical.RecordChange(*change)) + page, err := store.QueryIncidents(IncidentQuery{AlertIdentifier: "noted-alert"}) + require.NoError(t, err) + require.Len(t, page.Incidents, 1) + require.True(t, store.RecordNote("noted-alert", page.Incidents[0].ID, "Keep the old pool until its replacement is verified", "operator")) + context := store.FormatForAlert("noted-alert", 10) + require.Contains(t, context, "Note added by operator: Keep the old pool until its replacement is verified") + require.Contains(t, context, "source=operator_note") +} diff --git a/internal/ai/memory/incidents.go b/internal/ai/memory/incidents.go index 2ca35edb6..64cba24fc 100644 --- a/internal/ai/memory/incidents.go +++ b/internal/ai/memory/incidents.go @@ -22,6 +22,7 @@ type IncidentStatus string const ( IncidentStatusOpen IncidentStatus = "open" + IncidentStatusUnknown IncidentStatus = "unknown" IncidentStatusResolved IncidentStatus = "resolved" ) @@ -43,34 +44,37 @@ const ( // IncidentEvent represents a single timeline entry for an incident. type IncidentEvent struct { - ID string `json:"id"` - Type IncidentEventType `json:"type"` - Timestamp time.Time `json:"timestamp"` - Summary string `json:"summary"` - Details map[string]interface{} `json:"details,omitempty"` + Source string `json:"source,omitempty"` + Evidence *unifiedresources.ResourceChange `json:"evidence,omitempty"` + ID string `json:"id"` + Type IncidentEventType `json:"type"` + Timestamp time.Time `json:"timestamp"` + Summary string `json:"summary"` + Details map[string]interface{} `json:"details,omitempty"` } // Incident captures an alert occurrence and its investigation timeline. // It is an alert-scoped memory/projection for investigation support rather than // the canonical durable resource-change history. type Incident struct { - ID string `json:"id"` - AlertIdentifier string `json:"alertIdentifier"` - AlertType string `json:"alertType"` - Level string `json:"level"` - ResourceID string `json:"resourceId"` - ResourceName string `json:"resourceName"` - ResourceType string `json:"resourceType,omitempty"` - Node string `json:"node,omitempty"` - Instance string `json:"instance,omitempty"` - Message string `json:"message,omitempty"` - Status IncidentStatus `json:"status"` - OpenedAt time.Time `json:"openedAt"` - ClosedAt *time.Time `json:"closedAt,omitempty"` - Acknowledged bool `json:"acknowledged"` - AckUser string `json:"ackUser,omitempty"` - AckTime *time.Time `json:"ackTime,omitempty"` - Events []IncidentEvent `json:"events,omitempty"` + History *IncidentHistoryCoverage `json:"history,omitempty"` + ID string `json:"id"` + AlertIdentifier string `json:"alertIdentifier"` + AlertType string `json:"alertType"` + Level string `json:"level"` + ResourceID string `json:"resourceId"` + ResourceName string `json:"resourceName"` + ResourceType string `json:"resourceType,omitempty"` + Node string `json:"node,omitempty"` + Instance string `json:"instance,omitempty"` + Message string `json:"message,omitempty"` + Status IncidentStatus `json:"status"` + OpenedAt time.Time `json:"openedAt"` + ClosedAt *time.Time `json:"closedAt,omitempty"` + Acknowledged bool `json:"acknowledged"` + AckUser string `json:"ackUser,omitempty"` + AckTime *time.Time `json:"ackTime,omitempty"` + Events []IncidentEvent `json:"events,omitempty"` } type incidentShell struct { @@ -90,23 +94,24 @@ type incidentShell struct { } type incidentJSON struct { - ID string `json:"id"` - AlertIdentifier string `json:"alertIdentifier"` - AlertType string `json:"alertType"` - Level string `json:"level"` - ResourceID string `json:"resourceId"` - ResourceName string `json:"resourceName"` - ResourceType string `json:"resourceType,omitempty"` - Node string `json:"node,omitempty"` - Instance string `json:"instance,omitempty"` - Message string `json:"message,omitempty"` - Status IncidentStatus `json:"status"` - OpenedAt time.Time `json:"openedAt"` - ClosedAt *time.Time `json:"closedAt,omitempty"` - Acknowledged bool `json:"acknowledged"` - AckUser string `json:"ackUser,omitempty"` - AckTime *time.Time `json:"ackTime,omitempty"` - Events []IncidentEvent `json:"events,omitempty"` + History *IncidentHistoryCoverage `json:"history,omitempty"` + ID string `json:"id"` + AlertIdentifier string `json:"alertIdentifier"` + AlertType string `json:"alertType"` + Level string `json:"level"` + ResourceID string `json:"resourceId"` + ResourceName string `json:"resourceName"` + ResourceType string `json:"resourceType,omitempty"` + Node string `json:"node,omitempty"` + Instance string `json:"instance,omitempty"` + Message string `json:"message,omitempty"` + Status IncidentStatus `json:"status"` + OpenedAt time.Time `json:"openedAt"` + ClosedAt *time.Time `json:"closedAt,omitempty"` + Acknowledged bool `json:"acknowledged"` + AckUser string `json:"ackUser,omitempty"` + AckTime *time.Time `json:"ackTime,omitempty"` + Events []IncidentEvent `json:"events,omitempty"` } type incidentShellJSON struct { @@ -133,6 +138,7 @@ type incidentShellJSON struct { func (i Incident) MarshalJSON() ([]byte, error) { alertIdentifier := strings.TrimSpace(i.AlertIdentifier) return json.Marshal(incidentJSON{ + History: i.History, ID: i.ID, AlertIdentifier: alertIdentifier, AlertType: i.AlertType, @@ -162,6 +168,7 @@ func (i *Incident) UnmarshalJSON(data []byte) error { return err } *i = Incident{ + History: payload.History, ID: payload.ID, AlertIdentifier: strings.TrimSpace(payload.AlertIdentifier), AlertType: payload.AlertType, @@ -287,18 +294,19 @@ type IncidentStore struct { } const ( - defaultIncidentMaxIncidents = 500 - defaultIncidentMaxEvents = 120 - defaultIncidentMaxAgeDays = 90 - maxIncidentFileSize = 20 * 1024 * 1024 // 20MB - incidentStartMatchTolerance = time.Second - projectedIncidentChangeLimit = 256 - incidentSnapshotSource = "alert_history_snapshot" + defaultIncidentMaxIncidents = 500 + defaultIncidentMaxEvents = 120 + defaultIncidentMaxAgeDays = 90 + maxIncidentFileSize = 20 * 1024 * 1024 // 20MB + incidentStartMatchTolerance = time.Second + incidentSnapshotSource = "alert_history_snapshot" ) // IncidentTimelineStore exposes the canonical resource timeline used to derive // incident lifecycle and remediation history. type IncidentTimelineStore interface { + GetRecentChangesFiltered(string, time.Time, int, unifiedresources.ResourceChangeFilters) ([]unifiedresources.ResourceChange, error) + ResourceHistoryIDs(string) ([]string, error) GetRecentChanges(canonicalID string, since time.Time, limit int) ([]unifiedresources.ResourceChange, error) } @@ -473,7 +481,7 @@ func (s *IncidentStore) RecordAlertResolved(alert *alerts.Alert, resolvedAt time // EnsureAlertOccurrence materializes the minimum honest timeline carried by // an alert snapshot. It is the read-repair boundary for active alerts and // legacy history entries that predate the canonical resource timeline. When -// canonical events exist, projectIncident replaces these snapshot-derived +// canonical events exist, QueryIncidents replaces these snapshot-derived // lifecycle breadcrumbs with the durable projection. func (s *IncidentStore) EnsureAlertOccurrence(alert *alerts.Alert, resolvedAt *time.Time) *Incident { if s == nil || alert == nil || strings.TrimSpace(alert.ID) == "" { @@ -651,6 +659,15 @@ func (s *IncidentStore) RecordNote(alertIdentifier, incidentID, note, user strin return false } + var projected *Incident + if strings.HasPrefix(incidentID, "projected-") { + page, err := s.QueryIncidents(IncidentQuery{AlertIdentifier: alertIdentifier, IncidentID: incidentID, Limit: 1}) + if err != nil || len(page.Incidents) != 1 { + return false + } + projected = page.Incidents[0] + } + s.mu.Lock() defer s.mu.Unlock() @@ -660,6 +677,14 @@ func (s *IncidentStore) RecordNote(alertIdentifier, incidentID, note, user strin } else if alertIdentifier != "" { shell = s.findLatestIncidentByAlertIdentifierLocked(alertIdentifier) } + if shell == nil && projected != nil { + shell = &incidentShell{ID: projected.ID, AlertIdentifier: projected.AlertIdentifier, AlertType: projected.AlertType, Level: projected.Level, ResourceID: projected.ResourceID, ResourceName: projected.ResourceName, ResourceType: projected.ResourceType, Node: projected.Node, Instance: projected.Instance, Message: projected.Message, OpenedAt: projected.OpenedAt} + if projected.ClosedAt != nil { + closed := *projected.ClosedAt + shell.OccurrenceClosedAt = &closed + } + s.incidents = append(s.incidents, shell) + } if shell == nil { return false } @@ -681,42 +706,24 @@ func (s *IncidentStore) RecordNote(alertIdentifier, incidentID, note, user strin // GetTimelineByAlertIdentifier returns the most recent incident for the alert. func (s *IncidentStore) GetTimelineByAlertIdentifier(alertIdentifier string) *Incident { - if alertIdentifier == "" { - return nil - } - - s.mu.RLock() - shell := cloneIncidentShell(s.findLatestIncidentByAlertIdentifierLocked(alertIdentifier)) - timelineStore := s.resourceTimelineStore - maxAge := s.maxAge - s.mu.RUnlock() - - if shell == nil { - return s.projectIncidentFromCanonical(alertIdentifier, time.Time{}, timelineStore, maxAge) - } - return s.projectIncident(incidentFromShell(shell), timelineStore) + return s.GetTimelineByAlertAt(alertIdentifier, time.Time{}) } -// GetTimelineByAlertAt returns the incident closest to the provided start time for an alert. +// GetTimelineByAlertAt is retained for local writer callers. Runtime readers +// use QueryIncidents to distinguish an unavailable store from absent history. func (s *IncidentStore) GetTimelineByAlertAt(alertIdentifier string, startedAt time.Time) *Incident { - if alertIdentifier == "" { + if strings.TrimSpace(alertIdentifier) == "" { return nil } - if startedAt.IsZero() { - return s.GetTimelineByAlertIdentifier(alertIdentifier) + page, err := s.QueryIncidents(IncidentQuery{AlertIdentifier: alertIdentifier, StartedAt: startedAt, Limit: 1}) + if err != nil { + log.Warn().Err(err).Msg("incident history unavailable") + return nil } - - s.mu.RLock() - best, bestDelta := s.findClosestIncidentByAlertAtLocked(alertIdentifier, startedAt) - best = cloneIncidentShell(best) - timelineStore := s.resourceTimelineStore - maxAge := s.maxAge - s.mu.RUnlock() - - if best == nil || bestDelta > incidentStartMatchTolerance { - return s.projectIncidentFromCanonical(alertIdentifier, startedAt, timelineStore, maxAge) + if len(page.Incidents) == 0 { + return nil } - return s.projectIncident(incidentFromShell(best), timelineStore) + return page.Incidents[0] } func (s *IncidentStore) findIncidentByAlertAtLocked(alertIdentifier string, startedAt time.Time) *incidentShell { @@ -751,56 +758,58 @@ func (s *IncidentStore) findClosestIncidentByAlertAtLocked(alertIdentifier strin // ListIncidentsByResource returns recent incidents for a resource. func (s *IncidentStore) ListIncidentsByResource(resourceID string, limit int) []*Incident { - if resourceID == "" { + if strings.TrimSpace(resourceID) == "" { return nil } - - s.mu.RLock() - var matches []*Incident - for i := len(s.incidents) - 1; i >= 0; i-- { - shell := s.incidents[i] - if shell != nil && shell.ResourceID == resourceID { - matches = append(matches, incidentFromShell(cloneIncidentShell(shell))) - if limit > 0 && len(matches) >= limit { - break - } - } + page, err := s.QueryIncidents(IncidentQuery{ResourceID: resourceID, Limit: limit}) + if err != nil { + log.Warn().Err(err).Msg("incident history unavailable") + return nil } - timelineStore := s.resourceTimelineStore - s.mu.RUnlock() - - if timelineStore == nil { - return matches - } - projected := make([]*Incident, 0, len(matches)) - for _, incident := range matches { - projected = append(projected, s.projectIncident(incident, timelineStore)) - } - return projected + return page.Incidents } // FormatForAlert returns a condensed incident timeline for prompt injection. func (s *IncidentStore) FormatForAlert(alertIdentifier string, maxEvents int) string { - incident := s.GetTimelineByAlertIdentifier(alertIdentifier) - if incident == nil { - return "" + page, err := s.QueryIncidents(IncidentQuery{AlertIdentifier: alertIdentifier, Limit: 1}) + if err != nil { + return "\n\nIncident history unavailable: canonical evidence could not be read.\n" } + if len(page.Incidents) == 0 { + return formatIncidentCoverage(page) + } + incident := page.Incidents[0] var b strings.Builder b.WriteString("\n\n## Incident Memory\n") + b.WriteString(formatIncidentCoverage(page)) b.WriteString(fmt.Sprintf("Alert incident for %s (%s, %s)\n", incident.ResourceName, incident.AlertType, incident.Level)) b.WriteString(fmt.Sprintf("Status: %s\n", incident.Status)) + if incident.History != nil { + b.WriteString("Occurrence evidence source: " + incident.History.Source + "\n") + } events := incident.Events if maxEvents > 0 && len(events) > maxEvents { events = events[len(events)-maxEvents:] + b.WriteString(fmt.Sprintf("Showing latest %d of %d recorded events.\n", len(events), len(incident.Events))) } for _, evt := range events { b.WriteString("- ") - b.WriteString(evt.Timestamp.Format(time.RFC3339)) + b.WriteString(formatIncidentTime(evt.Timestamp)) b.WriteString(": ") b.WriteString(evt.Summary) + if evt.Type == IncidentEventNote { + if note, ok := stringMetadata(evt.Details, "note"); ok { + b.WriteString(": " + note) + } + } + if evt.Evidence != nil { + b.WriteString(fmt.Sprintf(" [record=%s observed=%s occurred=%s source=%s actor=%s]", evt.Evidence.ID, formatIncidentTime(evt.Evidence.ObservedAt), formatIncidentOccurredTime(evt.Evidence.OccurredAt), evt.Evidence.SourceAdapter, evt.Evidence.Actor)) + } else if evt.Source != "" { + b.WriteString(" [source=" + evt.Source + "]") + } b.WriteString("\n") } return b.String() @@ -808,21 +817,32 @@ func (s *IncidentStore) FormatForAlert(alertIdentifier string, maxEvents int) st // FormatForResource returns a condensed incident summary for a resource. func (s *IncidentStore) FormatForResource(resourceID string, limit int) string { - incidents := s.ListIncidentsByResource(resourceID, limit) + page, err := s.QueryIncidents(IncidentQuery{ResourceID: resourceID, Limit: limit}) + if err != nil { + return "\n\nIncident history unavailable: canonical evidence could not be read.\n" + } + return FormatIncidentPageForResource(page) +} + +// FormatIncidentPageForResource formats the same snapshot returned to an API +// caller, so a second read cannot silently describe a different history. +func FormatIncidentPageForResource(page IncidentPage) string { + incidents := page.Incidents if len(incidents) == 0 { - return "" + return formatIncidentCoverage(page) } var b strings.Builder b.WriteString("\n\n## Incident Memory\n") - b.WriteString("Recent incidents for this resource:\n") + b.WriteString(formatIncidentCoverage(page)) + b.WriteString("Recent incidents:\n") for _, incident := range incidents { status := string(incident.Status) if incident.Acknowledged && incident.Status == IncidentStatusOpen { status = "acknowledged" } b.WriteString("- ") - b.WriteString(incident.OpenedAt.Format(time.RFC3339)) + b.WriteString(formatIncidentTime(incident.OpenedAt)) b.WriteString(": ") b.WriteString(incident.AlertType) if incident.Level != "" { @@ -832,6 +852,9 @@ func (s *IncidentStore) FormatForResource(resourceID string, limit int) string { } b.WriteString(" - ") b.WriteString(status) + if incident.History != nil { + b.WriteString(" [source=" + incident.History.Source + "]") + } b.WriteString("\n") } return b.String() @@ -843,28 +866,18 @@ func (s *IncidentStore) FormatForPatrol(limit int) string { limit = 8 } - s.mu.RLock() - snapshot := make([]*Incident, 0, len(s.incidents)) - for i := len(s.incidents) - 1; i >= 0 && len(snapshot) < limit; i-- { - if shell := s.incidents[i]; shell != nil { - snapshot = append(snapshot, incidentFromShell(cloneIncidentShell(shell))) - } + page, err := s.QueryIncidents(IncidentQuery{Limit: limit}) + if err != nil { + return "\n\nIncident history unavailable: canonical evidence could not be read.\n" } - timelineStore := s.resourceTimelineStore - s.mu.RUnlock() - + snapshot := page.Incidents if len(snapshot) == 0 { - return "" - } - - if timelineStore != nil { - for i := range snapshot { - snapshot[i] = s.projectIncident(snapshot[i], timelineStore) - } + return formatIncidentCoverage(page) } var b strings.Builder b.WriteString("\n\n## Incident Memory\n") + b.WriteString(formatIncidentCoverage(page)) b.WriteString("Recent incidents across infrastructure:\n") for _, incident := range snapshot { @@ -879,7 +892,7 @@ func (s *IncidentStore) FormatForPatrol(limit int) string { } b.WriteString("- ") - b.WriteString(incident.OpenedAt.Format(time.RFC3339)) + b.WriteString(formatIncidentTime(incident.OpenedAt)) b.WriteString(": ") if incident.ResourceName != "" { b.WriteString(incident.ResourceName) @@ -895,6 +908,9 @@ func (s *IncidentStore) FormatForPatrol(limit int) string { } b.WriteString(" - ") b.WriteString(status) + if incident.History != nil { + b.WriteString(" [source=" + incident.History.Source + "]") + } if lastSummary != "" { b.WriteString(" - last: ") b.WriteString(truncateOutput(lastSummary, 80)) @@ -912,162 +928,34 @@ func (s *IncidentStore) projectsFromCanonicalLocked() bool { return s.resourceTimelineStore != nil } -func (s *IncidentStore) projectIncident(incident *Incident, timelineStore IncidentTimelineStore) *Incident { - if incident == nil || timelineStore == nil { - return incident - } - - projectedEvents := s.loadProjectedIncidentEvents(incident, timelineStore) - if len(projectedEvents) == 0 { - return incident - } - - projected := cloneIncident(incident) - resetDerivedIncidentState(projected) - filtered := make([]IncidentEvent, 0, len(projected.Events)+len(projectedEvents)) - for _, event := range projected.Events { - if isCanonicalProjectedIncidentEventType(event.Type) { - if !isSnapshotProjectionEvent(event) || hasIncidentEventType(projectedEvents, event.Type) { - continue - } - } - filtered = append(filtered, cloneIncidentEvent(event)) - } - filtered = append(filtered, projectedEvents...) - sortIncidentEvents(filtered) - projected.Events = filtered - applyProjectedIncidentState(projected, filtered) - return projected -} - -func (s *IncidentStore) projectIncidentFromCanonical(alertIdentifier string, startedAt time.Time, timelineStore IncidentTimelineStore, maxAge time.Duration) *Incident { - if timelineStore == nil || strings.TrimSpace(alertIdentifier) == "" { - return nil - } - - since := time.Now().Add(-defaultIncidentMaxAgeDays * 24 * time.Hour) - if maxAge > 0 { - since = time.Now().Add(-maxAge) - } - - changes, err := timelineStore.GetRecentChanges("", since, projectedIncidentChangeLimit) - if err != nil || len(changes) == 0 { - return nil - } - - events := make([]IncidentEvent, 0, len(changes)) - projected := &Incident{ - ID: "projected-" + strings.TrimSpace(alertIdentifier), - AlertIdentifier: strings.TrimSpace(alertIdentifier), - Status: IncidentStatusOpen, - } - for _, change := range changes { - if projectedAlertIdentifier(change) != projected.AlertIdentifier { - continue - } - event, ok := incidentEventFromResourceChange(change) - if !ok { - continue - } - events = append(events, event) - hydrateIncidentFromCanonicalChange(projected, change) - } - if len(events) == 0 { - return nil - } - - sortIncidentEvents(events) - projected.Events = events - if !startedAt.IsZero() { - openAt := projected.OpenedAt - if openAt.IsZero() { - openAt = events[0].Timestamp - } - delta := openAt.Sub(startedAt) - if delta < 0 { - delta = -delta - } - if delta > incidentStartMatchTolerance { - return nil - } - } - applyProjectedIncidentState(projected, events) - return projected -} - -func (s *IncidentStore) loadProjectedIncidentEvents(incident *Incident, timelineStore IncidentTimelineStore) []IncidentEvent { - if incident == nil || timelineStore == nil { - return nil - } - - resourceID := strings.TrimSpace(incident.ResourceID) - alertIdentifier := strings.TrimSpace(incident.AlertIdentifier) - if resourceID == "" || alertIdentifier == "" { - return nil - } - - since := incident.OpenedAt - if since.IsZero() { - since = time.Now().Add(-defaultIncidentMaxAgeDays * 24 * time.Hour) - } else { - since = since.Add(-incidentStartMatchTolerance) - } - - // Retained shells identify exact occurrences. Canonical history is keyed by - // alert identifier, so without an upper bound an old incident absorbs every - // later recurrence and can appear open again. Do not use the legacy read - // tolerance to include events from a preceding subsecond occurrence either. - var nextStart time.Time - s.mu.RLock() - for _, shell := range s.incidents { - if shell != nil && shell.AlertIdentifier == alertIdentifier && shell.ResourceID == resourceID && - shell.OpenedAt.After(incident.OpenedAt) && (nextStart.IsZero() || shell.OpenedAt.Before(nextStart)) { - nextStart = shell.OpenedAt - } - } - s.mu.RUnlock() - - changes, err := timelineStore.GetRecentChanges(resourceID, since, projectedIncidentChangeLimit) - if err != nil || len(changes) == 0 { - return nil - } - - events := make([]IncidentEvent, 0, len(changes)) - for _, change := range changes { - if projectedAlertIdentifier(change) != alertIdentifier { - continue - } - event, ok := incidentEventFromResourceChange(change) - if !ok { - continue - } - if (!incident.OpenedAt.IsZero() && event.Timestamp.Before(incident.OpenedAt)) || - (!nextStart.IsZero() && !event.Timestamp.Before(nextStart)) { - continue - } - events = append(events, event) - hydrateIncidentFromCanonicalChange(incident, change) - } - sortIncidentEvents(events) - return events -} - func hydrateIncidentFromCanonicalChange(incident *Incident, change unifiedresources.ResourceChange) { if incident == nil { return } - if resourceID := strings.TrimSpace(change.ResourceID); resourceID != "" && incident.ResourceID == "" { + // Alert evidence owns its target and risk context. A command may execute on + // a related host, so its target must not rename the alert's resource. + alertEvidence := false + switch change.Kind { + case unifiedresources.ChangeAlertFired, unifiedresources.ChangeAlertResolved, + unifiedresources.ChangeAlertAcknowledged, unifiedresources.ChangeAlertUnacknowledged, + unifiedresources.ChangeAlertSnoozed, unifiedresources.ChangeAlertUnsnoozed: + alertEvidence = true + } + if resourceID := strings.TrimSpace(change.ResourceID); resourceID != "" && (incident.ResourceID == "" || alertEvidence) { incident.ResourceID = resourceID } - if alertType, ok := stringMetadata(change.Metadata, unifiedresources.MetadataAlertType); ok && incident.AlertType == "" { + if alertType, ok := stringMetadata(change.Metadata, unifiedresources.MetadataAlertType); ok && (incident.AlertType == "" || alertEvidence) { incident.AlertType = alertType } - if level, ok := stringMetadata(change.Metadata, unifiedresources.MetadataAlertLevel); ok && incident.Level == "" { + if level, ok := stringMetadata(change.Metadata, unifiedresources.MetadataAlertLevel); ok && (incident.Level == "" || alertEvidence) { incident.Level = level } - if message, ok := stringMetadata(change.Metadata, unifiedresources.MetadataAlertMessage); ok && incident.Message == "" { + if message, ok := stringMetadata(change.Metadata, unifiedresources.MetadataAlertMessage); ok && (incident.Message == "" || alertEvidence) { incident.Message = message } + if resourceType, ok := stringMetadata(change.Metadata, "resourceType"); ok && (incident.ResourceType == "" || alertEvidence) { + incident.ResourceType = resourceType + } if incident.OpenedAt.IsZero() { incident.OpenedAt = incidentEventTimestamp(change) } @@ -1119,7 +1007,16 @@ func incidentEventFromResourceChange(change unifiedresources.ResourceChange) (In } } + evidence := change + evidence.Metadata = cloneIncidentEventDetails(change.Metadata) + evidence.RelatedResources = append([]string(nil), change.RelatedResources...) + if change.OccurredAt != nil { + occurred := *change.OccurredAt + evidence.OccurredAt = &occurred + } return IncidentEvent{ + Source: "canonical_resource_history", + Evidence: &evidence, ID: strings.TrimSpace(change.ID), Type: eventType, Timestamp: incidentEventTimestamp(change), @@ -1154,17 +1051,19 @@ func incidentEventTypeFromChangeKind(kind unifiedresources.ChangeKind) (Incident func incidentEventSummaryFromChange(change unifiedresources.ResourceChange, eventType IncidentEventType) string { switch eventType { case IncidentEventAlertFired: + // The source owns the alert condition and comparison direction. Numeric + // placeholders cannot establish either, including for legacy records. + if message, ok := stringMetadata(change.Metadata, unifiedresources.MetadataAlertMessage); ok { + return message + } + if reason := strings.TrimSpace(change.Reason); reason != "" { + return reason + } if alertType, ok := stringMetadata(change.Metadata, unifiedresources.MetadataAlertType); ok { - level, _ := stringMetadata(change.Metadata, unifiedresources.MetadataAlertLevel) - value, hasValue := floatMetadata(change.Metadata, unifiedresources.MetadataAlertValue) - threshold, hasThreshold := floatMetadata(change.Metadata, unifiedresources.MetadataAlertThreshold) - // Resource incidents carry numeric zero placeholders, not threshold evidence. - if alertType != "resource-incident" && (hasValue || hasThreshold) { - return fmt.Sprintf("Alert triggered: %s (%s %.1f >= %.1f)", alertType, level, value, threshold) - } - if level != "" { + if level, ok := stringMetadata(change.Metadata, unifiedresources.MetadataAlertLevel); ok { return fmt.Sprintf("Alert triggered: %s (%s)", alertType, level) } + return "Alert triggered: " + alertType } return "Alert triggered" case IncidentEventAlertAcknowledged: @@ -1192,7 +1091,7 @@ func incidentEventTimestamp(change unifiedresources.ResourceChange) time.Time { if !change.ObservedAt.IsZero() { return change.ObservedAt.UTC() } - return time.Now().UTC() + return time.Time{} } func projectedAlertIdentifier(change unifiedresources.ResourceChange) string { @@ -1219,28 +1118,6 @@ func stringMetadata(metadata map[string]any, key string) (string, bool) { return str, true } -func floatMetadata(metadata map[string]any, key string) (float64, bool) { - if len(metadata) == 0 { - return 0, false - } - value, ok := metadata[key] - if !ok { - return 0, false - } - switch typed := value.(type) { - case float64: - return typed, true - case float32: - return float64(typed), true - case int: - return float64(typed), true - case int64: - return float64(typed), true - default: - return 0, false - } -} - func eventActor(event IncidentEvent) string { if event.Details == nil { return "" @@ -1304,6 +1181,16 @@ func incidentEventTimestampRank(eventType IncidentEventType) int { func cloneIncidentEvent(event IncidentEvent) IncidentEvent { cloned := event cloned.Details = cloneIncidentEventDetails(event.Details) + if event.Evidence != nil { + evidence := *event.Evidence + evidence.Metadata = cloneIncidentEventDetails(evidence.Metadata) + evidence.RelatedResources = append([]string(nil), evidence.RelatedResources...) + if evidence.OccurredAt != nil { + occurred := *evidence.OccurredAt + evidence.OccurredAt = &occurred + } + cloned.Evidence = &evidence + } return cloned } @@ -1313,11 +1200,28 @@ func cloneIncidentEventDetails(details map[string]any) map[string]interface{} { } cloned := make(map[string]interface{}, len(details)) for key, value := range details { - cloned[key] = value + cloned[key] = cloneIncidentDetailValue(value) } return cloned } +func cloneIncidentDetailValue(value any) any { + switch v := value.(type) { + case map[string]any: + return cloneIncidentEventDetails(v) + case []any: + out := make([]any, len(v)) + for i, item := range v { + out[i] = cloneIncidentDetailValue(item) + } + return out + case []string: + return append([]string(nil), v...) + default: + return value + } +} + func newIncidentShellFromAlert(alert *alerts.Alert) *incidentShell { openedAt := alert.StartTime if openedAt.IsZero() { @@ -1415,6 +1319,13 @@ func (s *IncidentStore) addEventAtLocked(shell *incidentShell, eventType Inciden Summary: summary, Details: details, } + if eventType == IncidentEventNote { + event.Source = "operator_note" + } + if isSnapshotProjectionEvent(event) { + event.Source = incidentSnapshotSource + } + shell.Events = append(shell.Events, event) if s.maxEvents > 0 && len(shell.Events) > s.maxEvents { shell.Events = shell.Events[len(shell.Events)-s.maxEvents:] @@ -1660,7 +1571,16 @@ func incidentFromShell(shell *incidentShell) *Incident { OpenedAt: shell.OpenedAt, Events: cloneIncidentEvents(shell.Events), } + if shell.OpenedAt.IsZero() { + incident.Status = IncidentStatusUnknown + } + if shell.OccurrenceClosedAt != nil { + closed := *shell.OccurrenceClosedAt + incident.ClosedAt = &closed + incident.Status = IncidentStatusResolved + } applyProjectedIncidentState(incident, incident.Events) + incident.OpenedAt = shell.OpenedAt return incident } @@ -1677,20 +1597,12 @@ func cloneIncident(src *Incident) *Incident { t := *src.ClosedAt clone.ClosedAt = &t } - if len(src.Events) > 0 { - clone.Events = make([]IncidentEvent, len(src.Events)) - for i, event := range src.Events { - cloneEvent := event - if event.Details != nil { - detailsCopy := make(map[string]interface{}, len(event.Details)) - for key, value := range event.Details { - detailsCopy[key] = value - } - cloneEvent.Details = detailsCopy - } - clone.Events[i] = cloneEvent - } + clone.Events = cloneIncidentEvents(src.Events) + if src.History != nil { + history := *src.History + clone.History = &history } + return &clone } diff --git a/internal/ai/memory/incidents_test.go b/internal/ai/memory/incidents_test.go index f7d6a7423..0d218a9a2 100644 --- a/internal/ai/memory/incidents_test.go +++ b/internal/ai/memory/incidents_test.go @@ -838,7 +838,7 @@ func TestIncidentStore_FormatForResource(t *testing.T) { if !strings.Contains(result, "## Incident Memory") { t.Error("expected '## Incident Memory' header") } - if !strings.Contains(result, "Recent incidents for this resource") { + if !strings.Contains(result, "Recent incidents:") { t.Error("expected resource incidents header") } if !strings.Contains(result, "disk") { @@ -1209,7 +1209,9 @@ func TestIncidentStore_CanonicalProjectionOccurrenceBounds(t *testing.T) { // Checkpoint all retained boundaries, not just an isolated closed // shell. The canonical timeline remains in memory: this exercises // JSON recovery and projection, not a durable event-store restart. - before, err := json.Marshal(projected) + beforeProjection := *projected + beforeProjection.History = nil // Query coverage is refreshed on each read. + before, err := json.Marshal(beforeProjection) if err != nil { t.Fatal(err) } @@ -1230,7 +1232,21 @@ func TestIncidentStore_CanonicalProjectionOccurrenceBounds(t *testing.T) { if len(reloaded.incidents) != len(store.incidents) { t.Fatal("checkpoint replay duplicated a retained occurrence") } - after, err := json.Marshal(reloaded.GetTimelineByAlertAt(old.ID, start)) + afterProjection := reloaded.GetTimelineByAlertAt(old.ID, start) + if afterProjection == nil || projected.History == nil || afterProjection.History == nil { + t.Fatal("checkpoint replay lost the projection or its query coverage") + } + if afterProjection.History.ObservedBefore.Before(projected.History.ObservedBefore) || afterProjection.History.ObservedSince.Before(projected.History.ObservedSince) { + t.Fatal("fresh query coverage moved backwards") + } + beforeCoverage, afterCoverage := *projected.History, *afterProjection.History + afterCoverage.ObservedSince = beforeCoverage.ObservedSince + afterCoverage.ObservedBefore = beforeCoverage.ObservedBefore + if beforeCoverage != afterCoverage { + t.Fatalf("checkpoint replay changed coverage semantics: before %+v, after %+v", beforeCoverage, afterCoverage) + } + afterProjection.History = nil + after, err := json.Marshal(afterProjection) if err != nil { t.Fatal(err) } diff --git a/internal/ai/memory/provider_incident_summary_test.go b/internal/ai/memory/provider_incident_summary_test.go index 77812107c..881b3d9de 100644 --- a/internal/ai/memory/provider_incident_summary_test.go +++ b/internal/ai/memory/provider_incident_summary_test.go @@ -6,19 +6,30 @@ import ( ) func TestProviderIncidentSummaryDoesNotInventThreshold(t *testing.T) { - for _, typ := range []string{"resource-incident", "cpu"} { + for _, typ := range []string{"resource-incident", "backup-storage-incident", "cpu"} { change := unifiedresources.ResourceChange{Metadata: map[string]interface{}{ unifiedresources.MetadataAlertType: typ, unifiedresources.MetadataAlertLevel: "warning", unifiedresources.MetadataAlertValue: float64(0), unifiedresources.MetadataAlertThreshold: float64(0), }} - want := "Alert triggered: resource-incident (warning)" - if typ == "cpu" { - want = "Alert triggered: cpu (warning 0.0 >= 0.0)" - } + want := "Alert triggered: " + typ + " (warning)" if got := incidentEventSummaryFromChange(change, IncidentEventAlertFired); got != want { t.Fatalf("%s: got %q, want %q", typ, got, want) } } } + +func TestIncidentSummaryUsesSourceConditionWithoutInventingComparison(t *testing.T) { + for _, message := range []string{"Backup datastore is 90% full and protects seven workloads", "Free space fell below the configured minimum"} { + change := unifiedresources.ResourceChange{Reason: message, Metadata: map[string]any{ + unifiedresources.MetadataAlertType: "backup-storage-incident", + unifiedresources.MetadataAlertMessage: message, + unifiedresources.MetadataAlertValue: 0.0, + unifiedresources.MetadataAlertThreshold: 0.0, + }} + if got := incidentEventSummaryFromChange(change, IncidentEventAlertFired); got != message { + t.Fatalf("got %q, want recorded condition %q", got, message) + } + } +} diff --git a/internal/ai/service.go b/internal/ai/service.go index d3e0e8c6e..13a637f48 100644 --- a/internal/ai/service.go +++ b/internal/ai/service.go @@ -4999,11 +4999,15 @@ func (s *Service) lookupIncidentResourceIdentifier(alertIdentifier string) strin if store == nil { return "" } - timeline := store.GetTimelineByAlertIdentifier(alertIdentifier) - if timeline == nil { + page, err := store.QueryIncidents(memory.IncidentQuery{AlertIdentifier: alertIdentifier, Limit: 1}) + if err != nil { + log.Warn().Err(err).Msg("Canonical incident resource lookup unavailable") return "" } - return strings.TrimSpace(timeline.ResourceID) + if len(page.Incidents) == 0 { + return "" + } + return strings.TrimSpace(page.Incidents[0].ResourceID) } func (s *Service) recordCanonicalResourceChange(change *unifiedresources.ResourceChange) { diff --git a/internal/ai/tools/tools_knowledge.go b/internal/ai/tools/tools_knowledge.go index 21b82aac9..86bde2071 100644 --- a/internal/ai/tools/tools_knowledge.go +++ b/internal/ai/tools/tools_knowledge.go @@ -219,7 +219,7 @@ func (e *PulseToolExecutor) executeGetIncidentWindow(_ context.Context, args map "limit": limit, "has_more": hasMore, "coverage": "retained_records_only", - "evidence_limit": "These are retained observations, not continuous coverage. Empty history does not establish health or absence of incidents. ObservedAt is when Pulse observed a change, while OccurredAt is present only when its occurrence time is known. An alert resolving establishes that alert's recovery, not its cause or a verified action outcome.", + "evidence_limit": "These are retained observations, not continuous coverage. Empty history does not establish health or absence of incidents. ObservedAt is when Pulse observed a change, while OccurredAt is present only when its occurrence time is known. A resolution records alert closure, not its cause, workload recovery or a verified action outcome. RelatedResources identifies relationships, not additional targets of this event. This read does not query action records.", }), nil } diff --git a/internal/api/ai_intelligence_handlers.go b/internal/api/ai_intelligence_handlers.go index 7eba442a4..c2faafe8b 100644 --- a/internal/api/ai_intelligence_handlers.go +++ b/internal/api/ai_intelligence_handlers.go @@ -9,6 +9,7 @@ import ( "github.com/rcourtman/pulse-go-rewrite/internal/ai" "github.com/rcourtman/pulse-go-rewrite/internal/ai/baseline" + "github.com/rcourtman/pulse-go-rewrite/internal/ai/memory" "github.com/rcourtman/pulse-go-rewrite/internal/ai/unified" "github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources" "github.com/rcourtman/pulse-go-rewrite/internal/utils" @@ -1464,31 +1465,21 @@ func (h *AISettingsHandler) HandleGetRecentIncidents(w http.ResponseWriter, r *h // Get the resource ID filter if provided resourceID := r.URL.Query().Get("resource_id") - var incidents interface{} - if resourceID != "" { - incidents = incidentStore.ListIncidentsByResource(resourceID, limit) - } else { - // No direct method to list all incidents, use FormatForPatrol for now - // This is a limitation - we may want to add ListRecentIncidents to the store - incidentSummary := incidentStore.FormatForPatrol(limit) - if err := utils.WriteJSONResponse(w, map[string]interface{}{ - "incidents": []interface{}{}, - "incident_summary": incidentSummary, - "active_count": nil, - "active_count_status": "not_measured", - }); err != nil { - log.Error().Err(err).Msg("Failed to write incidents response") - } + page, err := incidentStore.QueryIncidents(memory.IncidentQuery{ResourceID: resourceID, Limit: limit}) + if err != nil { + http.Error(w, "Incident history unavailable", http.StatusServiceUnavailable) return } - if err := utils.WriteJSONResponse(w, map[string]interface{}{ - "incidents": incidents, + "incidents": page.Incidents, + "history": page.History, + "incident_summary": memory.FormatIncidentPageForResource(page), "active_count": nil, "active_count_status": "not_measured", }); err != nil { log.Error().Err(err).Msg("Failed to write incidents response") } + } // HandleGetIncidentData returns incident data for a specific resource (GET /api/ai/incidents/{resourceID}) @@ -1544,15 +1535,19 @@ func (h *AISettingsHandler) HandleGetIncidentData(w http.ResponseWriter, r *http return } - incidents := incidentStore.ListIncidentsByResource(resourceID, limit) - - // Also get formatted context for AI - formattedContext := incidentStore.FormatForResource(resourceID, limit) + page, err := incidentStore.QueryIncidents(memory.IncidentQuery{ResourceID: resourceID, Limit: limit}) + if err != nil { + http.Error(w, "Incident history unavailable", http.StatusServiceUnavailable) + return + } + incidents := page.Incidents + formattedContext := memory.FormatIncidentPageForResource(page) if err := utils.WriteJSONResponse(w, map[string]interface{}{ "resource_id": resourceID, "incidents": incidents, "formatted_context": formattedContext, + "history": page.History, }); err != nil { log.Error().Err(err).Msg("Failed to write incident data response") } diff --git a/internal/api/ai_intelligence_handlers_remediation_additional_test.go b/internal/api/ai_intelligence_handlers_remediation_additional_test.go index 243b31d14..96178101c 100644 --- a/internal/api/ai_intelligence_handlers_remediation_additional_test.go +++ b/internal/api/ai_intelligence_handlers_remediation_additional_test.go @@ -13,6 +13,7 @@ import ( "github.com/rcourtman/pulse-go-rewrite/internal/ai/memory" "github.com/rcourtman/pulse-go-rewrite/internal/alerts" "github.com/rcourtman/pulse-go-rewrite/internal/config" + "github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources" ) func TestHandleGetCircuitBreakerStatus(t *testing.T) { @@ -194,3 +195,38 @@ func TestHandleGetRecentIncidentsCountIsNotMeasured(t *testing.T) { }) } } + +func TestHandleGetRecentIncidentsCanonicalOnlyAndReadFailure(t *testing.T) { + handler, incidents := setupIncidentHandler(t) + canonical, err := unifiedresources.NewSQLiteResourceStore(t.TempDir(), "default") + if err != nil { + t.Fatal(err) + } + defer canonical.Close() + incidents.SetResourceTimelineStore(canonical) + change := unifiedresources.BuildAlertTimelineChange("canonical-only-resource", unifiedresources.ChangeAlertFired, time.Now().Add(-time.Minute), "", unifiedresources.AlertTimelineChange{AlertIdentifier: "canonical-only-alert", AlertType: "disk"}) + if err := canonical.RecordChange(*change); err != nil { + t.Fatal(err) + } + request := httptest.NewRequest(http.MethodGet, "/api/ai/incidents?resource_id=canonical-only-resource", nil) + response := httptest.NewRecorder() + handler.HandleGetRecentIncidents(response, request) + if response.Code != http.StatusOK { + t.Fatal(response.Code, response.Body.String()) + } + var page memory.IncidentPage + if err := json.Unmarshal(response.Body.Bytes(), &page); err != nil { + t.Fatal(err) + } + if len(page.Incidents) != 1 || page.Incidents[0].AlertIdentifier != "canonical-only-alert" || page.History.Source != "canonical_resource_history" { + t.Fatalf("canonical evidence missing: %#v", page) + } + if err := canonical.Close(); err != nil { + t.Fatal(err) + } + response = httptest.NewRecorder() + handler.HandleGetRecentIncidents(response, request) + if response.Code != http.StatusServiceUnavailable { + t.Fatal(response.Code, response.Body.String()) + } +} diff --git a/internal/api/alerting/alerts.go b/internal/api/alerting/alerts.go index 18296631b..2c19c30f2 100644 --- a/internal/api/alerting/alerts.go +++ b/internal/api/alerting/alerts.go @@ -22,6 +22,7 @@ import ( "github.com/rcourtman/pulse-go-rewrite/internal/models" "github.com/rcourtman/pulse-go-rewrite/internal/monitoring" "github.com/rcourtman/pulse-go-rewrite/internal/notifications" + "github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources" "github.com/rcourtman/pulse-go-rewrite/internal/utils" "github.com/rcourtman/pulse-go-rewrite/internal/websocket" "github.com/rs/zerolog/log" @@ -834,12 +835,16 @@ func (h *AlertHandlers) GetAlertIncidentTimeline(w http.ResponseWriter, r *http. return } - var incident *memory.Incident - if !startedAt.IsZero() { - incident = store.GetTimelineByAlertAt(alertID, startedAt) - } else { - incident = store.GetTimelineByAlertIdentifier(alertID) + page, err := store.QueryIncidents(memory.IncidentQuery{AlertIdentifier: alertID, StartedAt: startedAt, Limit: 1}) + if err != nil { + http.Error(w, "Incident history unavailable", http.StatusServiceUnavailable) + return } + var incident *memory.Incident + if len(page.Incidents) > 0 { + incident = page.Incidents[0] + } + if !startedAt.IsZero() && (incident == nil || len(incident.Events) == 0) { if alert, resolvedAt := findAlertOccurrenceForTimeline(h.getMonitor(r.Context()).GetAlertManager(), alertID, startedAt); alert != nil { incident = store.EnsureAlertOccurrence(alert, resolvedAt) @@ -868,8 +873,12 @@ func (h *AlertHandlers) GetAlertIncidentTimeline(w http.ResponseWriter, r *http. http.Error(w, "Incident store unavailable", http.StatusServiceUnavailable) return } - incidents := store.ListIncidentsByResource(resourceID, limit) - if err := utils.WriteJSONResponse(w, exportIncidents(incidents)); err != nil { + page, err := store.QueryIncidents(memory.IncidentQuery{ResourceID: resourceID, Limit: limit}) + if err != nil { + http.Error(w, "Incident history unavailable", http.StatusServiceUnavailable) + return + } + if err := utils.WriteJSONResponse(w, exportIncidents(page.Incidents)); err != nil { log.Error().Err(err).Msg("Failed to write incident list response") } return @@ -987,11 +996,13 @@ func (h *AlertHandlers) SaveAlertIncidentNote(w http.ResponseWriter, r *http.Req } type incidentEventView struct { - ID string `json:"id"` - Type memory.IncidentEventType `json:"type"` - Timestamp time.Time `json:"timestamp"` - Summary string `json:"summary"` - Details map[string]interface{} `json:"details"` + Source string `json:"source,omitempty"` + Evidence *unifiedresources.ResourceChange `json:"evidence,omitempty"` + ID string `json:"id"` + Type memory.IncidentEventType `json:"type"` + Timestamp time.Time `json:"timestamp"` + Summary string `json:"summary"` + Details map[string]interface{} `json:"details"` } func (v incidentEventView) NormalizeCollections() incidentEventView { @@ -1002,23 +1013,24 @@ func (v incidentEventView) NormalizeCollections() incidentEventView { } type incidentView struct { - ID string `json:"id"` - AlertIdentifier string `json:"alertIdentifier"` - AlertType string `json:"alertType"` - Level string `json:"level"` - ResourceID string `json:"resourceId"` - ResourceName string `json:"resourceName"` - ResourceType string `json:"resourceType,omitempty"` - Node string `json:"node,omitempty"` - Instance string `json:"instance,omitempty"` - Message string `json:"message,omitempty"` - Status memory.IncidentStatus `json:"status"` - OpenedAt time.Time `json:"openedAt"` - ClosedAt *time.Time `json:"closedAt,omitempty"` - Acknowledged bool `json:"acknowledged"` - AckUser string `json:"ackUser,omitempty"` - AckTime *time.Time `json:"ackTime,omitempty"` - Events []incidentEventView `json:"events"` + History *memory.IncidentHistoryCoverage `json:"history,omitempty"` + ID string `json:"id"` + AlertIdentifier string `json:"alertIdentifier"` + AlertType string `json:"alertType"` + Level string `json:"level"` + ResourceID string `json:"resourceId"` + ResourceName string `json:"resourceName"` + ResourceType string `json:"resourceType,omitempty"` + Node string `json:"node,omitempty"` + Instance string `json:"instance,omitempty"` + Message string `json:"message,omitempty"` + Status memory.IncidentStatus `json:"status"` + OpenedAt time.Time `json:"openedAt"` + ClosedAt *time.Time `json:"closedAt,omitempty"` + Acknowledged bool `json:"acknowledged"` + AckUser string `json:"ackUser,omitempty"` + AckTime *time.Time `json:"ackTime,omitempty"` + Events []incidentEventView `json:"events"` } func emptyIncidentView() incidentView { @@ -1042,6 +1054,7 @@ func exportIncident(incident *memory.Incident) *incidentView { events := make([]incidentEventView, 0, len(incident.Events)) for _, event := range incident.Events { events = append(events, incidentEventView{ + Source: event.Source, Evidence: event.Evidence, ID: event.ID, Type: event.Type, Timestamp: event.Timestamp, @@ -1052,6 +1065,7 @@ func exportIncident(incident *memory.Incident) *incidentView { alertIdentifier := strings.TrimSpace(incident.AlertIdentifier) view := emptyIncidentView() view.ID = incident.ID + view.History = incident.History view.AlertIdentifier = alertIdentifier view.AlertType = incident.AlertType view.Level = incident.Level diff --git a/internal/api/alerting/incident_query_test.go b/internal/api/alerting/incident_query_test.go new file mode 100644 index 000000000..509cb81ac --- /dev/null +++ b/internal/api/alerting/incident_query_test.go @@ -0,0 +1,49 @@ +package alerting + +import ( + "encoding/json" + "net/http" + "net/http/httptest" + "net/url" + "testing" + "time" + + "github.com/rcourtman/pulse-go-rewrite/internal/ai/memory" + "github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources" + "github.com/stretchr/testify/require" +) + +func TestIncidentQueryHTTPPreservesOccurrenceAndEvidence(t *testing.T) { + canonical, err := unifiedresources.NewSQLiteResourceStore(t.TempDir(), "default") + require.NoError(t, err) + defer canonical.Close() + incidents := memory.NewIncidentStore(memory.IncidentStoreConfig{}) + incidents.SetResourceTimelineStore(canonical) + monitor := new(MockAlertMonitor) + monitor.On("GetIncidentStore").Return(incidents) + handler := NewAlertHandlers(nil, monitor, nil) + first := time.Now().UTC().Add(-time.Hour).Truncate(time.Second) + for index, kind := range []unifiedresources.ChangeKind{unifiedresources.ChangeAlertFired, unifiedresources.ChangeAlertResolved, unifiedresources.ChangeAlertFired} { + at := first.Add(time.Duration(index) * time.Minute) + change := unifiedresources.BuildAlertTimelineChange("resource-a", kind, at, "operator", unifiedresources.AlertTimelineChange{AlertIdentifier: "repeated", AlertType: "disk", AlertLevel: "warning"}) + require.NoError(t, canonical.RecordChange(*change)) + } + request := httptest.NewRequest(http.MethodGet, "/api/alerts/incidents?alertIdentifier=repeated&started_at="+url.QueryEscape(first.Format(time.RFC3339)), nil) + response := httptest.NewRecorder() + handler.GetAlertIncidentTimeline(response, request) + require.Equal(t, http.StatusOK, response.Code, response.Body.String()) + var timeline memory.Incident + require.NoError(t, json.Unmarshal(response.Body.Bytes(), &timeline)) + require.Equal(t, memory.IncidentStatusResolved, timeline.Status) + require.Len(t, timeline.Events, 2) + require.NotNil(t, timeline.History) + require.NotNil(t, timeline.Events[0].Evidence) + require.Equal(t, "operator", timeline.Events[0].Evidence.Actor) + require.Equal(t, "canonical_resource_history", timeline.Events[0].Source) + // A failed read must not enter snapshot-repair fallback or return null/200. + require.NoError(t, canonical.Close()) + response = httptest.NewRecorder() + handler.GetAlertIncidentTimeline(response, request) + require.Equal(t, http.StatusServiceUnavailable, response.Code, response.Body.String()) + monitor.AssertNotCalled(t, "GetAlertManager") +} diff --git a/internal/monitoring/monitor_alert_handling_test.go b/internal/monitoring/monitor_alert_handling_test.go index f60cd108c..d8ae378fa 100644 --- a/internal/monitoring/monitor_alert_handling_test.go +++ b/internal/monitoring/monitor_alert_handling_test.go @@ -214,10 +214,11 @@ func TestMonitor_HandleAlertLifecycle_WritesCanonicalChanges(t *testing.T) { if len(changes) != 5 { t.Fatalf("expected 5 canonical changes, got %d", len(changes)) } + // Arrival order can differ from observation order during lifecycle replay. wantKinds := []unifiedresources.ChangeKind{ - unifiedresources.ChangeAlertUnacknowledged, unifiedresources.ChangeAlertUnsnoozed, unifiedresources.ChangeAlertSnoozed, + unifiedresources.ChangeAlertUnacknowledged, unifiedresources.ChangeAlertAcknowledged, unifiedresources.ChangeAlertFired, } diff --git a/internal/monitoring/monitor_alerts.go b/internal/monitoring/monitor_alerts.go index 3ac949d1b..e106a7a7a 100644 --- a/internal/monitoring/monitor_alerts.go +++ b/internal/monitoring/monitor_alerts.go @@ -521,7 +521,15 @@ func (m *Monitor) reconcileActiveAlertTimelines() { activeAlerts := m.alertManager.GetActiveAlerts() for i := range activeAlerts { alert := &activeAlerts[i] - timeline := m.incidentStore.GetTimelineByAlertAt(alert.ID, alert.StartTime) + page, err := m.incidentStore.QueryIncidents(memory.IncidentQuery{AlertIdentifier: alert.ID, StartedAt: alert.StartTime, Limit: 1}) + if err != nil { + log.Warn().Err(err).Msg("Skipping alert timeline reconciliation because canonical history is unavailable") + continue + } + var timeline *memory.Incident + if len(page.Incidents) > 0 { + timeline = page.Incidents[0] + } if timeline != nil { hasFired := false for _, event := range timeline.Events { diff --git a/internal/unifiedresources/changes.go b/internal/unifiedresources/changes.go index c4f80e432..ea702e36f 100644 --- a/internal/unifiedresources/changes.go +++ b/internal/unifiedresources/changes.go @@ -59,6 +59,12 @@ const ( // ResourceChangeFilters narrows the resource timeline to specific change kinds // and source origins while preserving the canonical change record shape. type ResourceChangeFilters struct { + // AlertIdentifiers matches exact canonical alert metadata before any limit. + // It is a history selector, never an action or resource authority binding. + AlertIdentifiers []string `json:"alertIdentifiers,omitempty"` + // ObservedBefore is an exclusive upper observation bound. Together with + // since it selects [since, before), without changing occurrence timestamps. + ObservedBefore *time.Time `json:"observedBefore,omitempty"` Kinds []ChangeKind `json:"kinds,omitempty"` SourceTypes []ChangeSourceType `json:"sourceTypes,omitempty"` SourceAdapters []ChangeSourceAdapter `json:"sourceAdapters,omitempty"` @@ -66,6 +72,25 @@ type ResourceChangeFilters struct { } func (filters ResourceChangeFilters) matches(change ResourceChange) bool { + if filters.ObservedBefore != nil && !change.ObservedAt.Before(*filters.ObservedBefore) { + return false + } + if len(filters.AlertIdentifiers) > 0 { + identifier, ok := change.Metadata[MetadataAlertIdentifier].(string) + if !ok { + return false + } + found := false + for _, wanted := range filters.AlertIdentifiers { + if identifier == wanted { + found = true + break + } + } + if !found { + return false + } + } if len(filters.Kinds) > 0 { match := false for _, kind := range filters.Kinds { diff --git a/internal/unifiedresources/incident_history_query_test.go b/internal/unifiedresources/incident_history_query_test.go new file mode 100644 index 000000000..41b2ec0e5 --- /dev/null +++ b/internal/unifiedresources/incident_history_query_test.go @@ -0,0 +1,147 @@ +package unifiedresources + +import ( + "fmt" + "strings" + "testing" + "time" + + "github.com/stretchr/testify/require" +) + +// These queries must not lose a relevant incident behind a fleet-wide limit. +// Use both durable and in-memory stores, including late observations and aliases. +func TestIncidentHistoryQueryFiltersBeforeLimit(t *testing.T) { + for _, backend := range []string{"memory", "sqlite"} { + t.Run(backend, func(t *testing.T) { + var store ResourceStore + if backend == "memory" { + store = NewMemoryStore() + } else { + sqlite, err := NewSQLiteResourceStore(t.TempDir(), "default") + require.NoError(t, err) + t.Cleanup(func() { require.NoError(t, sqlite.Close()) }) + store = sqlite + } + now := time.Now().UTC().Truncate(time.Second) + older := now.Add(-time.Hour) + record := func(id, alert string, observed time.Time) { + require.NoError(t, store.RecordChange(ResourceChange{ID: id, ResourceID: "resource-a", Kind: ChangeAlertFired, ObservedAt: observed, OccurredAt: &older, SourceType: SourceHeuristic, SourceAdapter: AdapterProxmox, Metadata: map[string]any{MetadataAlertIdentifier: alert}})) + } + record("target-a", "wanted", now) + record("target-b", "wanted", now) // equal timestamps have stable ID order + for i := 0; i < 300; i++ { + record(fmt.Sprintf("noise-%03d", i), "unrelated", now.Add(time.Second)) + } + record("late-insertion", "wanted", older) + record("upper-bound", "wanted", now.Add(2*time.Second)) + before := now.Add(2 * time.Second) + filters := ResourceChangeFilters{AlertIdentifiers: []string{"wanted"}, ObservedBefore: &before, Kinds: []ChangeKind{ChangeAlertFired}, SourceTypes: []ChangeSourceType{SourceHeuristic}, SourceAdapters: []ChangeSourceAdapter{AdapterProxmox}} + got, err := store.GetRecentChangesFiltered("", older, 2, filters) + require.NoError(t, err) + require.Len(t, got, 2) + require.Equal(t, "target-b", got[0].ID) + require.Equal(t, "target-a", got[1].ID) + adapter := NewMonitorAdapter(NewRegistry(store)) + forwarded, err := adapter.GetRecentChangesFiltered("", older, 2, filters) + require.NoError(t, err) + require.Equal(t, got, forwarded) + require.Equal(t, older, *got[0].OccurredAt) // observation bounds do not rewrite occurrence time + count, err := store.CountRecentChangesFiltered("", older, filters) + require.NoError(t, err) + require.Equal(t, 3, count) + kinds, err := store.CountRecentChangesByKindFiltered("", older, filters) + require.NoError(t, err) + require.Equal(t, 3, kinds[ChangeAlertFired]) + sources, err := store.CountRecentChangesBySourceTypeFiltered("", older, filters) + require.NoError(t, err) + require.Equal(t, 3, sources[SourceHeuristic]) + adapters, err := store.CountRecentChangesBySourceAdapterFiltered("", older, filters) + require.NoError(t, err) + require.Equal(t, 3, adapters[AdapterProxmox]) + + // A history alias selects the original evidence without rewriting its + // resource identity, or promoting the alias into action authority. + writer := store.(interface { + RecordChangeWithSourceIdentity(ResourceChange, string) error + }) + require.NoError(t, writer.RecordChangeWithSourceIdentity(ResourceChange{ID: "alias-event", ResourceID: "resource-canonical", Kind: ChangeAlertResolved, ObservedAt: now, Metadata: map[string]any{MetadataAlertIdentifier: "alias-alert"}}, "resource-legacy")) + aliasFilters := ResourceChangeFilters{AlertIdentifiers: []string{"alias-alert"}, ObservedBefore: &before} + for _, resource := range []string{"resource-legacy", "resource-canonical"} { + identities, err := adapter.ResourceHistoryIDs(resource) + require.NoError(t, err) + require.Contains(t, identities, "resource-legacy") + require.Contains(t, identities, "resource-canonical") + aliased, err := store.GetRecentChangesFiltered(resource, older, 1, aliasFilters) + require.NoError(t, err) + require.Len(t, aliased, 1) + require.Equal(t, "resource-canonical", aliased[0].ResourceID) + count, err := store.CountRecentChangesFiltered(resource, older, aliasFilters) + require.NoError(t, err) + require.Equal(t, 1, count) + } + // JSON numbers must not compare equal to textual alert identifiers. + require.NoError(t, store.RecordChange(ResourceChange{ID: "numeric-identity", ResourceID: "resource-a", Kind: ChangeAlertFired, ObservedAt: now, Metadata: map[string]any{MetadataAlertIdentifier: 123}})) + malformed, err := store.GetRecentChangesFiltered("", older, 10, ResourceChangeFilters{AlertIdentifiers: []string{"123"}}) + require.NoError(t, err) + require.Empty(t, malformed) + filters.AlertIdentifiers = []string{"wanted' OR 1=1 --"} + got, err = store.GetRecentChangesFiltered("", older, 2, filters) + require.NoError(t, err) + require.Empty(t, got) + }) + } +} + +func TestIncidentHistoryQueryReadFailureIsNotEmptyHistory(t *testing.T) { + store, err := NewSQLiteResourceStore(t.TempDir(), "default") + require.NoError(t, err) + require.NoError(t, store.Close()) + filters := ResourceChangeFilters{AlertIdentifiers: []string{"wanted"}} + _, err = store.GetRecentChangesFiltered("", time.Time{}, 1, filters) + require.Error(t, err) + _, err = store.CountRecentChangesFiltered("", time.Time{}, filters) + require.Error(t, err) + adapter := NewMonitorAdapter(NewRegistry(store)) + _, err = adapter.GetRecentChangesFiltered("", time.Time{}, 1, filters) + require.Error(t, err) + _, err = (&MonitorAdapter{}).GetRecentChangesFiltered("", time.Time{}, 1, filters) + require.ErrorContains(t, err, "unavailable") + _, err = (&MonitorAdapter{}).ResourceHistoryIDs("resource-a") + require.ErrorContains(t, err, "unavailable") +} + +func TestIncidentHistoryQueryInvalidLegacyMetadata(t *testing.T) { + store, err := NewSQLiteResourceStore(t.TempDir(), "default") + require.NoError(t, err) + defer store.Close() + require.NoError(t, store.RecordChange(ResourceChange{ID: "bad-json", ResourceID: "resource-a", Kind: ChangeAlertFired, ObservedAt: time.Now()})) + _, err = store.db.Exec(`UPDATE resource_changes SET metadata_json = '{invalid' WHERE id = 'bad-json'`) + require.NoError(t, err) + filters := ResourceChangeFilters{AlertIdentifiers: []string{"wanted"}} + got, err := store.GetRecentChangesFiltered("", time.Time{}, 1, filters) + require.NoError(t, err) + require.Empty(t, got) + count, err := store.CountRecentChangesFiltered("", time.Time{}, filters) + require.NoError(t, err) + require.Zero(t, count) +} + +func TestIncidentHistoryQueryUsesAlertIdentityIndex(t *testing.T) { + store, err := NewSQLiteResourceStore(t.TempDir(), "default") + require.NoError(t, err) + defer store.Close() + query, args := buildRecentChangeCountQuery(nil, time.Time{}, ResourceChangeFilters{AlertIdentifiers: []string{"wanted"}, Kinds: []ChangeKind{ChangeAlertFired, ChangeAlertResolved}}, "EXPLAIN QUERY PLAN SELECT COUNT(*) FROM resource_changes", store.resourceChangesObservedAtExpr(), store.resourceChangesSourceTypeExpr(), store.resourceChangesSourceAdapterExpr()) + rows, err := store.db.Query(query, args...) + require.NoError(t, err) + defer rows.Close() + var plans []string + for rows.Next() { + var id, parent, unused int + var detail string + require.NoError(t, rows.Scan(&id, &parent, &unused, &detail)) + plans = append(plans, detail) + } + require.NoError(t, rows.Err()) + require.Contains(t, strings.Join(plans, "\n"), "idx_resource_changes_alert_time") +} diff --git a/internal/unifiedresources/monitor_adapter.go b/internal/unifiedresources/monitor_adapter.go index edb0b0f4d..6c941e354 100644 --- a/internal/unifiedresources/monitor_adapter.go +++ b/internal/unifiedresources/monitor_adapter.go @@ -1,6 +1,7 @@ package unifiedresources import ( + "fmt" "sort" "strings" "sync" @@ -151,6 +152,30 @@ func (a *MonitorAdapter) GetRecentChanges(canonicalID string, since time.Time, l return registry.store.GetRecentChanges(canonicalID, since, limit) } +// GetRecentChangesFiltered preserves the canonical query predicates used by +// incident projections, including alert identity before the evidence limit. +func (a *MonitorAdapter) GetRecentChangesFiltered(resourceID string, since time.Time, limit int, filters ResourceChangeFilters) ([]ResourceChange, error) { + registry := a.currentRegistry() + if registry == nil || registry.store == nil { + return nil, fmt.Errorf("canonical resource history unavailable") + } + return registry.store.GetRecentChangesFiltered(resourceID, since, limit, filters) +} + +func (a *MonitorAdapter) ResourceHistoryIDs(resourceID string) ([]string, error) { + registry := a.currentRegistry() + if registry == nil || registry.store == nil { + return nil, fmt.Errorf("canonical resource history unavailable") + } + reader, ok := registry.store.(interface { + ResourceHistoryIDs(string) ([]string, error) + }) + if !ok { + return nil, fmt.Errorf("canonical history identities unavailable") + } + return reader.ResourceHistoryIDs(resourceID) +} + // GetResourceOperatorState exposes the durable operator-intent record through // the monitor adapter without widening monitoring's core resource-store // interface. Alert policy resolution discovers this capability optionally. diff --git a/internal/unifiedresources/store.go b/internal/unifiedresources/store.go index 36c57fe26..61c63ebfe 100644 --- a/internal/unifiedresources/store.go +++ b/internal/unifiedresources/store.go @@ -31,6 +31,7 @@ type ResourceStore interface { UpsertResourceIdentityPins(pins []ResourceIdentityPin) error ListResourceIdentityPins() ([]ResourceIdentityPin, error) RecordChange(change ResourceChange) error + ResourceHistoryIDs(resourceID string) ([]string, error) GetRecentChanges(canonicalID string, since time.Time, limit int) ([]ResourceChange, error) GetRecentChangesFiltered(canonicalID string, since time.Time, limit int, filters ResourceChangeFilters) ([]ResourceChange, error) CountRecentChanges(canonicalID string, since time.Time) (int, error) @@ -1033,6 +1034,7 @@ func (s *SQLiteResourceStore) migrateResourceChangesSchema() error { func (s *SQLiteResourceStore) ensureResourceChangesIndexes() error { indexes := []string{ + `CREATE INDEX IF NOT EXISTS idx_resource_changes_alert_time ON resource_changes(` + resourceChangesAlertIdentifierExpr() + `, observed_at DESC, id DESC)`, `CREATE INDEX IF NOT EXISTS idx_resource_changes_time ON resource_changes(observed_at DESC)`, `CREATE INDEX IF NOT EXISTS idx_resource_changes_canonical_time ON resource_changes(canonical_id, observed_at DESC)`, `CREATE INDEX IF NOT EXISTS idx_resource_changes_kind_time ON resource_changes(kind, observed_at DESC)`, @@ -1165,6 +1167,12 @@ func resourceChangesRelatedResourcesExpr() string { return "COALESCE(NULLIF(TRIM(related_resources), ''), '[]')" } +// The same guarded expression owns both the alert selector and its index. +// Invalid legacy metadata cannot establish an alert identity. +func resourceChangesAlertIdentifierExpr() string { + return "json_extract(CASE WHEN json_valid(metadata_json) THEN metadata_json ELSE '{}' END, '$.alert_identifier')" +} + func resourceChangesMetadataJSONExpr() string { return "COALESCE(NULLIF(TRIM(metadata_json), ''), '{}')" } @@ -1754,6 +1762,24 @@ func recordChangeSQL(execer sqlExecutor, change ResourceChange, includeTimestamp return nil } +// ResourceHistoryIDs expands read-only history identities. These aliases do +// not authorize actions or change the resource identity recorded on evidence. +func (s *SQLiteResourceStore) ResourceHistoryIDs(resourceID string) ([]string, error) { + return s.resourceChangeIDSet(resourceID) +} + +func (m *MemoryStore) ResourceHistoryIDs(resourceID string) ([]string, error) { + m.mu.RLock() + defer m.mu.RUnlock() + ids := m.resourceChangeIDSetLocked(CanonicalResourceID(resourceID)) + result := make([]string, 0, len(ids)) + for _, id := range ids { + result = append(result, id) + } + sort.Strings(result) + return result, nil +} + func (s *SQLiteResourceStore) GetRecentChanges(canonicalID string, since time.Time, limit int) ([]ResourceChange, error) { return s.GetRecentChangesFiltered(canonicalID, since, limit, ResourceChangeFilters{}) } @@ -1783,35 +1809,12 @@ func (s *SQLiteResourceStore) GetRecentChangesFiltered(canonicalID string, since conditions = append(conditions, observedAtExpr+" >= ?") args = append(args, since) } - if len(filters.Kinds) > 0 { - placeholders := make([]string, 0, len(filters.Kinds)) - for _, kind := range filters.Kinds { - placeholders = append(placeholders, "?") - args = append(args, string(kind)) - } - conditions = append(conditions, "kind IN ("+strings.Join(placeholders, ", ")+")") - } - if len(filters.SourceTypes) > 0 { - placeholders := make([]string, 0, len(filters.SourceTypes)) - for _, sourceType := range filters.SourceTypes { - placeholders = append(placeholders, "?") - args = append(args, string(sourceType)) - } - conditions = append(conditions, sourceTypeExpr+" IN ("+strings.Join(placeholders, ", ")+")") - } - if len(filters.SourceAdapters) > 0 { - placeholders := make([]string, 0, len(filters.SourceAdapters)) - for _, sourceAdapter := range filters.SourceAdapters { - placeholders = append(placeholders, "?") - args = append(args, string(sourceAdapter)) - } - conditions = append(conditions, sourceAdapterExpr+" IN ("+strings.Join(placeholders, ", ")+")") - } + conditions, args = appendRecentChangeFilterConditions(conditions, args, filters, observedAtExpr, sourceTypeExpr, sourceAdapterExpr) if len(conditions) > 0 { query += "\n\t\tWHERE " + strings.Join(conditions, " AND ") } query += ` - ORDER BY ` + observedAtExpr + ` DESC` + ORDER BY ` + observedAtExpr + ` DESC, id DESC` if limit > 0 { query += ` LIMIT ?` args = append(args, limit) @@ -3552,9 +3555,18 @@ func (m *MemoryStore) GetRecentChangesFiltered(canonicalID string, since time.Ti continue } out = append(out, change) - if limit > 0 && len(out) >= limit { - break + + } + // Late-arriving events need not be inserted in observation order. + // Apply the same stable order as SQLite before selecting the bounded page. + sort.Slice(out, func(i, j int) bool { + if out[i].ObservedAt.Equal(out[j].ObservedAt) { + return out[i].ID > out[j].ID } + return out[i].ObservedAt.After(out[j].ObservedAt) + }) + if limit > 0 && len(out) > limit { + out = out[:limit] } return out, nil } @@ -3676,6 +3688,29 @@ func buildRecentChangeCountQuery(canonicalIDs []string, since time.Time, filters if len(canonicalIDs) > 0 { conditions, args = appendRecentChangeResourceCondition(conditions, args, canonicalIDs, filters.IncludeRelated) } + conditions, args = appendRecentChangeFilterConditions(conditions, args, filters, observedAtExpr, sourceTypeExpr, sourceAdapterExpr) + query += ` WHERE ` + strings.Join(conditions, " AND ") + return query, args +} + +// appendRecentChangeFilterConditions keeps bounded reads and all aggregate +// counts on the same predicate. Metadata values are bound parameters, not SQL. +func appendRecentChangeFilterConditions(conditions []string, args []any, filters ResourceChangeFilters, observedAtExpr, sourceTypeExpr, sourceAdapterExpr string) ([]string, []any) { + if filters.ObservedBefore != nil { + conditions = append(conditions, observedAtExpr+" < ?") + args = append(args, *filters.ObservedBefore) + } + if len(filters.AlertIdentifiers) > 0 { + placeholders := make([]string, 0, len(filters.AlertIdentifiers)) + for _, identifier := range filters.AlertIdentifiers { + placeholders = append(placeholders, "?") + args = append(args, identifier) + } + // Invalid legacy JSON cannot establish a matching alert identity. Reject + // non-string JSON values rather than coercing them into an identifier. + metadata := "CASE WHEN json_valid(metadata_json) THEN metadata_json ELSE '{}' END" + conditions = append(conditions, "(json_type("+metadata+", '$.alert_identifier') = 'text' AND "+resourceChangesAlertIdentifierExpr()+" IN ("+strings.Join(placeholders, ", ")+"))") + } if len(filters.Kinds) > 0 { placeholders := make([]string, 0, len(filters.Kinds)) for _, kind := range filters.Kinds { @@ -3700,8 +3735,7 @@ func buildRecentChangeCountQuery(canonicalIDs []string, since time.Time, filters } conditions = append(conditions, sourceAdapterExpr+" IN ("+strings.Join(placeholders, ", ")+")") } - query += ` WHERE ` + strings.Join(conditions, " AND ") - return query, args + return conditions, args } func appendRecentChangeResourceCondition(conditions []string, args []any, canonicalIDs []string, includeRelated bool) ([]string, []any) { diff --git a/internal/unifiedresources/store_test.go b/internal/unifiedresources/store_test.go index fcf1bef43..92ec4d333 100644 --- a/internal/unifiedresources/store_test.go +++ b/internal/unifiedresources/store_test.go @@ -1318,8 +1318,8 @@ func TestMemoryStoreResourceChangeFiltersIncludeRelatedResources(t *testing.T) { if err != nil { t.Fatalf("GetRecentChangesFiltered include related: %v", err) } - if len(timeline) != 2 || timeline[0].ID != "mem-direct" || timeline[1].ID != "mem-related" { - t.Fatalf("relationship-aware memory timeline = %#v, want reverse insertion order direct plus related", timeline) + if len(timeline) != 2 || timeline[0].ID != "mem-related" || timeline[1].ID != "mem-direct" { + t.Fatalf("relationship-aware memory timeline = %#v, want newest observation first, as in the durable store", timeline) } count, err := store.CountRecentChangesFiltered("node:1", now.Add(-time.Hour), ResourceChangeFilters{IncludeRelated: true}) diff --git a/scripts/release_control/canonical_completion_guard_test.py b/scripts/release_control/canonical_completion_guard_test.py index 95db2565f..ec59d84c8 100644 --- a/scripts/release_control/canonical_completion_guard_test.py +++ b/scripts/release_control/canonical_completion_guard_test.py @@ -311,6 +311,7 @@ class CanonicalCompletionGuardTest(unittest.TestCase): "internal/monitoring/issue1638_dns_cache_test.go", "internal/monitoring/metric_window_provider_test.go", "internal/monitoring/monitor_additional_test.go", + "internal/monitoring/monitor_alert_handling_test.go", "internal/monitoring/monitor_alert_intent_test.go", "internal/monitoring/monitor_alert_override_migration_test.go", "internal/monitoring/monitor_backups_readstate_test.go",