diff --git a/docs/API.md b/docs/API.md index f2e3d6b6f5..002d29a1e2 100644 --- a/docs/API.md +++ b/docs/API.md @@ -836,9 +836,12 @@ delivery for all resources. - `GET /api/admin/reports/catalog` (admin, `settings:read`) - Returns the canonical reporting catalog for the settings surface, including locked-state teaser copy, enabled-surface guidance copy, performance-report options, canonical single-report filename subject, canonical fallback filename date style, and the nested VM inventory export definition. - Metadata route: readable without the `advanced_reporting` feature so locked admin surfaces can render the same reporting definition before upsell. -- `GET /api/admin/reports/generate` (admin, `settings:read`) - - Query params: `format` (pdf/csv, default `pdf`), `resourceType`, `resourceId`, `metricType` (optional), `start`/`end` (RFC3339, optional; defaults to last 24h), `title` (optional) +- `POST /api/admin/reports/generate` (admin, `settings:read`) + - Body fields: `format` (pdf/csv, default `pdf`), `resourceType`, `resourceId`, `metricType` (optional), `start`/`end` (RFC3339, optional; defaults to last 24h), `title` (optional) - If `title` is omitted, the backend applies the canonical default title for that resource report. + - When Pulse Assistant is configured, the configured AI provider narrates a PDF's executive summary; each narration counts against the AI budget and appears in AI usage. If the call fails, times out or the budget is spent, the PDF falls back to the deterministic summary. CSV output is never narrated. The settings UI uses this method. +- `GET /api/admin/reports/generate` (admin, `settings:read`) + - Takes the same fields as query params and returns the same report, except that it never calls an AI provider: a PDF carries the deterministic summary and a note saying AI narration needs `POST`. Other methods return `405` with `Allow: GET, POST`. - `POST /api/admin/reports/generate-multi` (admin, `settings:read`) - Body fields: `resources` (1-50 entries of `{resourceType,resourceId}`), `format`, `metricType` (optional), `start`/`end` (RFC3339, optional; defaults to last 24h), `title` (optional) - If `title` is omitted, the backend applies the canonical default fleet report title. @@ -851,8 +854,8 @@ Validation and limits: - Malformed `start`/`end` values, `end` not strictly after `start`, or report windows over 366 days return `400 invalid_time_range`. - `metricType` must match `[a-zA-Z0-9._:-]+` and be <= 64 chars, otherwise `400 invalid_metric_type`. - `title` must be <= 256 chars, otherwise `400 invalid_title`. -- Multi-report body max size is 1MB; oversized payloads return `400 body_too_large`. -- Multi-report bodies reject trailing payload and unknown JSON fields with `400 invalid_body`. +- Report body max size is 1MB; oversized payloads return `400 body_too_large`. +- Report bodies reject trailing payload and unknown JSON fields with `400 invalid_body`. - VM inventory export only accepts `csv`. Common reporting error codes: diff --git a/docs/MSP.md b/docs/MSP.md index 7561f520c6..9302b66a1d 100644 --- a/docs/MSP.md +++ b/docs/MSP.md @@ -260,10 +260,12 @@ Each client runtime (or organization) generates its own reports, scoped to that client's resources: - **UI**: Settings → Data & Reports. -- **API**: `GET /api/admin/reports/generate` (single resource) and +- **API**: `POST /api/admin/reports/generate` (single resource) and `POST /api/admin/reports/generate-multi` (up to 50 resources per report), returning PDF or CSV. In shared-process mode, scope with `X-Pulse-Org-ID` - or an org-bound token. + or an org-bound token. `GET /api/admin/reports/generate` with query + parameters still returns a single-resource report, but never adds the + Pulse Assistant narrative. - **Schedules**: `GET`/`POST /api/admin/reports/schedules`, `PUT`/`DELETE /api/admin/reports/schedules/{id}`, and `POST /api/admin/reports/schedules/{id}/run`. Schedules can target explicit diff --git a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md b/docs/release-control/v6/internal/subsystems/agent-lifecycle.md index 93f9a7544c..4715dcd9c6 100644 --- a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md +++ b/docs/release-control/v6/internal/subsystems/agent-lifecycle.md @@ -2480,12 +2480,13 @@ agent inventory, registration state, or command-channel readiness. or fleet-command evidence. Lifecycle flows must not recreate the retired Patrol quickstart bootstrap path, mint server-issued hosted-model tokens, or derive AI provider state from installation identity. - Per-request `/api/ai/chat` execution-mode overrides follow that same - boundary: lifecycle-adjacent consumers may rely on Assistant approval - semantics, but scoped `autonomous_mode:false` chat requests must not be - reinterpreted as agent registration, assignment, installer, or connection - lifecycle state. Patrol finding handoffs that force approval-required mode - from a non-empty `finding_id` are likewise AI/runtime governance, not an + The server-set `/api/ai/chat` execution mode follows that same boundary: + lifecycle-adjacent consumers may rely on Assistant approval semantics, but + the approval-required mode `HandleChat` installs on every chat request + (`ChatRequest` decodes no `autonomous_mode` field) must not be reinterpreted as agent + registration, assignment, installer, or connection lifecycle state. Patrol + finding handoffs running in that mode from a non-empty `finding_id` are + likewise AI/runtime governance, not an agent command grant or lifecycle authorization primitive. Patrol queued-fix approvals may now seed the shared action-audit store with planned and pending lifecycle evidence, but lifecycle surfaces must treat @@ -4344,6 +4345,22 @@ findings handed to the orchestrator, so investigation reasoning sees the same lock-against-remediation flag that the action broker enforces downstream — no possible drift between "what Patrol proposes" and "what the broker accepts." +That builder resolves a finding's resource reference through the +organization monitor's current read state before it reads operator +state. A finding still keyed by an agent's canonical ID after an +operator linked the agent into its guest therefore reads the merged +row's state while the read state folds the agent, as the resources API +does. A finding naming a saved host that has not reported since a +restart by its agent reference reads the row stored under that host's +canonical ID, because the read state overlays host continuity as the +resources API listing does. A saved host joined to its guest by a link +stays its own row there, without lending the guest its saved payload, +but the registry holds it under the guest, so its findings read the +guest's state, as they did before the restart. The +resolution only reads identity. It does not treat the folded agent as +removed, offline or unenrolled, does not turn a continuity-only host +into a live sighting, and grants no heartbeat, command, install or +repair authority. The same router wiring owns the alert-bridge patrol-trigger callback. It now receives the full alert payload as a struct and consults the operator's @@ -4926,8 +4943,8 @@ withheld) is an AI-runtime privacy concern; it is not agent enrollment config, installer readiness, command reachability, or any fleet-control capability signal. Patrol finding chat handoffs follow the same ownership split: when `/api/ai/chat` resolves a `finding_id` into model-only Patrol briefing, -resource, or action context, the backend-enforced `autonomous_mode:false` -clamp is Assistant action-governance, not agent readiness, fleet command +resource, or action context, the backend-enforced approval-required clamp +is Assistant action-governance, not agent readiness, fleet command reachability, or enrollment health. If the same request also carries recognized Patrol product handoff context, resources, or action references, the API handler may merge only same-finding @@ -7662,6 +7679,16 @@ bounded and classified temperature nodes, and leaves the host report intact when the helper is absent or incompatible. It does not treat Windows ACPI thermal zones as hardware sensor evidence, accept remote provider locations, or grant Pulse hardware-control authority. +### Single-resource report AI narration requires POST + +`internal/api/metrics_reporting_handlers.go`, `internal/api/router.go` and +`internal/api/ai_handlers.go` changed only how `/api/admin/reports/generate` +reads its fields (a `POST` JSON body or `GET` query parameters), which of those +may use the tenant's AI narrator (`POST` only), and how a `GET` finds Patrol +findings (`ExistingAIService`, which never constructs a tenant AI service). No +agent registration, enrolment, install, update, removal or report ingest path +changed. + ### Agent profile adoption is reported as a count only The usage telemetry snapshot reads `LoadAgentProfiles` and reports @@ -8969,6 +8996,45 @@ their own heartbeat cutoff (`fleethealth.AgentStaleThreshold`, which is separate from the monitoring reporting lease). Agent registration, enrolment, install, update, removal and report identity are unchanged. +### Safe methods refused on discovery settings and SAML logout + +`internal/api/method_guard.go`, `internal/api/discovery_handlers.go` and +`internal/api/saml_handlers.go` changed only so `/api/discovery/settings` and +`/api/saml/{id}/logout` answer `405` to every method outside their allowed set, +`GET` and `HEAD` included. The settings write still takes `PUT` or `POST` and +logout still takes `POST`. No agent registration, enrolment, install, update or +removal path changed. + +### SAML SLO responses bound to the logout that requested them + +`internal/api/saml_service.go` and `internal/api/saml_handlers.go` changed only +so `/api/saml/{id}/slo` accepts an IdP LogoutResponse when its signed +`InResponseTo` names an unexpired LogoutRequest that `/api/saml/{id}/logout` +issued, once, in a browser carrying no session or the session that logout +ended. Other responses are refused, the SLO handler writes no session or +cookie for any response (the logout request already cleared both), and +`/api/saml/{id}/logout` refuses a request that carries no session. No agent +registration, enrolment, install, update or removal path changed. + +### Manual SAML IdP certificate published as base64 DER + +`internal/api/saml_service.go` changed only so a SAML provider configured with +`idpSsoUrl` and a PEM `idpCertificate` or `idpCertFile` puts that certificate +into its in-memory IdP descriptor as base64 DER, which crewjam/saml can read +when it verifies the IdP's ACS Responses and LogoutResponses. No agent +registration, enrolment, install, update or removal path changed. + +### SAML logins bound to the browser that started them + +`internal/api/saml_service.go` and `internal/api/saml_handlers.go` changed only +so `/api/saml/{id}/login` records each SAML AuthnRequest against a short-lived +login cookie and `/api/saml/{id}/acs` accepts the IdP's signed answer to it +once, in that browser, at default settings, reposting the IdP's cross-site +delivery once from Pulse's origin so the cookie arrives. Signed content +carrying a prefixed attribute or namespace declaration named like a SAML +attribute is refused. No agent +registration, enrolment, install, update or removal path changed. + ### Demo write guard ignores websocket upgrade headers `internal/api/demo_middleware.go` changed only so a `POST`, `PUT`, `PATCH` or @@ -8985,3 +9051,24 @@ No agent registration, enrolment, install, update or removal path changed. for every method. No agent route lives under `/debug/`, so agent registration, enrolment, install, update, removal, report ingest and `/api/agent/ws` handshakes are unchanged on a demo instance. + +### Broadcast host coalescing honours operator splits + +`internal/monitoring/monitor.go` changed only so the websocket broadcast +coalesces host views with the operator's merge exclusions +(`MonitorAdapter.CoalesceForPresentation`), as the resources API's +`ListForPresentation` already did, so an agent host the operator split from +a same-named platform view by unlink or report-merge is coalesced on the +websocket as REST coalesces it (the filter's limits are recorded under +"Unlink replaces the pair's operator link" in the unified-resources +contract). Agent registration, enrolment, install, update, removal, report +identity and continuity are unchanged. + +### Report-merge splits a linked agent + +`internal/api/resourceapi/resources.go` changed only so report-merge on a +resource an operator linked an agent into (a VM, a node or a Docker host) +replaces that link, splitting the agent back out as unlink does. Agent +registration, enrolment, install, update, removal, report identity and +continuity are unchanged, and an agent's own declared node link still +ignores exclusions. diff --git a/docs/release-control/v6/internal/subsystems/ai-runtime.md b/docs/release-control/v6/internal/subsystems/ai-runtime.md index 08be24a5c8..0150ca214e 100644 --- a/docs/release-control/v6/internal/subsystems/ai-runtime.md +++ b/docs/release-control/v6/internal/subsystems/ai-runtime.md @@ -128,6 +128,25 @@ on component mount. Closing clears the registered input so later keyboard commands cannot target a detached composer. A handoff must leave Escape and keyboard input in Assistant, not the underlying alert search. +### Process-wide memory ID sequences + +`internal/ai/memory` mints incident, incident-event, remediation-record and +context-memory IDs from package-level sequences that every store in the process +shares. Multi-tenant monitoring gives each tenant monitor its own +`IncidentStore`, and their alert checks call `RecordAlertFired` from separate +goroutines, while a store's mutex covers only its own state. Each sequence is +therefore an `atomic.Int64` minted through `nextSequencedID` in +`internal/ai/memory/ids.go`, never a plain package counter. The ID shape stays +`-`, but `n` no longer wraps at 1000. The old +`counter%1000` suffix repeated an ID whenever one generator minted more than a +thousand in a wall-clock second, and incident notes resolve an incident by ID +while timeline merges deduplicate events by ID. Within one process IDs are now +unique by construction. Across restarts only the timestamp separates them, so a +repeated local second (DST fall-back or a backward clock step) can still meet a +restarted sequence's suffix. Persisted IDs and the stored format are unchanged. +`internal/ai/memory/ids_test.go` mints from every generator concurrently and +records alerts into two tenant stores concurrently; run it under `-race`. + ### Canonical incident-history queries Incident context reads `IncidentStore.QueryIncidents`, whose page contains @@ -2544,8 +2563,9 @@ deriving an older display status from `workflowStatusHistory`. popup/dialog only; it must not also close the Assistant drawer. Failed-turn retry is part of that same local chat-runtime boundary: a retryable in-memory assistant error may replay the original user turn's - structured mentions, finding id, approval override, handoff resources, - handoff actions, and handoff metadata, but must not reconstruct scoped + structured mentions, finding id, model route, handoff context, handoff + resources, handoff actions, and handoff metadata (execution mode is not + part of that request; `/api/ai/chat` sets it), but must not reconstruct scoped context from prompt history or saved transcript prose. Provider/model route recovery is explicit user-visible recovery, not hidden chat execution. A failed turn may expose route-and-retry actions through the existing drawer @@ -3537,7 +3557,7 @@ query...`, and `Reading storage...` before streamed tool arguments are browser-safe session summary, clear redo state when a new message or trim changes the transcript, and keep forked sessions independent from the source redo stack. The drawer must restore the removed prompt, structured mentions, - finding id, handoff metadata, approval/autonomous override, and selected + finding id, handoff context and metadata, and selected model route when available so the operator can edit and resend without reconstructing hidden context from transcript prose. Retry and regenerate are turn re-runs over the same undo boundary, not a @@ -3792,8 +3812,10 @@ query...`, and `Reading storage...` before streamed tool arguments are dispatches the shared evidence-first explanation request through `frontend-modern/src/components/AI/Chat/hooks/useExplanationRequest.ts`. It preserves drafts and current work, captures the selected context before - asynchronous initialization, and uses normal send, queue, and retry handling - with request-local `autonomousMode:false`. It must not prescribe a diagnosis + asynchronous initialization, and uses normal send, queue, and retry handling. + The store marks that context `autonomousMode:false` so the drawer discloses + approval-required posture; the request itself carries no execution-mode + field, because `/api/ai/chat` sets that mode server-side. It must not prescribe a diagnosis or a tool sequence or grant action authority. The drawer presentation must stay compact: source, status, one primary subject, and an optional safe route link. It must not render Patrol-authored remediation @@ -4176,10 +4198,14 @@ resolve canonical/source IDs and unique aliases before collection, reject human-facing labels in the settings payload must frame the operator boundary as Patrol mode rather than Patrol configuration. Monitor-only Patrol autonomy saves are part of the same runtime gate: - when the safe-remediation extension or entitlement is unavailable, both the - browser state owner and `internal/api/ai_handlers.go` must clear stale - full-mode unlock state while clamping autonomy back to `monitor`, so paid - remediation permission cannot survive through a free runtime save. + when the safe-remediation extension or entitlement is unavailable, the + API must clear stale full-mode unlock and Autopilot activation state while + clamping autonomy back to `monitor` (the monitor-only handler in + `internal/api/ai_handlers.go` without the extension, the + `SaveAutonomySettings` adapter in `internal/api/router_routes_ai_relay.go` + behind the paid handler), and the browser state owner must refuse a paid + level when a save starts, so paid remediation permission cannot survive + through a free runtime save. Event-triggered Patrol runtime policy is also part of this gate: when local development background automation or another runtime policy pauses automatic alert/anomaly-triggered checks, `internal/ai` must expose an explicit @@ -4607,10 +4633,11 @@ resolve canonical/source IDs and unique aliases before collection, reject completed, failed, expired, or otherwise historical action references must remain action context without being relabeled as requiring approval. When the Assistant drawer restores any session from that `handoff_summary`, - it must restore the scoped request-local approval boundary as well as the - safe visible briefing: the next chat turn must carry - `autonomous_mode:false` even when the summary is context-only and has no - queued action, while the visible badge/action copy must still reflect the + it must restore the scoped approval-required disclosure (the drawer-only + `autonomousMode:false` context flag) as well as the safe visible briefing, + even when the summary is context-only and has no queued action. The next + chat turn carries no execution-mode field; `/api/ai/chat` runs it + approval-required server-side. The visible badge/action copy must still reflect the actual last-known action state or Patrol assessment context instead of inventing a pending approval or restoring a Patrol recommendation. That restoration is success-bound: if the underlying session message load fails, @@ -4644,8 +4671,8 @@ resolve canonical/source IDs and unique aliases before collection, reject model-context text parsing must not resurrect them as a legacy fallback. After that send succeeds, the drawer must clear those request payloads while preserving the safe visible - briefing and request-local - approval-required posture; later turns must rely on backend-owned session + briefing and the drawer's + approval-required disclosure; later turns must rely on backend-owned session model-context hydration and current canonical stores instead of resending stale browser handoff payloads. Patrol approval-row Assistant entries are still Patrol finding handoffs, not local prompt-only shortcuts: live @@ -4657,34 +4684,38 @@ resolve canonical/source IDs and unique aliases before collection, reject Proposed-fix command text must stay out of both the persisted chat message and the model-only handoff context, and command payloads remain approval-context data, not conversational copy. - `/api/ai/chat` must also clamp Patrol finding handoffs to - approval-required mode when a request carries a non-empty `finding_id` or - resolves to model-only briefing, resource, or action context, by forcing the - request-local autonomous-mode override to false, even when a caller supplied - `autonomous_mode:true`. That clamp belongs to the + `/api/ai/chat` clamps every exchange to approval-required mode, Patrol + finding handoffs included: `ChatRequest` has no `autonomous_mode` field, so + caller JSON (a stale client's `autonomous_mode:true` included) is dropped at + decode, and `HandleChat` passes `AutonomousMode:false` into the chat service + whether or not the request carries a `finding_id` or model-only briefing, + resource, or action context. The browser sends no execution-mode field. + That clamp belongs to the backend/API execution boundary, does not mutate the user's persistent AI control setting, and prevents product-originated Patrol action context from becoming silent command authority. - The chat runtime must apply any request-local autonomous-mode override to + The chat runtime must apply that server-set request-local mode + (`chat.ExecuteRequest.AutonomousMode`) to both the per-request `AgenticLoop` and the cloned `PulseToolExecutor`; - persistent autonomous settings must not leak into scoped approval-required - handoffs through executor state. When such an override forces approval mode + persistent autonomous settings must not leak into approval-required + requests through executor state. When the request forces approval mode and the saved control level is autonomous, the executor clone must clamp its effective control level to controlled for that request only, so even - policy-allowed diagnostic commands require operator approval in scoped - handoffs without mutating the user's saved setting. + policy-allowed diagnostic commands require operator approval in public chat + without mutating the user's saved setting. The Assistant drawer may also render an attached context briefing for that handoff, but the briefing is runtime context visibility only: it must not mutate chat control settings, execute tools, or reveal raw command payloads. Resource-drawer Assistant entries use that same briefing path with `handoff_metadata.kind=resource_context`, a structured `handoff_resources` - reference, and `autonomous_mode:false`; they must not prefill or submit a + reference, and the drawer-only `autonomousMode:false` approval disclosure; + they must not prefill or submit a browser-authored prompt, and any rich resource facts must be hydrated by the backend context-pack path rather than reconstructed in the browser. Safe route-owned briefing actions may render as app links when the handoff includes an `actionHref`, but those links are navigation guidance only and do not grant tool execution or approval authority. - Request-local approval-required scoped handoffs must present that boundary + Scoped handoffs must present the approval-required boundary through compact source-named drawer state and the effective control label, so Patrol approval/finding handoffs and alert-investigation handoffs are named by their source rather than as generic dashboard briefs. @@ -6734,6 +6765,11 @@ The browser does not read the global learned-correlation list: `/api/ai/intelligence/correlations`, and neither the shared AI intelligence store nor the Patrol page loads it, because no page renders it. The resource drawer reads per-resource correlations from the resource-intelligence payload. +The browser does not read `GET /api/ai/circuit/status` either: +`frontend-modern/src/api/ai.ts` carries no client for it and the shared AI +intelligence store holds no breaker state, because an open breaker already +reaches Patrol as the `circuit_open` runtime block cause. The route stays a +backend API surface. AI-facing policy metadata must also be cloned through the shared unified- resource policy helper so chat and tools consumers do not maintain their own policy copy logic. Chat mention prefetch now calls that shared helper directly @@ -6950,19 +6986,22 @@ operator paragraph. Future overview or brief surfaces need a governed product owner first, must pass fact-bound structured context from owning Infrastructure, Workloads, Patrol, storage, recovery, and alert summaries, and must not let an unbounded prompt become a route's source of truth. -Future route-to-Assistant handoffs must also keep their execution mode scoped -to the request. When an overview brief opens Assistant, the drawer may prefill -only governed prompt/context data, but the submitted chat request must set -`autonomous_mode:false`, preserve the operator's persistent Assistant -control-level setting, and disclose the temporary approval-required mode in -the drawer instead of showing the generic Autonomous warning. +Future route-to-Assistant handoffs inherit the server-set execution mode +instead of carrying one. When an overview brief opens Assistant, the drawer may +prefill only governed prompt/context data; the submitted chat request carries +no execution-mode field (`/api/ai/chat` runs every request approval-required), +the operator's persistent Assistant control-level setting stays untouched, and +the drawer discloses the approval-required mode through the +`autonomousMode:false` context flag instead of showing the generic Autonomous +warning. Scoped Assistant handoffs that originate in owned product surfaces may also send bounded `handoff_context` text, structured `handoff_resources`, and safe structured `handoff_actions` through `frontend-modern/src/api/aiChat.ts` and `/api/ai/chat`. That context is model-only session metadata, not saved user-authored message text, and the backend must clamp the exchange to approval-required mode whenever such scoped handoff context, resources, or -action references are present. Patrol finding IDs remain stricter: when +action references are present (`HandleChat` currently clamps every exchange). +Patrol finding IDs remain stricter: when `finding_id` resolves, backend-refreshed durable Patrol context remains the canonical authority; the handler may merge only a recognized same-finding Patrol product handoff section as secondary model-only briefing, and it must @@ -7516,6 +7555,18 @@ prompt budget. Reporting is therefore an additive consumer of AI runtime, not a new ownership boundary, and the narrator/findings surfaces inherit the same governance the rest of the canonical AI runtime already enforces. +Report narration is caller-requested spend, so on the single-resource HTTP +route only `POST /api/admin/reports/generate` reaches it (fleet reports, from +`POST .../generate-multi` or a report schedule, use the fleet narrator). The +reporting narrator resolver goes through `AISettingsHandler.GetAIService`, +which constructs a tenant service on first use and can then list provider +models or start background discovery. A GET single-resource report never calls +that resolver: +it narrates deterministically and reads Patrol findings through +`AISettingsHandler.ExistingAIService`, which returns only a service that is +already running and never constructs one +(`TestAISettingsHandler_ExistingAIServiceNeverConstructs`, and through the +router's real wiring `TestContract_SingleReportGETDoesNotConstructTenantAIService`). The same canonical AI runtime now also owns the fleet-level report narrative through `report_fleet_narrator.go`. `Service` implements diff --git a/docs/release-control/v6/internal/subsystems/alerts.md b/docs/release-control/v6/internal/subsystems/alerts.md index 513b44c27d..bb6ec90d70 100644 --- a/docs/release-control/v6/internal/subsystems/alerts.md +++ b/docs/release-control/v6/internal/subsystems/alerts.md @@ -1609,6 +1609,34 @@ resolve overrides through the shared candidate set `buildProjectedOverrides` indexes platform alert rows under). Static metric-color defaults are only fallback presentation behavior for callers that do not have alert configuration in scope. +Guest bars follow `pulse-relaxed` the same way. `resolveMetricDisplayThresholds` +takes the tags the engine reads for a guest-scope resource and, when they +include `pulse-relaxed`, applies the `applyRelaxedGuestThresholds` step to +CPU, memory and disk before mapping to display thresholds: an unset threshold +gets the 95/92/95 floor with a clear five below it, a lower trigger is raised +to the floor and keeps its own clear point (a missing one filled five below, +as `ensureHysteresisThreshold` does) unless the raised trigger reaches it, and +an Off threshold stays Off. A relaxed guest's bar therefore turns critical +where its alert fires; 88% memory reads warning, not critical, under the +relaxed 92% trigger. The display resolver does not read filter-driven custom +rules, which the engine folds in before the override, so a guest a custom +rule matches can still colour by thresholds the engine does not use, relaxed +or not. Only Proxmox VMs and LXCs carry their tags into alert +policy, so `getWorkloadAlertPolicyTags` in +`frontend-modern/src/components/Workloads/workloadTopology.ts` passes tags for +those guests alone, and a vSphere VM or a container keeps its thresholds +whatever it is tagged. The Alerts thresholds row for a relaxed Proxmox guest +keeps showing the configured values its editor changes and adds one line +naming the tag and the thresholds it raises +(`getAlertResourceRelaxedTriggerSummary` in +`frontend-modern/src/components/Alerts/alertResourceTableModel.ts`, fed by +`pulseRelaxed` from `useThresholdsGuestData`); before it, the tag overrode +those settings with no visible sign in the UI (issue #863). Proof: the +pulse-relaxed cases in `frontend-modern/src/utils/__tests__/metricThresholds.test.ts`, +the alert policy tag cases in +`frontend-modern/src/components/Workloads/__tests__/workloadTopology.test.ts`, +and the pulse-relaxed guest cases in +`frontend-modern/src/components/Alerts/ResourceTable.test.tsx`. Docker container image-update alerts are lifecycle-governed by the alerts runtime. Disabling Docker update alerts globally, disabling alerts for a @@ -2642,7 +2670,7 @@ table subgroup-row contract. `AlertHistoryTableGroupRow` and grouped rows in `AlertResourceTableDesktop` must route their date/resource group bands through `frontend-modern/src/components/shared/groupedTableRowPresentation.ts` instead of local `bg-surface-alt` fills, so alert subgroup hierarchy stays visually -consistent with Infrastructure, Workloads, Storage, and Recovery tables. +consistent with Infrastructure, Workloads, and Storage tables. Alert history table shells must also rely on the shared `TableCard` frame and the shared `Table` primitive for horizontal overflow rather than adding alert-local bordered or `overflow-x-auto` wrappers inside the history section. @@ -3136,21 +3164,22 @@ issue sequential route writes. That reset removes all three query parameters in one navigation and clears any transient chart-bucket selection, so a search-only result set remains visibly resettable and an older URL write cannot resurrect another filter. -That same history surface now also owns the canonical resource-incident -handoff. `frontend-modern/src/features/alerts/AlertResourceIncidentsPanel.tsx` -must treat the selected incident resource as a unified-resource consumer, -linking back into canonical infrastructure/resource detail first and then into -shared workloads, storage, and recovery surfaces through -`frontend-modern/src/routing/resourceLinks.ts` rather than leaving the panel -as a dead-end investigation card or rebuilding provider-local route strings for -platforms such as TrueNAS. -That same alert handoff must now stay on the shared resolved-resource link -builder. `AlertResourceIncidentsPanel.tsx` must resolve its chip set through -`buildResolvedResourceSurfaceLinks(...)`, which owns exact unified-resource -handoffs plus the infrastructure fallback when alert history still references a -resource ID before the backing unified record has hydrated. Future incident-link -work must not reintroduce local infrastructure-link assembly, local dedupe, or -provider-local route strings inside the alert feature shell. +That same history surface also owns the resource-incident panel, and the panel +links nowhere. `frontend-modern/src/features/alerts/AlertResourceIncidentsPanel.tsx` +names the selected resource through the unified-resource lookup it is given +(`getResource`, falling back to the history state's lookup) and +`getPreferredInfrastructureDisplayName`, and shows the resource name the +selected history row carried while no unified record matches. It imports nothing from +`frontend-modern/src/routing/resourceLinks.ts`, and it offers no resource +cross-jump links: investigation continues in place through the incident +timeline cards or hands off to Pulse Assistant through +`IncidentAssistantHandoffButton`, as the retired cross-jump chip note above +describes. Future incident-link work +must not reintroduce a chip set, local infrastructure-link assembly, or +provider-local route strings inside the alert feature shell; +`frontend-modern/src/pages/__tests__/Alerts.helpers.test.ts` pins the panel +free of `buildResolvedResourceSurfaceLinks`, `buildResourceSurfaceLinksForResource`, +and `buildInfrastructureResourceLink`. Alert configuration load/save state, notification config reloads, and threshold override normalization now route through diff --git a/docs/release-control/v6/internal/subsystems/api-contracts.md b/docs/release-control/v6/internal/subsystems/api-contracts.md index 4c34abe088..2e4fb9e94c 100644 --- a/docs/release-control/v6/internal/subsystems/api-contracts.md +++ b/docs/release-control/v6/internal/subsystems/api-contracts.md @@ -2706,9 +2706,10 @@ a new API state machine, queue contract, or verification-accounting field. remediation-plan handoffs must use the same boundary for model-only context: plan status, risk, step labels, and command counts are allowed, while raw command and rollback command payloads remain in governed action - surfaces. Frontend Patrol finding-discussion handoffs must force a - request-local approval-required Assistant mode instead of inheriting the - user's persistent autonomous control setting; live approval, action artifact, + surfaces. Frontend Patrol finding-discussion handoffs run in the + approval-required Assistant mode that `/api/ai/chat` sets server-side for + every request, never the user's persistent autonomous control setting, and + send no execution-mode field of their own; live approval, action artifact, fix-outcome, and remediation-plan references only add structured action metadata, they are not the trigger for the boundary. Frontend-visible Patrol briefing payloads must stay compact and must not include suggested prompt @@ -2716,7 +2717,7 @@ a new API state machine, queue contract, or verification-accounting field. recommendation metadata. Frontend queued-fix recovery handoffs where the live approval or action artifact payload is unavailable must still carry that Patrol-owned finding briefing, current `fix_queued` posture, - request-local approval-required mode, and model-only evidence context; they must + approval-required disclosure, and model-only evidence context; they must not degrade into generic Assistant investigation chat or imply that execution can proceed from missing command payloads. Expired-approval recovery handoffs may use a still-available structured action artifact payload @@ -3329,6 +3330,7 @@ a new API state machine, queue contract, or verification-accounting field. and any frontend presentation of the AI summary's `policy_posture` counts, which must render that canonical snapshot through one shared component instead of page-local posture loops; no frontend surface renders those counts today, and the resource drawer stays on per-resource policy lines and the dedicated `frontend-modern/src/features/patrol/patrolInvestigationContextModel.ts` owner, so if Patrol recent-change, learned-correlation, or policy-coverage summary text returns, it is derived from the canonical AI payload in that one place instead of as hook-local count and pluralization logic; Patrol renders no such summary today and the model carries no such derivation and the Watch-only forward-path handoffs owned by `frontend-modern/src/features/patrol/patrolControlPresentation.ts` together with the Actions inbox empty-state read of `GET /api/ai/patrol/autonomy`, so finding-level and inbox-level mode guidance consumes the canonical Patrol autonomy read/save contract through the Patrol state hook (`handleAutonomyChange`) instead of introducing a second mode mutation path or a page-local autonomy dialect + and the Patrol finding-to-Assistant handoff owned by `frontend-modern/src/components/AI/FindingsPanel.tsx`, so the pending-approval, investigation, and proposed-fix reads behind that handoff happen once in the panel instead of in a second, unwired copy inside the Patrol state hook and that same Patrol investigation-context owner, so visible Assistant drawer handoffs may include live pending-approval metadata only as safe operator context: approval ID, status, risk, requested/expiry timestamps, @@ -3350,8 +3352,8 @@ a new API state machine, queue contract, or verification-accounting field. and the Patrol runtime-remediation destination shared with the AI settings endpoint, so run-history runtime-failure actions and runtime-finding actions may reuse the governed provider-settings route while still presenting that destination in Patrol as provider configuration instead of generic `AI Settings` copy and the Patrol route-shell destination itself, so the thin page shell at `frontend-modern/src/pages/AIIntelligence.tsx` may continue to bridge the shared AI-runtime payload boundary while exposing `/patrol` as the canonical product route and keeping retired `/ai` browser entry points unregistered and the Patrol route-shell accessibility boundary, so brand icons in `frontend-modern/src/features/patrol/PatrolIntelligenceHeader.tsx` stay decorative when the same heading already exposes visible Patrol text, preventing duplicate accessible names such as `Pulse Patrol Patrol` - and the Patrol mode selector boundary, so the default header and the - Patrol mode dialog compose the shared + and the Patrol mode selector boundary, so the header's `Mode and automation` + disclosure composes the shared `frontend-modern/src/components/shared/FilterButtonGroup.tsx` primitive for the visible `Watch only` / `Ask first` / `Safe auto-fix` / `Autopilot` presentation while the API contract remains the sole owner of @@ -3915,6 +3917,13 @@ counters exist to measure. onboarding telemetry. Customer diagnostics may expose runtime health, supportability, and sanitized troubleshooting state; admin analytics must stay behind admin-owned metrics routes. +25. Handlers that perform a caller-requested write (settings, sessions, + credentials, workflow or resource state) while answering `GET` or `HEAD`. + A route registered without a mux method pattern, or a sub-resource + dispatched by path, must refuse safe methods in the mutating handler + through `requireRequestMethod` in `internal/api/method_guard.go` (or an + equivalent explicit check) instead of relying on its callers, the + demo-mode read-only guard, or the CSRF check. ## Completion Obligations @@ -4031,24 +4040,34 @@ the authoritative analysis outcome. self-hosted v6 GA UI and the Patrol autonomy save contract, so Community/free runtime payloads may persist only `monitor` autonomy settings through - `/api/ai/patrol/autonomy`, while `approval`, `assisted`, and `full` return - the canonical license-required response instead of a generic save failure, - and Patrol frontend state owners must clamp stale paid autonomy to `monitor` - and send `full_mode_unlocked:false` before submitting that endpoint when the - safe-remediation entitlement is not effective. The monitor-only backend path - must likewise clear stale full-mode unlock state in its response and - persistence layer instead of preserving paid remediation state through a free - configuration save + `/api/ai/patrol/autonomy`, while `approval`, `assisted`, and an + acknowledged `full` return the canonical license-required response instead + of a generic save failure (the shared acknowledgement gate refuses a `full` + request without a current Autopilot acknowledgement first, for example with + `409 acknowledgement_required` when it carries none), and the Patrol + frontend state owner must + refuse any level other than `monitor` when a save starts while the + safe-remediation entitlement is not effective; its payload carries only + `autonomy_level`, the investigation budget and timeout, and (for Autopilot) + `acknowledgement_id`, never `full_mode_unlocked`. The monitor-only backend + path owns the unlock clamp: + it must clear stale full-mode unlock and Autopilot activation state in its + response and persistence layer instead of preserving paid remediation state + through a free configuration save and the Patrol settings-save readiness contract, so `/api/settings/ai/update` may save a selected Patrol provider/model even when that model is not ready for tool-backed Patrol execution, but it must echo `patrol_readiness` with stable `cause` metadata and execution routes - must continue to fail closed before model calls. Frontend Patrol settings + must continue to fail closed before model calls. Frontend settings consumers must surface that saved-but-not-ready response as a saved - configuration issue with the echoed provider, model, cause, and summary - instead of reporting the successful save as a failed save or hiding the - readiness blocker behind a generic notification; and the structured - investigation-record contract, so unified findings may + configuration issue rather than a failed save, and never as inline failure + state: every Pulse Intelligence settings page (Provider & Models, Patrol, + Assistant, Service context) raises a warning notification naming the echoed + summary, provider, and model, while the Patrol page's on/off save raises a + fixed not-ready warning and leaves the diagnosis to the page's own readiness + surfaces, such as the readiness banner that names the summary, provider, + and model while the header offers `Fix setup` when Patrol is active; and + the structured investigation-record contract, so unified findings may expose `investigation_record` only through the shared `aicontracts.InvestigationRecord` payload shape, with frontend API types and backend contract tests updated in the same slice as any field change. @@ -4297,7 +4316,7 @@ the authoritative analysis outcome. run-history consumers may pass a bounded `[Patrol Run Context]` block, scoped resource references, run outcome/coverage facts, sanitized analysis, and structured runtime failure summary/detail as model-only chat context - while forcing request-local approval-required mode and leaving retries, + under the server-set approval-required mode, leaving retries, configuration changes, and remediation authority outside the chat payload and the main Patrol page composition boundary, so once that governed secondary area exists inside the Current issues and history workspace the same payloads must not @@ -5161,6 +5180,134 @@ pass the bounded non-symlink regular-file boundary in `internal/api/sso_outbound.go`. Cloud handoff redirects consume the shared host-local redirect validator rather than carrying route-local prefix checks. +A handler that performs a caller-requested write behind a route whose mux +pattern carries no method must refuse `GET` and `HEAD` itself. +`requireRequestMethod` in `internal/api/method_guard.go` is the shared +handler-level guard: it admits only the listed methods and answers any other +method that reaches the handler with `405` and an `Allow` header naming them. +The demo-mode read-only guard and the CSRF check both admit `GET` and `HEAD` as +reads, and the `SameSite=Lax` session cookie rides cross-site top-level `GET` +navigations, so the method contract belongs to the mutating handler rather than +to its callers. `/api/discovery/settings` accepts `PUT`, its documented method, +and `POST`, which it took before the guard existed. SP-initiated SAML logout at +`/api/saml/{id}/logout` accepts only `POST`, matching the `/api/logout` +fallback it delegates to. The IdP's LogoutResponse keeps returning to +`/api/saml/{id}/slo`, which must accept the HTTP-Redirect binding's `GET` and +validates the signed response before it touches the session. +`TestDemoModeSafeMethodsCannotReachRouteMutations` in +`internal/api/ai_handlers_more_test.go` drives both routes through the full +`DEMO_MODE` router and proves the threshold and the session survive. +The rule does not cover `GET` flows the protocol or the browser handoff +requires (OIDC login and callback, SAML login initiation and SLO, magic-link +verify, cloud handoff, the checkout start and activation bridges, websocket +handshakes), or reads that run one-time lazy initialization or idempotent +reconciliation, such as the license and onboarding-overflow bootstrap behind +the entitlement reads and the action-expiry sweep behind the action reads. +Single-resource report generation at `GET /api/admin/reports/generate` still +runs the configured AI narrator, a paid provider call recorded in the cost +ledger; that is a known open exception, not part of this contract. + +The SAML SLO callback at `/api/saml/{id}/slo` accepts a LogoutResponse only as +the answer to a logout this SP started. crewjam/saml verifies the response's +signature, `Destination`, `Issuer` and `Status` and rejects one issued more +than 90 seconds ago, but never checks its `InResponseTo`, and the IdP signs its +answer to every user's logout, so a valid signature alone let one user replay +the answer to their own logout through a cross-site `GET` and clear another +user's session. `SAMLService.MakeLogoutRequest` in +`internal/api/saml_service.go` therefore records each LogoutRequest ID, on the +provider's own service, against the session-store hash of the session +`handleSAMLLogout` is ending. A record is honored for ten minutes unless the +1,024-record cap evicts it first, oldest first. `SAMLService.ValidateLogoutResponse` +reads `InResponseTo` from the same payload crewjam verified, taking only the +root element's attribute written without a prefix, the one the signature +covers (a namespace declaration such as `xmlns:InResponseTo` is ignored, +because exclusive canonicalization can leave an unused one out of the signed +form), and +spends the matching record. The response is accepted only when that record +exists and is unexpired, and the browser carries either no session cookie (the +normal case, because `handleSAMLLogout` clears the cookie before redirecting to +the IdP) or the session that request ended. Unsolicited, expired, evicted, +replayed and other-session responses are refused with `403` and a failed +`saml_slo_callback` audit event, and the handler leaves sessions and cookies +alone. A misdelivered response still spends its record, so each LogoutRequest +is answered at most once. The handler writes no session state or cookie for an +accepted response either: `handleSAMLLogout` already invalidated the session +and expired its cookies, and a cross-site HTTP-POST delivery withholds the +`SameSite=Lax` cookie, so a deletion cookie written there could log out a +browser whose session the request never presented. For the same reason +`handleSAMLLogout` answers `401` to a request that carries no session cookie, +API-token-only callers included, instead of falling back to a cookie-clearing +logout: the SAML routes are public and skip CSRF, so that fallback was a +cross-site forced logout of a kind a deployment with authentication configured +refuses on `/api/logout` as unauthenticated. The records live in memory, +so a restart or provider re-initialization during the IdP round trip leaves the +returning response unmatched and refused; the local session was already +cleared when logout began. `TestSAMLSLOAuditsLogoutResponseNotBoundToSession` +in `internal/api/audit_handlers_test.go` replays one user's signed response +into another user's browser and proves the victim's session survives and the +refusal is audited. + +The SAML ACS at `/api/saml/{id}/acs` accepts a Response, at default settings, +only as the IdP's answer to a login the presenting browser started. crewjam/saml +refuses a Response unless its `InResponseTo`, and that of each bearer +`SubjectConfirmationData`, is one of the request IDs its caller supplies, and +Pulse used to supply none, so with `allowIdpInitiated` off, the default, every +SAML login failed with "Authentication failed". `SAMLService.MakeAuthRequest` +in `internal/api/saml_service.go` therefore records each AuthnRequest ID, on +the provider's own service, with the sanitized `returnTo` and the hash of a +random token that `handleSAMLLogin` sets as the `HttpOnly`, `SameSite=Lax` +cookie `pulse_saml_login` (`__Host-pulse_saml_login` over HTTPS) for ten +minutes. A browser that already holds a token keeps it, so logins started one +after another in several tabs all stay valid. A record is honored for ten +minutes, judged when the Response spends it, unless the 1,024-record cap +evicts it first, oldest first. The IdP delivers its Response by a cross-site +HTTP-POST, which browsers send without `SameSite=Lax` cookies, so +`handleSAMLACS` answers a POST that carries a `SAMLResponse` or +`SAMLart` but no login cookie with a page that posts the same fields once more +from Pulse's own origin. The form has no `action`, so it returns to the URL +the IdP posted to, path prefix included, its script carries the request's CSP +nonce, and a marker field prevents a second repost. `SAMLService.ProcessResponse` +then has crewjam check the Response against every outstanding request ID and +reads the request it answers from the canonical form goxmldsig verified for +the signature covering the assertion crewjam returned, never from crewjam's +parsed fields, taking only an `InResponseTo` written without a prefix: the +signed `Response`'s, or that of the `Response` inside a signed +`ArtifactResponse`, when the IdP signed the message, and otherwise that of +every `SubjectConfirmationData` in the returned assertion's own `Subject`, +which must agree and of which there must be at least one, and no other signed +assertion may carry its ID. Other assertions in the Response, and assertions +in `Advice`, do not take part. That record is +spent only when it is bound to the presenting browser's token, and the login +returns to the recorded `returnTo`, not to the posted `RelayState`, which +`MakeAuthRequest` now escapes into the redirect query so a `returnTo` with its +own query string is not cut short at the IdP. A Response answering another browser's login, an unknown, +expired or already spent request, or none is refused with the +`saml_validation_failed` redirect and a failed `saml_login` audit event, and +refusing another browser's Response leaves that browser's record for it. With +`allowIdpInitiated` on, crewjam skips `InResponseTo`, so a Response answering +any request or none is accepted in any browser without the cookie or the +repost and returns to its posted `RelayState`, as before. In both modes the +signature verifier refuses signed content carrying a prefixed attribute or a +namespace declaration named like an un-namespaced attribute crewjam reads from +a Response or Assertion, such as `xmlns:IssueInstant`, `xmlns:NotOnOrAfter` or +`ext:ID`: encoding/xml fills an un-namespaced attribute field from any +attribute with that local name, and exclusive canonicalization leaves an +unused declaration out of the signed form, so appending one let a stale signed +Response pass crewjam's age checks, and an IdP-initiated login could be +replayed indefinitely. The records live in memory, so a +restart or a saved provider change (which replaces the provider's service; +a metadata refresh keeps the records) during the IdP round trip, or a login +started on a host that does not share cookies with the configured public +URL's, leaves the Response unmatched and the user starts the login again. Two logins started at +the same moment in a browser that holds no token yet each mint one and only +the last cookie set survives, so the other fails the same way, and, as with +OIDC login state, a flood of unauthenticated login starts can evict pending +records. +`TestSAMLACSAuditsResponseNotAnsweringThisBrowsersLogin` in +`internal/api/audit_handlers_test.go` delivers one browser's signed Response to +another browser with a login in progress and proves it is refused and audited +while its own browser completes the login. + Alert delivery diagnosis is a read-only monitoring API contract. `GET /api/alerts/delivery-diagnosis?alertIdentifier=` returns the alert manager's current delivery-policy projection for one active alert, including @@ -5421,16 +5568,51 @@ the projection to the Finding before calling the orchestrator. This is the one in-process write path that decides what the orchestrator sees about operator commitments; all consumers (in-process Patrol, external Claude Code, and MCP-speaking clients) read the same enriched shape. -The provider closure must tolerate the findings runtime's mixed -resource key spaces: unified-derived findings reference the hashed -canonical resource ID, but Patrol guest inventory rows reference the -node-scoped Proxmox source ID (`instance:node:vmid`). Operator state -is keyed by canonical ID only, so a reference that misses the store -directly resolves through the registry (`GetByReference`, which also -covers retired canonical-ID eras and node-scoped guest references -after #1669) before the provider reports no state — without that -hop, maintenance windows and intentionally-offline intent silently -never reach guest findings. Pinned by +The provider (`Router.patrolResourceOperatorStateProvider`) must +tolerate the findings runtime's mixed resource key spaces: +unified-derived findings reference the hashed canonical resource ID, +Patrol guest inventory rows reference the node-scoped Proxmox source +ID (`instance:node:vmid`), and a finding raised before an operator +link folded its resource into another keeps the folded canonical ID, +which Patrol re-projects as stored. Operator state is keyed by +canonical ID only, so every reference resolves before the store read, +through the identity-only `ResolveCanonicalResourceID` of the +organization monitor's current read state +(`GetUnifiedReadStateOrSnapshot`: the published registry with saved +hosts that have not reported since a restart overlaid, or the mock +estate). That read state is the listing the resources API seeds its +registry from, and resolution covers retired canonical-ID eras, +node-scoped guest references after #1669 and link-folded IDs. Only a +reference it cannot place is read as given. Resolving first is what +keeps Patrol on the row the resources API writes for the same +reference: an exact read first would let a row left under a folded +ID, even a default-valued one, hide the maintenance window the +operator then set on the merged resource. A saved host that has not +reported since a restart but is linked into a live resource resolves +alike everywhere: the read state keeps it as its own offline row, so its +saved telemetry never reaches the resource, but holds it under the +link's primary, and the resources API's registry, seeded from that +listing, holds it the same way instead of folding it. A finding, an +alert and an operator-state write naming the saved agent by its +canonical ID or `agent:` therefore all reach the resource the +agent was folded into before the restart. Only when the saved agent is +itself the link's primary and the resource is not its guest does each +keep its own identity until the agent reports. Two cases still resolve +differently in the resources API, whose registry re-ingests that +listing: references to records it replays from supplemental providers +on top of the listing, and manual links in mock mode, whose view +applies none. The read +state answers from a registry it keeps (a continuity overlay is shared +and reused for up to two seconds while the published registry and the +saved hosts are unchanged), so findings do not each clone the unified +listing the way a resources-API registry build does; only a monitor +left without a resource store, which the router never does, builds a +snapshot registry per call. Pinned by +`TestPatrolFindingOperatorStateFollowsLinkFoldedReference` and +`TestPatrolFindingOperatorStateResolvesContinuityBackedHost` in +`internal/api/ai_handlers_more_test.go`, +`TestResourcesAPIAndPatrolAgreeOnContinuityAgentLinkedIntoGuest` in +`internal/api/resources_link_continuity_test.go` and `TestContract_FindingsResourceOperatorStateProviderIsWired` in `internal/api/contract_test.go`. @@ -6446,6 +6628,25 @@ That same SSO boundary also owns manual SAML endpoint validation payloads. through the same validated absolute HTTP(S) helpers instead of letting the manual logout URL drift out of the request model or bypass the governed URL normalization path. +That same manual SAML configuration must verify signatures with the IdP +certificate the administrator supplied. With no IdP metadata, +`SAMLService.buildManualMetadata` in `internal/api/saml_service.go` builds the +IdP descriptor from `idpSsoUrl`, optional `idpSloUrl` and the PEM signing +certificate in `idpCertificate` or `idpCertFile`, and must put that certificate +in the signing `KeyDescriptor` as base64 DER, the `` content +IdP metadata carries. crewjam/saml base64-decodes that field to verify every +ACS Response and LogoutResponse, so the PEM block it once held failed each +check with "illegal base64 data" before any key was compared, and no manually +configured provider could accept a login or complete SLO. The stored +`idpCertificate` stays the PEM the administrator supplied; only the descriptor +rebuilt in memory on each load changed, so no saved configuration needs +migrating. `TestContract_SAMLManualIDPCertificateVerifiesIdPSignatures` in +`internal/api/contract_test.go` signs an ACS Response and a LogoutResponse with +one test IdP key and proves an inline-certificate provider, a certificate-file +provider and a metadata provider accept both alike, while a Response signed by +another key is refused. The proof runs at default settings, with each +Response answering an AuthnRequest that `SAMLService.MakeAuthRequest` +recorded for the test browser, as the SAML ACS binding above requires. That same SSO provider-detail boundary must return the non-secret nested provider configuration used by the settings edit form. `GET /api/security/sso/providers/{id}` may keep the flat list/card fields for @@ -7143,8 +7344,8 @@ unified-resource mentions as well; VMware network inventory does not create a provider-local mention family. That same `/api/ai/chat` payload boundary owns request-local execution mode, and the server sets it. `ChatRequest` in `internal/api/ai_handler.go` has no -`autonomous_mode` field, so the `autonomous_mode:false` that scoped handoffs -such as an explain-this-issue request still send is dropped at decode, and the +`autonomous_mode` field, so a stale client's `autonomous_mode` is dropped at +decode (the browser and the `internal/ai/eval` runner send none), and the handler passes approval-required mode into the chat service for every exchange; an Autonomous control level runs as Controlled for that request. That clamp is request-local and must not mutate the user's persistent AI @@ -7184,8 +7385,9 @@ status, and a runtime-failure flag; the backend rebuilds their context), but the must not turn their evidence or proposed remedies into user-authored message text. Ordinary attachment is context only. A labelled Explain action may submit the shared user-selected explanation request over the existing chat -transport, with the evidence kept in model-only handoff fields and -`autonomous_mode:false`. This starts explanation/review and grants no new +transport, with the evidence kept in model-only handoff fields and no +execution-mode field, since the handler already runs it approval-required. +This starts explanation/review and grants no new approval or execution authority. When a Patrol `finding_id` resolves, backend-refreshed durable finding context remains canonical; the handler may @@ -7205,10 +7407,10 @@ recommendation fields from legacy handoffs. Patrol finding handoffs keep that clamp on their own path: when a request carries a non-empty `finding_id` or resolves to model-only Patrol briefing, resource, or action context, `internal/api/ai_handler.go` must clamp the -request-local autonomous mode to false even if the caller supplied +request-local autonomous mode to false even if a caller's JSON carries `autonomous_mode:true`. That server-side clamp is part of the public API -contract because the frontend handoff setting is only advisory unless the -backend preserves the approval-required boundary. +contract because the drawer's `autonomousMode:false` flag is a disclosure only +and never reaches the request. That same backend API boundary now also owns the negative space around assistant control. Wiring native TrueNAS app actions into `internal/api/router.go`, `internal/api/ai_handler.go`, or adjacent backend @@ -7329,6 +7531,12 @@ no client for it, and neither the dashboard load bundle in fetches it. The resource drawer's correlation evidence comes from the resource-intelligence payload above. The route stays an API surface behind the `ai:execute` scope. +The frontend does not load `GET /api/ai/circuit/status` either. No surface +renders provider breaker state, so `frontend-modern/src/api/ai.ts` carries no +client or `CircuitBreakerStatus` type for it, and neither that dashboard bundle +nor the Patrol refresh path fetches it. An open breaker reaches the Patrol page +through `/api/ai/patrol/status` as the `circuit_open` blocked cause. The route +stays a backend API surface. That correlations route now reads through the canonical AI intelligence facade first, so the handler and its payload keep the detector behind one shared access layer instead of routing directly to Patrol-local correlation @@ -8729,7 +8937,7 @@ briefings may reuse that same safe metadata for factual action labels and safety while approval command text remains inside the governed approval/remediation surface. Patrol approval-row Assistant handoffs must use the same safe metadata boundary -and set `autonomousMode:false` for the request-local chat handoff; they must not +and set the drawer's `autonomousMode:false` approval disclosure; they must not paste raw approval or action command text into a chat prompt. Patrol remediation-plan or action-artifact Assistant handoffs must pass only safe status, risk, description, and command-count posture as non-authoritative context @@ -9635,19 +9843,22 @@ configuration from paid remediation autonomy. `GET /api/ai/patrol/autonomy` continues to clamp effective Community autonomy to `monitor`, and `PUT /api/ai/patrol/autonomy` in the open-source/free adapter must accept and persist only `monitor` settings while returning the canonical license-required -payload for investigation or remediation autonomy levels. Frontend Patrol state -owners must not rely on that 402 as normal control flow for stale local state: -when the current entitlement locks safe remediation, they submit `monitor` even -if older persisted settings or a previous entitlement left `approval`, -`assisted`, or `full` in memory. +payload for investigation or remediation autonomy levels (a `full` request +reaches that adapter only with a current Autopilot acknowledgement; otherwise +the shared acknowledgement gate refuses it first, for example with +`409 acknowledgement_required` when it carries none). Frontend Patrol +state owners must not rely on that 402 as normal control flow for stale local +state: when the current entitlement locks safe remediation, the Patrol header +presents `monitor` as the effective mode and the state owner starts no save but +an explicit `monitor` choice, even if older persisted settings or a previous +entitlement left `approval`, `assisted`, or `full` in memory. After a successful persisted update whose effective mode actually crosses from `monitor` to `approval`, `assisted`, or eligible `full`, the shared API mutation boundary invokes the bounded Patrol actionable-finding reconciliation pass. The pass runs only after `ApplyPatrolAutonomyConfig` has published the new runtime authority; failed writes, Community clamps, repeated paid-mode saves, budget-only edits, and demotions must not start backlog work. -The Patrol header and configuration-dialog presentation for that API boundary -must compose the shared +The Patrol header presentation for that API boundary must compose the shared `frontend-modern/src/components/shared/FilterButtonGroup.tsx` selector: the endpoint contract still owns the accepted autonomy values and license-required response shape, while the frontend owns only the option mapping, @@ -9686,6 +9897,32 @@ narrator must fail closed: nil provider, parse failure, timeout, or empty response causes the engine to fall back to the heuristic narrative without surfacing the AI failure to the caller, so reporting is never blocked by AI availability. +Only `POST /api/admin/reports/generate` may hand the tenant's AI narrator to +the engine, because a narrated PDF makes a paid provider call and appends a +`cost.UsageEvent` to the cost ledger. The single-resource handler accepts +`POST` with a JSON body (the `generate-multi` decoding rules: 1MB cap, unknown +fields and trailing payload rejected, through `decodeReportingRequestBody`) and +`GET` with the same fields as query parameters, and answers any other method +with `405` and `Allow: GET, POST`. GET never calls `resolveNarrator`, whose +`GetAIService` can construct a tenant AI service (and with it list provider +models or start background discovery): it narrates through `getReportNarrator`, +the heuristic summary whose `Disclaimer` replaces the "Configure Pulse +Assistant" tip with a note that AI narration needs POST, and reads Patrol +findings through `SetExistingFindingsResolver`, which `router.go` backs with +`AISettingsHandler.ExistingAIService` so only an already running service is +used. GET passes the demo-mode guard and the CSRF check, and a SameSite=Lax +session cookie rides a cross-site top-level GET navigation, so a link must not +be able to spend AI budget. The reporting catalog advertises the transport as +`performanceReport.singleResourceMethod` (`POST`), and the settings UI follows +it. `generate-multi` and the manual schedule run endpoint +(`POST /api/admin/reports/schedules/{id}/run`) were already POST-only; due +schedules run from the server's scheduler, not an HTTP request. +`TestContract_SingleReportAINarrationRequiresPOST` pins the rule on a real +engine: GET, HEAD, PUT and DELETE make no narrator call and no AI service +resolution, while POST makes one of each. +`TestContract_SingleReportGETDoesNotConstructTenantAIService` drives +`Router.wireReportingAIResolvers` with an uncached tenant: GET constructs no +tenant AI service and POST resolves one. Report alerts keep the alert engine's resolution. The node, VM and container enrichers in `internal/api/metrics_reporting_handlers.go` build every row through `reportActiveAlertInfo` and `reportResolvedAlertInfo`, @@ -11462,3 +11699,17 @@ Router's or the package-global limiters. Limits, windows, tenant selection, revocation and response payloads are unchanged. The connected lifetime and ownership controls are in `router_limiter_lifecycle_test.go`; the integration fixture's cleanup also runs on a failed listener/request assertion. + +### Report-merge undoes operator links + +`POST /api/resources/{id}/report-merge` keeps its request, status codes and +response shape. For a reported source that an operator link brought in, it +now records the exclusion against the link's own pair instead of a candidate +ID derived from the merged resource's type, so the store replaces the link +and the pair splits as after unlink (contract under "Report-merge splits +operator-linked resources" in unified-resources). `exclusions` counts those +pairs with the identity-match exclusions. `400 Resource is not merged` and +`400 No source targets found` now also need a resource with no applied link, +so two linked agents, which share their only source, can be reported. +`TestContract_ResourceReportMergeReplacesOperatorLink` pins the response +shape and the link's removal for a VM and the agent linked into it. diff --git a/docs/release-control/v6/internal/subsystems/cloud-paid.md b/docs/release-control/v6/internal/subsystems/cloud-paid.md index 693ed47bcc..471f052540 100644 --- a/docs/release-control/v6/internal/subsystems/cloud-paid.md +++ b/docs/release-control/v6/internal/subsystems/cloud-paid.md @@ -1161,18 +1161,21 @@ hands-on Patrol modes, issue investigation, verified fixes, and longer history`. absent platform families must not render as disabled placeholders in the paid or hosted app shell. Infrastructure is not an equal primary tab in that list. Workloads, Storage, and Recovery are not standalone aggregate - workspace tabs; their component surfaces remain reusable inside platform - pages via embedded `tableOnly` surfaces. Platform primary-tab settings + workspace tabs. Workload and storage rows render inside platform pages, + through the canonical `WorkloadsSurface` or `StorageSurface` where a page + embeds one and through platform-owned tables elsewhere; recovery rows + render only in platform-owned tabs such as Proxmox Backups and TrueNAS + Protection, because no Recovery surface exists. Platform primary-tab settings handoffs in `AppLayout.tsx` must target the canonical `/settings/infrastructure` workspace and must not retain retired settings aliases such as `/settings/workloads/docker` or nested - `/settings/infrastructure/platforms/*` paths. Each platform page must - remain chrome-only: routing plus sub-tab navigation that embeds the - canonical `WorkloadsSurface`, `StorageSurface`, or `RecoverySurface` in - `embedded tableOnly` mode with a forced platform/source filter. The shell must not introduce dashboard cards, - bespoke per-family tables, synthetic placeholder data, or reintroduce - Infrastructure as a primary navigation entry without a governed contract - decision recorded by the frontend-primitives owner. + `/settings/infrastructure/platforms/*` paths. Which platform tabs embed a + canonical surface and which render platform-owned tables is the + frontend-primitives platform-page rule; cloud-paid must not carry a + second copy of it. The paid or hosted shell must not introduce dashboard + cards, synthetic placeholder data, or forked per-family table primitives, + or reintroduce Infrastructure as a primary navigation entry, without a + governed contract decision recorded by the frontend-primitives owner. The presence of the user-facing `Machines` tab, backed by the internal `standalone` route/id, is not a new commercial usage unit: hosted and paid surfaces continue to meter the governed monitored-system grouping result @@ -1583,9 +1586,10 @@ follow the same canonical home. The hosted bootstrap / activation / recovery routes (`/pricing`, `/preview/setup-complete`, `/login`, settings panels under `/settings/...`) are unchanged. Pricing-handoff and hosted-signup upgrade paths must use platform routes as their canonical "back to product" -destination instead of the retired top-level Infrastructure route; aggregate -Workloads / Storage / Recovery pages are product workspaces, not hosted -acquisition destinations. The Docker / Podman runtime route remains `/docker` +destination instead of the retired top-level Infrastructure route. Platform +pages are product workspaces, not hosted acquisition destinations, and the +retired aggregate Workloads / Storage / Recovery pages are not destinations at +all. The Docker / Podman runtime route remains `/docker` for search familiarity, but the shell may label the destination as Docker so it reads as the recognizable container runtime lens rather than an owning infrastructure platform. diff --git a/docs/release-control/v6/internal/subsystems/deployment-installability.md b/docs/release-control/v6/internal/subsystems/deployment-installability.md index 3af74e0266..14ef915a65 100644 --- a/docs/release-control/v6/internal/subsystems/deployment-installability.md +++ b/docs/release-control/v6/internal/subsystems/deployment-installability.md @@ -6101,6 +6101,14 @@ summaries whose scope, format, and attachments the server fixes. The delivery surface (`/api/admin/reports/schedules`) and tenant-local storage are unchanged. +### Provider docs name POST for single-resource reports + +`docs/MSP.md` and its shipped copy now give `POST /api/admin/reports/generate` +for per-client single-resource reports and say a `GET` with query parameters +still returns the report without the Pulse Assistant narrative, matching the +server, which lets only `POST` narrate. Install, upgrade and provider bundle +delivery are unchanged. + ### Convergence log-reader compatibility (5 September 2026) Scheduled reconciliation run 33939926237 failed when `gh api` refused terminal diff --git a/docs/release-control/v6/internal/subsystems/frontend-primitives.md b/docs/release-control/v6/internal/subsystems/frontend-primitives.md index 355370e588..ebc14a10df 100644 --- a/docs/release-control/v6/internal/subsystems/frontend-primitives.md +++ b/docs/release-control/v6/internal/subsystems/frontend-primitives.md @@ -730,11 +730,14 @@ history, and must not discard the saved note. Platform-owned workload controls extend the shared `WorkloadsFilter` view options rather than creating page-local toolbar shells. Persistent presentation choices compose the shared `ViewOptionsDisclosure` instead of occupying the -primary filter rail: layout, metric style, chart visibility, memory basis, and -columns remain discoverable behind one `View` trigger. The history range stays +primary filter rail: layout, metric style, row hover, memory basis, inventory +totals, and columns remain discoverable behind one `View` trigger. The history range stays inline in both metric modes because bars now expose an intent-driven row -history lens and Trends keeps the same charts persistent. That inline range -must carry a visible contextual label. +history lens and Trends keeps the same charts persistent. In Bars the range +renders only at desktop widths while Row hover is set to `History`, the mode +that drives that lens; `Details` hover and the phone (`isMobile`) layout hide +it. The inline range carries the visible `History` label that +`WorkloadsFilter` passes to `MetricHistoryRangeSegmentedControl`. Controls inside the View disclosure must expand in place rather than opening nested absolute panels that can clip or create competing overlay stacks. The Proxmox page owns and persists the `Guest` / `Host` memory basis; @@ -2352,6 +2355,20 @@ not a replacement status card, CTA band, or page-local nested card. Proxmox backup source/state chips follow the same boundary: storage/recovery owns backup-source labels and state semantics, while the visible chip shell and tone vocabulary route through `MetadataBadge`. + Resource drawer identity chips (the IP address and alias chips and the + alias overflow count in `ResourceDetailSummary.tsx`) compose `MetadataBadge` + too. Badges stay on one line by default; an identity value that can outgrow + a narrow drawer cell opts into the badge's `wrap` mode, which caps the chip + at its container and breaks inside a token only where no space or hyphen + fits. These values differ at their ends (`...:pve1:113`, a container hash) + and a phone has no hover to show a title, so they wrap rather than + truncate. A one-line alias used to overflow its fixed-width value cell at + phone widths, where the cell's hidden overflow and the inline drawer's + clipping content shell cut it off without an ellipsis. + `tests/integration/tests/04-mobile.spec.ts` opens a Docker container drawer + at phone width and checks that no element or text in its Identity section + crosses either edge of those clipping boxes, apart from values that end in + their own ellipsis. Workload backup freshness follows a separate shared presentation boundary: `frontend-modern/src/utils/workloadGuestPresentation.ts` owns the canonical tone and icon mapping consumed by workload rows and drawers. A recorded @@ -2635,15 +2652,20 @@ not a replacement status card, CTA band, or page-local nested card. drifting off-screen from page-local flex behavior. The shared action rail must align to the trailing edge at wrapped desktop widths and remain separate from the filter-control wrap zone instead of waiting for a wide - breakpoint, so Recovery events, Workloads, Storage, Infrastructure, and + breakpoint, so Workloads, Storage, Infrastructure, and future dense toolbars do not strand Filter/Columns/Reset actions as an isolated second-row fragment. Shared `FilterToolbarPanel` owns - default filter-popover geometry, and `FilterToolbar` owns the shared chart - visibility display action: Workloads, Storage, Infrastructure, and future - summary-bearing pages must use `ChartVisibilityToggleButton` so the - affordance exposes one `Show charts` / `Hide charts` pressed-state contract - instead of rebuilding a one-option segmented control or an in-summary - collapse chevron page by page. Feature state hooks under + default filter-popover geometry. The workload summary chart section that the + shared chart show/hide action toggled is deleted and no other page exposes + one, so the shared toolbar carries no chart show/hide action: the + `ChartVisibilityToggleButton` primitive, its + `chart-visibility-toggle-button` rule, and its + `chart-visibility-local-toggle-labels` pattern guard are retired, and + `SharedPrimitives.guardrails.test.ts` keeps them out. A summary section that + returns through a governed product decision must register one shared + show/hide primitive with an explicit pressed state rather than a page-local + one-option segmented control or in-summary collapse chevron. + Feature state hooks under `frontend-modern/src/features/` own route-backed query state, selected item state, and data-window selection for their product surfaces; shared primitives and reusable presentation helpers may own viewport-safe chrome, @@ -3031,9 +3053,9 @@ Agent`), with the plain-language source phrase available through accessible than receiving an unfiltered copy of the estate. Opaque unified-resource ids are not part of that visible search vocabulary and must not retain a node when the normalized guest table has no corresponding match. -8. Keep summary interaction identity on one shared helper. Pages that expose row-hover, group-hover, or route-focus-driven summary emphasis, today the Workloads and Storage tables through `useSummaryPageInteractionState` in `frontend-modern/src/components/shared/summaryTableFocus.ts`, must derive page/group/entity scope through `frontend-modern/src/components/shared/summaryCardInteraction.ts` rather than letting each surface read `hovered || focused` from its own page-local ID source. Hovering or pinning a storage pool-group header, or a workload group header on a surface that enables inline group drawers (the Proxmox and VMware surfaces pass `groupNodeDrawerMode="disabled"` today), must scope the matching rows through that same shared contract instead of forking a page-local summary filter path. No summary card or chart strip renders today, and the bridge carries no chart-hover input; one that returns must consume that same resolved scope, and the chart under the pointer must enter its hovered series into `resolveSummaryScopeState` as a preview input that outranks row hover instead of keeping a chart-local hover island. Recovery is explicitly outside this interaction dialect: its retired posture-card strip must not return with row/group/chart hover behavior without a separate governed product decision. +8. Keep summary interaction identity on one shared helper. Pages that expose row-hover, group-hover, or route-focus-driven summary emphasis, today the Workloads and Storage tables through `useSummaryPageInteractionState` in `frontend-modern/src/components/shared/summaryTableFocus.ts`, must derive page/group/entity scope through `frontend-modern/src/components/shared/summaryCardInteraction.ts` rather than letting each surface read `hovered || focused` from its own page-local ID source. Hovering or pinning a storage pool-group header must scope the matching rows through that same shared contract instead of forking a page-local summary filter path. Workloads grouped host rows are static identity dividers and pass no group scope to the bridge; a grouped table that wants group hover or pin again needs a demand-ledger entry and must use this shared scope rather than reviving a Workloads-local path. No summary card or chart strip renders today, and the bridge carries no chart-hover input; one that returns must consume that same resolved scope, and the chart under the pointer must enter its hovered series into `resolveSummaryScopeState` as a preview input that outranks row hover instead of keeping a chart-local hover island. Recovery is explicitly outside this interaction dialect: its retired posture-card strip must not return with row/group/chart hover behavior without a separate governed product decision. 9. Keep any page summary page-scoped when table rows enter contextual focus. No infrastructure, workloads, or storage summary cards render today. If one returns, route-backed row selection may add a focused label and shared series emphasis, but the summary must continue to render the page-level series set instead of collapsing down to the selected row or replacing the global trend view with row-local empty states. -10. Keep contextual row focus on the shared primitives. Same-route table drill-ins must reuse `frontend-modern/src/components/shared/contextualFocus.ts` for local scroll preservation, off-screen reveal, inline-detail lookup, and deliberate inline-detail reveal, and must resolve the active series through `frontend-modern/src/components/shared/summaryCardInteraction.ts` behind `frontend-modern/src/components/shared/summaryTableFocus.ts`, instead of rebuilding page-local `Set` filters, focused-label scans, drawer-aware scroll math, or ad hoc scroll restoration in each surface. `frontend-modern/src/components/shared/summaryTableFocus.ts` measures and scrolls rows only for deliberate focus, a focused row's inline detail or a pinned group header, so it must not keep window-level scroll or resize listeners or re-measure the hovered row on hover changes and scroll frames; an off-screen affordance for the active row needs a rendered consumer before that cost returns. +10. Keep contextual row focus on the shared primitives. Same-route table drill-ins must reuse `frontend-modern/src/components/shared/contextualFocus.ts` for local scroll preservation, off-screen reveal, inline-detail lookup, and deliberate inline-detail reveal, and must resolve the active series through `frontend-modern/src/components/shared/summaryCardInteraction.ts` behind `frontend-modern/src/components/shared/summaryTableFocus.ts`, instead of rebuilding page-local `Set` filters, focused-label scans, drawer-aware scroll math, or ad hoc scroll restoration in each surface. `frontend-modern/src/components/shared/summaryTableFocus.ts` measures and scrolls rows only for deliberate focus, a focused row's inline detail or a pinned group header, and reveals a focused row once per focus change: its reveal effect subscribes to the focused series id and table root alone (a remounted table root starts a new reveal for the same focus) and runs the consumer's `revealActiveSeries` callback and the inline-detail scroll untracked, so page state the callback reads (live group lookups, expanded groups) cannot start another reveal, scroll the page back to the row, or reopen a group the operator collapsed. A reveal ends when its row and detail settle in view or its bounded retry window lapses, and filter changes do not re-run the consumer callback, so a row that stays filtered out past that window is not revealed when it returns. It must not keep window-level scroll or resize listeners or re-measure the hovered row on hover changes and scroll frames; an off-screen affordance for the active row needs a rendered consumer before that cost returns. 11. Keep summary-linked table row emphasis on the shared primitive contract. Workloads, infrastructure, and storage rows that mirror the active summary entity must expose that state through `data-summary-row-active` and let the shared presentation in `frontend-modern/src/index.css` render the row emphasis, rather than carrying page-local sky or blue fill classes inside each row renderer. Group-scoped preview and pin must use that same shared presentation boundary: child rows that belong to a hovered or pinned summary group should expose `data-summary-group-member-active="preview|pinned"` so the block-level emphasis stays subtle, consistent, and reversible instead of each table inventing its own outline, badge, or full-strength fill treatment. Static grouped row headers on workloads, infrastructure, storage, recovery, and future grouped tables must use `frontend-modern/src/components/shared/groupedTableRowPresentation.ts` plus the `.grouped-table-row` CSS contract in `frontend-modern/src/index.css`, rather than rebuilding local `bg-surface-alt` variants with subtly different light/dark behavior or page-local left-accent markers. That shared grouped-table primitive owns the subgroup cell padding, typography, small metadata, and badge treatment as well as the row background token, so a future adjustment to the subgroup visual language changes every grouped product table from one owner. Inline table detail rows on platform, workload, and infrastructure tables must compose `frontend-modern/src/components/shared/InlineDetailTableRow.tsx` for the full-width row, surface-alt cell, detail padding, and row-click containment instead of rebuilding page-local `TableRow` / `TableCell` / `div` shells around each drawer. Storage-backed reusable row presenters under `frontend-modern/src/features/storageBackups/` must also keep row height and alert accents on class/data-attribute presentation instead of runtime inline style maps, so the shared table contract stays CSP-safe on both steady-state and alert-highlighted routes. 12. Keep retained-value data loading honest at the ownership boundary. Helpers that prevent a feature surface from falling through the app-level Suspense @@ -3164,33 +3186,34 @@ Agent`), with the plain-language source phrase available through accessible operator action or selected Patrol context, uses distinct monitor labels, and does not become a nested card, generic dashboard strip, trust summary, or duplicate Patrol empty-work list. The - Patrol schedule and model drawer is part of that shared - feature-presentation boundary: it must stay viewport-bounded, expose an - accessible dialog label, keep the four-level control policy on the default - Patrol header, and keep provider model, schedule, trigger tuning, and - readiness validation inside the secondary disclosure. Backend save rejection reasons must pass - through as inline dialog state instead of being replaced with generic toast - copy, and that advanced disclosure must open when the inline state exists. - When the failure includes - Patrol readiness context, the inline state must expose the provider, model, - and readiness summary next to a direct provider-settings action instead of - hiding that diagnosis behind Assistant alone. The provider-model selector in - that popover must stay bound to the shared runtime settings/model catalog - even when the popover mounts after async catalog loading, but the full - catalog must stay behind an explicit change action so the default advanced - drawer leads with the current effective model summary rather than a raw - provider route list. A saved direct-provider Patrol model still renders as - that model instead of visually falling back to the default selection. - Successful provider-model saves that return a not-ready Patrol - readiness snapshot must use that same inline surface with `needs attention` - wording, and any Assistant handoff from that state describes a saved - configuration issue rather than a failed save. When governed fixes are locked, the same Patrol state - owner must clear stale full-mode unlock state before persisting the - monitor-only autonomy payload, so disabled paid controls cannot leak stale - permission into a save. If that inline state opens Assistant, the Patrol - feature must hand off - a source-named, model-only briefing and close the popover so the shared - Assistant drawer is not visually hidden behind feature chrome. When a + Patrol page keeps the four-level control policy on its header and has no + schedule and model drawer: schedule, trigger tuning, and the Patrol model + override live on Settings > Pulse Intelligence > Patrol, and providers and + the shared default model on Settings > Provider & Models. Those saves + report through the shared notification toasts, not inline state: a + rejected save raises an error toast carrying the backend's concrete reason + rather than generic fallback copy, and a saved response whose Patrol + readiness is not ready or degraded raises a warning toast naming the + readiness summary, provider, and model instead of a success toast. On the + Patrol page, an on/off save whose response reports Patrol not ready raises + a fixed warning toast, and a failed Patrol mode save raises an error toast + and restores the previous mode; neither renders inline save-failure state + or offers an Assistant handoff. If a Patrol settings popover or inline + save-failure state returns, it must stay viewport-bounded, expose an + accessible dialog label, bind any model selector to the shared runtime + settings and model catalog, show readiness context (provider, model, + summary) next to a direct provider-settings action rather than behind + Assistant alone, and describe a saved but not ready response as a saved + configuration issue rather than a failed save. An Assistant handoff from it + must send a source-named, model-only briefing and close the popover so the + shared Assistant drawer is not visually hidden behind feature chrome. When + governed fixes are locked, the Patrol header presents `Watch only` as the + effective mode with no enabled paid choice, and the Patrol state owner + refuses any other level when a save starts. Its autonomy payload + never carries `full_mode_unlocked`: `PUT /api/ai/patrol/autonomy` itself + clears stale full-mode unlock and Autopilot activation state on every save + below Autopilot, so disabled paid controls cannot leak stale permission + into a save. When a restored Patrol assessment session is attached, the shared Assistant drawer empty state must stay aligned with that source-named briefing and must not render generic @@ -3379,15 +3402,30 @@ Agent`), with the plain-language source phrase available through accessible default` instead of fusing provider and badge text such as `OpenRouterdefault`. Platform-first top-level pages registered through - `frontend-modern/src/App.tsx` must stay chrome-only and route through the - canonical app shell: each per-platform surface owns navigation and sub-tab - chrome, then embeds the canonical `WorkloadsSurface`, `StorageSurface`, or - `RecoverySurface` in `embedded tableOnly` mode with a forced platform or - source filter. Per-platform features must not - fork their own table primitives, header layouts, or summary cards when a - shared canonical surface already exists; new shared platform-page - primitives live under `frontend-modern/src/features/platformPage/` so the - chrome stays reusable across families. + `frontend-modern/src/App.tsx` route through the canonical app shell: each + per-platform page surface owns its route, `PlatformSectionTabs` sub-tab + navigation, and shared loading, empty, and error states, and its tabs are + built around tables rather than estate overview or attention summary + panels. Three tabs embed a cross-platform canonical surface with the + platform scope locked: the Proxmox and vSphere Overview tabs embed + `WorkloadsSurface` with `forcedPlatform` (the page owns the workloads + state and the one `WorkloadsFilter` toolbar), and the Proxmox Storage tab + embeds `StorageSurface` with `forcedSourceFilter`. Those tabs must keep + their guests, VMs, and storage rows on those embeds rather than a + platform-local copy. Neither surface has an `embedded` or `tableOnly` + mode, and no `RecoverySurface` exists: the aggregate Recovery surface was + deleted on 2026-05-26. Every other tab, including Docker containers, + Kubernetes pods, Machines, `/truenas/storage` + (`TrueNASStorageTopologyTable`), the Proxmox Backups tab, and the TrueNAS + Protection tab, renders a platform-owned table built from the shared + platform-page primitives in + `frontend-modern/src/features/platformPage/sharedPlatformPage.tsx`, even + where a canonical surface could list the same resource types. + Per-platform features must not fork their own table primitives, header + layouts, or summary cards when a shared canonical surface already exists; + new shared platform-page primitives live under + `frontend-modern/src/features/platformPage/` so the chrome stays reusable + across families. Source-specific platform product surfaces under `frontend-modern/src/features/`, such as the Proxmox Backups tab, may own domain IA and row models in their product subsystem while consuming shared @@ -3715,9 +3753,12 @@ production table, router and styles; it does not qualify full-app scrolling. IDs into setup payloads. The shared settings shell should let the backend resolve the effective BYOK model and then render that returned state rather than guessing a model in the modal. - Scoped Assistant handoffs must keep request-local execution overrides in - drawer context. Dashboard and other route-owned entry points may open the - Assistant drawer with source context and `autonomousMode:false`, but they + Scoped Assistant handoffs keep their approval-required disclosure in + drawer context: `autonomousMode:false` on `AIChatContext` is a drawer-only + flag the chat transport never sends, because `/api/ai/chat` sets + approval-required mode server-side for every request. Dashboard and other + route-owned entry points may open the + Assistant drawer with source context and that flag, but they must not infer a user task from an ordinary context-only open. Explicit labelled explanation actions use `aiChatStore.explain` and the shared explanation dispatcher, which captures the request context, waits for open @@ -3735,11 +3776,11 @@ production table, router and styles; it does not qualify full-app scrolling. assessment readout, but shared drawer/chrome primitives stay free of the `FindingsTrustSummary` shape so adding new trust signals goes through the contract first rather than per-shell branching. - Patrol header refresh controls stay on that same feature-owned shell + Patrol refresh controls stay on that same feature-owned shell boundary: `frontend-modern/src/features/patrol/usePatrolIntelligenceState.ts` must make the refresh affordance generation-aware and timeout-bounded, so - a slow supporting intelligence read cannot permanently disable the shared - Patrol header control while Patrol findings and status remain visible. + a slow supporting intelligence read cannot permanently disable the + stale-data Retry control while Patrol findings and status remain visible. That feature-owned presentation helper is the single emitter for investigation-record `impact` and `rollback` fields: when an investigation record exists but those fields @@ -3783,7 +3824,7 @@ production table, router and styles; it does not qualify full-app scrolling. chips in the drawer and must not become another primitive path for raw approval, command, or rollback command payload text. Missing-detail queued-fix recovery actions must still provide the feature-owned Patrol - briefing and request-local approval-required posture rather than opening the + briefing and approval-required disclosure rather than opening the shared drawer as context-free generic Assistant chat. If a feature-owned expired-approval recovery action still has structured action artifact metadata, the shared drawer may receive only safe summary fields and command counts; @@ -3810,16 +3851,16 @@ production table, router and styles; it does not qualify full-app scrolling. feature-owned finding handoff helper rather than hand-written prompt-only drawer opens: safe approval metadata, action artifact summaries, resource references, and bounded `handoff_actions` may enter the prompt and context, - but raw command text stays out and the scoped request must pass - `autonomousMode:false` instead of changing the user's persistent Assistant - control level. Patrol remediation-plan drawer handoffs must use the same + but raw command text stays out and the handoff must set the drawer's + `autonomousMode:false` disclosure instead of changing the user's persistent + Assistant control level. Patrol remediation-plan drawer handoffs must use the same primitive boundary: plan title/status/risk, step labels, and command counts may enter Assistant context; raw command and rollback command payloads must stay in the governed remediation/action panel. All Patrol finding discussion handoffs, including context-only findings without a live approval - or proposed fix, must pass `autonomousMode:false` as a request-local - override so the drawer shows approval-required posture without mutating the - persistent Assistant control setting. + or proposed fix, must set `autonomousMode:false` as that drawer-only flag + so the drawer shows the approval-required posture the server enforces, + without mutating the persistent Assistant control setting. 14. Keep shared filter primitives coherent with source-owned option hydration. Active platform/runtime pages and Settings infrastructure surfaces must keep canonical options visible in shared filter controls even when current @@ -3883,6 +3924,10 @@ production table, router and styles; it does not qualify full-app scrolling. finding-presentation destinations such as the Patrol provider-settings route, while `patrol_autonomy` and legacy Pro activation URLs remain parser aliases only and verified review links use the plain Patrol history anchor. + The Patrol finding-to-Assistant handoff has one owner, + `frontend-modern/src/components/AI/FindingsPanel.tsx`; the Patrol state + hook keeps no second copy of it, and the Patrol workspace passes no + handoff callback because nothing on the page reacts to a handoff. UI surfaces must not duplicate the `patrolControlStarter` query string or write Patrol control or legacy entry-point starter telemetry from local click handlers. @@ -4423,12 +4468,13 @@ labels) bounded — individual subject names and full PolicyRule contents stay outside the rendered surface, mirroring the agent and unified-resource contracts. -Embedded Recovery workspace controls now use the shared filter-toolbar -primitive boundary. Platform pages may choose a default Recovery workspace, -such as TrueNAS opening on protection coverage, but the compact -protection/events selector must use `FilterSegmentedControl` and the -recovery-owned `useRecoverySurfaceState` owner rather than page-local tabs, -nested cards, or independent protection/event state in the embedding surface. +No platform page embeds a Recovery workspace. The aggregate Recovery surface, +its protection/events selector, and `useRecoverySurfaceState` were deleted on +2026-05-26, and TrueNAS opens on its Overview tab. Recovery rows render in +platform-owned tabs whose controls stay on the shared filter-toolbar +primitives: the Proxmox Backups tab filters through the shared `FilterBar`, +and `TrueNASProtectionTable.tsx` through `PlatformTableToolbar`, rather than +page-local tab strips or nested cards. Cross-jump chip strips on alert and Patrol surfaces were retired on 2026-05-16 alongside the platform-first migration. The @@ -4472,10 +4518,13 @@ and `getActiveTabForPath` matcher must not recognize aggregate workspace URLs as owned shell destinations. New palette commands and shortcut chords must flow through the same shell owners; do not reintroduce hidden platform families or retired top-level aggregate routes by reanimating legacy paths. -The shared route-state helpers follow the same boundary: workload, storage, -and recovery helpers in `frontend-modern/src/routing/resourceLinks.ts` may -build query strings for an already-owned platform/runtime route, but must not -export pathname builders for `/workloads`, `/storage`, or `/recovery`. +The shared route-state helpers follow the same boundary: the workload and +storage helpers in `frontend-modern/src/routing/resourceLinks.ts` may build +query strings for an already-owned platform/runtime route, but must not export +pathname builders for `/workloads`, `/storage`, or `/recovery`. The module +exports no recovery query serializer: the Proxmox Backups tab owns its query +through `PROXMOX_BACKUPS_QUERY_PARAMS`, and the TrueNAS Protection tab keeps +its search, status filter, and sort out of the URL. The shared table chrome now allows `TableCardHeader` to expose a right-aligned action slot, currently used by the Workloads/Proxmox metric display control. @@ -4848,6 +4897,13 @@ performance-report transport (`/api/reporting` and `/api/reporting/generate-mult so the reporting panel does not go dead on mixed-version installs, but that compatibility path is intentionally report-only and must not invent the newer catalog-owned VM inventory export surface. +`reportingPanelModel.ts` picks the single-resource transport from the catalog's +`performanceReport.singleResourceMethod`: `POST` sends the request fields as a +JSON body, which is the only transport the server lets narrate a PDF with Pulse +Assistant, and `GET` sends the same fields as query parameters. +`reportingCatalogModel.ts` reads a missing field as `GET`, the only method a +backend that predates the field accepts, and the legacy compatibility catalog +names `GET` for `/api/reporting`. Fleet reports stay `POST` on every backend. `ReportingPanel.tsx` must therefore treat `vmInventoryExport` as optional when it renders a governed reporting catalog. A legacy compatibility catalog with no inventory export still owns a valid enabled reporting surface and must continue @@ -5676,6 +5732,13 @@ identity candidates) and pass them into the shared metric bar primitives (`ResponsiveMetricCell`, `StackedMemoryBar`, `StackedDiskBar`); the static `METRIC_THRESHOLDS` display constants remain fallback-only presentation for callers without alert configuration in scope. +Workload guest bars pass the guest's alert policy tags +(`getWorkloadAlertPolicyTags`) through that same `getMetricThresholds` call, +so a Proxmox guest tagged `pulse-relaxed` colours CPU, memory and disk by the +relaxed floor the engine applies to its configured thresholds; bar +primitives never read tags themselves. The Alerts thresholds table's shared row and mobile card render +the relaxed-trigger line from `getAlertResourceRelaxedTriggerSummary`, not +from per-section copy. The vSphere ESXi host table keeps power state distinct from aggregate resource health without spending a phone column on the repeated normal case. At phone widths the Power column is hidden and a shared `MetadataBadge` appears beside @@ -5767,11 +5830,6 @@ compose that primitive instead of copying rounded-full active selector styling. `ResourcePicker` report-domain filters are part of that boundary: the picker owns the reportable resource categories and labels, but the type selector shell must come from `FilterButtonGroup`. -Chart visibility display actions are registry-backed too. -`ChartVisibilityToggleButton` owns the `Show charts` / `Hide charts` label, -pressed-state, title, icon, and toolbar action styling for summary-bearing -filter surfaces. Pages must compose that primitive instead of recreating local -chart visibility buttons or one-option segmented controls. Column visibility controls are registry-backed too. `ColumnPicker` owns the column chooser trigger, panel title, reset action, empty-state copy, hidden count badge, dropdown width, and outside-click lifecycle through the shared @@ -5954,14 +6012,14 @@ pages that need workspace tabs or count chips next to search should route that through the shared `searchLeading` slot instead of recreating a second local header strip above the control bar. -Pages that filter a list-of-resources surface (Infrastructure, Workloads, -Storage, Recovery Protection coverage, Recovery events) compose the chip-based +Pages that filter a list-of-resources surface (Workloads, Storage, and the +Proxmox Backups tab) compose the chip-based `frontend-modern/src/components/shared/FilterBar/FilterBar.tsx` shell instead of `PageControls`. Each page declares a `FilterDef[]` catalog (label, options, value, defaultValue, group); `FilterBar` renders chips for active filters and exposes the rest behind a "+ Filter" menu, with type-ahead at both the menu and chip popovers (`AddFilterMenu` and `FilterChip`). Low-frequency view options -(grouping segmented control, charts toggle, columns picker, sort key) compose +(grouping segmented control, columns picker, sort key) compose the shared `ViewOptionsDisclosure` through `FilterBar`'s `viewOptions` prop instead of remaining as permanent toolbar controls. `FilterBar` owns the View trigger and inline disclosure; feature consumers pass only panel content and @@ -5971,9 +6029,9 @@ than passing an empty conditional wrapper that leaves a dead View trigger. Contextual frequent actions may use `leadingControls`, while table counters, active trend ranges, and other persistent orientation readouts may use `trailingControls`. Platform tables inherit the same ownership through -`PlatformTableToolbar`. Recovery is event-first and does not use equal -workspace subtabs for protected rollups versus event history; Storage subtabs -(Pools / Physical Disks) sit above the bar as navigation, not filters. +`PlatformTableToolbar`. View switches sit above the bar as navigation, not +filters: the Proxmox Backups tab's `By date` / `Coverage` `PlatformSectionTabs` and +the Storage subtabs (Pools / Physical Disks). `FilterBar` owns committed infrastructure search terms as removable pills, separate from its consumer-owned structured `FilterDef` chips. An exact infrastructure completion or a recognized abbreviated query clears the draft @@ -6213,6 +6271,13 @@ When a shared portal tooltip is already visible, that same owner must reschedule positioning on live coordinate and viewport changes so chart hover tooltips keep following the active pointer instead of sticking to their first anchor. +Only a visible tooltip holds a window resize listener. Hidden instances (one +per metric bar or sparkline on every table row) and portals suppressed on a +touch device register none and re-read the viewport when they open, so the +listener count no longer grows with rows. `TooltipPortal.test.tsx` proves that +hidden and touch-suppressed portals add no listener, that an open one re-clamps +on a window resize, and that one opened after an unheard resize starts from the +live viewport. Floating hover tooltips are a fine-pointer interaction only. The shared tooltip hook, portal, and singleton API must suppress them when the primary device reports no hover capability or a coarse pointer, because touch browsers @@ -6266,7 +6331,7 @@ an absent list as an empty audit history. That shared filter-option primitive is also the canonical owner for default `All ` option wording wherever a product surface exposes filter selects or segmented filter choices. Workloads filters, storage source -filters, recovery history and platform/type filters, Kubernetes namespace +filters, Kubernetes namespace drawers, resource-change timeline filters, and alert configuration options must call `frontend-modern/src/components/shared/filterOptionPresentation.ts` through their nearest presentation/model owner instead of hard-coding page-local `All @@ -6361,18 +6426,13 @@ landing fallback. Shared desktop and mobile navigation must tolerate a missing active tab for those paths while still highlighting canonical active routes such as Proxmox, Docker, Kubernetes, TrueNAS, vSphere, Machines, Alerts, Patrol, and Settings. -The recovery feature shell now also depends on the shared -`frontend-modern/src/components/shared/Subtabs.tsx` primitive for its primary -protected-items versus recovery-events workspace switch. The recovery lane may -own the active view and route-state semantics, but the top-level tab framing -must stay on the canonical shared subtabs control instead of reviving a -recovery-local switcher pattern. When recovery embeds that switcher inside the -page shell, it should follow the same ordering already used by storage: shared -subtabs row first, shared controls card second, and data card after that. The -contained styling should come from the same canonical subtabs shell, list, and -button class treatment already used by established Pulse surfaces rather than -from a recovery-only variant boundary, adjacent chip row, or recovery-local -filter-row embedding. +Recovery has no workspace switch of its own: the protected-items versus +recovery-events `Subtabs` switch left with the aggregate Recovery page on +2026-05-26. The Proxmox Backups tab switches between `/proxmox/backups/date` +and `/proxmox/backups/coverage` through the shared `PlatformSectionTabs` +row above its `FilterBar`, and the TrueNAS Protection tab is a single table. +A future recovery view switch belongs on those platform-page primitives, not +on a recovery-local switcher, chip row, or filter-row embedding. The shared table primitives now also need to preserve caller-owned separator styling. `TableHeader` and `TableBody` may provide canonical default borders and dividers, but when a caller supplies explicit border or divide classes the @@ -6393,63 +6453,38 @@ That same shared table boundary now owns CSP-safe fill rendering for metric bars: `frontend-modern/src/components/shared/ProgressBar.tsx` must render fill width through DOM attributes rather than inline width styles that break the public demo CSP. -That same recovery shell boundary now also owns one canonical top-level filter -controller in -`frontend-modern/src/features/recovery/useRecoverySurfaceState.ts`. Route-backed -recovery filters such as the provider-neutral `itemType` selector must be -derived, normalized, and fanned out to inventory, history, activity, facets, -and series consumers from that shared state owner rather than being recreated -as page-local toolbar state inside individual recovery sections. -That same shared recovery filter boundary also owns canonical recovery -item-type derivation through -`frontend-modern/src/utils/recoveryItemTypePresentation.ts`. Recovery shell -state, tables, summaries, and point-detail surfaces must resolve rollup and -point item types through the shared presenter helpers instead of repeating -`display.itemType` / `subjectType` / `subjectRef.type` fallback chains in -page-local consumers. -That same shared recovery decode boundary also owns canonical recovery display -shape. `frontend-modern/src/utils/recoveryPlatformModel.ts`, -`frontend-modern/src/hooks/useRecoveryPoints.ts`, and -`frontend-modern/src/hooks/useRecoveryRollups.ts` must normalize legacy -transport display aliases like `subjectLabel` and `subjectType` into canonical -runtime `itemLabel` and `itemType` fields before recovery presenters consume -the model. -The same shared recovery-column boundary must keep legacy `subject` and -`source` column ids at migration-only scope once -`frontend-modern/src/hooks/useColumnVisibility.ts` owns alias rewrites. -Recovery table runtime helpers and render switches should operate on canonical -`item` and `platform` ids rather than carrying the deleted ids as live cases. -That same shared recovery state owner now also keeps `platform` as the -canonical route and transport filter name for operator-facing recovery links, -while any accepted legacy `provider` aliases remain parser compatibility only. -Caller-facing shared recovery route builders must therefore stay -platform-first as well: compatibility `provider` aliases may be accepted while -parsing legacy links, but they should not remain a first-class input on new -recovery link construction helpers. -Recovery frontend decode and derived option builders must treat payload -`platform` / `platforms` as the canonical response fields and only fall back -to legacy `provider` / `providers` aliases for compatibility, so route, -filter, and table state do not keep backend-era vocabulary alive as the -default client model. -That normalization belongs at the shared recovery transport boundary in -`frontend-modern/src/hooks/useRecoveryPoints.ts` and -`frontend-modern/src/hooks/useRecoveryRollups.ts`, not in individual tables, -drawers, or summary cards. Recovery components should receive canonical -platform-first runtime models rather than re-deriving legacy alias fallback -locally. -Recovery section owners under `frontend-modern/src/components/Recovery/` must -consume that shared `platform` filter surface directly. They must not keep -recovery-local `provider` route/query vocabulary alive behind renamed labels, -or the UI will drift back to backend-shaped navigation even when the copy says -`Platform`. -That same shared recovery filter owner must also preserve route-owned platform -visibility while transport-backed options are still hydrating. If -`frontend-modern/src/features/recovery/useRecoverySurfaceState.ts` restores a -canonical `platform` selection such as `truenas` from the route before the -rollups, points, or facets payloads arrive, it must keep that selected -platform present in the option set so the shared `LabeledFilterSelect` shows -the owned value immediately instead of flashing back to `All Platforms` until -recovery data warms. +Recovery has no shell, filter controller, route serializer, or column model +left for frontend primitives to govern. The aggregate Recovery page, its +section owners under `frontend-modern/src/components/Recovery/`, and its state +owner `frontend-modern/src/features/recovery/useRecoverySurfaceState.ts` were +deleted on 2026-05-26 together with the rollup, facet, and series hooks +(`useRecoveryRollups.ts`, `useRecoveryPointsFacets.ts`, +`useRecoveryPointsSeries.ts`), and the recovery query serializer and +`recoveryItemTypePresentation.ts` that only it read have been deleted since. +Recovery rows render in two platform-owned tabs built from the shared +platform-page primitives: the Proxmox Backups tab and the TrueNAS Protection +tab. +The Proxmox Backups tab parses `/api/backups/pve` and `/api/backups/pbs` +inside `ProxmoxBackupsTable.tsx` and reads posture through +`frontend-modern/src/hooks/useProtectionPostures.ts`. The TrueNAS Protection +tab reads recovery points, and what binds that path is the decode boundary. +`frontend-modern/src/utils/recoveryPlatformModel.ts` maps legacy transport +aliases onto the canonical runtime model: `provider` to `platform`, +`subjectResourceId` to `itemResourceId`, `subjectRef` to `itemRef`, and +`display.subjectLabel` / `display.subjectType` to `display.itemLabel` / +`display.itemType`. `frontend-modern/src/hooks/useRecoveryPoints.ts`, the +only recovery-points reader, applies `normalizeRecoveryPointsResponse` to +every `/api/recovery/points` response, so the TrueNAS Protection tab receives +platform-first, item-first points. The `display.subjectLabel`, +`display.subjectType`, and `subjectRef` fallbacks that +`TrueNASProtectionTable.tsx` and `truenasPageModel.ts` still read are dead on +that path, because normalized points never carry those fields; new recovery +code reads the canonical fields only. New recovery filters, links, or view +switches belong to the platform tab that owns the rows, on the platform-page primitives, rather than in a revived +cross-platform recovery shell or a shared recovery serializer in +`frontend-modern/src/routing/resourceLinks.ts`. The rollup normalizer +(`normalizeRecoveryRollupsResponse`) has had no production reader since +`useRecoveryRollups.ts` was deleted. The retired dashboard action queue must not be reintroduced as a compact Patrol or infrastructure issue panel. Patrol-owned runtime findings remain governed by `frontend-modern/src/utils/aiFindingPresentation.ts` and their @@ -6664,6 +6699,16 @@ notice renders on. `collectOutdatedSensorSetupNodes` lists the PVE nodes that carry the registry's `proxmox.sensorSetupOutdated` verdict and takes no disk rows, because only the Proxmox Storage tab loads them; a predicate over them left the notice silent on Overview. +A notice names each affected resource by the label its page's table shows for +the row. The sensor-setup notice leads with the node's trimmed resource name +(the operator-set display name when one exists, else the Proxmox node name), +falling back to the id as the nodes table does, and adds the Proxmox node name +in parentheses unless the label already contains it as a whole word, ignoring +ASCII case, or as the host part of a domain name, as in +`West Production C (pve3)`. Both names are needed: the nodes table switches to +the Proxmox node name in its narrowest layout, and the setup script is re-run +on that Proxmox node, so the rule is stricter than `hasAlternateDisplayName` in +the guest table's node group header and keeps `pve-3` beside a `PVE3` label. Alert incident-event filter containers, labels, and chips must now route through the shared presentation helpers in @@ -6819,11 +6864,12 @@ That same Patrol shell ownership includes refresh affordance state: `frontend-modern/src/features/patrol/usePatrolIntelligenceState.ts` must keep operator refresh controls generation-aware, timeout-bounded, and separate from background polling state, so a slow supporting intelligence read cannot make the -shared Patrol header Refresh Patrol action spin indefinitely or stay disabled +Patrol stale-data Retry action spin indefinitely or stay disabled while Patrol findings and status remain visible. Neither the Patrol load and poll path (the store's `loadDashboardData` bundle) nor the Retry path's background supporting reads fetch the global -learned-correlation list, which no Patrol surface shows. +learned-correlation list or the provider circuit-breaker status, neither of +which any Patrol surface shows. That same Patrol shell should make scoped trigger policy legible without another navigation step. `frontend-modern/src/features/patrol/PatrolIntelligenceHeader.tsx` should keep actionable scoped-trigger state legible without promoting @@ -7417,20 +7463,19 @@ customer-facing action copy, while provider labels, empty states, and action/status messaging. Future settings copy changes in those areas should extend these helpers instead of inlining panel-local strings inside the shell or reactive state owners. -Shared infrastructure action-link framing now also owns recovery entry wording -for service resources. `frontend-modern/src/components/Infrastructure/serviceDetailLinks.ts` -must keep platform-service recovery links on canonical recovery-events -framing and route state, so upstream service surfaces do not drift back to -PBS-backup wording or inherit the page-default inventory workspace when they -are actually deep-linking into recovery activity. -That same shared primitive boundary also owns resource handoff chip framing for -cross-surface investigation UI. Alerts, Patrol, and similar feature shells may -choose which governed surfaces to show, but they must build those links through -the shared resolved-resource route helpers in -`frontend-modern/src/routing/resourceLinks.ts` instead of freezing raw route -strings, local link dedupe, or provider-specific link chips inside feature -panels. Shared chip styling belongs in the feature shell; canonical href and -label truth belongs in the shared route helper. +Shared infrastructure action-link framing owns service-resource action links. +`frontend-modern/src/components/Infrastructure/serviceDetailLinks.ts` returns +only the PMG thresholds link today: the PBS recovery-events link it used to +build was retired with the platform-first route migration, and PBS backups are +reached through the Proxmox Backups tab. A future service recovery link must +target the platform tab that owns those rows rather than a recovery-events +workspace. +Alerts, Patrol, and similar feature shells render no resource handoff chips +since the cross-jump strips were retired on 2026-05-16 (see the retired-helper +note above). A future cross-surface link must take its href and label from a +route helper in `frontend-modern/src/routing/resourceLinks.ts` instead of +freezing raw route strings, local link dedupe, or provider-specific link chips +inside a feature panel; chip styling belongs in the feature shell. That same shared primitive boundary now also owns persisted column-identity migration for governed surfaces. When a v6 surface canonicalizes saved column IDs, `frontend-modern/src/hooks/useColumnVisibility.ts` must accept explicit diff --git a/docs/release-control/v6/internal/subsystems/monitoring.md b/docs/release-control/v6/internal/subsystems/monitoring.md index 07179d8542..9f57b18290 100644 --- a/docs/release-control/v6/internal/subsystems/monitoring.md +++ b/docs/release-control/v6/internal/subsystems/monitoring.md @@ -2699,7 +2699,18 @@ truthfulness, not native thaw, containing-release or workload acceptance. ingestion so canonical host IDs stay stable across restarts (see the unified-resources contract's durable identity-pin obligation). Rebuild paths added to the adapter must keep that persistence step; ephemeral - snapshot-bridge adapters stay read-only. + snapshot-bridge adapters stay read-only. The rebuild, the live + supplemental refresh and the read-state overlay ingest records with the + adapter's configured stale thresholds, because record ingest joins + operator links and the freshness gate of every metric merge, a link's or + a source's into an existing row, reads them (unified-resources contract, + "Operator links reach record-ingested resources"). Regression + coverage: + `TestMonitorAdapterJoinsLinkedRecordsWithConfiguredStaleThresholds` and + `TestMonitorAdapterSourceMergesUseConfiguredStaleThresholds` in + `internal/unifiedresources/monitor_adapter_read_state_test.go` and + `TestManualLinkToSupplementalGuestHoldsWithAndWithoutContinuity` in + `internal/monitoring/issue1913_host_continuity_test.go`. 11. The TrueNAS provider projects pools with `Storage.Topology` fixed to `pool` and the ZFS data vdev layout in `Storage.VDevLayout`. The @@ -5374,6 +5385,42 @@ canonical aliases retain their precedence and ambiguous aliases remain unresolved. This avoids cloning a full resource for every policy lookup while keeping alert intent tied to the same identity rules as resource reads. +Alert intent resolves references and ancestors through the read state Patrol +resolves through and the resources API seeds from: the published registry with +saved hosts that have not reported since a restart overlaid +(`operatorIntentIdentity`). A saved host exists only there, so against the +published registry alone its alerts (`agent:`) read their literal +reference after a restart and missed intent set on the host, and, when an +operator link joins it to a live resource, intent set on the link primary that +the registry now resolves the saved host to. Alerts resolve under the alert +manager's lock, once per metric check, so the overlay is kept for the published +registry generation it was built from. Publication builds it before alert +evaluation, outside that lock; a lookup that races ahead of publication, or the +first one after leaving mock mode, builds it under the lock. A generation is +the registry the adapter publishes and when it last changed +(`MonitorAdapter.Generation`), since two publications can share a +`LastRebuiltAt`. A finished build is stored only if that generation is still +the published one and the mock-mode epoch it started in is current, checked and +stored inside the mock-mode fence, so a build that overlapped a mode switch is +discarded. A lookup made while mock mode is on drops the stored overlay; a +switch to mock and back with no lookup in between keeps it, since it still +describes the published generation. A saved-host change that arrives without a +new generation shows on the next one. Mock mode carries no saved hosts: lookups +then resolve against the published registry as before. +`monitor_alert_intent_test.go` +(`TestOperatorIntentIdentityKeepsOneOverlayPerGeneration`, +`TestOperatorIntentReachesSavedHostAlertsAfterRestart`) pins the reuse, a +generation sharing or preceding the last one's timestamp, the mock switch, a +rebuild made from inside an admitted fence call, and an unlinked saved host's +intent reaching its offline alert. +`internal/monitoring/issue1913_host_continuity_test.go` +(`TestManualLinkToSupplementalGuestHoldsWithAndWithoutContinuity`) pins a saved +agent's offline alert reading maintenance set on the guest it is linked into, +and `monitor_host_agent_removal_lifecycle_test.go` +(`TestHostAgentRemovalLifecycleHonorsSavedHostIntentAfterRestart`) pins the +restarted monitor's own host-offline check raising no alert for a saved host +the operator marked intentionally offline. + Alert restore precedes resource-store attachment during startup. Once the adapter attaches the persisted operator-policy resolver, monitoring asks Alerts to reconcile the restored set and refreshes shared alert state if it changed. @@ -5814,3 +5861,24 @@ that actually accepted the same occurrence. The connected HTTP 200/503 split, reopens the persistent queue, and requires the old accepted destination's recovery plus the new firing, with no recovery to the unannounced destination. Quiet hours and disabled recovery controls still apply. + +### Broadcast host coalescing honours operator splits + +The websocket broadcast coalesced top-level host views with +`CoalescePresentationHostResources`, which applies no merge exclusions, while +the resources API's `ResourceRegistry.ListForPresentation` applies the +registry's. A host pair the operator split (unlink or report-merge) could +therefore be one broadcast row and two REST rows. +`Monitor.buildBroadcastFrontendStateFromSnapshotWithClock` now calls +`coalesceResourcesForPresentation`, which asks the view's read state for +`MonitorAdapter.CoalesceForPresentation` when its registry is store-backed +(the resource store's adapter, or a host-continuity overlay that loaded the +store's exclusions afresh), and otherwise the resource store's adapter: the +mock-mode view lists through a store-less registry that carries no operator +decisions. Both surfaces share `presentationExclusionFilter`. Polling, rebuild cadence, +alert evaluation and the broadcast payload shape are unchanged. +`TestBroadcastPresentationCoalesceHonoursMergeExclusions` compares broadcast +and REST row counts with and without the split, and +`TestBroadcastPresentationCoalescePrefersTheListingReadState` pins the +listing read state over an older store generation and the store's exclusions +for a mock-style view. diff --git a/docs/release-control/v6/internal/subsystems/patrol-intelligence.md b/docs/release-control/v6/internal/subsystems/patrol-intelligence.md index e5ad3e0b55..ee881010cb 100644 --- a/docs/release-control/v6/internal/subsystems/patrol-intelligence.md +++ b/docs/release-control/v6/internal/subsystems/patrol-intelligence.md @@ -329,13 +329,14 @@ or historical proof/counting for resolved-only work. shared parent node, or explicit correlated finding IDs. Grouping must not infer causality from title or description prose, and the queue's affected resource count must continue to count canonical resources rather than - visual groups. Because a - contextual Assistant handoff from that workflow is still a first-party Patrol - starter for the same governed journey, it must record content-free workflow - prompt activity through the shared marker route with the `pulse_patrol` - surface before the drawer opens; it must not include finding IDs, prompt - text, resource context, model output, or direct-action payloads in that - marker. + visual groups. A contextual Assistant handoff from that workflow opens + through `frontend-modern/src/components/AI/FindingsPanel.tsx` and records no + workflow starter today, so the first-party `pulse_patrol` marker has no + browser producer. If that handoff starts recording one, it must write + content-free workflow prompt activity through the shared marker route with + the `pulse_patrol` surface before the drawer opens, with no finding IDs, + prompt text, resource context, model output, or direct-action payloads in + that marker. A compact Patrol work-group row may appear above the list only when it adds cross-source current-work grouping that the row title alone cannot express: pending approvals, failed governed actions, a failed/latest Patrol check, @@ -471,8 +472,11 @@ or historical proof/counting for resolved-only work. compact progress label must describe current work state (`ready`, `needs attention`, `approval needed`, `outcome verified`, `no active work`) instead of repeating the selected mode; the Patrol mode selector/header - owns current-mode copy, and local Patrol state must expose work evidence as + owns current-mode copy, and any local Patrol work evidence must be Patrol-owned issue/work counts rather than legacy proof counts. + No Patrol surface reads issue-evidence, governed-action decision, or + verified-outcome counts today, so `usePatrolIntelligenceState.ts` computes + none of them. When no active finding or pending approval remains, terminal verified or rejected outcomes must read as history behind a `no active work` current state; current-state copy must not ask operators to reconcile old approved @@ -771,12 +775,12 @@ attention`, `approval needed`, `outcome verified`, `no active work`) instead only block the selected run view while the run references finding ids that still need the direct Patrol findings payload. Patrol page refresh state must also separate background data loads from the - operator-clicked Update status action: slow or stalled supporting - intelligence reads may continue in the background, but they must not make the - shared header action spin or stay disabled once Patrol findings and status - remain visible. The header action is a status/history sync affordance; it - must not read like another Patrol run or imply that it changes - infrastructure. + operator-clicked Retry on the stale-data banner: background polls hold no + busy state, and slow or stalled supporting intelligence reads may continue + in the background, but they must not make Retry spin or stay disabled once + Patrol findings and status remain visible. Retry is a status/history sync + affordance; it must not read like another Patrol run or imply that it + changes infrastructure. Patrol trigger-status copy in the default header and activity strip must stay actionable. Runtime policy pauses for alert/anomaly-triggered background checks, such as the local development safety guard, may explain a blocked @@ -986,7 +990,7 @@ clear`, `Found N new issues`, `Fixed N issues`, `N issues still open`, or answer. Finding handoffs must be assembled through the Patrol-owned handoff model so the prompt, visible briefing, model-only finding context, resource reference, - bounded action reference, and request-local approval-required posture stay in + bounded action reference, and drawer approval-required disclosure stay in sync. The model-only context may include current finding status, recurrence, investigation record facts, evidence, verification, approval state, dry-run posture, existing action artifact @@ -997,8 +1001,8 @@ clear`, `Found N new issues`, `Fixed N issues`, `N issues still open`, or prompt-only local shortcut: pass approval ID/status/risk/target plus safe summary/count metadata as review context, attach the target resource reference, include bounded `handoff_actions` for live approvals or structured - action artifacts when present, force the request-local approval-required mode, - attach the Patrol-owned visible drawer briefing for the pending approval or + action artifacts when present, mark the drawer's approval-required + disclosure, attach the Patrol-owned visible drawer briefing for the pending approval or queued-fix recovery state, and never paste the approval command or action command text into the chat prompt. Existing remediation-plan or action-plan artifacts follow the same boundary: plan status, risk, and @@ -1006,11 +1010,12 @@ clear`, `Found N new issues`, `Fixed N issues`, `N issues still open`, or the LLM to critique, but visible handoffs must not render Patrol step lists or suggested prompt chips, and command or rollback command text stays in the governed remediation or approval surface. Generic finding - discussion handoffs must also force request-local approval-required mode for - any non-empty Patrol `finding_id`, including context-only findings and - findings that reference a live approval, action artifact, fix outcome, or - remediation plan, so default autonomous Assistant settings cannot bypass the - Patrol action-governance boundary. The assembled handoff must still pass + discussion handoffs for any non-empty Patrol `finding_id`, including + context-only findings and findings that reference a live approval, action + artifact, fix outcome, or remediation plan, run in the approval-required + mode `/api/ai/chat` sets server-side for every request; the browser sends no + execution-mode field, so default autonomous Assistant settings cannot bypass + the Patrol action-governance boundary. The assembled handoff must still pass through the Assistant runtime's resource-policy sanitizer before prompt injection, so Patrol-owned prose cannot leak governed resource names, IDs, aliases, nodes, paths, or @@ -1190,8 +1195,9 @@ fix`, or `Explain` based on current finding state), while secondary the model-only run context: it resolves the run ID from Patrol history, rebuilds bounded run facts, scoped resource references, sanitized analysis, and classified failure detail server-side, and rehydrates the same context - from stored metadata on follow-up turns. It must force request-local - approval-required mode, present a source-named visible drawer briefing, and + from stored metadata on follow-up turns. The exchange runs in the + server-set approval-required mode; the handoff must present a source-named + visible drawer briefing and frame Assistant as explanation and next-step review rather than execution or automatic retry authority. 9. Keep the normal Patrol status summary plain and operator-first rather @@ -1581,17 +1587,18 @@ suppress stale healthy summary headlines such as `Health A · 100/100` even if the last summary payload still looks healthy. Legacy `/ai` entry points must redirect into that same Patrol-owned shell rather than preserving a second canonical route. -That same browser proof now covers the Patrol control and advanced-settings -split. The advanced Patrol settings drawer must stay within the desktop -viewport, avoid duplicating the inline Patrol control policy, expose -provider/model, schedule, trigger, and user-level model checks directly, and surface -the backend's concrete license/validation reason when a settings change is -rejected instead of replacing it with a generic `Failed to save advanced -settings` toast. That inline failure no longer -opens Assistant. If it does again, the handoff is model-only explanation -context: raw command, script, credential, and provider-detail payloads stay -redacted, Assistant opens with `autonomousMode:false`, and the Patrol control -panel closes so the operator is not left behind an overlapping popover. +That same browser proof now covers where Patrol settings are saved. The +Patrol page has no settings drawer or popover: Patrol schedule, triggers, and +the Patrol model override live on Settings > Pulse Intelligence > Patrol +(`Save Patrol settings`), and providers and the shared default model on +Settings > Provider & Models. A rejected save there raises an error +notification carrying the backend's concrete license/validation reason +instead of generic fallback copy, and it offers no Assistant handoff. If a +Patrol settings popover or an Assistant handoff from a settings failure +returns, the handoff is model-only explanation context: raw command, script, +credential, and provider-detail payloads stay redacted, Assistant opens with +`autonomousMode:false`, and the popover closes so the operator is not left +behind it. Patrol-control save-failure sessions saved by earlier builds keep the compatible `handoff_metadata.kind=patrol_configuration_failure` plus only the runtime-failure boolean needed for drawer/session presentation, so they restore @@ -1626,27 +1633,32 @@ after a page render: recoverable Patrol provider/model settings saves must persist and echo structured readiness cause metadata, manual run requests must return the structured readiness reason if a stale UI still submits, and scheduled or scoped alert/anomaly runs must skip before calling the model while -preserving the blocked reason and cause in Patrol status. The Patrol -control state owner must also clamp stale investigation/remediation -autonomy back to findings-only `monitor` and clear stale full-mode unlock state -before persisting Patrol control when the safe-remediation entitlement is not -effective, so an expired or downgraded plan cannot turn a recoverable control -review into a Pro-only save failure. Patrol renders no inline error +preserving the blocked reason and cause in Patrol status. When the +safe-remediation entitlement is not effective, the Patrol header must present +findings-only `monitor` as the effective mode and the Patrol control state +owner must refuse any investigation/remediation level when a save starts; +the `monitor` save it does send is accepted without the entitlement, and the +API clears stale full-mode unlock and Autopilot activation state on it, so an +expired or downgraded plan cannot turn a recoverable control review into a +Pro-only save failure. Patrol renders no inline error surface for these failures today: switching Patrol on or off while it is not ready raises a fixed warning notification, a failed Patrol mode save raises an error notification, and neither offers an Assistant handoff. A provider or model save in Settings > Provider & Models reports the readiness summary, provider and model in its own save message. -The Patrol mode selector in that header and configuration dialog must -compose the shared `frontend-modern/src/components/shared/FilterButtonGroup.tsx` -instead of rebuilding a local active-button group. The wide default Patrol -header uses the segmented layout; the constrained configuration dialog may use -the shared prominent layout so all four mode labels remain readable without -inventing a Patrol-local selector. Patrol owns the default visible four-level +The Patrol mode selector in that header must compose the shared +`frontend-modern/src/components/shared/FilterButtonGroup.tsx` instead of +rebuilding a local active-button group. When the header shows Patrol mode +options, it renders the selector in the segmented layout inside its `Mode and +automation` disclosure; a constrained surface that hosts it may use the shared +prominent layout so all four mode labels remain readable without inventing a +Patrol-local selector. Patrol owns the default visible four-level policy presentation (`Watch only`, `Ask first`, `Safe auto-fix`, `Autopilot`), entitlement locks, and the rule that choosing the highest -Autopilot level sends `full_mode_unlocked:true` while choosing any lower level clears that -acknowledgement. The shared primitive owns pressed-state semantics, +Autopilot level opens the acknowledgement dialog and saves `full` only with a +recorded `acknowledgement_id` (the API derives the compatibility +`full_mode_unlocked` value itself), while saving any lower level lets the API +clear the Autopilot activation. The shared primitive owns pressed-state semantics, disabled-option behavior, and active/inactive selector styling. That same Patrol-owned presentation rule also applies to the findings empty state: `frontend-modern/src/components/AI/FindingsPanel.tsx` must not treat @@ -2189,10 +2201,13 @@ requester identity as safe metadata, without copying the approval command payload into Assistant. That same store now owns the Patrol dashboard load bundle as well, so the page refresh path stays aligned on a single orchestrated AI bundle instead of -repeating the individual summary, findings, circuit-breaker, and approval -fetches inline. Neither that bundle nor the Patrol refresh path fetches the -global learned-correlation list: the Patrol page renders no correlations, so -the store holds no copy of that list. +repeating the individual summary, findings, and approval fetches inline. +Neither that bundle nor the Patrol refresh path fetches the global +learned-correlation list: the Patrol page renders no correlations, so +the store holds no copy of that list. Nor do they fetch the provider +circuit-breaker status: an open breaker already reaches the page as the +`circuit_open` Patrol runtime block on `/api/ai/patrol/status`, so the store +holds no separate breaker state. The shared `frontend-modern/src/components/Infrastructure/ResourceChangeSummary.tsx` and `frontend-modern/src/components/Infrastructure/ResourceCorrelationSummary.tsx` diff --git a/docs/release-control/v6/internal/subsystems/performance-and-scalability.md b/docs/release-control/v6/internal/subsystems/performance-and-scalability.md index a2530e6a32..b772ef7eaa 100644 --- a/docs/release-control/v6/internal/subsystems/performance-and-scalability.md +++ b/docs/release-control/v6/internal/subsystems/performance-and-scalability.md @@ -1092,11 +1092,13 @@ change may globally weaken the Task 03 lifecycle-state idempotency invariant. mounted, report errors out of band, and must not rely on Solid `createResource` in a way that can bubble into the route-level Suspense fallback or the setup/welcome shell. - Workload row drawer expansion, local row focus, and summary group pinning - are local interaction state. Inbound Workloads deep links may hydrate - `resource` and `summaryGroup` into the focused row or group, but opening, - closing, or clearing an inline drawer must not write those params back to - route state, schedule router navigation, or trip the app-shell fallback. + Workload row drawer expansion and local row focus are local interaction + state. Inbound Workloads deep links may hydrate `resource` into the focused + row, but opening, closing, or clearing an inline drawer must not write that + param back to route state, schedule router navigation, or trip the + app-shell fallback. Workloads has no group focus, so it reads no + `summaryGroup` param: row expansion stopped writing it and no link builder + passes one, so its hydration was removed with the group pin. URL synchronization for Workloads stays limited to filter-owned scope in `frontend-modern/src/components/Workloads/useWorkloadUrlSync.ts`. The Workloads page may show a bounded partial-inventory banner when a @@ -1125,6 +1127,10 @@ change may globally weaken the Task 03 lifecycle-state idempotency invariant. 8. Render workload row identity directly from the shared canonical workload helper so row selection, hover, and fallback metadata lookup stay aligned with the same workload contract 9. Format infrastructure sensor labels through the shared `frontend-modern/src/utils/textPresentation.ts` presentation helper instead of maintaining a local title-casing implementation in `frontend-modern/src/components/Infrastructure/resourceDetailMappers.ts` 10. Extend workload row contract and per-row hot-path derivations through `frontend-modern/src/components/Workloads/guestRowModel.tsx` and `frontend-modern/src/components/Workloads/useGuestRowState.ts`, and extend tooltip-backed row cell presentation through `frontend-modern/src/components/Workloads/GuestRowCells.tsx`, rather than rebuilding column metadata, row identity, cell tooltips, or anomaly correlation inside `frontend-modern/src/components/Workloads/GuestRow.tsx` + Guest metric thresholds read the guest's alert policy tags through one + `getWorkloadAlertPolicyTags` memo per row and per drawer, beside the + existing scope and override-candidate memos, so `pulse-relaxed` bar + colouring adds no per-render tag parsing and no extra store reads. On coarse-pointer touch layouts, those cell details must not spawn floating hover tooltips from synthesized mouse events; a row tap remains the primary drawer action. Tooltip suppression belongs to the shared hover-capability @@ -1473,13 +1479,16 @@ change may globally weaken the Task 03 lifecycle-state idempotency invariant. `TableCardHeader` inside both the populated and filtered-empty table card. A platform page must not place a parallel floating title above its filters; the title/count band belongs to the same framed table surface as the rows. - `WorkloadPanel` owns the mutually exclusive host/guest drawer handoff: - clicking a grouped host row while a guest drawer is open must clear the - selected guest and keep/focus the host summary group so the node drawer can - replace the guest drawer. Platform pages that render a dedicated host - table above an embedded Workloads table may disable the grouped host drawer - through the explicit Workloads surface option, but standalone Workloads - must keep the inline grouped-row drawer as the default. + `WorkloadPanel` renders only the guest drawer. Grouped host rows are + static dividers: no click handler, hover preview, group pin, or inline + node drawer. They keep `data-summary-group-id` so a click on a divider + does not clear an open guest drawer. Host details open from the platform + page's own hosts table (`ProxmoxNodesTable` mounts `NodeDrawer`; + `VsphereHostsTable` owns its detail row). The former + `groupNodeDrawerMode` option defaulted to an inline grouped-row drawer + that only the retired standalone Workloads page used; both production + mounts passed `disabled`, so the option, the inline `NodeDrawer` mount + and the Workloads group hover/pin scope were removed. Compact icon headers inside `WorkloadTableHeader.tsx` may stay visually dense for responsive workload tables, but the icon must be decorative and the column label must remain present through an `sr-only` label so the @@ -1691,18 +1700,26 @@ shell clickable behind another overlay. scoped table metric-history interaction that requested them. Platform landing should stay route-module warm and data-light until the selected platform page owns its normal resource/table query. - Platform pages that embed `WorkloadsSurface` reuse the canonical - workloads filter toolbar through the `showFilterToolbar` + - `suppressPlatformFilter` props in `WorkloadsSurfaceProps`. The page - keeps `tableOnly` to hide the dashboard cards and summary strip but - opts in to the same shared `FilterBar`, `GroupedTableModeSegmentedControl`, - `ColumnPicker`, status/type chips, and search-history primitives that - the global Workloads page renders, so platform operators get - dense-table search, sort, grouping, view, status, and column controls - on every embedded workloads tab without spawning a forked toolbar. - The platform scope flows through `forcedPlatform` as a typed page - input; `suppressPlatformFilter` drops the now-redundant Platform chip - from the rendered toolbar so the user never sees a removable lock. + Two platform pages embed `WorkloadsSurface`: the Proxmox and vSphere + Overview tabs (`frontend-modern/src/features/proxmox/ProxmoxPageSurface.tsx` + and `frontend-modern/src/features/vmware/VmwarePageSurface.tsx`). Each page + builds the workloads state itself with `useWorkloadsState`, renders the + one shared `WorkloadsFilter` toolbar (search, status filter, a type + filter that vSphere suppresses, and the View options for grouping, metric display, and columns on the + shared `FilterBar`) from that state, and hands the same state to `WorkloadsSurface` through its + `state` prop with `suppressFilterToolbar`, so the surface skips its own + filter row and the page never stacks two toolbars wired to one state. + Proxmox also passes that state's search, plus its page-owned metric mode + and history range, to the nodes table above the guests, so one toolbar + drives both tables. `WorkloadsSurface` has no `tableOnly` mode, no + `showFilterToolbar` prop, and no dashboard cards or summary strip to + hide: it renders its filter row (unless suppressed or in kiosk mode), the + workloads table, or an empty-state table card. The platform scope flows + through `forcedPlatform` as a typed page input to `useWorkloadsState`, + the toolbar, and the surface. `suppressPlatformFilter: true` makes the + state expose no Platform filter config and both pages pass + `platformFilter={undefined}` to the toolbar, so the user never sees the + locked platform as a removable chip. ### Navigation resolves from a bounded request @@ -1788,6 +1805,22 @@ final measurements and host-load limitation remain recorded in `records/resource-payload-static-metadata-2026-08-24.md`. They do not establish a controlled performance improvement or satisfy the open SLO qualification. +### Window resize listeners do not grow with rows + +Table rows must not each register a window `resize` listener. +`frontend-modern/src/hooks/useBreakpoint.ts` keeps one module-level width +signal, fed by a single rAF-debounced listener while at least one caller is +mounted, and a new caller starts from the live `innerWidth`. Shared tooltip +state in `frontend-modern/src/components/shared/useTooltipState.ts` listens +only while its tooltip is visible. Measured on 2026-10-07 on a mock-mode +`/proxmox` page with 140 workload rows at 1440 px, the active listener count +fell from 859 (699 tooltip instances, 156 breakpoint callers) to 5; at 390 px +with 36 rows it fell from 203 to 5. An open tooltip adds one listener and +removes it on close, and crossing the phone breakpoint in either direction +still switches the workload table and row layout. `useBreakpoint.test.ts` +proves one listener for 50 callers, that back-to-back resizes leave one pending +frame, and that every caller crosses a breakpoint when that frame runs. + ### Large API responses negotiate gzip without corrupting edge cases The main listener compresses eligible JSON and frontend text assets for @@ -2003,7 +2036,10 @@ because both remaining production mounts (Proxmox and vSphere overview) set it unconditionally, so the non-compact path no longer rendered anywhere. A new embedded consumer that needs host stats beside its workloads must render its own hosts table rather than -re-growing metric cells in the shared group row. +re-growing metric cells in the shared group row. The same applies to host +interaction: the group row has no drawer, hover preview, or pin, and the +Workloads state no longer builds per-group summary scopes for every +filtered-guest change. WorkloadsSurface stays monitoring-first. It must not render a persistent aggregate banner just because running VMs or system containers lack an @@ -2066,7 +2102,7 @@ the hovered timestamp, so density does not remove point-in-time metric detail. Proxmox node thermals follow that same drawer-only rule: temperature remains a node-context metric, not a universal Workloads table column, and the node detail drawer may add one compact `Thermals` history group beside utilization and I/O -only after a grouped node row is selected. Host-agent CPU temperature must be +only after a node's drawer is opened from the Proxmox nodes table. Host-agent CPU temperature must be persisted into the same metrics-store history stream as other agent metrics, with current node temperature displayed only as a drawer-local current reading while history is still accumulating; it must not become a synthetic history @@ -2106,7 +2142,7 @@ still reads nothing extra. The Proxmox node drawer overview should follow the existing guest drawer compact detail-section pattern and expose node-specific context such as platform, kernel, hardware, raw capacity, telemetry, and thermal facts rather than -repeating the metric cells already visible in the grouped table row. +repeating the metric cells already visible in the Proxmox nodes table row. Object drawers across workload, node, Docker, and unified-resource surfaces must keep their default render bounded to active attention plus a small additive context projection. Full provider-specific support workflows remain @@ -2416,16 +2452,25 @@ nothing renders: the retired `Jump to row` visibility check did exactly that, costing 128 `getComputedStyle` calls across eight row hovers and 320 across ten scroll steps on the 12-node mock storage table, and none once removed. That same hot-path contract also owns the row-emphasis paint. Workload guest -and group rows and storage pool, group, and disk rows must expose +rows and storage pool, group, and disk rows must expose summary-linked activity through the shared `data-summary-row-active` marker and let the shared frontend primitive render the emphasis, instead of layering lane-local row-fill classes that diverge across pages or wash out inline metric bars. -`frontend-modern/src/components/Workloads/useWorkloadSelectionState.ts` must -write workload selection back into the workloads route through the shared -same-path route-state scheduler, but the actual shell-position handoff for -query-only row focus must go through `frontend-modern/src/utils/appShellScrollRestoration.ts` -plus the root `frontend-modern/src/App.tsx` shell so opening a focused -workload does not look like a full page reload. The shared same-path scheduler must also own cleanup +`frontend-modern/src/components/Workloads/useWorkloadSelectionState.ts` keeps +workload selection local and never writes it back into the route; an inbound +`resource` link only hydrates it. Local row focus keeps the scroll position +through `preserveScrollableAncestorVerticalOffset` in +`frontend-modern/src/components/shared/contextualFocus.ts` and reveals the +opened detail through `summaryTableFocus.ts`, so opening a focused workload +does not look like a full page reload. The root `frontend-modern/src/App.tsx` +shell applies a pending `frontend-modern/src/utils/appShellScrollRestoration.ts` +restore on the next route change, whatever caused it, so only the shared +same-path route-state scheduler may stage one, from the shell position at the +moment it navigates. Local row focus stages none: a position staged without a +navigation waits for an unrelated one, and before this rule a Type filter +change made at the top of the Proxmox overview jumped the shell back down to +the row toggled earlier, because the scheduler stages nothing from scrollTop 0. +The shared same-path scheduler must also own cleanup for every deferred scroll-restore timeout and animation frame it creates, so route-state cleanup cannot leave hot-path replay work running after the owning surface unmounts. @@ -2445,10 +2490,9 @@ That same hot-path ownership now also covers deliberate inline-detail reveal. When a focused workload or infrastructure row opens its inline detail, the hot path may preserve scroll across same-route state writes, but the actual reveal must still flow through the shared contextual-focus and summary-table helpers. -Direct row toggles that already have the row in view must capture the current -app-shell scroll position before the focus write and let the remounted root -shell restore that position, so the interaction stays anchored instead of -looking like a full refresh. Once the drawer is mounted, the shared reveal +Direct row toggles that already have the row in view stay anchored through +that contextual-focus offset preservation, not through a staged app-shell +restore, so the interaction does not look like a full refresh. Once the drawer is mounted, the shared reveal helper must still take over whenever the opened detail would land below the fold, marking that movement as deliberate so route-state restore does not replay over it and then scrolling only enough to keep the row header plus the @@ -2738,11 +2782,12 @@ can still lose the aggregate to a live agent while its own filesystem rows remain; such a guest stays on the Proxmox rule. The agent's OS, addresses and physical disk health carry no stale cue yet. Guest, node, and Docker-host drawer headers follow the same frontend-primitives dependency -boundary for collapse: Workloads owns which inline row is selected and the -close handler, while +boundary for collapse: the owning table (Workloads for guests, the Proxmox +nodes table for nodes, the Docker hosts table for Docker hosts) owns which +inline row is selected and the close handler, while `frontend-modern/src/components/shared/ObjectDrawerHeader.tsx` owns the full-width semantic header button, phone target height, focus treatment, and -collapse chevron. `WorkloadPanel` must pass the grouped node close handler into +collapse chevron. `ProxmoxNodesTable` must pass its node close handler into `NodeDrawer`, the Docker hosts table must pass its close handler into `DockerHostDrawer`, and those shells must not fall back to a small icon-only collapse target or wrap interactive header actions inside a local button. @@ -2800,20 +2845,23 @@ counts wrap independently so a narrow toolbar cannot strand View on a line by itself or force page-level horizontal overflow. The compact Add filter control joins that cluster only in the default state where no active menu chip, contextual Clear action, or leading action needs its own filter row; active -scope editing keeps Add filter beside the chips. The sparkline range remains -inline only while trends are active because it is the frequently changed -analytical window, and its compact control must include a visible `Trend range` -label so the values are not an orphaned set of durations. Columns must expand +scope editing keeps Add filter beside the chips. The sparkline range is a +trailing inline control, not a View option, because it is the frequently +changed analytical window. `WorkloadsFilter` renders it whenever the metric +mode is Trends, and in Bars only at desktop widths while Row hover is `History` +(the default); Bars with `Details` hover, or Bars below the 768px `isMobile` +breakpoint, hides it. Its compact control carries the visible `History` label +so the values are not an orphaned set of durations. Columns must expand as a full-width responsive checkbox row inside the View disclosure instead of opening a nested absolute panel or a tall single-column desktop list. On Proxmox, the non-default Host basis must remain visible in the workload memory column header after the View disclosure closes. No workload summary chart -section renders today, so the chart-visibility control in -`frontend-modern/src/components/Workloads/WorkloadsFilter.tsx` stays hidden -(no surface passes `onChartsToggle`). If a summary section returns, that -control must expose explicit `Show charts` / `Hide charts` pressed state, and -hiding charts must remove the summary section rather than leaving an empty -collapsed summary band on screen. +section renders today, and `WorkloadsFilter` carries no chart-visibility +control or `onChartsToggle` / `chartsCollapsed` props. A summary section that +returns through a governed product decision must bring its own visibility +control with explicit `Show charts` / `Hide charts` pressed state, and hiding +charts must remove the summary section rather than leaving an empty collapsed +summary band on screen. The Workloads-owned filter-config assembly now lives in `frontend-modern/src/components/Workloads/useWorkloadsState.ts`, so future filter runtime changes must extend through those owners instead of diff --git a/docs/release-control/v6/internal/subsystems/registry.json b/docs/release-control/v6/internal/subsystems/registry.json index 95466d4ebd..2703ddc73a 100644 --- a/docs/release-control/v6/internal/subsystems/registry.json +++ b/docs/release-control/v6/internal/subsystems/registry.json @@ -5836,6 +5836,7 @@ "frontend-modern/src/features/platformPage/__tests__/platformEstateOverviewModel.test.ts", "frontend-modern/src/features/platformPage/__tests__/platformOverviewLayout.guardrails.test.ts", "frontend-modern/src/features/platformPage/PlatformOutdatedAgentNotice.test.tsx", + "frontend-modern/src/features/platformPage/PlatformOutdatedSensorSetupNotice.test.tsx", "frontend-modern/src/features/platformPage/platformSearchSuggestions.test.ts" ] }, @@ -7822,7 +7823,6 @@ "frontend-modern/src/types/recovery.ts", "frontend-modern/src/utils/recoveryArtifactModePresentation.ts", "frontend-modern/src/utils/recoveryDatePresentation.ts", - "frontend-modern/src/utils/recoveryItemTypePresentation.ts", "frontend-modern/src/utils/recoveryOutcomePresentation.ts", "frontend-modern/src/utils/recoveryTimelineChartPresentation.ts", "frontend-modern/src/utils/recoveryTimelinePresentation.ts", @@ -7867,7 +7867,6 @@ "frontend-modern/src/types/recovery.ts", "frontend-modern/src/utils/recoveryArtifactModePresentation.ts", "frontend-modern/src/utils/recoveryDatePresentation.ts", - "frontend-modern/src/utils/recoveryItemTypePresentation.ts", "frontend-modern/src/utils/recoveryOutcomePresentation.ts", "frontend-modern/src/utils/recoveryTimelineChartPresentation.ts", "frontend-modern/src/utils/recoveryTimelinePresentation.ts" diff --git a/docs/release-control/v6/internal/subsystems/security-privacy.md b/docs/release-control/v6/internal/subsystems/security-privacy.md index 1524faa42c..5c15574ac3 100644 --- a/docs/release-control/v6/internal/subsystems/security-privacy.md +++ b/docs/release-control/v6/internal/subsystems/security-privacy.md @@ -2845,6 +2845,11 @@ the tenant resource store. Runtime reconciliation may visit every live monitor, but each alert manager resolves policy through its own tenant-scoped store, so a matching provider ID in another organization cannot import the mutation. The existing route scopes and authenticated actor attribution remain unchanged. +Patrol's finding operator-state provider follows the same rule: it resolves a +finding's reference through the read state of that organization's own monitor +(`resolveMonitorForOrg`) and then reads that organization's store, so a +source-native or link-folded reference never resolves through another tenant's +registry. ### Secret-bearing configuration transfer fails closed before data access @@ -3025,6 +3030,18 @@ Removing the disconnected recorder does not alter alert, action approval or operator authority. An unrelated organization receives no default archive fallback. +### Report GETs cannot spend AI budget + +`Router.wireReportingAIResolvers` in `internal/api/router.go` wires a second +reporting resolver, `SetExistingFindingsResolver`, backed by +`AISettingsHandler.ExistingAIService`. +A `GET /api/admin/reports/generate` uses only that resolver and the +deterministic narrator, so it makes no AI provider call, writes no cost-ledger +entry and never constructs a tenant AI service. GET passes the demo-mode guard +and the CSRF check, and a SameSite=Lax session cookie rides a cross-site +top-level navigation, so AI narration of a single-resource report now requires +a CSRF-checked `POST`. No authentication, scope or session rule changed. + ### Node setup action v7 compatibility Workflow consumers pin `actions/setup-node` to diff --git a/docs/release-control/v6/internal/subsystems/storage-recovery.md b/docs/release-control/v6/internal/subsystems/storage-recovery.md index d717f69894..ccb8f2ea1d 100644 --- a/docs/release-control/v6/internal/subsystems/storage-recovery.md +++ b/docs/release-control/v6/internal/subsystems/storage-recovery.md @@ -573,29 +573,28 @@ Compact Coverage age cells use the existing metadata-size text and reclaim numer 9. `frontend-modern/src/types/recovery.ts` 10. `frontend-modern/src/utils/recoveryDatePresentation.ts` 11. `frontend-modern/src/utils/recoveryTimelinePresentation.ts` -12. `frontend-modern/src/utils/recoveryItemTypePresentation.ts` -13. `frontend-modern/src/utils/textPresentation.ts` -14. `frontend-modern/src/utils/storageSummaryCache.ts` -15. `frontend-modern/src/components/Storage/useStorageSummaryCharts.ts` -16. `frontend-modern/src/features/storageBackups/storageCapacityDeltaPresentation.ts` -17. `frontend-modern/src/features/proxmox/BackupActivityChart.tsx` -18. `frontend-modern/src/features/proxmox/proxmoxBackupActivityPresentation.ts` -19. `frontend-modern/src/features/proxmox/proxmoxBackupRecoveryModel.ts` -20. `frontend-modern/src/features/proxmox/ProxmoxBackupsCoverageStrip.tsx` -21. `frontend-modern/src/features/proxmox/ProxmoxBackupServersTable.tsx` -22. `frontend-modern/src/features/proxmox/ProxmoxBackupsTable.tsx` -23. `frontend-modern/src/features/proxmox/ProxmoxPageSurface.tsx` -24. `frontend-modern/src/features/proxmox/ProxmoxRecoverableTable.tsx` -25. `frontend-modern/src/features/proxmox/proxmoxBackupsTableShared.tsx` -26. `frontend-modern/src/features/proxmox/proxmoxBackupSourcePresentation.ts` -27. `internal/recovery/model/posture.go` -28. `internal/recovery/posture.go` -29. `internal/recovery/store/store_posture.go` -30. `frontend-modern/src/hooks/useProtectionPostures.ts` -31. `frontend-modern/src/features/proxmox/ProxmoxCoverageTable.tsx` -32. `internal/mock/recovery_points.go` -33. `frontend-modern/src/features/proxmox/useProxmoxBackupTableWindowing.ts` -34. `frontend-modern/src/components/Storage/useStoragePoolsTableWindowing.ts` +12. `frontend-modern/src/utils/textPresentation.ts` +13. `frontend-modern/src/utils/storageSummaryCache.ts` +14. `frontend-modern/src/components/Storage/useStorageSummaryCharts.ts` +15. `frontend-modern/src/features/storageBackups/storageCapacityDeltaPresentation.ts` +16. `frontend-modern/src/features/proxmox/BackupActivityChart.tsx` +17. `frontend-modern/src/features/proxmox/proxmoxBackupActivityPresentation.ts` +18. `frontend-modern/src/features/proxmox/proxmoxBackupRecoveryModel.ts` +19. `frontend-modern/src/features/proxmox/ProxmoxBackupsCoverageStrip.tsx` +20. `frontend-modern/src/features/proxmox/ProxmoxBackupServersTable.tsx` +21. `frontend-modern/src/features/proxmox/ProxmoxBackupsTable.tsx` +22. `frontend-modern/src/features/proxmox/ProxmoxPageSurface.tsx` +23. `frontend-modern/src/features/proxmox/ProxmoxRecoverableTable.tsx` +24. `frontend-modern/src/features/proxmox/proxmoxBackupsTableShared.tsx` +25. `frontend-modern/src/features/proxmox/proxmoxBackupSourcePresentation.ts` +26. `internal/recovery/model/posture.go` +27. `internal/recovery/posture.go` +28. `internal/recovery/store/store_posture.go` +29. `frontend-modern/src/hooks/useProtectionPostures.ts` +30. `frontend-modern/src/features/proxmox/ProxmoxCoverageTable.tsx` +31. `internal/mock/recovery_points.go` +32. `frontend-modern/src/features/proxmox/useProxmoxBackupTableWindowing.ts` +33. `frontend-modern/src/components/Storage/useStoragePoolsTableWindowing.ts` ## Shared Boundaries @@ -980,10 +979,11 @@ states. This heading alignment changes neither workload backup scope nor the storage/recovery evidence carried by the adjacent Backup column and Backups tab. The guest table's grouped node rows are identity-only dividers. Node stats and -the node drawer stay with the Proxmox nodes table above it, so `ProxmoxPageSurface` -passes `groupNodeDrawerMode: 'disabled'` and no host-metric option to the -embedded Workloads state. Those dividers carry no backup or recovery evidence; -the per-guest Backup column remains the overview's protection signal. +the node drawer stay with the Proxmox nodes table above it: the embedded +Workloads table has no grouped-row drawer, hover preview, or group pin, and +`ProxmoxPageSurface` passes no host-metric option to the embedded Workloads +state. Those dividers carry no backup or recovery evidence; the per-guest +Backup column remains the overview's protection signal. The overview's node-row action selects the shared workload node scope and reveals the guest section. The title names the selected node and reports the filtered guest count, including zero. Activating the selected node again or @@ -1961,7 +1961,7 @@ recovery scope, or a storage/recovery-owned secret source. resources always withheld) is not a storage/recovery restore approval, backup freshness, recovery-scope, or restore-command signal. Patrol finding chat handoff execution controls in `internal/api/ai_handler.go` - follow the same boundary: backend-forced `autonomous_mode:false` for + follow the same boundary: the backend-forced approval-required mode for `finding_id` handoffs with model-only Patrol briefing, resource, or action context is Assistant action-governance, not a storage/recovery approval, recovery freshness, or restore-command signal. @@ -2184,7 +2184,7 @@ recovery scope, or a storage/recovery-owned secret source. recovery, SMART, or cleanup action. 22. Keep backend-native platform configuration reads on the adjacent AI/runtime and platform contracts. When `internal/api/` wires native TrueNAS app config for Assistant, storage and recovery may use that runtime shape during investigation, but they must not grow a parallel recovery-local config transport or provider-shaped configuration payload. 23. Keep provider-backed poll cadence and settings-runtime health on the adjacent platform-connections contract. When shared `internal/api/` and poller wiring expose TrueNAS last-sync status, failure summaries, discovered contribution counts, manual saved-test status refresh, or platform handoff links in settings, storage and recovery may consume the resulting datasets, apps, disks, and recovery artifacts but must not redefine those settings-runtime health semantics or connection-level handoffs in storage/recovery-local transport or page flows. -24. Keep recovery filter/query state on the shared route-state parsing contract without restoring standalone recovery navigation. When platform pages or other embedded owners expose TrueNAS recovery context, they may reuse the canonical recovery query vocabulary with owned `platform` and `node` fields, but they must land inside an owning platform/runtime route instead of inventing drawer-local recovery URLs, treating PBS services as the only recovery path, or sending operators to the retired Recovery aggregate route. +24. Keep recovery context inside the platform tab that owns it without restoring standalone recovery navigation. TrueNAS recovery context lives on the TrueNAS Protection tab and Proxmox recovery context on the Proxmox Backups tab; no shared recovery query vocabulary exists to reuse. A surface that exposes recovery context must land inside an owning platform/runtime route instead of inventing drawer-local recovery URLs, treating PBS services as the only recovery path, or sending operators to the retired Recovery aggregate route. That same shared route-helper boundary also owns exact workload handoffs when storage or recovery surfaces send operators back to node-scoped workloads for investigation context. Proxmox VM and system-container links @@ -2196,7 +2196,7 @@ recovery scope, or a storage/recovery-owned secret source. Patrol control anchor, must stay owned by their product surface and must not be reused as recovery entry points or storage/recovery navigation aliases. -25. Keep alert-side recovery drill-ins on that same embedded-owner route-state contract. When alert investigation surfaces such as resource-incident panels expose recovery follow-up links for TrueNAS or future API-backed platforms, they must route through an owning platform/runtime destination using canonical recovery query vocabulary instead of freezing alert-local recovery URLs, reviving the retired Recovery aggregate route, or introducing another provider-shaped recovery handoff vocabulary. +25. Keep alert-side recovery follow-up inside the owning platform tab. The resource-incident panel offers no recovery links today, and no shared recovery query vocabulary exists. A future alert-side recovery follow-up for TrueNAS or another API-backed platform must open the platform tab that owns those rows instead of freezing alert-local recovery URLs or reviving the retired Recovery aggregate route. 26. Keep VMware onboarding runtime and recovery semantics separate on that same adjacent platform-connections contract. When `internal/api/router.go`, `internal/api/router_routes_registration.go`, or `internal/api/vmware_handlers.go` evolve VMware connection CRUD, poller-owned `poll` / `observed` summary payloads, saved-test refresh, or observed datastore/VM snapshot visibility, storage and recovery may consume the resulting shared context but must not treat those onboarding/runtime payloads as canonical recovery artifacts, restore capability, or recovery-local control transport. A successful VMware connection test with `degraded` optional signal or performance diagnostics remains connection-readiness context only; storage @@ -2430,11 +2430,13 @@ recovery scope, or a storage/recovery-owned secret source. Storage and recovery UI must keep sourcing those signals from their existing canonical page models instead of polling the connections ledger for per-datastore or per-backup truth. - Platform-first top-level pages may embed `StorageSurface` and - `RecoverySurface` with `embedded tableOnly` and forced source or - platform filters (e.g. `forcedSourceFilter`, `forcedPlatformFilter`) - so platform-scoped storage and recovery rows render through the same - canonical surfaces rather than a forked per-platform table. + Platform-first top-level pages may embed `StorageSurface` + (`frontend-modern/src/components/Storage/Storage.tsx`) with a forced + source filter (`forcedSourceFilter`) so platform-scoped storage rows + render through the same canonical surface rather than a forked + per-platform table. The surface has no `embedded` or `tableOnly` mode + and no `forcedPlatformFilter`, and no `RecoverySurface` component + exists to embed alongside it. `frontend-modern/src/App.tsx` may carry the platform-page route registrations that mount those embedded canonical surfaces, but the routes themselves must derive their paths from the canonical builders @@ -2444,22 +2446,30 @@ recovery scope, or a storage/recovery-owned secret source. canonical surface that actually populates. The canonical TrueNAS adapter already emits the top-level TrueNAS system as a unified `agent` row tagged with the `truenas` platform, so the platform - page defaults to `/truenas/overview` (the Systems sub-tab) and the - embedded `StorageSurface` lives at `/truenas/storage`. The Source - filter chip in `StoragePageControls` is also suppressed when a - platform page locks source scope through `forcedSourceFilter` (via - `suppressSourceFilter`, auto-applied whenever `forcedSourceFilter` - is set), so the user never sees the platform's name pinned as a - removable filter chip inside the embedded surface. - Platform pages that embed `StorageSurface` reuse the canonical - `StoragePageControls` toolbar through the `showFilterToolbar` prop on - `StorageProps`. The page keeps `tableOnly` to hide the storage summary - section but opts in to the shared search, status, grouping, sort, - node, view, and chart-collapse controls so platform operators get - dense-table storage controls on every embedded storage tab without - forking the toolbar. The source scope flows through - `forcedSourceFilter` as a typed page input; the source filter remains - available in the toolbar only when not forced. The seven-state Storage + page defaults to `/truenas/overview`, whose Overview sub-tab lists + those systems. `/truenas/storage` renders the TrueNAS-owned + `TrueNASStorageTopologyTable`, not `StorageSurface`; the Proxmox + Storage tab in `frontend-modern/src/features/proxmox/ProxmoxPageSurface.tsx` + is the only page that embeds `StorageSurface` today. + An embedding page gets the canonical `StoragePageControls` toolbar + without opting in: `Storage.tsx` always renders it (only kiosk mode + hides it), there is no `showFilterToolbar` prop, and there is no storage + summary section to hide. The Proxmox embed passes `forcedSourceFilter`, + its own `resourceSource`, `suppressNodeFilter`, `filterAriaLabel`, and + Proxmox search placeholder and empty-history copy. It passes no + `forcedView`, so the Storage / Physical Disks view tabs stay. + `forcedSourceFilter` is a typed page input that the route state reads + as the source value and never writes back to the URL, and `Storage.tsx` + sets `suppressSourceFilter` whenever it is present. `StoragePageControls` + then drops the Source chip, so the user never sees the platform's name + pinned as a removable filter chip, and the locked source does not count + as an active filter that surfaces the toolbar's clear action. + `suppressNodeFilter` drops the Node chip so search carries host and + node scoping, and `Storage.tsx` resets any selected node to all nodes + while it is set. The embedded toolbar otherwise keeps search, the + Status filter on the Storage view (Role and Group filters on Physical + Disks once there is more than one choice), and the View popover's + grouping and sort; it has no chart-collapse control. The seven-state Storage status catalog stays in the shared Add filter menu instead of occupying a permanent segmented rail; a non-default status surfaces through the canonical FilterBar chip. Grouping and sort remain durable presentation @@ -2547,27 +2557,11 @@ take a correctness dependency on its contents. 6. Letting whitespace-padded storage route params hydrate non-canonical page state; shared storage URLs must trim and normalize `tab`, `source`, `status`, `node`, `group`, `sort`, `order`, `query`, and deep-link `resource` before the page model consumes them so pasted or hand-edited links resolve to the same canonical state as UI-authored routes without dropping adjacent unmanaged params 7. Letting storage `source` aliases or case drift survive in canonical route state; shared storage URLs must rewrite pasted values like `PVE`, `pbs`, or `ALL` to the owned source option values (for example `proxmox-pve`) or the canonical unset state so copied links match the same source filter values the storage toolbar presents 8. Letting explicit storage `all` sentinels survive in canonical route state; shared storage URLs must collapse case- or whitespace-variant `all` values for the managed `node` filter back to the canonical unset state so copied links do not preserve a fake active node filter -9. Letting whitespace-padded recovery timeline params fall off canonical route state; shared recovery URLs must trim and normalize `day`, `range`, `scope`, `status`, `verification`, `cluster`, `node`, `namespace`, `itemType`, and adjacent history filters before the page model validates them so pasted or hand-edited links resolve to the same canonical timeline and filter state as UI-authored routes -10. Letting explicit recovery `all` sentinels survive in canonical route state; shared recovery URLs must collapse case- or whitespace-variant `all` values for `cluster`, `node`, `namespace`, and `itemType` back to the canonical unset route state so copied links do not preserve fake active filters -11. Letting non-canonical recovery platform values survive in route or transport state; shared recovery URLs must collapse unsupported or fake `platform` values back to the canonical unset state, and only owned source-platform options or canonical legacy aliases may reach rollups, points, series, and facets transport filters - 11c. Letting route-owned recovery platform selections disappear while filter options are still hydrating; the recovery page state owner must keep the current canonical `platform` query value present in the platform option set until transport-backed facets and records arrive so shared filter selects keep the user-visible TrueNAS or other owned platform selection instead of flashing back to `All platforms` - 11d. Letting recovery filter default labels drift between protected inventory and recovery events; both recovery filter surfaces must consume the shared `recoveryTablePresentation` labels (`All item types`, `All platforms`, and `Any item`) instead of hard-coding title-case local variants. - 11a. Letting adjacent workload route-state changes in shared `frontend-modern/src/routing/resourceLinks.ts` perturb recovery parse/build semantics; expanding canonical workload platform scoping must not alter the owned recovery `platform` and `itemType` vocabulary, legacy alias rewrites, or recovery drill-down workspace selection - 11b. Letting adjacent storage route-state additions in shared `frontend-modern/src/routing/resourceLinks.ts` perturb recovery route semantics; expanding canonical storage deep links for unified resources must not reuse recovery-owned query names or alter the owned recovery parse/build contract while those surfaces continue sharing the same route-helper module - 11e. Letting adjacent platform-route additions in shared `frontend-modern/src/routing/resourceLinks.ts` perturb storage or recovery route semantics; adding canonical `/standalone/machines`, `/standalone/availability`, or other platform paths must not reuse storage/recovery query names, alter storage or recovery parse/build behavior, or convert agent-platform membership into storage/recovery ownership - 11f. Letting adjacent Patrol control starter route state in shared `frontend-modern/src/routing/resourceLinks.ts` perturb storage or recovery route semantics; `patrolControlStarter=patrol_control` is a first-party Patrol control handoff flag only, legacy `operationsLoopStarter=patrol_control`, `operationsLoopStarter=patrol_autonomy`, and `operationsLoopStarter=pulse_pro_activation` are only compatibility aliases, and storage/recovery parse-build contracts must not consume any of them as recovery state, storage focus, platform scope, or proof of backup/recovery posture - 11g. Letting adjacent Pulse Intelligence external-agent setup anchors in shared `frontend-modern/src/routing/resourceLinks.ts` perturb storage or recovery route semantics; `/settings/pulse-intelligence/assistant#external-agent-setup` is the canonical external-agent setup hash, `/settings/security/api#external-agent-setup` and `/settings/security/api#pulse-mcp-setup` are legacy compatibility hashes, and storage/recovery parse-build contracts must not consume any of them as recovery state, storage focus, platform scope, or proof of backup/recovery posture - 11h. Letting the Kubernetes platform page's route-local `cluster` scope perturb recovery's separate `cluster` filter merely because both constants live in shared `frontend-modern/src/routing/resourceLinks.ts`; `/kubernetes` may serialize `cluster=` for native inventory scoping, while `/recovery` continues to normalize and transport its own protected-item cluster vocabulary through the recovery parser/builders, with neither surface importing the other's option set or state owner -12. Letting protected-inventory protection posture overload recovery-event outcome filtering; the protected inventory protection-state control must drive the route-backed `state` field and local rollup posture filtering, while the recovery events `status` field remains the canonical outcome filter for points, series, and facets transport filters -13. Letting visible protected-item filters fall out of shared recovery links; protected inventory state such as stale, failed, warning, running, unknown, healthy, and never-succeeded must restore from the canonical recovery URL and rewrite to the owned `state=` route form, with legacy `stale=1` accepted only as compatibility input -14. Reintroducing stacked full-width recovery tables as the primary desktop layout; the governed recovery surface must expose one primary data region at a time with recovery events as the default workspace and protection coverage as an explicit secondary review so Pulse does not collapse back into a single-platform backup screen -15. Letting secondary recovery workspace state drift out of canonical route state; explicit `view=inventory` protection-coverage links must round-trip, while default recovery-events state should serialize without a redundant `view=events` query unless compatibility input is being normalized -16. Treating a selected protected-item rollup as row-click-only or header-only state instead of a canonical history filter; when a protected-item row focuses recovery history, the governed recovery events controls must surface that focus inside the shared filter surface through the same user-creatable item filter control, count it with the rest of the active filters, and let the same filter reset path clear it -17. Letting recovery-event focus leak as hidden state on the protection coverage surface; opening coverage must clear event-only `rollupId` and `day` context, and coverage drill-ins must open recovery events without preserving an invisible day filter that can make valid history look empty. -18. Letting the recovery details panel lead with transport-shaped payloads; operator-facing details must summarize outcome, artifact, target, restore readiness, and readable metadata labels first, while raw JSON and provider-specific keys stay behind an explicitly technical disclosure. -19. Letting protection coverage navigation bypass the canonical recovery workspace and route-state owner; coverage actions may focus stale inventory, attention inventory, or all protected items, but they must not mutate local-only filter state or revive passive posture-counter cards as the navigation owner. -20. Letting storage or recovery surfaces invoke retired self-hosted trial acquisition; `POST /api/license/trial/start` and `/auth/trial-activate` must stay closed on the ordinary self-hosted router, and storage/recovery-adjacent billing or support handoffs must not treat trial activation as recovery identity, restore proof, or backup transport state. -21. Treating an `AvailabilityData` facet on a storage resource as backup, protection, or recovery evidence; an agentless probe attached to a NAS, PBS, or datastore resource is monitoring reachability, not backup success, snapshot health, or restore readiness. Storage and recovery must read protection status from the canonical backup and storage resources, not from the availability probe facet. +9. Reintroducing an aggregate cross-platform Recovery page, a recovery route-state owner, or a shared recovery query serializer in `frontend-modern/src/routing/resourceLinks.ts` without a governed product decision. The aggregate page (`frontend-modern/src/components/Recovery/` with its state owner `frontend-modern/src/features/recovery/useRecoverySurfaceState.ts`) was deleted on 2026-05-26, and `RECOVERY_QUERY_PARAMS`, `parseRecoveryLinkSearch`, and `buildRecoveryRouteSearch` were deleted once nothing read them. Recovery rows render in the platform-owned Proxmox Backups and TrueNAS Protection tabs; the Proxmox Backups tab owns its route state through the `/proxmox/backups/:view` segment and `PROXMOX_BACKUPS_QUERY_PARAMS`, and the TrueNAS Protection tab keeps its search and status filter in component state and its sort in local storage, outside the URL. +10. Letting adjacent route state in shared `frontend-modern/src/routing/resourceLinks.ts` reuse storage or Proxmox Backups query names or alter their parse/build behavior. Workload platform scoping, unified-resource storage deep links, `buildStandalonePath()` platform paths, the Kubernetes page's route-local `KUBERNETES_QUERY_PARAMS.cluster` scope, the Patrol control starter (`patrolControlStarter=patrol_control`, with legacy `operationsLoopStarter=patrol_control`, `operationsLoopStarter=patrol_autonomy`, and `operationsLoopStarter=pulse_pro_activation` as compatibility aliases only), and the Pulse Intelligence external-agent setup anchors (`/settings/pulse-intelligence/assistant#external-agent-setup`, with `/settings/security/api#external-agent-setup` and `/settings/security/api#pulse-mcp-setup` as legacy compatibility hashes) share that module, and none of them is recovery state, storage focus, platform scope, or proof of backup or recovery posture. +11. Letting recovery row details on the Proxmox Backups or TrueNAS Protection tabs lead with transport-shaped payloads; operator-facing details must summarize outcome, artifact, target, and readable metadata labels first, and raw JSON or provider-specific keys must not lead the primary detail. +12. Letting storage or recovery surfaces invoke retired self-hosted trial acquisition; `POST /api/license/trial/start` and `/auth/trial-activate` must stay closed on the ordinary self-hosted router, and storage/recovery-adjacent billing or support handoffs must not treat trial activation as recovery identity, restore proof, or backup transport state. +13. Treating an `AvailabilityData` facet on a storage resource as backup, protection, or recovery evidence; an agentless probe attached to a NAS, PBS, or datastore resource is monitoring reachability, not backup success, snapshot health, or restore readiness. Storage and recovery must read protection status from the canonical backup and storage resources, not from the availability probe facet. ## Completion Obligations @@ -2709,7 +2703,7 @@ canonical pre-handler authorization proof covering both transfer routes. is likewise API/AI-owned transport: storage and recovery may consume the resulting incident context, but must not define storage-local approval argument keys or bypass the shared `internal/agentcapabilities` helper. -5. Keep recovery history table width budgeting derived from the canonical column specs in `frontend-modern/src/utils/recoveryTablePresentation.ts`, not from raw visible-column counts, so normalized subject labels and optional column sets cannot drift the right-edge badges and controls off-screen. Storage-pool, physical-disk, backup-server, coverage, and recoverable-artifact compact models must follow the same source-owned budgeting rule: expose three to five semantically distinct phone fields, allow a sixth only when measured content remains legible, fix identity at the shared 30-percent track, use explicit compact labels for narrow evidence fields, and normalize the selected tracks to the container without a horizontal rail. A status already conveyed by the identity indicator must not consume a duplicate phone column. Desktop column sets and storage/recovery evidence semantics must remain unchanged. +5. Keep storage and recovery table width budgeting source-owned rather than derived from raw visible-column counts, so long item labels and optional column sets cannot drift the right-edge badges and controls off-screen. Storage-pool, physical-disk, backup-server, coverage, and recoverable-artifact compact models must expose three to five semantically distinct phone fields, allow a sixth only when measured content remains legible, fix identity at the shared 30-percent track, use explicit compact labels for narrow evidence fields, and normalize the selected tracks to the container without a horizontal rail. A status already conveyed by the identity indicator must not consume a duplicate phone column. Desktop column sets and storage/recovery evidence semantics must remain unchanged. Below 360 pixels of content width, those models must promote identity to 40 percent and may demote the lowest-priority field through the shared `platform-table-narrow-hidden`/source-owned layout contract rather than @@ -2887,24 +2881,33 @@ canonical pre-handler authorization proof covering both transfer routes. Touch users still must not inherit synthetic hover branches, and storage must not keep a special trailing expand column once the shared leading action contract exists. - Static subgroup header emphasis for storage group rows and recovery history - day headers must also route through - `frontend-modern/src/components/shared/groupedTableRowPresentation.ts` and - the shared `.grouped-table-row` CSS contract in `frontend-modern/src/index.css`, - rather than storage- or recovery-local background classes or left-accent - marker variants. + Static subgroup header emphasis for storage group rows must also route + through `frontend-modern/src/components/shared/groupedTableRowPresentation.ts` + and the shared `.grouped-table-row` CSS contract in + `frontend-modern/src/index.css`, rather than storage-local background + classes or left-accent marker variants. The recovery history day headers + this rule also named went with the aggregate Recovery page; the Proxmox + Backups date feed's day dividers (`ProxmoxRecoverableTable.tsx`) style + their cell directly because a cell-level background renders reliably + across table layout engines. Storage pool rows must also keep sizing and alert accents on canonical class/data-attribute presentation rather than row-local inline style maps, so the public storage page stays CSP-safe under both normal and alert-highlighted demo/runtime states. -21. Keep recovery transport refreshes inside the recovery-owned feature state. - `frontend-modern/src/features/recovery/useRecoverySurfaceState.ts` and the - recovery data hooks may retain the last fulfilled rollups, points, facets, - and series while the next request is in flight through - the shared `frontend-modern/src/hooks/createNonSuspendingQuery.ts`, but - that retained-value behavior must stay route-owned and filter-owned through - the canonical recovery state model instead of recreating page-local - suspense escape hatches in `Recovery.tsx` or the recovery sections. +21. Keep recovery transport refreshes on the shared non-suspending query. + Every recovery read retains its last fulfilled response while the next + request is in flight through + `frontend-modern/src/hooks/createNonSuspendingQuery.ts`: + `frontend-modern/src/hooks/useRecoveryPoints.ts` (the TrueNAS Protection + tab's points) and `frontend-modern/src/hooks/useProtectionPostures.ts` + poll every 30 seconds, and `ProxmoxBackupsTable.tsx` reads + `/api/backups/pve` and `/api/backups/pbs` through the same primitive. + The points and posture reads stay keyed by the query or resource IDs + their tab passes, and the PVE and PBS inventory reads by fixed source + keys, instead of recreating page-local suspense escape hatches or a second + refetch state in the platform tables. The aggregate Recovery page's state owner, which + this item used to name, was deleted on 2026-05-26 with the rollup, facet, + and series hooks. 22. Keep storage/recovery-adjacent resource metadata on the shared unified resource contract. When canonical storage resources expose provider-backed identity such as Proxmox storage `pool`, storage and recovery consumers @@ -3476,36 +3479,44 @@ the platform-first migration. The chip strips that previously rendered them inside the alert resource-incidents panel and Patrol findings panel were retired in the same pass; storage and recovery drilldowns now stay inside the platform-page sub-tabs rather than offering external surface jumps. -Future cross-surface storage or recovery affordances must compose against -the embedded `StorageSurface` / `RecoverySurface` consumers rather than -reintroducing top-level URL builders. +Future cross-surface storage or recovery affordances must link into the +platform tab that owns those rows (the Proxmox Storage tab's embedded +`StorageSurface`, `/truenas/storage`, the Proxmox Backups tab, or the TrueNAS +Protection tab) rather than reintroducing top-level URL builders. The remaining storage and recovery route builders in -`frontend-modern/src/routing/resourceLinks.ts` are query-state serializers, not -destination builders: callers must append `buildStorageRouteSearch()` or -`buildRecoveryRouteSearch()` to the current platform-owned pathname such as -`/proxmox/storage`, `/proxmox/backups`, or `/truenas/protection`. They must not -emit `/storage` or `/recovery` as hidden compatibility paths. +`frontend-modern/src/routing/resourceLinks.ts` come down to one query-state +serializer, not a destination builder: `buildStorageRouteSearch()` is appended +to the current platform-owned pathname, today only by the Storage surface's +own filter state on the Proxmox Storage tab, and must not emit `/storage` as a +hidden compatibility path. The recovery serializer (`RECOVERY_QUERY_PARAMS`, +`parseRecoveryLinkSearch`, `buildRecoveryRouteSearch`) was deleted once +nothing read it: the Proxmox Backups tab keeps its view in the +`/proxmox/backups/:view` segment and its filters in +`PROXMOX_BACKUPS_QUERY_PARAMS`, and nothing emits `/recovery`. Adjacent route-query constants for other platform surfaces, including `STANDALONE_QUERY_PARAMS`, must remain namespaced and must not be reused as storage source, node, resource, workspace, rollup, or day state. Adding a shareable filter query to another surface must not broaden or reinterpret the -storage and recovery serializer contract. +storage serializer contract or the Proxmox Backups query. -Storage and Recovery can now be embedded by a platform page in table-only mode -with a forced platform source/filter. Proxmox uses that embedding for -source-scoped storage and recovery history, while the TrueNAS Protection tab -uses the same Recovery surface with a forced `truenas` platform filter and the -protection-coverage workspace as its default entry point. The embedded mode -suppresses standalone page chrome, summary charts, and full filter chrome, but -the Storage surface must keep the canonical Storage / Physical Disks view -selector inside the table header unless the embedding explicitly locks a -`forcedView`, and the Recovery surface must keep protection/events workspace -state in `frontend-modern/src/features/recovery/useRecoverySurfaceState.ts`. -The canonical route-backed filter state, fetch builders, table rendering, and -storage/recovery vocabulary remain owned by the Storage and Recovery surfaces. -Platform pages must compose those owners rather than cloning storage pools, -physical disks, recovery events, or protected-inventory tables under -platform-specific data contracts. +A platform page embeds `StorageSurface` with a forced source filter; there is +no table-only mode, and no Recovery surface exists to embed. The Proxmox +Storage tab is the only embedder: it passes `forcedSourceFilter`, and the +surface keeps the canonical Storage / Physical Disks view selector because +Proxmox locks no `forcedView`. Recovery rows render in platform-owned tabs. +The Proxmox Backups tab (`frontend-modern/src/features/proxmox/ProxmoxBackupsTable.tsx`) +reads `/api/backups/pve`, `/api/backups/pbs`, and protection postures itself, +and the TrueNAS Protection tab feeds +`frontend-modern/src/features/truenas/TrueNASProtectionTable.tsx` from +`useRecoveryPoints` with the `truenas` platform forced. `/truenas/storage` +renders `TrueNASStorageTopologyTable`, not `StorageSurface`. The Storage +surface keeps owning its route-backed filter state, fetch builders, table +rendering, and storage vocabulary for its embedder. The Proxmox Storage tab +must keep its storage pools and physical disks on that embed rather than a +Proxmox-local copy; TrueNAS storage is deliberately a TrueNAS-owned topology +table built from the TrueNAS page model. Platform recovery tables must read +the canonical backup, posture, and recovery-point transports rather than a +platform-specific recovery data contract. The investigation enrichment path reads operator-state from the in-memory provider already wired against the durable @@ -3513,6 +3524,13 @@ in-memory provider already wired against the durable commitments survive across restarts on the investigation read path the same way they do on the suppression read path — both flow through the same provider over the same durable table. +That provider resolves a finding's resource reference through the +organization monitor's current read state before it reads the table, +so a reference that read state folds into another resource through an +operator link reads the merged row. This changes which existing row is read, +not what is stored: no new persistence, migration or recovery path +opens, and a row left under a folded ID is neither rewritten nor +removed. The agent SSE stream at `/api/agent/events` is in-memory and stateless. No persistence; each connection starts fresh from the @@ -3732,11 +3750,15 @@ Storage physical-disk requirements copy now consumes the shared Infrastructure target label. Disk-health guidance may refer to Proxmox node requirements, but it must not revive removed nested Pulse settings paths such as `Settings → Infrastructure → Proxmox`. -Recovery item-type labels now route through -`frontend-modern/src/utils/recoveryItemTypePresentation.ts`. Recovery surfaces -must not render a bare `Cluster` item type for Kubernetes protected subjects; -use the canonical `K8s Cluster` label so protected inventory and event filters -do not confuse Kubernetes clusters with Proxmox clusters or table grouping. +The cross-platform recovery item-type columns and filters went with the +aggregate Recovery page on 2026-05-26, and +`frontend-modern/src/utils/recoveryItemTypePresentation.ts` was deleted once +its last reader, the recovery query serializer, was. The Proxmox Backups +tables label workload types (`VM`, `LXC`, `Host`, or `Guest`) from +`frontend-modern/src/features/proxmox/proxmoxBackupRecoveryModel.ts`. A future +cross-platform recovery item-type label must not render a bare `Cluster` for +Kubernetes protected items; it uses `K8s Cluster` so Kubernetes clusters are +not confused with Proxmox clusters or table grouping. This subsystem now sits under the dedicated storage and recovery lane so the operator-facing storage page, recovery timeline, and recovery-point persistence @@ -4269,8 +4291,8 @@ AI/runtime review identity only. Storage and recovery may display or link from the safe run ID, run type/status, runtime-failure flag, or scoped resource label as incident context, but they must not reinterpret it as backup freshness, restore proof, storage health authority, or recovery-local remediation state. -Patrol finding handoffs that force approval-required Assistant mode from a -non-empty `finding_id` follow the same adjacent API boundary: storage and +Patrol finding handoffs, which the backend runs in approval-required +Assistant mode from a non-empty `finding_id`, follow the same adjacent API boundary: storage and recovery may treat the resulting Assistant session as incident context, but must not reinterpret the finding ID or approval-required chat mode as backup freshness, restore authorization, storage remediation permission, or recovery @@ -4314,12 +4336,8 @@ controls, so those headings must not duplicate that interaction with buttons, their real button and `aria-sort` semantics, but their visible direction marker must come from the frontend-primitives-owned `tableSortPresentation.ts` helper; only the active backup sort column displays its direction. -Recovery all-history, all-item-type, and all-platform defaults follow the same -shared filter-option contract through -`frontend-modern/src/utils/recoveryTablePresentation.ts`, so recovery history -and protected-item tables do not invent separate default-filter wording. Physical-disk role and group filter defaults plus disk-type display labels -must likewise come from `frontend-modern/src/features/storageBackups/diskPresentation.ts`; +must come from `frontend-modern/src/features/storageBackups/diskPresentation.ts`; storage pages must not reintroduce local `All Roles`, `All Groups`, or `NVME Disk` strings that drift away from the shared filter-label and hardware acronym presentation contract. @@ -4370,7 +4388,26 @@ which reopens a focused pool's collapsed owning group, and a pinned pool-group header that sits off-screen scrolls into view through the same bridge. Focus only ever names a row in the active pools or disks view, so the callback never switches views; the view-switching branches that served the retired jump to -the active row are gone. +the active row are gone. Reveal runs once per deliberate focus change: the +bridge starts a reveal only when the focused series id or the table root +element changes, and runs the callback and the inline-detail scroll untracked, +so the storage state the callback reads (the pool-to-group lookup that every +live update rebuilds, the expanded group set) never starts another. A live +update therefore cannot pull the page back to an open pool the operator +scrolled away from, and collapsing the open pool's group does not bounce it +back open. Filter changes do not re-run the callback either, so a focused +pool whose row stays filtered out past the reveal's short retry window is not +revealed when the filter later clears; storage filters live in the URL and a +storage link without filter parameters resets them, so only a link carrying a +filter that excludes its own pool reaches that case. +`frontend-modern/src/components/Storage/useStorageExpansionState.ts` stores +only the groups the operator collapsed and derives the expanded set from the +current group keys, so a group that appears later opens by default and a +collapsed group stays collapsed across live updates until the operator reopens +it, focus moves to one of its pools, or the grouping changes. Group keys are +bare labels, so a grouping change clears the collapsed set rather than letting +a label carry collapse state into another grouping or hide the headerless +ungrouped `All` group. That same reveal contract now also owns inline-detail expansion. When a pool or disk row is deliberately focused and its inline detail opens on the storage page, the detail row must publish the same canonical summary series ID through @@ -4389,9 +4426,9 @@ That same recovery boundary now also assumes mock recovery context is projected from one canonical mock graph. `internal/mock/recovery_points.go` may synthesize inventory-only recovery artifacts for supported mock platforms, but those subjects must derive from the shared `internal/mock/fixture_graph.go` owner -instead of a separate hardcoded recovery cache, so recovery filters, rollups, -and shared route handoffs see the same platform set as settings and -infrastructure. +instead of a separate hardcoded recovery cache, so recovery points, rollups, +and the platform tabs that read them see the same platform set as settings +and infrastructure. That same graph-owned mock boundary also owns demo-readiness for storage and recovery surfaces. Mock seeded history and provider-backed storage/recovery counts must come from the same canonical fixture graph so @@ -4478,92 +4515,39 @@ That same shared helper layer also now assumes the Pulse Mobile relay runtime credential reaches only the explicit backend-owned route inventory, so storage- and recovery-adjacent transport work cannot accidentally widen that credential into a broader AI access bundle by touching neighboring routes. -The recovery frontend now also separates that ownership more explicitly: -`frontend-modern/src/features/recovery/useRecoverySurfaceState.ts` owns -canonical route parsing, filter/query state, transport hook inputs, and URL -synchronization, while `frontend-modern/src/components/Recovery/Recovery.tsx` -is the composition root for the operator-facing recovery surface and the split -section owners under `frontend-modern/src/components/Recovery/` hold the -protection coverage, activity, and history presentation layers. The history -surface is further split so `RecoveryHistorySection.tsx` owns the toolbar and -controller boundary, `useRecoveryHistorySectionState.ts` owns local section UI -state, and `RecoveryHistoryTable.tsx` owns the row/detail renderer. -That composition root now also owns one primary recovery workspace rather than -stacking protection-coverage and event-history tables on the same desktop page. -The governed default is event-first so operators land on concrete -backup/snapshot/replication history, while the header action and compatibility -`view=inventory` links open the secondary protection coverage review. -That same operator-facing workspace must lead with current protection status -rather than only the latest backup outcome. Protection coverage should surface -stale, never-succeeded, failed, warning, and running rollups as the primary -monitoring status so an item with an old successful point does not scan as -healthy when it still needs operator attention. -That same workspace contract also keeps Pulse's provider-neutral recovery model -explicit in the page language: recovery sections should talk about protected -items, recovery events, and latest points so PBS backups, TrueNAS snapshots, -Kubernetes artifacts, and future providers all fit the same first-class UI -frame without removing the source badges and row-level cues that make Proxmox -operators productive. +No aggregate Recovery page exists. Its composition root +`frontend-modern/src/components/Recovery/Recovery.tsx`, the section owners +beside it (`RecoveryActivitySection.tsx`, `RecoveryHistorySection.tsx`, +`RecoveryHistoryTable.tsx`, `RecoveryProtectedInventorySection.tsx`, +`RecoveryPointDetails.tsx`, `useRecoveryHistorySectionState.ts`), and its +route-state owner `frontend-modern/src/features/recovery/useRecoverySurfaceState.ts` +were deleted on 2026-05-26, and the event-first workspace, the secondary +protection-coverage review, and the `view=inventory` / `view=events` route +state went with them. Recovery rows render only in platform-owned tabs: the +Proxmox Backups tab, whose `/proxmox/backups/date` and +`/proxmox/backups/coverage` views are described under "Proxmox backup views +use canonical workflow routes", and the TrueNAS Protection tab, whose +`TrueNASProtectionTable.tsx` lists the points `useRecoveryPoints` returns with +the `truenas` platform forced. Bringing back a cross-platform recovery page is +a product decision that needs its own governed contract, not an extension of +either tab. License recovery stays reachable through the direct Plans & Billing route. `SelfHostedCommercialRecoverySection` inside `ProLicensePanel.tsx` remains available for explicit activation and recovery even when the ordinary free self-hosted navigation hides commercial surfaces. Recovery flows must not acquire proactive commercial prompts of their own. -Operator-facing filter and detail labels should likewise prefer `platform` -wording over implementation-facing `provider` wording, so the recovery surface -describes the monitored platform families Pulse covers rather than exposing -backend transport vocabulary as the primary UI model. -That same operator-facing vocabulary should also prefer `item` over backend -`subject` wording, and `platform` over generic `source` wording, across the -primary recovery headers, tables, filter controls, and detail metadata labels. -The data model can keep its internal subject/provider fields, but the page -frame that operators read should present one consistent protected-item and -platform model from summary through drill-in. Shared recovery URLs and -transport filters should likewise treat `platform` as the canonical -operator-facing query field, with legacy `provider` aliases accepted only as -compatibility input that rewrites back to canonical `platform` route state. -Shared recovery link builders should therefore accept canonical `platform` -inputs only; legacy `provider` belongs at parse-time compatibility boundaries, -not in new caller-facing recovery route helpers. -Cross-surface recovery drill-in links must also target the correct primary -workspace without relying on legacy inventory-first defaults. When a platform -service surface such as PBS links into recovery activity, that shared entry -point should land on the default recovery events workspace and describe the -destination as recovery events rather than reverting to PBS-backup wording. +Operator-facing recovery labels prefer `platform` over implementation-facing +`provider` and `item` over backend `subject`; the transport and data model may +keep subject/provider fields only as compatibility aliases. No shared recovery +query state exists any more (the Proxmox Backups tab's own query is its +`PROXMOX_BACKUPS_QUERY_PARAMS`), so a future cross-platform recovery filter or +link must treat `platform` as its canonical query field and accept legacy +`provider` only as parse-time compatibility input. That same recovery contract should keep response payloads canonical as well: recovery points and protected rollups should expose `platform` and `platforms` as the primary transport fields, while any legacy -`provider` / `providers` aliases stay compatibility-only so the page does not -silently drift back to backend-shaped vocabulary during decode. -That same shared recovery table contract should keep its runtime column model -canonical as well. Recovery inventory and event-history columns should use -`item` and `platform` identities rather than preserving `subject` and `source` -as the primary runtime model, and any saved legacy column IDs must migrate at -the shared column-visibility boundary instead of forcing recovery renderers to -carry deleted column identities indefinitely. Once that migration exists, -recovery tables and shared table presenters should not continue accepting -legacy `subject` and `source` ids in the live runtime path. -That same runtime-helper contract should prefer `item` terminology in shared -recovery presenters too. Helper exports that resolve labels or item-type badges -should expose canonical item-facing names, while any retained `subject` aliases -remain compatibility wrappers instead of the primary runtime boundary. -That same shared badge contract applies to table rendering too. Recovery item -type cells should use the same compact monitoring-table badge base that -workloads uses for `VM` and `Container`, rather than copying only the colors -and drifting on padding or visual weight. -The same rule applies inside recovery-owned helpers and selectors. Shared -summary helpers and platform filter renderers should use canonical `item` and -`platform` naming internally once compatibility boundaries already exist, -rather than keeping fresh `subject` or `provider` terminology alive in the -live recovery runtime path. -The same runtime vocabulary rule applies to cross-section recovery props too. -Live page-to-section boundaries should carry item-focused names like -`selectedHistoryItemLabel` instead of preserving `subject` labels after the -shared recovery presenters already expose canonical item terminology. -That same rule applies to recovery detail helpers. Provider-specific helper -names like `isPbsProvider` should become platform-specific helpers like -`isPbsPlatform` once the runtime recovery model is already canonically -platform-first. +`provider` / `providers` aliases stay compatibility-only so frontend decode +does not silently drift back to backend-shaped vocabulary. The same canonical boundary applies to linked-resource identifiers. Recovery API payloads, query filters, and normalized frontend runtime models should use `itemResourceId` as the canonical field while accepting or emitting @@ -4572,11 +4556,6 @@ That same canonical boundary also applies to external item references. Recovery API payloads and normalized frontend runtime models should use `itemRef` as the canonical item-reference field while treating `subjectRef` only as a compatibility alias during the transition. -That same presenter boundary should also own canonical item-type derivation. -Recovery surfaces must resolve rollup and point item types through one shared -item-type helper instead of repeating `display.itemType` / `subjectType` / -`subjectRef.type` fallback chains across state, summary, details, and table -renderers. That same recovery-store decode boundary must fail soft on malformed persisted metadata. If a stored recovery row contains bad `subject_ref_json`, `repository_ref_json`, or `details_json`, the list endpoints should log and @@ -4626,60 +4605,33 @@ derivation — sweeping protection-posture rows stranded under retired subject keys — so a live migration or a canonical-ID era change never forks one guest's protected item into stale and fresh rows. Rows without a recomputable guest identity keep their stored keys unchanged. -That same hook-boundary normalization also owns the runtime recovery display -model. Canonical recovery points and rollups must expose `display.itemLabel` -and `display.itemType` to recovery consumers, while legacy transport fields -such as `subjectLabel` and `subjectType` remain decode-only compatibility -aliases in the shared normalization layer instead of leaking into runtime -presenters. +The frontend decode boundary owns the runtime recovery display model: +`frontend-modern/src/utils/recoveryPlatformModel.ts`, which +`frontend-modern/src/hooks/useRecoveryPoints.ts` applies to every response, +exposes `display.itemLabel` and `display.itemType` to recovery consumers, while +legacy transport fields such as `subjectLabel` and `subjectType` remain +decode-only compatibility aliases in that normalization layer instead of +leaking into tables. That same canonical item-label boundary must prefer recognizable protected-item names over raw entity IDs. When unresolved Proxmox-backed recovery points only have a VMID/CTID in the subject ref but still carry a richer backup comment or notes label, the canonical recovery index and store backfill must promote that human-readable label into the persisted subject/item label instead of leaving protected inventory rows to lead with bare numeric IDs. -That same operator-facing row-identity rule should still preserve the governed -entity identifier as secondary context when it exists. Recovery inventory and -event rows should lead with the canonical item name, then show a muted -secondary compact `VMID`/`CTID`/`ID` cue when `display.entityIdLabel` is -available, so operators can disambiguate familiar names without turning the -primary scan path back into raw numeric identifiers or bloating the table with -an extra recovery-only identity row. -That same shared presentation layer also owns the distinction between -aggregate recovery-method language and single-record recovery-method language. -Timeline legends and daily breakdowns must use aggregate labels such as -`Snapshots`, `Local Copies`, and `Remote Copies`, while event rows, filters, -and point details must use the singular operator-facing forms `Snapshot`, -`Local Copy`, and `Remote Copy`. Recovery point detail summaries must also -humanize backend fields like kind, mode, outcome, and boolean state into -operator-facing labels such as `Point Type`, `Method`, `Outcome`, `Verified`, -and `Encrypted` instead of leaking raw transport values like `backup`, -`remote`, or lowercase outcome tokens into the primary drawer surface. -That primary workspace selection now also lives in canonical recovery route -state through `frontend-modern/src/routing/resourceLinks.ts` and -`frontend-modern/src/features/recovery/useRecoverySurfaceState.ts`, so copied -links and browser restores reopen explicit protection-coverage state instead -of silently falling back to page-local UI state. Focused recovery routes with -an active `rollupId` or `day` remain recovery-events-first by default, and -default event routes should omit redundant `view=events` query state. -That same shared route-helper contract now also has to preserve exact storage -and recovery handoffs for unified resources discovered outside the storage or -recovery pages. When alerts, Patrol, or infrastructure drawers route a -TrueNAS-backed disk, app, or system into an owning platform/runtime surface, -the shared helper must keep the owned `source`, `node`, `platform`, `view`, -and exact `resource` semantics intact instead of collapsing those handoffs -back to provider-local URLs, retired aggregate workspace routes, or generic -top-level tabs. -That history table layout now also derives its minimum width from the same -canonical column-width spec that owns the header sizing in -`frontend-modern/src/utils/recoveryTablePresentation.ts`, so longer governed -subject labels do not force the trailing outcome/status columns off-screen by -budget drift. -That same recovery product proof surface now also includes a browser-level -desktop layout guard in `tests/integration/tests/17-recovery-layout.spec.ts`, -which opens the recovery page against deterministic recovery payloads and -fails when the history table needs horizontal scrolling or lets the outcome -column drift outside the visible wrapper at desktop width. +Alerts, Patrol, and resource drawers hand no resources into storage or +recovery surfaces today: their cross-jump chips were retired on 2026-05-16, +and `buildStorageRouteSearch()` is read only by the Storage surface's own +filter state. A future handoff of a TrueNAS-backed disk, app, or system into +an owning platform tab must keep the owned `source`, `node`, `platform`, +`view`, and exact `resource` semantics instead of collapsing to provider-local +URLs, retired aggregate workspace routes, or generic top-level tabs. +The recovery product proof surface's browser-level layout guard is +`tests/integration/tests/17-proxmox-backups-layout.spec.ts`. Against the +mock-mode dataset it checks that selecting an activity day keeps the operator +on `/proxmox/backups` with an `N of M backups` count, and that a one-year +activity range keeps the page inside the horizontal viewport with the PBS +servers table's trailing column inside its wrapper; the activity chart may +scroll inside its own card. That same shared `internal/api/` dependency now also assumes tenant-scoped resource handlers seed registries from canonical unified resources only: recovery- and storage-adjacent API helpers may not fall back to raw tenant @@ -4865,383 +4817,72 @@ The storage page's readiness now stays route-owned as well: loading, reconnect, and disconnect presentation from the storage unified-resource fetch contract before consulting websocket churn so the storage surface does not present healthy REST-backed data as down or stale. -Meanwhile, -`frontend-modern/src/components/Recovery/` and the recovery hooks define the -event timeline, protection-coverage review, and recovery-history UX. The -governed page frame is event-first: operators land directly on concrete -backup/snapshot/replication history, with `RecoveryActivitySection.tsx` acting -as compact orientation for the visible event table. The former decorative -recovery summary-card strip is retired because workloads already owns the -object-level "has a backup" scan, while Recovery owns concrete recovery events -and an explicit secondary protection-coverage review. -That top recovery frame must rely on solid elevated operator panels and border -hierarchy rather than decorative gradients so the page reads like a monitoring -workspace instead of a marketing-style dashboard shell. The page must stay -compact enough to keep the activity chart and primary event table in the first -scroll window instead of stacking dashboard-like slabs above the work surface. -Protection coverage remains a secondary review, opened through the explicit -header action or canonical compatibility route state when the operator needs to -audit stale, failed, warning, running, unknown, or never-succeeded items. It -must expose a reciprocal page-header action back to Recovery events, so opening -coverage never leaves the operator dependent on the global Recovery nav item or -small table chrome to return to the primary page. It must not return as an -equal-weight subtab or a top-level posture-card action strip without a separate -governed product decision. -Recovery is also intentionally outside the interactive page/group/entity -summary-scope contract that the workloads and storage tables use. The recovery -route must not adopt `summaryCardInteraction.ts`, `summaryTableFocus.ts`, -row-driven summary scope, or a summary card strip simply because adjacent -monitoring pages use those primitives. Recovery may still show -coverage breadth and platform context inside the Protection coverage table and -event filters, but the page must read item-first so the unified recovery model -is not visually anchored to one platform family. -That same item-first rule also applies to the protection coverage table: -`RecoveryProtectedInventorySection.tsx` must surface protected item type as a -first-class column in the main inventory grid rather than leaving platform as -the only structural classifier beside the item label. Platform badges remain -important supporting operator context, especially for Proxmox-heavy fleets, -but the table frame itself must make protected item class explicit. -That same inventory contract must keep the protected-items grid operationally -bounded. The governed desktop recovery surface should not dump the entire -protected estate into one endless slab; it should page or otherwise bound the -primary inventory table so the workspace, filters, and adjacent activity panel -remain readable as one monitoring surface instead of dissolving into a raw list -dump. -That same protected-inventory surface should carry compact operator orientation -inside the table shell itself. `RecoveryProtectedInventorySection.tsx` should -expose the current bounded range, page, and sort state near the primary grid, -and the first column should carry enough secondary item metadata to read as a -monitored inventory row rather than a bare export line. -That same hierarchy rule also applies to the activity timeline. The governed -recovery surface should not append `RecoveryActivitySection.tsx` underneath the -default protected-items view as if trend telemetry were a second page bolted -onto inventory. The timeline owns recovery-event day selection, so it belongs -inside the `Recovery events` workspace and should read as history analysis for -the selected window rather than as a second copy of the page-level posture -summary. -The same owned vocabulary applies to recovery events as well: -`frontend-modern/src/utils/recoveryTablePresentation.ts` must keep the -history-table `type` column labeled as `Item Type` within recovery surfaces so -event history does not fall back to a generic shared `Type` header once the -recovery lane has already established item-first operator vocabulary. -That same item-first vocabulary must carry through the point-details drawer: -when a recovery point includes canonical item-class metadata, -`RecoveryPointDetails.tsx` must surface it as `Item Type` in the summary grid -instead of jumping directly from item identity to platform and point-method -metadata. -That same shared presentation layer also owns recovery placement vocabulary. -Cluster, node, and namespace facets remain valid supporting filters for -Proxmox-heavy and Kubernetes-heavy operators, but the governed recovery -surface must present them through platform-neutral labels such as -`Cluster / Site`, `Host / Agent`, and `Namespace / Group` across advanced -filters, active chips, table headers, column-picker entries, and point -details so the page treats placement as optional context inside a -multi-platform recovery model rather than a Proxmox-native spine. When -normalized display labels are present, the visible history rows must prefer -those labels over raw transport values for the same placement dimensions. -The recovery table presentation helper now owns the canonical subject-type -label fallback for recovery rows and delegates its title-casing to the shared -`frontend-modern/src/utils/textPresentation.ts` helper rather than keeping a -local recovery-only formatter, so subject and outcome labels stay aligned with -the shared frontend label contract. Protected-inventory and recovery-event -filters, table headers, and column-picker labels must use that helper for -artifact fields such as `Item Type`, so the recovery tabs do not drift into -near-identical page-local casing. -That same recovery drill-in surface now also keeps provider-specific metadata -inside a provider-neutral detail shell through -`frontend-modern/src/components/Recovery/RecoveryPointDetails.tsx`, so PBS -datastore and verification enrichments remain available without presenting the -details drawer as a PBS-only surface. -The point-details drawer also owns restore-safe operator guidance. It may -surface restore readiness, verification provenance, and chain coverage for the -selected point, but it must stay read-side until the backend exposes a governed -restore execution contract. The drawer must not present a freestanding restore -runbook or next-action path that reads as an approved restore workflow; target -confirmation and isolated test-restore planning belong in a future governed -action or restore flow, not in the evidence drawer. Chain context must be -derived from the current recovery result set only when at least two concrete -stages are visible, so mixed PVE/PBS/TrueNAS history can explain adjacent local -snapshot, local copy, and remote copy stages without filling the drawer with -missing-only cards. Raw transport IDs, provider refs, provider task IDs, and -raw JSON copy actions belong behind `Technical details`; the primary drawer -should keep human metadata, recorded verification provenance, target health, -and collapsed file lists without repeating the same verification fact in -provider-specific sections or rendering empty verifier/evidence placeholders -when no verification record exists. Verification provenance should translate -provider states such as PBS catalog `ok` into operator language instead of -surfacing raw transport status tokens. Container recovery points should present -container ids with operator vocabulary such as `CTID`, and duplicated placement -or target values should not be repeated under lower-priority metadata labels. -Provider-specific metadata must not recast the event drawer itself as if PBS -were the native recovery model. Provider-owned repository data should sit under -target-oriented wording such as `Target Details`, `Repository owner`, and -`Target Health`; when target-specific technical labels are surfaced, they -should prefer neutral wording such as `Target Ref` and `Target Resource`. -Those transport hooks are direct governed runtime surfaces, not just page -implementation detail: `frontend-modern/src/hooks/useRecoveryPoints.ts`, -`frontend-modern/src/hooks/useRecoveryPointsFacets.ts`, -`frontend-modern/src/hooks/useRecoveryPointsSeries.ts`, and -`frontend-modern/src/hooks/useRecoveryRollups.ts` must stay on the explicit -`recovery-product-surface` proof path instead of inheriting release-control -coverage only through a retired standalone Recovery page shell. -Those same hooks now also own recovery transport normalization at the frontend -boundary: raw compatibility fields such as `provider` / `providers` may be -accepted from older `/api/recovery/*` payloads, but the runtime values they -return to the rest of the recovery UI must be canonical `platform` / -`platforms` models. +Recovery has no page frame of its own. The event timeline, protection-coverage +review, recovery-history table, point-details drawer, activity strip, and the +recovery table, filter-chip, status, record, action, issue, and empty-state +presenters that served them were deleted with the aggregate Recovery page on +2026-05-26 (its summary strip had gone on 2026-05-14), along with +`useRecoveryRollups.ts`, +`useRecoveryPointsFacets.ts`, and `useRecoveryPointsSeries.ts`. The rules that +governed that frame's layout, density, workspace order, and default columns +have no surface to bind. Recovery evidence now reads through two +platform-owned tabs built on the shared platform-page table primitives: the +Proxmox Backups tab (`ProxmoxBackupsTable.tsx` and its activity chart, date +feed, coverage, recoverable-artifact, and backup-server tables) and the +TrueNAS Protection tab (`TrueNASProtectionTable.tsx`). Neither adopts the +interactive summary-scope primitives (`summaryCardInteraction.ts`, +`summaryTableFocus.ts`) or a summary card strip. +Recovery row details on those tabs stay read-side until the backend exposes a +governed restore execution contract. They may show verification and target +facts for the selected row, but must not present a freestanding restore +runbook or next-action path that reads as an approved restore workflow. +The Proxmox Backups activity chart keeps long ranges contained: +`BackupActivityChart.tsx` sizes the plot through +`getRecoveryTimelineChartMinWidthPx` (2,560 pixels for a year) and scrolls +inside its own card, so the page itself never overflows horizontally; +`tests/integration/tests/17-proxmox-backups-layout.spec.ts` asserts that at +desktop width. +The recovery transport hooks are direct governed runtime surfaces, not page +implementation detail: `frontend-modern/src/hooks/useRecoveryPoints.ts` and +`frontend-modern/src/hooks/useProtectionPostures.ts` stay on the explicit +`recovery-product-surface` proof path, as do the shared recovery helpers that +survived the page: `frontend-modern/src/types/recovery.ts`, +`frontend-modern/src/utils/recoveryDatePresentation.ts`, +`frontend-modern/src/utils/recoveryOutcomePresentation.ts`, +`frontend-modern/src/utils/recoveryArtifactModePresentation.ts`, +`frontend-modern/src/utils/recoveryTimelinePresentation.ts`, and +`frontend-modern/src/utils/recoveryTimelineChartPresentation.ts`. They must not +inherit release-control coverage only through the platform tabs that render +them. +`useRecoveryPoints.ts` also owns recovery transport normalization at the +frontend boundary: raw compatibility fields such as `provider` may still +arrive from older `/api/recovery/points` payloads, but the points it returns +are canonical `platform` models. The retired dashboard recovery and storage entry points must stay removed: `useDashboardRecovery`, `DashboardRecoveryStatusPanel`, `DashboardStoragePanel`, dashboard storage/recovery presentation helpers, and dashboard widget orchestration must not return as direct proof surfaces. New -storage or recovery summary proof must attach to the owning Storage, -Recovery, Infrastructure drawer, or shared summary component instead of -borrowing coverage through a broader dashboard shell. -The shared recovery type contract must be pinned the same way: -`frontend-modern/src/types/recovery.ts` must stay on the explicit -`recovery-product-surface` proof path instead of riding indirectly on route or -component coverage. -That same direct proof rule applies to the shared recovery date helper: -`frontend-modern/src/utils/recoveryDatePresentation.ts` must stay on the -explicit `recovery-product-surface` proof path instead of inheriting coverage -only through pages or higher-level recovery components. -That same direct proof rule also applies to the shared recovery status helper: -`frontend-modern/src/utils/recoveryStatusPresentation.ts` must stay on the -explicit `recovery-product-surface` proof path instead of inheriting coverage -only through pages or higher-level recovery components. -The default protected-inventory recovery route must also keep its primary -table shell on class-driven sizing (`table-fixed` plus owned width classes) -instead of inline `table-layout` / `min-width` styles, so the public recovery -surface stays CSP-safe without drifting from the shared table contract. -That same direct proof rule also applies to the shared recovery record helper: -`frontend-modern/src/utils/recoveryRecordPresentation.ts` must stay on the -explicit `recovery-product-surface` proof path instead of inheriting coverage -only through pages or higher-level recovery components. -That shared recovery record contract now also includes rollup-side display -payload continuity: the recovery backend must preserve the latest normalized -subject label on rollups, and recovery UI helpers must prefer that canonical -display label before raw subject ids whenever the live unified-resource map is -missing or only resolves to opaque machine identifiers. -That same direct proof rule also applies to the shared recovery outcome helper: -`frontend-modern/src/utils/recoveryOutcomePresentation.ts` must stay on the -explicit `recovery-product-surface` proof path instead of inheriting coverage -only through pages or higher-level recovery components. -That same direct proof rule also applies to the shared recovery action helper: -`frontend-modern/src/utils/recoveryActionPresentation.ts` must stay on the -explicit `recovery-product-surface` proof path instead of inheriting coverage -only through pages or higher-level recovery components. -That same direct proof rule also applies to the shared recovery artifact mode -helper: `frontend-modern/src/utils/recoveryArtifactModePresentation.ts` must -stay on the explicit `recovery-product-surface` proof path instead of -inheriting coverage only through pages or higher-level recovery components. -That same direct proof rule also applies to the shared recovery empty-state -helper: `frontend-modern/src/utils/recoveryEmptyStatePresentation.ts` must stay -on the explicit `recovery-product-surface` proof path instead of inheriting -coverage only through pages or higher-level recovery components. -That same direct proof rule also applies to the shared recovery filter-chip -helper: `frontend-modern/src/utils/recoveryFilterChipPresentation.ts` must stay -on the explicit `recovery-product-surface` proof path instead of inheriting -coverage only through pages or higher-level recovery components. -That same direct proof rule also applies to the shared recovery issue helper: -`frontend-modern/src/utils/recoveryIssuePresentation.ts` must stay on the -explicit `recovery-product-surface` proof path instead of inheriting coverage -only through pages or higher-level recovery components. -That same direct proof rule also applies to the shared recovery table helper: -`frontend-modern/src/utils/recoveryTablePresentation.ts` must stay on the -explicit `recovery-product-surface` proof path instead of inheriting coverage -only through pages or higher-level recovery components. -That same direct proof rule also applies to the shared recovery timeline-chart -helper: `frontend-modern/src/utils/recoveryTimelineChartPresentation.ts` must -stay on the explicit `recovery-product-surface` proof path instead of -inheriting coverage only through pages or higher-level recovery components. -That same direct proof rule also applies to the shared recovery timeline -helper: `frontend-modern/src/utils/recoveryTimelinePresentation.ts` must stay -on the explicit `recovery-product-surface` proof path instead of inheriting -coverage only through pages or higher-level recovery components. - -Those recovery transport surfaces now also share one normalized filter -contract: protection rollups, point history, facets, and chart series must -all honor the same canonical `platform`, canonical `itemType`, cluster, node, namespace, -workload-scope, verification, and route-backed free-text `q` filter so the -protection coverage list cannot drift from the timeline and facet state under the -same active recovery view. That same recovery filter contract now depends on -the canonical recovery index carrying a normalized `itemType` instead of -forcing each UI surface to re-derive protected item classes from raw +storage or recovery summary proof must attach to the owning Storage surface, +platform tab, or shared summary component instead of borrowing coverage +through a broader dashboard shell. +The recovery backend preserves rollup-side display payload continuity: it keeps +the latest normalized subject label on rollups, so a reader can prefer that +canonical display label over raw subject ids whenever the live +unified-resource map is missing or only resolves to opaque machine +identifiers. +The `/api/recovery/*` transports share one normalized filter contract: +protection rollups, point history, facets, and chart series all honor the same +canonical `platform`, canonical `itemType`, cluster, node, namespace, +workload-scope, verification, and free-text `q` filters, so any two of them +read under the same filters describe the same items. That filter contract +depends on the canonical recovery index carrying a normalized `itemType` +instead of forcing each reader to re-derive protected item classes from raw provider-native `subjectType` values. -That same recovery product surface keeps the primary workspace visually ahead -of secondary analytics: `frontend-modern/src/components/Recovery/Recovery.tsx` -must lead directly into the route-backed recovery events workspace, with -recovery events owning both the activity timeline and the event table together. -Protection coverage opens only through the explicit header action or -compatibility route state. The activity timeline remains required even when -point-history loading fails, but it belongs to the events workspace rather than -hanging beside coverage as a competing page-level peer. -That same recovery product surface must not reintroduce a four-card posture, -freshness, coverage, or activity strip as decorative orientation. Workloads is -the better object-level place to scan whether a resource has backup coverage; -Recovery should reserve its first viewport for concrete recovery history and -only expose coverage rollups when the operator asks for the secondary review. -That same page-level ownership applies to the recovery time window. The -canonical range selector now lives inside `RecoveryActivitySection.tsx` because -it controls the event timeline and the event query window, not a page-level KPI -strip. Range changes must clear selected-day focus and return the events table -to page one through the route-state owner so the chart and table remain aligned. -That same activity-section contract should stay compact and scan-first: -recovery-point volume, active-day count, stale count, and average rate may -appear as concise inline readouts beside the chart controls, but not as separate -top-level metric cards or nested mini-panels. The activity strip must also avoid -interpreted anomaly callouts such as "lowest active day" unless the recovery -model has canonical schedule expectations that make the signal actionable. -Item-type labels should continue to render through canonical workload/resource -badge classes in the event and coverage tables instead of adding recovery-only -wrapper chrome around VM, container, or other resource badges. -That same event-first shell rule should hand straight from the activity section -into the active events workspace without an extra page-local spacer band, -default tab row, or duplicate status strip that makes the surface softer than -storage or workloads. -That same scan-first rule applies to the secondary coverage surface. Recovery -should not show equal workspace tab labels and then repeat the same workspace -count as standalone text; protection coverage cues should focus on issues, -drill-in context, and active filters. -That same coverage surface should also follow the established monitoring-table -scan pattern in its first column. Coverage rows should lead with a clear -status cue, the primary item name, and compact badge-backed item/platform -context instead of relying on recovery-only rails or plain-text metadata lines -that make the table read like a report instead of an operational grid. -That same triage rule applies to the coverage sort. Protection coverage should -open with attention-state rollups first instead of defaulting to -newest-successful backups, so operators land on failed, never-succeeded, stale, -warning, and running items before the healthy catalog. -That same row contract should avoid duplicating context that already has a -dedicated column. When `Item Type` and `Platform` columns are visible, the -primary item cell should not restate those same badges on desktop; duplicate -context belongs only as a small-screen fallback when those columns collapse. -That same scan rule applies to the supporting columns themselves. Recovery -tables should keep one dominant identity cue and one canonical platform cue. -`Item Type` should use the same shared workload/resource badge treatment that -other Pulse tables use for `VM` and `Container`, while `Method` and similar -supporting fields stay on restrained metadata text instead of turning every -adjacent column into another colored badge. -That same item-identity contract also applies to synthetic Proxmox task -recovery points. When the persisted subject label is just a raw -`pve-task:*`/`UPID:*` identifier or `vmid=0`, the canonical recovery index -should derive a readable task label and `task` item type from point details so -recovery tables scan by operator meaning instead of transport IDs. -That same coverage surface should stay on the flat monitoring-table pattern -already used elsewhere in Pulse. Protection coverage should surface posture through -row-level status cues, outcome pills, and filters rather than inserting extra -`Needs Attention` / `Healthy Coverage` section rows that add height and turn -the table into a recovery-only grouped report. -That same protection-coverage table should also avoid recovery-local pagination -chrome. The workspace already holds the filtered rollups client-side, so it -should read as one continuous monitoring table with a simple coverage-item -count instead of introducing `Prev` / `Next` buttons and page counters that do -not match the canonical Pulse scan pattern. -That same table-shell contract must avoid duplicate framing once the summary -action has established the secondary coverage workspace. -`RecoveryProtectedInventorySection.tsx` should keep page/count/sort -orientation inside a slim table-shell status row and let the filter strip lead -directly into the grid instead of reintroducing a second large inventory header -card above the same table. -That same shell rule should also avoid low-signal bookkeeping above the grid. -The protected-items status row should surface the active workspace, protected -item count, and issue cues, but page-number and sort-direction bookkeeping -belongs in the table chrome itself rather than competing with the primary scan -path before operators even reach the rows. -That same table density rule also applies to recovery table chrome and filter -rows. Recovery inventory and event tables should use the same restrained -title-case header typography, compact control heights, and thin row density as -the established Pulse monitoring tables instead of drifting into report-style -uppercase headers or oversized filter chrome. -That same protected-items table contract should stay on the canonical shared -table separator treatment used by the rest of Pulse. Recovery inventory should -use the standard shared header/body dividers and avoid both local suppression -of those separators and local duplicate row or header borders that make the -lines read heavier than other monitoring tables. -That same workspace-shell rule should also avoid a dedicated recovery-only -status strip above the control bar. Recovery should use an event-first handoff: -activity context, a shared controls card, then a data card. Protection coverage -may use its own shared controls card and table when explicitly opened, but -Recovery should not collapse controls and content back into one fused workspace -slab or bury secondary workspace navigation inside the filter row. -That same strip should not repeat page-level counts or posture cues as a -replacement for the retired summary strip. Protection coverage controls should -stay focused on drill-in context and active filters instead of echoing page-wide -posture pills above the same table. -That same workspace handoff should stay on shared primitive styling too. -Recovery events and protection coverage should keep using shared `FilterBar`, -`TableCard`, and `TableCardHeader` primitives instead of inventing a -recovery-only variant or recovery-only class stack. -That same canonical-row rule also means the subtabs row should stand on its own -full-width shell instead of sharing a flex line with recovery-only chips or -adjacent badges that break the storage-style border and spacing treatment. -That same shared page-controls contract applies to recovery search width too. -The protected-items and recovery-events workspaces should keep the search field -on the standard full-width shared search row, and any counts or utility cues -should live in the toolbar actions instead of narrowing the search row through -recovery-local grid overrides or width hacks. Protected-items controls should -also use the same shared `Reset all` page-controls action pattern as storage -and workloads when visible filters are active, instead of forcing operators to -clear each inventory filter manually. -That same handoff should keep Recovery free of a first-viewport summary card -strip unless a new governed product decision reintroduces a first-viewport -summary owner. The current route should spend its top-level density budget on -the activity section, compact controls, and one primary data card. -That same shell rule applies to the recovery-events workspace. -`RecoveryHistorySection.tsx` should use the same slim status-row-plus-filter-row -pattern as the protected inventory surface, not a separate large titled header -bar plus another full toolbar slab. Event filter labels should also stay on the -canonical short Pulse vocabulary like `Platform` and `Status` instead of -recovery-only variants such as `History platform` or `History status`. Both -recovery toolbars should also stay on compact shared select sizing instead of -inflating the row with recovery-local min-width overrides that make the -controls denser and wider than storage for the same amount of operator input. -That same events-workspace rule should keep the activity strip as orientation -for the event list rather than burying it at the bottom. The events workspace -should move from the subtabs row to `RecoveryActivitySection.tsx`, then shared -controls, then the recovery history table as sibling sections, so the timeline -frames the event list without turning the page back into stacked primary -tables or embedding the activity strip inside the history card. -That same events-shell contract should avoid repeating page-state bookkeeping -ahead of the history grid. Recovery events should keep the toolbar utility area -focused on actual controls like advanced filters and column visibility instead -of passive `day groups` narration; day grouping should stay legible through the -history surface itself, while current page and other table bookkeeping remain -in the table footer instead of competing with the scan path above the filters. -That same activity panel should stay compact and analytical rather than -becoming a second dashboard header. `RecoveryActivitySection.tsx` should keep a -single slim telemetry header, compact active-filter chips, a shorter chart -frame, reduced vertical insets, and a smaller legend footprint so the events -workspace hands off quickly from activity context to the history table instead -of spending a disproportionate slice of the screen on chart chrome. The range -picker and legend should share one compact control row, and the activity strip -should not burn a separate descriptive subtitle row once the headline metrics -already explain the chart context. -That same timeline contract must keep long-range activity fully constrained to -the card width. Extended ranges such as `365d` should compress their day -columns to fit the available plot width instead of carrying per-column minimum -widths that make the chart overflow its containing card. -That telemetry header should also avoid derivative pace rows once the chart -already carries the rhythm. Total points, active days, and issue cues can -stay, but average-per-day style readouts should not re-expand the strip into a -second mini report above the event table. -That same events-table contract should also keep the default column set on a -monitoring-style scan path rather than a report-export path. Recovery events -should default to the concise columns operators need to triage quickly, while -secondary fields such as verification, size, target, and details remain -available through the shared column picker instead of crowding the baseline -desktop view. When the responsive event table collapses columns on mobile, the -primary item cell must retain enough method, platform, target, and verification -context to preserve restore-readiness meaning rather than hiding all secondary -evidence behind desktop-only columns in the desktop grid. -That same event-row scan rule should mirror the protected-items table in the -primary identity cell. `RecoveryHistoryTable.tsx` should lead each event row -with a compact outcome status cue plus the canonical item name, so operators -can scan event health by row without relying only on the far-right outcome -column. -That same density rule should also keep history grouping and badges restrained. -Day-group headers should read as slim dividers instead of banner rows, and -platform/method/outcome pills should stay compact enough that the event grid -still scans like a monitoring table rather than a report export. +The same item-identity contract applies to synthetic Proxmox task recovery +points. When the persisted subject label is just a raw `pve-task:*`/`UPID:*` +identifier or `vmid=0`, the canonical recovery index should derive a readable +task label and `task` item type from point details so recovery rows scan by +operator meaning instead of transport IDs. That shared unified-resource dependency now also includes policy-governed resource metadata on the frontend decode path: storage and recovery surfaces that route through `frontend-modern/src/hooks/useUnifiedResources.ts` must @@ -5797,29 +5438,12 @@ and proof surface describe one /api/auto-register path instead of a fake /api/auto-register/secure sibling, so adjacent transport and governed evidence do not drift onto a route split that the runtime does not actually expose. -That same filter contract applies to the advanced history facets transport as a -whole: changing node or namespace filters must narrow the facets request too, -so node and namespace option sets cannot drift back to the broader chart window -while the visible history table is already scoped to a smaller recovery slice. -That same narrowing rule now also applies when a timeline day is selected: the -facets request must use the same narrowed day window as the points request so -node and namespace option sets stay coherent with the visible history slice -instead of showing options from the full chart range while the table is already -scoped to a single day. -The recovery timeline drill-down now also treats day selection as a real -history transport boundary: choosing a day in the "Backups By Date" chart must -narrow the point-history request window to that selected local day rather than -only updating local selection chrome while the table remains on the broader -chart window. -That selected-day boundary must also be durable route state: the recovery URL -must preserve the active timeline day so reload, navigation, and shared links -reconstruct the same point-history window instead of silently widening back to -the broader chart range. -That same route continuity rule also applies to the selected chart window -itself: changing the recovery timeline range to `7d`, `90d`, or `1y` must stay -in canonical recovery route state so reload, navigation, and shared links -reconstruct the same rollup and series transport window instead of widening -back to the default `30d` range. +The facet, point-window, and range route-state rules that served the deleted +Recovery page's history table no longer have a reader. Day and range selection +now belong to the Proxmox Backups activity chart: choosing a day filters the +date feed in place and writes `PROXMOX_BACKUPS_QUERY_PARAMS.day`, so reload and +shared links reopen the same day, while the chart range (`7d`, `30d`, `90d`, +`1y`) is component state that starts at 30 days and is not written to the URL. This lane intentionally depends on other governed boundaries instead of overreaching into them. API transport and payload contract ownership remain in @@ -5989,16 +5613,19 @@ settings, but they must not reinterpret provider auth, provider connection, or model-selection causes as recovery-source health, backup readiness, or storage-control capability. That bounded projection is the current TrueNAS floor for storage and recovery: -operators can inspect TrueNAS pools, datasets, disks, snapshots, and -replication artifacts through the shared storage and recovery pages plus -cross-surface handoffs. Storage and recovery do not promise a TrueNAS-local -onboarding path, restore/control plane, or separate diagnostic transport; -backend-native app actions, logs, and config reads stay on the adjacent -AI/runtime path and only feed refreshed canonical recovery/state afterward. -The TrueNAS platform page may embed that same canonical recovery surface as a -scoped Protection tab, but it must keep the `truenas` platform filter forced, -reuse recovery-owned protection/event workspace state, and avoid growing a -TrueNAS-only snapshot or replication table contract. +operators can inspect TrueNAS pools, datasets, and disks on the TrueNAS +page's Storage tab (`TrueNASStorageTopologyTable`) and snapshots and +replication artifacts on its Protection tab. Storage and recovery do not +promise a TrueNAS-local onboarding path, restore/control plane, or separate +diagnostic transport; backend-native app actions, logs, and config reads stay +on the adjacent AI/runtime path and only feed refreshed canonical +recovery/state afterward. +The Protection tab is a platform-owned table, not an embedded recovery +surface: `TrueNASProtectionTable.tsx` lists the recovery points +`useRecoveryPoints` returns with the `truenas` platform filter forced. It must +keep that filter forced and keep reading the canonical recovery-points +transport rather than growing a TrueNAS-only snapshot or replication data +contract. VMware vSphere is the current admitted narrower phase-1 direction. Storage and recovery may consume vCenter-backed datastore inventory plus VM snapshot-tree visibility as shared storage and @@ -6077,10 +5704,9 @@ collector, agent-id/device fallback stream, or separate "real-time" history store once monitoring and `/api/metrics-store/history` already own the disk timeline. The same shell/runtime split now applies to websocket consumers: -`frontend-modern/src/components/Recovery/RecoveryPointDetails.tsx` and `frontend-modern/src/components/Storage/useStoragePageResources.ts` may consume websocket state only through `frontend-modern/src/contexts/appRuntime.ts`. -They must not import `@/App` or create storage/recovery-local shell coupling, +It must not import `@/App` or create storage/recovery-local shell coupling, because provider placement remains app-shell-owned and storage/recovery surfaces must stay lazy-load safe. The Storage page also keeps large-estate hydration and rendering bounded at @@ -6258,6 +5884,19 @@ is in `applyLicensedFeatureConfigSnapshot` (`pkg/server/telemetry_licensed_features.go`), pinned by `TestApplyLicensedFeatureConfigSnapshot_CountsScheduledReportingAndProfiles`. +### Single-resource report AI narration requires POST + +`internal/api/metrics_reporting_handlers.go` now accepts `POST` with a JSON +body for `/api/admin/reports/generate` and lets only that method use the +tenant's AI narrator; `GET` keeps working with query parameters and the +deterministic summary. A narrated PDF records the same AI cost-ledger entry as +before, and only a `POST` can now produce one. A `GET` reads Patrol findings +only from an AI service that is already running +(`AISettingsHandler.ExistingAIService`), so it no longer constructs a tenant AI +service, whose construction opens that tenant's SQLite resource export store. +Report content, backup and recovery enrichment, and tenant storage are +otherwise unchanged. + ### Audit-read activity is a bounded, content-free local history `audit_read_activity.json` follows the same shape as the existing external-agent @@ -6487,6 +6126,19 @@ no backup, snapshot, restore, retention, cleanup, or verification authority. The storage/recovery resource query and its admission, freshness, and persistence contracts are unchanged. +### Saved-host link holds open no recovery path + +`internal/unifiedresources/types.go` gains an unexported +`Resource.continuityOnly`: it marks a row saved-host continuity introduced (a +machine that has not reported since a restart), so an operator link holds the +row beside its primary instead of merging it, and, unless the saved row is +itself the link's effective primary (an agent linked into its guest makes the +guest primary), reference reads by its IDs resolve to the primary. It is +in-memory only, never serialized, persisted or exposed on the API, and storage +and recovery code neither reads it nor resolves references through the +registry, so recovery ownership, restore targeting and protection grouping are +unchanged. + ### Token creation and deletion preserve restart-time persistence truth The shared `internal/api` token creation path treats its persisted inventory as @@ -6878,6 +6530,44 @@ The performance report and reporting runtime snapshot handlers apply the same test before tabulating a disk temperature. That changes only which held value a report shows, not any storage or recovery path. +### Safe methods refused on discovery settings and SAML logout + +`internal/api/method_guard.go` adds `requireRequestMethod`, and +`internal/api/discovery_handlers.go` and `internal/api/saml_handlers.go` use it +so `/api/discovery/settings` (`PUT` or `POST`) and `/api/saml/{id}/logout` +(`POST`) answer `405` to any other method, `GET` and `HEAD` included, instead +of changing the discovery staleness threshold or clearing a session. No +storage, retention, backup, migration or recovery path is added or moved. + +### SAML SLO responses bound to the logout that requested them + +`internal/api/saml_service.go` keeps each outstanding SAML LogoutRequest ID in +memory on the provider's service, bound to the session-store hash of the +session being logged out, and honors it for ten minutes (at most 1,024 +records; expired ones are dropped on the next logout). `internal/api/saml_handlers.go` +refuses a LogoutResponse that does not answer one of them. The records are not +persisted, and the SLO handler no longer writes session state. No storage, +retention, backup, migration or recovery path is added or moved. + +### Manual SAML IdP certificate published as base64 DER + +`internal/api/saml_service.go` now encodes a manually configured SAML IdP +signing certificate as base64 DER in the IdP descriptor it rebuilds in memory +on each load. The saved settings are read as before: `idpCertificate` holds +the PEM the administrator pasted and `idpCertFile` the path of a PEM file, so +nothing persisted changes or needs migrating. No storage, retention, backup, +migration or recovery path is added or moved. + +### SAML logins bound to the browser that started them + +`internal/api/saml_service.go` keeps each outstanding SAML AuthnRequest ID in +memory on the provider's service, with the login's return path and the hash of +the browser's `pulse_saml_login` cookie token, and honors it for ten minutes +(at most 1,024 records; expired ones are dropped on the next login). The cookie +expires after ten minutes. Nothing is persisted, so a restart during the IdP +round trip refuses the returning Response and the user signs in again. No +storage, retention, backup, migration or recovery path is added or moved. + ### A linked agent's stale flag and guest disk owner open no storage path `internal/unifiedresources/types.go` now always sends `AgentData.Stale`, so a @@ -6905,3 +6595,13 @@ or recovery path is added or moved. hidden routes, so the demo guard on a `DEMO_MODE` instance answers it with `404`. That removes a diagnostic read surface on demo instances only; no storage, retention, backup, migration or recovery path is added or moved. + +### Report-merge link folds + +`internal/unifiedresources/types.go` adds an unexported, in-memory +`linkFolds` record of the operator links a resource absorbed, and +`internal/api/resourceapi/resources.go` makes report-merge exclude those +links' pairs, which deletes the pair's link row in the resource store as an +unlink does. Nothing new is persisted: the record is rebuilt with each +registry generation, and no schema, backup, retention, migration or +recovery path is added or moved. diff --git a/docs/release-control/v6/internal/subsystems/unified-resources.md b/docs/release-control/v6/internal/subsystems/unified-resources.md index f8cc5c1abe..c284a54e4a 100644 --- a/docs/release-control/v6/internal/subsystems/unified-resources.md +++ b/docs/release-control/v6/internal/subsystems/unified-resources.md @@ -1609,7 +1609,12 @@ Source freshness is a separate unified-resource status input, not a fixed global health timer. Snapshot rebuilds, resource seeding, supplemental record overlays, and cloned read-state overlays must preserve the monitoring-provided stale thresholds for Proxmox, PBS, and PMG sources so resources do not flap to -warning/degraded between successful configured poll cycles. +warning/degraded between successful configured poll cycles. A source merge +into an existing row judges metric freshness by the ingest's thresholds and +the stale pass's own threshold rule, so the merge's freshness gate never +calls a source current that the stale pass marks stale, or the reverse. The +gate decides only where no stronger rule does: a hypervisor-managed guest +keeps its platform CPU even while that source is stale. Service-discovery readiness is a unified-resource payload contract, not a drawer-local decoration. Resource list/detail payloads that expose a @@ -2484,8 +2489,10 @@ signal source instead of advertising empty object browsers. The canonical TrueNAS adapter (`internal/truenas/provider.go:: truenasRecordsFromSnapshot`) already emits the top-level TrueNAS appliance as a unified `agent` row tagged with the `truenas` -platform, so TrueNAS defaults to `/truenas/overview` (the Systems -sub-tab); the embedded `StorageSurface` lives at `/truenas/storage`. +platform, so TrueNAS defaults to `/truenas/overview`, whose Overview +sub-tab lists those systems in `TrueNASSystemsTable`. `/truenas/storage` +renders the TrueNAS-owned `TrueNASStorageTopologyTable`, not +`StorageSurface`; only the Proxmox Storage tab embeds `StorageSurface`. Any future platform that wants to default to a Systems / Hosts overview must first have its canonical resource adapter project the platform's top-level system as a unified resource so the builder @@ -2596,10 +2603,12 @@ application resource-provider or WebSocket lifecycle. in place through that shared runtime and the root app-shell restore path, instead of looking like a page refresh or remount. 12. Keep summary-chart visibility a display preference, not a unified-resource - filter. Platform/runtime pages may hide or restore chart sections through - shared presentation controls, but those controls must not mutate resource - identity, table membership, source scope, or summary-hover state. The - retired top-level `/infrastructure` page, its summary chart strip, and its + filter. The workload and infrastructure summary chart sections are + retired and no platform page exposes a chart show/hide control today. If a + governed product decision brings one back, hiding or restoring it must not + mutate resource identity, table membership, source scope, or summary-hover + state. The retired top-level + `/infrastructure` page, its summary chart strip, and its saved-view/route-state machinery must not be reintroduced for this purpose. 15. Keep operator-local resource search on the operator's own names. Resource-policy redaction is a transmission boundary (`docs/PRIVACY.md`), @@ -2763,7 +2772,7 @@ application resource-provider or WebSocket lifecycle. (`canonical_id` ↔ machine ID / DMI UUID / cluster / hostname) in the `resource_identities` table (`ResourceStore.UpsertResourceIdentityPins` / `ListResourceIdentityPins`, written diff-aware after monitor-adapter - rebuilds) and completes weak incoming identities from those pins + rebuilds; manual-link sides write through `ReplaceResourceIdentityPins`) and completes weak incoming identities from those pins before matching and ID derivation (`internal/unifiedresources/canonical_id_pins.go`). Persisted primary hostnames preserve their normalized dotted form. The pin index resolves @@ -2779,7 +2788,15 @@ application resource-provider or WebSocket lifecycle. must not mint host canonical IDs from snapshot-content-dependent identity subsets without consulting the pins, and pin writes stay on the durable store-backed registry (ephemeral per-request registries - consult, never write). Pinned hostnames preserve the full dotted name + consult, never write). A pin records the identity a resource derives + from its own sources rather than a manual link's merged projection: + each side of a link writes the pin captured before the link merged + them, keeping its row's other fields unless another resource holds that + key, and a pinnable folded side keeps its pin under its own canonical + ID, less any strong key a live resource or an earlier folded side + holds. Capture follows pin completion, so a key an earlier release + borrowed into a side's row can still reach the capture (see "Operator + links reach record-ingested resources"). Pinned hostnames preserve the full dotted name (`NormalizeFullHostname`): distinct machines that share a short hostname (`cloud.rnd-lax1` vs `cloud.gce-or1`) must keep distinct pins, distinct pin-index buckets, and distinct presentation host rows. @@ -2814,7 +2831,9 @@ application resource-provider or WebSocket lifecycle. ID and drops the superseded pin row (`ApplyCanonicalIDSuccessions` in `internal/unifiedresources/canonical_id_succession.go`). Successions never fire while the old ID still belongs to a live - resource, never follow a contradicting machine key (a reinstalled + resource or to one a manual link folded into its primary (see + "Operator links reach record-ingested resources"), never follow a + contradicting machine key (a reinstalled machine mints fresh, it does not absorb the old host's operator state), and never rewrite change-journal rows. Known limitation: machine-keyless hosts whose history already merged under a collapsed @@ -2895,8 +2914,9 @@ application resource-provider or WebSocket lifecycle. `ResourceRegistry.IngestRecords` applies the same `ApplyCanonicalIDSuccessions` semantics as pin-driven successions (operator state and action audits re-key, the superseded pin row - drops, never while the old ID belongs to a live resource, journal - rows never rewritten). Regression coverage: + drops, never while the old ID belongs to a live resource or one a + manual link folded into its primary, journal rows never rewritten). + Regression coverage: `TestRegistryIngestRecordsKeepsSameHostnameSystemsDistinct` / `TestIngestRecordsSucceedLegacyHostnameScopedCanonicalIDs` / `TestIngestRecordsDoNotCompleteTrueNASIdentityFromPins` in @@ -3045,10 +3065,11 @@ canonical WebSocket snapshot, including the public `vmware-vsphere` alias for the raw `vmware` source, rather than degrading a source-scoped page to periodic REST refreshes. VMware Overview passes that same source-scoped snapshot into the embedded Workloads state, so hosts and VMs share one inventory generation -and one explicit refresh path. Beyond disabling the grouped host drawer, it -passes no option that adds host metrics to grouped rows: the shared Workloads -group row carries host identity only, and per-host stats stay in the page's -own hosts table. +and one explicit refresh path. It passes no option that adds host metrics to +grouped rows: the shared Workloads group row carries host identity only, with +no drawer, hover preview, or group pin, and per-host stats and details stay in +the page's own hosts table. The Workloads route model no longer parses a +`summaryGroup` param, because no group focus remains to hydrate. Its page-owned workload toolbar consumes the complete shared `getWorkloadsMetricFilterProps` binding, so vSphere VMs expose the same Bars, Trends, Details, History, range, and first-use discovery contract as every @@ -3193,9 +3214,10 @@ in the same pass. The K8s Namespaces and Deployments drawer "Open Pods" / route to `/kubernetes/workloads` rather than building retired aggregate workload URLs; the legacy context/namespace query filter does not carry forward because the new Kubernetes Workloads tab does not -consume those parameters. Workloads, storage, and recovery route-state helpers -in `frontend-modern/src/routing/resourceLinks.ts` now serialize query state -only; the owning platform/runtime route supplies the pathname. New drawer or +consume those parameters. The workloads and storage route-state helpers in +`frontend-modern/src/routing/resourceLinks.ts` serialize query state only, and +the owning platform/runtime route supplies the pathname; the recovery +serializer was deleted once nothing read it. New drawer or correlation surfaces must anchor on platform routes (or stay-in-place drawer expansion) instead of resurrecting the retired top-level paths. The Proxmox Backups route-state vocabulary includes the opaque `location` @@ -3217,8 +3239,9 @@ recreate raw labelled `{{end}}

Finishing sign-in…

+ +`)) + +// writeSAMLACSRepost answers the IdP's HTTP-POST delivery of a Response that +// arrived without the login cookie. The IdP's page posts to the ACS from +// another site, and browsers withhold SameSite=Lax cookies from cross-site +// POSTs, so the cookie that binds the login to this browser is missing on +// arrival in every browser. The page posts the same fields once more from +// Pulse's own origin, which carries the cookie. The form has no action, so it +// returns to the URL the IdP posted to, path prefix included, and the marker +// field stops a second repost: a browser that still has no cookie started no +// login here, and the ACS refuses its Response. +func writeSAMLACSRepost(w http.ResponseWriter, req *http.Request) { + type field struct{ Name, Value string } + page := struct { + Fields []field + Nonce string + }{Nonce: CSPNonceFromContext(req.Context())} + for _, name := range []string{"SAMLResponse", "SAMLart", "RelayState"} { + if value := req.PostForm.Get(name); value != "" { + page.Fields = append(page.Fields, field{Name: name, Value: value}) + } + } + page.Fields = append(page.Fields, field{Name: samlACSRepostField, Value: "1"}) + + w.Header().Set("Content-Type", "text/html; charset=utf-8") + w.Header().Set("Cache-Control", "no-store") + if err := samlACSRepostPage.Execute(w, page); err != nil { + log.Error().Err(err).Msg("Failed to write SAML ACS repost page") + } +} + // clearSession clears the current session - properly invalidates server-side session // and clears both pulse_session and pulse_csrf cookies func (r *Router) clearSession(w http.ResponseWriter, req *http.Request) { diff --git a/internal/api/saml_handlers_more_test.go b/internal/api/saml_handlers_more_test.go index ff7ac9f9ac..ec1e80ca31 100644 --- a/internal/api/saml_handlers_more_test.go +++ b/internal/api/saml_handlers_more_test.go @@ -1,13 +1,26 @@ package api import ( + "bytes" + "compress/flate" "context" + "crypto/x509" + "encoding/base64" + "encoding/pem" + "encoding/xml" + "errors" + "fmt" + "html" + "io" "net/http" "net/http/httptest" + "net/url" + "regexp" "strings" "testing" "time" + "github.com/crewjam/saml" "github.com/rcourtman/pulse-go-rewrite/internal/config" ) @@ -246,8 +259,337 @@ func TestHandleSAMLSLO_RejectsIDPInitiatedLogoutRequest(t *testing.T) { } } +const samlSLOTestDestination = "https://pulse.example.com/api/saml/okta/slo" + +// samlSLOTestIdP signs LogoutResponses with a key the SP under test trusts +// through its IdP metadata, the way a real IdP answers a LogoutRequest. +type samlSLOTestIdP struct { + signer *saml.ServiceProvider +} + +// newSAMLSLOTestRouter returns a router whose "okta" provider supports SLO and +// trusts the returned IdP's signing key, with fresh session and CSRF stores. +func newSAMLSLOTestRouter(t *testing.T) (*Router, *samlSLOTestIdP) { + t.Helper() + dataPath := t.TempDir() + InitSessionStore(dataPath) + InitCSRFStore(dataPath) + + certPEM, _, key := generateTestCert(t) + block, _ := pem.Decode(certPEM) + if block == nil { + t.Fatal("decode IdP certificate PEM") + } + cert, err := x509.ParseCertificate(block.Bytes) + if err != nil { + t.Fatalf("parse IdP certificate: %v", err) + } + metadataXML := ` + + + + + ` + base64.StdEncoding.EncodeToString(block.Bytes) + ` + + + + + +` + + router := &Router{samlManager: NewSAMLServiceManager("https://pulse.example.com")} + router.samlManager.services["okta"] = newTestSAMLService(t, "okta", metadataXML) + return router, &samlSLOTestIdP{signer: &saml.ServiceProvider{ + EntityID: "idp", + Key: key, + Certificate: cert, + SignatureMethod: "http://www.w3.org/2001/04/xmldsig-more#rsa-sha256", + }} +} + +// redirectQuery returns the HTTP-Redirect binding query carrying a signed, +// successful LogoutResponse that answers requestID. +func (idp *samlSLOTestIdP) redirectQuery(t *testing.T, requestID string) string { + t.Helper() + resp, err := idp.signer.MakeLogoutResponse(samlSLOTestDestination, requestID) + if err != nil { + t.Fatalf("sign LogoutResponse: %v", err) + } + return resp.Redirect("").RawQuery +} + +// postForm returns the HTTP-POST binding form body carrying the same signed +// LogoutResponse as redirectQuery. +func (idp *samlSLOTestIdP) postForm(t *testing.T, requestID string) string { + t.Helper() + query, err := url.ParseQuery(idp.redirectQuery(t, requestID)) + if err != nil { + t.Fatalf("parse redirect query: %v", err) + } + signedXML := inflateSAMLTestPayload(t, query.Get("SAMLResponse")) + return url.Values{"SAMLResponse": {base64.StdEncoding.EncodeToString(signedXML)}}.Encode() +} + +func inflateSAMLTestPayload(t *testing.T, payload string) []byte { + t.Helper() + compressed, err := base64.StdEncoding.DecodeString(payload) + if err != nil { + t.Fatalf("decode SAML payload: %v", err) + } + raw, err := io.ReadAll(flate.NewReader(bytes.NewReader(compressed))) + if err != nil { + t.Fatalf("inflate SAML payload: %v", err) + } + return raw +} + +// newSAMLTestSession creates a live SAML session with the okta provider. +func newSAMLTestSession(t *testing.T, nameID string) string { + t.Helper() + token := generateSessionToken() + GetSessionStore().CreateSAMLSession(token, time.Hour, "agent", "127.0.0.1", nameID, &SAMLTokenInfo{ + ProviderID: "okta", + NameID: nameID, + SessionIndex: "sess-" + nameID, + }) + return token +} + +// startSAMLTestLogout runs SP-initiated logout for the session and returns the +// ID of the LogoutRequest sent to the IdP. +func startSAMLTestLogout(t *testing.T, router *Router, token string) string { + t.Helper() + req := httptest.NewRequest(http.MethodPost, "/api/saml/okta/logout", nil) + req.AddCookie(&http.Cookie{Name: "pulse_session", Value: token}) + rec := httptest.NewRecorder() + router.handleSAMLLogout(rec, req) + if rec.Code != http.StatusFound { + t.Fatalf("logout: expected status %d, got %d", http.StatusFound, rec.Code) + } + location, err := url.Parse(rec.Header().Get("Location")) + if err != nil || location.Host != "idp.example.com" { + t.Fatalf("logout: unexpected SLO redirect %q (%v)", rec.Header().Get("Location"), err) + } + var logoutRequest struct { + ID string `xml:"ID,attr"` + } + if err := xml.Unmarshal(inflateSAMLTestPayload(t, location.Query().Get("SAMLRequest")), &logoutRequest); err != nil { + t.Fatalf("parse LogoutRequest: %v", err) + } + if logoutRequest.ID == "" { + t.Fatal("LogoutRequest has no ID") + } + return logoutRequest.ID +} + +// deliverSAMLTestLogoutResponse sends an HTTP-Redirect binding LogoutResponse +// to /slo from a browser carrying the given session cookie ("" for none). +func deliverSAMLTestLogoutResponse(router *Router, query, token string) *httptest.ResponseRecorder { + req := httptest.NewRequest(http.MethodGet, "/api/saml/okta/slo?"+query, nil) + if token != "" { + req.AddCookie(&http.Cookie{Name: "pulse_session", Value: token}) + } + rec := httptest.NewRecorder() + router.handleSAMLSLO(rec, req) + return rec +} + +// postSAMLTestLogoutResponse sends an HTTP-POST binding form body to /slo as a +// cross-site form POST does: without the SameSite=Lax session cookie. +func postSAMLTestLogoutResponse(router *Router, form string) *httptest.ResponseRecorder { + req := httptest.NewRequest(http.MethodPost, "/api/saml/okta/slo", strings.NewReader(form)) + req.Header.Set("Content-Type", "application/x-www-form-urlencoded") + rec := httptest.NewRecorder() + router.handleSAMLSLO(rec, req) + return rec +} + +func assertSAMLSLORefused(t *testing.T, rec *httptest.ResponseRecorder, label string) { + t.Helper() + if rec.Code != http.StatusForbidden { + t.Fatalf("%s: expected status %d, got %d body=%q", label, http.StatusForbidden, rec.Code, rec.Body.String()) + } + if loc := rec.Header().Get("Location"); loc != "" { + t.Fatalf("%s: refused SLO must not redirect, got %q", label, loc) + } + if cookies := rec.Result().Cookies(); len(cookies) != 0 { + t.Fatalf("%s: refused SLO must not touch cookies, got %v", label, cookies) + } +} + +// The browser that started SP-initiated logout comes back from the IdP without +// a session cookie, because handleSAMLLogout cleared it before redirecting. +// The IdP's signed answer to that logout completes it, by either binding. +func TestHandleSAMLSLO_AcceptsAnswerToOutstandingLogout(t *testing.T) { + router, idp := newSAMLSLOTestRouter(t) + + token := newSAMLTestSession(t, "alice") + requestID := startSAMLTestLogout(t, router, token) + if ValidateSession(token) { + t.Fatal("SP-initiated logout must clear the local session before redirecting") + } + + rec := deliverSAMLTestLogoutResponse(router, idp.redirectQuery(t, requestID), "") + if rec.Code != http.StatusFound { + t.Fatalf("redirect binding: expected status %d, got %d body=%q", http.StatusFound, rec.Code, rec.Body.String()) + } + if loc := rec.Header().Get("Location"); loc != "/?logout=success" { + t.Fatalf("redirect binding: unexpected redirect %q", loc) + } + + token = newSAMLTestSession(t, "alice") + requestID = startSAMLTestLogout(t, router, token) + rec = postSAMLTestLogoutResponse(router, idp.postForm(t, requestID)) + if rec.Code != http.StatusFound { + t.Fatalf("POST binding: expected status %d, got %d body=%q", http.StatusFound, rec.Code, rec.Body.String()) + } + if loc := rec.Header().Get("Location"); loc != "/?logout=success" { + t.Fatalf("POST binding: unexpected redirect %q", loc) + } +} + +// Each LogoutRequest is answered once. Replaying the IdP's response, with or +// without a session cookie, is refused and leaves the presenting session alone. +func TestHandleSAMLSLO_RefusesReplayedLogoutResponse(t *testing.T) { + router, idp := newSAMLSLOTestRouter(t) + + requestID := startSAMLTestLogout(t, router, newSAMLTestSession(t, "alice")) + query := idp.redirectQuery(t, requestID) + if rec := deliverSAMLTestLogoutResponse(router, query, ""); rec.Code != http.StatusFound { + t.Fatalf("first delivery: expected status %d, got %d body=%q", http.StatusFound, rec.Code, rec.Body.String()) + } + + assertSAMLSLORefused(t, deliverSAMLTestLogoutResponse(router, query, ""), "replay without session") + + victim := newSAMLTestSession(t, "bob") + assertSAMLSLORefused(t, deliverSAMLTestLogoutResponse(router, query, victim), "replay into another session") + if !ValidateSession(victim) { + t.Fatal("replayed LogoutResponse cleared another user's session") + } +} + +// A response the IdP signed for one user's logout must not clear a different +// user's live session, even on its first delivery inside the issue window: a +// cross-site GET carries the victim's SameSite=Lax cookie and skips CSRF. The +// misdelivered response spends the request, so it cannot be retried either. +func TestHandleSAMLSLO_RefusesAnotherSessionsLogoutResponse(t *testing.T) { + router, idp := newSAMLSLOTestRouter(t) + + requestID := startSAMLTestLogout(t, router, newSAMLTestSession(t, "attacker")) + query := idp.redirectQuery(t, requestID) + + victim := newSAMLTestSession(t, "victim") + assertSAMLSLORefused(t, deliverSAMLTestLogoutResponse(router, query, victim), "another session's response") + if !ValidateSession(victim) { + t.Fatal("another user's LogoutResponse cleared the victim's session") + } + assertSAMLSLORefused(t, deliverSAMLTestLogoutResponse(router, query, ""), "retry after misdelivery") +} + +// A correctly signed LogoutResponse that answers no LogoutRequest this SP +// issued (an unsolicited one, or one answering a request the IdP was handed +// by someone else) is refused without touching the session. +func TestHandleSAMLSLO_RefusesUnsolicitedLogoutResponse(t *testing.T) { + router, idp := newSAMLSLOTestRouter(t) + victim := newSAMLTestSession(t, "victim") + + for _, requestID := range []string{"", "id-never-issued-by-this-sp"} { + label := "InResponseTo=" + requestID + assertSAMLSLORefused(t, deliverSAMLTestLogoutResponse(router, idp.redirectQuery(t, requestID), victim), label) + if !ValidateSession(victim) { + t.Fatalf("%s: unsolicited LogoutResponse cleared the session", label) + } + } +} + +// encoding/xml matches an un-namespaced attribute field to any attribute with +// that local name, a namespace declaration such as xmlns:InResponseTo included, +// and exclusive canonicalization can leave an unused declaration out of the +// signed form. Appending one to a signed response must not rebind it to another +// outstanding LogoutRequest. +func TestHandleSAMLSLO_InResponseToIgnoresNamespaceDeclarations(t *testing.T) { + router, idp := newSAMLSLOTestRouter(t) + ownRequestID := startSAMLTestLogout(t, router, newSAMLTestSession(t, "alice")) + otherRequestID := startSAMLTestLogout(t, router, newSAMLTestSession(t, "bob")) + + query, err := url.ParseQuery(idp.redirectQuery(t, ownRequestID)) + if err != nil { + t.Fatalf("parse redirect query: %v", err) + } + signedXML := string(inflateSAMLTestPayload(t, query.Get("SAMLResponse"))) + genuine := `InResponseTo="` + ownRequestID + `"` + if !strings.Contains(signedXML, genuine) { + t.Fatalf("signed response has no %s: %s", genuine, signedXML) + } + spoofed := strings.Replace(signedXML, genuine, genuine+` xmlns:InResponseTo="`+otherRequestID+`"`, 1) + form := url.Values{"SAMLResponse": {base64.StdEncoding.EncodeToString([]byte(spoofed))}}.Encode() + + // The signature still verifies, so the response completes the request it + // was signed for and nothing else. + if rec := postSAMLTestLogoutResponse(router, form); rec.Code != http.StatusFound { + t.Fatalf("spoofed delivery: expected status %d, got %d body=%q", http.StatusFound, rec.Code, rec.Body.String()) + } + if rec := deliverSAMLTestLogoutResponse(router, idp.redirectQuery(t, otherRequestID), ""); rec.Code != http.StatusFound { + t.Fatalf("a namespace declaration spent another LogoutRequest: genuine answer got %d body=%q", rec.Code, rec.Body.String()) + } + assertSAMLSLORefused(t, deliverSAMLTestLogoutResponse(router, idp.redirectQuery(t, ownRequestID), ""), "signed request after spoofed delivery") +} + +// A cross-site HTTP-POST delivery withholds the SameSite=Lax session cookie, +// so /slo cannot see which session the browser holds. Completing a logout +// must therefore write no cookie: a deletion cookie would log out the +// browser's live session even though the request never presented it. +func TestHandleSAMLSLO_CompletionWritesNoCookies(t *testing.T) { + router, idp := newSAMLSLOTestRouter(t) + + requestID := startSAMLTestLogout(t, router, newSAMLTestSession(t, "attacker")) + victim := newSAMLTestSession(t, "victim") + rec := postSAMLTestLogoutResponse(router, idp.postForm(t, requestID)) + if rec.Code != http.StatusFound { + t.Fatalf("cookieless POST binding: expected status %d, got %d body=%q", http.StatusFound, rec.Code, rec.Body.String()) + } + if cookies := rec.Result().Cookies(); len(cookies) != 0 { + t.Fatalf("cookieless POST binding: SLO completion wrote cookies %v", cookies) + } + if !ValidateSession(victim) { + t.Fatal("SLO completion invalidated an unrelated session") + } + + token := newSAMLTestSession(t, "alice") + requestID = startSAMLTestLogout(t, router, token) + rec = deliverSAMLTestLogoutResponse(router, idp.redirectQuery(t, requestID), token) + if rec.Code != http.StatusFound { + t.Fatalf("own session: expected status %d, got %d body=%q", http.StatusFound, rec.Code, rec.Body.String()) + } + if cookies := rec.Result().Cookies(); len(cookies) != 0 { + t.Fatalf("own session: SLO completion wrote cookies %v", cookies) + } +} + +// The SAML routes skip CSRF, and a cross-site POST withholds the +// SameSite=Lax session cookie, so SP-initiated logout must not answer a +// request that carries no session with the deletion cookies of a logout. +func TestHandleSAMLLogout_RefusesRequestsWithoutSession(t *testing.T) { + router, _ := newSAMLSLOTestRouter(t) + + for _, path := range []string{"/api/saml/okta/logout", "/api/saml/unknown/logout"} { + req := httptest.NewRequest(http.MethodPost, path, nil) + rec := httptest.NewRecorder() + router.handleSAMLLogout(rec, req) + + if rec.Code != http.StatusUnauthorized { + t.Fatalf("%s: expected status %d, got %d body=%q", path, http.StatusUnauthorized, rec.Code, rec.Body.String()) + } + if cookies := rec.Result().Cookies(); len(cookies) != 0 { + t.Fatalf("%s: cookieless logout wrote cookies %v", path, cookies) + } + } +} + func TestHandleSAMLLogout_SLOUnavailable(t *testing.T) { - InitSessionStore(t.TempDir()) + dataPath := t.TempDir() + InitSessionStore(dataPath) + InitCSRFStore(dataPath) router := &Router{samlManager: NewSAMLServiceManager("https://pulse.example.com")} metadataXML := ` @@ -265,7 +607,7 @@ func TestHandleSAMLLogout_SLOUnavailable(t *testing.T) { SessionIndex: "sess-1", }) - req := httptest.NewRequest(http.MethodGet, "/api/saml/okta/logout", nil) + req := httptest.NewRequest(http.MethodPost, "/api/saml/okta/logout", nil) req.AddCookie(&http.Cookie{Name: "pulse_session", Value: token}) rec := httptest.NewRecorder() @@ -280,7 +622,9 @@ func TestHandleSAMLLogout_SLOUnavailable(t *testing.T) { } func TestHandleSAMLLogout_SLOSuccess(t *testing.T) { - InitSessionStore(t.TempDir()) + dataPath := t.TempDir() + InitSessionStore(dataPath) + InitCSRFStore(dataPath) router := &Router{samlManager: NewSAMLServiceManager("https://pulse.example.com")} metadataXML := ` @@ -299,7 +643,7 @@ func TestHandleSAMLLogout_SLOSuccess(t *testing.T) { SessionIndex: "sess-1", }) - req := httptest.NewRequest(http.MethodGet, "/api/saml/okta/logout", nil) + req := httptest.NewRequest(http.MethodPost, "/api/saml/okta/logout", nil) req.AddCookie(&http.Cookie{Name: "pulse_session", Value: token}) rec := httptest.NewRecorder() @@ -314,6 +658,64 @@ func TestHandleSAMLLogout_SLOSuccess(t *testing.T) { } } +// A cross-site top-level GET carries the SameSite=Lax session cookie and skips +// the CSRF check, so SP-initiated logout must not clear the session on GET or +// HEAD. +func TestHandleSAMLLogout_RefusesSafeMethods(t *testing.T) { + dataPath := t.TempDir() + InitSessionStore(dataPath) + InitCSRFStore(dataPath) + + router := &Router{samlManager: NewSAMLServiceManager("https://pulse.example.com")} + metadataXML := ` + + + + + +` + router.samlManager.services["okta"] = newTestSAMLService(t, "okta", metadataXML) + + token := generateSessionToken() + GetSessionStore().CreateSAMLSession(token, time.Hour, "agent", "127.0.0.1", "user", &SAMLTokenInfo{ + ProviderID: "okta", + NameID: "name-id", + SessionIndex: "sess-1", + }) + + for _, method := range []string{http.MethodGet, http.MethodHead} { + req := httptest.NewRequest(method, "/api/saml/okta/logout", nil) + req.AddCookie(&http.Cookie{Name: "pulse_session", Value: token}) + rec := httptest.NewRecorder() + + router.handleSAMLLogout(rec, req) + + if rec.Code != http.StatusMethodNotAllowed { + t.Fatalf("%s: expected status %d, got %d", method, http.StatusMethodNotAllowed, rec.Code) + } + if allow := rec.Header().Get("Allow"); allow != http.MethodPost { + t.Fatalf("%s: expected Allow %q, got %q", method, http.MethodPost, allow) + } + if loc := rec.Header().Get("Location"); loc != "" { + t.Fatalf("%s: expected no SLO redirect, got %q", method, loc) + } + if !ValidateSession(token) { + t.Fatalf("%s: SAML logout cleared the session", method) + } + } + + req := httptest.NewRequest(http.MethodPost, "/api/saml/okta/logout", nil) + req.AddCookie(&http.Cookie{Name: "pulse_session", Value: token}) + rec := httptest.NewRecorder() + router.handleSAMLLogout(rec, req) + if rec.Code != http.StatusFound { + t.Fatalf("POST: expected status %d, got %d", http.StatusFound, rec.Code) + } + if ValidateSession(token) { + t.Fatal("POST: expected SAML logout to clear the session") + } +} + func TestExtractSAMLProviderID(t *testing.T) { if got := extractSAMLProviderID("/api/saml/okta/login", "login"); got != "okta" { t.Fatalf("expected okta, got %q", got) @@ -328,3 +730,522 @@ func TestExtractSAMLProviderID(t *testing.T) { t.Fatalf("expected empty provider for non-saml path, got %q", got) } } + +const samlManualCertTestIdPEntityID = "https://idp.example.com/metadata" + +// samlManualCertTestIdP holds an IdP signing key and its PEM certificate, the +// form an administrator pastes or points IDPCertFile at for a provider +// configured without IdP metadata. It signs ACS Responses and LogoutResponses +// the way the real IdP would. +type samlManualCertTestIdP struct { + certPEM []byte + idp *saml.IdentityProvider + slo *samlSLOTestIdP +} + +func newSAMLManualCertTestIdP(t *testing.T) *samlManualCertTestIdP { + t.Helper() + certPEM, _, key := generateTestCert(t) + block, _ := pem.Decode(certPEM) + if block == nil { + t.Fatal("decode IdP certificate PEM") + } + cert, err := x509.ParseCertificate(block.Bytes) + if err != nil { + t.Fatalf("parse IdP certificate: %v", err) + } + entityID, err := url.Parse(samlManualCertTestIdPEntityID) + if err != nil { + t.Fatalf("parse IdP entity ID: %v", err) + } + const signatureMethod = "http://www.w3.org/2001/04/xmldsig-more#rsa-sha256" + return &samlManualCertTestIdP{ + certPEM: certPEM, + idp: &saml.IdentityProvider{ + Key: key, + Certificate: cert, + MetadataURL: *entityID, + SignatureMethod: signatureMethod, + }, + slo: &samlSLOTestIdP{signer: &saml.ServiceProvider{ + EntityID: samlManualCertTestIdPEntityID, + Key: key, + Certificate: cert, + SignatureMethod: signatureMethod, + }}, + } +} + +// metadataXML publishes the same signing certificate as IdP metadata does, +// base64 DER inside . +func (idp *samlManualCertTestIdP) metadataXML() string { + return ` + + + + + ` + base64.StdEncoding.EncodeToString(idp.idp.Certificate.Raw) + ` + + + + + +` +} + +// acsPost returns the HTTP-POST binding request to the service's ACS carrying +// a signed, unsolicited (no InResponseTo) Response that identifies alice. +func (idp *samlManualCertTestIdP) acsPost(t *testing.T, service *SAMLService) *http.Request { + t.Helper() + return samlTestACSRequest(service, url.Values{"SAMLResponse": {base64.StdEncoding.EncodeToString(idp.responseXML(t, service, ""))}}) +} + +// acsPostAnswering returns the same request for an SP-initiated login: the +// Response and its SubjectConfirmationData answer the AuthnRequest requestID. +func (idp *samlManualCertTestIdP) acsPostAnswering(t *testing.T, service *SAMLService, requestID string) *http.Request { + t.Helper() + return samlTestACSRequest(service, url.Values{"SAMLResponse": {base64.StdEncoding.EncodeToString(idp.responseXML(t, service, requestID))}}) +} + +// responseXML returns the Response the IdP posts to the service's ACS after +// authenticating alice, answering requestID ("" for an unsolicited one). Both +// the Response and its Assertion are signed. +func (idp *samlManualCertTestIdP) responseXML(t *testing.T, service *SAMLService, requestID string) []byte { + t.Helper() + return idp.signedResponseXML(t, service, requestID, "alice", nil) +} + +// signedResponseXML is responseXML for the user nameID, with edit applied to +// the Assertion before the IdP signs it. +func (idp *samlManualCertTestIdP) signedResponseXML(t *testing.T, service *SAMLService, requestID, nameID string, edit func(*saml.Assertion)) []byte { + t.Helper() + spMetadata := service.sp.Metadata() + authn := &saml.IdpAuthnRequest{ + IDP: idp.idp, + HTTPRequest: httptest.NewRequest(http.MethodPost, "https://idp.example.com/sso", nil), + Request: saml.AuthnRequest{ID: requestID}, + ServiceProviderMetadata: spMetadata, + SPSSODescriptor: &spMetadata.SPSSODescriptors[0], + ACSEndpoint: &saml.IndexedEndpoint{Binding: saml.HTTPPostBinding, Location: service.sp.AcsURL.String()}, + Now: time.Now(), + } + if err := (saml.DefaultAssertionMaker{}).MakeAssertion(authn, &saml.Session{ + ID: "idp-session-" + nameID, + CreateTime: time.Now(), + ExpireTime: time.Now().Add(time.Hour), + Index: "idx-" + nameID, + NameID: nameID, + }); err != nil { + t.Fatalf("make assertion: %v", err) + } + if edit != nil { + edit(authn.Assertion) + } + form, err := authn.PostBinding() + if err != nil { + t.Fatalf("sign ACS Response: %v", err) + } + raw, err := base64.StdEncoding.DecodeString(form.SAMLResponse) + if err != nil { + t.Fatalf("decode ACS Response: %v", err) + } + return raw +} + +func samlTestACSRequest(service *SAMLService, form url.Values) *http.Request { + req := httptest.NewRequest(http.MethodPost, service.sp.AcsURL.String(), strings.NewReader(form.Encode())) + req.Header.Set("Content-Type", "application/x-www-form-urlencoded") + return req +} + +// samlTestAuthnRequestID returns the ID of the AuthnRequest carried by an +// HTTP-Redirect binding URL to the IdP. +func samlTestAuthnRequestID(t *testing.T, redirectURL string) string { + t.Helper() + location, err := url.Parse(redirectURL) + if err != nil || location.Host != "idp.example.com" { + t.Fatalf("unexpected AuthnRequest redirect %q (%v)", redirectURL, err) + } + var authnRequest struct { + ID string `xml:"ID,attr"` + } + if err := xml.Unmarshal(inflateSAMLTestPayload(t, location.Query().Get("SAMLRequest")), &authnRequest); err != nil { + t.Fatalf("parse AuthnRequest: %v", err) + } + if authnRequest.ID == "" { + t.Fatal("AuthnRequest has no ID") + } + return authnRequest.ID +} + +// samlTestXMLAttr appends attr="value" to the first start tag named tag (any +// prefix) in doc, the way an attacker edits a signed Response in transit. +func samlTestXMLAttr(t *testing.T, doc []byte, tag, attr, value string) []byte { + t.Helper() + re := regexp.MustCompile(`<([A-Za-z0-9]+:)?` + tag + `[\s>/]`) + loc := re.FindIndex(doc) + if loc == nil { + t.Fatalf("no <%s> in %s", tag, doc) + } + end := bytes.IndexByte(doc[loc[0]:], '>') + loc[0] + if doc[end-1] == '/' { + end-- + } + out := append([]byte{}, doc[:end]...) + out = append(out, []byte(` `+attr+`="`+value+`"`)...) + return append(out, doc[end:]...) +} + +// useSAMLTestAuthStores gives the test fresh session and CSRF stores and puts +// the previous ones back afterwards, so tests that run later still find the +// stores they were using. +func useSAMLTestAuthStores(t *testing.T) { + t.Helper() + dataPath := t.TempDir() + + sessionStoreMu.Lock() + previousSessionStore, previousSessionPath := sessionStore, sessionStoreDataPath + sessionStore, sessionStoreDataPath = nil, "" + sessionStoreMu.Unlock() + csrfStoreMu.Lock() + previousCSRFStore, previousCSRFPath := csrfStore, csrfStoreDataPath + csrfStore, csrfStoreDataPath = nil, "" + csrfStoreMu.Unlock() + InitSessionStore(dataPath) + InitCSRFStore(dataPath) + + t.Cleanup(func() { + resetSessionStoreForTests() + resetCSRFStoreForTests() + sessionStoreMu.Lock() + sessionStore, sessionStoreDataPath = previousSessionStore, previousSessionPath + sessionStoreMu.Unlock() + csrfStoreMu.Lock() + csrfStore, csrfStoreDataPath = previousCSRFStore, previousCSRFPath + csrfStoreMu.Unlock() + }) +} + +// samlTestUnsignedResponse removes the Response's own signature, leaving only +// its Assertion signed, as many IdPs send it. +func samlTestUnsignedResponse(t *testing.T, signed []byte) []byte { + t.Helper() + unsigned := regexp.MustCompile(`(?s)(]*>.*?)`).ReplaceAll(signed, []byte("$1")) + if bytes.Equal(unsigned, signed) { + t.Fatalf("could not strip the Response signature from %s", signed) + } + return unsigned +} + +const samlACSTestProvider = "okta" + +// newSAMLACSTestRouter returns a router whose "okta" SAML provider trusts idp +// through its metadata, at default settings unless allowIDPInitiated is set, +// with fresh session and CSRF stores. +func newSAMLACSTestRouter(t *testing.T, idp *samlManualCertTestIdP, allowIDPInitiated bool) (*Router, *SAMLService) { + t.Helper() + useSAMLTestAuthStores(t) + + cfg := &config.SAMLProviderConfig{IDPMetadataXML: idp.metadataXML(), AllowIDPInitiated: allowIDPInitiated} + service, err := NewSAMLService(context.Background(), samlACSTestProvider, cfg, "https://pulse.example.com") + if err != nil { + t.Fatalf("NewSAMLService: %v", err) + } + router := &Router{ + samlManager: NewSAMLServiceManager("https://pulse.example.com"), + ssoConfig: &config.SSOConfig{Providers: []config.SSOProvider{{ + ID: samlACSTestProvider, Name: "Test SAML", Type: config.SSOProviderTypeSAML, Enabled: true, SAML: cfg, + }}}, + } + router.samlManager.services[samlACSTestProvider] = service + return router, service +} + +// startSAMLTestLogin starts SP-initiated login in a browser holding loginToken +// ("" for a browser that has none yet) and returns the ID of the AuthnRequest +// sent to the IdP and the login token the browser holds afterwards. +func startSAMLTestLogin(t *testing.T, router *Router, loginToken, returnTo string) (requestID, heldToken string) { + t.Helper() + target := "/api/saml/okta/login" + if returnTo != "" { + target += "?returnTo=" + url.QueryEscape(returnTo) + } + req := httptest.NewRequest(http.MethodGet, target, nil) + if loginToken != "" { + req.AddCookie(&http.Cookie{Name: cookieNameSAMLLogin, Value: loginToken}) + } + rec := httptest.NewRecorder() + router.handleSAMLLogin(rec, req) + if rec.Code != http.StatusFound { + t.Fatalf("login: expected status %d, got %d body=%q", http.StatusFound, rec.Code, rec.Body.String()) + } + heldToken = loginToken + for _, cookie := range rec.Result().Cookies() { + if cookie.Name == cookieNameSAMLLogin { + if !cookie.HttpOnly || cookie.SameSite != http.SameSiteLaxMode || cookie.MaxAge <= 0 || cookie.Path != "/" { + t.Fatalf("login cookie must be an HttpOnly SameSite=Lax cookie that expires, got %+v", cookie) + } + heldToken = cookie.Value + } + } + if heldToken == "" { + t.Fatal("login set no login cookie") + } + return samlTestAuthnRequestID(t, rec.Header().Get("Location")), heldToken +} + +// postSAMLTestACS delivers an ACS form from a browser holding loginToken ("" +// for none). A cross-site HTTP-POST from the IdP carries no SameSite=Lax +// cookie, so pass "" to model the IdP's own delivery. +func postSAMLTestACS(router *Router, form url.Values, loginToken string) *httptest.ResponseRecorder { + req := httptest.NewRequest(http.MethodPost, "/api/saml/okta/acs", strings.NewReader(form.Encode())) + req.Header.Set("Content-Type", "application/x-www-form-urlencoded") + if loginToken != "" { + req.AddCookie(&http.Cookie{Name: cookieNameSAMLLogin, Value: loginToken}) + } + rec := httptest.NewRecorder() + router.handleSAMLACS(rec, req) + return rec +} + +// samlTestRepostForm returns the hidden fields of the ACS repost page. +func samlTestRepostForm(t *testing.T, rec *httptest.ResponseRecorder) url.Values { + t.Helper() + if rec.Code != http.StatusOK || !strings.HasPrefix(rec.Header().Get("Content-Type"), "text/html") { + t.Fatalf("expected the ACS repost page, got status %d type %q body=%q", rec.Code, rec.Header().Get("Content-Type"), rec.Body.String()) + } + form := url.Values{} + for _, match := range regexp.MustCompile(``).FindAllStringSubmatch(rec.Body.String(), -1) { + form.Add(match[1], html.UnescapeString(match[2])) + } + return form +} + +// deliverSAMLTestResponse delivers a Response the way the browser that holds +// loginToken receives it: the IdP's cross-site POST, which carries no login +// cookie and gets the repost page, then the page's same-origin repost. +func deliverSAMLTestResponse(t *testing.T, router *Router, form url.Values, loginToken string) *httptest.ResponseRecorder { + t.Helper() + return postSAMLTestACS(router, samlTestRepostForm(t, postSAMLTestACS(router, form, "")), loginToken) +} + +func assertSAMLLoginSucceeded(t *testing.T, rec *httptest.ResponseRecorder, wantLocation, label string) { + t.Helper() + if rec.Code != http.StatusFound || rec.Header().Get("Location") != wantLocation { + t.Fatalf("%s: expected redirect to %q, got status %d location %q body=%q", label, wantLocation, rec.Code, rec.Header().Get("Location"), rec.Body.String()) + } + for _, cookie := range rec.Result().Cookies() { + if cookie.Name == cookieNameSession && cookie.Value != "" && ValidateSession(cookie.Value) { + return + } + } + t.Fatalf("%s: login established no session, cookies %v", label, rec.Result().Cookies()) +} + +func assertSAMLLoginRefused(t *testing.T, rec *httptest.ResponseRecorder, label string) { + t.Helper() + if rec.Code != http.StatusFound || !strings.Contains(rec.Header().Get("Location"), "saml_error=saml_validation_failed") { + t.Fatalf("%s: expected the validation-failed redirect, got status %d location %q body=%q", label, rec.Code, rec.Header().Get("Location"), rec.Body.String()) + } + for _, cookie := range rec.Result().Cookies() { + if cookie.Name == cookieNameSession || cookie.Name == cookieNameSessionSecure { + t.Fatalf("%s: refused login wrote session cookie %+v", label, cookie) + } + } +} + +// At default settings, with IdP-initiated logins off, SP-initiated login +// completes: the IdP's cross-site POST of its signed answer gets the repost +// page, whose same-origin repost carries the login cookie, and the login +// returns to the recorded returnTo whatever RelayState the IdP posts. +func TestHandleSAMLACS_CompletesLoginStartedInThisBrowser(t *testing.T) { + idp := newSAMLManualCertTestIdP(t) + router, service := newSAMLACSTestRouter(t, idp, false) + + requestID, token := startSAMLTestLogin(t, router, "", "/alerts?tab=history&range=7d") + form := url.Values{ + "SAMLResponse": {base64.StdEncoding.EncodeToString(idp.responseXML(t, service, requestID))}, + "RelayState": {"/settings"}, + } + repost := postSAMLTestACS(router, form, "") + fields := samlTestRepostForm(t, repost) + if fields.Get("SAMLResponse") != form.Get("SAMLResponse") || fields.Get("RelayState") != "/settings" || fields.Get(samlACSRepostField) != "1" { + t.Fatalf("repost page must carry the posted fields and the repost marker, got %v", fields) + } + if cookies := repost.Result().Cookies(); len(cookies) != 0 { + t.Fatalf("repost page must not write cookies, got %v", cookies) + } + if body := repost.Body.String(); strings.Contains(body, " action=") || !strings.Contains(body, "document.forms[0].submit()") { + t.Fatalf("repost page must submit itself back to the URL the IdP posted to, got %q", body) + } + + assertSAMLLoginSucceeded(t, postSAMLTestACS(router, fields, token), "/alerts?range=7d&saml=success&tab=history", "repost with login cookie") +} + +// Each AuthnRequest is answered once, and only in the browser that started it. +// A Response answering another browser's request, an unknown request or none +// is refused, and refusing another browser leaves that browser's login intact. +func TestHandleSAMLACS_RefusesResponsesNotAnsweringThisBrowsersLogin(t *testing.T) { + idp := newSAMLManualCertTestIdP(t) + router, service := newSAMLACSTestRouter(t, idp, false) + + victimRequestID, victimToken := startSAMLTestLogin(t, router, "", "/") + attackerRequestID, attackerToken := startSAMLTestLogin(t, router, "", "/") + attackerForm := url.Values{"SAMLResponse": {base64.StdEncoding.EncodeToString(idp.responseXML(t, service, attackerRequestID))}} + + assertSAMLLoginRefused(t, deliverSAMLTestResponse(t, router, attackerForm, victimToken), "another browser's Response") + for _, requestID := range []string{"", "id-never-issued-by-this-sp"} { + form := url.Values{"SAMLResponse": {base64.StdEncoding.EncodeToString(idp.responseXML(t, service, requestID))}} + assertSAMLLoginRefused(t, deliverSAMLTestResponse(t, router, form, victimToken), "InResponseTo="+requestID) + } + assertSAMLLoginRefused(t, postSAMLTestACS(router, url.Values{ + "SAMLResponse": attackerForm["SAMLResponse"], + samlACSRepostField: {"1"}, + }, ""), "repost without a login cookie") + + assertSAMLLoginSucceeded(t, deliverSAMLTestResponse(t, router, attackerForm, attackerToken), "/?saml=success", "the Response's own browser") + assertSAMLLoginRefused(t, deliverSAMLTestResponse(t, router, attackerForm, attackerToken), "replay in the same browser") + + victimForm := url.Values{"SAMLResponse": {base64.StdEncoding.EncodeToString(idp.responseXML(t, service, victimRequestID))}} + assertSAMLLoginSucceeded(t, deliverSAMLTestResponse(t, router, victimForm, victimToken), "/?saml=success", "the victim's own login after the refusals") +} + +// encoding/xml fills an un-namespaced attribute field from a namespace +// declaration with the same local name, and exclusive canonicalization leaves +// an unused declaration out of the signed form. Declaring xmlns:InResponseTo +// on a signed Response or Assertion must not rebind the attacker's answer to +// the victim's outstanding AuthnRequest, whether the Response itself is signed +// or only its Assertion is. +func TestHandleSAMLACS_InResponseToIgnoresNamespaceDeclarations(t *testing.T) { + idp := newSAMLManualCertTestIdP(t) + router, service := newSAMLACSTestRouter(t, idp, false) + + victimRequestID, victimToken := startSAMLTestLogin(t, router, "", "/") + attackerRequestID, _ := startSAMLTestLogin(t, router, "", "/") + signed := idp.responseXML(t, service, attackerRequestID) + + spoofed := samlTestXMLAttr(t, signed, "Response", "xmlns:InResponseTo", victimRequestID) + spoofed = samlTestXMLAttr(t, spoofed, "SubjectConfirmationData", "xmlns:InResponseTo", victimRequestID) + assertSAMLLoginRefused(t, deliverSAMLTestResponse(t, router, url.Values{"SAMLResponse": {base64.StdEncoding.EncodeToString(spoofed)}}, victimToken), "signed Response") + + // Without its own signature the Response's attributes are the attacker's + // to write; only the signed Assertion says which request it answers. + assertionOnly := bytes.Replace(samlTestUnsignedResponse(t, signed), []byte(`InResponseTo="`+attackerRequestID+`"`), []byte(`InResponseTo="`+victimRequestID+`"`), 1) + assertionOnly = samlTestXMLAttr(t, assertionOnly, "SubjectConfirmationData", "xmlns:InResponseTo", victimRequestID) + assertSAMLLoginRefused(t, deliverSAMLTestResponse(t, router, url.Values{"SAMLResponse": {base64.StdEncoding.EncodeToString(assertionOnly)}}, victimToken), "assertion-only signature") + + victimForm := url.Values{"SAMLResponse": {base64.StdEncoding.EncodeToString(idp.responseXML(t, service, victimRequestID))}} + assertSAMLLoginSucceeded(t, deliverSAMLTestResponse(t, router, victimForm, victimToken), "/?saml=success", "the victim's own login after the spoofs") +} + +// Many IdPs sign only the Assertion. Its signed SubjectConfirmationData then +// names the request, and the login completes. A signed assertion that confirms +// no request cannot ride along with one that does: an IdP-signed assertion for +// another user without a SubjectConfirmation, placed ahead of the victim's own +// in an unsigned Response, is the assertion crewjam would return, and the +// victim's request must not vouch for it. +func TestHandleSAMLACS_BindsTheAssertionItReturns(t *testing.T) { + idp := newSAMLManualCertTestIdP(t) + router, service := newSAMLACSTestRouter(t, idp, false) + + victimRequestID, victimToken := startSAMLTestLogin(t, router, "", "/") + victimResponse := samlTestUnsignedResponse(t, idp.responseXML(t, service, victimRequestID)) + + unconfirmed := idp.signedResponseXML(t, service, "", "mallory", func(assertion *saml.Assertion) { + assertion.Subject.SubjectConfirmations = nil + }) + unconfirmedAssertion := regexp.MustCompile(`(?s)`).Find(unconfirmed) + at := bytes.Index(victimResponse, []byte("`}, + "RelayState": {"/"}, + "Unrelated": {"dropped"}, + }.Encode())) + req.Header.Set("Content-Type", "application/x-www-form-urlencoded") + req = req.WithContext(context.WithValue(req.Context(), cspNonceKey{}, "test-nonce")) + if err := req.ParseForm(); err != nil { + t.Fatalf("parse form: %v", err) + } + rec := httptest.NewRecorder() + writeSAMLACSRepost(rec, req) + + body := rec.Body.String() + if strings.Contains(body, "` { + t.Fatalf("repost must carry the exact posted value, got %v", fields) + } +} + +// samlRejectionDetail adds crewjam/saml's private reason to a rejected +// response, which its public error ("Authentication failed") hides. +func samlRejectionDetail(err error) string { + var invalid *saml.InvalidResponseError + if errors.As(err, &invalid) && invalid.PrivateErr != nil { + return err.Error() + ": " + invalid.PrivateErr.Error() + } + return fmt.Sprint(err) +} diff --git a/internal/api/saml_runtime_license_gating_test.go b/internal/api/saml_runtime_license_gating_test.go index 41e4ad6e0e..6a023b5dda 100644 --- a/internal/api/saml_runtime_license_gating_test.go +++ b/internal/api/saml_runtime_license_gating_test.go @@ -26,7 +26,7 @@ func TestSAMLRuntimeEndpointsUseCommunitySSOFeature(t *testing.T) { {http.MethodGet, "/api/saml/test-provider/login"}, {http.MethodPost, "/api/saml/test-provider/acs"}, {http.MethodGet, "/api/saml/test-provider/metadata"}, - {http.MethodGet, "/api/saml/test-provider/logout"}, + {http.MethodPost, "/api/saml/test-provider/logout"}, {http.MethodPost, "/api/saml/test-provider/slo"}, } diff --git a/internal/api/saml_service.go b/internal/api/saml_service.go index 01d6fbd187..fae6e23303 100644 --- a/internal/api/saml_service.go +++ b/internal/api/saml_service.go @@ -1,9 +1,12 @@ package api import ( + "bytes" + "compress/flate" "context" "crypto/rsa" "crypto/x509" + "encoding/base64" "encoding/pem" "encoding/xml" "errors" @@ -11,14 +14,18 @@ import ( "io" "net/http" "net/url" + "reflect" + "sort" "strings" "sync" "time" + "github.com/beevik/etree" "github.com/crewjam/saml" "github.com/rcourtman/pulse-go-rewrite/internal/config" "github.com/rcourtman/pulse-go-rewrite/internal/securityutil" "github.com/rs/zerolog/log" + dsig "github.com/russellhaering/goxmldsig" ) // SAMLService manages SAML Service Provider functionality for a single provider @@ -31,6 +38,141 @@ type SAMLService struct { httpClient *http.Client baseURL string lastRefresh time.Time + + // authnRequests holds the SP-initiated AuthnRequests still waiting for + // the IdP's Response, each bound to the browser that started the login. + authnRequests samlRequestStore + // logoutRequests holds the SP-initiated LogoutRequests still waiting for + // the IdP's LogoutResponse. + logoutRequests samlRequestStore +} + +const ( + // samlRequestTTL bounds how long the IdP round trip of an SP-initiated + // login or logout may take, matching the OIDC login state lifetime. + samlRequestTTL = 10 * time.Minute + maxSAMLRequestEntries = 1024 + // samlLogoutResponseMaxBytes matches crewjam/saml's inflate limit for the + // HTTP-Redirect binding. + samlLogoutResponseMaxBytes = 10 << 20 +) + +// errSAMLLogoutResponseUnbound marks a LogoutResponse that verified but does +// not answer an outstanding LogoutRequest this SP issued for the presenting +// session. +var errSAMLLogoutResponseUnbound = errors.New("logout response does not answer an outstanding logout request for this session") + +// errSAMLResponseUnbound marks an ACS Response that does not answer an +// outstanding AuthnRequest this SP issued to the presenting browser. +var errSAMLResponseUnbound = errors.New("response does not answer an outstanding authentication request from this browser") + +// samlRequestStore records the requests this SP issued and is still waiting +// for the IdP to answer, keyed by request ID. Each record carries the key of +// what the request is bound to: the browser that started an SP-initiated +// login, or the session an SP-initiated logout ended. +// +// crewjam/saml checks a LogoutResponse's signature, destination, issuer, +// status and age but never its InResponseTo, so without these records any +// recent signed response from the IdP, including one answering another user's +// logout, would be accepted. For logins crewjam does check InResponseTo, but +// only against the request IDs its caller supplies, and without records there +// were none to supply: every SP-initiated Response was refused. +type samlRequestStore struct { + mu sync.Mutex + entries map[string]samlRequestRecord +} + +type samlRequestRecord struct { + bindingKey string + // returnTo is the local path an SP-initiated login returns to. + returnTo string + expiresAt time.Time +} + +func (s *samlRequestStore) put(requestID string, record samlRequestRecord, now time.Time) { + s.mu.Lock() + defer s.mu.Unlock() + + if s.entries == nil { + s.entries = make(map[string]samlRequestRecord) + } + for id, entry := range s.entries { + if !now.Before(entry.expiresAt) { + delete(s.entries, id) + } + } + for len(s.entries) >= maxSAMLRequestEntries { + oldestID := "" + var oldestExpiry time.Time + for id, entry := range s.entries { + if oldestID == "" || entry.expiresAt.Before(oldestExpiry) || (entry.expiresAt.Equal(oldestExpiry) && id < oldestID) { + oldestID = id + oldestExpiry = entry.expiresAt + } + } + delete(s.entries, oldestID) + } + record.expiresAt = now.Add(samlRequestTTL) + s.entries[requestID] = record +} + +// consume spends the outstanding request a verified LogoutResponse answers. +// sessionKey identifies the session the presenting browser carries, or is +// empty when it carries none. The response may complete the logout only when +// it answers an unexpired request and the browser carries either no session +// or the session that request logged out. The request is spent on every +// verified presentation, so a response is honored at most once. +func (s *samlRequestStore) consume(requestID, sessionKey string, now time.Time) error { + s.mu.Lock() + defer s.mu.Unlock() + + entry, ok := s.entries[requestID] + if requestID == "" || !ok { + return fmt.Errorf("%w: no outstanding request matches InResponseTo %q", errSAMLLogoutResponseUnbound, requestID) + } + delete(s.entries, requestID) + if !now.Before(entry.expiresAt) { + return fmt.Errorf("%w: request %q expired", errSAMLLogoutResponseUnbound, requestID) + } + if sessionKey != "" && sessionKey != entry.bindingKey { + return fmt.Errorf("%w: request %q logged out a different session", errSAMLLogoutResponseUnbound, requestID) + } + return nil +} + +// outstanding returns every unexpired request, sorted, and whether any of +// them is bound to bindingKey. +func (s *samlRequestStore) outstanding(bindingKey string, now time.Time) (requestIDs []string, bound bool) { + s.mu.Lock() + defer s.mu.Unlock() + + for id, entry := range s.entries { + if now.Before(entry.expiresAt) { + requestIDs = append(requestIDs, id) + bound = bound || (bindingKey != "" && entry.bindingKey == bindingKey) + } + } + sort.Strings(requestIDs) + return requestIDs, bound +} + +// spend removes and returns the unexpired request a verified ACS Response +// answers, provided it is bound to bindingKey, the presenting browser. A +// request bound to another browser is left for that browser, so presenting a +// stolen Response elsewhere does not cost its owner the login. +func (s *samlRequestStore) spend(requestID, bindingKey string, now time.Time) (samlRequestRecord, error) { + s.mu.Lock() + defer s.mu.Unlock() + + entry, ok := s.entries[requestID] + if requestID == "" || bindingKey == "" || !ok || entry.bindingKey != bindingKey { + return samlRequestRecord{}, fmt.Errorf("%w: no outstanding request from this browser matches InResponseTo %q", errSAMLResponseUnbound, requestID) + } + delete(s.entries, requestID) + if !now.Before(entry.expiresAt) { + return samlRequestRecord{}, fmt.Errorf("%w: request %q expired", errSAMLResponseUnbound, requestID) + } + return entry, nil } func normalizeSAMLBaseURL(baseURL string) (string, error) { @@ -269,8 +411,11 @@ func (s *SAMLService) addIDPCertificate(metadata *saml.EntityDescriptor) error { Use: "signing", KeyInfo: saml.KeyInfo{ X509Data: saml.X509Data{ + // Base64 DER, as carries it in IdP + // metadata: crewjam/saml base64-decodes this field to + // verify signatures and cannot read a PEM block. X509Certificates: []saml.X509Certificate{ - {Data: string(pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: cert.Raw}))}, + {Data: base64.StdEncoding.EncodeToString(cert.Raw)}, }, }, }, @@ -451,25 +596,40 @@ func (s *SAMLService) loadSPCredentials() (*x509.Certificate, *rsa.PrivateKey, e return cert, key, nil } -// MakeAuthRequest creates a SAML AuthnRequest and returns the redirect URL -func (s *SAMLService) MakeAuthRequest(relayState string) (string, error) { +// MakeAuthRequest creates a SAML AuthnRequest and returns the HTTP-Redirect +// binding URL that sends it to the IdP. It records the request as outstanding +// for browserKey, the login binding key of the browser starting the login, +// together with returnTo, so that ProcessResponse accepts the IdP's answer to +// this request only once, only from that browser, and returns it to returnTo. +func (s *SAMLService) MakeAuthRequest(returnTo, browserKey string) (string, error) { s.mu.RLock() defer s.mu.RUnlock() if s.sp == nil { return "", errors.New("service provider not initialized") } + if browserKey == "" { + return "", errors.New("authentication request has no browser to bind") + } - if relayState == "" { - relayState = "/" + if returnTo == "" { + returnTo = "/" } if len(s.idpMetadata.IDPSSODescriptors) == 0 || len(s.idpMetadata.IDPSSODescriptors[0].SingleSignOnServices) == 0 { return "", errors.New("idp does not support single sign-on") } - // Use the simple redirect method - redirectURL, err := s.sp.MakeRedirectAuthenticationRequest(relayState) + // This is crewjam's MakeRedirectAuthenticationRequest, unrolled to learn + // the request's ID. + authnRequest, err := s.sp.MakeAuthenticationRequest(s.sp.GetSSOBindingLocation(saml.HTTPRedirectBinding), saml.HTTPRedirectBinding, saml.HTTPPostBinding) + if err != nil { + return "", fmt.Errorf("failed to create auth request: %w", err) + } + // Redirect appends RelayState to the query as given, so escape it: a + // returnTo with its own query string would otherwise split into stray + // parameters and come back from the IdP truncated. + redirectURL, err := authnRequest.Redirect(url.QueryEscape(returnTo), s.sp) if err != nil { return "", fmt.Errorf("failed to create auth request: %w", err) } @@ -483,11 +643,35 @@ func (s *SAMLService) MakeAuthRequest(relayState string) (string, error) { if err != nil { return "", fmt.Errorf("failed to validate auth redirect: %w", err) } + s.authnRequests.put(authnRequest.ID, samlRequestRecord{bindingKey: browserKey, returnTo: returnTo}, time.Now()) return validatedURL, nil } -// ProcessResponse processes a SAML response and extracts user information -func (s *SAMLService) ProcessResponse(r *http.Request) (*SAMLAuthResult, string, error) { +// AllowsIDPInitiated reports whether the provider accepts IdP-initiated +// (unsolicited) Responses, which answer no AuthnRequest and so cannot be bound +// to the browser that receives them. +func (s *SAMLService) AllowsIDPInitiated() bool { + s.mu.RLock() + defer s.mu.RUnlock() + return s.sp != nil && s.sp.AllowIDPInitiated +} + +// ProcessResponse verifies the IdP's Response posted to the ACS and extracts +// the user it authenticates, along with the local path the login returns to. +// +// Unless the provider allows IdP-initiated logins, the Response must answer +// an unexpired AuthnRequest from MakeAuthRequest that is bound to browserKey, +// the login binding key of the presenting browser ("" when it carries none), +// and it spends that request, so each login is completed at most once and only +// in the browser that started it. The request ID is read from what the IdP's +// signatures cover, never from crewjam/saml's parsed fields (see +// samlSignedContent). The returned path is the one recorded with the request, +// not the posted RelayState. Binding failures wrap errSAMLResponseUnbound. +// +// With IdP-initiated logins allowed, crewjam skips InResponseTo entirely, so a +// Response answering any request, or none, is accepted in any browser and +// returns to its posted RelayState, as before request binding existed. +func (s *SAMLService) ProcessResponse(r *http.Request, browserKey string) (*SAMLAuthResult, string, error) { s.mu.RLock() defer s.mu.RUnlock() @@ -502,18 +686,43 @@ func (s *SAMLService) ProcessResponse(r *http.Request) (*SAMLAuthResult, string, relayState := r.FormValue("RelayState") - // Allow IdP-initiated flow - possibleRequestIDs := []string{} - if s.sp.AllowIDPInitiated { - possibleRequestIDs = append(possibleRequestIDs, "") + // crewjam checks that the Response answers one of possibleRequestIDs, + // every outstanding AuthnRequest here; spend then decides whether it is + // this browser's. + now := time.Now() + possibleRequestIDs := []string{""} + if !s.sp.AllowIDPInitiated { + var bound bool + possibleRequestIDs, bound = s.authnRequests.outstanding(browserKey, now) + if !bound { + return nil, relayState, fmt.Errorf("%w: this browser has no login in progress", errSAMLResponseUnbound) + } } - // Parse and validate the SAML assertion - assertion, err := s.sp.ParseResponse(r, possibleRequestIDs) + // Parse and validate the SAML assertion on a copy of the service provider + // whose signature verifier keeps what each verified signature covers. + signed := &samlSignedContent{} + sp := *s.sp + sp.SignatureVerifier = signed + assertion, err := sp.ParseResponse(r, possibleRequestIDs) if err != nil { return nil, relayState, fmt.Errorf("failed to validate saml response: %w", err) } + if !s.sp.AllowIDPInitiated { + requestID, err := signed.inResponseTo(assertion.ID) + if err != nil { + return nil, relayState, fmt.Errorf("%w: %v", errSAMLResponseUnbound, err) + } + // Artifact resolution and signature checks take time, so judge + // the request's expiry now rather than when parsing began. + request, err := s.authnRequests.spend(requestID, browserKey, time.Now()) + if err != nil { + return nil, relayState, err + } + relayState = request.returnTo + } + // Extract user information from assertion result := &SAMLAuthResult{ Attributes: make(map[string][]string), @@ -594,14 +803,19 @@ func (s *SAMLService) GetMetadata() ([]byte, error) { return xml.MarshalIndent(metadata, "", " ") } -// MakeLogoutRequest creates a SAML LogoutRequest for SLO -func (s *SAMLService) MakeLogoutRequest(nameID, sessionIdx string) (string, error) { +// MakeLogoutRequest creates a SAML LogoutRequest for SLO and records its ID +// as outstanding for sessionKey, the session being logged out, so that +// ValidateLogoutResponse accepts only the IdP's answer to this request. +func (s *SAMLService) MakeLogoutRequest(nameID, sessionIdx, sessionKey string) (string, error) { s.mu.RLock() defer s.mu.RUnlock() if s.sp == nil { return "", errors.New("service provider not initialized") } + if sessionKey == "" { + return "", errors.New("logout request has no session to bind") + } // Check if IdP supports SLO if len(s.idpMetadata.IDPSSODescriptors) == 0 || @@ -619,21 +833,32 @@ func (s *SAMLService) MakeLogoutRequest(nameID, sessionIdx string) (string, erro // Build redirect URL redirectURL := req.Redirect("") - return validateSAMLRedirectTarget(redirectURL.String(), s.idpMetadata.IDPSSODescriptors[0].SingleLogoutServices) + validatedURL, err := validateSAMLRedirectTarget(redirectURL.String(), s.idpMetadata.IDPSSODescriptors[0].SingleLogoutServices) + if err != nil { + return "", err + } + s.logoutRequests.put(req.ID, samlRequestRecord{bindingKey: sessionKey}, time.Now()) + return validatedURL, nil } // ValidateLogoutResponse verifies an incoming IdP-signed SAML LogoutResponse, // checking the XML-DSig signature against the IdP's published certificate as -// well as the standard LogoutResponse temporal / target invariants. Returns -// nil only when the response is genuinely from the configured IdP and bound -// to the SAML state — anything that fails validation (no payload, bad -// signature, unknown issuer, expired) returns an error and the caller MUST -// NOT mutate any session state on the strength of the request. +// well as the standard LogoutResponse temporal / target invariants, and then +// binds it to the request it answers: its InResponseTo must name an +// unexpired LogoutRequest from MakeLogoutRequest, which it spends, and +// sessionKey (the presenting browser's session, or "" for none) must be empty +// or the session that request logged out. Returns nil only when all of that +// holds — anything else (no payload, bad signature, unknown issuer, expired, +// unsolicited, replayed, or another session's logout) returns an error and +// the caller MUST NOT mutate any session state on the strength of the +// request. Binding failures wrap errSAMLLogoutResponseUnbound. // -// Without this validation the SLO callback endpoint was an unauthenticated -// force-logout DoS: a cross-origin POST with no payload, or any forged -// payload, cleared the user's session. -func (s *SAMLService) ValidateLogoutResponse(req *http.Request) error { +// Without signature validation the SLO callback endpoint was an +// unauthenticated force-logout DoS: a cross-origin POST with no payload, or +// any forged payload, cleared the user's session. Before the binding, any +// user could replay the IdP's answer to their own logout through a cross-site +// GET to clear someone else's session. +func (s *SAMLService) ValidateLogoutResponse(req *http.Request, sessionKey string) error { s.mu.RLock() defer s.mu.RUnlock() @@ -641,7 +866,255 @@ func (s *SAMLService) ValidateLogoutResponse(req *http.Request) error { return errors.New("service provider not initialized") } - return s.sp.ValidateLogoutResponseRequest(req) + inResponseTo, err := s.verifiedLogoutResponseInResponseTo(req) + if err != nil { + return err + } + return s.logoutRequests.consume(inResponseTo, sessionKey, time.Now()) +} + +// verifiedLogoutResponseInResponseTo selects the SAMLResponse payload the way +// crewjam/saml's ValidateLogoutResponseRequest does (the HTTP-Redirect query +// parameter first, then the HTTP-POST form field), has crewjam verify exactly +// that payload, and reads InResponseTo from the same bytes. +func (s *SAMLService) verifiedLogoutResponseInResponseTo(req *http.Request) (string, error) { + var raw []byte + if data := req.URL.Query().Get("SAMLResponse"); data != "" { + if err := s.sp.ValidateLogoutResponseRedirect(data); err != nil { + return "", err + } + compressed, err := base64.StdEncoding.DecodeString(data) + if err != nil { + return "", fmt.Errorf("decode logout response: %w", err) + } + raw, err = io.ReadAll(io.LimitReader(flate.NewReader(bytes.NewReader(compressed)), samlLogoutResponseMaxBytes)) + if err != nil { + return "", fmt.Errorf("inflate logout response: %w", err) + } + } else { + if err := req.ParseForm(); err != nil { + return "", fmt.Errorf("parse logout response form: %w", err) + } + data := req.PostForm.Get("SAMLResponse") + if err := s.sp.ValidateLogoutResponseForm(data); err != nil { + return "", err + } + var err error + raw, err = base64.StdEncoding.DecodeString(data) + if err != nil { + return "", fmt.Errorf("decode logout response: %w", err) + } + } + return logoutResponseInResponseTo(raw) +} + +// logoutResponseInResponseTo returns the root LogoutResponse's InResponseTo +// attribute as written without a prefix, the one the IdP's enveloped signature +// covers, or "" when it has none. The attributes are read by hand, lexically, +// because encoding/xml would also fill an un-namespaced attribute field from a +// namespace declaration such as xmlns:InResponseTo="...", which exclusive +// canonicalization can leave out of the signed form when nothing uses it. crewjam +// has already checked that the root is a protocol LogoutResponse. +func logoutResponseInResponseTo(raw []byte) (string, error) { + decoder := xml.NewDecoder(bytes.NewReader(raw)) + for { + token, err := decoder.RawToken() + if err != nil { + return "", fmt.Errorf("parse logout response: %w", err) + } + start, ok := token.(xml.StartElement) + if !ok { + continue + } + if start.Name.Local != "LogoutResponse" { + return "", fmt.Errorf("parse logout response: unexpected root element %q", start.Name.Local) + } + inResponseTo, found := "", false + for _, attr := range start.Attr { + if attr.Name.Space != "" || attr.Name.Local != "InResponseTo" { + continue + } + if found { + return "", errors.New("parse logout response: duplicate InResponseTo") + } + inResponseTo, found = attr.Value, true + } + return inResponseTo, nil + } +} + +// samlSignedContent is the signature verifier ProcessResponse gives crewjam/saml +// for one ACS Response. It verifies each signature exactly as crewjam's default +// verifier does and keeps the content that signature covers: the canonical form +// goxmldsig digested, which holds only what the IdP signed. +// +// crewjam reads a Response's fields by unmarshalling the posted elements with +// encoding/xml, which fills an un-namespaced attribute field from any attribute +// with that local name, whatever its prefix, the last one winning. That +// includes a namespace declaration such as xmlns:IssueInstant, and exclusive +// canonicalization leaves a declaration nothing uses out of the signed form, +// so one can be appended to a signed element without breaking its signature +// and override what crewjam checks: InResponseTo, IssueInstant, NotOnOrAfter, +// Recipient, Destination. VerifySignature therefore refuses signed content +// carrying any prefixed attribute or declaration named like an un-namespaced +// attribute crewjam reads, so crewjam's fields match what was signed, and the +// binding decision still reads InResponseTo from the kept canonical content +// rather than from crewjam's fields. +type samlSignedContent struct { + elements []*etree.Element +} + +func (c *samlSignedContent) VerifySignature(validationContext *dsig.ValidationContext, el *etree.Element) error { + if shadow := samlShadowingAttribute(el); shadow != "" { + return fmt.Errorf("cannot validate signature on %s: %s shadows a SAML attribute", el.Tag, shadow) + } + signed, err := validationContext.Validate(el) + if err != nil { + return fmt.Errorf("cannot validate signature on %s: %v", el.Tag, err) + } + c.elements = append(c.elements, signed) + return nil +} + +// inResponseTo returns the AuthnRequest ID that the signature covering the +// assertion crewjam returned, identified by its ID, says it answers. +// +// When the Response, or an ArtifactResponse carrying it, is signed, the IdP +// signed the whole message, the returned assertion included, and the signed +// Response's InResponseTo names the request. Otherwise crewjam verified the +// returned assertion's own signature, and the request is named by that +// assertion's SubjectConfirmationData: every SubjectConfirmation of its +// Subject must carry the same one, and at least one must exist, because +// crewjam accepts a Subject without any. Other signed assertions in the same +// Response, and assertions in Advice, do not take part. +func (c *samlSignedContent) inResponseTo(assertionID string) (string, error) { + for _, el := range c.elements { + response := el + if el.Tag == "ArtifactResponse" { + response = samlChildElement(el, "Response") + } + if response == nil || response.Tag != "Response" { + continue + } + if requestID := samlUnprefixedAttr(response, "InResponseTo"); requestID != "" { + return requestID, nil + } + return "", errors.New("the signed Response answers no request") + } + var assertion *etree.Element + for _, el := range c.elements { + if el.Tag != "Assertion" || samlUnprefixedAttr(el, "ID") != assertionID { + continue + } + if assertion != nil { + return "", fmt.Errorf("several signed assertions carry ID %q", assertionID) + } + assertion = el + } + if assertion == nil { + return "", errors.New("no signature covers the returned assertion") + } + requestID := "" + if subject := samlChildElement(assertion, "Subject"); subject != nil { + for _, confirmation := range subject.ChildElements() { + if confirmation.Tag != "SubjectConfirmation" { + continue + } + // crewjam keeps the last SubjectConfirmationData of each + // confirmation, so every one must agree. + named := false + for _, data := range confirmation.ChildElements() { + if data.Tag != "SubjectConfirmationData" { + continue + } + value := samlUnprefixedAttr(data, "InResponseTo") + if value == "" || (requestID != "" && value != requestID) { + return "", errors.New("the signed assertion's subject confirmations do not name one request") + } + requestID, named = value, true + } + if !named { + return "", errors.New("a signed subject confirmation names no request") + } + } + } + if requestID == "" { + return "", errors.New("the signed assertion has no subject confirmation naming a request") + } + return requestID, nil +} + +// samlChildElement returns el's first child element with local name tag. +func samlChildElement(el *etree.Element, tag string) *etree.Element { + for _, child := range el.ChildElements() { + if child.Tag == tag { + return child + } + } + return nil +} + +// samlUnprefixedAttr returns el's attribute key written without a prefix, the +// one a signature over el covers, or "" when it has none. +func samlUnprefixedAttr(el *etree.Element, key string) string { + for _, attr := range el.Attr { + if attr.Space == "" && attr.Key == key { + return attr.Value + } + } + return "" +} + +// samlBoundAttributeNames holds the name of every un-namespaced XML attribute +// crewjam/saml unmarshals from a Response or an Assertion. Namespaced ones such +// as xsi:type only match their own namespace. +var samlBoundAttributeNames = func() map[string]bool { + names := make(map[string]bool) + seen := make(map[reflect.Type]bool) + var collect func(t reflect.Type) + collect = func(t reflect.Type) { + for t.Kind() == reflect.Pointer || t.Kind() == reflect.Slice || t.Kind() == reflect.Array { + t = t.Elem() + } + if t.Kind() != reflect.Struct || seen[t] { + return + } + seen[t] = true + for i := 0; i < t.NumField(); i++ { + field := t.Field(i) + name, options, _ := strings.Cut(field.Tag.Get("xml"), ",") + if strings.Contains(","+options+",", ",attr,") { + if name == "" { + name = field.Name + } + if !strings.Contains(name, " ") { + names[name] = true + } + } + collect(field.Type) + } + } + collect(reflect.TypeOf(saml.Response{})) + collect(reflect.TypeOf(saml.Assertion{})) + return names +}() + +// samlShadowingAttribute returns the first prefixed attribute or namespace +// declaration in el or its descendants whose local name is also an +// un-namespaced attribute crewjam/saml reads, as written, or "" when there is +// none. +func samlShadowingAttribute(el *etree.Element) string { + for _, attr := range el.Attr { + if attr.Space != "" && samlBoundAttributeNames[attr.Key] { + return attr.Space + ":" + attr.Key + } + } + for _, child := range el.ChildElements() { + if shadow := samlShadowingAttribute(child); shadow != "" { + return shadow + } + } + return "" } func validateSAMLRedirectTarget(rawURL string, allowedEndpoints []saml.Endpoint) (string, error) { diff --git a/internal/api/saml_service_additional_test.go b/internal/api/saml_service_additional_test.go index 4ce28c052d..3674b1d2ff 100644 --- a/internal/api/saml_service_additional_test.go +++ b/internal/api/saml_service_additional_test.go @@ -46,7 +46,7 @@ func TestProcessResponse_InvalidResponse(t *testing.T) { req := httptest.NewRequest(http.MethodPost, "/acs", body) req.Header.Set("Content-Type", "application/x-www-form-urlencoded") - _, relay, err := service.ProcessResponse(req) + _, relay, err := service.ProcessResponse(req, "") if err == nil { t.Fatal("expected error for invalid response") } diff --git a/internal/api/saml_service_test.go b/internal/api/saml_service_test.go index 7de7a75704..36f85490fb 100644 --- a/internal/api/saml_service_test.go +++ b/internal/api/saml_service_test.go @@ -7,13 +7,17 @@ import ( "crypto/rand" "crypto/rsa" "crypto/x509" + "encoding/base64" "encoding/pem" + "errors" + "fmt" "math/big" "net/http" "strings" "testing" "time" + "github.com/beevik/etree" "github.com/crewjam/saml" "github.com/rcourtman/pulse-go-rewrite/internal/config" ) @@ -88,6 +92,7 @@ func TestBuildManualMetadataAndCertificate(t *testing.T) { cfg.IDPIssuer = "issuer" certPEM, _, _ := generateTestCert(t) cfg.IDPCertificate = string(certPEM) + certBlock, _ := pem.Decode(certPEM) metadata, err := service.buildManualMetadata() if err != nil { @@ -102,6 +107,14 @@ func TestBuildManualMetadataAndCertificate(t *testing.T) { if len(metadata.IDPSSODescriptors[0].KeyDescriptors) == 0 { t.Fatal("expected key descriptor with certificate") } + // crewjam/saml reads content: base64 DER, never PEM. + certs := metadata.IDPSSODescriptors[0].KeyDescriptors[0].KeyInfo.X509Data.X509Certificates + if len(certs) != 1 { + t.Fatalf("expected one signing certificate, got %d", len(certs)) + } + if certs[0].Data != base64.StdEncoding.EncodeToString(certBlock.Bytes) { + t.Fatalf("expected the signing certificate as base64 DER, got %.40q", certs[0].Data) + } cfg.IDPSSOURL = "https://user:pass@idp.example.com/sso" if _, err := service.buildManualMetadata(); err == nil { @@ -175,7 +188,7 @@ func TestSAMLServiceBasicFlows(t *testing.T) { t.Fatalf("new service: %v", err) } - url, err := service.MakeAuthRequest("") + url, err := service.MakeAuthRequest("", sessionHash("login-token")) if err != nil || !strings.Contains(url, "SAMLRequest") { t.Fatalf("unexpected auth url: %v %s", err, url) } @@ -184,19 +197,19 @@ func TestSAMLServiceBasicFlows(t *testing.T) { t.Fatalf("metadata error: %v", err) } - logoutURL, err := service.MakeLogoutRequest("user", "sess") + logoutURL, err := service.MakeLogoutRequest("user", "sess", sessionHash("session-token")) if err != nil || !strings.Contains(logoutURL, "SAMLRequest") { t.Fatalf("unexpected logout url: %v %s", err, logoutURL) } service = &SAMLService{config: &config.SAMLProviderConfig{}} - if _, err := service.MakeAuthRequest(""); err == nil { + if _, err := service.MakeAuthRequest("", sessionHash("login-token")); err == nil { t.Fatal("expected error when sp missing") } if _, err := service.GetMetadata(); err == nil { t.Fatal("expected error when sp missing") } - if _, err := service.MakeLogoutRequest("user", "sess"); err == nil { + if _, err := service.MakeLogoutRequest("user", "sess", sessionHash("session-token")); err == nil { t.Fatal("expected error when sp missing") } if err := service.RefreshMetadata(context.Background()); err == nil { @@ -204,6 +217,175 @@ func TestSAMLServiceBasicFlows(t *testing.T) { } } +// Outstanding SLO requests expire after samlRequestTTL, and the store keeps +// at most maxSAMLRequestEntries by dropping the oldest. +func TestSAMLLogoutRequestStoreExpiresAndEvictsOldest(t *testing.T) { + var store samlRequestStore + now := time.Now() + key := sessionHash("session-token") + record := samlRequestRecord{bindingKey: key} + + store.put("id-expired", record, now) + if err := store.consume("id-expired", key, now.Add(samlRequestTTL)); !errors.Is(err, errSAMLLogoutResponseUnbound) { + t.Fatalf("expected an expired request to be refused, got %v", err) + } + store.put("id-live", record, now) + if err := store.consume("id-live", "", now.Add(samlRequestTTL-time.Second)); err != nil { + t.Fatalf("expected a live request to be accepted, got %v", err) + } + + for i := 0; i <= maxSAMLRequestEntries; i++ { + store.put(fmt.Sprintf("id-%04d", i), record, now.Add(time.Duration(i)*time.Millisecond)) + } + if len(store.entries) != maxSAMLRequestEntries { + t.Fatalf("expected %d outstanding requests, got %d", maxSAMLRequestEntries, len(store.entries)) + } + if err := store.consume("id-0000", key, now); !errors.Is(err, errSAMLLogoutResponseUnbound) { + t.Fatalf("expected the oldest request to be evicted, got %v", err) + } + if err := store.consume(fmt.Sprintf("id-%04d", maxSAMLRequestEntries), key, now); err != nil { + t.Fatalf("expected the newest request to be kept, got %v", err) + } +} + +// Outstanding AuthnRequests are spent only by the browser they are bound to. A +// request presented by another browser stays outstanding for its owner, and +// each request is spent once and only before it expires. +func TestSAMLAuthnRequestStoreSpendsOnlyForItsBrowser(t *testing.T) { + var store samlRequestStore + now := time.Now() + alice, bob := sessionHash("alice-login"), sessionHash("bob-login") + + store.put("id-alice-2", samlRequestRecord{bindingKey: alice, returnTo: "/alerts"}, now) + store.put("id-alice-1", samlRequestRecord{bindingKey: alice, returnTo: "/"}, now) + store.put("id-bob", samlRequestRecord{bindingKey: bob, returnTo: "/"}, now) + + if got, bound := store.outstanding(alice, now); strings.Join(got, ",") != "id-alice-1,id-alice-2,id-bob" || !bound { + t.Fatalf("outstanding(alice) = %v, %v; want every request, bound", got, bound) + } + if _, bound := store.outstanding("", now); bound { + t.Fatal("a browser without a login token has no request bound to it") + } + if _, err := store.spend("id-bob", alice, now); !errors.Is(err, errSAMLResponseUnbound) { + t.Fatalf("expected another browser's request to be refused, got %v", err) + } + if _, err := store.spend("id-bob", "", now); !errors.Is(err, errSAMLResponseUnbound) { + t.Fatalf("expected a browser without a login token to be refused, got %v", err) + } + if record, err := store.spend("id-bob", bob, now); err != nil || record.returnTo != "/" { + t.Fatalf("refusing other browsers must leave the request for its owner, got %+v %v", record, err) + } + record, err := store.spend("id-alice-2", alice, now) + if err != nil || record.returnTo != "/alerts" { + t.Fatalf("spend(id-alice-2) = %+v %v, want its recorded returnTo", record, err) + } + if _, err := store.spend("id-alice-2", alice, now); !errors.Is(err, errSAMLResponseUnbound) { + t.Fatalf("expected a spent request to be refused, got %v", err) + } + if got, bound := store.outstanding(alice, now.Add(samlRequestTTL)); len(got) != 0 || bound { + t.Fatalf("expired requests must not be outstanding, got %v, %v", got, bound) + } + if _, err := store.spend("id-alice-1", alice, now.Add(samlRequestTTL)); !errors.Is(err, errSAMLResponseUnbound) { + t.Fatalf("expected an expired request to be refused, got %v", err) + } +} + +// The binding decision reads InResponseTo from the canonical content of the +// signature that covers the assertion crewjam returned, taking only the +// attribute written without a prefix: the signed Response's when the message +// is signed, otherwise the returned assertion's own subject confirmations. +func TestSAMLSignedContentInResponseTo(t *testing.T) { + parse := func(doc string) *etree.Element { + t.Helper() + parsed := etree.NewDocument() + if err := parsed.ReadFromString(doc); err != nil { + t.Fatalf("parse %s: %v", doc, err) + } + return parsed.Root() + } + assertion := func(id, subject, advice string) string { + return `` + subject + `` + advice + `` + } + confirmation := func(attrs string) string { + return `` + } + response := func(attrs, body string) string { + return `` + body + `` + } + own, other := `InResponseTo="id-own"`, `InResponseTo="id-other"` + advice := `` + assertion("a9", confirmation(other), "") + `` + + for _, tc := range []struct { + name string + signed []string + want string + }{ + {name: "signed Response", signed: []string{response(`xmlns:InResponseTo="id-other" `+own, assertion("a1", confirmation(other), ""))}, want: "id-own"}, + {name: "signed Response with encrypted Assertion", signed: []string{response(own, ``)}, want: "id-own"}, + {name: "signed ArtifactResponse", signed: []string{`` + response(own, "") + ``}, want: "id-own"}, + {name: "signed Assertion", signed: []string{assertion("a1", confirmation(`xmlns:InResponseTo="id-other" `+own), advice)}, want: "id-own"}, + {name: "signed Response answering no request", signed: []string{response(`xmlns:InResponseTo="id-own"`, assertion("a1", confirmation(own), ""))}}, + {name: "subject confirmations disagree", signed: []string{assertion("a1", confirmation(own)+confirmation(other), "")}}, + {name: "subject confirmation naming no request", signed: []string{assertion("a1", confirmation(own)+confirmation(`Recipient="x"`), "")}}, + {name: "subject confirmation without data", signed: []string{assertion("a1", confirmation(own)+``, "")}}, + {name: "one subject confirmation with conflicting data", signed: []string{assertion("a1", ``, "")}}, + {name: "assertion without subject confirmation", signed: []string{assertion("a1", "", advice)}}, + {name: "another signed assertion names the request", signed: []string{assertion("a1", "", ""), assertion("a2", confirmation(own), "")}}, + {name: "returned assertion not signed", signed: []string{assertion("a2", confirmation(own), "")}}, + {name: "two signed assertions share the returned ID", signed: []string{assertion("a1", confirmation(own), ""), assertion("a1", confirmation(own), "")}}, + } { + t.Run(tc.name, func(t *testing.T) { + content := &samlSignedContent{} + for _, doc := range tc.signed { + content.elements = append(content.elements, parse(doc)) + } + got, err := content.inResponseTo("a1") + if tc.want == "" { + if err == nil { + t.Fatalf("expected a refusal, got %q", got) + } + return + } + if err != nil || got != tc.want { + t.Fatalf("inResponseTo() = %q, %v; want %q", got, err, tc.want) + } + }) + } +} + +// Signed content may not carry a prefixed attribute or namespace declaration +// named like an un-namespaced attribute crewjam/saml reads; the declarations +// and namespaced attributes real IdPs write are untouched. +func TestSAMLShadowingAttribute(t *testing.T) { + for _, name := range []string{"ID", "InResponseTo", "IssueInstant", "Destination", "NotBefore", "NotOnOrAfter", "Recipient", "SessionIndex"} { + if !samlBoundAttributeNames[name] { + t.Fatalf("expected %s among the attributes crewjam/saml reads, got %v", name, samlBoundAttributeNames) + } + } + for _, prefix := range []string{"saml", "samlp", "saml2", "saml2p", "ds", "dsig", "xs", "xsd", "xsi", "xenc", "ec", "md", "type"} { + if samlBoundAttributeNames[prefix] { + t.Fatalf("%q, a namespace prefix or namespaced attribute real IdPs write, is refused", prefix) + } + } + + const ns = `xmlns:saml="urn:oasis:names:tc:SAML:2.0:assertion" xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:ext="urn:example"` + for _, tc := range []struct { + doc, want string + }{ + {doc: ``, want: "xmlns:NotOnOrAfter"}, + {doc: ``, want: "ext:ID"}, + {doc: `a@example.com`}, + } { + doc := etree.NewDocument() + if err := doc.ReadFromString(tc.doc); err != nil { + t.Fatalf("parse %s: %v", tc.doc, err) + } + if got := samlShadowingAttribute(doc.Root()); got != tc.want { + t.Fatalf("samlShadowingAttribute(%s) = %q, want %q", tc.doc, got, tc.want) + } + } +} + func TestValidateSAMLRedirectTarget(t *testing.T) { allowed := []saml.Endpoint{ {Location: "https://idp.example.com/sso"}, diff --git a/internal/monitoring/canonical_guardrails_test.go b/internal/monitoring/canonical_guardrails_test.go index 0df5ff6094..e9646c53c8 100644 --- a/internal/monitoring/canonical_guardrails_test.go +++ b/internal/monitoring/canonical_guardrails_test.go @@ -650,7 +650,8 @@ func TestBroadcastResourceProjectionCoalescesSplitHostIdentities(t *testing.T) { for _, snippet := range []string{ "metricsTargetResolver := unifiedView.metricsTargets", "metricsTargetResolver = broadcastMetricsTargetResolver(unifiedView.readState)", - "broadcastResources := unifiedresources.CoalescePresentationHostResources(unifiedView.resources)", + "broadcastResources := m.coalesceResourcesForPresentation(unifiedView.readState, unifiedView.resources)", + "if coalesced, ok := coalescer.CoalesceForPresentation(resources); ok {", "broadcastFrontendResources, broadcastCatalogs := convertPresentationResourcesForBroadcast(broadcastResources)", "attachBroadcastMetricsTargetsInPlace(broadcastResources, metricsTargetResolver)", "frontendState.CapabilityCatalog = broadcastCatalogs.capabilities", diff --git a/internal/monitoring/issue1913_host_continuity_test.go b/internal/monitoring/issue1913_host_continuity_test.go index 5134592e7b..b5b93a56c5 100644 --- a/internal/monitoring/issue1913_host_continuity_test.go +++ b/internal/monitoring/issue1913_host_continuity_test.go @@ -4,6 +4,7 @@ import ( "testing" "time" + "github.com/rcourtman/pulse-go-rewrite/internal/alerts" "github.com/rcourtman/pulse-go-rewrite/internal/config" "github.com/rcourtman/pulse-go-rewrite/internal/models" "github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources" @@ -89,3 +90,161 @@ func TestHostContinuityPreservesLiveAgentAfterReenrollment(t *testing.T) { }) } } + +// Continuity hydration clones the registry through resource ingest, which +// applies operator links, while the rebuild ingests supplemental records after +// the snapshot. A link to a record-sourced guest (an agent inside a vSphere VM) +// must hold in the published inventory either way, or the state the websocket +// broadcasts and the resources API seeds from flips with an unrelated absent +// machine. A saved enrollment only fills an absent machine: when the linked +// agent itself is gone, its continuity row stays separate and the guest keeps +// what vSphere reports. The link still names one identity, so references to +// the agent reach the guest in every case: alert intent reads maintenance set +// on the guest for the agent's alerts, and the resources API, which seeds a +// registry from the read state and re-applies the links, agrees without +// merging the saved agent into the guest. +func TestManualLinkToSupplementalGuestHoldsWithAndWithoutContinuity(t *testing.T) { + now := time.Now().UTC() + guestAgent := models.Host{ + ID: "host-app-guest", MachineID: "machine-app-guest", Hostname: "app-guest", + Platform: "linux", Status: "online", LastSeen: now, IntervalSeconds: 30, + } + vmRecords := []unifiedresources.IngestRecord{{ + SourceID: "vc-1:vm:vm-42", + Resource: unifiedresources.Resource{ + Type: unifiedresources.ResourceTypeVM, + Technology: "vmware", + Name: "app-guest", + Status: unifiedresources.StatusOnline, + LastSeen: now, + VMware: &unifiedresources.VMwareData{ConnectionID: "vc-1", ManagedObjectID: "vm-42", EntityType: "vm"}, + }, + }} + + unlinked := unifiedresources.NewMonitorAdapter(unifiedresources.NewRegistry(nil)) + unlinked.PopulateSnapshotAndSupplemental( + models.StateSnapshot{Hosts: []models.Host{guestAgent}, LastUpdate: now}, + map[unifiedresources.DataSource][]unifiedresources.IngestRecord{unifiedresources.SourceVMware: vmRecords}, + ) + if len(unlinked.VMs()) != 1 || len(unlinked.Hosts()) != 1 { + t.Fatalf("unlinked estate = %d VMs, %d agents, want one of each", len(unlinked.VMs()), len(unlinked.Hosts())) + } + vmID, agentID := unlinked.VMs()[0].ID(), unlinked.Hosts()[0].ID() + + for _, tc := range []struct { + name string + reporting bool + continuity []config.HostContinuityEntry + }{ + {name: "no-continuity", reporting: true}, + { + name: "absent-machine-continuity", + reporting: true, + continuity: []config.HostContinuityEntry{{ + HostID: "absent-machine", MachineID: "other-machine", Hostname: "other-host", + Platform: "linux", LastSeen: now.Add(-time.Hour), + }}, + }, + { + name: "linked-agent-continuity-only", + continuity: []config.HostContinuityEntry{{ + HostID: guestAgent.ID, MachineID: guestAgent.MachineID, Hostname: guestAgent.Hostname, + Platform: "linux", LastSeen: now.Add(-10 * time.Minute), + }}, + }, + } { + t.Run(tc.name, func(t *testing.T) { + links := unifiedresources.NewMemoryStore() + if err := links.AddLink(unifiedresources.ResourceLink{ResourceA: vmID, ResourceB: agentID, PrimaryID: vmID}); err != nil { + t.Fatalf("add link: %v", err) + } + start, end := now.Add(-time.Hour), now.Add(time.Hour) + if err := links.SetResourceOperatorState(unifiedresources.ResourceOperatorState{ + CanonicalID: vmID, MaintenanceStartAt: &start, MaintenanceEndAt: &end, MaintenanceReason: "guest migration", + }); err != nil { + t.Fatalf("set guest maintenance: %v", err) + } + state := models.NewState() + if tc.reporting { + state.Hosts = []models.Host{guestAgent} + } + manager := alerts.NewManagerWithDataDir(t.TempDir()) + t.Cleanup(manager.Stop) + m := &Monitor{state: state, alertManager: manager} + if len(tc.continuity) > 0 { + continuity := config.NewHostContinuityStore(t.TempDir(), nil) + for _, entry := range tc.continuity { + if err := continuity.Upsert(entry); err != nil { + t.Fatal(err) + } + } + m.hostContinuityStore = continuity + } + m.SetResourceStore(unifiedresources.NewMonitorAdapter(unifiedresources.NewRegistry(links))) + m.SetSupplementalRecordsProvider(unifiedresources.SourceVMware, &testSupplementalInventoryReadinessProvider{ + source: unifiedresources.SourceVMware, + records: vmRecords, + settled: true, + readyAt: now, + }) + + resources, _ := m.UnifiedResourceSnapshot() + var guest, agent *unifiedresources.Resource + for i := range resources { + switch resources[i].ID { + case agentID: + agent = &resources[i] + case vmID: + guest = &resources[i] + } + } + if guest == nil { + t.Fatalf("linked VM %s missing from published inventory", vmID) + } + preview, err := manager.PreviewIntentPolicy(alerts.AlertIntentPolicyPreviewRequest{ + ResourceID: "agent:" + guestAgent.ID, ResourceType: "agent", + Signal: string(alerts.AlertIntentSignalOffline), ConditionActive: true, + }) + if err != nil { + t.Fatal(err) + } + if preview.Reason != "operator_maintenance" { + t.Fatalf("agent offline alert intent = %+v, want the guest's maintenance window", preview) + } + readState, ok := m.GetUnifiedReadStateOrSnapshot().(interface { + ResolveCanonicalResourceID(string) (string, bool) + }) + if !ok { + t.Fatal("monitor read state cannot resolve references") + } + if resolved, ok := readState.ResolveCanonicalResourceID("agent:" + guestAgent.ID); !ok || resolved != vmID { + t.Fatalf("read state resolved the agent to %q (%v), want the guest %s", resolved, ok, vmID) + } + resourcesAPI := unifiedresources.NewRegistry(links) + resourcesAPI.IngestResources(resources) + for _, ref := range []string{agentID, "agent:" + guestAgent.ID} { + if resolved, ok := resourcesAPI.ResolveReferenceID(ref); !ok || resolved != vmID { + t.Fatalf("resources API resolved %q to %q (%v), want the guest %s", ref, resolved, ok, vmID) + } + } + if apiGuest, ok := resourcesAPI.Get(vmID); !ok || (!tc.reporting && apiGuest.Agent != nil) { + t.Fatalf("resources API guest = %+v (listed=%v), want it free of the saved enrollment", apiGuest, ok) + } + if tc.reporting { + if agent != nil { + t.Fatalf("linked agent %s still published standalone", agentID) + } + if guest.VMware == nil || guest.Agent == nil { + t.Fatalf("published VM vmware=%t agent=%t, want the VM carrying its agent", guest.VMware != nil, guest.Agent != nil) + } + return + } + if agent == nil || agent.Status != unifiedresources.StatusOffline { + t.Fatalf("saved enrollment row = %+v, want its own offline row", agent) + } + if guest.Agent != nil || guest.Status != unifiedresources.StatusOnline { + t.Fatalf("published VM agent=%t status=%s, want vSphere's online VM untouched by the saved enrollment", guest.Agent != nil, guest.Status) + } + }) + } +} diff --git a/internal/monitoring/monitor.go b/internal/monitoring/monitor.go index 73f19993f8..550f922c64 100644 --- a/internal/monitoring/monitor.go +++ b/internal/monitoring/monitor.go @@ -1249,6 +1249,7 @@ type Monitor struct { metricsHistory *MetricsHistory metricsStore *metrics.Store // Persistent SQLite metrics storage alertManager *alerts.Manager + operatorIntentIdentity atomic.Pointer[operatorIntentIdentity] // read state alert intent resolves through (monitor_alert_intent.go) alertResolvedAICallback func(*alerts.Alert) alertTriggeredAICallback func(*alerts.Alert) alertPushCallback func(*alerts.Alert) @@ -4478,7 +4479,7 @@ func (m *Monitor) buildBroadcastFrontendStateFromSnapshotWithClock(snapshot mode if metricsTargetResolver == nil { metricsTargetResolver = broadcastMetricsTargetResolver(unifiedView.readState) } - broadcastResources := unifiedresources.CoalescePresentationHostResources(unifiedView.resources) + broadcastResources := m.coalesceResourcesForPresentation(unifiedView.readState, unifiedView.resources) // Coalescing owns the outer slice. Decorate that one projection in place, // not three full-resource copies; nested store data is still read-only. healthAlerts := resourceHealthAlerts(frontendState.ActiveAlerts) @@ -4501,6 +4502,39 @@ func (m *Monitor) buildBroadcastFrontendStateFromSnapshotWithClock(snapshot mode return frontendState } +// presentationCoalescingResourceStore is implemented by resource stores whose +// registry can hold the operator's merge exclusions (MonitorAdapter). +type presentationCoalescingResourceStore interface { + CoalesceForPresentation(resources []unifiedresources.Resource) ([]unifiedresources.Resource, bool) +} + +// coalesceResourcesForPresentation applies the presentation host coalesce with +// the operator's merge exclusions, which the resources API's +// ListForPresentation honours too: a host pair the operator split must not +// show as one broadcast row while REST lists two. The read state that listed +// the rows supplies them when its registry is store-backed, as the resource +// store's adapter and a host-continuity overlay are; the mock view and other +// views built from an unified list carry none, so the resource store's +// adapter supplies them instead. +func (m *Monitor) coalesceResourcesForPresentation(readState unifiedresources.ReadState, resources []unifiedresources.Resource) []unifiedresources.Resource { + if coalescer, ok := readState.(presentationCoalescingResourceStore); ok { + if coalesced, ok := coalescer.CoalesceForPresentation(resources); ok { + return coalesced + } + } + if m != nil { + m.mu.RLock() + store := m.resourceStore + m.mu.RUnlock() + if coalescer, ok := store.(presentationCoalescingResourceStore); ok { + if coalesced, ok := coalescer.CoalesceForPresentation(resources); ok { + return coalesced + } + } + } + return unifiedresources.CoalescePresentationHostResources(resources) +} + // GetLiveStateSnapshot returns the underlying monitor state snapshot without // applying global mock mode overrides. // diff --git a/internal/monitoring/monitor_alert_intent.go b/internal/monitoring/monitor_alert_intent.go index e00ff0cada..1b5648b4ad 100644 --- a/internal/monitoring/monitor_alert_intent.go +++ b/internal/monitoring/monitor_alert_intent.go @@ -2,9 +2,11 @@ package monitoring import ( "strings" + "sync" "time" "github.com/rcourtman/pulse-go-rewrite/internal/alerts" + "github.com/rcourtman/pulse-go-rewrite/internal/mock" "github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources" "github.com/rs/zerolog/log" ) @@ -27,9 +29,20 @@ func (m *Monitor) installOperatorIntentResolver(store ResourceStoreInterface) { if m == nil || m.alertManager == nil { return } + identity := m.newOperatorIntentIdentity(store) + m.operatorIntentIdentity.Store(identity) + identity.refresh() + identityStore := func() any { + if identity == nil { + return store + } + return identity.current() + } identityReader, hasIdentityReader := store.(resourceIntentIdentityReader) if hasIdentityReader && identityReader != nil { - m.alertManager.SetResourceIntentIdentityResolver(identityReader.ResolveCanonicalResourceID) + m.alertManager.SetResourceIntentIdentityResolver(func(ref string) (string, bool) { + return identityStore().(resourceIntentIdentityReader).ResolveCanonicalResourceID(ref) + }) } else { m.alertManager.SetResourceIntentIdentityResolver(nil) } @@ -38,10 +51,11 @@ func (m *Monitor) installOperatorIntentResolver(store ResourceStoreInterface) { m.alertManager.SetOperatorIntentContextResolver(nil) return } - ancestorReader, hasAncestorReader := store.(resourceIntentAncestorReader) + _, hasAncestorReader := store.(resourceIntentAncestorReader) m.alertManager.SetOperatorIntentContextResolver(func(resourceID string, now time.Time) (alerts.OperatorIntentContext, bool) { + identity := identityStore() if hasIdentityReader { - if canonicalID, found := identityReader.ResolveCanonicalResourceID(resourceID); found { + if canonicalID, found := identity.(resourceIntentIdentityReader).ResolveCanonicalResourceID(resourceID); found { resourceID = canonicalID } } @@ -81,7 +95,7 @@ func (m *Monitor) installOperatorIntentResolver(store ResourceStoreInterface) { applyActiveMaintenance(state, resourceID, false) } if hasAncestorReader { - for _, ancestorID := range ancestorReader.ResolveCanonicalResourceAncestors(resourceID) { + for _, ancestorID := range identity.(resourceIntentAncestorReader).ResolveCanonicalResourceAncestors(resourceID) { ancestorState, ancestorFound, ancestorErr := reader.GetResourceOperatorState(ancestorID) if ancestorErr != nil { log.Warn().Err(ancestorErr).Str("resourceID", resourceID).Str("ancestorID", ancestorID).Msg("Failed to read inherited operator maintenance state") @@ -102,6 +116,114 @@ func (m *Monitor) installOperatorIntentResolver(store ResourceStoreInterface) { } } +// operatorIntentIdentity is the store alert intent resolves resource +// references and ancestors through: the published registry with saved hosts +// that have not reported since a restart overlaid, the read state Patrol +// resolves through and the resources API seeds from. A saved host exists only +// there, and an operator link holds it under the link's primary, so the +// published registry alone left the saved host's alerts reading their literal +// reference and missed intent set on that host or on the primary. +// +// The alert manager resolves under its own lock, once per metric check, so +// the overlay is kept for the published generation it was built from. +// Publication builds it before evaluating alerts (refresh, outside that +// lock); a lookup that races ahead of publication builds it itself. A +// saved-host change between generations shows on the next one. Mock mode +// carries no saved hosts: the published registry answers and any real-mode +// overlay is dropped. +type operatorIntentIdentity struct { + monitor *Monitor + store ResourceStoreInterface + published unifiedresources.ReadState + generations interface { + Generation() unifiedresources.RegistryGeneration + } + + mu sync.Mutex + built bool + builtFor unifiedresources.RegistryGeneration + readState any +} + +// newOperatorIntentIdentity returns nil for a store without generations or +// the reference and ancestor readers, which alert intent then uses as is. +func (m *Monitor) newOperatorIntentIdentity(store ResourceStoreInterface) *operatorIntentIdentity { + published, isReadState := store.(unifiedresources.ReadState) + generations, versioned := store.(interface { + Generation() unifiedresources.RegistryGeneration + }) + _, resolves := store.(resourceIntentIdentityReader) + _, walksAncestors := store.(resourceIntentAncestorReader) + if !isReadState || !versioned || !resolves || !walksAncestors { + return nil + } + return &operatorIntentIdentity{monitor: m, store: store, published: published, generations: generations} +} + +// current returns the read state for the current published generation. +func (i *operatorIntentIdentity) current() any { + if mock.IsMockEnabled() { + i.mu.Lock() + i.built, i.readState = false, nil + i.mu.Unlock() + return i.store + } + generation := i.generations.Generation() + i.mu.Lock() + if i.built && i.builtFor == generation { + readState := i.readState + i.mu.Unlock() + return readState + } + i.mu.Unlock() + return i.build(generation) +} + +// refresh builds the read state for the current published generation unless +// it is already built. +func (i *operatorIntentIdentity) refresh() { + if i == nil || mock.IsMockEnabled() { + return + } + generation := i.generations.Generation() + i.mu.Lock() + fresh := i.built && i.builtFor == generation + i.mu.Unlock() + if !fresh { + i.build(generation) + } +} + +// build overlays saved hosts on the published registry outside i.mu, so a +// concurrent lookup waits on neither the overlay nor the store reads it makes. +// It keeps the result only while the mock-mode epoch it started in and the +// generation it read are still current. The check and the store run inside +// the fence, so a mode switch, which advances the fence after flipping the +// mode, either waits for them or makes them refuse: a build that read no +// saved hosts because mock mode was on never stands in for real mode. +func (i *operatorIntentIdentity) build(generation unifiedresources.RegistryGeneration) any { + scope := i.monitor.mockModeFence.begin() + if mock.IsMockEnabled() { + return i.store + } + readState := any(i.store) + overlay := i.monitor.readStateWithStandaloneHostContinuity(i.published) + if _, ok := overlay.(resourceIntentIdentityReader); ok { + if _, ok := overlay.(resourceIntentAncestorReader); ok { + readState = overlay + } + } + scope.run(func() { + if mock.IsMockEnabled() || i.generations.Generation() != generation { + return + } + i.mu.Lock() + i.built, i.builtFor, i.readState = true, generation, readState + i.mu.Unlock() + }) + return readState +} + func (m *Monitor) resolveBackupIntentContext(_ string, instance, node string, vmid int, now time.Time) (alerts.BackupIntentContext, bool) { if m == nil || m.state == nil || vmid <= 0 { return alerts.BackupIntentContext{}, false diff --git a/internal/monitoring/monitor_alert_intent_test.go b/internal/monitoring/monitor_alert_intent_test.go index 1b3c0485e2..2ec3c86355 100644 --- a/internal/monitoring/monitor_alert_intent_test.go +++ b/internal/monitoring/monitor_alert_intent_test.go @@ -5,6 +5,8 @@ import ( "time" "github.com/rcourtman/pulse-go-rewrite/internal/alerts" + "github.com/rcourtman/pulse-go-rewrite/internal/config" + "github.com/rcourtman/pulse-go-rewrite/internal/mock" "github.com/rcourtman/pulse-go-rewrite/internal/models" "github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources" "github.com/rcourtman/pulse-go-rewrite/pkg/proxmox" @@ -302,3 +304,150 @@ func TestSyncAlertsToStateCarriesLiveMetricStatusOfHeldAlert(t *testing.T) { t.Fatalf("live status = %+v, want latched at 78 clearing at 75", status) } } + +// After a restart, a host that has not reported again exists only through +// saved-host continuity, and the host-offline alert continuity raises names it +// as agent:. Intent the operator set on the saved host's row must +// reach that alert, as it does for Patrol findings and resources API reads, +// although the published registry alone does not know the host. +func TestOperatorIntentReachesSavedHostAlertsAfterRestart(t *testing.T) { + now := time.Now().UTC() + saved := config.HostContinuityEntry{ + HostID: "host-retired", MachineID: "machine-retired", Hostname: "retired", + Platform: "linux", LastSeen: now.Add(-10 * time.Minute), + } + continuity := config.NewHostContinuityStore(t.TempDir(), nil) + if err := continuity.Upsert(saved); err != nil { + t.Fatal(err) + } + store := unifiedresources.NewMemoryStore() + savedID := unifiedresources.MachineIdentityCanonicalID(unifiedresources.ResourceTypeAgent, saved.MachineID) + if err := store.SetResourceOperatorState(unifiedresources.ResourceOperatorState{ + CanonicalID: savedID, IntentionallyOffline: true, + }); err != nil { + t.Fatal(err) + } + adapter := unifiedresources.NewMonitorAdapter(unifiedresources.NewRegistry(store)) + adapter.PopulateFromSnapshot(models.StateSnapshot{LastUpdate: now}) + if _, ok := adapter.ResolveCanonicalResourceID("agent:" + saved.HostID); ok { + t.Fatal("published registry already knows the saved host; the case needs it absent") + } + + manager := alerts.NewManagerWithDataDir(t.TempDir()) + t.Cleanup(manager.Stop) + m := &Monitor{state: models.NewState(), alertManager: manager, hostContinuityStore: continuity} + m.SetResourceStore(adapter) + + preview, err := manager.PreviewIntentPolicy(alerts.AlertIntentPolicyPreviewRequest{ + ResourceID: "agent:" + saved.HostID, ResourceType: "agent", + Signal: string(alerts.AlertIntentSignalOffline), ConditionActive: true, + }) + if err != nil { + t.Fatal(err) + } + if preview.Reason != "operator_expected_offline" { + t.Fatalf("saved host offline alert intent = %+v, want the intentionally-offline state set on %s", preview, savedID) + } + + // The saved host reports again: the published registry knows it, and its + // alerts keep reading the same row. + adapter.PopulateFromSnapshot(models.StateSnapshot{ + Hosts: []models.Host{{ + ID: saved.HostID, MachineID: saved.MachineID, Hostname: saved.Hostname, + Platform: "linux", Status: "online", LastSeen: now, IntervalSeconds: 30, + }}, + LastUpdate: now.Add(time.Second), + }) + m.state.Hosts = []models.Host{{ID: saved.HostID, MachineID: saved.MachineID, Hostname: saved.Hostname, Status: "online", LastSeen: now}} + preview, err = manager.PreviewIntentPolicy(alerts.AlertIntentPolicyPreviewRequest{ + ResourceID: "agent:" + saved.HostID, ResourceType: "agent", + Signal: string(alerts.AlertIntentSignalOffline), ConditionActive: true, + }) + if err != nil { + t.Fatal(err) + } + if preview.Reason != "operator_expected_offline" { + t.Fatalf("reporting host offline alert intent = %+v, want the same intentionally-offline state", preview) + } +} + +// Alert intent keeps one saved-host overlay per published generation, so +// lookups made under the alert manager's lock reuse what publication built. +// A new publication replaces it even when it shares the last one's timestamp +// or is stamped earlier, and mock mode answers from the published registry +// and drops the real-mode overlay. +func TestOperatorIntentIdentityKeepsOneOverlayPerGeneration(t *testing.T) { + now := time.Now().UTC() + saved := config.HostContinuityEntry{ + HostID: "host-retired", MachineID: "machine-retired", Hostname: "retired", + Platform: "linux", LastSeen: now.Add(-10 * time.Minute), + } + continuity := config.NewHostContinuityStore(t.TempDir(), nil) + if err := continuity.Upsert(saved); err != nil { + t.Fatal(err) + } + adapter := unifiedresources.NewMonitorAdapter(unifiedresources.NewRegistry(unifiedresources.NewMemoryStore())) + adapter.PopulateFromSnapshot(models.StateSnapshot{LastUpdate: now.Add(time.Hour)}) + m := &Monitor{state: models.NewState(), hostContinuityStore: continuity} + identity := m.newOperatorIntentIdentity(adapter) + if identity == nil { + t.Fatal("monitor adapter offers no operator intent identity") + } + resolvesSavedHost := func(readState any) bool { + _, ok := readState.(resourceIntentIdentityReader).ResolveCanonicalResourceID("agent:" + saved.HostID) + return ok + } + + identity.refresh() + first := identity.current() + if first == any(adapter) || !resolvesSavedHost(first) { + t.Fatal("refresh did not build the saved-host overlay") + } + if again := identity.current(); again != first { + t.Fatal("lookup rebuilt the overlay within one generation") + } + + // A rebuild of a snapshot stamped in the future keeps its time, so two + // publications can share a timestamp. + adapter.PopulateFromSnapshot(models.StateSnapshot{LastUpdate: now.Add(time.Hour)}) + identity.refresh() + sameStamp := identity.current() + if sameStamp == first || !resolvesSavedHost(sameStamp) { + t.Fatal("a publication sharing the last one's timestamp kept its overlay") + } + + adapter.PopulateFromSnapshot(models.StateSnapshot{LastUpdate: now}) + identity.refresh() + second := identity.current() + if second == sameStamp || !resolvesSavedHost(second) { + t.Fatal("an earlier-stamped generation kept the previous overlay") + } + if again := identity.current(); again != second { + t.Fatal("lookups rebuild the overlay after an earlier-stamped generation") + } + + previous := mock.IsMockEnabled() + if err := mock.SetEnabled(true); err != nil { + t.Fatalf("enable mock mode: %v", err) + } + t.Cleanup(func() { _ = mock.SetEnabled(previous) }) + if got := identity.current(); got != any(adapter) { + t.Fatal("mock mode resolved through the real-mode saved-host overlay") + } + identity.refresh() + if err := mock.SetEnabled(false); err != nil { + t.Fatalf("disable mock mode: %v", err) + } + // Alert evaluation resolves from inside an admitted mock-mode fence call, + // so the rebuild's own fenced store nests in it. + var third any + if !m.mockModeFence.begin().run(func() { third = identity.current() }) { + t.Fatal("fence refused the lookup") + } + if third == any(adapter) || !resolvesSavedHost(third) { + t.Fatal("leaving mock mode kept answering from the published registry alone") + } + if again := identity.current(); again != third { + t.Fatal("the rebuild made inside a fenced call was not kept") + } +} diff --git a/internal/monitoring/monitor_alert_sync.go b/internal/monitoring/monitor_alert_sync.go index b85b085e15..a53f30a6e6 100644 --- a/internal/monitoring/monitor_alert_sync.go +++ b/internal/monitoring/monitor_alert_sync.go @@ -103,6 +103,11 @@ func (m *Monitor) syncUnifiedResourceAlertsToState(resources []unifiedresources. return } + // Alert intent resolves through the published registry with saved hosts + // overlaid, under the alert manager's lock. Build that overlay for the new + // generation here, outside the lock, so its lookups need not. + m.operatorIntentIdentity.Load().refresh() + // Resource publication can make persisted canonical policy resolvable for // restored native-ID alerts that preceded the first observation at startup. scope.run(func() { m.alertManager.ReconcileOperatorIntentState() }) diff --git a/internal/monitoring/monitor_host_agent_removal_lifecycle_test.go b/internal/monitoring/monitor_host_agent_removal_lifecycle_test.go index e9d81cf562..98837683c6 100644 --- a/internal/monitoring/monitor_host_agent_removal_lifecycle_test.go +++ b/internal/monitoring/monitor_host_agent_removal_lifecycle_test.go @@ -619,6 +619,35 @@ func TestHostAgentRemovalLifecycleKeepsOfflineRowRemovableAfterRestart(t *testin } } +// After a restart a host that has not reported again exists only as its saved +// continuity row, and its offline alert names it agent:. The intent +// the operator set on that row must still hold the alert back: alert intent +// resolves through the read state that overlays saved hosts, not through the +// published registry alone, which no longer lists the host. +func TestHostAgentRemovalLifecycleHonorsSavedHostIntentAfterRestart(t *testing.T) { + dataPath := t.TempDir() + now := time.Now().UTC() + token := &config.APITokenRecord{ID: "intent-restart-token", CreatedAt: now.Add(-time.Hour)} + report := hostRemovalLifecycleReport("intent-restart-host", "intent-restart-machine", "intent-restart-agent", "intent-restart.local", "linux", now) + if _, err := newHostRemovalLifecycleMonitor(t, dataPath).ApplyHostReport(report, token); err != nil { + t.Fatalf("initial ApplyHostReport: %v", err) + } + + restarted := newHostRemovalLifecycleMonitor(t, dataPath) + store := unifiedresources.NewMemoryStore() + savedID := unifiedresources.MachineIdentityCanonicalID(unifiedresources.ResourceTypeAgent, "intent-restart-machine") + if err := store.SetResourceOperatorState(unifiedresources.ResourceOperatorState{CanonicalID: savedID, IntentionallyOffline: true}); err != nil { + t.Fatalf("set saved host intent: %v", err) + } + restarted.SetResourceStore(unifiedresources.NewMonitorAdapter(unifiedresources.NewRegistry(store))) + for i := 0; i < 3; i++ { + restarted.evaluateHostAgents(now.Add(10*time.Minute + time.Duration(i)*time.Second)) + } + if alerts := restarted.alertManager.GetActiveAlerts(); len(alerts) != 0 { + t.Fatalf("intentionally offline saved host raised %+v after restart, want no alert", alerts) + } +} + func TestHostAgentRemovalLifecycleDoesNotPoisonDuplicateActiveIdentity(t *testing.T) { monitor := newHostRemovalLifecycleMonitor(t, t.TempDir()) now := time.Now().UTC() diff --git a/internal/monitoring/monitor_host_agents_test.go b/internal/monitoring/monitor_host_agents_test.go index f5abd19370..268d542f9d 100644 --- a/internal/monitoring/monitor_host_agents_test.go +++ b/internal/monitoring/monitor_host_agents_test.go @@ -6988,6 +6988,113 @@ func TestBroadcastProjectionListsRegistryOnceAndKeepsLiveChanges(t *testing.T) { } } +// presentationFixtureStore keeps the fixture registry: the broadcast's read +// refresh would otherwise rebuild it from the empty snapshot. +type presentationFixtureStore struct { + *unifiedresources.MonitorAdapter +} + +func (presentationFixtureStore) TryReplaceRegistryForRead(models.StateSnapshot, time.Duration, func() map[unifiedresources.DataSource][]unifiedresources.IngestRecord) bool { + return false +} + +// A host pair the operator split (unlink or report-merge) must be two +// broadcast rows exactly when the resources API lists two. The broadcast used +// to coalesce without the store's exclusions, so the websocket folded the +// pair the API kept apart. +func TestBroadcastPresentationCoalesceHonoursMergeExclusions(t *testing.T) { + now := time.Date(2026, 10, 7, 21, 0, 0, 0, time.UTC) + resources := []unifiedresources.Resource{ + { + ID: "agent-runtime-alpha", Type: unifiedresources.ResourceTypeAgent, Name: "alpha", + Status: unifiedresources.StatusOnline, LastSeen: now, + Sources: []unifiedresources.DataSource{unifiedresources.SourceAgent}, + Identity: unifiedresources.ResourceIdentity{Hostnames: []string{"alpha"}}, + Agent: &unifiedresources.AgentData{AgentID: "agent-alpha", Hostname: "alpha"}, + }, + { + ID: "agent-docker-alpha", Type: unifiedresources.ResourceTypeAgent, Name: "alpha", + Status: unifiedresources.StatusOnline, LastSeen: now, + Sources: []unifiedresources.DataSource{unifiedresources.SourceDocker}, + Identity: unifiedresources.ResourceIdentity{Hostnames: []string{"alpha"}}, + Docker: &unifiedresources.DockerData{Hostname: "alpha"}, + }, + } + for _, split := range []bool{false, true} { + store := unifiedresources.NewMemoryStore() + if split { + if err := store.AddExclusion(unifiedresources.ResourceExclusion{ResourceA: "agent-runtime-alpha", ResourceB: "agent-docker-alpha"}); err != nil { + t.Fatal(err) + } + } + registry := unifiedresources.NewRegistry(store) + registry.IngestResources(resources) + adapter := unifiedresources.NewMonitorAdapter(registry) + m := &Monitor{resourceStore: presentationFixtureStore{adapter}} + + state := m.buildBroadcastFrontendStateFromSnapshotWithClock(models.EmptyStateSnapshot(), m.mockModeFence.begin(), func() time.Time { return now }) + broadcastRows := 0 + for _, resource := range state.Resources { + if resource.Name == "alpha" { + broadcastRows++ + } + } + + // The resources API seeds a store-backed registry from the monitor's + // listing and presents it through ListForPresentation. + rest := unifiedresources.NewRegistry(store) + rest.IngestResources(adapter.GetAll()) + restRows := len(rest.ListForPresentation()) + + want := 1 + if split { + want = 2 + } + if broadcastRows != want || restRows != want { + t.Fatalf("split=%v: broadcast rows=%d REST rows=%d, want %d on both", split, broadcastRows, restRows, want) + } + } +} + +// The rows a broadcast coalesces were listed by its view's read state, such as +// a host-continuity overlay that loaded the store's exclusions afresh, so its +// exclusions decide even when the resource store's generation predates them. +func TestBroadcastPresentationCoalescePrefersTheListingReadState(t *testing.T) { + now := time.Date(2026, 10, 7, 21, 0, 0, 0, time.UTC) + resources := []unifiedresources.Resource{ + {ID: "agent-runtime-alpha", Type: unifiedresources.ResourceTypeAgent, Name: "alpha", Status: unifiedresources.StatusOnline, LastSeen: now, + Sources: []unifiedresources.DataSource{unifiedresources.SourceAgent}, Identity: unifiedresources.ResourceIdentity{Hostnames: []string{"alpha"}}, + Agent: &unifiedresources.AgentData{AgentID: "agent-alpha", Hostname: "alpha"}}, + {ID: "agent-docker-alpha", Type: unifiedresources.ResourceTypeAgent, Name: "alpha", Status: unifiedresources.StatusOnline, LastSeen: now, + Sources: []unifiedresources.DataSource{unifiedresources.SourceDocker}, Identity: unifiedresources.ResourceIdentity{Hostnames: []string{"alpha"}}, + Docker: &unifiedresources.DockerData{Hostname: "alpha"}}, + } + store := unifiedresources.NewMemoryStore() + base := unifiedresources.NewRegistry(store) + base.IngestResources(resources) + m := &Monitor{resourceStore: unifiedresources.NewMonitorAdapter(base)} + if err := store.AddExclusion(unifiedresources.ResourceExclusion{ResourceA: "agent-runtime-alpha", ResourceB: "agent-docker-alpha"}); err != nil { + t.Fatal(err) + } + overlay := unifiedresources.NewRegistry(store) + overlay.IngestResources(resources) + listing := unifiedresources.NewMonitorAdapter(overlay) + if got := len(m.coalesceResourcesForPresentation(listing, listing.GetAll())); got != 2 { + t.Fatalf("coalesced %d rows with the listing's exclusion, want 2", got) + } + if got := len(m.coalesceResourcesForPresentation(nil, listing.GetAll())); got != 1 { + t.Fatalf("without a listing read state the store generation decides: %d rows, want 1", got) + } + + // The mock view lists through a store-less registry, which carries no + // operator decisions, so the resource store's adapter supplies them. + mockView := monitorUnifiedStateViewFromResources(resources, now) + withSplit := &Monitor{resourceStore: listing} + if got := len(withSplit.coalesceResourcesForPresentation(mockView.readState, mockView.resources)); got != 2 { + t.Fatalf("mock-style view coalesced %d rows, want the store's split to keep 2", got) + } +} + // A host agent linked to a Proxmox node merges into one read-state row that // every PVE poll keeps online and fresh. Once the agent stops reporting, its // retained filesystem summary must stop standing in for the node's disk (and diff --git a/internal/unifiedresources/availability_link_test.go b/internal/unifiedresources/availability_link_test.go index fcb21d0289..e44edd3073 100644 --- a/internal/unifiedresources/availability_link_test.go +++ b/internal/unifiedresources/availability_link_test.go @@ -1,9 +1,11 @@ package unifiedresources import ( + "slices" "testing" "time" + "github.com/rcourtman/pulse-go-rewrite/internal/models" "github.com/rcourtman/pulse-go-rewrite/internal/operationaltrust" ) @@ -792,3 +794,294 @@ func TestProxmoxGuestAvailabilityLinkFollowsRetiredIDs(t *testing.T) { }) } } + +// assertAvailabilityCheckAttachedTo checks that the configured check targetID +// is projected onto resourceID by its explicit link and that the check's own +// endpoint row points there. +func assertAvailabilityCheckAttachedTo(t *testing.T, stage string, rr *ResourceRegistry, targetID, resourceID string) { + t.Helper() + target, ok := rr.Get(resourceID) + if !ok { + t.Fatalf("%s: check target %s missing from %v", stage, resourceID, resourceIDs(rr.List())) + } + attached := false + for _, check := range AvailabilityChecksForResource(*target) { + if check.TargetID == targetID { + attached = check.CorrelationState == AvailabilityCorrelationAttached && check.CorrelationRule == "explicit_resource_link" + } + } + if !attached { + t.Fatalf("%s: %s checks = %+v, want %s attached by its explicit link", stage, resourceID, AvailabilityChecksForResource(*target), targetID) + } + endpoint := availabilityEndpointByTarget(t, rr, targetID) + for _, relationship := range endpoint.Relationships { + if relationship.Type == RelChecks && relationship.TargetID == resourceID { + return + } + } + t.Fatalf("%s: endpoint relationships = %+v, want a checks edge to %s", stage, endpoint.Relationships, resourceID) +} + +func linkedGuestVMRecord(now time.Time) IngestRecord { + return IngestRecord{ + SourceID: "vc-1:vm:vm-42", + Resource: Resource{ + Type: ResourceTypeVM, + Technology: "vmware", + Name: "app-guest", + Status: StatusOnline, + LastSeen: now, + VMware: &VMwareData{ConnectionID: "vc-1", ManagedObjectID: "vm-42", EntityType: "vm"}, + }, + Identity: ResourceIdentity{Hostnames: []string{"app-guest"}}, + } +} + +func linkedGuestCheck(t *testing.T, targetID, address, linkedResourceID string, now time.Time) IngestRecord { + t.Helper() + return availabilityProbeRecord(targetID, address, &AvailabilityData{ + LinkedResourceID: linkedResourceID, + Address: address, + Protocol: "icmp", + Enabled: true, + Available: true, + LastChecked: &now, + Evidence: availabilityProbeEvidence(t, targetID, now), + }) +} + +// An operator link folds the agent inside a vSphere VM into the VM, so the +// agent's own canonical ID no longer names a row. A check configured against +// that ID watches the same machine and follows the agent to the merged row in +// every registry that lists it: the monitor's rebuild, a registry seeded from +// the monitor's listing that replays the checks (the resources API) and a +// read-state overlay. Reference reads (alert intent, operator state, API +// lookups) resolve the folded ID the same way. History does not, and the +// folded ID never becomes a superseded ID, which alert and availability +// migrations would turn into a permanent rewrite of the operator's config. +func TestAvailabilityLinkToALinkFoldedResourceFollowsItsPrimary(t *testing.T) { + now := time.Now().UTC() + snapshot := models.StateSnapshot{ + Hosts: []models.Host{{ID: "host-app-guest", Hostname: "app-guest", MachineID: "machine-app-guest", Status: "online", LastSeen: now}}, + LastUpdate: now, + } + agentID := MachineIdentityCanonicalID(ResourceTypeAgent, "machine-app-guest") + checks := []IngestRecord{linkedGuestCheck(t, "probe-guest", "192.0.2.60", agentID, now)} + records := map[DataSource][]IngestRecord{ + SourceVMware: {linkedGuestVMRecord(now)}, + SourceAvailability: checks, + } + + for _, agentPrimary := range []bool{false, true} { + name := "vm-primary" + if agentPrimary { + name = "agent-primary" + } + t.Run(name, func(t *testing.T) { + store := NewMemoryStore() + adapter := NewMonitorAdapter(NewRegistry(store)) + adapter.PopulateSnapshotAndSupplemental(snapshot, records) + assertAvailabilityCheckAttachedTo(t, "unlinked", adapter.currentRegistry(), "probe-guest", agentID) + + var vmID string + for _, resource := range adapter.GetAll() { + if resource.VMware != nil { + vmID = resource.ID + } + } + if vmID == "" { + t.Fatalf("unlinked estate = %v, want the vSphere VM", resourceIDs(adapter.GetAll())) + } + primaryID := vmID + if agentPrimary { + primaryID = agentID + } + if err := store.AddLink(ResourceLink{ResourceA: vmID, ResourceB: agentID, PrimaryID: primaryID}); err != nil { + t.Fatalf("add link: %v", err) + } + + // Pins persist on the first rebuild; damage from them shows on the second. + for i, stage := range []string{"rebuild", "second rebuild"} { + next := snapshot + next.LastUpdate = now.Add(time.Duration(i+1) * time.Second) + adapter.PopulateSnapshotAndSupplemental(next, records) + if _, listed := adapter.currentRegistry().Get(agentID); listed { + t.Fatalf("%s: agent %s still listed beside the VM it is linked into", stage, agentID) + } + + rest := NewRegistry(store) + rest.IngestResources(adapter.GetAll()) + rest.IngestRecords(SourceAvailability, checks) + overlay, ok := ReadStateWithRecords(adapter, SourceAvailability, checks).(*MonitorAdapter) + if !ok { + t.Fatalf("%s: overlay is not a monitor adapter", stage) + } + + for _, view := range []struct { + name string + rr *ResourceRegistry + }{ + {"monitor", adapter.currentRegistry()}, + {"resources API", rest}, + {"overlay", overlay.currentRegistry()}, + } { + label := stage + ", " + view.name + assertAvailabilityCheckAttachedTo(t, label, view.rr, "probe-guest", vmID) + // The agent's alert reference already reached the merged + // row; its canonical ID must agree. + for _, ref := range []string{agentID, "agent:host-app-guest"} { + if got, ok := view.rr.ResolveReferenceID(ref); !ok || got != vmID { + t.Fatalf("%s: ResolveReferenceID(%s) = %q, %t, want the VM %s", label, ref, got, ok, vmID) + } + } + // History bindings persist and join both journals in either + // direction, so they would outlive the link. + if got, claimed := view.rr.resolveHistoryReference(agentID); got != "" || claimed { + t.Fatalf("%s: history reference %s resolved to %q (claimed %t), want it left to its own journal", label, agentID, got, claimed) + } + vm, _ := view.rr.Get(vmID) + if slices.Contains(vm.SupersededCanonicalIDs, agentID) || + (vm.Canonical != nil && slices.Contains(vm.Canonical.SupersededIDs, agentID)) { + t.Fatalf("%s: folded agent %s is listed as superseded by the VM", label, agentID) + } + } + } + }) + } +} + +// An old enrollment of a guest linked into its current agent, and that agent +// linked into its VM: every ID folded along the chain resolves to the final +// primary, in the monitor and in a registry seeded from it. +func TestAvailabilityLinkFollowsChainedLinkFolds(t *testing.T) { + now := time.Now().UTC() + snapshot := models.StateSnapshot{ + Hosts: []models.Host{ + {ID: "host-app-guest", Hostname: "app-guest", MachineID: "machine-app-guest", Status: "online", LastSeen: now}, + {ID: "host-app-guest-old", Hostname: "app-guest-old", MachineID: "machine-app-guest-old", Status: "online", LastSeen: now}, + }, + LastUpdate: now, + } + agentID := MachineIdentityCanonicalID(ResourceTypeAgent, "machine-app-guest") + oldAgentID := MachineIdentityCanonicalID(ResourceTypeAgent, "machine-app-guest-old") + checks := []IngestRecord{linkedGuestCheck(t, "probe-old", "192.0.2.61", oldAgentID, now)} + records := map[DataSource][]IngestRecord{ + SourceVMware: {linkedGuestVMRecord(now)}, + SourceAvailability: checks, + } + + unlinked := NewMonitorAdapter(NewRegistry(nil)) + unlinked.PopulateSnapshotAndSupplemental(snapshot, records) + var vmID string + for _, resource := range unlinked.GetAll() { + if resource.VMware != nil { + vmID = resource.ID + } + } + listed := resourceIDs(unlinked.GetAll()) + if vmID == "" || !slices.Contains(listed, agentID) || !slices.Contains(listed, oldAgentID) { + t.Fatalf("unlinked estate = %v, want the VM and both agents", listed) + } + + store := NewMemoryStore() + for _, link := range []ResourceLink{ + {ResourceA: agentID, ResourceB: oldAgentID, PrimaryID: agentID}, + {ResourceA: vmID, ResourceB: agentID, PrimaryID: vmID}, + } { + if err := store.AddLink(link); err != nil { + t.Fatalf("add link: %v", err) + } + } + adapter := NewMonitorAdapter(NewRegistry(store)) + adapter.PopulateSnapshotAndSupplemental(snapshot, records) + listed = resourceIDs(adapter.GetAll()) + if slices.Contains(listed, agentID) || slices.Contains(listed, oldAgentID) { + t.Fatalf("linked estate = %v, want both agents folded into the VM %s", listed, vmID) + } + + rest := NewRegistry(store) + rest.IngestResources(adapter.GetAll()) + rest.IngestRecords(SourceAvailability, checks) + for _, view := range []struct { + name string + rr *ResourceRegistry + }{{"monitor", adapter.currentRegistry()}, {"resources API", rest}} { + assertAvailabilityCheckAttachedTo(t, view.name, view.rr, "probe-old", vmID) + for _, folded := range []string{agentID, oldAgentID} { + if got, ok := view.rr.ResolveReferenceID(folded); !ok || got != vmID { + t.Fatalf("%s: ResolveReferenceID(%s) = %q, %t, want the VM %s", view.name, folded, got, ok, vmID) + } + } + } +} + +// The fold index follows the rows holding each folded ID. A seeded listing can +// carry one folded ID on two rows, or on a row while a live row has that ID. A +// live row answers for itself; two holders resolve to nothing, without falling +// through to weaker matches such as a hostname alias, so an explicit check link +// stays fail-closed. Holders a link later merges, and a holder re-keyed in +// place, keep resolving. +func TestLinkFoldIndexFollowsItsHolders(t *testing.T) { + now := time.Now().UTC() + vm := func(id string, folded ...string) Resource { + folds := make([]ManualLinkFold, 0, len(folded)) + for _, foldedID := range folded { + folds = append(folds, ManualLinkFold{HolderID: id, FoldedID: foldedID, Sources: []DataSource{SourceAgent}}) + } + return Resource{ + ID: id, Type: ResourceTypeVM, Name: id, Status: StatusOnline, LastSeen: now, + Sources: []DataSource{SourceVMware}, linkFolds: folds, + } + } + + t.Run("ambiguous holders fail closed", func(t *testing.T) { + rr := NewRegistry(nil) + rr.IngestResources([]Resource{ + vm("vm-a", "agent-shared", "agent-live"), + vm("vm-b", "agent-shared"), + {ID: "agent-live", Type: ResourceTypeAgent, Name: "agent-live", Status: StatusOnline, LastSeen: now, Sources: []DataSource{SourceAgent}}, + { + ID: "vm-c", Type: ResourceTypeVM, Name: "vm-c", Status: StatusOnline, LastSeen: now, + Sources: []DataSource{SourceVMware}, Identity: ResourceIdentity{Hostnames: []string{"agent-shared"}}, + }, + }) + if got, ok := rr.ResolveReferenceID("agent-shared"); ok { + t.Fatalf("ResolveReferenceID(agent-shared) = %q, want no answer while two rows hold it", got) + } + if got, ok := rr.ResolveReferenceID("agent-live"); !ok || got != "agent-live" { + t.Fatalf("ResolveReferenceID(agent-live) = %q, %t, want the live row itself", got, ok) + } + rr.IngestRecords(SourceAvailability, []IngestRecord{linkedGuestCheck(t, "probe-shared", "192.0.2.62", "agent-shared", now)}) + endpoint := availabilityEndpointByTarget(t, rr, "probe-shared") + if endpoint.Availability.CorrelationState != AvailabilityCorrelationUnresolved { + t.Fatalf("ambiguous folded link correlation = %+v, want unresolved", endpoint.Availability) + } + }) + + t.Run("holders merged by a link resolve to the survivor", func(t *testing.T) { + store := NewMemoryStore() + if err := store.AddLink(ResourceLink{ResourceA: "vm-a", ResourceB: "vm-b", PrimaryID: "vm-a"}); err != nil { + t.Fatalf("add link: %v", err) + } + rr := NewRegistry(store) + rr.IngestResources([]Resource{vm("vm-a", "agent-shared"), vm("vm-b", "agent-shared")}) + for _, ref := range []string{"agent-shared", "vm-b"} { + if got, ok := rr.ResolveReferenceID(ref); !ok || got != "vm-a" { + t.Fatalf("ResolveReferenceID(%s) = %q, %t, want vm-a", ref, got, ok) + } + } + }) + + t.Run("re-keyed holder keeps its folds", func(t *testing.T) { + rr := NewRegistry(nil) + disk := vm("disk-old", "disk-folded") + disk.Type = ResourceTypePhysicalDisk + rr.IngestResources([]Resource{disk}) + rr.mu.Lock() + rr.rekeyPhysicalDiskLocked(rr.resources["disk-old"], "disk-new") + rr.mu.Unlock() + if got, ok := rr.ResolveReferenceID("disk-folded"); !ok || got != "disk-new" { + t.Fatalf("ResolveReferenceID(disk-folded) = %q, %t, want the re-keyed holder disk-new", got, ok) + } + }) +} diff --git a/internal/unifiedresources/canonical_id_pins.go b/internal/unifiedresources/canonical_id_pins.go index ece08ab9b7..847380b1bd 100644 --- a/internal/unifiedresources/canonical_id_pins.go +++ b/internal/unifiedresources/canonical_id_pins.go @@ -3,6 +3,7 @@ package unifiedresources import ( "log" "net" + "slices" "strings" ) @@ -239,7 +240,12 @@ func (rr *ResourceRegistry) loadIdentityPins() { // knows cluster+hostname) still derives the same canonical ID as a steady // state rebuild that knows the machine ID. Only missing fields are filled; an // incoming identity that already carries a machine ID is never overridden. -func (rr *ResourceRegistry) completeIdentityFromPins(source DataSource, resource Resource, identity ResourceIdentity) ResourceIdentity { +// A pin of a resource the operator split from this record (unlink or +// report-merge) completes nothing: the record would mint the pinned +// resource's machine-derived ID and fold into it by ID, which no exclusion +// check sees. A node pin that took a linked agent's machine key kept the pair +// merged after unlink that way. +func (rr *ResourceRegistry) completeIdentityFromPins(source DataSource, sourceID string, resource Resource, identity ResourceIdentity) ResourceIdentity { if CanonicalResourceType(resource.Type) != ResourceTypeAgent { return identity } @@ -256,6 +262,12 @@ func (rr *ResourceRegistry) completeIdentityFromPins(source DataSource, resource if !ok { return identity } + // The exclusion names the IDs the record holds without the pin. + if len(rr.exclusions) > 0 && + (rr.isExcluded(pin.CanonicalID, rr.chooseNewID(resource.Type, identity, source, sourceID)) || + rr.isExcluded(pin.CanonicalID, rr.sourceSpecificID(resource.Type, source, sourceID))) { + return identity + } if strings.TrimSpace(identity.MachineID) == "" { identity.MachineID = pin.MachineID } @@ -326,19 +338,114 @@ func (index *identityPinIndex) successionsFor(pin ResourceIdentityPin) []Canonic return successions } +// upsertedOnto returns the row UpsertResourceIdentityPins stores when pin is +// written over existing: an empty field keeps the stored value. +func (pin ResourceIdentityPin) upsertedOnto(existing ResourceIdentityPin) ResourceIdentityPin { + if pin.MachineID == "" { + pin.MachineID = existing.MachineID + } + if pin.DMIUUID == "" { + pin.DMIUUID = existing.DMIUUID + } + if pin.ClusterName == "" { + pin.ClusterName = existing.ClusterName + } + if pin.Hostname == "" { + pin.Hostname = existing.Hostname + } + return pin +} + +// keptOver returns the row a manual-link side writes over existing: like an +// upsert it keeps the stored fields its own pin leaves empty, so evidence +// from earlier observations survives a weaker one, but it never keeps a +// strong key another resource holds. Earlier releases pinned a link's +// primary with the merged identity, so such a key may be the other side's, +// and a key a resource reports now beats one a row only kept, as the store +// already hands a moved key to the new pin. +func (pin ResourceIdentityPin) keptOver(existing ResourceIdentityPin, held map[string]string) ResourceIdentityPin { + if pin.Hostname == "" && (pin.ClusterName == "" || !heldByOther(held, clusterPinKey(pin.ClusterName, existing.Hostname), pin.CanonicalID)) { + pin.Hostname = existing.Hostname + } + if pin.MachineID == "" && existing.MachineID != "" && !heldByOther(held, machinePinKey(existing.MachineID), pin.CanonicalID) { + pin.MachineID = existing.MachineID + } + if pin.DMIUUID == "" && existing.DMIUUID != "" && !heldByOther(held, dmiPinKey(existing.DMIUUID), pin.CanonicalID) { + pin.DMIUUID = existing.DMIUUID + } + if pin.ClusterName == "" && existing.ClusterName != "" && !heldByOther(held, clusterPinKey(existing.ClusterName, pin.Hostname), pin.CanonicalID) { + pin.ClusterName = existing.ClusterName + } + return pin +} + +// withoutKeysHeldElsewhere drops the strong keys another resource holds. +func (pin ResourceIdentityPin) withoutKeysHeldElsewhere(held map[string]string) ResourceIdentityPin { + if pin.MachineID != "" && heldByOther(held, machinePinKey(pin.MachineID), pin.CanonicalID) { + pin.MachineID = "" + } + if pin.DMIUUID != "" && heldByOther(held, dmiPinKey(pin.DMIUUID), pin.CanonicalID) { + pin.DMIUUID = "" + } + if pin.ClusterName != "" && heldByOther(held, clusterPinKey(pin.ClusterName, pin.Hostname), pin.CanonicalID) { + pin.ClusterName = "" + } + return pin +} + +func heldByOther(held map[string]string, key, canonicalID string) bool { + owner, taken := held[key] + return taken && owner != canonicalID +} + +// ownerKeys lists the strong keys the store lets only one pin hold, matching +// the conflict rule in UpsertResourceIdentityPins. +func (pin ResourceIdentityPin) ownerKeys() []string { + var keys []string + if pin.MachineID != "" { + keys = append(keys, machinePinKey(pin.MachineID)) + } + if pin.DMIUUID != "" { + keys = append(keys, dmiPinKey(pin.DMIUUID)) + } + if pin.ClusterName != "" { + keys = append(keys, clusterPinKey(pin.ClusterName, pin.Hostname)) + } + return keys +} + +func machinePinKey(machineID string) string { return "machine:" + machineID } + +func dmiPinKey(dmiUUID string) string { return "dmi:" + dmiUUID } + +func clusterPinKey(clusterName, hostname string) string { + return "cluster:" + clusterName + "\x00" + hostname +} + // PersistIdentityPins writes the identity pins for the registry's current // host resources into the resource store. Only new or changed pins are // written, so steady-state rebuild ticks cost no writes. Call this after a // rebuild on the durable store-backed registry; ephemeral per-request // registries consult pins but do not write them. // +// Each resource is pinned from its own sources. Both sides of a manual link +// are pinned from the identity captured before the link merged them +// (recordLinkOwnPin), and a pinnable folded side keeps its pin under its own +// canonical ID: a link is display intent, so the merged identity must not +// lend one machine's keys to the other's pin, or a later rebuild would +// complete those keys, mint the same canonical ID for both sides and keep +// them merged after the link is gone. A link side's row is rewritten whole +// (see keptOver), because an upsert would keep a key an earlier release +// borrowed into it from the merged identity. +// // When a new pin supersedes an earlier era's pin for the same physical host // (see successionsFor), the store re-keys the host's operator-owned rows to // the new canonical ID before the pin write, so operator intent // (never-auto-remediate, maintenance windows) and action-audit history // survive the era change. Successions are skipped while the old canonical ID -// still belongs to a live resource: a genuinely short-named host must not -// have its rows stolen by an FQDN sibling. Change-journal rows are never +// still belongs to a live resource, or to one a manual link folded into +// another: a genuinely short-named host must not have its rows stolen by an +// FQDN sibling. Change-journal rows are never // rewritten; EraIDs merges those at read time. func (rr *ResourceRegistry) PersistIdentityPins() { if rr.store == nil { @@ -346,27 +453,93 @@ func (rr *ResourceRegistry) PersistIdentityPins() { } rr.mu.RLock() - var pins []ResourceIdentityPin - var successions []CanonicalIDSuccession - for _, resource := range rr.resources { + // held maps each strong key to the resource that holds it this rebuild: + // a live resource's row (an upsert only ever widens it), each link + // side's own pin, and then the keys link sides keep from their rows. + held := make(map[string]string) + hold := func(pin ResourceIdentityPin) { + for _, key := range pin.ownerKeys() { + if _, taken := held[key]; !taken { + held[key] = pin.CanonicalID + } + } + } + var pins, linkPins []ResourceIdentityPin + for id, resource := range rr.resources { + if own, linked := rr.linkOwnPins[id]; linked { + if own != nil { + hold(*own) + linkPins = append(linkPins, *own) + } + continue + } pin, ok := identityPinForResource(resource) if !ok { continue } - if existing, known := rr.identityPins.byCanonicalID[pin.CanonicalID]; known && existing == pin { - continue + effective := pin + if existing, known := rr.identityPins.byCanonicalID[pin.CanonicalID]; known { + effective = pin.upsertedOnto(existing) } + hold(effective) pins = append(pins, pin) - for _, succession := range rr.identityPins.successionsFor(pin) { - if _, live := rr.resources[succession.OldCanonicalID]; live { - continue - } - successions = append(successions, succession) + } + folded := make([]string, 0, len(rr.linkOwnPins)) + for id, own := range rr.linkOwnPins { + if _, live := rr.resources[id]; !live && own != nil { + folded = append(folded, id) } } + slices.Sort(folded) + for _, id := range folded { + // The store gives each strong key one owner. A folded side drops a + // key a live resource, or an earlier folded side, already holds + // rather than taking it back on every other rebuild. A live non-link + // row is upserted, which cannot drop a field, so its kept fields + // hold their keys here. A side that would lose its machine evidence + // writes no pin: a cluster-only row reads as a machine-keyless + // host's, which cluster-slot succession lets a later machine absorb. + own := *rr.linkOwnPins[id] + pin := own.withoutKeysHeldElsewhere(held) + if !pin.hasStrongKey() || (pin.MachineID == "" && pin.DMIUUID == "" && (own.MachineID != "" || own.DMIUUID != "")) { + continue + } + hold(pin) + linkPins = append(linkPins, pin) + } + slices.SortFunc(linkPins, func(a, b ResourceIdentityPin) int { return strings.Compare(a.CanonicalID, b.CanonicalID) }) + for i, pin := range linkPins { + if existing, known := rr.identityPins.byCanonicalID[pin.CanonicalID]; known { + linkPins[i] = pin.keptOver(existing, held) + hold(linkPins[i]) + } + } + + var successions []CanonicalIDSuccession + changed := func(batch []ResourceIdentityPin) []ResourceIdentityPin { + var out []ResourceIdentityPin + for _, pin := range batch { + if existing, known := rr.identityPins.byCanonicalID[pin.CanonicalID]; known && existing == pin { + continue + } + out = append(out, pin) + for _, succession := range rr.identityPins.successionsFor(pin) { + // A resource a manual link folded into its primary is still + // observed. Succeeding it would re-key the link onto the + // primary itself, splitting the pair on the next rebuild. + if rr.canonicalIDObservedLocked(succession.OldCanonicalID) { + continue + } + successions = append(successions, succession) + } + } + return out + } + pins = changed(pins) + linkPins = changed(linkPins) rr.mu.RUnlock() - if len(pins) == 0 { + if len(pins) == 0 && len(linkPins) == 0 { return } if len(successions) > 0 { @@ -376,9 +549,20 @@ func (rr *ResourceRegistry) PersistIdentityPins() { } } } - if err := rr.store.UpsertResourceIdentityPins(pins); err != nil { - log.Printf("unifiedresources: failed to persist identity pins: %v", err) - return + // Upserts go first: one that deletes a link side's old row, which still + // held a key the side has now dropped, is then followed by the side's + // replace, which writes the row again. + if len(pins) > 0 { + if err := rr.store.UpsertResourceIdentityPins(pins); err != nil { + log.Printf("unifiedresources: failed to persist identity pins: %v", err) + return + } + } + if len(linkPins) > 0 { + if err := rr.store.ReplaceResourceIdentityPins(linkPins); err != nil { + log.Printf("unifiedresources: failed to persist manual-link identity pins: %v", err) + return + } } refreshed, err := rr.store.ListResourceIdentityPins() if err != nil { diff --git a/internal/unifiedresources/canonical_id_pins_test.go b/internal/unifiedresources/canonical_id_pins_test.go index 02aa36df11..172952866a 100644 --- a/internal/unifiedresources/canonical_id_pins_test.go +++ b/internal/unifiedresources/canonical_id_pins_test.go @@ -1,6 +1,11 @@ package unifiedresources -import "testing" +import ( + "testing" + "time" + + "github.com/rcourtman/pulse-go-rewrite/internal/models" +) // The #1559 shape: several standalone agents whose hostnames share a short // name (cloud.rnd-lax1, cloud.gce-or1, cloud.dmi-lax1). The pin index must @@ -74,3 +79,85 @@ func TestIdentityPinIndexLegacyCollapsedPinStillMatches(t *testing.T) { t.Fatalf("legacy collapsed pin should match its host's full hostname, got ok=%v pin=%q", ok, pin.CanonicalID) } } + +// A Proxmox node linked to a standalone agent: while linked, the merged +// resource's identity pin recorded the node's endpoint hostname with the +// agent's machine key. After unlink the node completed that key from the pin, +// minted the agent's machine-derived ID and folded into it by ID before any +// exclusion check ran, so the pair stayed merged. Each step is judged after +// three monitor rebuilds, since pins persist on one rebuild and steer the next. +func TestUnlinkSplitsHostPairWhosePinTookTheOtherSidesKey(t *testing.T) { + now := time.Now().UTC() + for _, tc := range []struct { + name string + nodeEndpoint string + agentPrimary bool + }{ + {"agent-primary", "https://10.0.0.5:8006", true}, + {"node-primary", "https://10.0.0.5:8006", false}, + {"agent-primary-fqdn", "https://pve1.example.lan:8006", true}, + {"node-primary-fqdn", "https://pve1.example.lan:8006", false}, + } { + t.Run(tc.name, func(t *testing.T) { + store, err := NewSQLiteResourceStore(t.TempDir(), "default") + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { _ = store.Close() }) + snapshot := models.StateSnapshot{ + LastUpdate: now, + Nodes: []models.Node{{ID: "lab-pve1", Name: "pve1", Instance: "lab", Host: tc.nodeEndpoint, Status: "online", LastSeen: now}}, + Hosts: []models.Host{{ID: "host-box", Hostname: "box-agent", MachineID: "fedcba9876543210", Status: "online", LastSeen: now}}, + } + adapter := NewMonitorAdapter(NewRegistry(store)) + rebuild := func() []Resource { + for i := 0; i < 3; i++ { + adapter.PopulateFromSnapshot(snapshot) + } + return adapter.GetAll() + } + var nodeID, agentID string + for _, resource := range rebuild() { + switch { + case resource.Proxmox != nil && resource.Agent == nil: + nodeID = resource.ID + case resource.Agent != nil && resource.Proxmox == nil: + agentID = resource.ID + } + } + if nodeID == "" || agentID == "" { + t.Fatalf("fixture did not produce a separate node and agent: %+v", adapter.GetAll()) + } + primaryID, otherID := nodeID, agentID + if tc.agentPrimary { + primaryID, otherID = agentID, nodeID + } + if err := store.AddLink(ResourceLink{ResourceA: primaryID, ResourceB: otherID, PrimaryID: primaryID, CreatedAt: time.Now().UTC()}); err != nil { + t.Fatal(err) + } + linked := rebuild() + if len(linked) != 1 { + t.Fatalf("link did not merge the pair: %d resources", len(linked)) + } + // Unlink as the API is called: on the merged resource, naming the + // side that disappeared into it. + mergedID := linked[0].ID + splitID := agentID + if mergedID == agentID { + splitID = nodeID + } + if err := store.AddExclusion(ResourceExclusion{ResourceA: mergedID, ResourceB: splitID, CreatedAt: time.Now().UTC()}); err != nil { + t.Fatal(err) + } + unlinked := rebuild() + if len(unlinked) != 2 { + t.Fatalf("unlink left %d resources, want the node and the agent apart: %+v", len(unlinked), unlinked) + } + for _, resource := range unlinked { + if resource.Proxmox != nil && resource.Agent != nil { + t.Fatalf("resource %s still carries both facets after unlink", resource.ID) + } + } + }) + } +} diff --git a/internal/unifiedresources/clone.go b/internal/unifiedresources/clone.go index 0beca03f35..bebac8b929 100644 --- a/internal/unifiedresources/clone.go +++ b/internal/unifiedresources/clone.go @@ -69,6 +69,7 @@ func cloneResourceData(in *Resource) Resource { out.Metrics = cloneResourceMetrics(in.Metrics) out.ParentID = cloneStringPtr(in.ParentID) out.parentBySource = cloneParentBySourceMap(in.parentBySource) + out.linkFolds = cloneManualLinkFolds(in.linkFolds) out.Tags = cloneStringSlice(in.Tags) out.Incidents = cloneResourceIncidentSlice(in.Incidents) out.Relationships = cloneResourceRelationshipSlice(in.Relationships) diff --git a/internal/unifiedresources/clone_test.go b/internal/unifiedresources/clone_test.go index 63b87b54a1..5e3a88e1b2 100644 --- a/internal/unifiedresources/clone_test.go +++ b/internal/unifiedresources/clone_test.go @@ -269,6 +269,25 @@ func TestCloneResource_MutateParentBySource(t *testing.T) { } } +// Operator link folds ride clones, so a registry seeded from another +// registry's listing can name each link's pair, and stay detached from them. +func TestCloneResource_CarriesLinkFolds(t *testing.T) { + original := &Resource{ + ID: "vm-1", + linkFolds: []ManualLinkFold{{HolderID: "vm-1", FoldedID: "agent-1", Sources: []DataSource{SourceAgent}}}, + } + cloned := cloneResource(original) + if len(cloned.linkFolds) != 1 || cloned.linkFolds[0].FoldedID != "agent-1" { + t.Fatalf("clone lost link folds: %+v", cloned.linkFolds) + } + + cloned.linkFolds[0].Sources[0] = SourceDocker + cloned.linkFolds[0].FoldedID = "MUTATED" + if original.linkFolds[0].Sources[0] != SourceAgent || original.linkFolds[0].FoldedID != "agent-1" { + t.Errorf("mutating cloned link folds affected the original: %+v", original.linkFolds) + } +} + func TestCloneResource_MutateAvailabilityTimes(t *testing.T) { checkedAt := time.Date(2026, time.July, 9, 12, 0, 0, 0, time.UTC) succeededAt := checkedAt.Add(-time.Minute) @@ -1031,3 +1050,18 @@ func TestCloneMaterializedResourceDetachesCanonicalPolicy(t *testing.T) { t.Fatal("nil materialized clone changed absence") } } + +// The resources API seeds its registry from the monitor's read state, so the +// marker on a row saved-host continuity introduced must ride the clone, or the +// API's link pass would fold the saved agent into its linked guest. +func TestCloneResourceKeepsContinuityMarker(t *testing.T) { + saved := Resource{ID: "agent-saved", Type: ResourceTypeAgent, continuityOnly: true} + if clone := cloneResource(&saved); !clone.continuityOnly { + t.Fatal("clone dropped the continuity marker") + } + rr := NewRegistry(nil) + rr.IngestResources([]Resource{saved}) + if seeded, ok := rr.Get(saved.ID); !ok || !seeded.continuityOnly { + t.Fatalf("seeded row = %+v (listed=%v), want the continuity marker kept", seeded, ok) + } +} diff --git a/internal/unifiedresources/code_standards_test.go b/internal/unifiedresources/code_standards_test.go index 6bf168aa27..696a6ce507 100644 --- a/internal/unifiedresources/code_standards_test.go +++ b/internal/unifiedresources/code_standards_test.go @@ -729,6 +729,28 @@ func TestResourceAPIUsesCanonicalTenantUnifiedSeed(t *testing.T) { } } +// Report-merge undoes an operator link through the registry's fold record. A +// candidate ID derived from the merged resource's type names neither side of +// a link, and the resources API's registry, seeded from the monitor's +// already-linked listing, never holds the folded row to derive one from. +func TestResourceAPIReportMergeExcludesRegistryLinkFolds(t *testing.T) { + data, err := os.ReadFile(filepath.Join("..", "api", "resourceapi", "resources.go")) + if err != nil { + t.Fatalf("failed to read resources.go: %v", err) + } + source := string(data) + + for _, snippet := range []string{ + "linkFolds := registry.ManualLinkFolds(path)", + "if len(resource.Sources) < 2 && len(linkFolds) == 0 {", + "pairs = append(pairs, exclusionPair{fold.HolderID, fold.FoldedID})", + } { + if !strings.Contains(source, snippet) { + t.Fatalf("internal/api/resourceapi/resources.go report-merge must contain %q", snippet) + } + } +} + func TestAgentlessAvailabilityTargetKindStaysCanonical(t *testing.T) { requiredSnippets := map[string][]string{ filepath.Join("..", "config", "availability.go"): { diff --git a/internal/unifiedresources/manual_link_decisions.go b/internal/unifiedresources/manual_link_decisions.go new file mode 100644 index 0000000000..250826ec2c --- /dev/null +++ b/internal/unifiedresources/manual_link_decisions.go @@ -0,0 +1,67 @@ +package unifiedresources + +import "time" + +// An operator decides each resource pair one way: a link folds the pair into +// one resource, an exclusion keeps it apart (the resources API's unlink and +// report-merge record exclusions). The later decision replaces the earlier +// one. Stores drop the opposite row when they write either, and their reads +// pass through effectiveManualPairDecisions, so a pair that still carries both +// (rows written before writes replaced each other, or IDs that canonical-ID +// succession rewrote onto one pair) is decided by its later write. Every +// registry loads its links and exclusions from the store and resolves the two +// reads together, since a decision can land between them, so the monitor's +// rebuilds, the resources API and the views built from the monitor's links +// all apply the same decision. + +// effectiveManualPairDecisions keeps, for each resource pair, only its latest +// decision: the newest link (a later row wins a tie, so a reversed-order copy +// that succession left behind cannot override a newer link's primary) unless +// an exclusion is at least as new. A link and an exclusion written at the +// same instant keep the pair apart: two separate rows lose no data and the +// operator can link again. +func effectiveManualPairDecisions(links []ResourceLink, exclusions []ResourceExclusion) ([]ResourceLink, []ResourceExclusion) { + if len(links) == 0 { + return links, exclusions + } + newestLink := make(map[string]int, len(links)) + for i, link := range links { + key := exclusionKey(link.ResourceA, link.ResourceB) + if j, ok := newestLink[key]; !ok || !links[j].CreatedAt.After(link.CreatedAt) { + newestLink[key] = i + } + } + latestExclusion := make(map[string]time.Time, len(exclusions)) + for _, exclusion := range exclusions { + key := exclusionKey(exclusion.ResourceA, exclusion.ResourceB) + if at, ok := latestExclusion[key]; !ok || exclusion.CreatedAt.After(at) { + latestExclusion[key] = exclusion.CreatedAt + } + } + + keptLinks := make([]ResourceLink, 0, len(newestLink)) + for i, link := range links { + key := exclusionKey(link.ResourceA, link.ResourceB) + if newestLink[key] != i { + continue + } + if at, ok := latestExclusion[key]; ok && !link.CreatedAt.After(at) { + continue + } + keptLinks = append(keptLinks, link) + } + keptExclusions := make([]ResourceExclusion, 0, len(exclusions)) + for _, exclusion := range exclusions { + if i, ok := newestLink[exclusionKey(exclusion.ResourceA, exclusion.ResourceB)]; ok && links[i].CreatedAt.After(exclusion.CreatedAt) { + continue + } + keptExclusions = append(keptExclusions, exclusion) + } + return keptLinks, keptExclusions +} + +// sameManualPair reports whether two ID pairs name the same resources in +// either order. +func sameManualPair(a1, b1, a2, b2 string) bool { + return exclusionKey(a1, b1) == exclusionKey(a2, b2) +} diff --git a/internal/unifiedresources/manual_link_folds.go b/internal/unifiedresources/manual_link_folds.go new file mode 100644 index 0000000000..4091d9e66a --- /dev/null +++ b/internal/unifiedresources/manual_link_folds.go @@ -0,0 +1,78 @@ +package unifiedresources + +// ManualLinkFold records one operator link that applyManualLinks applied: the +// resource FoldedID names was merged into the one HolderID names. The two IDs +// are the link row's own pair, so an exclusion naming them replaces the link +// (the store keeps one decision per pair). A candidate ID derived from the +// merged resource's type cannot: for an agent linked into a VM it is a vm- ID +// that names neither side. Sources lists what the folded side brought, +// including anything links had already folded into it. +type ManualLinkFold struct { + HolderID string + FoldedID string + Sources []DataSource +} + +// recordManualLinkFold notes on the link's merged resource that it took in +// other through the link pair (primaryID, otherID), together with the folds +// other already held, so each link along a chain keeps its own pair. The +// record is unexported and rides in-memory clones, so a registry seeded from +// another registry's listing, as the resources API's is, sees the folds the +// monitor applied although it never holds the folded rows itself. A pair is +// recorded once: record ingest can recreate a folded side whose source +// mapping points at a holder of another type and fold it again, possibly +// with fewer sources than it brought before, so a repeated pair adds its +// sources to the fold already recorded. The holder still carries what the +// earlier fold brought. +func recordManualLinkFold(primary *Resource, primaryID string, other *Resource, otherID string) { + added := make([]ManualLinkFold, 0, len(other.linkFolds)+1) + added = append(added, other.linkFolds...) + added = append(added, ManualLinkFold{ + HolderID: primaryID, + FoldedID: otherID, + Sources: cloneDataSourceSlice(other.Sources), + }) + folds := make([]ManualLinkFold, 0, len(primary.linkFolds)+len(added)) + recorded := make(map[string]int, len(primary.linkFolds)+len(added)) + record := func(fold ManualLinkFold) { + key := exclusionKey(fold.HolderID, fold.FoldedID) + if i, ok := recorded[key]; ok { + folds[i].Sources = addSources(cloneDataSourceSlice(folds[i].Sources), fold.Sources) + return + } + recorded[key] = len(folds) + folds = append(folds, fold) + } + for _, fold := range primary.linkFolds { + record(fold) + } + for _, fold := range added { + record(fold) + } + primary.linkFolds = folds +} + +// ManualLinkFolds lists the operator links folded into a resource, one per +// link pair: the resource's own earlier folds first, then each folded side's +// folds ahead of the link that took that side in. +func (rr *ResourceRegistry) ManualLinkFolds(resourceID string) []ManualLinkFold { + rr.mu.RLock() + defer rr.mu.RUnlock() + resource := rr.resources[CanonicalResourceID(resourceID)] + if resource == nil { + return nil + } + return cloneManualLinkFolds(resource.linkFolds) +} + +func cloneManualLinkFolds(in []ManualLinkFold) []ManualLinkFold { + if in == nil { + return nil + } + out := make([]ManualLinkFold, len(in)) + for i, fold := range in { + fold.Sources = cloneDataSourceSlice(fold.Sources) + out[i] = fold + } + return out +} diff --git a/internal/unifiedresources/monitor_adapter.go b/internal/unifiedresources/monitor_adapter.go index 6abf5dfdbf..cced6ca6e2 100644 --- a/internal/unifiedresources/monitor_adapter.go +++ b/internal/unifiedresources/monitor_adapter.go @@ -135,7 +135,7 @@ func readStateWithRecords(readState ReadState, source DataSource, records []Inge cloned := NewRegistry(registry.store) thresholds := adapter.currentStaleThresholds() cloned.IngestResourcesWithStaleThresholds(registry.List(), thresholds) - cloned.ingestRecords(source, records, onlyMissing) + cloned.ingestRecords(source, records, onlyMissing, thresholds) overlay := NewMonitorAdapterWithStaleThresholds(cloned, thresholds) if keyed { adapter.overlays.store(registry, rebuiltAt, key, now, overlay) @@ -372,6 +372,30 @@ func (a *MonitorAdapter) LastRebuiltAt() time.Time { return a.lastRebuiltAt } +// RegistryGeneration identifies one generation a MonitorAdapter published: +// the registry it serves and when that registry last changed. LastRebuiltAt +// alone does not, since two rebuilds of a snapshot stamped in the future +// share its time; a rebuild swaps the registry and an incremental update +// restamps it. +type RegistryGeneration struct { + registry *ResourceRegistry + rebuiltAt int64 +} + +// Generation returns the generation the adapter currently publishes. +func (a *MonitorAdapter) Generation() RegistryGeneration { + if a == nil { + return RegistryGeneration{} + } + a.mu.RLock() + defer a.mu.RUnlock() + generation := RegistryGeneration{registry: a.registry} + if !a.lastRebuiltAt.IsZero() { + generation.rebuiltAt = a.lastRebuiltAt.UnixNano() + } + return generation +} + // TryReplaceRegistryForRead rebuilds the registry on behalf of a read path // (/api/state, websocket hydrate). Unlike the ingest-driven populate methods // it never queues behind an in-flight rebuild and skips entirely while the @@ -450,7 +474,7 @@ func (a *MonitorAdapter) replaceRegistryLocked(snapshot models.StateSnapshot, re if len(records) == 0 || strings.TrimSpace(string(source)) == "" { continue } - rebuilt.IngestRecords(source, records) + rebuilt.IngestRecordsWithStaleThresholds(source, records, staleThresholds) } // IngestSnapshot runs its stale pass before the record sources above are // ingested, so record-sourced resources would otherwise keep their @@ -665,7 +689,7 @@ func (a *MonitorAdapter) PopulateSupplementalRecords(source DataSource, records return } before := registry.List() - registry.IngestRecords(source, records) + registry.IngestRecordsWithStaleThresholds(source, records, a.currentStaleThresholds()) rebuiltAt := time.Now().UTC() recordRegistryChanges(registry.store, before, registry.List(), rebuiltAt, nil, SourcePlatformEvent, changeSourceAdapterForDataSource(source)) registry.PersistIdentityPins() diff --git a/internal/unifiedresources/monitor_adapter_read_state_test.go b/internal/unifiedresources/monitor_adapter_read_state_test.go index 7eefcb5b99..55a8298d70 100644 --- a/internal/unifiedresources/monitor_adapter_read_state_test.go +++ b/internal/unifiedresources/monitor_adapter_read_state_test.go @@ -400,6 +400,234 @@ func TestReadStateWithRecordsPreservesConfiguredStaleThresholds(t *testing.T) { } } +// A link whose side arrives as a record joins during record ingest, so it +// must judge freshness by the adapter's thresholds as a snapshot-time link +// does. Under a five-minute vSphere threshold a VM polled three minutes ago +// is current, and its own memory reading outranks its in-guest agent's. +func TestMonitorAdapterJoinsLinkedRecordsWithConfiguredStaleThresholds(t *testing.T) { + now := time.Now().UTC() + snapshot := models.StateSnapshot{ + Hosts: []models.Host{{ + ID: "host-app-guest", + Hostname: "app-guest", + MachineID: "machine-app-guest", + Status: "online", + LastSeen: now, + Memory: models.Memory{Total: 8 << 30, Used: 6 << 30, Free: 2 << 30, Usage: 75}, + }}, + LastUpdate: now, + } + vmRecords := []IngestRecord{{ + SourceID: "vc-1:vm:vm-42", + Resource: Resource{ + Type: ResourceTypeVM, + Technology: "vmware", + Name: "app-guest", + Status: StatusOnline, + LastSeen: now.Add(-3 * time.Minute), + Metrics: &ResourceMetrics{Memory: &MetricValue{Percent: 40, Source: SourceVMware}}, + VMware: &VMwareData{ConnectionID: "vc-1", ManagedObjectID: "vm-42", EntityType: "vm"}, + }, + }} + thresholds := map[DataSource]time.Duration{SourceVMware: 5 * time.Minute} + + unlinked := NewMonitorAdapter(NewRegistry(nil)) + unlinked.PopulateSnapshotAndSupplemental(snapshot, map[DataSource][]IngestRecord{SourceVMware: vmRecords}) + vms, hosts := unlinked.VMs(), unlinked.Hosts() + if len(vms) != 1 || len(hosts) != 1 { + t.Fatalf("unlinked estate = %d VMs, %d agents, want one of each", len(vms), len(hosts)) + } + link := ResourceLink{ResourceA: vms[0].ID(), ResourceB: hosts[0].ID(), PrimaryID: hosts[0].ID()} + + for _, path := range []struct { + name string + read func(adapter *MonitorAdapter) ReadState + }{ + {"rebuild", func(adapter *MonitorAdapter) ReadState { + adapter.PopulateSnapshotAndSupplemental(snapshot, map[DataSource][]IngestRecord{SourceVMware: vmRecords}) + return adapter + }}, + {"supplemental", func(adapter *MonitorAdapter) ReadState { + adapter.PopulateFromSnapshot(snapshot) + adapter.PopulateSupplementalRecords(SourceVMware, vmRecords) + return adapter + }}, + {"overlay", func(adapter *MonitorAdapter) ReadState { + adapter.PopulateFromSnapshot(snapshot) + return ReadStateWithRecords(adapter, SourceVMware, vmRecords) + }}, + } { + t.Run(path.name, func(t *testing.T) { + store := NewMemoryStore() + if err := store.AddLink(link); err != nil { + t.Fatalf("add link: %v", err) + } + readState := path.read(NewMonitorAdapterWithStaleThresholds(NewRegistry(store), thresholds)) + + if hosts := readState.Hosts(); len(hosts) != 0 { + t.Fatalf("linked agent still listed standalone: %s", hosts[0].ID()) + } + vms := readState.VMs() + if len(vms) != 1 || vms[0].ID() != link.ResourceA { + t.Fatalf("VMs = %d, want the linked VM %s", len(vms), link.ResourceA) + } + if vms[0].r.Metrics == nil || vms[0].r.Metrics.Memory == nil { + t.Fatal("linked VM lost its memory reading") + } + if memory := vms[0].r.Metrics.Memory; memory.Source != SourceVMware || memory.Percent != 40 { + t.Fatalf("linked VM memory = %.0f%% from %s, want vSphere's current 40%%", memory.Percent, memory.Source) + } + }) + } +} + +// Every source merge into an existing row judges metric freshness by the +// adapter's configured thresholds, as the rebuild's links and stale pass do. +// A linked vSphere VM first reports no memory, so its in-guest agent supplies +// it; a refresh then brings vSphere memory observed three minutes ago, current +// under a five-minute threshold. The live supplemental refresh and the +// read-state overlay merge that record into the existing row, and must show +// vSphere's memory as a rebuild from the same records does. A Proxmox node +// polled every two minutes is likewise current at ninety seconds, so the +// rebuild keeps its readings over those of its auto-linked agent, silent for +// seventy-five seconds. +func TestMonitorAdapterSourceMergesUseConfiguredStaleThresholds(t *testing.T) { + now := time.Now().UTC() + thresholds := map[DataSource]time.Duration{SourceProxmox: 4 * time.Minute, SourceVMware: 5 * time.Minute} + + t.Run("linked vSphere VM refresh", func(t *testing.T) { + snapshot := models.StateSnapshot{ + Hosts: []models.Host{{ + ID: "host-app-guest", + Hostname: "app-guest", + MachineID: "machine-app-guest", + Status: "online", + LastSeen: now, + Memory: models.Memory{Total: 8 << 30, Used: 6 << 30, Free: 2 << 30, Usage: 75}, + }}, + LastUpdate: now, + } + vmRecord := func(lastSeen time.Time, metrics *ResourceMetrics) []IngestRecord { + return []IngestRecord{{ + SourceID: "vc-1:vm:vm-42", + Resource: Resource{ + Type: ResourceTypeVM, + Technology: "vmware", + Name: "app-guest", + Status: StatusOnline, + LastSeen: lastSeen, + Metrics: metrics, + VMware: &VMwareData{ConnectionID: "vc-1", ManagedObjectID: "vm-42", EntityType: "vm"}, + }, + }} + } + initial := vmRecord(now.Add(-4*time.Minute), &ResourceMetrics{CPU: &MetricValue{Percent: 10, Source: SourceVMware}}) + refreshedAt := now.Add(-3 * time.Minute) + refreshed := vmRecord(refreshedAt, &ResourceMetrics{ + CPU: &MetricValue{Percent: 10, Source: SourceVMware}, + Memory: &MetricValue{Percent: 40, Source: SourceVMware}, + }) + + unlinked := NewMonitorAdapter(NewRegistry(nil)) + unlinked.PopulateSnapshotAndSupplemental(snapshot, map[DataSource][]IngestRecord{SourceVMware: initial}) + vms, hosts := unlinked.VMs(), unlinked.Hosts() + if len(vms) != 1 || len(hosts) != 1 { + t.Fatalf("unlinked estate = %d VMs, %d agents, want one of each", len(vms), len(hosts)) + } + link := ResourceLink{ResourceA: vms[0].ID(), ResourceB: hosts[0].ID(), PrimaryID: hosts[0].ID()} + + for _, path := range []struct { + name string + read func(adapter *MonitorAdapter) ReadState + }{ + {"rebuild", func(adapter *MonitorAdapter) ReadState { + adapter.PopulateSnapshotAndSupplemental(snapshot, map[DataSource][]IngestRecord{SourceVMware: refreshed}) + return adapter + }}, + {"supplemental", func(adapter *MonitorAdapter) ReadState { + adapter.PopulateSnapshotAndSupplemental(snapshot, map[DataSource][]IngestRecord{SourceVMware: initial}) + adapter.PopulateSupplementalRecords(SourceVMware, refreshed) + return adapter + }}, + {"overlay", func(adapter *MonitorAdapter) ReadState { + adapter.PopulateSnapshotAndSupplemental(snapshot, map[DataSource][]IngestRecord{SourceVMware: initial}) + return ReadStateWithRecords(adapter, SourceVMware, refreshed) + }}, + } { + t.Run(path.name, func(t *testing.T) { + store := NewMemoryStore() + if err := store.AddLink(link); err != nil { + t.Fatalf("add link: %v", err) + } + readState := path.read(NewMonitorAdapterWithStaleThresholds(NewRegistry(store), thresholds)) + + vms := readState.VMs() + if len(vms) != 1 || vms[0].ID() != link.ResourceA || len(readState.Hosts()) != 0 { + t.Fatalf("estate = %d VMs, %d standalone agents, want only the linked VM %s", len(vms), len(readState.Hosts()), link.ResourceA) + } + if got := vms[0].r.SourceStatus[SourceVMware]; got.Status != "online" || !got.LastSeen.Equal(refreshedAt) { + t.Fatalf("vSphere sighting = %+v, want the refresh, current under the configured threshold", got) + } + if vms[0].r.Metrics == nil || vms[0].r.Metrics.Memory == nil { + t.Fatal("linked VM lost its memory reading") + } + if memory := vms[0].r.Metrics.Memory; memory.Source != SourceVMware || memory.Percent != 40 { + t.Fatalf("linked VM memory = %.0f%% from %s, want vSphere's current 40%%", memory.Percent, memory.Source) + } + }) + } + }) + + t.Run("node with auto-linked agent", func(t *testing.T) { + adapter := NewMonitorAdapterWithStaleThresholds(NewRegistry(nil), thresholds) + adapter.PopulateFromSnapshot(models.StateSnapshot{ + Nodes: []models.Node{{ + ID: "homelab-pve1", + Name: "pve1", + Instance: "homelab", + Status: "online", + LinkedAgentID: "host-pve1", + CPU: 0.12, + Memory: models.Memory{Total: 64 << 30, Used: 16 << 30, Free: 48 << 30, Usage: 25}, + LastSeen: now.Add(-90 * time.Second), + }}, + Hosts: []models.Host{{ + ID: "host-pve1", + MachineID: "machine-pve1", + Hostname: "pve1", + LinkedNodeID: "homelab-pve1", + Status: "online", + CPUUsage: 80, + Memory: models.Memory{Total: 64 << 30, Used: 48 << 30, Free: 16 << 30, Usage: 75}, + LastSeen: now.Add(-75 * time.Second), + IntervalSeconds: 30, + }}, + LastUpdate: now, + }) + + nodes := adapter.Nodes() + if len(nodes) != 1 { + t.Fatalf("nodes = %d, want the node merged with its agent", len(nodes)) + } + node := nodes[0].r + if poll := node.SourceStatus[SourceProxmox]; poll.Status != "online" || !poll.LastSeen.Equal(now.Add(-90*time.Second)) { + t.Fatalf("Proxmox sighting = %+v, want the current poll from ninety seconds ago", poll) + } + if got := node.SourceStatus[SourceAgent].Status; got != "stale" { + t.Fatalf("agent sighting = %q, want stale", got) + } + if node.Metrics == nil || node.Metrics.CPU == nil || node.Metrics.Memory == nil { + t.Fatalf("merged node lost its readings: %+v", node.Metrics) + } + if cpu := node.Metrics.CPU; cpu.Source != SourceProxmox || cpu.Percent != 12 { + t.Fatalf("node CPU = %.0f%% from %s, want the current poll's 12%%", cpu.Percent, cpu.Source) + } + if memory := node.Metrics.Memory; memory.Source != SourceProxmox || memory.Percent != 25 { + t.Fatalf("node memory = %.0f%% from %s, want the current poll's 25%%", memory.Percent, memory.Source) + } + }) +} + func TestHostContinuityCannotOverwriteCurrentResource(t *testing.T) { now := time.Now().UTC() for _, mode := range []string{"machine-identity", "source-identity", "future-timestamp", "provider-only"} { diff --git a/internal/unifiedresources/presentation_coalesce_test.go b/internal/unifiedresources/presentation_coalesce_test.go index 15c30f008a..a35335840a 100644 --- a/internal/unifiedresources/presentation_coalesce_test.go +++ b/internal/unifiedresources/presentation_coalesce_test.go @@ -358,6 +358,56 @@ func TestCoalescePresentationHostResourcesWithExclusionsHonorsManualSplit(t *tes } } +// The broadcast coalesces through the adapter, the resources API through +// ListForPresentation. Both must keep a host pair the operator split apart. +func TestMonitorAdapterCoalesceForPresentationHonoursExclusions(t *testing.T) { + now := time.Date(2026, 10, 7, 21, 0, 0, 0, time.UTC) + resources := []Resource{ + { + ID: "agent-runtime-alpha", Type: ResourceTypeAgent, Name: "alpha", Status: StatusOnline, LastSeen: now, + Sources: []DataSource{SourceAgent}, + Identity: ResourceIdentity{Hostnames: []string{"alpha"}}, + Agent: &AgentData{AgentID: "agent-alpha", Hostname: "alpha"}, + }, + { + ID: "agent-docker-alpha", Type: ResourceTypeAgent, Name: "alpha", Status: StatusOnline, LastSeen: now, + Sources: []DataSource{SourceDocker}, + Identity: ResourceIdentity{Hostnames: []string{"alpha"}}, + Docker: &DockerData{Hostname: "alpha"}, + }, + } + for _, split := range []bool{false, true} { + store := NewMemoryStore() + if split { + if err := store.AddExclusion(ResourceExclusion{ResourceA: "agent-docker-alpha", ResourceB: "agent-runtime-alpha"}); err != nil { + t.Fatal(err) + } + } + registry := NewRegistry(store) + registry.IngestResources(resources) + adapter := NewMonitorAdapter(registry) + want := 1 + if split { + want = 2 + } + rows, ok := adapter.CoalesceForPresentation(adapter.GetAll()) + if !ok || len(rows) != want { + t.Fatalf("split=%v: adapter coalesced %d rows (ok=%v), want %d", split, len(rows), ok, want) + } + if got := len(registry.ListForPresentation()); got != want { + t.Fatalf("split=%v: ListForPresentation gave %d rows, want %d", split, got, want) + } + } + + // A registry without a store holds no operator decisions, so it declines + // rather than coalescing as if none existed. + storeless := NewRegistry(nil) + storeless.IngestResources(resources) + if _, ok := NewMonitorAdapter(storeless).CoalesceForPresentation(resources); ok { + t.Fatal("a store-less adapter claimed to carry operator decisions") + } +} + // A Proxmox node whose Pulse Agent has gone offline must show the live PVE CPU, // not the agent's last (0) reading. The agent is the presentation primary, so // without a freshness gate its stale 0 CPU was kept and the live value dropped. diff --git a/internal/unifiedresources/presentation_exclusions.go b/internal/unifiedresources/presentation_exclusions.go new file mode 100644 index 0000000000..9ceb598389 --- /dev/null +++ b/internal/unifiedresources/presentation_exclusions.go @@ -0,0 +1,44 @@ +package unifiedresources + +// presentationExclusionFilter reports whether the presentation host coalesce +// must keep two resources apart because the operator split the pair (unlink +// or report-merge). It reads a copy of the registry's exclusions, so callers +// coalesce without holding the registry lock; nil means no pair is excluded. +func (rr *ResourceRegistry) presentationExclusionFilter() func(left, right Resource) bool { + rr.mu.RLock() + if len(rr.exclusions) == 0 { + rr.mu.RUnlock() + return nil + } + exclusions := make(map[string]struct{}, len(rr.exclusions)) + for key := range rr.exclusions { + exclusions[key] = struct{}{} + } + rr.mu.RUnlock() + + return func(left, right Resource) bool { + leftID := CanonicalResourceID(left.ID) + rightID := CanonicalResourceID(right.ID) + if leftID == "" || rightID == "" { + return false + } + _, ok := exclusions[exclusionKey(leftID, rightID)] + return ok + } +} + +// CoalesceForPresentation applies the presentation host coalesce to resources +// listed from this adapter's current registry generation, honouring the +// operator's merge exclusions through the same filter that +// ResourceRegistry.ListForPresentation uses, so the websocket broadcast keeps +// a pair the operator split apart as the resources API does. It reports +// false, leaving resources untouched, when the registry has no store and so +// carries no operator decisions: the mock view and other read states built +// from an already-unified list. +func (a *MonitorAdapter) CoalesceForPresentation(resources []Resource) ([]Resource, bool) { + registry := a.currentRegistry() + if registry == nil || registry.store == nil { + return resources, false + } + return CoalescePresentationHostResourcesWithExclusions(resources, registry.presentationExclusionFilter()), true +} diff --git a/internal/unifiedresources/registry.go b/internal/unifiedresources/registry.go index 48f2148fcd..02928ca3a0 100644 --- a/internal/unifiedresources/registry.go +++ b/internal/unifiedresources/registry.go @@ -5,6 +5,7 @@ import ( "encoding/hex" "fmt" "log" + "slices" "sort" "strings" "sync" @@ -95,6 +96,27 @@ type ResourceRegistry struct { exclusions map[string]struct{} identityPins *identityPinIndex pbsBackups []models.PBSBackup + // linkFoldIndex maps each canonical ID a manual link folded into another + // resource to the rows that took it in (Resource.linkFolds). Folded + // resources are still observed, so identity succession treats them as + // live, and references to them resolve to the row holding them. Holders + // carry their folds, so a registry seeded from another registry's listing + // indexes the same ones. Entries are rows, not IDs, and are checked on + // read (linkFoldHoldersLocked): a holder folded away, replaced or re-keyed + // needs no bookkeeping. + linkFoldIndex map[string][]*Resource + // linkOwnPins holds the identity pin each side of a manual link derives + // from its own sources, captured before the link merged them. A nil + // value records a side with no pinnable identity. PersistIdentityPins + // writes these instead of the merged projection, so a link never lends + // one machine's identity keys to another resource's durable pin. + linkOwnPins map[string]*ResourceIdentityPin + // linkHolds maps a resource that exists only through saved-host + // continuity, and that an operator link would otherwise fold away, to + // the link's primary (holdLinkedResourceLocked). Both rows stay listed + // with their own telemetry; references to the held row resolve to the + // primary. Every link pass recomputes it. + linkHolds map[string]string // supersededIndex maps record-declared retired canonical IDs to the live // resource that superseded them, so references persisted under a retired // ID (availability links, API reads) keep resolving. An empty value marks @@ -148,6 +170,13 @@ type ResourceRegistry struct { // what the full scan would see; scores stay live through the shared // pointers. agentNodeScanIndex map[string][]agentNodeCandidate + + // ingestStaleThresholds holds the caller-owned freshness thresholds of + // the snapshot or record ingest in progress, so each source merge into an + // existing row judges metric freshness as that ingest's manual links and + // stale pass do. nil outside an ingest and for callers without + // thresholds, where all three use the defaults. + ingestStaleThresholds map[DataSource]time.Duration } // agentNodeCandidate pairs a resources-map key with its entry so the indexed @@ -222,6 +251,9 @@ func (rr *ResourceRegistry) loadOverrides() { } exclusions, err := rr.store.GetExclusions() if err == nil { + // An unlink or link landing between the two reads would load both + // decisions for its pair (manual_link_decisions.go). + rr.links, exclusions = effectiveManualPairDecisions(rr.links, exclusions) for _, exclusion := range exclusions { key := exclusionKey(exclusion.ResourceA, exclusion.ResourceB) rr.exclusions[key] = struct{}{} @@ -243,6 +275,10 @@ func (rr *ResourceRegistry) IngestSnapshotWithStaleThresholds(snapshot models.St } func (rr *ResourceRegistry) ingestSnapshot(snapshot models.StateSnapshot, thresholds map[DataSource]time.Duration) { + rr.mu.Lock() + rr.ingestStaleThresholds = thresholds + rr.mu.Unlock() + hostByID := make(map[string]*models.Host, len(snapshot.Hosts)) for i := range snapshot.Hosts { host := snapshot.Hosts[i] @@ -606,15 +642,26 @@ func (rr *ResourceRegistry) ingestSnapshot(snapshot models.StateSnapshot, thresh rr.refreshCanonicalIdentitiesLocked() rr.markStaleLocked(time.Now().UTC(), thresholds) rr.invalidateViewsLocked() + rr.ingestStaleThresholds = nil rr.mu.Unlock() } // IngestRecords ingests normalized records for a single source. func (rr *ResourceRegistry) IngestRecords(source DataSource, records []IngestRecord) { - rr.ingestRecords(source, records, false) + rr.ingestRecords(source, records, false, nil) } -func (rr *ResourceRegistry) ingestRecords(source DataSource, records []IngestRecord, onlyMissing bool) { +// IngestRecordsWithStaleThresholds ingests normalized records for a single +// source and merges operator-linked resources with caller-owned thresholds. +func (rr *ResourceRegistry) IngestRecordsWithStaleThresholds(source DataSource, records []IngestRecord, thresholds map[DataSource]time.Duration) { + rr.ingestRecords(source, records, false, thresholds) +} + +func (rr *ResourceRegistry) ingestRecords(source DataSource, records []IngestRecord, onlyMissing bool, thresholds map[DataSource]time.Duration) { + rr.mu.Lock() + rr.ingestStaleThresholds = thresholds + rr.mu.Unlock() + var successions []CanonicalIDSuccession supersededSeen := make(map[string]struct{}) for _, record := range records { @@ -649,6 +696,17 @@ func (rr *ResourceRegistry) ingestRecords(source DataSource, records []IngestRec rr.applyRecordSuccessions(successions) rr.mu.Lock() + // Either side of an operator link can arrive as a record: a vSphere or + // TrueNAS VM with a Pulse agent inside, or an agent on a TrueNAS host. + // Snapshot and resource ingest apply the links, so record ingest must too, + // or the monitor's rebuilt registry keeps both rows while REST, which + // seeds from that registry, shows them merged. Continuity records only + // fill absent machines and never merge into a link (a saved enrollment + // joined to its linked guest would lend the guest its offline verdict + // and old agent payload), but the link still names their identity: the + // pass holds them beside the primary and resolves references to them + // there. + rr.applyManualLinks(thresholds) rr.refreshStorageConsumersLocked() rr.refreshPBSRollupsLocked() rr.refreshStoragePostureLocked() @@ -657,6 +715,7 @@ func (rr *ResourceRegistry) ingestRecords(source DataSource, records []IngestRec rr.refreshProxmoxSensorSetupLocked() rr.refreshCanonicalIdentitiesLocked() rr.invalidateViewsLocked() + rr.ingestStaleThresholds = nil rr.mu.Unlock() } @@ -746,12 +805,76 @@ func (rr *ResourceRegistry) proxmoxGuestResourceIDForSourceRefLocked(ref string) return uniqueID } +// canonicalIDObservedLocked reports whether a canonical ID still names an +// observed resource: a live row, or one a manual link folded into its primary. +// Succeeding an observed ID would hand its operator rows, and the link that +// folded it, to another resource. +func (rr *ResourceRegistry) canonicalIDObservedLocked(id string) bool { + if _, live := rr.resources[id]; live { + return true + } + return len(rr.linkFoldHoldersLocked(id)) > 0 +} + +// indexLinkFoldsLocked records that holder took in the canonical IDs its +// link folds name (Resource.linkFolds, recordManualLinkFold). +func (rr *ResourceRegistry) indexLinkFoldsLocked(holder *Resource) { + if holder == nil || len(holder.linkFolds) == 0 { + return + } + if rr.linkFoldIndex == nil { + rr.linkFoldIndex = make(map[string][]*Resource) + } + for _, fold := range holder.linkFolds { + if !slices.Contains(rr.linkFoldIndex[fold.FoldedID], holder) { + rr.linkFoldIndex[fold.FoldedID] = append(rr.linkFoldIndex[fold.FoldedID], holder) + } + } +} + +// holdsLinkFold reports whether a resource's link folds name foldedID. +func holdsLinkFold(resource *Resource, foldedID string) bool { + for _, fold := range resource.linkFolds { + if fold.FoldedID == foldedID { + return true + } + } + return false +} + +// linkFoldHoldersLocked lists the live rows that hold a canonical ID an +// operator link folded into them. More than one is ambiguous. +func (rr *ResourceRegistry) linkFoldHoldersLocked(ref string) []string { + ref = CanonicalResourceID(ref) + var holders []string + for _, holder := range rr.linkFoldIndex[ref] { + if rr.resources[holder.ID] != holder || !holdsLinkFold(holder, ref) || + slices.Contains(holders, holder.ID) { + continue + } + holders = append(holders, holder.ID) + } + return holders +} + +// linkFoldHolderLocked resolves a canonical ID an operator link folded into +// another resource to the row holding it. ambiguous reports two holders: the +// reference must then resolve to nothing, not fall through to weaker matches. +func (rr *ResourceRegistry) linkFoldHolderLocked(ref string) (holderID string, ambiguous bool) { + holders := rr.linkFoldHoldersLocked(ref) + if len(holders) != 1 { + return "", len(holders) > 1 + } + return holders[0], false +} + // applyRecordSuccessions re-keys operator-owned store rows from canonical IDs // that ingested records declared superseded (IngestRecord.SupersededCanonicalIDs) // onto the records' current canonical IDs. Mirrors the guards of pin-driven -// successions in PersistIdentityPins: a superseded ID still held by a live -// resource is skipped so a genuinely distinct sibling never has its rows -// stolen. Re-runs are cheap no-ops once the old rows are gone. +// successions in PersistIdentityPins: a superseded ID still observed, live or +// folded into another resource by a manual link, is skipped so a genuinely +// distinct sibling never has its rows stolen. Re-runs are cheap no-ops once +// the old rows are gone. func (rr *ResourceRegistry) applyRecordSuccessions(successions []CanonicalIDSuccession) { if len(successions) == 0 || rr.store == nil { return @@ -764,7 +887,7 @@ func (rr *ResourceRegistry) applyRecordSuccessions(successions []CanonicalIDSucc rr.mu.RLock() kept := make([]CanonicalIDSuccession, 0, len(successions)) for _, succession := range successions { - if _, live := rr.resources[succession.OldCanonicalID]; live { + if rr.canonicalIDObservedLocked(succession.OldCanonicalID) { continue } kept = append(kept, succession) @@ -842,6 +965,7 @@ func (rr *ResourceRegistry) ingestResources(resources []Resource, thresholds map rr.seedSourceMappingsFromResourceLocked(rr.resources[resourceID]) if resource := rr.resources[resourceID]; resource != nil { rr.indexSupersededCanonicalIDsLocked(resourceID, resource.SupersededCanonicalIDs) + rr.indexLinkFoldsLocked(resource) rr.indexPhysicalDiskHardwareLocked(resource) } } @@ -1382,23 +1506,7 @@ func (rr *ResourceRegistry) listMaterialized(withTargets bool) ([]Resource, map[ // merge exclusions. func (rr *ResourceRegistry) ListForPresentation() []Resource { resources := rr.List() - - rr.mu.RLock() - exclusions := make(map[string]struct{}, len(rr.exclusions)) - for key := range rr.exclusions { - exclusions[key] = struct{}{} - } - rr.mu.RUnlock() - - return CoalescePresentationHostResourcesWithExclusions(resources, func(left, right Resource) bool { - leftID := CanonicalResourceID(left.ID) - rightID := CanonicalResourceID(right.ID) - if leftID == "" || rightID == "" { - return false - } - _, ok := exclusions[exclusionKey(leftID, rightID)] - return ok - }) + return CoalescePresentationHostResourcesWithExclusions(resources, rr.presentationExclusionFilter()) } // ListByType returns all resources of the provided type. @@ -1483,8 +1591,18 @@ func (rr *ResourceRegistry) ResolveReferenceID(ref string) (string, bool) { } // The second result requests a one-time alias-index build after the read lock -// is released. Exact and source references never pay that cost. +// is released. Exact and source references never pay that cost. A reference +// to a saved link member held beside its primary resolves to the primary, +// as one to a folded member does: the operator's link names one identity. func (rr *ResourceRegistry) resolveReferenceIDLocked(ref string) (string, bool) { + resolvedID, needsAliasIndex := rr.resolveReferenceRowIDLocked(ref) + if resolvedID == "" { + return "", needsAliasIndex + } + return rr.linkHoldTargetLocked(resolvedID), false +} + +func (rr *ResourceRegistry) resolveReferenceRowIDLocked(ref string) (string, bool) { if ref == "" { return "", false } @@ -1494,6 +1612,11 @@ func (rr *ResourceRegistry) resolveReferenceIDLocked(ref string) (string, bool) if resolvedID := rr.supersededResourceIDLocked(ref); rr.resources[resolvedID] != nil { return resolvedID, false } + // A resource an operator link folded into another is that row now, as + // its agent alert references ("agent:") already are. + if resolvedID, ambiguous := rr.linkFoldHolderLocked(ref); resolvedID != "" || ambiguous { + return resolvedID, false + } if resolvedID := rr.uniqueSourceResourceIDLocked(ref); rr.resources[resolvedID] != nil { return resolvedID, false } @@ -1539,7 +1662,7 @@ func (rr *ResourceRegistry) uniqueCanonicalIdentityResourceIDLocked(ref string) matches := map[string]struct{}{} for resourceID, resource := range rr.resources { if resourceMatchesCanonicalIdentityReference(resource, ref) { - matches[resourceID] = struct{}{} + matches[rr.linkHoldTargetLocked(resourceID)] = struct{}{} } } return uniqueResourceIDMatch(matches) @@ -1772,26 +1895,15 @@ func sourceVerdictsRecorded(sightings map[DataSource]SourceStatus) bool { } func (rr *ResourceRegistry) markStaleLocked(now time.Time, thresholds map[DataSource]time.Duration) { - thresholds = effectiveStaleThresholds(thresholds) - changed := false for _, resource := range rr.resources { previousStatus := resource.Status staleFound := false for source, status := range resource.SourceStatus { - threshold, ok := thresholds[source] - if !ok { - threshold = 120 * time.Second - } - if status.ExpectedUpdateIntervalSeconds > 0 { - // Slow inventory polls have their own cadence. A source must miss - // two expected intervals before its retained observation is stale. - threshold = max(threshold, 2*time.Duration(status.ExpectedUpdateIntervalSeconds)*time.Second) - } if status.LastSeen.IsZero() { continue } - if now.Sub(status.LastSeen) > threshold { + if now.Sub(status.LastSeen) > sourceStaleThreshold(source, status, thresholds) { changed = changed || status.Status != "stale" status.Status = "stale" resource.SourceStatus[source] = status @@ -2884,7 +2996,7 @@ func (rr *ResourceRegistry) ingestRecord(source DataSource, sourceID string, res // pulse-agent's pin on a same-named host) completing the identity would // re-merge systems the connection scoping keeps apart. if source != SourceTrueNAS { - identity = rr.completeIdentityFromPins(source, resource, identity) + identity = rr.completeIdentityFromPins(source, sourceID, resource, identity) } resource.Identity = identity resource.Sources = []DataSource{source} @@ -2974,6 +3086,7 @@ func (rr *ResourceRegistry) ingestRecord(source DataSource, sourceID string, res if onlyMissing && rr.resources[resource.ID] != nil { return "" } + resource.continuityOnly = onlyMissing stampPhysicalDiskTemperatureReading(&resource, source, sourceID) rr.resources[resource.ID] = &resource rr.indexPhysicalDiskHardwareLocked(&resource) @@ -3000,6 +3113,7 @@ func (rr *ResourceRegistry) ingestRecord(source DataSource, sourceID string, res rr.matcher.Add(existing.ID, existing.Identity) return existing.ID } + resource.continuityOnly = onlyMissing stampPhysicalDiskTemperatureReading(&resource, source, sourceID) rr.resources[resource.ID] = &resource rr.indexPhysicalDiskHardwareLocked(&resource) @@ -3576,17 +3690,23 @@ func (rr *ResourceRegistry) resolveAvailabilityLinkedResource(ref string, incomi exactID := CanonicalResourceID(ref) if existing := rr.resources[exactID]; existing != nil { if !isAvailabilityOwnedResource(*existing) { - return exactID + // A saved link member held beside its primary answers there. + return rr.linkHoldTargetLocked(exactID) } return "" } - // References persisted under a retired canonical ID or under a - // node-scoped guest source ID follow the resource across identity eras - // and live migrations. These arms resolve provider-declared persistence - // keys, not display aliases, so the explicit link stays fail-closed. + // References persisted under a retired canonical ID, under a resource an + // operator link folded into another, or under a node-scoped guest source + // ID follow the resource across identity eras, links and live + // migrations. These arms resolve provider-declared persistence keys and + // operator links, not display aliases, so the explicit link stays + // fail-closed. eligible := func(candidateID string) string { candidateID = CanonicalResourceID(candidateID) + if candidateID != "" { + candidateID = rr.linkHoldTargetLocked(candidateID) + } existing := rr.resources[candidateID] if existing != nil && !isAvailabilityOwnedResource(*existing) { return candidateID @@ -3596,6 +3716,9 @@ func (rr *ResourceRegistry) resolveAvailabilityLinkedResource(ref string, incomi if candidateID := eligible(rr.supersededResourceIDLocked(exactID)); candidateID != "" { return candidateID } + if holderID, ambiguous := rr.linkFoldHolderLocked(exactID); holderID != "" || ambiguous { + return eligible(holderID) + } if candidateID := eligible(rr.uniqueSourceResourceIDLocked(ref)); candidateID != "" { return candidateID } @@ -3809,6 +3932,8 @@ func (rr *ResourceRegistry) mergeInto(existing *Resource, incoming Resource, sou if existing == nil { return } + // A source reported this row, so it is an observation again. + existing.continuityOnly = false rr.setSourceParent(existing, source, incoming.ParentID) @@ -3940,7 +4065,7 @@ func (rr *ResourceRegistry) mergeInto(existing *Resource, incoming Resource, sou existing.ParentID = rr.resolveCanonicalParentID(existing) existing.Status = chooseStatus(existing.Status, incoming.Status, source, existing.Sources) - existing.Metrics = mergeMetrics(existing, existing.Metrics, incoming.Metrics, source, now, existing.SourceStatus, nil) + existing.Metrics = mergeMetrics(existing, existing.Metrics, incoming.Metrics, source, now, existing.SourceStatus, rr.ingestStaleThresholds) existing.Metrics = clearUnavailableSourceMemoryMetric(existing.Metrics, &incoming, source) // Prefer agent naming when available @@ -4764,12 +4889,21 @@ func mergeVMwareData(existing *VMwareData, incoming *VMwareData) *VMwareData { } func (rr *ResourceRegistry) applyManualLinks(thresholds map[DataSource]time.Duration) { + if rr.linkHolds != nil { + rr.linkHolds = nil + rr.canonicalIdentityIndex = nil + } if len(rr.links) == 0 { return } for _, link := range rr.links { + // A link joins only the pair its row names. Canonical-ID succession + // can leave a row whose endpoint re-key collided with the successor's + // own row while its primary moved on to the successor; honouring that + // primary merged the successor through a row naming a retired ID, which + // an unlink or report-merge of the merged pair leaves behind. primaryID := link.PrimaryID - if primaryID == "" { + if primaryID != link.ResourceA && primaryID != link.ResourceB { primaryID = link.ResourceA } primary := rr.resources[primaryID] @@ -4808,13 +4942,105 @@ func (rr *ResourceRegistry) applyManualLinks(thresholds map[DataSource]time.Dura primary, other = other, primary primaryID, otherID = otherID, primaryID } + if primary.continuityOnly || other.continuityOnly { + rr.holdLinkedResourceLocked(primaryID, otherID, other) + continue + } + rr.recordLinkOwnPin(primaryID, primary) + rr.recordLinkOwnPin(otherID, other) rr.mergeResourceData(primary, other, thresholds) + recordManualLinkFold(primary, primaryID, other, otherID) delete(rr.resources, otherID) + // The fold record names every ID along a chain of links, so the chain + // resolves to its last primary; the folded row's own index entries + // lapse on read now that it has left the registry. + rr.indexLinkFoldsLocked(primary) rr.updateSourceMappings(otherID, primaryID) } } +// recordLinkOwnPin captures the identity pin a manual-link side derives from +// its own sources, before the link merge projects the other side's identity +// onto the primary. The first capture in a registry generation wins, so a +// primary that absorbs several links keeps the pin it had before any of them. +func (rr *ResourceRegistry) recordLinkOwnPin(id string, resource *Resource) { + if _, recorded := rr.linkOwnPins[id]; recorded { + return + } + if rr.linkOwnPins == nil { + rr.linkOwnPins = make(map[string]*ResourceIdentityPin) + } + if pin, ok := identityPinForResource(resource); ok { + rr.linkOwnPins[id] = &pin + return + } + rr.linkOwnPins[id] = nil +} + +// holdLinkedResourceLocked keeps a link whose member exists only through +// saved-host continuity out of the fold. A saved enrollment is not an +// observation: folded into a live resource it would lend that resource its +// offline verdict and old payload, and a live resource folded into it would +// hide live telemetry behind it. The link still names one identity, so a +// saved member the fold would have taken in answers to the primary, +// as a folded one does. A live member keeps its own references even when the +// saved member is the primary: a live row never resolves to a saved one. The +// first link to claim a member holds it, as the first fold takes it in. +func (rr *ResourceRegistry) holdLinkedResourceLocked(primaryID, otherID string, other *Resource) { + if !other.continuityOnly { + return + } + if _, held := rr.linkHolds[otherID]; held { + return + } + if rr.linkHolds == nil { + rr.linkHolds = make(map[string]string) + } + rr.linkHolds[otherID] = primaryID + rr.canonicalIdentityIndex = nil +} + +// linkHoldPrimaryLocked returns the row a held link member's references +// resolve to: its primary, followed through a later fold of the primary into +// another resource and through further holds (a saved primary held under +// another link). Empty when the row is not held or the chain does not end on +// a listed row. +func (rr *ResourceRegistry) linkHoldPrimaryLocked(resourceID string) string { + primaryID, held := rr.linkHolds[resourceID] + if !held { + return "" + } + seen := map[string]struct{}{resourceID: {}} + for { + if rr.resources[primaryID] == nil { + holderID, _ := rr.linkFoldHolderLocked(primaryID) + if holderID == "" { + return "" + } + primaryID = holderID + } + next, held := rr.linkHolds[primaryID] + if !held { + return primaryID + } + if _, cycle := seen[primaryID]; cycle { + return "" + } + seen[primaryID] = struct{}{} + primaryID = next + } +} + +// linkHoldTargetLocked maps a resolved row to the row its references answer +// to: the hold primary for a held link member, the row itself otherwise. +func (rr *ResourceRegistry) linkHoldTargetLocked(resourceID string) string { + if primaryID := rr.linkHoldPrimaryLocked(resourceID); primaryID != "" { + return primaryID + } + return resourceID +} + func (rr *ResourceRegistry) mergeResourceData(primary *Resource, other *Resource, thresholds map[DataSource]time.Duration) { if other == nil || primary == nil { return @@ -4876,6 +5102,12 @@ func (rr *ResourceRegistry) mergeResourceData(primary *Resource, other *Resource if primary.Ceph == nil { primary.Ceph = other.Ceph } + if primary.TrueNAS == nil { + primary.TrueNAS = other.TrueNAS + } + if primary.VMware == nil { + primary.VMware = other.VMware + } // Manual links combine already-normalized resources. Preserve each metric's // recorded source instead of flattening the linked resource to SourceAgent. @@ -5312,6 +5544,9 @@ func (rr *ResourceRegistry) buildCanonicalIdentityIndexLocked() { if resource.Canonical == nil { continue } + // A saved link member held beside its primary shares the primary's + // identity, so an alias both carry (a hostname) stays unambiguous. + resourceID = rr.linkHoldTargetLocked(resourceID) indexCandidate(resource.Canonical.PrimaryID, resourceID) indexCandidate(resource.Canonical.PlatformID, resourceID) for _, alias := range resource.Canonical.Aliases { @@ -6284,6 +6519,27 @@ func mergeMetrics( return &merged } +// sourceStaleThreshold is how long a source may go without delivering before +// its sighting is stale: the caller's threshold for the source, else the +// default. The stale pass and the metric merge's freshness gate both read it, +// so the gate never calls a source current that the pass marks stale, or the +// reverse. +func sourceStaleThreshold(source DataSource, sighting SourceStatus, thresholds map[DataSource]time.Duration) time.Duration { + threshold := thresholds[source] + if threshold <= 0 { + threshold = defaultStaleThresholds[source] + } + if threshold <= 0 { + threshold = 120 * time.Second + } + if sighting.ExpectedUpdateIntervalSeconds > 0 { + // Slow inventory polls have their own cadence. A source must miss + // two expected intervals before its retained observation is stale. + threshold = max(threshold, 2*time.Duration(sighting.ExpectedUpdateIntervalSeconds)*time.Second) + } + return threshold +} + // metricSourceStale reports whether a source's most recent report is older than // its stale threshold. A zero/unknown last-seen is treated as NOT stale so the // merge never demotes a source on missing information. @@ -6300,17 +6556,7 @@ func metricSourceStale( if !ok || st.LastSeen.IsZero() { return false } - threshold := time.Duration(0) - if configured := thresholds[source]; configured > 0 { - threshold = configured - } - if threshold <= 0 { - threshold = defaultStaleThresholds[source] - } - if threshold <= 0 { - threshold = 60 * time.Second - } - return now.Sub(st.LastSeen) > threshold + return now.Sub(st.LastSeen) > sourceStaleThreshold(source, st, thresholds) } func mergeMetric( diff --git a/internal/unifiedresources/registry_merge_policy_test.go b/internal/unifiedresources/registry_merge_policy_test.go index e4d7080211..b48ad5d8cc 100644 --- a/internal/unifiedresources/registry_merge_policy_test.go +++ b/internal/unifiedresources/registry_merge_policy_test.go @@ -2,6 +2,7 @@ package unifiedresources import ( "encoding/json" + "reflect" "slices" "strconv" "strings" @@ -247,6 +248,114 @@ func TestLeaseExpiredAgentOnLiveProxmoxNodeStaysOnline(t *testing.T) { } } +// A source merge into an existing row judges metric freshness by the ingest's +// stale thresholds, as the stale pass that ends the ingest does. With PVE +// polled every two minutes the monitor allows Proxmox four minutes, so a node +// polled ninety seconds ago is current while its agent, silent for seventy-five +// seconds, is stale, and the row shows the node's readings. Without thresholds +// both sources are stale by the defaults and the agent keeps its precedence. +func TestSourceMergesJudgeFreshnessByTheIngestStaleThresholds(t *testing.T) { + now := time.Now().UTC() + snapshot := models.StateSnapshot{ + Nodes: []models.Node{{ + ID: "homelab-pve1", + Name: "pve1", + Instance: "homelab", + Status: "online", + LinkedAgentID: "host-pve1", + CPU: 0.12, + Memory: models.Memory{Total: 64 << 30, Used: 16 << 30, Free: 48 << 30, Usage: 25}, + LastSeen: now.Add(-90 * time.Second), + }}, + Hosts: []models.Host{{ + ID: "host-pve1", + MachineID: "machine-pve1", + Hostname: "pve1", + LinkedNodeID: "homelab-pve1", + Status: "online", + CPUUsage: 80, + Memory: models.Memory{Total: 64 << 30, Used: 48 << 30, Free: 16 << 30, Usage: 75}, + LastSeen: now.Add(-75 * time.Second), + IntervalSeconds: 30, + }}, + } + + for _, tc := range []struct { + name string + thresholds map[DataSource]time.Duration + proxmoxPoll string + want DataSource + wantCPU float64 + wantMemory float64 + }{ + {"configured", map[DataSource]time.Duration{SourceProxmox: 4 * time.Minute}, "online", SourceProxmox, 12, 25}, + {"defaults", nil, "stale", SourceAgent, 80, 75}, + } { + t.Run(tc.name, func(t *testing.T) { + rr := NewRegistry(nil) + rr.IngestSnapshotWithStaleThresholds(snapshot, tc.thresholds) + + resource := onlyResourceOfType(t, rr, ResourceTypeAgent) + if got := resource.SourceStatus[SourceProxmox].Status; got != tc.proxmoxPoll { + t.Fatalf("Proxmox sighting = %q, want %q: %+v", got, tc.proxmoxPoll, resource.SourceStatus) + } + if got := resource.SourceStatus[SourceAgent].Status; got != "stale" { + t.Fatalf("agent sighting = %q, want stale", got) + } + if resource.Metrics == nil || resource.Metrics.CPU == nil || resource.Metrics.Memory == nil { + t.Fatalf("merged row lost its readings: %+v", resource.Metrics) + } + if cpu := resource.Metrics.CPU; cpu.Source != tc.want || cpu.Percent != tc.wantCPU { + t.Fatalf("CPU = %.0f%% from %s, want %.0f%% from %s", cpu.Percent, cpu.Source, tc.wantCPU, tc.want) + } + if memory := resource.Metrics.Memory; memory.Source != tc.want || memory.Percent != tc.wantMemory { + t.Fatalf("memory = %.0f%% from %s, want %.0f%% from %s", memory.Percent, memory.Source, tc.wantMemory, tc.want) + } + }) + } +} + +// The metric merge and the stale pass read one threshold rule, so a source +// sighting the merge treats as current is never one the pass marks stale. +func TestMetricMergeFreshnessMatchesTheStalePass(t *testing.T) { + now := time.Now().UTC() + configured := map[DataSource]time.Duration{SourceProxmox: 4 * time.Minute, SourceVMware: 5 * time.Minute} + for _, tc := range []struct { + name string + source DataSource + sighting SourceStatus + thresholds map[DataSource]time.Duration + wantStale bool + }{ + {"proxmox by default", SourceProxmox, SourceStatus{LastSeen: now.Add(-90 * time.Second)}, nil, true}, + {"proxmox by configuration", SourceProxmox, SourceStatus{LastSeen: now.Add(-90 * time.Second)}, configured, false}, + {"vsphere by default", SourceVMware, SourceStatus{LastSeen: now.Add(-3 * time.Minute)}, nil, true}, + {"vsphere by configuration", SourceVMware, SourceStatus{LastSeen: now.Add(-3 * time.Minute)}, configured, false}, + {"agent without its own threshold", SourceAgent, SourceStatus{LastSeen: now.Add(-75 * time.Second)}, configured, true}, + {"slow disk inventory", SourceProxmox, SourceStatus{LastSeen: now.Add(-5 * time.Minute), ExpectedUpdateIntervalSeconds: 600}, nil, false}, + {"missed two disk inventory polls", SourceProxmox, SourceStatus{LastSeen: now.Add(-21 * time.Minute), ExpectedUpdateIntervalSeconds: 600}, nil, true}, + } { + t.Run(tc.name, func(t *testing.T) { + status := map[DataSource]SourceStatus{tc.source: tc.sighting} + if got := metricSourceStale(now, status, tc.source, tc.thresholds); got != tc.wantStale { + t.Fatalf("metric merge stale = %v, want %v", got, tc.wantStale) + } + + rr := NewRegistry(nil) + rr.resources["resource"] = &Resource{ + ID: "resource", + Status: StatusOnline, + Sources: []DataSource{tc.source}, + SourceStatus: map[DataSource]SourceStatus{tc.source: tc.sighting}, + } + rr.MarkStale(now, tc.thresholds) + if got := rr.resources["resource"].SourceStatus[tc.source].Status == "stale"; got != tc.wantStale { + t.Fatalf("stale pass stale = %v, want %v", got, tc.wantStale) + } + }) + } +} + // Docker hosts hold a shorter lease than the registry's Docker stale // threshold, so a silent Docker host was offline for a while and then // flipped to warning once its sighting went stale. @@ -441,6 +550,914 @@ func TestManualLinkKeepsTheLiveSightingOfASharedSource(t *testing.T) { } } +// Either side of an operator link can arrive as a supplemental record: a +// vSphere VM with a Pulse agent inside, or an agent on a TrueNAS host. The +// monitor's rebuild ingests records after the snapshot, so it must join them +// there too, keep the record side's provider payload, and list what the +// resources API lists after seeding from it. +func TestManualLinksJoinRecordIngestedResourcesInMonitorRebuild(t *testing.T) { + now := time.Now().UTC() + snapshot := models.StateSnapshot{ + Hosts: []models.Host{ + {ID: "host-app-guest", Hostname: "app-guest", MachineID: "machine-app-guest", Status: "online", LastSeen: now}, + {ID: "host-nas", Hostname: "nas-a-mgmt", MachineID: "machine-nas-a", Status: "online", LastSeen: now}, + }, + LastUpdate: now, + } + vmRecord := IngestRecord{ + SourceID: "vc-1:vm:vm-42", + Resource: Resource{ + Type: ResourceTypeVM, + Technology: "vmware", + Name: "app-guest", + Status: StatusOnline, + LastSeen: now, + VMware: &VMwareData{ConnectionID: "vc-1", ManagedObjectID: "vm-42", EntityType: "vm"}, + }, + Identity: ResourceIdentity{Hostnames: []string{"app-guest"}}, + } + records := map[DataSource][]IngestRecord{ + SourceVMware: {vmRecord}, + SourceTrueNAS: { + { + SourceID: "system:tn-1", + Resource: Resource{ + Type: ResourceTypeAgent, + Name: "nas-a", + Status: StatusOnline, + LastSeen: now, + TrueNAS: &TrueNASData{Hostname: "nas-a"}, + }, + Identity: ResourceIdentity{Hostnames: []string{"nas-a"}}, + }, + { + SourceID: "system:tn-1:pool:tank", + ParentSourceID: "system:tn-1", + Resource: Resource{ + Type: ResourceTypeStorage, + Name: "tank", + Status: StatusOnline, + LastSeen: now, + Storage: &StorageMeta{Type: "zfs-pool", Platform: "truenas", Topology: "pool"}, + }, + }, + }, + } + + for _, agentPrimary := range []bool{false, true} { + name := "record-primary" + if agentPrimary { + name = "agent-primary" + } + t.Run(name, func(t *testing.T) { + store := NewMemoryStore() + adapter := NewMonitorAdapter(NewRegistry(store)) + adapter.PopulateSnapshotAndSupplemental(snapshot, records) + + var vmID, guestAgentID, systemID, nasAgentID string + for _, resource := range adapter.GetAll() { + switch { + case resource.VMware != nil: + vmID = resource.ID + case resource.TrueNAS != nil: + systemID = resource.ID + case resource.Name == "app-guest": + guestAgentID = resource.ID + case resource.Name == "nas-a-mgmt": + nasAgentID = resource.ID + } + } + if vmID == "" || guestAgentID == "" || systemID == "" || nasAgentID == "" { + t.Fatalf("unlinked estate = %v, want the VM, both agents and the TrueNAS system", resourceIDs(adapter.GetAll())) + } + guestPrimary, nasPrimary := vmID, systemID + if agentPrimary { + guestPrimary, nasPrimary = guestAgentID, nasAgentID + } + for _, link := range []ResourceLink{ + {ResourceA: vmID, ResourceB: guestAgentID, PrimaryID: guestPrimary}, + {ResourceA: systemID, ResourceB: nasAgentID, PrimaryID: nasPrimary}, + } { + if err := store.AddLink(link); err != nil { + t.Fatalf("add link: %v", err) + } + } + + assertJoined := func(stage string) { + t.Helper() + resources := adapter.GetAll() + if len(resources) != 3 { + t.Fatalf("%s: resources = %v, want the linked VM, the linked TrueNAS system and its pool", stage, resourceIDs(resources)) + } + // An agent inside a guest supplements the guest, whichever + // side the operator chose as primary. + guest, ok := adapter.currentRegistry().Get(vmID) + if !ok { + t.Fatalf("%s: linked vSphere VM %s missing from %v", stage, vmID, resourceIDs(resources)) + } + if guest.Type != ResourceTypeVM || guest.VMware == nil || guest.Agent == nil { + t.Fatalf("%s: linked VM type=%s vmware=%t agent=%t, want the VM carrying its agent", stage, guest.Type, guest.VMware != nil, guest.Agent != nil) + } + if !slices.Contains(guest.Sources, SourceVMware) || !slices.Contains(guest.Sources, SourceAgent) { + t.Fatalf("%s: linked VM sources = %v, want vmware and agent", stage, guest.Sources) + } + system, ok := adapter.currentRegistry().Get(nasPrimary) + if !ok { + t.Fatalf("%s: linked TrueNAS system %s missing from %v", stage, nasPrimary, resourceIDs(resources)) + } + if system.TrueNAS == nil || system.Agent == nil { + t.Fatalf("%s: linked system truenas=%t agent=%t, want the primary carrying both payloads", stage, system.TrueNAS != nil, system.Agent != nil) + } + if !slices.Contains(system.Sources, SourceTrueNAS) || !slices.Contains(system.Sources, SourceAgent) { + t.Fatalf("%s: linked system sources = %v, want truenas and agent", stage, system.Sources) + } + pool := onlyResourceOfType(t, adapter.currentRegistry(), ResourceTypeStorage) + if pool.ParentID == nil || *pool.ParentID != nasPrimary { + t.Fatalf("%s: pool parent = %v, want the linked system %s", stage, pool.ParentID, nasPrimary) + } + + // Identity pins persisted by the rebuild must not re-key a link + // onto its own primary. + links, err := store.GetLinks() + if err != nil { + t.Fatalf("%s: get links: %v", stage, err) + } + for _, link := range links { + if link.ResourceA == link.ResourceB { + t.Fatalf("%s: link collapsed onto %s", stage, link.ResourceA) + } + } + + // The resources API seeds its registry from this listing + // through resource ingest and must agree with it. + rest := NewRegistry(store) + rest.IngestResources(resources) + if got, want := resourceIDs(rest.List()), resourceIDs(resources); !slices.Equal(got, want) { + t.Fatalf("%s: a registry seeded from the monitor lists %v, monitor lists %v", stage, got, want) + } + } + + next := snapshot + next.LastUpdate = now.Add(time.Second) + adapter.PopulateSnapshotAndSupplemental(next, records) + assertJoined("rebuild") + + adapter.PopulateSupplementalRecords(SourceVMware, records[SourceVMware]) + adapter.PopulateSupplementalRecords(SourceTrueNAS, records[SourceTrueNAS]) + assertJoined("supplemental refresh") + + next.LastUpdate = now.Add(2 * time.Second) + adapter.PopulateSnapshotAndSupplemental(next, records) + assertJoined("second rebuild") + }) + } +} + +// A Proxmox node that absorbs a linked agent carries the agent's machine key, +// so the rebuild pins the node with it. The agent is still observed: that pin +// must not succeed the agent's ID onto the node, which re-keyed the link onto +// the node itself and split the pair on the next rebuild. +func TestManualLinkKeepsItsEndpointsThroughIdentityPinPersistence(t *testing.T) { + now := time.Now().UTC() + snapshot := models.StateSnapshot{ + Nodes: []models.Node{{ + ID: "lab-pve1", Name: "pve1", Instance: "lab", + Host: "https://192.0.2.10:8006", Status: "online", LastSeen: now, + }}, + Hosts: []models.Host{{ + ID: "host-box", Hostname: "box-x", MachineID: "machine-box", Status: "online", LastSeen: now, + }}, + LastUpdate: now, + } + store := NewMemoryStore() + adapter := NewMonitorAdapter(NewRegistry(store)) + adapter.PopulateSnapshotAndSupplemental(snapshot, nil) + var nodeID, agentID string + for _, resource := range adapter.GetAll() { + if resource.Proxmox != nil { + nodeID = resource.ID + } else if resource.Agent != nil { + agentID = resource.ID + } + } + if nodeID == "" || agentID == "" || nodeID == agentID { + t.Fatalf("unlinked estate = %v, want a node and a separate agent", resourceIDs(adapter.GetAll())) + } + if err := store.AddLink(ResourceLink{ResourceA: nodeID, ResourceB: agentID, PrimaryID: nodeID}); err != nil { + t.Fatalf("add link: %v", err) + } + + for rebuild := 1; rebuild <= 3; rebuild++ { + next := snapshot + next.LastUpdate = now.Add(time.Duration(rebuild) * time.Second) + adapter.PopulateSnapshotAndSupplemental(next, nil) + if got := resourceIDs(adapter.GetAll()); !slices.Equal(got, []string{nodeID}) { + t.Fatalf("rebuild %d: resources = %v, want only the linked node %s", rebuild, got, nodeID) + } + links, err := store.GetLinks() + if err != nil { + t.Fatalf("get links: %v", err) + } + if len(links) != 1 || links[0].ResourceA != nodeID || links[0].ResourceB != agentID { + t.Fatalf("rebuild %d: links = %+v, want %s -> %s kept", rebuild, links, nodeID, agentID) + } + } +} + +// pinCountingStore counts identity pin writes, so tests can tell a +// steady-state rebuild (no writes) from one that rewrites a row every tick. +type pinCountingStore struct { + *MemoryStore + pinWrites int +} + +func (s *pinCountingStore) UpsertResourceIdentityPins(pins []ResourceIdentityPin) error { + s.pinWrites++ + return s.MemoryStore.UpsertResourceIdentityPins(pins) +} + +func (s *pinCountingStore) ReplaceResourceIdentityPins(pins []ResourceIdentityPin) error { + s.pinWrites++ + return s.MemoryStore.ReplaceResourceIdentityPins(pins) +} + +// manualLinkPinSnapshot is a Proxmox node and an unrelated host agent. With a +// cluster name the node is a cluster member whose instance names the cluster, +// so its identity carries the cluster. +func manualLinkPinSnapshot(now time.Time, clusterName string) models.StateSnapshot { + instance := "lab" + if clusterName != "" { + instance = clusterName + } + return models.StateSnapshot{ + Nodes: []models.Node{{ + ID: instance + "-pve1", Name: "pve1", Instance: instance, ClusterName: clusterName, IsClusterMember: clusterName != "", + Host: "https://pve1.example.test:8006", Status: "online", LastSeen: now, + }}, + Hosts: []models.Host{{ + ID: "host-box", Hostname: "box-x", MachineID: "machine-box", Status: "online", LastSeen: now, + }}, + LastUpdate: now, + } +} + +// A manual link joins two resources for display; it does not make them one +// machine, so each side keeps pinning its own identity. A Proxmox node whose +// configured endpoint is a qualified name pins that name. Pinned with the +// linked agent's machine key instead, the node's next rebuild completed that +// key, minted the agent's machine-derived ID and merged with the agent before +// the link applied, and removing the link could not separate them. +func TestManualLinkPinsEachSideFromItsOwnIdentity(t *testing.T) { + for _, tc := range []struct { + name string + clusterName string + nodePrimary bool + }{ + {name: "standalone node primary", nodePrimary: true}, + {name: "standalone agent primary"}, + {name: "cluster node primary", clusterName: "lab-cluster", nodePrimary: true}, + {name: "cluster agent primary", clusterName: "lab-cluster"}, + } { + t.Run(tc.name, func(t *testing.T) { + now := time.Now().UTC() + snapshot := manualLinkPinSnapshot(now, tc.clusterName) + store := &pinCountingStore{MemoryStore: NewMemoryStore()} + adapter := NewMonitorAdapter(NewRegistry(store)) + rebuild := func(step int) { + next := snapshot + next.LastUpdate = now.Add(time.Duration(step) * time.Second) + adapter.PopulateSnapshotAndSupplemental(next, nil) + } + rebuild(0) + var nodeID, agentID string + for _, resource := range adapter.GetAll() { + if resource.Proxmox != nil { + nodeID = resource.ID + } else if resource.Agent != nil { + agentID = resource.ID + } + } + if nodeID == "" || agentID != MachineIdentityCanonicalID(ResourceTypeAgent, "machine-box") || nodeID == agentID { + t.Fatalf("unlinked estate = %v, want a node and the machine-keyed agent", resourceIDs(adapter.GetAll())) + } + assertOwnPins := func(step string) { + t.Helper() + pins, err := store.ListResourceIdentityPins() + if err != nil { + t.Fatalf("%s: list identity pins: %v", step, err) + } + agentPinned, nodePinned := false, false + for _, pin := range pins { + switch pin.CanonicalID { + case agentID: + if pin.MachineID != "machine-box" || pin.Hostname != "box-x" || pin.ClusterName != "" { + t.Fatalf("%s: agent pin = %+v, want its own machine key and hostname", step, pin) + } + agentPinned = true + case nodeID: + if pin.MachineID != "" || pin.ClusterName != tc.clusterName || pin.Hostname != "pve1.example.test" { + t.Fatalf("%s: node pin = %+v, want only its own cluster and endpoint", step, pin) + } + nodePinned = true + default: + t.Fatalf("%s: unexpected pin %+v", step, pin) + } + } + if !agentPinned { + t.Fatalf("%s: pins = %+v, want the agent's own machine-keyed pin", step, pins) + } + if tc.clusterName != "" && !nodePinned { + t.Fatalf("%s: pins = %+v, want the cluster node's own pin", step, pins) + } + } + assertOwnPins("unlinked") + + primaryID, otherID := agentID, nodeID + if tc.nodePrimary { + primaryID, otherID = nodeID, agentID + } + if err := store.AddLink(ResourceLink{ResourceA: primaryID, ResourceB: otherID, PrimaryID: primaryID}); err != nil { + t.Fatalf("add link: %v", err) + } + for step := 1; step <= 3; step++ { + writesBefore := store.pinWrites + rebuild(step) + if got := resourceIDs(adapter.GetAll()); !slices.Equal(got, []string{primaryID}) { + t.Fatalf("rebuild %d: resources = %v, want only the linked primary %s", step, got, primaryID) + } + links, err := store.GetLinks() + if err != nil { + t.Fatalf("get links: %v", err) + } + if len(links) != 1 || links[0].ResourceA != primaryID || links[0].ResourceB != otherID { + t.Fatalf("rebuild %d: links = %+v, want %s -> %s kept", step, links, primaryID, otherID) + } + assertOwnPins("linked rebuild") + if step > 1 && store.pinWrites != writesBefore { + t.Fatalf("rebuild %d: wrote identity pins %d time(s), want none in steady state", step, store.pinWrites-writesBefore) + } + } + + // Removing the link separates the pair on the next rebuild: + // nothing the link lent either side may outlive it. + store.mu.Lock() + store.links = nil + store.mu.Unlock() + rebuild(4) + if got, want := resourceIDs(adapter.GetAll()), resourceIDs([]Resource{{ID: nodeID}, {ID: agentID}}); !slices.Equal(got, want) { + t.Fatalf("after unlink: resources = %v, want %v", got, want) + } + for _, resource := range adapter.GetAll() { + if resource.ID == nodeID && (resource.Agent != nil || resource.Identity.MachineID != "") { + t.Fatalf("after unlink: node = %+v, want no agent payload or borrowed machine key", resource) + } + } + assertOwnPins("after unlink") + }) + } +} + +// Earlier releases pinned a link's primary with the merged identity, so an +// agent primary's row can still hold the linked node's cluster. No resource +// holds that cluster with the agent's hostname, so the row keeps it, as an +// upsert keeps any stored field, and is not rewritten on every rebuild. +func TestManualLinkSideKeepsAnUnclaimedStoredField(t *testing.T) { + now := time.Now().UTC() + snapshot := manualLinkPinSnapshot(now, "lab-cluster") + snapshot.Nodes[0].Host = "https://192.0.2.10:8006" + agentID := MachineIdentityCanonicalID(ResourceTypeAgent, "machine-box") + store := &pinCountingStore{MemoryStore: NewMemoryStore()} + widened := ResourceIdentityPin{ + CanonicalID: agentID, ResourceType: ResourceTypeAgent, + MachineID: "machine-box", ClusterName: "lab-cluster", Hostname: "box-x", + } + if err := store.UpsertResourceIdentityPins([]ResourceIdentityPin{widened}); err != nil { + t.Fatalf("seed widened pin: %v", err) + } + unlinked := NewMonitorAdapter(NewRegistry(nil)) + unlinked.PopulateSnapshotAndSupplemental(snapshot, nil) + var nodeID string + for _, resource := range unlinked.GetAll() { + if resource.Proxmox != nil { + nodeID = resource.ID + } + } + if nodeID == "" || nodeID == agentID { + t.Fatalf("unlinked estate = %v, want a node beside agent %s", resourceIDs(unlinked.GetAll()), agentID) + } + if err := store.AddLink(ResourceLink{ResourceA: agentID, ResourceB: nodeID, PrimaryID: agentID}); err != nil { + t.Fatalf("add link: %v", err) + } + + adapter := NewMonitorAdapter(NewRegistry(store)) + for step := 0; step < 3; step++ { + writesBefore := store.pinWrites + next := snapshot + next.LastUpdate = now.Add(time.Duration(step) * time.Second) + adapter.PopulateSnapshotAndSupplemental(next, nil) + if got := resourceIDs(adapter.GetAll()); !slices.Equal(got, []string{agentID}) { + t.Fatalf("rebuild %d: resources = %v, want only the linked agent %s", step, got, agentID) + } + if step > 0 && store.pinWrites != writesBefore { + t.Fatalf("rebuild %d: wrote identity pins %d time(s), want none in steady state", step, store.pinWrites-writesBefore) + } + pins, err := store.ListResourceIdentityPins() + if err != nil { + t.Fatalf("list identity pins: %v", err) + } + for _, pin := range pins { + if pin.CanonicalID == agentID && pin != widened { + t.Fatalf("rebuild %d: agent pin = %+v, want the stored row kept %+v", step, pin, widened) + } + } + } +} + +// The store gives each strong key one owner. Two hosts can keep separate +// machine IDs yet share a pool name and hostname (an old and a replacement +// enrollment); linked, the live primary keeps that key and the folded side +// keeps only its machine key, rather than the two taking the pool from each +// other every rebuild. A key the primary's row only kept from an earlier +// release's merged pin goes to the folded side that reports it. +func TestManualLinkFoldedSideNeverTakesALiveResourcesKey(t *testing.T) { + newPin := ResourceIdentityPin{CanonicalID: "agent-new", ResourceType: ResourceTypeAgent, MachineID: "machine-new", ClusterName: "pool-a", Hostname: "xcp1"} + oldPin := ResourceIdentityPin{CanonicalID: "agent-old", ResourceType: ResourceTypeAgent, MachineID: "machine-old", ClusterName: "pool-a", Hostname: "xcp1"} + ownNewPin, ownOldPin := newPin, oldPin + ownNewPin.ClusterName, ownOldPin.ClusterName = "", "" + for _, tc := range []struct { + name string + primaryPool string + seed []ResourceIdentityPin + want []ResourceIdentityPin + }{ + {name: "both report the pool", primaryPool: "pool-a", want: []ResourceIdentityPin{newPin, ownOldPin}}, + {name: "primary row borrowed the pool", seed: []ResourceIdentityPin{newPin}, want: []ResourceIdentityPin{ownNewPin, oldPin}}, + } { + t.Run(tc.name, func(t *testing.T) { + now := time.Now().UTC() + host := func(id, machineID, pool string) Resource { + return Resource{ + ID: id, Type: ResourceTypeAgent, Name: "xcp1", Status: StatusOnline, LastSeen: now, + Sources: []DataSource{SourceAgent}, + Agent: &AgentData{AgentID: id, Hostname: "xcp1"}, + Identity: ResourceIdentity{MachineID: machineID, ClusterName: pool, Hostnames: []string{"xcp1"}}, + } + } + store := &pinCountingStore{MemoryStore: NewMemoryStore()} + if err := store.UpsertResourceIdentityPins(tc.seed); err != nil { + t.Fatalf("seed pins: %v", err) + } + if err := store.AddLink(ResourceLink{ResourceA: "agent-new", ResourceB: "agent-old", PrimaryID: "agent-new"}); err != nil { + t.Fatalf("add link: %v", err) + } + for rebuild := 1; rebuild <= 3; rebuild++ { + writesBefore := store.pinWrites + registry := NewRegistry(store) + registry.IngestResources([]Resource{host("agent-new", "machine-new", tc.primaryPool), host("agent-old", "machine-old", "pool-a")}) + registry.PersistIdentityPins() + if rebuild > 1 && store.pinWrites != writesBefore { + t.Fatalf("rebuild %d: wrote identity pins %d time(s), want none in steady state", rebuild, store.pinWrites-writesBefore) + } + pins, err := store.ListResourceIdentityPins() + if err != nil { + t.Fatalf("list identity pins: %v", err) + } + slices.SortFunc(pins, func(a, b ResourceIdentityPin) int { return strings.Compare(a.CanonicalID, b.CanonicalID) }) + if !slices.Equal(pins, tc.want) { + t.Fatalf("rebuild %d: pins = %+v, want %+v", rebuild, pins, tc.want) + } + } + }) + } +} + +// Two hosts can share a cloned machine ID while an exclusion keeps them +// apart. A folded side whose machine key the other host holds writes no pin +// rather than a cluster-only one: cluster-slot succession treats a +// machine-keyless row as a host a later machine in that slot may absorb. +func TestManualLinkFoldedSideWritesNoPinWithoutItsMachineKey(t *testing.T) { + now := time.Now().UTC() + host := func(id, machineID, pool, hostname string) Resource { + return Resource{ + ID: id, Type: ResourceTypeAgent, Name: hostname, Status: StatusOnline, LastSeen: now, + Sources: []DataSource{SourceAgent}, + Agent: &AgentData{AgentID: id, Hostname: hostname}, + Identity: ResourceIdentity{MachineID: machineID, ClusterName: pool, Hostnames: []string{hostname}}, + } + } + store := NewMemoryStore() + if err := store.AddLink(ResourceLink{ResourceA: "agent-primary", ResourceB: "agent-clone", PrimaryID: "agent-primary"}); err != nil { + t.Fatalf("add link: %v", err) + } + registry := NewRegistry(store) + registry.IngestResources([]Resource{ + host("agent-original", "machine-cloned", "", "swarm-a"), + host("agent-clone", "machine-cloned", "swarm", "swarm-b"), + host("agent-primary", "machine-primary", "", "primary"), + }) + registry.PersistIdentityPins() + pins, err := store.ListResourceIdentityPins() + if err != nil { + t.Fatalf("list identity pins: %v", err) + } + for _, pin := range pins { + if pin.CanonicalID == "agent-clone" { + t.Fatalf("folded clone pinned as %+v, want no pin without its machine key", pin) + } + } + if len(pins) != 2 { + t.Fatalf("pins = %+v, want the original's and the primary's", pins) + } +} + +// A link side's row keeps a stored hostname its own pin lacks, but never +// where that hostname would complete a cluster key another resource holds: +// the store would delete that resource's whole row. +func TestManualLinkSideKeepsNoHostnameThatCompletesAHeldClusterKey(t *testing.T) { + now := time.Now().UTC() + host := func(id, machineID, pool string, hostnames ...string) Resource { + return Resource{ + ID: id, Type: ResourceTypeAgent, Name: id, Status: StatusOnline, LastSeen: now, + Sources: []DataSource{SourceAgent}, + Agent: &AgentData{AgentID: id}, + Identity: ResourceIdentity{MachineID: machineID, ClusterName: pool, Hostnames: hostnames}, + } + } + store := &pinCountingStore{MemoryStore: NewMemoryStore()} + if err := store.UpsertResourceIdentityPins([]ResourceIdentityPin{{ + CanonicalID: "agent-a", ResourceType: ResourceTypeAgent, MachineID: "machine-a", ClusterName: "pool-d", Hostname: "h1", + }}); err != nil { + t.Fatalf("seed pin: %v", err) + } + if err := store.AddLink(ResourceLink{ResourceA: "agent-a", ResourceB: "agent-f", PrimaryID: "agent-a"}); err != nil { + t.Fatalf("add link: %v", err) + } + want := []ResourceIdentityPin{ + {CanonicalID: "agent-a", ResourceType: ResourceTypeAgent, MachineID: "machine-a", ClusterName: "pool-c"}, + {CanonicalID: "agent-b", ResourceType: ResourceTypeAgent, MachineID: "machine-b", ClusterName: "pool-c", Hostname: "h1"}, + {CanonicalID: "agent-f", ResourceType: ResourceTypeAgent, MachineID: "machine-f", Hostname: "f1"}, + } + for rebuild := 1; rebuild <= 3; rebuild++ { + writesBefore := store.pinWrites + registry := NewRegistry(store) + registry.IngestResources([]Resource{ + host("agent-a", "machine-a", "pool-c"), + host("agent-b", "machine-b", "pool-c", "h1"), + host("agent-f", "machine-f", "", "f1"), + }) + registry.PersistIdentityPins() + if rebuild > 1 && store.pinWrites != writesBefore { + t.Fatalf("rebuild %d: wrote identity pins %d time(s), want none in steady state", rebuild, store.pinWrites-writesBefore) + } + pins, err := store.ListResourceIdentityPins() + if err != nil { + t.Fatalf("list identity pins: %v", err) + } + slices.SortFunc(pins, func(a, b ResourceIdentityPin) int { return strings.Compare(a.CanonicalID, b.CanonicalID) }) + if !slices.Equal(pins, want) { + t.Fatalf("rebuild %d: pins = %+v, want %+v", rebuild, pins, want) + } + } +} + +// A folded side's own pin can be narrower than a row an earlier release +// widened with a pool its primary now reports. The primary's new pin claims +// that pool and hostname, so the store deletes the folded side's whole row; +// the folded side's own pin is written after it, so the side stays pinned. +func TestManualLinkFoldedSideKeepsItsPinWhenAPrimaryClaimsItsWiderRow(t *testing.T) { + now := time.Now().UTC() + host := func(id, machineID, pool string) Resource { + return Resource{ + ID: id, Type: ResourceTypeAgent, Name: "xcp1", Status: StatusOnline, LastSeen: now, + Sources: []DataSource{SourceAgent}, + Agent: &AgentData{AgentID: id, Hostname: "xcp1"}, + Identity: ResourceIdentity{MachineID: machineID, ClusterName: pool, Hostnames: []string{"xcp1"}}, + } + } + store := &pinCountingStore{MemoryStore: NewMemoryStore()} + if err := store.UpsertResourceIdentityPins([]ResourceIdentityPin{{ + CanonicalID: "agent-old", ResourceType: ResourceTypeAgent, + MachineID: "machine-old", ClusterName: "pool-a", Hostname: "xcp1", + }}); err != nil { + t.Fatalf("seed widened pin: %v", err) + } + if err := store.AddLink(ResourceLink{ResourceA: "agent-new", ResourceB: "agent-old", PrimaryID: "agent-new"}); err != nil { + t.Fatalf("add link: %v", err) + } + want := []ResourceIdentityPin{ + {CanonicalID: "agent-new", ResourceType: ResourceTypeAgent, MachineID: "machine-new", ClusterName: "pool-a", Hostname: "xcp1"}, + {CanonicalID: "agent-old", ResourceType: ResourceTypeAgent, MachineID: "machine-old", Hostname: "xcp1"}, + } + for rebuild := 1; rebuild <= 3; rebuild++ { + writesBefore := store.pinWrites + registry := NewRegistry(store) + registry.IngestResources([]Resource{host("agent-new", "machine-new", "pool-a"), host("agent-old", "machine-old", "")}) + registry.PersistIdentityPins() + if rebuild > 1 && store.pinWrites != writesBefore { + t.Fatalf("rebuild %d: wrote identity pins %d time(s), want none in steady state", rebuild, store.pinWrites-writesBefore) + } + pins, err := store.ListResourceIdentityPins() + if err != nil { + t.Fatalf("list identity pins: %v", err) + } + slices.SortFunc(pins, func(a, b ResourceIdentityPin) int { return strings.Compare(a.CanonicalID, b.CanonicalID) }) + if !slices.Equal(pins, want) { + t.Fatalf("rebuild %d: pins = %+v, want %+v", rebuild, pins, want) + } + } +} + +// An earlier release pinned a linked node primary with the agent's machine +// key and dropped the agent's own pin. The node has nothing of its own to +// pin, so the agent's own pin takes the key back on the next rebuild. +func TestManualLinkReturnsABorrowedMachineKeyToItsAgent(t *testing.T) { + now := time.Now().UTC() + snapshot := manualLinkPinSnapshot(now, "") + snapshot.Nodes[0].Host = "https://192.0.2.10:8006" + unlinked := NewMonitorAdapter(NewRegistry(nil)) + unlinked.PopulateSnapshotAndSupplemental(snapshot, nil) + var nodeID string + for _, resource := range unlinked.GetAll() { + if resource.Proxmox != nil { + nodeID = resource.ID + } + } + agentID := MachineIdentityCanonicalID(ResourceTypeAgent, "machine-box") + if nodeID == "" || nodeID == agentID { + t.Fatalf("unlinked estate = %v, want a node beside agent %s", resourceIDs(unlinked.GetAll()), agentID) + } + + store := &pinCountingStore{MemoryStore: NewMemoryStore()} + if err := store.UpsertResourceIdentityPins([]ResourceIdentityPin{{ + CanonicalID: nodeID, ResourceType: ResourceTypeAgent, MachineID: "machine-box", Hostname: "pve1", + }}); err != nil { + t.Fatalf("seed borrowed pin: %v", err) + } + if err := store.AddLink(ResourceLink{ResourceA: nodeID, ResourceB: agentID, PrimaryID: nodeID}); err != nil { + t.Fatalf("add link: %v", err) + } + adapter := NewMonitorAdapter(NewRegistry(store)) + for step := 0; step < 2; step++ { + next := snapshot + next.LastUpdate = now.Add(time.Duration(step) * time.Second) + adapter.PopulateSnapshotAndSupplemental(next, nil) + if got := resourceIDs(adapter.GetAll()); !slices.Equal(got, []string{nodeID}) { + t.Fatalf("rebuild %d: resources = %v, want only the linked node %s", step, got, nodeID) + } + pins, err := store.ListResourceIdentityPins() + if err != nil { + t.Fatalf("list identity pins: %v", err) + } + if len(pins) != 1 || pins[0].CanonicalID != agentID || pins[0].MachineID != "machine-box" || pins[0].Hostname != "box-x" { + t.Fatalf("rebuild %d: pins = %+v, want only the agent's own pin", step, pins) + } + } + links, err := store.GetLinks() + if err != nil { + t.Fatalf("get links: %v", err) + } + if len(links) != 1 || links[0].ResourceA != nodeID || links[0].ResourceB != agentID { + t.Fatalf("links = %+v, want %s -> %s kept", links, nodeID, agentID) + } +} + +// A record may name an older canonical ID as superseded on every ingest. When +// that ID belongs to a resource a manual link folded into the record's own +// resource, it is still observed: succeeding it on a later refresh would +// re-key the link onto its primary. +func TestRecordDeclaredSuccessionSparesALinkFoldedResource(t *testing.T) { + now := time.Now().UTC() + snapshot := models.StateSnapshot{ + Hosts: []models.Host{{ID: "host-nas", Hostname: "nas-a-mgmt", MachineID: "machine-nas-a", Status: "online", LastSeen: now}}, + LastUpdate: now, + } + agentID := MachineIdentityCanonicalID(ResourceTypeAgent, "machine-nas-a") + records := []IngestRecord{{ + SourceID: "system:tn-1", + SupersededCanonicalIDs: []string{agentID}, + Resource: Resource{ + Type: ResourceTypeAgent, + Name: "nas-a", + Status: StatusOnline, + LastSeen: now, + TrueNAS: &TrueNASData{Hostname: "nas-a"}, + }, + Identity: ResourceIdentity{Hostnames: []string{"nas-a"}}, + }} + + unlinked := NewMonitorAdapter(NewRegistry(nil)) + unlinked.PopulateSnapshotAndSupplemental(snapshot, map[DataSource][]IngestRecord{SourceTrueNAS: records}) + var systemID string + for _, resource := range unlinked.GetAll() { + if resource.TrueNAS != nil { + systemID = resource.ID + } + } + if systemID == "" || !slices.Contains(resourceIDs(unlinked.GetAll()), agentID) { + t.Fatalf("unlinked estate = %v, want agent %s and a TrueNAS system", resourceIDs(unlinked.GetAll()), agentID) + } + + store := NewMemoryStore() + if err := store.AddLink(ResourceLink{ResourceA: systemID, ResourceB: agentID, PrimaryID: systemID}); err != nil { + t.Fatalf("add link: %v", err) + } + adapter := NewMonitorAdapter(NewRegistry(store)) + adapter.PopulateSnapshotAndSupplemental(snapshot, map[DataSource][]IngestRecord{SourceTrueNAS: records}) + adapter.PopulateSupplementalRecords(SourceTrueNAS, records) + + if got := resourceIDs(adapter.GetAll()); !slices.Equal(got, []string{systemID}) { + t.Fatalf("resources = %v, want only the linked system %s", got, systemID) + } + links, err := store.GetLinks() + if err != nil { + t.Fatalf("get links: %v", err) + } + if len(links) != 1 || links[0].ResourceA != systemID || links[0].ResourceB != agentID { + t.Fatalf("links = %+v, want %s -> %s kept", links, systemID, agentID) + } +} + +// A link merge folds the other resource's provider payloads into the primary +// where the primary has none, whichever side the operator chose. +func TestManualLinkKeepsProviderPayloadsThePrimaryLacks(t *testing.T) { + now := time.Now().UTC() + store := NewMemoryStore() + for _, link := range []ResourceLink{ + {ResourceA: "agent-nas", ResourceB: "truenas-system", PrimaryID: "agent-nas"}, + {ResourceA: "agent-esxi", ResourceB: "vmware-host", PrimaryID: "agent-esxi"}, + } { + if err := store.AddLink(link); err != nil { + t.Fatalf("add link: %v", err) + } + } + host := func(id, name string, sources ...DataSource) Resource { + status := make(map[DataSource]SourceStatus, len(sources)) + for _, source := range sources { + status[source] = SourceStatus{Status: "online", LastSeen: now} + } + return Resource{ID: id, Type: ResourceTypeAgent, Name: name, Status: StatusOnline, LastSeen: now, Sources: sources, SourceStatus: status} + } + agentNAS := host("agent-nas", "nas-a-mgmt", SourceAgent) + agentNAS.Agent = &AgentData{AgentID: "agent-nas", Hostname: "nas-a-mgmt"} + system := host("truenas-system", "nas-a", SourceTrueNAS) + system.TrueNAS = &TrueNASData{Hostname: "nas-a"} + agentESXi := host("agent-esxi", "esxi-01", SourceAgent) + agentESXi.Agent = &AgentData{AgentID: "agent-esxi", Hostname: "esxi-01"} + esxi := host("vmware-host", "esxi-01.lab", SourceVMware) + esxi.VMware = &VMwareData{ConnectionID: "vc-1", ManagedObjectID: "host-101", EntityType: "host"} + + rr := NewRegistry(store) + rr.IngestResources([]Resource{agentNAS, system, agentESXi, esxi}) + + if got := resourceIDs(rr.List()); !slices.Equal(got, []string{"agent-esxi", "agent-nas"}) { + t.Fatalf("resources = %v, want the two linked primaries", got) + } + if nas, _ := rr.Get("agent-nas"); nas.TrueNAS == nil || nas.Agent == nil { + t.Fatalf("linked TrueNAS system truenas=%t agent=%t, want both payloads", nas.TrueNAS != nil, nas.Agent != nil) + } + if host, _ := rr.Get("agent-esxi"); host.VMware == nil || host.Agent == nil { + t.Fatalf("linked vSphere host vmware=%t agent=%t, want both payloads", host.VMware != nil, host.Agent != nil) + } +} + +// Each operator link folded into a resource is recorded with the link row's +// own pair, also along a chain (a Docker host linked to an agent, the agent +// linked into a VM), and the record rides the monitor's listing into a +// registry seeded from it, which never holds the folded rows. +func TestManualLinkFoldsNameEachLinkPairAlongAChain(t *testing.T) { + now := time.Now().UTC() + store, err := NewSQLiteResourceStore(t.TempDir(), "default") + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { _ = store.Close() }) + snapshot := models.StateSnapshot{ + LastUpdate: now, + VMs: []models.VM{{ID: "lab:pve1:101", VMID: 101, Name: "web", Node: "pve1", Instance: "lab", Status: "running", Type: "qemu", LastSeen: now}}, + Hosts: []models.Host{{ID: "host-box", Hostname: "box-agent", MachineID: "fedcba9876543210", Status: "online", LastSeen: now}}, + DockerHosts: []models.DockerHost{{ID: "docker-box", Hostname: "dock-box", Status: "online", LastSeen: now}}, + } + adapter := NewMonitorAdapter(NewRegistry(store)) + adapter.PopulateFromSnapshot(snapshot) + var vmID, agentID, dockerID string + for _, resource := range adapter.GetAll() { + switch { + case resource.Type == ResourceTypeVM: + vmID = resource.ID + case resource.Agent != nil: + agentID = resource.ID + case resource.Docker != nil: + dockerID = resource.ID + } + if len(resource.linkFolds) != 0 { + t.Fatalf("unlinked resource %s carries link folds %+v", resource.ID, resource.linkFolds) + } + } + if vmID == "" || agentID == "" || dockerID == "" || len(adapter.GetAll()) != 3 { + t.Fatalf("fixture did not produce a separate VM, agent and Docker host: %+v", adapter.GetAll()) + } + + // Links apply in store order, so the agent takes in the Docker host before + // the VM takes in the agent. + if err := store.AddLink(ResourceLink{ResourceA: agentID, ResourceB: dockerID, PrimaryID: agentID, CreatedAt: now}); err != nil { + t.Fatal(err) + } + if err := store.AddLink(ResourceLink{ResourceA: agentID, ResourceB: vmID, PrimaryID: agentID, CreatedAt: now.Add(time.Second)}); err != nil { + t.Fatal(err) + } + adapter.PopulateFromSnapshot(snapshot) + merged := adapter.GetAll() + if len(merged) != 1 || merged[0].ID != vmID { + t.Fatalf("links did not fold the chain into the VM: %+v", merged) + } + + // The agent linked from its own side still folds into the guest it runs + // in, and the fold names the link's pair either way. + want := []ManualLinkFold{ + {HolderID: agentID, FoldedID: dockerID, Sources: []DataSource{SourceDocker}}, + {HolderID: vmID, FoldedID: agentID, Sources: []DataSource{SourceAgent, SourceDocker}}, + } + if got := adapter.currentRegistry().ManualLinkFolds(vmID); !reflect.DeepEqual(got, want) { + t.Fatalf("monitor folds = %+v, want %+v", got, want) + } + + seeded := NewRegistry(store) + seeded.IngestResources(adapter.GetAll()) + if got := seeded.ManualLinkFolds(vmID); !reflect.DeepEqual(got, want) { + t.Fatalf("seeded registry folds = %+v, want %+v", got, want) + } + if got := seeded.ManualLinkFolds(agentID); got != nil { + t.Fatalf("folded agent ID reports folds %+v", got) + } +} + +// Record ingest recreates a folded side whose source mapping points at a +// holder of another type (an agent-type TrueNAS record linked into a VM) and +// the record-ingest epilogue folds it again, so the pair must be recorded +// once however often the records arrive. +func TestManualLinkFoldsRecordEachPairOnceAcrossRecordIngests(t *testing.T) { + now := time.Now().UTC() + store, err := NewSQLiteResourceStore(t.TempDir(), "default") + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { _ = store.Close() }) + snapshot := models.StateSnapshot{ + LastUpdate: now, + VMs: []models.VM{{ID: "lab:pve1:101", VMID: 101, Name: "web", Node: "pve1", Instance: "lab", Status: "running", Type: "qemu", LastSeen: now}}, + } + records := []IngestRecord{{ + SourceID: "nas-1", + Resource: Resource{Type: ResourceTypeAgent, Name: "nas-1", Status: StatusOnline, LastSeen: now}, + Identity: ResourceIdentity{MachineID: "abcdef0123456789"}, + }} + probe := NewRegistry(store) + probe.IngestSnapshot(snapshot) + probe.IngestRecords(SourceTrueNAS, records) + var vmID, systemID string + for _, resource := range probe.List() { + switch resource.Type { + case ResourceTypeVM: + vmID = resource.ID + case ResourceTypeAgent: + systemID = resource.ID + } + } + if vmID == "" || systemID == "" { + t.Fatalf("fixture did not produce a separate VM and TrueNAS system: %+v", probe.List()) + } + if err := store.AddLink(ResourceLink{ResourceA: vmID, ResourceB: systemID, PrimaryID: vmID, CreatedAt: now}); err != nil { + t.Fatal(err) + } + + rr := NewRegistry(store) + rr.IngestSnapshot(snapshot) + for i := 0; i < 3; i++ { + rr.IngestRecords(SourceTrueNAS, records) + } + if listed := rr.List(); len(listed) != 1 { + t.Fatalf("link did not fold the TrueNAS system into the VM: %+v", listed) + } + want := []ManualLinkFold{{HolderID: vmID, FoldedID: systemID, Sources: []DataSource{SourceTrueNAS}}} + if got := rr.ManualLinkFolds(vmID); !reflect.DeepEqual(got, want) { + t.Fatalf("folds after repeated record ingests = %+v, want %+v", got, want) + } +} + +// A folded side recreated with fewer sources than it brought before (one of +// an identity-merged agent's records replayed) folds again; the holder still +// carries everything the earlier fold brought, so the fold keeps every +// source and a report naming any of them still selects the link. +func TestManualLinkFoldRepeatedPairKeepsEarlierSources(t *testing.T) { + holder := &Resource{ID: "vm-1"} + recordManualLinkFold(holder, "vm-1", &Resource{ID: "agent-1", Sources: []DataSource{SourceAgent, SourceDocker}}, "agent-1") + recordManualLinkFold(holder, "vm-1", &Resource{ID: "agent-1", Sources: []DataSource{SourceAgent}}, "agent-1") + want := []ManualLinkFold{{HolderID: "vm-1", FoldedID: "agent-1", Sources: []DataSource{SourceAgent, SourceDocker}}} + if !reflect.DeepEqual(holder.linkFolds, want) { + t.Fatalf("folds = %+v, want %+v", holder.linkFolds, want) + } +} + func TestLeaseExpiredKubernetesClusterStaysOffline(t *testing.T) { rr := NewRegistry(nil) rr.IngestSnapshot(models.StateSnapshot{ diff --git a/internal/unifiedresources/resolve_test.go b/internal/unifiedresources/resolve_test.go index d45bbc7e3a..ee236ddcc2 100644 --- a/internal/unifiedresources/resolve_test.go +++ b/internal/unifiedresources/resolve_test.go @@ -455,3 +455,270 @@ func TestResolveResourceContext_NotFound(t *testing.T) { t.Fatalf("expected no resolved resource for missing lookup, got %#v", resolved.Resource) } } + +// linkedGuestEstate is an agent inside a vSphere VM, joined by an operator +// link, as Pulse sees it after a restart before the agent reports again: the +// published registry has only the VM, and saved-host continuity fills in the +// agent. +type linkedGuestEstate struct { + saved models.Host + vmRecords []IngestRecord + agentID string + vmID string +} + +func newLinkedGuestEstate(t *testing.T) linkedGuestEstate { + t.Helper() + now := time.Now().UTC() + estate := linkedGuestEstate{ + saved: models.Host{ + ID: "host-app-guest", MachineID: "machine-app-guest", Hostname: "app-guest", + Platform: "linux", Status: "offline", LastSeen: now.Add(-10 * time.Minute), + Memory: models.Memory{Total: 8 << 30, Used: 7 << 30, Free: 1 << 30, Usage: 88}, + }, + vmRecords: []IngestRecord{{ + SourceID: "vc-1:vm:vm-42", + Resource: Resource{ + Type: ResourceTypeVM, + Technology: "vmware", + Name: "app-guest", + Status: StatusOnline, + LastSeen: now, + Metrics: &ResourceMetrics{Memory: &MetricValue{Percent: 40, Source: SourceVMware}}, + VMware: &VMwareData{ConnectionID: "vc-1", ManagedObjectID: "vm-42", EntityType: "vm"}, + }, + Identity: ResourceIdentity{Hostnames: []string{"app-guest"}}, + }}, + } + unlinked := NewMonitorAdapter(NewRegistry(nil)) + unlinked.PopulateSnapshotAndSupplemental( + models.StateSnapshot{Hosts: []models.Host{estate.saved}, LastUpdate: now}, + map[DataSource][]IngestRecord{SourceVMware: estate.vmRecords}, + ) + if len(unlinked.VMs()) != 1 || len(unlinked.Hosts()) != 1 { + t.Fatalf("unlinked estate = %d VMs, %d agents, want one of each", len(unlinked.VMs()), len(unlinked.Hosts())) + } + estate.vmID, estate.agentID = unlinked.VMs()[0].ID(), unlinked.Hosts()[0].ID() + return estate +} + +// readStateAfterRestart publishes the VM alone and overlays the saved agent, +// as Monitor.GetUnifiedReadStateOrSnapshot does. +func (e linkedGuestEstate) readStateAfterRestart(store ResourceStore) *MonitorAdapter { + published := NewMonitorAdapter(NewRegistry(store)) + published.PopulateSnapshotAndSupplemental( + models.StateSnapshot{LastUpdate: time.Now().UTC()}, + map[DataSource][]IngestRecord{SourceVMware: e.vmRecords}, + ) + return ReadStateWithHostContinuity(published, []IngestRecord{HostIngestRecord(e.saved)}).(*MonitorAdapter) +} + +// An operator link names one identity, so every reference to the saved +// agent resolves to the VM, as it does once the agent reports and folds in. +// The agent's saved telemetry stays on its own row in the read state and in +// the resources API, which seeds a registry from that listing and re-applies +// the links. +func TestContinuityAgentLinkedIntoLiveGuestAnswersToTheGuest(t *testing.T) { + estate := newLinkedGuestEstate(t) + store := NewMemoryStore() + if err := store.AddLink(ResourceLink{ResourceA: estate.vmID, ResourceB: estate.agentID, PrimaryID: estate.agentID}); err != nil { + t.Fatalf("add link: %v", err) + } + readState := estate.readStateAfterRestart(store) + resources := NewRegistry(store) + resources.IngestResources(readState.GetAll()) + + for _, registry := range []struct { + name string + registry *ResourceRegistry + }{ + {"read state", readState.registry}, + {"resources API", resources}, + } { + t.Run(registry.name, func(t *testing.T) { + rr := registry.registry + agent, ok := rr.Get(estate.agentID) + if !ok || agent.Status != StatusOffline { + t.Fatalf("saved agent row = %+v (listed=%v), want its own offline row", agent, ok) + } + vm, ok := rr.Get(estate.vmID) + if !ok { + t.Fatalf("linked VM %s missing", estate.vmID) + } + if vm.Agent != nil || vm.Status != StatusOnline || vm.Metrics == nil || vm.Metrics.Memory == nil || vm.Metrics.Memory.Source != SourceVMware { + t.Fatalf("VM agent=%v status=%s metrics=%+v, want vSphere's online VM without the saved agent's payload", vm.Agent != nil, vm.Status, vm.Metrics) + } + // The hostname both rows carry stays unambiguous: the saved row shares + // the VM's identity. + for _, ref := range []string{estate.agentID, "agent:" + estate.saved.ID, estate.saved.MachineID, "app-guest"} { + if resolved, ok := rr.ResolveReferenceID(ref); !ok || resolved != estate.vmID { + t.Fatalf("ResolveReferenceID(%q) = %q (%v), want the link primary %s", ref, resolved, ok, estate.vmID) + } + if _, resolved, ok := rr.GetByReference(ref); !ok || resolved != estate.vmID { + t.Fatalf("GetByReference(%q) = %q (%v), want the link primary %s", ref, resolved, ok, estate.vmID) + } + } + if resolved, ok := rr.ResolveReferenceID(estate.vmID); !ok || resolved != estate.vmID { + t.Fatalf("VM reference resolved to %q (%v)", resolved, ok) + } + }) + } + if resolved, ok := readState.ResolveCanonicalResourceID("agent:" + estate.saved.ID); !ok || resolved != estate.vmID { + t.Fatalf("read-state canonical resolution = %q (%v), want %s", resolved, ok, estate.vmID) + } + + // Once the agent reports, the link folds it in as before. + live := estate.saved + live.Status, live.LastSeen = "online", time.Now().UTC() + published := NewMonitorAdapter(NewRegistry(store)) + published.PopulateSnapshotAndSupplemental( + models.StateSnapshot{Hosts: []models.Host{live}, LastUpdate: time.Now().UTC()}, + map[DataSource][]IngestRecord{SourceVMware: estate.vmRecords}, + ) + if len(published.Hosts()) != 0 { + t.Fatal("reporting agent still listed beside its linked VM") + } + if resolved, ok := published.ResolveCanonicalResourceID(estate.agentID); !ok || resolved != estate.vmID { + t.Fatalf("folded agent resolved to %q (%v), want %s", resolved, ok, estate.vmID) + } +} + +// A saved primary never captures a live member: the live row keeps its own +// references and telemetry, and the saved row its own, until the primary +// reports and the link folds them. +func TestContinuityPrimaryLeavesLiveLinkMemberItsOwnIdentity(t *testing.T) { + now := time.Now().UTC() + live := models.Host{ID: "host-live", MachineID: "machine-live", Hostname: "live", Platform: "linux", Status: "online", LastSeen: now, CPUUsage: 23} + saved := models.Host{ID: "host-saved", MachineID: "machine-saved", Hostname: "saved", Platform: "linux", Status: "offline", LastSeen: now.Add(-time.Hour)} + + ids := NewMonitorAdapter(NewRegistry(nil)) + ids.PopulateFromSnapshot(models.StateSnapshot{Hosts: []models.Host{live, saved}, LastUpdate: now}) + if len(ids.Hosts()) != 2 { + t.Fatalf("unlinked estate has %d agents, want 2", len(ids.Hosts())) + } + agentIDs := map[string]string{} + for _, host := range ids.Hosts() { + agentIDs[host.AgentID()] = host.ID() + } + liveID, savedID := agentIDs[live.ID], agentIDs[saved.ID] + + store := NewMemoryStore() + if err := store.AddLink(ResourceLink{ResourceA: savedID, ResourceB: liveID, PrimaryID: savedID}); err != nil { + t.Fatalf("add link: %v", err) + } + published := NewMonitorAdapter(NewRegistry(store)) + published.PopulateFromSnapshot(models.StateSnapshot{Hosts: []models.Host{live}, LastUpdate: now}) + readState := ReadStateWithHostContinuity(published, []IngestRecord{HostIngestRecord(saved)}).(*MonitorAdapter) + resources := NewRegistry(store) + resources.IngestResources(readState.GetAll()) + + for name, rr := range map[string]*ResourceRegistry{"read state": readState.registry, "resources API": resources} { + liveRow, ok := rr.Get(liveID) + if !ok || liveRow.Status != StatusOnline || len(liveRow.Sources) != 1 { + t.Fatalf("%s: live member = %+v (listed=%v), want its own online row", name, liveRow, ok) + } + if _, ok := rr.Get(savedID); !ok { + t.Fatalf("%s: saved primary row missing", name) + } + for ref, want := range map[string]string{liveID: liveID, "agent:" + live.ID: liveID, savedID: savedID, "agent:" + saved.ID: savedID} { + if resolved, ok := rr.ResolveReferenceID(ref); !ok || resolved != want { + t.Fatalf("%s: %q resolved to %q (%v), want %s", name, ref, resolved, ok, want) + } + } + } +} + +// An observation that merges into a saved row makes it a live row again, so +// the link folds it: the resources API's record replay is one such path. +func TestObservationMergedIntoASavedRowLetsItsLinkFold(t *testing.T) { + estate := newLinkedGuestEstate(t) + store := NewMemoryStore() + if err := store.AddLink(ResourceLink{ResourceA: estate.vmID, ResourceB: estate.agentID, PrimaryID: estate.vmID}); err != nil { + t.Fatalf("add link: %v", err) + } + rr := estate.readStateAfterRestart(store).registry + if _, held := rr.Get(estate.agentID); !held { + t.Fatal("saved agent not listed before it reports") + } + live := estate.saved + live.Status, live.LastSeen = "online", time.Now().UTC() + rr.IngestRecords(SourceAgent, []IngestRecord{HostIngestRecord(live)}) + if _, listed := rr.Get(estate.agentID); listed { + t.Fatal("reported agent still listed beside its linked VM") + } + if vm, ok := rr.Get(estate.vmID); !ok || vm.Agent == nil { + t.Fatalf("linked VM = %+v (listed=%v), want it carrying the reported agent", vm, ok) + } +} + +// Holds follow the links one pass applies, whatever their order: a primary a +// later link folds away hands its held members to the row holding it, a saved +// member held under a saved primary answers to that primary's own target, and +// a cycle of saved members resolves each to itself. +func TestLinkHoldsFollowTheirPrimaryThroughThePass(t *testing.T) { + saved := func(id string) Resource { + return Resource{ID: id, Type: ResourceTypeAgent, Name: id, Status: StatusOffline, Sources: []DataSource{SourceAgent}, continuityOnly: true} + } + vm := func(id string) Resource { + return Resource{ID: id, Type: ResourceTypeVM, Name: id, Status: StatusOnline, Sources: []DataSource{SourceVMware}} + } + for _, tc := range []struct { + name string + resources []Resource + links []ResourceLink + want map[string]string + }{ + { + name: "primary folded later", + resources: []Resource{saved("agent-a"), vm("vm-v"), vm("vm-w")}, + links: []ResourceLink{ + {ResourceA: "agent-a", ResourceB: "vm-v", PrimaryID: "vm-v"}, + {ResourceA: "vm-v", ResourceB: "vm-w", PrimaryID: "vm-w"}, + }, + want: map[string]string{"agent-a": "vm-w", "vm-v": "vm-w", "vm-w": "vm-w"}, + }, + { + name: "saved primary held in turn", + resources: []Resource{saved("agent-x"), saved("agent-a"), vm("vm-v")}, + links: []ResourceLink{ + {ResourceA: "agent-x", ResourceB: "agent-a", PrimaryID: "agent-a"}, + {ResourceA: "agent-a", ResourceB: "vm-v", PrimaryID: "vm-v"}, + }, + want: map[string]string{"agent-x": "vm-v", "agent-a": "vm-v", "vm-v": "vm-v"}, + }, + { + name: "cycle", + resources: []Resource{saved("agent-a"), saved("agent-b"), saved("agent-c")}, + links: []ResourceLink{ + {ResourceA: "agent-a", ResourceB: "agent-b", PrimaryID: "agent-a"}, + {ResourceA: "agent-b", ResourceB: "agent-c", PrimaryID: "agent-b"}, + {ResourceA: "agent-c", ResourceB: "agent-a", PrimaryID: "agent-c"}, + }, + want: map[string]string{"agent-a": "agent-a", "agent-b": "agent-b", "agent-c": "agent-c"}, + }, + } { + t.Run(tc.name, func(t *testing.T) { + store := NewMemoryStore() + for _, link := range tc.links { + if err := store.AddLink(link); err != nil { + t.Fatalf("add link: %v", err) + } + } + rr := NewRegistry(store) + rr.IngestResources(tc.resources) + for ref, want := range tc.want { + if resolved, ok := rr.ResolveReferenceID(ref); !ok || resolved != want { + t.Fatalf("%q resolved to %q (%v), want %s", ref, resolved, ok, want) + } + } + for _, resource := range tc.resources { + if !resource.continuityOnly { + continue + } + if row, listed := rr.Get(resource.ID); !listed || row.Status != StatusOffline { + t.Fatalf("saved row %s = %+v (listed=%v), want it kept", resource.ID, row, listed) + } + } + }) + } +} diff --git a/internal/unifiedresources/store.go b/internal/unifiedresources/store.go index 06b03e9332..8f5d7fa85c 100644 --- a/internal/unifiedresources/store.go +++ b/internal/unifiedresources/store.go @@ -11,6 +11,7 @@ import ( "os" "path/filepath" "regexp" + "slices" "sort" "strings" "sync" @@ -29,6 +30,10 @@ type ResourceStore interface { // Identity pins keep canonical IDs for merged-source hosts stable across // restarts. See ResourceIdentityPin. UpsertResourceIdentityPins(pins []ResourceIdentityPin) error + // ReplaceResourceIdentityPins writes each pin as its row's whole + // content, clearing fields the pin leaves empty, where an upsert keeps + // them. Strong-key conflicts resolve as for an upsert. + ReplaceResourceIdentityPins(pins []ResourceIdentityPin) error ListResourceIdentityPins() ([]ResourceIdentityPin, error) RecordChange(change ResourceChange) error ResourceHistoryIDs(resourceID string) ([]string, error) @@ -1194,7 +1199,29 @@ func (s *SQLiteResourceStore) AddLink(link ResourceLink) error { a, b := normalizePair(link.ResourceA, link.ResourceB) s.mu.Lock() defer s.mu.Unlock() - _, err := s.db.Exec(` + tx, err := s.db.Begin() + if err != nil { + return fmt.Errorf("begin resource link %q<->%q: %w", a, b, err) + } + committed := false + defer func() { + if !committed { + _ = tx.Rollback() + } + }() + // A link replaces the pair's earlier decision (manual_link_decisions.go): + // its exclusion, and a link canonical-ID succession stored in the other + // order, which the upsert below would not replace. + if _, err := tx.Exec(`DELETE FROM resource_exclusions + WHERE (resource_a = ? AND resource_b = ?) OR (resource_a = ? AND resource_b = ?)`, a, b, b, a); err != nil { + return fmt.Errorf("clear resource exclusion %q<->%q replaced by link: %w", a, b, err) + } + if a != b { + if _, err := tx.Exec(`DELETE FROM resource_links WHERE resource_a = ? AND resource_b = ?`, b, a); err != nil { + return fmt.Errorf("clear reversed resource link %q<->%q: %w", b, a, err) + } + } + if _, err := tx.Exec(` INSERT INTO resource_links (resource_a, resource_b, primary_id, reason, created_by, created_at) VALUES (?, ?, ?, ?, ?, ?) ON CONFLICT(resource_a, resource_b) DO UPDATE SET @@ -1202,10 +1229,13 @@ func (s *SQLiteResourceStore) AddLink(link ResourceLink) error { reason=excluded.reason, created_by=excluded.created_by, created_at=excluded.created_at - `, a, b, link.PrimaryID, link.Reason, link.CreatedBy, link.CreatedAt) - if err != nil { + `, a, b, link.PrimaryID, link.Reason, link.CreatedBy, link.CreatedAt); err != nil { return fmt.Errorf("upsert resource link %q<->%q: %w", a, b, err) } + if err := tx.Commit(); err != nil { + return fmt.Errorf("commit resource link %q<->%q: %w", a, b, err) + } + committed = true return nil } @@ -1221,21 +1251,76 @@ func (s *SQLiteResourceStore) AddExclusion(exclusion ResourceExclusion) error { a, b := normalizePair(exclusion.ResourceA, exclusion.ResourceB) s.mu.Lock() defer s.mu.Unlock() - _, err := s.db.Exec(` + tx, err := s.db.Begin() + if err != nil { + return fmt.Errorf("begin resource exclusion %q<->%q: %w", a, b, err) + } + committed := false + defer func() { + if !committed { + _ = tx.Rollback() + } + }() + // An exclusion replaces the pair's earlier decision, so unlinking a linked + // pair splits it (manual_link_decisions.go): its links in either order, + // and an exclusion canonical-ID succession stored in the other order, + // which the upsert below would not replace. + if _, err := tx.Exec(`DELETE FROM resource_links + WHERE (resource_a = ? AND resource_b = ?) OR (resource_a = ? AND resource_b = ?)`, a, b, b, a); err != nil { + return fmt.Errorf("clear resource link %q<->%q replaced by exclusion: %w", a, b, err) + } + if a != b { + if _, err := tx.Exec(`DELETE FROM resource_exclusions WHERE resource_a = ? AND resource_b = ?`, b, a); err != nil { + return fmt.Errorf("clear reversed resource exclusion %q<->%q: %w", b, a, err) + } + } + if _, err := tx.Exec(` INSERT INTO resource_exclusions (resource_a, resource_b, reason, created_by, created_at) VALUES (?, ?, ?, ?, ?) ON CONFLICT(resource_a, resource_b) DO UPDATE SET reason=excluded.reason, created_by=excluded.created_by, created_at=excluded.created_at - `, a, b, exclusion.Reason, exclusion.CreatedBy, exclusion.CreatedAt) - if err != nil { + `, a, b, exclusion.Reason, exclusion.CreatedBy, exclusion.CreatedAt); err != nil { return fmt.Errorf("upsert resource exclusion %q<->%q: %w", a, b, err) } + if err := tx.Commit(); err != nil { + return fmt.Errorf("commit resource exclusion %q<->%q: %w", a, b, err) + } + committed = true return nil } -func (s *SQLiteResourceStore) GetLinks() (links []ResourceLink, err error) { +// GetLinks returns the links that are their pair's latest operator decision. +func (s *SQLiteResourceStore) GetLinks() ([]ResourceLink, error) { + links, err := s.storedLinks() + if err != nil || len(links) == 0 { + return links, err + } + exclusions, err := s.storedExclusions() + if err != nil { + return nil, err + } + links, _ = effectiveManualPairDecisions(links, exclusions) + return links, nil +} + +// GetExclusions returns the exclusions that are their pair's latest operator +// decision. +func (s *SQLiteResourceStore) GetExclusions() ([]ResourceExclusion, error) { + exclusions, err := s.storedExclusions() + if err != nil || len(exclusions) == 0 { + return exclusions, err + } + links, err := s.storedLinks() + if err != nil { + return nil, err + } + _, exclusions = effectiveManualPairDecisions(links, exclusions) + return exclusions, nil +} + +func (s *SQLiteResourceStore) storedLinks() (links []ResourceLink, err error) { rows, err := s.db.Query(`SELECT resource_a, resource_b, primary_id, reason, created_by, created_at FROM resource_links`) if err != nil { return nil, fmt.Errorf("query resource links: %w", err) @@ -1267,7 +1352,7 @@ func (s *SQLiteResourceStore) GetLinks() (links []ResourceLink, err error) { return links, nil } -func (s *SQLiteResourceStore) GetExclusions() (exclusions []ResourceExclusion, err error) { +func (s *SQLiteResourceStore) storedExclusions() (exclusions []ResourceExclusion, err error) { rows, err := s.db.Query(`SELECT resource_a, resource_b, reason, created_by, created_at FROM resource_exclusions`) if err != nil { return nil, fmt.Errorf("query resource exclusions: %w", err) @@ -1521,9 +1606,39 @@ func (s *SQLiteResourceStore) Close() error { } func (s *SQLiteResourceStore) UpsertResourceIdentityPins(pins []ResourceIdentityPin) error { + return s.writeResourceIdentityPins(pins, false) +} + +func (s *SQLiteResourceStore) ReplaceResourceIdentityPins(pins []ResourceIdentityPin) error { + return s.writeResourceIdentityPins(pins, true) +} + +// writeResourceIdentityPins upserts pins. An upsert keeps a stored field the +// pin leaves empty; a replace writes the pin as the whole row. +func (s *SQLiteResourceStore) writeResourceIdentityPins(pins []ResourceIdentityPin, replace bool) error { if len(pins) == 0 { return nil } + write := `INSERT INTO resource_identities (canonical_id, resource_type, machine_id, dmi_uuid, cluster_name, primary_hostname) + VALUES (?, ?, ?, ?, ?, ?) + ON CONFLICT(canonical_id) DO UPDATE SET + resource_type = excluded.resource_type, + machine_id = COALESCE(excluded.machine_id, resource_identities.machine_id), + dmi_uuid = COALESCE(excluded.dmi_uuid, resource_identities.dmi_uuid), + cluster_name = COALESCE(NULLIF(excluded.cluster_name, ''), resource_identities.cluster_name), + primary_hostname = COALESCE(NULLIF(excluded.primary_hostname, ''), resource_identities.primary_hostname), + updated_at = CURRENT_TIMESTAMP` + if replace { + write = `INSERT INTO resource_identities (canonical_id, resource_type, machine_id, dmi_uuid, cluster_name, primary_hostname) + VALUES (?, ?, ?, ?, ?, ?) + ON CONFLICT(canonical_id) DO UPDATE SET + resource_type = excluded.resource_type, + machine_id = excluded.machine_id, + dmi_uuid = excluded.dmi_uuid, + cluster_name = excluded.cluster_name, + primary_hostname = excluded.primary_hostname, + updated_at = CURRENT_TIMESTAMP` + } s.mu.Lock() defer s.mu.Unlock() @@ -1560,15 +1675,7 @@ func (s *SQLiteResourceStore) UpsertResourceIdentityPins(pins []ResourceIdentity ); err != nil { return fmt.Errorf("clear conflicting identity pins for %q: %w", pin.CanonicalID, err) } - if _, err := tx.Exec(`INSERT INTO resource_identities (canonical_id, resource_type, machine_id, dmi_uuid, cluster_name, primary_hostname) - VALUES (?, ?, ?, ?, ?, ?) - ON CONFLICT(canonical_id) DO UPDATE SET - resource_type = excluded.resource_type, - machine_id = COALESCE(excluded.machine_id, resource_identities.machine_id), - dmi_uuid = COALESCE(excluded.dmi_uuid, resource_identities.dmi_uuid), - cluster_name = COALESCE(NULLIF(excluded.cluster_name, ''), resource_identities.cluster_name), - primary_hostname = COALESCE(NULLIF(excluded.primary_hostname, ''), resource_identities.primary_hostname), - updated_at = CURRENT_TIMESTAMP`, + if _, err := tx.Exec(write, pin.CanonicalID, string(pin.ResourceType), nullIfEmptyArg(pin.MachineID), @@ -3413,6 +3520,14 @@ func NewMemoryStore() *MemoryStore { } func (m *MemoryStore) UpsertResourceIdentityPins(pins []ResourceIdentityPin) error { + return m.writeResourceIdentityPins(pins, false) +} + +func (m *MemoryStore) ReplaceResourceIdentityPins(pins []ResourceIdentityPin) error { + return m.writeResourceIdentityPins(pins, true) +} + +func (m *MemoryStore) writeResourceIdentityPins(pins []ResourceIdentityPin, replace bool) error { m.mu.Lock() defer m.mu.Unlock() for _, pin := range pins { @@ -3432,7 +3547,7 @@ func (m *MemoryStore) UpsertResourceIdentityPins(pins []ResourceIdentityPin) err delete(m.identityPins, canonicalID) } } - if existing, ok := m.identityPins[pin.CanonicalID]; ok { + if existing, ok := m.identityPins[pin.CanonicalID]; ok && !replace { if pin.MachineID == "" { pin.MachineID = existing.MachineID } @@ -3483,6 +3598,17 @@ func (m *MemoryStore) AddLink(link ResourceLink) error { link.ResourceA = CanonicalResourceID(link.ResourceA) link.ResourceB = CanonicalResourceID(link.ResourceB) link.PrimaryID = CanonicalResourceID(link.PrimaryID) + if link.CreatedAt.IsZero() { + link.CreatedAt = time.Now().UTC() + } + // A link replaces the pair's earlier decision, either way round + // (manual_link_decisions.go). + m.exclusions = slices.DeleteFunc(m.exclusions, func(exclusion ResourceExclusion) bool { + return sameManualPair(exclusion.ResourceA, exclusion.ResourceB, link.ResourceA, link.ResourceB) + }) + m.links = slices.DeleteFunc(m.links, func(existing ResourceLink) bool { + return sameManualPair(existing.ResourceA, existing.ResourceB, link.ResourceA, link.ResourceB) + }) m.links = append(m.links, link) return nil } @@ -3492,6 +3618,17 @@ func (m *MemoryStore) AddExclusion(exclusion ResourceExclusion) error { defer m.mu.Unlock() exclusion.ResourceA = CanonicalResourceID(exclusion.ResourceA) exclusion.ResourceB = CanonicalResourceID(exclusion.ResourceB) + if exclusion.CreatedAt.IsZero() { + exclusion.CreatedAt = time.Now().UTC() + } + // An exclusion replaces the pair's earlier decision, so unlinking a + // linked pair splits it (manual_link_decisions.go). + m.links = slices.DeleteFunc(m.links, func(link ResourceLink) bool { + return sameManualPair(link.ResourceA, link.ResourceB, exclusion.ResourceA, exclusion.ResourceB) + }) + m.exclusions = slices.DeleteFunc(m.exclusions, func(existing ResourceExclusion) bool { + return sameManualPair(existing.ResourceA, existing.ResourceB, exclusion.ResourceA, exclusion.ResourceB) + }) m.exclusions = append(m.exclusions, exclusion) return nil } @@ -3501,6 +3638,7 @@ func (m *MemoryStore) GetLinks() ([]ResourceLink, error) { defer m.mu.RUnlock() out := make([]ResourceLink, len(m.links)) copy(out, m.links) + out, _ = effectiveManualPairDecisions(out, m.exclusions) return out, nil } @@ -3509,6 +3647,7 @@ func (m *MemoryStore) GetExclusions() ([]ResourceExclusion, error) { defer m.mu.RUnlock() out := make([]ResourceExclusion, len(m.exclusions)) copy(out, m.exclusions) + _, out = effectiveManualPairDecisions(m.links, out) return out, nil } diff --git a/internal/unifiedresources/store_test.go b/internal/unifiedresources/store_test.go index 74649f8c83..9641df9ff6 100644 --- a/internal/unifiedresources/store_test.go +++ b/internal/unifiedresources/store_test.go @@ -644,6 +644,248 @@ func TestNewSQLiteResourceStore_MigratesLegacyStore(t *testing.T) { } } +func manualDecisionStores(t *testing.T) map[string]ResourceStore { + t.Helper() + sqlite, err := NewSQLiteResourceStore(t.TempDir(), "default") + if err != nil { + t.Fatalf("NewSQLiteResourceStore: %v", err) + } + t.Cleanup(func() { _ = sqlite.Close() }) + return map[string]ResourceStore{"sqlite": sqlite, "memory": NewMemoryStore()} +} + +// manualDecisionRows returns the rows a store holds before read resolution, +// so the tests see what a write replaced rather than what a read hides. +func manualDecisionRows(t *testing.T, store ResourceStore) ([]ResourceLink, []ResourceExclusion) { + t.Helper() + switch s := store.(type) { + case *SQLiteResourceStore: + links, err := s.storedLinks() + if err != nil { + t.Fatalf("storedLinks: %v", err) + } + exclusions, err := s.storedExclusions() + if err != nil { + t.Fatalf("storedExclusions: %v", err) + } + return links, exclusions + case *MemoryStore: + s.mu.RLock() + defer s.mu.RUnlock() + return append([]ResourceLink(nil), s.links...), append([]ResourceExclusion(nil), s.exclusions...) + } + t.Fatalf("unexpected store %T", store) + return nil, nil +} + +// seedReversedManualRows stores a link and an exclusion for a pair in the +// order canonical-ID succession can leave, which writes never produce. +func seedReversedManualRows(t *testing.T, store ResourceStore, a, b string, at time.Time) { + t.Helper() + switch s := store.(type) { + case *SQLiteResourceStore: + if _, err := s.db.Exec(`INSERT INTO resource_links (resource_a, resource_b, primary_id, reason, created_by, created_at) VALUES (?, ?, ?, '', '', ?)`, a, b, a, at); err != nil { + t.Fatal(err) + } + if _, err := s.db.Exec(`INSERT INTO resource_exclusions (resource_a, resource_b, reason, created_by, created_at) VALUES (?, ?, '', '', ?)`, a, b, at); err != nil { + t.Fatal(err) + } + case *MemoryStore: + s.mu.Lock() + s.links = append(s.links, ResourceLink{ResourceA: a, ResourceB: b, PrimaryID: a, CreatedAt: at}) + s.exclusions = append(s.exclusions, ResourceExclusion{ResourceA: a, ResourceB: b, CreatedAt: at}) + s.mu.Unlock() + default: + t.Fatalf("unexpected store %T", store) + } +} + +// Unlink records an exclusion; it must also delete the link, or every +// registry keeps folding the pair. Link must delete an earlier exclusion the +// same way. The IDs arrive in either order, as the API takes either side as +// the path, and a row succession stored in the other order is replaced too. +func TestManualPairDecisionReplacesThePreviousOne(t *testing.T) { + for name, store := range manualDecisionStores(t) { + t.Run(name, func(t *testing.T) { + at := time.Date(2026, 10, 7, 21, 0, 0, 0, time.UTC) + if err := store.AddLink(ResourceLink{ResourceA: "vm-web", ResourceB: "agent-web", PrimaryID: "vm-web", CreatedAt: at}); err != nil { + t.Fatal(err) + } + if err := store.AddLink(ResourceLink{ResourceA: "vm-db", ResourceB: "agent-db", PrimaryID: "vm-db", CreatedAt: at}); err != nil { + t.Fatal(err) + } + if links, exclusions := manualDecisionRows(t, store); len(links) != 2 || len(exclusions) != 0 { + t.Fatalf("after links: links=%+v exclusions=%+v", links, exclusions) + } + + seedReversedManualRows(t, store, "vm-web", "agent-web", at.Add(-time.Hour)) + if err := store.AddExclusion(ResourceExclusion{ResourceA: "agent-web", ResourceB: "vm-web", CreatedAt: at.Add(time.Minute)}); err != nil { + t.Fatal(err) + } + links, exclusions := manualDecisionRows(t, store) + if len(links) != 1 || !sameManualPair(links[0].ResourceA, links[0].ResourceB, "vm-db", "agent-db") { + t.Fatalf("unlink kept the wrong link rows: %+v", links) + } + if len(exclusions) != 1 || !exclusions[0].CreatedAt.Equal(at.Add(time.Minute)) { + t.Fatalf("unlink left exclusion rows %+v, want only its own", exclusions) + } + + seedReversedManualRows(t, store, "vm-web", "agent-web", at.Add(-time.Hour)) + if err := store.AddLink(ResourceLink{ResourceA: "agent-web", ResourceB: "vm-web", PrimaryID: "vm-web", CreatedAt: at.Add(2 * time.Minute)}); err != nil { + t.Fatal(err) + } + links, exclusions = manualDecisionRows(t, store) + if len(links) != 2 || len(exclusions) != 0 { + t.Fatalf("relink: links=%+v exclusions=%+v, want 2 links and no exclusions", links, exclusions) + } + + // Linking a linked pair again replaces its link, primary included. + if err := store.AddLink(ResourceLink{ResourceA: "agent-db", ResourceB: "vm-db", PrimaryID: "agent-db", CreatedAt: at.Add(3 * time.Minute)}); err != nil { + t.Fatal(err) + } + links, _ = manualDecisionRows(t, store) + primaries := map[string]string{} + for _, link := range links { + primaries[exclusionKey(link.ResourceA, link.ResourceB)] = link.PrimaryID + } + if len(links) != 2 || primaries[exclusionKey("vm-db", "agent-db")] != "agent-db" { + t.Fatalf("relinking kept stale rows: %+v", links) + } + + // A decision written without a time is stamped, as SQLite does, + // so it can be ordered against later ones. + if err := store.AddExclusion(ResourceExclusion{ResourceA: "vm-db", ResourceB: "agent-db"}); err != nil { + t.Fatal(err) + } + if _, exclusions = manualDecisionRows(t, store); len(exclusions) != 1 || exclusions[0].CreatedAt.IsZero() { + t.Fatalf("unstamped exclusion: %+v", exclusions) + } + }) + } +} + +// Stores written before unlink dropped the link still hold both rows for a +// pair, and canonical-ID succession can rewrite two pairs onto one in either +// order. Reads decide such a pair by its later write; a tie keeps it apart. +func TestManualPairDecisionReadsResolveStoredConflictsByRecency(t *testing.T) { + sqlite, err := NewSQLiteResourceStore(t.TempDir(), "default") + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { _ = sqlite.Close() }) + at := time.Date(2026, 9, 1, 12, 0, 0, 0, time.UTC) + rows := []struct { + table string + a, b string + createdAt time.Time + withPrimID bool + }{ + // Unlinked after linking: the pair stays apart. + {"resource_links", "agent-a", "vm-a", at, true}, + {"resource_exclusions", "agent-a", "vm-a", at.Add(time.Minute), false}, + // Linked again after an unlink: the pair is linked. + {"resource_exclusions", "agent-b", "vm-b", at, false}, + {"resource_links", "agent-b", "vm-b", at.Add(time.Minute), true}, + // Same instant: apart. + {"resource_links", "agent-c", "vm-c", at, true}, + {"resource_exclusions", "agent-c", "vm-c", at, false}, + // Succession stored the link in reverse order. + {"resource_links", "vm-d", "agent-d", at, true}, + {"resource_exclusions", "agent-d", "vm-d", at.Add(time.Minute), false}, + // Undecided elsewhere: untouched. + {"resource_links", "agent-e", "vm-e", at, true}, + {"resource_exclusions", "agent-f", "vm-f", at, false}, + // Succession left an older link in the other order: the newer + // link, and its primary, decide. + {"resource_links", "vm-g", "agent-g", at.Add(time.Minute), true}, + {"resource_links", "agent-g", "vm-g", at, true}, + } + for _, row := range rows { + var err error + if row.withPrimID { + _, err = sqlite.db.Exec(`INSERT INTO resource_links (resource_a, resource_b, primary_id, reason, created_by, created_at) VALUES (?, ?, ?, '', '', ?)`, row.a, row.b, row.b, row.createdAt) + } else { + _, err = sqlite.db.Exec(`INSERT INTO resource_exclusions (resource_a, resource_b, reason, created_by, created_at) VALUES (?, ?, '', '', ?)`, row.a, row.b, row.createdAt) + } + if err != nil { + t.Fatalf("seed %s %s<->%s: %v", row.table, row.a, row.b, err) + } + } + + links, err := sqlite.GetLinks() + if err != nil { + t.Fatal(err) + } + exclusions, err := sqlite.GetExclusions() + if err != nil { + t.Fatal(err) + } + linked := map[string]bool{} + for _, link := range links { + linked[exclusionKey(link.ResourceA, link.ResourceB)] = true + } + excluded := map[string]bool{} + for _, exclusion := range exclusions { + excluded[exclusionKey(exclusion.ResourceA, exclusion.ResourceB)] = true + } + for _, want := range []struct { + pair string + linked, excluded bool + }{ + {"a", false, true}, + {"b", true, false}, + {"c", false, true}, + {"d", false, true}, + {"e", true, false}, + {"f", false, true}, + {"g", true, false}, + } { + key := exclusionKey("agent-"+want.pair, "vm-"+want.pair) + if linked[key] != want.linked || excluded[key] != want.excluded { + t.Errorf("pair %s: linked=%v excluded=%v, want %v/%v", want.pair, linked[key], excluded[key], want.linked, want.excluded) + } + } + if len(links) != 3 || len(exclusions) != 4 { + t.Fatalf("links=%+v exclusions=%+v", links, exclusions) + } + for _, link := range links { + if sameManualPair(link.ResourceA, link.ResourceB, "agent-g", "vm-g") && link.PrimaryID != "agent-g" { + t.Fatalf("older reversed link won: %+v", link) + } + } +} + +// tornDecisionStore answers GetLinks from before an unlink and GetExclusions +// from after it, as two reads around a concurrent unlink do. +type tornDecisionStore struct { + *MemoryStore + staleLinks []ResourceLink +} + +func (s tornDecisionStore) GetLinks() ([]ResourceLink, error) { return s.staleLinks, nil } + +// A registry must not load both decisions for a pair when an unlink lands +// between its two reads: applyManualLinks does not consult exclusions, so the +// generation would stay merged. +func TestRegistryResolvesDecisionsAcrossItsTwoReads(t *testing.T) { + at := time.Date(2026, 10, 7, 21, 0, 0, 0, time.UTC) + store := tornDecisionStore{ + MemoryStore: NewMemoryStore(), + staleLinks: []ResourceLink{{ResourceA: "vm-web", ResourceB: "agent-web", PrimaryID: "vm-web", CreatedAt: at}}, + } + if err := store.AddExclusion(ResourceExclusion{ResourceA: "vm-web", ResourceB: "agent-web", CreatedAt: at.Add(time.Second)}); err != nil { + t.Fatal(err) + } + registry := NewRegistry(store) + registry.IngestResources([]Resource{ + {ID: "vm-web", Type: ResourceTypeVM, Name: "web", Status: StatusOnline, LastSeen: at, Sources: []DataSource{SourceProxmox}}, + {ID: "agent-web", Type: ResourceTypeAgent, Name: "web-agent", Status: StatusOnline, LastSeen: at, Sources: []DataSource{SourceAgent}, Agent: &AgentData{AgentID: "agent-web"}}, + }) + if got := len(registry.List()); got != 2 { + t.Fatalf("registry folded an unlinked pair: %d resources", got) + } +} + func TestNewSQLiteResourceStore_MigratesLegacyResourceChangesTable(t *testing.T) { dataDir := t.TempDir() legacyPath := filepath.Join(dataDir, "resources", resourceDBFileName) @@ -2864,6 +3106,66 @@ func TestSQLiteResourceStore_ResourceOperatorState_RoundTrips(t *testing.T) { } } +// An upsert keeps a stored field the pin leaves empty; a replace writes the +// pin as the whole row. Both give a strong key to the pin that carries it. +func TestResourceIdentityPinReplaceClearsFieldsUpsertKeeps(t *testing.T) { + sqliteStore, err := NewSQLiteResourceStore(t.TempDir(), "") + if err != nil { + t.Fatalf("open sqlite store: %v", err) + } + defer sqliteStore.Close() + + wide := ResourceIdentityPin{CanonicalID: "agent-a", ResourceType: ResourceTypeAgent, MachineID: "machine-a", DMIUUID: "dmi-a", ClusterName: "pool-a", Hostname: "host-a"} + narrow := ResourceIdentityPin{CanonicalID: "agent-a", ResourceType: ResourceTypeAgent, MachineID: "machine-a", Hostname: "host-a.lan"} + rival := ResourceIdentityPin{CanonicalID: "agent-b", ResourceType: ResourceTypeAgent, MachineID: "machine-b", ClusterName: "pool-a", Hostname: "host-a"} + for name, store := range map[string]ResourceStore{"memory": NewMemoryStore(), "sqlite": sqliteStore} { + t.Run(name, func(t *testing.T) { + pinFor := func(id string) (ResourceIdentityPin, bool) { + t.Helper() + pins, err := store.ListResourceIdentityPins() + if err != nil { + t.Fatalf("list pins: %v", err) + } + for _, pin := range pins { + if pin.CanonicalID == id { + return pin, true + } + } + return ResourceIdentityPin{}, false + } + if err := store.UpsertResourceIdentityPins([]ResourceIdentityPin{wide}); err != nil { + t.Fatalf("upsert wide pin: %v", err) + } + if err := store.UpsertResourceIdentityPins([]ResourceIdentityPin{narrow}); err != nil { + t.Fatalf("upsert narrow pin: %v", err) + } + kept := wide + kept.Hostname = narrow.Hostname + if got, _ := pinFor("agent-a"); got != kept { + t.Fatalf("after upsert: pin = %+v, want stored fields kept %+v", got, kept) + } + if err := store.ReplaceResourceIdentityPins([]ResourceIdentityPin{narrow}); err != nil { + t.Fatalf("replace narrow pin: %v", err) + } + if got, _ := pinFor("agent-a"); got != narrow { + t.Fatalf("after replace: pin = %+v, want exactly %+v", got, narrow) + } + if err := store.ReplaceResourceIdentityPins([]ResourceIdentityPin{wide}); err != nil { + t.Fatalf("replace wide pin: %v", err) + } + if err := store.ReplaceResourceIdentityPins([]ResourceIdentityPin{rival}); err != nil { + t.Fatalf("replace rival pin: %v", err) + } + if _, ok := pinFor("agent-a"); ok { + t.Fatalf("a replace claiming pool-a/host-a must delete the row holding that key") + } + if got, _ := pinFor("agent-b"); got != rival { + t.Fatalf("rival pin = %+v, want %+v", got, rival) + } + }) + } +} + func TestResourceChangeReadsMergeCanonicalIDEras(t *testing.T) { const machineID = "7d465a78-test-machine-id" steadyID := buildHashID(ResourceTypeAgent, "machine:"+machineID) diff --git a/internal/unifiedresources/types.go b/internal/unifiedresources/types.go index 1e54e94b23..6a049d8374 100644 --- a/internal/unifiedresources/types.go +++ b/internal/unifiedresources/types.go @@ -57,6 +57,16 @@ type Resource struct { ParentName string `json:"parentName,omitempty"` ChildCount int `json:"childCount,omitempty"` parentBySource map[DataSource]string + // linkFolds lists the operator links folded into this resource + // (recordManualLinkFold); report-merge excludes their pairs. + linkFolds []ManualLinkFold + + // continuityOnly marks a row saved-host continuity introduced: a machine + // no source has reported since a restart. It rides in-memory clones, so + // a registry seeded from a continuity read state still holds the row out + // of operator-link folds (holdLinkedResourceLocked). An observation that + // later merges into the row clears it. + continuityOnly bool Tags []string `json:"tags,omitempty"` CustomURL string `json:"customUrl,omitempty"` diff --git a/pkg/reporting/catalog.go b/pkg/reporting/catalog.go index 3820d821c2..fcf287cc09 100644 --- a/pkg/reporting/catalog.go +++ b/pkg/reporting/catalog.go @@ -2,6 +2,7 @@ package reporting import ( "fmt" + "net/http" "strings" "time" ) @@ -33,6 +34,7 @@ type PerformanceReportDefinition struct { Title string `json:"title"` Description string `json:"description"` SingleResourceEndpoint string `json:"singleResourceEndpoint"` + SingleResourceMethod string `json:"singleResourceMethod"` MultiResourceEndpoint string `json:"multiResourceEndpoint"` SingleFilenamePrefix string `json:"singleFilenamePrefix"` SingleFilenameSubject string `json:"singleFilenameSubject"` @@ -150,6 +152,7 @@ func DescribePerformanceReport() PerformanceReportDefinition { Title: "Performance Reports", Description: "Generate PDF summaries or CSV metric exports from historical monitoring data for one or more selected resources.", SingleResourceEndpoint: "/api/admin/reports/generate", + SingleResourceMethod: http.MethodPost, MultiResourceEndpoint: "/api/admin/reports/generate-multi", SingleFilenamePrefix: "report", SingleFilenameSubject: FilenameSubjectResourceID, diff --git a/scripts/check-patrol-assistant-journey.mjs b/scripts/check-patrol-assistant-journey.mjs index df417be86e..0f63f78c33 100644 --- a/scripts/check-patrol-assistant-journey.mjs +++ b/scripts/check-patrol-assistant-journey.mjs @@ -311,7 +311,7 @@ try { "Disk usage is 95 percent.", ); expect(requests.at(-1).handoff_resources[0].id).toBe(resourceId); - expect(requests.at(-1).autonomous_mode).toBe(false); + expect(requests.at(-1)).not.toHaveProperty("autonomous_mode"); await expect( page.getByText("No changes were made.", { exact: false }), ).toBeVisible(); @@ -386,7 +386,7 @@ try { "Source: Pulse Alerts active alert", ); expect(requests.at(-1).handoff_resources.length).toBeGreaterThan(0); - expect(requests.at(-1).autonomous_mode).toBe(false); + expect(requests.at(-1)).not.toHaveProperty("autonomous_mode"); await expect( page.getByText("No changes were made.", { exact: false }), ).toBeVisible(); @@ -476,7 +476,7 @@ try { expect(requests.at(-1).handoff_context).toContain(finding.id); expect(requests.at(-1).handoff_context).toContain(uncertainConclusion); expect(requests.at(-1).handoff_context).toContain(failedRead); - expect(requests.at(-1).autonomous_mode).toBe(false); + expect(requests.at(-1)).not.toHaveProperty("autonomous_mode"); await expect( page .getByTestId("assistant-message-list") diff --git a/tests/integration/e2e-tiering.mjs b/tests/integration/e2e-tiering.mjs index 1ea7e10de8..0e5d4fd37e 100644 --- a/tests/integration/e2e-tiering.mjs +++ b/tests/integration/e2e-tiering.mjs @@ -10,7 +10,6 @@ export const QUARANTINED_SPECS = [ // spec turns that open bug into a permanent red. '**/03-multi-tenant.spec.ts', '**/47-inline-selection-scroll-stability.spec.ts', - '**/48-summary-hover-selection.spec.ts', ]; /** diff --git a/tests/integration/tests/04-mobile.spec.ts b/tests/integration/tests/04-mobile.spec.ts index 431ff6294b..6e1dccce53 100644 --- a/tests/integration/tests/04-mobile.spec.ts +++ b/tests/integration/tests/04-mobile.spec.ts @@ -1,5 +1,9 @@ import { test, expect, devices } from "@playwright/test"; -import { ensureAuthenticated, setMockMode } from "./helpers"; +import { + ensureAuthenticated, + setMockMode, + waitForDefaultMockRuntimeReady, +} from "./helpers"; const getViewportWidth = async ( page: import("@playwright/test").Page, @@ -397,6 +401,131 @@ test.describe("Mobile viewport flows", () => { ); }); + test("Drawer identity chips wrap inside the drawer content instead of being clipped", async ({ + page, + }) => { + await waitForDefaultMockRuntimeReady(page); + await page.goto("/docker"); + + // Container drawers carry long generated aliases such as + // app-container- that are wider than the identity value cell. + const containerRow = page.locator("tr[data-docker-container-row]").first(); + if ( + ["1", "true", "yes", "on"].includes( + String(process.env.PULSE_E2E_REQUIRE_DEFAULT_MOCK_READY || "") + .trim() + .toLowerCase(), + ) + ) { + // CI runs the default mock estate, which always has Docker containers. + await expect(containerRow).toBeVisible({ timeout: 15_000 }); + } else { + const containerRowVisible = await containerRow + .waitFor({ state: "visible", timeout: 15_000 }) + .then( + () => true, + () => false, + ); + if (!containerRowVisible) { + test.skip(true, "No Docker container rows available to expand"); + } + } + // On phones the whole row is the disclosure target. + await containerRow.locator("td").first().click(); + await expect( + containerRow.getByRole("button", { name: /^Collapse details for / }), + ).toHaveAttribute("aria-expanded", "true"); + + const identity = page + .locator('[data-testid="resource-identity-section"]') + .first(); + await expect(identity).toBeVisible(); + await expect(identity.getByText("Aliases")).toBeVisible(); + + const cutOff = await identity.evaluate((section) => { + // Below lg the inline drawer's content shell clips anything past it. + let shell: HTMLElement | null = section.parentElement; + while (shell && window.getComputedStyle(shell).overflowX !== "clip") { + shell = shell.parentElement; + } + if (!shell) return ["no clipping drawer content shell"]; + + const offenders: string[] = []; + const walker = document.createTreeWalker( + section, + NodeFilter.SHOW_ELEMENT | NodeFilter.SHOW_TEXT, + ); + for (let node = walker.nextNode(); node; node = walker.nextNode()) { + let rect: DOMRect; + let inlineLevel: boolean; + if (node.nodeType === Node.TEXT_NODE) { + if (!node.textContent?.trim()) continue; + const range = document.createRange(); + range.selectNodeContents(node); + rect = range.getBoundingClientRect(); + inlineLevel = true; + } else { + rect = (node as Element).getBoundingClientRect(); + inlineLevel = window + .getComputedStyle(node as Element) + .display.startsWith("inline"); + } + const left = rect.left; + let right = rect.right; + // Inline content that truncates with an ellipsis is meant to end + // inside the block whose line it sits on (a truncating value span or + // label cell). The fixed-layout detail cell also declares an ellipsis, + // but that only marks its own line: a block child such as the chip + // row crosses the cell edge with no ellipsis and is silently cut. + if (inlineLevel) { + let line = node.parentElement; + while ( + line && + line !== shell && + ["inline", "contents"].includes( + window.getComputedStyle(line).display, + ) + ) { + line = line.parentElement; + } + const lineStyle = line ? window.getComputedStyle(line) : null; + if ( + line && + lineStyle?.textOverflow === "ellipsis" && + lineStyle.overflowX !== "visible" && + ["block", "inline-block", "table-cell", "list-item", "flow-root"].includes( + lineStyle.display, + ) + ) { + right = Math.min(right, line.getBoundingClientRect().right); + } + } + // Any clipping box on the way up (the detail cell, the table wrapper, + // the drawer shell) hides whatever still crosses either of its edges. + for ( + let box: HTMLElement | null = node.parentElement; + box; + box = box === shell ? null : box.parentElement + ) { + const style = window.getComputedStyle(box); + if (style.overflowX === "visible" || style.display === "contents") { + continue; + } + const edges = box.getBoundingClientRect(); + const past = Math.max(right - edges.right, edges.left - left); + if (past > 1) { + offenders.push( + `${node.textContent?.trim().slice(0, 60)} (${Math.round(past)}px past ${box.tagName.toLowerCase()})`, + ); + break; + } + } + } + return offenders; + }); + expect(cutOff).toEqual([]); + }); + test("Infrastructure landing loads without horizontal overflow at mobile viewport", async ({ page, }) => { diff --git a/tests/integration/tests/18-patrol-runtime-state.spec.ts b/tests/integration/tests/18-patrol-runtime-state.spec.ts index da3a1deccb..6a1b5a31ec 100644 --- a/tests/integration/tests/18-patrol-runtime-state.spec.ts +++ b/tests/integration/tests/18-patrol-runtime-state.spec.ts @@ -349,20 +349,6 @@ async function mockBlockedPatrolRuntimeState( }); }); - await page.route("**/api/ai/circuit/status", async (route) => { - await route.fulfill({ - status: 200, - contentType: "application/json", - body: JSON.stringify({ - state: "closed", - can_patrol: true, - consecutive_failures: 0, - total_successes: 42, - total_failures: 0, - }), - }); - }); - await page.route("**/api/ai/approvals", async (route) => { await route.fulfill({ status: 200, @@ -523,20 +509,6 @@ async function mockScopedTriggerPatrolRuntimeState( }); }); - await page.route("**/api/ai/circuit/status", async (route) => { - await route.fulfill({ - status: 200, - contentType: "application/json", - body: JSON.stringify({ - state: "closed", - can_patrol: true, - consecutive_failures: 0, - total_successes: 42, - total_failures: 0, - }), - }); - }); - await page.route("**/api/ai/approvals", async (route) => { await route.fulfill({ status: 200, diff --git a/tests/integration/tests/48-summary-hover-selection.spec.ts b/tests/integration/tests/48-summary-hover-selection.spec.ts deleted file mode 100644 index 933ba904ed..0000000000 --- a/tests/integration/tests/48-summary-hover-selection.spec.ts +++ /dev/null @@ -1,1320 +0,0 @@ -import fs from "node:fs"; -import path from "node:path"; -import { fileURLToPath } from "node:url"; -import { expect, test as base } from "@playwright/test"; - -import { - createAuthenticatedStorageState, - getMockMode, - setMockMode, -} from "./helpers"; - -const __dirname = path.dirname(fileURLToPath(import.meta.url)); - -type WorkerFixtures = { - authStorageStatePath: string; -}; - -let mockModeWasEnabled: boolean | null = null; - -const test = base.extend, WorkerFixtures>({ - storageState: async ({ authStorageStatePath }, use) => { - await use(authStorageStatePath); - }, - authStorageStatePath: [ - async ({ browser }, use, workerInfo) => { - const storageStatePath = path.resolve( - __dirname, - "..", - "..", - "tmp", - "playwright-auth", - `summary-hover-selection-${workerInfo.project.name}.json`, - ); - fs.mkdirSync(path.dirname(storageStatePath), { recursive: true }); - await createAuthenticatedStorageState(browser, storageStatePath); - try { - await use(storageStatePath); - } finally { - fs.rmSync(storageStatePath, { force: true }); - } - }, - { scope: "worker" }, - ], -}); - -async function ensureMockModeEnabled( - page: import("@playwright/test").Page, -): Promise { - const state = await getMockMode(page); - if (mockModeWasEnabled === null) { - mockModeWasEnabled = state.enabled; - } - if (!state.enabled) { - await setMockMode(page, true); - } -} - -async function expectSummaryHighlightCount( - summary: import("@playwright/test").Locator, - resourceId: string, - expectedCount: number, -): Promise { - await expect - .poll(async () => - summary - .locator("[data-highlight-series-id]") - .evaluateAll( - (nodes, expectedId) => - nodes.filter( - (node) => - node.getAttribute("data-highlight-series-id") === expectedId && - node.getAttribute("data-highlight-series-active") === "true", - ).length, - resourceId, - ), - ) - .toBe(expectedCount); -} - -async function readSummaryHighlightCount( - summary: import("@playwright/test").Locator, - resourceId: string, -): Promise { - return summary - .locator("[data-highlight-series-id]") - .evaluateAll( - (nodes, expectedId) => - nodes.filter( - (node) => - node.getAttribute("data-highlight-series-id") === expectedId && - node.getAttribute("data-highlight-series-active") === "true", - ).length, - resourceId, - ); -} - -async function hoverRowUntilSummaryHighlights( - page: import("@playwright/test").Page, - rows: import("@playwright/test").Locator, - summary: import("@playwright/test").Locator, - fallbackAttribute: string, - expectedCount: number, -): Promise<{ index: number; resourceId: string }> { - const rowCount = await rows.count(); - for (let index = 0; index < rowCount; index += 1) { - const row = rows.nth(index); - if (!(await row.isVisible())) { - continue; - } - const resourceId = await readSummarySeriesId(row, fallbackAttribute); - if (!resourceId) { - continue; - } - await row.hover(); - try { - await expect - .poll(() => readSummaryHighlightCount(summary, resourceId), { - timeout: 1_500, - }) - .toBe(expectedCount); - return { index, resourceId }; - } catch { - await page.mouse.move(1, 1); - } - } - - throw new Error("Unable to find a row that maps to summary highlight state"); -} - -async function dispatchRowHover( - row: import("@playwright/test").Locator, -): Promise { - await row.dispatchEvent("mouseenter"); - await row.dispatchEvent("mouseover"); - await row.dispatchEvent("focusin"); -} - -async function hoverSummaryChartUntilActiveId( - page: import("@playwright/test").Page, - summary: import("@playwright/test").Locator, - chartRoot: import("@playwright/test").Locator, -): Promise { - const surface = chartRoot.locator("svg, canvas").first(); - await expect(surface).toBeVisible(); - const box = await surface.boundingBox(); - expect(box).not.toBeNull(); - if (!box) { - throw new Error("Chart surface has no bounding box"); - } - - const probePoints = [ - { x: 0.88, y: 0.25 }, - { x: 0.88, y: 0.5 }, - { x: 0.72, y: 0.25 }, - { x: 0.72, y: 0.5 }, - { x: 0.56, y: 0.25 }, - { x: 0.56, y: 0.5 }, - ]; - - for (const point of probePoints) { - await page.mouse.move( - box.x + box.width * point.x, - box.y + box.height * point.y, - ); - const activeIds = await summary - .locator("[data-highlight-series-id]") - .evaluateAll((nodes) => - Array.from( - new Set( - nodes - .filter( - (node) => - node.getAttribute("data-highlight-series-active") === "true", - ) - .map( - (node) => node.getAttribute("data-highlight-series-id") || "", - ) - .filter(Boolean), - ), - ), - ); - if (activeIds.length === 1) { - return activeIds[0]; - } - } - - throw new Error("Unable to activate a summary chart hover state"); -} - -async function expectActiveIsolatedLineCards( - summary: import("@playwright/test").Locator, - expectedCount: number, -): Promise { - await expect - .poll(async () => - summary - .locator('[data-active-series-display="isolate"]') - .evaluateAll( - (nodes) => - nodes.filter( - (node) => - node.getAttribute("data-highlight-series-active") === "true" && - node.getAttribute("data-rendered-series-count") === "1", - ).length, - ), - ) - .toBe(expectedCount); -} - -async function expectActiveDensityMapsPreserveOverview( - summary: import("@playwright/test").Locator, - resourceId: string, - expectedCount: number, -): Promise { - await expect - .poll(async () => - summary - .locator('[data-summary-chart-kind="density-map"]') - .evaluateAll((nodes, expectedId) => { - const activeNodes = nodes.filter( - (node) => - node.getAttribute("data-highlight-series-id") === expectedId && - node.getAttribute("data-highlight-series-active") === "true", - ); - return { - activeCount: activeNodes.length, - preservedOverview: activeNodes.every( - (node) => - Number(node.getAttribute("data-rendered-series-count") || "0") > - 1, - ), - }; - }, resourceId), - ) - .toEqual({ - activeCount: expectedCount, - preservedOverview: true, - }); -} - -async function expectSummarySyncedReadoutCount( - summary: import("@playwright/test").Locator, - expectedCount: number, -): Promise { - await expect - .poll(async () => summary.locator('[data-summary-sync-readout="true"]').count()) - .toBe(expectedCount); -} - -async function expectOnlyOneSummaryHoverTooltip( - page: import("@playwright/test").Page, -): Promise { - await expect - .poll(async () => - page - .locator('[data-sparkline-tooltip="true"], [data-density-map-tooltip="true"]') - .count(), - ) - .toBe(1); -} - -async function hoverDensityMapUntilTooltipAppears( - page: import("@playwright/test").Page, - densityMap: import("@playwright/test").Locator, -): Promise { - const canvas = densityMap.locator("canvas").first(); - await expect(canvas).toBeVisible(); - const box = await canvas.boundingBox(); - expect(box).not.toBeNull(); - if (!box) { - throw new Error("Density map canvas has no bounding box"); - } - - for (const point of [ - { x: 0.84, y: 0.2 }, - { x: 0.7, y: 0.35 }, - { x: 0.56, y: 0.5 }, - ]) { - await page.mouse.move( - box.x + box.width * point.x, - box.y + box.height * point.y, - ); - const tooltip = page.locator('[data-density-map-tooltip="true"]').last(); - if (await tooltip.isVisible().catch(() => false)) { - return; - } - } - - throw new Error("Unable to activate a density-map tooltip"); -} - -async function expectSummaryHoverTimestampsAligned( - summary: import("@playwright/test").Locator, - expectedCount: number, -): Promise { - await expect - .poll(async () => - summary.locator("[data-active-hover-timestamp]").evaluateAll((nodes) => { - const timestamps = nodes - .map((node) => node.getAttribute("data-active-hover-timestamp") || "") - .filter(Boolean); - return { - count: timestamps.length, - uniqueCount: new Set(timestamps).size, - }; - }), - ) - .toEqual({ - count: expectedCount, - uniqueCount: 1, - }); -} - -async function expectSparklineSourceCursorTracksLocalScrub( - page: import("@playwright/test").Page, - sparklineRoot: import("@playwright/test").Locator, -): Promise { - const surface = sparklineRoot.locator("svg").first(); - await expect(surface).toBeVisible(); - const box = await surface.boundingBox(); - expect(box).not.toBeNull(); - if (!box) { - throw new Error("Sparkline surface has no bounding box"); - } - - const samples: Array<{ cursorX: number; lineX: number }> = []; - for (const ratio of [0.15, 0.45, 0.75]) { - await page.mouse.move( - box.x + box.width * ratio, - box.y + box.height * 0.45, - ); - await expect - .poll(async () => - sparklineRoot.evaluate((node) => { - const activeCursor = Number.parseFloat( - node.getAttribute("data-active-hover-cursor-x") || "NaN", - ); - const svgCursor = Number.parseFloat( - node - .querySelector('line[stroke-dasharray="3 3"]') - ?.getAttribute("x1") || "NaN", - ); - return Number.isFinite(activeCursor) && Number.isFinite(svgCursor); - }), - ) - .toBe(true); - - const sample = await sparklineRoot.evaluate((node) => ({ - cursorX: Number.parseFloat( - node.getAttribute("data-active-hover-cursor-x") || "NaN", - ), - lineX: Number.parseFloat( - node.querySelector('line[stroke-dasharray="3 3"]')?.getAttribute("x1") || - "NaN", - ), - lineY1: Number.parseFloat( - node.querySelector('line[stroke-dasharray="3 3"]')?.getAttribute("y1") || - "NaN", - ), - lineY2: Number.parseFloat( - node.querySelector('line[stroke-dasharray="3 3"]')?.getAttribute("y2") || - "NaN", - ), - })); - expect(sample.cursorX).toBeCloseTo(sample.lineX, 1); - expect(sample.lineY1).toBe(0); - expect(sample.lineY2).toBe(100); - samples.push(sample); - } - - expect(samples[0].lineX).toBeLessThan(samples[1].lineX); - expect(samples[1].lineX).toBeLessThan(samples[2].lineX); - await page.mouse.move(1, 1); -} - -async function expectSparklineTooltipTracksPointer( - page: import("@playwright/test").Page, - sparklineRoot: import("@playwright/test").Locator, -): Promise { - const surface = sparklineRoot.locator("svg").first(); - await expect(surface).toBeVisible(); - const box = await surface.boundingBox(); - expect(box).not.toBeNull(); - if (!box) { - throw new Error("Sparkline surface has no bounding box"); - } - - const tooltip = page.locator('[data-sparkline-tooltip="true"]').last(); - const readTooltipTop = async () => - tooltip.evaluate((node) => - Number.parseFloat(node.closest("foreignObject")?.getAttribute("y") || "NaN"), - ); - - await page.mouse.move(box.x + box.width * 0.45, box.y + box.height * 0.2); - await expect(tooltip).toBeVisible(); - const firstTop = await readTooltipTop(); - - await page.mouse.move(box.x + box.width * 0.45, box.y + box.height * 0.75); - await expect(tooltip).toBeVisible(); - const secondTop = await readTooltipTop(); - - expect(secondTop).toBeGreaterThan(firstTop + 40); - await page.mouse.move(1, 1); -} - -async function readSummarySeriesId( - row: import("@playwright/test").Locator, - fallbackAttribute: string, -): Promise { - return ( - (await row.getAttribute("data-summary-series-id")) ?? - (await row.getAttribute(fallbackAttribute)) ?? - "" - ); -} - -async function readRenderedSeriesCounts( - summary: import("@playwright/test").Locator, -): Promise { - return summary.locator("[data-rendered-series-count]").evaluateAll((nodes) => - nodes.map((node) => - Number.parseInt(node.getAttribute("data-rendered-series-count") || "0", 10), - ), - ); -} - -async function readSummaryCardHeights( - summary: import("@playwright/test").Locator, -): Promise { - return summary.locator("[data-summary-card-state]").evaluateAll((nodes) => - nodes.map((node) => node.getBoundingClientRect().height), - ); -} - -async function expectSummaryCardHeightsStableAcrossRowHover( - page: import("@playwright/test").Page, - summary: import("@playwright/test").Locator, - rows: import("@playwright/test").Locator, - sampleCount = 8, - expectedCardCount = 4, -): Promise { - await expect - .poll(async () => { - const heights = await readSummaryCardHeights(summary); - return ( - heights.length === expectedCardCount && - heights.every((height) => Number.isFinite(height) && height >= 180) - ); - }) - .toBe(true); - - const baselineHeights = await readSummaryCardHeights(summary); - expect(baselineHeights.length).toBeGreaterThan(0); - - const rowCount = await rows.count(); - let exercisedRows = 0; - for (let index = 0; index < rowCount && exercisedRows < sampleCount; index += 1) { - const row = rows.nth(index); - if (!(await row.isVisible())) { - continue; - } - await row.hover(); - await page.waitForTimeout(120); - const currentHeights = await readSummaryCardHeights(summary); - expect(currentHeights).toHaveLength(baselineHeights.length); - currentHeights.forEach((height, cardIndex) => { - expect(height).toBeCloseTo(baselineHeights[cardIndex], 0); - }); - exercisedRows += 1; - } - - expect(exercisedRows).toBeGreaterThan(0); - await page.mouse.move(1, 1); - const settledHeights = await readSummaryCardHeights(summary); - settledHeights.forEach((height, cardIndex) => { - expect(height).toBeCloseTo(baselineHeights[cardIndex], 0); - }); -} - -async function scrollPrimaryViewportToBottom( - page: import("@playwright/test").Page, -): Promise { - await page.evaluate(() => { - const shell = document.querySelector(".app-scroll-shell"); - if (shell) { - shell.scrollTo({ top: shell.scrollHeight }); - return; - } - window.scrollTo(0, document.body.scrollHeight); - }); -} - -test.describe.serial("Summary hover selection", () => { - test.setTimeout(180_000); - - test.afterAll(async ({ browser }) => { - if (mockModeWasEnabled === null) return; - - const context = await browser.newContext(); - const page = await context.newPage(); - try { - const current = await getMockMode(page); - if (current.enabled !== mockModeWasEnabled) { - await setMockMode(page, mockModeWasEnabled); - } - } finally { - await context.close(); - } - }); - - test("keeps summary chart hover selection coherent across infrastructure, workloads, and storage", async ({ - page, - }, testInfo) => { - test.skip( - testInfo.project.name.startsWith("mobile-"), - "Desktop runtime proof", - ); - - await ensureMockModeEnabled(page); - - await page.goto("/infrastructure", { waitUntil: "domcontentloaded" }); - const infrastructureSummary = page.getByTestId("infrastructure-summary"); - await expect(infrastructureSummary).toBeVisible(); - const infrastructureRows = page.locator( - '[data-testid="infrastructure-page"] tr[data-row-id]', - ); - const firstInfrastructureRow = infrastructureRows.first(); - await expect(firstInfrastructureRow).toBeVisible(); - const { index: infrastructureRowIndex, resourceId: infrastructureRowId } = - await hoverRowUntilSummaryHighlights( - page, - infrastructureRows, - infrastructureSummary, - "data-row-id", - 4, - ); - const infrastructureRow = infrastructureRows.nth(infrastructureRowIndex); - expect(infrastructureRowId).not.toBe(""); - await expectActiveIsolatedLineCards(infrastructureSummary, 2); - await infrastructureRow.click(); - await infrastructureSummary.hover(); - await expectSummaryHighlightCount( - infrastructureSummary, - infrastructureRowId, - 4, - ); - await expectActiveIsolatedLineCards(infrastructureSummary, 2); - await expectActiveDensityMapsPreserveOverview( - infrastructureSummary, - infrastructureRowId, - 2, - ); - const infrastructureSparklines = infrastructureSummary.locator( - "[data-active-series-display]", - ); - await expectSparklineSourceCursorTracksLocalScrub( - page, - infrastructureSparklines.nth(0), - ); - await expectSparklineTooltipTracksPointer( - page, - infrastructureSparklines.nth(0), - ); - await expectSparklineSourceCursorTracksLocalScrub( - page, - infrastructureSparklines.nth(1), - ); - await expectSparklineTooltipTracksPointer( - page, - infrastructureSparklines.nth(1), - ); - - await page.goto("/workloads", { waitUntil: "domcontentloaded" }); - const workloadsSummary = page.getByTestId("workloads-summary"); - await expect(workloadsSummary).toBeVisible(); - const workloadRows = page.locator("tr[data-guest-id]"); - const firstWorkloadRow = workloadRows.first(); - await expect(firstWorkloadRow).toBeVisible(); - const { resourceId: workloadRowId } = await hoverRowUntilSummaryHighlights( - page, - workloadRows, - workloadsSummary, - "data-guest-id", - 4, - ); - expect(workloadRowId).not.toBe(""); - await expectActiveIsolatedLineCards(workloadsSummary, 2); - await expectActiveDensityMapsPreserveOverview( - workloadsSummary, - workloadRowId, - 2, - ); - const workloadSparklines = workloadsSummary.locator( - "[data-active-series-display]", - ); - await expectSparklineSourceCursorTracksLocalScrub( - page, - workloadSparklines.nth(0), - ); - await expectSparklineTooltipTracksPointer( - page, - workloadSparklines.nth(0), - ); - await expectSparklineSourceCursorTracksLocalScrub( - page, - workloadSparklines.nth(1), - ); - await expectSparklineTooltipTracksPointer( - page, - workloadSparklines.nth(1), - ); - - const vmwareWorkloadRow = page - .locator('tr[data-guest-id^="vm-"]', { hasText: "warehouse-api-01" }) - .first(); - await expect(vmwareWorkloadRow).toBeVisible(); - const vmwareWorkloadId = await readSummarySeriesId( - vmwareWorkloadRow, - "data-guest-id", - ); - expect(vmwareWorkloadId).not.toBe(""); - await dispatchRowHover(vmwareWorkloadRow); - await expectSummaryHighlightCount(workloadsSummary, vmwareWorkloadId, 4); - await expectActiveIsolatedLineCards(workloadsSummary, 2); - await expectActiveDensityMapsPreserveOverview( - workloadsSummary, - vmwareWorkloadId, - 2, - ); - - await page.goto("/storage", { waitUntil: "domcontentloaded" }); - const storageSummary = page.getByTestId("storage-summary"); - await expect(storageSummary).toBeVisible(); - const stickyStorageSummary = storageSummary.locator( - 'xpath=ancestor::*[@data-sticky-summary="true"][1]', - ); - await expect(stickyStorageSummary).toHaveAttribute( - "data-sticky-summary-desktop-only", - "false", - ); - - const storagePoolRows = page.locator("tr[data-row-id]"); - const firstStoragePoolRow = storagePoolRows.first(); - await expect(firstStoragePoolRow).toBeVisible(); - const { resourceId: storagePoolRowId } = - await hoverRowUntilSummaryHighlights( - page, - storagePoolRows, - storageSummary, - "data-row-id", - 3, - ); - expect(storagePoolRowId).not.toBe(""); - await expectActiveIsolatedLineCards(storageSummary, 3); - - await page.getByRole("tab", { name: "Physical Disks" }).click(); - const storageDiskRows = page.locator("tr[data-row-id]"); - const firstStorageDiskRow = storageDiskRows.first(); - await expect(firstStorageDiskRow).toBeVisible(); - const { resourceId: storageDiskRowId } = - await hoverRowUntilSummaryHighlights( - page, - storageDiskRows, - storageSummary, - "data-row-id", - 1, - ); - expect(storageDiskRowId).not.toBe(""); - await expectActiveIsolatedLineCards(storageSummary, 1); - - await scrollPrimaryViewportToBottom(page); - await page.waitForTimeout(250); - const stickyBox = await stickyStorageSummary.boundingBox(); - expect(stickyBox).not.toBeNull(); - expect(stickyBox?.y ?? Number.POSITIVE_INFINITY).toBeLessThanOrEqual(96); - }); - - test("synchronizes chart hover across summary cards on infrastructure, workloads, and storage", async ({ - page, - }, testInfo) => { - test.skip( - testInfo.project.name.startsWith("mobile-"), - "Desktop runtime proof", - ); - - await ensureMockModeEnabled(page); - - await page.goto("/infrastructure", { waitUntil: "domcontentloaded" }); - const infrastructureSummary = page.getByTestId("infrastructure-summary"); - await expect(infrastructureSummary).toBeVisible(); - const infrastructureChartId = await hoverSummaryChartUntilActiveId( - page, - infrastructureSummary, - infrastructureSummary - .locator('[data-active-series-display="isolate"]') - .first(), - ); - expect(infrastructureChartId).not.toBe(""); - await expectSummaryHighlightCount( - infrastructureSummary, - infrastructureChartId, - 4, - ); - await expectSummaryHoverTimestampsAligned(infrastructureSummary, 4); - await expectActiveIsolatedLineCards(infrastructureSummary, 2); - await expectActiveDensityMapsPreserveOverview( - infrastructureSummary, - infrastructureChartId, - 2, - ); - await expectSummarySyncedReadoutCount(infrastructureSummary, 3); - await expectOnlyOneSummaryHoverTooltip(page); - - await page.goto("/workloads", { waitUntil: "domcontentloaded" }); - const workloadsSummary = page.getByTestId("workloads-summary"); - await expect(workloadsSummary).toBeVisible(); - const workloadChartId = await hoverSummaryChartUntilActiveId( - page, - workloadsSummary, - workloadsSummary - .locator('[data-active-series-display="isolate"]') - .first(), - ); - expect(workloadChartId).not.toBe(""); - await expectSummaryHighlightCount(workloadsSummary, workloadChartId, 4); - await expectSummaryHoverTimestampsAligned(workloadsSummary, 4); - await expectActiveIsolatedLineCards(workloadsSummary, 2); - await expectActiveDensityMapsPreserveOverview( - workloadsSummary, - workloadChartId, - 2, - ); - await expectSummarySyncedReadoutCount(workloadsSummary, 3); - await expectOnlyOneSummaryHoverTooltip(page); - - await page.goto("/storage", { waitUntil: "domcontentloaded" }); - const storageSummary = page.getByTestId("storage-summary"); - await expect(storageSummary).toBeVisible(); - const storageChartId = await hoverSummaryChartUntilActiveId( - page, - storageSummary, - storageSummary.locator('[data-active-series-display="isolate"]').first(), - ); - expect(storageChartId).not.toBe(""); - await expectSummaryHighlightCount(storageSummary, storageChartId, 3); - await expectSummaryHoverTimestampsAligned(storageSummary, 4); - await expectActiveIsolatedLineCards(storageSummary, 3); - await expectSummarySyncedReadoutCount(storageSummary, 2); - await expectOnlyOneSummaryHoverTooltip(page); - }); - - test("keeps summary card geometry stable while row hover changes active series", async ({ - page, - }, testInfo) => { - test.skip( - testInfo.project.name.startsWith("mobile-"), - "Desktop runtime proof", - ); - - await ensureMockModeEnabled(page); - - await page.goto("/infrastructure", { waitUntil: "domcontentloaded" }); - const infrastructureSummary = page.getByTestId("infrastructure-summary"); - await expect(infrastructureSummary).toBeVisible(); - await expectSummaryCardHeightsStableAcrossRowHover( - page, - infrastructureSummary, - page.locator("tr[data-summary-series-id]"), - ); - - await page.goto("/workloads", { waitUntil: "domcontentloaded" }); - const workloadsSummary = page.getByTestId("workloads-summary"); - await expect(workloadsSummary).toBeVisible(); - await expectSummaryCardHeightsStableAcrossRowHover( - page, - workloadsSummary, - page.locator("tr[data-guest-id]"), - ); - }); - - test("keeps table coordination non-destructive and reversible", async ({ - page, - }, testInfo) => { - test.skip( - testInfo.project.name.startsWith("mobile-"), - "Desktop runtime proof", - ); - - await ensureMockModeEnabled(page); - - await page.goto("/infrastructure", { waitUntil: "domcontentloaded" }); - const infrastructureSummary = page.getByTestId("infrastructure-summary"); - await expect(infrastructureSummary).toBeVisible(); - const infrastructureChartId = await hoverSummaryChartUntilActiveId( - page, - infrastructureSummary, - infrastructureSummary - .locator('[data-active-series-display="isolate"]') - .first(), - ); - const highlightedInfrastructureRow = page.locator( - `tr[data-summary-series-id="${infrastructureChartId}"]`, - ); - const infrastructureJumpButton = page.getByRole("button", { - name: "Jump to row", - }); - if (await highlightedInfrastructureRow.first().isVisible()) { - await expect(highlightedInfrastructureRow.first()).toHaveAttribute( - "data-summary-row-active", - "true", - ); - } else { - await expect(infrastructureJumpButton).toBeVisible(); - await infrastructureJumpButton.click(); - await expect(highlightedInfrastructureRow.first()).toBeVisible(); - } - - await page.goto("/workloads", { waitUntil: "domcontentloaded" }); - const workloadsSummary = page.getByTestId("workloads-summary"); - await expect(workloadsSummary).toBeVisible(); - const focusedWorkloadRow = page.locator("tr[data-guest-id]").first(); - const focusedWorkloadId = await readSummarySeriesId( - focusedWorkloadRow, - "data-guest-id", - ); - expect(focusedWorkloadId).not.toBe(""); - await focusedWorkloadRow.click(); - await scrollPrimaryViewportToBottom(page); - const jumpButton = page.getByRole("button", { name: "Jump to row" }); - const focusedSummaryRow = page - .locator(`tr[data-summary-series-id="${focusedWorkloadId}"]`) - .first(); - if (await jumpButton.isVisible()) { - await jumpButton.click(); - await expect(focusedSummaryRow).toBeVisible(); - } else { - await expect(focusedSummaryRow).toBeVisible(); - } - }); - - test("treats workload group headers as first-class summary scope", async ({ - page, - }, testInfo) => { - test.skip( - testInfo.project.name.startsWith("mobile-"), - "Desktop runtime proof", - ); - - await ensureMockModeEnabled(page); - - await page.goto("/workloads", { waitUntil: "domcontentloaded" }); - const workloadsSummary = page.getByTestId("workloads-summary"); - await expect(workloadsSummary).toBeVisible(); - - await expect - .poll(async () => { - const counts = await readRenderedSeriesCounts(workloadsSummary); - return counts.length === 4 && counts.every((count) => count > 1); - }) - .toBe(true); - const resolvedBaselineCounts = await readRenderedSeriesCounts(workloadsSummary); - - const baselineVisibleRows = await page.locator("tr[data-guest-id]").count(); - const groupRows = page.locator("tr[data-summary-group-id]"); - const groupRowCount = await groupRows.count(); - expect(groupRowCount).toBeGreaterThan(0); - - let matchedGroupRow: import("@playwright/test").Locator | null = null; - let matchedGroupSeriesCount = 0; - for (let index = 0; index < groupRowCount; index += 1) { - const row = groupRows.nth(index); - if (!(await row.isVisible())) { - continue; - } - const rawSeriesCount = await row.getAttribute("data-summary-group-series-count"); - const seriesCount = Number.parseInt(rawSeriesCount || "0", 10); - if (!Number.isFinite(seriesCount) || seriesCount < 1) { - continue; - } - if (resolvedBaselineCounts.every((count) => count <= seriesCount)) { - continue; - } - - matchedGroupRow = row; - matchedGroupSeriesCount = seriesCount; - break; - } - - expect(matchedGroupRow).not.toBeNull(); - expect(matchedGroupSeriesCount).toBeGreaterThan(0); - if (!matchedGroupRow) { - return; - } - - await matchedGroupRow.hover(); - await expect(matchedGroupRow).toHaveAttribute("data-summary-row-active", "true"); - await expect(page.locator('tr[data-summary-group-member-active="preview"]')).toHaveCount( - matchedGroupSeriesCount, - ); - await expect - .poll(() => readRenderedSeriesCounts(workloadsSummary)) - .toEqual(new Array(4).fill(matchedGroupSeriesCount)); - await expect(page.locator("tr[data-guest-id]")).toHaveCount(baselineVisibleRows); - - await page.mouse.move(1, 1); - await expect(matchedGroupRow).toHaveAttribute("data-summary-row-active", "false"); - await expect(page.locator('tr[data-summary-group-member-active="preview"]')).toHaveCount(0); - await expect - .poll(() => readRenderedSeriesCounts(workloadsSummary)) - .toEqual(resolvedBaselineCounts); - }); - - test("treats infrastructure cluster headers as first-class summary scope", async ({ - page, - }, testInfo) => { - test.skip( - testInfo.project.name.startsWith("mobile-"), - "Desktop runtime proof", - ); - - await ensureMockModeEnabled(page); - - await page.goto("/infrastructure", { waitUntil: "domcontentloaded" }); - const infrastructureSummary = page.getByTestId("infrastructure-summary"); - await expect(infrastructureSummary).toBeVisible(); - - await expect - .poll(async () => { - const counts = await readRenderedSeriesCounts(infrastructureSummary); - return counts.length === 4 && counts.every((count) => count > 1); - }) - .toBe(true); - const resolvedBaselineCounts = await readRenderedSeriesCounts( - infrastructureSummary, - ); - - const baselineVisibleRows = await page.locator("tr[data-summary-series-id]").count(); - const groupRows = page.locator("tr[data-summary-group-id]"); - const groupRowCount = await groupRows.count(); - expect(groupRowCount).toBeGreaterThan(0); - - let matchedGroupRow: import("@playwright/test").Locator | null = null; - let matchedGroupSeriesCount = 0; - for (let index = 0; index < groupRowCount; index += 1) { - const row = groupRows.nth(index); - if (!(await row.isVisible())) { - continue; - } - const rawSeriesCount = await row.getAttribute("data-summary-group-series-count"); - const seriesCount = Number.parseInt(rawSeriesCount || "0", 10); - if (!Number.isFinite(seriesCount) || seriesCount < 1) { - continue; - } - if (resolvedBaselineCounts.every((count) => count <= seriesCount)) { - continue; - } - - matchedGroupRow = row; - matchedGroupSeriesCount = seriesCount; - break; - } - - expect(matchedGroupRow).not.toBeNull(); - expect(matchedGroupSeriesCount).toBeGreaterThan(0); - if (!matchedGroupRow) { - return; - } - - await matchedGroupRow.hover(); - await expect(matchedGroupRow).toHaveAttribute("data-summary-row-active", "true"); - await expect(page.locator('tr[data-summary-group-member-active="preview"]')).toHaveCount( - matchedGroupSeriesCount, - ); - await expect - .poll(() => readRenderedSeriesCounts(infrastructureSummary)) - .toEqual(new Array(4).fill(matchedGroupSeriesCount)); - await expect(page.locator("tr[data-summary-series-id]")).toHaveCount( - baselineVisibleRows, - ); - - await page.mouse.move(1, 1); - await expect(matchedGroupRow).toHaveAttribute("data-summary-row-active", "false"); - await expect(page.locator('tr[data-summary-group-member-active="preview"]')).toHaveCount(0); - await expect - .poll(() => readRenderedSeriesCounts(infrastructureSummary)) - .toEqual(resolvedBaselineCounts); - }); - - test("pins workload and infrastructure group scope into reversible route-backed focus", async ({ - page, - }, testInfo) => { - test.skip( - testInfo.project.name.startsWith("mobile-"), - "Desktop runtime proof", - ); - - await ensureMockModeEnabled(page); - - for (const surface of [ - { - path: "/workloads", - summaryTestId: "workloads-summary", - tableSurfaceTestId: "workloads-table-surface", - tableRowSelector: "tr[data-guest-id]", - }, - { - path: "/infrastructure", - summaryTestId: "infrastructure-summary", - tableSurfaceTestId: "infrastructure-table-surface", - tableRowSelector: "tr[data-summary-series-id]", - }, - ] as const) { - await page.goto(surface.path, { waitUntil: "domcontentloaded" }); - const summary = page.getByTestId(surface.summaryTestId); - await expect(summary).toBeVisible(); - await expect(page.getByText("Pinned to")).toHaveCount(0); - - await expect - .poll(async () => { - const counts = await readRenderedSeriesCounts(summary); - return counts.length === 4 && counts.every((count) => count > 1); - }) - .toBe(true); - - const baselineCounts = await readRenderedSeriesCounts(summary); - const baselineVisibleRows = await page.locator(surface.tableRowSelector).count(); - const groupRows = page.locator("tr[data-summary-group-id]"); - const groupRowCount = await groupRows.count(); - expect(groupRowCount).toBeGreaterThan(0); - - let matchedGroupRow: import("@playwright/test").Locator | null = null; - let matchedGroupSeriesCount = 0; - let matchedGroupId = ""; - for (let index = 0; index < groupRowCount; index += 1) { - const row = groupRows.nth(index); - if (!(await row.isVisible())) { - continue; - } - const rawSeriesCount = await row.getAttribute("data-summary-group-series-count"); - const seriesCount = Number.parseInt(rawSeriesCount || "0", 10); - const groupId = (await row.getAttribute("data-summary-group-id")) || ""; - if (!Number.isFinite(seriesCount) || seriesCount < 1 || !groupId) { - continue; - } - if (baselineCounts.every((count) => count <= seriesCount)) { - continue; - } - matchedGroupRow = row; - matchedGroupSeriesCount = seriesCount; - matchedGroupId = groupId; - } - - expect(matchedGroupRow).not.toBeNull(); - expect(matchedGroupSeriesCount).toBeGreaterThan(0); - expect(matchedGroupId).not.toBe(""); - if (!matchedGroupRow) { - return; - } - - await matchedGroupRow.hover(); - await expect(page.getByText("Pinned to")).toHaveCount(0); - await expect(page.locator('tr[data-summary-group-member-active="preview"]')).toHaveCount( - matchedGroupSeriesCount, - ); - await page.mouse.move(1, 1); - await expect(page.getByText("Pinned to")).toHaveCount(0); - await expect(page.locator('tr[data-summary-group-member-active="preview"]')).toHaveCount(0); - - await matchedGroupRow.click(); - await expect - .poll(() => new URL(page.url()).searchParams.get("summaryGroup")) - .toBe(matchedGroupId); - await expect(page.getByText("Pinned to")).toHaveCount(0); - await expect(page.locator('tr[data-summary-group-member-active="preview"]')).toHaveCount( - matchedGroupSeriesCount, - ); - await expect - .poll(() => readRenderedSeriesCounts(summary)) - .toEqual(new Array(4).fill(matchedGroupSeriesCount)); - await expect(page.locator(surface.tableRowSelector)).toHaveCount( - baselineVisibleRows, - ); - - await page.mouse.move(1, 1); - await expect(page.locator('tr[data-summary-group-member-active="preview"]')).toHaveCount(0); - await expect(page.locator('tr[data-summary-group-member-active="pinned"]')).toHaveCount( - matchedGroupSeriesCount, - ); - await expect - .poll(() => readRenderedSeriesCounts(summary)) - .toEqual(new Array(4).fill(matchedGroupSeriesCount)); - - await page.evaluate(() => { - const shell = document.querySelector(".app-scroll-shell"); - if (shell) { - shell.scrollTo({ top: 0 }); - return; - } - window.scrollTo(0, 0); - }); - await expect - .poll(async () => - page.evaluate((groupId) => { - const row = document.querySelector( - `tr[data-summary-group-id="${groupId}"]`, - ); - if (!row) { - return false; - } - const rect = row.getBoundingClientRect(); - return rect.bottom <= 0 || rect.top >= window.innerHeight; - }, matchedGroupId), - ) - .toBe(true); - await expect(page.getByText("Pinned to")).toHaveCount(0); - - await page - .getByTestId(surface.tableSurfaceTestId) - .getByRole("button", { name: "Clear selection" }) - .click(); - await expect - .poll(() => new URL(page.url()).searchParams.get("summaryGroup")) - .toBeNull(); - await expect(page.getByText("Pinned to")).toHaveCount(0); - await expect(page.locator('tr[data-summary-group-member-active="pinned"]')).toHaveCount(0); - await page.mouse.move(1, 1); - await expect - .poll(() => readRenderedSeriesCounts(summary)) - .toEqual(baselineCounts); - - await matchedGroupRow.scrollIntoViewIfNeeded(); - await matchedGroupRow.click(); - await expect - .poll(() => new URL(page.url()).searchParams.get("summaryGroup")) - .toBe(matchedGroupId); - await page.keyboard.press("Escape"); - await expect - .poll(() => new URL(page.url()).searchParams.get("summaryGroup")) - .toBeNull(); - await expect(page.locator('tr[data-summary-group-member-active="pinned"]')).toHaveCount(0); - await page.mouse.move(1, 1); - await expect - .poll(() => readRenderedSeriesCounts(summary)) - .toEqual(baselineCounts); - } - }); - - test("pins storage pool-group scope into reversible route-backed focus without collapsing the table", async ({ - page, - }, testInfo) => { - test.skip( - testInfo.project.name.startsWith("mobile-"), - "Desktop runtime proof", - ); - - await ensureMockModeEnabled(page); - - await page.goto("/storage?group=node", { waitUntil: "domcontentloaded" }); - const storageSummary = page.getByTestId("storage-summary"); - await expect(storageSummary).toBeVisible(); - await expect(page.getByText("Pinned to")).toHaveCount(0); - - await expect - .poll(async () => { - const counts = await readRenderedSeriesCounts(storageSummary); - return counts.length >= 3 && counts.some((count) => count > 1); - }) - .toBe(true); - - const baselineCounts = await readRenderedSeriesCounts(storageSummary); - const baselineVisibleRows = await page.locator("tr[data-row-id]").count(); - const groupRows = page.locator("tr[data-summary-group-id]"); - const groupRowCount = await groupRows.count(); - expect(groupRowCount).toBeGreaterThan(0); - - let matchedGroupRow: import("@playwright/test").Locator | null = null; - let matchedGroupId = ""; - let matchedGroupSeriesCount = 0; - for (let index = 0; index < groupRowCount; index += 1) { - const row = groupRows.nth(index); - if (!(await row.isVisible())) { - continue; - } - const rawSeriesCount = await row.getAttribute("data-summary-group-series-count"); - const seriesCount = Number.parseInt(rawSeriesCount || "0", 10); - const groupId = (await row.getAttribute("data-summary-group-id")) || ""; - if (!Number.isFinite(seriesCount) || seriesCount < 1 || !groupId) { - continue; - } - if (baselineCounts.every((count) => count <= seriesCount)) { - continue; - } - matchedGroupRow = row; - matchedGroupId = groupId; - matchedGroupSeriesCount = seriesCount; - } - - expect(matchedGroupRow).not.toBeNull(); - expect(matchedGroupId).not.toBe(""); - expect(matchedGroupSeriesCount).toBeGreaterThan(0); - if (!matchedGroupRow) { - return; - } - - await matchedGroupRow.hover(); - await expect(page.getByText("Pinned to")).toHaveCount(0); - await expect(page.locator('tr[data-summary-group-member-active="preview"]')).toHaveCount( - matchedGroupSeriesCount, - ); - await page.mouse.move(1, 1); - await expect(page.getByText("Pinned to")).toHaveCount(0); - await expect(page.locator('tr[data-summary-group-member-active="preview"]')).toHaveCount(0); - - await matchedGroupRow.click(); - await expect - .poll(() => new URL(page.url()).searchParams.get("summaryGroup")) - .toBe(matchedGroupId); - await expect(page.getByText("Pinned to")).toHaveCount(0); - await expect(page.locator('tr[data-summary-group-member-active="preview"]')).toHaveCount( - matchedGroupSeriesCount, - ); - await expect(page.locator("tr[data-row-id]")).toHaveCount(baselineVisibleRows); - - await expect - .poll(() => readRenderedSeriesCounts(storageSummary)) - .not.toEqual(baselineCounts); - - await page.mouse.move(1, 1); - await expect(page.locator('tr[data-summary-group-member-active="preview"]')).toHaveCount(0); - await expect(page.locator('tr[data-summary-group-member-active="pinned"]')).toHaveCount( - matchedGroupSeriesCount, - ); - await expect(page.locator("tr[data-row-id]")).toHaveCount(baselineVisibleRows); - - await page.evaluate(() => { - const shell = document.querySelector(".app-scroll-shell"); - if (shell) { - shell.scrollTo({ top: 0 }); - return; - } - window.scrollTo(0, 0); - }); - await expect - .poll(async () => - page.evaluate((groupId) => { - const row = document.querySelector( - `tr[data-summary-group-id="${groupId}"]`, - ); - if (!row) { - return false; - } - const rect = row.getBoundingClientRect(); - return rect.bottom <= 0 || rect.top >= window.innerHeight; - }, matchedGroupId), - ) - .toBe(true); - await expect(page.getByText("Pinned to")).toHaveCount(0); - - await page - .getByTestId("storage-content-surface") - .getByRole("button", { name: "Clear selection" }) - .click(); - await expect(page.locator('tr[data-summary-group-member-active="pinned"]')).toHaveCount(0); - await expect - .poll(() => new URL(page.url()).searchParams.get("summaryGroup")) - .toBeNull(); - await expect(page.getByText("Pinned to")).toHaveCount(0); - await page.mouse.move(1, 1); - await expect - .poll(() => readRenderedSeriesCounts(storageSummary)) - .toEqual(baselineCounts); - - await matchedGroupRow.scrollIntoViewIfNeeded(); - await matchedGroupRow.click(); - await expect - .poll(() => new URL(page.url()).searchParams.get("summaryGroup")) - .toBe(matchedGroupId); - await page.keyboard.press("Escape"); - await expect(page.locator('tr[data-summary-group-member-active="pinned"]')).toHaveCount(0); - await expect - .poll(() => new URL(page.url()).searchParams.get("summaryGroup")) - .toBeNull(); - await page.mouse.move(1, 1); - await expect - .poll(() => readRenderedSeriesCounts(storageSummary)) - .toEqual(baselineCounts); - }); - - test("keeps density-map detail inside the hover tooltip without extra chart chrome", async ({ - page, - }, testInfo) => { - test.skip( - testInfo.project.name.startsWith("mobile-"), - "Desktop runtime proof", - ); - - await ensureMockModeEnabled(page); - - await page.goto("/workloads", { waitUntil: "domcontentloaded" }); - const workloadsSummary = page.getByTestId("workloads-summary"); - await expect(workloadsSummary).toBeVisible(); - await expect( - workloadsSummary.locator('[data-density-map-focus-detail="true"]'), - ).toHaveCount(0); - await expect(workloadsSummary).not.toContainText("Top activity overview"); - - const firstDensityMap = workloadsSummary - .locator('[data-summary-chart-kind="density-map"]') - .first(); - await hoverDensityMapUntilTooltipAppears(page, firstDensityMap); - - const tooltip = page.locator('[data-density-map-tooltip="true"]').last(); - await expect(tooltip).toBeVisible(); - await expect(tooltip).toContainText("Current"); - await expect(tooltip).toContainText("Peak"); - await expect( - tooltip.locator('[data-density-map-tooltip-sparkline="true"]'), - ).toHaveCount(0); - }); -}); diff --git a/tests/integration/tests/51-quickstart-cross-surface.spec.ts b/tests/integration/tests/51-quickstart-cross-surface.spec.ts index 3efb255114..69b6b2e960 100644 --- a/tests/integration/tests/51-quickstart-cross-surface.spec.ts +++ b/tests/integration/tests/51-quickstart-cross-surface.spec.ts @@ -285,20 +285,6 @@ async function mockRetiredQuickstartSurface( }); }); - await page.route("**/api/ai/circuit/status", async (route) => { - await route.fulfill({ - status: 200, - contentType: "application/json", - body: JSON.stringify({ - state: "closed", - can_patrol: false, - consecutive_failures: 0, - total_successes: 0, - total_failures: 0, - }), - }); - }); - await page.route("**/api/ai/approvals", async (route) => { await route.fulfill({ status: 200, diff --git a/tests/integration/tests/73-patrol-assistant-operator-briefing.spec.ts b/tests/integration/tests/73-patrol-assistant-operator-briefing.spec.ts index e122ddf30e..9d5f76a8cd 100644 --- a/tests/integration/tests/73-patrol-assistant-operator-briefing.spec.ts +++ b/tests/integration/tests/73-patrol-assistant-operator-briefing.spec.ts @@ -541,20 +541,6 @@ test.describe("Patrol Assistant operator briefing", () => { }); }); - await page.route("**/api/ai/circuit/status", async (route) => { - await route.fulfill({ - status: 200, - contentType: "application/json", - body: JSON.stringify({ - state: "closed", - can_patrol: true, - consecutive_failures: 0, - total_successes: 42, - total_failures: 0, - }), - }); - }); - await page.route("**/api/ai/approvals", async (route) => { await route.fulfill({ status: 200, diff --git a/tests/integration/tests/78-monitor-first-patrol-workbench.spec.ts b/tests/integration/tests/78-monitor-first-patrol-workbench.spec.ts index 976cdc1cf2..ebc74eb263 100644 --- a/tests/integration/tests/78-monitor-first-patrol-workbench.spec.ts +++ b/tests/integration/tests/78-monitor-first-patrol-workbench.spec.ts @@ -476,20 +476,6 @@ async function mockMonitorFirstPatrolWorkbench( }); }); - await page.route("**/api/ai/circuit/status", async (route) => { - await route.fulfill({ - status: 200, - contentType: "application/json", - body: JSON.stringify({ - state: "closed", - can_patrol: true, - consecutive_failures: 0, - total_successes: 4, - total_failures: 0, - }), - }); - }); - await page.route("**/api/ai/approvals", async (route) => { const approvals = resolved.pendingApprovalCount > 0 diff --git a/tests/integration/tests/91-operational-trust-attention-workbench.spec.ts b/tests/integration/tests/91-operational-trust-attention-workbench.spec.ts index 94d85f0e78..dd374fec6f 100644 --- a/tests/integration/tests/91-operational-trust-attention-workbench.spec.ts +++ b/tests/integration/tests/91-operational-trust-attention-workbench.spec.ts @@ -493,19 +493,6 @@ async function mockAttention( }), }); }); - await page.route("**/api/ai/circuit/status", async (route) => { - await route.fulfill({ - status: 200, - contentType: "application/json", - body: JSON.stringify({ - state: "closed", - can_patrol: true, - consecutive_failures: 0, - total_successes: 1, - total_failures: 0, - }), - }); - }); await page.route("**/api/ai/approvals", async (route) => { await route.fulfill({ status: 200, diff --git a/tests/integration/tests/92-operational-trust-availability-facet.spec.ts b/tests/integration/tests/92-operational-trust-availability-facet.spec.ts index b5d96d83ed..238badf3dc 100644 --- a/tests/integration/tests/92-operational-trust-availability-facet.spec.ts +++ b/tests/integration/tests/92-operational-trust-availability-facet.spec.ts @@ -730,15 +730,6 @@ async function routePatrolSupport( }, }); }); - await page.route("**/api/ai/circuit/status", async (route) => { - await fulfillJSON(route, { - state: "closed", - can_patrol: true, - consecutive_failures: 0, - total_successes: 1, - total_failures: 0, - }); - }); await page.route("**/api/ai/approvals", async (route) => { await fulfillJSON(route, { approvals: [] }); });