diff --git a/docs/API.md b/docs/API.md index 296d4cc116..002d29a1e2 100644 --- a/docs/API.md +++ b/docs/API.md @@ -836,9 +836,12 @@ delivery for all resources. - `GET /api/admin/reports/catalog` (admin, `settings:read`) - Returns the canonical reporting catalog for the settings surface, including locked-state teaser copy, enabled-surface guidance copy, performance-report options, canonical single-report filename subject, canonical fallback filename date style, and the nested VM inventory export definition. - Metadata route: readable without the `advanced_reporting` feature so locked admin surfaces can render the same reporting definition before upsell. -- `GET /api/admin/reports/generate` (admin, `settings:read`) - - Query params: `format` (pdf/csv, default `pdf`), `resourceType`, `resourceId`, `metricType` (optional), `start`/`end` (RFC3339, optional; defaults to last 24h), `title` (optional) +- `POST /api/admin/reports/generate` (admin, `settings:read`) + - Body fields: `format` (pdf/csv, default `pdf`), `resourceType`, `resourceId`, `metricType` (optional), `start`/`end` (RFC3339, optional; defaults to last 24h), `title` (optional) - If `title` is omitted, the backend applies the canonical default title for that resource report. + - When Pulse Assistant is configured, the configured AI provider narrates a PDF's executive summary; each narration counts against the AI budget and appears in AI usage. If the call fails, times out or the budget is spent, the PDF falls back to the deterministic summary. CSV output is never narrated. The settings UI uses this method. +- `GET /api/admin/reports/generate` (admin, `settings:read`) + - Takes the same fields as query params and returns the same report, except that it never calls an AI provider: a PDF carries the deterministic summary and a note saying AI narration needs `POST`. Other methods return `405` with `Allow: GET, POST`. - `POST /api/admin/reports/generate-multi` (admin, `settings:read`) - Body fields: `resources` (1-50 entries of `{resourceType,resourceId}`), `format`, `metricType` (optional), `start`/`end` (RFC3339, optional; defaults to last 24h), `title` (optional) - If `title` is omitted, the backend applies the canonical default fleet report title. @@ -851,8 +854,8 @@ Validation and limits: - Malformed `start`/`end` values, `end` not strictly after `start`, or report windows over 366 days return `400 invalid_time_range`. - `metricType` must match `[a-zA-Z0-9._:-]+` and be <= 64 chars, otherwise `400 invalid_metric_type`. - `title` must be <= 256 chars, otherwise `400 invalid_title`. -- Multi-report body max size is 1MB; oversized payloads return `400 body_too_large`. -- Multi-report bodies reject trailing payload and unknown JSON fields with `400 invalid_body`. +- Report body max size is 1MB; oversized payloads return `400 body_too_large`. +- Report bodies reject trailing payload and unknown JSON fields with `400 invalid_body`. - VM inventory export only accepts `csv`. Common reporting error codes: @@ -891,7 +894,9 @@ For an existing incident whose notifications should be paused without clearing it, use the snooze and unsnooze endpoints below. - `GET /api/alerts/config` -- `PUT /api/alerts/config` +- `PUT /api/alerts/config` โ€” replaces only the top-level keys the body + carries; a key left out keeps its stored value, and the merged config is + normalized as before. The body must be a JSON object. - `GET /api/alerts/deadman/config` โ€” returns only whether an external watchdog is configured; `pingUrl` is `***REDACTED***` when present and never returns the credential-bearing URL diff --git a/docs/MSP.md b/docs/MSP.md index 7561f520c6..9302b66a1d 100644 --- a/docs/MSP.md +++ b/docs/MSP.md @@ -260,10 +260,12 @@ Each client runtime (or organization) generates its own reports, scoped to that client's resources: - **UI**: Settings โ†’ Data & Reports. -- **API**: `GET /api/admin/reports/generate` (single resource) and +- **API**: `POST /api/admin/reports/generate` (single resource) and `POST /api/admin/reports/generate-multi` (up to 50 resources per report), returning PDF or CSV. In shared-process mode, scope with `X-Pulse-Org-ID` - or an org-bound token. + or an org-bound token. `GET /api/admin/reports/generate` with query + parameters still returns a single-resource report, but never adds the + Pulse Assistant narrative. - **Schedules**: `GET`/`POST /api/admin/reports/schedules`, `PUT`/`DELETE /api/admin/reports/schedules/{id}`, and `POST /api/admin/reports/schedules/{id}/run`. Schedules can target explicit diff --git a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md b/docs/release-control/v6/internal/subsystems/agent-lifecycle.md index 07295760aa..21997c6afe 100644 --- a/docs/release-control/v6/internal/subsystems/agent-lifecycle.md +++ b/docs/release-control/v6/internal/subsystems/agent-lifecycle.md @@ -3690,6 +3690,14 @@ Agent` secondary handoff against the live setup wizard instead of relying ## Current State +### Alert settings saves do not touch agent lifecycle + +`PUT /api/alerts/config` in `internal/api/alerting/alerts.go` now keeps stored +values for the top-level keys a client leaves out. Agent alert defaults +(`agentDefaults`) follow the same rule as every other key; enrollment, +reporting leases, commands and agent removal are unchanged. +`internal/api/alerting/alerts_test.go` pins the handler. + ### VM guest execution admission (backup precaution) `AgentRegisterPayload.guest_exec_guard_version` is optional; version 1 on a @@ -7657,6 +7665,16 @@ bounded and classified temperature nodes, and leaves the host report intact when the helper is absent or incompatible. It does not treat Windows ACPI thermal zones as hardware sensor evidence, accept remote provider locations, or grant Pulse hardware-control authority. +### Single-resource report AI narration requires POST + +`internal/api/metrics_reporting_handlers.go`, `internal/api/router.go` and +`internal/api/ai_handlers.go` changed only how `/api/admin/reports/generate` +reads its fields (a `POST` JSON body or `GET` query parameters), which of those +may use the tenant's AI narrator (`POST` only), and how a `GET` finds Patrol +findings (`ExistingAIService`, which never constructs a tenant AI service). No +agent registration, enrolment, install, update, removal or report ingest path +changed. + ### Agent profile adoption is reported as a count only The usage telemetry snapshot reads `LoadAgentProfiles` and reports @@ -8825,6 +8843,25 @@ in `internal/monitoring/monitor_host_agents_test.go` `internal/monitoring/issue1595_collection_trust_test.go` (`TestAgentDiskHistoryFollowsTheSerialItsUnraidRowReports`). +### Unraid temperatures chart agent disk history + +`internal/monitoring/monitor_agents.go` changed only so the SMART history +writer charts the temperature a host agent's disk shows. A SMART row without +its own reading shows the one the host's Unraid inventory reports for the disk, +when that inventory row describes the disk (`HostSMARTDiskTemperature`), and a +disk only the Unraid inventory reports, +such as a member the agent's SMART collection skips under `--disk-exclude`, +shows that row's reading (`HostUnraidDiskTemperature`). Both are now written +as `smart_temp` under the key the disk's metrics target reads +(`HostSMARTDiskMetricID`, `HostUnraidDiskMetricID`); a reading the disk does +not show as collected, a spun-down disk's or an expired host's, is not. Agent +registration, enrolment, install, update, removal, report identity and disk +source IDs are unchanged. Focused proof lives in +`internal/monitoring/monitor_host_agents_test.go` +(`TestApplyHostReportChartsUnraidTemperatureOfDiskWithoutSMART`) and +`internal/monitoring/issue1595_collection_trust_test.go` +(`TestAgentDiskChartsTheUnraidTemperatureItShows`). + ### Windows braced MachineGuid does not abort agent startup The Windows unified agent resolves host information through gopsutil's combined @@ -8953,3 +8990,11 @@ like any other write instead of always reaching its handler. Agent command websocket handshakes to `/api/agent/ws` are HTTP/1.1 `GET` requests, the only transport its gorilla upgrader serves, and still connect on a demo instance. No agent registration, enrolment, install, update or removal path changed. + +### Demo mode hides the pprof routes + +`internal/api/demo_mode_operations.go` changed only so the demo guard on a +`DEMO_MODE` instance answers `/debug/pprof` and every path below it with `404` +for every method. No agent route lives under `/debug/`, so agent registration, +enrolment, install, update, removal, report ingest and `/api/agent/ws` +handshakes are unchanged on a demo instance. diff --git a/docs/release-control/v6/internal/subsystems/ai-runtime.md b/docs/release-control/v6/internal/subsystems/ai-runtime.md index 08be24a5c8..bc4f13bd3f 100644 --- a/docs/release-control/v6/internal/subsystems/ai-runtime.md +++ b/docs/release-control/v6/internal/subsystems/ai-runtime.md @@ -128,6 +128,25 @@ on component mount. Closing clears the registered input so later keyboard commands cannot target a detached composer. A handoff must leave Escape and keyboard input in Assistant, not the underlying alert search. +### Process-wide memory ID sequences + +`internal/ai/memory` mints incident, incident-event, remediation-record and +context-memory IDs from package-level sequences that every store in the process +shares. Multi-tenant monitoring gives each tenant monitor its own +`IncidentStore`, and their alert checks call `RecordAlertFired` from separate +goroutines, while a store's mutex covers only its own state. Each sequence is +therefore an `atomic.Int64` minted through `nextSequencedID` in +`internal/ai/memory/ids.go`, never a plain package counter. The ID shape stays +`-`, but `n` no longer wraps at 1000. The old +`counter%1000` suffix repeated an ID whenever one generator minted more than a +thousand in a wall-clock second, and incident notes resolve an incident by ID +while timeline merges deduplicate events by ID. Within one process IDs are now +unique by construction. Across restarts only the timestamp separates them, so a +repeated local second (DST fall-back or a backward clock step) can still meet a +restarted sequence's suffix. Persisted IDs and the stored format are unchanged. +`internal/ai/memory/ids_test.go` mints from every generator concurrently and +records alerts into two tenant stores concurrently; run it under `-race`. + ### Canonical incident-history queries Incident context reads `IncidentStore.QueryIncidents`, whose page contains @@ -7516,6 +7535,18 @@ prompt budget. Reporting is therefore an additive consumer of AI runtime, not a new ownership boundary, and the narrator/findings surfaces inherit the same governance the rest of the canonical AI runtime already enforces. +Report narration is caller-requested spend, so on the single-resource HTTP +route only `POST /api/admin/reports/generate` reaches it (fleet reports, from +`POST .../generate-multi` or a report schedule, use the fleet narrator). The +reporting narrator resolver goes through `AISettingsHandler.GetAIService`, +which constructs a tenant service on first use and can then list provider +models or start background discovery. A GET single-resource report never calls +that resolver: +it narrates deterministically and reads Patrol findings through +`AISettingsHandler.ExistingAIService`, which returns only a service that is +already running and never constructs one +(`TestAISettingsHandler_ExistingAIServiceNeverConstructs`, and through the +router's real wiring `TestContract_SingleReportGETDoesNotConstructTenantAIService`). The same canonical AI runtime now also owns the fleet-level report narrative through `report_fleet_narrator.go`. `Service` implements diff --git a/docs/release-control/v6/internal/subsystems/alerts.md b/docs/release-control/v6/internal/subsystems/alerts.md index 40604c2962..513b44c27d 100644 --- a/docs/release-control/v6/internal/subsystems/alerts.md +++ b/docs/release-control/v6/internal/subsystems/alerts.md @@ -1114,6 +1114,43 @@ history and affirmative recovery. This is source-fixture lifecycle proof, not an installed registry/destination result or a diagnosis of a reporter's daily resolve/reopen cycle. +### A settings save keeps the alert settings it did not send + +`PUT /api/alerts/config` applies the body to the stored configuration through +`Manager.ApplyConfigUpdate` and `ApplyAlertConfigUpdate` in +`internal/alerts/config/update.go`: a top-level key the client sent replaces +that setting whole, a key it left out keeps the stored value, and the result +is normalized as any update is. The thresholds page has no control for +flapping detection (`flappingEnabled` and its window, threshold and cooldown), +alert TTL cleanup (`maxAlertAgeDays`, `maxAcknowledgedAgeDays`, +`autoAcknowledgeAfterHours`) or `customRules`, so it never sends them. The +handler used to decode the body into a zero `AlertConfig`, so every thresholds +save turned flapping detection and TTL cleanup off, reset the flapping tuning +to its defaults, dropped custom rules, and persisted that. The page's +`flapping`, `aggregation` and `ioNormalization` payload objects are not +`AlertConfig` fields and never enabled anything. A stored config keeps what it +holds, including values an earlier save zeroed; nothing rewrites them. + +The body is decoded on its own, so sent keys mean exactly what the decoder +makes of them, duplicates and case-folded keys included. Unsent fields are +copied from a JSON round trip of the stored config, so the result shares no +maps or slices with it. `ApplyConfigUpdate` reads, merges and applies under +the manager lock, so two concurrent partial saves cannot revert each other in +memory, and returns a JSON-cloned snapshot so that persistence, which +normalizes the config it is handed in place, never writes into the live +config's maps. + +`internal/alerts/config/update_test.go` covers unsent, explicit-off, +case-folded, duplicate-key, malformed and full-body updates, a sent object +replacing its stored value whole (a sent `truenasDiskDefaults: {}` does not +inherit the stored temperature) and the no-sharing rule. +`TestUpdateConfigKeepsSettingsAThresholdsSaveDidNotSend` and +`TestApplyConfigUpdateKeepsConcurrentPartialUpdates` in +`config_validation_test.go` run saves through the manager, and +`TestUpdateAlertConfig_KeepsStoredValuesForUnsentKeys` in +`internal/api/alerting/alerts_test.go` drives the handler against a real +manager. + ### Confirmed empty storage is recovery evidence Static storage capacity evaluation must admit a zero usage observation when @@ -1997,6 +2034,61 @@ alert. `shouldPreserveAlertOutsideNodeCleanup` now keeps any alert carrying `internal/alerts/alerts_test.go` pins both signals, and its sibling pins that a guest alert on a removed Proxmox node is still removed. +### Agent disk temperature alerts judge the reading the disk shows + +`CheckHost` judges each agent disk on the temperature, collection state and +disk type the disk shows, read by `unifiedresources.HostDiskTemperatureReadings` +from the same adapters the registry builds the disks with. A SMART row shows its +own reading or, when the agent's probe returned none, the one the host's Unraid +inventory reports for that disk (`HostSMARTDiskTemperature`), and keeps its +`agent:/disk_temp:` resource. An Unraid inventory row whose disk +key no SMART row carries, a disk only the inventory lists, shows that row's +reading (`HostUnraidDiskTemperature`) and its own transport as disk type, under +its kernel block device token, the resource a plain Linux SMART row for the +same device uses. A SMART row and an Unraid row with one key are one registry +disk even under different device labels, such as a controller member +(`0 [megaraid,0]`) and its Unraid device (`sda`), and alert once, under the +SMART row's resource. When smartctl and Unraid name +one device with different serials the registry shows two disks on it, and the +device's one alert judges the reading that stands highest against its own disk +type's threshold (`judgedHostDiskTemperature`): over its trigger first, then +inside its recovery band, then the larger margin over the trigger, with a tie +keeping the reading listed first. +Before this `CheckHost` read only the SMART row's own field, while the Physical +Disks Temp cell, the Running Hot verdict ("at or above its alert threshold") +and Patrol judged the Unraid reading against the same trigger, so such a disk +could read as at its alert threshold with no alert. Unraid's own warning and +critical disk temperature notifications do not make this redundant: Pulse +already alerts on the smartctl readings and array state of the same hosts, +which Unraid also notifies on, and a user who routes alerts through Pulse sets +these thresholds here. + +Only a reading collected now fires or resolves +(`HostDiskTemperatureReading.Collected`: no standby row, and +`diskinventory.TemperatureCollected`), so a spun-down disk holds the alert +like a listed disk without a temperature. A disk only the +inventory lists whose device matches the agent's `--disk-exclude` patterns +(`fsfilters.MatchesDeviceExclude`) raises nothing, and an alert it raised before +the exclusion resolves on the next report (`clearHostDiskTemperatureResources`), +whatever the SMART list holds, unless another listed reading uses the same +resource: `docs/UNIFIED_AGENT.md` documents exclusion as +removing the disk from monitoring, and linked Proxmox hosts already judge an +excluded disk healthy. A disk only the inventory lists is also judged in a +report whose SMART list is empty, because the disk shows the inventory's +reading in that report. The absence cleanup above still runs only on a +non-empty SMART list, and such a report neither counts toward nor restarts a +disk's three, so a disk that leaves the inventory while every report's SMART +list stays empty keeps its alert until the 24-hour stale-alert sweep. +`TestCheckHostJudgesTheUnraidTemperatureADiskShows` and +`TestCheckHostHoldsUnraidDiskTemperatureAlertUntilAReadingOrDeparture` in +`internal/alerts/host_unraid_lifecycle_test.go` pin the fallback, the +inventory-only disk, the empty SMART list, two disks on one device (also of +different types), a controller member under another device label, standby, +exclusion, the disk type +and host override thresholds, the holds, the handover to a SMART reading, +departure, and the exclusion clear with and without SMART rows and beside a +listed SMART row. `TestJudgedHostDiskTemperatureOutranks` pins the ranking. + ### Configured flapping thresholds remain reachable Every accepted positive `FlappingThreshold`, including values above ten, must diff --git a/docs/release-control/v6/internal/subsystems/api-contracts.md b/docs/release-control/v6/internal/subsystems/api-contracts.md index 64978f4e99..b0b7fb977f 100644 --- a/docs/release-control/v6/internal/subsystems/api-contracts.md +++ b/docs/release-control/v6/internal/subsystems/api-contracts.md @@ -4039,12 +4039,16 @@ the authoritative analysis outcome. `/api/settings/ai/update` may save a selected Patrol provider/model even when that model is not ready for tool-backed Patrol execution, but it must echo `patrol_readiness` with stable `cause` metadata and execution routes - must continue to fail closed before model calls. Frontend Patrol settings + must continue to fail closed before model calls. Frontend settings consumers must surface that saved-but-not-ready response as a saved - configuration issue with the echoed provider, model, cause, and summary - instead of reporting the successful save as a failed save or hiding the - readiness blocker behind a generic notification; and the structured - investigation-record contract, so unified findings may + configuration issue rather than a failed save, and never as inline failure + state: every Pulse Intelligence settings page (Provider & Models, Patrol, + Assistant, Service context) raises a warning notification naming the echoed + summary, provider, and model, while the Patrol page's on/off save raises a + fixed not-ready warning and leaves the diagnosis to the page's own readiness + surfaces, such as the readiness banner that names the summary, provider, + and model while the header offers `Fix setup` when Patrol is active; and + the structured investigation-record contract, so unified findings may expose `investigation_record` only through the shared `aicontracts.InvestigationRecord` payload shape, with frontend API types and backend contract tests updated in the same slice as any field change. @@ -7137,12 +7141,14 @@ introducing VMware-only mention payloads or provider-local inventory reads under `/api/vmware/*`. Runtime-specific container/app mentions remain shared unified-resource mentions as well; VMware network inventory does not create a provider-local mention family. -That same `/api/ai/chat` payload boundary owns per-request execution-mode -overrides. Dashboard Pulse Brief and other scoped handoffs may include -`autonomous_mode:false` on the chat request to force approval-required command -execution for that exchange, but the transport must treat the field as a -request override only and must not mutate the user's persistent AI control -setting. +That same `/api/ai/chat` payload boundary owns request-local execution mode, +and the server sets it. `ChatRequest` in `internal/api/ai_handler.go` has no +`autonomous_mode` field, so the `autonomous_mode:false` that scoped handoffs +such as an explain-this-issue request still send is dropped at decode, and the +handler passes approval-required mode into the chat service for every +exchange; an Autonomous control level runs as Controlled for that request. +That clamp is request-local and must not mutate the user's persistent AI +control setting. That same chat transport boundary owns new-session anchoring. When the request omits `session_id`, the handler may generate and stream a session ID immediately so the browser can anchor the visible turn, but that generated ID is @@ -7196,7 +7202,7 @@ not serialize the model-only `handoff_context`, runtime failure detail, action preflight/result bodies, remediation descriptions, raw commands, or approval command payloads, and it must not preserve Patrol-authored next-step recommendation fields from legacy handoffs. -Patrol finding handoffs are stricter than ordinary chat requests: when a request +Patrol finding handoffs keep that clamp on their own path: when a request carries a non-empty `finding_id` or resolves to model-only Patrol briefing, resource, or action context, `internal/api/ai_handler.go` must clamp the request-local autonomous mode to false even if the caller supplied @@ -7653,6 +7659,23 @@ read-only demo account. That same hidden read-side boundary includes `GET` and `HEAD` reads for `/api/admin/users` and manual discovery at `/api/discover`; public demo mode may block writes generically, but it must not reveal that admin-user inventory or manual-discovery read routes exist. +The same policy hides the Go runtime profiling family for every method: on a +`DEMO_MODE` instance the demo guard answers `/debug/pprof`, `/debug/pprof/` +and every path below it with `404`, ahead of route authentication. +`registerDebugRoutes` gates those handlers on `RequireAdmin` plus a +`settings:read` scope that session logins skip, so a demo that signs visitors +in as the configured admin would otherwise hand every visitor goroutine and +heap dumps, the process command line, and CPU profiles, execution traces and +forced garbage collection that cost the whole process. `PULSE_PPROF_DISABLED` +still removes the routes on any instance. +`TestPublicDemoAdminOperationsPolicyHidesEveryDebugRoute` fails when a literal +route the parsed router files register under `/debug/`, with or without a +method prefix, escapes that policy for any common method. +`TestContract_DemoModeHidesPprofFromDemoAdminSession` in +`internal/api/contract_test.go` signs in as that admin through the full +router, proves the same session still reads `/debug/pprof/cmdline` below the +demo guard, and gets `404` through it for eight pprof paths with `GET`, +`HEAD`, `OPTIONS` and `POST`. That generic write block keys on the request method alone. The demo middleware admits `GET`, `HEAD` and `OPTIONS` plus the login, logout, AI execute and OIDC login/callback allowlist; HTTP/1.1 websocket handshakes for @@ -9668,6 +9691,32 @@ narrator must fail closed: nil provider, parse failure, timeout, or empty response causes the engine to fall back to the heuristic narrative without surfacing the AI failure to the caller, so reporting is never blocked by AI availability. +Only `POST /api/admin/reports/generate` may hand the tenant's AI narrator to +the engine, because a narrated PDF makes a paid provider call and appends a +`cost.UsageEvent` to the cost ledger. The single-resource handler accepts +`POST` with a JSON body (the `generate-multi` decoding rules: 1MB cap, unknown +fields and trailing payload rejected, through `decodeReportingRequestBody`) and +`GET` with the same fields as query parameters, and answers any other method +with `405` and `Allow: GET, POST`. GET never calls `resolveNarrator`, whose +`GetAIService` can construct a tenant AI service (and with it list provider +models or start background discovery): it narrates through `getReportNarrator`, +the heuristic summary whose `Disclaimer` replaces the "Configure Pulse +Assistant" tip with a note that AI narration needs POST, and reads Patrol +findings through `SetExistingFindingsResolver`, which `router.go` backs with +`AISettingsHandler.ExistingAIService` so only an already running service is +used. GET passes the demo-mode guard and the CSRF check, and a SameSite=Lax +session cookie rides a cross-site top-level GET navigation, so a link must not +be able to spend AI budget. The reporting catalog advertises the transport as +`performanceReport.singleResourceMethod` (`POST`), and the settings UI follows +it. `generate-multi` and the manual schedule run endpoint +(`POST /api/admin/reports/schedules/{id}/run`) were already POST-only; due +schedules run from the server's scheduler, not an HTTP request. +`TestContract_SingleReportAINarrationRequiresPOST` pins the rule on a real +engine: GET, HEAD, PUT and DELETE make no narrator call and no AI service +resolution, while POST makes one of each. +`TestContract_SingleReportGETDoesNotConstructTenantAIService` drives +`Router.wireReportingAIResolvers` with an uncached tenant: GET constructs no +tenant AI service and POST resolves one. Report alerts keep the alert engine's resolution. The node, VM and container enrichers in `internal/api/metrics_reporting_handlers.go` build every row through `reportActiveAlertInfo` and `reportResolvedAlertInfo`, @@ -10293,6 +10342,22 @@ boundaries. Alert evaluation after that save remains alerts-owned and must use the persisted trigger/recovery pair plus a valid derived critical threshold; the API must not invent a TrueNAS-specific threshold sidecar. +`PUT /api/alerts/config` replaces only the top-level keys the body carries +and keeps the stored value of every key it leaves out, so a client that does +not know a setting cannot turn it off by leaving it out: the thresholds page +has no flapping-detection or alert-TTL controls, and an older browser bundle +or API script predates later fields. A sent key replaces its stored value +whole, so `"overrides": {}` removes every override rather than merging, and +the merged config then goes through the same normalization as any update (for +example, activation state is never cleared). The read, merge and apply run +under the alert manager's lock, so two concurrent partial saves cannot revert +each other in memory; the handler runs one save at a time and persists a +snapshot of the applied config, so successful saves through this endpoint +reach `alerts.json` in the order they were applied. A body that is not a JSON object, `null` included, is a 400 and changes +nothing. `TestUpdateAlertConfig_KeepsStoredValuesForUnsentKeys` +and `TestUpdateAlertConfig_RejectsNonObjectBody` in +`internal/api/alerting/alerts_test.go` prove the transport. + ### Alert intent and UDP availability transport `GET /api/alerts/intent-policies` and diff --git a/docs/release-control/v6/internal/subsystems/deployment-installability.md b/docs/release-control/v6/internal/subsystems/deployment-installability.md index 3af74e0266..14ef915a65 100644 --- a/docs/release-control/v6/internal/subsystems/deployment-installability.md +++ b/docs/release-control/v6/internal/subsystems/deployment-installability.md @@ -6101,6 +6101,14 @@ summaries whose scope, format, and attachments the server fixes. The delivery surface (`/api/admin/reports/schedules`) and tenant-local storage are unchanged. +### Provider docs name POST for single-resource reports + +`docs/MSP.md` and its shipped copy now give `POST /api/admin/reports/generate` +for per-client single-resource reports and say a `GET` with query parameters +still returns the report without the Pulse Assistant narrative, matching the +server, which lets only `POST` narrate. Install, upgrade and provider bundle +delivery are unchanged. + ### Convergence log-reader compatibility (5 September 2026) Scheduled reconciliation run 33939926237 failed when `gh api` refused terminal diff --git a/docs/release-control/v6/internal/subsystems/frontend-primitives.md b/docs/release-control/v6/internal/subsystems/frontend-primitives.md index bd363c2cab..018ed65630 100644 --- a/docs/release-control/v6/internal/subsystems/frontend-primitives.md +++ b/docs/release-control/v6/internal/subsystems/frontend-primitives.md @@ -294,6 +294,14 @@ expose only a raw reason. A same-VM fresh observation withdraws the notice and updates values without remounting the row. Permission, disabled-agent and actual unavailable cases retain their own explanations. +When a linked Pulse agent's filesystems fill in for a VM with no Proxmox guest +filesystems (`disksFromAgent`), the reading follows that agent instead of the +Proxmox reason: current while it reports, last known once it is stale, and +unavailable once the guest is stopped. `getWorkloadGuestDiskRead` in +`workloadGuestPresentation.ts` owns that rule for the workload row's disk cell, +the guest drawer's filesystem rows and its History, so the three cannot +disagree about one guest; `workloadGuestPresentation.test.ts` pins it. + Verification: `workloadGuestPresentation.test.ts`, the mounted `GuestRow`, `StackedDiskBar` and `GuestDrawerOverview.filesystems` regressions, and `browser-tests/guest-disk-deferral.cjs` cover the fixed reasons, retained/no @@ -3023,9 +3031,9 @@ Agent`), with the plain-language source phrase available through accessible than receiving an unfiltered copy of the estate. Opaque unified-resource ids are not part of that visible search vocabulary and must not retain a node when the normalized guest table has no corresponding match. -8. Keep summary interaction identity on one shared helper. Pages that expose row-hover, group-hover, or route-focus-driven summary emphasis, today the Workloads and Storage tables through `useSummaryPageInteractionState` in `frontend-modern/src/components/shared/summaryTableFocus.ts`, must derive page/group/entity scope through `frontend-modern/src/components/shared/summaryCardInteraction.ts` rather than letting each surface read `hovered || focused` from its own page-local ID source. Hovering or pinning a storage pool-group header, or a workload group header on a surface that enables inline group drawers (the Proxmox and VMware surfaces pass `groupNodeDrawerMode="disabled"` today), must scope the matching rows through that same shared contract instead of forking a page-local summary filter path. No summary card or chart strip renders today; one that returns must consume that same resolved scope, and the chart under the pointer must promote its series into the shared active entity through the bridge's `setChartHoverSync` input instead of keeping a chart-local hover island. Recovery is explicitly outside this interaction dialect: its retired posture-card strip must not return with row/group/chart hover behavior without a separate governed product decision. +8. Keep summary interaction identity on one shared helper. Pages that expose row-hover, group-hover, or route-focus-driven summary emphasis, today the Workloads and Storage tables through `useSummaryPageInteractionState` in `frontend-modern/src/components/shared/summaryTableFocus.ts`, must derive page/group/entity scope through `frontend-modern/src/components/shared/summaryCardInteraction.ts` rather than letting each surface read `hovered || focused` from its own page-local ID source. Hovering or pinning a storage pool-group header, or a workload group header on a surface that enables inline group drawers (the Proxmox and VMware surfaces pass `groupNodeDrawerMode="disabled"` today), must scope the matching rows through that same shared contract instead of forking a page-local summary filter path. No summary card or chart strip renders today, and the bridge carries no chart-hover input; one that returns must consume that same resolved scope, and the chart under the pointer must enter its hovered series into `resolveSummaryScopeState` as a preview input that outranks row hover instead of keeping a chart-local hover island. Recovery is explicitly outside this interaction dialect: its retired posture-card strip must not return with row/group/chart hover behavior without a separate governed product decision. 9. Keep any page summary page-scoped when table rows enter contextual focus. No infrastructure, workloads, or storage summary cards render today. If one returns, route-backed row selection may add a focused label and shared series emphasis, but the summary must continue to render the page-level series set instead of collapsing down to the selected row or replacing the global trend view with row-local empty states. -10. Keep contextual row focus on the shared primitives. Same-route table drill-ins must reuse `frontend-modern/src/components/shared/contextualFocus.ts` for local scroll preservation, off-screen reveal, inline-detail lookup, and deliberate inline-detail reveal, and must resolve the active series through `frontend-modern/src/components/shared/summaryCardInteraction.ts` behind `frontend-modern/src/components/shared/summaryTableFocus.ts`, instead of rebuilding page-local `Set` filters, focused-label scans, drawer-aware scroll math, or ad hoc scroll restoration in each surface. +10. Keep contextual row focus on the shared primitives. Same-route table drill-ins must reuse `frontend-modern/src/components/shared/contextualFocus.ts` for local scroll preservation, off-screen reveal, inline-detail lookup, and deliberate inline-detail reveal, and must resolve the active series through `frontend-modern/src/components/shared/summaryCardInteraction.ts` behind `frontend-modern/src/components/shared/summaryTableFocus.ts`, instead of rebuilding page-local `Set` filters, focused-label scans, drawer-aware scroll math, or ad hoc scroll restoration in each surface. `frontend-modern/src/components/shared/summaryTableFocus.ts` measures and scrolls rows only for deliberate focus, a focused row's inline detail or a pinned group header, so it must not keep window-level scroll or resize listeners or re-measure the hovered row on hover changes and scroll frames; an off-screen affordance for the active row needs a rendered consumer before that cost returns. 11. Keep summary-linked table row emphasis on the shared primitive contract. Workloads, infrastructure, and storage rows that mirror the active summary entity must expose that state through `data-summary-row-active` and let the shared presentation in `frontend-modern/src/index.css` render the row emphasis, rather than carrying page-local sky or blue fill classes inside each row renderer. Group-scoped preview and pin must use that same shared presentation boundary: child rows that belong to a hovered or pinned summary group should expose `data-summary-group-member-active="preview|pinned"` so the block-level emphasis stays subtle, consistent, and reversible instead of each table inventing its own outline, badge, or full-strength fill treatment. Static grouped row headers on workloads, infrastructure, storage, recovery, and future grouped tables must use `frontend-modern/src/components/shared/groupedTableRowPresentation.ts` plus the `.grouped-table-row` CSS contract in `frontend-modern/src/index.css`, rather than rebuilding local `bg-surface-alt` variants with subtly different light/dark behavior or page-local left-accent markers. That shared grouped-table primitive owns the subgroup cell padding, typography, small metadata, and badge treatment as well as the row background token, so a future adjustment to the subgroup visual language changes every grouped product table from one owner. Inline table detail rows on platform, workload, and infrastructure tables must compose `frontend-modern/src/components/shared/InlineDetailTableRow.tsx` for the full-width row, surface-alt cell, detail padding, and row-click containment instead of rebuilding page-local `TableRow` / `TableCell` / `div` shells around each drawer. Storage-backed reusable row presenters under `frontend-modern/src/features/storageBackups/` must also keep row height and alert accents on class/data-attribute presentation instead of runtime inline style maps, so the shared table contract stays CSP-safe on both steady-state and alert-highlighted routes. 12. Keep retained-value data loading honest at the ownership boundary. Helpers that prevent a feature surface from falling through the app-level Suspense @@ -3156,33 +3164,30 @@ Agent`), with the plain-language source phrase available through accessible operator action or selected Patrol context, uses distinct monitor labels, and does not become a nested card, generic dashboard strip, trust summary, or duplicate Patrol empty-work list. The - Patrol schedule and model drawer is part of that shared - feature-presentation boundary: it must stay viewport-bounded, expose an - accessible dialog label, keep the four-level control policy on the default - Patrol header, and keep provider model, schedule, trigger tuning, and - readiness validation inside the secondary disclosure. Backend save rejection reasons must pass - through as inline dialog state instead of being replaced with generic toast - copy, and that advanced disclosure must open when the inline state exists. - When the failure includes - Patrol readiness context, the inline state must expose the provider, model, - and readiness summary next to a direct provider-settings action instead of - hiding that diagnosis behind Assistant alone. The provider-model selector in - that popover must stay bound to the shared runtime settings/model catalog - even when the popover mounts after async catalog loading, but the full - catalog must stay behind an explicit change action so the default advanced - drawer leads with the current effective model summary rather than a raw - provider route list. A saved direct-provider Patrol model still renders as - that model instead of visually falling back to the default selection. - Successful provider-model saves that return a not-ready Patrol - readiness snapshot must use that same inline surface with `needs attention` - wording, and any Assistant handoff from that state describes a saved - configuration issue rather than a failed save. When governed fixes are locked, the same Patrol state - owner must clear stale full-mode unlock state before persisting the - monitor-only autonomy payload, so disabled paid controls cannot leak stale - permission into a save. If that inline state opens Assistant, the Patrol - feature must hand off - a source-named, model-only briefing and close the popover so the shared - Assistant drawer is not visually hidden behind feature chrome. When a + Patrol page keeps the four-level control policy on its header and has no + schedule and model drawer: schedule, trigger tuning, and the Patrol model + override live on Settings > Pulse Intelligence > Patrol, and providers and + the shared default model on Settings > Provider & Models. Those saves + report through the shared notification toasts, not inline state: a + rejected save raises an error toast carrying the backend's concrete reason + rather than generic fallback copy, and a saved response whose Patrol + readiness is not ready or degraded raises a warning toast naming the + readiness summary, provider, and model instead of a success toast. On the + Patrol page, an on/off save whose response reports Patrol not ready raises + a fixed warning toast, and a failed Patrol mode save raises an error toast + and restores the previous mode; neither renders inline save-failure state + or offers an Assistant handoff. If a Patrol settings popover or inline + save-failure state returns, it must stay viewport-bounded, expose an + accessible dialog label, bind any model selector to the shared runtime + settings and model catalog, show readiness context (provider, model, + summary) next to a direct provider-settings action rather than behind + Assistant alone, and describe a saved but not ready response as a saved + configuration issue rather than a failed save. An Assistant handoff from it + must send a source-named, model-only briefing and close the popover so the + shared Assistant drawer is not visually hidden behind feature chrome. When + governed fixes are locked, the Patrol state owner must clear stale + full-mode unlock state before persisting the monitor-only autonomy payload, + so disabled paid controls cannot leak stale permission into a save. When a restored Patrol assessment session is attached, the shared Assistant drawer empty state must stay aligned with that source-named briefing and must not render generic @@ -4840,6 +4845,13 @@ performance-report transport (`/api/reporting` and `/api/reporting/generate-mult so the reporting panel does not go dead on mixed-version installs, but that compatibility path is intentionally report-only and must not invent the newer catalog-owned VM inventory export surface. +`reportingPanelModel.ts` picks the single-resource transport from the catalog's +`performanceReport.singleResourceMethod`: `POST` sends the request fields as a +JSON body, which is the only transport the server lets narrate a PDF with Pulse +Assistant, and `GET` sends the same fields as query parameters. +`reportingCatalogModel.ts` reads a missing field as `GET`, the only method a +backend that predates the field accepts, and the legacy compatibility catalog +names `GET` for `/api/reporting`. Fleet reports stay `POST` on every backend. `ReportingPanel.tsx` must therefore treat `vmInventoryExport` as optional when it renders a governed reporting catalog. A legacy compatibility catalog with no inventory export still owns a valid enabled reporting surface and must continue @@ -5140,10 +5152,10 @@ KPI, problem-resource, or card shells. Workload-table and guest-row fallback copy that lives under `frontend-modern/src/components/Workloads/` must keep using `frontend-modern/src/utils/workloadEmptyStatePresentation.ts` and `frontend-modern/src/utils/workloadGuestPresentation.ts`. New route-level empty -states, tone mapping, or compact issue copy must extend the shared -`emptyStatePresentation`, `semanticTonePresentation`, and -`problemResourcePresentation` helpers instead of reviving deleted -dashboard-only KPI, metric, storage, recovery, or trend presentation helpers. +states or tone mapping must extend the shared `emptyStatePresentation` and +`semanticTonePresentation` helpers instead of reviving deleted dashboard-only +KPI, metric, storage, recovery, problem-resource, or trend presentation +helpers. That shell must also stay passive with respect to data ownership: future overview trend cards may render summary-range controls and operator-facing empty or error copy only after they have a governed owner, and they must not @@ -6449,20 +6461,6 @@ rollups, points, or facets payloads arrive, it must keep that selected platform present in the option set so the shared `LabeledFilterSelect` shows the owned value immediately instead of flashing back to `All Platforms` until recovery data warms. -`frontend-modern/src/utils/problemResourcePresentation.ts` now also belongs to -that same dashboard overview boundary so the problem-resource severity contract -stays shared with `ProblemResourcesTable.tsx` instead of floating as an -unowned helper. -Problem-resource table readability belongs to that same owner. Repeated rows -may collapse only when they share the same governed display label, resource -type, and problem signal; the header count and Pulse Brief counts must continue -to represent the underlying affected resources, and grouped links must route to -the broad owning surface rather than inventing a synthetic resource target. -Problem Resources and Pulse Brief wording must not amplify generic -status-shaped names such as `storage (offline)` into first-viewport prose or -grouped-row sublabels; when the resource name is only a type plus status, the -surface should summarize the type-level issue in operator language instead of -repeating raw backend-shaped labels. The retired dashboard action queue must not be reintroduced as a compact Patrol or infrastructure issue panel. Patrol-owned runtime findings remain governed by `frontend-modern/src/utils/aiFindingPresentation.ts` and their diff --git a/docs/release-control/v6/internal/subsystems/monitoring.md b/docs/release-control/v6/internal/subsystems/monitoring.md index 67a6f98337..d213a427c3 100644 --- a/docs/release-control/v6/internal/subsystems/monitoring.md +++ b/docs/release-control/v6/internal/subsystems/monitoring.md @@ -2699,7 +2699,18 @@ truthfulness, not native thaw, containing-release or workload acceptance. ingestion so canonical host IDs stay stable across restarts (see the unified-resources contract's durable identity-pin obligation). Rebuild paths added to the adapter must keep that persistence step; ephemeral - snapshot-bridge adapters stay read-only. + snapshot-bridge adapters stay read-only. The rebuild, the live + supplemental refresh and the read-state overlay ingest records with the + adapter's configured stale thresholds, because record ingest joins + operator links and the freshness gate of every metric merge, a link's or + a source's into an existing row, reads them (unified-resources contract, + "Operator links reach record-ingested resources"). Regression + coverage: + `TestMonitorAdapterJoinsLinkedRecordsWithConfiguredStaleThresholds` and + `TestMonitorAdapterSourceMergesUseConfiguredStaleThresholds` in + `internal/unifiedresources/monitor_adapter_read_state_test.go` and + `TestManualLinkToSupplementalGuestHoldsWithAndWithoutContinuity` in + `internal/monitoring/issue1913_host_continuity_test.go`. 11. The TrueNAS provider projects pools with `Storage.Topology` fixed to `pool` and the ZFS data vdev layout in `Storage.VDevLayout`. The @@ -4514,6 +4525,13 @@ disk usage when the last VM snapshot is still recent guest-agent truth rather than an already carried-forward fallback. That keeps transient guest-agent or status-call failures from regressing a VM back to misleading allocated-disk data while still avoiding indefinite replay of stale disk summaries. +The previous VM that `previousVMFromView` builds from the unified read state +carries disk usage only when it is Proxmox's own reading. When a manually +linked Pulse agent's disk filled in because Proxmox had no guest filesystems +(`VMView.DiskFromLinkedAgent`), carrying it would relabel the agent's value as +a Proxmox `prev-` read, and with Proxmox outranking the agent on guests that +copy would then freeze over the agent's live disk. +`TestPreviousVMFromViewKeepsLinkedAgentDiskOutOfProxmoxCarry` pins both owners. That compatibility boundary also applies to historical snapshot labels that may still exist in tests, live in-memory state, or pre-canonical diagnostic paths: legacy aliases such as `rrd-available`, `rrd-data`, `node-status-available`, @@ -4823,8 +4841,10 @@ timeline instead of creating a second drawer-only or mock-only disk history path. The SMART-resolved id is the serial the disk resource carries: when a SMART row reports none, `HostSMARTDiskMetricID` takes the one the host's Unraid inventory -reports for that disk, as the unified-resources adapter does, unless several -SMART rows share the disk's device path, and `hostDiskIOMetricResourceID` keys +reports for that disk, as the unified-resources adapter does, unless the Unraid +row does not describe the disk (`unraidDiskDescribesSMARTRow`: a controller +member, or several SMART rows on the disk's device path), and +`hostDiskIOMetricResourceID` keys a device with no non-standby SMART row by its Unraid serial (`HostUnraidDeviceMetricID`, which refuses a device several Unraid rows name) before the linked Proxmox node's disks and the `:` fallback. @@ -4835,6 +4855,21 @@ their counter under no member. Proof: `TestAgentDiskHistoryFollowsTheSerialItsUnraidRowReports` and `TestHostDiskIOMetricResourceIDFallbacks`. +The agent's `smart_temp` series holds the reading the disk resource shows, +not only the SMART row's own. `writeHostSMARTMetrics` writes +`HostSMARTDiskTemperature`, which falls back to the host's Unraid inventory +reading when the row has none, and `writeHostUnraidDiskTemperatures` writes +`HostUnraidDiskTemperature` under `HostUnraidDiskMetricID` for each Unraid row +whose key is not one of the host's SMART rows' keys, such as a `--disk-exclude` +member, so an Unraid reading never overwrites a SMART row's own at the same +timestamp. +Unraid rows sharing a key are one disk in the registry, which shows the latest +of their readings, so the last row with a reading decides that key's single +sample. `diskinventory.TemperatureCollected` keeps a spun-down disk's or an +expired host's leftover reading out of history. +Proof: +`TestAgentDiskChartsTheUnraidTemperatureItShows` and +`TestApplyHostReportChartsUnraidTemperatureOfDiskWithoutSMART`. That same monitoring-owned disk-health boundary also includes shared storage risk assessment in `internal/storagehealth/`. When providers or host agents emit structured storage topology such as Unraid per-disk state, the shared diff --git a/docs/release-control/v6/internal/subsystems/patrol-intelligence.md b/docs/release-control/v6/internal/subsystems/patrol-intelligence.md index e5ad3e0b55..c6b2cb9fd7 100644 --- a/docs/release-control/v6/internal/subsystems/patrol-intelligence.md +++ b/docs/release-control/v6/internal/subsystems/patrol-intelligence.md @@ -1581,17 +1581,18 @@ suppress stale healthy summary headlines such as `Health A ยท 100/100` even if the last summary payload still looks healthy. Legacy `/ai` entry points must redirect into that same Patrol-owned shell rather than preserving a second canonical route. -That same browser proof now covers the Patrol control and advanced-settings -split. The advanced Patrol settings drawer must stay within the desktop -viewport, avoid duplicating the inline Patrol control policy, expose -provider/model, schedule, trigger, and user-level model checks directly, and surface -the backend's concrete license/validation reason when a settings change is -rejected instead of replacing it with a generic `Failed to save advanced -settings` toast. That inline failure no longer -opens Assistant. If it does again, the handoff is model-only explanation -context: raw command, script, credential, and provider-detail payloads stay -redacted, Assistant opens with `autonomousMode:false`, and the Patrol control -panel closes so the operator is not left behind an overlapping popover. +That same browser proof now covers where Patrol settings are saved. The +Patrol page has no settings drawer or popover: Patrol schedule, triggers, and +the Patrol model override live on Settings > Pulse Intelligence > Patrol +(`Save Patrol settings`), and providers and the shared default model on +Settings > Provider & Models. A rejected save there raises an error +notification carrying the backend's concrete license/validation reason +instead of generic fallback copy, and it offers no Assistant handoff. If a +Patrol settings popover or an Assistant handoff from a settings failure +returns, the handoff is model-only explanation context: raw command, script, +credential, and provider-detail payloads stay redacted, Assistant opens with +`autonomousMode:false`, and the popover closes so the operator is not left +behind it. Patrol-control save-failure sessions saved by earlier builds keep the compatible `handoff_metadata.kind=patrol_configuration_failure` plus only the runtime-failure boolean needed for drawer/session presentation, so they restore @@ -1637,12 +1638,13 @@ ready raises a fixed warning notification, a failed Patrol mode save raises an e notification, and neither offers an Assistant handoff. A provider or model save in Settings > Provider & Models reports the readiness summary, provider and model in its own save message. -The Patrol mode selector in that header and configuration dialog must -compose the shared `frontend-modern/src/components/shared/FilterButtonGroup.tsx` -instead of rebuilding a local active-button group. The wide default Patrol -header uses the segmented layout; the constrained configuration dialog may use -the shared prominent layout so all four mode labels remain readable without -inventing a Patrol-local selector. Patrol owns the default visible four-level +The Patrol mode selector in that header must compose the shared +`frontend-modern/src/components/shared/FilterButtonGroup.tsx` instead of +rebuilding a local active-button group. When the header shows Patrol mode +options, it renders the selector in the segmented layout inside its `Mode and +automation` disclosure; a constrained surface that hosts it may use the shared +prominent layout so all four mode labels remain readable without inventing a +Patrol-local selector. Patrol owns the default visible four-level policy presentation (`Watch only`, `Ask first`, `Safe auto-fix`, `Autopilot`), entitlement locks, and the rule that choosing the highest Autopilot level sends `full_mode_unlocked:true` while choosing any lower level clears that diff --git a/docs/release-control/v6/internal/subsystems/performance-and-scalability.md b/docs/release-control/v6/internal/subsystems/performance-and-scalability.md index 10dfe511ed..9feade4637 100644 --- a/docs/release-control/v6/internal/subsystems/performance-and-scalability.md +++ b/docs/release-control/v6/internal/subsystems/performance-and-scalability.md @@ -2424,6 +2424,12 @@ workloads and storage tables. Row hover may highlight the matching row in place, but the hot path must not auto-scroll the page or rebuild the table into a one-row filtered view on transient hover; only a deliberate row focus may reveal an off-screen row, through the shared summary-table focus bridge. +The bridge measures row position only inside that deliberate reveal. It must +not keep window-level scroll or resize listeners, or re-measure the hovered +row on every hover change and scroll frame, for an off-screen affordance that +nothing renders: the retired `Jump to row` visibility check did exactly that, +costing 128 `getComputedStyle` calls across eight row hovers and 320 across +ten scroll steps on the 12-node mock storage table, and none once removed. That same hot-path contract also owns the row-emphasis paint. Workload guest and group rows and storage pool, group, and disk rows must expose summary-linked activity through the shared `data-summary-row-active` marker and let the shared frontend @@ -2728,9 +2734,26 @@ unavailable once the guest is stopped, whether its status or the Proxmox `vm-stopped` reason says so, and Proxmox's own guest filesystems keep their Proxmox read state. The two flags ride the existing row mapping and add no query or poll; `GuestDrawer.test.tsx` and `useWorkloads.test.ts` pin -both sources. The workload table's disk cell and the drawer History still read -the aggregate disk metric and the Proxmox reason, and the agent's OS, addresses -and physical disk health carry no stale cue yet. +both sources. The workload table's disk cell (`useGuestRowState.ts`, +`GuestRow.tsx`) and the drawer History (`guestDrawerModel.ts`) judge the same +guest through the drawer rows' rule, `getWorkloadGuestDiskRead` in +`workloadGuestPresentation.ts`: a silent agent's bars carry the Last known +label, a live agent's bars stay current under Proxmox `agent-not-running`, and +History reads the aggregate disk metric as current, last known or unavailable +the same way. The rule covers the aggregate because Proxmox records no guest +disk usage without guest filesystems, so a manually linked guest then keeps the +agent's disk metric, and the poller no longer carries that agent reading +forward as Proxmox's own `prev-` read (see the monitoring contract). +`TestResourceRegistry_ManualGuestAgentLinkDiskMetricFollowsProxmoxGuestFilesystems` +pins that merge; `GuestRow.test.tsx` and `GuestDrawer.test.tsx` pin the cell and +History. On the websocket path (`resourceStateAdapters.ts`) a native Proxmox VM +facet's new guest-read outcome drops filesystems it no longer reports, a native +agent facet without `disks` drops the agent's, and the agent facet always +carries `stale`, so the rows switch source and lose the Last known label when +the agent resumes, without a reload. A stale Proxmox source +can still lose the aggregate to a live agent while its own filesystem rows +remain; such a guest stays on the Proxmox rule. The agent's OS, addresses and +physical disk health carry no stale cue yet. Guest, node, and Docker-host drawer headers follow the same frontend-primitives dependency boundary for collapse: Workloads owns which inline row is selected and the close handler, while diff --git a/docs/release-control/v6/internal/subsystems/security-privacy.md b/docs/release-control/v6/internal/subsystems/security-privacy.md index 1524faa42c..bdc77ad4f1 100644 --- a/docs/release-control/v6/internal/subsystems/security-privacy.md +++ b/docs/release-control/v6/internal/subsystems/security-privacy.md @@ -3025,6 +3025,18 @@ Removing the disconnected recorder does not alter alert, action approval or operator authority. An unrelated organization receives no default archive fallback. +### Report GETs cannot spend AI budget + +`Router.wireReportingAIResolvers` in `internal/api/router.go` wires a second +reporting resolver, `SetExistingFindingsResolver`, backed by +`AISettingsHandler.ExistingAIService`. +A `GET /api/admin/reports/generate` uses only that resolver and the +deterministic narrator, so it makes no AI provider call, writes no cost-ledger +entry and never constructs a tenant AI service. GET passes the demo-mode guard +and the CSRF check, and a SameSite=Lax session cookie rides a cross-site +top-level navigation, so AI narration of a single-resource report now requires +a CSRF-checked `POST`. No authentication, scope or session rule changed. + ### Node setup action v7 compatibility Workflow consumers pin `actions/setup-node` to diff --git a/docs/release-control/v6/internal/subsystems/storage-recovery.md b/docs/release-control/v6/internal/subsystems/storage-recovery.md index 27c4f95c7b..4737de1395 100644 --- a/docs/release-control/v6/internal/subsystems/storage-recovery.md +++ b/docs/release-control/v6/internal/subsystems/storage-recovery.md @@ -3008,6 +3008,15 @@ recovery evidence. Unremembered overflow deliveries remain auditable with `reason=capacity`; cache expiry, overflow, or restart is not a successful storage operation or a reason to delete historical events. +### Alert settings saves keep unsent keys without a new storage path + +`PUT /api/alerts/config` now applies only the top-level keys the body carries +and keeps stored values for the rest, so a thresholds save no longer turns off +alert TTL cleanup. It still writes `alerts.json` through the existing +`SaveAlertConfig` path, opens no storage or recovery route, and leaves backup +and snapshot alert settings under the same replace-what-was-sent rule as every +other key. `internal/api/alerting/alerts_test.go` pins the handler. + ### Container diagnostics shares private bootstrap transport (1 October 2026) Recovery-adjacent diagnostics now reuse the canonical complete installer @@ -4362,10 +4371,13 @@ one that returns must stay page-scoped instead of collapsing to the expanded row or replacing the page overview with row-local empty states. That same storage ownership now also governs reveal. Row hover may highlight the matching row in place, but storage hover must not auto-filter or -auto-scroll the table. Reveal belongs only to deliberate focus: a focused -pool or disk row may switch to the owning view or expand the owning group -through the bridge's `revealActiveSeries` callback, and a pinned pool-group -header that sits off-screen scrolls into view through the same bridge. +auto-scroll the table. Reveal belongs only to deliberate focus: the bridge +hands the focused pool or disk series to the `revealActiveSeries` callback, +which reopens a focused pool's collapsed owning group, and a pinned pool-group +header that sits off-screen scrolls into view through the same bridge. Focus +only ever names a row in the active pools or disks view, so the callback never +switches views; the view-switching branches that served the retired jump to +the active row are gone. That same reveal contract now also owns inline-detail expansion. When a pool or disk row is deliberately focused and its inline detail opens on the storage page, the detail row must publish the same canonical summary series ID through @@ -6253,6 +6265,19 @@ is in `applyLicensedFeatureConfigSnapshot` (`pkg/server/telemetry_licensed_features.go`), pinned by `TestApplyLicensedFeatureConfigSnapshot_CountsScheduledReportingAndProfiles`. +### Single-resource report AI narration requires POST + +`internal/api/metrics_reporting_handlers.go` now accepts `POST` with a JSON +body for `/api/admin/reports/generate` and lets only that method use the +tenant's AI narrator; `GET` keeps working with query parameters and the +deterministic summary. A narrated PDF records the same AI cost-ledger entry as +before, and only a `POST` can now produce one. A `GET` reads Patrol findings +only from an AI service that is already running +(`AISettingsHandler.ExistingAIService`), so it no longer constructs a tenant AI +service, whose construction opens that tenant's SQLite resource export store. +Report content, backup and recovery enrichment, and tenant storage are +otherwise unchanged. + ### Audit-read activity is a bounded, content-free local history `audit_read_activity.json` follows the same shape as the existing external-agent @@ -6873,6 +6898,15 @@ The performance report and reporting runtime snapshot handlers apply the same test before tabulating a disk temperature. That changes only which held value a report shows, not any storage or recovery path. +### A linked agent's stale flag and guest disk owner open no storage path + +`internal/unifiedresources/types.go` now always sends `AgentData.Stale`, so a +resumed agent clears its stopped-reporting mark in browsers that merge agent +facets field by field, and `VMView.DiskFromLinkedAgent` keeps the poller from +carrying a linked agent's guest disk forward as a Proxmox read. Both describe +which source's reading is current; no backup, retention, migration or recovery +path is added or moved. + ### Demo write guard ignores websocket upgrade headers `internal/api/demo_middleware.go` no longer exempts every request carrying @@ -6883,3 +6917,11 @@ allowlist it gets the generic demo `403`, or `404` on a hidden route, and that includes config import at `/api/config/import`. HTTP/1.1 websocket handshakes are `GET` requests and still connect. No storage, retention, backup, migration or recovery path is added or moved. + +### Demo mode hides the pprof routes + +`internal/api/demo_mode_operations.go` adds the Go runtime profiling family +(`/debug/pprof` and every path below it, every method) to the public-demo +hidden routes, so the demo guard on a `DEMO_MODE` instance answers it with +`404`. That removes a diagnostic read surface on demo instances only; no +storage, retention, backup, migration or recovery path is added or moved. diff --git a/docs/release-control/v6/internal/subsystems/unified-resources.md b/docs/release-control/v6/internal/subsystems/unified-resources.md index 7f476226af..7ed66c3921 100644 --- a/docs/release-control/v6/internal/subsystems/unified-resources.md +++ b/docs/release-control/v6/internal/subsystems/unified-resources.md @@ -1609,7 +1609,12 @@ Source freshness is a separate unified-resource status input, not a fixed global health timer. Snapshot rebuilds, resource seeding, supplemental record overlays, and cloned read-state overlays must preserve the monitoring-provided stale thresholds for Proxmox, PBS, and PMG sources so resources do not flap to -warning/degraded between successful configured poll cycles. +warning/degraded between successful configured poll cycles. A source merge +into an existing row judges metric freshness by the ingest's thresholds and +the stale pass's own threshold rule, so the merge's freshness gate never +calls a source current that the stale pass marks stale, or the reverse. The +gate decides only where no stronger rule does: a hypervisor-managed guest +keeps its platform CPU even while that source is stale. Service-discovery readiness is a unified-resource payload contract, not a drawer-local decoration. Resource list/detail payloads that expose a @@ -2303,6 +2308,16 @@ evidence of what the array was doing when the agent went quiet, but without live status colour. Where a retained rebuild percentage is shown, it reads "Rebuild was at N%" with no speed or progress bar, and a member that was not healthy names its state in its badge. +A Pulse agent manually linked into a guest carries the same signal there. +`AgentData.Stale` is always on the wire, because browsers merge agent facets +field by field (`resourceStateAdapters.ts`) and would keep a resumed agent's +omitted false as true; `TestAgentDataAlwaysSendsStale` pins it. That always +present `stale` also marks a native agent facet, so one without `disks` has +withdrawn them, and a native Proxmox VM facet's new guest-read outcome likewise +drops `disks` it no longer reports; either way the other source's filesystems +take over without a reload (`resourceStateAdapters.test.ts`). `VMView.DiskFromLinkedAgent` names the agent +as the owner of a guest's selected disk metric, which the monitoring poller +reads before it carries a Proxmox disk reading forward. For SMART disk temperatures on rows that still render, the provenance travels on `agent.sensors.smart[].collection`: the Machines temperature cell @@ -2803,7 +2818,9 @@ application resource-provider or WebSocket lifecycle. ID and drops the superseded pin row (`ApplyCanonicalIDSuccessions` in `internal/unifiedresources/canonical_id_succession.go`). Successions never fire while the old ID still belongs to a live - resource, never follow a contradicting machine key (a reinstalled + resource or to one a manual link folded into its primary (see + "Operator links reach record-ingested resources"), never follow a + contradicting machine key (a reinstalled machine mints fresh, it does not absorb the old host's operator state), and never rewrite change-journal rows. Known limitation: machine-keyless hosts whose history already merged under a collapsed @@ -2884,8 +2901,9 @@ application resource-provider or WebSocket lifecycle. `ResourceRegistry.IngestRecords` applies the same `ApplyCanonicalIDSuccessions` semantics as pin-driven successions (operator state and action audits re-key, the superseded pin row - drops, never while the old ID belongs to a live resource, journal - rows never rewritten). Regression coverage: + drops, never while the old ID belongs to a live resource or one a + manual link folded into its primary, journal rows never rewritten). + Regression coverage: `TestRegistryIngestRecordsKeepsSameHostnameSystemsDistinct` / `TestIngestRecordsSucceedLegacyHostnameScopedCanonicalIDs` / `TestIngestRecordsDoNotCompleteTrueNASIdentityFromPins` in @@ -5464,6 +5482,43 @@ A row's own serial is kept even when it is a placeholder, as the adapter keeps it, and the Unraid row then stays a disk of its own. Proof: `TestHostSMARTDiskMetricIDTakesTheSerialItsUnraidRowReports`, `TestAgentDiskHistoryFollowsTheSerialItsUnraidRowReports`. +The adapters' disk temperature rule is shared with the agent disk history +writer. `HostSMARTDiskTemperature` returns the temperature and collection state +a SMART row's disk shows: the row's own reading, or, when it has none, its +Unraid row's under `unraidDiskTemperatureStatus`. `matchUnraidDisk` finds that +Unraid row by the SMART row's usable serial anywhere in the inventory before +any device path, and a placeholder serial matches nothing. The fallback, like +the serial a row without one takes (`hostSMARTDiskSerial`), comes only from an +Unraid row that describes the row's disk (`unraidDiskDescribesSMARTRow`): one +carrying the row's usable serial, or one naming the row's device path when the +row is that whole kernel block device. A controller member, even the only one +reported, and a row sharing its path with other SMART rows are not, because +the Unraid row describes the block device as a whole. `HostUnraidDiskTemperature` +returns what a disk built from an Unraid row alone shows, and +`HostUnraidDiskMetricID` returns the key that disk's metrics target reads: its +usable serial, else its `HostUnraidDiskSourceID`, and nothing for a row without +a device, which is not ingested. The writer used to take only the SMART row's +own reading, so a temperature shown from the Unraid inventory, for a disk whose +SMART probe returned none or one excluded from SMART collection, was never +charted. Proof: `TestAgentDiskChartsTheUnraidTemperatureItShows` and +`TestHostUnraidDiskMetricIDMatchesItsDiskMetricsTarget`. +`HostDiskTemperatureReadings` lists the reading each disk of a host agent shows +for consumers that judge disk heat, the agent disk temperature alerts in +`CheckHost`: one per SMART row the registry ingests (no virtual block device), +then one per Unraid row whose disk key (`HostUnraidDiskMetricID`) no SMART row's +`HostSMARTDiskMetricID` equals, the history writer's rule, with a row without a +device skipped as the registry skips it. Rows with one key are one registry +disk whatever device labels they carry, such as a controller member +(`0 [megaraid,0]`) and its Unraid device (`sda`). When smartctl and Unraid +report different usable serials for one device the registry shows both disks, +and both are listed. Each reading is taken from the disk resource +`resourceFromHostSMARTDisk` or `resourceFromHostUnraidPhysicalDisk` builds, so +its temperature, collection state and disk type are the ones that row's disk +shows. Unraid rows on different devices that share a key and no SMART row are +one registry disk but keep a reading each, since alerts are keyed by device. +Proof: `TestHostDiskTemperatureReadingsMatchTheDisksTheRegistryShows` compares +each reading with the ingested disk, including two disks on one device and a +controller member Unraid lists under its block device. That same canonical physical-disk view must also expose source-independent host context. When a disk is API-backed rather than node-backed, typed views should fall back to canonical host identity such as `identity.hostnames` instead of @@ -5580,12 +5635,14 @@ wording stays aligned with the drawer's recent-change cards and timeline. Timeline cards in that drawer surface change metadata when it is present, so the history view preserves the richer provenance already carried by the unified-resource model instead of flattening those fields away. -The same Infrastructure resource-only links now also default through the -shared `frontend-modern/src/components/Infrastructure/ResourceChangeSummary.tsx` +The shared `frontend-modern/src/components/Infrastructure/ResourceChangeSummary.tsx` and `frontend-modern/src/components/Infrastructure/ResourceCorrelationSummary.tsx` -cards from the Patrol page, resource drawer, and problem-resource dashboard -panels, so canonical resource-filter path construction stays owned by the -shared summary cards rather than being duplicated per surface. +cards keep resource links behind an optional `buildResourceHref` input with no +default, following the 2026-05-16 cross-resource drilldown retirement above. +The resource drawer passes none, so its change and correlation labels render +as plain text; a surface that needs resource links must pass a platform-route +builder into the shared card rather than rebuilding resource-filter paths per +surface. Platform tables supply the resource-label resolver to the resource drawer through `PlatformResourceDetailTableRow`. `createPlatformResourceLabelResolver(...)` in `frontend-modern/src/features/platformPage/PlatformResourceDetailTableRow.tsx` @@ -6547,6 +6604,75 @@ full heading. This shared drawer rule applies across resource types. The 390px browser qualification checks heading fit and keyboard expansion; no data model, authorisation, action dispatch or resource admission contract changes. +### Operator links reach record-ingested resources + +Every registry ingest path applies the operator's manual links: snapshot +ingest, resource ingest and record ingest (`IngestRecords`, +`IngestRecordsWithStaleThresholds` and the read-state overlay's record pass). +Either side of a link can arrive only as a supplemental record, such as a +vSphere or TrueNAS VM with a Pulse agent inside, or an agent on a TrueNAS host, +and the monitor's rebuild ingests those records after the snapshot. While only +snapshot and resource ingest applied links, the rebuilt registry behind the +websocket broadcast and `/api/state` kept both rows. The resources API, which +seeds through resource ingest, showed them merged, and so did the published +inventory whenever an unrelated absent machine made continuity hydration +clone the registry. Mock mode builds its unified view from fixtures without +the link store, so links do not reach the mock broadcast; that boundary is +separate. + +- The monitor adapter passes its configured stale thresholds to record + ingest, so a link joined there judges each side's metrics the way a + snapshot-time link does. Ordinary source merges into an existing row use + the same thresholds on every ingest path: snapshot and record ingest hold + the caller's thresholds while they run, and `mergeInto` judges each + metric's freshness with them through `sourceStaleThreshold`, the rule the + stale pass reads. Before this, those merges used the defaults, so a + Proxmox node polled every two minutes showed a silent auto-linked agent's + readings while the stale pass called the node's poll current, and a + supplemental refresh of a linked vSphere VM kept the agent's memory over + vSphere's current reading. Outside mock mode the monitor configures only + Proxmox, PBS and PMG; mock mode also configures TrueNAS, vSphere and + availability. Regression coverage: + `TestMonitorAdapterSourceMergesUseConfiguredStaleThresholds` (rebuild, + supplemental refresh, overlay, node with auto-linked agent), + `TestSourceMergesJudgeFreshnessByTheIngestStaleThresholds` and + `TestMetricMergeFreshnessMatchesTheStalePass`. +- Continuity records are the one record ingest that never joins links. A + saved enrollment only fills an absent machine, so it stays its own row + instead of lending its linked guest an offline verdict and an old agent + payload. +- A link merge keeps the TrueNAS and vSphere payloads the primary lacks, so + an agent chosen as primary over a TrueNAS system still carries the + system's facet, and the system's pools re-parent to the merged row. +- The registry remembers each resource a manual link folded into its + primary. A folded resource is still observed, so neither pin-driven nor + record-declared succession may succeed its ID. An agent-type primary + carries the folded resource's machine keys and is pinned with them, and + succeeding the folded ID re-keyed the link onto the primary itself, which + split the pair on the next rebuild. That already broke snapshot links whose + primary is a machine-keyless agent-type row, such as a Proxmox node. +- Fold history is registry-local: resource seeding and overlay clones do not + carry it. A reference to the folded resource's canonical ID, such as an + availability check configured against an agent later linked into its + guest, does not resolve to the primary, so the check loses its target in + the broadcast as it already did in the resources API. + +`registry_merge_policy_test.go` pins the record links in both directions +through two rebuilds and the live supplemental refresh, and checks that a +registry seeded from the monitor's listing through resource ingest, as the +resources API seeds, lists the same rows +(`TestManualLinksJoinRecordIngestedResourcesInMonitorRebuild`). The same file +pins the node case (`TestManualLinkKeepsItsEndpointsThroughIdentityPinPersistence`), +record-declared succession (`TestRecordDeclaredSuccessionSparesALinkFoldedResource`) +and both payloads (`TestManualLinkKeepsProviderPayloadsThePrimaryLacks`). +`monitor_adapter_read_state_test.go` +(`TestMonitorAdapterJoinsLinkedRecordsWithConfiguredStaleThresholds`) pins the +thresholds on the rebuild, supplemental and overlay paths, and +`internal/monitoring/issue1913_host_continuity_test.go` +(`TestManualLinkToSupplementalGuestHoldsWithAndWithoutContinuity`) pins the +published inventory with no continuity, an unrelated absent machine and the +linked agent's own saved enrollment. + ### Provider link network corroboration An existing reciprocal host/provider link remains authoritative, including a diff --git a/frontend-modern/browser-verification/0f6190e732d7b132.json b/frontend-modern/browser-verification/0f6190e732d7b132.json new file mode 100644 index 0000000000..bd6ecf244f --- /dev/null +++ b/frontend-modern/browser-verification/0f6190e732d7b132.json @@ -0,0 +1,44 @@ +{ + "version": 1, + "base_sha": "421d1e5423e5b83b7923ea8c3b133ad520cd5f60", + "verified_at": "2026-10-07T15:20:38Z", + "result": "passed", + "changed_paths": [ + "frontend-modern/src/features/patrol/patrolInvestigationContextModel.ts", + "frontend-modern/src/features/patrol/usePatrolIntelligenceState.ts" + ], + "content_sha256": { + "frontend-modern/src/features/patrol/patrolInvestigationContextModel.ts": "f1e249c36f0f0f9810330a37cc9a80cb43baa1cd123db1defd3f27720d39912a", + "frontend-modern/src/features/patrol/usePatrolIntelligenceState.ts": "cb0a6962ed929238a974a3bbe5f69f4c89961ff30c88c2c94398073b9ef99a1c" + }, + "routes": [ + "/patrol", + "/patrol (Mode and automation disclosure)", + "/settings/pulse-intelligence/patrol", + "/settings/pulse-intelligence" + ], + "viewports": [ + { + "width": 1440, + "height": 900 + }, + { + "width": 390, + "height": 844 + } + ], + "states": [ + "Embedded production builds (vite build into internal/api/frontend-modern/dist, go build) of the parent tree (333e51d2de; rebased onto 421d1e5423, which changes only contract docs) and this tree, each on a private DEMO_MODE mock stack with a fresh data dir; the same Playwright script ran against both and produced identical results on all 31 observed fields at both widths.", + "Patrol on/off save whose PUT /api/settings/ai/update response (page.route) reports patrol_readiness not_ready: switching off and back on each raise only the warning toast 'Patrol setting was saved, but Patrol is not ready to run.' (role=alert inside the Notifications region, close button only); no role=alert outside the toast region, no 'needs attention' text, no readiness summary rendered inline, no Assistant action.", + "Failed Patrol mode save (Pro mode selector unlocked by adding ai_autofix to runtime-capabilities via page.route; PUT /api/ai/patrol/autonomy answered 500 {error}): only the error toast with the server reason; mode badge returns to Watch only; no inline error or 'needs attention' state.", + "Successful Patrol mode save (PUT 200, reconcile GET returns approval): Ask first becomes pressed and the badge reads Ask first; no new toast and no inline alert.", + "Settings > Pulse Intelligence > Patrol save (Schedule -> Every hour) and Settings > Provider & Models save with a not_ready response: only the warning toast ', but Patrol is not ready: The selected model cannot run Patrol tools. \u00b7 Provider: OpenRouter \u00b7 Model: openrouter:deepseek/deepseek-r1'; no inline alert.", + "Patrol status with readiness not_ready (page.route on /api/ai/patrol/status): the page's own 'Patrol model issue' banner names summary, provider and model, and the header shows Fix setup, at both widths.", + "Dist proof: 187 files each side; after normalising chunk hashes and SRI strings only assets/AIIntelligence.js differs (935 bytes smaller): the removed failure builder, details helper and surface callback replaced by one status check, toast string byte-identical; check-bundle-size.mjs passes." + ], + "interactions": [ + "Demo sign-in; on /patrol click Toggle Patrol twice (off, then on) with a not-ready settings response; open 'Mode and automation' and click Ask first with a rejected save, then again with an accepted save; desktop mouse and phone touch (isMobile) contexts.", + "On /settings/pulse-intelligence/patrol select Schedule 'Every hour' and click Save Patrol settings; on /settings/pulse-intelligence click Save provider settings; both with a not-ready response.", + "Screenshots of each toast state captured and inspected at both widths." + ] +} diff --git a/frontend-modern/browser-verification/1214e26434830968.json b/frontend-modern/browser-verification/1214e26434830968.json new file mode 100644 index 0000000000..00fd597c11 --- /dev/null +++ b/frontend-modern/browser-verification/1214e26434830968.json @@ -0,0 +1,60 @@ +{ + "version": 1, + "base_sha": "48c2d6cdce8d019151e3567873de88e21b76bb22", + "verified_at": "2026-10-07T19:21:21Z", + "result": "passed", + "changed_paths": [ + "frontend-modern/src/components/Workloads/GuestDrawerOverview.tsx", + "frontend-modern/src/components/Workloads/GuestRow.tsx", + "frontend-modern/src/components/Workloads/guestDrawerModel.ts", + "frontend-modern/src/components/Workloads/useGuestRowState.ts", + "frontend-modern/src/utils/resourceStateAdapters.ts", + "frontend-modern/src/utils/workloadGuestPresentation.ts" + ], + "content_sha256": { + "frontend-modern/src/components/Workloads/GuestDrawerOverview.tsx": "1cf03b76a8d8732db0fda9b2ec30c0245c350eadcb36a36909493a9f4d6e7831", + "frontend-modern/src/components/Workloads/GuestRow.tsx": "c7b8bb81f158d8d64195be65fa69b8ee18019414e602043c5413693efc06d1d6", + "frontend-modern/src/components/Workloads/guestDrawerModel.ts": "a30e4435523c0d30f618eb7f0e803cfba9a3a5203851d642f1d5b25c2f33b10d", + "frontend-modern/src/components/Workloads/useGuestRowState.ts": "1c41975ec9447f3657cd4e1cf0e9fc6ce187fdca18e7f050f882ac5d483379c9", + "frontend-modern/src/utils/resourceStateAdapters.ts": "35e7e01344e2f78cfcf550cc0b277aa06e42c544a110a6c33b054d97d9b16765", + "frontend-modern/src/utils/workloadGuestPresentation.ts": "3d1458ee02891047f722d643e74978fac6ec4edae5a24f5eb1808331de2e3b8d" + }, + "routes": [ + "/proxmox?q=checkout-web-01", + "/proxmox?q=checkout-web-02", + "/proxmox?q=analytics-batch-01", + "/proxmox?q=checkout-web-193", + "/standalone" + ], + "viewports": [ + { + "width": 1440, + "height": 900 + }, + { + "width": 1920, + "height": 1000 + }, + { + "width": 390, + "height": 844 + } + ], + "states": [ + "Private mock stack (PULSE_MOCK_MODE, embedded production build of this tree on origin/main dace9295c6, rebased onto 48c2d6cdce which changes none of the verified frontend sources) with a verification-only generator overlay: Proxmox guest filesystems stripped to usage -1 + agent-not-running (or vm-stopped) for checkout-web-01/02/193, and standalone agents Apollo-114 and Meridian-731 held offline from a control file", + "Agents linked into VMs through POST /api/resources/{vm}/link: Apollo-114 into checkout-web-01 (silent), Spectrum-709 into checkout-web-02 (live), Meridian-848 into analytics-batch-01 (Proxmox filesystems kept, control), Meridian-731 into stopped checkout-web-193 (silent); /api/resources confirmed agent.stale, agent.disks, absent proxmox.disks and the agent-sourced metrics.disk", + "Mock-mode websocket state does not apply manual links (record-ingested resources skip applyManualLinks; chipped separately), so guest passes ran the Proxmox page on the linked REST resources with the websocket held by Playwright routeWebSocket", + "Before (main frontend + backend, same data): checkout-web-01 and checkout-web-02 disk cells read '- Unavailable' with 'Guest agent not running' and History deferred the disk on that message at 1440 and 390", + "After, live agent (checkout-web-02): agent filesystem bars with no read-status label, nothing deferred in History, drawer filesystem rows current (1440 and 390)", + "After, silent agent (checkout-web-01): bars plus 'Last known' (phone 'Prior') with 'Using last known disk stats. The Pulse Agent in this guest stopped reporting.', drawer filesystem rows last known, History disk deferred on the same message (1440 and 390)", + "After, stopped guest (checkout-web-193): '- Unavailable' / 'N/A' with the vm-stopped message in cell, drawer and History; analytics-batch-01 with Proxmox filesystems unchanged", + "Machines (/standalone at 1920): Apollo-114 held silent read '(stale)' with the stopped-reporting title; after releasing the hold mid-session without a reload, main kept '(stale)' although /api/state had dropped stale, while this build received stale:false and cleared it with metrics resuming", + "No page errors or console errors on /proxmox (guest rows, drawers, History) and /standalone, with and without the websocket" + ], + "interactions": [ + "Playwright 1440x900 and 390x844: open /proxmox?q= for four guests, read the disk cell text, read-status notice and bar, open the guest drawer (phone: second visible cell), read the filesystem status line, switch to History and read the disk current/last-known/deferred legends; screenshots of rows and History", + "Same pass against a before binary built from main's frontend and backend on the same data dir", + "Playwright 1920x1000 on /standalone: watch Apollo-114 while the control file releases its silent hold, polling /api/state for the agent's stale flag and re-reading the row without reloading", + "API: GET /api/resources/{vm} and /api/state before and after linking and after backend restarts" + ] +} diff --git a/frontend-modern/browser-verification/2cb610e77c51ad5f.json b/frontend-modern/browser-verification/2cb610e77c51ad5f.json new file mode 100644 index 0000000000..afde001442 --- /dev/null +++ b/frontend-modern/browser-verification/2cb610e77c51ad5f.json @@ -0,0 +1,56 @@ +{ + "version": 1, + "base_sha": "48c2d6cdce8d019151e3567873de88e21b76bb22", + "verified_at": "2026-10-07T18:53:57Z", + "result": "passed", + "changed_paths": [ + "frontend-modern/src/components/Storage/useStoragePageModel.ts", + "frontend-modern/src/components/shared/contextualFocus.ts", + "frontend-modern/src/components/shared/summaryCardInteraction.ts", + "frontend-modern/src/components/shared/summaryTableFocus.ts" + ], + "content_sha256": { + "frontend-modern/src/components/Storage/useStoragePageModel.ts": "9f4734cf74cb44b7db9222bd08d25fa2ae52d787e192bd07e36640e2e81cde6d", + "frontend-modern/src/components/shared/contextualFocus.ts": "4dc89355f978c9dfee90cb0c780976504ac4e7e158bbe7ddf02d566c57211fff", + "frontend-modern/src/components/shared/summaryCardInteraction.ts": "9e4f4f9275cd01f2a19b98c1c795eec5c7f37be8f5a002170a94f400bdae4625", + "frontend-modern/src/components/shared/summaryTableFocus.ts": "52c46c227696b5dc9713d0cc3d9b76e434fecafeeda46edb064858b0328732ee" + }, + "routes": [ + "/proxmox/storage", + "/proxmox/storage?tab=disks", + "/proxmox/storage?group=node&resource=ceph-rbd", + "/proxmox/storage?resource=ceph-rbd", + "/proxmox/storage?group=node", + "/proxmox/storage?group=node&summaryGroup=storage:node:West%20Production%20F", + "/proxmox", + "/proxmox?resource=" + ], + "viewports": [ + { + "width": 1440, + "height": 900 + }, + { + "width": 390, + "height": 844 + } + ], + "states": [ + "Private 12-node mock stack (backend :7783, Vite :5323 from the landing clone). One Playwright scenario script ran with the four runtime files swapped back to origin/main (before) and with this change (after) at 1440x900 and 390x844, then again after rebasing onto 71a723abb1 with the backend rebuilt from that tree; the final parent adds only #2677 (Patrol correlation fetch), which touches none of these files; no page errors in any run", + "Storage pools: hovering 8 rows marks each data-summary-row-active and never moves the page; clicking a lower pool opens its inline detail and scrolls just enough to show the row header and top of the detail; Escape closes it; identical before and after", + "Storage deep link ?resource=ceph-rbd opens the Ceph pool detail in view; with group=node, clicking a group header pins it (summaryGroup in the URL, header marked active); a summaryGroup deep link pins and scrolls toward the last group, landing the header in or just below view with the same timing spread before and after", + "Storage Physical Disks: clicking a lower disk opens its detail with the same reveal, and focus stays on the Physical Disks view; pool focus stays on the pools view; identical before and after", + "Storage in-app link: with group=node, collapsing the shared group with its disclosure button hides ceph-rbd; a client-side navigation to ?resource=ceph-rbd then reopens the group and lands the Ceph pool detail in view, at both widths, identical before and after", + "Workloads (/proxmox): hovering guest rows marks them active; clicking a lower guest opens its inline drawer in view and Escape closes it; a ?resource deep link opens the drawer in view; identical before and after", + "Cost: window scroll listeners drop from 2 to 1 and resize by one on both pages; getComputedStyle calls during 8 row hovers fall 128 to 0 (Storage) and 175-193 to 13-39 (Workloads), during 10 scroll steps 320 to 0 and 274-472 to 130", + "Pre-existing on main and unchanged here: with a pool open, Storage scrolls itself back to it a few seconds after scrolling away (disks view and Workloads do not), and collapsing the group holding the open pool bounces it open; traced to the reveal effect tracking reads inside revealActiveSeries, left for a follow-up" + ], + "interactions": [ + "Hovered 8 rows then scrolled the app shell 10 steps on Storage and Workloads with window listener and getComputedStyle counters installed before load", + "Clicked a pool row, a physical disk row and a guest row near the bottom, measured the row and detail rectangles, pressed Escape", + "Loaded the ceph-rbd, summaryGroup and guest resource deep links, repeating the summaryGroup link four times per width", + "Collapsed the shared storage group via its disclosure button, then pushed /proxmox/storage?group=node&resource=ceph-rbd through the router", + "Clicked a storage group header to pin it, and clicked the header of the group holding an open pool", + "Scrolled the shell to the top with a pool, a disk and a guest open and sampled the scroll position every 500 ms for 12 s" + ] +} diff --git a/frontend-modern/browser-verification/71a0ce6b24830582.json b/frontend-modern/browser-verification/71a0ce6b24830582.json new file mode 100644 index 0000000000..0b4623136d --- /dev/null +++ b/frontend-modern/browser-verification/71a0ce6b24830582.json @@ -0,0 +1,39 @@ +{ + "version": 1, + "base_sha": "48c2d6cdce8d019151e3567873de88e21b76bb22", + "verified_at": "2026-10-07T19:05:33Z", + "result": "passed", + "changed_paths": [ + "frontend-modern/src/types/aiIntelligence.ts" + ], + "content_sha256": { + "frontend-modern/src/types/aiIntelligence.ts": "d9287de8171cee10d72c5df0a898c28e47e649be3f9dee8fa791a7406c0adc51" + }, + "routes": [ + "/patrol", + "/kubernetes" + ], + "viewports": [ + { + "width": 1280, + "height": 800 + }, + { + "width": 390, + "height": 844 + } + ], + "states": [ + "Type-only deletion: TypeScript erases CorrelationsResponse, and npm run build before and after the change emitted the same 187 dist files with identical SHA-256 sums, so the shipped bundle is byte-identical", + "Embedded production build in a private demo-mode mock stack (port 7763, fresh data dir)", + "Patrol inbox: Patrol heading and 5 attention rows render at 1280x800 and 390x844", + "Kubernetes deployment account-portal drawer opens with header, technical summary, runtime context and identity sections at both widths", + "No console errors and no page errors on either route at either width; no /api/ai/intelligence/correlations request", + "Relationship map (ResourceCorrelationSummary, the surviving ResourceCorrelation consumer) not rendered in this pass: table-row drawers do not fetch per-resource facets or resource intelligence (enableRemoteHistory is off for table-row presentation); its code ships unchanged in the byte-identical bundle" + ], + "interactions": [ + "Demo sign-in, then navigate to /patrol and wait for the Patrol heading", + "Navigate to /kubernetes and click the account-portal deployment row to open its inline resource drawer", + "Repeat the same pass at 1280x800 and 390x844" + ] +} diff --git a/frontend-modern/browser-verification/dad3b270f160419c.json b/frontend-modern/browser-verification/dad3b270f160419c.json new file mode 100644 index 0000000000..5cee659013 --- /dev/null +++ b/frontend-modern/browser-verification/dad3b270f160419c.json @@ -0,0 +1,42 @@ +{ + "version": 1, + "base_sha": "77d21197927a670076a74e87634e73ea999a566a", + "verified_at": "2026-10-07T19:05:00Z", + "result": "passed", + "changed_paths": [ + "frontend-modern/src/components/Settings/reportingCatalogModel.ts", + "frontend-modern/src/components/Settings/reportingPanelModel.ts" + ], + "content_sha256": { + "frontend-modern/src/components/Settings/reportingCatalogModel.ts": "fc3ab83f49ac62aaf7ef97b415a0bff1c4e445ca17771efc18304295e94e3df0", + "frontend-modern/src/components/Settings/reportingPanelModel.ts": "e4366a8c50ea897dcb4907b4e488a2c5f758e710d7a9be0d5fe0d3e047e2d9c5" + }, + "routes": [ + "/settings/support/reporting", + "/api/admin/reports/generate" + ], + "viewports": [ + { + "width": 1280, + "height": 800 + }, + { + "width": 390, + "height": 844 + } + ], + "states": [ + "Pro dev licence active, Pulse Assistant enabled on a local stub OpenAI-compatible provider that logs each call, real admin session login (no admin bypass)", + "Data & Reports panel with one mock agent selected, then two", + "Downloaded single-resource PDF opened: POST copy carries the stub AI narrative and provenance footer; GET copy carries the deterministic summary and the new GET note in place of the Configure Pulse Assistant tip", + "AI usage summary after the run: report_narrative recorded only for the two UI POST downloads" + ], + "interactions": [ + "Generate Report (1 resource) as PDF at 1280x800 and 390x844: one POST /api/admin/reports/generate with a JSON body, 200 application/pdf, download saved, stub provider called once each", + "Generate Report (1 resource) as CSV at both widths: POST, 200 text/csv, stub provider not called", + "Generate Report (2 resources) as PDF at both widths: POST /api/admin/reports/generate-multi unchanged, 200 application/pdf", + "Top-level browser navigation to GET /api/admin/reports/generate?format=pdf with the session cookie: PDF downloaded, stub provider not called", + "HEAD, PUT, DELETE and PATCH return 405 with Allow GET, POST; a session POST without the CSRF header returns 403", + "No console errors on the reporting panel at either width" + ] +} diff --git a/frontend-modern/public/docs/API.md b/frontend-modern/public/docs/API.md index 296d4cc116..002d29a1e2 100644 --- a/frontend-modern/public/docs/API.md +++ b/frontend-modern/public/docs/API.md @@ -836,9 +836,12 @@ delivery for all resources. - `GET /api/admin/reports/catalog` (admin, `settings:read`) - Returns the canonical reporting catalog for the settings surface, including locked-state teaser copy, enabled-surface guidance copy, performance-report options, canonical single-report filename subject, canonical fallback filename date style, and the nested VM inventory export definition. - Metadata route: readable without the `advanced_reporting` feature so locked admin surfaces can render the same reporting definition before upsell. -- `GET /api/admin/reports/generate` (admin, `settings:read`) - - Query params: `format` (pdf/csv, default `pdf`), `resourceType`, `resourceId`, `metricType` (optional), `start`/`end` (RFC3339, optional; defaults to last 24h), `title` (optional) +- `POST /api/admin/reports/generate` (admin, `settings:read`) + - Body fields: `format` (pdf/csv, default `pdf`), `resourceType`, `resourceId`, `metricType` (optional), `start`/`end` (RFC3339, optional; defaults to last 24h), `title` (optional) - If `title` is omitted, the backend applies the canonical default title for that resource report. + - When Pulse Assistant is configured, the configured AI provider narrates a PDF's executive summary; each narration counts against the AI budget and appears in AI usage. If the call fails, times out or the budget is spent, the PDF falls back to the deterministic summary. CSV output is never narrated. The settings UI uses this method. +- `GET /api/admin/reports/generate` (admin, `settings:read`) + - Takes the same fields as query params and returns the same report, except that it never calls an AI provider: a PDF carries the deterministic summary and a note saying AI narration needs `POST`. Other methods return `405` with `Allow: GET, POST`. - `POST /api/admin/reports/generate-multi` (admin, `settings:read`) - Body fields: `resources` (1-50 entries of `{resourceType,resourceId}`), `format`, `metricType` (optional), `start`/`end` (RFC3339, optional; defaults to last 24h), `title` (optional) - If `title` is omitted, the backend applies the canonical default fleet report title. @@ -851,8 +854,8 @@ Validation and limits: - Malformed `start`/`end` values, `end` not strictly after `start`, or report windows over 366 days return `400 invalid_time_range`. - `metricType` must match `[a-zA-Z0-9._:-]+` and be <= 64 chars, otherwise `400 invalid_metric_type`. - `title` must be <= 256 chars, otherwise `400 invalid_title`. -- Multi-report body max size is 1MB; oversized payloads return `400 body_too_large`. -- Multi-report bodies reject trailing payload and unknown JSON fields with `400 invalid_body`. +- Report body max size is 1MB; oversized payloads return `400 body_too_large`. +- Report bodies reject trailing payload and unknown JSON fields with `400 invalid_body`. - VM inventory export only accepts `csv`. Common reporting error codes: @@ -891,7 +894,9 @@ For an existing incident whose notifications should be paused without clearing it, use the snooze and unsnooze endpoints below. - `GET /api/alerts/config` -- `PUT /api/alerts/config` +- `PUT /api/alerts/config` โ€” replaces only the top-level keys the body + carries; a key left out keeps its stored value, and the merged config is + normalized as before. The body must be a JSON object. - `GET /api/alerts/deadman/config` โ€” returns only whether an external watchdog is configured; `pingUrl` is `***REDACTED***` when present and never returns the credential-bearing URL diff --git a/frontend-modern/public/docs/MSP.md b/frontend-modern/public/docs/MSP.md index 7561f520c6..9302b66a1d 100644 --- a/frontend-modern/public/docs/MSP.md +++ b/frontend-modern/public/docs/MSP.md @@ -260,10 +260,12 @@ Each client runtime (or organization) generates its own reports, scoped to that client's resources: - **UI**: Settings โ†’ Data & Reports. -- **API**: `GET /api/admin/reports/generate` (single resource) and +- **API**: `POST /api/admin/reports/generate` (single resource) and `POST /api/admin/reports/generate-multi` (up to 50 resources per report), returning PDF or CSV. In shared-process mode, scope with `X-Pulse-Org-ID` - or an org-bound token. + or an org-bound token. `GET /api/admin/reports/generate` with query + parameters still returns a single-resource report, but never adds the + Pulse Assistant narrative. - **Schedules**: `GET`/`POST /api/admin/reports/schedules`, `PUT`/`DELETE /api/admin/reports/schedules/{id}`, and `POST /api/admin/reports/schedules/{id}/run`. Schedules can target explicit diff --git a/frontend-modern/src/components/Settings/__tests__/ReportingPanel.test.tsx b/frontend-modern/src/components/Settings/__tests__/ReportingPanel.test.tsx index cf86494145..f947e2968c 100644 --- a/frontend-modern/src/components/Settings/__tests__/ReportingPanel.test.tsx +++ b/frontend-modern/src/components/Settings/__tests__/ReportingPanel.test.tsx @@ -35,6 +35,7 @@ const baseCatalog = { title: 'Performance Reports', description: 'Historical performance reporting', singleResourceEndpoint: '/api/admin/reports/generate', + singleResourceMethod: 'POST', multiResourceEndpoint: '/api/admin/reports/generate-multi', singleFilenamePrefix: 'report', singleFilenameSubject: 'resource_id', diff --git a/frontend-modern/src/components/Settings/__tests__/reportingCatalogModel.branchcov2.test.ts b/frontend-modern/src/components/Settings/__tests__/reportingCatalogModel.branchcov2.test.ts index b451dc65fd..1b38e0882f 100644 --- a/frontend-modern/src/components/Settings/__tests__/reportingCatalogModel.branchcov2.test.ts +++ b/frontend-modern/src/components/Settings/__tests__/reportingCatalogModel.branchcov2.test.ts @@ -42,6 +42,7 @@ const basePerformanceReport = () => ({ title: 'Performance Reports', description: 'Historical performance reporting', singleResourceEndpoint: '/api/admin/reports/generate', + singleResourceMethod: 'POST', multiResourceEndpoint: '/api/admin/reports/generate-multi', singleFilenamePrefix: 'report', singleFilenameSubject: 'resource_id', @@ -453,6 +454,19 @@ describe('parseReportingPerformanceReportDefinition (via catalog.performanceRepo }); }); + it('throws when singleResourceMethod is neither GET nor POST', () => { + expectInvalid({ + ...baseCatalog(), + performanceReport: { ...basePerformanceReport(), singleResourceMethod: 'PUT' }, + }); + }); + + it('reads a missing singleResourceMethod as GET, the only method older backends accept', () => { + const { singleResourceMethod: _omitted, ...withoutMethod } = basePerformanceReport(); + const parsed = parseReportingCatalog({ ...baseCatalog(), performanceReport: withoutMethod }); + expect(parsed.performanceReport.singleResourceMethod).toBe('GET'); + }); + it('throws when multiResourceEndpoint is not a string', () => { expectInvalid({ ...baseCatalog(), diff --git a/frontend-modern/src/components/Settings/__tests__/reportingCatalogModel.test.ts b/frontend-modern/src/components/Settings/__tests__/reportingCatalogModel.test.ts index 97add3f9ae..2b1d289a3c 100644 --- a/frontend-modern/src/components/Settings/__tests__/reportingCatalogModel.test.ts +++ b/frontend-modern/src/components/Settings/__tests__/reportingCatalogModel.test.ts @@ -22,6 +22,7 @@ const baseCatalogPayload = { title: 'Performance Reports', description: 'Historical performance reporting', singleResourceEndpoint: '/api/admin/reports/generate', + singleResourceMethod: 'POST', multiResourceEndpoint: '/api/admin/reports/generate-multi', singleFilenamePrefix: 'report', singleFilenameSubject: 'resource_id', @@ -74,6 +75,7 @@ describe('reporting catalog model', () => { const catalog = buildLegacyReportingCatalogFallback(); expect(catalog.performanceReport.singleResourceEndpoint).toBe('/api/reporting'); + expect(catalog.performanceReport.singleResourceMethod).toBe('GET'); expect(catalog.performanceReport.multiResourceEndpoint).toBe('/api/reporting/generate-multi'); expect(catalog.vmInventoryExport).toBeNull(); }); diff --git a/frontend-modern/src/components/Settings/__tests__/reportingPanelModel.test.ts b/frontend-modern/src/components/Settings/__tests__/reportingPanelModel.test.ts index a058281e36..708b9fe8cb 100644 --- a/frontend-modern/src/components/Settings/__tests__/reportingPanelModel.test.ts +++ b/frontend-modern/src/components/Settings/__tests__/reportingPanelModel.test.ts @@ -14,6 +14,7 @@ const performanceDefinition: ReportingPerformanceReportDefinition = { title: 'Performance Reports', description: 'Historical performance reporting', singleResourceEndpoint: '/api/admin/reports/generate', + singleResourceMethod: 'POST', multiResourceEndpoint: '/api/admin/reports/generate-multi', singleFilenamePrefix: 'report', singleFilenameSubject: 'resource_id', @@ -75,12 +76,52 @@ describe('reporting panel model', () => { ); expect(request.filename).toBe('report-agent-1-20260320.pdf'); + // POST, never GET: only POST may narrate the PDF with Pulse Assistant. + expect(request.request.url).toBe('/api/admin/reports/generate'); + expect(request.request.init).toMatchObject({ + method: 'POST', + headers: { 'Content-Type': 'application/json' }, + }); + expect(JSON.parse(String(request.request.init?.body))).toEqual({ + resourceType: 'agent', + resourceId: 'agent-1', + format: 'pdf', + start: '2026-03-19T12:34:56.000Z', + end: now.toISOString(), + metricType: 'cpu', + }); + }); + + it('keeps the query-string GET transport for backends whose catalog names GET', () => { + const now = new Date('2026-03-20T12:34:56.000Z'); + const request = buildReportingRequest( + { + end: now.toISOString(), + format: 'csv', + metricType: 'cpu', + now, + resources: [{ id: 'vm-1', type: 'vm', name: 'vm-a' }], + start: '2026-03-19T12:34:56.000Z', + title: '', + }, + { + ...performanceDefinition, + singleResourceEndpoint: '/api/reporting', + singleResourceMethod: 'GET', + }, + ); + expect(request.request.init).toBeUndefined(); - expect(request.request.url).toContain('/api/admin/reports/generate?'); - expect(request.request.url).toContain('resourceType=agent'); - expect(request.request.url).toContain('resourceId=agent-1'); - expect(request.request.url).toContain('metricType=cpu'); - expect(request.request.url).not.toContain('title='); + const url = new URL(request.request.url, 'http://pulse.local'); + expect(url.pathname).toBe('/api/reporting'); + expect(Object.fromEntries(url.searchParams)).toEqual({ + resourceType: 'vm', + resourceId: 'vm-1', + format: 'csv', + start: '2026-03-19T12:34:56.000Z', + end: now.toISOString(), + metricType: 'cpu', + }); }); it('builds a fleet reporting request body and filename', () => { @@ -158,9 +199,9 @@ describe('reporting panel model', () => { }, ); - expect(request.request.url).not.toContain('title='); - expect(request.request.url).not.toContain('metricType='); - expect(request.request.url).not.toContain('Custom+fleet+title'); + const body = JSON.parse(String(request.request.init?.body)); + expect(body).not.toHaveProperty('title'); + expect(body).not.toHaveProperty('metricType'); }); it('passes through explicit custom titles without inventing defaults', () => { @@ -186,7 +227,7 @@ describe('reporting panel model', () => { performanceDefinition, ); - expect(request.request.url).toContain('title=Custom+report+title'); + expect(JSON.parse(String(request.request.init?.body)).title).toBe('Custom report title'); }); it('derives canonical range starts from the selected preset', () => { diff --git a/frontend-modern/src/components/Settings/__tests__/useReportingPanelState.test.ts b/frontend-modern/src/components/Settings/__tests__/useReportingPanelState.test.ts index aaff7556e7..5a3caa8bfc 100644 --- a/frontend-modern/src/components/Settings/__tests__/useReportingPanelState.test.ts +++ b/frontend-modern/src/components/Settings/__tests__/useReportingPanelState.test.ts @@ -28,6 +28,7 @@ const catalogPayload = { title: 'Performance Reports', description: 'Historical performance reporting', singleResourceEndpoint: '/api/admin/reports/generate', + singleResourceMethod: 'POST', multiResourceEndpoint: '/api/admin/reports/generate-multi', singleFilenamePrefix: 'report', singleFilenameSubject: 'resource_id', @@ -478,6 +479,48 @@ describe('useReportingPanelState', () => { dispose(); }); + it('posts a single-resource report as a JSON body through apiFetch', async () => { + apiFetchMock.mockImplementation((url: string) => { + if (url === '/api/admin/reports/generate') { + return Promise.resolve(new Response('%PDF', { status: 200 })); + } + if (url === '/api/admin/reports/schedules') { + return Promise.resolve(jsonResponse(schedulesPayload)); + } + return Promise.resolve(jsonResponse(catalogPayload)); + }); + const { createObjectURL, revokeObjectURL } = URL; + URL.createObjectURL = vi.fn(() => 'blob:report'); + URL.revokeObjectURL = vi.fn(); + try { + const { hookState, dispose } = mountHook(); + await flushAsync(); + + hookState.setSelectedResources([{ id: 'vm-1', type: 'vm', name: 'vm-a' }]); + await hookState.handleGenerate(); + + const generate = apiFetchMock.mock.calls.filter(([url]) => + String(url).startsWith('/api/admin/reports/generate'), + ); + expect(generate).toHaveLength(1); + expect(generate[0][0]).toBe('/api/admin/reports/generate'); + expect(generate[0][1]).toMatchObject({ + method: 'POST', + headers: { 'Content-Type': 'application/json' }, + }); + expect(JSON.parse(String(generate[0][1].body))).toMatchObject({ + resourceType: 'vm', + resourceId: 'vm-1', + format: catalogPayload.performanceReport.defaultFormat, + }); + expect(showWarningMock).not.toHaveBeenCalled(); + dispose(); + } finally { + URL.createObjectURL = createObjectURL; + URL.revokeObjectURL = revokeObjectURL; + } + }); + it('saves a Patrol weekly summary without a resource scope', async () => { const savedSchedule = { id: 'digest-1', diff --git a/frontend-modern/src/components/Settings/reportingCatalogModel.ts b/frontend-modern/src/components/Settings/reportingCatalogModel.ts index 52749a55de..33b73fafa6 100644 --- a/frontend-modern/src/components/Settings/reportingCatalogModel.ts +++ b/frontend-modern/src/components/Settings/reportingCatalogModel.ts @@ -6,6 +6,7 @@ import { export type ReportingFormat = 'pdf' | 'csv'; export type ReportingFilenameDateStyle = 'utc_yyyymmdd'; export type ReportingFilenameSubject = 'resource_id'; +export type ReportingSingleResourceMethod = 'GET' | 'POST'; export interface ReportingFormatDefinition { value: ReportingFormat; @@ -24,6 +25,7 @@ export interface ReportingPerformanceReportDefinition { title: string; description: string; singleResourceEndpoint: string; + singleResourceMethod: ReportingSingleResourceMethod; multiResourceEndpoint: string; singleFilenamePrefix: string; singleFilenameSubject: ReportingFilenameSubject; @@ -85,6 +87,7 @@ export function buildLegacyReportingCatalogFallback(): ReportingCatalog { description: 'Generate PDF summaries or CSV metric exports from historical monitoring data for one or more selected resources.', singleResourceEndpoint: '/api/reporting', + singleResourceMethod: 'GET', multiResourceEndpoint: '/api/reporting/generate-multi', singleFilenamePrefix: 'report', singleFilenameSubject: 'resource_id', @@ -176,6 +179,9 @@ function parseReportingPerformanceReportDefinition( typeof candidate.title !== 'string' || typeof candidate.description !== 'string' || typeof candidate.singleResourceEndpoint !== 'string' || + (candidate.singleResourceMethod !== undefined && + candidate.singleResourceMethod !== 'GET' && + candidate.singleResourceMethod !== 'POST') || typeof candidate.multiResourceEndpoint !== 'string' || typeof candidate.singleFilenamePrefix !== 'string' || candidate.singleFilenameSubject !== 'resource_id' || @@ -211,6 +217,8 @@ function parseReportingPerformanceReportDefinition( title: candidate.title, description: candidate.description, singleResourceEndpoint: candidate.singleResourceEndpoint, + // Backends that predate the field accept only GET. + singleResourceMethod: candidate.singleResourceMethod ?? 'GET', multiResourceEndpoint: candidate.multiResourceEndpoint, singleFilenamePrefix: candidate.singleFilenamePrefix, singleFilenameSubject: candidate.singleFilenameSubject, diff --git a/frontend-modern/src/components/Settings/reportingPanelModel.ts b/frontend-modern/src/components/Settings/reportingPanelModel.ts index fbaa61d26a..45a6671e68 100644 --- a/frontend-modern/src/components/Settings/reportingPanelModel.ts +++ b/frontend-modern/src/components/Settings/reportingPanelModel.ts @@ -96,6 +96,7 @@ export function buildReportingRequest( | 'singleFilenamePrefix' | 'singleFilenameSubject' | 'singleResourceEndpoint' + | 'singleResourceMethod' | 'supportsCustomTitle' | 'supportsMetricFilter' >, @@ -105,23 +106,41 @@ export function buildReportingRequest( if (context.resources.length === 1) { const resource = context.resources[0]; - const params = new URLSearchParams({ + const fields = { resourceType: toReportingResourceType(resource.type), resourceId: resource.id, format: context.format, start: context.start, end: context.end, - }); + title: customTitle || undefined, + metricType: metricType || undefined, + }; + const filename = buildReportingFilename(context.format, resource.id, context.now, definition); - if (metricType) { - params.append('metricType', metricType); - } - if (customTitle) { - params.append('title', customTitle); + // Only POST may narrate a PDF with Pulse Assistant; the catalog names + // GET only for backends that accept nothing else. + if (definition.singleResourceMethod === 'POST') { + return { + filename, + request: { + url: definition.singleResourceEndpoint, + init: { + method: 'POST', + headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify(fields), + }, + }, + }; } + const params = new URLSearchParams(); + for (const [key, value] of Object.entries(fields)) { + if (value) { + params.append(key, value); + } + } return { - filename: buildReportingFilename(context.format, resource.id, context.now, definition), + filename, request: { url: `${definition.singleResourceEndpoint}?${params.toString()}`, }, diff --git a/frontend-modern/src/components/Storage/__tests__/Storage.test.tsx b/frontend-modern/src/components/Storage/__tests__/Storage.test.tsx index 5b4b65d38e..277c0171a2 100644 --- a/frontend-modern/src/components/Storage/__tests__/Storage.test.tsx +++ b/frontend-modern/src/components/Storage/__tests__/Storage.test.tsx @@ -1,4 +1,5 @@ import { cleanup, fireEvent, render, screen, waitFor, within } from '@solidjs/testing-library'; +import { createSignal } from 'solid-js'; import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; import { ChartsAPI } from '@/api/charts'; import storageSummarySource from '@/components/Storage/useStorageSummaryCharts.ts?raw'; @@ -57,6 +58,9 @@ if (typeof HTMLCanvasElement.prototype.getContext === 'function') { let mockLocationSearch = ''; let mockLocationPath = '/proxmox/storage'; +// Most tests read a location snapshot; set liveLocation to follow in-app navigation. +let liveLocation = false; +const [locationVersion, setLocationVersion] = createSignal(0); const navigateSpy = vi.fn(); let wsConnected = true; @@ -296,7 +300,19 @@ vi.mock('@solidjs/router', async () => { const actual = await vi.importActual('@solidjs/router'); return { ...actual, - useLocation: () => ({ pathname: mockLocationPath, search: mockLocationSearch }), + useLocation: () => + liveLocation + ? { + get pathname() { + locationVersion(); + return mockLocationPath; + }, + get search() { + locationVersion(); + return mockLocationSearch; + }, + } + : { pathname: mockLocationPath, search: mockLocationSearch }, useNavigate: () => navigateSpy, }; }); @@ -388,6 +404,7 @@ describe('Storage', () => { fetchMock.mockClear(); mockLocationPath = '/proxmox/storage'; mockLocationSearch = ''; + liveLocation = false; navigateSpy.mockReset(); wsConnected = true; @@ -825,6 +842,41 @@ describe('Storage', () => { ).not.toBeInTheDocument(); }); + it('reopens the collapsed group of a pool focused by an in-app storage link', async () => { + liveLocation = true; + hookResources = [ + buildStorageResource('storage-ceph-link', 'Ceph-Link-Store', 'pve1', { + storageType: 'cephfs', + }), + buildStorageResource('storage-other', 'Other-Store', 'pve2'), + ]; + mockLocationSearch = '?group=node'; + + render(() => ); + + fireEvent.click(await screen.findByRole('button', { name: 'Collapse pve1' })); + await waitFor(() => { + expect(screen.queryByText('Ceph-Link-Store')).not.toBeInTheDocument(); + }); + + mockLocationSearch = '?group=node&resource=storage-ceph-link'; + setLocationVersion((version) => version + 1); + + await waitFor(() => { + expect(screen.getByRole('button', { name: 'Collapse pve1' })).toBeInTheDocument(); + const poolRow = screen.getByText('Ceph-Link-Store').closest('tr[data-summary-series-id]'); + expect( + document + .querySelector('tr[data-inline-detail-for]') + ?.getAttribute('data-inline-detail-for'), + ).toBe(poolRow?.getAttribute('data-summary-series-id')); + }); + expect(screen.getAllByRole('tab', { name: 'Storage' })[0]).toHaveAttribute( + 'aria-selected', + 'true', + ); + }); + it('clears pinned storage group scope from the content-card header action', async () => { hookResources = [ buildStorageResource('storage-1', 'Node-Store', 'pve1'), diff --git a/frontend-modern/src/components/Storage/useStoragePageModel.ts b/frontend-modern/src/components/Storage/useStoragePageModel.ts index d288072e34..3cc8e78cec 100644 --- a/frontend-modern/src/components/Storage/useStoragePageModel.ts +++ b/frontend-modern/src/components/Storage/useStoragePageModel.ts @@ -209,13 +209,6 @@ export const useStoragePageModel = (options: UseStoragePageModelOptions = {}) => } return keys; }); - const physicalDiskSeriesIds = createMemo(() => { - const ids = new Set(); - for (const disk of physicalDisks()) { - ids.add(resolvePhysicalDiskMetricResourceId(disk)); - } - return ids; - }); const storageSummaryGroupScopes = createMemo>(() => { if (view() !== 'pools' || groupBy() === 'none') { return new Map(); @@ -409,21 +402,13 @@ export const useStoragePageModel = (options: UseStoragePageModelOptions = {}) => focusedSeriesId: focusedStorageResourceId, onEscapeClear: clearAllPageStateOnEscape, revealActiveSeries: (seriesId) => { - if (physicalDiskSeriesIds().has(seriesId)) { - if (view() !== 'disks') { - setView('disks'); - } - return; - } - - const groupKey = storageGroupKeyByMetricSeriesId().get(seriesId); - if (!groupKey) { - return; - } + // Focus only ever names a row in the active view, so reveal never + // switches views; it reopens a focused pool's collapsed group. if (view() !== 'pools') { - setView('pools'); + return; } - if (!expandedGroups().has(groupKey)) { + const groupKey = storageGroupKeyByMetricSeriesId().get(seriesId); + if (groupKey && !expandedGroups().has(groupKey)) { toggleGroup(groupKey); } }, diff --git a/frontend-modern/src/components/Workloads/GuestDrawer.test.tsx b/frontend-modern/src/components/Workloads/GuestDrawer.test.tsx index 2e4f080e38..9b3ba8e5e3 100644 --- a/frontend-modern/src/components/Workloads/GuestDrawer.test.tsx +++ b/frontend-modern/src/components/Workloads/GuestDrawer.test.tsx @@ -440,6 +440,54 @@ describe('GuestDrawer', () => { } }); + it("judges a linked agent's disk metric in History by that agent, not the Proxmox reason", async () => { + chartsApiMocks.getMetricsHistory.mockResolvedValue({ + resourceType: 'vm', + resourceId: 'inst1:node1:100', + range: '24h', + start: 1, + end: 3, + metrics: {}, + source: 'store', + }); + // Proxmox has no guest filesystems, so the agent's filesystems and disk + // metric fill in while Proxmox still reports its own agent-not-running. + const [guest, setGuest] = createSignal( + makeGuest({ + disk: { total: 100, used: 40, free: 60, usage: 40 }, + disks: [{ total: 100, used: 40, free: 60, usage: 40, mountpoint: '/' }], + disksFromAgent: true, + diskStatusReason: 'agent-not-running', + }), + ); + render(() => ); + fireEvent.click(screen.getByRole('tab', { name: 'History' })); + await waitFor(() => + expect(screen.getAllByText('No stored history in this range')).toHaveLength(3), + ); + const history = screen.getAllByTestId('guest-history-group-chart')[0]; + expect(history.querySelector('[data-history-current="disk"]')).toHaveTextContent( + 'Disk40.0%current', + ); + expect(history.querySelector('[data-history-deferred="disk"]')).toBeNull(); + + setGuest({ ...guest(), agentStale: true }); + expect(history.querySelector('[data-history-current="disk"]')).toBeNull(); + expect(history.querySelector('[data-history-last-known="disk"]')).toHaveTextContent( + 'Disk40.0%last known', + ); + expect(history.querySelector('[data-history-deferred="disk"]')).toHaveTextContent( + 'The Pulse Agent in this guest stopped reporting.', + ); + + setGuest({ ...guest(), status: 'stopped' }); + expect(history.querySelector('[data-history-last-known="disk"]')).toBeNull(); + expect(history.querySelector('[data-history-current="disk"]')).toBeNull(); + expect(history.querySelector('[data-history-deferred="disk"]')).toHaveTextContent( + 'Guest filesystem stats unavailable while the VM is stopped.', + ); + }); + it('uses the shared discovery loading fallback instead of a drawer-local spinner row', () => { expect(guestDrawerSource).toContain('DiscoveryLoadingFallback'); expect(guestDrawerSource).not.toContain( diff --git a/frontend-modern/src/components/Workloads/GuestDrawerOverview.tsx b/frontend-modern/src/components/Workloads/GuestDrawerOverview.tsx index 7218755137..bce936019d 100644 --- a/frontend-modern/src/components/Workloads/GuestDrawerOverview.tsx +++ b/frontend-modern/src/components/Workloads/GuestDrawerOverview.tsx @@ -15,7 +15,7 @@ import type { DiscoveryIdentifiedSummary } from '@/utils/discoveryPresentation'; import { formatBytes } from '@/utils/format'; import type { MetricDisplayThresholds } from '@/utils/metricThresholds'; import { - getWorkloadGuestDiskStatusMessage, + getWorkloadGuestDiskRead, getWorkloadsGuestProtectionPresentation, } from '@/utils/workloadGuestPresentation'; @@ -152,44 +152,19 @@ export function GuestDrawerOverview(props: GuestDrawerOverviewProps) { props.guest.vmware?.datacenterName || props.guest.vmware?.clusterName, ); - // Rows the linked Pulse agent supplied follow that agent's freshness, not - // the Proxmox reason for its own guest reads. A stopped guest has no - // current filesystems from either source, with or without the Proxmox - // vm-stopped reason, which it gives only for a guest with allocated disk. - const vmDiskReason = isGuestDrawerVM(props.guest) ? props.guest.diskStatusReason : undefined; - const agentDisks = props.guest.disksFromAgent === true; - const agentDisksStopped = - agentDisks && (props.guest.status === 'stopped' || vmDiskReason === 'vm-stopped'); - const diskReason = agentDisks ? undefined : vmDiskReason; - const diskReadState = agentDisks - ? agentDisksStopped + // Rows the linked Pulse agent supplied follow that agent's freshness, the + // same rule the workload table's disk cell and History read. + const diskRead = getWorkloadGuestDiskRead(props.guest, isGuestDrawerVM(props.guest)); + const diskReadState = + !props.guest.disksFromAgent && props.guest.telemetryAvailability?.disk === false ? 'unavailable' - : agentLastKnownReason() - ? 'last-known' - : 'current' - : props.guest.telemetryAvailability?.disk === false - ? 'unavailable' - : diskReason - ? diskReason.startsWith('prev-') - ? 'last-known' - : 'unavailable' - : 'current'; + : diskRead.state; const diskRows = (props.guest.disks ?? []).map((disk, index) => buildWorkloadsDiskPresentation(disk, index, props.diskThresholds, diskReadState), ); - const diskStatusMessage = agentDisks - ? agentDisksStopped - ? isGuestDrawerVM(props.guest) - ? getWorkloadGuestDiskStatusMessage('vm-stopped') - : 'Filesystem usage is unavailable.' - : agentLastKnownReason() - ? 'Using last known disk stats. The Pulse Agent in this guest stopped reporting.' - : null - : diskReason - ? getWorkloadGuestDiskStatusMessage(diskReason) - : diskReadState === 'unavailable' - ? 'Filesystem usage is unavailable.' - : null; + const diskStatusMessage = + diskRead.message ?? + (diskReadState === 'unavailable' ? 'Filesystem usage is unavailable.' : null); return compactDetailSections([ discovery diff --git a/frontend-modern/src/components/Workloads/GuestRow.tsx b/frontend-modern/src/components/Workloads/GuestRow.tsx index ef0d60564d..fbdfce3e53 100644 --- a/frontend-modern/src/components/Workloads/GuestRow.tsx +++ b/frontend-modern/src/components/Workloads/GuestRow.tsx @@ -54,10 +54,6 @@ export { GUEST_COLUMNS, VIEW_MODE_COLUMNS } from './guestRowModel'; export type { GuestRowProps, WorkloadIOEmphasis } from './guestRowModel'; import { getGuestColumnStyle } from './guestRowModel'; -// Disk-read reasons that point at something broken the user can fix, as -// opposed to a guest that simply has no agent or is not running. -const DISK_READ_ACTIONABLE_REASONS = new Set(['permission-denied', 'agent-error']); - export function GuestRow(props: GuestRowProps) { const [rowActionProps] = splitProps(props, ['onClick']); const selectionGuard = createRowTextSelectionGuard(); @@ -82,6 +78,7 @@ export function GuestRow(props: GuestRowProps) { discoveryReadinessPresentation, diskRead, diskThresholds, + diskUsageRead, diskWrite, displayId, dockerHostId, @@ -286,7 +283,9 @@ export function GuestRow(props: GuestRowProps) { }; const getDiskStatusTooltip = () => { - if (!isVM(props.guest)) return 'Disk stats unavailable'; + const message = diskUsageRead().message; + if (message) return message; + if (!isVM(props.guest) || props.guest.disksFromAgent) return 'Disk stats unavailable'; const vm = props.guest as VM; return getWorkloadGuestDiskStatusMessage(vm.diskStatusReason); @@ -294,10 +293,9 @@ export function GuestRow(props: GuestRowProps) { // A native title or bar tooltip is not a visible freshness cue on touch, // and disappears from the chart's accessible value when the history lens opens. const diskReadStatus = createMemo(() => { - const guest = props.guest; - if (!isVM(guest) || !guest.diskStatusReason) return undefined; - const reason = guest.diskStatusReason; - const retained = reason.startsWith('prev-'); + const read = diskUsageRead(); + if (read.state === 'current') return undefined; + const retained = read.state === 'last-known'; return { label: retained ? usesCompactTableLayout() @@ -306,11 +304,11 @@ export function GuestRow(props: GuestRowProps) { : usesCompactTableLayout() ? 'N/A' : 'Unavailable', - message: getDiskStatusTooltip(), + message: read.message, valueLabelContext: retained ? ('last known' as const) : ('current' as const), // Most reasons are a setup gap (no guest agent, VM stopped) rather than a // fault, so they stay muted; only a broken read the user can fix is amber. - needsAction: DISK_READ_ACTIONABLE_REASONS.has(reason.replace(/^prev-/, '')), + needsAction: read.needsAction, }; }); const diskValueLabelContext = (): MetricMiniSparklineValueLabelContext => @@ -632,11 +630,7 @@ export function GuestRow(props: GuestRowProps) { aggregateDisk={props.guest.disk} anomaly={diskAnomaly()} thresholds={diskThresholds()} - statusMessage={ - isVM(props.guest) && props.guest.diskStatusReason - ? getDiskStatusTooltip() - : undefined - } + statusMessage={diskUsageRead().message ?? undefined} /> } > diff --git a/frontend-modern/src/components/Workloads/__tests__/GuestRow.test.tsx b/frontend-modern/src/components/Workloads/__tests__/GuestRow.test.tsx index 36317abf96..384b5ce602 100644 --- a/frontend-modern/src/components/Workloads/__tests__/GuestRow.test.tsx +++ b/frontend-modern/src/components/Workloads/__tests__/GuestRow.test.tsx @@ -1076,6 +1076,61 @@ describe('GuestRow', () => { ); expect(container.querySelector('[data-workload-disk-read-status]')).toBeNull(); }); + + it("follows a linked agent's filesystems, not the Proxmox read reason", () => { + // Proxmox has no guest filesystems, so the agent's fill in while + // Proxmox still reports its own agent-not-running. + const agentGuest = (overrides: Partial = {}) => + makeGuest({ + diskStatusReason: 'agent-not-running', + disksFromAgent: true, + disks: [makeDisk({ mountpoint: '/' })], + ...overrides, + }); + const [guest, setGuest] = createSignal(agentGuest()); + const { container } = render(() => ( + + + + +
+ )); + expect(screen.getByTestId('disk-bar')).not.toHaveAttribute('data-status-message'); + expect(container.querySelector('[data-workload-disk-read-status]')).toBeNull(); + + const lastKnown = + 'Using last known disk stats. The Pulse Agent in this guest stopped reporting.'; + setGuest(agentGuest({ agentStale: true })); + expect(screen.getByTestId('disk-bar')).toHaveAttribute('data-status-message', lastKnown); + let notice = container.querySelector('[data-workload-disk-read-status]'); + expect(notice?.querySelector('[aria-hidden="true"]')).toHaveTextContent('Last known'); + expect(notice).toHaveAttribute('title', lastKnown); + expect(notice?.className).toContain('text-muted'); + + setGuest(agentGuest({ agentStale: true, status: 'stopped', diskStatusReason: 'vm-stopped' })); + expect(screen.queryByTestId('disk-bar')).toBeNull(); + notice = container.querySelector('[data-workload-disk-read-status]'); + expect(notice?.querySelector('[aria-hidden="true"]')).toHaveTextContent('Unavailable'); + expect(notice).toHaveAttribute( + 'title', + 'Guest filesystem stats unavailable while the VM is stopped.', + ); + }); + + it("does not announce a silent linked agent's sparkline reading as current", () => { + renderGuestRow({ + guest: makeGuest({ + diskStatusReason: 'agent-not-running', + disksFromAgent: true, + agentStale: true, + }), + visibleColumnIds: ['name', 'disk'], + metricDisplayMode: 'sparklines', + }); + expect( + screen.getByRole('img', { name: 'test-vm disk usage history, last known 50%' }), + ).toBeInTheDocument(); + }); }); describe('lock label', () => { diff --git a/frontend-modern/src/components/Workloads/guestDrawerModel.ts b/frontend-modern/src/components/Workloads/guestDrawerModel.ts index 27fa776ad3..8f6c8c0aa5 100644 --- a/frontend-modern/src/components/Workloads/guestDrawerModel.ts +++ b/frontend-modern/src/components/Workloads/guestDrawerModel.ts @@ -13,6 +13,7 @@ import { } from '@/features/alerts/metricAlertPresentation'; import { formatBytes, formatPercent, getBackupInfo, type BackupThresholds } from '@/utils/format'; import { + getWorkloadGuestDiskRead, getWorkloadGuestDiskStatusMessage, getWorkloadsGuestBackupStatusPresentation, } from '@/utils/workloadGuestPresentation'; @@ -153,12 +154,13 @@ export const getGuestDrawerDeferredMetrics = ( guest: Guest, ): Record => { const metrics: Record = {}; - if (isGuestDrawerVM(guest) && guest.diskStatusReason) { + // History judges the aggregate disk metric by the same rule as the + // filesystem rows, so a linked agent's reading follows that agent. + const disk = getWorkloadGuestDiskRead(guest, isGuestDrawerVM(guest)); + if (disk.state !== 'current') { metrics.disk = { - lastKnownValue: guest.diskStatusReason.startsWith('prev-') - ? getGuestDrawerDiskUsage(guest) - : undefined, - message: getWorkloadGuestDiskStatusMessage(guest.diskStatusReason), + lastKnownValue: disk.state === 'last-known' ? getGuestDrawerDiskUsage(guest) : undefined, + message: disk.message, }; } const memory = getGuestDrawerMemoryReading(guest); @@ -188,7 +190,9 @@ export const getGuestDrawerCurrentMetrics = (guest: Guest): Record typeof value === 'number' && Number.isFinite(value) ? value : undefined; return { diff --git a/frontend-modern/src/components/Workloads/useGuestRowState.ts b/frontend-modern/src/components/Workloads/useGuestRowState.ts index 7b5d58bfc5..bdbc76c2b2 100644 --- a/frontend-modern/src/components/Workloads/useGuestRowState.ts +++ b/frontend-modern/src/components/Workloads/useGuestRowState.ts @@ -9,6 +9,7 @@ import { buildMetricKey } from '@/utils/metricsKeys'; import { getGuestHealthIndicator, isGuestRunning } from '@/utils/status'; import { formatBytes } from '@/utils/format'; import { getContainerRuntimeBadgeForRuntime } from '@/utils/resourceBadgePresentation'; +import { getWorkloadGuestDiskRead } from '@/utils/workloadGuestPresentation'; import { getCanonicalWorkloadId, getWorkloadPlatformScopes, @@ -225,21 +226,21 @@ export function useGuestRowState(props: GuestRowProps) { return ((disk.used ?? 0) / total) * 100; }); + // The cell's filesystem bars and aggregate follow the drawer's rule: a + // linked agent's filesystems follow that agent, Proxmox's its read reason. + const diskUsageRead = createMemo(() => + getWorkloadGuestDiskRead(props.guest, workloadType() === 'vm'), + ); + const hasDiskUsage = createMemo(() => { const guest = props.guest; if (!guest.disk || guest.telemetryAvailability?.disk === false) return false; const total = guest.disk.total ?? 0; if (!Number.isFinite(total) || total <= 0) return false; // A generic canonical merge can retain the previous numeric metric while - // the provider explicitly reports no usable guest reading. Only prev- - // reasons allow that number to remain visible as last-known evidence. - if ( - workloadType() === 'vm' && - 'diskStatusReason' in guest && - guest.diskStatusReason && - !guest.diskStatusReason.startsWith('prev-') - ) - return false; + // its source reports no usable guest reading. Only a last known reading + // may keep that number visible. + if (diskUsageRead().state === 'unavailable') return false; const percent = diskPercent(); return Number.isFinite(percent) && percent >= 0; }); @@ -315,6 +316,7 @@ export function useGuestRowState(props: GuestRowProps) { discoveryReadinessPresentation, diskRead, diskThresholds, + diskUsageRead, diskWrite, displayId, dockerHostId, diff --git a/frontend-modern/src/components/shared/SharedPrimitives.guardrails.test.ts b/frontend-modern/src/components/shared/SharedPrimitives.guardrails.test.ts index e5823364b6..fc0c57503f 100644 --- a/frontend-modern/src/components/shared/SharedPrimitives.guardrails.test.ts +++ b/frontend-modern/src/components/shared/SharedPrimitives.guardrails.test.ts @@ -1350,7 +1350,6 @@ describe('shared primitive guardrails', () => { expect(contextualFocusSource).toContain('export const revealInlineDetailInViewport'); expect(contextualFocusSource).toContain('markRouteStateDeliberateScroll'); expect(contextualFocusSource).toContain('data-inline-detail-for'); - expect(summaryCardInteractionSource).toContain('chartHoveredSeriesId'); expect(summaryCardInteractionSource).toContain('SummarySeriesGroupScope'); expect(summaryCardInteractionSource).toContain('resolveSummaryGroupScope'); expect(summaryCardInteractionSource).toContain('resolveSummaryGroupMemberInteractionState'); @@ -1364,7 +1363,6 @@ describe('shared primitive guardrails', () => { expect(summaryTableFocusSource).toContain('export function useSummaryTableFocusBridge'); expect(summaryTableFocusSource).toContain('export function useSummaryPageInteractionState'); expect(summaryTableFocusSource).toContain('resolveSummaryActiveSeriesId'); - expect(summaryTableFocusSource).toContain('activeScopeState'); expect(summaryTableFocusSource).toContain('focusedSeriesId'); expect(summaryTableFocusSource).toContain('findInlineDetailElement'); expect(summaryTableFocusSource).toContain('revealInlineDetailInViewport'); @@ -1376,8 +1374,9 @@ describe('shared primitive guardrails', () => { expect(summaryTableFocusSource).toContain("event.key !== 'Escape'"); expect(summaryTableFocusSource).toContain('querySelector('); expect(summaryTableFocusSource).toContain( - "row.scrollIntoView({ behavior: 'smooth', block: 'center' })", + "row.scrollIntoView({ behavior: 'smooth', block: 'nearest' })", ); + expect(summaryTableFocusSource).not.toContain("window.addEventListener('scroll'"); expect(summaryTableFocusSource).not.toContain('useNavigate('); }); diff --git a/frontend-modern/src/components/shared/__tests__/summaryCardInteraction.branchcov0724pm.test.ts b/frontend-modern/src/components/shared/__tests__/summaryCardInteraction.branchcov0724pm.test.ts index 33283ace20..8b995327f5 100644 --- a/frontend-modern/src/components/shared/__tests__/summaryCardInteraction.branchcov0724pm.test.ts +++ b/frontend-modern/src/components/shared/__tests__/summaryCardInteraction.branchcov0724pm.test.ts @@ -185,12 +185,12 @@ describe('summaryCardInteraction.branchcov0724pm', () => { }); }); - it('prefers a valid chartHoveredSeriesId over the hoveredGroupScope group return, confirming b37 is precedence-gated (regression guard)', () => { - // chartHoveredSeriesId 'alpha' is in the resolved scope -> entity/preview + it('prefers a hovered series inside the scope over the hoveredGroupScope group return, confirming b37 is precedence-gated (regression guard)', () => { + // hoveredSeriesId 'alpha' is in the resolved scope -> entity/preview // return wins; the hoveredGroupScope group arm (b37) is NOT taken here. expect( resolveSummaryScopeState({ - chartHoveredSeriesId: 'alpha', + hoveredSeriesId: 'alpha', hoveredGroupScope: validScope, }), ).toEqual({ diff --git a/frontend-modern/src/components/shared/__tests__/summaryCardInteraction.test.ts b/frontend-modern/src/components/shared/__tests__/summaryCardInteraction.test.ts index a1fe5684e5..9d8c2d4417 100644 --- a/frontend-modern/src/components/shared/__tests__/summaryCardInteraction.test.ts +++ b/frontend-modern/src/components/shared/__tests__/summaryCardInteraction.test.ts @@ -23,14 +23,6 @@ describe('summaryCardInteraction', () => { }); it('prefers hovered series ids and falls back to focused ids', () => { - expect( - resolveSummaryActiveSeriesId({ - chartHoveredSeriesId: 'gamma', - hoveredSeriesId: 'beta', - focusedSeriesId: 'alpha', - }), - ).toBe('gamma'); - expect( resolveSummaryActiveSeriesId({ hoveredSeriesId: 'beta', @@ -71,8 +63,7 @@ describe('summaryCardInteraction', () => { expect( resolveSummaryScopeState({ - chartHoveredSeriesId: 'beta', - hoveredSeriesId: 'alpha', + hoveredSeriesId: 'beta', focusedSeriesId: 'alpha', hoveredGroupScope: groupScope, focusedGroupScope: groupScope, diff --git a/frontend-modern/src/components/shared/__tests__/summaryTableFocus.branchcov0723pm.test.ts b/frontend-modern/src/components/shared/__tests__/summaryTableFocus.branchcov0723pm.test.ts index 09d1b4455f..5b5c58a922 100644 --- a/frontend-modern/src/components/shared/__tests__/summaryTableFocus.branchcov0723pm.test.ts +++ b/frontend-modern/src/components/shared/__tests__/summaryTableFocus.branchcov0723pm.test.ts @@ -9,12 +9,9 @@ import { // Branch-coverage tests for the UNCOVERED guard arms of summaryTableFocus.ts. // The existing spec (summaryTableFocus.test.tsx) exercises the happy paths; // this file targets the null/empty/early-return arms that never fire there: -// - scheduleViewportRefresh (scroll/resize -> viewportVersion tick) โ€” the -// primary measured gap; never invoked by the existing suite. -// - activeRow / focusedGroupRow guards: missing container ref, empty id, -// querySelector miss, attribute-selector escaping. -// - jumpToActiveRow: no-active-id no-op + frame-exhaustion arm. -// - shouldShowJumpToActiveRow: empty summary collections + already-visible. +// - focusedGroupRow guards: missing container ref, empty id, querySelector +// miss, attribute-selector escaping. +// - page-default and hover-only active series resolution. // - Escape handler: defaultPrevented, every modifier arm, dialog target. // Every asserted value below is hand-computed against the source in // src/components/shared/summaryTableFocus.ts โ€” no snapshots, no constant- @@ -50,326 +47,140 @@ describe('summaryTableFocus.branchcov0723pm', () => { }); // ------------------------------------------------------------------------- - // PRIMARY TARGET: scheduleViewportRefresh - // (createEffect at summaryTableFocus.ts:226). The existing spec never - // dispatches scroll/resize after mounting a table root, so the entire - // effect body โ€” listener registration, the rAF callback that ticks - // viewportVersion, the dedup guard, and the cancel-on-cleanup arm โ€” is - // uncovered. + // focusedGroupRow guard arms: missing container ref, empty id, querySelector + // miss, attribute-selector escaping. // ------------------------------------------------------------------------- - describe('scheduleViewportRefresh', () => { - it('ticks viewportVersion via the rAF callback after a scroll event, recomputing isActiveRowVisible', () => { - // Deferred rAF: capture the callback so we control when viewportVersion - // increments (the synchronous default stub would tick instantly and hide - // the dedup arm tested below). - const scheduled: FrameRequestCallback[] = []; - let nextId = 7; - vi.stubGlobal('requestAnimationFrame', (cb: FrameRequestCallback) => { - scheduled.push(cb); - return nextId++; - }); - - const root = document.createElement('div'); - const row = document.createElement('div'); - row.setAttribute('data-summary-series-id', 'workload-a'); - let rowTop = 1200; // below innerHeight(800) -> not visible - Object.defineProperty(row, 'getBoundingClientRect', { - configurable: true, - value: () => buildRect(rowTop), - }); - root.appendChild(row); - document.body.appendChild(root); - - const [hoveredSeriesId] = createSignal('workload-a'); - const { result } = renderHook(() => useSummaryPageInteractionState({ hoveredSeriesId })); - result.setTableRootRef(root); - - // Row below the viewport -> jump affordance shown. - expect(result.shouldShowJumpToActiveRow()).toBe(true); - - // Slide the row into the viewport. The memo must NOT pick this up yet - // because viewportVersion has not ticked (no rAF has fired). - rowTop = 64; - expect(result.shouldShowJumpToActiveRow()).toBe(true); - - // scroll -> scheduleViewportRefresh schedules exactly one rAF. - window.dispatchEvent(new Event('scroll')); - expect(scheduled).toHaveLength(1); - - // Fire the rAF -> viewportVersion++ -> isActiveRowVisible recomputes -> - // row is now in-viewport -> affordance hidden. - scheduled[0](0); - expect(result.shouldShowJumpToActiveRow()).toBe(false); - }); - - it('coalesces back-to-back scroll events into a single rAF (rafId !== undefined dedup arm)', () => { - const scheduled: FrameRequestCallback[] = []; - let nextId = 3; - vi.stubGlobal('requestAnimationFrame', (cb: FrameRequestCallback) => { - scheduled.push(cb); - return nextId++; - }); - - const root = document.createElement('div'); - document.body.appendChild(root); - - const { result } = renderHook(() => - useSummaryTableFocusBridge({ activeSeriesId: () => null }), - ); - result.setTableRootRef(root); - - window.dispatchEvent(new Event('scroll')); - window.dispatchEvent(new Event('scroll')); - window.dispatchEvent(new Event('scroll')); - - // Only the first scroll schedules an rAF; subsequent ones hit - // `if (rafId !== undefined) return;` while the callback is pending. - expect(scheduled).toHaveLength(1); - }); - - it('refreshes viewportVersion on window resize as well as scroll (resize listener arm)', () => { - const scheduled: FrameRequestCallback[] = []; - let nextId = 11; - vi.stubGlobal('requestAnimationFrame', (cb: FrameRequestCallback) => { - scheduled.push(cb); - return nextId++; - }); - - const root = document.createElement('div'); - const row = document.createElement('div'); - row.setAttribute('data-summary-series-id', 'workload-a'); - let rowTop = 900; - Object.defineProperty(row, 'getBoundingClientRect', { - configurable: true, - value: () => buildRect(rowTop), - }); - root.appendChild(row); - document.body.appendChild(root); - - const [hoveredSeriesId] = createSignal('workload-a'); - const { result } = renderHook(() => useSummaryPageInteractionState({ hoveredSeriesId })); - result.setTableRootRef(root); - expect(result.shouldShowJumpToActiveRow()).toBe(true); - - rowTop = 50; - window.dispatchEvent(new Event('resize')); - expect(scheduled).toHaveLength(1); - scheduled[0](0); - expect(result.shouldShowJumpToActiveRow()).toBe(false); - }); - - it('cancels the pending rAF id when the owner is disposed (cancelAnimationFrame cleanup arm)', () => { - const scheduled: FrameRequestCallback[] = []; - const cancelSpy = vi.fn(); - const pendingId = 42; - vi.stubGlobal('requestAnimationFrame', (cb: FrameRequestCallback) => { - scheduled.push(cb); - return pendingId; - }); - vi.stubGlobal('cancelAnimationFrame', cancelSpy); - - const root = document.createElement('div'); - document.body.appendChild(root); - - const { result, cleanup } = renderHook(() => - useSummaryTableFocusBridge({ activeSeriesId: () => null }), - ); - result.setTableRootRef(root); - - window.dispatchEvent(new Event('scroll')); - expect(cancelSpy).not.toHaveBeenCalled(); - - cleanup(); - - // The rAF id returned by the deferred stub (42) is cancelled on dispose. - expect(cancelSpy).toHaveBeenCalledWith(pendingId); - }); - - it('does not attach scroll/resize listeners when tableRoot ref is missing (missing container ref arm)', () => { + describe('focusedGroupRow guard arms', () => { + it('does not attempt a reveal when setTableRootRef receives undefined (setter `element ?? null` right arm)', () => { const rafSpy = vi.fn((cb: FrameRequestCallback) => { cb(0); return 1; }); vi.stubGlobal('requestAnimationFrame', rafSpy); - renderHook(() => useSummaryTableFocusBridge({ activeSeriesId: () => null })); + const [focusedGroupId] = createSignal('group-a'); + const { result } = renderHook(() => useSummaryTableFocusBridge({ focusedGroupId })); + result.setTableRootRef(undefined); - // No setTableRootRef -> effect hits `if (!root ...) return;` -> no - // listeners -> scroll/resize schedule no rAF. - window.dispatchEvent(new Event('scroll')); - window.dispatchEvent(new Event('resize')); + // tableRoot stays null -> the reveal effect returns before scheduling. expect(rafSpy).not.toHaveBeenCalled(); }); - }); - // ------------------------------------------------------------------------- - // activeRow guard arms: missing container ref, empty id, querySelector miss, - // attribute-selector escaping. - // ------------------------------------------------------------------------- - describe('activeRow guard arms', () => { - it('returns null when the tableRoot ref has never been set (missing container ref)', () => { - const { result } = renderHook(() => - useSummaryTableFocusBridge({ activeSeriesId: () => 'workload-a' }), - ); - expect(result.activeRow()).toBeNull(); - }); + it('does not attempt a reveal for a whitespace-only focused group id (normalizeSeriesId guard)', () => { + const rafSpy = vi.fn((cb: FrameRequestCallback) => { + cb(0); + return 1; + }); + vi.stubGlobal('requestAnimationFrame', rafSpy); - it('returns null when setTableRootRef receives undefined (setter `element ?? null` right arm)', () => { - const { result } = renderHook(() => - useSummaryTableFocusBridge({ activeSeriesId: () => 'workload-a' }), - ); - result.setTableRootRef(undefined); - expect(result.activeRow()).toBeNull(); - }); - - it('returns null for null / whitespace-only active ids and the row for a real id (normalizedActiveSeriesId guard)', () => { const root = document.createElement('div'); - const row = document.createElement('div'); - row.setAttribute('data-summary-series-id', 'workload-a'); - root.appendChild(row); document.body.appendChild(root); - const [activeSeriesId, setActiveSeriesId] = createSignal(null); - const { result } = renderHook(() => useSummaryTableFocusBridge({ activeSeriesId })); + const [focusedGroupId] = createSignal(' '); + const { result } = renderHook(() => useSummaryTableFocusBridge({ focusedGroupId })); result.setTableRootRef(root); - expect(result.activeRow()).toBeNull(); - setActiveSeriesId(' '); - expect(result.activeRow()).toBeNull(); - setActiveSeriesId('workload-a'); - expect(result.activeRow()).toBe(row); + expect(rafSpy).not.toHaveBeenCalled(); }); - it('returns null when no descendant carries the matching data attribute (querySelector miss)', () => { + it('does not scroll a decoy row when no group row carries the matching id (querySelector miss)', () => { + const scrollIntoView = vi.fn(); const root = document.createElement('div'); const decoy = document.createElement('div'); - decoy.setAttribute('data-summary-series-id', 'other'); + decoy.setAttribute('data-summary-group-id', 'other'); + Object.defineProperty(decoy, 'getBoundingClientRect', { + configurable: true, + value: () => buildRect(1200, 40), + }); + Object.defineProperty(decoy, 'scrollIntoView', { + configurable: true, + value: scrollIntoView, + }); root.appendChild(decoy); document.body.appendChild(root); - const { result } = renderHook(() => - useSummaryTableFocusBridge({ activeSeriesId: () => 'workload-a' }), - ); + const [focusedGroupId] = createSignal('group-a'); + const { result } = renderHook(() => useSummaryTableFocusBridge({ focusedGroupId })); result.setTableRootRef(root); - expect(result.activeRow()).toBeNull(); + expect(scrollIntoView).not.toHaveBeenCalled(); }); - it('escapes backslashes and double quotes in the series id before injecting it into the attribute selector', () => { + it('escapes backslashes and double quotes in the group id before injecting it into the attribute selector', () => { + const scrollIntoView = vi.fn(); const root = document.createElement('div'); const row = document.createElement('div'); // Literal id containing both a double-quote and a backslash โ€” the exact // characters escapeAttributeSelectorValue rewrites. - row.setAttribute('data-summary-series-id', 'id"with\\special'); + row.setAttribute('data-summary-group-id', 'id"with\\special'); + Object.defineProperty(row, 'getBoundingClientRect', { + configurable: true, + value: () => buildRect(1200, 40), + }); + Object.defineProperty(row, 'scrollIntoView', { + configurable: true, + value: scrollIntoView, + }); root.appendChild(row); document.body.appendChild(root); - const { result } = renderHook(() => - useSummaryTableFocusBridge({ activeSeriesId: () => 'id"with\\special' }), - ); + const [focusedGroupId] = createSignal('id"with\\special'); + const { result } = renderHook(() => useSummaryTableFocusBridge({ focusedGroupId })); result.setTableRootRef(root); // If escaping were broken the selector would not match the literal id. - expect(result.activeRow()).toBe(row); + expect(scrollIntoView).toHaveBeenCalledWith({ behavior: 'smooth', block: 'nearest' }); }); }); // ------------------------------------------------------------------------- - // jumpToActiveRow guard arms + // Active series resolution โ€” page default and transient hover // ------------------------------------------------------------------------- - describe('jumpToActiveRow guard arms', () => { - it('is a no-op when there is no active series id (normalizedActiveSeriesId falsy -> early return)', () => { - const revealActiveSeries = vi.fn(); - const rafSpy = vi.fn((cb: FrameRequestCallback) => { - cb(0); - return 1; - }); - vi.stubGlobal('requestAnimationFrame', rafSpy); - - const root = document.createElement('div'); - document.body.appendChild(root); - - const { result } = renderHook(() => - useSummaryTableFocusBridge({ - activeSeriesId: () => null, - revealActiveSeries, - }), - ); - result.setTableRootRef(root); - - result.jumpToActiveRow(); - - expect(revealActiveSeries).not.toHaveBeenCalled(); - expect(rafSpy).not.toHaveBeenCalled(); - }); - - it('exhausts the 6-frame retry budget without scrolling when the active row never mounts (no matching row)', () => { - const revealActiveSeries = vi.fn(); - const rafSpy = vi.fn((cb: FrameRequestCallback) => { - cb(0); - return 1; - }); - vi.stubGlobal('requestAnimationFrame', rafSpy); - - const root = document.createElement('div'); - document.body.appendChild(root); - - const { result } = renderHook(() => - useSummaryTableFocusBridge({ - activeSeriesId: () => 'workload-a', - revealActiveSeries, - }), - ); - result.setTableRootRef(root); - - result.jumpToActiveRow(); - - // revealActiveSeries fires once before the scroll retry loop. - expect(revealActiveSeries).toHaveBeenCalledWith('workload-a'); - expect(revealActiveSeries).toHaveBeenCalledTimes(1); - // attemptScroll(6) recurses once per synchronous rAF until remainingHits - // 0 -> 6 rAF calls total, then the `remainingFrames <= 0` arm returns. - expect(rafSpy).toHaveBeenCalledTimes(6); - }); - }); - - // ------------------------------------------------------------------------- - // shouldShowJumpToActiveRow โ€” empty summary collections + already-visible row - // ------------------------------------------------------------------------- - describe('shouldShowJumpToActiveRow arms', () => { - it('is false when no hover/focus/group signal resolves an active series (empty summary collections / page-default state)', () => { + describe('active series resolution arms', () => { + it('resolves no active series or group when no hover/focus/group signal is set (page-default state)', () => { const root = document.createElement('div'); document.body.appendChild(root); const { result } = renderHook(() => useSummaryPageInteractionState({})); result.setTableRootRef(root); - // resolveSummaryScopeState returns { kind: 'page', seriesId: null }. expect(result.activeSeriesId()).toBeNull(); expect(result.activeGroupScope()).toBeNull(); - // Boolean(null) && !isActiveRowVisible() -> false && ... -> false. - expect(result.shouldShowJumpToActiveRow()).toBe(false); }); - it('is false when the active row is already inside the viewport (already-focused / visible arm)', () => { + it('highlights a hovered off-screen row without moving the page (hover is not a reveal)', () => { + const scrollIntoView = vi.fn(); + const scrollTo = vi.fn(); + Object.defineProperty(window, 'scrollTo', { + configurable: true, + value: scrollTo, + }); const root = document.createElement('div'); const row = document.createElement('div'); row.setAttribute('data-summary-series-id', 'workload-a'); - // top=200, height=40, innerHeight=800 -> 0 < 200 and 240 < 800 -> visible. + // top=1200 > innerHeight(800) -> off-screen. Object.defineProperty(row, 'getBoundingClientRect', { configurable: true, - value: () => buildRect(200, 40), + value: () => buildRect(1200, 40), + }); + Object.defineProperty(row, 'scrollIntoView', { + configurable: true, + value: scrollIntoView, }); root.appendChild(row); document.body.appendChild(root); + const revealActiveSeries = vi.fn(); const [hoveredSeriesId] = createSignal('workload-a'); - const { result } = renderHook(() => useSummaryPageInteractionState({ hoveredSeriesId })); + const { result } = renderHook(() => + useSummaryPageInteractionState({ hoveredSeriesId, revealActiveSeries }), + ); result.setTableRootRef(root); expect(result.activeSeriesId()).toBe('workload-a'); - // Boolean('workload-a') && !true -> false. - expect(result.shouldShowJumpToActiveRow()).toBe(false); + expect(revealActiveSeries).not.toHaveBeenCalled(); + expect(scrollIntoView).not.toHaveBeenCalled(); + expect(scrollTo).not.toHaveBeenCalled(); }); }); @@ -395,9 +206,7 @@ describe('summaryTableFocus.branchcov0723pm', () => { document.body.appendChild(root); const [focusedGroupId] = createSignal('group-a'); - const { result } = renderHook(() => - useSummaryTableFocusBridge({ activeSeriesId: () => null, focusedGroupId }), - ); + const { result } = renderHook(() => useSummaryTableFocusBridge({ focusedGroupId })); result.setTableRootRef(root); // `!isElementVisibleWithinViewport(row) && ...` short-circuits on the @@ -422,9 +231,7 @@ describe('summaryTableFocus.branchcov0723pm', () => { document.body.appendChild(root); const [focusedGroupId] = createSignal('group-a'); - const { result } = renderHook(() => - useSummaryTableFocusBridge({ activeSeriesId: () => null, focusedGroupId }), - ); + const { result } = renderHook(() => useSummaryTableFocusBridge({ focusedGroupId })); result.setTableRootRef(root); expect(scrollIntoView).toHaveBeenCalledWith({ @@ -444,9 +251,7 @@ describe('summaryTableFocus.branchcov0723pm', () => { document.body.appendChild(root); const [focusedGroupId] = createSignal('group-a'); - const { result } = renderHook(() => - useSummaryTableFocusBridge({ activeSeriesId: () => null, focusedGroupId }), - ); + const { result } = renderHook(() => useSummaryTableFocusBridge({ focusedGroupId })); result.setTableRootRef(root); // Initial rAF + one per decrement: remainingFrames 12->0 = 13 calls. diff --git a/frontend-modern/src/components/shared/__tests__/summaryTableFocus.test.tsx b/frontend-modern/src/components/shared/__tests__/summaryTableFocus.test.tsx index a995072b60..1448e4de40 100644 --- a/frontend-modern/src/components/shared/__tests__/summaryTableFocus.test.tsx +++ b/frontend-modern/src/components/shared/__tests__/summaryTableFocus.test.tsx @@ -1,4 +1,4 @@ -import { renderHook } from '@solidjs/testing-library'; +import { renderHook, waitFor } from '@solidjs/testing-library'; import { createSignal } from 'solid-js'; import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; import { useSummaryPageInteractionState } from '@/components/shared/summaryTableFocus'; @@ -30,8 +30,8 @@ describe('useSummaryPageInteractionState', () => { vi.unstubAllGlobals(); }); - it('prefers chart hover over row hover and route focus for the active series id', () => { - const [hoveredSeriesId] = createSignal('row-hovered'); + it('prefers row hover over route focus for the active series id', () => { + const [hoveredSeriesId, setHoveredSeriesId] = createSignal('row-hovered'); const [focusedSeriesId] = createSignal('route-focused'); const { result } = renderHook(() => @@ -43,80 +43,72 @@ describe('useSummaryPageInteractionState', () => { expect(result.activeSeriesId()).toBe('row-hovered'); - result.setChartHoverSync({ - sourceKey: 'cpu', - seriesId: 'chart-hovered', - timestamp: 123, - }); + setHoveredSeriesId(null); - expect(result.activeSeriesId()).toBe('chart-hovered'); + expect(result.activeSeriesId()).toBe('route-focused'); }); - it('shows a jump affordance when the active row is mounted but outside the viewport', () => { + it('does not subscribe to window scroll or resize', () => { + const addEventListener = vi.spyOn(window, 'addEventListener'); const [hoveredSeriesId] = createSignal('workload-a'); - const scrollIntoView = vi.fn(); + const [focusedSeriesId] = createSignal('workload-b'); const root = document.createElement('div'); - const row = document.createElement('div'); - row.setAttribute('data-summary-series-id', 'workload-a'); - Object.defineProperty(row, 'getBoundingClientRect', { - configurable: true, - value: () => buildRect(1200), - }); - Object.defineProperty(row, 'scrollIntoView', { - configurable: true, - value: scrollIntoView, - }); - root.appendChild(row); document.body.appendChild(root); const { result } = renderHook(() => useSummaryPageInteractionState({ hoveredSeriesId, + focusedSeriesId, }), ); result.setTableRootRef(root); - expect(result.shouldShowJumpToActiveRow()).toBe(true); - - result.jumpToActiveRow(); - - expect(scrollIntoView).toHaveBeenCalledWith({ behavior: 'smooth', block: 'center' }); + const subscribed = addEventListener.mock.calls.map(([type]) => type); + addEventListener.mockRestore(); + expect(subscribed).not.toContain('scroll'); + expect(subscribed).not.toContain('resize'); }); - it('reveals an unmapped row before attempting to scroll to it', () => { - const [hoveredSeriesId] = createSignal('pool-alpha'); - const scrollIntoView = vi.fn(); + it('reveals a focused row that only mounts once its owning view opens', async () => { + const [focusedSeriesId] = createSignal('pool-alpha'); + const scrollTo = vi.fn(); const root = document.createElement('div'); const row = document.createElement('div'); - row.setAttribute('data-summary-series-id', 'pool-alpha'); - Object.defineProperty(row, 'getBoundingClientRect', { - configurable: true, - value: () => buildRect(64), - }); - Object.defineProperty(row, 'scrollIntoView', { - configurable: true, - value: scrollIntoView, - }); + const detail = document.createElement('div'); + Object.defineProperty(window, 'scrollTo', { + configurable: true, + value: scrollTo, + }); + vi.stubGlobal('scrollY', 0); + + row.setAttribute('data-summary-series-id', 'pool-alpha'); + row.getBoundingClientRect = vi.fn(() => buildRect(680, 40)); + detail.setAttribute('data-inline-detail-for', 'pool-alpha'); + detail.getBoundingClientRect = vi.fn(() => buildRect(724, 220)); + document.body.appendChild(root); + + // The owning view renders the row after the reveal request returns, so the + // bridge must pick it up through its mutation observer. const revealActiveSeries = vi.fn(() => { - root.appendChild(row); + window.setTimeout(() => root.append(row, detail), 0); }); const { result } = renderHook(() => useSummaryPageInteractionState({ - hoveredSeriesId, + focusedSeriesId, revealActiveSeries, }), ); result.setTableRootRef(root); - expect(result.shouldShowJumpToActiveRow()).toBe(true); - - result.jumpToActiveRow(); expect(revealActiveSeries).toHaveBeenCalledWith('pool-alpha'); - expect(scrollIntoView).toHaveBeenCalledWith({ behavior: 'smooth', block: 'center' }); + expect(scrollTo).not.toHaveBeenCalled(); + await waitFor(() => { + expect(scrollTo).toHaveBeenCalledWith({ top: 456, behavior: 'smooth' }); + }); }); it('reveals focused inline detail without hard-centering the row', () => { diff --git a/frontend-modern/src/components/shared/contextualFocus.ts b/frontend-modern/src/components/shared/contextualFocus.ts index ad063d931b..8b19aaad6d 100644 --- a/frontend-modern/src/components/shared/contextualFocus.ts +++ b/frontend-modern/src/components/shared/contextualFocus.ts @@ -4,12 +4,6 @@ const INLINE_DETAIL_TARGET_TOP_RATIO = 0.28; const INLINE_DETAIL_MIN_TOP_MARGIN = 72; const INLINE_DETAIL_MIN_DETAIL_PEEK = 160; -export interface SummaryChartHoverSync { - sourceKey: string; - seriesId: string; - timestamp: number; -} - const resolveScrollableAncestor = (element: Element | null | undefined): HTMLElement | null => { let scroller = element instanceof HTMLElement ? element : null; while (scroller) { diff --git a/frontend-modern/src/components/shared/summaryCardInteraction.ts b/frontend-modern/src/components/shared/summaryCardInteraction.ts index bce85138b4..7de92e64d7 100644 --- a/frontend-modern/src/components/shared/summaryCardInteraction.ts +++ b/frontend-modern/src/components/shared/summaryCardInteraction.ts @@ -88,7 +88,6 @@ export const resolveSummaryGroupMemberInteractionState = (options: { }; export const resolveSummaryScopeState = (options: { - chartHoveredSeriesId?: string | null; hoveredSeriesId?: string | null; focusedSeriesId?: string | null; hoveredGroupScope?: SummarySeriesGroupScope | null; @@ -102,19 +101,6 @@ export const resolveSummaryScopeState = (options: { focusedGroupScope: options.focusedGroupScope, }); - const chartHoveredSeriesId = normalizeSeriesId(options.chartHoveredSeriesId); - if ( - chartHoveredSeriesId && - (!groupScope || isSummarySeriesInGroupScope(groupScope, chartHoveredSeriesId)) - ) { - return { - groupScope, - kind: 'entity', - seriesId: chartHoveredSeriesId, - source: 'preview', - }; - } - const hoveredSeriesId = normalizeSeriesId(options.hoveredSeriesId); if ( hoveredSeriesId && @@ -170,7 +156,6 @@ export const resolveSummaryScopeState = (options: { }; export function resolveSummaryActiveSeriesId(options: { - chartHoveredSeriesId?: string | null; hoveredSeriesId?: string | null; focusedSeriesId?: string | null; groupScope?: SummarySeriesGroupScope | null; diff --git a/frontend-modern/src/components/shared/summaryTableFocus.ts b/frontend-modern/src/components/shared/summaryTableFocus.ts index cf4936f775..68d73100cb 100644 --- a/frontend-modern/src/components/shared/summaryTableFocus.ts +++ b/frontend-modern/src/components/shared/summaryTableFocus.ts @@ -1,15 +1,9 @@ import { createEffect, createMemo, createSignal, onCleanup, type Accessor } from 'solid-js'; -import { - findInlineDetailElement, - revealInlineDetailInViewport, - type SummaryChartHoverSync, -} from './contextualFocus'; +import { findInlineDetailElement, revealInlineDetailInViewport } from './contextualFocus'; import { resolveSummaryActiveSeriesId, - resolveSummaryScopeState, resolveSummaryGroupScope, type SummarySeriesGroupScope, - type SummaryScopeState, } from './summaryCardInteraction'; const normalizeSeriesId = (value: string | null | undefined): string => value?.trim() || ''; @@ -69,7 +63,6 @@ const isElementVisibleWithinViewport = (element: HTMLElement): boolean => { }; export interface UseSummaryTableFocusBridgeOptions { - activeSeriesId: Accessor; focusedSeriesId?: Accessor; focusedGroupId?: Accessor; clearPinnedScope?: () => void; @@ -80,26 +73,9 @@ export interface UseSummaryTableFocusBridgeOptions { export function useSummaryTableFocusBridge(options: UseSummaryTableFocusBridgeOptions) { const [tableRoot, setTableRoot] = createSignal(null); const [clearSurfaceRoot, setClearSurfaceRoot] = createSignal(null); - const [viewportVersion, setViewportVersion] = createSignal(0); const focusedSeriesId = options.focusedSeriesId ?? (() => null); const focusedGroupId = options.focusedGroupId ?? (() => null); - const normalizedActiveSeriesId = createMemo(() => { - const normalized = normalizeSeriesId(options.activeSeriesId()); - return normalized || null; - }); - - const activeRow = (): HTMLElement | null => { - const root = tableRoot(); - const activeSeriesId = normalizedActiveSeriesId(); - if (!root || !activeSeriesId) { - return null; - } - return root.querySelector( - `[data-summary-series-id="${escapeAttributeSelectorValue(activeSeriesId)}"]`, - ); - }; - const focusedRow = (): HTMLElement | null => { const root = tableRoot(); const focusedId = normalizeSeriesId(focusedSeriesId()); @@ -122,19 +98,6 @@ export function useSummaryTableFocusBridge(options: UseSummaryTableFocusBridgeOp ); }; - const isActiveRowVisible = createMemo(() => { - viewportVersion(); - const row = activeRow(); - if (!row) { - return false; - } - return isElementVisibleWithinViewport(row); - }); - - const shouldShowJumpToActiveRow = createMemo(() => { - return Boolean(normalizedActiveSeriesId()) && !isActiveRowVisible(); - }); - createEffect(() => { const root = clearSurfaceRoot() ?? tableRoot(); const lookupRoot = tableRoot(); @@ -200,58 +163,6 @@ export function useSummaryTableFocusBridge(options: UseSummaryTableFocusBridgeOp }); }); - const jumpToActiveRow = () => { - const activeSeriesId = normalizedActiveSeriesId(); - if (!activeSeriesId) { - return; - } - - options.revealActiveSeries?.(activeSeriesId); - - const attemptScroll = (remainingFrames: number) => { - const row = activeRow(); - if (row) { - row.scrollIntoView({ behavior: 'smooth', block: 'center' }); - return; - } - if (remainingFrames <= 0 || typeof window === 'undefined') { - return; - } - window.requestAnimationFrame(() => attemptScroll(remainingFrames - 1)); - }; - - attemptScroll(6); - }; - - createEffect(() => { - const root = tableRoot(); - if (!root || typeof window === 'undefined') { - return; - } - - let rafId: number | undefined; - const scheduleViewportRefresh = () => { - if (rafId !== undefined) { - return; - } - rafId = window.requestAnimationFrame(() => { - rafId = undefined; - setViewportVersion((current) => current + 1); - }); - }; - - window.addEventListener('scroll', scheduleViewportRefresh, true); - window.addEventListener('resize', scheduleViewportRefresh); - - onCleanup(() => { - window.removeEventListener('scroll', scheduleViewportRefresh, true); - window.removeEventListener('resize', scheduleViewportRefresh); - if (rafId !== undefined) { - window.cancelAnimationFrame(rafId); - } - }); - }); - createEffect(() => { const focusedId = normalizeSeriesId(focusedSeriesId()); const root = tableRoot(); @@ -384,13 +295,9 @@ export function useSummaryTableFocusBridge(options: UseSummaryTableFocusBridgeOp }); return { - activeRow, - isActiveRowVisible, - jumpToActiveRow, setClearSurfaceRootRef: (element: HTMLElement | undefined) => setClearSurfaceRoot(element ?? null), setTableRootRef: (element: HTMLElement | undefined) => setTableRoot(element ?? null), - shouldShowJumpToActiveRow, } as const; } @@ -406,7 +313,6 @@ export interface UseSummaryPageInteractionStateOptions { } export function useSummaryPageInteractionState(options: UseSummaryPageInteractionStateOptions) { - const [chartHoverSync, setChartHoverSync] = createSignal(null); const hoveredSeriesId = options.hoveredSeriesId ?? (() => null); const focusedSeriesId = options.focusedSeriesId ?? (() => null); const focusedGroupId = options.focusedGroupId ?? (() => null); @@ -420,20 +326,8 @@ export function useSummaryPageInteractionState(options: UseSummaryPageInteractio }), ); - const activeScopeState = createMemo(() => - resolveSummaryScopeState({ - chartHoveredSeriesId: chartHoverSync()?.seriesId ?? null, - hoveredSeriesId: hoveredSeriesId(), - focusedSeriesId: focusedSeriesId(), - hoveredGroupScope: hoveredGroupScope(), - focusedGroupScope: focusedGroupScope(), - groupScope: activeGroupScope(), - }), - ); - const activeSeriesId = createMemo(() => resolveSummaryActiveSeriesId({ - chartHoveredSeriesId: chartHoverSync()?.seriesId ?? null, hoveredSeriesId: hoveredSeriesId(), focusedSeriesId: focusedSeriesId(), groupScope: activeGroupScope(), @@ -441,26 +335,17 @@ export function useSummaryPageInteractionState(options: UseSummaryPageInteractio ); const tableFocus = useSummaryTableFocusBridge({ - activeSeriesId, clearPinnedScope: options.clearPinnedScope, focusedSeriesId, focusedGroupId, - onEscapeClear: () => { - setChartHoverSync(null); - options.onEscapeClear?.(); - }, + onEscapeClear: options.onEscapeClear, revealActiveSeries: options.revealActiveSeries, }); return { activeGroupScope, - activeScopeState, activeSeriesId, - chartHoverSync, - jumpToActiveRow: tableFocus.jumpToActiveRow, setClearSurfaceRootRef: tableFocus.setClearSurfaceRootRef, - setChartHoverSync, setTableRootRef: tableFocus.setTableRootRef, - shouldShowJumpToActiveRow: tableFocus.shouldShowJumpToActiveRow, } as const; } diff --git a/frontend-modern/src/features/patrol/__tests__/usePatrolIntelligenceState.test.ts b/frontend-modern/src/features/patrol/__tests__/usePatrolIntelligenceState.test.ts index 4dcef31829..a7e54782cb 100644 --- a/frontend-modern/src/features/patrol/__tests__/usePatrolIntelligenceState.test.ts +++ b/frontend-modern/src/features/patrol/__tests__/usePatrolIntelligenceState.test.ts @@ -3,7 +3,7 @@ import { afterEach, describe, expect, it, vi } from 'vitest'; import { PATROL_MANUAL_SYNC_TIMEOUT_MS, PATROL_REFRESH_TIMEOUT_MS, - buildPatrolSettingsReadinessFailure, + getPatrolSavedReadinessWarning, openPatrolAssistantWorkflowHandoff, patrolStartFailureMessage, recordPatrolControlStarterActivity, @@ -338,24 +338,25 @@ describe('usePatrolIntelligenceState', () => { }); }); - describe('buildPatrolSettingsReadinessFailure', () => { - it('ignores settings snapshots that do not block Patrol readiness', () => { + describe('getPatrolSavedReadinessWarning', () => { + it('stays quiet when the saved settings do not block Patrol readiness', () => { + expect(getPatrolSavedReadinessWarning(null)).toBeNull(); expect( - buildPatrolSettingsReadinessFailure({ - settings: settingsWithReadiness({ + getPatrolSavedReadinessWarning( + settingsWithReadiness({ status: 'warning', ready: true, summary: 'Patrol can run with reduced confidence.', checks: [], }), - }), + ), ).toBeNull(); }); - it('builds a saved configuration issue from a not-ready settings response', () => { + it('warns that a not-ready Patrol setting was still saved', () => { expect( - buildPatrolSettingsReadinessFailure({ - settings: settingsWithReadiness({ + getPatrolSavedReadinessWarning( + settingsWithReadiness({ status: 'not_ready', ready: false, cause: 'model_unsupported_tools', @@ -364,40 +365,8 @@ describe('usePatrolIntelligenceState', () => { model: 'ollama:deepseek-r1:7b', checks: [], }), - autonomyLevel: 'monitor', - fullModeUnlocked: false, - investigationBudget: 15, - investigationTimeoutSec: 300, - runtimeState: 'blocked', - blockedReason: 'Connect a tool-capable Patrol model.', - }), - ).toEqual({ - message: 'The selected model cannot run Patrol tools.', - code: 'patrol_readiness_not_ready', - status: 409, - saved: true, - details: { - status: 'not_ready', - cause: 'model_unsupported_tools', - summary: 'The selected model cannot run Patrol tools.', - provider: 'ollama', - model: 'ollama:deepseek-r1:7b', - }, - autonomyLevel: 'monitor', - fullModeUnlocked: false, - investigationBudget: 15, - investigationTimeoutSec: 300, - readiness: { - status: 'not_ready', - cause: 'model_unsupported_tools', - summary: 'The selected model cannot run Patrol tools.', - provider: 'ollama', - model: 'ollama:deepseek-r1:7b', - }, - runtimeState: 'blocked', - blockedReason: 'Connect a tool-capable Patrol model.', - blockedCause: undefined, - }); + ), + ).toBe('Patrol setting was saved, but Patrol is not ready to run.'); }); }); @@ -410,6 +379,8 @@ describe('usePatrolIntelligenceState', () => { 'provider_not_configured', ); expect(resolvePatrolBlockedActionCause('', '')).toBeUndefined(); - expect(patrolIntelligenceStateSource).toContain('blockedCause: patrolStatus()?.blocked_cause'); + expect(patrolIntelligenceStateSource).toContain( + 'const blockedCause = createMemo(() => patrolStatus()?.blocked_cause);', + ); }); }); diff --git a/frontend-modern/src/features/patrol/patrolInvestigationContextModel.ts b/frontend-modern/src/features/patrol/patrolInvestigationContextModel.ts index ab961032a5..ec242837ed 100644 --- a/frontend-modern/src/features/patrol/patrolInvestigationContextModel.ts +++ b/frontend-modern/src/features/patrol/patrolInvestigationContextModel.ts @@ -182,27 +182,6 @@ export interface PatrolRunAssistantHandoff { context: AIChatContext; } -export interface PatrolConfigurationFailureInput { - message: string; - code?: string; - status?: number; - saved?: boolean; - details?: Record; - autonomyLevel?: string; - fullModeUnlocked?: boolean; - investigationBudget?: number; - investigationTimeoutSec?: number; - readiness?: { - status?: string; - cause?: string; - summary?: string; - provider?: string; - model?: string; - } | null; - runtimeState?: string; - blockedReason?: string; - blockedCause?: string; -} const MAX_PATROL_RUN_HANDOFF_RESOURCES = 8; export function buildPatrolInvestigationRecordPresentation( diff --git a/frontend-modern/src/features/patrol/usePatrolIntelligenceState.ts b/frontend-modern/src/features/patrol/usePatrolIntelligenceState.ts index 9bf1c0a20f..3b6f509208 100644 --- a/frontend-modern/src/features/patrol/usePatrolIntelligenceState.ts +++ b/frontend-modern/src/features/patrol/usePatrolIntelligenceState.ts @@ -62,7 +62,6 @@ import { patrolAssistantFindingHandoffRequiresApprovalMode, type PatrolAssistantApprovalBriefingInput, type PatrolAssistantFindingHandoff, - type PatrolConfigurationFailureInput, } from './patrolInvestigationContextModel'; import { schedulePatrolRunAcceptanceReconciliation } from './patrolRunAcceptance'; @@ -141,19 +140,6 @@ export const patrolStartFailureMessage = (error: unknown): string => ? `Could not reach Pulse to start Patrol: ${patrolErrorMessage(error, 'network request failed')}` : patrolErrorMessage(error, 'Pulse rejected the Patrol run'); -const buildReadinessDetails = ( - readiness: NonNullable, -): Record => { - const details: Record = { - status: readiness.status, - }; - if (readiness.cause?.trim()) details.cause = readiness.cause.trim(); - if (readiness.summary?.trim()) details.summary = readiness.summary.trim(); - if (readiness.provider?.trim()) details.provider = readiness.provider.trim(); - if (readiness.model?.trim()) details.model = readiness.model.trim(); - return details; -}; - export function resolvePatrolAutonomyLevelForSave( level: PatrolAutonomyLevel, fullModeUnlocked: boolean, @@ -184,52 +170,16 @@ export function resolvePatrolAutonomySettingsForSave({ }; } -export function buildPatrolSettingsReadinessFailure({ - settings, - message, - autonomyLevel, - fullModeUnlocked, - investigationBudget, - investigationTimeoutSec, - runtimeState, - blockedReason, - blockedCause, -}: { - settings: AISettings | null | undefined; - message?: string; - autonomyLevel?: string; - fullModeUnlocked?: boolean; - investigationBudget?: number; - investigationTimeoutSec?: number; - runtimeState?: string; - blockedReason?: string; - blockedCause?: string; -}): PatrolConfigurationFailureInput | null { - const readiness = settings?.patrol_readiness; - if (!readiness || readiness.status !== 'not_ready') return null; - - return { - message: - message || readiness.summary || 'Patrol settings were saved, but Patrol is not ready to run.', - code: 'patrol_readiness_not_ready', - status: 409, - saved: true, - details: buildReadinessDetails(readiness), - autonomyLevel, - fullModeUnlocked, - investigationBudget, - investigationTimeoutSec, - readiness: { - status: readiness.status, - cause: readiness.cause, - summary: readiness.summary, - provider: readiness.provider, - model: readiness.model, - }, - runtimeState, - blockedReason, - blockedCause, - }; +/** + * A Patrol on/off save whose response reports Patrol not ready was still + * saved, so it raises a warning rather than an error. + */ +export function getPatrolSavedReadinessWarning( + settings: AISettings | null | undefined, +): string | null { + return settings?.patrol_readiness?.status === 'not_ready' + ? 'Patrol setting was saved, but Patrol is not ready to run.' + : null; } /** @@ -380,26 +330,6 @@ export function usePatrolIntelligenceState() { applyPatrolAISettings(aiRuntimeSettings()); }); - const surfaceSavedPatrolReadinessIssue = ( - settings: AISettings | null | undefined, - message?: string, - ) => { - const failure = buildPatrolSettingsReadinessFailure({ - settings, - message, - autonomyLevel: autonomyLevel(), - fullModeUnlocked: fullModeUnlocked(), - investigationBudget: investigationBudget(), - investigationTimeoutSec: investigationTimeout(), - runtimeState: runtimeState(), - blockedReason: blockedReason(), - blockedCause: patrolStatus()?.blocked_cause, - }); - if (failure) { - notificationStore.warning(failure.message); - } - }; - async function handleTogglePatrol() { if (isTogglingPatrol()) return; setIsTogglingPatrol(true); @@ -419,10 +349,10 @@ export function usePatrolIntelligenceState() { } else { setPatrolEnabledLocal(newValue); } - surfaceSavedPatrolReadinessIssue( - data, - 'Patrol setting was saved, but Patrol is not ready to run.', - ); + const readinessWarning = getPatrolSavedReadinessWarning(data); + if (readinessWarning) { + notificationStore.warning(readinessWarning); + } if (refetchPatrolStatus) { refetchPatrolStatus(); } diff --git a/frontend-modern/src/types/aiIntelligence.ts b/frontend-modern/src/types/aiIntelligence.ts index 48237e884d..f20f83c533 100644 --- a/frontend-modern/src/types/aiIntelligence.ts +++ b/frontend-modern/src/types/aiIntelligence.ts @@ -109,14 +109,6 @@ export interface ResourceCorrelation { description: string; } -export interface CorrelationsResponse { - correlations: ResourceCorrelation[]; - count: number; - message?: string; - license_required?: boolean; - upgrade_url?: string; -} - export interface IntelligenceSummary { timestamp: string; overall_health: IntelligenceHealthScore; diff --git a/frontend-modern/src/utils/__tests__/docsLinks.test.ts b/frontend-modern/src/utils/__tests__/docsLinks.test.ts index 0ccae13f89..88dc5447c7 100644 --- a/frontend-modern/src/utils/__tests__/docsLinks.test.ts +++ b/frontend-modern/src/utils/__tests__/docsLinks.test.ts @@ -175,6 +175,17 @@ describe('docsLinks', () => { expect(handler).toContain('agentID != "" && len(*compatibility.ObservationLocationIDs) == 1'); }); + it('documents alert configuration saves as replacing only the keys they send', () => { + const api = readFileSync(path.join(repoRoot, 'docs', 'API.md'), 'utf8'); + expect(api).toContain('`PUT /api/alerts/config` โ€” replaces only the top-level keys the body'); + expect(api).toContain('a key left out keeps its stored value'); + const handler = readFileSync( + path.join(repoRoot, 'internal', 'api', 'alerting', 'alerts.go'), + 'utf8', + ); + expect(handler).toContain('GetAlertManager().ApplyConfigUpdate(body)'); + }); + it('separates server removal from persistent-data erasure', () => { const installation = readFileSync(path.join(repoRoot, 'docs', 'INSTALL.md'), 'utf8'); const removal = installation.split('## ๐Ÿ—‘๏ธ Uninstall')[1]; @@ -566,6 +577,27 @@ describe('docsLinks', () => { expect(apiReference).not.toContain('Mobile Remote Access'); }); + it('documents POST as the only report transport that uses Pulse Assistant', () => { + const handler = readFileSync( + path.join(repoRoot, 'internal', 'api', 'metrics_reporting_handlers.go'), + 'utf8', + ); + expect(handler).toContain( + 'if r.Method == http.MethodPost {\n\t\tnarrator, _, findings = h.resolveNarrator(r.Context())', + ); + expect(handler).toContain('req.Narrator = getReportNarrator{}'); + for (const guide of ['API.md', 'MSP.md']) { + const rootDoc = readFileSync(path.join(repoRoot, 'docs', guide), 'utf8'); + expect(readFileSync(path.join(frontendRoot, 'public', 'docs', guide), 'utf8')).toBe(rootDoc); + expect(rootDoc).toContain('`POST /api/admin/reports/generate` ('); + } + const apiReference = readFileSync(path.join(repoRoot, 'docs', 'API.md'), 'utf8'); + expect(apiReference).toContain('it never calls an AI provider'); + expect(apiReference).toContain('`405` with `Allow: GET, POST`'); + const msp = readFileSync(path.join(repoRoot, 'docs', 'MSP.md'), 'utf8'); + expect(msp).not.toContain('`GET /api/admin/reports/generate` (single resource)'); + }); + it('ships credential-free Proxmox setup instructions and a separate token prompt', () => { const apiReference = readFileSync(path.join(repoRoot, 'docs', 'API.md'), 'utf8'); const shipped = readFileSync(path.join(frontendRoot, 'public', 'docs', 'API.md'), 'utf8'); diff --git a/frontend-modern/src/utils/__tests__/frontendResourceTypeBoundaries.test.ts b/frontend-modern/src/utils/__tests__/frontendResourceTypeBoundaries.test.ts index f96ca80244..a2de774cb6 100644 --- a/frontend-modern/src/utils/__tests__/frontendResourceTypeBoundaries.test.ts +++ b/frontend-modern/src/utils/__tests__/frontendResourceTypeBoundaries.test.ts @@ -3609,7 +3609,7 @@ describe('frontend resource type boundaries', () => { expect(patrolIntelligenceStateSource).toMatch( /import\s*\{[^}]*\bbuildPatrolAssistantFindingHandoffFromUnifiedFinding\b[^}]*\}\s*from '\.\/patrolInvestigationContextModel';/, ); - expect(patrolIntelligenceStateSource).toContain('buildPatrolSettingsReadinessFailure'); + expect(patrolIntelligenceStateSource).toContain('getPatrolSavedReadinessWarning'); expect(patrolIntelligenceStateSource).not.toContain('recent_changes?.length'); expect(patrolIntelligenceStateSource).not.toContain('governed resource${'); expect(patrolInvestigationContextModelSource).toContain( diff --git a/frontend-modern/src/utils/__tests__/resourceStateAdapters.test.ts b/frontend-modern/src/utils/__tests__/resourceStateAdapters.test.ts index 9673cfd24b..9d9502f7b8 100644 --- a/frontend-modern/src/utils/__tests__/resourceStateAdapters.test.ts +++ b/frontend-modern/src/utils/__tests__/resourceStateAdapters.test.ts @@ -1706,6 +1706,82 @@ describe('fast merge path for metrics-only delta patches', () => { } }); + it("lets a linked agent's filesystems and freshness replace withdrawn ones", () => { + const raw = createPveGuestRaw(); + raw.sources = ['proxmox-pve', 'agent'] as Resource['sources']; + raw.platformData = { ...raw.platformData, sources: ['proxmox-pve', 'agent'] }; + raw.proxmox = { + ...raw.proxmox, + runtimeStatus: 'running', + guestAgentStatus: 'available', + disks: [{ mountpoint: '/', total: 2048, used: 512 }], + }; + raw.agent = { + agentId: 'agent-in-vm', + stale: true, + disks: [{ mountpoint: '/', total: 4096, used: 1024 }], + } as Resource['agent']; + raw.platformData = { ...raw.platformData, agent: raw.agent }; + const previous = seedDisplayRows([raw]); + // The guest agent stopped, so Proxmox withdraws its filesystems, and the + // linked Pulse agent reports again with an explicit stale=false. + const incoming = structuredClone(raw); + incoming.proxmox = { + vmid: 100, + nodeName: 'pve-node-1', + runtimeStatus: 'running', + guestAgentStatus: 'expected-unreachable', + diskStatusReason: 'agent-not-running', + }; + incoming.agent = { ...raw.agent, stale: false } as Resource['agent']; + incoming.platformData = { ...incoming.platformData, agent: incoming.agent }; + const changed = new Set([raw.id]); + const full = mergeCanonicalResourceSnapshot([incoming], previous); + const delta = mergeCanonicalResourceDeltaSnapshot([incoming], previous, changed); + const fast = mergeCanonicalResourceDeltaSnapshot( + [incoming], + previous, + changed, + new Map([[raw.id, ['proxmox', 'agent']]]), + ); + for (const [merged] of [full, delta, fast]) { + expect(merged.proxmox?.disks).toBeUndefined(); + expect(merged.proxmox?.diskStatusReason).toBe('agent-not-running'); + expect(merged.agent?.stale).toBe(false); + expect(merged.agent?.disks).toEqual(raw.agent?.disks); + } + + // A partial facet without a native outcome keeps the prior filesystems. + const partial = { ...raw, proxmox: { vmid: 100, uptime: 1001 } } as Resource; + const [kept] = mergeCanonicalResourceSnapshot([partial], previous); + expect(kept.proxmox?.disks).toEqual(raw.proxmox?.disks); + + // A native agent report (it always carries stale) without disks has + // withdrawn them; a legacy facet without stale keeps the prior list. + const withdrawn = structuredClone(raw); + withdrawn.agent = { agentId: 'agent-in-vm', stale: false } as Resource['agent']; + withdrawn.platformData = { ...withdrawn.platformData, agent: withdrawn.agent }; + for (const [merged] of [ + mergeCanonicalResourceSnapshot([withdrawn], previous), + mergeCanonicalResourceDeltaSnapshot([withdrawn], previous, changed), + mergeCanonicalResourceDeltaSnapshot( + [withdrawn], + previous, + changed, + new Map([[raw.id, ['agent']]]), + ), + ]) { + expect(merged.agent?.disks).toBeUndefined(); + expect(merged.agent?.stale).toBe(false); + // The compatibility mirror that host threshold rows read drops them too. + const mirror = (merged.platformData as { agent?: { disks?: unknown } } | undefined)?.agent; + expect(mirror?.disks).toBeUndefined(); + } + const legacy = { ...raw, agent: { agentId: 'agent-in-vm' } } as Resource; + const [legacyKept] = mergeCanonicalResourceSnapshot([legacy], previous); + expect(legacyKept.agent?.disks).toEqual(raw.agent?.disks); + }); + it('keeps explicit lock and deferral evidence on a new outcome, including false expected', () => { const raw = createPveGuestRaw(); raw.proxmox = { diff --git a/frontend-modern/src/utils/__tests__/workloadGuestPresentation.test.ts b/frontend-modern/src/utils/__tests__/workloadGuestPresentation.test.ts index 7edf413296..8c4dff0eb3 100644 --- a/frontend-modern/src/utils/__tests__/workloadGuestPresentation.test.ts +++ b/frontend-modern/src/utils/__tests__/workloadGuestPresentation.test.ts @@ -4,6 +4,7 @@ import { getWorkloadsGuestBackupStatusPresentation, getWorkloadsGuestBackupTooltip, getWorkloadsGuestProtectionPresentation, + getWorkloadGuestDiskRead, getWorkloadGuestDiskStatusMessage, getWorkloadsGuestNetworkEmptyState, } from '@/utils/workloadGuestPresentation'; @@ -104,4 +105,63 @@ describe('workloadGuestPresentation', () => { 'Using last known disk stats. No filesystems found. VM may be booting or using a Live ISO.', ); }); + + it("judges a linked agent's filesystems by that agent, not the Proxmox read reason", () => { + const agent = { diskStatusReason: 'agent-not-running', disksFromAgent: true }; + expect(getWorkloadGuestDiskRead({ ...agent, status: 'running' }, true)).toEqual({ + state: 'current', + message: null, + needsAction: false, + }); + expect( + getWorkloadGuestDiskRead({ ...agent, status: 'running', agentStale: true }, true), + ).toEqual({ + state: 'last-known', + message: 'Using last known disk stats. The Pulse Agent in this guest stopped reporting.', + needsAction: false, + }); + // A stopped guest has no current filesystems from either source, with or + // without the Proxmox vm-stopped reason. + for (const guest of [ + { ...agent, status: 'stopped', agentStale: true }, + { ...agent, status: 'running', diskStatusReason: 'vm-stopped' }, + ]) { + expect(getWorkloadGuestDiskRead(guest, true)).toEqual({ + state: 'unavailable', + message: 'Guest filesystem stats unavailable while the VM is stopped.', + needsAction: false, + }); + } + expect(getWorkloadGuestDiskRead({ disksFromAgent: true, status: 'stopped' }, false)).toEqual({ + state: 'unavailable', + message: 'Filesystem usage is unavailable.', + needsAction: false, + }); + }); + + it('keeps Proxmox read reasons for its own guest filesystems', () => { + expect(getWorkloadGuestDiskRead({ status: 'running' }, true).state).toBe('current'); + expect(getWorkloadGuestDiskRead({ diskStatusReason: 'prev-vm-locked' }, true)).toEqual({ + state: 'last-known', + message: getWorkloadGuestDiskStatusMessage('prev-vm-locked'), + needsAction: false, + }); + expect(getWorkloadGuestDiskRead({ diskStatusReason: 'agent-not-running' }, true)).toEqual({ + state: 'unavailable', + message: getWorkloadGuestDiskStatusMessage('agent-not-running'), + needsAction: false, + }); + expect( + getWorkloadGuestDiskRead({ diskStatusReason: 'prev-agent-error' }, true).needsAction, + ).toBe(true); + expect( + getWorkloadGuestDiskRead({ diskStatusReason: 'permission-denied' }, true).needsAction, + ).toBe(true); + // Only a VM carries a Proxmox guest read reason. + expect(getWorkloadGuestDiskRead({ diskStatusReason: 'prev-vm-locked' }, false).state).toBe( + 'current', + ); + // An agent's staleness says nothing about Proxmox's own filesystems. + expect(getWorkloadGuestDiskRead({ agentStale: true }, true).state).toBe('current'); + }); }); diff --git a/frontend-modern/src/utils/resourceStateAdapters.ts b/frontend-modern/src/utils/resourceStateAdapters.ts index 72efec2fb5..fcd3bac486 100644 --- a/frontend-modern/src/utils/resourceStateAdapters.ts +++ b/frontend-modern/src/utils/resourceStateAdapters.ts @@ -124,8 +124,9 @@ const mergeRecord = (incoming?: T, existing?: T): T | unde }; // Native VM facets carry a runtime state and an explicit guest-agent outcome. -// Their optional read reason, lock and false expected flag are omitempty on -// wire. A new outcome supersedes those fields, not unrelated richer metadata. +// Their optional read reason, lock, false expected flag and guest filesystems +// are omitempty on wire. A new outcome supersedes those fields, not unrelated +// richer metadata; withdrawn filesystems must not hide a linked agent's. // Partial/legacy facets without that evidence must retain the prior facts. const mergeProxmoxFacet = (incoming?: T, existing?: T): T | undefined => { const merged = mergeRecord(incoming, existing); @@ -142,12 +143,30 @@ const mergeProxmoxFacet = (incoming?: T, existing?: T): T ) return merged; const next = { ...merged }; - for (const key of ['diskStatusReason', 'guestAgentExpected', 'lock']) { + for (const key of ['diskStatusReason', 'guestAgentExpected', 'lock', 'disks']) { if (!Object.prototype.hasOwnProperty.call(incoming, key)) delete next[key]; } return next; }; +// Native agent facets always carry `stale`; facets synthesized from legacy +// platform data never do. Their filesystem list is omitempty on wire, so a +// native report without `disks` has withdrawn them, while a partial or legacy +// facet keeps the prior list. +const mergeAgentFacet = (incoming?: T, existing?: T): T | undefined => { + const merged = mergeRecord(incoming, existing); + if ( + !incoming || + !merged || + typeof incoming.stale !== 'boolean' || + Object.prototype.hasOwnProperty.call(incoming, 'disks') + ) + return merged; + const next = { ...merged }; + delete next.disks; + return next; +}; + const mergePlatformData = ( incomingValue: Resource['platformData'], existingValue: Resource['platformData'], @@ -180,7 +199,9 @@ const mergePlatformData = ( const nested = key === 'proxmox' ? mergeProxmoxFacet(asRecord(incoming[key]), asRecord(existing[key])) - : mergeRecord(asRecord(incoming[key]), asRecord(existing[key])); + : key === 'agent' + ? mergeAgentFacet(asRecord(incoming[key]), asRecord(existing[key])) + : mergeRecord(asRecord(incoming[key]), asRecord(existing[key])); if (key === 'docker') { if (hasDockerFacetEvidence(nested)) { merged[key] = nested; @@ -895,7 +916,9 @@ const mergeCanonicalSourceFacet = ( shouldKeepSourceFacet(incomingSources, ...sourceCandidates) ? sourceCandidates.includes('proxmox-pve') ? mergeProxmoxFacet(incomingFacet, existingFacet) - : mergeRecord(incomingFacet, existingFacet) + : sourceCandidates.includes('agent') + ? mergeAgentFacet(incomingFacet, existingFacet) + : mergeRecord(incomingFacet, existingFacet) : incomingFacet; // A missing field alone can be a partial snapshot. Explicit unavailable raw diff --git a/frontend-modern/src/utils/workloadGuestPresentation.ts b/frontend-modern/src/utils/workloadGuestPresentation.ts index 36fbe7d28f..b6be31269c 100644 --- a/frontend-modern/src/utils/workloadGuestPresentation.ts +++ b/frontend-modern/src/utils/workloadGuestPresentation.ts @@ -146,3 +146,61 @@ export function getWorkloadGuestDiskStatusMessage(reason?: string): string { return carriedForward ? `Using last known disk stats. ${message}` : message; } + +export interface WorkloadGuestDiskReadSource { + status?: string; + /** Proxmox's reason for its own guest filesystem read. */ + diskStatusReason?: string; + /** `disks` came from the linked Pulse agent because Proxmox reported none. */ + disksFromAgent?: boolean; + agentStale?: boolean; +} + +export type WorkloadGuestDiskRead = + | { state: 'current'; message: null; needsAction: false } + | { + state: 'last-known' | 'unavailable'; + /** Why the reading is not current. */ + message: string; + /** A broken read the user can fix, rather than a setup gap. */ + needsAction: boolean; + }; + +// Disk-read reasons that point at something broken the user can fix. +const DISK_READ_ACTIONABLE_REASONS = new Set(['permission-denied', 'agent-error']); + +// One freshness rule for a guest's filesystem rows and aggregate disk metric, +// shared by the workload table cell, the drawer and its History. Proxmox +// records no guest disk usage without guest filesystems, so when the linked +// Pulse agent's filesystems fill in, the registry keeps the agent's disk +// metric too and the reading follows that agent, not the Proxmox reason. +export function getWorkloadGuestDiskRead( + guest: WorkloadGuestDiskReadSource, + isVM: boolean, +): WorkloadGuestDiskRead { + const reason = isVM ? guest.diskStatusReason : undefined; + if (guest.disksFromAgent) { + if (guest.status === 'stopped' || reason === 'vm-stopped') { + return { + state: 'unavailable', + message: isVM + ? getWorkloadGuestDiskStatusMessage('vm-stopped') + : 'Filesystem usage is unavailable.', + needsAction: false, + }; + } + return guest.agentStale + ? { + state: 'last-known', + message: 'Using last known disk stats. The Pulse Agent in this guest stopped reporting.', + needsAction: false, + } + : { state: 'current', message: null, needsAction: false }; + } + if (!reason) return { state: 'current', message: null, needsAction: false }; + return { + state: reason.startsWith('prev-') ? 'last-known' : 'unavailable', + message: getWorkloadGuestDiskStatusMessage(reason), + needsAction: DISK_READ_ACTIONABLE_REASONS.has(reason.replace(/^prev-/, '')), + }; +} diff --git a/internal/ai/memory/changes.go b/internal/ai/memory/changes.go index 0771e9cd71..3b0a4e4bc0 100644 --- a/internal/ai/memory/changes.go +++ b/internal/ai/memory/changes.go @@ -131,17 +131,3 @@ func (d *ChangeDetector) loadFromDisk() error { d.trimChanges() return nil } - -// Helper functions - -func intToString(n int) string { - if n == 0 { - return "0" - } - var result string - for n > 0 { - result = string(rune('0'+n%10)) + result - n /= 10 - } - return result -} diff --git a/internal/ai/memory/context.go b/internal/ai/memory/context.go index 1b5547d882..9a42d03758 100644 --- a/internal/ai/memory/context.go +++ b/internal/ai/memory/context.go @@ -711,18 +711,15 @@ func (s *ContextStore) ForceSave() error { var memoryCounter, incidentMemCounter, patternMemCounter atomic.Int64 func generateMemoryID() string { - n := memoryCounter.Add(1) - return fmt.Sprintf("mem-%s-%d", time.Now().Format("20060102150405"), n%1000) + return nextSequencedID("mem-", &memoryCounter) } func generateIncidentMemoryID() string { - n := incidentMemCounter.Add(1) - return fmt.Sprintf("inc-mem-%s-%d", time.Now().Format("20060102150405"), n%1000) + return nextSequencedID("inc-mem-", &incidentMemCounter) } func generatePatternMemoryID() string { - n := patternMemCounter.Add(1) - return fmt.Sprintf("pat-mem-%s-%d", time.Now().Format("20060102150405"), n%1000) + return nextSequencedID("pat-mem-", &patternMemCounter) } func calculatePatternConfidence(occurrences int) float64 { diff --git a/internal/ai/memory/ids.go b/internal/ai/memory/ids.go new file mode 100644 index 0000000000..2be409a1da --- /dev/null +++ b/internal/ai/memory/ids.go @@ -0,0 +1,16 @@ +package memory + +import ( + "strconv" + "sync/atomic" + "time" +) + +// nextSequencedID mints prefix + local wall-clock second + "-" + the next value +// of seq. Each sequence is package-level and shared by every store in the +// process, including one store per tenant monitor, so it must stay atomic. +// The suffix never wraps, so one sequence never repeats an ID within a process +// however many IDs share a second. +func nextSequencedID(prefix string, seq *atomic.Int64) string { + return prefix + time.Now().Format("20060102150405") + "-" + strconv.FormatInt(seq.Add(1), 10) +} diff --git a/internal/ai/memory/ids_test.go b/internal/ai/memory/ids_test.go new file mode 100644 index 0000000000..530381625e --- /dev/null +++ b/internal/ai/memory/ids_test.go @@ -0,0 +1,122 @@ +package memory + +import ( + "fmt" + "regexp" + "strings" + "sync" + "testing" + "time" + + "github.com/rcourtman/pulse-go-rewrite/internal/alerts" +) + +// TestSequencedIDsStayUniqueUnderConcurrentMinting mints from every +// package-level ID sequence on many goroutines at once. Under -race it proves +// the sequences are synchronized. The uniqueness check proves the suffix does +// not wrap: more than a thousand IDs from one generator land in the same +// wall-clock second, which the old counter%1000 suffix turned into duplicates. +func TestSequencedIDsStayUniqueUnderConcurrentMinting(t *testing.T) { + generators := []struct { + prefix string + mint func() string + }{ + {"inc-", generateIncidentID}, + {"inc-evt-", generateIncidentEventID}, + {"rem-", generateRecordID}, + {"mem-", generateMemoryID}, + {"inc-mem-", generateIncidentMemoryID}, + {"pat-mem-", generatePatternMemoryID}, + } + const goroutines, perGoroutine = 16, 150 + + for _, gen := range generators { + t.Run(strings.TrimSuffix(gen.prefix, "-"), func(t *testing.T) { + shape := regexp.MustCompile("^" + regexp.QuoteMeta(gen.prefix) + `\d{14}-[1-9]\d*$`) + minted := make([][]string, goroutines) + start := make(chan struct{}) + var wg sync.WaitGroup + for g := range goroutines { + wg.Add(1) + go func() { + defer wg.Done() + <-start + batch := make([]string, perGoroutine) + for i := range batch { + batch[i] = gen.mint() + } + minted[g] = batch + }() + } + close(start) + wg.Wait() + + seen := make(map[string]bool, goroutines*perGoroutine) + for _, batch := range minted { + for _, id := range batch { + if !shape.MatchString(id) { + t.Fatalf("ID %q does not match %s", id, shape) + } + if seen[id] { + t.Fatalf("duplicate ID %q among %d concurrent mints", id, goroutines*perGoroutine) + } + seen[id] = true + } + } + }) + } +} + +// TestIncidentStoresMintIDsConcurrentlyAcrossTenants mirrors multi-tenant +// monitoring: each tenant monitor owns its own IncidentStore, and their alert +// checks call RecordAlertFired from separate goroutines. Each store's mutex +// covers only its own state, so the shared ID sequence must be synchronized on +// its own. Run under -race. +func TestIncidentStoresMintIDsConcurrentlyAcrossTenants(t *testing.T) { + const tenants, checkersPerTenant, alertsPerChecker = 2, 4, 50 + stores := make([]*IncidentStore, tenants) + for i := range stores { + stores[i] = NewIncidentStore(IncidentStoreConfig{MaxIncidents: checkersPerTenant * alertsPerChecker}) + } + + startedAt := time.Now().Add(-time.Minute) + start := make(chan struct{}) + var wg sync.WaitGroup + for tenant, store := range stores { + for checker := range checkersPerTenant { + wg.Add(1) + go func() { + defer wg.Done() + <-start + for n := range alertsPerChecker { + store.RecordAlertFired(&alerts.Alert{ + ID: fmt.Sprintf("tenant-%d-checker-%d-alert-%d", tenant, checker, n), + Type: "cpu", + Level: alerts.AlertLevelWarning, + ResourceID: fmt.Sprintf("tenant-%d-vm-%d-%d", tenant, checker, n), + ResourceName: "vm", + StartTime: startedAt, + }) + } + }() + } + } + close(start) + wg.Wait() + + seen := make(map[string]bool, tenants*checkersPerTenant*alertsPerChecker) + for tenant, store := range stores { + store.mu.RLock() + incidents := len(store.incidents) + for _, shell := range store.incidents { + if seen[shell.ID] { + t.Errorf("tenant %d incident ID %q was already minted", tenant, shell.ID) + } + seen[shell.ID] = true + } + store.mu.RUnlock() + if incidents != checkersPerTenant*alertsPerChecker { + t.Errorf("tenant %d recorded %d incidents, want %d", tenant, incidents, checkersPerTenant*alertsPerChecker) + } + } +} diff --git a/internal/ai/memory/incidents.go b/internal/ai/memory/incidents.go index 8729c83cef..5c763b2d92 100644 --- a/internal/ai/memory/incidents.go +++ b/internal/ai/memory/incidents.go @@ -1747,18 +1747,14 @@ func cloneTime(value time.Time) *time.Time { return &cloned } -var incidentCounter int64 +var incidentCounter, incidentEventCounter atomic.Int64 func generateIncidentID() string { - incidentCounter++ - return "inc-" + time.Now().Format("20060102150405") + "-" + intToString(int(incidentCounter%1000)) + return nextSequencedID("inc-", &incidentCounter) } -var incidentEventCounter int64 - func generateIncidentEventID() string { - incidentEventCounter++ - return "inc-evt-" + time.Now().Format("20060102150405") + "-" + intToString(int(incidentEventCounter%1000)) + return nextSequencedID("inc-evt-", &incidentEventCounter) } func formatAlertSummary(alert *alerts.Alert) string { diff --git a/internal/ai/memory/memory_coverage_test.go b/internal/ai/memory/memory_coverage_test.go index 653f05796d..7ae826ea6a 100644 --- a/internal/ai/memory/memory_coverage_test.go +++ b/internal/ai/memory/memory_coverage_test.go @@ -11,18 +11,11 @@ import ( "github.com/rcourtman/pulse-go-rewrite/internal/alerts" ) -func TestChangeDetector_DefaultsAndHelpers(t *testing.T) { +func TestChangeDetector_Defaults(t *testing.T) { detector := NewChangeDetector(ChangeDetectorConfig{}) if detector.maxChanges != 1000 { t.Fatalf("expected default maxChanges=1000, got %d", detector.maxChanges) } - - if got := intToString(0); got != "0" { - t.Errorf("intToString(0) = %q", got) - } - if got := intToString(42); got != "42" { - t.Errorf("intToString(42) = %q", got) - } } func TestNewChangeDetector_LoadsFromDisk(t *testing.T) { diff --git a/internal/ai/memory/remediation.go b/internal/ai/memory/remediation.go index da93f5f4f5..97d2c8ac81 100644 --- a/internal/ai/memory/remediation.go +++ b/internal/ai/memory/remediation.go @@ -5,6 +5,7 @@ import ( "fmt" "os" "sync" + "sync/atomic" "time" "github.com/rcourtman/pulse-go-rewrite/internal/utils" @@ -349,11 +350,10 @@ func (r *RemediationLog) loadFromDisk() error { // Helper functions -var recordCounter int64 +var recordCounter atomic.Int64 func generateRecordID() string { - recordCounter++ - return "rem-" + time.Now().Format("20060102150405") + "-" + intToString(int(recordCounter%1000)) + return nextSequencedID("rem-", &recordCounter) } func truncateOutput(output string, maxLen int) string { diff --git a/internal/alerts/config/update.go b/internal/alerts/config/update.go new file mode 100644 index 0000000000..0cc9fd334b --- /dev/null +++ b/internal/alerts/config/update.go @@ -0,0 +1,90 @@ +package config + +import ( + "encoding/json" + "errors" + "fmt" + "reflect" + "strings" +) + +var errAlertConfigUpdateNotObject = errors.New("alert configuration update must be a JSON object") + +// ApplyAlertConfigUpdate decodes a client's alert configuration update onto +// the stored configuration. A top-level key the client sent replaces that +// setting whole; a key it left out keeps the stored value. +// +// Clients only send what they know about. The settings page has no control +// for flapping detection, alert TTL cleanup or custom rules, and an older +// browser bundle or API script predates any field added since. Decoding the +// body into a zero AlertConfig used to turn every one of those settings off +// on each save. +// +// The body is decoded on its own, so the keys it carries mean exactly what +// encoding/json makes of them. Each setting it left out is copied from a JSON +// round trip of stored, so the result shares no maps or slices with it. +func ApplyAlertConfigUpdate(stored AlertConfig, update []byte) (AlertConfig, error) { + var sent map[string]json.RawMessage + if err := json.Unmarshal(update, &sent); err != nil { + return AlertConfig{}, err + } + if sent == nil { + return AlertConfig{}, errAlertConfigUpdateNotObject + } + var result AlertConfig + if err := json.Unmarshal(update, &result); err != nil { + return AlertConfig{}, err + } + + kept, err := CloneAlertConfig(stored) + if err != nil { + return AlertConfig{}, err + } + + resultValue := reflect.ValueOf(&result).Elem() + keptValue := reflect.ValueOf(kept) + for i := 0; i < resultValue.NumField(); i++ { + if !alertConfigKeySent(sent, alertConfigJSONKey(resultValue.Type().Field(i))) { + resultValue.Field(i).Set(keptValue.Field(i)) + } + } + return result, nil +} + +// CloneAlertConfig returns a copy of config that shares no maps, slices or +// pointers with it, by the same JSON round trip config takes through +// alerts.json. +func CloneAlertConfig(config AlertConfig) (AlertConfig, error) { + data, err := json.Marshal(config) + if err != nil { + return AlertConfig{}, fmt.Errorf("encode alert config: %w", err) + } + var clone AlertConfig + if err := json.Unmarshal(data, &clone); err != nil { + return AlertConfig{}, fmt.Errorf("decode alert config: %w", err) + } + return clone, nil +} + +// alertConfigJSONKey returns the key encoding/json reads a field from. +func alertConfigJSONKey(field reflect.StructField) string { + name, _, _ := strings.Cut(field.Tag.Get("json"), ",") + if name == "" { + return field.Name + } + return name +} + +// alertConfigKeySent matches keys the way encoding/json matches them to +// fields: exactly, or else case-insensitively. +func alertConfigKeySent(sent map[string]json.RawMessage, key string) bool { + if _, ok := sent[key]; ok { + return true + } + for sentKey := range sent { + if strings.EqualFold(sentKey, key) { + return true + } + } + return false +} diff --git a/internal/alerts/config/update_test.go b/internal/alerts/config/update_test.go new file mode 100644 index 0000000000..9d1347f870 --- /dev/null +++ b/internal/alerts/config/update_test.go @@ -0,0 +1,257 @@ +package config + +import ( + "encoding/json" + "reflect" + "strings" + "testing" + "time" +) + +func storedAlertConfigForUpdate() AlertConfig { + activated := time.Date(2026, 10, 1, 9, 30, 0, 0, time.UTC) + note := "keep" + return AlertConfig{ + IdentitySchemaVersion: 2, + Enabled: true, + ActivationState: ActivationActive, + ActivationTime: &activated, + ObservationWindowHours: 24, + GuestDefaults: ThresholdConfig{CPU: &HysteresisThreshold{Trigger: 80, Clear: 75}}, + StorageDefault: HysteresisThreshold{Trigger: 85, Clear: 80}, + DiskTempByType: map[string]HysteresisThreshold{"nvme": {Trigger: 70, Clear: 65}}, + IgnoredGuestPrefixes: []string{"test-"}, + Overrides: map[string]ThresholdConfig{"node:pve1": {Disabled: true, Note: ¬e}}, + CustomRules: []CustomAlertRule{{ID: "rule-1", Name: "Busy web", Enabled: true}}, + Schedule: ScheduleConfig{Cooldown: 5, MaxAlertsHour: 10, InitialNotify: "all"}, + MinimumDelta: 3, + SuppressionWindow: 10, + HysteresisMargin: 4, + TimeThresholds: map[string]int{"guest": 5}, + MaxAlertAgeDays: 7, + MaxAcknowledgedAgeDays: 1, + AutoAcknowledgeAfterHours: 24, + FlappingEnabled: true, + FlappingWindowSeconds: 600, + FlappingThreshold: 4, + FlappingCooldownMinutes: 20, + } +} + +func TestApplyAlertConfigUpdateKeepsSettingsTheClientDidNotSend(t *testing.T) { + stored := storedAlertConfigForUpdate() + + // The settings page sends thresholds and schedule but has no control for + // flapping detection, alert TTL cleanup or custom rules. + got, err := ApplyAlertConfigUpdate(stored, []byte(`{ + "enabled": true, + "guestDefaults": {"cpu": {"trigger": 90, "clear": 85}}, + "overrides": {}, + "flapping": {"enabled": true, "threshold": 5} + }`)) + if err != nil { + t.Fatalf("ApplyAlertConfigUpdate: %v", err) + } + + if got.GuestDefaults.CPU == nil || got.GuestDefaults.CPU.Trigger != 90 { + t.Fatalf("guest CPU default = %+v, want the sent trigger 90", got.GuestDefaults.CPU) + } + if len(got.Overrides) != 0 { + t.Fatalf("overrides = %v, want the sent empty map to replace the stored overrides", got.Overrides) + } + if !got.FlappingEnabled || got.FlappingWindowSeconds != 600 || got.FlappingThreshold != 4 || got.FlappingCooldownMinutes != 20 { + t.Fatalf("flapping = %v/%d/%d/%d, want the stored true/600/4/20", + got.FlappingEnabled, got.FlappingWindowSeconds, got.FlappingThreshold, got.FlappingCooldownMinutes) + } + if got.MaxAlertAgeDays != 7 || got.MaxAcknowledgedAgeDays != 1 || got.AutoAcknowledgeAfterHours != 24 { + t.Fatalf("TTL cleanup = %d/%d/%d, want the stored 7/1/24", + got.MaxAlertAgeDays, got.MaxAcknowledgedAgeDays, got.AutoAcknowledgeAfterHours) + } + if !reflect.DeepEqual(got.CustomRules, stored.CustomRules) { + t.Fatalf("custom rules = %+v, want the stored rules", got.CustomRules) + } + if got.MinimumDelta != 3 || got.SuppressionWindow != 10 || got.HysteresisMargin != 4 { + t.Fatalf("general settings = %v/%d/%v, want the stored 3/10/4", + got.MinimumDelta, got.SuppressionWindow, got.HysteresisMargin) + } +} + +func TestApplyAlertConfigUpdateEmptyObjectChangesNothing(t *testing.T) { + stored := storedAlertConfigForUpdate() + + got, err := ApplyAlertConfigUpdate(stored, []byte(`{}`)) + if err != nil { + t.Fatalf("ApplyAlertConfigUpdate({}): %v", err) + } + if !reflect.DeepEqual(got, stored) { + t.Fatalf("ApplyAlertConfigUpdate({}) changed the config:\n got %+v\nwant %+v", got, stored) + } +} + +func TestApplyAlertConfigUpdateHonorsExplicitOffAndClear(t *testing.T) { + stored := storedAlertConfigForUpdate() + + got, err := ApplyAlertConfigUpdate(stored, []byte(`{ + "flappingEnabled": false, + "maxAlertAgeDays": 0, + "maxAcknowledgedAgeDays": 0, + "autoAcknowledgeAfterHours": 0, + "customRules": null, + "ignoredGuestPrefixes": [] + }`)) + if err != nil { + t.Fatalf("ApplyAlertConfigUpdate: %v", err) + } + + if got.FlappingEnabled || got.MaxAlertAgeDays != 0 || got.MaxAcknowledgedAgeDays != 0 || got.AutoAcknowledgeAfterHours != 0 { + t.Fatalf("explicit off was not applied: flapping=%v ttl=%d/%d/%d", + got.FlappingEnabled, got.MaxAlertAgeDays, got.MaxAcknowledgedAgeDays, got.AutoAcknowledgeAfterHours) + } + if got.CustomRules != nil { + t.Fatalf("custom rules = %+v, want null to clear them", got.CustomRules) + } + if len(got.IgnoredGuestPrefixes) != 0 { + t.Fatalf("ignored guest prefixes = %v, want the sent empty list", got.IgnoredGuestPrefixes) + } + if !got.Enabled || got.ActivationState != ActivationActive { + t.Fatalf("unsent enabled/activation = %v/%q, want the stored true/active", got.Enabled, got.ActivationState) + } +} + +// The merge is top-level only: a sent object replaces the stored one whole, +// so a sent truenasDiskDefaults without a temperature must not inherit the +// stored temperature. +func TestApplyAlertConfigUpdateReplacesSentObjectsWhole(t *testing.T) { + stored := storedAlertConfigForUpdate() + stored.TrueNASDiskDefaults = ThresholdConfig{Temperature: &HysteresisThreshold{Trigger: 62, Clear: 57}} + + got, err := ApplyAlertConfigUpdate(stored, []byte(`{"truenasDiskDefaults": {}}`)) + if err != nil { + t.Fatalf("ApplyAlertConfigUpdate: %v", err) + } + if got.TrueNASDiskDefaults.Temperature != nil { + t.Fatalf("TrueNAS disk temperature = %+v, want the sent empty object to clear it", got.TrueNASDiskDefaults.Temperature) + } +} + +// encoding/json matches keys to fields case-insensitively, so a key in another +// casing still names the field and must not leave the stored value in place. +func TestApplyAlertConfigUpdateMatchesKeysLikeTheDecoder(t *testing.T) { + stored := storedAlertConfigForUpdate() + + got, err := ApplyAlertConfigUpdate(stored, []byte(`{"FlappingEnabled": false}`)) + if err != nil { + t.Fatalf("ApplyAlertConfigUpdate: %v", err) + } + if got.FlappingEnabled { + t.Fatal("FlappingEnabled sent as false stayed true") + } +} + +// Sent keys mean what decoding the body alone makes of them, duplicates and +// all. +func TestApplyAlertConfigUpdateDecodesSentKeysLikeTheBodyAlone(t *testing.T) { + stored := storedAlertConfigForUpdate() + + for _, test := range []struct { + body string + field func(AlertConfig) any + }{ + {`{"enabled": false, "enabled": null}`, func(c AlertConfig) any { return c.Enabled }}, + {`{"flappingEnabled": false, "FlappingEnabled": true}`, func(c AlertConfig) any { return c.FlappingEnabled }}, + { + `{"guestDefaults": {"cpu": {"trigger": 90, "clear": 85}}, "guestDefaults": {"memory": {"trigger": 70, "clear": 65}}}`, + func(c AlertConfig) any { return c.GuestDefaults }, + }, + } { + var alone AlertConfig + if err := json.Unmarshal([]byte(test.body), &alone); err != nil { + t.Fatalf("decode %s alone: %v", test.body, err) + } + got, err := ApplyAlertConfigUpdate(stored, []byte(test.body)) + if err != nil { + t.Fatalf("ApplyAlertConfigUpdate(%s): %v", test.body, err) + } + if !reflect.DeepEqual(test.field(got), test.field(alone)) { + t.Fatalf("ApplyAlertConfigUpdate(%s) = %+v, decoding the body alone gives %+v", + test.body, test.field(got), test.field(alone)) + } + } + + if _, err := ApplyAlertConfigUpdate(stored, []byte(`{"enabled": "yes", "enabled": true}`)); err == nil { + t.Fatal("a type error in a duplicated key succeeded, want the decoder's error") + } +} + +// The merge copies unsent settings field by field under their JSON keys, so +// an embedded, unexported or untagged field would be skipped or matched +// under the wrong key. +func TestAlertConfigFieldsHaveExplicitJSONKeys(t *testing.T) { + fields := reflect.TypeOf(AlertConfig{}) + keys := make([]string, fields.NumField()) + for i := 0; i < fields.NumField(); i++ { + field := fields.Field(i) + keys[i], _, _ = strings.Cut(field.Tag.Get("json"), ",") + if field.Anonymous || !field.IsExported() || keys[i] == "" || keys[i] == "-" { + t.Errorf("AlertConfig.%s needs to be an exported, named field with an explicit json key", field.Name) + } + // Keys that fold equal would both count as sent for either key. + for j := 0; j < i; j++ { + if strings.EqualFold(keys[i], keys[j]) { + t.Errorf("AlertConfig.%s and AlertConfig.%s have json keys that fold equal", + fields.Field(j).Name, field.Name) + } + } + } +} + +func TestApplyAlertConfigUpdateSharesNoStateWithStored(t *testing.T) { + stored := storedAlertConfigForUpdate() + + got, err := ApplyAlertConfigUpdate(stored, []byte(`{"enabled": true}`)) + if err != nil { + t.Fatalf("ApplyAlertConfigUpdate: %v", err) + } + got.TimeThresholds["guest"] = 99 + got.DiskTempByType["nvme"] = HysteresisThreshold{Trigger: 1, Clear: 0} + got.IgnoredGuestPrefixes[0] = "changed-" + *got.ActivationTime = time.Time{} + + if stored.TimeThresholds["guest"] != 5 || stored.DiskTempByType["nvme"].Trigger != 70 || + stored.IgnoredGuestPrefixes[0] != "test-" || stored.ActivationTime.IsZero() { + t.Fatal("editing the merged config changed the stored config it was merged from") + } +} + +func TestApplyAlertConfigUpdateRejectsMalformedBodies(t *testing.T) { + stored := storedAlertConfigForUpdate() + + for _, body := range []string{`null`, `[]`, `"config"`, `{"flappingEnabled": "yes"}`, `{`} { + if _, err := ApplyAlertConfigUpdate(stored, []byte(body)); err == nil { + t.Fatalf("ApplyAlertConfigUpdate(%s) succeeded, want a decode error", body) + } + } +} + +// A config marshaled by a Go client still replaces every field it writes. +// Empty omitempty fields are left out of that body, so they keep their +// stored values like any other unsent key. +func TestApplyAlertConfigUpdateFullBodyReplacesWrittenFields(t *testing.T) { + stored := storedAlertConfigForUpdate() + replacement := AlertConfig{Enabled: true, Overrides: map[string]ThresholdConfig{}, TimeThresholds: map[string]int{}} + + body, err := json.Marshal(replacement) + if err != nil { + t.Fatalf("marshal replacement: %v", err) + } + got, err := ApplyAlertConfigUpdate(stored, body) + if err != nil { + t.Fatalf("ApplyAlertConfigUpdate: %v", err) + } + if got.FlappingEnabled || got.MaxAlertAgeDays != 0 || got.MinimumDelta != 0 || len(got.Overrides) != 0 { + t.Fatalf("full body left written fields at their stored values: %+v", got) + } + if !reflect.DeepEqual(got.CustomRules, stored.CustomRules) { + t.Fatalf("custom rules = %+v, want the stored rules for the omitted key", got.CustomRules) + } +} diff --git a/internal/alerts/config_runtime.go b/internal/alerts/config_runtime.go index d609874c77..722515faee 100644 --- a/internal/alerts/config_runtime.go +++ b/internal/alerts/config_runtime.go @@ -1,6 +1,8 @@ package alerts import ( + "errors" + "fmt" "strings" "time" @@ -13,7 +15,36 @@ import ( func (m *Manager) UpdateConfig(config AlertConfig) { m.mu.Lock() defer m.mu.Unlock() + m.updateConfigLocked(config) +} +// ErrConfigSnapshot reports an update that was applied but could not be copied +// for the caller. +var ErrConfigSnapshot = errors.New("snapshot applied alert configuration") + +// ApplyConfigUpdate applies a client's JSON update to the stored +// configuration: keys the update carries replace their settings and the rest +// keep their stored values (see alertconfig.ApplyAlertConfigUpdate). Reading, +// merging and applying under one lock keeps two partial updates from +// reverting each other's settings. It returns a snapshot of the applied +// config that the caller owns, so persisting it cannot write into the live +// config's maps. +func (m *Manager) ApplyConfigUpdate(update []byte) (AlertConfig, error) { + m.mu.Lock() + defer m.mu.Unlock() + config, err := alertconfig.ApplyAlertConfigUpdate(m.config, update) + if err != nil { + return AlertConfig{}, err + } + m.updateConfigLocked(config) + snapshot, err := alertconfig.CloneAlertConfig(m.config) + if err != nil { + return AlertConfig{}, fmt.Errorf("%w: %w", ErrConfigSnapshot, err) + } + return snapshot, nil +} + +func (m *Manager) updateConfigLocked(config AlertConfig) { // Clients and rollback-era config writers may not know about the additive // identity schema marker. Never lower a version already held by this // process; an older binary can still omit it on disk and the migration will diff --git a/internal/alerts/config_validation_test.go b/internal/alerts/config_validation_test.go index 82c579abe3..3b4f7e30b3 100644 --- a/internal/alerts/config_validation_test.go +++ b/internal/alerts/config_validation_test.go @@ -205,3 +205,79 @@ func TestUpdateConfigNormalizesExactEscalationRoutingAndRepeatPolicy(t *testing. } } } + +// The thresholds page sends no flapping, alert TTL or custom-rule keys. A save +// through ApplyConfigUpdate must leave those settings as stored, also after +// the merged config is normalized in place. +func TestUpdateConfigKeepsSettingsAThresholdsSaveDidNotSend(t *testing.T) { + m := newTestManager(t) + stored := m.GetConfig() + stored.FlappingEnabled = true + stored.FlappingWindowSeconds = 600 + stored.MaxAlertAgeDays = 7 + stored.MaxAcknowledgedAgeDays = 1 + stored.AutoAcknowledgeAfterHours = 24 + stored.CustomRules = []CustomAlertRule{{ID: "rule-1", Name: "Busy web", Enabled: true}} + m.UpdateConfig(stored) + + snapshot, err := m.ApplyConfigUpdate([]byte(`{ + "enabled": true, + "nodeDefaults": {"cpu": {"trigger": 82, "clear": 77}}, + "overrides": {} + }`)) + if err != nil { + t.Fatalf("ApplyConfigUpdate: %v", err) + } + snapshot.TimeThresholds["guest"] = 999 + if m.GetConfig().TimeThresholds["guest"] == 999 { + t.Fatal("the returned snapshot shares its maps with the live config") + } + + got := m.GetConfig() + if got.NodeDefaults.CPU == nil || got.NodeDefaults.CPU.Trigger != 82 { + t.Fatalf("node CPU default = %+v, want the saved trigger 82", got.NodeDefaults.CPU) + } + if !got.FlappingEnabled || got.FlappingWindowSeconds != 600 { + t.Fatalf("flapping = %v/%d after a thresholds save, want true/600", got.FlappingEnabled, got.FlappingWindowSeconds) + } + if got.MaxAlertAgeDays != 7 || got.MaxAcknowledgedAgeDays != 1 || got.AutoAcknowledgeAfterHours != 24 { + t.Fatalf("TTL cleanup = %d/%d/%d after a thresholds save, want 7/1/24", + got.MaxAlertAgeDays, got.MaxAcknowledgedAgeDays, got.AutoAcknowledgeAfterHours) + } + if len(got.CustomRules) != 1 || got.CustomRules[0].ID != "rule-1" { + t.Fatalf("custom rules = %+v after a thresholds save, want the stored rule", got.CustomRules) + } +} + +// Each partial update reads, merges and applies under the manager lock, so +// two saves that touch different settings cannot revert each other. +func TestApplyConfigUpdateKeepsConcurrentPartialUpdates(t *testing.T) { + m := newTestManager(t) + + for round := 0; round < 50; round++ { + base := m.GetConfig() + base.FlappingEnabled = false + base.MaxAlertAgeDays = 0 + m.UpdateConfig(base) + + bodies := []string{`{"flappingEnabled": true}`, `{"maxAlertAgeDays": 9}`} + done := make(chan struct{}, len(bodies)) + for _, body := range bodies { + go func(body string) { + defer func() { done <- struct{}{} }() + if _, err := m.ApplyConfigUpdate([]byte(body)); err != nil { + t.Errorf("ApplyConfigUpdate(%s): %v", body, err) + } + }(body) + } + for range bodies { + <-done + } + + got := m.GetConfig() + if !got.FlappingEnabled || got.MaxAlertAgeDays != 9 { + t.Fatalf("round %d: flapping=%v maxAlertAgeDays=%d, want both concurrent updates kept", + round, got.FlappingEnabled, got.MaxAlertAgeDays) + } + } +} diff --git a/internal/alerts/host.go b/internal/alerts/host.go index db47b5bdf6..be3d9aa092 100644 --- a/internal/alerts/host.go +++ b/internal/alerts/host.go @@ -10,6 +10,7 @@ import ( "github.com/rcourtman/pulse-go-rewrite/internal/models" "github.com/rcourtman/pulse-go-rewrite/internal/storagehealth" "github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources" + "github.com/rcourtman/pulse-go-rewrite/pkg/fsfilters" "github.com/rs/zerolog/log" ) @@ -419,43 +420,79 @@ func (m *Manager) CheckHost(host models.Host) { } if thresholds.DiskTemperature != nil && thresholds.DiskTemperature.Trigger > 0 { + // Each disk is judged on the temperature it shows: its SMART row's + // reading, or the one the host's Unraid inventory reports for it when + // the agent's probe returned none, and the inventory's reading for a + // disk only the inventory lists. + readings := unifiedresources.HostDiskTemperatureReadings(host) + seenDiskTemps := make(map[string]struct{}, len(readings)) + excludedDiskTemps := make(map[string]struct{}) + judgedDiskTemps := make(map[string]judgedHostDiskTemperature, len(readings)) + judgedOrder := make([]string, 0, len(readings)) + for _, disk := range readings { + tempResourceID := hostDiskTemperatureResourceID(host.ID, disk.Device) + // A disk the agent's --disk-exclude patterns match is out of + // monitoring, so the Unraid inventory's reading raises nothing + // for it either. + if disk.UnraidOnly && fsfilters.MatchesDeviceExclude(disk.Device, host.DiskExclude) { + excludedDiskTemps[tempResourceID] = struct{}{} + continue + } + // A listed disk in standby, or without a current temperature after + // a failed probe, is still present. Its alert holds until a fresh + // reading resolves it instead of clearing and re-raising. + seenDiskTemps[tempResourceID] = struct{}{} + if !disk.Collected() { + continue + } + m.mu.RLock() + threshold := m.hostDiskTemperatureThresholdNoLock(thresholds.DiskTemperature, diskTempOverridden, disk.DiskType) + m.mu.RUnlock() + // Sources that disagree on a disk's identity can show two disks + // on one device. Its alert judges the reading that stands highest + // against its own disk type's threshold. + candidate := judgedHostDiskTemperature{reading: disk, threshold: threshold} + if judged, ok := judgedDiskTemps[tempResourceID]; !ok { + judgedOrder = append(judgedOrder, tempResourceID) + } else if !candidate.outranks(judged) { + continue + } + judgedDiskTemps[tempResourceID] = candidate + } + for _, tempResourceID := range judgedOrder { + disk := judgedDiskTemps[tempResourceID].reading + effectiveTempThreshold := judgedDiskTemps[tempResourceID].threshold + + tempResourceName := fmt.Sprintf("%s (%s Temp)", hostDisplayName(host), disk.Device) + + diskTempMetadata := hostChildAlertMetadata(baseMetadata, hostDiskAlertResourceType) + diskTempMetadata["metric"] = "diskTemperature" + diskTempMetadata["device"] = disk.Device + diskTempMetadata["temperature"] = disk.Temperature + diskTempMetadata["model"] = disk.Model + diskTempMetadata["diskType"] = disk.DiskType + spec, err := buildCanonicalMetricSpec(tempResourceID, tempResourceName, unifiedresources.ResourceType("agent-disk"), "diskTemperature", effectiveTempThreshold) + if err != nil { + log.Warn(). + Err(err). + Str("resourceID", tempResourceID). + Str("host", resourceName). + Str("device", disk.Device). + Msg("Skipping invalid canonical host disk temperature metric spec") + continue + } + + m.checkMetricWithCanonicalSpec(spec, tempResourceName, nodeName, disk.Device, "agent", float64(disk.Temperature), effectiveTempThreshold, &metricOptions{Metadata: diskTempMetadata}) + } + // An excluded disk's alert from before the exclusion resolves now, + // whatever the SMART list holds, unless another listed disk shares it. + for tempResourceID := range seenDiskTemps { + delete(excludedDiskTemps, tempResourceID) + } + m.clearHostDiskTemperatureResources(excludedDiskTemps) // An empty SMART list means collection failed or is unsupported for // this report, not that every disk left, so existing alerts are held. if len(host.Sensors.SMART) > 0 { - seenDiskTemps := make(map[string]struct{}, len(host.Sensors.SMART)) - for _, disk := range host.Sensors.SMART { - // A listed disk in standby, or without a temperature after a - // failed probe, is still present. Its alert holds until a fresh - // reading resolves it instead of clearing and re-raising. - tempResourceID := hostDiskTemperatureResourceID(host.ID, disk.Device) - seenDiskTemps[tempResourceID] = struct{}{} - if disk.Temperature > 0 && !disk.Standby { - m.mu.RLock() - effectiveTempThreshold := m.hostDiskTemperatureThresholdNoLock(thresholds.DiskTemperature, diskTempOverridden, disk.Type) - m.mu.RUnlock() - - tempResourceName := fmt.Sprintf("%s (%s Temp)", hostDisplayName(host), disk.Device) - - diskTempMetadata := hostChildAlertMetadata(baseMetadata, hostDiskAlertResourceType) - diskTempMetadata["metric"] = "diskTemperature" - diskTempMetadata["device"] = disk.Device - diskTempMetadata["temperature"] = disk.Temperature - diskTempMetadata["model"] = disk.Model - diskTempMetadata["diskType"] = disk.Type - spec, err := buildCanonicalMetricSpec(tempResourceID, tempResourceName, unifiedresources.ResourceType("agent-disk"), "diskTemperature", effectiveTempThreshold) - if err != nil { - log.Warn(). - Err(err). - Str("resourceID", tempResourceID). - Str("host", resourceName). - Str("device", disk.Device). - Msg("Skipping invalid canonical host disk temperature metric spec") - continue - } - - m.checkMetricWithCanonicalSpec(spec, tempResourceName, nodeName, disk.Device, "agent", float64(disk.Temperature), effectiveTempThreshold, &metricOptions{Metadata: diskTempMetadata}) - } - } // A disk missing from consecutive non-empty reports was removed, // replaced or renamed, so no later reading will resolve its alert. m.cleanupHostDiskTemperatureAlerts(host.ID, seenDiskTemps) diff --git a/internal/alerts/host_disk_temperature_readings.go b/internal/alerts/host_disk_temperature_readings.go new file mode 100644 index 0000000000..f1e997d8fc --- /dev/null +++ b/internal/alerts/host_disk_temperature_readings.go @@ -0,0 +1,72 @@ +package alerts + +import "github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources" + +// clearHostDiskTemperatureResources resolves the disk temperature alerts +// raised under the given resource IDs and drops their pending threshold runs. +// CheckHost calls it for disks its host still lists but the agent's +// --disk-exclude patterns took out of monitoring, so no reading will resolve +// them. +func (m *Manager) clearHostDiskTemperatureResources(resourceIDs map[string]struct{}) { + if len(resourceIDs) == 0 { + return + } + + m.mu.Lock() + defer m.mu.Unlock() + + var storageKeys []string + for storageKey, alert := range m.activeAlerts { + if alert == nil { + continue + } + if _, ok := resourceIDs[alert.ResourceID]; ok { + storageKeys = append(storageKeys, storageKey) + } + } + for resourceID := range resourceIDs { + m.core.DropPendingForResource(resourceID) + } + for _, storageKey := range storageKeys { + m.clearAlertNoLock(storageKey) + } +} + +// judgedHostDiskTemperature is a disk temperature reading with the threshold +// its disk type resolves to under the host's policy. +type judgedHostDiskTemperature struct { + reading unifiedresources.HostDiskTemperatureReading + threshold *HysteresisThreshold +} + +// standing ranks the reading against its own threshold: 2 at or above the +// trigger, 1 inside the recovery band, where an open alert holds, 0 below it, +// and -1 when the threshold is off. margin is how far the reading sits above +// the trigger. +func (j judgedHostDiskTemperature) standing() (rank int, margin float64) { + threshold := j.threshold + if threshold == nil || threshold.Trigger <= 0 { + return -1, 0 + } + temperature := float64(j.reading.Temperature) + margin = temperature - threshold.Trigger + switch { + case temperature >= threshold.Trigger: + return 2, margin + case threshold.Clear > 0 && threshold.Clear < threshold.Trigger && temperature > threshold.Clear: + return 1, margin + default: + return 0, margin + } +} + +// outranks reports whether j should be judged instead of other for their +// shared alert. A tie keeps the reading listed first. +func (j judgedHostDiskTemperature) outranks(other judgedHostDiskTemperature) bool { + rank, margin := j.standing() + otherRank, otherMargin := other.standing() + if rank != otherRank { + return rank > otherRank + } + return margin > otherMargin +} diff --git a/internal/alerts/host_unraid_lifecycle_test.go b/internal/alerts/host_unraid_lifecycle_test.go index 1dd52d59b6..a3c217fb20 100644 --- a/internal/alerts/host_unraid_lifecycle_test.go +++ b/internal/alerts/host_unraid_lifecycle_test.go @@ -7,6 +7,7 @@ import ( "time" "github.com/rcourtman/pulse-go-rewrite/internal/models" + "github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources" ) func TestHostCustomSensorAlertLifecycle(t *testing.T) { @@ -546,6 +547,303 @@ func TestConfigSaveKeepsAlertTheEvaluatorKeepsWithoutRecoveryBand(t *testing.T) } } +const unraidTempHostID = "host-unraid-temp" + +// unraidTempHost is an Unraid agent report with the given SMART rows and +// Unraid inventory rows. +func unraidTempHost(smart []models.HostDiskSMART, unraid ...models.HostUnraidDisk) models.Host { + host := hostWithSMARTDiskTemp(unraidTempHostID, "sata", 40) + host.Sensors.SMART = smart + host.Unraid = &models.HostUnraidStorage{ArrayStarted: true, ArrayState: "STARTED", Disks: unraid} + return host +} + +func unraidTempAlertID(device string) string { + return canonicalMetricStateID(hostDiskTemperatureResourceID(unraidTempHostID, device), "diskTemperature") +} + +func TestCheckHostJudgesTheUnraidTemperatureADiskShows(t *testing.T) { + cool := models.HostDiskSMART{Device: "sda", Model: "cool-disk", Serial: "COOL-1", Type: "sata", Temperature: 40} + coolUnraid := models.HostUnraidDisk{Name: "disk1", Device: "/dev/sda", Serial: "COOL-1", Transport: "sata", Temperature: 40} + hot := models.HostUnraidDisk{Name: "disk2", Device: "/dev/sdc", Model: "hot-disk", Serial: "HOT-1", Transport: "sata", Temperature: 60} + + cases := []struct { + name string + host func() models.Host + configure func(m *Manager) + device string + wantAlert bool + // quiet lists devices that must raise nothing. + quiet []string + }{ + { + // smartctl returned no temperature, and the disk shows Unraid's. + name: "SMART row without a reading", + host: func() models.Host { + return unraidTempHost([]models.HostDiskSMART{ + {Device: "sdc", Model: "hot-disk", Serial: "HOT-1", Type: "sata"}, + }, hot) + }, + device: "sdc", + wantAlert: true, + }, + { + name: "disk only the Unraid inventory lists", + host: func() models.Host { + return unraidTempHost([]models.HostDiskSMART{cool}, coolUnraid, hot) + }, + device: "sdc", + wantAlert: true, + }, + { + name: "Unraid disk while the SMART list is empty", + host: func() models.Host { + return unraidTempHost(nil, coolUnraid, hot) + }, + device: "sdc", + wantAlert: true, + }, + { + // smartctl and Unraid name the device with different serials, so + // two disks show on it; its alert judges the hotter reading. + name: "two disks shown on one device", + host: func() models.Host { + return unraidTempHost([]models.HostDiskSMART{ + cool, + {Device: "sdc", Model: "bridge", Serial: "BRIDGE-9", Type: "sata", Temperature: 41}, + }, coolUnraid, hot) + }, + device: "sdc", + wantAlert: true, + }, + { + // 64C is under the sas trigger (65); 60C is over the sata one (55). + name: "two disks of different types on one device", + host: func() models.Host { + return unraidTempHost([]models.HostDiskSMART{ + cool, + {Device: "sdc", Model: "bridge", Serial: "BRIDGE-9", Type: "sas", Temperature: 64}, + }, coolUnraid, hot) + }, + device: "sdc", + wantAlert: true, + }, + { + // A controller member and its Unraid device are one registry + // disk, alerted once under the member's own resource. + name: "controller member under another device label", + host: func() models.Host { + member := models.HostDiskSMART{Device: "0 [megaraid,0]", Model: "hot-disk", Serial: "HOT-1", Type: "sata", Controller: "0", Target: "megaraid,0", Temperature: 60} + return unraidTempHost([]models.HostDiskSMART{cool, member}, coolUnraid, hot) + }, + device: "0 [megaraid,0]", + wantAlert: true, + quiet: []string{"sdc"}, + }, + { + name: "spun-down Unraid disk", + host: func() models.Host { + spunDown := hot + spunDown.SpunDown = true + return unraidTempHost([]models.HostDiskSMART{cool}, coolUnraid, spunDown) + }, + device: "sdc", + }, + { + name: "Unraid disk the agent excludes", + host: func() models.Host { + host := unraidTempHost([]models.HostDiskSMART{cool}, coolUnraid, hot) + host.DiskExclude = []string{"/dev/sdc"} + return host + }, + device: "sdc", + }, + { + // 65C is under the nvme trigger (70) and over the agent default (55). + name: "Unraid disk judged by its disk type", + host: func() models.Host { + nvme := hot + nvme.Transport = "nvme" + nvme.Temperature = 65 + return unraidTempHost([]models.HostDiskSMART{cool}, coolUnraid, nvme) + }, + device: "sdc", + }, + { + name: "Unraid disk under a host override", + host: func() models.Host { + return unraidTempHost([]models.HostDiskSMART{cool}, coolUnraid, hot) + }, + configure: func(m *Manager) { + m.config.Overrides[unraidTempHostID] = ThresholdConfig{DiskTemperature: &HysteresisThreshold{Trigger: 65, Clear: 60}} + }, + device: "sdc", + }, + } + + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + m := configureDiskTempTypeHostManager(t) + if tc.configure != nil { + m.mu.Lock() + tc.configure(m) + m.mu.Unlock() + } + m.CheckHost(tc.host()) + + alert, exists := testLookupActiveAlert(t, m, unraidTempAlertID(tc.device)) + if exists != tc.wantAlert { + t.Fatalf("disk temperature alert for %s active = %v, want %v; active: %v", tc.device, exists, tc.wantAlert, alertKeys(m)) + } + for _, device := range append([]string{"sda"}, tc.quiet...) { + if _, exists := testLookupActiveAlert(t, m, unraidTempAlertID(device)); exists { + t.Fatalf("%s raised a temperature alert, active: %v", device, alertKeys(m)) + } + } + if !tc.wantAlert { + return + } + if alert.Value != 60 || alert.Metadata["temperature"] != 60 || alert.Metadata["diskType"] != "sata" || alert.Metadata["model"] != "hot-disk" { + t.Fatalf("alert value/metadata = %v/%v, want the 60C sata reading of hot-disk", alert.Value, alert.Metadata) + } + }) + } +} + +func TestCheckHostHoldsUnraidDiskTemperatureAlertUntilAReadingOrDeparture(t *testing.T) { + cool := models.HostDiskSMART{Device: "sda", Model: "cool-disk", Serial: "COOL-1", Type: "sata", Temperature: 40} + hot := models.HostUnraidDisk{Name: "disk2", Device: "/dev/sdc", Model: "hot-disk", Serial: "HOT-1", Transport: "sata", Temperature: 60} + alertID := unraidTempAlertID("sdc") + // CheckHost clears a disk's alert once three non-empty reports in a row + // omit it. + const absentReports = 3 + active := func(m *Manager) bool { + _, exists := testLookupActiveAlert(t, m, alertID) + return exists + } + hotHost := func(t *testing.T) *Manager { + m := configureDiskTempTypeHostManager(t) + m.CheckHost(unraidTempHost([]models.HostDiskSMART{cool}, hot)) + if !active(m) { + t.Fatalf("expected the Unraid reading to raise a disk temperature alert, active: %v", alertKeys(m)) + } + return m + } + + t.Run("listed without a reading", func(t *testing.T) { + m := hotHost(t) + spunDown := hot + spunDown.SpunDown = true + for i := 0; i < 4; i++ { + m.CheckHost(unraidTempHost([]models.HostDiskSMART{cool}, spunDown)) + } + if !active(m) { + t.Fatalf("a spun-down disk still in the inventory lost its alert, active: %v", alertKeys(m)) + } + }) + + t.Run("cooled reading resolves it", func(t *testing.T) { + m := hotHost(t) + cooled := hot + cooled.Temperature = 45 + m.CheckHost(unraidTempHost([]models.HostDiskSMART{cool}, cooled)) + if active(m) { + t.Fatalf("a 45C Unraid reading (clear 50) left the alert active: %v", alertKeys(m)) + } + }) + + t.Run("smartctl reading takes over the same alert", func(t *testing.T) { + m := hotHost(t) + withSMART := []models.HostDiskSMART{cool, {Device: "sdc", Model: "hot-disk", Serial: "HOT-1", Type: "sata", Temperature: 62}} + m.CheckHost(unraidTempHost(withSMART, hot)) + if !active(m) { + t.Fatalf("the disk's SMART row did not carry its Unraid-raised alert, active: %v", alertKeys(m)) + } + withSMART[1].Temperature = 45 + m.CheckHost(unraidTempHost(withSMART, hot)) + if active(m) { + t.Fatalf("a 45C SMART reading left the alert active: %v", alertKeys(m)) + } + }) + + t.Run("left the inventory", func(t *testing.T) { + m := hotHost(t) + for i := 1; i < absentReports; i++ { + m.CheckHost(unraidTempHost([]models.HostDiskSMART{cool})) + if !active(m) { + t.Fatalf("alert cleared after %d report(s) without the disk, want %d", i, absentReports) + } + } + m.CheckHost(unraidTempHost([]models.HostDiskSMART{cool})) + if active(m) { + t.Fatalf("alert outlived a disk gone from %d reports, active: %v", absentReports, alertKeys(m)) + } + }) + + t.Run("excluded row on a device a SMART row lists", func(t *testing.T) { + m := hotHost(t) + excluded := unraidTempHost([]models.HostDiskSMART{ + cool, + {Device: "sdc", Model: "bridge", Serial: "BRIDGE-9", Type: "sata", Temperature: 62}, + }, hot) + excluded.DiskExclude = []string{"sdc"} + m.CheckHost(excluded) + if !active(m) { + t.Fatalf("the exclusion cleared an alert the device's SMART row still raises, active: %v", alertKeys(m)) + } + }) + + // The disk is still listed but out of monitoring, so its alert resolves + // on the first report, even one with an empty SMART list. + for name, smart := range map[string][]models.HostDiskSMART{ + "excluded after it fired": {cool}, + "excluded after it fired, SMART list empty": nil, + } { + t.Run(name, func(t *testing.T) { + m := hotHost(t) + excluded := unraidTempHost(smart, hot) + excluded.DiskExclude = []string{"sdc"} + m.CheckHost(excluded) + if active(m) { + t.Fatalf("alert outlived the disk's exclusion from monitoring, active: %v", alertKeys(m)) + } + }) + } +} + +func TestJudgedHostDiskTemperatureOutranks(t *testing.T) { + sata := &HysteresisThreshold{Trigger: 55, Clear: 50} + sas := &HysteresisThreshold{Trigger: 65, Clear: 60} + noBand := &HysteresisThreshold{Trigger: 55, Clear: 55} + off := &HysteresisThreshold{Trigger: 0} + judged := func(temperature int, threshold *HysteresisThreshold) judgedHostDiskTemperature { + return judgedHostDiskTemperature{ + reading: unifiedresources.HostDiskTemperatureReading{Temperature: temperature}, + threshold: threshold, + } + } + + cases := []struct { + name string + first, second judgedHostDiskTemperature + secondWins bool + }{ + {name: "over its trigger beats a hotter reading under its own", first: judged(64, sas), second: judged(56, sata), secondWins: true}, + {name: "in its recovery band beats one below its clear", first: judged(49, sata), second: judged(62, sas), secondWins: true}, + {name: "no band means under the trigger is below it", first: judged(54, noBand), second: judged(51, sata), secondWins: true}, + {name: "further over its trigger wins", first: judged(56, sata), second: judged(70, sas), secondWins: true}, + {name: "a threshold that is off never wins", first: judged(40, sata), second: judged(90, off)}, + {name: "a tie keeps the first", first: judged(60, sata), second: judged(60, sata)}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + if got := tc.second.outranks(tc.first); got != tc.secondWins { + t.Fatalf("second outranks first = %v, want %v", got, tc.secondWins) + } + }) + } +} + func hasAlertType(alerts []Alert, alertType string) bool { for _, alert := range alerts { if alert.Type == alertType { diff --git a/internal/api/ai_handlers.go b/internal/api/ai_handlers.go index 62ee996eff..0986b24b6d 100644 --- a/internal/api/ai_handlers.go +++ b/internal/api/ai_handlers.go @@ -385,6 +385,25 @@ func (h *AISettingsHandler) loadAIConfig(ctx context.Context) (*config.AIConfig, } // GetAIService returns the underlying AI service +// ExistingAIService returns the org's AI service only when one already exists +// and never constructs one. GetAIService builds and configures a tenant service +// on first use, which can list provider models and start background discovery, +// so read paths that must not reach an AI provider use this instead. +func (h *AISettingsHandler) ExistingAIService(ctx context.Context) *ai.Service { + if h == nil { + return nil + } + mtPersistence, mtMonitor, _, _, _, _ := h.stateRefs() + defaultAIService := h.providerSnapshot().defaultAIService + orgID := GetOrgID(ctx) + if orgID == "default" || orgID == "" || (mtPersistence == nil && mtMonitor == nil) { + return defaultAIService + } + h.aiServicesMu.RLock() + defer h.aiServicesMu.RUnlock() + return h.aiServices[orgID] +} + func (h *AISettingsHandler) GetAIService(ctx context.Context) *ai.Service { mtPersistence, mtMonitor, _, _, _, _ := h.stateRefs() providers := h.providerSnapshot() diff --git a/internal/api/ai_handlers_more_test.go b/internal/api/ai_handlers_more_test.go index 3e81a3e649..53a64dbe18 100644 --- a/internal/api/ai_handlers_more_test.go +++ b/internal/api/ai_handlers_more_test.go @@ -306,6 +306,36 @@ func TestAISettingsHandler_GetAIService_MultiTenantProviders(t *testing.T) { } } +// Report GETs read Patrol findings through ExistingAIService, so it must never +// construct a tenant service: construction can list provider models and start +// background discovery. +func TestAISettingsHandler_ExistingAIServiceNeverConstructs(t *testing.T) { + handler := NewAISettingsHandler(config.NewMultiTenantPersistence(t.TempDir()), nil, nil) + t.Cleanup(handler.StopServices) + tenant := context.WithValue(context.Background(), OrgIDContextKey, "tenant-1") + + if svc := handler.ExistingAIService(tenant); svc != nil { + t.Fatalf("expected no service before the tenant's first GetAIService, got %p", svc) + } + handler.aiServicesMu.RLock() + built := len(handler.aiServices) + handler.aiServicesMu.RUnlock() + if built != 0 { + t.Fatalf("ExistingAIService constructed %d tenant services, want 0", built) + } + + created := handler.GetAIService(tenant) + if created == nil { + t.Fatal("expected GetAIService to construct the tenant service") + } + if got := handler.ExistingAIService(tenant); got != created { + t.Fatalf("expected the running tenant service %p, got %p", created, got) + } + if got, want := handler.ExistingAIService(context.Background()), handler.GetAIService(context.Background()); got != want { + t.Fatalf("expected the default service %p, got %p", want, got) + } +} + func TestAISettingsHandler_GetAIService_UsesTenantReadState(t *testing.T) { tmp := t.TempDir() mtp := config.NewMultiTenantPersistence(tmp) diff --git a/internal/api/alerting/alerts.go b/internal/api/alerting/alerts.go index 2c19c30f20..8cc4165417 100644 --- a/internal/api/alerting/alerts.go +++ b/internal/api/alerting/alerts.go @@ -32,6 +32,7 @@ import ( type AlertManager interface { GetConfig() alerts.AlertConfig UpdateConfig(alerts.AlertConfig) + ApplyConfigUpdate(update []byte) (alerts.AlertConfig, error) GetActiveAlerts() []alerts.Alert DiagnoseAlertDelivery(alertIdentifier string) (alerts.AlertDeliveryDiagnosis, bool) DiagnoseActiveAlertDeliveries() []alerts.AlertDeliveryDiagnosis @@ -80,6 +81,7 @@ type AlertMonitor interface { type AlertHandlers struct { stateMu sync.RWMutex intentPolicyMu sync.Mutex + configSaveMu sync.Mutex mtMonitor *monitoring.MultiTenantMonitor defaultMonitor AlertMonitor wsHub *websocket.Hub @@ -292,13 +294,28 @@ func (h *AlertHandlers) UpdateAlertConfig(w http.ResponseWriter, r *http.Request // from instances with a few hundred disabled containers (#1601). r.Body = http.MaxBytesReader(w, r.Body, 1<<20) - var config alerts.AlertConfig - if err := json.NewDecoder(r.Body).Decode(&config); err != nil { + body, err := io.ReadAll(r.Body) + if err != nil { + http.Error(w, err.Error(), http.StatusBadRequest) + return + } + // One save at a time, so a slower save cannot write an older config to + // disk after a newer one was applied. + h.configSaveMu.Lock() + defer h.configSaveMu.Unlock() + + // Keys the client left out keep their stored values, so a thresholds + // save no longer turns off flapping detection or alert TTL cleanup. + updatedConfig, err := h.getMonitor(r.Context()).GetAlertManager().ApplyConfigUpdate(body) + if errors.Is(err, alerts.ErrConfigSnapshot) { + log.Error().Err(err).Msg("Failed to snapshot applied alert configuration") + http.Error(w, err.Error(), http.StatusInternalServerError) + return + } + if err != nil { http.Error(w, err.Error(), http.StatusBadRequest) return } - h.getMonitor(r.Context()).GetAlertManager().UpdateConfig(config) - updatedConfig := h.getMonitor(r.Context()).GetAlertManager().GetConfig() // Update notification manager with schedule settings notificationMgr := h.getMonitor(r.Context()).GetNotificationManager() diff --git a/internal/api/alerting/alerts_test.go b/internal/api/alerting/alerts_test.go index 1c3df1f2f9..ff40a931ee 100644 --- a/internal/api/alerting/alerts_test.go +++ b/internal/api/alerting/alerts_test.go @@ -74,6 +74,11 @@ func (m *MockAlertManager) UpdateConfig(cfg alerts.AlertConfig) { m.Called(cfg) } +func (m *MockAlertManager) ApplyConfigUpdate(update []byte) (alerts.AlertConfig, error) { + args := m.Called(update) + return args.Get(0).(alerts.AlertConfig), args.Error(1) +} + func (m *MockAlertManager) GetActiveAlerts() []alerts.Alert { args := m.Called() return args.Get(0).([]alerts.Alert) @@ -238,7 +243,7 @@ func TestUpdateAlertConfig(t *testing.T) { h := NewAlertHandlers(nil, mockMonitor, nil) cfg := alerts.AlertConfig{Enabled: true, ActivationState: alerts.ActivationPending} - mockManager.On("UpdateConfig", testifymock.Anything).Return() + mockManager.On("ApplyConfigUpdate", testifymock.Anything).Return(cfg, nil) mockManager.On("GetConfig").Return(cfg) mockPersist.On("SaveAlertConfig", testifymock.Anything).Return(nil) @@ -252,6 +257,88 @@ func TestUpdateAlertConfig(t *testing.T) { assert.False(t, notificationMgr.IsEnabled()) } +// The Thresholds page has no control for flapping detection or alert TTL +// cleanup and never sends those keys; decoding its body into a zero config +// turned both off on every save. +func TestUpdateAlertConfig_KeepsStoredValuesForUnsentKeys(t *testing.T) { + manager := alerts.NewManagerWithDataDir(t.TempDir()) + defer manager.Stop() + stored := manager.GetConfig() + stored.MaxAlertAgeDays = 7 + stored.MaxAcknowledgedAgeDays = 1 + stored.AutoAcknowledgeAfterHours = 24 + stored.FlappingEnabled = true + manager.UpdateConfig(stored) + + mockMonitor := new(MockAlertMonitor) + mockPersist := new(MockConfigPersistence) + notificationMgr := notifications.NewNotificationManagerWithDataDir("", t.TempDir()) + defer notificationMgr.Stop() + mockMonitor.On("GetAlertManager").Return(manager) + mockMonitor.On("GetConfigPersistence").Return(mockPersist) + mockMonitor.On("GetNotificationManager").Return(notificationMgr) + var persisted alerts.AlertConfig + mockPersist.On("SaveAlertConfig", testifymock.Anything).Run(func(args testifymock.Arguments) { + persisted = args.Get(0).(alerts.AlertConfig) + }).Return(nil) + + h := NewAlertHandlers(nil, mockMonitor, nil) + + body := `{"enabled":true,"guestDefaults":{"cpu":{"trigger":90,"clear":85}},"overrides":{},` + + `"flapping":{"enabled":true,"threshold":5,"window":10,"suppressionTime":30,"minStability":0.8}}` + req := httptest.NewRequest(http.MethodPut, "/api/alerts/config", strings.NewReader(body)) + w := httptest.NewRecorder() + + h.UpdateAlertConfig(w, req) + + if w.Code != http.StatusOK { + t.Fatalf("status = %d, want 200: %s", w.Code, w.Body.String()) + } + // Persistence normalizes the config it is handed in place, so it must get + // a snapshot rather than the live config's maps. + persisted.TimeThresholds["guest"] = 999 + if manager.GetConfig().TimeThresholds["guest"] == 999 { + t.Fatal("the persisted config shares its maps with the live alert config") + } + persisted.TimeThresholds["guest"] = manager.GetConfig().TimeThresholds["guest"] + for name, cfg := range map[string]alerts.AlertConfig{"live": manager.GetConfig(), "persisted": persisted} { + if cfg.GuestDefaults.CPU == nil || cfg.GuestDefaults.CPU.Trigger != 90 { + t.Fatalf("%s guest CPU default = %+v, want the sent trigger 90", name, cfg.GuestDefaults.CPU) + } + if !cfg.FlappingEnabled { + t.Fatalf("thresholds save turned %s flapping detection off", name) + } + if cfg.MaxAlertAgeDays != 7 || cfg.MaxAcknowledgedAgeDays != 1 || cfg.AutoAcknowledgeAfterHours != 24 { + t.Fatalf("%s TTL cleanup = %d/%d/%d after a thresholds save, want 7/1/24", + name, cfg.MaxAlertAgeDays, cfg.MaxAcknowledgedAgeDays, cfg.AutoAcknowledgeAfterHours) + } + } +} + +func TestUpdateAlertConfig_RejectsNonObjectBody(t *testing.T) { + manager := alerts.NewManagerWithDataDir(t.TempDir()) + defer manager.Stop() + before := manager.GetConfig() + + mockMonitor := new(MockAlertMonitor) + mockMonitor.On("GetAlertManager").Return(manager) + h := NewAlertHandlers(nil, mockMonitor, nil) + + for _, body := range []string{`null`, `[]`, `"config"`} { + req := httptest.NewRequest(http.MethodPut, "/api/alerts/config", strings.NewReader(body)) + w := httptest.NewRecorder() + + h.UpdateAlertConfig(w, req) + + if w.Code != http.StatusBadRequest { + t.Fatalf("status = %d for body %s, want 400", w.Code, body) + } + } + if after := manager.GetConfig(); after.Enabled != before.Enabled || after.ActivationState != before.ActivationState { + t.Fatalf("a rejected body changed the config: %+v", after) + } +} + // Regression for #1341: persistence failures used to be silently swallowed // (logged but reported as "saved successfully"), so a user setting a custom // override would see the value stick in memory and revert on reload with @@ -271,7 +358,7 @@ func TestUpdateAlertConfig_PersistenceFailureSurfacesAsError(t *testing.T) { h := NewAlertHandlers(nil, mockMonitor, nil) cfg := alerts.AlertConfig{Enabled: true, ActivationState: alerts.ActivationPending} - mockManager.On("UpdateConfig", testifymock.Anything).Return() + mockManager.On("ApplyConfigUpdate", testifymock.Anything).Return(cfg, nil) mockManager.On("GetConfig").Return(cfg) mockPersist.On("SaveAlertConfig", testifymock.Anything).Return(errors.New("permission denied")) diff --git a/internal/api/contract_test.go b/internal/api/contract_test.go index 953ed0fba2..87b9781699 100644 --- a/internal/api/contract_test.go +++ b/internal/api/contract_test.go @@ -5081,6 +5081,7 @@ func TestContract_ReportingCatalogJSONSnapshot(t *testing.T) { "title":"Performance Reports", "description":"Generate PDF summaries or CSV metric exports from historical monitoring data for one or more selected resources.", "singleResourceEndpoint":"/api/admin/reports/generate", + "singleResourceMethod":"POST", "multiResourceEndpoint":"/api/admin/reports/generate-multi", "singleFilenamePrefix":"report", "singleFilenameSubject":"resource_id", @@ -5227,6 +5228,169 @@ func TestContract_PerformanceReportTransportUsesCatalogDefinition(t *testing.T) if engine.lastReq.MetricType != "cpu" || engine.lastReq.Title != "Node report" { t.Fatalf("expected trimmed canonical optional fields, got %+v", engine.lastReq) } + + post := httptest.NewRequest( + http.MethodPost, + "/api/reporting", + strings.NewReader(`{"resourceType":"node","resourceId":"node-1","metricType":" cpu ","title":" Node report "}`), + ) + rec = httptest.NewRecorder() + handler.HandleGenerateReport(rec, post) + if rec.Code != http.StatusOK { + t.Fatalf("expected POST 200, got %d body=%s", rec.Code, rec.Body.String()) + } + if engine.lastReq.ResourceID != "node-1" || engine.lastReq.Format != definition.DefaultFormat || + engine.lastReq.MetricType != "cpu" || engine.lastReq.Title != "Node report" { + t.Fatalf("expected POST body to carry the same canonical fields as GET, got %+v", engine.lastReq) + } +} + +type recordingReportEngine struct { + *reporting.ReportEngine + requests []reporting.MetricReportRequest +} + +func (e *recordingReportEngine) Generate(req reporting.MetricReportRequest) ([]byte, string, error) { + e.requests = append(e.requests, req) + return e.ReportEngine.Generate(req) +} + +type countingReportNarrator struct{ calls atomic.Int32 } + +func (n *countingReportNarrator) Narrate(context.Context, reporting.NarrativeInput) (reporting.Narrative, error) { + n.calls.Add(1) + return reporting.Narrative{Source: reporting.NarrativeSourceAI, HealthStatus: "HEALTHY", HealthMessage: "narrated"}, nil +} + +// A PDF single-resource report may be narrated by the tenant's Pulse +// Assistant, a paid provider call recorded in the cost ledger. GET passes the +// demo-mode guard and the CSRF check, and SameSite=Lax session cookies ride +// cross-site top-level GET navigations, so only POST, the settings UI +// transport, may reach the AI narrator. GET still returns the report, with +// the deterministic summary and a note in place of the "Configure Pulse +// Assistant" tip. +func TestContract_SingleReportAINarrationRequiresPOST(t *testing.T) { + store, err := metrics.NewStore(metrics.DefaultConfig(t.TempDir())) + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { _ = store.Close() }) + engine := &recordingReportEngine{ReportEngine: reporting.NewReportEngine(reporting.EngineConfig{MetricsStore: store})} + original := reporting.GetEngine() + reporting.SetEngine(engine) + t.Cleanup(func() { reporting.SetEngine(original) }) + + // The narrator resolver stands for GetAIService, which can construct a + // tenant AI service; GET must use only the existing-service findings path. + narrator := &countingReportNarrator{} + var narratorResolves, findingsResolves atomic.Int32 + handler := NewReportingHandlers(nil, nil) + handler.SetNarratorResolver(func(context.Context) (reporting.Narrator, reporting.FleetNarrator, reporting.FindingsProvider) { + narratorResolves.Add(1) + return narrator, nil, nil + }) + handler.SetExistingFindingsResolver(func(context.Context) reporting.FindingsProvider { + findingsResolves.Add(1) + return nil + }) + generate := func(req *http.Request) *httptest.ResponseRecorder { + rec := httptest.NewRecorder() + handler.HandleGenerateReport(rec, req) + return rec + } + const query = "/api/admin/reports/generate?format=pdf&resourceType=node&resourceId=node-1" + + rec := generate(httptest.NewRequest(http.MethodGet, query, nil)) + if rec.Code != http.StatusOK || rec.Header().Get("Content-Type") != "application/pdf" { + t.Fatalf("GET report: status %d, content type %q, body=%s", rec.Code, rec.Header().Get("Content-Type"), rec.Body.String()) + } + if got := narrator.calls.Load(); got != 0 { + t.Fatalf("GET report called the AI narrator %d times, want 0", got) + } + narrative, err := engine.NarrativeFor(engine.requests[len(engine.requests)-1]) + if err != nil { + t.Fatalf("GET report narrative: %v", err) + } + if narrative.Source != reporting.NarrativeSourceHeuristic || narrative.Disclaimer != reportGETNarrativeNote { + t.Fatalf("GET report narrative = source %q disclaimer %q, want the deterministic summary with the GET note", narrative.Source, narrative.Disclaimer) + } + + for _, method := range []string{http.MethodHead, http.MethodPut, http.MethodDelete} { + rec := generate(httptest.NewRequest(method, query, nil)) + if rec.Code != http.StatusMethodNotAllowed || rec.Header().Get("Allow") != "GET, POST" { + t.Fatalf("%s report: status %d, Allow %q, want 405 with Allow GET, POST", method, rec.Code, rec.Header().Get("Allow")) + } + } + if got := narrator.calls.Load(); got != 0 { + t.Fatalf("GET, HEAD, PUT and DELETE called the AI narrator %d times, want 0", got) + } + if got := narratorResolves.Load(); got != 0 { + t.Fatalf("GET, HEAD, PUT and DELETE resolved the AI service %d times, want 0", got) + } + if got := findingsResolves.Load(); got != 1 { + t.Fatalf("GET looked up existing Patrol findings %d times, want 1", got) + } + + rec = generate(httptest.NewRequest( + http.MethodPost, + "/api/admin/reports/generate", + strings.NewReader(`{"format":"pdf","resourceType":"node","resourceId":"node-1"}`), + )) + if rec.Code != http.StatusOK || rec.Header().Get("Content-Type") != "application/pdf" { + t.Fatalf("POST report: status %d, content type %q, body=%s", rec.Code, rec.Header().Get("Content-Type"), rec.Body.String()) + } + if got := narrator.calls.Load(); got != 1 { + t.Fatalf("POST report called the AI narrator %d times, want 1", got) + } + if got := narratorResolves.Load(); got != 1 { + t.Fatalf("POST report resolved the AI service %d times, want 1", got) + } +} + +// The router's real resolvers: a GET report for a tenant whose AI service is +// not running must not construct one, because construction can list provider +// models and start background discovery. POST resolves it as before. +func TestContract_SingleReportGETDoesNotConstructTenantAIService(t *testing.T) { + engine := &stubReportingEngine{data: []byte("report"), contentType: "application/pdf"} + original := reporting.GetEngine() + reporting.SetEngine(engine) + t.Cleanup(func() { reporting.SetEngine(original) }) + + router := &Router{ + aiSettingsHandler: NewAISettingsHandler(config.NewMultiTenantPersistence(t.TempDir()), nil, nil), + reportingHandlers: NewReportingHandlers(nil, nil), + } + router.wireReportingAIResolvers() + t.Cleanup(router.aiSettingsHandler.StopServices) + tenantServices := func() int { + router.aiSettingsHandler.aiServicesMu.RLock() + defer router.aiSettingsHandler.aiServicesMu.RUnlock() + return len(router.aiSettingsHandler.aiServices) + } + generate := func(req *http.Request) *httptest.ResponseRecorder { + rec := httptest.NewRecorder() + router.reportingHandlers.HandleGenerateReport(rec, req.WithContext(context.WithValue(req.Context(), OrgIDContextKey, "tenant-1"))) + return rec + } + + rec := generate(httptest.NewRequest(http.MethodGet, "/api/admin/reports/generate?format=pdf&resourceType=node&resourceId=node-1", nil)) + if rec.Code != http.StatusOK { + t.Fatalf("GET report: status %d, body=%s", rec.Code, rec.Body.String()) + } + if got := tenantServices(); got != 0 { + t.Fatalf("GET report constructed %d tenant AI services, want 0", got) + } + if _, ok := engine.lastReq.Narrator.(getReportNarrator); !ok || engine.lastReq.FindingsProvider != nil { + t.Fatalf("GET report narrator %T, findings %v; want the deterministic narrator and no findings without a running service", engine.lastReq.Narrator, engine.lastReq.FindingsProvider) + } + + rec = generate(httptest.NewRequest(http.MethodPost, "/api/admin/reports/generate", strings.NewReader(`{"format":"pdf","resourceType":"node","resourceId":"node-1"}`))) + if rec.Code != http.StatusOK { + t.Fatalf("POST report: status %d, body=%s", rec.Code, rec.Body.String()) + } + if got := tenantServices(); got != 1 { + t.Fatalf("POST report resolved %d tenant AI services, want 1", got) + } } // Normalization keeps a disk's last-known temperature when the current @@ -9212,6 +9376,11 @@ func TestContract_DemoModeCommercialSurfacePolicy(t *testing.T) { {method: http.MethodGet, path: "/api/discover/"}, {method: http.MethodHead, path: "/api/discover"}, {method: http.MethodGet, path: licensePurchaseStartPath}, + {method: http.MethodGet, path: "/debug/pprof"}, + {method: http.MethodGet, path: "/debug/pprof/"}, + {method: http.MethodGet, path: "/debug/pprof/heap"}, + {method: http.MethodPost, path: "/debug/pprof/symbol"}, + {method: http.MethodOptions, path: "/debug/pprof/trace"}, } for _, tc := range testCases { @@ -9375,6 +9544,95 @@ func TestContract_DemoModeUpgradeHeadersDoNotExemptWrites(t *testing.T) { } } +// The pprof routes are gated only on admin auth and session logins skip token +// scopes, so the public demo's shared login reaches them like any admin. The +// demo guard must answer every pprof path with 404 for that session, whatever +// the method, while the handlers below it stay registered. +func TestContract_DemoModeHidesPprofFromDemoAdminSession(t *testing.T) { + t.Setenv("PULSE_PPROF_DISABLED", "") + + dataDir := t.TempDir() + hashedPass, err := authpkg.HashPassword("demo") + if err != nil { + t.Fatalf("hash password: %v", err) + } + cfg := &config.Config{ + DataPath: dataDir, + ConfigPath: dataDir, + DemoMode: true, + AuthUser: "demo", + AuthPass: hashedPass, + } + + router := NewRouter(cfg, nil, nil, nil, nil, "1.0.0") + cleanupTestRouter(t, router) + server := newIPv4HTTPServer(t, router.Handler()) + t.Cleanup(server.Close) + belowDemoGuard := newIPv4HTTPServer(t, router.mux) + t.Cleanup(belowDemoGuard.Close) + + loginResp, err := http.Post(server.URL+"/api/login", "application/json", strings.NewReader(`{"username":"demo","password":"demo"}`)) + if err != nil { + t.Fatalf("demo login: %v", err) + } + loginResp.Body.Close() + if loginResp.StatusCode != http.StatusOK { + t.Fatalf("demo login status = %d, want %d", loginResp.StatusCode, http.StatusOK) + } + var cookiePairs []string + for _, cookie := range loginResp.Cookies() { + cookiePairs = append(cookiePairs, cookie.Name+"="+cookie.Value) + } + if len(cookiePairs) == 0 { + t.Fatal("expected session cookies after demo login") + } + sessionCookies := strings.Join(cookiePairs, "; ") + + client := &http.Client{CheckRedirect: func(*http.Request, []*http.Request) error { + return http.ErrUseLastResponse + }} + send := func(baseURL, method, path string) (int, string) { + t.Helper() + req, err := http.NewRequest(method, baseURL+path, nil) + if err != nil { + t.Fatalf("build %s %s: %v", method, path, err) + } + req.Header.Set("Cookie", sessionCookies) + resp, err := client.Do(req) + if err != nil { + t.Fatalf("%s %s: %v", method, path, err) + } + defer resp.Body.Close() + var payload bytes.Buffer + _, _ = payload.ReadFrom(resp.Body) + return resp.StatusCode, payload.String() + } + + status, body := send(belowDemoGuard.URL, http.MethodGet, "/debug/pprof/cmdline") + if status != http.StatusOK || body == "" { + t.Fatalf("demo session below the demo guard: pprof cmdline status = %d, want %d with a body", status, http.StatusOK) + } + + paths := []string{ + "/debug/pprof", + "/debug/pprof/", + "/debug/pprof/heap?gc=1", + "/debug/pprof/goroutine?debug=2", + "/debug/pprof/cmdline", + "/debug/pprof/profile?seconds=1", + "/debug/pprof/symbol", + "/debug/pprof/trace?seconds=1", + } + methods := []string{http.MethodGet, http.MethodHead, http.MethodOptions, http.MethodPost} + for _, path := range paths { + for _, method := range methods { + if status, _ := send(server.URL, method, path); status != http.StatusNotFound { + t.Errorf("demo session %s %s status = %d, want %d", method, path, status, http.StatusNotFound) + } + } + } +} + func TestContract_ReleaseDemoFixtureRuntimeGuardrailsRemainCanonical(t *testing.T) { requiredSnippets := map[string][]string{ "licensing_handlers.go": { diff --git a/internal/api/demo_mode_operations.go b/internal/api/demo_mode_operations.go index bad26c86c4..d661fc2204 100644 --- a/internal/api/demo_mode_operations.go +++ b/internal/api/demo_mode_operations.go @@ -1,6 +1,9 @@ package api -import "net/http" +import ( + "net/http" + "strings" +) var publicDemoAdminOperationsPolicies = []publicDemoCommercialRoutePolicy{ { @@ -43,6 +46,17 @@ var publicDemoAdminOperationsPolicies = []publicDemoCommercialRoutePolicy{ exposure: publicDemoCommercialExposureHidden, matches: readDemoCommercialPath("/api/discover"), }, + // Go runtime profiling. These routes are gated only on admin auth, which + // a demo login that is the configured admin passes, so every visitor could + // otherwise pull goroutine and heap dumps or the command line, or start + // CPU profiles and traces that cost the whole process. Hidden for every + // method, so a write probe such as the POST symbol lookup gets the same + // 404 instead of revealing the route. + { + route: "/debug/pprof", + exposure: publicDemoCommercialExposureHidden, + matches: familyDemoCommercialPath("/debug/pprof"), + }, } func publicDemoAdminOperationsPolicyForRequest( @@ -55,3 +69,15 @@ func publicDemoAdminOperationsPolicyForRequest( } return "", false } + +// familyDemoCommercialPath matches root and every path below it, for any +// method. +func familyDemoCommercialPath(root string) func(*http.Request) bool { + return func(r *http.Request) bool { + if r == nil || r.URL == nil { + return false + } + path := normalizeDemoCommercialPath(r.URL.Path) + return path == root || strings.HasPrefix(path, root+"/") + } +} diff --git a/internal/api/demo_mode_operations_test.go b/internal/api/demo_mode_operations_test.go index 2596013369..907702ccc5 100644 --- a/internal/api/demo_mode_operations_test.go +++ b/internal/api/demo_mode_operations_test.go @@ -3,6 +3,7 @@ package api import ( "net/http" "net/http/httptest" + "strings" "testing" ) @@ -23,6 +24,16 @@ func TestPublicDemoAdminOperationsPolicyForRequest(t *testing.T) { {name: "admin users head", method: http.MethodHead, path: "/api/admin/users"}, {name: "manual discovery read", method: http.MethodGet, path: "/api/discover"}, {name: "manual discovery head", method: http.MethodHead, path: "/api/discover"}, + {name: "pprof root", method: http.MethodGet, path: "/debug/pprof"}, + {name: "pprof index", method: http.MethodGet, path: "/debug/pprof/"}, + {name: "pprof heap with forced gc", method: http.MethodGet, path: "/debug/pprof/heap?gc=1"}, + {name: "pprof goroutine dump", method: http.MethodGet, path: "/debug/pprof/goroutine?debug=2"}, + {name: "pprof cmdline", method: http.MethodGet, path: "/debug/pprof/cmdline"}, + {name: "pprof cpu profile", method: http.MethodGet, path: "/debug/pprof/profile?seconds=30"}, + {name: "pprof trace", method: http.MethodGet, path: "/debug/pprof/trace?seconds=5"}, + {name: "pprof symbol lookup", method: http.MethodPost, path: "/debug/pprof/symbol"}, + {name: "pprof head", method: http.MethodHead, path: "/debug/pprof/allocs"}, + {name: "pprof options", method: http.MethodOptions, path: "/debug/pprof/"}, } for _, tc := range hiddenRoutes { @@ -48,6 +59,7 @@ func TestPublicDemoAdminOperationsPolicyForRequest(t *testing.T) { {name: "runtime capabilities", method: http.MethodGet, path: "/api/license/runtime-capabilities"}, {name: "admin users write", method: http.MethodPost, path: "/api/admin/users"}, {name: "manual discovery write", method: http.MethodPost, path: "/api/discover"}, + {name: "pprof lookalike", method: http.MethodGet, path: "/debug/pprofile"}, } for _, tc := range allowedRoutes { @@ -59,3 +71,52 @@ func TestPublicDemoAdminOperationsPolicyForRequest(t *testing.T) { }) } } + +// Every literal route the router files register under /debug/, with or without +// a method prefix, must stay hidden on the public demo whatever the method, so +// a new debug handler cannot ship reachable to the demo admin session by +// default. +func TestPublicDemoAdminOperationsPolicyHidesEveryDebugRoute(t *testing.T) { + literalRoutes, _, _ := parseRouterRoutes(t) + + methods := []string{ + http.MethodGet, + http.MethodHead, + http.MethodOptions, + http.MethodPost, + http.MethodPut, + http.MethodPatch, + http.MethodDelete, + } + + var debugRoutes int + for _, route := range literalRoutes { + // ServeMux separates a method from its path with spaces or tabs. + pattern := route + if i := strings.IndexAny(route, " \t"); i > 0 { + if rest := strings.TrimLeft(route[i+1:], " \t"); strings.HasPrefix(rest, "/") { + pattern = rest + } + } + if !strings.HasPrefix(pattern, "/debug/") { + continue + } + debugRoutes++ + paths := []string{pattern} + if strings.HasSuffix(pattern, "/") { + paths = append(paths, pattern+"goroutine") + } + for _, path := range paths { + for _, method := range methods { + req := httptest.NewRequest(method, path, nil) + exposure, ok := publicDemoAdminOperationsPolicyForRequest(req) + if !ok || exposure != publicDemoCommercialExposureHidden { + t.Errorf("%s %s is not hidden on the public demo (matched=%v exposure=%q)", method, path, ok, exposure) + } + } + } + } + if debugRoutes == 0 { + t.Fatal("no /debug/ routes found in the router inventory") + } +} diff --git a/internal/api/metrics_reporting_handlers.go b/internal/api/metrics_reporting_handlers.go index 08e4c7b240..f918817170 100644 --- a/internal/api/metrics_reporting_handlers.go +++ b/internal/api/metrics_reporting_handlers.go @@ -7,6 +7,7 @@ import ( "fmt" "io" "net/http" + "net/url" "os" "regexp" "strings" @@ -99,6 +100,7 @@ type ReportingHandlers struct { mtMonitor *monitoring.MultiTenantMonitor recoveryManager *recoverymanager.Manager narratorResolver func(ctx context.Context) (reporting.Narrator, reporting.FleetNarrator, reporting.FindingsProvider) + existingFindingsResolver func(ctx context.Context) reporting.FindingsProvider settingsStore reportingSystemSettingsStore scheduleRunMu sync.Mutex commercialLicenseResolver func(ctx context.Context) *licenseService @@ -168,6 +170,23 @@ func (h *ReportingHandlers) resolveNarrator(ctx context.Context) (reporting.Narr return h.narratorResolver(ctx) } +// SetExistingFindingsResolver wires the Patrol findings lookup that GET reports +// use. Unlike the narrator resolver it must return only an AI service that is +// already running, never construct one. +func (h *ReportingHandlers) SetExistingFindingsResolver(resolver func(ctx context.Context) reporting.FindingsProvider) { + if h == nil { + return + } + h.existingFindingsResolver = resolver +} + +func (h *ReportingHandlers) resolveExistingFindings(ctx context.Context) reporting.FindingsProvider { + if h == nil || h.existingFindingsResolver == nil { + return nil + } + return h.existingFindingsResolver(ctx) +} + func (h *ReportingHandlers) resolveReportBranding(ctx context.Context) reporting.ReportBranding { service := getLicenseServiceForContext(ctx) entitled := service != nil && service.HasFeature(featureWhiteLabelValue) @@ -300,28 +319,33 @@ func normalizePerformanceReportTimeRange( return start, end, nil } -func decodeMultiReportRequestBody(w http.ResponseWriter, r *http.Request) (multiReportRequestBody, bool) { +// decodeReportingRequestBody strictly decodes one JSON report request body: +// at most 1MB, no unknown fields, no trailing payload. +func decodeReportingRequestBody[T any](w http.ResponseWriter, r *http.Request) (T, bool) { r.Body = http.MaxBytesReader(w, r.Body, reportingMultiReportBodyMax) - var body multiReportRequestBody + var body, zero T decoder := json.NewDecoder(r.Body) decoder.DisallowUnknownFields() - if err := decoder.Decode(&body); err != nil { + err := decoder.Decode(&body) + if err == nil { + // Anything after the value is invalid, and reading past it can also + // hit the size cap. + if trailing := decoder.Decode(&struct{}{}); trailing != io.EOF { + err = errors.New("trailing payload") + if trailing != nil { + err = trailing + } + } + } + if err != nil { var maxBytesErr *http.MaxBytesError if errors.As(err, &maxBytesErr) { writeErrorResponse(w, http.StatusBadRequest, "body_too_large", "Request body must be 1MB or less", nil) - return multiReportRequestBody{}, false + return zero, false } writeErrorResponse(w, http.StatusBadRequest, "invalid_body", "Invalid request body", nil) - return multiReportRequestBody{}, false - } - if decoder.More() { - writeErrorResponse(w, http.StatusBadRequest, "invalid_body", "Invalid request body", nil) - return multiReportRequestBody{}, false - } - if err := decoder.Decode(&struct{}{}); err != io.EOF { - writeErrorResponse(w, http.StatusBadRequest, "invalid_body", "Invalid request body", nil) - return multiReportRequestBody{}, false + return zero, false } return body, true @@ -539,9 +563,67 @@ func (h *ReportingHandlers) listBackupsForReport(ctx context.Context, orgID stri return out } -// HandleGenerateReport generates a report +// singleReportRequestBody carries the single-resource report fields. POST +// sends them as a JSON body; GET sends the same names as query parameters. +type singleReportRequestBody struct { + ResourceType string `json:"resourceType"` + ResourceID string `json:"resourceId"` + Format string `json:"format"` + Start string `json:"start"` + End string `json:"end"` + Title string `json:"title"` + MetricType string `json:"metricType"` +} + +func singleReportRequestFromQuery(q url.Values) singleReportRequestBody { + return singleReportRequestBody{ + ResourceType: q.Get("resourceType"), + ResourceID: q.Get("resourceId"), + Format: q.Get("format"), + Start: q.Get("start"), + End: q.Get("end"), + Title: q.Get("title"), + MetricType: q.Get("metricType"), + } +} + +// reportGETNarrativeNote takes the place of the PDF's "Configure Pulse +// Assistant" tip on a GET report, which stays deterministic whatever the +// Assistant settings. +const reportGETNarrativeNote = "Tip: This report was requested with GET, which never uses Pulse Assistant. Generate it from Settings, or request it with POST, to include AI narration when Pulse Assistant is configured." + +// getReportNarrator narrates GET single-resource reports with the +// deterministic summary and never calls an AI provider. +type getReportNarrator struct{} + +func (getReportNarrator) Narrate(ctx context.Context, in reporting.NarrativeInput) (reporting.Narrative, error) { + out, err := reporting.HeuristicNarrator{}.Narrate(ctx, in) + out.Source = reporting.NarrativeSourceHeuristic + out.Disclaimer = reportGETNarrativeNote + return out, err +} + +// HandleGenerateReport generates a single-resource report. POST with a JSON +// body is the canonical transport, used by the settings UI, and the only one +// that may narrate a PDF with the tenant's Pulse Assistant, which spends AI +// budget and records usage in the cost ledger. GET takes the same fields as +// query parameters and always narrates deterministically: GET passes the +// demo-mode guard and the CSRF check, and SameSite=Lax session cookies ride +// cross-site top-level GET navigations, so a link must not be able to spend +// that budget. func (h *ReportingHandlers) HandleGenerateReport(w http.ResponseWriter, r *http.Request) { - if r.Method != http.MethodGet { + var params singleReportRequestBody + switch r.Method { + case http.MethodGet: + params = singleReportRequestFromQuery(r.URL.Query()) + case http.MethodPost: + body, ok := decodeReportingRequestBody[singleReportRequestBody](w, r) + if !ok { + return + } + params = body + default: + w.Header().Set("Allow", "GET, POST") http.Error(w, "Method not allowed", http.StatusMethodNotAllowed) return } @@ -553,15 +635,14 @@ func (h *ReportingHandlers) HandleGenerateReport(w http.ResponseWriter, r *http. } definition := performanceReportDefinition() - q := r.URL.Query() - format, err := normalizePerformanceReportFormat(q.Get("format"), definition) + format, err := normalizePerformanceReportFormat(params.Format, definition) if err != nil { writeErrorResponse(w, http.StatusBadRequest, "invalid_format", err.Error(), nil) return } - resourceTypeRaw := q.Get("resourceType") - resourceID := q.Get("resourceId") + resourceTypeRaw := params.ResourceType + resourceID := params.ResourceID if resourceTypeRaw == "" || resourceID == "" { writeErrorResponse(w, http.StatusBadRequest, "missing_params", "resourceType and resourceId are required", nil) return @@ -584,8 +665,8 @@ func (h *ReportingHandlers) HandleGenerateReport(w http.ResponseWriter, r *http. metricType, title, err := normalizePerformanceReportOptionalFields( definition, - q.Get("metricType"), - q.Get("title"), + params.MetricType, + params.Title, ) if err != nil { code := "invalid_metric_type" @@ -599,8 +680,8 @@ func (h *ReportingHandlers) HandleGenerateReport(w http.ResponseWriter, r *http. start, end, err := normalizePerformanceReportTimeRange( definition, - q.Get("start"), - q.Get("end"), + params.Start, + params.End, time.Now(), ) if err != nil { @@ -628,9 +709,18 @@ func (h *ReportingHandlers) HandleGenerateReport(w http.ResponseWriter, r *http. // Wire the per-tenant AI narrator and Patrol findings provider when // configured. Both are nil-safe at the engine layer; absence falls // back to the heuristic narrator with no findings section. Single- - // resource reports do not use the FleetNarrator. - narrator, _, findings := h.resolveNarrator(r.Context()) - req.Narrator = narrator + // resource reports do not use the FleetNarrator. Only POST resolves the + // AI service; GET narrates deterministically and reads findings only + // from an AI service that is already running. + var narrator reporting.Narrator + var findings reporting.FindingsProvider + if r.Method == http.MethodPost { + narrator, _, findings = h.resolveNarrator(r.Context()) + req.Narrator = narrator + } else { + findings = h.resolveExistingFindings(r.Context()) + req.Narrator = getReportNarrator{} + } req.FindingsProvider = findings data, contentType, err := engine.Generate(req) @@ -647,6 +737,7 @@ func (h *ReportingHandlers) HandleGenerateReport(w http.ResponseWriter, r *http. log.Info(). Str("event", "reporting.single.generated"). Str("org_id", orgID). + Str("method", r.Method). Str("resource_type", resourceType). Str("metric_type", metricType). Str("format", string(format)). @@ -1187,7 +1278,7 @@ func (h *ReportingHandlers) HandleGenerateMultiReport(w http.ResponseWriter, r * http.Error(w, "Method not allowed", http.StatusMethodNotAllowed) return } - body, ok := decodeMultiReportRequestBody(w, r) + body, ok := decodeReportingRequestBody[multiReportRequestBody](w, r) if !ok { return } diff --git a/internal/api/reporting_handlers_test.go b/internal/api/reporting_handlers_test.go index 00851e4a56..f5adb84e9d 100644 --- a/internal/api/reporting_handlers_test.go +++ b/internal/api/reporting_handlers_test.go @@ -75,7 +75,7 @@ func (p reportBrandLicenseProvider) Service(context.Context) *pkglicensing.Servi func TestReportingHandlers_MethodNotAllowed(t *testing.T) { handler := NewReportingHandlers(nil, nil) - req := httptest.NewRequest(http.MethodPost, "/api/reporting", nil) + req := httptest.NewRequest(http.MethodPut, "/api/reporting", nil) rr := httptest.NewRecorder() handler.HandleGenerateReport(rr, req) @@ -83,6 +83,47 @@ func TestReportingHandlers_MethodNotAllowed(t *testing.T) { if rr.Code != http.StatusMethodNotAllowed { t.Fatalf("expected status %d, got %d", http.StatusMethodNotAllowed, rr.Code) } + if got := rr.Header().Get("Allow"); got != "GET, POST" { + t.Fatalf("expected Allow GET, POST, got %q", got) + } +} + +func TestReportingHandlers_GenerateReportRejectsInvalidPOSTBody(t *testing.T) { + engine := &stubReportingEngine{data: []byte("report"), contentType: "application/pdf"} + original := reporting.GetEngine() + reporting.SetEngine(engine) + t.Cleanup(func() { reporting.SetEngine(original) }) + + handler := NewReportingHandlers(nil, nil) + for name, body := range map[string]string{ + "empty": ``, + "unknown field": `{"resourceType":"node","resourceId":"node-1","narrate":true}`, + "trailing value": `{"resourceType":"node","resourceId":"node-1"} {}`, + "query shape": `resourceType=node&resourceId=node-1`, + } { + req := httptest.NewRequest(http.MethodPost, "/api/reporting", strings.NewReader(body)) + rr := httptest.NewRecorder() + handler.HandleGenerateReport(rr, req) + if rr.Code != http.StatusBadRequest || !strings.Contains(rr.Body.String(), "invalid_body") { + t.Fatalf("%s: expected 400 invalid_body, got %d body=%s", name, rr.Code, rr.Body.String()) + } + } + + req := httptest.NewRequest(http.MethodPost, "/api/reporting?resourceType=node&resourceId=node-1", strings.NewReader(`{}`)) + rr := httptest.NewRecorder() + handler.HandleGenerateReport(rr, req) + if rr.Code != http.StatusBadRequest || !strings.Contains(rr.Body.String(), "missing_params") { + t.Fatalf("POST must read fields from the body, not the query: got %d body=%s", rr.Code, rr.Body.String()) + } + + // A valid value followed by padding past the cap is too large, not malformed. + padded := `{"resourceType":"node","resourceId":"node-1"}` + strings.Repeat(" ", reportingMultiReportBodyMax) + req = httptest.NewRequest(http.MethodPost, "/api/reporting", strings.NewReader(padded)) + rr = httptest.NewRecorder() + handler.HandleGenerateReport(rr, req) + if rr.Code != http.StatusBadRequest || !strings.Contains(rr.Body.String(), "body_too_large") { + t.Fatalf("expected 400 body_too_large for an oversized body, got %d body=%s", rr.Code, rr.Body.String()) + } } func TestReportingHandlers_EngineUnavailable(t *testing.T) { diff --git a/internal/api/router.go b/internal/api/router.go index 6f77a32878..60dcd7343d 100644 --- a/internal/api/router.go +++ b/internal/api/router.go @@ -805,33 +805,7 @@ func (r *Router) setupRoutes() { // tenant services are refreshed when Router.SetMonitor replaces a monitor. r.configureMetadataProviderFactory() - // Wire the per-tenant AI narrator, fleet narrator, and Patrol - // findings provider into reporting. The AI service implements all - // three interfaces; when not configured for the tenant the engine - // falls back to the heuristic narrators with no findings section. - if r.reportingHandlers != nil { - settings := r.aiSettingsHandler - r.reportingHandlers.SetPatrolDigestResolver(func(ctx context.Context, days int) (ai.PatrolDigest, bool) { - if settings == nil { - return ai.PatrolDigest{}, false - } - return settings.BuildPatrolDigest(ctx, days) - }) - r.reportingHandlers.SetNarratorResolver(func(ctx context.Context) (reporting.Narrator, reporting.FleetNarrator, reporting.FindingsProvider) { - if settings == nil { - return nil, nil, nil - } - svc := settings.GetAIService(ctx) - if svc == nil { - return nil, nil, nil - } - cfg := svc.GetAIConfig() - if cfg == nil || !cfg.Enabled { - return nil, nil, nil - } - return svc, svc, svc - }) - } + r.wireReportingAIResolvers() // AI chat handler r.aiHandler = NewAIHandler(r.multiTenant, r.mtMonitor, r.agentExecServer) @@ -1031,6 +1005,50 @@ func (r *Router) setupRoutes() { // See issue #334 - ServeMux redirects empty path to "./" which breaks reverse proxies } +// wireReportingAIResolvers connects reporting to the per-tenant AI service. +func (r *Router) wireReportingAIResolvers() { + // Wire the per-tenant AI narrator, fleet narrator, and Patrol + // findings provider into reporting. The AI service implements all + // three interfaces; when not configured for the tenant the engine + // falls back to the heuristic narrators with no findings section. + if r.reportingHandlers != nil { + settings := r.aiSettingsHandler + r.reportingHandlers.SetPatrolDigestResolver(func(ctx context.Context, days int) (ai.PatrolDigest, bool) { + if settings == nil { + return ai.PatrolDigest{}, false + } + return settings.BuildPatrolDigest(ctx, days) + }) + r.reportingHandlers.SetNarratorResolver(func(ctx context.Context) (reporting.Narrator, reporting.FleetNarrator, reporting.FindingsProvider) { + if settings == nil { + return nil, nil, nil + } + svc := settings.GetAIService(ctx) + if svc == nil { + return nil, nil, nil + } + cfg := svc.GetAIConfig() + if cfg == nil || !cfg.Enabled { + return nil, nil, nil + } + return svc, svc, svc + }) + r.reportingHandlers.SetExistingFindingsResolver(func(ctx context.Context) reporting.FindingsProvider { + if settings == nil { + return nil + } + svc := settings.ExistingAIService(ctx) + if svc == nil { + return nil + } + if cfg := svc.GetAIConfig(); cfg == nil || !cfg.Enabled { + return nil + } + return svc + }) + } +} + // CleanupTenant removes all per-tenant resources (RBAC, AI, License) for a deleted org. func (r *Router) CleanupTenant(ctx context.Context, orgID string) error { orgID = strings.TrimSpace(orgID) diff --git a/internal/monitoring/issue1595_collection_trust_test.go b/internal/monitoring/issue1595_collection_trust_test.go index 6a47558c65..e3de00161e 100644 --- a/internal/monitoring/issue1595_collection_trust_test.go +++ b/internal/monitoring/issue1595_collection_trust_test.go @@ -681,6 +681,167 @@ func TestAgentDiskHistoryFollowsTheSerialItsUnraidRowReports(t *testing.T) { } } +// An Unraid host's disk can show a temperature only its Unraid inventory +// reports: a member the agent's SMART collection skips (--disk-exclude), or +// one Unraid reads with its own per-disk SMART settings while the agent's +// probe returns no temperature. The chart must hold the reading the disk +// shows, under the key its metrics target reads, and nothing for a reading the +// disk does not show as collected (a spun-down disk, a host past its lease) or +// does not show at all (a controller member behind the device Unraid reads). +func TestAgentDiskChartsTheUnraidTemperatureItShows(t *testing.T) { + storeConfig := metrics.DefaultConfig(t.TempDir()) + storeConfig.FlushInterval = time.Hour + store, err := metrics.NewStore(storeConfig) + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { _ = store.Close() }) + m := &Monitor{metricsStore: store, rateTracker: NewRateTracker()} + + now := time.Now().UTC().Truncate(time.Second) + host := models.Host{ID: "host-tower", Hostname: "tower", MachineID: "machine-tower", Status: "online", LastSeen: now} + host.Sensors.SMART = []models.HostDiskSMART{ + // The agent's identity-only row for a disk its probe got nothing from. + {Device: "sdb", Serial: "SER-B", Health: "UNKNOWN", Collection: &diskinventory.CollectionStatus{ + Temperature: diskinventory.Unavailable("smartctl", "SMART probe returned no usable temperature data")}}, + // A row with its own reading keeps it; the Unraid one is not written too. + {Device: "sdc", WWN: "0x5000c500aaaa0003", Health: "PASSED", Temperature: 41, Collection: &diskinventory.CollectionStatus{ + Temperature: diskinventory.Available("smartctl")}}, + // The agent's native standby row for a disk Unraid reports spun down. + {Device: "sdd", Serial: "SER-D", Health: "UNKNOWN", Standby: true, Collection: &diskinventory.CollectionStatus{ + Temperature: diskinventory.Unavailable("unraid", "disk is reported spun down")}}, + // Controller members behind one kernel block device, which the Unraid + // row describes as a whole: neither member takes its temperature. + {Device: "sdh", Controller: "sdh", Target: "megaraid,0", WWN: "0x5000c500bbbb0001", Health: "PASSED", Collection: &diskinventory.CollectionStatus{ + Temperature: diskinventory.Unavailable("smartctl", "SMART probe returned no usable temperature data")}}, + {Device: "sdh", Controller: "sdh", Target: "megaraid,1", WWN: "0x5000c500bbbb0002", Health: "PASSED", Collection: &diskinventory.CollectionStatus{ + Temperature: diskinventory.Unavailable("smartctl", "SMART probe returned no usable temperature data")}}, + // A member whose own serial is the Unraid row's is the disk that row + // describes, though its path is shared. + {Device: "sdi", Controller: "sdi", Target: "megaraid,0", Serial: "SER-I", Health: "UNKNOWN", Collection: &diskinventory.CollectionStatus{ + Temperature: diskinventory.Unavailable("smartctl", "SMART probe returned no usable temperature data")}}, + {Device: "sdi", Controller: "sdi", Target: "megaraid,1", Serial: "SER-I2", Health: "UNKNOWN", Collection: &diskinventory.CollectionStatus{ + Temperature: diskinventory.Unavailable("smartctl", "SMART probe returned no usable temperature data")}}, + // A placeholder serial is no evidence of which member the row is. + {Device: "sdn", Controller: "sdn", Target: "megaraid,0", Serial: "UNKNOWN", WWN: "0x5000c500cccc0001", Health: "UNKNOWN", Collection: &diskinventory.CollectionStatus{ + Temperature: diskinventory.Unavailable("smartctl", "SMART probe returned no usable temperature data")}}, + {Device: "sdn", Controller: "sdn", Target: "megaraid,1", Serial: "UNKNOWN", WWN: "0x5000c500cccc0002", Health: "UNKNOWN", Collection: &diskinventory.CollectionStatus{ + Temperature: diskinventory.Unavailable("smartctl", "SMART probe returned no usable temperature data")}}, + // A placeholder serial matches no Unraid row; the row's path finds its own. + {Device: "sdo", Serial: "0123456789", WWN: "0x5000c500eeee0001", Health: "UNKNOWN", Collection: &diskinventory.CollectionStatus{ + Temperature: diskinventory.Unavailable("smartctl", "SMART probe returned no usable temperature data")}}, + // The only reported member behind a controller path is still a member. + {Device: "sdq", Controller: "sdq", Target: "megaraid,0", WWN: "0x5000c500ffff0001", Health: "UNKNOWN", Collection: &diskinventory.CollectionStatus{ + Temperature: diskinventory.Unavailable("smartctl", "SMART probe returned no usable temperature data")}}, + // Without a serial or WWN, a member charts its own reading under its + // topology key. + {Device: "sdr", Controller: "sdr", Target: "megaraid,0", Health: "PASSED", Temperature: 35, Collection: &diskinventory.CollectionStatus{ + Temperature: diskinventory.Available("smartctl")}}, + // A SMART row's key is its own: the merged disk shows the row's reading, + // here not collected, and the Unraid one is not charted over it. + {Device: "sdm", WWN: "0x5000c500dddd0001", Health: "PASSED", Temperature: 40, Collection: &diskinventory.CollectionStatus{ + Temperature: diskinventory.Unavailable("smartctl", "disk is in standby")}}, + } + host.Unraid = &models.HostUnraidStorage{ArrayStarted: true, Disks: []models.HostUnraidDisk{ + {Device: "sdb", Serial: "SER-B", Name: "disk1", Role: "data", Status: "online", Temperature: 38}, + {Device: "sdc", Serial: "SER-C", Name: "disk2", Role: "data", Status: "online", Temperature: 39}, + {Device: "sdd", Serial: "SER-D", Name: "disk3", Role: "data", Status: "online", Temperature: 35, SpunDown: true}, + // Members with no SMART row at all. + {Device: "sde", Serial: "SER-E", Name: "disk4", Role: "data", Status: "online", Temperature: 36}, + {Device: "sdf", Serial: "SER-F", Name: "disk5", Role: "data", Status: "online", Temperature: 33, SpunDown: true}, + {Device: "sdg", Name: "disk6", Role: "data", Status: "online", Temperature: 37}, + {Device: "sdh", Serial: "SER-H", Name: "disk7", Role: "data", Status: "online", Temperature: 42}, + {Device: "sdi", Serial: "SER-I", Name: "disk8", Role: "data", Status: "online", Temperature: 44}, + // Rows sharing one history key are one disk, showing the latest reading. + {Device: "sdj", Name: "disk9", Role: "data", Status: "online"}, + {Device: "sdj", Name: "disk10", Role: "data", Status: "online", Temperature: 43}, + {Device: "sdk", Name: "disk11", Role: "data", Status: "online", Temperature: 38}, + {Device: "sdk", Name: "disk12", Role: "data", Status: "online", Temperature: 39}, + {Device: "sdl", Name: "disk13", Role: "data", Status: "online", Temperature: 40}, + {Device: "sdl", Name: "disk14", Role: "data", Status: "online", Temperature: 32, SpunDown: true}, + {Device: "sdn", Serial: "UNKNOWN", Name: "disk15", Role: "data", Status: "online", Temperature: 45}, + {Device: "sdm", Serial: "SER-M", Name: "disk16", Role: "data", Status: "online", Temperature: 37}, + {Device: "sdp", Serial: "0123456789", Name: "disk17", Role: "data", Status: "online", Temperature: 30}, + {Device: "sdo", Serial: "0123456789", Name: "disk18", Role: "data", Status: "online", Temperature: 46}, + {Device: "sdq", Serial: "SER-Q", Name: "disk19", Role: "data", Status: "online", Temperature: 47}, + {Device: "sdr", Serial: "SER-R", Name: "disk20", Role: "data", Status: "online", Temperature: 36}, + }} + // Disks by usable serial, else WWN, else device path, and the reading each + // shows as collected. + want := map[string]float64{ + "SER-B": 38, "SER-C": 41, "SER-D": 0, "SER-E": 36, "SER-F": 0, "sdg": 37, + "SER-H": 42, "0x5000c500bbbb0001": 0, "0x5000c500bbbb0002": 0, + "SER-I": 44, "SER-I2": 0, "sdj": 43, "sdk": 39, "sdl": 0, + "sdn": 45, "0x5000c500cccc0001": 0, "0x5000c500cccc0002": 0, "SER-M": 0, + "0x5000c500eeee0001": 46, "sdo": 46, "sdp": 30, "0x5000c500ffff0001": 0, "SER-Q": 47, + "sdr": 35, "SER-R": 36, + } + + m.writeHostSMARTMetrics(host, now) + // Once the host's lease expires its last report is kept, offline, as + // context; nothing it carries is a current reading. + reported := host + reported.Sensors.SMART = append([]models.HostDiskSMART(nil), host.Sensors.SMART...) + state := &models.State{Hosts: []models.Host{reported}} + offline, expired := state.ExpireHostTelemetry(host.ID, host.LastSeen) + if !expired { + t.Fatal("host telemetry did not expire") + } + m.writeHostSMARTMetrics(offline, now.Add(30*time.Second)) + store.Flush() + + live := unifiedresources.NewRegistry(nil) + live.IngestSnapshot(models.StateSnapshot{Hosts: []models.Host{host}}) + rehydrated := unifiedresources.NewRegistry(nil) + rehydrated.IngestResources(live.List()) + for _, tc := range []struct { + name string + registry *unifiedresources.ResourceRegistry + }{{"live", live}, {"rehydrated", rehydrated}} { + disks := tc.registry.ListByType(unifiedresources.ResourceTypePhysicalDisk) + if len(disks) != len(want) { + t.Fatalf("%s: disks = %d, want %d", tc.name, len(disks), len(want)) + } + for _, disk := range disks { + device := disk.PhysicalDisk.Serial + if !diskinventory.IsUsableHardwareID(device) { + device = "" + } + if device == "" { + device = disk.PhysicalDisk.WWN + } + if device == "" { + device = disk.PhysicalDisk.DevPath + } + wantTemp, ok := want[device] + if !ok { + t.Fatalf("%s: unexpected disk %s at %q", tc.name, disk.ID, device) + } + shown := 0.0 + if diskinventory.TemperatureCollected(disk.PhysicalDisk.Temperature, disk.PhysicalDisk.Collection) { + shown = float64(disk.PhysicalDisk.Temperature) + } + if shown != wantTemp { + t.Fatalf("%s: disk %s shows %vยฐC as collected, want %vยฐC", tc.name, device, shown, wantTemp) + } + target := tc.registry.MetricsTarget(disk.ID) + if target == nil { + t.Fatalf("%s: disk %s has no metrics target", tc.name, device) + } + points, err := store.Query(target.ResourceType, target.ResourceID, "smart_temp", now.Add(-time.Minute), now.Add(time.Minute), 0) + if err != nil { + t.Fatal(err) + } + switch { + case wantTemp == 0 && len(points) != 0: + t.Errorf("%s: disk %s shows no collected temperature but target %+v charts %+v", tc.name, device, *target, points) + case wantTemp > 0 && (len(points) != 1 || points[0].Value != wantTemp || !points[0].Timestamp.Equal(now)): + t.Errorf("%s: disk %s shows %vยฐC but target %+v charts %+v, want one sample at the online report", tc.name, device, wantTemp, *target, points) + } + } + } +} + // A SMART row found only by device path may describe the slot's previous // occupant. It is refused when its identity contradicts the Proxmox disk's, // judged only on what both producers report like for like: WWNs for every diff --git a/internal/monitoring/issue1913_host_continuity_test.go b/internal/monitoring/issue1913_host_continuity_test.go index 5134592e7b..192540bc5c 100644 --- a/internal/monitoring/issue1913_host_continuity_test.go +++ b/internal/monitoring/issue1913_host_continuity_test.go @@ -89,3 +89,120 @@ func TestHostContinuityPreservesLiveAgentAfterReenrollment(t *testing.T) { }) } } + +// Continuity hydration clones the registry through resource ingest, which +// applies operator links, while the rebuild ingests supplemental records after +// the snapshot. A link to a record-sourced guest (an agent inside a vSphere VM) +// must hold in the published inventory either way, or the state the websocket +// broadcasts and the resources API seeds from flips with an unrelated absent +// machine. A saved enrollment only fills an absent machine: when the linked +// agent itself is gone, its continuity row stays separate and the guest keeps +// what vSphere reports. +func TestManualLinkToSupplementalGuestHoldsWithAndWithoutContinuity(t *testing.T) { + now := time.Now().UTC() + guestAgent := models.Host{ + ID: "host-app-guest", MachineID: "machine-app-guest", Hostname: "app-guest", + Platform: "linux", Status: "online", LastSeen: now, IntervalSeconds: 30, + } + vmRecords := []unifiedresources.IngestRecord{{ + SourceID: "vc-1:vm:vm-42", + Resource: unifiedresources.Resource{ + Type: unifiedresources.ResourceTypeVM, + Technology: "vmware", + Name: "app-guest", + Status: unifiedresources.StatusOnline, + LastSeen: now, + VMware: &unifiedresources.VMwareData{ConnectionID: "vc-1", ManagedObjectID: "vm-42", EntityType: "vm"}, + }, + }} + + unlinked := unifiedresources.NewMonitorAdapter(unifiedresources.NewRegistry(nil)) + unlinked.PopulateSnapshotAndSupplemental( + models.StateSnapshot{Hosts: []models.Host{guestAgent}, LastUpdate: now}, + map[unifiedresources.DataSource][]unifiedresources.IngestRecord{unifiedresources.SourceVMware: vmRecords}, + ) + if len(unlinked.VMs()) != 1 || len(unlinked.Hosts()) != 1 { + t.Fatalf("unlinked estate = %d VMs, %d agents, want one of each", len(unlinked.VMs()), len(unlinked.Hosts())) + } + vmID, agentID := unlinked.VMs()[0].ID(), unlinked.Hosts()[0].ID() + + for _, tc := range []struct { + name string + reporting bool + continuity []config.HostContinuityEntry + }{ + {name: "no-continuity", reporting: true}, + { + name: "absent-machine-continuity", + reporting: true, + continuity: []config.HostContinuityEntry{{ + HostID: "absent-machine", MachineID: "other-machine", Hostname: "other-host", + Platform: "linux", LastSeen: now.Add(-time.Hour), + }}, + }, + { + name: "linked-agent-continuity-only", + continuity: []config.HostContinuityEntry{{ + HostID: guestAgent.ID, MachineID: guestAgent.MachineID, Hostname: guestAgent.Hostname, + Platform: "linux", LastSeen: now.Add(-10 * time.Minute), + }}, + }, + } { + t.Run(tc.name, func(t *testing.T) { + links := unifiedresources.NewMemoryStore() + if err := links.AddLink(unifiedresources.ResourceLink{ResourceA: vmID, ResourceB: agentID, PrimaryID: vmID}); err != nil { + t.Fatalf("add link: %v", err) + } + state := models.NewState() + if tc.reporting { + state.Hosts = []models.Host{guestAgent} + } + m := &Monitor{state: state} + if len(tc.continuity) > 0 { + continuity := config.NewHostContinuityStore(t.TempDir(), nil) + for _, entry := range tc.continuity { + if err := continuity.Upsert(entry); err != nil { + t.Fatal(err) + } + } + m.hostContinuityStore = continuity + } + m.SetResourceStore(unifiedresources.NewMonitorAdapter(unifiedresources.NewRegistry(links))) + m.SetSupplementalRecordsProvider(unifiedresources.SourceVMware, &testSupplementalInventoryReadinessProvider{ + source: unifiedresources.SourceVMware, + records: vmRecords, + settled: true, + readyAt: now, + }) + + resources, _ := m.UnifiedResourceSnapshot() + var guest, agent *unifiedresources.Resource + for i := range resources { + switch resources[i].ID { + case agentID: + agent = &resources[i] + case vmID: + guest = &resources[i] + } + } + if guest == nil { + t.Fatalf("linked VM %s missing from published inventory", vmID) + } + if tc.reporting { + if agent != nil { + t.Fatalf("linked agent %s still published standalone", agentID) + } + if guest.VMware == nil || guest.Agent == nil { + t.Fatalf("published VM vmware=%t agent=%t, want the VM carrying its agent", guest.VMware != nil, guest.Agent != nil) + } + return + } + if agent == nil || agent.Status != unifiedresources.StatusOffline { + t.Fatalf("saved enrollment row = %+v, want its own offline row", agent) + } + if guest.Agent != nil || guest.Status != unifiedresources.StatusOnline { + t.Fatalf("published VM agent=%t status=%s, want vSphere's online VM untouched by the saved enrollment", guest.Agent != nil, guest.Status) + } + }) + } +} diff --git a/internal/monitoring/monitor_agents.go b/internal/monitoring/monitor_agents.go index 85599d39e7..598c6b3bd5 100644 --- a/internal/monitoring/monitor_agents.go +++ b/internal/monitoring/monitor_agents.go @@ -4081,14 +4081,18 @@ func (m *Monitor) writeHostSMARTMetrics(host models.Host, now time.Time) { return } + smartMetricIDs := make(map[string]struct{}, len(host.Sensors.SMART)) for _, disk := range host.Sensors.SMART { resourceID := unifiedresources.HostSMARTDiskMetricID(host, disk) if resourceID == "" { continue } + smartMetricIDs[resourceID] = struct{}{} - if diskinventory.TemperatureCollected(disk.Temperature, disk.Collection) { - m.metricsStore.Write("disk", resourceID, "smart_temp", float64(disk.Temperature), now) + // Chart the reading the disk shows: a row without its own carries the + // host's Unraid reading for the disk. + if temperature, collection := unifiedresources.HostSMARTDiskTemperature(host, disk); diskinventory.TemperatureCollected(temperature, collection) { + m.metricsStore.Write("disk", resourceID, "smart_temp", float64(temperature), now) } attrs := disk.Attributes @@ -4127,6 +4131,50 @@ func (m *Monitor) writeHostSMARTMetrics(host models.Host, now time.Time) { m.metricsStore.Write("disk", resourceID, "smart_unsafe_shutdowns", float64(*attrs.UnsafeShutdowns), now) } } + + m.writeHostUnraidDiskTemperatures(host, smartMetricIDs, now) +} + +// writeHostUnraidDiskTemperatures charts the temperature of each disk only the +// host's Unraid inventory reports, such as a member the agent's SMART +// collection skips under --disk-exclude. smartMetricIDs holds the history keys +// of the host's SMART rows. A disk under one of those keys is that row's to +// chart, and the row already carries the Unraid reading when it has none of +// its own. Unraid rows sharing a key are one disk in the registry, which keeps +// the latest of their readings, so the last row with a reading decides that +// key's sample. +func (m *Monitor) writeHostUnraidDiskTemperatures(host models.Host, smartMetricIDs map[string]struct{}, now time.Time) { + if host.Unraid == nil { + return + } + type reading struct { + temperature int + collection *diskinventory.CollectionStatus + } + var keys []string + readings := make(map[string]reading, len(host.Unraid.Disks)) + for _, disk := range host.Unraid.Disks { + resourceID := unifiedresources.HostUnraidDiskMetricID(host, disk) + if resourceID == "" { + continue + } + if _, ok := smartMetricIDs[resourceID]; ok { + continue + } + temperature, collection := unifiedresources.HostUnraidDiskTemperature(host, disk) + if temperature <= 0 { + continue + } + if _, ok := readings[resourceID]; !ok { + keys = append(keys, resourceID) + } + readings[resourceID] = reading{temperature: temperature, collection: collection} + } + for _, resourceID := range keys { + if r := readings[resourceID]; diskinventory.TemperatureCollected(r.temperature, r.collection) { + m.metricsStore.Write("disk", resourceID, "smart_temp", float64(r.temperature), now) + } + } } func normalizeHostDiskDevice(device string) string { diff --git a/internal/monitoring/monitor_host_agents_test.go b/internal/monitoring/monitor_host_agents_test.go index baef427792..f5abd19370 100644 --- a/internal/monitoring/monitor_host_agents_test.go +++ b/internal/monitoring/monitor_host_agents_test.go @@ -3017,6 +3017,71 @@ func TestApplyHostReportPersistsSMARTMetricsForAgentDisks(t *testing.T) { } } +// An Unraid member the agent sends no SMART row for (a --disk-exclude match) +// still shows the temperature its Unraid inventory row reports, so the report +// charts it under the disk's serial. A disk with its own SMART reading keeps +// it, and a spun-down member's leftover reading is not charted. +func TestApplyHostReportChartsUnraidTemperatureOfDiskWithoutSMART(t *testing.T) { + storeCfg := metrics.DefaultConfig(t.TempDir()) + storeCfg.WriteBufferSize = 1 + store, err := metrics.NewStore(storeCfg) + if err != nil { + t.Fatalf("NewStore: %v", err) + } + defer store.Close() + + monitor := &Monitor{ + state: models.NewState(), + alertManager: alerts.NewManager(), + hostTokenBindings: make(map[string]string), + config: &config.Config{}, + rateTracker: NewRateTracker(), + metricsStore: store, + } + t.Cleanup(func() { monitor.alertManager.Stop() }) + + report := agentshost.Report{ + Agent: agentshost.AgentInfo{ID: "agent-tower", Version: "1.0.0", IntervalSeconds: 30}, + Host: agentshost.HostInfo{ID: "machine-tower", Hostname: "tower", MachineID: "machine-tower"}, + Metrics: agentshost.Metrics{ + Memory: agentshost.MemoryMetric{TotalBytes: 1024, UsedBytes: 512, FreeBytes: 512, Usage: 50}, + }, + Sensors: agentshost.Sensors{ + SMART: []agentshost.DiskSMART{ + {Device: "/dev/sda", Model: "IronWolf", Serial: "SERIAL-TOWER-1", Temperature: 41}, + }, + }, + Unraid: &agentshost.UnraidStorage{ArrayStarted: true, Disks: []agentshost.UnraidDisk{ + {Name: "disk1", Device: "sda", Role: "data", Status: "online", Serial: "SERIAL-TOWER-1", Temperature: 40}, + {Name: "disk2", Device: "sdb", Role: "data", Status: "online", Serial: "SERIAL-TOWER-2", Temperature: 36}, + {Name: "disk3", Device: "sdc", Role: "data", Status: "online", Serial: "SERIAL-TOWER-3", Temperature: 34, SpunDown: true}, + }}, + Timestamp: time.Now().UTC(), + } + + if _, err := monitor.ApplyHostReport(report, nil); err != nil { + t.Fatalf("ApplyHostReport: %v", err) + } + store.Flush() + + points := waitForStoredDiskMetric(t, store, "SERIAL-TOWER-2", "smart_temp") + if len(points) != 1 || points[0].Value != 36 { + t.Fatalf("SERIAL-TOWER-2 smart_temp = %+v, want the Unraid reading 36", points) + } + points = waitForStoredDiskMetric(t, store, "SERIAL-TOWER-1", "smart_temp") + if len(points) != 1 || points[0].Value != 41 { + t.Fatalf("SERIAL-TOWER-1 smart_temp = %+v, want its own SMART reading 41", points) + } + now := time.Now().UTC() + points, err = store.Query("disk", "SERIAL-TOWER-3", "smart_temp", now.Add(-time.Hour), now.Add(time.Hour), 60) + if err != nil { + t.Fatalf("Query smart_temp: %v", err) + } + if len(points) != 0 { + t.Fatalf("spun-down SERIAL-TOWER-3 smart_temp = %+v, want none", points) + } +} + func TestApplyHostReportPersistsAgentTemperatureMetric(t *testing.T) { t.Helper() @@ -7124,3 +7189,47 @@ func TestHostAgentSMARTRowForSwappedOutDiskDoesNotLendItsIdentity(t *testing.T) }) } } + +// A Pulse agent linked into a VM fills the guest disk metric when Proxmox has +// no guest filesystems. The next poll must not carry that agent reading +// forward as Proxmox's own last known guest read: with Proxmox outranking the +// agent on guests, the carried copy would freeze over the agent's live disk. +func TestPreviousVMFromViewKeepsLinkedAgentDiskOutOfProxmoxCarry(t *testing.T) { + now := time.Now() + view := func(source unifiedresources.DataSource) *unifiedresources.VMView { + used, total := int64(400), int64(1000) + v := unifiedresources.NewVMView(&unifiedresources.Resource{ + ID: "vm-pve-node1-101", + Type: unifiedresources.ResourceTypeVM, + Name: "app-101", + LastSeen: now, + Proxmox: &unifiedresources.ProxmoxData{ + Instance: "pve", + NodeName: "node1", + VMID: 101, + RuntimeStatus: "running", + DiskStatusReason: "agent-not-running", + }, + // The linked agent's addresses count as recent guest evidence. + Identity: unifiedresources.ResourceIdentity{IPAddresses: []string{"10.0.0.5"}}, + Metrics: &unifiedresources.ResourceMetrics{ + Disk: &unifiedresources.MetricValue{Used: &used, Total: &total, Percent: 40, Source: source}, + }, + }) + return &v + } + carry := func(source unifiedresources.DataSource) (float64, string) { + prev := previousVMFromView(view(source)) + _, _, _, usage, _, reason := stabilizeGuestLowTrustDisk( + &prev, "running", 32<<30, 0, 32<<30, -1, nil, "agent-not-running", false, now, + ) + return usage, reason + } + + if usage, reason := carry(unifiedresources.SourceAgent); usage != -1 || reason != "agent-not-running" { + t.Fatalf("linked agent disk carried as Proxmox's: usage=%v reason=%q", usage, reason) + } + if usage, reason := carry(unifiedresources.SourceProxmox); usage != 40 || reason != "prev-agent-not-running" { + t.Fatalf("Proxmox's own guest disk not carried: usage=%v reason=%q", usage, reason) + } +} diff --git a/internal/monitoring/monitor_previous_state.go b/internal/monitoring/monitor_previous_state.go index a186647fab..6f33e6e8f9 100644 --- a/internal/monitoring/monitor_previous_state.go +++ b/internal/monitoring/monitor_previous_state.go @@ -124,25 +124,32 @@ func previousVMFromView(vm *unifiedresources.VMView) models.VM { instance := vm.Instance() node := vm.Node() vmid := vm.VMID() - return models.VM{ - ID: makeGuestID(instance, node, vmid), - Instance: instance, - Node: node, - VMID: vmid, - Name: vm.Name(), - Type: "qemu", - Status: vm.RuntimeStatus(), - IPAddresses: vm.IPAddresses(), - OSName: vm.OSName(), - OSVersion: vm.OSVersion(), - AgentVersion: vm.AgentVersion(), - Lock: vm.Lock(), - Disk: models.Disk{ + // Only Proxmox's own guest disk reading may be carried forward as last + // known. A linked agent's disk that filled in stays the agent's, or the + // carry would relabel it as Proxmox's and freeze it over the agent's. + var disk models.Disk + if !vm.DiskFromLinkedAgent() { + disk = models.Disk{ Used: vm.DiskUsed(), Total: vm.DiskTotal(), Free: max(0, vm.DiskTotal()-vm.DiskUsed()), Usage: vm.DiskPercent(), - }, + } + } + return models.VM{ + ID: makeGuestID(instance, node, vmid), + Instance: instance, + Node: node, + VMID: vmid, + Name: vm.Name(), + Type: "qemu", + Status: vm.RuntimeStatus(), + IPAddresses: vm.IPAddresses(), + OSName: vm.OSName(), + OSVersion: vm.OSVersion(), + AgentVersion: vm.AgentVersion(), + Lock: vm.Lock(), + Disk: disk, NetworkInterfaces: guestNetworkInterfacesFromReadStateView(vm.NetworkInterfaces()), Disks: guestDisksFromReadStateView(vm.Disks()), DiskStatusReason: vm.DiskStatusReason(), diff --git a/internal/unifiedresources/adapters.go b/internal/unifiedresources/adapters.go index 167af4c6b1..e5220c3800 100644 --- a/internal/unifiedresources/adapters.go +++ b/internal/unifiedresources/adapters.go @@ -806,10 +806,7 @@ func resourceFromHostUnraidPhysicalDisk(host models.Host, disk models.HostUnraid health := unraidPhysicalDiskHealth(disk) assessment := assessUnraidPhysicalDisk(disk) sizeBytes := unraidDiskSizeBytes(host, disk) - var collection *diskinventory.CollectionStatus - if status := unraidDiskTemperatureStatus(host, disk); status.State != "" { - collection = &diskinventory.CollectionStatus{Temperature: status} - } + temperature, collection := HostUnraidDiskTemperature(host, disk) resource := Resource{ Type: ResourceTypePhysicalDisk, Name: name, @@ -824,7 +821,7 @@ func resourceFromHostUnraidPhysicalDisk(host models.Host, disk models.HostUnraid SizeBytes: sizeBytes, Health: health, Wearout: -1, - Temperature: disk.Temperature, + Temperature: temperature, Used: unraidDiskMountPath(disk), StorageRole: unraidDiskRole(&disk), StorageGroup: unraidDiskGroup(&disk), @@ -869,6 +866,46 @@ func unraidDiskTemperatureStatus(host models.Host, disk models.HostUnraidDisk) d } } +// HostUnraidDiskTemperature returns the temperature the disk resource built +// from a host's Unraid inventory row shows, with its collection state. The +// agent disk history writer charts this reading. +func HostUnraidDiskTemperature(host models.Host, disk models.HostUnraidDisk) (int, *diskinventory.CollectionStatus) { + status := unraidDiskTemperatureStatus(host, disk) + if status.State == "" { + return disk.Temperature, nil + } + return disk.Temperature, &diskinventory.CollectionStatus{Temperature: status} +} + +// HostSMARTDiskTemperature returns the temperature the disk resource built +// from a host agent's SMART row shows, with its collection state. The agent +// disk history writer charts this reading. +func HostSMARTDiskTemperature(host models.Host, disk models.HostDiskSMART) (int, *diskinventory.CollectionStatus) { + return hostSMARTDiskTemperature(host, disk, matchUnraidDisk(host.Unraid, disk)) +} + +// hostSMARTDiskTemperature returns the temperature the disk resource built +// from a host agent's SMART row shows, with the row's collection state: the +// row's own reading, or, when the row has none, the one the host's Unraid +// inventory reports for the disk. Unraid reads its members' temperatures with +// its own per-disk SMART settings, so it can have one when the agent's probe +// returned none. The reading is taken only from an Unraid row that describes +// the row's disk (unraidDiskDescribesSMARTRow). +func hostSMARTDiskTemperature(host models.Host, disk models.HostDiskSMART, unraidDisk *models.HostUnraidDisk) (int, *diskinventory.CollectionStatus) { + collection := diskinventory.CloneStatus(disk.Collection) + if disk.Temperature > 0 || !unraidDiskDescribesSMARTRow(host, disk, unraidDisk) { + return disk.Temperature, collection + } + // The reading is now the Unraid inventory's, and so is its state. + if status := unraidDiskTemperatureStatus(host, *unraidDisk); status.State != "" { + if collection == nil { + collection = &diskinventory.CollectionStatus{} + } + collection.Temperature = status + } + return unraidDisk.Temperature, collection +} + func resourceFromHostSMARTDisk(host models.Host, disk models.HostDiskSMART) (Resource, ResourceIdentity) { name := strings.TrimSpace(disk.Model) if name == "" { @@ -907,8 +944,7 @@ func resourceFromHostSMARTDisk(host models.Host, disk models.HostDiskSMART) (Res model := strings.TrimSpace(disk.Model) serial := hostSMARTDiskSerial(host, disk, unraidDisk) diskType := strings.TrimSpace(disk.Type) - temperature := disk.Temperature - collection := diskinventory.CloneStatus(disk.Collection) + temperature, collection := hostSMARTDiskTemperature(host, disk, unraidDisk) health := strings.TrimSpace(disk.Health) if unraidDisk != nil { if model == "" { @@ -920,16 +956,6 @@ func resourceFromHostSMARTDisk(host models.Host, disk models.HostDiskSMART) (Res if sizeBytes <= 0 { sizeBytes = unraidDiskSizeBytes(host, *unraidDisk) } - if temperature <= 0 { - temperature = unraidDisk.Temperature - // The reading is now the Unraid inventory's, and so is its state. - if status := unraidDiskTemperatureStatus(host, *unraidDisk); status.State != "" { - if collection == nil { - collection = &diskinventory.CollectionStatus{} - } - collection.Temperature = status - } - } if health == "" || strings.EqualFold(health, "UNKNOWN") { health = unraidPhysicalDiskHealth(*unraidDisk) } @@ -1219,21 +1245,30 @@ func maxInt64() int64 { return int64(^uint64(0) >> 1) } +// matchUnraidDisk returns the host's Unraid row for a SMART row: the one +// carrying the row's usable serial, else the one at its device path. A serial +// names the drive and a path only where the kernel put it, so a serial match +// anywhere in the inventory outranks a path match. A placeholder serial, which +// several bridges report alike, is no match at all. func matchUnraidDisk(unraid *models.HostUnraidStorage, disk models.HostDiskSMART) *models.HostUnraidDisk { if unraid == nil || len(unraid.Disks) == 0 { return nil } - normalizedDevice := strings.ToLower(normalizePhysicalDiskDeviceToken(disk.Device)) - normalizedSerial := strings.TrimSpace(strings.ToLower(disk.Serial)) - for i := range unraid.Disks { - candidate := &unraid.Disks[i] - if normalizedSerial != "" && strings.EqualFold(strings.TrimSpace(candidate.Serial), normalizedSerial) { - return candidate + if serial := strings.TrimSpace(disk.Serial); diskinventory.IsUsableHardwareID(serial) { + for i := range unraid.Disks { + if strings.EqualFold(strings.TrimSpace(unraid.Disks[i].Serial), serial) { + return &unraid.Disks[i] + } } - candidateDevice := strings.ToLower(normalizePhysicalDiskDeviceToken(candidate.Device)) - if normalizedDevice != "" && candidateDevice != "" && candidateDevice == normalizedDevice { - return candidate + } + normalizedDevice := strings.ToLower(normalizePhysicalDiskDeviceToken(disk.Device)) + if normalizedDevice == "" { + return nil + } + for i := range unraid.Disks { + if strings.ToLower(normalizePhysicalDiskDeviceToken(unraid.Disks[i].Device)) == normalizedDevice { + return &unraid.Disks[i] } } return nil diff --git a/internal/unifiedresources/canonical_id_pins.go b/internal/unifiedresources/canonical_id_pins.go index ece08ab9b7..c48aa0cee3 100644 --- a/internal/unifiedresources/canonical_id_pins.go +++ b/internal/unifiedresources/canonical_id_pins.go @@ -337,8 +337,9 @@ func (index *identityPinIndex) successionsFor(pin ResourceIdentityPin) []Canonic // the new canonical ID before the pin write, so operator intent // (never-auto-remediate, maintenance windows) and action-audit history // survive the era change. Successions are skipped while the old canonical ID -// still belongs to a live resource: a genuinely short-named host must not -// have its rows stolen by an FQDN sibling. Change-journal rows are never +// still belongs to a live resource, or to one a manual link folded into +// another: a genuinely short-named host must not have its rows stolen by an +// FQDN sibling. Change-journal rows are never // rewritten; EraIDs merges those at read time. func (rr *ResourceRegistry) PersistIdentityPins() { if rr.store == nil { @@ -358,7 +359,11 @@ func (rr *ResourceRegistry) PersistIdentityPins() { } pins = append(pins, pin) for _, succession := range rr.identityPins.successionsFor(pin) { - if _, live := rr.resources[succession.OldCanonicalID]; live { + // A primary that absorbed a linked resource also carries its + // machine keys. That resource is still observed, and succeeding + // it would re-key the link onto the primary itself, splitting + // the pair again on the next rebuild. + if rr.canonicalIDObservedLocked(succession.OldCanonicalID) { continue } successions = append(successions, succession) diff --git a/internal/unifiedresources/canonical_ids_types_test.go b/internal/unifiedresources/canonical_ids_types_test.go index c8ed1bedb5..293d664ca8 100644 --- a/internal/unifiedresources/canonical_ids_types_test.go +++ b/internal/unifiedresources/canonical_ids_types_test.go @@ -503,6 +503,50 @@ func TestHostUnraidDiskSourceIDNormalizesDeviceAndPrefersSerial(t *testing.T) { } } +// An Unraid inventory row's history key is the one the metrics target of the +// disk resource built from it reads: its usable serial, unscoped like every +// physical-disk history key, or else its host/device source ID. A row without +// a device is not ingested as a disk and has no key. +func TestHostUnraidDiskMetricIDMatchesItsDiskMetricsTarget(t *testing.T) { + host := models.Host{ID: "host-tower", Hostname: "tower"} + tests := []struct { + name string + disk models.HostUnraidDisk + want string + }{ + {"serial", models.HostUnraidDisk{Device: "/dev/sdb", Serial: "SERIAL-DATA"}, "SERIAL-DATA"}, + {"placeholder serial", models.HostUnraidDisk{Device: "sdc", Serial: "N/A"}, "host-tower:sdc"}, + {"no serial", models.HostUnraidDisk{Device: "sdd [sat]"}, "host-tower:sdd"}, + {"no device", models.HostUnraidDisk{Name: "disk9", Serial: "SERIAL-MISSING"}, ""}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + got := HostUnraidDiskMetricID(host, tt.disk) + if got != tt.want { + t.Fatalf("HostUnraidDiskMetricID(%+v) = %q, want %q", tt.disk, got, tt.want) + } + if tt.want == "" { + return + } + disk := tt.disk + disk.Name, disk.Role, disk.Status = "disk1", "data", "online" + registry := NewRegistry(nil) + registry.IngestSnapshot(models.StateSnapshot{Hosts: []models.Host{{ + ID: host.ID, Hostname: host.Hostname, Status: "online", + Unraid: &models.HostUnraidStorage{ArrayStarted: true, Disks: []models.HostUnraidDisk{disk}}, + }}}) + disks := registry.ListByType(ResourceTypePhysicalDisk) + if len(disks) != 1 { + t.Fatalf("disks = %d, want 1", len(disks)) + } + target := registry.MetricsTarget(disks[0].ID) + if target == nil || target.ResourceType != "disk" || target.ResourceID != got { + t.Fatalf("metrics target = %+v, want disk %q", target, got) + } + }) + } +} + // An agent disk's source ID carries its host, because a usable serial or WWN // names the drive but not the machine; its metrics key is the hardware ID // alone. Without hardware identity the two keep one host/device/topology key. @@ -1007,3 +1051,21 @@ func TestResourceIncidentNativeSeverityJSONContract(t *testing.T) { } } } + +// Browsers merge agent facets field by field and keep an omitted field, so a +// resumed agent must send stale=false or it stays marked as stopped reporting. +func TestAgentDataAlwaysSendsStale(t *testing.T) { + for _, stale := range []bool{false, true} { + raw, err := json.Marshal(AgentData{AgentID: "agent-1", Stale: stale}) + if err != nil { + t.Fatalf("marshal: %v", err) + } + var decoded map[string]any + if err := json.Unmarshal(raw, &decoded); err != nil { + t.Fatalf("unmarshal: %v", err) + } + if got, ok := decoded["stale"]; !ok || got != stale { + t.Fatalf("stale = %v (present %v), want explicit %v in %s", got, ok, stale, raw) + } + } +} diff --git a/internal/unifiedresources/host_disk_temperatures.go b/internal/unifiedresources/host_disk_temperatures.go new file mode 100644 index 0000000000..12c34d802a --- /dev/null +++ b/internal/unifiedresources/host_disk_temperatures.go @@ -0,0 +1,97 @@ +package unifiedresources + +import ( + "github.com/rcourtman/pulse-go-rewrite/internal/models" + "github.com/rcourtman/pulse-go-rewrite/pkg/diskinventory" + "github.com/rcourtman/pulse-go-rewrite/pkg/fsfilters" +) + +// HostDiskTemperatureReading is the temperature one physical disk of a host +// agent shows, taken from the disk resource the registry builds for it. +type HostDiskTemperatureReading struct { + // Device names the disk on the host: a SMART row's device label, or the + // kernel block device token of an Unraid inventory row. + Device string + // MetricID is the disk's history key (HostSMARTDiskMetricID or + // HostUnraidDiskMetricID). + MetricID string + Model string + DiskType string + Temperature int + Collection *diskinventory.CollectionStatus + // Standby reports a SMART row the agent marked as in standby. + Standby bool + // UnraidOnly reports a disk built from the host's Unraid inventory row + // alone, because no SMART row of the host carries its disk key. + UnraidOnly bool +} + +// Collected reports whether the reading is a temperature collected now: not +// a disk in standby, and not a reading its collection state marks as kept +// from earlier (diskinventory.TemperatureCollected). +func (r HostDiskTemperatureReading) Collected() bool { + return !r.Standby && diskinventory.TemperatureCollected(r.Temperature, r.Collection) +} + +// HostDiskTemperatureReadings returns the temperature each physical disk of a +// host agent shows, in report order: one reading per SMART row the registry +// ingests, then one per Unraid inventory row whose disk key no SMART row of +// the host carries. The registry merges a SMART row and an Unraid row with +// one key into one disk whatever device labels they use, as a controller +// member ("0 [megaraid,0]") and its Unraid device ("sda") do. A SMART row +// without a reading of its own shows the one its Unraid row reports +// (HostSMARTDiskTemperature), and a disk only the Unraid inventory lists, such +// as a --disk-exclude match, shows that row's (HostUnraidDiskTemperature). +// When smartctl and Unraid report different usable serials for one device the +// registry shows both disks, so both are listed. The readings come from the +// adapters the registry builds the disks with, so a consumer judging them +// judges the temperature, collection state and disk type each row's disk +// shows. +func HostDiskTemperatureReadings(host models.Host) []HostDiskTemperatureReading { + readings := make([]HostDiskTemperatureReading, 0, len(host.Sensors.SMART)) + smartKeys := make(map[string]struct{}, len(host.Sensors.SMART)) + for _, disk := range host.Sensors.SMART { + // The registry ingests no disk for a virtual block device. + if fsfilters.IsVirtualBlockDevice(disk.Device) { + continue + } + key := HostSMARTDiskMetricID(host, disk) + if key != "" { + smartKeys[key] = struct{}{} + } + resource, _ := resourceFromHostSMARTDisk(host, disk) + reading := hostDiskTemperatureReading(disk.Device, key, resource) + reading.Standby = disk.Standby + readings = append(readings, reading) + } + if host.Unraid == nil { + return readings + } + for _, disk := range host.Unraid.Disks { + // HostUnraidDiskMetricID is empty for a row without a device, which + // the registry ingests no disk for. + key := HostUnraidDiskMetricID(host, disk) + if key == "" { + continue + } + if _, ok := smartKeys[key]; ok { + continue + } + resource, _ := resourceFromHostUnraidPhysicalDisk(host, disk) + reading := hostDiskTemperatureReading(normalizePhysicalDiskDeviceToken(disk.Device), key, resource) + reading.UnraidOnly = true + readings = append(readings, reading) + } + return readings +} + +func hostDiskTemperatureReading(device, metricID string, resource Resource) HostDiskTemperatureReading { + reading := HostDiskTemperatureReading{Device: device, MetricID: metricID} + if disk := resource.PhysicalDisk; disk != nil { + reading.Model = disk.Model + reading.DiskType = disk.DiskType + reading.Temperature = disk.Temperature + reading.Collection = diskinventory.CloneStatus(disk.Collection) + } + return reading +} diff --git a/internal/unifiedresources/monitor_adapter.go b/internal/unifiedresources/monitor_adapter.go index 6abf5dfdbf..910a61f5a3 100644 --- a/internal/unifiedresources/monitor_adapter.go +++ b/internal/unifiedresources/monitor_adapter.go @@ -135,7 +135,7 @@ func readStateWithRecords(readState ReadState, source DataSource, records []Inge cloned := NewRegistry(registry.store) thresholds := adapter.currentStaleThresholds() cloned.IngestResourcesWithStaleThresholds(registry.List(), thresholds) - cloned.ingestRecords(source, records, onlyMissing) + cloned.ingestRecords(source, records, onlyMissing, thresholds) overlay := NewMonitorAdapterWithStaleThresholds(cloned, thresholds) if keyed { adapter.overlays.store(registry, rebuiltAt, key, now, overlay) @@ -450,7 +450,7 @@ func (a *MonitorAdapter) replaceRegistryLocked(snapshot models.StateSnapshot, re if len(records) == 0 || strings.TrimSpace(string(source)) == "" { continue } - rebuilt.IngestRecords(source, records) + rebuilt.IngestRecordsWithStaleThresholds(source, records, staleThresholds) } // IngestSnapshot runs its stale pass before the record sources above are // ingested, so record-sourced resources would otherwise keep their @@ -665,7 +665,7 @@ func (a *MonitorAdapter) PopulateSupplementalRecords(source DataSource, records return } before := registry.List() - registry.IngestRecords(source, records) + registry.IngestRecordsWithStaleThresholds(source, records, a.currentStaleThresholds()) rebuiltAt := time.Now().UTC() recordRegistryChanges(registry.store, before, registry.List(), rebuiltAt, nil, SourcePlatformEvent, changeSourceAdapterForDataSource(source)) registry.PersistIdentityPins() diff --git a/internal/unifiedresources/monitor_adapter_read_state_test.go b/internal/unifiedresources/monitor_adapter_read_state_test.go index 7eefcb5b99..55a8298d70 100644 --- a/internal/unifiedresources/monitor_adapter_read_state_test.go +++ b/internal/unifiedresources/monitor_adapter_read_state_test.go @@ -400,6 +400,234 @@ func TestReadStateWithRecordsPreservesConfiguredStaleThresholds(t *testing.T) { } } +// A link whose side arrives as a record joins during record ingest, so it +// must judge freshness by the adapter's thresholds as a snapshot-time link +// does. Under a five-minute vSphere threshold a VM polled three minutes ago +// is current, and its own memory reading outranks its in-guest agent's. +func TestMonitorAdapterJoinsLinkedRecordsWithConfiguredStaleThresholds(t *testing.T) { + now := time.Now().UTC() + snapshot := models.StateSnapshot{ + Hosts: []models.Host{{ + ID: "host-app-guest", + Hostname: "app-guest", + MachineID: "machine-app-guest", + Status: "online", + LastSeen: now, + Memory: models.Memory{Total: 8 << 30, Used: 6 << 30, Free: 2 << 30, Usage: 75}, + }}, + LastUpdate: now, + } + vmRecords := []IngestRecord{{ + SourceID: "vc-1:vm:vm-42", + Resource: Resource{ + Type: ResourceTypeVM, + Technology: "vmware", + Name: "app-guest", + Status: StatusOnline, + LastSeen: now.Add(-3 * time.Minute), + Metrics: &ResourceMetrics{Memory: &MetricValue{Percent: 40, Source: SourceVMware}}, + VMware: &VMwareData{ConnectionID: "vc-1", ManagedObjectID: "vm-42", EntityType: "vm"}, + }, + }} + thresholds := map[DataSource]time.Duration{SourceVMware: 5 * time.Minute} + + unlinked := NewMonitorAdapter(NewRegistry(nil)) + unlinked.PopulateSnapshotAndSupplemental(snapshot, map[DataSource][]IngestRecord{SourceVMware: vmRecords}) + vms, hosts := unlinked.VMs(), unlinked.Hosts() + if len(vms) != 1 || len(hosts) != 1 { + t.Fatalf("unlinked estate = %d VMs, %d agents, want one of each", len(vms), len(hosts)) + } + link := ResourceLink{ResourceA: vms[0].ID(), ResourceB: hosts[0].ID(), PrimaryID: hosts[0].ID()} + + for _, path := range []struct { + name string + read func(adapter *MonitorAdapter) ReadState + }{ + {"rebuild", func(adapter *MonitorAdapter) ReadState { + adapter.PopulateSnapshotAndSupplemental(snapshot, map[DataSource][]IngestRecord{SourceVMware: vmRecords}) + return adapter + }}, + {"supplemental", func(adapter *MonitorAdapter) ReadState { + adapter.PopulateFromSnapshot(snapshot) + adapter.PopulateSupplementalRecords(SourceVMware, vmRecords) + return adapter + }}, + {"overlay", func(adapter *MonitorAdapter) ReadState { + adapter.PopulateFromSnapshot(snapshot) + return ReadStateWithRecords(adapter, SourceVMware, vmRecords) + }}, + } { + t.Run(path.name, func(t *testing.T) { + store := NewMemoryStore() + if err := store.AddLink(link); err != nil { + t.Fatalf("add link: %v", err) + } + readState := path.read(NewMonitorAdapterWithStaleThresholds(NewRegistry(store), thresholds)) + + if hosts := readState.Hosts(); len(hosts) != 0 { + t.Fatalf("linked agent still listed standalone: %s", hosts[0].ID()) + } + vms := readState.VMs() + if len(vms) != 1 || vms[0].ID() != link.ResourceA { + t.Fatalf("VMs = %d, want the linked VM %s", len(vms), link.ResourceA) + } + if vms[0].r.Metrics == nil || vms[0].r.Metrics.Memory == nil { + t.Fatal("linked VM lost its memory reading") + } + if memory := vms[0].r.Metrics.Memory; memory.Source != SourceVMware || memory.Percent != 40 { + t.Fatalf("linked VM memory = %.0f%% from %s, want vSphere's current 40%%", memory.Percent, memory.Source) + } + }) + } +} + +// Every source merge into an existing row judges metric freshness by the +// adapter's configured thresholds, as the rebuild's links and stale pass do. +// A linked vSphere VM first reports no memory, so its in-guest agent supplies +// it; a refresh then brings vSphere memory observed three minutes ago, current +// under a five-minute threshold. The live supplemental refresh and the +// read-state overlay merge that record into the existing row, and must show +// vSphere's memory as a rebuild from the same records does. A Proxmox node +// polled every two minutes is likewise current at ninety seconds, so the +// rebuild keeps its readings over those of its auto-linked agent, silent for +// seventy-five seconds. +func TestMonitorAdapterSourceMergesUseConfiguredStaleThresholds(t *testing.T) { + now := time.Now().UTC() + thresholds := map[DataSource]time.Duration{SourceProxmox: 4 * time.Minute, SourceVMware: 5 * time.Minute} + + t.Run("linked vSphere VM refresh", func(t *testing.T) { + snapshot := models.StateSnapshot{ + Hosts: []models.Host{{ + ID: "host-app-guest", + Hostname: "app-guest", + MachineID: "machine-app-guest", + Status: "online", + LastSeen: now, + Memory: models.Memory{Total: 8 << 30, Used: 6 << 30, Free: 2 << 30, Usage: 75}, + }}, + LastUpdate: now, + } + vmRecord := func(lastSeen time.Time, metrics *ResourceMetrics) []IngestRecord { + return []IngestRecord{{ + SourceID: "vc-1:vm:vm-42", + Resource: Resource{ + Type: ResourceTypeVM, + Technology: "vmware", + Name: "app-guest", + Status: StatusOnline, + LastSeen: lastSeen, + Metrics: metrics, + VMware: &VMwareData{ConnectionID: "vc-1", ManagedObjectID: "vm-42", EntityType: "vm"}, + }, + }} + } + initial := vmRecord(now.Add(-4*time.Minute), &ResourceMetrics{CPU: &MetricValue{Percent: 10, Source: SourceVMware}}) + refreshedAt := now.Add(-3 * time.Minute) + refreshed := vmRecord(refreshedAt, &ResourceMetrics{ + CPU: &MetricValue{Percent: 10, Source: SourceVMware}, + Memory: &MetricValue{Percent: 40, Source: SourceVMware}, + }) + + unlinked := NewMonitorAdapter(NewRegistry(nil)) + unlinked.PopulateSnapshotAndSupplemental(snapshot, map[DataSource][]IngestRecord{SourceVMware: initial}) + vms, hosts := unlinked.VMs(), unlinked.Hosts() + if len(vms) != 1 || len(hosts) != 1 { + t.Fatalf("unlinked estate = %d VMs, %d agents, want one of each", len(vms), len(hosts)) + } + link := ResourceLink{ResourceA: vms[0].ID(), ResourceB: hosts[0].ID(), PrimaryID: hosts[0].ID()} + + for _, path := range []struct { + name string + read func(adapter *MonitorAdapter) ReadState + }{ + {"rebuild", func(adapter *MonitorAdapter) ReadState { + adapter.PopulateSnapshotAndSupplemental(snapshot, map[DataSource][]IngestRecord{SourceVMware: refreshed}) + return adapter + }}, + {"supplemental", func(adapter *MonitorAdapter) ReadState { + adapter.PopulateSnapshotAndSupplemental(snapshot, map[DataSource][]IngestRecord{SourceVMware: initial}) + adapter.PopulateSupplementalRecords(SourceVMware, refreshed) + return adapter + }}, + {"overlay", func(adapter *MonitorAdapter) ReadState { + adapter.PopulateSnapshotAndSupplemental(snapshot, map[DataSource][]IngestRecord{SourceVMware: initial}) + return ReadStateWithRecords(adapter, SourceVMware, refreshed) + }}, + } { + t.Run(path.name, func(t *testing.T) { + store := NewMemoryStore() + if err := store.AddLink(link); err != nil { + t.Fatalf("add link: %v", err) + } + readState := path.read(NewMonitorAdapterWithStaleThresholds(NewRegistry(store), thresholds)) + + vms := readState.VMs() + if len(vms) != 1 || vms[0].ID() != link.ResourceA || len(readState.Hosts()) != 0 { + t.Fatalf("estate = %d VMs, %d standalone agents, want only the linked VM %s", len(vms), len(readState.Hosts()), link.ResourceA) + } + if got := vms[0].r.SourceStatus[SourceVMware]; got.Status != "online" || !got.LastSeen.Equal(refreshedAt) { + t.Fatalf("vSphere sighting = %+v, want the refresh, current under the configured threshold", got) + } + if vms[0].r.Metrics == nil || vms[0].r.Metrics.Memory == nil { + t.Fatal("linked VM lost its memory reading") + } + if memory := vms[0].r.Metrics.Memory; memory.Source != SourceVMware || memory.Percent != 40 { + t.Fatalf("linked VM memory = %.0f%% from %s, want vSphere's current 40%%", memory.Percent, memory.Source) + } + }) + } + }) + + t.Run("node with auto-linked agent", func(t *testing.T) { + adapter := NewMonitorAdapterWithStaleThresholds(NewRegistry(nil), thresholds) + adapter.PopulateFromSnapshot(models.StateSnapshot{ + Nodes: []models.Node{{ + ID: "homelab-pve1", + Name: "pve1", + Instance: "homelab", + Status: "online", + LinkedAgentID: "host-pve1", + CPU: 0.12, + Memory: models.Memory{Total: 64 << 30, Used: 16 << 30, Free: 48 << 30, Usage: 25}, + LastSeen: now.Add(-90 * time.Second), + }}, + Hosts: []models.Host{{ + ID: "host-pve1", + MachineID: "machine-pve1", + Hostname: "pve1", + LinkedNodeID: "homelab-pve1", + Status: "online", + CPUUsage: 80, + Memory: models.Memory{Total: 64 << 30, Used: 48 << 30, Free: 16 << 30, Usage: 75}, + LastSeen: now.Add(-75 * time.Second), + IntervalSeconds: 30, + }}, + LastUpdate: now, + }) + + nodes := adapter.Nodes() + if len(nodes) != 1 { + t.Fatalf("nodes = %d, want the node merged with its agent", len(nodes)) + } + node := nodes[0].r + if poll := node.SourceStatus[SourceProxmox]; poll.Status != "online" || !poll.LastSeen.Equal(now.Add(-90*time.Second)) { + t.Fatalf("Proxmox sighting = %+v, want the current poll from ninety seconds ago", poll) + } + if got := node.SourceStatus[SourceAgent].Status; got != "stale" { + t.Fatalf("agent sighting = %q, want stale", got) + } + if node.Metrics == nil || node.Metrics.CPU == nil || node.Metrics.Memory == nil { + t.Fatalf("merged node lost its readings: %+v", node.Metrics) + } + if cpu := node.Metrics.CPU; cpu.Source != SourceProxmox || cpu.Percent != 12 { + t.Fatalf("node CPU = %.0f%% from %s, want the current poll's 12%%", cpu.Percent, cpu.Source) + } + if memory := node.Metrics.Memory; memory.Source != SourceProxmox || memory.Percent != 25 { + t.Fatalf("node memory = %.0f%% from %s, want the current poll's 25%%", memory.Percent, memory.Source) + } + }) +} + func TestHostContinuityCannotOverwriteCurrentResource(t *testing.T) { now := time.Now().UTC() for _, mode := range []string{"machine-identity", "source-identity", "future-timestamp", "provider-only"} { diff --git a/internal/unifiedresources/physical_disk_ids.go b/internal/unifiedresources/physical_disk_ids.go index 4ba7cefa98..9d210e9e8c 100644 --- a/internal/unifiedresources/physical_disk_ids.go +++ b/internal/unifiedresources/physical_disk_ids.go @@ -61,17 +61,37 @@ func hostSMARTDiskKey(host models.Host, disk models.HostDiskSMART, serial, wwn s // row's own, or, when the row reports none, the one the host's Unraid // inventory reports for the disk. smartctl reports no serial for a disk in // standby, and the Unraid row still names it. A row without a serial matches -// its Unraid row by device path alone, so it takes no serial from a path -// several of the host's rows share: controller members behind one kernel block -// device, which the Unraid row describes as a whole. +// its Unraid row by device path alone, so it takes the serial only where that +// row describes its disk (unraidDiskDescribesSMARTRow). func hostSMARTDiskSerial(host models.Host, disk models.HostDiskSMART, unraidDisk *models.HostUnraidDisk) string { serial := strings.TrimSpace(disk.Serial) - if serial != "" || unraidDisk == nil || hostSMARTDevicePathShared(host, disk) { + if serial != "" || !unraidDiskDescribesSMARTRow(host, disk, unraidDisk) { return serial } return strings.TrimSpace(unraidDisk.Serial) } +// unraidDiskDescribesSMARTRow reports whether the Unraid row matched to a SMART +// row (matchUnraidDisk) describes that row's disk. An Unraid row describes a +// kernel block device as a whole, so it describes the row when it carries the +// row's usable serial, or when it names the row's device path and the row is +// that whole device: not a controller member behind it, even the only one +// reported, nor one of several rows sharing the path. +func unraidDiskDescribesSMARTRow(host models.Host, disk models.HostDiskSMART, unraidDisk *models.HostUnraidDisk) bool { + if unraidDisk == nil { + return false + } + if serial := strings.TrimSpace(disk.Serial); diskinventory.IsUsableHardwareID(serial) && + strings.EqualFold(serial, strings.TrimSpace(unraidDisk.Serial)) { + return true + } + device := strings.ToLower(normalizePhysicalDiskDeviceToken(disk.Device)) + if device == "" || device != strings.ToLower(normalizePhysicalDiskDeviceToken(unraidDisk.Device)) { + return false + } + return !diskinventory.IsControllerMemberTarget(disk.Target) && !hostSMARTDevicePathShared(host, disk) +} + // hostSMARTDevicePathShared reports whether more than one of the host's SMART // rows, the row itself among them, names the row's device path. func hostSMARTDevicePathShared(host models.Host, disk models.HostDiskSMART) bool { @@ -130,6 +150,17 @@ func HostUnraidDiskSourceID(host models.Host, disk models.HostUnraidDisk) string return "" } +// HostUnraidDiskMetricID returns the history key of the disk resource built +// from a host's Unraid inventory row: the key its metrics target reads, the +// row's usable serial or else its source ID. It returns "" for a row the +// registry does not ingest as a disk, one without a device. +func HostUnraidDiskMetricID(host models.Host, disk models.HostUnraidDisk) string { + if strings.TrimSpace(disk.Device) == "" { + return "" + } + return PreferredPhysicalDiskMetricID(disk.Serial, "", HostUnraidDiskSourceID(host, disk)) +} + const hostPhysicalDiskSourceIDMarker = "/physical-disk:" func hostPhysicalDiskSourceID(hostID, hardwareID string) string { diff --git a/internal/unifiedresources/registry.go b/internal/unifiedresources/registry.go index 48f2148fcd..ac5891cacd 100644 --- a/internal/unifiedresources/registry.go +++ b/internal/unifiedresources/registry.go @@ -95,6 +95,10 @@ type ResourceRegistry struct { exclusions map[string]struct{} identityPins *identityPinIndex pbsBackups []models.PBSBackup + // linkMergedIDs holds the resources a manual link folded into their + // primary. They are still observed, so identity succession treats them + // as live. + linkMergedIDs map[string]struct{} // supersededIndex maps record-declared retired canonical IDs to the live // resource that superseded them, so references persisted under a retired // ID (availability links, API reads) keep resolving. An empty value marks @@ -148,6 +152,13 @@ type ResourceRegistry struct { // what the full scan would see; scores stay live through the shared // pointers. agentNodeScanIndex map[string][]agentNodeCandidate + + // ingestStaleThresholds holds the caller-owned freshness thresholds of + // the snapshot or record ingest in progress, so each source merge into an + // existing row judges metric freshness as that ingest's manual links and + // stale pass do. nil outside an ingest and for callers without + // thresholds, where all three use the defaults. + ingestStaleThresholds map[DataSource]time.Duration } // agentNodeCandidate pairs a resources-map key with its entry so the indexed @@ -243,6 +254,10 @@ func (rr *ResourceRegistry) IngestSnapshotWithStaleThresholds(snapshot models.St } func (rr *ResourceRegistry) ingestSnapshot(snapshot models.StateSnapshot, thresholds map[DataSource]time.Duration) { + rr.mu.Lock() + rr.ingestStaleThresholds = thresholds + rr.mu.Unlock() + hostByID := make(map[string]*models.Host, len(snapshot.Hosts)) for i := range snapshot.Hosts { host := snapshot.Hosts[i] @@ -606,15 +621,26 @@ func (rr *ResourceRegistry) ingestSnapshot(snapshot models.StateSnapshot, thresh rr.refreshCanonicalIdentitiesLocked() rr.markStaleLocked(time.Now().UTC(), thresholds) rr.invalidateViewsLocked() + rr.ingestStaleThresholds = nil rr.mu.Unlock() } // IngestRecords ingests normalized records for a single source. func (rr *ResourceRegistry) IngestRecords(source DataSource, records []IngestRecord) { - rr.ingestRecords(source, records, false) + rr.ingestRecords(source, records, false, nil) } -func (rr *ResourceRegistry) ingestRecords(source DataSource, records []IngestRecord, onlyMissing bool) { +// IngestRecordsWithStaleThresholds ingests normalized records for a single +// source and merges operator-linked resources with caller-owned thresholds. +func (rr *ResourceRegistry) IngestRecordsWithStaleThresholds(source DataSource, records []IngestRecord, thresholds map[DataSource]time.Duration) { + rr.ingestRecords(source, records, false, thresholds) +} + +func (rr *ResourceRegistry) ingestRecords(source DataSource, records []IngestRecord, onlyMissing bool, thresholds map[DataSource]time.Duration) { + rr.mu.Lock() + rr.ingestStaleThresholds = thresholds + rr.mu.Unlock() + var successions []CanonicalIDSuccession supersededSeen := make(map[string]struct{}) for _, record := range records { @@ -649,6 +675,16 @@ func (rr *ResourceRegistry) ingestRecords(source DataSource, records []IngestRec rr.applyRecordSuccessions(successions) rr.mu.Lock() + // Either side of an operator link can arrive as a record: a vSphere or + // TrueNAS VM with a Pulse agent inside, or an agent on a TrueNAS host. + // Snapshot and resource ingest apply the links, so record ingest must too, + // or the monitor's rebuilt registry keeps both rows while REST, which + // seeds from that registry, shows them merged. Continuity records only + // fill absent machines: a saved enrollment joined to its linked guest + // would lend the guest its offline verdict and old agent payload. + if !onlyMissing { + rr.applyManualLinks(thresholds) + } rr.refreshStorageConsumersLocked() rr.refreshPBSRollupsLocked() rr.refreshStoragePostureLocked() @@ -657,6 +693,7 @@ func (rr *ResourceRegistry) ingestRecords(source DataSource, records []IngestRec rr.refreshProxmoxSensorSetupLocked() rr.refreshCanonicalIdentitiesLocked() rr.invalidateViewsLocked() + rr.ingestStaleThresholds = nil rr.mu.Unlock() } @@ -746,12 +783,25 @@ func (rr *ResourceRegistry) proxmoxGuestResourceIDForSourceRefLocked(ref string) return uniqueID } +// canonicalIDObservedLocked reports whether a canonical ID still names an +// observed resource: a live row, or one a manual link folded into its primary. +// Succeeding an observed ID would hand its operator rows, and the link that +// folded it, to another resource. +func (rr *ResourceRegistry) canonicalIDObservedLocked(id string) bool { + if _, live := rr.resources[id]; live { + return true + } + _, folded := rr.linkMergedIDs[id] + return folded +} + // applyRecordSuccessions re-keys operator-owned store rows from canonical IDs // that ingested records declared superseded (IngestRecord.SupersededCanonicalIDs) // onto the records' current canonical IDs. Mirrors the guards of pin-driven -// successions in PersistIdentityPins: a superseded ID still held by a live -// resource is skipped so a genuinely distinct sibling never has its rows -// stolen. Re-runs are cheap no-ops once the old rows are gone. +// successions in PersistIdentityPins: a superseded ID still observed, live or +// folded into another resource by a manual link, is skipped so a genuinely +// distinct sibling never has its rows stolen. Re-runs are cheap no-ops once +// the old rows are gone. func (rr *ResourceRegistry) applyRecordSuccessions(successions []CanonicalIDSuccession) { if len(successions) == 0 || rr.store == nil { return @@ -764,7 +814,7 @@ func (rr *ResourceRegistry) applyRecordSuccessions(successions []CanonicalIDSucc rr.mu.RLock() kept := make([]CanonicalIDSuccession, 0, len(successions)) for _, succession := range successions { - if _, live := rr.resources[succession.OldCanonicalID]; live { + if rr.canonicalIDObservedLocked(succession.OldCanonicalID) { continue } kept = append(kept, succession) @@ -1772,26 +1822,15 @@ func sourceVerdictsRecorded(sightings map[DataSource]SourceStatus) bool { } func (rr *ResourceRegistry) markStaleLocked(now time.Time, thresholds map[DataSource]time.Duration) { - thresholds = effectiveStaleThresholds(thresholds) - changed := false for _, resource := range rr.resources { previousStatus := resource.Status staleFound := false for source, status := range resource.SourceStatus { - threshold, ok := thresholds[source] - if !ok { - threshold = 120 * time.Second - } - if status.ExpectedUpdateIntervalSeconds > 0 { - // Slow inventory polls have their own cadence. A source must miss - // two expected intervals before its retained observation is stale. - threshold = max(threshold, 2*time.Duration(status.ExpectedUpdateIntervalSeconds)*time.Second) - } if status.LastSeen.IsZero() { continue } - if now.Sub(status.LastSeen) > threshold { + if now.Sub(status.LastSeen) > sourceStaleThreshold(source, status, thresholds) { changed = changed || status.Status != "stale" status.Status = "stale" resource.SourceStatus[source] = status @@ -3940,7 +3979,7 @@ func (rr *ResourceRegistry) mergeInto(existing *Resource, incoming Resource, sou existing.ParentID = rr.resolveCanonicalParentID(existing) existing.Status = chooseStatus(existing.Status, incoming.Status, source, existing.Sources) - existing.Metrics = mergeMetrics(existing, existing.Metrics, incoming.Metrics, source, now, existing.SourceStatus, nil) + existing.Metrics = mergeMetrics(existing, existing.Metrics, incoming.Metrics, source, now, existing.SourceStatus, rr.ingestStaleThresholds) existing.Metrics = clearUnavailableSourceMemoryMetric(existing.Metrics, &incoming, source) // Prefer agent naming when available @@ -4811,6 +4850,10 @@ func (rr *ResourceRegistry) applyManualLinks(thresholds map[DataSource]time.Dura rr.mergeResourceData(primary, other, thresholds) delete(rr.resources, otherID) + if rr.linkMergedIDs == nil { + rr.linkMergedIDs = make(map[string]struct{}) + } + rr.linkMergedIDs[otherID] = struct{}{} rr.updateSourceMappings(otherID, primaryID) } } @@ -4876,6 +4919,12 @@ func (rr *ResourceRegistry) mergeResourceData(primary *Resource, other *Resource if primary.Ceph == nil { primary.Ceph = other.Ceph } + if primary.TrueNAS == nil { + primary.TrueNAS = other.TrueNAS + } + if primary.VMware == nil { + primary.VMware = other.VMware + } // Manual links combine already-normalized resources. Preserve each metric's // recorded source instead of flattening the linked resource to SourceAgent. @@ -6284,6 +6333,27 @@ func mergeMetrics( return &merged } +// sourceStaleThreshold is how long a source may go without delivering before +// its sighting is stale: the caller's threshold for the source, else the +// default. The stale pass and the metric merge's freshness gate both read it, +// so the gate never calls a source current that the pass marks stale, or the +// reverse. +func sourceStaleThreshold(source DataSource, sighting SourceStatus, thresholds map[DataSource]time.Duration) time.Duration { + threshold := thresholds[source] + if threshold <= 0 { + threshold = defaultStaleThresholds[source] + } + if threshold <= 0 { + threshold = 120 * time.Second + } + if sighting.ExpectedUpdateIntervalSeconds > 0 { + // Slow inventory polls have their own cadence. A source must miss + // two expected intervals before its retained observation is stale. + threshold = max(threshold, 2*time.Duration(sighting.ExpectedUpdateIntervalSeconds)*time.Second) + } + return threshold +} + // metricSourceStale reports whether a source's most recent report is older than // its stale threshold. A zero/unknown last-seen is treated as NOT stale so the // merge never demotes a source on missing information. @@ -6300,17 +6370,7 @@ func metricSourceStale( if !ok || st.LastSeen.IsZero() { return false } - threshold := time.Duration(0) - if configured := thresholds[source]; configured > 0 { - threshold = configured - } - if threshold <= 0 { - threshold = defaultStaleThresholds[source] - } - if threshold <= 0 { - threshold = 60 * time.Second - } - return now.Sub(st.LastSeen) > threshold + return now.Sub(st.LastSeen) > sourceStaleThreshold(source, st, thresholds) } func mergeMetric( diff --git a/internal/unifiedresources/registry_merge_policy_test.go b/internal/unifiedresources/registry_merge_policy_test.go index e4d7080211..6f04cb36b7 100644 --- a/internal/unifiedresources/registry_merge_policy_test.go +++ b/internal/unifiedresources/registry_merge_policy_test.go @@ -247,6 +247,114 @@ func TestLeaseExpiredAgentOnLiveProxmoxNodeStaysOnline(t *testing.T) { } } +// A source merge into an existing row judges metric freshness by the ingest's +// stale thresholds, as the stale pass that ends the ingest does. With PVE +// polled every two minutes the monitor allows Proxmox four minutes, so a node +// polled ninety seconds ago is current while its agent, silent for seventy-five +// seconds, is stale, and the row shows the node's readings. Without thresholds +// both sources are stale by the defaults and the agent keeps its precedence. +func TestSourceMergesJudgeFreshnessByTheIngestStaleThresholds(t *testing.T) { + now := time.Now().UTC() + snapshot := models.StateSnapshot{ + Nodes: []models.Node{{ + ID: "homelab-pve1", + Name: "pve1", + Instance: "homelab", + Status: "online", + LinkedAgentID: "host-pve1", + CPU: 0.12, + Memory: models.Memory{Total: 64 << 30, Used: 16 << 30, Free: 48 << 30, Usage: 25}, + LastSeen: now.Add(-90 * time.Second), + }}, + Hosts: []models.Host{{ + ID: "host-pve1", + MachineID: "machine-pve1", + Hostname: "pve1", + LinkedNodeID: "homelab-pve1", + Status: "online", + CPUUsage: 80, + Memory: models.Memory{Total: 64 << 30, Used: 48 << 30, Free: 16 << 30, Usage: 75}, + LastSeen: now.Add(-75 * time.Second), + IntervalSeconds: 30, + }}, + } + + for _, tc := range []struct { + name string + thresholds map[DataSource]time.Duration + proxmoxPoll string + want DataSource + wantCPU float64 + wantMemory float64 + }{ + {"configured", map[DataSource]time.Duration{SourceProxmox: 4 * time.Minute}, "online", SourceProxmox, 12, 25}, + {"defaults", nil, "stale", SourceAgent, 80, 75}, + } { + t.Run(tc.name, func(t *testing.T) { + rr := NewRegistry(nil) + rr.IngestSnapshotWithStaleThresholds(snapshot, tc.thresholds) + + resource := onlyResourceOfType(t, rr, ResourceTypeAgent) + if got := resource.SourceStatus[SourceProxmox].Status; got != tc.proxmoxPoll { + t.Fatalf("Proxmox sighting = %q, want %q: %+v", got, tc.proxmoxPoll, resource.SourceStatus) + } + if got := resource.SourceStatus[SourceAgent].Status; got != "stale" { + t.Fatalf("agent sighting = %q, want stale", got) + } + if resource.Metrics == nil || resource.Metrics.CPU == nil || resource.Metrics.Memory == nil { + t.Fatalf("merged row lost its readings: %+v", resource.Metrics) + } + if cpu := resource.Metrics.CPU; cpu.Source != tc.want || cpu.Percent != tc.wantCPU { + t.Fatalf("CPU = %.0f%% from %s, want %.0f%% from %s", cpu.Percent, cpu.Source, tc.wantCPU, tc.want) + } + if memory := resource.Metrics.Memory; memory.Source != tc.want || memory.Percent != tc.wantMemory { + t.Fatalf("memory = %.0f%% from %s, want %.0f%% from %s", memory.Percent, memory.Source, tc.wantMemory, tc.want) + } + }) + } +} + +// The metric merge and the stale pass read one threshold rule, so a source +// sighting the merge treats as current is never one the pass marks stale. +func TestMetricMergeFreshnessMatchesTheStalePass(t *testing.T) { + now := time.Now().UTC() + configured := map[DataSource]time.Duration{SourceProxmox: 4 * time.Minute, SourceVMware: 5 * time.Minute} + for _, tc := range []struct { + name string + source DataSource + sighting SourceStatus + thresholds map[DataSource]time.Duration + wantStale bool + }{ + {"proxmox by default", SourceProxmox, SourceStatus{LastSeen: now.Add(-90 * time.Second)}, nil, true}, + {"proxmox by configuration", SourceProxmox, SourceStatus{LastSeen: now.Add(-90 * time.Second)}, configured, false}, + {"vsphere by default", SourceVMware, SourceStatus{LastSeen: now.Add(-3 * time.Minute)}, nil, true}, + {"vsphere by configuration", SourceVMware, SourceStatus{LastSeen: now.Add(-3 * time.Minute)}, configured, false}, + {"agent without its own threshold", SourceAgent, SourceStatus{LastSeen: now.Add(-75 * time.Second)}, configured, true}, + {"slow disk inventory", SourceProxmox, SourceStatus{LastSeen: now.Add(-5 * time.Minute), ExpectedUpdateIntervalSeconds: 600}, nil, false}, + {"missed two disk inventory polls", SourceProxmox, SourceStatus{LastSeen: now.Add(-21 * time.Minute), ExpectedUpdateIntervalSeconds: 600}, nil, true}, + } { + t.Run(tc.name, func(t *testing.T) { + status := map[DataSource]SourceStatus{tc.source: tc.sighting} + if got := metricSourceStale(now, status, tc.source, tc.thresholds); got != tc.wantStale { + t.Fatalf("metric merge stale = %v, want %v", got, tc.wantStale) + } + + rr := NewRegistry(nil) + rr.resources["resource"] = &Resource{ + ID: "resource", + Status: StatusOnline, + Sources: []DataSource{tc.source}, + SourceStatus: map[DataSource]SourceStatus{tc.source: tc.sighting}, + } + rr.MarkStale(now, tc.thresholds) + if got := rr.resources["resource"].SourceStatus[tc.source].Status == "stale"; got != tc.wantStale { + t.Fatalf("stale pass stale = %v, want %v", got, tc.wantStale) + } + }) + } +} + // Docker hosts hold a shorter lease than the registry's Docker stale // threshold, so a silent Docker host was offline for a while and then // flipped to warning once its sighting went stale. @@ -441,6 +549,319 @@ func TestManualLinkKeepsTheLiveSightingOfASharedSource(t *testing.T) { } } +// Either side of an operator link can arrive as a supplemental record: a +// vSphere VM with a Pulse agent inside, or an agent on a TrueNAS host. The +// monitor's rebuild ingests records after the snapshot, so it must join them +// there too, keep the record side's provider payload, and list what the +// resources API lists after seeding from it. +func TestManualLinksJoinRecordIngestedResourcesInMonitorRebuild(t *testing.T) { + now := time.Now().UTC() + snapshot := models.StateSnapshot{ + Hosts: []models.Host{ + {ID: "host-app-guest", Hostname: "app-guest", MachineID: "machine-app-guest", Status: "online", LastSeen: now}, + {ID: "host-nas", Hostname: "nas-a-mgmt", MachineID: "machine-nas-a", Status: "online", LastSeen: now}, + }, + LastUpdate: now, + } + vmRecord := IngestRecord{ + SourceID: "vc-1:vm:vm-42", + Resource: Resource{ + Type: ResourceTypeVM, + Technology: "vmware", + Name: "app-guest", + Status: StatusOnline, + LastSeen: now, + VMware: &VMwareData{ConnectionID: "vc-1", ManagedObjectID: "vm-42", EntityType: "vm"}, + }, + Identity: ResourceIdentity{Hostnames: []string{"app-guest"}}, + } + records := map[DataSource][]IngestRecord{ + SourceVMware: {vmRecord}, + SourceTrueNAS: { + { + SourceID: "system:tn-1", + Resource: Resource{ + Type: ResourceTypeAgent, + Name: "nas-a", + Status: StatusOnline, + LastSeen: now, + TrueNAS: &TrueNASData{Hostname: "nas-a"}, + }, + Identity: ResourceIdentity{Hostnames: []string{"nas-a"}}, + }, + { + SourceID: "system:tn-1:pool:tank", + ParentSourceID: "system:tn-1", + Resource: Resource{ + Type: ResourceTypeStorage, + Name: "tank", + Status: StatusOnline, + LastSeen: now, + Storage: &StorageMeta{Type: "zfs-pool", Platform: "truenas", Topology: "pool"}, + }, + }, + }, + } + + for _, agentPrimary := range []bool{false, true} { + name := "record-primary" + if agentPrimary { + name = "agent-primary" + } + t.Run(name, func(t *testing.T) { + store := NewMemoryStore() + adapter := NewMonitorAdapter(NewRegistry(store)) + adapter.PopulateSnapshotAndSupplemental(snapshot, records) + + var vmID, guestAgentID, systemID, nasAgentID string + for _, resource := range adapter.GetAll() { + switch { + case resource.VMware != nil: + vmID = resource.ID + case resource.TrueNAS != nil: + systemID = resource.ID + case resource.Name == "app-guest": + guestAgentID = resource.ID + case resource.Name == "nas-a-mgmt": + nasAgentID = resource.ID + } + } + if vmID == "" || guestAgentID == "" || systemID == "" || nasAgentID == "" { + t.Fatalf("unlinked estate = %v, want the VM, both agents and the TrueNAS system", resourceIDs(adapter.GetAll())) + } + guestPrimary, nasPrimary := vmID, systemID + if agentPrimary { + guestPrimary, nasPrimary = guestAgentID, nasAgentID + } + for _, link := range []ResourceLink{ + {ResourceA: vmID, ResourceB: guestAgentID, PrimaryID: guestPrimary}, + {ResourceA: systemID, ResourceB: nasAgentID, PrimaryID: nasPrimary}, + } { + if err := store.AddLink(link); err != nil { + t.Fatalf("add link: %v", err) + } + } + + assertJoined := func(stage string) { + t.Helper() + resources := adapter.GetAll() + if len(resources) != 3 { + t.Fatalf("%s: resources = %v, want the linked VM, the linked TrueNAS system and its pool", stage, resourceIDs(resources)) + } + // An agent inside a guest supplements the guest, whichever + // side the operator chose as primary. + guest, ok := adapter.currentRegistry().Get(vmID) + if !ok { + t.Fatalf("%s: linked vSphere VM %s missing from %v", stage, vmID, resourceIDs(resources)) + } + if guest.Type != ResourceTypeVM || guest.VMware == nil || guest.Agent == nil { + t.Fatalf("%s: linked VM type=%s vmware=%t agent=%t, want the VM carrying its agent", stage, guest.Type, guest.VMware != nil, guest.Agent != nil) + } + if !slices.Contains(guest.Sources, SourceVMware) || !slices.Contains(guest.Sources, SourceAgent) { + t.Fatalf("%s: linked VM sources = %v, want vmware and agent", stage, guest.Sources) + } + system, ok := adapter.currentRegistry().Get(nasPrimary) + if !ok { + t.Fatalf("%s: linked TrueNAS system %s missing from %v", stage, nasPrimary, resourceIDs(resources)) + } + if system.TrueNAS == nil || system.Agent == nil { + t.Fatalf("%s: linked system truenas=%t agent=%t, want the primary carrying both payloads", stage, system.TrueNAS != nil, system.Agent != nil) + } + if !slices.Contains(system.Sources, SourceTrueNAS) || !slices.Contains(system.Sources, SourceAgent) { + t.Fatalf("%s: linked system sources = %v, want truenas and agent", stage, system.Sources) + } + pool := onlyResourceOfType(t, adapter.currentRegistry(), ResourceTypeStorage) + if pool.ParentID == nil || *pool.ParentID != nasPrimary { + t.Fatalf("%s: pool parent = %v, want the linked system %s", stage, pool.ParentID, nasPrimary) + } + + // Identity pins persisted by the rebuild must not re-key a link + // onto its own primary. + links, err := store.GetLinks() + if err != nil { + t.Fatalf("%s: get links: %v", stage, err) + } + for _, link := range links { + if link.ResourceA == link.ResourceB { + t.Fatalf("%s: link collapsed onto %s", stage, link.ResourceA) + } + } + + // The resources API seeds its registry from this listing + // through resource ingest and must agree with it. + rest := NewRegistry(store) + rest.IngestResources(resources) + if got, want := resourceIDs(rest.List()), resourceIDs(resources); !slices.Equal(got, want) { + t.Fatalf("%s: a registry seeded from the monitor lists %v, monitor lists %v", stage, got, want) + } + } + + next := snapshot + next.LastUpdate = now.Add(time.Second) + adapter.PopulateSnapshotAndSupplemental(next, records) + assertJoined("rebuild") + + adapter.PopulateSupplementalRecords(SourceVMware, records[SourceVMware]) + adapter.PopulateSupplementalRecords(SourceTrueNAS, records[SourceTrueNAS]) + assertJoined("supplemental refresh") + + next.LastUpdate = now.Add(2 * time.Second) + adapter.PopulateSnapshotAndSupplemental(next, records) + assertJoined("second rebuild") + }) + } +} + +// A Proxmox node that absorbs a linked agent carries the agent's machine key, +// so the rebuild pins the node with it. The agent is still observed: that pin +// must not succeed the agent's ID onto the node, which re-keyed the link onto +// the node itself and split the pair on the next rebuild. +func TestManualLinkKeepsItsEndpointsThroughIdentityPinPersistence(t *testing.T) { + now := time.Now().UTC() + snapshot := models.StateSnapshot{ + Nodes: []models.Node{{ + ID: "lab-pve1", Name: "pve1", Instance: "lab", + Host: "https://192.0.2.10:8006", Status: "online", LastSeen: now, + }}, + Hosts: []models.Host{{ + ID: "host-box", Hostname: "box-x", MachineID: "machine-box", Status: "online", LastSeen: now, + }}, + LastUpdate: now, + } + store := NewMemoryStore() + adapter := NewMonitorAdapter(NewRegistry(store)) + adapter.PopulateSnapshotAndSupplemental(snapshot, nil) + var nodeID, agentID string + for _, resource := range adapter.GetAll() { + if resource.Proxmox != nil { + nodeID = resource.ID + } else if resource.Agent != nil { + agentID = resource.ID + } + } + if nodeID == "" || agentID == "" || nodeID == agentID { + t.Fatalf("unlinked estate = %v, want a node and a separate agent", resourceIDs(adapter.GetAll())) + } + if err := store.AddLink(ResourceLink{ResourceA: nodeID, ResourceB: agentID, PrimaryID: nodeID}); err != nil { + t.Fatalf("add link: %v", err) + } + + for rebuild := 1; rebuild <= 3; rebuild++ { + next := snapshot + next.LastUpdate = now.Add(time.Duration(rebuild) * time.Second) + adapter.PopulateSnapshotAndSupplemental(next, nil) + if got := resourceIDs(adapter.GetAll()); !slices.Equal(got, []string{nodeID}) { + t.Fatalf("rebuild %d: resources = %v, want only the linked node %s", rebuild, got, nodeID) + } + links, err := store.GetLinks() + if err != nil { + t.Fatalf("get links: %v", err) + } + if len(links) != 1 || links[0].ResourceA != nodeID || links[0].ResourceB != agentID { + t.Fatalf("rebuild %d: links = %+v, want %s -> %s kept", rebuild, links, nodeID, agentID) + } + } +} + +// A record may name an older canonical ID as superseded on every ingest. When +// that ID belongs to a resource a manual link folded into the record's own +// resource, it is still observed: succeeding it on a later refresh would +// re-key the link onto its primary. +func TestRecordDeclaredSuccessionSparesALinkFoldedResource(t *testing.T) { + now := time.Now().UTC() + snapshot := models.StateSnapshot{ + Hosts: []models.Host{{ID: "host-nas", Hostname: "nas-a-mgmt", MachineID: "machine-nas-a", Status: "online", LastSeen: now}}, + LastUpdate: now, + } + agentID := MachineIdentityCanonicalID(ResourceTypeAgent, "machine-nas-a") + records := []IngestRecord{{ + SourceID: "system:tn-1", + SupersededCanonicalIDs: []string{agentID}, + Resource: Resource{ + Type: ResourceTypeAgent, + Name: "nas-a", + Status: StatusOnline, + LastSeen: now, + TrueNAS: &TrueNASData{Hostname: "nas-a"}, + }, + Identity: ResourceIdentity{Hostnames: []string{"nas-a"}}, + }} + + unlinked := NewMonitorAdapter(NewRegistry(nil)) + unlinked.PopulateSnapshotAndSupplemental(snapshot, map[DataSource][]IngestRecord{SourceTrueNAS: records}) + var systemID string + for _, resource := range unlinked.GetAll() { + if resource.TrueNAS != nil { + systemID = resource.ID + } + } + if systemID == "" || !slices.Contains(resourceIDs(unlinked.GetAll()), agentID) { + t.Fatalf("unlinked estate = %v, want agent %s and a TrueNAS system", resourceIDs(unlinked.GetAll()), agentID) + } + + store := NewMemoryStore() + if err := store.AddLink(ResourceLink{ResourceA: systemID, ResourceB: agentID, PrimaryID: systemID}); err != nil { + t.Fatalf("add link: %v", err) + } + adapter := NewMonitorAdapter(NewRegistry(store)) + adapter.PopulateSnapshotAndSupplemental(snapshot, map[DataSource][]IngestRecord{SourceTrueNAS: records}) + adapter.PopulateSupplementalRecords(SourceTrueNAS, records) + + if got := resourceIDs(adapter.GetAll()); !slices.Equal(got, []string{systemID}) { + t.Fatalf("resources = %v, want only the linked system %s", got, systemID) + } + links, err := store.GetLinks() + if err != nil { + t.Fatalf("get links: %v", err) + } + if len(links) != 1 || links[0].ResourceA != systemID || links[0].ResourceB != agentID { + t.Fatalf("links = %+v, want %s -> %s kept", links, systemID, agentID) + } +} + +// A link merge folds the other resource's provider payloads into the primary +// where the primary has none, whichever side the operator chose. +func TestManualLinkKeepsProviderPayloadsThePrimaryLacks(t *testing.T) { + now := time.Now().UTC() + store := NewMemoryStore() + for _, link := range []ResourceLink{ + {ResourceA: "agent-nas", ResourceB: "truenas-system", PrimaryID: "agent-nas"}, + {ResourceA: "agent-esxi", ResourceB: "vmware-host", PrimaryID: "agent-esxi"}, + } { + if err := store.AddLink(link); err != nil { + t.Fatalf("add link: %v", err) + } + } + host := func(id, name string, sources ...DataSource) Resource { + status := make(map[DataSource]SourceStatus, len(sources)) + for _, source := range sources { + status[source] = SourceStatus{Status: "online", LastSeen: now} + } + return Resource{ID: id, Type: ResourceTypeAgent, Name: name, Status: StatusOnline, LastSeen: now, Sources: sources, SourceStatus: status} + } + agentNAS := host("agent-nas", "nas-a-mgmt", SourceAgent) + agentNAS.Agent = &AgentData{AgentID: "agent-nas", Hostname: "nas-a-mgmt"} + system := host("truenas-system", "nas-a", SourceTrueNAS) + system.TrueNAS = &TrueNASData{Hostname: "nas-a"} + agentESXi := host("agent-esxi", "esxi-01", SourceAgent) + agentESXi.Agent = &AgentData{AgentID: "agent-esxi", Hostname: "esxi-01"} + esxi := host("vmware-host", "esxi-01.lab", SourceVMware) + esxi.VMware = &VMwareData{ConnectionID: "vc-1", ManagedObjectID: "host-101", EntityType: "host"} + + rr := NewRegistry(store) + rr.IngestResources([]Resource{agentNAS, system, agentESXi, esxi}) + + if got := resourceIDs(rr.List()); !slices.Equal(got, []string{"agent-esxi", "agent-nas"}) { + t.Fatalf("resources = %v, want the two linked primaries", got) + } + if nas, _ := rr.Get("agent-nas"); nas.TrueNAS == nil || nas.Agent == nil { + t.Fatalf("linked TrueNAS system truenas=%t agent=%t, want both payloads", nas.TrueNAS != nil, nas.Agent != nil) + } + if host, _ := rr.Get("agent-esxi"); host.VMware == nil || host.Agent == nil { + t.Fatalf("linked vSphere host vmware=%t agent=%t, want both payloads", host.VMware != nil, host.Agent != nil) + } +} + func TestLeaseExpiredKubernetesClusterStaysOffline(t *testing.T) { rr := NewRegistry(nil) rr.IngestSnapshot(models.StateSnapshot{ diff --git a/internal/unifiedresources/registry_test.go b/internal/unifiedresources/registry_test.go index 56a426e60c..271c2c4a36 100644 --- a/internal/unifiedresources/registry_test.go +++ b/internal/unifiedresources/registry_test.go @@ -2857,6 +2857,101 @@ func TestResourceRegistry_IngestSnapshotCreatesUnraidDisksWithoutSMART(t *testin } } +func TestHostDiskTemperatureReadingsMatchTheDisksTheRegistryShows(t *testing.T) { + now := time.Date(2026, 10, 7, 12, 0, 0, 0, time.UTC) + host := models.Host{ + ID: "host-tower", + Hostname: "tower", + Status: "online", + LastSeen: now, + Sensors: models.HostSensorSummary{ + SMART: []models.HostDiskSMART{ + // smartctl read this disk itself. + {Device: "sda", Model: "WDC A", Serial: "SER-A", Type: "sata", Temperature: 38}, + // smartctl returned no temperature; Unraid read one. + {Device: "sdb", Model: "WDC B", Serial: "SER-B", Type: "sata"}, + // smartctl and Unraid name this device with different serials. + {Device: "sde", Model: "Bridge E", Serial: "SER-E1", Type: "sata", Temperature: 41}, + // The registry ingests no virtual device. + {Device: "zram0", Temperature: 70}, + // A controller member Unraid lists under its block device. + {Device: "0 [megaraid,0]", Model: "Seagate F", Serial: "SER-F", Type: "sata", Controller: "0", Target: "megaraid,0", Temperature: 44}, + }, + }, + Unraid: &models.HostUnraidStorage{ + ArrayStarted: true, + ArrayState: "STARTED", + Disks: []models.HostUnraidDisk{ + {Name: "disk1", Device: "/dev/sda", Role: "data", Status: "online", Serial: "SER-A", Transport: "sata", Temperature: 36}, + {Name: "disk2", Device: "/dev/sdb", Role: "data", Status: "online", Serial: "SER-B", Transport: "sata", Temperature: 47}, + // No SMART row, as for a --disk-exclude match. + {Name: "disk3", Device: "/dev/nvme0n1", Role: "data", Status: "online", Model: "Samsung C", Serial: "SER-C", Transport: "nvme", Temperature: 52}, + {Name: "disk4", Device: "/dev/sdd", Role: "data", Status: "online", Model: "WDC D", Serial: "SER-D", Transport: "sata", Temperature: 30, SpunDown: true}, + {Name: "disk5", Device: "/dev/sde", Role: "data", Status: "online", Model: "WDC E", Serial: "SER-E2", Transport: "sata", Temperature: 43}, + {Name: "disk7", Device: "/dev/sdf", Role: "data", Status: "online", Model: "Seagate F", Serial: "SER-F", Transport: "sata", Temperature: 44}, + // An empty slot: the registry ingests no disk for it. + {Name: "disk6", Role: "data", Status: "missing"}, + }, + }, + } + + type want struct { + device string + diskType string + temperature int + collected bool + unraidOnly bool + } + wants := map[string]want{ + "SER-A": {device: "sda", diskType: "sata", temperature: 38, collected: true}, + "SER-B": {device: "sdb", diskType: "sata", temperature: 47, collected: true}, + "SER-E1": {device: "sde", diskType: "sata", temperature: 41, collected: true}, + "SER-F": {device: "0 [megaraid,0]", diskType: "sata", temperature: 44, collected: true}, + "SER-C": {device: "nvme0n1", diskType: "nvme", temperature: 52, collected: true, unraidOnly: true}, + "SER-D": {device: "sdd", diskType: "sata", temperature: 30, unraidOnly: true}, + "SER-E2": {device: "sde", diskType: "sata", temperature: 43, collected: true, unraidOnly: true}, + } + + readings := HostDiskTemperatureReadings(host) + if len(readings) != len(wants) { + t.Fatalf("expected %d readings, got %d: %+v", len(wants), len(readings), readings) + } + byKey := make(map[string]HostDiskTemperatureReading, len(readings)) + for _, reading := range readings { + w, ok := wants[reading.MetricID] + if !ok { + t.Fatalf("unexpected reading %+v", reading) + } + if reading.Device != w.device || reading.DiskType != w.diskType || reading.Temperature != w.temperature || + reading.Collected() != w.collected || reading.UnraidOnly != w.unraidOnly { + t.Fatalf("reading for %s = %+v (collected %v), want %+v", reading.MetricID, reading, reading.Collected(), w) + } + byKey[reading.MetricID] = reading + } + + // The readings are the ones the disks the registry builds show, one each. + rr := NewRegistry(nil) + rr.IngestSnapshot(models.StateSnapshot{Hosts: []models.Host{host}}) + disks := rr.ListByType(ResourceTypePhysicalDisk) + if len(disks) != len(wants) { + t.Fatalf("expected %d physical disks, got %d: %+v", len(wants), len(disks), disks) + } + for _, disk := range disks { + meta := disk.PhysicalDisk + if meta == nil { + t.Fatalf("physical disk without meta: %+v", disk) + } + reading, ok := byKey[meta.Serial] + if !ok { + t.Fatalf("registry disk %q has no reading", meta.Serial) + } + if meta.Temperature != reading.Temperature || meta.DiskType != reading.DiskType || + diskinventory.TemperatureCollected(meta.Temperature, meta.Collection) != diskinventory.TemperatureCollected(reading.Temperature, reading.Collection) { + t.Fatalf("registry disk %q shows %dC %q (collection %+v), reading %+v", meta.Serial, meta.Temperature, meta.DiskType, meta.Collection, reading) + } + } +} + func TestResourceRegistry_IngestSnapshotCreatesUnraidStorageResource(t *testing.T) { rr := NewRegistry(nil) now := time.Date(2026, 3, 7, 12, 0, 0, 0, time.UTC) @@ -5138,6 +5233,118 @@ func TestResourceRegistry_ManualGuestAgentLinkUsesAgentCPUOnlyWhenPlatformHasNoC } } +// The workload table, guest drawer and History judge a guest's disk reading by +// the linked agent's freshness when the agent's filesystems fill in for +// Proxmox's (useWorkloads disksFromAgent). That needs the registry to keep the +// agent's disk metric while Proxmox has no guest disk usage, also once the +// agent is silent, and Proxmox's own fresh reading otherwise. +func TestResourceRegistry_ManualGuestAgentLinkDiskMetricFollowsProxmoxGuestFilesystems(t *testing.T) { + now := time.Now().UTC() + agentDisks := []models.Disk{{Total: 1000, Used: 400, Free: 600, Usage: 40, Mountpoint: "/"}} + cases := []struct { + name string + vmDisk models.Disk + vmDisks []models.Disk + reason string + agentStatus string + agentLastSeen time.Time + wantSource DataSource + wantPercent float64 + wantStale bool + }{ + { + name: "no proxmox filesystems, live agent", + vmDisk: models.Disk{Total: 32 << 30, Usage: -1}, + reason: "agent-not-running", + agentStatus: "online", + agentLastSeen: now, + wantSource: SourceAgent, + wantPercent: 40, + }, + { + name: "no proxmox filesystems, silent agent keeps its last disk", + vmDisk: models.Disk{Total: 32 << 30, Usage: -1}, + reason: "agent-not-running", + agentStatus: "offline", + agentLastSeen: now.Add(-10 * time.Minute), + wantSource: SourceAgent, + wantPercent: 40, + wantStale: true, + }, + { + name: "proxmox guest filesystems keep proxmox disk", + vmDisk: models.Disk{Total: 1000, Used: 700, Free: 300, Usage: 70}, + vmDisks: []models.Disk{{Total: 1000, Used: 700, Free: 300, Usage: 70, Mountpoint: "/"}}, + agentStatus: "online", + agentLastSeen: now, + wantSource: SourceProxmox, + wantPercent: 70, + }, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + snapshot := models.StateSnapshot{ + VMs: []models.VM{{ + ID: "pve-a:node-1:402", + VMID: 402, + Name: "vm-402", + Node: "node-1", + Instance: "pve-a", + Type: "qemu", + Status: "running", + LastSeen: now, + Disk: tc.vmDisk, + Disks: tc.vmDisks, + DiskStatusReason: tc.reason, + }}, + Hosts: []models.Host{{ + ID: "agent-402", + Hostname: "guest-402", + Status: tc.agentStatus, + LastSeen: tc.agentLastSeen, + Disks: agentDisks, + }}, + } + store := NewMemoryStore() + unlinked := NewRegistry(store) + unlinked.IngestSnapshot(snapshot) + var vmID, agentID string + for _, resource := range unlinked.List() { + switch resource.Type { + case ResourceTypeVM: + vmID = resource.ID + case ResourceTypeAgent: + agentID = resource.ID + } + } + if vmID == "" || agentID == "" { + t.Fatalf("vm %q / agent %q not ingested", vmID, agentID) + } + if err := store.AddLink(ResourceLink{ResourceA: vmID, ResourceB: agentID, PrimaryID: vmID}); err != nil { + t.Fatalf("add link: %v", err) + } + + rr := NewRegistry(store) + rr.IngestSnapshot(snapshot) + got, ok := rr.Get(vmID) + if !ok || got.Metrics == nil || got.Metrics.Disk == nil { + t.Fatalf("linked guest lost its disk metric: %+v", got) + } + if got.Metrics.Disk.Source != tc.wantSource || got.Metrics.Disk.Percent != tc.wantPercent { + t.Fatalf("disk = %+v, want %s at %.0f%%", got.Metrics.Disk, tc.wantSource, tc.wantPercent) + } + agentFilesystemsFillIn := got.Proxmox != nil && len(got.Proxmox.Disks) == 0 && + got.Agent != nil && len(got.Agent.Disks) > 0 + if agentFilesystemsFillIn != (tc.wantSource == SourceAgent) { + t.Fatalf("agent filesystems fill in = %v, disk metric source %s", agentFilesystemsFillIn, tc.wantSource) + } + if got.Agent == nil || got.Agent.Stale != tc.wantStale { + t.Fatalf("agent stale = %+v, want %v", got.Agent, tc.wantStale) + } + }) + } +} + func TestResourceRegistry_ManualGuestAgentLinkReconnectRestoresFreshPlatformCPU(t *testing.T) { now := time.Now().UTC() const ( diff --git a/internal/unifiedresources/types.go b/internal/unifiedresources/types.go index d54c5e831b..1e54e94b23 100644 --- a/internal/unifiedresources/types.go +++ b/internal/unifiedresources/types.go @@ -967,8 +967,10 @@ type AgentData struct { // offline by the staleness evaluator) even though the row itself may stay // online via another source such as the Proxmox API poll. It lets the UI // present the agent and its version as not-reporting instead of a - // healthy-looking stale value. - Stale bool `json:"stale,omitempty"` + // healthy-looking stale value. It is always sent: browsers merge agent + // facets field by field, so an omitted false would keep a resumed agent + // marked as stopped reporting. + Stale bool `json:"stale"` // LastReportAt is the agent's own last successful report time. On a // multi-source row (for example a Proxmox node also polled over the PVE // API) this differs from the row's LastSeen, which reflects the freshest diff --git a/internal/unifiedresources/views.go b/internal/unifiedresources/views.go index 81667e27e1..7f9d4a890c 100644 --- a/internal/unifiedresources/views.go +++ b/internal/unifiedresources/views.go @@ -364,6 +364,14 @@ func (v VMView) DiskPercent() float64 { return viewMetricPercent(v.r.Metrics, selectMetricsDisk) } +// DiskFromLinkedAgent reports whether the selected disk metric is the linked +// Pulse agent's. It fills in when Proxmox has no guest filesystems of its own, +// so it is never Proxmox's guest reading. +func (v VMView) DiskFromLinkedAgent() bool { + return v.r != nil && v.r.Metrics != nil && v.r.Metrics.Disk != nil && + v.r.Metrics.Disk.Source == SourceAgent +} + func (v VMView) NetIn() float64 { if v.r == nil { return 0 diff --git a/internal/unifiedresources/views_test.go b/internal/unifiedresources/views_test.go index 8c8bcc98cc..7cf1ce2e93 100644 --- a/internal/unifiedresources/views_test.go +++ b/internal/unifiedresources/views_test.go @@ -2374,3 +2374,19 @@ func TestContainerViewLinkedAgentMemory(t *testing.T) { t.Fatal("nil view has memory") } } + +func TestVMViewDiskFromLinkedAgentFollowsSelectedMetricSource(t *testing.T) { + for _, tc := range []struct { + metrics *ResourceMetrics + want bool + }{ + {metrics: &ResourceMetrics{Disk: &MetricValue{Percent: 40, Source: SourceAgent}}, want: true}, + {metrics: &ResourceMetrics{Disk: &MetricValue{Percent: 40, Source: SourceProxmox}}}, + {metrics: &ResourceMetrics{Memory: &MetricValue{Percent: 40, Source: SourceAgent}}}, + {metrics: nil}, + } { + if got := NewVMView(&Resource{Type: ResourceTypeVM, Metrics: tc.metrics}).DiskFromLinkedAgent(); got != tc.want { + t.Fatalf("DiskFromLinkedAgent(%+v) = %v, want %v", tc.metrics, got, tc.want) + } + } +} diff --git a/pkg/reporting/catalog.go b/pkg/reporting/catalog.go index 3820d821c2..fcf287cc09 100644 --- a/pkg/reporting/catalog.go +++ b/pkg/reporting/catalog.go @@ -2,6 +2,7 @@ package reporting import ( "fmt" + "net/http" "strings" "time" ) @@ -33,6 +34,7 @@ type PerformanceReportDefinition struct { Title string `json:"title"` Description string `json:"description"` SingleResourceEndpoint string `json:"singleResourceEndpoint"` + SingleResourceMethod string `json:"singleResourceMethod"` MultiResourceEndpoint string `json:"multiResourceEndpoint"` SingleFilenamePrefix string `json:"singleFilenamePrefix"` SingleFilenameSubject string `json:"singleFilenameSubject"` @@ -150,6 +152,7 @@ func DescribePerformanceReport() PerformanceReportDefinition { Title: "Performance Reports", Description: "Generate PDF summaries or CSV metric exports from historical monitoring data for one or more selected resources.", SingleResourceEndpoint: "/api/admin/reports/generate", + SingleResourceMethod: http.MethodPost, MultiResourceEndpoint: "/api/admin/reports/generate-multi", SingleFilenamePrefix: "report", SingleFilenameSubject: FilenameSubjectResourceID, diff --git a/scripts/release_control/subsystem_lookup_test.py b/scripts/release_control/subsystem_lookup_test.py index ae75d43aa3..b2b903baa8 100644 --- a/scripts/release_control/subsystem_lookup_test.py +++ b/scripts/release_control/subsystem_lookup_test.py @@ -672,6 +672,15 @@ class SubsystemLookupTest(unittest.TestCase): self.assertEqual(file_entry["classification"], "runtime") self.assertEqual(file_entry["matches"], []) + def test_lookup_paths_reports_retired_problem_resource_helper_as_unowned(self) -> None: + path = "frontend-modern/src/utils/problemResourcePresentation.ts" + result = lookup_paths([path]) + self.assertEqual(result["unowned_runtime_files"], [path]) + self.assertEqual(result["impacted_subsystems"], []) + file_entry = result["files"][0] + self.assertEqual(file_entry["classification"], "runtime") + self.assertEqual(file_entry["matches"], []) + def test_lookup_paths_assigns_recovery_points_hook_to_storage_recovery(self) -> None: result = lookup_paths(["frontend-modern/src/hooks/useRecoveryPoints.ts"]) self.assertEqual(result["unowned_runtime_files"], [])