mirror of
https://github.com/rcourtman/Pulse.git
synced 2026-10-24 04:33:17 +00:00
e7cabc72f1c31dc8be4d35c7f64550fe2bdf3c57
13050 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 rcourtman
|
e7cabc72f1 |
Hide pprof routes on demo-mode instances (#2708)
The `/debug/pprof` handlers are gated on `RequireAdmin` plus a `settings:read` scope that session logins skip. A `DEMO_MODE` instance that signs visitors in as the configured admin therefore served every visitor goroutine and heap dumps, the process command line, and CPU profiles and execution traces that load the whole process. Demo mode already hides diagnostics, logs, admin users and manual discovery through the admin-operations policy; pprof was missing from it. ## Change - `internal/api/demo_mode_operations.go`: through the admin-operations policy, the demo guard answers `/debug/pprof` and every path below it with `404` for every method, ahead of route authentication, so a write probe such as the `POST` symbol lookup gets the same `404` instead of the generic demo `403`. Instances outside demo mode keep the routes, and `PULSE_PPROF_DISABLED` still removes them everywhere. - Contract notes in api-contracts (owner), agent-lifecycle and storage-recovery. ## Proof - `TestContract_DemoModeHidesPprofFromDemoAdminSession` signs in as the demo admin through the full router. The same session gets `200` from `/debug/pprof/cmdline` below the demo guard, and `404` for eight pprof paths × `GET`/`HEAD`/`OPTIONS`/`POST` through the served handler. Against the previous policy (`go test -overlay`) `GET` and `HEAD` returned pprof output (`200`, or the root's `301` into the index), `OPTIONS` got the router's preflight `200` and `POST` the generic demo `403`. - `TestPublicDemoAdminOperationsPolicyHidesEveryDebugRoute` parses the router files and fails if a literal route under `/debug/`, with or without a method prefix, escapes the policy for any of seven methods, so a new debug handler there cannot ship reachable on a demo by default. A temporary `GET /debug/vars` registration made it fail as expected. - Path-handling probe (not committed): 210 raw requests (35 request targets × six methods), covering trailing and double slashes, dot segments, `%2F`/`%2e` encodings, absolute-form targets and `CONNECT`, run against a copy of the router's mux dispatch. None reached a pprof handler; paths the mux would clean onto pprof only receive a redirect, and the policy catches the follow-up request. |
||
|
rcourtman
|
6263c580f7 |
Judge a linked agent's guest disks by that agent everywhere (#2712)
Follow-up to #2669, which made the guest drawer's filesystem rows follow a linked Pulse agent's freshness. The workload table's disk cell and the drawer History still judged the same guest by the Proxmox read reason, so one guest disagreed with itself across the three. ## What changes - **One rule for three readers.** `getWorkloadGuestDiskRead` (`workloadGuestPresentation.ts`) now drives the table disk cell, the drawer's filesystem rows and History. When the linked agent's filesystems fill in for a VM that has no Proxmox guest filesystems, the reading is current while the agent reports, last known once it is stale, and unavailable once the guest is stopped. Proxmox's own guest filesystems keep their read reason. - **Where the aggregate disk metric comes from.** On a manually linked VM, Proxmox records no guest disk usage without guest filesystems, so the registry's link merge keeps the agent's disk metric. A registry test pins that. - **Poller carry-forward.** `previousVMFromView` built the previous VM from the merged metric, so the next poll carried the agent's disk forward as a Proxmox `prev-agent-not-running` read. Proxmox priority then froze it over the agent's live value. Only Proxmox's own disk reading is carried now (`VMView.DiskFromLinkedAgent`). - **`agent.stale` is always on the wire.** The browser merges agent facets field by field, so an omitted false kept a resumed agent "stopped reporting" until a reload. This affected Machines rows and node drawers too. - **Withdrawn filesystems.** The websocket merge kept filesystems that a native Proxmox VM facet or a native agent facet (which always carries `stale`) no longer reported, including in the `platformData.agent` mirror that host threshold rows read. ## Verification - New tests, each failing on main: `GuestRow.test.tsx`, `GuestDrawer.test.tsx`, `workloadGuestPresentation.test.ts`, `resourceStateAdapters.test.ts`, `TestPreviousVMFromViewKeepsLinkedAgentDiskOutOfProxmoxCarry`, `TestAgentDataAlwaysSendsStale`, `TestVMViewDiskFromLinkedAgentFollowsSelectedMetricSource`, and `TestResourceRegistry_ManualGuestAgentLinkDiskMetricFollowsProxmoxGuestFilesystems`. The registry test is a characterization test; it fails when the agent outranks Proxmox on guest disk. - Private mock stack at 1440 and 390, with agents linked into VMs through `POST /api/resources/{vm}/link`: - Live agent, no QEMU guest agent: before, "- Unavailable · Guest agent not running", with History deferred; after, the agent's bars, nothing deferred. - Silent agent: before, "- Unavailable"; after, bars plus "Last known" (phone: "Prior"), with History deferred on the agent message. - Stopped guest: Unavailable. - A guest with Proxmox filesystems: unchanged. - Machines row for a standalone agent released from a silent hold, watched without a reload: on main it kept "(stale)"; now it clears. - Bundle: WorkloadsSurface 61.93 kB gzip against a 70.23 kB cap. ## Known residuals - Mock mode's websocket state does not apply manual links to resources ingested as records (`ingestRecords` never runs `applyManualLinks`), so the browser pass ran the Proxmox page on REST data. In production this likely affects links to vSphere or TrueNAS resources; it is tracked separately. - A stale Proxmox source can lose the aggregate to a live agent while its own filesystem rows and `prev-` reason remain. That guest stays on the Proxmox rule and reads its live aggregate as last known, which is the same as main and errs toward caution. A guest with no platform read reason can likewise keep a retained aggregate after the agent withdraws its filesystems, also as on main. Both need aggregate provenance on the wire. |
||
|
rcourtman
|
15322e1573 |
Drop contract prose for the deleted problem-resource surfaces (#2706)
Contract prose still described surfaces that no longer exist: - `frontend-primitives.md` told new route-level copy to extend `problemResourcePresentation` (deleted in |
||
|
rcourtman
|
23bf99e1ba |
Keep alert settings a config save leaves out (#2689)
## Problem
Saving anything on Alerts > Thresholds turned flapping detection and
alert TTL cleanup off. The page's `PUT /api/alerts/config` body never
carries `flappingEnabled`, `maxAlertAgeDays`, `maxAcknowledgedAgeDays`,
`autoAcknowledgeAfterHours` or `customRules` (the page has no control
for any of them). `UpdateAlertConfig` decoded the body into a zero
`AlertConfig`, so every save wrote `false`/`0` for those fields, reset
the flapping tuning to defaults, dropped custom rules, and persisted
that. The `flapping`, `aggregation` and `ioNormalization` objects the
page sends are not `AlertConfig` fields and never enabled anything.
## Fix
- `PUT /api/alerts/config` now replaces only the top-level keys the body
carries. A key left out keeps its stored value. A sent key replaces its
setting whole, so `"overrides": {}` still clears overrides, and the
usual normalization still runs. A body that is not a JSON object
(including `null`) is a 400 and changes nothing.
- `alertconfig.ApplyAlertConfigUpdate` decodes the body on its own, so
sent keys get exact `encoding/json` semantics, including duplicates and
case-folded keys. It copies every unsent top-level field from a JSON
clone of the stored config.
- `Manager.ApplyConfigUpdate` reads, merges and applies under the
manager lock, so concurrent partial saves cannot revert each other. It
returns an owned snapshot, which the handler persists, so
`SaveAlertConfig`'s in-place normalization no longer writes into the
live config's maps. The handler runs one save at a time, so successful
saves reach `alerts.json` in the order they were applied.
- Stored configs are not rewritten; a value an earlier save zeroed stays
as it is.
## Verification
- Unit: `internal/alerts/config/update_test.go`; manager save path and
concurrent partial updates in
`internal/alerts/config_validation_test.go`; handler against a real
manager in `internal/api/alerting/alerts_test.go`. Each new regression
fails against main's code (overlay negative controls). The concurrency
test fails at round 0 against a non-atomic variant.
- `go test ./internal/alerts/... ./internal/api/alerting/` and the
`internal/api` alert endpoint, tenant and scope tests pass; the alert
tests pass under `-race`.
- Private stack (mock mode, Vite on main's frontend):
- On main, a node CPU change saved from the Thresholds page took
flapping and all three TTLs to false/0, in memory and in `alerts.json`.
- With this change, the same UI save kept them (true/7/1/24) and stored
the new threshold.
- The API repro (GET, drop the four keys, PUT) keeps them; explicit
`false`/`0` still applies; `null` and `[]` return 400.
- Five rounds of concurrent partial PUTs kept both changes in memory and
on disk.
## Follow-ups
- The thresholds page still sends the dead
`flapping`/`aggregation`/`ioNormalization` objects and hard-coded
`minimumDelta`/`suppressionWindow`/`hysteresisMargin`. A frontend
cleanup will follow once the open PRs touching
`alertsConfigurationModel.ts` land.
- `SaveAlertConfig` still normalizes caller-owned maps in place for
other callers (`ActivateAlerts`, startup paths), and the GET handler
encodes a shallow copy. That pre-existing race is tracked separately.
|
||
|
rcourtman
|
36d3f8c557 |
Judge pulse-relaxed guests by one threshold on poll and save (#2705)
Stacked on #2700 (which carries #2696 and #2692). Merge order is enforced by the stack: this change must not reach main before #2700's per-filesystem reevaluation (see "Why stacked" below). ## Problem `pulse-relaxed` lifts a guest's CPU, memory and disk triggers to a 95/92/95 floor. Only `CheckGuest` applied it. Config-save reevaluation rebuilt the guest from alert metadata without its tags (`guestSnapshotFromAlert`), so a save judged a relaxed guest's alert by its unrelaxed thresholds: - A relaxed guest with an unset memory threshold alerts at 92 while polling. On save that reads as no threshold, so the save resolved the alert and sent a recovery notification, and the next poll raised it again. - The floor also lifted an Off threshold (trigger 0, or -1 from the editors) like any low trigger, so tagging a guest relaxed turned an Off alert on, with no recovery band (`{95, 0}`). A save then re-judged it Off and resolved it, and the next poll raised it again. ## Change - The relaxed step moves into `effectiveAlertPolicyNoLock`'s guest branch, so `CheckGuest`, config-save reevaluation (including #2700's `resolveGuestAlertThresholdsNoLock`) and posture checks share one resolver. - `guestSnapshotFromAlert` reads the recorded `tags` back, as `[]string` live or `[]interface{}` after a restart, through a new `metadataStringsValue` reader. `storagePolicyAliases` now uses the same reader. - An Off threshold (trigger <= 0) stays Off under `pulse-relaxed`; an unset one still gets the floor, as the existing tests pin. The tag is documented (issue #863) as relaxing thresholds, so it must never make a guest more sensitive than its config. - `metricOptionsWithTags` drops the `tags` key from an open alert once its resource has no tags. Reevaluation now reads that key, so a removed `pulse-relaxed` left behind would judge the alert by the relaxed trigger while polling uses the guest's own, and the alert would flap. Notification tag routing reads the same key, so it also stops routing on removed tags. ## Why stacked A filesystem's own override replaces the relaxed guest disk threshold in `CheckGuest`. On main, reevaluation misses filesystem overrides (fixed by #2700). With this change alone, a save would judge a relaxed guest's filesystem alert by the 95 floor instead of, say, an 85 override, and flap it. `TestConfigSaveKeepsRelaxedGuestFilesystemOverride` fails on main plus this change and passes on this stack. ## Proof New tests in `internal/alerts/alerts_test.go` and `internal/alerts/unified_eval_test.go`. With `go test -overlay` mapping the six runtime files back to #2700's head, these fail: - `TestConfigSaveJudgesRelaxedGuestAlertByRelaxedThresholds` (the save resolved the alert) - `TestPulseRelaxedKeepsOffGuestThresholdOff` (a disk alert was raised with disk Off) - `TestApplyRelaxedGuestThresholds/off_thresholds_stay_off` (Off became `{95, 0}` / `{92, 0}`) - `TestCheckGuestDropsRemovedTagsFromOpenAlerts` (stale tags; with only the metadata assertion relaxed, the save resolves the alert) `go test ./internal/alerts/... ./internal/notifications/` and the monitoring alert tests pass. |
||
|
rcourtman
|
b7049ff778 |
Raise Proxmox disk temperature alerts under the per-type policy (#2680)
**Stacked on #2675** (TrueNAS disk temperature alerts by disk type). Merge that first; this branch contains its commit. ## Problem A Proxmox physical disk that only the node's own sensors report never raised a temperature alert: - `CheckDiskHealth` judges health and wearout only; - unified evaluation types TrueNAS physical disks, not PVE ones. So Physical Disks could read a disk **Running Hot** under the per-type thresholds while alerts stayed silent. Whether a disk alerted depended on whether a Pulse agent ran on the node: agent-covered disks alert through `CheckHost` at the same thresholds. #1347 shows users look for disk temperature thresholds in the Proxmox tabs. ## Change - **`CheckProxmoxDiskTemperature`** (`internal/alerts/proxmox_disk_temperature.go`) raises a `diskTemperature` metric alert under the disk's existing PVE alert reference. It is judged by the disk temperature policy: the disk type's `DiskTempByType` entry under the agent Disk Temp default, which switches it off for every type. The alert carries the disk's serial, WWN and `diskType`, like the health and wearout alerts. - **One alert per disk.** `mergeHostAgentSMARTIntoDisks` marks a disk that a still-reporting linked agent lists in its SMART report (`AgentSMARTReported`, internal, never serialized). That disk is the agent's: `CheckHost` alerts on it, and an open PVE alert closes with `moved_to_agent`. A lapsed agent owns nothing, so the node's own current readings keep alerting while its agent is silent. - **Holds and clears.** Only a reading collected this poll is judged (`diskinventory.TemperatureCollected`), so a retained value holds the alert. An excluded device, or a switched-off policy, closes it. - **Stability.** A new `proxmox-disk` alert delay defaults to 5s, so warnings wait the same 300s noisy-gauge window as agent and TrueNAS disk temperatures. Critical readings bypass it. - **Config save.** `reevaluateActiveAlertsLocked` re-judges the alert per disk type. Without this it would fall through to guest thresholds, which have no disk temperature, and resolve on every save. An alert without a recorded type uses the lowest per-type trigger. - **History.** `ProxmoxPhysicalDiskAlertIdentifiers` lists the new `<ref>::metric-threshold:diskTemperature` identifier, so resource history owns its rows by recorded hardware identity, as #2646 does for health and wearout. - **Copy.** The Agents "Disk temperature by type" card says it covers agent, TrueNAS and Proxmox disks. ## Evidence Private mock stack (8 PVE nodes) with a local-only overlay pinning PVE disk readings: | Disk | Physical Disks | Alert | |---|---|---| | pve1 NVMe 63°C | Healthy | none | | pve2 SATA 56°C | Running Hot | warning, trigger 55, after the 300s window | | pve3 NVMe 72°C | Running Hot | warning, trigger 70, after the 300s window | | pve3 SAS 63°C | Healthy | none | | pve3 SATA 66°C | Running Hot | critical at once | The alert card reads: "Samsung 870 EVO 1TB (/dev/nvme2) · Temperature · Temperature 66°C, above the 55°C alert level · Clears once it drops to 50°C or lower and stays there for 5 minutes." ## Tests - `internal/alerts/alerts_test.go`: - the policy (per type, raised trigger); - ownership with `moved_to_agent`, the hold without a reading, exclusion, the off switch; - the stability window; - re-judging on save, including the lowest-trigger hold; - history identifiers. - `internal/monitoring/physical_disk_roundtrip_test.go`: the marker for reporting agents only (not a lapsed one), the SMART fallback path, and that `checkPhysicalDiskAlerts` passes collected readings, ownership and exclusion. - `internal/unifiedresources/store_test.go`: a temperature alert row is owned by its disk and still found by a read by its reference. - `internal/models/deepcopy_test.go`: the marker survives cloning and stays out of JSON. - `ThresholdsTable.test.tsx`: the card copy. - Ten wiring points were mutation-checked (marker, lease condition, collected reading, ownership, exclusion, re-judge branch, lowest trigger, resolution, history identifier); each removal fails a test. ## Second opinion - Fixed: an offline agent's retained SMART rows kept ownership, so neither side alerted while the agent was silent. Ownership now needs a reporting lease. - Kept, documented in `alerts.md`: on the first poll after an agent is newly linked to a disk with an open PVE alert, both alerts can be open until the next disk poll. This is the same window as the node-to-agent usage alert handover. ## Not in this PR - Like the health and wearout alerts, a disk that leaves the inventory keeps its temperature alert until the 24-hour stale sweep. - Per-host Disk Temp overrides do not reach PVE-only disks, which have no agent host. |
||
|
rcourtman
|
b814ccdd81 |
Count down to future Patrol times instead of reading just now (#2702)
`formatRelativeTime` is past-only: a time ahead of now reads "just now"
(compact) or "0s ago". Three Patrol lines handed it future times, so
each read wrong for its whole duration:
- Attention detail, suppressed issue: "Returns to active attention just
now." (also for an issue marked reviewed, which expiry hands back to
Reviewed, not active attention)
- Findings, will-fix-later: "Reminding just now"
- Findings, snoozed: "snoozed until just now"
Supersedes #2613, rebuilt on main now that #2598 has landed (#2613 was
stacked on two early #2598 commits whose contract wording main has since
replaced).
## Change
- `formatTimeUntil` in `utils/format.ts`, beside `formatRelativeTime`:
counts down ("in 3h", "in 1d"), rounds to the nearest unit (a 24-hour
suppression set a moment ago reads "in 1d", not "in 23h"), and returns
`dueText` (default "now") once the time arrives. `formatRelativeTime`
stays past-only and its doc points future times at the new helper.
- The three lines use it with the shared relative-time clock
(`useRelativeTimeNow`), since none of those times change while the panel
stays open, and keep the exact time on hover.
- Suppression line: "Returns to active attention in 1d." for an open
issue, "Suppression ends in 1d. It stays reviewed." for one marked
reviewed, "Suppression has ended." once due. It takes its words from
`getAttentionSuppressionEndPresentation` (added on main by
|
||
|
rcourtman
|
0b6473affc |
Fill the table's visible width with phone drawers (#2703)
Phone and tablet table drawers capped their sticky content at
`calc(100vw - 3.5rem)`, a guess at the surrounding page chrome. On pages
with less chrome than that, every drawer stopped short of the table and
left an empty strip on the right: 27px at 390px, measured on the
Availability checks, Proxmox node and guest, Docker host and container,
Kubernetes node and Machines drawers. At tablet widths a table wider
than its shell got a drawer wider than the visible area (712px in
695px).
The table scroll shell already declares `container-type: inline-size`
(the responsive column priorities use it), so `InlineDetailTableRow` now
caps the default content shell at `100cqi`, the shell's own visible
width. Measured live: content equals the visible table width at 320,
390, 768 and 1000px, with no right gap; desktop keeps the static
uncapped layout. The availability drawer's hostname now fits instead of
truncating. Callers passing their own `contentClass` are unaffected.
Not changed here: `.table-fixed td { overflow: hidden }` makes the
sticky offset inert, so a sideways-scrolled table still moves its drawer
(same before and after). Long unwrapped identity chips that were already
clipped are clipped less now; wrapping them is a separate follow-up.
Proof: `InlineDetailTableRow.test.tsx` and the shared-primitive
guardrail pin the new cap and reject the old one; the guardrail already
pins the scroll shell's `container-type`. Browser receipt covers five
routes at 320 to 1440px, before and after.
|
||
|
rcourtman
|
3fb267333d |
Require POST for Assistant session mutation routes (#2670)
The session router dispatches `/api/ai/sessions/{id}/abort`,
`summarize`, `fork`, `undo` and `redo` by path alone, and none of those
handlers checked the HTTP method, so a GET or HEAD cancelled a running
turn, compacted or rewound a transcript, or persisted a fork.
- **Demo mode:** `DemoModeMiddleware` admits every GET/HEAD as a read,
so a public-demo session could mutate Assistant sessions despite the
read-only guard.
- **CSRF:** `CheckCSRF` skips GET/HEAD, and session cookies default to
SameSite=Lax, which browsers send on cross-site top-level GET
navigations. A link carrying a session ID could therefore trigger these
mutations for a signed-in user.
## Change
- `requireAssistantSessionMutationMethod` in
`internal/api/ai_handler.go`: abort, summarize, fork, undo, redo and
steer admit only POST and answer anything else with `405` + `Allow:
POST` before resolving the Assistant service. Steer already refused
non-POST but sent no `Allow` header; it now uses the same guard.
- The guard sits in the handlers, matching how chat, steer and the
approval approve/deny handlers already own their method contracts.
- `messages` (GET) and the unsupported `diff`/`revert`/`unrevert` routes
are unchanged. Session rename/delete and the collection route already
dispatch on the method.
- Callers: `frontend-modern/src/api/aiChat.ts` uses POST for all six;
pulse-mobile's abort call and its OTA compatibility probe use POST.
## Proof
- `TestAssistantSessionMutationsRequirePost`: GET/HEAD/PUT/DELETE get
405 + `Allow: POST` for every mutation, with no mutation call and no
`IsRunning` check reaching the service. POST still dispatches.
- `TestAssistantSessionReadSubresourcesKeepTheirMethods`: GET `messages`
returns 200 and GET `diff` returns 501, with no mutation either way.
- `TestDemoModeRouterRejectsAssistantSessionMutationsOverSafeMethods`:
drives `NewRouter(...).Handler()` with `DemoMode` on. GET/HEAD to every
mutation pass the demo guard and get 405, POST stays 403, nothing
mutates, and GET `messages` still reads.
- Negative controls via `go test -overlay`: on main's `ai_handler.go`,
every non-POST case fails and the demo test reports `GET abort status =
200`. With abort's guard moved after the `IsRunning` check, the ordering
assertion fails.
|
||
|
rcourtman
|
1ac600ec2b |
Judge filesystem alerts by their own threshold on config save (#2700)
Stacked on #2696, which is stacked on #2692. The unique commit is the
last one,
|
||
|
rcourtman
|
77d2119792 |
Remove unread Patrol summaryStats and its audit rule (#2690)
Stacked on #2686, which sits on #2682 and #2674; its own change is the top commit, |
||
|
rcourtman
|
1231392e8d |
Judge a linked agent's guest disks by that agent everywhere
PR 2669 made the guest drawer's filesystem rows follow a linked Pulse agent's freshness when its filesystems fill in for a VM that has no Proxmox guest filesystems. The workload table's disk cell and the drawer History still judged the same reading only by the Proxmox read reason. So one guest read as current in the drawer and as "Unavailable: Guest agent not running" in the table and History, and a silent agent's reading could not show as last known there. getWorkloadGuestDiskRead now owns one rule for the table cell, the drawer rows and History: current while the agent reports, last known once it is stale, unavailable once the guest is stopped. Proxmox's own guest filesystems keep their read reason. The rule also covers the aggregate disk metric, because Proxmox records no guest disk usage without guest filesystems, so the manual link merge keeps the agent's metric. Three paths broke that: - The poller built its previous VM from the merged metric, so the next poll carried the agent's disk forward as a Proxmox prev- read, and Proxmox priority then froze it over the agent's live value. Only Proxmox's own disk reading is carried now. - agent.stale was omitempty, and the browser merges agent facets field by field, so a resumed agent stayed "stopped reporting" until a reload. It is always sent now. - The same merge kept filesystems a native Proxmox VM facet or a native agent facet had withdrawn, so the rows did not switch source. |
||
|
rcourtman
|
0180b33431 |
Hide pprof routes on demo-mode instances
The /debug/pprof handlers are gated on RequireAdmin plus a settings:read scope that session logins skip. The public demo signs every visitor in as the configured admin, so any visitor could fetch goroutine and heap dumps and the process command line, or start CPU profiles and execution traces that load the whole process. DEMO_MODE already hides diagnostics, logs, admin users and manual discovery through the admin-operations policy; pprof was missing from it. The policy now has the demo guard answer /debug/pprof and every path below it with 404 for every method, ahead of route authentication. Instances outside demo mode keep the routes, and PULSE_PPROF_DISABLED still removes them everywhere. TestContract_DemoModeHidesPprofFromDemoAdminSession signs in as the demo admin through the full router, shows the same session still reads /debug/pprof/cmdline below the demo guard, and gets 404 for eight pprof paths through the served handler. On the previous policy GET and HEAD returned pprof output (200, or the root's 301 into the index) and POST got the generic demo 403. A policy test also fails if a literal route the router files register under /debug/, with or without a method prefix, escapes the policy for any common method. |
||
|
rcourtman
|
48c2d6cdce |
Stop Patrol fetching the correlation list nobody reads (#2677)
The Patrol page fetched `GET /api/ai/intelligence/correlations` on every mount, every 60 s poll, every return to the tab and on Retry, and the store's dashboard bundle fetched it again for FindingsPanel's refresh. Nothing has read the list since the Patrol supporting-context block was removed in 27ba85ddc6; the resource drawer reads per-resource correlations from the resource-intelligence payload. ## Change - `stores/aiIntelligence.ts`: remove the correlation signal, `correlations` getter, `correlationsSignal`, `loadCorrelations()`, its org-switch reset, and its entry in `loadDashboardData()`. - `features/patrol/usePatrolIntelligenceState.ts`: drop the read from the Retry path's background supporting reads. - `api/ai.ts`: remove `AIAPI.getCorrelations` (no other caller). The backend route stays as an API surface behind `ai:execute`. - Tests: the store bundle test now pins every AIAPI read the bundle makes; the manual-run test stalls the post-start refresh through a remaining read so it still proves "Starting" clears without waiting (negative controls: awaiting the refresh, or removing it, both fail the test). Dead correlation route stubs removed from six E2E specs. - Contract prose in api-contracts, patrol-intelligence, unified-resources, ai-runtime and frontend-primitives no longer says the Patrol page or store loads or renders correlations or recent changes. The remaining stale Patrol correlation paragraphs (patrol-intelligence ~L2135/~L2197, ai-runtime ~L6854) are rewritten by #2674; the two PRs split hunks by agreement and merge cleanly together. `CorrelationsResponse` stays for now: `patrolInvestigationContextModel.ts` still imports it until #2674 lands. It goes in a follow-up once both are on main. ## Evidence - Demo-mode stack, headless Chromium at 1440x900 and 390x844: correlation requests go from 1 to 0 per Patrol load, browser reload, 60 s poll and tab hide/show. Every other `/api/ai/*` read per phase is identical; no console errors. Re-run on the final base. - Production build: the route string is gone from the bundle; `check-bundle-size.mjs` passes with no chunk renamed (JS gzip -0.18 kB on the same base). - `tsc --noEmit` passes; vitest for src/stores, src/api, src/features/patrol, src/components/AI and the Patrol page test passes; contract, registry and status audits pass. |
||
|
rcourtman
|
71a723abb1 |
Read controller-member disk history at its written key (#2694)
## Problem A physical disk behind a RAID controller (target such as `megaraid,3`) with no usable serial or WWN had its SMART history written under one key and read under another, so its temperature and SMART charts stayed empty. - Host agent: writes under `HostSMARTDiskMetricID`, e.g. `host-pve:sdd@ctrl0/megaraid,3`. - Proxmox: writes under `PhysicalDiskMetricID`, which returns `ProxmoxPhysicalDiskSourceID`, e.g. `pve-node1--dev-sdx:sdx@/megaraid,0`. - Reader: `BuildMetricsTarget` passed those source IDs, which already name the member, to `PhysicalDiskMetaMetricID`. That function appended the topology again: `host-pve:sdd@ctrl0/megaraid,3:sdd@ctrl0/megaraid,3`. The same happened in a registry rehydrated from persisted resources. ## Fix `PhysicalDiskMetaMetricID` now follows the Proxmox writer. A fallback that already ends in the member's topology is returned unchanged. The topology is appended only to a fallback without it: - a Proxmox source ID from before members were scoped, which `PhysicalDiskMetricID` scopes the same way; - the canonical resource ID that `PhysicalDiskView.MetricResourceID` falls back to. TrueNAS disks carry no controller target and are unaffected. ## Existing history - **SMART:** no SMART writer used the doubled key, so samples already stored become readable without a migration. - **Disk I/O:** a linked node's agent with no SMART reading for a member, such as a standby row, files the device's I/O under the Proxmox disk's metrics target. That target was the doubled key. The I/O now lands beside the member's SMART history. Samples already stored under the doubled key are left to age out: nothing reads them, and they are the block path's counter filed under its one merged member. - Mock seeding keys these disks by `disk.ID`, the single-scoped key, so it already matched. ## Proof - `TestIdentitylessControllerMembersReadTheirWritersHistory` (`internal/monitoring/issue1595_collection_trust_test.go`) drives the real writers: `writeHostSMARTMetrics`, `writeHostPhysicalDiskIOMetrics` and the Proxmox `writeSMARTMetrics`. It then queries the store at `MetricsTarget` for five members in a live registry and in one rehydrated through a JSON round trip. The members are two agent members sharing a path, a lone agent member with I/O, a Proxmox member, and a standby member merged into its linked node's Proxmox row. Before the fix all ten targets read the doubled key. - `TestPhysicalDiskMetaMetricIDScopesAControllerMemberOnce` (`internal/unifiedresources/canonical_ids_types_test.go`) pins each fallback shape against its writer: agent, smartctl-labelled agent, Proxmox, legacy Proxmox, canonical resource ID, and a disk that is no controller member. Before the fix, three cases failed with the doubled key. ## Stacking This branch sits on #2685, which sits on #2679, because both edit `physical_disk_ids.go` and the proof file. It supersedes #2661, which carried the same reader fix but conflicted with this stack and with main. |
||
|
rcourtman
|
843daf7ae3 |
Remove dead Patrol assessment and save-failure handoffs (#2682)
Stacked on #2674 (branch `refactor/drop-patrol-context-summary`); this branch contains that commit. ## What Removes the two Patrol-to-Assistant handoff builders in `frontend-modern/src/features/patrol/patrolInvestigationContextModel.ts` that no production code calls: - `buildPatrolAssessmentAssistantHandoff`: its caller, the summary card's Discuss handler, went in |
||
|
rcourtman
|
2eef42c512 |
Keep Off alert defaults off after saving (#2684)
## Summary
Switching a metric Off in a **Global Defaults** row (Alerts >
Thresholds) did not stick. After Save and a reload the metric was back
On at its factory threshold and kept alerting. This affected every
section whose factory threshold is positive: Machines CPU, memory, disk
and disk temperature, Proxmox node temperature, PBS, Docker, Kubernetes,
TrueNAS, vSphere and storage.
The editors stage `-1` for Off, and the save payload passed it through
as `{trigger: -1}`. Backend config normalization reads a negative
default trigger as unset and restores the factory value. Guest and node
CPU, memory and disk escaped only because nothing normalizes them.
- `buildAlertsConfigurationPayload` now writes every Off default as
`{trigger: 0, clear: 0}`, the value the engine and normalization both
read as off. Per-resource overrides are serialized separately and keep
`-1`, which the override path stores as written. A `-1` already stored
on a guest or node default becomes `0` on the next save.
- The Global Defaults **Custom** badge now treats `-1` and `0` as the
same Off value. Without that, a fresh install showed Custom on
Kubernetes, TrueNAS Systems and vSphere, and VMs & Containers turned
Custom after any save, though nothing had changed. An unset factory
value stays distinct from Off: an unset TrueNAS disk temperature follows
Disk temperature by type.
- A row's custom-threshold marker still compares raw values. It drives
the row's revert control and override removal, so a stored `-1` override
must stay visible against an Off default. A test pins that.
- The alerts contract no longer claims Off is written as `-1`
everywhere.
## Verification
- New payload test: every section's Off default is written as `{0, 0}`
and reads back Off. It fails on `main` with `guestDefaults.cpu: expected
{ trigger: -1 }`.
- New backend test: a `{0, 0}` default stays off through `UpdateConfig`
in every section, and a host at 99% raises no CPU, memory or disk alert
when the agent defaults are off. Both fail when the defaults are written
as `-1`.
- Live on a private mock stack. On `main`, Proxmox CPU and Temp switched
Off saved as `{-1}` and `{80, 75}`, and after reload Temp read On. With
the fix:
- Machines CPU at 1% raised 8 alerts. Switching it Off stored `{0, 0}`
and resolved all 8.
- Proxmox Temp Off resolved the node temperature alert.
- Nothing re-fired over two minutes.
- Values stayed Off after a backend restart and a reload. Machines,
Proxmox, Kubernetes, TrueNAS and vSphere were each checked at desktop or
phone width.
- Custom badge on a fresh install: no Proxmox, Kubernetes, TrueNAS or
vSphere group reads Custom, and after an unrelated save only the group
actually changed does.
|
||
|
rcourtman
|
60527fa8ab |
Drop contract prose for the deleted problem-resource surfaces
frontend-primitives.md still told new route-level copy to extend problemResourcePresentation, deleted in |
||
|
rcourtman
|
e3a93efcee |
Judge pulse-relaxed guests by one threshold on poll and save
CheckGuest lifted a pulse-relaxed guest's CPU, memory and disk triggers
to the 95/92/95 floor itself, but config-save reevaluation rebuilt the
guest from alert metadata without its tags and judged the alert by the
unrelaxed thresholds. A relaxed guest whose memory threshold was unset
got the floor of 92 while polling, read as no threshold on save, and
the save resolved the alert with a recovery notification; the next poll
raised it again.
The floor also lifted an Off threshold (trigger 0, or -1 from the
editors) like any low trigger, so tagging a guest relaxed turned on an
alert its config switched off, with no recovery band at all ({95, 0}).
The tag is documented as relaxing thresholds; it must never make a
guest more sensitive than its config.
- Apply the relaxed step in effectiveAlertPolicyNoLock's guest branch,
so CheckGuest, reevaluation and any later resolver share it.
- Read the recorded tags back in guestSnapshotFromAlert ([]string live,
[]interface{} after a restart) through a shared metadata reader that
storagePolicyAliases now uses too.
- Leave an Off threshold off; an unset one still gets the floor.
- Drop the tags key from an open alert once its resource has no tags.
Reevaluation now reads it, and a removed pulse-relaxed left behind
would judge the alert by the relaxed trigger while polling uses the
guest's own.
This stacks on the per-filesystem reevaluation fix. Without it, a save
judges a filesystem alert by the guest's relaxed floor instead of the
filesystem's own lower override, and flaps it;
TestConfigSaveKeepsRelaxedGuestFilesystemOverride guards that.
The other new tests fail against the base branch's runtime files via
-overlay.
|
||
|
rcourtman
|
dace9295c6 |
Remove exports orphaned by the summary component deletions (#2671)
Stacked on #2662 (its commits are ancestors of this branch, so this PR shows them until #2662 merges). The May cleanup deleted InfrastructureSummary, StorageSummary, the shared summary table and the SummaryMetricCard, InteractiveSparkline, DensityMap and SummaryPanel primitives, but left the helpers only they called. This removes them: - `getMetricsChartKeyCandidatesFromResource` (utils/agentResources.ts) - `getNormalizedIdentityLookupVariants` and `getAgentLikeIdentityAliases` (utils/resourceIdentity.ts) - `getPBSStatusIndicator` (utils/status.ts) - `useSummaryContextualFocusState` (components/shared/contextualFocus.ts), plus `filterSummarySeriesByGroupScope`, `resolveSummaryCardInteractionState` and `SummaryCardInteractionState` that only it used - the deprecated `ChartsAPI.getInfrastructureCharts` delegate; its branch tests now exercise `getInfrastructureSummaryCharts` Proof of deadness: a TypeScript import/export graph from `src/index.tsx` finds no production importer of any removed symbol, and a repo-wide grep finds no remaining reference. The production build differs from its parent only in the shared chunk that carries ChartsAPI, which loses the delegate's static method (99 bytes); after normalising chunk hashes, SRI hashes and minified identifiers every other file is identical. Tests: `src/utils`, `src/components/shared`, `src/api`, `src/features`, `src/hooks`, `src/__tests__`, Workloads, Storage and Infrastructure suites pass (only the known Mac-only `agentInstallCommand` failures). Contracts: frontend-primitives, storage-recovery and performance-and-scalability still described the deleted summary cards, sparkline, density map, summary panel, sticky summary section and Jump to row button as live owners, and credited contextualFocus.ts with the removed hook's series filtering. Those clauses now name the live owners (summaryCardInteraction.ts behind summaryTableFocus.ts, the storage summary cache behind the pool Growth column, TooltipPortal for the mini sparkline and backup activity chart tooltips), become conditional on a summary strip returning, or go. api-contracts and unified-resources record the removed chart client and chart-key helper. Browser pass (embedded production build, demo mock stack, 1440x900 and 390x844): Proxmox node-to-guests reveal, guest row hover emphasis and inline detail reveal, storage Growth column, pool row hover and inline detail, pool-group hover preview, pin and Escape clear. Receipt: `frontend-modern/browser-verification/232e5c23490f8cd4.json`. |
||
|
rcourtman
|
864a088db6 |
Fill the table's visible width with phone drawers
Inline table drawers below the lg breakpoint capped their sticky content at calc(100vw - 3.5rem), a guess at the page chrome. Pages with less chrome than that left an empty strip beside every drawer (27px at 390px on the Availability checks, Proxmox, Docker, Kubernetes and Machines tables), and a table wider than its shell at tablet widths got a drawer wider than the visible area. The table scroll shell is already a size container, so the content now caps at 100cqi: exactly the table's visible width at every width, measured live at 320, 390, 768 and 1000px. Desktop keeps the static uncapped layout. |
||
|
rcourtman
|
e747f2e57c |
Count down to future Patrol times instead of reading just now
formatRelativeTime is past-only: any time ahead of now reads "just now"
(compact) or "0s ago", which is right for clock skew on an observed
time. Patrol handed it three future times, so each read wrong for its
whole duration: a suppressed issue said "Returns to active attention
just now.", a will-fix-later finding said "Reminding just now", and a
snoozed finding said "snoozed until just now".
formatTimeUntil in utils/format.ts is the future counterpart. It counts
down ("in 3h", "in 1d"), rounds to the nearest unit so a 24-hour
suppression set a moment ago reads "in 1d" rather than "in 23h", and
returns "now" (or the caller's dueText) once the time arrives. The three
Patrol lines use it with the shared relative-time clock, because none of
those times change while the panel stays open, and keep the exact time
on hover.
The suppression line names the state expiry restores. An open issue
reads "Returns to active attention in 1d."; one marked reviewed reads
"Suppression ends in 1d. It stays reviewed.", because expiry restores
Acknowledged, which is not active attention; a passed expiry reads
"Suppression has ended.". The countdown takes its words from
getAttentionSuppressionEndPresentation, which already words the
early-end button and the duration field by the same reviewed split, so
all three agree.
A reminder reads "Reminder overdue" once one overdue id list, sampled on
the shared clock, holds it. The Overdue commitments chip and filter
count the same list, so the label, chip and filter agree even for a row
that mounts between ticks (it reads "Reminder due" until the next tick),
and the list only notifies when its ids change, so a tick that moves no
deadline past now does not rebuild open rows. The reminder hover no
longer promises the finding returns only "if it is still tripping": the
hourly reminder sweep brings it back regardless. An ended snooze reads
"snooze ended" until the next findings re-read drops the snoozed status.
The existing countdowns did not fit: CountdownTimer runs its own
one-second interval and has no day unit (a 7-day suppression would read
"167h 59m"), and the replication Next sync helper is private to that
table with sync-specific overdue tones.
An audit of every formatRelativeTime and
formatPlatformTableRelativeTimeValue caller found no other future
timestamps; the autopilot and alert snooze expiries already show an
absolute time.
This rebuilds PR 2613 on main after PR 2598 landed. PR 2613 was stacked
on two early PR 2598 commits whose relative-age contract wording main
has since replaced (every read of the shared clock now returns the wall
clock), so the future-time note sits under main's wording and before the
read-time age exception, and that exception now opens "The relative-age
rule" so it does not read as an exception to the future-time note.
|
||
|
rcourtman
|
539eadfab2 |
Judge filesystem alerts by their own threshold on config save
A config save re-judges every active alert. Agent filesystem alerts were re-judged by the host disk threshold, ignoring the filesystem's own override and the per-type DiskFillByType threshold CheckHost uses. Guest filesystem alerts were resolved by their "<guestID>-disk-<key>" resource ID, which matches no guest-disk override and names no guest, so the guest's own override was missed too. A filesystem override of 70 raised an alert at 80 under a default of 90; saving any alert setting resolved it and sent a recovery, and the next poll raised it again as a new alert. CheckHost and CheckGuest now resolve a filesystem's threshold through hostDiskUsageThresholdNoLock and guestDiskUsageThresholdNoLock, and reevaluation calls the same helpers: resolveHostAlertThresholdsNoLock for agent filesystem alerts, and a new resolveGuestAlertThresholdsNoLock that resolves a guest filesystem alert's guest by the ID parseGuestAlertIdentity reads from its resource ID. The helpers return override and per-type thresholds as owned copies, so evaluation still never normalizes the live config. TestConfigSaveJudgesFilesystemAlertsByTheirOwnThreshold covers an agent filesystem override, an NVMe fill threshold, a guest filesystem override and a guest override. With host.go, guest.go and config_runtime.go overlaid by their origin/main or #2696 copies, all four fail: the unchanged save resolves the alert. Stacked on #2696. |
||
|
rcourtman
|
aa7053a07e | Merge remote-tracking branch 'origin/main' into fix/alert-config-keep-unsent-fields | ||
|
rcourtman
|
55e865ecb8 |
Move the config save regressions to the end of their file
main added TestUpdateConfigKeepsLowTriggerClearBelowTrigger (#2678) at the spot these tests were inserted, with its own import block, so the PR no longer merged. The tests now sit at the end of the file and the concurrency test waits on a channel instead of sync.WaitGroup, leaving the import line as main has it. |
||
|
rcourtman
|
7eb32f3181 |
Keep a low alert default's clear below its trigger (#2678)
## Problem
Setting an alert global default to a positive trigger of 5 or less, for
example Machines CPU at 1% in Alerts > Thresholds, stored
`agentDefaults.cpu` as `{trigger: 1, clear: 75}`, a clear far above its
trigger.
The thresholds page sends `clear = max(0, trigger - 5)`, which is 0 for
those triggers. `NormalizeAgentDefaults` derived `trigger - 5`, found it
non-positive, and fell back to the factory clear (75/80/85).
`ValidateHysteresisThresholds` never checked agent CPU, memory or disk,
so the pair survived. PBS CPU/memory, node temperature and
`internal/config/persistence.go` had the same fallback. On the save
path, validation repaired PBS and node temperature, but
`LoadAlertConfig` still turned a stored node temperature clear of 0 into
75; at startup `UpdateConfig` repaired that in memory.
Per-type `DiskFillByType` / `DiskTempByType` entries were worse: any
clear ≤0 reset the whole pair. A SATA disk temperature trigger of 4 from
the thresholds page was stored as 55, and an API-written fill entry of
`{88, 0}` became `{90, 85}`.
## What a user saw
- **Firing and resolving did not change for these pairs.**
`buildCanonicalMetricSpec` drops a clear at or above the trigger, and
the reducer then clears at the trigger. That held both before and after
this change. There was no flapping and no alert that never re-armed.
- `/api/alerts/config` and `alerts.json` served the factory clear above
the trigger.
- Firing agent alerts carried it as `metadata.clearThreshold`: 75 on a
1% CPU alert, and 85 on a disk alert still firing at 81.8%.
- A per-type disk trigger of 1–5 was silently replaced by the type
default.
- Machines metric bars: `resolveThreshold` in `metricThresholds.ts`
clamps the warning level to `min(clear, trigger)`, so a 1–5% agent
trigger had no warning band. Readings below the trigger, including 0,
now show amber instead of green, which matches guest, node, Docker and
Kubernetes bars at the same setting. This one is read from the code; I
did not exercise it in a browser.
## Fix
- PBS, node temperature, agent and per-type disk defaults now use
`normalizeThresholdPointer`, the normalizer Kubernetes, TrueNAS and
vSphere already use. A missing clear sits five points under the trigger,
floored at 0. A clear at or above the trigger is repaired. Zero still
means off for global defaults, and nil or negative still takes the
factory default. For the canonical per-type keys (the types the page
edits), a non-positive trigger still resets to the type default, because
the page cannot represent a per-type Off; other per-type keys are stored
as written, as before.
- `internal/config/persistence.go` drops its duplicate
agent/node-temperature and storage helpers and calls the same normalizer
through a new `NormalizeHysteresisThreshold`. A config saved before this
fix loads repaired.
- Agent CPU/memory/disk/disk temperature are added to
`ValidateHysteresisThresholds`, as every other family already is.
- One deliberate behaviour change for invalid pairs above the margin: an
API-written `{50, 80}` is repaired to `{50, 45}`, so it now has a real
recovery band where it used to clear at the trigger. This matches what
`EnsureValidHysteresis` already does for every other family.
- An alerts contract clause records the rule.
## Verification
- `internal/alerts/config/normalize_low_trigger_clear_test.go` runs
every default family and both per-type maps through the `UpdateConfig`
normalizer chain.
- `TestUpdateConfigKeepsLowTriggerClearBelowTrigger` pins the saved
config and a firing alert's `clearThreshold`.
- `persistence_test.go` checks the written `alerts.json` itself as well
as load repair.
- All new tests fail against the previous normalizers. Coverage cases
that pinned the old fallback (including one labelled "suspected source
bug") are updated.
- `go test ./internal/alerts/... ./internal/config/ ./internal/ai/`
passes.
- Live, on private mock stacks (baseline binary vs this branch): I saved
the payload the thresholds page sends, then read back
`/api/alerts/config`.
- Baseline: agent `{1,75}`, `{3,80}`, `{5,85}`; `diskTempByType.sata`
`{55,50}` for a 4°C trigger; firing agent alerts reported
`clearThreshold` 75/80/85.
- This branch: every pair `{t, 0}`, `diskTempByType.sata` `{4, 0}`, and
alerts reported 1/3/5.
- After a restart, this branch also repaired a hand-written
`alerts.json` holding `{1,75}` and node temperature `{2,75}`.
|
||
|
rcourtman
|
71a470129f |
Stop alert evaluation writing live thresholds outside the lock
Host and guest evaluation filled in the Clear of a per-disk override's threshold after dropping m.mu, writing through the live config's pointer. GetConfig clones that pointer under the read lock, which does not exclude the writer, and the saved override silently took whatever the last poll filled in. Docker evaluation pointed into m.config.DockerDefaults, and read the restart-loop and memory-limit settings, with no lock held while UpdateConfig replaced m.config. Clone the override's Disk threshold under m.mu before normalizing it, resolve Docker container thresholds from cloned defaults plus the override under the lock, and read the restart-loop and memory-limit settings under it. Drop getThresholdForMetricFromConfig: nothing has called it since the v6 split, and it normalized in place. TestAlertEvaluationLeavesTheLiveConfigToTheLock runs host, guest and Docker evaluation against GetConfig() and UpdateConfig(GetConfig()) under -race. With host.go, guest.go, docker.go and metric_runtime.go overlaid by their origin/main copies it fails: the live override's clear reads 85, and -race reports every site above. Stacked on #2692, which made GetConfig return an owned copy. |
||
|
rcourtman
|
5c55197900 |
Remove test-only Patrol context summary helpers (#2674)
Removes two exports from
`frontend-modern/src/features/patrol/patrolInvestigationContextModel.ts`
that only tests still called, and corrects the contract prose that
described the removed Patrol supporting-context surface as live.
## Why
`buildPatrolInvestigationContextSummary` (with its input type) and
`selectPatrolSupportingRecentChanges` lost their last production caller
in
|
||
|
rcourtman
|
a543907130 |
Type the Patrol run test's summary mock restore
The manual-run test restored the stalled loadIntelligenceSummary mock with mockResolvedValue(undefined), which the frontend type-check rejects because the store method resolves IntelligenceSummary | null. Restore it with null instead; the test behaviour is unchanged. |
||
|
rcourtman
|
ef45926b12 | Merge commit 'cd4e564fbf' into fix/proxmox-disk-temperature-alerts | ||
|
rcourtman
|
cd4e564fbf | Merge remote-tracking branch 'origin/main' into fix/disk-heat-remaining-judges | ||
|
rcourtman
|
b1fbee1338 |
Move TrueNAS re-judge tests clear of the cleanup tests
#2672 added its node cleanup tests right after TestCleanupAlertsForNodes, where this branch had added its TrueNAS disk temperature re-judge tests, so the branch no longer merged with main. Place the TrueNAS tests just before TestCleanupAlertsForNodes instead; no test changes. |
||
|
rcourtman
|
409c5482ee |
Read controller-member disk history at its written key
A physical disk behind a RAID controller (target such as megaraid,3) with no usable serial or WWN had its SMART history written under one key and read under another, so its temperature and SMART charts stayed empty. The host agent writes under HostSMARTDiskMetricID and Proxmox under PhysicalDiskMetricID, and both keys already scope the member (host-pve:sdd@ctrl0/megaraid,3). BuildMetricsTarget passed that source ID to PhysicalDiskMetaMetricID, which appended the topology again (host-pve:sdd@ctrl0/megaraid,3:sdd@ctrl0/megaraid,3). The reader now does what the Proxmox writer does: a fallback that already names the member is returned unchanged, and only a fallback without the topology gets it appended. That covers a Proxmox source ID from before members were scoped and the canonical resource ID a view falls back to. TrueNAS disks carry no controller target and are unaffected. SMART history needs no migration: no SMART writer used the doubled key, so samples already stored become readable. A linked node's agent with no SMART reading for a member, such as a standby row, files the device's I/O under the Proxmox disk's metrics target, which was the doubled key. That I/O now lands beside the member's SMART history, and samples already stored under the doubled key are left to age out. This carries forward the fix from PR 2661, whose branch conflicts with the disk ID stack it now sits on, and proves it against the real SMART and I/O writers in the live and rehydrated registries. |
||
|
rcourtman
|
ad1e34277d |
Stop alert config saves writing into the live config
SaveAlertConfig normalized the config it was handed in place, and its
callers hand it alerts.Manager.GetConfig(), a shallow copy whose maps
and threshold pointers were the running config's own. Alert activation
and the alert identity migration therefore wrote the live delay map and
threshold pointers outside the manager lock while evaluation read them.
The shallow copy also let UpdateConfig, which normalizes the override
map it is handed in place, rewrite the live overrides while
GET /api/alerts/config encoded another snapshot. A concurrent map read
or iteration plus a write is a fatal Go runtime error, so these paths
could kill Pulse.
AlertConfig.Clone is a typed deep copy. GetConfig returns one, so
callers own what they encode, persist and edit, and SaveAlertConfig
clones before normalizing, because a config just applied with
UpdateConfig is the live config. GetConfig has no per-resource caller
(once per poll cycle, backup check, request or Patrol run); a clone
with 1,000 overrides takes about 0.3 ms. A test fills every field and
fails when a new map, slice or pointer is not copied.
Persistence also stored and loaded an explicit 0 type-level alert delay
("no delay") as the 5-second factory delay, which the noisy-gauge rule
stretches to 300 seconds for memory and temperature, while the manager
keeps 0. The shared map write was what made the running manager agree
with the file; without it, memory would say 0 and disk 5. Persistence
now keeps 0 on save and load like NormalizeTimeThresholds, and only a
missing or negative delay takes the default. Pulse saves never wrote a
0, and the first v6 start re-saves alerts.json, so only a newly sent or
hand-edited 0 changes meaning.
On main, a reproducer taking the activation, migration and GET steps
against live evaluation died with "fatal error: concurrent map
iteration and map write" in 5 of 5 plain runs and reported 16 races
under -race. It passes with this change.
|
||
|
rcourtman
|
22b8b995ca |
Read the shared wall clock on Alerts overview cards (#2673)
The overview hook kept its own minute signal and handed its value to each card as "now". A card that mounted between ticks measured from up to a minute earlier: an alert raised 70 seconds ago read "this minute", a reading already past the 10-minute stale cut-off led the card as the live one, a just-raised alert stayed out of Triggered (24h) until the next tick, and the stale flip and 24h drop landed up to a minute late. The card now reads useRelativeTimeNow for its started age and the live reading's stale cut-off, and the hook reads it for the 24h count, so they measure from the wall clock and move with the shared 30-second tick. The minute interval stays only for the delivery diagnoses refresh, a server read that should not double its rate. |
||
|
rcourtman
|
2d6a2937fc |
Keep platform metric alerts through Proxmox node cleanup (#2672)
`CleanupAlertsForNodes` runs on every Proxmox node poll
(`monitor_pve_storage.go`) and on every mock alert tick. It removes
alerts whose `Node` is not a current Proxmox node or PBS instance, and
keeps the types `shouldPreserveAlertOutsideNodeCleanup` lists.
Kubernetes, TrueNAS and vSphere metric alerts carry a cluster or
platform host as `Node`, and a display label (`TrueNAS Disk`,
`Kubernetes Pod`, `vSphere VM`) in `metadata.resourceType`. The preserve
list only had the bare words `truenas`, `kubernetes`, `k8s` and
`vmware`, so none of these labels matched. In an estate with Proxmox
plus one of those platforms:
- every Proxmox poll deleted these alerts without resolving them;
- their reducer incident stayed firing, so the next unified evaluation
re-created each one and handed it to the alert callback again as a new
alert.
## Change
`shouldPreserveAlertOutsideNodeCleanup` keeps an alert that carries
`metadata.platformType` (set on every unified Kubernetes/TrueNAS/vSphere
metric alert), or whose label maps through `CanonicalAlertResourceType`
to one of those platform types. The label path covers alerts persisted
before `platformType` existed. A guest alert whose Proxmox node left the
estate is still removed.
## Evidence
- Unit probe on main: a TrueNAS disk temperature alert, a Kubernetes pod
CPU alert and a vSphere VM CPU alert were all deleted by
`CleanupAlertsForNodes({"pve1": true})`. Over three
cleanup-then-evaluate cycles the TrueNAS alert went to the alert
callback 3 times; with this change, 0 times and it stays active.
- Private mock stack: TrueNAS disk temperature alerts kept vanishing
between mock alert ticks, while the disks kept their readings in
`/api/resources`. With this change they stay open.
## Tests
`TestCleanupAlertsForNodesKeepsPlatformMetricAlerts` (in
`internal/alerts/alerts_test.go`) raises a TrueNAS disk, a TrueNAS pool,
a Kubernetes pod and a vSphere VM alert, then runs the cleanup three
ways: as raised, with only `platformType` (label unrecognised), and with
only the label (`platformType` removed).
`TestCleanupAlertsForNodesStillRemovesMissingProxmoxNodeAlerts` keeps
the removal of a guest alert on a removed Proxmox node. Each preserve
condition was mutation-checked: removing either one fails the first
test.
|
||
|
rcourtman
|
e30b44ea8f |
Read a silent in-guest agent's guest drawer readings as last known (#2669)
A Pulse agent linked into a VM keeps its last report on the VM resource after it stops reporting: the registry copies the agent's whole facet, `agent.stale` included, onto the guest. The guest drawer never read that flag. A quiet agent's Guest RAID kept its live colours and "Rebuild: 42% · 118M/sec", its network rows and version read as current, and nothing said the agent had gone. ## Change - `useWorkloads` maps `agent.stale` onto the workload as `agentStale`, and sets `disksFromAgent` when Proxmox reported no guest filesystems and the agent's own `disks` filled in. Filesystem rows are the only agent reading with a second source. - When the agent is stale, the guest drawer: - passes `HOST_AGENT_STOPPED_REPORTING_REASON` to `RaidCard` (`lastKnownReason`, #2652); - puts a status line over muted Network rows; - marks the Pulse Agent row "· stopped reporting"; - reads agent-supplied filesystem rows through the existing `last-known` disk read state. - Agent-supplied filesystem rows now follow the agent's freshness instead of the Proxmox guest-read reason. Before, a VM with a **live** linked agent but no qemu-guest-agent showed the agent's filesystems as "Usage unavailable · ?/328 GB" with advice to install qemu-guest-agent. A stopped guest still reads those rows as unavailable, whether its status or the Proxmox `vm-stopped` reason says so. Proxmox's own guest filesystems keep their read state. ## Verification - Private mock stack built from `main` plus this change. Two host agents were held past their reporting lease and two left live, then each was linked into a VM through `POST /api/resources/{vm}/link`. The two silent VMs serve `agent.stale: true` with the agent's raid, disks and interfaces; the two live ones carry no stale flag. - Playwright at 1440 and 390: - silent VMs read as described above; - the live VM with agent-only filesystems shows usage with bars; - the live control is unchanged; - no horizontal overflow. - New tests in `GuestDrawer.test.tsx` cover: - silent agent; - live agent with Proxmox agent-not-running; - Proxmox filesystems under a silent agent; - stopped guest, with and without the reason; - stale flip in both directions on an open drawer. The new tests fail against the old drawer. `useWorkloads.test.ts` asserts source selection with distinct mountpoints. - Frontend suites over Workloads, hooks, utils, shared, platformPage and Infrastructure pass, apart from the Mac-only PTY tests in `agentInstallCommand.test.ts`. - WorkloadsSurface grows by 0.22 kB gzip (64.85 kB against a ~71.9 kB cap), and the chunk set is unchanged. ## Not covered - The workload table's disk cell and the drawer History still read the aggregate disk metric with the Proxmox reason. - The agent's OS, its addresses in the identity rows, and the lazily loaded Physical Disks card's health and wearout carry no stale cue yet. |
||
|
rcourtman
|
6d8dce2cae |
Remove unread Patrol summaryStats and its audit rule
usePatrolIntelligenceState() still returned summaryStats, a closure that
counted critical, warning and active Patrol findings plus fixed outcomes.
Its only reader was PatrolIntelligenceSummary.tsx, deleted in
|
||
|
rcourtman
|
f7ac26b573 |
Keep alert settings a config save leaves out
The Thresholds page builds its PUT /api/alerts/config body from the settings it edits. It has no control for flapping detection, alert TTL cleanup or custom rules, so it never sends flappingEnabled, maxAlertAgeDays, maxAcknowledgedAgeDays, autoAcknowledgeAfterHours or customRules. The handler decoded the body into a zero AlertConfig, so every thresholds save turned flapping detection and TTL cleanup off, reset the flapping tuning, dropped custom rules, and persisted that. The page's flapping, aggregation and ioNormalization objects are not AlertConfig fields and never enabled anything. The handler now hands the body to Manager.ApplyConfigUpdate, which reads the stored config, merges and applies it under the manager lock, so two concurrent partial saves cannot revert each other. The body is decoded on its own, so the keys it carries mean exactly what encoding/json makes of them; every top-level field it left out is copied from a JSON round trip of the stored config, sharing no maps with it. ApplyConfigUpdate returns a cloned snapshot, which the handler persists, because SaveAlertConfig normalizes the config it is handed in place and used to write into the live config's maps. Saves run one at a time so alerts.json ends with the last applied config. A body that is not a JSON object, null included, is a 400. Stored configs are not rewritten, so a value an earlier save zeroed stays as it is. On a private stack, a node CPU change saved from the Thresholds page zeroed all four fields in memory and in alerts.json on main; with this change the same save kept them and stored the new threshold. |
||
|
rcourtman
|
421d1e5423 |
Remove test-only Patrol summary presenters
The Patrol summary card was deleted in
|
||
|
rcourtman
|
58347f0c4f |
Chart history for disks named by their Unraid serial
A SMART row that reports no serial, smartctl's standby row among them, takes the serial the host's Unraid inventory reports for the same disk: the adapter shows it and the disk's metrics target reads it. The SMART and disk I/O writers keyed the row on its own WWN or host/device key instead, so the chart read a key nothing wrote and the disk's temperature and I/O history never appeared. The I/O writer did the same for an Unraid disk with no SMART reading. HostSMARTDiskMetricID now takes the serial through the adapter's own rule (hostSMARTDiskSerial over matchUnraidDisk), and the I/O writer keys a device without a SMART reading by its Unraid serial before the Proxmox and host/device fallbacks. Source IDs keep the row's own identity, so registry keys and rehydration are unchanged. That rule matches by device path alone, so it now skips a path several SMART rows share. Controller members behind one block device would otherwise all take the device's serial and merge into one disk, and the I/O writer's guard against filing an aggregate counter under a member would stop working. That guard now compares every row on the path by its own identity, standby rows and legacy "sdc [sat]" labels included, so an aggregate counter never lands under one member. The I/O writer also refuses a device that several Unraid rows name. Samples already written under the old keys stay where they are. The chart did not read them while the Unraid row named the disk, and a device-keyed series can hold another disk's samples once device letters move, so they are not folded into the serial's history. |
||
|
rcourtman
|
9bf72eb407 | Merge remote-tracking branch 'origin/main' into fix/global-default-off | ||
|
rcourtman
|
564ea41a6e |
Keep Off alert defaults off after saving
Switching a Global Defaults metric Off on the thresholds page staged -1, and the save payload passed it through as trigger -1. Alert config normalization reads a negative default trigger as unset and restores the factory threshold, so every Off default with a positive factory value came back On after save and reload: agent CPU, memory, disk and disk temperature, Proxmox node temperature, PBS, Docker, Kubernetes, TrueNAS, vSphere and storage. Guest and node CPU, memory and disk escaped only because nothing normalizes them. The payload builder now writes every Off default as trigger 0, which the engine and normalization both read as off. Overrides keep -1, which the override path stores as written. A -1 already stored on a guest or node default becomes 0 on the next save. With -1 and 0 both live Off values, the Global Defaults Custom badge compares Off values as equal, so the factory -1 I/O defaults no longer read Custom against a saved 0 and tell users they changed something they had not. A row's custom marker keeps comparing raw values: it drives the row's revert control and override removal, so a stored -1 override must stay visible against an Off default. Contract-Neutral: Global Defaults Off now persists; the Off wire value and Custom badge rule are recorded in the alerts contract, and frontend-primitives gains no rule |
||
|
rcourtman
|
7432995fe8 |
Stop websocket upgrade headers exempting demo-mode writes (#2666)
`DemoModeMiddleware` let any request carrying `Upgrade: websocket`
through, whatever its method or path. On a `DEMO_MODE` instance a
`POST`/`PUT`/`PATCH`/`DELETE` with that header reached its handler, and
the read-only guard did nothing.
## How exposed the public demo was
- The public demo (`demo.pulserelay.pro`, v6.5.0) runs `DEMO_MODE=true`
with `PULSE_AUTH_USER=demo`, and its login page signs visitors in as
demo/demo. No proxy auth is configured. Caddy is a plain `reverse_proxy`
with no method filtering, and it forwards `Connection: Upgrade` and
`Upgrade: websocket` unchanged.
- That session is the configured admin.
`sessionUserHasSettingsAdminPrivileges` matches `cfg.AuthUser`, so
`RequireAdmin` accepts it, and session requests skip token scope checks.
Nothing in the handlers checks demo mode on its own, so this middleware
was the only write guard.
- Anyone using the public credentials could therefore make any admin
write by adding the two headers. That covers switching the demo out of
mock mode (`POST /api/system/mock-mode`), changing system settings,
creating organisations, importing config, changing the password and
adding notification targets. A CSRF token was needed, but the attacker
gets one from their own login.
- On the old code, the new contract test reproduces this end to end. A
demo/demo session's mock-mode POST with upgrade headers returned 200 and
turned mock mode off.
- I found no sign it was used. Over the last 30 days the demo journal
shows every "mock mode disabled" line paired with a process restart (new
PID). Config file mtimes match deploys. 5,782 ordinary writes were
blocked over the same period. Caddy keeps no access log for the demo, so
this is the absence of evidence, not proof.
## Change
The exemption is deleted rather than narrowed. RFC 6455 handshakes are
`GET` requests, gorilla's `Upgrader` refuses any other method, and `GET`
already passes the read-only branch above it. A narrowed `GET`-only
exemption would never run. `/ws` and `/api/agent/ws` keep connecting,
and the request method alone decides.
## Tests
- `TestContract_DemoModeUpgradeHeadersDoNotExemptWrites` (registered
api-contracts proof) runs against the full router. It signs in as the
configured demo admin and checks that the session passes `RequireAdmin`
for a read. A mock-mode POST carrying `Connection: Upgrade` and
`Upgrade: websocket` now gets the demo 403 and leaves mock mode on, and
a real `/ws` handshake from the same session still upgrades.
- `demo_middleware_test.go`: the table row that asserted the bypass now
expects 403. POST/PUT/PATCH/DELETE with handshake headers to mock-mode,
settings, orgs, `/ws` and `/api/agent/ws` all get 403. Real gorilla
handshakes to `/ws` and `/api/agent/ws` through the middleware still
upgrade and echo.
- Negative control: run against the old middleware via `go test
-overlay`, the new contract test and the upgrade-header test fail, and
the handshake test passes on both versions.
## Not changed here
- `auth.go` `isWebSocketUpgrade` and `configapi`
`explicitAPITokenFromRequest` still accept a `?token=` query credential
on any request with `Upgrade: websocket`. A valid, scoped token is still
required, so no access is gained, but the credential can travel in a
URL. The `ErrorHandler` skip in `middleware.go` only bypasses recovery
and metrics.
- Older and separate:
`/api/ai/sessions/{id}/{abort,summarize,fork,undo,redo}` change state on
`GET`, which the demo guard admits as reads. Tracked separately.
|
||
|
rcourtman
|
cb5ffc2efe |
Remove dead Patrol assessment and save-failure handoffs
buildPatrolAssessmentAssistantHandoff lost its last caller when the Patrol summary card's Discuss handler was removed in |
||
|
rcourtman
|
b9c2a90ee7 |
Raise Proxmox disk temperature alerts under the per-type policy
A Proxmox disk that only the node's sensors report never raised a temperature alert: CheckDiskHealth judges health and wearout only, and unified evaluation covers TrueNAS disks, not PVE ones. Physical Disks could read the disk Running Hot under the per-type thresholds while alerts stayed silent, and whether a disk alerted depended on whether a Pulse agent happened to run on the node. The PVE disk poller now hands each disk's current reading to CheckProxmoxDiskTemperature, which judges it by the same disk temperature policy as agent and TrueNAS disks (DiskTempByType under the agent Disk Temp default) under the disk's existing PVE alert reference. A disk a still-reporting linked agent lists in its SMART report is the agent's: CheckHost alerts on it and the PVE alert closes as moved to the agent, so no disk alerts twice. A retained reading holds the alert, an excluded device or a switched-off policy closes it, warnings wait the same 300s stability window as other disk temperatures, a config save re-judges the alert per disk type, and resource history owns its rows by the disk's recorded identity like the health and wearout alerts. |
||
|
rcourtman
|
75263ea589 |
Refuse a seeded disk slot mapping when the disk was replaced (#2667)
`ingestRecord` reuses an existing source-key mapping before any identity matching. For physical disks it checked only the type and parent scope. Proxmox keys a disk by its slot (`ProxmoxPhysicalDiskSourceID`, e.g. `pve1-node1--dev-sdb`), so in a registry seeded from persisted unified resources, a disk swapped into the slot mapped straight back to the previous disk's canonical resource. Where an agent report had been merged in, `mergeInto` also kept the old disk's serial, WWN, temperature and FAILED health. This was found while reviewing #2658, and its contract note said the seeded mapping was not covered. **Change.** A Proxmox disk observation now refuses the mapping when the two disks carry hardware identities naming different disks (`diskinventory.HardwareIdentityConflict` from #2658): - Only fields reported on both sides count. A missing serial or WWN, including Proxmox's literal `unknown`, keeps the mapping. - Differing serials do not count for SAS, because Proxmox may report the SAS address there (#1595). - They also do not count when the mapped disk was merged with an agent report and Proxmox did not read the drive's own serial. This is the same rule monitoring's `hostAgentSMARTSerialComparable` applies: NVMe on both sides, or agent SATA under Proxmox vendor `ATA`, and never a Proxmox `usb` disk. - The refused key falls through to the linked-disk join, `findMatch` and `chooseNewID`, but none of them may merge it back into the refused disk. An ID landing on it is replaced with the key's source-specific ID. **Reachability.** No current caller replays Proxmox disk records over a seeded mapping. The monitor rebuilds a fresh registry each poll. The resource API, host-continuity and Patrol registries seeded from unified resources do not replay Proxmox disk records. This closes the registry contract rather than a reported symptom. **Proof.** `TestRegistrySeededSlotMappingRefusesReplacedDisk` (in `internal/monitoring/issue1595_collection_trust_test.go`) builds the previous disk through `IngestSnapshot` with fixtures shaped like `monitor_pve`'s output, rehydrates with `IngestResources(List())`, polls Proxmox, then rehydrates a second time. - 5 replacement cases: SATA with and without a new WWN, SAS, a USB-enclosure swap ingested Proxmox-first, and a disk Proxmox alone reports. - 8 same-disk controls: re-ingested, no serial, a SAS address serial, SAS without a WWN, a USB bridge serial, a bridge serial under vendor `ATA`, a SCSI designator, and a SCSI designator from an agent without collection status. Negative controls via `go test -overlay` each fail only their intended cases: the whole fix removed, the refused-disk exclusion removed, serial comparability always true or always false, the Proxmox-only branch removed, a collection-status provenance rule instead, and the USB veto removed. The combined tree of main, #2660 and this change also passes `internal/unifiedresources` and the monitoring disk tests. **Residuals (in the unified-resources contract).** - Drives that share a serial they do not own, such as a USB bridge's, still read as one disk, as they do in identity matching. - The refused disk keeps its Proxmox facet until a rebuild, so the slot's PVE alert reference resolves as ambiguous rather than to either disk. |
||
|
rcourtman
|
080ace2e97 |
Show the outdated sensor setup notice on every Proxmox tab (#2668)
Since the per-tab Proxmox queries landed, only the Storage tab loads `physical_disk` rows. So the outdated sensor setup notice, which judged nodes from the active tab's disk rows, never showed on Overview. The registry now derives the verdict on each Proxmox node (`proxmox.sensorSetupOutdated`, an explicit true or false on every node) right after child counts resolve each disk's canonical parent. A node is flagged when its temperature payload is available and in the legacy `sensors -j` format and a SATA/SAS disk under it has no reading collected now (`diskinventory.TemperatureCollected`). The page lists flagged nodes from the rows Overview, Storage, Replication and Backups already load, so no tab hydrates disks for it. Rule fixes that came with the move: - Proxmox's disk inventory types SATA/SAS disks as `hdd`/`ssd`, so the old `sata`/`sas`-only check never matched a node without a host agent. - A retained reading no longer counts as current (carried over from #2659's frontend rule). - The verdict is sent as explicit false, because the browser's field-by-field facet merge would otherwise keep an earlier true after the condition clears. Stacked on #2659 (shares its `sensorSetup.ts` edit, which this replaces). Verified on a private mock stack at 1440px and 390px: - Overview, Storage, Replication and Backups show the notice for legacy-format nodes with an `hdd` disk without a reading, a retained SATA reading, and a SATA disk without a reading. - Overview makes no `physical_disk` request. - With `main`'s frontend swapped in, Overview showed nothing and Storage named one node of three. - A node whose disk starts reporting drops out on the next websocket update without a reload. |
||
|
rcourtman
|
3e4ea03433 |
Keep a chart for every host's copy of a same-serial disk
The registry keeps one resource per source ID, and an agent disk's source ID was its bare serial or WWN. Since same-serial disks on different machines stay apart (#2660), a dual-ported SAS shelf, cloned VMs with an explicit serial or fixed-serial USB bridges left every disk but the last one ingested without its agent source target. A disk only the agent reports then had no metrics target and no chart, both in the live registry and in the one the resource API seeds from unified resources. The agent SMART and Unraid source ID is now <agent ID>/physical-disk:<serial or WWN>, and seeding rebuilds it from the agent host above the disk, directly or through an Unraid array or cache pool. Seeding an identity-less array disk now reproduces its live host:device key too, instead of a storage-scoped key no writer uses. Canonical IDs do not move: SourceSpecificID hashes an agent disk's bare serial or WWN, so a disk split off onto its source-specific ID, and the exclusion that split it, keep their recorded IDs. Every host's split copy shares that ID, and two agent IDs on one machine no longer share a source key, so findMatch's split branch now takes it keyed to the machine when another machine's disk holds it, and joins a holder on the same machine, instead of overwriting either. History stays keyed by the serial or WWN, which every disk writer and reader shares, so a drive keeps one history across sources and host moves; a host-scoped key would orphan the stored history of every agent disk with a serial. Distinct drives sharing a serial across hosts still interleave in one series, a residual the contract records. |
||
|
rcourtman
|
400d62ca04 |
Name the agent disk history key apart from its source ID
The host agent's SMART and disk I/O writers asked HostSMARTDiskSourceID for their history key, so one function named both the registry source key and the metrics key. Scoping the source key to its host must not move the history key with it, so HostSMARTDiskMetricID now names the history key and the writers call it. Both functions return the same value here, so no written key changes. Contract-Neutral: writers keep the same history keys; only the helper that names them changes |