mirror of
https://github.com/rcourtman/Pulse.git
synced 2026-10-24 04:33:17 +00:00
4e95ca45cb9cba97bb259311b3ddacbe11c8ece9
6162 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 rcourtman
|
4e95ca45cb |
Stop Storage scrolling back to open pools and reopening groups
On the Storage pools view, an open pool pulled the page back to itself a second or two after the operator scrolled away, and collapsing the open pool's group bounced it straight back open. The summary focus bridge ran the consumer's revealActiveSeries callback inside a tracked effect, so everything Storage's callback reads (the pool-to-group lookup that every live update rebuilds, the expanded group set) re-ran the reveal and its inline-detail scroll. The reveal now subscribes only to the focused series id and the table root, and runs the callback and scroll untracked, so it runs once per deliberate focus change. Its cleanup is registered up front, so a focus change or unmount also cancels pending settle frames. Fixing that exposed a second cause of the bounce: the expansion state re-added every group key missing from its expanded set whenever a live update rebuilt the groups, so any collapsed group reopened within about ten seconds. It now stores only the groups the operator collapsed and derives the expanded set, so new groups still open by default. Group keys are bare labels, so a grouping change clears the collapsed set rather than carrying collapse state into another grouping or hiding the headerless "All" group. Measured on a 12-node mock stack: after scrolling away from an open pool the shell stays at scrollTop 0 for 12 s (was 0 -> 1111 within ~1.5 s), and a group collapsed by its button or header stays collapsed for 16 s. The new bridge, Storage and expansion tests fail on the previous code. Filter changes do not re-run the reveal either, so a focused pool whose row stays filtered out past the reveal's retry window is not revealed when the filter clears; storage links reset the URL-owned filters, so only a link that carries an excluding filter gets there. |
||
|
rcourtman
|
1a094646d7 |
Let only POST narrate single-resource reports with Pulse Assistant (#2716)
## What
Single-resource report generation (`/api/admin/reports/generate`) now
takes `POST` with a JSON body as its canonical transport, and only
`POST` may narrate a PDF with Pulse Assistant.
- **POST**: the same fields as before (`resourceType`, `resourceId`,
`format`, `metricType`, `start`, `end`, `title`) in a JSON body, decoded
with the `generate-multi` rules (1MB cap, unknown fields and trailing
payload rejected). When Pulse Assistant is configured, a PDF's executive
summary is narrated by the AI provider as before.
- **GET**: still accepts the same fields as query parameters and returns
the same report, but always uses the deterministic summary. The PDF
shows a short note in place of the "Configure Pulse Assistant" tip,
which would be wrong for a tenant that already has the Assistant
configured.
- **GET never resolves the AI service at all.** It reads Patrol findings
only from a tenant AI service that is already running, through a new
`AISettingsHandler.ExistingAIService` lookup. Constructing a tenant
service through `GetAIService` can list provider models or start
background discovery, so a report GET must not trigger it. The router's
reporting wiring moves into `Router.wireReportingAIResolvers` so a test
can drive it.
- **Other methods**: `405` with `Allow: GET, POST`.
- The shared report body decoder now also returns `body_too_large`,
rather than `invalid_body`, when the cap is hit while reading past a
valid value.
- The reporting catalog advertises
`performanceReport.singleResourceMethod: "POST"`, and the settings UI
follows it. A catalog without the field, and the legacy fallback
catalog, keep the GET query transport that older backends accept. Fleet
reports were already `POST`.
- `docs/API.md` and `docs/MSP.md` (and their shipped copies) document
both methods.
## Why
Before this change, a GET for a PDF resolved the tenant's AI narrator.
That made a paid provider call and recorded a usage event in the AI cost
ledger. GET passes the demo-mode guard and the CSRF check, and a
SameSite=Lax session cookie is sent on cross-site top-level GET
navigations. So following a link could spend a signed-in admin's AI
budget.
GET stays available rather than returning `405` because the API
reference and the MSP guide documented it for scripted exports. Breaking
those scripts would add no protection: CSV is never narrated, and GET no
longer reaches the provider.
## Verification
- `TestContract_SingleReportAINarrationRequiresPOST` drives a real
report engine with a counting narrator. GET, HEAD, PUT and DELETE make
zero narrator calls and POST makes one. The GET narrative is heuristic
and carries the note. As a negative control, the same test fails against
the handler on `main` via `go test -overlay` ("GET report called the AI
narrator 1 times, want 0").
- `TestContract_SingleReportGETDoesNotConstructTenantAIService` drives
the router's real resolvers with an uncached tenant: GET constructs no
tenant AI service, while POST resolves one. A negative control that
swaps `ExistingAIService` back to `GetAIService` fails it ("GET report
constructed 1 tenant AI services, want 0").
`TestAISettingsHandler_ExistingAIServiceNeverConstructs` covers the
lookup itself.
- Handler tests cover the POST body rules (empty, unknown field,
trailing value, form-encoded body, fields only in the query, oversized
padding) and the `Allow` header. The catalog contract snapshot includes
the new field.
- Frontend tests cover POST bodies, the GET transport for catalogs that
name GET, parser defaults and validation, and the legacy fallback.
`docsLinks.test.ts` ties the docs to the handler.
- Browser pass on a private stack: real login, dev Pro licence, and
Pulse Assistant pointed at a local stub OpenAI-compatible provider that
logs every call. Checked at 1280×800 and 390×844:
- Generating a single-resource PDF from Settings → Data & Reports sends
one `POST` with a JSON body, downloads the PDF and calls the provider
once. The downloaded PDF carries the AI narrative.
- CSV makes no provider call. The fleet report still posts to
`generate-multi`.
- A top-level browser navigation to the GET URL with the session cookie
downloads the deterministic PDF, with the new note, and makes no
provider call.
- HEAD, PUT, DELETE and PATCH return `405`, and a session POST without
the CSRF header returns `403`.
- The AI usage summary lists `report_narrative` only for the UI POSTs.
- Full `./internal/api/...` and `./pkg/reporting` run on the dev VM:
everything passed apart from two tests that fail there for environment
reasons in code this change does not touch
(`TestServerInfoEndpointReportsDevelopment` needs `.git`; a `configapi`
shell-hangup bootstrap test).
The same class of issue on the Assistant session routes
(`/api/ai/sessions/{id}/summarize` and friends) is handled separately in
#2670.
|
||
|
rcourtman
|
fd36869223 |
Judge source merges by the ingest's stale thresholds (#2717)
Stacked on #2715 (its commit is the parent of this one); either can land first. ## Problem `mergeInto` passed nil thresholds to `mergeMetrics`, so every source merge into an existing row (snapshot and record ingest alike) judged metric freshness by the registry defaults, while the stale pass and manual links used the monitor's configured thresholds. Two visible results: - A Proxmox node polled every two minutes (four-minute threshold) merged with its auto-linked agent showed the silent agent's CPU and memory, while the stale pass called the node's poll current. - A supplemental refresh (`MonitorAdapter.PopulateSupplementalRecords`, and the read-state overlay) of a linked vSphere VM kept the in-guest agent's memory over vSphere's current reading, where a full rebuild from the same records shows vSphere's. ## Change - Snapshot and record ingest hold the caller's thresholds on the registry while they run (the same per-pass pattern as the guest loop's agent index), and `mergeInto` passes them to `mergeMetrics`. The snapshot helpers that reach `mergeInto` keep their signatures. - The stale pass and the metric merge share one threshold rule, `sourceStaleThreshold`, so the merge also honours a sighting's expected update interval (slow Proxmox disk inventory), as the stale pass already did. - Outside mock mode the monitor configures only Proxmox, PBS and PMG, so those are the production sources this changes; mock mode also configures TrueNAS, vSphere and availability. ## Proof - `TestMonitorAdapterSourceMergesUseConfiguredStaleThresholds`: rebuild, supplemental refresh and overlay of a linked vSphere VM, plus a node with an auto-linked agent, through `NewMonitorAdapterWithStaleThresholds`. - `TestSourceMergesJudgeFreshnessByTheIngestStaleThresholds`: snapshot node+agent merge with and without thresholds. - `TestMetricMergeFreshnessMatchesTheStalePass`: the merge and the stale pass agree for default, configured and expected-interval sightings. - Negative control with #2715's `registry.go`: the supplemental, overlay, node+agent, snapshot and expected-interval cases fail; the rebuild case and the defaults case pass on both. - `go test ./internal/unifiedresources/... ./internal/monitoring/... ./internal/api/resourceapi/... ./internal/mock/...` on pulse-dev. |
||
|
rcourtman
|
7f45148d15 |
Drop the unused Patrol readiness-failure object behind the toggle toast (#2687)
Stacked on #2686 (which stacks on #2682 and #2674); this branch carries
their commits until they land. Only the top two commits are new.
## What
`buildPatrolSettingsReadinessFailure` built a full
`PatrolConfigurationFailureInput` for the Patrol page's inline "needs
attention" state. That state lost its render in
|
||
|
rcourtman
|
796a70999b |
Stop tenant incident stores racing on shared ID counters (#2711)
Each tenant monitor owns its own `IncidentStore`, but incident and incident-event IDs came from package-level `int64` counters bumped with `++` under nothing but the calling store's mutex. Two tenants recording alerts at once raced on them. `go test -race` failed `TestLeavingMockModeClearsEveryRunningTenantMonitor` on #2698 ([run](https://github.com/rcourtman/Pulse/actions/runs/37652317098/job/112919080928)) at `incidents.go:1753`. The remediation record counter in `remediation.go` had the same shape, and `startPatrolForContext` creates one remediation log per org. **Fix** - All six `internal/ai/memory` ID generators (incident, incident event, remediation record, memory, incident memory, pattern memory) now mint through one atomic helper, `nextSequencedID` in `ids.go`. - The numeric suffix no longer wraps at 1000. Even with atomic counters, a generator that minted more than a thousand IDs in one wall-clock second repeated an ID. That matters because `RecordNote` finds an incident by ID and the incident query deduplicates merged events by ID. The `<prefix><YYYYMMDDhhmmss>-<n>` layout is unchanged. Only `n` can now exceed three digits. - `intToString` lost its last runtime callers and is removed. **Proof** - `internal/ai/memory/ids_test.go` mints 2,400 IDs from every generator on 16 goroutines and records alerts into two tenant stores concurrently. - Against the `origin/main` runtime files (via `-overlay`), `-race` reports DATA RACE at `incidents.go:1753/1760` and `remediation.go:355`, and every generator fails the uniqueness/shape check: four produce duplicate IDs, and `mem`/`pat-mem` produce a wrapped `-0` suffix. - With the fix, `go test -race ./internal/ai/memory/` passes, and `go test -race -count=5 -run TestLeavingMockModeClearsEveryRunningTenantMonitor ./internal/monitoring/` passes. **Residual:** across restarts, only the timestamp separates IDs. If the local clock repeats a second (DST fall-back or a backward step), a restarted sequence could still meet an earlier suffix. Persisted IDs and the stored format are unchanged. The `ai-runtime` contract records this. |
||
|
rcourtman
|
c5416b6800 |
Alert on the Unraid temperature an agent disk shows (#2701)
Stacked on #2698 (which sits on #2694, #2685 and #2679); this PR's own change is the top commit, "Alert on the Unraid temperature an agent disk shows", plus a merge of main for an alerts.md anchor both touched. ## Why A host agent's physical disk can show a temperature that only the host's Unraid inventory reports: - a SMART row whose smartctl probe returned no temperature takes its Unraid row's reading (`HostSMARTDiskTemperature`), and - a disk only the inventory lists, such as a `--disk-exclude` match, is built from its Unraid row alone (`HostUnraidDiskTemperature`). Physical Disks (Temp cell and the `Running Hot` verdict, "at or above its alert threshold") and Patrol judge that reading against the alert disk temperature policy, but `CheckHost` only read the SMART row's own field. A disk could read as at its alert threshold while no alert fired. Unraid's own warning/critical disk temperature notifications do not make this redundant: Pulse already alerts on the smartctl readings and array state of the same hosts, which Unraid also notifies on. ## What changes - `unifiedresources.HostDiskTemperatureReadings(host)` lists the reading each of a host's disks shows, built with the registry's own adapters: one per SMART row the registry ingests, then one per Unraid row whose disk key no SMART row carries (the same rule as the history writer in #2698). - `CheckHost` judges those readings: - SMART rows keep their `agent:<host>/disk_temp:<device>` resource IDs. An inventory-only disk uses its block device token, which is the ID a plain Linux SMART row for that device uses, so a later SMART reading continues the same alert. - Per-type thresholds and host overrides use the disk type the disk shows. - Only a reading collected now fires or resolves, so a spun-down disk holds its alert. - When conflicting serials put two disks on one device, its alert judges the reading that stands highest against its own type's threshold. - An inventory-only disk matching the agent's `--disk-exclude` raises nothing, and an alert it raised before the exclusion resolves on the next report. This matches the documented exclusion from monitoring and how linked Proxmox hosts already treat excluded disks. - Absence cleanup still runs only on a non-empty SMART list. ## Verification - New alert tests in `internal/alerts/host_unraid_lifecycle_test.go` cover: the fallback, an inventory-only disk, an empty SMART list, two disks on one device (also of different types), a controller member under another device label, standby, exclusion, disk-type and host-override thresholds, holds, handover to a SMART reading, departure, and the exclusion clear. - Each fails on the previous `host.go`. - Removing any guard (exclusion skip, exclusion clear, disk type, collected-now, threshold ranking) fails its own test. - `TestHostDiskTemperatureReadingsMatchTheDisksTheRegistryShows` compares every reading with the disk the live registry ingests, including conflicting serials, a virtual device and a MegaRAID member Unraid lists under its block device. - `go test ./internal/alerts/... ./internal/unifiedresources/...` passes on the merged tree, as do the Unraid/SMART/temperature tests in `internal/monitoring`. |
||
|
rcourtman
|
a14584d9f1 |
Drop the summary focus bridge members nothing reads (#2709)
The jump-to-row affordance lost its only consumer when the summary components were deleted, yet the bridge kept computing it on every Workloads and Storage page: a capture-phase window scroll listener and a resize listener re-measured the hovered or focused row on each frame and hover change, walking its ancestors with getComputedStyle. On the 12-node mock storage table that was 128 calls across eight row hovers and 320 across ten scroll steps, now none. Storage's reveal callback kept branches that switched between the pools and disks views for that jump. Focus only ever names a row in the active view, so drop them and the disk-id memo that served them. The chart-hover input went the same way. No summary chart renders, so setChartHoverSync was never called and the resolver's chart branch was always fed null. Remove the signal, its type and the resolver option, and say in the contract how a returning chart re-enters the shared scope instead. Focus reveal, pinned-group reveal, Escape and row emphasis behave as before at desktop and phone widths; the browser receipt records the before/after pass. |
||
|
rcourtman
|
e7cabc72f1 |
Hide pprof routes on demo-mode instances (#2708)
The `/debug/pprof` handlers are gated on `RequireAdmin` plus a `settings:read` scope that session logins skip. A `DEMO_MODE` instance that signs visitors in as the configured admin therefore served every visitor goroutine and heap dumps, the process command line, and CPU profiles and execution traces that load the whole process. Demo mode already hides diagnostics, logs, admin users and manual discovery through the admin-operations policy; pprof was missing from it. ## Change - `internal/api/demo_mode_operations.go`: through the admin-operations policy, the demo guard answers `/debug/pprof` and every path below it with `404` for every method, ahead of route authentication, so a write probe such as the `POST` symbol lookup gets the same `404` instead of the generic demo `403`. Instances outside demo mode keep the routes, and `PULSE_PPROF_DISABLED` still removes them everywhere. - Contract notes in api-contracts (owner), agent-lifecycle and storage-recovery. ## Proof - `TestContract_DemoModeHidesPprofFromDemoAdminSession` signs in as the demo admin through the full router. The same session gets `200` from `/debug/pprof/cmdline` below the demo guard, and `404` for eight pprof paths × `GET`/`HEAD`/`OPTIONS`/`POST` through the served handler. Against the previous policy (`go test -overlay`) `GET` and `HEAD` returned pprof output (`200`, or the root's `301` into the index), `OPTIONS` got the router's preflight `200` and `POST` the generic demo `403`. - `TestPublicDemoAdminOperationsPolicyHidesEveryDebugRoute` parses the router files and fails if a literal route under `/debug/`, with or without a method prefix, escapes the policy for any of seven methods, so a new debug handler there cannot ship reachable on a demo by default. A temporary `GET /debug/vars` registration made it fail as expected. - Path-handling probe (not committed): 210 raw requests (35 request targets × six methods), covering trailing and double slashes, dot segments, `%2F`/`%2e` encodings, absolute-form targets and `CONNECT`, run against a copy of the router's mux dispatch. None reached a pprof handler; paths the mux would clean onto pprof only receive a redirect, and the policy catches the follow-up request. |
||
|
rcourtman
|
6263c580f7 |
Judge a linked agent's guest disks by that agent everywhere (#2712)
Follow-up to #2669, which made the guest drawer's filesystem rows follow a linked Pulse agent's freshness. The workload table's disk cell and the drawer History still judged the same guest by the Proxmox read reason, so one guest disagreed with itself across the three. ## What changes - **One rule for three readers.** `getWorkloadGuestDiskRead` (`workloadGuestPresentation.ts`) now drives the table disk cell, the drawer's filesystem rows and History. When the linked agent's filesystems fill in for a VM that has no Proxmox guest filesystems, the reading is current while the agent reports, last known once it is stale, and unavailable once the guest is stopped. Proxmox's own guest filesystems keep their read reason. - **Where the aggregate disk metric comes from.** On a manually linked VM, Proxmox records no guest disk usage without guest filesystems, so the registry's link merge keeps the agent's disk metric. A registry test pins that. - **Poller carry-forward.** `previousVMFromView` built the previous VM from the merged metric, so the next poll carried the agent's disk forward as a Proxmox `prev-agent-not-running` read. Proxmox priority then froze it over the agent's live value. Only Proxmox's own disk reading is carried now (`VMView.DiskFromLinkedAgent`). - **`agent.stale` is always on the wire.** The browser merges agent facets field by field, so an omitted false kept a resumed agent "stopped reporting" until a reload. This affected Machines rows and node drawers too. - **Withdrawn filesystems.** The websocket merge kept filesystems that a native Proxmox VM facet or a native agent facet (which always carries `stale`) no longer reported, including in the `platformData.agent` mirror that host threshold rows read. ## Verification - New tests, each failing on main: `GuestRow.test.tsx`, `GuestDrawer.test.tsx`, `workloadGuestPresentation.test.ts`, `resourceStateAdapters.test.ts`, `TestPreviousVMFromViewKeepsLinkedAgentDiskOutOfProxmoxCarry`, `TestAgentDataAlwaysSendsStale`, `TestVMViewDiskFromLinkedAgentFollowsSelectedMetricSource`, and `TestResourceRegistry_ManualGuestAgentLinkDiskMetricFollowsProxmoxGuestFilesystems`. The registry test is a characterization test; it fails when the agent outranks Proxmox on guest disk. - Private mock stack at 1440 and 390, with agents linked into VMs through `POST /api/resources/{vm}/link`: - Live agent, no QEMU guest agent: before, "- Unavailable · Guest agent not running", with History deferred; after, the agent's bars, nothing deferred. - Silent agent: before, "- Unavailable"; after, bars plus "Last known" (phone: "Prior"), with History deferred on the agent message. - Stopped guest: Unavailable. - A guest with Proxmox filesystems: unchanged. - Machines row for a standalone agent released from a silent hold, watched without a reload: on main it kept "(stale)"; now it clears. - Bundle: WorkloadsSurface 61.93 kB gzip against a 70.23 kB cap. ## Known residuals - Mock mode's websocket state does not apply manual links to resources ingested as records (`ingestRecords` never runs `applyManualLinks`), so the browser pass ran the Proxmox page on REST data. In production this likely affects links to vSphere or TrueNAS resources; it is tracked separately. - A stale Proxmox source can lose the aggregate to a live agent while its own filesystem rows and `prev-` reason remain. That guest stays on the Proxmox rule and reads its live aggregate as last known, which is the same as main and errs toward caution. A guest with no platform read reason can likewise keep a retained aggregate after the agent withdraws its filesystems, also as on main. Both need aggregate provenance on the wire. |
||
|
rcourtman
|
15322e1573 |
Drop contract prose for the deleted problem-resource surfaces (#2706)
Contract prose still described surfaces that no longer exist: - `frontend-primitives.md` told new route-level copy to extend `problemResourcePresentation` (deleted in |
||
|
rcourtman
|
23bf99e1ba |
Keep alert settings a config save leaves out (#2689)
## Problem
Saving anything on Alerts > Thresholds turned flapping detection and
alert TTL cleanup off. The page's `PUT /api/alerts/config` body never
carries `flappingEnabled`, `maxAlertAgeDays`, `maxAcknowledgedAgeDays`,
`autoAcknowledgeAfterHours` or `customRules` (the page has no control
for any of them). `UpdateAlertConfig` decoded the body into a zero
`AlertConfig`, so every save wrote `false`/`0` for those fields, reset
the flapping tuning to defaults, dropped custom rules, and persisted
that. The `flapping`, `aggregation` and `ioNormalization` objects the
page sends are not `AlertConfig` fields and never enabled anything.
## Fix
- `PUT /api/alerts/config` now replaces only the top-level keys the body
carries. A key left out keeps its stored value. A sent key replaces its
setting whole, so `"overrides": {}` still clears overrides, and the
usual normalization still runs. A body that is not a JSON object
(including `null`) is a 400 and changes nothing.
- `alertconfig.ApplyAlertConfigUpdate` decodes the body on its own, so
sent keys get exact `encoding/json` semantics, including duplicates and
case-folded keys. It copies every unsent top-level field from a JSON
clone of the stored config.
- `Manager.ApplyConfigUpdate` reads, merges and applies under the
manager lock, so concurrent partial saves cannot revert each other. It
returns an owned snapshot, which the handler persists, so
`SaveAlertConfig`'s in-place normalization no longer writes into the
live config's maps. The handler runs one save at a time, so successful
saves reach `alerts.json` in the order they were applied.
- Stored configs are not rewritten; a value an earlier save zeroed stays
as it is.
## Verification
- Unit: `internal/alerts/config/update_test.go`; manager save path and
concurrent partial updates in
`internal/alerts/config_validation_test.go`; handler against a real
manager in `internal/api/alerting/alerts_test.go`. Each new regression
fails against main's code (overlay negative controls). The concurrency
test fails at round 0 against a non-atomic variant.
- `go test ./internal/alerts/... ./internal/api/alerting/` and the
`internal/api` alert endpoint, tenant and scope tests pass; the alert
tests pass under `-race`.
- Private stack (mock mode, Vite on main's frontend):
- On main, a node CPU change saved from the Thresholds page took
flapping and all three TTLs to false/0, in memory and in `alerts.json`.
- With this change, the same UI save kept them (true/7/1/24) and stored
the new threshold.
- The API repro (GET, drop the four keys, PUT) keeps them; explicit
`false`/`0` still applies; `null` and `[]` return 400.
- Five rounds of concurrent partial PUTs kept both changes in memory and
on disk.
## Follow-ups
- The thresholds page still sends the dead
`flapping`/`aggregation`/`ioNormalization` objects and hard-coded
`minimumDelta`/`suppressionWindow`/`hysteresisMargin`. A frontend
cleanup will follow once the open PRs touching
`alertsConfigurationModel.ts` land.
- `SaveAlertConfig` still normalizes caller-owned maps in place for
other callers (`ActivateAlerts`, startup paths), and the GET handler
encodes a shallow copy. That pre-existing race is tracked separately.
|
||
|
rcourtman
|
36d3f8c557 |
Judge pulse-relaxed guests by one threshold on poll and save (#2705)
Stacked on #2700 (which carries #2696 and #2692). Merge order is enforced by the stack: this change must not reach main before #2700's per-filesystem reevaluation (see "Why stacked" below). ## Problem `pulse-relaxed` lifts a guest's CPU, memory and disk triggers to a 95/92/95 floor. Only `CheckGuest` applied it. Config-save reevaluation rebuilt the guest from alert metadata without its tags (`guestSnapshotFromAlert`), so a save judged a relaxed guest's alert by its unrelaxed thresholds: - A relaxed guest with an unset memory threshold alerts at 92 while polling. On save that reads as no threshold, so the save resolved the alert and sent a recovery notification, and the next poll raised it again. - The floor also lifted an Off threshold (trigger 0, or -1 from the editors) like any low trigger, so tagging a guest relaxed turned an Off alert on, with no recovery band (`{95, 0}`). A save then re-judged it Off and resolved it, and the next poll raised it again. ## Change - The relaxed step moves into `effectiveAlertPolicyNoLock`'s guest branch, so `CheckGuest`, config-save reevaluation (including #2700's `resolveGuestAlertThresholdsNoLock`) and posture checks share one resolver. - `guestSnapshotFromAlert` reads the recorded `tags` back, as `[]string` live or `[]interface{}` after a restart, through a new `metadataStringsValue` reader. `storagePolicyAliases` now uses the same reader. - An Off threshold (trigger <= 0) stays Off under `pulse-relaxed`; an unset one still gets the floor, as the existing tests pin. The tag is documented (issue #863) as relaxing thresholds, so it must never make a guest more sensitive than its config. - `metricOptionsWithTags` drops the `tags` key from an open alert once its resource has no tags. Reevaluation now reads that key, so a removed `pulse-relaxed` left behind would judge the alert by the relaxed trigger while polling uses the guest's own, and the alert would flap. Notification tag routing reads the same key, so it also stops routing on removed tags. ## Why stacked A filesystem's own override replaces the relaxed guest disk threshold in `CheckGuest`. On main, reevaluation misses filesystem overrides (fixed by #2700). With this change alone, a save would judge a relaxed guest's filesystem alert by the 95 floor instead of, say, an 85 override, and flap it. `TestConfigSaveKeepsRelaxedGuestFilesystemOverride` fails on main plus this change and passes on this stack. ## Proof New tests in `internal/alerts/alerts_test.go` and `internal/alerts/unified_eval_test.go`. With `go test -overlay` mapping the six runtime files back to #2700's head, these fail: - `TestConfigSaveJudgesRelaxedGuestAlertByRelaxedThresholds` (the save resolved the alert) - `TestPulseRelaxedKeepsOffGuestThresholdOff` (a disk alert was raised with disk Off) - `TestApplyRelaxedGuestThresholds/off_thresholds_stay_off` (Off became `{95, 0}` / `{92, 0}`) - `TestCheckGuestDropsRemovedTagsFromOpenAlerts` (stale tags; with only the metadata assertion relaxed, the save resolves the alert) `go test ./internal/alerts/... ./internal/notifications/` and the monitoring alert tests pass. |
||
|
rcourtman
|
b7049ff778 |
Raise Proxmox disk temperature alerts under the per-type policy (#2680)
**Stacked on #2675** (TrueNAS disk temperature alerts by disk type). Merge that first; this branch contains its commit. ## Problem A Proxmox physical disk that only the node's own sensors report never raised a temperature alert: - `CheckDiskHealth` judges health and wearout only; - unified evaluation types TrueNAS physical disks, not PVE ones. So Physical Disks could read a disk **Running Hot** under the per-type thresholds while alerts stayed silent. Whether a disk alerted depended on whether a Pulse agent ran on the node: agent-covered disks alert through `CheckHost` at the same thresholds. #1347 shows users look for disk temperature thresholds in the Proxmox tabs. ## Change - **`CheckProxmoxDiskTemperature`** (`internal/alerts/proxmox_disk_temperature.go`) raises a `diskTemperature` metric alert under the disk's existing PVE alert reference. It is judged by the disk temperature policy: the disk type's `DiskTempByType` entry under the agent Disk Temp default, which switches it off for every type. The alert carries the disk's serial, WWN and `diskType`, like the health and wearout alerts. - **One alert per disk.** `mergeHostAgentSMARTIntoDisks` marks a disk that a still-reporting linked agent lists in its SMART report (`AgentSMARTReported`, internal, never serialized). That disk is the agent's: `CheckHost` alerts on it, and an open PVE alert closes with `moved_to_agent`. A lapsed agent owns nothing, so the node's own current readings keep alerting while its agent is silent. - **Holds and clears.** Only a reading collected this poll is judged (`diskinventory.TemperatureCollected`), so a retained value holds the alert. An excluded device, or a switched-off policy, closes it. - **Stability.** A new `proxmox-disk` alert delay defaults to 5s, so warnings wait the same 300s noisy-gauge window as agent and TrueNAS disk temperatures. Critical readings bypass it. - **Config save.** `reevaluateActiveAlertsLocked` re-judges the alert per disk type. Without this it would fall through to guest thresholds, which have no disk temperature, and resolve on every save. An alert without a recorded type uses the lowest per-type trigger. - **History.** `ProxmoxPhysicalDiskAlertIdentifiers` lists the new `<ref>::metric-threshold:diskTemperature` identifier, so resource history owns its rows by recorded hardware identity, as #2646 does for health and wearout. - **Copy.** The Agents "Disk temperature by type" card says it covers agent, TrueNAS and Proxmox disks. ## Evidence Private mock stack (8 PVE nodes) with a local-only overlay pinning PVE disk readings: | Disk | Physical Disks | Alert | |---|---|---| | pve1 NVMe 63°C | Healthy | none | | pve2 SATA 56°C | Running Hot | warning, trigger 55, after the 300s window | | pve3 NVMe 72°C | Running Hot | warning, trigger 70, after the 300s window | | pve3 SAS 63°C | Healthy | none | | pve3 SATA 66°C | Running Hot | critical at once | The alert card reads: "Samsung 870 EVO 1TB (/dev/nvme2) · Temperature · Temperature 66°C, above the 55°C alert level · Clears once it drops to 50°C or lower and stays there for 5 minutes." ## Tests - `internal/alerts/alerts_test.go`: - the policy (per type, raised trigger); - ownership with `moved_to_agent`, the hold without a reading, exclusion, the off switch; - the stability window; - re-judging on save, including the lowest-trigger hold; - history identifiers. - `internal/monitoring/physical_disk_roundtrip_test.go`: the marker for reporting agents only (not a lapsed one), the SMART fallback path, and that `checkPhysicalDiskAlerts` passes collected readings, ownership and exclusion. - `internal/unifiedresources/store_test.go`: a temperature alert row is owned by its disk and still found by a read by its reference. - `internal/models/deepcopy_test.go`: the marker survives cloning and stays out of JSON. - `ThresholdsTable.test.tsx`: the card copy. - Ten wiring points were mutation-checked (marker, lease condition, collected reading, ownership, exclusion, re-judge branch, lowest trigger, resolution, history identifier); each removal fails a test. ## Second opinion - Fixed: an offline agent's retained SMART rows kept ownership, so neither side alerted while the agent was silent. Ownership now needs a reporting lease. - Kept, documented in `alerts.md`: on the first poll after an agent is newly linked to a disk with an open PVE alert, both alerts can be open until the next disk poll. This is the same window as the node-to-agent usage alert handover. ## Not in this PR - Like the health and wearout alerts, a disk that leaves the inventory keeps its temperature alert until the 24-hour stale sweep. - Per-host Disk Temp overrides do not reach PVE-only disks, which have no agent host. |
||
|
rcourtman
|
b814ccdd81 |
Count down to future Patrol times instead of reading just now (#2702)
`formatRelativeTime` is past-only: a time ahead of now reads "just now"
(compact) or "0s ago". Three Patrol lines handed it future times, so
each read wrong for its whole duration:
- Attention detail, suppressed issue: "Returns to active attention just
now." (also for an issue marked reviewed, which expiry hands back to
Reviewed, not active attention)
- Findings, will-fix-later: "Reminding just now"
- Findings, snoozed: "snoozed until just now"
Supersedes #2613, rebuilt on main now that #2598 has landed (#2613 was
stacked on two early #2598 commits whose contract wording main has since
replaced).
## Change
- `formatTimeUntil` in `utils/format.ts`, beside `formatRelativeTime`:
counts down ("in 3h", "in 1d"), rounds to the nearest unit (a 24-hour
suppression set a moment ago reads "in 1d", not "in 23h"), and returns
`dueText` (default "now") once the time arrives. `formatRelativeTime`
stays past-only and its doc points future times at the new helper.
- The three lines use it with the shared relative-time clock
(`useRelativeTimeNow`), since none of those times change while the panel
stays open, and keep the exact time on hover.
- Suppression line: "Returns to active attention in 1d." for an open
issue, "Suppression ends in 1d. It stays reviewed." for one marked
reviewed, "Suppression has ended." once due. It takes its words from
`getAttentionSuppressionEndPresentation` (added on main by
|
||
|
rcourtman
|
0b6473affc |
Fill the table's visible width with phone drawers (#2703)
Phone and tablet table drawers capped their sticky content at
`calc(100vw - 3.5rem)`, a guess at the surrounding page chrome. On pages
with less chrome than that, every drawer stopped short of the table and
left an empty strip on the right: 27px at 390px, measured on the
Availability checks, Proxmox node and guest, Docker host and container,
Kubernetes node and Machines drawers. At tablet widths a table wider
than its shell got a drawer wider than the visible area (712px in
695px).
The table scroll shell already declares `container-type: inline-size`
(the responsive column priorities use it), so `InlineDetailTableRow` now
caps the default content shell at `100cqi`, the shell's own visible
width. Measured live: content equals the visible table width at 320,
390, 768 and 1000px, with no right gap; desktop keeps the static
uncapped layout. The availability drawer's hostname now fits instead of
truncating. Callers passing their own `contentClass` are unaffected.
Not changed here: `.table-fixed td { overflow: hidden }` makes the
sticky offset inert, so a sideways-scrolled table still moves its drawer
(same before and after). Long unwrapped identity chips that were already
clipped are clipped less now; wrapping them is a separate follow-up.
Proof: `InlineDetailTableRow.test.tsx` and the shared-primitive
guardrail pin the new cap and reject the old one; the guardrail already
pins the scroll shell's `container-type`. Browser receipt covers five
routes at 320 to 1440px, before and after.
|
||
|
rcourtman
|
3fb267333d |
Require POST for Assistant session mutation routes (#2670)
The session router dispatches `/api/ai/sessions/{id}/abort`,
`summarize`, `fork`, `undo` and `redo` by path alone, and none of those
handlers checked the HTTP method, so a GET or HEAD cancelled a running
turn, compacted or rewound a transcript, or persisted a fork.
- **Demo mode:** `DemoModeMiddleware` admits every GET/HEAD as a read,
so a public-demo session could mutate Assistant sessions despite the
read-only guard.
- **CSRF:** `CheckCSRF` skips GET/HEAD, and session cookies default to
SameSite=Lax, which browsers send on cross-site top-level GET
navigations. A link carrying a session ID could therefore trigger these
mutations for a signed-in user.
## Change
- `requireAssistantSessionMutationMethod` in
`internal/api/ai_handler.go`: abort, summarize, fork, undo, redo and
steer admit only POST and answer anything else with `405` + `Allow:
POST` before resolving the Assistant service. Steer already refused
non-POST but sent no `Allow` header; it now uses the same guard.
- The guard sits in the handlers, matching how chat, steer and the
approval approve/deny handlers already own their method contracts.
- `messages` (GET) and the unsupported `diff`/`revert`/`unrevert` routes
are unchanged. Session rename/delete and the collection route already
dispatch on the method.
- Callers: `frontend-modern/src/api/aiChat.ts` uses POST for all six;
pulse-mobile's abort call and its OTA compatibility probe use POST.
## Proof
- `TestAssistantSessionMutationsRequirePost`: GET/HEAD/PUT/DELETE get
405 + `Allow: POST` for every mutation, with no mutation call and no
`IsRunning` check reaching the service. POST still dispatches.
- `TestAssistantSessionReadSubresourcesKeepTheirMethods`: GET `messages`
returns 200 and GET `diff` returns 501, with no mutation either way.
- `TestDemoModeRouterRejectsAssistantSessionMutationsOverSafeMethods`:
drives `NewRouter(...).Handler()` with `DemoMode` on. GET/HEAD to every
mutation pass the demo guard and get 405, POST stays 403, nothing
mutates, and GET `messages` still reads.
- Negative controls via `go test -overlay`: on main's `ai_handler.go`,
every non-POST case fails and the demo test reports `GET abort status =
200`. With abort's guard moved after the `IsRunning` check, the ordering
assertion fails.
|
||
|
rcourtman
|
3a482c610c |
Judge source merges by the ingest's stale thresholds
mergeInto always passed nil thresholds to mergeMetrics, so every source merge into an existing row judged metric freshness by the registry defaults while the stale pass, the manual links and the monitor's configuration used the caller's thresholds. A Proxmox node polled every two minutes (four-minute threshold) merged with its auto-linked agent showed the silent agent's readings while the stale pass called the node's poll current, and a supplemental refresh of a linked vSphere VM kept the in-guest agent's memory over vSphere's current reading. Snapshot and record ingest now hold the caller's thresholds on the registry while they run, the way the guest loop holds its agent index, and mergeInto reads them. The ~50 snapshot helpers that reach mergeInto through ingest() keep their signatures. The stale pass and the metric merge now share one threshold rule, sourceStaleThreshold, so the merge also honours a sighting's expected update interval (slow Proxmox disk inventory) as the stale pass already did, and a source without a default falls back to 120s in both (the merge used 60s). Every defined source has a default, so that fallback reaches none. Outside mock mode the monitor configures only Proxmox, PBS and PMG, so those are the production sources this changes; mock mode also configures TrueNAS, vSphere and availability. |
||
|
rcourtman
|
1ac600ec2b |
Judge filesystem alerts by their own threshold on config save (#2700)
Stacked on #2696, which is stacked on #2692. The unique commit is the
last one,
|
||
|
rcourtman
|
9bd1b6e582 |
Let only POST narrate single-resource reports with Pulse Assistant
GET /api/admin/reports/generate resolved the tenant's AI narrator, so a PDF requested with GET made a paid provider call and appended a usage event to the AI cost ledger. GET passes the demo-mode guard and the CSRF check, and a SameSite=Lax session cookie rides a cross-site top-level GET navigation, so a link could spend a signed-in admin's AI budget. POST with a JSON body, decoded with the generate-multi rules, is now the canonical single-resource transport and the only one that may use the AI narrator. The settings UI sends it, following a new performanceReport.singleResourceMethod catalog field; a catalog without the field, and the legacy fallback catalog, keep the GET transport that older backends accept. GET stays available with the same query parameters but narrates deterministically, printing a note in place of the "Configure Pulse Assistant" tip, and other methods get 405 with Allow: GET, POST. GET also stops resolving the AI service at all: it reads Patrol findings through a new ExistingAIService lookup, because GetAIService constructs a tenant service on first use and that can list provider models or start background discovery. The router's reporting wiring moves into wireReportingAIResolvers so a test can drive it. Keeping GET rather than answering 405 avoids breaking documented API scripts (MSP.md pointed providers at it) for no security gain: CSV is never narrated and GET no longer reaches the provider. |
||
|
rcourtman
|
b0590c0ccf |
Join operator links on resources that arrive as records
The monitor rebuilds its registry from the state snapshot and then ingests supplemental records (vSphere, TrueNAS). Snapshot and resource ingest applied the operator's manual links, but record ingest did not, so a link whose side exists only as a record (an agent inside a vSphere or TrueNAS VM, an agent on a TrueNAS host) never joined in the registry behind the websocket broadcast and /api/state. The resources API seeds through resource ingest and showed the same pair merged, and the published inventory also flipped to merged whenever continuity hydration cloned the registry for an unrelated absent machine. Record ingest now applies the links, with the adapter's stale thresholds on the rebuild, live supplemental and overlay paths. Continuity records stay out of it: a saved enrollment only fills an absent machine and must not lend its linked guest an offline verdict. A link merge keeps the TrueNAS and vSphere payloads the primary lacks. Joining links in the rebuild exposed how identity treats a folded resource. Its ID was no longer live, so once an agent-type primary was pinned with the folded agent's machine key, pin succession re-keyed the link onto the primary itself and the next rebuild split the pair. That already broke snapshot links with a Proxmox node as primary. The registry now remembers each folded resource, and pin-driven and record-declared successions treat it as observed. Mock mode still builds its view from fixtures without the link store, so the mock broadcast is unchanged here. |
||
|
rcourtman
|
77d2119792 |
Remove unread Patrol summaryStats and its audit rule (#2690)
Stacked on #2686, which sits on #2682 and #2674; its own change is the top commit, |
||
|
rcourtman
|
1231392e8d |
Judge a linked agent's guest disks by that agent everywhere
PR 2669 made the guest drawer's filesystem rows follow a linked Pulse agent's freshness when its filesystems fill in for a VM that has no Proxmox guest filesystems. The workload table's disk cell and the drawer History still judged the same reading only by the Proxmox read reason. So one guest read as current in the drawer and as "Unavailable: Guest agent not running" in the table and History, and a silent agent's reading could not show as last known there. getWorkloadGuestDiskRead now owns one rule for the table cell, the drawer rows and History: current while the agent reports, last known once it is stale, unavailable once the guest is stopped. Proxmox's own guest filesystems keep their read reason. The rule also covers the aggregate disk metric, because Proxmox records no guest disk usage without guest filesystems, so the manual link merge keeps the agent's metric. Three paths broke that: - The poller built its previous VM from the merged metric, so the next poll carried the agent's disk forward as a Proxmox prev- read, and Proxmox priority then froze it over the agent's live value. Only Proxmox's own disk reading is carried now. - agent.stale was omitempty, and the browser merges agent facets field by field, so a resumed agent stayed "stopped reporting" until a reload. It is always sent now. - The same merge kept filesystems a native Proxmox VM facet or a native agent facet had withdrawn, so the rows did not switch source. |
||
|
rcourtman
|
239cd06933 | Merge origin/main into fix/unraid-disk-temperature-alerts | ||
|
rcourtman
|
88d44099aa |
Stop tenant incident stores racing on shared ID counters
Each tenant monitor owns its own IncidentStore, but incident and incident-event IDs came from package-level int64 counters bumped with ++ under nothing but the calling store's mutex. Two tenants recording alerts at once raced on them: go test -race failed TestLeavingMockModeClearsEveryRunningTenantMonitor on PR 2698 at incidents.go:1753. The remediation record counter had the same shape, and remediation logs are created per org. All six internal/ai/memory ID generators now mint through one atomic helper. The numeric suffix also stops wrapping at 1000: even with atomic counters, one generator minting more than a thousand IDs in a wall-clock second repeated an ID, and incident notes find their incident by ID. The prefix and timestamp layout is unchanged, and intToString lost its last callers. ids_test.go mints 2,400 IDs from every generator on 16 goroutines and records alerts into two tenant stores concurrently. Against the old files it reports DATA RACE at incidents.go:1753/1760 and remediation.go:355, and every generator mints duplicate or wrapped IDs. |
||
|
rcourtman
|
aafa4793c3 |
Drop the summary focus bridge members nothing reads
The jump-to-row affordance lost its only consumer when the summary components were deleted, yet the bridge kept computing it on every Workloads and Storage page: a capture-phase window scroll listener and a resize listener re-measured the hovered or focused row on each frame and hover change, walking its ancestors with getComputedStyle. On the 12-node mock storage table that was 128 calls across eight row hovers and 320 across ten scroll steps, now none. Storage's reveal callback kept branches that switched between the pools and disks views for that jump. Focus only ever names a row in the active view, so drop them and the disk-id memo that served them. The chart-hover input went the same way. No summary chart renders, so setChartHoverSync was never called and the resolver's chart branch was always fed null. Remove the signal, its type and the resolver option, and say in the contract how a returning chart re-enters the shared scope instead. Focus reveal, pinned-group reveal, Escape and row emphasis behave as before at desktop and phone widths; the browser receipt records the before/after pass. |
||
|
rcourtman
|
0180b33431 |
Hide pprof routes on demo-mode instances
The /debug/pprof handlers are gated on RequireAdmin plus a settings:read scope that session logins skip. The public demo signs every visitor in as the configured admin, so any visitor could fetch goroutine and heap dumps and the process command line, or start CPU profiles and execution traces that load the whole process. DEMO_MODE already hides diagnostics, logs, admin users and manual discovery through the admin-operations policy; pprof was missing from it. The policy now has the demo guard answer /debug/pprof and every path below it with 404 for every method, ahead of route authentication. Instances outside demo mode keep the routes, and PULSE_PPROF_DISABLED still removes them everywhere. TestContract_DemoModeHidesPprofFromDemoAdminSession signs in as the demo admin through the full router, shows the same session still reads /debug/pprof/cmdline below the demo guard, and gets 404 for eight pprof paths through the served handler. On the previous policy GET and HEAD returned pprof output (200, or the root's 301 into the index) and POST got the generic demo 403. A policy test also fails if a literal route the router files register under /debug/, with or without a method prefix, escapes the policy for any common method. |
||
|
rcourtman
|
48c2d6cdce |
Stop Patrol fetching the correlation list nobody reads (#2677)
The Patrol page fetched `GET /api/ai/intelligence/correlations` on every mount, every 60 s poll, every return to the tab and on Retry, and the store's dashboard bundle fetched it again for FindingsPanel's refresh. Nothing has read the list since the Patrol supporting-context block was removed in 27ba85ddc6; the resource drawer reads per-resource correlations from the resource-intelligence payload. ## Change - `stores/aiIntelligence.ts`: remove the correlation signal, `correlations` getter, `correlationsSignal`, `loadCorrelations()`, its org-switch reset, and its entry in `loadDashboardData()`. - `features/patrol/usePatrolIntelligenceState.ts`: drop the read from the Retry path's background supporting reads. - `api/ai.ts`: remove `AIAPI.getCorrelations` (no other caller). The backend route stays as an API surface behind `ai:execute`. - Tests: the store bundle test now pins every AIAPI read the bundle makes; the manual-run test stalls the post-start refresh through a remaining read so it still proves "Starting" clears without waiting (negative controls: awaiting the refresh, or removing it, both fail the test). Dead correlation route stubs removed from six E2E specs. - Contract prose in api-contracts, patrol-intelligence, unified-resources, ai-runtime and frontend-primitives no longer says the Patrol page or store loads or renders correlations or recent changes. The remaining stale Patrol correlation paragraphs (patrol-intelligence ~L2135/~L2197, ai-runtime ~L6854) are rewritten by #2674; the two PRs split hunks by agreement and merge cleanly together. `CorrelationsResponse` stays for now: `patrolInvestigationContextModel.ts` still imports it until #2674 lands. It goes in a follow-up once both are on main. ## Evidence - Demo-mode stack, headless Chromium at 1440x900 and 390x844: correlation requests go from 1 to 0 per Patrol load, browser reload, 60 s poll and tab hide/show. Every other `/api/ai/*` read per phase is identical; no console errors. Re-run on the final base. - Production build: the route string is gone from the bundle; `check-bundle-size.mjs` passes with no chunk renamed (JS gzip -0.18 kB on the same base). - `tsc --noEmit` passes; vitest for src/stores, src/api, src/features/patrol, src/components/AI and the Patrol page test passes; contract, registry and status audits pass. |
||
|
rcourtman
|
71a723abb1 |
Read controller-member disk history at its written key (#2694)
## Problem A physical disk behind a RAID controller (target such as `megaraid,3`) with no usable serial or WWN had its SMART history written under one key and read under another, so its temperature and SMART charts stayed empty. - Host agent: writes under `HostSMARTDiskMetricID`, e.g. `host-pve:sdd@ctrl0/megaraid,3`. - Proxmox: writes under `PhysicalDiskMetricID`, which returns `ProxmoxPhysicalDiskSourceID`, e.g. `pve-node1--dev-sdx:sdx@/megaraid,0`. - Reader: `BuildMetricsTarget` passed those source IDs, which already name the member, to `PhysicalDiskMetaMetricID`. That function appended the topology again: `host-pve:sdd@ctrl0/megaraid,3:sdd@ctrl0/megaraid,3`. The same happened in a registry rehydrated from persisted resources. ## Fix `PhysicalDiskMetaMetricID` now follows the Proxmox writer. A fallback that already ends in the member's topology is returned unchanged. The topology is appended only to a fallback without it: - a Proxmox source ID from before members were scoped, which `PhysicalDiskMetricID` scopes the same way; - the canonical resource ID that `PhysicalDiskView.MetricResourceID` falls back to. TrueNAS disks carry no controller target and are unaffected. ## Existing history - **SMART:** no SMART writer used the doubled key, so samples already stored become readable without a migration. - **Disk I/O:** a linked node's agent with no SMART reading for a member, such as a standby row, files the device's I/O under the Proxmox disk's metrics target. That target was the doubled key. The I/O now lands beside the member's SMART history. Samples already stored under the doubled key are left to age out: nothing reads them, and they are the block path's counter filed under its one merged member. - Mock seeding keys these disks by `disk.ID`, the single-scoped key, so it already matched. ## Proof - `TestIdentitylessControllerMembersReadTheirWritersHistory` (`internal/monitoring/issue1595_collection_trust_test.go`) drives the real writers: `writeHostSMARTMetrics`, `writeHostPhysicalDiskIOMetrics` and the Proxmox `writeSMARTMetrics`. It then queries the store at `MetricsTarget` for five members in a live registry and in one rehydrated through a JSON round trip. The members are two agent members sharing a path, a lone agent member with I/O, a Proxmox member, and a standby member merged into its linked node's Proxmox row. Before the fix all ten targets read the doubled key. - `TestPhysicalDiskMetaMetricIDScopesAControllerMemberOnce` (`internal/unifiedresources/canonical_ids_types_test.go`) pins each fallback shape against its writer: agent, smartctl-labelled agent, Proxmox, legacy Proxmox, canonical resource ID, and a disk that is no controller member. Before the fix, three cases failed with the doubled key. ## Stacking This branch sits on #2685, which sits on #2679, because both edit `physical_disk_ids.go` and the proof file. It supersedes #2661, which carried the same reader fix but conflicted with this stack and with main. |
||
|
rcourtman
|
843daf7ae3 |
Remove dead Patrol assessment and save-failure handoffs (#2682)
Stacked on #2674 (branch `refactor/drop-patrol-context-summary`); this branch contains that commit. ## What Removes the two Patrol-to-Assistant handoff builders in `frontend-modern/src/features/patrol/patrolInvestigationContextModel.ts` that no production code calls: - `buildPatrolAssessmentAssistantHandoff`: its caller, the summary card's Discuss handler, went in |
||
|
rcourtman
|
2eef42c512 |
Keep Off alert defaults off after saving (#2684)
## Summary
Switching a metric Off in a **Global Defaults** row (Alerts >
Thresholds) did not stick. After Save and a reload the metric was back
On at its factory threshold and kept alerting. This affected every
section whose factory threshold is positive: Machines CPU, memory, disk
and disk temperature, Proxmox node temperature, PBS, Docker, Kubernetes,
TrueNAS, vSphere and storage.
The editors stage `-1` for Off, and the save payload passed it through
as `{trigger: -1}`. Backend config normalization reads a negative
default trigger as unset and restores the factory value. Guest and node
CPU, memory and disk escaped only because nothing normalizes them.
- `buildAlertsConfigurationPayload` now writes every Off default as
`{trigger: 0, clear: 0}`, the value the engine and normalization both
read as off. Per-resource overrides are serialized separately and keep
`-1`, which the override path stores as written. A `-1` already stored
on a guest or node default becomes `0` on the next save.
- The Global Defaults **Custom** badge now treats `-1` and `0` as the
same Off value. Without that, a fresh install showed Custom on
Kubernetes, TrueNAS Systems and vSphere, and VMs & Containers turned
Custom after any save, though nothing had changed. An unset factory
value stays distinct from Off: an unset TrueNAS disk temperature follows
Disk temperature by type.
- A row's custom-threshold marker still compares raw values. It drives
the row's revert control and override removal, so a stored `-1` override
must stay visible against an Off default. A test pins that.
- The alerts contract no longer claims Off is written as `-1`
everywhere.
## Verification
- New payload test: every section's Off default is written as `{0, 0}`
and reads back Off. It fails on `main` with `guestDefaults.cpu: expected
{ trigger: -1 }`.
- New backend test: a `{0, 0}` default stays off through `UpdateConfig`
in every section, and a host at 99% raises no CPU, memory or disk alert
when the agent defaults are off. Both fail when the defaults are written
as `-1`.
- Live on a private mock stack. On `main`, Proxmox CPU and Temp switched
Off saved as `{-1}` and `{80, 75}`, and after reload Temp read On. With
the fix:
- Machines CPU at 1% raised 8 alerts. Switching it Off stored `{0, 0}`
and resolved all 8.
- Proxmox Temp Off resolved the node temperature alert.
- Nothing re-fired over two minutes.
- Values stayed Off after a backend restart and a reload. Machines,
Proxmox, Kubernetes, TrueNAS and vSphere were each checked at desktop or
phone width.
- Custom badge on a fresh install: no Proxmox, Kubernetes, TrueNAS or
vSphere group reads Custom, and after an unrelated save only the group
actually changed does.
|
||
|
rcourtman
|
60527fa8ab |
Drop contract prose for the deleted problem-resource surfaces
frontend-primitives.md still told new route-level copy to extend problemResourcePresentation, deleted in |
||
|
rcourtman
|
e3a93efcee |
Judge pulse-relaxed guests by one threshold on poll and save
CheckGuest lifted a pulse-relaxed guest's CPU, memory and disk triggers
to the 95/92/95 floor itself, but config-save reevaluation rebuilt the
guest from alert metadata without its tags and judged the alert by the
unrelaxed thresholds. A relaxed guest whose memory threshold was unset
got the floor of 92 while polling, read as no threshold on save, and
the save resolved the alert with a recovery notification; the next poll
raised it again.
The floor also lifted an Off threshold (trigger 0, or -1 from the
editors) like any low trigger, so tagging a guest relaxed turned on an
alert its config switched off, with no recovery band at all ({95, 0}).
The tag is documented as relaxing thresholds; it must never make a
guest more sensitive than its config.
- Apply the relaxed step in effectiveAlertPolicyNoLock's guest branch,
so CheckGuest, reevaluation and any later resolver share it.
- Read the recorded tags back in guestSnapshotFromAlert ([]string live,
[]interface{} after a restart) through a shared metadata reader that
storagePolicyAliases now uses too.
- Leave an Off threshold off; an unset one still gets the floor.
- Drop the tags key from an open alert once its resource has no tags.
Reevaluation now reads it, and a removed pulse-relaxed left behind
would judge the alert by the relaxed trigger while polling uses the
guest's own.
This stacks on the per-filesystem reevaluation fix. Without it, a save
judges a filesystem alert by the guest's relaxed floor instead of the
filesystem's own lower override, and flaps it;
TestConfigSaveKeepsRelaxedGuestFilesystemOverride guards that.
The other new tests fail against the base branch's runtime files via
-overlay.
|
||
|
rcourtman
|
dace9295c6 |
Remove exports orphaned by the summary component deletions (#2671)
Stacked on #2662 (its commits are ancestors of this branch, so this PR shows them until #2662 merges). The May cleanup deleted InfrastructureSummary, StorageSummary, the shared summary table and the SummaryMetricCard, InteractiveSparkline, DensityMap and SummaryPanel primitives, but left the helpers only they called. This removes them: - `getMetricsChartKeyCandidatesFromResource` (utils/agentResources.ts) - `getNormalizedIdentityLookupVariants` and `getAgentLikeIdentityAliases` (utils/resourceIdentity.ts) - `getPBSStatusIndicator` (utils/status.ts) - `useSummaryContextualFocusState` (components/shared/contextualFocus.ts), plus `filterSummarySeriesByGroupScope`, `resolveSummaryCardInteractionState` and `SummaryCardInteractionState` that only it used - the deprecated `ChartsAPI.getInfrastructureCharts` delegate; its branch tests now exercise `getInfrastructureSummaryCharts` Proof of deadness: a TypeScript import/export graph from `src/index.tsx` finds no production importer of any removed symbol, and a repo-wide grep finds no remaining reference. The production build differs from its parent only in the shared chunk that carries ChartsAPI, which loses the delegate's static method (99 bytes); after normalising chunk hashes, SRI hashes and minified identifiers every other file is identical. Tests: `src/utils`, `src/components/shared`, `src/api`, `src/features`, `src/hooks`, `src/__tests__`, Workloads, Storage and Infrastructure suites pass (only the known Mac-only `agentInstallCommand` failures). Contracts: frontend-primitives, storage-recovery and performance-and-scalability still described the deleted summary cards, sparkline, density map, summary panel, sticky summary section and Jump to row button as live owners, and credited contextualFocus.ts with the removed hook's series filtering. Those clauses now name the live owners (summaryCardInteraction.ts behind summaryTableFocus.ts, the storage summary cache behind the pool Growth column, TooltipPortal for the mini sparkline and backup activity chart tooltips), become conditional on a summary strip returning, or go. api-contracts and unified-resources record the removed chart client and chart-key helper. Browser pass (embedded production build, demo mock stack, 1440x900 and 390x844): Proxmox node-to-guests reveal, guest row hover emphasis and inline detail reveal, storage Growth column, pool row hover and inline detail, pool-group hover preview, pin and Escape clear. Receipt: `frontend-modern/browser-verification/232e5c23490f8cd4.json`. |
||
|
rcourtman
|
864a088db6 |
Fill the table's visible width with phone drawers
Inline table drawers below the lg breakpoint capped their sticky content at calc(100vw - 3.5rem), a guess at the page chrome. Pages with less chrome than that left an empty strip beside every drawer (27px at 390px on the Availability checks, Proxmox, Docker, Kubernetes and Machines tables), and a table wider than its shell at tablet widths got a drawer wider than the visible area. The table scroll shell is already a size container, so the content now caps at 100cqi: exactly the table's visible width at every width, measured live at 320, 390, 768 and 1000px. Desktop keeps the static uncapped layout. |
||
|
rcourtman
|
e747f2e57c |
Count down to future Patrol times instead of reading just now
formatRelativeTime is past-only: any time ahead of now reads "just now"
(compact) or "0s ago", which is right for clock skew on an observed
time. Patrol handed it three future times, so each read wrong for its
whole duration: a suppressed issue said "Returns to active attention
just now.", a will-fix-later finding said "Reminding just now", and a
snoozed finding said "snoozed until just now".
formatTimeUntil in utils/format.ts is the future counterpart. It counts
down ("in 3h", "in 1d"), rounds to the nearest unit so a 24-hour
suppression set a moment ago reads "in 1d" rather than "in 23h", and
returns "now" (or the caller's dueText) once the time arrives. The three
Patrol lines use it with the shared relative-time clock, because none of
those times change while the panel stays open, and keep the exact time
on hover.
The suppression line names the state expiry restores. An open issue
reads "Returns to active attention in 1d."; one marked reviewed reads
"Suppression ends in 1d. It stays reviewed.", because expiry restores
Acknowledged, which is not active attention; a passed expiry reads
"Suppression has ended.". The countdown takes its words from
getAttentionSuppressionEndPresentation, which already words the
early-end button and the duration field by the same reviewed split, so
all three agree.
A reminder reads "Reminder overdue" once one overdue id list, sampled on
the shared clock, holds it. The Overdue commitments chip and filter
count the same list, so the label, chip and filter agree even for a row
that mounts between ticks (it reads "Reminder due" until the next tick),
and the list only notifies when its ids change, so a tick that moves no
deadline past now does not rebuild open rows. The reminder hover no
longer promises the finding returns only "if it is still tripping": the
hourly reminder sweep brings it back regardless. An ended snooze reads
"snooze ended" until the next findings re-read drops the snoozed status.
The existing countdowns did not fit: CountdownTimer runs its own
one-second interval and has no day unit (a 7-day suppression would read
"167h 59m"), and the replication Next sync helper is private to that
table with sync-specific overdue tones.
An audit of every formatRelativeTime and
formatPlatformTableRelativeTimeValue caller found no other future
timestamps; the autopilot and alert snooze expiries already show an
absolute time.
This rebuilds PR 2613 on main after PR 2598 landed. PR 2613 was stacked
on two early PR 2598 commits whose relative-age contract wording main
has since replaced (every read of the shared clock now returns the wall
clock), so the future-time note sits under main's wording and before the
read-time age exception, and that exception now opens "The relative-age
rule" so it does not read as an exception to the future-time note.
|
||
|
rcourtman
|
d02d6441dd |
Merge origin/main into fix/unraid-disk-temperature-alerts
Main's Proxmox node cleanup section landed at the same alerts.md anchor as the Unraid disk temperature section; both are kept, main's first. |
||
|
rcourtman
|
539eadfab2 |
Judge filesystem alerts by their own threshold on config save
A config save re-judges every active alert. Agent filesystem alerts were re-judged by the host disk threshold, ignoring the filesystem's own override and the per-type DiskFillByType threshold CheckHost uses. Guest filesystem alerts were resolved by their "<guestID>-disk-<key>" resource ID, which matches no guest-disk override and names no guest, so the guest's own override was missed too. A filesystem override of 70 raised an alert at 80 under a default of 90; saving any alert setting resolved it and sent a recovery, and the next poll raised it again as a new alert. CheckHost and CheckGuest now resolve a filesystem's threshold through hostDiskUsageThresholdNoLock and guestDiskUsageThresholdNoLock, and reevaluation calls the same helpers: resolveHostAlertThresholdsNoLock for agent filesystem alerts, and a new resolveGuestAlertThresholdsNoLock that resolves a guest filesystem alert's guest by the ID parseGuestAlertIdentity reads from its resource ID. The helpers return override and per-type thresholds as owned copies, so evaluation still never normalizes the live config. TestConfigSaveJudgesFilesystemAlertsByTheirOwnThreshold covers an agent filesystem override, an NVMe fill threshold, a guest filesystem override and a guest override. With host.go, guest.go and config_runtime.go overlaid by their origin/main or #2696 copies, all four fail: the unchanged save resolves the alert. Stacked on #2696. |
||
|
rcourtman
|
aa7053a07e | Merge remote-tracking branch 'origin/main' into fix/alert-config-keep-unsent-fields | ||
|
rcourtman
|
ac631eadb8 |
Alert on the Unraid temperature an agent disk shows
A host agent's disk can show a temperature only its Unraid inventory reports: a SMART row whose smartctl probe returned none takes its Unraid row's reading, and a disk only the inventory lists is built from that row alone. The Physical Disks Temp cell, the Running Hot verdict and Patrol judge that reading against the alert disk temperature policy, but CheckHost read only the SMART row's own field, so a disk could read as at its alert threshold while no alert fired. unifiedresources.HostDiskTemperatureReadings now lists the reading each of a host's disks shows, built with the registry's own adapters: one per SMART row, then one per Unraid row whose disk key no SMART row carries (the history writer's rule). CheckHost judges those. SMART rows keep their alert resource IDs, an inventory-only disk uses its block device token, and per-type thresholds follow the disk type the disk shows. Only a reading collected now fires or resolves, so spun-down disks hold their alerts. When conflicting serials put two disks on one device, its alert judges the reading that stands highest against its own type's threshold. An inventory-only disk the agent's --disk-exclude patterns match raises nothing, and an alert it raised before the exclusion resolves on the next report, as the documented exclusion from monitoring already suppresses linked Proxmox health and wear alerts. Absence cleanup still runs only on a non-empty SMART list. Unraid's own disk temperature notifications do not make this redundant: Pulse already alerts on the smartctl readings and array state of the same hosts, which Unraid also notifies on. |
||
|
Richard Courtman
|
7b884bc98a |
Chart the Unraid temperature an agent disk shows
A host agent's disk can show a temperature only its Unraid inventory reports. The adapter takes the Unraid row's reading when a SMART row has none, which happens when the agent's smartctl probe returns no temperature for a disk Unraid reads with its own per-disk SMART settings, and a disk the agent sends no SMART row for, such as a --disk-exclude match, is built from its Unraid row alone. The only agent disk temperature writer took the SMART row's own reading, and the unified sync skips agent disks as natively written, so these temperatures were shown live but never charted. writeHostSMARTMetrics now writes the reading the disk shows (HostSMARTDiskTemperature, the adapter's own rule), and a new pass writes each Unraid row's reading under HostUnraidDiskMetricID, the key its disk's metrics target reads, unless that key is one of the host's SMART rows' keys. An Unraid reading therefore never overwrites a SMART row's own at the same timestamp. Unraid rows sharing a key are one disk in the registry, which shows the latest of their readings, so the last row with a reading decides the sample. TemperatureCollected still keeps spun-down and expired-host readings out of history. Charting the fallback makes its row matching matter, so the fallback and the serial a row without one takes now need an Unraid row that describes the disk: one carrying the row's usable serial, or one naming the row's device path when the row is that whole block device, not a controller member (even the only one reported) or one of several rows on the path. matchUnraidDisk now tries usable serials across the whole inventory before device paths, and a placeholder serial matches nothing, so a row no longer takes another disk's Unraid row through a shared placeholder serial or an earlier path match. A lone member without a serial or WWN now charts under its own topology key, which its metrics target reads as written. |
||
|
rcourtman
|
7eb32f3181 |
Keep a low alert default's clear below its trigger (#2678)
## Problem
Setting an alert global default to a positive trigger of 5 or less, for
example Machines CPU at 1% in Alerts > Thresholds, stored
`agentDefaults.cpu` as `{trigger: 1, clear: 75}`, a clear far above its
trigger.
The thresholds page sends `clear = max(0, trigger - 5)`, which is 0 for
those triggers. `NormalizeAgentDefaults` derived `trigger - 5`, found it
non-positive, and fell back to the factory clear (75/80/85).
`ValidateHysteresisThresholds` never checked agent CPU, memory or disk,
so the pair survived. PBS CPU/memory, node temperature and
`internal/config/persistence.go` had the same fallback. On the save
path, validation repaired PBS and node temperature, but
`LoadAlertConfig` still turned a stored node temperature clear of 0 into
75; at startup `UpdateConfig` repaired that in memory.
Per-type `DiskFillByType` / `DiskTempByType` entries were worse: any
clear ≤0 reset the whole pair. A SATA disk temperature trigger of 4 from
the thresholds page was stored as 55, and an API-written fill entry of
`{88, 0}` became `{90, 85}`.
## What a user saw
- **Firing and resolving did not change for these pairs.**
`buildCanonicalMetricSpec` drops a clear at or above the trigger, and
the reducer then clears at the trigger. That held both before and after
this change. There was no flapping and no alert that never re-armed.
- `/api/alerts/config` and `alerts.json` served the factory clear above
the trigger.
- Firing agent alerts carried it as `metadata.clearThreshold`: 75 on a
1% CPU alert, and 85 on a disk alert still firing at 81.8%.
- A per-type disk trigger of 1–5 was silently replaced by the type
default.
- Machines metric bars: `resolveThreshold` in `metricThresholds.ts`
clamps the warning level to `min(clear, trigger)`, so a 1–5% agent
trigger had no warning band. Readings below the trigger, including 0,
now show amber instead of green, which matches guest, node, Docker and
Kubernetes bars at the same setting. This one is read from the code; I
did not exercise it in a browser.
## Fix
- PBS, node temperature, agent and per-type disk defaults now use
`normalizeThresholdPointer`, the normalizer Kubernetes, TrueNAS and
vSphere already use. A missing clear sits five points under the trigger,
floored at 0. A clear at or above the trigger is repaired. Zero still
means off for global defaults, and nil or negative still takes the
factory default. For the canonical per-type keys (the types the page
edits), a non-positive trigger still resets to the type default, because
the page cannot represent a per-type Off; other per-type keys are stored
as written, as before.
- `internal/config/persistence.go` drops its duplicate
agent/node-temperature and storage helpers and calls the same normalizer
through a new `NormalizeHysteresisThreshold`. A config saved before this
fix loads repaired.
- Agent CPU/memory/disk/disk temperature are added to
`ValidateHysteresisThresholds`, as every other family already is.
- One deliberate behaviour change for invalid pairs above the margin: an
API-written `{50, 80}` is repaired to `{50, 45}`, so it now has a real
recovery band where it used to clear at the trigger. This matches what
`EnsureValidHysteresis` already does for every other family.
- An alerts contract clause records the rule.
## Verification
- `internal/alerts/config/normalize_low_trigger_clear_test.go` runs
every default family and both per-type maps through the `UpdateConfig`
normalizer chain.
- `TestUpdateConfigKeepsLowTriggerClearBelowTrigger` pins the saved
config and a firing alert's `clearThreshold`.
- `persistence_test.go` checks the written `alerts.json` itself as well
as load repair.
- All new tests fail against the previous normalizers. Coverage cases
that pinned the old fallback (including one labelled "suspected source
bug") are updated.
- `go test ./internal/alerts/... ./internal/config/ ./internal/ai/`
passes.
- Live, on private mock stacks (baseline binary vs this branch): I saved
the payload the thresholds page sends, then read back
`/api/alerts/config`.
- Baseline: agent `{1,75}`, `{3,80}`, `{5,85}`; `diskTempByType.sata`
`{55,50}` for a 4°C trigger; firing agent alerts reported
`clearThreshold` 75/80/85.
- This branch: every pair `{t, 0}`, `diskTempByType.sata` `{4, 0}`, and
alerts reported 1/3/5.
- After a restart, this branch also repaired a hand-written
`alerts.json` holding `{1,75}` and node temperature `{2,75}`.
|
||
|
rcourtman
|
71a470129f |
Stop alert evaluation writing live thresholds outside the lock
Host and guest evaluation filled in the Clear of a per-disk override's threshold after dropping m.mu, writing through the live config's pointer. GetConfig clones that pointer under the read lock, which does not exclude the writer, and the saved override silently took whatever the last poll filled in. Docker evaluation pointed into m.config.DockerDefaults, and read the restart-loop and memory-limit settings, with no lock held while UpdateConfig replaced m.config. Clone the override's Disk threshold under m.mu before normalizing it, resolve Docker container thresholds from cloned defaults plus the override under the lock, and read the restart-loop and memory-limit settings under it. Drop getThresholdForMetricFromConfig: nothing has called it since the v6 split, and it normalized in place. TestAlertEvaluationLeavesTheLiveConfigToTheLock runs host, guest and Docker evaluation against GetConfig() and UpdateConfig(GetConfig()) under -race. With host.go, guest.go, docker.go and metric_runtime.go overlaid by their origin/main copies it fails: the live override's clear reads 85, and -race reports every site above. Stacked on #2692, which made GetConfig return an owned copy. |
||
|
rcourtman
|
5c55197900 |
Remove test-only Patrol context summary helpers (#2674)
Removes two exports from
`frontend-modern/src/features/patrol/patrolInvestigationContextModel.ts`
that only tests still called, and corrects the contract prose that
described the removed Patrol supporting-context surface as live.
## Why
`buildPatrolInvestigationContextSummary` (with its input type) and
`selectPatrolSupportingRecentChanges` lost their last production caller
in
|
||
|
rcourtman
|
ef45926b12 | Merge commit 'cd4e564fbf' into fix/proxmox-disk-temperature-alerts | ||
|
rcourtman
|
cd4e564fbf | Merge remote-tracking branch 'origin/main' into fix/disk-heat-remaining-judges | ||
|
rcourtman
|
409c5482ee |
Read controller-member disk history at its written key
A physical disk behind a RAID controller (target such as megaraid,3) with no usable serial or WWN had its SMART history written under one key and read under another, so its temperature and SMART charts stayed empty. The host agent writes under HostSMARTDiskMetricID and Proxmox under PhysicalDiskMetricID, and both keys already scope the member (host-pve:sdd@ctrl0/megaraid,3). BuildMetricsTarget passed that source ID to PhysicalDiskMetaMetricID, which appended the topology again (host-pve:sdd@ctrl0/megaraid,3:sdd@ctrl0/megaraid,3). The reader now does what the Proxmox writer does: a fallback that already names the member is returned unchanged, and only a fallback without the topology gets it appended. That covers a Proxmox source ID from before members were scoped and the canonical resource ID a view falls back to. TrueNAS disks carry no controller target and are unaffected. SMART history needs no migration: no SMART writer used the doubled key, so samples already stored become readable. A linked node's agent with no SMART reading for a member, such as a standby row, files the device's I/O under the Proxmox disk's metrics target, which was the doubled key. That I/O now lands beside the member's SMART history, and samples already stored under the doubled key are left to age out. This carries forward the fix from PR 2661, whose branch conflicts with the disk ID stack it now sits on, and proves it against the real SMART and I/O writers in the live and rehydrated registries. |
||
|
rcourtman
|
ad1e34277d |
Stop alert config saves writing into the live config
SaveAlertConfig normalized the config it was handed in place, and its
callers hand it alerts.Manager.GetConfig(), a shallow copy whose maps
and threshold pointers were the running config's own. Alert activation
and the alert identity migration therefore wrote the live delay map and
threshold pointers outside the manager lock while evaluation read them.
The shallow copy also let UpdateConfig, which normalizes the override
map it is handed in place, rewrite the live overrides while
GET /api/alerts/config encoded another snapshot. A concurrent map read
or iteration plus a write is a fatal Go runtime error, so these paths
could kill Pulse.
AlertConfig.Clone is a typed deep copy. GetConfig returns one, so
callers own what they encode, persist and edit, and SaveAlertConfig
clones before normalizing, because a config just applied with
UpdateConfig is the live config. GetConfig has no per-resource caller
(once per poll cycle, backup check, request or Patrol run); a clone
with 1,000 overrides takes about 0.3 ms. A test fills every field and
fails when a new map, slice or pointer is not copied.
Persistence also stored and loaded an explicit 0 type-level alert delay
("no delay") as the 5-second factory delay, which the noisy-gauge rule
stretches to 300 seconds for memory and temperature, while the manager
keeps 0. The shared map write was what made the running manager agree
with the file; without it, memory would say 0 and disk 5. Persistence
now keeps 0 on save and load like NormalizeTimeThresholds, and only a
missing or negative delay takes the default. Pulse saves never wrote a
0, and the first v6 start re-saves alerts.json, so only a newly sent or
hand-edited 0 changes meaning.
On main, a reproducer taking the activation, migration and GET steps
against live evaluation died with "fatal error: concurrent map
iteration and map write" in 5 of 5 plain runs and reported 16 races
under -race. It passes with this change.
|
||
|
rcourtman
|
22b8b995ca |
Read the shared wall clock on Alerts overview cards (#2673)
The overview hook kept its own minute signal and handed its value to each card as "now". A card that mounted between ticks measured from up to a minute earlier: an alert raised 70 seconds ago read "this minute", a reading already past the 10-minute stale cut-off led the card as the live one, a just-raised alert stayed out of Triggered (24h) until the next tick, and the stale flip and 24h drop landed up to a minute late. The card now reads useRelativeTimeNow for its started age and the live reading's stale cut-off, and the hook reads it for the 24h count, so they measure from the wall clock and move with the shared 30-second tick. The minute interval stays only for the delivery diagnoses refresh, a server read that should not double its rate. |
||
|
rcourtman
|
2d6a2937fc |
Keep platform metric alerts through Proxmox node cleanup (#2672)
`CleanupAlertsForNodes` runs on every Proxmox node poll
(`monitor_pve_storage.go`) and on every mock alert tick. It removes
alerts whose `Node` is not a current Proxmox node or PBS instance, and
keeps the types `shouldPreserveAlertOutsideNodeCleanup` lists.
Kubernetes, TrueNAS and vSphere metric alerts carry a cluster or
platform host as `Node`, and a display label (`TrueNAS Disk`,
`Kubernetes Pod`, `vSphere VM`) in `metadata.resourceType`. The preserve
list only had the bare words `truenas`, `kubernetes`, `k8s` and
`vmware`, so none of these labels matched. In an estate with Proxmox
plus one of those platforms:
- every Proxmox poll deleted these alerts without resolving them;
- their reducer incident stayed firing, so the next unified evaluation
re-created each one and handed it to the alert callback again as a new
alert.
## Change
`shouldPreserveAlertOutsideNodeCleanup` keeps an alert that carries
`metadata.platformType` (set on every unified Kubernetes/TrueNAS/vSphere
metric alert), or whose label maps through `CanonicalAlertResourceType`
to one of those platform types. The label path covers alerts persisted
before `platformType` existed. A guest alert whose Proxmox node left the
estate is still removed.
## Evidence
- Unit probe on main: a TrueNAS disk temperature alert, a Kubernetes pod
CPU alert and a vSphere VM CPU alert were all deleted by
`CleanupAlertsForNodes({"pve1": true})`. Over three
cleanup-then-evaluate cycles the TrueNAS alert went to the alert
callback 3 times; with this change, 0 times and it stays active.
- Private mock stack: TrueNAS disk temperature alerts kept vanishing
between mock alert ticks, while the disks kept their readings in
`/api/resources`. With this change they stay open.
## Tests
`TestCleanupAlertsForNodesKeepsPlatformMetricAlerts` (in
`internal/alerts/alerts_test.go`) raises a TrueNAS disk, a TrueNAS pool,
a Kubernetes pod and a vSphere VM alert, then runs the cleanup three
ways: as raised, with only `platformType` (label unrecognised), and with
only the label (`platformType` removed).
`TestCleanupAlertsForNodesStillRemovesMissingProxmoxNodeAlerts` keeps
the removal of a guest alert on a removed Proxmox node. Each preserve
condition was mutation-checked: removing either one fails the first
test.
|
||
|
rcourtman
|
e30b44ea8f |
Read a silent in-guest agent's guest drawer readings as last known (#2669)
A Pulse agent linked into a VM keeps its last report on the VM resource after it stops reporting: the registry copies the agent's whole facet, `agent.stale` included, onto the guest. The guest drawer never read that flag. A quiet agent's Guest RAID kept its live colours and "Rebuild: 42% · 118M/sec", its network rows and version read as current, and nothing said the agent had gone. ## Change - `useWorkloads` maps `agent.stale` onto the workload as `agentStale`, and sets `disksFromAgent` when Proxmox reported no guest filesystems and the agent's own `disks` filled in. Filesystem rows are the only agent reading with a second source. - When the agent is stale, the guest drawer: - passes `HOST_AGENT_STOPPED_REPORTING_REASON` to `RaidCard` (`lastKnownReason`, #2652); - puts a status line over muted Network rows; - marks the Pulse Agent row "· stopped reporting"; - reads agent-supplied filesystem rows through the existing `last-known` disk read state. - Agent-supplied filesystem rows now follow the agent's freshness instead of the Proxmox guest-read reason. Before, a VM with a **live** linked agent but no qemu-guest-agent showed the agent's filesystems as "Usage unavailable · ?/328 GB" with advice to install qemu-guest-agent. A stopped guest still reads those rows as unavailable, whether its status or the Proxmox `vm-stopped` reason says so. Proxmox's own guest filesystems keep their read state. ## Verification - Private mock stack built from `main` plus this change. Two host agents were held past their reporting lease and two left live, then each was linked into a VM through `POST /api/resources/{vm}/link`. The two silent VMs serve `agent.stale: true` with the agent's raid, disks and interfaces; the two live ones carry no stale flag. - Playwright at 1440 and 390: - silent VMs read as described above; - the live VM with agent-only filesystems shows usage with bars; - the live control is unchanged; - no horizontal overflow. - New tests in `GuestDrawer.test.tsx` cover: - silent agent; - live agent with Proxmox agent-not-running; - Proxmox filesystems under a silent agent; - stopped guest, with and without the reason; - stale flip in both directions on an open drawer. The new tests fail against the old drawer. `useWorkloads.test.ts` asserts source selection with distinct mountpoints. - Frontend suites over Workloads, hooks, utils, shared, platformPage and Infrastructure pass, apart from the Mac-only PTY tests in `agentInstallCommand.test.ts`. - WorkloadsSurface grows by 0.22 kB gzip (64.85 kB against a ~71.9 kB cap), and the chunk set is unchanged. ## Not covered - The workload table's disk cell and the drawer History still read the aggregate disk metric with the Proxmox reason. - The agent's OS, its addresses in the identity rows, and the lazily loaded Physical Disks card's health and wearout carry no stale cue yet. |