mirror of
https://github.com/rcourtman/Pulse.git
synced 2026-10-26 05:29:45 +00:00
Preserve reviewed monitoring repairs with current upstream thresholds
Additively incorporate exact upstream36d3f8c557while retaining all accepted local commits and proposal31eccb559e. Resolve the single adjacent insertion conflict by retaining both reviewed observation-gap interruption and upstream metric tag cleanup. Preserve upstream shared guest and disk-type threshold judgement, phone drawer sizing, future Patrol timing and POST-only Assistant mutations. Exact final validation and outstanding browser/security dispositions remain required before advancing publication. Change-source: pulse-maintainer
This commit is contained in:
pulse-triage[bot]
@@ -275,6 +275,18 @@ Without comparable identity (a SAS, SCSI or USB replacement with no WWN on one
|
||||
side) the row still matches. Report admission and the SMART wire contract are
|
||||
unchanged; the regression is
|
||||
`TestHostAgentSMARTRowForSwappedOutDiskDoesNotLendItsIdentity`.
|
||||
A PVE disk that a still-reporting linked Agent's SMART row matches is marked
|
||||
`AgentSMARTReported` (internal poll evidence on `models.PhysicalDisk`, never
|
||||
serialized); disks the poller builds from that report when the Proxmox disk
|
||||
query fails go through the same merge, and an Agent whose lease lapsed owns
|
||||
none. The Agent then owns the disk's temperature alert
|
||||
through `CheckHost`, and the PVE disk temperature check closes its own as
|
||||
moved to the Agent, so one disk never alerts twice. Report admission, the SMART
|
||||
wire contract and Agent authority are unchanged;
|
||||
`TestMergeHostAgentSMARTIntoDisksMarksDisksTheAgentReports` in
|
||||
`internal/monitoring/physical_disk_roundtrip_test.go` and
|
||||
`TestPhysicalDiskAgentSMARTReportedStaysInternal` in
|
||||
`internal/models/deepcopy_test.go` pin the marker.
|
||||
|
||||
Assistant historical metric wiring uses the current monitor's retained store
|
||||
and registry metrics coordinates. Historical reads do not alter enrollment,
|
||||
@@ -4797,6 +4809,12 @@ agent-lifecycle operations. If those routes are called directly, the API must
|
||||
fail them as unsupported rather than presenting file diffs or reverts as
|
||||
agent command rollback, enrollment repair, update rollback, or fleet-control
|
||||
authority.
|
||||
Assistant session mutations (`/abort`, `/summarize`, `/fork`, `/undo`,
|
||||
`/redo`, `/steer` under `/api/ai/sessions/{id}`) accept only `POST`; any other
|
||||
method that reaches those handlers gets `405` before touching session state.
|
||||
That method guard
|
||||
is AI-runtime transport hardening in `internal/api/ai_handler.go`; it adds no
|
||||
agent command, enrollment, update, or fleet-control path.
|
||||
That same shared dependency now also assumes hosted cloud handoff authorizes
|
||||
tenant org access before browser lifecycle continues. Lifecycle-adjacent opens
|
||||
into hosted workspaces may depend on `internal/api/cloud_handoff_handlers.go`,
|
||||
|
||||
@@ -3570,6 +3570,12 @@ query...`, and `Reading storage...` before streamed tool arguments are
|
||||
answer (nothing streaming, no error block, no pending approval or
|
||||
question, and a user prompt available to re-send) because the undo
|
||||
boundary only operates on the last durable turn.
|
||||
Session mutations stay POST-only at the handler: abort, summarize, fork,
|
||||
undo, redo, and steer refuse every other method with `405` and
|
||||
`Allow: POST` before resolving the Assistant service. A `GET` or `HEAD`
|
||||
that the demo-mode read-only guard or the CSRF check admits as a read
|
||||
therefore cannot cancel a running turn, compact or rewind a transcript,
|
||||
or persist a fork.
|
||||
The empty Assistant drawer may surface recent non-empty sessions as direct
|
||||
resume actions using the backend session list already owned by the drawer;
|
||||
it must not create a parallel recent-chat store or product-authored prompt
|
||||
|
||||
@@ -1529,7 +1529,8 @@ That same guest-threshold owner also governs guest-derived lifecycle and
|
||||
posture alerts. Snapshot age, backup age, powered-off state, and
|
||||
configuration-change reevaluation must all construct a canonical lightweight
|
||||
guest snapshot and route threshold resolution through the shared
|
||||
guest-defaults → filter-driven custom rules → guest-override chain.
|
||||
guest-defaults → filter-driven custom rules → guest-override → `pulse-relaxed`
|
||||
chain.
|
||||
That canonical guest context must preserve the live guest name and tags for
|
||||
snapshot and backup posture evaluation. Ignored prefixes, `pulse-no-alerts`,
|
||||
configured ignored tags, and required-tag filtering must resolve through the
|
||||
@@ -1539,6 +1540,27 @@ operator's suppression policy.
|
||||
Passing `nil` guest context or resolving only overrides/defaults is forbidden
|
||||
because it silently bypasses custom guest rules and makes guest lifecycle
|
||||
alerting diverge from running-guest metric truth.
|
||||
The `pulse-relaxed` step lives in `effectiveAlertPolicyNoLock`, not in
|
||||
`CheckGuest`, so polling and config-save reevaluation judge a relaxed guest's
|
||||
alert by one threshold. Reevaluation rebuilds the guest from alert metadata,
|
||||
so `guestSnapshotFromAlert` reads back the `tags` the evaluator recorded
|
||||
(`[]string` live, `[]interface{}` after a restart). Without them a save judged
|
||||
the alert by unrelaxed thresholds, resolved one raised under an unset
|
||||
threshold's relaxed floor with a recovery notification, and the next poll
|
||||
raised it again. Those tags must stay current: `metricOptionsWithTags` drops
|
||||
the key from an open alert once its resource has no tags, or a removed
|
||||
`pulse-relaxed` would keep judging the alert by the relaxed trigger. The tag
|
||||
lifts CPU, memory and disk triggers to a 95/92/95 floor and gives an unset
|
||||
threshold that floor, but an Off threshold (trigger <= 0) stays Off: relaxing
|
||||
a guest must never raise an alert its config turned off. A filesystem's own
|
||||
override still replaces the relaxed guest disk threshold for that filesystem,
|
||||
on save as while polling. Proof:
|
||||
`TestConfigSaveJudgesRelaxedGuestAlertByRelaxedThresholds`,
|
||||
`TestConfigSaveKeepsRelaxedGuestFilesystemOverride`,
|
||||
`TestPulseRelaxedKeepsOffGuestThresholdOff` and the Off case of
|
||||
`TestApplyRelaxedGuestThresholds` in `internal/alerts/alerts_test.go`, and
|
||||
`TestCheckGuestDropsRemovedTagsFromOpenAlerts` in
|
||||
`internal/alerts/unified_eval_test.go`.
|
||||
That same guest-alert owner also has to retire per-disk guest alerts when the
|
||||
guest stops, disk alerting is disabled, or the reported disk set changes.
|
||||
Canonical guest disk identity is only valid while the guest still exposes that
|
||||
@@ -1907,14 +1929,84 @@ A disk is hot from its trigger. Alerts and Patrol findings stay open until the
|
||||
reading falls to the clear value, or under the trigger when there is no band
|
||||
below it. The Physical Disks verdict and the TrueNAS Health cell judge only the
|
||||
current reading against the trigger. PDF reports colour disk temperatures by
|
||||
the same thresholds. Judges that still differ: TrueNAS disk temperature alerts
|
||||
use `TrueNASDiskDefaults.Temperature` (a flat 55/50), and the TrueNAS disk
|
||||
drawer tones the reading from a fixed 55C.
|
||||
the same thresholds.
|
||||
`TestDiskTemperatureThresholdMatchesCheckHostPolicy` in
|
||||
`internal/alerts/threshold_resolution_shared_test.go` pins per-type resolution, a raised
|
||||
NVMe trigger that `CheckHost` also honours, the copy, the disabled default and
|
||||
the nil manager.
|
||||
|
||||
TrueNAS disk temperature alerts follow the policy too:
|
||||
`trueNASDiskTemperatureDefaultNoLock`
|
||||
(`internal/alerts/truenas_disk_temperature.go`) is the type-default tier of
|
||||
`effectiveAlertPolicyNoLock` for `truenas-disk`, so an unset
|
||||
`TrueNASDiskDefaults.Temperature` resolves per disk type (the
|
||||
`alertPolicyQuery.DiskType` the unified input carries from
|
||||
`PhysicalDisk.DiskType`). A TrueNAS-wide value the user saved under TrueNAS
|
||||
Disks replaces it for every TrueNAS disk, as a host Disk Temp override does for
|
||||
an agent's disks, 0 meaning off, and a per-disk override beats both. The alert
|
||||
records `diskType` so a config save re-judges it per type; one without it is
|
||||
held to the lowest per-type trigger. A retained reading
|
||||
(`diskinventory.TemperatureCollected` false) is no evidence and raises nothing.
|
||||
The factory value is unset. `normalizeTrueNASDiskTemperature` drops a stored
|
||||
55/50 only from a config without `TrueNASDiskTemperatureByType`, because the
|
||||
thresholds page wrote that flat factory value back on every save; the page
|
||||
writes the marker, so a 55 typed later is kept. Every config `GetConfig`
|
||||
returns carries the marker, so a client that reads and writes back keeps it.
|
||||
Without the marker (an old saved file, or a page loaded before the upgrade)
|
||||
55/50 is read as that unchosen factory value; a client building a config from
|
||||
scratch with an explicit TrueNAS-wide 55 has to send the marker. The TrueNAS
|
||||
Disks Global Defaults cell reads `By type` while unset
|
||||
(`globalDefaultFallbacks` on `ResourceTable`,
|
||||
`resolveAlertResourceGlobalDefaultCell`), or Off when the agent Disk Temp
|
||||
default switches the policy off, and switching it on then stages an explicit
|
||||
TrueNAS-wide 55 because unset would stay off. Each disk row inherits its type's
|
||||
trigger from the unsaved editor state (`resolveTrueNASDiskTemperatureDefault`).
|
||||
The TrueNAS disk drawer tones a current reading from the same per-type trigger
|
||||
(`getDiskTemperatureThresholds` passed into `buildTrueNASDetailSections`), so it
|
||||
no longer judges heat from a fixed 55C. Judges that still differ: the TrueNAS
|
||||
storage table and drawer do not apply a TrueNAS-wide value or per-disk
|
||||
override.
|
||||
`TestTrueNASDiskTemperatureAlertsFollowDiskTemperaturePolicy` and
|
||||
`TestTrueNASDiskTemperatureAlertIgnoresRetainedReading` in
|
||||
`internal/alerts/unified_eval_test.go` pin the tiers and retained readings;
|
||||
`TestTrueNASDiskTemperatureAlertsReevaluatePerDiskType` and
|
||||
`TestTrueNASDiskTemperatureAlertWithoutDiskTypeHeldAtLowestTrigger` in
|
||||
`internal/alerts/alerts_test.go` pin re-judging on a config save;
|
||||
`TestNormalizeTrueNASDiskTemperatureKeepsOnlyChosenValues` pins the
|
||||
saved-config migration.
|
||||
|
||||
Proxmox physical disks alert on temperature under the same policy.
|
||||
`CheckProxmoxDiskTemperature` (`internal/alerts/proxmox_disk_temperature.go`)
|
||||
raises a `diskTemperature` metric alert under the disk's PVE alert reference
|
||||
(`ProxmoxPhysicalDiskAlertResourceID`, resource type `proxmox-disk`), judged by
|
||||
`proxmoxDiskTemperatureThresholdNoLock`: the disk type's `DiskTempByType` entry
|
||||
under the agent Disk Temp default, which switches it off for every type. It
|
||||
carries the disk's serial, WWN and `diskType` like the health and wearout
|
||||
alerts, so `ProxmoxPhysicalDiskAlertIdentifiers` lists its
|
||||
`<reference>::metric-threshold:diskTemperature` identifier for history
|
||||
ownership, and `reevaluateActiveAlertsLocked` re-judges it per disk type (the
|
||||
lowest per-type trigger when the type was never recorded) instead of falling
|
||||
through to guest thresholds that have no disk temperature. A disk a
|
||||
still-reporting linked agent lists in its SMART report is the agent's: its
|
||||
alert closes with the `moved_to_agent` resolution and `CheckHost` raises the
|
||||
agent's own. The handover happens on the next disk poll, so for that one
|
||||
interval an agent newly linked to a disk with an open PVE alert can hold its
|
||||
own alert beside it. A lapsed agent owns nothing, so the node's own readings
|
||||
keep alerting while its agent is silent. An
|
||||
excluded disk, or a policy switched off, closes the alert; a poll without a
|
||||
current reading holds it. Like the health and wearout alerts, a disk that
|
||||
leaves the inventory keeps its alert until the 24-hour stale sweep. The
|
||||
`proxmox-disk` alert delay defaults to the factory 5 seconds, so a warning
|
||||
waits out the same 300-second noisy-gauge stability window agent and TrueNAS
|
||||
disk temperature warnings do, and a critical reading bypasses it.
|
||||
`TestProxmoxDiskTemperatureAlertsFollowDiskTemperaturePolicy`,
|
||||
`TestProxmoxDiskTemperatureAlertOwnershipAndHolds`,
|
||||
`TestProxmoxDiskTemperatureWarningWaitsOutStabilityWindow`,
|
||||
`TestProxmoxDiskTemperatureAlertsReevaluatePerDiskType` and
|
||||
`TestCheckDiskHealthAlertsCarryHistoryOwnershipIdentity` in
|
||||
`internal/alerts/alerts_test.go` pin the policy, ownership, holds, re-judging
|
||||
and identifiers.
|
||||
|
||||
### Agent disk temperature alerts clear when their disk leaves the report
|
||||
|
||||
`CheckHost` also clears a disk temperature alert, and drops its pending
|
||||
|
||||
@@ -2161,6 +2161,20 @@ payload shape change when the portal presents compact client rows.
|
||||
index; clients must treat the endpoint's `accepted:true` as inbox
|
||||
receipt, not delivery. Browser clients must use the shared
|
||||
`AIChatAPI.steerSession(sessionId, { prompt, clientMessageId })` helper.
|
||||
Every Assistant session mutation sub-resource under
|
||||
`/api/ai/sessions/{id}` (`/abort`, `/summarize`, `/fork`, `/undo`,
|
||||
`/redo`, `/steer`) accepts only `POST`. Any other method that reaches the
|
||||
handler returns `405` with `Allow: POST` through
|
||||
`requireAssistantSessionMutationMethod` in `internal/api/ai_handler.go`,
|
||||
before the session service is resolved (earlier layers still answer
|
||||
first: `Router.ServeHTTP` returns `200` for `OPTIONS` preflight, and
|
||||
auth, scope, CSRF, and demo-mode guards keep their own refusals). The
|
||||
session router dispatches these sub-resources by path alone, and the
|
||||
demo-mode read-only guard and the CSRF check both treat `GET` and `HEAD`
|
||||
as reads, so the method contract belongs to the mutating handler rather
|
||||
than to its callers. `GET /api/ai/sessions/{id}/messages` stays the read
|
||||
path, and the unsupported `/diff`, `/revert`, and `/unrevert` handlers
|
||||
keep answering `501` for every method that reaches them.
|
||||
OpenCode-style file diff/revert session routes are deliberately not part
|
||||
of Pulse's supported Assistant session contract: Pulse sessions do not own
|
||||
local code-file edits, and infrastructure mutations must be reviewed
|
||||
@@ -4933,6 +4947,10 @@ transition and evidence times) from the frontend-primitives shared
|
||||
relative-time clock. That is presentation over the same `GET
|
||||
/api/ai/patrol/status`, run history and `GET /api/ai/patrol/attention/{id}`
|
||||
payloads: no field, request or polling cadence changes.
|
||||
The attention detail's suppression `expiresAt` and a finding's `remind_at` and
|
||||
`snoozed_until` from `GET /api/ai/patrol/findings` are future times, so the
|
||||
page presents them as countdowns from the same clock ("in 1d") instead of
|
||||
ages. That is presentation only: no field, request or polling cadence changes.
|
||||
|
||||
### Resource-list facets preserve scoped navigation evidence
|
||||
|
||||
|
||||
@@ -2206,6 +2206,18 @@ the same record; the table renders defaults once, in both desktop table and
|
||||
narrow card layouts, and per-section metric columns must resolve to metric
|
||||
keys the shared column normalizer produces so the defaults editor reads and
|
||||
writes the same record keys the section persists.
|
||||
A global default that another setting decides while it is unset passes a
|
||||
`globalDefaultFallbacks` entry for that metric (label, title, whether the other
|
||||
setting is itself off, and the value switching on stages when it is). The
|
||||
TrueNAS Disks temperature uses it to read `By type`, following Disk
|
||||
temperature by type, with `features/alerts/thresholds/trueNASDiskTemperature.ts`
|
||||
giving each disk row its type's trigger from the unsaved editor state.
|
||||
`resolveAlertResourceGlobalDefaultCell` decides the cell for both layouts:
|
||||
unset follows the fallback, switching on from Off returns to unset unless the
|
||||
fallback is off, and an unset default with no fallback stays Off as the engine
|
||||
reads it. `ResourceTable.tsx` hands its props to the shared table state through
|
||||
getters, so the Custom badge and reset control follow defaults that load after
|
||||
the table mounts.
|
||||
Platform sub-routes that add native provider inventory must stay on the shared
|
||||
platform page and table primitives. The vSphere Networks surface routes through
|
||||
`/vmware/networks`, the shared platform tab model, the command palette
|
||||
@@ -5345,7 +5357,19 @@ too. Every read of the clock returns the wall clock; the 30-second tick only
|
||||
tells readers to re-read, so a cell that mounts between ticks never measures
|
||||
from a stale time and a timestamp from the last few seconds never reads as a
|
||||
future time.
|
||||
The rule has one deliberate exception. The age of a latest reading (last used,
|
||||
Future times have their own formatter. `formatRelativeTime` is past-only: it
|
||||
reads a time ahead of `now` as "just now" (compact) or "0s ago", which is right
|
||||
for clock skew on something already observed and wrong for an expiry, a
|
||||
reminder or a schedule.
|
||||
Those go through `formatTimeUntil` (`frontend-modern/src/utils/format.ts`),
|
||||
which counts down ("in 3h", "in 1d"), rounds to the nearest unit so a duration
|
||||
just chosen reads as chosen, and returns `dueText` (default "now") once the
|
||||
time arrives. A countdown target does not change while its surface stays open,
|
||||
so the countdown passes `now` from the shared clock, as the Patrol suppression
|
||||
expiry and the Patrol findings reminder and snooze lines do. The replication
|
||||
Next sync column and the Patrol header's next-check `CountdownTimer` keep their
|
||||
own minute- and second-precision countdowns.
|
||||
The relative-age rule has one deliberate exception. The age of a latest reading (last used,
|
||||
last seen, last success, last checked) on data the surface reads once and does
|
||||
not re-read stays the age at read time, because a moving age over a snapshot
|
||||
that never refreshes claims the reading stopped when it may not have. Such a
|
||||
@@ -5860,7 +5884,15 @@ the surface-alt detail row shell locally. The content shell must clip
|
||||
horizontal paint below the large breakpoint without becoming a scroll
|
||||
container, reset the parent table's `whitespace-nowrap` inheritance, and allow
|
||||
its descendants to shrink, then restore visible overflow for the static
|
||||
desktop layout. Long operator-state copy must wrap inside the shared row border
|
||||
desktop layout. Below that breakpoint the default content shell is capped at
|
||||
the table scroll shell's content-box inline size (`max-w-[100cqi]`; the shell
|
||||
is a size container), never a viewport estimate of the page chrome, so a phone
|
||||
drawer does not leave an empty strip beside its content (the retired
|
||||
`100vw-3.5rem` cap left 27px at 390px) and is never wider than the table's
|
||||
visible area. Callers that pass their own `contentClass` own their width.
|
||||
Fixed-layout cells clip overflow, so the sticky offset is inert and horizontal
|
||||
scrolling still moves the drawer with its cell.
|
||||
Long operator-state copy must wrap inside the shared row border
|
||||
instead of painting beneath adjacent controls or disappearing at the clip edge.
|
||||
When focused detail content is removed, `InlineDetailTableRow` restores focus
|
||||
to its current `aria-controls` disclosure with `preventScroll`; live refresh,
|
||||
|
||||
@@ -2124,6 +2124,29 @@ boundary around it:
|
||||
matches node names only, so it relies on fixture node names being unique
|
||||
across instances, which they are.
|
||||
|
||||
`checkPhysicalDiskAlerts` also hands each disk's temperature to
|
||||
`CheckProxmoxDiskTemperature`, so a Proxmox disk that only the node's sensors
|
||||
report alerts under the disk temperature policy, as its Running Hot verdict
|
||||
already says it should. It passes the reading only when this poll collected it
|
||||
(`collectedPhysicalDiskTemperature`, `diskinventory.TemperatureCollected`), so
|
||||
a retained last-known value holds the alert rather than judging it, and it
|
||||
passes `AgentSMARTReported`, which `mergeHostAgentSMARTIntoDisks` sets on a
|
||||
disk a still-reporting linked agent lists in its SMART report (disks built
|
||||
from that report when the Proxmox query fails go through the same merge). A
|
||||
disk the agent lists is the agent's: `CheckHost` raises its temperature
|
||||
alert, and the PVE disk check closes its own as moved to the agent, so no
|
||||
disk alerts twice. Once the agent's lease lapses (`ExpireHostTelemetry` marks
|
||||
the host offline) its retained rows still enrich the disk, but ownership
|
||||
returns to the PVE check, which judges the node's own current readings; an
|
||||
agent newly linked to a disk with an open PVE alert takes it over on the next
|
||||
disk poll. An excluded device closes its temperature
|
||||
alert with the health and wearout ones. Fixture disks carry no agent SMART
|
||||
merge, so on a mock estate every hot disk on an online node alerts.
|
||||
`TestMergeHostAgentSMARTIntoDisksMarksDisksTheAgentReports` and
|
||||
`TestCheckPhysicalDiskAlertsRaisesProxmoxDiskTemperatureAlerts` in
|
||||
`internal/monitoring/physical_disk_roundtrip_test.go` pin the marker,
|
||||
the collected reading, agent ownership and exclusion.
|
||||
|
||||
Switching mock mode fences the alert evaluations that read mode-dependent data
|
||||
(`mockModeFence`, `internal/monitoring/mock_mode_fence.go`). `GetState`, the
|
||||
fixture graph, the unified read view, the recovery rollups and the connection
|
||||
|
||||
@@ -2715,6 +2715,36 @@ clock while it stays open. Its `Last seen` and flapping window are latest
|
||||
readings of a record that may have moved since, so they stay the ages at
|
||||
read time, and the queue rows, summary, recent work, run history and digest
|
||||
re-derive on their own 30- or 60-second re-reads.
|
||||
Future times in Patrol count down instead of reading as ages, because the
|
||||
past-only `formatRelativeTime` reads each of them as "just now" for their whole
|
||||
duration. The attention detail's suppression line counts down to the expiry
|
||||
with the exact time on hover and names the state expiry restores in the
|
||||
early-end control's words, from the same
|
||||
`getAttentionSuppressionEndPresentation` split on
|
||||
`operationalRecord.acknowledgement`: an open issue reads `Returns to active
|
||||
attention in 1d.`, one marked reviewed reads `Suppression ends in 1d. It stays
|
||||
reviewed.` (expiry restores Acknowledged, which is not active attention), and a
|
||||
passed expiry reads `Suppression has ended.`
|
||||
(`getAttentionSuppressionExpiryLabel`). The findings list reads `Reminding in
|
||||
6d` for a will_fix_later reminder and `snoozed, returns in 3h` for a snooze,
|
||||
both through `formatTimeUntil` on the shared clock with the exact time on
|
||||
hover. A reminder reads `Reminder overdue` once the panel's overdue id list
|
||||
holds it. That list is sampled on the shared clock and is what the Overdue
|
||||
commitments chip and filter count, so the label, chip and filter agree: a row
|
||||
mounted or re-read between the deadline and the tick that adds it reads
|
||||
`Reminder due`, and a row already on screen keeps its last countdown until that
|
||||
tick. The list only notifies when its ids change, so a clock tick that moves no
|
||||
deadline past now does not rebuild the open Overdue rows. The hourly reminder
|
||||
sweep then brings the finding back whether or not it still trips. An ended
|
||||
snooze reads `snooze ended` until the next findings re-read drops the snoozed
|
||||
status: `GET /api/ai/patrol/findings` sends no status, so the frontend
|
||||
`aiIntelligence` store (`normalizeFindingStatus`) derives `snoozed` on each
|
||||
read only while `snoozed_until` is ahead, and the finding then shows as Active
|
||||
unless it has since resolved or been dismissed.
|
||||
`getFindingReminderPresentation` and `getFindingSnoozePresentation` in
|
||||
`aiFindingPresentation.ts` own the findings copy.
|
||||
`PatrolAttentionWorkbench.test.tsx`, `FindingsPanel.links.test.tsx` and
|
||||
`FindingsPanel.test.ts` pin it.
|
||||
|
||||
### This week card answers what Patrol did for the customer
|
||||
|
||||
|
||||
@@ -1429,6 +1429,12 @@ state only. A restored prompt or restored message count may help an operator
|
||||
continue a protected-item investigation, but it must not become backup coverage
|
||||
evidence, recovery freshness, restore entitlement, storage-owner identity,
|
||||
approval policy, or a provider-local recovery command handoff.
|
||||
The POST-only guard on Assistant session mutations (`/abort`,
|
||||
`/summarize`, `/fork`, `/undo`, `/redo`, `/steer`; `405` with `Allow: POST`
|
||||
for any other method that reaches the handler) is transport hardening in
|
||||
`internal/api/ai_handler.go`.
|
||||
It opens no storage, backup, or recovery read or write path, and a rejected
|
||||
request never reaches session state.
|
||||
Approved Assistant tool execution through `internal/api/router_routes_ai_relay.go`
|
||||
is also adjacent API/AI action plumbing for storage/recovery consumers.
|
||||
`AssistantToolExecutor` / `ApprovedAssistantToolExecutor` may execute an already
|
||||
|
||||
@@ -6112,7 +6112,12 @@ the reference and is never retried. An owned row records its alert's
|
||||
reference as `alert_resource_id`. A read by the reference returns the rows
|
||||
journaled under it plus the rows owned away from it, matched through the
|
||||
indexed alert identifier (`ProxmoxPhysicalDiskAlertIdentifiers`) and that
|
||||
recorded reference, and every count uses the same predicate. The Alerts
|
||||
recorded reference, and every count uses the same predicate. Those
|
||||
identifiers cover the health and wearout specs and the disk temperature
|
||||
metric spec (`<reference>::metric-threshold:diskTemperature`), since PVE disks
|
||||
also raise temperature alerts under the same reference
|
||||
(`TestProxmoxDiskTemperatureAlertRowsFollowRecordedHardwareIdentity` in
|
||||
`internal/unifiedresources/store_test.go`). The Alerts
|
||||
history Resource action therefore still lists every occurrence raised under
|
||||
the path, across the disks that held it, and alert-centric reads keep their
|
||||
occurrences (see the AI runtime contract's incident-history queries). Rows
|
||||
@@ -6268,6 +6273,13 @@ presentation. `frontend-modern/src/components/Infrastructure/`
|
||||
placement, signal, and snapshot context through the canonical resource drawer
|
||||
and debug/source sections rather than introducing a VMware-only detail route,
|
||||
drawer tab, or provider-local investigation shell.
|
||||
The TrueNAS physical-disk drawer judges a current temperature reading by the
|
||||
disk temperature policy, like the TrueNAS storage table:
|
||||
`useResourceDetailDrawerDerivedState.ts` passes the alerts store's
|
||||
`getDiskTemperatureThresholds` into `buildTrueNASDetailSections`, and the
|
||||
Temperature row takes a warning tone only when `isPhysicalDiskRunningHot`
|
||||
says the reading reached its disk type's alert trigger. A retained reading
|
||||
keeps its muted last-known row, and without a resolver heat is not judged.
|
||||
That same infrastructure consumer boundary also owns source selection
|
||||
continuity. Settings infrastructure panels and platform/runtime pages must
|
||||
keep canonical sources such as `truenas` and `availability` present in their
|
||||
|
||||
@@ -0,0 +1,69 @@
|
||||
{
|
||||
"version": 1,
|
||||
"base_sha": "bdf0f76b7bf61d550b1cb8574497eef75b354acb",
|
||||
"verified_at": "2026-10-07T14:11:27Z",
|
||||
"result": "passed",
|
||||
"changed_paths": [
|
||||
"frontend-modern/src/components/Alerts/AlertResourceTableDesktop.tsx",
|
||||
"frontend-modern/src/components/Alerts/AlertResourceTableMobile.tsx",
|
||||
"frontend-modern/src/components/Alerts/ResourceTable.tsx",
|
||||
"frontend-modern/src/components/Alerts/ThresholdsTableAgentsResourcesSection.tsx",
|
||||
"frontend-modern/src/components/Alerts/ThresholdsTableTrueNASTab.tsx",
|
||||
"frontend-modern/src/components/Alerts/alertResourceTableModel.ts",
|
||||
"frontend-modern/src/components/Infrastructure/resourceDetailDrawerTrueNASModel.ts",
|
||||
"frontend-modern/src/components/Infrastructure/useResourceDetailDrawerDerivedState.ts",
|
||||
"frontend-modern/src/features/alerts/alertsConfigurationModel.ts",
|
||||
"frontend-modern/src/features/alerts/thresholds/hooks/useThresholdsPlatformData.ts",
|
||||
"frontend-modern/src/features/alerts/thresholds/trueNASDiskTemperature.ts",
|
||||
"frontend-modern/src/types/alerts.ts",
|
||||
"frontend-modern/src/utils/alertThresholdDefaults.ts"
|
||||
],
|
||||
"content_sha256": {
|
||||
"frontend-modern/src/components/Alerts/AlertResourceTableDesktop.tsx": "4d09dd97adc2868889e256455c207b6b9089ea311713878f0d6ff00efb6b7123",
|
||||
"frontend-modern/src/components/Alerts/AlertResourceTableMobile.tsx": "ceaf136da9361a5b1756350626a833da4dfebcb4f211c7269d7d89d37fb7e085",
|
||||
"frontend-modern/src/components/Alerts/ResourceTable.tsx": "7e80a6c789b3069fcadcb86e0a6689e5319e7b0ac5d5bfa1a596520daec2d7e1",
|
||||
"frontend-modern/src/components/Alerts/ThresholdsTableAgentsResourcesSection.tsx": "fc4e9f4d05c0bf5799acbbe64c37b901e104869ff120819342432563789ad0e9",
|
||||
"frontend-modern/src/components/Alerts/ThresholdsTableTrueNASTab.tsx": "a22b86c28b65ec18ba81171fb1206a24107a0d97db19d7dd488bdfd0a689a956",
|
||||
"frontend-modern/src/components/Alerts/alertResourceTableModel.ts": "5881aeaf7ee575589997bf67fb713a01784ce7ca16ecb5a5eaea77f36010837a",
|
||||
"frontend-modern/src/components/Infrastructure/resourceDetailDrawerTrueNASModel.ts": "b60ba59688d6ea0cabfb2b413b8adcaa85c8d41524c1d1f0df9bdb9e6d85733c",
|
||||
"frontend-modern/src/components/Infrastructure/useResourceDetailDrawerDerivedState.ts": "63b07de3f354f8328298348a8892ad73fde2f11ec6768e9c0f6fc37eb4c43516",
|
||||
"frontend-modern/src/features/alerts/alertsConfigurationModel.ts": "0e116e65c992a72e21d700835b0cb0db7173be9ec9c4980382958506d4e75976",
|
||||
"frontend-modern/src/features/alerts/thresholds/hooks/useThresholdsPlatformData.ts": "8525b0b012198ff87e29063908a9907b7fa1080cd84187202b1cd3cf198a3b46",
|
||||
"frontend-modern/src/features/alerts/thresholds/trueNASDiskTemperature.ts": "80433d42dd4658a1449c0f0c378d7fa52cd6dfab117d626195099768e750352f",
|
||||
"frontend-modern/src/types/alerts.ts": "d072175ab98e3785eae643d655be4c2d54cac0eddc8b55803a2841a312062e5a",
|
||||
"frontend-modern/src/utils/alertThresholdDefaults.ts": "077e291d5c224063a373a5e9f5555d9b470f585801b0cfdd9dc1acacf233af28"
|
||||
},
|
||||
"routes": [
|
||||
"/alerts/thresholds/truenas",
|
||||
"/truenas/storage"
|
||||
],
|
||||
"viewports": [
|
||||
{
|
||||
"width": 1366,
|
||||
"height": 900
|
||||
},
|
||||
{
|
||||
"width": 375,
|
||||
"height": 812
|
||||
}
|
||||
],
|
||||
"states": [
|
||||
"Private mock stack (8 nodes, mock TrueNAS truenas-main) with a local-only generator overlay pinning TrueNAS disk temperatures: nvme0n1 60C, nvme1n1 72C, sda 57C (SATA), sdc 63C (SAS); after = this branch, before = main logic in the same tree",
|
||||
"Thresholds > TrueNAS > Disks with no saved TrueNAS-wide value: header reads 'Defaults: NVMe 70\u00b0C \u00b7 SAS 65\u00b0C \u00b7 SATA 55\u00b0C', note 'Unless set here, each disk alerts at Disk temperature by type, set under Agents.', Global Defaults Temp cell empty with placeholder 'By type' (fits after widening to w-20) and On, disk rows nvme0n1/nvme1n1 70\u00b0C, sda/sdb 55\u00b0C, sdc-sdf 65\u00b0C",
|
||||
"Same section at 375px: Global Defaults card shows TEMP \u00b0C 'By type' with On, disk cards show their type triggers, no horizontal overflow",
|
||||
"Agent Disk Temp default switched off through the API: header 'Defaults: all metric alerts off', Global Defaults cell Off with title 'Off because the agent Disk Temp default under Agents is off...', disk rows Off",
|
||||
"Saved TrueNAS-wide 62: header 'Defaults: Temp 62\u00b0C', every disk row 62\u00b0C, Custom badge and reset control after reload; saved 55 survives save and reload as 55 (stored with truenasDiskTemperatureByType)",
|
||||
"Alerts with factory per-type thresholds: after stack raises nvme1n1 72 (trigger 70) and sda 57 (trigger 55) with diskType metadata, none for nvme0n1 60 or sdc 63; before (main) raised all four at a flat 55",
|
||||
"TrueNAS storage table agrees: Attention 'Disk temperature is 72\u00b0C, at or above its 70\u00b0C alert threshold.' on nvme1n1 and '57\u00b0C ... 55\u00b0C' on sda, none on nvme0n1",
|
||||
"TrueNAS disk drawer (Platform details > Show TrueNAS > Health): nvme0n1 Temperature 60\u00b0C in default text colour (main toned it amber from the fixed 55C); sda 57\u00b0C amber rgb(252,211,77) at 1366 and at 375 with no overflow"
|
||||
],
|
||||
"interactions": [
|
||||
"Typed 62 into the TrueNAS Disks Global Defaults Temp cell, saved, reloaded: API truenasDiskDefaults {trigger 62, clear 57}, cell shows 62 with Custom",
|
||||
"Triple-clicked and typed 55, saved, reloaded: API keeps {55, 50}, cell shows 55 with Custom and the reset control",
|
||||
"Clicked Reset to factory defaults: cell returns to 'By type'; saved: API truenasDiskDefaults {}",
|
||||
"On the 375px card, clicked On to switch the default Off, saved: API stores {trigger 0, clear 0} and the after stack's TrueNAS disk alerts resolve; clicked again: returns to 'By type', saved: {}",
|
||||
"With the agent Disk Temp default off, clicked the Off badge: stages an explicit 55 (Custom, rows 55\u00b0C); discarded",
|
||||
"Loaded an old saved config (flat 55/50, no marker) with the new binary: GET returns truenasDiskDefaults {} and truenasDiskTemperatureByType true",
|
||||
"Expanded nvme0n1 and sda rows on /truenas/storage, opened Platform details and Show TrueNAS, read the Temperature row value and computed colour at 1366 and 375"
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,52 @@
|
||||
{
|
||||
"version": 1,
|
||||
"base_sha": "7eb32f318100fef045d63414fd63304c62acfdef",
|
||||
"verified_at": "2026-10-07T16:55:00Z",
|
||||
"result": "passed",
|
||||
"changed_paths": [
|
||||
"frontend-modern/src/components/shared/InlineDetailTableRow.tsx"
|
||||
],
|
||||
"content_sha256": {
|
||||
"frontend-modern/src/components/shared/InlineDetailTableRow.tsx": "e5b27178cb3268132595943058849b3f619da22fcbd7916dfc301d4e852ef244"
|
||||
},
|
||||
"routes": [
|
||||
"/standalone/availability",
|
||||
"/standalone/machines",
|
||||
"/proxmox",
|
||||
"/docker",
|
||||
"/kubernetes"
|
||||
],
|
||||
"viewports": [
|
||||
{
|
||||
"width": 1440,
|
||||
"height": 900
|
||||
},
|
||||
{
|
||||
"width": 1000,
|
||||
"height": 800
|
||||
},
|
||||
{
|
||||
"width": 768,
|
||||
"height": 1024
|
||||
},
|
||||
{
|
||||
"width": 390,
|
||||
"height": 844
|
||||
},
|
||||
{
|
||||
"width": 320,
|
||||
"height": 700
|
||||
}
|
||||
],
|
||||
"states": [
|
||||
"Private stacks served by Vite from this tree (Playwright 1.56.1 headless Chromium): a non-mock stack with one HTTP availability check, then a mock estate (3 Proxmox nodes, 2 Docker hosts, 1 Kubernetes cluster, 2 generic hosts).",
|
||||
"Before (origin/main class max-w-[calc(100vw-3.5rem)]): at 390px every opened drawer (availability check, Proxmox node, Proxmox guest, Docker host, Docker container, Kubernetes node, Machines agent) was 334px inside a 361px visible table, leaving a 27px empty strip; at 768px a table forced wider than its shell got a 712px drawer in a 695px visible area.",
|
||||
"After (max-w-[100cqi]): drawer content equals the scroll shell's visible width at 320 (291px), 390 (361px), 768 (695px) and 1000 (916px) with no right gap, including the forced-wider table; 1440px keeps the static lg layout (max-width none, 1320px). The availability drawer's hostname now shows in full instead of truncating.",
|
||||
"Long unwrapped identity chips (e.g. proxmox-lxc-docker:Production West:pve1:107) were already clipped at the content edge before (Kubernetes 20px, Docker 8px and 83px) and are clipped less after (7px, 70px); chip wrapping is a separate follow-up.",
|
||||
"Sticky positioning is inert in both versions because .table-fixed td sets overflow hidden; not changed here. This is not an installed-build, physical-phone or release acceptance."
|
||||
],
|
||||
"interactions": [
|
||||
"Opened each drawer by its disclosure button (row tap at phone width where the button is hidden), measured the scroll shell, detail cell, content shell and every descendant's right edge, and captured screenshots at 390px before and after.",
|
||||
"Forced a 900px table inside its shell at 390px and 768px and scrolled it 300px sideways to compare drawer width against the visible area for both classes."
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,51 @@
|
||||
{
|
||||
"version": 1,
|
||||
"base_sha": "7eb32f318100fef045d63414fd63304c62acfdef",
|
||||
"verified_at": "2026-10-07T16:47:38Z",
|
||||
"result": "passed",
|
||||
"changed_paths": [
|
||||
"frontend-modern/src/components/AI/FindingsPanel.tsx",
|
||||
"frontend-modern/src/features/patrol/PatrolAttentionWorkbench.tsx",
|
||||
"frontend-modern/src/utils/aiFindingPresentation.ts",
|
||||
"frontend-modern/src/utils/format.ts"
|
||||
],
|
||||
"content_sha256": {
|
||||
"frontend-modern/src/components/AI/FindingsPanel.tsx": "6b50ebbf3bbe24e31eebdb3c8be8ac6ca9c08405352facbfcd294a1361d8a2f5",
|
||||
"frontend-modern/src/features/patrol/PatrolAttentionWorkbench.tsx": "9055b69986cd4d320fd2194dba2819c0643e63326724952fd1021325530e9f5b",
|
||||
"frontend-modern/src/utils/aiFindingPresentation.ts": "29e212a8d0aed4c1eb17697bbed86519ed82f64bc2940239db4a79470a23a090",
|
||||
"frontend-modern/src/utils/format.ts": "abaec8a1f40c159e9a58702b726ccaea196bf6b72825c97addbee00af3bd7809"
|
||||
},
|
||||
"routes": [
|
||||
"/patrol",
|
||||
"/patrol/activity"
|
||||
],
|
||||
"viewports": [
|
||||
{
|
||||
"width": 1440,
|
||||
"height": 900
|
||||
},
|
||||
{
|
||||
"width": 390,
|
||||
"height": 844
|
||||
}
|
||||
],
|
||||
"states": [
|
||||
"Private mock stack (backend :7793 built from this tree on origin/main 7eb32f3181, Vite :5303 serving it) with Playwright page.clock installed before load and the data held: the /ws proxy dropped every server frame and answered the client heartbeat with a data-free pong, every GET /api/ read was replayed from its cached body (Patrol findings reads left pending), and the clock was moved with setSystemTime plus a 31-second runFor so only the shared 30-second tick fired; one websocket connection and no attention detail re-read for the whole hold",
|
||||
"Unreviewed suppressed issue (ops-prometheus Docker service health, suppressed through the real form for the default 24 hours; the form's duration read 'Return it to active attention after'): Lifecycle reads 'Returns to active attention in 1d.' with hover 'Suppressed until 8 Oct 17:36' and button 'Return to active attention', then 'in 20h.' at +4h, 'in 29m.' at +23h20m (28m at 390) and 'Suppression has ended.' at +25h, at 1440 and 390",
|
||||
"Reviewed then suppressed issue (esxi-07.lab.local VMware host health, Mark reviewed then suppressed for 24 hours from its detail; the form's duration read 'End suppression after'): Lifecycle reads 'Suppression ends in 1d. It stays reviewed.' with button 'End suppression', then 'in 20h' at +4h, 'in 29m' at +23h20m (28m at 390) and 'Suppression has ended.' at +25h, at 1440 and 390",
|
||||
"Patrol activity, Finding options and history, Resolved: mail-gateway-us dismissed through Finding options, Dismiss: Later reads '(will fix later) \u00b7 Reminding in 7d' with hover 'Reminder due 14 Oct 17:36. Pulse brings this finding back on its first reminder check after that.'; at +7d1m it reads 'Reminder overdue' and the 'Overdue commitments (1)' chip appears on that tick with findings reads still pending, and the chip filters to that finding, at 1440 and 390",
|
||||
"Between ticks: page clock set 5 seconds past that reminder's deadline with no timer fired, then switching from Active to Resolved mounts the row reading 'Reminder due' with no Overdue chip; one 31-second runFor later the row reads 'Reminder overdue' and the 'Overdue commitments (1)' chip appears together, at 1440 and 390",
|
||||
"Same list, dr-vault snoozed for 1 hour through POST /api/ai/patrol/snooze (Patrol rows hide the snooze buttons): reads 'snoozed, returns in 49m' with hover 'Snoozed until 7 Oct 18:36', 'in 9m' at +40m (8m at 390) and 'snooze ended' at +70m; after releasing the hold the next findings read lists dr-vault under Active, at 1440 and 390",
|
||||
"No 'just now' on any future time at either viewport (the only 'just now' left is the attention detail's read-time Last seen age)",
|
||||
"Same-session controls under the running Vite: with FindingsPanel.tsx and aiFindingPresentation.ts swapped to the first draft of this change, the between-ticks row read 'Reminder overdue' with no Overdue chip; with FindingsPanel.tsx, PatrolAttentionWorkbench.tsx, aiFindingPresentation.ts and format.ts swapped back to origin/main, the reviewed issue read 'Returns to active attention just now.' before and after +4h, the reminder 'Reminding just now' before and after +7d with no Overdue chip, and the snooze 'snoozed until just now'"
|
||||
],
|
||||
"interactions": [
|
||||
"Opened a decision by its Patrol attention link, filled the suppression reason and submitted Suppress temporarily with the default 24 hours",
|
||||
"Opened a second decision by its attention link, clicked Mark reviewed, reopened it and suppressed it for 24 hours",
|
||||
"Opened Finding options and history, opened Finding options on a finding, chose Dismiss: Later and Confirm Dismiss",
|
||||
"Opened Reviewed and suppressed, opened each suppressed issue from the list, read the Lifecycle line, its hover title and the early-end button",
|
||||
"Switched the findings filter between Active and Resolved, read reminder and snooze hover titles, stepped the page clock and clicked the Overdue commitments chip",
|
||||
"Released the hold and switched to Active to see the ended snooze re-derived",
|
||||
"Repeated every pass at 1440x900 and 390x844, then reran with the first draft and with origin/main's components as controls"
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,39 @@
|
||||
{
|
||||
"version": 1,
|
||||
"base_sha": "8e1a564d7e229bda51c021c7e6bde920e0ca0596",
|
||||
"verified_at": "2026-10-07T14:35:42Z",
|
||||
"result": "passed",
|
||||
"changed_paths": [
|
||||
"frontend-modern/src/components/Alerts/ThresholdsTableAgentsResourcesSection.tsx"
|
||||
],
|
||||
"content_sha256": {
|
||||
"frontend-modern/src/components/Alerts/ThresholdsTableAgentsResourcesSection.tsx": "3052a190a76e885e98ed6601b3e83c8959e1c656d12270b87e074fffa95860ac"
|
||||
},
|
||||
"routes": [
|
||||
"/alerts/thresholds/systems",
|
||||
"/alerts/overview",
|
||||
"/proxmox/storage?tab=disks"
|
||||
],
|
||||
"viewports": [
|
||||
{
|
||||
"width": 1366,
|
||||
"height": 900
|
||||
},
|
||||
{
|
||||
"width": 375,
|
||||
"height": 812
|
||||
}
|
||||
],
|
||||
"states": [
|
||||
"Private mock stack (8 PVE nodes) on this branch with a local-only generator overlay pinning PVE disks: pve1 NVMe 63C, pve2 SATA 56C, pve3 NVMe 72C, pve3 SAS 63C, pve3 SATA 66C",
|
||||
"Alerts > Thresholds > Systems (Agents section): 'Disk temperature by type' card reads 'Alert trigger in \u00b0C for each disk type, for agent, TrueNAS and Proxmox disks. ... a TrueNAS Disks value replaces them for TrueNAS disks.' at 1366, and at 375 the card spans 34-341px with no horizontal overflow",
|
||||
"Alerts overview: 'Samsung 870 EVO 1TB (/dev/nvme2) (Temperature) CRITICAL on West Production C \u2014 Temperature 66\u00b0C, above the 55\u00b0C alert level; Clears once it drops to 50\u00b0C or lower and stays there for 5 minutes', Alert level 55\u00b0C, Clear level 50\u00b0C",
|
||||
"Warning-level PVE disk temperature alerts (SATA 56C trigger 55, NVMe 72C trigger 70) appear after the 300s stability window (first breach 15:26:32, firing by 15:31:46); the critical 66C fires at once; none for NVMe 63C or SAS 63C",
|
||||
"Physical Disks filtered to West Production: exactly the three alerting disks read Running Hot (Crucial MX500 56\u00b0C, WD Black SN850X 72\u00b0C, Samsung 870 EVO 66\u00b0C); the 870 EVO drawer reads 'Running Hot \u2014 Disk temperature is 66\u00b0C, at or above its 55\u00b0C alert threshold.'"
|
||||
],
|
||||
"interactions": [
|
||||
"Opened Alerts overview and read the PVE disk temperature alert card",
|
||||
"Filtered Physical Disks by 'West Production C', expanded the Samsung 870 EVO row",
|
||||
"Opened Thresholds > Systems, read the Disk temperature by type card at 1366 and at 375"
|
||||
]
|
||||
}
|
||||
@@ -39,6 +39,7 @@ import { AIAPI, type RemediationPlan } from '@/api/ai';
|
||||
import { createSuppressionRuleFromFinding, reinvestigateFinding } from '@/api/patrol';
|
||||
import type { PatrolRunRecord, PatrolRuntimeState, PatrolAutonomyLevel } from '@/api/patrol';
|
||||
import { formatRelativeTime } from '@/utils/format';
|
||||
import { useRelativeTimeNow } from '@/utils/relativeTimeClock';
|
||||
import {
|
||||
getFindingAlertIdentifier,
|
||||
hasTriggeringAlert,
|
||||
@@ -84,6 +85,8 @@ import {
|
||||
getFindingSubjectPresentation,
|
||||
getFindingTitlePresentation,
|
||||
getFindingRecencyPresentation,
|
||||
getFindingReminderPresentation,
|
||||
getFindingSnoozePresentation,
|
||||
getFindingEvidencePresentation,
|
||||
hasFindingInvestigationDetails,
|
||||
hasFindingInvestigationHandoffPointer,
|
||||
@@ -194,6 +197,10 @@ function getFindingSuppressionRuleScope(finding: UnifiedFinding) {
|
||||
export const FindingsPanel: Component<FindingsPanelProps> = (props) => {
|
||||
const location = useLocation();
|
||||
const [filter, setFilter] = createSignal<FindingsPanelFilter>(props.filterOverride ?? 'active');
|
||||
// A reminder or snooze end never changes while its row stays on screen, so
|
||||
// the countdown to it, and whether a reminder is overdue, read the shared
|
||||
// clock.
|
||||
const now = useRelativeTimeNow();
|
||||
const [sortBy, setSortBy] = createSignal<'severity' | 'time'>('severity');
|
||||
const [expandedId, setExpandedId] = createSignal<string | null>(null);
|
||||
let panelRoot: HTMLDivElement | undefined;
|
||||
@@ -338,6 +345,32 @@ export const FindingsPanel: Component<FindingsPanelProps> = (props) => {
|
||||
}
|
||||
});
|
||||
|
||||
// Will_fix_later commitments whose RemindAt deadline has passed. The Go
|
||||
// store's proactive sweep (SweepWillFixLaterReminders) wakes these on a 1h
|
||||
// timer; until then the Overdue commitments chip and filter surface them on
|
||||
// demand. The list is sampled on the shared clock and notifies only when its
|
||||
// ids change, so a tick that moves no deadline past now does not rebuild the
|
||||
// filtered rows. Each row's "Reminder overdue" label reads the same list, so
|
||||
// a row that mounts between ticks agrees with the chip and filter.
|
||||
const overdueFindingIds = createMemo(
|
||||
() => {
|
||||
const nowMs = now();
|
||||
return sourceFindings()
|
||||
.filter((f) => {
|
||||
if (f.dismissedReason !== 'will_fix_later' || !f.remindAt) return false;
|
||||
const due = Date.parse(f.remindAt);
|
||||
return Number.isFinite(due) && due <= nowMs;
|
||||
})
|
||||
.map((f) => f.id);
|
||||
},
|
||||
[],
|
||||
{
|
||||
equals: (prev, next) =>
|
||||
prev.length === next.length && prev.every((id, index) => id === next[index]),
|
||||
},
|
||||
);
|
||||
const overdueFindingIdSet = createMemo(() => new Set(overdueFindingIds()));
|
||||
|
||||
// Filter and sort findings
|
||||
const filteredFindings = createMemo(() => {
|
||||
if (hasUnknownRunSnapshot()) {
|
||||
@@ -365,22 +398,8 @@ export const FindingsPanel: Component<FindingsPanelProps> = (props) => {
|
||||
const approvalFindingIds = new Set(sourceFindingsWithPendingApprovals().map((f) => f.id));
|
||||
findings = findings.filter((f) => approvalFindingIds.has(f.id));
|
||||
} else if (filter() === 'overdue') {
|
||||
// Overdue surfaces will_fix_later commitments whose RemindAt
|
||||
// deadline has already passed. The Go store's proactive sweep
|
||||
// (SweepWillFixLaterReminders) wakes these on a 1h timer, but
|
||||
// this filter lets the operator see them on demand — including
|
||||
// any that the sweep has not yet promoted on this cycle.
|
||||
const nowMs = Date.now();
|
||||
findings = findings.filter((f) => {
|
||||
if (f.dismissedReason !== 'will_fix_later') {
|
||||
return false;
|
||||
}
|
||||
if (!f.remindAt) {
|
||||
return false;
|
||||
}
|
||||
const due = Date.parse(f.remindAt);
|
||||
return Number.isFinite(due) && due <= nowMs;
|
||||
});
|
||||
const overdueIds = overdueFindingIdSet();
|
||||
findings = findings.filter((f) => overdueIds.has(f.id));
|
||||
}
|
||||
|
||||
// Filter by specific finding IDs if provided
|
||||
@@ -468,17 +487,8 @@ export const FindingsPanel: Component<FindingsPanelProps> = (props) => {
|
||||
? scopedPendingApprovalCount()
|
||||
: sourceFindingsWithPendingApprovals().length,
|
||||
}));
|
||||
// Count of will_fix_later commitments whose RemindAt deadline has
|
||||
// already passed. Drives the Overdue commitments chip below.
|
||||
const overdueCount = createMemo(() => {
|
||||
const nowMs = Date.now();
|
||||
return sourceFindings().filter((f) => {
|
||||
if (f.dismissedReason !== 'will_fix_later') return false;
|
||||
if (!f.remindAt) return false;
|
||||
const due = Date.parse(f.remindAt);
|
||||
return Number.isFinite(due) && due <= nowMs;
|
||||
}).length;
|
||||
});
|
||||
// Drives the Overdue commitments chip below.
|
||||
const overdueCount = createMemo(() => overdueFindingIds().length);
|
||||
// Active findings that restate an active alert (backend-stamped mirror).
|
||||
// They are demoted into a collapsed group below the list rather than shown
|
||||
// as separate items: the alert already owns that condition on the Patrol
|
||||
@@ -1279,18 +1289,27 @@ export const FindingsPanel: Component<FindingsPanelProps> = (props) => {
|
||||
{formatOperatorStateDismissCauseLabel(getOperatorStateDismissCause(finding))}
|
||||
</span>
|
||||
</Show>
|
||||
<Show when={finding.dismissedReason === 'will_fix_later' && finding.remindAt}>
|
||||
<span
|
||||
class="ml-2 text-amber-600 dark:text-amber-400"
|
||||
title="Pulse will surface this finding again on this date if it is still tripping."
|
||||
>
|
||||
{' · '}Reminding {formatTime(finding.remindAt!)}
|
||||
</span>
|
||||
<Show
|
||||
when={getFindingReminderPresentation(
|
||||
finding,
|
||||
now(),
|
||||
overdueFindingIdSet().has(finding.id),
|
||||
)}
|
||||
>
|
||||
{(reminder) => (
|
||||
<span class="ml-2 text-amber-600 dark:text-amber-400" title={reminder().title}>
|
||||
{' · '}
|
||||
{reminder().label}
|
||||
</span>
|
||||
)}
|
||||
</Show>
|
||||
<Show when={finding.status === 'snoozed' && finding.snoozedUntil}>
|
||||
<span class="ml-2 text-blue-500 dark:text-blue-400">
|
||||
{' · '}snoozed until {formatTime(finding.snoozedUntil!)}
|
||||
</span>
|
||||
<Show when={getFindingSnoozePresentation(finding, now())}>
|
||||
{(snooze) => (
|
||||
<span class="ml-2 text-blue-500 dark:text-blue-400" title={snooze().title}>
|
||||
{' · '}
|
||||
{snooze().label}
|
||||
</span>
|
||||
)}
|
||||
</Show>
|
||||
<Show when={finding.acknowledgedAt && finding.status === 'active'}>
|
||||
<span class="ml-2 text-muted">
|
||||
|
||||
@@ -1,8 +1,9 @@
|
||||
import { fireEvent, render, screen, waitFor } from '@solidjs/testing-library';
|
||||
import type { JSX } from 'solid-js';
|
||||
import { createSignal, type JSX } from 'solid-js';
|
||||
import { beforeEach, describe, expect, it, vi } from 'vitest';
|
||||
|
||||
import { FindingsPanel } from '../FindingsPanel';
|
||||
import { RELATIVE_TIME_TICK_MS } from '@/utils/relativeTimeClock';
|
||||
|
||||
const mockState = vi.hoisted(() => {
|
||||
const loadFindings = vi.fn();
|
||||
@@ -784,3 +785,163 @@ describe('FindingsPanel investigate action gating', () => {
|
||||
expect(busyButton).toBeDisabled();
|
||||
});
|
||||
});
|
||||
|
||||
describe('FindingsPanel future times', () => {
|
||||
const hour = 60 * 60 * 1000;
|
||||
|
||||
beforeEach(() => {
|
||||
mockState.findings = [...mockState.initialFindings];
|
||||
mockState.loadPatrolFindings.mockClear();
|
||||
mockState.findingsLoading = false;
|
||||
mockState.findingsError = null;
|
||||
mockState.patrolFindingsLoading = false;
|
||||
mockState.patrolFindingsError = null;
|
||||
mockState.patrolPendingApprovals = [];
|
||||
});
|
||||
|
||||
it('counts reminders and snoozes down on the shared clock instead of reading just now', () => {
|
||||
vi.useFakeTimers({
|
||||
toFake: ['Date', 'setInterval', 'clearInterval'],
|
||||
now: Date.parse('2026-03-30T12:00:00Z'),
|
||||
});
|
||||
mockState.patrolFindings = [
|
||||
{
|
||||
...mockState.initialPatrolFindings[0],
|
||||
id: 'finding-later',
|
||||
title: 'Backup job keeps slipping',
|
||||
status: 'dismissed',
|
||||
dismissedReason: 'will_fix_later',
|
||||
remindAt: '2026-04-06T12:00:00Z',
|
||||
},
|
||||
{
|
||||
...mockState.initialPatrolFindings[0],
|
||||
id: 'finding-snoozed',
|
||||
title: 'Swap pressure on vm-100',
|
||||
status: 'snoozed',
|
||||
snoozedUntil: '2026-03-30T15:00:00Z',
|
||||
},
|
||||
];
|
||||
|
||||
try {
|
||||
render(() => <FindingsPanel findingsSource="patrol" filterOverride="resolved" />);
|
||||
|
||||
const reminder = screen.getByText(/Reminding in 7d/);
|
||||
const snooze = screen.getByText(/snoozed, returns in 3h/);
|
||||
expect(reminder).toHaveAttribute(
|
||||
'title',
|
||||
expect.stringMatching(
|
||||
/^Reminder due .+\. Pulse brings this finding back on its first reminder check after that\.$/,
|
||||
),
|
||||
);
|
||||
expect(snooze).toHaveAttribute('title', expect.stringMatching(/^Snoozed until /));
|
||||
expect(screen.queryByText(/just now/)).not.toBeInTheDocument();
|
||||
expect(screen.queryByRole('button', { name: /Overdue commitments/ })).not.toBeInTheDocument();
|
||||
|
||||
// The findings are not re-read: only the shared clock moves.
|
||||
vi.advanceTimersByTime(2 * hour);
|
||||
expect(snooze).toHaveTextContent('snoozed, returns in 1h');
|
||||
|
||||
vi.advanceTimersByTime(hour);
|
||||
expect(snooze).toHaveTextContent('snooze ended');
|
||||
|
||||
vi.setSystemTime(Date.parse('2026-04-06T12:30:00Z'));
|
||||
vi.advanceTimersByTime(RELATIVE_TIME_TICK_MS);
|
||||
expect(reminder).toHaveTextContent('Reminder overdue');
|
||||
// The Overdue commitments chip reads the same clock as the label, with no
|
||||
// findings re-read.
|
||||
expect(screen.getByRole('button', { name: /Overdue commitments \(1\)/ })).toBeInTheDocument();
|
||||
expect(reminder).toHaveAttribute('title', expect.stringMatching(/^Reminder was due /));
|
||||
} finally {
|
||||
vi.useRealTimers();
|
||||
}
|
||||
});
|
||||
|
||||
it('keeps overdue rows mounted across clock ticks and adds a reminder once it falls due', () => {
|
||||
vi.useFakeTimers({
|
||||
toFake: ['Date', 'setInterval', 'clearInterval'],
|
||||
now: Date.parse('2026-03-30T12:00:00Z'),
|
||||
});
|
||||
mockState.patrolFindings = [
|
||||
{
|
||||
...mockState.initialPatrolFindings[0],
|
||||
id: 'finding-overdue',
|
||||
title: 'Certificate renewal keeps slipping',
|
||||
status: 'dismissed',
|
||||
dismissedReason: 'will_fix_later',
|
||||
remindAt: '2026-03-30T11:00:00Z',
|
||||
},
|
||||
{
|
||||
...mockState.initialPatrolFindings[0],
|
||||
id: 'finding-due-soon',
|
||||
title: 'Backup job keeps slipping',
|
||||
status: 'dismissed',
|
||||
dismissedReason: 'will_fix_later',
|
||||
remindAt: '2026-03-30T12:10:00Z',
|
||||
},
|
||||
];
|
||||
|
||||
try {
|
||||
render(() => <FindingsPanel findingsSource="patrol" filterOverride="resolved" />);
|
||||
fireEvent.click(screen.getByRole('button', { name: /Overdue commitments \(1\)/ }));
|
||||
|
||||
const review = screen.getByRole('button', {
|
||||
name: 'Review issue for Certificate renewal keeps slipping',
|
||||
});
|
||||
review.focus();
|
||||
expect(screen.queryByText('Backup job keeps slipping')).not.toBeInTheDocument();
|
||||
|
||||
// A tick that moves no deadline past now leaves the rows alone.
|
||||
vi.advanceTimersByTime(RELATIVE_TIME_TICK_MS);
|
||||
expect(review.isConnected).toBe(true);
|
||||
expect(document.activeElement).toBe(review);
|
||||
|
||||
// Once the second reminder falls due it joins the open Overdue filter
|
||||
// and the chip count, on the same clock as the row labels.
|
||||
vi.advanceTimersByTime(10 * 60 * 1000);
|
||||
expect(screen.getByText('Backup job keeps slipping')).toBeInTheDocument();
|
||||
expect(screen.getByRole('button', { name: /Overdue commitments \(2\)/ })).toBeInTheDocument();
|
||||
} finally {
|
||||
vi.useRealTimers();
|
||||
}
|
||||
});
|
||||
|
||||
it('calls a reminder overdue only once the Overdue commitments list holds it', () => {
|
||||
vi.useFakeTimers({
|
||||
toFake: ['Date', 'setInterval', 'clearInterval'],
|
||||
now: Date.parse('2026-03-30T12:00:00Z'),
|
||||
});
|
||||
mockState.patrolFindings = [
|
||||
{
|
||||
...mockState.initialPatrolFindings[0],
|
||||
id: 'finding-due',
|
||||
title: 'Certificate renewal keeps slipping',
|
||||
status: 'dismissed',
|
||||
dismissedReason: 'will_fix_later',
|
||||
remindAt: '2026-03-30T12:00:05Z',
|
||||
},
|
||||
];
|
||||
const [filter, setFilter] = createSignal<'active' | 'resolved'>('active');
|
||||
|
||||
try {
|
||||
render(() => <FindingsPanel findingsSource="patrol" filterOverride={filter()} />);
|
||||
expect(screen.queryByText('Certificate renewal keeps slipping')).not.toBeInTheDocument();
|
||||
|
||||
// The deadline passes between ticks of the shared clock, and the row
|
||||
// first mounts before the next tick: it must not call itself overdue
|
||||
// while the chip and filter do not count it.
|
||||
vi.setSystemTime(Date.parse('2026-03-30T12:00:10Z'));
|
||||
setFilter('resolved');
|
||||
expect(screen.getByText('Certificate renewal keeps slipping')).toBeInTheDocument();
|
||||
expect(screen.getByText(/Reminder due/)).toBeInTheDocument();
|
||||
expect(screen.queryByText(/Reminder overdue/)).not.toBeInTheDocument();
|
||||
expect(screen.queryByRole('button', { name: /Overdue commitments/ })).not.toBeInTheDocument();
|
||||
|
||||
// The next tick moves the label, the chip and the filter together.
|
||||
vi.advanceTimersByTime(RELATIVE_TIME_TICK_MS);
|
||||
expect(screen.getByText(/Reminder overdue/)).toBeInTheDocument();
|
||||
expect(screen.getByRole('button', { name: /Overdue commitments \(1\)/ })).toBeInTheDocument();
|
||||
} finally {
|
||||
vi.useRealTimers();
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
@@ -22,6 +22,8 @@ import {
|
||||
getFindingSeveritySortOrder,
|
||||
getFindingResolutionReason,
|
||||
getFindingRecencyPresentation,
|
||||
getFindingReminderPresentation,
|
||||
getFindingSnoozePresentation,
|
||||
getFindingLoopStateBadgeClasses,
|
||||
getFindingLoopStateBadgeTone,
|
||||
getFindingSeverityBadgeClasses,
|
||||
@@ -369,11 +371,14 @@ describe('FindingsPanel assistant handoff', () => {
|
||||
// Once a finding is dismissed as will_fix_later, the row must surface the
|
||||
// pending reminder so the operator knows the commitment exists; otherwise
|
||||
// the only place the deadline lives is the lifecycle log. The amber tone
|
||||
// signals "pending operator action" rather than a generic muted note.
|
||||
expect(findingsPanelSource).toContain(
|
||||
"finding.dismissedReason === 'will_fix_later' && finding.remindAt",
|
||||
// signals "pending operator action" rather than a generic muted note. The
|
||||
// deadline is in the future, so it counts down on the shared clock rather
|
||||
// than going through the past-only age formatter, and it reads overdue from
|
||||
// the same list as the Overdue commitments chip.
|
||||
expect(findingsPanelSource).toMatch(
|
||||
/getFindingReminderPresentation\(\s*finding,\s*now\(\),\s*overdueFindingIdSet\(\)\.has\(finding\.id\),?\s*\)/,
|
||||
);
|
||||
expect(findingsPanelSource).toContain('Reminding {formatTime(finding.remindAt!)}');
|
||||
expect(findingsPanelSource).not.toContain('formatTime(finding.remindAt');
|
||||
expect(findingsPanelSource).toContain('text-amber-600 dark:text-amber-400');
|
||||
});
|
||||
|
||||
@@ -652,6 +657,92 @@ describe('aiFindingPresentation', () => {
|
||||
});
|
||||
});
|
||||
|
||||
describe('findingReminderAndSnoozePresentation', () => {
|
||||
const now = Date.parse('2026-03-25T12:00:00Z');
|
||||
|
||||
it('counts down to a will_fix_later reminder instead of reading it as just now', () => {
|
||||
const reminder = getFindingReminderPresentation(
|
||||
{ dismissedReason: 'will_fix_later', remindAt: '2026-03-31T12:00:00Z' },
|
||||
now,
|
||||
false,
|
||||
);
|
||||
expect(reminder?.label).toBe('Reminding in 6d');
|
||||
expect(reminder?.title).toMatch(
|
||||
/^Reminder due .+\. Pulse brings this finding back on its first reminder check after that\.$/,
|
||||
);
|
||||
expect(reminder?.title).not.toContain('still tripping');
|
||||
expect(
|
||||
getFindingReminderPresentation(
|
||||
{ dismissedReason: 'will_fix_later', remindAt: '2026-03-25T15:00:00Z' },
|
||||
now,
|
||||
false,
|
||||
)?.label,
|
||||
).toBe('Reminding in 3h');
|
||||
});
|
||||
|
||||
it('calls a reminder overdue once the Overdue commitments list holds it', () => {
|
||||
const reminder = getFindingReminderPresentation(
|
||||
{ dismissedReason: 'will_fix_later', remindAt: '2026-03-25T11:00:00Z' },
|
||||
now,
|
||||
true,
|
||||
);
|
||||
expect(reminder?.label).toBe('Reminder overdue');
|
||||
expect(reminder?.title).toMatch(
|
||||
/^Reminder was due .+\. Pulse brings this finding back on its next reminder check\.$/,
|
||||
);
|
||||
// Past the deadline but not yet on the list (it is sampled on the next
|
||||
// clock tick), so the chip and filter would not count it yet.
|
||||
expect(
|
||||
getFindingReminderPresentation(
|
||||
{ dismissedReason: 'will_fix_later', remindAt: '2026-03-25T11:59:50Z' },
|
||||
now,
|
||||
false,
|
||||
)?.label,
|
||||
).toBe('Reminder due');
|
||||
});
|
||||
|
||||
it('shows no reminder for other dismissals or an unusable remind time', () => {
|
||||
expect(
|
||||
getFindingReminderPresentation(
|
||||
{ dismissedReason: 'not_an_issue', remindAt: '2026-03-31T12:00:00Z' },
|
||||
now,
|
||||
false,
|
||||
),
|
||||
).toBeNull();
|
||||
expect(
|
||||
getFindingReminderPresentation({ dismissedReason: 'will_fix_later' }, now, false),
|
||||
).toBeNull();
|
||||
expect(
|
||||
getFindingReminderPresentation(
|
||||
{ dismissedReason: 'will_fix_later', remindAt: 'not a date' },
|
||||
now,
|
||||
true,
|
||||
),
|
||||
).toBeNull();
|
||||
});
|
||||
|
||||
it('counts down to the end of a snooze and keeps the exact time on hover', () => {
|
||||
const snooze = getFindingSnoozePresentation(
|
||||
{ status: 'snoozed', snoozedUntil: '2026-03-25T15:00:00Z' },
|
||||
now,
|
||||
);
|
||||
expect(snooze?.label).toBe('snoozed, returns in 3h');
|
||||
expect(snooze?.title).toMatch(/^Snoozed until .+/);
|
||||
expect(
|
||||
getFindingSnoozePresentation(
|
||||
{ status: 'snoozed', snoozedUntil: '2026-03-25T11:00:00Z' },
|
||||
now,
|
||||
)?.label,
|
||||
).toBe('snooze ended');
|
||||
expect(
|
||||
getFindingSnoozePresentation(
|
||||
{ status: 'active', snoozedUntil: '2026-03-25T15:00:00Z' },
|
||||
now,
|
||||
),
|
||||
).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
describe('patrolFindingClassification', () => {
|
||||
it('describes grouped Patrol findings as issues instead of internal signals', () => {
|
||||
expect(getPatrolFindingIssueCountLabel(0)).toBe('0 issues');
|
||||
@@ -997,7 +1088,8 @@ describe('aiFindingPresentation', () => {
|
||||
it('uses explicit textual separators in finding metadata instead of relying on visual spacing', () => {
|
||||
expect(findingsPanelSource).toContain("{' · '}acknowledged");
|
||||
expect(findingsPanelSource).toContain("{' · '}last investigated");
|
||||
expect(findingsPanelSource).toContain("{' · '}snoozed until");
|
||||
expect(findingsPanelSource).toMatch(/\{' · '\}\s*\{snooze\(\)\.label\}/);
|
||||
expect(findingsPanelSource).toMatch(/\{' · '\}\s*\{reminder\(\)\.label\}/);
|
||||
});
|
||||
|
||||
it('keeps the Open work heading free of a duplicate count badge', () => {
|
||||
|
||||
@@ -43,12 +43,11 @@ import {
|
||||
ALERT_RESOURCE_METRIC_OFF_VALUE,
|
||||
getAlertResourceColumnKind,
|
||||
getAlertResourceColumnHeaderTooltip,
|
||||
getAlertResourceEnabledDefault,
|
||||
getAlertResourceMetricBounds,
|
||||
getAlertResourceMetricDelayOverride,
|
||||
getAlertResourceMetricStep,
|
||||
isAlertResourceMetricOff,
|
||||
normalizeAlertResourceMetricKey,
|
||||
resolveAlertResourceGlobalDefaultCell,
|
||||
} from './alertResourceTableModel';
|
||||
import type { OfflineState, Resource, ResourceTableProps } from './ResourceTable';
|
||||
|
||||
@@ -210,7 +209,10 @@ export function AlertResourceTableDesktop(props: AlertResourceTableDesktopProps)
|
||||
{(column) => (
|
||||
<TableHead
|
||||
class={`${getPlatformTableHeadClassForKind(getAlertResourceColumnKind(column))} whitespace-normal wrap-break-word`}
|
||||
title={getAlertResourceColumnHeaderTooltip(column)}
|
||||
title={
|
||||
props.table.columnTooltips?.[column] ??
|
||||
getAlertResourceColumnHeaderTooltip(column)
|
||||
}
|
||||
>
|
||||
{column}
|
||||
</TableHead>
|
||||
@@ -281,10 +283,14 @@ export function AlertResourceTableDesktop(props: AlertResourceTableDesktopProps)
|
||||
{(column) => {
|
||||
const metric = normalizeAlertResourceMetricKey(column);
|
||||
const bounds = getAlertResourceMetricBounds(metric);
|
||||
const value = () => props.table.globalDefaults?.[metric] ?? 0;
|
||||
const value = () => props.table.globalDefaults?.[metric];
|
||||
const fallback = () => props.table.globalDefaultFallbacks?.[metric];
|
||||
// An unset default and a stored 0 are both disabled to the
|
||||
// alert engine, so neither may render as On.
|
||||
const isOff = () => isAlertResourceMetricOff(value());
|
||||
// alert engine, so neither may render as On, unless the
|
||||
// metric's unset default follows another setting.
|
||||
const cell = () =>
|
||||
resolveAlertResourceGlobalDefaultCell(metric, value(), fallback());
|
||||
const isOff = () => cell().isOff;
|
||||
|
||||
return (
|
||||
<TableCell
|
||||
@@ -297,8 +303,12 @@ export function AlertResourceTableDesktop(props: AlertResourceTableDesktopProps)
|
||||
min={bounds.min}
|
||||
max={bounds.max}
|
||||
step={getAlertResourceMetricStep(metric)}
|
||||
value={isOff() ? '' : value()}
|
||||
placeholder={getAlertResourceTableMetricPlaceholder(isOff())}
|
||||
value={isOff() || cell().follows ? '' : (value() ?? '')}
|
||||
placeholder={
|
||||
cell().follows && !isOff()
|
||||
? fallback()?.label
|
||||
: getAlertResourceTableMetricPlaceholder(isOff())
|
||||
}
|
||||
disabled={isOff()}
|
||||
onInput={(e) => {
|
||||
const nextValue = parseFloat(e.currentTarget.value);
|
||||
@@ -318,12 +328,16 @@ export function AlertResourceTableDesktop(props: AlertResourceTableDesktopProps)
|
||||
}));
|
||||
props.table.setHasUnsavedChanges?.(true);
|
||||
}}
|
||||
class={`w-16 px-2 py-0.5 text-sm text-center border rounded ${
|
||||
class={`${cell().follows && !isOff() ? 'w-20' : 'w-16'} px-2 py-0.5 text-sm text-center border rounded ${
|
||||
isOff()
|
||||
? 'border-border bg-surface-alt text-muted italic placeholder: dark:placeholder: placeholder:opacity-60 pointer-events-none'
|
||||
: 'border-border text-base-content focus:border-blue-500 focus:ring-1 focus:ring-blue-500'
|
||||
}`}
|
||||
title={getAlertResourceTableMetricInputTitle(isOff())}
|
||||
title={
|
||||
cell().follows
|
||||
? fallback()?.title
|
||||
: getAlertResourceTableMetricInputTitle(isOff())
|
||||
}
|
||||
/>
|
||||
<Show when={isOff()}>
|
||||
<button
|
||||
@@ -332,7 +346,7 @@ export function AlertResourceTableDesktop(props: AlertResourceTableDesktopProps)
|
||||
onClick={() => {
|
||||
props.table.setGlobalDefaults?.((prev) => ({
|
||||
...prev,
|
||||
[metric]: getAlertResourceEnabledDefault(metric),
|
||||
[metric]: cell().enableValue,
|
||||
}));
|
||||
props.table.setHasUnsavedChanges?.(true);
|
||||
}}
|
||||
@@ -348,7 +362,7 @@ export function AlertResourceTableDesktop(props: AlertResourceTableDesktopProps)
|
||||
props.table.setGlobalDefaults?.((prev) => ({
|
||||
...prev,
|
||||
[metric]: isOff()
|
||||
? getAlertResourceEnabledDefault(metric)
|
||||
? cell().enableValue
|
||||
: ALERT_RESOURCE_METRIC_OFF_VALUE,
|
||||
}));
|
||||
props.table.setHasUnsavedChanges?.(true);
|
||||
|
||||
@@ -26,7 +26,6 @@ import {
|
||||
ALERT_RESOURCE_METRIC_OFF_VALUE,
|
||||
alertResourceSupportsMetric,
|
||||
buildAlertResourceEditPayload,
|
||||
getAlertResourceEnabledDefault,
|
||||
getAlertResourceLabel,
|
||||
getAlertResourceMetricBounds,
|
||||
getAlertResourceMetricDisplayValue,
|
||||
@@ -34,6 +33,7 @@ import {
|
||||
isAlertResourceMetricOff,
|
||||
isAlertResourceMetricOverridden,
|
||||
normalizeAlertResourceMetricKey,
|
||||
resolveAlertResourceGlobalDefaultCell,
|
||||
resolveAlertResourceMetricEnableValue,
|
||||
} from './alertResourceTableModel';
|
||||
import type { Resource } from '@/features/alerts/thresholds/tableTypes';
|
||||
@@ -212,10 +212,14 @@ export function AlertResourceTableMobile(props: AlertResourceTableMobileProps) {
|
||||
{(column) => {
|
||||
const metric = normalizeAlertResourceMetricKey(column);
|
||||
const bounds = getAlertResourceMetricBounds(metric);
|
||||
const value = () => props.table.globalDefaults?.[metric] ?? 0;
|
||||
const value = () => props.table.globalDefaults?.[metric];
|
||||
const fallback = () => props.table.globalDefaultFallbacks?.[metric];
|
||||
// An unset default and a stored 0 are both disabled to the alert
|
||||
// engine, so neither may render as On.
|
||||
const isOff = () => isAlertResourceMetricOff(value());
|
||||
// engine, so neither may render as On, unless the metric's unset
|
||||
// default follows another setting.
|
||||
const cell = () =>
|
||||
resolveAlertResourceGlobalDefaultCell(metric, value(), fallback());
|
||||
const isOff = () => cell().isOff;
|
||||
|
||||
return (
|
||||
<div class="p-2 bg-surface rounded-sm border border-border-subtle flex flex-col gap-1">
|
||||
@@ -227,8 +231,13 @@ export function AlertResourceTableMobile(props: AlertResourceTableMobileProps) {
|
||||
min={bounds.min}
|
||||
max={bounds.max}
|
||||
step={getAlertResourceMetricStep(metric)}
|
||||
value={isOff() ? '' : value()}
|
||||
placeholder={getAlertResourceTableMetricPlaceholder(isOff())}
|
||||
value={isOff() || cell().follows ? '' : (value() ?? '')}
|
||||
placeholder={
|
||||
cell().follows && !isOff()
|
||||
? fallback()?.label
|
||||
: getAlertResourceTableMetricPlaceholder(isOff())
|
||||
}
|
||||
title={cell().follows ? fallback()?.title : undefined}
|
||||
disabled={isOff()}
|
||||
class={`min-h-11 w-full rounded-sm border p-1 text-center text-sm sm:min-h-0 ${isOff() ? 'bg-surface-hover' : ' border-border'}`}
|
||||
onInput={(e) => {
|
||||
@@ -257,7 +266,7 @@ export function AlertResourceTableMobile(props: AlertResourceTableMobileProps) {
|
||||
onClick={() => {
|
||||
props.table.setGlobalDefaults?.((prev) => ({
|
||||
...prev,
|
||||
[metric]: getAlertResourceEnabledDefault(metric),
|
||||
[metric]: cell().enableValue,
|
||||
}));
|
||||
props.table.setHasUnsavedChanges?.(true);
|
||||
}}
|
||||
@@ -271,7 +280,7 @@ export function AlertResourceTableMobile(props: AlertResourceTableMobileProps) {
|
||||
props.table.setGlobalDefaults?.((prev) => ({
|
||||
...prev,
|
||||
[metric]: isOff()
|
||||
? getAlertResourceEnabledDefault(metric)
|
||||
? cell().enableValue
|
||||
: ALERT_RESOURCE_METRIC_OFF_VALUE,
|
||||
}));
|
||||
props.table.setHasUnsavedChanges?.(true);
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest';
|
||||
import { createSignal } from 'solid-js';
|
||||
import { cleanup, fireEvent, render, screen } from '@solidjs/testing-library';
|
||||
import { ALERT_BULK_EDIT_CLEAR_LABEL } from '@/utils/alertBulkEditPresentation';
|
||||
import resourceTableSource from '@/components/Alerts/ResourceTable.tsx?raw';
|
||||
@@ -1007,6 +1008,144 @@ describe('ResourceTable', () => {
|
||||
expect(updater({ cpu: 0 })).toEqual({ cpu: 80 });
|
||||
});
|
||||
|
||||
it.each([
|
||||
['desktop', false],
|
||||
['mobile', true],
|
||||
])('lets an unset %s global default follow its fallback', (_layout, mobile) => {
|
||||
mockIsMobile.mockReturnValue(mobile);
|
||||
const setGlobalDefaults = vi.fn();
|
||||
const props = makeProps({
|
||||
resources: [],
|
||||
columns: ['Temp °C'],
|
||||
globalDefaults: { temperature: undefined },
|
||||
globalDefaultFallbacks: {
|
||||
temperature: { label: 'By type', title: 'Follows Disk temperature by type' },
|
||||
},
|
||||
setGlobalDefaults,
|
||||
setHasUnsavedChanges: vi.fn(),
|
||||
});
|
||||
render(() => <ResourceTable {...props} />);
|
||||
|
||||
// Unset reads On and names where the value comes from.
|
||||
expect(screen.getByTestId('status-badge')).toHaveTextContent('On');
|
||||
const input = screen.getByPlaceholderText('By type') as HTMLInputElement;
|
||||
expect(input.value).toBe('');
|
||||
expect(input.disabled).toBe(false);
|
||||
expect(input.title).toBe('Follows Disk temperature by type');
|
||||
|
||||
fireEvent.click(screen.getByTestId('status-badge'));
|
||||
const updater = setGlobalDefaults.mock.calls[0][0] as (
|
||||
prev: Record<string, number | undefined>,
|
||||
) => Record<string, number | undefined>;
|
||||
expect(updater({ temperature: undefined })).toEqual({ temperature: -1 });
|
||||
});
|
||||
|
||||
it.each([
|
||||
['desktop', false],
|
||||
['mobile', true],
|
||||
])('switches an Off %s fallback default back to following', (_layout, mobile) => {
|
||||
mockIsMobile.mockReturnValue(mobile);
|
||||
const setGlobalDefaults = vi.fn();
|
||||
const props = makeProps({
|
||||
resources: [],
|
||||
columns: ['Temp °C'],
|
||||
globalDefaults: { temperature: -1 },
|
||||
globalDefaultFallbacks: {
|
||||
temperature: { label: 'By type', title: 'Follows Disk temperature by type' },
|
||||
},
|
||||
setGlobalDefaults,
|
||||
setHasUnsavedChanges: vi.fn(),
|
||||
});
|
||||
render(() => <ResourceTable {...props} />);
|
||||
|
||||
expect(screen.getByTestId('status-badge')).toHaveTextContent('Off');
|
||||
fireEvent.click(screen.getByTestId('status-badge'));
|
||||
const updater = setGlobalDefaults.mock.calls[0][0] as (
|
||||
prev: Record<string, number | undefined>,
|
||||
) => Record<string, number | undefined>;
|
||||
expect(updater({ temperature: -1 })).toEqual({ temperature: undefined });
|
||||
});
|
||||
|
||||
it.each([
|
||||
['desktop', false],
|
||||
['mobile', true],
|
||||
])('reads an unset %s default Off when its fallback is off', (_layout, mobile) => {
|
||||
mockIsMobile.mockReturnValue(mobile);
|
||||
const setGlobalDefaults = vi.fn();
|
||||
const props = makeProps({
|
||||
resources: [],
|
||||
columns: ['Temp °C'],
|
||||
globalDefaults: { temperature: undefined },
|
||||
globalDefaultFallbacks: {
|
||||
temperature: { label: 'By type', title: 'Off by type', off: true, enableValue: 55 },
|
||||
},
|
||||
setGlobalDefaults,
|
||||
setHasUnsavedChanges: vi.fn(),
|
||||
});
|
||||
render(() => <ResourceTable {...props} />);
|
||||
|
||||
expect(screen.getByTestId('status-badge')).toHaveTextContent('Off');
|
||||
expect(screen.queryByPlaceholderText('By type')).toBeNull();
|
||||
|
||||
// Unset would still be off, so switching on stages an explicit value.
|
||||
fireEvent.click(screen.getByTestId('status-badge'));
|
||||
const updater = setGlobalDefaults.mock.calls[0][0] as (
|
||||
prev: Record<string, number | undefined>,
|
||||
) => Record<string, number | undefined>;
|
||||
expect(updater({ temperature: undefined })).toEqual({ temperature: 55 });
|
||||
});
|
||||
|
||||
it('shows a set fallback global default as its number', () => {
|
||||
const props = makeProps({
|
||||
resources: [],
|
||||
columns: ['Temp °C'],
|
||||
globalDefaults: { temperature: 62 },
|
||||
globalDefaultFallbacks: {
|
||||
temperature: { label: 'By type', title: 'Follows Disk temperature by type' },
|
||||
},
|
||||
setGlobalDefaults: vi.fn(),
|
||||
setHasUnsavedChanges: vi.fn(),
|
||||
});
|
||||
render(() => <ResourceTable {...props} />);
|
||||
|
||||
expect(screen.getByDisplayValue('62')).toBeInTheDocument();
|
||||
expect(screen.queryByPlaceholderText('By type')).toBeNull();
|
||||
});
|
||||
|
||||
it('follows global defaults that arrive after the table mounts', () => {
|
||||
const [globalDefaults, setGlobalDefaults] = createSignal<Record<string, number | undefined>>({
|
||||
temperature: undefined,
|
||||
});
|
||||
const props = makeProps({
|
||||
resources: [],
|
||||
columns: ['Temp °C'],
|
||||
factoryDefaults: { temperature: undefined },
|
||||
onResetDefaults: vi.fn(),
|
||||
setGlobalDefaults: vi.fn(),
|
||||
setHasUnsavedChanges: vi.fn(),
|
||||
});
|
||||
render(() => <ResourceTable {...props} globalDefaults={globalDefaults()} />);
|
||||
|
||||
expect(screen.queryByText('Custom')).toBeNull();
|
||||
// The saved config loads after mount.
|
||||
setGlobalDefaults({ temperature: 62 });
|
||||
expect(screen.getByText('Custom')).toBeInTheDocument();
|
||||
});
|
||||
|
||||
it('uses a table column tooltip over the generic one', () => {
|
||||
const props = makeProps({
|
||||
resources: [],
|
||||
columns: ['Temp °C'],
|
||||
columnTooltips: { 'Temp °C': 'Disk temperature that raises an alert.' },
|
||||
});
|
||||
render(() => <ResourceTable {...props} />);
|
||||
|
||||
expect(screen.getByText('Temp °C', { selector: 'th' })).toHaveAttribute(
|
||||
'title',
|
||||
'Disk temperature that raises an alert.',
|
||||
);
|
||||
});
|
||||
|
||||
it('stages an explicit threshold from the mobile row editor', () => {
|
||||
mockIsMobile.mockReturnValue(true);
|
||||
const setEditingThresholds = vi.fn();
|
||||
|
||||
@@ -11,6 +11,7 @@ import { ActionIconButton } from '@/components/shared/Button';
|
||||
import { useObservedElementWidth } from '@/hooks/useObservedElementWidth';
|
||||
import { useAlertResourceTableState } from './useAlertResourceTableState';
|
||||
import type { GroupHeaderMeta, Resource } from '@/features/alerts/thresholds/tableTypes';
|
||||
import type { AlertResourceGlobalDefaultFallback } from './alertResourceTableModel';
|
||||
import { AlertResourceTableDesktop } from './AlertResourceTableDesktop';
|
||||
import { AlertResourceTableMobile } from './AlertResourceTableMobile';
|
||||
|
||||
@@ -49,6 +50,10 @@ export interface ResourceTableProps {
|
||||
formatMetricValue: (metric: string, value: number | undefined) => string;
|
||||
hasActiveAlert: (resourceId: string, metric: string) => boolean;
|
||||
globalDefaults?: Record<string, number | undefined>;
|
||||
/** Metrics whose unset global default follows another setting. */
|
||||
globalDefaultFallbacks?: Record<string, AlertResourceGlobalDefaultFallback>;
|
||||
/** Header tooltips that replace the generic one for a column label. */
|
||||
columnTooltips?: Record<string, string>;
|
||||
setGlobalDefaults?: (
|
||||
value:
|
||||
| Record<string, number | undefined>
|
||||
@@ -97,10 +102,20 @@ export function ResourceTable(props: ResourceTableProps) {
|
||||
someSelected,
|
||||
clearSelectedIds,
|
||||
} = useAlertResourceTableState({
|
||||
resources: props.resources,
|
||||
groupedResources: props.groupedResources,
|
||||
globalDefaults: props.globalDefaults,
|
||||
factoryDefaults: props.factoryDefaults,
|
||||
// Getters keep the state reactive: saved defaults arrive after mount, and
|
||||
// the Custom badge and reset control must follow them.
|
||||
get resources() {
|
||||
return props.resources;
|
||||
},
|
||||
get groupedResources() {
|
||||
return props.groupedResources;
|
||||
},
|
||||
get globalDefaults() {
|
||||
return props.globalDefaults;
|
||||
},
|
||||
get factoryDefaults() {
|
||||
return props.factoryDefaults;
|
||||
},
|
||||
});
|
||||
const totalColumnCount = createMemo(
|
||||
() =>
|
||||
|
||||
@@ -82,9 +82,10 @@ export function ThresholdsTableAgentsResourcesSection(props: ThresholdsTableSect
|
||||
<Card padding="md" tone="card" class="mt-4">
|
||||
<h3 class="text-sm font-semibold text-base-content">Disk temperature by type</h3>
|
||||
<p class="mt-1 text-xs text-muted">
|
||||
Alert trigger in °C for each disk type. Warning colors start 5°C below the trigger.
|
||||
Setting a Disk Temp override on a host above replaces these for all of that host's
|
||||
disks.
|
||||
Alert trigger in °C for each disk type, for agent, TrueNAS and Proxmox disks. Warning
|
||||
colors start 5°C below the trigger. A Disk Temp override on a host above replaces
|
||||
these for all of that host's disks, and a TrueNAS Disks value replaces them for
|
||||
TrueNAS disks.
|
||||
</p>
|
||||
<div class="mt-4 grid gap-4 sm:grid-cols-3">
|
||||
<For each={DISK_TEMP_TYPE_FIELDS}>
|
||||
|
||||
@@ -8,6 +8,9 @@ import { CollapsibleSection } from './Thresholds/sections/CollapsibleSection';
|
||||
import { formatMetricValue } from '@/features/alerts/thresholds/helpers';
|
||||
import type { ThresholdsTableSectionProps } from '@/features/alerts/thresholds/thresholdsTableSectionProps';
|
||||
import type { Resource } from '@/features/alerts/thresholds/tableTypes';
|
||||
import { getDiskTemperatureByTypeItems } from '@/features/alerts/thresholds/trueNASDiskTemperature';
|
||||
import { FACTORY_AGENT_DEFAULTS } from '@/utils/alertThresholdDefaults';
|
||||
import type { AlertResourceGlobalDefaultFallback } from './alertResourceTableModel';
|
||||
import { getAlertThresholdsDefaultsSummary } from '@/utils/alertThresholdsSectionPresentation';
|
||||
|
||||
const TRUENAS_SYSTEM_COLUMNS = [
|
||||
@@ -22,6 +25,7 @@ const TRUENAS_SYSTEM_COLUMNS = [
|
||||
];
|
||||
const TRUENAS_STORAGE_COLUMNS = ['Usage %'];
|
||||
const TRUENAS_DISK_COLUMNS = ['Temp °C'];
|
||||
const TRUENAS_DISK_COLUMN_TOOLTIPS = { 'Temp °C': 'Disk temperature that raises an alert.' };
|
||||
|
||||
function TrueNASResourceSection(
|
||||
props: ThresholdsTableSectionProps & {
|
||||
@@ -39,6 +43,10 @@ function TrueNASResourceSection(
|
||||
| ((prev: Record<string, number | undefined>) => Record<string, number | undefined>),
|
||||
) => void;
|
||||
onResetDefaults?: () => void;
|
||||
defaultsSummary?: string;
|
||||
globalDefaultFallbacks?: Record<string, AlertResourceGlobalDefaultFallback>;
|
||||
columnTooltips?: Record<string, string>;
|
||||
note?: string;
|
||||
},
|
||||
) {
|
||||
const { state, tableProps } = props;
|
||||
@@ -48,7 +56,9 @@ function TrueNASResourceSection(
|
||||
<CollapsibleSection
|
||||
id={props.id}
|
||||
title={props.title}
|
||||
defaultsSummary={getAlertThresholdsDefaultsSummary(props.columns, props.defaults)}
|
||||
defaultsSummary={
|
||||
props.defaultsSummary ?? getAlertThresholdsDefaultsSummary(props.columns, props.defaults)
|
||||
}
|
||||
resourceCount={props.resources().length}
|
||||
collapsed={state.isCollapsed(props.id)}
|
||||
onToggle={() => state.toggleSection(props.id)}
|
||||
@@ -57,6 +67,9 @@ function TrueNASResourceSection(
|
||||
emptyMessage="No TrueNAS alert targets match the current filters."
|
||||
>
|
||||
<div ref={state.registerSection(props.id)} class="scroll-mt-24">
|
||||
<Show when={props.note}>
|
||||
<p class="mb-3 text-xs text-muted">{props.note}</p>
|
||||
</Show>
|
||||
<ResourceTable
|
||||
title=""
|
||||
onConfigureResourceIntent={tableProps.onConfigureResourceIntent}
|
||||
@@ -79,6 +92,8 @@ function TrueNASResourceSection(
|
||||
formatMetricValue={formatMetricValue}
|
||||
hasActiveAlert={state.hasActiveAlert}
|
||||
globalDefaults={props.defaults}
|
||||
globalDefaultFallbacks={props.globalDefaultFallbacks}
|
||||
columnTooltips={props.columnTooltips}
|
||||
setGlobalDefaults={props.setGlobalDefaults}
|
||||
setHasUnsavedChanges={tableProps.setHasUnsavedChanges}
|
||||
globalDisableFlag={tableProps.disableAllTrueNAS}
|
||||
@@ -104,6 +119,38 @@ export function ThresholdsTableTrueNASTab(props: ThresholdsTableSectionProps) {
|
||||
const trueNASDefaults = () => props.tableProps.trueNASDefaults ?? {};
|
||||
const trueNASStorageDefaults = () => ({ usage: trueNASDefaults().usage ?? 85 });
|
||||
const trueNASDiskDefaults = () => props.tableProps.trueNASDiskDefaults ?? {};
|
||||
const diskTemperatureByType = () =>
|
||||
getDiskTemperatureByTypeItems({
|
||||
agentDiskTemperature: props.tableProps.agentDefaults.diskTemperature,
|
||||
diskTempByType: props.tableProps.diskTempByType,
|
||||
});
|
||||
// Unset, TrueNAS disks follow Disk temperature by type, so the header names
|
||||
// the per-type triggers rather than one number.
|
||||
const trueNASDiskDefaultsSummary = () => {
|
||||
if (trueNASDiskDefaults().temperature !== undefined) return undefined;
|
||||
return getAlertThresholdsDefaultsSummary(
|
||||
TRUENAS_DISK_COLUMNS,
|
||||
diskTemperatureByType().length > 0 ? {} : { temperature: 0 },
|
||||
{ ruleItems: diskTemperatureByType() },
|
||||
);
|
||||
};
|
||||
// When the agent Disk Temp default is off, Disk temperature by type is off
|
||||
// too, so the unset cell reads Off and switching it on stages a TrueNAS-wide
|
||||
// value at the factory disk temperature trigger.
|
||||
const trueNASDiskTemperatureFallback = (): Record<string, AlertResourceGlobalDefaultFallback> => {
|
||||
const items = diskTemperatureByType();
|
||||
return {
|
||||
temperature: {
|
||||
label: 'By type',
|
||||
title:
|
||||
items.length > 0
|
||||
? `Follows Disk temperature by type under Agents: ${items.join(', ')}. Enter a value to use one temperature for every TrueNAS disk.`
|
||||
: 'Off because the agent Disk Temp default under Agents is off. Turn it on here to alert on TrueNAS disks anyway.',
|
||||
off: items.length === 0,
|
||||
enableValue: FACTORY_AGENT_DEFAULTS.diskTemperature,
|
||||
},
|
||||
};
|
||||
};
|
||||
|
||||
return (
|
||||
<>
|
||||
@@ -175,6 +222,10 @@ export function ThresholdsTableTrueNASTab(props: ThresholdsTableSectionProps) {
|
||||
icon={<HardDrive class="w-5 h-5" />}
|
||||
typeKey="truenas-disk"
|
||||
defaults={trueNASDiskDefaults()}
|
||||
defaultsSummary={trueNASDiskDefaultsSummary()}
|
||||
globalDefaultFallbacks={trueNASDiskTemperatureFallback()}
|
||||
columnTooltips={TRUENAS_DISK_COLUMN_TOOLTIPS}
|
||||
note="Unless set here, each disk alerts at Disk temperature by type, set under Agents."
|
||||
factoryDefaults={props.tableProps.factoryTrueNASDiskDefaults}
|
||||
setGlobalDefaults={props.tableProps.setTrueNASDiskDefaults}
|
||||
onResetDefaults={props.tableProps.resetTrueNASDiskDefaults}
|
||||
|
||||
@@ -457,6 +457,25 @@ describe('ThresholdsTable navigation and redirection', () => {
|
||||
});
|
||||
});
|
||||
|
||||
it('says Disk temperature by type covers agent, TrueNAS and Proxmox disks', async () => {
|
||||
setPathname('/alerts/thresholds/systems');
|
||||
const host: Agent = {
|
||||
id: 'legacy-h1',
|
||||
hostname: 'legacy-host',
|
||||
displayName: 'Legacy Host',
|
||||
status: 'online',
|
||||
lastSeen: 123,
|
||||
memory: { total: 100, used: 50, free: 50, usage: 50 },
|
||||
};
|
||||
|
||||
render(() => <ThresholdsTable {...(baseProps() as any)} agents={[host]} />);
|
||||
|
||||
const heading = await screen.findByText('Disk temperature by type');
|
||||
expect(heading.parentElement?.textContent).toContain(
|
||||
'for each disk type, for agent, TrueNAS and Proxmox disks',
|
||||
);
|
||||
});
|
||||
|
||||
it('navigates to correct route when tabs are clicked', () => {
|
||||
render(() => <ThresholdsTable {...(baseProps() as any)} />);
|
||||
|
||||
|
||||
@@ -244,6 +244,49 @@ export function resolveAlertResourceMetricEnableValue(
|
||||
return undefined;
|
||||
}
|
||||
|
||||
/**
|
||||
* A global default that may stay unset because another setting decides it,
|
||||
* such as TrueNAS disk temperature following Disk temperature by type. Unset,
|
||||
* the cell is empty with `label` as its placeholder and `title` explaining
|
||||
* where the value comes from. `off` says the other setting is itself off, so
|
||||
* the unset cell reads Off and switching it on has to stage `enableValue`.
|
||||
*/
|
||||
export interface AlertResourceGlobalDefaultFallback {
|
||||
label: string;
|
||||
title: string;
|
||||
off?: boolean;
|
||||
enableValue?: number;
|
||||
}
|
||||
|
||||
export interface AlertResourceGlobalDefaultCell {
|
||||
/** Unset and decided by the fallback. */
|
||||
follows: boolean;
|
||||
isOff: boolean;
|
||||
/** Value staged when the cell is switched back on from Off. */
|
||||
enableValue: number | undefined;
|
||||
}
|
||||
|
||||
/**
|
||||
* State of one Global Defaults metric cell. An unset default is Off, as the
|
||||
* alert engine reads it, unless the metric has a fallback; then it reads as
|
||||
* the fallback does. Switching it on from Off returns it to unset when the
|
||||
* fallback is on, and stages an explicit value when the fallback is off,
|
||||
* since unset would still be off.
|
||||
*/
|
||||
export function resolveAlertResourceGlobalDefaultCell(
|
||||
metric: string,
|
||||
value: number | undefined,
|
||||
fallback: AlertResourceGlobalDefaultFallback | undefined,
|
||||
): AlertResourceGlobalDefaultCell {
|
||||
const follows = value === undefined && fallback !== undefined;
|
||||
const explicitEnableValue = fallback?.enableValue ?? getAlertResourceEnabledDefault(metric);
|
||||
return {
|
||||
follows,
|
||||
isOff: follows ? Boolean(fallback?.off) : isAlertResourceMetricOff(value ?? 0),
|
||||
enableValue: fallback && !fallback.off ? undefined : explicitEnableValue,
|
||||
};
|
||||
}
|
||||
|
||||
export function getAlertResourceMetricDelayOverride(
|
||||
metricDelaySeconds: Record<string, number> | undefined,
|
||||
metric: string,
|
||||
|
||||
+29
-6
@@ -830,16 +830,39 @@ describe('buildTrueNASVMSections additional branches', () => {
|
||||
// =========================================================================
|
||||
|
||||
describe('buildTrueNASDiskSections additional branches', () => {
|
||||
it('sets Temperature tone to warning at 55 degrees', () => {
|
||||
expect(findRow(diskRes({ temperature: 55 }), 'Temperature')?.tone).toBe('warning');
|
||||
// The drawer judges heat by the disk type's alert trigger, through the
|
||||
// resolver the alerts store provides; these are the factory triggers.
|
||||
const factoryTriggers: Record<string, number> = { nvme: 70, sas: 65, sata: 55 };
|
||||
const resolveThresholds = (diskType: string) => {
|
||||
const critical = factoryTriggers[diskType.toLowerCase()] ?? 55;
|
||||
return { warning: critical - 5, critical };
|
||||
};
|
||||
const temperatureTone = (disk: Partial<ResourcePhysicalDiskMeta>) =>
|
||||
buildTrueNASDetailSections(diskRes(disk), resolveThresholds)
|
||||
.flatMap((section) => section.rows)
|
||||
.find((row) => row.label === 'Temperature')?.tone;
|
||||
|
||||
it('sets Temperature tone to warning from the disk type trigger', () => {
|
||||
expect(temperatureTone({ diskType: 'sata', temperature: 55 })).toBe('warning');
|
||||
expect(temperatureTone({ diskType: 'sas', temperature: 65 })).toBe('warning');
|
||||
expect(temperatureTone({ diskType: 'nvme', temperature: 70 })).toBe('warning');
|
||||
});
|
||||
|
||||
it('sets Temperature tone to warning above 55 degrees', () => {
|
||||
expect(findRow(diskRes({ temperature: 60 }), 'Temperature')?.tone).toBe('warning');
|
||||
it('keeps a reading under its type trigger at default tone', () => {
|
||||
expect(temperatureTone({ diskType: 'nvme', temperature: 60 })).toBe('default');
|
||||
expect(temperatureTone({ diskType: 'sas', temperature: 63 })).toBe('default');
|
||||
expect(temperatureTone({ diskType: 'sata', temperature: 40 })).toBe('default');
|
||||
});
|
||||
|
||||
it('sets Temperature tone to default below 55 degrees', () => {
|
||||
expect(findRow(diskRes({ temperature: 40 }), 'Temperature')?.tone).toBe('default');
|
||||
it('judges no heat without thresholds', () => {
|
||||
expect(findRow(diskRes({ diskType: 'sata', temperature: 80 }), 'Temperature')?.tone).toBe(
|
||||
'default',
|
||||
);
|
||||
expect(
|
||||
buildTrueNASDetailSections(diskRes({ diskType: 'sata', temperature: 80 }), () => null)
|
||||
.flatMap((section) => section.rows)
|
||||
.find((row) => row.label === 'Temperature')?.tone,
|
||||
).toBe('default');
|
||||
});
|
||||
|
||||
it('formats Wearout percentage for a valid value', () => {
|
||||
|
||||
+2
-1
@@ -381,8 +381,9 @@ describe('resourceDetailDrawerTrueNASModel', () => {
|
||||
},
|
||||
},
|
||||
});
|
||||
const sataTrigger = () => ({ warning: 50, critical: 55 });
|
||||
const temperatureRow = (resource: Resource) =>
|
||||
buildTrueNASDetailSections(resource)
|
||||
buildTrueNASDetailSections(resource, sataTrigger)
|
||||
.find((section) => section.label === 'Health')
|
||||
?.rows.find((row) => row.label === 'Temperature');
|
||||
|
||||
|
||||
@@ -20,7 +20,11 @@ import {
|
||||
type DetailSection,
|
||||
type DetailValueTone,
|
||||
} from '@/components/shared/detailSectionModel';
|
||||
import { getPhysicalDiskTemperaturePresentation } from '@/features/storageBackups/diskTemperaturePresentation';
|
||||
import {
|
||||
getPhysicalDiskTemperaturePresentation,
|
||||
isPhysicalDiskRunningHot,
|
||||
} from '@/features/storageBackups/diskTemperaturePresentation';
|
||||
import type { TrueNASDiskTemperatureThresholdResolver } from '@/features/truenas/truenasPageModel';
|
||||
import { hasImpairedResourceSource } from '@/utils/resourceSourceHealth';
|
||||
|
||||
export type ResourceDetailDrawerTrueNASRowTone = DetailValueTone;
|
||||
@@ -399,9 +403,13 @@ const diskTemperature = (disk: ResourcePhysicalDiskMeta) =>
|
||||
});
|
||||
|
||||
// A retained reading, such as a silent host agent's on a merged disk row, is
|
||||
// not current: it takes no heat tone and its title says why.
|
||||
// not current: it takes no heat tone and its title says why. A current reading
|
||||
// is hot from its disk type's alert trigger, the threshold the TrueNAS storage
|
||||
// table and the disk's temperature alert judge it by. Without a resolver, heat
|
||||
// is not judged.
|
||||
const diskTemperatureRow = (
|
||||
disk: ResourcePhysicalDiskMeta,
|
||||
resolveDiskTemperatureThresholds?: TrueNASDiskTemperatureThresholdResolver,
|
||||
): ResourceDetailDrawerTrueNASRow | null => {
|
||||
const reading = diskTemperature(disk);
|
||||
if (!reading) return null;
|
||||
@@ -411,9 +419,12 @@ const diskTemperatureRow = (
|
||||
tone: 'muted',
|
||||
});
|
||||
}
|
||||
return row('Temperature', reading.label, {
|
||||
tone: (disk.temperature ?? 0) >= 55 ? 'warning' : 'default',
|
||||
});
|
||||
const thresholds = resolveDiskTemperatureThresholds?.(disk.diskType ?? '') ?? null;
|
||||
const hot = isPhysicalDiskRunningHot(
|
||||
{ temperature: disk.temperature ?? 0, collection: disk.collection },
|
||||
thresholds,
|
||||
);
|
||||
return row('Temperature', reading.label, { tone: hot ? 'warning' : 'default' });
|
||||
};
|
||||
|
||||
const currentDiskTemperatureLabel = (disk: ResourcePhysicalDiskMeta): string | null => {
|
||||
@@ -424,6 +435,7 @@ const currentDiskTemperatureLabel = (disk: ResourcePhysicalDiskMeta): string | n
|
||||
const buildTrueNASDiskSections = (
|
||||
resource: Resource,
|
||||
disk: ResourcePhysicalDiskMeta,
|
||||
resolveDiskTemperatureThresholds?: TrueNASDiskTemperatureThresholdResolver,
|
||||
): ResourceDetailDrawerTrueNASSection[] => {
|
||||
const riskReasons = (disk.risk?.reasons ?? [])
|
||||
.map((reason) => asString(reason.summary))
|
||||
@@ -439,7 +451,7 @@ const buildTrueNASDiskSections = (
|
||||
|
||||
const healthRows = compactRows([
|
||||
row('Health', normalizeDelimitedLabel(disk.health), { tone: diskStateTone(disk) }),
|
||||
diskTemperatureRow(disk),
|
||||
diskTemperatureRow(disk, resolveDiskTemperatureThresholds),
|
||||
row(
|
||||
'Wearout',
|
||||
disk.wearout === undefined || disk.wearout < 0 ? null : formatPercent(disk.wearout),
|
||||
@@ -746,6 +758,7 @@ const buildTrueNASShareSections = (
|
||||
|
||||
export const buildTrueNASDetailSections = (
|
||||
resource: Resource,
|
||||
resolveDiskTemperatureThresholds?: TrueNASDiskTemperatureThresholdResolver,
|
||||
): ResourceDetailDrawerTrueNASSection[] => {
|
||||
if (resource.truenas?.share) return buildTrueNASShareSections(resource.truenas.share);
|
||||
if (resource.truenas?.vm) return buildTrueNASVMSections(resource, resource.truenas.vm);
|
||||
@@ -754,7 +767,11 @@ export const buildTrueNASDetailSections = (
|
||||
return buildTrueNASStorageSections(resource, resource.storage);
|
||||
}
|
||||
if (isTrueNASScopedResource(resource) && resource.physicalDisk) {
|
||||
return buildTrueNASDiskSections(resource, resource.physicalDisk);
|
||||
return buildTrueNASDiskSections(
|
||||
resource,
|
||||
resource.physicalDisk,
|
||||
resolveDiskTemperatureThresholds,
|
||||
);
|
||||
}
|
||||
if (isTrueNASScopedResource(resource) && resource.truenas) {
|
||||
return buildTrueNASSystemSections(resource, resource.truenas);
|
||||
|
||||
+5
-1
@@ -23,6 +23,7 @@ import {
|
||||
} from '@/utils/resourceIdentity';
|
||||
import { isPulseAgentPlatformResource } from '@/utils/agentResources';
|
||||
import { areSystemSettingsLoaded, shouldHideDockerUpdateActions } from '@/stores/systemSettings';
|
||||
import { useAlertsActivation } from '@/stores/alertsActivation';
|
||||
import {
|
||||
getResourcePolicyBadges,
|
||||
getResourcePolicyGovernedSummary,
|
||||
@@ -381,7 +382,10 @@ export const useResourceDetailDrawerDerivedState = (
|
||||
const vmwareDetailsSummary = createMemo(() =>
|
||||
buildVMwareDetailsSummary(resource().type, vmwareData()),
|
||||
);
|
||||
const trueNASDetailSections = createMemo(() => buildTrueNASDetailSections(resource()));
|
||||
const { getDiskTemperatureThresholds } = useAlertsActivation();
|
||||
const trueNASDetailSections = createMemo(() =>
|
||||
buildTrueNASDetailSections(resource(), getDiskTemperatureThresholds),
|
||||
);
|
||||
const hasTrueNASDetails = createMemo(() => trueNASDetailSections().length > 0);
|
||||
const trueNASDetailsSummary = createMemo(() => buildTrueNASDetailsSummary(resource()));
|
||||
const kubernetesDetailSections = createMemo(() => buildKubernetesDetailSections(resource()));
|
||||
|
||||
@@ -5,9 +5,12 @@ import { TableCell, TableRow, type TableRowProps } from './Table';
|
||||
export const INLINE_DETAIL_TABLE_CELL_CLASS = 'p-0 border-b border-border bg-surface-alt';
|
||||
// overflow-x-clip keeps over-wide content from painting outside the row's
|
||||
// border below the lg breakpoint (issue #1622) without creating a scroll
|
||||
// container the way overflow-hidden would.
|
||||
// container the way overflow-hidden would. Below lg the sticky content is as
|
||||
// wide as the table's visible area: 100cqi resolves against the table scroll
|
||||
// shell (a size container), so a guess at the page chrome cannot leave an
|
||||
// empty strip beside the drawer or push it past the visible edge.
|
||||
export const INLINE_DETAIL_TABLE_CONTENT_CLASS =
|
||||
'sticky left-0 min-w-0 max-w-[calc(100vw-3.5rem)] whitespace-normal overflow-x-clip px-2 py-3 sm:px-4 sm:py-4 lg:static lg:max-w-none lg:overflow-x-visible';
|
||||
'sticky left-0 min-w-0 max-w-[100cqi] whitespace-normal overflow-x-clip px-2 py-3 sm:px-4 sm:py-4 lg:static lg:max-w-none lg:overflow-x-visible';
|
||||
|
||||
export interface InlineDetailTableRowProps extends TableRowProps {
|
||||
cellId?: string;
|
||||
|
||||
@@ -8757,7 +8757,10 @@ describe('shared primitive guardrails', () => {
|
||||
expect(summaryRowActionButtonSource).toContain('h-9 w-9');
|
||||
expect(searchFieldSource).toContain('min-h-11');
|
||||
expect(filterToolbarSource).toContain('min-h-11');
|
||||
expect(inlineDetailTableRowSource).toContain('max-w-[calc(100vw-3.5rem)]');
|
||||
// Phone and tablet drawers fill the table's visible width (the scroll
|
||||
// shell is a size container), not a viewport guess at the page chrome.
|
||||
expect(inlineDetailTableRowSource).toContain('max-w-[100cqi]');
|
||||
expect(inlineDetailTableRowSource).not.toContain('100vw-3.5rem');
|
||||
});
|
||||
|
||||
it('keeps shared settings panels dense without forking their desktop framing', () => {
|
||||
|
||||
@@ -39,7 +39,7 @@ describe('InlineDetailTableRow', () => {
|
||||
expect(detail.parentElement).toHaveClass('left-0');
|
||||
expect(detail.parentElement).toHaveClass('min-w-0');
|
||||
expect(detail.parentElement).toHaveClass('whitespace-normal');
|
||||
expect(detail.parentElement).toHaveClass('max-w-[calc(100vw-3.5rem)]');
|
||||
expect(detail.parentElement).toHaveClass('max-w-[100cqi]');
|
||||
expect(detail.parentElement).toHaveClass('lg:static');
|
||||
expect(detail.parentElement).toHaveClass('lg:max-w-none');
|
||||
});
|
||||
|
||||
+49
-2
@@ -56,7 +56,8 @@ describe('readAlertsConfigurationSnapshot — absent optional sections', () => {
|
||||
expect(snapshot.trueNASDefaults.cpu).toBe(80);
|
||||
expect(snapshot.trueNASDefaults.usage).toBe(85);
|
||||
expect(snapshot.trueNASDefaults.diskRead).toBe(-1);
|
||||
expect(snapshot.trueNASDiskDefaults.temperature).toBe(55);
|
||||
// Unset: each TrueNAS disk follows Disk temperature by type.
|
||||
expect(snapshot.trueNASDiskDefaults.temperature).toBeUndefined();
|
||||
expect(snapshot.vmwareDefaults.cpu).toBe(80);
|
||||
expect(snapshot.vmwareDefaults.usage).toBe(85);
|
||||
});
|
||||
@@ -232,7 +233,8 @@ describe('readAlertsConfigurationSnapshot — factory fallback (undefined-trigge
|
||||
expect(snapshot.trueNASDefaults.diskWrite).toBe(-1);
|
||||
expect(snapshot.trueNASDefaults.networkIn).toBe(-1);
|
||||
expect(snapshot.trueNASDefaults.networkOut).toBe(-1);
|
||||
expect(snapshot.trueNASDiskDefaults.temperature).toBe(55);
|
||||
// No usable trigger leaves TrueNAS disks on Disk temperature by type.
|
||||
expect(snapshot.trueNASDiskDefaults.temperature).toBeUndefined();
|
||||
});
|
||||
|
||||
it('falls back to factory defaults for vmwareDefaults and agentDefaults', () => {
|
||||
@@ -1526,3 +1528,48 @@ describe('dockerDefaults.updateAlertDelayHours', () => {
|
||||
expect(alertConfig?.dockerDefaults?.updateAlertDelayHours).toBe(24);
|
||||
});
|
||||
});
|
||||
|
||||
describe('TrueNAS disk temperature round trip', () => {
|
||||
const buildTrueNASDiskPayload = (temperature: number | undefined) => {
|
||||
const snapshot = createDefaultAlertsConfigurationSnapshot();
|
||||
snapshot.trueNASDiskDefaults = { temperature };
|
||||
return buildAlertsConfigurationPayload({
|
||||
snapshot,
|
||||
rawOverridesConfig: {},
|
||||
alertsActivationState: null,
|
||||
alertsActivationConfig: null,
|
||||
}).alertConfig;
|
||||
};
|
||||
|
||||
it('omits an unset TrueNAS disk temperature so disks follow Disk temperature by type', () => {
|
||||
const alertConfig = buildTrueNASDiskPayload(undefined);
|
||||
expect(alertConfig?.truenasDiskDefaults).toEqual({});
|
||||
expect(alertConfig?.truenasDiskTemperatureByType).toBe(true);
|
||||
});
|
||||
|
||||
it('writes Off as a zero trigger, which the backend keeps as off', () => {
|
||||
expect(buildTrueNASDiskPayload(-1)?.truenasDiskDefaults).toEqual({
|
||||
temperature: { trigger: 0, clear: 0 },
|
||||
});
|
||||
});
|
||||
|
||||
it('writes a TrueNAS-wide value, 55 included, as a trigger', () => {
|
||||
expect(buildTrueNASDiskPayload(55)?.truenasDiskDefaults).toEqual({
|
||||
temperature: { trigger: 55, clear: 50 },
|
||||
});
|
||||
});
|
||||
|
||||
it('reads an absent temperature as unset and a zero trigger as off', () => {
|
||||
const base = { overrides: {} } as unknown as AlertConfig;
|
||||
expect(
|
||||
readAlertsConfigurationSnapshot({ ...base, truenasDiskDefaults: {} }).trueNASDiskDefaults
|
||||
.temperature,
|
||||
).toBeUndefined();
|
||||
expect(
|
||||
readAlertsConfigurationSnapshot({
|
||||
...base,
|
||||
truenasDiskDefaults: { temperature: { trigger: 0, clear: 0 } },
|
||||
}).trueNASDiskDefaults.temperature,
|
||||
).toBe(0);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -185,6 +185,15 @@ const createHysteresisThreshold = (trigger: number | undefined, clearMargin = 5)
|
||||
};
|
||||
};
|
||||
|
||||
// Unset leaves TrueNAS disks on Disk temperature by type, so it is omitted
|
||||
// rather than written as a trigger. Off is written as 0: the backend reads a
|
||||
// negative trigger as unset.
|
||||
const buildTrueNASDiskDefaultsConfig = (temperature: number | undefined) => {
|
||||
if (temperature === undefined) return {};
|
||||
if (temperature <= 0) return { temperature: { trigger: 0, clear: 0 } };
|
||||
return { temperature: createHysteresisThreshold(temperature) };
|
||||
};
|
||||
|
||||
const normalizeGap = (value: unknown, fallback: number) => {
|
||||
const numeric = Number(value);
|
||||
if (!Number.isFinite(numeric)) {
|
||||
@@ -413,10 +422,10 @@ export function readAlertsConfigurationSnapshot(config: AlertConfig): AlertsConf
|
||||
}
|
||||
|
||||
if (config.truenasDiskDefaults) {
|
||||
// Unset means each TrueNAS disk follows Disk temperature by type.
|
||||
const temperature = config.truenasDiskDefaults.temperature;
|
||||
snapshot.trueNASDiskDefaults = {
|
||||
temperature:
|
||||
getTriggerValue(config.truenasDiskDefaults.temperature) ??
|
||||
FACTORY_TRUENAS_DISK_DEFAULTS.temperature,
|
||||
temperature: temperature === undefined ? undefined : getTriggerValue(temperature),
|
||||
};
|
||||
}
|
||||
|
||||
@@ -847,9 +856,8 @@ export function buildAlertsConfigurationPayload({
|
||||
networkIn: createHysteresisThreshold(snapshot.trueNASDefaults.networkIn),
|
||||
networkOut: createHysteresisThreshold(snapshot.trueNASDefaults.networkOut),
|
||||
},
|
||||
truenasDiskDefaults: {
|
||||
temperature: createHysteresisThreshold(snapshot.trueNASDiskDefaults.temperature),
|
||||
},
|
||||
truenasDiskDefaults: buildTrueNASDiskDefaultsConfig(snapshot.trueNASDiskDefaults.temperature),
|
||||
truenasDiskTemperatureByType: true,
|
||||
vmwareDefaults: {
|
||||
cpu: createHysteresisThreshold(snapshot.vmwareDefaults.cpu),
|
||||
memory: createHysteresisThreshold(snapshot.vmwareDefaults.memory),
|
||||
|
||||
+44
@@ -0,0 +1,44 @@
|
||||
import { describe, expect, it } from 'vitest';
|
||||
import {
|
||||
getDiskTemperatureByTypeItems,
|
||||
resolveDiskTemperatureTriggerForType,
|
||||
resolveTrueNASDiskTemperatureDefault,
|
||||
} from '../trueNASDiskTemperature';
|
||||
|
||||
const factoryPolicy = {
|
||||
agentDiskTemperature: 55,
|
||||
diskTempByType: { nvme: 70, sas: 65, sata: 55 },
|
||||
};
|
||||
|
||||
describe('TrueNAS disk temperature in the thresholds editor', () => {
|
||||
it('gives each disk its type trigger, and unknown types the agent default', () => {
|
||||
expect(resolveDiskTemperatureTriggerForType(factoryPolicy, 'NVMe')).toBe(70);
|
||||
expect(resolveDiskTemperatureTriggerForType(factoryPolicy, 'sas')).toBe(65);
|
||||
expect(resolveDiskTemperatureTriggerForType(factoryPolicy, 'sata')).toBe(55);
|
||||
expect(resolveDiskTemperatureTriggerForType(factoryPolicy, 'hdd')).toBe(55);
|
||||
expect(
|
||||
resolveDiskTemperatureTriggerForType({ ...factoryPolicy, agentDiskTemperature: 60 }, ''),
|
||||
).toBe(60);
|
||||
});
|
||||
|
||||
it('switches every type off with the agent Disk Temp default', () => {
|
||||
const off = { ...factoryPolicy, agentDiskTemperature: -1 };
|
||||
expect(resolveDiskTemperatureTriggerForType(off, 'nvme')).toBe(0);
|
||||
expect(getDiskTemperatureByTypeItems(off)).toEqual([]);
|
||||
});
|
||||
|
||||
it('lets a TrueNAS-wide value, Off included, replace the type trigger', () => {
|
||||
expect(resolveTrueNASDiskTemperatureDefault(undefined, factoryPolicy, 'nvme')).toBe(70);
|
||||
expect(resolveTrueNASDiskTemperatureDefault(62, factoryPolicy, 'nvme')).toBe(62);
|
||||
expect(resolveTrueNASDiskTemperatureDefault(-1, factoryPolicy, 'nvme')).toBe(-1);
|
||||
});
|
||||
|
||||
it('names the per-type triggers', () => {
|
||||
expect(
|
||||
getDiskTemperatureByTypeItems({
|
||||
...factoryPolicy,
|
||||
diskTempByType: { nvme: 75, sas: 65, sata: 55 },
|
||||
}),
|
||||
).toEqual(['NVMe 75°C', 'SAS 65°C', 'SATA 55°C']);
|
||||
});
|
||||
});
|
||||
@@ -9,6 +9,7 @@ import type { Resource } from '@/types/resource';
|
||||
import type { Resource as TableResource } from '../tableTypes';
|
||||
import type { Override } from '../types';
|
||||
import type { ThresholdsDataInputs } from '../thresholdsResourceModel';
|
||||
import { resolveTrueNASDiskTemperatureDefault } from '../trueNASDiskTemperature';
|
||||
import {
|
||||
createOverridesMap,
|
||||
findOverrideByCandidates,
|
||||
@@ -307,7 +308,18 @@ export function useThresholdsPlatformData(inputs: ThresholdsDataInputs) {
|
||||
resource,
|
||||
type: 'truenasDisk',
|
||||
resourceType: 'TrueNAS Disk',
|
||||
defaults: trueNASDiskDefaults(),
|
||||
// Each disk inherits the trigger saving would apply to it: the
|
||||
// TrueNAS-wide value, else its type's disk temperature trigger.
|
||||
defaults: {
|
||||
temperature: resolveTrueNASDiskTemperatureDefault(
|
||||
trueNASDiskDefaults().temperature,
|
||||
{
|
||||
agentDiskTemperature: props.agentDefaults?.diskTemperature,
|
||||
diskTempByType: props.diskTempByType ?? {},
|
||||
},
|
||||
resource.physicalDisk?.diskType,
|
||||
),
|
||||
},
|
||||
overridesMap,
|
||||
node: trueNASSystemName(resource),
|
||||
instance: readString(resource.physicalDisk?.diskType) || 'TrueNAS',
|
||||
|
||||
@@ -0,0 +1,51 @@
|
||||
import { formatMetricValue } from './helpers';
|
||||
import { FACTORY_AGENT_DEFAULTS } from '@/utils/alertThresholdDefaults';
|
||||
|
||||
// The thresholds editor's view of the TrueNAS disk temperature trigger. It
|
||||
// mirrors the backend (trueNASDiskTemperatureDefaultNoLock): a TrueNAS-wide
|
||||
// value the user saved applies to every TrueNAS disk; unset, each disk follows
|
||||
// Disk temperature by type, which the agent Disk Temp default switches off for
|
||||
// every type when it is off. It reads the unsaved editor state, so a row shows
|
||||
// what saving would apply.
|
||||
|
||||
export const DISK_TEMPERATURE_TYPE_FIELDS: readonly { key: string; label: string }[] = [
|
||||
{ key: 'nvme', label: 'NVMe' },
|
||||
{ key: 'sas', label: 'SAS' },
|
||||
{ key: 'sata', label: 'SATA' },
|
||||
];
|
||||
|
||||
export interface DiskTemperaturePolicyState {
|
||||
/** The agent Disk Temp default trigger; zero or less is off. */
|
||||
agentDiskTemperature: number | undefined;
|
||||
diskTempByType: Record<string, number>;
|
||||
}
|
||||
|
||||
/** The disk temperature policy's trigger for a disk type; zero or less is off. */
|
||||
export function resolveDiskTemperatureTriggerForType(
|
||||
policy: DiskTemperaturePolicyState,
|
||||
diskType: string | null | undefined,
|
||||
): number {
|
||||
const base = policy.agentDiskTemperature ?? FACTORY_AGENT_DEFAULTS.diskTemperature;
|
||||
if (base <= 0) return 0;
|
||||
const normalizedType = (diskType ?? '').trim().toLowerCase();
|
||||
const byType = normalizedType ? policy.diskTempByType[normalizedType] : undefined;
|
||||
return typeof byType === 'number' && byType > 0 ? byType : base;
|
||||
}
|
||||
|
||||
/** The trigger a TrueNAS disk without its own override inherits. */
|
||||
export function resolveTrueNASDiskTemperatureDefault(
|
||||
trueNASDiskTemperature: number | undefined,
|
||||
policy: DiskTemperaturePolicyState,
|
||||
diskType: string | null | undefined,
|
||||
): number {
|
||||
if (trueNASDiskTemperature !== undefined) return trueNASDiskTemperature;
|
||||
return resolveDiskTemperatureTriggerForType(policy, diskType);
|
||||
}
|
||||
|
||||
/** "NVMe 70°C", one per disk type, or none when the policy is off. */
|
||||
export function getDiskTemperatureByTypeItems(policy: DiskTemperaturePolicyState): string[] {
|
||||
return DISK_TEMPERATURE_TYPE_FIELDS.flatMap((field) => {
|
||||
const trigger = resolveDiskTemperatureTriggerForType(policy, field.key);
|
||||
return trigger > 0 ? [`${field.label} ${formatMetricValue('temperature', trigger)}`] : [];
|
||||
});
|
||||
}
|
||||
@@ -64,7 +64,7 @@ import {
|
||||
import type { EvidenceEnvelope } from '@/types/operationalTrust';
|
||||
import type { ActionDetailResponse } from '@/types/actionAudit';
|
||||
import { getAlertResourceIncidentAcknowledgedByLabel } from '@/utils/alertIncidentPresentation';
|
||||
import { formatRelativeTime } from '@/utils/format';
|
||||
import { formatAbsoluteTime, formatRelativeTime, formatTimeUntil } from '@/utils/format';
|
||||
import { useRelativeTimeNow } from '@/utils/relativeTimeClock';
|
||||
import { copyToClipboard } from '@/utils/clipboard';
|
||||
import type { PatrolAutonomyLevel } from '@/api/patrol';
|
||||
@@ -153,14 +153,31 @@ function getAttentionSuppressionEndPresentation(reviewed: boolean) {
|
||||
durationLabel: 'End suppression after',
|
||||
actionLabel: 'End suppression',
|
||||
successLabel: 'Suppression ended. It stays reviewed',
|
||||
countdownLabel: (remaining: string) => `Suppression ends ${remaining}. It stays reviewed.`,
|
||||
}
|
||||
: {
|
||||
durationLabel: 'Return it to active attention after',
|
||||
actionLabel: 'Return to active attention',
|
||||
successLabel: 'Returned to decision inbox',
|
||||
countdownLabel: (remaining: string) => `Returns to active attention ${remaining}.`,
|
||||
};
|
||||
}
|
||||
|
||||
// The suppression line counts down to the expiry in the same words as the
|
||||
// early-end button, and says so once the expiry has passed.
|
||||
export function getAttentionSuppressionExpiryLabel(
|
||||
expiresAt: string,
|
||||
reviewed: boolean,
|
||||
now: number,
|
||||
): string {
|
||||
const expiry = Date.parse(expiresAt);
|
||||
if (!Number.isFinite(expiry)) return '';
|
||||
if (expiry <= now) return 'Suppression has ended.';
|
||||
return getAttentionSuppressionEndPresentation(reviewed).countdownLabel(
|
||||
formatTimeUntil(expiry, { compact: true, now }),
|
||||
);
|
||||
}
|
||||
|
||||
export function PatrolAttentionWorkbench(
|
||||
props: {
|
||||
autonomyLevel?: PatrolAutonomyLevel;
|
||||
@@ -1285,8 +1302,9 @@ function AttentionLifecycleControls(props: {
|
||||
onSuppress: (itemId: string, reason: string, expiresAt: string) => Promise<void>;
|
||||
onUnsuppress: (itemId: string, reviewed: boolean) => Promise<void>;
|
||||
}) {
|
||||
// The acknowledgement time never changes while the detail stays open, so its
|
||||
// age reads the shared clock.
|
||||
// The acknowledgement and suppression expiry times never change while the
|
||||
// detail stays open, so the acknowledgement age and the countdown to the
|
||||
// expiry read the shared clock.
|
||||
const now = useRelativeTimeNow();
|
||||
const [showSuppression, setShowSuppression] = createSignal(false);
|
||||
const [reason, setReason] = createSignal('');
|
||||
@@ -1329,10 +1347,13 @@ function AttentionLifecycleControls(props: {
|
||||
</p>
|
||||
<Show when={suppression().expiresAt}>
|
||||
{(expiresAt) => (
|
||||
<p>
|
||||
Returns to active attention {formatRelativeTime(expiresAt(), { compact: true })}
|
||||
.
|
||||
</p>
|
||||
<Show when={getAttentionSuppressionExpiryLabel(expiresAt(), reviewed(), now())}>
|
||||
{(label) => (
|
||||
<p title={`Suppressed until ${formatAbsoluteTime(Date.parse(expiresAt()))}`}>
|
||||
{label()}
|
||||
</p>
|
||||
)}
|
||||
</Show>
|
||||
)}
|
||||
</Show>
|
||||
</div>
|
||||
|
||||
@@ -91,6 +91,7 @@ vi.mock('@/features/actions/ActionReviewDialog', async () => {
|
||||
});
|
||||
|
||||
import {
|
||||
getAttentionSuppressionExpiryLabel,
|
||||
getDistinctPatrolImpact,
|
||||
getPatrolDecisionDisplayTitle,
|
||||
PatrolAttentionWorkbench,
|
||||
@@ -519,6 +520,74 @@ describe('PatrolAttentionWorkbench', () => {
|
||||
}
|
||||
});
|
||||
|
||||
it('counts down to when a suppressed issue returns to active attention', async () => {
|
||||
vi.useFakeTimers({
|
||||
toFake: ['Date', 'setInterval', 'clearInterval'],
|
||||
now: Date.parse('2026-07-19T08:10:00Z'),
|
||||
});
|
||||
const suppressed = item({ state: 'suppressed' });
|
||||
const suppressedDetail = detail(suppressed);
|
||||
// A 24-hour suppression set just now.
|
||||
suppressedDetail.operationalRecord.suppression = {
|
||||
at: '2026-07-19T08:10:00Z',
|
||||
by: 'operator',
|
||||
reason: 'Planned storage maintenance',
|
||||
expiresAt: '2026-07-20T08:10:00Z',
|
||||
};
|
||||
const handled = summary({ suppressedCount: 1 });
|
||||
apiMocks.getList.mockImplementation((scope: string) =>
|
||||
Promise.resolve(listResponse(scope === 'all' ? [suppressed] : [], handled)),
|
||||
);
|
||||
apiMocks.getDetail.mockResolvedValue(suppressedDetail);
|
||||
|
||||
try {
|
||||
renderWorkbench();
|
||||
fireEvent.click(await screen.findByRole('button', { name: 'Reviewed and suppressed (1)' }));
|
||||
fireEvent.click(
|
||||
await screen.findByRole('button', { name: 'Open Database VM · Disk pressure' }),
|
||||
);
|
||||
const detailRegion = await screen.findByRole('complementary', {
|
||||
name: 'Database VM · Disk pressure',
|
||||
});
|
||||
const expiry = await within(detailRegion).findByText(/Returns to active attention/);
|
||||
expect(expiry).toHaveTextContent('Returns to active attention in 1d.');
|
||||
expect(expiry).toHaveAttribute('title', expect.stringMatching(/^Suppressed until /));
|
||||
|
||||
// The detail is not re-read: its suppression never changes and only the
|
||||
// clock moves.
|
||||
vi.advanceTimersByTime(4 * 60 * 60 * 1000);
|
||||
expect(expiry).toHaveTextContent('Returns to active attention in 20h.');
|
||||
|
||||
vi.setSystemTime(Date.parse('2026-07-20T07:30:00Z'));
|
||||
vi.advanceTimersByTime(RELATIVE_TIME_TICK_MS);
|
||||
expect(expiry).toHaveTextContent('Returns to active attention in 40m.');
|
||||
|
||||
vi.setSystemTime(Date.parse('2026-07-20T08:11:00Z'));
|
||||
vi.advanceTimersByTime(RELATIVE_TIME_TICK_MS);
|
||||
expect(expiry).toHaveTextContent('Suppression has ended.');
|
||||
expect(detailRegion).not.toHaveTextContent('just now');
|
||||
expect(apiMocks.getDetail).toHaveBeenCalledTimes(1);
|
||||
} finally {
|
||||
vi.useRealTimers();
|
||||
}
|
||||
});
|
||||
|
||||
it('says a reviewed issue stays reviewed when its suppression ends', () => {
|
||||
const now = Date.parse('2026-07-19T08:10:00Z');
|
||||
// Expiry restores Acknowledged for a reviewed alert, which is not active
|
||||
// attention, so only an unreviewed issue "returns to active attention".
|
||||
expect(getAttentionSuppressionExpiryLabel('2026-07-20T08:10:00Z', true, now)).toBe(
|
||||
'Suppression ends in 1d. It stays reviewed.',
|
||||
);
|
||||
expect(getAttentionSuppressionExpiryLabel('2026-07-19T11:10:00Z', false, now)).toBe(
|
||||
'Returns to active attention in 3h.',
|
||||
);
|
||||
expect(getAttentionSuppressionExpiryLabel('2026-07-19T08:00:00Z', true, now)).toBe(
|
||||
'Suppression has ended.',
|
||||
);
|
||||
expect(getAttentionSuppressionExpiryLabel('not a date', false, now)).toBe('');
|
||||
});
|
||||
|
||||
it('keeps the detail header and queue controls pinned while a long decision scrolls', async () => {
|
||||
const active = item();
|
||||
apiMocks.getList.mockResolvedValue(
|
||||
|
||||
@@ -147,6 +147,9 @@ export interface AlertConfig {
|
||||
kubernetesDefaults?: AlertThresholds;
|
||||
truenasDefaults?: AlertThresholds;
|
||||
truenasDiskDefaults?: AlertThresholds;
|
||||
// Set on every config written since an unset truenasDiskDefaults
|
||||
// temperature means each TrueNAS disk follows Disk temperature by type.
|
||||
truenasDiskTemperatureByType?: boolean;
|
||||
vmwareDefaults?: AlertThresholds;
|
||||
snapshotDefaults?: SnapshotAlertConfig;
|
||||
backupDefaults?: BackupAlertConfig;
|
||||
|
||||
@@ -96,7 +96,8 @@ describe('alertThresholdDefaults', () => {
|
||||
|
||||
describe('FACTORY_TRUENAS_DISK_DEFAULTS', () => {
|
||||
it('only carries a disk temperature trigger', () => {
|
||||
expect(FACTORY_TRUENAS_DISK_DEFAULTS).toEqual({ temperature: 55 });
|
||||
// Unset: each TrueNAS disk follows Disk temperature by type.
|
||||
expect(FACTORY_TRUENAS_DISK_DEFAULTS).toEqual({ temperature: undefined });
|
||||
expect(Object.keys(FACTORY_TRUENAS_DISK_DEFAULTS)).toEqual(['temperature']);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -13,6 +13,7 @@ import {
|
||||
formatUptime,
|
||||
formatAbsoluteTime,
|
||||
formatRelativeTime,
|
||||
formatTimeUntil,
|
||||
getBackupInfo,
|
||||
} from '@/utils/format';
|
||||
|
||||
@@ -263,6 +264,67 @@ describe('formatRelativeTime', () => {
|
||||
});
|
||||
});
|
||||
|
||||
describe('formatTimeUntil', () => {
|
||||
const now = Date.parse('2024-03-15T12:00:00Z');
|
||||
const minute = 60 * 1000;
|
||||
const hour = 60 * minute;
|
||||
const day = 24 * hour;
|
||||
|
||||
it('returns empty text for falsy or unparseable input', () => {
|
||||
expect(formatTimeUntil(undefined, { now })).toBe('');
|
||||
expect(formatTimeUntil(0, { now, emptyText: '-' })).toBe('-');
|
||||
expect(formatTimeUntil('not a date', { now, emptyText: '-' })).toBe('-');
|
||||
});
|
||||
|
||||
it('counts down to a future time instead of reading it as just now', () => {
|
||||
expect(formatTimeUntil(now + 30 * 1000, { now })).toBe('in under a minute');
|
||||
expect(formatTimeUntil(now + 5 * minute, { now, compact: true })).toBe('in 5m');
|
||||
expect(formatTimeUntil(now + 3 * hour, { now, compact: true })).toBe('in 3h');
|
||||
expect(formatTimeUntil(now + 6 * day, { now, compact: true })).toBe('in 6d');
|
||||
expect(formatTimeUntil(now + 45 * day, { now, compact: true })).toBe('in 45d');
|
||||
});
|
||||
|
||||
it('spells units out in the long form', () => {
|
||||
expect(formatTimeUntil(now + minute, { now })).toBe('in 1 min');
|
||||
expect(formatTimeUntil(now + 5 * minute, { now })).toBe('in 5 mins');
|
||||
expect(formatTimeUntil(now + hour, { now })).toBe('in 1 hour');
|
||||
expect(formatTimeUntil(now + 5 * hour, { now })).toBe('in 5 hours');
|
||||
expect(formatTimeUntil(now + day, { now })).toBe('in 1 day');
|
||||
expect(formatTimeUntil(now + 7 * day, { now })).toBe('in 7 days');
|
||||
expect(formatTimeUntil(now + 60 * day, { now })).toBe('in 2 months');
|
||||
expect(formatTimeUntil(now + 365 * day, { now })).toBe('in 1 year');
|
||||
});
|
||||
|
||||
it('rounds to the nearest unit, so a duration just set reads as chosen', () => {
|
||||
// A 24-hour, 1-hour and 7-day suppression set a few seconds ago.
|
||||
expect(formatTimeUntil(now + day - 5000, { now, compact: true })).toBe('in 1d');
|
||||
expect(formatTimeUntil(now + hour - 5000, { now, compact: true })).toBe('in 1h');
|
||||
expect(formatTimeUntil(now + 7 * day - 5000, { now, compact: true })).toBe('in 7d');
|
||||
expect(formatTimeUntil(now + 2 * hour + 10 * minute, { now, compact: true })).toBe('in 2h');
|
||||
expect(formatTimeUntil(now + 2 * hour + 50 * minute, { now, compact: true })).toBe('in 3h');
|
||||
// Months round from the remaining time itself, not from rounded days.
|
||||
expect(formatTimeUntil(now + 44 * day + 12 * hour, { now })).toBe('in 1 month');
|
||||
expect(formatTimeUntil(now + 45 * day + hour, { now })).toBe('in 2 months');
|
||||
});
|
||||
|
||||
it('reads as due once the time has arrived or passed', () => {
|
||||
expect(formatTimeUntil(now, { now })).toBe('now');
|
||||
expect(formatTimeUntil(now - hour, { now, compact: true })).toBe('now');
|
||||
expect(formatTimeUntil(now - hour, { now, dueText: 'overdue' })).toBe('overdue');
|
||||
});
|
||||
|
||||
it('accepts ISO strings and Date objects and defaults now to the current time', () => {
|
||||
vi.useFakeTimers();
|
||||
vi.setSystemTime(now);
|
||||
try {
|
||||
expect(formatTimeUntil('2024-03-15T15:00:00Z', { compact: true })).toBe('in 3h');
|
||||
expect(formatTimeUntil(new Date(now + 10 * minute), { compact: true })).toBe('in 10m');
|
||||
} finally {
|
||||
vi.useRealTimers();
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe('getBackupInfo', () => {
|
||||
beforeEach(() => {
|
||||
vi.useFakeTimers();
|
||||
|
||||
@@ -507,7 +507,7 @@ describe('metricThresholds', () => {
|
||||
usage: 85,
|
||||
temperature: 80,
|
||||
});
|
||||
expect(FACTORY_TRUENAS_DISK_DEFAULTS).toEqual({ temperature: 55 });
|
||||
expect(FACTORY_TRUENAS_DISK_DEFAULTS).toEqual({ temperature: undefined });
|
||||
expect(FACTORY_VMWARE_DEFAULTS).toEqual({
|
||||
cpu: 80,
|
||||
memory: 85,
|
||||
|
||||
@@ -5,7 +5,7 @@ import type { MetadataBadgeTone } from '@/components/shared/MetadataBadge';
|
||||
import { isLivePendingApproval } from '@/utils/approvalState';
|
||||
import { getPatrolProviderSettingsAction } from '@/utils/patrolRuntimeActions';
|
||||
import { formatIdentifierLabel, titleCaseDelimitedLabel } from '@/utils/textPresentation';
|
||||
import { formatBytes } from '@/utils/format';
|
||||
import { formatAbsoluteTime, formatBytes, formatTimeUntil } from '@/utils/format';
|
||||
|
||||
const DEFAULT_BADGE_CLASSES = 'border-border bg-surface-alt text-muted';
|
||||
const DEFAULT_LOOP_STATE_CLASSES = 'border-border bg-surface-alt text-muted';
|
||||
@@ -275,6 +275,11 @@ export interface FindingRecencyPresentation {
|
||||
timestamp: string;
|
||||
}
|
||||
|
||||
export interface FindingCountdownPresentation {
|
||||
label: string;
|
||||
title: string;
|
||||
}
|
||||
|
||||
export interface PatrolFindingClassification {
|
||||
kind: 'runtime' | 'infrastructure';
|
||||
label: string;
|
||||
@@ -913,6 +918,49 @@ export const getFindingRecencyPresentation = (
|
||||
};
|
||||
};
|
||||
|
||||
// A will_fix_later reminder and the end of a snooze are future times, so the
|
||||
// row counts down to them ("Reminding in 6d") from the `now` the panel passes
|
||||
// from the shared relative-time clock, and keeps the exact time on hover. A
|
||||
// reminder reads overdue when `overdue` says the panel's overdue list holds it,
|
||||
// the same list the Overdue commitments chip and filter count, so the three
|
||||
// agree; read between the deadline and the clock tick that adds it there, it
|
||||
// says "Reminder due". It stays overdue until the hourly reminder sweep brings
|
||||
// the finding back, whether or not it still trips. A passed snooze has ended,
|
||||
// so neither keeps counting.
|
||||
export const getFindingReminderPresentation = (
|
||||
finding: Pick<UnifiedFinding, 'dismissedReason' | 'remindAt'>,
|
||||
now: number,
|
||||
overdue: boolean,
|
||||
): FindingCountdownPresentation | null => {
|
||||
if (finding.dismissedReason !== 'will_fix_later' || !finding.remindAt) return null;
|
||||
const due = Date.parse(finding.remindAt);
|
||||
if (!Number.isFinite(due)) return null;
|
||||
const at = formatAbsoluteTime(due);
|
||||
if (overdue) {
|
||||
return {
|
||||
label: 'Reminder overdue',
|
||||
title: `Reminder was due ${at}. Pulse brings this finding back on its next reminder check.`,
|
||||
};
|
||||
}
|
||||
return {
|
||||
label:
|
||||
due <= now ? 'Reminder due' : `Reminding ${formatTimeUntil(due, { compact: true, now })}`,
|
||||
title: `Reminder due ${at}. Pulse brings this finding back on its first reminder check after that.`,
|
||||
};
|
||||
};
|
||||
|
||||
export const getFindingSnoozePresentation = (
|
||||
finding: Pick<UnifiedFinding, 'status' | 'snoozedUntil'>,
|
||||
now: number,
|
||||
): FindingCountdownPresentation | null => {
|
||||
if (finding.status !== 'snoozed' || !finding.snoozedUntil) return null;
|
||||
const until = Date.parse(finding.snoozedUntil);
|
||||
if (!Number.isFinite(until)) return null;
|
||||
const title = `Snoozed until ${formatAbsoluteTime(until)}`;
|
||||
if (until <= now) return { label: 'snooze ended', title };
|
||||
return { label: `snoozed, returns ${formatTimeUntil(until, { compact: true, now })}`, title };
|
||||
};
|
||||
|
||||
export const getInvestigationStatusBadgeClasses = (status: InvestigationStatus): string =>
|
||||
INVESTIGATION_STATUS_CLASSES[status] || DEFAULT_BADGE_CLASSES;
|
||||
|
||||
|
||||
@@ -44,8 +44,12 @@ export const FACTORY_TRUENAS_DEFAULTS = {
|
||||
networkOut: -1,
|
||||
};
|
||||
|
||||
export const FACTORY_TRUENAS_DISK_DEFAULTS = {
|
||||
temperature: 55,
|
||||
// TrueNAS disk temperature has no flat factory value. Unset, each TrueNAS disk
|
||||
// follows Disk temperature by type, as the backend resolves it
|
||||
// (trueNASDiskTemperatureDefaultNoLock); a saved value applies to every
|
||||
// TrueNAS disk.
|
||||
export const FACTORY_TRUENAS_DISK_DEFAULTS: Record<string, number | undefined> = {
|
||||
temperature: undefined,
|
||||
};
|
||||
|
||||
export const FACTORY_VMWARE_DEFAULTS = {
|
||||
|
||||
@@ -169,9 +169,18 @@ export function formatAbsoluteTime(timestamp: number | undefined): string {
|
||||
return `${day} ${month} ${hours}:${minutes}`;
|
||||
}
|
||||
|
||||
function relativeTimeInputMs(timestamp: number | string | Date): number {
|
||||
if (typeof timestamp === 'number') return timestamp;
|
||||
if (typeof timestamp === 'string') return new Date(timestamp).getTime();
|
||||
return timestamp.getTime();
|
||||
}
|
||||
|
||||
/**
|
||||
* Format a timestamp as a human-readable relative time string.
|
||||
* ALL relative time formatting MUST use this function.
|
||||
* Format a timestamp as a human-readable relative age ("5m ago").
|
||||
* ALL relative ages MUST use this function. It is past-only: a time ahead of
|
||||
* `now` is clock skew on something already observed and reads as "just now".
|
||||
* A time meant to be in the future (an expiry, a reminder, a schedule) uses
|
||||
* formatTimeUntil instead.
|
||||
*
|
||||
* @param timestamp - Unix ms number, ISO date string, Date object, or undefined
|
||||
* @param options.compact - Use short format: "5m ago" instead of "5 mins ago"
|
||||
@@ -185,19 +194,51 @@ export function formatRelativeTime(
|
||||
): string {
|
||||
if (!timestamp) return options?.emptyText ?? '';
|
||||
|
||||
let ms: number;
|
||||
if (typeof timestamp === 'number') {
|
||||
ms = timestamp;
|
||||
} else if (typeof timestamp === 'string') {
|
||||
ms = new Date(timestamp).getTime();
|
||||
} else {
|
||||
ms = timestamp.getTime();
|
||||
}
|
||||
|
||||
const diffMs = (options?.now ?? Date.now()) - ms;
|
||||
const diffMs = (options?.now ?? Date.now()) - relativeTimeInputMs(timestamp);
|
||||
return formatTimeDiff(diffMs, options?.compact);
|
||||
}
|
||||
|
||||
/**
|
||||
* Format a future timestamp as a countdown: "in 3h" (compact) or "in 3 hours".
|
||||
* ALL relative future times (expiries, reminders, schedules) MUST use this
|
||||
* function, because formatRelativeTime reads any future time as "just now".
|
||||
*
|
||||
* Unlike an age, a countdown rounds to the nearest unit, so a 24-hour
|
||||
* suppression set a moment ago reads "in 1d" rather than "in 23h".
|
||||
*
|
||||
* @param timestamp - Unix ms number, ISO date string, Date object, or undefined
|
||||
* @param options.compact - Use short format: "in 5m" instead of "in 5 mins"
|
||||
* @param options.emptyText - Text for falsy or unparseable input (default: '')
|
||||
* @param options.dueText - Text once the time has arrived or passed (default: 'now')
|
||||
* @param options.now - The time to count down from (default: Date.now()). A
|
||||
* countdown that must keep moving passes the shared relative-time clock.
|
||||
*/
|
||||
export function formatTimeUntil(
|
||||
timestamp: number | string | Date | undefined,
|
||||
options?: { compact?: boolean; emptyText?: string; dueText?: string; now?: number },
|
||||
): string {
|
||||
if (!timestamp) return options?.emptyText ?? '';
|
||||
|
||||
const remainingMs = relativeTimeInputMs(timestamp) - (options?.now ?? Date.now());
|
||||
if (!Number.isFinite(remainingMs)) return options?.emptyText ?? '';
|
||||
if (remainingMs <= 0) return options?.dueText ?? 'now';
|
||||
if (remainingMs < 60_000) return 'in under a minute';
|
||||
|
||||
const compact = options?.compact;
|
||||
const inUnits = (value: number, short: string, singular: string, plural: string) =>
|
||||
compact ? `in ${value}${short}` : `in ${value} ${value === 1 ? singular : plural}`;
|
||||
|
||||
const minutes = Math.round(remainingMs / 60_000);
|
||||
if (minutes < 60) return inUnits(minutes, 'm', 'min', 'mins');
|
||||
const hours = Math.round(remainingMs / 3_600_000);
|
||||
if (hours < 24) return inUnits(hours, 'h', 'hour', 'hours');
|
||||
const days = Math.round(remainingMs / 86_400_000);
|
||||
if (compact || days < 30) return inUnits(days, 'd', 'day', 'days');
|
||||
const months = Math.round(remainingMs / (30 * 86_400_000));
|
||||
if (months < 12) return inUnits(months, '', 'month', 'months');
|
||||
return inUnits(Math.round(remainingMs / (365 * 86_400_000)), '', 'year', 'years');
|
||||
}
|
||||
|
||||
/**
|
||||
* Formats a time difference in milliseconds to a human-readable string.
|
||||
* Internal helper used by formatRelativeTime and formatBackupAge.
|
||||
|
||||
@@ -19,6 +19,8 @@ package alerts
|
||||
// 4. the per-resource override, through the identity-aware lookup for
|
||||
// the kind (guest stable keys, storage aliases, canonical registry
|
||||
// translation)
|
||||
// 5. a guest's pulse-relaxed tag, which lifts its CPU/memory/disk
|
||||
// triggers to a 95/92/95 floor
|
||||
|
||||
// alertPolicyQuery identifies one resource asking for its effective
|
||||
// alert policy.
|
||||
@@ -28,13 +30,20 @@ type alertPolicyQuery struct {
|
||||
TypeKey string
|
||||
ResourceID string
|
||||
// Guest carries the live guest model when the query is for a guest
|
||||
// kind: filter-scoped custom rules and the clustered override
|
||||
// identity need it. Nil is valid — filter rules then never match and
|
||||
// override lookup falls back to the raw resource ID.
|
||||
// kind: filter-scoped custom rules, the clustered override identity and
|
||||
// the pulse-relaxed tag need it. Nil is valid — filter rules then never
|
||||
// match, override lookup falls back to the raw resource ID, and no tag
|
||||
// relaxes the thresholds.
|
||||
Guest any
|
||||
// StorageAliases are the storage resource's alias IDs, honored by the
|
||||
// storage override lookup.
|
||||
StorageAliases []string
|
||||
// DiskType is a physical disk's type ("nvme", "sas", "sata"), which
|
||||
// picks its disk temperature threshold. DiskTypeUnknown marks a disk
|
||||
// whose type was never recorded, such as an alert persisted before it
|
||||
// carried one; it is judged against the lowest per-type trigger.
|
||||
DiskType string
|
||||
DiskTypeUnknown bool
|
||||
}
|
||||
|
||||
// EffectiveAlertPolicy is the resolved policy for one resource.
|
||||
@@ -47,7 +56,8 @@ type EffectiveAlertPolicy struct {
|
||||
// connectivity opt-outs are Thresholds.DisableConnectivity.
|
||||
OfflineDisabled bool
|
||||
// Thresholds is the folded threshold block: type defaults, then
|
||||
// custom rules, then the per-resource override.
|
||||
// custom rules, then the per-resource override, then a guest's
|
||||
// pulse-relaxed floor.
|
||||
Thresholds ThresholdConfig
|
||||
}
|
||||
|
||||
@@ -116,10 +126,19 @@ func (m *Manager) effectiveAlertPolicyNoLock(q alertPolicyQuery) EffectiveAlertP
|
||||
policy.AllDisabled, policy.OfflineDisabled = m.alertPolicyTypeSwitchesNoLock(q.TypeKey)
|
||||
|
||||
thresholds := m.defaultThresholdsForResourceType(q.TypeKey)
|
||||
if q.TypeKey == "truenas-disk" {
|
||||
thresholds.Temperature = m.trueNASDiskTemperatureDefaultNoLock(q.DiskType, q.DiskTypeUnknown)
|
||||
}
|
||||
switch {
|
||||
case isGuestThresholdResourceType(q.TypeKey):
|
||||
thresholds = m.customRuleThresholdsNoLock(thresholds, q.Guest)
|
||||
thresholds = m.resolveGuestThresholdOverride(thresholds, q.Guest, q.ResourceID)
|
||||
// Polling and config-save reevaluation both resolve through here,
|
||||
// so a relaxed guest's alert is judged by one threshold. Applied
|
||||
// only by CheckGuest, a save re-judged it by the unrelaxed one.
|
||||
if guest, ok := extractGuestSnapshot(q.Guest); ok && parsePulseTags(guest.Tags).Relaxed {
|
||||
thresholds = applyRelaxedGuestThresholds(thresholds)
|
||||
}
|
||||
case q.TypeKey == "storage":
|
||||
thresholds = m.resolveStorageThresholdOverride(thresholds, q.ResourceID, q.StorageAliases)
|
||||
default:
|
||||
|
||||
@@ -784,6 +784,137 @@ func TestPulseRelaxedThresholdsIncreaseCpuTrigger(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// pulse-relaxed lifted an Off trigger to its floor like any low trigger, so a
|
||||
// relaxed guest with disk alerts off raised one at 96%. A config save then
|
||||
// re-judged it by the Off threshold and resolved it, and the next poll raised
|
||||
// it again. Relaxing a guest must not raise an alert its config turned off.
|
||||
func TestPulseRelaxedKeepsOffGuestThresholdOff(t *testing.T) {
|
||||
m := newTestManager(t)
|
||||
m.mu.Lock()
|
||||
m.config.Enabled = true
|
||||
m.config.TimeThresholds = map[string]int{}
|
||||
m.config.GuestDefaults = ThresholdConfig{
|
||||
CPU: &HysteresisThreshold{Trigger: 80, Clear: 75},
|
||||
Memory: &HysteresisThreshold{Trigger: 85, Clear: 80},
|
||||
Disk: &HysteresisThreshold{Trigger: 0, Clear: 0},
|
||||
}
|
||||
m.mu.Unlock()
|
||||
|
||||
vm := models.VM{
|
||||
ID: "inst/qemu/105",
|
||||
Name: "relaxed-vm",
|
||||
Node: "node1",
|
||||
Instance: "inst",
|
||||
Status: "running",
|
||||
Memory: models.Memory{Usage: 40},
|
||||
Disk: models.Disk{Usage: 96},
|
||||
Tags: []string{"pulse-relaxed"},
|
||||
}
|
||||
|
||||
m.CheckGuest(vm, "inst")
|
||||
if testHasActiveAlert(t, m, canonicalMetricStateID(vm.ID, "disk")) {
|
||||
t.Fatalf("expected no disk alert on a relaxed guest whose disk threshold is off")
|
||||
}
|
||||
}
|
||||
|
||||
// A config save re-judges a relaxed guest's alert by the thresholds CheckGuest
|
||||
// raised it under. Reevaluation rebuilt the guest from alert metadata without
|
||||
// its tags, so an unset memory threshold, which pulse-relaxed sets to 92, read
|
||||
// as no threshold there: the save resolved the alert with a recovery
|
||||
// notification and the next poll raised it again. The tags are read back in
|
||||
// the shape a restart restores them.
|
||||
func TestConfigSaveJudgesRelaxedGuestAlertByRelaxedThresholds(t *testing.T) {
|
||||
m := newTestManager(t)
|
||||
m.mu.Lock()
|
||||
m.config.Enabled = true
|
||||
m.config.TimeThresholds = map[string]int{}
|
||||
m.config.GuestDefaults = ThresholdConfig{}
|
||||
m.mu.Unlock()
|
||||
|
||||
vm := models.VM{
|
||||
ID: "inst/qemu/106",
|
||||
Name: "relaxed-vm",
|
||||
Node: "node1",
|
||||
Instance: "inst",
|
||||
Status: "running",
|
||||
Memory: models.Memory{Usage: 95},
|
||||
Tags: []string{"pulse-relaxed"},
|
||||
}
|
||||
memoryAlertID := canonicalMetricStateID(vm.ID, "memory")
|
||||
|
||||
m.CheckGuest(vm, "inst")
|
||||
alert := testRequireActiveAlert(t, m, memoryAlertID)
|
||||
if alert.Threshold != 92 {
|
||||
t.Fatalf("memory alert threshold = %v, want the relaxed 92", alert.Threshold)
|
||||
}
|
||||
|
||||
m.mu.Lock()
|
||||
raw, err := json.Marshal(alert.Metadata)
|
||||
if err != nil {
|
||||
m.mu.Unlock()
|
||||
t.Fatalf("marshal metadata: %v", err)
|
||||
}
|
||||
var restored map[string]interface{}
|
||||
if err := json.Unmarshal(raw, &restored); err != nil {
|
||||
m.mu.Unlock()
|
||||
t.Fatalf("unmarshal metadata: %v", err)
|
||||
}
|
||||
alert.Metadata = restored
|
||||
m.mu.Unlock()
|
||||
if _, ok := restored["tags"].([]interface{}); !ok {
|
||||
t.Fatalf("restored tags = %T, want []interface{}", restored["tags"])
|
||||
}
|
||||
|
||||
m.UpdateConfig(m.GetConfig())
|
||||
if !testHasActiveAlert(t, m, memoryAlertID) {
|
||||
t.Fatalf("expected a config save to keep a relaxed guest's memory alert over its relaxed trigger")
|
||||
}
|
||||
}
|
||||
|
||||
// pulse-relaxed lifts a guest's disk threshold, but a filesystem's own
|
||||
// override still decides that filesystem, while polling and on a config save.
|
||||
// Judged by the guest's relaxed floor on save, a filesystem alert over its
|
||||
// lower override resolved and the next poll raised it again.
|
||||
func TestConfigSaveKeepsRelaxedGuestFilesystemOverride(t *testing.T) {
|
||||
m := newTestManager(t)
|
||||
guestID := BuildGuestKey("pve1", "node1", 108)
|
||||
m.mu.Lock()
|
||||
m.config.Enabled = true
|
||||
m.config.TimeThresholds = map[string]int{}
|
||||
m.config.GuestDefaults = ThresholdConfig{
|
||||
Disk: &HysteresisThreshold{Trigger: 90, Clear: 85},
|
||||
}
|
||||
m.config.Overrides = map[string]ThresholdConfig{
|
||||
"guest-disk:guest:pve1:108/disk:boot-dev-vda1": {
|
||||
Disk: &HysteresisThreshold{Trigger: 85, Clear: 80},
|
||||
},
|
||||
}
|
||||
m.mu.Unlock()
|
||||
|
||||
vm := models.VM{
|
||||
ID: guestID,
|
||||
VMID: 108,
|
||||
Name: "relaxed-fs",
|
||||
Node: "node1",
|
||||
Instance: "pve1",
|
||||
Status: "running",
|
||||
Disks: []models.Disk{
|
||||
{Mountpoint: "/boot", Device: "/dev/vda1", Usage: 92, Total: 100, Used: 92, Free: 8},
|
||||
},
|
||||
Tags: []string{"pulse-relaxed"},
|
||||
}
|
||||
bootAlertID := canonicalMetricStateID(guestID+"-disk-boot-dev-vda1", "disk")
|
||||
|
||||
m.CheckGuest(vm, "pve1")
|
||||
if !testHasActiveAlert(t, m, bootAlertID) {
|
||||
t.Fatalf("expected a /boot alert over its 85%% override on a relaxed guest")
|
||||
}
|
||||
m.UpdateConfig(m.GetConfig())
|
||||
if !testHasActiveAlert(t, m, bootAlertID) {
|
||||
t.Fatalf("expected a config save to keep a relaxed guest's /boot alert over its own 85%% override")
|
||||
}
|
||||
}
|
||||
|
||||
func TestClearAlertMarksResolutionAndReturnsStatus(t *testing.T) {
|
||||
m := newTestManager(t)
|
||||
m.ClearActiveAlerts()
|
||||
@@ -6349,6 +6480,22 @@ func TestApplyRelaxedGuestThresholds(t *testing.T) {
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("off thresholds stay off", func(t *testing.T) {
|
||||
cfg := ThresholdConfig{
|
||||
CPU: &HysteresisThreshold{Trigger: 0, Clear: 0},
|
||||
Memory: &HysteresisThreshold{Trigger: -1, Clear: -1},
|
||||
Disk: &HysteresisThreshold{Trigger: 0, Clear: 0},
|
||||
}
|
||||
|
||||
result := applyRelaxedGuestThresholds(cfg)
|
||||
|
||||
for name, th := range map[string]*HysteresisThreshold{"CPU": result.CPU, "Memory": result.Memory, "Disk": result.Disk} {
|
||||
if th == nil || th.Trigger > 0 {
|
||||
t.Errorf("%s = %+v, want it to stay off", name, th)
|
||||
}
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("original config unchanged", func(t *testing.T) {
|
||||
// t.Parallel()
|
||||
original := ThresholdConfig{
|
||||
@@ -12395,6 +12542,65 @@ func TestCheckEscalations(t *testing.T) {
|
||||
})
|
||||
}
|
||||
|
||||
// A config save re-judges open TrueNAS disk temperature alerts against the
|
||||
// threshold the next evaluation applies to each disk's type.
|
||||
func TestTrueNASDiskTemperatureAlertsReevaluatePerDiskType(t *testing.T) {
|
||||
m := newTestManager(t)
|
||||
cfg := unifiedEvalBaseConfig()
|
||||
cfg.TrueNASDiskDefaults = ThresholdConfig{}
|
||||
configureUnifiedEvalManager(t, m, cfg)
|
||||
|
||||
nvme := trueNASTemperatureDisk("nvme0n1", "nvme", 72)
|
||||
sata := trueNASTemperatureDisk("sda", "sata", 57)
|
||||
checkTrueNASTemperatureDisk(t, m, nvme)
|
||||
checkTrueNASTemperatureDisk(t, m, sata)
|
||||
for _, disk := range []unifiedresources.Resource{nvme, sata} {
|
||||
if _, firing := trueNASDiskTemperatureAlert(t, m, disk); !firing {
|
||||
t.Fatalf("%s did not fire before the save: %v", disk.ID, alertKeys(m))
|
||||
}
|
||||
}
|
||||
|
||||
// Raising only the NVMe trigger resolves the NVMe alert and leaves the
|
||||
// SATA alert, still over its own trigger, open.
|
||||
m.mu.Lock()
|
||||
m.config.DiskTempByType["nvme"] = HysteresisThreshold{Trigger: 75, Clear: 70}
|
||||
m.reevaluateActiveAlertsLocked()
|
||||
m.mu.Unlock()
|
||||
if _, firing := trueNASDiskTemperatureAlert(t, m, nvme); firing {
|
||||
t.Fatalf("NVMe alert at 72C stayed open under a 75C NVMe trigger")
|
||||
}
|
||||
if _, firing := trueNASDiskTemperatureAlert(t, m, sata); !firing {
|
||||
t.Fatalf("SATA alert at 57C resolved under its unchanged 55C trigger")
|
||||
}
|
||||
}
|
||||
|
||||
// An alert raised before it recorded its disk type is held to the lowest
|
||||
// per-type trigger, so a save never resolves an alert the next evaluation
|
||||
// raises again.
|
||||
func TestTrueNASDiskTemperatureAlertWithoutDiskTypeHeldAtLowestTrigger(t *testing.T) {
|
||||
m := newTestManager(t)
|
||||
cfg := unifiedEvalBaseConfig()
|
||||
cfg.TrueNASDiskDefaults = ThresholdConfig{}
|
||||
cfg.AgentDefaults.DiskTemperature = &HysteresisThreshold{Trigger: 70, Clear: 65}
|
||||
cfg.DiskTempByType = map[string]HysteresisThreshold{"nvme": {Trigger: 60, Clear: 55}}
|
||||
configureUnifiedEvalManager(t, m, cfg)
|
||||
|
||||
nvme := trueNASTemperatureDisk("nvme0n1", "nvme", 66)
|
||||
checkTrueNASTemperatureDisk(t, m, nvme)
|
||||
alert, firing := trueNASDiskTemperatureAlert(t, m, nvme)
|
||||
if !firing {
|
||||
t.Fatalf("NVMe at 66C did not fire under a 60C NVMe trigger: %v", alertKeys(m))
|
||||
}
|
||||
|
||||
m.mu.Lock()
|
||||
delete(alert.Metadata, "diskType")
|
||||
m.reevaluateActiveAlertsLocked()
|
||||
m.mu.Unlock()
|
||||
if _, firing := trueNASDiskTemperatureAlert(t, m, nvme); !firing {
|
||||
t.Fatalf("an alert without a disk type was resolved by the 70C agent default although its NVMe trigger is 60C")
|
||||
}
|
||||
}
|
||||
|
||||
func TestCleanupAlertsForNodes(t *testing.T) {
|
||||
// t.Parallel()
|
||||
|
||||
@@ -22285,14 +22491,20 @@ func TestStaleCleanupKeepsThresholdAlertStillBeingEvaluated(t *testing.T) {
|
||||
|
||||
// Resource history owns each PVE disk alert row by the hardware identity the
|
||||
// alert records, and reads by the path reference find those rows by alert
|
||||
// identifier. Both must match what CheckDiskHealth produces.
|
||||
// identifier. Both must match what CheckDiskHealth and
|
||||
// CheckProxmoxDiskTemperature produce.
|
||||
func TestCheckDiskHealthAlertsCarryHistoryOwnershipIdentity(t *testing.T) {
|
||||
m := newTestManager(t)
|
||||
m.ClearActiveAlerts()
|
||||
m.mu.Lock()
|
||||
m.config.TimeThresholds = map[string]int{}
|
||||
m.config.MetricTimeThresholds = nil
|
||||
m.mu.Unlock()
|
||||
disk := proxmox.Disk{DevPath: "/dev/sdb", Model: "Crucial MX500", Serial: "2117E59AB123", WWN: "0x5002538f12345678",
|
||||
Type: "ssd", Health: "FAILED", Wearout: 3}
|
||||
for i := 0; i < 3; i++ {
|
||||
m.CheckDiskHealth("lab", "pve1", disk)
|
||||
m.CheckProxmoxDiskTemperature("lab", "pve1", ProxmoxDiskTemperatureReading{Disk: disk, Celsius: 80})
|
||||
}
|
||||
|
||||
ref := unifiedresources.ProxmoxPhysicalDiskAlertResourceID("lab", "pve1", disk.DevPath)
|
||||
@@ -22313,3 +22525,187 @@ func TestCheckDiskHealthAlertsCarryHistoryOwnershipIdentity(t *testing.T) {
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func proxmoxTemperatureDisk(devPath, diskType string) proxmox.Disk {
|
||||
return proxmox.Disk{DevPath: devPath, Model: "Test Disk", Serial: "SER-" + devPath, Type: diskType, Health: "PASSED"}
|
||||
}
|
||||
|
||||
func proxmoxDiskTemperatureAlert(t *testing.T, m *Manager, devPath string) (*Alert, bool) {
|
||||
t.Helper()
|
||||
ref := unifiedresources.ProxmoxPhysicalDiskAlertResourceID("lab", "pve1", devPath)
|
||||
return testLookupActiveAlert(t, m, canonicalMetricStateID(ref, proxmoxDiskTemperatureMetric))
|
||||
}
|
||||
|
||||
func proxmoxDiskTemperatureResolution(m *Manager, devPath string) (*models.AlertResolution, bool) {
|
||||
ref := unifiedresources.ProxmoxPhysicalDiskAlertResourceID("lab", "pve1", devPath)
|
||||
id := canonicalMetricStateID(ref, proxmoxDiskTemperatureMetric)
|
||||
for _, resolved := range m.GetRecentlyResolved() {
|
||||
if resolved.ID == id {
|
||||
return resolved.Resolution, true
|
||||
}
|
||||
}
|
||||
return nil, false
|
||||
}
|
||||
|
||||
// TestProxmoxDiskTemperatureAlertsFollowDiskTemperaturePolicy pins Proxmox disk
|
||||
// temperature alerts to the disk temperature policy that judges an agent's
|
||||
// disks, TrueNAS disks and the Physical Disks verdict.
|
||||
func TestProxmoxDiskTemperatureAlertsFollowDiskTemperaturePolicy(t *testing.T) {
|
||||
m := newTestManager(t)
|
||||
configureUnifiedEvalManager(t, m, unifiedEvalBaseConfig())
|
||||
|
||||
check := func(devPath, diskType string, celsius int) {
|
||||
m.CheckProxmoxDiskTemperature("lab", "pve1", ProxmoxDiskTemperatureReading{Disk: proxmoxTemperatureDisk(devPath, diskType), Celsius: celsius})
|
||||
}
|
||||
check("/dev/nvme0n1", "nvme", 63)
|
||||
check("/dev/nvme1n1", "nvme", 72)
|
||||
check("/dev/sda", "sata", 56)
|
||||
check("/dev/sdb", "sas", 63)
|
||||
|
||||
if _, firing := proxmoxDiskTemperatureAlert(t, m, "/dev/nvme0n1"); firing {
|
||||
t.Fatalf("NVMe at 63C fired under its 70C trigger: %v", alertKeys(m))
|
||||
}
|
||||
if _, firing := proxmoxDiskTemperatureAlert(t, m, "/dev/sdb"); firing {
|
||||
t.Fatalf("SAS at 63C fired under its 65C trigger: %v", alertKeys(m))
|
||||
}
|
||||
for devPath, trigger := range map[string]float64{"/dev/nvme1n1": 70, "/dev/sda": 55} {
|
||||
alert, firing := proxmoxDiskTemperatureAlert(t, m, devPath)
|
||||
if !firing {
|
||||
t.Fatalf("%s did not fire: %v", devPath, alertKeys(m))
|
||||
}
|
||||
if alert.Threshold != trigger {
|
||||
t.Fatalf("%s trigger = %v, want %v", devPath, alert.Threshold, trigger)
|
||||
}
|
||||
if alert.Type != proxmoxDiskTemperatureMetric || alert.Metadata["resourceType"] != proxmoxDiskResourceType {
|
||||
t.Fatalf("%s alert type %q, resourceType %v", devPath, alert.Type, alert.Metadata["resourceType"])
|
||||
}
|
||||
if alert.Metadata["diskType"] == nil || alert.Metadata[unifiedresources.MetadataDiskSerial] != "SER-"+devPath {
|
||||
t.Fatalf("%s metadata = %v", devPath, alert.Metadata)
|
||||
}
|
||||
}
|
||||
if alert, _ := proxmoxDiskTemperatureAlert(t, m, "/dev/sda"); alert.Message != "Disk temperature at 56.0°C" {
|
||||
t.Fatalf("message = %q", alert.Message)
|
||||
}
|
||||
|
||||
// A raised NVMe trigger moves NVMe disks with it.
|
||||
m.mu.Lock()
|
||||
m.config.DiskTempByType["nvme"] = HysteresisThreshold{Trigger: 75, Clear: 70}
|
||||
m.mu.Unlock()
|
||||
check("/dev/nvme2n1", "nvme", 72)
|
||||
if _, firing := proxmoxDiskTemperatureAlert(t, m, "/dev/nvme2n1"); firing {
|
||||
t.Fatalf("NVMe at 72C fired under a 75C trigger")
|
||||
}
|
||||
}
|
||||
|
||||
// A disk the linked agent reports is the agent's: its open alert closes as
|
||||
// moved to the agent, and the disk never alerts twice. An excluded disk, or a
|
||||
// policy the agent Disk Temp default switched off, closes it as a plain clear.
|
||||
// A disk with no current reading keeps its alert.
|
||||
func TestProxmoxDiskTemperatureAlertOwnershipAndHolds(t *testing.T) {
|
||||
m := newTestManager(t)
|
||||
configureUnifiedEvalManager(t, m, unifiedEvalBaseConfig())
|
||||
disk := proxmoxTemperatureDisk("/dev/sda", "sata")
|
||||
read := func(reading ProxmoxDiskTemperatureReading) {
|
||||
reading.Disk = disk
|
||||
m.CheckProxmoxDiskTemperature("lab", "pve1", reading)
|
||||
}
|
||||
firing := func() bool {
|
||||
_, ok := proxmoxDiskTemperatureAlert(t, m, "/dev/sda")
|
||||
return ok
|
||||
}
|
||||
|
||||
read(ProxmoxDiskTemperatureReading{Celsius: 60})
|
||||
if !firing() {
|
||||
t.Fatalf("SATA at 60C did not fire: %v", alertKeys(m))
|
||||
}
|
||||
read(ProxmoxDiskTemperatureReading{Celsius: 0})
|
||||
if !firing() {
|
||||
t.Fatalf("a poll without a current reading resolved the alert")
|
||||
}
|
||||
|
||||
read(ProxmoxDiskTemperatureReading{Celsius: 60, AgentOwned: true})
|
||||
if firing() {
|
||||
t.Fatalf("an agent-owned disk kept its Proxmox temperature alert")
|
||||
}
|
||||
if resolution, ok := proxmoxDiskTemperatureResolution(m, "/dev/sda"); !ok || resolution == nil || resolution.Reason != string(AlertResolutionMovedToAgent) {
|
||||
t.Fatalf("handover resolution = %+v (resolved %v), want moved_to_agent", resolution, ok)
|
||||
}
|
||||
read(ProxmoxDiskTemperatureReading{Celsius: 80, AgentOwned: true})
|
||||
if firing() {
|
||||
t.Fatalf("an agent-owned disk raised a Proxmox temperature alert")
|
||||
}
|
||||
|
||||
read(ProxmoxDiskTemperatureReading{Celsius: 60})
|
||||
if !firing() {
|
||||
t.Fatalf("the disk did not alert again once the agent stopped listing it")
|
||||
}
|
||||
read(ProxmoxDiskTemperatureReading{Celsius: 60, Excluded: true})
|
||||
if firing() {
|
||||
t.Fatalf("an excluded disk kept its temperature alert")
|
||||
}
|
||||
|
||||
read(ProxmoxDiskTemperatureReading{Celsius: 60})
|
||||
m.mu.Lock()
|
||||
m.config.AgentDefaults.DiskTemperature = &HysteresisThreshold{Trigger: 0, Clear: 0}
|
||||
m.mu.Unlock()
|
||||
read(ProxmoxDiskTemperatureReading{Celsius: 60})
|
||||
if firing() {
|
||||
t.Fatalf("a switched-off agent Disk Temp default kept the Proxmox disk alert")
|
||||
}
|
||||
}
|
||||
|
||||
// A Proxmox disk temperature warning waits out the factory stability window for
|
||||
// a noisy gauge, as agent and TrueNAS disk temperature warnings do; a critical
|
||||
// reading bypasses it.
|
||||
func TestProxmoxDiskTemperatureWarningWaitsOutStabilityWindow(t *testing.T) {
|
||||
m := newTestManager(t)
|
||||
cfg := unifiedEvalBaseConfig()
|
||||
m.UpdateConfig(cfg)
|
||||
if delay := m.GetConfig().TimeThresholds[proxmoxDiskResourceType]; delay != 5 {
|
||||
t.Fatalf("proxmox-disk delay = %d, want the 5s factory default", delay)
|
||||
}
|
||||
m.mu.RLock()
|
||||
grace := m.getTimeThreshold("lab:pve1:disk:dev-sda", proxmoxDiskResourceType, proxmoxDiskTemperatureMetric)
|
||||
m.mu.RUnlock()
|
||||
if grace != defaultNoisyGaugeStabilitySeconds {
|
||||
t.Fatalf("Proxmox disk temperature delay = %ds, want the %ds stability window", grace, defaultNoisyGaugeStabilitySeconds)
|
||||
}
|
||||
|
||||
m.CheckProxmoxDiskTemperature("lab", "pve1", ProxmoxDiskTemperatureReading{Disk: proxmoxTemperatureDisk("/dev/sda", "sata"), Celsius: 57})
|
||||
if _, firing := proxmoxDiskTemperatureAlert(t, m, "/dev/sda"); firing {
|
||||
t.Fatalf("a 57C SATA warning fired before its stability window")
|
||||
}
|
||||
m.CheckProxmoxDiskTemperature("lab", "pve1", ProxmoxDiskTemperatureReading{Disk: proxmoxTemperatureDisk("/dev/sdb", "sata"), Celsius: 70})
|
||||
if _, firing := proxmoxDiskTemperatureAlert(t, m, "/dev/sdb"); !firing {
|
||||
t.Fatalf("a 70C SATA critical reading waited out the stability window")
|
||||
}
|
||||
}
|
||||
|
||||
// A config save re-judges an open Proxmox disk temperature alert against its
|
||||
// disk type's trigger, as the next poll does; one without a recorded disk type
|
||||
// is held to the lowest per-type trigger.
|
||||
func TestProxmoxDiskTemperatureAlertsReevaluatePerDiskType(t *testing.T) {
|
||||
m := newTestManager(t)
|
||||
cfg := unifiedEvalBaseConfig()
|
||||
cfg.AgentDefaults.DiskTemperature = &HysteresisThreshold{Trigger: 70, Clear: 65}
|
||||
configureUnifiedEvalManager(t, m, cfg)
|
||||
m.CheckProxmoxDiskTemperature("lab", "pve1", ProxmoxDiskTemperatureReading{Disk: proxmoxTemperatureDisk("/dev/nvme0n1", "nvme"), Celsius: 72})
|
||||
m.CheckProxmoxDiskTemperature("lab", "pve1", ProxmoxDiskTemperatureReading{Disk: proxmoxTemperatureDisk("/dev/sda", "sata"), Celsius: 57})
|
||||
sata, ok := proxmoxDiskTemperatureAlert(t, m, "/dev/sda")
|
||||
if !ok {
|
||||
t.Fatalf("SATA at 57C did not fire: %v", alertKeys(m))
|
||||
}
|
||||
|
||||
m.mu.Lock()
|
||||
m.config.DiskTempByType["nvme"] = HysteresisThreshold{Trigger: 75, Clear: 70}
|
||||
delete(sata.Metadata, "diskType")
|
||||
m.reevaluateActiveAlertsLocked()
|
||||
m.mu.Unlock()
|
||||
|
||||
if _, firing := proxmoxDiskTemperatureAlert(t, m, "/dev/nvme0n1"); firing {
|
||||
t.Fatalf("NVMe alert at 72C survived a save raising its trigger to 75C")
|
||||
}
|
||||
if _, firing := proxmoxDiskTemperatureAlert(t, m, "/dev/sda"); !firing {
|
||||
t.Fatalf("a SATA alert without a disk type was resolved by the 70C agent default although its trigger is 55C")
|
||||
}
|
||||
}
|
||||
|
||||
@@ -794,7 +794,6 @@ func TestBranchCovNormalizeTrueNASDefaults(t *testing.T) {
|
||||
{"DiskWrite", t1.DiskWrite, alertconfig.HysteresisThreshold{Trigger: 0, Clear: 0}},
|
||||
{"NetworkIn", t1.NetworkIn, alertconfig.HysteresisThreshold{Trigger: 0, Clear: 0}},
|
||||
{"NetworkOut", t1.NetworkOut, alertconfig.HysteresisThreshold{Trigger: 0, Clear: 0}},
|
||||
{"Disk.Temperature", cfg.TrueNASDiskDefaults.Temperature, alertconfig.HysteresisThreshold{Trigger: 55, Clear: 50}},
|
||||
} {
|
||||
if f.got == nil {
|
||||
t.Fatalf("%s is nil, want non-nil", f.name)
|
||||
@@ -803,6 +802,11 @@ func TestBranchCovNormalizeTrueNASDefaults(t *testing.T) {
|
||||
t.Fatalf("%s = %+v, want %+v", f.name, *f.got, f.want)
|
||||
}
|
||||
}
|
||||
// TrueNAS disks follow the disk temperature policy until the user
|
||||
// saves a TrueNAS-wide value.
|
||||
if cfg.TrueNASDiskDefaults.Temperature != nil {
|
||||
t.Fatalf("Disk.Temperature = %+v, want unset", *cfg.TrueNASDiskDefaults.Temperature)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("derivation and operator-enabled IO metric via CPU/DiskRead", func(t *testing.T) {
|
||||
|
||||
@@ -388,9 +388,35 @@ func NormalizeTrueNASDefaults(config *AlertConfig) {
|
||||
config.TrueNASDefaults.NetworkIn = normalizeThresholdPointer(config.TrueNASDefaults.NetworkIn, 0, 0, "truenas.networkIn")
|
||||
config.TrueNASDefaults.NetworkOut = normalizeThresholdPointer(config.TrueNASDefaults.NetworkOut, 0, 0, "truenas.networkOut")
|
||||
|
||||
config.TrueNASDiskDefaults.Temperature = normalizeThresholdPointer(config.TrueNASDiskDefaults.Temperature, 55, 50, "truenas.disk.temperature")
|
||||
config.TrueNASDiskDefaults.Temperature = normalizeTrueNASDiskTemperature(config.TrueNASDiskDefaults.Temperature, config.TrueNASDiskTemperatureByType)
|
||||
config.TrueNASDiskTemperatureByType = true
|
||||
}
|
||||
|
||||
// normalizeTrueNASDiskTemperature keeps the TrueNAS Disks temperature default
|
||||
// unset unless the user chose one. Unset, each TrueNAS disk follows the disk
|
||||
// temperature policy for its type. Before that, the factory default was a
|
||||
// flat 55/50 and the thresholds page wrote it back on every save, so in a
|
||||
// config written before (byType false) a stored 55/50 marks a value nobody
|
||||
// chose and is migrated to unset. Any other value is the user's TrueNAS-wide
|
||||
// choice and is kept, 0 meaning off. Once byType is set, 55 is a choice too.
|
||||
func normalizeTrueNASDiskTemperature(current *HysteresisThreshold, byType bool) *HysteresisThreshold {
|
||||
if current == nil || current.Trigger < 0 {
|
||||
return nil
|
||||
}
|
||||
if !byType && current.Trigger == legacyTrueNASDiskTemperatureTrigger &&
|
||||
(current.Clear == legacyTrueNASDiskTemperatureClear || current.Clear <= 0) {
|
||||
return nil
|
||||
}
|
||||
return normalizeThresholdPointer(current, 0, 0, "truenas.disk.temperature")
|
||||
}
|
||||
|
||||
// The flat TrueNAS disk temperature factory default before TrueNAS disks
|
||||
// followed the disk temperature policy.
|
||||
const (
|
||||
legacyTrueNASDiskTemperatureTrigger = 55
|
||||
legacyTrueNASDiskTemperatureClear = 50
|
||||
)
|
||||
|
||||
func NormalizeVMwareDefaults(config *AlertConfig) {
|
||||
config.VMwareDefaults.CPU = normalizeThresholdPointer(config.VMwareDefaults.CPU, 80, 75, "vmware.cpu")
|
||||
config.VMwareDefaults.Memory = normalizeThresholdPointer(config.VMwareDefaults.Memory, 85, 80, "vmware.memory")
|
||||
@@ -554,6 +580,10 @@ func NormalizeTimeThresholds(config *AlertConfig) {
|
||||
ensureDelay("truenas-pool")
|
||||
ensureDelay("truenas-dataset")
|
||||
ensureDelay("truenas-disk")
|
||||
// Proxmox disk temperature alerts: the factory stability window for a
|
||||
// noisy gauge applies to this type's unchanged default, as it does to
|
||||
// agent and TrueNAS disk temperatures.
|
||||
ensureDelay("proxmox-disk")
|
||||
ensureDelay("vmware-host")
|
||||
ensureDelay("vmware-vm")
|
||||
ensureDelay("vmware-datastore")
|
||||
|
||||
@@ -0,0 +1,58 @@
|
||||
package config
|
||||
|
||||
import "testing"
|
||||
|
||||
// TestNormalizeTrueNASDiskTemperatureKeepsOnlyChosenValues pins the saved
|
||||
// config migration that lets TrueNAS disks follow the disk temperature policy.
|
||||
// In a config written before it, the old flat factory default was written
|
||||
// back on every save, so it is not a choice; anything else the user saved is
|
||||
// kept.
|
||||
func TestNormalizeTrueNASDiskTemperatureKeepsOnlyChosenValues(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
stored *HysteresisThreshold
|
||||
want *HysteresisThreshold
|
||||
}{
|
||||
{name: "unset stays unset", stored: nil, want: nil},
|
||||
{name: "old factory default is migrated to unset", stored: &HysteresisThreshold{Trigger: 55, Clear: 50}, want: nil},
|
||||
{name: "old factory trigger without a clear is migrated to unset", stored: &HysteresisThreshold{Trigger: 55}, want: nil},
|
||||
{name: "negative trigger is unset", stored: &HysteresisThreshold{Trigger: -1}, want: nil},
|
||||
{name: "off is kept", stored: &HysteresisThreshold{Trigger: 0, Clear: 10}, want: &HysteresisThreshold{Trigger: 0, Clear: 0}},
|
||||
{name: "custom value is kept", stored: &HysteresisThreshold{Trigger: 62, Clear: 58}, want: &HysteresisThreshold{Trigger: 62, Clear: 58}},
|
||||
{name: "custom value gets a clear", stored: &HysteresisThreshold{Trigger: 62}, want: &HysteresisThreshold{Trigger: 62, Clear: 57}},
|
||||
{name: "55 with a custom clear is a choice", stored: &HysteresisThreshold{Trigger: 55, Clear: 45}, want: &HysteresisThreshold{Trigger: 55, Clear: 45}},
|
||||
}
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
cfg := &AlertConfig{TrueNASDiskDefaults: ThresholdConfig{Temperature: tt.stored}}
|
||||
NormalizeTrueNASDefaults(cfg)
|
||||
if !cfg.TrueNASDiskTemperatureByType {
|
||||
t.Fatalf("normalized config is not marked as following disk type")
|
||||
}
|
||||
got := cfg.TrueNASDiskDefaults.Temperature
|
||||
switch {
|
||||
case tt.want == nil && got != nil:
|
||||
t.Fatalf("Temperature = %+v, want unset", *got)
|
||||
case tt.want != nil && got == nil:
|
||||
t.Fatalf("Temperature unset, want %+v", *tt.want)
|
||||
case tt.want != nil && *got != *tt.want:
|
||||
t.Fatalf("Temperature = %+v, want %+v", *got, *tt.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// Once a config follows disk type, 55 is a value the user typed, and a save
|
||||
// or restart must keep it.
|
||||
func TestNormalizeTrueNASDiskTemperatureKeepsChosen55(t *testing.T) {
|
||||
cfg := &AlertConfig{
|
||||
TrueNASDiskDefaults: ThresholdConfig{Temperature: &HysteresisThreshold{Trigger: 55, Clear: 50}},
|
||||
TrueNASDiskTemperatureByType: true,
|
||||
}
|
||||
NormalizeTrueNASDefaults(cfg)
|
||||
NormalizeTrueNASDefaults(cfg)
|
||||
got := cfg.TrueNASDiskDefaults.Temperature
|
||||
if got == nil || *got != (HysteresisThreshold{Trigger: 55, Clear: 50}) {
|
||||
t.Fatalf("Temperature = %+v, want the chosen 55/50 kept", got)
|
||||
}
|
||||
}
|
||||
@@ -388,6 +388,13 @@ type AlertConfig struct {
|
||||
FlappingWindowSeconds int `json:"flappingWindowSeconds"` // Time window for counting state changes
|
||||
FlappingThreshold int `json:"flappingThreshold"` // Number of state changes to trigger flapping
|
||||
FlappingCooldownMinutes int `json:"flappingCooldownMinutes"` // Cooldown period after flapping detected
|
||||
|
||||
// TrueNASDiskTemperatureByType marks a config written since an unset
|
||||
// TrueNASDiskDefaults.Temperature means each TrueNAS disk follows the
|
||||
// disk temperature policy for its type. An older config carries the
|
||||
// former flat factory value, which nobody chose, and normalization drops
|
||||
// it once.
|
||||
TrueNASDiskTemperatureByType bool `json:"truenasDiskTemperatureByType,omitempty"`
|
||||
}
|
||||
|
||||
// UnmarshalJSON accepts canonical v6 alert config keys.
|
||||
|
||||
@@ -417,7 +417,17 @@ func (m *Manager) reevaluateActiveAlertsLocked() {
|
||||
alertsToResolve = append(alertsToResolve, alertID)
|
||||
continue
|
||||
}
|
||||
thresholds := m.resolveResourceThresholds(primaryResourceType, resourceID)
|
||||
query := alertPolicyQuery{TypeKey: primaryResourceType, ResourceID: resourceID}
|
||||
if primaryResourceType == "truenas-disk" {
|
||||
// Judge the alert against its disk type's threshold, as the
|
||||
// next evaluation will. An alert raised before it recorded a
|
||||
// disk type is held to the lowest per-type trigger, so a save
|
||||
// never resolves an alert its type would raise again.
|
||||
diskType, known := alert.Metadata["diskType"].(string)
|
||||
query.DiskType = diskType
|
||||
query.DiskTypeUnknown = !known
|
||||
}
|
||||
thresholds := m.effectiveAlertPolicyNoLock(query).Thresholds
|
||||
if thresholds.Disabled {
|
||||
alertsToResolve = append(alertsToResolve, alertID)
|
||||
continue
|
||||
@@ -425,6 +435,19 @@ func (m *Manager) reevaluateActiveAlertsLocked() {
|
||||
threshold = getThresholdForMetric(thresholds, metricType)
|
||||
}
|
||||
|
||||
// A Proxmox disk temperature alert is judged by the disk temperature
|
||||
// policy for its disk type, as CheckProxmoxDiskTemperature judges it.
|
||||
// The node and guest policies below have no disk temperature, so a
|
||||
// save would otherwise resolve an alert the next poll raises again.
|
||||
if primaryResourceType == proxmoxDiskResourceType && metricType == proxmoxDiskTemperatureMetric {
|
||||
diskType, known := alert.Metadata["diskType"].(string)
|
||||
threshold = m.proxmoxDiskTemperatureThresholdNoLock(diskType, !known)
|
||||
if threshold == nil || threshold.Trigger <= 0 {
|
||||
alertsToResolve = append(alertsToResolve, alertID)
|
||||
continue
|
||||
}
|
||||
}
|
||||
|
||||
isAgentResource := alertResourceTypeKeysContain(resourceTypeKeys, "agent")
|
||||
if !handledModernPlatformType && isAgentResource {
|
||||
if allDisabled, _ := m.alertPolicyTypeSwitchesNoLock("agent"); allDisabled {
|
||||
|
||||
@@ -106,9 +106,11 @@ func defaultAlertConfig() AlertConfig {
|
||||
NetworkIn: &HysteresisThreshold{Trigger: 0, Clear: 0},
|
||||
NetworkOut: &HysteresisThreshold{Trigger: 0, Clear: 0},
|
||||
},
|
||||
TrueNASDiskDefaults: ThresholdConfig{
|
||||
Temperature: &HysteresisThreshold{Trigger: 55, Clear: 50},
|
||||
},
|
||||
// TrueNAS disk temperature is unset: each disk follows the disk
|
||||
// temperature policy for its type (DiskTempByType) until the user
|
||||
// saves a TrueNAS-wide value.
|
||||
TrueNASDiskDefaults: ThresholdConfig{},
|
||||
TrueNASDiskTemperatureByType: true,
|
||||
VMwareDefaults: ThresholdConfig{
|
||||
CPU: &HysteresisThreshold{Trigger: 80, Clear: 75},
|
||||
Memory: &HysteresisThreshold{Trigger: 85, Clear: 80},
|
||||
@@ -148,6 +150,7 @@ func defaultAlertConfig() AlertConfig {
|
||||
"truenas-pool": 5,
|
||||
"truenas-dataset": 5,
|
||||
"truenas-disk": 5,
|
||||
"proxmox-disk": 5,
|
||||
"vmware-host": 5,
|
||||
"vmware-vm": 5,
|
||||
"vmware-datastore": 5,
|
||||
|
||||
+10
-11
@@ -141,8 +141,7 @@ func (m *Manager) CheckGuest(guest any, instanceName string) {
|
||||
return
|
||||
}
|
||||
|
||||
settings := policy.TagSettings
|
||||
monitorOnly := settings.MonitorOnly
|
||||
monitorOnly := policy.TagSettings.MonitorOnly
|
||||
if monitorOnly || m.guestHasMonitorOnlyAlerts(guestID) {
|
||||
log.Debug().
|
||||
Str("guest", name).
|
||||
@@ -215,19 +214,12 @@ func (m *Manager) CheckGuest(guest any, instanceName string) {
|
||||
// If guest is running, clear any powered-off alert
|
||||
m.clearGuestPoweredOffAlert(guestID, name)
|
||||
|
||||
// Get thresholds (check custom rules, then overrides, then defaults)
|
||||
// Get thresholds (defaults, custom rules, overrides, then the
|
||||
// pulse-relaxed floor)
|
||||
m.mu.RLock()
|
||||
thresholds := m.getGuestThresholds(guest, guestID)
|
||||
m.mu.RUnlock()
|
||||
|
||||
if settings.Relaxed {
|
||||
thresholds = applyRelaxedGuestThresholds(thresholds)
|
||||
log.Info().
|
||||
Str("guest", name).
|
||||
Float64("trigger", thresholds.CPU.Trigger).
|
||||
Msg("Applied relaxed thresholds for pulse-relaxed tag")
|
||||
}
|
||||
|
||||
// If alerts are disabled for this guest, clear any existing alerts and return
|
||||
if thresholds.Disabled {
|
||||
if alertsCleared := m.clearGuestMetricAlerts(guestID); alertsCleared > 0 {
|
||||
@@ -691,6 +683,10 @@ func parsePulseTags(tags []string) pulseTagSettings {
|
||||
return settings
|
||||
}
|
||||
|
||||
// applyRelaxedGuestThresholds lifts a pulse-relaxed guest's CPU, memory and
|
||||
// disk triggers to a 95/92/95 floor. An unset threshold gets the floor; one
|
||||
// switched Off (trigger <= 0) stays Off, since relaxing a guest must never
|
||||
// raise an alert its config turned off.
|
||||
func applyRelaxedGuestThresholds(cfg ThresholdConfig) ThresholdConfig {
|
||||
relaxed := cloneThresholdConfig(cfg)
|
||||
|
||||
@@ -699,6 +695,9 @@ func applyRelaxedGuestThresholds(cfg ThresholdConfig) ThresholdConfig {
|
||||
*th = &HysteresisThreshold{Trigger: minTrigger, Clear: minTrigger - 5}
|
||||
return
|
||||
}
|
||||
if (*th).Trigger <= 0 {
|
||||
return
|
||||
}
|
||||
ensureHysteresisThreshold(*th)
|
||||
if (*th).Trigger < minTrigger {
|
||||
(*th).Trigger = minTrigger
|
||||
|
||||
@@ -251,6 +251,9 @@ func guestSnapshotFromAlert(alert *Alert, resourceID string) guestSnapshot {
|
||||
if snapshot.VMID <= 0 {
|
||||
snapshot.VMID = metadataIntValue(alert.Metadata["guestVmid"])
|
||||
}
|
||||
// Tags carry pulse-relaxed, so a config save re-judges the alert by the
|
||||
// thresholds CheckGuest raised it under.
|
||||
snapshot.Tags, _ = metadataStringsValue(alert.Metadata, "tags")
|
||||
return snapshot.normalizeCollections()
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,110 @@
|
||||
package alerts
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"strings"
|
||||
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources"
|
||||
"github.com/rcourtman/pulse-go-rewrite/pkg/proxmox"
|
||||
"github.com/rs/zerolog/log"
|
||||
)
|
||||
|
||||
// proxmoxDiskTemperatureMetric is the metric of a Proxmox physical disk's
|
||||
// temperature alert, the same metric CheckHost raises for an agent's disks.
|
||||
const proxmoxDiskTemperatureMetric = "diskTemperature"
|
||||
|
||||
// proxmoxDiskResourceType is the resource type PVE disk alerts carry in their
|
||||
// spec and metadata (see CheckDiskHealth).
|
||||
const proxmoxDiskResourceType = "proxmox-disk"
|
||||
|
||||
// ProxmoxDiskTemperatureReading is one Proxmox physical disk's temperature as
|
||||
// the PVE disk poller judged it.
|
||||
type ProxmoxDiskTemperatureReading struct {
|
||||
Disk proxmox.Disk
|
||||
// Celsius is the reading collected this poll, or 0 when there is none:
|
||||
// no sensor reports the disk, it is in standby, or normalization kept a
|
||||
// last-known value it did not collect now.
|
||||
Celsius int
|
||||
// AgentOwned says a linked Pulse agent lists the disk in its SMART
|
||||
// report, so the agent's CheckHost owns its temperature alert.
|
||||
AgentOwned bool
|
||||
// Excluded says the linked agent's --disk-exclude patterns match the
|
||||
// device, which excludes it from disk alerts.
|
||||
Excluded bool
|
||||
}
|
||||
|
||||
// CheckProxmoxDiskTemperature raises a Proxmox physical disk's temperature
|
||||
// alert under the disk temperature policy (DiskTemperatureThreshold): the
|
||||
// same per-type thresholds that judge an agent's disks, TrueNAS disks and the
|
||||
// Physical Disks verdict. Without it a disk that only the node's sensors
|
||||
// report could read Running Hot and never alert.
|
||||
//
|
||||
// A disk a linked agent lists is the agent's: its open alert here closes as
|
||||
// moved to the agent rather than recovered, so the disk never alerts twice.
|
||||
// An excluded disk, or a policy switched off by the agent Disk Temp default,
|
||||
// closes its alert. A disk with no current reading keeps its alert as it is,
|
||||
// since no reading shows it cooled.
|
||||
func (m *Manager) CheckProxmoxDiskTemperature(instance, node string, reading ProxmoxDiskTemperatureReading) {
|
||||
if m == nil {
|
||||
return
|
||||
}
|
||||
disk := reading.Disk
|
||||
resourceID := proxmoxDiskCanonicalResourceID(instance, node, disk.DevPath)
|
||||
resourceName := fmt.Sprintf("%s (%s)", disk.Model, disk.DevPath)
|
||||
diskType := strings.ToLower(strings.TrimSpace(disk.Type))
|
||||
|
||||
m.mu.RLock()
|
||||
enabled := m.config.Enabled
|
||||
threshold := m.proxmoxDiskTemperatureThresholdNoLock(diskType, false)
|
||||
m.mu.RUnlock()
|
||||
if !enabled {
|
||||
return
|
||||
}
|
||||
|
||||
if reading.AgentOwned || reading.Excluded || threshold == nil || threshold.Trigger <= 0 {
|
||||
spec, err := buildCanonicalMetricSpec(resourceID, resourceName, unifiedresources.ResourceType(proxmoxDiskResourceType), proxmoxDiskTemperatureMetric, nil)
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
var resolution *AlertResolution
|
||||
if reading.AgentOwned && !reading.Excluded {
|
||||
resolution = &AlertResolution{Reason: AlertResolutionMovedToAgent}
|
||||
}
|
||||
m.releaseCanonicalMetricAlert(spec, resourceName, node, instance, proxmoxDiskResourceType, 0, resolution)
|
||||
return
|
||||
}
|
||||
if reading.Celsius <= 0 {
|
||||
return
|
||||
}
|
||||
|
||||
spec, err := buildCanonicalMetricSpec(resourceID, resourceName, unifiedresources.ResourceType(proxmoxDiskResourceType), proxmoxDiskTemperatureMetric, threshold)
|
||||
if err != nil {
|
||||
log.Warn().
|
||||
Err(err).
|
||||
Str("resourceID", resourceID).
|
||||
Str("node", node).
|
||||
Str("disk", disk.DevPath).
|
||||
Msg("Skipping invalid canonical Proxmox disk temperature metric spec")
|
||||
return
|
||||
}
|
||||
metadata := proxmoxDiskAlertMetadata(disk)
|
||||
metadata["metric"] = proxmoxDiskTemperatureMetric
|
||||
// A config save re-judges the open alert against its disk type.
|
||||
metadata["diskType"] = diskType
|
||||
value := float64(reading.Celsius)
|
||||
m.checkMetricWithCanonicalSpec(spec, resourceName, node, instance, proxmoxDiskResourceType, value, threshold, &metricOptions{
|
||||
Metadata: metadata,
|
||||
Message: fmt.Sprintf("Disk temperature at %s", formatMetricValue(value, "°C")),
|
||||
})
|
||||
}
|
||||
|
||||
// proxmoxDiskTemperatureThresholdNoLock is the disk temperature policy for a
|
||||
// Proxmox disk of diskType, or the lowest per-type trigger for an alert whose
|
||||
// disk type was never recorded, so a config save never resolves an alert the
|
||||
// next poll raises again. Callers must hold m.mu.
|
||||
func (m *Manager) proxmoxDiskTemperatureThresholdNoLock(diskType string, typeUnknown bool) *HysteresisThreshold {
|
||||
if typeUnknown {
|
||||
return cloneThreshold(m.lowestHostDiskTemperatureThresholdNoLock(m.config.AgentDefaults.DiskTemperature))
|
||||
}
|
||||
return cloneThreshold(m.hostDiskTemperatureThresholdNoLock(m.config.AgentDefaults.DiskTemperature, false, diskType))
|
||||
}
|
||||
@@ -196,6 +196,24 @@ func metadataStringValue(metadata map[string]interface{}, key string) string {
|
||||
}
|
||||
}
|
||||
|
||||
// metadataStringsValue reads a string list from alert metadata. A live alert
|
||||
// holds a []string; one reloaded from disk holds a []interface{}.
|
||||
func metadataStringsValue(metadata map[string]interface{}, key string) ([]string, bool) {
|
||||
switch values := metadata[key].(type) {
|
||||
case []string:
|
||||
return append([]string(nil), values...), true
|
||||
case []interface{}:
|
||||
result := make([]string, 0, len(values))
|
||||
for _, value := range values {
|
||||
if s, ok := value.(string); ok {
|
||||
result = append(result, s)
|
||||
}
|
||||
}
|
||||
return result, true
|
||||
}
|
||||
return nil, false
|
||||
}
|
||||
|
||||
func metadataBoolValue(metadata map[string]interface{}, key string) bool {
|
||||
if metadata == nil {
|
||||
return false
|
||||
|
||||
@@ -76,17 +76,8 @@ func storagePolicyAliases(alert *Alert) []string {
|
||||
if alert == nil {
|
||||
return nil
|
||||
}
|
||||
switch aliases := alert.Metadata[storagePolicyAliasesKey].(type) {
|
||||
case []string:
|
||||
return append([]string(nil), aliases...)
|
||||
case []interface{}:
|
||||
result := make([]string, 0, len(aliases))
|
||||
for _, value := range aliases {
|
||||
if alias, ok := value.(string); ok {
|
||||
result = append(result, alias)
|
||||
}
|
||||
}
|
||||
return result
|
||||
if aliases, ok := metadataStringsValue(alert.Metadata, storagePolicyAliasesKey); ok {
|
||||
return aliases
|
||||
}
|
||||
// Older PBS snapshots predate alias metadata. The poller records the
|
||||
// instance and datastore name separately: only reconstruct an alias when
|
||||
|
||||
@@ -0,0 +1,18 @@
|
||||
package alerts
|
||||
|
||||
// trueNASDiskTemperatureDefaultNoLock returns the default tier of a TrueNAS
|
||||
// disk's temperature threshold. A TrueNAS-wide value the user saved under
|
||||
// TrueNAS Disks applies to every TrueNAS disk, as a host Disk Temp override
|
||||
// does for an agent's disks. Without one the disk follows the disk
|
||||
// temperature policy for its type, the threshold DiskTemperatureThreshold
|
||||
// returns, so its alert agrees with the TrueNAS storage table and Patrol. A
|
||||
// per-disk override still beats both. Callers must hold m.mu.
|
||||
func (m *Manager) trueNASDiskTemperatureDefaultNoLock(diskType string, typeUnknown bool) *HysteresisThreshold {
|
||||
if explicit := m.config.TrueNASDiskDefaults.Temperature; explicit != nil {
|
||||
return cloneThreshold(explicit)
|
||||
}
|
||||
if typeUnknown {
|
||||
return cloneThreshold(m.lowestHostDiskTemperatureThresholdNoLock(m.config.AgentDefaults.DiskTemperature))
|
||||
}
|
||||
return cloneThreshold(m.hostDiskTemperatureThresholdNoLock(m.config.AgentDefaults.DiskTemperature, false, diskType))
|
||||
}
|
||||
@@ -6,6 +6,7 @@ import (
|
||||
|
||||
alertspecs "github.com/rcourtman/pulse-go-rewrite/internal/alerts/specs"
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources"
|
||||
"github.com/rcourtman/pulse-go-rewrite/pkg/diskinventory"
|
||||
"github.com/rs/zerolog/log"
|
||||
)
|
||||
|
||||
@@ -57,6 +58,9 @@ type UnifiedResourceInput struct {
|
||||
Temperature *UnifiedResourceMetric
|
||||
|
||||
StorageAliases []string // Durable policy lookup identities for storage alerts.
|
||||
// DiskType is a physical disk's type ("nvme", "sas", "sata"). It picks
|
||||
// the disk temperature threshold of a TrueNAS disk.
|
||||
DiskType string
|
||||
}
|
||||
|
||||
type unifiedMetricCandidate struct {
|
||||
@@ -297,6 +301,11 @@ func (m *Manager) evaluateUnifiedMetrics(input *UnifiedResourceInput, thresholds
|
||||
if platformType != "" {
|
||||
metadata[alertPlatformTypeKey] = platformType
|
||||
}
|
||||
if input.Type == "truenas-disk" {
|
||||
// A config save re-judges the open temperature alert against
|
||||
// the threshold for this disk type.
|
||||
metadata["diskType"] = input.DiskType
|
||||
}
|
||||
merged.Metadata = metadata
|
||||
opts = &merged
|
||||
}
|
||||
@@ -357,15 +366,19 @@ func (m *Manager) interruptUnobservedUnifiedMetrics(input *UnifiedResourceInput,
|
||||
}
|
||||
}
|
||||
|
||||
// metricOptionsWithTags records a resource's tags on its metric alerts. With
|
||||
// no tags it drops any an open alert still carries: config-save reevaluation
|
||||
// reads pulse-relaxed back from them, and a removed tag left behind judged
|
||||
// the alert by thresholds the evaluator no longer applies.
|
||||
func metricOptionsWithTags(opts *metricOptions, tags []string) *metricOptions {
|
||||
if len(tags) == 0 {
|
||||
return opts
|
||||
}
|
||||
|
||||
merged := metricOptions{}
|
||||
if opts != nil {
|
||||
merged = *opts
|
||||
}
|
||||
if len(tags) == 0 {
|
||||
merged.RemoveMetadata = append(append([]string(nil), merged.RemoveMetadata...), "tags")
|
||||
return &merged
|
||||
}
|
||||
metadata := make(map[string]interface{}, len(merged.Metadata)+1)
|
||||
for key, value := range merged.Metadata {
|
||||
metadata[key] = value
|
||||
@@ -603,10 +616,15 @@ func UnifiedResourceInputFromResource(resource unifiedresources.Resource) (*Unif
|
||||
}
|
||||
if resource.Temperature != nil {
|
||||
input.Temperature = &UnifiedResourceMetric{Value: *resource.Temperature, Percent: *resource.Temperature}
|
||||
} else if resource.PhysicalDisk != nil && resource.PhysicalDisk.Temperature > 0 {
|
||||
} else if resource.PhysicalDisk != nil &&
|
||||
diskinventory.TemperatureCollected(resource.PhysicalDisk.Temperature, resource.PhysicalDisk.Collection) {
|
||||
// A retained last-known reading is not current evidence of heat.
|
||||
value := float64(resource.PhysicalDisk.Temperature)
|
||||
input.Temperature = &UnifiedResourceMetric{Value: value, Percent: value}
|
||||
}
|
||||
if resource.PhysicalDisk != nil {
|
||||
input.DiskType = strings.ToLower(strings.TrimSpace(resource.PhysicalDisk.DiskType))
|
||||
}
|
||||
|
||||
return input, true
|
||||
}
|
||||
@@ -850,7 +868,7 @@ func (m *Manager) CheckUnifiedResourceWithCapacityTrend(input *UnifiedResourceIn
|
||||
return
|
||||
}
|
||||
|
||||
thresholds := m.resolveResourceThresholds(input.Type, input.ID)
|
||||
thresholds := m.effectiveAlertPolicyNoLock(alertPolicyQuery{TypeKey: input.Type, ResourceID: input.ID, DiskType: input.DiskType}).Thresholds
|
||||
m.mu.RUnlock()
|
||||
|
||||
if thresholds.Disabled {
|
||||
|
||||
@@ -10,6 +10,7 @@ import (
|
||||
alertspecs "github.com/rcourtman/pulse-go-rewrite/internal/alerts/specs"
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/models"
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources"
|
||||
"github.com/rcourtman/pulse-go-rewrite/pkg/diskinventory"
|
||||
)
|
||||
|
||||
func unifiedEvalBaseConfig() AlertConfig {
|
||||
@@ -901,6 +902,51 @@ func TestCheckGuestPerDiskAnnotatesCanonicalSpecMetadata(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// A guest that drops its last tag leaves none on its open alerts. Config-save
|
||||
// reevaluation reads pulse-relaxed back from alert metadata, so a removed tag
|
||||
// left behind judged the alert by the relaxed trigger: the save resolved a
|
||||
// reading still over the guest's own trigger, and the next poll raised it
|
||||
// again.
|
||||
func TestCheckGuestDropsRemovedTagsFromOpenAlerts(t *testing.T) {
|
||||
m := newTestManager(t)
|
||||
configureUnifiedEvalManager(t, m, unifiedEvalBaseConfig())
|
||||
|
||||
guestID := BuildGuestKey("pve1", "node1", 102)
|
||||
vm := models.VM{
|
||||
ID: guestID,
|
||||
VMID: 102,
|
||||
Name: "app02",
|
||||
Node: "node1",
|
||||
Instance: "pve1",
|
||||
Status: "running",
|
||||
CPU: 0.97,
|
||||
Memory: models.Memory{Usage: 40},
|
||||
Disk: models.Disk{Usage: 40},
|
||||
Tags: []string{"pulse-relaxed"},
|
||||
}
|
||||
alertID := canonicalMetricStateID(guestID, "cpu")
|
||||
m.CheckGuest(vm, "pve1")
|
||||
if !testHasActiveAlert(t, m, alertID) {
|
||||
t.Fatalf("expected CPU alert %q over the relaxed trigger", alertID)
|
||||
}
|
||||
|
||||
vm.Tags = nil
|
||||
vm.CPU = 0.90
|
||||
m.CheckGuest(vm, "pve1")
|
||||
alert := testRequireActiveAlert(t, m, alertID)
|
||||
m.mu.RLock()
|
||||
tags, hasTags := alert.Metadata["tags"]
|
||||
m.mu.RUnlock()
|
||||
if hasTags {
|
||||
t.Fatalf("alert metadata tags = %v, want none once the guest dropped them", tags)
|
||||
}
|
||||
|
||||
m.UpdateConfig(m.GetConfig())
|
||||
if !testHasActiveAlert(t, m, alertID) {
|
||||
t.Fatalf("expected a config save to keep a CPU alert over the guest's own trigger after pulse-relaxed is removed")
|
||||
}
|
||||
}
|
||||
|
||||
func TestCheckGuestPerDiskCleansUpRemovedDiskAlerts(t *testing.T) {
|
||||
m := newTestManager(t)
|
||||
configureUnifiedEvalManager(t, m, unifiedEvalBaseConfig())
|
||||
@@ -1812,3 +1858,167 @@ func TestBuildMetricAlertStatusCarriesWindowAndMonotonicRecovery(t *testing.T) {
|
||||
t.Fatalf("status without clear level = %+v", noClear)
|
||||
}
|
||||
}
|
||||
|
||||
func trueNASTemperatureDisk(device, diskType string, celsius int) unifiedresources.Resource {
|
||||
return unifiedresources.Resource{
|
||||
ID: "physical-disk:truenas-main/" + device,
|
||||
Type: unifiedresources.ResourceTypePhysicalDisk,
|
||||
Name: device,
|
||||
ParentName: "truenas-main",
|
||||
Sources: []unifiedresources.DataSource{unifiedresources.SourceTrueNAS},
|
||||
PhysicalDisk: &unifiedresources.PhysicalDiskMeta{
|
||||
DevPath: "/dev/" + device,
|
||||
DiskType: diskType,
|
||||
Temperature: celsius,
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
func trueNASDiskTemperatureAlertID(resource unifiedresources.Resource) string {
|
||||
return canonicalMetricStateID(resource.ID, "temperature")
|
||||
}
|
||||
|
||||
func checkTrueNASTemperatureDisk(t *testing.T, m *Manager, resource unifiedresources.Resource) {
|
||||
t.Helper()
|
||||
input, ok := UnifiedResourceInputFromResource(resource)
|
||||
if !ok {
|
||||
t.Fatalf("TrueNAS disk %s did not become alert input", resource.ID)
|
||||
}
|
||||
m.CheckUnifiedResource(input)
|
||||
}
|
||||
|
||||
func trueNASDiskTemperatureAlert(t *testing.T, m *Manager, resource unifiedresources.Resource) (*Alert, bool) {
|
||||
t.Helper()
|
||||
return testLookupActiveAlert(t, m, trueNASDiskTemperatureAlertID(resource))
|
||||
}
|
||||
|
||||
// TestTrueNASDiskTemperatureAlertsFollowDiskTemperaturePolicy pins TrueNAS
|
||||
// disk temperature alerts to the disk temperature policy that the TrueNAS
|
||||
// storage table, Physical Disks and Patrol judge heat by: per disk type,
|
||||
// unless the user saved a TrueNAS-wide value, with a per-disk override first.
|
||||
func TestTrueNASDiskTemperatureAlertsFollowDiskTemperaturePolicy(t *testing.T) {
|
||||
nvme63 := trueNASTemperatureDisk("nvme0n1", "nvme", 63)
|
||||
nvme72 := trueNASTemperatureDisk("nvme1n1", "nvme", 72)
|
||||
sata56 := trueNASTemperatureDisk("sda", "sata", 56)
|
||||
sata60 := trueNASTemperatureDisk("sdb", "sata", 60)
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
mutate func(*AlertConfig)
|
||||
disks []unifiedresources.Resource
|
||||
firing []unifiedresources.Resource
|
||||
trigger map[string]float64
|
||||
}{
|
||||
{
|
||||
name: "factory thresholds judge each disk by its type",
|
||||
disks: []unifiedresources.Resource{nvme63, nvme72, sata56},
|
||||
firing: []unifiedresources.Resource{nvme72, sata56},
|
||||
trigger: map[string]float64{nvme72.ID: 70, sata56.ID: 55},
|
||||
},
|
||||
{
|
||||
name: "a raised NVMe trigger moves NVMe alerts with it",
|
||||
mutate: func(cfg *AlertConfig) {
|
||||
cfg.DiskTempByType = map[string]HysteresisThreshold{"nvme": {Trigger: 75, Clear: 70}}
|
||||
},
|
||||
disks: []unifiedresources.Resource{nvme72, sata56},
|
||||
firing: []unifiedresources.Resource{sata56},
|
||||
},
|
||||
{
|
||||
name: "a saved TrueNAS-wide value applies to every TrueNAS disk",
|
||||
mutate: func(cfg *AlertConfig) {
|
||||
cfg.TrueNASDiskDefaults.Temperature = &HysteresisThreshold{Trigger: 62, Clear: 57}
|
||||
},
|
||||
disks: []unifiedresources.Resource{nvme63, sata60},
|
||||
firing: []unifiedresources.Resource{nvme63},
|
||||
trigger: map[string]float64{nvme63.ID: 62},
|
||||
},
|
||||
{
|
||||
name: "a per-disk override beats the TrueNAS-wide value",
|
||||
mutate: func(cfg *AlertConfig) {
|
||||
cfg.TrueNASDiskDefaults.Temperature = &HysteresisThreshold{Trigger: 62, Clear: 57}
|
||||
cfg.Overrides = map[string]ThresholdConfig{
|
||||
sata56.ID: {Temperature: &HysteresisThreshold{Trigger: 50, Clear: 45}},
|
||||
}
|
||||
},
|
||||
disks: []unifiedresources.Resource{sata56},
|
||||
firing: []unifiedresources.Resource{sata56},
|
||||
trigger: map[string]float64{sata56.ID: 50},
|
||||
},
|
||||
{
|
||||
name: "a switched-off agent Disk Temp default switches TrueNAS disks off",
|
||||
mutate: func(cfg *AlertConfig) {
|
||||
cfg.AgentDefaults.DiskTemperature = &HysteresisThreshold{Trigger: 0, Clear: 0}
|
||||
},
|
||||
disks: []unifiedresources.Resource{nvme72, sata60},
|
||||
},
|
||||
{
|
||||
name: "a saved TrueNAS-wide value still applies when the agent default is off",
|
||||
mutate: func(cfg *AlertConfig) {
|
||||
cfg.AgentDefaults.DiskTemperature = &HysteresisThreshold{Trigger: 0, Clear: 0}
|
||||
cfg.TrueNASDiskDefaults.Temperature = &HysteresisThreshold{Trigger: 58, Clear: 53}
|
||||
},
|
||||
disks: []unifiedresources.Resource{sata60},
|
||||
firing: []unifiedresources.Resource{sata60},
|
||||
},
|
||||
{
|
||||
name: "a saved TrueNAS-wide off switches TrueNAS disks off",
|
||||
mutate: func(cfg *AlertConfig) {
|
||||
cfg.TrueNASDiskDefaults.Temperature = &HysteresisThreshold{Trigger: 0, Clear: 0}
|
||||
},
|
||||
disks: []unifiedresources.Resource{nvme72, sata60},
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
m := newTestManager(t)
|
||||
cfg := unifiedEvalBaseConfig()
|
||||
cfg.TrueNASDiskDefaults = ThresholdConfig{}
|
||||
if tt.mutate != nil {
|
||||
tt.mutate(&cfg)
|
||||
}
|
||||
configureUnifiedEvalManager(t, m, cfg)
|
||||
|
||||
for _, disk := range tt.disks {
|
||||
checkTrueNASTemperatureDisk(t, m, disk)
|
||||
}
|
||||
|
||||
want := make(map[string]bool, len(tt.firing))
|
||||
for _, disk := range tt.firing {
|
||||
want[disk.ID] = true
|
||||
}
|
||||
for _, disk := range tt.disks {
|
||||
alert, firing := trueNASDiskTemperatureAlert(t, m, disk)
|
||||
if firing != want[disk.ID] {
|
||||
t.Fatalf("%s firing = %v, want %v (active: %v)", disk.ID, firing, want[disk.ID], alertKeys(m))
|
||||
}
|
||||
if !firing {
|
||||
continue
|
||||
}
|
||||
if got := alert.Metadata["diskType"]; got != disk.PhysicalDisk.DiskType {
|
||||
t.Fatalf("%s diskType metadata = %v, want %q", disk.ID, got, disk.PhysicalDisk.DiskType)
|
||||
}
|
||||
if trigger, ok := tt.trigger[disk.ID]; ok && alert.Threshold != trigger {
|
||||
t.Fatalf("%s threshold = %v, want %v", disk.ID, alert.Threshold, trigger)
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// A last-known reading the source no longer collects is not evidence of heat.
|
||||
func TestTrueNASDiskTemperatureAlertIgnoresRetainedReading(t *testing.T) {
|
||||
m := newTestManager(t)
|
||||
cfg := unifiedEvalBaseConfig()
|
||||
cfg.TrueNASDiskDefaults = ThresholdConfig{}
|
||||
configureUnifiedEvalManager(t, m, cfg)
|
||||
|
||||
disk := trueNASTemperatureDisk("sda", "sata", 80)
|
||||
disk.PhysicalDisk.Collection = &diskinventory.CollectionStatus{
|
||||
Temperature: diskinventory.Unavailable("truenas", "disk is in standby"),
|
||||
}
|
||||
checkTrueNASTemperatureDisk(t, m, disk)
|
||||
if _, firing := trueNASDiskTemperatureAlert(t, m, disk); firing {
|
||||
t.Fatalf("retained 80C reading raised an alert: %v", alertKeys(m))
|
||||
}
|
||||
}
|
||||
|
||||
@@ -2971,8 +2971,25 @@ func (h *AIHandler) HandleMessages(w http.ResponseWriter, r *http.Request, sessi
|
||||
json.NewEncoder(w).Encode(messages)
|
||||
}
|
||||
|
||||
// requireAssistantSessionMutationMethod admits only POST to an Assistant
|
||||
// session mutation (abort, summarize, fork, undo, redo, steer). The session
|
||||
// router dispatches these sub-resources by path alone, and GET/HEAD pass both
|
||||
// the demo-mode read-only guard and the CSRF check while SameSite=Lax cookies
|
||||
// ride cross-site top-level GET navigations, so each mutation owns its method.
|
||||
func requireAssistantSessionMutationMethod(w http.ResponseWriter, r *http.Request) bool {
|
||||
if r.Method == http.MethodPost {
|
||||
return true
|
||||
}
|
||||
w.Header().Set("Allow", http.MethodPost)
|
||||
http.Error(w, "Method not allowed", http.StatusMethodNotAllowed)
|
||||
return false
|
||||
}
|
||||
|
||||
// HandleAbort handles POST /api/ai/sessions/{id}/abort
|
||||
func (h *AIHandler) HandleAbort(w http.ResponseWriter, r *http.Request, sessionID string) {
|
||||
if !requireAssistantSessionMutationMethod(w, r) {
|
||||
return
|
||||
}
|
||||
ctx := r.Context()
|
||||
if !h.IsRunning(ctx) {
|
||||
http.Error(w, "Pulse Assistant is not running", http.StatusServiceUnavailable)
|
||||
@@ -3187,6 +3204,9 @@ func (h *AIHandler) recordFirstPartyWorkflowPromptActivity(ctx context.Context,
|
||||
// HandleSummarize handles POST /api/ai/sessions/{id}/summarize
|
||||
// Compresses context when nearing model limits
|
||||
func (h *AIHandler) HandleSummarize(w http.ResponseWriter, r *http.Request, sessionID string) {
|
||||
if !requireAssistantSessionMutationMethod(w, r) {
|
||||
return
|
||||
}
|
||||
ctx := r.Context()
|
||||
if !h.IsRunning(ctx) {
|
||||
http.Error(w, "Pulse Assistant is not running", http.StatusServiceUnavailable)
|
||||
@@ -3218,6 +3238,9 @@ func (h *AIHandler) HandleDiff(w http.ResponseWriter, r *http.Request, sessionID
|
||||
// HandleFork handles POST /api/ai/sessions/{id}/fork
|
||||
// Creates a branch point in the conversation
|
||||
func (h *AIHandler) HandleFork(w http.ResponseWriter, r *http.Request, sessionID string) {
|
||||
if !requireAssistantSessionMutationMethod(w, r) {
|
||||
return
|
||||
}
|
||||
ctx := r.Context()
|
||||
if !h.IsRunning(ctx) {
|
||||
http.Error(w, "Pulse Assistant is not running", http.StatusServiceUnavailable)
|
||||
@@ -3244,6 +3267,9 @@ func (h *AIHandler) HandleFork(w http.ResponseWriter, r *http.Request, sessionID
|
||||
// The body is optional; retry/regenerate sends SessionTurnUndoOptions with the
|
||||
// prompt being re-run so a stale retry can never remove a different turn.
|
||||
func (h *AIHandler) HandleUndoLastTurn(w http.ResponseWriter, r *http.Request, sessionID string) {
|
||||
if !requireAssistantSessionMutationMethod(w, r) {
|
||||
return
|
||||
}
|
||||
ctx := r.Context()
|
||||
if !h.IsRunning(ctx) {
|
||||
http.Error(w, "Pulse Assistant is not running", http.StatusServiceUnavailable)
|
||||
@@ -3276,6 +3302,9 @@ func (h *AIHandler) HandleUndoLastTurn(w http.ResponseWriter, r *http.Request, s
|
||||
|
||||
// HandleRedoLastTurn handles POST /api/ai/sessions/{id}/redo.
|
||||
func (h *AIHandler) HandleRedoLastTurn(w http.ResponseWriter, r *http.Request, sessionID string) {
|
||||
if !requireAssistantSessionMutationMethod(w, r) {
|
||||
return
|
||||
}
|
||||
ctx := r.Context()
|
||||
if !h.IsRunning(ctx) {
|
||||
http.Error(w, "Pulse Assistant is not running", http.StatusServiceUnavailable)
|
||||
@@ -3303,8 +3332,7 @@ func (h *AIHandler) HandleRedoLastTurn(w http.ResponseWriter, r *http.Request, s
|
||||
// loop; accepted=false with a reason is a normal outcome and the client
|
||||
// falls back to the ordinary follow-up queue.
|
||||
func (h *AIHandler) HandleSteerSession(w http.ResponseWriter, r *http.Request, sessionID string) {
|
||||
if r.Method != http.MethodPost {
|
||||
http.Error(w, "Method not allowed", http.StatusMethodNotAllowed)
|
||||
if !requireAssistantSessionMutationMethod(w, r) {
|
||||
return
|
||||
}
|
||||
ctx := r.Context()
|
||||
|
||||
@@ -7,6 +7,7 @@ import (
|
||||
"net/http/httptest"
|
||||
"reflect"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
"unsafe"
|
||||
@@ -14,6 +15,7 @@ import (
|
||||
"github.com/gorilla/websocket"
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/agentexec"
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/ai"
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/ai/chat"
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/ai/cost"
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/ai/memory"
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/alerts"
|
||||
@@ -22,6 +24,7 @@ import (
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/monitoring"
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/servicediscovery"
|
||||
unifiedresources "github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources"
|
||||
internalauth "github.com/rcourtman/pulse-go-rewrite/pkg/auth"
|
||||
)
|
||||
|
||||
type stubMetadataProvider struct{}
|
||||
@@ -795,3 +798,204 @@ func TestHandleExportAICostHistoryCarriesPromptCacheBuckets(t *testing.T) {
|
||||
t.Fatalf("csv row missing cache buckets or cache-rate pricing: %s", lines[1])
|
||||
}
|
||||
}
|
||||
|
||||
// sessionMutationRecordingAIService records every Assistant session mutation
|
||||
// and every running check that reaches the service, so a rejected request can
|
||||
// be proven to stop before the handler touches the service at all.
|
||||
type sessionMutationRecordingAIService struct {
|
||||
capturingAIService
|
||||
mu sync.Mutex
|
||||
mutations []string
|
||||
runningCalls int
|
||||
}
|
||||
|
||||
func newSessionMutationRecordingAIService() *sessionMutationRecordingAIService {
|
||||
return &sessionMutationRecordingAIService{capturingAIService: capturingAIService{running: true}}
|
||||
}
|
||||
|
||||
func (s *sessionMutationRecordingAIService) record(op, sessionID string) {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
s.mutations = append(s.mutations, op+":"+sessionID)
|
||||
}
|
||||
|
||||
func (s *sessionMutationRecordingAIService) recorded() []string {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
return append([]string(nil), s.mutations...)
|
||||
}
|
||||
|
||||
func (s *sessionMutationRecordingAIService) IsRunning() bool {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
s.runningCalls++
|
||||
return s.capturingAIService.running
|
||||
}
|
||||
|
||||
func (s *sessionMutationRecordingAIService) runningChecks() int {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
return s.runningCalls
|
||||
}
|
||||
|
||||
func (s *sessionMutationRecordingAIService) AbortSession(ctx context.Context, sessionID string) error {
|
||||
s.record("abort", sessionID)
|
||||
return nil
|
||||
}
|
||||
|
||||
func (s *sessionMutationRecordingAIService) SummarizeSession(ctx context.Context, sessionID string) (map[string]interface{}, error) {
|
||||
s.record("summarize", sessionID)
|
||||
return map[string]interface{}{"success": true}, nil
|
||||
}
|
||||
|
||||
func (s *sessionMutationRecordingAIService) ForkSession(ctx context.Context, sessionID string) (*chat.Session, error) {
|
||||
s.record("fork", sessionID)
|
||||
return &chat.Session{ID: sessionID + "-fork"}, nil
|
||||
}
|
||||
|
||||
func (s *sessionMutationRecordingAIService) UndoLastTurn(ctx context.Context, sessionID string, opts chat.SessionTurnUndoOptions) (*chat.SessionTurnUndoResult, error) {
|
||||
s.record("undo", sessionID)
|
||||
return &chat.SessionTurnUndoResult{Success: true, SessionID: sessionID}, nil
|
||||
}
|
||||
|
||||
func (s *sessionMutationRecordingAIService) RedoLastTurn(ctx context.Context, sessionID string) (*chat.SessionTurnRedoResult, error) {
|
||||
s.record("redo", sessionID)
|
||||
return &chat.SessionTurnRedoResult{Success: true, SessionID: sessionID}, nil
|
||||
}
|
||||
|
||||
func (s *sessionMutationRecordingAIService) SteerSession(ctx context.Context, sessionID string, req chat.SessionSteerRequest) (*chat.SessionSteerResult, error) {
|
||||
s.record("steer", sessionID)
|
||||
return &chat.SessionSteerResult{Accepted: true, SessionID: sessionID}, nil
|
||||
}
|
||||
|
||||
var assistantSessionMutationSubresources = []string{"abort", "summarize", "fork", "undo", "redo", "steer"}
|
||||
|
||||
func TestAssistantSessionMutationsRequirePost(t *testing.T) {
|
||||
for _, sub := range assistantSessionMutationSubresources {
|
||||
for _, method := range []string{http.MethodGet, http.MethodHead, http.MethodPut, http.MethodDelete} {
|
||||
t.Run(sub+"/"+method, func(t *testing.T) {
|
||||
svc := newSessionMutationRecordingAIService()
|
||||
handler := &AIHandler{}
|
||||
setUnexportedField(t, handler, "defaultService", svc)
|
||||
router := &Router{aiHandler: handler}
|
||||
|
||||
req := httptest.NewRequest(method, "/api/ai/sessions/session-1/"+sub, nil)
|
||||
rec := httptest.NewRecorder()
|
||||
router.routeAISessions(rec, req)
|
||||
|
||||
if rec.Code != http.StatusMethodNotAllowed {
|
||||
t.Fatalf("%s %s status = %d, want %d", method, sub, rec.Code, http.StatusMethodNotAllowed)
|
||||
}
|
||||
if allow := rec.Header().Get("Allow"); allow != http.MethodPost {
|
||||
t.Fatalf("%s %s Allow = %q, want %q", method, sub, allow, http.MethodPost)
|
||||
}
|
||||
if got := svc.recorded(); len(got) != 0 {
|
||||
t.Fatalf("%s %s reached the service: %v", method, sub, got)
|
||||
}
|
||||
if checks := svc.runningChecks(); checks != 0 {
|
||||
t.Fatalf("%s %s resolved the service (%d running checks) before rejecting the method", method, sub, checks)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
t.Run(sub+"/POST", func(t *testing.T) {
|
||||
svc := newSessionMutationRecordingAIService()
|
||||
handler := &AIHandler{}
|
||||
setUnexportedField(t, handler, "defaultService", svc)
|
||||
router := &Router{aiHandler: handler}
|
||||
|
||||
body := ""
|
||||
if sub == "steer" {
|
||||
body = `{"prompt":"check the backup job too"}`
|
||||
}
|
||||
req := httptest.NewRequest(http.MethodPost, "/api/ai/sessions/session-1/"+sub, strings.NewReader(body))
|
||||
rec := httptest.NewRecorder()
|
||||
router.routeAISessions(rec, req)
|
||||
|
||||
if rec.Code != http.StatusOK {
|
||||
t.Fatalf("POST %s status = %d, want %d: %s", sub, rec.Code, http.StatusOK, rec.Body.String())
|
||||
}
|
||||
if got := svc.recorded(); !reflect.DeepEqual(got, []string{sub + ":session-1"}) {
|
||||
t.Fatalf("POST %s mutations = %v, want [%s:session-1]", sub, got, sub)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestAssistantSessionReadSubresourcesKeepTheirMethods(t *testing.T) {
|
||||
svc := newSessionMutationRecordingAIService()
|
||||
handler := &AIHandler{}
|
||||
setUnexportedField(t, handler, "defaultService", svc)
|
||||
router := &Router{aiHandler: handler}
|
||||
|
||||
messagesReq := httptest.NewRequest(http.MethodGet, "/api/ai/sessions/session-1/messages", nil)
|
||||
messagesRec := httptest.NewRecorder()
|
||||
router.routeAISessions(messagesRec, messagesReq)
|
||||
if messagesRec.Code != http.StatusOK {
|
||||
t.Fatalf("GET messages status = %d, want %d: %s", messagesRec.Code, http.StatusOK, messagesRec.Body.String())
|
||||
}
|
||||
|
||||
diffReq := httptest.NewRequest(http.MethodGet, "/api/ai/sessions/session-1/diff", nil)
|
||||
diffRec := httptest.NewRecorder()
|
||||
router.routeAISessions(diffRec, diffReq)
|
||||
if diffRec.Code != http.StatusNotImplemented {
|
||||
t.Fatalf("GET diff status = %d, want %d: %s", diffRec.Code, http.StatusNotImplemented, diffRec.Body.String())
|
||||
}
|
||||
|
||||
if got := svc.recorded(); len(got) != 0 {
|
||||
t.Fatalf("read sub-resources mutated the session: %v", got)
|
||||
}
|
||||
}
|
||||
|
||||
// TestDemoModeRouterRejectsAssistantSessionMutationsOverSafeMethods drives the
|
||||
// production handler chain: the demo-mode guard admits GET and HEAD as reads,
|
||||
// so the session mutation itself must refuse them.
|
||||
func TestDemoModeRouterRejectsAssistantSessionMutationsOverSafeMethods(t *testing.T) {
|
||||
tempDir := t.TempDir()
|
||||
hashed, err := internalauth.HashPassword("Password!1")
|
||||
if err != nil {
|
||||
t.Fatalf("hash password: %v", err)
|
||||
}
|
||||
cfg := &config.Config{
|
||||
DataPath: tempDir,
|
||||
ConfigPath: tempDir,
|
||||
AuthUser: "admin",
|
||||
AuthPass: hashed,
|
||||
DemoMode: true,
|
||||
}
|
||||
router := NewRouter(cfg, nil, nil, nil, nil, "1.0.0")
|
||||
t.Cleanup(router.shutdownBackgroundWorkers)
|
||||
svc := newSessionMutationRecordingAIService()
|
||||
setUnexportedField(t, router.aiHandler, "defaultService", svc)
|
||||
|
||||
serve := func(method, path string) *httptest.ResponseRecorder {
|
||||
t.Helper()
|
||||
req := httptest.NewRequest(method, path, nil)
|
||||
req.SetBasicAuth("admin", "Password!1")
|
||||
rec := httptest.NewRecorder()
|
||||
router.Handler().ServeHTTP(rec, req)
|
||||
return rec
|
||||
}
|
||||
|
||||
for _, sub := range assistantSessionMutationSubresources {
|
||||
for _, method := range []string{http.MethodGet, http.MethodHead} {
|
||||
rec := serve(method, "/api/ai/sessions/session-1/"+sub)
|
||||
if rec.Header().Get("X-Demo-Mode") != "true" {
|
||||
t.Fatalf("%s %s did not pass through the demo-mode guard", method, sub)
|
||||
}
|
||||
if rec.Code != http.StatusMethodNotAllowed {
|
||||
t.Fatalf("demo %s %s status = %d, want %d: %s", method, sub, rec.Code, http.StatusMethodNotAllowed, rec.Body.String())
|
||||
}
|
||||
}
|
||||
if rec := serve(http.MethodPost, "/api/ai/sessions/session-1/"+sub); rec.Code != http.StatusForbidden {
|
||||
t.Fatalf("demo POST %s status = %d, want %d: %s", sub, rec.Code, http.StatusForbidden, rec.Body.String())
|
||||
}
|
||||
}
|
||||
if got := svc.recorded(); len(got) != 0 {
|
||||
t.Fatalf("demo-mode requests mutated Assistant sessions: %v", got)
|
||||
}
|
||||
|
||||
if rec := serve(http.MethodGet, "/api/ai/sessions/session-1/messages"); rec.Code != http.StatusOK {
|
||||
t.Fatalf("demo GET messages status = %d, want %d: %s", rec.Code, http.StatusOK, rec.Body.String())
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,6 +1,8 @@
|
||||
package models
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
@@ -512,3 +514,22 @@ func TestCloneHostXCPNGInventoryIsolation(t *testing.T) {
|
||||
t.Fatal("cloned XCP-ng inventory aliases source VMs")
|
||||
}
|
||||
}
|
||||
|
||||
// AgentSMARTReported is internal poll evidence: a state snapshot copy keeps it
|
||||
// for the alert check, and it never reaches an API or websocket payload.
|
||||
func TestPhysicalDiskAgentSMARTReportedStaysInternal(t *testing.T) {
|
||||
disk := PhysicalDisk{ID: "lab-pve1-sda", DevPath: "/dev/sda", AgentSMARTReported: true}
|
||||
if clone := clonePhysicalDisk(disk); !clone.AgentSMARTReported {
|
||||
t.Fatalf("clone dropped AgentSMARTReported")
|
||||
}
|
||||
if clones := clonePhysicalDisks([]PhysicalDisk{disk}); !clones[0].AgentSMARTReported {
|
||||
t.Fatalf("slice clone dropped AgentSMARTReported")
|
||||
}
|
||||
payload, err := json.Marshal(disk)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if strings.Contains(strings.ToLower(string(payload)), "agentsmartreported") {
|
||||
t.Fatalf("AgentSMARTReported leaked into JSON: %s", payload)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -2766,7 +2766,11 @@ type PhysicalDisk struct {
|
||||
IO *DiskIO `json:"io,omitempty"`
|
||||
Collection *diskinventory.CollectionStatus `json:"collection,omitempty"`
|
||||
ExpectedUpdateInterval time.Duration `json:"-"` // Collector schedule, independent of general node polling
|
||||
LastChecked time.Time `json:"lastChecked"`
|
||||
// AgentSMARTReported marks a disk a linked host agent lists in its SMART
|
||||
// report, so the agent's CheckHost owns its temperature alert. Internal
|
||||
// poll evidence, set by the PVE disk poller's agent SMART merge.
|
||||
AgentSMARTReported bool `json:"-"`
|
||||
LastChecked time.Time `json:"lastChecked"`
|
||||
}
|
||||
|
||||
// PBSInstance represents a Proxmox Backup Server instance
|
||||
|
||||
@@ -513,6 +513,7 @@ func mergeHostAgentSMARTIntoDisks(disks []models.PhysicalDisk, nodes []models.No
|
||||
|
||||
// Build a map of node name to linked host's SMART data
|
||||
smartByNodeName := make(map[string][]models.HostDiskSMART)
|
||||
reportingNodeNames := make(map[string]bool)
|
||||
for _, node := range nodes {
|
||||
if node.LinkedAgentID == "" {
|
||||
continue
|
||||
@@ -522,6 +523,11 @@ func mergeHostAgentSMARTIntoDisks(disks []models.PhysicalDisk, nodes []models.No
|
||||
continue
|
||||
}
|
||||
smartByNodeName[node.Name] = host.Sensors.SMART
|
||||
// Only an agent still reporting owns its disks' temperature alerts;
|
||||
// a silent one's retained rows still enrich the disk.
|
||||
if !strings.EqualFold(strings.TrimSpace(host.Status), "offline") {
|
||||
reportingNodeNames[node.Name] = true
|
||||
}
|
||||
log.Debug().
|
||||
Str("nodeName", node.Name).
|
||||
Str("hostAgentID", node.LinkedAgentID).
|
||||
@@ -608,6 +614,11 @@ func mergeHostAgentSMARTIntoDisks(disks []models.PhysicalDisk, nodes []models.No
|
||||
if matched == nil {
|
||||
continue
|
||||
}
|
||||
// The linked agent reports this disk, so while it keeps reporting its
|
||||
// CheckHost owns the disk's temperature alert and the PVE disk check
|
||||
// stands aside. Once its lease lapses, CheckHost no longer alerts on
|
||||
// the disk and the PVE check judges the node's own readings again.
|
||||
updated[i].AgentSMARTReported = reportingNodeNames[updated[i].Node]
|
||||
|
||||
if strings.TrimSpace(updated[i].Model) == "" && strings.TrimSpace(matched.Model) != "" {
|
||||
updated[i].Model = strings.TrimSpace(matched.Model)
|
||||
|
||||
@@ -10,6 +10,7 @@ import (
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/alerts"
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/config"
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/logging"
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/models"
|
||||
@@ -1416,19 +1417,35 @@ func preserveUnavailablePhysicalDiskEvidence(current, previous models.PhysicalDi
|
||||
return current
|
||||
}
|
||||
|
||||
// checkPhysicalDiskAlerts evaluates one Proxmox physical disk's health and
|
||||
// wearout alerts. A device matched by the linked host agent's --disk-exclude
|
||||
// patterns is evaluated as healthy, so excluding a disk also resolves the
|
||||
// alerts it already raised.
|
||||
// checkPhysicalDiskAlerts evaluates one Proxmox physical disk's health,
|
||||
// wearout and temperature alerts. A device matched by the linked host agent's
|
||||
// --disk-exclude patterns is evaluated as healthy and without a temperature
|
||||
// alert, so excluding a disk also resolves the alerts it already raised.
|
||||
func (m *Monitor) checkPhysicalDiskAlerts(instance string, disk models.PhysicalDisk, excludePatterns []string) {
|
||||
if len(excludePatterns) > 0 && fsfilters.MatchesDeviceExclude(disk.DevPath, excludePatterns) {
|
||||
excluded := len(excludePatterns) > 0 && fsfilters.MatchesDeviceExclude(disk.DevPath, excludePatterns)
|
||||
if excluded {
|
||||
healthyDisk := proxmoxDiskFromPhysicalDisk(disk)
|
||||
healthyDisk.Health = "PASSED"
|
||||
healthyDisk.Wearout = 100
|
||||
m.alertManager.CheckDiskHealth(instance, disk.Node, healthyDisk)
|
||||
return
|
||||
} else {
|
||||
m.alertManager.CheckDiskHealth(instance, disk.Node, proxmoxDiskFromPhysicalDisk(disk))
|
||||
}
|
||||
m.alertManager.CheckDiskHealth(instance, disk.Node, proxmoxDiskFromPhysicalDisk(disk))
|
||||
m.alertManager.CheckProxmoxDiskTemperature(instance, disk.Node, alerts.ProxmoxDiskTemperatureReading{
|
||||
Disk: proxmoxDiskFromPhysicalDisk(disk),
|
||||
Celsius: collectedPhysicalDiskTemperature(disk),
|
||||
AgentOwned: disk.AgentSMARTReported,
|
||||
Excluded: excluded,
|
||||
})
|
||||
}
|
||||
|
||||
// collectedPhysicalDiskTemperature is the disk's temperature when this poll
|
||||
// collected it, or 0 for a last-known value normalization kept.
|
||||
func collectedPhysicalDiskTemperature(disk models.PhysicalDisk) int {
|
||||
if !diskinventory.TemperatureCollected(disk.Temperature, disk.Collection) {
|
||||
return 0
|
||||
}
|
||||
return disk.Temperature
|
||||
}
|
||||
|
||||
func proxmoxDiskFromPhysicalDisk(disk models.PhysicalDisk) proxmox.Disk {
|
||||
|
||||
@@ -1121,3 +1121,96 @@ func TestTrueNASPlaceholderDiskSerialsStayPerApplianceAndShareOneHistoryKey(t *t
|
||||
t.Fatalf("canonical TrueNAS disks = %d, want %d (one per appliance and device)", disks, diskCount)
|
||||
}
|
||||
}
|
||||
|
||||
// The agent SMART merge marks each Proxmox disk a reporting linked agent
|
||||
// lists, so the disk's temperature alert is left to the agent's CheckHost.
|
||||
// Disks the agent does not list, disks on nodes without an agent, and disks
|
||||
// of an agent whose lease lapsed stay unmarked. Disks the poller builds from
|
||||
// the agent's SMART report when the Proxmox query fails go through the same
|
||||
// merge.
|
||||
func TestMergeHostAgentSMARTIntoDisksMarksDisksTheAgentReports(t *testing.T) {
|
||||
disks := []models.PhysicalDisk{
|
||||
{Instance: "pve1", Node: "node1", DevPath: "/dev/sda"},
|
||||
{Instance: "pve1", Node: "node1", DevPath: "/dev/sdb"},
|
||||
{Instance: "pve1", Node: "node2", DevPath: "/dev/sda"},
|
||||
{Instance: "pve1", Node: "node3", DevPath: "/dev/sda"},
|
||||
}
|
||||
nodes := []models.Node{
|
||||
{Name: "node1", LinkedAgentID: "host-node1"},
|
||||
{Name: "node2"},
|
||||
{Name: "node3", LinkedAgentID: "host-node3"},
|
||||
}
|
||||
smart := []models.HostDiskSMART{{Device: "sda", Serial: "SER-SDA", Type: "sata", Temperature: 40}}
|
||||
hosts := []models.Host{
|
||||
{ID: "host-node1", Status: "online", Sensors: models.HostSensorSummary{SMART: smart}},
|
||||
{ID: "host-node3", Status: "offline", Sensors: models.HostSensorSummary{SMART: smart}},
|
||||
}
|
||||
|
||||
merged := mergeHostAgentSMARTIntoDisks(disks, nodes, hosts)
|
||||
for i, want := range []bool{true, false, false, false} {
|
||||
if merged[i].AgentSMARTReported != want {
|
||||
t.Fatalf("%s/%s AgentSMARTReported = %v, want %v", merged[i].Node, merged[i].DevPath, merged[i].AgentSMARTReported, want)
|
||||
}
|
||||
}
|
||||
if disks[0].AgentSMARTReported {
|
||||
t.Fatalf("merge modified the input slice")
|
||||
}
|
||||
if merged[3].Serial != "SER-SDA" {
|
||||
t.Fatalf("a silent agent's retained row no longer enriches the disk: %+v", merged[3])
|
||||
}
|
||||
|
||||
fallback := mergeHostAgentSMARTIntoDisks(physicalDisksFromHostAgentSMART("pve1", "node1", smart), nodes, hosts)
|
||||
if len(fallback) != 1 || !fallback[0].AgentSMARTReported {
|
||||
t.Fatalf("a disk built from a reporting agent's SMART report is not the agent's: %+v", fallback)
|
||||
}
|
||||
}
|
||||
|
||||
// checkPhysicalDiskAlerts raises a Proxmox disk's temperature alert from a
|
||||
// reading collected this poll, under the disk temperature policy, and leaves
|
||||
// alone a retained reading, a disk the linked agent reports and an excluded
|
||||
// device.
|
||||
func TestCheckPhysicalDiskAlertsRaisesProxmoxDiskTemperatureAlerts(t *testing.T) {
|
||||
manager := alerts.NewManagerWithDataDir(t.TempDir())
|
||||
m := &Monitor{alertManager: manager}
|
||||
const instance, node = "pve1", "node1"
|
||||
alertID := func(devPath string) string {
|
||||
return unifiedresources.ProxmoxPhysicalDiskAlertResourceID(instance, node, devPath) + "::metric-threshold:diskTemperature"
|
||||
}
|
||||
active := func(devPath string) bool {
|
||||
for _, alert := range manager.GetActiveAlerts() {
|
||||
if alert.ID == alertID(devPath) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
// 70C is critical for SATA (trigger 55, critical 65), which fires without
|
||||
// the stability delay a warning waits out.
|
||||
sata := func(devPath string) models.PhysicalDisk {
|
||||
return models.PhysicalDisk{Instance: instance, Node: node, DevPath: devPath, Model: "Test SATA", Type: "sata", Health: "PASSED", Wearout: -1, Temperature: 70}
|
||||
}
|
||||
|
||||
m.checkPhysicalDiskAlerts(instance, sata("/dev/sda"), nil)
|
||||
if !active("/dev/sda") {
|
||||
t.Fatalf("a SATA disk at 70C raised no temperature alert: %+v", manager.GetActiveAlerts())
|
||||
}
|
||||
|
||||
retained := sata("/dev/sdb")
|
||||
retained.Collection = &diskinventory.CollectionStatus{Temperature: diskinventory.Unavailable("proxmox_node_smart", "disk is in standby")}
|
||||
m.checkPhysicalDiskAlerts(instance, retained, nil)
|
||||
if active("/dev/sdb") {
|
||||
t.Fatalf("a retained 70C reading raised a temperature alert")
|
||||
}
|
||||
|
||||
agentOwned := sata("/dev/sdc")
|
||||
agentOwned.AgentSMARTReported = true
|
||||
m.checkPhysicalDiskAlerts(instance, agentOwned, nil)
|
||||
if active("/dev/sdc") {
|
||||
t.Fatalf("a disk the linked agent reports raised a second temperature alert")
|
||||
}
|
||||
|
||||
m.checkPhysicalDiskAlerts(instance, sata("/dev/sdd"), []string{"sdd"})
|
||||
if active("/dev/sdd") {
|
||||
t.Fatalf("an excluded device raised a temperature alert")
|
||||
}
|
||||
}
|
||||
|
||||
@@ -856,7 +856,7 @@ func TestProxmoxPhysicalDiskAlertReferenceShape(t *testing.T) {
|
||||
ProxmoxPhysicalDiskAlertResourceID("lab", "pve1", "/"),
|
||||
} {
|
||||
require.True(t, isProxmoxPhysicalDiskAlertReference(ref), ref)
|
||||
require.Len(t, ProxmoxPhysicalDiskAlertIdentifiers(ref), 2, ref)
|
||||
require.Len(t, ProxmoxPhysicalDiskAlertIdentifiers(ref), 3, ref)
|
||||
}
|
||||
for _, ref := range []string{
|
||||
ProxmoxPhysicalDiskAlertResourceID("lab", "pve1", ""),
|
||||
|
||||
@@ -28,6 +28,11 @@ const (
|
||||
// "<reference>::<reference><suffix>" (buildCanonicalStateID over the spec ID).
|
||||
var proxmoxDiskAlertSpecSuffixes = []string{"-health", "-wearout"}
|
||||
|
||||
// proxmoxDiskTemperatureAlertSpecID is the PVE disk temperature alert's spec,
|
||||
// a metric threshold spec (canonicalMetricSpecID over "diskTemperature"), so
|
||||
// its identifier is "<reference>::metric-threshold:diskTemperature".
|
||||
const proxmoxDiskTemperatureAlertSpecID = "metric-threshold:diskTemperature"
|
||||
|
||||
// ProxmoxPhysicalDiskAlertIdentifiers lists the alert identifiers of the PVE
|
||||
// disk alerts raised under ref, or nil when ref is not shaped like a PVE disk
|
||||
// alert reference.
|
||||
@@ -35,11 +40,11 @@ func ProxmoxPhysicalDiskAlertIdentifiers(ref string) []string {
|
||||
if !isProxmoxPhysicalDiskAlertReference(ref) {
|
||||
return nil
|
||||
}
|
||||
identifiers := make([]string, 0, len(proxmoxDiskAlertSpecSuffixes))
|
||||
identifiers := make([]string, 0, len(proxmoxDiskAlertSpecSuffixes)+1)
|
||||
for _, suffix := range proxmoxDiskAlertSpecSuffixes {
|
||||
identifiers = append(identifiers, ref+"::"+ref+suffix)
|
||||
}
|
||||
return identifiers
|
||||
return append(identifiers, ref+"::"+proxmoxDiskTemperatureAlertSpecID)
|
||||
}
|
||||
|
||||
// isProxmoxPhysicalDiskAlertReference matches "<instance>:<node>:disk:<key>"
|
||||
|
||||
@@ -14,6 +14,7 @@ import (
|
||||
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/models"
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/operationreceipt"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
func TestSQLiteResourceStoreDoesNotPersistRawOperationReceiptProtocolOrReplayAuthority(t *testing.T) {
|
||||
@@ -3328,3 +3329,32 @@ func TestSuccessionMergesGuestChangeJournalEras(t *testing.T) {
|
||||
t.Fatalf("legacy-era journal row missing from new-ID read: %+v", changes)
|
||||
}
|
||||
}
|
||||
|
||||
// A PVE disk temperature alert row is owned by the hardware identity it
|
||||
// records, like the health and wearout rows, and a read by the path reference
|
||||
// still finds it through ProxmoxPhysicalDiskAlertIdentifiers.
|
||||
func TestProxmoxDiskTemperatureAlertRowsFollowRecordedHardwareIdentity(t *testing.T) {
|
||||
store, err := NewSQLiteResourceStore(t.TempDir(), "default")
|
||||
require.NoError(t, err)
|
||||
t.Cleanup(func() { require.NoError(t, store.Close()) })
|
||||
now := time.Now().UTC().Truncate(time.Second)
|
||||
ref := ProxmoxPhysicalDiskAlertResourceID("lab", "pve1", "/dev/sda")
|
||||
snapshot := proxmoxHistoryIdentitySnapshot(now)
|
||||
snapshot.PhysicalDisks = []models.PhysicalDisk{pveDiskHistoryTestDisk("pve1", "/dev/sda", "ZA1A2B3C", "", now)}
|
||||
adapter := NewMonitorAdapter(NewRegistry(store))
|
||||
adapter.PopulateFromSnapshot(snapshot)
|
||||
diskID, ok := adapter.currentRegistry().ResolveReferenceID(ref)
|
||||
require.True(t, ok)
|
||||
|
||||
change := BuildAlertTimelineChange(ref, ChangeAlertFired, now, "", AlertTimelineChange{
|
||||
AlertIdentifier: ref + "::metric-threshold:diskTemperature", AlertStartedAt: now, AlertType: "diskTemperature",
|
||||
AlertMetadata: map[string]any{"disk_path": "/dev/sda", MetadataDiskSerial: "ZA1A2B3C", MetadataDiskWWN: ""},
|
||||
})
|
||||
change.ID = "temperature"
|
||||
require.NoError(t, adapter.RecordChange(*change))
|
||||
|
||||
require.ElementsMatch(t, []string{"temperature"}, pveDiskHistoryIDs(pveDiskHistoryRows(t, store, diskID)))
|
||||
byRef := pveDiskHistoryRows(t, store, ref)
|
||||
require.ElementsMatch(t, []string{"temperature"}, pveDiskHistoryIDs(byRef))
|
||||
require.Equal(t, ref, OwnedAlertReference(byRef["temperature"]))
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user