mirror of
https://github.com/rcourtman/Pulse.git
synced 2026-10-24 04:33:17 +00:00
Raise Proxmox disk temperature alerts under the per-type policy (#2680)
**Stacked on #2675** (TrueNAS disk temperature alerts by disk type). Merge that first; this branch contains its commit. ## Problem A Proxmox physical disk that only the node's own sensors report never raised a temperature alert: - `CheckDiskHealth` judges health and wearout only; - unified evaluation types TrueNAS physical disks, not PVE ones. So Physical Disks could read a disk **Running Hot** under the per-type thresholds while alerts stayed silent. Whether a disk alerted depended on whether a Pulse agent ran on the node: agent-covered disks alert through `CheckHost` at the same thresholds. #1347 shows users look for disk temperature thresholds in the Proxmox tabs. ## Change - **`CheckProxmoxDiskTemperature`** (`internal/alerts/proxmox_disk_temperature.go`) raises a `diskTemperature` metric alert under the disk's existing PVE alert reference. It is judged by the disk temperature policy: the disk type's `DiskTempByType` entry under the agent Disk Temp default, which switches it off for every type. The alert carries the disk's serial, WWN and `diskType`, like the health and wearout alerts. - **One alert per disk.** `mergeHostAgentSMARTIntoDisks` marks a disk that a still-reporting linked agent lists in its SMART report (`AgentSMARTReported`, internal, never serialized). That disk is the agent's: `CheckHost` alerts on it, and an open PVE alert closes with `moved_to_agent`. A lapsed agent owns nothing, so the node's own current readings keep alerting while its agent is silent. - **Holds and clears.** Only a reading collected this poll is judged (`diskinventory.TemperatureCollected`), so a retained value holds the alert. An excluded device, or a switched-off policy, closes it. - **Stability.** A new `proxmox-disk` alert delay defaults to 5s, so warnings wait the same 300s noisy-gauge window as agent and TrueNAS disk temperatures. Critical readings bypass it. - **Config save.** `reevaluateActiveAlertsLocked` re-judges the alert per disk type. Without this it would fall through to guest thresholds, which have no disk temperature, and resolve on every save. An alert without a recorded type uses the lowest per-type trigger. - **History.** `ProxmoxPhysicalDiskAlertIdentifiers` lists the new `<ref>::metric-threshold:diskTemperature` identifier, so resource history owns its rows by recorded hardware identity, as #2646 does for health and wearout. - **Copy.** The Agents "Disk temperature by type" card says it covers agent, TrueNAS and Proxmox disks. ## Evidence Private mock stack (8 PVE nodes) with a local-only overlay pinning PVE disk readings: | Disk | Physical Disks | Alert | |---|---|---| | pve1 NVMe 63°C | Healthy | none | | pve2 SATA 56°C | Running Hot | warning, trigger 55, after the 300s window | | pve3 NVMe 72°C | Running Hot | warning, trigger 70, after the 300s window | | pve3 SAS 63°C | Healthy | none | | pve3 SATA 66°C | Running Hot | critical at once | The alert card reads: "Samsung 870 EVO 1TB (/dev/nvme2) · Temperature · Temperature 66°C, above the 55°C alert level · Clears once it drops to 50°C or lower and stays there for 5 minutes." ## Tests - `internal/alerts/alerts_test.go`: - the policy (per type, raised trigger); - ownership with `moved_to_agent`, the hold without a reading, exclusion, the off switch; - the stability window; - re-judging on save, including the lowest-trigger hold; - history identifiers. - `internal/monitoring/physical_disk_roundtrip_test.go`: the marker for reporting agents only (not a lapsed one), the SMART fallback path, and that `checkPhysicalDiskAlerts` passes collected readings, ownership and exclusion. - `internal/unifiedresources/store_test.go`: a temperature alert row is owned by its disk and still found by a read by its reference. - `internal/models/deepcopy_test.go`: the marker survives cloning and stays out of JSON. - `ThresholdsTable.test.tsx`: the card copy. - Ten wiring points were mutation-checked (marker, lease condition, collected reading, ownership, exclusion, re-judge branch, lowest trigger, resolution, history identifier); each removal fails a test. ## Second opinion - Fixed: an offline agent's retained SMART rows kept ownership, so neither side alerted while the agent was silent. Ownership now needs a reporting lease. - Kept, documented in `alerts.md`: on the first poll after an agent is newly linked to a disk with an open PVE alert, both alerts can be open until the next disk poll. This is the same window as the node-to-agent usage alert handover. ## Not in this PR - Like the health and wearout alerts, a disk that leaves the inventory keeps its temperature alert until the 24-hour stale sweep. - Per-host Disk Temp overrides do not reach PVE-only disks, which have no agent host.
This commit is contained in:
rcourtman
@@ -248,6 +248,18 @@ Without comparable identity (a SAS, SCSI or USB replacement with no WWN on one
|
||||
side) the row still matches. Report admission and the SMART wire contract are
|
||||
unchanged; the regression is
|
||||
`TestHostAgentSMARTRowForSwappedOutDiskDoesNotLendItsIdentity`.
|
||||
A PVE disk that a still-reporting linked Agent's SMART row matches is marked
|
||||
`AgentSMARTReported` (internal poll evidence on `models.PhysicalDisk`, never
|
||||
serialized); disks the poller builds from that report when the Proxmox disk
|
||||
query fails go through the same merge, and an Agent whose lease lapsed owns
|
||||
none. The Agent then owns the disk's temperature alert
|
||||
through `CheckHost`, and the PVE disk temperature check closes its own as
|
||||
moved to the Agent, so one disk never alerts twice. Report admission, the SMART
|
||||
wire contract and Agent authority are unchanged;
|
||||
`TestMergeHostAgentSMARTIntoDisksMarksDisksTheAgentReports` in
|
||||
`internal/monitoring/physical_disk_roundtrip_test.go` and
|
||||
`TestPhysicalDiskAgentSMARTReportedStaysInternal` in
|
||||
`internal/models/deepcopy_test.go` pin the marker.
|
||||
|
||||
Assistant historical metric wiring uses the current monitor's retained store
|
||||
and registry metrics coordinates. Historical reads do not alter enrollment,
|
||||
|
||||
@@ -1851,14 +1851,84 @@ A disk is hot from its trigger. Alerts and Patrol findings stay open until the
|
||||
reading falls to the clear value, or under the trigger when there is no band
|
||||
below it. The Physical Disks verdict and the TrueNAS Health cell judge only the
|
||||
current reading against the trigger. PDF reports colour disk temperatures by
|
||||
the same thresholds. Judges that still differ: TrueNAS disk temperature alerts
|
||||
use `TrueNASDiskDefaults.Temperature` (a flat 55/50), and the TrueNAS disk
|
||||
drawer tones the reading from a fixed 55C.
|
||||
the same thresholds.
|
||||
`TestDiskTemperatureThresholdMatchesCheckHostPolicy` in
|
||||
`internal/alerts/threshold_resolution_shared_test.go` pins per-type resolution, a raised
|
||||
NVMe trigger that `CheckHost` also honours, the copy, the disabled default and
|
||||
the nil manager.
|
||||
|
||||
TrueNAS disk temperature alerts follow the policy too:
|
||||
`trueNASDiskTemperatureDefaultNoLock`
|
||||
(`internal/alerts/truenas_disk_temperature.go`) is the type-default tier of
|
||||
`effectiveAlertPolicyNoLock` for `truenas-disk`, so an unset
|
||||
`TrueNASDiskDefaults.Temperature` resolves per disk type (the
|
||||
`alertPolicyQuery.DiskType` the unified input carries from
|
||||
`PhysicalDisk.DiskType`). A TrueNAS-wide value the user saved under TrueNAS
|
||||
Disks replaces it for every TrueNAS disk, as a host Disk Temp override does for
|
||||
an agent's disks, 0 meaning off, and a per-disk override beats both. The alert
|
||||
records `diskType` so a config save re-judges it per type; one without it is
|
||||
held to the lowest per-type trigger. A retained reading
|
||||
(`diskinventory.TemperatureCollected` false) is no evidence and raises nothing.
|
||||
The factory value is unset. `normalizeTrueNASDiskTemperature` drops a stored
|
||||
55/50 only from a config without `TrueNASDiskTemperatureByType`, because the
|
||||
thresholds page wrote that flat factory value back on every save; the page
|
||||
writes the marker, so a 55 typed later is kept. Every config `GetConfig`
|
||||
returns carries the marker, so a client that reads and writes back keeps it.
|
||||
Without the marker (an old saved file, or a page loaded before the upgrade)
|
||||
55/50 is read as that unchosen factory value; a client building a config from
|
||||
scratch with an explicit TrueNAS-wide 55 has to send the marker. The TrueNAS
|
||||
Disks Global Defaults cell reads `By type` while unset
|
||||
(`globalDefaultFallbacks` on `ResourceTable`,
|
||||
`resolveAlertResourceGlobalDefaultCell`), or Off when the agent Disk Temp
|
||||
default switches the policy off, and switching it on then stages an explicit
|
||||
TrueNAS-wide 55 because unset would stay off. Each disk row inherits its type's
|
||||
trigger from the unsaved editor state (`resolveTrueNASDiskTemperatureDefault`).
|
||||
The TrueNAS disk drawer tones a current reading from the same per-type trigger
|
||||
(`getDiskTemperatureThresholds` passed into `buildTrueNASDetailSections`), so it
|
||||
no longer judges heat from a fixed 55C. Judges that still differ: the TrueNAS
|
||||
storage table and drawer do not apply a TrueNAS-wide value or per-disk
|
||||
override.
|
||||
`TestTrueNASDiskTemperatureAlertsFollowDiskTemperaturePolicy` and
|
||||
`TestTrueNASDiskTemperatureAlertIgnoresRetainedReading` in
|
||||
`internal/alerts/unified_eval_test.go` pin the tiers and retained readings;
|
||||
`TestTrueNASDiskTemperatureAlertsReevaluatePerDiskType` and
|
||||
`TestTrueNASDiskTemperatureAlertWithoutDiskTypeHeldAtLowestTrigger` in
|
||||
`internal/alerts/alerts_test.go` pin re-judging on a config save;
|
||||
`TestNormalizeTrueNASDiskTemperatureKeepsOnlyChosenValues` pins the
|
||||
saved-config migration.
|
||||
|
||||
Proxmox physical disks alert on temperature under the same policy.
|
||||
`CheckProxmoxDiskTemperature` (`internal/alerts/proxmox_disk_temperature.go`)
|
||||
raises a `diskTemperature` metric alert under the disk's PVE alert reference
|
||||
(`ProxmoxPhysicalDiskAlertResourceID`, resource type `proxmox-disk`), judged by
|
||||
`proxmoxDiskTemperatureThresholdNoLock`: the disk type's `DiskTempByType` entry
|
||||
under the agent Disk Temp default, which switches it off for every type. It
|
||||
carries the disk's serial, WWN and `diskType` like the health and wearout
|
||||
alerts, so `ProxmoxPhysicalDiskAlertIdentifiers` lists its
|
||||
`<reference>::metric-threshold:diskTemperature` identifier for history
|
||||
ownership, and `reevaluateActiveAlertsLocked` re-judges it per disk type (the
|
||||
lowest per-type trigger when the type was never recorded) instead of falling
|
||||
through to guest thresholds that have no disk temperature. A disk a
|
||||
still-reporting linked agent lists in its SMART report is the agent's: its
|
||||
alert closes with the `moved_to_agent` resolution and `CheckHost` raises the
|
||||
agent's own. The handover happens on the next disk poll, so for that one
|
||||
interval an agent newly linked to a disk with an open PVE alert can hold its
|
||||
own alert beside it. A lapsed agent owns nothing, so the node's own readings
|
||||
keep alerting while its agent is silent. An
|
||||
excluded disk, or a policy switched off, closes the alert; a poll without a
|
||||
current reading holds it. Like the health and wearout alerts, a disk that
|
||||
leaves the inventory keeps its alert until the 24-hour stale sweep. The
|
||||
`proxmox-disk` alert delay defaults to the factory 5 seconds, so a warning
|
||||
waits out the same 300-second noisy-gauge stability window agent and TrueNAS
|
||||
disk temperature warnings do, and a critical reading bypasses it.
|
||||
`TestProxmoxDiskTemperatureAlertsFollowDiskTemperaturePolicy`,
|
||||
`TestProxmoxDiskTemperatureAlertOwnershipAndHolds`,
|
||||
`TestProxmoxDiskTemperatureWarningWaitsOutStabilityWindow`,
|
||||
`TestProxmoxDiskTemperatureAlertsReevaluatePerDiskType` and
|
||||
`TestCheckDiskHealthAlertsCarryHistoryOwnershipIdentity` in
|
||||
`internal/alerts/alerts_test.go` pin the policy, ownership, holds, re-judging
|
||||
and identifiers.
|
||||
|
||||
### Agent disk temperature alerts clear when their disk leaves the report
|
||||
|
||||
`CheckHost` also clears a disk temperature alert, and drops its pending
|
||||
|
||||
@@ -2116,6 +2116,18 @@ the same record; the table renders defaults once, in both desktop table and
|
||||
narrow card layouts, and per-section metric columns must resolve to metric
|
||||
keys the shared column normalizer produces so the defaults editor reads and
|
||||
writes the same record keys the section persists.
|
||||
A global default that another setting decides while it is unset passes a
|
||||
`globalDefaultFallbacks` entry for that metric (label, title, whether the other
|
||||
setting is itself off, and the value switching on stages when it is). The
|
||||
TrueNAS Disks temperature uses it to read `By type`, following Disk
|
||||
temperature by type, with `features/alerts/thresholds/trueNASDiskTemperature.ts`
|
||||
giving each disk row its type's trigger from the unsaved editor state.
|
||||
`resolveAlertResourceGlobalDefaultCell` decides the cell for both layouts:
|
||||
unset follows the fallback, switching on from Off returns to unset unless the
|
||||
fallback is off, and an unset default with no fallback stays Off as the engine
|
||||
reads it. `ResourceTable.tsx` hands its props to the shared table state through
|
||||
getters, so the Custom badge and reset control follow defaults that load after
|
||||
the table mounts.
|
||||
Platform sub-routes that add native provider inventory must stay on the shared
|
||||
platform page and table primitives. The vSphere Networks surface routes through
|
||||
`/vmware/networks`, the shared platform tab model, the command palette
|
||||
|
||||
@@ -1825,6 +1825,29 @@ boundary around it:
|
||||
matches node names only, so it relies on fixture node names being unique
|
||||
across instances, which they are.
|
||||
|
||||
`checkPhysicalDiskAlerts` also hands each disk's temperature to
|
||||
`CheckProxmoxDiskTemperature`, so a Proxmox disk that only the node's sensors
|
||||
report alerts under the disk temperature policy, as its Running Hot verdict
|
||||
already says it should. It passes the reading only when this poll collected it
|
||||
(`collectedPhysicalDiskTemperature`, `diskinventory.TemperatureCollected`), so
|
||||
a retained last-known value holds the alert rather than judging it, and it
|
||||
passes `AgentSMARTReported`, which `mergeHostAgentSMARTIntoDisks` sets on a
|
||||
disk a still-reporting linked agent lists in its SMART report (disks built
|
||||
from that report when the Proxmox query fails go through the same merge). A
|
||||
disk the agent lists is the agent's: `CheckHost` raises its temperature
|
||||
alert, and the PVE disk check closes its own as moved to the agent, so no
|
||||
disk alerts twice. Once the agent's lease lapses (`ExpireHostTelemetry` marks
|
||||
the host offline) its retained rows still enrich the disk, but ownership
|
||||
returns to the PVE check, which judges the node's own current readings; an
|
||||
agent newly linked to a disk with an open PVE alert takes it over on the next
|
||||
disk poll. An excluded device closes its temperature
|
||||
alert with the health and wearout ones. Fixture disks carry no agent SMART
|
||||
merge, so on a mock estate every hot disk on an online node alerts.
|
||||
`TestMergeHostAgentSMARTIntoDisksMarksDisksTheAgentReports` and
|
||||
`TestCheckPhysicalDiskAlertsRaisesProxmoxDiskTemperatureAlerts` in
|
||||
`internal/monitoring/physical_disk_roundtrip_test.go` pin the marker,
|
||||
the collected reading, agent ownership and exclusion.
|
||||
|
||||
Switching mock mode fences the alert evaluations that read mode-dependent data
|
||||
(`mockModeFence`, `internal/monitoring/mock_mode_fence.go`). `GetState`, the
|
||||
fixture graph, the unified read view, the recovery rollups and the connection
|
||||
|
||||
@@ -6076,7 +6076,12 @@ the reference and is never retried. An owned row records its alert's
|
||||
reference as `alert_resource_id`. A read by the reference returns the rows
|
||||
journaled under it plus the rows owned away from it, matched through the
|
||||
indexed alert identifier (`ProxmoxPhysicalDiskAlertIdentifiers`) and that
|
||||
recorded reference, and every count uses the same predicate. The Alerts
|
||||
recorded reference, and every count uses the same predicate. Those
|
||||
identifiers cover the health and wearout specs and the disk temperature
|
||||
metric spec (`<reference>::metric-threshold:diskTemperature`), since PVE disks
|
||||
also raise temperature alerts under the same reference
|
||||
(`TestProxmoxDiskTemperatureAlertRowsFollowRecordedHardwareIdentity` in
|
||||
`internal/unifiedresources/store_test.go`). The Alerts
|
||||
history Resource action therefore still lists every occurrence raised under
|
||||
the path, across the disks that held it, and alert-centric reads keep their
|
||||
occurrences (see the AI runtime contract's incident-history queries). Rows
|
||||
@@ -6232,6 +6237,13 @@ presentation. `frontend-modern/src/components/Infrastructure/`
|
||||
placement, signal, and snapshot context through the canonical resource drawer
|
||||
and debug/source sections rather than introducing a VMware-only detail route,
|
||||
drawer tab, or provider-local investigation shell.
|
||||
The TrueNAS physical-disk drawer judges a current temperature reading by the
|
||||
disk temperature policy, like the TrueNAS storage table:
|
||||
`useResourceDetailDrawerDerivedState.ts` passes the alerts store's
|
||||
`getDiskTemperatureThresholds` into `buildTrueNASDetailSections`, and the
|
||||
Temperature row takes a warning tone only when `isPhysicalDiskRunningHot`
|
||||
says the reading reached its disk type's alert trigger. A retained reading
|
||||
keeps its muted last-known row, and without a resolver heat is not judged.
|
||||
That same infrastructure consumer boundary also owns source selection
|
||||
continuity. Settings infrastructure panels and platform/runtime pages must
|
||||
keep canonical sources such as `truenas` and `availability` present in their
|
||||
|
||||
@@ -0,0 +1,69 @@
|
||||
{
|
||||
"version": 1,
|
||||
"base_sha": "bdf0f76b7bf61d550b1cb8574497eef75b354acb",
|
||||
"verified_at": "2026-10-07T14:11:27Z",
|
||||
"result": "passed",
|
||||
"changed_paths": [
|
||||
"frontend-modern/src/components/Alerts/AlertResourceTableDesktop.tsx",
|
||||
"frontend-modern/src/components/Alerts/AlertResourceTableMobile.tsx",
|
||||
"frontend-modern/src/components/Alerts/ResourceTable.tsx",
|
||||
"frontend-modern/src/components/Alerts/ThresholdsTableAgentsResourcesSection.tsx",
|
||||
"frontend-modern/src/components/Alerts/ThresholdsTableTrueNASTab.tsx",
|
||||
"frontend-modern/src/components/Alerts/alertResourceTableModel.ts",
|
||||
"frontend-modern/src/components/Infrastructure/resourceDetailDrawerTrueNASModel.ts",
|
||||
"frontend-modern/src/components/Infrastructure/useResourceDetailDrawerDerivedState.ts",
|
||||
"frontend-modern/src/features/alerts/alertsConfigurationModel.ts",
|
||||
"frontend-modern/src/features/alerts/thresholds/hooks/useThresholdsPlatformData.ts",
|
||||
"frontend-modern/src/features/alerts/thresholds/trueNASDiskTemperature.ts",
|
||||
"frontend-modern/src/types/alerts.ts",
|
||||
"frontend-modern/src/utils/alertThresholdDefaults.ts"
|
||||
],
|
||||
"content_sha256": {
|
||||
"frontend-modern/src/components/Alerts/AlertResourceTableDesktop.tsx": "4d09dd97adc2868889e256455c207b6b9089ea311713878f0d6ff00efb6b7123",
|
||||
"frontend-modern/src/components/Alerts/AlertResourceTableMobile.tsx": "ceaf136da9361a5b1756350626a833da4dfebcb4f211c7269d7d89d37fb7e085",
|
||||
"frontend-modern/src/components/Alerts/ResourceTable.tsx": "7e80a6c789b3069fcadcb86e0a6689e5319e7b0ac5d5bfa1a596520daec2d7e1",
|
||||
"frontend-modern/src/components/Alerts/ThresholdsTableAgentsResourcesSection.tsx": "fc4e9f4d05c0bf5799acbbe64c37b901e104869ff120819342432563789ad0e9",
|
||||
"frontend-modern/src/components/Alerts/ThresholdsTableTrueNASTab.tsx": "a22b86c28b65ec18ba81171fb1206a24107a0d97db19d7dd488bdfd0a689a956",
|
||||
"frontend-modern/src/components/Alerts/alertResourceTableModel.ts": "5881aeaf7ee575589997bf67fb713a01784ce7ca16ecb5a5eaea77f36010837a",
|
||||
"frontend-modern/src/components/Infrastructure/resourceDetailDrawerTrueNASModel.ts": "b60ba59688d6ea0cabfb2b413b8adcaa85c8d41524c1d1f0df9bdb9e6d85733c",
|
||||
"frontend-modern/src/components/Infrastructure/useResourceDetailDrawerDerivedState.ts": "63b07de3f354f8328298348a8892ad73fde2f11ec6768e9c0f6fc37eb4c43516",
|
||||
"frontend-modern/src/features/alerts/alertsConfigurationModel.ts": "0e116e65c992a72e21d700835b0cb0db7173be9ec9c4980382958506d4e75976",
|
||||
"frontend-modern/src/features/alerts/thresholds/hooks/useThresholdsPlatformData.ts": "8525b0b012198ff87e29063908a9907b7fa1080cd84187202b1cd3cf198a3b46",
|
||||
"frontend-modern/src/features/alerts/thresholds/trueNASDiskTemperature.ts": "80433d42dd4658a1449c0f0c378d7fa52cd6dfab117d626195099768e750352f",
|
||||
"frontend-modern/src/types/alerts.ts": "d072175ab98e3785eae643d655be4c2d54cac0eddc8b55803a2841a312062e5a",
|
||||
"frontend-modern/src/utils/alertThresholdDefaults.ts": "077e291d5c224063a373a5e9f5555d9b470f585801b0cfdd9dc1acacf233af28"
|
||||
},
|
||||
"routes": [
|
||||
"/alerts/thresholds/truenas",
|
||||
"/truenas/storage"
|
||||
],
|
||||
"viewports": [
|
||||
{
|
||||
"width": 1366,
|
||||
"height": 900
|
||||
},
|
||||
{
|
||||
"width": 375,
|
||||
"height": 812
|
||||
}
|
||||
],
|
||||
"states": [
|
||||
"Private mock stack (8 nodes, mock TrueNAS truenas-main) with a local-only generator overlay pinning TrueNAS disk temperatures: nvme0n1 60C, nvme1n1 72C, sda 57C (SATA), sdc 63C (SAS); after = this branch, before = main logic in the same tree",
|
||||
"Thresholds > TrueNAS > Disks with no saved TrueNAS-wide value: header reads 'Defaults: NVMe 70\u00b0C \u00b7 SAS 65\u00b0C \u00b7 SATA 55\u00b0C', note 'Unless set here, each disk alerts at Disk temperature by type, set under Agents.', Global Defaults Temp cell empty with placeholder 'By type' (fits after widening to w-20) and On, disk rows nvme0n1/nvme1n1 70\u00b0C, sda/sdb 55\u00b0C, sdc-sdf 65\u00b0C",
|
||||
"Same section at 375px: Global Defaults card shows TEMP \u00b0C 'By type' with On, disk cards show their type triggers, no horizontal overflow",
|
||||
"Agent Disk Temp default switched off through the API: header 'Defaults: all metric alerts off', Global Defaults cell Off with title 'Off because the agent Disk Temp default under Agents is off...', disk rows Off",
|
||||
"Saved TrueNAS-wide 62: header 'Defaults: Temp 62\u00b0C', every disk row 62\u00b0C, Custom badge and reset control after reload; saved 55 survives save and reload as 55 (stored with truenasDiskTemperatureByType)",
|
||||
"Alerts with factory per-type thresholds: after stack raises nvme1n1 72 (trigger 70) and sda 57 (trigger 55) with diskType metadata, none for nvme0n1 60 or sdc 63; before (main) raised all four at a flat 55",
|
||||
"TrueNAS storage table agrees: Attention 'Disk temperature is 72\u00b0C, at or above its 70\u00b0C alert threshold.' on nvme1n1 and '57\u00b0C ... 55\u00b0C' on sda, none on nvme0n1",
|
||||
"TrueNAS disk drawer (Platform details > Show TrueNAS > Health): nvme0n1 Temperature 60\u00b0C in default text colour (main toned it amber from the fixed 55C); sda 57\u00b0C amber rgb(252,211,77) at 1366 and at 375 with no overflow"
|
||||
],
|
||||
"interactions": [
|
||||
"Typed 62 into the TrueNAS Disks Global Defaults Temp cell, saved, reloaded: API truenasDiskDefaults {trigger 62, clear 57}, cell shows 62 with Custom",
|
||||
"Triple-clicked and typed 55, saved, reloaded: API keeps {55, 50}, cell shows 55 with Custom and the reset control",
|
||||
"Clicked Reset to factory defaults: cell returns to 'By type'; saved: API truenasDiskDefaults {}",
|
||||
"On the 375px card, clicked On to switch the default Off, saved: API stores {trigger 0, clear 0} and the after stack's TrueNAS disk alerts resolve; clicked again: returns to 'By type', saved: {}",
|
||||
"With the agent Disk Temp default off, clicked the Off badge: stages an explicit 55 (Custom, rows 55\u00b0C); discarded",
|
||||
"Loaded an old saved config (flat 55/50, no marker) with the new binary: GET returns truenasDiskDefaults {} and truenasDiskTemperatureByType true",
|
||||
"Expanded nvme0n1 and sda rows on /truenas/storage, opened Platform details and Show TrueNAS, read the Temperature row value and computed colour at 1366 and 375"
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,39 @@
|
||||
{
|
||||
"version": 1,
|
||||
"base_sha": "8e1a564d7e229bda51c021c7e6bde920e0ca0596",
|
||||
"verified_at": "2026-10-07T14:35:42Z",
|
||||
"result": "passed",
|
||||
"changed_paths": [
|
||||
"frontend-modern/src/components/Alerts/ThresholdsTableAgentsResourcesSection.tsx"
|
||||
],
|
||||
"content_sha256": {
|
||||
"frontend-modern/src/components/Alerts/ThresholdsTableAgentsResourcesSection.tsx": "3052a190a76e885e98ed6601b3e83c8959e1c656d12270b87e074fffa95860ac"
|
||||
},
|
||||
"routes": [
|
||||
"/alerts/thresholds/systems",
|
||||
"/alerts/overview",
|
||||
"/proxmox/storage?tab=disks"
|
||||
],
|
||||
"viewports": [
|
||||
{
|
||||
"width": 1366,
|
||||
"height": 900
|
||||
},
|
||||
{
|
||||
"width": 375,
|
||||
"height": 812
|
||||
}
|
||||
],
|
||||
"states": [
|
||||
"Private mock stack (8 PVE nodes) on this branch with a local-only generator overlay pinning PVE disks: pve1 NVMe 63C, pve2 SATA 56C, pve3 NVMe 72C, pve3 SAS 63C, pve3 SATA 66C",
|
||||
"Alerts > Thresholds > Systems (Agents section): 'Disk temperature by type' card reads 'Alert trigger in \u00b0C for each disk type, for agent, TrueNAS and Proxmox disks. ... a TrueNAS Disks value replaces them for TrueNAS disks.' at 1366, and at 375 the card spans 34-341px with no horizontal overflow",
|
||||
"Alerts overview: 'Samsung 870 EVO 1TB (/dev/nvme2) (Temperature) CRITICAL on West Production C \u2014 Temperature 66\u00b0C, above the 55\u00b0C alert level; Clears once it drops to 50\u00b0C or lower and stays there for 5 minutes', Alert level 55\u00b0C, Clear level 50\u00b0C",
|
||||
"Warning-level PVE disk temperature alerts (SATA 56C trigger 55, NVMe 72C trigger 70) appear after the 300s stability window (first breach 15:26:32, firing by 15:31:46); the critical 66C fires at once; none for NVMe 63C or SAS 63C",
|
||||
"Physical Disks filtered to West Production: exactly the three alerting disks read Running Hot (Crucial MX500 56\u00b0C, WD Black SN850X 72\u00b0C, Samsung 870 EVO 66\u00b0C); the 870 EVO drawer reads 'Running Hot \u2014 Disk temperature is 66\u00b0C, at or above its 55\u00b0C alert threshold.'"
|
||||
],
|
||||
"interactions": [
|
||||
"Opened Alerts overview and read the PVE disk temperature alert card",
|
||||
"Filtered Physical Disks by 'West Production C', expanded the Samsung 870 EVO row",
|
||||
"Opened Thresholds > Systems, read the Disk temperature by type card at 1366 and at 375"
|
||||
]
|
||||
}
|
||||
@@ -43,12 +43,11 @@ import {
|
||||
ALERT_RESOURCE_METRIC_OFF_VALUE,
|
||||
getAlertResourceColumnKind,
|
||||
getAlertResourceColumnHeaderTooltip,
|
||||
getAlertResourceEnabledDefault,
|
||||
getAlertResourceMetricBounds,
|
||||
getAlertResourceMetricDelayOverride,
|
||||
getAlertResourceMetricStep,
|
||||
isAlertResourceMetricOff,
|
||||
normalizeAlertResourceMetricKey,
|
||||
resolveAlertResourceGlobalDefaultCell,
|
||||
} from './alertResourceTableModel';
|
||||
import type { OfflineState, Resource, ResourceTableProps } from './ResourceTable';
|
||||
|
||||
@@ -210,7 +209,10 @@ export function AlertResourceTableDesktop(props: AlertResourceTableDesktopProps)
|
||||
{(column) => (
|
||||
<TableHead
|
||||
class={`${getPlatformTableHeadClassForKind(getAlertResourceColumnKind(column))} whitespace-normal wrap-break-word`}
|
||||
title={getAlertResourceColumnHeaderTooltip(column)}
|
||||
title={
|
||||
props.table.columnTooltips?.[column] ??
|
||||
getAlertResourceColumnHeaderTooltip(column)
|
||||
}
|
||||
>
|
||||
{column}
|
||||
</TableHead>
|
||||
@@ -281,10 +283,14 @@ export function AlertResourceTableDesktop(props: AlertResourceTableDesktopProps)
|
||||
{(column) => {
|
||||
const metric = normalizeAlertResourceMetricKey(column);
|
||||
const bounds = getAlertResourceMetricBounds(metric);
|
||||
const value = () => props.table.globalDefaults?.[metric] ?? 0;
|
||||
const value = () => props.table.globalDefaults?.[metric];
|
||||
const fallback = () => props.table.globalDefaultFallbacks?.[metric];
|
||||
// An unset default and a stored 0 are both disabled to the
|
||||
// alert engine, so neither may render as On.
|
||||
const isOff = () => isAlertResourceMetricOff(value());
|
||||
// alert engine, so neither may render as On, unless the
|
||||
// metric's unset default follows another setting.
|
||||
const cell = () =>
|
||||
resolveAlertResourceGlobalDefaultCell(metric, value(), fallback());
|
||||
const isOff = () => cell().isOff;
|
||||
|
||||
return (
|
||||
<TableCell
|
||||
@@ -297,8 +303,12 @@ export function AlertResourceTableDesktop(props: AlertResourceTableDesktopProps)
|
||||
min={bounds.min}
|
||||
max={bounds.max}
|
||||
step={getAlertResourceMetricStep(metric)}
|
||||
value={isOff() ? '' : value()}
|
||||
placeholder={getAlertResourceTableMetricPlaceholder(isOff())}
|
||||
value={isOff() || cell().follows ? '' : (value() ?? '')}
|
||||
placeholder={
|
||||
cell().follows && !isOff()
|
||||
? fallback()?.label
|
||||
: getAlertResourceTableMetricPlaceholder(isOff())
|
||||
}
|
||||
disabled={isOff()}
|
||||
onInput={(e) => {
|
||||
const nextValue = parseFloat(e.currentTarget.value);
|
||||
@@ -318,12 +328,16 @@ export function AlertResourceTableDesktop(props: AlertResourceTableDesktopProps)
|
||||
}));
|
||||
props.table.setHasUnsavedChanges?.(true);
|
||||
}}
|
||||
class={`w-16 px-2 py-0.5 text-sm text-center border rounded ${
|
||||
class={`${cell().follows && !isOff() ? 'w-20' : 'w-16'} px-2 py-0.5 text-sm text-center border rounded ${
|
||||
isOff()
|
||||
? 'border-border bg-surface-alt text-muted italic placeholder: dark:placeholder: placeholder:opacity-60 pointer-events-none'
|
||||
: 'border-border text-base-content focus:border-blue-500 focus:ring-1 focus:ring-blue-500'
|
||||
}`}
|
||||
title={getAlertResourceTableMetricInputTitle(isOff())}
|
||||
title={
|
||||
cell().follows
|
||||
? fallback()?.title
|
||||
: getAlertResourceTableMetricInputTitle(isOff())
|
||||
}
|
||||
/>
|
||||
<Show when={isOff()}>
|
||||
<button
|
||||
@@ -332,7 +346,7 @@ export function AlertResourceTableDesktop(props: AlertResourceTableDesktopProps)
|
||||
onClick={() => {
|
||||
props.table.setGlobalDefaults?.((prev) => ({
|
||||
...prev,
|
||||
[metric]: getAlertResourceEnabledDefault(metric),
|
||||
[metric]: cell().enableValue,
|
||||
}));
|
||||
props.table.setHasUnsavedChanges?.(true);
|
||||
}}
|
||||
@@ -348,7 +362,7 @@ export function AlertResourceTableDesktop(props: AlertResourceTableDesktopProps)
|
||||
props.table.setGlobalDefaults?.((prev) => ({
|
||||
...prev,
|
||||
[metric]: isOff()
|
||||
? getAlertResourceEnabledDefault(metric)
|
||||
? cell().enableValue
|
||||
: ALERT_RESOURCE_METRIC_OFF_VALUE,
|
||||
}));
|
||||
props.table.setHasUnsavedChanges?.(true);
|
||||
|
||||
@@ -26,7 +26,6 @@ import {
|
||||
ALERT_RESOURCE_METRIC_OFF_VALUE,
|
||||
alertResourceSupportsMetric,
|
||||
buildAlertResourceEditPayload,
|
||||
getAlertResourceEnabledDefault,
|
||||
getAlertResourceLabel,
|
||||
getAlertResourceMetricBounds,
|
||||
getAlertResourceMetricDisplayValue,
|
||||
@@ -34,6 +33,7 @@ import {
|
||||
isAlertResourceMetricOff,
|
||||
isAlertResourceMetricOverridden,
|
||||
normalizeAlertResourceMetricKey,
|
||||
resolveAlertResourceGlobalDefaultCell,
|
||||
resolveAlertResourceMetricEnableValue,
|
||||
} from './alertResourceTableModel';
|
||||
import type { Resource } from '@/features/alerts/thresholds/tableTypes';
|
||||
@@ -212,10 +212,14 @@ export function AlertResourceTableMobile(props: AlertResourceTableMobileProps) {
|
||||
{(column) => {
|
||||
const metric = normalizeAlertResourceMetricKey(column);
|
||||
const bounds = getAlertResourceMetricBounds(metric);
|
||||
const value = () => props.table.globalDefaults?.[metric] ?? 0;
|
||||
const value = () => props.table.globalDefaults?.[metric];
|
||||
const fallback = () => props.table.globalDefaultFallbacks?.[metric];
|
||||
// An unset default and a stored 0 are both disabled to the alert
|
||||
// engine, so neither may render as On.
|
||||
const isOff = () => isAlertResourceMetricOff(value());
|
||||
// engine, so neither may render as On, unless the metric's unset
|
||||
// default follows another setting.
|
||||
const cell = () =>
|
||||
resolveAlertResourceGlobalDefaultCell(metric, value(), fallback());
|
||||
const isOff = () => cell().isOff;
|
||||
|
||||
return (
|
||||
<div class="p-2 bg-surface rounded-sm border border-border-subtle flex flex-col gap-1">
|
||||
@@ -227,8 +231,13 @@ export function AlertResourceTableMobile(props: AlertResourceTableMobileProps) {
|
||||
min={bounds.min}
|
||||
max={bounds.max}
|
||||
step={getAlertResourceMetricStep(metric)}
|
||||
value={isOff() ? '' : value()}
|
||||
placeholder={getAlertResourceTableMetricPlaceholder(isOff())}
|
||||
value={isOff() || cell().follows ? '' : (value() ?? '')}
|
||||
placeholder={
|
||||
cell().follows && !isOff()
|
||||
? fallback()?.label
|
||||
: getAlertResourceTableMetricPlaceholder(isOff())
|
||||
}
|
||||
title={cell().follows ? fallback()?.title : undefined}
|
||||
disabled={isOff()}
|
||||
class={`min-h-11 w-full rounded-sm border p-1 text-center text-sm sm:min-h-0 ${isOff() ? 'bg-surface-hover' : ' border-border'}`}
|
||||
onInput={(e) => {
|
||||
@@ -257,7 +266,7 @@ export function AlertResourceTableMobile(props: AlertResourceTableMobileProps) {
|
||||
onClick={() => {
|
||||
props.table.setGlobalDefaults?.((prev) => ({
|
||||
...prev,
|
||||
[metric]: getAlertResourceEnabledDefault(metric),
|
||||
[metric]: cell().enableValue,
|
||||
}));
|
||||
props.table.setHasUnsavedChanges?.(true);
|
||||
}}
|
||||
@@ -271,7 +280,7 @@ export function AlertResourceTableMobile(props: AlertResourceTableMobileProps) {
|
||||
props.table.setGlobalDefaults?.((prev) => ({
|
||||
...prev,
|
||||
[metric]: isOff()
|
||||
? getAlertResourceEnabledDefault(metric)
|
||||
? cell().enableValue
|
||||
: ALERT_RESOURCE_METRIC_OFF_VALUE,
|
||||
}));
|
||||
props.table.setHasUnsavedChanges?.(true);
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest';
|
||||
import { createSignal } from 'solid-js';
|
||||
import { cleanup, fireEvent, render, screen } from '@solidjs/testing-library';
|
||||
import { ALERT_BULK_EDIT_CLEAR_LABEL } from '@/utils/alertBulkEditPresentation';
|
||||
import resourceTableSource from '@/components/Alerts/ResourceTable.tsx?raw';
|
||||
@@ -1007,6 +1008,144 @@ describe('ResourceTable', () => {
|
||||
expect(updater({ cpu: 0 })).toEqual({ cpu: 80 });
|
||||
});
|
||||
|
||||
it.each([
|
||||
['desktop', false],
|
||||
['mobile', true],
|
||||
])('lets an unset %s global default follow its fallback', (_layout, mobile) => {
|
||||
mockIsMobile.mockReturnValue(mobile);
|
||||
const setGlobalDefaults = vi.fn();
|
||||
const props = makeProps({
|
||||
resources: [],
|
||||
columns: ['Temp °C'],
|
||||
globalDefaults: { temperature: undefined },
|
||||
globalDefaultFallbacks: {
|
||||
temperature: { label: 'By type', title: 'Follows Disk temperature by type' },
|
||||
},
|
||||
setGlobalDefaults,
|
||||
setHasUnsavedChanges: vi.fn(),
|
||||
});
|
||||
render(() => <ResourceTable {...props} />);
|
||||
|
||||
// Unset reads On and names where the value comes from.
|
||||
expect(screen.getByTestId('status-badge')).toHaveTextContent('On');
|
||||
const input = screen.getByPlaceholderText('By type') as HTMLInputElement;
|
||||
expect(input.value).toBe('');
|
||||
expect(input.disabled).toBe(false);
|
||||
expect(input.title).toBe('Follows Disk temperature by type');
|
||||
|
||||
fireEvent.click(screen.getByTestId('status-badge'));
|
||||
const updater = setGlobalDefaults.mock.calls[0][0] as (
|
||||
prev: Record<string, number | undefined>,
|
||||
) => Record<string, number | undefined>;
|
||||
expect(updater({ temperature: undefined })).toEqual({ temperature: -1 });
|
||||
});
|
||||
|
||||
it.each([
|
||||
['desktop', false],
|
||||
['mobile', true],
|
||||
])('switches an Off %s fallback default back to following', (_layout, mobile) => {
|
||||
mockIsMobile.mockReturnValue(mobile);
|
||||
const setGlobalDefaults = vi.fn();
|
||||
const props = makeProps({
|
||||
resources: [],
|
||||
columns: ['Temp °C'],
|
||||
globalDefaults: { temperature: -1 },
|
||||
globalDefaultFallbacks: {
|
||||
temperature: { label: 'By type', title: 'Follows Disk temperature by type' },
|
||||
},
|
||||
setGlobalDefaults,
|
||||
setHasUnsavedChanges: vi.fn(),
|
||||
});
|
||||
render(() => <ResourceTable {...props} />);
|
||||
|
||||
expect(screen.getByTestId('status-badge')).toHaveTextContent('Off');
|
||||
fireEvent.click(screen.getByTestId('status-badge'));
|
||||
const updater = setGlobalDefaults.mock.calls[0][0] as (
|
||||
prev: Record<string, number | undefined>,
|
||||
) => Record<string, number | undefined>;
|
||||
expect(updater({ temperature: -1 })).toEqual({ temperature: undefined });
|
||||
});
|
||||
|
||||
it.each([
|
||||
['desktop', false],
|
||||
['mobile', true],
|
||||
])('reads an unset %s default Off when its fallback is off', (_layout, mobile) => {
|
||||
mockIsMobile.mockReturnValue(mobile);
|
||||
const setGlobalDefaults = vi.fn();
|
||||
const props = makeProps({
|
||||
resources: [],
|
||||
columns: ['Temp °C'],
|
||||
globalDefaults: { temperature: undefined },
|
||||
globalDefaultFallbacks: {
|
||||
temperature: { label: 'By type', title: 'Off by type', off: true, enableValue: 55 },
|
||||
},
|
||||
setGlobalDefaults,
|
||||
setHasUnsavedChanges: vi.fn(),
|
||||
});
|
||||
render(() => <ResourceTable {...props} />);
|
||||
|
||||
expect(screen.getByTestId('status-badge')).toHaveTextContent('Off');
|
||||
expect(screen.queryByPlaceholderText('By type')).toBeNull();
|
||||
|
||||
// Unset would still be off, so switching on stages an explicit value.
|
||||
fireEvent.click(screen.getByTestId('status-badge'));
|
||||
const updater = setGlobalDefaults.mock.calls[0][0] as (
|
||||
prev: Record<string, number | undefined>,
|
||||
) => Record<string, number | undefined>;
|
||||
expect(updater({ temperature: undefined })).toEqual({ temperature: 55 });
|
||||
});
|
||||
|
||||
it('shows a set fallback global default as its number', () => {
|
||||
const props = makeProps({
|
||||
resources: [],
|
||||
columns: ['Temp °C'],
|
||||
globalDefaults: { temperature: 62 },
|
||||
globalDefaultFallbacks: {
|
||||
temperature: { label: 'By type', title: 'Follows Disk temperature by type' },
|
||||
},
|
||||
setGlobalDefaults: vi.fn(),
|
||||
setHasUnsavedChanges: vi.fn(),
|
||||
});
|
||||
render(() => <ResourceTable {...props} />);
|
||||
|
||||
expect(screen.getByDisplayValue('62')).toBeInTheDocument();
|
||||
expect(screen.queryByPlaceholderText('By type')).toBeNull();
|
||||
});
|
||||
|
||||
it('follows global defaults that arrive after the table mounts', () => {
|
||||
const [globalDefaults, setGlobalDefaults] = createSignal<Record<string, number | undefined>>({
|
||||
temperature: undefined,
|
||||
});
|
||||
const props = makeProps({
|
||||
resources: [],
|
||||
columns: ['Temp °C'],
|
||||
factoryDefaults: { temperature: undefined },
|
||||
onResetDefaults: vi.fn(),
|
||||
setGlobalDefaults: vi.fn(),
|
||||
setHasUnsavedChanges: vi.fn(),
|
||||
});
|
||||
render(() => <ResourceTable {...props} globalDefaults={globalDefaults()} />);
|
||||
|
||||
expect(screen.queryByText('Custom')).toBeNull();
|
||||
// The saved config loads after mount.
|
||||
setGlobalDefaults({ temperature: 62 });
|
||||
expect(screen.getByText('Custom')).toBeInTheDocument();
|
||||
});
|
||||
|
||||
it('uses a table column tooltip over the generic one', () => {
|
||||
const props = makeProps({
|
||||
resources: [],
|
||||
columns: ['Temp °C'],
|
||||
columnTooltips: { 'Temp °C': 'Disk temperature that raises an alert.' },
|
||||
});
|
||||
render(() => <ResourceTable {...props} />);
|
||||
|
||||
expect(screen.getByText('Temp °C', { selector: 'th' })).toHaveAttribute(
|
||||
'title',
|
||||
'Disk temperature that raises an alert.',
|
||||
);
|
||||
});
|
||||
|
||||
it('stages an explicit threshold from the mobile row editor', () => {
|
||||
mockIsMobile.mockReturnValue(true);
|
||||
const setEditingThresholds = vi.fn();
|
||||
|
||||
@@ -11,6 +11,7 @@ import { ActionIconButton } from '@/components/shared/Button';
|
||||
import { useObservedElementWidth } from '@/hooks/useObservedElementWidth';
|
||||
import { useAlertResourceTableState } from './useAlertResourceTableState';
|
||||
import type { GroupHeaderMeta, Resource } from '@/features/alerts/thresholds/tableTypes';
|
||||
import type { AlertResourceGlobalDefaultFallback } from './alertResourceTableModel';
|
||||
import { AlertResourceTableDesktop } from './AlertResourceTableDesktop';
|
||||
import { AlertResourceTableMobile } from './AlertResourceTableMobile';
|
||||
|
||||
@@ -49,6 +50,10 @@ export interface ResourceTableProps {
|
||||
formatMetricValue: (metric: string, value: number | undefined) => string;
|
||||
hasActiveAlert: (resourceId: string, metric: string) => boolean;
|
||||
globalDefaults?: Record<string, number | undefined>;
|
||||
/** Metrics whose unset global default follows another setting. */
|
||||
globalDefaultFallbacks?: Record<string, AlertResourceGlobalDefaultFallback>;
|
||||
/** Header tooltips that replace the generic one for a column label. */
|
||||
columnTooltips?: Record<string, string>;
|
||||
setGlobalDefaults?: (
|
||||
value:
|
||||
| Record<string, number | undefined>
|
||||
@@ -97,10 +102,20 @@ export function ResourceTable(props: ResourceTableProps) {
|
||||
someSelected,
|
||||
clearSelectedIds,
|
||||
} = useAlertResourceTableState({
|
||||
resources: props.resources,
|
||||
groupedResources: props.groupedResources,
|
||||
globalDefaults: props.globalDefaults,
|
||||
factoryDefaults: props.factoryDefaults,
|
||||
// Getters keep the state reactive: saved defaults arrive after mount, and
|
||||
// the Custom badge and reset control must follow them.
|
||||
get resources() {
|
||||
return props.resources;
|
||||
},
|
||||
get groupedResources() {
|
||||
return props.groupedResources;
|
||||
},
|
||||
get globalDefaults() {
|
||||
return props.globalDefaults;
|
||||
},
|
||||
get factoryDefaults() {
|
||||
return props.factoryDefaults;
|
||||
},
|
||||
});
|
||||
const totalColumnCount = createMemo(
|
||||
() =>
|
||||
|
||||
@@ -82,9 +82,10 @@ export function ThresholdsTableAgentsResourcesSection(props: ThresholdsTableSect
|
||||
<Card padding="md" tone="card" class="mt-4">
|
||||
<h3 class="text-sm font-semibold text-base-content">Disk temperature by type</h3>
|
||||
<p class="mt-1 text-xs text-muted">
|
||||
Alert trigger in °C for each disk type. Warning colors start 5°C below the trigger.
|
||||
Setting a Disk Temp override on a host above replaces these for all of that host's
|
||||
disks.
|
||||
Alert trigger in °C for each disk type, for agent, TrueNAS and Proxmox disks. Warning
|
||||
colors start 5°C below the trigger. A Disk Temp override on a host above replaces
|
||||
these for all of that host's disks, and a TrueNAS Disks value replaces them for
|
||||
TrueNAS disks.
|
||||
</p>
|
||||
<div class="mt-4 grid gap-4 sm:grid-cols-3">
|
||||
<For each={DISK_TEMP_TYPE_FIELDS}>
|
||||
|
||||
@@ -8,6 +8,9 @@ import { CollapsibleSection } from './Thresholds/sections/CollapsibleSection';
|
||||
import { formatMetricValue } from '@/features/alerts/thresholds/helpers';
|
||||
import type { ThresholdsTableSectionProps } from '@/features/alerts/thresholds/thresholdsTableSectionProps';
|
||||
import type { Resource } from '@/features/alerts/thresholds/tableTypes';
|
||||
import { getDiskTemperatureByTypeItems } from '@/features/alerts/thresholds/trueNASDiskTemperature';
|
||||
import { FACTORY_AGENT_DEFAULTS } from '@/utils/alertThresholdDefaults';
|
||||
import type { AlertResourceGlobalDefaultFallback } from './alertResourceTableModel';
|
||||
import { getAlertThresholdsDefaultsSummary } from '@/utils/alertThresholdsSectionPresentation';
|
||||
|
||||
const TRUENAS_SYSTEM_COLUMNS = [
|
||||
@@ -22,6 +25,7 @@ const TRUENAS_SYSTEM_COLUMNS = [
|
||||
];
|
||||
const TRUENAS_STORAGE_COLUMNS = ['Usage %'];
|
||||
const TRUENAS_DISK_COLUMNS = ['Temp °C'];
|
||||
const TRUENAS_DISK_COLUMN_TOOLTIPS = { 'Temp °C': 'Disk temperature that raises an alert.' };
|
||||
|
||||
function TrueNASResourceSection(
|
||||
props: ThresholdsTableSectionProps & {
|
||||
@@ -39,6 +43,10 @@ function TrueNASResourceSection(
|
||||
| ((prev: Record<string, number | undefined>) => Record<string, number | undefined>),
|
||||
) => void;
|
||||
onResetDefaults?: () => void;
|
||||
defaultsSummary?: string;
|
||||
globalDefaultFallbacks?: Record<string, AlertResourceGlobalDefaultFallback>;
|
||||
columnTooltips?: Record<string, string>;
|
||||
note?: string;
|
||||
},
|
||||
) {
|
||||
const { state, tableProps } = props;
|
||||
@@ -48,7 +56,9 @@ function TrueNASResourceSection(
|
||||
<CollapsibleSection
|
||||
id={props.id}
|
||||
title={props.title}
|
||||
defaultsSummary={getAlertThresholdsDefaultsSummary(props.columns, props.defaults)}
|
||||
defaultsSummary={
|
||||
props.defaultsSummary ?? getAlertThresholdsDefaultsSummary(props.columns, props.defaults)
|
||||
}
|
||||
resourceCount={props.resources().length}
|
||||
collapsed={state.isCollapsed(props.id)}
|
||||
onToggle={() => state.toggleSection(props.id)}
|
||||
@@ -57,6 +67,9 @@ function TrueNASResourceSection(
|
||||
emptyMessage="No TrueNAS alert targets match the current filters."
|
||||
>
|
||||
<div ref={state.registerSection(props.id)} class="scroll-mt-24">
|
||||
<Show when={props.note}>
|
||||
<p class="mb-3 text-xs text-muted">{props.note}</p>
|
||||
</Show>
|
||||
<ResourceTable
|
||||
title=""
|
||||
onConfigureResourceIntent={tableProps.onConfigureResourceIntent}
|
||||
@@ -79,6 +92,8 @@ function TrueNASResourceSection(
|
||||
formatMetricValue={formatMetricValue}
|
||||
hasActiveAlert={state.hasActiveAlert}
|
||||
globalDefaults={props.defaults}
|
||||
globalDefaultFallbacks={props.globalDefaultFallbacks}
|
||||
columnTooltips={props.columnTooltips}
|
||||
setGlobalDefaults={props.setGlobalDefaults}
|
||||
setHasUnsavedChanges={tableProps.setHasUnsavedChanges}
|
||||
globalDisableFlag={tableProps.disableAllTrueNAS}
|
||||
@@ -104,6 +119,38 @@ export function ThresholdsTableTrueNASTab(props: ThresholdsTableSectionProps) {
|
||||
const trueNASDefaults = () => props.tableProps.trueNASDefaults ?? {};
|
||||
const trueNASStorageDefaults = () => ({ usage: trueNASDefaults().usage ?? 85 });
|
||||
const trueNASDiskDefaults = () => props.tableProps.trueNASDiskDefaults ?? {};
|
||||
const diskTemperatureByType = () =>
|
||||
getDiskTemperatureByTypeItems({
|
||||
agentDiskTemperature: props.tableProps.agentDefaults.diskTemperature,
|
||||
diskTempByType: props.tableProps.diskTempByType,
|
||||
});
|
||||
// Unset, TrueNAS disks follow Disk temperature by type, so the header names
|
||||
// the per-type triggers rather than one number.
|
||||
const trueNASDiskDefaultsSummary = () => {
|
||||
if (trueNASDiskDefaults().temperature !== undefined) return undefined;
|
||||
return getAlertThresholdsDefaultsSummary(
|
||||
TRUENAS_DISK_COLUMNS,
|
||||
diskTemperatureByType().length > 0 ? {} : { temperature: 0 },
|
||||
{ ruleItems: diskTemperatureByType() },
|
||||
);
|
||||
};
|
||||
// When the agent Disk Temp default is off, Disk temperature by type is off
|
||||
// too, so the unset cell reads Off and switching it on stages a TrueNAS-wide
|
||||
// value at the factory disk temperature trigger.
|
||||
const trueNASDiskTemperatureFallback = (): Record<string, AlertResourceGlobalDefaultFallback> => {
|
||||
const items = diskTemperatureByType();
|
||||
return {
|
||||
temperature: {
|
||||
label: 'By type',
|
||||
title:
|
||||
items.length > 0
|
||||
? `Follows Disk temperature by type under Agents: ${items.join(', ')}. Enter a value to use one temperature for every TrueNAS disk.`
|
||||
: 'Off because the agent Disk Temp default under Agents is off. Turn it on here to alert on TrueNAS disks anyway.',
|
||||
off: items.length === 0,
|
||||
enableValue: FACTORY_AGENT_DEFAULTS.diskTemperature,
|
||||
},
|
||||
};
|
||||
};
|
||||
|
||||
return (
|
||||
<>
|
||||
@@ -175,6 +222,10 @@ export function ThresholdsTableTrueNASTab(props: ThresholdsTableSectionProps) {
|
||||
icon={<HardDrive class="w-5 h-5" />}
|
||||
typeKey="truenas-disk"
|
||||
defaults={trueNASDiskDefaults()}
|
||||
defaultsSummary={trueNASDiskDefaultsSummary()}
|
||||
globalDefaultFallbacks={trueNASDiskTemperatureFallback()}
|
||||
columnTooltips={TRUENAS_DISK_COLUMN_TOOLTIPS}
|
||||
note="Unless set here, each disk alerts at Disk temperature by type, set under Agents."
|
||||
factoryDefaults={props.tableProps.factoryTrueNASDiskDefaults}
|
||||
setGlobalDefaults={props.tableProps.setTrueNASDiskDefaults}
|
||||
onResetDefaults={props.tableProps.resetTrueNASDiskDefaults}
|
||||
|
||||
@@ -457,6 +457,25 @@ describe('ThresholdsTable navigation and redirection', () => {
|
||||
});
|
||||
});
|
||||
|
||||
it('says Disk temperature by type covers agent, TrueNAS and Proxmox disks', async () => {
|
||||
setPathname('/alerts/thresholds/systems');
|
||||
const host: Agent = {
|
||||
id: 'legacy-h1',
|
||||
hostname: 'legacy-host',
|
||||
displayName: 'Legacy Host',
|
||||
status: 'online',
|
||||
lastSeen: 123,
|
||||
memory: { total: 100, used: 50, free: 50, usage: 50 },
|
||||
};
|
||||
|
||||
render(() => <ThresholdsTable {...(baseProps() as any)} agents={[host]} />);
|
||||
|
||||
const heading = await screen.findByText('Disk temperature by type');
|
||||
expect(heading.parentElement?.textContent).toContain(
|
||||
'for each disk type, for agent, TrueNAS and Proxmox disks',
|
||||
);
|
||||
});
|
||||
|
||||
it('navigates to correct route when tabs are clicked', () => {
|
||||
render(() => <ThresholdsTable {...(baseProps() as any)} />);
|
||||
|
||||
|
||||
@@ -244,6 +244,49 @@ export function resolveAlertResourceMetricEnableValue(
|
||||
return undefined;
|
||||
}
|
||||
|
||||
/**
|
||||
* A global default that may stay unset because another setting decides it,
|
||||
* such as TrueNAS disk temperature following Disk temperature by type. Unset,
|
||||
* the cell is empty with `label` as its placeholder and `title` explaining
|
||||
* where the value comes from. `off` says the other setting is itself off, so
|
||||
* the unset cell reads Off and switching it on has to stage `enableValue`.
|
||||
*/
|
||||
export interface AlertResourceGlobalDefaultFallback {
|
||||
label: string;
|
||||
title: string;
|
||||
off?: boolean;
|
||||
enableValue?: number;
|
||||
}
|
||||
|
||||
export interface AlertResourceGlobalDefaultCell {
|
||||
/** Unset and decided by the fallback. */
|
||||
follows: boolean;
|
||||
isOff: boolean;
|
||||
/** Value staged when the cell is switched back on from Off. */
|
||||
enableValue: number | undefined;
|
||||
}
|
||||
|
||||
/**
|
||||
* State of one Global Defaults metric cell. An unset default is Off, as the
|
||||
* alert engine reads it, unless the metric has a fallback; then it reads as
|
||||
* the fallback does. Switching it on from Off returns it to unset when the
|
||||
* fallback is on, and stages an explicit value when the fallback is off,
|
||||
* since unset would still be off.
|
||||
*/
|
||||
export function resolveAlertResourceGlobalDefaultCell(
|
||||
metric: string,
|
||||
value: number | undefined,
|
||||
fallback: AlertResourceGlobalDefaultFallback | undefined,
|
||||
): AlertResourceGlobalDefaultCell {
|
||||
const follows = value === undefined && fallback !== undefined;
|
||||
const explicitEnableValue = fallback?.enableValue ?? getAlertResourceEnabledDefault(metric);
|
||||
return {
|
||||
follows,
|
||||
isOff: follows ? Boolean(fallback?.off) : isAlertResourceMetricOff(value ?? 0),
|
||||
enableValue: fallback && !fallback.off ? undefined : explicitEnableValue,
|
||||
};
|
||||
}
|
||||
|
||||
export function getAlertResourceMetricDelayOverride(
|
||||
metricDelaySeconds: Record<string, number> | undefined,
|
||||
metric: string,
|
||||
|
||||
+29
-6
@@ -830,16 +830,39 @@ describe('buildTrueNASVMSections additional branches', () => {
|
||||
// =========================================================================
|
||||
|
||||
describe('buildTrueNASDiskSections additional branches', () => {
|
||||
it('sets Temperature tone to warning at 55 degrees', () => {
|
||||
expect(findRow(diskRes({ temperature: 55 }), 'Temperature')?.tone).toBe('warning');
|
||||
// The drawer judges heat by the disk type's alert trigger, through the
|
||||
// resolver the alerts store provides; these are the factory triggers.
|
||||
const factoryTriggers: Record<string, number> = { nvme: 70, sas: 65, sata: 55 };
|
||||
const resolveThresholds = (diskType: string) => {
|
||||
const critical = factoryTriggers[diskType.toLowerCase()] ?? 55;
|
||||
return { warning: critical - 5, critical };
|
||||
};
|
||||
const temperatureTone = (disk: Partial<ResourcePhysicalDiskMeta>) =>
|
||||
buildTrueNASDetailSections(diskRes(disk), resolveThresholds)
|
||||
.flatMap((section) => section.rows)
|
||||
.find((row) => row.label === 'Temperature')?.tone;
|
||||
|
||||
it('sets Temperature tone to warning from the disk type trigger', () => {
|
||||
expect(temperatureTone({ diskType: 'sata', temperature: 55 })).toBe('warning');
|
||||
expect(temperatureTone({ diskType: 'sas', temperature: 65 })).toBe('warning');
|
||||
expect(temperatureTone({ diskType: 'nvme', temperature: 70 })).toBe('warning');
|
||||
});
|
||||
|
||||
it('sets Temperature tone to warning above 55 degrees', () => {
|
||||
expect(findRow(diskRes({ temperature: 60 }), 'Temperature')?.tone).toBe('warning');
|
||||
it('keeps a reading under its type trigger at default tone', () => {
|
||||
expect(temperatureTone({ diskType: 'nvme', temperature: 60 })).toBe('default');
|
||||
expect(temperatureTone({ diskType: 'sas', temperature: 63 })).toBe('default');
|
||||
expect(temperatureTone({ diskType: 'sata', temperature: 40 })).toBe('default');
|
||||
});
|
||||
|
||||
it('sets Temperature tone to default below 55 degrees', () => {
|
||||
expect(findRow(diskRes({ temperature: 40 }), 'Temperature')?.tone).toBe('default');
|
||||
it('judges no heat without thresholds', () => {
|
||||
expect(findRow(diskRes({ diskType: 'sata', temperature: 80 }), 'Temperature')?.tone).toBe(
|
||||
'default',
|
||||
);
|
||||
expect(
|
||||
buildTrueNASDetailSections(diskRes({ diskType: 'sata', temperature: 80 }), () => null)
|
||||
.flatMap((section) => section.rows)
|
||||
.find((row) => row.label === 'Temperature')?.tone,
|
||||
).toBe('default');
|
||||
});
|
||||
|
||||
it('formats Wearout percentage for a valid value', () => {
|
||||
|
||||
+2
-1
@@ -381,8 +381,9 @@ describe('resourceDetailDrawerTrueNASModel', () => {
|
||||
},
|
||||
},
|
||||
});
|
||||
const sataTrigger = () => ({ warning: 50, critical: 55 });
|
||||
const temperatureRow = (resource: Resource) =>
|
||||
buildTrueNASDetailSections(resource)
|
||||
buildTrueNASDetailSections(resource, sataTrigger)
|
||||
.find((section) => section.label === 'Health')
|
||||
?.rows.find((row) => row.label === 'Temperature');
|
||||
|
||||
|
||||
@@ -20,7 +20,11 @@ import {
|
||||
type DetailSection,
|
||||
type DetailValueTone,
|
||||
} from '@/components/shared/detailSectionModel';
|
||||
import { getPhysicalDiskTemperaturePresentation } from '@/features/storageBackups/diskTemperaturePresentation';
|
||||
import {
|
||||
getPhysicalDiskTemperaturePresentation,
|
||||
isPhysicalDiskRunningHot,
|
||||
} from '@/features/storageBackups/diskTemperaturePresentation';
|
||||
import type { TrueNASDiskTemperatureThresholdResolver } from '@/features/truenas/truenasPageModel';
|
||||
import { hasImpairedResourceSource } from '@/utils/resourceSourceHealth';
|
||||
|
||||
export type ResourceDetailDrawerTrueNASRowTone = DetailValueTone;
|
||||
@@ -399,9 +403,13 @@ const diskTemperature = (disk: ResourcePhysicalDiskMeta) =>
|
||||
});
|
||||
|
||||
// A retained reading, such as a silent host agent's on a merged disk row, is
|
||||
// not current: it takes no heat tone and its title says why.
|
||||
// not current: it takes no heat tone and its title says why. A current reading
|
||||
// is hot from its disk type's alert trigger, the threshold the TrueNAS storage
|
||||
// table and the disk's temperature alert judge it by. Without a resolver, heat
|
||||
// is not judged.
|
||||
const diskTemperatureRow = (
|
||||
disk: ResourcePhysicalDiskMeta,
|
||||
resolveDiskTemperatureThresholds?: TrueNASDiskTemperatureThresholdResolver,
|
||||
): ResourceDetailDrawerTrueNASRow | null => {
|
||||
const reading = diskTemperature(disk);
|
||||
if (!reading) return null;
|
||||
@@ -411,9 +419,12 @@ const diskTemperatureRow = (
|
||||
tone: 'muted',
|
||||
});
|
||||
}
|
||||
return row('Temperature', reading.label, {
|
||||
tone: (disk.temperature ?? 0) >= 55 ? 'warning' : 'default',
|
||||
});
|
||||
const thresholds = resolveDiskTemperatureThresholds?.(disk.diskType ?? '') ?? null;
|
||||
const hot = isPhysicalDiskRunningHot(
|
||||
{ temperature: disk.temperature ?? 0, collection: disk.collection },
|
||||
thresholds,
|
||||
);
|
||||
return row('Temperature', reading.label, { tone: hot ? 'warning' : 'default' });
|
||||
};
|
||||
|
||||
const currentDiskTemperatureLabel = (disk: ResourcePhysicalDiskMeta): string | null => {
|
||||
@@ -424,6 +435,7 @@ const currentDiskTemperatureLabel = (disk: ResourcePhysicalDiskMeta): string | n
|
||||
const buildTrueNASDiskSections = (
|
||||
resource: Resource,
|
||||
disk: ResourcePhysicalDiskMeta,
|
||||
resolveDiskTemperatureThresholds?: TrueNASDiskTemperatureThresholdResolver,
|
||||
): ResourceDetailDrawerTrueNASSection[] => {
|
||||
const riskReasons = (disk.risk?.reasons ?? [])
|
||||
.map((reason) => asString(reason.summary))
|
||||
@@ -439,7 +451,7 @@ const buildTrueNASDiskSections = (
|
||||
|
||||
const healthRows = compactRows([
|
||||
row('Health', normalizeDelimitedLabel(disk.health), { tone: diskStateTone(disk) }),
|
||||
diskTemperatureRow(disk),
|
||||
diskTemperatureRow(disk, resolveDiskTemperatureThresholds),
|
||||
row(
|
||||
'Wearout',
|
||||
disk.wearout === undefined || disk.wearout < 0 ? null : formatPercent(disk.wearout),
|
||||
@@ -746,6 +758,7 @@ const buildTrueNASShareSections = (
|
||||
|
||||
export const buildTrueNASDetailSections = (
|
||||
resource: Resource,
|
||||
resolveDiskTemperatureThresholds?: TrueNASDiskTemperatureThresholdResolver,
|
||||
): ResourceDetailDrawerTrueNASSection[] => {
|
||||
if (resource.truenas?.share) return buildTrueNASShareSections(resource.truenas.share);
|
||||
if (resource.truenas?.vm) return buildTrueNASVMSections(resource, resource.truenas.vm);
|
||||
@@ -754,7 +767,11 @@ export const buildTrueNASDetailSections = (
|
||||
return buildTrueNASStorageSections(resource, resource.storage);
|
||||
}
|
||||
if (isTrueNASScopedResource(resource) && resource.physicalDisk) {
|
||||
return buildTrueNASDiskSections(resource, resource.physicalDisk);
|
||||
return buildTrueNASDiskSections(
|
||||
resource,
|
||||
resource.physicalDisk,
|
||||
resolveDiskTemperatureThresholds,
|
||||
);
|
||||
}
|
||||
if (isTrueNASScopedResource(resource) && resource.truenas) {
|
||||
return buildTrueNASSystemSections(resource, resource.truenas);
|
||||
|
||||
+5
-1
@@ -23,6 +23,7 @@ import {
|
||||
} from '@/utils/resourceIdentity';
|
||||
import { isPulseAgentPlatformResource } from '@/utils/agentResources';
|
||||
import { areSystemSettingsLoaded, shouldHideDockerUpdateActions } from '@/stores/systemSettings';
|
||||
import { useAlertsActivation } from '@/stores/alertsActivation';
|
||||
import {
|
||||
getResourcePolicyBadges,
|
||||
getResourcePolicyGovernedSummary,
|
||||
@@ -381,7 +382,10 @@ export const useResourceDetailDrawerDerivedState = (
|
||||
const vmwareDetailsSummary = createMemo(() =>
|
||||
buildVMwareDetailsSummary(resource().type, vmwareData()),
|
||||
);
|
||||
const trueNASDetailSections = createMemo(() => buildTrueNASDetailSections(resource()));
|
||||
const { getDiskTemperatureThresholds } = useAlertsActivation();
|
||||
const trueNASDetailSections = createMemo(() =>
|
||||
buildTrueNASDetailSections(resource(), getDiskTemperatureThresholds),
|
||||
);
|
||||
const hasTrueNASDetails = createMemo(() => trueNASDetailSections().length > 0);
|
||||
const trueNASDetailsSummary = createMemo(() => buildTrueNASDetailsSummary(resource()));
|
||||
const kubernetesDetailSections = createMemo(() => buildKubernetesDetailSections(resource()));
|
||||
|
||||
+49
-2
@@ -56,7 +56,8 @@ describe('readAlertsConfigurationSnapshot — absent optional sections', () => {
|
||||
expect(snapshot.trueNASDefaults.cpu).toBe(80);
|
||||
expect(snapshot.trueNASDefaults.usage).toBe(85);
|
||||
expect(snapshot.trueNASDefaults.diskRead).toBe(-1);
|
||||
expect(snapshot.trueNASDiskDefaults.temperature).toBe(55);
|
||||
// Unset: each TrueNAS disk follows Disk temperature by type.
|
||||
expect(snapshot.trueNASDiskDefaults.temperature).toBeUndefined();
|
||||
expect(snapshot.vmwareDefaults.cpu).toBe(80);
|
||||
expect(snapshot.vmwareDefaults.usage).toBe(85);
|
||||
});
|
||||
@@ -232,7 +233,8 @@ describe('readAlertsConfigurationSnapshot — factory fallback (undefined-trigge
|
||||
expect(snapshot.trueNASDefaults.diskWrite).toBe(-1);
|
||||
expect(snapshot.trueNASDefaults.networkIn).toBe(-1);
|
||||
expect(snapshot.trueNASDefaults.networkOut).toBe(-1);
|
||||
expect(snapshot.trueNASDiskDefaults.temperature).toBe(55);
|
||||
// No usable trigger leaves TrueNAS disks on Disk temperature by type.
|
||||
expect(snapshot.trueNASDiskDefaults.temperature).toBeUndefined();
|
||||
});
|
||||
|
||||
it('falls back to factory defaults for vmwareDefaults and agentDefaults', () => {
|
||||
@@ -1526,3 +1528,48 @@ describe('dockerDefaults.updateAlertDelayHours', () => {
|
||||
expect(alertConfig?.dockerDefaults?.updateAlertDelayHours).toBe(24);
|
||||
});
|
||||
});
|
||||
|
||||
describe('TrueNAS disk temperature round trip', () => {
|
||||
const buildTrueNASDiskPayload = (temperature: number | undefined) => {
|
||||
const snapshot = createDefaultAlertsConfigurationSnapshot();
|
||||
snapshot.trueNASDiskDefaults = { temperature };
|
||||
return buildAlertsConfigurationPayload({
|
||||
snapshot,
|
||||
rawOverridesConfig: {},
|
||||
alertsActivationState: null,
|
||||
alertsActivationConfig: null,
|
||||
}).alertConfig;
|
||||
};
|
||||
|
||||
it('omits an unset TrueNAS disk temperature so disks follow Disk temperature by type', () => {
|
||||
const alertConfig = buildTrueNASDiskPayload(undefined);
|
||||
expect(alertConfig?.truenasDiskDefaults).toEqual({});
|
||||
expect(alertConfig?.truenasDiskTemperatureByType).toBe(true);
|
||||
});
|
||||
|
||||
it('writes Off as a zero trigger, which the backend keeps as off', () => {
|
||||
expect(buildTrueNASDiskPayload(-1)?.truenasDiskDefaults).toEqual({
|
||||
temperature: { trigger: 0, clear: 0 },
|
||||
});
|
||||
});
|
||||
|
||||
it('writes a TrueNAS-wide value, 55 included, as a trigger', () => {
|
||||
expect(buildTrueNASDiskPayload(55)?.truenasDiskDefaults).toEqual({
|
||||
temperature: { trigger: 55, clear: 50 },
|
||||
});
|
||||
});
|
||||
|
||||
it('reads an absent temperature as unset and a zero trigger as off', () => {
|
||||
const base = { overrides: {} } as unknown as AlertConfig;
|
||||
expect(
|
||||
readAlertsConfigurationSnapshot({ ...base, truenasDiskDefaults: {} }).trueNASDiskDefaults
|
||||
.temperature,
|
||||
).toBeUndefined();
|
||||
expect(
|
||||
readAlertsConfigurationSnapshot({
|
||||
...base,
|
||||
truenasDiskDefaults: { temperature: { trigger: 0, clear: 0 } },
|
||||
}).trueNASDiskDefaults.temperature,
|
||||
).toBe(0);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -185,6 +185,15 @@ const createHysteresisThreshold = (trigger: number | undefined, clearMargin = 5)
|
||||
};
|
||||
};
|
||||
|
||||
// Unset leaves TrueNAS disks on Disk temperature by type, so it is omitted
|
||||
// rather than written as a trigger. Off is written as 0: the backend reads a
|
||||
// negative trigger as unset.
|
||||
const buildTrueNASDiskDefaultsConfig = (temperature: number | undefined) => {
|
||||
if (temperature === undefined) return {};
|
||||
if (temperature <= 0) return { temperature: { trigger: 0, clear: 0 } };
|
||||
return { temperature: createHysteresisThreshold(temperature) };
|
||||
};
|
||||
|
||||
const normalizeGap = (value: unknown, fallback: number) => {
|
||||
const numeric = Number(value);
|
||||
if (!Number.isFinite(numeric)) {
|
||||
@@ -413,10 +422,10 @@ export function readAlertsConfigurationSnapshot(config: AlertConfig): AlertsConf
|
||||
}
|
||||
|
||||
if (config.truenasDiskDefaults) {
|
||||
// Unset means each TrueNAS disk follows Disk temperature by type.
|
||||
const temperature = config.truenasDiskDefaults.temperature;
|
||||
snapshot.trueNASDiskDefaults = {
|
||||
temperature:
|
||||
getTriggerValue(config.truenasDiskDefaults.temperature) ??
|
||||
FACTORY_TRUENAS_DISK_DEFAULTS.temperature,
|
||||
temperature: temperature === undefined ? undefined : getTriggerValue(temperature),
|
||||
};
|
||||
}
|
||||
|
||||
@@ -847,9 +856,8 @@ export function buildAlertsConfigurationPayload({
|
||||
networkIn: createHysteresisThreshold(snapshot.trueNASDefaults.networkIn),
|
||||
networkOut: createHysteresisThreshold(snapshot.trueNASDefaults.networkOut),
|
||||
},
|
||||
truenasDiskDefaults: {
|
||||
temperature: createHysteresisThreshold(snapshot.trueNASDiskDefaults.temperature),
|
||||
},
|
||||
truenasDiskDefaults: buildTrueNASDiskDefaultsConfig(snapshot.trueNASDiskDefaults.temperature),
|
||||
truenasDiskTemperatureByType: true,
|
||||
vmwareDefaults: {
|
||||
cpu: createHysteresisThreshold(snapshot.vmwareDefaults.cpu),
|
||||
memory: createHysteresisThreshold(snapshot.vmwareDefaults.memory),
|
||||
|
||||
+44
@@ -0,0 +1,44 @@
|
||||
import { describe, expect, it } from 'vitest';
|
||||
import {
|
||||
getDiskTemperatureByTypeItems,
|
||||
resolveDiskTemperatureTriggerForType,
|
||||
resolveTrueNASDiskTemperatureDefault,
|
||||
} from '../trueNASDiskTemperature';
|
||||
|
||||
const factoryPolicy = {
|
||||
agentDiskTemperature: 55,
|
||||
diskTempByType: { nvme: 70, sas: 65, sata: 55 },
|
||||
};
|
||||
|
||||
describe('TrueNAS disk temperature in the thresholds editor', () => {
|
||||
it('gives each disk its type trigger, and unknown types the agent default', () => {
|
||||
expect(resolveDiskTemperatureTriggerForType(factoryPolicy, 'NVMe')).toBe(70);
|
||||
expect(resolveDiskTemperatureTriggerForType(factoryPolicy, 'sas')).toBe(65);
|
||||
expect(resolveDiskTemperatureTriggerForType(factoryPolicy, 'sata')).toBe(55);
|
||||
expect(resolveDiskTemperatureTriggerForType(factoryPolicy, 'hdd')).toBe(55);
|
||||
expect(
|
||||
resolveDiskTemperatureTriggerForType({ ...factoryPolicy, agentDiskTemperature: 60 }, ''),
|
||||
).toBe(60);
|
||||
});
|
||||
|
||||
it('switches every type off with the agent Disk Temp default', () => {
|
||||
const off = { ...factoryPolicy, agentDiskTemperature: -1 };
|
||||
expect(resolveDiskTemperatureTriggerForType(off, 'nvme')).toBe(0);
|
||||
expect(getDiskTemperatureByTypeItems(off)).toEqual([]);
|
||||
});
|
||||
|
||||
it('lets a TrueNAS-wide value, Off included, replace the type trigger', () => {
|
||||
expect(resolveTrueNASDiskTemperatureDefault(undefined, factoryPolicy, 'nvme')).toBe(70);
|
||||
expect(resolveTrueNASDiskTemperatureDefault(62, factoryPolicy, 'nvme')).toBe(62);
|
||||
expect(resolveTrueNASDiskTemperatureDefault(-1, factoryPolicy, 'nvme')).toBe(-1);
|
||||
});
|
||||
|
||||
it('names the per-type triggers', () => {
|
||||
expect(
|
||||
getDiskTemperatureByTypeItems({
|
||||
...factoryPolicy,
|
||||
diskTempByType: { nvme: 75, sas: 65, sata: 55 },
|
||||
}),
|
||||
).toEqual(['NVMe 75°C', 'SAS 65°C', 'SATA 55°C']);
|
||||
});
|
||||
});
|
||||
@@ -9,6 +9,7 @@ import type { Resource } from '@/types/resource';
|
||||
import type { Resource as TableResource } from '../tableTypes';
|
||||
import type { Override } from '../types';
|
||||
import type { ThresholdsDataInputs } from '../thresholdsResourceModel';
|
||||
import { resolveTrueNASDiskTemperatureDefault } from '../trueNASDiskTemperature';
|
||||
import {
|
||||
createOverridesMap,
|
||||
findOverrideByCandidates,
|
||||
@@ -307,7 +308,18 @@ export function useThresholdsPlatformData(inputs: ThresholdsDataInputs) {
|
||||
resource,
|
||||
type: 'truenasDisk',
|
||||
resourceType: 'TrueNAS Disk',
|
||||
defaults: trueNASDiskDefaults(),
|
||||
// Each disk inherits the trigger saving would apply to it: the
|
||||
// TrueNAS-wide value, else its type's disk temperature trigger.
|
||||
defaults: {
|
||||
temperature: resolveTrueNASDiskTemperatureDefault(
|
||||
trueNASDiskDefaults().temperature,
|
||||
{
|
||||
agentDiskTemperature: props.agentDefaults?.diskTemperature,
|
||||
diskTempByType: props.diskTempByType ?? {},
|
||||
},
|
||||
resource.physicalDisk?.diskType,
|
||||
),
|
||||
},
|
||||
overridesMap,
|
||||
node: trueNASSystemName(resource),
|
||||
instance: readString(resource.physicalDisk?.diskType) || 'TrueNAS',
|
||||
|
||||
@@ -0,0 +1,51 @@
|
||||
import { formatMetricValue } from './helpers';
|
||||
import { FACTORY_AGENT_DEFAULTS } from '@/utils/alertThresholdDefaults';
|
||||
|
||||
// The thresholds editor's view of the TrueNAS disk temperature trigger. It
|
||||
// mirrors the backend (trueNASDiskTemperatureDefaultNoLock): a TrueNAS-wide
|
||||
// value the user saved applies to every TrueNAS disk; unset, each disk follows
|
||||
// Disk temperature by type, which the agent Disk Temp default switches off for
|
||||
// every type when it is off. It reads the unsaved editor state, so a row shows
|
||||
// what saving would apply.
|
||||
|
||||
export const DISK_TEMPERATURE_TYPE_FIELDS: readonly { key: string; label: string }[] = [
|
||||
{ key: 'nvme', label: 'NVMe' },
|
||||
{ key: 'sas', label: 'SAS' },
|
||||
{ key: 'sata', label: 'SATA' },
|
||||
];
|
||||
|
||||
export interface DiskTemperaturePolicyState {
|
||||
/** The agent Disk Temp default trigger; zero or less is off. */
|
||||
agentDiskTemperature: number | undefined;
|
||||
diskTempByType: Record<string, number>;
|
||||
}
|
||||
|
||||
/** The disk temperature policy's trigger for a disk type; zero or less is off. */
|
||||
export function resolveDiskTemperatureTriggerForType(
|
||||
policy: DiskTemperaturePolicyState,
|
||||
diskType: string | null | undefined,
|
||||
): number {
|
||||
const base = policy.agentDiskTemperature ?? FACTORY_AGENT_DEFAULTS.diskTemperature;
|
||||
if (base <= 0) return 0;
|
||||
const normalizedType = (diskType ?? '').trim().toLowerCase();
|
||||
const byType = normalizedType ? policy.diskTempByType[normalizedType] : undefined;
|
||||
return typeof byType === 'number' && byType > 0 ? byType : base;
|
||||
}
|
||||
|
||||
/** The trigger a TrueNAS disk without its own override inherits. */
|
||||
export function resolveTrueNASDiskTemperatureDefault(
|
||||
trueNASDiskTemperature: number | undefined,
|
||||
policy: DiskTemperaturePolicyState,
|
||||
diskType: string | null | undefined,
|
||||
): number {
|
||||
if (trueNASDiskTemperature !== undefined) return trueNASDiskTemperature;
|
||||
return resolveDiskTemperatureTriggerForType(policy, diskType);
|
||||
}
|
||||
|
||||
/** "NVMe 70°C", one per disk type, or none when the policy is off. */
|
||||
export function getDiskTemperatureByTypeItems(policy: DiskTemperaturePolicyState): string[] {
|
||||
return DISK_TEMPERATURE_TYPE_FIELDS.flatMap((field) => {
|
||||
const trigger = resolveDiskTemperatureTriggerForType(policy, field.key);
|
||||
return trigger > 0 ? [`${field.label} ${formatMetricValue('temperature', trigger)}`] : [];
|
||||
});
|
||||
}
|
||||
@@ -147,6 +147,9 @@ export interface AlertConfig {
|
||||
kubernetesDefaults?: AlertThresholds;
|
||||
truenasDefaults?: AlertThresholds;
|
||||
truenasDiskDefaults?: AlertThresholds;
|
||||
// Set on every config written since an unset truenasDiskDefaults
|
||||
// temperature means each TrueNAS disk follows Disk temperature by type.
|
||||
truenasDiskTemperatureByType?: boolean;
|
||||
vmwareDefaults?: AlertThresholds;
|
||||
snapshotDefaults?: SnapshotAlertConfig;
|
||||
backupDefaults?: BackupAlertConfig;
|
||||
|
||||
@@ -96,7 +96,8 @@ describe('alertThresholdDefaults', () => {
|
||||
|
||||
describe('FACTORY_TRUENAS_DISK_DEFAULTS', () => {
|
||||
it('only carries a disk temperature trigger', () => {
|
||||
expect(FACTORY_TRUENAS_DISK_DEFAULTS).toEqual({ temperature: 55 });
|
||||
// Unset: each TrueNAS disk follows Disk temperature by type.
|
||||
expect(FACTORY_TRUENAS_DISK_DEFAULTS).toEqual({ temperature: undefined });
|
||||
expect(Object.keys(FACTORY_TRUENAS_DISK_DEFAULTS)).toEqual(['temperature']);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -507,7 +507,7 @@ describe('metricThresholds', () => {
|
||||
usage: 85,
|
||||
temperature: 80,
|
||||
});
|
||||
expect(FACTORY_TRUENAS_DISK_DEFAULTS).toEqual({ temperature: 55 });
|
||||
expect(FACTORY_TRUENAS_DISK_DEFAULTS).toEqual({ temperature: undefined });
|
||||
expect(FACTORY_VMWARE_DEFAULTS).toEqual({
|
||||
cpu: 80,
|
||||
memory: 85,
|
||||
|
||||
@@ -44,8 +44,12 @@ export const FACTORY_TRUENAS_DEFAULTS = {
|
||||
networkOut: -1,
|
||||
};
|
||||
|
||||
export const FACTORY_TRUENAS_DISK_DEFAULTS = {
|
||||
temperature: 55,
|
||||
// TrueNAS disk temperature has no flat factory value. Unset, each TrueNAS disk
|
||||
// follows Disk temperature by type, as the backend resolves it
|
||||
// (trueNASDiskTemperatureDefaultNoLock); a saved value applies to every
|
||||
// TrueNAS disk.
|
||||
export const FACTORY_TRUENAS_DISK_DEFAULTS: Record<string, number | undefined> = {
|
||||
temperature: undefined,
|
||||
};
|
||||
|
||||
export const FACTORY_VMWARE_DEFAULTS = {
|
||||
|
||||
@@ -35,6 +35,12 @@ type alertPolicyQuery struct {
|
||||
// StorageAliases are the storage resource's alias IDs, honored by the
|
||||
// storage override lookup.
|
||||
StorageAliases []string
|
||||
// DiskType is a physical disk's type ("nvme", "sas", "sata"), which
|
||||
// picks its disk temperature threshold. DiskTypeUnknown marks a disk
|
||||
// whose type was never recorded, such as an alert persisted before it
|
||||
// carried one; it is judged against the lowest per-type trigger.
|
||||
DiskType string
|
||||
DiskTypeUnknown bool
|
||||
}
|
||||
|
||||
// EffectiveAlertPolicy is the resolved policy for one resource.
|
||||
@@ -116,6 +122,9 @@ func (m *Manager) effectiveAlertPolicyNoLock(q alertPolicyQuery) EffectiveAlertP
|
||||
policy.AllDisabled, policy.OfflineDisabled = m.alertPolicyTypeSwitchesNoLock(q.TypeKey)
|
||||
|
||||
thresholds := m.defaultThresholdsForResourceType(q.TypeKey)
|
||||
if q.TypeKey == "truenas-disk" {
|
||||
thresholds.Temperature = m.trueNASDiskTemperatureDefaultNoLock(q.DiskType, q.DiskTypeUnknown)
|
||||
}
|
||||
switch {
|
||||
case isGuestThresholdResourceType(q.TypeKey):
|
||||
thresholds = m.customRuleThresholdsNoLock(thresholds, q.Guest)
|
||||
|
||||
@@ -12395,6 +12395,65 @@ func TestCheckEscalations(t *testing.T) {
|
||||
})
|
||||
}
|
||||
|
||||
// A config save re-judges open TrueNAS disk temperature alerts against the
|
||||
// threshold the next evaluation applies to each disk's type.
|
||||
func TestTrueNASDiskTemperatureAlertsReevaluatePerDiskType(t *testing.T) {
|
||||
m := newTestManager(t)
|
||||
cfg := unifiedEvalBaseConfig()
|
||||
cfg.TrueNASDiskDefaults = ThresholdConfig{}
|
||||
configureUnifiedEvalManager(t, m, cfg)
|
||||
|
||||
nvme := trueNASTemperatureDisk("nvme0n1", "nvme", 72)
|
||||
sata := trueNASTemperatureDisk("sda", "sata", 57)
|
||||
checkTrueNASTemperatureDisk(t, m, nvme)
|
||||
checkTrueNASTemperatureDisk(t, m, sata)
|
||||
for _, disk := range []unifiedresources.Resource{nvme, sata} {
|
||||
if _, firing := trueNASDiskTemperatureAlert(t, m, disk); !firing {
|
||||
t.Fatalf("%s did not fire before the save: %v", disk.ID, alertKeys(m))
|
||||
}
|
||||
}
|
||||
|
||||
// Raising only the NVMe trigger resolves the NVMe alert and leaves the
|
||||
// SATA alert, still over its own trigger, open.
|
||||
m.mu.Lock()
|
||||
m.config.DiskTempByType["nvme"] = HysteresisThreshold{Trigger: 75, Clear: 70}
|
||||
m.reevaluateActiveAlertsLocked()
|
||||
m.mu.Unlock()
|
||||
if _, firing := trueNASDiskTemperatureAlert(t, m, nvme); firing {
|
||||
t.Fatalf("NVMe alert at 72C stayed open under a 75C NVMe trigger")
|
||||
}
|
||||
if _, firing := trueNASDiskTemperatureAlert(t, m, sata); !firing {
|
||||
t.Fatalf("SATA alert at 57C resolved under its unchanged 55C trigger")
|
||||
}
|
||||
}
|
||||
|
||||
// An alert raised before it recorded its disk type is held to the lowest
|
||||
// per-type trigger, so a save never resolves an alert the next evaluation
|
||||
// raises again.
|
||||
func TestTrueNASDiskTemperatureAlertWithoutDiskTypeHeldAtLowestTrigger(t *testing.T) {
|
||||
m := newTestManager(t)
|
||||
cfg := unifiedEvalBaseConfig()
|
||||
cfg.TrueNASDiskDefaults = ThresholdConfig{}
|
||||
cfg.AgentDefaults.DiskTemperature = &HysteresisThreshold{Trigger: 70, Clear: 65}
|
||||
cfg.DiskTempByType = map[string]HysteresisThreshold{"nvme": {Trigger: 60, Clear: 55}}
|
||||
configureUnifiedEvalManager(t, m, cfg)
|
||||
|
||||
nvme := trueNASTemperatureDisk("nvme0n1", "nvme", 66)
|
||||
checkTrueNASTemperatureDisk(t, m, nvme)
|
||||
alert, firing := trueNASDiskTemperatureAlert(t, m, nvme)
|
||||
if !firing {
|
||||
t.Fatalf("NVMe at 66C did not fire under a 60C NVMe trigger: %v", alertKeys(m))
|
||||
}
|
||||
|
||||
m.mu.Lock()
|
||||
delete(alert.Metadata, "diskType")
|
||||
m.reevaluateActiveAlertsLocked()
|
||||
m.mu.Unlock()
|
||||
if _, firing := trueNASDiskTemperatureAlert(t, m, nvme); !firing {
|
||||
t.Fatalf("an alert without a disk type was resolved by the 70C agent default although its NVMe trigger is 60C")
|
||||
}
|
||||
}
|
||||
|
||||
func TestCleanupAlertsForNodes(t *testing.T) {
|
||||
// t.Parallel()
|
||||
|
||||
@@ -22285,14 +22344,20 @@ func TestStaleCleanupKeepsThresholdAlertStillBeingEvaluated(t *testing.T) {
|
||||
|
||||
// Resource history owns each PVE disk alert row by the hardware identity the
|
||||
// alert records, and reads by the path reference find those rows by alert
|
||||
// identifier. Both must match what CheckDiskHealth produces.
|
||||
// identifier. Both must match what CheckDiskHealth and
|
||||
// CheckProxmoxDiskTemperature produce.
|
||||
func TestCheckDiskHealthAlertsCarryHistoryOwnershipIdentity(t *testing.T) {
|
||||
m := newTestManager(t)
|
||||
m.ClearActiveAlerts()
|
||||
m.mu.Lock()
|
||||
m.config.TimeThresholds = map[string]int{}
|
||||
m.config.MetricTimeThresholds = nil
|
||||
m.mu.Unlock()
|
||||
disk := proxmox.Disk{DevPath: "/dev/sdb", Model: "Crucial MX500", Serial: "2117E59AB123", WWN: "0x5002538f12345678",
|
||||
Type: "ssd", Health: "FAILED", Wearout: 3}
|
||||
for i := 0; i < 3; i++ {
|
||||
m.CheckDiskHealth("lab", "pve1", disk)
|
||||
m.CheckProxmoxDiskTemperature("lab", "pve1", ProxmoxDiskTemperatureReading{Disk: disk, Celsius: 80})
|
||||
}
|
||||
|
||||
ref := unifiedresources.ProxmoxPhysicalDiskAlertResourceID("lab", "pve1", disk.DevPath)
|
||||
@@ -22313,3 +22378,187 @@ func TestCheckDiskHealthAlertsCarryHistoryOwnershipIdentity(t *testing.T) {
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func proxmoxTemperatureDisk(devPath, diskType string) proxmox.Disk {
|
||||
return proxmox.Disk{DevPath: devPath, Model: "Test Disk", Serial: "SER-" + devPath, Type: diskType, Health: "PASSED"}
|
||||
}
|
||||
|
||||
func proxmoxDiskTemperatureAlert(t *testing.T, m *Manager, devPath string) (*Alert, bool) {
|
||||
t.Helper()
|
||||
ref := unifiedresources.ProxmoxPhysicalDiskAlertResourceID("lab", "pve1", devPath)
|
||||
return testLookupActiveAlert(t, m, canonicalMetricStateID(ref, proxmoxDiskTemperatureMetric))
|
||||
}
|
||||
|
||||
func proxmoxDiskTemperatureResolution(m *Manager, devPath string) (*models.AlertResolution, bool) {
|
||||
ref := unifiedresources.ProxmoxPhysicalDiskAlertResourceID("lab", "pve1", devPath)
|
||||
id := canonicalMetricStateID(ref, proxmoxDiskTemperatureMetric)
|
||||
for _, resolved := range m.GetRecentlyResolved() {
|
||||
if resolved.ID == id {
|
||||
return resolved.Resolution, true
|
||||
}
|
||||
}
|
||||
return nil, false
|
||||
}
|
||||
|
||||
// TestProxmoxDiskTemperatureAlertsFollowDiskTemperaturePolicy pins Proxmox disk
|
||||
// temperature alerts to the disk temperature policy that judges an agent's
|
||||
// disks, TrueNAS disks and the Physical Disks verdict.
|
||||
func TestProxmoxDiskTemperatureAlertsFollowDiskTemperaturePolicy(t *testing.T) {
|
||||
m := newTestManager(t)
|
||||
configureUnifiedEvalManager(t, m, unifiedEvalBaseConfig())
|
||||
|
||||
check := func(devPath, diskType string, celsius int) {
|
||||
m.CheckProxmoxDiskTemperature("lab", "pve1", ProxmoxDiskTemperatureReading{Disk: proxmoxTemperatureDisk(devPath, diskType), Celsius: celsius})
|
||||
}
|
||||
check("/dev/nvme0n1", "nvme", 63)
|
||||
check("/dev/nvme1n1", "nvme", 72)
|
||||
check("/dev/sda", "sata", 56)
|
||||
check("/dev/sdb", "sas", 63)
|
||||
|
||||
if _, firing := proxmoxDiskTemperatureAlert(t, m, "/dev/nvme0n1"); firing {
|
||||
t.Fatalf("NVMe at 63C fired under its 70C trigger: %v", alertKeys(m))
|
||||
}
|
||||
if _, firing := proxmoxDiskTemperatureAlert(t, m, "/dev/sdb"); firing {
|
||||
t.Fatalf("SAS at 63C fired under its 65C trigger: %v", alertKeys(m))
|
||||
}
|
||||
for devPath, trigger := range map[string]float64{"/dev/nvme1n1": 70, "/dev/sda": 55} {
|
||||
alert, firing := proxmoxDiskTemperatureAlert(t, m, devPath)
|
||||
if !firing {
|
||||
t.Fatalf("%s did not fire: %v", devPath, alertKeys(m))
|
||||
}
|
||||
if alert.Threshold != trigger {
|
||||
t.Fatalf("%s trigger = %v, want %v", devPath, alert.Threshold, trigger)
|
||||
}
|
||||
if alert.Type != proxmoxDiskTemperatureMetric || alert.Metadata["resourceType"] != proxmoxDiskResourceType {
|
||||
t.Fatalf("%s alert type %q, resourceType %v", devPath, alert.Type, alert.Metadata["resourceType"])
|
||||
}
|
||||
if alert.Metadata["diskType"] == nil || alert.Metadata[unifiedresources.MetadataDiskSerial] != "SER-"+devPath {
|
||||
t.Fatalf("%s metadata = %v", devPath, alert.Metadata)
|
||||
}
|
||||
}
|
||||
if alert, _ := proxmoxDiskTemperatureAlert(t, m, "/dev/sda"); alert.Message != "Disk temperature at 56.0°C" {
|
||||
t.Fatalf("message = %q", alert.Message)
|
||||
}
|
||||
|
||||
// A raised NVMe trigger moves NVMe disks with it.
|
||||
m.mu.Lock()
|
||||
m.config.DiskTempByType["nvme"] = HysteresisThreshold{Trigger: 75, Clear: 70}
|
||||
m.mu.Unlock()
|
||||
check("/dev/nvme2n1", "nvme", 72)
|
||||
if _, firing := proxmoxDiskTemperatureAlert(t, m, "/dev/nvme2n1"); firing {
|
||||
t.Fatalf("NVMe at 72C fired under a 75C trigger")
|
||||
}
|
||||
}
|
||||
|
||||
// A disk the linked agent reports is the agent's: its open alert closes as
|
||||
// moved to the agent, and the disk never alerts twice. An excluded disk, or a
|
||||
// policy the agent Disk Temp default switched off, closes it as a plain clear.
|
||||
// A disk with no current reading keeps its alert.
|
||||
func TestProxmoxDiskTemperatureAlertOwnershipAndHolds(t *testing.T) {
|
||||
m := newTestManager(t)
|
||||
configureUnifiedEvalManager(t, m, unifiedEvalBaseConfig())
|
||||
disk := proxmoxTemperatureDisk("/dev/sda", "sata")
|
||||
read := func(reading ProxmoxDiskTemperatureReading) {
|
||||
reading.Disk = disk
|
||||
m.CheckProxmoxDiskTemperature("lab", "pve1", reading)
|
||||
}
|
||||
firing := func() bool {
|
||||
_, ok := proxmoxDiskTemperatureAlert(t, m, "/dev/sda")
|
||||
return ok
|
||||
}
|
||||
|
||||
read(ProxmoxDiskTemperatureReading{Celsius: 60})
|
||||
if !firing() {
|
||||
t.Fatalf("SATA at 60C did not fire: %v", alertKeys(m))
|
||||
}
|
||||
read(ProxmoxDiskTemperatureReading{Celsius: 0})
|
||||
if !firing() {
|
||||
t.Fatalf("a poll without a current reading resolved the alert")
|
||||
}
|
||||
|
||||
read(ProxmoxDiskTemperatureReading{Celsius: 60, AgentOwned: true})
|
||||
if firing() {
|
||||
t.Fatalf("an agent-owned disk kept its Proxmox temperature alert")
|
||||
}
|
||||
if resolution, ok := proxmoxDiskTemperatureResolution(m, "/dev/sda"); !ok || resolution == nil || resolution.Reason != string(AlertResolutionMovedToAgent) {
|
||||
t.Fatalf("handover resolution = %+v (resolved %v), want moved_to_agent", resolution, ok)
|
||||
}
|
||||
read(ProxmoxDiskTemperatureReading{Celsius: 80, AgentOwned: true})
|
||||
if firing() {
|
||||
t.Fatalf("an agent-owned disk raised a Proxmox temperature alert")
|
||||
}
|
||||
|
||||
read(ProxmoxDiskTemperatureReading{Celsius: 60})
|
||||
if !firing() {
|
||||
t.Fatalf("the disk did not alert again once the agent stopped listing it")
|
||||
}
|
||||
read(ProxmoxDiskTemperatureReading{Celsius: 60, Excluded: true})
|
||||
if firing() {
|
||||
t.Fatalf("an excluded disk kept its temperature alert")
|
||||
}
|
||||
|
||||
read(ProxmoxDiskTemperatureReading{Celsius: 60})
|
||||
m.mu.Lock()
|
||||
m.config.AgentDefaults.DiskTemperature = &HysteresisThreshold{Trigger: 0, Clear: 0}
|
||||
m.mu.Unlock()
|
||||
read(ProxmoxDiskTemperatureReading{Celsius: 60})
|
||||
if firing() {
|
||||
t.Fatalf("a switched-off agent Disk Temp default kept the Proxmox disk alert")
|
||||
}
|
||||
}
|
||||
|
||||
// A Proxmox disk temperature warning waits out the factory stability window for
|
||||
// a noisy gauge, as agent and TrueNAS disk temperature warnings do; a critical
|
||||
// reading bypasses it.
|
||||
func TestProxmoxDiskTemperatureWarningWaitsOutStabilityWindow(t *testing.T) {
|
||||
m := newTestManager(t)
|
||||
cfg := unifiedEvalBaseConfig()
|
||||
m.UpdateConfig(cfg)
|
||||
if delay := m.GetConfig().TimeThresholds[proxmoxDiskResourceType]; delay != 5 {
|
||||
t.Fatalf("proxmox-disk delay = %d, want the 5s factory default", delay)
|
||||
}
|
||||
m.mu.RLock()
|
||||
grace := m.getTimeThreshold("lab:pve1:disk:dev-sda", proxmoxDiskResourceType, proxmoxDiskTemperatureMetric)
|
||||
m.mu.RUnlock()
|
||||
if grace != defaultNoisyGaugeStabilitySeconds {
|
||||
t.Fatalf("Proxmox disk temperature delay = %ds, want the %ds stability window", grace, defaultNoisyGaugeStabilitySeconds)
|
||||
}
|
||||
|
||||
m.CheckProxmoxDiskTemperature("lab", "pve1", ProxmoxDiskTemperatureReading{Disk: proxmoxTemperatureDisk("/dev/sda", "sata"), Celsius: 57})
|
||||
if _, firing := proxmoxDiskTemperatureAlert(t, m, "/dev/sda"); firing {
|
||||
t.Fatalf("a 57C SATA warning fired before its stability window")
|
||||
}
|
||||
m.CheckProxmoxDiskTemperature("lab", "pve1", ProxmoxDiskTemperatureReading{Disk: proxmoxTemperatureDisk("/dev/sdb", "sata"), Celsius: 70})
|
||||
if _, firing := proxmoxDiskTemperatureAlert(t, m, "/dev/sdb"); !firing {
|
||||
t.Fatalf("a 70C SATA critical reading waited out the stability window")
|
||||
}
|
||||
}
|
||||
|
||||
// A config save re-judges an open Proxmox disk temperature alert against its
|
||||
// disk type's trigger, as the next poll does; one without a recorded disk type
|
||||
// is held to the lowest per-type trigger.
|
||||
func TestProxmoxDiskTemperatureAlertsReevaluatePerDiskType(t *testing.T) {
|
||||
m := newTestManager(t)
|
||||
cfg := unifiedEvalBaseConfig()
|
||||
cfg.AgentDefaults.DiskTemperature = &HysteresisThreshold{Trigger: 70, Clear: 65}
|
||||
configureUnifiedEvalManager(t, m, cfg)
|
||||
m.CheckProxmoxDiskTemperature("lab", "pve1", ProxmoxDiskTemperatureReading{Disk: proxmoxTemperatureDisk("/dev/nvme0n1", "nvme"), Celsius: 72})
|
||||
m.CheckProxmoxDiskTemperature("lab", "pve1", ProxmoxDiskTemperatureReading{Disk: proxmoxTemperatureDisk("/dev/sda", "sata"), Celsius: 57})
|
||||
sata, ok := proxmoxDiskTemperatureAlert(t, m, "/dev/sda")
|
||||
if !ok {
|
||||
t.Fatalf("SATA at 57C did not fire: %v", alertKeys(m))
|
||||
}
|
||||
|
||||
m.mu.Lock()
|
||||
m.config.DiskTempByType["nvme"] = HysteresisThreshold{Trigger: 75, Clear: 70}
|
||||
delete(sata.Metadata, "diskType")
|
||||
m.reevaluateActiveAlertsLocked()
|
||||
m.mu.Unlock()
|
||||
|
||||
if _, firing := proxmoxDiskTemperatureAlert(t, m, "/dev/nvme0n1"); firing {
|
||||
t.Fatalf("NVMe alert at 72C survived a save raising its trigger to 75C")
|
||||
}
|
||||
if _, firing := proxmoxDiskTemperatureAlert(t, m, "/dev/sda"); !firing {
|
||||
t.Fatalf("a SATA alert without a disk type was resolved by the 70C agent default although its trigger is 55C")
|
||||
}
|
||||
}
|
||||
|
||||
@@ -794,7 +794,6 @@ func TestBranchCovNormalizeTrueNASDefaults(t *testing.T) {
|
||||
{"DiskWrite", t1.DiskWrite, alertconfig.HysteresisThreshold{Trigger: 0, Clear: 0}},
|
||||
{"NetworkIn", t1.NetworkIn, alertconfig.HysteresisThreshold{Trigger: 0, Clear: 0}},
|
||||
{"NetworkOut", t1.NetworkOut, alertconfig.HysteresisThreshold{Trigger: 0, Clear: 0}},
|
||||
{"Disk.Temperature", cfg.TrueNASDiskDefaults.Temperature, alertconfig.HysteresisThreshold{Trigger: 55, Clear: 50}},
|
||||
} {
|
||||
if f.got == nil {
|
||||
t.Fatalf("%s is nil, want non-nil", f.name)
|
||||
@@ -803,6 +802,11 @@ func TestBranchCovNormalizeTrueNASDefaults(t *testing.T) {
|
||||
t.Fatalf("%s = %+v, want %+v", f.name, *f.got, f.want)
|
||||
}
|
||||
}
|
||||
// TrueNAS disks follow the disk temperature policy until the user
|
||||
// saves a TrueNAS-wide value.
|
||||
if cfg.TrueNASDiskDefaults.Temperature != nil {
|
||||
t.Fatalf("Disk.Temperature = %+v, want unset", *cfg.TrueNASDiskDefaults.Temperature)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("derivation and operator-enabled IO metric via CPU/DiskRead", func(t *testing.T) {
|
||||
|
||||
@@ -388,9 +388,35 @@ func NormalizeTrueNASDefaults(config *AlertConfig) {
|
||||
config.TrueNASDefaults.NetworkIn = normalizeThresholdPointer(config.TrueNASDefaults.NetworkIn, 0, 0, "truenas.networkIn")
|
||||
config.TrueNASDefaults.NetworkOut = normalizeThresholdPointer(config.TrueNASDefaults.NetworkOut, 0, 0, "truenas.networkOut")
|
||||
|
||||
config.TrueNASDiskDefaults.Temperature = normalizeThresholdPointer(config.TrueNASDiskDefaults.Temperature, 55, 50, "truenas.disk.temperature")
|
||||
config.TrueNASDiskDefaults.Temperature = normalizeTrueNASDiskTemperature(config.TrueNASDiskDefaults.Temperature, config.TrueNASDiskTemperatureByType)
|
||||
config.TrueNASDiskTemperatureByType = true
|
||||
}
|
||||
|
||||
// normalizeTrueNASDiskTemperature keeps the TrueNAS Disks temperature default
|
||||
// unset unless the user chose one. Unset, each TrueNAS disk follows the disk
|
||||
// temperature policy for its type. Before that, the factory default was a
|
||||
// flat 55/50 and the thresholds page wrote it back on every save, so in a
|
||||
// config written before (byType false) a stored 55/50 marks a value nobody
|
||||
// chose and is migrated to unset. Any other value is the user's TrueNAS-wide
|
||||
// choice and is kept, 0 meaning off. Once byType is set, 55 is a choice too.
|
||||
func normalizeTrueNASDiskTemperature(current *HysteresisThreshold, byType bool) *HysteresisThreshold {
|
||||
if current == nil || current.Trigger < 0 {
|
||||
return nil
|
||||
}
|
||||
if !byType && current.Trigger == legacyTrueNASDiskTemperatureTrigger &&
|
||||
(current.Clear == legacyTrueNASDiskTemperatureClear || current.Clear <= 0) {
|
||||
return nil
|
||||
}
|
||||
return normalizeThresholdPointer(current, 0, 0, "truenas.disk.temperature")
|
||||
}
|
||||
|
||||
// The flat TrueNAS disk temperature factory default before TrueNAS disks
|
||||
// followed the disk temperature policy.
|
||||
const (
|
||||
legacyTrueNASDiskTemperatureTrigger = 55
|
||||
legacyTrueNASDiskTemperatureClear = 50
|
||||
)
|
||||
|
||||
func NormalizeVMwareDefaults(config *AlertConfig) {
|
||||
config.VMwareDefaults.CPU = normalizeThresholdPointer(config.VMwareDefaults.CPU, 80, 75, "vmware.cpu")
|
||||
config.VMwareDefaults.Memory = normalizeThresholdPointer(config.VMwareDefaults.Memory, 85, 80, "vmware.memory")
|
||||
@@ -554,6 +580,10 @@ func NormalizeTimeThresholds(config *AlertConfig) {
|
||||
ensureDelay("truenas-pool")
|
||||
ensureDelay("truenas-dataset")
|
||||
ensureDelay("truenas-disk")
|
||||
// Proxmox disk temperature alerts: the factory stability window for a
|
||||
// noisy gauge applies to this type's unchanged default, as it does to
|
||||
// agent and TrueNAS disk temperatures.
|
||||
ensureDelay("proxmox-disk")
|
||||
ensureDelay("vmware-host")
|
||||
ensureDelay("vmware-vm")
|
||||
ensureDelay("vmware-datastore")
|
||||
|
||||
@@ -0,0 +1,58 @@
|
||||
package config
|
||||
|
||||
import "testing"
|
||||
|
||||
// TestNormalizeTrueNASDiskTemperatureKeepsOnlyChosenValues pins the saved
|
||||
// config migration that lets TrueNAS disks follow the disk temperature policy.
|
||||
// In a config written before it, the old flat factory default was written
|
||||
// back on every save, so it is not a choice; anything else the user saved is
|
||||
// kept.
|
||||
func TestNormalizeTrueNASDiskTemperatureKeepsOnlyChosenValues(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
stored *HysteresisThreshold
|
||||
want *HysteresisThreshold
|
||||
}{
|
||||
{name: "unset stays unset", stored: nil, want: nil},
|
||||
{name: "old factory default is migrated to unset", stored: &HysteresisThreshold{Trigger: 55, Clear: 50}, want: nil},
|
||||
{name: "old factory trigger without a clear is migrated to unset", stored: &HysteresisThreshold{Trigger: 55}, want: nil},
|
||||
{name: "negative trigger is unset", stored: &HysteresisThreshold{Trigger: -1}, want: nil},
|
||||
{name: "off is kept", stored: &HysteresisThreshold{Trigger: 0, Clear: 10}, want: &HysteresisThreshold{Trigger: 0, Clear: 0}},
|
||||
{name: "custom value is kept", stored: &HysteresisThreshold{Trigger: 62, Clear: 58}, want: &HysteresisThreshold{Trigger: 62, Clear: 58}},
|
||||
{name: "custom value gets a clear", stored: &HysteresisThreshold{Trigger: 62}, want: &HysteresisThreshold{Trigger: 62, Clear: 57}},
|
||||
{name: "55 with a custom clear is a choice", stored: &HysteresisThreshold{Trigger: 55, Clear: 45}, want: &HysteresisThreshold{Trigger: 55, Clear: 45}},
|
||||
}
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
cfg := &AlertConfig{TrueNASDiskDefaults: ThresholdConfig{Temperature: tt.stored}}
|
||||
NormalizeTrueNASDefaults(cfg)
|
||||
if !cfg.TrueNASDiskTemperatureByType {
|
||||
t.Fatalf("normalized config is not marked as following disk type")
|
||||
}
|
||||
got := cfg.TrueNASDiskDefaults.Temperature
|
||||
switch {
|
||||
case tt.want == nil && got != nil:
|
||||
t.Fatalf("Temperature = %+v, want unset", *got)
|
||||
case tt.want != nil && got == nil:
|
||||
t.Fatalf("Temperature unset, want %+v", *tt.want)
|
||||
case tt.want != nil && *got != *tt.want:
|
||||
t.Fatalf("Temperature = %+v, want %+v", *got, *tt.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// Once a config follows disk type, 55 is a value the user typed, and a save
|
||||
// or restart must keep it.
|
||||
func TestNormalizeTrueNASDiskTemperatureKeepsChosen55(t *testing.T) {
|
||||
cfg := &AlertConfig{
|
||||
TrueNASDiskDefaults: ThresholdConfig{Temperature: &HysteresisThreshold{Trigger: 55, Clear: 50}},
|
||||
TrueNASDiskTemperatureByType: true,
|
||||
}
|
||||
NormalizeTrueNASDefaults(cfg)
|
||||
NormalizeTrueNASDefaults(cfg)
|
||||
got := cfg.TrueNASDiskDefaults.Temperature
|
||||
if got == nil || *got != (HysteresisThreshold{Trigger: 55, Clear: 50}) {
|
||||
t.Fatalf("Temperature = %+v, want the chosen 55/50 kept", got)
|
||||
}
|
||||
}
|
||||
@@ -388,6 +388,13 @@ type AlertConfig struct {
|
||||
FlappingWindowSeconds int `json:"flappingWindowSeconds"` // Time window for counting state changes
|
||||
FlappingThreshold int `json:"flappingThreshold"` // Number of state changes to trigger flapping
|
||||
FlappingCooldownMinutes int `json:"flappingCooldownMinutes"` // Cooldown period after flapping detected
|
||||
|
||||
// TrueNASDiskTemperatureByType marks a config written since an unset
|
||||
// TrueNASDiskDefaults.Temperature means each TrueNAS disk follows the
|
||||
// disk temperature policy for its type. An older config carries the
|
||||
// former flat factory value, which nobody chose, and normalization drops
|
||||
// it once.
|
||||
TrueNASDiskTemperatureByType bool `json:"truenasDiskTemperatureByType,omitempty"`
|
||||
}
|
||||
|
||||
// UnmarshalJSON accepts canonical v6 alert config keys.
|
||||
|
||||
@@ -409,7 +409,17 @@ func (m *Manager) reevaluateActiveAlertsLocked() {
|
||||
alertsToResolve = append(alertsToResolve, alertID)
|
||||
continue
|
||||
}
|
||||
thresholds := m.resolveResourceThresholds(primaryResourceType, resourceID)
|
||||
query := alertPolicyQuery{TypeKey: primaryResourceType, ResourceID: resourceID}
|
||||
if primaryResourceType == "truenas-disk" {
|
||||
// Judge the alert against its disk type's threshold, as the
|
||||
// next evaluation will. An alert raised before it recorded a
|
||||
// disk type is held to the lowest per-type trigger, so a save
|
||||
// never resolves an alert its type would raise again.
|
||||
diskType, known := alert.Metadata["diskType"].(string)
|
||||
query.DiskType = diskType
|
||||
query.DiskTypeUnknown = !known
|
||||
}
|
||||
thresholds := m.effectiveAlertPolicyNoLock(query).Thresholds
|
||||
if thresholds.Disabled {
|
||||
alertsToResolve = append(alertsToResolve, alertID)
|
||||
continue
|
||||
@@ -417,6 +427,19 @@ func (m *Manager) reevaluateActiveAlertsLocked() {
|
||||
threshold = getThresholdForMetric(thresholds, metricType)
|
||||
}
|
||||
|
||||
// A Proxmox disk temperature alert is judged by the disk temperature
|
||||
// policy for its disk type, as CheckProxmoxDiskTemperature judges it.
|
||||
// The node and guest policies below have no disk temperature, so a
|
||||
// save would otherwise resolve an alert the next poll raises again.
|
||||
if primaryResourceType == proxmoxDiskResourceType && metricType == proxmoxDiskTemperatureMetric {
|
||||
diskType, known := alert.Metadata["diskType"].(string)
|
||||
threshold = m.proxmoxDiskTemperatureThresholdNoLock(diskType, !known)
|
||||
if threshold == nil || threshold.Trigger <= 0 {
|
||||
alertsToResolve = append(alertsToResolve, alertID)
|
||||
continue
|
||||
}
|
||||
}
|
||||
|
||||
isAgentResource := alertResourceTypeKeysContain(resourceTypeKeys, "agent")
|
||||
if !handledModernPlatformType && isAgentResource {
|
||||
if allDisabled, _ := m.alertPolicyTypeSwitchesNoLock("agent"); allDisabled {
|
||||
|
||||
@@ -106,9 +106,11 @@ func defaultAlertConfig() AlertConfig {
|
||||
NetworkIn: &HysteresisThreshold{Trigger: 0, Clear: 0},
|
||||
NetworkOut: &HysteresisThreshold{Trigger: 0, Clear: 0},
|
||||
},
|
||||
TrueNASDiskDefaults: ThresholdConfig{
|
||||
Temperature: &HysteresisThreshold{Trigger: 55, Clear: 50},
|
||||
},
|
||||
// TrueNAS disk temperature is unset: each disk follows the disk
|
||||
// temperature policy for its type (DiskTempByType) until the user
|
||||
// saves a TrueNAS-wide value.
|
||||
TrueNASDiskDefaults: ThresholdConfig{},
|
||||
TrueNASDiskTemperatureByType: true,
|
||||
VMwareDefaults: ThresholdConfig{
|
||||
CPU: &HysteresisThreshold{Trigger: 80, Clear: 75},
|
||||
Memory: &HysteresisThreshold{Trigger: 85, Clear: 80},
|
||||
@@ -148,6 +150,7 @@ func defaultAlertConfig() AlertConfig {
|
||||
"truenas-pool": 5,
|
||||
"truenas-dataset": 5,
|
||||
"truenas-disk": 5,
|
||||
"proxmox-disk": 5,
|
||||
"vmware-host": 5,
|
||||
"vmware-vm": 5,
|
||||
"vmware-datastore": 5,
|
||||
|
||||
@@ -0,0 +1,110 @@
|
||||
package alerts
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"strings"
|
||||
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources"
|
||||
"github.com/rcourtman/pulse-go-rewrite/pkg/proxmox"
|
||||
"github.com/rs/zerolog/log"
|
||||
)
|
||||
|
||||
// proxmoxDiskTemperatureMetric is the metric of a Proxmox physical disk's
|
||||
// temperature alert, the same metric CheckHost raises for an agent's disks.
|
||||
const proxmoxDiskTemperatureMetric = "diskTemperature"
|
||||
|
||||
// proxmoxDiskResourceType is the resource type PVE disk alerts carry in their
|
||||
// spec and metadata (see CheckDiskHealth).
|
||||
const proxmoxDiskResourceType = "proxmox-disk"
|
||||
|
||||
// ProxmoxDiskTemperatureReading is one Proxmox physical disk's temperature as
|
||||
// the PVE disk poller judged it.
|
||||
type ProxmoxDiskTemperatureReading struct {
|
||||
Disk proxmox.Disk
|
||||
// Celsius is the reading collected this poll, or 0 when there is none:
|
||||
// no sensor reports the disk, it is in standby, or normalization kept a
|
||||
// last-known value it did not collect now.
|
||||
Celsius int
|
||||
// AgentOwned says a linked Pulse agent lists the disk in its SMART
|
||||
// report, so the agent's CheckHost owns its temperature alert.
|
||||
AgentOwned bool
|
||||
// Excluded says the linked agent's --disk-exclude patterns match the
|
||||
// device, which excludes it from disk alerts.
|
||||
Excluded bool
|
||||
}
|
||||
|
||||
// CheckProxmoxDiskTemperature raises a Proxmox physical disk's temperature
|
||||
// alert under the disk temperature policy (DiskTemperatureThreshold): the
|
||||
// same per-type thresholds that judge an agent's disks, TrueNAS disks and the
|
||||
// Physical Disks verdict. Without it a disk that only the node's sensors
|
||||
// report could read Running Hot and never alert.
|
||||
//
|
||||
// A disk a linked agent lists is the agent's: its open alert here closes as
|
||||
// moved to the agent rather than recovered, so the disk never alerts twice.
|
||||
// An excluded disk, or a policy switched off by the agent Disk Temp default,
|
||||
// closes its alert. A disk with no current reading keeps its alert as it is,
|
||||
// since no reading shows it cooled.
|
||||
func (m *Manager) CheckProxmoxDiskTemperature(instance, node string, reading ProxmoxDiskTemperatureReading) {
|
||||
if m == nil {
|
||||
return
|
||||
}
|
||||
disk := reading.Disk
|
||||
resourceID := proxmoxDiskCanonicalResourceID(instance, node, disk.DevPath)
|
||||
resourceName := fmt.Sprintf("%s (%s)", disk.Model, disk.DevPath)
|
||||
diskType := strings.ToLower(strings.TrimSpace(disk.Type))
|
||||
|
||||
m.mu.RLock()
|
||||
enabled := m.config.Enabled
|
||||
threshold := m.proxmoxDiskTemperatureThresholdNoLock(diskType, false)
|
||||
m.mu.RUnlock()
|
||||
if !enabled {
|
||||
return
|
||||
}
|
||||
|
||||
if reading.AgentOwned || reading.Excluded || threshold == nil || threshold.Trigger <= 0 {
|
||||
spec, err := buildCanonicalMetricSpec(resourceID, resourceName, unifiedresources.ResourceType(proxmoxDiskResourceType), proxmoxDiskTemperatureMetric, nil)
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
var resolution *AlertResolution
|
||||
if reading.AgentOwned && !reading.Excluded {
|
||||
resolution = &AlertResolution{Reason: AlertResolutionMovedToAgent}
|
||||
}
|
||||
m.releaseCanonicalMetricAlert(spec, resourceName, node, instance, proxmoxDiskResourceType, 0, resolution)
|
||||
return
|
||||
}
|
||||
if reading.Celsius <= 0 {
|
||||
return
|
||||
}
|
||||
|
||||
spec, err := buildCanonicalMetricSpec(resourceID, resourceName, unifiedresources.ResourceType(proxmoxDiskResourceType), proxmoxDiskTemperatureMetric, threshold)
|
||||
if err != nil {
|
||||
log.Warn().
|
||||
Err(err).
|
||||
Str("resourceID", resourceID).
|
||||
Str("node", node).
|
||||
Str("disk", disk.DevPath).
|
||||
Msg("Skipping invalid canonical Proxmox disk temperature metric spec")
|
||||
return
|
||||
}
|
||||
metadata := proxmoxDiskAlertMetadata(disk)
|
||||
metadata["metric"] = proxmoxDiskTemperatureMetric
|
||||
// A config save re-judges the open alert against its disk type.
|
||||
metadata["diskType"] = diskType
|
||||
value := float64(reading.Celsius)
|
||||
m.checkMetricWithCanonicalSpec(spec, resourceName, node, instance, proxmoxDiskResourceType, value, threshold, &metricOptions{
|
||||
Metadata: metadata,
|
||||
Message: fmt.Sprintf("Disk temperature at %s", formatMetricValue(value, "°C")),
|
||||
})
|
||||
}
|
||||
|
||||
// proxmoxDiskTemperatureThresholdNoLock is the disk temperature policy for a
|
||||
// Proxmox disk of diskType, or the lowest per-type trigger for an alert whose
|
||||
// disk type was never recorded, so a config save never resolves an alert the
|
||||
// next poll raises again. Callers must hold m.mu.
|
||||
func (m *Manager) proxmoxDiskTemperatureThresholdNoLock(diskType string, typeUnknown bool) *HysteresisThreshold {
|
||||
if typeUnknown {
|
||||
return cloneThreshold(m.lowestHostDiskTemperatureThresholdNoLock(m.config.AgentDefaults.DiskTemperature))
|
||||
}
|
||||
return cloneThreshold(m.hostDiskTemperatureThresholdNoLock(m.config.AgentDefaults.DiskTemperature, false, diskType))
|
||||
}
|
||||
@@ -0,0 +1,18 @@
|
||||
package alerts
|
||||
|
||||
// trueNASDiskTemperatureDefaultNoLock returns the default tier of a TrueNAS
|
||||
// disk's temperature threshold. A TrueNAS-wide value the user saved under
|
||||
// TrueNAS Disks applies to every TrueNAS disk, as a host Disk Temp override
|
||||
// does for an agent's disks. Without one the disk follows the disk
|
||||
// temperature policy for its type, the threshold DiskTemperatureThreshold
|
||||
// returns, so its alert agrees with the TrueNAS storage table and Patrol. A
|
||||
// per-disk override still beats both. Callers must hold m.mu.
|
||||
func (m *Manager) trueNASDiskTemperatureDefaultNoLock(diskType string, typeUnknown bool) *HysteresisThreshold {
|
||||
if explicit := m.config.TrueNASDiskDefaults.Temperature; explicit != nil {
|
||||
return cloneThreshold(explicit)
|
||||
}
|
||||
if typeUnknown {
|
||||
return cloneThreshold(m.lowestHostDiskTemperatureThresholdNoLock(m.config.AgentDefaults.DiskTemperature))
|
||||
}
|
||||
return cloneThreshold(m.hostDiskTemperatureThresholdNoLock(m.config.AgentDefaults.DiskTemperature, false, diskType))
|
||||
}
|
||||
@@ -6,6 +6,7 @@ import (
|
||||
|
||||
alertspecs "github.com/rcourtman/pulse-go-rewrite/internal/alerts/specs"
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources"
|
||||
"github.com/rcourtman/pulse-go-rewrite/pkg/diskinventory"
|
||||
"github.com/rs/zerolog/log"
|
||||
)
|
||||
|
||||
@@ -57,6 +58,9 @@ type UnifiedResourceInput struct {
|
||||
Temperature *UnifiedResourceMetric
|
||||
|
||||
StorageAliases []string // Durable policy lookup identities for storage alerts.
|
||||
// DiskType is a physical disk's type ("nvme", "sas", "sata"). It picks
|
||||
// the disk temperature threshold of a TrueNAS disk.
|
||||
DiskType string
|
||||
}
|
||||
|
||||
type unifiedMetricCandidate struct {
|
||||
@@ -297,6 +301,11 @@ func (m *Manager) evaluateUnifiedMetrics(input *UnifiedResourceInput, thresholds
|
||||
if platformType != "" {
|
||||
metadata[alertPlatformTypeKey] = platformType
|
||||
}
|
||||
if input.Type == "truenas-disk" {
|
||||
// A config save re-judges the open temperature alert against
|
||||
// the threshold for this disk type.
|
||||
metadata["diskType"] = input.DiskType
|
||||
}
|
||||
merged.Metadata = metadata
|
||||
opts = &merged
|
||||
}
|
||||
@@ -551,10 +560,15 @@ func UnifiedResourceInputFromResource(resource unifiedresources.Resource) (*Unif
|
||||
}
|
||||
if resource.Temperature != nil {
|
||||
input.Temperature = &UnifiedResourceMetric{Value: *resource.Temperature, Percent: *resource.Temperature}
|
||||
} else if resource.PhysicalDisk != nil && resource.PhysicalDisk.Temperature > 0 {
|
||||
} else if resource.PhysicalDisk != nil &&
|
||||
diskinventory.TemperatureCollected(resource.PhysicalDisk.Temperature, resource.PhysicalDisk.Collection) {
|
||||
// A retained last-known reading is not current evidence of heat.
|
||||
value := float64(resource.PhysicalDisk.Temperature)
|
||||
input.Temperature = &UnifiedResourceMetric{Value: value, Percent: value}
|
||||
}
|
||||
if resource.PhysicalDisk != nil {
|
||||
input.DiskType = strings.ToLower(strings.TrimSpace(resource.PhysicalDisk.DiskType))
|
||||
}
|
||||
|
||||
return input, true
|
||||
}
|
||||
@@ -798,7 +812,7 @@ func (m *Manager) CheckUnifiedResourceWithCapacityTrend(input *UnifiedResourceIn
|
||||
return
|
||||
}
|
||||
|
||||
thresholds := m.resolveResourceThresholds(input.Type, input.ID)
|
||||
thresholds := m.effectiveAlertPolicyNoLock(alertPolicyQuery{TypeKey: input.Type, ResourceID: input.ID, DiskType: input.DiskType}).Thresholds
|
||||
m.mu.RUnlock()
|
||||
|
||||
if thresholds.Disabled {
|
||||
|
||||
@@ -10,6 +10,7 @@ import (
|
||||
alertspecs "github.com/rcourtman/pulse-go-rewrite/internal/alerts/specs"
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/models"
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources"
|
||||
"github.com/rcourtman/pulse-go-rewrite/pkg/diskinventory"
|
||||
)
|
||||
|
||||
func unifiedEvalBaseConfig() AlertConfig {
|
||||
@@ -1778,3 +1779,167 @@ func TestBuildMetricAlertStatusCarriesWindowAndMonotonicRecovery(t *testing.T) {
|
||||
t.Fatalf("status without clear level = %+v", noClear)
|
||||
}
|
||||
}
|
||||
|
||||
func trueNASTemperatureDisk(device, diskType string, celsius int) unifiedresources.Resource {
|
||||
return unifiedresources.Resource{
|
||||
ID: "physical-disk:truenas-main/" + device,
|
||||
Type: unifiedresources.ResourceTypePhysicalDisk,
|
||||
Name: device,
|
||||
ParentName: "truenas-main",
|
||||
Sources: []unifiedresources.DataSource{unifiedresources.SourceTrueNAS},
|
||||
PhysicalDisk: &unifiedresources.PhysicalDiskMeta{
|
||||
DevPath: "/dev/" + device,
|
||||
DiskType: diskType,
|
||||
Temperature: celsius,
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
func trueNASDiskTemperatureAlertID(resource unifiedresources.Resource) string {
|
||||
return canonicalMetricStateID(resource.ID, "temperature")
|
||||
}
|
||||
|
||||
func checkTrueNASTemperatureDisk(t *testing.T, m *Manager, resource unifiedresources.Resource) {
|
||||
t.Helper()
|
||||
input, ok := UnifiedResourceInputFromResource(resource)
|
||||
if !ok {
|
||||
t.Fatalf("TrueNAS disk %s did not become alert input", resource.ID)
|
||||
}
|
||||
m.CheckUnifiedResource(input)
|
||||
}
|
||||
|
||||
func trueNASDiskTemperatureAlert(t *testing.T, m *Manager, resource unifiedresources.Resource) (*Alert, bool) {
|
||||
t.Helper()
|
||||
return testLookupActiveAlert(t, m, trueNASDiskTemperatureAlertID(resource))
|
||||
}
|
||||
|
||||
// TestTrueNASDiskTemperatureAlertsFollowDiskTemperaturePolicy pins TrueNAS
|
||||
// disk temperature alerts to the disk temperature policy that the TrueNAS
|
||||
// storage table, Physical Disks and Patrol judge heat by: per disk type,
|
||||
// unless the user saved a TrueNAS-wide value, with a per-disk override first.
|
||||
func TestTrueNASDiskTemperatureAlertsFollowDiskTemperaturePolicy(t *testing.T) {
|
||||
nvme63 := trueNASTemperatureDisk("nvme0n1", "nvme", 63)
|
||||
nvme72 := trueNASTemperatureDisk("nvme1n1", "nvme", 72)
|
||||
sata56 := trueNASTemperatureDisk("sda", "sata", 56)
|
||||
sata60 := trueNASTemperatureDisk("sdb", "sata", 60)
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
mutate func(*AlertConfig)
|
||||
disks []unifiedresources.Resource
|
||||
firing []unifiedresources.Resource
|
||||
trigger map[string]float64
|
||||
}{
|
||||
{
|
||||
name: "factory thresholds judge each disk by its type",
|
||||
disks: []unifiedresources.Resource{nvme63, nvme72, sata56},
|
||||
firing: []unifiedresources.Resource{nvme72, sata56},
|
||||
trigger: map[string]float64{nvme72.ID: 70, sata56.ID: 55},
|
||||
},
|
||||
{
|
||||
name: "a raised NVMe trigger moves NVMe alerts with it",
|
||||
mutate: func(cfg *AlertConfig) {
|
||||
cfg.DiskTempByType = map[string]HysteresisThreshold{"nvme": {Trigger: 75, Clear: 70}}
|
||||
},
|
||||
disks: []unifiedresources.Resource{nvme72, sata56},
|
||||
firing: []unifiedresources.Resource{sata56},
|
||||
},
|
||||
{
|
||||
name: "a saved TrueNAS-wide value applies to every TrueNAS disk",
|
||||
mutate: func(cfg *AlertConfig) {
|
||||
cfg.TrueNASDiskDefaults.Temperature = &HysteresisThreshold{Trigger: 62, Clear: 57}
|
||||
},
|
||||
disks: []unifiedresources.Resource{nvme63, sata60},
|
||||
firing: []unifiedresources.Resource{nvme63},
|
||||
trigger: map[string]float64{nvme63.ID: 62},
|
||||
},
|
||||
{
|
||||
name: "a per-disk override beats the TrueNAS-wide value",
|
||||
mutate: func(cfg *AlertConfig) {
|
||||
cfg.TrueNASDiskDefaults.Temperature = &HysteresisThreshold{Trigger: 62, Clear: 57}
|
||||
cfg.Overrides = map[string]ThresholdConfig{
|
||||
sata56.ID: {Temperature: &HysteresisThreshold{Trigger: 50, Clear: 45}},
|
||||
}
|
||||
},
|
||||
disks: []unifiedresources.Resource{sata56},
|
||||
firing: []unifiedresources.Resource{sata56},
|
||||
trigger: map[string]float64{sata56.ID: 50},
|
||||
},
|
||||
{
|
||||
name: "a switched-off agent Disk Temp default switches TrueNAS disks off",
|
||||
mutate: func(cfg *AlertConfig) {
|
||||
cfg.AgentDefaults.DiskTemperature = &HysteresisThreshold{Trigger: 0, Clear: 0}
|
||||
},
|
||||
disks: []unifiedresources.Resource{nvme72, sata60},
|
||||
},
|
||||
{
|
||||
name: "a saved TrueNAS-wide value still applies when the agent default is off",
|
||||
mutate: func(cfg *AlertConfig) {
|
||||
cfg.AgentDefaults.DiskTemperature = &HysteresisThreshold{Trigger: 0, Clear: 0}
|
||||
cfg.TrueNASDiskDefaults.Temperature = &HysteresisThreshold{Trigger: 58, Clear: 53}
|
||||
},
|
||||
disks: []unifiedresources.Resource{sata60},
|
||||
firing: []unifiedresources.Resource{sata60},
|
||||
},
|
||||
{
|
||||
name: "a saved TrueNAS-wide off switches TrueNAS disks off",
|
||||
mutate: func(cfg *AlertConfig) {
|
||||
cfg.TrueNASDiskDefaults.Temperature = &HysteresisThreshold{Trigger: 0, Clear: 0}
|
||||
},
|
||||
disks: []unifiedresources.Resource{nvme72, sata60},
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
m := newTestManager(t)
|
||||
cfg := unifiedEvalBaseConfig()
|
||||
cfg.TrueNASDiskDefaults = ThresholdConfig{}
|
||||
if tt.mutate != nil {
|
||||
tt.mutate(&cfg)
|
||||
}
|
||||
configureUnifiedEvalManager(t, m, cfg)
|
||||
|
||||
for _, disk := range tt.disks {
|
||||
checkTrueNASTemperatureDisk(t, m, disk)
|
||||
}
|
||||
|
||||
want := make(map[string]bool, len(tt.firing))
|
||||
for _, disk := range tt.firing {
|
||||
want[disk.ID] = true
|
||||
}
|
||||
for _, disk := range tt.disks {
|
||||
alert, firing := trueNASDiskTemperatureAlert(t, m, disk)
|
||||
if firing != want[disk.ID] {
|
||||
t.Fatalf("%s firing = %v, want %v (active: %v)", disk.ID, firing, want[disk.ID], alertKeys(m))
|
||||
}
|
||||
if !firing {
|
||||
continue
|
||||
}
|
||||
if got := alert.Metadata["diskType"]; got != disk.PhysicalDisk.DiskType {
|
||||
t.Fatalf("%s diskType metadata = %v, want %q", disk.ID, got, disk.PhysicalDisk.DiskType)
|
||||
}
|
||||
if trigger, ok := tt.trigger[disk.ID]; ok && alert.Threshold != trigger {
|
||||
t.Fatalf("%s threshold = %v, want %v", disk.ID, alert.Threshold, trigger)
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// A last-known reading the source no longer collects is not evidence of heat.
|
||||
func TestTrueNASDiskTemperatureAlertIgnoresRetainedReading(t *testing.T) {
|
||||
m := newTestManager(t)
|
||||
cfg := unifiedEvalBaseConfig()
|
||||
cfg.TrueNASDiskDefaults = ThresholdConfig{}
|
||||
configureUnifiedEvalManager(t, m, cfg)
|
||||
|
||||
disk := trueNASTemperatureDisk("sda", "sata", 80)
|
||||
disk.PhysicalDisk.Collection = &diskinventory.CollectionStatus{
|
||||
Temperature: diskinventory.Unavailable("truenas", "disk is in standby"),
|
||||
}
|
||||
checkTrueNASTemperatureDisk(t, m, disk)
|
||||
if _, firing := trueNASDiskTemperatureAlert(t, m, disk); firing {
|
||||
t.Fatalf("retained 80C reading raised an alert: %v", alertKeys(m))
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,6 +1,8 @@
|
||||
package models
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
@@ -512,3 +514,22 @@ func TestCloneHostXCPNGInventoryIsolation(t *testing.T) {
|
||||
t.Fatal("cloned XCP-ng inventory aliases source VMs")
|
||||
}
|
||||
}
|
||||
|
||||
// AgentSMARTReported is internal poll evidence: a state snapshot copy keeps it
|
||||
// for the alert check, and it never reaches an API or websocket payload.
|
||||
func TestPhysicalDiskAgentSMARTReportedStaysInternal(t *testing.T) {
|
||||
disk := PhysicalDisk{ID: "lab-pve1-sda", DevPath: "/dev/sda", AgentSMARTReported: true}
|
||||
if clone := clonePhysicalDisk(disk); !clone.AgentSMARTReported {
|
||||
t.Fatalf("clone dropped AgentSMARTReported")
|
||||
}
|
||||
if clones := clonePhysicalDisks([]PhysicalDisk{disk}); !clones[0].AgentSMARTReported {
|
||||
t.Fatalf("slice clone dropped AgentSMARTReported")
|
||||
}
|
||||
payload, err := json.Marshal(disk)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if strings.Contains(strings.ToLower(string(payload)), "agentsmartreported") {
|
||||
t.Fatalf("AgentSMARTReported leaked into JSON: %s", payload)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -2766,7 +2766,11 @@ type PhysicalDisk struct {
|
||||
IO *DiskIO `json:"io,omitempty"`
|
||||
Collection *diskinventory.CollectionStatus `json:"collection,omitempty"`
|
||||
ExpectedUpdateInterval time.Duration `json:"-"` // Collector schedule, independent of general node polling
|
||||
LastChecked time.Time `json:"lastChecked"`
|
||||
// AgentSMARTReported marks a disk a linked host agent lists in its SMART
|
||||
// report, so the agent's CheckHost owns its temperature alert. Internal
|
||||
// poll evidence, set by the PVE disk poller's agent SMART merge.
|
||||
AgentSMARTReported bool `json:"-"`
|
||||
LastChecked time.Time `json:"lastChecked"`
|
||||
}
|
||||
|
||||
// PBSInstance represents a Proxmox Backup Server instance
|
||||
|
||||
@@ -513,6 +513,7 @@ func mergeHostAgentSMARTIntoDisks(disks []models.PhysicalDisk, nodes []models.No
|
||||
|
||||
// Build a map of node name to linked host's SMART data
|
||||
smartByNodeName := make(map[string][]models.HostDiskSMART)
|
||||
reportingNodeNames := make(map[string]bool)
|
||||
for _, node := range nodes {
|
||||
if node.LinkedAgentID == "" {
|
||||
continue
|
||||
@@ -522,6 +523,11 @@ func mergeHostAgentSMARTIntoDisks(disks []models.PhysicalDisk, nodes []models.No
|
||||
continue
|
||||
}
|
||||
smartByNodeName[node.Name] = host.Sensors.SMART
|
||||
// Only an agent still reporting owns its disks' temperature alerts;
|
||||
// a silent one's retained rows still enrich the disk.
|
||||
if !strings.EqualFold(strings.TrimSpace(host.Status), "offline") {
|
||||
reportingNodeNames[node.Name] = true
|
||||
}
|
||||
log.Debug().
|
||||
Str("nodeName", node.Name).
|
||||
Str("hostAgentID", node.LinkedAgentID).
|
||||
@@ -608,6 +614,11 @@ func mergeHostAgentSMARTIntoDisks(disks []models.PhysicalDisk, nodes []models.No
|
||||
if matched == nil {
|
||||
continue
|
||||
}
|
||||
// The linked agent reports this disk, so while it keeps reporting its
|
||||
// CheckHost owns the disk's temperature alert and the PVE disk check
|
||||
// stands aside. Once its lease lapses, CheckHost no longer alerts on
|
||||
// the disk and the PVE check judges the node's own readings again.
|
||||
updated[i].AgentSMARTReported = reportingNodeNames[updated[i].Node]
|
||||
|
||||
if strings.TrimSpace(updated[i].Model) == "" && strings.TrimSpace(matched.Model) != "" {
|
||||
updated[i].Model = strings.TrimSpace(matched.Model)
|
||||
|
||||
@@ -10,6 +10,7 @@ import (
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/alerts"
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/config"
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/logging"
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/models"
|
||||
@@ -1416,19 +1417,35 @@ func preserveUnavailablePhysicalDiskEvidence(current, previous models.PhysicalDi
|
||||
return current
|
||||
}
|
||||
|
||||
// checkPhysicalDiskAlerts evaluates one Proxmox physical disk's health and
|
||||
// wearout alerts. A device matched by the linked host agent's --disk-exclude
|
||||
// patterns is evaluated as healthy, so excluding a disk also resolves the
|
||||
// alerts it already raised.
|
||||
// checkPhysicalDiskAlerts evaluates one Proxmox physical disk's health,
|
||||
// wearout and temperature alerts. A device matched by the linked host agent's
|
||||
// --disk-exclude patterns is evaluated as healthy and without a temperature
|
||||
// alert, so excluding a disk also resolves the alerts it already raised.
|
||||
func (m *Monitor) checkPhysicalDiskAlerts(instance string, disk models.PhysicalDisk, excludePatterns []string) {
|
||||
if len(excludePatterns) > 0 && fsfilters.MatchesDeviceExclude(disk.DevPath, excludePatterns) {
|
||||
excluded := len(excludePatterns) > 0 && fsfilters.MatchesDeviceExclude(disk.DevPath, excludePatterns)
|
||||
if excluded {
|
||||
healthyDisk := proxmoxDiskFromPhysicalDisk(disk)
|
||||
healthyDisk.Health = "PASSED"
|
||||
healthyDisk.Wearout = 100
|
||||
m.alertManager.CheckDiskHealth(instance, disk.Node, healthyDisk)
|
||||
return
|
||||
} else {
|
||||
m.alertManager.CheckDiskHealth(instance, disk.Node, proxmoxDiskFromPhysicalDisk(disk))
|
||||
}
|
||||
m.alertManager.CheckDiskHealth(instance, disk.Node, proxmoxDiskFromPhysicalDisk(disk))
|
||||
m.alertManager.CheckProxmoxDiskTemperature(instance, disk.Node, alerts.ProxmoxDiskTemperatureReading{
|
||||
Disk: proxmoxDiskFromPhysicalDisk(disk),
|
||||
Celsius: collectedPhysicalDiskTemperature(disk),
|
||||
AgentOwned: disk.AgentSMARTReported,
|
||||
Excluded: excluded,
|
||||
})
|
||||
}
|
||||
|
||||
// collectedPhysicalDiskTemperature is the disk's temperature when this poll
|
||||
// collected it, or 0 for a last-known value normalization kept.
|
||||
func collectedPhysicalDiskTemperature(disk models.PhysicalDisk) int {
|
||||
if !diskinventory.TemperatureCollected(disk.Temperature, disk.Collection) {
|
||||
return 0
|
||||
}
|
||||
return disk.Temperature
|
||||
}
|
||||
|
||||
func proxmoxDiskFromPhysicalDisk(disk models.PhysicalDisk) proxmox.Disk {
|
||||
|
||||
@@ -1121,3 +1121,96 @@ func TestTrueNASPlaceholderDiskSerialsStayPerApplianceAndShareOneHistoryKey(t *t
|
||||
t.Fatalf("canonical TrueNAS disks = %d, want %d (one per appliance and device)", disks, diskCount)
|
||||
}
|
||||
}
|
||||
|
||||
// The agent SMART merge marks each Proxmox disk a reporting linked agent
|
||||
// lists, so the disk's temperature alert is left to the agent's CheckHost.
|
||||
// Disks the agent does not list, disks on nodes without an agent, and disks
|
||||
// of an agent whose lease lapsed stay unmarked. Disks the poller builds from
|
||||
// the agent's SMART report when the Proxmox query fails go through the same
|
||||
// merge.
|
||||
func TestMergeHostAgentSMARTIntoDisksMarksDisksTheAgentReports(t *testing.T) {
|
||||
disks := []models.PhysicalDisk{
|
||||
{Instance: "pve1", Node: "node1", DevPath: "/dev/sda"},
|
||||
{Instance: "pve1", Node: "node1", DevPath: "/dev/sdb"},
|
||||
{Instance: "pve1", Node: "node2", DevPath: "/dev/sda"},
|
||||
{Instance: "pve1", Node: "node3", DevPath: "/dev/sda"},
|
||||
}
|
||||
nodes := []models.Node{
|
||||
{Name: "node1", LinkedAgentID: "host-node1"},
|
||||
{Name: "node2"},
|
||||
{Name: "node3", LinkedAgentID: "host-node3"},
|
||||
}
|
||||
smart := []models.HostDiskSMART{{Device: "sda", Serial: "SER-SDA", Type: "sata", Temperature: 40}}
|
||||
hosts := []models.Host{
|
||||
{ID: "host-node1", Status: "online", Sensors: models.HostSensorSummary{SMART: smart}},
|
||||
{ID: "host-node3", Status: "offline", Sensors: models.HostSensorSummary{SMART: smart}},
|
||||
}
|
||||
|
||||
merged := mergeHostAgentSMARTIntoDisks(disks, nodes, hosts)
|
||||
for i, want := range []bool{true, false, false, false} {
|
||||
if merged[i].AgentSMARTReported != want {
|
||||
t.Fatalf("%s/%s AgentSMARTReported = %v, want %v", merged[i].Node, merged[i].DevPath, merged[i].AgentSMARTReported, want)
|
||||
}
|
||||
}
|
||||
if disks[0].AgentSMARTReported {
|
||||
t.Fatalf("merge modified the input slice")
|
||||
}
|
||||
if merged[3].Serial != "SER-SDA" {
|
||||
t.Fatalf("a silent agent's retained row no longer enriches the disk: %+v", merged[3])
|
||||
}
|
||||
|
||||
fallback := mergeHostAgentSMARTIntoDisks(physicalDisksFromHostAgentSMART("pve1", "node1", smart), nodes, hosts)
|
||||
if len(fallback) != 1 || !fallback[0].AgentSMARTReported {
|
||||
t.Fatalf("a disk built from a reporting agent's SMART report is not the agent's: %+v", fallback)
|
||||
}
|
||||
}
|
||||
|
||||
// checkPhysicalDiskAlerts raises a Proxmox disk's temperature alert from a
|
||||
// reading collected this poll, under the disk temperature policy, and leaves
|
||||
// alone a retained reading, a disk the linked agent reports and an excluded
|
||||
// device.
|
||||
func TestCheckPhysicalDiskAlertsRaisesProxmoxDiskTemperatureAlerts(t *testing.T) {
|
||||
manager := alerts.NewManagerWithDataDir(t.TempDir())
|
||||
m := &Monitor{alertManager: manager}
|
||||
const instance, node = "pve1", "node1"
|
||||
alertID := func(devPath string) string {
|
||||
return unifiedresources.ProxmoxPhysicalDiskAlertResourceID(instance, node, devPath) + "::metric-threshold:diskTemperature"
|
||||
}
|
||||
active := func(devPath string) bool {
|
||||
for _, alert := range manager.GetActiveAlerts() {
|
||||
if alert.ID == alertID(devPath) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
// 70C is critical for SATA (trigger 55, critical 65), which fires without
|
||||
// the stability delay a warning waits out.
|
||||
sata := func(devPath string) models.PhysicalDisk {
|
||||
return models.PhysicalDisk{Instance: instance, Node: node, DevPath: devPath, Model: "Test SATA", Type: "sata", Health: "PASSED", Wearout: -1, Temperature: 70}
|
||||
}
|
||||
|
||||
m.checkPhysicalDiskAlerts(instance, sata("/dev/sda"), nil)
|
||||
if !active("/dev/sda") {
|
||||
t.Fatalf("a SATA disk at 70C raised no temperature alert: %+v", manager.GetActiveAlerts())
|
||||
}
|
||||
|
||||
retained := sata("/dev/sdb")
|
||||
retained.Collection = &diskinventory.CollectionStatus{Temperature: diskinventory.Unavailable("proxmox_node_smart", "disk is in standby")}
|
||||
m.checkPhysicalDiskAlerts(instance, retained, nil)
|
||||
if active("/dev/sdb") {
|
||||
t.Fatalf("a retained 70C reading raised a temperature alert")
|
||||
}
|
||||
|
||||
agentOwned := sata("/dev/sdc")
|
||||
agentOwned.AgentSMARTReported = true
|
||||
m.checkPhysicalDiskAlerts(instance, agentOwned, nil)
|
||||
if active("/dev/sdc") {
|
||||
t.Fatalf("a disk the linked agent reports raised a second temperature alert")
|
||||
}
|
||||
|
||||
m.checkPhysicalDiskAlerts(instance, sata("/dev/sdd"), []string{"sdd"})
|
||||
if active("/dev/sdd") {
|
||||
t.Fatalf("an excluded device raised a temperature alert")
|
||||
}
|
||||
}
|
||||
|
||||
@@ -848,7 +848,7 @@ func TestProxmoxPhysicalDiskAlertReferenceShape(t *testing.T) {
|
||||
ProxmoxPhysicalDiskAlertResourceID("lab", "pve1", "/"),
|
||||
} {
|
||||
require.True(t, isProxmoxPhysicalDiskAlertReference(ref), ref)
|
||||
require.Len(t, ProxmoxPhysicalDiskAlertIdentifiers(ref), 2, ref)
|
||||
require.Len(t, ProxmoxPhysicalDiskAlertIdentifiers(ref), 3, ref)
|
||||
}
|
||||
for _, ref := range []string{
|
||||
ProxmoxPhysicalDiskAlertResourceID("lab", "pve1", ""),
|
||||
|
||||
@@ -26,6 +26,11 @@ const (
|
||||
// "<reference>::<reference><suffix>" (buildCanonicalStateID over the spec ID).
|
||||
var proxmoxDiskAlertSpecSuffixes = []string{"-health", "-wearout"}
|
||||
|
||||
// proxmoxDiskTemperatureAlertSpecID is the PVE disk temperature alert's spec,
|
||||
// a metric threshold spec (canonicalMetricSpecID over "diskTemperature"), so
|
||||
// its identifier is "<reference>::metric-threshold:diskTemperature".
|
||||
const proxmoxDiskTemperatureAlertSpecID = "metric-threshold:diskTemperature"
|
||||
|
||||
// ProxmoxPhysicalDiskAlertIdentifiers lists the alert identifiers of the PVE
|
||||
// disk alerts raised under ref, or nil when ref is not shaped like a PVE disk
|
||||
// alert reference.
|
||||
@@ -33,11 +38,11 @@ func ProxmoxPhysicalDiskAlertIdentifiers(ref string) []string {
|
||||
if !isProxmoxPhysicalDiskAlertReference(ref) {
|
||||
return nil
|
||||
}
|
||||
identifiers := make([]string, 0, len(proxmoxDiskAlertSpecSuffixes))
|
||||
identifiers := make([]string, 0, len(proxmoxDiskAlertSpecSuffixes)+1)
|
||||
for _, suffix := range proxmoxDiskAlertSpecSuffixes {
|
||||
identifiers = append(identifiers, ref+"::"+ref+suffix)
|
||||
}
|
||||
return identifiers
|
||||
return append(identifiers, ref+"::"+proxmoxDiskTemperatureAlertSpecID)
|
||||
}
|
||||
|
||||
// isProxmoxPhysicalDiskAlertReference matches "<instance>:<node>:disk:<key>"
|
||||
|
||||
@@ -14,6 +14,7 @@ import (
|
||||
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/models"
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/operationreceipt"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
func TestSQLiteResourceStoreDoesNotPersistRawOperationReceiptProtocolOrReplayAuthority(t *testing.T) {
|
||||
@@ -3328,3 +3329,32 @@ func TestSuccessionMergesGuestChangeJournalEras(t *testing.T) {
|
||||
t.Fatalf("legacy-era journal row missing from new-ID read: %+v", changes)
|
||||
}
|
||||
}
|
||||
|
||||
// A PVE disk temperature alert row is owned by the hardware identity it
|
||||
// records, like the health and wearout rows, and a read by the path reference
|
||||
// still finds it through ProxmoxPhysicalDiskAlertIdentifiers.
|
||||
func TestProxmoxDiskTemperatureAlertRowsFollowRecordedHardwareIdentity(t *testing.T) {
|
||||
store, err := NewSQLiteResourceStore(t.TempDir(), "default")
|
||||
require.NoError(t, err)
|
||||
t.Cleanup(func() { require.NoError(t, store.Close()) })
|
||||
now := time.Now().UTC().Truncate(time.Second)
|
||||
ref := ProxmoxPhysicalDiskAlertResourceID("lab", "pve1", "/dev/sda")
|
||||
snapshot := proxmoxHistoryIdentitySnapshot(now)
|
||||
snapshot.PhysicalDisks = []models.PhysicalDisk{pveDiskHistoryTestDisk("pve1", "/dev/sda", "ZA1A2B3C", "", now)}
|
||||
adapter := NewMonitorAdapter(NewRegistry(store))
|
||||
adapter.PopulateFromSnapshot(snapshot)
|
||||
diskID, ok := adapter.currentRegistry().ResolveReferenceID(ref)
|
||||
require.True(t, ok)
|
||||
|
||||
change := BuildAlertTimelineChange(ref, ChangeAlertFired, now, "", AlertTimelineChange{
|
||||
AlertIdentifier: ref + "::metric-threshold:diskTemperature", AlertStartedAt: now, AlertType: "diskTemperature",
|
||||
AlertMetadata: map[string]any{"disk_path": "/dev/sda", MetadataDiskSerial: "ZA1A2B3C", MetadataDiskWWN: ""},
|
||||
})
|
||||
change.ID = "temperature"
|
||||
require.NoError(t, adapter.RecordChange(*change))
|
||||
|
||||
require.ElementsMatch(t, []string{"temperature"}, pveDiskHistoryIDs(pveDiskHistoryRows(t, store, diskID)))
|
||||
byRef := pveDiskHistoryRows(t, store, ref)
|
||||
require.ElementsMatch(t, []string{"temperature"}, pveDiskHistoryIDs(byRef))
|
||||
require.Equal(t, ref, OwnedAlertReference(byRef["temperature"]))
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user