diff --git a/docs/release-control/v6/internal/subsystems/ai-runtime.md b/docs/release-control/v6/internal/subsystems/ai-runtime.md index e2534c99cd..48b93c4c1a 100644 --- a/docs/release-control/v6/internal/subsystems/ai-runtime.md +++ b/docs/release-control/v6/internal/subsystems/ai-runtime.md @@ -329,10 +329,17 @@ retained value as "last known 41C (disk is in standby)". Proofs: `internal/ai/patrol_ai_more_test.go`. They judge that collected reading against the alert disk temperature policy, -not a fixed line. `ThresholdProvider.GetDiskTemperatureThreshold(diskType)` -(the alert manager's per-type trigger and clear) resolves each disk's limits -once in `patrolPhysicalDiskRows`, and the run state carries the provider -(`patrolRuntimeState.thresholdProvider`). With no provider the factory alert +not a fixed line. `ThresholdProvider.GetDiskTemperatureThreshold(host, diskType)` +(the alert manager's `HostDiskTemperatureThreshold` trigger and clear) resolves +each disk's limits once in `patrolPhysicalDiskRows` and in the AI chat context, +for the host agent that reports the disk. `physicalDiskTemperatureHost` finds it +by walking the disk's parent to the agent machine, through the pool for an +Unraid disk, so that agent's Disk Temp override (or one inherited from its +linked node or guest) replaces the per-type trigger, and an override or agent +default that switches the host's alerts off leaves no heat to judge. A disk no +agent reports gets the per-type policy. The global alerts switch and the agent +alert-type switch stop `CheckHost` without changing this policy, as before. +The run state carries the provider (`patrolRuntimeState.thresholdProvider`). With no provider the factory alert configuration applies. A disk is hot from its trigger: a triage warning, a disk issue and an AI chat "needing attention" entry, as its temperature alert and Physical Disks Running Hot verdict are. A disk-high finding recovers at or @@ -341,8 +348,10 @@ the alert does. With no current reading, a disk last seen hot leaves that verification unknown rather than recovered. Scoped runs keep the provider, and a provider set before Patrol starts is handed to it. Below the trigger Patrol flags nothing, so an NVMe at 63C stays quiet and a SATA disk at 56C is flagged. Proofs: -`internal/ai/patrol_disk_temperature_test.go` and -`TestAlertThresholdAdapter_DiskTemperatureFollowsAlertPolicy` in +`internal/ai/patrol_disk_temperature_test.go` (a real registry with one +overridden agent in `TestPatrolJudgesAgentDisksByTheAgentDiskTemperatureOverride`), +`TestAlertThresholdAdapter_DiskTemperatureFollowsAlertPolicy` and +`TestAlertThresholdAdapter_DiskTemperatureHonoursHostOverrides` in `internal/ai/alert_threshold_adapter_test.go`. Retained summaries disclose that point and bucket timestamps describe returned diff --git a/docs/release-control/v6/internal/subsystems/alerts.md b/docs/release-control/v6/internal/subsystems/alerts.md index 2690c5f5ea..5df9ac3bd2 100644 --- a/docs/release-control/v6/internal/subsystems/alerts.md +++ b/docs/release-control/v6/internal/subsystems/alerts.md @@ -1789,14 +1789,26 @@ an alert the next evaluation raises again. The policy `CheckHost` applies to a SMART disk (the agent Disk Temp default, refined by the disk type's `DiskTempByType` entry, and switched off for every type when that default is off) is the canonical answer to "is this disk too -hot". `diskTemperatureThresholdForType` in `internal/alerts/host.go` holds it, -and `Manager.DiskTemperatureThreshold(diskType)` exposes it, without host -overrides, to Patrol through `AlertThresholdAdapter.GetDiskTemperatureThreshold`. +hot". `diskTemperatureThresholdForType` in `internal/alerts/host.go` holds it. +`Manager.HostDiskTemperatureThreshold(host, diskType)` resolves it for the host +agent that reports a disk exactly as `CheckHost` does: an explicit Disk Temp +override on that host, or one inherited from its linked node or guest, replaces +the per-type entry, the first override in that chain decides even when it sets +no Disk Temp, and an override that switches the host's alerts off leaves no +threshold. `TestHostDiskTemperatureThresholdMatchesCheckHostOverrides` pins each +case against what `CheckHost` fires, including an override under the canonical +resource the agent ID resolves to and one inherited from a linked guest. `Manager.DiskTemperatureThreshold(diskType)` +is the hostless case. Patrol reads the policy through +`AlertThresholdAdapter.GetDiskTemperatureThreshold`. `DefaultDiskTemperatureThreshold` serves callers with no manager. Disk risk (`internal/storagehealth`) judges no temperature. The Physical Disks Temp cell -and Health verdict mirror the per-type triggers and a switched-off default in -`resolveDiskTemperatureDisplayThresholds`; host overrides do not reach that -page yet. That resolver returns null when the agent Disk Temp default is off. +and Health verdict mirror the policy in `resolveDiskTemperatureDisplayThresholds` +under the override keys of the machine whose agent reports the disk, in the +order `CheckHost` reads them as far as the browser can see that chain +(`storage-recovery.md` lists where it cannot). That resolver returns null when the agent Disk +Temp default is off or an override or agent default switches the host's alerts +off. The global alerts switch and the agent alert-type switch stop `CheckHost` +without changing the policy. The displays that read it treat null as off: no threshold colour (`getTemperatureTextClass` with `diskTemperature`) and never hot. A `DiskTempByType` entry cannot be switched off on its own, because diff --git a/docs/release-control/v6/internal/subsystems/frontend-primitives.md b/docs/release-control/v6/internal/subsystems/frontend-primitives.md index 7019a6713d..4cf6c245b6 100644 --- a/docs/release-control/v6/internal/subsystems/frontend-primitives.md +++ b/docs/release-control/v6/internal/subsystems/frontend-primitives.md @@ -429,7 +429,9 @@ The physical-disk verdict words and their phone forms are `Needs Attention` (`Attention`), `Running Hot` (`Hot`) and `Replace Now` (`Replace`), all from `getPhysicalDiskHealthCompactLabel` in `diskPresentation.ts`. `Running Hot` is red and starts at the disk's alert trigger, from the same thresholds object -that colours its Temp cell (`getPhysicalDiskTemperatureThresholds`). +that colours its Temp cell (`getPhysicalDiskTemperatureThresholds`). Callers +pass the override keys of the machine that reports the disk, so that machine's +Disk Temp override sets those thresholds. Controller expansions carry the supplied kind-specific targets, absolute times, duration and cluster/namespace when the phone row omits those columns. Proxmox diff --git a/docs/release-control/v6/internal/subsystems/performance-and-scalability.md b/docs/release-control/v6/internal/subsystems/performance-and-scalability.md index 9ac7408aa9..55a9ba06fa 100644 --- a/docs/release-control/v6/internal/subsystems/performance-and-scalability.md +++ b/docs/release-control/v6/internal/subsystems/performance-and-scalability.md @@ -3448,6 +3448,10 @@ whole-estate physical-disk work to Workloads. The guest's existing agent RAID snapshot remains local to the opened drawer; an empty or failed child query must not imply that host disks belong to the guest. Focused hook and drawer tests pin the query and no-data/error presentations. +The drawer judges those disks' heat under the guest agent's alert overrides: +its `agentId`, which `useWorkloads` copies from `resource.agent.agentId` onto +the workload row, then the guest's own override keys. That reads the alert +configuration already loaded and adds no query, poll or per-row work. ### Configuration transfer authorization stays off persistence hot paths diff --git a/docs/release-control/v6/internal/subsystems/storage-recovery.md b/docs/release-control/v6/internal/subsystems/storage-recovery.md index 66185fb88d..53b6bd3c34 100644 --- a/docs/release-control/v6/internal/subsystems/storage-recovery.md +++ b/docs/release-control/v6/internal/subsystems/storage-recovery.md @@ -6789,13 +6789,36 @@ Alerts > Thresholds, `diskTempByType`, else the agent Disk Temp default). Disk risk never carries heat, because every registry that rebuilds it lacks the alert configuration. `extractPhysicalDiskPresentationData` takes the alerts store's `getDiskTemperatureThresholds` resolver, so the Temp cell colour and the -verdict read one thresholds object. The band below the trigger is a Temp cell +verdict read one thresholds object. It resolves them under the alert override +keys of the machine whose agent reports the disk, as disk temperature alerts +are evaluated under that host. `getPhysicalDiskAlertResourceIds` +(`components/Storage/physicalDiskAlertOwner.ts`) follows the disk's parent to +the agent machine, through the pool for an Unraid array or cache disk, and +reads that machine's keys in the order `CheckHost` reads them +(`getAgentMachineAlertOverrideKeys`: the agent ID, the machine's canonical ID, +then the node or guest the agent is merged with). +A Disk Temp override set on the machine therefore replaces the per-type trigger +for every disk it reports, on Physical Disks, the disk detail and the pool +drawer's linked disks. A guest drawer passes the guest agent's ID, then the +guest's own keys. An override that switches the machine's alerts off leaves its +disks uncoloured and never hot. The band below the trigger is a Temp cell colour only, and a retained (not current) reading is never heat. The verdict judges the current reading, with no memory of an alert. A disk temperature alert held between its clear value and trigger shows an amber Temp cell, not `Running Hot`. A switched-off agent Disk Temp default resolves to null, which leaves the reading uncoloured and never hot, as alerting is off for it. -Per-host Disk Temp overrides do not yet reach Physical Disks. PDF performance +The browser rebuilds that chain from what it loads, so it can differ from +alerts and Patrol, which read the agent's own link hints and resolve every key +to its canonical resource. On the Storage page, a disk whose agent an operator +manually linked into a VM keeps the per-type trigger, because the page loads +agents and pools, not guests. An override the agent inherits from a node or +guest it is linked to without a merge, or one stored under that node's own +canonical resource ID, does not reach the browser, because agent link hints +are not published. The guest drawer applies the guest's own override keys, +which `CheckHost` reads only when the agent is also linked to that guest +automatically. Overrides set on the reporting agent's own row, or on the node +or guest it is merged with, reach the browser through the keys above. PDF +performance reports colour a disk reading by the same per-type thresholds, which `reporting.DiskInfo` carries. `frontend-modern/src/features/storageBackups/__tests__/diskPresentation.test.ts` @@ -6803,7 +6826,10 @@ pins cell-and-verdict agreement for every type from 40C to 80C under factory and raised triggers, plus retained readings, failure evidence, sorting and filtering. `frontend-modern/src/components/Storage/__tests__/DiskList.test.tsx` covers NVMe 63C, SATA 56C and a raised NVMe trigger loaded through the alerts -store. +store, and NVMe disks at 72C on three machines judged by an 80C override, the +per-type trigger and switched-off alerts. +`frontend-modern/src/components/Storage/__tests__/physicalDiskAlertOwner.test.ts` +pins the owner walk and the key order. ### Pool-to-physical-disk ownership in Storage details diff --git a/frontend-modern/browser-verification/7db0b8c4fce0b08a.json b/frontend-modern/browser-verification/7db0b8c4fce0b08a.json new file mode 100644 index 0000000000..6ac99d08f0 --- /dev/null +++ b/frontend-modern/browser-verification/7db0b8c4fce0b08a.json @@ -0,0 +1,79 @@ +{ + "version": 1, + "base_sha": "38ce73c24a7fdf9f3523fbf7f9f9a23ac2be7276", + "verified_at": "2026-10-07T10:53:35Z", + "result": "passed", + "changed_paths": [ + "frontend-modern/src/components/Storage/DiskDetail.tsx", + "frontend-modern/src/components/Storage/DiskList.tsx", + "frontend-modern/src/components/Storage/Storage.tsx", + "frontend-modern/src/components/Storage/StorageContentCard.tsx", + "frontend-modern/src/components/Storage/StoragePoolDetail.tsx", + "frontend-modern/src/components/Storage/StoragePoolRow.tsx", + "frontend-modern/src/components/Storage/StoragePoolsTable.tsx", + "frontend-modern/src/components/Storage/physicalDiskAlertOwner.ts", + "frontend-modern/src/components/Storage/useDiskDetailModel.ts", + "frontend-modern/src/components/Storage/useDiskListModel.ts", + "frontend-modern/src/components/Storage/useStoragePageModel.ts", + "frontend-modern/src/components/Storage/useStoragePageResources.ts", + "frontend-modern/src/components/Storage/useStoragePoolDetailModel.ts", + "frontend-modern/src/components/Workloads/GuestDrawerOverview.tsx", + "frontend-modern/src/components/Workloads/GuestPhysicalDisks.tsx", + "frontend-modern/src/features/storageBackups/diskPresentation.ts", + "frontend-modern/src/features/storageBackups/storagePoolDetailPresentation.ts", + "frontend-modern/src/hooks/useWorkloads.ts", + "frontend-modern/src/types/workloads.ts", + "frontend-modern/src/utils/metricThresholds.ts" + ], + "content_sha256": { + "frontend-modern/src/components/Storage/DiskDetail.tsx": "f6617192db42acb3da59297ef86e03e2d01dafc16cb5744ad87f84ffd37f6773", + "frontend-modern/src/components/Storage/DiskList.tsx": "0a457320ac9aea99618f9c1be12cd85a59d0536341436b225eeaeadec09896f3", + "frontend-modern/src/components/Storage/Storage.tsx": "c1ecf5d710c427f0acc8230728e15648376628b43131f8504f12e0ff841dccff", + "frontend-modern/src/components/Storage/StorageContentCard.tsx": "faa6896330d7ea74b6886fc3d449fe8bab3b6275f00bf15e7c9d8bb24a1b2d8e", + "frontend-modern/src/components/Storage/StoragePoolDetail.tsx": "0a6401690187f4b3fa886ece2862edb34cf99122e175a1c4806384cdc8e4fed4", + "frontend-modern/src/components/Storage/StoragePoolRow.tsx": "ce177b691b800852f35ec66267150e6259d3bf9cc65b3492bbe7055879dade7c", + "frontend-modern/src/components/Storage/StoragePoolsTable.tsx": "ad1bc1967c8e4561a943ec12629f3c64ccb0423ebe6551d4a8ccd2c6afe72872", + "frontend-modern/src/components/Storage/physicalDiskAlertOwner.ts": "279aa8cd48705a31e91579bdbcf4964495301bb7478a4fa0c4549959596dd9fa", + "frontend-modern/src/components/Storage/useDiskDetailModel.ts": "b47926cc53cca725a2bca4300742b5ad211103140b809c86140648affa6d5b36", + "frontend-modern/src/components/Storage/useDiskListModel.ts": "84aa1e3c298f614a0e2e7da012ea0cc4f7f8993d9fe686d67c7865cd84298de5", + "frontend-modern/src/components/Storage/useStoragePageModel.ts": "d274908f0a7fa531921f226936bab93166676904dc8544717f7fa2172ddb37d3", + "frontend-modern/src/components/Storage/useStoragePageResources.ts": "adf35ccbe0fcae8d7559177419be0e9086720b6e039632b9c1ea829f7f7bd6c1", + "frontend-modern/src/components/Storage/useStoragePoolDetailModel.ts": "7654fbb97eaf9d1ed20b7ef41c73a0061f83f29cb989ded5e3e9b4b65c0e23ef", + "frontend-modern/src/components/Workloads/GuestDrawerOverview.tsx": "9a35f11b805b2c067afd9aeff08a2cc017caa345a72f1512a8d1bf0fc98b5615", + "frontend-modern/src/components/Workloads/GuestPhysicalDisks.tsx": "74f6b660e4d91bffad9c1c8e7580a7085036bff68e15daaf7709b4c3f615d5e3", + "frontend-modern/src/features/storageBackups/diskPresentation.ts": "ce492c2348a7e2a690b2613d0e8c0284b6de37dd5b0c0e63cd32c22bdfe150db", + "frontend-modern/src/features/storageBackups/storagePoolDetailPresentation.ts": "75765a1f902eb717aacc995143d750f0fa5957e6217442995566d00592f9effa", + "frontend-modern/src/hooks/useWorkloads.ts": "f25d948645c4de8984ff53242f0426c42ee24f2e5f916b4906c80faf50772be1", + "frontend-modern/src/types/workloads.ts": "013a5de46caf8c2d968fd1459383f0fcf0fb09efbc924b46c9aa781683d6c5e9", + "frontend-modern/src/utils/metricThresholds.ts": "cbe2299b0a2e3b917d7fa8c0c8ccabb2f7a0862708736636c74908f969e5c9ea" + }, + "routes": [ + "/proxmox/storage?tab=disks&q=West%20Production" + ], + "viewports": [ + { + "width": 1440, + "height": 900 + }, + { + "width": 375, + "height": 812 + } + ], + "states": [ + "Private mock stack (8 nodes, each with a linked Pulse agent): Vite dev server on this exact staged frontend tree, backend built from the same tree with a local-only generator overlay (never committed) pinning West Production A/B/C NVMe disks at 72C and a pve3 SATA disk at 41C. Each disk's parent is its node's merged agent machine (agent IDs host-node-mock-cluster-1-pve1/2/3)", + "Overrides set through GET/PUT /api/alerts/config: host-node-mock-cluster-1-pve1 diskTemperature 80/75, host-node-mock-cluster-1-pve3 disabled; pve2 none. diskTempByType nvme 70/65", + "Same-session control with the changed runtime files swapped to origin/main under the running Vite (then restored, full diff cmp-identical): all three NVMe disks read 'Running Hot' with red 72C cells, including pve1 under its 80C override (detail: 'Disk temperature is 72\u00b0C, at or above its 70\u00b0C alert threshold.') and pve3 with alerts off", + "After, 1440: pve1 NVMe 72C 'Healthy' with a green cell (rgb(22,163,74)); pve2 NVMe 72C 'Running Hot' with a red cell (rgb(220,38,38)); pve3 NVMe 72C uncoloured (text-base-content) and not hot (its mock SMART health is UNKNOWN, so the verdict reads 'Unknown'), and pve3's other disks lose their threshold colour too; no horizontal overflow (scrollWidth 1440)", + "After, 375: same verdicts in compact form (pve1 'Healthy' green, pve2 'Hot' red, pve3 uncoloured); scrollWidth 375", + "After, inline disk detail for pve1's NVMe: health 'Healthy' and Temperature 72\u00b0C, agreeing with its row", + "After, override moved to pve2's linked Proxmox node ID (mock-cluster-1-pve2, diskTemperature 80/75): pve2's NVMe reads 'Healthy' with a green cell, as CheckHost inherits a linked-node override; config restored afterwards", + "Not reachable on the mock estate and covered by unit tests instead: pool-drawer linked disks (storagePoolDetailPresentation/useStoragePoolDetailModel tests), the guest drawer for an agent merged into a VM (GuestPhysicalDisks.test.tsx), and the owner walk through an Unraid pool (physicalDiskAlertOwner.test.ts)" + ], + "interactions": [ + "Loaded Physical Disks filtered to West Production at 1440x900 and 375x812 and read each pinned row's verdict, Temp cell class and computed colour, before (main frontend) and after", + "Clicked pve1's NVMe row to open its inline disk detail and read its health and temperature", + "Changed alert overrides through GET/PUT /api/alerts/config (host-keyed, then node-keyed) and reloaded Physical Disks after each", + "Measured document scrollWidth at both widths" + ] +} diff --git a/frontend-modern/src/components/Storage/DiskDetail.tsx b/frontend-modern/src/components/Storage/DiskDetail.tsx index 3343c8cfd5..984a5b80ab 100644 --- a/frontend-modern/src/components/Storage/DiskDetail.tsx +++ b/frontend-modern/src/components/Storage/DiskDetail.tsx @@ -46,6 +46,8 @@ import { useDiskDetailModel } from './useDiskDetailModel'; interface DiskDetailProps { disk: Resource; nodes: Resource[]; + /** Alert override keys of the machine whose agent reports the disk. */ + alertResourceIds?: string[]; } type DiskDetailTab = 'overview' | 'history'; @@ -65,6 +67,7 @@ export const DiskDetail: Component = (props) => { liveIOAvailable, } = useDiskDetailModel({ disk: () => props.disk, + alertResourceIds: () => props.alertResourceIds, }); const rangeOptions = createMemo(() => getUnlockedHistoryRangeOptions(DISK_DETAIL_HISTORY_RANGE_OPTIONS, maxHistoryDays()), diff --git a/frontend-modern/src/components/Storage/DiskList.tsx b/frontend-modern/src/components/Storage/DiskList.tsx index db5c65b4b0..5aef960a95 100644 --- a/frontend-modern/src/components/Storage/DiskList.tsx +++ b/frontend-modern/src/components/Storage/DiskList.tsx @@ -86,6 +86,7 @@ import { getPhysicalDiskTableLayoutModeForContainer, getPhysicalDiskTemperaturePresentation, isPhysicalDiskColumnVisible, + type PhysicalDiskAlertResourceIdResolver, type PhysicalDiskTableColumnId, } from '@/features/storageBackups/diskPresentation'; import { useObservedElementWidth } from '@/hooks/useObservedElementWidth'; @@ -98,6 +99,8 @@ import { useDiskListModel } from './useDiskListModel'; interface DiskListProps { disks: Resource[]; + /** Alert override keys of the machine reporting each disk; absent means none. */ + getDiskAlertResourceIds?: PhysicalDiskAlertResourceIdResolver; nodes: Resource[]; selectedNode: string | null; sourceFilter?: string; @@ -134,6 +137,7 @@ export const DiskList: Component = (props) => { getPhysicalDiskColumnWidthStyle(layoutMode(), columnId); const model = useDiskListModel({ disks: () => props.disks, + getDiskAlertResourceIds: (disk) => props.getDiskAlertResourceIds?.(disk) ?? [], nodes: () => props.nodes, selectedNode: () => props.selectedNode, sourceFilter: () => props.sourceFilter ?? 'all', @@ -569,7 +573,11 @@ export const DiskList: Component = (props) => { colSpan={9} class={PHYSICAL_DISK_DETAIL_ROW_CELL_CLASS} > - + diff --git a/frontend-modern/src/components/Storage/Storage.tsx b/frontend-modern/src/components/Storage/Storage.tsx index 240d8291c7..c7f7f2d7ff 100644 --- a/frontend-modern/src/components/Storage/Storage.tsx +++ b/frontend-modern/src/components/Storage/Storage.tsx @@ -57,6 +57,7 @@ const Storage: Component = (props) => { focusedSummaryStorageGroupId, hoveredSummaryStorageGroupScope, physicalDisks, + getDiskAlertResourceIds, nodes, groupedRecords, expandedGroups, @@ -139,6 +140,7 @@ const Storage: Component = (props) => { StorageView; physicalDisks: () => Resource[]; + getDiskAlertResourceIds: PhysicalDiskAlertResourceIdResolver; nodes: () => Resource[]; sourceFilter: () => string; healthFilter: () => StorageHealthFilter; @@ -75,6 +77,7 @@ export const StorageContentCard: Component = (props) =>
= (props) => storageGrowthBySeriesId={props.storageGrowthBySeriesId()} storageGrowthColumnLabel={props.storageGrowthColumnLabel()} physicalDisks={props.physicalDisks()} + getDiskAlertResourceIds={props.getDiskAlertResourceIds} nodeOnlineByLabel={props.nodeOnlineByLabel()} highlightedRecordId={props.highlightedRecordId()} getRecordAlertState={props.getRecordAlertState} diff --git a/frontend-modern/src/components/Storage/StoragePoolDetail.tsx b/frontend-modern/src/components/Storage/StoragePoolDetail.tsx index ddecaebb0e..5de78815cb 100644 --- a/frontend-modern/src/components/Storage/StoragePoolDetail.tsx +++ b/frontend-modern/src/components/Storage/StoragePoolDetail.tsx @@ -39,6 +39,7 @@ import { STORAGE_DETAIL_SECTION_TITLE_SPACED_CLASS, } from '@/features/storageBackups/detailPresentation'; import type { StorageRecord } from '@/features/storageBackups/models'; +import type { PhysicalDiskAlertResourceIdResolver } from '@/features/storageBackups/diskPresentation'; import type { Resource } from '@/types/resource'; import { formatTemperature } from '@/utils/temperature'; import { StorageDetailKeyValueRow } from './StorageDetailKeyValueRow'; @@ -47,6 +48,7 @@ import { useStoragePoolDetailModel } from './useStoragePoolDetailModel'; interface StoragePoolDetailProps { record: StorageRecord; physicalDisks: Resource[]; + getDiskAlertResourceIds?: PhysicalDiskAlertResourceIdResolver; summarySeriesId: string; controlsId?: string; } @@ -67,6 +69,7 @@ export const StoragePoolDetail: Component = (props) => { } = useStoragePoolDetailModel({ record: () => props.record, physicalDisks: () => props.physicalDisks, + getDiskAlertResourceIds: props.getDiskAlertResourceIds, }); const rangeOptions = createMemo(() => getUnlockedHistoryRangeOptions(STORAGE_POOL_DETAIL_HISTORY_RANGE_OPTIONS, maxHistoryDays()), @@ -285,7 +288,10 @@ export const StoragePoolDetail: Component = (props) => { ? PHYSICAL_DISK_TEMPERATURE_LAST_KNOWN_CLASS : getLinkedDiskTemperatureTextClass( disk.temperature, - getDiskTemperatureThresholds(disk.diskType), + getDiskTemperatureThresholds( + disk.diskType, + disk.alertResourceIds, + ), ) }`} title={disk.temperatureLastKnownTitle} diff --git a/frontend-modern/src/components/Storage/StoragePoolRow.tsx b/frontend-modern/src/components/Storage/StoragePoolRow.tsx index ab435d43f3..bf98501879 100644 --- a/frontend-modern/src/components/Storage/StoragePoolRow.tsx +++ b/frontend-modern/src/components/Storage/StoragePoolRow.tsx @@ -24,6 +24,7 @@ import { STORAGE_POOL_ROW_USAGE_WRAP_CLASS, } from '@/features/storageBackups/storagePoolRowPresentation'; import type { StorageCapacityDeltaPresentation } from '@/features/storageBackups/storageCapacityDeltaPresentation'; +import type { PhysicalDiskAlertResourceIdResolver } from '@/features/storageBackups/diskPresentation'; import type { Resource } from '@/types/resource'; import { getStoragePoolCellPaddingClass, @@ -57,6 +58,7 @@ interface StoragePoolRowProps { alertHeadlineCompact?: string | null; alertHeadlineClass?: string; physicalDisks: Resource[]; + getDiskAlertResourceIds?: PhysicalDiskAlertResourceIdResolver; alertDataAttrs: { 'data-row-id': string; 'data-alert-state': string; @@ -247,6 +249,7 @@ export const StoragePoolRow: Component = (props) => { diff --git a/frontend-modern/src/components/Storage/StoragePoolsTable.tsx b/frontend-modern/src/components/Storage/StoragePoolsTable.tsx index 3b1feb13a0..3bfc614191 100644 --- a/frontend-modern/src/components/Storage/StoragePoolsTable.tsx +++ b/frontend-modern/src/components/Storage/StoragePoolsTable.tsx @@ -25,6 +25,7 @@ import { useObservedElementWidth } from '@/hooks/useObservedElementWidth'; import { resolveStorageRecordMetricResourceId } from '@/features/storageBackups/storageMetricsIdentity'; import type { StorageCapacityDeltaPresentation } from '@/features/storageBackups/storageCapacityDeltaPresentation'; import type { StorageAlertRowState } from '@/features/storageBackups/storageAlertState'; +import type { PhysicalDiskAlertResourceIdResolver } from '@/features/storageBackups/diskPresentation'; import type { Resource } from '@/types/resource'; import { StorageGroupRow } from './StorageGroupRow'; import { StoragePoolRow } from './StoragePoolRow'; @@ -45,6 +46,7 @@ type StoragePoolsTableProps = { storageGrowthBySeriesId: Map; storageGrowthColumnLabel: string; physicalDisks: Resource[]; + getDiskAlertResourceIds?: PhysicalDiskAlertResourceIdResolver; nodeOnlineByLabel: Map; highlightedRecordId: string | null; getRecordAlertState: (recordId: string) => StorageAlertRowState; @@ -195,6 +197,7 @@ export const StoragePoolsTable: Component = (props) => { alertHeadlineCompact={rowModel().alertHeadlineCompact} alertHeadlineClass={rowModel().alertHeadlineClass} physicalDisks={props.physicalDisks} + getDiskAlertResourceIds={props.getDiskAlertResourceIds} alertDataAttrs={rowModel().alertDataAttrs} /> ); diff --git a/frontend-modern/src/components/Storage/__tests__/DiskList.test.tsx b/frontend-modern/src/components/Storage/__tests__/DiskList.test.tsx index e91f30ba64..704574be10 100644 --- a/frontend-modern/src/components/Storage/__tests__/DiskList.test.tsx +++ b/frontend-modern/src/components/Storage/__tests__/DiskList.test.tsx @@ -69,12 +69,14 @@ describe('DiskList', () => { nodes: Resource[]; selectedNode: string | null; searchTerm: string; + getDiskAlertResourceIds?: (disk: Resource) => string[]; }) => render(() => { const [selectedDiskId, setSelectedDiskId] = createSignal(null); return ( { } }); + it('judges each disk by the Disk Temp override of the machine that reports it', async () => { + const getConfig = vi.spyOn(AlertsAPI, 'getConfig').mockResolvedValue({ + enabled: true, + activationState: 'active', + agentDefaults: { diskTemperature: { trigger: 55, clear: 50 } }, + diskTempByType: { nvme: { trigger: 70, clear: 65 } }, + overrides: { + 'host-pve3': { diskTemperature: { trigger: 80, clear: 75 } }, + 'host-pve5': { disabled: true }, + }, + } as unknown as AlertConfig); + try { + const view = renderDiskList({ + disks: [ + buildDisk('nvme-pve3', 'pve3', { diskType: 'nvme', temperature: 72 }), + buildDisk('nvme-pve4', 'pve4', { diskType: 'nvme', temperature: 72 }), + buildDisk('nvme-pve5', 'pve5', { diskType: 'nvme', temperature: 72 }), + ], + nodes: [], + selectedNode: null, + searchTerm: '', + getDiskAlertResourceIds: (disk) => [`host-${disk.parentId?.replace('node-', '')}`], + }); + const row = (id: string) => + within(view.container.querySelector(`[data-row-id="${id}"]`) as HTMLElement); + + await useAlertsActivation().refreshConfig(); + // pve3's 80C override puts 72C under its warning band: healthy and green. + await waitFor(() => expect(row('nvme-pve3').getByText('Healthy')).toBeInTheDocument()); + expect(row('nvme-pve3').getByText('72°C')).toHaveClass('text-green-600'); + // pve4 has no override, so the 70C NVMe trigger judges its disk. + expect(row('nvme-pve4').getByText('Running Hot')).toBeInTheDocument(); + expect(row('nvme-pve4').getByText('72°C')).toHaveClass('text-red-600'); + // pve5's alerts are switched off, so its disk is never judged hot. + expect(row('nvme-pve5').queryByText('Running Hot')).not.toBeInTheDocument(); + expect(row('nvme-pve5').getByText('Healthy')).toBeInTheDocument(); + } finally { + getConfig.mockRestore(); + eventBus.emit('org_switched', 'default'); + } + }); + it('renders SSD life and falls back to the Proxmox usage string for Belongs', () => { renderDiskList({ disks: [ diff --git a/frontend-modern/src/components/Storage/__tests__/physicalDiskAlertOwner.test.ts b/frontend-modern/src/components/Storage/__tests__/physicalDiskAlertOwner.test.ts new file mode 100644 index 0000000000..fb46f5fbb7 --- /dev/null +++ b/frontend-modern/src/components/Storage/__tests__/physicalDiskAlertOwner.test.ts @@ -0,0 +1,65 @@ +import { describe, expect, it } from 'vitest'; +import type { Resource } from '@/types/resource'; +import { + getAgentMachineAlertOverrideKeys, + getPhysicalDiskAlertResourceIds, + indexPhysicalDiskOwners, +} from '@/components/Storage/physicalDiskAlertOwner'; + +const resource = (fields: Record): Resource => + ({ name: String(fields.id), status: 'online', lastSeen: 0, ...fields }) as unknown as Resource; + +const agentMachine = resource({ + id: 'agent-tower', + type: 'agent', + agent: { agentId: 'host-tower' }, + proxmox: { sourceId: 'lab-tower', nodeName: 'tower' }, +}); +const nodeWithoutAgent = resource({ + id: 'node-bare', + type: 'agent', + proxmox: { sourceId: 'lab-bare', nodeName: 'bare' }, +}); +const unraidArray = resource({ id: 'storage-array', type: 'storage', parentId: 'agent-tower' }); +const disk = (parentId?: string) => + resource({ id: `disk-${parentId}`, type: 'physical_disk', parentId }); + +describe('physical disk alert owner', () => { + const owners = indexPhysicalDiskOwners([ + agentMachine, + nodeWithoutAgent, + unraidArray, + disk('agent-tower'), + ]); + + it('indexes only agent machines and storage pools', () => { + expect([...owners.keys()].sort()).toEqual(['agent-tower', 'storage-array']); + }); + + it('reads the reporting machine override keys in the order CheckHost reads them', () => { + // Agent ID, the canonical ID it resolves to, then the merged node. + const expected = ['host-tower', 'agent-tower', 'lab-tower']; + expect(getAgentMachineAlertOverrideKeys(agentMachine)).toEqual(expected); + expect(getPhysicalDiskAlertResourceIds(disk('agent-tower'), owners)).toEqual(expected); + // An Unraid array or cache disk hangs off its pool, which hangs off the machine. + expect(getPhysicalDiskAlertResourceIds(disk('storage-array'), owners)).toEqual(expected); + }); + + it('follows an agent merged into a guest to the guest override keys', () => { + const guest = resource({ + id: 'vm-unraid', + type: 'vm', + agent: { agentId: 'host-unraid' }, + proxmox: { instance: 'lab', node: 'tower', nodeName: 'tower', vmid: 105 }, + }); + const keys = getAgentMachineAlertOverrideKeys(guest); + expect(keys.slice(0, 2)).toEqual(['host-unraid', 'vm-unraid']); + expect(keys).toContain('lab:tower:105'); + }); + + it('leaves disks no loaded agent reports on the per-type thresholds', () => { + expect(getPhysicalDiskAlertResourceIds(disk('node-bare'), owners)).toEqual([]); + expect(getPhysicalDiskAlertResourceIds(disk('vm-not-loaded'), owners)).toEqual([]); + expect(getPhysicalDiskAlertResourceIds(disk(undefined), owners)).toEqual([]); + }); +}); diff --git a/frontend-modern/src/components/Storage/__tests__/useStoragePoolDetailModel.test.ts b/frontend-modern/src/components/Storage/__tests__/useStoragePoolDetailModel.test.ts index 9206ffef31..6fc799bc3f 100644 --- a/frontend-modern/src/components/Storage/__tests__/useStoragePoolDetailModel.test.ts +++ b/frontend-modern/src/components/Storage/__tests__/useStoragePoolDetailModel.test.ts @@ -94,6 +94,7 @@ describe('useStoragePoolDetailModel', () => { devPath: '/dev/sda', model: 'Disk A', diskType: '', + alertResourceIds: [], temperature: 44, hasIssue: false, errorCount: 0, diff --git a/frontend-modern/src/components/Storage/physicalDiskAlertOwner.ts b/frontend-modern/src/components/Storage/physicalDiskAlertOwner.ts new file mode 100644 index 0000000000..9864945c62 --- /dev/null +++ b/frontend-modern/src/components/Storage/physicalDiskAlertOwner.ts @@ -0,0 +1,66 @@ +import { guestOverrideIdCandidates } from '@/features/alerts/guestOverrideIdentity'; +import type { Resource } from '@/types/resource'; +import { getExplicitAgentIdFromResource } from '@/utils/agentResources'; + +/** + * Indexes the resources a physical disk can reach the machine that reports it + * through: machines running a Pulse agent, and the storage pools disks are + * parented to. + */ +export const indexPhysicalDiskOwners = (resources: readonly Resource[]): Map => { + const owners = new Map(); + for (const resource of resources) { + if (resource.type === 'storage' || getExplicitAgentIdFromResource(resource)) { + owners.set(resource.id, resource); + } + } + return owners; +}; + +const uniqueKeys = (...keys: Array): string[] => [ + ...new Set(keys.map((key) => key?.trim() ?? '').filter((key) => key.length > 0)), +]; + +/** + * The alert override keys of a machine running a Pulse agent, in the order + * CheckHost reads them for that agent: its own ID, the machine's canonical + * resource ID (which the backend resolves the agent ID to), then the Proxmox + * node or guest the agent is merged with. The first key with an override + * decides. + */ +export const getAgentMachineAlertOverrideKeys = (machine: Resource): string[] => { + const linkedGuestKeys = + machine.type === 'vm' || machine.type === 'system-container' + ? guestOverrideIdCandidates(machine) + : []; + return uniqueKeys( + getExplicitAgentIdFromResource(machine), + machine.id, + machine.type === 'agent' ? machine.proxmox?.sourceId : undefined, + ...linkedGuestKeys, + ); +}; + +/** + * The alert override keys that judge a physical disk's heat: those of the + * machine whose Pulse agent reports the disk, which its disk temperature + * alerts are evaluated under. That is the agent machine the disk is parented + * to or, for an Unraid array or cache disk parented to its pool, the pool's + * machine. A Disk Temp override set on that machine therefore replaces the + * per-type thresholds for every disk it reports. Empty when no agent reports + * the disk or its machine is not loaded, which leaves the per-type thresholds. + */ +export const getPhysicalDiskAlertResourceIds = ( + disk: Resource, + owners: ReadonlyMap, +): string[] => { + let parentId = disk.parentId?.trim(); + for (let hops = 0; parentId && hops < 2; hops += 1) { + const parent = owners.get(parentId); + if (!parent) break; + if (getExplicitAgentIdFromResource(parent)) return getAgentMachineAlertOverrideKeys(parent); + if (parent.type !== 'storage') break; + parentId = parent.parentId?.trim(); + } + return []; +}; diff --git a/frontend-modern/src/components/Storage/useDiskDetailModel.ts b/frontend-modern/src/components/Storage/useDiskDetailModel.ts index e8c31c049b..b5b6560850 100644 --- a/frontend-modern/src/components/Storage/useDiskDetailModel.ts +++ b/frontend-modern/src/components/Storage/useDiskDetailModel.ts @@ -17,6 +17,8 @@ import { resolvePhysicalDiskHistoryResourceId } from './diskResourceUtils'; type UseDiskDetailModelOptions = { disk: Accessor; + /** Alert override keys of the machine whose agent reports the disk. */ + alertResourceIds?: Accessor; }; export const useDiskDetailModel = (options: UseDiskDetailModelOptions) => { @@ -24,7 +26,11 @@ export const useDiskDetailModel = (options: UseDiskDetailModelOptions) => { const { getDiskTemperatureThresholds } = useAlertsActivation(); const diskData = createMemo(() => - extractPhysicalDiskPresentationData(options.disk(), getDiskTemperatureThresholds), + extractPhysicalDiskPresentationData( + options.disk(), + getDiskTemperatureThresholds, + options.alertResourceIds?.(), + ), ); const historyResourceId = createMemo(() => resolvePhysicalDiskHistoryResourceId(options.disk())); const attributeCards = createMemo(() => diff --git a/frontend-modern/src/components/Storage/useDiskListModel.ts b/frontend-modern/src/components/Storage/useDiskListModel.ts index 93428b1a15..f4acf983e3 100644 --- a/frontend-modern/src/components/Storage/useDiskListModel.ts +++ b/frontend-modern/src/components/Storage/useDiskListModel.ts @@ -7,6 +7,7 @@ import { buildPhysicalDiskRoleFilterOptions, extractPhysicalDiskPresentationData, filterAndSortPhysicalDisks, + type PhysicalDiskAlertResourceIdResolver, type PhysicalDiskPresentationData, } from '@/features/storageBackups/diskPresentation'; import { useAlertsActivation } from '@/stores/alertsActivation'; @@ -14,6 +15,7 @@ import { matchesPhysicalDiskNode } from './diskResourceUtils'; type UseDiskListModelOptions = { disks: () => Resource[]; + getDiskAlertResourceIds?: PhysicalDiskAlertResourceIdResolver; nodes: () => Resource[]; selectedNode: () => string | null; sourceFilter?: () => string; @@ -30,16 +32,25 @@ export const useDiskListModel = (options: UseDiskListModelOptions) => { const { getDiskTemperatureThresholds } = useAlertsActivation(); // Each disk's heat is judged by the user's alert disk temperature thresholds, - // so the presentation data recomputes when the alert configuration loads. + // under the Disk Temp override of the machine that reports it, so the + // presentation data recomputes when the alert configuration loads. const diskDataById = createMemo(() => - buildPhysicalDiskPresentationDataMap(options.disks(), getDiskTemperatureThresholds), + buildPhysicalDiskPresentationDataMap( + options.disks(), + getDiskTemperatureThresholds, + options.getDiskAlertResourceIds, + ), ); const roleFilterOptions = createMemo(() => buildPhysicalDiskRoleFilterOptions(options.disks())); const groupFilterOptions = createMemo(() => buildPhysicalDiskGroupFilterOptions(options.disks())); const getDiskData = (disk: Resource): PhysicalDiskPresentationData => diskDataById().get(disk.id) ?? - extractPhysicalDiskPresentationData(disk, getDiskTemperatureThresholds); + extractPhysicalDiskPresentationData( + disk, + getDiskTemperatureThresholds, + options.getDiskAlertResourceIds?.(disk), + ); const selectedNodeResource = createMemo( () => options.nodes().find((node) => node.id === options.selectedNode()) ?? null, diff --git a/frontend-modern/src/components/Storage/useStoragePageModel.ts b/frontend-modern/src/components/Storage/useStoragePageModel.ts index 90ed4045f4..d288072e34 100644 --- a/frontend-modern/src/components/Storage/useStoragePageModel.ts +++ b/frontend-modern/src/components/Storage/useStoragePageModel.ts @@ -66,6 +66,7 @@ export const useStoragePageModel = (options: UseStoragePageModelOptions = {}) => storageResources, nodes, physicalDisks, + getDiskAlertResourceIds, cephResources, alertsEnabled, } = useStoragePageResources({ resourceSource: options.resourceSource }); @@ -471,6 +472,7 @@ export const useStoragePageModel = (options: UseStoragePageModelOptions = {}) => initialDataReceived: surfaceInitialDataReceived, nodeOptions, physicalDisks, + getDiskAlertResourceIds, nodes, groupedRecords, expandedGroups, diff --git a/frontend-modern/src/components/Storage/useStoragePageResources.ts b/frontend-modern/src/components/Storage/useStoragePageResources.ts index 721516b6a2..9e343d5c5b 100644 --- a/frontend-modern/src/components/Storage/useStoragePageResources.ts +++ b/frontend-modern/src/components/Storage/useStoragePageResources.ts @@ -3,6 +3,7 @@ import { useWebSocket } from '@/contexts/appRuntime'; import { useUnifiedResources } from '@/hooks/useUnifiedResources'; import { useAlertsActivation } from '@/stores/alertsActivation'; import type { Resource, ResourceType } from '@/types/resource'; +import { getPhysicalDiskAlertResourceIds, indexPhysicalDiskOwners } from './physicalDiskAlertOwner'; const STORAGE_PAGE_RESOURCES_QUERY = 'type=agent,pbs,storage,physical_disk,ceph'; @@ -42,6 +43,11 @@ export const useStoragePageResources = (options: UseStoragePageResourcesOptions ...resourceSource, resources: storageResourceRows, }; + // A disk's heat is judged under the alert overrides of the machine whose + // agent reports it; this page loads those machines and pools with the disks. + const physicalDiskOwners = createMemo(() => indexPhysicalDiskOwners(resourceSource.resources())); + const getDiskAlertResourceIds = (disk: Resource): string[] => + getPhysicalDiskAlertResourceIds(disk, physicalDiskOwners()); const alertsEnabled = alertsActivation.detectionEnabled; return { @@ -53,6 +59,7 @@ export const useStoragePageResources = (options: UseStoragePageResourcesOptions reconnect, nodes, physicalDisks, + getDiskAlertResourceIds, cephResources, storageResources, alertsEnabled, diff --git a/frontend-modern/src/components/Storage/useStoragePoolDetailModel.ts b/frontend-modern/src/components/Storage/useStoragePoolDetailModel.ts index bf22c321b0..13bcad1060 100644 --- a/frontend-modern/src/components/Storage/useStoragePoolDetailModel.ts +++ b/frontend-modern/src/components/Storage/useStoragePoolDetailModel.ts @@ -1,6 +1,7 @@ import { Accessor, createMemo, createSignal } from 'solid-js'; import type { HistoryTimeRange } from '@/api/charts'; import type { StorageRecord } from '@/features/storageBackups/models'; +import type { PhysicalDiskAlertResourceIdResolver } from '@/features/storageBackups/diskPresentation'; import { buildStoragePoolDetailConfigRows, buildStoragePoolDetailTopologyRows, @@ -13,6 +14,7 @@ import type { Resource } from '@/types/resource'; type UseStoragePoolDetailModelOptions = { record: Accessor; physicalDisks: Accessor; + getDiskAlertResourceIds?: PhysicalDiskAlertResourceIdResolver; }; export const useStoragePoolDetailModel = (options: UseStoragePoolDetailModelOptions) => { @@ -22,7 +24,11 @@ export const useStoragePoolDetailModel = (options: UseStoragePoolDetailModelOpti const configRows = createMemo(() => buildStoragePoolDetailConfigRows(options.record())); const zfsSummary = createMemo(() => buildStoragePoolDetailZfsSummary(options.record())); const linkedDisks = createMemo(() => - getStoragePoolLinkedDisks(options.record(), options.physicalDisks()), + getStoragePoolLinkedDisks( + options.record(), + options.physicalDisks(), + options.getDiskAlertResourceIds, + ), ); const topologyRows = createMemo(() => buildStoragePoolDetailTopologyRows(options.record(), linkedDisks()), diff --git a/frontend-modern/src/components/Workloads/GuestDrawerOverview.tsx b/frontend-modern/src/components/Workloads/GuestDrawerOverview.tsx index 9b30b81b1f..072c4c3e4a 100644 --- a/frontend-modern/src/components/Workloads/GuestDrawerOverview.tsx +++ b/frontend-modern/src/components/Workloads/GuestDrawerOverview.tsx @@ -29,6 +29,7 @@ import { } from './guestDrawerModel'; import type { NestedWorkloadContext } from './nestedWorkloadContext'; import { WORKLOAD_ACTION_AGENT_LABEL } from './workloadAgentReadiness'; +import { getWorkloadAlertResourceIdCandidates } from './workloadTopology'; import { RaidCard } from '@/components/shared/cards/RaidCard'; import type { GuestDrawerProps } from './guestDrawerModel'; @@ -326,7 +327,14 @@ export function GuestDrawerOverview(props: GuestDrawerOverviewProps) { } > - + diff --git a/frontend-modern/src/components/Workloads/GuestPhysicalDisks.tsx b/frontend-modern/src/components/Workloads/GuestPhysicalDisks.tsx index 2b057ff711..fa6f2877d7 100644 --- a/frontend-modern/src/components/Workloads/GuestPhysicalDisks.tsx +++ b/frontend-modern/src/components/Workloads/GuestPhysicalDisks.tsx @@ -19,10 +19,15 @@ const LazyDiskDetail = lazy(() => import('@/components/Storage/DiskDetail').then(({ DiskDetail }) => ({ default: DiskDetail })), ); -const GuestPhysicalDiskRow: Component<{ disk: Resource }> = (props) => { +const GuestPhysicalDiskRow: Component<{ disk: Resource; alertResourceIds: string[] }> = (props) => { const [expanded, setExpanded] = createSignal(false); const { getDiskTemperatureThresholds } = useAlertsActivation(); - const data = () => extractPhysicalDiskPresentationData(props.disk, getDiskTemperatureThresholds); + const data = () => + extractPhysicalDiskPresentationData( + props.disk, + getDiskTemperatureThresholds, + props.alertResourceIds, + ); const health = () => getPhysicalDiskHealthStatus(data()); const temperature = () => getPhysicalDiskTemperaturePresentation(data()); const label = () => data().model || data().devPath || props.disk.name; @@ -68,7 +73,11 @@ const GuestPhysicalDiskRow: Component<{ disk: Resource }> = (props) => {

} > - +
@@ -76,7 +85,15 @@ const GuestPhysicalDiskRow: Component<{ disk: Resource }> = (props) => { ); }; -export const GuestPhysicalDisks: Component<{ parentId: string }> = (props) => { +export const GuestPhysicalDisks: Component<{ + parentId: string; + /** + * Alert override keys judging these disks' heat, in the order the backend + * reads them: the guest's own agent reports them, so its Disk Temp override + * applies, then the guest's canonical ID and override keys. + */ + alertResourceIds?: string[]; +}> = (props) => { // The Proxmox Overview snapshot deliberately excludes physical disks. Query // only the open guest's direct children rather than hydrating every disk in // a large estate (or mistaking the VMID-based table key for the resource ID). @@ -104,7 +121,11 @@ export const GuestPhysicalDisks: Component<{ parentId: string }> = (props) => { Physical Disks & SMART ({disks().length})
- {(disk) => } + + {(disk) => ( + + )} +
diff --git a/frontend-modern/src/components/Workloads/__tests__/GuestPhysicalDisks.test.tsx b/frontend-modern/src/components/Workloads/__tests__/GuestPhysicalDisks.test.tsx index 0b316595db..67dfde10e2 100644 --- a/frontend-modern/src/components/Workloads/__tests__/GuestPhysicalDisks.test.tsx +++ b/frontend-modern/src/components/Workloads/__tests__/GuestPhysicalDisks.test.tsx @@ -1,6 +1,10 @@ -import { cleanup, fireEvent, render, screen, within } from '@solidjs/testing-library'; +import { cleanup, fireEvent, render, screen, waitFor, within } from '@solidjs/testing-library'; import { createSignal } from 'solid-js'; import { afterEach, beforeAll, beforeEach, describe, expect, it, vi } from 'vitest'; +import { AlertsAPI } from '@/api/alerts'; +import { useAlertsActivation } from '@/stores/alertsActivation'; +import { eventBus } from '@/stores/events'; +import type { AlertConfig } from '@/types/alerts'; import type { Resource } from '@/types/resource'; import { GuestPhysicalDisks } from '../GuestPhysicalDisks'; @@ -83,6 +87,38 @@ describe('GuestPhysicalDisks', () => { expect(within(card).getAllByTestId('guest-physical-disk')).toHaveLength(1); }); + it("judges the guest's disks by its own agent's Disk Temp override", async () => { + const getConfig = vi.spyOn(AlertsAPI, 'getConfig').mockResolvedValue({ + enabled: true, + activationState: 'active', + agentDefaults: { diskTemperature: { trigger: 55, clear: 50 } }, + diskTempByType: { sata: { trigger: 55, clear: 50 } }, + overrides: { 'host-guest-101': { diskTemperature: { trigger: 65, clear: 60 } } }, + } as unknown as AlertConfig); + try { + queryState.resources = [ + { ...disk, physicalDisk: { ...disk.physicalDisk!, temperature: 58 } } as Resource, + ]; + await useAlertsActivation().refreshConfig(); + const guestCard = render(() => ( + + )); + // 58C passes the 55C SATA trigger but not the agent's 65C override. + await waitFor(() => + expect(within(guestCard.container).getByText('Healthy')).toBeInTheDocument(), + ); + guestCard.unmount(); + + const unmatched = render(() => ( + + )); + expect(within(unmatched.container).getByText('Running Hot')).toBeInTheDocument(); + } finally { + getConfig.mockRestore(); + eventBus.emit('org_switched', 'default'); + } + }); + it('does not add an empty storage card to an uninstrumented guest', () => { render(() => ); expect(screen.queryByTestId('guest-physical-disks')).toBeNull(); diff --git a/frontend-modern/src/features/storageBackups/__tests__/diskPresentation.test.ts b/frontend-modern/src/features/storageBackups/__tests__/diskPresentation.test.ts index cb94ecd9f7..f927939f4a 100644 --- a/frontend-modern/src/features/storageBackups/__tests__/diskPresentation.test.ts +++ b/frontend-modern/src/features/storageBackups/__tests__/diskPresentation.test.ts @@ -1,4 +1,5 @@ import { describe, expect, it } from 'vitest'; +import type { AlertConfig } from '@/types/alerts'; import type { Resource } from '@/types/resource'; import { buildPhysicalDiskPresentationDataMap, @@ -401,6 +402,31 @@ describe('diskPresentation', () => { } }); + it('judges a disk under the Disk Temp override of the machine that reports it', () => { + const config = { + agentDefaults: { diskTemperature: { trigger: 55, clear: 50 } }, + diskTempByType: { nvme: { trigger: 70, clear: 65 } }, + overrides: { 'host-raised': { diskTemperature: { trigger: 80, clear: 75 } } }, + } as unknown as AlertConfig; + const resolve = (diskType: string, alertResourceIds?: string[]) => + resolveDiskTemperatureDisplayThresholds(config, diskType, alertResourceIds); + const raisedHostDisk = buildDisk('nvme0n1', { diskType: 'nvme', temperature: 72 }); + const plainHostDisk = buildDisk('nvme1n1', { diskType: 'nvme', temperature: 72 }); + const dataMap = buildPhysicalDiskPresentationDataMap( + [raisedHostDisk, plainHostDisk], + resolve, + (disk) => (disk === raisedHostDisk ? ['host-raised'] : ['host-plain']), + ); + + expect(dataMap.get('nvme0n1')!.temperatureThresholds).toEqual({ warning: 75, critical: 80 }); + expect(getPhysicalDiskHealthStatus(dataMap.get('nvme0n1')!).label).toBe('Healthy'); + expect(getPhysicalDiskHealthStatus(dataMap.get('nvme1n1')!).label).toBe('Running Hot'); + expect( + extractPhysicalDiskPresentationData(raisedHostDisk, resolve, ['host-raised']) + .temperatureThresholds, + ).toEqual({ warning: 75, critical: 80 }); + }); + it('calls no disk hot when its temperature alerting is off', () => { const off = () => null; const data = extractPhysicalDiskPresentationData(buildDisk('sda', { temperature: 80 }), off); diff --git a/frontend-modern/src/features/storageBackups/__tests__/storagePoolDetailPresentation.test.ts b/frontend-modern/src/features/storageBackups/__tests__/storagePoolDetailPresentation.test.ts index 42a4b15b04..9e26cb4f33 100644 --- a/frontend-modern/src/features/storageBackups/__tests__/storagePoolDetailPresentation.test.ts +++ b/frontend-modern/src/features/storageBackups/__tests__/storagePoolDetailPresentation.test.ts @@ -231,6 +231,7 @@ describe('storagePoolDetailPresentation', () => { state: 'online', sizeLabel: '1000 B', diskType: '', + alertResourceIds: [], temperature: 44, hasIssue: true, spunDown: false, @@ -313,9 +314,13 @@ describe('storagePoolDetailPresentation', () => { }, ] as Resource[]; - const linkedDisks = getStoragePoolLinkedDisks(record, disks); + const linkedDisks = getStoragePoolLinkedDisks(record, disks, (disk) => [`owner-of-${disk.id}`]); expect(linkedDisks.map((disk) => disk.role)).toEqual(['parity', 'data']); + // Each linked disk carries the override keys its Temp colour is judged under. + expect(linkedDisks.map((disk) => disk.alertResourceIds)).toEqual( + linkedDisks.map((disk) => [`owner-of-${disk.id}`]), + ); expect(linkedDisks[0].temperatureLastKnownTitle).toBeUndefined(); expect(linkedDisks[1]).toEqual( expect.objectContaining({ diff --git a/frontend-modern/src/features/storageBackups/diskPresentation.ts b/frontend-modern/src/features/storageBackups/diskPresentation.ts index 4d8c996f08..ea16f6468b 100644 --- a/frontend-modern/src/features/storageBackups/diskPresentation.ts +++ b/frontend-modern/src/features/storageBackups/diskPresentation.ts @@ -76,9 +76,10 @@ export interface PhysicalDiskPresentationData { target?: string; temperature: number; /** - * The alert disk temperature thresholds for this disk's type. The Temp cell - * colours the reading by them and the Health verdict judges heat by them. - * Absent means the factory thresholds for the type. + * The alert disk temperature thresholds for this disk's type, or the Disk + * Temp override of the machine whose agent reports it. The Temp cell colours + * the reading by them and the Health verdict judges heat by them. Absent + * means the factory thresholds for the type. */ temperatureThresholds?: MetricDisplayThresholds | null; rpm: number; @@ -410,14 +411,23 @@ export function getPhysicalDiskHostLabel( } /** - * Resolves the alert disk temperature thresholds for a disk type. Callers with - * the user's alert configuration pass `getDiskTemperatureThresholds` from the - * alerts activation store; the default is the factory configuration. + * Resolves the alert disk temperature thresholds for a disk type, under the + * alert override keys of the machine whose agent reports the disk. Callers + * with the user's alert configuration pass `getDiskTemperatureThresholds` from + * the alerts activation store; the default is the factory configuration. */ export type PhysicalDiskTemperatureThresholdResolver = ( diskType: string, + alertResourceIds?: string[], ) => MetricDisplayThresholds | null; +/** + * Returns the alert override keys of the machine whose agent reports a disk, + * so that machine's Disk Temp override judges the disk's heat. Empty when no + * agent reports it. + */ +export type PhysicalDiskAlertResourceIdResolver = (disk: Resource) => string[]; + const resolveFactoryDiskTemperatureThresholds: PhysicalDiskTemperatureThresholdResolver = ( diskType, ) => resolveDiskTemperatureDisplayThresholds(null, diskType); @@ -433,6 +443,7 @@ export const getPhysicalDiskTemperatureThresholds = ( export function extractPhysicalDiskPresentationData( resource: Resource, resolveTemperatureThresholds: PhysicalDiskTemperatureThresholdResolver = resolveFactoryDiskTemperatureThresholds, + alertResourceIds: string[] = [], ): PhysicalDiskPresentationData { const pd = resource.physicalDisk || ((resource.platformData as any)?.physicalDisk ?? {}); const diskNode = getPhysicalDiskNodeIdentity(resource); @@ -467,7 +478,7 @@ export function extractPhysicalDiskPresentationData( health: pd.health || 'UNKNOWN', wearout: pd.wearout ?? -1, temperature: pd.temperature ?? 0, - temperatureThresholds: resolveTemperatureThresholds(pd.diskType || ''), + temperatureThresholds: resolveTemperatureThresholds(pd.diskType || '', alertResourceIds), rpm: pd.rpm ?? 0, used: pd.used || '', storageRole: pd.storageRole, @@ -541,10 +552,18 @@ export function getPhysicalDiskCollectionMessages(disk: PhysicalDiskPresentation export function buildPhysicalDiskPresentationDataMap( disks: Resource[], resolveTemperatureThresholds?: PhysicalDiskTemperatureThresholdResolver, + getAlertResourceIds?: PhysicalDiskAlertResourceIdResolver, ): Map { const map = new Map(); for (const disk of disks || []) { - map.set(disk.id, extractPhysicalDiskPresentationData(disk, resolveTemperatureThresholds)); + map.set( + disk.id, + extractPhysicalDiskPresentationData( + disk, + resolveTemperatureThresholds, + getAlertResourceIds?.(disk), + ), + ); } return map; } diff --git a/frontend-modern/src/features/storageBackups/storagePoolDetailPresentation.ts b/frontend-modern/src/features/storageBackups/storagePoolDetailPresentation.ts index cca8d9a7f4..560a98d001 100644 --- a/frontend-modern/src/features/storageBackups/storagePoolDetailPresentation.ts +++ b/frontend-modern/src/features/storageBackups/storagePoolDetailPresentation.ts @@ -1,5 +1,8 @@ import type { HistoryTimeRange, ResourceType as HistoryChartResourceType } from '@/api/charts'; -import { getPhysicalDiskTemperaturePresentation } from '@/features/storageBackups/diskPresentation'; +import { + getPhysicalDiskTemperaturePresentation, + type PhysicalDiskAlertResourceIdResolver, +} from '@/features/storageBackups/diskPresentation'; import type { StorageRecord } from '@/features/storageBackups/models'; import { getStorageRecordContent, @@ -27,6 +30,8 @@ export type StoragePoolDetailLinkedDisk = { state: string; sizeLabel: string; diskType: string; + /** Alert override keys of the machine whose agent reports the disk. */ + alertResourceIds?: string[]; temperature: number; /** Set only when the temperature is a retained, not current, reading. */ temperatureLastKnownTitle?: string; @@ -402,6 +407,7 @@ const normalizeDevicePath = (value: string): string => value.trim().replace(/^\/ export function getStoragePoolLinkedDisks( record: StorageRecord, physicalDisks: Resource[], + getAlertResourceIds?: PhysicalDiskAlertResourceIdResolver, ): StoragePoolDetailLinkedDisk[] { const pool = getStorageRecordZfsPool(record); const unraidStorageGroup = getUnraidStorageGroup(record).toLowerCase(); @@ -432,6 +438,7 @@ export function getStoragePoolLinkedDisks( state: readDiskState(disk), sizeLabel: readDiskSizeLabel(disk), diskType: readDiskType(disk), + alertResourceIds: getAlertResourceIds?.(disk) ?? [], temperature: readDiskTemperature(disk), temperatureLastKnownTitle: readDiskTemperatureLastKnownTitle(disk), hasIssue: readDiskHasIssue(disk), diff --git a/frontend-modern/src/hooks/__tests__/useWorkloads.test.ts b/frontend-modern/src/hooks/__tests__/useWorkloads.test.ts index b7f5265e88..8a19338644 100644 --- a/frontend-modern/src/hooks/__tests__/useWorkloads.test.ts +++ b/frontend-modern/src/hooks/__tests__/useWorkloads.test.ts @@ -99,11 +99,12 @@ describe('useWorkloads', () => { }, ); - it('preserves linked-agent RAID arrays from the VM resource for guest detail', async () => { + it('preserves linked-agent RAID arrays and agent ID from the VM resource for guest detail', async () => { const [snapshot] = createSignal([ { ...sampleResource, agent: { + agentId: 'host-guest-agent', agentVersion: '6.4.5', raid: [ { @@ -130,6 +131,8 @@ describe('useWorkloads', () => { try { await flushAsync(); expect(result!.workloads()[0]?.canonicalResourceId).toBe(sampleResource.id); + // The guest drawer judges the agent's disks under this ID's overrides. + expect(result!.workloads()[0]?.agentId).toBe('host-guest-agent'); expect(result!.workloads()[0]?.agentRaid).toMatchObject([ { device: '/dev/md0', level: 'raid1', state: 'clean' }, ]); diff --git a/frontend-modern/src/hooks/useWorkloads.ts b/frontend-modern/src/hooks/useWorkloads.ts index 5f53347254..35f238e8cb 100644 --- a/frontend-modern/src/hooks/useWorkloads.ts +++ b/frontend-modern/src/hooks/useWorkloads.ts @@ -137,6 +137,7 @@ type APIResource = { vcpus?: number; }; agent?: { + agentId?: string; hostname?: string; uptimeSeconds?: number; osName?: string; @@ -608,6 +609,7 @@ const mapResourceToWorkload = (resource: APIResource): WorkloadGuest | null => { osVersion: resource.agent?.osVersion ?? resource.proxmox?.osVersion, agentVersion: resource.agent?.agentVersion, agentKind: resource.agent?.agentVersion ? 'pulse' : undefined, + agentId: resource.agent?.agentId, agentRaid: resource.agent?.raid, networkInterfaces: mapNetworkInterfaces(resource.agent?.networkInterfaces), networkIn: finiteMetricNumber(resource.metrics?.netIn?.value) ?? 0, diff --git a/frontend-modern/src/types/workloads.ts b/frontend-modern/src/types/workloads.ts index 00425a6612..411d3c3777 100644 --- a/frontend-modern/src/types/workloads.ts +++ b/frontend-modern/src/types/workloads.ts @@ -56,6 +56,8 @@ export type WorkloadGuest = (VM | Container) & { containerRuntime?: string; /** Identifies the producer of agentVersion instead of inferring it from VM type. */ agentKind?: 'pulse' | 'qemu-guest'; + /** ID of the Pulse agent running in this guest, the key of its alert overrides. */ + agentId?: string; /** RAID arrays reported by the linked Pulse agent, not Proxmox virtual disks. */ agentRaid?: HostRAIDArray[]; updateStatus?: DockerContainerUpdateStatus; diff --git a/frontend-modern/src/utils/__tests__/metricThresholds.test.ts b/frontend-modern/src/utils/__tests__/metricThresholds.test.ts index 59cd271bc2..a0ed2a6380 100644 --- a/frontend-modern/src/utils/__tests__/metricThresholds.test.ts +++ b/frontend-modern/src/utils/__tests__/metricThresholds.test.ts @@ -397,6 +397,50 @@ describe('metricThresholds', () => { }); }); + it('judges a disk by the alert overrides of the machine that reports it', () => { + const config = { + enabled: true, + guestDefaults: {}, + nodeDefaults: {}, + agentDefaults: { diskTemperature: { trigger: 55, clear: 50 } }, + diskTempByType: { nvme: { trigger: 70, clear: 65 } }, + storageDefault: { trigger: 85, clear: 80 }, + overrides: { + 'host-raised': { diskTemperature: { trigger: 80, clear: 75 } }, + 'host-memory-only': { memory: { trigger: 95, clear: 90 } }, + 'host-off': { disabled: true }, + }, + } as AlertConfig; + // The machine's first matching key decides, as CheckHost's chain does. + expect( + resolveDiskTemperatureDisplayThresholds(config, 'nvme', [ + 'agent:host-raised', + 'host-raised', + ]), + ).toEqual({ warning: 75, critical: 80 }); + expect(resolveDiskTemperatureDisplayThresholds(config, 'nvme', ['host-memory-only'])).toEqual( + { + warning: 65, + critical: 70, + }, + ); + expect(resolveDiskTemperatureDisplayThresholds(config, 'nvme', ['host-plain'])).toEqual({ + warning: 65, + critical: 70, + }); + // A host whose alerts are switched off raises no disk temperature alert, + // so none of its disks is judged hot. + expect(resolveDiskTemperatureDisplayThresholds(config, 'nvme', ['host-off'])).toBeNull(); + // An agent default that switches every host's alerts off does the same. + const agentsOff = { + ...config, + agentDefaults: { ...config.agentDefaults, disabled: true }, + } as AlertConfig; + expect( + resolveDiskTemperatureDisplayThresholds(agentsOff, 'nvme', ['host-raised']), + ).toBeNull(); + }); + it('falls back to seeded per-type disk temperature defaults without config', () => { expect(resolveDiskTemperatureDisplayThresholds(null, 'nvme')).toEqual({ warning: 65, diff --git a/frontend-modern/src/utils/metricThresholds.ts b/frontend-modern/src/utils/metricThresholds.ts index ee533c19ae..a892bb4ef2 100644 --- a/frontend-modern/src/utils/metricThresholds.ts +++ b/frontend-modern/src/utils/metricThresholds.ts @@ -271,7 +271,10 @@ export const resolveMetricDisplayThresholds = ( * host (or inherited linked resource) wins, then the per-type map * (diskTempByType: nvme/sas/sata), then the global agent default. A * switched-off agent default switches every type off, as disk temperature - * alerts do. Null means disk temperature alerting is off for this disk. + * alerts do, and so does an override (or agent default) that switches the + * host's alerts off. Null means disk temperature alerting is off for this + * disk. `resourceIds` are the alert override keys of the machine whose agent + * reports the disk, in the order the backend reads them. */ export const resolveDiskTemperatureDisplayThresholds = ( config: AlertConfig | null, @@ -282,6 +285,9 @@ export const resolveDiskTemperatureDisplayThresholds = ( const normalizedType = (diskType ?? '').trim().toLowerCase(); const override = findOverride(config?.overrides, resourceIds); + if (override?.disabled || config?.agentDefaults?.disabled === true) { + return null; + } const overrideValue = getOverrideValue(override, 'diskTemperature'); if (overrideValue !== undefined) { return resolveThreshold(overrideValue, FACTORY_AGENT_DEFAULTS.diskTemperature, margin); diff --git a/internal/ai/alert_threshold_adapter.go b/internal/ai/alert_threshold_adapter.go index 8509f06899..53163e2f3b 100644 --- a/internal/ai/alert_threshold_adapter.go +++ b/internal/ai/alert_threshold_adapter.go @@ -69,12 +69,13 @@ func (a *AlertThresholdAdapter) GetGuestDiskThreshold() float64 { } // GetDiskTemperatureThreshold returns the disk temperature alert trigger and -// clear values for a disk type, resolved exactly as disk temperature alerts -// resolve them (alerts.Manager.DiskTemperatureThreshold). -func (a *AlertThresholdAdapter) GetDiskTemperatureThreshold(diskType string) (float64, float64) { +// clear values for a disk of a type that a host agent reports, resolved +// exactly as disk temperature alerts resolve them +// (alerts.Manager.HostDiskTemperatureThreshold), host overrides included. +func (a *AlertThresholdAdapter) GetDiskTemperatureThreshold(host alerts.DiskTemperatureHost, diskType string) (float64, float64) { var threshold *alerts.HysteresisThreshold if a.manager != nil { - threshold = a.manager.DiskTemperatureThreshold(diskType) + threshold = a.manager.HostDiskTemperatureThreshold(host, diskType) } else { threshold = alerts.DefaultDiskTemperatureThreshold(diskType) } diff --git a/internal/ai/alert_threshold_adapter_test.go b/internal/ai/alert_threshold_adapter_test.go index f65840ecbd..67347911f6 100644 --- a/internal/ai/alert_threshold_adapter_test.go +++ b/internal/ai/alert_threshold_adapter_test.go @@ -84,7 +84,7 @@ func TestAlertThresholdAdapter_Fallbacks(t *testing.T) { } func TestAlertThresholdAdapter_DiskTemperatureFollowsAlertPolicy(t *testing.T) { - if trigger, clear := NewAlertThresholdAdapter(nil).GetDiskTemperatureThreshold("nvme"); trigger != 70 || clear != 65 { + if trigger, clear := NewAlertThresholdAdapter(nil).GetDiskTemperatureThreshold(alerts.DiskTemperatureHost{}, "nvme"); trigger != 70 || clear != 65 { t.Fatalf("no manager: nvme = %v/%v, want the factory 70/65", trigger, clear) } @@ -93,17 +93,42 @@ func TestAlertThresholdAdapter_DiskTemperatureFollowsAlertPolicy(t *testing.T) { cfg.DiskTempByType["nvme"] = alerts.HysteresisThreshold{Trigger: 75, Clear: 70} mgr.UpdateConfig(cfg) a := NewAlertThresholdAdapter(mgr) - if trigger, clear := a.GetDiskTemperatureThreshold("nvme"); trigger != 75 || clear != 70 { + if trigger, clear := a.GetDiskTemperatureThreshold(alerts.DiskTemperatureHost{}, "nvme"); trigger != 75 || clear != 70 { t.Fatalf("raised nvme = %v/%v, want 75/70", trigger, clear) } - if trigger, clear := a.GetDiskTemperatureThreshold("sata"); trigger != 55 || clear != 50 { + if trigger, clear := a.GetDiskTemperatureThreshold(alerts.DiskTemperatureHost{}, "sata"); trigger != 55 || clear != 50 { t.Fatalf("sata = %v/%v, want 55/50", trigger, clear) } cfg = mgr.GetConfig() cfg.AgentDefaults.DiskTemperature = &alerts.HysteresisThreshold{Trigger: 0, Clear: 0} mgr.UpdateConfig(cfg) - if trigger, clear := a.GetDiskTemperatureThreshold("nvme"); trigger != 0 || clear != 0 { + if trigger, clear := a.GetDiskTemperatureThreshold(alerts.DiskTemperatureHost{}, "nvme"); trigger != 0 || clear != 0 { t.Fatalf("disk temperature alerting off: nvme = %v/%v, want 0/0", trigger, clear) } } + +// A host's Disk Temp override reaches Patrol for that host's disks only, and a +// host whose alerts are switched off has no disk heat policy at all. +func TestAlertThresholdAdapter_DiskTemperatureHonoursHostOverrides(t *testing.T) { + mgr := alerts.NewManager() + cfg := mgr.GetConfig() + cfg.Overrides = map[string]alerts.ThresholdConfig{ + "host-raised": {DiskTemperature: &alerts.HysteresisThreshold{Trigger: 80, Clear: 75}}, + "host-off": {Disabled: true}, + } + mgr.UpdateConfig(cfg) + a := NewAlertThresholdAdapter(mgr) + + for host, want := range map[string][2]float64{ + "host-raised": {80, 75}, + "host-plain": {70, 65}, + "host-off": {0, 0}, + "": {70, 65}, + } { + trigger, clear := a.GetDiskTemperatureThreshold(alerts.DiskTemperatureHost{ID: host}, "nvme") + if trigger != want[0] || clear != want[1] { + t.Errorf("host %q nvme = %v/%v, want %v/%v", host, trigger, clear, want[0], want[1]) + } + } +} diff --git a/internal/ai/mock_test.go b/internal/ai/mock_test.go index 91524141ab..6fb53b6a55 100644 --- a/internal/ai/mock_test.go +++ b/internal/ai/mock_test.go @@ -63,7 +63,7 @@ func (m *mockThresholdProvider) GetGuestCPUThreshold() float64 { return 0 } func (m *mockThresholdProvider) GetGuestMemoryThreshold() float64 { return m.guestMem } func (m *mockThresholdProvider) GetGuestDiskThreshold() float64 { return m.guestDisk } func (m *mockThresholdProvider) GetStorageThreshold() float64 { return m.storage } -func (m *mockThresholdProvider) GetDiskTemperatureThreshold(diskType string) (float64, float64) { +func (m *mockThresholdProvider) GetDiskTemperatureThreshold(_ alerts.DiskTemperatureHost, diskType string) (float64, float64) { if limits, ok := m.diskTemperature[diskType]; ok { return limits[0], limits[1] } diff --git a/internal/ai/patrol.go b/internal/ai/patrol.go index a630554703..d349e60417 100644 --- a/internal/ai/patrol.go +++ b/internal/ai/patrol.go @@ -67,9 +67,10 @@ type ThresholdProvider interface { // GetStorageThreshold returns the usage alert trigger threshold for storage (0-100%) GetStorageThreshold() float64 // GetDiskTemperatureThreshold returns the disk temperature alert trigger - // and clear values (°C) for a disk type. A zero trigger means disk - // temperature alerting is off. - GetDiskTemperatureThreshold(diskType string) (trigger, clear float64) + // and clear values (°C) for a disk of the given type that the given host + // agent reports. A host with no ID means no agent reports the disk. A zero + // trigger means disk temperature alerting is off for it. + GetDiskTemperatureThreshold(host alerts.DiskTemperatureHost, diskType string) (trigger, clear float64) } // AlertResolver provides the ability to review and resolve alerts diff --git a/internal/ai/patrol_ai.go b/internal/ai/patrol_ai.go index a14000ab2e..e8462ae8f8 100644 --- a/internal/ai/patrol_ai.go +++ b/internal/ai/patrol_ai.go @@ -24,6 +24,7 @@ import ( "github.com/rcourtman/pulse-go-rewrite/internal/ai/memory" "github.com/rcourtman/pulse-go-rewrite/internal/ai/providers" "github.com/rcourtman/pulse-go-rewrite/internal/ai/tools" + "github.com/rcourtman/pulse-go-rewrite/internal/alerts" "github.com/rcourtman/pulse-go-rewrite/internal/config" "github.com/rcourtman/pulse-go-rewrite/internal/models" "github.com/rcourtman/pulse-go-rewrite/internal/servicediscovery" @@ -2447,6 +2448,10 @@ func patrolPhysicalDiskRows(snap patrolRuntimeState, scopedSet map[string]bool) if urp != nil { diskResources := urp.GetByType(unifiedresources.ResourceTypePhysicalDisk) rows := make([]patrolPhysicalDiskRow, 0, len(diskResources)) + var owners map[string]unifiedresources.Resource + if len(diskResources) > 0 { + owners = physicalDiskOwnerIndex(urp) + } for _, r := range diskResources { if !seedIsInScope(scopedSet, r.ID) || r.PhysicalDisk == nil { continue @@ -2479,7 +2484,7 @@ func patrolPhysicalDiskRows(snap patrolRuntimeState, scopedSet map[string]bool) temperature: tools.SplitDiskTemperature(r.PhysicalDisk.Temperature, r.PhysicalDisk.Collection), smartEvidence: unifiedPhysicalDiskSMARTIssueParts(r.PhysicalDisk.SMART), } - row.temperatureLimits = snap.diskTemperatureLimits(row.diskType) + row.temperatureLimits = snap.diskTemperatureLimits(physicalDiskTemperatureHost(r, owners), row.diskType) rows = append(rows, row) } return rows @@ -2522,7 +2527,7 @@ func patrolPhysicalDiskRows(snap patrolRuntimeState, scopedSet map[string]bool) temperature: tools.SplitDiskTemperature(d.Temperature, d.Collection), smartEvidence: modelPhysicalDiskSMARTIssueParts(d.SmartAttributes), } - row.temperatureLimits = snap.diskTemperatureLimits(row.diskType) + row.temperatureLimits = snap.diskTemperatureLimits(alerts.DiskTemperatureHost{}, row.diskType) rows = append(rows, row) } return rows diff --git a/internal/ai/patrol_disk_temperature.go b/internal/ai/patrol_disk_temperature.go index 53e58fabe3..1af32bb0f4 100644 --- a/internal/ai/patrol_disk_temperature.go +++ b/internal/ai/patrol_disk_temperature.go @@ -1,9 +1,14 @@ package ai -import "github.com/rcourtman/pulse-go-rewrite/internal/alerts" +import ( + "strings" -// diskTemperatureLimits is the alert disk temperature policy for one disk -// type, the same policy disk temperature alerts and the Physical Disks Health + "github.com/rcourtman/pulse-go-rewrite/internal/alerts" + "github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources" +) + +// diskTemperatureLimits is the alert disk temperature policy for one disk, +// the same policy disk temperature alerts and the Physical Disks Health // verdict judge heat by. A disk is hot at the trigger and stays hot until it // cools to the clear value. A zero trigger means disk temperature alerting is // off, so no reading counts as heat. @@ -12,13 +17,14 @@ type diskTemperatureLimits struct { clear float64 } -// diskTemperatureLimitsFor resolves the policy from the user's alert -// configuration, or from the factory alert configuration when Patrol has no -// threshold provider. -func diskTemperatureLimitsFor(provider ThresholdProvider, diskType string) diskTemperatureLimits { +// diskTemperatureLimitsFor resolves the policy for a disk of the given type +// that the given host agent reports from the user's alert configuration, so +// that host's Disk Temp override applies, or from the factory alert +// configuration when Patrol has no threshold provider. +func diskTemperatureLimitsFor(provider ThresholdProvider, host alerts.DiskTemperatureHost, diskType string) diskTemperatureLimits { var trigger, clear float64 if provider != nil { - trigger, clear = provider.GetDiskTemperatureThreshold(diskType) + trigger, clear = provider.GetDiskTemperatureThreshold(host, diskType) } else if threshold := alerts.DefaultDiskTemperatureThreshold(diskType); threshold != nil { trigger, clear = threshold.Trigger, threshold.Clear } @@ -48,3 +54,44 @@ func (l diskTemperatureLimits) cooled(temperature int) bool { } return float64(temperature) <= l.clear } + +// physicalDiskOwnerIndex indexes, by resource ID, the resources a physical disk +// can reach its reporting host agent through: agent-bearing machines and the +// storage pools disks hang off. +func physicalDiskOwnerIndex(urp UnifiedResourceProvider) map[string]unifiedresources.Resource { + index := make(map[string]unifiedresources.Resource) + for _, resource := range urp.GetAll() { + if resource.Agent != nil || unifiedresources.CanonicalResourceType(resource.Type) == unifiedresources.ResourceTypeStorage { + index[resource.ID] = resource + } + } + return index +} + +// physicalDiskTemperatureHost returns the host agent that reports a physical +// disk, the host its disk temperature alerts are evaluated under: the agent on +// the machine the disk is parented to or, for a disk parented to a storage +// pool (Unraid array and cache disks), the agent on the pool's machine. A disk +// no agent reports gets the hostless policy. +func physicalDiskTemperatureHost(disk unifiedresources.Resource, owners map[string]unifiedresources.Resource) alerts.DiskTemperatureHost { + parentID := disk.ParentID + for hops := 0; parentID != nil && hops < 2; hops++ { + parent, ok := owners[strings.TrimSpace(*parentID)] + if !ok { + break + } + if parent.Agent != nil && strings.TrimSpace(parent.Agent.AgentID) != "" { + return alerts.DiskTemperatureHost{ + ID: strings.TrimSpace(parent.Agent.AgentID), + LinkedNodeID: parent.Agent.LinkedNodeID, + LinkedVMID: parent.Agent.LinkedVMID, + LinkedContainerID: parent.Agent.LinkedContainerID, + } + } + if unifiedresources.CanonicalResourceType(parent.Type) != unifiedresources.ResourceTypeStorage { + break + } + parentID = parent.ParentID + } + return alerts.DiskTemperatureHost{} +} diff --git a/internal/ai/patrol_disk_temperature_test.go b/internal/ai/patrol_disk_temperature_test.go index d8c433eae4..dc3260378f 100644 --- a/internal/ai/patrol_disk_temperature_test.go +++ b/internal/ai/patrol_disk_temperature_test.go @@ -6,6 +6,7 @@ import ( "testing" "time" + "github.com/rcourtman/pulse-go-rewrite/internal/alerts" "github.com/rcourtman/pulse-go-rewrite/internal/models" "github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources" "github.com/rcourtman/pulse-go-rewrite/pkg/aicontracts" @@ -33,7 +34,7 @@ func TestDiskTemperatureLimitsFollowTheAlertPolicy(t *testing.T) { {name: "alerting off", provider: &mockThresholdProvider{diskTemperature: map[string][2]float64{"sata": {0, 0}}}, diskType: "sata", want: diskTemperatureLimits{}}, {name: "clear above trigger", provider: &mockThresholdProvider{diskTemperature: map[string][2]float64{"sas": {60, 64}}}, diskType: "sas", want: diskTemperatureLimits{trigger: 60, clear: 60}}, } { - if got := diskTemperatureLimitsFor(tc.provider, tc.diskType); got != tc.want { + if got := diskTemperatureLimitsFor(tc.provider, alerts.DiskTemperatureHost{}, tc.diskType); got != tc.want { t.Errorf("%s: limits = %+v, want %+v", tc.name, got, tc.want) } } @@ -171,7 +172,7 @@ func TestPatrolRunStateCarriesTheConfiguredDiskTemperaturePolicy(t *testing.T) { {ID: "nvme-72", DevPath: "/dev/nvme0n1", Type: "nvme", Health: "PASSED", Wearout: 90, Temperature: 72}, }} ps := NewPatrolService(nil, &mockStateProvider{state: snapshot}) - if got := ps.currentPatrolRuntimeState().diskTemperatureLimits("nvme"); got.trigger != 70 { + if got := ps.currentPatrolRuntimeState().diskTemperatureLimits(alerts.DiskTemperatureHost{}, "nvme"); got.trigger != 70 { t.Fatalf("no provider: nvme trigger = %v, want the factory 70", got.trigger) } @@ -180,7 +181,7 @@ func TestPatrolRunStateCarriesTheConfiguredDiskTemperaturePolicy(t *testing.T) { "current": ps.currentPatrolRuntimeState(), "snapshot": ps.patrolRuntimeStateForSnapshot(snapshot), } { - if got := state.diskTemperatureLimits("nvme"); got.trigger != 75 || got.clear != 70 { + if got := state.diskTemperatureLimits(alerts.DiskTemperatureHost{}, "nvme"); got.trigger != 75 || got.clear != 70 { t.Errorf("%s state: nvme limits = %+v, want the configured 75/70", name, got) } if flags := triageDiskHealthChecksState(state, nil); len(flags) != 0 { @@ -194,7 +195,7 @@ func TestPatrolRunStateCarriesTheConfiguredDiskTemperaturePolicy(t *testing.T) { func TestScopedPatrolStateKeepsTheConfiguredDiskTemperaturePolicy(t *testing.T) { state := diskTemperatureTriageState(raisedNVMeThresholds()) scoped := filterPatrolStateByScopeState(state, PatrolScope{ResourceIDs: []string{"nvme-72"}}) - if got := scoped.diskTemperatureLimits("nvme"); got.trigger != 75 || got.clear != 70 { + if got := scoped.diskTemperatureLimits(alerts.DiskTemperatureHost{}, "nvme"); got.trigger != 75 || got.clear != 70 { t.Fatalf("scoped state nvme limits = %+v, want the configured 75/70", got) } if flags := triageDiskHealthChecksState(scoped, nil); len(flags) != 0 { @@ -212,19 +213,19 @@ func TestPatrolStartsWithAThresholdProviderSetBeforeIt(t *testing.T) { if patrol == nil { t.Fatal("expected Patrol to start once a state provider exists") } - if got := patrol.currentPatrolRuntimeState().diskTemperatureLimits("nvme"); got.trigger != 75 { + if got := patrol.currentPatrolRuntimeState().diskTemperatureLimits(alerts.DiskTemperatureHost{}, "nvme"); got.trigger != 75 { t.Fatalf("Patrol nvme trigger = %v, want the provider's 75", got.trigger) } } // A heat finding recovers at the clear value, where its alert recovers. func TestDiskHeatRecoversAtTheClearValue(t *testing.T) { - limits := diskTemperatureLimitsFor(nil, "nvme") + limits := diskTemperatureLimitsFor(nil, alerts.DiskTemperatureHost{}, "nvme") if !limits.cooled(65) || limits.cooled(66) { t.Fatalf("nvme limits %+v: want 65C cooled and 66C still warm", limits) } // With no band below the trigger, a reading at the trigger is still hot. - noBand := diskTemperatureLimitsFor(&mockThresholdProvider{diskTemperature: map[string][2]float64{"sas": {70, 70}}}, "sas") + noBand := diskTemperatureLimitsFor(&mockThresholdProvider{diskTemperature: map[string][2]float64{"sas": {70, 70}}}, alerts.DiskTemperatureHost{}, "sas") if !noBand.hot(70) || noBand.cooled(70) || !noBand.cooled(69) { t.Fatalf("no-band limits %+v: want 70C hot and not cooled, 69C cooled", noBand) } @@ -247,3 +248,98 @@ func TestDiskHeatRecoveryNeedsACurrentReading(t *testing.T) { t.Fatalf("retained 40C: recovered = %v, err = %v, want recovered", recovered, err) } } + +// overriddenAgentDisksRegistry is two host agents, each reporting one NVMe +// disk hotter than the NVMe trigger, with a Disk Temp override of 80C (clear +// 75) set on agent-cool only. Alerts judge agent-cool's disk by the override. +func overriddenAgentDisksRegistry(t *testing.T) (*unifiedresources.ResourceRegistry, *AlertThresholdAdapter) { + t.Helper() + now := time.Now() + agent := func(id string, temperature int) models.Host { + return models.Host{ + ID: id, Hostname: id, DisplayName: id, Status: "online", LastSeen: now, IntervalSeconds: 30, + Sensors: models.HostSensorSummary{SMART: []models.HostDiskSMART{{ + Device: "nvme0n1", Model: "Samsung 990", Serial: id + "-serial", Type: "nvme", + Health: "PASSED", Temperature: temperature, + }}}, + } + } + registry := unifiedresources.NewRegistry(nil) + registry.IngestSnapshot(models.StateSnapshot{Hosts: []models.Host{agent("agent-cool", 76), agent("agent-plain", 72)}}) + + mgr := alerts.NewManager() + cfg := mgr.GetConfig() + cfg.Overrides = map[string]alerts.ThresholdConfig{ + "agent-cool": {DiskTemperature: &alerts.HysteresisThreshold{Trigger: 80, Clear: 75}}, + } + mgr.UpdateConfig(cfg) + return registry, NewAlertThresholdAdapter(mgr) +} + +// A host agent's Disk Temp override replaces the per-type trigger for every +// disk that agent reports, in Patrol exactly as in its alerts: agent-cool's +// NVMe at 76C is under its 80C override, agent-plain's at 72C is over the +// NVMe trigger of 70C. +func TestPatrolJudgesAgentDisksByTheAgentDiskTemperatureOverride(t *testing.T) { + registry, provider := overriddenAgentDisksRegistry(t) + state := newPatrolRuntimeStateWithProviders(models.StateSnapshot{}, nil, unifiedresources.NewUnifiedAIAdapter(registry)) + state.thresholdProvider = provider + + rows := patrolPhysicalDiskRows(state, nil) + if len(rows) != 2 { + t.Fatalf("rows = %+v, want one disk per agent", rows) + } + for _, row := range rows { + want, wantIssue := diskTemperatureLimits{trigger: 70, clear: 65}, true + if row.temperature.Collected == 76 { + want, wantIssue = diskTemperatureLimits{trigger: 80, clear: 75}, false + } + if row.temperatureLimits != want { + t.Errorf("disk at %dC: limits = %+v, want %+v", row.temperature.Collected, row.temperatureLimits, want) + } + if got := patrolPhysicalDiskHealthIssue(row); got != wantIssue { + t.Errorf("disk at %dC: health issue = %v, want %v", row.temperature.Collected, got, wantIssue) + } + } + + s := &Service{} + s.SetUnifiedResourceProvider(unifiedresources.NewUnifiedAIAdapter(registry)) + s.SetPatrolThresholdProvider(provider) + _, section, _ := strings.Cut(s.buildUnifiedResourceContext(), "Physical Disks Needing Attention") + section, _, _ = strings.Cut(section, "\n\n") + if !strings.Contains(section, "Temp: 72C") || strings.Contains(section, "Temp: 76C") { + t.Fatalf("attention section = %q, want agent-plain's 72C disk only", section) + } +} + +// A disk reaches its reporting agent through its parent: the agent's machine +// directly, or the storage pool an Unraid array or cache disk hangs off. A +// disk on a machine with no agent gets the hostless policy. +func TestPhysicalDiskTemperatureHostFollowsTheParentChain(t *testing.T) { + ref := func(id string) *string { return &id } + owners := map[string]unifiedresources.Resource{ + "agent-1": {ID: "agent-1", Type: unifiedresources.ResourceTypeAgent, Agent: &unifiedresources.AgentData{ + AgentID: "host-1", LinkedNodeID: "lab-pve1", + }}, + "vm-1": {ID: "vm-1", Type: unifiedresources.ResourceTypeVM, Agent: &unifiedresources.AgentData{ + AgentID: "host-vm", LinkedVMID: "lab-pve1-101", + }}, + "array-1": {ID: "array-1", Type: unifiedresources.ResourceTypeStorage, ParentID: ref("agent-1")}, + "node-1": {ID: "node-1", Type: unifiedresources.ResourceTypeAgent}, + } + for parent, want := range map[string]alerts.DiskTemperatureHost{ + "agent-1": {ID: "host-1", LinkedNodeID: "lab-pve1"}, + "vm-1": {ID: "host-vm", LinkedVMID: "lab-pve1-101"}, + "array-1": {ID: "host-1", LinkedNodeID: "lab-pve1"}, + "node-1": {}, + "missing": {}, + } { + disk := unifiedresources.Resource{ID: "disk-under-" + parent, Type: unifiedresources.ResourceTypePhysicalDisk, ParentID: ref(parent)} + if got := physicalDiskTemperatureHost(disk, owners); got != want { + t.Errorf("disk under %s: host = %+v, want %+v", parent, got, want) + } + } + if got := physicalDiskTemperatureHost(unifiedresources.Resource{ID: "orphan"}, owners); got != (alerts.DiskTemperatureHost{}) { + t.Errorf("parentless disk: host = %+v, want hostless", got) + } +} diff --git a/internal/ai/patrol_state.go b/internal/ai/patrol_state.go index b1cf74e2e0..4b8b3d9b52 100644 --- a/internal/ai/patrol_state.go +++ b/internal/ai/patrol_state.go @@ -100,9 +100,9 @@ type patrolRuntimeState struct { } // diskTemperatureLimits resolves the alert disk temperature policy for one -// disk type. -func (s patrolRuntimeState) diskTemperatureLimits(diskType string) diskTemperatureLimits { - return diskTemperatureLimitsFor(s.thresholdProvider, diskType) +// disk of the given type that the given host agent reports. +func (s patrolRuntimeState) diskTemperatureLimits(host alerts.DiskTemperatureHost, diskType string) diskTemperatureLimits { + return diskTemperatureLimitsFor(s.thresholdProvider, host, diskType) } func newPatrolRuntimeState(snapshot models.StateSnapshot) patrolRuntimeState { diff --git a/internal/ai/patrol_thresholds_test.go b/internal/ai/patrol_thresholds_test.go index 53ad2439db..883795f24d 100644 --- a/internal/ai/patrol_thresholds_test.go +++ b/internal/ai/patrol_thresholds_test.go @@ -3,6 +3,7 @@ package ai import ( "testing" + "github.com/rcourtman/pulse-go-rewrite/internal/alerts" "github.com/stretchr/testify/assert" ) @@ -20,7 +21,7 @@ func (m MockThresholdProvider) GetNodeMemoryThreshold() float64 { return m.Node func (m MockThresholdProvider) GetGuestMemoryThreshold() float64 { return m.GuestMem } func (m MockThresholdProvider) GetGuestDiskThreshold() float64 { return m.GuestDisk } func (m MockThresholdProvider) GetStorageThreshold() float64 { return m.Storage } -func (m MockThresholdProvider) GetDiskTemperatureThreshold(string) (float64, float64) { +func (m MockThresholdProvider) GetDiskTemperatureThreshold(alerts.DiskTemperatureHost, string) (float64, float64) { return 0, 0 } diff --git a/internal/ai/resource_context.go b/internal/ai/resource_context.go index 31218ace81..5d7778688e 100644 --- a/internal/ai/resource_context.go +++ b/internal/ai/resource_context.go @@ -409,6 +409,7 @@ func (s *Service) buildUnifiedResourceContextForModel(destinationModel string) s if len(physicalDisks) > 0 { attention := make([]unifiedresources.Resource, 0) + owners := physicalDiskOwnerIndex(urp) for _, disk := range physicalDisks { health := "" hot := false @@ -416,7 +417,7 @@ func (s *Service) buildUnifiedResourceContextForModel(destinationModel string) s health = strings.ToUpper(strings.TrimSpace(disk.PhysicalDisk.Health)) // A retained last-known reading is not current heat. collected := tools.SplitDiskTemperature(disk.PhysicalDisk.Temperature, disk.PhysicalDisk.Collection).Collected - hot = diskTemperatureLimitsFor(thresholdProvider, disk.PhysicalDisk.DiskType).hot(collected) + hot = diskTemperatureLimitsFor(thresholdProvider, physicalDiskTemperatureHost(disk, owners), disk.PhysicalDisk.DiskType).hot(collected) } if disk.Status != unifiedresources.StatusOnline || (health != "" && health != "PASSED" && health != "UNKNOWN") || hot { attention = append(attention, disk) diff --git a/internal/alerts/host.go b/internal/alerts/host.go index 9347925507..65ae1cbd40 100644 --- a/internal/alerts/host.go +++ b/internal/alerts/host.go @@ -1017,14 +1017,41 @@ func diskTemperatureThresholdForType(byType map[string]HysteresisThreshold, host // temperature alerts, Patrol and the Physical Disks Health verdict all judge // heat against it; disk risk does not. The disk is hot at the trigger, and the // clear value is where it stops being hot. A nil threshold or a non-positive -// trigger means disk temperature alerting is off. +// trigger means disk temperature alerting is off. A disk a host agent reports +// is judged by HostDiskTemperatureThreshold instead. func (m *Manager) DiskTemperatureThreshold(diskType string) *HysteresisThreshold { + return m.HostDiskTemperatureThreshold(DiskTemperatureHost{}, diskType) +} + +// DiskTemperatureHost is the host agent whose report carries a physical disk, +// with the node and guests that agent is linked to. CheckHost evaluates the +// disk's temperature under that host's thresholds. +type DiskTemperatureHost struct { + ID string + LinkedNodeID string + LinkedVMID string + LinkedContainerID string +} + +// HostDiskTemperatureThreshold is the disk heat policy for a disk the given +// host agent reports, resolved as CheckHost resolves it. An explicit disk +// temperature override on the host, or one inherited from its linked node or +// guest, replaces the disk type's entry. An override that switches the host's +// alerts off switches its disk heat off too, so the result is nil. A host with +// no ID resolves as DiskTemperatureThreshold. +func (m *Manager) HostDiskTemperatureThreshold(host DiskTemperatureHost, diskType string) *HysteresisThreshold { if m == nil { return DefaultDiskTemperatureThreshold(diskType) } m.mu.RLock() defer m.mu.RUnlock() - return cloneThreshold(m.hostDiskTemperatureThresholdNoLock(m.config.AgentDefaults.DiskTemperature, false, diskType)) + thresholds := m.resolveHostThresholdsNoLock(host.ID, host.LinkedNodeID, host.LinkedVMID, host.LinkedContainerID) + if thresholds.Disabled { + return nil + } + override, exists := m.hostThresholdOverrideNoLock(host.ID, host.LinkedNodeID, host.LinkedVMID, host.LinkedContainerID) + overridden := exists && override.DiskTemperature != nil + return cloneThreshold(m.hostDiskTemperatureThresholdNoLock(thresholds.DiskTemperature, overridden, diskType)) } // DefaultDiskTemperatureThreshold is DiskTemperatureThreshold under the diff --git a/internal/alerts/threshold_resolution_shared_test.go b/internal/alerts/threshold_resolution_shared_test.go index 673a1bab50..cf040367e4 100644 --- a/internal/alerts/threshold_resolution_shared_test.go +++ b/internal/alerts/threshold_resolution_shared_test.go @@ -1445,3 +1445,148 @@ func TestDiskTemperatureThresholdMatchesCheckHostPolicy(t *testing.T) { t.Fatalf("nil manager threshold = %+v, want the factory nvme 70", got) } } + +// HostDiskTemperatureThreshold is the disk heat policy for a disk a host agent +// reports, so it must agree with CheckHost on every host-level override: an +// explicit Disk Temp override on the host (or the canonical resource its ID +// resolves to), or inherited from its linked node or guest, replaces the +// per-type entry; the first override in the chain decides even when it sets no +// Disk Temp; and a host whose alerts are off judges no heat. +func TestHostDiskTemperatureThresholdMatchesCheckHostOverrides(t *testing.T) { + const hostID = "host-temp-policy" + const nodeID = "pve1-node1" + const vmID = "pve1-node1-105" + const canonicalID = "agent-canonical-temp-policy" + perType := &HysteresisThreshold{Trigger: 70, Clear: 65} + + for _, tc := range []struct { + name string + overrides map[string]ThresholdConfig + linkNode bool + linkVM bool + canonical bool + want *HysteresisThreshold + off bool + }{ + {name: "no override", want: perType}, + { + name: "host disk temp override", + overrides: map[string]ThresholdConfig{hostID: {DiskTemperature: &HysteresisThreshold{Trigger: 80, Clear: 75}}}, + want: &HysteresisThreshold{Trigger: 80, Clear: 75}, + }, + { + name: "host override without disk temp", + overrides: map[string]ThresholdConfig{hostID: {Memory: &HysteresisThreshold{Trigger: 90, Clear: 85}}}, + want: perType, + }, + { + name: "linked node disk temp override", + overrides: map[string]ThresholdConfig{nodeID: {DiskTemperature: &HysteresisThreshold{Trigger: 78, Clear: 72}}}, + linkNode: true, + want: &HysteresisThreshold{Trigger: 78, Clear: 72}, + }, + { + name: "host override shadows linked node", + overrides: map[string]ThresholdConfig{ + hostID: {Memory: &HysteresisThreshold{Trigger: 90, Clear: 85}}, + nodeID: {DiskTemperature: &HysteresisThreshold{Trigger: 78, Clear: 72}}, + }, + linkNode: true, + want: perType, + }, + { + // The agent ID resolves to the machine's canonical resource ID. + name: "canonical machine disk temp override", + overrides: map[string]ThresholdConfig{canonicalID: {DiskTemperature: &HysteresisThreshold{Trigger: 82, Clear: 77}}}, + canonical: true, + want: &HysteresisThreshold{Trigger: 82, Clear: 77}, + }, + { + name: "linked guest disk temp override", + overrides: map[string]ThresholdConfig{vmID: {DiskTemperature: &HysteresisThreshold{Trigger: 76, Clear: 71}}}, + linkVM: true, + want: &HysteresisThreshold{Trigger: 76, Clear: 71}, + }, + { + name: "linked guest alerts disabled", + overrides: map[string]ThresholdConfig{vmID: {Disabled: true}}, + linkVM: true, + }, + { + name: "host disk temp switched off", + overrides: map[string]ThresholdConfig{hostID: {DiskTemperature: &HysteresisThreshold{}}}, + off: true, + }, + { + name: "host alerts disabled", + overrides: map[string]ThresholdConfig{hostID: {Disabled: true}}, + }, + { + name: "linked node alerts disabled", + overrides: map[string]ThresholdConfig{nodeID: {Disabled: true}}, + linkNode: true, + }, + } { + t.Run(tc.name, func(t *testing.T) { + m := configureDiskTempTypeHostManager(t) + m.mu.Lock() + for id, override := range tc.overrides { + m.config.Overrides[id] = override + } + m.mu.Unlock() + if tc.canonical { + m.SetResourceIntentIdentityResolver(func(resourceID string) (string, bool) { + return canonicalID, resourceID == hostID + }) + } + + owner := DiskTemperatureHost{ID: hostID} + if tc.linkNode { + owner.LinkedNodeID = nodeID + } + if tc.linkVM { + owner.LinkedVMID = vmID + } + got := m.HostDiskTemperatureThreshold(owner, "nvme") + switch { + case tc.off: + if got == nil || got.Trigger > 0 { + t.Fatalf("HostDiskTemperatureThreshold = %+v, want a switched-off threshold", got) + } + case (got == nil) != (tc.want == nil) || (got != nil && *got != *tc.want): + t.Fatalf("HostDiskTemperatureThreshold = %+v, want %+v", got, tc.want) + } + + check := func(temperature int) bool { + host := hostWithSMARTDiskTemp(hostID, "nvme", temperature) + host.LinkedNodeID = owner.LinkedNodeID + host.LinkedVMID = owner.LinkedVMID + m.CheckHost(host) + _, exists := testLookupActiveAlert(t, m, hostDiskTempAlertID(host)) + return exists + } + if got == nil || got.Trigger <= 0 { + if check(99) { + t.Fatalf("CheckHost alerted at 99C on a host the policy judges no heat for, active: %v", alertKeys(m)) + } + return + } + trigger := int(got.Trigger) + if check(trigger - 1) { + t.Fatalf("CheckHost alerted at %dC, under the policy trigger %v", trigger-1, got.Trigger) + } + if !check(trigger) { + t.Fatalf("CheckHost stayed quiet at the policy trigger %dC, active: %v", trigger, alertKeys(m)) + } + }) + } + + m := configureDiskTempTypeHostManager(t) + if got, want := m.HostDiskTemperatureThreshold(DiskTemperatureHost{}, "sas"), m.DiskTemperatureThreshold("sas"); got == nil || want == nil || *got != *want { + t.Fatalf("hostless policy = %+v, want DiskTemperatureThreshold %+v", got, want) + } + var nilManager *Manager + if got := nilManager.HostDiskTemperatureThreshold(DiskTemperatureHost{ID: hostID}, "nvme"); got == nil || got.Trigger != 70 { + t.Fatalf("nil manager host policy = %+v, want the factory nvme 70", got) + } +} diff --git a/internal/api/ai_handlers_concurrency_test.go b/internal/api/ai_handlers_concurrency_test.go index ca04c61635..0c5276fa05 100644 --- a/internal/api/ai_handlers_concurrency_test.go +++ b/internal/api/ai_handlers_concurrency_test.go @@ -7,6 +7,7 @@ import ( "time" "github.com/rcourtman/pulse-go-rewrite/internal/ai" + "github.com/rcourtman/pulse-go-rewrite/internal/alerts" "github.com/rcourtman/pulse-go-rewrite/internal/config" "github.com/rcourtman/pulse-go-rewrite/internal/models" "github.com/rcourtman/pulse-go-rewrite/internal/monitoring" @@ -67,7 +68,7 @@ func (concurrencyThresholdProvider) GetGuestMemoryThreshold() float64 { } func (concurrencyThresholdProvider) GetGuestDiskThreshold() float64 { return 95 } func (concurrencyThresholdProvider) GetStorageThreshold() float64 { return 92 } -func (concurrencyThresholdProvider) GetDiskTemperatureThreshold(string) (float64, float64) { +func (concurrencyThresholdProvider) GetDiskTemperatureThreshold(alerts.DiskTemperatureHost, string) (float64, float64) { return 70, 65 } diff --git a/internal/api/ai_handlers_more_test.go b/internal/api/ai_handlers_more_test.go index 128483e0c1..52339f4893 100644 --- a/internal/api/ai_handlers_more_test.go +++ b/internal/api/ai_handlers_more_test.go @@ -16,6 +16,7 @@ import ( "github.com/rcourtman/pulse-go-rewrite/internal/ai" "github.com/rcourtman/pulse-go-rewrite/internal/ai/cost" "github.com/rcourtman/pulse-go-rewrite/internal/ai/memory" + "github.com/rcourtman/pulse-go-rewrite/internal/alerts" "github.com/rcourtman/pulse-go-rewrite/internal/config" "github.com/rcourtman/pulse-go-rewrite/internal/models" "github.com/rcourtman/pulse-go-rewrite/internal/monitoring" @@ -154,7 +155,7 @@ func (stubThresholdProvider) GetGuestMemoryThreshold() float64 { } func (stubThresholdProvider) GetGuestDiskThreshold() float64 { return 95 } func (stubThresholdProvider) GetStorageThreshold() float64 { return 92 } -func (stubThresholdProvider) GetDiskTemperatureThreshold(string) (float64, float64) { +func (stubThresholdProvider) GetDiskTemperatureThreshold(alerts.DiskTemperatureHost, string) (float64, float64) { return 70, 65 }