5161 Commits

Author	SHA1	Message	Date
rcourtman	080ace2e97	Show the outdated sensor setup notice on every Proxmox tab (#2668) ... Since the per-tab Proxmox queries landed, only the Storage tab loads `physical_disk` rows. So the outdated sensor setup notice, which judged nodes from the active tab's disk rows, never showed on Overview. The registry now derives the verdict on each Proxmox node (`proxmox.sensorSetupOutdated`, an explicit true or false on every node) right after child counts resolve each disk's canonical parent. A node is flagged when its temperature payload is available and in the legacy `sensors -j` format and a SATA/SAS disk under it has no reading collected now (`diskinventory.TemperatureCollected`). The page lists flagged nodes from the rows Overview, Storage, Replication and Backups already load, so no tab hydrates disks for it. Rule fixes that came with the move: - Proxmox's disk inventory types SATA/SAS disks as `hdd`/`ssd`, so the old `sata`/`sas`-only check never matched a node without a host agent. - A retained reading no longer counts as current (carried over from #2659's frontend rule). - The verdict is sent as explicit false, because the browser's field-by-field facet merge would otherwise keep an earlier true after the condition clears. Stacked on #2659 (shares its `sensorSetup.ts` edit, which this replaces). Verified on a private mock stack at 1440px and 390px: - Overview, Storage, Replication and Backups show the notice for legacy-format nodes with an `hdd` disk without a reading, a retained SATA reading, and a SATA disk without a reading. - Overview makes no `physical_disk` request. - With `main`'s frontend swapped in, Overview showed nothing and Storage named one node of three. - A node whose disk starts reporting drops out on the next websocket update without a reload.	2026-10-07 15:38:27 +01:00
rcourtman	cfa026249f	Switch every running tenant monitor when mock mode flips (#2665) ... Follow-up to #2656 (the per-monitor mock-mode epoch fence), which documented this gap as not covered. ## Problem Mock mode is one process-wide flag (`mock.SetEnabled`), but `Monitor.SetMockMode` ended the fence epoch, cleared the alerts, forgot the fixture agents and reset the state of only the monitor it was called on. The server runs one `Monitor` per organization (`MultiTenantMonitor.GetMonitor` → `New` + `Start`), and each runs its own mock alert pass into its own alert manager. A switch made through one monitor (the mock-mode API uses the requesting tenant's monitor; the demo-fixture licence sync uses the default one) left every other running monitor holding what it raised in the mode the process had left: fixture alerts and fixture-agent node links in live mode, live alerts in mock mode, until the 24-hour stale sweep. ## Fix - `internal/monitoring/mock_mode_switch.go`: a process-wide set of running monitors, guarded by the existing `mockModeSwitchMu`. `Start` joins it and leaves when its loop returns. While joining it picks the mock metrics sampler or discovery under the lock (now `startModeRuntime`), so no switch lands between reading the mode and starting the runtime for it. - `SetMockMode` flips the flag once, then for the receiver and every running monitor: ends the epoch, waits for admitted calls, clears alerts, forgets fixture agents (when leaving mock mode) and resets state (`endMockModeEpoch`). Only after that does it restart the sampler, polls and discovery of each monitor in the set (`resumeAfterMockModeSwitch`). A monitor that has not joined yet gets its runtime from its own `Start`. - Each monitor records the mode its alerts and state belong to (`mockModeAligned`, set in `New` from the same read that decides whether persisted alerts are restored) and the number of switches made so far (`mockModeSwitchesSeen`). If it missed switches between `New` and `Start`, it leaves what it holds before joining, even after a round trip that brought the flag back. A call for the mode the flag already holds still moves any reached monitor that belongs to the other mode, and leaves the rest alone. Callers are unchanged and no API surface moves. ## Is this reachable in release builds? Barely. In a `release` build `mockruntime.ValidateEnablement` refuses mock fixtures unless `SetReleaseFixturesAuthorized(true)` was called. Its only caller is `syncReleaseDemoFixtureRuntime`, which requires `DemoMode` plus the `demo_fixtures` entitlement and runs only for the default organization. Several tenant monitors in mock mode would also need more than one organization's monitor running on that demo instance. Demo mode's read-only middleware normally blocks creating organizations and the mock-mode API's POST. It does forward any request carrying `Upgrade: websocket` whatever its method; that bypass predates this change and is being fixed separately. So in practice this is a dev/mock-stack problem, and the fix stays inside the monitoring package. ## Tests New tests in `internal/monitoring/monitor_mock_alerts_test.go`: - `TestLeavingMockModeClearsEveryRunningTenantMonitor` and `TestEnteringMockModeClearsEveryRunningTenantMonitor` start the default and one other tenant monitor through `MultiTenantMonitor`, as the server does, and wait until both have joined. They raise alerts in both, switch through one, and check both. - `TestMonitorStartedAfterAMockModeSwitchLeavesTheModeItWasBuiltIn` covers a switch that lands between `New` and `Start`. - `TestMonitorStartedAfterAMockModeRoundTripDropsWhatItRaisedMeanwhile` covers two switches in that window, with fixture alerts raised in between. - `TestSetMockModeAlignsAMonitorWhoseModeTheFlagAlreadyHolds` covers a call for the mode the flag already holds. - `TestMockModeSwitchLeavesAnUnjoinedMonitorsRuntimeToItsStart` covers a switch reaching a monitor that has not joined. One more is in `internal/monitoring/monitor_host_agent_removal_lifecycle_test.go`: `TestLeavingMockModeReleasesFixtureAgentNodeLinksOnEveryRunningMonitor`. It switches through the default monitor and checks that the other tenant's fixture node is no longer suppressed by a fixture agent's leftover node link. Negative controls (`go test -overlay`): | Control | Failing tests | What happened | |---|---|---| | Fence-PR runtime | Leaving, entering, node-link | Default monitor kept 69–104 fixture alerts (varies by run) and 69 fixture agents; org-b kept its live alert; org-b's `mock-cluster-1-pve1` stayed suppressed | | Set kept, switch reaches only the receiver | Same three | Same failures | | No realign at join | `TestMonitorStartedAfterAMockModeSwitchLeavesTheModeItWasBuiltIn` | | | Join compares only the mode, not the switch count | `TestMonitorStartedAfterAMockModeRoundTripDropsWhatItRaisedMeanwhile` | Started in live mode with the fixture node's memory alert | | Plain early return when the flag already matches | `TestSetMockModeAlignsAMonitorWhoseModeTheFlagAlreadyHolds` | | | Resume ignores set membership | `TestMockModeSwitchLeavesAnUnjoinedMonitorsRuntimeToItsStart` | | Every control failed only the tests it targets. Also run: the full `internal/monitoring` package; `-race -count=3` on the new tests plus the existing `TestLeavingMockMode*` and `TestMockModeFence*`; and the mock-mode tests in `internal/api` and `internal/api/configapi`.	2026-10-07 15:27:59 +01:00
rcourtman	900d620abd	Rank a confirmed availability outage first in the summary check (#2664) ... A machine with several availability checks shows one summary check, and its health reads only that summary. The summary ranked every observed failure equally and broke ties by target id. So a check that had failed once, below its threshold, could hide a later-sorting check with a confirmed outage. The outage check's own row carried `availability_unreachable` while the machine it monitors read healthy. - `availabilityCheckPriority` now ranks a confirmed outage first, then any other observed failure, then unchecked/ambiguous/unresolved, then passing. Ties still go to the first target id. - `availabilityOutageConfirmed` is the one confirmation rule, shared by the ranking and `EvaluateResourceHealth`. It mirrors the poller's incident gate: observed failures at the threshold, an unavailable aggregate when one is reported, and no indeterminate outcome. A probe agent that stops reporting keeps its old failure count but confirms nothing, on its own row or on the target. - Tests in `availability_link_test.go` go through the registry projection path: a confirmed outage behind an earlier unconfirmed failure (and its recovery), and a silent probe agent's stale failure count. Both fail without the change. `health_test.go`'s legacy fixture gains the `LastChecked` every producer sets. - The `unified-resources.md` contract describes the summary ranking and the shared rule. Drawers already list every check from `availabilityChecks`, and availability incidents live on each check's own row, so alert synthesis is unchanged.	2026-10-07 15:23:46 +01:00
rcourtman	0f6d1e4180	Stop a failing availability check keeping a node online (#2663) ... Follows #2643, which made the stale pass read each source's own verdict but left availability checks counting by delivery. ## Problem Once every source with a verdict had gone quiet, any current availability sighting kept its target online. A node that `preserveOrExpireNodes` expired stayed online while its HTTP check failed, and a quiet failing check lifted that node's offline verdict to warning. One sighting stands for every check on a target and carries whichever check was projected last, so a failing check that had just run also vouched for another check's old pass. ## Change - **Registry verdict** (`aggregateStatus`, `availabilityChecksProveOnline`): each check on a monitored resource is judged by its own evidence window (`Evidence.ValidUntil`). An enabled, passing check with current evidence proves the resource answers and reads online once one of the resource's own sources has gone quiet and none that is current has a verdict. Every other check abstains, current or quiet. A failing probe never makes a guest count as stopped (workloads summary, Proxmox page counts). A check's own row keeps the ordinary rules. - **Ownership**: only a target's own quiet sources hand the stale pass its status. Projecting, re-running or retargeting a check re-applies the stale pass's rule to targets it owns, because the resources API replays checks after its stale pass. A manual link judges the merged resource by the checks it carries. - **Poller evidence** (`internal/monitoring`): - A local location beside other locations ages out like a silent remote path ("no recent local check"). - An available check's evidence lasts only as long as its newest reachable path. - After an entitlement lapse, an adopted remote result keeps the earlier of its agent check time and the server receipt time. - A multi-location aggregate is never adopted as a local observation; the target reads as never checked until the local poller runs. Contracts: `unified-resources.md` ("Status follows each source's own verdict") replaces the residual paragraph, and `monitoring.md` covers the multi-location rules. ## Verification - New tests in `registry_test.go`, `registry_merge_policy_test.go` and `availability_probe_agent_test.go`. Each fails against the base runtime via `go test -overlay`, including both projection orders. - Full `internal/unifiedresources` and `internal/monitoring` suites pass, plus the availability, connection, workload and resource-state tests in `internal/api`, `resourceapi` and `chartapi`.	2026-10-07 15:23:03 +01:00
rcourtman	156d2e274d	Keep same-serial disks on different machines apart (#2660) ... ## Problem `ingestRecord` refuses a physical-disk match under another parent (`findMatch` skips scope-incompatible candidates), then `chooseNewID` mints the same hardware-keyed canonical ID (`machine:<serial>`) and the ingest merges into whatever disk already holds it. Disks that report one usable serial on several machines collapse into a single resource parented to whichever machine ingested last, and every other machine loses the disk from inventory. Real triggers: dual-ported SAS shelves seen by two nodes, cloned VMs with an explicit identical serial, fixed-serial USB bridges, TrueNAS systems. Repro: `IngestSnapshot` with three PVE nodes in one instance, each with a `/dev/sda` of the same serial, gave one disk. ## Fix - A new disk resolves its parent to a machine: the nearest host ancestor, so a TrueNAS pool or Unraid array counts as its system or host. - Two disks with one identity stay apart only on evidence: one reporter placing the serial on two machines (cloned VMs keep both serial and hostname), or machines whose hostnames share nothing. Reporters on one box still join across grouping parents and unmerged host resources, so the TrueNAS API and the agent on that box stay one disk. An unknown machine never splits a disk, so a PVE row whose node row is missing still joins its host's agent disk. - Once a serial spans machines, every copy takes the serial keyed to its own machine, and the holder of the unscoped ID is re-keyed with its source mappings, retired-ID claims and relationship endpoints. IDs don't depend on machine ingest order, and the unscoped ID never passes between disks that coexist. A disk only one machine reports keeps its current ID. - Operator splits ("Split merged resource" exclusions) now hold on this path; the old collision merge ignored them. A split recorded against any ID the disk can hold applies, so it survives a re-key. - The registry indexes disks by hardware key (`physicalDisksByHardware`) because `IdentityMatcher` keeps one resource per machine ID. ## ID stability decision Machine-scoped IDs only for identities that actually span machines, rather than a pre-pass. A pre-pass needs final parent scopes before ingest, but host/node merging happens during ingest. Giving the first-ingested machine the unscoped ID would hand one machine's ID, and its operator state, to another machine when an earlier-sorting machine appears. With the chosen design a disk's ID changes when a same-serial disk first appears on another machine. That orphans ID-keyed state rather than misapplying it. ## Residuals (documented in unified-resources.md) - Like any serial-keyed ID, the unscoped ID still passes to a same-serial disk that appears once the previous holder is gone. - Alert-history rows owned under the unscoped ID don't follow a re-key. The PVE disk alert owner (#2646) already leaves shared-serial rows unbound unless a WWN tells them apart. - Agent SMART and Unraid disk source IDs are the bare serial. Same-serial agent disks on different hosts share one `SourceAgent` mapping, and only the last ingested keeps the agent source target. This is tracked as a separate task, because the fix re-keys persisted mappings. - Same-serial disks share one metrics history key. - Across reporters, hostnames that normalize alike still join. ## Verification - New tests in `registry_merge_policy_test.go`: per-machine disks for PVE-only and linked-agent clusters, including reversed ingest order; rehydration with a machine joining; agent + TrueNAS on one box versus a cloned TrueNAS with the same hostname; a disk moving into a pool; a parentless PVE row on a shared serial; exclusions honoured, surviving a re-key, with per-machine fallback; serialized-JSON rehydration; retired-ID claims carried across a re-key. - Every new test fails against the unfixed registry or with its fix removed (`-overlay` negative controls). - `go test ./internal/unifiedresources` passes, as do the disk/SMART/Unraid/TrueNAS tests in `internal/monitoring`, `internal/api` and `internal/truenas`. - Five rounds of independent review; the last round found no P1/P2 issues.	2026-10-07 14:50:15 +01:00
rcourtman	a85bdcdc9a	Stop a stale agent SMART row lending a replaced disk its identity (#2658) ... ## Problem PR #2648 stopped the poller's previous-evidence lookup from handing a replaced disk the old disk's serial. The linked host agent's SMART merge runs before that lookup and had the same flaw. Its last-resort match is the device path, and a path names a slot, not a disk. When a disk is swapped before the agent reports again, or after the agent went silent and kept its last rows, the agent's row for the old disk matched the replacement. It filled the old serial or, for SAS, promoted it over the Proxmox SAS address. The registry then keyed the replacement to the old disk's canonical resource and SMART history, and it also took the old disk's temperature and FAILED health. For SAS, the registry's same-path join (`resolveLinkedPhysicalDisk`) did the same at the canonical level even when monitoring kept the rows apart. Proxmox's disk inventory (`PVE::Diskmanage`, checked on a PVE 9 node) writes the literal `unknown` for a serial or WWN udev cannot read. The poller recorded it as a collected serial, so the collection state hid the gap, and the merge's `Serial == ""` fill never let the agent's real serial replace it. ## Change - `mergeHostAgentSMARTIntoDisks` refuses a device-path row whose identity contradicts the Proxmox disk's. The new `diskinventory.HardwareIdentityConflict` defines a contradiction: nothing agrees across serial and WWN, and a field both producers report like for like disagrees. - WWNs count for every disk. - Serials count only where Proxmox's serial is the drive's own, as smartctl's is: a disk typed `nvme`, or one presented with SCSI vendor `ATA` (libata, or a SAS HBA's SCSI-ATA translation) and not typed `usb`/`sas`. Elsewhere the Proxmox serial may be a SAS address or SCSI designator (#1595) or a USB bridge's own serial, so it only counts toward agreement. - Spelling differences are not disagreement: udev's underscores for whitespace, a T10 designator for an ATA drive (`ATA`, the model, then the serial), the agent's NAA field spelling, and udev's 64-bit prefix of an NAA 6 WWN. - The refusal runs after the unique-path check, so a second row on the path keeps the match ambiguous. - The merge's WWN step compares normalized identity (`HardwareIdentityMatch`), so the agent's `5-c50-a1b2c3d4` finds the Proxmox `0x5000c500a1b2c3d4` disk. - `resolveLinkedPhysicalDisk` refuses the SAS same-path fallback when the two WWNs name different disks. - The poller records Proxmox's `unknown` serial/WWN as empty, so the serial reads `missing` and the agent can fill it. Retained evidence carries a previous serial only when it is usable, so a pre-upgrade record cannot restore the placeholder. Identity keys already ignored the placeholder. When the agent now fills a disk's serial, that disk's Proxmox-poll SMART samples move from its WWN or path key to the serial key, where the agent's own samples already go. ## Tests - `TestHostAgentSMARTRowForSwappedOutDiskDoesNotLendItsIdentity` (sata and sas) drives the real poller and registry. Proxmox reports `unknown`, the agent fills the serial, the agent goes silent, and the disk is swapped. On main, the SATA replacement takes the old serial and resource; the SAS replacement keeps its own serial in state, but the registry joins it to the old resource with its temperature and FAILED health. - `TestHostAgentSMARTMergeRefusesPathMatchAcrossContradictingIdentity` has one row per producer rule: SAS/SATA/NVMe replacements refused; same-disk SAS without an agent WWN, a renamed SAS disk in the agent's WWN spelling, udev's serial spelling, a USB enclosure (typed `usb` and typed `ssd`), a T10 designator and an NAA 6 volume merged; and an ambiguous path. - `TestRegistrySASPathJoinRefusesContradictingWWN`, `TestProxmoxUnknownDiskIdentityIsRecordedAsUnreported` and `TestHardwareIdentityConflict` cover the rest. - Negative controls via `go test -overlay`: removing each rule fails only its own case. These cover the guard, the WWN normalization, the transport/vendor rule, compaction, the designator rule, the NAA 6 prefix, the registry guard, the ingest normalization and the usable-serial carry. Against main's `monitor.go` and `registry.go`, the end-to-end, merge and registry tests fail. - `TestMergeHostAgentSMARTIntoDisks_PreservesExistingProxmoxIdentity` merged a row with a different WWN by path alone, which the guard now rightly refuses. Its row is now the same disk in the agent's WWN spelling, with all assertions unchanged. - `internal/monitoring`, `internal/unifiedresources` and `pkg/diskinventory` pass in full. ## Not covered - A replacement with no comparable identity still matches its path: no usable serial or WWN at all, or a SAS/SCSI/USB disk with a WWN on one side only. - The registry's seeded source mapping, reused after a restart by the slot-shaped Proxmox source ID, can still hand a swapped disk the old canonical resource. That's a separate follow-up.	2026-10-07 14:01:19 +01:00
rcourtman	447807526f	Show the outdated sensor setup notice on every Proxmox tab ... The Proxmox page warns when a node's SSH temperature monitoring still runs the pre-rc.6 setup, which locks the key to `sensors -j` so SATA/SAS disk temperatures never arrive. The notice judged that from the disk rows of the active tab, and since each tab hydrates only its own resource families, Overview loads no disks and never showed it. The registry now derives the verdict on the node itself, after child counts resolve each disk's canonical parent: a node with an available, legacy-format temperature payload is flagged when a SMART-only disk under it has no reading collected now. The page lists flagged nodes from the rows every tab already loads, so Overview needs no disk inventory. Every Proxmox node carries an explicit true or false, because the browser merges a facet field by field and would keep an earlier true if a cleared verdict were omitted. Two rule fixes ride along with the move. Proxmox's own disk inventory types SATA/SAS disks as hdd or ssd, so the old sata/sas-only set never matched a node without a host agent. And a retained reading kept after a disk stopped being read no longer counts as present (diskinventory.TemperatureCollected).	2026-10-07 13:58:33 +01:00
rcourtman	43a80dafd9	Judge disk heat under the reporting host's Disk Temp override (#2654) ... #2636 made the alert disk temperature policy the only judge of disk heat, but it resolved that policy by disk type alone. Disk temperature alerts (`CheckHost`) evaluate each SMART disk under the thresholds of the host agent that reports it. An explicit Disk Temp override on that host, or one inherited from its linked node or guest, replaces the per-type trigger, and a host whose alerts are switched off raises nothing. So a host with an 80C Disk Temp override still read **Running Hot** on Physical Disks at the 70C NVMe trigger. Patrol flagged it too, and a host with alerts switched off still read hot. The Alerts thresholds page promises the opposite: "Setting a Disk Temp override on a host above replaces these for all of that host's disks". ## What changes - **Backend policy:** `alerts.Manager.HostDiskTemperatureThreshold(host, diskType)` resolves the policy for a host the way `CheckHost` does, from the same helpers (`resolveHostThresholdsNoLock`, `hostThresholdOverrideNoLock`, `hostDiskTemperatureThresholdNoLock`). A disabled override gives no threshold. `DiskTemperatureThreshold(diskType)` is now its hostless case. - **Patrol and the AI chat context:** `ThresholdProvider.GetDiskTemperatureThreshold(host, diskType)`. Each unified physical disk finds its host by walking its parent to the agent machine, through the pool for Unraid array and cache disks. - **Physical Disks, the disk detail, the pool drawer's linked disks and the guest drawer** pass the reporting machine's override keys to `resolveDiskTemperatureDisplayThresholds`. Keys follow `CheckHost`'s order: agent ID, canonical resource ID, then the merged node or guest. The resolver returns null (uncoloured, never hot) for an override or agent default that switches the host's alerts off. The browser rebuilds `CheckHost`'s key chain from what it loads, so a few identity edge cases can still differ from alerts and Patrol (listed in `storage-recovery.md`): on the Storage page, a disk whose agent an operator manually linked into a VM; an override inherited through agent link hints the API does not publish (an unmerged linked node or guest), or stored under a linked node's own canonical ID; and guest keys in the guest drawer for an agent merged into a guest it is not automatically linked to. Overrides set on the agent's own row, or on the node or guest it is merged with, reach every surface. ## Verification - `TestHostDiskTemperatureThresholdMatchesCheckHostOverrides` checks each case against what `CheckHost` actually fires: no override, host override, host override without Disk Temp, an override under the canonical resource the agent ID resolves to, linked-node and linked-guest overrides, a host override shadowing a node one, a switched-off Disk Temp, and a disabled host, linked node or linked guest. - `TestPatrolJudgesAgentDisksByTheAgentDiskTemperatureOverride` uses a real registry with two agents, one overridden. A mutation that drops the owner lookup fails it. - Frontend tests cover the resolver, the owner walk and key order, DiskList with three hosts, the presentation map, pool-drawer linked disks and the guest drawer. - Private mock stack: three NVMe disks pinned at 72C. Before (main frontend): all three read Running Hot. After: - pve1, with an 80C override, reads Healthy (green). - pve2 still reads Running Hot. - pve3, with alerts off, is uncoloured. - Moving pve2's override onto its linked node ID turns it Healthy. - Checked at 1440 and 375 with no overflow. The receipt is in `frontend-modern/browser-verification/`. - A read-only second-opinion review ran twice. Round 1's findings (frontend key order, guest drawer canonical ID, disabled agent default, doc wording) are fixed in the first commit. Round 2's are the parity cases and the documented browser limits in the second commit.	2026-10-07 13:49:00 +01:00
rcourtman	27e440339b	Switch every running tenant monitor when mock mode flips ... Mock mode is one process-wide flag, but SetMockMode ended the epoch, cleared the alerts, forgot the fixture agents and reset the state of only the monitor it was called on. The server runs one monitor per organization, each with its own alert manager and its own mock alert pass. A switch made through one of them (the mock-mode API uses the requesting tenant's monitor, the demo-fixture licence sync the default one) left every other running monitor holding what it raised in the mode the process had left: fixture alerts and fixture-agent node links in live mode, live alerts in mock mode, until the 24-hour stale sweep. Monitors now join a process-wide set of running monitors when Start runs and leave it when Start returns, under the same mutex that already serialized switches. SetMockMode flips the flag once, then ends the epoch of the monitor it was called on and of every running monitor: it waits for their admitted calls, clears their alerts, forgets their fixture agents and resets their state. Only then does it restart each running monitor's sampler, polls and discovery. Start picks the mock metrics sampler or discovery while it holds that mutex, so no switch can land between reading the mode and starting the runtime for it, and a switch leaves the runtime of a monitor that has not joined yet to its own Start. A monitor records the mode and the switch count it was built under. If it missed switches between New and Start, it leaves what it holds before it joins, dropping live alerts New restored or fixture alerts a read path raised meanwhile, even when a round trip brought the flag back. A call for the mode the flag already holds still moves a reached monitor whose alerts belong to the other mode, and leaves the rest alone. Release builds reach this only on an instance that runs DemoMode with the demo_fixtures entitlement, since release ValidateEnablement refuses mock fixtures otherwise, and that has more than one organization's monitor running. The licence sync only switches the default organization. So the fix stays inside the monitoring package. The callers are unchanged and no API surface moves. New tests start the default and one other tenant monitor through MultiTenantMonitor, as the server does, and switch through one of them. Against the per-monitor switch, the default monitor kept its fixture alerts (69 to 104 across runs) and all 69 fixture agents after mock mode was left through the other tenant. That tenant kept its live alert after mock mode was entered through the default monitor, and its fixture node stayed suppressed by a fixture agent's leftover node link. Four more tests fail if Start joins without leaving the mode its monitor was built in, if Start compares only the mode after a round trip, if a call for the flag's current mode returns without moving such a monitor, or if a switch starts the runtime of a monitor that has not joined.	2026-10-07 13:48:10 +01:00
rcourtman	ade799307a	Rank a confirmed availability outage first in the summary check ... A resource with several attached availability checks exposes one compatibility summary, Resource.Availability, and health reads only that. The summary ranked every observed failure equally and broke ties by target id, so a check that had failed once below its threshold could hide a later-sorting check with a confirmed outage: the outage check's own row carried availability_unreachable while the machine it monitors read healthy. Rank a confirmed outage above any other failure and have health use the same per-check rule, so target health and row presentation follow a confirmed outage on any attached check. The rule mirrors the poller's incident gate: observed failures at the threshold, an unavailable aggregate when one is reported, and no indeterminate outcome. A probe agent that stops reporting keeps its old failure count but reads indeterminate with an unknown aggregate, so it confirms nothing, on its own row or on the target. Drawers already list every check from availabilityChecks; the summary keeps its worst-check meaning.	2026-10-07 13:40:46 +01:00
rcourtman	83ec32ee71	Keep fixture alerts from outliving a mock-mode switch (#2656) ... Leaving mock mode cleared active alerts, but every path that evaluates mode-dependent data checked the mode once, up front. An evaluation already holding fixture data when the switch happened landed its alerts after the clear, in live mode, where nothing evaluates or resolves fixture resources, so they stayed until the 24-hour stale sweep. A pass-level lock around `checkMockAlerts` was tried and reverted: it closed only that path and made the mock-mode API wait behind a whole pass of synchronous event-log writes. **What changes** - `mockModeFence` (`internal/monitoring/mock_mode_fence.go`): each path takes a scope before reading anything mode-dependent and sends each alert-manager call through `scope.run`. Covered: `checkMockAlerts` (every loop, host agents, disks, snapshot cache), `checkBackupAlerts`, `pollGuestSnapshots`, `checkConnectionAlerts`, `pruneStaleDockerAlerts`, and the resource-store refresh (`updateResourceStore`, `updateResourceStoreForRead` and `syncUnifiedResourceAlertsToState` now take the scope of the state they rebuild from). - `SetMockMode` is serialized, flips the mode, ends the epoch, waits only for calls already admitted, then clears. Unified metrics go one resource per call, so the wait covers single calls, not passes. Admission never blocks. - Registry rebuilds are tracked, not waited on: a refresh evaluates the registry it read back only if no rebuild from an ended epoch overlapped it. Readers that act on the inventory (Docker pruning, backup and snapshot guest lookups, the pollers' previous-state carry) use current state until an undisturbed rebuild in the new epoch replaces the fixture registry. - `mock.SupplementalRecords`/`SupplementalChanges` serve nothing outside mock mode. The router swaps its mock adapters for the real pollers only after the switch, refreshing between sources, and the adapters used to fall back to default fixtures, so a single switch to live mode raised fixture vSphere incidents. **Verified** - Each new test parks a real evaluation at a fixed point (a held lock, a blocking supplier or a gated registry read), switches modes and releases it, with bounded waits. Against the unfenced code: the mock pass left about 40 fixture alerts, the frontend refresh 5 vSphere incidents, the backup check 137 backup-age alerts, and a mock-mode connection observation counted toward a live connection's confirmation. The drain, republish, pruning and carry tests fail against a fence without its wait, overlap check or registry tracking; the republish test also fails with the old mock supplemental defaults. - 995 tests from the touched files and related areas pass, plus `internal/mock` and the `internal/api` mock/platform tests; the new tests pass under `-race` and over repeated runs. **Not covered** - The fence and clear are per monitor while mock mode is process-wide; other tenant monitors neither drain nor clear on a switch (unchanged, noted in the monitoring contract). - Report admission and live pollers outside the fenced paths can still evaluate after the clear when entering mock mode; their alerts can persist until mock mode is left.	2026-10-07 13:40:00 +01:00
rcourtman	5814c0a44c	Stop a failing availability check keeping a node online ... Once every source with a verdict had gone quiet, the stale pass counted any current availability sighting as online. A node that preserveOrExpireNodes expired stayed online while its HTTP check failed, and a quiet failing check lifted that node's offline verdict to warning. One sighting stands for every check on a target and carries whichever check was projected last, so a failing check that had just run also vouched for another check's old pass. Status aggregation now judges each check on a monitored resource by its own evidence window. A passing check whose evidence is still valid proves the resource answers, and reads online when no current source has a verdict. Every other check abstains, current or quiet: a failing check proves only that one port or service does not answer, so it never makes a guest count as stopped in the workloads summary or the Proxmox page. A check's own row keeps the ordinary rules, and a manual link judges the merged resource by the checks it carries. Projecting or retargeting a check now re-applies the stale pass's rule to targets whose status that pass owns, since the resources API replays checks after its stale pass and never runs another. Only a target's own quiet sources give the pass that ownership, never its checks' sighting. The poller now keeps each check's evidence tied to the observation that passed. It ages out a local location that stopped running beside other locations, as it already did for silent remote paths: the check took its freshness from the newest location, so a failing remote report kept an hour-old local pass counted as reachable with current evidence. An available check's evidence lasts only as long as its newest reachable path, and a remote result adopted as local after an entitlement lapse keeps the server's receipt time rather than the agent's clock.	2026-10-07 13:38:52 +01:00
rcourtman	11cc232133	Stop reporting unavailable storage as disabled to AI tools (#2657) ... The storage pool rows the AI tools build (pulse_storage pools and pulse_query storage pools) derived both enabled and active from "status is not offline". Storage that reported itself unavailable, such as Proxmox storage with active=0, reached Patrol and Assistant as administratively disabled even when it is enabled in Proxmox. Enabled now comes from the storage facet's own flag. Active needs the facet's flag and a row that is not offline, because PBS datastores report Active even while unavailable. Rows without a storage facet keep the status-derived value. TrueNAS pools and datasets were the only storage producers that left both flags at their zero value, so reading the facet alone would have turned every TrueNAS pool into a disabled one. They now report Enabled, and Active from the pool state or the dataset's mount and lock state.	2026-10-07 13:35:59 +01:00
rcourtman	bc15c3500b	Stop QEMU default disk serials merging nested Proxmox disks (#2653) ... ## Problem Nested Proxmox clusters (each PVE node is itself a QEMU VM) lose physical disks from inventory. QEMU gives a virtual disk configured without a serial a default one built from its drive ID or a per-VM counter, so every VM built the same way reports the same value: - Proxmox's `/nodes/{node}/disks/list` serial is udev's `ID_SERIAL_SHORT` (`PVE/Diskmanage.pm`). A virtio-scsi disk on a Proxmox VM reports `drive-scsi0` with no `ID_WWN`; on a libvirt VM it is `drive-scsi0-0-0-0`. A `-drive` given no ID falls back to QEMU's automatic ID, `scsi0-hd0` for `if=scsi` and `none0` for `if=none`. - An emulated ATA disk's serial is `QM` plus a five-digit counter (`QM00001`). `diskinventory.IsUsableHardwareID` accepted all of these, so `resourceFromPhysicalDisk` set `MachineID = "drive-scsi0"` on every node and the registry minted one `machine:drive-scsi0` canonical disk and one SMART series for the whole cluster. Three nodes with two such disks each produced 2 canonical disks instead of 6. ## Change - `IsUsableHardwareID` rejects exactly QEMU's default shapes: `drive-scsi` plus digits with at most three `-digits` groups, `scsi<n>-hd<n>`, `none<n>`, and `QM` plus five digits. Real serials that only share a prefix (`DRIVE-ASSET123`, `drive-sata0`, `QM0001`) stay identity. It also strips `naa.`/`eui.`/`wwn-`/`0x` framing before its all-zero and all-F test, so a `0x0000000000000000` WWN is a placeholder too. - TrueNAS provider: a disk took its canonical identity from any non-empty serial, so a placeholder merged disks across appliances. It now uses only a usable serial, and lists superseded IDs for every disk without one. - TrueNAS history keys: for a serial that was already rejected, such as `UNKNOWN`, native TrueNAS history used TrueNAS's own disk identifier, and the unified SMART writer (`syncUnifiedPhysicalDiskMetrics`) substituted the metrics target only for an empty serial, so the samples went to the canonical resource ID. Neither matched the chart reader's key, and rejecting QEMU serials would have put TrueNAS VMs on that path. Native history now falls back to the disk's source ID, and the writer always writes under the resolved metrics target, so writer, native history and reader agree. History lookup keys also drop placeholder serials. ## Upgrade effect A disk reporting one of these values (nested PVE nodes, TrueNAS or agent hosts running as QEMU VMs with virtual disks, TrueNAS disks reporting `UNKNOWN`) changes canonical ID and SMART metric key once. Samples recorded under the old key are not carried over, because that key mixed every disk that shared the value. Path-shaped alert references (Proxmox disk health and wearout, host-agent SMART) are unaffected; anything keyed on the old canonical ID or metric key starts again under the new one. This is recorded in the monitoring and agent-lifecycle contracts. ## Tests - `TestIsUsableHardwareIDRejectsQEMUDefaultDiskSerials`: each default shape is rejected, falls back to the scoped topology ID and never matches itself; prefix-sharing real serials stay usable. `TestPreferredIDRejectsPlaceholderHardwareIdentity` gains the framed all-zero/all-F WWNs. - `TestNestedProxmoxDefaultQEMUSerialsStayPerNode`: 3 nodes × (`drive-scsi0`, `QM00005`, `drive-scsi0-0-0-1`) give 9 canonical disks, 9 slots and 9 metric keys, every SMART writer key is a key readers resolve, and agent SMART source IDs stay per host. - `TestTrueNASPlaceholderDiskSerialsStayPerApplianceAndShareOneHistoryKey`: two appliances with `UNKNOWN` and `drive-scsiN` serials keep one canonical disk each, and the SMART writer's samples are readable under each disk's metrics target. - `TestProviderPlaceholderDiskSerialsKeyNeitherIdentityNorHistory`: placeholder serials key no identity, list superseded canonical IDs, drop out of native history lookup keys, and native history lands under the registry's metrics target. - Negative controls via `go test -overlay`, one runtime file reverted at a time: base `identity.go` fails all three round trips (3 canonical disks instead of 9, 9 TrueNAS disks instead of 16); base `provider.go` fails the identity and history assertions; base `monitor.go` fails the SMART-history-under-reader-key assertion; reverting only the native history key, the superseded-ID gate or the lookup-key filter fails its own provider assertion. - `pkg/diskinventory`, `internal/truenas/...`, `internal/unifiedresources/...` and `internal/monitoring` pass in full; `internal/api/...` disk, hardware, serial and TrueNAS tests pass. Stacked on #2648 (`fix/pve-disk-serial-carry`), whose commit this branch carries until it merges. Not in this change: the registry's merge of disks across parents on a canonical-ID collision (separate change), and a pre-existing writer/reader key mismatch for controller-member disks without a usable serial or WWN (`PhysicalDiskMetaMetricID` appends topology to a source ID that already carries it), which is tracked separately.	2026-10-07 13:22:34 +01:00
rcourtman	425dd0d45b	Keep same-serial disks on different machines apart ... findMatch refuses a physical-disk candidate under another parent, but ingestRecord then minted the same hardware-keyed canonical ID from the serial and merged into its holder on the collision. Disks that report one usable serial on several machines (dual-ported SAS shelves, cloned VMs with an explicit serial, fixed-serial USB bridges, TrueNAS systems) collapsed into one resource parented to whichever machine ingested last, and every other machine lost the disk from inventory. A new disk now resolves its parent to a machine (the nearest host ancestor, so pools and Unraid arrays count as their host) and stays apart only on evidence: one reporter placing the serial on two machines, or machines whose hostnames share nothing. Reporters on one box still join across grouping parents, so the TrueNAS API and the agent on that box stay one disk, and an unknown machine never splits a disk. Once a serial spans machines, every copy takes the serial keyed to its own machine and the unscoped holder is re-keyed, so IDs do not depend on machine ingest order and the unscoped ID never passes between disks that coexist. A disk only one machine reports keeps its ID. Operator splits now hold on this path; the old collision merge ignored them. The registry indexes disks by hardware key because the identity matcher keeps one resource per machine ID.	2026-10-07 12:56:47 +01:00
rcourtman	8d00d54740	Stop a stale agent SMART row lending a replaced disk its identity ... The linked host agent's SMART merge falls back to matching a Proxmox disk by device path, and a path names a slot, not a disk. When a disk is swapped before the agent reports again, or after the agent went silent and kept its last rows, the row for the old disk matched the replacement. It filled or, for SAS, promoted the old serial onto it, which keyed the replacement to the old disk's canonical resource and metric history, and lent it the old disk's temperature and failed health. For SAS the registry's same-path join did the same at the canonical level even when monitoring kept the rows apart. Both now refuse a row whose identity contradicts the disk's, judged only on what both producers report like for like. WWNs count for every disk. Serials count only where Proxmox's serial is the drive's own, as smartctl's is: NVMe namespaces, and disks presented with SCSI vendor "ATA", whose serial udev's ata_id reads from IDENTIFY. Elsewhere Proxmox's serial may be a SAS address, a SCSI designator or a USB bridge's own serial (#1595). Reporter spellings are not disagreement: udev writes whitespace in serials as underscores, its T10 designator for an ATA drive puts "ATA" and the model before the drive serial, and its ID_WWN keeps only the first 64 bits of an NAA 6 identifier. The WWN match step compares normalized identity too, so the agent's 5-c50-a1b2c3d4 spelling finds the Proxmox 0x5000c500a1b2c3d4 disk. Proxmox's disk inventory reports a serial or WWN udev cannot read as the literal "unknown". The poller recorded it as a collected serial, so the collection state hid the gap and the agent's serial never filled it. It is now recorded as empty, and retained evidence carries only a usable previous serial so an older record cannot restore it. Identity keys already ignored the placeholder.	2026-10-07 12:16:36 +01:00
rcourtman	6fb9752397	Stop reporting unavailable storage as disabled to AI tools ... The storage pool rows the AI tools build (pulse_storage pools and pulse_query storage pools) derived both enabled and active from "status is not offline". Storage that reported itself unavailable, such as Proxmox storage with active=0, reached Patrol and Assistant as administratively disabled even when it is enabled in Proxmox. Enabled now comes from the storage facet's own flag. Active needs the facet's flag and a row that is not offline, because PBS datastores report Active even while unavailable. Rows without a storage facet keep the status-derived value. TrueNAS pools and datasets were the only storage producers that left both flags at their zero value, so reading the facet alone would have turned every TrueNAS pool into a disabled one. They now report Enabled, and Active from the pool state or the dataset's mount and lock state.	2026-10-07 12:15:44 +01:00
rcourtman	41cea2ffab	Stop a replaced PVE disk inheriting the old disk's evidence (#2648) ... ## Problem The Proxmox disk poller keeps evidence it could not collect this poll by looking up the previous poll's record of the disk: by serial, then WWN, then the Proxmox source ID (`instance-node-devpath`), then the device token. The source ID and device token name a slot, not a disk, and a replacement in the same slot keeps both. So: - a replacement reporting a different serial or WWN took the old disk's retained temperature, controller, pool and I/O; - a replacement record with an empty serial also took the old disk's serial. The registry keys disks on serial, so the new disk became the old disk's canonical resource and SMART metric series, and every later poll carried the borrowed serial forward. Current Proxmox reports a missing serial as the literal `unknown`, which the poller records as reported, so the serial copy needs a producer that sends an empty serial, such as the host-agent fallback rows. The evidence copy happens with real Proxmox data. ## Change - `previousPhysicalDiskEvidence` matches by hardware identity through `diskinventory.HardwareIdentityMatch` (serial first, then WWN, each against both fields of the earlier record), and refuses a slot match when both records carry a serial or WWN and none agrees. Missing identity is still not a conflict: placeholders such as `unknown`, and an empty serial, keep the slot's evidence as before. - The registry's merged disk view keeps the host agent's WWN, which the agent spells as unpadded NAA 5 fields (`5-c50-a1b2c3d4`, `internal/hostagent/smartctl.go` `formatWWN`) while Proxmox reports `0x5000c500a1b2c3d4`. `HardwareIdentityMatch` now re-pads that spelling for WWN fields only, so one disk seen by both reporters is not taken for a replacement. The registry's linked-disk join uses the same comparison. ## Tests - `TestPhysicalDiskReplacementInSameSlotDoesNotInheritPreviousSerial` drives the poller and registry across two polls after a swap; on base the replacement inherits the old serial and canonical resource. Its controls keep a serial-omitting same disk on its resource. - `TestPreviousPhysicalDiskEvidenceRejectsSlotMatchAcrossConflictingIdentity` covers renamed devices matched only by identity, PVE vs agent WWN spellings, SAS address serials with a matching WWN, placeholders, and conflicting serial/WWN by source ID and by device token. - Negative controls via `go test -overlay`: on base the replacement, conflict and renamed-device cases fail; with the guard but without the WWN re-padding, the same-disk PVE/agent WWN cases fail. - `pkg/diskinventory`, `internal/unifiedresources` and `internal/monitoring` pass in full. Follow-ups, not in this change: the host-agent SMART merge's device-path fallback can still fill or promote a serial from a stale SMART row; QEMU default serials (`drive-scsi0`) collapse nested Proxmox nodes' disks into one resource; and the registry merges disks across parents on a canonical-ID collision.	2026-10-07 12:12:23 +01:00
rcourtman	73a2d68cf9	Pin disk heat parity for canonical and linked-guest overrides ... A second review of the per-host Disk Temp change found two gaps. The parity test did not cover an override stored under the canonical resource an agent ID resolves to, or one inherited from a linked guest. The documented browser limits also listed only two cases where the browser cannot see CheckHost's chain. The parity test now checks both overrides, and a disabled linked guest, against what CheckHost fires. storage-recovery.md lists every case where the browser's keys can differ from alerts and Patrol: manually merged agent and VM disks on the Storage page, overrides inherited through unpublished link hints or stored under a linked node's canonical ID, and guest keys for an agent merged into a guest it is not automatically linked to.	2026-10-07 12:08:03 +01:00
rcourtman	06b7f5a0b9	Keep fixture alerts from outliving a mock-mode switch ... SetMockMode(false) switched the mock package off and then cleared active alerts, but every path that evaluates mode-dependent data checked the mode once, up front. A mock alert pass (started from the ticker with go, so passes overlap), a frontend or ingest refresh evaluating a GetState snapshot, a backup-age check over fixture recovery rollups, or a connection check over the mock ledger could already hold fixture data when the switch happened. Its evaluations then landed after the clear, in live mode, where nothing evaluates or resolves fixture resources, so the alerts stayed until the 24-hour stale sweep. A pass-level lock around checkMockAlerts was tried and reverted: it left the other paths open and made the mock-mode API wait behind a whole pass of synchronous event-log writes. The monitor now owns a mock-mode epoch fence. The mock pass, backup-age and snapshot checks, connection checks, Docker pruning and the resource-store refresh take a scope before reading anything mode-dependent and send each alert-manager call through it. SetMockMode, now serialized, flips the mode, ends the epoch, waits only for calls already admitted, and then clears. Unified metrics are evaluated one resource per call, so the wait covers single calls, not passes. Registry rebuilds are tracked rather than waited on. A refresh evaluates the registry it read back only if no rebuild from an ended epoch overlapped it, and the readers that open or remove alerts from the inventory use current state until a clean rebuild in the new epoch replaces the previous mode's registry. The mock supplemental fixture sources now serve nothing outside mock mode: the router swaps them for the real pollers only after the switch, refreshing between sources, so a single switch to live mode used to raise fixture vSphere incidents. The new tests park a real evaluation at a fixed point (a held lock or a blocking supplier), switch modes and release it. Against the unfenced code the mock pass left about 40 fixture alerts, the frontend refresh 5 vSphere incidents and the backup check 137 backup-age alerts in live mode, and a mock-mode connection observation counted toward a live connection's confirmation. The drain, republish and pruning tests fail against a fence without its wait, overlap check or registry tracking. Switching mock mode with several tenant monitors running remains uncovered and is noted in the monitoring contract.	2026-10-07 12:03:38 +01:00
rcourtman	a28422ae26	Judge disk heat under the reporting host's Disk Temp override ... Disk temperature alerts evaluate each SMART disk under the thresholds of the host agent that reports it. An explicit Disk Temp override on that host, or one inherited from its linked node or guest, replaces the per-type trigger, and a host whose alerts are switched off raises none. Since #2636 the Physical Disks verdict and Patrol judged heat by disk type alone. A host with an 80C override still read Running Hot at the 70C NVMe trigger, and a host with its alerts off still read hot. - alerts.Manager.HostDiskTemperatureThreshold resolves the policy for a host exactly as CheckHost does, from the same helpers. DiskTemperatureThreshold is its hostless case. - Patrol and the AI chat context find each disk's host by walking its parent to the agent machine (through the pool for Unraid disks) and ask the threshold adapter for that host's limits. - Physical Disks, the disk detail, the pool drawer's linked disks and the guest drawer pass the reporting machine's override keys to the thresholds resolver, in the order CheckHost reads them: agent ID, canonical ID, then the merged node or guest. The resolver treats an override or agent default that switches alerts off as no heat. Two cases still fall back to the per-type trigger in the browser while alerts and Patrol apply the override: on the Storage page, a disk whose agent was manually linked into a VM, and an override an agent inherits from a guest it is linked to without a merge.	2026-10-07 11:54:53 +01:00
rcourtman	044dc091ec	Own PVE disk alert history rows by recorded hardware (#2646) ... Follows #2627, which left PVE disk alert references out of resource history: binding the path reference failed review because a binding belongs to the reference string, so any disk that takes over the path inherits every row and every read of it. A physical disk's history (facets, timeline and intelligence APIs, Assistant resource context, Patrol's scoped feed) therefore never showed its own SMART health and wearout alerts. This owns each lifecycle row instead of binding the reference. **Write** - `proxmoxDiskAlertMetadata` now records `disk_wwn` next to `disk_serial`. - `MonitorAdapter.RecordChange` writes a PVE disk alert row under the one physical disk the registry knows by that serial or WWN (`proxmoxDiskAlertOwner`), wherever it sits now. A WWN only breaks a tie between two matches; it never rules a disk out, because the registry keeps a merged agent's smartctl-framed WWN (`5-c500-…`), which doesn't compare equal to PVE's (`0x5000c500…`). The registry only translates identity into a canonical ID. The path never breaks a tie, so a generation that still places another disk at the path cannot gain a row that disk does not match by identity, and no check precedes a later write. A disk not yet in inventory, or removed, gets the ID the registry mints for its identity unless another resource holds it. A row with no usable identity names only the one identity-less disk at its path. Anything else stays under the reference. - An owned row records the alert's reference as `alert_resource_id`. No alias is written, the reference is never queued for the legacy backfill, and `resolveHistoryReference` refuses it. - Rows journaled before this change stay under the reference, unrewritten (journal rows are never rewritten; retention is 30 days). **Read** - A store read by the reference (rows and every count, SQLite and MemoryStore) also returns the rows owned away from it. They are matched through the indexed alert identifier plus the recorded reference, so the Alerts history Resource action lists the same occurrences as before. - `QueryIncidents` matches owned rows to occurrences under their alert reference, while their evidence keeps the owning disk. Reads by alert identifier and by the reference return the same occurrences as before, including one occurrence whose firing and resolution two disks own. A read by a disk admits the saved shells of its own occurrences. An owned row whose firing the read lacks takes the occurrence or saved shell at its recorded start (shells within the usual tolerance), or opens it under the alert-centric read's ID, before events are assigned. That boundary is not canonical, so capped reads keep their guard. Notes recorded from a disk read reach the occurrence the alert-centric read shows. **Known residuals (in the unified-resources contract)** - History follows inventory where inventory merges: same-serial disks, a serial carried at a path when a poll reports none, and disks correlated with an agent observation by path. - When PVE and the agent share no identifier for such a disk (a SAS address reported as the serial), or the alert fired before the PVE observation reached the registry, the row carries the minted ID. The disk's history omits it, and reads by the reference still return it. - A disk the registry keys by its agent's WWN alone also carries PVE's serial, so a different disk reporting that same serial, before its first poll reaches the registry, has its row written under the first disk. - Two occurrences of one alert starting within the shell tolerance: the alert-centric read assigns a saved shell by firing order, which a disk read missing a firing cannot reproduce. **Tests** - `TestProxmoxDiskAlertRowsFollowRecordedHardwareIdentity` (SQLite and memory): reboot reorder before the registry catches up, replacement before and after its first poll, WWN-only, path-only, placeholder serial, counts. - `TestProxmoxDiskAlertOwnerDecisions`: shared serials, WWN tie-break, agent-framed WWN, RAID volume (full NAA serial, truncated WWN), minted ID already held, identity-less paths. - `TestHistoryIdentityNeverBindsProxmoxDiskAlertReferences`, which replaces `TestHistoryIdentityLeavesProxmoxDiskAlertReferencesUnbound`. - `TestProxmoxDiskAlertReferenceReadUsesIndexes`: the read by reference uses the canonical and alert-identifier indexes, not a scan. - `TestIncidentQueryKeepsOccurrencesOfHardwareOwnedDiskAlerts` compares the journaled and owned layouts for alert-identifier, `StartedAt` and reference reads, plus disk reads, notes, evidence and a split occurrence with no shell, an exact shell and an offset shell, on both stores. `TestIncidentQueryTruncatedReadKeepsOwnedOccurrencesGuarded` covers capped reads. - `TestCheckDiskHealthAlertsCarryHistoryOwnershipIdentity`: the producer's alert identifiers match `ProxmoxPhysicalDiskAlertIdentifiers`. - `TestProxmoxDiskAlertTimelineFollowsRecordedHardware`: real alert manager, reboot reorder, incidents by reference and by disk. - Each new test fails against the code it guards (checked by reverting it). **Live check** (private mock-mode build, fresh data dir): 6 fired rows and 1 acknowledgement, all owned by the disk their serial names. No aliases. Each disk's timeline API returns its rows. `/api/ai/incidents?resource_id=<reference>` and `?resource_id=<disk>` return the same incident (same saved ID; fired and acknowledged events whose evidence names the disk).	2026-10-07 11:53:57 +01:00
rcourtman	dd8bfa4db8	Stop QEMU default disk serials merging nested Proxmox disks ... QEMU gives a virtual disk configured without a serial one built from its drive ID or a per-VM counter, so every VM built the same way reports the same value. Proxmox's disks/list serial is udev's ID_SERIAL_SHORT, which for a virtio-scsi disk is "drive-scsi0" (libvirt: "drive-scsi0-0-0-0"), and an ATA disk's serial is "QM00001". IsUsableHardwareID accepted these as hardware identity, so every node of a nested Proxmox cluster minted one machine:drive-scsi0 canonical disk and one SMART series, and all but one node lost the disk from inventory. IsUsableHardwareID now rejects exactly QEMU's default shapes: drive-scsi plus digits with at most three -digits groups, QEMU's automatic drive IDs scsiN-hdN and noneN, and QM plus five digits. Real serials that share a prefix, such as DRIVE-ASSET123, stay identity. It also strips WWN framing before the all-zero test, so a 0x0000000000000000 WWN is a placeholder too. The TrueNAS provider set a disk's canonical identity from any non-empty serial, so placeholders merged disks across appliances. It now uses only a usable serial. For a serial that was already rejected, such as UNKNOWN, native TrueNAS history and the unified SMART writer disagreed with the chart reader: native history used TrueNAS's disk identifier, and the writer substituted the metrics target only for an empty serial, so a placeholder sent samples to the canonical resource ID. Rejecting QEMU serials would have put TrueNAS VMs on that path. Both now use the key the registry's metrics target resolves, the disk's source ID, and native history lookups no longer route by placeholder serials. Disks that report one of these values change canonical ID and SMART metric key once; samples under the old key are not carried over because that key mixed every disk that shared the value.	2026-10-07 11:51:16 +01:00
rcourtman	9b644b3d49	Stop a replaced PVE disk inheriting the old disk's evidence ... The disk poller keeps evidence it could not collect this poll by looking up the previous record, first by serial and WWN, then by the Proxmox source ID and device token. Those last two name a slot, not a disk, and a replacement in the same slot keeps both. So a replacement reporting a different serial or WWN took the old disk's retained temperature, controller, pool and I/O. A replacement record with an empty serial also took the old serial. The registry keys disks on serial, so the new disk became the old disk's canonical resource and SMART metric series, and each later poll carried the borrowed serial forward. Match on hardware identity through diskinventory.HardwareIdentityMatch, serial first then WWN, and refuse a slot match when both records carry a serial or WWN that share no value. Missing identity is still not a conflict, so Proxmox's literal "unknown" and other placeholders keep the slot's evidence as before. The registry's merged disk view keeps the host agent's WWN, which the agent spells as unpadded NAA 5 fields ("5-c50-a1b2c3d4") while Proxmox reports "0x5000c500a1b2c3d4". HardwareIdentityMatch now re-pads that spelling for WWN fields, so the same disk seen by both reporters is not mistaken for a replacement.	2026-10-07 11:23:34 +01:00
rcourtman	1f2ff618b6	Own PVE disk alert history rows by recorded hardware ... PVE disk health and wearout alerts reference the disk by instance, node and device path, so a physical disk's history never showed its own SMART and wearout alerts. Binding the path reference failed review: a binding belongs to the reference string, so any disk that takes over the path inherits every row and every read of it. Own each lifecycle row instead. The alert now records the disk's WWN next to its serial, and the history writer puts the row under the disk the registry knows by that hardware identity, wherever it sits now. The registry only translates identity into a canonical ID, so a generation that still places another disk at the path cannot claim the row, and no check precedes a later write. A disk not yet in inventory, or removed, gets the ID the registry mints for its identity. Rows without usable or unambiguous identity stay under the reference, as do rows journaled before this change; journal rows are never rewritten. Reads by the reference still return every row raised under it, through the alert identifier index, and the incident projection keeps owned rows under their alert reference, so the Alerts history Resource action and incident timelines show the same occurrences as before. A read by the disk returns its own occurrences with their saved shells and notes.	2026-10-07 11:18:45 +01:00
Richard Courtman	ff67e38968	Pair a merged disk's temperature with its rows' own states ... The registry merges a physical disk's rows one step at a time, choosing the temperature by source preference and merging collection states separately. The pairing rule from #2618 was exact for two rows only: with three or more, an earlier step's single merged state stood in for its rows. A silent agent, an offline Unraid inventory and a stale Proxmox copy could leave a withdrawn 41 labelled collected, and a legacy agent's stateless reading could pick up another row's "unsupported". The merged disk now keeps each row's current reading and state (merge-time only, never serialized), and the shown value takes its state from the readings of that value: a collected reading first, then the agent's own report over a copy. A copy of an agent source's state gives way to the agent's own withdrawal, including a legacy agent's sourceless one, and a value source preference keeps after its row reported again keeps its presented state under that row's later withdrawal. Value selection is unchanged.	2026-10-07 11:18:10 +01:00
rcourtman	8b36dec0a2	Keep a source's own offline verdict through the stale pass (#2643) ... ## Problem The unified registry's stale pass recomputed a resource's status from its source sightings, and read a sighting that had *delivered recently* as *online*. A sighting only says the source delivered, not what it reported: - A Proxmox node the cluster reports offline on a live poll came back `online` once its linked host agent went silent, which is exactly what a powered-off machine's agent does (the #1441 / discussion #1135 symptom for users with an agent on the node). The Proxmox nodes table showed an amber dot next to its own "Offline" label while the drawer led with a critical "Node is offline" alert. - A node `preserveOrExpireNodes` expired after failed polls read `warning` ("Degraded"), and the stopped guests on it were counted under "Attention". Node alerts were not affected: `CheckNode` reads the poller's own `models.Node`. Everything that reads the projection was (`/api/resources`, websocket state, `NodesSnapshot`, Patrol and AI tools, connections grouping, reporting, charts). ## Fix Each sighting records the status its source reported (an unexported verdict that replaces the narrower lease marker from #2634), and `aggregateStatus` reads it: - While any source with a verdict is current, only the current sources decide, in the priority order the merge uses (agent, then Proxmox or vSphere, then Docker), so a live agent still keeps a node online when the Proxmox API path to it fails. - Once every source is quiet, an `offline` verdict survives and anything else reads `warning`, as before. - Sightings still report delivery, so health keeps its `telemetry_stale` reason. Availability checks projected onto a target keep counting by delivery, exactly as on main. Giving them a verdict needs each check's own freshness, and that is left to a follow-up (documented in unified-resources.md). ## Verification - Registry tests in `registry_merge_policy_test.go`: live poll reports offline with a silent agent (inside and past its lease), poller-expired node alone and with its agent, stopped and running guests on a quiet poll, JSON round trip, in-memory copy keeping a facet without a verdict, and an aggregation table. The node, poller-expired, guest and round-trip tests fail on main with the bug's exact symptoms; all of #2634's lease tests still pass. - Private mock stack (demo estate, one node held offline on a fresh poll with its agent silent): before, both offline nodes showed amber ("Offline"/"Degraded" pills) and 44 stopped guests counted as Attention; after, both read Offline and the guests moved to Stopped. A full status diff of ~1,780 resources showed no other systematic change. - `go test` for unifiedresources, monitoring, ai, alerts, mock, api, api/resourceapi and api/chartapi passes.	2026-10-07 10:56:28 +01:00
rcourtman	ea1a07a721	Keep a source's own offline verdict through the stale pass ... The registry's stale pass recomputed a resource's status from its source sightings, and read a sighting that had delivered recently as "online". A sighting only says the source delivered, not what it reported, so: - A Proxmox node the cluster reports offline on a live poll came back online once its linked host agent went silent, which is what a powered-off machine's agent does. The Proxmox nodes table showed an amber dot beside its own "Offline" label while the drawer led with a critical "Node is offline" alert. - A node the poller expired after failed polls read as a warning ("Degraded"), and the stopped guests on it counted as needing attention. Each sighting now records the status its source reported, and the aggregation reads that verdict. Current sources decide in the same priority order the merge uses, so a live agent still keeps a node online when the Proxmox API path to it fails. A source that went quiet drops out while another is current. Once every source is quiet, an offline verdict survives and anything else reads as warning, as before. This generalises the lease marker that kept a silent agent offline, so a stopped guest keeps its offline verdict too when its only source goes quiet. A running guest whose only source goes quiet still reads as warning. Sightings still report delivery, so health keeps its telemetry_stale reason. An in-memory copy keeps its verdicts; only a copy that lost them all, a serialized one, takes its stored status as each source's verdict. Availability checks projected onto a target still count by delivery, as before. Giving them a verdict needs each check's own freshness, so it is left to a follow-up.	2026-10-07 10:17:49 +01:00
rcourtman	79b6a0974e	Name disk wearout as life remaining and log disk alerts once ... The Proxmox disk wearout alert read "SSD has less than 10% life remaining (7% wearout)". The number is percent life remaining, so "7% wearout" reads as 7% worn and contradicts the first clause, and the Alerts overview shows that text verbatim. It now reads "SSD life remaining is 7%", the wording host-agent SMART wearout alerts already use. The threshold stays in the alert's Threshold field, which the card and notifications show beside the message; spelling it into the sentence as "below 10%" would be false at exactly 10%, which alerts. CheckDiskHealth also logged a failed disk at error and a worn disk at warn on every evaluation, not when the alert opened. The physical disk poll runs every 5 minutes by default, so one failed disk logged an error 288 times a day for as long as it stayed failed, and mock mode repeated it on every alert tick. Both now log at error or warn only on the evaluation that opens the alert and at debug afterwards. The new log test captures through a hook installed in TestMain: assigning log.Logger inside a test would race with managers that earlier tests leave running. The alerts contract also said wearout recovery needs readings of at least 10%; a 10% reading still matches, so recovery needs readings above it.	2026-10-07 10:10:46 +01:00
rcourtman	a21e0ef8f8	Stop reports and resource history calling a moved alert recovered (#2607) ... Follows #2591. A node alert that moves to its linked Pulse agent closes with a `moved_to_agent` resolution (#2580). Two readers still presented that close as a recovery. **Performance reports.** The node, VM and container enrichers copied recently resolved alerts into `reporting.AlertInfo` without the resolution, so a report generated in the five-minute window after a handover read HEALTHY, "All systems operating normally", counted the alert as "triggered and resolved", and sent the AI narrator `resolved: true`. - Report rows now carry the alert's resource and resolution (`reportActiveAlertInfo` / `reportResolvedAlertInfo`). - `pkg/reporting` has one rule (`unresolvedAlerts`): an alert that closed without recovering stays live unless the report also shows its successor's reading for that metric, meaning an open alert or a recovery since the handover. That way the condition is counted once, and fleets match handovers across all their resources. - The single-resource PDF card reads WARNING (or CRITICAL at the alert's level) with "Alert moved to <agent> - check the agent for the current reading". The active list names the agent, the alerts table shows "Moved" with the engine's summary as a note, and recommendations say to check the agent. - Fleets count the alert as active and name the agent. Their all-clear pattern and "No fleet-wide action required" no longer appear while any alert is active. - The narrator payload carries `resolved: false`, the summary and `successor_alert_listed`, with prompt rules for both. **Resource history.** `alert_resolved` changes were labelled by kind alone ("Alert resolved", green) with that label prefixed to the headline. `getResourceChangePresentation` now reads the `alert_resolution` metadata: the change reads "Alert moved" in the neutral blue of the history badge, and its summary is the whole headline in the drawer, `ResourceChangeSummary` and Patrol's assessment handoff. **Verification** - A private mock stack running this branch, with a local-only overlay that closes a real pve1 memory alert through `clearAlertWithResolution`. Within the window, `/api/admin/reports/generate?resourceType=node` rendered CRITICAL with the handover message and a "Moved" row, while the ordinary CPU recovery stayed green. - In the browser, a full drawer (Availability checks, Fleet view) showed the producer-made rows as "Alert moved" next to "Alert resolved" at 1280 and 375. A Proxmox node's own drawer uses the `table-row` presentation, which fetches no change history (see below). - Tests in registered proof files: `pkg/reporting/alert_state_test.go` (verdict, successor timing, fleet pool, PDF text and width), `internal/api/contract_test.go`, `internal/ai/report_narrator_test.go`, `resourceChangePresentation.test.ts`, `ResourceDetailDrawer.history.test.tsx` and `patrolInvestigationContextModel.test.ts`. Each failed with the fix reverted. - Second opinion: two read-only review rounds. The temporal match, fleet-wide pool, narrator dedupe signal, fleet all-clear fallbacks and contract wording came from them. **Not fixed here (separate changes)** - Reports requested as `agent` carry no alert rows at all. That is how the Reports page addresses every Proxmox node in v6, so such a report reads HEALTHY even with active critical alerts. This change makes the shared rule correct for when those reports attach the machine's alerts. - Node and guest alert lifecycle changes are recorded under legacy alert resource IDs, which canonical history queries never match. Facet kind counts still group moved closes under "Alert resolved".	2026-10-07 09:10:57 +01:00
rcourtman	097858bc41	Merge remote-tracking branch 'origin/main' into fix/alert-card-live-reading	2026-10-07 08:45:23 +01:00
rcourtman	bea0cec8b9	Keep PVE disk alert references out of resource history (#2627) ... PVE disk health and wearout alerts reference the disk by instance, node and device path (`<instance>:<node>:disk:<device key>`). Operator mutes resolve that reference to the disk at the path (#2112), but resource history leaves it unbound, so a physical disk's history (facets, timeline and intelligence APIs, Assistant resource context) does not show its own SMART and wearout alerts. This records why that stays the decision, and pins it with a test. - A device path is not hardware identity. A replacement disk in the same slot, or a reboot that reorders devices, takes over the path, and disk alerts are evaluated before their poll's disks reach the registry, so an event can resolve against a generation that still places another disk there. - A history binding belongs to the reference string and moves to its latest target, carrying every row journaled under the reference and every read of it. Bound, the path would put a replaced disk's alerts into its successor's history, and the Alerts history Resource action (which reads incidents by the alert's own reference) would show the successor's incidents for the predecessor's alert. - A binding guarded by the serial each disk alert records was built and measured on a mock estate (mock mode needed a build overlay to evaluate PVE disk alerts at all). It bound correctly while disks kept their paths, but review found rejected events still joining the currently bound disk, a removed retained target accepting a successor's event, no WWN in the alert metadata, a non-atomic check and write, and retry expiry abandoning rows. That attempt is not in this PR. Binding these alerts needs ownership per row (each event under the disk its recorded hardware identity names, rows without usable identity left under the reference, and a decided read for the Resource action), not a per-reference binding. The contract paragraph in `unified-resources.md` says so. Test: `TestHistoryIdentityLeavesProxmoxDiskAlertReferencesUnbound` (live event and backfilled legacy row stay under the reference and out of the disk's history, while `ResolveReferenceID` still resolves the reference for mutes). It fails against the reverted binding implementation.	2026-10-07 02:58:35 +01:00
pulse-triage[bot]	809629954d	Merge branch 'main' into fix/docker-alert-unique-refs	2026-10-07 02:16:52 +01:00
pulse-triage[bot]	d4ab4a2324	Merge branch 'main' into fix/pve-disk-alert-history	2026-10-07 02:16:49 +01:00
rcourtman	4fd6b483cb	Use a SMART warning, not heat, in the cluster member disk test ... TestContract_ConnectionSystemMembersIgnoreOwnedPhysicalDisks proves a disk's health never folds into its cluster node's liveness. Its warning disk came from heat: an NVMe at 63C, which the flat 60C disk risk rule flagged. Disk risk no longer judges temperature, and 63C is under the NVMe alert trigger, so that disk is now online and the precondition failed. Reallocated sectors give the disk the same warning status through real SMART evidence, which keeps the test on its subject.	2026-10-07 01:42:54 +01:00
rcourtman	396bb5d517	Keep a disk heat finding open without a current reading ... Patrol verifies a disk-high finding as recovered once the disk's temperature cools to its alert clear value. A disk in standby, or one whose host agent stopped reporting, has no current reading, and a missing reading counted as cooled. A finding raised for a SATA disk at 56C would close as soon as the disk spun down, while its temperature alert stayed open. When a disk has no current reading and its last known reading is still at or above the clear value, verification now reports the recovery as unknown instead of proven. A disk last seen cool still recovers.	2026-10-07 01:19:36 +01:00
rcourtman	35bf55af23	Colour report disk temperatures by the alert thresholds ... The PDF performance report's disk table coloured a temperature amber from 50C and red from 60C for every disk type, the last fixed disk heat line after disk heat moved to the alert disk temperature policy. An NVMe at 63C, well under its 70C alert trigger, printed red. A user's raised trigger changed nothing. Each reported disk now carries the tenant's alert thresholds for its type (the clear value and the trigger, from alerts.Manager.DiskTemperatureThreshold). The PDF colours the reading amber from the clear value and red from the trigger, as the Physical Disks Temp column does, and leaves it plain when disk temperature alerting is off.	2026-10-07 01:19:36 +01:00
rcourtman	3fd5411ecc	Judge disk heat by the alert disk temperature thresholds ... Pulse decided whether a physical disk was too hot in three places with three thresholds. Disk risk added temperature_high at a flat 60/70C for every disk type and drove the disk status and the Physical Disks Health verdict. Patrol flagged anything over 55C and the AI chat context listed disks from 50C. Alerts and the Temp column used the per-type thresholds users tune in Alerts (NVMe 70, SAS 65, SATA 55 by default). An NVMe at 63C read Running Hot beside a green Temp cell. A SATA disk at 56C read Healthy beside a red one. A user who raised the NVMe trigger to 75 still got a hot verdict at 70. The alert disk temperature policy now decides disk heat. alerts.Manager.DiskTemperatureThreshold exposes the per-type resolution CheckHost already applies. - Disk risk no longer judges temperature. Every registry that rebuilds risk lacks the alert configuration, so no rule there could honour it. A hot disk keeps an online status. - The Health verdict reads Running Hot when a current reading reaches the disk type's trigger. It uses the thresholds object that colours the Temp cell, so the two cannot disagree. The amber band below the trigger stays a cell colour, not a verdict. - The TrueNAS storage table lost the heat it used to get from risk. It now judges heat the same way: an Attention row with the reason, and a warning dot for phones. - Patrol triage, its disk issue filter, finding recovery and the chat context's attention list judge the collected reading against the same trigger, including scoped runs and a provider set before Patrol starts. - A switched-off agent Disk Temp default resolves to no thresholds, which the displays that read them now treat as off. Per-host Disk Temp overrides do not reach these surfaces yet. TrueNAS disk alerts, the TrueNAS disk drawer and PDF reports keep their own fixed temperatures for now.	2026-10-07 01:19:31 +01:00
rcourtman	1b6a9b45d7	Keep retained disk temperatures out of AI, Patrol and reports (#2625) ... Normalization keeps a disk's last-known temperature when the current observation did not collect one (a disk in standby, a host agent past its reporting lease) and marks it non-available. These readers still took `PhysicalDisk.Temperature` at face value; each now goes through `diskinventory.TemperatureCollected`: - **AI chat context**: "Physical Disks Needing Attention" no longer flags a disk for heat from a retained reading. A disk listed for another reason shows it as `last known 52C (host agent stopped reporting)`. - **Patrol**: the disk health-issue gate, triage flags and `disk-high` finding verification judge only a collected reading. Both seed tables label a retained one as last known. - **AI tools**: `pulse_query` / `pulse_metrics` physical disks and `pulse_storage disk_health` put a retained value in `last_known_temperature` + `last_known_temperature_reason` instead of `temperature`. - **Reports**: the performance report's disk table and the reporting runtime snapshot leave a retained value out (no reading). - **Disk temperature charts**: a short series is padded out to now only with a collected reading. Otherwise stored samples are returned as they are. - `monitor.go`'s private copy of the predicate is replaced by the shared helper. Each new test reproduces against the pre-change code: retained `unavailable` temperatures (standby / silent agent) beside a collected one.	2026-10-07 01:18:08 +01:00
rcourtman	684c0dc598	Show a host whose agent stopped reporting as offline (#2634) ... A host agent that stops reporting is never shown as offline in the Machines table. Its row renders its last report as if it were current. ## Why The monitor marks a host agent offline once its reporting lease expires, and raises "Host is offline". The registry's stale pass marks the agent sighting stale after 60 seconds, and `aggregateStatus` ranked stale above offline. So the machine reached every consumer as `warning`. On the Machines table that meant an amber dot. CPU, memory, disk, network, disk I/O, uptime and temperature all rendered as live, with threshold colours. Meanwhile the row's own drawer led with the critical offline alert and read "Observed state: Degraded". Docker hosts and Kubernetes clusters hold the same kind of lease, and theirs is shorter than the 120-second stale threshold. A silent Docker host therefore read offline at first, then flipped to warning once its sighting went stale. ## What changes - **Backend.** Each sighting records when a push reporter delivered its machine, Docker host or cluster as offline. The reporters are the host agent, Docker and Kubernetes. Status aggregation counts that sighting as offline whatever its age. - The sighting itself still reads `stale`, so health keeps its `telemetry_stale` reason and monitored-system reasons are unchanged. - A live source still carries a merged row. A Proxmox node whose agent died stays online through the PVE poll. - Pull sources, and the guests, containers, pods and disks that push collectors report about, keep the stale-to-warning rule. - A JSON copy re-derives the marker from its stored status. - A manual link joining two resources from one source keeps the fresher sighting. - **Frontend.** Offline rows already blank CPU, memory, disk, network and disk I/O through the danger gate. - The Machines Uptime and Temp cells and the Docker hosts Temp cell had no gate, and now use it. - The drawer drops an offline resource's uptime and a silent agent's uptime. - The drawer marks a silent agent's non-disk Thermals rows "(last known)" with the reason as title, as retained disk rows already are. ## Verification - Go tests cover the agent, Docker, Kubernetes, mixed-source, Proxmox-node, JSON round-trip and manual-link cases, plus the unchanged late-agent and stale-PVE boundaries. Each fix's test fails without it. The `unifiedresources` package passes. - The `api`, `monitoring`, `ai`, `alerts` and `agentcontext` suites ran on pulse-dev. One `internal/api` test failed there, `TestServerInfoEndpointReportsDevelopment`, and it fails identically on the pre-fix build in that scratch tree. - Frontend tests cover the Machines offline row, the Docker offline Temp cell, the drawer thermals marking and the drawer uptime rows. The adjacent suites pass. - Live check on a private mock stack, holding one standalone Linux agent and one Docker host past their leases, at 1440 and 390 wide. Before: amber row with live-looking readings. After: red row with every reading blanked, the drawer reading "Offline", and thermals marked last known. - The browser receipt is included. Bundle sizes are within budget: WorkloadsSurface is unchanged, Standalone +0.04 kB, Docker +0.01 kB. Builds on rcourtman/Pulse#2630 and updates its contract sentence, which described the lapsed agent as reaching the table as `warning`. Known follow-up: the drawer's Disks card still threshold-colours a silent agent's retained filesystem usage.	2026-10-07 01:03:44 +01:00
rcourtman	c06af69fcb	Show a host whose agent stopped reporting as offline ... A host agent past its reporting lease is offline: the monitor marks it so and raises "Host is offline". The registry's stale pass ranked the agent's stale sighting above offline, though, so the machine reached every consumer as a warning. The Machines table showed it amber and rendered its last CPU, memory, disk and temperature as current readings, while its own drawer led with the critical offline alert. A Docker host was offline only until its sighting crossed the 120-second stale threshold, then flipped to warning, and Kubernetes clusters likewise. Each sighting now records when a push reporter (host agent, Docker, Kubernetes) delivered its machine, Docker host or cluster as offline, and status aggregation counts that as offline whatever the sighting's age. The sighting itself still reads stale, and a live source still carries a merged row, so a Proxmox node with a dead agent stays online. A copy that went through JSON re-derives the marker from its stored status, and a manual link joining two resources from one source keeps the fresher sighting, so a retired agent cannot pull a live one down. Offline rows then take the existing danger gate. The Machines Uptime and Temp cells and the Docker hosts Temp cell had no gate and now follow it. The drawer drops a silent agent's uptime and marks its non-disk Thermals rows "(last known)", as retained disk rows already are.	2026-10-07 00:14:49 +01:00
rcourtman	344f1f8531	Raise disk health alerts for mock physical disks ... checkMockAlerts evaluated guests, agents, nodes, storage, PBS, PMG and Docker against the mock estate but never physical disks. The fixture keeps a FAILED cohort and worn SSDs stable on purpose, so mock and demo stacks showed failing and "Replace Now" disks with no disk-health or disk-wearout alert, and the disk alert lifecycle could not be exercised without real hardware. The pass applies the physical disk poller's boundary: only disks on online nodes, matched by instance and node name, with the linked agent's --disk-exclude patterns forcing a matched device healthy. The per-disk evaluation moves into checkPhysicalDiskAlerts so the poller and the mock pass share one rule. The poller's agent-link settle deferral is not applied, because fixture links and exclusions are complete from the first pass. Like the other loops in checkMockAlerts, the disk loop runs after the pass's single mode check, so the existing race with leaving mock mode now covers disk alerts too. That race also involves other fixture evaluation paths and is left for a monitor-wide fix. On the public demo's estate (8 nodes) this adds one warning-level disk-wearout alert and no disk-health alert: that estate has no FAILED Proxmox disk.	2026-10-06 23:43:53 +01:00
rcourtman	38954b0b71	Give Docker containers and services without IDs their own alert refs ... A container reported without an ID alerted under its Docker host's own reference (docker:<host>). Its alerts showed as host alerts, joined the host's resource history, were never cleared when the container left the report, and made the host's Patrol scope alias ambiguous. A Swarm service without an ID put its normalized name in the service-ID slot, so a name equal to another service's ID shared that service's alerts and history, and the history resolver had to treat the whole cluster as a conflict. Both now alert under their name, marked in the ID position (docker:<host>/name:<name>, docker:<host>/service/name:<name>). Docker and Swarm IDs never contain a colon, so these cannot equal a host or ID reference, and one with neither an ID nor a name raises no alert. The policy switches, cleanup, Patrol aliases and the history resolver use the same builders and parse the same shapes, so the resolver no longer needs the cluster-wide conflict. A pre-upgrade container alert left on the host's reference is cleared on the host's next report, and a pending image update recorded there keeps its age. Canonical state IDs were split at the first "::", but Docker hosts that collide are disambiguated as "<base>::<suffix>". Every container alert of such a host was rewritten to the reference of the host named <base>, so it joined that host's history and skipped its own host's cleanup. State IDs now split before the alert's recorded spec ID, or after its own resource ID, so alerts saved that way also recover on restore.	2026-10-06 23:27:50 +01:00
rcourtman	18f0fbd826	Keep PVE disk alert references out of resource history ... PVE disk health and wearout alerts reference the disk by instance, node and device path. Operator mutes resolve that reference to the disk at the path (#2112), but resource history must not bind it: a replacement disk in the same slot, or a reboot that reorders devices, takes over the path, and disk alerts are evaluated before their poll's disks reach the registry. A history binding belongs to the reference string, so it would carry a replaced disk's journaled alerts into its successor's history and make the Alerts history Resource action show the successor's incidents for its predecessor's alert. A binding guarded by each alert's recorded disk serial was built and measured on a mock estate. It bound correctly while the disk stayed at its path, but rejected events still joined whichever disk the reference was bound to, and the serial check cannot be atomic with journal writes. Record the decision in the unified-resources contract and pin it with a test, so binding these alerts later goes through ownership per row rather than a per-reference binding.	2026-10-06 23:20:20 +01:00
rcourtman	fb7dd955ed	Keep retained disk temperatures out of AI, Patrol and reports ... Normalization keeps a disk's last-known temperature when the current observation did not collect one (a disk in standby, a host agent past its reporting lease) and marks it non-available. Several readers still took PhysicalDisk.Temperature at face value. The AI chat context listed a standby disk as needing attention for heat. Patrol raised health issues and triage flags from the retained value. The AI query and disk-health tools handed it to the model as the disk's temperature. Performance reports and the reporting runtime snapshot tabulated it as measured, and the disk temperature charts padded a series out to now with it. Each of them now goes through diskinventory.TemperatureCollected. AI surfaces keep a retained value as context, labelled last known with its reason. Reports and the charts leave it out. monitor.go's private copy of the predicate is replaced by the shared helper.	2026-10-06 23:09:52 +01:00
rcourtman	d9ffb39ce8	Keep a merged disk's temperature under its own row's state (#2618) ... Stacked on #2609; until it merges this branch also carries its commit, and the change here is the last commit. ## Why The registry merges a physical disk's rows (Proxmox, host agent SMART, Unraid inventory), choosing the temperature by source preference and merging collection states separately. When two rows carry different temperatures, the shown value could sit under the other row's `available` state. A real case: a node whose disk temperature Proxmox collects now (`proxmox_node_smart`, 40) and whose linked host agent has gone silent (retained 72, `unavailable`). In either ingest order the merged disk showed **72 labelled available**, so risk, history and the UI treated the frozen reading as current. ## What - `pairPhysicalDiskTemperatureState` runs at the end of each merge step. When the merged state says collected but the row the shown value came from says it was not, the merged state becomes that row's own. - It never changes a value and never grants availability, so it cannot revive a withdrawn reading or override a resumed agent's fresh one. - Rows with the same value keep the merged state, so the agent's own withdrawal still supersedes the Proxmox copy (#2593). - With that guarantee the Unraid-native row no longer needs to withhold `available` (#2609 restricted it), so it carries the same derived state as a SMART row borrowing its reading. ## Known limit This is exact for the two rows of one merge step. The merged disk keeps one state per field, so with three or more rows for one disk, a withdrawal that an earlier step replaced is invisible to a later step. Sol found two such counterexamples, both needing three rows (one also needs coincidentally equal values, the other a legacy agent plus two Proxmox rows). Closing that needs each reading's own state tracked across merges, and that redesign is filed separately. Preferring another source's current value instead of the retained one was tried and dropped: it needs an ordering between same-source states that the rows do not record, and each version reopened a withdrawn reading somewhere. ## Tests - `TestPhysicalDiskMergePairsTemperatureWithItsCollectionState`, in both ingest orders: - silent agent against a collected Proxmox reading, and against no Proxmox reading - reporting agent - silent agent against a Proxmox copy of its earlier reading, with and without SMART attributes - resumed agent against a Proxmox copy of its withdrawal - standby agent against a Proxmox copy of its pre-sleep reading - `TestPhysicalDiskMergeNeverPresentsAWithdrawnTemperatureAsCollected`: three distinct-valued withdrawn or stale rows in all six orders. - `TestUnraidInventoryRowDoesNotMaskTheSMARTRowState`: the Unraid inventory's `available` no longer lends itself to a SMART row's uncollected reading. - Without the rule, the peer-reported case, the Unraid masking case and one three-way order fail.	2026-10-06 23:05:53 +01:00
rcourtman	dcb69cfb02	Record Docker host, service and sub-resource alert history (#2620) ... Follows #2608, which made `MonitorAdapter.RecordChange` resolve Proxmox and agent alert references to canonical resources. Three kinds of alert reference still journaled under strings no resource answers to: - Docker host alerts (`docker:<host ID>`) - Swarm service alerts (`docker:<host ID>/service/<ID>`) - sub-resource alerts such as ZFS pool and device health (`<storage ID>/zfs-pool:<pool>[/device:<dev>]`) and host filesystem, SMART, RAID and sensor alerts (`agent:<host ID>/disk:…` etc.) Resource facets, timelines, resource intelligence and Assistant context read history by canonical ID, so these alerts never appeared there. ## What changes - **Docker host refs** resolve through the registry's Docker source ID. - **Swarm service refs** resolve through the host's Swarm cluster to the service with that exact ID, from any manager that reports it. A service with no ID is referenced by its name in the same slot, so while a cluster has such a service, every service ref there is a conflict and binds nothing. - **Never bind:** container names, shortened container IDs and hostless `docker-service:<name>` refs never bind, and are no longer retried against later inventory. - **Sub-resource refs** bind to the owner whose durable ID they carry: - ZFS pool and device alerts go to the storage. - The Unraid array alert goes to its array storage. - Filesystem, SMART, disk temperature, RAID and custom-sensor alerts go to the host. - The owner must resolve to exactly one resource of the expected type. An ambiguous or incompatible owner is a conflict. - Mount and kernel device labels can name a different disk after a reboot, so these alerts never bind a physical disk. - **Retries:** a reference that hits a conflict at write time now enters the backfill retry set, the same as one found at startup. ## Known residuals (producer-side, follow-up proposed) - A container with no ID alerts under its host's reference (`DockerResourceID(host, "")`), so its events join that host's history (never another container's). - An ID-less service whose name equals another service's ID shares that service's reference. Rows it journaled can join the other service's history once a later generation or event binds the reference. Swarm always reports service IDs, so this needs a malformed report. Both are documented in the contract. Separating them needs a producer change in `internal/alerts`. ## Evidence Private `PULSE_MOCK_MODE` stack, fresh data dir. The binary at #2608's tip ran first, then this change on the same dir. Alert rows were classified by source reference (from `alert_identifier`) and checked against each owner's `/api/resources/{id}/timeline`: | ref shape | rows readable via owner timeline, before | after | |---|---|---| | Docker host | 0 / 1 | 3 / 3 | | Swarm service | 0 / 5 | 19 / 19 | | ZFS pool | 0 / 9 | 26 / 26 | | ZFS device | 0 / 9 | 27 / 27 | | agent filesystem | 0 / 1 | 3 / 3 | | Docker container (mock short IDs) | 0 / 11 | 0 / 30, no container alias written | Tests: - `TestHistoryIdentityMonitorAdapterResolvesDockerHostAndServiceReferences` covers multi-manager services, a Docker host merged into its agent machine, legacy-row backfill, recovery after removal, ID-less service conflicts, and names and short IDs staying unbound. - `TestHistoryIdentityMonitorAdapterResolvesSubResourceReferences` covers every owner shape, wrong-type and ambiguous owners, retained bindings, write-time conflict retry, and physical disks staying unbound. - `TestOwnerAlertTimelinesUseCanonicalHistoryIdentity` drives the real alert manager for Docker host offline, Swarm service gap, ZFS pool/device and host filesystem alerts. It fails on #2608's tip. UI reach: the Docker, storage and platform host-row drawers don't request per-resource history. This fix reaches the facets/timeline/intelligence APIs and Assistant resource context, not a drawer.	2026-10-06 22:57:41 +01:00
rcourtman	0fb5ccb62b	Keep node temperatures off same-named nodes at other sites (#2617) ... Two Proxmox connections can each have a node with the same name, which is common for MSPs running one `px1` per site. With one host agent linked to site A's `px1`, `getHostAgentTemperatureByID("siteB-px1", "px1")` returned site A's temperature. The hostname fallback took the first agent with that hostname, even one linked to another node, and even when the name was ambiguous enough that the agent linker had refused to link it. The cluster sensor cache leaked the same way: SSH readings an agent collects for its cluster siblings were keyed by bare node name. Site B's `px1` then showed, alerted on and recorded history for another machine's sensors. ## Change - The poller hands the lookup the polled node (`getHostAgentTemperatureForNode`, `carriedTemperatureOutlivesAgentLease`), and the lookup works on that node's **slot**: its own source ID, plus, when the read state does not hold that ID, the one same-named node proven to be the same machine. - The state folds the views of one machine reached through two connections (a cluster added twice, or a multi-homed host added by each address) into a single node kept under one view's ID, and an automatic agent link follows it. A narrowed fallback on its own would leave those views without a temperature permanently; that was reproduced with real `UpdateNodesForInstance` folds. - Same-machine proof comes from the state's own evidence: new `models.NodeObservationsSameMachine` and `models.HostAgentBridgesNodeViews` (`internal/models/node_machine_identity.go`). `UpdateNodesForInstance`'s agent-corroboration check now shares its rule with them. Two or more candidates are ambiguous and none is taken, as the state declines an ambiguous alias. - An agent linked to the slot is the node's agent. The hostname fallback only takes an agent with no node, VM or container link, and only when no node outside the slot and no second unlinked agent has the name. - Cluster sensor cache entries record their reporting agent. A reading serves a node only when that agent's linked node shares one of the slot's connections, or belongs to a connection whose configured view of the node has a matching TLS fingerprint (the other connection of a cluster added twice). `LinkedNodeID` holds the Proxmox source node ID for every link source (automatic `findLinkedProxmoxEntityWithHints` → `NodeView.SourceID()`, manual `LinkHostAgentToNode` validating against `models.Node.ID`, reconciliation, continuity restore), so the link match needs no namespace normalization. A manual link stays pinned to the ID the operator chose (the state never transfers intent to a replacement provider identity), so a node held under the other view's ID has no linked agent. The physical-disk SMART merge already matches by `node.LinkedAgentID` within one instance and is unchanged. ## Tests - `TestNodeTemperatureStaysWithinSameNamedNodesSite` (registered monitoring proof, `monitor_additional_test.go`), through `collectNodeTemperatureData` against registry-backed and state-folded read states: agent linked to the other site's node; ambiguous unlinked agents; unambiguous unlinked agent (still served); cluster added through two connections (alternating real state folds); multi-homed host via two standalone connections; ambiguous address alias; a namesake agent with a contradicting fingerprint; sibling readings across both connections of one cluster; sibling reading from the other site. The leaking cases fail on main, and each folded-view guard was checked with an overlay negative control. - `TestApplyHostReportScopesClusterSensorsToReportersConnection` (`monitor_host_agents_test.go`): real `ApplyHostReport` ingest of cluster sensors. - `TestNodeObservationsSameMachineMatchesStateFold` and `TestHostAgentBridgesNodeViewsMatchesStateFold` pin the predicates against `UpdateNodesForInstance`'s actual fold. - Full `internal/monitoring`, `internal/models` and `internal/unifiedresources` pass on pulse-dev.	2026-10-06 22:40:51 +01:00
rcourtman	8259a74c9d	Show the live reading of threshold alerts held below trigger (#2567) ... A threshold alert stays open after its reading drops below the trigger. It holds until the reading reaches the clear level, then must stay there for the recovery delay. While it held, the evaluator never touched the alert, so every surface kept showing the last breach. On minipc the drawer said "Node temperature at 80.0°C" next to a green 72°C in the node table, with nothing explaining why the warning was still there. #2068 reports the same thing for a disk alert. **What changes** - Open threshold alerts carry a volatile `metricStatus`: the value Pulse is comparing now (the average for rolling-average rules, with the latest sample beside it), the phase (`breaching`, `latched` or `recovering`), the trigger and clear levels, and recovery progress from the reducer's monotonic run. `value`, `message` and `lastSeen` keep meaning the last breach, and the hold branch still never notifies. - The status is stripped from the active-state checkpoint, the recovery mirror and the degraded recovery projection, so holding adds no durable writes. Resolved records carry none. - Stale-alert cleanup and the retention TTLs use the latest observation. Previously an alert held below its trigger for more than a day was auto-resolved as "resource no longer being monitored". - Node, guest and Mail Gateway drawer rows lead with the reading now and say what clears the alert, with the last breach on hover. The Proxmox TEMP cell keeps its own open temperature alert's tone and explains it on hover. Storage rows stop saying "Over N%" once usage is back under the limit, and the Assistant handoff no longer labels the breach value as current. **Verified** - Unit tests drive a node temperature alert through 85 → 78 → 72 → 75 → 77 → 72 → resolved, and fail against the old evaluator. - Private mock stack with pve2 stepped 88 → 78 → 72°C at desktop and phone widths; see the browser receipt in this PR. Refs #2068	2026-10-06 22:23:05 +01:00
rcourtman	38a5334b90	Judge disk risk only on a temperature collected now (#2611) ... Since #2593, a disk whose host agent stops reporting keeps its last SMART temperature as a last-known value marked `unavailable`, and a disk in standby keeps its pre-sleep reading the same way. The shared disk risk assessment still fed that value into its temperature rule (>=70C critical, >=60C warning). So a silent agent's frozen 72°C made the Physical Disks row read **Replace Now** ("Disk temperature is 72C") with a `warning` resource status, while #2603 makes the temperature cell beside it say the reading is not current. ## Change - `storagehealth.CollectedTemperature(temperature, collection)` passes a temperature through only when its `collection.temperature` state is `available`, or when the source predates collection state (nil or empty state). Otherwise it returns 0 before risk judges it. - Three places use it: - `AssessPhysicalDisk`, for Proxmox inventory disks, through the unified adapter; - `AssessHostSMARTDisk(WithThresholds)`, for a host agent's own SMART rows; - `physicalDiskAssessmentFromMeta`, the registry's recompute after it merges sources. - The retained value stays on the resource as last-known history. Health, wearout and SMART counters are unchanged: they are durable evidence and keep judging as before. Alerts are unaffected: - SMART disk-health alerts already discard `temperature_high` (`internal/alerts/host.go`). - Disk temperature alerts evaluate the agent's own fresh report. - No fresh collector emits a positive temperature with a non-available state. ## Evidence On a private mock stack, with a local generator overlay stamping real collector states: - Before: the silent-agent disk (72°C retained) read **Replace Now** and the API reported `status: warning`, `risk: critical`. - After: it reads **Healthy**, the API reports `online` with no risk, and its temperature shows as last known. - A genuinely hot disk (72°C `available`) still reads Replace Now. Whether "Replace Now" is the right word for a heat-only verdict is a separate question. ## Tests - `internal/storagehealth/risk_test.go`: `TestCollectedTemperatureKeepsOnlyCurrentReadings` and `TestDiskAssessmentsIgnoreRetainedTemperature`. - `internal/unifiedresources/registry_test.go`: `TestPhysicalDiskRiskIgnoresRetainedTemperature`. It covers Proxmox disks in every collection state through the monitor adapter, plus a real agent+Proxmox merge of the same disk, collected and retained. - Mutation-checked: removing any of the three gates fails a case, and the merged case fails without the registry-recompute gate. - Full `internal/storagehealth`, `internal/unifiedresources`, `internal/api/resourceapi` and `internal/truenas` pass, plus the SMART/disk/host subsets of `internal/alerts` and `internal/monitoring`. ## Related - The Unraid inventory temperature path in `adapters.go` (`assessUnraidPhysicalDisk`) gets the same gate in the Unraid change that stamps its collection state. - A registry merge can still pair a retained agent temperature with a Proxmox row's `available` state (`registry.go`, Proxmox-over-agent carry). That is being fixed separately.	2026-10-06 22:10:42 +01:00