mirror of
https://github.com/rcourtman/Pulse.git
synced 2026-10-23 20:25:56 +00:00
0f6d1e41801f8e7dfdf843592e4a05b09581d7e2
12981 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 rcourtman
|
0f6d1e4180 |
Stop a failing availability check keeping a node online (#2663)
Follows #2643, which made the stale pass read each source's own verdict but left availability checks counting by delivery. ## Problem Once every source with a verdict had gone quiet, any current availability sighting kept its target online. A node that `preserveOrExpireNodes` expired stayed online while its HTTP check failed, and a quiet failing check lifted that node's offline verdict to warning. One sighting stands for every check on a target and carries whichever check was projected last, so a failing check that had just run also vouched for another check's old pass. ## Change - **Registry verdict** (`aggregateStatus`, `availabilityChecksProveOnline`): each check on a monitored resource is judged by its own evidence window (`Evidence.ValidUntil`). An enabled, passing check with current evidence proves the resource answers and reads online once one of the resource's own sources has gone quiet and none that is current has a verdict. Every other check abstains, current or quiet. A failing probe never makes a guest count as stopped (workloads summary, Proxmox page counts). A check's own row keeps the ordinary rules. - **Ownership**: only a target's own quiet sources hand the stale pass its status. Projecting, re-running or retargeting a check re-applies the stale pass's rule to targets it owns, because the resources API replays checks after its stale pass. A manual link judges the merged resource by the checks it carries. - **Poller evidence** (`internal/monitoring`): - A local location beside other locations ages out like a silent remote path ("no recent local check"). - An available check's evidence lasts only as long as its newest reachable path. - After an entitlement lapse, an adopted remote result keeps the earlier of its agent check time and the server receipt time. - A multi-location aggregate is never adopted as a local observation; the target reads as never checked until the local poller runs. Contracts: `unified-resources.md` ("Status follows each source's own verdict") replaces the residual paragraph, and `monitoring.md` covers the multi-location rules. ## Verification - New tests in `registry_test.go`, `registry_merge_policy_test.go` and `availability_probe_agent_test.go`. Each fails against the base runtime via `go test -overlay`, including both projection orders. - Full `internal/unifiedresources` and `internal/monitoring` suites pass, plus the availability, connection, workload and resource-state tests in `internal/api`, `resourceapi` and `chartapi`. |
||
|
rcourtman
|
bdf0f76b7b |
Show retained disk temperatures as last known in drawers and tables (#2659)
Normalization keeps a disk's last temperature when the current observation did not collect one, such as a disk in standby or a host agent past its reporting lease. #2603 taught the Storage presenters to show that value as last known. Four surfaces still read the raw number as a current reading. Each now goes through the canonical `getPhysicalDiskTemperaturePresentation` / `isPhysicalDiskTemperatureCurrent` instead of adding a predicate. - **TrueNAS disk drawer** (`resourceDetailDrawerTrueNASModel.ts`): the Temperature row took the 55°C warning tone from a retained reading, and the summary listed it as a reading. It now reads "61°C (last known)" with the reason as title and no tone, and the summary leaves it out. A current reading keeps the existing tone. Replacing the fixed 55 with the alert policy is a separate follow-up. - **Drawer History fallback** (`resourceDetailDrawerMetricsHistoryModel.ts`): with no stored points, a retained reading was labelled "current". It now goes through the deferred-metric slot, so the legend says "last known" and the reason shows above the chart. A disk's value is read from `physicalDisk` beside its collection state. Websocket rows carry no top-level `temperature` for a disk, so on the TrueNAS page the fallback had shown "-" even for a current disk. - **TrueNAS storage table** (`TrueNASStorageTopologyTable.tsx`): the Temp cell showed a retained reading plainly, and hottest-first sorted it to the top of its group. It now renders muted with its reason, and the sort ranks it as no reading. - **Proxmox outdated sensor setup notice** (`sensorSetup.ts`): a retained reading counted as a temperature arriving, so the notice could hide for a SATA disk Pulse cannot read now. Verification: a regression test per surface fails on main and passes here. A private mock stack with a local-only overlay stamped one TrueNAS disk and one pve3 SATA disk with an unavailable temperature state ("host agent stopped reporting") and marked pve3's sensor payload legacy. Playwright at 1440×900 and 390×844 compared the change against the same session with main's runtime files. The receipt is `frontend-modern/browser-verification/294409f678589ef2.json`. |
||
|
rcourtman
|
bab38a25b9 |
Drop the summary chart caches nothing reads (#2662)
The Workloads table history hook was the last caller of `infrastructureSummaryCache.ts` and `workloadsSummaryCache.ts`, and nothing has read either cache since the InfrastructureSummary card and the standalone Workloads summary were deleted on 2026-05-26. The hook runs both fetchers inside `createNonSuspendingQuery`, which always passes an abort signal, so their in-flight dedupe was skipped too. Each 30-second Trends poll still serialized the full response on the main thread and wrote it to localStorage or, above the 900k-character cap, deleted the key. Measured on embedded builds against the 50-node, 929-guest mock estate at the 1h range: | | Before | After | |---|---|---| | JSON serialized per poll | 9,629,902 + 924,249 chars (5-31 ms each) | none over 200k | | Summary-cache storage calls | `removeItem` on both keys every poll | none | | `/api/charts/workloads` + `/api/charts/infrastructure` | one each on load and per poll | same | | Rows drawing history (1440 / 390 wide) | 12 nodes + 140 guests / 6 + 36 | same | The hook now calls `ChartsAPI.getWorkloadCharts` and `ChartsAPI.getInfrastructureSummaryCharts` directly with the same range, signal, `maxPoints`, node scope and metric list. The node, agent and Docker-host series fold moves into `workloadMetricHistoryModel.ts` with its merge rule unchanged. The query layer already retains reads across remounts for five minutes and clears them on org switch. The hook replaces the deleted cache under the `workloads-hot-path` policy. The unified-resources, performance-and-scalability and frontend-primitives contracts still described InfrastructureSummary, its model and state hook, the shared summary table, selector and details drawer, WorkloadsSummary, throughputPresentation, resourceBadges and the two caches as live owners. Each clause now names the live owner, becomes a conditional rule, or is removed where nothing it governs remains. Stale prose for the other summary primitives deleted in May (SummaryMetricCard, InteractiveSparkline, DensityMap, StorageSummary) and their test-only helpers is left to a separate change. Checks: affected vitest suites (10,053 tests; the only failures are the Mac-only `agentInstallCommand` suite), contract, registry and status audits, and the browser verification receipt. |
||
|
rcourtman
|
156d2e274d |
Keep same-serial disks on different machines apart (#2660)
## Problem
`ingestRecord` refuses a physical-disk match under another parent
(`findMatch` skips scope-incompatible candidates), then `chooseNewID`
mints the same hardware-keyed canonical ID (`machine:<serial>`) and the
ingest merges into whatever disk already holds it. Disks that report one
usable serial on several machines collapse into a single resource
parented to whichever machine ingested last, and every other machine
loses the disk from inventory. Real triggers: dual-ported SAS shelves
seen by two nodes, cloned VMs with an explicit identical serial,
fixed-serial USB bridges, TrueNAS systems. Repro: `IngestSnapshot` with
three PVE nodes in one instance, each with a `/dev/sda` of the same
serial, gave one disk.
## Fix
- A new disk resolves its parent to a machine: the nearest host
ancestor, so a TrueNAS pool or Unraid array counts as its system or
host.
- Two disks with one identity stay apart only on evidence: one reporter
placing the serial on two machines (cloned VMs keep both serial and
hostname), or machines whose hostnames share nothing. Reporters on one
box still join across grouping parents and unmerged host resources, so
the TrueNAS API and the agent on that box stay one disk. An unknown
machine never splits a disk, so a PVE row whose node row is missing
still joins its host's agent disk.
- Once a serial spans machines, every copy takes the serial keyed to its
own machine, and the holder of the unscoped ID is re-keyed with its
source mappings, retired-ID claims and relationship endpoints. IDs don't
depend on machine ingest order, and the unscoped ID never passes between
disks that coexist. A disk only one machine reports keeps its current
ID.
- Operator splits ("Split merged resource" exclusions) now hold on this
path; the old collision merge ignored them. A split recorded against any
ID the disk can hold applies, so it survives a re-key.
- The registry indexes disks by hardware key (`physicalDisksByHardware`)
because `IdentityMatcher` keeps one resource per machine ID.
## ID stability decision
Machine-scoped IDs only for identities that actually span machines,
rather than a pre-pass. A pre-pass needs final parent scopes before
ingest, but host/node merging happens during ingest. Giving the
first-ingested machine the unscoped ID would hand one machine's ID, and
its operator state, to another machine when an earlier-sorting machine
appears. With the chosen design a disk's ID changes when a same-serial
disk first appears on another machine. That orphans ID-keyed state
rather than misapplying it.
## Residuals (documented in unified-resources.md)
- Like any serial-keyed ID, the unscoped ID still passes to a
same-serial disk that appears once the previous holder is gone.
- Alert-history rows owned under the unscoped ID don't follow a re-key.
The PVE disk alert owner (#2646) already leaves shared-serial rows
unbound unless a WWN tells them apart.
- Agent SMART and Unraid disk source IDs are the bare serial.
Same-serial agent disks on different hosts share one `SourceAgent`
mapping, and only the last ingested keeps the agent source target. This
is tracked as a separate task, because the fix re-keys persisted
mappings.
- Same-serial disks share one metrics history key.
- Across reporters, hostnames that normalize alike still join.
## Verification
- New tests in `registry_merge_policy_test.go`: per-machine disks for
PVE-only and linked-agent clusters, including reversed ingest order;
rehydration with a machine joining; agent + TrueNAS on one box versus a
cloned TrueNAS with the same hostname; a disk moving into a pool; a
parentless PVE row on a shared serial; exclusions honoured, surviving a
re-key, with per-machine fallback; serialized-JSON rehydration;
retired-ID claims carried across a re-key.
- Every new test fails against the unfixed registry or with its fix
removed (`-overlay` negative controls).
- `go test ./internal/unifiedresources` passes, as do the
disk/SMART/Unraid/TrueNAS tests in `internal/monitoring`, `internal/api`
and `internal/truenas`.
- Five rounds of independent review; the last round found no P1/P2
issues.
|
||
|
rcourtman
|
49d4100385 |
Keep watchdog, delivery log and Patrol ages moving (#2598)
Follow-up to #2578: the remaining Alerts and Patrol panels whose relative ages froze while the panel stayed open over unchanged data. **External watchdog (Alerts › Destinations).** The status was read once at mount, so Last success and Monitor progress kept their read-time age through a real stall, which is the case the panel exists to show. It now re-reads `GET /api/alerts/deadman/status` every 30 seconds in the background (the Refresh button is untouched; a failed re-read keeps the last status and marks it unavailable), and both ages read the shared clock. **Delivery log.** Each row's hover age reads the clock. A logged attempt's time never changes, so its age is right over a log that is only re-read on demand. **Patrol.** The paused banner kept "Blocked 2m ago" because each status re-read returns the same `blocked_at`. The selected-run banner kept its age because the selected run is held as picked. The attention detail is read once per selection, so its acknowledgement, transition and evidence times now read the clock. Its Last seen and flapping window describe a record that may have moved since, so they stay read-time (the new frontend-primitives paragraph states that exception). Queue rows, summary, run history and the digest already re-derive on their own 30/60-second re-reads, so they are unchanged. **Verification.** Fake-timer tests per surface (each fails on main and passes here). Live on a private mock stack with Playwright `page.clock` and the data stream held (one live socket kept by a data-free keep-alive per fake minute), at 1440 and 390 wide. After ten held minutes: watchdog 33s → 10 mins ago with 20 background re-reads; delivery tooltip 5 → 15 mins ago; "Blocked 2m → 12m ago"; "Patrol run 3m → 13m ago"; attention detail 3/5/7m → 13/15/17m ago. The same-session control on main's files stayed frozen everywhere. Found while auditing, filed separately: future times ("snoozed until", "Returns to active attention") render as "just now" because `formatRelativeTime` is past-only. **Review follow-ups (second-opinion review, three passes).** The status read has no timeout, so the first poll design could misbehave: - **Stuck Refresh:** a hung background read blocked Refresh; separately, a Refresh superseded by a hung read stayed disabled. - **Slow reads:** reads slower than the 30s poll were all discarded, failures included, so an outage could read "Heartbeat healthy". - **False recovery:** an older late success could clear a newer failure. Each read now gets a sequence number, and an outcome applies only when it is newer than the last read that settled. A hung Refresh is freed at the first poll once it has been pending 29s (one second of slack, because live runs showed ticks landing a few ms short of 30s). Unit tests cover each case and fail on the earlier versions. Each case was also checked live with held reads answered in and out of order, with an earlier-version control. Main is merged in, with a receipt for the combined `PatrolAttentionWorkbench.tsx` (#2591). |
||
|
rcourtman
|
4e88ae110c |
Lead open History rows with the live reading of held alerts (#2638)
Builds on #2637 (alert card live reading), so this branch carries its commits until that PR merges. ## Why Alerts > History is a log of occurrences, but its open row is the one occurrence still in progress: it shows an Active badge and a running duration. Beside them, the breach message "Node temperature at 95.0°C" reads as the reading now, while the node sits at 78°C under its 80°C trigger and the alert is held until it clears. With #2637 the Alerts card one tab away says "Temperature 78°C now, back under the 80°C alert level", so the two pages disagreed about the same alert (the confusion in #2068). ## What changes - An active History row for a threshold alert with a usable live status leads with the same presentation summary as the card (`getMetricAlertPresentation`, via a new `getAlertHistoryRowCopy` in `alertHistoryModel.ts`). - Desktop: the clear rule and the last breach move into the message cell's hover. Phone: they show as muted lines under the reading, since a phone has no hover. - The row's `description` stays the recorded message, so search ("95.0" still finds the held row) and the Assistant handoff read the record. - Closed rows, and open rows without a live status (non-threshold alerts, Pulse system alerts, the moments after a restart), are unchanged. - The reading rechecks on the shared relative-time clock, read only for open rows, so it turns stale ("Last reading: …, 12 mins ago") when evaluations stop without re-rendering closed rows on each tick. ## Verification - Private mock stack built from this tree with a local-only generator overlay pinning pve2's CPU temperature: walked 95 → 78 → 72 °C at 1440×900 and 390×844, light and dark; earlier pass also covered breaching, °F, and the alert resolving out of the list. - New tests: model copy and fallbacks (`alertHistoryModel.test.ts`), the phone card (`AlertHistoryMobileList.test.tsx`), and the desktop row following a live store update and turning stale on the clock (`AlertHistoryTableAlertRow.test.tsx`). - Receipt: `frontend-modern/browser-verification/9c0e53268f0d014f.json`. Refs #2068 |
||
|
rcourtman
|
a85bdcdc9a |
Stop a stale agent SMART row lending a replaced disk its identity (#2658)
## Problem PR #2648 stopped the poller's previous-evidence lookup from handing a replaced disk the old disk's serial. The linked host agent's SMART merge runs before that lookup and had the same flaw. Its last-resort match is the device path, and a path names a slot, not a disk. When a disk is swapped before the agent reports again, or after the agent went silent and kept its last rows, the agent's row for the old disk matched the replacement. It filled the old serial or, for SAS, promoted it over the Proxmox SAS address. The registry then keyed the replacement to the old disk's canonical resource and SMART history, and it also took the old disk's temperature and FAILED health. For SAS, the registry's same-path join (`resolveLinkedPhysicalDisk`) did the same at the canonical level even when monitoring kept the rows apart. Proxmox's disk inventory (`PVE::Diskmanage`, checked on a PVE 9 node) writes the literal `unknown` for a serial or WWN udev cannot read. The poller recorded it as a collected serial, so the collection state hid the gap, and the merge's `Serial == ""` fill never let the agent's real serial replace it. ## Change - `mergeHostAgentSMARTIntoDisks` refuses a device-path row whose identity contradicts the Proxmox disk's. The new `diskinventory.HardwareIdentityConflict` defines a contradiction: nothing agrees across serial and WWN, and a field both producers report like for like disagrees. - WWNs count for every disk. - Serials count only where Proxmox's serial is the drive's own, as smartctl's is: a disk typed `nvme`, or one presented with SCSI vendor `ATA` (libata, or a SAS HBA's SCSI-ATA translation) and not typed `usb`/`sas`. Elsewhere the Proxmox serial may be a SAS address or SCSI designator (#1595) or a USB bridge's own serial, so it only counts toward agreement. - Spelling differences are not disagreement: udev's underscores for whitespace, a T10 designator for an ATA drive (`ATA`, the model, then the serial), the agent's NAA field spelling, and udev's 64-bit prefix of an NAA 6 WWN. - The refusal runs after the unique-path check, so a second row on the path keeps the match ambiguous. - The merge's WWN step compares normalized identity (`HardwareIdentityMatch`), so the agent's `5-c50-a1b2c3d4` finds the Proxmox `0x5000c500a1b2c3d4` disk. - `resolveLinkedPhysicalDisk` refuses the SAS same-path fallback when the two WWNs name different disks. - The poller records Proxmox's `unknown` serial/WWN as empty, so the serial reads `missing` and the agent can fill it. Retained evidence carries a previous serial only when it is usable, so a pre-upgrade record cannot restore the placeholder. Identity keys already ignored the placeholder. When the agent now fills a disk's serial, that disk's Proxmox-poll SMART samples move from its WWN or path key to the serial key, where the agent's own samples already go. ## Tests - `TestHostAgentSMARTRowForSwappedOutDiskDoesNotLendItsIdentity` (sata and sas) drives the real poller and registry. Proxmox reports `unknown`, the agent fills the serial, the agent goes silent, and the disk is swapped. On main, the SATA replacement takes the old serial and resource; the SAS replacement keeps its own serial in state, but the registry joins it to the old resource with its temperature and FAILED health. - `TestHostAgentSMARTMergeRefusesPathMatchAcrossContradictingIdentity` has one row per producer rule: SAS/SATA/NVMe replacements refused; same-disk SAS without an agent WWN, a renamed SAS disk in the agent's WWN spelling, udev's serial spelling, a USB enclosure (typed `usb` and typed `ssd`), a T10 designator and an NAA 6 volume merged; and an ambiguous path. - `TestRegistrySASPathJoinRefusesContradictingWWN`, `TestProxmoxUnknownDiskIdentityIsRecordedAsUnreported` and `TestHardwareIdentityConflict` cover the rest. - Negative controls via `go test -overlay`: removing each rule fails only its own case. These cover the guard, the WWN normalization, the transport/vendor rule, compaction, the designator rule, the NAA 6 prefix, the registry guard, the ingest normalization and the usable-serial carry. Against main's `monitor.go` and `registry.go`, the end-to-end, merge and registry tests fail. - `TestMergeHostAgentSMARTIntoDisks_PreservesExistingProxmoxIdentity` merged a row with a different WWN by path alone, which the guard now rightly refuses. Its row is now the same disk in the agent's WWN spelling, with all assertions unchanged. - `internal/monitoring`, `internal/unifiedresources` and `pkg/diskinventory` pass in full. ## Not covered - A replacement with no comparable identity still matches its path: no usable serial or WWN at all, or a SAS/SCSI/USB disk with a WWN on one side only. - The registry's seeded source mapping, reused after a restart by the slot-shaped Proxmox source ID, can still hand a swapped disk the old canonical resource. That's a separate follow-up. |
||
|
rcourtman
|
43a80dafd9 |
Judge disk heat under the reporting host's Disk Temp override (#2654)
#2636 made the alert disk temperature policy the only judge of disk heat, but it resolved that policy by disk type alone. Disk temperature alerts (`CheckHost`) evaluate each SMART disk under the thresholds of the host agent that reports it. An explicit Disk Temp override on that host, or one inherited from its linked node or guest, replaces the per-type trigger, and a host whose alerts are switched off raises nothing. So a host with an 80C Disk Temp override still read **Running Hot** on Physical Disks at the 70C NVMe trigger. Patrol flagged it too, and a host with alerts switched off still read hot. The Alerts thresholds page promises the opposite: "Setting a Disk Temp override on a host above replaces these for all of that host's disks". ## What changes - **Backend policy:** `alerts.Manager.HostDiskTemperatureThreshold(host, diskType)` resolves the policy for a host the way `CheckHost` does, from the same helpers (`resolveHostThresholdsNoLock`, `hostThresholdOverrideNoLock`, `hostDiskTemperatureThresholdNoLock`). A disabled override gives no threshold. `DiskTemperatureThreshold(diskType)` is now its hostless case. - **Patrol and the AI chat context:** `ThresholdProvider.GetDiskTemperatureThreshold(host, diskType)`. Each unified physical disk finds its host by walking its parent to the agent machine, through the pool for Unraid array and cache disks. - **Physical Disks, the disk detail, the pool drawer's linked disks and the guest drawer** pass the reporting machine's override keys to `resolveDiskTemperatureDisplayThresholds`. Keys follow `CheckHost`'s order: agent ID, canonical resource ID, then the merged node or guest. The resolver returns null (uncoloured, never hot) for an override or agent default that switches the host's alerts off. The browser rebuilds `CheckHost`'s key chain from what it loads, so a few identity edge cases can still differ from alerts and Patrol (listed in `storage-recovery.md`): on the Storage page, a disk whose agent an operator manually linked into a VM; an override inherited through agent link hints the API does not publish (an unmerged linked node or guest), or stored under a linked node's own canonical ID; and guest keys in the guest drawer for an agent merged into a guest it is not automatically linked to. Overrides set on the agent's own row, or on the node or guest it is merged with, reach every surface. ## Verification - `TestHostDiskTemperatureThresholdMatchesCheckHostOverrides` checks each case against what `CheckHost` actually fires: no override, host override, host override without Disk Temp, an override under the canonical resource the agent ID resolves to, linked-node and linked-guest overrides, a host override shadowing a node one, a switched-off Disk Temp, and a disabled host, linked node or linked guest. - `TestPatrolJudgesAgentDisksByTheAgentDiskTemperatureOverride` uses a real registry with two agents, one overridden. A mutation that drops the owner lookup fails it. - Frontend tests cover the resolver, the owner walk and key order, DiskList with three hosts, the presentation map, pool-drawer linked disks and the guest drawer. - Private mock stack: three NVMe disks pinned at 72C. Before (main frontend): all three read Running Hot. After: - pve1, with an 80C override, reads Healthy (green). - pve2 still reads Running Hot. - pve3, with alerts off, is uncoloured. - Moving pve2's override onto its linked node ID turns it Healthy. - Checked at 1440 and 375 with no overflow. The receipt is in `frontend-modern/browser-verification/`. - A read-only second-opinion review ran twice. Round 1's findings (frontend key order, guest drawer canonical ID, disabled agent default, doc wording) are fixed in the first commit. Round 2's are the parity cases and the documented browser limits in the second commit. |
||
|
rcourtman
|
83ec32ee71 |
Keep fixture alerts from outliving a mock-mode switch (#2656)
Leaving mock mode cleared active alerts, but every path that evaluates mode-dependent data checked the mode once, up front. An evaluation already holding fixture data when the switch happened landed its alerts after the clear, in live mode, where nothing evaluates or resolves fixture resources, so they stayed until the 24-hour stale sweep. A pass-level lock around `checkMockAlerts` was tried and reverted: it closed only that path and made the mock-mode API wait behind a whole pass of synchronous event-log writes. **What changes** - `mockModeFence` (`internal/monitoring/mock_mode_fence.go`): each path takes a scope before reading anything mode-dependent and sends each alert-manager call through `scope.run`. Covered: `checkMockAlerts` (every loop, host agents, disks, snapshot cache), `checkBackupAlerts`, `pollGuestSnapshots`, `checkConnectionAlerts`, `pruneStaleDockerAlerts`, and the resource-store refresh (`updateResourceStore`, `updateResourceStoreForRead` and `syncUnifiedResourceAlertsToState` now take the scope of the state they rebuild from). - `SetMockMode` is serialized, flips the mode, ends the epoch, waits only for calls already admitted, then clears. Unified metrics go one resource per call, so the wait covers single calls, not passes. Admission never blocks. - Registry rebuilds are tracked, not waited on: a refresh evaluates the registry it read back only if no rebuild from an ended epoch overlapped it. Readers that act on the inventory (Docker pruning, backup and snapshot guest lookups, the pollers' previous-state carry) use current state until an undisturbed rebuild in the new epoch replaces the fixture registry. - `mock.SupplementalRecords`/`SupplementalChanges` serve nothing outside mock mode. The router swaps its mock adapters for the real pollers only after the switch, refreshing between sources, and the adapters used to fall back to default fixtures, so a single switch to live mode raised fixture vSphere incidents. **Verified** - Each new test parks a real evaluation at a fixed point (a held lock, a blocking supplier or a gated registry read), switches modes and releases it, with bounded waits. Against the unfenced code: the mock pass left about 40 fixture alerts, the frontend refresh 5 vSphere incidents, the backup check 137 backup-age alerts, and a mock-mode connection observation counted toward a live connection's confirmation. The drain, republish, pruning and carry tests fail against a fence without its wait, overlap check or registry tracking; the republish test also fails with the old mock supplemental defaults. - 995 tests from the touched files and related areas pass, plus `internal/mock` and the `internal/api` mock/platform tests; the new tests pass under `-race` and over repeated runs. **Not covered** - The fence and clear are per monitor while mock mode is process-wide; other tenant monitors neither drain nor clear on a switch (unchanged, noted in the monitoring contract). - Report admission and live pollers outside the fenced paths can still evaluate after the clear when entering mock mode; their alerts can persist until mock mode is left. |
||
|
rcourtman
|
5814c0a44c |
Stop a failing availability check keeping a node online
Once every source with a verdict had gone quiet, the stale pass counted any current availability sighting as online. A node that preserveOrExpireNodes expired stayed online while its HTTP check failed, and a quiet failing check lifted that node's offline verdict to warning. One sighting stands for every check on a target and carries whichever check was projected last, so a failing check that had just run also vouched for another check's old pass. Status aggregation now judges each check on a monitored resource by its own evidence window. A passing check whose evidence is still valid proves the resource answers, and reads online when no current source has a verdict. Every other check abstains, current or quiet: a failing check proves only that one port or service does not answer, so it never makes a guest count as stopped in the workloads summary or the Proxmox page. A check's own row keeps the ordinary rules, and a manual link judges the merged resource by the checks it carries. Projecting or retargeting a check now re-applies the stale pass's rule to targets whose status that pass owns, since the resources API replays checks after its stale pass and never runs another. Only a target's own quiet sources give the pass that ownership, never its checks' sighting. The poller now keeps each check's evidence tied to the observation that passed. It ages out a local location that stopped running beside other locations, as it already did for silent remote paths: the check took its freshness from the newest location, so a failing remote report kept an hour-old local pass counted as reachable with current evidence. An available check's evidence lasts only as long as its newest reachable path, and a remote result adopted as local after an entitlement lapse keeps the server's receipt time rather than the agent's clock. |
||
|
rcourtman
|
b03a32d8ae |
Show a silent agent's retained RAID state as last known (#2652)
Since #2644 the machine drawer reads a silent host agent's Thermals and Disks as last known, but the RAID card between them kept presenting the agent's last report as current: a green "clean" array, an amber "clean, degraded, recovering" array with "Rebuild: 45% · 98000K/sec", and coloured member badges, as if the array were still rebuilding. The Machines table's RAID tooltip did the same with an amber rebuild bar. What changes - `RaidCard` takes the drawer's `lastKnownReason` (`agentReadingsLastKnownReason`, from `agent.stale`). Every array and member state stays as evidence, but reads "<state> (last known)" in muted text beside a muted dot, titled "Last known reading, not current: host agent stopped reporting". A rebuild reads "Rebuild was at N%" without its speed. Member badges lose their status colour, so a member that was not healthy names its state ("/dev/sdb2 · faulty"). - The Machines RAID column (hidden by default) and its tooltip apply the same rule from `agent.stale`: a muted summary with ", last known" for screen readers, and a tooltip headed "RAID Arrays (last known)" without the rebuild bar or speed. - mdadm reports healthy members as "active sync", which the badge helper only matched as an exact "active", so every healthy software RAID member wore the amber warning badge. `raidPresentation.ts` now reads it as healthy (failure states still win). - RaidCard's empty guard ran once at mount, so a drawer opened before the first RAID report never showed the card. It is now tracked. - The array header row wraps, so at 390 wide the longer last-known state takes its own line instead of truncating the array name to "m..". Not changed: the guest drawer's Guest RAID card. The workload model maps `agentRaid` but not the in-guest agent's staleness, so wiring it needs a separate change across `useWorkloads.ts` and the other agent-sourced guest rows. Verification - Private mock stack with a local-only overlay holding Apollo-114 offline past its lease, seeded with a clean md0 and a degraded, rebuilding md1. The live control host Spectrum-709 carries the same arrays. Checked the drawer card at 1920, 1440 and 390, the Machines RAID cell and tooltip at 1920 (the column is container-hidden below that), and the drawer in dark mode. Silent host: no emerald, amber or red classes anywhere in the card or tooltip. Live host: unchanged colours and rebuild bar, with healthy members now green. No console errors and no horizontal scroll. Receipt: `frontend-modern/browser-verification/05a685ccb29f5883.json`. - New tests: `RaidCard.test.tsx` (current, last known, transitions both ways, empty to populated), `raidPresentation.test.ts` (active sync and last-known labels), an `AgentsMachinesTable.test.tsx` case (muted summary, dot, no colour classes, no rebuild bar), and a `SharedPrimitives.guardrails.test.ts` pin. All five fail against the base source. - `vitest` over `components/shared`, `components/Infrastructure`, `components/Workloads`, `features/standalone`, `features/platformPage` and `utils` passes, apart from the PTY tests in `agentInstallCommand.test.ts`, which fail on macOS regardless of this change. The pre-commit battery passes. |
||
|
rcourtman
|
11cc232133 |
Stop reporting unavailable storage as disabled to AI tools (#2657)
The storage pool rows the AI tools build (pulse_storage pools and pulse_query storage pools) derived both enabled and active from "status is not offline". Storage that reported itself unavailable, such as Proxmox storage with active=0, reached Patrol and Assistant as administratively disabled even when it is enabled in Proxmox. Enabled now comes from the storage facet's own flag. Active needs the facet's flag and a row that is not offline, because PBS datastores report Active even while unavailable. Rows without a storage facet keep the status-derived value. TrueNAS pools and datasets were the only storage producers that left both flags at their zero value, so reading the facet alone would have turned every TrueNAS pool into a disabled one. They now report Enabled, and Active from the pool state or the dataset's mount and lock state. |
||
|
rcourtman
|
5b3f507a1c |
Remove status and disk helpers left unused by the selector removal (#2651)
Stacked on #2649 (base commit
|
||
|
rcourtman
|
2f5027a09d | Merge remote-tracking branch 'origin/main' into perf/drop-write-only-summary-chart-caches | ||
|
rcourtman
|
bc15c3500b |
Stop QEMU default disk serials merging nested Proxmox disks (#2653)
## Problem
Nested Proxmox clusters (each PVE node is itself a QEMU VM) lose
physical disks from inventory. QEMU gives a virtual disk configured
without a serial a default one built from its drive ID or a per-VM
counter, so every VM built the same way reports the same value:
- Proxmox's `/nodes/{node}/disks/list` serial is udev's
`ID_SERIAL_SHORT` (`PVE/Diskmanage.pm`). A virtio-scsi disk on a Proxmox
VM reports `drive-scsi0` with no `ID_WWN`; on a libvirt VM it is
`drive-scsi0-0-0-0`. A `-drive` given no ID falls back to QEMU's
automatic ID, `scsi0-hd0` for `if=scsi` and `none0` for `if=none`.
- An emulated ATA disk's serial is `QM` plus a five-digit counter
(`QM00001`).
`diskinventory.IsUsableHardwareID` accepted all of these, so
`resourceFromPhysicalDisk` set `MachineID = "drive-scsi0"` on every node
and the registry minted one `machine:drive-scsi0` canonical disk and one
SMART series for the whole cluster. Three nodes with two such disks each
produced 2 canonical disks instead of 6.
## Change
- `IsUsableHardwareID` rejects exactly QEMU's default shapes:
`drive-scsi` plus digits with at most three `-digits` groups,
`scsi<n>-hd<n>`, `none<n>`, and `QM` plus five digits. Real serials that
only share a prefix (`DRIVE-ASSET123`, `drive-sata0`, `QM0001`) stay
identity. It also strips `naa.`/`eui.`/`wwn-`/`0x` framing before its
all-zero and all-F test, so a `0x0000000000000000` WWN is a placeholder
too.
- TrueNAS provider: a disk took its canonical identity from any
non-empty serial, so a placeholder merged disks across appliances. It
now uses only a usable serial, and lists superseded IDs for every disk
without one.
- TrueNAS history keys: for a serial that was already rejected, such as
`UNKNOWN`, native TrueNAS history used TrueNAS's own disk identifier,
and the unified SMART writer (`syncUnifiedPhysicalDiskMetrics`)
substituted the metrics target only for an empty serial, so the samples
went to the canonical resource ID. Neither matched the chart reader's
key, and rejecting QEMU serials would have put TrueNAS VMs on that path.
Native history now falls back to the disk's source ID, and the writer
always writes under the resolved metrics target, so writer, native
history and reader agree. History lookup keys also drop placeholder
serials.
## Upgrade effect
A disk reporting one of these values (nested PVE nodes, TrueNAS or agent
hosts running as QEMU VMs with virtual disks, TrueNAS disks reporting
`UNKNOWN`) changes canonical ID and SMART metric key once. Samples
recorded under the old key are not carried over, because that key mixed
every disk that shared the value. Path-shaped alert references (Proxmox
disk health and wearout, host-agent SMART) are unaffected; anything
keyed on the old canonical ID or metric key starts again under the new
one. This is recorded in the monitoring and agent-lifecycle contracts.
## Tests
- `TestIsUsableHardwareIDRejectsQEMUDefaultDiskSerials`: each default
shape is rejected, falls back to the scoped topology ID and never
matches itself; prefix-sharing real serials stay usable.
`TestPreferredIDRejectsPlaceholderHardwareIdentity` gains the framed
all-zero/all-F WWNs.
- `TestNestedProxmoxDefaultQEMUSerialsStayPerNode`: 3 nodes ×
(`drive-scsi0`, `QM00005`, `drive-scsi0-0-0-1`) give 9 canonical disks,
9 slots and 9 metric keys, every SMART writer key is a key readers
resolve, and agent SMART source IDs stay per host.
-
`TestTrueNASPlaceholderDiskSerialsStayPerApplianceAndShareOneHistoryKey`:
two appliances with `UNKNOWN` and `drive-scsiN` serials keep one
canonical disk each, and the SMART writer's samples are readable under
each disk's metrics target.
- `TestProviderPlaceholderDiskSerialsKeyNeitherIdentityNorHistory`:
placeholder serials key no identity, list superseded canonical IDs, drop
out of native history lookup keys, and native history lands under the
registry's metrics target.
- Negative controls via `go test -overlay`, one runtime file reverted at
a time: base `identity.go` fails all three round trips (3 canonical
disks instead of 9, 9 TrueNAS disks instead of 16); base `provider.go`
fails the identity and history assertions; base `monitor.go` fails the
SMART-history-under-reader-key assertion; reverting only the native
history key, the superseded-ID gate or the lookup-key filter fails its
own provider assertion.
- `pkg/diskinventory`, `internal/truenas/...`,
`internal/unifiedresources/...` and `internal/monitoring` pass in full;
`internal/api/...` disk, hardware, serial and TrueNAS tests pass.
Stacked on #2648 (`fix/pve-disk-serial-carry`), whose commit this branch
carries until it merges. Not in this change: the registry's merge of
disks across parents on a canonical-ID collision (separate change), and
a pre-existing writer/reader key mismatch for controller-member disks
without a usable serial or WWN (`PhysicalDiskMetaMetricID` appends
topology to a source ID that already carries it), which is tracked
separately.
|
||
|
rcourtman
|
e89f1cec9e |
Drop the summary chart caches nothing reads
infrastructureSummaryCache.ts and workloadsSummaryCache.ts lost their last readers on 2026-05-26, when the InfrastructureSummary card and the standalone Workloads summary were deleted. Their only remaining caller, the Workloads table history hook, runs both fetchers inside createNonSuspendingQuery, which always passes an abort signal, so the in-flight dedupe was skipped as well. What was left ran on every 30-second Trends poll: serialize the whole response on the main thread, then write it to localStorage or, above the 900k-character cap, delete the key. On the 50-node, 929-guest mock estate that was 9.6 MB of workloads JSON and 0.9 MB of infrastructure JSON per poll (5-31 ms each), thrown away every time. The hook now calls ChartsAPI.getWorkloadCharts and getInfrastructureSummaryCharts directly with the same range, signal, maxPoints, node scope and metric list, and the node/agent/Docker-host series fold moves into workloadMetricHistoryModel. The query layer already retains reads across remounts for five minutes and clears them on org switch, so nothing else changes for users. A before/after run on embedded builds showed the same chart requests and sparklines on every node and guest row, with no storage traffic and no multi-megabyte serialization afterwards. The hook takes the deleted cache's place under the workloads-hot-path policy. The subsystem contracts still described InfrastructureSummary, its model and state hook, the shared summary table, selector and details drawer, WorkloadsSummary, throughputPresentation and the two caches as live owners. Those clauses now name the live owner (useWorkloadTableMetricHistory, ProxmoxNodesTable, nodeFromResource, ResponsiveMetricCell, storageSummaryCache, summaryTableFocus), become conditional rules, or are removed where nothing they govern remains. |
||
|
rcourtman
|
94a7573426 | Merge remote-tracking branch 'origin/main' into fix/relative-age-watchdog-patrol | ||
|
rcourtman
|
f165bff68e |
Record the browser pass for the merged Patrol lifecycle card
Merging main brought the suppression-end labels that follow the review state into PatrolAttentionWorkbench.tsx, next to this branch's relative-time clock for the acknowledgement age. Neither commit's receipt covers the combined file, so this records the browser pass on the merged tree. |
||
|
rcourtman
|
bd4f6e36c4 |
Remove contract prose and badge helpers for deleted surfaces (#2655)
Stacked on #2649, which carries its commit |
||
|
rcourtman
|
26bee50a8a |
Restate the read-time age exception after the wall-clock rule
A latest-reading age over data a surface reads once stays the age at read time unless the surface re-reads it; immutable event times read the clock. Main reworded the rule this exception follows, so it comes back after the merge, next to main's wording. |
||
|
rcourtman
|
6a12761bb3 | Merge remote-tracking branch 'origin/main' into fix/relative-age-watchdog-patrol | ||
|
rcourtman
|
8526011cbd |
Take main's wall-clock wording for the relative-age rule
Main replaced the note about the relative-time clock trailing the wall clock: every read now returns the wall clock. Adopt that paragraph as it stands so the merge with main is clean; the read-time exception is restated after it once main is merged. |
||
|
rcourtman
|
425dd0d45b |
Keep same-serial disks on different machines apart
findMatch refuses a physical-disk candidate under another parent, but ingestRecord then minted the same hardware-keyed canonical ID from the serial and merged into its holder on the collision. Disks that report one usable serial on several machines (dual-ported SAS shelves, cloned VMs with an explicit serial, fixed-serial USB bridges, TrueNAS systems) collapsed into one resource parented to whichever machine ingested last, and every other machine lost the disk from inventory. A new disk now resolves its parent to a machine (the nearest host ancestor, so pools and Unraid arrays count as their host) and stays apart only on evidence: one reporter placing the serial on two machines, or machines whose hostnames share nothing. Reporters on one box still join across grouping parents, so the TrueNAS API and the agent on that box stay one disk, and an unknown machine never splits a disk. Once a serial spans machines, every copy takes the serial keyed to its own machine and the unscoped holder is re-keyed, so IDs do not depend on machine ingest order and the unscoped ID never passes between disks that coexist. A disk only one machine reports keeps its ID. Operator splits now hold on this path; the old collision merge ignored them. The registry indexes disks by hardware key because the identity matcher keeps one resource per machine ID. |
||
|
rcourtman
|
59209647f1 |
Remove the unreachable resource policy summary card (#2650)
Stacked on #2649 (it edits the same util, test, registry and
unified-resources contract); this branch carries that commit, so
whichever lands first, the other still merges cleanly.
ResourcePolicySummary lost its last production mount when
|
||
|
rcourtman
|
b43bd0e69d |
Remove exports orphaned by the unified resource table removal (#2649)
Follow-up to #2612, which deleted the never-mounted
`UnifiedResourceTable` and left exports whose only production callers
were the deleted files. An import graph from
`frontend-modern/src/index.tsx` (TypeScript compiler, static and dynamic
imports) confirms none of them is reachable on current main.
## What goes
- **`components/Infrastructure/infrastructureSelectors.ts`** and its two
test files. The one live caller was `workloadSelectors.ts`, which used
`computeIOScale` for Net I/O and Disk I/O outlier emphasis by building
fake `Resource` objects from guests and casting them. The median, MAD
and percentile math now lives privately in `workloadSelectors.ts` and
reads guest throughput directly. Output is identical for every input,
including null, negative, NaN and infinite throughput and an empty list.
The distribution cases from the deleted suites now live in
`workloadSelectors.test.ts`.
- **`utils/resourcePolicyPresentation.ts`**:
`getResourcePolicyTableBadges`, `hasBlockingResourcePolicyPosture` and
`shouldShowResourceAlternateName`. They only fed the deleted table
cards.
- **`ResourceFacetSummary`**: the `maxVisibleBadges` and `class` props
and the "+N" overflow chip. Only the deleted rows passed them; the
detail drawer, the one live caller, passes neither and still renders
every chip.
## Governance
- `registry.json`: the deleted file leaves `owned_files`, the
`workloads-hot-path` and `resource-consumers` match and proof lists, and
the shared ownership entry. `status.json` lane L10 drops it from
evidence; the derived lane status is unchanged.
- `performance-and-scalability.md` and `unified-resources.md`: the
deleted file's Canonical File and Shared Boundary entries are removed,
along with the selector search, status-ordering and source-filter
paragraphs. Prose that described table-only chip limits, row policy
badges and alternate-name suppression as live is now conditional or
scoped to the drawer. Completion Obligation 15 now states the
operator-local raw-name search rule from
|
||
|
rcourtman
|
e02edd9989 |
Show retained disk temperatures as last known in drawers and tables
Normalization keeps a disk's last temperature when the current observation did not collect one, such as a disk in standby or a host agent past its reporting lease. PR #2603 taught the Storage presenters to show that value as last known. Four surfaces still read the raw number as a current reading: - The TrueNAS disk drawer's Temperature row took the 55C warning tone from a retained reading, and its summary listed it as a reading. The row now says "(last known)" with the reason as title and no tone. The summary drops it because it has no room for the reason. - The drawer History fallback fed a retained reading into smart_temp and labelled it "current" when no points were stored. It now goes through the deferred-metric slot, so the legend says "last known" and the reason shows above the chart. The disk's value is read from physicalDisk beside its collection state, because websocket rows carry no top-level temperature for a disk and showed "-" instead. - The TrueNAS storage table's Temp cell showed a retained reading plainly, and hottest-first sorted it to the top. It now renders muted with its reason, and the sort ranks it as no reading. - The Proxmox outdated sensor setup notice counted a retained reading as a temperature arriving, so it could hide for a SATA disk Pulse cannot read now. All four reuse getPhysicalDiskTemperaturePresentation or isPhysicalDiskTemperatureCurrent instead of adding a predicate. |
||
|
rcourtman
|
8d00d54740 |
Stop a stale agent SMART row lending a replaced disk its identity
The linked host agent's SMART merge falls back to matching a Proxmox disk by device path, and a path names a slot, not a disk. When a disk is swapped before the agent reports again, or after the agent went silent and kept its last rows, the row for the old disk matched the replacement. It filled or, for SAS, promoted the old serial onto it, which keyed the replacement to the old disk's canonical resource and metric history, and lent it the old disk's temperature and failed health. For SAS the registry's same-path join did the same at the canonical level even when monitoring kept the rows apart. Both now refuse a row whose identity contradicts the disk's, judged only on what both producers report like for like. WWNs count for every disk. Serials count only where Proxmox's serial is the drive's own, as smartctl's is: NVMe namespaces, and disks presented with SCSI vendor "ATA", whose serial udev's ata_id reads from IDENTIFY. Elsewhere Proxmox's serial may be a SAS address, a SCSI designator or a USB bridge's own serial (#1595). Reporter spellings are not disagreement: udev writes whitespace in serials as underscores, its T10 designator for an ATA drive puts "ATA" and the model before the drive serial, and its ID_WWN keeps only the first 64 bits of an NAA 6 identifier. The WWN match step compares normalized identity too, so the agent's 5-c50-a1b2c3d4 spelling finds the Proxmox 0x5000c500a1b2c3d4 disk. Proxmox's disk inventory reports a serial or WWN udev cannot read as the literal "unknown". The poller recorded it as a collected serial, so the collection state hid the gap and the agent's serial never filled it. It is now recorded as empty, and retained evidence carries only a usable previous serial so an older record cannot restore it. Identity keys already ignored the placeholder. |
||
|
rcourtman
|
6fb9752397 |
Stop reporting unavailable storage as disabled to AI tools
The storage pool rows the AI tools build (pulse_storage pools and pulse_query storage pools) derived both enabled and active from "status is not offline". Storage that reported itself unavailable, such as Proxmox storage with active=0, reached Patrol and Assistant as administratively disabled even when it is enabled in Proxmox. Enabled now comes from the storage facet's own flag. Active needs the facet's flag and a row that is not offline, because PBS datastores report Active even while unavailable. Rows without a storage facet keep the status-derived value. TrueNAS pools and datasets were the only storage producers that left both flags at their zero value, so reading the facet alone would have turned every TrueNAS pool into a disabled one. They now report Enabled, and Active from the pool state or the dataset's mount and lock state. |
||
|
rcourtman
|
41cea2ffab |
Stop a replaced PVE disk inheriting the old disk's evidence (#2648)
## Problem The Proxmox disk poller keeps evidence it could not collect this poll by looking up the previous poll's record of the disk: by serial, then WWN, then the Proxmox source ID (`instance-node-devpath`), then the device token. The source ID and device token name a slot, not a disk, and a replacement in the same slot keeps both. So: - a replacement reporting a different serial or WWN took the old disk's retained temperature, controller, pool and I/O; - a replacement record with an empty serial also took the old disk's serial. The registry keys disks on serial, so the new disk became the old disk's canonical resource and SMART metric series, and every later poll carried the borrowed serial forward. Current Proxmox reports a missing serial as the literal `unknown`, which the poller records as reported, so the serial copy needs a producer that sends an empty serial, such as the host-agent fallback rows. The evidence copy happens with real Proxmox data. ## Change - `previousPhysicalDiskEvidence` matches by hardware identity through `diskinventory.HardwareIdentityMatch` (serial first, then WWN, each against both fields of the earlier record), and refuses a slot match when both records carry a serial or WWN and none agrees. Missing identity is still not a conflict: placeholders such as `unknown`, and an empty serial, keep the slot's evidence as before. - The registry's merged disk view keeps the host agent's WWN, which the agent spells as unpadded NAA 5 fields (`5-c50-a1b2c3d4`, `internal/hostagent/smartctl.go` `formatWWN`) while Proxmox reports `0x5000c500a1b2c3d4`. `HardwareIdentityMatch` now re-pads that spelling for WWN fields only, so one disk seen by both reporters is not taken for a replacement. The registry's linked-disk join uses the same comparison. ## Tests - `TestPhysicalDiskReplacementInSameSlotDoesNotInheritPreviousSerial` drives the poller and registry across two polls after a swap; on base the replacement inherits the old serial and canonical resource. Its controls keep a serial-omitting same disk on its resource. - `TestPreviousPhysicalDiskEvidenceRejectsSlotMatchAcrossConflictingIdentity` covers renamed devices matched only by identity, PVE vs agent WWN spellings, SAS address serials with a matching WWN, placeholders, and conflicting serial/WWN by source ID and by device token. - Negative controls via `go test -overlay`: on base the replacement, conflict and renamed-device cases fail; with the guard but without the WWN re-padding, the same-disk PVE/agent WWN cases fail. - `pkg/diskinventory`, `internal/unifiedresources` and `internal/monitoring` pass in full. Follow-ups, not in this change: the host-agent SMART merge's device-path fallback can still fill or promote a serial from a stale SMART row; QEMU default serials (`drive-scsi0`) collapse nested Proxmox nodes' disks into one resource; and the registry merges disks across parents on a canonical-ID collision. |
||
|
rcourtman
|
73a2d68cf9 |
Pin disk heat parity for canonical and linked-guest overrides
A second review of the per-host Disk Temp change found two gaps. The parity test did not cover an override stored under the canonical resource an agent ID resolves to, or one inherited from a linked guest. The documented browser limits also listed only two cases where the browser cannot see CheckHost's chain. The parity test now checks both overrides, and a disabled linked guest, against what CheckHost fires. storage-recovery.md lists every case where the browser's keys can differ from alerts and Patrol: manually merged agent and VM disks on the Storage page, overrides inherited through unpublished link hints or stored under a linked node's canonical ID, and guest keys for an agent merged into a guest it is not automatically linked to. |
||
|
rcourtman
|
06b7f5a0b9 |
Keep fixture alerts from outliving a mock-mode switch
SetMockMode(false) switched the mock package off and then cleared active alerts, but every path that evaluates mode-dependent data checked the mode once, up front. A mock alert pass (started from the ticker with go, so passes overlap), a frontend or ingest refresh evaluating a GetState snapshot, a backup-age check over fixture recovery rollups, or a connection check over the mock ledger could already hold fixture data when the switch happened. Its evaluations then landed after the clear, in live mode, where nothing evaluates or resolves fixture resources, so the alerts stayed until the 24-hour stale sweep. A pass-level lock around checkMockAlerts was tried and reverted: it left the other paths open and made the mock-mode API wait behind a whole pass of synchronous event-log writes. The monitor now owns a mock-mode epoch fence. The mock pass, backup-age and snapshot checks, connection checks, Docker pruning and the resource-store refresh take a scope before reading anything mode-dependent and send each alert-manager call through it. SetMockMode, now serialized, flips the mode, ends the epoch, waits only for calls already admitted, and then clears. Unified metrics are evaluated one resource per call, so the wait covers single calls, not passes. Registry rebuilds are tracked rather than waited on. A refresh evaluates the registry it read back only if no rebuild from an ended epoch overlapped it, and the readers that open or remove alerts from the inventory use current state until a clean rebuild in the new epoch replaces the previous mode's registry. The mock supplemental fixture sources now serve nothing outside mock mode: the router swaps them for the real pollers only after the switch, refreshing between sources, so a single switch to live mode used to raise fixture vSphere incidents. The new tests park a real evaluation at a fixed point (a held lock or a blocking supplier), switch modes and release it. Against the unfenced code the mock pass left about 40 fixture alerts, the frontend refresh 5 vSphere incidents and the backup check 137 backup-age alerts in live mode, and a mock-mode connection observation counted toward a live connection's confirmation. The drain, republish and pruning tests fail against a fence without its wait, overlap check or registry tracking. Switching mock mode with several tenant monitors running remains uncovered and is noted in the monitoring contract. |
||
|
rcourtman
|
bf29201b19 |
Remove contract prose and badge helpers for deleted surfaces
PR 2612 deleted the never-mounted UnifiedResourceTable, and
|
||
|
rcourtman
|
d5c945241d |
Show a remote unacknowledge in a tab that acknowledged locally (#2647)
After acknowledging an alert on Alerts > Overview, an unacknowledge from another session never brought the card back in that tab until a reload, although `/api/alerts/active` and the store both had it unacknowledged. A pure remote acknowledge/unacknowledge round trip (no click in the tab) already worked. ## Cause `useAlertAcknowledgementState` kept its own acknowledgement override and spread it over the store alerts. Its release effect never read the override signal and only re-ran when the `Object.values` memo over the keyed alert store changed, which in-place store writes (`updateAlert`, `reconcile` payloads) do not trigger. The override outlived the server's confirmation and then masked the server's next change. Whether it cleared depended on unrelated alerts appearing or resolving, so the bug was intermittent. The override was redundant: the websocket store's `updateAlert` already writes the optimistic state and holds back stale payloads until the server confirms. The hook now writes through it and keeps no copy. `updateAlert` is required, and its only caller always passed it. ## Store hold fixes (`frontend-modern/src/stores/websocket.ts`) - **Broadcast before response:** the backend broadcasts after writing the HTTP response, so the confirming payload can reach the socket before the response resolves. The hold then waited for a second confirmation, skipped a following remote unacknowledge, and after 15s showed "Server did not confirm the alert acknowledgment in time" without actually re-syncing. Reproduced live. A change the stored alert already shows now sets no hold and keeps the server's `ackTime`/`ackUser`. - **Give-up re-sync:** a hold that gives up now re-syncs from `/api/alerts/active`. If it joins a recovery that the revision fence then discards, it fetches again, since a quiet estate may send no further alert delta. - **Overlapping REST recovery:** a local acknowledgement write supersedes a REST recovery already in flight, whose response may predate the write. The store discards that response and fetches again once it settles. ## Tests - **Hook (`useAlertAcknowledgementState.test.tsx`), over a real `createStore` + `reconcile` store:** - local acknowledge, then remote unacknowledge - local restore, then remote acknowledge - acknowledge, restore and bulk acknowledge The first two fail on the old hook at the remote-change step. - **Overview state (`useAlertOverviewState.test.tsx`):** now backed by a keyed store, and checks that the card returns after a remote unacknowledge. Fails on the old hook. - **Store (`websocket-unified.test.ts`):** covers the broadcast-before-response race, the give-up re-sync, the overlapping-recovery supersede, a give-up joining a fenced-out recovery, and a recovery that predates a held acknowledge. The first two fail on the old store. For the recovery races, removing each mechanism (supersede, its fresh fetch, the re-sync chain) fails the test that pins it. ## Browser check Private mock stack at 1280×900 and 375×812: - **Before:** the card stayed hidden while the header read "2 critical, Acknowledged 1" against a nav badge of 3 critical. - **Reported flow:** a local Acknowledge, then a `curl` unacknowledge, brings the card back within 2s. - **Restore direction:** Unacknowledge in the tab, then a remote acknowledge, moves it back to Acknowledged within 2s. - **Race:** with the acknowledge response held 1.5s so the socket wins, the card returns at once with no toast. With only `websocket.ts` reverted to main, it stayed hidden about 16s and then the toast fired. ## Left as is If the give-up's own REST fetch fails, the tab keeps the local state until the next alert payload. That matches main, where a give-up never re-synced at all. |
||
|
rcourtman
|
a28422ae26 |
Judge disk heat under the reporting host's Disk Temp override
Disk temperature alerts evaluate each SMART disk under the thresholds of the host agent that reports it. An explicit Disk Temp override on that host, or one inherited from its linked node or guest, replaces the per-type trigger, and a host whose alerts are switched off raises none. Since #2636 the Physical Disks verdict and Patrol judged heat by disk type alone. A host with an 80C override still read Running Hot at the 70C NVMe trigger, and a host with its alerts off still read hot. - alerts.Manager.HostDiskTemperatureThreshold resolves the policy for a host exactly as CheckHost does, from the same helpers. DiskTemperatureThreshold is its hostless case. - Patrol and the AI chat context find each disk's host by walking its parent to the agent machine (through the pool for Unraid disks) and ask the threshold adapter for that host's limits. - Physical Disks, the disk detail, the pool drawer's linked disks and the guest drawer pass the reporting machine's override keys to the thresholds resolver, in the order CheckHost reads them: agent ID, canonical ID, then the merged node or guest. The resolver treats an override or agent default that switches alerts off as no heat. Two cases still fall back to the per-type trigger in the browser while alerts and Patrol apply the override: on the Storage page, a disk whose agent was manually linked into a VM, and an override an agent inherits from a guest it is linked to without a merge. |
||
|
rcourtman
|
044dc091ec |
Own PVE disk alert history rows by recorded hardware (#2646)
Follows #2627, which left PVE disk alert references out of resource history: binding the path reference failed review because a binding belongs to the reference string, so any disk that takes over the path inherits every row and every read of it. A physical disk's history (facets, timeline and intelligence APIs, Assistant resource context, Patrol's scoped feed) therefore never showed its own SMART health and wearout alerts. This owns each lifecycle row instead of binding the reference. **Write** - `proxmoxDiskAlertMetadata` now records `disk_wwn` next to `disk_serial`. - `MonitorAdapter.RecordChange` writes a PVE disk alert row under the one physical disk the registry knows by that serial or WWN (`proxmoxDiskAlertOwner`), wherever it sits now. A WWN only breaks a tie between two matches; it never rules a disk out, because the registry keeps a merged agent's smartctl-framed WWN (`5-c500-…`), which doesn't compare equal to PVE's (`0x5000c500…`). The registry only translates identity into a canonical ID. The path never breaks a tie, so a generation that still places another disk at the path cannot gain a row that disk does not match by identity, and no check precedes a later write. A disk not yet in inventory, or removed, gets the ID the registry mints for its identity unless another resource holds it. A row with no usable identity names only the one identity-less disk at its path. Anything else stays under the reference. - An owned row records the alert's reference as `alert_resource_id`. No alias is written, the reference is never queued for the legacy backfill, and `resolveHistoryReference` refuses it. - Rows journaled before this change stay under the reference, unrewritten (journal rows are never rewritten; retention is 30 days). **Read** - A store read by the reference (rows and every count, SQLite and MemoryStore) also returns the rows owned away from it. They are matched through the indexed alert identifier plus the recorded reference, so the Alerts history Resource action lists the same occurrences as before. - `QueryIncidents` matches owned rows to occurrences under their alert reference, while their evidence keeps the owning disk. Reads by alert identifier and by the reference return the same occurrences as before, including one occurrence whose firing and resolution two disks own. A read by a disk admits the saved shells of its own occurrences. An owned row whose firing the read lacks takes the occurrence or saved shell at its recorded start (shells within the usual tolerance), or opens it under the alert-centric read's ID, before events are assigned. That boundary is not canonical, so capped reads keep their guard. Notes recorded from a disk read reach the occurrence the alert-centric read shows. **Known residuals (in the unified-resources contract)** - History follows inventory where inventory merges: same-serial disks, a serial carried at a path when a poll reports none, and disks correlated with an agent observation by path. - When PVE and the agent share no identifier for such a disk (a SAS address reported as the serial), or the alert fired before the PVE observation reached the registry, the row carries the minted ID. The disk's history omits it, and reads by the reference still return it. - A disk the registry keys by its agent's WWN alone also carries PVE's serial, so a different disk reporting that same serial, before its first poll reaches the registry, has its row written under the first disk. - Two occurrences of one alert starting within the shell tolerance: the alert-centric read assigns a saved shell by firing order, which a disk read missing a firing cannot reproduce. **Tests** - `TestProxmoxDiskAlertRowsFollowRecordedHardwareIdentity` (SQLite and memory): reboot reorder before the registry catches up, replacement before and after its first poll, WWN-only, path-only, placeholder serial, counts. - `TestProxmoxDiskAlertOwnerDecisions`: shared serials, WWN tie-break, agent-framed WWN, RAID volume (full NAA serial, truncated WWN), minted ID already held, identity-less paths. - `TestHistoryIdentityNeverBindsProxmoxDiskAlertReferences`, which replaces `TestHistoryIdentityLeavesProxmoxDiskAlertReferencesUnbound`. - `TestProxmoxDiskAlertReferenceReadUsesIndexes`: the read by reference uses the canonical and alert-identifier indexes, not a scan. - `TestIncidentQueryKeepsOccurrencesOfHardwareOwnedDiskAlerts` compares the journaled and owned layouts for alert-identifier, `StartedAt` and reference reads, plus disk reads, notes, evidence and a split occurrence with no shell, an exact shell and an offset shell, on both stores. `TestIncidentQueryTruncatedReadKeepsOwnedOccurrencesGuarded` covers capped reads. - `TestCheckDiskHealthAlertsCarryHistoryOwnershipIdentity`: the producer's alert identifiers match `ProxmoxPhysicalDiskAlertIdentifiers`. - `TestProxmoxDiskAlertTimelineFollowsRecordedHardware`: real alert manager, reboot reorder, incidents by reference and by disk. - Each new test fails against the code it guards (checked by reverting it). **Live check** (private mock-mode build, fresh data dir): 6 fired rows and 1 acknowledgement, all owned by the disk their serial names. No aliases. Each disk's timeline API returns its rows. `/api/ai/incidents?resource_id=<reference>` and `?resource_id=<disk>` return the same incident (same saved ID; fired and acknowledged events whose evidence names the disk). |
||
|
rcourtman
|
853e172fc2 |
Pair a merged disk's temperature with its rows' own states (#2645)
Follow-up to #2618. ## Why The registry merges a physical disk's rows (Proxmox disk, host agent SMART, Unraid inventory, TrueNAS) one step at a time. It chooses the temperature by source preference and merges collection states separately. #2618 paired the shown value with its row's state, but only exactly for two rows: with three or more, an earlier step's single merged state stood in for its rows. Two counterexamples (same serial, no SMART attributes on the agent row): 1. Silent agent SMART 72 (`unavailable`, smartctl), offline Unraid inventory 41 (`unavailable`, unraid), stale Proxmox copy 41 (`available`, smartctl): the merged disk showed **41 as collected**. 2. A legacy agent's fresh 50 (no state) next to a Proxmox inventory row (`unsupported`) showed **50 as unsupported**. With a second Proxmox row collecting 40, which state the 50 showed depended on ingest order. The end-to-end test added here also shows a production case on main today. A pre-6.2 agent linked to a PVE node whose host goes down withdraws its reading without a source. Its copy on the Proxmox disk carries `host_agent`, and no disk poll refreshes that copy. Through the production rebuild (`PopulateFromSnapshot`), the canonical disk then shows the retained 38 as `available(host_agent)`. With equal values, #2618's pairing skipped, and `MergeReportedStatus` could not match `""` to `host_agent`. ## What - The merged disk keeps each row's current temperature reading and state. This is merge-time only, never serialized, and a row that reports again replaces its reading. - The shown value's state comes from the readings of that value. A collected reading wins over one that isn't, and the agent's own report wins over a copy. While a current reading holds the shown value, the order in which different rows arrive no longer changes the state. - A state another row copied from an agent source gives way to the agent's own withdrawal of that source, in either order. That covers a copied `available` and a copied older withdrawal. - A legacy agent's sourceless withdrawal now matches its `host_agent` copy. The legacy-agent rule moves into `diskinventory.LegacyHostAgentStatus`: monitoring stamps the Proxmox copy with it, and the registry matches the withdrawal with it, so the two cannot drift. - A second agent reporting the same disk is not a copy, so its withdrawal no longer withdraws another agent's own collected reading. Before this change, the result depended on ingest order. - When source preference keeps a value that no current reading holds, the value keeps the state it was presented with and the row that state belonged to. That happens when its row went into standby, or reported again without the SMART attributes that made it preferred. The value takes that row's later withdrawal, and the agent's withdrawal of its source still supersedes it. - Value selection is unchanged. A silent agent's 72 next to Proxmox's collected 40 still shows 72 as last-known, as the existing test pins. - Boundaries: - A disk entering as a new row is that row alone. - A disk seeded from a clone continues from its readings only while it still presents the value and state they were built for. - Otherwise it stands as one reading of unknown rows. An agent's withdrawal of its source supersedes that reading, and the reading still withdraws for the agent when the agent is among its sources. ## Known limits - JSON keeps only the presented state. A withdrawal the state does not show is lost across a round trip, but the production rebuild never merges across JSON. - A manual link keeps the primary disk's readings and drops the other disk's. - A kept value's state depends on the history of reports, as the kept value itself does. It stays `available` when its row now reports a different collected value (same as before). Kept values need repeated reports from one row, so they arise only in incremental ingest, never in the per-poll rebuild. ## Tests - `TestPhysicalDiskMergePairsEveryShownTemperatureWithItsRowState`: every ingest order of a catalog of 8 agent, 3 Unraid (plus none), 7 Proxmox and 1 TrueNAS (plus none) row variants. It checks the shown state against a set-based oracle, including the collected-first and own-report-first preference. - `TestPhysicalDiskMergeNeverPresentsAWithdrawnTemperatureAsCollected` now covers counterexample 1 with equal values in all orders. `TestPhysicalDiskMergeKeepsALegacyAgentReadingCollected` covers counterexample 2. - `TestPhysicalDiskMergePairsTemperatureWithItsCollectionState` gains two cases: equal-value resumption against a Proxmox copy that has SMART attributes, and a legacy agent past its lease. - `TestPhysicalDiskMergeTemperatureReadingsAcrossIngestBoundaries` covers: - clones that are seeded back, edited, or re-ingested as a Proxmox row; - presented disks against newer agent reports; - same-row re-reports and kept values; - two agents sharing a source. Against `origin/main`'s registry, every case passes except four this change improves: clone continuation, a kept value from another source, and the two two-agent cases. - `TestSilentLegacyAgentCanonicalDiskFollowsItsWithdrawal` (monitoring, end to end through `PopulateFromSnapshot`): a legacy agent's disk is collected while it reports, not collected once its lease expires with a stale Proxmox copy, and collected again when it resumes. It fails on main. - `TestLegacyHostAgentStatusFillsTheLegacyAgentProvenance` (diskinventory) covers the shared helper. - The existing pairing, withdrawal, Unraid masking and `physical_disk_roundtrip_test.go` tests stay green. - A sweep of 22 code mutations is caught by these tests. It covers dropped supersession, preference order, stand-in flags, the legacy key, record validation, stamping, kept-value handling and owner tracking. Contracts updated: unified-resources (merge rule and its limits), monitoring and agent-lifecycle (legacy withdrawal matching), storage-recovery (what storage surfaces read from a merged disk). |
||
|
rcourtman
|
393c6ca6b4 |
Remove status and disk helpers left unused by the selector removal
Deleting infrastructureSelectors.ts removed the only production caller of three more exports. getInfrastructureSystemIdentitySortLabel is handled in a separate change; this removes the other two: STATUS_SORT_ORDER in utils/status.ts and getResourceDiskSummary in utils/format.ts, along with the tests that only pinned them. Neither is used inside its own module, and the helpers getResourceDiskSummary relied on (normalizeDiskArray, NormalizedDisk) keep their other callers. Rollup was already tree-shaking both, so the production build is byte-identical to its parent's. |
||
|
rcourtman
|
eb2b859d22 |
Show a silent agent's retained RAID state as last known
Since #2644 the machine drawer reads a silent host agent's Thermals and Disks as last known, but the RAID card between them kept presenting the agent's last report as current. On a mock stack the held host showed a green "clean" array and an amber "clean, degraded, recovering" array with "Rebuild: 45% · 98000K/sec" and coloured member badges, as if the array were still rebuilding now. The Machines table's RAID tooltip did the same, with an amber rebuild bar. RaidCard now takes the drawer's lastKnownReason. Every array and member state stays, because an array that degraded before the machine went quiet is evidence, but it reads "(last known)" in muted text beside a muted dot, titled with the reason. A rebuild reads "Rebuild was at N%" without its speed, and member badges lose their status colour, so a member that was not healthy names its state ("/dev/sdb2 · faulty"). The Machines RAID column and tooltip apply the same rule from agent.stale. The guest drawer's Guest RAID card stays as it is: the workload model carries agentRaid but no agent staleness yet. Two fixes ride along in the same primitive. mdadm reports a healthy member as "active sync", which the badge helper only matched as exact "active", so every healthy software RAID member wore the amber warning badge; raidPresentation now reads it as healthy. RaidCard's empty guard ran once at mount, so a drawer opened before the first RAID report never showed the card; it is now tracked. The array header row also wraps, so on a phone the longer last-known state no longer truncates the name. |
||
|
rcourtman
|
dd8bfa4db8 |
Stop QEMU default disk serials merging nested Proxmox disks
QEMU gives a virtual disk configured without a serial one built from its drive ID or a per-VM counter, so every VM built the same way reports the same value. Proxmox's disks/list serial is udev's ID_SERIAL_SHORT, which for a virtio-scsi disk is "drive-scsi0" (libvirt: "drive-scsi0-0-0-0"), and an ATA disk's serial is "QM00001". IsUsableHardwareID accepted these as hardware identity, so every node of a nested Proxmox cluster minted one machine:drive-scsi0 canonical disk and one SMART series, and all but one node lost the disk from inventory. IsUsableHardwareID now rejects exactly QEMU's default shapes: drive-scsi plus digits with at most three -digits groups, QEMU's automatic drive IDs scsiN-hdN and noneN, and QM plus five digits. Real serials that share a prefix, such as DRIVE-ASSET123, stay identity. It also strips WWN framing before the all-zero test, so a 0x0000000000000000 WWN is a placeholder too. The TrueNAS provider set a disk's canonical identity from any non-empty serial, so placeholders merged disks across appliances. It now uses only a usable serial. For a serial that was already rejected, such as UNKNOWN, native TrueNAS history and the unified SMART writer disagreed with the chart reader: native history used TrueNAS's disk identifier, and the writer substituted the metrics target only for an empty serial, so a placeholder sent samples to the canonical resource ID. Rejecting QEMU serials would have put TrueNAS VMs on that path. Both now use the key the registry's metrics target resolves, the disk's source ID, and native history lookups no longer route by placeholder serials. Disks that report one of these values change canonical ID and SMART metric key once; samples under the old key are not carried over because that key mixed every disk that shared the value. |
||
|
rcourtman
|
155442c846 |
Remove the unreachable resource policy summary card
ResourcePolicySummary lost its last production mount when
|
||
|
rcourtman
|
38ce73c24a |
Show a silent agent's retained disk usage as last known (#2644)
Since #2634 a host agent past its reporting lease reads offline, and the machine drawer marks its Thermals rows "(last known)". The Disks card in the same Machine details still presented the agent's retained filesystem usage as current. On a mock stack the silent host's root mount read 93% in red with a red aggregate bar, beside Thermals reading "45°C (last known)". **Change** - `DisksCard` takes an optional `lastKnownReason`. With one, the total and each mount keep their figures, because a disk that filled before the machine went quiet is evidence. They read "Last known N%" with used and total bytes, in muted text titled "Last known reading, not current: host agent stopped reporting", without threshold colour, the aggregate `StackedDiskBar` or per-mount bars. The guest drawer already treats retained filesystem rows this way. - The drawer derives one `agentReadingsLastKnownReason` from `agent.stale` for Thermals and Disks alike, so a Proxmox node that stays online with a silent linked agent gets the same treatment. - Mount rows may now wrap. On a phone the row's label keeps its full width, so a long mountpoint squeezed the value into a sliver and clipped its byte figures. The label now takes its own line with the value below. **Verification** - Private mock stack with one standalone host held offline past its lease (root mount at 93%, plus a long mountpoint), at 1440 and 390 wide, light and dark. A live host was checked as a control and is unchanged. Receipt: `frontend-modern/browser-verification/34bdf8cf77873482.json`. - Component tests cover current vs last-known rendering and a stale→current flip. A drawer test covers offline and online rows with a stale agent and fails when the wiring is removed. A guardrail pins the branch and the drawer wiring. The RAID card beside it still shows a silent agent's last RAID state as current. That is tracked separately. |
||
|
rcourtman
|
9b644b3d49 |
Stop a replaced PVE disk inheriting the old disk's evidence
The disk poller keeps evidence it could not collect this poll by looking
up the previous record, first by serial and WWN, then by the Proxmox
source ID and device token. Those last two name a slot, not a disk, and
a replacement in the same slot keeps both. So a replacement reporting a
different serial or WWN took the old disk's retained temperature,
controller, pool and I/O. A replacement record with an empty serial
also took the old serial. The registry keys disks on serial, so the
new disk became the old disk's canonical resource and SMART metric
series, and each later poll carried the borrowed serial forward.
Match on hardware identity through diskinventory.HardwareIdentityMatch,
serial first then WWN, and refuse a slot match when both records carry
a serial or WWN that share no value. Missing identity is still not a
conflict, so Proxmox's literal "unknown" and other placeholders keep
the slot's evidence as before.
The registry's merged disk view keeps the host agent's WWN, which the
agent spells as unpadded NAA 5 fields ("5-c50-a1b2c3d4") while Proxmox
reports "0x5000c500a1b2c3d4". HardwareIdentityMatch now re-pads that
spelling for WWN fields, so the same disk seen by both reporters is
not mistaken for a replacement.
|
||
|
rcourtman
|
df9514df37 |
Show a remote unacknowledge in a tab that acknowledged locally
After acknowledging an alert on Alerts > Overview, an unacknowledge from another session never brought the card back in that tab until a reload, although /api/alerts/active and the store both had it unacknowledged. useAlertAcknowledgementState kept its own acknowledgement override over the store alerts. Its release effect never read the override signal and only re-ran when the Object.values memo over the keyed store changed, which in-place store writes do not trigger. The override outlived the server's confirmation and then masked the server's next change; whether it cleared depended on unrelated alerts appearing or resolving. The websocket store's updateAlert already writes the optimistic state and holds back stale payloads until the server confirms, so the hook now writes through it and keeps no copy (its only caller always passed it). The store hold had gaps of its own: - The backend broadcasts after writing the HTTP response, so the confirming payload can arrive before the response resolves. The hold then waited for a second confirmation, skipped a following remote unacknowledge, and after 15s showed "Server did not confirm the alert acknowledgment in time" without re-syncing. A change the stored alert already shows now sets no hold and keeps the server's ackTime/ackUser. - A hold that gives up now re-syncs from /api/alerts/active, fetching again if a recovery it joins is fenced out, since a quiet estate may send no further alert delta. - A local acknowledgement write supersedes a REST recovery in flight, whose response may predate the write: the store discards it and fetches again once it settles. Tests: hook tests over a real createStore/reconcile store (local acknowledge then remote unacknowledge, local restore then remote acknowledge, bulk) and the overview-state test backed by a keyed store fail on the old hook at the remote-change step. The store tests for the broadcast-before-response race and the give-up fail on the old store; the recovery races are each pinned by a test that fails when its mechanism is removed. Browser check on a private mock stack at 1280x900 and 375x812: a local acknowledge then a curl unacknowledge brings the card back within 2s (before, it stayed hidden while the nav badge already counted it); a local restore then a remote acknowledge moves it back to Acknowledged; with the acknowledge response delayed 1.5s so the socket wins, the card returns at once with no toast, where main's store held it back for about 16s and then showed the toast. |
||
|
rcourtman
|
1f2ff618b6 |
Own PVE disk alert history rows by recorded hardware
PVE disk health and wearout alerts reference the disk by instance, node and device path, so a physical disk's history never showed its own SMART and wearout alerts. Binding the path reference failed review: a binding belongs to the reference string, so any disk that takes over the path inherits every row and every read of it. Own each lifecycle row instead. The alert now records the disk's WWN next to its serial, and the history writer puts the row under the disk the registry knows by that hardware identity, wherever it sits now. The registry only translates identity into a canonical ID, so a generation that still places another disk at the path cannot claim the row, and no check precedes a later write. A disk not yet in inventory, or removed, gets the ID the registry mints for its identity. Rows without usable or unambiguous identity stay under the reference, as do rows journaled before this change; journal rows are never rewritten. Reads by the reference still return every row raised under it, through the alert identifier index, and the incident projection keeps owned rows under their alert reference, so the Alerts history Resource action and incident timelines show the same occurrences as before. A read by the disk returns its own occurrences with their saved shells and notes. |
||
|
Richard Courtman
|
ff67e38968 |
Pair a merged disk's temperature with its rows' own states
The registry merges a physical disk's rows one step at a time, choosing the temperature by source preference and merging collection states separately. The pairing rule from #2618 was exact for two rows only: with three or more, an earlier step's single merged state stood in for its rows. A silent agent, an offline Unraid inventory and a stale Proxmox copy could leave a withdrawn 41 labelled collected, and a legacy agent's stateless reading could pick up another row's "unsupported". The merged disk now keeps each row's current reading and state (merge-time only, never serialized), and the shown value takes its state from the readings of that value: a collected reading first, then the agent's own report over a copy. A copy of an agent source's state gives way to the agent's own withdrawal, including a legacy agent's sourceless one, and a value source preference keeps after its row reported again keeps its presented state under that row's later withdrawal. Value selection is unchanged. |
||
|
rcourtman
|
812c712abb | Merge remote-tracking branch 'origin/main' into fix/stale-agent-disks-card | ||
|
rcourtman
|
8b36dec0a2 |
Keep a source's own offline verdict through the stale pass (#2643)
## Problem The unified registry's stale pass recomputed a resource's status from its source sightings, and read a sighting that had *delivered recently* as *online*. A sighting only says the source delivered, not what it reported: - A Proxmox node the cluster reports offline on a live poll came back `online` once its linked host agent went silent, which is exactly what a powered-off machine's agent does (the #1441 / discussion #1135 symptom for users with an agent on the node). The Proxmox nodes table showed an amber dot next to its own "Offline" label while the drawer led with a critical "Node is offline" alert. - A node `preserveOrExpireNodes` expired after failed polls read `warning` ("Degraded"), and the stopped guests on it were counted under "Attention". Node alerts were not affected: `CheckNode` reads the poller's own `models.Node`. Everything that reads the projection was (`/api/resources`, websocket state, `NodesSnapshot`, Patrol and AI tools, connections grouping, reporting, charts). ## Fix Each sighting records the status its source reported (an unexported verdict that replaces the narrower lease marker from #2634), and `aggregateStatus` reads it: - While any source with a verdict is current, only the current sources decide, in the priority order the merge uses (agent, then Proxmox or vSphere, then Docker), so a live agent still keeps a node online when the Proxmox API path to it fails. - Once every source is quiet, an `offline` verdict survives and anything else reads `warning`, as before. - Sightings still report delivery, so health keeps its `telemetry_stale` reason. Availability checks projected onto a target keep counting by delivery, exactly as on main. Giving them a verdict needs each check's own freshness, and that is left to a follow-up (documented in unified-resources.md). ## Verification - Registry tests in `registry_merge_policy_test.go`: live poll reports offline with a silent agent (inside and past its lease), poller-expired node alone and with its agent, stopped and running guests on a quiet poll, JSON round trip, in-memory copy keeping a facet without a verdict, and an aggregation table. The node, poller-expired, guest and round-trip tests fail on main with the bug's exact symptoms; all of #2634's lease tests still pass. - Private mock stack (demo estate, one node held offline on a fresh poll with its agent silent): before, both offline nodes showed amber ("Offline"/"Degraded" pills) and 44 stopped guests counted as Attention; after, both read Offline and the guests moved to Stopped. A full status diff of ~1,780 resources showed no other systematic change. - `go test` for unifiedresources, monitoring, ai, alerts, mock, api, api/resourceapi and api/chartapi passes. |
||
|
rcourtman
|
8ad0cbbe26 |
Move static inline styles to classes the production CSP allows (#2642)
## Why
The production server sends `style-src 'self' 'nonce-...'` with no
`'unsafe-inline'` (`internal/api/security.go`). Solid compiles static
style values, such as `style="..."` or a literal entry of a `style={{
... }}` object, into template HTML. The browser reports each one as a
`style-src-attr` violation. Opening a row's inline detail on Machines or
Docker Images logged eight of them. The HelpIcon popover, the Assistant
typing dots and the alert threshold slider logged their own.
Chromium still applied the styles after Solid cloned the template, so
the visible effect there was console noise. Whether a stricter browser
applies them is not guaranteed. The Vite dev server allows inline
styles, so none of this showed up in development.
## What changed
- 24 drawer tab-panel `overflow-anchor` styles (ResourceDetailDrawer,
GuestDrawer, NodeDrawer, DockerHostDrawer, StoragePoolDetail,
DiskDetail) became the `[overflow-anchor:none]` class.
- The Assistant typing-dot delays became `[animation-delay:...]`
classes. `animate-bounce` already sets the 1s duration.
- ThresholdSlider uses `touch-none`.
- The HelpIcon popover keeps its dynamic position in `style` and takes
`min-w-[200px]` as a class.
- The boot-failure fallback in `index.tsx` is built from DOM nodes and
classes, which also stops the error text being parsed as markup.
- A new guard in `SharedPrimitives.guardrails.test.ts` compiles every
runtime `.tsx` that mentions `style` with the client-build Solid preset.
It fails on any `style` attribute or `<style>` element in the resulting
templates, nested `<template>` content included. The regex scans I used
first missed a mixed style object in HelpIcon; this check caught it.
- A note in `frontend-primitives.md` records the rule.
## Verification
- Production build of this branch embedded in a mock-mode backend that
serves the real nonce CSP, compared with an origin/main build on the
same setup. Checked at 1440x900 and 390x844, plus 1920/2560 for the
desktop thresholds table.
- Before: 8 violations per drawer open on `/standalone/machines` and
`/docker/images`, 3 on Docker host and Proxmox guest drawers, 1 on the
HelpIcon popover, 5 on the typing dots, 1 on the threshold slider.
- After: 0 violations on all of those, plus the Proxmox node drawer,
storage pool detail and disk detail.
- Computed styles match the old ones: `overflow-anchor: none` on every
visible panel, dot delays of 0/0.12/0.24s and 0/0.15/0.3s,
`touch-action: none`, popover `min-width: 200px`.
- Rows opened below the fold (drawer 215 to 351px past it) did not move.
Other drift readings were identical in both builds.
- Vitest: 933 files pass across the affected directories. The only
failure is `agentInstallCommand.test.ts`, which fails on macOS
regardless of this change.
- A new MessageItem test fails on origin/main and passes here.
|
||
|
rcourtman
|
dcd409f9da |
Remove exports orphaned by the unified resource table removal
PR 2612 deleted the never-mounted UnifiedResourceTable, which left
exports whose only production callers were the deleted files. An import
graph from src/index.tsx confirms none of them is reachable now.
infrastructureSelectors.ts served one live caller: workloadSelectors used
computeIOScale for Net I/O and Disk I/O outlier emphasis, by building
fake Resource objects from guests and casting them. The median, MAD and
percentile math now lives in workloadSelectors and reads guest throughput
directly, so the file and its search, sort, grouping and summary-scope
helpers go. Output is unchanged for every input, including null,
negative and non-finite throughput.
getResourcePolicyTableBadges, hasBlockingResourcePolicyPosture and
shouldShowResourceAlternateName only fed the deleted table cards, and
ResourceFacetSummary's maxVisibleBadges and class props (with the +N
overflow chip) were only passed by the deleted rows. The detail drawer,
the component's one live caller, still renders every chip.
Registry ownership, proof entries and lane L10 evidence drop the deleted
file. Contract prose that described the selector pipeline or table-only
behaviour as live is removed or made conditional. Completion Obligation
15 now states the operator-local raw-name search rule that
|
||
|
rcourtman
|
905bae8c28 |
Name disk wearout as life remaining and log disk alerts once (#2640)
Two defects in `Manager.CheckDiskHealth`
(`internal/alerts/disk_health.go`).
**Wearout copy.** The Proxmox disk wearout alert read "SSD has less than
10% life remaining (7% wearout)". `disk.Wearout` is percent life
*remaining*, so "7% wearout" reads as 7% worn and contradicts the first
clause; the Alerts overview card shows the text verbatim. It now reads
"SSD life remaining is 7%", the same wording host-agent SMART wearout
alerts use (`storagehealth` `wearout_low`). The 10% threshold stays in
the alert's `Threshold` field, which the card ("Alert level 10%") and
notifications already render; writing "below 10%" into the sentence
would be false at exactly 10%, which still alerts (below-direction
thresholds match `<=`).
**Per-poll logging.** A failed disk logged `Disk health alert created`
at error, and a worn disk logged at warn, on every evaluation instead of
when the alert opened. With the default 5-minute disk poll that is 288
error lines a day per failed disk, and mock mode repeated it every ~10 s
alert tick. Both now log at error/warn only when
`evaluateCanonicalStatefulAlert` returns the `activated` transition (the
idiom ZFS pool and PMG alerts already use) and at debug on later polls.
Verification:
- `TestCheckDiskHealthLogsAlertOpeningOnce` counts log lines over three
polls of a failed disk at exactly the 10% wearout boundary, then a
recovery and re-failure. It fails on the old code (3 error lines for 3
polls) and with the debug gate mutated to `<`. Logs are captured through
a hook installed in a new package `TestMain`, because assigning
`log.Logger` inside a test races with managers that other tests leave
running.
- Two existing wearout tests now assert the message text.
- `go test -race -shuffle=on ./internal/alerts/` passes.
- A private demo-mode mock stack logged one error per failed disk and
one warn for the worn disk, then debug on every tick (5 error lines vs
130 debug repeats over ~4 minutes). The Alerts overview card read "SSD
life remaining is 7%" at desktop and phone widths.
The alerts contract also said wearout recovery needs readings of "at
least 10%"; a 10% reading still matches, so it now says above 10%.
|