mirror of
https://github.com/rcourtman/Pulse.git
synced 2026-10-23 20:25:56 +00:00
843daf7ae34033febbc132cf270670cf296f4acc
13011 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 rcourtman
|
843daf7ae3 |
Remove dead Patrol assessment and save-failure handoffs (#2682)
Stacked on #2674 (branch `refactor/drop-patrol-context-summary`); this branch contains that commit. ## What Removes the two Patrol-to-Assistant handoff builders in `frontend-modern/src/features/patrol/patrolInvestigationContextModel.ts` that no production code calls: - `buildPatrolAssessmentAssistantHandoff`: its caller, the summary card's Discuss handler, went in |
||
|
rcourtman
|
2eef42c512 |
Keep Off alert defaults off after saving (#2684)
## Summary
Switching a metric Off in a **Global Defaults** row (Alerts >
Thresholds) did not stick. After Save and a reload the metric was back
On at its factory threshold and kept alerting. This affected every
section whose factory threshold is positive: Machines CPU, memory, disk
and disk temperature, Proxmox node temperature, PBS, Docker, Kubernetes,
TrueNAS, vSphere and storage.
The editors stage `-1` for Off, and the save payload passed it through
as `{trigger: -1}`. Backend config normalization reads a negative
default trigger as unset and restores the factory value. Guest and node
CPU, memory and disk escaped only because nothing normalizes them.
- `buildAlertsConfigurationPayload` now writes every Off default as
`{trigger: 0, clear: 0}`, the value the engine and normalization both
read as off. Per-resource overrides are serialized separately and keep
`-1`, which the override path stores as written. A `-1` already stored
on a guest or node default becomes `0` on the next save.
- The Global Defaults **Custom** badge now treats `-1` and `0` as the
same Off value. Without that, a fresh install showed Custom on
Kubernetes, TrueNAS Systems and vSphere, and VMs & Containers turned
Custom after any save, though nothing had changed. An unset factory
value stays distinct from Off: an unset TrueNAS disk temperature follows
Disk temperature by type.
- A row's custom-threshold marker still compares raw values. It drives
the row's revert control and override removal, so a stored `-1` override
must stay visible against an Off default. A test pins that.
- The alerts contract no longer claims Off is written as `-1`
everywhere.
## Verification
- New payload test: every section's Off default is written as `{0, 0}`
and reads back Off. It fails on `main` with `guestDefaults.cpu: expected
{ trigger: -1 }`.
- New backend test: a `{0, 0}` default stays off through `UpdateConfig`
in every section, and a host at 99% raises no CPU, memory or disk alert
when the agent defaults are off. Both fail when the defaults are written
as `-1`.
- Live on a private mock stack. On `main`, Proxmox CPU and Temp switched
Off saved as `{-1}` and `{80, 75}`, and after reload Temp read On. With
the fix:
- Machines CPU at 1% raised 8 alerts. Switching it Off stored `{0, 0}`
and resolved all 8.
- Proxmox Temp Off resolved the node temperature alert.
- Nothing re-fired over two minutes.
- Values stayed Off after a backend restart and a reload. Machines,
Proxmox, Kubernetes, TrueNAS and vSphere were each checked at desktop or
phone width.
- Custom badge on a fresh install: no Proxmox, Kubernetes, TrueNAS or
vSphere group reads Custom, and after an unrelated save only the group
actually changed does.
|
||
|
rcourtman
|
dace9295c6 |
Remove exports orphaned by the summary component deletions (#2671)
Stacked on #2662 (its commits are ancestors of this branch, so this PR shows them until #2662 merges). The May cleanup deleted InfrastructureSummary, StorageSummary, the shared summary table and the SummaryMetricCard, InteractiveSparkline, DensityMap and SummaryPanel primitives, but left the helpers only they called. This removes them: - `getMetricsChartKeyCandidatesFromResource` (utils/agentResources.ts) - `getNormalizedIdentityLookupVariants` and `getAgentLikeIdentityAliases` (utils/resourceIdentity.ts) - `getPBSStatusIndicator` (utils/status.ts) - `useSummaryContextualFocusState` (components/shared/contextualFocus.ts), plus `filterSummarySeriesByGroupScope`, `resolveSummaryCardInteractionState` and `SummaryCardInteractionState` that only it used - the deprecated `ChartsAPI.getInfrastructureCharts` delegate; its branch tests now exercise `getInfrastructureSummaryCharts` Proof of deadness: a TypeScript import/export graph from `src/index.tsx` finds no production importer of any removed symbol, and a repo-wide grep finds no remaining reference. The production build differs from its parent only in the shared chunk that carries ChartsAPI, which loses the delegate's static method (99 bytes); after normalising chunk hashes, SRI hashes and minified identifiers every other file is identical. Tests: `src/utils`, `src/components/shared`, `src/api`, `src/features`, `src/hooks`, `src/__tests__`, Workloads, Storage and Infrastructure suites pass (only the known Mac-only `agentInstallCommand` failures). Contracts: frontend-primitives, storage-recovery and performance-and-scalability still described the deleted summary cards, sparkline, density map, summary panel, sticky summary section and Jump to row button as live owners, and credited contextualFocus.ts with the removed hook's series filtering. Those clauses now name the live owners (summaryCardInteraction.ts behind summaryTableFocus.ts, the storage summary cache behind the pool Growth column, TooltipPortal for the mini sparkline and backup activity chart tooltips), become conditional on a summary strip returning, or go. api-contracts and unified-resources record the removed chart client and chart-key helper. Browser pass (embedded production build, demo mock stack, 1440x900 and 390x844): Proxmox node-to-guests reveal, guest row hover emphasis and inline detail reveal, storage Growth column, pool row hover and inline detail, pool-group hover preview, pin and Escape clear. Receipt: `frontend-modern/browser-verification/232e5c23490f8cd4.json`. |
||
|
rcourtman
|
7eb32f3181 |
Keep a low alert default's clear below its trigger (#2678)
## Problem
Setting an alert global default to a positive trigger of 5 or less, for
example Machines CPU at 1% in Alerts > Thresholds, stored
`agentDefaults.cpu` as `{trigger: 1, clear: 75}`, a clear far above its
trigger.
The thresholds page sends `clear = max(0, trigger - 5)`, which is 0 for
those triggers. `NormalizeAgentDefaults` derived `trigger - 5`, found it
non-positive, and fell back to the factory clear (75/80/85).
`ValidateHysteresisThresholds` never checked agent CPU, memory or disk,
so the pair survived. PBS CPU/memory, node temperature and
`internal/config/persistence.go` had the same fallback. On the save
path, validation repaired PBS and node temperature, but
`LoadAlertConfig` still turned a stored node temperature clear of 0 into
75; at startup `UpdateConfig` repaired that in memory.
Per-type `DiskFillByType` / `DiskTempByType` entries were worse: any
clear ≤0 reset the whole pair. A SATA disk temperature trigger of 4 from
the thresholds page was stored as 55, and an API-written fill entry of
`{88, 0}` became `{90, 85}`.
## What a user saw
- **Firing and resolving did not change for these pairs.**
`buildCanonicalMetricSpec` drops a clear at or above the trigger, and
the reducer then clears at the trigger. That held both before and after
this change. There was no flapping and no alert that never re-armed.
- `/api/alerts/config` and `alerts.json` served the factory clear above
the trigger.
- Firing agent alerts carried it as `metadata.clearThreshold`: 75 on a
1% CPU alert, and 85 on a disk alert still firing at 81.8%.
- A per-type disk trigger of 1–5 was silently replaced by the type
default.
- Machines metric bars: `resolveThreshold` in `metricThresholds.ts`
clamps the warning level to `min(clear, trigger)`, so a 1–5% agent
trigger had no warning band. Readings below the trigger, including 0,
now show amber instead of green, which matches guest, node, Docker and
Kubernetes bars at the same setting. This one is read from the code; I
did not exercise it in a browser.
## Fix
- PBS, node temperature, agent and per-type disk defaults now use
`normalizeThresholdPointer`, the normalizer Kubernetes, TrueNAS and
vSphere already use. A missing clear sits five points under the trigger,
floored at 0. A clear at or above the trigger is repaired. Zero still
means off for global defaults, and nil or negative still takes the
factory default. For the canonical per-type keys (the types the page
edits), a non-positive trigger still resets to the type default, because
the page cannot represent a per-type Off; other per-type keys are stored
as written, as before.
- `internal/config/persistence.go` drops its duplicate
agent/node-temperature and storage helpers and calls the same normalizer
through a new `NormalizeHysteresisThreshold`. A config saved before this
fix loads repaired.
- Agent CPU/memory/disk/disk temperature are added to
`ValidateHysteresisThresholds`, as every other family already is.
- One deliberate behaviour change for invalid pairs above the margin: an
API-written `{50, 80}` is repaired to `{50, 45}`, so it now has a real
recovery band where it used to clear at the trigger. This matches what
`EnsureValidHysteresis` already does for every other family.
- An alerts contract clause records the rule.
## Verification
- `internal/alerts/config/normalize_low_trigger_clear_test.go` runs
every default family and both per-type maps through the `UpdateConfig`
normalizer chain.
- `TestUpdateConfigKeepsLowTriggerClearBelowTrigger` pins the saved
config and a firing alert's `clearThreshold`.
- `persistence_test.go` checks the written `alerts.json` itself as well
as load repair.
- All new tests fail against the previous normalizers. Coverage cases
that pinned the old fallback (including one labelled "suspected source
bug") are updated.
- `go test ./internal/alerts/... ./internal/config/ ./internal/ai/`
passes.
- Live, on private mock stacks (baseline binary vs this branch): I saved
the payload the thresholds page sends, then read back
`/api/alerts/config`.
- Baseline: agent `{1,75}`, `{3,80}`, `{5,85}`; `diskTempByType.sata`
`{55,50}` for a 4°C trigger; firing agent alerts reported
`clearThreshold` 75/80/85.
- This branch: every pair `{t, 0}`, `diskTempByType.sata` `{4, 0}`, and
alerts reported 1/3/5.
- After a restart, this branch also repaired a hand-written
`alerts.json` holding `{1,75}` and node temperature `{2,75}`.
|
||
|
rcourtman
|
5c55197900 |
Remove test-only Patrol context summary helpers (#2674)
Removes two exports from
`frontend-modern/src/features/patrol/patrolInvestigationContextModel.ts`
that only tests still called, and corrects the contract prose that
described the removed Patrol supporting-context surface as live.
## Why
`buildPatrolInvestigationContextSummary` (with its input type) and
`selectPatrolSupportingRecentChanges` lost their last production caller
in
|
||
|
rcourtman
|
22b8b995ca |
Read the shared wall clock on Alerts overview cards (#2673)
The overview hook kept its own minute signal and handed its value to each card as "now". A card that mounted between ticks measured from up to a minute earlier: an alert raised 70 seconds ago read "this minute", a reading already past the 10-minute stale cut-off led the card as the live one, a just-raised alert stayed out of Triggered (24h) until the next tick, and the stale flip and 24h drop landed up to a minute late. The card now reads useRelativeTimeNow for its started age and the live reading's stale cut-off, and the hook reads it for the 24h count, so they measure from the wall clock and move with the shared 30-second tick. The minute interval stays only for the delivery diagnoses refresh, a server read that should not double its rate. |
||
|
rcourtman
|
2d6a2937fc |
Keep platform metric alerts through Proxmox node cleanup (#2672)
`CleanupAlertsForNodes` runs on every Proxmox node poll
(`monitor_pve_storage.go`) and on every mock alert tick. It removes
alerts whose `Node` is not a current Proxmox node or PBS instance, and
keeps the types `shouldPreserveAlertOutsideNodeCleanup` lists.
Kubernetes, TrueNAS and vSphere metric alerts carry a cluster or
platform host as `Node`, and a display label (`TrueNAS Disk`,
`Kubernetes Pod`, `vSphere VM`) in `metadata.resourceType`. The preserve
list only had the bare words `truenas`, `kubernetes`, `k8s` and
`vmware`, so none of these labels matched. In an estate with Proxmox
plus one of those platforms:
- every Proxmox poll deleted these alerts without resolving them;
- their reducer incident stayed firing, so the next unified evaluation
re-created each one and handed it to the alert callback again as a new
alert.
## Change
`shouldPreserveAlertOutsideNodeCleanup` keeps an alert that carries
`metadata.platformType` (set on every unified Kubernetes/TrueNAS/vSphere
metric alert), or whose label maps through `CanonicalAlertResourceType`
to one of those platform types. The label path covers alerts persisted
before `platformType` existed. A guest alert whose Proxmox node left the
estate is still removed.
## Evidence
- Unit probe on main: a TrueNAS disk temperature alert, a Kubernetes pod
CPU alert and a vSphere VM CPU alert were all deleted by
`CleanupAlertsForNodes({"pve1": true})`. Over three
cleanup-then-evaluate cycles the TrueNAS alert went to the alert
callback 3 times; with this change, 0 times and it stays active.
- Private mock stack: TrueNAS disk temperature alerts kept vanishing
between mock alert ticks, while the disks kept their readings in
`/api/resources`. With this change they stay open.
## Tests
`TestCleanupAlertsForNodesKeepsPlatformMetricAlerts` (in
`internal/alerts/alerts_test.go`) raises a TrueNAS disk, a TrueNAS pool,
a Kubernetes pod and a vSphere VM alert, then runs the cleanup three
ways: as raised, with only `platformType` (label unrecognised), and with
only the label (`platformType` removed).
`TestCleanupAlertsForNodesStillRemovesMissingProxmoxNodeAlerts` keeps
the removal of a guest alert on a removed Proxmox node. Each preserve
condition was mutation-checked: removing either one fails the first
test.
|
||
|
rcourtman
|
e30b44ea8f |
Read a silent in-guest agent's guest drawer readings as last known (#2669)
A Pulse agent linked into a VM keeps its last report on the VM resource after it stops reporting: the registry copies the agent's whole facet, `agent.stale` included, onto the guest. The guest drawer never read that flag. A quiet agent's Guest RAID kept its live colours and "Rebuild: 42% · 118M/sec", its network rows and version read as current, and nothing said the agent had gone. ## Change - `useWorkloads` maps `agent.stale` onto the workload as `agentStale`, and sets `disksFromAgent` when Proxmox reported no guest filesystems and the agent's own `disks` filled in. Filesystem rows are the only agent reading with a second source. - When the agent is stale, the guest drawer: - passes `HOST_AGENT_STOPPED_REPORTING_REASON` to `RaidCard` (`lastKnownReason`, #2652); - puts a status line over muted Network rows; - marks the Pulse Agent row "· stopped reporting"; - reads agent-supplied filesystem rows through the existing `last-known` disk read state. - Agent-supplied filesystem rows now follow the agent's freshness instead of the Proxmox guest-read reason. Before, a VM with a **live** linked agent but no qemu-guest-agent showed the agent's filesystems as "Usage unavailable · ?/328 GB" with advice to install qemu-guest-agent. A stopped guest still reads those rows as unavailable, whether its status or the Proxmox `vm-stopped` reason says so. Proxmox's own guest filesystems keep their read state. ## Verification - Private mock stack built from `main` plus this change. Two host agents were held past their reporting lease and two left live, then each was linked into a VM through `POST /api/resources/{vm}/link`. The two silent VMs serve `agent.stale: true` with the agent's raid, disks and interfaces; the two live ones carry no stale flag. - Playwright at 1440 and 390: - silent VMs read as described above; - the live VM with agent-only filesystems shows usage with bars; - the live control is unchanged; - no horizontal overflow. - New tests in `GuestDrawer.test.tsx` cover: - silent agent; - live agent with Proxmox agent-not-running; - Proxmox filesystems under a silent agent; - stopped guest, with and without the reason; - stale flip in both directions on an open drawer. The new tests fail against the old drawer. `useWorkloads.test.ts` asserts source selection with distinct mountpoints. - Frontend suites over Workloads, hooks, utils, shared, platformPage and Infrastructure pass, apart from the Mac-only PTY tests in `agentInstallCommand.test.ts`. - WorkloadsSurface grows by 0.22 kB gzip (64.85 kB against a ~71.9 kB cap), and the chunk set is unchanged. ## Not covered - The workload table's disk cell and the drawer History still read the aggregate disk metric with the Proxmox reason. - The agent's OS, its addresses in the identity rows, and the lazily loaded Physical Disks card's health and wearout carry no stale cue yet. |
||
|
rcourtman
|
9bf72eb407 | Merge remote-tracking branch 'origin/main' into fix/global-default-off | ||
|
rcourtman
|
564ea41a6e |
Keep Off alert defaults off after saving
Switching a Global Defaults metric Off on the thresholds page staged -1, and the save payload passed it through as trigger -1. Alert config normalization reads a negative default trigger as unset and restores the factory threshold, so every Off default with a positive factory value came back On after save and reload: agent CPU, memory, disk and disk temperature, Proxmox node temperature, PBS, Docker, Kubernetes, TrueNAS, vSphere and storage. Guest and node CPU, memory and disk escaped only because nothing normalizes them. The payload builder now writes every Off default as trigger 0, which the engine and normalization both read as off. Overrides keep -1, which the override path stores as written. A -1 already stored on a guest or node default becomes 0 on the next save. With -1 and 0 both live Off values, the Global Defaults Custom badge compares Off values as equal, so the factory -1 I/O defaults no longer read Custom against a saved 0 and tell users they changed something they had not. A row's custom marker keeps comparing raw values: it drives the row's revert control and override removal, so a stored -1 override must stay visible against an Off default. Contract-Neutral: Global Defaults Off now persists; the Off wire value and Custom badge rule are recorded in the alerts contract, and frontend-primitives gains no rule |
||
|
rcourtman
|
7432995fe8 |
Stop websocket upgrade headers exempting demo-mode writes (#2666)
`DemoModeMiddleware` let any request carrying `Upgrade: websocket`
through, whatever its method or path. On a `DEMO_MODE` instance a
`POST`/`PUT`/`PATCH`/`DELETE` with that header reached its handler, and
the read-only guard did nothing.
## How exposed the public demo was
- The public demo (`demo.pulserelay.pro`, v6.5.0) runs `DEMO_MODE=true`
with `PULSE_AUTH_USER=demo`, and its login page signs visitors in as
demo/demo. No proxy auth is configured. Caddy is a plain `reverse_proxy`
with no method filtering, and it forwards `Connection: Upgrade` and
`Upgrade: websocket` unchanged.
- That session is the configured admin.
`sessionUserHasSettingsAdminPrivileges` matches `cfg.AuthUser`, so
`RequireAdmin` accepts it, and session requests skip token scope checks.
Nothing in the handlers checks demo mode on its own, so this middleware
was the only write guard.
- Anyone using the public credentials could therefore make any admin
write by adding the two headers. That covers switching the demo out of
mock mode (`POST /api/system/mock-mode`), changing system settings,
creating organisations, importing config, changing the password and
adding notification targets. A CSRF token was needed, but the attacker
gets one from their own login.
- On the old code, the new contract test reproduces this end to end. A
demo/demo session's mock-mode POST with upgrade headers returned 200 and
turned mock mode off.
- I found no sign it was used. Over the last 30 days the demo journal
shows every "mock mode disabled" line paired with a process restart (new
PID). Config file mtimes match deploys. 5,782 ordinary writes were
blocked over the same period. Caddy keeps no access log for the demo, so
this is the absence of evidence, not proof.
## Change
The exemption is deleted rather than narrowed. RFC 6455 handshakes are
`GET` requests, gorilla's `Upgrader` refuses any other method, and `GET`
already passes the read-only branch above it. A narrowed `GET`-only
exemption would never run. `/ws` and `/api/agent/ws` keep connecting,
and the request method alone decides.
## Tests
- `TestContract_DemoModeUpgradeHeadersDoNotExemptWrites` (registered
api-contracts proof) runs against the full router. It signs in as the
configured demo admin and checks that the session passes `RequireAdmin`
for a read. A mock-mode POST carrying `Connection: Upgrade` and
`Upgrade: websocket` now gets the demo 403 and leaves mock mode on, and
a real `/ws` handshake from the same session still upgrades.
- `demo_middleware_test.go`: the table row that asserted the bypass now
expects 403. POST/PUT/PATCH/DELETE with handshake headers to mock-mode,
settings, orgs, `/ws` and `/api/agent/ws` all get 403. Real gorilla
handshakes to `/ws` and `/api/agent/ws` through the middleware still
upgrade and echo.
- Negative control: run against the old middleware via `go test
-overlay`, the new contract test and the upgrade-header test fail, and
the handshake test passes on both versions.
## Not changed here
- `auth.go` `isWebSocketUpgrade` and `configapi`
`explicitAPITokenFromRequest` still accept a `?token=` query credential
on any request with `Upgrade: websocket`. A valid, scoped token is still
required, so no access is gained, but the credential can travel in a
URL. The `ErrorHandler` skip in `middleware.go` only bypasses recovery
and metrics.
- Older and separate:
`/api/ai/sessions/{id}/{abort,summarize,fork,undo,redo}` change state on
`GET`, which the demo guard admits as reads. Tracked separately.
|
||
|
rcourtman
|
cb5ffc2efe |
Remove dead Patrol assessment and save-failure handoffs
buildPatrolAssessmentAssistantHandoff lost its last caller when the Patrol summary card's Discuss handler was removed in |
||
|
rcourtman
|
75263ea589 |
Refuse a seeded disk slot mapping when the disk was replaced (#2667)
`ingestRecord` reuses an existing source-key mapping before any identity matching. For physical disks it checked only the type and parent scope. Proxmox keys a disk by its slot (`ProxmoxPhysicalDiskSourceID`, e.g. `pve1-node1--dev-sdb`), so in a registry seeded from persisted unified resources, a disk swapped into the slot mapped straight back to the previous disk's canonical resource. Where an agent report had been merged in, `mergeInto` also kept the old disk's serial, WWN, temperature and FAILED health. This was found while reviewing #2658, and its contract note said the seeded mapping was not covered. **Change.** A Proxmox disk observation now refuses the mapping when the two disks carry hardware identities naming different disks (`diskinventory.HardwareIdentityConflict` from #2658): - Only fields reported on both sides count. A missing serial or WWN, including Proxmox's literal `unknown`, keeps the mapping. - Differing serials do not count for SAS, because Proxmox may report the SAS address there (#1595). - They also do not count when the mapped disk was merged with an agent report and Proxmox did not read the drive's own serial. This is the same rule monitoring's `hostAgentSMARTSerialComparable` applies: NVMe on both sides, or agent SATA under Proxmox vendor `ATA`, and never a Proxmox `usb` disk. - The refused key falls through to the linked-disk join, `findMatch` and `chooseNewID`, but none of them may merge it back into the refused disk. An ID landing on it is replaced with the key's source-specific ID. **Reachability.** No current caller replays Proxmox disk records over a seeded mapping. The monitor rebuilds a fresh registry each poll. The resource API, host-continuity and Patrol registries seeded from unified resources do not replay Proxmox disk records. This closes the registry contract rather than a reported symptom. **Proof.** `TestRegistrySeededSlotMappingRefusesReplacedDisk` (in `internal/monitoring/issue1595_collection_trust_test.go`) builds the previous disk through `IngestSnapshot` with fixtures shaped like `monitor_pve`'s output, rehydrates with `IngestResources(List())`, polls Proxmox, then rehydrates a second time. - 5 replacement cases: SATA with and without a new WWN, SAS, a USB-enclosure swap ingested Proxmox-first, and a disk Proxmox alone reports. - 8 same-disk controls: re-ingested, no serial, a SAS address serial, SAS without a WWN, a USB bridge serial, a bridge serial under vendor `ATA`, a SCSI designator, and a SCSI designator from an agent without collection status. Negative controls via `go test -overlay` each fail only their intended cases: the whole fix removed, the refused-disk exclusion removed, serial comparability always true or always false, the Proxmox-only branch removed, a collection-status provenance rule instead, and the USB veto removed. The combined tree of main, #2660 and this change also passes `internal/unifiedresources` and the monitoring disk tests. **Residuals (in the unified-resources contract).** - Drives that share a serial they do not own, such as a USB bridge's, still read as one disk, as they do in identity matching. - The refused disk keeps its Proxmox facet until a rebuild, so the slot's PVE alert reference resolves as ambiguous rather than to either disk. |
||
|
rcourtman
|
080ace2e97 |
Show the outdated sensor setup notice on every Proxmox tab (#2668)
Since the per-tab Proxmox queries landed, only the Storage tab loads `physical_disk` rows. So the outdated sensor setup notice, which judged nodes from the active tab's disk rows, never showed on Overview. The registry now derives the verdict on each Proxmox node (`proxmox.sensorSetupOutdated`, an explicit true or false on every node) right after child counts resolve each disk's canonical parent. A node is flagged when its temperature payload is available and in the legacy `sensors -j` format and a SATA/SAS disk under it has no reading collected now (`diskinventory.TemperatureCollected`). The page lists flagged nodes from the rows Overview, Storage, Replication and Backups already load, so no tab hydrates disks for it. Rule fixes that came with the move: - Proxmox's disk inventory types SATA/SAS disks as `hdd`/`ssd`, so the old `sata`/`sas`-only check never matched a node without a host agent. - A retained reading no longer counts as current (carried over from #2659's frontend rule). - The verdict is sent as explicit false, because the browser's field-by-field facet merge would otherwise keep an earlier true after the condition clears. Stacked on #2659 (shares its `sensorSetup.ts` edit, which this replaces). Verified on a private mock stack at 1440px and 390px: - Overview, Storage, Replication and Backups show the notice for legacy-format nodes with an `hdd` disk without a reading, a retained SATA reading, and a SATA disk without a reading. - Overview makes no `physical_disk` request. - With `main`'s frontend swapped in, Overview showed nothing and Storage named one node of three. - A node whose disk starts reporting drops out on the next websocket update without a reload. |
||
|
rcourtman
|
cfa026249f |
Switch every running tenant monitor when mock mode flips (#2665)
Follow-up to #2656 (the per-monitor mock-mode epoch fence), which documented this gap as not covered. ## Problem Mock mode is one process-wide flag (`mock.SetEnabled`), but `Monitor.SetMockMode` ended the fence epoch, cleared the alerts, forgot the fixture agents and reset the state of only the monitor it was called on. The server runs one `Monitor` per organization (`MultiTenantMonitor.GetMonitor` → `New` + `Start`), and each runs its own mock alert pass into its own alert manager. A switch made through one monitor (the mock-mode API uses the requesting tenant's monitor; the demo-fixture licence sync uses the default one) left every other running monitor holding what it raised in the mode the process had left: fixture alerts and fixture-agent node links in live mode, live alerts in mock mode, until the 24-hour stale sweep. ## Fix - `internal/monitoring/mock_mode_switch.go`: a process-wide set of running monitors, guarded by the existing `mockModeSwitchMu`. `Start` joins it and leaves when its loop returns. While joining it picks the mock metrics sampler or discovery under the lock (now `startModeRuntime`), so no switch lands between reading the mode and starting the runtime for it. - `SetMockMode` flips the flag once, then for the receiver and every running monitor: ends the epoch, waits for admitted calls, clears alerts, forgets fixture agents (when leaving mock mode) and resets state (`endMockModeEpoch`). Only after that does it restart the sampler, polls and discovery of each monitor in the set (`resumeAfterMockModeSwitch`). A monitor that has not joined yet gets its runtime from its own `Start`. - Each monitor records the mode its alerts and state belong to (`mockModeAligned`, set in `New` from the same read that decides whether persisted alerts are restored) and the number of switches made so far (`mockModeSwitchesSeen`). If it missed switches between `New` and `Start`, it leaves what it holds before joining, even after a round trip that brought the flag back. A call for the mode the flag already holds still moves any reached monitor that belongs to the other mode, and leaves the rest alone. Callers are unchanged and no API surface moves. ## Is this reachable in release builds? Barely. In a `release` build `mockruntime.ValidateEnablement` refuses mock fixtures unless `SetReleaseFixturesAuthorized(true)` was called. Its only caller is `syncReleaseDemoFixtureRuntime`, which requires `DemoMode` plus the `demo_fixtures` entitlement and runs only for the default organization. Several tenant monitors in mock mode would also need more than one organization's monitor running on that demo instance. Demo mode's read-only middleware normally blocks creating organizations and the mock-mode API's POST. It does forward any request carrying `Upgrade: websocket` whatever its method; that bypass predates this change and is being fixed separately. So in practice this is a dev/mock-stack problem, and the fix stays inside the monitoring package. ## Tests New tests in `internal/monitoring/monitor_mock_alerts_test.go`: - `TestLeavingMockModeClearsEveryRunningTenantMonitor` and `TestEnteringMockModeClearsEveryRunningTenantMonitor` start the default and one other tenant monitor through `MultiTenantMonitor`, as the server does, and wait until both have joined. They raise alerts in both, switch through one, and check both. - `TestMonitorStartedAfterAMockModeSwitchLeavesTheModeItWasBuiltIn` covers a switch that lands between `New` and `Start`. - `TestMonitorStartedAfterAMockModeRoundTripDropsWhatItRaisedMeanwhile` covers two switches in that window, with fixture alerts raised in between. - `TestSetMockModeAlignsAMonitorWhoseModeTheFlagAlreadyHolds` covers a call for the mode the flag already holds. - `TestMockModeSwitchLeavesAnUnjoinedMonitorsRuntimeToItsStart` covers a switch reaching a monitor that has not joined. One more is in `internal/monitoring/monitor_host_agent_removal_lifecycle_test.go`: `TestLeavingMockModeReleasesFixtureAgentNodeLinksOnEveryRunningMonitor`. It switches through the default monitor and checks that the other tenant's fixture node is no longer suppressed by a fixture agent's leftover node link. Negative controls (`go test -overlay`): | Control | Failing tests | What happened | |---|---|---| | Fence-PR runtime | Leaving, entering, node-link | Default monitor kept 69–104 fixture alerts (varies by run) and 69 fixture agents; org-b kept its live alert; org-b's `mock-cluster-1-pve1` stayed suppressed | | Set kept, switch reaches only the receiver | Same three | Same failures | | No realign at join | `TestMonitorStartedAfterAMockModeSwitchLeavesTheModeItWasBuiltIn` | | | Join compares only the mode, not the switch count | `TestMonitorStartedAfterAMockModeRoundTripDropsWhatItRaisedMeanwhile` | Started in live mode with the fixture node's memory alert | | Plain early return when the flag already matches | `TestSetMockModeAlignsAMonitorWhoseModeTheFlagAlreadyHolds` | | | Resume ignores set membership | `TestMockModeSwitchLeavesAnUnjoinedMonitorsRuntimeToItsStart` | | Every control failed only the tests it targets. Also run: the full `internal/monitoring` package; `-race -count=3` on the new tests plus the existing `TestLeavingMockMode*` and `TestMockModeFence*`; and the mock-mode tests in `internal/api` and `internal/api/configapi`. |
||
|
rcourtman
|
7a026891c6 |
Keep a low alert default's clear below its trigger
Setting a global default to a positive trigger of 5 or less (Machines CPU
at 1%) stored agentDefaults.cpu as {trigger: 1, clear: 75}. The thresholds
page sends clear max(0, trigger - 5), which is 0 there. NormalizeAgentDefaults
then derived trigger - 5, found it non-positive and fell back to the
factory clear, and ValidateHysteresisThresholds never checked agent
CPU/memory/disk. PBS, node temperature and the persistence normalizer had
the same fallback (validation repaired PBS and node temperature on save).
Per-type disk fill and disk temperature entries were worse: a clear of 0
reset the whole pair, so a SATA disk temperature trigger of 4 was stored
as 55.
Alert firing and resolving did not change for these pairs:
buildCanonicalMetricSpec drops a clear at or above the trigger and the
reducer then clears at the trigger. The wrong level showed in
/api/alerts/config and alerts.json, and firing agent alerts reported it as
clearThreshold (75 on a 1% CPU alert, 85 on a disk alert still firing at
81.8%).
PBS, node temperature, agent and per-type disk defaults now use
normalizeThresholdPointer, which Kubernetes, TrueNAS and vSphere already
use: a missing clear sits five points under the trigger floored at 0, and
a clear at or above the trigger is repaired, so an API-written {50, 80}
now gets a real recovery band at 45. A non-positive per-type trigger still
resets to the type default. Persistence calls the same normalizer through
a new NormalizeHysteresisThreshold instead of its own copies, so configs
saved before this load repaired.
|
||
|
rcourtman
|
900d620abd |
Rank a confirmed availability outage first in the summary check (#2664)
A machine with several availability checks shows one summary check, and its health reads only that summary. The summary ranked every observed failure equally and broke ties by target id. So a check that had failed once, below its threshold, could hide a later-sorting check with a confirmed outage. The outage check's own row carried `availability_unreachable` while the machine it monitors read healthy. - `availabilityCheckPriority` now ranks a confirmed outage first, then any other observed failure, then unchecked/ambiguous/unresolved, then passing. Ties still go to the first target id. - `availabilityOutageConfirmed` is the one confirmation rule, shared by the ranking and `EvaluateResourceHealth`. It mirrors the poller's incident gate: observed failures at the threshold, an unavailable aggregate when one is reported, and no indeterminate outcome. A probe agent that stops reporting keeps its old failure count but confirms nothing, on its own row or on the target. - Tests in `availability_link_test.go` go through the registry projection path: a confirmed outage behind an earlier unconfirmed failure (and its recovery), and a silent probe agent's stale failure count. Both fail without the change. `health_test.go`'s legacy fixture gains the `LastChecked` every producer sets. - The `unified-resources.md` contract describes the summary ranking and the shared rule. Drawers already list every check from `availabilityChecks`, and availability incidents live on each check's own row, so alert synthesis is unchanged. |
||
|
rcourtman
|
0f6d1e4180 |
Stop a failing availability check keeping a node online (#2663)
Follows #2643, which made the stale pass read each source's own verdict but left availability checks counting by delivery. ## Problem Once every source with a verdict had gone quiet, any current availability sighting kept its target online. A node that `preserveOrExpireNodes` expired stayed online while its HTTP check failed, and a quiet failing check lifted that node's offline verdict to warning. One sighting stands for every check on a target and carries whichever check was projected last, so a failing check that had just run also vouched for another check's old pass. ## Change - **Registry verdict** (`aggregateStatus`, `availabilityChecksProveOnline`): each check on a monitored resource is judged by its own evidence window (`Evidence.ValidUntil`). An enabled, passing check with current evidence proves the resource answers and reads online once one of the resource's own sources has gone quiet and none that is current has a verdict. Every other check abstains, current or quiet. A failing probe never makes a guest count as stopped (workloads summary, Proxmox page counts). A check's own row keeps the ordinary rules. - **Ownership**: only a target's own quiet sources hand the stale pass its status. Projecting, re-running or retargeting a check re-applies the stale pass's rule to targets it owns, because the resources API replays checks after its stale pass. A manual link judges the merged resource by the checks it carries. - **Poller evidence** (`internal/monitoring`): - A local location beside other locations ages out like a silent remote path ("no recent local check"). - An available check's evidence lasts only as long as its newest reachable path. - After an entitlement lapse, an adopted remote result keeps the earlier of its agent check time and the server receipt time. - A multi-location aggregate is never adopted as a local observation; the target reads as never checked until the local poller runs. Contracts: `unified-resources.md` ("Status follows each source's own verdict") replaces the residual paragraph, and `monitoring.md` covers the multi-location rules. ## Verification - New tests in `registry_test.go`, `registry_merge_policy_test.go` and `availability_probe_agent_test.go`. Each fails against the base runtime via `go test -overlay`, including both projection orders. - Full `internal/unifiedresources` and `internal/monitoring` suites pass, plus the availability, connection, workload and resource-state tests in `internal/api`, `resourceapi` and `chartapi`. |
||
|
rcourtman
|
d67bef7519 | Merge remote-tracking branch 'origin/main' into refactor/drop-patrol-context-summary | ||
|
rcourtman
|
6a5b7d04c0 |
Remove test-only Patrol context summary helpers
buildPatrolInvestigationContextSummary and selectPatrolSupportingRecentChanges
lost their last production caller when the Patrol page's supporting-context
block, with its Details toggle for recent changes, learned correlations and
policy coverage, was removed in
|
||
|
rcourtman
|
732bb6856a | Merge remote-tracking branch 'origin/main' into refactor/drop-summary-export-orphans | ||
|
rcourtman
|
8676ec687f |
Read the shared wall clock on Alerts overview cards
The overview hook kept its own minute signal and handed its value to each card as "now". A card that mounted between ticks measured from up to a minute earlier: an alert raised 70 seconds ago read "this minute", a reading already past the 10-minute stale cut-off led the card as the live one, a just-raised alert stayed out of Triggered (24h) until the next tick, and the stale flip and 24h drop landed up to a minute late. The card now reads useRelativeTimeNow for its started age and the live reading's stale cut-off, and the hook reads it for the 24h count, so they measure from the wall clock and move with the shared 30-second tick. The minute interval stays only for the delivery diagnoses refresh, a server read that should not double its rate. |
||
|
rcourtman
|
bdf0f76b7b |
Show retained disk temperatures as last known in drawers and tables (#2659)
Normalization keeps a disk's last temperature when the current observation did not collect one, such as a disk in standby or a host agent past its reporting lease. #2603 taught the Storage presenters to show that value as last known. Four surfaces still read the raw number as a current reading. Each now goes through the canonical `getPhysicalDiskTemperaturePresentation` / `isPhysicalDiskTemperatureCurrent` instead of adding a predicate. - **TrueNAS disk drawer** (`resourceDetailDrawerTrueNASModel.ts`): the Temperature row took the 55°C warning tone from a retained reading, and the summary listed it as a reading. It now reads "61°C (last known)" with the reason as title and no tone, and the summary leaves it out. A current reading keeps the existing tone. Replacing the fixed 55 with the alert policy is a separate follow-up. - **Drawer History fallback** (`resourceDetailDrawerMetricsHistoryModel.ts`): with no stored points, a retained reading was labelled "current". It now goes through the deferred-metric slot, so the legend says "last known" and the reason shows above the chart. A disk's value is read from `physicalDisk` beside its collection state. Websocket rows carry no top-level `temperature` for a disk, so on the TrueNAS page the fallback had shown "-" even for a current disk. - **TrueNAS storage table** (`TrueNASStorageTopologyTable.tsx`): the Temp cell showed a retained reading plainly, and hottest-first sorted it to the top of its group. It now renders muted with its reason, and the sort ranks it as no reading. - **Proxmox outdated sensor setup notice** (`sensorSetup.ts`): a retained reading counted as a temperature arriving, so the notice could hide for a SATA disk Pulse cannot read now. Verification: a regression test per surface fails on main and passes here. A private mock stack with a local-only overlay stamped one TrueNAS disk and one pve3 SATA disk with an unavailable temperature state ("host agent stopped reporting") and marked pve3's sensor payload legacy. Playwright at 1440×900 and 390×844 compared the change against the same session with main's runtime files. The receipt is `frontend-modern/browser-verification/294409f678589ef2.json`. |
||
|
rcourtman
|
7a8376c055 |
Keep platform metric alerts through Proxmox node cleanup
CleanupAlertsForNodes runs on every Proxmox node poll and removes alerts whose node is not a current Proxmox node or PBS instance. Kubernetes, TrueNAS and vSphere metric alerts carry a cluster or platform host as their node and a display label such as "TrueNAS Disk" in metadata.resourceType, which the preserve list's platform words never matched. So in an estate with Proxmox and one of those platforms, every poll deleted these alerts silently while their reducer incident stayed firing, and the next evaluation re-created each one and handed it to the alert callback again as a new alert. Keep alerts that carry metadata.platformType, or whose label maps to a Kubernetes, TrueNAS or vSphere type. A guest alert on a removed Proxmox node is still removed. |
||
|
rcourtman
|
bab38a25b9 |
Drop the summary chart caches nothing reads (#2662)
The Workloads table history hook was the last caller of `infrastructureSummaryCache.ts` and `workloadsSummaryCache.ts`, and nothing has read either cache since the InfrastructureSummary card and the standalone Workloads summary were deleted on 2026-05-26. The hook runs both fetchers inside `createNonSuspendingQuery`, which always passes an abort signal, so their in-flight dedupe was skipped too. Each 30-second Trends poll still serialized the full response on the main thread and wrote it to localStorage or, above the 900k-character cap, deleted the key. Measured on embedded builds against the 50-node, 929-guest mock estate at the 1h range: | | Before | After | |---|---|---| | JSON serialized per poll | 9,629,902 + 924,249 chars (5-31 ms each) | none over 200k | | Summary-cache storage calls | `removeItem` on both keys every poll | none | | `/api/charts/workloads` + `/api/charts/infrastructure` | one each on load and per poll | same | | Rows drawing history (1440 / 390 wide) | 12 nodes + 140 guests / 6 + 36 | same | The hook now calls `ChartsAPI.getWorkloadCharts` and `ChartsAPI.getInfrastructureSummaryCharts` directly with the same range, signal, `maxPoints`, node scope and metric list. The node, agent and Docker-host series fold moves into `workloadMetricHistoryModel.ts` with its merge rule unchanged. The query layer already retains reads across remounts for five minutes and clears them on org switch. The hook replaces the deleted cache under the `workloads-hot-path` policy. The unified-resources, performance-and-scalability and frontend-primitives contracts still described InfrastructureSummary, its model and state hook, the shared summary table, selector and details drawer, WorkloadsSummary, throughputPresentation, resourceBadges and the two caches as live owners. Each clause now names the live owner, becomes a conditional rule, or is removed where nothing it governs remains. Stale prose for the other summary primitives deleted in May (SummaryMetricCard, InteractiveSparkline, DensityMap, StorageSummary) and their test-only helpers is left to a separate change. Checks: affected vitest suites (10,053 tests; the only failures are the Mac-only `agentInstallCommand` suite), contract, registry and status audits, and the browser verification receipt. |
||
|
rcourtman
|
156d2e274d |
Keep same-serial disks on different machines apart (#2660)
## Problem
`ingestRecord` refuses a physical-disk match under another parent
(`findMatch` skips scope-incompatible candidates), then `chooseNewID`
mints the same hardware-keyed canonical ID (`machine:<serial>`) and the
ingest merges into whatever disk already holds it. Disks that report one
usable serial on several machines collapse into a single resource
parented to whichever machine ingested last, and every other machine
loses the disk from inventory. Real triggers: dual-ported SAS shelves
seen by two nodes, cloned VMs with an explicit identical serial,
fixed-serial USB bridges, TrueNAS systems. Repro: `IngestSnapshot` with
three PVE nodes in one instance, each with a `/dev/sda` of the same
serial, gave one disk.
## Fix
- A new disk resolves its parent to a machine: the nearest host
ancestor, so a TrueNAS pool or Unraid array counts as its system or
host.
- Two disks with one identity stay apart only on evidence: one reporter
placing the serial on two machines (cloned VMs keep both serial and
hostname), or machines whose hostnames share nothing. Reporters on one
box still join across grouping parents and unmerged host resources, so
the TrueNAS API and the agent on that box stay one disk. An unknown
machine never splits a disk, so a PVE row whose node row is missing
still joins its host's agent disk.
- Once a serial spans machines, every copy takes the serial keyed to its
own machine, and the holder of the unscoped ID is re-keyed with its
source mappings, retired-ID claims and relationship endpoints. IDs don't
depend on machine ingest order, and the unscoped ID never passes between
disks that coexist. A disk only one machine reports keeps its current
ID.
- Operator splits ("Split merged resource" exclusions) now hold on this
path; the old collision merge ignored them. A split recorded against any
ID the disk can hold applies, so it survives a re-key.
- The registry indexes disks by hardware key (`physicalDisksByHardware`)
because `IdentityMatcher` keeps one resource per machine ID.
## ID stability decision
Machine-scoped IDs only for identities that actually span machines,
rather than a pre-pass. A pre-pass needs final parent scopes before
ingest, but host/node merging happens during ingest. Giving the
first-ingested machine the unscoped ID would hand one machine's ID, and
its operator state, to another machine when an earlier-sorting machine
appears. With the chosen design a disk's ID changes when a same-serial
disk first appears on another machine. That orphans ID-keyed state
rather than misapplying it.
## Residuals (documented in unified-resources.md)
- Like any serial-keyed ID, the unscoped ID still passes to a
same-serial disk that appears once the previous holder is gone.
- Alert-history rows owned under the unscoped ID don't follow a re-key.
The PVE disk alert owner (#2646) already leaves shared-serial rows
unbound unless a WWN tells them apart.
- Agent SMART and Unraid disk source IDs are the bare serial.
Same-serial agent disks on different hosts share one `SourceAgent`
mapping, and only the last ingested keeps the agent source target. This
is tracked as a separate task, because the fix re-keys persisted
mappings.
- Same-serial disks share one metrics history key.
- Across reporters, hostnames that normalize alike still join.
## Verification
- New tests in `registry_merge_policy_test.go`: per-machine disks for
PVE-only and linked-agent clusters, including reversed ingest order;
rehydration with a machine joining; agent + TrueNAS on one box versus a
cloned TrueNAS with the same hostname; a disk moving into a pool; a
parentless PVE row on a shared serial; exclusions honoured, surviving a
re-key, with per-machine fallback; serialized-JSON rehydration;
retired-ID claims carried across a re-key.
- Every new test fails against the unfixed registry or with its fix
removed (`-overlay` negative controls).
- `go test ./internal/unifiedresources` passes, as do the
disk/SMART/Unraid/TrueNAS tests in `internal/monitoring`, `internal/api`
and `internal/truenas`.
- Five rounds of independent review; the last round found no P1/P2
issues.
|
||
|
rcourtman
|
61c639223d |
Remove exports orphaned by the summary component deletions
The May cleanup deleted InfrastructureSummary, StorageSummary, the shared summary table and the SummaryMetricCard, InteractiveSparkline and DensityMap primitives, but left the helpers only they called. Each one below has had no production caller since then and survived only through its own tests and raw-source guard pins: - getMetricsChartKeyCandidatesFromResource (agentResources.ts) - getNormalizedIdentityLookupVariants and getAgentLikeIdentityAliases (resourceIdentity.ts); the variants helper was briefly reused for PBS history and dropped again on 2026-09-30 - getPBSStatusIndicator (status.ts) - useSummaryContextualFocusState (contextualFocus.ts), which takes filterSummarySeriesByGroupScope, resolveSummaryCardInteractionState and SummaryCardInteractionState with it - the deprecated ChartsAPI.getInfrastructureCharts delegate; its branch tests now exercise getInfrastructureSummaryCharts directly A production build differs from its parent only by that delegate's static method; every other removal was already tree-shaken. The frontend-primitives, storage-recovery and performance contracts still described the deleted summary cards, sparkline, density map and summary panel as live owners, and credited contextualFocus.ts with the removed hook's series filtering. Those clauses now name the live owners (summaryCardInteraction.ts behind summaryTableFocus.ts, the storage summary cache behind the pool Growth column, TooltipPortal for chart tooltips), become conditional on a summary strip returning, or go. |
||
|
rcourtman
|
49d4100385 |
Keep watchdog, delivery log and Patrol ages moving (#2598)
Follow-up to #2578: the remaining Alerts and Patrol panels whose relative ages froze while the panel stayed open over unchanged data. **External watchdog (Alerts › Destinations).** The status was read once at mount, so Last success and Monitor progress kept their read-time age through a real stall, which is the case the panel exists to show. It now re-reads `GET /api/alerts/deadman/status` every 30 seconds in the background (the Refresh button is untouched; a failed re-read keeps the last status and marks it unavailable), and both ages read the shared clock. **Delivery log.** Each row's hover age reads the clock. A logged attempt's time never changes, so its age is right over a log that is only re-read on demand. **Patrol.** The paused banner kept "Blocked 2m ago" because each status re-read returns the same `blocked_at`. The selected-run banner kept its age because the selected run is held as picked. The attention detail is read once per selection, so its acknowledgement, transition and evidence times now read the clock. Its Last seen and flapping window describe a record that may have moved since, so they stay read-time (the new frontend-primitives paragraph states that exception). Queue rows, summary, run history and the digest already re-derive on their own 30/60-second re-reads, so they are unchanged. **Verification.** Fake-timer tests per surface (each fails on main and passes here). Live on a private mock stack with Playwright `page.clock` and the data stream held (one live socket kept by a data-free keep-alive per fake minute), at 1440 and 390 wide. After ten held minutes: watchdog 33s → 10 mins ago with 20 background re-reads; delivery tooltip 5 → 15 mins ago; "Blocked 2m → 12m ago"; "Patrol run 3m → 13m ago"; attention detail 3/5/7m → 13/15/17m ago. The same-session control on main's files stayed frozen everywhere. Found while auditing, filed separately: future times ("snoozed until", "Returns to active attention") render as "just now" because `formatRelativeTime` is past-only. **Review follow-ups (second-opinion review, three passes).** The status read has no timeout, so the first poll design could misbehave: - **Stuck Refresh:** a hung background read blocked Refresh; separately, a Refresh superseded by a hung read stayed disabled. - **Slow reads:** reads slower than the 30s poll were all discarded, failures included, so an outage could read "Heartbeat healthy". - **False recovery:** an older late success could clear a newer failure. Each read now gets a sequence number, and an outcome applies only when it is newer than the last read that settled. A hung Refresh is freed at the first poll once it has been pending 29s (one second of slack, because live runs showed ticks landing a few ms short of 30s). Unit tests cover each case and fail on the earlier versions. Each case was also checked live with held reads answered in and out of order, with an earlier-version control. Main is merged in, with a receipt for the combined `PatrolAttentionWorkbench.tsx` (#2591). |
||
|
rcourtman
|
4e88ae110c |
Lead open History rows with the live reading of held alerts (#2638)
Builds on #2637 (alert card live reading), so this branch carries its commits until that PR merges. ## Why Alerts > History is a log of occurrences, but its open row is the one occurrence still in progress: it shows an Active badge and a running duration. Beside them, the breach message "Node temperature at 95.0°C" reads as the reading now, while the node sits at 78°C under its 80°C trigger and the alert is held until it clears. With #2637 the Alerts card one tab away says "Temperature 78°C now, back under the 80°C alert level", so the two pages disagreed about the same alert (the confusion in #2068). ## What changes - An active History row for a threshold alert with a usable live status leads with the same presentation summary as the card (`getMetricAlertPresentation`, via a new `getAlertHistoryRowCopy` in `alertHistoryModel.ts`). - Desktop: the clear rule and the last breach move into the message cell's hover. Phone: they show as muted lines under the reading, since a phone has no hover. - The row's `description` stays the recorded message, so search ("95.0" still finds the held row) and the Assistant handoff read the record. - Closed rows, and open rows without a live status (non-threshold alerts, Pulse system alerts, the moments after a restart), are unchanged. - The reading rechecks on the shared relative-time clock, read only for open rows, so it turns stale ("Last reading: …, 12 mins ago") when evaluations stop without re-rendering closed rows on each tick. ## Verification - Private mock stack built from this tree with a local-only generator overlay pinning pve2's CPU temperature: walked 95 → 78 → 72 °C at 1440×900 and 390×844, light and dark; earlier pass also covered breaching, °F, and the alert resolving out of the list. - New tests: model copy and fallbacks (`alertHistoryModel.test.ts`), the phone card (`AlertHistoryMobileList.test.tsx`), and the desktop row following a live store update and turning stale on the clock (`AlertHistoryTableAlertRow.test.tsx`). - Receipt: `frontend-modern/browser-verification/9c0e53268f0d014f.json`. Refs #2068 |
||
|
rcourtman
|
c9d4cdebf0 |
Read a silent in-guest agent's guest drawer readings as last known
A Pulse agent linked into a VM keeps its last report on the VM resource after it stops reporting: the registry copies the agent's whole facet, agent.stale included, onto the guest. The guest drawer never read that flag, so a quiet agent's RAID kept its live colours and "Rebuild: 42% · 118M/sec", its network rows and version read as current, and nothing said the agent had gone. - useWorkloads maps agent.stale onto the row as agentStale, and sets disksFromAgent when Proxmox reported no guest filesystems and the agent's own disks filled in. - The guest drawer passes the stopped-reporting reason to RaidCard, puts a status line over muted network rows, marks the Pulse Agent row as stopped reporting, and reads agent-supplied filesystem rows as last known. - Agent-supplied filesystem rows follow the agent's freshness instead of the Proxmox guest-read reason. A VM with a live linked agent but no qemu-guest-agent showed the agent's filesystems as "Usage unavailable" with advice to install qemu-guest-agent. A stopped VM keeps the Proxmox reason, and Proxmox's own guest filesystems keep their read state. The workload table's disk cell, the drawer History, and the agent's OS, addresses and physical disk health still carry no stale cue. |
||
|
rcourtman
|
a5b2a5a4c4 |
Refuse a seeded disk slot mapping when the disk was replaced
ingestRecord reuses an existing source-key mapping before any identity matching, and for physical disks it checked only the type and parent scope. Proxmox keys a disk by its slot, so in a registry seeded from persisted unified resources a disk swapped into the slot mapped straight back to the previous disk's canonical resource. Where an agent report had been merged in, mergeInto also kept that disk's serial, WWN, temperature and failed health. A Proxmox disk observation now refuses the mapping when the two disks carry hardware identities naming different disks (diskinventory.HardwareIdentityConflict). Only fields reported on both sides count. Differing serials do not count for SAS (Proxmox may report the SAS address, #1595), or when the mapped disk was merged with an agent report and Proxmox did not read the drive's own serial, by the rule monitoring's SMART merge already uses. Collection status cannot decide that: a legacy agent's restored serial keeps Proxmox's status. The refused key falls through to the linked-disk join, findMatch and chooseNewID, but none of them may merge it back into the refused disk: an ID landing on it is replaced with the key's source-specific ID. Without that, a disk keyed by a USB bridge serial re-merged at once. No current caller replays Proxmox disk records over seeded mappings: the monitor rebuilds a fresh registry each poll. This closes the registry contract the seeded-mapping comment promises. |
||
|
rcourtman
|
a85bdcdc9a |
Stop a stale agent SMART row lending a replaced disk its identity (#2658)
## Problem PR #2648 stopped the poller's previous-evidence lookup from handing a replaced disk the old disk's serial. The linked host agent's SMART merge runs before that lookup and had the same flaw. Its last-resort match is the device path, and a path names a slot, not a disk. When a disk is swapped before the agent reports again, or after the agent went silent and kept its last rows, the agent's row for the old disk matched the replacement. It filled the old serial or, for SAS, promoted it over the Proxmox SAS address. The registry then keyed the replacement to the old disk's canonical resource and SMART history, and it also took the old disk's temperature and FAILED health. For SAS, the registry's same-path join (`resolveLinkedPhysicalDisk`) did the same at the canonical level even when monitoring kept the rows apart. Proxmox's disk inventory (`PVE::Diskmanage`, checked on a PVE 9 node) writes the literal `unknown` for a serial or WWN udev cannot read. The poller recorded it as a collected serial, so the collection state hid the gap, and the merge's `Serial == ""` fill never let the agent's real serial replace it. ## Change - `mergeHostAgentSMARTIntoDisks` refuses a device-path row whose identity contradicts the Proxmox disk's. The new `diskinventory.HardwareIdentityConflict` defines a contradiction: nothing agrees across serial and WWN, and a field both producers report like for like disagrees. - WWNs count for every disk. - Serials count only where Proxmox's serial is the drive's own, as smartctl's is: a disk typed `nvme`, or one presented with SCSI vendor `ATA` (libata, or a SAS HBA's SCSI-ATA translation) and not typed `usb`/`sas`. Elsewhere the Proxmox serial may be a SAS address or SCSI designator (#1595) or a USB bridge's own serial, so it only counts toward agreement. - Spelling differences are not disagreement: udev's underscores for whitespace, a T10 designator for an ATA drive (`ATA`, the model, then the serial), the agent's NAA field spelling, and udev's 64-bit prefix of an NAA 6 WWN. - The refusal runs after the unique-path check, so a second row on the path keeps the match ambiguous. - The merge's WWN step compares normalized identity (`HardwareIdentityMatch`), so the agent's `5-c50-a1b2c3d4` finds the Proxmox `0x5000c500a1b2c3d4` disk. - `resolveLinkedPhysicalDisk` refuses the SAS same-path fallback when the two WWNs name different disks. - The poller records Proxmox's `unknown` serial/WWN as empty, so the serial reads `missing` and the agent can fill it. Retained evidence carries a previous serial only when it is usable, so a pre-upgrade record cannot restore the placeholder. Identity keys already ignored the placeholder. When the agent now fills a disk's serial, that disk's Proxmox-poll SMART samples move from its WWN or path key to the serial key, where the agent's own samples already go. ## Tests - `TestHostAgentSMARTRowForSwappedOutDiskDoesNotLendItsIdentity` (sata and sas) drives the real poller and registry. Proxmox reports `unknown`, the agent fills the serial, the agent goes silent, and the disk is swapped. On main, the SATA replacement takes the old serial and resource; the SAS replacement keeps its own serial in state, but the registry joins it to the old resource with its temperature and FAILED health. - `TestHostAgentSMARTMergeRefusesPathMatchAcrossContradictingIdentity` has one row per producer rule: SAS/SATA/NVMe replacements refused; same-disk SAS without an agent WWN, a renamed SAS disk in the agent's WWN spelling, udev's serial spelling, a USB enclosure (typed `usb` and typed `ssd`), a T10 designator and an NAA 6 volume merged; and an ambiguous path. - `TestRegistrySASPathJoinRefusesContradictingWWN`, `TestProxmoxUnknownDiskIdentityIsRecordedAsUnreported` and `TestHardwareIdentityConflict` cover the rest. - Negative controls via `go test -overlay`: removing each rule fails only its own case. These cover the guard, the WWN normalization, the transport/vendor rule, compaction, the designator rule, the NAA 6 prefix, the registry guard, the ingest normalization and the usable-serial carry. Against main's `monitor.go` and `registry.go`, the end-to-end, merge and registry tests fail. - `TestMergeHostAgentSMARTIntoDisks_PreservesExistingProxmoxIdentity` merged a row with a different WWN by path alone, which the guard now rightly refuses. Its row is now the same disk in the agent's WWN spelling, with all assertions unchanged. - `internal/monitoring`, `internal/unifiedresources` and `pkg/diskinventory` pass in full. ## Not covered - A replacement with no comparable identity still matches its path: no usable serial or WWN at all, or a SAS/SCSI/USB disk with a WWN on one side only. - The registry's seeded source mapping, reused after a restart by the slot-shaped Proxmox source ID, can still hand a swapped disk the old canonical resource. That's a separate follow-up. |
||
|
rcourtman
|
447807526f |
Show the outdated sensor setup notice on every Proxmox tab
The Proxmox page warns when a node's SSH temperature monitoring still runs the pre-rc.6 setup, which locks the key to `sensors -j` so SATA/SAS disk temperatures never arrive. The notice judged that from the disk rows of the active tab, and since each tab hydrates only its own resource families, Overview loads no disks and never showed it. The registry now derives the verdict on the node itself, after child counts resolve each disk's canonical parent: a node with an available, legacy-format temperature payload is flagged when a SMART-only disk under it has no reading collected now. The page lists flagged nodes from the rows every tab already loads, so Overview needs no disk inventory. Every Proxmox node carries an explicit true or false, because the browser merges a facet field by field and would keep an earlier true if a cleared verdict were omitted. Two rule fixes ride along with the move. Proxmox's own disk inventory types SATA/SAS disks as hdd or ssd, so the old sata/sas-only set never matched a node without a host agent. And a retained reading kept after a disk stopped being read no longer counts as present (diskinventory.TemperatureCollected). |
||
|
rcourtman
|
19558f955d | Merge remote-tracking branch 'origin/main' into refactor/drop-summary-export-orphans | ||
|
rcourtman
|
43a80dafd9 |
Judge disk heat under the reporting host's Disk Temp override (#2654)
#2636 made the alert disk temperature policy the only judge of disk heat, but it resolved that policy by disk type alone. Disk temperature alerts (`CheckHost`) evaluate each SMART disk under the thresholds of the host agent that reports it. An explicit Disk Temp override on that host, or one inherited from its linked node or guest, replaces the per-type trigger, and a host whose alerts are switched off raises nothing. So a host with an 80C Disk Temp override still read **Running Hot** on Physical Disks at the 70C NVMe trigger. Patrol flagged it too, and a host with alerts switched off still read hot. The Alerts thresholds page promises the opposite: "Setting a Disk Temp override on a host above replaces these for all of that host's disks". ## What changes - **Backend policy:** `alerts.Manager.HostDiskTemperatureThreshold(host, diskType)` resolves the policy for a host the way `CheckHost` does, from the same helpers (`resolveHostThresholdsNoLock`, `hostThresholdOverrideNoLock`, `hostDiskTemperatureThresholdNoLock`). A disabled override gives no threshold. `DiskTemperatureThreshold(diskType)` is now its hostless case. - **Patrol and the AI chat context:** `ThresholdProvider.GetDiskTemperatureThreshold(host, diskType)`. Each unified physical disk finds its host by walking its parent to the agent machine, through the pool for Unraid array and cache disks. - **Physical Disks, the disk detail, the pool drawer's linked disks and the guest drawer** pass the reporting machine's override keys to `resolveDiskTemperatureDisplayThresholds`. Keys follow `CheckHost`'s order: agent ID, canonical resource ID, then the merged node or guest. The resolver returns null (uncoloured, never hot) for an override or agent default that switches the host's alerts off. The browser rebuilds `CheckHost`'s key chain from what it loads, so a few identity edge cases can still differ from alerts and Patrol (listed in `storage-recovery.md`): on the Storage page, a disk whose agent an operator manually linked into a VM; an override inherited through agent link hints the API does not publish (an unmerged linked node or guest), or stored under a linked node's own canonical ID; and guest keys in the guest drawer for an agent merged into a guest it is not automatically linked to. Overrides set on the agent's own row, or on the node or guest it is merged with, reach every surface. ## Verification - `TestHostDiskTemperatureThresholdMatchesCheckHostOverrides` checks each case against what `CheckHost` actually fires: no override, host override, host override without Disk Temp, an override under the canonical resource the agent ID resolves to, linked-node and linked-guest overrides, a host override shadowing a node one, a switched-off Disk Temp, and a disabled host, linked node or linked guest. - `TestPatrolJudgesAgentDisksByTheAgentDiskTemperatureOverride` uses a real registry with two agents, one overridden. A mutation that drops the owner lookup fails it. - Frontend tests cover the resolver, the owner walk and key order, DiskList with three hosts, the presentation map, pool-drawer linked disks and the guest drawer. - Private mock stack: three NVMe disks pinned at 72C. Before (main frontend): all three read Running Hot. After: - pve1, with an 80C override, reads Healthy (green). - pve2 still reads Running Hot. - pve3, with alerts off, is uncoloured. - Moving pve2's override onto its linked node ID turns it Healthy. - Checked at 1440 and 375 with no overflow. The receipt is in `frontend-modern/browser-verification/`. - A read-only second-opinion review ran twice. Round 1's findings (frontend key order, guest drawer canonical ID, disabled agent default, doc wording) are fixed in the first commit. Round 2's are the parity cases and the documented browser limits in the second commit. |
||
|
rcourtman
|
27e440339b |
Switch every running tenant monitor when mock mode flips
Mock mode is one process-wide flag, but SetMockMode ended the epoch, cleared the alerts, forgot the fixture agents and reset the state of only the monitor it was called on. The server runs one monitor per organization, each with its own alert manager and its own mock alert pass. A switch made through one of them (the mock-mode API uses the requesting tenant's monitor, the demo-fixture licence sync the default one) left every other running monitor holding what it raised in the mode the process had left: fixture alerts and fixture-agent node links in live mode, live alerts in mock mode, until the 24-hour stale sweep. Monitors now join a process-wide set of running monitors when Start runs and leave it when Start returns, under the same mutex that already serialized switches. SetMockMode flips the flag once, then ends the epoch of the monitor it was called on and of every running monitor: it waits for their admitted calls, clears their alerts, forgets their fixture agents and resets their state. Only then does it restart each running monitor's sampler, polls and discovery. Start picks the mock metrics sampler or discovery while it holds that mutex, so no switch can land between reading the mode and starting the runtime for it, and a switch leaves the runtime of a monitor that has not joined yet to its own Start. A monitor records the mode and the switch count it was built under. If it missed switches between New and Start, it leaves what it holds before it joins, dropping live alerts New restored or fixture alerts a read path raised meanwhile, even when a round trip brought the flag back. A call for the mode the flag already holds still moves a reached monitor whose alerts belong to the other mode, and leaves the rest alone. Release builds reach this only on an instance that runs DemoMode with the demo_fixtures entitlement, since release ValidateEnablement refuses mock fixtures otherwise, and that has more than one organization's monitor running. The licence sync only switches the default organization. So the fix stays inside the monitoring package. The callers are unchanged and no API surface moves. New tests start the default and one other tenant monitor through MultiTenantMonitor, as the server does, and switch through one of them. Against the per-monitor switch, the default monitor kept its fixture alerts (69 to 104 across runs) and all 69 fixture agents after mock mode was left through the other tenant. That tenant kept its live alert after mock mode was entered through the default monitor, and its fixture node stayed suppressed by a fixture agent's leftover node link. Four more tests fail if Start joins without leaving the mode its monitor was built in, if Start compares only the mode after a round trip, if a call for the flag's current mode returns without moving such a monitor, or if a switch starts the runtime of a monitor that has not joined. |
||
|
rcourtman
|
27854a6da5 |
Stop websocket upgrade headers exempting demo-mode writes
DemoModeMiddleware admitted any request carrying "Upgrade: websocket", whatever its method or path, so a POST/PUT/PATCH/DELETE with that header reached its handler on a read-only demo instance. The public demo signs visitors in as the configured admin user, which RequireAdmin accepts and which skips token scope checks, so the middleware was the only write guard: adding two headers let a demo session switch the instance out of mock mode, change system settings or create organisations. The exemption never served a real handshake. RFC 6455 handshakes are GET requests, gorilla's Upgrader refuses any other method, and GET already passes the read-only branch above it, so /ws and /api/agent/ws keep connecting. Removing the branch leaves the method alone deciding. TestContract_DemoModeUpgradeHeadersDoNotExemptWrites signs in as the demo admin through the full router, proves the session passes RequireAdmin, gets 403 for a mock-mode POST with upgrade headers and still opens /ws. On the old middleware the same POST returned 200 and turned mock mode off. |
||
|
rcourtman
|
ade799307a |
Rank a confirmed availability outage first in the summary check
A resource with several attached availability checks exposes one compatibility summary, Resource.Availability, and health reads only that. The summary ranked every observed failure equally and broke ties by target id, so a check that had failed once below its threshold could hide a later-sorting check with a confirmed outage: the outage check's own row carried availability_unreachable while the machine it monitors read healthy. Rank a confirmed outage above any other failure and have health use the same per-check rule, so target health and row presentation follow a confirmed outage on any attached check. The rule mirrors the poller's incident gate: observed failures at the threshold, an unavailable aggregate when one is reported, and no indeterminate outcome. A probe agent that stops reporting keeps its old failure count but reads indeterminate with an unknown aggregate, so it confirms nothing, on its own row or on the target. Drawers already list every check from availabilityChecks; the summary keeps its worst-check meaning. |
||
|
rcourtman
|
83ec32ee71 |
Keep fixture alerts from outliving a mock-mode switch (#2656)
Leaving mock mode cleared active alerts, but every path that evaluates mode-dependent data checked the mode once, up front. An evaluation already holding fixture data when the switch happened landed its alerts after the clear, in live mode, where nothing evaluates or resolves fixture resources, so they stayed until the 24-hour stale sweep. A pass-level lock around `checkMockAlerts` was tried and reverted: it closed only that path and made the mock-mode API wait behind a whole pass of synchronous event-log writes. **What changes** - `mockModeFence` (`internal/monitoring/mock_mode_fence.go`): each path takes a scope before reading anything mode-dependent and sends each alert-manager call through `scope.run`. Covered: `checkMockAlerts` (every loop, host agents, disks, snapshot cache), `checkBackupAlerts`, `pollGuestSnapshots`, `checkConnectionAlerts`, `pruneStaleDockerAlerts`, and the resource-store refresh (`updateResourceStore`, `updateResourceStoreForRead` and `syncUnifiedResourceAlertsToState` now take the scope of the state they rebuild from). - `SetMockMode` is serialized, flips the mode, ends the epoch, waits only for calls already admitted, then clears. Unified metrics go one resource per call, so the wait covers single calls, not passes. Admission never blocks. - Registry rebuilds are tracked, not waited on: a refresh evaluates the registry it read back only if no rebuild from an ended epoch overlapped it. Readers that act on the inventory (Docker pruning, backup and snapshot guest lookups, the pollers' previous-state carry) use current state until an undisturbed rebuild in the new epoch replaces the fixture registry. - `mock.SupplementalRecords`/`SupplementalChanges` serve nothing outside mock mode. The router swaps its mock adapters for the real pollers only after the switch, refreshing between sources, and the adapters used to fall back to default fixtures, so a single switch to live mode raised fixture vSphere incidents. **Verified** - Each new test parks a real evaluation at a fixed point (a held lock, a blocking supplier or a gated registry read), switches modes and releases it, with bounded waits. Against the unfenced code: the mock pass left about 40 fixture alerts, the frontend refresh 5 vSphere incidents, the backup check 137 backup-age alerts, and a mock-mode connection observation counted toward a live connection's confirmation. The drain, republish, pruning and carry tests fail against a fence without its wait, overlap check or registry tracking; the republish test also fails with the old mock supplemental defaults. - 995 tests from the touched files and related areas pass, plus `internal/mock` and the `internal/api` mock/platform tests; the new tests pass under `-race` and over repeated runs. **Not covered** - The fence and clear are per monitor while mock mode is process-wide; other tenant monitors neither drain nor clear on a switch (unchanged, noted in the monitoring contract). - Report admission and live pollers outside the fenced paths can still evaluate after the clear when entering mock mode; their alerts can persist until mock mode is left. |
||
|
rcourtman
|
5814c0a44c |
Stop a failing availability check keeping a node online
Once every source with a verdict had gone quiet, the stale pass counted any current availability sighting as online. A node that preserveOrExpireNodes expired stayed online while its HTTP check failed, and a quiet failing check lifted that node's offline verdict to warning. One sighting stands for every check on a target and carries whichever check was projected last, so a failing check that had just run also vouched for another check's old pass. Status aggregation now judges each check on a monitored resource by its own evidence window. A passing check whose evidence is still valid proves the resource answers, and reads online when no current source has a verdict. Every other check abstains, current or quiet: a failing check proves only that one port or service does not answer, so it never makes a guest count as stopped in the workloads summary or the Proxmox page. A check's own row keeps the ordinary rules, and a manual link judges the merged resource by the checks it carries. Projecting or retargeting a check now re-applies the stale pass's rule to targets whose status that pass owns, since the resources API replays checks after its stale pass and never runs another. Only a target's own quiet sources give the pass that ownership, never its checks' sighting. The poller now keeps each check's evidence tied to the observation that passed. It ages out a local location that stopped running beside other locations, as it already did for silent remote paths: the check took its freshness from the newest location, so a failing remote report kept an hour-old local pass counted as reachable with current evidence. An available check's evidence lasts only as long as its newest reachable path, and a remote result adopted as local after an entitlement lapse keeps the server's receipt time rather than the agent's clock. |
||
|
rcourtman
|
b03a32d8ae |
Show a silent agent's retained RAID state as last known (#2652)
Since #2644 the machine drawer reads a silent host agent's Thermals and Disks as last known, but the RAID card between them kept presenting the agent's last report as current: a green "clean" array, an amber "clean, degraded, recovering" array with "Rebuild: 45% · 98000K/sec", and coloured member badges, as if the array were still rebuilding. The Machines table's RAID tooltip did the same with an amber rebuild bar. What changes - `RaidCard` takes the drawer's `lastKnownReason` (`agentReadingsLastKnownReason`, from `agent.stale`). Every array and member state stays as evidence, but reads "<state> (last known)" in muted text beside a muted dot, titled "Last known reading, not current: host agent stopped reporting". A rebuild reads "Rebuild was at N%" without its speed. Member badges lose their status colour, so a member that was not healthy names its state ("/dev/sdb2 · faulty"). - The Machines RAID column (hidden by default) and its tooltip apply the same rule from `agent.stale`: a muted summary with ", last known" for screen readers, and a tooltip headed "RAID Arrays (last known)" without the rebuild bar or speed. - mdadm reports healthy members as "active sync", which the badge helper only matched as an exact "active", so every healthy software RAID member wore the amber warning badge. `raidPresentation.ts` now reads it as healthy (failure states still win). - RaidCard's empty guard ran once at mount, so a drawer opened before the first RAID report never showed the card. It is now tracked. - The array header row wraps, so at 390 wide the longer last-known state takes its own line instead of truncating the array name to "m..". Not changed: the guest drawer's Guest RAID card. The workload model maps `agentRaid` but not the in-guest agent's staleness, so wiring it needs a separate change across `useWorkloads.ts` and the other agent-sourced guest rows. Verification - Private mock stack with a local-only overlay holding Apollo-114 offline past its lease, seeded with a clean md0 and a degraded, rebuilding md1. The live control host Spectrum-709 carries the same arrays. Checked the drawer card at 1920, 1440 and 390, the Machines RAID cell and tooltip at 1920 (the column is container-hidden below that), and the drawer in dark mode. Silent host: no emerald, amber or red classes anywhere in the card or tooltip. Live host: unchanged colours and rebuild bar, with healthy members now green. No console errors and no horizontal scroll. Receipt: `frontend-modern/browser-verification/05a685ccb29f5883.json`. - New tests: `RaidCard.test.tsx` (current, last known, transitions both ways, empty to populated), `raidPresentation.test.ts` (active sync and last-known labels), an `AgentsMachinesTable.test.tsx` case (muted summary, dot, no colour classes, no rebuild bar), and a `SharedPrimitives.guardrails.test.ts` pin. All five fail against the base source. - `vitest` over `components/shared`, `components/Infrastructure`, `components/Workloads`, `features/standalone`, `features/platformPage` and `utils` passes, apart from the PTY tests in `agentInstallCommand.test.ts`, which fail on macOS regardless of this change. The pre-commit battery passes. |
||
|
rcourtman
|
11cc232133 |
Stop reporting unavailable storage as disabled to AI tools (#2657)
The storage pool rows the AI tools build (pulse_storage pools and pulse_query storage pools) derived both enabled and active from "status is not offline". Storage that reported itself unavailable, such as Proxmox storage with active=0, reached Patrol and Assistant as administratively disabled even when it is enabled in Proxmox. Enabled now comes from the storage facet's own flag. Active needs the facet's flag and a row that is not offline, because PBS datastores report Active even while unavailable. Rows without a storage facet keep the status-derived value. TrueNAS pools and datasets were the only storage producers that left both flags at their zero value, so reading the facet alone would have turned every TrueNAS pool into a disabled one. They now report Enabled, and Active from the pool state or the dataset's mount and lock state. |
||
|
rcourtman
|
5b3f507a1c |
Remove status and disk helpers left unused by the selector removal (#2651)
Stacked on #2649 (base commit
|
||
|
rcourtman
|
2f5027a09d | Merge remote-tracking branch 'origin/main' into perf/drop-write-only-summary-chart-caches | ||
|
rcourtman
|
bc15c3500b |
Stop QEMU default disk serials merging nested Proxmox disks (#2653)
## Problem
Nested Proxmox clusters (each PVE node is itself a QEMU VM) lose
physical disks from inventory. QEMU gives a virtual disk configured
without a serial a default one built from its drive ID or a per-VM
counter, so every VM built the same way reports the same value:
- Proxmox's `/nodes/{node}/disks/list` serial is udev's
`ID_SERIAL_SHORT` (`PVE/Diskmanage.pm`). A virtio-scsi disk on a Proxmox
VM reports `drive-scsi0` with no `ID_WWN`; on a libvirt VM it is
`drive-scsi0-0-0-0`. A `-drive` given no ID falls back to QEMU's
automatic ID, `scsi0-hd0` for `if=scsi` and `none0` for `if=none`.
- An emulated ATA disk's serial is `QM` plus a five-digit counter
(`QM00001`).
`diskinventory.IsUsableHardwareID` accepted all of these, so
`resourceFromPhysicalDisk` set `MachineID = "drive-scsi0"` on every node
and the registry minted one `machine:drive-scsi0` canonical disk and one
SMART series for the whole cluster. Three nodes with two such disks each
produced 2 canonical disks instead of 6.
## Change
- `IsUsableHardwareID` rejects exactly QEMU's default shapes:
`drive-scsi` plus digits with at most three `-digits` groups,
`scsi<n>-hd<n>`, `none<n>`, and `QM` plus five digits. Real serials that
only share a prefix (`DRIVE-ASSET123`, `drive-sata0`, `QM0001`) stay
identity. It also strips `naa.`/`eui.`/`wwn-`/`0x` framing before its
all-zero and all-F test, so a `0x0000000000000000` WWN is a placeholder
too.
- TrueNAS provider: a disk took its canonical identity from any
non-empty serial, so a placeholder merged disks across appliances. It
now uses only a usable serial, and lists superseded IDs for every disk
without one.
- TrueNAS history keys: for a serial that was already rejected, such as
`UNKNOWN`, native TrueNAS history used TrueNAS's own disk identifier,
and the unified SMART writer (`syncUnifiedPhysicalDiskMetrics`)
substituted the metrics target only for an empty serial, so the samples
went to the canonical resource ID. Neither matched the chart reader's
key, and rejecting QEMU serials would have put TrueNAS VMs on that path.
Native history now falls back to the disk's source ID, and the writer
always writes under the resolved metrics target, so writer, native
history and reader agree. History lookup keys also drop placeholder
serials.
## Upgrade effect
A disk reporting one of these values (nested PVE nodes, TrueNAS or agent
hosts running as QEMU VMs with virtual disks, TrueNAS disks reporting
`UNKNOWN`) changes canonical ID and SMART metric key once. Samples
recorded under the old key are not carried over, because that key mixed
every disk that shared the value. Path-shaped alert references (Proxmox
disk health and wearout, host-agent SMART) are unaffected; anything
keyed on the old canonical ID or metric key starts again under the new
one. This is recorded in the monitoring and agent-lifecycle contracts.
## Tests
- `TestIsUsableHardwareIDRejectsQEMUDefaultDiskSerials`: each default
shape is rejected, falls back to the scoped topology ID and never
matches itself; prefix-sharing real serials stay usable.
`TestPreferredIDRejectsPlaceholderHardwareIdentity` gains the framed
all-zero/all-F WWNs.
- `TestNestedProxmoxDefaultQEMUSerialsStayPerNode`: 3 nodes ×
(`drive-scsi0`, `QM00005`, `drive-scsi0-0-0-1`) give 9 canonical disks,
9 slots and 9 metric keys, every SMART writer key is a key readers
resolve, and agent SMART source IDs stay per host.
-
`TestTrueNASPlaceholderDiskSerialsStayPerApplianceAndShareOneHistoryKey`:
two appliances with `UNKNOWN` and `drive-scsiN` serials keep one
canonical disk each, and the SMART writer's samples are readable under
each disk's metrics target.
- `TestProviderPlaceholderDiskSerialsKeyNeitherIdentityNorHistory`:
placeholder serials key no identity, list superseded canonical IDs, drop
out of native history lookup keys, and native history lands under the
registry's metrics target.
- Negative controls via `go test -overlay`, one runtime file reverted at
a time: base `identity.go` fails all three round trips (3 canonical
disks instead of 9, 9 TrueNAS disks instead of 16); base `provider.go`
fails the identity and history assertions; base `monitor.go` fails the
SMART-history-under-reader-key assertion; reverting only the native
history key, the superseded-ID gate or the lookup-key filter fails its
own provider assertion.
- `pkg/diskinventory`, `internal/truenas/...`,
`internal/unifiedresources/...` and `internal/monitoring` pass in full;
`internal/api/...` disk, hardware, serial and TrueNAS tests pass.
Stacked on #2648 (`fix/pve-disk-serial-carry`), whose commit this branch
carries until it merges. Not in this change: the registry's merge of
disks across parents on a canonical-ID collision (separate change), and
a pre-existing writer/reader key mismatch for controller-member disks
without a usable serial or WWN (`PhysicalDiskMetaMetricID` appends
topology to a source ID that already carries it), which is tracked
separately.
|
||
|
rcourtman
|
e89f1cec9e |
Drop the summary chart caches nothing reads
infrastructureSummaryCache.ts and workloadsSummaryCache.ts lost their last readers on 2026-05-26, when the InfrastructureSummary card and the standalone Workloads summary were deleted. Their only remaining caller, the Workloads table history hook, runs both fetchers inside createNonSuspendingQuery, which always passes an abort signal, so the in-flight dedupe was skipped as well. What was left ran on every 30-second Trends poll: serialize the whole response on the main thread, then write it to localStorage or, above the 900k-character cap, delete the key. On the 50-node, 929-guest mock estate that was 9.6 MB of workloads JSON and 0.9 MB of infrastructure JSON per poll (5-31 ms each), thrown away every time. The hook now calls ChartsAPI.getWorkloadCharts and getInfrastructureSummaryCharts directly with the same range, signal, maxPoints, node scope and metric list, and the node/agent/Docker-host series fold moves into workloadMetricHistoryModel. The query layer already retains reads across remounts for five minutes and clears them on org switch, so nothing else changes for users. A before/after run on embedded builds showed the same chart requests and sparklines on every node and guest row, with no storage traffic and no multi-megabyte serialization afterwards. The hook takes the deleted cache's place under the workloads-hot-path policy. The subsystem contracts still described InfrastructureSummary, its model and state hook, the shared summary table, selector and details drawer, WorkloadsSummary, throughputPresentation and the two caches as live owners. Those clauses now name the live owner (useWorkloadTableMetricHistory, ProxmoxNodesTable, nodeFromResource, ResponsiveMetricCell, storageSummaryCache, summaryTableFocus), become conditional rules, or are removed where nothing they govern remains. |
||
|
rcourtman
|
94a7573426 | Merge remote-tracking branch 'origin/main' into fix/relative-age-watchdog-patrol | ||
|
rcourtman
|
f165bff68e |
Record the browser pass for the merged Patrol lifecycle card
Merging main brought the suppression-end labels that follow the review state into PatrolAttentionWorkbench.tsx, next to this branch's relative-time clock for the acknowledgement age. Neither commit's receipt covers the combined file, so this records the browser pass on the merged tree. |
||
|
rcourtman
|
bd4f6e36c4 |
Remove contract prose and badge helpers for deleted surfaces (#2655)
Stacked on #2649, which carries its commit |
||
|
rcourtman
|
26bee50a8a |
Restate the read-time age exception after the wall-clock rule
A latest-reading age over data a surface reads once stays the age at read time unless the surface re-reads it; immutable event times read the clock. Main reworded the rule this exception follows, so it comes back after the merge, next to main's wording. |