Merge remote-tracking branch 'origin/main' into fix/global-default-off

This commit is contained in:
rcourtman
2026-10-07 16:08:21 +01:00
46 changed files with 2889 additions and 535 deletions
@@ -765,6 +765,15 @@ not fenced: a live report already past its mock-mode check when mock mode is
switched on can still be evaluated after the clear, and its alerts can persist
until mock mode is left.
Every tenant monitor registers its own fixture agents, and their node links,
in its own alert manager, so the toggle forgets the fixture agents of every
running monitor, not only those of the monitor it was called through. A link
left behind would keep owning its fixture node's usage alerts after mock mode
ends. `TestLeavingMockModeReleasesFixtureAgentNodeLinksOnEveryRunningMonitor`
in `internal/monitoring/monitor_host_agent_removal_lifecycle_test.go` switches
through the default monitor and fails if the other tenant's node stays
suppressed.
Physical-disk evidence collected by a host agent must survive projection back
into monitoring's models. Absent evidence has to carry its declared sentinel
rather than a zero value that reads as a real measurement: an absent
@@ -8901,3 +8910,12 @@ agent from its last report instead of from the PVE poll, and mark it stale at
their own heartbeat cutoff (`fleethealth.AgentStaleThreshold`, which is
separate from the monitoring reporting lease). Agent registration, enrolment,
install, update, removal and report identity are unchanged.
### Demo write guard ignores websocket upgrade headers
`internal/api/demo_middleware.go` changed only so a `POST`, `PUT`, `PATCH` or
`DELETE` carrying `Upgrade: websocket` is judged by the demo read-only guard
like any other write instead of always reaching its handler. Agent command
websocket handshakes to `/api/agent/ws` are HTTP/1.1 `GET` requests, the only
transport its gorilla upgrader serves, and still connect on a demo instance.
No agent registration, enrolment, install, update or removal path changed.
@@ -7651,6 +7651,17 @@ read-only demo account. That same hidden read-side boundary includes `GET` and
`HEAD` reads for `/api/admin/users` and manual discovery at `/api/discover`;
public demo mode may block writes generically, but it must not reveal that
admin-user inventory or manual-discovery read routes exist.
That generic write block keys on the request method alone. The demo
middleware admits `GET`, `HEAD` and `OPTIONS` plus the login, logout, AI
execute and OIDC login/callback allowlist; HTTP/1.1 websocket handshakes for
`/ws` and `/api/agent/ws` are `GET` requests and connect through that read
branch, so an `Upgrade: websocket` or `Connection: Upgrade` header never
carries a `POST`, `PUT`, `PATCH` or `DELETE` past the guard. A demo deployment that signs
visitors in as the configured admin, as the public demo does, has only this
guard between that session and admin writes such as `/api/system/mock-mode`.
`TestContract_DemoModeUpgradeHeadersDoNotExemptWrites` in
`internal/api/contract_test.go` signs in as that admin through the full router
and pins both halves.
That shared infrastructure install boundary now also preserves copied shell
command payload continuity: any privilege-escalation wrapper applied at
`frontend-modern/src/components/Settings/InfrastructureInstallerSection.tsx`
@@ -6666,6 +6666,11 @@ icon/content layout, and action-link chrome. The
outdated-sensor notices, and the
`platform-inline-notice-local-amber-shell` pattern guard blocks future
`platformPage` files from reintroducing page-local amber notice shells.
A notice's render predicate reads what its page hydrates on every tab the
notice renders on. `collectOutdatedSensorSetupNodes` lists the PVE nodes that
carry the registry's `proxmox.sensorSetupOutdated` verdict and takes no disk
rows, because only the Proxmox Storage tab loads them; a predicate over them
left the notice silent on Overview.
Alert incident-event filter containers, labels, and chips must now route
through the shared presentation helpers in
@@ -7886,6 +7891,25 @@ When the cell's value is a retained host-agent SMART temperature, it renders
muted with a dotted underline and screen-reader "last known", as does a
retained value in the guest card. Tooltip and Thermals rows say "(last known)",
and the Thermals row carries the collection reason as its title.
The shared drawer History reads the same decision. For a physical disk,
`resourceDetailDrawerMetricsHistoryModel.ts` takes the reading from
`physicalDisk.temperature`, beside the collection state that qualifies it,
because websocket rows carry no top-level `temperature` for a disk. When that
state is set and is not `available` (an empty or absent state predates the
contract and stays current), `getResourceMetricsHistoryCurrentMetrics` offers
no current `smart_temp`. A positive retained reading then reaches
`getResourceMetricsHistoryDeferredMetrics` as a deferred metric: with no
history points the legend says "last known", never "current", and the reason
renders above the chart.
TrueNAS disk rows follow it too: the drawer's Temperature row reads
"(last known)" with the reason as title and no heat tone, and the one-line
summary leaves a retained reading out because it has no room for the reason.
The Proxmox outdated sensor setup notice reads the registry's per-node
`proxmox.sensorSetupOutdated` verdict, which counts a SATA or SAS disk as having
a temperature only when the reading was collected now
(`diskinventory.TemperatureCollected` in
`internal/unifiedresources/proxmox_sensor_setup.go`), so a retained one cannot
hide the notice for a disk Pulse cannot read now.
The focused browser proofs are
`frontend-modern/src/features/patrol/__tests__/patrolRunAcceptance.test.ts`,
@@ -1899,11 +1899,33 @@ a live report or poll already past its mock-mode check when mock mode is
switched on can still evaluate after that clear, and its alerts can persist
until mock mode is left.
The fence and the clear are per monitor while the mode is process-wide. A
tenant monitor other than the one `SetMockMode` was called on neither drains
nor clears, so fixture alerts it raised outlive the switch as they did before
the fence; switching mock mode with several tenant monitors running is not
covered.
The mode is process-wide while each monitor keeps its own fence, alert manager,
fixture agents and state, and the server runs one monitor per organization, so
`SetMockMode` switches every running monitor, whichever tenant's monitor it is
called on (the mock-mode API calls the requesting tenant's, the demo-fixture
licence sync the default organization's). `Start` joins a process-wide set of
running monitors under `mockModeSwitchMu`, and leaves it when its loop returns.
A switch flips the mode once, then ends each monitor's epoch, waits for its
admitted calls, clears its alerts, forgets its fixture agents when leaving mock
mode and resets its state, and only then restarts the mode-dependent runtime of
each monitor in the set. `Start` sets its runtime context before it joins and
chooses the mock metrics sampler or discovery while it joins, under the same
lock, so no switch lands between reading the mode and starting the runtime for
it, and a switch that reaches a monitor not yet joined leaves the runtime to
its `Start`. A monitor that missed switches between `New` and `Start` first
leaves what it holds: `New` records the mode and the number of switches made so
far, and `Start` compares both, so even a round trip that brought the flag back
drops the fixture alerts a read path raised meanwhile, as well as the live
alerts `New` restored outside mock mode. A monitor that is not running is
switched only when the switch is made through it; a call for the mode the flag
already holds still moves any reached monitor whose alerts belong to the other
mode, and leaves the rest alone.
Release builds reach several tenant monitors in mock mode only on an instance
running `DemoMode` with the `demo_fixtures` entitlement (release
`ValidateEnablement` refuses mock fixtures otherwise) and with more than one
organization's monitor running; the licence sync switches only for the
default organization.
The proofs park a real evaluation at a fixed point, switch modes and release
it. `TestLeavingMockModeRefusesTheRestOfAnInFlightPass` holds a mock pass at its
@@ -1933,7 +1955,25 @@ previous-state carry returns fixture nodes or guests. All eight are in
`TestStoreRefreshTakesItsMockModeScopeBeforeReadingState` in
`internal/monitoring/canonical_guardrails_test.go` rejects a call that reads
state in an argument ahead of `m.mockModeFence.begin()` and pins the
end-epoch-then-clear order in `SetMockMode`.
end-epoch-then-clear order in `endMockModeEpoch`, which `SetMockMode` runs for
every monitor it reaches.
`TestLeavingMockModeClearsEveryRunningTenantMonitor` and
`TestEnteringMockModeClearsEveryRunningTenantMonitor` start the default and one
other tenant monitor through `MultiTenantMonitor`, wait until both have joined,
raise alerts in both, switch through one of them and check both. Against a
switch that reached only its own monitor, the default monitor kept its fixture
alerts (69 to 104 across runs) and all 69 fixture agents after leaving mock
mode through the other tenant, and that tenant kept its live alert after mock
mode was entered through the default monitor. `TestMonitorStartedAfterAMockModeSwitchLeavesTheModeItWasBuiltIn`
fails if `Start` joins without leaving the mode its monitor was built in, and
`TestSetMockModeAlignsAMonitorWhoseModeTheFlagAlreadyHolds` fails if a call for
the flag's current mode returns without moving such a monitor, and
`TestMockModeSwitchLeavesAnUnjoinedMonitorsRuntimeToItsStart` fails if a switch
starts the runtime of a monitor whose `Start` has not joined, and
`TestMonitorStartedAfterAMockModeRoundTripDropsWhatItRaisedMeanwhile` fails if
`Start` compares only the mode. All six are in
`internal/monitoring/monitor_mock_alerts_test.go`.
The disk evaluation runs on the mock alert tick, not the poller's disk
interval (five minutes by default, `PhysicalDiskPollingMinutes` per instance).
@@ -3175,7 +3215,20 @@ Monitoring retains one current status per target and location, using server
receipt time for remote freshness while keeping the agent-authored observation
time as evidence metadata. A stale or disconnected agent path derives as
indeterminate at read time; slow or fast agent clocks cannot manufacture or
conceal a disconnect. Aggregation is conservative: all reachable paths are
conceal a disconnect. A local path beside other locations that has not run
within the same window derives as indeterminate too ("no recent local check").
The target takes its freshness from its newest location, so otherwise a
failing remote report kept an old local pass counted as reachable and the
check read available with current evidence. For the same reason an available
check's evidence lasts only as long as its newest reachable path, not a
fresher path that failed. When the external-probe entitlement lapses, a
single remote result stands in for the local location until the local poller
runs, with the earlier of its agent check time and the server's receipt time
as its check time, so a fast agent clock cannot keep it current. An aggregate
over several locations never stands in: a target without a retained local
observation reads as never checked, and not connected, until the local poller
runs.
Aggregation is conservative: all reachable paths are
healthy, mixed reachable and failed/unknown paths are degraded, all current
paths unreachable are unavailable, and no reachable path with incomplete or
indeterminate coverage is unknown. Only aggregate unavailability advances the
@@ -944,6 +944,15 @@ Patrol coverage, schedule, finding, and approval state belongs on Patrol-owned
surfaces or explicit Patrol affordances; Proxmox overview must not treat it as
backup coverage, restore readiness, PBS verification/protection proof, or a
replacement for the Proxmox Backups tab and workload Backup column.
The outdated sensor setup notice above the Proxmox tabs is temperature
guidance, not storage or recovery evidence. It reads the registry's per-node
`proxmox.sensorSetupOutdated` verdict from the node rows the active tab already
loads (Overview, Storage, Replication and Backups load nodes; Ceph and Mail do
not, so it does not render there), so the Overview query must not add
`physical_disk` rows to feed it; the derived `ProxmoxData` flag opens no storage
or recovery path.
`ProxmoxPageSurface.contract.test.tsx` pins the notice on Overview with no disk
rows and an Overview query without `physical_disk`.
The Proxmox inline estate totals are likewise an adjacent frontend-primitives
and unified-resource projection, not backup or recovery evidence.
`ProxmoxPageSurface` must flow the already-loaded workload inventory into the
@@ -6886,3 +6895,14 @@ migration or recovery authority is added or moved.
The performance report and reporting runtime snapshot handlers apply the same
test before tabulating a disk temperature. That changes only which held value a
report shows, not any storage or recovery path.
### Demo write guard ignores websocket upgrade headers
`internal/api/demo_middleware.go` no longer exempts every request carrying
`Upgrade: websocket` from the demo read-only guard. On a `DEMO_MODE` instance a
`POST`, `PUT`, `PATCH` or `DELETE` with websocket handshake headers is now
judged like any other write: outside the login, logout, AI execute and OIDC
allowlist it gets the generic demo `403`, or `404` on a hidden route, and that
includes config import at `/api/config/import`. HTTP/1.1 websocket handshakes
are `GET` requests and still connect. No storage, retention, backup, migration
or recovery path is added or moved.
@@ -1121,6 +1121,13 @@ for the Physical Disks verdict. The heat reason follows any native TrueNAS
alert text, and a hot disk's status dot turns warning though its source state
stays `online`, because phones show only the dot.
`truenasPageModel.test.ts` and `TrueNASStorageTopologyTable.test.tsx` pin it.
The Temp column reads the same collection state through
`getPhysicalDiskTemperaturePresentation`. A disk temperature that is not
current, such as a host agent's last reading kept on a merged disk row after
the agent stopped reporting, renders as last known: muted, with its reason as
the title and screen-reader "last known" text. The Temp sort ranks it as no
reading, and the tree sort orders each sibling group, so a retained value
never leads its group hottest first.
The vSphere Datastores and Networks tables follow the same exception-first
Health column. `getVmwareDatastoreIssue` and `getVmwareNetworkIssue` return
nothing for a green row; otherwise the reasons are, in order, an impaired
@@ -2980,6 +2987,36 @@ or force every provider type back into the active route payload. Docker,
Kubernetes, TrueNAS, and VMware surface contract tests pin their route query
maps, and the shared hook test pins facet normalization.
### Proxmox node verdicts over disks arrive on the node
The Proxmox page hydrates one query per workflow, and only its Storage tab
loads `physical_disk` rows, so a node verdict that depends on the node's disks
is derived in the registry and carried on the node, never rebuilt in the page
from disk rows. `ProxmoxData.SensorSetupOutdated` (`proxmox.sensorSetupOutdated`
on REST and websocket rows) is that verdict for the outdated sensor setup
notice. `refreshProxmoxSensorSetupLocked`
(`internal/unifiedresources/proxmox_sensor_setup.go`) runs after
`buildChildCounts` on every ingest path and gives every Proxmox node an
explicit true or false (other resources omit it), because browser facet merges
read an omitted field as a partial snapshot and would keep an earlier true. It
is true when the node's temperature payload is available and in the legacy
`sensors -j` format (`models.Temperature.LegacySensorsFormat`) and a disk whose
canonical parent is the node has no reading collected now
(`diskinventory.TemperatureCollected`) and a type whose temperature a PVE node
gets only through SMART: `sata`, `sas`, or the `hdd` and `ssd` form factors
Proxmox's own inventory reports. Proxmox also types a non-rotational USB device
`ssd`, so a USB device without SMART counts. A retained reading does not count
as current, including one kept while a host agent reports the disk in standby;
a reading a linked host agent collects now clears the disk. NVMe disks never
set it, because kernel hwmon reports them even on a legacy setup, and a host
agent the registry has not linked to the node contributes no disks. The
verdict reflects the registry's node and disk state at each ingest.
`TestProxmoxNodeSensorSetupOutdatedFromItsDisks` and
`TestProxmoxNodeSensorSetupOutdatedFollowsWhereTheReadingArrives` in
`internal/unifiedresources/registry_test.go` pin the rule, the linked-agent
case, the explicit false on the wire and the presented JSON;
`resourceStateAdapters.test.ts` pins that a false clears a merged true.
### Canonical REST facets preserve realtime workload evidence
The frontend REST projection retains the complete source-authored Proxmox
@@ -3424,6 +3461,20 @@ It owns probe status, incidents, history, evidence, and the outgoing `checks`
relationship. The matched resource carries every correlated check in the canonical
`availabilityChecks` facet, keyed by saved target id, while `availability`
remains an additive singular compatibility summary selected from that set.
The summary is the worst check: a confirmed outage first, then any other
observed failure, then an unchecked, ambiguous or unresolved check, then a
passing one, ties going to the first saved target id. A confirmed outage is an
enabled check whose observed failures reach its failure threshold, with an
unavailable aggregate when one is reported and an outcome that is not
indeterminate. That is the gate the availability poller applies before it
raises `availability_unreachable`, so a probe agent that stops reporting keeps
its old failure count but confirms nothing. `availabilityOutageConfirmed` in
`internal/unifiedresources/availability.go` holds that rule for both the
ranking and `EvaluateResourceHealth`, which reads the summary. A confirmed
outage on any attached check therefore makes the resource's health critical
even when a check that sorts earlier has failed below its threshold, with the
`availability_failed` reason unless a critical alert explains the verdict
first.
Adding a second explicit or unambiguously correlated check must retain both
source-owned endpoint rows, project both facets onto the same resource, and
emit one `checks` relationship per target from the check to that resource.
@@ -6177,9 +6228,20 @@ pass and manual links both apply that function:
the best of equal ones. A quiet source drops out of that decision.
- Once every source is quiet, an `offline` verdict survives and any other
verdict reads `warning`. The best of those wins.
- A current facet sighting carries no verdict (the PBS host-agent association,
an availability check projected onto its target). It counts as `online` only
when no current source has a verdict, and reads `warning` once quiet.
- A current facet sighting carries no verdict (the PBS host-agent association).
It counts as `online` only when no current source has a verdict, and reads
`warning` once quiet.
- Availability checks projected onto a monitored resource rank like that facet,
but their shared sighting is not their verdict. `availabilityChecksProveOnline`
judges each check at the pass's `now` by its own evidence window
(`Evidence.ValidUntil`: two poll intervals for a single local check, the
probe report window for a remote or multi-location one). An enabled,
passing check with current evidence proves the resource answers and reads
`online` once one of the resource's own sources has gone quiet and none that
is current has a verdict. Every other check abstains, current or quiet: a
failing check proves only that one port or service does not answer, and a
pass whose evidence lapsed proves nothing now. A check's own row keeps the ordinary
rules.
The stale pass used to read delivery as the verdict. A Proxmox node the cluster
reported offline on a live poll came back `online` once its linked agent fell
@@ -6203,24 +6265,52 @@ online stays `online` when its linked agent stops reporting. A live agent keeps
a node `online` when the Proxmox poll reports it offline, which the Proxmox
nodes table reads with `connectionHealth: error` as a stale provider. When a
guest's or container's only source goes quiet, a running one stays `warning`
and a stopped one stays `offline`.
and a stopped one stays `offline`. In each quiet case a projected availability
check with current evidence lifts the resource to `online`, as described
below.
The sighting still reads `stale`, so health keeps its `telemetry_stale` reason
and monitored-system reasons are unchanged. In-memory clones keep the verdicts,
including a facet's missing one. A serialized copy carries none, so
`IngestResources` gives each sighting of a copy that lost them all the stored
resource status; a merged row's separate source verdicts do not survive that
round trip. A manual link that joins two resources reported by one source keeps
round trip, and a copy whose status a passing check supplied gives that status
to every source. A manual link that joins two resources reported by one source keeps
the fresher sighting. `registry_merge_policy_test.go` pins the node,
poller-expired, guest, round-trip, copy, agent, Docker, Kubernetes,
mixed-source and manual-link cases, and the aggregation table.
Availability checks still count by delivery. With every source that has a
verdict quiet, a current check keeps its target `online` even when the check
fails, as it did before. Several checks share one sighting whose freshness
comes from whichever check was projected last, so an availability verdict
needs each check's own freshness, including local checks that miss their
cadence, and has to survive manual links. That is left to a follow-up.
Availability checks used to count by delivery too. With every source that
has a verdict quiet, a current failing check kept a node `preserveOrExpireNodes`
expired `online`, and a quiet one lifted it to `warning`. Several checks share
one sighting whose freshness comes from whichever check was projected last, so
a failing check that just ran could also vouch for another check's old pass.
Each check is now judged by its own evidence. A failing probe no longer holds
an expired node `online`, and it never makes a guest read stopped to the
consumers that count `offline` as stopped (the workloads summary, the Proxmox
page counts). The failure stays on the check's own row, which raises an
outage incident once the check confirms one (past its failure threshold, from
every location). A check observed from several locations relies on the poller
ageing out a location that stopped running and lasting an available check's
evidence only as long as its newest reachable path (monitoring.md), so one
location's fresh failure cannot lend its freshness to another location's old
pass.
The verdict is read from the checks on the resource when status is
aggregated. Projecting, re-running or retargeting a check also re-applies the
stale pass's rule to every target it touches whose status that pass owns,
because a check is not a delivery from the target's own sources and the
resources API replays checks after its stale pass. The pass owns a status only
through the target's own sightings: the checks' sighting going quiet neither
hands it the target nor, once removed, takes the target away, so a target
whose own sources have not gone quiet keeps the status they gave it. A check
retargeted away therefore takes its verdict with it at once. A
manual link keeps only the primary's own checks, so the merged resource is
judged by the checks it shows and never by the linked resource's fresher
sighting. `registry_test.go` pins the expired-node and quiet-guest cases, both
projection orders, re-judging on check changes, ownership through the
target's own sightings and the manual link; the aggregation table pins the
check rules.
### Canonical object drawer hierarchy
@@ -6427,9 +6517,52 @@ conflict): the agent's row may be the slot's previous occupant, retained by a
silent agent, and the join would hand the replacement that disk's serial,
readings and canonical resource. A hardware identity match still joins on its
own, and without a WWN on either side the path fallback stands. The seeded
source mapping reused after a restart is a separate path this does not cover.
`TestRegistrySASPathJoinRefusesContradictingWWN` covers the stale row and both
same-disk shapes; monitoring's SMART merge applies the matching guard first.
source mapping a rehydrated registry reuses applies the same refusal (next
section). `TestRegistrySASPathJoinRefusesContradictingWWN` covers the stale
row and both same-disk shapes; monitoring's SMART merge applies the matching
guard first.
### Seeded slot mapping refuses a replaced disk
`ingestRecord` honours a source key's existing mapping, such as one
`IngestResources` seeds from persisted unified resources, before any
identity matching. Proxmox keys a disk by its slot
(`ProxmoxPhysicalDiskSourceID`), so a disk swapped into the slot arrived
under the previous disk's key, and `mergeInto` kept that disk's canonical ID
and, where an agent report had been merged in, its serial, WWN, temperature
and failed health. A Proxmox disk observation now refuses the mapping when
the two carry hardware identities naming different disks
(`proxmoxDiskSlotHoldsAnotherDisk`: `HardwareIdentityConflict` after
`HardwareIdentityMatch`). Only fields reported on both sides count, so a
missing serial or WWN, including Proxmox's literal `unknown`, is no evidence.
Differing serials are no evidence when either side is SAS, because Proxmox
may report the SAS address there (#1595), or when the mapped disk was merged
with an agent report, whose serial `mergeInto` may have kept (collection
status does not reliably say which serial it holds), unless Proxmox read the
drive's own serial too, by the rule monitoring's
`hostAgentSMARTSerialComparable` applies: NVMe on both sides, or an agent
SATA disk under Proxmox vendor `ATA` that Proxmox does not type `usb`, not a
SCSI designator or a USB bridge's serial. A refused key falls through to the linked-disk join,
`findMatch` and `chooseNewID` like an unmapped one, except that none of them
may merge it into the refused disk: an ID that lands on it is replaced with
the key's source-specific ID. Agreement still wins, as it does in identity
matching, so drives that share a serial they do not own, such as a USB
bridge's, read as one disk: a second swap in the same enclosure joins the
first replacement, which reported the same bridge serial. That belongs to
identity matching, not this guard. The refused disk keeps its Proxmox
facet and is not updated, so until the registry is rebuilt both disks answer
to the slot's PVE alert reference, and canonical reference resolution treats
that reference as ambiguous rather than handing it to either. A later
rehydration may seed the slot key to the old disk again, and the refusal
repeats. No current caller ingests Proxmox disk records over a seeded
mapping: the monitor rebuilds a fresh registry each poll, and the resource
API, host-continuity and Patrol registries seeded from unified resources do
not replay Proxmox disk records. This closes the registry contract, not a
reported symptom. `TestRegistrySeededSlotMappingRefusesReplacedDisk` covers
SATA (with and without a new WWN), SAS, USB-enclosure and Proxmox-only
replacements, plus eight same-disk controls (including a SCSI designator
from an agent without collection status and a USB bridge serial under
Proxmox vendor `ATA`), each through two rehydrations.
### Drawer tab selection survives a transient snapshot change (#1723)