The full installer race suite exposed report deletion despite the subshell-counter guard. Bind EXIT cleanup to the original process using the direct child shell parent PID, including before the inherited counter is initialised. Retain strict and unchanged-graph verdicts, exact arguments and parent cleanup. Align the stale channel-copy assertion with the existing non-stable message; executable consent controls remain unchanged.
Change-source: pulse-maintainer
Require a successful bounded executable identity probe before release discovery. Keep Pro and unknown runtime editions untouched even when VERSION offers a usable community version, and do not defer a broken primary binary to a legacy one. Exercise the real shell classifier and main boundary with adverse controls and a TERM-resistant deadline.
Change-source: pulse-maintainer
Preserve exact Delivery commits 697608448a and 745c1f9ce5. Same graph and endpoints; only the observed transport-only error can retry. Refusals, integrity failures, unknown output, timeouts and signals remain terminal.
Change-source: pulse-maintainer
Additively incorporate published demo-write admission, disk-replacement identity and Proxmox sensor guidance. Resolve the storage-recovery contract append conflict by retaining both the silent-source History rule and the demo-write rule unchanged.
Change-source: pulse-maintainer
`DemoModeMiddleware` let any request carrying `Upgrade: websocket`
through, whatever its method or path. On a `DEMO_MODE` instance a
`POST`/`PUT`/`PATCH`/`DELETE` with that header reached its handler, and
the read-only guard did nothing.
## How exposed the public demo was
- The public demo (`demo.pulserelay.pro`, v6.5.0) runs `DEMO_MODE=true`
with `PULSE_AUTH_USER=demo`, and its login page signs visitors in as
demo/demo. No proxy auth is configured. Caddy is a plain `reverse_proxy`
with no method filtering, and it forwards `Connection: Upgrade` and
`Upgrade: websocket` unchanged.
- That session is the configured admin.
`sessionUserHasSettingsAdminPrivileges` matches `cfg.AuthUser`, so
`RequireAdmin` accepts it, and session requests skip token scope checks.
Nothing in the handlers checks demo mode on its own, so this middleware
was the only write guard.
- Anyone using the public credentials could therefore make any admin
write by adding the two headers. That covers switching the demo out of
mock mode (`POST /api/system/mock-mode`), changing system settings,
creating organisations, importing config, changing the password and
adding notification targets. A CSRF token was needed, but the attacker
gets one from their own login.
- On the old code, the new contract test reproduces this end to end. A
demo/demo session's mock-mode POST with upgrade headers returned 200 and
turned mock mode off.
- I found no sign it was used. Over the last 30 days the demo journal
shows every "mock mode disabled" line paired with a process restart (new
PID). Config file mtimes match deploys. 5,782 ordinary writes were
blocked over the same period. Caddy keeps no access log for the demo, so
this is the absence of evidence, not proof.
## Change
The exemption is deleted rather than narrowed. RFC 6455 handshakes are
`GET` requests, gorilla's `Upgrader` refuses any other method, and `GET`
already passes the read-only branch above it. A narrowed `GET`-only
exemption would never run. `/ws` and `/api/agent/ws` keep connecting,
and the request method alone decides.
## Tests
- `TestContract_DemoModeUpgradeHeadersDoNotExemptWrites` (registered
api-contracts proof) runs against the full router. It signs in as the
configured demo admin and checks that the session passes `RequireAdmin`
for a read. A mock-mode POST carrying `Connection: Upgrade` and
`Upgrade: websocket` now gets the demo 403 and leaves mock mode on, and
a real `/ws` handshake from the same session still upgrades.
- `demo_middleware_test.go`: the table row that asserted the bypass now
expects 403. POST/PUT/PATCH/DELETE with handshake headers to mock-mode,
settings, orgs, `/ws` and `/api/agent/ws` all get 403. Real gorilla
handshakes to `/ws` and `/api/agent/ws` through the middleware still
upgrade and echo.
- Negative control: run against the old middleware via `go test
-overlay`, the new contract test and the upgrade-header test fail, and
the handshake test passes on both versions.
## Not changed here
- `auth.go` `isWebSocketUpgrade` and `configapi`
`explicitAPITokenFromRequest` still accept a `?token=` query credential
on any request with `Upgrade: websocket`. A valid, scoped token is still
required, so no access is gained, but the credential can travel in a
URL. The `ErrorHandler` skip in `middleware.go` only bypasses recovery
and metrics.
- Older and separate:
`/api/ai/sessions/{id}/{abort,summarize,fork,undo,redo}` change state on
`GET`, which the demo guard admits as reads. Tracked separately.
A TERM-only timeout relies on the child exiting. Keep a fixed five-second KILL backstop, preserve every timeout as a failure and bind the shell/Go command proofs to the actual bounded invocation.
Contract-Neutral: Internal build timeout containment; source graph, endpoints, checksum trust and artifact acceptance are unchanged.
Change-source: pulse-maintainer
Keep the published process-wide tenant switch while using the retained removal lifecycle for both agent and Docker fixture hosts in every ended epoch. Preserve both lifecycle regression tests; no candidate source is re-parented.
Change-source: pulse-maintainer
`ingestRecord` reuses an existing source-key mapping before any identity
matching. For physical disks it checked only the type and parent scope.
Proxmox keys a disk by its slot (`ProxmoxPhysicalDiskSourceID`, e.g.
`pve1-node1--dev-sdb`), so in a registry seeded from persisted unified
resources, a disk swapped into the slot mapped straight back to the
previous disk's canonical resource. Where an agent report had been
merged in, `mergeInto` also kept the old disk's serial, WWN, temperature
and FAILED health. This was found while reviewing #2658, and its
contract note said the seeded mapping was not covered.
**Change.** A Proxmox disk observation now refuses the mapping when the
two disks carry hardware identities naming different disks
(`diskinventory.HardwareIdentityConflict` from #2658):
- Only fields reported on both sides count. A missing serial or WWN,
including Proxmox's literal `unknown`, keeps the mapping.
- Differing serials do not count for SAS, because Proxmox may report the
SAS address there (#1595).
- They also do not count when the mapped disk was merged with an agent
report and Proxmox did not read the drive's own serial. This is the same
rule monitoring's `hostAgentSMARTSerialComparable` applies: NVMe on both
sides, or agent SATA under Proxmox vendor `ATA`, and never a Proxmox
`usb` disk.
- The refused key falls through to the linked-disk join, `findMatch` and
`chooseNewID`, but none of them may merge it back into the refused disk.
An ID landing on it is replaced with the key's source-specific ID.
**Reachability.** No current caller replays Proxmox disk records over a
seeded mapping. The monitor rebuilds a fresh registry each poll. The
resource API, host-continuity and Patrol registries seeded from unified
resources do not replay Proxmox disk records. This closes the registry
contract rather than a reported symptom.
**Proof.** `TestRegistrySeededSlotMappingRefusesReplacedDisk` (in
`internal/monitoring/issue1595_collection_trust_test.go`) builds the
previous disk through `IngestSnapshot` with fixtures shaped like
`monitor_pve`'s output, rehydrates with `IngestResources(List())`, polls
Proxmox, then rehydrates a second time.
- 5 replacement cases: SATA with and without a new WWN, SAS, a
USB-enclosure swap ingested Proxmox-first, and a disk Proxmox alone
reports.
- 8 same-disk controls: re-ingested, no serial, a SAS address serial,
SAS without a WWN, a USB bridge serial, a bridge serial under vendor
`ATA`, a SCSI designator, and a SCSI designator from an agent without
collection status.
Negative controls via `go test -overlay` each fail only their intended
cases: the whole fix removed, the refused-disk exclusion removed, serial
comparability always true or always false, the Proxmox-only branch
removed, a collection-status provenance rule instead, and the USB veto
removed. The combined tree of main, #2660 and this change also passes
`internal/unifiedresources` and the monitoring disk tests.
**Residuals (in the unified-resources contract).**
- Drives that share a serial they do not own, such as a USB bridge's,
still read as one disk, as they do in identity matching.
- The refused disk keeps its Proxmox facet until a rebuild, so the
slot's PVE alert reference resolves as ambiguous rather than to either
disk.
Since the per-tab Proxmox queries landed, only the Storage tab loads
`physical_disk` rows. So the outdated sensor setup notice, which judged
nodes from the active tab's disk rows, never showed on Overview.
The registry now derives the verdict on each Proxmox node
(`proxmox.sensorSetupOutdated`, an explicit true or false on every node)
right after child counts resolve each disk's canonical parent. A node is
flagged when its temperature payload is available and in the legacy
`sensors -j` format and a SATA/SAS disk under it has no reading
collected now (`diskinventory.TemperatureCollected`). The page lists
flagged nodes from the rows Overview, Storage, Replication and Backups
already load, so no tab hydrates disks for it.
Rule fixes that came with the move:
- Proxmox's disk inventory types SATA/SAS disks as `hdd`/`ssd`, so the
old `sata`/`sas`-only check never matched a node without a host agent.
- A retained reading no longer counts as current (carried over from
#2659's frontend rule).
- The verdict is sent as explicit false, because the browser's
field-by-field facet merge would otherwise keep an earlier true after
the condition clears.
Stacked on #2659 (shares its `sensorSetup.ts` edit, which this
replaces).
Verified on a private mock stack at 1440px and 390px:
- Overview, Storage, Replication and Backups show the notice for
legacy-format nodes with an `hdd` disk without a reading, a retained
SATA reading, and a SATA disk without a reading.
- Overview makes no `physical_disk` request.
- With `main`'s frontend swapped in, Overview showed nothing and Storage
named one node of three.
- A node whose disk starts reporting drops out on the next websocket
update without a reload.
Keep the same pinned Go graph and endpoint. Retry only the observed transport-only diagnostic, at most three attempts; stop on refusals, integrity errors, unknown output, timeouts and signals. Execute the actual Docker download command against controlled outcomes and retain its original-base rejection.
Contract-Neutral: Build-only transport recovery; source graph, endpoints, checksum trust and artifact acceptance are unchanged.
Change-source: pulse-maintainer
Match automatic system-mount exclusions at directory boundaries rather than textual prefixes. Local /snapshots, /runtime and /var/lib/docker-data volumes retain capacity and usage through agent collection, both report paths and QGA aggregation; genuine system and remote/container mounts remain filtered. Regression controls include host usage alert onset and recovery.
Contract-Neutral: Restore existing filesystem eligibility without changing schemas, identities, explicit operator patterns or guest-command safety policy.
Change-source: pulse-maintainer
Follow-up to #2656 (the per-monitor mock-mode epoch fence), which
documented this gap as not covered.
## Problem
Mock mode is one process-wide flag (`mock.SetEnabled`), but
`Monitor.SetMockMode` ended the fence epoch, cleared the alerts, forgot
the fixture agents and reset the state of only the monitor it was called
on. The server runs one `Monitor` per organization
(`MultiTenantMonitor.GetMonitor` → `New` + `Start`), and each runs its
own mock alert pass into its own alert manager. A switch made through
one monitor (the mock-mode API uses the requesting tenant's monitor; the
demo-fixture licence sync uses the default one) left every other running
monitor holding what it raised in the mode the process had left: fixture
alerts and fixture-agent node links in live mode, live alerts in mock
mode, until the 24-hour stale sweep.
## Fix
- `internal/monitoring/mock_mode_switch.go`: a process-wide set of
running monitors, guarded by the existing `mockModeSwitchMu`. `Start`
joins it and leaves when its loop returns. While joining it picks the
mock metrics sampler or discovery under the lock (now
`startModeRuntime`), so no switch lands between reading the mode and
starting the runtime for it.
- `SetMockMode` flips the flag once, then for the receiver and every
running monitor: ends the epoch, waits for admitted calls, clears
alerts, forgets fixture agents (when leaving mock mode) and resets state
(`endMockModeEpoch`). Only after that does it restart the sampler, polls
and discovery of each monitor in the set (`resumeAfterMockModeSwitch`).
A monitor that has not joined yet gets its runtime from its own `Start`.
- Each monitor records the mode its alerts and state belong to
(`mockModeAligned`, set in `New` from the same read that decides whether
persisted alerts are restored) and the number of switches made so far
(`mockModeSwitchesSeen`). If it missed switches between `New` and
`Start`, it leaves what it holds before joining, even after a round trip
that brought the flag back. A call for the mode the flag already holds
still moves any reached monitor that belongs to the other mode, and
leaves the rest alone.
Callers are unchanged and no API surface moves.
## Is this reachable in release builds?
Barely. In a `release` build `mockruntime.ValidateEnablement` refuses
mock fixtures unless `SetReleaseFixturesAuthorized(true)` was called.
Its only caller is `syncReleaseDemoFixtureRuntime`, which requires
`DemoMode` plus the `demo_fixtures` entitlement and runs only for the
default organization. Several tenant monitors in mock mode would also
need more than one organization's monitor running on that demo instance.
Demo mode's read-only middleware normally blocks creating organizations
and the mock-mode API's POST. It does forward any request carrying
`Upgrade: websocket` whatever its method; that bypass predates this
change and is being fixed separately. So in practice this is a
dev/mock-stack problem, and the fix stays inside the monitoring package.
## Tests
New tests in `internal/monitoring/monitor_mock_alerts_test.go`:
- `TestLeavingMockModeClearsEveryRunningTenantMonitor` and
`TestEnteringMockModeClearsEveryRunningTenantMonitor` start the default
and one other tenant monitor through `MultiTenantMonitor`, as the server
does, and wait until both have joined. They raise alerts in both, switch
through one, and check both.
- `TestMonitorStartedAfterAMockModeSwitchLeavesTheModeItWasBuiltIn`
covers a switch that lands between `New` and `Start`.
- `TestMonitorStartedAfterAMockModeRoundTripDropsWhatItRaisedMeanwhile`
covers two switches in that window, with fixture alerts raised in
between.
- `TestSetMockModeAlignsAMonitorWhoseModeTheFlagAlreadyHolds` covers a
call for the mode the flag already holds.
- `TestMockModeSwitchLeavesAnUnjoinedMonitorsRuntimeToItsStart` covers a
switch reaching a monitor that has not joined.
One more is in
`internal/monitoring/monitor_host_agent_removal_lifecycle_test.go`:
`TestLeavingMockModeReleasesFixtureAgentNodeLinksOnEveryRunningMonitor`.
It switches through the default monitor and checks that the other
tenant's fixture node is no longer suppressed by a fixture agent's
leftover node link.
Negative controls (`go test -overlay`):
| Control | Failing tests | What happened |
|---|---|---|
| Fence-PR runtime | Leaving, entering, node-link | Default monitor kept
69–104 fixture alerts (varies by run) and 69 fixture agents; org-b kept
its live alert; org-b's `mock-cluster-1-pve1` stayed suppressed |
| Set kept, switch reaches only the receiver | Same three | Same
failures |
| No realign at join |
`TestMonitorStartedAfterAMockModeSwitchLeavesTheModeItWasBuiltIn` | |
| Join compares only the mode, not the switch count |
`TestMonitorStartedAfterAMockModeRoundTripDropsWhatItRaisedMeanwhile` |
Started in live mode with the fixture node's memory alert |
| Plain early return when the flag already matches |
`TestSetMockModeAlignsAMonitorWhoseModeTheFlagAlreadyHolds` | |
| Resume ignores set membership |
`TestMockModeSwitchLeavesAnUnjoinedMonitorsRuntimeToItsStart` | |
Every control failed only the tests it targets.
Also run: the full `internal/monitoring` package; `-race -count=3` on
the new tests plus the existing `TestLeavingMockMode*` and
`TestMockModeFence*`; and the mock-mode tests in `internal/api` and
`internal/api/configapi`.
Replace the old build and commercial narrative with setup, recipient safety, schedule checks and honest retained-data limits in the existing Intelligence guide. Keep the old repository guide as a pointer and verify the shipped Markdown with meaningful actual-parent controls.
Contract-Neutral: Existing-surface help and rendered-documentation verification only; no runtime, route, permission or interface change.
Change-source: pulse-maintainer
The reviewed Core candidate has completed pure capacity checks, but its synthetic polling and History lifecycle was not executed. Correct the contract note so fixture presence or compilation cannot be read as source-polling or native recovery acceptance. Runtime source and retained adverse evidence are unchanged.
Change-source: pulse-maintainer
Preserve the exact Core commit. Retained exact-final local race checks and actual-parent failures adequately cover this pure aggregation change, including Linux, alias and overflow controls. Polling and History execution, native recovery and the earlier broad timeout remain unproved or adverse; this is not acceptance of those outcomes.
Change-source: pulse-maintainer
Preserve the exact three Delivery commits. Retained source-8bf8uicz proves the unchanged final shell gate and its 26 consent cases under race with syntax and vet checks. The updater-only proof mapping preserves unrelated installer requirements; installed timer and update acceptance remain separate.
Change-source: pulse-maintainer
Bring the general Unix private-file bootstrap up to the existing PBS guide's safety floor. Preserve token and installer files, ignore curl configuration, require verified HTTPS and HTTP 200 before publishing the script, and exercise refusal and partial-download paths.
Contract-Neutral: Documentation and executable documentation tests only; no installer, API, runtime or capability contract changes.
Change-source: pulse-maintainer
A machine with several availability checks shows one summary check, and
its health reads only that summary. The summary ranked every observed
failure equally and broke ties by target id. So a check that had failed
once, below its threshold, could hide a later-sorting check with a
confirmed outage. The outage check's own row carried
`availability_unreachable` while the machine it monitors read healthy.
- `availabilityCheckPriority` now ranks a confirmed outage first, then
any other observed failure, then unchecked/ambiguous/unresolved, then
passing. Ties still go to the first target id.
- `availabilityOutageConfirmed` is the one confirmation rule, shared by
the ranking and `EvaluateResourceHealth`. It mirrors the poller's
incident gate: observed failures at the threshold, an unavailable
aggregate when one is reported, and no indeterminate outcome. A probe
agent that stops reporting keeps its old failure count but confirms
nothing, on its own row or on the target.
- Tests in `availability_link_test.go` go through the registry
projection path: a confirmed outage behind an earlier unconfirmed
failure (and its recovery), and a silent probe agent's stale failure
count. Both fail without the change. `health_test.go`'s legacy fixture
gains the `LastChecked` every producer sets.
- The `unified-resources.md` contract describes the summary ranking and
the shared rule.
Drawers already list every check from `availabilityChecks`, and
availability incidents live on each check's own row, so alert synthesis
is unchanged.
Follows #2643, which made the stale pass read each source's own verdict
but left availability checks counting by delivery.
## Problem
Once every source with a verdict had gone quiet, any current
availability sighting kept its target online. A node that
`preserveOrExpireNodes` expired stayed online while its HTTP check
failed, and a quiet failing check lifted that node's offline verdict to
warning. One sighting stands for every check on a target and carries
whichever check was projected last, so a failing check that had just run
also vouched for another check's old pass.
## Change
- **Registry verdict** (`aggregateStatus`,
`availabilityChecksProveOnline`): each check on a monitored resource is
judged by its own evidence window (`Evidence.ValidUntil`). An enabled,
passing check with current evidence proves the resource answers and
reads online once one of the resource's own sources has gone quiet and
none that is current has a verdict. Every other check abstains, current
or quiet. A failing probe never makes a guest count as stopped
(workloads summary, Proxmox page counts). A check's own row keeps the
ordinary rules.
- **Ownership**: only a target's own quiet sources hand the stale pass
its status. Projecting, re-running or retargeting a check re-applies the
stale pass's rule to targets it owns, because the resources API replays
checks after its stale pass. A manual link judges the merged resource by
the checks it carries.
- **Poller evidence** (`internal/monitoring`):
- A local location beside other locations ages out like a silent remote
path ("no recent local check").
- An available check's evidence lasts only as long as its newest
reachable path.
- After an entitlement lapse, an adopted remote result keeps the earlier
of its agent check time and the server receipt time.
- A multi-location aggregate is never adopted as a local observation;
the target reads as never checked until the local poller runs.
Contracts: `unified-resources.md` ("Status follows each source's own
verdict") replaces the residual paragraph, and `monitoring.md` covers
the multi-location rules.
## Verification
- New tests in `registry_test.go`, `registry_merge_policy_test.go` and
`availability_probe_agent_test.go`. Each fails against the base runtime
via `go test -overlay`, including both projection orders.
- Full `internal/unifiedresources` and `internal/monitoring` suites
pass, plus the availability, connection, workload and resource-state
tests in `internal/api`, `resourceapi` and `chartapi`.
Retain the exact reviewed documentation and formatting commits. Existing update help separates checkout builds from installed executables, preserves edition and state, and removes shared temporary configuration writes and assumed recovery. Six retained rendered-help passes and original-input controls remain applicable; no runtime change or native/browser acceptance is claimed.
Change-source: pulse-maintainer
Register the real shell verifier for the updater without admitting it as proof for unrelated installers. Preserve exact installer requirements and test missing/unrelated proof rejection.
Change-source: pulse-maintainer
The broader installer race/vet VM attempt reached readiness but returned no command verdict at its observation deadline. Keep the same consent cases in the normal installer suite and make their committed file standalone, with only standard-library imports and the actual shell gate. This enables proportionate exact-source Bash regression proof without building unrelated application dependencies; the broader failed attempt remains retained.
Change-source: pulse-maintainer
Add the exact current upstream without rewriting accepted commits. Retain reviewed monitoring, history and documentation repairs alongside published disk identity, retained-temperature, age refresh and cache cleanup changes. Publication and final-source proof remain separate.
Change-source: pulse-maintainer
Normalization keeps a disk's last temperature when the current
observation did not collect one, such as a disk in standby or a host
agent past its reporting lease. #2603 taught the Storage presenters to
show that value as last known. Four surfaces still read the raw number
as a current reading. Each now goes through the canonical
`getPhysicalDiskTemperaturePresentation` /
`isPhysicalDiskTemperatureCurrent` instead of adding a predicate.
- **TrueNAS disk drawer** (`resourceDetailDrawerTrueNASModel.ts`): the
Temperature row took the 55°C warning tone from a retained reading, and
the summary listed it as a reading. It now reads "61°C (last known)"
with the reason as title and no tone, and the summary leaves it out. A
current reading keeps the existing tone. Replacing the fixed 55 with the
alert policy is a separate follow-up.
- **Drawer History fallback**
(`resourceDetailDrawerMetricsHistoryModel.ts`): with no stored points, a
retained reading was labelled "current". It now goes through the
deferred-metric slot, so the legend says "last known" and the reason
shows above the chart. A disk's value is read from `physicalDisk` beside
its collection state. Websocket rows carry no top-level `temperature`
for a disk, so on the TrueNAS page the fallback had shown "-" even for a
current disk.
- **TrueNAS storage table** (`TrueNASStorageTopologyTable.tsx`): the
Temp cell showed a retained reading plainly, and hottest-first sorted it
to the top of its group. It now renders muted with its reason, and the
sort ranks it as no reading.
- **Proxmox outdated sensor setup notice** (`sensorSetup.ts`): a
retained reading counted as a temperature arriving, so the notice could
hide for a SATA disk Pulse cannot read now.
Verification: a regression test per surface fails on main and passes
here. A private mock stack with a local-only overlay stamped one TrueNAS
disk and one pve3 SATA disk with an unavailable temperature state ("host
agent stopped reporting") and marked pve3's sensor payload legacy.
Playwright at 1440×900 and 390×844 compared the change against the same
session with main's runtime files. The receipt is
`frontend-modern/browser-verification/294409f678589ef2.json`.
The Workloads table history hook was the last caller of
`infrastructureSummaryCache.ts` and `workloadsSummaryCache.ts`, and
nothing has read either cache since the InfrastructureSummary card and
the standalone Workloads summary were deleted on 2026-05-26. The hook
runs both fetchers inside `createNonSuspendingQuery`, which always
passes an abort signal, so their in-flight dedupe was skipped too. Each
30-second Trends poll still serialized the full response on the main
thread and wrote it to localStorage or, above the 900k-character cap,
deleted the key.
Measured on embedded builds against the 50-node, 929-guest mock estate
at the 1h range:
| | Before | After |
|---|---|---|
| JSON serialized per poll | 9,629,902 + 924,249 chars (5-31 ms each) |
none over 200k |
| Summary-cache storage calls | `removeItem` on both keys every poll |
none |
| `/api/charts/workloads` + `/api/charts/infrastructure` | one each on
load and per poll | same |
| Rows drawing history (1440 / 390 wide) | 12 nodes + 140 guests / 6 +
36 | same |
The hook now calls `ChartsAPI.getWorkloadCharts` and
`ChartsAPI.getInfrastructureSummaryCharts` directly with the same range,
signal, `maxPoints`, node scope and metric list. The node, agent and
Docker-host series fold moves into `workloadMetricHistoryModel.ts` with
its merge rule unchanged. The query layer already retains reads across
remounts for five minutes and clears them on org switch. The hook
replaces the deleted cache under the `workloads-hot-path` policy.
The unified-resources, performance-and-scalability and
frontend-primitives contracts still described InfrastructureSummary, its
model and state hook, the shared summary table, selector and details
drawer, WorkloadsSummary, throughputPresentation, resourceBadges and the
two caches as live owners. Each clause now names the live owner, becomes
a conditional rule, or is removed where nothing it governs remains.
Stale prose for the other summary primitives deleted in May
(SummaryMetricCard, InteractiveSparkline, DensityMap, StorageSummary)
and their test-only helpers is left to a separate change.
Checks: affected vitest suites (10,053 tests; the only failures are the
Mac-only `agentInstallCommand` suite), contract, registry and status
audits, and the browser verification receipt.
Count QGA Windows volume GUIDs rather than their shared physical drive; without a GUID retain distinct drive paths. Preserve visible mount metadata, Linux deduplication and byte-overflow containment. Exercise QEMU-only polling, removal and measured-zero History, adapting guest fixtures to the current captured mock-mode scope.
Contract-Neutral: Correct existing guest filesystem capacity aggregation and fixture scope compatibility without changing schemas, identity, command admission or backup policy.
Change-source: pulse-maintainer
## Problem
`ingestRecord` refuses a physical-disk match under another parent
(`findMatch` skips scope-incompatible candidates), then `chooseNewID`
mints the same hardware-keyed canonical ID (`machine:<serial>`) and the
ingest merges into whatever disk already holds it. Disks that report one
usable serial on several machines collapse into a single resource
parented to whichever machine ingested last, and every other machine
loses the disk from inventory. Real triggers: dual-ported SAS shelves
seen by two nodes, cloned VMs with an explicit identical serial,
fixed-serial USB bridges, TrueNAS systems. Repro: `IngestSnapshot` with
three PVE nodes in one instance, each with a `/dev/sda` of the same
serial, gave one disk.
## Fix
- A new disk resolves its parent to a machine: the nearest host
ancestor, so a TrueNAS pool or Unraid array counts as its system or
host.
- Two disks with one identity stay apart only on evidence: one reporter
placing the serial on two machines (cloned VMs keep both serial and
hostname), or machines whose hostnames share nothing. Reporters on one
box still join across grouping parents and unmerged host resources, so
the TrueNAS API and the agent on that box stay one disk. An unknown
machine never splits a disk, so a PVE row whose node row is missing
still joins its host's agent disk.
- Once a serial spans machines, every copy takes the serial keyed to its
own machine, and the holder of the unscoped ID is re-keyed with its
source mappings, retired-ID claims and relationship endpoints. IDs don't
depend on machine ingest order, and the unscoped ID never passes between
disks that coexist. A disk only one machine reports keeps its current
ID.
- Operator splits ("Split merged resource" exclusions) now hold on this
path; the old collision merge ignored them. A split recorded against any
ID the disk can hold applies, so it survives a re-key.
- The registry indexes disks by hardware key (`physicalDisksByHardware`)
because `IdentityMatcher` keeps one resource per machine ID.
## ID stability decision
Machine-scoped IDs only for identities that actually span machines,
rather than a pre-pass. A pre-pass needs final parent scopes before
ingest, but host/node merging happens during ingest. Giving the
first-ingested machine the unscoped ID would hand one machine's ID, and
its operator state, to another machine when an earlier-sorting machine
appears. With the chosen design a disk's ID changes when a same-serial
disk first appears on another machine. That orphans ID-keyed state
rather than misapplying it.
## Residuals (documented in unified-resources.md)
- Like any serial-keyed ID, the unscoped ID still passes to a
same-serial disk that appears once the previous holder is gone.
- Alert-history rows owned under the unscoped ID don't follow a re-key.
The PVE disk alert owner (#2646) already leaves shared-serial rows
unbound unless a WWN tells them apart.
- Agent SMART and Unraid disk source IDs are the bare serial.
Same-serial agent disks on different hosts share one `SourceAgent`
mapping, and only the last ingested keeps the agent source target. This
is tracked as a separate task, because the fix re-keys persisted
mappings.
- Same-serial disks share one metrics history key.
- Across reporters, hostnames that normalize alike still join.
## Verification
- New tests in `registry_merge_policy_test.go`: per-machine disks for
PVE-only and linked-agent clusters, including reversed ingest order;
rehydration with a machine joining; agent + TrueNAS on one box versus a
cloned TrueNAS with the same hostname; a disk moving into a pool; a
parentless PVE row on a shared serial; exclusions honoured, surviving a
re-key, with per-machine fallback; serialized-JSON rehydration;
retired-ID claims carried across a re-key.
- Every new test fails against the unfixed registry or with its fix
removed (`-overlay` negative controls).
- `go test ./internal/unifiedresources` passes, as do the
disk/SMART/Unraid/TrueNAS tests in `internal/monitoring`, `internal/api`
and `internal/truenas`.
- Five rounds of independent review; the last round found no P1/P2
issues.
Retain the exact German and Spanish documentation candidate and its rendered safety controls. Runtime frontend and browser receipts are unchanged.
Change-source: pulse-maintainer
Text matching admitted truncated configuration, nested opt-in and padded RC channels. Parse one configuration object with the installed jq dependency and fail closed before downloads when consent cannot be verified. Preserve the legacy no-config timer opt-in, manual updates and disabled-timer gate; add executed-shell regression cases and the deployment contract.
Change-source: pulse-maintainer
Follow-up to #2578: the remaining Alerts and Patrol panels whose
relative ages froze while the panel stayed open over unchanged data.
**External watchdog (Alerts › Destinations).** The status was read once
at mount, so Last success and Monitor progress kept their read-time age
through a real stall, which is the case the panel exists to show. It now
re-reads `GET /api/alerts/deadman/status` every 30 seconds in the
background (the Refresh button is untouched; a failed re-read keeps the
last status and marks it unavailable), and both ages read the shared
clock.
**Delivery log.** Each row's hover age reads the clock. A logged
attempt's time never changes, so its age is right over a log that is
only re-read on demand.
**Patrol.** The paused banner kept "Blocked 2m ago" because each status
re-read returns the same `blocked_at`. The selected-run banner kept its
age because the selected run is held as picked. The attention detail is
read once per selection, so its acknowledgement, transition and evidence
times now read the clock. Its Last seen and flapping window describe a
record that may have moved since, so they stay read-time (the new
frontend-primitives paragraph states that exception). Queue rows,
summary, run history and the digest already re-derive on their own
30/60-second re-reads, so they are unchanged.
**Verification.** Fake-timer tests per surface (each fails on main and
passes here). Live on a private mock stack with Playwright `page.clock`
and the data stream held (one live socket kept by a data-free keep-alive
per fake minute), at 1440 and 390 wide. After ten held minutes: watchdog
33s → 10 mins ago with 20 background re-reads; delivery tooltip 5 → 15
mins ago; "Blocked 2m → 12m ago"; "Patrol run 3m → 13m ago"; attention
detail 3/5/7m → 13/15/17m ago. The same-session control on main's files
stayed frozen everywhere.
Found while auditing, filed separately: future times ("snoozed until",
"Returns to active attention") render as "just now" because
`formatRelativeTime` is past-only.
**Review follow-ups (second-opinion review, three passes).** The status
read has no timeout, so the first poll design could misbehave:
- **Stuck Refresh:** a hung background read blocked Refresh; separately,
a Refresh superseded by a hung read stayed disabled.
- **Slow reads:** reads slower than the 30s poll were all discarded,
failures included, so an outage could read "Heartbeat healthy".
- **False recovery:** an older late success could clear a newer failure.
Each read now gets a sequence number, and an outcome applies only when
it is newer than the last read that settled. A hung Refresh is freed at
the first poll once it has been pending 29s (one second of slack,
because live runs showed ticks landing a few ms short of 30s). Unit
tests cover each case and fail on the earlier versions. Each case was
also checked live with held reads answered in and out of order, with an
earlier-version control. Main is merged in, with a receipt for the
combined `PatrolAttentionWorkbench.tsx` (#2591).
Retain exact reviewed Core commit 6dcff07343. Its dedicated regression tests, both parent-production controls, full Proxmox race result and focused monitoring race/vet receipts support the existing counter invariant repair. The broad monitoring timeout remains adverse evidence, not whole-suite acceptance; native QEMU-only recovery and containing publication remain separate.
Change-source: pulse-maintainer
Compose the upstream frontier observed at this coordination turn's launch. Preserve its reviewed commit identities and exact frontend content; do not reconcile later upstream movement in this bounded batch.
Change-source: pulse-maintainer
Builds on #2637 (alert card live reading), so this branch carries its
commits until that PR merges.
## Why
Alerts > History is a log of occurrences, but its open row is the one
occurrence still in progress: it shows an Active badge and a running
duration. Beside them, the breach message "Node temperature at 95.0°C"
reads as the reading now, while the node sits at 78°C under its 80°C
trigger and the alert is held until it clears. With #2637 the Alerts
card one tab away says "Temperature 78°C now, back under the 80°C alert
level", so the two pages disagreed about the same alert (the confusion
in #2068).
## What changes
- An active History row for a threshold alert with a usable live status
leads with the same presentation summary as the card
(`getMetricAlertPresentation`, via a new `getAlertHistoryRowCopy` in
`alertHistoryModel.ts`).
- Desktop: the clear rule and the last breach move into the message
cell's hover. Phone: they show as muted lines under the reading, since a
phone has no hover.
- The row's `description` stays the recorded message, so search ("95.0"
still finds the held row) and the Assistant handoff read the record.
- Closed rows, and open rows without a live status (non-threshold
alerts, Pulse system alerts, the moments after a restart), are
unchanged.
- The reading rechecks on the shared relative-time clock, read only for
open rows, so it turns stale ("Last reading: …, 12 mins ago") when
evaluations stop without re-rendering closed rows on each tick.
## Verification
- Private mock stack built from this tree with a local-only generator
overlay pinning pve2's CPU temperature: walked 95 → 78 → 72 °C at
1440×900 and 390×844, light and dark; earlier pass also covered
breaching, °F, and the alert resolving out of the list.
- New tests: model copy and fallbacks (`alertHistoryModel.test.ts`), the
phone card (`AlertHistoryMobileList.test.tsx`), and the desktop row
following a live store update and turning stale on the clock
(`AlertHistoryTableAlertRow.test.tsx`).
- Receipt: `frontend-modern/browser-verification/9c0e53268f0d014f.json`.
Refs #2068
Preserve exact reviewed tip b5b1fa9777 and upstream a85bdcdc9a additively. Retain both alert threshold test sets. Combine upstream mock-mode epoch fencing with reviewed structural-revision checks and Docker departure ownership; add a regression that a current revision cannot revive an ended epoch. Resolve the monitoring contract to describe both boundaries, without rewriting accepted history.
Publication remains deferred for genuine final-content browser receipts and independent CodeQL disposition. Composition checks do not establish native recovery, deployment, exact-head hosted acceptance or release readiness.
Change-source: pulse-maintainer
ingestRecord reuses an existing source-key mapping before any identity
matching, and for physical disks it checked only the type and parent
scope. Proxmox keys a disk by its slot, so in a registry seeded from
persisted unified resources a disk swapped into the slot mapped straight
back to the previous disk's canonical resource. Where an agent report had
been merged in, mergeInto also kept that disk's serial, WWN, temperature
and failed health.
A Proxmox disk observation now refuses the mapping when the two disks
carry hardware identities naming different disks
(diskinventory.HardwareIdentityConflict). Only fields reported on both
sides count. Differing serials do not count for SAS (Proxmox may report
the SAS address, #1595), or when the mapped disk was merged with an agent
report and Proxmox did not read the drive's own serial, by the rule
monitoring's SMART merge already uses. Collection status cannot decide
that: a legacy agent's restored serial keeps Proxmox's status.
The refused key falls through to the linked-disk join, findMatch and
chooseNewID, but none of them may merge it back into the refused disk:
an ID landing on it is replaced with the key's source-specific ID.
Without that, a disk keyed by a USB bridge serial re-merged at once.
No current caller replays Proxmox disk records over seeded mappings: the
monitor rebuilds a fresh registry each poll. This closes the registry
contract the seeded-mapping comment promises.
Separate development build output from the installed service. Retire the legacy shared-temporary-file and unconditional rollback recipes in favour of the current deployment-aware guide, with rendered link and safety checks.
Contract-Neutral: Documentation and rendered-help verification only; no updater, service, API, build or recovery behaviour changes.
Change-source: pulse-maintainer
## Problem
PR #2648 stopped the poller's previous-evidence lookup from handing a
replaced disk the old disk's serial. The linked host agent's SMART merge
runs before that lookup and had the same flaw. Its last-resort match is
the device path, and a path names a slot, not a disk. When a disk is
swapped before the agent reports again, or after the agent went silent
and kept its last rows, the agent's row for the old disk matched the
replacement. It filled the old serial or, for SAS, promoted it over the
Proxmox SAS address. The registry then keyed the replacement to the old
disk's canonical resource and SMART history, and it also took the old
disk's temperature and FAILED health. For SAS, the registry's same-path
join (`resolveLinkedPhysicalDisk`) did the same at the canonical level
even when monitoring kept the rows apart.
Proxmox's disk inventory (`PVE::Diskmanage`, checked on a PVE 9 node)
writes the literal `unknown` for a serial or WWN udev cannot read. The
poller recorded it as a collected serial, so the collection state hid
the gap, and the merge's `Serial == ""` fill never let the agent's real
serial replace it.
## Change
- `mergeHostAgentSMARTIntoDisks` refuses a device-path row whose
identity contradicts the Proxmox disk's. The new
`diskinventory.HardwareIdentityConflict` defines a contradiction:
nothing agrees across serial and WWN, and a field both producers report
like for like disagrees.
- WWNs count for every disk.
- Serials count only where Proxmox's serial is the drive's own, as
smartctl's is: a disk typed `nvme`, or one presented with SCSI vendor
`ATA` (libata, or a SAS HBA's SCSI-ATA translation) and not typed
`usb`/`sas`. Elsewhere the Proxmox serial may be a SAS address or SCSI
designator (#1595) or a USB bridge's own serial, so it only counts
toward agreement.
- Spelling differences are not disagreement: udev's underscores for
whitespace, a T10 designator for an ATA drive (`ATA`, the model, then
the serial), the agent's NAA field spelling, and udev's 64-bit prefix of
an NAA 6 WWN.
- The refusal runs after the unique-path check, so a second row on the
path keeps the match ambiguous.
- The merge's WWN step compares normalized identity
(`HardwareIdentityMatch`), so the agent's `5-c50-a1b2c3d4` finds the
Proxmox `0x5000c500a1b2c3d4` disk.
- `resolveLinkedPhysicalDisk` refuses the SAS same-path fallback when
the two WWNs name different disks.
- The poller records Proxmox's `unknown` serial/WWN as empty, so the
serial reads `missing` and the agent can fill it. Retained evidence
carries a previous serial only when it is usable, so a pre-upgrade
record cannot restore the placeholder. Identity keys already ignored the
placeholder. When the agent now fills a disk's serial, that disk's
Proxmox-poll SMART samples move from its WWN or path key to the serial
key, where the agent's own samples already go.
## Tests
- `TestHostAgentSMARTRowForSwappedOutDiskDoesNotLendItsIdentity` (sata
and sas) drives the real poller and registry. Proxmox reports `unknown`,
the agent fills the serial, the agent goes silent, and the disk is
swapped. On main, the SATA replacement takes the old serial and
resource; the SAS replacement keeps its own serial in state, but the
registry joins it to the old resource with its temperature and FAILED
health.
- `TestHostAgentSMARTMergeRefusesPathMatchAcrossContradictingIdentity`
has one row per producer rule: SAS/SATA/NVMe replacements refused;
same-disk SAS without an agent WWN, a renamed SAS disk in the agent's
WWN spelling, udev's serial spelling, a USB enclosure (typed `usb` and
typed `ssd`), a T10 designator and an NAA 6 volume merged; and an
ambiguous path.
- `TestRegistrySASPathJoinRefusesContradictingWWN`,
`TestProxmoxUnknownDiskIdentityIsRecordedAsUnreported` and
`TestHardwareIdentityConflict` cover the rest.
- Negative controls via `go test -overlay`: removing each rule fails
only its own case. These cover the guard, the WWN normalization, the
transport/vendor rule, compaction, the designator rule, the NAA 6
prefix, the registry guard, the ingest normalization and the
usable-serial carry. Against main's `monitor.go` and `registry.go`, the
end-to-end, merge and registry tests fail.
- `TestMergeHostAgentSMARTIntoDisks_PreservesExistingProxmoxIdentity`
merged a row with a different WWN by path alone, which the guard now
rightly refuses. Its row is now the same disk in the agent's WWN
spelling, with all assertions unchanged.
- `internal/monitoring`, `internal/unifiedresources` and
`pkg/diskinventory` pass in full.
## Not covered
- A replacement with no comparable identity still matches its path: no
usable serial or WWN at all, or a SAS/SCSI/USB disk with a WWN on one
side only.
- The registry's seeded source mapping, reused after a restart by the
slot-shaped Proxmox source ID, can still hand a swapped disk the old
canonical resource. That's a separate follow-up.
The Proxmox page warns when a node's SSH temperature monitoring still
runs the pre-rc.6 setup, which locks the key to `sensors -j` so SATA/SAS
disk temperatures never arrive. The notice judged that from the disk
rows of the active tab, and since each tab hydrates only its own
resource families, Overview loads no disks and never showed it.
The registry now derives the verdict on the node itself, after child
counts resolve each disk's canonical parent: a node with an available,
legacy-format temperature payload is flagged when a SMART-only disk
under it has no reading collected now. The page lists flagged nodes from
the rows every tab already loads, so Overview needs no disk inventory.
Every Proxmox node carries an explicit true or false, because the browser
merges a facet field by field and would keep an earlier true if a
cleared verdict were omitted.
Two rule fixes ride along with the move. Proxmox's own disk inventory
types SATA/SAS disks as hdd or ssd, so the old sata/sas-only set never
matched a node without a host agent. And a retained reading kept after
a disk stopped being read no longer counts as present
(diskinventory.TemperatureCollected).
Keep API-only monitoring from being mistaken for a backup-safe guest-agent-free path. Link the existing planned precaution and retain independent post-backup thaw, every-filesystem writes, liveness and prior-active restoration in both shipped quickstarts.
Contract-Neutral: Documentation and rendered-content tests only; no runtime, API, permissions or safety-contract change.
Change-source: pulse-maintainer
#2636 made the alert disk temperature policy the only judge of disk
heat, but it resolved that policy by disk type alone. Disk temperature
alerts (`CheckHost`) evaluate each SMART disk under the thresholds of
the host agent that reports it. An explicit Disk Temp override on that
host, or one inherited from its linked node or guest, replaces the
per-type trigger, and a host whose alerts are switched off raises
nothing. So a host with an 80C Disk Temp override still read **Running
Hot** on Physical Disks at the 70C NVMe trigger. Patrol flagged it too,
and a host with alerts switched off still read hot. The Alerts
thresholds page promises the opposite: "Setting a Disk Temp override on
a host above replaces these for all of that host's disks".
## What changes
- **Backend policy:** `alerts.Manager.HostDiskTemperatureThreshold(host,
diskType)` resolves the policy for a host the way `CheckHost` does, from
the same helpers (`resolveHostThresholdsNoLock`,
`hostThresholdOverrideNoLock`, `hostDiskTemperatureThresholdNoLock`). A
disabled override gives no threshold.
`DiskTemperatureThreshold(diskType)` is now its hostless case.
- **Patrol and the AI chat context:**
`ThresholdProvider.GetDiskTemperatureThreshold(host, diskType)`. Each
unified physical disk finds its host by walking its parent to the agent
machine, through the pool for Unraid array and cache disks.
- **Physical Disks, the disk detail, the pool drawer's linked disks and
the guest drawer** pass the reporting machine's override keys to
`resolveDiskTemperatureDisplayThresholds`. Keys follow `CheckHost`'s
order: agent ID, canonical resource ID, then the merged node or guest.
The resolver returns null (uncoloured, never hot) for an override or
agent default that switches the host's alerts off.
The browser rebuilds `CheckHost`'s key chain from what it loads, so a
few identity edge cases can still differ from alerts and Patrol (listed
in `storage-recovery.md`): on the Storage page, a disk whose agent an
operator manually linked into a VM; an override inherited through agent
link hints the API does not publish (an unmerged linked node or guest),
or stored under a linked node's own canonical ID; and guest keys in the
guest drawer for an agent merged into a guest it is not automatically
linked to. Overrides set on the agent's own row, or on the node or guest
it is merged with, reach every surface.
## Verification
- `TestHostDiskTemperatureThresholdMatchesCheckHostOverrides` checks
each case against what `CheckHost` actually fires: no override, host
override, host override without Disk Temp, an override under the
canonical resource the agent ID resolves to, linked-node and
linked-guest overrides, a host override shadowing a node one, a
switched-off Disk Temp, and a disabled host, linked node or linked
guest.
- `TestPatrolJudgesAgentDisksByTheAgentDiskTemperatureOverride` uses a
real registry with two agents, one overridden. A mutation that drops the
owner lookup fails it.
- Frontend tests cover the resolver, the owner walk and key order,
DiskList with three hosts, the presentation map, pool-drawer linked
disks and the guest drawer.
- Private mock stack: three NVMe disks pinned at 72C. Before (main
frontend): all three read Running Hot. After:
- pve1, with an 80C override, reads Healthy (green).
- pve2 still reads Running Hot.
- pve3, with alerts off, is uncoloured.
- Moving pve2's override onto its linked node ID turns it Healthy.
- Checked at 1440 and 375 with no overflow. The receipt is in
`frontend-modern/browser-verification/`.
- A read-only second-opinion review ran twice. Round 1's findings
(frontend key order, guest drawer canonical ID, disabled agent default,
doc wording) are fixed in the first commit. Round 2's are the parity
cases and the documented browser limits in the second commit.
Mock mode is one process-wide flag, but SetMockMode ended the epoch,
cleared the alerts, forgot the fixture agents and reset the state of
only the monitor it was called on. The server runs one monitor per
organization, each with its own alert manager and its own mock alert
pass. A switch made through one of them (the mock-mode API uses the
requesting tenant's monitor, the demo-fixture licence sync the default
one) left every other running monitor holding what it raised in the mode
the process had left: fixture alerts and fixture-agent node links in
live mode, live alerts in mock mode, until the 24-hour stale sweep.
Monitors now join a process-wide set of running monitors when Start runs
and leave it when Start returns, under the same mutex that already
serialized switches. SetMockMode flips the flag once, then ends the
epoch of the monitor it was called on and of every running monitor: it
waits for their admitted calls, clears their alerts, forgets their
fixture agents and resets their state. Only then does it restart each
running monitor's sampler, polls and discovery. Start picks the mock
metrics sampler or discovery while it holds that mutex, so no switch can
land between reading the mode and starting the runtime for it, and a
switch leaves the runtime of a monitor that has not joined yet to its
own Start. A monitor records the mode and the switch count it was built
under. If it missed switches between New and Start, it leaves what it
holds before it joins, dropping live alerts New restored or fixture
alerts a read path raised meanwhile, even when a round trip brought the
flag back. A call for the mode the flag already holds still moves a
reached monitor whose alerts belong to the other mode, and leaves the
rest alone.
Release builds reach this only on an instance that runs DemoMode with
the demo_fixtures entitlement, since release ValidateEnablement refuses
mock fixtures otherwise, and that has more than one organization's
monitor running. The licence sync only switches the default
organization. So the fix stays inside the monitoring package. The
callers are unchanged and no API surface moves.
New tests start the default and one other tenant monitor through
MultiTenantMonitor, as the server does, and switch through one of them.
Against the per-monitor switch, the default monitor kept its fixture
alerts (69 to 104 across runs) and all 69 fixture agents after mock mode
was left through the other tenant. That tenant kept its live alert after
mock mode was entered through the default monitor, and its fixture node
stayed suppressed by a fixture agent's leftover node link. Four more
tests fail if Start joins without leaving the mode its monitor was built
in, if Start compares only the mode after a round trip, if a call for
the flag's current mode returns without moving such a monitor, or if a
switch starts the runtime of a monitor that has not joined.
DemoModeMiddleware admitted any request carrying "Upgrade: websocket",
whatever its method or path, so a POST/PUT/PATCH/DELETE with that header
reached its handler on a read-only demo instance. The public demo signs
visitors in as the configured admin user, which RequireAdmin accepts and
which skips token scope checks, so the middleware was the only write
guard: adding two headers let a demo session switch the instance out of
mock mode, change system settings or create organisations.
The exemption never served a real handshake. RFC 6455 handshakes are GET
requests, gorilla's Upgrader refuses any other method, and GET already
passes the read-only branch above it, so /ws and /api/agent/ws keep
connecting. Removing the branch leaves the method alone deciding.
TestContract_DemoModeUpgradeHeadersDoNotExemptWrites signs in as the demo
admin through the full router, proves the session passes RequireAdmin,
gets 403 for a mock-mode POST with upgrade headers and still opens /ws.
On the old middleware the same POST returned 200 and turned mock mode off.
A resource with several attached availability checks exposes one
compatibility summary, Resource.Availability, and health reads only that.
The summary ranked every observed failure equally and broke ties by
target id, so a check that had failed once below its threshold could
hide a later-sorting check with a confirmed outage: the outage check's
own row carried availability_unreachable while the machine it monitors
read healthy.
Rank a confirmed outage above any other failure and have health use the
same per-check rule, so target health and row presentation follow a
confirmed outage on any attached check. The rule mirrors the poller's
incident gate: observed failures at the threshold, an unavailable
aggregate when one is reported, and no indeterminate outcome. A probe
agent that stops reporting keeps its old failure count but reads
indeterminate with an unknown aggregate, so it confirms nothing, on its
own row or on the target. Drawers already list every check from
availabilityChecks; the summary keeps its worst-check meaning.
Leaving mock mode cleared active alerts, but every path that evaluates
mode-dependent data checked the mode once, up front. An evaluation
already holding fixture data when the switch happened landed its alerts
after the clear, in live mode, where nothing evaluates or resolves
fixture resources, so they stayed until the 24-hour stale sweep.
A pass-level lock around `checkMockAlerts` was tried and reverted: it
closed only that path and made the mock-mode API wait behind a whole
pass of synchronous event-log writes.
**What changes**
- `mockModeFence` (`internal/monitoring/mock_mode_fence.go`): each path
takes a scope before reading anything mode-dependent and sends each
alert-manager call through `scope.run`. Covered: `checkMockAlerts`
(every loop, host agents, disks, snapshot cache), `checkBackupAlerts`,
`pollGuestSnapshots`, `checkConnectionAlerts`, `pruneStaleDockerAlerts`,
and the resource-store refresh (`updateResourceStore`,
`updateResourceStoreForRead` and `syncUnifiedResourceAlertsToState` now
take the scope of the state they rebuild from).
- `SetMockMode` is serialized, flips the mode, ends the epoch, waits
only for calls already admitted, then clears. Unified metrics go one
resource per call, so the wait covers single calls, not passes.
Admission never blocks.
- Registry rebuilds are tracked, not waited on: a refresh evaluates the
registry it read back only if no rebuild from an ended epoch overlapped
it. Readers that act on the inventory (Docker pruning, backup and
snapshot guest lookups, the pollers' previous-state carry) use current
state until an undisturbed rebuild in the new epoch replaces the fixture
registry.
- `mock.SupplementalRecords`/`SupplementalChanges` serve nothing outside
mock mode. The router swaps its mock adapters for the real pollers only
after the switch, refreshing between sources, and the adapters used to
fall back to default fixtures, so a single switch to live mode raised
fixture vSphere incidents.
**Verified**
- Each new test parks a real evaluation at a fixed point (a held lock, a
blocking supplier or a gated registry read), switches modes and releases
it, with bounded waits. Against the unfenced code: the mock pass left
about 40 fixture alerts, the frontend refresh 5 vSphere incidents, the
backup check 137 backup-age alerts, and a mock-mode connection
observation counted toward a live connection's confirmation. The drain,
republish, pruning and carry tests fail against a fence without its
wait, overlap check or registry tracking; the republish test also fails
with the old mock supplemental defaults.
- 995 tests from the touched files and related areas pass, plus
`internal/mock` and the `internal/api` mock/platform tests; the new
tests pass under `-race` and over repeated runs.
**Not covered**
- The fence and clear are per monitor while mock mode is process-wide;
other tenant monitors neither drain nor clear on a switch (unchanged,
noted in the monitoring contract).
- Report admission and live pollers outside the fenced paths can still
evaluate after the clear when entering mock mode; their alerts can
persist until mock mode is left.
Once every source with a verdict had gone quiet, the stale pass counted
any current availability sighting as online. A node that
preserveOrExpireNodes expired stayed online while its HTTP check failed,
and a quiet failing check lifted that node's offline verdict to warning.
One sighting stands for every check on a target and carries whichever
check was projected last, so a failing check that had just run also
vouched for another check's old pass.
Status aggregation now judges each check on a monitored resource by its
own evidence window. A passing check whose evidence is still valid
proves the resource answers, and reads online when no current source has
a verdict. Every other check abstains, current or quiet: a failing check
proves only that one port or service does not answer, so it never makes
a guest count as stopped in the workloads summary or the Proxmox page.
A check's own row keeps the ordinary rules, and a manual link judges the
merged resource by the checks it carries.
Projecting or retargeting a check now re-applies the stale pass's rule
to targets whose status that pass owns, since the resources API replays
checks after its stale pass and never runs another. Only a target's own
quiet sources give the pass that ownership, never its checks' sighting.
The poller now keeps each check's evidence tied to the observation that
passed. It ages out a local location that stopped running beside other
locations, as it already did for silent remote paths: the check took its
freshness from the newest location, so a failing remote report kept an
hour-old local pass counted as reachable with current evidence. An
available check's evidence lasts only as long as its newest reachable
path, and a remote result adopted as local after an entitlement lapse
keeps the server's receipt time rather than the agent's clock.
Since #2644 the machine drawer reads a silent host agent's Thermals and
Disks as last known, but the RAID card between them kept presenting the
agent's last report as current: a green "clean" array, an amber "clean,
degraded, recovering" array with "Rebuild: 45% · 98000K/sec", and
coloured member badges, as if the array were still rebuilding. The
Machines table's RAID tooltip did the same with an amber rebuild bar.
What changes
- `RaidCard` takes the drawer's `lastKnownReason`
(`agentReadingsLastKnownReason`, from `agent.stale`). Every array and
member state stays as evidence, but reads "<state> (last known)" in
muted text beside a muted dot, titled "Last known reading, not current:
host agent stopped reporting". A rebuild reads "Rebuild was at N%"
without its speed. Member badges lose their status colour, so a member
that was not healthy names its state ("/dev/sdb2 · faulty").
- The Machines RAID column (hidden by default) and its tooltip apply the
same rule from `agent.stale`: a muted summary with ", last known" for
screen readers, and a tooltip headed "RAID Arrays (last known)" without
the rebuild bar or speed.
- mdadm reports healthy members as "active sync", which the badge helper
only matched as an exact "active", so every healthy software RAID member
wore the amber warning badge. `raidPresentation.ts` now reads it as
healthy (failure states still win).
- RaidCard's empty guard ran once at mount, so a drawer opened before
the first RAID report never showed the card. It is now tracked.
- The array header row wraps, so at 390 wide the longer last-known state
takes its own line instead of truncating the array name to "m..".
Not changed: the guest drawer's Guest RAID card. The workload model maps
`agentRaid` but not the in-guest agent's staleness, so wiring it needs a
separate change across `useWorkloads.ts` and the other agent-sourced
guest rows.
Verification
- Private mock stack with a local-only overlay holding Apollo-114
offline past its lease, seeded with a clean md0 and a degraded,
rebuilding md1. The live control host Spectrum-709 carries the same
arrays. Checked the drawer card at 1920, 1440 and 390, the Machines RAID
cell and tooltip at 1920 (the column is container-hidden below that),
and the drawer in dark mode. Silent host: no emerald, amber or red
classes anywhere in the card or tooltip. Live host: unchanged colours
and rebuild bar, with healthy members now green. No console errors and
no horizontal scroll. Receipt:
`frontend-modern/browser-verification/05a685ccb29f5883.json`.
- New tests: `RaidCard.test.tsx` (current, last known, transitions both
ways, empty to populated), `raidPresentation.test.ts` (active sync and
last-known labels), an `AgentsMachinesTable.test.tsx` case (muted
summary, dot, no colour classes, no rebuild bar), and a
`SharedPrimitives.guardrails.test.ts` pin. All five fail against the
base source.
- `vitest` over `components/shared`, `components/Infrastructure`,
`components/Workloads`, `features/standalone`, `features/platformPage`
and `utils` passes, apart from the PTY tests in
`agentInstallCommand.test.ts`, which fail on macOS regardless of this
change. The pre-commit battery passes.
The storage pool rows the AI tools build (pulse_storage pools and
pulse_query storage pools) derived both enabled and active from "status
is not offline". Storage that reported itself unavailable, such as
Proxmox storage with active=0, reached Patrol and Assistant as
administratively disabled even when it is enabled in Proxmox.
Enabled now comes from the storage facet's own flag. Active needs the
facet's flag and a row that is not offline, because PBS datastores
report Active even while unavailable. Rows without a storage facet keep
the status-derived value.
TrueNAS pools and datasets were the only storage producers that left
both flags at their zero value, so reading the facet alone would have
turned every TrueNAS pool into a disabled one. They now report Enabled,
and Active from the pool state or the dataset's mount and lock state.
pulse-triage[bot]