Compose the exact current upstream shared-clock, last-known guest readings and platform-alert ownership repairs without rewriting the reviewed batch. Source proofs, final-content browser evidence and hosted checks remain separate publication prerequisites.
Change-source: pulse-maintainer
The overview hook kept its own minute signal and handed its value to
each card as "now". A card that mounted between ticks measured from up
to a minute earlier: an alert raised 70 seconds ago read "this minute",
a reading already past the 10-minute stale cut-off led the card as the
live one, a just-raised alert stayed out of Triggered (24h) until the
next tick, and the stale flip and 24h drop landed up to a minute late.
The card now reads useRelativeTimeNow for its started age and the live
reading's stale cut-off, and the hook reads it for the 24h count, so
they measure from the wall clock and move with the shared 30-second
tick. The minute interval stays only for the delivery diagnoses
refresh, a server read that should not double its rate.
`CleanupAlertsForNodes` runs on every Proxmox node poll
(`monitor_pve_storage.go`) and on every mock alert tick. It removes
alerts whose `Node` is not a current Proxmox node or PBS instance, and
keeps the types `shouldPreserveAlertOutsideNodeCleanup` lists.
Kubernetes, TrueNAS and vSphere metric alerts carry a cluster or
platform host as `Node`, and a display label (`TrueNAS Disk`,
`Kubernetes Pod`, `vSphere VM`) in `metadata.resourceType`. The preserve
list only had the bare words `truenas`, `kubernetes`, `k8s` and
`vmware`, so none of these labels matched. In an estate with Proxmox
plus one of those platforms:
- every Proxmox poll deleted these alerts without resolving them;
- their reducer incident stayed firing, so the next unified evaluation
re-created each one and handed it to the alert callback again as a new
alert.
## Change
`shouldPreserveAlertOutsideNodeCleanup` keeps an alert that carries
`metadata.platformType` (set on every unified Kubernetes/TrueNAS/vSphere
metric alert), or whose label maps through `CanonicalAlertResourceType`
to one of those platform types. The label path covers alerts persisted
before `platformType` existed. A guest alert whose Proxmox node left the
estate is still removed.
## Evidence
- Unit probe on main: a TrueNAS disk temperature alert, a Kubernetes pod
CPU alert and a vSphere VM CPU alert were all deleted by
`CleanupAlertsForNodes({"pve1": true})`. Over three
cleanup-then-evaluate cycles the TrueNAS alert went to the alert
callback 3 times; with this change, 0 times and it stays active.
- Private mock stack: TrueNAS disk temperature alerts kept vanishing
between mock alert ticks, while the disks kept their readings in
`/api/resources`. With this change they stay open.
## Tests
`TestCleanupAlertsForNodesKeepsPlatformMetricAlerts` (in
`internal/alerts/alerts_test.go`) raises a TrueNAS disk, a TrueNAS pool,
a Kubernetes pod and a vSphere VM alert, then runs the cleanup three
ways: as raised, with only `platformType` (label unrecognised), and with
only the label (`platformType` removed).
`TestCleanupAlertsForNodesStillRemovesMissingProxmoxNodeAlerts` keeps
the removal of a guest alert on a removed Proxmox node. Each preserve
condition was mutation-checked: removing either one fails the first
test.
A Pulse agent linked into a VM keeps its last report on the VM resource
after it stops reporting: the registry copies the agent's whole facet,
`agent.stale` included, onto the guest. The guest drawer never read that
flag. A quiet agent's Guest RAID kept its live colours and "Rebuild: 42%
· 118M/sec", its network rows and version read as current, and nothing
said the agent had gone.
## Change
- `useWorkloads` maps `agent.stale` onto the workload as `agentStale`,
and sets `disksFromAgent` when Proxmox reported no guest filesystems and
the agent's own `disks` filled in. Filesystem rows are the only agent
reading with a second source.
- When the agent is stale, the guest drawer:
- passes `HOST_AGENT_STOPPED_REPORTING_REASON` to `RaidCard`
(`lastKnownReason`, #2652);
- puts a status line over muted Network rows;
- marks the Pulse Agent row "· stopped reporting";
- reads agent-supplied filesystem rows through the existing `last-known`
disk read state.
- Agent-supplied filesystem rows now follow the agent's freshness
instead of the Proxmox guest-read reason. Before, a VM with a **live**
linked agent but no qemu-guest-agent showed the agent's filesystems as
"Usage unavailable · ?/328 GB" with advice to install qemu-guest-agent.
A stopped guest still reads those rows as unavailable, whether its
status or the Proxmox `vm-stopped` reason says so. Proxmox's own guest
filesystems keep their read state.
## Verification
- Private mock stack built from `main` plus this change. Two host agents
were held past their reporting lease and two left live, then each was
linked into a VM through `POST /api/resources/{vm}/link`. The two silent
VMs serve `agent.stale: true` with the agent's raid, disks and
interfaces; the two live ones carry no stale flag.
- Playwright at 1440 and 390:
- silent VMs read as described above;
- the live VM with agent-only filesystems shows usage with bars;
- the live control is unchanged;
- no horizontal overflow.
- New tests in `GuestDrawer.test.tsx` cover:
- silent agent;
- live agent with Proxmox agent-not-running;
- Proxmox filesystems under a silent agent;
- stopped guest, with and without the reason;
- stale flip in both directions on an open drawer.
The new tests fail against the old drawer. `useWorkloads.test.ts`
asserts source selection with distinct mountpoints.
- Frontend suites over Workloads, hooks, utils, shared, platformPage and
Infrastructure pass, apart from the Mac-only PTY tests in
`agentInstallCommand.test.ts`.
- WorkloadsSurface grows by 0.22 kB gzip (64.85 kB against a ~71.9 kB
cap), and the chunk set is unchanged.
## Not covered
- The workload table's disk cell and the drawer History still read the
aggregate disk metric with the Proxmox reason.
- The agent's OS, its addresses in the identity rows, and the lazily
loaded Physical Disks card's health and wearout carry no stale cue yet.
source-2h58772x exposed an outdated copy assertion left behind by 53e1bff13c. Check the current severity/tag guidance and its real help link, retaining both filter conjunction requirements. No reader, guide or routing behaviour changes.
Contract-Neutral: Documentation test repair for existing routing guidance; no runtime or subsystem contract changes.
Change-source: pulse-maintainer
Preserve exact Delivery commits 697608448a and 745c1f9ce5. Same graph and endpoints; only the observed transport-only error can retry. Refusals, integrity failures, unknown output, timeouts and signals remain terminal.
Change-source: pulse-maintainer
Apply the exact locked formatter output returned by source-jl9uxa9s; no guide or runtime content changes.
Contract-Neutral: Test formatting only; no subsystem contract delta.
Change-source: pulse-maintainer
Remove the record-limited but indefinitely blocking duplicate commands. Use the existing unfiltered Pulse readers, preserve NAS-specific safety guidance, and exercise the linked source and rendered help against failures, real deadlines, TERM resistance and missing timeout.
Contract-Neutral: Documentation and diagnostic-recipe verification only; no runtime, API, permission or subsystem behaviour changes.
Change-source: pulse-maintainer
Additively incorporate published demo-write admission, disk-replacement identity and Proxmox sensor guidance. Resolve the storage-recovery contract append conflict by retaining both the silent-source History rule and the demo-write rule unchanged.
Change-source: pulse-maintainer
`DemoModeMiddleware` let any request carrying `Upgrade: websocket`
through, whatever its method or path. On a `DEMO_MODE` instance a
`POST`/`PUT`/`PATCH`/`DELETE` with that header reached its handler, and
the read-only guard did nothing.
## How exposed the public demo was
- The public demo (`demo.pulserelay.pro`, v6.5.0) runs `DEMO_MODE=true`
with `PULSE_AUTH_USER=demo`, and its login page signs visitors in as
demo/demo. No proxy auth is configured. Caddy is a plain `reverse_proxy`
with no method filtering, and it forwards `Connection: Upgrade` and
`Upgrade: websocket` unchanged.
- That session is the configured admin.
`sessionUserHasSettingsAdminPrivileges` matches `cfg.AuthUser`, so
`RequireAdmin` accepts it, and session requests skip token scope checks.
Nothing in the handlers checks demo mode on its own, so this middleware
was the only write guard.
- Anyone using the public credentials could therefore make any admin
write by adding the two headers. That covers switching the demo out of
mock mode (`POST /api/system/mock-mode`), changing system settings,
creating organisations, importing config, changing the password and
adding notification targets. A CSRF token was needed, but the attacker
gets one from their own login.
- On the old code, the new contract test reproduces this end to end. A
demo/demo session's mock-mode POST with upgrade headers returned 200 and
turned mock mode off.
- I found no sign it was used. Over the last 30 days the demo journal
shows every "mock mode disabled" line paired with a process restart (new
PID). Config file mtimes match deploys. 5,782 ordinary writes were
blocked over the same period. Caddy keeps no access log for the demo, so
this is the absence of evidence, not proof.
## Change
The exemption is deleted rather than narrowed. RFC 6455 handshakes are
`GET` requests, gorilla's `Upgrader` refuses any other method, and `GET`
already passes the read-only branch above it. A narrowed `GET`-only
exemption would never run. `/ws` and `/api/agent/ws` keep connecting,
and the request method alone decides.
## Tests
- `TestContract_DemoModeUpgradeHeadersDoNotExemptWrites` (registered
api-contracts proof) runs against the full router. It signs in as the
configured demo admin and checks that the session passes `RequireAdmin`
for a read. A mock-mode POST carrying `Connection: Upgrade` and
`Upgrade: websocket` now gets the demo 403 and leaves mock mode on, and
a real `/ws` handshake from the same session still upgrades.
- `demo_middleware_test.go`: the table row that asserted the bypass now
expects 403. POST/PUT/PATCH/DELETE with handshake headers to mock-mode,
settings, orgs, `/ws` and `/api/agent/ws` all get 403. Real gorilla
handshakes to `/ws` and `/api/agent/ws` through the middleware still
upgrade and echo.
- Negative control: run against the old middleware via `go test
-overlay`, the new contract test and the upgrade-header test fail, and
the handshake test passes on both versions.
## Not changed here
- `auth.go` `isWebSocketUpgrade` and `configapi`
`explicitAPITokenFromRequest` still accept a `?token=` query credential
on any request with `Upgrade: websocket`. A valid, scoped token is still
required, so no access is gained, but the credential can travel in a
URL. The `ErrorHandler` skip in `middleware.go` only bypasses recovery
and metrics.
- Older and separate:
`/api/ai/sessions/{id}/{abort,summarize,fork,undo,redo}` change state on
`GET`, which the demo guard admits as reads. Tracked separately.
A TERM-only timeout relies on the child exiting. Keep a fixed five-second KILL backstop, preserve every timeout as a failure and bind the shell/Go command proofs to the actual bounded invocation.
Contract-Neutral: Internal build timeout containment; source graph, endpoints, checksum trust and artifact acceptance are unchanged.
Change-source: pulse-maintainer
Keep the published process-wide tenant switch while using the retained removal lifecycle for both agent and Docker fixture hosts in every ended epoch. Preserve both lifecycle regression tests; no candidate source is re-parented.
Change-source: pulse-maintainer
`ingestRecord` reuses an existing source-key mapping before any identity
matching. For physical disks it checked only the type and parent scope.
Proxmox keys a disk by its slot (`ProxmoxPhysicalDiskSourceID`, e.g.
`pve1-node1--dev-sdb`), so in a registry seeded from persisted unified
resources, a disk swapped into the slot mapped straight back to the
previous disk's canonical resource. Where an agent report had been
merged in, `mergeInto` also kept the old disk's serial, WWN, temperature
and FAILED health. This was found while reviewing #2658, and its
contract note said the seeded mapping was not covered.
**Change.** A Proxmox disk observation now refuses the mapping when the
two disks carry hardware identities naming different disks
(`diskinventory.HardwareIdentityConflict` from #2658):
- Only fields reported on both sides count. A missing serial or WWN,
including Proxmox's literal `unknown`, keeps the mapping.
- Differing serials do not count for SAS, because Proxmox may report the
SAS address there (#1595).
- They also do not count when the mapped disk was merged with an agent
report and Proxmox did not read the drive's own serial. This is the same
rule monitoring's `hostAgentSMARTSerialComparable` applies: NVMe on both
sides, or agent SATA under Proxmox vendor `ATA`, and never a Proxmox
`usb` disk.
- The refused key falls through to the linked-disk join, `findMatch` and
`chooseNewID`, but none of them may merge it back into the refused disk.
An ID landing on it is replaced with the key's source-specific ID.
**Reachability.** No current caller replays Proxmox disk records over a
seeded mapping. The monitor rebuilds a fresh registry each poll. The
resource API, host-continuity and Patrol registries seeded from unified
resources do not replay Proxmox disk records. This closes the registry
contract rather than a reported symptom.
**Proof.** `TestRegistrySeededSlotMappingRefusesReplacedDisk` (in
`internal/monitoring/issue1595_collection_trust_test.go`) builds the
previous disk through `IngestSnapshot` with fixtures shaped like
`monitor_pve`'s output, rehydrates with `IngestResources(List())`, polls
Proxmox, then rehydrates a second time.
- 5 replacement cases: SATA with and without a new WWN, SAS, a
USB-enclosure swap ingested Proxmox-first, and a disk Proxmox alone
reports.
- 8 same-disk controls: re-ingested, no serial, a SAS address serial,
SAS without a WWN, a USB bridge serial, a bridge serial under vendor
`ATA`, a SCSI designator, and a SCSI designator from an agent without
collection status.
Negative controls via `go test -overlay` each fail only their intended
cases: the whole fix removed, the refused-disk exclusion removed, serial
comparability always true or always false, the Proxmox-only branch
removed, a collection-status provenance rule instead, and the USB veto
removed. The combined tree of main, #2660 and this change also passes
`internal/unifiedresources` and the monitoring disk tests.
**Residuals (in the unified-resources contract).**
- Drives that share a serial they do not own, such as a USB bridge's,
still read as one disk, as they do in identity matching.
- The refused disk keeps its Proxmox facet until a rebuild, so the
slot's PVE alert reference resolves as ambiguous rather than to either
disk.
Since the per-tab Proxmox queries landed, only the Storage tab loads
`physical_disk` rows. So the outdated sensor setup notice, which judged
nodes from the active tab's disk rows, never showed on Overview.
The registry now derives the verdict on each Proxmox node
(`proxmox.sensorSetupOutdated`, an explicit true or false on every node)
right after child counts resolve each disk's canonical parent. A node is
flagged when its temperature payload is available and in the legacy
`sensors -j` format and a SATA/SAS disk under it has no reading
collected now (`diskinventory.TemperatureCollected`). The page lists
flagged nodes from the rows Overview, Storage, Replication and Backups
already load, so no tab hydrates disks for it.
Rule fixes that came with the move:
- Proxmox's disk inventory types SATA/SAS disks as `hdd`/`ssd`, so the
old `sata`/`sas`-only check never matched a node without a host agent.
- A retained reading no longer counts as current (carried over from
#2659's frontend rule).
- The verdict is sent as explicit false, because the browser's
field-by-field facet merge would otherwise keep an earlier true after
the condition clears.
Stacked on #2659 (shares its `sensorSetup.ts` edit, which this
replaces).
Verified on a private mock stack at 1440px and 390px:
- Overview, Storage, Replication and Backups show the notice for
legacy-format nodes with an `hdd` disk without a reading, a retained
SATA reading, and a SATA disk without a reading.
- Overview makes no `physical_disk` request.
- With `main`'s frontend swapped in, Overview showed nothing and Storage
named one node of three.
- A node whose disk starts reporting drops out on the next websocket
update without a reload.
Keep the same pinned Go graph and endpoint. Retry only the observed transport-only diagnostic, at most three attempts; stop on refusals, integrity errors, unknown output, timeouts and signals. Execute the actual Docker download command against controlled outcomes and retain its original-base rejection.
Contract-Neutral: Build-only transport recovery; source graph, endpoints, checksum trust and artifact acceptance are unchanged.
Change-source: pulse-maintainer
Match automatic system-mount exclusions at directory boundaries rather than textual prefixes. Local /snapshots, /runtime and /var/lib/docker-data volumes retain capacity and usage through agent collection, both report paths and QGA aggregation; genuine system and remote/container mounts remain filtered. Regression controls include host usage alert onset and recovery.
Contract-Neutral: Restore existing filesystem eligibility without changing schemas, identities, explicit operator patterns or guest-command safety policy.
Change-source: pulse-maintainer
Apply the exact locked Prettier output returned by the offline source-proof VM. The earlier check stopped before lint or renderer tests; this changes only the test assertion layout.
Contract-Neutral: Test formatting only; documentation, installer, API and runtime contracts are unchanged.
Change-source: pulse-maintainer
Follow-up to #2656 (the per-monitor mock-mode epoch fence), which
documented this gap as not covered.
## Problem
Mock mode is one process-wide flag (`mock.SetEnabled`), but
`Monitor.SetMockMode` ended the fence epoch, cleared the alerts, forgot
the fixture agents and reset the state of only the monitor it was called
on. The server runs one `Monitor` per organization
(`MultiTenantMonitor.GetMonitor` → `New` + `Start`), and each runs its
own mock alert pass into its own alert manager. A switch made through
one monitor (the mock-mode API uses the requesting tenant's monitor; the
demo-fixture licence sync uses the default one) left every other running
monitor holding what it raised in the mode the process had left: fixture
alerts and fixture-agent node links in live mode, live alerts in mock
mode, until the 24-hour stale sweep.
## Fix
- `internal/monitoring/mock_mode_switch.go`: a process-wide set of
running monitors, guarded by the existing `mockModeSwitchMu`. `Start`
joins it and leaves when its loop returns. While joining it picks the
mock metrics sampler or discovery under the lock (now
`startModeRuntime`), so no switch lands between reading the mode and
starting the runtime for it.
- `SetMockMode` flips the flag once, then for the receiver and every
running monitor: ends the epoch, waits for admitted calls, clears
alerts, forgets fixture agents (when leaving mock mode) and resets state
(`endMockModeEpoch`). Only after that does it restart the sampler, polls
and discovery of each monitor in the set (`resumeAfterMockModeSwitch`).
A monitor that has not joined yet gets its runtime from its own `Start`.
- Each monitor records the mode its alerts and state belong to
(`mockModeAligned`, set in `New` from the same read that decides whether
persisted alerts are restored) and the number of switches made so far
(`mockModeSwitchesSeen`). If it missed switches between `New` and
`Start`, it leaves what it holds before joining, even after a round trip
that brought the flag back. A call for the mode the flag already holds
still moves any reached monitor that belongs to the other mode, and
leaves the rest alone.
Callers are unchanged and no API surface moves.
## Is this reachable in release builds?
Barely. In a `release` build `mockruntime.ValidateEnablement` refuses
mock fixtures unless `SetReleaseFixturesAuthorized(true)` was called.
Its only caller is `syncReleaseDemoFixtureRuntime`, which requires
`DemoMode` plus the `demo_fixtures` entitlement and runs only for the
default organization. Several tenant monitors in mock mode would also
need more than one organization's monitor running on that demo instance.
Demo mode's read-only middleware normally blocks creating organizations
and the mock-mode API's POST. It does forward any request carrying
`Upgrade: websocket` whatever its method; that bypass predates this
change and is being fixed separately. So in practice this is a
dev/mock-stack problem, and the fix stays inside the monitoring package.
## Tests
New tests in `internal/monitoring/monitor_mock_alerts_test.go`:
- `TestLeavingMockModeClearsEveryRunningTenantMonitor` and
`TestEnteringMockModeClearsEveryRunningTenantMonitor` start the default
and one other tenant monitor through `MultiTenantMonitor`, as the server
does, and wait until both have joined. They raise alerts in both, switch
through one, and check both.
- `TestMonitorStartedAfterAMockModeSwitchLeavesTheModeItWasBuiltIn`
covers a switch that lands between `New` and `Start`.
- `TestMonitorStartedAfterAMockModeRoundTripDropsWhatItRaisedMeanwhile`
covers two switches in that window, with fixture alerts raised in
between.
- `TestSetMockModeAlignsAMonitorWhoseModeTheFlagAlreadyHolds` covers a
call for the mode the flag already holds.
- `TestMockModeSwitchLeavesAnUnjoinedMonitorsRuntimeToItsStart` covers a
switch reaching a monitor that has not joined.
One more is in
`internal/monitoring/monitor_host_agent_removal_lifecycle_test.go`:
`TestLeavingMockModeReleasesFixtureAgentNodeLinksOnEveryRunningMonitor`.
It switches through the default monitor and checks that the other
tenant's fixture node is no longer suppressed by a fixture agent's
leftover node link.
Negative controls (`go test -overlay`):
| Control | Failing tests | What happened |
|---|---|---|
| Fence-PR runtime | Leaving, entering, node-link | Default monitor kept
69–104 fixture alerts (varies by run) and 69 fixture agents; org-b kept
its live alert; org-b's `mock-cluster-1-pve1` stayed suppressed |
| Set kept, switch reaches only the receiver | Same three | Same
failures |
| No realign at join |
`TestMonitorStartedAfterAMockModeSwitchLeavesTheModeItWasBuiltIn` | |
| Join compares only the mode, not the switch count |
`TestMonitorStartedAfterAMockModeRoundTripDropsWhatItRaisedMeanwhile` |
Started in live mode with the fixture node's memory alert |
| Plain early return when the flag already matches |
`TestSetMockModeAlignsAMonitorWhoseModeTheFlagAlreadyHolds` | |
| Resume ignores set membership |
`TestMockModeSwitchLeavesAnUnjoinedMonitorsRuntimeToItsStart` | |
Every control failed only the tests it targets.
Also run: the full `internal/monitoring` package; `-race -count=3` on
the new tests plus the existing `TestLeavingMockMode*` and
`TestMockModeFence*`; and the mock-mode tests in `internal/api` and
`internal/api/configapi`.
Replace the old build and commercial narrative with setup, recipient safety, schedule checks and honest retained-data limits in the existing Intelligence guide. Keep the old repository guide as a pointer and verify the shipped Markdown with meaningful actual-parent controls.
Contract-Neutral: Existing-surface help and rendered-documentation verification only; no runtime, route, permission or interface change.
Change-source: pulse-maintainer
The reviewed Core candidate has completed pure capacity checks, but its synthetic polling and History lifecycle was not executed. Correct the contract note so fixture presence or compilation cannot be read as source-polling or native recovery acceptance. Runtime source and retained adverse evidence are unchanged.
Change-source: pulse-maintainer
Preserve the exact Core commit. Retained exact-final local race checks and actual-parent failures adequately cover this pure aggregation change, including Linux, alias and overflow controls. Polling and History execution, native recovery and the earlier broad timeout remain unproved or adverse; this is not acceptance of those outcomes.
Change-source: pulse-maintainer
Preserve the exact three Delivery commits. Retained source-8bf8uicz proves the unchanged final shell gate and its 26 consent cases under race with syntax and vet checks. The updater-only proof mapping preserves unrelated installer requirements; installed timer and update acceptance remain separate.
Change-source: pulse-maintainer
Bring the general Unix private-file bootstrap up to the existing PBS guide's safety floor. Preserve token and installer files, ignore curl configuration, require verified HTTPS and HTTP 200 before publishing the script, and exercise refusal and partial-download paths.
Contract-Neutral: Documentation and executable documentation tests only; no installer, API, runtime or capability contract changes.
Change-source: pulse-maintainer
A machine with several availability checks shows one summary check, and
its health reads only that summary. The summary ranked every observed
failure equally and broke ties by target id. So a check that had failed
once, below its threshold, could hide a later-sorting check with a
confirmed outage. The outage check's own row carried
`availability_unreachable` while the machine it monitors read healthy.
- `availabilityCheckPriority` now ranks a confirmed outage first, then
any other observed failure, then unchecked/ambiguous/unresolved, then
passing. Ties still go to the first target id.
- `availabilityOutageConfirmed` is the one confirmation rule, shared by
the ranking and `EvaluateResourceHealth`. It mirrors the poller's
incident gate: observed failures at the threshold, an unavailable
aggregate when one is reported, and no indeterminate outcome. A probe
agent that stops reporting keeps its old failure count but confirms
nothing, on its own row or on the target.
- Tests in `availability_link_test.go` go through the registry
projection path: a confirmed outage behind an earlier unconfirmed
failure (and its recovery), and a silent probe agent's stale failure
count. Both fail without the change. `health_test.go`'s legacy fixture
gains the `LastChecked` every producer sets.
- The `unified-resources.md` contract describes the summary ranking and
the shared rule.
Drawers already list every check from `availabilityChecks`, and
availability incidents live on each check's own row, so alert synthesis
is unchanged.
Follows #2643, which made the stale pass read each source's own verdict
but left availability checks counting by delivery.
## Problem
Once every source with a verdict had gone quiet, any current
availability sighting kept its target online. A node that
`preserveOrExpireNodes` expired stayed online while its HTTP check
failed, and a quiet failing check lifted that node's offline verdict to
warning. One sighting stands for every check on a target and carries
whichever check was projected last, so a failing check that had just run
also vouched for another check's old pass.
## Change
- **Registry verdict** (`aggregateStatus`,
`availabilityChecksProveOnline`): each check on a monitored resource is
judged by its own evidence window (`Evidence.ValidUntil`). An enabled,
passing check with current evidence proves the resource answers and
reads online once one of the resource's own sources has gone quiet and
none that is current has a verdict. Every other check abstains, current
or quiet. A failing probe never makes a guest count as stopped
(workloads summary, Proxmox page counts). A check's own row keeps the
ordinary rules.
- **Ownership**: only a target's own quiet sources hand the stale pass
its status. Projecting, re-running or retargeting a check re-applies the
stale pass's rule to targets it owns, because the resources API replays
checks after its stale pass. A manual link judges the merged resource by
the checks it carries.
- **Poller evidence** (`internal/monitoring`):
- A local location beside other locations ages out like a silent remote
path ("no recent local check").
- An available check's evidence lasts only as long as its newest
reachable path.
- After an entitlement lapse, an adopted remote result keeps the earlier
of its agent check time and the server receipt time.
- A multi-location aggregate is never adopted as a local observation;
the target reads as never checked until the local poller runs.
Contracts: `unified-resources.md` ("Status follows each source's own
verdict") replaces the residual paragraph, and `monitoring.md` covers
the multi-location rules.
## Verification
- New tests in `registry_test.go`, `registry_merge_policy_test.go` and
`availability_probe_agent_test.go`. Each fails against the base runtime
via `go test -overlay`, including both projection orders.
- Full `internal/unifiedresources` and `internal/monitoring` suites
pass, plus the availability, connection, workload and resource-state
tests in `internal/api`, `resourceapi` and `chartapi`.
Retain the exact reviewed documentation and formatting commits. Existing update help separates checkout builds from installed executables, preserves edition and state, and removes shared temporary configuration writes and assumed recovery. Six retained rendered-help passes and original-input controls remain applicable; no runtime change or native/browser acceptance is claimed.
Change-source: pulse-maintainer
Register the real shell verifier for the updater without admitting it as proof for unrelated installers. Preserve exact installer requirements and test missing/unrelated proof rejection.
Change-source: pulse-maintainer
The broader installer race/vet VM attempt reached readiness but returned no command verdict at its observation deadline. Keep the same consent cases in the normal installer suite and make their committed file standalone, with only standard-library imports and the actual shell gate. This enables proportionate exact-source Bash regression proof without building unrelated application dependencies; the broader failed attempt remains retained.
Change-source: pulse-maintainer
Add the exact current upstream without rewriting accepted commits. Retain reviewed monitoring, history and documentation repairs alongside published disk identity, retained-temperature, age refresh and cache cleanup changes. Publication and final-source proof remain separate.
Change-source: pulse-maintainer
The overview hook kept its own minute signal and handed its value to
each card as "now". A card that mounted between ticks measured from up
to a minute earlier: an alert raised 70 seconds ago read "this minute",
a reading already past the 10-minute stale cut-off led the card as the
live one, a just-raised alert stayed out of Triggered (24h) until the
next tick, and the stale flip and 24h drop landed up to a minute late.
The card now reads useRelativeTimeNow for its started age and the live
reading's stale cut-off, and the hook reads it for the 24h count, so
they measure from the wall clock and move with the shared 30-second
tick. The minute interval stays only for the delivery diagnoses
refresh, a server read that should not double its rate.
Normalization keeps a disk's last temperature when the current
observation did not collect one, such as a disk in standby or a host
agent past its reporting lease. #2603 taught the Storage presenters to
show that value as last known. Four surfaces still read the raw number
as a current reading. Each now goes through the canonical
`getPhysicalDiskTemperaturePresentation` /
`isPhysicalDiskTemperatureCurrent` instead of adding a predicate.
- **TrueNAS disk drawer** (`resourceDetailDrawerTrueNASModel.ts`): the
Temperature row took the 55°C warning tone from a retained reading, and
the summary listed it as a reading. It now reads "61°C (last known)"
with the reason as title and no tone, and the summary leaves it out. A
current reading keeps the existing tone. Replacing the fixed 55 with the
alert policy is a separate follow-up.
- **Drawer History fallback**
(`resourceDetailDrawerMetricsHistoryModel.ts`): with no stored points, a
retained reading was labelled "current". It now goes through the
deferred-metric slot, so the legend says "last known" and the reason
shows above the chart. A disk's value is read from `physicalDisk` beside
its collection state. Websocket rows carry no top-level `temperature`
for a disk, so on the TrueNAS page the fallback had shown "-" even for a
current disk.
- **TrueNAS storage table** (`TrueNASStorageTopologyTable.tsx`): the
Temp cell showed a retained reading plainly, and hottest-first sorted it
to the top of its group. It now renders muted with its reason, and the
sort ranks it as no reading.
- **Proxmox outdated sensor setup notice** (`sensorSetup.ts`): a
retained reading counted as a temperature arriving, so the notice could
hide for a SATA disk Pulse cannot read now.
Verification: a regression test per surface fails on main and passes
here. A private mock stack with a local-only overlay stamped one TrueNAS
disk and one pve3 SATA disk with an unavailable temperature state ("host
agent stopped reporting") and marked pve3's sensor payload legacy.
Playwright at 1440×900 and 390×844 compared the change against the same
session with main's runtime files. The receipt is
`frontend-modern/browser-verification/294409f678589ef2.json`.
CleanupAlertsForNodes runs on every Proxmox node poll and removes alerts
whose node is not a current Proxmox node or PBS instance. Kubernetes,
TrueNAS and vSphere metric alerts carry a cluster or platform host as
their node and a display label such as "TrueNAS Disk" in
metadata.resourceType, which the preserve list's platform words never
matched. So in an estate with Proxmox and one of those platforms, every
poll deleted these alerts silently while their reducer incident stayed
firing, and the next evaluation re-created each one and handed it to the
alert callback again as a new alert.
Keep alerts that carry metadata.platformType, or whose label maps to a
Kubernetes, TrueNAS or vSphere type. A guest alert on a removed Proxmox
node is still removed.
The Workloads table history hook was the last caller of
`infrastructureSummaryCache.ts` and `workloadsSummaryCache.ts`, and
nothing has read either cache since the InfrastructureSummary card and
the standalone Workloads summary were deleted on 2026-05-26. The hook
runs both fetchers inside `createNonSuspendingQuery`, which always
passes an abort signal, so their in-flight dedupe was skipped too. Each
30-second Trends poll still serialized the full response on the main
thread and wrote it to localStorage or, above the 900k-character cap,
deleted the key.
Measured on embedded builds against the 50-node, 929-guest mock estate
at the 1h range:
| | Before | After |
|---|---|---|
| JSON serialized per poll | 9,629,902 + 924,249 chars (5-31 ms each) |
none over 200k |
| Summary-cache storage calls | `removeItem` on both keys every poll |
none |
| `/api/charts/workloads` + `/api/charts/infrastructure` | one each on
load and per poll | same |
| Rows drawing history (1440 / 390 wide) | 12 nodes + 140 guests / 6 +
36 | same |
The hook now calls `ChartsAPI.getWorkloadCharts` and
`ChartsAPI.getInfrastructureSummaryCharts` directly with the same range,
signal, `maxPoints`, node scope and metric list. The node, agent and
Docker-host series fold moves into `workloadMetricHistoryModel.ts` with
its merge rule unchanged. The query layer already retains reads across
remounts for five minutes and clears them on org switch. The hook
replaces the deleted cache under the `workloads-hot-path` policy.
The unified-resources, performance-and-scalability and
frontend-primitives contracts still described InfrastructureSummary, its
model and state hook, the shared summary table, selector and details
drawer, WorkloadsSummary, throughputPresentation, resourceBadges and the
two caches as live owners. Each clause now names the live owner, becomes
a conditional rule, or is removed where nothing it governs remains.
Stale prose for the other summary primitives deleted in May
(SummaryMetricCard, InteractiveSparkline, DensityMap, StorageSummary)
and their test-only helpers is left to a separate change.
Checks: affected vitest suites (10,053 tests; the only failures are the
Mac-only `agentInstallCommand` suite), contract, registry and status
audits, and the browser verification receipt.
Count QGA Windows volume GUIDs rather than their shared physical drive; without a GUID retain distinct drive paths. Preserve visible mount metadata, Linux deduplication and byte-overflow containment. Exercise QEMU-only polling, removal and measured-zero History, adapting guest fixtures to the current captured mock-mode scope.
Contract-Neutral: Correct existing guest filesystem capacity aggregation and fixture scope compatibility without changing schemas, identity, command admission or backup policy.
Change-source: pulse-maintainer
## Problem
`ingestRecord` refuses a physical-disk match under another parent
(`findMatch` skips scope-incompatible candidates), then `chooseNewID`
mints the same hardware-keyed canonical ID (`machine:<serial>`) and the
ingest merges into whatever disk already holds it. Disks that report one
usable serial on several machines collapse into a single resource
parented to whichever machine ingested last, and every other machine
loses the disk from inventory. Real triggers: dual-ported SAS shelves
seen by two nodes, cloned VMs with an explicit identical serial,
fixed-serial USB bridges, TrueNAS systems. Repro: `IngestSnapshot` with
three PVE nodes in one instance, each with a `/dev/sda` of the same
serial, gave one disk.
## Fix
- A new disk resolves its parent to a machine: the nearest host
ancestor, so a TrueNAS pool or Unraid array counts as its system or
host.
- Two disks with one identity stay apart only on evidence: one reporter
placing the serial on two machines (cloned VMs keep both serial and
hostname), or machines whose hostnames share nothing. Reporters on one
box still join across grouping parents and unmerged host resources, so
the TrueNAS API and the agent on that box stay one disk. An unknown
machine never splits a disk, so a PVE row whose node row is missing
still joins its host's agent disk.
- Once a serial spans machines, every copy takes the serial keyed to its
own machine, and the holder of the unscoped ID is re-keyed with its
source mappings, retired-ID claims and relationship endpoints. IDs don't
depend on machine ingest order, and the unscoped ID never passes between
disks that coexist. A disk only one machine reports keeps its current
ID.
- Operator splits ("Split merged resource" exclusions) now hold on this
path; the old collision merge ignored them. A split recorded against any
ID the disk can hold applies, so it survives a re-key.
- The registry indexes disks by hardware key (`physicalDisksByHardware`)
because `IdentityMatcher` keeps one resource per machine ID.
## ID stability decision
Machine-scoped IDs only for identities that actually span machines,
rather than a pre-pass. A pre-pass needs final parent scopes before
ingest, but host/node merging happens during ingest. Giving the
first-ingested machine the unscoped ID would hand one machine's ID, and
its operator state, to another machine when an earlier-sorting machine
appears. With the chosen design a disk's ID changes when a same-serial
disk first appears on another machine. That orphans ID-keyed state
rather than misapplying it.
## Residuals (documented in unified-resources.md)
- Like any serial-keyed ID, the unscoped ID still passes to a
same-serial disk that appears once the previous holder is gone.
- Alert-history rows owned under the unscoped ID don't follow a re-key.
The PVE disk alert owner (#2646) already leaves shared-serial rows
unbound unless a WWN tells them apart.
- Agent SMART and Unraid disk source IDs are the bare serial.
Same-serial agent disks on different hosts share one `SourceAgent`
mapping, and only the last ingested keeps the agent source target. This
is tracked as a separate task, because the fix re-keys persisted
mappings.
- Same-serial disks share one metrics history key.
- Across reporters, hostnames that normalize alike still join.
## Verification
- New tests in `registry_merge_policy_test.go`: per-machine disks for
PVE-only and linked-agent clusters, including reversed ingest order;
rehydration with a machine joining; agent + TrueNAS on one box versus a
cloned TrueNAS with the same hostname; a disk moving into a pool; a
parentless PVE row on a shared serial; exclusions honoured, surviving a
re-key, with per-machine fallback; serialized-JSON rehydration;
retired-ID claims carried across a re-key.
- Every new test fails against the unfixed registry or with its fix
removed (`-overlay` negative controls).
- `go test ./internal/unifiedresources` passes, as do the
disk/SMART/Unraid/TrueNAS tests in `internal/monitoring`, `internal/api`
and `internal/truenas`.
- Five rounds of independent review; the last round found no P1/P2
issues.
Retain the exact German and Spanish documentation candidate and its rendered safety controls. Runtime frontend and browser receipts are unchanged.
Change-source: pulse-maintainer
Text matching admitted truncated configuration, nested opt-in and padded RC channels. Parse one configuration object with the installed jq dependency and fail closed before downloads when consent cannot be verified. Preserve the legacy no-config timer opt-in, manual updates and disabled-timer gate; add executed-shell regression cases and the deployment contract.
Change-source: pulse-maintainer
Follow-up to #2578: the remaining Alerts and Patrol panels whose
relative ages froze while the panel stayed open over unchanged data.
**External watchdog (Alerts › Destinations).** The status was read once
at mount, so Last success and Monitor progress kept their read-time age
through a real stall, which is the case the panel exists to show. It now
re-reads `GET /api/alerts/deadman/status` every 30 seconds in the
background (the Refresh button is untouched; a failed re-read keeps the
last status and marks it unavailable), and both ages read the shared
clock.
**Delivery log.** Each row's hover age reads the clock. A logged
attempt's time never changes, so its age is right over a log that is
only re-read on demand.
**Patrol.** The paused banner kept "Blocked 2m ago" because each status
re-read returns the same `blocked_at`. The selected-run banner kept its
age because the selected run is held as picked. The attention detail is
read once per selection, so its acknowledgement, transition and evidence
times now read the clock. Its Last seen and flapping window describe a
record that may have moved since, so they stay read-time (the new
frontend-primitives paragraph states that exception). Queue rows,
summary, run history and the digest already re-derive on their own
30/60-second re-reads, so they are unchanged.
**Verification.** Fake-timer tests per surface (each fails on main and
passes here). Live on a private mock stack with Playwright `page.clock`
and the data stream held (one live socket kept by a data-free keep-alive
per fake minute), at 1440 and 390 wide. After ten held minutes: watchdog
33s → 10 mins ago with 20 background re-reads; delivery tooltip 5 → 15
mins ago; "Blocked 2m → 12m ago"; "Patrol run 3m → 13m ago"; attention
detail 3/5/7m → 13/15/17m ago. The same-session control on main's files
stayed frozen everywhere.
Found while auditing, filed separately: future times ("snoozed until",
"Returns to active attention") render as "just now" because
`formatRelativeTime` is past-only.
**Review follow-ups (second-opinion review, three passes).** The status
read has no timeout, so the first poll design could misbehave:
- **Stuck Refresh:** a hung background read blocked Refresh; separately,
a Refresh superseded by a hung read stayed disabled.
- **Slow reads:** reads slower than the 30s poll were all discarded,
failures included, so an outage could read "Heartbeat healthy".
- **False recovery:** an older late success could clear a newer failure.
Each read now gets a sequence number, and an outcome applies only when
it is newer than the last read that settled. A hung Refresh is freed at
the first poll once it has been pending 29s (one second of slack,
because live runs showed ticks landing a few ms short of 30s). Unit
tests cover each case and fail on the earlier versions. Each case was
also checked live with held reads answered in and out of order, with an
earlier-version control. Main is merged in, with a receipt for the
combined `PatrolAttentionWorkbench.tsx` (#2591).
Retain exact reviewed Core commit 6dcff07343. Its dedicated regression tests, both parent-production controls, full Proxmox race result and focused monitoring race/vet receipts support the existing counter invariant repair. The broad monitoring timeout remains adverse evidence, not whole-suite acceptance; native QEMU-only recovery and containing publication remain separate.
Change-source: pulse-maintainer
Compose the upstream frontier observed at this coordination turn's launch. Preserve its reviewed commit identities and exact frontend content; do not reconcile later upstream movement in this bounded batch.
Change-source: pulse-maintainer
Apply the locked formatter to the new rendered-help checks; documentation and test behaviour are unchanged.
Contract-Neutral: Formatting only in documentation verification; no runtime or contract delta.
Change-source: pulse-maintainer
Builds on #2637 (alert card live reading), so this branch carries its
commits until that PR merges.
## Why
Alerts > History is a log of occurrences, but its open row is the one
occurrence still in progress: it shows an Active badge and a running
duration. Beside them, the breach message "Node temperature at 95.0°C"
reads as the reading now, while the node sits at 78°C under its 80°C
trigger and the alert is held until it clears. With #2637 the Alerts
card one tab away says "Temperature 78°C now, back under the 80°C alert
level", so the two pages disagreed about the same alert (the confusion
in #2068).
## What changes
- An active History row for a threshold alert with a usable live status
leads with the same presentation summary as the card
(`getMetricAlertPresentation`, via a new `getAlertHistoryRowCopy` in
`alertHistoryModel.ts`).
- Desktop: the clear rule and the last breach move into the message
cell's hover. Phone: they show as muted lines under the reading, since a
phone has no hover.
- The row's `description` stays the recorded message, so search ("95.0"
still finds the held row) and the Assistant handoff read the record.
- Closed rows, and open rows without a live status (non-threshold
alerts, Pulse system alerts, the moments after a restart), are
unchanged.
- The reading rechecks on the shared relative-time clock, read only for
open rows, so it turns stale ("Last reading: …, 12 mins ago") when
evaluations stop without re-rendering closed rows on each tick.
## Verification
- Private mock stack built from this tree with a local-only generator
overlay pinning pve2's CPU temperature: walked 95 → 78 → 72 °C at
1440×900 and 390×844, light and dark; earlier pass also covered
breaching, °F, and the alert resolving out of the list.
- New tests: model copy and fallbacks (`alertHistoryModel.test.ts`), the
phone card (`AlertHistoryMobileList.test.tsx`), and the desktop row
following a live store update and turning stale on the clock
(`AlertHistoryTableAlertRow.test.tsx`).
- Receipt: `frontend-modern/browser-verification/9c0e53268f0d014f.json`.
Refs #2068
Preserve exact reviewed tip b5b1fa9777 and upstream a85bdcdc9a additively. Retain both alert threshold test sets. Combine upstream mock-mode epoch fencing with reviewed structural-revision checks and Docker departure ownership; add a regression that a current revision cannot revive an ended epoch. Resolve the monitoring contract to describe both boundaries, without rewriting accepted history.
Publication remains deferred for genuine final-content browser receipts and independent CodeQL disposition. Composition checks do not establish native recovery, deployment, exact-head hosted acceptance or release readiness.
Change-source: pulse-maintainer
A Pulse agent linked into a VM keeps its last report on the VM resource
after it stops reporting: the registry copies the agent's whole facet,
agent.stale included, onto the guest. The guest drawer never read that
flag, so a quiet agent's RAID kept its live colours and "Rebuild: 42% ·
118M/sec", its network rows and version read as current, and nothing
said the agent had gone.
- useWorkloads maps agent.stale onto the row as agentStale, and sets
disksFromAgent when Proxmox reported no guest filesystems and the
agent's own disks filled in.
- The guest drawer passes the stopped-reporting reason to RaidCard, puts
a status line over muted network rows, marks the Pulse Agent row as
stopped reporting, and reads agent-supplied filesystem rows as last
known.
- Agent-supplied filesystem rows follow the agent's freshness instead of
the Proxmox guest-read reason. A VM with a live linked agent but no
qemu-guest-agent showed the agent's filesystems as "Usage unavailable"
with advice to install qemu-guest-agent. A stopped VM keeps the Proxmox
reason, and Proxmox's own guest filesystems keep their read state.
The workload table's disk cell, the drawer History, and the agent's OS,
addresses and physical disk health still carry no stale cue.
ingestRecord reuses an existing source-key mapping before any identity
matching, and for physical disks it checked only the type and parent
scope. Proxmox keys a disk by its slot, so in a registry seeded from
persisted unified resources a disk swapped into the slot mapped straight
back to the previous disk's canonical resource. Where an agent report had
been merged in, mergeInto also kept that disk's serial, WWN, temperature
and failed health.
A Proxmox disk observation now refuses the mapping when the two disks
carry hardware identities naming different disks
(diskinventory.HardwareIdentityConflict). Only fields reported on both
sides count. Differing serials do not count for SAS (Proxmox may report
the SAS address, #1595), or when the mapped disk was merged with an agent
report and Proxmox did not read the drive's own serial, by the rule
monitoring's SMART merge already uses. Collection status cannot decide
that: a legacy agent's restored serial keeps Proxmox's status.
The refused key falls through to the linked-disk join, findMatch and
chooseNewID, but none of them may merge it back into the refused disk:
an ID landing on it is replaced with the key's source-specific ID.
Without that, a disk keyed by a USB bridge serial re-merged at once.
No current caller replays Proxmox disk records over seeded mappings: the
monitor rebuilds a fresh registry each poll. This closes the registry
contract the seeded-mapping comment promises.
Separate development build output from the installed service. Retire the legacy shared-temporary-file and unconditional rollback recipes in favour of the current deployment-aware guide, with rendered link and safety checks.
Contract-Neutral: Documentation and rendered-help verification only; no updater, service, API, build or recovery behaviour changes.
Change-source: pulse-maintainer
pulse-triage[bot]