Carry cancellation through PVE/PBS construction, cluster probes and retained sessions, startup retries and port fallback. Preserve prior health/errors on caller termination; accepted clients retain independent ordinary reads and full timeouts. Cover real pinned HTTPS cancellation, deadlines, form fallback and first-poll publication.
Contract-Neutral: Restore existing caller cancellation and usable-session guarantees; no wire/API schema, agent lifecycle, TLS trust, guest safety, polling cadence or History contract delta.
Change-source: pulse-maintainer
Stage both enumeration routes' guest readings, snapshots, alerts, backup eligibility and History until their complete cycle is admitted. Retirement, runtime replacement and transport changes cancel obsolete reads and serialize state withdrawal against admitted effects without holding the monitor mutex through callbacks.
Retain authoritative inventory after detail-budget expiry, independent agents and accepted cache safety; cancellation is not command completion. Add deterministic late-reply and publication-order controls.
Change-source: pulse-maintainer
Invalidate successful and negative QGA memory caches on guest unavailability, observed stops and connection retirement. Retain and cancel obsolete read owners until completion, discard late/cancelled samples and coalesce overlapping refreshes without weakening transport uncertainty. Verify both collectors, served readings, History and independent linked-agent evidence.
Change-source: pulse-maintainer
Cancel obsolete enrichment contexts on explicit cache invalidation and PVE instance retirement while retaining ownership until completion. Serialize publication and retry scheduling with invalidation, re-read fallback metadata within the reservation, and discard late network, OS and version outcomes without weakening shared guest-command uncertainty fences. Cover late readers, slot waiting, direct/cluster HTTP cancellation and ordinary independent recovery.
Change-source: pulse-maintainer
Separate metadata-cache activity from successful observation age so empty or failing guest reads cannot erase OS-info suppression or its accumulated failures. Preserve useful and OS evidence ages, ordinary backoff and guest-command fences; expire only idle non-in-flight state and clean activity on explicit invalidation and instance retirement.
Change-source: pulse-maintainer
Close background catch-up admission atomically with its wait group, cancel between whole lifecycle events and keep the applied watermark before closing history. Start joins replay admitted by router attachment; Stop also covers a monitor that never started. Retain blocked-writer ownership through the existing tenant deletion budget, and resume the unapplied tail in a new monitor.
Contract-Neutral: Preserve canonical occurrence history, notification policy, tenant isolation and bounded deletion refusal; make existing background projection writers obey monitor lifetime without changing public API or delivery behaviour.
Change-source: pulse-maintainer
Capture store, tenant and provider selection once across source ownership, records and consume-once changes. Keep provider callbacks outside locks, refuse replaced collections, and order non-queueing read refreshes with ingest. Preserve admitted history and leave skipped changes pending.
Contract-Neutral: Internal dependency/publication ownership only; resource schemas, tenant authorization, notification policy, thresholds, retention and shutdown semantics are unchanged.
Change-source: pulse-maintainer
Apply saved notification policy before initial resource publication, batch supplemental bindings and retain explicit withdrawals for existing and future tenants. Serialize complete router initialization and refuse old binding collections before ordered writes; preserve admitted alert joins without clearing active alerts. Keep native and demo provider records separated across the mock-switch hook gap.
Contract-Neutral: Internal resource dependency wiring and publication ordering only; existing resource/API schemas, authorization and tenant boundaries, alert thresholds, retention and shutdown policy are unchanged.
Change-source: pulse-maintainer
Monitor.Start launched mock and connection alert passes without joining them. Tenant deletion could close stores and remove its directory while a pass still emitted alert writes. Wait for admitted passes, refuse new checks after cancellation, and retain the existing bounded-deletion refusal. Log real persistence deletion errors without exposing them in the API response.
Contract-Neutral: Preserve token scopes, tenant isolation, API responses and deletion safety contracts; bind existing internal alert passes to their monitor lifetime and retain failure diagnostics.
Change-source: pulse-maintainer
The mock-mode unified view caches its linked build keyed on
sameManualLinks, which compared each link's resource pair and primary
and said link metadata, time included, does not change a fold. Among
otherwise equal candidates, chained links now fold into the primary of
the earliest-created link (linkChainRootLocked), so that is false for
the creation time: a list that differs only in a time can leave a
different row standing. In a cycle of a Proxmox storage, a TrueNAS VM
and a vSphere VM the TrueNAS VM keeps its row while its link is older
than the vSphere VM's and the vSphere VM keeps it once it is newer, yet
with the fixture data version unchanged the cached view kept serving
the old row while the live rebuild, which reloads the links from the
store, served the new one.
Compare CreatedAt too. The comparison has three readers and all take the
fix: the data-version cache hit, the decision to keep the previous
freshness when a view is published, and the structure view that serves
callers reading identity and topology.
The regression test replaces the TrueNAS VM's link with one that differs
only in its time, with no fixture tick, and expects the surviving row a
mock.UnifiedResourceSnapshotWithLinks build keeps: through the structure
view and the data-version view when the time moves later, and through the
data-version view when it moves back. Removing the CreatedAt term fails
it, and so does reverting any one of the three readers on its own. The
monitoring and unified-resources contracts now say the key includes the
creation time.
Follow-up to #2778. Stacks cleanly on main; the contract prose
cross-references the wiring change in #2796, so land that first if both
are open.
## What was wrong
In mock mode the unified view is keyed on the fixture data version,
which advances on every metric tick, and building it for the default
1,776-resource estate costs about half a second of CPU (many seconds on
a starved process). #2778 moved the metrics-target lookup onto the
fixture structure revision. The other callers that read only what the
estate lists and how it connects still rebuilt the whole view after each
tick: at startup they made 42 view requests, all misses (the
Docker-alert prune 18, the connection snapshots 13, the host-agent
evaluation 8, the chart prewarm 2, the metrics-target lookup 1), and
with no client connected the monitor tick alone rebuilt the estate about
every nine seconds.
## What changes
- Callers that read identity and topology share the view built for the
current structure revision and operator links
(`currentStructureUnifiedStateView`, `GetUnifiedStructureReadState`,
`currentModeStructureReadState`, in
`internal/monitoring/unified_structure_view.go`; `mockMetricsTarget`
uses the same cache reader): the Docker-alert prune, the workload chart
prewarm, the demo Patrol operator-state resolver, and the host-agent
evaluation. `evaluateHostAgents`, which runs on every monitor tick, now
lists nodes only when an offline host names a linked node.
- The connection-degraded alert feed keeps only platform rows
(`snapshotConnectionsForAlerts` drops every agent row, and a PBS row's
reported node name only becomes a host alias that its snapshot does not
carry). It now builds the aggregator inputs with `platformRowsOnly`, so
it no longer reads the hosts, their desired-config fingerprints or the
PBS instances. The connections endpoint, diagnostics and runtime
inventory still read them.
- Deliberately not moved:
- The mock storage chart (`resolveMockStorageTotal`): a linked pool
takes its capacity from whichever source was seen last, which a tick can
move without a structure change. It stays on the data-version view, read
once per request instead of once per uncached pool.
- `linkedHostForNode`, `getHostAgentTemperatureForNode` and
`carriedTemperatureOutlivesAgentLease` (only reachable with
`PULSE_MOCK_KEEP_REAL_POLLING`): they read a host's status, disks,
sensors and report lease, which a tick moves. Hoisting them would need
the read state threaded through the node poller, and they run after
startup, so they stay as they were.
Real mode is unchanged.
## Measured (pulse-dev, probe builds, one pinned core, GOMAXPROCS=1,
host load 30 to 110, single runs)
| | #2796 alone | with this change |
|---|---|---|
| estate view builds before the listener answers | 2 | 1 |
| estate builds, no client connected, 299 s of life | 33 | 2 (both at
startup) |
| process CPU over a 180 s steady window | 12.3 s | 3.3 s |
A separate run of the pre-#2796 build used 9.0 s over that window, so
read 12.3 against 3.3 as the comparison rather than as exact
percentages. The public demo host runs a mock estate, so the
steady-state saving is the one that matters there.
## Verification
- New tests (in registered proof files): no consumer of the structure
view rebuilds the view after a fixture tick (Docker prune, structure
node listing, workload prewarm, host-agent evaluation including an
offline host that names a linked node); the data-version view still
rebuilds; the structure view follows a structural fixture change and an
operator link; lazy correlation lists nodes only for a linked host; a
storage chart reads the estate only when it is not cached; the alert
feed does not rebuild the view, its platform rows equal the full
aggregator's, and configured PVE/PBS/PMG rows survive without the hosts;
the Patrol resolver does not rebuild the view.
- Negative controls (each fails the named test): prune, prewarm, eager
node listing, structure node listing, alert feed and Patrol resolver
each moved back to the data-version view, and a storage chart that reads
the estate before checking its cache.
- `go vet` and the targeted tests on pulse-dev.
## Known limits
- The structure view can be several ticks old, so a caller on it must
not read status, metric values, timestamps, sensors or anything a link
merges by freshness; the contract says so. The fixture sets pod node
names (which feed the platform aliases a Patrol reference can resolve
through) when the estate is built, not on a tick.
- `pbsInstanceFromReadStateView` never copies `NodeName`, so
`pbsReportedNodeNamesByInstance` appears to feed empty host aliases on
the connections endpoint. That is a separate bug, not changed here.
- The batch sharing of the storage chart's read (`sync.OnceValue` in
`GetStorageMetricsForChartBatch`) is not pinned by a test; the
single-chart laziness is.
Not a feature: no demand-ledger entry.
In mock mode the unified view is keyed on the fixture data version, which
advances on every metric tick, and building it for the default
1,776-resource estate costs about half a second of CPU (many seconds on a
starved process). #2778 moved the metrics-target lookup onto the fixture
structure revision; the other callers that read only what the estate
lists and how it connects still rebuilt the whole view after each tick.
At startup they made 42 view requests, all misses, and with no client
connected the monitor tick alone rebuilt the estate about every nine
seconds.
Callers that read identity and topology now share the view built for the
current structure revision and operator links (currentStructureUnifiedStateView,
GetUnifiedStructureReadState): the Docker-alert prune, the workload chart
prewarm, the demo Patrol resolver, and the host-agent evaluation, which
also lists nodes only when an offline host names a linked node. The
connection-degraded alert feed keeps only platform rows, so it no longer
reads the hosts, their desired-config fingerprints or the PBS instances
that only feed agent rows and a host alias its snapshot drops.
The mock storage chart is not one of them: a linked pool takes its
capacity from whichever source was seen last, which a tick can move
without a structure change. It keeps the data-version view, read once per
request instead of once per uncached pool. The per-node polling lookups
of a monitor that keeps real polling in mock mode (linkedHostForNode,
getHostAgentTemperatureForNode, carriedTemperatureOutlivesAgentLease)
read a host's status, disks and sensors, which a tick moves, so they stay
on the data-version view.
On pulse-dev (one pinned core, host load 30 to 110, probe builds): a
start built the estate view twice before the listener answered with the
wiring change alone and once with this; with no client connected the
process built the estate 33 times in 299 s and used 12.3 s of CPU over a
180 s window, against twice and 3.3 s with this change.
The PBS poll records the hostname the node reports about itself on
PBSInstance.NodeName so /api/connections can alias the PBS connection by
it and group the host agent running on that machine into the PBS row.
The name reaches the unified PBS resource (PBSData.NodeName), but
Monitor.PBSInstancesSnapshot() rebuilds each instance from the read-state
view, and the view had no accessor for it, so the snapshot always carried
an empty NodeName. The ledger's reported-name map was therefore always
empty and a PBS machine configured by IP or DNS alias still showed as an
API row plus a separate agent row. The unit tests missed it because they
hand the aggregator its inputs instead of going through the monitor.
Add PBSInstanceView.NodeName() and copy it in pbsInstanceFromReadStateView.
The new ledger test drives a real monitor through the input builder, the
connection aliases and the system grouping: with the conversion line
removed it fails on the missing alias, with it the agent attaches to the
PBS system.
Only direct root@pam sessions can read the node name (API-token
connections get none), so this restores grouping for password-auth PBS
connections; token-auth ones still match only on what they were
configured with.
Splitting a merged physical disk (the drawer's "Split merged resource",
POST /api/resources/{id}/report-merge) recorded its exclusions and the
registry kept two rows, but the PVE disk poller had already paired the
linked agent's SMART row with the Proxmox disk by WWN, serial or path
before the registry saw either, and read no exclusions. The split Proxmox
row still carried the agent's attributes, I/O, health, any serial, WWN,
type or temperature it filled, and AgentSMARTReported, which hands the
disk's temperature alert to the agent's CheckHost. #2748 listed this as
the part of the split it did not reach.
mergeHostAgentSMARTIntoDisks now asks the resource store
(MonitorAdapter.ProxmoxDiskAgentSMARTSplit) once per matched row. The
answer is physicalDiskSplitLocked's, applied to the agent's disk as the
registry generation holds it and to the Proxmox observation under its own
ID, so the exclusions that keep the registry's rows apart keep the
poller's pair apart: an exclusion naming only the agent's candidate
splits nothing, a replacement in the slot inherits no split recorded
against the disk it replaced, and a manual link that leaves the
generation holding both observations as one disk pairs them again.
A split disk takes nothing from the row. That includes the node-sensor
temperature the agent supplied: the node poll takes its SMART list from a
reporting agent whose report carries a usable reading, so that reading is
the agent's row of the other disk; an SSH-supplied reading stays and the
PVE disk check judges it. The record is marked AgentSMARTSplit, so the
poll's evidence retention restores nothing into it, and the
Proxmox-query-failure fallback gives it no stand-in row.
The split applies on the first full disk poll that collects the node from
Proxmox and matches the agent's row; the unified-resources contract names
that lag and the other limits.
An operator's split of a Proxmox node and its pulse-agent (report-merge
or unlink) split the registry's rows in #2741, but the monitor's own
link (Node.LinkedAgentID, Host.LinkedNodeID) stayed. Everything that
reads that link kept treating two machines as one: the agent's SMART
inventory as the node's disk fallback, its --disk-exclude patterns on
the node's disks, guest discovery through the agent, deployment
reporting the node as already running an agent, service discovery, the
agent's container filesystem readings, the node's temperatures and the
node/agent alert correlation.
The resource store's exclusion is now the one record of the split.
MonitorAdapter decides it with the registry's own rule for a node's link
and confirms a split against the store's latest decisions, and
models.State holds both sides of a split link back on every link write,
before a link can steer where node views fold. The agents API's node
link is the operator's newer decision, so it removes the pair's split
from the store (TakeExclusion) before recording manual intent, and
puts it back, unless the operator decided again meanwhile, if the
intent fails to persist (RestoreExclusion). A split recorded after a
manual link ends that intent on the agent's next report that finds the
split confirmed, so the node is no longer reserved for that agent.
A rebuild that cannot read the store's decisions still loads none, in
the registry as in monitoring; that needs a change in registry.go and
monitor_adapter.go and is left for a follow-up.
Alert evaluation resolves a metrics target for every resource and
windowed metric. In mock mode that went through the unified view cache,
keyed on the fixture data version, which advances on every 2 s metric
tick. Building the view for the default 1,776-resource estate costs about
half a second of CPU, so a process starved of CPU published each build
under an already-stale version and the next lookup built again. Startup
runs the first alert pass before the listener opens, so it never got
there: 675 of 929 view requests missed at a 16 s average build, and the
process had not served at 606 s.
A target depends only on a resource's type and source mappings, which a
metric tick never changes. Key the lookup on a new structure revision
(mock.FixtureStructureRevision, the existing structural-only counter)
and the link list instead, and answer from the targets the cached view
captured. An ID the estate does not list resolves to nothing without a
build. The view itself stays keyed on the data version for every other
consumer. The fixture now advances its data version before its structure
revision, and a cache fill reads the revision first, so a reader that sees
a new revision cannot cache the previous estate's snapshot under it.
This is not a regression from 1a094646d7..d2ac0c5112: both commits stall
identically under equal CPU contention and both start in 14 to 16 s at
normal load. Real mode does not pay the cost in steady state either; a
full store pass over 1,694 resources takes about 108 ms.
On pulse-dev, with the process pinned to one core behind four busy
loops, the fixed binary serves at 148 s where the previous behaviour had
not served at 606 s. The remaining startup time there is one view build
per consumer per tick, and the monitor's repeated full resource-store
passes; neither is changed here.
## Problem
In mock mode, operator manual links (`POST /api/resources/{id}/link`)
never reached the websocket broadcast or `/api/state` `resources`, even
after a restart, while `GET /api/resources/{id}` showed the pair merged.
`Monitor.currentUnifiedStateView` builds the mock view from the fixture
graph, and both the fixture build and the view's re-ingest used
`unifiedresources.NewRegistry(nil)`, so no links were loaded. The
resources API seeds its own store-backed registry from that same list,
which is why it showed the merge.
## Change
- `mock.UnifiedResourceSnapshotWithLinks(links)` builds the fixture
graph in `unifiedresources.NewRegistryWithManualLinks(links)`, a
registry with the links but no store. The links apply at each ingest
stage (snapshot, TrueNAS and vSphere records, then availability), as in
the live rebuild, so an availability check that names the folded
resource by source still projects onto the primary. Nothing the fixture
build ingests can write identity pins, successions or change rows.
Without links the shared memoized snapshot is used, as before.
- The monitor takes the links from its resource store's current
generation (`MonitorAdapter.ManualLinks`). It keys the cached mock view
on the fixture data version and the link list, so a link change rebuilds
the view without waiting for a fixture tick.
- The view's freshness is the key for the resources API's registry
cache, so it is now assigned in publication order. A view with the
published fixture freshness and links keeps the published value. Any
other view gets a value later than every view before it, so a slow build
that publishes late cannot reuse one.
- The real-mode fence view behind `currentModeReadState` still leaves
the links out. It only covers the short window after leaving mock mode,
before the registry rebuilds. Its comment now says why.
## Verification
- Private mock stack, websocket not blocked. Linked standalone agent
Apollo-114 into VM checkout-web-01.
- Before: REST showed the agent facet while `/api/state` and the
websocket `initialState` still listed the agent as its own row. Machines
still listed both linked agents.
- After: REST, `/api/state` and `initialState` agree, and they still
agree after a restart.
- On a websocket held open across the link, a `resourceDelta` upserted
the VM with sources `[proxmox, agent]` and removed the agent.
- A headless browser kept the Machines table open while the link was
posted from outside the page. The agent left the table in 3.6 s, with no
reload.
- New tests:
- `TestMockUnifiedViewAppliesOperatorManualLinks`
(monitor_host_agents_test.go)
- `TestFixtureGraphAppliesManualLinksAtEachIngestStage` and
`TestUnifiedResourceSnapshotWithLinksLeavesTheSharedSnapshotUnlinked`
(platform_fixtures_test.go)
- `TestRegistryWithManualLinksFoldsLikeTheStoreBackedRegistry`
(registry_test.go)
- Negative controls, run with `go test -overlay`. Each one fails at the
expected assertion:
- main's monitor.go
- a `NewRegistryWithManualLinks` that drops its links
- monitor.go without the publication-order freshness rule
- `internal/unifiedresources/...`, `internal/monitoring/...` and
`internal/mock/...` pass on pulse-dev. In `internal/api/...` there are
two failures, and neither is from this change.
`TestServerInfoEndpointReportsDevelopment` fails because the synced tree
has no `.git`. `TestContainerBootstrapFailuresStopAndClean/hangup`
failed under load and passed 3 of 3 runs alone.
## Interplay with #2725 and #2739
Those landed while this was open. Unlink now deletes the pair's link, so
it reaches the mock view through `ManualLinks` at the adapter's next
rebuild, the same timing as live. The broadcast's presentation
coalescing now takes exclusions from the resource store adapter for
store-less views, and the mock view is one of those. The merge commit
updates the unified-resources notes for both changes.
Resolve agent-lifecycle.md by keeping both sides (additive notes), and
bring the unified-resources notes in line with #2725 and #2739: unlink now
reaches the mock view through ManualLinks, and broadcast presentation
coalescing honours exclusions for store-less views.
Additively incorporate the exact main repair that landed during this publication turn. Preserve every starting reviewed commit and the deferred browser qualification boundary.
Change-source: pulse-maintainer
The resources API rebuilds its registry from the monitor's unified
seed, but built it with NewRegistry and plain ingests, so its stale
pass used the registry defaults while the monitor judges each source
by its configured polling cadence. A Proxmox node polled every two
minutes read stale, with warning status, in /api/resources ninety
seconds after its poll while the websocket called it online.
Presentation coalescing had the same gap in both views. When it merges
an agent row the registry kept apart into its Proxmox node by hostname,
it picked each reading's source by the default thresholds, so the row
showed a silent agent's readings over a poll the registry calls
current.
The seed now carries the thresholds its registry generation was judged
by. The monitor adapter records them per generation: thresholds set
since the last rebuild wait for the rebuild that applies them, and the
read-state overlays and live supplemental refreshes of a generation are
judged by its thresholds, since re-judging a listing cannot clear a
stale verdict it already carries. The monitor lists a generation with
its thresholds from one registry pointer, and a view built from mock
fixtures or a bare snapshot reports none (the defaults it was judged
by). The monitor hands seed, freshness and thresholds over from one
view, per tenant too. The resources API builds
its registry with them (NewRegistryWithStaleThresholds: ingests and
stale passes that pass none use them) and keys its cache on its own
copy. The coalesce judges metric freshness with metricSourceStale, the
rule the stale pass and the merge already share, and both the
broadcast and ListForPresentation pass their view's thresholds. The
broadcast coalesce also honours the operator's merge exclusions, so the
adapter's CoalesceForPresentation takes the listing's thresholds next
to them.
Patrol's scoped runtime state still rebuilds registries with the
defaults.
Preserve the exact upstream commits already present at coordination start; later upstream reconciliation remains with the publication gate.
Change-source: pulse-maintainer
After a restart, an agent an operator linked into a live resource, such
as an agent inside a vSphere VM, exists only through saved-host
continuity until it reports again. Three consumers then disagreed about
what a reference to it meant. The read state kept the saved agent as its
own offline row and resolved references to it there. The resources API,
which seeds a registry from that listing and re-applies the links,
folded it into the VM and merged its stale telemetry. Alert intent
resolved through the published registry, where the agent is absent, so
its agent:<host ID> alerts kept their literal reference. Maintenance or
NeverAutoRemediate set on the VM reached neither the agent's alerts nor
its Patrol findings in that window, while the API said it applied.
An operator link names one identity; telemetry separation is the only
reason the saved row stays apart. Link application now holds a saved
row beside the link's primary instead of merging it, in every registry.
A reference that resolves to the saved row (its canonical ID,
agent:<host ID>, its machine ID or an alias it shares with the primary)
resolves to the primary, as a folded row's references do, while each
row keeps its own telemetry. An in-memory marker on rows continuity
introduced rides clones, so the resources API's registry holds them the
same way. A saved primary leaves a live member its own identity.
Alert intent now resolves references and ancestors through the published
registry with saved hosts overlaid, the read state Patrol resolves
through. That also lets intent set on an unlinked saved host reach its
offline alert. The resolver runs under the alert manager's lock once
per metric check, so the overlay is kept per published registry
generation, publication builds it before alerts are evaluated, and a
build that overlaps a mock-mode switch is discarded.
Compose upstream b7bdf1b4eb available at batch start without re-parenting published or reviewed commits. Resolve concurrent append-only contract and regression-test additions by retaining both the existing partial-inventory controls and published presentation-split controls. No frontend source changes.
Change-source: pulse-maintainer
In mock mode the websocket broadcast and /api/state serve a view of the
fixture graph, and the fixture graph was built in a registry with no
store, so it never loaded the operator's manual links. The resources API
seeds its own store-backed registry from that list and applied them, so
a standalone agent linked into a VM showed merged in /api/resources and
stayed a separate row in the broadcast, even after a restart. Peers
verifying link UI on mock stacks had to block the websocket to see it.
The mock view now builds the fixture graph with the links the monitor
adapter's current registry generation loaded (MonitorAdapter.ManualLinks),
through NewRegistryWithManualLinks: a registry with those links and no
store, so fixture ingest cannot persist identity pins, successions or
change rows. The links apply at each ingest stage, as in the live
rebuild, so an availability check that names the folded resource by
source still projects onto the primary. Without links the shared
memoized snapshot is used as before.
The cached view is keyed on the fixture version and the link list, so a
link change rebuilds it without a fixture tick. Its freshness, which the
resources API keys its registry cache on, is assigned in publication
order: a view with the published fixture freshness and links keeps the
published value, and any other view gets a later one than every view
before it.
The fence view behind currentModeReadState still omits links; it only
covers the window after leaving mock mode, before the registry rebuilds.
POST /api/resources/{id}/unlink recorded only an exclusion. The link row
stayed, and applyManualLinks never consults exclusions, so the monitor's
rebuilds, the resources API and the websocket kept the pair folded while
the API answered "Resources unlinked".
A pair now carries one operator decision. AddExclusion deletes the pair's
links and AddLink its exclusions, in one transaction and in either ID
order. Reads keep each pair's latest decision, so rows written before
this change, or rewritten onto one pair by canonical-ID succession,
resolve by their later write, and a registry resolves its two reads
together so an unlink landing between them cannot load both.
While a Proxmox node was linked to an agent, the merged resource's
identity pin could take the agent's machine key. After unlink the node
completed that key, minted the agent's ID and folded into it by ID, where
no exclusion check runs. A pin no longer completes a record the operator
split from the pinned resource.
The broadcast coalesced host views without exclusions while REST's
ListForPresentation honoured them, so a split host pair was one websocket
row and two REST rows. The broadcast now coalesces through the read state
that listed the rows, with the same exclusion filter.
Additively incorporate exact upstream 36d3f8c557 while retaining all accepted local commits and proposal 31eccb559e. Resolve the single adjacent insertion conflict by retaining both reviewed observation-gap interruption and upstream metric tag cleanup. Preserve upstream shared guest and disk-type threshold judgement, phone drawer sizing, future Patrol timing and POST-only Assistant mutations. Exact final validation and outstanding browser/security dispositions remain required before advancing publication.
Change-source: pulse-maintainer
Use a dedicated shared mutex for every monitor random-source access, including lazy poll-backoff initialisation. Preserve the existing seeded jitter, intervals and nil-source metadata policy. Cover deterministic source ownership, mixed concurrent access and independent real HTTP guest enrichment.
Contract-Neutral: Correct internal random-source ownership without changing public interfaces, refresh or backoff durations, distributions, guest commands, lock checks or uncertainty fences.
Change-source: pulse-maintainer
Honour cold-cache retry backoff and retain a per-key in-flight reservation until enrichment returns, rather than treating a timed hold as completion. Avoid duplicate command sequences and stale safety-state writers while preserving existing lock, uncertainty, observation and refresh policies. Cover blocked real HTTP reads, early deferral, slot cancellation and ordinary recovery.
Contract-Neutral: Restore the existing per-key metadata reservation and retry scheduling semantics; no public API, configuration, guest command, lock, uncertainty fence, observation or numeric duration changes.
Change-source: pulse-maintainer
A Proxmox disk that only the node's sensors report never raised a
temperature alert: CheckDiskHealth judges health and wearout only, and
unified evaluation covers TrueNAS disks, not PVE ones. Physical Disks
could read the disk Running Hot under the per-type thresholds while
alerts stayed silent, and whether a disk alerted depended on whether a
Pulse agent happened to run on the node.
The PVE disk poller now hands each disk's current reading to
CheckProxmoxDiskTemperature, which judges it by the same disk
temperature policy as agent and TrueNAS disks (DiskTempByType under the
agent Disk Temp default) under the disk's existing PVE alert reference.
A disk a still-reporting linked agent lists in its SMART report is the
agent's: CheckHost alerts on it and the PVE alert closes as moved to the
agent, so no disk alerts twice. A retained reading holds the alert,
an excluded device or a switched-off policy closes it, warnings wait the
same 300s stability window as other disk temperatures, a config save
re-judges the alert per disk type, and resource history owns its rows by
the disk's recorded identity like the health and wearout alerts.
Keep the published process-wide tenant switch while using the retained removal lifecycle for both agent and Docker fixture hosts in every ended epoch. Preserve both lifecycle regression tests; no candidate source is re-parented.
Change-source: pulse-maintainer
Follow-up to #2656 (the per-monitor mock-mode epoch fence), which
documented this gap as not covered.
## Problem
Mock mode is one process-wide flag (`mock.SetEnabled`), but
`Monitor.SetMockMode` ended the fence epoch, cleared the alerts, forgot
the fixture agents and reset the state of only the monitor it was called
on. The server runs one `Monitor` per organization
(`MultiTenantMonitor.GetMonitor` → `New` + `Start`), and each runs its
own mock alert pass into its own alert manager. A switch made through
one monitor (the mock-mode API uses the requesting tenant's monitor; the
demo-fixture licence sync uses the default one) left every other running
monitor holding what it raised in the mode the process had left: fixture
alerts and fixture-agent node links in live mode, live alerts in mock
mode, until the 24-hour stale sweep.
## Fix
- `internal/monitoring/mock_mode_switch.go`: a process-wide set of
running monitors, guarded by the existing `mockModeSwitchMu`. `Start`
joins it and leaves when its loop returns. While joining it picks the
mock metrics sampler or discovery under the lock (now
`startModeRuntime`), so no switch lands between reading the mode and
starting the runtime for it.
- `SetMockMode` flips the flag once, then for the receiver and every
running monitor: ends the epoch, waits for admitted calls, clears
alerts, forgets fixture agents (when leaving mock mode) and resets state
(`endMockModeEpoch`). Only after that does it restart the sampler, polls
and discovery of each monitor in the set (`resumeAfterMockModeSwitch`).
A monitor that has not joined yet gets its runtime from its own `Start`.
- Each monitor records the mode its alerts and state belong to
(`mockModeAligned`, set in `New` from the same read that decides whether
persisted alerts are restored) and the number of switches made so far
(`mockModeSwitchesSeen`). If it missed switches between `New` and
`Start`, it leaves what it holds before joining, even after a round trip
that brought the flag back. A call for the mode the flag already holds
still moves any reached monitor that belongs to the other mode, and
leaves the rest alone.
Callers are unchanged and no API surface moves.
## Is this reachable in release builds?
Barely. In a `release` build `mockruntime.ValidateEnablement` refuses
mock fixtures unless `SetReleaseFixturesAuthorized(true)` was called.
Its only caller is `syncReleaseDemoFixtureRuntime`, which requires
`DemoMode` plus the `demo_fixtures` entitlement and runs only for the
default organization. Several tenant monitors in mock mode would also
need more than one organization's monitor running on that demo instance.
Demo mode's read-only middleware normally blocks creating organizations
and the mock-mode API's POST. It does forward any request carrying
`Upgrade: websocket` whatever its method; that bypass predates this
change and is being fixed separately. So in practice this is a
dev/mock-stack problem, and the fix stays inside the monitoring package.
## Tests
New tests in `internal/monitoring/monitor_mock_alerts_test.go`:
- `TestLeavingMockModeClearsEveryRunningTenantMonitor` and
`TestEnteringMockModeClearsEveryRunningTenantMonitor` start the default
and one other tenant monitor through `MultiTenantMonitor`, as the server
does, and wait until both have joined. They raise alerts in both, switch
through one, and check both.
- `TestMonitorStartedAfterAMockModeSwitchLeavesTheModeItWasBuiltIn`
covers a switch that lands between `New` and `Start`.
- `TestMonitorStartedAfterAMockModeRoundTripDropsWhatItRaisedMeanwhile`
covers two switches in that window, with fixture alerts raised in
between.
- `TestSetMockModeAlignsAMonitorWhoseModeTheFlagAlreadyHolds` covers a
call for the mode the flag already holds.
- `TestMockModeSwitchLeavesAnUnjoinedMonitorsRuntimeToItsStart` covers a
switch reaching a monitor that has not joined.
One more is in
`internal/monitoring/monitor_host_agent_removal_lifecycle_test.go`:
`TestLeavingMockModeReleasesFixtureAgentNodeLinksOnEveryRunningMonitor`.
It switches through the default monitor and checks that the other
tenant's fixture node is no longer suppressed by a fixture agent's
leftover node link.
Negative controls (`go test -overlay`):
| Control | Failing tests | What happened |
|---|---|---|
| Fence-PR runtime | Leaving, entering, node-link | Default monitor kept
69–104 fixture alerts (varies by run) and 69 fixture agents; org-b kept
its live alert; org-b's `mock-cluster-1-pve1` stayed suppressed |
| Set kept, switch reaches only the receiver | Same three | Same
failures |
| No realign at join |
`TestMonitorStartedAfterAMockModeSwitchLeavesTheModeItWasBuiltIn` | |
| Join compares only the mode, not the switch count |
`TestMonitorStartedAfterAMockModeRoundTripDropsWhatItRaisedMeanwhile` |
Started in live mode with the fixture node's memory alert |
| Plain early return when the flag already matches |
`TestSetMockModeAlignsAMonitorWhoseModeTheFlagAlreadyHolds` | |
| Resume ignores set membership |
`TestMockModeSwitchLeavesAnUnjoinedMonitorsRuntimeToItsStart` | |
Every control failed only the tests it targets.
Also run: the full `internal/monitoring` package; `-race -count=3` on
the new tests plus the existing `TestLeavingMockMode*` and
`TestMockModeFence*`; and the mock-mode tests in `internal/api` and
`internal/api/configapi`.
Preserve exact reviewed tip b5b1fa9777 and upstream a85bdcdc9a additively. Retain both alert threshold test sets. Combine upstream mock-mode epoch fencing with reviewed structural-revision checks and Docker departure ownership; add a regression that a current revision cannot revive an ended epoch. Resolve the monitoring contract to describe both boundaries, without rewriting accepted history.
Publication remains deferred for genuine final-content browser receipts and independent CodeQL disposition. Composition checks do not establish native recovery, deployment, exact-head hosted acceptance or release readiness.
Change-source: pulse-maintainer
## Problem
PR #2648 stopped the poller's previous-evidence lookup from handing a
replaced disk the old disk's serial. The linked host agent's SMART merge
runs before that lookup and had the same flaw. Its last-resort match is
the device path, and a path names a slot, not a disk. When a disk is
swapped before the agent reports again, or after the agent went silent
and kept its last rows, the agent's row for the old disk matched the
replacement. It filled the old serial or, for SAS, promoted it over the
Proxmox SAS address. The registry then keyed the replacement to the old
disk's canonical resource and SMART history, and it also took the old
disk's temperature and FAILED health. For SAS, the registry's same-path
join (`resolveLinkedPhysicalDisk`) did the same at the canonical level
even when monitoring kept the rows apart.
Proxmox's disk inventory (`PVE::Diskmanage`, checked on a PVE 9 node)
writes the literal `unknown` for a serial or WWN udev cannot read. The
poller recorded it as a collected serial, so the collection state hid
the gap, and the merge's `Serial == ""` fill never let the agent's real
serial replace it.
## Change
- `mergeHostAgentSMARTIntoDisks` refuses a device-path row whose
identity contradicts the Proxmox disk's. The new
`diskinventory.HardwareIdentityConflict` defines a contradiction:
nothing agrees across serial and WWN, and a field both producers report
like for like disagrees.
- WWNs count for every disk.
- Serials count only where Proxmox's serial is the drive's own, as
smartctl's is: a disk typed `nvme`, or one presented with SCSI vendor
`ATA` (libata, or a SAS HBA's SCSI-ATA translation) and not typed
`usb`/`sas`. Elsewhere the Proxmox serial may be a SAS address or SCSI
designator (#1595) or a USB bridge's own serial, so it only counts
toward agreement.
- Spelling differences are not disagreement: udev's underscores for
whitespace, a T10 designator for an ATA drive (`ATA`, the model, then
the serial), the agent's NAA field spelling, and udev's 64-bit prefix of
an NAA 6 WWN.
- The refusal runs after the unique-path check, so a second row on the
path keeps the match ambiguous.
- The merge's WWN step compares normalized identity
(`HardwareIdentityMatch`), so the agent's `5-c50-a1b2c3d4` finds the
Proxmox `0x5000c500a1b2c3d4` disk.
- `resolveLinkedPhysicalDisk` refuses the SAS same-path fallback when
the two WWNs name different disks.
- The poller records Proxmox's `unknown` serial/WWN as empty, so the
serial reads `missing` and the agent can fill it. Retained evidence
carries a previous serial only when it is usable, so a pre-upgrade
record cannot restore the placeholder. Identity keys already ignored the
placeholder. When the agent now fills a disk's serial, that disk's
Proxmox-poll SMART samples move from its WWN or path key to the serial
key, where the agent's own samples already go.
## Tests
- `TestHostAgentSMARTRowForSwappedOutDiskDoesNotLendItsIdentity` (sata
and sas) drives the real poller and registry. Proxmox reports `unknown`,
the agent fills the serial, the agent goes silent, and the disk is
swapped. On main, the SATA replacement takes the old serial and
resource; the SAS replacement keeps its own serial in state, but the
registry joins it to the old resource with its temperature and FAILED
health.
- `TestHostAgentSMARTMergeRefusesPathMatchAcrossContradictingIdentity`
has one row per producer rule: SAS/SATA/NVMe replacements refused;
same-disk SAS without an agent WWN, a renamed SAS disk in the agent's
WWN spelling, udev's serial spelling, a USB enclosure (typed `usb` and
typed `ssd`), a T10 designator and an NAA 6 volume merged; and an
ambiguous path.
- `TestRegistrySASPathJoinRefusesContradictingWWN`,
`TestProxmoxUnknownDiskIdentityIsRecordedAsUnreported` and
`TestHardwareIdentityConflict` cover the rest.
- Negative controls via `go test -overlay`: removing each rule fails
only its own case. These cover the guard, the WWN normalization, the
transport/vendor rule, compaction, the designator rule, the NAA 6
prefix, the registry guard, the ingest normalization and the
usable-serial carry. Against main's `monitor.go` and `registry.go`, the
end-to-end, merge and registry tests fail.
- `TestMergeHostAgentSMARTIntoDisks_PreservesExistingProxmoxIdentity`
merged a row with a different WWN by path alone, which the guard now
rightly refuses. Its row is now the same disk in the agent's WWN
spelling, with all assertions unchanged.
- `internal/monitoring`, `internal/unifiedresources` and
`pkg/diskinventory` pass in full.
## Not covered
- A replacement with no comparable identity still matches its path: no
usable serial or WWN at all, or a SAS/SCSI/USB disk with a WWN on one
side only.
- The registry's seeded source mapping, reused after a restart by the
slot-shaped Proxmox source ID, can still hand a swapped disk the old
canonical resource. That's a separate follow-up.
Mock mode is one process-wide flag, but SetMockMode ended the epoch,
cleared the alerts, forgot the fixture agents and reset the state of
only the monitor it was called on. The server runs one monitor per
organization, each with its own alert manager and its own mock alert
pass. A switch made through one of them (the mock-mode API uses the
requesting tenant's monitor, the demo-fixture licence sync the default
one) left every other running monitor holding what it raised in the mode
the process had left: fixture alerts and fixture-agent node links in
live mode, live alerts in mock mode, until the 24-hour stale sweep.
Monitors now join a process-wide set of running monitors when Start runs
and leave it when Start returns, under the same mutex that already
serialized switches. SetMockMode flips the flag once, then ends the
epoch of the monitor it was called on and of every running monitor: it
waits for their admitted calls, clears their alerts, forgets their
fixture agents and resets their state. Only then does it restart each
running monitor's sampler, polls and discovery. Start picks the mock
metrics sampler or discovery while it holds that mutex, so no switch can
land between reading the mode and starting the runtime for it, and a
switch leaves the runtime of a monitor that has not joined yet to its
own Start. A monitor records the mode and the switch count it was built
under. If it missed switches between New and Start, it leaves what it
holds before it joins, dropping live alerts New restored or fixture
alerts a read path raised meanwhile, even when a round trip brought the
flag back. A call for the mode the flag already holds still moves a
reached monitor whose alerts belong to the other mode, and leaves the
rest alone.
Release builds reach this only on an instance that runs DemoMode with
the demo_fixtures entitlement, since release ValidateEnablement refuses
mock fixtures otherwise, and that has more than one organization's
monitor running. The licence sync only switches the default
organization. So the fix stays inside the monitoring package. The
callers are unchanged and no API surface moves.
New tests start the default and one other tenant monitor through
MultiTenantMonitor, as the server does, and switch through one of them.
Against the per-monitor switch, the default monitor kept its fixture
alerts (69 to 104 across runs) and all 69 fixture agents after mock mode
was left through the other tenant. That tenant kept its live alert after
mock mode was entered through the default monitor, and its fixture node
stayed suppressed by a fixture agent's leftover node link. Four more
tests fail if Start joins without leaving the mode its monitor was built
in, if Start compares only the mode after a round trip, if a call for
the flag's current mode returns without moving such a monitor, or if a
switch starts the runtime of a monitor that has not joined.
Leaving mock mode cleared active alerts, but every path that evaluates
mode-dependent data checked the mode once, up front. An evaluation
already holding fixture data when the switch happened landed its alerts
after the clear, in live mode, where nothing evaluates or resolves
fixture resources, so they stayed until the 24-hour stale sweep.
A pass-level lock around `checkMockAlerts` was tried and reverted: it
closed only that path and made the mock-mode API wait behind a whole
pass of synchronous event-log writes.
**What changes**
- `mockModeFence` (`internal/monitoring/mock_mode_fence.go`): each path
takes a scope before reading anything mode-dependent and sends each
alert-manager call through `scope.run`. Covered: `checkMockAlerts`
(every loop, host agents, disks, snapshot cache), `checkBackupAlerts`,
`pollGuestSnapshots`, `checkConnectionAlerts`, `pruneStaleDockerAlerts`,
and the resource-store refresh (`updateResourceStore`,
`updateResourceStoreForRead` and `syncUnifiedResourceAlertsToState` now
take the scope of the state they rebuild from).
- `SetMockMode` is serialized, flips the mode, ends the epoch, waits
only for calls already admitted, then clears. Unified metrics go one
resource per call, so the wait covers single calls, not passes.
Admission never blocks.
- Registry rebuilds are tracked, not waited on: a refresh evaluates the
registry it read back only if no rebuild from an ended epoch overlapped
it. Readers that act on the inventory (Docker pruning, backup and
snapshot guest lookups, the pollers' previous-state carry) use current
state until an undisturbed rebuild in the new epoch replaces the fixture
registry.
- `mock.SupplementalRecords`/`SupplementalChanges` serve nothing outside
mock mode. The router swaps its mock adapters for the real pollers only
after the switch, refreshing between sources, and the adapters used to
fall back to default fixtures, so a single switch to live mode raised
fixture vSphere incidents.
**Verified**
- Each new test parks a real evaluation at a fixed point (a held lock, a
blocking supplier or a gated registry read), switches modes and releases
it, with bounded waits. Against the unfenced code: the mock pass left
about 40 fixture alerts, the frontend refresh 5 vSphere incidents, the
backup check 137 backup-age alerts, and a mock-mode connection
observation counted toward a live connection's confirmation. The drain,
republish, pruning and carry tests fail against a fence without its
wait, overlap check or registry tracking; the republish test also fails
with the old mock supplemental defaults.
- 995 tests from the touched files and related areas pass, plus
`internal/mock` and the `internal/api` mock/platform tests; the new
tests pass under `-race` and over repeated runs.
**Not covered**
- The fence and clear are per monitor while mock mode is process-wide;
other tenant monitors neither drain nor clear on a switch (unchanged,
noted in the monitoring contract).
- Report admission and live pollers outside the fenced paths can still
evaluate after the clear when entering mock mode; their alerts can
persist until mock mode is left.
Preserve exact published upstream commits at the batch-start frontier. Resolve the TrueNAS history conflict by passing scoped disk identity through the existing ambiguous-alias and in-flight inventory guards, keeping both reviewed protections. No frontend resolution or receipt changes.
Change-source: pulse-maintainer
## Problem
Nested Proxmox clusters (each PVE node is itself a QEMU VM) lose
physical disks from inventory. QEMU gives a virtual disk configured
without a serial a default one built from its drive ID or a per-VM
counter, so every VM built the same way reports the same value:
- Proxmox's `/nodes/{node}/disks/list` serial is udev's
`ID_SERIAL_SHORT` (`PVE/Diskmanage.pm`). A virtio-scsi disk on a Proxmox
VM reports `drive-scsi0` with no `ID_WWN`; on a libvirt VM it is
`drive-scsi0-0-0-0`. A `-drive` given no ID falls back to QEMU's
automatic ID, `scsi0-hd0` for `if=scsi` and `none0` for `if=none`.
- An emulated ATA disk's serial is `QM` plus a five-digit counter
(`QM00001`).
`diskinventory.IsUsableHardwareID` accepted all of these, so
`resourceFromPhysicalDisk` set `MachineID = "drive-scsi0"` on every node
and the registry minted one `machine:drive-scsi0` canonical disk and one
SMART series for the whole cluster. Three nodes with two such disks each
produced 2 canonical disks instead of 6.
## Change
- `IsUsableHardwareID` rejects exactly QEMU's default shapes:
`drive-scsi` plus digits with at most three `-digits` groups,
`scsi<n>-hd<n>`, `none<n>`, and `QM` plus five digits. Real serials that
only share a prefix (`DRIVE-ASSET123`, `drive-sata0`, `QM0001`) stay
identity. It also strips `naa.`/`eui.`/`wwn-`/`0x` framing before its
all-zero and all-F test, so a `0x0000000000000000` WWN is a placeholder
too.
- TrueNAS provider: a disk took its canonical identity from any
non-empty serial, so a placeholder merged disks across appliances. It
now uses only a usable serial, and lists superseded IDs for every disk
without one.
- TrueNAS history keys: for a serial that was already rejected, such as
`UNKNOWN`, native TrueNAS history used TrueNAS's own disk identifier,
and the unified SMART writer (`syncUnifiedPhysicalDiskMetrics`)
substituted the metrics target only for an empty serial, so the samples
went to the canonical resource ID. Neither matched the chart reader's
key, and rejecting QEMU serials would have put TrueNAS VMs on that path.
Native history now falls back to the disk's source ID, and the writer
always writes under the resolved metrics target, so writer, native
history and reader agree. History lookup keys also drop placeholder
serials.
## Upgrade effect
A disk reporting one of these values (nested PVE nodes, TrueNAS or agent
hosts running as QEMU VMs with virtual disks, TrueNAS disks reporting
`UNKNOWN`) changes canonical ID and SMART metric key once. Samples
recorded under the old key are not carried over, because that key mixed
every disk that shared the value. Path-shaped alert references (Proxmox
disk health and wearout, host-agent SMART) are unaffected; anything
keyed on the old canonical ID or metric key starts again under the new
one. This is recorded in the monitoring and agent-lifecycle contracts.
## Tests
- `TestIsUsableHardwareIDRejectsQEMUDefaultDiskSerials`: each default
shape is rejected, falls back to the scoped topology ID and never
matches itself; prefix-sharing real serials stay usable.
`TestPreferredIDRejectsPlaceholderHardwareIdentity` gains the framed
all-zero/all-F WWNs.
- `TestNestedProxmoxDefaultQEMUSerialsStayPerNode`: 3 nodes ×
(`drive-scsi0`, `QM00005`, `drive-scsi0-0-0-1`) give 9 canonical disks,
9 slots and 9 metric keys, every SMART writer key is a key readers
resolve, and agent SMART source IDs stay per host.
-
`TestTrueNASPlaceholderDiskSerialsStayPerApplianceAndShareOneHistoryKey`:
two appliances with `UNKNOWN` and `drive-scsiN` serials keep one
canonical disk each, and the SMART writer's samples are readable under
each disk's metrics target.
- `TestProviderPlaceholderDiskSerialsKeyNeitherIdentityNorHistory`:
placeholder serials key no identity, list superseded canonical IDs, drop
out of native history lookup keys, and native history lands under the
registry's metrics target.
- Negative controls via `go test -overlay`, one runtime file reverted at
a time: base `identity.go` fails all three round trips (3 canonical
disks instead of 9, 9 TrueNAS disks instead of 16); base `provider.go`
fails the identity and history assertions; base `monitor.go` fails the
SMART-history-under-reader-key assertion; reverting only the native
history key, the superseded-ID gate or the lookup-key filter fails its
own provider assertion.
- `pkg/diskinventory`, `internal/truenas/...`,
`internal/unifiedresources/...` and `internal/monitoring` pass in full;
`internal/api/...` disk, hardware, serial and TrueNAS tests pass.
Stacked on #2648 (`fix/pve-disk-serial-carry`), whose commit this branch
carries until it merges. Not in this change: the registry's merge of
disks across parents on a canonical-ID collision (separate change), and
a pre-existing writer/reader key mismatch for controller-member disks
without a usable serial or WWN (`PhysicalDiskMetaMetricID` appends
topology to a source ID that already carries it), which is tracked
separately.
The linked host agent's SMART merge falls back to matching a Proxmox
disk by device path, and a path names a slot, not a disk. When a disk
is swapped before the agent reports again, or after the agent went
silent and kept its last rows, the row for the old disk matched the
replacement. It filled or, for SAS, promoted the old serial onto it,
which keyed the replacement to the old disk's canonical resource and
metric history, and lent it the old disk's temperature and failed
health. For SAS the registry's same-path join did the same at the
canonical level even when monitoring kept the rows apart.
Both now refuse a row whose identity contradicts the disk's, judged
only on what both producers report like for like. WWNs count for every
disk. Serials count only where Proxmox's serial is the drive's own, as
smartctl's is: NVMe namespaces, and disks presented with SCSI vendor
"ATA", whose serial udev's ata_id reads from IDENTIFY. Elsewhere
Proxmox's serial may be a SAS address, a SCSI designator or a USB
bridge's own serial (#1595). Reporter spellings are not disagreement:
udev writes whitespace in serials as underscores, its T10 designator
for an ATA drive puts "ATA" and the model before the drive serial, and
its ID_WWN keeps only the first 64 bits of an NAA 6 identifier. The WWN match step compares normalized identity too,
so the agent's 5-c50-a1b2c3d4 spelling finds the Proxmox
0x5000c500a1b2c3d4 disk.
Proxmox's disk inventory reports a serial or WWN udev cannot read as
the literal "unknown". The poller recorded it as a collected serial, so
the collection state hid the gap and the agent's serial never filled
it. It is now recorded as empty, and retained evidence carries only a
usable previous serial so an older record cannot restore it. Identity
keys already ignored the placeholder.
Additively incorporate exact upstream d5c945241d without changing reviewed identities or ancestry. Resolve only overlapping tests by retaining both real metric-owner handover and recorded-hardware disk-history controls, plus acknowledgement hiding/counting and remote unacknowledgement assertions. Final-source validation and independent browser/security disposition remain separate.
Change-source: pulse-maintainer
SetMockMode(false) switched the mock package off and then cleared active
alerts, but every path that evaluates mode-dependent data checked the
mode once, up front. A mock alert pass (started from the ticker with go,
so passes overlap), a frontend or ingest refresh evaluating a GetState
snapshot, a backup-age check over fixture recovery rollups, or a
connection check over the mock ledger could already hold fixture data
when the switch happened. Its evaluations then landed after the clear,
in live mode, where nothing evaluates or resolves fixture resources, so
the alerts stayed until the 24-hour stale sweep.
A pass-level lock around checkMockAlerts was tried and reverted: it left
the other paths open and made the mock-mode API wait behind a whole pass
of synchronous event-log writes.
The monitor now owns a mock-mode epoch fence. The mock pass, backup-age
and snapshot checks, connection checks, Docker pruning and the
resource-store refresh take a scope before reading anything
mode-dependent and send each alert-manager call through it. SetMockMode,
now serialized, flips the mode, ends the epoch, waits only for calls
already admitted, and then clears. Unified metrics are evaluated one
resource per call, so the wait covers single calls, not passes.
Registry rebuilds are tracked rather than waited on. A refresh evaluates
the registry it read back only if no rebuild from an ended epoch
overlapped it, and the readers that open or remove alerts from the
inventory use current state until a clean rebuild in the new epoch
replaces the previous mode's registry. The mock supplemental fixture
sources now serve nothing outside mock mode: the router swaps them for
the real pollers only after the switch, refreshing between sources, so
a single switch to live mode used to raise fixture vSphere incidents.
The new tests park a real evaluation at a fixed point (a held lock or a
blocking supplier), switch modes and release it. Against the unfenced
code the mock pass left about 40 fixture alerts, the frontend refresh 5
vSphere incidents and the backup check 137 backup-age alerts in live
mode, and a mock-mode connection observation counted toward a live
connection's confirmation. The drain, republish and pruning tests fail
against a fence without its wait, overlap check or registry tracking.
Switching mock mode with several tenant monitors running remains
uncovered and is noted in the monitoring contract.
QEMU gives a virtual disk configured without a serial one built from
its drive ID or a per-VM counter, so every VM built the same way reports
the same value. Proxmox's disks/list serial is udev's ID_SERIAL_SHORT,
which for a virtio-scsi disk is "drive-scsi0" (libvirt:
"drive-scsi0-0-0-0"), and an ATA disk's serial is "QM00001".
IsUsableHardwareID accepted these as hardware identity, so every node of
a nested Proxmox cluster minted one machine:drive-scsi0 canonical disk
and one SMART series, and all but one node lost the disk from inventory.
IsUsableHardwareID now rejects exactly QEMU's default shapes: drive-scsi
plus digits with at most three -digits groups, QEMU's automatic drive
IDs scsiN-hdN and noneN, and QM plus five digits. Real serials that
share a prefix, such as DRIVE-ASSET123, stay identity. It also strips
WWN framing before the all-zero test, so a 0x0000000000000000 WWN is a
placeholder too.
The TrueNAS provider set a disk's canonical identity from any non-empty
serial, so placeholders merged disks across appliances. It now uses only
a usable serial. For a serial that was already rejected, such as
UNKNOWN, native TrueNAS history and the unified SMART writer disagreed
with the chart reader: native history used TrueNAS's disk identifier,
and the writer substituted the metrics target only for an empty serial,
so a placeholder sent samples to the canonical resource ID. Rejecting
QEMU serials would have put TrueNAS VMs on that path. Both now use the
key the registry's metrics target resolves, the disk's source ID, and
native history lookups no longer route by placeholder serials.
Disks that report one of these values change canonical ID and SMART
metric key once; samples under the old key are not carried over because
that key mixed every disk that shared the value.
The registry merges a physical disk's rows one step at a time, choosing
the temperature by source preference and merging collection states
separately. The pairing rule from #2618 was exact for two rows only: with
three or more, an earlier step's single merged state stood in for its
rows. A silent agent, an offline Unraid inventory and a stale Proxmox
copy could leave a withdrawn 41 labelled collected, and a legacy agent's
stateless reading could pick up another row's "unsupported".
The merged disk now keeps each row's current reading and state
(merge-time only, never serialized), and the shown value takes its state
from the readings of that value: a collected reading first, then the
agent's own report over a copy. A copy of an agent source's state gives
way to the agent's own withdrawal, including a legacy agent's sourceless
one, and a value source preference keeps after its row reported again
keeps its presented state under that row's later withdrawal. Value
selection is unchanged.
Dispatch replication independently of the efficient guest path, coalesce bounded runtime-scoped reads and fence obsolete client/runtime completions. Preserve prior inventory on unavailable input or cancellation; only an explicitly complete empty read clears it. Bind scheduler, replacement and HTTP cancellation controls to this repair.
Contract-Neutral: Repair existing replication scheduling and cancellation ownership; no wire, endpoint, cadence, permission or host-agent lifecycle contract changes.
Change-source: pulse-maintainer
Add exact published upstream 1b6a9b45d7 without rewriting reviewed history. Retain its collected-temperature protection for AI, Patrol, reports and metric recording. Resolve the chart overlap without restoring fabricated live two-point history: no samples means no series, and a single observed sample retains its original timestamp and value. Extend the upstream regression to cover both retained and live sparse histories.
Change-source: pulse-maintainer
Normalization keeps a disk's last-known temperature when the current
observation did not collect one (a disk in standby, a host agent past
its reporting lease) and marks it non-available. Several readers still
took PhysicalDisk.Temperature at face value. The AI chat context listed
a standby disk as needing attention for heat. Patrol raised health
issues and triage flags from the retained value. The AI query and
disk-health tools handed it to the model as the disk's temperature.
Performance reports and the reporting runtime snapshot tabulated it as
measured, and the disk temperature charts padded a series out to now
with it.
Each of them now goes through diskinventory.TemperatureCollected. AI
surfaces keep a retained value as context, labelled last known with
its reason. Reports and the charts leave it out. monitor.go's private
copy of the predicate is replaced by the shared helper.
Additively integrate published main dcb69cfb02 while retaining reviewed tip 52fe83d266 and every accepted commit. Preserve both subsystem contracts and independent alert tests. Reconcile the reactive row-history helper with reviewed memory provenance and visible freshness cues; exact source, browser and protected checks remain required.
Change-source: pulse-maintainer
Port rcourtman’s Pulse PR #2592 at 5a37a72f76 onto the assigned current source. Departed fixture hosts use the live removal boundary even without state readers; structural revision checks prevent stale overlapping passes from reinstating retired hosts or node links.
The source tree is identical to rejected candidate 2540bfdae621ddad72063463e0ef8dea6451247f. Its contributor-history merge failed maintainer provenance, so this is a new attributed maintainer port, not an amendment or relabelling of the original commits. Fresh exact-source proof and independent protected review are still required.
Change-source: pulse-maintainer
Two Proxmox connections can each have a node with the same name, which
is common for MSPs running one px1 per site. The node temperature
lookup fell back to the first host agent whose hostname matched the
node name, even when that agent was linked to the other site's node,
and even when the name was ambiguous enough that the agent linker had
refused to link it. Site B's px1 then showed, alerted on and recorded
history for site A's sensors. The cluster sensor cache leaked the same
way: an agent's SSH readings for its cluster siblings were keyed by
bare node name, so a site A sibling reading fed site B's node.
The poller now hands the lookup the polled node, and the lookup works
on that node's slot. The state folds the views of one machine reached
through two connections (a cluster added twice, or a multi-homed host
added by each address) into one node held under one view's ID, and
the agent link follows it. Those views used to find their agent only
through the loose hostname match, so a polled view the read state does
not hold also takes the one same-named node proven to be the same
machine. models.NodeObservationsSameMachine and
models.HostAgentBridgesNodeViews give that proof from the evidence
UpdateNodesForInstance folds on, and the state's agent corroboration
check now shares its rule with them. Ambiguous candidates are declined,
as the state declines an ambiguous alias.
An agent linked to the slot is the node's agent. The hostname fallback
only takes an agent with no link when no node outside the slot and no
second unlinked agent has the name. Cluster sensor cache entries record
their reporting agent and serve a node only when that agent's linked
node shares one of the slot's connections, or belongs to a connection
whose configured view of the node has a matching TLS fingerprint.
LinkedNodeID holds the Proxmox source node ID for automatic, manual and
continuity-restored links, so the link match needs no normalization.
The physical-disk SMART merge already matches agents by
node.LinkedAgentID within one instance and is unchanged.
pulse-triage[bot]