Carry exact duplicate-row conflicts into owned client trust before map deduplication. Keep unresolved saved trust through initialization, missing-client retry and topology refresh without replacing pins or stopping trusted siblings. Cover actual monitor lifecycle routes and member projections alongside client TLS controls.
Change-source: pulse-maintainer
Three rules chose a node's single CPU temperature. Node history took
CPUMax first, agent history took cpu_package first, and the node drawer
recomputed the maximum of package, CPUMax and every core in the
browser. On an agent-fed node CPUMax is built from cores alone and can
sit below the package, and elsewhere a core can run above the package,
so the drawer's CPU temperature headline could disagree with the line
drawn beneath it and with the History tab. Node alerts and the CPU
low/record tracking used yet another package-first rule.
models.PrimaryCPUTemperatureCelsius is now the one rule: the hottest
usable package, CPU-maximum or core reading, with a sensor-map form for
agents. Node and agent history, the router's live fallbacks, the
unified resource temperature, node temperature alerts and the CPU
low/record tracking all call it. The drawer reads the resource value
through a new Node.cpuTemperature field instead of recomputing it, and
getCpuTemperature is removed.
Mock mode wrote a second package reading under a cpu.package key into
every agent host, including node-linked ones, from a series unrelated
to the node's static reading. A node and its linked agent now carry one
reading, sampled from the agent's series under the agent's real keys,
and both histories seed from that series, so the current reading
continues the line.
Reject ambiguous names and VMIDs before reading provider configuration, honour partial placement constraints, and read current inventory rather than stale state. Use the chosen typed guest view for policy metadata so shared names cannot select another installation. Preserve direct reads, original provider errors, guest safety and authority; cover both VM and LXC paths.
Change-source: pulse-maintainer
Saturate duration arithmetic before doubling so long failure episodes cannot overflow into immediate retries. Preserve existing thresholds, early delays and recovery, with independent mathematical and real scheduler lifecycle controls.
Change-source: pulse-maintainer
Additively integrate published main dcb69cfb02 while retaining reviewed tip 52fe83d266 and every accepted commit. Preserve both subsystem contracts and independent alert tests. Reconcile the reactive row-history helper with reviewed memory provenance and visible freshness cues; exact source, browser and protected checks remain required.
Change-source: pulse-maintainer
Incorporate the exact observed upstream head additively, retaining reviewed tip 1c1ba27fe6 and every accepted commit. Preserve published source-keyed node carry-over, silent-agent temperature bounds and memory identity. PR2576 remains unchanged while the owned combined-Patrol browser proof is unavailable; composition does not establish validation, publication or installed recovery.
Change-source: pulse-maintainer
Two Proxmox connections can each have a node with the same name, which
is common for MSPs running one px1 per site. The node temperature
lookup fell back to the first host agent whose hostname matched the
node name, even when that agent was linked to the other site's node,
and even when the name was ambiguous enough that the agent linker had
refused to link it. Site B's px1 then showed, alerted on and recorded
history for site A's sensors. The cluster sensor cache leaked the same
way: an agent's SSH readings for its cluster siblings were keyed by
bare node name, so a site A sibling reading fed site B's node.
The poller now hands the lookup the polled node, and the lookup works
on that node's slot. The state folds the views of one machine reached
through two connections (a cluster added twice, or a multi-homed host
added by each address) into one node held under one view's ID, and
the agent link follows it. Those views used to find their agent only
through the loose hostname match, so a polled view the read state does
not hold also takes the one same-named node proven to be the same
machine. models.NodeObservationsSameMachine and
models.HostAgentBridgesNodeViews give that proof from the evidence
UpdateNodesForInstance folds on, and the state's agent corroboration
check now shares its rule with them. Ambiguous candidates are declined,
as the state declines an ambiguous alias.
An agent linked to the slot is the node's agent. The hostname fallback
only takes an agent with no link when no node outside the slot and no
second unlinked agent has the name. Cluster sensor cache entries record
their reporting agent and serve a node only when that agent's linked
node shares one of the slot's connections, or belongs to a connection
whose configured view of the node has a matching TLS fingerprint.
LinkedNodeID holds the Proxmox source node ID for automatic, manual and
continuity-restored links, so the link match needs no normalization.
The physical-disk SMART merge already matches agents by
node.LinkedAgentID within one instance and is unchanged.
The poller reads the previous poll's nodes back from the unified read
state, but previousNodeFromView returned the registry's unified resource
ID and no temperature. Every lookup keyed on the Proxmox source ID
missed: nodeLastOnline in the offline grace policy, the temperature
carry, the CPU low/record tracking (reset to the current reading every
poll) and the node memory carry. When a whole instance poll failed,
markPVEInstanceNodesUnreachable wrote the projected nodes back, so each
failed poll re-ingested the node under a new unified ID. Cluster members
also lost the cluster identity their canonical ID derives from, so they
changed ID for the length of an outage.
The projection now returns the source ID, the stored temperature
details, the cluster name and the linked agent ID, and
previousNodesForInstance restores the config-derived identity stamps
(provider-scoped flag, native aliases, TLS fingerprint) the way
pollPVENode writes them.
Reviving those paths needed four bounds:
- The node memory carry uses the poller's own validated snapshot rather
than the read state, which holds the agent's reading on a node merged
with a host agent, and keeps used + cache + free equal to the total.
- A carried temperature keeps a lapsed host agent's lease. A reading
stamped no later than the agent's last report is not carried, so the
five-minute carry window cannot hold a silent agent's reading in
alerts and history past its lease, which #2582 guarantees.
- A carried reading keeps the node's CPU and GPU temperatures only. Its
SMART and NVMe rows also stamp the node's physical disks, so a disk
that went to standby would be stamped and recorded with its pre-sleep
value.
- Outage preservation and authoritative membership reconciliation keep a
node's temperature only under those rules and drop it when the node
goes offline.
TestPollPVEInstanceMarksStaleNodesOfflineWhenGetNodesReturnsEmpty relied
on the dead nodeLastOnline lookup and now ages that sighting too. The
silent-agent test from #2582 now rebuilds the store as production does
and feeds the polled node back, which is what exercises the carry.
Prevent optional Linux memory and metadata failures from starving supported filesystem usage. Carry shared admission deferral independently of a completed disk sample. Preserve backup locks, uncertain completion, no replay and original observation ages; bind both poll paths and truthful History to connected controls. This is source repair, not a native discussion #2538 cause or installed recovery.
Change-source: pulse-maintainer
The deletion-budget note and its test said only that a loop still
starting up can be slow to stop. The CI failures have a specific
cause: in mock mode Monitor.Start runs startMockMetricsSampler
synchronously before its select loop, and prepareMockMetricsHistory
seeds seven days of fixture history without a context, so the loop
saw its cancelled context only once seeding finished ("Mock metrics
history sampler started" and "monitoring loop stopped" log in the same
second in runs 37329726585 and 37331626618). Say so in the contract
and the test comment.
244d322dce made organization deletion refuse, and keep the data, when
the tenant's monitoring loop has not exited. That is the right guard,
but it reused tenantMonitorShutdownTimeout, a two-second budget that
had only ever been a best-effort wait before logging and carrying on.
A loop that is still starting up or finishing a tick can take longer
to see its cancelled context. In E2E run 37329726585 a just-used
organization's loop took 8-10 seconds, so both attempts of
13-api-token-scope's org cleanup got 503 tenant_shutdown_incomplete.
Any real delete shortly after a tenant was busy hits the same refusal
until a retry.
Deletion now waits on its own tenantDeletionShutdownTimeout (30s)
before it refuses. Process shutdown and RemoveTenant keep the
two-second budget, and a loop that never exits still refuses and
retains the data. Both budgets are variables so tests can shorten
them; the never-stopping-loop test no longer waits on the real budget.
Extend the #1962 linked-agent memory fallback to Proxmox LXC containers (#2148). A correlated online Pulse agent running inside a container now supplies the container's memory metric instead of the cache-inclusive cluster/resources fallback, which can badly under-report shared-memory workloads. The agent sample is only trusted when its total matches the container's configured limit, so an agent without lxcfs cannot leak host memory into the guest row. Threads the previous-guest agent map through the efficient and node-by-node container builders and reads agent-owned memory from merged ContainerView resources.
Change-source: pulse-maintainer
Issue #1962 describes automatic correlation into a hybrid VM, while the retained-link regression only exercises a manual merge. Exercise the hostname link selector through source snapshot reconstruction and next-poll memory resolution, preserving standalone host identity, cross-instance VMID isolation and ambiguous-name refusal. This narrows the acceptance boundary without changing matching behaviour or claiming installed recovery.
Change-source: pulse-maintainer
Keep stale/offline/unavailable controls, diagnostic and alert agreement, and verify isolation with a simultaneous duplicate VMID in another instance. Test-only follow-through for issue #1962.
Change-source: pulse-maintainer
Read agent-owned memory and source freshness from VM views when correlation removes the standalone host row. Preserve preferred guest memory sources and reject stale, offline or unavailable agent evidence. Reproduce issue #1962 through registry merge, next-poll diagnostics, card projection and memory alerts.
Change-source: pulse-maintainer
Commit link and unlink journal updates before publishing state, preserve manual selections across report and provider refresh boundaries, and reserve dormant owners across restart. Re-evaluate known automatic associations using provider names while retaining legacy unknown links.
Change-source: pulse-maintainer
The thresholds UI stores per-resource overrides keyed by unified
registry resource IDs, while the engine check paths look up legacy
monitor IDs, so a per-node override on an API-only PVE node saved,
displayed as Custom, and never applied: the node kept alerting at the
global defaults. Reproduced live against mock: memory at 81% with a 42%
override stored under the registry ID raised nothing.
Route the node, PMG, PBS, and host/linked-node override lookups through
the canonical registry identity resolver the intent-policy path already
uses, with the exact legacy key still winning so overrides stored by
older UI versions keep resolving unchanged.
Proof: TestNodeThresholdOverrideStoredUnderRegistryIDApplies drives the
real populate path for an API-only node, stores a memory override under
the registry resource ID the UI uses, and asserts CheckNode fires at the
override threshold while controls without the override or without the
resolver stay quiet.
Refs #1738
Contract-Neutral: behavioral fix#1738: threshold override lookup resolves registry identity, no public contract delta
Read paths refresh canonical identity on every clone, so views always
showed the derived aliases, but the stored registry entries never carried
them: snapshot ingest built resources without the canonical block and only
the IngestResources seed path stored refreshed clones. GetByReference's
canonical-alias fallback reads the stored entries directly, so prefixed
refs like "agent:{hostID}" never resolved on a production rebuild and
per-resource alert grace overrides on merged PVE-node + pulse-agent hosts
silently fell back to factory policy. The earlier alias fix (dd008f1c2)
only proved the IngestResources-seeded path. Recompute identities in each
ingest finalize pass, after links and merges settle.
Proof: TestMergedProxmoxHostAgentGraceOverrideHoldsCPUAlert drives the
real populate path (node plus linked agent merging into one resource),
saves a 600s metric.cpu override under the registry resource ID the UI
uses, and asserts CheckHost holds the alert while a control manager
without the override fires immediately.
Refs #1497
Contract-Neutral: behavioral fix#1497: populate derived canonical identity on stored registry entries, clones already exposed identical data, no public contract delta
matchZFSPoolForStorage's sole-pool fallback attached the node's only ZFS
pool to every storage the per-node endpoint returned, including NFS, CIFS,
PBS, and RBD entries that can never be backed by a local pool. One failing
device then raised a duplicate ZFS device alert per shared storage. Gate
the matcher on isInherentlySharedStorageType so remote-backed storages are
never matched, while dir-type storages keep the single-pool fallback.
Refs #1731
An explicit connection address override on a cluster member was silently
discarded when VerifySSL was enabled and the member had no per-endpoint
fingerprint: the hostname-for-TLS preference displaced the operator's
address, so overriding an undialable discovered hostname changed nothing.
The override now wins in every TLS mode, and the hostname preference
applies only to auto-discovered addresses.
Failed endpoint recovery attempts also logged their cause at debug level
only, leaving the recurring 'No endpoints recovered' warning without a
reason. The warning now carries per-endpoint failure reasons and the
sanitized error is stored so the UI health status shows it too.
Refs #1665
Contract-Neutral: behavioral bug fix refs #1665: cluster member IPOverride honored in all TLS modes, recovery failure reasons surfaced; no public contract delta
Proxmox's per-node storage endpoint GET /nodes/{node}/storage does not
return what the node can actually use. It returns every storage in the
datacenter config and reports the ones the node is excluded from with
enabled:0/active:0 instead of leaving them out. pollStorageWithNodes
ingested every row it got back, so a storage restricted to one node
appeared on all the others as a disabled entry, which the UI renders as
Offline.
The poller already fetches the datacenter config once via GetAllStorage
and keeps it keyed by storage name, so the restriction is in hand at the
point each per-node row is turned into a models.Storage. Skip the row
when the config carries a non-empty nodes restriction that does not
include the node being polled. Node names are compared case-insensitively
to match how node identity is compared elsewhere in the poller.
Only the restriction drops a row. A storage with no restriction still
shows up on every node it is reported from, including when it is disabled
everywhere, so a datacenter-wide disabled storage stays visible as
disabled rather than vanishing. Because the shared-storage aggregation
builds Nodes/NodeIDs/NodeCount from the surviving per-node rows, this also
stops a restricted shared storage from claiming cluster members that
cannot mount it.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The discovery-policy check resolved endpoint hostnames with a bare
net.LookupIP while the actual dials went through pkg/tlsutil's process-global
cached resolver, so the policy and the connection reasoned about two different
DNS views. That split is why 108aa4e20 had to skip resolution entirely for the
default policy, which left the injected 169.254.0.0/16 blocklist enforced only
against literal IPs: a hostname endpoint pointed at the metadata range walked
straight through.
Resolve through tlsutil.LookupHostCached instead. The shared resolver caches
answers and lookup failures alike until its next refresh, so repeat poll cycles
cost a cache hit rather than a query and the per-poll DNS volume that opened
#1638 stays gone. With that in place the default-policy skip is removed and the
blocklist applies to resolved addresses again, and the five-minute decision
cache is dropped rather than kept: it bought nothing on top of the resolver
cache, made the verdict trail the configuration, and memoized the fail-open
"resolution failed, allow" outcome for minutes even with an explicit allowlist
configured. Its claim to match a DNS refresh interval that operators configure
through DNS_CACHE_TIMEOUT goes with it.
The SSH backoffs now only escalate for work that ran. A knownhosts manager
suppressing a call inside its own window reports ErrKeyscanSuppressed, and the
temperature layer neither records a failure nor pays for the RPi fallback in
that case. An expired collection deadline is our own budget rather than
evidence about the host, so it holds the window at the floor. Both backoffs
decay once a retry deadline is more than one window past, and replacing the
temperature SSH key on disk clears both maps so a repaired key is tried on the
next cycle instead of after fifteen minutes.
Refs discussion #1638.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The cluster-endpoint discovery-policy check ran a raw net.LookupIP per node
per poll cycle since c5f5af7ab, bypassing the process-global cached resolver,
and the injected default subnet blocklist (169.254.0.0/16) made the
zero-policy fast path unreachable so even unconfigured installs generated
that DNS volume. Evaluate the default link-local-only policy against literal
endpoint IPs without resolution, and memoize custom-policy verdicts per
endpoint for the shared 5-minute DNS-cache TTL so repeat polls stay off the
resolver. Also cache ssh-keyscan failures with doubling backoff in the
knownhosts manager and back off temperature SSH collection per host after
failures instead of re-executing ssh twice per node every 10s cycle.
Refs discussion #1638.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ac0fb263c changed preserved guests to carry their real runtime status
("running") instead of the stringified aggregate status ("online"). The
three guest metric recording loops gate on Status == "running", so guests
carried forward while their node sits in the grace period started passing
that gate. The carried-forward projection has no counters, so every cycle
wrote CPU, disk and network zeroes into the history and the persistent
store. Sparklines and stored metrics showed a collapse to zero for a guest
Pulse could not see, rather than a gap. Memory escaped only by accident,
because the projection leaves usage unknown and historyMemoryUsage returns
the -1 sentinel.
Freshly built guests are stamped with the cycle's sample time while
preserved guests keep the LastSeen of the cycle that observed them, so that
field already distinguishes the two. All three loops now gate on
guestObservedInCycle: the efficient path in recordGuestMetrics, and the
traditional VM and container paths, which had the same defect inline.
The guard fails open when LastSeen is absent. Dropping a real sample is the
worse error and the harder one to notice, so no evidence means record.
Regression coverage asserts both directions, and both were verified by
breaking them: removing the guard reports the fabricated zero sample, and
inverting it reports the observed guest losing its sample entirely.
Keep the configured cluster URL as primary and recover discovered member
endpoints asynchronously while it is healthy. Reset member reachability
evidence when the effective address changes.
Refs #1437 and #1493
Back-port v5 fix ffaeea18d to v6. clusterEndpointEffectiveURL now derives
hasFingerprint strictly from the endpoint's own Fingerprint and ignores
the cluster-level value the callers pass. Previously, when only the
primary node had a fingerprint, every fingerprint-less cluster member was
routed to its per-node IP — bypassing hostname TLS certificate validation
while effectively pinning the wrong cert. Adds direct regression cases.
(v6 has no buildClusterClientEndpoints, so the v5 builder-level test is
covered by the unit cases instead.)
pulse-triage[bot]