The poller reads the previous poll's nodes back from the unified read
state, but previousNodeFromView returned the registry's unified resource
ID and no temperature. Every lookup keyed on the Proxmox source ID
missed: nodeLastOnline in the offline grace policy, the temperature
carry, the CPU low/record tracking (reset to the current reading every
poll) and the node memory carry. When a whole instance poll failed,
markPVEInstanceNodesUnreachable wrote the projected nodes back, so each
failed poll re-ingested the node under a new unified ID. Cluster members
also lost the cluster identity their canonical ID derives from, so they
changed ID for the length of an outage.
The projection now returns the source ID, the stored temperature
details, the cluster name and the linked agent ID, and
previousNodesForInstance restores the config-derived identity stamps
(provider-scoped flag, native aliases, TLS fingerprint) the way
pollPVENode writes them.
Reviving those paths needed four bounds:
- The node memory carry uses the poller's own validated snapshot rather
than the read state, which holds the agent's reading on a node merged
with a host agent, and keeps used + cache + free equal to the total.
- A carried temperature keeps a lapsed host agent's lease. A reading
stamped no later than the agent's last report is not carried, so the
five-minute carry window cannot hold a silent agent's reading in
alerts and history past its lease, which #2582 guarantees.
- A carried reading keeps the node's CPU and GPU temperatures only. Its
SMART and NVMe rows also stamp the node's physical disks, so a disk
that went to standby would be stamped and recorded with its pre-sleep
value.
- Outage preservation and authoritative membership reconciliation keep a
node's temperature only under those rules and drop it when the node
goes offline.
TestPollPVEInstanceMarksStaleNodesOfflineWhenGetNodesReturnsEmpty relied
on the dead nodeLastOnline lookup and now ages that sighting too. The
silent-agent test from #2582 now rebuilds the store as production does
and feeds the polled node back, which is what exercises the carry.
The config layer already refuses to consolidate two same-named clusters
whose TOFU-captured TLS fingerprints contradict, and node aggregation
keeps any two same-named clusters from different connection instances
apart unconditionally. Two gaps remained one layer down. First, the
endpoint-IP agent match bypassed the contradiction guard entirely, so
two sites reusing RFC1918 addressing (the MSP support case: pve01 on
192.168.1.11 at both sites) still bound the second site's node to the
first site's host agent, attaching the wrong machine's telemetry.
Second, the unconditional split had no way to recognize the legitimate
duplicate - the same cluster added twice through different member
addresses with no config-level endpoint overlap could never fold back
into one node slot.
The aggregation layer now receives the config layer's identity
evidence: each PVE node carries the TLS certificate fingerprint of its
own named endpoint record (standalone nodes carry the instance
fingerprint; a cluster member never inherits the instance-level
fingerprint, which pins whichever member the connection URL reaches).
Same-named clusters from different instances merge only when both views
carry the same fingerprint; contradicting or unknown evidence keeps the
fail-safe split. Agent binding applies the identical doctrine: hostname
and address matches are rejected when the candidate agent's linked
nodes live in a different named cluster or carry a different
fingerprint, closing the previously unguarded endpoint-IP path.
Reported via support by an MSP whose sites reuse cluster names, node
names, and RFC1918 ranges.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
rcourtman