The thresholds UI stores per-resource overrides keyed by unified
registry resource IDs, while the engine check paths look up legacy
monitor IDs, so a per-node override on an API-only PVE node saved,
displayed as Custom, and never applied: the node kept alerting at the
global defaults. Reproduced live against mock: memory at 81% with a 42%
override stored under the registry ID raised nothing.
Route the node, PMG, PBS, and host/linked-node override lookups through
the canonical registry identity resolver the intent-policy path already
uses, with the exact legacy key still winning so overrides stored by
older UI versions keep resolving unchanged.
Proof: TestNodeThresholdOverrideStoredUnderRegistryIDApplies drives the
real populate path for an API-only node, stores a memory override under
the registry resource ID the UI uses, and asserts CheckNode fires at the
override threshold while controls without the override or without the
resolver stay quiet.
Refs #1738
Contract-Neutral: behavioral fix#1738: threshold override lookup resolves registry identity, no public contract delta
Read paths refresh canonical identity on every clone, so views always
showed the derived aliases, but the stored registry entries never carried
them: snapshot ingest built resources without the canonical block and only
the IngestResources seed path stored refreshed clones. GetByReference's
canonical-alias fallback reads the stored entries directly, so prefixed
refs like "agent:{hostID}" never resolved on a production rebuild and
per-resource alert grace overrides on merged PVE-node + pulse-agent hosts
silently fell back to factory policy. The earlier alias fix (dd008f1c2)
only proved the IngestResources-seeded path. Recompute identities in each
ingest finalize pass, after links and merges settle.
Proof: TestMergedProxmoxHostAgentGraceOverrideHoldsCPUAlert drives the
real populate path (node plus linked agent merging into one resource),
saves a 600s metric.cpu override under the registry resource ID the UI
uses, and asserts CheckHost holds the alert while a control manager
without the override fires immediately.
Refs #1497
Contract-Neutral: behavioral fix#1497: populate derived canonical identity on stored registry entries, clones already exposed identical data, no public contract delta
matchZFSPoolForStorage's sole-pool fallback attached the node's only ZFS
pool to every storage the per-node endpoint returned, including NFS, CIFS,
PBS, and RBD entries that can never be backed by a local pool. One failing
device then raised a duplicate ZFS device alert per shared storage. Gate
the matcher on isInherentlySharedStorageType so remote-backed storages are
never matched, while dir-type storages keep the single-pool fallback.
Refs #1731
An explicit connection address override on a cluster member was silently
discarded when VerifySSL was enabled and the member had no per-endpoint
fingerprint: the hostname-for-TLS preference displaced the operator's
address, so overriding an undialable discovered hostname changed nothing.
The override now wins in every TLS mode, and the hostname preference
applies only to auto-discovered addresses.
Failed endpoint recovery attempts also logged their cause at debug level
only, leaving the recurring 'No endpoints recovered' warning without a
reason. The warning now carries per-endpoint failure reasons and the
sanitized error is stored so the UI health status shows it too.
Refs #1665
Contract-Neutral: behavioral bug fix refs #1665: cluster member IPOverride honored in all TLS modes, recovery failure reasons surfaced; no public contract delta
Proxmox's per-node storage endpoint GET /nodes/{node}/storage does not
return what the node can actually use. It returns every storage in the
datacenter config and reports the ones the node is excluded from with
enabled:0/active:0 instead of leaving them out. pollStorageWithNodes
ingested every row it got back, so a storage restricted to one node
appeared on all the others as a disabled entry, which the UI renders as
Offline.
The poller already fetches the datacenter config once via GetAllStorage
and keeps it keyed by storage name, so the restriction is in hand at the
point each per-node row is turned into a models.Storage. Skip the row
when the config carries a non-empty nodes restriction that does not
include the node being polled. Node names are compared case-insensitively
to match how node identity is compared elsewhere in the poller.
Only the restriction drops a row. A storage with no restriction still
shows up on every node it is reported from, including when it is disabled
everywhere, so a datacenter-wide disabled storage stays visible as
disabled rather than vanishing. Because the shared-storage aggregation
builds Nodes/NodeIDs/NodeCount from the surviving per-node rows, this also
stops a restricted shared storage from claiming cluster members that
cannot mount it.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The discovery-policy check resolved endpoint hostnames with a bare
net.LookupIP while the actual dials went through pkg/tlsutil's process-global
cached resolver, so the policy and the connection reasoned about two different
DNS views. That split is why 108aa4e20 had to skip resolution entirely for the
default policy, which left the injected 169.254.0.0/16 blocklist enforced only
against literal IPs: a hostname endpoint pointed at the metadata range walked
straight through.
Resolve through tlsutil.LookupHostCached instead. The shared resolver caches
answers and lookup failures alike until its next refresh, so repeat poll cycles
cost a cache hit rather than a query and the per-poll DNS volume that opened
#1638 stays gone. With that in place the default-policy skip is removed and the
blocklist applies to resolved addresses again, and the five-minute decision
cache is dropped rather than kept: it bought nothing on top of the resolver
cache, made the verdict trail the configuration, and memoized the fail-open
"resolution failed, allow" outcome for minutes even with an explicit allowlist
configured. Its claim to match a DNS refresh interval that operators configure
through DNS_CACHE_TIMEOUT goes with it.
The SSH backoffs now only escalate for work that ran. A knownhosts manager
suppressing a call inside its own window reports ErrKeyscanSuppressed, and the
temperature layer neither records a failure nor pays for the RPi fallback in
that case. An expired collection deadline is our own budget rather than
evidence about the host, so it holds the window at the floor. Both backoffs
decay once a retry deadline is more than one window past, and replacing the
temperature SSH key on disk clears both maps so a repaired key is tried on the
next cycle instead of after fifteen minutes.
Refs discussion #1638.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The cluster-endpoint discovery-policy check ran a raw net.LookupIP per node
per poll cycle since c5f5af7ab, bypassing the process-global cached resolver,
and the injected default subnet blocklist (169.254.0.0/16) made the
zero-policy fast path unreachable so even unconfigured installs generated
that DNS volume. Evaluate the default link-local-only policy against literal
endpoint IPs without resolution, and memoize custom-policy verdicts per
endpoint for the shared 5-minute DNS-cache TTL so repeat polls stay off the
resolver. Also cache ssh-keyscan failures with doubling backoff in the
knownhosts manager and back off temperature SSH collection per host after
failures instead of re-executing ssh twice per node every 10s cycle.
Refs discussion #1638.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ac0fb263c changed preserved guests to carry their real runtime status
("running") instead of the stringified aggregate status ("online"). The
three guest metric recording loops gate on Status == "running", so guests
carried forward while their node sits in the grace period started passing
that gate. The carried-forward projection has no counters, so every cycle
wrote CPU, disk and network zeroes into the history and the persistent
store. Sparklines and stored metrics showed a collapse to zero for a guest
Pulse could not see, rather than a gap. Memory escaped only by accident,
because the projection leaves usage unknown and historyMemoryUsage returns
the -1 sentinel.
Freshly built guests are stamped with the cycle's sample time while
preserved guests keep the LastSeen of the cycle that observed them, so that
field already distinguishes the two. All three loops now gate on
guestObservedInCycle: the efficient path in recordGuestMetrics, and the
traditional VM and container paths, which had the same defect inline.
The guard fails open when LastSeen is absent. Dropping a real sample is the
worse error and the harder one to notice, so no evidence means record.
Regression coverage asserts both directions, and both were verified by
breaking them: removing the guard reports the fabricated zero sample, and
inverting it reports the observed guest losing its sample entirely.
Keep the configured cluster URL as primary and recover discovered member
endpoints asynchronously while it is healthy. Reset member reachability
evidence when the effective address changes.
Refs #1437 and #1493
Back-port v5 fix ffaeea18d to v6. clusterEndpointEffectiveURL now derives
hasFingerprint strictly from the endpoint's own Fingerprint and ignores
the cluster-level value the callers pass. Previously, when only the
primary node had a fingerprint, every fingerprint-less cluster member was
routed to its per-node IP — bypassing hostname TLS certificate validation
while effectively pinning the wrong cert. Adds direct regression cases.
(v6 has no buildClusterClientEndpoints, so the v5 builder-level test is
covered by the unit cases instead.)
rcourtman