Two root causes behind #1516's remaining reports:
- A node whose Proxmox disks/list query fails (PVE probes SMART per disk
inside that call, so dozens of disks can exceed the API window) now
falls back to the linked host agent's smartctl inventory instead of
leaving the Physical Disks view empty. Each node also gets its own
attempt window so one slow node no longer starves the rest of the
cluster, and a poll that runs out of budget saves partial results.
- The -n standby probe guard is dropped for positively confirmed
non-rotational devices in both the host agent and the node sensor
wrapper. The guard exists to avoid spinning up sleeping HDDs; an SSD
has nothing to spin up, and some SATA SSDs answer CHECK POWER MODE
with a bogus standby state that permanently hid their temperature,
attributes and history.
Refs #1516
The alert engine fired powered-off alerts for every stopped VM and
container regardless of whether the guest was configured to autostart.
This meant intentionally-stopped guests (onboot=0) generated alarm
fatigue — in the live lab, 7 of 12 alerts were noise from stopped VMs
that are deliberately off.
Now the alert engine checks the Proxmox onboot setting before firing:
- onboot=true → stopped is unexpected, alert (preserved behavior)
- onboot=false → stopped is expected, suppress
- onboot=nil → unknown, alert (preserved behavior)
For containers, onboot is parsed from the already-fetched container
config in enrichContainerMetadata (no new API call). For VMs, the
config is fetched only for stopped VMs via GetVMConfig (one extra call
per stopped VM, running VMs are unaffected).
Models: add OnBoot *bool to VM and Container.
Alerts: add onboot-aware suppression branch in CheckGuest.
Monitoring: add parseProxmoxOnBoot helper and fetchVMOnBoot.
Ingest stamped every resource's SourceStatus "online" and replaced a zero
LastSeen with time.Now(), so a source that never delivered a resource
(synthesized offline PVE placeholders, never-polled instances) reported a
fresh sighting and a healthy source forever: markStaleLocked skips zero
LastSeen entries, so the stamp was permanently exempt from stale-marking.
That fabricated freshness fed monitor previous-state reads (the
resurrection loop mitigated in 8372a22c5), the frontend lastSeen display,
the monitored-systems ledger, and AI/agent context.
- ingest/mergeInto/IngestResources/presentation coalesce: stamp per-source
delivery status from the actual sighting ("unknown" when the source has
never delivered, "online" otherwise) and preserve zero LastSeen instead
of substituting ingest time.
- replaceRegistry: run MarkStale over the fully assembled registry. The
IngestSnapshot stale pass ran before record sources (TrueNAS, VMware)
were ingested, leaving them stamped "online" at any age.
- monitorLastSeenUnix: report 0 for never-seen instead of time.Now(), so
the frontend renders a dash rather than "just now".
- connected infrastructure: clamp zero LastSeen to 0 ms instead of the
negative UnixMilli of the zero time.
preserveOrExpireNodes keeps requiring a real online sighting for grace;
the registry no longer manufactures the timestamps that made that guard
necessary, so the 8372a22c5 failure mode is impossible by construction.
A configured PVE instance that was already down when Pulse started had
no node state, so the unreachable-poll path returned without writing
anything and the instance was invisible on the Proxmox overview until
its first successful poll. Synthesize offline node entries from the
instance config in that case: cluster configs get one row per
discovered member endpoint with the same node ID a live poll would
assign, standalone instances get a single row named after the
configured instance. The next successful poll replaces them wholesale.
Also stop the offline grace policy from resurrecting such placeholders.
Previous node state round-trips through the unified registry, which
stamps zero LastSeen values with the ingest time, so a never-seen
placeholder came back "recently seen" every cycle and flipped to a
permanent online/degraded state. Grace now requires a real online
sighting: a nodeLastOnline entry from this process, or a previous
snapshot that itself reports the node online with a recent poll
timestamp.
Verified live on the dev stack: a synthetic instance pointing at an
unreachable TEST-NET address appears as an offline node row on the
Proxmox overview immediately after a fresh-state backend start and
stays offline past the grace and stale windows.
Fixes#1433
pollPVEInstance returned early on a GetNodes error, so a host that
stopped answering entirely (shutdown, network loss) kept its last
successful snapshot in state forever: node row frozen at online with
stale uptime. The existing 60s offline grace policy only ran for
cluster-reported peer-down and empty-poll cycles, never for a whole
instance going dark.
Route the poll error path through the same grace policy: nodes of an
unreachable instance stay online (connection degraded) within the
grace window and flip to offline with cleared uptime and CPU once it
lapses, matching the cluster-reported node-down behavior.
Verified live: registered a synthetic PVE instance, killed it, and the
node flips to offline one poll cycle after the grace period.
Fixes#1441
Tighten v5-to-v6 upgrade safety, release installability, provider MSP mode handling, AI cost accounting, metrics flushing, and frontend guardrails for the v6.0.0 GA candidate.
The deferred recordTaskResult call was passing pollErr as a function
argument, so it captured the value at defer-time (always nil) instead
of the value at execution time. Result: the per-instance pollStatusMap
treated every poll as a success — LastSuccess was set to "now" on
every cycle, ConsecutiveFailures stayed at zero, and the circuit
breaker never opened, even when the staleness tracker (which used a
proper closure) recorded the same poll as a failure.
The connections aggregator derives state from PollStatus.LastSuccess,
so the Connections UI reported broken PVE/PBS/PMG instances as
"active / verified / healthy" while no data was ingested. Wrap the
recordTaskResult call in a defer closure so it reads the live pollErr
at execution time, matching the pollMetrics and stalenessTracker
defers immediately above.
Introduces GET /api/connections and POST /api/connections/probe as the
backend half of the one-ledger / one-editor connection redesign.
- GET /api/connections aggregates PVE/PBS/PMG/VMware/TrueNAS/agent rows
into a unified Connection shape with derived state (active, paused,
unauthorized, unreachable, stale, pending) computed from in-memory
scheduler health plus agent Host.LastSeen. No new persisted state.
- POST /api/connections/probe fingerprints a host across the five
supported products in parallel (2s dial + 1s read, 3s total, max 5
concurrent). Admin-gated (RequireAdmin + ScopeSettingsWrite) to block
unauthenticated SSRF against internal hosts.
- Disabled bool on PVEInstance/PBSInstance/PMGInstance (zero-value =
enabled, preserves existing nodes.json); pollers skip disabled
instances at client init, reconnect, and per-node iteration.
- NodeConfigRequest/Response gain Enabled; write path translates
*bool -> Disabled so omitted field leaves state untouched.
- ConnectionsAPI frontend client (list/probe) typed off the Go shape.
Contracts updated: api-contracts, monitoring, agent-lifecycle,
performance-and-scalability, storage-recovery. Proofs added:
contract_test.go JSON snapshot for Connection and ProbeResponse,
monitoring guardrails for the Disabled-skip behavior, and a vitest
mock-client test for ConnectionsAPI.
Frontend editor / drawer / table rewrite lands in a separate block.
rcourtman