The alert engine fired powered-off alerts for every stopped VM and
container regardless of whether the guest was configured to autostart.
This meant intentionally-stopped guests (onboot=0) generated alarm
fatigue — in the live lab, 7 of 12 alerts were noise from stopped VMs
that are deliberately off.
Now the alert engine checks the Proxmox onboot setting before firing:
- onboot=true → stopped is unexpected, alert (preserved behavior)
- onboot=false → stopped is expected, suppress
- onboot=nil → unknown, alert (preserved behavior)
For containers, onboot is parsed from the already-fetched container
config in enrichContainerMetadata (no new API call). For VMs, the
config is fetched only for stopped VMs via GetVMConfig (one extra call
per stopped VM, running VMs are unaffected).
Models: add OnBoot *bool to VM and Container.
Alerts: add onboot-aware suppression branch in CheckGuest.
Monitoring: add parseProxmoxOnBoot helper and fetchVMOnBoot.
Force stale negative LXC Docker detections to re-run after the Docker checker is configured so explicit Proxmox guest Docker inventory repopulates after backend restarts.
getConfiguredHostIPs() and the new discoveryConfigSnapshot() now deep-copy config under a brief RLock and release it before resolving Proxmox/PBS/PMG hostnames via the injectable lookupConfiguredHostIP seam, so slow or blocked DNS can no longer stall monitor writers. The discovery IPBlocklist is the deduplicated merge of the operator-configured blocklist and the resolved configured-host IPs. Adds lock-freedom and merge coverage tests; updates the monitoring contract obligation.
f62f35e24 restored the v5 used | cache | free memory split for Proxmox
nodes and guests, but standalone host agents still reported a flat
used/free pair, so the Machines page memory bar could not show the
reclaimable segment. Flagged by the Machines page v5 parity audit.
- Host agent reports cacheBytes (gopsutil Available minus Free); the
ZFS ARC adjustment recomputes free so used + cache + free still
covers the total.
- ApplyHostReport maps the field into models.Memory.Cache and clamps
inconsistent or older-agent reports so used + cache never exceeds
total.
- AgentMemoryMeta carries cache onto unified resources so the frontend
agent payload exposes it.
- Mock generic hosts split a third of non-used pages as cache, and the
node-linked host conversion now holds the invariant instead of
stacking the node's cache on top of a recomputed free.
- Contracts: monitoring, unified-resources, and storage-recovery now
document the split (also covering the f62f35e24 node/guest surface,
which landed without contract deltas).
The alert engine stamps metadata.resourceType on every alert, but the
websocket state path converts alerts.Alert to models.Alert, which had no
Metadata field, so every active alert reached the frontend stripped. The
history Type badge then fell back to unified-store lookups that miss
nodes (alert.resourceId is the platform-native node ID while unified
resources mint canonical ids, and alert.resourceName is the raw node
name while unified resources prefer the display name), rendering
Unknown. In mock mode the generated history rows had the same gap.
models.Alert gained the Metadata field in f62f35e24 (it rode along with
the memory-cache commit); this completes the transport:
- copy Metadata in activeAlertsSnapshot (websocket active alerts),
GetRecentlyResolved (resolved alerts to state), and the mock
UpdateAlertSnapshots conversion; sources are deep clones already
- deep-copy Metadata in models cloneAlert to keep the snapshot
clone contract honest
- stamp resourceType in the mock history generator using the real
engine vocabulary (node, vm, system-container)
- recognize system-container in the history Type badge map; that is
what the v6 engine stamps for LXC guests
Verified live in mock mode: history previously resolved 277 of 780
rows to Unknown (all node alerts); now 718/718 rows and 19/19 active
alerts carry resourceType and zero badges render Unknown.
v5 modeled memory as used | cache | free (Memory.Cache, 'reclaimable
buff/cache') and the memory bar's tooltip carried a 'Shown in Proxmox'
row explaining why Pulse's percentage reads lower than the Proxmox UI's
cache-inclusive number — a recurring support question. The v6 rebuild
deleted the field from the backend model, so the split and the
reconciliation vanished product-wide. Flagged by the Proxmox overview
parity audit.
Backend: re-add Memory.Cache; split it out via a shared
splitReclaimableMemory helper at the node resolver (node status reports
truly-free directly) and the VM builder (when guest meminfo reported
free pages); transport as proxmox.memoryCache on unified resources
alongside swap/balloon; mock mode populates plausible cache for nodes
and VMs.
Frontend: cache prop on StackedMemoryBar with the v5 muted-amber
segment between active and balloon, tooltip rows for reclaimable cache,
truly-free Free (balloon-capped), and the 'Shown in Proxmox'
reconciliation; guest and node memory adapters normalize free to
truly-free at the boundary; guest and node drawers grow a Reclaimable
cache row; the Proxmox nodes table passes cache and node swap through.
The unified-resource adapter flattened models.ZFSPool to four scalars
(state + R/W/C totals), so scan status and the per-device report never
reached the frontend. The storage detail's ZFS card, the usage bar's
scrub/resilver overlay and ZFS tooltip, and ZFS-device-to-disk matching
were all dead code on the unified path: a degraded pool rendered as a
one-word State cell with no way to see which device failed or that a
resilver was running. Found by the v5-parity audit of the pools table
(same pipeline disease as #1471 on the disks table).
- Add StorageMeta.ZFSPool and populate it in resourceFromStorage
(normalized copy, not aliased).
- Serialize it in monitorStoragePlatformData so the /api/state path
carries it alongside the existing flat scalars.
- Map it through normalizeStorageMeta and the storage record adapter
(meta first, flat platformData fallback).
- Render the per-device report (name, vdev type, state, R/W/C errors,
message) in the detail's ZFS Pool card - the v6 home for what v5
showed via the health-map dots and the degraded-pool warning row.
- Pin the end-to-end path in code_standards_test and update the
unified-resources, monitoring, storage-recovery, and
frontend-primitives contracts.
Verified live in mock mode: degraded pool now shows the resilver scan,
pool error totals, per-device states, and the bar pulse overlay.
Nodes that set up SSH temperature monitoring before v6.0.0-rc.6 have an
authorized_keys entry locked to `sensors -j`, so the pulse-sensors wrapper
never runs and SMART (SATA/SAS) disk temperatures silently never arrive;
only NVMe shows via kernel hwmon (what the #1471 reporters hit). The server
already detected the legacy payload but only logged it.
- models.Temperature gains LegacySensorsFormat, set by parseSensorsJSON
when the payload is raw sensors -j, and preserved across the host-agent
merge; it rides the existing temperatureDetails projection on unified
node resources.
- New PlatformOutdatedSensorSetupNotice on the Proxmox page, data-gated:
collection succeeded AND payload was legacy AND the node has SATA/SAS
disks without temperatures, so nodes whose disk temps arrive via a host
agent stay clean. Copy points at re-running the node setup script, which
replaces the locked key entry and installs the wrapper.
- monitoring.md contract extended with the legacy-format marker behavior.
The liveness handler called SchedulerHealth(), which refreshes provider
instance caches under the monitor's write lock and snapshots queues and
breakers on every probe — it only needed the dead-letter count. Add
Monitor.DeadLetterCount() (queue-local mutex only) and use it.
The hot-dev watchdog killed the backend after 2 missed 3s probes; on a
dev box running race-detector test suites and builds that is routine
CPU starvation, and each kill destroyed any in-flight Assistant chat
stream (observed repeatedly during the chat-feel audit). Default the
miss threshold to 4 (~20s) — genuine wedges still die quickly.
A full local go test run on 2026-06-11 failed pkg/metrics while vite
builds were saturating the machine. Reproduced under synthetic 8-core
load: only the TestSLO_* wall-clock p95 assertions failed, never the
functional tests. A latency budget measured on a shared dev machine
cannot distinguish host contention from a code regression; load storms
inflated medians up to ~4x with no code change, so no absolute
threshold separates the two.
Route the latency SLO assertions (pkg/metrics, internal/api,
internal/monitoring) and the load/stress perf assertions (internal/api)
through helpers that keep strict enforcement on GitHub Actions runners
(controlled environment, existing hosted-runner envelopes unchanged)
and skip locally on overrun, printing the full p50/p95/p99
distribution in the skip message. A local pass still means the budget
was genuinely met. Error-response and correctness checks remain hard
failures. CI -race runs are unaffected: these tests already skip under
the race detector.
Verified: pkg/metrics, internal/api, and internal/monitoring all pass
with 8 CPU burners saturating the host, the scenario that previously
turned TestSLO_RollupTierBatchedFleet and four other budget tests red.
resourceFromStorage and resourceFromDockerContainer stamped LastSeen with
time.Now() at conversion because their source models carried no poll
timestamp. The registry rebuilds from the retained state snapshot every
cycle, so those resources re-reported a fresh sighting each rebuild even
after their upstream source (PVE instance, docker host agent) stopped
delivering, and their per-source SourceStatus could never go stale via
markStaleLocked. 53faa4e46 fixed this fabrication at the ingest layer but
left these two adapter-level stamps.
- models.Storage gains LastSeen (omitzero), stamped where entries are
built: the PVE storage poll (poll start time, including synthesized
cluster-shared entries; preserved entries for unpolled nodes keep their
old stamp), the PBS datastore conversion (PBS instance sighting), Ceph
pool projection (cluster LastUpdated), and the mock generator (offline
mock nodes get a backdated stamp so the stale path renders).
- resourceFromStorage passes storage.LastSeen through; zero stays zero
("never seen") instead of becoming conversion time.
- resourceFromDockerContainer uses host.LastSeen: containers are delivered
wholesale with each host report, so the host report timestamp is the
container sighting. This matches every other docker sub-resource adapter
(services, tasks, volumes, networks, images already use host.LastSeen).
- ingestStorage routes PBS-poller datastore entries (instance "pbs-<name>",
type pbs) to SourcePBS, parented to the PBS instance. Keying them
SourceProxmox would judge their freshness against the 60s Proxmox stale
threshold while PBS polls every 60s by default, flapping healthy
datastores stale between polls; SourcePBS carries the cadence-matched
120s threshold. PVE-reported pbs-typed storage.cfg backends stay
SourceProxmox. Side effect: syncUnifiedStorageMetrics no longer skips
PBS datastore storage, so those entries gain usage history.
- storageFromReadStateView round-trips LastSeen so the legacy storage API
reports the honest sighting; mock refresh re-stamps available storage on
each simulated poll.
The parent host/node staleness was already honest, so platform pages
reflected outages at the parent level; this makes the per-resource source
freshness honest too.
Ingest stamped every resource's SourceStatus "online" and replaced a zero
LastSeen with time.Now(), so a source that never delivered a resource
(synthesized offline PVE placeholders, never-polled instances) reported a
fresh sighting and a healthy source forever: markStaleLocked skips zero
LastSeen entries, so the stamp was permanently exempt from stale-marking.
That fabricated freshness fed monitor previous-state reads (the
resurrection loop mitigated in 8372a22c5), the frontend lastSeen display,
the monitored-systems ledger, and AI/agent context.
- ingest/mergeInto/IngestResources/presentation coalesce: stamp per-source
delivery status from the actual sighting ("unknown" when the source has
never delivered, "online" otherwise) and preserve zero LastSeen instead
of substituting ingest time.
- replaceRegistry: run MarkStale over the fully assembled registry. The
IngestSnapshot stale pass ran before record sources (TrueNAS, VMware)
were ingested, leaving them stamped "online" at any age.
- monitorLastSeenUnix: report 0 for never-seen instead of time.Now(), so
the frontend renders a dash rather than "just now".
- connected infrastructure: clamp zero LastSeen to 0 ms instead of the
negative UnixMilli of the zero time.
preserveOrExpireNodes keeps requiring a real online sighting for grace;
the registry no longer manufactures the timestamps that made that guard
necessary, so the 8372a22c5 failure mode is impossible by construction.
A configured PVE instance that was already down when Pulse started had
no node state, so the unreachable-poll path returned without writing
anything and the instance was invisible on the Proxmox overview until
its first successful poll. Synthesize offline node entries from the
instance config in that case: cluster configs get one row per
discovered member endpoint with the same node ID a live poll would
assign, standalone instances get a single row named after the
configured instance. The next successful poll replaces them wholesale.
Also stop the offline grace policy from resurrecting such placeholders.
Previous node state round-trips through the unified registry, which
stamps zero LastSeen values with the ingest time, so a never-seen
placeholder came back "recently seen" every cycle and flipped to a
permanent online/degraded state. Grace now requires a real online
sighting: a nodeLastOnline entry from this process, or a previous
snapshot that itself reports the node online with a recent poll
timestamp.
Verified live on the dev stack: a synthetic instance pointing at an
unreachable TEST-NET address appears as an offline node row on the
Proxmox overview immediately after a fresh-state backend start and
stays offline past the grace and stale windows.
Fixes#1433
Two regressions surfaced by generating a real-mode agent report after a
backend restart:
- Report subject lookups used Monitor.GetUnifiedResources (the raw
resource store), but the raw store's canonical IDs depend on per-boot
ingest order for merged-source hosts: after a restart the same host
resolved to a different agent-<hash> than the one the UI and
/api/state advertise, so reports lost the resource name, availability,
and metrics translation entirely. Subject enrichment now reads
Monitor.UnifiedResourceSnapshot and MetricsTargetForResource resolves
through GetUnifiedReadStateOrSnapshot first (raw store as fallback) -
the same re-ingested registry every other read surface uses.
- The performance summary card grid positions cards absolutely and never
paginated: an agent host reporting 8+ metric families walked off the
page bottom, fought fpdf's auto page break, and scattered one orphan
element per page (a 7-day delly report rendered 18 pages, ten of them
near-blank). The grid now starts a new page before a row that will not
fit; the same report renders 9 pages with intact cards.
Verified live: real-mode delly report shows name, 288 data points,
availability, charts, and correctly paginated cards. The underlying
canonical-ID instability (raw store vs re-ingested view, and the
resource_changes journal fragmenting across boot eras) is a separate
root issue tracked for its own fix.
Mock mode seeded rich in-memory chart history but passed a nil store to the
seeder and the live tick, so the sqlite metrics store held almost nothing:
vmware/truenas/docker resources got one row per poll from the platform
ingestion paths and mock Proxmox guests got nothing at all (the unified sync
skips every resource in mock mode). Performance reports rendered
"Data Points: 0" for PVE guests and "Samples: 1" elsewhere, with no charts.
Re-enable store seeding behind an explicit PULSE_MOCK_SEED_METRICS_STORE
opt-in that scripts/hot-dev.sh and scripts/toggle-mock.sh export exactly
where they point PULSE_DATA_DIR at the isolated tmp/mock-data dir. Without
the opt-in (a production install flipping PULSE_MOCK_MODE on its real data
dir) the store stays untouched, which is what the old nil guard protected.
Replace the dormant dense seeding policy (every in-memory timestamp written
to both hourly and daily tiers, ~11M rows at current fixture scale) with
tier-correct backfill driven by the deterministic mock.SampleMetric runtime:
daily 30d at 4h spacing, hourly 7d at 2h, minute 24h at 15m, raw left to the
live tick (now also store-connected under the opt-in). Timestamps sit on the
spacing grid and a new Store.MaxTimestampsForTier coverage query fills only
the gap since the previous boot, so restarts neither duplicate rows nor
re-pay the seed: a fresh seed wrote 264,960 rows in 8.4s (71MB) and a
restart wrote only the gap plus fixture series whose mock IDs are not
boot-stable (k8s pod names, ceph FSID, a pre-existing generator defect).
Verified live in mock mode: 30-day PDF reports for a mock PVE VM
(checkout-web-01), a vSphere VM, the TrueNAS host, and a Docker container
now show 1260/1260/1260/540 data points with rendered time-series charts.
Performance reports answered 'what were the averages' but never 'was my
infrastructure up' - the question a managed-service client reads a
monthly report for. Reports now carry an Availability summary derived
from the recorded resource change timeline (state_transition entries
keyed by the canonical unified ID):
- uptime percent over the observed portion of the window, outage count,
total downtime, and longest outage, rendered in the executive summary
with an explicit semantics note; fleet summaries gain a per-resource
Uptime column and CSV exports gain availability header lines
- absent/unknown spans are unobserved time: excluded from the uptime
math entirely and disclosed as coverage, never counted as downtime.
The journal records a registry absence for every monitor restart, so
treating gaps as outages would invent fleet-wide downtime every time
the operator restarts Pulse
- warning states count as up (the resource is reachable and serving);
the uptime label clamps rounding so any real downtime can never
display as a clean 100%
- resources with no timeline render no availability section at all
rather than a fabricated number
Verified live against a real 7-day window: uptime/outage/downtime
figures reconcile with the raw resource_changes journal.
Performance reports were structurally disconnected from the v6 ID
space: the UI (and any API caller working from /api/state) addresses
resources by canonical unified ID, while the metrics store is keyed by
each platform's native source ID (the resource's metricsTarget). The
engine queried the store with the unified ID verbatim, so every report
rendered 'Data Points: 0' regardless of how much history existed, and
covers showed raw hash IDs a report reader cannot map to a machine.
- MetricReportRequest gains MetricsResourceID: handlers resolve the
unified ID through the tenant monitor's resource store (new
Monitor.MetricsTargetForResource accessor; the registry computes
targets on demand, they are not persisted on snapshot structs) and
the engine uses it for store queries only. Recovery points and
Patrol findings stay keyed by the unified ID.
- Legacy snapshot models and their alerts are keyed by the metrics
target ID, so enrichment now matches either ID space and resource
names/status resolve again on covers, headers, and fleet rows.
- Fleet summaries mirror the single-report guard: zero data points
across the fleet renders a muted NO DATA card instead of a green
HEALTHY 'All systems operating normally' - false reassurance is the
worst failure mode for a client-facing stability report.
- Em dashes in PDF-bound literals become hyphens; fpdf core fonts are
cp1252 and rendered them as mojibake.
Cluster endpoints were discovered once at add time and never re-read:
detectClusterMembership returned immediately for instances already marked
IsCluster, so when a cluster moved subnets Pulse kept dialing the dead
per-node addresses for failover and kept displaying them on Settings ->
Infrastructure, even though monitoring still worked through the main host
fallback. The manual /refresh-cluster endpoint exists but is not wired to
any UI control, so nothing ever corrected the stored addresses.
The 5-minute cluster re-check now also covers cluster instances: it
re-reads /cluster/status through the configured main host, and when a
node's address or the node set actually changed (volatile fields like
Online/LastSeen are ignored) it updates the stored endpoints, persists the
config, and rebuilds the failover client so polling moves to the new
addresses. User-managed fields (IP overrides, guest URLs, fingerprints)
are preserved by the existing discovery helpers, and fields the cluster
status API omits are inherited from the stored endpoint rather than
erased.
Refs #1493
recordTaskResult reset the failure counters on a successful poll but left
LastErrorAt/LastErrorMessage/LastErrorCategory in place forever. The
connections aggregator surfaces that field as a live error banner on
Settings -> Infrastructure and derives the Unauthorized state from it, so
one transient cluster outage (or startup blip) pinned a red "no healthy
nodes available" banner onto a connection that polls fine, and a single
past 401 could pin a healthy connection to Unauthorized.
LastError now means "current outstanding failure since the last success":
recorded on failure, cleared on success. Connections that are genuinely
failing (no successful poll) keep their banner.
Refs #1493
pollPVEInstance returned early on a GetNodes error, so a host that
stopped answering entirely (shutdown, network loss) kept its last
successful snapshot in state forever: node row frozen at online with
stale uptime. The existing 60s offline grace policy only ran for
cluster-reported peer-down and empty-poll cycles, never for a whole
instance going dark.
Route the poll error path through the same grace policy: nodes of an
unreachable instance stay online (connection degraded) within the
grace window and flip to offline with cleared uptime and CPU once it
lapses, matching the cluster-reported node-down behavior.
Verified live: registered a synthetic PVE instance, killed it, and the
node flips to offline one poll cycle after the grace period.
Fixes#1441
Two gaps found by exercising the MSP pilot path live on a throwaway
multi-tenant instance:
1. CheckAccess granted any authenticated principal access to the default
org, so a token bound to a client org could read the provider's own
default-org estate if it leaked from a client site. Org-bound tokens
now fall through to the explicit binding check for the default org;
authenticated users and legacy unbound tokens keep default-org access,
and binding "default" explicitly still grants it.
2. The webhook private-target allowlist (instance-wide system setting)
only ever reached the default org's notification manager on
startup/reload, and only the request-context org on settings update.
Tenant orgs' webhooks to private targets (per-client Gotify over VPN,
the canonical MSP alert route) failed SSRF validation with no org-side
remedy, and any allowlist died with a restart. Settings updates and
reloads now fan out to every live tenant manager via the new
MultiTenantMonitor.ForEachMonitor, and tenant monitors inherit the
persisted allowlist and public URL at creation.
Both fixes verified live: org-bound token vs default org returns 403;
client-org webhooks to a private target succeed after restart and for
orgs created after the allowlist was saved. MSP.md validation checklist
gains the default-org probe and the allowlist guidance; MULTI_TENANT.md
documents the binding semantics. Contracts updated for api-contracts,
security-privacy, and monitoring with adjacency notes for
agent-lifecycle, storage-recovery, and performance-and-scalability.
Alert webhook payloads now carry the tenant that fired them, so MSP/PSA
receivers (ConnectWise and similar) can route tickets by client without
inferring the tenant from which webhook endpoint fired.
- WebhookPayloadData gains TenantID/TenantName, exposed to custom
templates as {{.TenantID}}/{{.TenantName}}.
- Defaults come from PULSE_TENANT_ID/PULSE_TENANT_NAME (already injected
into provider-hosted client runtimes; name falls back to ID).
- Shared-process multi-tenant orgs override via a lazy org-backed
resolver wired in MultiTenantMonitor.GetMonitor, so display-name
renames are picked up without restart.
- Generic service template emits a tenant block when identity is set,
omits it otherwise; single-tenant payloads are unchanged.
- Notifications and monitoring subsystem contracts updated with the
tenant-identity ownership boundary; guardrail test pins the org
wiring.
Clears the six dupl pairs in internal/monitoring:
- poll_providers.go: PVE/PBS/PMG listInstances / describeInstances /
connectionStatus closures collapse into generic sortedClientNames,
describeProviderInstances, and providerConnectionStatuses helpers; the
PBS/PMG adapters are now built by newPrefixedPollProvider from a
prefixedPollProviderSpec (prefix drives status keys, health keys, and
the fallback instance name).
- truenas_poller.go / vmware_poller.go: the Start scheduling loop
(double-start guard, stopped-channel handshake, sync+poll cadence)
moves to startPollerLoop, and the active-connection config policy
(enabled instances keyed by trimmed connection ID, defaults applied)
moves to loadActiveInstanceConfigs, both in the new
platform_poller_shared.go.
- monitor_polling_vm.go / monitor_polling_containers.go: traditional
polling now records guest series via the existing canonical
recordGuestMetric helper (io/network sentinels preserve the historical
cpu/memory/disk-only behavior on this path); the source-shape
guardrails in canonical_guardrails_test.go and
memory_source_catalog_test.go pin the new delegation.
- mock_metrics_history.go: native and TrueNAS disk seeding share one
seedDiskTelemetry closure.
Proof: new TestTrueNASPollerDoubleStartKeepsSingleLoop pins the shared
lifecycle loop; new TestSeedMockMetricsHistory_DiskTelemetryParityAcrossNativeAndTrueNAS
pins four-series parity for both disk sources. Contract Extension Points
name the shared wiring layer and poller scaffold.
Full internal/monitoring test suite passes.
golangci-lint run ./... failed on ~190 pre-existing errcheck violations and
5 unformatted files, burying any new regression in noise. Fix all of them:
- Test files that hand-rolled mock-mode set/restore (vmware, truenas, and
friends) now use the canonical setMockModeForTest/testutil.SetMockMode
helper instead of drift copies that ignored SetEnabled errors.
- internal/mock and internal/monitoring tests get package-local
mustSetEnabled/mustSetMockEnabled/mustSetMonitorMockMode helpers that
fail the test on toggle errors.
- pkg/auth/sqlite_manager.go, pkg/metrics/store.go, pkg/server/server.go:
rollbacks in defers use the explicit-discard idiom, migration renames and
rollup commits log failures, the hosted reaper goroutine logs an error
exit, shutdown mock-disable logs failures.
- Remaining test sites check errors with t.Fatalf/t.Errorf or explicitly
discard best-effort calls (restore-chmods, handler-closure unmarshals)
per existing repo style.
- gofmt: internal/api/maintenance_verification.go, internal/ai/demo.go and
three findings test files.
Only dupl findings remain (44 pre-existing production-code duplication
pairs) — those need real refactors, not mechanical fixes.
Full test suites pass for every touched package.
Tighten v5-to-v6 upgrade safety, release installability, provider MSP mode handling, AI cost accounting, metrics flushing, and frontend guardrails for the v6.0.0 GA candidate.
Back-port v5 fix 333e66a8e to v6, adapted to v6's DockerHostView accessors.
findMatchingDockerHost now skips an agentID/token match when
dockerHostIdentityConflicts detects a different MachineID, Swarm NodeID, or
(absent stronger IDs) hostname, so two physically different Docker nodes
that share a token+agentID are not collapsed into one host record. Adds a
regression test.
Back-port v5 fix ae6b663e9 to v6. Removes the 'only fetch ZFS pools when a
zfspool/zfs/local-zfs storage exists' gate (you cannot tell whether a dir
storage sits on a ZFS dataset without the pool list; fetch is gated by
PULSE_DISABLE_ZFS_MONITORING and empty-fast on non-ZFS nodes) and the
type guard before matchZFSPoolForStorage, so a dir storage whose path
resolves to a pool (e.g. /rpool/data) now gets ZFS health attached. The
guardrail-pinned 'if pool := matchZFSPoolForStorage(...)' call form is
preserved. Adds a poll integration test.
Back-port v5 fix 22f8dd3bb to v6. Adds normalizeSMARTDeviceIdentifier
(strips a trailing ' [scsi]'/'[ata]'-style type annotation and the /dev/
prefix) and uses it in the device-path match for both mergeNVMeTempsIntoDisks
and mergeHostAgentSMARTIntoDisks, with a non-empty guard. Without it, disks
whose SMART entries carry a typed label never matched by device path, so
their temperature stayed 0. Function signatures/call sites are unchanged
(canonical guardrail still passes). Adds a unit test.
Back-port v5 fix ffaeea18d to v6. clusterEndpointEffectiveURL now derives
hasFingerprint strictly from the endpoint's own Fingerprint and ignores
the cluster-level value the callers pass. Previously, when only the
primary node had a fingerprint, every fingerprint-less cluster member was
routed to its per-node IP — bypassing hostname TLS certificate validation
while effectively pinning the wrong cert. Adds direct regression cases.
(v6 has no buildClusterClientEndpoints, so the v5 builder-level test is
covered by the unit cases instead.)
Back-port v5 fix 0dca8a037 to v6. Extract pollPVEBackupsAndSnapshots:
the two backup-inventory scans share a bounded backupCtx, which is then
cancelled, and pollGuestSnapshots runs on the parent context. Because
the parent has no deadline, pollGuestSnapshots establishes its own
60s-4min budget instead of inheriting an already-exhausted backup
deadline and skipping entirely (the v6 early-return at
monitor_backups.go made this strictly worse). Adds a regression test
proving snapshots still poll after the storage scan exhausts its budget.
Back-port v5 fix 398ef8117 to v6, adapted to v6's canonical alert keying.
pollStorageWithNodes now calls SyncStorageAlertsForInstance after the
per-storage checks, clearing storage-inventory alerts whose ResourceID is
no longer present. Matches by ResourceID (every v6 storage alert sets
ResourceID=storage.ID) rather than reconstructed alert keys, so it is
robust to v6's composite alert-state IDs (e.g. ...::metric-threshold:usage).
Conservative: live storage keeps its alert (ResourceID still valid),
non-storage alerts are untouched, Ceph alias IDs are treated as valid.
Without this, a storage that disappears or changes identity left its
alert visible until the multi-day stale-alert TTL. Adds unit + poll
integration regression tests.
Dead-code sweep. Functions flagged unreachable by golang.org/x/tools/cmd/deadcode
and confirmed unused across pulse, pulse-enterprise, pulse-pro and pulse-mobile by
adversarial cross-repo verification. Cross-module reachability was checked
explicitly (only pkg/ exported symbols are importable by other modules; internal/
packages and _test.go files are not). go build, go vet and test-compile all pass.
When the same physical Ceph cluster is reported by both the Proxmox API
poller and a Pulse host-agent (same FSID), v5 stored it under two pool-ID
namespaces and ran alert evaluation on the raw, un-deduplicated list,
producing duplicate/flapping Ceph pool alerts and a per-pool override that
appeared to revert between identities.
v6 closes this without the v5 'agent:'-prefix patch: both ingest paths
reconcile by FSID before alert evaluation. State.UpsertCephCluster
(host-agent) and State.UpdateCephClustersForInstance (Proxmox) both funnel
through upsertCephClusterInSlice, which collapses same-FSID clusters into a
single reconciled entry, and both call sites run checkCephPoolStorage on the
STORED cluster. The per-pool override resolves across source aliases via
storageThresholdLookupIDs/cephPoolStorageSourceAliasID, and
clearStorageAliasAlerts removes any alias-keyed alert.
This adds a regression test asserting the observable outcome for the exact
dual-source scenario (host-agent + Proxmox API, same FSID, one pool at ~61%
with a 50% per-pool override): exactly one active Ceph pool usage alert,
override honored, no second/flapping alert across either ingest order or
repeated interleaved cycles. No behavior change.
rcourtman