Cluster endpoints were discovered once at add time and never re-read:
detectClusterMembership returned immediately for instances already marked
IsCluster, so when a cluster moved subnets Pulse kept dialing the dead
per-node addresses for failover and kept displaying them on Settings ->
Infrastructure, even though monitoring still worked through the main host
fallback. The manual /refresh-cluster endpoint exists but is not wired to
any UI control, so nothing ever corrected the stored addresses.
The 5-minute cluster re-check now also covers cluster instances: it
re-reads /cluster/status through the configured main host, and when a
node's address or the node set actually changed (volatile fields like
Online/LastSeen are ignored) it updates the stored endpoints, persists the
config, and rebuilds the failover client so polling moves to the new
addresses. User-managed fields (IP overrides, guest URLs, fingerprints)
are preserved by the existing discovery helpers, and fields the cluster
status API omits are inherited from the stored endpoint rather than
erased.
Refs #1493
recordTaskResult reset the failure counters on a successful poll but left
LastErrorAt/LastErrorMessage/LastErrorCategory in place forever. The
connections aggregator surfaces that field as a live error banner on
Settings -> Infrastructure and derives the Unauthorized state from it, so
one transient cluster outage (or startup blip) pinned a red "no healthy
nodes available" banner onto a connection that polls fine, and a single
past 401 could pin a healthy connection to Unauthorized.
LastError now means "current outstanding failure since the last success":
recorded on failure, cleared on success. Connections that are genuinely
failing (no successful poll) keep their banner.
Refs #1493
pollPVEInstance returned early on a GetNodes error, so a host that
stopped answering entirely (shutdown, network loss) kept its last
successful snapshot in state forever: node row frozen at online with
stale uptime. The existing 60s offline grace policy only ran for
cluster-reported peer-down and empty-poll cycles, never for a whole
instance going dark.
Route the poll error path through the same grace policy: nodes of an
unreachable instance stay online (connection degraded) within the
grace window and flip to offline with cleared uptime and CPU once it
lapses, matching the cluster-reported node-down behavior.
Verified live: registered a synthetic PVE instance, killed it, and the
node flips to offline one poll cycle after the grace period.
Fixes#1441
Two gaps found by exercising the MSP pilot path live on a throwaway
multi-tenant instance:
1. CheckAccess granted any authenticated principal access to the default
org, so a token bound to a client org could read the provider's own
default-org estate if it leaked from a client site. Org-bound tokens
now fall through to the explicit binding check for the default org;
authenticated users and legacy unbound tokens keep default-org access,
and binding "default" explicitly still grants it.
2. The webhook private-target allowlist (instance-wide system setting)
only ever reached the default org's notification manager on
startup/reload, and only the request-context org on settings update.
Tenant orgs' webhooks to private targets (per-client Gotify over VPN,
the canonical MSP alert route) failed SSRF validation with no org-side
remedy, and any allowlist died with a restart. Settings updates and
reloads now fan out to every live tenant manager via the new
MultiTenantMonitor.ForEachMonitor, and tenant monitors inherit the
persisted allowlist and public URL at creation.
Both fixes verified live: org-bound token vs default org returns 403;
client-org webhooks to a private target succeed after restart and for
orgs created after the allowlist was saved. MSP.md validation checklist
gains the default-org probe and the allowlist guidance; MULTI_TENANT.md
documents the binding semantics. Contracts updated for api-contracts,
security-privacy, and monitoring with adjacency notes for
agent-lifecycle, storage-recovery, and performance-and-scalability.
Alert webhook payloads now carry the tenant that fired them, so MSP/PSA
receivers (ConnectWise and similar) can route tickets by client without
inferring the tenant from which webhook endpoint fired.
- WebhookPayloadData gains TenantID/TenantName, exposed to custom
templates as {{.TenantID}}/{{.TenantName}}.
- Defaults come from PULSE_TENANT_ID/PULSE_TENANT_NAME (already injected
into provider-hosted client runtimes; name falls back to ID).
- Shared-process multi-tenant orgs override via a lazy org-backed
resolver wired in MultiTenantMonitor.GetMonitor, so display-name
renames are picked up without restart.
- Generic service template emits a tenant block when identity is set,
omits it otherwise; single-tenant payloads are unchanged.
- Notifications and monitoring subsystem contracts updated with the
tenant-identity ownership boundary; guardrail test pins the org
wiring.
Clears the six dupl pairs in internal/monitoring:
- poll_providers.go: PVE/PBS/PMG listInstances / describeInstances /
connectionStatus closures collapse into generic sortedClientNames,
describeProviderInstances, and providerConnectionStatuses helpers; the
PBS/PMG adapters are now built by newPrefixedPollProvider from a
prefixedPollProviderSpec (prefix drives status keys, health keys, and
the fallback instance name).
- truenas_poller.go / vmware_poller.go: the Start scheduling loop
(double-start guard, stopped-channel handshake, sync+poll cadence)
moves to startPollerLoop, and the active-connection config policy
(enabled instances keyed by trimmed connection ID, defaults applied)
moves to loadActiveInstanceConfigs, both in the new
platform_poller_shared.go.
- monitor_polling_vm.go / monitor_polling_containers.go: traditional
polling now records guest series via the existing canonical
recordGuestMetric helper (io/network sentinels preserve the historical
cpu/memory/disk-only behavior on this path); the source-shape
guardrails in canonical_guardrails_test.go and
memory_source_catalog_test.go pin the new delegation.
- mock_metrics_history.go: native and TrueNAS disk seeding share one
seedDiskTelemetry closure.
Proof: new TestTrueNASPollerDoubleStartKeepsSingleLoop pins the shared
lifecycle loop; new TestSeedMockMetricsHistory_DiskTelemetryParityAcrossNativeAndTrueNAS
pins four-series parity for both disk sources. Contract Extension Points
name the shared wiring layer and poller scaffold.
Full internal/monitoring test suite passes.
golangci-lint run ./... failed on ~190 pre-existing errcheck violations and
5 unformatted files, burying any new regression in noise. Fix all of them:
- Test files that hand-rolled mock-mode set/restore (vmware, truenas, and
friends) now use the canonical setMockModeForTest/testutil.SetMockMode
helper instead of drift copies that ignored SetEnabled errors.
- internal/mock and internal/monitoring tests get package-local
mustSetEnabled/mustSetMockEnabled/mustSetMonitorMockMode helpers that
fail the test on toggle errors.
- pkg/auth/sqlite_manager.go, pkg/metrics/store.go, pkg/server/server.go:
rollbacks in defers use the explicit-discard idiom, migration renames and
rollup commits log failures, the hosted reaper goroutine logs an error
exit, shutdown mock-disable logs failures.
- Remaining test sites check errors with t.Fatalf/t.Errorf or explicitly
discard best-effort calls (restore-chmods, handler-closure unmarshals)
per existing repo style.
- gofmt: internal/api/maintenance_verification.go, internal/ai/demo.go and
three findings test files.
Only dupl findings remain (44 pre-existing production-code duplication
pairs) — those need real refactors, not mechanical fixes.
Full test suites pass for every touched package.
Tighten v5-to-v6 upgrade safety, release installability, provider MSP mode handling, AI cost accounting, metrics flushing, and frontend guardrails for the v6.0.0 GA candidate.
Back-port v5 fix 333e66a8e to v6, adapted to v6's DockerHostView accessors.
findMatchingDockerHost now skips an agentID/token match when
dockerHostIdentityConflicts detects a different MachineID, Swarm NodeID, or
(absent stronger IDs) hostname, so two physically different Docker nodes
that share a token+agentID are not collapsed into one host record. Adds a
regression test.
Back-port v5 fix ae6b663e9 to v6. Removes the 'only fetch ZFS pools when a
zfspool/zfs/local-zfs storage exists' gate (you cannot tell whether a dir
storage sits on a ZFS dataset without the pool list; fetch is gated by
PULSE_DISABLE_ZFS_MONITORING and empty-fast on non-ZFS nodes) and the
type guard before matchZFSPoolForStorage, so a dir storage whose path
resolves to a pool (e.g. /rpool/data) now gets ZFS health attached. The
guardrail-pinned 'if pool := matchZFSPoolForStorage(...)' call form is
preserved. Adds a poll integration test.
Back-port v5 fix 22f8dd3bb to v6. Adds normalizeSMARTDeviceIdentifier
(strips a trailing ' [scsi]'/'[ata]'-style type annotation and the /dev/
prefix) and uses it in the device-path match for both mergeNVMeTempsIntoDisks
and mergeHostAgentSMARTIntoDisks, with a non-empty guard. Without it, disks
whose SMART entries carry a typed label never matched by device path, so
their temperature stayed 0. Function signatures/call sites are unchanged
(canonical guardrail still passes). Adds a unit test.
Back-port v5 fix ffaeea18d to v6. clusterEndpointEffectiveURL now derives
hasFingerprint strictly from the endpoint's own Fingerprint and ignores
the cluster-level value the callers pass. Previously, when only the
primary node had a fingerprint, every fingerprint-less cluster member was
routed to its per-node IP — bypassing hostname TLS certificate validation
while effectively pinning the wrong cert. Adds direct regression cases.
(v6 has no buildClusterClientEndpoints, so the v5 builder-level test is
covered by the unit cases instead.)
Back-port v5 fix 0dca8a037 to v6. Extract pollPVEBackupsAndSnapshots:
the two backup-inventory scans share a bounded backupCtx, which is then
cancelled, and pollGuestSnapshots runs on the parent context. Because
the parent has no deadline, pollGuestSnapshots establishes its own
60s-4min budget instead of inheriting an already-exhausted backup
deadline and skipping entirely (the v6 early-return at
monitor_backups.go made this strictly worse). Adds a regression test
proving snapshots still poll after the storage scan exhausts its budget.
Back-port v5 fix 398ef8117 to v6, adapted to v6's canonical alert keying.
pollStorageWithNodes now calls SyncStorageAlertsForInstance after the
per-storage checks, clearing storage-inventory alerts whose ResourceID is
no longer present. Matches by ResourceID (every v6 storage alert sets
ResourceID=storage.ID) rather than reconstructed alert keys, so it is
robust to v6's composite alert-state IDs (e.g. ...::metric-threshold:usage).
Conservative: live storage keeps its alert (ResourceID still valid),
non-storage alerts are untouched, Ceph alias IDs are treated as valid.
Without this, a storage that disappears or changes identity left its
alert visible until the multi-day stale-alert TTL. Adds unit + poll
integration regression tests.
Dead-code sweep. Functions flagged unreachable by golang.org/x/tools/cmd/deadcode
and confirmed unused across pulse, pulse-enterprise, pulse-pro and pulse-mobile by
adversarial cross-repo verification. Cross-module reachability was checked
explicitly (only pkg/ exported symbols are importable by other modules; internal/
packages and _test.go files are not). go build, go vet and test-compile all pass.
When the same physical Ceph cluster is reported by both the Proxmox API
poller and a Pulse host-agent (same FSID), v5 stored it under two pool-ID
namespaces and ran alert evaluation on the raw, un-deduplicated list,
producing duplicate/flapping Ceph pool alerts and a per-pool override that
appeared to revert between identities.
v6 closes this without the v5 'agent:'-prefix patch: both ingest paths
reconcile by FSID before alert evaluation. State.UpsertCephCluster
(host-agent) and State.UpdateCephClustersForInstance (Proxmox) both funnel
through upsertCephClusterInSlice, which collapses same-FSID clusters into a
single reconciled entry, and both call sites run checkCephPoolStorage on the
STORED cluster. The per-pool override resolves across source aliases via
storageThresholdLookupIDs/cephPoolStorageSourceAliasID, and
clearStorageAliasAlerts removes any alias-keyed alert.
This adds a regression test asserting the observable outcome for the exact
dual-source scenario (host-agent + Proxmox API, same FSID, one pool at ~61%
with a 50% per-pool override): exactly one active Ceph pool usage alert,
override honored, no second/flapping alert across either ingest order or
repeated interleaved cycles. No behavior change.
On a Proxmox node, physical disks collected by the host agent were keyed by
the NVMe controller (e.g. "nvme0 [nvme]") instead of the namespace, reported
sizeBytes 0 (or a stale filesystem-usage value), and flickered as the agent
reading intermittently replaced the authoritative Proxmox disks/list reading.
Root causes:
- smartctl --scan-open reports NVMe disks by their controller char device
(/dev/nvme0), and that scan label became the reported devPath.
- DiskSMART carried no capacity, so the server backfilled size by matching the
SMART device against host filesystem-usage entries, which never match a whole
partitioned/LVM/ZFS disk, leaving size 0.
- The unified-resource merge let the agent's controller label overwrite the
canonical Proxmox /dev/... devPath.
Fixes:
- The agent now reports the canonical block device (an NVMe controller resolves
to its namespace) and the authoritative capacity from /sys/block, with the
smartctl user_capacity / nvme_total_capacity as a cross-platform fallback.
Disks behind multiplexing controllers (megaraid, cciss, areca) keep their
disambiguating label and smartctl-reported size.
- SizeBytes flows through the agent report, host model, and adapter; the
filesystem-usage match is demoted to a legacy fallback.
- The merge keeps a canonical /dev/<device> devPath and never downgrades it to
a scan label, so an un-updated agent can no longer corrupt Proxmox data.
Refs #1483.
Prefer QEMU guest-agent MemAvailable when Proxmox reports saturated VM memory without guest free fields.
Add regression coverage for the issue #1319 Windows fsinfo volume payload so usable C/E/F volumes remain counted while System Reserved partitions are skipped.
Refs #1319
Fixes#1341
Forward-port of the release/5.1 fix (ce1607694).
When a Pulse host-agent reports Ceph data, ApplyHostReport upserts the
cluster into state but only the Proxmox-API polling path ran
checkCephPoolStorage. Users with agent-reported Ceph (instance prefix
"agent:hostname") saved per-pool overrides under agent-prefixed IDs
that the alert manager never evaluated. The threshold appeared to
save (and showed Custom in the UI), but the polling cycle was
checking a different storage ID, so the alert silently stayed
dormant. Run checkCephPoolStorage right after the agent upsert so
the override key actually drives evaluation.
Two real bugs that surfaced once the 20m test timeout let the
internal/api and internal/monitoring packages run to completion.
cloneVMwareInventoryMetrics omitted four fields:
Commit 23ea4e487 (Surface vSphere VM uptime and guest disk usage)
added UptimeSeconds, DiskUsedBytes, DiskTotalBytes, and DiskPercent
to vmware.InventoryMetrics but did not extend
cloneVMwareInventoryMetrics. The clone left those pointer fields
aliased to the source struct, so the mock fixture refresh path
(refreshVMwareInventoryMetrics writing through metrics.UptimeSeconds
via ensureInt64Ptr) and the snapshot read path
(inventoryUptimeSeconds dereferencing metrics.UptimeSeconds) raced
on the same heap-allocated int64.
TestMonitorBuildBroadcastFrontendStateUsesCanonicalMockUnifiedResources
exemption:
The test asserts broadcast state does not publish the lowercase-
hyphenated legacy docker host label so canonical docker hosts
surface their human-readable DisplayName. Commit 89abed099
(2026-05-24) added the docker-swarm-node resource type whose Name
is the swarm node hostname (matching how Docker Swarm identifies
node members), which collided with the legacy-label rejection.
Refine the assertion to apply only to host-type resources
(docker-host, agent, node).
handleWorkloadCharts: add a 3s TTL response cache + per-key singleflight
so repeated sparkline polls from dashboards share work. TestSLO_WorkloadCharts
p95 drops from 188ms (over the 90ms target) to 838µs because most polls
hit cache or coalesce. Cold-call cost is unchanged; the win is dedup of
the steady-state polling pattern.
Cache + singleflight state live on the Router struct, not as globals, so
tenants don't cross-contaminate and tests don't pollute each other. Same
treatment for stateComputeGroup added in the previous commit.
monitor_metrics.mergeMetricHistory: drop the defensive cloneMetricPointMap
of base and cloneMetricSeries of candidate inside the merge. Every caller
already passes a freshly-owned map (from filterChartMetricMap,
GetAllGuestMetrics, GetAllStorageMetrics, queryStore*), so the internal
clones were pure waste. Caller-side cloneMetricPointMap is removed in
the three sites where the input is owned. ~6% allocation reduction on
BenchmarkHandleWorkloadCharts_StoreBacked.
Contract-neutral: no endpoint, response body, header, or wire-format
change.
Closes the only API-coverage gap from the Docker / Kubernetes IA
maturity review: Roles, ClusterRoles, RoleBindings, and
ClusterRoleBindings now flow from the Kubernetes agent through the
canonical resource registry into the Kubernetes platform-page
Configuration tab.
Agent: pkg/agents/kubernetes/report.go gains four new report struct
types that carry summary counts plus subject-kind sets; individual
subject names and full PolicyRule contents are deliberately omitted
so Pulse stays a "what permissions exist where" surface, not an RBAC
enumeration tool. internal/kubernetesagent/agent.go gains four
collectors that call rbacv1.RoleList/ClusterRoleList/etc. through the
existing runKubernetesCallWithRetry wrapper, matching the
ServiceAccount collector's RBAC-forbidden retry pattern.
Canonical: internal/models mirrors with NormalizeCollections coverage;
convert* funcs in internal/monitoring/kubernetes_agents.go translate
agent report -> model; ResourceTypeK8sRole / K8sClusterRole /
K8sRoleBinding / K8sClusterRoleBinding join the canonical type set;
registry ingest* + adapter resourceFrom* functions emit one Resource
per RBAC object with ruleCount / roleKind / roleName / subjectCount /
subjectKinds / aggregationLabels on the K8s meta; search mapping in
internal/api/resources.go and the privacy allow-list in
internal/api/org_handlers.go pick up the four new type tokens; the
K8s privacy category in unifiedresources/policy_metadata.go classifies
them like the rest of K8s.
Frontend: ResourceType union + ResourceKubernetesMeta carry the new
kinds and RBAC summary fields; KubernetesPageSurface query asks for
them; the page model buckets them into the Configuration group;
KubernetesConfigTable renders Role / ClusterRole rule counts and the
aggregated flag, plus RoleBinding / ClusterRoleBinding role refs and
"N subjects · Kind1, Kind2 +overflow" subject summaries.
Curated demo seeds per-namespace Roles + RoleBindings plus an
aggregated ClusterRole + ClusterRoleBinding for pulse-demo-monitoring
in each cluster so the Configuration tab renders 18 RBAC rows across
the three demo clusters.
Contracts updated for the canonical-shape guard: monitoring,
api-contracts, unified-resources, frontend-primitives,
organization-settings (canonical) plus agent-lifecycle and
storage-recovery (dependent via Extension Points). Verification
proofs extended: kubernetes_registry_test.go, kubernetes_agents_test.go,
agent_inventory_test.go (new TestCollectRBACInventoryReportsSummaryCountsOnly
that pins the subject-name-omission contract), demo_scenarios_test.go,
adapter_coverage_test.go, contract_test.go, org_handlers_test.go,
resourceIdentity.test.ts, reportingResourceTypes.test.ts,
KubernetesConfigTable.test.tsx, and the
subsystem_lookup_test.py line-anchor bumps that the contract edits
shifted (api-contracts 246 -> 253, organization-settings 92 -> 93).
Verified:
- go build ./internal/... ./cmd/... clean
- go test ./internal/unifiedresources/..., ./internal/mock/...,
./internal/kubernetesagent/..., ./internal/api/...,
the K8s subset of ./internal/monitoring/... all clean (three
pre-existing unrelated monitoring failures noted earlier remain
unchanged by this commit)
- npm run type-check, lint:eslint, lint:theme,
lint:canonical-platforms clean
- vitest: 70 K8s frontend tests pass including the new RBAC render
coverage in KubernetesConfigTable.test.tsx
- browser proof on /kubernetes/configuration: 36 config rows
including 18 RBAC rows across three clusters; ClusterRole
"pulse-demo-monitoring" shows "12 rules · Aggregated";
ClusterRoleBinding shows "3 subjects · Group, ServiceAccount +1"
Collect native Kubernetes config, policy, and autoscaling objects.
Project the new resource types through API filters, unified resources, mock fixtures, and Kubernetes tabs.
Keep Secret inventory metadata-only and route k8s-secret policy as restricted local-only.
VsphereHostsTable was missing the Version (ESXi build) and Uptime
columns that Proxmox / Docker / Kubernetes / TrueNAS host
equivalents carry. The data was already piped through the
canonical projection — ESXi version on resource.agent.osVersion
(e.g. "8.0.3"), host uptime on resource.uptime (lifted from
InventoryMetrics.UptimeSeconds via the sys.uptime.latest
PerformanceManager counter wired in 23ea4e487 this morning) —
the table just didn't render columns for them.
VsphereHostsTable now renders Version and Uptime cells, ordered
Version-adjacent-to-Host and Uptime before vCenter to mirror the
Proxmox Nodes layout. Column widths trimmed to fit. Uptime uses
the shared formatUptime util in condensed form (e.g. "163d")
with the full label as the cell title attribute. Detail row
colspan bumps 9 to 11 to keep the drawer spanning the full row.
Plumbing the canonical Resource.Uptime through to the frontend
required two fallback extensions surfaced by the column audit:
- useUnifiedResources.ts toResource: the uptime fallback chain
ended on platform-specific carve-outs (agent.uptimeSeconds,
proxmox.uptime, pbs/pmg/kubernetes); vSphere populates only
the canonical Resource.Uptime, so the chain has to land on
v2.uptime. Same shape as the workloads-hook fix in e5b31f484.
- internal/monitoring/monitor.go monitorUptime: the websocket
broadcast converter walked the same platform-specific chain
and silently dropped vSphere host/VM uptime, then merge-
clobbered the REST-loaded value once WS reconnect replayed
the broadcast. Add the canonical resource.Uptime fallback so
the broadcast payload carries uptime for VMware-backed rows
consistently with the REST contract. Carve-outs still take
precedence so existing platforms keep prior behavior.
Verified: vSphere overview Hosts table now renders esxi-01..07
with Version 8.0.3 and Uptime 147d-167d cells. No regression in
Proxmox / Docker / Kubernetes / TrueNAS uptime paths.
Contracts:
- monitoring.md documents the canonical Resource.Uptime fallback
in monitorUptime and the carve-out precedence.
- unified-resources.md adds the same canonical-uptime fallback
rule to the toResource consumer-side contract.
- storage-recovery.md amends rule 29 to call out that the
canonical Resource.Uptime fallback is descriptive host/VM
uptime only and must not be reinterpreted as backup recency
or recovery cadence.
Proofs:
- internal/monitoring/canonical_guardrails_test.go locks the
monitorUptime fallback contract: canonical Resource.Uptime is
surfaced when no carve-out is set, carve-outs take precedence,
nil when nothing populates.
- frontend-modern/src/hooks/__tests__/useUnifiedResources.test.ts
asserts the toResource v2.uptime fallback for a vSphere-shaped
payload with no platform-specific uptime carve-out.
Project vCenter network inventory through canonical resources and add the vSphere Networks table backed by vCenter network topology. Align resource presentation coalescing so state and resource APIs share the same host contract.
Add a global resource timeline endpoint for provider activity and wire vSphere Activity to VMware timeline changes. Seed mock VMware activity through the same supplemental-change path and keep the relevant resource contract tests current.
Add VM and share counts to the TrueNAS observed summary contract and systems table.
Keep TrueNAS overview inventory table-first with a platform guardrail.
Scope TrueNAS child source IDs under the system source key so duplicate pool, dataset, app, VM, share, and disk names from different appliances do not merge.
Use the resulting hierarchy for per-system TrueNAS overview counts and scoped mock metric targets.
monitorFrontendStatus's "pod" branch was re-deriving status from
Kubernetes.PodPhase alone, so a CrashLoopBackOff pod (Phase=Running
with unready containers) was reported as "running" even though
statusFromKubernetesPod had already classified it as StatusWarning.
Drop the unconditional phase=running -> "running" mapping and fall
through to the unified-status switch, which surfaces StatusWarning as
"degraded". Phase-based mappings for pending/unknown/succeeded/failed
are kept since they still cover edge cases where the unified status
is StatusOnline but phase carries useful information.
Adds two regression cases in TestMonitorFrontendAndMetricHelpers:
- pod warning running degraded (CrashLoopBackOff)
- pod online running running (healthy baseline)
rcourtman