Two PVE clusters whose nodes share hostnames (pve01/pve02 on different
subnets) collapsed into one: the agent-link hostname fallback bound the
second cluster's node to the first cluster's agent despite contradicting
IP evidence, and the shared LinkedAgentID then overrode the
cluster-scoped merge key, so the fresher node overwrote the first
cluster's slot and relabeled it. The endpoint-host merge alias had the
same hole when TLS settings degrade node endpoints to bare node names.
Hostname matches now reject candidate agents whose linked nodes live in
a different named cluster or whose reported IPs exclude the node's
endpoint IP, and neither a shared agent identity nor an endpoint alias
can merge two nodes whose named clusters contradict. Empty cluster
names still merge freely so standalone views keep folding into their
cluster node.
Reported via support by an MSP running two clusters plus a standalone
that reuse pve01/pve02 across sites.
Removed-agent records kept no platform identity, so the fleet doctor
emitted an empty platform for removed rows and the Agent Doctor
uninstall handoff could never tighten to a single platform command.
Host and Docker removal now capture the agent's last-known reported
platform onto the removed record, and the fleet diagnostic resolves the
retained value through the same strict normalization as live subjects,
so /api/agents/diagnostics reports a platform for removed agents and
the handoff shipped in 4fc86ec18 scopes to one command automatically.
A retained value that does not normalize to a known platform still
yields an empty diagnostic platform rather than a guess, and removed
Kubernetes clusters retain nothing because the cluster report never
carries a platform. The field is additive and optional, so removed
lists serialized before it existed load unchanged. Monitoring contract
updated; proofs staged in agent_fleet_doctor_test.go and
monitor_host_agents_test.go.
The connections ledger derives agent rows from the unified fabric, which
includes machines whose telemetry comes from platform integrations (vSphere
ESXi hosts, TrueNAS). Agent Doctor rendered every one as a permanent
'Unknown / no structured reason' row, while agents the ledger does not carry
(Docker-only, Kubernetes-only) were silently dropped from the fleet view.
- Expose HostView.IntegrationSource() (source-set based: only SourceAgent
ingest counts, since integration providers fabricate an Agent payload) and
plumb it through models.Host to the connections ledger as the optional
integrationSource field.
- Agent Doctor skips integration-backed connections and appends
diagnostics-only agents, honoring scope, so the doctor covers exactly the
real Pulse Agent fleet.
- Update readiness agent checks no longer count integration-backed machines
as registered agents.
- Humanize doctor copy: plain-language stale message with '10m 2s'-style
durations, offline wording without enum leakage, no 'Supported target:
Unknown' cell when no target is published, host-local command banner only
when a command is actually offered, and a compact non-zero summary strip.
Contracts updated for unified-resources, monitoring, api-contracts,
agent-lifecycle, and dependent storage-recovery; verification via
views_test.go, monitor_host_agents_test.go, state_host_test.go,
contract_test.go, and the frontend connections API test.
New *_branchcov0718_test.go files extend the Go coverage wave to three more
areas of previously-uncovered pure logic. Covered areas include the domain-type
NormalizeCollections normalizers in models.go, the cloud control-plane pure
helpers for hosted-entitlement org-id normalization, magic-link URL redaction
and default MSP backup paths, and the actionlifecycle dispatch-attempt context
round-trip plus the typed-error Error and Unwrap methods.
Test-only, with no runtime or subsystem-contract change. Impure targets that
need a live Docker manager or tenant registry were skipped per the purity gate
and noted in the wave reports. Verified in a clean worktree at HEAD with go vet
and package tests green, gofmt clean, and every covered target moved from 0
percent toward full coverage.
New *_branchcov0718_test.go files raise coverage of previously-uncovered pure
functions across ten packages. Covered areas include securityutil SSRF and URL
validation, truenas path and telemetry parse helpers, storagehealth SMART and
physical-disk risk assessment, vmware inventory sort keys and error classifiers,
servicediscovery token filtering and readiness, telemetry evidence-from-history,
models ToFrontend converters and frontend NormalizeCollections normalizers,
actionplanner type predicates and canonical resource-id sort, config API-token
accessors, and licensing state accessors.
Test-only, with no runtime or subsystem-contract change. Verified in a clean
worktree at HEAD with go vet and package tests green, gofmt clean, and every
named target function moved from 0 percent to covered.
Two more newly added *_w0716_coverage_test.go files from the same
branch-coverage wave, touching no source and no existing test.
- internal/models deep-copy clone independence for Docker, Kubernetes,
ZFS and PBS clones, each mutating the original and asserting the clone
stays unchanged
- internal/ai/qualification lab shell-quoting and resource-name
rendering plus report replay predicates and markdown renderers
Cloned VMs that keep the same /etc/machine-id collapse into a single
Docker host in Pulse, with each clone's report silently overwriting the
other (#1584). The server now tracks per-identity hostname and machine
ID observations and flags a conflict when a value flaps back to one
already seen inside a 15-minute window, a signature a one-time rename
never produces. The conflict rides the DockerHost model through unified
resources, and the Docker page shows a warning naming the flapping
hostnames with the machine-id remedy. The warning self-clears once one
clone stops reporting for the window.
Adds table-driven branch-coverage unit tests for previously uncovered pure
functions across internal/models, internal/config, pkg/securityutil and
pkg/aicontracts. New test files only, with no source changes.
Covers model collection normalizers, reflect-based metric deep-copy,
removed-host agent bookkeeping, report-schedule and entitlement-billing
normalization, effective AI control and Patrol autonomy resolution,
websocket-origin host normalization, and command-safety classifiers. 61
TestBranchCov functions in 8 files, all vet and gofmt clean.
Adds table-driven branch-coverage unit tests for previously untested pure
helper functions across internal/alerts/config, internal/config,
internal/ai/safety, internal/ai/modelresolution, internal/operationreceipt,
internal/models, internal/securityutil and pkg/securityutil. New test files
only, with no source changes.
Covers alert-config normalization and validation, sensitive-path and
redaction classifiers, URL normalizers, provider model resolution,
operation-receipt decoding, credential masking, and account-to-org role
mapping. 57 TestBranchCov functions in 12 files, all vet and gofmt clean.
The alert engine fired powered-off alerts for every stopped VM and
container regardless of whether the guest was configured to autostart.
This meant intentionally-stopped guests (onboot=0) generated alarm
fatigue — in the live lab, 7 of 12 alerts were noise from stopped VMs
that are deliberately off.
Now the alert engine checks the Proxmox onboot setting before firing:
- onboot=true → stopped is unexpected, alert (preserved behavior)
- onboot=false → stopped is expected, suppress
- onboot=nil → unknown, alert (preserved behavior)
For containers, onboot is parsed from the already-fetched container
config in enrichContainerMetadata (no new API call). For VMs, the
config is fetched only for stopped VMs via GetVMConfig (one extra call
per stopped VM, running VMs are unaffected).
Models: add OnBoot *bool to VM and Container.
Alerts: add onboot-aware suppression branch in CheckGuest.
Monitoring: add parseProxmoxOnBoot helper and fetchVMOnBoot.
The alert engine stamps metadata.resourceType on every alert, but the
websocket state path converts alerts.Alert to models.Alert, which had no
Metadata field, so every active alert reached the frontend stripped. The
history Type badge then fell back to unified-store lookups that miss
nodes (alert.resourceId is the platform-native node ID while unified
resources mint canonical ids, and alert.resourceName is the raw node
name while unified resources prefer the display name), rendering
Unknown. In mock mode the generated history rows had the same gap.
models.Alert gained the Metadata field in f62f35e24 (it rode along with
the memory-cache commit); this completes the transport:
- copy Metadata in activeAlertsSnapshot (websocket active alerts),
GetRecentlyResolved (resolved alerts to state), and the mock
UpdateAlertSnapshots conversion; sources are deep clones already
- deep-copy Metadata in models cloneAlert to keep the snapshot
clone contract honest
- stamp resourceType in the mock history generator using the real
engine vocabulary (node, vm, system-container)
- recognize system-container in the history Type badge map; that is
what the v6 engine stamps for LXC guests
Verified live in mock mode: history previously resolved 277 of 780
rows to Unknown (all node alerts); now 718/718 rows and 19/19 active
alerts carry resourceType and zero badges render Unknown.
v5 modeled memory as used | cache | free (Memory.Cache, 'reclaimable
buff/cache') and the memory bar's tooltip carried a 'Shown in Proxmox'
row explaining why Pulse's percentage reads lower than the Proxmox UI's
cache-inclusive number — a recurring support question. The v6 rebuild
deleted the field from the backend model, so the split and the
reconciliation vanished product-wide. Flagged by the Proxmox overview
parity audit.
Backend: re-add Memory.Cache; split it out via a shared
splitReclaimableMemory helper at the node resolver (node status reports
truly-free directly) and the VM builder (when guest meminfo reported
free pages); transport as proxmox.memoryCache on unified resources
alongside swap/balloon; mock mode populates plausible cache for nodes
and VMs.
Frontend: cache prop on StackedMemoryBar with the v5 muted-amber
segment between active and balloon, tooltip rows for reclaimable cache,
truly-free Free (balloon-capped), and the 'Shown in Proxmox'
reconciliation; guest and node memory adapters normalize free to
truly-free at the boundary; guest and node drawers grow a Reclaimable
cache row; the Proxmox nodes table passes cache and node swap through.
Nodes that set up SSH temperature monitoring before v6.0.0-rc.6 have an
authorized_keys entry locked to `sensors -j`, so the pulse-sensors wrapper
never runs and SMART (SATA/SAS) disk temperatures silently never arrive;
only NVMe shows via kernel hwmon (what the #1471 reporters hit). The server
already detected the legacy payload but only logged it.
- models.Temperature gains LegacySensorsFormat, set by parseSensorsJSON
when the payload is raw sensors -j, and preserved across the host-agent
merge; it rides the existing temperatureDetails projection on unified
node resources.
- New PlatformOutdatedSensorSetupNotice on the Proxmox page, data-gated:
collection succeeded AND payload was legacy AND the node has SATA/SAS
disks without temperatures, so nodes whose disk temps arrive via a host
agent stay clean. Copy points at re-running the node setup script, which
replaces the locked key entry and installs the wrapper.
- monitoring.md contract extended with the legacy-format marker behavior.
resourceFromStorage and resourceFromDockerContainer stamped LastSeen with
time.Now() at conversion because their source models carried no poll
timestamp. The registry rebuilds from the retained state snapshot every
cycle, so those resources re-reported a fresh sighting each rebuild even
after their upstream source (PVE instance, docker host agent) stopped
delivering, and their per-source SourceStatus could never go stale via
markStaleLocked. 53faa4e46 fixed this fabrication at the ingest layer but
left these two adapter-level stamps.
- models.Storage gains LastSeen (omitzero), stamped where entries are
built: the PVE storage poll (poll start time, including synthesized
cluster-shared entries; preserved entries for unpolled nodes keep their
old stamp), the PBS datastore conversion (PBS instance sighting), Ceph
pool projection (cluster LastUpdated), and the mock generator (offline
mock nodes get a backdated stamp so the stale path renders).
- resourceFromStorage passes storage.LastSeen through; zero stays zero
("never seen") instead of becoming conversion time.
- resourceFromDockerContainer uses host.LastSeen: containers are delivered
wholesale with each host report, so the host report timestamp is the
container sighting. This matches every other docker sub-resource adapter
(services, tasks, volumes, networks, images already use host.LastSeen).
- ingestStorage routes PBS-poller datastore entries (instance "pbs-<name>",
type pbs) to SourcePBS, parented to the PBS instance. Keying them
SourceProxmox would judge their freshness against the 60s Proxmox stale
threshold while PBS polls every 60s by default, flapping healthy
datastores stale between polls; SourcePBS carries the cadence-matched
120s threshold. PVE-reported pbs-typed storage.cfg backends stay
SourceProxmox. Side effect: syncUnifiedStorageMetrics no longer skips
PBS datastore storage, so those entries gain usage history.
- storageFromReadStateView round-trips LastSeen so the legacy storage API
reports the honest sighting; mock refresh re-stamps available storage on
each simulated poll.
The parent host/node staleness was already honest, so platform pages
reflected outages at the parent level; this makes the per-resource source
freshness honest too.
Tighten v5-to-v6 upgrade safety, release installability, provider MSP mode handling, AI cost accounting, metrics flushing, and frontend guardrails for the v6.0.0 GA candidate.
Dead-code sweep. Functions flagged unreachable by golang.org/x/tools/cmd/deadcode
and confirmed unused across pulse, pulse-enterprise, pulse-pro and pulse-mobile by
adversarial cross-repo verification. Cross-module reachability was checked
explicitly (only pkg/ exported symbols are importable by other modules; internal/
packages and _test.go files are not). go build, go vet and test-compile all pass.
On a Proxmox node, physical disks collected by the host agent were keyed by
the NVMe controller (e.g. "nvme0 [nvme]") instead of the namespace, reported
sizeBytes 0 (or a stale filesystem-usage value), and flickered as the agent
reading intermittently replaced the authoritative Proxmox disks/list reading.
Root causes:
- smartctl --scan-open reports NVMe disks by their controller char device
(/dev/nvme0), and that scan label became the reported devPath.
- DiskSMART carried no capacity, so the server backfilled size by matching the
SMART device against host filesystem-usage entries, which never match a whole
partitioned/LVM/ZFS disk, leaving size 0.
- The unified-resource merge let the agent's controller label overwrite the
canonical Proxmox /dev/... devPath.
Fixes:
- The agent now reports the canonical block device (an NVMe controller resolves
to its namespace) and the authoritative capacity from /sys/block, with the
smartctl user_capacity / nvme_total_capacity as a cross-platform fallback.
Disks behind multiplexing controllers (megaraid, cciss, areca) keep their
disambiguating label and smartctl-reported size.
- SizeBytes flows through the agent report, host model, and adapter; the
filesystem-usage match is demoted to a legacy fallback.
- The merge keeps a canonical /dev/<device> devPath and never downgrades it to
a scan label, so an un-updated agent can no longer corrupt Proxmox data.
Refs #1483.
KubernetesCluster RBAC slices were not deep-cloned
cloneKubernetesCluster cloned Nodes, Namespaces, Pods, Deployments,
and 20+ other slices via dedicated helpers but left Roles,
ClusterRoles, RoleBindings, and ClusterRoleBindings aliased to the
source slice through the dest := src shallow copy. The final
dest.NormalizeCollections() call then iterates over those four
slices and writes c.Roles[i] = c.Roles[i].NormalizeCollections()
via index assignment, which races with any concurrent clone (or
read of the same source). The race detector caught it once the
k8s cluster count was bumped from 1 to 3 in 7938f28de, which made
the contention window wide enough to hit under -race. Fix by
deep-cloning the four RBAC slices with append([]T(nil), src...)
following the same pattern as the inline slice copies elsewhere
in cloneKubernetesCluster.
SECURITY.md sensor-wrapper alignment
The SMART/SSH feature shipped in 8769f07ee updated the shipped
public security doc at frontend-modern/public/docs/SECURITY.md to
document the new Pulse-owned /usr/local/sbin/pulse-sensors wrapper
forced-command shape for the legacy SSH temperature collection
flow, but the source SECURITY.md at the repo root still described
the prior command="sensors -j" forced command. The docsLinks
test (which compares the two for byte equality) flagged the drift.
Align root SECURITY.md and re-sync the shipped copy so both
describe the wrapper contract that the setup-script and runtime
collector now own.
Closes the only API-coverage gap from the Docker / Kubernetes IA
maturity review: Roles, ClusterRoles, RoleBindings, and
ClusterRoleBindings now flow from the Kubernetes agent through the
canonical resource registry into the Kubernetes platform-page
Configuration tab.
Agent: pkg/agents/kubernetes/report.go gains four new report struct
types that carry summary counts plus subject-kind sets; individual
subject names and full PolicyRule contents are deliberately omitted
so Pulse stays a "what permissions exist where" surface, not an RBAC
enumeration tool. internal/kubernetesagent/agent.go gains four
collectors that call rbacv1.RoleList/ClusterRoleList/etc. through the
existing runKubernetesCallWithRetry wrapper, matching the
ServiceAccount collector's RBAC-forbidden retry pattern.
Canonical: internal/models mirrors with NormalizeCollections coverage;
convert* funcs in internal/monitoring/kubernetes_agents.go translate
agent report -> model; ResourceTypeK8sRole / K8sClusterRole /
K8sRoleBinding / K8sClusterRoleBinding join the canonical type set;
registry ingest* + adapter resourceFrom* functions emit one Resource
per RBAC object with ruleCount / roleKind / roleName / subjectCount /
subjectKinds / aggregationLabels on the K8s meta; search mapping in
internal/api/resources.go and the privacy allow-list in
internal/api/org_handlers.go pick up the four new type tokens; the
K8s privacy category in unifiedresources/policy_metadata.go classifies
them like the rest of K8s.
Frontend: ResourceType union + ResourceKubernetesMeta carry the new
kinds and RBAC summary fields; KubernetesPageSurface query asks for
them; the page model buckets them into the Configuration group;
KubernetesConfigTable renders Role / ClusterRole rule counts and the
aggregated flag, plus RoleBinding / ClusterRoleBinding role refs and
"N subjects · Kind1, Kind2 +overflow" subject summaries.
Curated demo seeds per-namespace Roles + RoleBindings plus an
aggregated ClusterRole + ClusterRoleBinding for pulse-demo-monitoring
in each cluster so the Configuration tab renders 18 RBAC rows across
the three demo clusters.
Contracts updated for the canonical-shape guard: monitoring,
api-contracts, unified-resources, frontend-primitives,
organization-settings (canonical) plus agent-lifecycle and
storage-recovery (dependent via Extension Points). Verification
proofs extended: kubernetes_registry_test.go, kubernetes_agents_test.go,
agent_inventory_test.go (new TestCollectRBACInventoryReportsSummaryCountsOnly
that pins the subject-name-omission contract), demo_scenarios_test.go,
adapter_coverage_test.go, contract_test.go, org_handlers_test.go,
resourceIdentity.test.ts, reportingResourceTypes.test.ts,
KubernetesConfigTable.test.tsx, and the
subsystem_lookup_test.py line-anchor bumps that the contract edits
shifted (api-contracts 246 -> 253, organization-settings 92 -> 93).
Verified:
- go build ./internal/... ./cmd/... clean
- go test ./internal/unifiedresources/..., ./internal/mock/...,
./internal/kubernetesagent/..., ./internal/api/...,
the K8s subset of ./internal/monitoring/... all clean (three
pre-existing unrelated monitoring failures noted earlier remain
unchanged by this commit)
- npm run type-check, lint:eslint, lint:theme,
lint:canonical-platforms clean
- vitest: 70 K8s frontend tests pass including the new RBAC render
coverage in KubernetesConfigTable.test.tsx
- browser proof on /kubernetes/configuration: 36 config rows
including 18 RBAC rows across three clusters; ClusterRole
"pulse-demo-monitoring" shows "12 rules · Aggregated";
ClusterRoleBinding shows "3 subjects · Group, ServiceAccount +1"
Collect native Kubernetes config, policy, and autoscaling objects.
Project the new resource types through API filters, unified resources, mock fixtures, and Kubernetes tabs.
Keep Secret inventory metadata-only and route k8s-secret policy as restricted local-only.
rcourtman