Ceph Quincy and later (including Squid on PVE 9) dropped the monmap
mons array and the mgrmap active_name/standbys arrays from ceph status
output, replacing them with monmap.num_mons and mgrmap.num_standbys,
with quorum membership reported at the top level of the payload. Both
the host agent parser and the Proxmox API path only understood the
legacy arrays, so modern clusters showed 0 monitors and undercounted
managers.
- hostagent: read num_mons/num_standbys and top-level quorum data,
taking the largest available signal, and base the mon/mgr service
rows on the same counts
- pkg/proxmox: decode mgrmap num_standbys and top-level
quorum_names/quorum on CephStatus
- monitoring: fall back to the new fields when counting MON/MGR
daemons, and log Ceph 401/403 failures at warn level with a hint to
grant Sys.Audit on / instead of hiding them at debug
- models: prefer the larger non-zero MON/MGR counts when merging Ceph
cluster records from multiple sources
Fixes#1626, Refs discussion #1290
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A second pass targeting PARTIALLY covered functions rather than untouched
ones, so every case here is an arm the existing suites never reached.
Percentages are per-function coverage, measured before and after.
- pkg/securityutil: the SSRF guards, which is where the uncovered arms
actually matter. isCarrierGradeNATIPv4 and isLocalNetworkIP are pinned at
the first and last address of 100.64.0.0/10 and just outside both ends,
across loopback, link-local, every RFC1918 range and IPv6 unique-local.
joinURLPath, IsLocalNetworkHost, resolveOutboundIPAddrs and
cloneRestrictedTransport to 100 percent, resolvePermittedOutboundIPs to
96.8, with the transport clone asserted independent of its source.
- internal/models: eleven deep-copy helpers from as low as 25 percent to 100.
Every one asserts real independence, mutating each nested slice, map and
pointer field of the clone and checking the original is untouched, which is
the failure mode a deep-copy helper actually has.
- internal/alerts: metricClearThreshold 28.6 to 100, resourceTypeLabel and
alertspecsMetricTriggered 50 to 100, the four canonical spec-id and
tracking-key builders 66.7 to 100, inferCanonicalKindFromLegacyAlert to 100,
and the backup-snapshot and ack-identity predicates.
- internal/servicediscovery: the four fingerprint generators to 100, each
asserted for both stability and sensitivity; the three command builders and
ValidateResourceID on their exact output and each rejection reason.
- internal/storagehealth: zfsScanActive and firstNonEmpty.
cephClusterSourceRank is deliberately left at 75 percent: its default arm is
unreachable because normalizeCephClusterSource can only return the two cases
above it. That is recorded rather than faked.
No source file is modified. Adversarial review returned no rejects and flagged
nine re-hit subtests; all nine were removed and every target function
re-measured at an identical percentage, proving they carried nothing.
PULSE_ALLOW_CONTRACT_NEUTRAL_COMMIT=test-only branch coverage, no source or contract change
Seventeen files closing the zero-coverage functions the current source drop
left behind. Every named target was measured off 0.0 percent by a per-function
coverage delta, re-measured against current main.
- config: the five new durable Proxmox cluster-node identity helpers
(deterministic id, endpoint equality, alias lookup, id existence, lookup by
id) to 100 percent; VMware and agent-profile persistence round-trips under
t.TempDir including that AppendProfileChangeLog appends rather than
replaces; AI chat session save, load, delete, per-user scoping and age
cleanup, with explicit timestamps rather than time.Now-relative fuzz;
PVEInstance.DeepCopy asserted for nested independence.
- truenas: incidentFromPoolStatus over every pool health string,
RecordsFromSnapshot over nil, empty and populated snapshots, both
TransportStatus accessors, and the RPC handshake and auth typed errors
through errors.Is and errors.As.
- unifiedresources: the maintenance-window operator-state lifecycle on
MemoryStore including the not-found and already-cleared arms, plus the
four remaining View accessors asserted on their exact formatted output.
- api: restoreAgentExecMetadata, buildAlertConnectionSnapshotsWithRuntimeSources
and both mock series generators, asserted on shape, ordering and
determinism rather than non-emptiness.
- cmd/pulse-control-plane: the four remaining MSP and mobile proof report
printers, asserted on the concrete strings in captured stdout.
- ai: cost.EmptySummary, approval.emptyExecutionState, demo.IsDemoRuntimeIntended
and tools.findCanonicalAppContainerResourceByReferences across no-match,
first-match, later-match and ambiguous references.
- monitoring, models, alerts: trueNASAppRunning,
supplementalProviderOwnedSourcesForOrg, IOCounterPresence.Effective,
ValidAlertIntentSignal and intentTimePointer.
No source file is modified. Adversarial review returned no rejects across all
seventeen files and flagged four padding cases plus one dead table field; all
were removed and the per-function coverage re-measured as identical, proving
they carried nothing.
PULSE_ALLOW_CONTRACT_NEUTRAL_COMMIT=test-only branch coverage, no source or contract change
Two PVE clusters whose nodes share hostnames (pve01/pve02 on different
subnets) collapsed into one: the agent-link hostname fallback bound the
second cluster's node to the first cluster's agent despite contradicting
IP evidence, and the shared LinkedAgentID then overrode the
cluster-scoped merge key, so the fresher node overwrote the first
cluster's slot and relabeled it. The endpoint-host merge alias had the
same hole when TLS settings degrade node endpoints to bare node names.
Hostname matches now reject candidate agents whose linked nodes live in
a different named cluster or whose reported IPs exclude the node's
endpoint IP, and neither a shared agent identity nor an endpoint alias
can merge two nodes whose named clusters contradict. Empty cluster
names still merge freely so standalone views keep folding into their
cluster node.
Reported via support by an MSP running two clusters plus a standalone
that reuse pve01/pve02 across sites.
Removed-agent records kept no platform identity, so the fleet doctor
emitted an empty platform for removed rows and the Agent Doctor
uninstall handoff could never tighten to a single platform command.
Host and Docker removal now capture the agent's last-known reported
platform onto the removed record, and the fleet diagnostic resolves the
retained value through the same strict normalization as live subjects,
so /api/agents/diagnostics reports a platform for removed agents and
the handoff shipped in 4fc86ec18 scopes to one command automatically.
A retained value that does not normalize to a known platform still
yields an empty diagnostic platform rather than a guess, and removed
Kubernetes clusters retain nothing because the cluster report never
carries a platform. The field is additive and optional, so removed
lists serialized before it existed load unchanged. Monitoring contract
updated; proofs staged in agent_fleet_doctor_test.go and
monitor_host_agents_test.go.
The connections ledger derives agent rows from the unified fabric, which
includes machines whose telemetry comes from platform integrations (vSphere
ESXi hosts, TrueNAS). Agent Doctor rendered every one as a permanent
'Unknown / no structured reason' row, while agents the ledger does not carry
(Docker-only, Kubernetes-only) were silently dropped from the fleet view.
- Expose HostView.IntegrationSource() (source-set based: only SourceAgent
ingest counts, since integration providers fabricate an Agent payload) and
plumb it through models.Host to the connections ledger as the optional
integrationSource field.
- Agent Doctor skips integration-backed connections and appends
diagnostics-only agents, honoring scope, so the doctor covers exactly the
real Pulse Agent fleet.
- Update readiness agent checks no longer count integration-backed machines
as registered agents.
- Humanize doctor copy: plain-language stale message with '10m 2s'-style
durations, offline wording without enum leakage, no 'Supported target:
Unknown' cell when no target is published, host-local command banner only
when a command is actually offered, and a compact non-zero summary strip.
Contracts updated for unified-resources, monitoring, api-contracts,
agent-lifecycle, and dependent storage-recovery; verification via
views_test.go, monitor_host_agents_test.go, state_host_test.go,
contract_test.go, and the frontend connections API test.
New *_branchcov0718_test.go files extend the Go coverage wave to three more
areas of previously-uncovered pure logic. Covered areas include the domain-type
NormalizeCollections normalizers in models.go, the cloud control-plane pure
helpers for hosted-entitlement org-id normalization, magic-link URL redaction
and default MSP backup paths, and the actionlifecycle dispatch-attempt context
round-trip plus the typed-error Error and Unwrap methods.
Test-only, with no runtime or subsystem-contract change. Impure targets that
need a live Docker manager or tenant registry were skipped per the purity gate
and noted in the wave reports. Verified in a clean worktree at HEAD with go vet
and package tests green, gofmt clean, and every covered target moved from 0
percent toward full coverage.
New *_branchcov0718_test.go files raise coverage of previously-uncovered pure
functions across ten packages. Covered areas include securityutil SSRF and URL
validation, truenas path and telemetry parse helpers, storagehealth SMART and
physical-disk risk assessment, vmware inventory sort keys and error classifiers,
servicediscovery token filtering and readiness, telemetry evidence-from-history,
models ToFrontend converters and frontend NormalizeCollections normalizers,
actionplanner type predicates and canonical resource-id sort, config API-token
accessors, and licensing state accessors.
Test-only, with no runtime or subsystem-contract change. Verified in a clean
worktree at HEAD with go vet and package tests green, gofmt clean, and every
named target function moved from 0 percent to covered.
Two more newly added *_w0716_coverage_test.go files from the same
branch-coverage wave, touching no source and no existing test.
- internal/models deep-copy clone independence for Docker, Kubernetes,
ZFS and PBS clones, each mutating the original and asserting the clone
stays unchanged
- internal/ai/qualification lab shell-quoting and resource-name
rendering plus report replay predicates and markdown renderers
Cloned VMs that keep the same /etc/machine-id collapse into a single
Docker host in Pulse, with each clone's report silently overwriting the
other (#1584). The server now tracks per-identity hostname and machine
ID observations and flags a conflict when a value flaps back to one
already seen inside a 15-minute window, a signature a one-time rename
never produces. The conflict rides the DockerHost model through unified
resources, and the Docker page shows a warning naming the flapping
hostnames with the machine-id remedy. The warning self-clears once one
clone stops reporting for the window.
Adds table-driven branch-coverage unit tests for previously uncovered pure
functions across internal/models, internal/config, pkg/securityutil and
pkg/aicontracts. New test files only, with no source changes.
Covers model collection normalizers, reflect-based metric deep-copy,
removed-host agent bookkeeping, report-schedule and entitlement-billing
normalization, effective AI control and Patrol autonomy resolution,
websocket-origin host normalization, and command-safety classifiers. 61
TestBranchCov functions in 8 files, all vet and gofmt clean.
Adds table-driven branch-coverage unit tests for previously untested pure
helper functions across internal/alerts/config, internal/config,
internal/ai/safety, internal/ai/modelresolution, internal/operationreceipt,
internal/models, internal/securityutil and pkg/securityutil. New test files
only, with no source changes.
Covers alert-config normalization and validation, sensitive-path and
redaction classifiers, URL normalizers, provider model resolution,
operation-receipt decoding, credential masking, and account-to-org role
mapping. 57 TestBranchCov functions in 12 files, all vet and gofmt clean.
The alert engine fired powered-off alerts for every stopped VM and
container regardless of whether the guest was configured to autostart.
This meant intentionally-stopped guests (onboot=0) generated alarm
fatigue — in the live lab, 7 of 12 alerts were noise from stopped VMs
that are deliberately off.
Now the alert engine checks the Proxmox onboot setting before firing:
- onboot=true → stopped is unexpected, alert (preserved behavior)
- onboot=false → stopped is expected, suppress
- onboot=nil → unknown, alert (preserved behavior)
For containers, onboot is parsed from the already-fetched container
config in enrichContainerMetadata (no new API call). For VMs, the
config is fetched only for stopped VMs via GetVMConfig (one extra call
per stopped VM, running VMs are unaffected).
Models: add OnBoot *bool to VM and Container.
Alerts: add onboot-aware suppression branch in CheckGuest.
Monitoring: add parseProxmoxOnBoot helper and fetchVMOnBoot.
The alert engine stamps metadata.resourceType on every alert, but the
websocket state path converts alerts.Alert to models.Alert, which had no
Metadata field, so every active alert reached the frontend stripped. The
history Type badge then fell back to unified-store lookups that miss
nodes (alert.resourceId is the platform-native node ID while unified
resources mint canonical ids, and alert.resourceName is the raw node
name while unified resources prefer the display name), rendering
Unknown. In mock mode the generated history rows had the same gap.
models.Alert gained the Metadata field in f62f35e24 (it rode along with
the memory-cache commit); this completes the transport:
- copy Metadata in activeAlertsSnapshot (websocket active alerts),
GetRecentlyResolved (resolved alerts to state), and the mock
UpdateAlertSnapshots conversion; sources are deep clones already
- deep-copy Metadata in models cloneAlert to keep the snapshot
clone contract honest
- stamp resourceType in the mock history generator using the real
engine vocabulary (node, vm, system-container)
- recognize system-container in the history Type badge map; that is
what the v6 engine stamps for LXC guests
Verified live in mock mode: history previously resolved 277 of 780
rows to Unknown (all node alerts); now 718/718 rows and 19/19 active
alerts carry resourceType and zero badges render Unknown.
v5 modeled memory as used | cache | free (Memory.Cache, 'reclaimable
buff/cache') and the memory bar's tooltip carried a 'Shown in Proxmox'
row explaining why Pulse's percentage reads lower than the Proxmox UI's
cache-inclusive number — a recurring support question. The v6 rebuild
deleted the field from the backend model, so the split and the
reconciliation vanished product-wide. Flagged by the Proxmox overview
parity audit.
Backend: re-add Memory.Cache; split it out via a shared
splitReclaimableMemory helper at the node resolver (node status reports
truly-free directly) and the VM builder (when guest meminfo reported
free pages); transport as proxmox.memoryCache on unified resources
alongside swap/balloon; mock mode populates plausible cache for nodes
and VMs.
Frontend: cache prop on StackedMemoryBar with the v5 muted-amber
segment between active and balloon, tooltip rows for reclaimable cache,
truly-free Free (balloon-capped), and the 'Shown in Proxmox'
reconciliation; guest and node memory adapters normalize free to
truly-free at the boundary; guest and node drawers grow a Reclaimable
cache row; the Proxmox nodes table passes cache and node swap through.
Nodes that set up SSH temperature monitoring before v6.0.0-rc.6 have an
authorized_keys entry locked to `sensors -j`, so the pulse-sensors wrapper
never runs and SMART (SATA/SAS) disk temperatures silently never arrive;
only NVMe shows via kernel hwmon (what the #1471 reporters hit). The server
already detected the legacy payload but only logged it.
- models.Temperature gains LegacySensorsFormat, set by parseSensorsJSON
when the payload is raw sensors -j, and preserved across the host-agent
merge; it rides the existing temperatureDetails projection on unified
node resources.
- New PlatformOutdatedSensorSetupNotice on the Proxmox page, data-gated:
collection succeeded AND payload was legacy AND the node has SATA/SAS
disks without temperatures, so nodes whose disk temps arrive via a host
agent stay clean. Copy points at re-running the node setup script, which
replaces the locked key entry and installs the wrapper.
- monitoring.md contract extended with the legacy-format marker behavior.
resourceFromStorage and resourceFromDockerContainer stamped LastSeen with
time.Now() at conversion because their source models carried no poll
timestamp. The registry rebuilds from the retained state snapshot every
cycle, so those resources re-reported a fresh sighting each rebuild even
after their upstream source (PVE instance, docker host agent) stopped
delivering, and their per-source SourceStatus could never go stale via
markStaleLocked. 53faa4e46 fixed this fabrication at the ingest layer but
left these two adapter-level stamps.
- models.Storage gains LastSeen (omitzero), stamped where entries are
built: the PVE storage poll (poll start time, including synthesized
cluster-shared entries; preserved entries for unpolled nodes keep their
old stamp), the PBS datastore conversion (PBS instance sighting), Ceph
pool projection (cluster LastUpdated), and the mock generator (offline
mock nodes get a backdated stamp so the stale path renders).
- resourceFromStorage passes storage.LastSeen through; zero stays zero
("never seen") instead of becoming conversion time.
- resourceFromDockerContainer uses host.LastSeen: containers are delivered
wholesale with each host report, so the host report timestamp is the
container sighting. This matches every other docker sub-resource adapter
(services, tasks, volumes, networks, images already use host.LastSeen).
- ingestStorage routes PBS-poller datastore entries (instance "pbs-<name>",
type pbs) to SourcePBS, parented to the PBS instance. Keying them
SourceProxmox would judge their freshness against the 60s Proxmox stale
threshold while PBS polls every 60s by default, flapping healthy
datastores stale between polls; SourcePBS carries the cadence-matched
120s threshold. PVE-reported pbs-typed storage.cfg backends stay
SourceProxmox. Side effect: syncUnifiedStorageMetrics no longer skips
PBS datastore storage, so those entries gain usage history.
- storageFromReadStateView round-trips LastSeen so the legacy storage API
reports the honest sighting; mock refresh re-stamps available storage on
each simulated poll.
The parent host/node staleness was already honest, so platform pages
reflected outages at the parent level; this makes the per-resource source
freshness honest too.
Tighten v5-to-v6 upgrade safety, release installability, provider MSP mode handling, AI cost accounting, metrics flushing, and frontend guardrails for the v6.0.0 GA candidate.
Dead-code sweep. Functions flagged unreachable by golang.org/x/tools/cmd/deadcode
and confirmed unused across pulse, pulse-enterprise, pulse-pro and pulse-mobile by
adversarial cross-repo verification. Cross-module reachability was checked
explicitly (only pkg/ exported symbols are importable by other modules; internal/
packages and _test.go files are not). go build, go vet and test-compile all pass.
On a Proxmox node, physical disks collected by the host agent were keyed by
the NVMe controller (e.g. "nvme0 [nvme]") instead of the namespace, reported
sizeBytes 0 (or a stale filesystem-usage value), and flickered as the agent
reading intermittently replaced the authoritative Proxmox disks/list reading.
Root causes:
- smartctl --scan-open reports NVMe disks by their controller char device
(/dev/nvme0), and that scan label became the reported devPath.
- DiskSMART carried no capacity, so the server backfilled size by matching the
SMART device against host filesystem-usage entries, which never match a whole
partitioned/LVM/ZFS disk, leaving size 0.
- The unified-resource merge let the agent's controller label overwrite the
canonical Proxmox /dev/... devPath.
Fixes:
- The agent now reports the canonical block device (an NVMe controller resolves
to its namespace) and the authoritative capacity from /sys/block, with the
smartctl user_capacity / nvme_total_capacity as a cross-platform fallback.
Disks behind multiplexing controllers (megaraid, cciss, areca) keep their
disambiguating label and smartctl-reported size.
- SizeBytes flows through the agent report, host model, and adapter; the
filesystem-usage match is demoted to a legacy fallback.
- The merge keeps a canonical /dev/<device> devPath and never downgrades it to
a scan label, so an un-updated agent can no longer corrupt Proxmox data.
Refs #1483.
KubernetesCluster RBAC slices were not deep-cloned
cloneKubernetesCluster cloned Nodes, Namespaces, Pods, Deployments,
and 20+ other slices via dedicated helpers but left Roles,
ClusterRoles, RoleBindings, and ClusterRoleBindings aliased to the
source slice through the dest := src shallow copy. The final
dest.NormalizeCollections() call then iterates over those four
slices and writes c.Roles[i] = c.Roles[i].NormalizeCollections()
via index assignment, which races with any concurrent clone (or
read of the same source). The race detector caught it once the
k8s cluster count was bumped from 1 to 3 in 7938f28de, which made
the contention window wide enough to hit under -race. Fix by
deep-cloning the four RBAC slices with append([]T(nil), src...)
following the same pattern as the inline slice copies elsewhere
in cloneKubernetesCluster.
SECURITY.md sensor-wrapper alignment
The SMART/SSH feature shipped in 8769f07ee updated the shipped
public security doc at frontend-modern/public/docs/SECURITY.md to
document the new Pulse-owned /usr/local/sbin/pulse-sensors wrapper
forced-command shape for the legacy SSH temperature collection
flow, but the source SECURITY.md at the repo root still described
the prior command="sensors -j" forced command. The docsLinks
test (which compares the two for byte equality) flagged the drift.
Align root SECURITY.md and re-sync the shipped copy so both
describe the wrapper contract that the setup-script and runtime
collector now own.
courtmanr@gmail.com