Five new branch-coverage tests over the packages the telemetry lifecycle drop
touched, taking twenty-two functions to full coverage without touching any
source or existing test.
internal/telemetry: the deployment method precedence chain, the duration bucket
asserted one nanosecond either side of every boundary so the strict comparison
is pinned, every activation stage in its precedence order with the rank
ordering proved strictly increasing, the estate size bucket at each inclusive
boundary, the monitored resource count proved to draw on each counted field
individually, and the update failure classifier through every category arm
including the status short-circuit that wins over error text.
The lifecycle and install-id records are exercised as write-then-read round
trips under a temp directory and a frozen clock: a first-ever ping with nothing
on disk, a later ping against an existing record, an unreadable record, a
directory that cannot be written, and a reset proved to change the identifier
and persist the new one.
internal/monitoring: the alert outcome accumulation across the cutoff boundary
and each classification, and the install outcome accumulation over the arms
reachable without a live monitor.
internal/api: the read state snapshot mapping asserted field by field with
collection independence, both diagnostics collection normalizers, the in-memory
zip entry extraction across a missing entry, a non-zip payload and a zero-length
entry, and the trusted proxy check over IPv4, IPv6 and malformed input.
Contract-Neutral: test-only branch coverage, no contract surface touched
Five new branch-coverage tests taking previously unreached functions from zero
to covered, with no source or existing test touched.
cmd/pulse-control-plane: the MSP status failure lookup across exact, absent,
case-differing and substring inputs, and the status state ordering proved
deterministic across repeated runs over the same map so a random map iteration
order cannot pass by luck.
internal/ai/eval: the signal coverage assertion driven through both the lazy
quality evaluation path and the supplied-quality path, at the minimum rate
boundary and either side of it, asserting the formatted message rather than
only the pass or fail outcome, plus the approval write command builder.
internal/api: the setup script server name derivation including the fallback
arm, the patrol autonomy level validator, the approval risk assessment across
every level it can return, and the two typed error messages.
internal/hostagent: the token already exists classifier over each phrase it
recognises, a wrapped error carrying the phrase, and near-miss messages that
must not match.
internal/monitoring: the limited temperature buffer including its truncation
boundary and its aliasing behaviour, and the mock series generators asserted on
determinism under a fixed seed and on their value bounds.
Reported for follow-up, not fixed here: the node escalation arm in the approval
risk assessment is unreachable for the current command set, generatePlateauSeries
panics on a negative point count and leaves its tail unclamped, and the limited
temperature buffer returns a slice that aliases its internal storage.
Contract-Neutral: test-only branch coverage, no contract surface touched
Removed-agent records kept no platform identity, so the fleet doctor
emitted an empty platform for removed rows and the Agent Doctor
uninstall handoff could never tighten to a single platform command.
Host and Docker removal now capture the agent's last-known reported
platform onto the removed record, and the fleet diagnostic resolves the
retained value through the same strict normalization as live subjects,
so /api/agents/diagnostics reports a platform for removed agents and
the handoff shipped in 4fc86ec18 scopes to one command automatically.
A retained value that does not normalize to a known platform still
yields an empty diagnostic platform rather than a guess, and removed
Kubernetes clusters retain nothing because the cluster report never
carries a platform. The field is additive and optional, so removed
lists serialized before it existed load unchanged. Monitoring contract
updated; proofs staged in agent_fleet_doctor_test.go and
monitor_host_agents_test.go.
In-place updates never rewrite the systemd unit, so installs upgraded
past v6.1.0-rc.1 keep a unit without AmbientCapabilities=CAP_NET_RAW
and every ICMP probe fails with ping's raw stderr. Detect the
capability failure and point at the unit and its fix instead (#1554).
The connections aggregator, alerts and grouping helpers plus the monitoring
group-key, availability-key and read-state mapper helpers were all at zero
coverage. Each of these is a pure value-in to value-out function on a path
that decides how a connection is grouped, whether an alert snapshot is
emitted, and how host thermal state crosses the read-state boundary, so a
silent regression there is invisible until it reaches a user.
Adds branch coverage for buildPMGConnection, connectionTypeForAlerts,
snapshotConnectionsForAlerts, uniqueMatch, connectedInfrastructureGroupKey,
availabilityConnectionKey, ConnectionHealthKey, pingArgs,
hostThermalStateFromReadStateView, latestMetricPoint and MemorySourceTrust,
including nil and whitespace arms, candidate precedence order, and clone
independence in both directions for the mapper. Every named target moves
from zero to full statement coverage apart from the pingArgs arms that are
unreachable on this platform.
Test-only change.
PULSE_ALLOW_CONTRACT_NEUTRAL_COMMIT=test-only branch coverage, no contract surface touched
The connections ledger derives agent rows from the unified fabric, which
includes machines whose telemetry comes from platform integrations (vSphere
ESXi hosts, TrueNAS). Agent Doctor rendered every one as a permanent
'Unknown / no structured reason' row, while agents the ledger does not carry
(Docker-only, Kubernetes-only) were silently dropped from the fleet view.
- Expose HostView.IntegrationSource() (source-set based: only SourceAgent
ingest counts, since integration providers fabricate an Agent payload) and
plumb it through models.Host to the connections ledger as the optional
integrationSource field.
- Agent Doctor skips integration-backed connections and appends
diagnostics-only agents, honoring scope, so the doctor covers exactly the
real Pulse Agent fleet.
- Update readiness agent checks no longer count integration-backed machines
as registered agents.
- Humanize doctor copy: plain-language stale message with '10m 2s'-style
durations, offline wording without enum leakage, no 'Supported target:
Unknown' cell when no target is published, host-local command banner only
when a command is actually offered, and a compact non-zero summary strip.
Contracts updated for unified-resources, monitoring, api-contracts,
agent-lifecycle, and dependent storage-recovery; verification via
views_test.go, monitor_host_agents_test.go, state_host_test.go,
contract_test.go, and the frontend connections API test.
PVE's disks/list endpoint labels healthy SCSI/SAS drives OK while ATA
drives say PASSED, and failing ATA drives come back as FAILED! with the
bang. Pulse ingested those raw strings, so a healthy SAS drive rendered
as health Unknown even though the Proxmox UI showed S.M.A.R.T. OK. The
host agent and TrueNAS paths already normalize their health text; the
PVE ingestion was the only entry that did not.
Map OK/PASS to PASSED and FAIL-containing values to FAILED at ingestion,
and accept OK as healthy in the disk presentation layer as defense for
state produced by older servers.
Refs #1595
Contract-Neutral: behavioral fix: normalize PVE disk health strings at ingestion, no public contract delta (#1595)
smartctl reports SAS drives with device protocol SCSI, so detectDiskType
fell through to its blanket sata default, and that non-empty type also
masked the text-output transport evidence the fallback parser had
already extracted. The wrong sata label then blocked the merge layer
from promoting the smartctl serial over the SAS transport address
Proxmox reports.
Classify SCSI-protocol devices via the scsi_transport_protocol
descriptor, let the text and sysfs refinements upgrade a generic scsi
label, and apply the legacy sata default only after all evidence is
exhausted. Parse the SCSI log-page fields (power-on hours, grown defect
count, endurance used) that SCSI drives report instead of an ATA
attribute table, and let agent-reported sas replace coarse hdd/ssd/sata
types during the disk merge.
Refs #1595
Contract-Neutral: behavioral fix: SAS transport detection and SCSI attribute parsing in host agent SMART collection; no public contract delta (#1595)
Contract-Neutral: RC qualification fixes preserve existing public API, tenant, monitoring, and organization contracts while correcting canonical runtime ownership and test fixtures.
Wiping the agent state dir regenerates the agent ID and a fresh install
command mints a fresh token, so Docker host identity resolution correctly
refuses to adopt the old record (a foreign token must never take over a
live host) and creates a new one. Nothing ever reaped the old record, so
it sat in state forever with a stale agent version, stale containers and
stale image digests. That is the source of the outdated-agent banner that
contradicts the update dialog in #1586, the doubled container rows in
#1564, and plausibly the digest-mismatch preflight refusal there too.
Reap a matching record only when it stopped reporting before the
superseding token was minted, the same explicit-reenroll-intent rule the
removal block uses (#1581). A record that is still reporting keeps
advancing LastSeen and is never touched, and supersession sets no
resurrection block.
Refs #1586#1564
PULSE_ALLOW_CONTRACT_NEUTRAL_COMMIT: docker host state bug fix; live Codex agent WIP holds the contract docs
A PBS datastore reached the unified registry twice, once from the PBS
snapshot adapter under the canonical <instance-id>/<name> source ID and
once from the PBS poller's models.Storage conversion under the legacy
<instance-id>-<name> storage ID. The thresholds page rendered two cards
per datastore and their alert overrides split across both key formats,
with only the legacy key honoured at evaluation time.
Skip the redundant storage-conversion ingest when the canonical
datastore resource is already registered, carry the canonical ID as a
storage alias so evaluation accepts overrides in either format, and add
the legacy dash key to the frontend candidate list so pre-existing
overrides stay bound to the single surviving card and re-home onto the
canonical key on the next save.
Fixes#1591
Contract-Neutral: behavior fix with regression tests across registry/alerts/frontend; contract docs carry another agent's WIP on the shared tree, delta deferred
The PVE backup poll classified storage as PBS-backed by the pbs- name
prefix as well as the storage type. A dir storage merely named pbs-*
(content backup, is_mountpoint) lost all of its vzdump backups from the
Backups view whenever a direct PBS connection was also configured,
because the poll skipped its content as 'PBS is authoritative'. Classify
by storage type only.
Fixes#1592
Contract-Neutral: behavior fix with regression test; monitoring contract docs carry another agent's WIP on the shared tree, delta deferred
A pool built from /dev/disk/by-id/nvme-MODEL_SERIAL_1-part1 references
(systemd appends the _1 namespace token) made serialFromByID take the
last underscore token, so it extracted the namespace digit instead of
the serial and the disk fell back to the generic 'ZFS' usage string
while an identically-built node without the suffix resolved its pool
name. Reported with exact zpool output in issue #1540.
Contract-Neutral: bugfix restoring intended disk-to-pool annotation (#1540); no contract surface changes
wilddev65's rc.2 retest confirmed the connection-keying fix split the two
same-serial CORE boxes but left four symptoms. All four trace to Pulse
reading fields the TrueNAS API has never served, which the pre-fix
identity flapping had masked behind bigger breakage:
- Datasets all showed Offline: pool.dataset.query returns no "mounted"
field on any TrueNAS version (CORE 13 and SCALE both strip it from the
property allowlist), and the parser read absence as unmounted. A listed
dataset now defaults to mounted unless "locked" (encrypted, key
unloaded) or an explicit mounted=false says otherwise.
- Disks all showed Attention/Unknown: disk.query carries no status or
smart_status field either, so every disk landed on UNKNOWN health,
which the storage table bucketed as Attention. Per-disk ZFS state
(ONLINE/DEGRADED/FAULTED/...) and pool membership are now derived from
the pool.query topology that both platforms attach unconditionally --
disk.query's extra.pools cannot cross the REST bridge, whose query
params turn into filters that silently match nothing. Disks therefore
nest under their pools again, and a genuinely unreadable health renders
as neutral Unknown instead of amber Attention.
- Disk temperatures were blank: disk.temperatures takes parameters, so
REST v2.0 has only ever served it as POST with a body keyed by param
name; the client's GET failed on every version, and CORE (no JSON-RPC
endpoint) has no reporting fallback to rescue it. The ladder now tries
native JSON-RPC reporting first and falls back to the POST form, and
the provider surfaces API-reported disk temps as SMART sensor entries
so the host Thermals card lists disks for API-backed systems.
- The storage table ordered identically-named pools (a DR pair both
named "tank") by map iteration order, flipping between refreshes;
ordering now tie-breaks on system name then id.
Verified end-to-end against two stub TrueNAS CORE 13 REST servers
reporting the same placeholder DMI serial: storage tab nests both
systems' pools/datasets/disks correctly with temperatures and Healthy
states, and the system drawer Thermals card lists per-disk readings.
checkMockAlerts checked guests, nodes, storage, PBS, and PMG but never
looped over the mock estate's DockerHosts, so no docker container state,
health, metric, update, or swarm service alert could ever fire against
mock data. The generator deliberately keeps one container unhealthy per
estate, and the docker alert lifecycle (disable-all sweeps, update-alert
delay, config reconciliation) was unexercisable in the mock QA surface.
Verified live on a scratch mock instance: canonical docker metric alerts
fire, the Containers global-defaults toggle clears all of them via the
DisableAllDockerContainers sweep (the #1545 fix path), and re-enabling
re-fires them on the next cycle.
Contract-Neutral: mock-mode docker alert evaluation wiring fix; no public contract delta
Release-level regression pass over the alert lifecycle found two seams
with store/scheduler-level coverage but untested wiring:
- ingestRecoveryPointsBestEffort (#1580): a complete enumeration deletes
in-scope points the source stopped reporting, an empty enumeration
clears the scope, and a failed upsert skips reconciliation so a broken
batch cannot delete points the source still reports.
- availabilityPollProvider.FixedInstanceInterval (#1582): the configured
poll interval reaches the scheduler descriptor exactly (with the
10s..1h clamp), 0 for disabled or unknown targets, so the adaptive
scheduler cannot collapse the availability detection window again.
Cloned VMs that keep the same /etc/machine-id collapse into a single
Docker host in Pulse, with each clone's report silently overwriting the
other (#1584). The server now tracks per-identity hostname and machine
ID observations and flags a conflict when a value flaps back to one
already seen inside a 15-minute window, a signature a one-time rename
never produces. The conflict rides the DockerHost model through unified
resources, and the Docker page shows a warning naming the flapping
hostnames with the machine-id remedy. The warning self-clears once one
clone stops reporting for the window.
Adds branch-coverage unit tests for previously uncovered pure functions in
internal/monitoring and internal/ai/tools. New test files only, with no source
changes.
Covers existing-cluster IP-override and fingerprint lookup, storage-summary
capacity-trend building, nouveau GPU temperature parsing, resolved-resource
control identity, and available-agent-host formatting. 32 TestBranchCov
functions in 2 files, all vet and gofmt clean.
Adds table-driven branch-coverage unit tests for previously uncovered pure
functions across internal/alerts/config, internal/ai/modelresolution,
internal/monitoring, internal/ai, internal/ai/qualification and pkg/licensing.
New test files only, with no source changes.
Covers per-subsystem alert-default normalization, configured model and provider
resolution, fleet-doctor identity and cluster-endpoint helpers, docker-state and
infrastructure-key mapping, investigation and SMART issue helpers, JSON and
autonomy normalization, and licensing feature-tier resolution. 65 TestBranchCov
functions in 7 files, all vet and gofmt clean.
A full monitor reload tears the state down and rebuilds it empty, so
agent-reported host rows vanish until each agent's next report lands.
Config fetches in that window 404ed with a perfectly valid token, which
showed up as rare correlated agent_config_fetch failures across
unrelated agents (#1570).
When the live snapshot has no match, resolve the host from the
persisted continuity store with the same semantics as the live path. A
report-scoped token resolves only its bound host and manage-scoped
tokens resolve by host ID. Deliberately removed hosts stay 404 because
removal deletes their continuity entry.
Deleting a host writes a machine-id-keyed removal block that rejected
every future report with HTTP 400, and the error pointed at an Allow
reconnect control that is not wired into the UI, leaving the machine
permanently unable to enroll without changing its machine-id (#1581).
Three holes made the block effectively immortal:
- The 24h TTL sweep only iterated the in-memory removal maps, which
reset on every restart, so persisted blocks never expired. Sweep the
persisted entries by their own RemovedAt for host agents, Docker
hosts, and Kubernetes clusters.
- AllowHostAgentReenroll (and the Docker and Kubernetes equivalents)
bailed out when the ID was missing from the in-memory map, so even
the API escape hatch stopped clearing persisted blocks after a
restart. Check and clear the persisted store independently.
- A report presenting an API token created after the removal is
explicit re-add intent (the user generated a fresh install command),
so clear the block and accept it. A still-running old agent keeps
presenting its pre-removal token and stays blocked.
Also reword the rejection to describe the two working recovery paths
instead of the unwired Settings control.
The recovery store was upsert-only: backups and snapshots deleted at
the source lingered as recovery_points rows until the 90-day retention
prune. ListRollups kept returning a rollup with a frozen LastSuccessAt,
so the backup-age alert for a deleted guest re-raised every poll cycle
and acknowledging or clearing it could never stick (#1580).
Each backup poll already publishes a complete per-instance enumeration
(partial failures early-return or carry previous entries forward), so
attach a reconcile scope to that ingest batch. After the upsert, points
in the scope (provider + id class + instance) that were not part of the
enumeration are deleted, which lets the existing per-cycle alert sweep
resolve the alert. An empty enumeration is meaningful and clears the
scope, covering the delete-all-backups case from the report.
Also make the async ingest queue batches instead of overwriting the
single pending slot, which silently dropped a full poll cycle whenever
two sources coalesced behind an active batch.
An availability target's configured poll interval only seeded the
adaptive scheduler: BuildPlan derived every instance's cadence from the
global adaptive bounds, and a failing probe raised the staleness score
and error penalty, collapsing the probe interval toward the global
5-second minimum. With interval 120s and failure threshold 4 the alert
was promised after ~8 minutes of downtime but fired within the first
minute because the four consecutive failures accumulated at the
collapsed cadence (#1582).
Availability checks promise pollInterval x failureThreshold as the
detection window, so the cadence is a user contract, not a scheduling
hint. Add a FixedIntervalPollProvider extension that pins an instance
to its configured interval, implement it for availability targets, and
bypass adaptive selection wherever the next run is computed (plan
building, rescheduling, and the non-adaptive fallback paths).
A pool built from /dev/disk/by-id/nvme-eui.<hex> references (the
installer's device naming when identical NVMe models share a box)
produced no serial key in the disk-to-pool matcher, and the WWN lookup
never stripped the eui. prefix smartctl reports, so the disk fell back
to the generic Proxmox usage string and showed 'ZFS' instead of its
pool name while an identically-built node showed 'local-zfs' (issue
#1540).
URLs saved through the resource drawer historically landed in the docker
store under the runtime container key, which any stable record (including
an intentionally empty cleared one) outranks in the unified customUrl
projection, and which orphans on container recreation. On report ingest,
copy those records into the stable app-container guest key when it is
missing (cleared links stay cleared), healing saves stranded before the
drawer moved to the stable identity. Also read the runtime key before its
copy-if-missing container-name snapshot so the freshest write wins among
the docker-store fallbacks.
Refs #1556
Two TrueNAS systems that report the same hostname collapsed into one
flapping resource (#1573, #1575): systemSourceID keyed the system by the
snapshot-reported hostname, every child pool/dataset/app/VM/share/disk
was scoped under it, and the client minted the system's machine key from
the DMI serial with a hostname fallback, so serial-less systems sharing
a hostname (and DR clones sharing a serial) also fully merged in the
identity matcher.
The system source ID now scopes to the connection ID the poller passes
through NewLiveProviderForConnection; the hostname arm survives only for
fixture snapshots, which carry no connection. The ingest identity drops
the machine key entirely (DR clones share DMI serials, and vendor
placeholder serials collide across unrelated machines), the client no
longer falls back to the hostname for MachineID, and ingest skips
identity-pin completion for SourceTrueNAS so a stale pre-fix pin or a
same-named agent host's pin cannot lend the system a machine key and
re-merge what connection scoping keeps apart. Agent.AgentID and the
native metric history keys follow the source ID minus its system:
prefix, so BuildMetricsTarget keeps resolving one series.
Rows minted under the retired hostname-keyed derivation re-key once via
record-declared succession: records name their old canonical IDs in
IngestRecord.SupersededCanonicalIDs and IngestRecords applies the
existing ApplyCanonicalIDSuccessions semantics (operator state and
action audits re-key, the superseded pin drops, never while the old ID
still belongs to a live resource, journal rows are never rewritten).
Alert identities and persisted metric series under old child IDs are
not re-keyed: active alerts re-arm under the new IDs and TrueNAS host
charts are backed by native read-through history.
When fetchVMFSInfo fails (agent not running, timeout, permission
denied, no filesystems), the builder passed the cluster/resources
numbers through — and PVE always reports 0 used for QEMU — so the UI
rendered a confident "0% (0 B/<allocated>)" for every affected VM
instead of the unavailable dash plus diskStatusReason tooltip. The
agent-disabled path already used the -1 sentinel for exactly this;
apply it on the error path too. The stabilizer can still replace the
sentinel with the previous good sample when recent agent evidence
exists.
Reported by Massimo Simoni (support, 2026-07-13): screenshot showed
every VM at 0% disk with only containers reporting real usage.
An alert that resolved while its firing notification was still in the
grouping window or waiting in the persistent queue (alert delay pushes
activation close to resolution; quiet-hours replay defers delivery)
produced a recovery-only notification: CancelAlert dropped the queued
firing, but LastNotified had been set optimistically at dispatch, so
the resolved-notification gate believed the firing had been sent.
CancelAlert now reports whether it cancelled a firing notification that
had not been delivered (grouping window entries and pending queue rows;
mid-send rows are excluded because their delivery may still complete),
and handleAlertResolved suppresses the recovery in that case. This also
covers the quiet-hours replay bypass: a recovery only follows a deferred
firing if the replay was actually delivered.
Addresses #1553
Two root causes behind #1516's remaining reports:
- A node whose Proxmox disks/list query fails (PVE probes SMART per disk
inside that call, so dozens of disks can exceed the API window) now
falls back to the linked host agent's smartctl inventory instead of
leaving the Physical Disks view empty. Each node also gets its own
attempt window so one slow node no longer starves the rest of the
cluster, and a poll that runs out of budget saves partial results.
- The -n standby probe guard is dropped for positively confirmed
non-rotational devices in both the host agent and the node sensor
wrapper. The guard exists to avoid spinning up sleeping HDDs; an SSD
has nothing to spin up, and some SATA SSDs answer CHECK POWER MODE
with a bogus standby state that permanently hid their temperature,
attributes and history.
Refs #1516
rcourtman