Contract-Neutral: RC qualification fixes preserve existing public API, tenant, monitoring, and organization contracts while correcting canonical runtime ownership and test fixtures.
Wiping the agent state dir regenerates the agent ID and a fresh install
command mints a fresh token, so Docker host identity resolution correctly
refuses to adopt the old record (a foreign token must never take over a
live host) and creates a new one. Nothing ever reaped the old record, so
it sat in state forever with a stale agent version, stale containers and
stale image digests. That is the source of the outdated-agent banner that
contradicts the update dialog in #1586, the doubled container rows in
#1564, and plausibly the digest-mismatch preflight refusal there too.
Reap a matching record only when it stopped reporting before the
superseding token was minted, the same explicit-reenroll-intent rule the
removal block uses (#1581). A record that is still reporting keeps
advancing LastSeen and is never touched, and supersession sets no
resurrection block.
Refs #1586#1564
PULSE_ALLOW_CONTRACT_NEUTRAL_COMMIT: docker host state bug fix; live Codex agent WIP holds the contract docs
A PBS datastore reached the unified registry twice, once from the PBS
snapshot adapter under the canonical <instance-id>/<name> source ID and
once from the PBS poller's models.Storage conversion under the legacy
<instance-id>-<name> storage ID. The thresholds page rendered two cards
per datastore and their alert overrides split across both key formats,
with only the legacy key honoured at evaluation time.
Skip the redundant storage-conversion ingest when the canonical
datastore resource is already registered, carry the canonical ID as a
storage alias so evaluation accepts overrides in either format, and add
the legacy dash key to the frontend candidate list so pre-existing
overrides stay bound to the single surviving card and re-home onto the
canonical key on the next save.
Fixes#1591
Contract-Neutral: behavior fix with regression tests across registry/alerts/frontend; contract docs carry another agent's WIP on the shared tree, delta deferred
The PVE backup poll classified storage as PBS-backed by the pbs- name
prefix as well as the storage type. A dir storage merely named pbs-*
(content backup, is_mountpoint) lost all of its vzdump backups from the
Backups view whenever a direct PBS connection was also configured,
because the poll skipped its content as 'PBS is authoritative'. Classify
by storage type only.
Fixes#1592
Contract-Neutral: behavior fix with regression test; monitoring contract docs carry another agent's WIP on the shared tree, delta deferred
A pool built from /dev/disk/by-id/nvme-MODEL_SERIAL_1-part1 references
(systemd appends the _1 namespace token) made serialFromByID take the
last underscore token, so it extracted the namespace digit instead of
the serial and the disk fell back to the generic 'ZFS' usage string
while an identically-built node without the suffix resolved its pool
name. Reported with exact zpool output in issue #1540.
Contract-Neutral: bugfix restoring intended disk-to-pool annotation (#1540); no contract surface changes
wilddev65's rc.2 retest confirmed the connection-keying fix split the two
same-serial CORE boxes but left four symptoms. All four trace to Pulse
reading fields the TrueNAS API has never served, which the pre-fix
identity flapping had masked behind bigger breakage:
- Datasets all showed Offline: pool.dataset.query returns no "mounted"
field on any TrueNAS version (CORE 13 and SCALE both strip it from the
property allowlist), and the parser read absence as unmounted. A listed
dataset now defaults to mounted unless "locked" (encrypted, key
unloaded) or an explicit mounted=false says otherwise.
- Disks all showed Attention/Unknown: disk.query carries no status or
smart_status field either, so every disk landed on UNKNOWN health,
which the storage table bucketed as Attention. Per-disk ZFS state
(ONLINE/DEGRADED/FAULTED/...) and pool membership are now derived from
the pool.query topology that both platforms attach unconditionally --
disk.query's extra.pools cannot cross the REST bridge, whose query
params turn into filters that silently match nothing. Disks therefore
nest under their pools again, and a genuinely unreadable health renders
as neutral Unknown instead of amber Attention.
- Disk temperatures were blank: disk.temperatures takes parameters, so
REST v2.0 has only ever served it as POST with a body keyed by param
name; the client's GET failed on every version, and CORE (no JSON-RPC
endpoint) has no reporting fallback to rescue it. The ladder now tries
native JSON-RPC reporting first and falls back to the POST form, and
the provider surfaces API-reported disk temps as SMART sensor entries
so the host Thermals card lists disks for API-backed systems.
- The storage table ordered identically-named pools (a DR pair both
named "tank") by map iteration order, flipping between refreshes;
ordering now tie-breaks on system name then id.
Verified end-to-end against two stub TrueNAS CORE 13 REST servers
reporting the same placeholder DMI serial: storage tab nests both
systems' pools/datasets/disks correctly with temperatures and Healthy
states, and the system drawer Thermals card lists per-disk readings.
checkMockAlerts checked guests, nodes, storage, PBS, and PMG but never
looped over the mock estate's DockerHosts, so no docker container state,
health, metric, update, or swarm service alert could ever fire against
mock data. The generator deliberately keeps one container unhealthy per
estate, and the docker alert lifecycle (disable-all sweeps, update-alert
delay, config reconciliation) was unexercisable in the mock QA surface.
Verified live on a scratch mock instance: canonical docker metric alerts
fire, the Containers global-defaults toggle clears all of them via the
DisableAllDockerContainers sweep (the #1545 fix path), and re-enabling
re-fires them on the next cycle.
Contract-Neutral: mock-mode docker alert evaluation wiring fix; no public contract delta
Release-level regression pass over the alert lifecycle found two seams
with store/scheduler-level coverage but untested wiring:
- ingestRecoveryPointsBestEffort (#1580): a complete enumeration deletes
in-scope points the source stopped reporting, an empty enumeration
clears the scope, and a failed upsert skips reconciliation so a broken
batch cannot delete points the source still reports.
- availabilityPollProvider.FixedInstanceInterval (#1582): the configured
poll interval reaches the scheduler descriptor exactly (with the
10s..1h clamp), 0 for disabled or unknown targets, so the adaptive
scheduler cannot collapse the availability detection window again.
Cloned VMs that keep the same /etc/machine-id collapse into a single
Docker host in Pulse, with each clone's report silently overwriting the
other (#1584). The server now tracks per-identity hostname and machine
ID observations and flags a conflict when a value flaps back to one
already seen inside a 15-minute window, a signature a one-time rename
never produces. The conflict rides the DockerHost model through unified
resources, and the Docker page shows a warning naming the flapping
hostnames with the machine-id remedy. The warning self-clears once one
clone stops reporting for the window.
Adds branch-coverage unit tests for previously uncovered pure functions in
internal/monitoring and internal/ai/tools. New test files only, with no source
changes.
Covers existing-cluster IP-override and fingerprint lookup, storage-summary
capacity-trend building, nouveau GPU temperature parsing, resolved-resource
control identity, and available-agent-host formatting. 32 TestBranchCov
functions in 2 files, all vet and gofmt clean.
Adds table-driven branch-coverage unit tests for previously uncovered pure
functions across internal/alerts/config, internal/ai/modelresolution,
internal/monitoring, internal/ai, internal/ai/qualification and pkg/licensing.
New test files only, with no source changes.
Covers per-subsystem alert-default normalization, configured model and provider
resolution, fleet-doctor identity and cluster-endpoint helpers, docker-state and
infrastructure-key mapping, investigation and SMART issue helpers, JSON and
autonomy normalization, and licensing feature-tier resolution. 65 TestBranchCov
functions in 7 files, all vet and gofmt clean.
A full monitor reload tears the state down and rebuilds it empty, so
agent-reported host rows vanish until each agent's next report lands.
Config fetches in that window 404ed with a perfectly valid token, which
showed up as rare correlated agent_config_fetch failures across
unrelated agents (#1570).
When the live snapshot has no match, resolve the host from the
persisted continuity store with the same semantics as the live path. A
report-scoped token resolves only its bound host and manage-scoped
tokens resolve by host ID. Deliberately removed hosts stay 404 because
removal deletes their continuity entry.
Deleting a host writes a machine-id-keyed removal block that rejected
every future report with HTTP 400, and the error pointed at an Allow
reconnect control that is not wired into the UI, leaving the machine
permanently unable to enroll without changing its machine-id (#1581).
Three holes made the block effectively immortal:
- The 24h TTL sweep only iterated the in-memory removal maps, which
reset on every restart, so persisted blocks never expired. Sweep the
persisted entries by their own RemovedAt for host agents, Docker
hosts, and Kubernetes clusters.
- AllowHostAgentReenroll (and the Docker and Kubernetes equivalents)
bailed out when the ID was missing from the in-memory map, so even
the API escape hatch stopped clearing persisted blocks after a
restart. Check and clear the persisted store independently.
- A report presenting an API token created after the removal is
explicit re-add intent (the user generated a fresh install command),
so clear the block and accept it. A still-running old agent keeps
presenting its pre-removal token and stays blocked.
Also reword the rejection to describe the two working recovery paths
instead of the unwired Settings control.
The recovery store was upsert-only: backups and snapshots deleted at
the source lingered as recovery_points rows until the 90-day retention
prune. ListRollups kept returning a rollup with a frozen LastSuccessAt,
so the backup-age alert for a deleted guest re-raised every poll cycle
and acknowledging or clearing it could never stick (#1580).
Each backup poll already publishes a complete per-instance enumeration
(partial failures early-return or carry previous entries forward), so
attach a reconcile scope to that ingest batch. After the upsert, points
in the scope (provider + id class + instance) that were not part of the
enumeration are deleted, which lets the existing per-cycle alert sweep
resolve the alert. An empty enumeration is meaningful and clears the
scope, covering the delete-all-backups case from the report.
Also make the async ingest queue batches instead of overwriting the
single pending slot, which silently dropped a full poll cycle whenever
two sources coalesced behind an active batch.
An availability target's configured poll interval only seeded the
adaptive scheduler: BuildPlan derived every instance's cadence from the
global adaptive bounds, and a failing probe raised the staleness score
and error penalty, collapsing the probe interval toward the global
5-second minimum. With interval 120s and failure threshold 4 the alert
was promised after ~8 minutes of downtime but fired within the first
minute because the four consecutive failures accumulated at the
collapsed cadence (#1582).
Availability checks promise pollInterval x failureThreshold as the
detection window, so the cadence is a user contract, not a scheduling
hint. Add a FixedIntervalPollProvider extension that pins an instance
to its configured interval, implement it for availability targets, and
bypass adaptive selection wherever the next run is computed (plan
building, rescheduling, and the non-adaptive fallback paths).
A pool built from /dev/disk/by-id/nvme-eui.<hex> references (the
installer's device naming when identical NVMe models share a box)
produced no serial key in the disk-to-pool matcher, and the WWN lookup
never stripped the eui. prefix smartctl reports, so the disk fell back
to the generic Proxmox usage string and showed 'ZFS' instead of its
pool name while an identically-built node showed 'local-zfs' (issue
#1540).
URLs saved through the resource drawer historically landed in the docker
store under the runtime container key, which any stable record (including
an intentionally empty cleared one) outranks in the unified customUrl
projection, and which orphans on container recreation. On report ingest,
copy those records into the stable app-container guest key when it is
missing (cleared links stay cleared), healing saves stranded before the
drawer moved to the stable identity. Also read the runtime key before its
copy-if-missing container-name snapshot so the freshest write wins among
the docker-store fallbacks.
Refs #1556
Two TrueNAS systems that report the same hostname collapsed into one
flapping resource (#1573, #1575): systemSourceID keyed the system by the
snapshot-reported hostname, every child pool/dataset/app/VM/share/disk
was scoped under it, and the client minted the system's machine key from
the DMI serial with a hostname fallback, so serial-less systems sharing
a hostname (and DR clones sharing a serial) also fully merged in the
identity matcher.
The system source ID now scopes to the connection ID the poller passes
through NewLiveProviderForConnection; the hostname arm survives only for
fixture snapshots, which carry no connection. The ingest identity drops
the machine key entirely (DR clones share DMI serials, and vendor
placeholder serials collide across unrelated machines), the client no
longer falls back to the hostname for MachineID, and ingest skips
identity-pin completion for SourceTrueNAS so a stale pre-fix pin or a
same-named agent host's pin cannot lend the system a machine key and
re-merge what connection scoping keeps apart. Agent.AgentID and the
native metric history keys follow the source ID minus its system:
prefix, so BuildMetricsTarget keeps resolving one series.
Rows minted under the retired hostname-keyed derivation re-key once via
record-declared succession: records name their old canonical IDs in
IngestRecord.SupersededCanonicalIDs and IngestRecords applies the
existing ApplyCanonicalIDSuccessions semantics (operator state and
action audits re-key, the superseded pin drops, never while the old ID
still belongs to a live resource, journal rows are never rewritten).
Alert identities and persisted metric series under old child IDs are
not re-keyed: active alerts re-arm under the new IDs and TrueNAS host
charts are backed by native read-through history.
When fetchVMFSInfo fails (agent not running, timeout, permission
denied, no filesystems), the builder passed the cluster/resources
numbers through — and PVE always reports 0 used for QEMU — so the UI
rendered a confident "0% (0 B/<allocated>)" for every affected VM
instead of the unavailable dash plus diskStatusReason tooltip. The
agent-disabled path already used the -1 sentinel for exactly this;
apply it on the error path too. The stabilizer can still replace the
sentinel with the previous good sample when recent agent evidence
exists.
Reported by Massimo Simoni (support, 2026-07-13): screenshot showed
every VM at 0% disk with only containers reporting real usage.
An alert that resolved while its firing notification was still in the
grouping window or waiting in the persistent queue (alert delay pushes
activation close to resolution; quiet-hours replay defers delivery)
produced a recovery-only notification: CancelAlert dropped the queued
firing, but LastNotified had been set optimistically at dispatch, so
the resolved-notification gate believed the firing had been sent.
CancelAlert now reports whether it cancelled a firing notification that
had not been delivered (grouping window entries and pending queue rows;
mid-send rows are excluded because their delivery may still complete),
and handleAlertResolved suppresses the recovery in that case. This also
covers the quiet-hours replay bypass: a recovery only follows a deferred
firing if the replay was actually delivered.
Addresses #1553
Two root causes behind #1516's remaining reports:
- A node whose Proxmox disks/list query fails (PVE probes SMART per disk
inside that call, so dozens of disks can exceed the API window) now
falls back to the linked host agent's smartctl inventory instead of
leaving the Physical Disks view empty. Each node also gets its own
attempt window so one slow node no longer starves the rest of the
cluster, and a poll that runs out of budget saves partial results.
- The -n standby probe guard is dropped for positively confirmed
non-rotational devices in both the host agent and the node sensor
wrapper. The guard exists to avoid spinning up sleeping HDDs; an SSD
has nothing to spin up, and some SATA SSDs answer CHECK POWER MODE
with a bogus standby state that permanently hid their temperature,
attributes and history.
Refs #1516
The websocket broadcast path applies docker metadata (container
customUrl) via applyDockerMetadataToUnifiedResources, but the REST
/api/resources registry seed read the raw unified state view, so the
two payloads drifted: a container web-interface URL saved in the
drawer never appeared in REST-hydrated tables. Apply the same
hydration at the UnifiedResourceSnapshot provider boundary.
The AI action broker treated an unreadable operator lock as unlocked:
isResourceRemediationLocked returned (false, nil) with no audit store
wired, and the caller logged store errors then dispatched anyway. An
operator's NeverAutoRemediate=true could be silently ignored whenever
the policy store was missing or erroring, which is unacceptable while
Patrol and Assistant run at assisted or full autonomy.
Posture change at the dispatch decision point:
- isResourceRemediationLocked now reports unknown state (nil store or
lookup failure) as an ErrRemediationLockStateUnknown-wrapped error
instead of silently defaulting to unlocked.
- New checkRemediationLockForDispatch gate: dispatches without an
approved human decision fail CLOSED on unknown lock state and
surface "remediation lock state unknown; operator approval
required". Human-approved dispatches keep the historical fail-open
behavior with a warning log. A confirmed lock still refuses even
approved dispatches, as before.
- executeNativeActionWithAudit (TrueNAS app start/stop/restart) now
enforces the lock too; it previously skipped the check entirely.
- Refusals persist Failed audit records with stable
remediation_lock_state_unknown: / resource_remediation_locked:
ErrorMessage prefixes.
- ai-runtime subsystem contract updated to pin the new posture.
Tests cover store-error and nil-store at both autonomy postures on
both dispatch paths; routing/control tests now wire an in-memory
audit store since autonomous dispatch without one is refused.
Two AI write paths asserted unverified success while the Proxmox guest
and Docker start/stop/restart handlers already do read-after-write
checks. Bring both in line with that idiom:
- pulse_kubernetes scale: re-read the deployment's spec/ready replicas
via kubectl through the same agent (bounded settle-and-retry window)
and return a JSON response with a verification block instead of
"Action complete - no verification needed".
- pulse_docker update: the docker agent already recreates the container,
health-checks it, rolls back on failure, and acks a terminal command
status; expose that status through a new Monitor lookup
(GetDockerCommandStatus) plus UpdatesProvider.GetCommandStatus, and
poll it within a bounded window. Responses now report verified
success, verified failure (is_error), or an explicit inconclusive,
never unverified success.
kubernetes_control_test.go also carries a small in-flight fix from the
parallel remediation-lock work (in-memory ActionAuditStore in the test
helper) that these tests require to run on this tree.
Stamp LastSeen and rate-tracker samples with server receipt time instead
of the agent-reported timestamp, matching the host agent fix. A drifted
agent clock could otherwise mark a reporting Docker host or Kubernetes
cluster permanently stale and loop offline/recovery alerts.
Refs #1519
The RC3 memory bound for PBS backup polling summarized any group with
more than 8 snapshots into a single synthesized entry built from group
metadata. A synthesized entry has no verification, size, file, or
per-snapshot time data, so most real deployments saw every backup as
Unverified with no size, PBS files not listed, and a backup timeline
collapsed onto the latest backup day.
Keep the issue #1524 memory bounds but derive them from real data:
always fetch snapshots for stale groups, retain the newest bounded set
per group (limit raised from 8 to 100 to cover real keep policies), and
keep the newest-first global live-state cap. Remove the synthesized
group placeholder path entirely and update the monitoring subsystem
contract and tests to pin real-snapshot bounding.
Fixes#1541
Refs #1524
rcourtman