A pool built from /dev/disk/by-id/nvme-eui.<hex> references (the
installer's device naming when identical NVMe models share a box)
produced no serial key in the disk-to-pool matcher, and the WWN lookup
never stripped the eui. prefix smartctl reports, so the disk fell back
to the generic Proxmox usage string and showed 'ZFS' instead of its
pool name while an identically-built node showed 'local-zfs' (issue
#1540).
URLs saved through the resource drawer historically landed in the docker
store under the runtime container key, which any stable record (including
an intentionally empty cleared one) outranks in the unified customUrl
projection, and which orphans on container recreation. On report ingest,
copy those records into the stable app-container guest key when it is
missing (cleared links stay cleared), healing saves stranded before the
drawer moved to the stable identity. Also read the runtime key before its
copy-if-missing container-name snapshot so the freshest write wins among
the docker-store fallbacks.
Refs #1556
Two TrueNAS systems that report the same hostname collapsed into one
flapping resource (#1573, #1575): systemSourceID keyed the system by the
snapshot-reported hostname, every child pool/dataset/app/VM/share/disk
was scoped under it, and the client minted the system's machine key from
the DMI serial with a hostname fallback, so serial-less systems sharing
a hostname (and DR clones sharing a serial) also fully merged in the
identity matcher.
The system source ID now scopes to the connection ID the poller passes
through NewLiveProviderForConnection; the hostname arm survives only for
fixture snapshots, which carry no connection. The ingest identity drops
the machine key entirely (DR clones share DMI serials, and vendor
placeholder serials collide across unrelated machines), the client no
longer falls back to the hostname for MachineID, and ingest skips
identity-pin completion for SourceTrueNAS so a stale pre-fix pin or a
same-named agent host's pin cannot lend the system a machine key and
re-merge what connection scoping keeps apart. Agent.AgentID and the
native metric history keys follow the source ID minus its system:
prefix, so BuildMetricsTarget keeps resolving one series.
Rows minted under the retired hostname-keyed derivation re-key once via
record-declared succession: records name their old canonical IDs in
IngestRecord.SupersededCanonicalIDs and IngestRecords applies the
existing ApplyCanonicalIDSuccessions semantics (operator state and
action audits re-key, the superseded pin drops, never while the old ID
still belongs to a live resource, journal rows are never rewritten).
Alert identities and persisted metric series under old child IDs are
not re-keyed: active alerts re-arm under the new IDs and TrueNAS host
charts are backed by native read-through history.
When fetchVMFSInfo fails (agent not running, timeout, permission
denied, no filesystems), the builder passed the cluster/resources
numbers through — and PVE always reports 0 used for QEMU — so the UI
rendered a confident "0% (0 B/<allocated>)" for every affected VM
instead of the unavailable dash plus diskStatusReason tooltip. The
agent-disabled path already used the -1 sentinel for exactly this;
apply it on the error path too. The stabilizer can still replace the
sentinel with the previous good sample when recent agent evidence
exists.
Reported by Massimo Simoni (support, 2026-07-13): screenshot showed
every VM at 0% disk with only containers reporting real usage.
An alert that resolved while its firing notification was still in the
grouping window or waiting in the persistent queue (alert delay pushes
activation close to resolution; quiet-hours replay defers delivery)
produced a recovery-only notification: CancelAlert dropped the queued
firing, but LastNotified had been set optimistically at dispatch, so
the resolved-notification gate believed the firing had been sent.
CancelAlert now reports whether it cancelled a firing notification that
had not been delivered (grouping window entries and pending queue rows;
mid-send rows are excluded because their delivery may still complete),
and handleAlertResolved suppresses the recovery in that case. This also
covers the quiet-hours replay bypass: a recovery only follows a deferred
firing if the replay was actually delivered.
Addresses #1553
Two root causes behind #1516's remaining reports:
- A node whose Proxmox disks/list query fails (PVE probes SMART per disk
inside that call, so dozens of disks can exceed the API window) now
falls back to the linked host agent's smartctl inventory instead of
leaving the Physical Disks view empty. Each node also gets its own
attempt window so one slow node no longer starves the rest of the
cluster, and a poll that runs out of budget saves partial results.
- The -n standby probe guard is dropped for positively confirmed
non-rotational devices in both the host agent and the node sensor
wrapper. The guard exists to avoid spinning up sleeping HDDs; an SSD
has nothing to spin up, and some SATA SSDs answer CHECK POWER MODE
with a bogus standby state that permanently hid their temperature,
attributes and history.
Refs #1516
The websocket broadcast path applies docker metadata (container
customUrl) via applyDockerMetadataToUnifiedResources, but the REST
/api/resources registry seed read the raw unified state view, so the
two payloads drifted: a container web-interface URL saved in the
drawer never appeared in REST-hydrated tables. Apply the same
hydration at the UnifiedResourceSnapshot provider boundary.
The AI action broker treated an unreadable operator lock as unlocked:
isResourceRemediationLocked returned (false, nil) with no audit store
wired, and the caller logged store errors then dispatched anyway. An
operator's NeverAutoRemediate=true could be silently ignored whenever
the policy store was missing or erroring, which is unacceptable while
Patrol and Assistant run at assisted or full autonomy.
Posture change at the dispatch decision point:
- isResourceRemediationLocked now reports unknown state (nil store or
lookup failure) as an ErrRemediationLockStateUnknown-wrapped error
instead of silently defaulting to unlocked.
- New checkRemediationLockForDispatch gate: dispatches without an
approved human decision fail CLOSED on unknown lock state and
surface "remediation lock state unknown; operator approval
required". Human-approved dispatches keep the historical fail-open
behavior with a warning log. A confirmed lock still refuses even
approved dispatches, as before.
- executeNativeActionWithAudit (TrueNAS app start/stop/restart) now
enforces the lock too; it previously skipped the check entirely.
- Refusals persist Failed audit records with stable
remediation_lock_state_unknown: / resource_remediation_locked:
ErrorMessage prefixes.
- ai-runtime subsystem contract updated to pin the new posture.
Tests cover store-error and nil-store at both autonomy postures on
both dispatch paths; routing/control tests now wire an in-memory
audit store since autonomous dispatch without one is refused.
Two AI write paths asserted unverified success while the Proxmox guest
and Docker start/stop/restart handlers already do read-after-write
checks. Bring both in line with that idiom:
- pulse_kubernetes scale: re-read the deployment's spec/ready replicas
via kubectl through the same agent (bounded settle-and-retry window)
and return a JSON response with a verification block instead of
"Action complete - no verification needed".
- pulse_docker update: the docker agent already recreates the container,
health-checks it, rolls back on failure, and acks a terminal command
status; expose that status through a new Monitor lookup
(GetDockerCommandStatus) plus UpdatesProvider.GetCommandStatus, and
poll it within a bounded window. Responses now report verified
success, verified failure (is_error), or an explicit inconclusive,
never unverified success.
kubernetes_control_test.go also carries a small in-flight fix from the
parallel remediation-lock work (in-memory ActionAuditStore in the test
helper) that these tests require to run on this tree.
Stamp LastSeen and rate-tracker samples with server receipt time instead
of the agent-reported timestamp, matching the host agent fix. A drifted
agent clock could otherwise mark a reporting Docker host or Kubernetes
cluster permanently stale and loop offline/recovery alerts.
Refs #1519
The RC3 memory bound for PBS backup polling summarized any group with
more than 8 snapshots into a single synthesized entry built from group
metadata. A synthesized entry has no verification, size, file, or
per-snapshot time data, so most real deployments saw every backup as
Unverified with no size, PBS files not listed, and a backup timeline
collapsed onto the latest backup day.
Keep the issue #1524 memory bounds but derive them from real data:
always fetch snapshots for stale groups, retain the newest bounded set
per group (limit raised from 8 to 100 to cover real keep policies), and
keep the newest-first global live-state cap. Remove the synthesized
group placeholder path entirely and update the monitoring subsystem
contract and tests to pin real-snapshot bounding.
Fixes#1541
Refs #1524
Refs #1510
Refs #1501
Refs #1507
Refs #1442
- persist scoped workloads status filters across platform navigation
- derive host memory pressure from available memory
- reapply system settings after every monitor reload path
- bound PBS backup snapshot polling workers during large backup scans
getDockerCommandPayload returned dispatched commands on every report
fetch, causing the agent to re-execute check-updates on every poll
cycle. When the ack also failed, the report was buffered and retried,
creating an infinite loop.
- Only return command payload on the queued->dispatched transition;
subsequent fetches return nil (agent already received it).
- Don't propagate ack errors from handleCheckUpdatesCommand; the report
was delivered and check-updates is fire-and-forget. Command expires
if ack never succeeds.
Refs #1504
When the Proxmox API's meminfo/status payload omits Available, Buffers,
and Cached (common for QEMU guest-agent and node status responses), the
code derived 'available' from Free alone — producing used = Total-Free
which counts reclaimable page cache as used memory (e.g. 94% instead of
the correct 76%).
Guest path (deriveGuestMemInfoAvailable): return 0 when cache metrics
are completely missing and only Free is available, so resolveGuestStatusMemory
tries better sources: guest-agent /proc/meminfo file-read (returns
MemAvailable), RRD memavailable, or the linked host agent.
Node path (resolveNodeMemory): always try RRD memavailable when cache
metrics are missing, not only when effectiveAvailable == 0 — previously
a non-zero Free value blocked the RRD fallback.
Refs #1501
writeHostPhysicalDiskIOMetrics gated ALL per-disk I/O metrics on
host.Sensors.SMART being non-empty. When the agent's SMART collection
fails (smartctl not installed, LXC container can't see /sys/block),
DiskIO data from gopsutil is valid but silently discarded.
Fix: remove the SMART requirement. When no SMART match is found for a
DiskIO entry, try matching against Proxmox API physical disks by device
name (via LinkedNodeID), then fall back to hostID:device as the metric
resource ID. This matches the fallback scheme already used by
HostSMARTDiskSourceID and PhysicalDiskMetaMetricID.
Refs #1487
TCP probes to fast local-network services can complete in under 1ms,
causing latency.Milliseconds() to return 0. Combined with omitempty on
the poller status struct, this made some probes show 'Online' with no
latency in the settings panel. Now successful probes always report at
least 1ms.
All TCP and HTTP availability probes were failing with 'no route to host'
because the Pulse process's Go networking stack cannot reach local subnet
IPs when a Tailscale system extension is active. The ICMP probe already
worked because it uses exec.Command('ping'). Apply the same pattern:
- probeTCP: try Go net.Dialer first, fall back to exec.Command('nc')
- probeHTTP: try Go http.Client first, fall back to exec.Command('curl')
- Both fallbacks use OS-appropriate timeout flags (macOS: -G, Linux: -w)
- HTTP probes now skip TLS verification (availability check, not identity)
- Add AvailabilityProbeHTTPS protocol constant and wire it through
Validate, HTTPURL, Normalize, and the probe switch
- Replace bulky TooltipPortal tooltip on AvailabilityProbeCell with a
clean native title attribute ('14 ms - checked 3s ago')
Before: 0/20 probes green (all 'no route to host')
After: 14/20 probes green (6 legitimately down hosts)
TestMonitorPollGuestSnapshots_RefreshesStaleCanonicalStoreForClusterGuest
passed context.Background() to pollGuestSnapshots, which creates an
internal 60s snapshot timeout. The backupStorageTimeoutSnapshotClient
mock blocks GetStorage until context cancellation, so the test wasted
60s waiting for the timeout to expire. Pass a 200ms deadline context
so pollGuestSnapshots caps its budget down — test now runs in 0.2s
instead of 60s and no longer times out under -race.
rcourtman