The AI action broker treated an unreadable operator lock as unlocked:
isResourceRemediationLocked returned (false, nil) with no audit store
wired, and the caller logged store errors then dispatched anyway. An
operator's NeverAutoRemediate=true could be silently ignored whenever
the policy store was missing or erroring, which is unacceptable while
Patrol and Assistant run at assisted or full autonomy.
Posture change at the dispatch decision point:
- isResourceRemediationLocked now reports unknown state (nil store or
lookup failure) as an ErrRemediationLockStateUnknown-wrapped error
instead of silently defaulting to unlocked.
- New checkRemediationLockForDispatch gate: dispatches without an
approved human decision fail CLOSED on unknown lock state and
surface "remediation lock state unknown; operator approval
required". Human-approved dispatches keep the historical fail-open
behavior with a warning log. A confirmed lock still refuses even
approved dispatches, as before.
- executeNativeActionWithAudit (TrueNAS app start/stop/restart) now
enforces the lock too; it previously skipped the check entirely.
- Refusals persist Failed audit records with stable
remediation_lock_state_unknown: / resource_remediation_locked:
ErrorMessage prefixes.
- ai-runtime subsystem contract updated to pin the new posture.
Tests cover store-error and nil-store at both autonomy postures on
both dispatch paths; routing/control tests now wire an in-memory
audit store since autonomous dispatch without one is refused.
Two AI write paths asserted unverified success while the Proxmox guest
and Docker start/stop/restart handlers already do read-after-write
checks. Bring both in line with that idiom:
- pulse_kubernetes scale: re-read the deployment's spec/ready replicas
via kubectl through the same agent (bounded settle-and-retry window)
and return a JSON response with a verification block instead of
"Action complete - no verification needed".
- pulse_docker update: the docker agent already recreates the container,
health-checks it, rolls back on failure, and acks a terminal command
status; expose that status through a new Monitor lookup
(GetDockerCommandStatus) plus UpdatesProvider.GetCommandStatus, and
poll it within a bounded window. Responses now report verified
success, verified failure (is_error), or an explicit inconclusive,
never unverified success.
kubernetes_control_test.go also carries a small in-flight fix from the
parallel remediation-lock work (in-memory ActionAuditStore in the test
helper) that these tests require to run on this tree.
Stamp LastSeen and rate-tracker samples with server receipt time instead
of the agent-reported timestamp, matching the host agent fix. A drifted
agent clock could otherwise mark a reporting Docker host or Kubernetes
cluster permanently stale and loop offline/recovery alerts.
Refs #1519
The RC3 memory bound for PBS backup polling summarized any group with
more than 8 snapshots into a single synthesized entry built from group
metadata. A synthesized entry has no verification, size, file, or
per-snapshot time data, so most real deployments saw every backup as
Unverified with no size, PBS files not listed, and a backup timeline
collapsed onto the latest backup day.
Keep the issue #1524 memory bounds but derive them from real data:
always fetch snapshots for stale groups, retain the newest bounded set
per group (limit raised from 8 to 100 to cover real keep policies), and
keep the newest-first global live-state cap. Remove the synthesized
group placeholder path entirely and update the monitoring subsystem
contract and tests to pin real-snapshot bounding.
Fixes#1541
Refs #1524
Refs #1510
Refs #1501
Refs #1507
Refs #1442
- persist scoped workloads status filters across platform navigation
- derive host memory pressure from available memory
- reapply system settings after every monitor reload path
- bound PBS backup snapshot polling workers during large backup scans
getDockerCommandPayload returned dispatched commands on every report
fetch, causing the agent to re-execute check-updates on every poll
cycle. When the ack also failed, the report was buffered and retried,
creating an infinite loop.
- Only return command payload on the queued->dispatched transition;
subsequent fetches return nil (agent already received it).
- Don't propagate ack errors from handleCheckUpdatesCommand; the report
was delivered and check-updates is fire-and-forget. Command expires
if ack never succeeds.
Refs #1504
When the Proxmox API's meminfo/status payload omits Available, Buffers,
and Cached (common for QEMU guest-agent and node status responses), the
code derived 'available' from Free alone — producing used = Total-Free
which counts reclaimable page cache as used memory (e.g. 94% instead of
the correct 76%).
Guest path (deriveGuestMemInfoAvailable): return 0 when cache metrics
are completely missing and only Free is available, so resolveGuestStatusMemory
tries better sources: guest-agent /proc/meminfo file-read (returns
MemAvailable), RRD memavailable, or the linked host agent.
Node path (resolveNodeMemory): always try RRD memavailable when cache
metrics are missing, not only when effectiveAvailable == 0 — previously
a non-zero Free value blocked the RRD fallback.
Refs #1501
writeHostPhysicalDiskIOMetrics gated ALL per-disk I/O metrics on
host.Sensors.SMART being non-empty. When the agent's SMART collection
fails (smartctl not installed, LXC container can't see /sys/block),
DiskIO data from gopsutil is valid but silently discarded.
Fix: remove the SMART requirement. When no SMART match is found for a
DiskIO entry, try matching against Proxmox API physical disks by device
name (via LinkedNodeID), then fall back to hostID:device as the metric
resource ID. This matches the fallback scheme already used by
HostSMARTDiskSourceID and PhysicalDiskMetaMetricID.
Refs #1487
TCP probes to fast local-network services can complete in under 1ms,
causing latency.Milliseconds() to return 0. Combined with omitempty on
the poller status struct, this made some probes show 'Online' with no
latency in the settings panel. Now successful probes always report at
least 1ms.
All TCP and HTTP availability probes were failing with 'no route to host'
because the Pulse process's Go networking stack cannot reach local subnet
IPs when a Tailscale system extension is active. The ICMP probe already
worked because it uses exec.Command('ping'). Apply the same pattern:
- probeTCP: try Go net.Dialer first, fall back to exec.Command('nc')
- probeHTTP: try Go http.Client first, fall back to exec.Command('curl')
- Both fallbacks use OS-appropriate timeout flags (macOS: -G, Linux: -w)
- HTTP probes now skip TLS verification (availability check, not identity)
- Add AvailabilityProbeHTTPS protocol constant and wire it through
Validate, HTTPURL, Normalize, and the probe switch
- Replace bulky TooltipPortal tooltip on AvailabilityProbeCell with a
clean native title attribute ('14 ms - checked 3s ago')
Before: 0/20 probes green (all 'no route to host')
After: 14/20 probes green (6 legitimately down hosts)
TestMonitorPollGuestSnapshots_RefreshesStaleCanonicalStoreForClusterGuest
passed context.Background() to pollGuestSnapshots, which creates an
internal 60s snapshot timeout. The backupStorageTimeoutSnapshotClient
mock blocks GetStorage until context cancellation, so the test wasted
60s waiting for the timeout to expire. Pass a 200ms deadline context
so pollGuestSnapshots caps its budget down — test now runs in 0.2s
instead of 60s and no longer times out under -race.
Agentless availability checks (ICMP/TCP/HTTP) were always minting
standalone network-endpoint resources, leaving them disconnected from
the known Proxmox/Docker guest they actually monitor. This made
availability evidence invisible on the platform resource row where the
user expects it, per the performance-and-scalability bounded-row
contract.
Backend (unified-resources ingest):
- Add LinkedResourceID field to AvailabilityData and AvailabilityTarget
- resolveAvailabilityLink: explicit link first, then exact-IP unambiguous
correlation; skip hostname-only (lossy); guard against overwriting a
different target's facet
- Unlinked/unmatched probes still mint network-endpoint (fallback)
Frontend:
- Relax getAvailabilityProbePresentation for any resource with availability
- Add compact protocol badge to UnifiedResourceHostTableCard name cell
- Add optional 'Link to resource' field to availability target form
- Add linkedResourceId to frontend types
Contracts: api-contracts, unified-resources, monitoring,
performance-and-scalability, storage-recovery.
Governance: coverage_gap + candidate_lane in status.json.
parseProxmoxOnBoot returned nil when the onboot key was absent from a
valid config, but Proxmox's default is onboot=0 (do not start on boot).
This caused stopped VMs without an explicit onboot line (like
windows-runner and tails-anon) to still generate powered-off alerts
despite not being configured to autostart.
Now returns false for absent keys in non-empty configs. nil is reserved
for empty/failed config fetches and unrecognised values, preserving the
'unknown' fallback for genuine uncertainty.
TestAlertLifecycleCanonicalChangesRemainWritable used a hardcoded date
(2026-03-20) that is now >90 days old, exceeding the incident store's
maxAge retention. trimLocked() evicted the incident shell immediately
after creation, causing GetTimelineByAlertIdentifier to return nil.
Switch to time.Now().UTC().Add(-1h) so the test never goes stale.
The alert engine fired powered-off alerts for every stopped VM and
container regardless of whether the guest was configured to autostart.
This meant intentionally-stopped guests (onboot=0) generated alarm
fatigue — in the live lab, 7 of 12 alerts were noise from stopped VMs
that are deliberately off.
Now the alert engine checks the Proxmox onboot setting before firing:
- onboot=true → stopped is unexpected, alert (preserved behavior)
- onboot=false → stopped is expected, suppress
- onboot=nil → unknown, alert (preserved behavior)
For containers, onboot is parsed from the already-fetched container
config in enrichContainerMetadata (no new API call). For VMs, the
config is fetched only for stopped VMs via GetVMConfig (one extra call
per stopped VM, running VMs are unaffected).
Models: add OnBoot *bool to VM and Container.
Alerts: add onboot-aware suppression branch in CheckGuest.
Monitoring: add parseProxmoxOnBoot helper and fetchVMOnBoot.
Force stale negative LXC Docker detections to re-run after the Docker checker is configured so explicit Proxmox guest Docker inventory repopulates after backend restarts.
getConfiguredHostIPs() and the new discoveryConfigSnapshot() now deep-copy config under a brief RLock and release it before resolving Proxmox/PBS/PMG hostnames via the injectable lookupConfiguredHostIP seam, so slow or blocked DNS can no longer stall monitor writers. The discovery IPBlocklist is the deduplicated merge of the operator-configured blocklist and the resolved configured-host IPs. Adds lock-freedom and merge coverage tests; updates the monitoring contract obligation.
rcourtman