When a deep scan was attempted for a workload (container/VM) but produced no
command output — e.g. the host agent rejects exec — the AI analyzer was still
asked to identify the service from metadata alone. It confabulated confident,
false identities: an ESPHome LXC and an influxdb-telegraf LXC were both
"identified" as Pi-hole at 0.95 confidence, with invented facts carrying
fabricated command sources (source: "pihole -v" for a command that never ran)
and a docker exec CLI for an LXC.
When a command scan was attempted for a command-dependent resource type but
yielded nothing, abstain: skip the analyzer entirely and return a not-determined
result (no service, no facts, zero confidence) with guidance to enable Pulse
Commands. Host agents are unaffected (identified from their own metadata), and
deployments without command scanning keep the metadata-only path. Abstaining
also avoids a wasted model call.
Verified live: re-running discovery on the esphome LXC now returns an empty,
zero-confidence result with the correct container CLI instead of fabricated
Pi-hole facts. Adds a regression test.
On a provider/transport failure the chat rendered the raw upstream error. An
OpenRouter 402, for example, dumped the provider's billing JSON and a
workspace-key dashboard URL straight into the message: unreadable, and a leak
of provider routing plus a key fragment.
Classify known failures (billing/402, auth, rate-limit, timeout, cancel) into
clean actionable messages, and for anything else strip the embedded JSON body
and any URL so raw provider payloads never reach the user. Applied to both the
in-stream error-event path and the transport-failure fallback. Adds tests,
including the exact OpenRouter 402 string.
Verified live: the 402 now reads "The AI provider rejected the request for
billing or quota reasons..." instead of the raw JSON+URL dump.
OpenRouter and other OpenAI-compatible gateways normalize chain-of-thought
into a "reasoning" delta field, while DeepSeek's direct API uses
"reasoning_content". The OpenAI-compatible client only read
"reasoning_content", so reasoning models routed via OpenRouter (e.g.
deepseek/deepseek-v4-pro, the configured Assistant default) had every
thinking token dropped. The Assistant showed a long dead pause and then the
answer instead of a live thinking stream, which read as "DeepSeek doesn't
stream".
Parse "reasoning" on both the streaming delta and the non-streaming message,
and surface it as a thinking event alongside the existing reasoning_content
path. Direct DeepSeek (reasoning_content) is unchanged. Adds a regression
test feeding an OpenRouter-style reasoning stream.
Tighten v5-to-v6 upgrade safety, release installability, provider MSP mode handling, AI cost accounting, metrics flushing, and frontend guardrails for the v6.0.0 GA candidate.
Two follow-ups caught by independent re-review:
- #29 (8855b78c0): GetConnectionStatuses keys a name-less instance by host,
but the diagnostics lookup used 'pve-'/'pbs-'+Name only, so the
monitor-state merge was a silent no-op for unnamed instances. Fall back
to host to match.
- #22 (d9da9b18f, #1254): add a full-router test asserting an UNBOUND
agent:report token gets 404 (not 200) on config fetch — continuously
verifying the token-binding security boundary that the scope change
relies on.
Follow-up to a85ec40a4 (#19), caught by independent re-review. Ceph pool
storage ("<instance>-ceph-pool-<name>") is raised/cleared by the separate
Ceph poll path (checkCephPoolStorage) and is NOT in the allStorage
inventory passed to SyncStorageAlertsForInstance — which runs before
pollCephCluster. Without a guard the sweep cleared the ceph pool usage
alert every storage poll and pollCephCluster re-raised it every cycle
(flapping: notification spam + StartTime reset). Skip any alert whose
ResourceID contains '-ceph-pool-'. Adds a regression test.
#23 (a6b773a85) made Restart load the AI config once and pass it to
startWithConfig, so the start path no longer re-loads it. The test still
expected LoadAIConfig().Twice(); updated to .Once() to match the fixed
single-load behavior. (Caught by running the full internal/api suite,
which the per-fix targeted runs had missed.)
Reverts b39cfc3a0. That port of v5 fix#1360 made QuickAnalysis build a
per-model provider via NewForModel, but v6 deliberately uses the configured
s.provider (it already has createPatrolProviderForModel for the per-model
case elsewhere), and NewForModel resolves unknown prefixes to a real Ollama
client rather than erroring — so the fallback never fired. This broke the
existing TestPatrolService_AskAIAboutAlert tests (panic via an empty
response) and changed v6's intended behavior. Restored to v6's original
QuickAnalysis. #1360 needs re-evaluation against v6's provider model before
any re-attempt.
Back-port the core of v5 fix 1de1392c9 to v6 (minimal, not the full
dedup refactor). The model list API now sets ModelInfo.Provider (stamped
in prefixProviderModels and passed through HandleListModels), and the
frontend groupModelsByProvider prefers the server-supplied provider over
splitting the model id, so models with opaque ids (e.g. Ollama-hosted
llama3-8b) group under the correct provider. Adds a grouping regression
test.
Back-port v5 fix 333e66a8e to v6, adapted to v6's DockerHostView accessors.
findMatchingDockerHost now skips an agentID/token match when
dockerHostIdentityConflicts detects a different MachineID, Swarm NodeID, or
(absent stronger IDs) hostname, so two physically different Docker nodes
that share a token+agentID are not collapsed into one host record. Adds a
regression test.
Back-port v5 fix 716340dcd to v6. Extracts startWithConfig from Start so
Restart passes the config it already loaded down to the start path, instead
of Start re-reading loadAIConfig a second time. Avoids starting the AI
service with a stale/inconsistent config if it changed between Restart's
load and the start. Both Restart start paths now use startWithConfig.
Back-port v5 fix ae6b663e9 to v6. Removes the 'only fetch ZFS pools when a
zfspool/zfs/local-zfs storage exists' gate (you cannot tell whether a dir
storage sits on a ZFS dataset without the pool list; fetch is gated by
PULSE_DISABLE_ZFS_MONITORING and empty-fast on non-ZFS nodes) and the
type guard before matchZFSPoolForStorage, so a dir storage whose path
resolves to a pool (e.g. /rpool/data) now gets ZFS health attached. The
guardrail-pinned 'if pool := matchZFSPoolForStorage(...)' call form is
preserved. Adds a poll integration test.
Back-port the agentupdate half of v5 fix 8600706da to v6 (the install.sh
FreeBSD rc.d supervisor-pidfile change is already present in substance).
performUpdate now resolves the executable via resolveExecutablePath, which
falls back to an absolute, existing os.Args[0] when os.Executable() fails
or returns empty — the condition that breaks agent self-update on
FreeBSD/OPNsense. Adds an injectable osArgsFn and a unit test.
Back-port v5 fix 48bdfdc30 to v6. In node scope, collectSwarmDataFromManager
now derives each service's Desired/Running/Completed task counts from the
node-local task list and drops services with no tasks on this node, instead
of reusing the cluster-wide ServiceStatus and a 'keep all when none match'
fallback. Without this, a node-scoped Swarm view showed wrong counts and
services that aren't running on that node. Adds two regression tests.
Back-port v5 fix 5f372e257 to v6. QuickAnalysis now builds a provider for
the configured PatrolModel via providers.NewForModel (matching the 8 other
call sites that already do so), falling back to the default provider if it
cannot. Previously a patrol model on a different provider than the default
(e.g. default Gemini, patrol OpenAI) was sent to the wrong backend. Guard
changed from provider==nil to cfg disabled, with a re-check after provider
selection. Updates the existing fallback test and adds a positive
patrol-provider test (httptest OpenAI server).
Back-port v5 fix d310c257a to v6, adapted to v6's connection-status key
format. computeDiagnostics now merges a failed PVE/PBS diagnostics probe
with the monitor's live connection state: if the long-running poller still
reports the instance connected, a transient probe failure (network blip,
TLS re-check) no longer flips it to 'disconnected' in the UI. Uses v6's
'pve-<name>'/'pbs-<name>' status keys (v5 used a bare node name, which
would not match in v6). Adds a merge-logic regression test.
Back-port v5 fix 22f8dd3bb to v6. Adds normalizeSMARTDeviceIdentifier
(strips a trailing ' [scsi]'/'[ata]'-style type annotation and the /dev/
prefix) and uses it in the device-path match for both mergeNVMeTempsIntoDisks
and mergeHostAgentSMARTIntoDisks, with a non-empty guard. Without it, disks
whose SMART entries carry a typed label never matched by device path, so
their temperature stayed 0. Function signatures/call sites are unchanged
(canonical guardrail still passes). Adds a unit test.
Back-port the smartctl regex half of v5 fix 0c2de2938 to v6 (the
prerelease-aware CompareVersions half is already present). smartctl text
fallback now matches 'Current Drive Temperature:' in addition to
'Current Temperature:', so OPNsense/pfSense disk temperatures are read
instead of reported as missing. Adds a regression test.
Back-port v5 fix b83f717ce to v6. zfsMountpointScore now gives top
priority (score 0) to a dataset whose mountpoint is exactly '/', even a
child dataset like rpool/ROOT/pve-1, so it outranks a pool-root dataset
mounted elsewhere (e.g. rpool at /mnt/storage). Previously a pool-root
dataset won unconditionally, so the pool summary could reflect the wrong
dataset's usage. Updates scoring tests and adds a regression case.
Back-port v5 fix 117ec901d to v6, adapted to v6's config subpackage.
Adds NormalizePBSDefaults (called from UpdateConfig alongside the other
Normalize*Defaults) and pbs.cpu/pbs.memory to ValidateHysteresisThresholds.
PBS was the only monitored resource type whose thresholds were not
normalized on UpdateConfig, so user-supplied configs with nil/negative
PBS thresholds were never defaulted and a Trigger=0 'disable' sentinel
did not clear the Clear value. Adds regression test.
Back-port v5 fix a4834ed80 to v6. hostDisplayName now appends
'(Host Agent)' (or returns 'Host Agent' for an unnamed agent) when the
host is linked to a Proxmox node/VM/container, and the disk-temperature
resource name uses hostDisplayName(host) instead of host.DisplayName.
Without this, a host agent running alongside the Proxmox node/VM it is
linked to produced alerts with an identical name, making them
indistinguishable. Adapted to v6's 'Agent' fallback. Adds test cases.
Back-port v5 fixes 5997fd81f and 0a7b93a84 to v6:
- GetWebhooks list response now includes the configured mention so the UI
shows it after reload instead of blanking it.
- sendResolvedWebhook now assigns data.Mention (v6 set it for grouped/
firing webhooks but dropped it on resolved), and the Discord/Slack/Teams/
Mattermost ResolvedPayloadTemplate strings gained {{if .Mention}} guards.
Without these, a configured @everyone/@channel was silently omitted from
resolved/cleared notifications. Adds list-API and per-service resolved
mention regression tests.
Back-port v5 fix ffaeea18d to v6. clusterEndpointEffectiveURL now derives
hasFingerprint strictly from the endpoint's own Fingerprint and ignores
the cluster-level value the callers pass. Previously, when only the
primary node had a fingerprint, every fingerprint-less cluster member was
routed to its per-node IP — bypassing hostname TLS certificate validation
while effectively pinning the wrong cert. Adds direct regression cases.
(v6 has no buildClusterClientEndpoints, so the v5 builder-level test is
covered by the unit cases instead.)
Back-port v5 fix 0dca8a037 to v6. Extract pollPVEBackupsAndSnapshots:
the two backup-inventory scans share a bounded backupCtx, which is then
cancelled, and pollGuestSnapshots runs on the parent context. Because
the parent has no deadline, pollGuestSnapshots establishes its own
60s-4min budget instead of inheriting an already-exhausted backup
deadline and skipping entirely (the v6 early-return at
monitor_backups.go made this strictly worse). Adds a regression test
proving snapshots still poll after the storage scan exhausts its budget.
Back-port v5 fix 398ef8117 to v6, adapted to v6's canonical alert keying.
pollStorageWithNodes now calls SyncStorageAlertsForInstance after the
per-storage checks, clearing storage-inventory alerts whose ResourceID is
no longer present. Matches by ResourceID (every v6 storage alert sets
ResourceID=storage.ID) rather than reconstructed alert keys, so it is
robust to v6's composite alert-state IDs (e.g. ...::metric-threshold:usage).
Conservative: live storage keeps its alert (ResourceID still valid),
non-storage alerts are untouched, Ceph alias IDs are treated as valid.
Without this, a storage that disappears or changes identity left its
alert visible until the multi-day stale-alert TTL. Adds unit + poll
integration regression tests.
Back-port v5 fix 9fb76579c to v6. CheckBackupsWithInventory now filters
same-VMID guest candidates by backup kind (qemu/lxc) before attributing
a backup, in both the canonical-key lookup and the bare-VMID secondary
path. Without this, a vm backup with vmid=101 was attributed to an lxc
container with the same vmid (or vice-versa), suppressing the
orphaned-backup alert and attaching the wrong guest identity.
Adds guestMatchesBackupType/filterGuestsByBackupType helpers and a
regression test.
Back-port v5 fix 04a828264 to v6. canReadConfig now accepts
ScopeAgentReport (legacy host-agent:report) in addition to the config/
manage/settings scopes; resolveConfigAgent already restricts a
report-only token to the host it is bound to, so it can only read its
own config. Without this, a v5 agent upgraded to a v6 server while still
holding a report-only token gets HTTP 403 on config fetch and breaks.
Retargets TestHandleConfigMissingConfigScope to monitoring:read and adds
TestHandleConfigAllowsBoundAgentReportScope as a #1254 regression guard.
Back-port v5 fix af486b6f5 to v6. LoadAlertConfig now back-fills
Schedule.Cooldown=5, MaxAlertsHour=10, NotifyOnResolve=true and the
Grouping defaults for legacy alerts.json files saved before these
fields existed, probing raw JSON so an explicit 0/false is preserved.
Without this, a v5 user migrating to v6 loads with zeroed cooldown and
rate limit, silently losing notification-spam protection.
Copy guest metadata at the store boundary so asynchronous monitor persistence cannot race with caller-owned metadata during release backend race tests.
Verified via deadcode/ts-prune detection + 317-agent cross-repo verification swarm.
Gated: go build + go vet + test-compile (pulse module), tsc --noEmit (full project
incl tests), npm run lint. Net -2549 lines across 161 files.
Apply max alerts/hour to cooldown and critical re-notification paths so active alerts cannot bypass the UI delivery cap.
Persist LastNotified on active alert records when dispatching, including clone dispatches after reload.
Refs #1444
rcourtman