Production telemetry (30d) shows Patrol averaging ~9.4 billed AI calls
per run while only ~1-2 come from the analysis passes. The remainder is
the alert auto-resolve review: one QuickAnalysis call per active alert
(>=10min old) on every run, re-asking the model about the same standing
alert every interval even when the trigger condition demonstrably still
holds.
Two structural changes, both resolution-neutral:
- Still-firing gate: skip the model review for alerts whose current
snapshot still shows the trigger condition (metric at/above threshold,
offline resource still offline). A correct review could only answer
KEEP, so the question is not worth a billed call. The gate never
resolves anything locally - uncertain cases (unknown types, missing
resources, unmapped metrics) still go to the model, which retains
sole authority over resolution.
- Batched review: the remaining candidates are reviewed in one model
call per 20 alerts (numbered verdict lines, unparseable -> KEEP,
bare-RESOLVE fallback for single-alert batches) instead of one call
per alert.
Steady-state runs with standing alerts drop from N review calls to 0;
runs where conditions may have cleared pay ceil(K/20) instead of K.
QuickAnalysisRequest gains a TargetType tag (recorded on the usage
event) so cost telemetry can decompose alert-review spend from the
main patrol pass going forward. Contract updated in-commit
(ai-runtime: model-owned resolve direction, cost-gated keep direction,
batched review, fail-safe parsing, alert_autoresolve usage tagging).
Production telemetry (7d, 2026-07-17) shows a persistent tail of live
installs on 6.0.0-rc.x and 6.0.3/6.0.4 while 6.0.5 has been stable since
2026-07-09. Root cause for the reachable stranding class: release
selection trusted GitHub's created_at ordering, and this repo interleaves
v5-line maintenance releases with v6 releases (v5.1.36 was created the
day before v6.0.5). Whenever a v5.1.x release is the most recently
created stable:
- stable channel (getLatestReleaseForChannel) returned the first
non-prerelease in list order -> "no update" for every v6 install
until the next v6 release ships
- the RSS rate-limit fallback returned the first feed entry matching the
channel, same failure, both channels
- the unattended updater trusted /releases/latest, which GitHub defines
as most-recently-created stable -> timer no-ops in the same window
All three now select the highest eligible version: the API path tracks
newest stable + newest prerelease in one pass (prerelease = GitHub flag
OR tag shape), the feed fallback picks the max matching the channel, and
pulse-auto-update.sh scans /releases pairing each tag with its own
draft/prerelease flags before the fail-closed shape filter, keeping
/releases/latest as fallback. helm-chart-* tags fail semver parsing and
are dropped everywhere.
Verified not broken (pinned by new tests): rc-channel installs are
offered the newer stable (6.0.0-rc.6 -> 6.0.5) and move onto the next rc
line when one opens (-> 6.1.0-rc.2); the auto-update prerelease filter
(fail-closed is_prerelease_tag, metadata-flag refusal) still refuses
prerelease targets on stable. Go tests fail on the pre-fix code
(returned 5.1.37 / "no update").
Contract: deployment-installability now pins version-max release
selection with proofs in manager_stranded_upgrade_test.go and
pulse_auto_update_test.go.
Telemetry shows thousands of installs configure an AI provider but almost
none ever use the interactive Assistant. A live first-session exercise
(fresh install, Ollama qwen3:8b quickstart) found why: after enabling,
nothing changes on screen — the launcher and handoff buttons are gated on
sessionCapabilities.assistantEnabled, which was only read at page load;
the empty transcript was blank; and the blessed Ollama+qwen3:8b path
reported Patrol degraded while telling the user to pull the model they
had just selected.
- Setup-modal success now opens the Assistant drawer, and the AI settings
save paths refresh the assistantEnabled capability in place
(aiChatStore.refreshEnabledFromServer) so entry points appear without a
reload; toasts point at the Assistant instead of back at settings.
- The empty transcript owns a plain-language welcome and three suggested
prompts that dispatch as real turns (ASSISTANT_SUGGESTED_PROMPTS).
- Patrol static readiness: the blessed Ollama Patrol model is Ready;
other Ollama models keep the warning, now naming the selected model.
- pulse_summarize fleet argument errors instruct the model to enumerate
resources itself instead of interrogating the operator (observed live:
'how is my machine doing?' ended in a resource-ID elicitation).
Contracts: ai-runtime and frontend-primitives Current State updated.
Tests: full internal/ai + internal/api suites green; vitest ChatMessages,
AISettings, aiChat store, and settingsArchitecture suites green; flow
verified live end-to-end.
Every mock-mode boot (e2e/CI managed backends, qual runs, demo containers)
was sending startup/heartbeat pings describing the synthetic fixture fleet,
registering as real installs in the license-server telemetry DB and
dominating the VMware/Kubernetes/TrueNAS adoption aggregates (346 of 6,225
weekly-active installs matched the mock signature, incl. 120 of 227 on
6.1.0-rc.1).
- internal/telemetry: route startup and heartbeat sends through sendEvent,
which drops the ping while mock.IsMockEnabled() is true; checked per event
so runtime mock toggles take effect immediately. Disclosed in the package
doc and both PRIVACY.md copies.
- scripts/telemetry_adoption_report.py: exclude historical mock-fleet rows
(kubernetes_pods = 120×N with vmware_hosts = 7×N, the internal/mock
fixture template) by default, report the excluded row/install counts, and
add --include-mock-fleet to audit them. Versions through 6.1.0-rc.2 keep
pinging the mock fleet until upgraded, so report-side filtering stays
necessary.
- security-privacy contract: record the mock-mode suppression boundary and
the report-side mock-fleet exclusion as subsystem invariants.
- Tests: Go sendEvent suppression/sending pair against an httptest endpoint;
Python signature-matcher and summarize_rows exclusion coverage.
make lint (dupl) flagged two clone pairs: shouldRestartAIChat vs
aiSettingsUpdateTouchesPatrolReadiness shared the forty-field provider
config block, and the approved/rejected action decision-ID collectors in
Pulse Intelligence telemetry differed only by lifecycle state and audit
predicate.
Extract aiSettingsUpdateTouchesProviderConfig for the shared field
check and parameterize pulseIntelligenceActionDecisionIDs by state,
label, and audit predicate, keeping the two named wrappers. The
decision-telemetry contract tests now pin the wrapper call sites, which
still fix the lifecycle state and audit predicate per decision kind.
Contract-Neutral: behavior-preserving dedup of internal helpers to clear dupl lint findings; no public-contract delta
The upstream removal of HandleRunCommand's dead legacy body (ec5e44da0)
deleted TestNormalizeRunCommandApprovalTarget with it, but the v6
host-alias standards test still required that test's snippet in
ai_handlers_test.go, so TestV6DirectHostAliasValidatorCoverage failed.
Drop only the retired validator's snippet; the surviving execute and
investigate-alert validator pins keep host-alias rejection covered.
The raw-command endpoint answers 410 Gone unconditionally; the old
implementation below the retirement return was unreachable (govet) and
kept normalizeRunCommandApprovalTarget, normalizeAIExecuteTargetType,
and AIRunCommandRequest alive with no live callers. Remove them and the
tests that only exercised the removed helpers.
Contract-Neutral: remove unreachable dead code below the 410 retirement return in HandleRunCommand; endpoint behavior unchanged
checkMockAlerts checked guests, nodes, storage, PBS, and PMG but never
looped over the mock estate's DockerHosts, so no docker container state,
health, metric, update, or swarm service alert could ever fire against
mock data. The generator deliberately keeps one container unhealthy per
estate, and the docker alert lifecycle (disable-all sweeps, update-alert
delay, config reconciliation) was unexercisable in the mock QA surface.
Verified live on a scratch mock instance: canonical docker metric alerts
fire, the Containers global-defaults toggle clears all of them via the
DisableAllDockerContainers sweep (the #1545 fix path), and re-enabling
re-fires them on the next cycle.
Contract-Neutral: mock-mode docker alert evaluation wiring fix; no public contract delta
Release-level regression pass over the alert lifecycle found two seams
with store/scheduler-level coverage but untested wiring:
- ingestRecoveryPointsBestEffort (#1580): a complete enumeration deletes
in-scope points the source stopped reporting, an empty enumeration
clears the scope, and a failed upsert skips reconciliation so a broken
batch cannot delete points the source still reports.
- availabilityPollProvider.FixedInstanceInterval (#1582): the configured
poll interval reaches the scheduler descriptor exactly (with the
10s..1h clamp), 0 for disabled or unknown targets, so the adaptive
scheduler cannot collapse the availability detection window again.
'This agent does not support durable typed-operation receipts' told users
nothing they could act on. Say the agent is on an older version and how to
resolve it. Applies to docker container, host update and host storage
cleanup executors.
The DisableAllDockerContainers and DisableAllDockerServices config sweeps
matched only legacy docker-container-/docker-service- alert ID prefixes.
Canonical stateful alerts (image-update, state, health, replica-gap) are
stored under <resourceID>::<specID> state IDs, so existing alerts survived
the toggle and lingered indefinitely, re-notifying past cooldown.
Match on the docker: resource ID scheme as well as the legacy prefixes.
The docker container lifecycle codec in agentexec had a contract test that never
exercised the codec functions at runtime, leaving decode, bind, validate and
identity helpers at zero coverage. This adds a table-driven test taking each to
near or full coverage including every validation error arm. In aicontracts,
CloneActionReference, IsNilAlertPayload and DefaultEngineConfig were uncovered,
so this covers the nil and populated arms and asserts CloneActionReference
produces an independent deep copy.
Both are new test-only files. No source changed. Verified with go test, gofmt
and go vet, with each target function confirmed to move off zero coverage.
Two more newly added *_w0716_coverage_test.go files from the same
branch-coverage wave, touching no source and no existing test.
- internal/models deep-copy clone independence for Docker, Kubernetes,
ZFS and PBS clones, each mutating the original and asserting the clone
stays unchanged
- internal/ai/qualification lab shell-quoting and resource-name
rendering plus report replay predicates and markdown renderers
Cover previously-unexecuted branches in modules that gained behavior
since the last wave. All six are newly added *_w0716_coverage_test.go
files that modify no source and no existing test.
- internal/ai/chat investigation-budget injection plus the agentic
tool-event and string helpers
- internal/ai/qualification manifest Validate error arms and the Patrol
docker-predicate and finding-prerequisite validators
- internal/unifiedresources adapters transforms and ContractResourceType
- internal/config AIConfig nil-receiver default branches
- pkg/aicontracts orchestrator error methods and investigation config
Cloned VMs that keep the same /etc/machine-id collapse into a single
Docker host in Pulse, with each clone's report silently overwriting the
other (#1584). The server now tracks per-identity hostname and machine
ID observations and flags a conflict when a value flaps back to one
already seen inside a 15-minute window, a signature a one-time rename
never produces. The conflict rides the DockerHost model through unified
resources, and the Docker page shows a warning naming the flapping
hostnames with the machine-id remedy. The warning self-clears once one
clone stops reporting for the window.
A vSphere 6.x target fails the connect test at the Automation session
step with a generic "HTTP 500" message, because /api on those releases
routes to the JSON-RPC servlet. When /api/session fails with a non-auth
status, probe the legacy /rest CIS session API with the same
credentials; if that login works the target predates the JSON APIs
Pulse uses, so the test now reports the unsupported-version warning
naming the vCenter 8.0U1 floor instead of pointing at credentials. The
probe deletes the session it creates. Also reword the frontend guidance
for that category in plain terms.
Related to #1585.
Update checks against registries without a hardcoded token endpoint
(lscr.io and other spec-compliant registries) failed with a blanket
"Check failed" because the manifest HEAD went out anonymously and the
401 was terminal. Parse the Bearer challenge on the 401, fetch a pull
token from the advertised realm, and retry once. Token endpoints that
answer with access_token instead of token are now accepted too.
Fixes#1583.
Adds branch-coverage unit tests for previously uncovered pure functions in
internal/monitoring and internal/ai/tools. New test files only, with no source
changes.
Covers existing-cluster IP-override and fingerprint lookup, storage-summary
capacity-trend building, nouveau GPU temperature parsing, resolved-resource
control identity, and available-agent-host formatting. 32 TestBranchCov
functions in 2 files, all vet and gofmt clean.
Adds table-driven branch-coverage unit tests for previously uncovered pure
functions across internal/alerts/config, internal/ai/modelresolution,
internal/monitoring, internal/ai, internal/ai/qualification and pkg/licensing.
New test files only, with no source changes.
Covers per-subsystem alert-default normalization, configured model and provider
resolution, fleet-doctor identity and cluster-endpoint helpers, docker-state and
infrastructure-key mapping, investigation and SMART issue helpers, JSON and
autonomy normalization, and licensing feature-tier resolution. 65 TestBranchCov
functions in 7 files, all vet and gofmt clean.
Adds table-driven branch-coverage unit tests for previously uncovered pure
functions across internal/models, internal/config, pkg/securityutil and
pkg/aicontracts. New test files only, with no source changes.
Covers model collection normalizers, reflect-based metric deep-copy,
removed-host agent bookkeeping, report-schedule and entitlement-billing
normalization, effective AI control and Patrol autonomy resolution,
websocket-origin host normalization, and command-safety classifiers. 61
TestBranchCov functions in 8 files, all vet and gofmt clean.
Adds table-driven branch-coverage unit tests for previously untested pure
helper functions across internal/alerts/config, internal/config,
internal/ai/safety, internal/ai/modelresolution, internal/operationreceipt,
internal/models, internal/securityutil and pkg/securityutil. New test files
only, with no source changes.
Covers alert-config normalization and validation, sensitive-path and
redaction classifiers, URL normalizers, provider model resolution,
operation-receipt decoding, credential masking, and account-to-org role
mapping. 57 TestBranchCov functions in 12 files, all vet and gofmt clean.
Every ID token verification failure rendered the issuer-mismatch
advice, sending users with audience or clock problems down the wrong
path and telling users with a genuine issuer mismatch nothing they had
not already checked (#1533). Map the distinct verification failures
(issuer, audience, expiry) to their own error codes and give each
accurate login-page copy pointing at the server log's got/want detail.
A full monitor reload tears the state down and rebuilds it empty, so
agent-reported host rows vanish until each agent's next report lands.
Config fetches in that window 404ed with a perfectly valid token, which
showed up as rare correlated agent_config_fetch failures across
unrelated agents (#1570).
When the live snapshot has no match, resolve the host from the
persisted continuity store with the same semantics as the live path. A
report-scoped token resolves only its bound host and manage-scoped
tokens resolve by host ID. Deliberately removed hosts stay 404 because
removal deletes their continuity entry.
rcourtman