- add a forced run action to pulse_discovery for known resources
- make discovery progress describe model-backed evidence analysis rather than a live Assistant chat
- keep shared select hydration stable for persisted discovery intervals
Small Ollama models (qwen2.5:11b, qwen2.5:14b, similar) frequently emit
Pulse tool invocations as plain JSON inside content instead of routing
through the structured tool_calls channel. Users saw raw payloads like
`{"name": "pulse_query", "parameters": {...}}` as the assistant's final
response.
Extend cleanToolCallArtifacts and containsToolCallMarker with a new pass
that detects this leak shape, gated on a closed allowlist of canonical
tool names sourced from the runtime registry. The allowlist auto-syncs
when registerTools() gains a tool, so no separate hand-maintained list.
Anchored on (?:^|\n) and a leading `"name"` key so prose containing JSON
fragments, unrelated objects (`{"foo":"bar"}`), or named resources
(`{"name":"my-vm","cpu":50}`) are left untouched.
The reporting feature now ships across two surfaces (PDF/CSV export
and pulse_summarize chat tool) and three modes (single-resource,
fleet, summarize). Without usage telemetry we can't tell whether the
work earns its place — operator demand, AI-vs-heuristic adoption,
range/format preferences are all invisible. Stops further feature
investment from being pure speculation.
Three new info-level log events, structured so an agent can grep
transcripts and group by dimension without a separate metrics
pipeline (matches the "agent owns ops analysis, human gets outcomes"
posture in MEMORY.md):
reporting.single.generated — single-resource PDF/CSV
reporting.fleet.generated — multi-resource fleet PDF/CSV
reporting.summarize.invoked — pulse_summarize chat tool (both modes)
Common dimensions: org_id, format/action, range, ai_configured,
findings_configured, window_start/end. Single-resource adds
resource_type + metric_type + bytes; fleet adds resource_count +
bytes; summarize adds resource_type + resource_count (fleet mode) +
narrative_source (so we can audit AI-fallback rate).
Includes rangeLabel() helper that maps a window to the canonical
catalog range token (24h/7d/30d) with a 1h tolerance, falling back
to "<hours>h" so non-standard windows still group. Tested.
TestReportingTelemetryEventNames pins the canonical event names as
a contract — an agent grepping logs depends on them being stable;
changing them silently would break audit tooling on the consumer
side.
The reporting engine already logs the resolved narrative source
(heuristic/ai) at debug level via the existing "Generating report"
line, useful for diagnosing why a specific report fell back. Kept
at debug; the new info-level events cover the operator surface.
v1 of pulse_summarize (1fe5d6853) shipped with heuristic narrative
only. The follow-up wiring promised in that commit now lands: the
chat session carries optional report-narration providers that the
tool's handler reads when building requests, so AI-narrated synthesis
flows into chat using the same provider, sanitizer, model selection,
cost ledger, and budget gate the report PDF endpoint already uses.
Pipeline:
- pkg/reporting Narrator / FleetNarrator / FindingsProvider interfaces
are already implemented by internal/ai.Service. No new
implementations.
- tools.ExecutorConfig + PulseToolExecutor gain three optional fields
(ReportNarrator, ReportFleetNarrator, ReportFindingsProvider).
Clone() copies them so per-session executors inherit the wiring.
- chat.Config gains the same three fields; NewService threads them
into ExecutorConfig.
- tools_summarize.go reads e.reportNarrator/FleetNarrator/
FindingsProvider and populates MetricReportRequest /
MultiReportRequest. The engine already accepts these on the request
and falls back to heuristic when they are nil — no engine changes
needed.
- AIHandler gains SetReportNarratorResolver(ctx -> narrators); both
per-tenant and default chat.Config construction sites invoke the
resolver. Router wires the resolver to AISettingsHandler.GetAIService
with the same Enabled-gate the reporting handler uses.
Unconfigured tenants are unchanged: the resolver returns nil, the
tool returns heuristic narrative — identical to today. Configured
tenants get AI synthesis in chat that matches what their report PDF
already carries, billed and budget-gated the same way.
The reporting synthesis layer (observations, recommendations,
outliers, period comparison) shipped trapped behind the PDF/CSV
export. Operators who chat with Assistant could not ask "what's been
happening with pve1 this week" — the data path existed but had no
non-PDF surface. This commit adds a single new tool, pulse_summarize,
that wraps the engine's non-rendering entry points (NarrativeFor /
FleetNarrativeFor) so that question gets answered in chat.
The tool takes an action parameter (resource | fleet) and routes
accordingly:
- resource mode requires resource_type + resource_id and returns the
same Narrative the single-resource report carries (health status,
observations, recommendations, period comparison).
- fleet mode requires resource_type + a comma-separated resource_ids
string (PropertySchema does not currently support array items, and
CSV is LLM-friendly enough) and returns the FleetNarrative
(outliers, patterns, recommendations). Capped at the same
multi-report ceiling (50) as the API endpoint.
The tool is read-only — no control level requirement, no approval
gate — and uses the global reporting engine the rest of the app
already shares. Returns a JSON envelope so chat can render it or
hand it back to the model for follow-up framing.
v1 ships with heuristic narrative only. The AI narrator wiring
through the chat session (Narrator/FleetNarrator/FindingsProvider
threaded via chat.Config -> tools.ExecutorConfig -> PulseToolExecutor)
is a focused follow-up; it lets the same tool inherit the per-tenant
AI service the report PDF endpoint already uses. The seam is
already in place because NarrativeFor/FleetNarrativeFor take an
optional narrator on the request — v1 passes nil, v2 populates it.
Closes the certainty loop for agents watching the substrate's push
channel. The action audit's read-after-write probe outcome was
already persisted on the audit record, but agents watching
action.completed only learned "the action ran" — they had to fetch
/api/actions/{id} to know whether the read-back probe confirmed
the intended state. That defeated the substrate's
push-notification guarantee for dispatch certainty.
The new agent-stable AgentResourceActionVerification projection
(ran, success, command, note, ranAt — output stays in the audit
record, deliberately omitted from events to keep payloads small)
is now carried on both:
- the action.completed SSE payload, projected from
record.Result.Verification by the router-side bridge in
wireAIChatDependenciesForService, and
- the resource-context bundle's recentActions surface, via the
same shared projectAgentResourceVerification helper
so the bundle (depth) and the doorbell (push) speak the same
vocabulary. Refused-before-dispatch failures omit verification
(the probe never runs) so agents branch on field presence to
distinguish "no probe attempted" from "probe ran with empty
result". Three contract pins lock the symmetry: payload field
present, router bridge populates it, bundle parallels.
The capabilities manifest's subscribe_events description now
mentions the verification block so external agents discover the
field through the same path they already use to learn the rest
of the agent surface.
The "+N" overflow indicator on TagBadges was styled with
`cursor-pointer`, which signals a clickable affordance — but the
element only listens for mouseenter/mouseleave to show a tooltip and
has no click handler. Switch to `cursor-help` so the cursor matches
the actual interaction (hover for more info), avoiding a phantom
click expectation.
Third compounding slice on per-resource operator state, completing the
suppression triangle: maintenance window (slice 31), intentionally
offline (slice 32), and now NeverAutoRemediate at the action-broker
boundary. When the operator has set NeverAutoRemediate=true on a
resource, executeCommandWithAudit refuses the dispatch even with a
valid approval and matching plan hash — per-resource operator intent
outranks per-action approval.
The check sits next to the drift refusal in the same audit hot path:
fail-open on store-lookup error (logged but doesn't block), persist a
Failed audit record with `resource_remediation_locked:` prefix on the
ErrorMessage so audit-UI filters and alert rules can branch on the
stable token. Returns ErrResourceRemediationLocked sentinel for
caller-side branching.
Reuses the existing actionAuditStore handle (which is the unified
ResourceStore from slice 29), so no new provider plumbing needed —
the broker already has the data path it needs.
Pulse's primary monitored platform is Proxmox, but the per-command-class
preflight catalog only covered systemd services, Docker containers, and
Kubernetes deployments. Operators approving a Patrol-proposed
qm restart/qm stop/qm shutdown saw the generic preflight without the
operational nuance that distinguishes those verbs (qm stop is a hard halt,
qm shutdown is graceful with a 60s ACPI timeout, etc.).
Adds eight new classes covering qm and pct lifecycle (reboot/restart,
stop, start, shutdown). Each gets hand-authored safety and verification
copy that names the actual semantics — pct stop is destructive vs pct
shutdown's lxc-attach handoff — so the operator sees concrete context at
approval time.
Broker-level VerificationCommandForCommand intentionally does not derive
qm status / pct status from these classes — pulse_control's
verifyGuestAction already runs those checks at the tool layer, so adding
a parallel broker dispatch would double-run. The preflight copy still
names what the tool-layer verification will read.
Closes the read-after-write piece of the user's act-with-control loop:
"Act with control: ask for approval, execute through Pulse tools, then
verify." Until now Pulse authored verification narrative copy at
approval time but never actually executed the check.
Adds ActionVerificationResult to internal/unifiedresources/actions.go
and embeds it as ExecutionResult.Verification so the existing
result_json column persists the outcome without a schema migration.
The struct records: did the check run, what command did Pulse send,
what came back, did it succeed, when, and any failure note.
Adds VerificationCommandForCommand in tools_control.go that derives
the per-class read-after-write check used by the broker:
- service-restart / service-start / service-reload / service-stop
→ systemctl is-active <unit>
Container classes (container-restart, container-stop) are
intentionally deferred to pulse_docker's existing tool-level
docker inspect verification — adding a broker-level dispatch would
double-run the same check.
executeCommandWithAudit now runs the derived verification command via
the same agent path immediately after a successful dispatch, captures
output and exit code, and writes the result onto
ExecutionResult.Verification before recordActionExecutionResult
persists. If the check returns non-zero, Verification.Success=false
with a Note explaining the exit code so the audit history honestly
shows that Pulse ran the action but couldn't confirm it took.
Verification is best-effort and class-scoped: unknown command shapes
leave Verification nil rather than fabricating a verified=true entry,
matching the no-fabrication boundary from Impact / preflight authoring.
Tests cover:
- VerificationCommandForCommand derives the right check per class,
shell-escapes single quotes in unit names, and returns "", false
for docker (deferred) and unknown commands.
- TestExecuteCommandWithAuditRunsClassDerivedVerificationAfterDispatch
- TestExecuteCommandWithAuditMarksVerificationFailedWhenReadbackDoesNotConfirm
- TestExecuteCommandWithAuditSkipsVerificationForUnclassifiedCommands
Updates ai-runtime contract with the read-after-write rule and
container-class deferral. Adds Completion Obligation #21 to
unified-resources pinning ExecutionResult.Verification shape and the
no-fabrication boundary. Extends TestActionExecutionContractStaysAPIOwned
to pin ActionVerificationResult struct and the Verification field on
ExecutionResult.
Operators about to approve a Pulse-driven action need to know what the
command actually touches before saying yes. Until now the preflight
shown at approval time was the same generic boilerplate for every
action: "approval is scoped, hash must match, etc." — useful safety
posture but no operational specifics. The dry-run summary said the
same thing for 99% of actions.
Adds classifyApprovalCommand and approvalCommandClassPreflightAdditions
in tools_control.go that bucket common Pulse remediation actions and
return hand-authored safety + verification additions per class:
- service-restart (systemctl restart, service restart)
- service-stop (systemctl stop, service stop)
- service-start (systemctl start, service start)
- service-reload (systemctl reload)
- container-restart (docker / podman restart)
- container-stop (docker / podman stop)
- k8s-rollout-restart (kubectl rollout restart)
The additions name concrete operational facts the operator needs:
- For service-restart: "Service will be briefly unavailable; no other
unit dependencies altered" plus verification "Read back
systemctl is-active <unit>" and "tail journal for crash patterns".
- For container-restart: "Container will be briefly unavailable;
image and volume mounts unchanged" plus "Read back via docker
inspect" verification.
- For k8s-rollout-restart: "Pods rolled in waves per deployment
strategy; PodDisruptionBudget continues to apply" plus "watch
kubectl rollout status" verification.
approvalPreflight now appends class-specific safety and verification
content onto the existing default content rather than replacing it,
so the broker's structural safety posture (org scope, hash match,
single-use approval) remains visible alongside the operational copy.
Unknown command classes return nil/nil — no fabricated padding for
commands the bucket does not recognize. The default preflight stands
on its own, matching the no-fabrication rule that runs through the
rest of the trust-record arc.
Tests cover: each known class returns non-empty additions with the
expected operational tokens; unknown commands return nil; the
end-to-end approvalPreflight merge surfaces both default and
class-specific safety/verification entries. ai-runtime contract
pinned with the per-class enrichment rule and the no-fabrication
boundary for unknown classes.
Slice 13 added the drift refusal path but only logged at WARN level —
the audit history showed nothing, so an operator reviewing the action
trail could not see "Pulse caught this drift attempt." Now the drift
branch writes a Failed audit record with Result.ErrorMessage prefixed
"plan_drift:" and dispatches a Failed lifecycle event before returning
ErrActionPlanDrift.
The record carries the same Request, Plan, and Approvals snapshots that
a normal audit record would, so the operator-facing audit row shows
exactly what was attempted and what was approved. The "plan_drift:"
prefix is a stable token for downstream surfaces (audit UI filters,
alert rules) to distinguish drift refusals from generic execution
failures.
Extends TestExecuteCommandWithAuditRefusesPayloadDriftAgainstApprovedPlan
to assert the audit record exists with State=Failed and the plan_drift
error message after refusal. Updates the code-standards snippet check
for ErrActionPlanDrift to match gofmt's actual alignment in actions.go.
ai-runtime contract pinned with the audit-record-on-drift rule.
PlanHash existed on ActionPlan as the contract for "the operator
approved exactly this (command, target, reason) combination" but the
broker never validated it: at execute time the freshly-recomputed hash
was overwritten by the approved plan's hash via mergeApprovedActionPlan
without comparison. A drifted payload (LLM re-emits with different
command, agent ID changes between approval and execute, malicious
injection) would run under a stale approval.
Fix: at the dispatch boundary in executeCommandWithAudit, recompute the
approval-equivalent hash from the actual payload using approvalPlanHash
(same function used at approval-creation time, so direct comparison is
meaningful), compare to plan.PlanHash, and refuse with a new
ErrActionPlanDrift error when they differ. The drift refusal also logs
at WARN level with action_id, approval_id, and both hashes so audit
review can see when drift was caught.
When approvedHash is empty (older approval records or contract paths
that did not author one), validation is skipped and existing behavior
is preserved.
Adds two tests:
- TestExecuteCommandWithAuditRefusesPayloadDriftAgainstApprovedPlan:
approval is for "systemctl restart workload"; payload at execute
time is "rm -rf /var/log/pulse"; expects ErrActionPlanDrift and no
agent dispatch.
- TestExecuteCommandWithAuditAllowsMatchingPlanHash: same payload
matches the approved hash; expects normal dispatch.
Updates two pre-existing tests that used stub PlanHash strings:
- TestExecuteCommandWithDeniedApprovalDoesNotDispatch: was testing the
denial path with PlanHash:"sha256:test". Now uses the real approval-
equivalent hash so denial fires (not drift) and the test still
isolates the denial behavior.
- ControlledConsumesApprovedCommandWithResolvedRoutingTarget: same fix
pattern.
Extends TestActionExecutionContractStaysAPIOwned in code_standards
test to pin ErrActionPlanDrift's existence in actions.go so future
refactors cannot silently downgrade drift into a generic error kind.
Native action path (executeNativeActionWithAudit) is left for a
follow-up: it has the same drift gap but uses actionPlanHashForParams
which is shaped differently from approvalPlanHash, so a sound fix
needs a coherent canonical hash function rather than just adding the
check. Contracts pinned in ai-runtime and unified-resources (new
Completion Obligation #20).
Extend the patrol_report_finding LLM tool schema with an optional
impact parameter and propagate it through PatrolFindingInput,
patrolFindingCreatorAdapter.CreateFinding, and into Finding.Impact so
LLM-authored Patrol findings carry consequence-if-ignored copy at
detection time alongside the curated catalogs already in place for
runtime failures and threshold alerts.
Updates the patrol system prompt with an "Authoring Impact" section
that instructs the LLM to write concrete operational consequences
(named workloads, jobs, recovery windows) rather than echoing severity
or category, and to leave impact empty rather than fabricate a
consequence when one is genuinely unknown. The eval-pass prompt gets a
shorter version of the same guidance.
Adds two unit tests: one covering an authored impact passing through
to PatrolFindingInput.Impact, and one covering the omitted-impact case
where the contract is honored verbatim with no synthesized default.
Updates the ai-runtime contract to pin the tool-schema authoring rule
and the no-fabrication invariant.
- derive approval requester identity from investigation-fix approvals
- stamp pending action-audit lifecycle events with Patrol as the producer
- document the requester boundary for Patrol handoffs and timelines
Parse the /proc/mdstat operation keyword for mdadm arrays and propagate it through host reports, models, unified resources, monitoring views, alert metadata, and AI storage summaries.
Treat recovery and reshape as rebuild signals while silencing routine check and resync maintenance, with fallback rebuild detection only when no mdstat operation is available.
Tests cover mdstat operation parsing plus recovery, check, and resync alert behavior.
Fixes#1446
Persist approval requests as governed planned actions with context confidence and stream that plan into Pulse Assistant approvals.
Reuse the planned action identity when approved actions execute so the unified audit trail stays continuous.
rcourtman