Clears the ten dupl pairs across internal/ai:
- patrol_intelligence.go, tools_query.go, tools_storage.go: VM and LXC
system-container paths collapse into generics over read-state view
method subsets (gatherGuestIntelligenceFromViews, canonicalGuestGetResult
+ guestViewGetResult, addCanonicalGuestSearchMatches +
addGuestViewSearchMatches, appendGuestDiskSummaries).
- tools_file.go: append/write share executeFileMutation driven by
fileMutationSpec (approval-command text, shell redirect, verification
strategy stay per-action and verbatim).
- tools_kubernetes.go: deployment restart / pod delete share
executeKubernetesResourceAction driven by kubernetesResourceAction.
- providers/anthropic.go + anthropic_oauth.go: message conversion shared
via convertMessagesToAnthropic (the OAuth copy was annotated 'same as
regular client').
- memory/changes.go + memory/remediation.go: history loading shared via
the generic loadMemoryHistory in memory/paths.go (10 MiB cap, sort,
missing-file semantics preserved via a found flag).
- findings.go and unified/alerts.go: Finding/findingJSON and
UnifiedFinding/unifiedFindingJSON are deliberate marshal-mirror twins
(AlertIdentifier json:"-" vs alert_identifier round-trip); merging
would break every public literal. Suppressed with nolint:dupl and
enforced instead by new reflect-based mirror-sync tests.
Contract Extension Points name the shared helpers and the mirror
invariant. Full ./internal/ai/... test tree passes.
The finding lifecycle was asymmetric around the deterministic-resolve
gate: the gate correctly blocks LLM resolves of event/persistent
findings (backup, reliability, security, general) when the verifier
still detects the signal, but nothing ever cleared such a finding when
the underlying issue WAS fixed — absence-based stale auto-resolve only
covers performance/capacity, so a fixed backup stayed an active finding
indefinitely unless the LLM happened to call patrol_resolve_finding.
reconcileStaleFindings now runs the deterministic verifier for seeded,
unreported event/persistent findings whose key has one, and resolves
ONLY on an affirmative "signal gone" verification. Still-present or
inconclusive results fail closed (same standard as the resolve gate);
verifications are capped per run (3) with deferred candidates logged
and retried next run. Test seam: PatrolService.verifyFixResolvedFn.
Two enabling repairs found during verification:
- hasDeterministicVerifierForKey listed 2 of the 7 keys the
verifyFixDeterministically dispatch handles, so the LLM-resolve gate
silently skipped existing verifiers for backup-stale and
guest-unreachable findings. Now aligned; documented as the single
source of truth for both consumers.
- The finding-key vocabulary was forked: the patrol_report_finding tool
suggested keys (high-cpu, ...) the verifier switch never matched, so
verification rarely engaged for new findings. normalizeFindingKey now
aliases unambiguous directional synonyms onto the canonical verifier
vocabulary (high-cpu -> cpu-high etc.; pbs-job-failed and node-offline
deliberately NOT aliased — different resource models), and the tool
description teaches the canonical keys.
NOT changed: the synthetic ai-patrol-error finding — verification showed
the reported "accumulation" is a non-problem (deterministic ID, store
merges repeats as heartbeats, successful runs auto-resolve it).
Teeth: 8 new tests in patrol_reconcile_test.go including the
idempotence invariant (repeat reconcile over unchanged state produces
zero resolves and zero lifecycle growth) and the cap-defers case.
Contract: verified stale-resolve clause added beside the
deterministic-resolve-gate in ai-runtime.md. Full internal/ai tree green.
buildDiscoveryToolResponse rebuilt each fact as {category,key,value} and dropped
the Source (the command/origin that produced it) and Confidence the DiscoveryFact
already carries — so the model could state facts it couldn't attribute or weight.
Serialize both per fact (omitted when empty) so the Assistant can report "Debian
12, per /etc/os-release" instead of a bare, untraceable claim.
This is the first piece of the provenance/trust groundwork: make the model
*receive* the source metadata it needs to cite. Surfacing it to the user (UI) and
push-path discovery freshness are follow-ups.
- Test: TestBuildDiscoveryToolResponse_IncludesFactProvenance (source+confidence
present when set, omitted when empty).
- Contract: ai-runtime section 8 documents the pulse_discovery fact provenance.
In the trace for 'hows esphome', the model called get with the canonical handle
'system-container-599a2e3...' as resource_id and no resource_type, and it failed
twice with 'resource_type is required' before recovering with 'get 102'. The user
sees those failed tool calls in the chat.
A canonical handle already encodes the type (unifiedresources/ids.go builds ids
as '<type>-<hash>'), so executeGetResource now infers resource_type from the
handle when it's omitted, via resourceTypeFromCanonicalID (the trailing hex hash
segment is unambiguous since no type word is all-hex). A bare numeric VMID still
requires an explicit type. Test TestResourceTypeFromCanonicalID covers it; full
internal/ai/tools green.
User on a cloud model saw 'redacted by policy' everywhere. Root cause: the
default classification (classifyResourceSensitivity) treated every VM, container,
pod, k8s workload, and docker service as 'Sensitive', which redacts their
hostname/IP/alias/path for cloud models. For Pulse's homelab/SMB audience that
crippled the cloud Assistant — a workload named 'grafana' isn't a secret, and its
private LAN IP isn't either.
Recalibrate: compute workloads classify as 'Internal' (cloud-summary, no
redaction) so cloud models can see their names/IPs. Escalation to
Sensitive/Restricted is by tag (database, backup, customer-data, secret, ...) or
by genuinely sensitive TYPE: storage/data-at-rest (storage, PBS, Ceph,
physical-disk, network-share, network, k8s PV/PVC/StorageClass), configuration
(docker-config, k8s-configmap), and security (k8s RBAC, secrets, PMG). Secrets
and PMG stay Restricted; the tag-based escalation is unchanged.
Tests: new TestRefreshPolicyMetadata_PlainComputeWorkloadsAreInternalNotRedacted
+ TestComputeWorkloadPolicyIsInternalUnlessEscalated lock it in. ~13 AI-subsystem
redaction tests that assumed plain compute = Sensitive updated to tag their
fixtures so they still exercise redaction on a genuinely-sensitive resource (no
assertions weakened). Contract: unified-resources Extension Points documents the
recalibrated classification. internal/ai/... + internal/unifiedresources/... green.
FormatForRemediation surfaced config and log paths but not data paths,
while FormatForAIContext (chat) does. For remediation those matter —
backup targets, disk-full triage, restore points (e.g. a database data
dir or HA's /config/.storage). Add a Data Directories section, matching
the chat pack. Extends the remediation test to assert a data path
reaches it; teeth-checked. Full servicediscovery package green.
Dead-code sweep. Functions flagged unreachable by golang.org/x/tools/cmd/deadcode
and confirmed unused across pulse, pulse-enterprise, pulse-pro and pulse-mobile by
adversarial cross-repo verification. Cross-module reachability was checked
explicitly (only pkg/ exported symbols are importable by other modules; internal/
packages and _test.go files are not). go build, go vet and test-compile all pass.
- add a forced run action to pulse_discovery for known resources
- make discovery progress describe model-backed evidence analysis rather than a live Assistant chat
- keep shared select hydration stable for persisted discovery intervals
Small Ollama models (qwen2.5:11b, qwen2.5:14b, similar) frequently emit
Pulse tool invocations as plain JSON inside content instead of routing
through the structured tool_calls channel. Users saw raw payloads like
`{"name": "pulse_query", "parameters": {...}}` as the assistant's final
response.
Extend cleanToolCallArtifacts and containsToolCallMarker with a new pass
that detects this leak shape, gated on a closed allowlist of canonical
tool names sourced from the runtime registry. The allowlist auto-syncs
when registerTools() gains a tool, so no separate hand-maintained list.
Anchored on (?:^|\n) and a leading `"name"` key so prose containing JSON
fragments, unrelated objects (`{"foo":"bar"}`), or named resources
(`{"name":"my-vm","cpu":50}`) are left untouched.
The reporting feature now ships across two surfaces (PDF/CSV export
and pulse_summarize chat tool) and three modes (single-resource,
fleet, summarize). Without usage telemetry we can't tell whether the
work earns its place — operator demand, AI-vs-heuristic adoption,
range/format preferences are all invisible. Stops further feature
investment from being pure speculation.
Three new info-level log events, structured so an agent can grep
transcripts and group by dimension without a separate metrics
pipeline (matches the "agent owns ops analysis, human gets outcomes"
posture in MEMORY.md):
reporting.single.generated — single-resource PDF/CSV
reporting.fleet.generated — multi-resource fleet PDF/CSV
reporting.summarize.invoked — pulse_summarize chat tool (both modes)
Common dimensions: org_id, format/action, range, ai_configured,
findings_configured, window_start/end. Single-resource adds
resource_type + metric_type + bytes; fleet adds resource_count +
bytes; summarize adds resource_type + resource_count (fleet mode) +
narrative_source (so we can audit AI-fallback rate).
Includes rangeLabel() helper that maps a window to the canonical
catalog range token (24h/7d/30d) with a 1h tolerance, falling back
to "<hours>h" so non-standard windows still group. Tested.
TestReportingTelemetryEventNames pins the canonical event names as
a contract — an agent grepping logs depends on them being stable;
changing them silently would break audit tooling on the consumer
side.
The reporting engine already logs the resolved narrative source
(heuristic/ai) at debug level via the existing "Generating report"
line, useful for diagnosing why a specific report fell back. Kept
at debug; the new info-level events cover the operator surface.
v1 of pulse_summarize (1fe5d6853) shipped with heuristic narrative
only. The follow-up wiring promised in that commit now lands: the
chat session carries optional report-narration providers that the
tool's handler reads when building requests, so AI-narrated synthesis
flows into chat using the same provider, sanitizer, model selection,
cost ledger, and budget gate the report PDF endpoint already uses.
Pipeline:
- pkg/reporting Narrator / FleetNarrator / FindingsProvider interfaces
are already implemented by internal/ai.Service. No new
implementations.
- tools.ExecutorConfig + PulseToolExecutor gain three optional fields
(ReportNarrator, ReportFleetNarrator, ReportFindingsProvider).
Clone() copies them so per-session executors inherit the wiring.
- chat.Config gains the same three fields; NewService threads them
into ExecutorConfig.
- tools_summarize.go reads e.reportNarrator/FleetNarrator/
FindingsProvider and populates MetricReportRequest /
MultiReportRequest. The engine already accepts these on the request
and falls back to heuristic when they are nil — no engine changes
needed.
- AIHandler gains SetReportNarratorResolver(ctx -> narrators); both
per-tenant and default chat.Config construction sites invoke the
resolver. Router wires the resolver to AISettingsHandler.GetAIService
with the same Enabled-gate the reporting handler uses.
Unconfigured tenants are unchanged: the resolver returns nil, the
tool returns heuristic narrative — identical to today. Configured
tenants get AI synthesis in chat that matches what their report PDF
already carries, billed and budget-gated the same way.
The reporting synthesis layer (observations, recommendations,
outliers, period comparison) shipped trapped behind the PDF/CSV
export. Operators who chat with Assistant could not ask "what's been
happening with pve1 this week" — the data path existed but had no
non-PDF surface. This commit adds a single new tool, pulse_summarize,
that wraps the engine's non-rendering entry points (NarrativeFor /
FleetNarrativeFor) so that question gets answered in chat.
The tool takes an action parameter (resource | fleet) and routes
accordingly:
- resource mode requires resource_type + resource_id and returns the
same Narrative the single-resource report carries (health status,
observations, recommendations, period comparison).
- fleet mode requires resource_type + a comma-separated resource_ids
string (PropertySchema does not currently support array items, and
CSV is LLM-friendly enough) and returns the FleetNarrative
(outliers, patterns, recommendations). Capped at the same
multi-report ceiling (50) as the API endpoint.
The tool is read-only — no control level requirement, no approval
gate — and uses the global reporting engine the rest of the app
already shares. Returns a JSON envelope so chat can render it or
hand it back to the model for follow-up framing.
v1 ships with heuristic narrative only. The AI narrator wiring
through the chat session (Narrator/FleetNarrator/FindingsProvider
threaded via chat.Config -> tools.ExecutorConfig -> PulseToolExecutor)
is a focused follow-up; it lets the same tool inherit the per-tenant
AI service the report PDF endpoint already uses. The seam is
already in place because NarrativeFor/FleetNarrativeFor take an
optional narrator on the request — v1 passes nil, v2 populates it.
Closes the certainty loop for agents watching the substrate's push
channel. The action audit's read-after-write probe outcome was
already persisted on the audit record, but agents watching
action.completed only learned "the action ran" — they had to fetch
/api/actions/{id} to know whether the read-back probe confirmed
the intended state. That defeated the substrate's
push-notification guarantee for dispatch certainty.
The new agent-stable AgentResourceActionVerification projection
(ran, success, command, note, ranAt — output stays in the audit
record, deliberately omitted from events to keep payloads small)
is now carried on both:
- the action.completed SSE payload, projected from
record.Result.Verification by the router-side bridge in
wireAIChatDependenciesForService, and
- the resource-context bundle's recentActions surface, via the
same shared projectAgentResourceVerification helper
so the bundle (depth) and the doorbell (push) speak the same
vocabulary. Refused-before-dispatch failures omit verification
(the probe never runs) so agents branch on field presence to
distinguish "no probe attempted" from "probe ran with empty
result". Three contract pins lock the symmetry: payload field
present, router bridge populates it, bundle parallels.
The capabilities manifest's subscribe_events description now
mentions the verification block so external agents discover the
field through the same path they already use to learn the rest
of the agent surface.
The "+N" overflow indicator on TagBadges was styled with
`cursor-pointer`, which signals a clickable affordance — but the
element only listens for mouseenter/mouseleave to show a tooltip and
has no click handler. Switch to `cursor-help` so the cursor matches
the actual interaction (hover for more info), avoiding a phantom
click expectation.
Third compounding slice on per-resource operator state, completing the
suppression triangle: maintenance window (slice 31), intentionally
offline (slice 32), and now NeverAutoRemediate at the action-broker
boundary. When the operator has set NeverAutoRemediate=true on a
resource, executeCommandWithAudit refuses the dispatch even with a
valid approval and matching plan hash — per-resource operator intent
outranks per-action approval.
The check sits next to the drift refusal in the same audit hot path:
fail-open on store-lookup error (logged but doesn't block), persist a
Failed audit record with `resource_remediation_locked:` prefix on the
ErrorMessage so audit-UI filters and alert rules can branch on the
stable token. Returns ErrResourceRemediationLocked sentinel for
caller-side branching.
Reuses the existing actionAuditStore handle (which is the unified
ResourceStore from slice 29), so no new provider plumbing needed —
the broker already has the data path it needs.
Pulse's primary monitored platform is Proxmox, but the per-command-class
preflight catalog only covered systemd services, Docker containers, and
Kubernetes deployments. Operators approving a Patrol-proposed
qm restart/qm stop/qm shutdown saw the generic preflight without the
operational nuance that distinguishes those verbs (qm stop is a hard halt,
qm shutdown is graceful with a 60s ACPI timeout, etc.).
Adds eight new classes covering qm and pct lifecycle (reboot/restart,
stop, start, shutdown). Each gets hand-authored safety and verification
copy that names the actual semantics — pct stop is destructive vs pct
shutdown's lxc-attach handoff — so the operator sees concrete context at
approval time.
Broker-level VerificationCommandForCommand intentionally does not derive
qm status / pct status from these classes — pulse_control's
verifyGuestAction already runs those checks at the tool layer, so adding
a parallel broker dispatch would double-run. The preflight copy still
names what the tool-layer verification will read.
Closes the read-after-write piece of the user's act-with-control loop:
"Act with control: ask for approval, execute through Pulse tools, then
verify." Until now Pulse authored verification narrative copy at
approval time but never actually executed the check.
Adds ActionVerificationResult to internal/unifiedresources/actions.go
and embeds it as ExecutionResult.Verification so the existing
result_json column persists the outcome without a schema migration.
The struct records: did the check run, what command did Pulse send,
what came back, did it succeed, when, and any failure note.
Adds VerificationCommandForCommand in tools_control.go that derives
the per-class read-after-write check used by the broker:
- service-restart / service-start / service-reload / service-stop
→ systemctl is-active <unit>
Container classes (container-restart, container-stop) are
intentionally deferred to pulse_docker's existing tool-level
docker inspect verification — adding a broker-level dispatch would
double-run the same check.
executeCommandWithAudit now runs the derived verification command via
the same agent path immediately after a successful dispatch, captures
output and exit code, and writes the result onto
ExecutionResult.Verification before recordActionExecutionResult
persists. If the check returns non-zero, Verification.Success=false
with a Note explaining the exit code so the audit history honestly
shows that Pulse ran the action but couldn't confirm it took.
Verification is best-effort and class-scoped: unknown command shapes
leave Verification nil rather than fabricating a verified=true entry,
matching the no-fabrication boundary from Impact / preflight authoring.
Tests cover:
- VerificationCommandForCommand derives the right check per class,
shell-escapes single quotes in unit names, and returns "", false
for docker (deferred) and unknown commands.
- TestExecuteCommandWithAuditRunsClassDerivedVerificationAfterDispatch
- TestExecuteCommandWithAuditMarksVerificationFailedWhenReadbackDoesNotConfirm
- TestExecuteCommandWithAuditSkipsVerificationForUnclassifiedCommands
Updates ai-runtime contract with the read-after-write rule and
container-class deferral. Adds Completion Obligation #21 to
unified-resources pinning ExecutionResult.Verification shape and the
no-fabrication boundary. Extends TestActionExecutionContractStaysAPIOwned
to pin ActionVerificationResult struct and the Verification field on
ExecutionResult.
Operators about to approve a Pulse-driven action need to know what the
command actually touches before saying yes. Until now the preflight
shown at approval time was the same generic boilerplate for every
action: "approval is scoped, hash must match, etc." — useful safety
posture but no operational specifics. The dry-run summary said the
same thing for 99% of actions.
Adds classifyApprovalCommand and approvalCommandClassPreflightAdditions
in tools_control.go that bucket common Pulse remediation actions and
return hand-authored safety + verification additions per class:
- service-restart (systemctl restart, service restart)
- service-stop (systemctl stop, service stop)
- service-start (systemctl start, service start)
- service-reload (systemctl reload)
- container-restart (docker / podman restart)
- container-stop (docker / podman stop)
- k8s-rollout-restart (kubectl rollout restart)
The additions name concrete operational facts the operator needs:
- For service-restart: "Service will be briefly unavailable; no other
unit dependencies altered" plus verification "Read back
systemctl is-active <unit>" and "tail journal for crash patterns".
- For container-restart: "Container will be briefly unavailable;
image and volume mounts unchanged" plus "Read back via docker
inspect" verification.
- For k8s-rollout-restart: "Pods rolled in waves per deployment
strategy; PodDisruptionBudget continues to apply" plus "watch
kubectl rollout status" verification.
approvalPreflight now appends class-specific safety and verification
content onto the existing default content rather than replacing it,
so the broker's structural safety posture (org scope, hash match,
single-use approval) remains visible alongside the operational copy.
Unknown command classes return nil/nil — no fabricated padding for
commands the bucket does not recognize. The default preflight stands
on its own, matching the no-fabrication rule that runs through the
rest of the trust-record arc.
Tests cover: each known class returns non-empty additions with the
expected operational tokens; unknown commands return nil; the
end-to-end approvalPreflight merge surfaces both default and
class-specific safety/verification entries. ai-runtime contract
pinned with the per-class enrichment rule and the no-fabrication
boundary for unknown classes.
Slice 13 added the drift refusal path but only logged at WARN level —
the audit history showed nothing, so an operator reviewing the action
trail could not see "Pulse caught this drift attempt." Now the drift
branch writes a Failed audit record with Result.ErrorMessage prefixed
"plan_drift:" and dispatches a Failed lifecycle event before returning
ErrActionPlanDrift.
The record carries the same Request, Plan, and Approvals snapshots that
a normal audit record would, so the operator-facing audit row shows
exactly what was attempted and what was approved. The "plan_drift:"
prefix is a stable token for downstream surfaces (audit UI filters,
alert rules) to distinguish drift refusals from generic execution
failures.
Extends TestExecuteCommandWithAuditRefusesPayloadDriftAgainstApprovedPlan
to assert the audit record exists with State=Failed and the plan_drift
error message after refusal. Updates the code-standards snippet check
for ErrActionPlanDrift to match gofmt's actual alignment in actions.go.
ai-runtime contract pinned with the audit-record-on-drift rule.
PlanHash existed on ActionPlan as the contract for "the operator
approved exactly this (command, target, reason) combination" but the
broker never validated it: at execute time the freshly-recomputed hash
was overwritten by the approved plan's hash via mergeApprovedActionPlan
without comparison. A drifted payload (LLM re-emits with different
command, agent ID changes between approval and execute, malicious
injection) would run under a stale approval.
Fix: at the dispatch boundary in executeCommandWithAudit, recompute the
approval-equivalent hash from the actual payload using approvalPlanHash
(same function used at approval-creation time, so direct comparison is
meaningful), compare to plan.PlanHash, and refuse with a new
ErrActionPlanDrift error when they differ. The drift refusal also logs
at WARN level with action_id, approval_id, and both hashes so audit
review can see when drift was caught.
When approvedHash is empty (older approval records or contract paths
that did not author one), validation is skipped and existing behavior
is preserved.
Adds two tests:
- TestExecuteCommandWithAuditRefusesPayloadDriftAgainstApprovedPlan:
approval is for "systemctl restart workload"; payload at execute
time is "rm -rf /var/log/pulse"; expects ErrActionPlanDrift and no
agent dispatch.
- TestExecuteCommandWithAuditAllowsMatchingPlanHash: same payload
matches the approved hash; expects normal dispatch.
Updates two pre-existing tests that used stub PlanHash strings:
- TestExecuteCommandWithDeniedApprovalDoesNotDispatch: was testing the
denial path with PlanHash:"sha256:test". Now uses the real approval-
equivalent hash so denial fires (not drift) and the test still
isolates the denial behavior.
- ControlledConsumesApprovedCommandWithResolvedRoutingTarget: same fix
pattern.
Extends TestActionExecutionContractStaysAPIOwned in code_standards
test to pin ErrActionPlanDrift's existence in actions.go so future
refactors cannot silently downgrade drift into a generic error kind.
Native action path (executeNativeActionWithAudit) is left for a
follow-up: it has the same drift gap but uses actionPlanHashForParams
which is shaped differently from approvalPlanHash, so a sound fix
needs a coherent canonical hash function rather than just adding the
check. Contracts pinned in ai-runtime and unified-resources (new
Completion Obligation #20).
Extend the patrol_report_finding LLM tool schema with an optional
impact parameter and propagate it through PatrolFindingInput,
patrolFindingCreatorAdapter.CreateFinding, and into Finding.Impact so
LLM-authored Patrol findings carry consequence-if-ignored copy at
detection time alongside the curated catalogs already in place for
runtime failures and threshold alerts.
Updates the patrol system prompt with an "Authoring Impact" section
that instructs the LLM to write concrete operational consequences
(named workloads, jobs, recovery windows) rather than echoing severity
or category, and to leave impact empty rather than fabricate a
consequence when one is genuinely unknown. The eval-pass prompt gets a
shorter version of the same guidance.
Adds two unit tests: one covering an authored impact passing through
to PatrolFindingInput.Impact, and one covering the omitted-impact case
where the contract is honored verbatim with no synthesized default.
Updates the ai-runtime contract to pin the tool-schema authoring rule
and the no-fabrication invariant.
- derive approval requester identity from investigation-fix approvals
- stamp pending action-audit lifecycle events with Patrol as the producer
- document the requester boundary for Patrol handoffs and timelines
Parse the /proc/mdstat operation keyword for mdadm arrays and propagate it through host reports, models, unified resources, monitoring views, alert metadata, and AI storage summaries.
Treat recovery and reshape as rebuild signals while silencing routine check and resync maintenance, with fallback rebuild detection only when no mdstat operation is available.
Tests cover mdstat operation parsing plus recovery, check, and resync alert behavior.
Fixes#1446
Persist approval requests as governed planned actions with context confidence and stream that plan into Pulse Assistant approvals.
Reuse the planned action identity when approved actions execute so the unified audit trail stays continuous.
rcourtman