chat.Service.ExecuteStream was a long-standing cost-ledger gap: the
agentic loop accumulated token counts via stream callbacks (see
GetTotalInputTokens / GetTotalOutputTokens in agentic_control.go)
and surfaced them in the SSE done envelope to the frontend, but
nothing on the server side recorded a cost.UsageEvent. Patrol,
discovery, QuickAnalysis, and the report narrators all record; only
chat — the bulk of AI token spend — did not. The operator's AI
usage dashboard was therefore understating cost dramatically.
Found while extending the cost-recording mindset across subpackages
after fixing QuickAnalysis (08491b9f4). Initially spawned as a
separate task but the right shape and scope became clear, so landing
it directly here.
Pipeline:
- Service.CostStore() exposes the per-tenant cost store handle.
- chat.Config gains optional CostStore *cost.Store field, threaded
into chat.Service.costStore at NewService time.
- chat.Service.recordChatTurnCost records a UsageEvent with
UseCase="chat" after every loop.ExecuteWithTools return (success
OR error — operator was billed regardless of clean response).
Skips when costStore is nil or zero tokens accumulated.
- ai_handler.go's two chatCfg construction sites populate CostStore
via h.resolveCostStore(ctx).
- router wires the resolver to AISettingsHandler.GetAIService(ctx).CostStore()
with no Enabled gate — even brief chat usage while AI was being
configured should appear in the dashboard.
ExecutePatrolStream is deliberately not changed. It creates a
separate tempLoop and its caller (patrol_ai.go) records cost via
its own helper at line 887. Recording in ExecuteStream only avoids
double-counting on the patrol-via-chat path.
Tests in chat/cost_recording_test.go cover: recording when store
configured, no-op when store nil, no-op on zero tokens (early
failures), graceful handling of model strings missing the
provider prefix.
Service.Narrate (b84b87d8d) and Service.NarrateFleet (d4463a615)
fixed missing cost-record calls in the report-narrative path. Auditing
the rest of internal/ai for the same bug class found one more:
Service.QuickAnalysis. It is used for alert auto-resolve and similar
lightweight decisions, so production token spend on auto-resolve
analysis was invisible in the AI usage dashboard.
Mirror the same fix: capture costStore under the read lock alongside
provider/cfg, and after provider.Chat returns, record a UsageEvent
labelled with the request's UseCase (defaulting to "quick_analysis"
when the caller leaves it blank). Recording happens before the
empty-content guard so failed-but-billed calls are still visible.
Adds cost_recording_audit_test.go: an AST-level audit that walks
internal/ai/*.go (excluding _test.go and sub-packages), finds every
function calling .Chat() on a providers.Provider value, and asserts
each function body also references .Record() on a cost store.
Exemption is allowed via a //cost-recording-exempt: <reason> doc
comment. RunPatrolToolPreflight is annotated as exempt — it is a
connectivity self-test, not user workload, and should not pollute
the operator's cost dashboard.
The audit is intentionally local (function-scoped, not
interprocedural). A passthrough wrapper that calls a recording
function rather than calling Record itself would need an explicit
exemption naming the wrapped callee. Keeping the scan local makes
new Chat callers loud rather than letting silent gaps creep in via
indirection.
Future Chat callers must either record cost or carry the exemption
marker. The audit fails CI otherwise, so the regression that shipped
in b2bd9d114 (Narrate) and would have shipped again in d4463a615
(fleet) cannot recur silently.
The Verify Patrol button reset its result to empty on every
page load — the operator had to re-click to see the verified
state, even though nothing had changed. This commit adds the
observability layer of the auto-preflight plan: every
RunPatrolToolPreflight result is now cached on the AI Service
and surfaced through /api/settings/ai as patrol_preflight, so
the inline result panel rehydrates on page load with the
most-recent outcome and a "last verified Xs ago" indicator.
Backend: patrolPreflightCache (mutex-guarded) on Service with
defensive-copy CachedPatrolPreflight() accessor; every
RunPatrolToolPreflight branch (success, soft warning, classified
failure, validation early-return) records into the cache.
PatrolPreflightSnapshot projects the cached result onto the
AI settings response. Tests cover both success-then-failure
supersession and the defensive-copy invariant.
Frontend: PatrolPreflightSnapshot type mirrors the wire shape;
hydratePatrolPreflightFromSettings(data) projects the snapshot
into the same response shape the manual button writes;
loadSettings and updateSettings flows call it. The result
panel renders a "last verified Xs ago" line under the
provider/model row when recorded_at_unix is present.
End-to-end smoke verified against deepseek-v4-flash: panel
rehydrates as green "Tool calling verified · last verified
just now" after page reload.
Auto-preflight on save (the trigger half of the resilience
plan) follows in the next commit.
Contracts: ai-runtime, api-contracts, agent-lifecycle (dep),
storage-recovery (dep), frontend-primitives all updated to
reflect the new patrol_preflight surface and hydration
contract. Verification artifacts: settingsArchitecture +
patrolPreflight client tests.
Before this change, all three dismissal reasons funnelled through the same
"swallow re-detection at same/lower severity" path in FindingsStore.Add, so
will_fix_later was functionally identical to expected_behavior — Pulse stayed
quiet forever despite the dismiss_finding tool literally telling the LLM
"Pulse Patrol will continue to monitor this issue."
Now will_fix_later sets Finding.RemindAt (default 7 days) at dismissal time.
Once RemindAt has passed, the next re-detection clears the dismissal and emits
a `reminded` lifecycle event so the operator sees their lapsed commitment
instead of a swallowed finding. expected_behavior keeps acknowledged-forever
semantics; not_an_issue keeps Suppressed=true. The LLM tool response now
surfaces the remind-at date so Patrol's conversational explanations stay
aligned with the contract.
Patrol runs, evaluation passes, and QuickAnalysis calls were consuming
LLM tokens without recording them in the cost store. This made the
cost_budget_usd_30d budget setting ineffective since enforceBudget()
never saw patrol spend.
- Add RecordUsage() to ai.Service for thread-safe cost recording
- Add recordPatrolUsage() helper to PatrolService, called on both
success and error paths for main patrol and evaluation pass
- Record QuickAnalysis token usage in cost store
- Return partial PatrolResponse (with token counts) on error instead
of nil, so callers can always record consumed tokens
- Propagate partial response through chat_service_adapter on error
- Fix API-only mode to accept Bearer tokens and query params
- Fix data race in API token validation using fine-grained locking
- Fix unified agent download serving wrong binary for invalid arch
- Fix AI infra discovery running when AI disabled and missing stop mechanism
- Update patrol.go to use chat service for AI execution
- Update service.go with chat service provider integration
- Add patrol streaming endpoint to router
rcourtman