Surface and self-heal Patrol's provider-unavailable state

Field telemetry showed installs with Patrol enabled recording weeks of
empty error runs (runs_30d=122, ai_calls=0, findings=0): provider
initialization failed once at boot (model resolution can need the
provider's live catalog, so Pulse racing a booting Ollama server loses)
and was never retried, while the run loop kept recording "Patrol
provider not configured" errors that told operators who had configured
a provider to configure one.

- Retry provider initialization on every scheduled run, so a boot-time
  race strands Patrol for at most one interval instead of until the
  next settings save. LoadConfig records the redacted init failure.
- Name the real failure in the blocked reason and run record when the
  configured provider failed to initialize, instead of claiming no
  provider is configured.
- Raise the deduped Patrol runtime finding when scheduled runs are
  skipped by a persistent readiness blocker or missing provider, so the
  state reaches the findings surfaces and alert notification channels
  once, instead of living only on the Patrol page banner. Transient
  circuit-breaker blocks stay finding-free; the attempts that opened
  the breaker already raised their own.
- Resolve the runtime finding when Patrol is turned off; opting out is
  a resolution, not a state to keep nagging about.
- Record the extended runtime-failure surface in the ai-runtime
  subsystem contract.
This commit is contained in:
rcourtman
2026-08-21 21:05:54 +01:00
parent 9ef94418e8
commit 7c37a85cb2
6 changed files with 373 additions and 41 deletions
@@ -6394,6 +6394,12 @@ summary.
When missing provider configuration blocks Patrol, `blocked_reason` must point
to Pulse Intelligence > Provider & Models settings and tool-capable Patrol model
selection.
When a provider is configured but its initialization failed — model resolution
can require the provider's live catalog, so Pulse booting before a local model
server loses that race — `blocked_reason` must name the initialization failure
instead of claiming no provider is configured, and the runtime must retry
provider initialization on each scheduled run so a boot-time race strands
Patrol for at most one interval rather than until the next settings save.
That runtime-state contract must be derived from live Patrol runtime inputs,
not only from the last failed run attempt, and the backend must clear any stale
managed-credit block once a provider or local model configuration returns.
@@ -6494,6 +6500,18 @@ snooze, dismiss, resolve, and suppress actions against synthetic `ai-service`
runtime findings. The canonical recovery path is to correct Patrol provider
configuration in Pulse Intelligence > Provider & Models settings and let Patrol
re-evaluate the runtime condition on the next run.
Runs Patrol never attempts are part of that same finding surface. A scheduled
or scoped run skipped by a persistent readiness blocker or an unavailable
provider must raise the deduped synthetic runtime finding, so an enabled-but-
inert Patrol reaches the shared findings surfaces and alert notification
channels once instead of living only on the Patrol page banner; field
telemetry showed installs recording weeks of empty error runs before anyone
noticed. A transiently open circuit breaker must stay finding-free, because
the attempts that opened it already raised their classified finding.
Turning Patrol off is a Patrol-owned resolution of that synthetic runtime
finding: the disable transition must resolve it rather than leave an
unactionable warning nagging about a state the operator just opted out of,
and infrastructure findings are untouched by that transition.
The shared findings lifecycle must also treat a regressed issue as a new active
occurrence. When a resolved finding reappears, `internal/ai/findings.go` must
clear any stale acknowledgement timestamp from the prior occurrence instead of
@@ -0,0 +1,164 @@
package ai
// Tests for the blocked-run runtime finding and provider-init retry. A Patrol
// that is enabled but cannot use a provider must surface one deduped,
// self-resolving finding on the shared findings surfaces instead of skipping
// runs with no evidence outside the Patrol page: field telemetry showed
// installs recording weeks of empty error runs (provider init failed at boot
// and was never retried) before anyone noticed.
import (
"context"
"strings"
"testing"
"github.com/rcourtman/pulse-go-rewrite/internal/config"
"github.com/rcourtman/pulse-go-rewrite/internal/models"
)
func patrolRuntimeFindingIDForTest() string {
return generateFindingID(patrolRuntimeResourceID, "reliability", patrolRuntimeFindingKey)
}
func blockedRunStateProvider() *mockStateProvider {
return &mockStateProvider{
state: models.StateSnapshot{
Nodes: []models.Node{
{ID: "node1", Name: "node1", Status: "online"},
},
},
}
}
func TestPatrol_BlockedScheduledRun_RaisesDedupedRuntimeFinding(t *testing.T) {
ps := NewPatrolService(nil, blockedRunStateProvider())
notified := 0
ps.SetFindingNotifyCallback(func(f *Finding) { notified++ })
ps.runPatrol(context.Background())
ps.runPatrol(context.Background())
stored := ps.findings.Get(patrolRuntimeFindingIDForTest())
if stored == nil {
t.Fatal("expected blocked scheduled run to raise the runtime finding")
}
if stored.IsResolved() {
t.Fatal("runtime finding should stay active while runs remain blocked")
}
if stored.FailureCause != string(PatrolFailureCauseProviderNotConfigured) {
t.Fatalf("failure cause = %q, want %q", stored.FailureCause, PatrolFailureCauseProviderNotConfigured)
}
if notified != 1 {
t.Fatalf("finding notified %d times across two blocked runs, want exactly once", notified)
}
}
func TestPatrol_ReadinessBlockedRun_RaisesRuntimeFindingWithoutRunRecord(t *testing.T) {
ps := NewPatrolService(nil, blockedRunStateProvider())
cfg := DefaultPatrolConfig()
cfg.RuntimeBlockedReason = "No AI provider is configured yet. Add a provider API key or an Ollama server on the Provider & Models settings page."
cfg.RuntimeBlockedCause = PatrolFailureCauseProviderNotConfigured
ps.SetConfig(cfg)
ps.runPatrol(context.Background())
if got := len(ps.GetRunHistory(10)); got != 0 {
t.Fatalf("readiness-blocked scheduled run recorded %d run(s), want none", got)
}
stored := ps.findings.Get(patrolRuntimeFindingIDForTest())
if stored == nil {
t.Fatal("expected readiness-blocked run to raise the runtime finding")
}
if !strings.Contains(stored.Description, "No AI provider is configured yet") {
t.Fatalf("finding description %q should carry the readiness reason", stored.Description)
}
}
func TestPatrol_ProviderInitFailure_UsesHonestBlockedReason(t *testing.T) {
svc := NewService(nil, nil)
// Configured (Ollama base URL) but the explicit model routes to a provider
// with no credentials, so provider init fails deterministically offline.
svc.cfg = &config.AIConfig{
Enabled: true,
OllamaBaseURL: "http://127.0.0.1:1",
Model: "anthropic:claude-model",
}
ps := NewPatrolService(svc, blockedRunStateProvider())
ps.runPatrol(context.Background())
status := ps.GetStatus()
if !strings.Contains(status.BlockedReason, "failed to initialize") {
t.Fatalf("blocked reason %q should name the provider init failure, not claim no provider is configured", status.BlockedReason)
}
if status.BlockedCause != PatrolFailureCauseProviderConnection {
t.Fatalf("blocked cause = %q, want %q", status.BlockedCause, PatrolFailureCauseProviderConnection)
}
stored := ps.findings.Get(patrolRuntimeFindingIDForTest())
if stored == nil {
t.Fatal("expected provider init failure to raise the runtime finding")
}
if !strings.Contains(stored.Description, "failed to initialize") {
t.Fatalf("finding description %q should name the provider init failure", stored.Description)
}
}
func TestService_RetryProviderInit(t *testing.T) {
ctx := context.Background()
unconfigured := NewService(nil, nil)
unconfigured.cfg = &config.AIConfig{}
if unconfigured.RetryProviderInit(ctx) {
t.Fatal("retry must fail while the config has no enabled, configured provider")
}
// A keyless OpenAI-compatible custom endpoint constructs offline.
recoverable := NewService(nil, nil)
recoverable.cfg = &config.AIConfig{
Enabled: true,
OpenAIBaseURL: "http://127.0.0.1:9/v1",
Model: "openai:test-model",
}
if !recoverable.RetryProviderInit(ctx) {
t.Fatal("retry should build a provider for a constructible config")
}
if !recoverable.IsEnabled() {
t.Fatal("service should report enabled after a successful retry")
}
if got := recoverable.ProviderInitError(); got != "" {
t.Fatalf("provider init error = %q after successful retry, want empty", got)
}
failing := NewService(nil, nil)
failing.cfg = &config.AIConfig{
Enabled: true,
OllamaBaseURL: "http://127.0.0.1:1",
Model: "anthropic:claude-model",
}
if failing.RetryProviderInit(ctx) {
t.Fatal("retry must fail when the selected model's provider has no credentials")
}
if failing.ProviderInitError() == "" {
t.Fatal("failed retry should record the provider init error")
}
}
func TestPatrol_DisablingPatrolResolvesRuntimeFinding(t *testing.T) {
ps := NewPatrolService(nil, blockedRunStateProvider())
ps.runPatrol(context.Background())
if stored := ps.findings.Get(patrolRuntimeFindingIDForTest()); stored == nil || stored.IsResolved() {
t.Fatal("expected an active runtime finding before disabling Patrol")
}
cfg := DefaultPatrolConfig()
cfg.Enabled = false
ps.SetConfig(cfg)
stored := ps.findings.Get(patrolRuntimeFindingIDForTest())
if stored == nil || !stored.IsResolved() {
t.Fatal("disabling Patrol should resolve the runtime finding instead of leaving it to nag")
}
}
+9
View File
@@ -337,6 +337,7 @@ func (p *PatrolService) SetConfig(cfg PatrolConfig) {
oldInterval := p.config.GetInterval()
oldBlockedReason := strings.TrimSpace(p.config.RuntimeBlockedReason)
oldBlockedCause := p.config.RuntimeBlockedCause
wasEnabled := p.config.Enabled
p.config = cfg
newInterval := cfg.GetInterval()
configCh := p.configChanged
@@ -353,6 +354,14 @@ func (p *PatrolService) SetConfig(cfg PatrolConfig) {
}
p.mu.Unlock()
// Turning Patrol off is a deliberate resolution of "Patrol cannot run":
// leaving the runtime finding active would nag about a state the operator
// just opted out of. Infrastructure findings are untouched — they describe
// the estate, not the Patrol runtime.
if wasEnabled && !cfg.Enabled {
p.resolvePatrolRuntimeFailureFinding("patrol_disabled")
}
// Signal config change if patrol is running and interval changed
if configCh != nil && newInterval != oldInterval {
select {
+13 -6
View File
@@ -385,6 +385,7 @@ func (p *PatrolService) runPatrolWithTriggerStart(ctx context.Context, trigger T
p.endRun()
}
p.setBlockedReasonWithCause(reason, cfg.RuntimeBlockedCause)
p.raiseBlockedRunFinding(reason, cfg.RuntimeBlockedCause)
log.Info().Str("reason", reason).Str("cause", string(cfg.RuntimeBlockedCause)).Msg("AI Patrol: Skipping run - runtime readiness blocked")
return
}
@@ -567,7 +568,7 @@ func (p *PatrolService) runPatrolWithTriggerStart(ctx context.Context, trigger T
// Acquire a breaker probe only when this run is actually ready to call the
// provider. Early collection/scope exits must not strand a half-open probe.
aiServiceEnabled := p.aiService != nil && p.aiService.IsEnabled()
aiServiceEnabled := p.retryProviderInitIfNeeded(ctx)
llmAllowed := true
if aiServiceEnabled && breaker != nil {
llmAllowed = breaker.Allow()
@@ -579,12 +580,17 @@ func (p *PatrolService) runPatrolWithTriggerStart(ctx context.Context, trigger T
// Check if we can run LLM analysis (AI-only patrol)
if !canRunLLM {
reason := patrolProviderNotConfiguredReason
cause := PatrolFailureCauseProviderNotConfigured
reason, cause := p.providerUnavailableReason()
if aiServiceEnabled && !llmAllowed {
reason = "circuit breaker is open"
cause = PatrolFailureCauseCircuitOpen
GetPatrolMetrics().RecordCircuitBlock()
} else {
// A run skipped for a persistent provider state (never attempted)
// surfaces as the deduped runtime finding; a transiently open
// breaker does not, because the attempts that opened it already
// raised their own specific finding.
trackFinding(newPatrolRuntimeFailureFinding(patrolBlockedRunFailure(reason, cause), time.Now()))
}
p.setBlockedReasonWithCause(reason, cause)
log.Info().Str("reason", reason).Str("cause", string(cause)).Msg("AI Patrol: Skipping run - AI unavailable")
@@ -1076,7 +1082,7 @@ func (p *PatrolService) runScopedPatrolWithStart(ctx context.Context, scope Patr
// Acquire a breaker probe only after the requested scope has resolved to
// real work. This keeps collection-only failures outside provider health.
aiServiceEnabled := p.aiService != nil && p.aiService.IsEnabled()
aiServiceEnabled := p.retryProviderInitIfNeeded(ctx)
llmAllowed := true
if aiServiceEnabled && breaker != nil {
llmAllowed = breaker.Allow()
@@ -1087,12 +1093,13 @@ func (p *PatrolService) runScopedPatrolWithStart(ctx context.Context, scope Patr
canRunLLM := aiServiceEnabled && llmAllowed
if !canRunLLM {
reason := patrolProviderNotConfiguredReason
cause := PatrolFailureCauseProviderNotConfigured
reason, cause := p.providerUnavailableReason()
if aiServiceEnabled && !llmAllowed {
reason = "circuit breaker is open"
cause = PatrolFailureCauseCircuitOpen
GetPatrolMetrics().RecordCircuitBlock()
} else {
p.raiseBlockedRunFinding(reason, cause)
}
p.setBlockedReasonWithCause(reason, cause)
log.Info().Str("reason", reason).Str("cause", string(cause)).Msg("AI Patrol: Skipping scoped run - AI unavailable")
+71 -1
View File
@@ -431,6 +431,76 @@ func summarizePatrolRuntimeFailureDetail(raw string, cancelled bool) string {
}
}
// patrolBlockedRunFailure describes a scheduled run Patrol skipped before any
// provider attempt — a persistent readiness blocker (no provider configured,
// no usable Patrol model) or a provider that failed to initialize. Raising it
// as the deduped runtime finding gives the operator one nudge on the surfaces
// they actually watch (findings list, attention badge, alert notification
// channels); the Patrol page banner alone left field installs blocked for
// weeks without anyone noticing. reason must already be operator-honest: it
// becomes the finding description.
func patrolBlockedRunFailure(reason string, cause PatrolFailureCause) patrolRuntimeFailure {
description := strings.TrimSpace(redactPatrolRuntimeFailureDetail(reason))
if description == "" {
description = "Patrol is enabled and scheduled, but its AI runtime is not ready, so analysis runs are being skipped."
}
if cause == "" {
cause = PatrolFailureCauseProviderNotConfigured
}
return patrolRuntimeFailure{
Title: "Pulse Patrol: Runs are being skipped",
Summary: "Patrol is enabled but runs are being skipped",
Cause: cause,
Description: description,
Impact: patrolRuntimeFailureImpact,
Recommendation: "Open the Provider & Models settings page and complete the provider setup, or turn Patrol off if you do not plan to use AI analysis.",
}
}
// retryProviderInitIfNeeded reports whether the AI service can run LLM
// analysis, first retrying provider initialization when the config claims a
// configured provider but the boot-time build failed. Provider construction
// can depend on a live model-catalog call, so Pulse starting before a local
// Ollama server must strand Patrol for at most one interval, not until the
// next settings save.
func (p *PatrolService) retryProviderInitIfNeeded(ctx context.Context) bool {
if p == nil || p.aiService == nil {
return false
}
if p.aiService.IsEnabled() {
return true
}
if p.aiService.RetryProviderInit(ctx) {
log.Info().Msg("AI Patrol: Provider initialization recovered on scheduled retry")
return true
}
return false
}
// providerUnavailableReason returns the operator-facing reason Patrol cannot
// use a provider right now. When the config claims a configured provider but
// initialization failed, the reason names that failure instead of telling an
// operator who has configured a provider to configure one.
func (p *PatrolService) providerUnavailableReason() (string, PatrolFailureCause) {
if p != nil && p.aiService != nil {
if initErr := strings.TrimSpace(p.aiService.ProviderInitError()); initErr != "" {
return fmt.Sprintf("The configured AI provider failed to initialize, so Patrol runs are being skipped. Pulse retries on every scheduled run. Provider error: %s", initErr), PatrolFailureCauseProviderConnection
}
}
return patrolProviderNotConfiguredReason, PatrolFailureCauseProviderNotConfigured
}
// raiseBlockedRunFinding records the deduped runtime finding for a scheduled
// run skipped by a persistent readiness blocker. The stable runtime finding ID
// means repeated blocked ticks notify at most once, and the finding
// auto-resolves on the next successful provider-backed run.
func (p *PatrolService) raiseBlockedRunFinding(reason string, cause PatrolFailureCause) {
if p == nil || p.findings == nil {
return
}
p.recordFinding(newPatrolRuntimeFailureFinding(patrolBlockedRunFailure(reason, cause), time.Now()))
}
func newPatrolRuntimeFailureFinding(failure patrolRuntimeFailure, now time.Time) *Finding {
return &Finding{
ID: generateFindingID(patrolRuntimeResourceID, "reliability", patrolRuntimeFindingKey),
@@ -464,6 +534,6 @@ func (p *PatrolService) resolvePatrolRuntimeFailureFinding(reason string) bool {
if resolver := p.unifiedFindingResolver; resolver != nil {
resolver(errorFindingID)
}
log.Info().Str("reason", reason).Msg("AI Patrol: Auto-resolved previous patrol runtime finding after successful provider-backed run")
log.Info().Str("reason", reason).Msg("AI Patrol: Resolved patrol runtime finding")
return true
}
+98 -34
View File
@@ -235,10 +235,16 @@ type PatrolStreamResponse struct {
// Service orchestrates AI interactions
type Service struct {
mu sync.RWMutex
orgID string
persistence *config.ConfigPersistence
provider providers.Provider
mu sync.RWMutex
orgID string
persistence *config.ConfigPersistence
provider providers.Provider
// providerInitErr records why the default provider could not be built even
// though the config claims an enabled, configured provider (for example a
// live model-catalog resolution against an Ollama server that was still
// booting). Patrol readiness reports it instead of the misleading
// "provider not configured" copy, and RetryProviderInit clears it.
providerInitErr string
cfg *config.AIConfig
agentServer AgentServer
policy CommandPolicy
@@ -1745,44 +1751,18 @@ func (s *Service) LoadConfig() error {
if err != nil {
return fmt.Errorf("failed to load Pulse Assistant config: %w", err)
}
modelResolutionCtx := context.Background()
var providerClient providers.Provider
var providerInitErr string
// Don't initialize provider if AI is not enabled or not configured
if cfg == nil || !cfg.Enabled || !cfg.IsConfigured() {
} else {
selectedModel, resolveErr := ResolveConfiguredModel(modelResolutionCtx, cfg)
if resolveErr != nil {
log.Warn().Err(resolveErr).Str("orgID", orgID).Msg("AI enabled but no effective provider model could be resolved")
} else {
cfg.Model = selectedModel
selectedProvider, _ := config.ParseModelString(selectedModel)
nextProvider, providerErr := providers.NewForModel(cfg, selectedModel)
if providerErr != nil {
log.Warn().
Err(providerErr).
Str("selected_model", selectedModel).
Str("selected_provider", selectedProvider).
Strs("configured_providers", cfg.GetConfiguredProviders()).
Msg("AI enabled but selected provider could not be initialized")
}
providerClient = nextProvider
if providerClient != nil {
log.Info().
Str("provider", selectedProvider).
Str("model", selectedModel).
Str("control_level", cfg.GetControlLevel()).
Bool("autonomous", cfg.IsAutonomous()).
Msg("AI service initialized")
}
}
if cfg != nil && cfg.Enabled && cfg.IsConfigured() {
providerClient, providerInitErr = buildConfiguredProvider(context.Background(), cfg, orgID, true)
}
s.mu.Lock()
s.cfg = cfg
s.provider = providerClient
s.providerInitErr = providerInitErr
s.initInfraDiscoveryServiceLocked()
s.initDiscoveryServiceLocked()
s.mu.Unlock()
@@ -1793,6 +1773,90 @@ func (s *Service) LoadConfig() error {
return nil
}
// buildConfiguredProvider resolves the effective model and constructs the
// default provider client for an enabled, configured AI config. It returns a
// nil provider plus a redacted failure summary when either step fails — for
// example when model resolution needs the provider's live catalog and that
// server is still booting. Callers own storing the result under s.mu.
// stampModel controls whether the resolved model is written back to cfg.Model:
// LoadConfig owns the config instance it just loaded, while retry callers
// share s.cfg and must not mutate it without the lock.
func buildConfiguredProvider(ctx context.Context, cfg *config.AIConfig, orgID string, stampModel bool) (providers.Provider, string) {
selectedModel, resolveErr := ResolveConfiguredModel(ctx, cfg)
if resolveErr != nil {
log.Warn().Err(resolveErr).Str("orgID", orgID).Msg("AI enabled but no effective provider model could be resolved")
return nil, summarizePatrolRuntimeFailureDetail(resolveErr.Error(), false)
}
if stampModel {
cfg.Model = selectedModel
}
selectedProvider, _ := config.ParseModelString(selectedModel)
nextProvider, providerErr := providers.NewForModel(cfg, selectedModel)
if providerErr != nil {
log.Warn().
Err(providerErr).
Str("selected_model", selectedModel).
Str("selected_provider", selectedProvider).
Strs("configured_providers", cfg.GetConfiguredProviders()).
Msg("AI enabled but selected provider could not be initialized")
return nil, summarizePatrolRuntimeFailureDetail(providerErr.Error(), false)
}
log.Info().
Str("provider", selectedProvider).
Str("model", selectedModel).
Str("control_level", cfg.GetControlLevel()).
Bool("autonomous", cfg.IsAutonomous()).
Msg("AI service initialized")
return nextProvider, ""
}
// ProviderInitError returns the redacted reason the default provider could not
// be initialized, or "" when the provider is healthy or intentionally absent.
func (s *Service) ProviderInitError() string {
s.mu.RLock()
defer s.mu.RUnlock()
return s.providerInitErr
}
// RetryProviderInit rebuilds the default provider when the config claims an
// enabled, configured provider but initialization previously failed. Provider
// construction can depend on a live model-catalog call, so a boot-time race
// (Pulse starting before a local Ollama server) must not strand the runtime
// until the next settings save; Patrol calls this once per scheduled run.
// Returns true when a usable provider is available afterwards.
func (s *Service) RetryProviderInit(ctx context.Context) bool {
s.mu.RLock()
cfg := s.cfg
provider := s.provider
orgID := s.orgID
s.mu.RUnlock()
if provider != nil {
return true
}
if cfg == nil || !cfg.Enabled || !cfg.IsConfigured() {
return false
}
nextProvider, initErr := buildConfiguredProvider(ctx, cfg, orgID, false)
s.mu.Lock()
defer s.mu.Unlock()
// A concurrent LoadConfig may have replaced provider or cfg; never clobber
// a healthy provider with a failed retry.
if s.provider != nil {
return true
}
if s.cfg != cfg {
return s.provider != nil
}
s.provider = nextProvider
s.providerInitErr = initErr
return nextProvider != nil
}
// IsEnabled returns true if AI is enabled and configured
func (s *Service) IsEnabled() bool {
// In demo mode, AI is always considered enabled (using mock backend)