When a user deletes an API token that was migrated from .env, track
the hash in a suppression list to prevent it from being re-migrated
on the next restart.
Changes:
- Add SuppressedEnvMigrations field to Config
- Add env_token_suppressions.json persistence
- Check suppression list during env token migration
- Record suppressed hash when deleting "Migrated from .env" tokens
- Update RemoveAPIToken to return the removed record
Related to #871
- Add validateAIFindings() that cross-checks AI findings against actual metrics
- Filter out low-confidence findings (CPU <50%, memory <60%, disk <70%)
- Always allow critical findings, backup issues, and reliability findings through
- Update AI system prompt with stricter thresholds and explicit noise examples
- Add 'before creating a finding' checklist for AI (the 3am test)
- Update AI.md docs with clear value proposition and expectations
- Add comprehensive tests for the validation layer
This ensures paying users get immediate value without noise.
PBS datastores are now displayed in the Storage overview alongside PVE
storage. Each PBS datastore is converted to a Storage entry with:
- type: 'pbs'
- content: 'backup'
- shared: true
- active: based on PBS instance status
This provides a complete picture of all storage resources in one view
while keeping detailed PBS info in the dedicated PBS section.
Closes#869
- Fix PVE nodes: buildNodeUrl in ProxmoxNodesSection.tsx now prioritizes
guestURL over host (was ignoring guestURL entirely)
- Add PBS support: GuestURL field added to PBSInstance config, model,
and API handlers
- Add PMG support: GuestURL field added to PMGInstance config, model,
and API handlers
- Update NodeSummaryTable to use guestURL for PBS nodes
- Frontend types updated for PBS/PMG guestURL support
The Guest URL setting in node configuration now works correctly across
all node types. When set, it takes priority over the Host URL when
clicking on node names to navigate to the Proxmox/PBS/PMG web UI.
Closes#870
- Add IsMockResource() helper to detect mock data resources by naming patterns
- Filter out heuristic findings from mock resources when PULSE_MOCK_MODE is off
- Mock findings still appear when demo mode is enabled
- Ensures users don't see mock data findings in production
The AI demo/mock findings were using a non-existent MOCK_ENABLED
env var. Changed to PULSE_MOCK_MODE which is the actual env var
used by the mock data system.
- Add AnalyzeKubernetes to PatrolConfig and enable by default
- Add analyzeKubernetesCluster() for K8s heuristic analysis
- Detect offline clusters, nodes not ready, CrashLoopBackOff pods
- Detect failed/pending pods and unavailable deployments
- Add K8s clusters to LLM context in buildInfrastructureSummary()
- Add kubernetes_cluster as valid resource type in AI prompt
- Enhance analyzeDockerHost() for Docker/Podman
- Podman-aware messaging based on runtime
- Add unhealthy container detection (health checks)
- Add exited container with error detection
- Add high CPU detection (>90%)
- Add stale host detection (10+ minutes)
- Tiered severity for restarts (>10 = critical)
- Add 20 new tests for K8s and Docker analysis
When MOCK_ENABLED=true, Pulse now injects realistic AI patrol
findings to showcase the AI features without requiring actual
LLM API calls. This enables the demo instance to demonstrate:
- Critical/warning/info findings with realistic content
- Patrol run history
- Actionable recommendations
Also includes refinements to dismissal logic from earlier work:
- Only 'not_an_issue' creates permanent suppression
- 'expected_behavior' and 'will_fix_later' just acknowledge
1. Fixed TestNewConfigPersistenceFailsWhenEncryptedDataPresentWithoutKey
- Test was picking up real encryption key from /etc/pulse during migration
- Now temporarily moves system key during test for proper isolation
- Uses t.Cleanup to ensure key is restored even on failure
2. Cleaned up console.log statements in production code
- Dashboard.tsx: replaced console.log with logger.debug for metadata events
- CompleteStep.tsx: removed verbose agent detection debug logs
These changes reduce log noise in production while maintaining debug
capability in development mode.
Addresses #866 - agents were logging 'WebSocket connection failed' warnings
even during normal reconnection scenarios (server restart, network blip, etc).
Changes:
- Normal close errors (1000, 1001, connection reset) now log at Debug level
- Only log Warning after 3+ consecutive failures
- Changed 'Connecting to Pulse' from Info to Debug to reduce noise
- Successful connections still log at Info level
The WebSocket is only used for AI command execution, not metrics, so
transient disconnections don't affect monitoring functionality.
User feedback fields (DismissedReason, UserNote, TimesRaised, Suppressed, Source)
were not being saved to disk, causing 'expected behavior' dismissals to be lost
after Pulse restarted.
- Add missing fields to AIFindingRecord in persistence.go
- Update FindingsPersistenceAdapter to save/load these fields
- Add comprehensive tests for dismissal persistence round-trip
Fixes issue where Frigate storage warning kept reappearing despite being
marked as expected behavior.
Since watch/info findings are filtered from the UI and never shown
to users, don't include them in the patrol run status summary.
This makes the summary consistent with what users actually see.
The LLM was confusing VMIDs because they weren't included in the
context. Now the formatted context shows:
### Container: ollama (VMID 200) on minipc
This prevents the AI from referencing the wrong VMID when generating
findings and recommendations.
When the service restarts, it now checks if a patrol ran within the
last hour. If so, it skips the initial patrol to avoid wasting API
tokens during development/maintenance when the service is restarted
frequently.
The scheduled patrol runs (every 6 hours) are not affected.
100 samples was causing 326k+ input tokens which is expensive.
24 samples (hourly resolution) still provides good pattern visibility
while significantly reducing token cost.
Estimated reduction: ~75% fewer metric tokens.
When AI patrol fails due to API issues like insufficient balance, invalid
API key, or rate limiting, we now create a finding that appears in the
AI Insights tab. This makes the issue visible to users rather than hidden
in logs.
The finding includes:
- Clear description of the issue (e.g., 'Insufficient API credits')
- Recommendation for how to fix it
- Evidence showing the actual error message
When a patrol run encounters errors (e.g., LLM call failed), don't
display 'All healthy' in the summary as that's misleading - the
analysis didn't complete properly.
Now shows 'Analysis incomplete (N errors)' instead, which correctly
explains why the status badge shows red/error.
Modern LLMs have 100k+ token contexts. 100 samples over 24h gives
~15 minute resolution while adding minimal token overhead.
This lets the LLM see fine-grained patterns, short spikes, and
accurately distinguish anomalies from normal behavior.
The in-memory MetricsHistory only retains 24 hours of data, not 7 days.
Changed computeGuestMetricSamples to use trendWindow24h instead of
trendWindow7d, and reduced sample count from 24 to 12 points.
This ensures the LLM actually receives metric samples in the context,
which wasn't happening before because the 7-day query returned empty data.
Bug Fixes:
- Fix boolean fields with 'omitempty' not persisting false values
- AlertTriggeredAnalysis, PatrolAnalyzeNodes/Guests/Docker/Storage
- omitempty causes Go to skip false (zero value) when marshaling JSON
- On reload, NewDefaultAIConfig() sets true, and missing field stays true
- Fix model dropdown losing selection after save (SolidJS reactivity issue)
- Added explicit 'selected' attribute to option elements
- Ensures browser maintains selection with optgroups during re-renders
Improvements:
- Change patrol type label from 'Quick' to 'Patrol' in history table
- Add chat_model and patrol_model to AI settings update log
- Add alert_triggered_analysis to AI config load log for debugging
Instead of relying on pre-computed trend heuristics (which can be misleading
for edge cases like step changes vs continuous growth), we now pass downsampled
raw data points to the LLM so it can interpret patterns directly.
Changes:
- Add MetricSamples field to ResourceContext
- Add DownsampleMetrics() to reduce data points for LLM consumption
- Add formatMetricSamples() to format data compactly (e.g., 'Disk: 26→26→31%')
- Add computeGuestMetricSamples() to gather 7-day sampled history
- Populate MetricSamples for VMs and containers during context build
- Add History section to formatted context output
The LLM now sees actual patterns like 'stable for 6 days then jumped' rather
than just '45.8%/day growth rate' - allowing for much more nuanced interpretation.
This approach:
- Leverages LLM's pattern recognition instead of hard-coded heuristics
- Provides 7 days of data (~24 samples) for context on normal behavior
- Uses minimal tokens due to compact formatting with deduplication
- Is more future-proof as LLMs improve
Example output:
**History (7d sampled, oldest→newest)**: Disk: 26→26→26→26→26→31%
Refs: Frigate disk usage false positive investigation
Filter out 'watch' and 'info' severity findings from the API response.
These lower-severity findings were mostly noise:
- 'watch': CPU is 35% instead of 11% (who cares)
- 'info': Stopped container exists (knew that)
Now only showing actionable findings:
- critical: Something is broken NOW
- warning: Something needs attention soon
Users prefer silence to noise.
Runbooks were a half-built feature that provided no value:
- Only 3 runbooks existed
- AI dynamic remediation already covers the same ground
- Added UI complexity without benefit
Removed:
- runbooks.go and runbooks_test.go
- Handler functions in ai_handlers.go
- Routes in router.go
- Test cases in ai_handlers_test.go
- Auto-fix call in patrol.go
Kept (dead code but harmless):
- Frontend types/API calls (will 404)
- RecordIncidentRunbook function (unused)
Less code = easier to maintain.
Updated LLM prompt with explicit guidance on what NOT to report:
- Small baseline deviations (7% vs 4% is normal variance)
- Low utilization (under 50% CPU or 60% memory is fine)
- Stopped containers that aren't autostart
- 'Elevated' metrics still well under limits
Severity guidelines made more specific:
- CRITICAL: disk >95%, service down, data loss
- WARNING: disk >85%, memory >90%, failures
- WATCH: Only for trends projected to hit critical in <7 days
- INFO: Context/observations
Key message to LLM: 'Users prefer silence to noise'
Only flag things that require operator action.
Smarter anomaly detection to reduce false positives:
**Learning Window:** 7 days → 14 days
- Captures weekly patterns (weekday vs weekend)
**Metric-Specific Thresholds:**
CPU:
- Only report if usage >70% AND >2x baseline
- Low CPU variance (5% vs 10%) is not actionable
Memory:
- Report if >80% OR (>1.5x baseline AND >60%)
- Memory is more stable, lower threshold makes sense
Disk:
- Report if >85% usage OR +15 percentage points growth
- Disk problems are critical, use absolute thresholds
Other metrics:
- Use 2x threshold as default
This dramatically reduces 'noise' anomalies while catching
actual problems that need operator attention.
More aggressive noise filtering:
1. Anomaly threshold raised from 1.5x to 2x
- 1.5x is too borderline to be actionable
- Now requires genuinely significant deviation
2. Filter out 'Ran diagnostic' and 'Executed command' fallback items
- These are generic summaries that provide no value
- Only show remediations with specific, meaningful descriptions
Goal: If something shows in AI Intelligence, it should demand attention.
Critical changes to surface only actionable insights:
1. Anomalies now require at least 50% deviation from baseline
- '1.0x baseline' values filtered out (statistically significant but not actionable)
- Must be >1.5x above OR <0.5x below baseline to report
2. Status changes filter out startup noise
- 'unknown → running' is just system starting, not a real state change
- Backups removed from main list (they have dedicated section)
3. Only show genuinely interesting changes:
- Config changes, migrations, restarts, deletions
- Things that require operator attention
This massively reduces noise while keeping high-signal alerts.
Critical fixes to show only actionable insights:
1. Skip stopped VMs/containers from anomaly detection
- '0.0x baseline' for stopped resources is expected, not an anomaly
- Only check anomalies for status='running'
2. Filter correlations by confidence (>=70%)
- Low confidence correlations are likely coincidental
- Only show high-confidence, actionable dependencies
This reduces noise and surfaces genuinely useful intelligence.
Free Features (no license required):
- Anomaly detection - removed license gating, purely statistical analysis
- Learning status endpoint - GET /api/ai/intelligence/learning
Learning Status Response:
- resources_baselined: count of resources with learned baselines
- total_metrics: total metric baselines (cpu + memory + disk)
- metric_breakdown: {cpu: X, memory: Y, disk: Z}
- status: 'waiting' | 'learning' | 'active'
- message: human-readable description
This makes the AI intelligence features visible to all users,
encouraging upgrades for the full LLM-powered patrol experience.
AIStatusIndicator:
- Now shows BOTH patrol findings AND baseline anomalies
- Displays even when only anomaly detection is active (no patrol)
- Badge count includes both findings + anomalies
- Tooltip provides detailed breakdown by severity
Trend Prediction (backend):
- Add TrendPrediction struct for resource exhaustion forecasting
- CalculateTrend() uses linear regression on sample history
- Predicts days until resource is full (or if declining/stable)
- Severity: critical (<7 days), warning (<30 days), info (>30 days)
- Human-readable descriptions like 'full in ~2 weeks (+0.5% per day)'
This creates a more cohesive intelligence experience where anomaly
detection works independently of the pro/patrol features, making
value visible immediately to all users.
Add /api/ai/intelligence/anomalies endpoint that compares live metrics
against learned baselines to surface deviations - all deterministic
(no LLM required).
Backend:
- Add AnomalyReport struct with severity classification
- Add CheckResourceAnomalies method to baseline store
- Add HandleGetAnomalies API handler
- Add GetStateProvider getter to AI service
Frontend:
- Add AnomalyReport and AnomaliesResponse types
- Add getAnomalies API function
- Add AnomalySeverity type
This is the first step toward surfacing deterministic intelligence
directly in the UI without requiring LLM interaction.
- Create Intelligence struct that aggregates all AI subsystems
- Add /api/ai/intelligence endpoint for system-wide and per-resource insights
- Wire Intelligence into PatrolService as a facade (not replacement)
- Add TypeScript types and API client for frontend
- Add unit tests for Intelligence orchestrator
- Fix pre-existing test failures using diagnostic commands instead of actionable ones
The Intelligence orchestrator provides:
- System-wide health scoring (A-F grades)
- Aggregated findings, predictions, correlations
- Per-resource context generation for AI prompts
- Learning progress tracking
This unifies access to AI subsystems without replacing existing code paths.
IMPORTANT: This disables the encryption key deletion during migration.
Previously, when migrating from /etc/pulse to a new data directory, the code
would DELETE the original key after copying it. This was causing mysterious
key loss bugs in dev environments.
Changes:
- Commented out the os.Remove() call that deletes the encryption key
- Keep both copies of the key for safety (old location is just unused)
- Updated test to skip when production key exists (test isolation issue)
The old key at /etc/pulse will now be preserved even after migration.
This is safe because:
1. The new key location is checked first
2. Having a backup is better than risking data loss
3. Users can manually clean up the old key if desired
Added extensive logging to crypto.go to trace when the encryption key
migration code runs and when it deletes the key. This is to diagnose
a recurring bug where the encryption key mysteriously disappears.
The logs will show:
- When migration is being considered (dataDir != /etc/pulse)
- When migration is skipped (dataDir == /etc/pulse)
- CRITICAL log when key is about to be deleted
- CRITICAL log when key has been deleted
This will help identify whether it's the Go code or something external
deleting the key.
Backend:
- Enhanced buildEnrichedResourceContext to ALWAYS show learned baselines with
status indicators (normal/elevated/anomaly) instead of only when anomalous
- This makes Pulse Pro's 'moat' visible - users can see the AI understands
their infrastructure's normal behavior patterns
- Added baseline import to service.go
Frontend (user changes):
- Added incident event type filtering with toggle buttons
- Added resource incident panel to view all incidents for a resource
- Added timeline expand/collapse functionality in alert history
- Added incident note saving with proper incidentId tracking
- Added startedAt parameter for proper incident timeline loading
rcourtman