Commit Graph

14 Commits

Author SHA1 Message Date
rcourtman 186ce504c8 Retire disconnected incident recording and preserve archives
The fleet sampler and coordinator had no production alert trigger and could
repeat cached values as fresh incident evidence. Preserve saved recordings
through explicit read-only lookups, propagate read failures and report the
former live incident count as unmeasured. Keep historical status and duration
units explicit without rewriting archived observations.

Integrate main's alert dispatch wording and startup replay qualification.
Canonical incident listing and real-model outcome qualification remain open.
2026-09-06 17:42:24 +01:00
rcourtman 29ad80870d Preserve per-provider semantics on batch metric misses
A second adversarial review round surfaced two robustness gaps. The
incident recorder treated any ID absent from a metrics batch as an
empty successful sample, which silently rewrites the semantics of a
provider whose per-ID method errors or synthesizes for unknown IDs;
batch misses now fall through to the per-ID call, keeping the batch
for the hot path only. And one bySource mutation site, the PBS host
disk association helper, predated the source-target cache invalidation
invariant; it now invalidates like every other mutation path. The gap
was unobservable today because the adjacent ingest already invalidated,
but the invariant is only useful if it is total.

Contract-Neutral: robustness fixes inside existing cache and sampling semantics, no resolved values change
2026-08-16 23:03:12 +01:00
rcourtman 81de801e58 Batch incident recorder metrics lookups
The incident recorder asks for current metrics for every monitored
resource every five seconds, and the adapter answered each request by
scanning all views with a formatted VMID comparison per candidate. At
thousands of resources that is millions of comparisons and allocations
per tick, forever. The adapter now exposes a batch method that builds
the full lookup in one pass over the views, keyed by every ID form the
per-ID method matches with the same first-match precedence, and the
recorder uses it when the provider offers it. Per-ID behavior is pinned
to the batch by an equivalence test, including VMID collisions.

Contract-Neutral: performance-only batch lookup addition, per-ID metrics resolution is pinned unchanged by equivalence tests
2026-08-16 21:48:31 +01:00
rcourtman 6bdb3c9c8c Coordinate incident recorder async persistence 2026-07-04 13:13:41 +01:00
rcourtman faefe6edc8 Remove 198 unreachable Go functions
Dead-code sweep. Functions flagged unreachable by golang.org/x/tools/cmd/deadcode
and confirmed unused across pulse, pulse-enterprise, pulse-pro and pulse-mobile by
adversarial cross-repo verification. Cross-module reachability was checked
explicitly (only pkg/ exported symbols are importable by other modules; internal/
packages and _test.go files are not). go build, go vet and test-compile all pass.
2026-06-03 12:29:37 +01:00
rcourtman 778a2577b6 feat: Pulse v6 release 2026-03-18 16:06:30 +00:00
rcourtman 7f7edfceb4 test: expand backend coverage 2026-01-25 21:08:44 +00:00
rcourtman 27f1a11acb feat: add AI Intelligence system with investigation and forecasting
Major new AI capabilities for infrastructure monitoring:

Investigation System:
- Autonomous finding investigation with configurable autonomy levels
- Investigation orchestrator with rate limiting and guardrails
- Safety checks for read-only mode enforcement
- Chat-based investigation with approval workflows

Forecasting & Remediation:
- Trend forecasting for resource capacity planning
- Remediation engine for generating fix proposals
- Circuit breaker for AI operation protection

Unified Findings:
- Unified store bridging alerts and AI findings
- Correlation and root cause analysis
- Incident coordinator with metrics recording

New Frontend:
- AI Intelligence page with patrol controls
- Investigation drawer for finding details
- Unified findings panel with actions

Supporting Infrastructure:
- Learning store for user preference tracking
- Proxmox event ingestion and correlation
- Enhanced patrol with investigation triggers
2026-01-24 22:41:43 +00:00
rcourtman 2a8f55d719 feat(enterprise): add Advanced Reporting and Audit Webhooks integration
This commit adds enterprise-grade reporting and audit capabilities:

Reporting:
- Refactored metrics store from internal/ to pkg/ for enterprise access
- Added pkg/reporting with shared interfaces for report generation
- Created API endpoint: GET /api/admin/reports/generate
- New ReportingPanel.tsx for PDF/CSV report configuration

Audit Webhooks:
- Extended pkg/audit with webhook URL management interface
- Added API endpoint: GET/POST /api/admin/webhooks/audit
- New AuditWebhookPanel.tsx for webhook configuration
- Updated Settings.tsx with Reporting and Webhooks tabs

Server Hardening:
- Enterprise hooks now execute outside mutex with panic recovery
- Removed dbPath from metrics Stats API to prevent path disclosure
- Added storage metrics persistence to polling loop

Documentation:
- Updated README.md feature table
- Updated docs/API.md with new endpoints
- Updated docs/PULSE_PRO.md with feature descriptions
- Updated docs/WEBHOOKS.md with audit webhooks section
2026-01-09 21:31:49 +00:00
rcourtman 67bde72c93 Improve test coverage 2025-12-17 12:00:59 +00:00
rcourtman bcd7b550d4 AI Problem Solver implementation and various fixes
- Implement 'Show Problems Only' toggle combining degraded status, high CPU/memory alerts, and needs backup filters
- Add 'Investigate with AI' button to filter bar for problematic guests
- Fix dashboard column sizing inconsistencies between bars and sparklines view modes
- Fix PBS backups display and polling
- Refine AI prompt for general-purpose usage
- Fix frontend flickering and reload loops during initial load
- Integrate persistent SQLite metrics store with Monitor
- Fortify AI command routing with improved validation and logging
- Fix CSRF token handling for note deletion
- Debug and fix AI command execution issues
- Various AI reliability improvements and command safety enhancements
2025-12-06 23:46:08 +00:00
rcourtman 6a8258be14 chore: remove dead code and unused exports
Remove ~900 lines of unused code identified by static analysis:

Go:
- internal/logging: Remove 10 unused functions (InitFromConfig, New,
  FromContext, WithLogger, etc.) that were built but never integrated
- cmd/pulse-sensor-proxy: Remove 7 dead validation functions for a
  removed command execution feature
- internal/metrics: Remove 8 unused notification metric functions and
  10 Prometheus metrics that were never wired up

Frontend:
- Delete ActivationBanner.tsx stub component
- Remove unused exports: stopMetricsSampler, getSamplerStatus,
  formatSpeedCompact, parseMetricKey, getResourceAlerts
2025-11-27 13:17:39 +00:00
rcourtman 63043adf3b test: add unit tests for internal/metrics
Add 14 tests covering:
- RecordAlertFired, RecordAlertResolved, RecordAlertAcknowledged
- RecordNotificationSent with success/failure
- RecordNotificationRetry, RecordNotificationDLQ
- RecordGroupedNotification with various counts
- RecordAlertEscalation with various levels
- RecordAlertSuppressed with various reasons
- UpdateQueueDepth, UpdateHistorySize
- RecordHistorySaveError
- Metric vector initialization verification
2025-11-26 14:27:54 +00:00
rcourtman c8e0281953 Add comprehensive alert system reliability improvements
This commit implements critical reliability features to prevent data loss
and improve alert system robustness:

**Persistent Notification Queue:**
- SQLite-backed queue with WAL journaling for crash recovery
- Dead Letter Queue (DLQ) for notifications that exhaust retries
- Exponential backoff retry logic (100ms → 200ms → 400ms)
- Full audit trail for all notification delivery attempts
- New file: internal/notifications/queue.go (661 lines)

**DLQ Management API:**
- GET /api/notifications/dlq - Retrieve DLQ items
- GET /api/notifications/queue/stats - Queue statistics
- POST /api/notifications/dlq/retry - Retry failed notifications
- POST /api/notifications/dlq/delete - Delete DLQ items
- New file: internal/api/notification_queue.go (145 lines)

**Prometheus Metrics:**
- 18 comprehensive metrics for alerts and notifications
- Metric hooks integrated via function pointers to avoid import cycles
- /metrics endpoint exposed for Prometheus scraping
- New file: internal/metrics/alert_metrics.go (193 lines)

**Alert History Reliability:**
- Exponential backoff retry for history saves (3 attempts)
- Automatic backup restoration on write failure
- Modified: internal/alerts/history.go

**Flapping Detection:**
- Detects and suppresses rapidly oscillating alerts
- Configurable window (default: 5 minutes)
- Configurable threshold (default: 5 state changes)
- Configurable cooldown (default: 15 minutes)
- Automatic cleanup of inactive flapping history

**Alert TTL & Auto-Cleanup:**
- MaxAlertAgeDays: Auto-cleanup old alerts (default: 7 days)
- MaxAcknowledgedAgeDays: Faster cleanup for acked alerts (default: 1 day)
- AutoAcknowledgeAfterHours: Auto-ack long-running alerts (default: 24 hours)
- Prevents memory leaks from long-running alerts

**WebSocket Broadcast Sequencer:**
- Channel-based sequencing ensures ordered message delivery
- 100ms coalescing window for rapid state updates
- Prevents race conditions in WebSocket broadcasts
- Modified: internal/websocket/hub.go

**Configuration Fields Added:**
- FlappingEnabled, FlappingWindowSeconds, FlappingThreshold, FlappingCooldownMinutes
- MaxAlertAgeDays, MaxAcknowledgedAgeDays, AutoAcknowledgeAfterHours

All features are production-ready and build successfully.
2025-11-06 16:46:30 +00:00