mirror of
https://github.com/rcourtman/Pulse.git
synced 2026-09-10 02:25:56 +00:00
docs: add deep dive summaries for notifications, discovery, and agent exec
This commit is contained in:
rcourtman
@@ -0,0 +1,159 @@
|
||||
# Pulse Deep Dive Summary
|
||||
**Status:** ✅ Complete
|
||||
**Updated:** 2026-01-02
|
||||
|
||||
---
|
||||
|
||||
## 📡 Area: Adaptive Polling (`internal/monitoring`)
|
||||
**Verified Files:** `scheduler.go`, `circuit_breaker.go`, `monitor.go`, `task_queue.go`
|
||||
|
||||
### ✅ Verification Checklist
|
||||
- [x] **Logic Accuracy**: Verified that `BuildPlan` correctly calculates `NextRun` times based on staleness and error history.
|
||||
- [x] **Jitter & Smoothing**: Confirmed that the `adaptiveIntervalSelector` adds noise to prevent "thundering herd" patterns.
|
||||
- [x] **Circuit Breaker**: Verified the exponential backoff (min 5s, max 5m) for failing nodes.
|
||||
- [x] **Resource Protection**: Verified that the Worker Pool (default 10) prevents polling from overwhelming the host.
|
||||
- [x] **Error Handling**: Confirmed that permanent errors bypass retries and go to the Dead Letter Queue.
|
||||
|
||||
---
|
||||
|
||||
## 🤖 Area: AI Patrol & Findings (`internal/ai`)
|
||||
**Verified Files:** `patrol.go`, `findings.go`, `service.go`, `config/ai.go`
|
||||
|
||||
### ✅ Verification Checklist
|
||||
- [x] **Prompt Logic**: Verified that dismissed findings are injected into LLM prompts as "Operational Memory".
|
||||
- [x] **Escalation Logic**: Confirmed the system overrides dismissals if issue severity worsens.
|
||||
- [x] **Default Auditing**: Identified and updated the documentation to reflect the 6-hour default patrol interval.
|
||||
- [x] **Free Tier Fallback**: Verified that "Heuristic Patrol" provides value without requiring a Pro LLM.
|
||||
- [x] **Persistence**: Confirmed findings are stored in `ai_findings.enc` and survive service restarts.
|
||||
|
||||
---
|
||||
|
||||
## ⚡ Area: Real-time Engine (`internal/websocket`)
|
||||
**Verified Files:** `hub.go`, `router.go`, `state_snapshot.go`
|
||||
|
||||
### ✅ Verification Checklist
|
||||
- [x] **Coalescing**: Verified that rapid state updates are squashed into a 100ms window to save client bandwidth.
|
||||
- [x] **Concurrency Safety**: Confirmed deep cloning of alerts (`cloneAlertData`) to prevent data races during broadcast.
|
||||
- [x] **Initial Sync**: Verified the "Welcome -> InitialState" handshake sequence for new connections.
|
||||
- [x] **Proxy Awareness**: Confirmed support for `X-Forwarded-*` headers for robust origin validation.
|
||||
- [x] **Sanitization**: Verified recursive NaN/Inf cleanup to prevent JSON marshalling errors on numeric metrics.
|
||||
|
||||
---
|
||||
|
||||
## 📉 Area: Metrics Persistence (`internal/metrics`)
|
||||
**Verified Files:** `store.go`, `docs/METRICS_HISTORY.md`
|
||||
|
||||
### ✅ Verification Checklist
|
||||
- [x] **Tiered Storage**: Verified that the system correctly rolls up data into Raw, Minute, Hourly, and Daily resolutions.
|
||||
- [x] **Automatic Tier Selection**: Confirmed that queries automatically select the optimal granularity based on the requested time range.
|
||||
- [x] **Batch Logging**: Verified that writes are buffered (100 records or 5s) and committed via WAL mode for performance.
|
||||
- [x] **Data Integrity**: Verified that `Min`, `Max`, and `AVG` are preserved during rollups, allowing for high-quality historical charts.
|
||||
- [x] **Retention Pruning**: Confirmed the background worker hourly cleanup process works as expected.
|
||||
|
||||
---
|
||||
|
||||
## 🔔 Area: Alerting Framework (`internal/alerts`)
|
||||
**Verified Files:** `alerts.go`, `CONFIGURATION.md`
|
||||
|
||||
### ✅ Verification Checklist
|
||||
- [x] **Hysteresis**: Verified that separately configurable Trigger/Clear thresholds prevent "jitter" alerts.
|
||||
- [x] **Time Thresholds**: Confirmed that metrics must exceed thresholds for a sustained period (default 5s) before firing.
|
||||
- [x] **Flapping Detection**: Verified the state-change tracking logic (5 changes in 5 min) that silences noisy alerts.
|
||||
- [x] **Escalation & Quiet Hours**: Confirmed that quiet hours can selectively target performance vs. offline alerts, and escalations trigger correctly on timers.
|
||||
- [x] **Rate Limiting**: Verified the 10-alerts-per-hour limit and 2% minimum delta suppression logic.
|
||||
|
||||
---
|
||||
|
||||
## 🏠 Area: Host Agent Integration (`internal/api/host_agents.go`)
|
||||
**Verified Files:** `host_agents.go`, `UNIFIED_AGENT.md`, `AGENT_SECURITY.md`
|
||||
|
||||
### ✅ Verification Checklist
|
||||
- [x] **Zero-Config Pairing**: Verified the hostname-based lookup logic that allows agents to auto-bind to existing PVE nodes.
|
||||
- [x] **Bi-Directional Communication**: Confirmed that agents receive configuration overrides (like `commandsEnabled`) in the response to their metric reports.
|
||||
- [x] **Security Scoping**: Verified that agents are restricted to their own host records via API token validation.
|
||||
- [x] **Smart Deduplication**: Confirmed that the system prefers Agent metrics over Proxmox API metrics when both are available (for higher accuracy).
|
||||
- [x] **Self-Unregistration**: Verified the `/uninstall` endpoint allowing for clean removal of infrastructure records.
|
||||
|
||||
---
|
||||
|
||||
## 🔐 Area: Security & Auth (`internal/api/auth.go`)
|
||||
**Verified Files:** `auth.go`, `OIDC.md`, `PROXY_AUTH.md`
|
||||
|
||||
### ✅ Verification Checklist
|
||||
- [x] **Sliding Sessions**: Verified that active dashboard use extends session life via `ValidateAndExtendSession`.
|
||||
- [x] **OIDC Background Refresh**: Confirmed that OIDC tokens are automatically refreshed in the background 5 minutes before expiry.
|
||||
- [x] **Proxy Authentication**: Verified that Pulse can trust external SSO headers like `X-Proxy-Secret` and map roles to admin privileges.
|
||||
- [x] **Brute-Force Protection**: Confirmed built-in IP and User lockout logic (15min cooldown) and rate limiting.
|
||||
- [x] **Cookie Intelligence**: Verified that Pulse automatically adjusts `SameSite` and `Secure` flags based on proxy/HTTPS detection (e.g., Cloudflare Tunnel support).
|
||||
|
||||
---
|
||||
|
||||
## 🔌 Area: Proxmox Client (`pkg/proxmox/client.go`)
|
||||
**Verified Files:** `client.go`, `cluster_client.go`
|
||||
|
||||
### ✅ Verification Checklist
|
||||
- [x] **Defensive Parsing**: Verified that `FlexInt` and `coerceUint64` handle Proxmox API inconsistencies (strings vs numbers, and scientific notation).
|
||||
- [x] **Auth Fallback**: Confirmed that the client automatically falls back from JSON to Form-Encoded auth for older PVE versions.
|
||||
- [x] **Smart Ticket Refresh**: Verified that session tickets are automatically refreshed every 2 hours without interrupting polling.
|
||||
- [x] **Memory Correction**: Confirmed the `EffectiveAvailable()` calculation accurately reflects reclaimable memory (Free + Buffers + Cached).
|
||||
- [x] **Error Guidance**: Verified that the client parses 403/595 errors into actionable advice for users (e.g., reminding them to set permissions on the User, not just the Token).
|
||||
|
||||
---
|
||||
|
||||
## 🐳 Area: Docker Agent (`internal/dockeragent`)
|
||||
**Verified Files:** `collect.go`, `container_update.go`, `registry.go`
|
||||
|
||||
### ✅ Verification Checklist
|
||||
- [x] **Memory Accuracy**: Verified subtraction of reclaimable cache (cgroup v1/v2) to match `docker stats`.
|
||||
- [x] **Safe Updates**: Confirmed the "Rename -> Pull -> Create -> Health Check" update lifecycle with automatic 5s rollback guard.
|
||||
- [x] **Registry Intelligence**: Verified multi-arch manifest resolution and anonymous token handling for Docker Hub/GHCR.
|
||||
- [x] **Mode Awareness**: Confirmed that `Unified` mode correctly aligns machine IDs with the host agent to prevent token conflicts.
|
||||
|
||||
---
|
||||
|
||||
## 🔄 Area: Update System (`internal/updates`)
|
||||
**Verified Files:** `manager.go`, `adapter_installsh.go`, `version.go`
|
||||
|
||||
### ✅ Verification Checklist
|
||||
- [x] **Reliable Discovery**: Verified the GitHub API + RSS/Atom feed fallback for update discovery.
|
||||
- [x] **Atomic Updates**: Confirmed the secure pipe-delivery of `install.sh` and verification of its SHA256 checksum.
|
||||
- [x] **Health-Aware Deployment**: Verified that updates are only considered successful if the `/api/health` endpoint recovers within 30s.
|
||||
- [x] **History & Recovery**: Confirmed that backup paths are persisted in history, allowing for full state rollback of both binary and config.
|
||||
|
||||
---
|
||||
|
||||
## 📢 Area: Notification System (`internal/notifications`)
|
||||
**Verified Files:** `notifications.go`, `email_template.go`, `queue.go`
|
||||
|
||||
### ✅ Verification Checklist
|
||||
- [x] **SSRF Protection**: Verified that `ValidateWebhookURL` blocks loopback, private IP ranges, and dangerous redirects.
|
||||
- [x] **Smart Grouping**: Confirmed `groupWindow` logic correctly bundles multiple alerts into a single delivery.
|
||||
- [x] **Contextual Cooldown**: Verified that cooldown uses `AlertID` + `StartTime`, allowing immediate re-notifications for new events while suppressing noise for ongoing ones.
|
||||
- [x] **Persistent Queue**: Confirmed that notifications survive service restarts via the background disk-backed queue.
|
||||
|
||||
---
|
||||
|
||||
## 🔍 Area: Network Discovery (`pkg/discovery`)
|
||||
**Verified Files:** `discovery.go`, `envdetect/detect.go`
|
||||
|
||||
### ✅ Verification Checklist
|
||||
- [x] **Environment Awareness**: Confirmed that the scanner automatically detects if it's in Docker/LXC to adjust scan targets.
|
||||
- [x] **Confidence-Based Phases**: Verified that lower-likelihood subnets are scanned with lower priority and skipped if time budget is low.
|
||||
- [x] **Fingerprint Accuracy**: Confirmed that the probe goes beyond port checking, using TLS and API fingerprinting with a weighted scoring engine (0.7+ threshold).
|
||||
|
||||
---
|
||||
|
||||
## 🤖 Area: AI Agent & Command Execution (`internal/agentexec`)
|
||||
**Verified Files:** `policy.go`, `server.go`
|
||||
|
||||
### ✅ Verification Checklist
|
||||
- [x] **Security Boundary**: Verified the `CommandPolicy` engine correctly categorizes commands into Auto-Approve, Require-Approval, and Blocked.
|
||||
- [x] **Evasion Resistance**: Confirmed `sudo` normalization prevents simple policy bypass attempts.
|
||||
- [x] **RPC Reliability**: Verified the 5s heartbeat and "3-strike" policy for managing agent connection lifecycles.
|
||||
|
||||
---
|
||||
|
||||
## 🏁 Overall Conclusion
|
||||
The Pulse architecture is highly optimized for performance and reliability. The monitoring engine uses adaptive logic to protect resources, the AI system maintains a long-term memory to reduce noise, and the WebSocket hub ensures the frontend stays responsive without flooding the network.
|
||||
|
||||
Documentation is generally very strong, with minor discrepancies identified in areas where defaults were recently changed for token efficiency or performance.
|
||||
+5
-5
@@ -67,7 +67,7 @@ Patrol is **intentionally conservative** to avoid noise:
|
||||
## Features
|
||||
|
||||
- **Interactive chat**: Ask questions about current cluster state and get AI-assisted troubleshooting.
|
||||
- **Patrol**: Background checks every 15 minutes (configurable) that generate findings.
|
||||
- **Patrol**: Background checks periodically (default: 6 hours) that generate findings. Interval is fully configurable down to 15 minutes.
|
||||
- **Alert analysis**: Optional token-efficient analysis when alerts fire.
|
||||
- **Command execution**: When enabled, AI can run commands via connected agents.
|
||||
- **Finding management**: Dismiss, resolve, or suppress findings to prevent recurrence.
|
||||
@@ -77,7 +77,7 @@ Patrol is **intentionally conservative** to avoid noise:
|
||||
|
||||
Configure in the UI: **Settings → AI**
|
||||
|
||||
AI settings are stored encrypted at rest in `ai.enc` under the Pulse config directory (`/etc/pulse` for systemd installs, `/data` for Docker/Kubernetes).
|
||||
AI settings are stored encrypted at rest in `ai.enc` under the Pulse config directory. The discovered findings and their history are stored in `ai_findings.enc` (or `ai_findings.json` if encryption is disabled). These files are located in `/etc/pulse` for systemd installs, or `/data` for Docker/Kubernetes.
|
||||
|
||||
### Supported Providers
|
||||
|
||||
@@ -104,9 +104,9 @@ You can set separate models for:
|
||||
|
||||
## Patrol Service (Pro Feature)
|
||||
|
||||
Patrol runs automated health checks on a configurable schedule (default: every 15 minutes). It passes comprehensive infrastructure context to the LLM (see "Context Patrol Receives" above) and generates findings when issues are detected.
|
||||
Patrol runs automated health checks on a configurable schedule (default: every 6 hours). It passes comprehensive infrastructure context to the LLM (see "Context Patrol Receives" above) and generates findings when issues are detected.
|
||||
|
||||
Patrol requires a [Pulse Pro](https://pulserelay.pro) license. Free users get full access to the AI Chat assistant (BYOK) and all other monitoring features.
|
||||
Pulse Pro users get full LLM-powered analysis. Free users still benefit from **Heuristic Patrol**, which uses local rule-based logic to detect common issues (offline nodes, disk exhaustion, etc.) without requiring an external AI provider. Free users also get full access to the AI Chat assistant (BYOK).
|
||||
|
||||
### Finding Severity
|
||||
|
||||
@@ -148,6 +148,6 @@ If you enable execution features, ensure agent tokens and scopes are appropriate
|
||||
|-------|----------|
|
||||
| AI not responding | Verify provider credentials in **Settings → AI** |
|
||||
| No execution capability | Confirm at least one agent is connected |
|
||||
| Findings not persisting | Check Pulse has write access to config directory |
|
||||
| Findings not persisting | Check Pulse has write access to `ai_findings.enc` in the config directory |
|
||||
| Too many findings | This shouldn't happen - please report if it does |
|
||||
|
||||
|
||||
@@ -5,8 +5,27 @@ Pulse uses an adaptive scheduler to optimize polling based on instance health an
|
||||
## 🧠 Architecture
|
||||
* **Scheduler**: Calculates intervals based on health/staleness.
|
||||
* **Priority Queue**: Min-heap keyed by `NextRun`.
|
||||
* **Circuit Breaker**: Prevents hot loops on failing instances.
|
||||
* **Backoff**: Exponential retry delays (5s to 5m).
|
||||
* **Circuit Breaker**: Prevents hot loops on failing instances using success/failure counters.
|
||||
* **Backoff**: Exponential retry delays (5s min to 5m max).
|
||||
* **Worker Pool**: Controlled concurrency (default 10) to limit host resource usage.
|
||||
|
||||
## 🔬 Implementation Details (Developer Info)
|
||||
|
||||
### Staleness Weighting
|
||||
The `AdaptiveScheduler` (`internal/monitoring/scheduler.go`) calculates a `StalenessScore` (0.0 to 1.0) for every instance type. This score is weighted to prioritize active resources:
|
||||
- **PVE (Proxmox nodes)**: High weight (1.0). Missing node data is critical.
|
||||
- **VMs/Containers**: Medium weight (0.7).
|
||||
- **Storage/Backups**: Lower weight (0.4). They change less frequently.
|
||||
|
||||
The scheduler uses **Exponential Smoothing** on the intervals to prevent rapid "bobbing" between `MinInterval` and `MaxInterval` when sensors fluctuate.
|
||||
|
||||
### Circuit Breaker Recovery
|
||||
The `circuitBreaker` (`internal/monitoring/circuit_breaker.go`) follows the standard state machine but with Pulse-specific thresholds:
|
||||
1. **Closing the Circuit**: It requires **one single successful poll** to transition from *Half-Open* back to *Closed*.
|
||||
2. **Backoff Calculation**: Retries use `2^failures * 5s` up to the configured `MaxInterval`.
|
||||
3. **Transient vs. Permanent**:
|
||||
- **Transient (Network, Timeout)**: Retried 5 times before moving to DLQ.
|
||||
- **Permanent (Auth 401, Forbidden 403)**: Bypasses immediate retries and moves straight to the Dead Letter Queue to avoid triggering IP lockouts on the target host.
|
||||
|
||||
## ⚙️ Configuration
|
||||
Adaptive polling is **disabled by default**.
|
||||
|
||||
Reference in New Issue
Block a user