diff --git a/README.md b/README.md index 2481f0401..9267d39a4 100644 --- a/README.md +++ b/README.md @@ -274,44 +274,54 @@ Configure email notifications in **Settings → Alerts → Email Destinations** 2. Use your email as username and app password as password 3. Server: smtp-mail.outlook.com, Port: 587, Enable STARTTLS -### Alert Suppression Using VM/CT Tags +### Alert Configuration -Control alert behavior per VM/container using Proxmox tags (no UI configuration needed): +Pulse provides two complementary approaches for managing alerts: -#### Available Tags -- **`pulse-no-alerts`** - Completely disables all alerts for this VM/CT -- **`pulse-monitor-only`** - Shows in UI but suppresses notifications (email, webhooks) -- **`pulse-relaxed`** - Sets thresholds to 95% for CPU/RAM, 98% for disk +#### Custom Alert Rules (Permanent Policy) +Configure persistent alert policies in **Settings → Alerts → Custom Rules**: +- Define thresholds for specific VMs/containers based on name patterns +- Set different thresholds for production vs development environments +- Create complex rules with AND/OR logic +- Manage all rules through the UI with priority ordering -#### Priority Order -Tags take precedence over other configurations: -1. `pulse-no-alerts` tag - Overrides everything, no alerts at all -2. `pulse-relaxed` tag - Overrides to 95%/98% regardless of custom rules -3. Custom Alert Rules from UI - Applied if no tags present -4. Default global thresholds - Used if nothing else is configured +**Use for:** Long-term alert policies like "all database VMs should alert at 90%" -**Note:** `pulse-monitor-only` works with any threshold source. Tags are checked every 30-60 seconds, no restart needed. +#### Proxmox Tags (Temporary Overrides) +Apply operational overrides directly in Proxmox using VM/CT tags: -#### How to Use -1. In Proxmox, edit your VM/CT -2. Add one or more tags in the Options tab -3. Pulse picks up changes within 30-60 seconds (no restart needed) +| Tag | Purpose | Use Case | +|-----|---------|----------| +| `pulse-no-alerts` | Maintenance mode | Suppress all alerts during maintenance windows | +| `pulse-monitor-only` | Silent monitoring | Development/staging VMs you want to watch but not get paged for | +| `pulse-relaxed` | Temporary tolerance | Expected high load periods (backups, batch jobs) | -#### Examples +**How tags work:** +- Tags are operational overrides that take priority over custom rules +- Changes apply within 30-60 seconds (no restart needed) +- `pulse-relaxed` sets fixed thresholds: 95% CPU/RAM, 98% disk +- Multiple tags can be combined (e.g., relaxed + monitor-only) + +**Examples:** ```bash -# Via Proxmox CLI -pvesh set /nodes/pve/lxc/100/config -tags 'production,pulse-monitor-only' -pvesh set /nodes/pve/qemu/200/config -tags 'dev,pulse-no-alerts' -pvesh set /nodes/pve/lxc/300/config -tags 'database,pulse-relaxed' +# Maintenance window - suppress all alerts +pvesh set /nodes/pve/lxc/100/config -tags 'prod,pulse-no-alerts' -# Multiple tags work together -pvesh set /nodes/pve/lxc/400/config -tags 'pulse-relaxed,pulse-monitor-only' +# Development VM - monitor but don't notify +pvesh set /nodes/pve/qemu/200/config -tags 'dev,pulse-monitor-only' + +# Backup server during backup window - relax thresholds +pvesh set /nodes/pve/lxc/300/config -tags 'backup-server,pulse-relaxed' + +# Remove tag after maintenance +pvesh set /nodes/pve/lxc/100/config -tags 'prod' ``` -Perfect for: -- Development VMs that shouldn't trigger alerts -- Services that naturally run hot (databases, media servers) -- Temporary maintenance windows (add tag, remove when done) +**Best practices:** +- Use Custom Rules for "what should normally happen" +- Use Tags for "temporary exceptions to normal" +- Remove tags when temporary need ends +- Document tag usage in your runbooks ### HTTPS/TLS Configuration Enable HTTPS by setting these environment variables: diff --git a/internal/alerts/alerts.go b/internal/alerts/alerts.go index 97dbf5647..612e517f3 100644 --- a/internal/alerts/alerts.go +++ b/internal/alerts/alerts.go @@ -433,15 +433,16 @@ func (m *Manager) CheckGuest(guest interface{}, instanceName string) { return } - // Check for Pulse-specific tags for alert suppression + // Check for Pulse-specific tags (operational overrides - highest priority) + // Tags are meant for temporary operational control, complementing (not replacing) custom rules var suppressAlerts, monitorOnly, useRelaxedThresholds bool for _, tag := range tags { switch tag { - case "pulse-no-alerts": + case "pulse-no-alerts": // Maintenance mode - completely suppress all alerts suppressAlerts = true - case "pulse-monitor-only": - monitorOnly = true - case "pulse-relaxed": + case "pulse-monitor-only": // Operational mode - show in UI but suppress notifications + monitorOnly = true + case "pulse-relaxed": // Temporary override - relax thresholds during expected high load useRelaxedThresholds = true } } @@ -455,7 +456,7 @@ func (m *Manager) CheckGuest(guest interface{}, instanceName string) { log.Info(). Str("alertID", alertID). Str("guest", name). - Msg("Cleared alert for guest with pulse:no-alerts tag") + Msg("Cleared alert - guest has pulse-no-alerts tag (maintenance mode)") } } m.mu.Unlock() @@ -502,7 +503,7 @@ func (m *Manager) CheckGuest(guest interface{}, instanceName string) { } log.Info(). Str("guest", name). - Msg("Applied relaxed thresholds due to pulse-relaxed tag (95% CPU/RAM, 98% disk)") + Msg("Applied pulse-relaxed tag override (95% CPU/RAM, 98% disk) - temporary operational override") } // Check each metric