mirror of
https://github.com/Studio-Saelix/sencho.git
synced 2026-08-06 17:08:10 +00:00
Anso
0a94df318a
* fix(alerts): harden with security fixes, design compliance, and test coverage Add authMiddleware to all alert endpoints, validate notification test dispatch inputs, fix restart_count metric via Docker inspect, correct network metric units, replace any types with DockerContainerStats interface, add webhook timeouts and dispatch error tracking. Frontend: migrate Select to Combobox, add ScrollArea and delete confirmation AlertDialog, fix icon strokeWidth to 1.5. Add update availability notifications for both Sencho version updates (6-hour check in MonitorService) and stack image updates (state transition detection in ImageUpdateService). Extract shared version fetch logic into utils/version-check.ts. Add diagnostic logging gated behind developer_mode for MonitorService breach state machine and NotificationService dispatch routing. Tests: 24 new alert API integration tests, restart_count and version check unit tests (688 total passing). Docs updated with HTTPS requirement, update notifications section, and troubleshooting guide. * fix(alerts): remove unused TEST_USERNAME import in alerts-api tests
176 lines
8.1 KiB
Plaintext
176 lines
8.1 KiB
Plaintext
---
|
|
title: Alerts & Notifications
|
|
description: Set threshold-based alerts on container metrics and route them to Discord, Slack, or any webhook.
|
|
---
|
|
|
|
Sencho can watch your containers for resource anomalies and notify you when thresholds are breached. Alerts are defined per stack, and notifications are delivered through external channels you configure.
|
|
|
|
## Setting up notification channels
|
|
|
|
At least one channel must be enabled before alerts can be delivered externally. Go to **Settings > Notifications**.
|
|
|
|
<Frame>
|
|
<img src="/images/alerts-notifications/notifications-settings.png" alt="Notifications & Alerts settings showing Discord, Slack, and Webhook tabs" />
|
|
</Frame>
|
|
|
|
Three channel types are available, each configured with a webhook URL and an enable/disable toggle:
|
|
|
|
### Discord
|
|
|
|
1. In Discord, go to your server's **Settings > Integrations > Webhooks**
|
|
2. Click **New Webhook**, choose a channel, and copy the webhook URL
|
|
3. In Sencho, open **Settings > Notifications > Discord**, paste the URL, enable the toggle, and click **Save**
|
|
4. Click **Test** to send a test message and verify the connection
|
|
|
|
### Slack
|
|
|
|
1. In Slack, go to **api.slack.com/apps**, create an app, and add the **Incoming Webhooks** feature
|
|
2. Activate it and copy the generated webhook URL for your chosen channel
|
|
3. In Sencho, open **Settings > Notifications > Slack**, paste the URL, enable the toggle, and click **Save**
|
|
|
|
### Generic Webhook
|
|
|
|
Any HTTPS endpoint that accepts a POST with a JSON body can receive Sencho alerts. Go to **Settings > Notifications > Webhook**, enter the URL, enable the toggle, and click **Save**.
|
|
|
|
<Note>
|
|
All webhook URLs (Discord, Slack, and generic) must use HTTPS. HTTP URLs are rejected.
|
|
</Note>
|
|
|
|
The payload format is:
|
|
|
|
```json
|
|
{
|
|
"level": "warning",
|
|
"message": "cpu_percent exceeded 90% for 5 minutes on stack my-app",
|
|
"timestamp": "2026-03-22T10:00:00.000Z"
|
|
}
|
|
```
|
|
|
|
## Creating stack alerts
|
|
|
|
Stack alerts are configured per stack. Right-click a stack in the sidebar (or click the three-dot menu) and select **Alerts**. A panel slides open showing existing rules and a form to create new ones.
|
|
|
|
<Frame>
|
|
<img src="/images/alerts-notifications/alert-panel.png" alt="Stack Alerts panel showing the notification status banner and alert rule form" />
|
|
</Frame>
|
|
|
|
The panel includes:
|
|
|
|
- **Notification status banner** at the top, showing whether channels are configured. If no channels are enabled, a warning explains that rules will be evaluated but no external notifications will be sent.
|
|
- **Existing Rules** section listing all active rules for this stack, each showing the metric, condition, duration, and cooldown. Hover over a rule to reveal the delete button.
|
|
- **Add New Rule** form with the fields described below.
|
|
|
|
### Alert fields
|
|
|
|
| Field | Description |
|
|
|-------|-------------|
|
|
| **Metric** | The container metric to watch (see table below) |
|
|
| **Operator** | Comparison: Greater than, Greater or equal, Less than, Less or equal, Equals |
|
|
| **Threshold** | The numerical value to compare against |
|
|
| **Duration (mins)** | How long the condition must hold before firing (default: 5) |
|
|
| **Cooldown (mins)** | Minimum time between repeated notifications for this rule (default: 60) |
|
|
|
|
### Available metrics
|
|
|
|
| Metric | Description |
|
|
|--------|-------------|
|
|
| CPU Usage (%) | CPU usage relative to total host cores |
|
|
| Memory Usage (%) | Memory used as a fraction of the host total |
|
|
| Memory Usage (MB) | RSS memory used by the container |
|
|
| Network In (MB) | Cumulative inbound network bytes (in MB) |
|
|
| Network Out (MB) | Cumulative outbound network bytes (in MB) |
|
|
| Restart Count | Number of times the container has restarted |
|
|
|
|
### Example: alert on high CPU
|
|
|
|
To alert when any container in a stack uses more than 80% CPU for over 5 consecutive minutes, with a 60-minute cooldown:
|
|
|
|
| Field | Value |
|
|
|-------|-------|
|
|
| Metric | CPU Usage (%) |
|
|
| Operator | Greater than |
|
|
| Threshold | 80 |
|
|
| Duration | 5 |
|
|
| Cooldown | 60 |
|
|
|
|
## Notification history
|
|
|
|
All dispatched notifications appear in the **notification bell** in the top-right corner of the navigation bar. A red dot pulses on the bell when unread notifications exist.
|
|
|
|
<Frame>
|
|
<img src="/images/alerts-notifications/notification-popover.png" alt="Notification popover showing recent alert entries with level badges and timestamps" />
|
|
</Frame>
|
|
|
|
Click the bell to open the notification popover, which shows:
|
|
|
|
- **Level badge** for each entry (ERROR in red, WARNING in amber, INFO in default)
|
|
- **Node name** badge for notifications from remote nodes
|
|
- **Timestamp** of when the alert was triggered
|
|
- **Alert message** describing what was breached
|
|
|
|
The popover header includes:
|
|
|
|
| Action | What it does |
|
|
|--------|--------------|
|
|
| **Mark all as read** | Marks all notifications as read (removes the red dot) |
|
|
| **Clear all** | Deletes all notifications from the list |
|
|
|
|
You can also dismiss individual notifications by hovering over them and clicking the dismiss button.
|
|
|
|
<Note>
|
|
Notifications only reach external channels (Discord, Slack, Webhook) if at least one channel is enabled. Dashboard notifications appear regardless.
|
|
</Note>
|
|
|
|
## Alerts on remote nodes
|
|
|
|
Alerts work the same way on remote nodes as they do locally. When you switch to a remote node and open a stack's alerts panel, you are managing rules on that remote instance.
|
|
|
|
Key details:
|
|
|
|
- **Alert rules are stored on each node independently.** Rules created while a remote node is selected are saved on that remote instance, not your primary instance.
|
|
- **Monitoring runs locally on each node.** Each Sencho instance evaluates its own alert rules against its own container metrics.
|
|
- **Notifications are sent by the node that detects the breach.** Make sure notification channels are configured on each remote node where you want to receive alerts, since channel settings are per-instance.
|
|
|
|
The alert panel shows a blue info banner when you are configuring alerts on a remote node, including which channels are active on that node.
|
|
|
|
### Setup checklist for remote alerts
|
|
|
|
1. On the **remote** Sencho instance, go to **Settings > Notifications** and configure at least one channel
|
|
2. From your **primary** instance, switch to the remote node
|
|
3. Right-click a stack and select **Alerts** to create rules
|
|
4. The remote instance handles monitoring and notification delivery independently
|
|
|
|
## Update availability notifications
|
|
|
|
Sencho can notify you when software updates are available, both for Sencho itself and for your stack images.
|
|
|
|
### Sencho version updates
|
|
|
|
When a newer version of Sencho is published, an informational notification is dispatched through your configured channels. This check runs periodically and only notifies once per new version. After you update, the cycle resets and you will be notified when the next release is available.
|
|
|
|
### Stack image updates
|
|
|
|
When the periodic image check (every 6 hours) detects that a stack has new upstream images available, a notification is dispatched for each affected stack. Notifications are sent only on state transitions: you will be notified once when a new update appears, not on every check cycle. After you update the stack, the status resets.
|
|
|
|
Both notification types use the same channel routing as alerts: if notification routes are configured for a stack, those channels receive the message; otherwise, global notification channels are used as a fallback.
|
|
|
|
## Troubleshooting
|
|
|
|
### Notifications not being delivered
|
|
|
|
- Verify at least one notification channel is enabled in **Settings > Notifications**
|
|
- Click **Test** on the channel to confirm the webhook URL is reachable
|
|
- Check that the webhook URL uses HTTPS
|
|
- If using notification routing (Admiral tier), verify the route pattern matches the stack name
|
|
|
|
### Alert not firing
|
|
|
|
- Confirm the alert rule exists by opening the stack's alert panel
|
|
- Check that the **duration** has elapsed; the condition must hold continuously for the configured duration before an alert fires
|
|
- Check the **cooldown** period; after an alert fires, it will not fire again until the cooldown expires
|
|
- Verify the metric is being collected; container must be running for stats to be gathered
|
|
|
|
### Delete confirmation dialog
|
|
|
|
Deleting an alert rule now requires confirmation. Click the trash icon next to a rule, then confirm in the dialog that appears.
|