sencho/docs/features/alerts-notifications.mdx

---
title: Alerts & Notifications
description: Set threshold-based alerts on container metrics and route them to Discord, Slack, or any webhook.
---

Sencho can watch your containers for resource anomalies and notify you when thresholds are breached. Alerts are defined per stack, and notifications are delivered through external channels you configure.

## Setting up notification channels

At least one channel must be enabled before alerts can be delivered externally. Go to **Settings > Notifications**.

<Frame>
  <img src="/images/alerts-notifications/notifications-settings.png" alt="Notifications & Alerts settings showing Discord, Slack, and Webhook tabs" />
</Frame>

Three channel types are available, each configured with a webhook URL and an enable/disable toggle:

### Discord

1. In Discord, go to your server's **Settings > Integrations > Webhooks**
2. Click **New Webhook**, choose a channel, and copy the webhook URL
3. In Sencho, open **Settings > Notifications > Discord**, paste the URL, enable the toggle, and click **Save**
4. Click **Test** to send a test message and verify the connection

### Slack

1. In Slack, go to **api.slack.com/apps**, create an app, and add the **Incoming Webhooks** feature
2. Activate it and copy the generated webhook URL for your chosen channel
3. In Sencho, open **Settings > Notifications > Slack**, paste the URL, enable the toggle, and click **Save**

### Generic Webhook

Any HTTPS endpoint that accepts a POST with a JSON body can receive Sencho alerts. Go to **Settings > Notifications > Webhook**, enter the URL, enable the toggle, and click **Save**.

<Note>
  All webhook URLs (Discord, Slack, and generic) must use HTTPS. HTTP URLs are rejected.
</Note>

The payload format is:

```json
{
  "level": "warning",
  "message": "cpu_percent exceeded 90% for 5 minutes on stack my-app",
  "timestamp": "2026-03-22T10:00:00.000Z"
}
```

## Creating stack alerts

Stack alerts are configured per stack. Right-click a stack in the sidebar (or click the three-dot menu) and select **Alerts**. A panel slides open showing existing rules and a form to create new ones.

<Frame>
  <img src="/images/alerts-notifications/alert-panel.png" alt="Stack Alerts panel showing the notification status banner and alert rule form" />
</Frame>

The panel includes:

- **Notification status banner** at the top, showing whether channels are configured. If no channels are enabled, a warning explains that rules will be evaluated but no external notifications will be sent.
- **Existing Rules** section listing all active rules for this stack, each showing the metric, condition, duration, and cooldown. Hover over a rule to reveal the delete button.
- **Add New Rule** form with the fields described below.

### Alert fields

| Field | Description |
|-------|-------------|
| **Metric** | The container metric to watch (see table below) |
| **Operator** | Comparison: Greater than, Greater or equal, Less than, Less or equal, Equals |
| **Threshold** | The numerical value to compare against |
| **Duration (mins)** | How long the condition must hold before firing (default: 5) |
| **Cooldown (mins)** | Minimum time between repeated notifications for this rule (default: 60) |

### Available metrics

| Metric | Description |
|--------|-------------|
| CPU Usage (%) | CPU usage relative to total host cores |
| Memory Usage (%) | Memory used as a fraction of the host total |
| Memory Usage (MB) | RSS memory used by the container |
| Network In (MB) | Cumulative inbound network bytes (in MB) |
| Network Out (MB) | Cumulative outbound network bytes (in MB) |
| Restart Count | Number of times the container has restarted |

### Example: alert on high CPU

To alert when any container in a stack uses more than 80% CPU for over 5 consecutive minutes, with a 60-minute cooldown:

| Field | Value |
|-------|-------|
| Metric | CPU Usage (%) |
| Operator | Greater than |
| Threshold | 80 |
| Duration | 5 |
| Cooldown | 60 |

## Notification history

All dispatched notifications appear in the **notification bell** in the top-right corner of the navigation bar. A red dot pulses on the bell when unread notifications exist.

<Frame>
  <img src="/images/alerts-notifications/notification-popover.png" alt="Notification popover showing recent alert entries with level badges and timestamps" />
</Frame>

Click the bell to open the notification popover, which shows:

- **Level badge** for each entry (ERROR in red, WARNING in amber, INFO in default)
- **Node name** badge for notifications from remote nodes
- **Timestamp** of when the alert was triggered
- **Alert message** describing what was breached

The popover header includes:

| Action | What it does |
|--------|--------------|
| **Mark all as read** | Marks all notifications as read (removes the red dot) |
| **Clear all** | Deletes all notifications from the list |

You can also dismiss individual notifications by hovering over them and clicking the dismiss button.

<Note>
  Notifications only reach external channels (Discord, Slack, Webhook) if at least one channel is enabled. Dashboard notifications appear regardless.
</Note>

## Alerts on remote nodes

Alerts work the same way on remote nodes as they do locally. When you switch to a remote node and open a stack's alerts panel, you are managing rules on that remote instance.

Key details:

- **Alert rules are stored on each node independently.** Rules created while a remote node is selected are saved on that remote instance, not your primary instance.
- **Monitoring runs locally on each node.** Each Sencho instance evaluates its own alert rules against its own container metrics.
- **Notifications are sent by the node that detects the breach.** Make sure notification channels are configured on each remote node where you want to receive alerts, since channel settings are per-instance.

The alert panel shows a blue info banner when you are configuring alerts on a remote node, including which channels are active on that node.

### Setup checklist for remote alerts

1. On the **remote** Sencho instance, go to **Settings > Notifications** and configure at least one channel
2. From your **primary** instance, switch to the remote node
3. Right-click a stack and select **Alerts** to create rules
4. The remote instance handles monitoring and notification delivery independently

## Update availability notifications

Sencho can notify you when software updates are available, both for Sencho itself and for your stack images.

### Sencho version updates

When a newer version of Sencho is published, an informational notification is dispatched through your configured channels. Each Sencho instance runs its own version check roughly once every 6 hours and notifies exactly once per new version, so remote nodes self-report their own availability alerts. After you update, the cycle resets and you will be notified when the next release becomes available.

The notification message includes the version you are running and the version that was detected, for example:

```
Sencho 0.47.0 is available (currently running 0.46.16). Visit the Fleet dashboard to update.
```

### Stack image updates

When the periodic image check (every 6 hours) detects that a stack has new upstream images available, a notification is dispatched for each affected stack. Notifications are sent only on state transitions: you will be notified once when an update first appears, not on every check cycle. After you update the stack, the status resets so a future release can notify again.

<Note>
  If you upgraded to a Sencho version that added image update notifications and you already had stacks flagged as having updates, the first check after the upgrade sends one catch-up notification per affected stack so the in-app bell stays in sync with the blue update indicator.
</Note>

Both notification types use the same channel routing as alerts: if notification routes are configured for a stack, those channels receive the message; otherwise, global notification channels are used as a fallback.

## Troubleshooting

### Notifications not being delivered

- Verify at least one notification channel is enabled in **Settings > Notifications**
- Click **Test** on the channel to confirm the webhook URL is reachable
- Check that the webhook URL uses HTTPS
- If using notification routing (Admiral tier), verify the route pattern matches the stack name

### Alert not firing

- Confirm the alert rule exists by opening the stack's alert panel
- Check that the **duration** has elapsed; the condition must hold continuously for the configured duration before an alert fires
- Check the **cooldown** period; after an alert fires, it will not fire again until the cooldown expires
- Verify the metric is being collected; container must be running for stats to be gathered

### Delete confirmation dialog

Deleting an alert rule now requires confirmation. Click the trash icon next to a rule, then confirm in the dialog that appears.

### Version notification shows "currently running 0.0.0"

This affected older Sencho builds that read the running version from an environment variable which was not always populated. Update to the current release; the running version is now resolved from the packaged manifest and the notification will report it correctly the next time a new release is detected.

### A stack shows the blue update indicator but no notification was received

If the blue indicator was already visible before you upgraded to a Sencho version that supports image update notifications, the first image check after the upgrade sends one catch-up notification per affected stack. Subsequent checks notify only when a stack newly transitions from "no updates" to "updates available".

If the indicator appeared after the upgrade but no notification followed, open the notification bell. Dispatch failures (for example, a misconfigured Discord or Slack webhook) are logged as error entries in the bell so they are visible without tailing logs. Confirm at least one channel is enabled in **Settings > Notifications**; the in-app bell receives all notifications regardless of external channel configuration.