sencho/docs/features/alerts-notifications.mdx

---
title: Alerts & Notifications
description: Set threshold-based alerts on container metrics and route them to Discord, Slack, or any webhook.
---

Sencho can watch your containers for resource anomalies and notify you when thresholds are breached. Alerts are defined per stack, and notifications are delivered through external channels you configure.

## Setting up notification channels

At least one channel must be enabled before alerts can be delivered externally. Go to **Settings > Notifications**.

<Frame>
  <img src="/images/alerts-notifications/notifications-settings.png" alt="Notifications & Alerts settings showing Discord, Slack, and Webhook tabs" />
</Frame>

Three channel types are available, each configured with a webhook URL and an enable/disable toggle:

### Discord

1. In Discord, go to your server's **Settings > Integrations > Webhooks**
2. Click **New Webhook**, choose a channel, and copy the webhook URL
3. In Sencho, open **Settings > Notifications > Discord**, paste the URL, enable the toggle, and click **Save**
4. Click **Test** to send a test message and verify the connection

### Slack

1. In Slack, go to **api.slack.com/apps**, create an app, and add the **Incoming Webhooks** feature
2. Activate it and copy the generated webhook URL for your chosen channel
3. In Sencho, open **Settings > Notifications > Slack**, paste the URL, enable the toggle, and click **Save**

### Generic Webhook

Any HTTPS endpoint that accepts a POST with a JSON body can receive Sencho alerts. Go to **Settings > Notifications > Webhook**, enter the URL, enable the toggle, and click **Save**.

<Note>
  All webhook URLs (Discord, Slack, and generic) must use HTTPS. HTTP URLs are rejected.
</Note>

The payload format is:

```json
{
  "level": "warning",
  "message": "cpu_percent exceeded 90% for 5 minutes on stack my-app",
  "timestamp": "2026-03-22T10:00:00.000Z"
}
```

## Creating stack alerts

Stack alerts are configured per stack. Right-click a stack in the sidebar (or click the three-dot menu) and select **Alerts**. A panel slides open showing existing rules and a form to create new ones.

<Frame>
  <img src="/images/alerts-notifications/alert-panel.png" alt="Stack Alerts panel showing the notification status banner and alert rule form" />
</Frame>

The panel includes:

- **Notification status banner** at the top, showing whether channels are configured. If no channels are enabled, a warning explains that rules will be evaluated but no external notifications will be sent.
- **Existing Rules** section listing all active rules for this stack, each showing the metric, condition, duration, and cooldown. Hover over a rule to reveal the delete button.
- **Add New Rule** form with the fields described below.

### Alert fields

| Field | Description |
|-------|-------------|
| **Metric** | The container metric to watch (see table below) |
| **Operator** | Comparison: Greater than, Greater or equal, Less than, Less or equal, Equals |
| **Threshold** | The numerical value to compare against |
| **Duration (mins)** | How long the condition must hold before firing (default: 5) |
| **Cooldown (mins)** | Minimum time between repeated notifications for this rule (default: 60) |

### Available metrics

| Metric | Description |
|--------|-------------|
| CPU Usage (%) | CPU usage relative to total host cores |
| Memory Usage (%) | Memory used as a fraction of the host total |
| Memory Usage (MB) | RSS memory used by the container |
| Network In (MB) | Cumulative inbound network bytes (in MB) |
| Network Out (MB) | Cumulative outbound network bytes (in MB) |
| Restart Count | Number of times the container has restarted |

### Example: alert on high CPU

To alert when any container in a stack uses more than 80% CPU for over 5 consecutive minutes, with a 60-minute cooldown:

| Field | Value |
|-------|-------|
| Metric | CPU Usage (%) |
| Operator | Greater than |
| Threshold | 80 |
| Duration | 5 |
| Cooldown | 60 |

## Notification history

All dispatched notifications appear in the **notification bell** in the top-right corner of the navigation bar. A red dot pulses on the bell when unread notifications exist.

<Frame>
  <img src="/images/alerts-notifications/notification-popover.png" alt="Notification popover showing recent alert entries with level badges and timestamps" />
</Frame>

Click the bell to open the notification popover, which shows:

- **Level badge** for each entry (ERROR in red, WARNING in amber, INFO in default)
- **Node name** badge for notifications from remote nodes
- **Timestamp** of when the alert was triggered
- **Alert message** describing what was breached

The popover header includes:

| Action | What it does |
|--------|--------------|
| **Mark all as read** | Marks all notifications as read (removes the red dot) |
| **Clear all** | Deletes all notifications from the list |

You can also dismiss individual notifications by hovering over them and clicking the dismiss button.

<Note>
  Notifications only reach external channels (Discord, Slack, Webhook) if at least one channel is enabled. Dashboard notifications appear regardless.
</Note>

## Alerts on remote nodes

Alerts work the same way on remote nodes as they do locally. When you switch to a remote node and open a stack's alerts panel, you are managing rules on that remote instance.

Key details:

- **Alert rules are stored on each node independently.** Rules created while a remote node is selected are saved on that remote instance, not your primary instance.
- **Monitoring runs locally on each node.** Each Sencho instance evaluates its own alert rules against its own container metrics.
- **Notifications are sent by the node that detects the breach.** Make sure notification channels are configured on each remote node where you want to receive alerts, since channel settings are per-instance.

The alert panel shows a blue info banner when you are configuring alerts on a remote node, including which channels are active on that node.

### Setup checklist for remote alerts

You can configure a remote node's notification channels directly from the control plane:

1. From your **primary** instance, switch to the remote node using the node picker
2. Open **Settings > Notifications** and configure the remote's Discord, Slack, or Webhook channels. Values saved here apply only to the selected node.
3. Right-click a stack and select **Alerts** to create rules
4. The remote instance handles monitoring and notification delivery independently

## Update availability notifications

Sencho can notify you when software updates are available, both for Sencho itself and for your stack images.

### Sencho version updates

When a newer version of Sencho is published, an informational notification is dispatched through your configured channels. Each Sencho instance runs its own version check roughly once every 6 hours and notifies exactly once per new version, so remote nodes self-report their own availability alerts. After you update, the cycle resets and you will be notified when the next release becomes available.

The notification message includes the version you are running and the version that was detected, for example:

```
Sencho 0.47.0 is available (currently running 0.46.16). Visit the Fleet dashboard to update.
```

### Stack image updates

When the periodic image check (every 6 hours) detects that a stack has new upstream images available, a notification is dispatched for each affected stack. Notifications are sent only on state transitions: you will be notified once when an update first appears, not on every check cycle. After you update the stack, the status resets so a future release can notify again.

Both notification types use the same channel routing as alerts: if notification routes are configured for a stack, those channels receive the message; otherwise, global notification channels are used as a fallback.

## Scheduled scan completion

Recurring [vulnerability scans](/features/vulnerability-scanning) dispatch a notification whenever a run finishes so you do not have to check the task history manually.

- **Info** when every image scanned successfully.
- **Warning** when one or more images failed to scan during the run.
- **Error** when the run itself could not start (for example, Trivy is not installed on the target node).

The message includes the scheduled task name, a summary of how many images were scanned, cached, and failed, and a breakdown of findings by severity. A typical clean-run message looks like:

```
Scheduled scan "nightly-scan" completed: Scanned 12 image(s); 3 skipped (cached). Found 2 critical, 5 high, 10 medium.
```

If no critical, high, or medium findings are present, the message ends with `No critical, high, or medium findings.` so the outcome is still explicit. When the target node has nothing to scan, the message reads `No images to scan.`, and when every image was already covered by a recent cached scan it reads `All N image(s) already scanned recently (cache hit).` In both cases the notification still fires so you know the run executed.

<Note>
  Severity counts reflect the current security posture of the node, aggregated across both freshly scanned images and cached scan results. They are not a delta of what changed on this run.
</Note>

Failures are typically transient (registry timeouts, missing credentials) and do not stop the rest of the run from completing.

## Container crash detection

Sencho watches every container on each of your nodes in real time and notifies you when something exits unexpectedly. Detection is causal: Sencho distinguishes crashes from intentional stops, so stopping a stack, restarting it, or running `docker compose down` will not produce a false crash alert.

### What triggers a crash alert

| Situation | Alert |
|---|---|
| Container exits with a non-zero exit code without being asked to stop | **Crash** alert (level: error) |
| Container is killed by the kernel for exceeding its memory limit | **OOM Kill** alert (level: error) |
| Healthcheck reports the container as unhealthy | **Healthcheck failed** alert (level: error) |

Alerts arrive within a couple of seconds of the event, not on a polling interval.

### What does not trigger a crash alert

- Stopping, restarting, updating, or removing a stack from Sencho
- Running `docker stop`, `docker restart`, or `docker compose down` from a terminal on the host
- A container exiting cleanly with exit code `0`
- A container being replaced during an image update

### Global toggle

Crash and unhealthy alerts are gated by the **Global Crash Detection** toggle in **Settings > System**. When disabled, Sencho stops dispatching crash and unhealthy notifications on all nodes. Stack metric alerts and update notifications are unaffected by this toggle.

### Docker daemon interruptions

If the Docker daemon becomes unreachable (daemon restart, socket lost, network issue on a remote node), Sencho sends a single **Lost connection to Docker daemon** warning and pauses crash detection on the affected node. When the connection is restored, Sencho reconciles container state against a pre-disconnect snapshot:

- If most of your containers are still running, individual gap exits are classified and alerts are sent as normal.
- If a large share of containers exited during the outage (for example after a daemon restart), Sencho consolidates them into a single **Docker daemon interruption detected** informational notification instead of paging you for every container.

A matching **Reconnected to Docker daemon** info notification confirms monitoring has resumed.

### High-churn events

When many crashes land in a short window (for example, a large stack coming down unexpectedly), Sencho dispatches an initial batch and summarizes the remainder in a single **N additional containers crashed in the last minute** entry so your notification channels are not flooded.

## Troubleshooting

### Notifications not being delivered

- Verify at least one notification channel is enabled in **Settings > Notifications**
- Click **Test** on the channel to confirm the webhook URL is reachable
- Check that the webhook URL uses HTTPS
- If using notification routing (Admiral tier), verify the route pattern matches the stack name

### Alert not firing

- Confirm the alert rule exists by opening the stack's alert panel
- Check that the **duration** has elapsed; the condition must hold continuously for the configured duration before an alert fires
- Check the **cooldown** period; after an alert fires, it will not fire again until the cooldown expires
- Verify the metric is being collected; container must be running for stats to be gathered

### I stopped a stack but got a crash alert

This should not happen. Sencho distinguishes intentional stops from crashes by correlating each container exit with the action that triggered it. If you do see a crash alert for a stack you stopped, confirm Sencho can reach the Docker daemon on the affected node: the **Lost connection to Docker daemon** warning is dispatched when the socket is unreachable, and exits observed during an outage may be classified as crashes on reconnect.

### After a Docker daemon restart I only got one summary notification

Expected. When a large share of containers exits during a Docker daemon interruption, Sencho consolidates them into a single informational notification instead of paging per container. Individual crashes that happen after the reconnect are alerted on as normal.

### Crash alerts stopped arriving and nothing else looks wrong

Check, in order:

1. Open **Settings > System** and confirm **Global Crash Detection** is on. When it is off, crash, OOM, and unhealthy notifications stop on all nodes while stack metric alerts and update notifications continue as normal.
2. Look in the in-app notifications panel for a **Lost connection to Docker daemon** warning. If present, Sencho is not receiving events from the affected node and is retrying in the background; a **Reconnected to Docker daemon** info entry will appear when monitoring resumes.
3. Confirm the affected node is reachable from the host running Sencho. For remote nodes, the remote Sencho instance is responsible for its own crash detection; check its status from that instance.

### A burst of crashes happened but I only see about twenty alerts

Expected, and by design. Sencho caps crash notifications at around 20 per minute per node so a runaway restart loop or a stack-wide failure cannot flood your notification channels. Any crashes beyond the cap in that window are rolled up into a single **N additional containers crashed in the last minute** warning. The complete, unredacted list of events is always visible in the in-app notifications panel; only external channel delivery (Discord, Slack, webhooks) is rate-limited.

### Delete confirmation dialog

Deleting an alert rule now requires confirmation. Click the trash icon next to a rule, then confirm in the dialog that appears.

### Fleet shows an update button but I never received a Sencho version notification

Fleet and the in-app bell share the same version lookup. Each Sencho instance checks for a new release roughly every 6 hours and notifies exactly once per version, so if a check already fired for the latest release the bell will not notify again for that same version.

If the update button is visible but no notification is in the bell, a transient network failure during the last successful check can briefly delay delivery; the next evaluation cycle (about every 30 seconds) retries automatically. To confirm external channels (Discord, Slack, webhooks) are reachable, open **Settings > Notifications** and click **Test** on each. The in-app bell receives every notification regardless of external channel configuration.

### A stack shows the blue update indicator but no notification was received

Open the notification bell. Dispatch failures (for example, a misconfigured Discord or Slack webhook) are logged as error entries in the bell so they are visible without tailing logs. Confirm at least one channel is enabled in **Settings > Notifications**; the in-app bell receives all notifications regardless of external channel configuration.