---
title: Alerts & Notifications
description: Threshold and event alerts for your fleet, dispatched to Discord, Slack, or any webhook, with per-stack rules and Skipper routing.
---
Sencho watches each node it manages for container crashes, host pressure, scheduled-task results, and update availability, then surfaces every signal in two places: the in-app notification bell at the top of the shell and one of three external channels you configure. This page covers everything from configuring channels to writing per-stack threshold rules, routing alerts to dedicated channels with Skipper routing rules, and tuning retention.
## Notification channels
Open **Settings · Notifications** to configure the three channel types. Each channel is per-node, so switching the active node via the node picker reloads the panel against that node's stored settings. The masthead carries a `CHANNELS` stat showing how many of the three slots are enabled.
Each tab carries the same controls: an **Enabled** toggle (helper: `Send Sencho events to this channel.`), a **Webhook URL** input (placeholder `https://...`, helper: `Sencho posts JSON payloads here. Use a private channel.`), and the **Test** and **Save** buttons. The kicker on each tab toggles between `enabled` and `off` so you can see at a glance which slots are wired up.
Webhook URLs must use HTTPS. Sencho rejects `http://` and any string that does not parse as a URL. The same rule applies to test sends and to routing-rule URLs.
### Discord
Paste an incoming webhook URL from your Discord channel's **Edit Channel · Integrations · Webhooks** view. Sencho posts a single embed per alert, with the title `Sencho Alert []`, the message in the description, and the embed color set by severity (info blue, warning yellow, error red).
### Slack
Paste an incoming webhook URL from a Slack app installed in your workspace (URL shape `https://hooks.slack.com/services/...`). Sencho posts a single text message: ` *Sencho Alert []*` followed by the message body, with the emoji set per severity (`ℹ️` info, `⚠️` warning, `🚨` error).
### Generic webhook
Use the Generic Webhook tab when you have your own receiver: a Mattermost or Teams adapter, a serverless function that fans out to email or SMS, or a logging endpoint. Sencho posts JSON of the shape:
```json
{
"level": "warning",
"message": "[Node: Local] The cpu_percent for plex has exceeded your threshold of 80% (Currently: 91%).",
"timestamp": "2026-05-08T22:14:09.812Z",
"source": "sencho"
}
```
`level` is one of `info`, `warning`, `error`. `source` is always the literal string `sencho`. `timestamp` is ISO-8601 with millisecond precision.
### Test sends and delivery semantics
The **Test** button on each tab dispatches the literal message `🔌 Test Notification from Sencho!` at level `info` through the same path a real alert would take. Test sends require the admin role; the server returns 403 if an operator or viewer submits one.
Each dispatch is a single-shot HTTP POST with a 10-second `AbortSignal.timeout`. There is no retry queue. If your endpoint is down at the moment of the dispatch, the alert is recorded in the bell with an internal `dispatch_error` field set and is not redelivered.
## Notification Routing
Notification Routing requires a **Sencho Skipper or Admiral** license. Admin role is required to create, edit, or delete routes.
Routing lets you direct alerts that match specific criteria to dedicated channels. Production crashes can land in `#prod-incidents` on Slack while staging notifications go to a less urgent Discord channel, all without juggling per-channel webhook URLs across teams.
### How routing fits into dispatch
For every alert Sencho dispatches, the routing engine evaluates every enabled route. A route matches when **all of its non-empty matchers** match the alert: the **Node**, **Stacks**, **Labels**, and **Categories** filters compose with AND. An empty matcher is treated as match-anything.
If at least one route matches, every matching route fires and the **global channels are skipped** for that alert. If zero routes match, the alert falls back to the global channels configured in the previous section.
A route with all four matchers left empty matches every alert and intercepts global delivery entirely. Stack-less alerts (host CPU, RAM, disk, fleet-sync warnings, and scheduled-task results without a stack target) almost always miss any populated `Stacks` filter, so they fall back to global by default.
### Creating a routing rule
Open **Settings · Routing** and click **+ Add Route**.
| Field | Purpose |
|-------|---------|
| **Name** | A human label, up to 100 characters. Shown on the rule card. |
| **Node scope** | Either `Any node` or a specific node. When set to a node, the rule only matches alerts originating from that node. |
| **Stacks** *(optional)* | A combobox of stacks on the active node. Selected stacks appear as removable pills. Empty matches any stack. |
| **Labels** *(optional)* | A combobox of stack labels on the active node. Empty matches any label. |
| **Categories** *(optional)* | A combobox of notification categories. The helper line reads `Leave blank to match all categories. All non-empty filters must match (AND).` |
| **Channel** | Tabs for Discord, Slack, and Webhook with a URL input below. URL must use HTTPS. |
| **Priority** | A number used to sort the rule list. Lower numbers appear higher up. Priority does not gate dispatch: when multiple rules match the same alert, every matching rule fires concurrently. |
| **Enabled** | Toggle the rule on or off without deleting it. |
The modal kicker reads `ROUTING · NEW RULE` when adding and `ROUTING · EDIT RULE` when modifying.
### Managing rules
Each rule renders as a card on the Routing page with the rule name, channel-type badge, an `ON` / `OFF` pill, then a row of small badges naming each matcher: one mono badge per **Stack**, one outline badge per **Label**, one outline mono badge per **Category** label. When all matchers are empty, a single muted `Matches all alerts` line replaces the badge row. After the badges, a vertical bar separator is followed by the truncated channel URL, then a second separator and a `Priority: N` chip when priority is non-zero.
Three icon actions appear on the right edge of each card:
- **Lightning** sends a test message through the rule's channel using the same payload shape as the global Test button.
- **Pencil** opens the edit modal pre-filled with the current values.
- **Trash** deletes the rule after a confirmation dialog.
The masthead carries `SCOPE` (`global`), `ROUTES` (total), and `ENABLED` (count).
## Notification categories
Every alert Sencho dispatches carries a category that you can filter on in the bell, target with a routing rule's **Categories** matcher, or reason about when reading audit history.
| Category | Label in the bell | Trigger |
|----------|-------------------|---------|
| `deploy_success` | Deploy success | Stack deploy completed without error |
| `deploy_failure` | Deploy failure | Stack deploy returned a non-zero exit |
| `stack_started` | Stack started | Stack started via the dashboard or API |
| `stack_stopped` | Stack stopped | Stack stopped via the dashboard or API |
| `stack_restarted` | Stack restarted | Stack restarted via the dashboard or API |
| `image_update_available` | Update available | Image-update poll found a newer digest |
| `image_update_applied` | Update applied | Manual or scheduled auto-update applied new images |
| `autoheal_triggered` | Auto-heal | Auto-heal restarted, failed to restart, or auto-disabled a policy |
| `monitor_alert` | Monitor alert | Per-stack threshold breach, host CPU/RAM/disk warning, or healthcheck failure |
| `scan_finding` | Scan finding | Vulnerability-scan completion, per-violation alert, post-deploy scan failure, or auto-update gate block |
| `system` | System | Sencho version, Trivy auto-update, fleet sync, daemon connectivity, scheduled-task lifecycle, cloud-backup upload failure |
| `blueprint_deployed` | `blueprint_deployed` | Blueprint provisioned a new deployment (Admiral) |
| `blueprint_deployment_failed` | `blueprint_deployment_failed` | Blueprint deployment errored out (Admiral) |
| `blueprint_drift_detected` | `blueprint_drift_detected` | Blueprint drift detected in `suggest` or `enforce` mode (Admiral) |
| `blueprint_drift_correction_failed` | `blueprint_drift_correction_failed` | Blueprint enforce-mode redeploy failed (Admiral) |
The four `blueprint_*` categories are accepted by routing rules but render as raw category strings in the bell because the frontend label map omits them.
## Per-stack alert rules
Each stack carries its own set of threshold rules that fire when a metric stays above (or below) a value for a configurable window. Rules live on the node where the stack runs and are evaluated locally on a 30-second tick.
Open the rules editor by right-clicking a stack in the sidebar and choosing **Alerts**, or by pressing `A` while focused on a stack.
### Alert fields
| Field | Purpose |
|-------|---------|
| **Metric** | The system resource or metric to monitor. |
| **Operator** | Comparison: `Greater than`, `Greater or eq`, `Less than`, `Less or eq`, `Equals`. |
| **Threshold** | A number ≥ 0. The unit follows the chosen metric. |
| **Duration (mins)** | How long the breach must persist before firing. Default `5`, range 0 to 1440. |
| **Cooldown (mins)** | Silence window between fires after a rule triggers. Default `60`, range 0 to 10080. |
Sencho tracks the start of each breach in memory; the rule fires only after the breach has lasted for the full **Duration**, and only if the previous fire is older than **Cooldown**.
### Available metrics
| Metric | UI label | Unit |
|--------|----------|------|
| `cpu_percent` | CPU Usage (%) | percent of all cores |
| `memory_percent` | Memory Usage (%) | percent of container memory limit |
| `memory_mb` | Memory Usage (MB) | MB of resident set size, minus cache |
| `net_rx` | Network In (MB/s) | MB per second, computed as a delta between consecutive samples |
| `net_tx` | Network Out (MB/s) | MB per second, computed as a delta between consecutive samples |
| `restart_count` | Restart Count | integer count of Docker-reported restarts |
### Channel banner states
The **NOTIFICATION CHANNELS** banner above the rules list reflects what dispatch will look like for this stack:
- **Loading** is a spinner with `Checking notification channels...` while Sencho asks the target node for its agent state.
- **Remote node** is a blue banner reading `Remote node: `, with the body `Alert rules are stored and evaluated on this remote instance. Notifications are dispatched using that node's configured channels.` A sub-line reports whether the remote has any channels configured.
- **No channels** is an amber banner reading `No notification channels configured`, body `Alert rules will be saved and evaluated, but no notifications will be dispatched. Configure Discord, Slack, or a webhook in Settings → Notifications.`
- **Active** is a green banner reading `Notifications active via Discord, Slack, …` with the configured channels listed.
### Example: alert on high CPU
To page when CPU on a stack stays above 80% for at least five minutes, with no more than one alert per hour:
| Field | Value |
|-------|-------|
| **Metric** | CPU Usage (%) |
| **Operator** | Greater than |
| **Threshold** | `80` |
| **Duration (mins)** | `5` |
| **Cooldown (mins)** | `60` |
### Permissions and validation
Operators and viewers see existing rules read-only; only admins see the add and delete affordances. Submitting an empty threshold surfaces `Please enter a threshold.` Successful saves toast `Alert rule added.` and `Alert rule deleted.` A failed POST surfaces `Network error. Could not reach the node.`
The delete confirmation dialog reads `Delete Alert Rule` / `This will permanently remove this alert rule. Notifications for this condition will no longer be sent.` with a destructive **Delete** button.
## Notification history (the bell)
The bell icon at the top of the shell is the live feed of every alert across the fleet. A pulsing red dot surfaces unread items.
### Anatomy
A title bar with `Notifications` in the italic display face, the unread count to the right (` UNREAD`), and a row of controls below.
The segmented control switches between `All`, `Unread` (with badge), and `Alerts` (rows where level is `warning` or `error`). To the right, three icon buttons:
- **SlidersHorizontal** toggles a hidden filter row. A small dot on the icon means at least one filter is active.
- **CheckCheck** is `Mark all read`.
- **Trash2** is `Clear all`. The action issues `DELETE /api/notifications` against every node that contributed a row.
The hidden filter row has two combobox dropdowns: **Filter by node** (only rendered with two or more nodes registered) and **Filter by category**, listing all the user-facing labels from the categories table above.
### Row anatomy
Each row carries:
- A **3 px left rail** colored by severity: brand cyan for `info`, amber for `warning`, destructive rose for `error`. Rail saturation drops to about 30% once the row is read.
- A **severity icon** in the same hue: `Info`, `AlertTriangle`, or `AlertOctagon`.
- The **message body**, slightly dimmer once read.
- A **kicker line** below the message with an optional **node-name pill** (only rendered for multi-node fleets), a `·` separator, and a relative timestamp (`just now`, `Nm ago`, `Nh ago`, `yesterday`, then a short locale date).
- A **dismiss** target on hover, top-right.
There is no level chip in the row; the rail color and icon do that work. Rows whose payload carries a `stack_name` become click targets that jump to the stack and, when a `container_name` is present, surface that container's logs.
### Day groupings
Rows are grouped under uppercase day-band headers: `TODAY`, `YESTERDAY`, `THIS WEEK`, `EARLIER`.
### Empty states
The popover renders three different empty states:
- All filters, no rows ever: `You're all caught up` over `New notifications appear here in real time.`
- Unread filter, all read: `No unread notifications` over `Everything in your feed has been read.`
- Alerts filter, no warnings or errors: `No active alerts` over `Warnings and errors will surface here when they occur.`
### What you won't see in the bell
Sencho deliberately suppresses one class of notification from the popover: rows where the category is one of `deploy_success`, `stack_started`, `stack_stopped`, `stack_restarted`, or `image_update_applied`, AND the row carries an `actor_username` other than `system`. These are confirmations of an action you just clicked, surfaced as a toast at the moment of the action; replaying them in the bell would be noise. The rows are still persisted to `notification_history` and still dispatched to global channels and matching routes; only the bell render hides them.
### Limits
The popover holds the latest 50 rows per node per fetch. The backend caps `notification_history` at 100 rows per node, regardless of the user-tunable retention setting; the cap evicts the oldest rows on every insert. There is no `Load more` affordance; older rows roll off the cap.
## Cross-node delivery
The bell aggregates across the entire fleet by hitting `/api/notifications` against every registered node and opening one WebSocket per node: `/ws/notifications` for the local instance and `/ws/notifications?nodeId=` for every remote. A new alert on any remote arrives in the bell within about a second; a 60-second safety-net poll catches anything the WebSocket dropped.
Per-stack alert rules and channel configuration are **stored on the node where the stack runs**. To configure a rule on a remote, switch the active node via the picker, open the stack's **Monitor** sheet, and the form's POST is forwarded to the remote.
Crash detection runs only on local Docker; remote nodes run their own copy of `DockerEventService` and emit through the proxy. Each emitted message is prefixed with `[Node: ]` so the source is unambiguous in the channel.
Switching the active node only affects per-stack rule editing and the Settings panels. The bell aggregates every node regardless of which one is active.
## Alerts emitted by the system
Sencho dispatches notifications from many code paths. The list below covers every emitter in current code, organized by source. Each message is prefixed with `[Node: ]` when emitted by a node-scoped service.
### Container crash and health
Real-time on the Docker event stream:
- **Crash** at `error`/`monitor_alert`: `Container Crash Detected: exited unexpectedly (Code: ).`
- **OOM kill** at `error`/`monitor_alert`: `Container OOM Kill: was killed by the OOM killer (out of memory).` When a `die` arrives with exit code 137 but no preceding `oom` event, Sencho inspects the container and reclassifies as OOM if the `OOMKilled` flag is set.
- **Healthcheck failure** at `error`/`monitor_alert`: `Healthcheck failed: is unhealthy.`
- **Mass exit** kicks in when a daemon-disconnect gap is followed by 20% or more of containers exiting on reconnect, in which case a single `info`/`system` summary `Docker daemon interruption detected: N containers exited during connection gap.` is emitted instead of N crash alerts.
- **Rate limit** caps crash dispatches at 20 per 60-second window per node, then a single `warning`/`monitor_alert` `N additional containers crashed in the last minute.`
- **Dedup** is 60 minutes per container after a non-rate-suppressed dispatch.
### Daemon connectivity
- `warning`/`system`: `Lost connection to Docker daemon; monitoring paused.` (one-shot until reconnect)
- `info`/`system`: `Reconnected to Docker daemon.`
- `warning`/`system`: `Received malformed Docker event payloads. Monitoring continues but some events may be skipped.` (when more than 10 parse errors in a minute)
### Host thresholds
`warning`/`monitor_alert` for host CPU, RAM, and disk when the configured threshold is exceeded. 5-minute cooldown per signal. Example: `Host CPU utilization is critically high: 92% (Threshold: 90%)`. Evaluated on the 30-second monitor tick.
### Docker janitor
`info`/`system`: `Node "" has accumulated GB of unused Docker data. Consider using the Janitor tool.` 24-hour cooldown.
### Sencho version availability
`info`/`system`: `Sencho X.Y.Z is available (currently running A.B.C). Visit the Fleet dashboard to update.` Polled every 6 hours; one-shot dedup until the running version reaches or passes the notified version.
### Image update availability
`info`/`image_update_available`: `Stack "" has image updates available.` Polled every 6 hours, with a 2-minute startup delay and a 2-minute cooldown on manual refresh. Notifies on **state transition only**; pre-existing `has_update` rows are backfilled silently on first run.
### Auto-update execution
`info`/`image_update_applied`: `Auto-update: stack "" updated with new images`. If a block-on-deploy policy gates the auto-update: `warning`/`scan_finding` `Policy "" blocked auto-update: N image(s) exceed `. See [Auto-update policies](/features/auto-update-policies).
### Auto-heal
- `info`/`autoheal_triggered`: `Auto-Heal: Restarted on stack after being unhealthy for minute(s).`
- `warning`/`autoheal_triggered`: `Auto-Heal: Failed to restart on stack . Error: `
- `warning`/`autoheal_triggered`: `Auto-Heal: Policy for [/] has been auto-disabled after consecutive failures.`
See [Auto-heal policies](/features/auto-heal-policies).
### Vulnerability scanning
- **Per-violation, during a scheduled scan** is `warning`/`scan_finding`. One alert per violation: `Policy "" violated by : exceeds `.
- **Scan completion** is `info`/`scan_finding` (or `warning` if any image failed): `Scheduled scan "" completed: