---
title: Alerts & Notifications
sidebarTitle: Alerts and notifications
description: Threshold and event alerts for your fleet, dispatched to Discord, Slack, Apprise, ntfy, or any webhook, with stack and service rules and channel routing.
---

Sencho watches each node it manages for container crashes, host pressure, scheduled-task results, and update availability, then surfaces every signal in two places: the in-app notification bell at the top of the shell and an external channel you configure. This page covers everything from configuring channels to writing stack and service threshold rules, routing alerts to dedicated channels with routing rules, and tuning retention.

<Frame>
  <img src="/images/alerts-notifications/notifications-settings.png" alt="Settings · Notifications · Channels panel with NODE Local in the header, Discord Slack Webhook Apprise and ntfy tabs with Apprise selected, Enabled off, an empty Apprise endpoint placeholder, Test beside Save, and a Delivery retries section below showing Extra attempts 0 and a Save retries button." />
</Frame>

## Notification channels

Open **Settings · Notifications · Channels** to configure Discord, Slack, custom webhook, Apprise, and ntfy channels. Each channel is per-node, so switching the active node via the node picker reloads the panel against that node's stored settings. The masthead carries a `CHANNELS` stat showing how many of the slots are enabled.

Below the channel tabs, **Delivery retries** sets how many extra in-process attempts (0 to 3) Sencho makes after a transient delivery failure on that node. The default is `0` (single-shot). Extra attempts wait a fixed one second between tries. Admin role is required to change the value.

Each Discord, Slack, and Webhook tab carries an **Enabled** toggle, a **Webhook URL** input (HTTPS only), and **Test** / **Save**. The Apprise tab uses an **Apprise endpoint**: keyed `/notify/{key}` shows optional **Tags**; stateless `/notify` shows **Destination URLs**. The ntfy tab accepts an ntfy server and topic URL with an **Enabled** toggle and **Test** / **Save**. The kicker on each tab toggles between `enabled` and `off` so you can see at a glance which slots are wired up.

<Note>
  Discord, Slack, and webhook URLs must use HTTPS. Apprise and ntfy endpoints may use HTTP or HTTPS. Every endpoint must parse as a URL.
</Note>

### Discord

Paste an incoming webhook URL from your Discord channel's **Edit Channel · Integrations · Webhooks** view. Sencho posts a single embed per alert, with the title `Sencho Alert [<LEVEL>]`, the message in the description, and the embed color set by severity (info blue, warning yellow, error red).

### Slack

Paste an incoming webhook URL from a Slack app installed in your workspace (URL shape `https://hooks.slack.com/services/...`). Sencho posts a single text message: `<emoji> *Sencho Alert [<LEVEL>]*` followed by the message body, with the emoji set per severity (`ℹ️` info, `⚠️` warning, `🚨` error).

### Generic webhook

Use the Generic Webhook tab when you have your own receiver: a Mattermost or Teams adapter, a serverless function that fans out to email or SMS, or a logging endpoint. Sencho posts JSON of the shape:

```json
{
  "level": "warning",
  "message": "The **CPU usage** for **api** in **plex** (container **plex-api-1**) has exceeded your threshold of **80%** (Currently: 91%).",
  "timestamp": "2026-05-08T22:14:09.812Z",
  "source": "sencho"
}
```

`level` is one of `info`, `warning`, `error`. `source` is always the literal string `sencho`. `timestamp` is ISO-8601 with millisecond precision.

### Apprise

Use a keyed Apprise endpoint ending in `/notify/<key>` with optional tags, or a stateless endpoint ending in `/notify` with one or more destination URLs. Separate stateless destination URLs with commas or whitespace. Apprise endpoints may use HTTP for a local gateway.

Sencho sends the alert title, body, severity, and either tags or destination URLs to Apprise. An HTTP 204 response means Apprise accepted no delivery and is reported as a failed test or dispatch. Configuration reads expose only destination provider names and counts, never destination URLs or endpoint keys.

### Test sends and delivery semantics

The **Test** button on each tab dispatches the literal message `🔌 Test Notification from Sencho!` at level `info` through the same path a real alert would take, including the node's delivery-retries setting. Test sends require the admin role; the server returns 403 for any non-admin user.

Each delivery attempt is an HTTP POST with a 10-second `AbortSignal.timeout`. By default (`Delivery retries` = 0) Sencho makes one attempt. You can allow up to three extra in-process attempts with a fixed one-second delay between them. Retries apply only to classified transient failures (for example HTTP 5xx or network timeouts). Client errors such as HTTP 4xx and Apprise HTTP 204 are not retried. There is no durable retry queue: if the process exits mid-dispatch, remaining attempts are not persisted. Delivery is at-least-once under ambiguous timeouts or connection resets, so a receiver that accepted a request whose response was lost can receive a duplicate. If every attempt fails, the alert remains in the bell with `dispatch_error` set.

### Payload templates

Each channel tab carries an **Edit Payload** toggle below the **Enabled** switch. Opening it reveals a JSON editor. When you save a template, it replaces Sencho's built-in body for that channel: every alert is posted as your JSON with the variables below substituted in. Leaving the editor blank restores the built-in payload.

The following variables are available:

| Variable | Content |
|----------|---------|
| `{{level}}` | `info`, `warning`, or `error` |
| `{{message}}` | The alert message text |
| `{{category}}` | The notification category (for example `deploy_failure` or `monitor_alert`) |
| `{{timestamp}}` | ISO-8601 timestamp of the alert |
| `{{stack_name}}` | The stack the alert concerns, empty when none |
| `{{actor}}` | The user who triggered the action, empty for automated alerts |

Variables are replaced with JSON-escaped values, so quotes or newlines inside a value cannot break the document. A variable can stand alone as a whole string (`"{{message}}"`) or appear inside a string (`"status: {{level}}"`). A variable with no context (for example `{{stack_name}}` on a node-level alert) becomes an empty string. `container_name` is not available.

Sencho validates the template on save: it must be valid JSON after variable substitution, it may only use the variables above, and it may be at most 8000 characters. Unknown variables and malformed JSON are rejected before anything is saved.

Two channel-specific behaviors:

- **ntfy**: with a template, Sencho publishes JSON instead of the usual plain-text message. The Title, Priority, and Tags headers are not sent.
- **Apprise**: destination URLs (stateless endpoints) and tags (keyed endpoints) stay managed by the channel fields. They are merged into your rendered body automatically, and a template cannot set `urls` or `tag` itself. The template must be a JSON object.

The **Test** button uses the template currently in the editor, so you can validate a payload before saving it. Notification routes always use the built-in payload for their channel; templates apply to the global channels in this section.

For example, a webhook receiver that wants a flat structure:

```json
{
  "title": "{{level}}",
  "body": "{{message}}",
  "category": "{{category}}",
  "timestamp": "{{timestamp}}",
  "stack": "{{stack_name}}",
  "actor": "{{actor}}"
}
```

An ntfy topic that publishes JSON:

```json
{
  "topic": "sencho-alerts",
  "message": "[{{level}}] {{message}}",
  "tags": ["warning"]
}
```

## Notification Routing

<Note>
  Admin role is required to create, edit, or delete routes.
</Note>

Routing lets you direct alerts that match specific criteria to dedicated channels. Production crashes can land in `#prod-incidents` on Slack while staging notifications go to a less urgent Discord channel, all without juggling per-channel webhook URLs across teams.

### How routing fits into dispatch

For every alert Sencho dispatches, the routing engine evaluates every enabled route. A route matches when **all of its non-empty matchers** match the alert: the **Node**, **Stacks**, **Labels**, **Categories**, and **Severity** filters compose with AND. An empty matcher is treated as match-anything.

If at least one route matches, every matching route fires and the **global channels are skipped** for that alert. If zero routes match, the alert falls back to the global channels configured in the previous section.

A route with all five matchers left unconstrained (`Node` = any, and empty Stacks, Labels, Categories, and Severity) matches every alert and intercepts global delivery entirely. A route scoped to a node with the other four matchers empty matches every alert on that node. Stack-less alerts (host CPU, RAM, disk, fleet-sync warnings, and scheduled-task results without a stack target) almost always miss any populated `Stacks` filter, so they fall back to global by default.

### Creating a routing rule

Open **Settings · Notifications · Routing** and click **+ Add route**.

| Field | Purpose |
|-------|---------|
| **Name** | A human label, up to 100 characters. Shown on the rule card. |
| **Node scope** | Either `Any node` or a specific node. When set to a node, the rule only matches alerts originating from that node. |
| **Stacks** *(optional)* | Free-form stack patterns with `*` as the only wildcard (for example `prod-*`), plus an optional picker that inserts a known exact stack name as a chip. Empty matches any stack. |
| **Labels** *(optional)* | A combobox of stack labels on the active node. Empty matches any label. |
| **Categories** *(optional)* | A combobox of notification categories. The helper line reads `Leave blank to match all categories. All non-empty filters must match (AND).` |
| **Severity** *(optional)* | One or more of info, warning, or error. Empty matches any severity. |
| **Channel** | Tabs for Discord, Slack, Webhook, Apprise, and ntfy. Discord, Slack, and Webhook require HTTPS. Apprise and ntfy accept HTTP or HTTPS. |
| **Priority** | A number used to sort the rule list. Lower numbers appear higher up. Priority does not gate dispatch: when multiple rules match the same alert, every matching rule fires concurrently. |
| **Enabled** | Toggle the rule on or off without deleting it. |

The modal kicker reads `ROUTING · NEW RULE` when adding and `ROUTING · EDIT RULE` when modifying.

### Managing rules

Each rule renders as a card on the Routing page with the rule name, channel-type badge, optional node badge, an `ON` / `OFF` pill, then a row of small badges naming each matcher: one mono badge per **Stack** pattern, one outline badge per **Label**, one outline mono badge per **Category**, and one outline badge per **Severity**. When Node is any and the other four matchers are empty, a muted `Matches all alerts` line replaces the badge row. When the rule is node-scoped and the other four are empty, the card shows `Matches all alerts on this node` instead. After the badges, a vertical bar separator is followed by the truncated channel URL, then a second separator and a `Priority: N` chip when priority is non-zero.

Three icon actions appear on the right edge of each card:

- **Lightning** sends a test message through the rule's channel using the same payload shape as the global Test button.
- **Pencil** opens the edit modal pre-filled with the current values.
- **Trash** deletes the rule after a confirmation dialog.

The masthead carries `SCOPE` (`global`), `ROUTES` (total), and `ENABLED` (count).

## Mute Rules

<Note>
  Admin role is required to create, edit, or delete mute rules.
</Note>

Notification suppression rules **hide or drop** matching alerts. They do not send alerts elsewhere. Routing and suppression are separate: a rule can match the same alert as a route, but suppression is evaluated first and can block bell delivery, external channels, or both.

Open **Settings · Notifications · Mute Rules** and click **+ Add mute rule**.

| Field | Purpose |
|-------|---------|
| **Name** | A human label, up to 100 characters. |
| **Node scope** | `Any node` or a specific fleet node. Limits which node emits the alert before the rule can match. |
| **Stacks** *(optional)* | Free-form stack patterns with `*` as the only wildcard (for example `prod-*`), plus an optional known-stack picker. Empty matches any stack. |
| **Labels** *(optional)* | Stack labels. Empty matches any label. |
| **Categories** *(optional)* | Notification categories. Empty matches any category. |
| **Severity** *(optional)* | One or more of info, warning, or error. Empty matches any severity. |
| **Apply to** | **Bell**, **External**, or **Both**. Bell skips the notification popover WebSocket push. External skips routing rules and global channels. |
| **Expiration** | Forever, 1 hour, 24 hours, or a custom date. Expired rules stop matching automatically. |
| **Weekly window (UTC)** *(optional)* | Recurring weekly maintenance window. Choose one or more start days and a UTC start/end time. The rule only mutes while the window is active. Same-day windows use an inclusive start and exclusive end. Cross-midnight windows (for example Sat 22:00 to Sun 02:00) list only the start day. Leave off for always-on mute (subject to expiration). |
| **Enabled** | Toggle the rule without deleting it. |

All non-empty matchers must match (AND). Suppressed alerts are still written to stack activity history; only delivery is affected. When a mute rule blocks delivery, the activity row shows a **Suppressed** badge with the matched rule name in a tooltip.

Rules you create on the control instance replicate to remote nodes so alerts emitted on a remote stack honor the same suppression. Replication is best-effort. Remotes that do not support weekly windows do not receive scheduled replicas as all-day mutes: Sencho skips the push and attempts to remove any prior replica. If a remote is unreachable, cleanup may stay pending until connectivity returns and you re-save or clear the rule. Before downgrading a node past a release that introduced weekly windows, clear every weekly window (or disable those rules) so an older binary does not treat them as always-on.

From the bell, admins can open a row menu and choose **Mute this category**, **Mute notifications like this**, or **Mute this stack** to create a quick rule with default **Both** targeting and no weekly window. The same presets are available from stack menus (sidebar, stack header, activity tab), fleet node cards, and label groups.

The masthead carries `SCOPE` (`global`), `RULES` (total), and `ACTIVE` (count).

### Built-in bell quieting (not a mute rule)

Sencho also hides one class of notification from the popover without a user rule: `info`-level rows where the category is one of `deploy_success`, `stack_started`, `stack_stopped`, `stack_restarted`, `stack_taken_down`, or `image_update_applied`, and the row carries an `actor_username` other than `system`. These are confirmations of an action you just clicked. The rows are still persisted and still dispatched externally; only the bell render hides them. Use mute rules when you need configurable, user-controlled muting.

## Notification categories

Every alert Sencho dispatches carries a category that you can filter on in the bell, target with a routing rule's **Categories** matcher, or reason about when reading audit history.

| Category | Label in the bell | Trigger |
|----------|-------------------|---------|
| `deploy_success` | Deploy success | Stack deploy completed without error |
| `deploy_failure` | Deploy failure | Stack deploy returned a non-zero exit |
| `stack_started` | Stack started | Stack started via the dashboard or API |
| `stack_stopped` | Stack stopped | Stack stopped via the dashboard or API |
| `stack_restarted` | Stack restarted | Stack restarted via the dashboard or API |
| `stack_taken_down` | Stack taken down | Stack taken down via the dashboard or API |
| `image_update_available` | Update available | Image-update poll found a newer digest |
| `node_update_available` | Node update | Sencho self-update available for this node |
| `image_update_applied` | Update applied | Manual or scheduled auto-update applied new images |
| `autoheal_triggered` | Auto-heal | Auto-heal restarted, failed to restart, or auto-disabled a policy |
| `monitor_alert` | Monitor alert | Per-stack threshold breach, host CPU/RAM/disk warning, or healthcheck failure |
| `scan_finding` | Scan finding | Vulnerability-scan completion, per-violation alert, post-deploy scan failure, or auto-update gate block |
| `system` | System | Trivy auto-update, fleet sync, daemon connectivity, scheduled-task lifecycle, cloud-backup upload failure |
| `blueprint_deployed` | `blueprint_deployed` | Blueprint provisioned a new deployment |
| `blueprint_deployment_failed` | `blueprint_deployment_failed` | Blueprint deployment errored out |
| `blueprint_drift_detected` | `blueprint_drift_detected` | Blueprint drift detected in `suggest` or `enforce` mode |
| `blueprint_drift_correction_failed` | `blueprint_drift_correction_failed` | Blueprint enforce-mode redeploy failed |

The four `blueprint_*` categories are accepted by routing rules but render as raw category strings in the bell because the frontend label map omits them.

Seven further categories are written to notification history and the stack activity timeline, and can be targeted by a mute rule, but are never dispatched to external channels or accepted by a routing rule: `drift_detected` (Drift detected), `drift_resolved` (Drift resolved), `update_started` (Update started), `health_gate_passed` (Health gate passed), `health_gate_failed` (Health gate failed), `rollback_generation_released` (Rollback protection released), and `network_auto_created` (renders as a raw category string, same reason as the blueprint categories above).

## Stack alert rules

Each stack carries its own set of threshold rules that fire when a metric stays above (or below) a value for a configurable window. A rule can target **All services** in the stack or one Compose service. Metrics are evaluated per running container (not as a stack aggregate): each container has its own duration timer, and cooldown is tracked per Compose service so replicas of the same service share one silence window while different services can alert independently.

Rules live on the node where the stack runs and are evaluated locally on a 30-second tick.

Open the rules editor from any of these entry points:

- Stack header **More actions** → **Monitor** (Alerts tab)
- Container card **Monitor** (prefers that Compose service in add forms)
- Sidebar context menu **Alerts**, or press `A` while focused on a stack

<Frame>
  <img src="/images/alerts-notifications/alert-panel.png" alt="The Stack › PLEX › MONITOR sheet with the Alerts tab active, a green notification-channel banner under the NOTIFICATION CHANNELS heading, an ACTIVE RULES section listing 'CPU Usage (%) > 80' with the secondary line 'Trigger after 5m • Cooldown 60m', and an ADD NEW RULE form with Metric, Operator, Threshold, Duration, Cooldown fields and an Add Rule submit button." />
</Frame>

### Alert fields

| Field | Purpose |
|-------|---------|
| **Service** | **All services** (default) or one Compose service from the stack. All services evaluates every matching container independently. |
| **Metric** | The system resource or metric to monitor. |
| **Operator** | Comparison: `Greater than`, `Greater or eq`, `Less than`, `Less or eq`, `Equals`. |
| **Threshold** | A number ≥ 0. The unit follows the chosen metric. |
| **Duration (mins)** | How long a container's breach must persist before firing. Default `5`, range 0 to 1440. |
| **Cooldown (mins)** | Silence window between fires for the same Compose service after a rule triggers. Default `60`, range 0 to 10080. |

Sencho tracks the start of each breach per container in memory; the rule fires only after that container's breach has lasted for the full **Duration**, and only if the previous fire for the same Compose service is older than **Cooldown**. The notification names the triggering service and container. Containers without a Compose service label are grouped under one shared cooldown key and appear as **unknown service** in the message; that label is not selectable as a rule target. A scoped rule whose service is no longer listed in the compose file shows **Not in compose** in the rules list; evaluation still matches running containers that carry that Compose service label.

### Available metrics

| Metric | UI label | Unit |
|--------|----------|------|
| `cpu_percent` | CPU Usage (%) | percent of all cores |
| `memory_percent` | Memory Usage (%) | percent of container memory limit |
| `memory_mb` | Memory Usage (MB) | MB of resident set size, minus cache |
| `net_rx` | Network In (MB/s) | MB per second, computed as a delta between consecutive samples |
| `net_tx` | Network Out (MB/s) | MB per second, computed as a delta between consecutive samples |
| `restart_count` | Restart Count | integer count of Docker-reported restarts |

### Channel banner states

The **NOTIFICATION CHANNELS** banner above the rules list reflects what dispatch will look like for this stack:

- **Loading** is a spinner with `Checking notification channels...` while Sencho asks the target node for its agent state.
- **Remote node** is a blue banner reading `Remote node: <name>`, with the body `Alert rules are stored and evaluated on this remote instance. Notifications are dispatched using that node's configured channels.` A sub-line reports whether the remote has any channels configured.
- **No channels** is an amber banner reading `No notification channels configured`, body `Alert rules will be saved and evaluated, but no notifications will be dispatched. Configure Discord, Slack, Apprise, or a webhook in Settings → Notifications → Channels.`
- **Active** is a green banner reading `Notifications active via Discord, Slack, …` with the configured channels listed.

<Frame>
  <img src="/images/alerts-notifications/alert-panel-remote-banner.png" alt="The Stack › nakama › MONITOR sheet on a remote node, with the blue 'Remote node: Pitt-Moba' banner explaining that rules are stored and evaluated on the remote, plus a follow-up amber line noting that no notification channels are configured on that remote." />
</Frame>

### Example: alert on high CPU

To page when CPU on a stack stays above 80% for at least five minutes, with no more than one alert per hour:

| Field | Value |
|-------|-------|
| **Metric** | CPU Usage (%) |
| **Operator** | Greater than |
| **Threshold** | `80` |
| **Duration (mins)** | `5` |
| **Cooldown (mins)** | `60` |

### Permissions and validation

Stack alert rules are gated by the `stack:edit` permission on the target stack, not a fixed admin check. A user sees the **Add new rule** form and the delete action on each row only if their role (or a scoped grant) carries `stack:edit` for that stack; a user with only `stack:read` sees existing rules read-only. Built-in roles: `admin` and `node-admin` have `stack:edit` on every stack; `deployer`, `viewer`, and `auditor` do not, so they see a read-only list. Submitting an empty threshold surfaces `Please enter a threshold.` Successful saves toast `Alert rule added.` and `Alert rule deleted.` A failed POST surfaces `Network error. Could not reach the node.`

The delete confirmation dialog reads `Delete Alert Rule` / `This will permanently remove this alert rule. Notifications for this condition will no longer be sent.` with a destructive **Delete** button.

## Notification history (the bell)

The bell icon at the top of the shell is the live feed of every alert across the fleet. A pulsing red dot surfaces unread items.

<Frame>
  <img src="/images/alerts-notifications/notification-popover.png" alt="The notification bell open, with a 'Notifications' italic title, '1 UNREAD' uppercase mono caption, an All / Unread / Alerts segmented control, the filter-toggle, mark-all-read, and clear-all icon actions, a 'TODAY' day-band header, and a warning-tinted row reporting a CPU threshold breach on the plex stack with a relative timestamp and a per-row dismiss target." />
</Frame>

### Anatomy

A title bar with `Notifications` in the italic display face, the unread count to the right (`<n> UNREAD`), and a row of controls below.

The segmented control switches between `All`, `Unread` (with badge), and `Alerts` (rows where level is `warning` or `error`). To the right, three icon buttons:

- **SlidersHorizontal** toggles a hidden filter row. A small dot on the icon means at least one filter is active.
- **CheckCheck** is `Mark all read`.
- **Trash2** is `Clear all`. The action issues `DELETE /api/notifications` against every node that contributed a row.

The hidden filter row has two combobox dropdowns: **Filter by node** (only rendered with two or more nodes registered) and **Filter by category**, listing all the user-facing labels from the categories table above.

<Frame>
  <img src="/images/alerts-notifications/bell-filter-row.png" alt="The bell with its filter row expanded, the 'ALL TYPES' combobox open over a long popover listing every notification category from 'All types' through 'System', and the underlying notification rows visible beneath." />
</Frame>

### Row anatomy

Each row carries:

- A **3 px left rail** colored by severity: brand cyan for `info`, amber for `warning`, destructive rose for `error`. Rail saturation drops to about 30% once the row is read.
- A **severity icon** in the same hue: `Info`, `AlertTriangle`, or `AlertOctagon`.
- The **message body**, slightly dimmer once read.
- A **kicker line** below the message with an optional **node-name pill** (only rendered for multi-node fleets), a `·` separator, and a relative timestamp (`just now`, `Nm ago`, `Nh ago`, `yesterday`, then a short locale date).
- A **dismiss** target on hover, top-right.

There is no level chip in the row; the rail color and icon do that work. Rows whose payload carries a `stack_name` become click targets that jump to the stack and, when a `container_name` is present, surface that container's logs.

### Day groupings

Rows are grouped under uppercase day-band headers: `TODAY`, `YESTERDAY`, `THIS WEEK`, `EARLIER`.

### Empty states

<Frame>
  <img src="/images/alerts-notifications/bell-empty.png" alt="The bell open in its empty state with a strikethrough bell icon, the headline 'You're all caught up', and the mono caption 'New notifications appear here in real time.'" />
</Frame>

The popover renders three different empty states:

- All filters, no rows ever: `You're all caught up` over `New notifications appear here in real time.`
- Unread filter, all read: `No unread notifications` over `Everything in your feed has been read.`
- Alerts filter, no warnings or errors: `No active alerts` over `Warnings and errors will surface here when they occur.`

### What you won't see in the bell

Sencho deliberately suppresses one class of notification from the popover: `info`-level rows where the category is one of `deploy_success`, `stack_started`, `stack_stopped`, `stack_restarted`, `stack_taken_down`, or `image_update_applied`, AND the row carries an `actor_username` other than `system`. These are confirmations of an action you just clicked, surfaced as a toast at the moment of the action; replaying them in the bell would be noise. The rows are still persisted to `notification_history` and still dispatched to global channels and matching routes; only the bell render hides them.

### Limits

The popover holds the latest 50 rows per node per fetch. There is no `Load more` affordance; older rows roll off the caps described in [Retention and limits](#retention-and-limits) below.

## Cross-node delivery

The bell aggregates across the entire fleet by hitting `/api/notifications` against every registered node and opening one WebSocket per node: `/ws/notifications` for the local instance and `/ws/notifications?nodeId=<id>` for every remote. A new alert on any remote arrives in the bell within about a second; a 60-second safety-net poll catches anything the WebSocket dropped.

Per-stack alert rules and channel configuration are **stored on the node where the stack runs**. To configure a rule on a remote, switch the active node via the picker, open the stack's **Monitor** sheet, and the form's POST is forwarded to the remote.

Crash detection runs only on local Docker; remote nodes run their own copy of `DockerEventService` and emit through the proxy. On a multi-node fleet the bell attributes remotes with a node-name badge rather than embedding the satellite-local name in the message. External channel payloads (Slack, Discord, Apprise, custom webhook) receive the same sanitized body with no structured source-node id.

Switching the active node only affects per-stack rule editing and the Settings panels. The bell aggregates every node regardless of which one is active.

## Alerts emitted by the system

Sencho dispatches notifications from many code paths. The list below covers every emitter in current code, organized by source. Message bodies are node-neutral for local targets so fleet aggregation does not contradict the bell badge; remote-target names stay when a hub names an authoritative remote.

### Container crash and health

Real-time on the Docker event stream:

- **Crash** at `error`/`monitor_alert`: `Container Crash Detected: <name> exited unexpectedly (Code: <N>).`
- **OOM kill** at `error`/`monitor_alert`: `Container OOM Kill: <name> was killed by the OOM killer (out of memory).` When a `die` arrives with exit code 137 but no preceding `oom` event, Sencho inspects the container and reclassifies as OOM if the `OOMKilled` flag is set.
- **Healthcheck failure** at `error`/`monitor_alert`: `Healthcheck failed: <name> is unhealthy.` Emitted only on the transition into Docker `unhealthy` (not on every repeated unhealthy event).
- **Mass exit** kicks in when a daemon-disconnect gap is followed by 20% or more of containers exiting on reconnect, in which case a single `info`/`system` summary `Docker daemon interruption detected: N containers exited during connection gap.` is emitted instead of N crash alerts.
- **Rate limit** caps combined crash and health dispatches at 20 per fixed 60-second window per node. Overflow does not write individual history rows; at the end of the window Sencho emits a single `warning`/`monitor_alert` roll-up such as `N additional container crash alerts were rate-limited in the last minute.`, `N additional container health alerts were rate-limited in the last minute.`, or `N additional container alerts were rate-limited in the last minute (X crash, Y health).`
- **Crash dedup** is 60 minutes per container after a non-rate-suppressed crash dispatch. Cleared when that container starts again.
- **Health dedup** is 60 minutes per container after a health alert that was persisted to notification history. It survives healthy/starting recovery and the 10-minute prune of in-memory container tracking state (dedup lives in a separate map), and clears only when the window expires or the Sencho process restarts.

### Daemon connectivity

- `warning`/`system`: `Lost connection to Docker daemon; monitoring paused.` (one-shot until reconnect)
- `info`/`system`: `Reconnected to Docker daemon.`
- `warning`/`system`: `Received malformed Docker event payloads. Monitoring continues but some events may be skipped.` (when more than 10 parse errors in a minute)

### Host thresholds

`warning`/`monitor_alert` for host CPU, RAM, and disk when the configured threshold is exceeded. Evaluated on the 30-second monitor tick. Example: `Host CPU utilization is critically high: 92% (Threshold: 90%)`.

Each metric (CPU, RAM, disk) carries its own suppression window. The first time a metric crosses its threshold, one notification fires immediately. While the metric stays over threshold within the window, additional cycles are silently counted but not dispatched. The next dispatch after the window elapses carries a summary suffix: `Suppressed 119 alerts in the last 60m; first over threshold at 14:23 UTC.` When the metric drops below threshold, the counter resets so the next breach fires fresh.

The suppression window defaults to 60 minutes and is configured per node in **Settings → Monitoring → Host Alerts → Alert suppression**.

The entire host threshold evaluation can be silenced per node from **Settings · Monitoring · Host Alerts · Host threshold alerts** while keeping the configured limit values. This toggle affects only the CPU, RAM, and disk threshold checks; crash capture, stack alert rules, and health gate checks all continue to run independently.

### Reclaimable Docker data

`info`/`system`: `This node has accumulated <N> GB of unused Docker data. Open Resources to reclaim space, or set up a Prune Node Resources schedule.` 24-hour cooldown. On a multi-node fleet the bell attributes remotes with a node-name badge rather than embedding the name in the message.

### Sencho version availability

`info`/`node_update_available`: `Sencho X.Y.Z is available (currently running A.B.C). Visit the Fleet dashboard to update.` Monitor evaluates every 30 seconds against a shared version cache (refreshes every 30 minutes when published, or every 3 minutes while publish is pending). One-shot dedup until the running version reaches or passes the notified version.

### Image update availability

`info`/`image_update_available`: `Stack "<name>" has image updates available.` On multi-service stacks the message can name the services that have updates. By default, checks every two hours in interval mode, with a two-minute startup delay. The cadence can be changed or replaced with a cron schedule. Manual refresh carries a two-minute cooldown. Notifies on **state transition only**; the first run also emits catch-up notifications for stacks already known to have updates.

### Auto-update execution

`info`/`image_update_applied`: `Auto-update: stack "<name>" updated with new images`. If a block-on-deploy policy gates the auto-update: `warning`/`scan_finding` `Policy "<name>" blocked auto-update: N image(s) exceed <severity>`. See [Auto-update policies](/features/auto-update-policies).

### Auto-heal

- `info`/`autoheal_triggered`: `Auto-Heal: Restarted <container> on stack <stack> after being unhealthy for <N> minute(s).`
- `warning`/`autoheal_triggered`: `Auto-Heal: Failed to restart <container> on stack <stack>. Error: <err>`
- `warning`/`autoheal_triggered`: `Auto-Heal: Policy for <stack>[/<svc>] has been auto-disabled after <N> consecutive failures.`

See [Auto-heal policies](/features/auto-heal-policies).

### Vulnerability scanning

- **Per-violation, during a scheduled scan** is `warning`/`scan_finding`. One alert per violation: `Policy "<name>" violated by <imageRef>: <severity> exceeds <maxSeverity>`.
- **Scan completion** is `info`/`scan_finding` (or `warning` if any image failed): `Scheduled scan "<task name>" completed: <output>` where `<output>` is one of `Scanned N image(s); X skipped (cached). Found A critical, B high, C medium.`, `No critical, high, or medium findings.`, `No images to scan`, or `All N image(s) already scanned recently (cache hit)`.
- **Pre-deploy gate when Trivy is missing** is `warning`/`scan_finding`: `Pre-deploy scan for "<stack>" skipped: Trivy not installed on this node`.
- **Post-deploy scan finding** is `error`/`scan_finding` if criticals are present, `warning` otherwise: `Vulnerability scan for <imageRef>: <N> critical, <M> high`.
- **Post-deploy scan failure** is `warning`/`scan_finding`: `Post-deploy scan failed for <imageRef> (<stack>): <msg>`.
- **Trivy auto-update** is `info`/`system`: `Trivy updated from vX to vY` or `Trivy update available: vX (currently vY)`.

See [Vulnerability scanning](/features/vulnerability-scanning).

### Scheduled tasks

- `error`/`system`: `Scheduled task "<name>" (<action>) failed: <err>`
- `info`/`system` recovery: `Scheduled task "<name>" (<action>) recovered successfully`

Local-target failure reasons stay node-neutral (`Target node is offline`, `Container "<n>" not found on this node. ...`). When the task targets a remote node, the failure reason keeps that node's roster name.

### Recovery Vault upload failure

`warning`/`system`: `Cloud backup failed for scheduled snapshot <id>: <message>`. See [Fleet backups](/features/fleet-backups).

### Blueprints

- `warning`/`blueprint_drift_detected`: `Blueprint "<n>" drifted on this node: <reason>` for a local target, or `Blueprint "<n>" drifted on node "<remote>": <reason>` for a remote. Stateful marker-loss uses the same local/remote clause.
- `error`/`blueprint_drift_correction_failed`: `Auto-fix for "<n>" on this node failed: <err>` (local) or `Auto-fix for "<n>" on node "<remote>" failed: <err>` (remote).

See [Blueprints](/features/blueprint-model).

### Fleet sync

Fleet Sync requires Admiral on the control instance.

- `warning`/`system` identity drift: `Fleet self-identity changed from "X" to "Y". Identity-scoped policies are reapplied on the next sync.`
- `warning`/`system` truncation: `Fleet sync truncated <resource> to <N> rows (<dropped> not replicated). Reduce the local set or contact support.`
- `warning`/`system` stale: `Fleet sync to node "<n>" has been failing for over an hour for <resource>. Check the node's connectivity and API token.`

## Retention and limits

**Settings · Operations · Data Retention** hosts the notification and audit retention windows alongside two vulnerability-scan retention controls (**Scan history per digest** and **Remove scans for deleted images and stacks**); those two are covered in [Vulnerability scanning](/features/vulnerability-scanning), not here.

<Frame>
  <img src="/images/alerts-notifications/developer-retention.png" alt="Settings · Operations · Data Retention card showing Container metrics with a 24 HRS field, Notification log with a 30 DAYS field, Scan history per digest with a 50 SCANS field, a Remove scans for deleted images and stacks toggle in the ON state, and Audit log with a 90 DAYS field, plus a SAVE SETTINGS button." />
</Frame>

| Control | Range | Default | What it prunes |
|---------|-------|---------|----------------|
| **Container metrics** | 1 to 8760 hours | 24 hours | Per-container CPU, RAM, and network history used by the dashboard sparklines |
| **Notification log** | 1 to 365 days | 30 days | The bell's `notification_history` table |
| **Audit log** *(Admiral)* | 1 to 365 days | 90 days | Audit trail entries |

Beyond the time-based window above, `notification_history` also carries two hard row caps that a cleanup pass enforces on every monitor tick: up to 500 rows per (node, stack) pair, and up to 1,000 rows per node for stack-less system events (host thresholds, fleet-sync warnings, scheduled-task results). A chatty stack cannot evict a quieter stack's history because each stack has its own 500-row quota. These caps apply on top of your retention window, whichever prunes first.

A separate rate limit applies to crash and health alerts only: 20 emits per fixed 60-second window per node (not a rolling window). When the cap is hit, individual overflow alerts are not written to history or the bell; at the end of the window Sencho emits a single `warning`/`monitor_alert` roll-up that names crash-only, health-only, or mixed suppression.

## Crash detection toggle

The global crash-capture switch lives under **Settings · Monitoring · Container Alerts**.

<Frame>
  <img src="/images/alerts-notifications/system-crash-toggle.png" alt="Settings · Monitoring · Container Alerts panel showing a Container crash & health alerts toggle in the ON state with the helper line about unexpected exits, OOM kills, and healthcheck failures." />
</Frame>

The **Container crash & health alerts** toggle controls whether `DockerEventService` raises crash, OOM, and healthcheck alerts on the active node. Helper text: `Send alerts for unexpected container exits, OOM kills, and Docker healthcheck failures. Auto-Heal can still observe crash signals independently.` Defaults to on; if the database read fails, Sencho falls back to default-deny so the system never leaks alerts you cannot turn off.

The **Host Alerts** panel carries the **Host thresholds** rows (CPU limit, RAM limit, Disk limit, all expressed as percent) that drive the host-level monitor warnings. Container crash and health alerts live in the **Container Alerts** panel. The **Reclaimable Docker data threshold** (in GiB) that drives the unused-Docker-data alert lives in the **Docker & Storage** panel.

## Refresh cadence

| Surface | Cadence |
|---------|---------|
| Crash, OOM, and healthcheck events | Real-time over the Docker event stream |
| Host CPU / RAM / disk threshold checks | 30 seconds |
| Per-stack alert rule evaluation | 30 seconds |
| Image update poll | Configurable (default every 2 hours), with a 2-minute startup delay and a 2-minute cooldown on manual refresh |
| Sencho version check | Monitor evaluation every 30 seconds; shared version cache refreshes every 30 minutes when published or every 3 minutes while publish is pending |
| Notification fanout to channels | One attempt by default; optional 0-3 extra in-process attempts with a fixed 1s delay; 10-second timeout per attempt; no durable queue |
| Bell live updates | Pushed live over the notifications WebSocket per node |
| Bell safety-net reconcile | 60 seconds |
| Crash dedup window per container | 60 minutes (cleared on container start) |
| Health dedup window per container | 60 minutes after a persisted health alert; survives recovery; process-local |
| Crash/health rate-limit window | 20 alerts per fixed 60 seconds per node, then a single typed roll-up (overflow individuals are not persisted) |

Switching the active node tears down per-stack rule editors and reloads channel state, but does not affect the bell, which keeps every node's WebSocket open in parallel.

## Troubleshooting

<AccordionGroup>
  <Accordion title="Notifications never arrive">
    Check three things in order. First, the channel toggle in **Settings · Notifications · Channels** must be on; the kicker on each tab reads `enabled` or `off`. Second, Discord, Slack, and webhook URLs must use HTTPS (the form rejects plain `http://` for those channels); Apprise and ntfy endpoints may use HTTP or HTTPS. Third, a routing rule with unconstrained Node plus empty Stacks, Labels, Categories, and Severity matchers will intercept every alert and skip the global channels. Use the per-channel **Test** button to issue a one-shot dispatch and watch your endpoint for the literal message `🔌 Test Notification from Sencho!` Sencho records the failure reason in `notification_history.dispatch_error` when delivery throws, so a row that appears in the bell with no follow-up at the endpoint usually means a 4xx or timeout at the receiver.
  </Accordion>
  <Accordion title="An alert rule never fires even when the threshold is breached">
    Three causes account for almost every case. First, the rule's **Duration** has not elapsed yet: that container's breach must persist for the full duration before the rule fires (a healthy sibling service does not clear another container's timer). Second, the same Compose service is still in cooldown after a previous fire. Third, the panel's banner is not green: a remote-node banner means the rule was saved on a remote whose channels you may not have configured, and an amber `No notification channels configured` banner means the rule evaluates fine but Sencho has nowhere to send the alert. The evaluator runs on a 30-second tick, so expect up to 30 seconds of latency between the breach starting and the timer engaging.
  </Accordion>
  <Accordion title="I stopped a stack but got a crash alert anyway">
    The Docker event service defers `die` classification by 500 ms to absorb out-of-order `kill` events from the daemon, then asks the lifecycle classifier whether the exit was intentional, clean (exit 0), a crash, or an OOM kill. If your stop happened far enough outside that window, or the daemon emitted the events without the kill marker the classifier looks for, the exit can be classified as a crash. The classifier favors avoiding silent crashes over avoiding noisy false positives. Compare the alert timestamp against your `docker compose down` time; entries within a second of each other are usually the same event seen from two angles.
  </Accordion>
  <Accordion title="After a Docker daemon restart, the bell only has one summary line">
    When the daemon disconnects and reconnects, Sencho snapshots every container at the moment of reconnect. If at least 20% exited during the gap, the service emits a single `info`/`system` summary `Docker daemon interruption detected: N containers exited during connection gap.` instead of one crash alert per container. Below the threshold, every gap exit is classified individually.
  </Accordion>
  <Accordion title="Crash alerts stopped arriving and nothing else looks wrong">
    Check **Settings · Monitoring · Container Alerts · Container crash & health alerts**. The toggle is the master switch for the Docker event service. The panel cache reads the database every 500 ms; if the database read errors, Sencho defaults the toggle to off so the failure mode is silent rather than spammy. Repair the toggle, save, and the next Docker event reaches the dispatcher.
  </Accordion>
  <Accordion title="A burst of crashes or health flaps happened but I only see about twenty alerts">
    Sencho rate-limits crash and health dispatches together to 20 emits per fixed 60-second window per node, then emits a single `warning`/`monitor_alert` roll-up such as `N additional container crash alerts were rate-limited in the last minute.`, a health-only variant, or a mixed `(X crash, Y health)` summary once the window closes. Rate-limited overflow individuals are not written to `notification_history` and do not appear in the bell; only the roll-up is persisted for that overflow. Separately, health flaps on the same container are deduped for 60 minutes after a persisted unhealthy alert, even if Docker recovered and failed again inside that window.
  </Accordion>
  <Accordion title="My deploy success doesn't appear in the bell, only as a toast">
    Sencho deliberately hides `info`-level rows whose category is one of `deploy_success`, `stack_started`, `stack_stopped`, `stack_restarted`, `stack_taken_down`, or `image_update_applied` AND whose `actor_username` is set to a real user. The reasoning: those are confirmations of the action you just clicked and are already shown as a toast. The rows are still persisted to `notification_history` and still dispatched to global channels and matching routes; only the bell render hides them.
  </Accordion>
  <Accordion title="A routing rule is set up but the global channel still fires">
    Routing matchers AND together: every non-empty matcher must match the alert. A rule with **Stacks** set to `prod-api` will not match a `monitor_alert` for a different stack, and a rule with a populated **Stacks** matcher will not match host-level alerts (which carry no stack target). Stack patterns may use `*` as a wildcard (for example `prod-*`). When zero rules match, Sencho falls back to global channels. To intercept everything, leave Node as any and leave Stacks, Labels, Categories, and Severity empty. Also confirm the rule's **Enabled** pill is `ON`.
  </Accordion>
  <Accordion title="Fleet shows an update button but I never received a Sencho version notification">
    Fleet and the notification path share the same version lookup. When a newer published release is in that cache, Monitor dispatches a single `info`/`node_update_available` alert and records it under `last_sencho_update_notified_version`, so each release produces one alert per node. If you already upgraded to (or past) that version, the dedup self-heals and the alert does not fire. Check **Settings · Notifications · Mute Rules** for a rule that mutes `node_update_available` in the bell. A newly published release may take up to the cache TTL (about 30 minutes when published, or about 3 minutes while registry publish is still pending) before Fleet and the bell both observe it.
  </Accordion>
  <Accordion title="A stack shows the blue update indicator but no notification was received">
    The image-update service emits on **state transitions only** after its first run. The first run also emits catch-up notifications for stacks already known to have updates, then sets a backfill flag. Once that flag is set, only a false-to-true transition triggers another alert. If you dismissed the catch-up notification, a new one appears the next time the stack goes from up-to-date back to behind.
  </Accordion>
</AccordionGroup>
