gsadmin/sencho
1
0
Fork 0
mirror of https://github.com/Studio-Saelix/sencho.git synced 2026-07-26 11:49:16 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
a3edee5e6a6c6319bb292f2e37d96aa3260acf3f
sencho/docs/features/alerts-notifications.mdx
T
Anso a3edee5e6a feat: weekly UTC maintenance windows for mute rules (#1661) 
* feat: add weekly UTC maintenance windows to mute rules

Let mute rules suppress only during recurring UTC windows, normalize
replica node identity, and fail-open when remotes lack schedule support
so older nodes never keep an all-day scheduled mute after a successful cleanup DELETE.

* fix: fail closed on corrupt mute schedules and clean invalid replicas

Empty or whitespace stored schedules no longer act as all-day mutes. Invalid schedules trigger remote DELETE cleanup, and the weekly-window form gains accessibility attributes plus component coverage.

* fix: require explicit repair before clearing a corrupt mute schedule

The suppression engine already fails closed on an unreadable stored
schedule (scheduleInvalid), but the frontend never surfaced that flag:
a corrupt rule looked identical to an ordinary unscheduled one, and
opening Edit then clicking Update sent an explicit schedule: null,
silently turning the corruption into a valid all-day mute. Add the
flag to the rule type, show an Invalid schedule badge on the card, and
block saving in the edit form until the operator explicitly touches
the weekly window (configures a new one, or toggles it to confirm the
clear).

* fix: correct contradictory toggle-sequence copy in schedule-repair toast

The blocking toast told operators to toggle the weekly window "off then
on" to confirm clearing a corrupt schedule, but the toggle starts off
for a corrupt rule, so that sequence leaves it on and trips the
no-selected-day validation instead. The correct, tested sequence is on
then off, matching the inline hint below the toggle. Also add a
regression test confirming the invalid-schedule save gate resets
cleanly across edit sessions on different rules.

* fix: enforce replica node_id and guard fleet sync against stale writes

Two hardenings to the suppression-rule fleet sync path found during
review: the /replica endpoint trusted the payload's node_id instead of
forcing it to null server-side, so a direct proxy-authenticated caller
could persist a scoped replica; and upsertNotificationSuppressionRuleReplica
overwrote unconditionally with no ordering check, so a delayed older
POST arriving after a newer one could downgrade the stored rule. Force
node_id to null on every replica write, and skip (with a warning log)
any incoming write whose updated_at is not newer than what's stored.

* test: assert the exact-tie updated_at case in the fleet sync stale-write guard

The staleness guard added in c31458a1 uses >= (ties are ignored, not
just strictly older writes); add the missing assertion for that
boundary and make the comment explicit about it.

* fix: bump vulnerable transitive backend dependencies

npm audit flagged body-parser, fast-uri, and protobufjs (one high
severity: fast-uri host confusion via failed IDN canonicalization).
All three have patch/minor fixes within existing semver ranges;
npm audit fix resolves all three with no package.json changes.

* fix: sanitize suppression replica fields before logging

Log entries built from fleet-sync replica payloads embedded rule id
and timestamp values directly, allowing a compromised peer to forge
log lines via control characters.

* fix: prevent delayed replica writes from resurrecting deleted mute rules

A network-reordered replica POST arriving after a DELETE fell into the
insert-when-absent branch with no protection, since the staleness guard
only compares against a row that still exists. Add a permanent
per-id tombstone (safe because rule ids are AUTOINCREMENT and never
reused): every delete records one, and the replica upsert refuses to
recreate a tombstoned id regardless of the incoming updated_at.
2026-07-21 23:17:52 -04:00

471 lines
41 KiB
Plaintext
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
---
title: Alerts & Notifications
sidebarTitle: Alerts and notifications
description: Threshold and event alerts for your fleet, dispatched to Discord, Slack, Apprise, or any webhook, with per-stack rules and channel routing.
---
Sencho watches each node it manages for container crashes, host pressure, scheduled-task results, and update availability, then surfaces every signal in two places: the in-app notification bell at the top of the shell and an external channel you configure. This page covers everything from configuring channels to writing per-stack threshold rules, routing alerts to dedicated channels with routing rules, and tuning retention.
<Frame>
<img src="/images/alerts-notifications/notifications-settings.png" alt="Settings · Notifications · Channels panel with NODE Local in the header, a Delivery retries row showing Extra attempts 0 and a Save retries button, Discord Slack Webhook and Apprise tabs with Apprise selected, Enabled off, an empty Apprise endpoint placeholder, and Test beside Save." />
</Frame>
## Notification channels
Open **Settings · Notifications · Channels** to configure Discord, Slack, custom webhook, and Apprise channels. Each channel is per-node, so switching the active node via the node picker reloads the panel against that node's stored settings. The masthead carries a `CHANNELS` stat showing how many of the four slots are enabled.
Above the channel tabs, **Delivery retries** sets how many extra in-process attempts (0 to 3) Sencho makes after a transient delivery failure on that node. The default is `0` (single-shot). Extra attempts wait a fixed one second between tries. Admin role is required to change the value.
Each Discord, Slack, and Webhook tab carries an **Enabled** toggle, a **Webhook URL** input (HTTPS only), and **Test** / **Save**. The Apprise tab uses an **Apprise endpoint** instead: keyed `/notify/{key}` shows optional **Tags**; stateless `/notify` shows **Destination URLs**. The kicker on each tab toggles between `enabled` and `off` so you can see at a glance which slots are wired up.
<Note>
Discord, Slack, and webhook URLs must use HTTPS. Apprise endpoints may use HTTP or HTTPS. Every endpoint must parse as a URL.
</Note>
### Discord
Paste an incoming webhook URL from your Discord channel's **Edit Channel · Integrations · Webhooks** view. Sencho posts a single embed per alert, with the title `Sencho Alert [<LEVEL>]`, the message in the description, and the embed color set by severity (info blue, warning yellow, error red).
### Slack
Paste an incoming webhook URL from a Slack app installed in your workspace (URL shape `https://hooks.slack.com/services/...`). Sencho posts a single text message: `<emoji> *Sencho Alert [<LEVEL>]*` followed by the message body, with the emoji set per severity (`ℹ️` info, `⚠️` warning, `🚨` error).
### Generic webhook
Use the Generic Webhook tab when you have your own receiver: a Mattermost or Teams adapter, a serverless function that fans out to email or SMS, or a logging endpoint. Sencho posts JSON of the shape:
```json
{
"level": "warning",
"message": "The **CPU usage** for **plex** has exceeded your threshold of **80%** (Currently: 91%).",
"timestamp": "2026-05-08T22:14:09.812Z",
"source": "sencho"
}
```
`level` is one of `info`, `warning`, `error`. `source` is always the literal string `sencho`. `timestamp` is ISO-8601 with millisecond precision.
### Apprise
Use a keyed Apprise endpoint ending in `/notify/<key>` with optional tags, or a stateless endpoint ending in `/notify` with one or more destination URLs. Separate stateless destination URLs with commas or whitespace. Apprise endpoints may use HTTP for a local gateway.
Sencho sends the alert title, body, severity, and either tags or destination URLs to Apprise. An HTTP 204 response means Apprise accepted no delivery and is reported as a failed test or dispatch. Configuration reads expose only destination provider names and counts, never destination URLs or endpoint keys.
### Test sends and delivery semantics
The **Test** button on each tab dispatches the literal message `🔌 Test Notification from Sencho!` at level `info` through the same path a real alert would take, including the node's delivery-retries setting. Test sends require the admin role; the server returns 403 if an operator or viewer submits one.
Each delivery attempt is an HTTP POST with a 10-second `AbortSignal.timeout`. By default (`Delivery retries` = 0) Sencho makes one attempt. You can allow up to three extra in-process attempts with a fixed one-second delay between them. Retries apply only to classified transient failures (for example HTTP 5xx or network timeouts). Client errors such as HTTP 4xx and Apprise HTTP 204 are not retried. There is no durable retry queue: if the process exits mid-dispatch, remaining attempts are not persisted. Delivery is at-least-once under ambiguous timeouts or connection resets, so a receiver that accepted a request whose response was lost can receive a duplicate. If every attempt fails, the alert remains in the bell with `dispatch_error` set.
## Notification Routing
<Note>
Admin role is required to create, edit, or delete routes.
</Note>
Routing lets you direct alerts that match specific criteria to dedicated channels. Production crashes can land in `#prod-incidents` on Slack while staging notifications go to a less urgent Discord channel, all without juggling per-channel webhook URLs across teams.
### How routing fits into dispatch
For every alert Sencho dispatches, the routing engine evaluates every enabled route. A route matches when **all of its non-empty matchers** match the alert: the **Node**, **Stacks**, **Labels**, **Categories**, and **Severity** filters compose with AND. An empty matcher is treated as match-anything.
If at least one route matches, every matching route fires and the **global channels are skipped** for that alert. If zero routes match, the alert falls back to the global channels configured in the previous section.
A route with all five matchers left unconstrained (`Node` = any, and empty Stacks, Labels, Categories, and Severity) matches every alert and intercepts global delivery entirely. A route scoped to a node with the other four matchers empty matches every alert on that node. Stack-less alerts (host CPU, RAM, disk, fleet-sync warnings, and scheduled-task results without a stack target) almost always miss any populated `Stacks` filter, so they fall back to global by default.
### Creating a routing rule
Open **Settings · Notifications · Notification Routing** and click **+ Add Route**.
| Field | Purpose |
|-------|---------|
| **Name** | A human label, up to 100 characters. Shown on the rule card. |
| **Node scope** | Either `Any node` or a specific node. When set to a node, the rule only matches alerts originating from that node. |
| **Stacks** *(optional)* | Free-form stack patterns with `*` as the only wildcard (for example `prod-*`), plus an optional picker that inserts a known exact stack name as a chip. Empty matches any stack. |
| **Labels** *(optional)* | A combobox of stack labels on the active node. Empty matches any label. |
| **Categories** *(optional)* | A combobox of notification categories. The helper line reads `Leave blank to match all categories. All non-empty filters must match (AND).` |
| **Severity** *(optional)* | One or more of info, warning, or error. Empty matches any severity. |
| **Channel** | Tabs for Discord, Slack, Webhook, and Apprise. Discord, Slack, and Webhook require HTTPS. Apprise accepts HTTP or HTTPS. |
| **Priority** | A number used to sort the rule list. Lower numbers appear higher up. Priority does not gate dispatch: when multiple rules match the same alert, every matching rule fires concurrently. |
| **Enabled** | Toggle the rule on or off without deleting it. |
The modal kicker reads `ROUTING · NEW RULE` when adding and `ROUTING · EDIT RULE` when modifying.
### Managing rules
Each rule renders as a card on the Routing page with the rule name, channel-type badge, optional node badge, an `ON` / `OFF` pill, then a row of small badges naming each matcher: one mono badge per **Stack** pattern, one outline badge per **Label**, one outline mono badge per **Category**, and one outline badge per **Severity**. When Node is any and the other four matchers are empty, a muted `Matches all alerts` line replaces the badge row. When the rule is node-scoped and the other four are empty, the card shows `Matches all alerts on this node` instead. After the badges, a vertical bar separator is followed by the truncated channel URL, then a second separator and a `Priority: N` chip when priority is non-zero.
Three icon actions appear on the right edge of each card:
- **Lightning** sends a test message through the rule's channel using the same payload shape as the global Test button.
- **Pencil** opens the edit modal pre-filled with the current values.
- **Trash** deletes the rule after a confirmation dialog.
The masthead carries `SCOPE` (`global`), `ROUTES` (total), and `ENABLED` (count).
## Mute Rules
<Note>
Admin role is required to create, edit, or delete mute rules.
</Note>
Notification suppression rules **hide or drop** matching alerts. They do not send alerts elsewhere. Routing and suppression are separate: a rule can match the same alert as a route, but suppression is evaluated first and can block bell delivery, external channels, or both.
Open **Settings · Notifications · Mute Rules** and click **+ Add mute rule**.
| Field | Purpose |
|-------|---------|
| **Name** | A human label, up to 100 characters. |
| **Node scope** | `Any node` or a specific fleet node. Limits which node emits the alert before the rule can match. |
| **Stacks** *(optional)* | Free-form stack patterns with `*` as the only wildcard (for example `prod-*`), plus an optional known-stack picker. Empty matches any stack. |
| **Labels** *(optional)* | Stack labels. Empty matches any label. |
| **Categories** *(optional)* | Notification categories. Empty matches any category. |
| **Severity** *(optional)* | One or more of info, warning, or error. Empty matches any severity. |
| **Apply to** | **Bell**, **External**, or **Both**. Bell skips the notification popover WebSocket push. External skips routing rules and global channels. |
| **Expiration** | Forever, 1 hour, 24 hours, or a custom date. Expired rules stop matching automatically. |
| **Weekly window (UTC)** *(optional)* | Recurring weekly maintenance window. Choose one or more start days and a UTC start/end time. The rule only mutes while the window is active. Same-day windows use an inclusive start and exclusive end. Cross-midnight windows (for example Sat 22:00 to Sun 02:00) list only the start day. Leave off for always-on mute (subject to expiration). |
| **Enabled** | Toggle the rule without deleting it. |
All non-empty matchers must match (AND). Suppressed alerts are still written to stack activity history; only delivery is affected. When a mute rule blocks delivery, the activity row shows a **Suppressed** badge with the matched rule name in a tooltip.
Rules you create on the control instance replicate to remote nodes so alerts emitted on a remote stack honor the same suppression. Replication is best-effort. Remotes that do not support weekly windows do not receive scheduled replicas as all-day mutes: Sencho skips the push and attempts to remove any prior replica. If a remote is unreachable, cleanup may stay pending until connectivity returns and you re-save or clear the rule. Before downgrading a node past a release that introduced weekly windows, clear every weekly window (or disable those rules) so an older binary does not treat them as always-on.
From the bell, admins can open a row menu and choose **Mute this category**, **Mute notifications like this**, or **Mute this stack** to create a quick rule with default **Both** targeting and no weekly window. The same presets are available from stack menus (sidebar, stack header, activity tab), fleet node cards, and label groups.
### Built-in bell quieting (not a mute rule)
Sencho also hides one class of notification from the popover without a user rule: rows where the category is one of `deploy_success`, `stack_started`, `stack_stopped`, `stack_restarted`, or `image_update_applied`, and the row carries an `actor_username` other than `system`. These are confirmations of an action you just clicked. The rows are still persisted and still dispatched externally; only the bell render hides them. Use mute rules when you need configurable, operator-controlled muting.
## Notification categories
Every alert Sencho dispatches carries a category that you can filter on in the bell, target with a routing rule's **Categories** matcher, or reason about when reading audit history.
| Category | Label in the bell | Trigger |
|----------|-------------------|---------|
| `deploy_success` | Deploy success | Stack deploy completed without error |
| `deploy_failure` | Deploy failure | Stack deploy returned a non-zero exit |
| `stack_started` | Stack started | Stack started via the dashboard or API |
| `stack_stopped` | Stack stopped | Stack stopped via the dashboard or API |
| `stack_restarted` | Stack restarted | Stack restarted via the dashboard or API |
| `image_update_available` | Update available | Image-update poll found a newer digest |
| `node_update_available` | Node update | Sencho self-update available for this node |
| `image_update_applied` | Update applied | Manual or scheduled auto-update applied new images |
| `autoheal_triggered` | Auto-heal | Auto-heal restarted, failed to restart, or auto-disabled a policy |
| `monitor_alert` | Monitor alert | Per-stack threshold breach, host CPU/RAM/disk warning, or healthcheck failure |
| `scan_finding` | Scan finding | Vulnerability-scan completion, per-violation alert, post-deploy scan failure, or auto-update gate block |
| `system` | System | Trivy auto-update, fleet sync, daemon connectivity, scheduled-task lifecycle, cloud-backup upload failure |
| `blueprint_deployed` | `blueprint_deployed` | Blueprint provisioned a new deployment |
| `blueprint_deployment_failed` | `blueprint_deployment_failed` | Blueprint deployment errored out |
| `blueprint_drift_detected` | `blueprint_drift_detected` | Blueprint drift detected in `suggest` or `enforce` mode |
| `blueprint_drift_correction_failed` | `blueprint_drift_correction_failed` | Blueprint enforce-mode redeploy failed |
The four `blueprint_*` categories are accepted by routing rules but render as raw category strings in the bell because the frontend label map omits them.
## Per-stack alert rules
Each stack carries its own set of threshold rules that fire when a metric stays above (or below) a value for a configurable window. Rules live on the node where the stack runs and are evaluated locally on a 30-second tick.
Open the rules editor by right-clicking a stack in the sidebar and choosing **Alerts**, or by pressing `A` while focused on a stack.
<Frame>
<img src="/images/alerts-notifications/alert-panel.png" alt="The Stack › PLEX › MONITOR sheet with the Alerts tab active, a green notification-channel banner under the NOTIFICATION CHANNELS heading, an ACTIVE RULES section listing 'CPU Usage (%) > 80' with the secondary line 'Trigger after 5m • Cooldown 60m', and an ADD NEW RULE form with Metric, Operator, Threshold, Duration, Cooldown fields and an Add Rule submit button." />
</Frame>
### Alert fields
| Field | Purpose |
|-------|---------|
| **Metric** | The system resource or metric to monitor. |
| **Operator** | Comparison: `Greater than`, `Greater or eq`, `Less than`, `Less or eq`, `Equals`. |
| **Threshold** | A number ≥ 0. The unit follows the chosen metric. |
| **Duration (mins)** | How long the breach must persist before firing. Default `5`, range 0 to 1440. |
| **Cooldown (mins)** | Silence window between fires after a rule triggers. Default `60`, range 0 to 10080. |
Sencho tracks the start of each breach in memory; the rule fires only after the breach has lasted for the full **Duration**, and only if the previous fire is older than **Cooldown**.
### Available metrics
| Metric | UI label | Unit |
|--------|----------|------|
| `cpu_percent` | CPU Usage (%) | percent of all cores |
| `memory_percent` | Memory Usage (%) | percent of container memory limit |
| `memory_mb` | Memory Usage (MB) | MB of resident set size, minus cache |
| `net_rx` | Network In (MB/s) | MB per second, computed as a delta between consecutive samples |
| `net_tx` | Network Out (MB/s) | MB per second, computed as a delta between consecutive samples |
| `restart_count` | Restart Count | integer count of Docker-reported restarts |
### Channel banner states
The **NOTIFICATION CHANNELS** banner above the rules list reflects what dispatch will look like for this stack:
- **Loading** is a spinner with `Checking notification channels...` while Sencho asks the target node for its agent state.
- **Remote node** is a blue banner reading `Remote node: <name>`, with the body `Alert rules are stored and evaluated on this remote instance. Notifications are dispatched using that node's configured channels.` A sub-line reports whether the remote has any channels configured.
- **No channels** is an amber banner reading `No notification channels configured`, body `Alert rules will be saved and evaluated, but no notifications will be dispatched. Configure Discord, Slack, Apprise, or a webhook in Settings → Notifications → Channels.`
- **Active** is a green banner reading `Notifications active via Discord, Slack, …` with the configured channels listed.
<Frame>
<img src="/images/alerts-notifications/alert-panel-remote-banner.png" alt="The Stack › SAELIX-DB › MONITOR sheet on a remote node, with the blue 'Remote node: node-a' banner explaining that rules are stored and evaluated on the remote, plus a follow-up amber line noting that no notification channels are configured on that remote." />
</Frame>
### Example: alert on high CPU
To page when CPU on a stack stays above 80% for at least five minutes, with no more than one alert per hour:
| Field | Value |
|-------|-------|
| **Metric** | CPU Usage (%) |
| **Operator** | Greater than |
| **Threshold** | `80` |
| **Duration (mins)** | `5` |
| **Cooldown (mins)** | `60` |
### Permissions and validation
Operators and viewers see existing rules read-only; only admins see the add and delete affordances. Submitting an empty threshold surfaces `Please enter a threshold.` Successful saves toast `Alert rule added.` and `Alert rule deleted.` A failed POST surfaces `Network error. Could not reach the node.`
The delete confirmation dialog reads `Delete Alert Rule` / `This will permanently remove this alert rule. Notifications for this condition will no longer be sent.` with a destructive **Delete** button.
## Notification history (the bell)
The bell icon at the top of the shell is the live feed of every alert across the fleet. A pulsing red dot surfaces unread items.
<Frame>
<img src="/images/alerts-notifications/notification-popover.png" alt="The notification bell open, with a 'Notifications' italic title, '9 UNREAD' uppercase mono caption, an All / Unread / Alerts segmented control, the filter-toggle, mark-all-read, and clear-all icon actions, a 'TODAY' day-band header, and a stack of severity-tinted rows describing Sencho version updates and a recent stack stop, each with a node-name pill, a relative timestamp, and a per-row dismiss target." />
</Frame>
### Anatomy
A title bar with `Notifications` in the italic display face, the unread count to the right (`<n> UNREAD`), and a row of controls below.
The segmented control switches between `All`, `Unread` (with badge), and `Alerts` (rows where level is `warning` or `error`). To the right, three icon buttons:
- **SlidersHorizontal** toggles a hidden filter row. A small dot on the icon means at least one filter is active.
- **CheckCheck** is `Mark all read`.
- **Trash2** is `Clear all`. The action issues `DELETE /api/notifications` against every node that contributed a row.
The hidden filter row has two combobox dropdowns: **Filter by node** (only rendered with two or more nodes registered) and **Filter by category**, listing all the user-facing labels from the categories table above.
<Frame>
<img src="/images/alerts-notifications/bell-filter-row.png" alt="The bell with its filter row expanded, the 'ALL TYPES' combobox open over a long popover listing every notification category from 'All types' through 'System', and the underlying notification rows visible beneath." />
</Frame>
### Row anatomy
Each row carries:
- A **3 px left rail** colored by severity: brand cyan for `info`, amber for `warning`, destructive rose for `error`. Rail saturation drops to about 30% once the row is read.
- A **severity icon** in the same hue: `Info`, `AlertTriangle`, or `AlertOctagon`.
- The **message body**, slightly dimmer once read.
- A **kicker line** below the message with an optional **node-name pill** (only rendered for multi-node fleets), a `·` separator, and a relative timestamp (`just now`, `Nm ago`, `Nh ago`, `yesterday`, then a short locale date).
- A **dismiss** target on hover, top-right.
There is no level chip in the row; the rail color and icon do that work. Rows whose payload carries a `stack_name` become click targets that jump to the stack and, when a `container_name` is present, surface that container's logs.
### Day groupings
Rows are grouped under uppercase day-band headers: `TODAY`, `YESTERDAY`, `THIS WEEK`, `EARLIER`.
### Empty states
<Frame>
<img src="/images/alerts-notifications/bell-empty.png" alt="The bell open in its empty state with a strikethrough bell icon, the headline 'You're all caught up', and the mono caption 'New notifications appear here in real time.'" />
</Frame>
The popover renders three different empty states:
- All filters, no rows ever: `You're all caught up` over `New notifications appear here in real time.`
- Unread filter, all read: `No unread notifications` over `Everything in your feed has been read.`
- Alerts filter, no warnings or errors: `No active alerts` over `Warnings and errors will surface here when they occur.`
### What you won't see in the bell
Sencho deliberately suppresses one class of notification from the popover: rows where the category is one of `deploy_success`, `stack_started`, `stack_stopped`, `stack_restarted`, or `image_update_applied`, AND the row carries an `actor_username` other than `system`. These are confirmations of an action you just clicked, surfaced as a toast at the moment of the action; replaying them in the bell would be noise. The rows are still persisted to `notification_history` and still dispatched to global channels and matching routes; only the bell render hides them.
### Limits
The popover holds the latest 50 rows per node per fetch. The backend caps `notification_history` at 100 rows per node, regardless of the user-tunable retention setting; the cap evicts the oldest rows on every insert. There is no `Load more` affordance; older rows roll off the cap.
## Cross-node delivery
The bell aggregates across the entire fleet by hitting `/api/notifications` against every registered node and opening one WebSocket per node: `/ws/notifications` for the local instance and `/ws/notifications?nodeId=<id>` for every remote. A new alert on any remote arrives in the bell within about a second; a 60-second safety-net poll catches anything the WebSocket dropped.
Per-stack alert rules and channel configuration are **stored on the node where the stack runs**. To configure a rule on a remote, switch the active node via the picker, open the stack's **Monitor** sheet, and the form's POST is forwarded to the remote.
Crash detection runs only on local Docker; remote nodes run their own copy of `DockerEventService` and emit through the proxy. On a multi-node fleet the bell attributes remotes with a node-name badge rather than embedding the satellite-local name in the message. External channel payloads (Slack, Discord, Apprise, custom webhook) receive the same sanitized body with no structured source-node id.
Switching the active node only affects per-stack rule editing and the Settings panels. The bell aggregates every node regardless of which one is active.
## Alerts emitted by the system
Sencho dispatches notifications from many code paths. The list below covers every emitter in current code, organized by source. Message bodies are node-neutral for local targets so fleet aggregation does not contradict the bell badge; remote-target names stay when a hub names an authoritative remote.
### Container crash and health
Real-time on the Docker event stream:
- **Crash** at `error`/`monitor_alert`: `Container Crash Detected: <name> exited unexpectedly (Code: <N>).`
- **OOM kill** at `error`/`monitor_alert`: `Container OOM Kill: <name> was killed by the OOM killer (out of memory).` When a `die` arrives with exit code 137 but no preceding `oom` event, Sencho inspects the container and reclassifies as OOM if the `OOMKilled` flag is set.
- **Healthcheck failure** at `error`/`monitor_alert`: `Healthcheck failed: <name> is unhealthy.`
- **Mass exit** kicks in when a daemon-disconnect gap is followed by 20% or more of containers exiting on reconnect, in which case a single `info`/`system` summary `Docker daemon interruption detected: N containers exited during connection gap.` is emitted instead of N crash alerts.
- **Rate limit** caps crash dispatches at 20 per 60-second window per node, then a single `warning`/`monitor_alert` `N additional containers crashed in the last minute.`
- **Dedup** is 60 minutes per container after a non-rate-suppressed dispatch.
### Daemon connectivity
- `warning`/`system`: `Lost connection to Docker daemon; monitoring paused.` (one-shot until reconnect)
- `info`/`system`: `Reconnected to Docker daemon.`
- `warning`/`system`: `Received malformed Docker event payloads. Monitoring continues but some events may be skipped.` (when more than 10 parse errors in a minute)
### Host thresholds
`warning`/`monitor_alert` for host CPU, RAM, and disk when the configured threshold is exceeded. Evaluated on the 30-second monitor tick. Example: `Host CPU utilization is critically high: 92% (Threshold: 90%)`.
Each metric (CPU, RAM, disk) carries its own suppression window. The first time a metric crosses its threshold, one notification fires immediately. While the metric stays over threshold within the window, additional cycles are silently counted but not dispatched. The next dispatch after the window elapses carries a summary suffix: `Suppressed 119 alerts in the last 60m; first over threshold at 14:23 UTC.` When the metric drops below threshold, the counter resets so the next breach fires fresh.
The suppression window defaults to 60 minutes and is configured per node in **Settings → Monitoring → Host Alerts → Alert suppression**.
The entire host threshold evaluation can be silenced per node from **Settings · Monitoring · Host Alerts · Host threshold alerts** while keeping the configured limit values. This toggle affects only the CPU, RAM, and disk threshold checks; crash capture, stack alert rules, and health gate checks all continue to run independently.
### Reclaimable Docker data
`info`/`system`: `This node has accumulated <N> GB of unused Docker data. Open Resources to reclaim space, or set up a Prune Node Resources schedule.` 24-hour cooldown. On a multi-node fleet the bell attributes remotes with a node-name badge rather than embedding the name in the message.
### Sencho version availability
`info`/`node_update_available`: `Sencho X.Y.Z is available (currently running A.B.C). Visit the Fleet dashboard to update.` Monitor evaluates every 30 seconds against a shared version cache (refreshes every 30 minutes when published, or every 3 minutes while publish is pending). One-shot dedup until the running version reaches or passes the notified version.
### Image update availability
`info`/`image_update_available`: `Stack "<name>" has image updates available.` On multi-service stacks the message can name the services that have updates. By default, checks every two hours in interval mode, with a two-minute startup delay. The cadence can be changed or replaced with a cron schedule. Manual refresh carries a two-minute cooldown. Notifies on **state transition only**; the first run also emits catch-up notifications for stacks already known to have updates.
### Auto-update execution
`info`/`image_update_applied`: `Auto-update: stack "<name>" updated with new images`. If a block-on-deploy policy gates the auto-update: `warning`/`scan_finding` `Policy "<name>" blocked auto-update: N image(s) exceed <severity>`. See [Auto-update policies](/features/auto-update-policies).
### Auto-heal
- `info`/`autoheal_triggered`: `Auto-Heal: Restarted <container> on stack <stack> after being unhealthy for <N> minute(s).`
- `warning`/`autoheal_triggered`: `Auto-Heal: Failed to restart <container> on stack <stack>. Error: <err>`
- `warning`/`autoheal_triggered`: `Auto-Heal: Policy for <stack>[/<svc>] has been auto-disabled after <N> consecutive failures.`
See [Auto-heal policies](/features/auto-heal-policies).
### Vulnerability scanning
- **Per-violation, during a scheduled scan** is `warning`/`scan_finding`. One alert per violation: `Policy "<name>" violated by <imageRef>: <severity> exceeds <maxSeverity>`.
- **Scan completion** is `info`/`scan_finding` (or `warning` if any image failed): `Scheduled scan "<task name>" completed: <output>` where `<output>` is one of `Scanned N image(s); X skipped (cached). Found A critical, B high, C medium.`, `No critical, high, or medium findings.`, `No images to scan`, or `All N image(s) already scanned recently (cache hit)`.
- **Pre-deploy gate when Trivy is missing** is `warning`/`scan_finding`: `Pre-deploy scan for "<stack>" skipped: Trivy not installed on this node`.
- **Post-deploy scan finding** is `error`/`scan_finding` if criticals are present, `warning` otherwise: `Vulnerability scan for <imageRef>: <N> critical, <M> high`.
- **Post-deploy scan failure** is `warning`/`scan_finding`: `Post-deploy scan failed for <imageRef> (<stack>): <msg>`.
- **Trivy auto-update** is `info`/`system`: `Trivy updated from vX to vY` or `Trivy update available: vX (currently vY)`.
See [Vulnerability scanning](/features/vulnerability-scanning).
### Scheduled tasks
- `error`/`system`: `Scheduled task "<name>" (<action>) failed: <err>`
- `info`/`system` recovery: `Scheduled task "<name>" (<action>) recovered successfully`
Local-target failure reasons stay node-neutral (`Target node is offline`, `Container "<n>" not found on this node. ...`). When the task targets a remote node, the failure reason keeps that node's roster name.
### Recovery Vault upload failure
`warning`/`system`: `Cloud backup failed for scheduled snapshot <id>: <message>`. See [Fleet backups](/features/fleet-backups).
### Blueprints
- `warning`/`blueprint_drift_detected`: `Blueprint "<n>" drifted on this node: <reason>` for a local target, or `Blueprint "<n>" drifted on node "<remote>": <reason>` for a remote. Stateful marker-loss uses the same local/remote clause.
- `error`/`blueprint_drift_correction_failed`: `Auto-fix for "<n>" on this node failed: <err>` (local) or `Auto-fix for "<n>" on node "<remote>" failed: <err>` (remote).
See [Blueprints](/features/blueprint-model).
### Fleet sync
Fleet Sync requires Admiral on the control instance.
- `warning`/`system` identity drift: `Fleet self-identity changed from "X" to "Y". Identity-scoped policies are reapplied on the next sync.`
- `warning`/`system` truncation: `Fleet sync truncated <resource> to <N> rows (<dropped> not replicated). Reduce the local set or contact support.`
- `warning`/`system` stale: `Fleet sync to node "<n>" has been failing for over an hour for <resource>. Check the node's connectivity and API token.`
## Retention and limits
Three retention controls live under **Settings · Operations · Data Retention**.
<Frame>
<img src="/images/alerts-notifications/developer-retention.png" alt="Settings · Operations · Data Retention card showing three rows: Container metrics with a 24 HRS field, Notification log with a 30 DAYS field, and Audit log (Admiral) with a 90 DAYS field, plus a SAVE SETTINGS button." />
</Frame>
| Control | Range | Default | What it prunes |
|---------|-------|---------|----------------|
| **Container metrics** | 1 to 8760 hours | 24 hours | Per-container CPU, RAM, and network history used by the dashboard sparklines |
| **Notification log** | 1 to 365 days | 30 days | The bell's `notification_history` table |
| **Audit log** *(Admiral)* | 1 to 365 days | 90 days | Audit trail entries |
A hard 100-row per-node cap inside `notification_history` always applies on top of the user-tunable retention; new inserts evict the oldest rows beyond 100 even if your retention window is longer.
A separate rate limit applies to crash and health alerts only: 20 emits per 60-second window per node, with a single `warning`/`monitor_alert` roll-up `N additional containers crashed in the last minute.` issued at the end of the window.
## Crash detection toggle
The global crash-capture switch lives under **Settings · Monitoring · Container Alerts**.
<Frame>
<img src="/images/alerts-notifications/system-crash-toggle.png" alt="Settings · Monitoring · Container Alerts panel showing a Container crash & health alerts toggle in the ON state with the helper line about unexpected exits, OOM kills, and healthcheck failures." />
</Frame>
The **Container crash & health alerts** toggle controls whether `DockerEventService` raises crash, OOM, and healthcheck alerts on the active node. Helper text: `Send alerts for unexpected container exits, OOM kills, and Docker healthcheck failures. Auto-Heal can still observe crash signals independently.` Defaults to on; if the database read fails, Sencho falls back to default-deny so the system never leaks alerts you cannot turn off.
The **Host Alerts** panel carries the **Host thresholds** rows (CPU limit, RAM limit, Disk limit, all expressed as percent) that drive the host-level monitor warnings. Container crash and health alerts live in the **Container Alerts** panel. The **Reclaimable Docker data threshold** (in GiB) that drives the unused-Docker-data alert lives in the **Docker & Storage** panel.
## Refresh cadence
| Surface | Cadence |
|---------|---------|
| Crash, OOM, and healthcheck events | Real-time over the Docker event stream |
| Host CPU / RAM / disk threshold checks | 30 seconds |
| Per-stack alert rule evaluation | 30 seconds |
| Image update poll | Configurable (default every 2 hours), with a 2-minute startup delay and a 2-minute cooldown on manual refresh |
| Sencho version check | Monitor evaluation every 30 seconds; shared version cache refreshes every 30 minutes when published or every 3 minutes while publish is pending |
| Notification fanout to channels | One attempt by default; optional 0-3 extra in-process attempts with a fixed 1s delay; 10-second timeout per attempt; no durable queue |
| Bell live updates | Pushed live over the notifications WebSocket per node |
| Bell safety-net reconcile | 60 seconds |
| Crash dedup window per container | 60 minutes |
| Crash rate-limit window | 20 alerts per 60 seconds per node, then a single roll-up |
Switching the active node tears down per-stack rule editors and reloads channel state, but does not affect the bell, which keeps every node's WebSocket open in parallel.
## Troubleshooting
<AccordionGroup>
<Accordion title="Notifications never arrive">
Check three things in order. First, the channel toggle in **Settings · Notifications · Channels** must be on; the kicker on each tab reads `enabled` or `off`. Second, Discord, Slack, and webhook URLs must use HTTPS (the form rejects plain `http://` for those channels); Apprise endpoints may use HTTP or HTTPS. Third, a routing rule with unconstrained Node plus empty Stacks, Labels, Categories, and Severity matchers will intercept every alert and skip the global channels. Use the per-channel **Test** button to issue a one-shot dispatch and watch your endpoint for the literal message `🔌 Test Notification from Sencho!` Sencho records the failure reason in `notification_history.dispatch_error` when delivery throws, so a row that appears in the bell with no follow-up at the endpoint usually means a 4xx or timeout at the receiver.
</Accordion>
<Accordion title="An alert rule never fires even when the threshold is breached">
Three causes account for almost every case. First, the rule's **Duration** has not elapsed yet: the breach must persist for the full duration before the rule fires. Second, the rule is still in cooldown after a previous fire. Third, the panel's banner is not green: a remote-node banner means the rule was saved on a remote whose channels you may not have configured, and an amber `No notification channels configured` banner means the rule evaluates fine but Sencho has nowhere to send the alert. The evaluator runs on a 30-second tick, so expect up to 30 seconds of latency between the breach starting and the timer engaging.
</Accordion>
<Accordion title="I stopped a stack but got a crash alert anyway">
The Docker event service defers `die` classification by 500 ms to absorb out-of-order `kill` events from the daemon, then asks the lifecycle classifier whether the exit was intentional, clean (exit 0), a crash, or an OOM kill. If your stop happened far enough outside that window, or the daemon emitted the events without the kill marker the classifier looks for, the exit can be classified as a crash. The classifier favors avoiding silent crashes over avoiding noisy false positives. Compare the alert timestamp against your `docker compose down` time; entries within a second of each other are usually the same event seen from two angles.
</Accordion>
<Accordion title="After a Docker daemon restart, the bell only has one summary line">
When the daemon disconnects and reconnects, Sencho snapshots every container at the moment of reconnect. If at least 20% exited during the gap, the service emits a single `info`/`system` summary `Docker daemon interruption detected: N containers exited during connection gap.` instead of one crash alert per container. Below the threshold, every gap exit is classified individually.
</Accordion>
<Accordion title="Crash alerts stopped arriving and nothing else looks wrong">
Check **Settings · Monitoring · Container Alerts · Container crash & health alerts**. The toggle is the master switch for the Docker event service. The panel cache reads the database every 500 ms; if the database read errors, Sencho defaults the toggle to off so the failure mode is silent rather than spammy. Repair the toggle, save, and the next Docker event reaches the dispatcher.
</Accordion>
<Accordion title="A burst of crashes happened but I only see about twenty alerts">
Sencho rate-limits crash and health dispatches to 20 emits per rolling 60-second window per node, then emits a single `warning`/`monitor_alert` roll-up `N additional containers crashed in the last minute.` once the window closes. Every alert is still persisted to `notification_history` and visible in the bell up to the 100-row per-node cap; only the channel fanout is throttled.
</Accordion>
<Accordion title="My deploy success doesn't appear in the bell, only as a toast">
Sencho deliberately hides rows whose category is one of `deploy_success`, `stack_started`, `stack_stopped`, `stack_restarted`, or `image_update_applied` AND whose `actor_username` is set to a real user. The reasoning: those are confirmations of the action you just clicked and are already shown as a toast. The rows are still persisted to `notification_history` and still dispatched to global channels and matching routes; only the bell render hides them.
</Accordion>
<Accordion title="A routing rule is set up but the global channel still fires">
Routing matchers AND together: every non-empty matcher must match the alert. A rule with **Stacks** set to `prod-api` will not match a `monitor_alert` for a different stack, and a rule with a populated **Stacks** matcher will not match host-level alerts (which carry no stack target). Stack patterns may use `*` as a wildcard (for example `prod-*`). When zero rules match, Sencho falls back to global channels. To intercept everything, leave Node as any and leave Stacks, Labels, Categories, and Severity empty. Also confirm the rule's **Enabled** pill is `ON`.
</Accordion>
<Accordion title="Fleet shows an update button but I never received a Sencho version notification">
Fleet and the notification path share the same version lookup. When a newer published release is in that cache, Monitor dispatches a single `info`/`node_update_available` alert and records it under `last_sencho_update_notified_version`, so each release produces one alert per node. If you already upgraded to (or past) that version, the dedup self-heals and the alert does not fire. Check **Settings · Notifications · Mute Rules** for a rule that mutes `node_update_available` in the bell. A newly published release may take up to the cache TTL (about 30 minutes when published, or about 3 minutes while registry publish is still pending) before Fleet and the bell both observe it.
</Accordion>
<Accordion title="A stack shows the blue update indicator but no notification was received">
The image-update service emits on **state transitions only** after its first run. The first run also emits catch-up notifications for stacks already known to have updates, then sets a backfill flag. Once that flag is set, only a false-to-true transition triggers another alert. If you dismissed the catch-up notification, a new one appears the next time the stack goes from up-to-date back to behind.
</Accordion>
</AccordionGroup>
Reference in New Issue View Git Blame Copy Permalink