diff --git a/docs/docs.json b/docs/docs.json
index e5c5f388..761b6b94 100644
--- a/docs/docs.json
+++ b/docs/docs.json
@@ -118,7 +118,6 @@
"features/global-search",
"features/global-observability",
"features/alerts-notifications",
- "features/notification-routing",
"features/audit-log"
]
},
@@ -174,7 +173,6 @@
"features/licensing",
"features/node-compatibility",
"security",
- "reference/security-advisories",
"reference/contact"
]
},
diff --git a/docs/features/alerts-notifications.mdx b/docs/features/alerts-notifications.mdx
index 418bdf38..a9b4c95d 100644
--- a/docs/features/alerts-notifications.mdx
+++ b/docs/features/alerts-notifications.mdx
@@ -1,275 +1,424 @@
---
title: Alerts & Notifications
-description: Set threshold-based alerts on container metrics and route them to Discord, Slack, or any webhook.
+description: Threshold and event alerts for your fleet, dispatched to Discord, Slack, or any webhook, with per-stack rules and Admiral routing.
---
-Sencho can watch your containers for resource anomalies and notify you when thresholds are breached. Alerts are defined per stack, and notifications are delivered through external channels you configure.
-
-## Setting up notification channels
-
-At least one channel must be enabled before alerts can be delivered externally. Go to **Settings > Notifications**.
+Sencho watches each node it manages for container crashes, host pressure, scheduled-task results, and update availability, then surfaces every signal in two places: the in-app notification bell at the top of the shell and one of three external channels you configure. This page covers everything from configuring channels to writing per-stack threshold rules, routing alerts to dedicated channels with Admiral routing rules, and tuning retention.
- 
+
-Three channel types are available, each configured with a webhook URL and an enable/disable toggle:
+## Notification channels
+
+Open **Settings · Notifications** to configure the three channel types. Each channel is per-node, so switching the active node via the node picker reloads the panel against that node's stored settings. The masthead carries a `CHANNELS` stat showing how many of the three slots are enabled.
+
+Each tab carries the same controls: an **Enabled** toggle (helper: `Send Sencho events to this channel.`), a **Webhook URL** input (placeholder `https://...`, helper: `Sencho posts JSON payloads here. Use a private channel.`), and the **Test** and **Save** buttons. The kicker on each tab toggles between `enabled` and `off` so you can see at a glance which slots are wired up.
+
+
+ Webhook URLs must use HTTPS. Sencho rejects `http://` and any string that does not parse as a URL. The same rule applies to test sends and to routing-rule URLs.
+
### Discord
-1. In Discord, go to your server's **Settings > Integrations > Webhooks**
-2. Click **New Webhook**, choose a channel, and copy the webhook URL
-3. In Sencho, open **Settings > Notifications > Discord**, paste the URL, enable the toggle, and click **Save**
-4. Click **Test** to send a test message and verify the connection
+Paste an incoming webhook URL from your Discord channel's **Edit Channel · Integrations · Webhooks** view. Sencho posts a single embed per alert, with the title `Sencho Alert []`, the message in the description, and the embed color set by severity (info blue, warning yellow, error red).
### Slack
-1. In Slack, go to **api.slack.com/apps**, create an app, and add the **Incoming Webhooks** feature
-2. Activate it and copy the generated webhook URL for your chosen channel
-3. In Sencho, open **Settings > Notifications > Slack**, paste the URL, enable the toggle, and click **Save**
+Paste an incoming webhook URL from a Slack app installed in your workspace (URL shape `https://hooks.slack.com/services/...`). Sencho posts a single text message: ` *Sencho Alert []*` followed by the message body, with the emoji set per severity (`ℹ️` info, `⚠️` warning, `🚨` error).
-### Generic Webhook
+### Generic webhook
-Any HTTPS endpoint that accepts a POST with a JSON body can receive Sencho alerts. Go to **Settings > Notifications > Webhook**, enter the URL, enable the toggle, and click **Save**.
-
-
- All webhook URLs (Discord, Slack, and generic) must use HTTPS. HTTP URLs are rejected.
-
-
-The payload format is:
+Use the Generic Webhook tab when you have your own receiver: a Mattermost or Teams adapter, a serverless function that fans out to email or SMS, or a logging endpoint. Sencho posts JSON of the shape:
```json
{
"level": "warning",
- "message": "cpu_percent exceeded 90% for 5 minutes on stack my-app",
- "timestamp": "2026-03-22T10:00:00.000Z"
+ "message": "[Node: Local] The cpu_percent for plex has exceeded your threshold of 80% (Currently: 91%).",
+ "timestamp": "2026-05-08T22:14:09.812Z",
+ "source": "sencho"
}
```
-## Creating stack alerts
+`level` is one of `info`, `warning`, `error`. `source` is always the literal string `sencho`. `timestamp` is ISO-8601 with millisecond precision.
-Stack alerts are configured per stack. Right-click a stack in the sidebar (or click the three-dot menu) and select **Alerts**. A panel slides open showing existing rules and a form to create new ones.
+### Test sends and delivery semantics
+
+The **Test** button on each tab dispatches the literal message `🔌 Test Notification from Sencho!` at level `info` through the same path a real alert would take. Test sends require the admin role; the server returns 403 if an operator or viewer submits one.
+
+Each dispatch is a single-shot HTTP POST with a 10-second `AbortSignal.timeout`. There is no retry queue. If your endpoint is down at the moment of the dispatch, the alert is recorded in the bell with an internal `dispatch_error` field set and is not redelivered.
+
+## Notification Routing
+
+
+ Notification Routing requires a **Sencho Admiral** license. Admin role is required to create, edit, or delete routes.
+
+
+Routing lets you direct alerts that match specific criteria to dedicated channels. Production crashes can land in `#prod-incidents` on Slack while staging notifications go to a less urgent Discord channel, all without juggling per-channel webhook URLs across teams.
- 
+ 
-The panel includes:
+### How routing fits into dispatch
-- **Notification status banner** at the top, showing whether channels are configured. If no channels are enabled, a warning explains that rules will be evaluated but no external notifications will be sent.
-- **Existing Rules** section listing all active rules for this stack, each showing the metric, condition, duration, and cooldown. Hover over a rule to reveal the delete button.
-- **Add New Rule** form with the fields described below.
+For every alert Sencho dispatches, the routing engine evaluates every enabled route. A route matches when **all of its non-empty matchers** match the alert: the **Node**, **Stacks**, **Labels**, and **Categories** filters compose with AND. An empty matcher is treated as match-anything.
+
+If at least one route matches, every matching route fires and the **global channels are skipped** for that alert. If zero routes match, the alert falls back to the global channels configured in the previous section.
+
+A route with all four matchers left empty matches every alert and intercepts global delivery entirely. Stack-less alerts (host CPU, RAM, disk, fleet-sync warnings, and scheduled-task results without a stack target) almost always miss any populated `Stacks` filter, so they fall back to global by default.
+
+### Creating a routing rule
+
+Open **Settings · Routing** and click **+ Add Route**.
+
+
+ 
+
+
+| Field | Purpose |
+|-------|---------|
+| **Name** | A human label, up to 100 characters. Shown on the rule card. |
+| **Node scope** | Either `Any node` or a specific node. When set to a node, the rule only matches alerts originating from that node. |
+| **Stacks** *(optional)* | A combobox of stacks on the active node. Selected stacks appear as removable pills. Empty matches any stack. |
+| **Labels** *(optional)* | A combobox of stack labels on the active node. Empty matches any label. |
+| **Categories** *(optional)* | A combobox of notification categories. The helper line reads `Leave blank to match all categories. All non-empty filters must match (AND).` |
+| **Channel** | Tabs for Discord, Slack, and Webhook with a URL input below. URL must use HTTPS. |
+| **Priority** | A number used to sort the rule list. Lower numbers appear higher up. Priority does not gate dispatch: when multiple rules match the same alert, every matching rule fires concurrently. |
+| **Enabled** | Toggle the rule on or off without deleting it. |
+
+The modal kicker reads `ROUTING · NEW RULE` when adding and `ROUTING · EDIT RULE` when modifying.
+
+### Managing rules
+
+Each rule renders as a card on the Routing page with the rule name, channel-type badge, an `ON` / `OFF` pill, then a row of small badges naming each matcher: one mono badge per **Stack**, one outline badge per **Label**, one outline mono badge per **Category** label. When all matchers are empty, a single muted `Matches all alerts` line replaces the badge row. After the badges, a vertical bar separator is followed by the truncated channel URL, then a second separator and a `Priority: N` chip when priority is non-zero.
+
+Three icon actions appear on the right edge of each card:
+
+- **Lightning** sends a test message through the rule's channel using the same payload shape as the global Test button.
+- **Pencil** opens the edit modal pre-filled with the current values.
+- **Trash** deletes the rule after a confirmation dialog.
+
+The masthead carries `SCOPE` (`global`), `ROUTES` (total), and `ENABLED` (count).
+
+## Notification categories
+
+Every alert Sencho dispatches carries a category that you can filter on in the bell, target with a routing rule's **Categories** matcher, or reason about when reading audit history.
+
+| Category | Label in the bell | Trigger |
+|----------|-------------------|---------|
+| `deploy_success` | Deploy success | Stack deploy completed without error |
+| `deploy_failure` | Deploy failure | Stack deploy returned a non-zero exit |
+| `stack_started` | Stack started | Stack started via the dashboard or API |
+| `stack_stopped` | Stack stopped | Stack stopped via the dashboard or API |
+| `stack_restarted` | Stack restarted | Stack restarted via the dashboard or API |
+| `image_update_available` | Update available | Image-update poll found a newer digest |
+| `image_update_applied` | Update applied | Manual or scheduled auto-update applied new images |
+| `autoheal_triggered` | Auto-heal | Auto-heal restarted, failed to restart, or auto-disabled a policy |
+| `monitor_alert` | Monitor alert | Per-stack threshold breach, host CPU/RAM/disk warning, or healthcheck failure |
+| `scan_finding` | Scan finding | Vulnerability-scan completion, per-violation alert, post-deploy scan failure, or auto-update gate block |
+| `system` | System | Sencho version, Trivy auto-update, fleet sync, daemon connectivity, scheduled-task lifecycle, cloud-backup upload failure |
+| `blueprint_deployed` | `blueprint_deployed` | Blueprint provisioned a new deployment (Admiral) |
+| `blueprint_deployment_failed` | `blueprint_deployment_failed` | Blueprint deployment errored out (Admiral) |
+| `blueprint_drift_detected` | `blueprint_drift_detected` | Blueprint drift detected in `suggest` or `enforce` mode (Admiral) |
+| `blueprint_drift_correction_failed` | `blueprint_drift_correction_failed` | Blueprint enforce-mode redeploy failed (Admiral) |
+
+The four `blueprint_*` categories are accepted by routing rules but render as raw category strings in the bell because the frontend label map omits them.
+
+## Per-stack alert rules
+
+Each stack carries its own set of threshold rules that fire when a metric stays above (or below) a value for a configurable window. Rules live on the node where the stack runs and are evaluated locally on a 30-second tick.
+
+Open the rules editor by right-clicking a stack in the sidebar and choosing **Alerts**, or by pressing `A` while focused on a stack.
+
+
+
+
### Alert fields
-| Field | Description |
-|-------|-------------|
-| **Metric** | The container metric to watch (see table below) |
-| **Operator** | Comparison: Greater than, Greater or equal, Less than, Less or equal, Equals |
-| **Threshold** | The numerical value to compare against |
-| **Duration (mins)** | How long the condition must hold before firing (default: 5) |
-| **Cooldown (mins)** | Minimum time between repeated notifications for this rule (default: 60) |
+| Field | Purpose |
+|-------|---------|
+| **Metric** | The system resource or metric to monitor. |
+| **Operator** | Comparison: `Greater than`, `Greater or eq`, `Less than`, `Less or eq`, `Equals`. |
+| **Threshold** | A number ≥ 0. The unit follows the chosen metric. |
+| **Duration (mins)** | How long the breach must persist before firing. Default `5`, range 0 to 1440. |
+| **Cooldown (mins)** | Silence window between fires after a rule triggers. Default `60`, range 0 to 10080. |
+
+Sencho tracks the start of each breach in memory; the rule fires only after the breach has lasted for the full **Duration**, and only if the previous fire is older than **Cooldown**.
### Available metrics
-| Metric | Description |
-|--------|-------------|
-| CPU Usage (%) | CPU usage relative to total host cores |
-| Memory Usage (%) | Memory used as a fraction of the host total |
-| Memory Usage (MB) | RSS memory used by the container |
-| Network In (MB) | Cumulative inbound network bytes (in MB) |
-| Network Out (MB) | Cumulative outbound network bytes (in MB) |
-| Restart Count | Number of times the container has restarted |
+| Metric | UI label | Unit |
+|--------|----------|------|
+| `cpu_percent` | CPU Usage (%) | percent of all cores |
+| `memory_percent` | Memory Usage (%) | percent of container memory limit |
+| `memory_mb` | Memory Usage (MB) | MB of resident set size, minus cache |
+| `net_rx` | Network In (MB/s) | MB per second, computed as a delta between consecutive samples |
+| `net_tx` | Network Out (MB/s) | MB per second, computed as a delta between consecutive samples |
+| `restart_count` | Restart Count | integer count of Docker-reported restarts |
+
+### Channel banner states
+
+The **NOTIFICATION CHANNELS** banner above the rules list reflects what dispatch will look like for this stack:
+
+- **Loading** is a spinner with `Checking notification channels...` while Sencho asks the target node for its agent state.
+- **Remote node** is a blue banner reading `Remote node: `, with the body `Alert rules are stored and evaluated on this remote instance. Notifications are dispatched using that node's configured channels.` A sub-line reports whether the remote has any channels configured.
+- **No channels** is an amber banner reading `No notification channels configured`, body `Alert rules will be saved and evaluated, but no notifications will be dispatched. Configure Discord, Slack, or a webhook in Settings → Notifications.`
+- **Active** is a green banner reading `Notifications active via Discord, Slack, …` with the configured channels listed.
+
+
+ 
+
### Example: alert on high CPU
-To alert when any container in a stack uses more than 80% CPU for over 5 consecutive minutes, with a 60-minute cooldown:
+To page when CPU on a stack stays above 80% for at least five minutes, with no more than one alert per hour:
| Field | Value |
|-------|-------|
-| Metric | CPU Usage (%) |
-| Operator | Greater than |
-| Threshold | 80 |
-| Duration | 5 |
-| Cooldown | 60 |
+| **Metric** | CPU Usage (%) |
+| **Operator** | Greater than |
+| **Threshold** | `80` |
+| **Duration (mins)** | `5` |
+| **Cooldown (mins)** | `60` |
-## Notification history
+### Permissions and validation
-All dispatched notifications appear in the **notification bell** in the top-right corner of the navigation bar. A red dot pulses on the bell when unread notifications exist.
+Operators and viewers see existing rules read-only; only admins see the add and delete affordances. Submitting an empty threshold surfaces `Please enter a threshold.` Successful saves toast `Alert rule added.` and `Alert rule deleted.` A failed POST surfaces `Network error. Could not reach the node.`
+
+The delete confirmation dialog reads `Delete Alert Rule` / `This will permanently remove this alert rule. Notifications for this condition will no longer be sent.` with a destructive **Delete** button.
+
+## Notification history (the bell)
+
+The bell icon at the top of the shell is the live feed of every alert across the fleet. A pulsing red dot surfaces unread items.
- 
+
-Click the bell to open the notification popover. Each entry shows:
+### Anatomy
-- **Level badge** (ERROR in red, WARNING in amber, INFO in default)
-- **Node name** badge for notifications from remote nodes
-- **Timestamp** of when the alert was triggered
-- **Alert message** describing what was breached
+A title bar with `Notifications` in the italic display face, the unread count to the right (` UNREAD`), and a row of controls below.
-A toolbar above the list provides filtering and bulk actions:
+The segmented control switches between `All`, `Unread` (with badge), and `Alerts` (rows where level is `warning` or `error`). To the right, three icon buttons:
-| Control | What it does |
-|---------|--------------|
-| **All / Unread / Alerts** | Switches the list between every notification, unread only, or alert-level only |
-| **Filter toggle** | Reveals dropdowns to filter by node and by notification type; an accent dot on the icon indicates an active filter |
-| **Mark all as read** | Marks every notification as read (removes the red dot) |
-| **Clear all** | Deletes every notification from the list |
+- **SlidersHorizontal** toggles a hidden filter row. A small dot on the icon means at least one filter is active.
+- **CheckCheck** is `Mark all read`.
+- **Trash2** is `Clear all`. The action issues `DELETE /api/notifications` against every node that contributed a row.
-You can also dismiss individual notifications by hovering over them and clicking the dismiss button.
+The hidden filter row has two combobox dropdowns: **Filter by node** (only rendered with two or more nodes registered) and **Filter by category**, listing all the user-facing labels from the categories table above.
-
- Notifications only reach external channels (Discord, Slack, Webhook) if at least one channel is enabled. Dashboard notifications appear regardless.
-
+
+ 
+
-## Alerts on remote nodes
+### Row anatomy
-Alerts work the same way on remote nodes as they do locally. When you switch to a remote node and open a stack's alerts panel, you are managing rules on that remote instance.
+Each row carries:
-Key details:
+- A **3 px left rail** colored by severity: brand cyan for `info`, amber for `warning`, destructive rose for `error`. Rail saturation drops to about 30% once the row is read.
+- A **severity icon** in the same hue: `Info`, `AlertTriangle`, or `AlertOctagon`.
+- The **message body**, slightly dimmer once read.
+- A **kicker line** below the message with an optional **node-name pill** (only rendered for multi-node fleets), a `·` separator, and a relative timestamp (`just now`, `Nm ago`, `Nh ago`, `yesterday`, then a short locale date).
+- A **dismiss** target on hover, top-right.
-- **Alert rules are stored on each node independently.** Rules created while a remote node is selected are saved on that remote instance, not your primary instance.
-- **Monitoring runs locally on each node.** Each Sencho instance evaluates its own alert rules against its own container metrics.
-- **Notifications are sent by the node that detects the breach.** Make sure notification channels are configured on each remote node where you want to receive alerts, since channel settings are per-instance.
+There is no level chip in the row; the rail color and icon do that work. Rows whose payload carries a `stack_name` become click targets that jump to the stack and, when a `container_name` is present, surface that container's logs.
-The alert panel shows a blue info banner when you are configuring alerts on a remote node, including which channels are active on that node.
+### Day groupings
-### Setup checklist for remote alerts
+Rows are grouped under uppercase day-band headers: `TODAY`, `YESTERDAY`, `THIS WEEK`, `EARLIER`.
-You can configure a remote node's notification channels directly from the control plane:
+### Empty states
-1. From your **primary** instance, switch to the remote node using the node picker
-2. Open **Settings > Notifications** and configure the remote's Discord, Slack, or Webhook channels. Values saved here apply only to the selected node.
-3. Right-click a stack and select **Alerts** to create rules
-4. The remote instance handles monitoring and notification delivery independently
+
+ 
+
-## Update availability notifications
+The popover renders three different empty states:
-Sencho can notify you when software updates are available, both for Sencho itself and for your stack images.
+- All filters, no rows ever: `You're all caught up` over `New notifications appear here in real time.`
+- Unread filter, all read: `No unread notifications` over `Everything in your feed has been read.`
+- Alerts filter, no warnings or errors: `No active alerts` over `Warnings and errors will surface here when they occur.`
-### Sencho version updates
+### What you won't see in the bell
-When a newer version of Sencho is published, an informational notification is dispatched through your configured channels. Each Sencho instance runs its own version check roughly once every 6 hours and notifies exactly once per new version, so remote nodes self-report their own availability alerts. After you update, the cycle resets and you will be notified when the next release becomes available.
+Sencho deliberately suppresses one class of notification from the popover: rows where the category is one of `deploy_success`, `stack_started`, `stack_stopped`, `stack_restarted`, or `image_update_applied`, AND the row carries an `actor_username` other than `system`. These are confirmations of an action you just clicked, surfaced as a toast at the moment of the action; replaying them in the bell would be noise. The rows are still persisted to `notification_history` and still dispatched to global channels and matching routes; only the bell render hides them.
-The notification message includes the version you are running and the version that was detected, for example:
+### Limits
-```
-Sencho 0.47.0 is available (currently running 0.46.16). Visit the Fleet dashboard to update.
-```
+The popover holds the latest 50 rows per node per fetch. The backend caps `notification_history` at 100 rows per node, regardless of the user-tunable retention setting; the cap evicts the oldest rows on every insert. There is no `Load more` affordance; older rows roll off the cap.
-### Stack image updates
+## Cross-node delivery
-When the periodic image check (every 6 hours) detects that a stack has new upstream images available, a notification is dispatched for each affected stack. Notifications are sent only on state transitions: you will be notified once when an update first appears, not on every check cycle. After you update the stack, the status resets so a future release can notify again.
+The bell aggregates across the entire fleet by hitting `/api/notifications` against every registered node and opening one WebSocket per node: `/ws/notifications` for the local instance and `/ws/notifications?nodeId=` for every remote. A new alert on any remote arrives in the bell within about a second; a 60-second safety-net poll catches anything the WebSocket dropped.
-Both notification types use the same channel routing as alerts: if notification routes are configured for a stack, those channels receive the message; otherwise, global notification channels are used as a fallback.
+Per-stack alert rules and channel configuration are **stored on the node where the stack runs**. To configure a rule on a remote, switch the active node via the picker, open the stack's **Monitor** sheet, and the form's POST is forwarded to the remote.
-## Scheduled scan completion
+Crash detection runs only on local Docker; remote nodes run their own copy of `DockerEventService` and emit through the proxy. Each emitted message is prefixed with `[Node: ]` so the source is unambiguous in the channel.
-Recurring [vulnerability scans](/features/vulnerability-scanning) dispatch a notification whenever a run finishes so you do not have to check the task history manually.
+Switching the active node only affects per-stack rule editing and the Settings panels. The bell aggregates every node regardless of which one is active.
-- **Info** when every image scanned successfully.
-- **Warning** when one or more images failed to scan during the run.
-- **Error** when the run itself could not start (for example, Trivy is not installed on the target node).
+## Alerts emitted by the system
-The message includes the scheduled task name, a summary of how many images were scanned, cached, and failed, and a breakdown of findings by severity. A typical clean-run message looks like:
+Sencho dispatches notifications from many code paths. The list below covers every emitter in current code, organized by source. Each message is prefixed with `[Node: ]` when emitted by a node-scoped service.
-```
-Scheduled scan "nightly-scan" completed: Scanned 12 image(s); 3 skipped (cached). Found 2 critical, 5 high, 10 medium.
-```
+### Container crash and health
-If no critical, high, or medium findings are present, the message ends with `No critical, high, or medium findings.` so the outcome is still explicit. When the target node has nothing to scan, the message reads `No images to scan.`, and when every image was already covered by a recent cached scan it reads `All N image(s) already scanned recently (cache hit).` In both cases the notification still fires so you know the run executed.
+Real-time on the Docker event stream:
-
- Severity counts reflect the current security posture of the node, aggregated across both freshly scanned images and cached scan results. They are not a delta of what changed on this run.
-
+- **Crash** at `error`/`monitor_alert`: `Container Crash Detected: exited unexpectedly (Code: ).`
+- **OOM kill** at `error`/`monitor_alert`: `Container OOM Kill: was killed by the OOM killer (out of memory).` When a `die` arrives with exit code 137 but no preceding `oom` event, Sencho inspects the container and reclassifies as OOM if the `OOMKilled` flag is set.
+- **Healthcheck failure** at `error`/`monitor_alert`: `Healthcheck failed: is unhealthy.`
+- **Mass exit** kicks in when a daemon-disconnect gap is followed by 20% or more of containers exiting on reconnect, in which case a single `info`/`system` summary `Docker daemon interruption detected: N containers exited during connection gap.` is emitted instead of N crash alerts.
+- **Rate limit** caps crash dispatches at 20 per 60-second window per node, then a single `warning`/`monitor_alert` `N additional containers crashed in the last minute.`
+- **Dedup** is 60 minutes per container after a non-rate-suppressed dispatch.
-Failures are typically transient (registry timeouts, missing credentials) and do not stop the rest of the run from completing.
+### Daemon connectivity
-## Container crash detection
+- `warning`/`system`: `Lost connection to Docker daemon; monitoring paused.` (one-shot until reconnect)
+- `info`/`system`: `Reconnected to Docker daemon.`
+- `warning`/`system`: `Received malformed Docker event payloads. Monitoring continues but some events may be skipped.` (when more than 10 parse errors in a minute)
-Sencho watches every container on each of your nodes in real time and notifies you when something exits unexpectedly. Detection is causal: Sencho distinguishes crashes from intentional stops, so stopping a stack, restarting it, or running `docker compose down` will not produce a false crash alert.
+### Host thresholds
-### What triggers a crash alert
+`warning`/`monitor_alert` for host CPU, RAM, and disk when the configured threshold is exceeded. 5-minute cooldown per signal. Example: `Host CPU utilization is critically high: 92% (Threshold: 90%)`. Evaluated on the 30-second monitor tick.
-| Situation | Alert |
-|---|---|
-| Container exits with a non-zero exit code without being asked to stop | **Crash** alert (level: error) |
-| Container is killed by the kernel for exceeding its memory limit | **OOM Kill** alert (level: error) |
-| Healthcheck reports the container as unhealthy | **Healthcheck failed** alert (level: error) |
+### Docker janitor
-Alerts arrive within a couple of seconds of the event, not on a polling interval.
+`info`/`system`: `Node "" has accumulated GB of unused Docker data. Consider using the Janitor tool.` 24-hour cooldown.
-### What does not trigger a crash alert
+### Sencho version availability
-- Stopping, restarting, updating, or removing a stack from Sencho
-- Running `docker stop`, `docker restart`, or `docker compose down` from a terminal on the host
-- A container exiting cleanly with exit code `0`
-- A container being replaced during an image update
+`info`/`system`: `Sencho X.Y.Z is available (currently running A.B.C). Visit the Fleet dashboard to update.` Polled every 6 hours; one-shot dedup until the running version reaches or passes the notified version.
-### Global toggle
+### Image update availability
-Crash and unhealthy alerts are gated by the **Global Crash Detection** toggle in **Settings > System**. When disabled, Sencho stops dispatching crash and unhealthy notifications on all nodes. Stack metric alerts and update notifications are unaffected by this toggle.
+`info`/`image_update_available`: `Stack "" has image updates available.` Polled every 6 hours, with a 2-minute startup delay and a 2-minute cooldown on manual refresh. Notifies on **state transition only**; pre-existing `has_update` rows are backfilled silently on first run.
-### Docker daemon interruptions
+### Auto-update execution
-If the Docker daemon becomes unreachable (daemon restart, socket lost, network issue on a remote node), Sencho sends a single **Lost connection to Docker daemon** warning and pauses crash detection on the affected node. When the connection is restored, Sencho reconciles container state against a pre-disconnect snapshot:
+`info`/`image_update_applied`: `Auto-update: stack "" updated with new images`. If a block-on-deploy policy gates the auto-update: `warning`/`scan_finding` `Policy "" blocked auto-update: N image(s) exceed `. See [Auto-update policies](/features/auto-update-policies).
-- If most of your containers are still running, individual gap exits are classified and alerts are sent as normal.
-- If a large share of containers exited during the outage (for example after a daemon restart), Sencho consolidates them into a single **Docker daemon interruption detected** informational notification instead of paging you for every container.
+### Auto-heal
-A matching **Reconnected to Docker daemon** info notification confirms monitoring has resumed.
+- `info`/`autoheal_triggered`: `Auto-Heal: Restarted on stack after being unhealthy for minute(s).`
+- `warning`/`autoheal_triggered`: `Auto-Heal: Failed to restart on stack . Error: `
+- `warning`/`autoheal_triggered`: `Auto-Heal: Policy for [/] has been auto-disabled after consecutive failures.`
-### High-churn events
+See [Auto-heal policies](/features/auto-heal-policies).
-When many crashes land in a short window (for example, a large stack coming down unexpectedly), Sencho dispatches an initial batch and summarizes the remainder in a single **N additional containers crashed in the last minute** entry so your notification channels are not flooded.
+### Vulnerability scanning
+
+- **Per-violation, during a scheduled scan** is `warning`/`scan_finding`. One alert per violation: `Policy "" violated by : exceeds `.
+- **Scan completion** is `info`/`scan_finding` (or `warning` if any image failed): `Scheduled scan "" completed: