mirror of
https://github.com/Studio-Saelix/sencho.git
synced 2026-07-26 11:49:16 +00:00
feat: add Apprise as a fourth notification channel (#1644)
* feat: add Apprise as a fourth notification channel Support keyed and stateless Apprise endpoints with secret-safe public DTOs, fail-closed malformed config, and mode-specific Settings UI. Docs and screenshots updated for four-channel Channels and routing. * fix: harden Apprise secrets at rest and preserve-on-write saves Encrypt Apprise endpoint and config with CryptoService so a downgrade cannot leak via SELECT *. Align channel and routing saves so blank destination fields omit config on same-mode URL edits, enforce keyed notify IDs, and keep secrets_redacted truthful. * fix: harden Apprise route type changes and mixed-version config UI Require a raw channel_url when switching notification-route types so ciphertext cannot strand under Discord/Slack/webhook. Default missing remote apprise status, replace Channels state on node switch, and exercise the production config-column migrator. * fix: tolerate stub fleet configuration payloads without agents Normalize remote Apprise agent status only when notifications.agents is present so successful Pilot/stub fetches stay online instead of throwing into the offline catch path. * fix: correct TypeScript in configuration normalize tests * fix: ignore stale Channels agent bodies after node switch Compare the active node after response JSON parsing so a slow body cannot overwrite the newly selected node's channel state. * fix: isolate corrupt Apprise crypto and keep keyed Tags visible Decrypt failures on one Apprise row no longer 500 agent/route lists or suppress sibling channel dispatch. Treat public /notify/<redacted> as keyed so Tags remain editable after reload.
This commit is contained in:
Anso
@@ -1,23 +1,23 @@
|
||||
---
|
||||
title: Alerts & Notifications
|
||||
sidebarTitle: Alerts and notifications
|
||||
description: Threshold and event alerts for your fleet, dispatched to Discord, Slack, or any webhook, with per-stack rules and channel routing.
|
||||
description: Threshold and event alerts for your fleet, dispatched to Discord, Slack, Apprise, or any webhook, with per-stack rules and channel routing.
|
||||
---
|
||||
|
||||
Sencho watches each node it manages for container crashes, host pressure, scheduled-task results, and update availability, then surfaces every signal in two places: the in-app notification bell at the top of the shell and one of three external channels you configure. This page covers everything from configuring channels to writing per-stack threshold rules, routing alerts to dedicated channels with routing rules, and tuning retention.
|
||||
Sencho watches each node it manages for container crashes, host pressure, scheduled-task results, and update availability, then surfaces every signal in two places: the in-app notification bell at the top of the shell and an external channel you configure. This page covers everything from configuring channels to writing per-stack threshold rules, routing alerts to dedicated channels with routing rules, and tuning retention.
|
||||
|
||||
<Frame>
|
||||
<img src="/images/alerts-notifications/notifications-settings.png" alt="Settings · Notifications · Channels panel showing the Discord, Slack, and Webhook tabs with the masthead breadcrumb, the CHANNELS 3/3 stat, the active Discord tab with its Enabled toggle on, the Webhook URL input, and the Test and Save actions." />
|
||||
<img src="/images/alerts-notifications/notifications-settings.png" alt="Settings · Notifications · Channels panel with Discord, Slack, Webhook, and Apprise tabs, CHANNELS 1/4 in the masthead, the Apprise tab active with Enabled on, a redacted Apprise endpoint, an empty Tags field, and Test disabled beside Save." />
|
||||
</Frame>
|
||||
|
||||
## Notification channels
|
||||
|
||||
Open **Settings · Notifications · Channels** to configure the three channel types. Each channel is per-node, so switching the active node via the node picker reloads the panel against that node's stored settings. The masthead carries a `CHANNELS` stat showing how many of the three slots are enabled.
|
||||
Open **Settings · Notifications · Channels** to configure Discord, Slack, custom webhook, and Apprise channels. Each channel is per-node, so switching the active node via the node picker reloads the panel against that node's stored settings. The masthead carries a `CHANNELS` stat showing how many of the four slots are enabled.
|
||||
|
||||
Each tab carries the same controls: an **Enabled** toggle (helper: `Send Sencho events to this <name> channel.`), a **Webhook URL** input (placeholder `https://...`, helper: `Sencho posts JSON payloads here. Use a private channel.`), and the **Test** and **Save** buttons. The kicker on each tab toggles between `enabled` and `off` so you can see at a glance which slots are wired up.
|
||||
Each Discord, Slack, and Webhook tab carries an **Enabled** toggle, a **Webhook URL** input (HTTPS only), and **Test** / **Save**. The Apprise tab uses an **Apprise endpoint** instead: keyed `/notify/{key}` shows optional **Tags**; stateless `/notify` shows **Destination URLs**. The kicker on each tab toggles between `enabled` and `off` so you can see at a glance which slots are wired up.
|
||||
|
||||
<Note>
|
||||
Webhook URLs must use HTTPS. Sencho rejects `http://` and any string that does not parse as a URL. The same rule applies to test sends and to routing-rule URLs.
|
||||
Discord, Slack, and webhook URLs must use HTTPS. Apprise endpoints may use HTTP or HTTPS. Every endpoint must parse as a URL.
|
||||
</Note>
|
||||
|
||||
### Discord
|
||||
@@ -43,6 +43,12 @@ Use the Generic Webhook tab when you have your own receiver: a Mattermost or Tea
|
||||
|
||||
`level` is one of `info`, `warning`, `error`. `source` is always the literal string `sencho`. `timestamp` is ISO-8601 with millisecond precision.
|
||||
|
||||
### Apprise
|
||||
|
||||
Use a keyed Apprise endpoint ending in `/notify/<key>` with optional tags, or a stateless endpoint ending in `/notify` with one or more destination URLs. Separate stateless destination URLs with commas or whitespace. Apprise endpoints may use HTTP for a local gateway.
|
||||
|
||||
Sencho sends the alert title, body, severity, and either tags or destination URLs to Apprise. An HTTP 204 response means Apprise accepted no delivery and is reported as a failed test or dispatch. Configuration reads expose only destination provider names and counts, never destination URLs or endpoint keys.
|
||||
|
||||
### Test sends and delivery semantics
|
||||
|
||||
The **Test** button on each tab dispatches the literal message `🔌 Test Notification from Sencho!` at level `info` through the same path a real alert would take. Test sends require the admin role; the server returns 403 if an operator or viewer submits one.
|
||||
@@ -74,7 +80,7 @@ A route with all four matchers left empty matches every alert and intercepts glo
|
||||
Open **Settings · Notifications · Notification Routing** and click **+ Add Route**.
|
||||
|
||||
<Frame>
|
||||
<img src="/images/alerts-notifications/routing-modal.png" alt="The 'New routing rule' modal with the 'ROUTING · NEW RULE' kicker, a Name input filled with 'Production alerts', a Node scope select reading 'Any node', empty Stacks, Labels, and Categories combobox pickers, the helper line 'Leave blank to match all categories. All non-empty filters must match (AND).', a Channel tab strip with Discord selected and a webhook URL filled in, and Priority and Enabled fields with Cancel and CREATE actions in the footer." />
|
||||
<img src="/images/alerts-notifications/routing-modal.png" alt="The New routing rule modal with empty Name, Node scope set to Any node, empty Stacks, Labels, and Categories fields, Channel tabs with Apprise selected, and Cancel and CREATE actions. Priority and Enabled may be scrolled out of the cropped frame." />
|
||||
</Frame>
|
||||
|
||||
| Field | Purpose |
|
||||
@@ -84,7 +90,7 @@ Open **Settings · Notifications · Notification Routing** and click **+ Add Rou
|
||||
| **Stacks** *(optional)* | A combobox of stacks on the active node. Selected stacks appear as removable pills. Empty matches any stack. |
|
||||
| **Labels** *(optional)* | A combobox of stack labels on the active node. Empty matches any label. |
|
||||
| **Categories** *(optional)* | A combobox of notification categories. The helper line reads `Leave blank to match all categories. All non-empty filters must match (AND).` |
|
||||
| **Channel** | Tabs for Discord, Slack, and Webhook with a URL input below. URL must use HTTPS. |
|
||||
| **Channel** | Tabs for Discord, Slack, Webhook, and Apprise. Discord, Slack, and Webhook require HTTPS. Apprise accepts HTTP or HTTPS. |
|
||||
| **Priority** | A number used to sort the rule list. Lower numbers appear higher up. Priority does not gate dispatch: when multiple rules match the same alert, every matching rule fires concurrently. |
|
||||
| **Enabled** | Toggle the rule on or off without deleting it. |
|
||||
|
||||
@@ -164,7 +170,7 @@ Each stack carries its own set of threshold rules that fire when a metric stays
|
||||
Open the rules editor by right-clicking a stack in the sidebar and choosing **Alerts**, or by pressing `A` while focused on a stack.
|
||||
|
||||
<Frame>
|
||||
<img src="/images/alerts-notifications/alert-panel.png" alt="The Stack › PLEX › MONITOR sheet with the Alerts tab active, a green 'Notifications active via Discord, Slack, Webhook' banner under the NOTIFICATION CHANNELS heading, an ACTIVE RULES section listing 'CPU Usage (%) > 80' with the secondary line 'Trigger after 5m • Cooldown 60m', and an ADD NEW RULE form with Metric, Operator, Threshold, Duration, Cooldown fields and an Add Rule submit button." />
|
||||
<img src="/images/alerts-notifications/alert-panel.png" alt="The Stack › PLEX › MONITOR sheet with the Alerts tab active, a green notification-channel banner under the NOTIFICATION CHANNELS heading, an ACTIVE RULES section listing 'CPU Usage (%) > 80' with the secondary line 'Trigger after 5m • Cooldown 60m', and an ADD NEW RULE form with Metric, Operator, Threshold, Duration, Cooldown fields and an Add Rule submit button." />
|
||||
</Frame>
|
||||
|
||||
### Alert fields
|
||||
@@ -196,7 +202,7 @@ The **NOTIFICATION CHANNELS** banner above the rules list reflects what dispatch
|
||||
|
||||
- **Loading** is a spinner with `Checking notification channels...` while Sencho asks the target node for its agent state.
|
||||
- **Remote node** is a blue banner reading `Remote node: <name>`, with the body `Alert rules are stored and evaluated on this remote instance. Notifications are dispatched using that node's configured channels.` A sub-line reports whether the remote has any channels configured.
|
||||
- **No channels** is an amber banner reading `No notification channels configured`, body `Alert rules will be saved and evaluated, but no notifications will be dispatched. Configure Discord, Slack, or a webhook in Settings → Notifications → Channels.`
|
||||
- **No channels** is an amber banner reading `No notification channels configured`, body `Alert rules will be saved and evaluated, but no notifications will be dispatched. Configure Discord, Slack, Apprise, or a webhook in Settings → Notifications → Channels.`
|
||||
- **Active** is a green banner reading `Notifications active via Discord, Slack, …` with the configured channels listed.
|
||||
|
||||
<Frame>
|
||||
@@ -287,7 +293,7 @@ The bell aggregates across the entire fleet by hitting `/api/notifications` agai
|
||||
|
||||
Per-stack alert rules and channel configuration are **stored on the node where the stack runs**. To configure a rule on a remote, switch the active node via the picker, open the stack's **Monitor** sheet, and the form's POST is forwarded to the remote.
|
||||
|
||||
Crash detection runs only on local Docker; remote nodes run their own copy of `DockerEventService` and emit through the proxy. On a multi-node fleet the bell attributes remotes with a node-name badge rather than embedding the satellite-local name in the message. External channel payloads (Slack, Discord, custom webhook) receive the same sanitized body with no structured source-node id.
|
||||
Crash detection runs only on local Docker; remote nodes run their own copy of `DockerEventService` and emit through the proxy. On a multi-node fleet the bell attributes remotes with a node-name badge rather than embedding the satellite-local name in the message. External channel payloads (Slack, Discord, Apprise, custom webhook) receive the same sanitized body with no structured source-node id.
|
||||
|
||||
Switching the active node only affects per-stack rule editing and the Settings panels. The bell aggregates every node regardless of which one is active.
|
||||
|
||||
@@ -433,8 +439,8 @@ Switching the active node tears down per-stack rule editors and reloads channel
|
||||
## Troubleshooting
|
||||
|
||||
<AccordionGroup>
|
||||
<Accordion title="Notifications never arrive in Discord, Slack, or my webhook">
|
||||
Check three things in order. First, the channel toggle in **Settings · Notifications · Channels** must be on; the kicker on each tab reads `enabled` or `off`. Second, the URL must use HTTPS; the form rejects plain `http://` outright. Third, a routing rule with empty `Stacks`, `Labels`, and `Categories` matchers will intercept every alert and skip the global channels. Use the per-channel **Test** button to issue a one-shot dispatch and watch your endpoint for the literal message `🔌 Test Notification from Sencho!` Sencho records the failure reason in `notification_history.dispatch_error` when delivery throws, so a row that appears in the bell with no follow-up at the endpoint usually means a 4xx or timeout at the receiver.
|
||||
<Accordion title="Notifications never arrive">
|
||||
Check three things in order. First, the channel toggle in **Settings · Notifications · Channels** must be on; the kicker on each tab reads `enabled` or `off`. Second, Discord, Slack, and webhook URLs must use HTTPS (the form rejects plain `http://` for those channels); Apprise endpoints may use HTTP or HTTPS. Third, a routing rule with empty `Stacks`, `Labels`, and `Categories` matchers will intercept every alert and skip the global channels. Use the per-channel **Test** button to issue a one-shot dispatch and watch your endpoint for the literal message `🔌 Test Notification from Sencho!` Sencho records the failure reason in `notification_history.dispatch_error` when delivery throws, so a row that appears in the bell with no follow-up at the endpoint usually means a 4xx or timeout at the receiver.
|
||||
</Accordion>
|
||||
<Accordion title="An alert rule never fires even when the threshold is breached">
|
||||
Three causes account for almost every case. First, the rule's **Duration** has not elapsed yet: the breach must persist for the full duration before the rule fires. Second, the rule is still in cooldown after a previous fire. Third, the panel's banner is not green: a remote-node banner means the rule was saved on a remote whose channels you may not have configured, and an amber `No notification channels configured` banner means the rule evaluates fine but Sencho has nowhere to send the alert. The evaluator runs on a 30-second tick, so expect up to 30 seconds of latency between the breach starting and the timer engaging.
|
||||
|
||||
@@ -87,7 +87,7 @@ The card is divided into four sections.
|
||||
|
||||
| Row | What it shows |
|
||||
|-----|---------------|
|
||||
| **Notification agents** | The list of enabled delivery agents from `Discord`, `Slack`, `Webhook`, joined by commas; reads `None` when no agent is enabled |
|
||||
| **Notification agents** | The list of enabled delivery agents from `Discord`, `Slack`, `Webhook`, and `Apprise`, joined by commas; reads `None` when no agent is enabled |
|
||||
| **Alert rules** | Total per-stack alert rules in effect, formatted `<n> rules` |
|
||||
| **Notification routing** | Number of enabled routing rules that direct categories to specific agents, formatted `<n> routes` |
|
||||
|
||||
|
||||
@@ -36,12 +36,12 @@ See [the pricing page](https://sencho.io/pricing) for current pricing.
|
||||
- Atomic deployments with automatic rollback, and one-click rollback to the previous deployment
|
||||
- Auto-update policies for stack images and auto-heal policies for failed containers
|
||||
- Scheduled operations across the full action catalog (lifecycle, updates, scans, snapshots, prune)
|
||||
- Webhooks (incoming, to trigger deploys from CI/CD) and notification routing (per-stack and per-category rules to Discord, Slack, or any webhook)
|
||||
- Webhooks (incoming, to trigger deploys from CI/CD) and notification routing (per-stack and per-category rules to Discord, Slack, Apprise, or any webhook)
|
||||
- Custom S3-compatible backup target (bring your own AWS S3, Cloudflare R2, MinIO, Backblaze B2, or Wasabi bucket)
|
||||
- Vulnerability scanning: install, update, uninstall, and auto-update the managed Trivy binary, on-demand scans for vulnerabilities, secrets, and misconfigurations, scan comparison, CVE suppressions, and single-scan SBOM export (SPDX, CycloneDX)
|
||||
- Private registry credentials for Docker Hub, GitHub Container Registry (GHCR), and custom or self-hosted registries (admin role required)
|
||||
- A 14-day recent-activity audit log with Stream and Table views and filtering
|
||||
- Alert rules with Discord, Slack, and webhook targets
|
||||
- Alert rules with Discord, Slack, Apprise, and webhook targets
|
||||
- API tokens for CI/CD pipelines and scripts (admin role required)
|
||||
- Unlimited accounts with the Admin and Viewer roles
|
||||
- Two-factor authentication (TOTP plus backup codes)
|
||||
|
||||
@@ -189,7 +189,7 @@ When you select a remote node in the switcher, the Settings hub filters to the p
|
||||
|-------|:-----:|-------|
|
||||
| Appearance | Per browser | Theme and density preferences are stored in your browser, not on the node. |
|
||||
| Host Alerts | Per node | Host CPU, RAM, and disk thresholds for the selected node. |
|
||||
| Channels | Per node | Discord, Slack, and Webhook channels fire from the node that detects the event. |
|
||||
| Channels | Per node | Discord, Slack, Webhook, and Apprise channels fire from the node that detects the event. |
|
||||
| Labels | Per node | Stack and container label palettes. |
|
||||
| Vulnerability Scanning | Per node | Trivy install state and scanner readiness for the selected node. |
|
||||
| Developer | Per node | Retention windows for metrics and logs, plus Developer Mode. |
|
||||
|
||||
@@ -147,7 +147,7 @@ The **Logs** view aggregates output from every container across every stack into
|
||||
|
||||
### Alerts and notifications
|
||||
|
||||
Configure threshold-based alerts per stack and route notifications to Discord, Slack, or any generic webhook. Per-stack [routing rules](/features/alerts-notifications#notification-routing) send each stack's alerts to the right channel. [Learn more →](/features/alerts-notifications)
|
||||
Configure threshold-based alerts per stack and route notifications to Discord, Slack, Apprise, or any generic webhook. Per-stack [routing rules](/features/alerts-notifications#notification-routing) send each stack's alerts to the right channel. [Learn more →](/features/alerts-notifications)
|
||||
|
||||
### Audit log
|
||||
|
||||
|
||||
@@ -216,7 +216,7 @@ Use the **Enabled** toggle in the row to pause a schedule without losing its con
|
||||
|
||||
## Failure notifications
|
||||
|
||||
When a scheduled task fails, Sencho dispatches an `error`-level notification in the `system` category through your configured channels (Discord, Slack, custom webhooks, and the in-app bell). The notification carries the task name, the action, and the error message so you can diagnose without opening the run history.
|
||||
When a scheduled task fails, Sencho dispatches an `error`-level notification in the `system` category through your configured channels (Discord, Slack, Apprise, custom webhooks, and the in-app bell). The notification carries the task name, the action, and the error message so you can diagnose without opening the run history.
|
||||
|
||||
When a task that previously failed succeeds again, Sencho dispatches an `info`-level recovery notification in the same category to confirm the issue is resolved. Tasks that succeed on every run do not produce per-run notifications: the recovery-only approach keeps the channel quiet during steady-state operation.
|
||||
|
||||
|
||||
Binary file not shown.
|
Before Width: | Height: | Size: 112 KiB After Width: | Height: | Size: 103 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 102 KiB After Width: | Height: | Size: 51 KiB |
@@ -501,7 +501,7 @@ docker compose pull && docker compose up -d
|
||||
|
||||
## Notification routing rule matches but alerts still go to global channels
|
||||
|
||||
**Symptom:** You created a routing rule for a stack, but alerts from that stack still arrive on the global Discord/Slack/Webhook channels.
|
||||
**Symptom:** You created a routing rule for a stack, but alerts from that stack still arrive on the global channels.
|
||||
|
||||
**Checks:**
|
||||
|
||||
|
||||
@@ -378,7 +378,7 @@ Quick reference:
|
||||
|
||||
**Scope:** Per-node (each node has its own notification agents; remote nodes dispatch alerts through their own channels)
|
||||
|
||||
Configure external destinations for alert notifications. Three agent types are available on separate tabs: **Discord**, **Slack**, and **Webhook**. The masthead publishes a **CHANNELS** pill showing how many agents are enabled (for example, `2/3`).
|
||||
Configure external destinations for alert notifications. Four agent types are available on separate tabs: **Discord**, **Slack**, **Webhook**, and **Apprise**. The masthead publishes a **CHANNELS** pill showing how many agents are enabled (for example, `2/4`).
|
||||
|
||||
For each agent:
|
||||
|
||||
@@ -386,6 +386,7 @@ For each agent:
|
||||
|-------|-------------|
|
||||
| **Enabled** toggle | Activates or deactivates this agent. Disabled agents receive no messages even if a URL is saved. |
|
||||
| **Webhook URL** | The endpoint Sencho will POST to when an alert fires. |
|
||||
| **Apprise** | Use a keyed `/notify/<key>` endpoint with optional tags, or a stateless `/notify` endpoint with destination URLs. Apprise accepts HTTP or HTTPS. |
|
||||
|
||||
Click **Save** to persist changes. Click **Test** to send a test payload immediately and verify delivery.
|
||||
|
||||
@@ -410,9 +411,9 @@ Create routing rules that direct specific alert types to specific notification c
|
||||
| **Stack patterns** | Glob patterns to match stack names (multi-select). |
|
||||
| **Labels** | Apply this rule to stacks that carry any of the selected labels. |
|
||||
| **Categories** | Event categories that trigger the rule (crash, deploy, vulnerability, etc.). |
|
||||
| **Channel type** | `discord`, `slack`, or `webhook`. |
|
||||
| **Channel type** | `discord`, `slack`, `webhook`, or `apprise`. |
|
||||
| **Channel URL** | The destination endpoint for this rule. |
|
||||
| **Priority** | Order in which rules are evaluated; the first match wins. |
|
||||
| **Priority** | Sort order among matching rules. Every matching route fires; priority does not stop later matches. |
|
||||
| **Enabled** toggle | Mute a rule without deleting it. |
|
||||
|
||||
Each rule keeps an execution history (collapsible per row) showing which alerts triggered it, when, and whether the delivery succeeded.
|
||||
|
||||
Reference in New Issue
Block a user