mirror of
https://github.com/Studio-Saelix/sencho.git
synced 2026-07-26 11:49:16 +00:00
feat: split Host Alerts into Host Alerts, Container Alerts, and Stacks guardrails (#1461)
* feat: split Host Alerts into Host Alerts, Container Alerts, and Stacks guardrails Move global_crash from Host Alerts to new Monitoring > Container Alerts section. Move health gate and env deploy guardrails from Host Alerts to Infrastructure > Stacks > Deploy Guardrails subsection. Host Alerts now contains only host threshold settings (CPU, RAM, disk, alert suppression, and the master host_alerts_enabled toggle). Stacks gains a Deploy Guardrails subsection (node-scoped, admin-gated) alongside the existing Workflow controls (browser-local). Dashboard Crash detection row now routes to Container Alerts. * docs: update crash detection toggle description to match new Container Alerts section
This commit is contained in:
Anso
@@ -367,15 +367,15 @@ A separate rate limit applies to crash and health alerts only: 20 emits per 60-s
|
||||
|
||||
## Crash detection toggle
|
||||
|
||||
The global crash-capture switch lives under **Settings · Monitoring · Host Alerts**.
|
||||
The global crash-capture switch lives under **Settings · Monitoring · Container Alerts**.
|
||||
|
||||
<Frame>
|
||||
<img src="/images/alerts-notifications/system-crash-toggle.png" alt="Settings · Monitoring · Host Alerts panel showing a HOST THRESHOLDS section with CPU limit, RAM limit, and Disk limit, then a CRASH CAPTURE section with a Global crash capture toggle in the ON state with the helper line 'Watch every managed container for unexpected exits.'" />
|
||||
<img src="/images/alerts-notifications/system-crash-toggle.png" alt="Settings · Monitoring · Container Alerts panel showing a Container crash & health alerts toggle in the ON state with the helper line about unexpected exits, OOM kills, and healthcheck failures." />
|
||||
</Frame>
|
||||
|
||||
The **Global crash capture** toggle controls whether `DockerEventService` raises crash, OOM, and healthcheck alerts on the active node. Helper text: `Watch every managed container for unexpected exits.` Defaults to on; if the database read fails, Sencho falls back to default-deny so the system never leaks alerts you cannot turn off.
|
||||
The **Container crash & health alerts** toggle controls whether `DockerEventService` raises crash, OOM, and healthcheck alerts on the active node. Helper text: `Send alerts for unexpected container exits, OOM kills, and Docker healthcheck failures. Auto-Heal can still observe crash signals independently.` Defaults to on; if the database read fails, Sencho falls back to default-deny so the system never leaks alerts you cannot turn off.
|
||||
|
||||
The **Host Alerts** panel also carries the **Host thresholds** rows (CPU limit, RAM limit, Disk limit, all expressed as percent) that drive the host-level monitor warnings. The **Janitor threshold** (in GiB) that drives the unused-Docker-data alert lives in the **Docker & Storage** panel.
|
||||
The **Host Alerts** panel carries the **Host thresholds** rows (CPU limit, RAM limit, Disk limit, all expressed as percent) that drive the host-level monitor warnings. Container crash and health alerts live in the **Container Alerts** panel. The **Janitor threshold** (in GiB) that drives the unused-Docker-data alert lives in the **Docker & Storage** panel.
|
||||
|
||||
## Refresh cadence
|
||||
|
||||
@@ -410,7 +410,7 @@ Switching the active node tears down per-stack rule editors and reloads channel
|
||||
When the daemon disconnects and reconnects, Sencho snapshots every container at the moment of reconnect. If at least 20% exited during the gap, the service emits a single `info`/`system` summary `Docker daemon interruption detected: N containers exited during connection gap.` instead of one crash alert per container. Below the threshold, every gap exit is classified individually.
|
||||
</Accordion>
|
||||
<Accordion title="Crash alerts stopped arriving and nothing else looks wrong">
|
||||
Check **Settings · Monitoring · Host Alerts · Global crash capture**. The toggle is the master switch for the Docker event service. The panel cache reads the database every 500 ms; if the database read errors, Sencho defaults the toggle to off so the failure mode is silent rather than spammy. Repair the toggle, save, and the next Docker event reaches the dispatcher.
|
||||
Check **Settings · Monitoring · Container Alerts · Container crash & health alerts**. The toggle is the master switch for the Docker event service. The panel cache reads the database every 500 ms; if the database read errors, Sencho defaults the toggle to off so the failure mode is silent rather than spammy. Repair the toggle, save, and the next Docker event reaches the dispatcher.
|
||||
</Accordion>
|
||||
<Accordion title="A burst of crashes happened but I only see about twenty alerts">
|
||||
Sencho rate-limits crash and health dispatches to 20 emits per rolling 60-second window per node, then emits a single `warning`/`monitor_alert` roll-up `N additional containers crashed in the last minute.` once the window closes. Every alert is still persisted to `notification_history` and visible in the bell up to the 100-row per-node cap; only the channel fanout is throttled.
|
||||
|
||||
@@ -46,7 +46,7 @@ When a service declares an `env_file:` that does not exist in the stack director
|
||||
|
||||
A `${VAR:?message}` reference tells Compose the variable is required: the deploy fails if it is unset or empty. By default Sencho surfaces that as an advisory finding and lets Compose report it at deploy time.
|
||||
|
||||
If you would rather fail fast with a clear message before anything runs, turn on **Block deploy on missing required env vars** under **Settings → Host alerts → Deploy guardrails**. With it on, a deploy or update is refused up front when a required variable is unset or empty, before any backup, image pull, or container change happens. It is off by default, applies to the node you set it on, and requires an admin to change.
|
||||
If you would rather fail fast with a clear message before anything runs, turn on **Block deploy on missing required env vars** under **Settings → Infrastructure → Stacks → Deploy Guardrails**. With it on, a deploy or update is refused up front when a required variable is unset or empty, before any backup, image pull, or container change happens. It is off by default, applies to the node you set it on, and requires an admin to change.
|
||||
|
||||
## Opening the inventory
|
||||
|
||||
|
||||
@@ -67,13 +67,13 @@ The gate also runs for updates you did not click: scheduled image updates, webho
|
||||
|
||||
### Tuning or disabling the gate
|
||||
|
||||
Open **Settings > Host Alerts > Update health gate** on the node you want to configure:
|
||||
Open **Settings > Infrastructure > Stacks > Deploy Guardrails** on the node you want to configure:
|
||||
|
||||
- **Observe health after updates** turns the gate on or off for that node. On by default; turning it off only stops the observation and its timeline events, never the update itself.
|
||||
- **Observation window** sets how long containers are watched, from 15 to 600 seconds. The default is 90 seconds; raise it for stacks that take a while to settle, since a healthcheck still starting at the end of the window records an unknown verdict instead of a pass.
|
||||
|
||||
<Frame>
|
||||
<img src="/images/health-gated-updates/settings.png" alt="Host Alerts settings page with the Update health gate group, showing the Observe health after updates toggle and the Observation window field" />
|
||||
<img src="/images/health-gated-updates/settings.png" alt="Stacks settings page with the Deploy Guardrails subsection, showing the Observe health after updates toggle and the Observation window field" />
|
||||
</Frame>
|
||||
|
||||
## Rollback readiness
|
||||
@@ -102,7 +102,7 @@ When a deploy or update fails, Sencho classifies the failure from the compose ou
|
||||
Unknown means one of the verdict-affecting signals could not be verified, most often because Docker or the node did not answer in time. The update path is never blocked by an unknown verdict; check the node's connection if it persists, and proceed when you are confident in the stack's state.
|
||||
</Accordion>
|
||||
<Accordion title="The health gate failed but my app seems fine">
|
||||
The gate fails on the first clear problem it observes: a container exit, an unhealthy healthcheck, a restart loop, or a container disappearing mid-window. Check the named container's logs from the stack page. If the container legitimately restarts during startup (a migration step, for example), raise the observation window under **Settings > Host Alerts > Update health gate** so the gate watches past the settle period.
|
||||
The gate fails on the first clear problem it observes: a container exit, an unhealthy healthcheck, a restart loop, or a container disappearing mid-window. Check the named container's logs from the stack page. If the container legitimately restarts during startup (a migration step, for example), raise the observation window under **Settings > Infrastructure > Stacks > Deploy Guardrails** so the gate watches past the settle period.
|
||||
</Accordion>
|
||||
<Accordion title="The health gate result is unknown">
|
||||
Unknown means the observation could not finish: Sencho restarted mid-window, a newer update superseded the observation, Docker became unreachable, a healthcheck was still starting when the window ended, or no containers appeared to observe. The stack timeline records the reason. Check the containers directly; an unknown verdict makes no claim either way.
|
||||
|
||||
@@ -185,7 +185,7 @@ When you select a remote node in the switcher, the Settings hub filters to the p
|
||||
| Panel | Scope | Notes |
|
||||
|-------|:-----:|-------|
|
||||
| Appearance | Per browser | Theme and density preferences are stored in your browser, not on the node. |
|
||||
| Host Alerts | Per node | Host CPU, RAM, disk, and crash thresholds for the selected node. |
|
||||
| Host Alerts | Per node | Host CPU, RAM, and disk thresholds for the selected node. |
|
||||
| Channels | Per node | Discord, Slack, and Webhook channels fire from the node that detects the event. |
|
||||
| Labels | Per node | Stack and container label palettes. |
|
||||
| Vulnerability Scanning | Per node | Trivy install state and scanner readiness for the selected node. |
|
||||
|
||||
+35
-13
@@ -16,7 +16,7 @@ Open the Settings Hub by clicking the **Profile** icon in the top bar and select
|
||||
| **Personal** | Account, Appearance |
|
||||
| **Access** | License, Users, SSO, API Tokens |
|
||||
| **Infrastructure** | Nodes, Fleet Mesh, Registries, Cloud Backup, App Store, Stacks |
|
||||
| **Monitoring** | Host Alerts, Docker & Storage |
|
||||
| **Monitoring** | Host Alerts, Container Alerts, Docker & Storage |
|
||||
| **Notifications** | Channels, Notification Routing |
|
||||
| **Automation** | Webhooks |
|
||||
| **Organization** | Labels |
|
||||
@@ -33,7 +33,7 @@ Every section renders inside the same masthead-and-sidebar layout. The masthead
|
||||
|
||||
| Pill | Meaning |
|
||||
|------|---------|
|
||||
| **SCOPE** `operator` / `browser` / `global` | Setting applies to your account (`operator`), to this browser only (`browser`, for browser-local sections such as Appearance and Stacks), or to the whole instance (`global`, every other non-node group) |
|
||||
| **SCOPE** `operator` / `browser` / `global` | Setting applies to your account (`operator`), to this browser only (`browser`, for browser-local sections such as Appearance), or to the whole instance (`global`, every other non-node group) |
|
||||
| **NODE** `<node name>` | Setting is per-node and is currently being edited against this node |
|
||||
| **EDITED** `<count>` pending / `saved` | The current section has unsaved changes |
|
||||
| Section-specific stats | Each section can publish its own pills: `2FA on`/`off` and `BACKUP <n> left` (Account); `PLAN`, `TRIAL <n>d left`, `RENEWS`, `STATUS` (License); `OPERATORS` (Users); `CHANNELS` (Channels); `ROUTES` (Notification Routing); `WEBHOOKS` and `ENABLED` (Webhooks); `LABELS` (Labels); `PROVIDER`, `USED`, `SNAPSHOTS` (Cloud Backup); `DEV MODE` (Developer Diagnostics) |
|
||||
@@ -225,14 +225,12 @@ See [API Tokens](/features/api-tokens) for the full walkthrough.
|
||||
|
||||
**Scope:** Per-node (applies to the currently selected node)
|
||||
|
||||
Configure the host resource thresholds that trigger warnings, the suppression cadence for repeated alerts, and whether unexpected container exits raise a crash alert. These are alerting thresholds, not enforcement: Sencho warns when a metric crosses a line, it never throttles or kills containers. The masthead **NODE** pill names which node you are configuring; the **EDITED** pill counts unsaved changes.
|
||||
Configure the host resource thresholds that trigger warnings and the suppression cadence for repeated alerts. These are alerting thresholds, not enforcement: Sencho warns when a metric crosses a line, it never throttles or kills containers. The masthead **NODE** pill names which node you are configuring; the **EDITED** pill counts unsaved changes.
|
||||
|
||||
<Frame>
|
||||
<img src="/images/settings/settings-host-alerts.png" alt="Host Alerts section showing the Host thresholds and Crash capture subsections" />
|
||||
<img src="/images/settings/settings-host-alerts.png" alt="Host Alerts section showing the Host thresholds subsection with the master toggle and CPU, RAM, Disk, and Alert suppression controls" />
|
||||
</Frame>
|
||||
|
||||
### Host thresholds
|
||||
|
||||
| Setting | Default | Description |
|
||||
|---------|---------|-------------|
|
||||
| **Host threshold alerts** | On | Master switch for CPU, RAM, and disk threshold alerts only. When OFF, no host threshold checks run and the controls below are inactive. |
|
||||
@@ -241,13 +239,21 @@ Configure the host resource thresholds that trigger warnings, the suppression ca
|
||||
| **Disk limit** | 90% | Low free space slows image pulls and backups. |
|
||||
| **Alert suppression** | 60 min | How long to wait before resending a host alert while the metric stays over threshold. The follow-up message includes a count of suppressed cycles. Range 1 to 1,440 minutes. |
|
||||
|
||||
### Crash capture
|
||||
Click **Save alerts** to apply.
|
||||
|
||||
---
|
||||
|
||||
## Container Alerts
|
||||
|
||||
**Scope:** Per-node (applies to the currently selected node)
|
||||
|
||||
Controls whether Sencho watches every managed container on this node for unexpected exits, OOM kills, and Docker healthcheck failures, and dispatches alerts for each. Auto-Heal observes crash signals independently regardless of this toggle, so turning it off stops alerts but does not stop Auto-Heal.
|
||||
|
||||
| Setting | Default | Description |
|
||||
|---------|---------|-------------|
|
||||
| **Global crash capture** | On | Watch every managed container for unexpected exits. When on, Sencho sends a notification whenever a container exits with a non-zero status outside a planned stop. |
|
||||
| **Container crash & health alerts** | On | Send alerts for unexpected container exits, OOM kills, and Docker healthcheck failures. Auto-Heal can still observe crash signals independently. |
|
||||
|
||||
Click **Save alerts** to apply.
|
||||
Click **Save settings** to apply.
|
||||
|
||||
---
|
||||
|
||||
@@ -487,11 +493,13 @@ See [App Store](/features/app-store#custom-template-registry) for more on custom
|
||||
|
||||
## Stacks
|
||||
|
||||
Stack editor, lifecycle workflow preferences, and deploy guardrails. The Workflow controls are browser-local (each browser remembers its own choices). The Deploy Guardrails are node-scoped backend settings. The masthead shows the **NODE** pill for this section.
|
||||
|
||||
### Workflow
|
||||
|
||||
**Scope:** This browser (preferences are saved to local storage)
|
||||
|
||||
Stack editor and lifecycle workflow preferences. Each browser remembers its own choices, so they follow you across tabs but never change what other operators see. The masthead reads `SCOPE browser`.
|
||||
|
||||
### Deploy progress
|
||||
#### Deploy progress
|
||||
|
||||
Sencho streams live output whenever you deploy, restart, update, install, or run a Git operation. It is on by default; turn it off to run operations without it. When it is on, **Progress style** chooses how it appears: **Modal** (a centered overlay that closes automatically on success or stays open on failure) or **Inline** (a quiet status band on the stack detail). See [Deploy Progress](/features/deploy-progress) for the full reference.
|
||||
|
||||
@@ -500,7 +508,7 @@ Sencho streams live output whenever you deploy, restart, update, install, or run
|
||||
| **Enabled** (default) | A progress surface (the Modal overlay or the Inline band, per Progress style) shows for every long-running operation |
|
||||
| **Disabled** | Operations run without it; results surface via toast notifications, and a failed operation still shows recovery actions on the stack page |
|
||||
|
||||
### Diff preview before save
|
||||
#### Diff preview before save
|
||||
|
||||
When enabled, clicking **Save & Deploy** or **Save Only** in the compose or env editor opens a side-by-side diff modal before writing anything to disk. The left pane shows the current on-disk content; the right pane shows your unsaved edits, with additions highlighted green and removals highlighted red.
|
||||
|
||||
@@ -511,6 +519,20 @@ When enabled, clicking **Save & Deploy** or **Save Only** in the compose or env
|
||||
|
||||
If there are no unsaved changes the modal is skipped and the save proceeds immediately. See [Diff preview before save](/features/editor#diff-preview-before-save) in the Editor guide for the full workflow.
|
||||
|
||||
### Deploy Guardrails
|
||||
|
||||
**Scope:** Per-node (saved on the active node, admin-only)
|
||||
|
||||
Node-level safety checks and post-deploy observation used during stack deploys and updates.
|
||||
|
||||
| Setting | Default | Description |
|
||||
|---------|---------|-------------|
|
||||
| **Observe health after updates** | On | After a stack deploy or update succeeds, watch its containers for the observation window and record a passed or failed verdict on the stack timeline. Observational only: nothing is restarted or rolled back automatically. |
|
||||
| **Observation window** | 90 s | How long to watch containers before declaring the update healthy. Raise it for stacks that take a while to settle. Range 15 to 600 seconds. |
|
||||
| **Block deploy on missing required env vars** | Off | When on, a deploy or update is refused before it starts if a required `${VAR:?message}` variable is unset or empty, so the stack fails fast with a clear message instead of mid-deploy. |
|
||||
|
||||
Click **Save settings** to apply.
|
||||
|
||||
---
|
||||
|
||||
## Support
|
||||
|
||||
Reference in New Issue
Block a user