gsadmin/sencho
1
0
Fork 0
mirror of https://github.com/Studio-Saelix/sencho.git synced 2026-08-10 18:56:53 +00:00
Code Issues Packages Projects Releases Wiki Activity

fix: dedupe healthcheck alerts and share crash rate limits (#1690)

Browse Source 
* fix: dedupe healthcheck alerts and share crash rate limits

Health flaps no longer spam the bell: emit only on transition into unhealthy, keep a prune-surviving 60m dedup stamp that advances only when history persists, and share the fixed-window rate cap with crash alerts using typed roll-up copy. Docs now match fixed-window and non-persisted overflow behavior.

* fix: harden health alert rate refund and shutdown cleanup

Bind rate-token refunds to the issuing fixed window, clear deferred-retry markers on recovery/destroy/shutdown, and refuse deferred health dispatches after the service stops so the 20/min cap and no-duplicate guarantees hold under async races.
This commit is contained in:
Anso
2026-07-23 23:14:25 -04:00
committed by GitHub
parent a89498ae5b
commit ec0f59a85e
3 changed files with 823 additions and 57 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/features/alerts-notifications.mdx
+10 -8
View File
@@ -308,10 +308,11 @@ Real-time on the Docker event stream:
- **Crash** at `error`/`monitor_alert`: `Container Crash Detected: <name> exited unexpectedly (Code: <N>).`
- **OOM kill** at `error`/`monitor_alert`: `Container OOM Kill: <name> was killed by the OOM killer (out of memory).` When a `die` arrives with exit code 137 but no preceding `oom` event, Sencho inspects the container and reclassifies as OOM if the `OOMKilled` flag is set.
- **Healthcheck failure** at `error`/`monitor_alert`: `Healthcheck failed: <name> is unhealthy.`
- **Healthcheck failure** at `error`/`monitor_alert`: `Healthcheck failed: <name> is unhealthy.` Emitted only on the transition into Docker `unhealthy` (not on every repeated unhealthy event).
- **Mass exit** kicks in when a daemon-disconnect gap is followed by 20% or more of containers exiting on reconnect, in which case a single `info`/`system` summary `Docker daemon interruption detected: N containers exited during connection gap.` is emitted instead of N crash alerts.
- **Rate limit** caps crash dispatches at 20 per 60-second window per node, then a single `warning`/`monitor_alert` `N additional containers crashed in the last minute.`
- **Dedup** is 60 minutes per container after a non-rate-suppressed dispatch.
- **Rate limit** caps combined crash and health dispatches at 20 per fixed 60-second window per node. Overflow does not write individual history rows; at the end of the window Sencho emits a single `warning`/`monitor_alert` roll-up such as `N additional container crash alerts were rate-limited in the last minute.`, `N additional container health alerts were rate-limited in the last minute.`, or `N additional container alerts were rate-limited in the last minute (X crash, Y health).`
- **Crash dedup** is 60 minutes per container after a non-rate-suppressed crash dispatch. Cleared when that container starts again.
- **Health dedup** is 60 minutes per container after a health alert that was persisted to notification history. It survives healthy/starting recovery and the 10-minute prune of in-memory container tracking state (dedup lives in a separate map), and clears only when the window expires or the Sencho process restarts.
### Daemon connectivity
@@ -406,7 +407,7 @@ Three retention controls live under **Settings · Operations · Data Retention**
A hard 100-row per-node cap inside `notification_history` always applies on top of the user-tunable retention; new inserts evict the oldest rows beyond 100 even if your retention window is longer.
A separate rate limit applies to crash and health alerts only: 20 emits per 60-second window per node, with a single `warning`/`monitor_alert` roll-up `N additional containers crashed in the last minute.` issued at the end of the window.
A separate rate limit applies to crash and health alerts only: 20 emits per fixed 60-second window per node (not a rolling window). When the cap is hit, individual overflow alerts are not written to history or the bell; at the end of the window Sencho emits a single `warning`/`monitor_alert` roll-up that names crash-only, health-only, or mixed suppression.
## Crash detection toggle
@@ -432,8 +433,9 @@ The **Host Alerts** panel carries the **Host thresholds** rows (CPU limit, RAM l
| Notification fanout to channels | One attempt by default; optional 0-3 extra in-process attempts with a fixed 1s delay; 10-second timeout per attempt; no durable queue |
| Bell live updates | Pushed live over the notifications WebSocket per node |
| Bell safety-net reconcile | 60 seconds |
| Crash dedup window per container | 60 minutes |
| Crash rate-limit window | 20 alerts per 60 seconds per node, then a single roll-up |
| Crash dedup window per container | 60 minutes (cleared on container start) |
| Health dedup window per container | 60 minutes after a persisted health alert; survives recovery; process-local |
| Crash/health rate-limit window | 20 alerts per fixed 60 seconds per node, then a single typed roll-up (overflow individuals are not persisted) |
Switching the active node tears down per-stack rule editors and reloads channel state, but does not affect the bell, which keeps every node's WebSocket open in parallel.
@@ -455,8 +457,8 @@ Switching the active node tears down per-stack rule editors and reloads channel
<Accordion title="Crash alerts stopped arriving and nothing else looks wrong">
Check **Settings · Monitoring · Container Alerts · Container crash & health alerts**. The toggle is the master switch for the Docker event service. The panel cache reads the database every 500 ms; if the database read errors, Sencho defaults the toggle to off so the failure mode is silent rather than spammy. Repair the toggle, save, and the next Docker event reaches the dispatcher.
</Accordion>
<Accordion title="A burst of crashes happened but I only see about twenty alerts">
Sencho rate-limits crash and health dispatches to 20 emits per rolling 60-second window per node, then emits a single `warning`/`monitor_alert` roll-up `N additional containers crashed in the last minute.` once the window closes. Every alert is still persisted to `notification_history` and visible in the bell up to the 100-row per-node cap; only the channel fanout is throttled.
<Accordion title="A burst of crashes or health flaps happened but I only see about twenty alerts">
Sencho rate-limits crash and health dispatches together to 20 emits per fixed 60-second window per node, then emits a single `warning`/`monitor_alert` roll-up such as `N additional container crash alerts were rate-limited in the last minute.`, a health-only variant, or a mixed `(X crash, Y health)` summary once the window closes. Rate-limited overflow individuals are not written to `notification_history` and do not appear in the bell; only the roll-up is persisted for that overflow. Separately, health flaps on the same container are deduped for 60 minutes after a persisted unhealthy alert, even if Docker recovered and failed again inside that window.
</Accordion>
<Accordion title="My deploy success doesn't appear in the bell, only as a toast">
Sencho deliberately hides rows whose category is one of `deploy_success`, `stack_started`, `stack_stopped`, `stack_restarted`, or `image_update_applied` AND whose `actor_username` is set to a real user. The reasoning: those are confirmations of the action you just clicked and are already shown as a toast. The rows are still persisted to `notification_history` and still dispatched to global channels and matching routes; only the bell render hides them.