gsadmin/sencho
1
0
Fork 0
mirror of https://github.com/Studio-Saelix/sencho.git synced 2026-08-26 02:06:49 +00:00
Code Issues Packages Projects Releases Wiki Activity

fix(notifications): replace polling with Docker event stream for container lifecycle detection (#588)

Browse Source 
* fix(notifications): replace polling with Docker event stream for container lifecycle detection

Replaces the 30-second MonitorService crash-detection poll with a causal,
per-node Docker events stream. Eliminates false crash alerts on intentional
stops (docker stop, compose down, stack restart/update), detects OOM kills
as a distinct alert category, and surfaces real crashes in real time.

A new DockerEventManager spawns one DockerEventService per local node. Each
service consumes the filtered container event stream, classifies die events
against recent kill/oom state, and reconciles container state via snapshot
diffing on connect and reconnect. Rate limiting, exponential backoff with
jitter, and parse-error tolerance keep the stream resilient under load and
during daemon interruptions.

MonitorService retains host limits, janitor, version check, and stack metric
alerts; crash and healthcheck detection move out entirely.

* fix(tests): silence require-imports lint in hoisted mock factory
This commit is contained in:
Anso
2026-04-14 13:47:48 -04:00
committed by GitHub
parent 6ffac2a0db
commit ad9a6859e6
11 changed files with 1720 additions and 124 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/features/alerts-notifications.mdx
+50
View File
@@ -164,6 +164,44 @@ When the periodic image check (every 6 hours) detects that a stack has new upstr
Both notification types use the same channel routing as alerts: if notification routes are configured for a stack, those channels receive the message; otherwise, global notification channels are used as a fallback.
## Container crash detection
Sencho watches every container on each of your nodes in real time and notifies you when something exits unexpectedly. Detection is causal: Sencho distinguishes crashes from intentional stops, so stopping a stack, restarting it, or running `docker compose down` will not produce a false crash alert.
### What triggers a crash alert
| Situation | Alert |
|---|---|
| Container exits with a non-zero exit code without being asked to stop | **Crash** alert (level: error) |
| Container is killed by the kernel for exceeding its memory limit | **OOM Kill** alert (level: error) |
| Healthcheck reports the container as unhealthy | **Healthcheck failed** alert (level: error) |
Alerts arrive within a couple of seconds of the event, not on a polling interval.
### What does not trigger a crash alert
- Stopping, restarting, updating, or removing a stack from Sencho
- Running `docker stop`, `docker restart`, or `docker compose down` from a terminal on the host
- A container exiting cleanly with exit code `0`
- A container being replaced during an image update
### Global toggle
Crash and unhealthy alerts are gated by the **Global Crash Detection** toggle in **Settings > System**. When disabled, Sencho stops dispatching crash and unhealthy notifications on all nodes. Stack metric alerts and update notifications are unaffected by this toggle.
### Docker daemon interruptions
If the Docker daemon becomes unreachable (daemon restart, socket lost, network issue on a remote node), Sencho sends a single **Lost connection to Docker daemon** warning and pauses crash detection on the affected node. When the connection is restored, Sencho reconciles container state against a pre-disconnect snapshot:
- If most of your containers are still running, individual gap exits are classified and alerts are sent as normal.
- If a large share of containers exited during the outage (for example after a daemon restart), Sencho consolidates them into a single **Docker daemon interruption detected** informational notification instead of paging you for every container.
A matching **Reconnected to Docker daemon** info notification confirms monitoring has resumed.
### High-churn events
When many crashes land in a short window (for example, a large stack coming down unexpectedly), Sencho dispatches an initial batch and summarizes the remainder in a single **N additional containers crashed in the last minute** entry so your notification channels are not flooded.
## Troubleshooting
### Notifications not being delivered
@@ -180,6 +218,18 @@ Both notification types use the same channel routing as alerts: if notification
- Check the **cooldown** period; after an alert fires, it will not fire again until the cooldown expires
- Verify the metric is being collected; container must be running for stats to be gathered
### I stopped a stack but got a crash alert
This should not happen on current versions. If it does, confirm Sencho can reach the Docker daemon on the affected node (the **Lost connection to Docker daemon** warning is dispatched when the socket is unreachable). Previously, Sencho relied on polling and could mistake intentional stops for crashes; detection is now causal and in real time.
### I'm seeing fewer crash alerts after upgrading
Expected. Intentional stops, `docker stop` from a host terminal, and scheduled stack recreations no longer trigger crash alerts. Real crashes, OOM kills, and failing healthchecks still alert. If you want to opt out entirely, toggle **Global Crash Detection** off in **Settings > System**.
### After a Docker daemon restart I only got one summary notification
Expected. When a large share of containers exits during a Docker daemon interruption, Sencho consolidates them into a single informational notification instead of paging per container. Individual crashes that happen after the reconnect are alerted on as normal.
### Delete confirmation dialog
Deleting an alert rule now requires confirmation. Click the trash icon next to a rule, then confirm in the dialog that appears.