fix(notifications): align Sencho update alerts with Fleet cache (#1620)

Remove the MonitorService 6-hour version-check cooldown so node_update_available
fires when the shared version cache observes a published release, matching the
Fleet update button. Dedup and publish-pending gating still prevent spam.
This commit is contained in:
Anso
2026-07-14 12:20:25 -04:00
committed by GitHub
parent 381ed2a91f
commit 4079cb9198
3 changed files with 90 additions and 149 deletions
+5 -4
View File
@@ -144,11 +144,12 @@ Every alert Sencho dispatches carries a category that you can filter on in the b
| `stack_stopped` | Stack stopped | Stack stopped via the dashboard or API |
| `stack_restarted` | Stack restarted | Stack restarted via the dashboard or API |
| `image_update_available` | Update available | Image-update poll found a newer digest |
| `node_update_available` | Node update | Sencho self-update available for this node |
| `image_update_applied` | Update applied | Manual or scheduled auto-update applied new images |
| `autoheal_triggered` | Auto-heal | Auto-heal restarted, failed to restart, or auto-disabled a policy |
| `monitor_alert` | Monitor alert | Per-stack threshold breach, host CPU/RAM/disk warning, or healthcheck failure |
| `scan_finding` | Scan finding | Vulnerability-scan completion, per-violation alert, post-deploy scan failure, or auto-update gate block |
| `system` | System | Sencho version, Trivy auto-update, fleet sync, daemon connectivity, scheduled-task lifecycle, cloud-backup upload failure |
| `system` | System | Trivy auto-update, fleet sync, daemon connectivity, scheduled-task lifecycle, cloud-backup upload failure |
| `blueprint_deployed` | `blueprint_deployed` | Blueprint provisioned a new deployment |
| `blueprint_deployment_failed` | `blueprint_deployment_failed` | Blueprint deployment errored out |
| `blueprint_drift_detected` | `blueprint_drift_detected` | Blueprint drift detected in `suggest` or `enforce` mode |
@@ -327,7 +328,7 @@ The entire host threshold evaluation can be silenced per node from **Settings ·
### Sencho version availability
`info`/`system`: `Sencho X.Y.Z is available (currently running A.B.C). Visit the Fleet dashboard to update.` Polled every 6 hours; one-shot dedup until the running version reaches or passes the notified version.
`info`/`node_update_available`: `Sencho X.Y.Z is available (currently running A.B.C). Visit the Fleet dashboard to update.` Monitor evaluates every 30 seconds against a shared version cache (refreshes every 30 minutes when published, or every 3 minutes while publish is pending). One-shot dedup until the running version reaches or passes the notified version.
### Image update availability
@@ -416,7 +417,7 @@ The **Host Alerts** panel carries the **Host thresholds** rows (CPU limit, RAM l
| Host CPU / RAM / disk threshold checks | 30 seconds |
| Per-stack alert rule evaluation | 30 seconds |
| Image update poll | Configurable (default every 2 hours), with a 2-minute startup delay and a 2-minute cooldown on manual refresh |
| Sencho version check | 6 hours |
| Sencho version check | Monitor evaluation every 30 seconds; shared version cache refreshes every 30 minutes when published or every 3 minutes while publish is pending |
| Notification fanout to channels | Single shot per dispatch, 10-second timeout, no retries |
| Bell live updates | Pushed live over the notifications WebSocket per node |
| Bell safety-net reconcile | 60 seconds |
@@ -453,7 +454,7 @@ Switching the active node tears down per-stack rule editors and reloads channel
Routing matchers AND together: every non-empty matcher must match the alert. A rule with **Stacks** set to `prod-api` will not match a `monitor_alert` for a different stack, and a rule with a populated **Stacks** matcher will not match host-level alerts (which carry no stack target). When zero rules match, Sencho falls back to global channels. To intercept everything, leave all four matcher fields empty on the rule. Also confirm the rule's **Enabled** pill is `ON`.
</Accordion>
<Accordion title="Fleet shows an update button but I never received a Sencho version notification">
The version poll runs every 6 hours and dedups against the `last_sencho_update_notified_version` system-state key, so each newly-released version produces exactly one alert per node. If you upgraded Sencho between the release and the poll, the dedup self-heals; the alert simply does not fire because the running version already matches. To force a fresh check, restart the Sencho container; the version poll runs once on startup after a 2-minute delay.
Fleet and the notification path share the same version lookup. When a newer published release is in that cache, Monitor dispatches a single `info`/`node_update_available` alert and records it under `last_sencho_update_notified_version`, so each release produces one alert per node. If you already upgraded to (or past) that version, the dedup self-heals and the alert does not fire. Check **Settings · Notifications · Mute Rules** for a rule that mutes `node_update_available` in the bell. A newly published release may take up to the cache TTL (about 30 minutes when published, or about 3 minutes while registry publish is still pending) before Fleet and the bell both observe it.
</Accordion>
<Accordion title="A stack shows the blue update indicator but no notification was received">
The image-update service emits on **state transitions only**. The first run of the service backfills a flag against rows that already had `has_update: true` so they do not all re-fire on first install. Once the flag is set, only the false-to-true transition triggers an alert. If a stack was already showing the indicator before backfill ran, a notification will only appear the next time it goes from up-to-date back to behind.