Commit Graph

8 Commits

Author SHA1 Message Date
Anso 44dba59cab feat(notifications): add structured category enum to dispatcher and history (#774)
Introduce a NotificationCategory string-literal union (11 values) and
thread it through dispatchAlert as a required second argument. All
callers (DockerEventService, AutoHealService, ImageUpdateService,
MonitorService, PolicyEnforcement, policyGate, SchedulerService,
imageUpdates route) pass an explicit category at every call site,
giving TypeScript compile-time enforcement that no new emit site can
be added without choosing a category.

DatabaseService gains an idempotent migration that adds a nullable
category TEXT column to notification_history; existing rows keep
category=NULL (displayed as Uncategorized in the UI). The
getNotificationHistory method accepts an optional category filter
that is forwarded from the GET /api/notifications/history route via
a ?category= query param.

NotificationPanel gains a category Select dropdown so users can
filter history by category. The frontend types mirror the backend
union so API responses are type-safe end-to-end.

All 75 test files (1410 tests) updated to the new 4-arg dispatchAlert
signature and passing.
2026-04-25 13:55:07 -04:00
Anso 6b8c369745 fix(notifications): stop Sencho version notifications from silently skipping (#594)
* fix(notifications): stop Sencho version notifications from silently skipping

Three independent defects combined to make version-update notifications
silently fail even while Fleet overview correctly surfaced an update button:

- The in-memory 6-hour cooldown was advanced before the network fetch,
  so a single transient failure at boot could lock the check for the
  rest of the container lifetime. Moved the cooldown update inside the
  success branch so failures retry on the next eval cycle.
- MonitorService called the raw version fetch directly, bypassing the
  CacheService wrapper (TTL, inflight dedup, stale-on-error) that Fleet
  uses, so the two paths could diverge. Unified both on a shared
  getLatestVersion() helper in utils/version-check.ts.
- The dedup key could carry stale state from a previous build and never
  self-clear. It now self-heals when the running version reaches the
  previously-notified version, so future releases re-fire as expected.

Added diagnostic logs gated on debug mode for each skip branch, plus
three regression tests covering cooldown-on-failure, cooldown-on-success,
and dedup self-heal.

* docs(notifications): drop legacy-upgrade framing from alerts troubleshooting

Sencho has not shipped publicly, so troubleshooting entries written in
'this used to happen but now does Y' mode reference a past that does not
exist for any reader. Rewrote the version-notification, image-update,
and crash-alert troubleshooting entries to describe current behavior
positively without referring to prior builds, upgrade paths, or legacy
fixes.

* chore(security): accept CVE-2026-33810 in bundled Docker CLI 29.4.0

Trivy now flags CVE-2026-33810 (Go stdlib crypto/x509 DNS constraint
bypass, fixed in Go 1.26.2) in the Docker CLI static binary we ship.
Docker CLI 29.4.0 is the latest upstream release and still links Go
1.26.1; no newer static binary exists yet.

Same exposure profile as the already-accepted CVE-2026-32280: the
Docker CLI and compose plugin only validate certificates from
well-known registry CAs and the local Docker socket, not from
attacker-controlled CAs with crafted DNS name constraints. Revisit
on the next Docker CLI release that rebuilds against Go 1.26.2 or
later.
2026-04-14 16:34:28 -04:00
Anso ad9a6859e6 fix(notifications): replace polling with Docker event stream for container lifecycle detection (#588)
* fix(notifications): replace polling with Docker event stream for container lifecycle detection

Replaces the 30-second MonitorService crash-detection poll with a causal,
per-node Docker events stream. Eliminates false crash alerts on intentional
stops (docker stop, compose down, stack restart/update), detects OOM kills
as a distinct alert category, and surfaces real crashes in real time.

A new DockerEventManager spawns one DockerEventService per local node. Each
service consumes the filtered container event stream, classifies die events
against recent kill/oom state, and reconciles container state via snapshot
diffing on connect and reconnect. Rate limiting, exponential backoff with
jitter, and parse-error tolerance keep the stream resilient under load and
during daemon interruptions.

MonitorService retains host limits, janitor, version check, and stack metric
alerts; crash and healthcheck detection move out entirely.

* fix(tests): silence require-imports lint in hoisted mock factory
2026-04-14 13:47:48 -04:00
Anso 4a0319331a fix(notifications): resolve version notification showing 0.0.0 and backfill missing image update notifications (#586)
- Read running Sencho version from the packaged manifest via getSenchoVersion() instead of process.env.npm_package_version, which is undefined when launched via node dist/index.js (production Docker). Skip the update check entirely when the version cannot be resolved.
- Add a one-time backfill pass in ImageUpdateService so users who upgraded to a Sencho version with the notification pipeline receive a catch-up entry for stacks already flagged as having updates before the upgrade.
- Surface dispatch failures as error-level entries in the in-app notification bell via a direct notification_history write, so misconfigured webhooks are visible without tailing logs.
- Extend test coverage for both paths: mock getSenchoVersion (including the null-version case) and add dispatch-path tests for transition, no-re-fire, backfill, and error surfacing.
- Expand alerts-notifications docs with per-instance 6-hour check cadence, an example version message, a backfill note, and three new troubleshooting entries.
2026-04-14 11:51:30 -04:00
Anso 0a94df318a fix(alerts): harden with security fixes, design compliance, and test coverage (#570)
* fix(alerts): harden with security fixes, design compliance, and test coverage

Add authMiddleware to all alert endpoints, validate notification test
dispatch inputs, fix restart_count metric via Docker inspect, correct
network metric units, replace any types with DockerContainerStats
interface, add webhook timeouts and dispatch error tracking.

Frontend: migrate Select to Combobox, add ScrollArea and delete
confirmation AlertDialog, fix icon strokeWidth to 1.5.

Add update availability notifications for both Sencho version updates
(6-hour check in MonitorService) and stack image updates (state
transition detection in ImageUpdateService). Extract shared version
fetch logic into utils/version-check.ts.

Add diagnostic logging gated behind developer_mode for MonitorService
breach state machine and NotificationService dispatch routing.

Tests: 24 new alert API integration tests, restart_count and version
check unit tests (688 total passing). Docs updated with HTTPS
requirement, update notifications section, and troubleshooting guide.

* fix(alerts): remove unused TEST_USERNAME import in alerts-api tests
2026-04-13 21:40:53 -04:00
Anso 9db97107aa fix(dashboard): harden real-time dashboard with bug fixes and design compliance (#517)
* fix(dashboard): harden real-time dashboard with bug fixes and design compliance

- Fix host alert spam: add 5-minute cooldown for CPU/RAM/disk threshold
  alerts, preventing duplicate notifications every 30s during sustained
  breaches. Extract shared dispatchWithCooldown helper (also used by
  Docker janitor alerts).
- Fix memory metric inflation: subtract filesystem cache from stored
  memory_mb values, matching the existing calculateMemoryPercent logic.
- Fix crash detection reliability: replace fragile 'seconds ago' string
  matching with a tracked Set of alerted container IDs. Containers are
  only alerted once per crash event, with automatic cleanup when they
  start running again or after a 1-hour TTL.
- Fix health status bar: exited containers now trigger 'degraded' state
  independently of unread error notifications.
- Fix CPU chart Y-axis: auto-scale when aggregate container CPU exceeds
  100% instead of silently clipping at the hardcoded domain ceiling.
- Fix grammar: 'actives' to 'active' in container count label.
- Add shadow-card-bevel to all dashboard cards per design system.
- Update dashboard docs to reflect revised health status thresholds.

* test(dashboard): update monitor service tests for new alert signatures

- Add container Id fields to crash detection test fixtures
- Update host alert assertions to match dispatchWithCooldown 3-arg call
- Fix unhealthy container test to use State: 'unhealthy' instead of
  State: 'running' (running containers are now skipped in crash detect)
2026-04-12 04:25:41 -04:00
Anso 1b573f542a feat(notifications): add shared notification routing rules (Admiral tier) (#347)
Route stack alerts to specific Discord, Slack, or webhook channels instead
of the single global endpoint. Includes per-rule enable/disable, priority
ordering, and automatic fallback to global agents when no rule matches.

- Add notification_routes table, interface, and CRUD in DatabaseService
- Add routing logic in NotificationService.dispatchAlert with optional stackName
- Pass stack context from MonitorService (crash/health) and SchedulerService
- Add 5 API endpoints gated with requireAdmin + requireAdmiral
- Add NotificationRoutingSection UI with Combobox stack picker, channel tabs
- Parallel webhook dispatch via Promise.allSettled
- 10 unit tests covering routing, fallback, and edge cases
- Documentation with screenshots at docs/features/notification-routing.mdx
2026-04-02 22:22:27 -04:00
Anso 93ae147ec1 test: expand backend test coverage for stability gaps (#329)
Add 5 new test files (116 test cases) targeting previously untested
service-layer logic:

- database-metrics: metrics CRUD, minute-bucket aggregation, cleanup
  retention, notification auto-cap at 100, stack alerts CRUD, and
  stress tests with 1000+ metrics
- monitor-service: CPU/memory/network calculation helpers, all 6
  alert condition operators, breach state machine lifecycle, global
  crash detection, host limit thresholds, cleanup delegation, and
  isProcessing concurrency guard
- docker-controller: validateApiData error detection, state-safe
  start/stop (304 handling), batch container removal with partial
  failures, disk usage calculation, resource classification
  (managed/unmanaged/system), orphan detection, and daemon
  unreachable error propagation
- scheduler-service: cron parsing, license tier gating, concurrent
  task prevention via runningTasks Set, manual trigger, all 4 task
  types (restart/snapshot/prune/update), wildcard targets, error
  recording, recovery notifications, and cleanup
- compose-service: subprocess spawn/exit handling, WebSocket output,
  deploy with health probe and atomic rollback, registry auth temp
  dir lifecycle, and downStack teardown
2026-04-01 22:43:35 -04:00