Introduce a NotificationCategory string-literal union (11 values) and
thread it through dispatchAlert as a required second argument. All
callers (DockerEventService, AutoHealService, ImageUpdateService,
MonitorService, PolicyEnforcement, policyGate, SchedulerService,
imageUpdates route) pass an explicit category at every call site,
giving TypeScript compile-time enforcement that no new emit site can
be added without choosing a category.
DatabaseService gains an idempotent migration that adds a nullable
category TEXT column to notification_history; existing rows keep
category=NULL (displayed as Uncategorized in the UI). The
getNotificationHistory method accepts an optional category filter
that is forwarded from the GET /api/notifications/history route via
a ?category= query param.
NotificationPanel gains a category Select dropdown so users can
filter history by category. The frontend types mirror the backend
union so API responses are type-safe end-to-end.
All 75 test files (1410 tests) updated to the new 4-arg dispatchAlert
signature and passing.
Dashboard and sidebar status indicators previously only refreshed on a
5-30 second polling cadence: a container restart, a degraded -> healthy
transition, or a stack update was invisible until the next tick.
Add a lightweight, non-persisted "state-invalidate" envelope on the
existing /ws/notifications WebSocket:
Backend
- NotificationService.broadcastEvent: sibling of dispatchAlert that
pushes an arbitrary {type, ...} envelope to every subscriber WITHOUT
writing to the alerts history (these are pure ephemeral signals).
- DockerEventService.handleEvent: emit the envelope for state-changing
container actions (start/die/kill/destroy/create/restart/pause/
unpause/health_status/rename/update). Carries node id, stack name
(from the compose project label), container id, action, and
timestamp.
Frontend
- EditorLayout's two notification WebSocket handlers (local plus
per-remote-node) branch on type. On state-invalidate they re-emit a
window CustomEvent and trigger a debounced (250ms) refreshStacks so
a burst of events from compose recreating multiple services
collapses to one refetch. The refresh callback is held in a ref so
the long-lived WS effect never closes over a stale function.
- useDashboardData listens for the same window event and refetches
/stats, /system/stats, and /stacks/statuses on every signal.
Historical metrics stay on their 60s polling cadence (10-minute
trend data, not a live indicator).
Tests
- Three new docker-event-service cases assert broadcastEvent fires on
start and health_status events with the correct envelope shape, and
does not fire on non-state actions like exec_create.
- Existing 28 cases updated with the broadcastEvent mock so the
subscriber stub matches the new shape.
Polling stays as a safety net at the same intervals; the WS path is
the fast path. Multi-node fleets benefit on the local node today;
extending the remote forwarder to relay state-invalidate is a
recommended follow-up.
Add stack_name and container_name columns to notification_history so bell
rows can act as jump points. Producers (AutoHeal, Docker events) pass the
container context through dispatchAlert; the panel renders routable rows
as buttons that load the target stack and, when a container name is
present, open its logs modal. Non-structural notifications stay as
passive display rows.
* feat(db): add auto_heal_policies and auto_heal_history schema and CRUD
Adds two new SQLite tables (auto_heal_policies, auto_heal_history) to
DatabaseService.initSchema() and exposes CRUD methods: getAutoHealPolicies,
getAutoHealPolicy, addAutoHealPolicy, updateAutoHealPolicy,
deleteAutoHealPolicy, recordAutoHealHistory, getAutoHealHistory,
incrementConsecutiveFailures, resetConsecutiveFailures, setPolicyEnabled.
Also adds AutoHealPolicy and AutoHealHistoryEntry TypeScript interfaces.
* feat(events): track health-status duration and expose state accessors
- Add healthStatus and unhealthySince fields to InternalContainerState
- onHealthStatus now records unhealthySince timestamp on first transition
to unhealthy, and clears it when the container recovers or restarts
- onStart resets both fields so a restarted container begins from 'starting'
- Add listContainerStates() and getContainerState() public accessors for
use by the upcoming AutoHealService evaluator
* fix(auto-heal): key allowlist in updateAutoHealPolicy, cascade delete, extract ContainerHealthSnapshot
* feat: add AutoHealService evaluator singleton
Polls every 30 s, matches containers to enabled policies via Compose
labels, and restarts containers that have been unhealthy beyond the
configured threshold. Enforces cooldown, per-hour rate cap, and
recent-user-action suppression; auto-disables policies after repeated
consecutive failures. Also adds DockerEventManager.getService() accessor
required by the evaluator.
* fix(auto-heal): prune stale restartTimestamps, guard undefined policy id
- Prune restartTimestamps entries for containers no longer running after
each container list fetch, preventing unbounded map growth from dead
container IDs.
- Guard against policies with undefined id at the start of the per-policy
loop; warn and skip rather than proceed with a non-null assertion.
- Extract handleAutoDisable private helper to bring executeHeal under 30
lines and isolate the auto-disable side-effect sequence.
- Move ContainerInfo type to module scope.
* feat: add auto-heal API routes and wire AutoHealService lifecycle
Registers five REST endpoints under /api/auto-heal/policies (list, create,
patch, delete, history) with requirePaid + requireAdmin guards and Zod
validation. Wires AutoHealService.start()/stop() into the server startup
and graceful-shutdown blocks alongside MonitorService.
* test: add AutoHealService and DatabaseService auto-heal unit tests
- 15 unit tests for AutoHealService.shouldHeal covering all decision branches
(healthy state, duration threshold, user-action suppression, cooldown,
rate limiting, and correct skipReason values)
- 13 integration tests for DatabaseService auto-heal CRUD: policy round-trip,
stack-name filter, partial update, cascade delete, history ordering/limit,
consecutive failure counters, and setPolicyEnabled toggle
* fix: log AutoHealService shutdown errors consistently
* fix(api): requireAdmin-first guard order and try/catch on auto-heal routes
* feat(ui): add StackAutoHealSheet component
* feat(ui): add Auto-Heal context menu item to EditorLayout
* fix(ui): StackAutoHealSheet label, token, a11y, and useEffect fixes
- Rename 'All services in stack' to 'All services' in combobox options and placeholder
- Replace text-green-600 with text-success design token in actionColorClass
- Add htmlFor/id pairs to all four numeric form inputs for accessibility
- Inline fetch logic into useEffect, removing stale closure risk and eslint-disable comment
- Remove now-unused fetchPolicies and fetchServices standalone functions
- Update 'Auto-disable after' label to 'Auto-disable after (failures)' for clarity
- Add toast.error in policy fetch failure path; services fetch silently skips as before
* docs: add auto-heal-policies feature documentation
* test(e2e): add auto-heal policies CRUD spec
* fix(docs): correct auto-heal-policies nav position in docs.json
* fix(docker-events): harden crash detection against edge cases
- Isolate per-container failures in the reconcile path so one failed
container inspect cannot abort classification for the rest of the
batch after a reconnect.
- Fall back to inspecting State.OOMKilled when a container exits with
code 137 and no oom event preceded the die, so cgroup OOM kills that
lose the oom event are still classified correctly. The dedup check
runs before the fallback so crashloops do not hammer the daemon.
- Guard the intentional-kill window against wildly future-dated Docker
timestamps by switching to a signed age comparison with a bounded
negative-skew tolerance, so clock skew cannot flip a genuine crash
into an intentional stop.
- Align diagnostic logging with the codebase [Service:diag] convention
and add one informational lifecycle log on boot and shutdown so
operators running in developer mode can confirm the watcher started.
- New tests cover gap-inspect isolation, the OOM inspect fallback (both
success and failure paths), duplicate die events collapsing within
the grace window, and clock-skew bounds on the classifier.
* docs(alerts): add troubleshooting entries for crash detection toggle and rate limits
* fix(notifications): replace polling with Docker event stream for container lifecycle detection
Replaces the 30-second MonitorService crash-detection poll with a causal,
per-node Docker events stream. Eliminates false crash alerts on intentional
stops (docker stop, compose down, stack restart/update), detects OOM kills
as a distinct alert category, and surfaces real crashes in real time.
A new DockerEventManager spawns one DockerEventService per local node. Each
service consumes the filtered container event stream, classifies die events
against recent kill/oom state, and reconciles container state via snapshot
diffing on connect and reconnect. Rate limiting, exponential backoff with
jitter, and parse-error tolerance keep the stream resilient under load and
during daemon interruptions.
MonitorService retains host limits, janitor, version check, and stack metric
alerts; crash and healthcheck detection move out entirely.
* fix(tests): silence require-imports lint in hoisted mock factory
Anso