* feat(auto-heal): restart crashed containers and harden the heal loop
Auto-Heal now restarts containers that crash (non-zero exit) and stay down
past the policy threshold, in addition to those that fail their Docker
healthcheck. Crash detection reuses the container event classifier so a
container that exits cleanly or that an operator stopped is never restarted;
only classified crashes set the heal signal.
Also hardens the existing loop:
- A paid controlling instance refreshes proxied remotes' entitlement on a
background interval so a remote node's policies keep evaluating between
operator visits instead of lapsing a few minutes after the sheet was last
opened. A node that stays unreachable surfaces a warning.
- Overlapping policies (all-services plus a service-specific one) restart a
given container at most once per evaluation pass, so the hourly cap holds.
- A failed restart now counts toward the cooldown and hourly cap, so a broken
setup is retried on the cooldown interval rather than every poll.
- Diagnostic logging behind developer mode for evaluation, heal decisions,
timing, and lease refresh.
* docs(auto-heal): document crash healing and refresh troubleshooting
Cover the two heal conditions (unhealthy and crashed), note that clean exits
and operator stops are never restarted and that crash healing acts on crashes
observed while Sencho is running, and update the troubleshooting and tab
visibility entries accordingly.
* fix(auto-heal): close stale crash-signal race and harden lease refresh
A crash signal could outlive the crash it described. The exit classifier is
deferred 500ms, so an immediate restart could let it stamp the crash marker
after the container was already running, and a later clean or operator-initiated
exit did not clear it; the next poll could then restart a container that had
exited cleanly. Now a clean or intentional exit always clears the marker, a die
that a start has superseded is not stamped, and the die's own time is captured at
arrival rather than at the deferred classification so the supersede check is
accurate.
Also:
- Crash state survives the event service's idle-prune window, so crash healing
works for any configured threshold rather than only short ones.
- An exited or dead container is matched before any health-text parsing, so it
can never fall into the healthcheck path.
- A remote with no reachable proxy target counts toward the lease-refresh
failure warning instead of being silently skipped.
Introduce a NotificationCategory string-literal union (11 values) and
thread it through dispatchAlert as a required second argument. All
callers (DockerEventService, AutoHealService, ImageUpdateService,
MonitorService, PolicyEnforcement, policyGate, SchedulerService,
imageUpdates route) pass an explicit category at every call site,
giving TypeScript compile-time enforcement that no new emit site can
be added without choosing a category.
DatabaseService gains an idempotent migration that adds a nullable
category TEXT column to notification_history; existing rows keep
category=NULL (displayed as Uncategorized in the UI). The
getNotificationHistory method accepts an optional category filter
that is forwarded from the GET /api/notifications/history route via
a ?category= query param.
NotificationPanel gains a category Select dropdown so users can
filter history by category. The frontend types mirror the backend
union so API responses are type-safe end-to-end.
All 75 test files (1410 tests) updated to the new 4-arg dispatchAlert
signature and passing.
Dashboard and sidebar status indicators previously only refreshed on a
5-30 second polling cadence: a container restart, a degraded -> healthy
transition, or a stack update was invisible until the next tick.
Add a lightweight, non-persisted "state-invalidate" envelope on the
existing /ws/notifications WebSocket:
Backend
- NotificationService.broadcastEvent: sibling of dispatchAlert that
pushes an arbitrary {type, ...} envelope to every subscriber WITHOUT
writing to the alerts history (these are pure ephemeral signals).
- DockerEventService.handleEvent: emit the envelope for state-changing
container actions (start/die/kill/destroy/create/restart/pause/
unpause/health_status/rename/update). Carries node id, stack name
(from the compose project label), container id, action, and
timestamp.
Frontend
- EditorLayout's two notification WebSocket handlers (local plus
per-remote-node) branch on type. On state-invalidate they re-emit a
window CustomEvent and trigger a debounced (250ms) refreshStacks so
a burst of events from compose recreating multiple services
collapses to one refetch. The refresh callback is held in a ref so
the long-lived WS effect never closes over a stale function.
- useDashboardData listens for the same window event and refetches
/stats, /system/stats, and /stacks/statuses on every signal.
Historical metrics stay on their 60s polling cadence (10-minute
trend data, not a live indicator).
Tests
- Three new docker-event-service cases assert broadcastEvent fires on
start and health_status events with the correct envelope shape, and
does not fire on non-state actions like exec_create.
- Existing 28 cases updated with the broadcastEvent mock so the
subscriber stub matches the new shape.
Polling stays as a safety net at the same intervals; the WS path is
the fast path. Multi-node fleets benefit on the local node today;
extending the remote forwarder to relay state-invalidate is a
recommended follow-up.
Add stack_name and container_name columns to notification_history so bell
rows can act as jump points. Producers (AutoHeal, Docker events) pass the
container context through dispatchAlert; the panel renders routable rows
as buttons that load the target stack and, when a container name is
present, open its logs modal. Non-structural notifications stay as
passive display rows.
* fix(docker-events): harden crash detection against edge cases
- Isolate per-container failures in the reconcile path so one failed
container inspect cannot abort classification for the rest of the
batch after a reconnect.
- Fall back to inspecting State.OOMKilled when a container exits with
code 137 and no oom event preceded the die, so cgroup OOM kills that
lose the oom event are still classified correctly. The dedup check
runs before the fallback so crashloops do not hammer the daemon.
- Guard the intentional-kill window against wildly future-dated Docker
timestamps by switching to a signed age comparison with a bounded
negative-skew tolerance, so clock skew cannot flip a genuine crash
into an intentional stop.
- Align diagnostic logging with the codebase [Service:diag] convention
and add one informational lifecycle log on boot and shutdown so
operators running in developer mode can confirm the watcher started.
- New tests cover gap-inspect isolation, the OOM inspect fallback (both
success and failure paths), duplicate die events collapsing within
the grace window, and clock-skew bounds on the classifier.
* docs(alerts): add troubleshooting entries for crash detection toggle and rate limits
* fix(notifications): replace polling with Docker event stream for container lifecycle detection
Replaces the 30-second MonitorService crash-detection poll with a causal,
per-node Docker events stream. Eliminates false crash alerts on intentional
stops (docker stop, compose down, stack restart/update), detects OOM kills
as a distinct alert category, and surfaces real crashes in real time.
A new DockerEventManager spawns one DockerEventService per local node. Each
service consumes the filtered container event stream, classifies die events
against recent kill/oom state, and reconciles container state via snapshot
diffing on connect and reconnect. Rate limiting, exponential backoff with
jitter, and parse-error tolerance keep the stream resilient under load and
during daemon interruptions.
MonitorService retains host limits, janitor, version check, and stack metric
alerts; crash and healthcheck detection move out entirely.
* fix(tests): silence require-imports lint in hoisted mock factory
Anso