mirror of
https://github.com/Studio-Saelix/sencho.git
synced 2026-08-07 17:34:23 +00:00
41bf075eb06c2991eaaab670cffc91698a8c042b
12 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Anso
|
ec0f59a85e |
fix: dedupe healthcheck alerts and share crash rate limits (#1690)
* fix: dedupe healthcheck alerts and share crash rate limits Health flaps no longer spam the bell: emit only on transition into unhealthy, keep a prune-surviving 60m dedup stamp that advances only when history persists, and share the fixed-window rate cap with crash alerts using typed roll-up copy. Docs now match fixed-window and non-persisted overflow behavior. * fix: harden health alert rate refund and shutdown cleanup Bind rate-token refunds to the issuing fixed window, clear deferred-retry markers on recovery/destroy/shutdown, and refuse deferred health dispatches after the service stops so the 20/min cap and no-duplicate guarantees hold under async races. |
||
|
Anso
|
85842cc547 |
feat: add service-scoped stack alert rules (#1681)
* feat: add service-scoped stack alert rules
Stack alerts can target one Compose service or all services. Breach timers
are per container and cooldowns are per service so a healthy sibling no
longer clears another container's timer or silences a different service.
* fix: gate remote scoped alert creates without losing the body
Remote hops skip JSON parsing so the proxy stream stays pipeable, which
left service_name invisible to the capability gate. Buffer POST /alerts
bodies for inspection, fail closed when the remote lacks the capability,
and rewrite the buffered bytes on forward. Restore alert-panel alt text
to match the unchanged screenshot.
* fix: bound remote alert body buffer and reject encoded JSON
Cap proxied POST /alerts buffering at the local 100KB JSON limit with
structured 413 cleanup, reject non-identity Content-Encoding with 415 so
compressed scoped bodies cannot bypass the mixed-version gate, and cover
oversized, chunked, and gzip regressions.
* fix: harden service-scoped alert delete, cooldown, and proxy gates
Reject non-digit alert ids, dual-write last_fired_at for rollback safety,
gate cooldown on persisted notification history, fail-fast oversized proxy
bodies with 413, and clarify Not in compose UI semantics.
* test: expect dispatchAlert persisted result in crash-safety cases
Update notification-routing assertions for the new { persisted } return
shape so CI matches the cooldown-gating contract.
|
||
|
Anso
|
d8e4ede94f |
fix(notifications): neutralize satellite-local node names in alert bodies (#1640)
* fix(notifications): neutralize satellite-local node names in alert bodies Fleet-aggregated alerts embedded each instance seed name (often Local) while the hub badge already named the remote. Drop identity prefixes and use type-aware local wording so attribution stays on the badge. * fix(docs): correct image-update default check cadence Operator docs still said six-hour polling; the seeded default is two hours in interval mode, and the cadence is configurable or cron-based. * test(notifications): assert hub stamps roster name on neutral remote bodies Cover the fan-in path that attaches hub roster identity while leaving the satellite message body unchanged. |
||
|
Anso
|
dbb7fe8215 |
feat(auto-heal): restart crashed containers and harden the heal loop (#1258)
* feat(auto-heal): restart crashed containers and harden the heal loop Auto-Heal now restarts containers that crash (non-zero exit) and stay down past the policy threshold, in addition to those that fail their Docker healthcheck. Crash detection reuses the container event classifier so a container that exits cleanly or that an operator stopped is never restarted; only classified crashes set the heal signal. Also hardens the existing loop: - A paid controlling instance refreshes proxied remotes' entitlement on a background interval so a remote node's policies keep evaluating between operator visits instead of lapsing a few minutes after the sheet was last opened. A node that stays unreachable surfaces a warning. - Overlapping policies (all-services plus a service-specific one) restart a given container at most once per evaluation pass, so the hourly cap holds. - A failed restart now counts toward the cooldown and hourly cap, so a broken setup is retried on the cooldown interval rather than every poll. - Diagnostic logging behind developer mode for evaluation, heal decisions, timing, and lease refresh. * docs(auto-heal): document crash healing and refresh troubleshooting Cover the two heal conditions (unhealthy and crashed), note that clean exits and operator stops are never restarted and that crash healing acts on crashes observed while Sencho is running, and update the troubleshooting and tab visibility entries accordingly. * fix(auto-heal): close stale crash-signal race and harden lease refresh A crash signal could outlive the crash it described. The exit classifier is deferred 500ms, so an immediate restart could let it stamp the crash marker after the container was already running, and a later clean or operator-initiated exit did not clear it; the next poll could then restart a container that had exited cleanly. Now a clean or intentional exit always clears the marker, a die that a start has superseded is not stamped, and the die's own time is captured at arrival rather than at the deferred classification so the supersede check is accurate. Also: - Crash state survives the event service's idle-prune window, so crash healing works for any configured threshold rather than only short ones. - An exited or dead container is matched before any health-text parsing, so it can never fall into the healthcheck path. - A remote with no reachable proxy target counts toward the lease-refresh failure warning instead of being silently skipped. |
||
|
Anso
|
265fece988 |
fix(notifications): prevent self-container stack routing (#1242)
* fix(notifications): prevent self-container stack routing * fix(stack-files): stabilize download metrics in CI |
||
|
Anso
|
2d56ea958a |
fix(stack-activity): per-stack history integrity, attribution, sanitization (#1228)
* fix(stack-activity): per-stack history integrity, attribution, sanitization Address the Stack Activity audit findings (PR 1 of 2): - Per-stack history integrity: drop the per-insert 100-row prune in addNotificationHistory that evicted quieter stacks' history whenever another stack got chatty. Periodic cleanupOldNotifications now caps per (node, stack) at 500 rows and per-node unattached system events at 1000 rows, on top of the existing 30-day retention. Signature takes an options bag and returns a per-stage summary so MonitorService can log what actually ran each cycle. - Actor attribution: thread req.user?.username through every notifyActionFailure call site and add synthetic actors at service emit sites (system:autoheal, system:scheduler, system:image-update, system:docker-events, system:blueprint, system:monitor, system:policy). The timeline renders system actors as "via <Label>" so an autoheal redeploy is no longer indistinguishable from a user redeploy. - Message sanitization: new sanitizeNotificationMessage at NotificationService.dispatchAlert strips KEY=VALUE pairs whose key ends in TOKEN/KEY/PASSWORD/SECRET/CREDENTIALS/AUTH, scrubs HTTP basic auth in URLs and Bearer tokens, collapses COMPOSE_DIR paths, and truncates to 1000 chars. Applied to the stored history and to every downstream Discord/Slack/webhook channel. The ImageUpdateService recovery-path direct DB write also runs through the sanitizer. - Composite pagination cursor: getStackActivity now accepts a (timestamp, id) cursor (?before=&beforeId=). The legacy timestamp-only form silently dropped events when a single compose up emitted many events sharing one millisecond. Route rejects beforeId without before. - Frontend hardening: distinct error state with retry button (initial fetch failure no longer renders as the genuine empty state), strict positive-integer parsing on cursor params, overrequest-by-1 pagination so the last page does not leave a dead "Load more" click, runtime guard on liveEvents merge that validates the level union, per-minute day-bucket recompute so an open panel does not stay on "Today" past midnight. No tier, role, or capability gate touched. Route permission gate remains stack:read on the named stack. * fix(stack-activity): sanitizer covers lowercase env vars and per-node compose dir External review surfaced two leak paths in the message sanitizer: - The sensitive-key regex was uppercase-only. Compose env names are conventionally uppercase but lowercase forms (db_password, jwt_secret, github_token) are valid and do leak through the same Docker and compose-parse error paths. Make the regex case-insensitive and tighten it to also catch bare TOKEN= / KEY= / PASSWORD= without a prefix word, while still leaving BYPASS, COMPASS, and similar non-secret keys alone. - The compose-dir path collapse only read process.env.COMPOSE_DIR, but the real resolution chain is node.compose_dir (per-node DB override) -> process.env.COMPOSE_DIR -> /app/compose. A node with a custom compose_dir could still leak absolute paths into stored history and downstream channels. Route both the dispatchAlert call and the ImageUpdateService recovery-path direct write through NodeRegistry.getInstance().getComposeDir(localNodeId) so the collapse covers every resolution outcome. Tests now assert lowercase keys are redacted and that BYPASS-style non-secrets stay intact in both cases. notification-routing mock extended to stub the new getComposeDir call. * chore(stack-activity): a11y roles, visibility-aware tick, live-disconnect signal Close three small follow-ups on the per-stack activity timeline: - A11y: each day-group gets role="list" and each event row gets role="listitem" so screen readers traverse the timeline as a list instead of a wall of text. The day-group container also carries an aria-label naming the bucket. - Visibility-aware day-bucket tick: the 60s setInterval that re-derives Today/Yesterday/Earlier now short-circuits when document.hidden, so a backgrounded panel does not re-render every minute for no visible effect. - Live-disconnect signal: useNotifications dispatches a sencho:notifications-connection custom event on WebSocket open and close. The timeline listens and, when explicitly disconnected, shows a one-line "Live updates offline; reconnecting…" hint above the list. The sidebar ticker already surfaces fleet-wide connection state; this adds an in-context cue for users who are focused on a single stack. Stack-name case normalization was considered and rejected: stack names are case-permissive per the isValidStackName validator, and lowercasing on read or write would silently rename or hide a user's "MyApp" stack. * ci(stack-activity): drop unnecessary escape in URL_BASIC_AUTH regex ESLint no-useless-escape errored on \- inside the character class [a-zA-Z0-9+.\-] at notificationMessage.ts:14. Move the dash to the end of the class so it's an unambiguous literal and the escape is no longer required. Behavior is identical; sanitizer tests still pass. * revert(stack-activity): drop unvalidated E2E spec from this PR The spec was committed without ever running against a real Docker daemon, then failed in CI when it ran for the first time: deploy returned 200 but no notification appeared on the activity endpoint within the polling window, suggesting either a deploy-notification race or a node-id resolution mismatch in the CI environment. Backend unit tests (route + composite cursor + sanitizer) and frontend component tests cover the same logic. The E2E spec will land in a dedicated follow-up once it has been authored against a working CI environment. |
||
|
Anso
|
44dba59cab |
feat(notifications): add structured category enum to dispatcher and history (#774)
Introduce a NotificationCategory string-literal union (11 values) and thread it through dispatchAlert as a required second argument. All callers (DockerEventService, AutoHealService, ImageUpdateService, MonitorService, PolicyEnforcement, policyGate, SchedulerService, imageUpdates route) pass an explicit category at every call site, giving TypeScript compile-time enforcement that no new emit site can be added without choosing a category. DatabaseService gains an idempotent migration that adds a nullable category TEXT column to notification_history; existing rows keep category=NULL (displayed as Uncategorized in the UI). The getNotificationHistory method accepts an optional category filter that is forwarded from the GET /api/notifications/history route via a ?category= query param. NotificationPanel gains a category Select dropdown so users can filter history by category. The frontend types mirror the backend union so API responses are type-safe end-to-end. All 75 test files (1410 tests) updated to the new 4-arg dispatchAlert signature and passing. |
||
|
Anso
|
5c5021846a |
feat(events): broadcast state-invalidate on docker events so dashboard updates live (#768)
Dashboard and sidebar status indicators previously only refreshed on a
5-30 second polling cadence: a container restart, a degraded -> healthy
transition, or a stack update was invisible until the next tick.
Add a lightweight, non-persisted "state-invalidate" envelope on the
existing /ws/notifications WebSocket:
Backend
- NotificationService.broadcastEvent: sibling of dispatchAlert that
pushes an arbitrary {type, ...} envelope to every subscriber WITHOUT
writing to the alerts history (these are pure ephemeral signals).
- DockerEventService.handleEvent: emit the envelope for state-changing
container actions (start/die/kill/destroy/create/restart/pause/
unpause/health_status/rename/update). Carries node id, stack name
(from the compose project label), container id, action, and
timestamp.
Frontend
- EditorLayout's two notification WebSocket handlers (local plus
per-remote-node) branch on type. On state-invalidate they re-emit a
window CustomEvent and trigger a debounced (250ms) refreshStacks so
a burst of events from compose recreating multiple services
collapses to one refetch. The refresh callback is held in a ref so
the long-lived WS effect never closes over a stale function.
- useDashboardData listens for the same window event and refetches
/stats, /system/stats, and /stacks/statuses on every signal.
Historical metrics stay on their 60s polling cadence (10-minute
trend data, not a live indicator).
Tests
- Three new docker-event-service cases assert broadcastEvent fires on
start and health_status events with the correct envelope shape, and
does not fire on non-state actions like exec_create.
- Existing 28 cases updated with the broadcastEvent mock so the
subscriber stub matches the new shape.
Polling stays as a safety net at the same intervals; the WS path is
the fast path. Multi-node fleets benefit on the local node today;
extending the remote forwarder to relay state-invalidate is a
recommended follow-up.
|
||
|
Anso
|
ed2a16af79 |
feat(notifications): deep-link bell rows to source stack and container logs (#692)
Add stack_name and container_name columns to notification_history so bell rows can act as jump points. Producers (AutoHeal, Docker events) pass the container context through dispatchAlert; the panel renders routable rows as buttons that load the target stack and, when a container name is present, open its logs modal. Non-structural notifications stay as passive display rows. |
||
|
Anso
|
5bb4b01953 |
feat: auto-heal policies for unhealthy containers (#671)
* feat(db): add auto_heal_policies and auto_heal_history schema and CRUD Adds two new SQLite tables (auto_heal_policies, auto_heal_history) to DatabaseService.initSchema() and exposes CRUD methods: getAutoHealPolicies, getAutoHealPolicy, addAutoHealPolicy, updateAutoHealPolicy, deleteAutoHealPolicy, recordAutoHealHistory, getAutoHealHistory, incrementConsecutiveFailures, resetConsecutiveFailures, setPolicyEnabled. Also adds AutoHealPolicy and AutoHealHistoryEntry TypeScript interfaces. * feat(events): track health-status duration and expose state accessors - Add healthStatus and unhealthySince fields to InternalContainerState - onHealthStatus now records unhealthySince timestamp on first transition to unhealthy, and clears it when the container recovers or restarts - onStart resets both fields so a restarted container begins from 'starting' - Add listContainerStates() and getContainerState() public accessors for use by the upcoming AutoHealService evaluator * fix(auto-heal): key allowlist in updateAutoHealPolicy, cascade delete, extract ContainerHealthSnapshot * feat: add AutoHealService evaluator singleton Polls every 30 s, matches containers to enabled policies via Compose labels, and restarts containers that have been unhealthy beyond the configured threshold. Enforces cooldown, per-hour rate cap, and recent-user-action suppression; auto-disables policies after repeated consecutive failures. Also adds DockerEventManager.getService() accessor required by the evaluator. * fix(auto-heal): prune stale restartTimestamps, guard undefined policy id - Prune restartTimestamps entries for containers no longer running after each container list fetch, preventing unbounded map growth from dead container IDs. - Guard against policies with undefined id at the start of the per-policy loop; warn and skip rather than proceed with a non-null assertion. - Extract handleAutoDisable private helper to bring executeHeal under 30 lines and isolate the auto-disable side-effect sequence. - Move ContainerInfo type to module scope. * feat: add auto-heal API routes and wire AutoHealService lifecycle Registers five REST endpoints under /api/auto-heal/policies (list, create, patch, delete, history) with requirePaid + requireAdmin guards and Zod validation. Wires AutoHealService.start()/stop() into the server startup and graceful-shutdown blocks alongside MonitorService. * test: add AutoHealService and DatabaseService auto-heal unit tests - 15 unit tests for AutoHealService.shouldHeal covering all decision branches (healthy state, duration threshold, user-action suppression, cooldown, rate limiting, and correct skipReason values) - 13 integration tests for DatabaseService auto-heal CRUD: policy round-trip, stack-name filter, partial update, cascade delete, history ordering/limit, consecutive failure counters, and setPolicyEnabled toggle * fix: log AutoHealService shutdown errors consistently * fix(api): requireAdmin-first guard order and try/catch on auto-heal routes * feat(ui): add StackAutoHealSheet component * feat(ui): add Auto-Heal context menu item to EditorLayout * fix(ui): StackAutoHealSheet label, token, a11y, and useEffect fixes - Rename 'All services in stack' to 'All services' in combobox options and placeholder - Replace text-green-600 with text-success design token in actionColorClass - Add htmlFor/id pairs to all four numeric form inputs for accessibility - Inline fetch logic into useEffect, removing stale closure risk and eslint-disable comment - Remove now-unused fetchPolicies and fetchServices standalone functions - Update 'Auto-disable after' label to 'Auto-disable after (failures)' for clarity - Add toast.error in policy fetch failure path; services fetch silently skips as before * docs: add auto-heal-policies feature documentation * test(e2e): add auto-heal policies CRUD spec * fix(docs): correct auto-heal-policies nav position in docs.json |
||
|
Anso
|
44a89d9d2e |
fix(docker-events): harden crash detection against edge cases (#591)
* fix(docker-events): harden crash detection against edge cases - Isolate per-container failures in the reconcile path so one failed container inspect cannot abort classification for the rest of the batch after a reconnect. - Fall back to inspecting State.OOMKilled when a container exits with code 137 and no oom event preceded the die, so cgroup OOM kills that lose the oom event are still classified correctly. The dedup check runs before the fallback so crashloops do not hammer the daemon. - Guard the intentional-kill window against wildly future-dated Docker timestamps by switching to a signed age comparison with a bounded negative-skew tolerance, so clock skew cannot flip a genuine crash into an intentional stop. - Align diagnostic logging with the codebase [Service:diag] convention and add one informational lifecycle log on boot and shutdown so operators running in developer mode can confirm the watcher started. - New tests cover gap-inspect isolation, the OOM inspect fallback (both success and failure paths), duplicate die events collapsing within the grace window, and clock-skew bounds on the classifier. * docs(alerts): add troubleshooting entries for crash detection toggle and rate limits |
||
|
Anso
|
ad9a6859e6 |
fix(notifications): replace polling with Docker event stream for container lifecycle detection (#588)
* fix(notifications): replace polling with Docker event stream for container lifecycle detection Replaces the 30-second MonitorService crash-detection poll with a causal, per-node Docker events stream. Eliminates false crash alerts on intentional stops (docker stop, compose down, stack restart/update), detects OOM kills as a distinct alert category, and surfaces real crashes in real time. A new DockerEventManager spawns one DockerEventService per local node. Each service consumes the filtered container event stream, classifies die events against recent kill/oom state, and reconciles container state via snapshot diffing on connect and reconnect. Rate limiting, exponential backoff with jitter, and parse-error tolerance keep the stream resilient under load and during daemon interruptions. MonitorService retains host limits, janitor, version check, and stack metric alerts; crash and healthcheck detection move out entirely. * fix(tests): silence require-imports lint in hoisted mock factory |