mirror of
https://github.com/Studio-Saelix/sencho.git
synced 2026-09-01 13:18:08 +00:00
275c654407d44d6af0dbf4b7a2c8deed3f24695a
24 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Anso
|
275c654407 |
feat(fleet): detect and update from a new sencho-dev:dev build (#1871)
* feat(fleet): add self dev-build detection primitives Split compareLocalToRemoteTag into compareLocalToRemoteTagDetailed (returns the probe's primary digest alongside the match/update/error verdict) with compareLocalToRemoteTag now a thin wrapper, so a caller that needs both the verdict and the digest no longer has to probe the same mutable tag twice. Add detectSelfDevBuildUpdate, which compares the running container's own image against the rolling ghcr.io/studio-saelix/sencho-dev:dev tag using the new detailed comparison, laying the groundwork for surfacing dev-build updates in Fleet. * feat: add isSenchoDevRepository and isSenchoDevFloatingTag predicates Add two pure predicate functions to helpers/selfUpdateCompose.ts for identifying Sencho dev repository references and floating tag variants: - isSenchoDevRepository: checks if a reference is to the ghcr.io/studio-saelix/sencho-dev repository, including digest-pinned and dev-<sha> tag variants - isSenchoDevFloatingTag: checks if a reference is specifically the floating :dev tag on the Sencho dev repository (not digest-pinned, not immutable dev-<sha>) Both functions reuse existing parsing patterns (normalizeImageRepository for repository extraction, classifyImagePin idiom for digest and tag detection) to maintain consistency. Add comprehensive test coverage in self-update-compose.test.ts covering all specified test cases including edge cases (malformed refs, unrelated repos, digest pins, etc.). * feat(gitops): wire dev-build detection into MonitorService Adds a dev_build_update_available notification category and a new checkSenchoDevBuild() cycle in MonitorService that detects when the running container has fallen behind the rolling ghcr.io/studio-saelix/sencho-dev:dev build it is pinned to, using detectSelfDevBuildUpdate() and isSenchoDevFloatingTag(). Availability state is written unconditionally so the Fleet update affordance never depends on notification delivery succeeding, while a separate dedup key prevents re-notifying for a digest already announced. Also guards checkSenchoVersion() so a dev-repo pin no longer produces a false positive stable-release update notification. * feat(fleet): surface dev-image status and build availability Fleet's GET /update-status now reports isDevImage (any reference to the sencho-dev repository, including digest pins) and devBuildUpdateAvailable (the exact floating :dev tag with a newer build observed, read from the system-state key MonitorService already maintains). A dev-pinned local node forces updateAvailable to false and clears any stale stable-release skip, since that skip was computed before image-pin classification and would otherwise leak a bogus "Skipped" state onto a dev row. Made MonitorService's SENCHO_DEV_BUILD_AVAILABLE_KEY constant public so both call sites share one string instead of duplicating it. * fix(fleet): omit targetVersion for a dev-image update trigger updateRequestInit() always forwarded latestVersion (the latest stable release) as targetVersion whenever it was valid semver, even for a dev-pinned node. The backend already ignores targetVersion safely for a floating pin, so this never caused an actual repin, but it produced a misleading "Update to X.Y.Z" button label and confirm-dialog copy for an update that installs the dev image, not that stable release. * feat(fleet): add integration-image badge and dev build update button NodeCard now shows a persistent "Integration image" badge whenever a node's compose image is any sencho-dev reference, independent of update availability, visible to every role. When a newer dev build is available, a solid brand-colored "Update dev build" button appears alongside it, admin-only, reusing the existing update trigger and requireAdmin route. Styled distinctly from the neutral stable "Update to X.Y.Z" button so an operator always knows which channel they're acting on. * feat(fleet): add dev-image copy to the local update confirm dialog LocalUpdateConfirmDialog now recognizes isDevImage and shows a distinct LOCAL - DEV UPDATE kicker plus copy stating the sencho-dev:dev reference will be pulled without rewriting the compose image, and that integration images are unsigned and carry no release attestations. Without this, a dev-pinned node's update confirmation fell through to the generic "Pulls Sencho the latest release" copy. FleetView.tsx threads isDevImage from the node's update status through to the dialog, same source as its other pin fields. * feat(fleet): separate dev and stable availability in the Node Updates sheet The sheet counted stable and dev availability together via the same updateAvailable field, so a dev-pinned node with a build available fell into neither the summary counts nor any row action, and would have misleadingly rendered as "Up to date" once devBuildUpdateAvailable existed. stableAvailable and devAvailable are now tracked separately: the changelog dot lights only from stableAvailable (a dev build has no release changelog), the summary and meta text report the combined total, a dev row shows "Integration build" instead of a stable version in the Latest column, and the existing Update button/badge now also fires for devBuildUpdateAvailable. Update all and Skip stay stable-only, since both already gate on fields a dev row never satisfies. * feat(fleet): bring dev-build detection and update to Mobile Fleet Mobile Fleet previously had no update capability at all: it only polled /fleet/overview and never called useFleetUpdateStatus, so it could not show the stable update flow either. It now fetches update status alongside the overview poll, shows the same "integration" marker as desktop on any dev-pinned node's card (visible to every role), and gives admins a dev-build update action. The action renders as a sibling of the card's own button rather than nested inside it, since the card is itself a <button> and a nested button is invalid HTML with broken touch semantics. It reuses the exact same triggerNodeUpdate/confirmLocalUpdate flow and LocalUpdateConfirmDialog /ReconnectingOverlay components desktop already renders, so there is no parallel API implementation to keep in sync. * feat(notifications): wire dev_build_update_available through the frontend Adds the category to the frontend NotificationCategory union, its bell label, the per-node "mute update notifications" bundle, and the bell's friendly dot-color memo. The changelog navigation and "View changelog" button stay scoped to node_update_available only: a dev build has no release changelog entry to navigate to. * docs: document dev-build detection and update on Fleet Adds the dev_build_update_available notification category, the persistent Integration image marker, and the dev-build update action (desktop and mobile) to the alerts-notifications, verifying-images, fleet-view, remote-updates, and upgrade pages. States the detection cadence explicitly: it polls on a fixed interval and reflects the newest build observed, not necessarily every individual build. * fix(gitops): sanitize the inconclusive-reason debug log for log injection CodeQL flagged the dev-build check's debug log as depending on a user-influenced value (a registry probe failure reason can trace back to external input). Wraps it with sanitizeForLog(), the existing repo-wide remediation for this class of finding, matching how registry-api.ts already handles the same pattern. * test(gitops): cover the no-repin invariant on a dev-build self-update Proves triggerUpdate(), called with neither targetVersion nor targetImageRef (the exact dev-build update call), pulls the current compose-declared ref unchanged and never stages a compose rewrite. * fix(fleet): use the shared busy-button pattern on Mobile Fleet's dev update action Replaces the local Loader2 plus boolean pending logic with BusyButton so busy behavior and interaction locking stay in sync with the rest of the app's async click surfaces. * test(gitops): exercise the production call shape in the no-repin regression Fleet substitutes the stable compare target when the request body omits one, so SelfUpdateService receives a targetVersion even for a dev-build update. The guard that protects a :dev install is therefore the semver check inside the repin branch, not the absence of a target. Drives triggerUpdate with a forwarded target against a floating :dev pin and asserts the reference is pulled unchanged with no staged patch, and pairs it with a semver case so the negative assertions cannot pass vacuously. |
||
|
Anso
|
85842cc547 |
feat: add service-scoped stack alert rules (#1681)
* feat: add service-scoped stack alert rules
Stack alerts can target one Compose service or all services. Breach timers
are per container and cooldowns are per service so a healthy sibling no
longer clears another container's timer or silences a different service.
* fix: gate remote scoped alert creates without losing the body
Remote hops skip JSON parsing so the proxy stream stays pipeable, which
left service_name invisible to the capability gate. Buffer POST /alerts
bodies for inspection, fail closed when the remote lacks the capability,
and rewrite the buffered bytes on forward. Restore alert-panel alt text
to match the unchanged screenshot.
* fix: bound remote alert body buffer and reject encoded JSON
Cap proxied POST /alerts buffering at the local 100KB JSON limit with
structured 413 cleanup, reject non-identity Content-Encoding with 415 so
compressed scoped bodies cannot bypass the mixed-version gate, and cover
oversized, chunked, and gzip regressions.
* fix: harden service-scoped alert delete, cooldown, and proxy gates
Reject non-digit alert ids, dual-write last_fired_at for rollback safety,
gate cooldown on persisted notification history, fail-fast oversized proxy
bodies with 413, and clarify Not in compose UI semantics.
* test: expect dispatchAlert persisted result in crash-safety cases
Update notification-routing assertions for the new { persisted } return
shape so CI matches the cooldown-gating contract.
|
||
|
Anso
|
d8e4ede94f |
fix(notifications): neutralize satellite-local node names in alert bodies (#1640)
* fix(notifications): neutralize satellite-local node names in alert bodies Fleet-aggregated alerts embedded each instance seed name (often Local) while the hub badge already named the remote. Drop identity prefixes and use type-aware local wording so attribution stays on the badge. * fix(docs): correct image-update default check cadence Operator docs still said six-hour polling; the seeded default is two hours in interval mode, and the cadence is configurable or cron-based. * test(notifications): assert hub stamps roster name on neutral remote bodies Cover the fan-in path that attaches hub roster identity while leaving the satellite message body unchanged. |
||
|
Anso
|
678c198faa |
fix(notifications): stop embedding Local in janitor alerts (#1631)
Fleet-aggregated reclaim alerts showed Node Local in the body while the badge already named the remote. Use a node-neutral message so identity comes from the hub badge; leave stored rows unchanged. |
||
|
Anso
|
4079cb9198 |
fix(notifications): align Sencho update alerts with Fleet cache (#1620)
Remove the MonitorService 6-hour version-check cooldown so node_update_available fires when the shared version cache observes a published release, matching the Fleet update button. Dedup and publish-pending gating still prevent spam. |
||
|
Anso
|
10fb93dcb1 |
feat(scheduler): schedule container restart, stop, and start (#1526)
* feat(scheduler): schedule container restart, stop, and start Add container as a scheduled-task target type so operators can automate lifecycle actions against standalone containers by node and name, with matching UI pickers, validation, execution on local and remote nodes, and tests. * fix(scheduler): stack service matching and container picker hygiene Backfill Service on smartFallback containers so per-service stack restarts work when container_name is set. Match services by compose label and container name in stack routes and scheduled restarts. Exclude Sencho from GET /api/containers lists. Hide the Restart Stack service picker when a stack has only one service. * test(scheduler): scope service checkbox assertion to Services block The create dialog also has a Delete after run checkbox. Count checkboxes only inside the Services section so CI does not include unrelated form controls. * fix(scheduler): narrow closest() result to HTMLElement in schedule test The service-checkbox assertion passed an Element from closest() into within(), which requires an HTMLElement, failing tsc -b in the frontend build and Docker build stages. Use the closest<HTMLElement>() type argument so the value type-checks without an unsafe cast. * fix(scheduler): hide Sencho container on remote node picker lists Remote container lists are proxied from peer Sencho instances, so id-only self filtering missed peers on older builds. Await SelfIdentity init, match ImageID, and drop official saelix/sencho images. Apply the same heuristic in the scheduled-operations UI and when the hub fetches remote containers for scheduled runs. * test(monitor): add missing DatabaseService mocks for scan history cleanup * test(scheduler): add missing markStaleScansAsFailed mock SchedulerService.tick() calls db.markStaleScansAsFailed() to sweep stale vulnerability scans. The scheduler-service test was missing this method in its DatabaseService mock, causing TypeError failures during test initialization. Added mockMarkStaleScansAsFailed to hoisted mocks and DatabaseService mock object, returning safe default of 0 scans marked as failed. * test(compose): add missing FileSystemService mocks for getStackContent/getEnvContent * test(containers-route): mock SelfIdentityService to prevent initialize() crash The excludeSelfContainers() helper calls SelfIdentityService.initialize(), which tries to access DockerController. Without a proper SelfIdentityService mock, the initialize() call fails silently, causing a 500 error on GET /api/containers. Added SelfIdentityService mock with initialize(), isOwnContainer(), and isOwnImage() methods to prevent the crash. |
||
|
Anso
|
98667e0d6f | fix: make host memory usage ZFS ARC-aware (#1547) | ||
|
Anso
|
624b586887 |
chore: gate self-update prompts on published registry images (#1519)
GitHub Releases appear before docker-publish.yml finishes pushing images. Probe Docker Hub and GHCR manifests before advertising a version as available. Sanitize registry probe debug logs for CodeQL log-injection. |
||
|
Anso
|
26d557a701 |
feat: purge scan data for deleted images and stacks (#1467)
Vulnerability scan rows were never cleaned up when their image was removed from Docker or their stack was deleted, so the Security Overview (including the Top exploit-risk findings card) kept surfacing findings for artifacts that no longer exist. Scan results now reflect what is still on the host: - Deleting a stack immediately purges its stack:<name> compose-config scan. - A background reconciliation in the monitor janitor removes scans whose image is gone from the node, or whose stack folder no longer exists. It is fail-safe: a scan is only removed when its artifact is positively known to be gone, the Docker image list is read with a timeout (skipped on failure), and stack scans are reconciled only when the stack list is non-empty. - An opt-out "Remove scans for deleted images and stacks" setting (on by default, per-node) lets operators retain scan history for removed artifacts. Scan deletes remove child findings explicitly, since SQLite foreign-key cascade is not enabled on the connection. |
||
|
Anso
|
315e8b6379 |
feat: add node update alerts with changelog tab and skip-version handling (#1463)
* feat: add node update alerts with changelog tab and skip-version handling - Add node_update_available notification category with blue/brand bell dot - Route node_update_available notifications to Fleet -> Node updates sheet - Add Changelog tab to NodeUpdatesSheet with GitHub release notes - Add per-node skip-version persistence (node_update_skips table) - Skip hides update CTA on node card and sheet; re-surfaces on newer version - Skipped nodes excluded from Update all backend filter - Add pulsating dot indicator on Changelog tab when updates available - Always-visible View changelog action in notification row bottom - Admin-only for all mutating controls (skip, unskip, update) - Backend tests for skip-version semantics (15 tests) - Update fleet-view.mdx, remote-updates.mdx, and OpenAPI spec * fix: address audit findings - nested button, stale changelog, semver normalization, mobile intent - Move View changelog button outside routable button (sibling element) - Fix aria-label for node_update_available notification rows - Support ?recheck=true on release-notes endpoint - Invalidate release notes cache on forced recheck - Store normalized semver (semver.valid strips v prefix) - Skip fleetUpdatesIntent on mobile (desktop only) - Add v-prefix normalization test * fix: restore View changelog on same line as timestamp, opposite sides The button is always visible at the bottom right of the notification card, on the same row as the timestamp (just now), using justify-between layout. * fix: update tests for node_update_available category and release-notes fetch - Backend: monitor-service tests now expect node_update_available instead of system - Frontend: NodeUpdatesSheet tests mock release-notes API call to prevent undefined then() * fix: resolve ci lint failures |
||
|
Anso
|
b7dd9dc1b0 |
feat: add ON/OFF toggle for host threshold alerts (#1456)
* feat: add ON/OFF toggle for host threshold alerts Add host_alerts_enabled setting (default ON) as a master switch for CPU, RAM, and disk host threshold evaluation. When OFF, the four threshold controls in Settings > Host Alerts are disabled and MonitorService skips the systeminformation calls and alert dispatch entirely, while clearing stale suppression state so re-enabling starts fresh. The dashboard Configuration Status card shows "Off" when host threshold alerts are disabled. Crash capture, health gate, deploy guardrails, stack alert rules, and the Docker janitor are all unaffected. * fix: exit NumberChip edit mode when externally disabled When the host threshold alerts master toggle is turned OFF while a NumberChip is in edit mode, force-exit edit mode so the chip renders the greyed-out button state consistently with the other chips. |
||
|
Anso
|
42fc8048fe |
fix(metrics): host memory usage excludes reclaimable page cache (#1297)
* fix(metrics): host memory usage excludes reclaimable page cache Host RAM was computed as mem.used / mem.total via systeminformation, but mem.used counts reclaimable buffers/cache as used, so a busy Linux host read ~99%. Switch the dashboard stats, the fleet node self-report, and the host-RAM alert threshold to mem.active / mem.total (cache-excluded), and report used: mem.active and free: mem.available so the byte readout stays consistent with the percentage. Add regression tests for a cache-heavy host (no false alert) and a genuinely busy host (alert still fires). * test(metrics): assert system stats memory excludes reclaimable cache The cached /api/system/stats test mocked si.mem() without active/available, so after the route switched to the cache-excluded fields it produced a NaN percentage that the shape-only assertions did not catch. Fill the mock with a realistic shape and assert the route reports the active working set, not the cache-inclusive used/free. |
||
|
Anso
|
2d56ea958a |
fix(stack-activity): per-stack history integrity, attribution, sanitization (#1228)
* fix(stack-activity): per-stack history integrity, attribution, sanitization Address the Stack Activity audit findings (PR 1 of 2): - Per-stack history integrity: drop the per-insert 100-row prune in addNotificationHistory that evicted quieter stacks' history whenever another stack got chatty. Periodic cleanupOldNotifications now caps per (node, stack) at 500 rows and per-node unattached system events at 1000 rows, on top of the existing 30-day retention. Signature takes an options bag and returns a per-stage summary so MonitorService can log what actually ran each cycle. - Actor attribution: thread req.user?.username through every notifyActionFailure call site and add synthetic actors at service emit sites (system:autoheal, system:scheduler, system:image-update, system:docker-events, system:blueprint, system:monitor, system:policy). The timeline renders system actors as "via <Label>" so an autoheal redeploy is no longer indistinguishable from a user redeploy. - Message sanitization: new sanitizeNotificationMessage at NotificationService.dispatchAlert strips KEY=VALUE pairs whose key ends in TOKEN/KEY/PASSWORD/SECRET/CREDENTIALS/AUTH, scrubs HTTP basic auth in URLs and Bearer tokens, collapses COMPOSE_DIR paths, and truncates to 1000 chars. Applied to the stored history and to every downstream Discord/Slack/webhook channel. The ImageUpdateService recovery-path direct DB write also runs through the sanitizer. - Composite pagination cursor: getStackActivity now accepts a (timestamp, id) cursor (?before=&beforeId=). The legacy timestamp-only form silently dropped events when a single compose up emitted many events sharing one millisecond. Route rejects beforeId without before. - Frontend hardening: distinct error state with retry button (initial fetch failure no longer renders as the genuine empty state), strict positive-integer parsing on cursor params, overrequest-by-1 pagination so the last page does not leave a dead "Load more" click, runtime guard on liveEvents merge that validates the level union, per-minute day-bucket recompute so an open panel does not stay on "Today" past midnight. No tier, role, or capability gate touched. Route permission gate remains stack:read on the named stack. * fix(stack-activity): sanitizer covers lowercase env vars and per-node compose dir External review surfaced two leak paths in the message sanitizer: - The sensitive-key regex was uppercase-only. Compose env names are conventionally uppercase but lowercase forms (db_password, jwt_secret, github_token) are valid and do leak through the same Docker and compose-parse error paths. Make the regex case-insensitive and tighten it to also catch bare TOKEN= / KEY= / PASSWORD= without a prefix word, while still leaving BYPASS, COMPASS, and similar non-secret keys alone. - The compose-dir path collapse only read process.env.COMPOSE_DIR, but the real resolution chain is node.compose_dir (per-node DB override) -> process.env.COMPOSE_DIR -> /app/compose. A node with a custom compose_dir could still leak absolute paths into stored history and downstream channels. Route both the dispatchAlert call and the ImageUpdateService recovery-path direct write through NodeRegistry.getInstance().getComposeDir(localNodeId) so the collapse covers every resolution outcome. Tests now assert lowercase keys are redacted and that BYPASS-style non-secrets stay intact in both cases. notification-routing mock extended to stub the new getComposeDir call. * chore(stack-activity): a11y roles, visibility-aware tick, live-disconnect signal Close three small follow-ups on the per-stack activity timeline: - A11y: each day-group gets role="list" and each event row gets role="listitem" so screen readers traverse the timeline as a list instead of a wall of text. The day-group container also carries an aria-label naming the bucket. - Visibility-aware day-bucket tick: the 60s setInterval that re-derives Today/Yesterday/Earlier now short-circuits when document.hidden, so a backgrounded panel does not re-render every minute for no visible effect. - Live-disconnect signal: useNotifications dispatches a sencho:notifications-connection custom event on WebSocket open and close. The timeline listens and, when explicitly disconnected, shows a one-line "Live updates offline; reconnecting…" hint above the list. The sidebar ticker already surfaces fleet-wide connection state; this adds an in-context cue for users who are focused on a single stack. Stack-name case normalization was considered and rejected: stack names are case-permissive per the isValidStackName validator, and lowercasing on read or write would silently rename or hide a user's "MyApp" stack. * ci(stack-activity): drop unnecessary escape in URL_BASIC_AUTH regex ESLint no-useless-escape errored on \- inside the character class [a-zA-Z0-9+.\-] at notificationMessage.ts:14. Move the dash to the end of the class so it's an unambiguous literal and the escape is no longer required. Behavior is identical; sanitizer tests still pass. * revert(stack-activity): drop unvalidated E2E spec from this PR The spec was committed without ever running against a real Docker daemon, then failed in CI when it ran for the first time: deploy returned 200 but no notification appeared on the activity endpoint within the polling window, suggesting either a deploy-notification race or a node-id resolution mismatch in the CI environment. Backend unit tests (route + composite cursor + sanitizer) and frontend component tests cover the same logic. The E2E spec will land in a dedicated follow-up once it has been authored against a working CI environment. |
||
|
Anso
|
fcff8e9047 |
fix(monitor): collapse repeated host-metric alerts into per-window summary (F-11) (#1175)
* fix(monitor): collapse repeated host-metric alerts into per-window summary (F-11) A host metric over threshold previously dispatched one notification every 5 minutes for the duration of the breach, producing 7+ identical messages in 35 minutes and spamming Discord/Slack routes. Replace the hardcoded 5-minute cooldown for CPU/RAM/disk with a per-metric suppression window (default 60 min, configurable via host_alert_suppression_mins). The first breach fires immediately; subsequent cycles within the window are silently counted; the next dispatch after the window elapses carries a summary suffix listing how many cycles were suppressed and when the breach first crossed threshold. Recovery clears the counter so re-breach fires fresh. The pattern mirrors PolicyEnforcement.notifyTrivyMissingOnce: module-scope Map, in-memory only, in-cycle dedup, with a test-reset helper. The existing system_state row keeps post-restart re-fires bounded. Janitor and per-stack alert rules are unchanged; they already have adequate cadence and per-rule cooldown respectively. * fix(ci): restore backend and frontend checks * fix(e2e): remove create button timing race * fix(e2e): harden create double-click test * fix(monitor): clear persisted F-11 timestamp on recovery + clamp suppression window Independent audit on the previous commit surfaced two issues. 1. clearHostMetricSuppression early-returned on missing in-memory state, leaving a stale system_state.last_host_*_alert_ts row alive after a process restart. Scenario: breach fires + persists timestamp, process restarts, metric recovers before another evaluate cycle re-seeds the in-memory Map, recovery cleanup early-returns. Next re-breach inside the original window hits the restart-survivability branch and is silently suppressed instead of firing fresh. Fix: read persisted state in clearHostMetricSuppression and reset to '0' independently of in-memory presence. The read-before-write also skips redundant writes when the row is already cleared. 2. host_alert_suppression_mins is validated by zod on the bulk PATCH path but the single-key POST /api/settings path accepts allowlisted keys without re-validation. A 999999999-minute value would silence host alerts for centuries. Add MAX_HOST_ALERT_SUPPRESSION_MIN = 1440 mirroring the zod max, and clamp via Math.min in evaluateGlobalSettings. Two new vitest cases (restart-then-recovery-then-rebreach; the 1440 clamp) confirmed failing before the fix, passing after. The existing "metric drop" case updated to use a mock-backed persistence pattern consistent with the new restart-scenario tests. 73/73 monitor-service tests green; full backend suite 2507/2510 (same pre-existing Windows EBUSY flake on filesystem-backup.test.ts as baseline). |
||
|
Anso
|
a51547a158 |
fix(monitor): decouple janitor disk-usage check from 30s cycle (F-6) (#1164)
* fix(monitor): decouple janitor disk-usage check from 30s cycle (F-6) `docker system df` (called by the MonitorService janitor check) can take 30+ seconds on Docker Desktop with many volumes. Running it on the 30s evaluate cycle compounded with the per-container stats fan-out and pushed the cycle to 140s+, blocking subsequent monitoring work. This change: - Moves the janitor disk-usage check into its own 15-minute cycle with a tight 8s timeout. A circuit breaker opens after 3 consecutive timeouts (60-minute cooldown) so a sick daemon stops pinning Dockerode sockets every tick. The first janitor tick is deferred 45 seconds past boot to avoid head-of-line collision with the initial monitor cycle's stats fan-out. - Adds a paired 8s `withTimeout` wrap to the admin prune-estimate routes (`/api/system/prune/estimate` and the dry-run path of `/api/system/prune/system`) so a slow df does not hang the admin tab. Both routes respond 503 with code `docker_df_slow` on timeout. - Factors `withTimeout` and `TimeoutError` into `utils/withTimeout.ts` so the route layer does not have to import from a service module. - Adds 10 unit tests covering the decoupling guardrail, breaker open/close, cooldown, threshold gate, the 100 MB reclaimable floor, re-entrancy, recovery logging, non-timeout error handling, and the full timer-cleanup contract of `stop()`. - Adds 4 integration tests for the prune routes covering the 503 timeout response, the success path, and the non-timeout 5xx path. * fix(fleet,monitor): extend F-6 timeout to fleet prune routes; close breaker-recovery log gap Codex audit findings on PR #1164: Major. The fleet routes that fan out prune-estimate work on local nodes (`POST /api/fleet/labels/fleet-prune` dry-run path and `POST /api/fleet/prune/estimate`) called `estimateSystemReclaim` without a timeout, so a slow local Docker daemon could still hang the fleet admin tab even though the system-maintenance routes were already bounded. Wrap both call sites with the shared `withTimeout(..., 8s)` and surface a "Docker daemon is busy" message via the per-target and per-node error channels the routes already used for other failures. The destructive (non-dry-run) prune path stays unwrapped because it calls `pruneSystem` / `pruneManagedOnly`, not `df`. Minor. The janitor circuit breaker zeroed `janitorConsecutiveTimeouts` when it opened, so a successful call after a full breaker-open cooldown slipped past the `if (counter > 0)` recovery-log branch and never emitted `[Monitor] Janitor disk-usage check recovered`. The operator observability signal was missing exactly when it mattered most. Extend the predicate to also trip on `janitorBreakerUntil > 0` (which stays set to its past timestamp after cooldown until the next success clears it), so recovery logs symmetrically for both partial-failure and post-breaker recovery paths. Added a dedicated test. Three new integration tests cover the fleet routes (timeout, success, and the estimate endpoint's per-node unreachable shape). |
||
|
Anso
|
2dda1bee1f |
perf(monitor): parallelize per-container stats fetch in evaluation cycle (#1045)
The MonitorService.evaluateStackAlerts loop processed containers serially, awaiting one Docker stats call per container. Each Docker stats call inherently waits ~1s for the engine to produce a sample, so cycle time scaled linearly with container count and exceeded the 25s warning threshold on production nodes with ~17+ running containers. Changes: - Extract the per-container body into processContainer(). - Replace the serial for-loop with a bounded-concurrency fan-out (runWithConcurrency, cap of 10) so the cycle runs in roughly max(per-container time) instead of sum. - Add a per-cycle dedup set (firedThisCycle) so a stack rule fires exactly once per cycle even when multiple containers in the stack all breach. Without it, parallel workers race past the cooldown check before any DB write lands and dispatch the same alert N times on first breach. - Tests cover wall-time parallelism, per-container failure isolation, the concurrency cap, and the per-cycle dispatch dedup. |
||
|
Anso
|
44dba59cab |
feat(notifications): add structured category enum to dispatcher and history (#774)
Introduce a NotificationCategory string-literal union (11 values) and thread it through dispatchAlert as a required second argument. All callers (DockerEventService, AutoHealService, ImageUpdateService, MonitorService, PolicyEnforcement, policyGate, SchedulerService, imageUpdates route) pass an explicit category at every call site, giving TypeScript compile-time enforcement that no new emit site can be added without choosing a category. DatabaseService gains an idempotent migration that adds a nullable category TEXT column to notification_history; existing rows keep category=NULL (displayed as Uncategorized in the UI). The getNotificationHistory method accepts an optional category filter that is forwarded from the GET /api/notifications/history route via a ?category= query param. NotificationPanel gains a category Select dropdown so users can filter history by category. The frontend types mirror the backend union so API responses are type-safe end-to-end. All 75 test files (1410 tests) updated to the new 4-arg dispatchAlert signature and passing. |
||
|
Anso
|
6b8c369745 |
fix(notifications): stop Sencho version notifications from silently skipping (#594)
* fix(notifications): stop Sencho version notifications from silently skipping Three independent defects combined to make version-update notifications silently fail even while Fleet overview correctly surfaced an update button: - The in-memory 6-hour cooldown was advanced before the network fetch, so a single transient failure at boot could lock the check for the rest of the container lifetime. Moved the cooldown update inside the success branch so failures retry on the next eval cycle. - MonitorService called the raw version fetch directly, bypassing the CacheService wrapper (TTL, inflight dedup, stale-on-error) that Fleet uses, so the two paths could diverge. Unified both on a shared getLatestVersion() helper in utils/version-check.ts. - The dedup key could carry stale state from a previous build and never self-clear. It now self-heals when the running version reaches the previously-notified version, so future releases re-fire as expected. Added diagnostic logs gated on debug mode for each skip branch, plus three regression tests covering cooldown-on-failure, cooldown-on-success, and dedup self-heal. * docs(notifications): drop legacy-upgrade framing from alerts troubleshooting Sencho has not shipped publicly, so troubleshooting entries written in 'this used to happen but now does Y' mode reference a past that does not exist for any reader. Rewrote the version-notification, image-update, and crash-alert troubleshooting entries to describe current behavior positively without referring to prior builds, upgrade paths, or legacy fixes. * chore(security): accept CVE-2026-33810 in bundled Docker CLI 29.4.0 Trivy now flags CVE-2026-33810 (Go stdlib crypto/x509 DNS constraint bypass, fixed in Go 1.26.2) in the Docker CLI static binary we ship. Docker CLI 29.4.0 is the latest upstream release and still links Go 1.26.1; no newer static binary exists yet. Same exposure profile as the already-accepted CVE-2026-32280: the Docker CLI and compose plugin only validate certificates from well-known registry CAs and the local Docker socket, not from attacker-controlled CAs with crafted DNS name constraints. Revisit on the next Docker CLI release that rebuilds against Go 1.26.2 or later. |
||
|
Anso
|
ad9a6859e6 |
fix(notifications): replace polling with Docker event stream for container lifecycle detection (#588)
* fix(notifications): replace polling with Docker event stream for container lifecycle detection Replaces the 30-second MonitorService crash-detection poll with a causal, per-node Docker events stream. Eliminates false crash alerts on intentional stops (docker stop, compose down, stack restart/update), detects OOM kills as a distinct alert category, and surfaces real crashes in real time. A new DockerEventManager spawns one DockerEventService per local node. Each service consumes the filtered container event stream, classifies die events against recent kill/oom state, and reconciles container state via snapshot diffing on connect and reconnect. Rate limiting, exponential backoff with jitter, and parse-error tolerance keep the stream resilient under load and during daemon interruptions. MonitorService retains host limits, janitor, version check, and stack metric alerts; crash and healthcheck detection move out entirely. * fix(tests): silence require-imports lint in hoisted mock factory |
||
|
Anso
|
4a0319331a |
fix(notifications): resolve version notification showing 0.0.0 and backfill missing image update notifications (#586)
- Read running Sencho version from the packaged manifest via getSenchoVersion() instead of process.env.npm_package_version, which is undefined when launched via node dist/index.js (production Docker). Skip the update check entirely when the version cannot be resolved. - Add a one-time backfill pass in ImageUpdateService so users who upgraded to a Sencho version with the notification pipeline receive a catch-up entry for stacks already flagged as having updates before the upgrade. - Surface dispatch failures as error-level entries in the in-app notification bell via a direct notification_history write, so misconfigured webhooks are visible without tailing logs. - Extend test coverage for both paths: mock getSenchoVersion (including the null-version case) and add dispatch-path tests for transition, no-re-fire, backfill, and error surfacing. - Expand alerts-notifications docs with per-instance 6-hour check cadence, an example version message, a backfill note, and three new troubleshooting entries. |
||
|
Anso
|
0a94df318a |
fix(alerts): harden with security fixes, design compliance, and test coverage (#570)
* fix(alerts): harden with security fixes, design compliance, and test coverage Add authMiddleware to all alert endpoints, validate notification test dispatch inputs, fix restart_count metric via Docker inspect, correct network metric units, replace any types with DockerContainerStats interface, add webhook timeouts and dispatch error tracking. Frontend: migrate Select to Combobox, add ScrollArea and delete confirmation AlertDialog, fix icon strokeWidth to 1.5. Add update availability notifications for both Sencho version updates (6-hour check in MonitorService) and stack image updates (state transition detection in ImageUpdateService). Extract shared version fetch logic into utils/version-check.ts. Add diagnostic logging gated behind developer_mode for MonitorService breach state machine and NotificationService dispatch routing. Tests: 24 new alert API integration tests, restart_count and version check unit tests (688 total passing). Docs updated with HTTPS requirement, update notifications section, and troubleshooting guide. * fix(alerts): remove unused TEST_USERNAME import in alerts-api tests |
||
|
Anso
|
9db97107aa |
fix(dashboard): harden real-time dashboard with bug fixes and design compliance (#517)
* fix(dashboard): harden real-time dashboard with bug fixes and design compliance - Fix host alert spam: add 5-minute cooldown for CPU/RAM/disk threshold alerts, preventing duplicate notifications every 30s during sustained breaches. Extract shared dispatchWithCooldown helper (also used by Docker janitor alerts). - Fix memory metric inflation: subtract filesystem cache from stored memory_mb values, matching the existing calculateMemoryPercent logic. - Fix crash detection reliability: replace fragile 'seconds ago' string matching with a tracked Set of alerted container IDs. Containers are only alerted once per crash event, with automatic cleanup when they start running again or after a 1-hour TTL. - Fix health status bar: exited containers now trigger 'degraded' state independently of unread error notifications. - Fix CPU chart Y-axis: auto-scale when aggregate container CPU exceeds 100% instead of silently clipping at the hardcoded domain ceiling. - Fix grammar: 'actives' to 'active' in container count label. - Add shadow-card-bevel to all dashboard cards per design system. - Update dashboard docs to reflect revised health status thresholds. * test(dashboard): update monitor service tests for new alert signatures - Add container Id fields to crash detection test fixtures - Update host alert assertions to match dispatchWithCooldown 3-arg call - Fix unhealthy container test to use State: 'unhealthy' instead of State: 'running' (running containers are now skipped in crash detect) |
||
|
Anso
|
1b573f542a |
feat(notifications): add shared notification routing rules (Admiral tier) (#347)
Route stack alerts to specific Discord, Slack, or webhook channels instead of the single global endpoint. Includes per-rule enable/disable, priority ordering, and automatic fallback to global agents when no rule matches. - Add notification_routes table, interface, and CRUD in DatabaseService - Add routing logic in NotificationService.dispatchAlert with optional stackName - Pass stack context from MonitorService (crash/health) and SchedulerService - Add 5 API endpoints gated with requireAdmin + requireAdmiral - Add NotificationRoutingSection UI with Combobox stack picker, channel tabs - Parallel webhook dispatch via Promise.allSettled - 10 unit tests covering routing, fallback, and edge cases - Documentation with screenshots at docs/features/notification-routing.mdx |
||
|
Anso
|
93ae147ec1 |
test: expand backend test coverage for stability gaps (#329)
Add 5 new test files (116 test cases) targeting previously untested service-layer logic: - database-metrics: metrics CRUD, minute-bucket aggregation, cleanup retention, notification auto-cap at 100, stack alerts CRUD, and stress tests with 1000+ metrics - monitor-service: CPU/memory/network calculation helpers, all 6 alert condition operators, breach state machine lifecycle, global crash detection, host limit thresholds, cleanup delegation, and isProcessing concurrency guard - docker-controller: validateApiData error detection, state-safe start/stop (304 handling), batch container removal with partial failures, disk usage calculation, resource classification (managed/unmanaged/system), orphan detection, and daemon unreachable error propagation - scheduler-service: cron parsing, license tier gating, concurrent task prevention via runningTasks Set, manual trigger, all 4 task types (restart/snapshot/prune/update), wildcard targets, error recording, recovery notifications, and cleanup - compose-service: subprocess spawn/exit handling, WebSocket output, deploy with health probe and atomic rollback, registry auth temp dir lifecycle, and downStack teardown |