Commit Graph

21 Commits

Author SHA1 Message Date
Anso 678c198faa fix(notifications): stop embedding Local in janitor alerts (#1631)
Fleet-aggregated reclaim alerts showed Node Local in the body while the
badge already named the remote. Use a node-neutral message so identity
comes from the hub badge; leave stored rows unchanged.
2026-07-14 18:28:01 -04:00
Anso 4079cb9198 fix(notifications): align Sencho update alerts with Fleet cache (#1620)
Remove the MonitorService 6-hour version-check cooldown so node_update_available
fires when the shared version cache observes a published release, matching the
Fleet update button. Dedup and publish-pending gating still prevent spam.
2026-07-14 12:20:25 -04:00
Anso 10fb93dcb1 feat(scheduler): schedule container restart, stop, and start (#1526)
* feat(scheduler): schedule container restart, stop, and start

Add container as a scheduled-task target type so operators can automate lifecycle actions against standalone containers by node and name, with matching UI pickers, validation, execution on local and remote nodes, and tests.

* fix(scheduler): stack service matching and container picker hygiene

Backfill Service on smartFallback containers so per-service stack restarts work when container_name is set. Match services by compose label and container name in stack routes and scheduled restarts. Exclude Sencho from GET /api/containers lists. Hide the Restart Stack service picker when a stack has only one service.

* test(scheduler): scope service checkbox assertion to Services block

The create dialog also has a Delete after run checkbox. Count checkboxes only inside the Services section so CI does not include unrelated form controls.

* fix(scheduler): narrow closest() result to HTMLElement in schedule test

The service-checkbox assertion passed an Element from closest() into
within(), which requires an HTMLElement, failing tsc -b in the frontend
build and Docker build stages. Use the closest<HTMLElement>() type
argument so the value type-checks without an unsafe cast.

* fix(scheduler): hide Sencho container on remote node picker lists

Remote container lists are proxied from peer Sencho instances, so id-only self filtering missed peers on older builds. Await SelfIdentity init, match ImageID, and drop official saelix/sencho images. Apply the same heuristic in the scheduled-operations UI and when the hub fetches remote containers for scheduled runs.

* test(monitor): add missing DatabaseService mocks for scan history cleanup

* test(scheduler): add missing markStaleScansAsFailed mock

SchedulerService.tick() calls db.markStaleScansAsFailed() to sweep stale
vulnerability scans. The scheduler-service test was missing this method in
its DatabaseService mock, causing TypeError failures during test initialization.

Added mockMarkStaleScansAsFailed to hoisted mocks and DatabaseService mock
object, returning safe default of 0 scans marked as failed.

* test(compose): add missing FileSystemService mocks for getStackContent/getEnvContent

* test(containers-route): mock SelfIdentityService to prevent initialize() crash

The excludeSelfContainers() helper calls SelfIdentityService.initialize(), which tries to access DockerController. Without a proper SelfIdentityService mock, the initialize() call fails silently, causing a 500 error on GET /api/containers.

Added SelfIdentityService mock with initialize(), isOwnContainer(), and isOwnImage() methods to prevent the crash.
2026-07-02 22:31:29 -04:00
Anso 98667e0d6f fix: make host memory usage ZFS ARC-aware (#1547) 2026-07-01 23:08:49 -04:00
Anso 624b586887 chore: gate self-update prompts on published registry images (#1519)
GitHub Releases appear before docker-publish.yml finishes pushing images.

Probe Docker Hub and GHCR manifests before advertising a version as available.

Sanitize registry probe debug logs for CodeQL log-injection.
2026-06-29 16:29:56 -04:00
Anso 26d557a701 feat: purge scan data for deleted images and stacks (#1467)
Vulnerability scan rows were never cleaned up when their image was removed
from Docker or their stack was deleted, so the Security Overview (including
the Top exploit-risk findings card) kept surfacing findings for artifacts that
no longer exist.

Scan results now reflect what is still on the host:

- Deleting a stack immediately purges its stack:<name> compose-config scan.
- A background reconciliation in the monitor janitor removes scans whose image
  is gone from the node, or whose stack folder no longer exists. It is
  fail-safe: a scan is only removed when its artifact is positively known to be
  gone, the Docker image list is read with a timeout (skipped on failure), and
  stack scans are reconciled only when the stack list is non-empty.
- An opt-out "Remove scans for deleted images and stacks" setting (on by
  default, per-node) lets operators retain scan history for removed artifacts.

Scan deletes remove child findings explicitly, since SQLite foreign-key cascade
is not enabled on the connection.
2026-06-26 11:59:37 -04:00
Anso 315e8b6379 feat: add node update alerts with changelog tab and skip-version handling (#1463)
* feat: add node update alerts with changelog tab and skip-version handling

- Add node_update_available notification category with blue/brand bell dot
- Route node_update_available notifications to Fleet -> Node updates sheet
- Add Changelog tab to NodeUpdatesSheet with GitHub release notes
- Add per-node skip-version persistence (node_update_skips table)
- Skip hides update CTA on node card and sheet; re-surfaces on newer version
- Skipped nodes excluded from Update all backend filter
- Add pulsating dot indicator on Changelog tab when updates available
- Always-visible View changelog action in notification row bottom
- Admin-only for all mutating controls (skip, unskip, update)
- Backend tests for skip-version semantics (15 tests)
- Update fleet-view.mdx, remote-updates.mdx, and OpenAPI spec

* fix: address audit findings - nested button, stale changelog, semver normalization, mobile intent

- Move View changelog button outside routable button (sibling element)
- Fix aria-label for node_update_available notification rows
- Support ?recheck=true on release-notes endpoint
- Invalidate release notes cache on forced recheck
- Store normalized semver (semver.valid strips v prefix)
- Skip fleetUpdatesIntent on mobile (desktop only)
- Add v-prefix normalization test

* fix: restore View changelog on same line as timestamp, opposite sides

The button is always visible at the bottom right of the notification card,
on the same row as the timestamp (just now), using justify-between layout.

* fix: update tests for node_update_available category and release-notes fetch

- Backend: monitor-service tests now expect node_update_available instead of system
- Frontend: NodeUpdatesSheet tests mock release-notes API call to prevent undefined then()

* fix: resolve ci lint failures
2026-06-26 00:07:51 -04:00
Anso b7dd9dc1b0 feat: add ON/OFF toggle for host threshold alerts (#1456)
* feat: add ON/OFF toggle for host threshold alerts

Add host_alerts_enabled setting (default ON) as a master switch for CPU,
RAM, and disk host threshold evaluation. When OFF, the four threshold
controls in Settings > Host Alerts are disabled and MonitorService skips
the systeminformation calls and alert dispatch entirely, while clearing
stale suppression state so re-enabling starts fresh.

The dashboard Configuration Status card shows "Off" when host threshold
alerts are disabled. Crash capture, health gate, deploy guardrails,
stack alert rules, and the Docker janitor are all unaffected.

* fix: exit NumberChip edit mode when externally disabled

When the host threshold alerts master toggle is turned OFF while a
NumberChip is in edit mode, force-exit edit mode so the chip renders
the greyed-out button state consistently with the other chips.
2026-06-25 16:05:16 -04:00
Anso 42fc8048fe fix(metrics): host memory usage excludes reclaimable page cache (#1297)
* fix(metrics): host memory usage excludes reclaimable page cache

Host RAM was computed as mem.used / mem.total via systeminformation, but mem.used counts
reclaimable buffers/cache as used, so a busy Linux host read ~99%. Switch the dashboard stats,
the fleet node self-report, and the host-RAM alert threshold to mem.active / mem.total
(cache-excluded), and report used: mem.active and free: mem.available so the byte readout stays
consistent with the percentage. Add regression tests for a cache-heavy host (no false alert) and
a genuinely busy host (alert still fires).

* test(metrics): assert system stats memory excludes reclaimable cache

The cached /api/system/stats test mocked si.mem() without active/available, so after
the route switched to the cache-excluded fields it produced a NaN percentage that the
shape-only assertions did not catch. Fill the mock with a realistic shape and assert the
route reports the active working set, not the cache-inclusive used/free.
2026-06-03 16:11:00 -04:00
Anso 2d56ea958a fix(stack-activity): per-stack history integrity, attribution, sanitization (#1228)
* fix(stack-activity): per-stack history integrity, attribution, sanitization

Address the Stack Activity audit findings (PR 1 of 2):

- Per-stack history integrity: drop the per-insert 100-row prune in
  addNotificationHistory that evicted quieter stacks' history whenever
  another stack got chatty. Periodic cleanupOldNotifications now caps
  per (node, stack) at 500 rows and per-node unattached system events
  at 1000 rows, on top of the existing 30-day retention. Signature
  takes an options bag and returns a per-stage summary so MonitorService
  can log what actually ran each cycle.

- Actor attribution: thread req.user?.username through every
  notifyActionFailure call site and add synthetic actors at service
  emit sites (system:autoheal, system:scheduler, system:image-update,
  system:docker-events, system:blueprint, system:monitor, system:policy).
  The timeline renders system actors as "via <Label>" so an autoheal
  redeploy is no longer indistinguishable from a user redeploy.

- Message sanitization: new sanitizeNotificationMessage at
  NotificationService.dispatchAlert strips KEY=VALUE pairs whose key
  ends in TOKEN/KEY/PASSWORD/SECRET/CREDENTIALS/AUTH, scrubs HTTP basic
  auth in URLs and Bearer tokens, collapses COMPOSE_DIR paths, and
  truncates to 1000 chars. Applied to the stored history and to every
  downstream Discord/Slack/webhook channel. The ImageUpdateService
  recovery-path direct DB write also runs through the sanitizer.

- Composite pagination cursor: getStackActivity now accepts a
  (timestamp, id) cursor (?before=&beforeId=). The legacy timestamp-only
  form silently dropped events when a single compose up emitted many
  events sharing one millisecond. Route rejects beforeId without before.

- Frontend hardening: distinct error state with retry button (initial
  fetch failure no longer renders as the genuine empty state), strict
  positive-integer parsing on cursor params, overrequest-by-1 pagination
  so the last page does not leave a dead "Load more" click, runtime
  guard on liveEvents merge that validates the level union, per-minute
  day-bucket recompute so an open panel does not stay on "Today" past
  midnight.

No tier, role, or capability gate touched. Route permission gate
remains stack:read on the named stack.

* fix(stack-activity): sanitizer covers lowercase env vars and per-node compose dir

External review surfaced two leak paths in the message sanitizer:

- The sensitive-key regex was uppercase-only. Compose env names are
  conventionally uppercase but lowercase forms (db_password, jwt_secret,
  github_token) are valid and do leak through the same Docker and
  compose-parse error paths. Make the regex case-insensitive and tighten
  it to also catch bare TOKEN= / KEY= / PASSWORD= without a prefix word,
  while still leaving BYPASS, COMPASS, and similar non-secret keys alone.

- The compose-dir path collapse only read process.env.COMPOSE_DIR, but
  the real resolution chain is node.compose_dir (per-node DB override)
  -> process.env.COMPOSE_DIR -> /app/compose. A node with a custom
  compose_dir could still leak absolute paths into stored history and
  downstream channels. Route both the dispatchAlert call and the
  ImageUpdateService recovery-path direct write through
  NodeRegistry.getInstance().getComposeDir(localNodeId) so the
  collapse covers every resolution outcome.

Tests now assert lowercase keys are redacted and that BYPASS-style
non-secrets stay intact in both cases. notification-routing mock
extended to stub the new getComposeDir call.

* chore(stack-activity): a11y roles, visibility-aware tick, live-disconnect signal

Close three small follow-ups on the per-stack activity timeline:

- A11y: each day-group gets role="list" and each event row gets
  role="listitem" so screen readers traverse the timeline as a list
  instead of a wall of text. The day-group container also carries an
  aria-label naming the bucket.

- Visibility-aware day-bucket tick: the 60s setInterval that re-derives
  Today/Yesterday/Earlier now short-circuits when document.hidden, so a
  backgrounded panel does not re-render every minute for no visible
  effect.

- Live-disconnect signal: useNotifications dispatches a
  sencho:notifications-connection custom event on WebSocket open and
  close. The timeline listens and, when explicitly disconnected, shows
  a one-line "Live updates offline; reconnecting…" hint above the list.
  The sidebar ticker already surfaces fleet-wide connection state; this
  adds an in-context cue for users who are focused on a single stack.

Stack-name case normalization was considered and rejected: stack names
are case-permissive per the isValidStackName validator, and lowercasing
on read or write would silently rename or hide a user's "MyApp" stack.

* ci(stack-activity): drop unnecessary escape in URL_BASIC_AUTH regex

ESLint no-useless-escape errored on \- inside the character class
[a-zA-Z0-9+.\-] at notificationMessage.ts:14. Move the dash to the
end of the class so it's an unambiguous literal and the escape is no
longer required. Behavior is identical; sanitizer tests still pass.

* revert(stack-activity): drop unvalidated E2E spec from this PR

The spec was committed without ever running against a real Docker
daemon, then failed in CI when it ran for the first time: deploy
returned 200 but no notification appeared on the activity endpoint
within the polling window, suggesting either a deploy-notification
race or a node-id resolution mismatch in the CI environment.

Backend unit tests (route + composite cursor + sanitizer) and
frontend component tests cover the same logic. The E2E spec will
land in a dedicated follow-up once it has been authored against a
working CI environment.
2026-05-25 21:09:00 -04:00
Anso fcff8e9047 fix(monitor): collapse repeated host-metric alerts into per-window summary (F-11) (#1175)
* fix(monitor): collapse repeated host-metric alerts into per-window summary (F-11)

A host metric over threshold previously dispatched one notification every 5
minutes for the duration of the breach, producing 7+ identical messages
in 35 minutes and spamming Discord/Slack routes. Replace the hardcoded
5-minute cooldown for CPU/RAM/disk with a per-metric suppression window
(default 60 min, configurable via host_alert_suppression_mins). The first
breach fires immediately; subsequent cycles within the window are silently
counted; the next dispatch after the window elapses carries a summary
suffix listing how many cycles were suppressed and when the breach first
crossed threshold. Recovery clears the counter so re-breach fires fresh.

The pattern mirrors PolicyEnforcement.notifyTrivyMissingOnce: module-scope
Map, in-memory only, in-cycle dedup, with a test-reset helper. The
existing system_state row keeps post-restart re-fires bounded.

Janitor and per-stack alert rules are unchanged; they already have
adequate cadence and per-rule cooldown respectively.

* fix(ci): restore backend and frontend checks

* fix(e2e): remove create button timing race

* fix(e2e): harden create double-click test

* fix(monitor): clear persisted F-11 timestamp on recovery + clamp suppression window

Independent audit on the previous commit surfaced two issues.

1. clearHostMetricSuppression early-returned on missing in-memory state,
   leaving a stale system_state.last_host_*_alert_ts row alive after a
   process restart. Scenario: breach fires + persists timestamp, process
   restarts, metric recovers before another evaluate cycle re-seeds the
   in-memory Map, recovery cleanup early-returns. Next re-breach inside
   the original window hits the restart-survivability branch and is
   silently suppressed instead of firing fresh. Fix: read persisted
   state in clearHostMetricSuppression and reset to '0' independently
   of in-memory presence. The read-before-write also skips redundant
   writes when the row is already cleared.

2. host_alert_suppression_mins is validated by zod on the bulk PATCH
   path but the single-key POST /api/settings path accepts allowlisted
   keys without re-validation. A 999999999-minute value would silence
   host alerts for centuries. Add MAX_HOST_ALERT_SUPPRESSION_MIN = 1440
   mirroring the zod max, and clamp via Math.min in evaluateGlobalSettings.

Two new vitest cases (restart-then-recovery-then-rebreach; the 1440
clamp) confirmed failing before the fix, passing after. The existing
"metric drop" case updated to use a mock-backed persistence pattern
consistent with the new restart-scenario tests. 73/73 monitor-service
tests green; full backend suite 2507/2510 (same pre-existing Windows
EBUSY flake on filesystem-backup.test.ts as baseline).
2026-05-23 06:28:01 -04:00
Anso a51547a158 fix(monitor): decouple janitor disk-usage check from 30s cycle (F-6) (#1164)
* fix(monitor): decouple janitor disk-usage check from 30s cycle (F-6)

`docker system df` (called by the MonitorService janitor check) can take
30+ seconds on Docker Desktop with many volumes. Running it on the 30s
evaluate cycle compounded with the per-container stats fan-out and pushed
the cycle to 140s+, blocking subsequent monitoring work.

This change:

- Moves the janitor disk-usage check into its own 15-minute cycle with
  a tight 8s timeout. A circuit breaker opens after 3 consecutive
  timeouts (60-minute cooldown) so a sick daemon stops pinning Dockerode
  sockets every tick. The first janitor tick is deferred 45 seconds past
  boot to avoid head-of-line collision with the initial monitor cycle's
  stats fan-out.
- Adds a paired 8s `withTimeout` wrap to the admin prune-estimate routes
  (`/api/system/prune/estimate` and the dry-run path of
  `/api/system/prune/system`) so a slow df does not hang the admin tab.
  Both routes respond 503 with code `docker_df_slow` on timeout.
- Factors `withTimeout` and `TimeoutError` into `utils/withTimeout.ts`
  so the route layer does not have to import from a service module.
- Adds 10 unit tests covering the decoupling guardrail, breaker
  open/close, cooldown, threshold gate, the 100 MB reclaimable floor,
  re-entrancy, recovery logging, non-timeout error handling, and the
  full timer-cleanup contract of `stop()`.
- Adds 4 integration tests for the prune routes covering the 503
  timeout response, the success path, and the non-timeout 5xx path.

* fix(fleet,monitor): extend F-6 timeout to fleet prune routes; close breaker-recovery log gap

Codex audit findings on PR #1164:

Major. The fleet routes that fan out prune-estimate work on local nodes
(`POST /api/fleet/labels/fleet-prune` dry-run path and
`POST /api/fleet/prune/estimate`) called `estimateSystemReclaim` without
a timeout, so a slow local Docker daemon could still hang the fleet
admin tab even though the system-maintenance routes were already
bounded. Wrap both call sites with the shared `withTimeout(..., 8s)`
and surface a "Docker daemon is busy" message via the per-target and
per-node error channels the routes already used for other failures.
The destructive (non-dry-run) prune path stays unwrapped because it
calls `pruneSystem` / `pruneManagedOnly`, not `df`.

Minor. The janitor circuit breaker zeroed `janitorConsecutiveTimeouts`
when it opened, so a successful call after a full breaker-open cooldown
slipped past the `if (counter > 0)` recovery-log branch and never
emitted `[Monitor] Janitor disk-usage check recovered`. The operator
observability signal was missing exactly when it mattered most.
Extend the predicate to also trip on `janitorBreakerUntil > 0` (which
stays set to its past timestamp after cooldown until the next success
clears it), so recovery logs symmetrically for both partial-failure
and post-breaker recovery paths. Added a dedicated test.

Three new integration tests cover the fleet routes (timeout, success,
and the estimate endpoint's per-node unreachable shape).
2026-05-23 00:04:05 -04:00
Anso 2dda1bee1f perf(monitor): parallelize per-container stats fetch in evaluation cycle (#1045)
The MonitorService.evaluateStackAlerts loop processed containers serially,
awaiting one Docker stats call per container. Each Docker stats call
inherently waits ~1s for the engine to produce a sample, so cycle time
scaled linearly with container count and exceeded the 25s warning
threshold on production nodes with ~17+ running containers.

Changes:
- Extract the per-container body into processContainer().
- Replace the serial for-loop with a bounded-concurrency fan-out
  (runWithConcurrency, cap of 10) so the cycle runs in roughly
  max(per-container time) instead of sum.
- Add a per-cycle dedup set (firedThisCycle) so a stack rule fires
  exactly once per cycle even when multiple containers in the stack
  all breach. Without it, parallel workers race past the cooldown
  check before any DB write lands and dispatch the same alert N
  times on first breach.
- Tests cover wall-time parallelism, per-container failure isolation,
  the concurrency cap, and the per-cycle dispatch dedup.
2026-05-14 10:20:49 -04:00
Anso 44dba59cab feat(notifications): add structured category enum to dispatcher and history (#774)
Introduce a NotificationCategory string-literal union (11 values) and
thread it through dispatchAlert as a required second argument. All
callers (DockerEventService, AutoHealService, ImageUpdateService,
MonitorService, PolicyEnforcement, policyGate, SchedulerService,
imageUpdates route) pass an explicit category at every call site,
giving TypeScript compile-time enforcement that no new emit site can
be added without choosing a category.

DatabaseService gains an idempotent migration that adds a nullable
category TEXT column to notification_history; existing rows keep
category=NULL (displayed as Uncategorized in the UI). The
getNotificationHistory method accepts an optional category filter
that is forwarded from the GET /api/notifications/history route via
a ?category= query param.

NotificationPanel gains a category Select dropdown so users can
filter history by category. The frontend types mirror the backend
union so API responses are type-safe end-to-end.

All 75 test files (1410 tests) updated to the new 4-arg dispatchAlert
signature and passing.
2026-04-25 13:55:07 -04:00
Anso 6b8c369745 fix(notifications): stop Sencho version notifications from silently skipping (#594)
* fix(notifications): stop Sencho version notifications from silently skipping

Three independent defects combined to make version-update notifications
silently fail even while Fleet overview correctly surfaced an update button:

- The in-memory 6-hour cooldown was advanced before the network fetch,
  so a single transient failure at boot could lock the check for the
  rest of the container lifetime. Moved the cooldown update inside the
  success branch so failures retry on the next eval cycle.
- MonitorService called the raw version fetch directly, bypassing the
  CacheService wrapper (TTL, inflight dedup, stale-on-error) that Fleet
  uses, so the two paths could diverge. Unified both on a shared
  getLatestVersion() helper in utils/version-check.ts.
- The dedup key could carry stale state from a previous build and never
  self-clear. It now self-heals when the running version reaches the
  previously-notified version, so future releases re-fire as expected.

Added diagnostic logs gated on debug mode for each skip branch, plus
three regression tests covering cooldown-on-failure, cooldown-on-success,
and dedup self-heal.

* docs(notifications): drop legacy-upgrade framing from alerts troubleshooting

Sencho has not shipped publicly, so troubleshooting entries written in
'this used to happen but now does Y' mode reference a past that does not
exist for any reader. Rewrote the version-notification, image-update,
and crash-alert troubleshooting entries to describe current behavior
positively without referring to prior builds, upgrade paths, or legacy
fixes.

* chore(security): accept CVE-2026-33810 in bundled Docker CLI 29.4.0

Trivy now flags CVE-2026-33810 (Go stdlib crypto/x509 DNS constraint
bypass, fixed in Go 1.26.2) in the Docker CLI static binary we ship.
Docker CLI 29.4.0 is the latest upstream release and still links Go
1.26.1; no newer static binary exists yet.

Same exposure profile as the already-accepted CVE-2026-32280: the
Docker CLI and compose plugin only validate certificates from
well-known registry CAs and the local Docker socket, not from
attacker-controlled CAs with crafted DNS name constraints. Revisit
on the next Docker CLI release that rebuilds against Go 1.26.2 or
later.
2026-04-14 16:34:28 -04:00
Anso ad9a6859e6 fix(notifications): replace polling with Docker event stream for container lifecycle detection (#588)
* fix(notifications): replace polling with Docker event stream for container lifecycle detection

Replaces the 30-second MonitorService crash-detection poll with a causal,
per-node Docker events stream. Eliminates false crash alerts on intentional
stops (docker stop, compose down, stack restart/update), detects OOM kills
as a distinct alert category, and surfaces real crashes in real time.

A new DockerEventManager spawns one DockerEventService per local node. Each
service consumes the filtered container event stream, classifies die events
against recent kill/oom state, and reconciles container state via snapshot
diffing on connect and reconnect. Rate limiting, exponential backoff with
jitter, and parse-error tolerance keep the stream resilient under load and
during daemon interruptions.

MonitorService retains host limits, janitor, version check, and stack metric
alerts; crash and healthcheck detection move out entirely.

* fix(tests): silence require-imports lint in hoisted mock factory
2026-04-14 13:47:48 -04:00
Anso 4a0319331a fix(notifications): resolve version notification showing 0.0.0 and backfill missing image update notifications (#586)
- Read running Sencho version from the packaged manifest via getSenchoVersion() instead of process.env.npm_package_version, which is undefined when launched via node dist/index.js (production Docker). Skip the update check entirely when the version cannot be resolved.
- Add a one-time backfill pass in ImageUpdateService so users who upgraded to a Sencho version with the notification pipeline receive a catch-up entry for stacks already flagged as having updates before the upgrade.
- Surface dispatch failures as error-level entries in the in-app notification bell via a direct notification_history write, so misconfigured webhooks are visible without tailing logs.
- Extend test coverage for both paths: mock getSenchoVersion (including the null-version case) and add dispatch-path tests for transition, no-re-fire, backfill, and error surfacing.
- Expand alerts-notifications docs with per-instance 6-hour check cadence, an example version message, a backfill note, and three new troubleshooting entries.
2026-04-14 11:51:30 -04:00
Anso 0a94df318a fix(alerts): harden with security fixes, design compliance, and test coverage (#570)
* fix(alerts): harden with security fixes, design compliance, and test coverage

Add authMiddleware to all alert endpoints, validate notification test
dispatch inputs, fix restart_count metric via Docker inspect, correct
network metric units, replace any types with DockerContainerStats
interface, add webhook timeouts and dispatch error tracking.

Frontend: migrate Select to Combobox, add ScrollArea and delete
confirmation AlertDialog, fix icon strokeWidth to 1.5.

Add update availability notifications for both Sencho version updates
(6-hour check in MonitorService) and stack image updates (state
transition detection in ImageUpdateService). Extract shared version
fetch logic into utils/version-check.ts.

Add diagnostic logging gated behind developer_mode for MonitorService
breach state machine and NotificationService dispatch routing.

Tests: 24 new alert API integration tests, restart_count and version
check unit tests (688 total passing). Docs updated with HTTPS
requirement, update notifications section, and troubleshooting guide.

* fix(alerts): remove unused TEST_USERNAME import in alerts-api tests
2026-04-13 21:40:53 -04:00
Anso 9db97107aa fix(dashboard): harden real-time dashboard with bug fixes and design compliance (#517)
* fix(dashboard): harden real-time dashboard with bug fixes and design compliance

- Fix host alert spam: add 5-minute cooldown for CPU/RAM/disk threshold
  alerts, preventing duplicate notifications every 30s during sustained
  breaches. Extract shared dispatchWithCooldown helper (also used by
  Docker janitor alerts).
- Fix memory metric inflation: subtract filesystem cache from stored
  memory_mb values, matching the existing calculateMemoryPercent logic.
- Fix crash detection reliability: replace fragile 'seconds ago' string
  matching with a tracked Set of alerted container IDs. Containers are
  only alerted once per crash event, with automatic cleanup when they
  start running again or after a 1-hour TTL.
- Fix health status bar: exited containers now trigger 'degraded' state
  independently of unread error notifications.
- Fix CPU chart Y-axis: auto-scale when aggregate container CPU exceeds
  100% instead of silently clipping at the hardcoded domain ceiling.
- Fix grammar: 'actives' to 'active' in container count label.
- Add shadow-card-bevel to all dashboard cards per design system.
- Update dashboard docs to reflect revised health status thresholds.

* test(dashboard): update monitor service tests for new alert signatures

- Add container Id fields to crash detection test fixtures
- Update host alert assertions to match dispatchWithCooldown 3-arg call
- Fix unhealthy container test to use State: 'unhealthy' instead of
  State: 'running' (running containers are now skipped in crash detect)
2026-04-12 04:25:41 -04:00
Anso 1b573f542a feat(notifications): add shared notification routing rules (Admiral tier) (#347)
Route stack alerts to specific Discord, Slack, or webhook channels instead
of the single global endpoint. Includes per-rule enable/disable, priority
ordering, and automatic fallback to global agents when no rule matches.

- Add notification_routes table, interface, and CRUD in DatabaseService
- Add routing logic in NotificationService.dispatchAlert with optional stackName
- Pass stack context from MonitorService (crash/health) and SchedulerService
- Add 5 API endpoints gated with requireAdmin + requireAdmiral
- Add NotificationRoutingSection UI with Combobox stack picker, channel tabs
- Parallel webhook dispatch via Promise.allSettled
- 10 unit tests covering routing, fallback, and edge cases
- Documentation with screenshots at docs/features/notification-routing.mdx
2026-04-02 22:22:27 -04:00
Anso 93ae147ec1 test: expand backend test coverage for stability gaps (#329)
Add 5 new test files (116 test cases) targeting previously untested
service-layer logic:

- database-metrics: metrics CRUD, minute-bucket aggregation, cleanup
  retention, notification auto-cap at 100, stack alerts CRUD, and
  stress tests with 1000+ metrics
- monitor-service: CPU/memory/network calculation helpers, all 6
  alert condition operators, breach state machine lifecycle, global
  crash detection, host limit thresholds, cleanup delegation, and
  isProcessing concurrency guard
- docker-controller: validateApiData error detection, state-safe
  start/stop (304 handling), batch container removal with partial
  failures, disk usage calculation, resource classification
  (managed/unmanaged/system), orphan detection, and daemon
  unreachable error propagation
- scheduler-service: cron parsing, license tier gating, concurrent
  task prevention via runningTasks Set, manual trigger, all 4 task
  types (restart/snapshot/prune/update), wildcard targets, error
  recording, recovery notifications, and cleanup
- compose-service: subprocess spawn/exit handling, WebSocket output,
  deploy with health probe and atomic rollback, registry auth temp
  dir lifecycle, and downStack teardown
2026-04-01 22:43:35 -04:00