* fix(notifications): neutralize satellite-local node names in alert bodies
Fleet-aggregated alerts embedded each instance seed name (often Local)
while the hub badge already named the remote. Drop identity prefixes and
use type-aware local wording so attribution stays on the badge.
* fix(docs): correct image-update default check cadence
Operator docs still said six-hour polling; the seeded default is two
hours in interval mode, and the cadence is configurable or cron-based.
* test(notifications): assert hub stamps roster name on neutral remote bodies
Cover the fan-in path that attaches hub roster identity while leaving the
satellite message body unchanged.
Fleet-aggregated reclaim alerts showed Node Local in the body while the
badge already named the remote. Use a node-neutral message so identity
comes from the hub badge; leave stored rows unchanged.
Remove the MonitorService 6-hour version-check cooldown so node_update_available
fires when the shared version cache observes a published release, matching the
Fleet update button. Dedup and publish-pending gating still prevent spam.
GitHub Releases appear before docker-publish.yml finishes pushing images.
Probe Docker Hub and GHCR manifests before advertising a version as available.
Sanitize registry probe debug logs for CodeQL log-injection.
Rename "Janitor threshold" to "Reclaimable Docker data threshold" in Settings
and update the unused-Docker-data alert to point users to the Resources view
and the Prune Node Resources scheduled action, which are the real UI surfaces
for reclaiming disk space. Remove "janitor" from settings search keywords.
The internal docker_janitor_gb database key and JANITOR_* constants are
unchanged.
Vulnerability scan rows were never cleaned up when their image was removed
from Docker or their stack was deleted, so the Security Overview (including
the Top exploit-risk findings card) kept surfacing findings for artifacts that
no longer exist.
Scan results now reflect what is still on the host:
- Deleting a stack immediately purges its stack:<name> compose-config scan.
- A background reconciliation in the monitor janitor removes scans whose image
is gone from the node, or whose stack folder no longer exists. It is
fail-safe: a scan is only removed when its artifact is positively known to be
gone, the Docker image list is read with a timeout (skipped on failure), and
stack scans are reconciled only when the stack list is non-empty.
- An opt-out "Remove scans for deleted images and stacks" setting (on by
default, per-node) lets operators retain scan history for removed artifacts.
Scan deletes remove child findings explicitly, since SQLite foreign-key cascade
is not enabled on the connection.
* feat: add node update alerts with changelog tab and skip-version handling
- Add node_update_available notification category with blue/brand bell dot
- Route node_update_available notifications to Fleet -> Node updates sheet
- Add Changelog tab to NodeUpdatesSheet with GitHub release notes
- Add per-node skip-version persistence (node_update_skips table)
- Skip hides update CTA on node card and sheet; re-surfaces on newer version
- Skipped nodes excluded from Update all backend filter
- Add pulsating dot indicator on Changelog tab when updates available
- Always-visible View changelog action in notification row bottom
- Admin-only for all mutating controls (skip, unskip, update)
- Backend tests for skip-version semantics (15 tests)
- Update fleet-view.mdx, remote-updates.mdx, and OpenAPI spec
* fix: address audit findings - nested button, stale changelog, semver normalization, mobile intent
- Move View changelog button outside routable button (sibling element)
- Fix aria-label for node_update_available notification rows
- Support ?recheck=true on release-notes endpoint
- Invalidate release notes cache on forced recheck
- Store normalized semver (semver.valid strips v prefix)
- Skip fleetUpdatesIntent on mobile (desktop only)
- Add v-prefix normalization test
* fix: restore View changelog on same line as timestamp, opposite sides
The button is always visible at the bottom right of the notification card,
on the same row as the timestamp (just now), using justify-between layout.
* fix: update tests for node_update_available category and release-notes fetch
- Backend: monitor-service tests now expect node_update_available instead of system
- Frontend: NodeUpdatesSheet tests mock release-notes API call to prevent undefined then()
* fix: resolve ci lint failures
* feat: add ON/OFF toggle for host threshold alerts
Add host_alerts_enabled setting (default ON) as a master switch for CPU,
RAM, and disk host threshold evaluation. When OFF, the four threshold
controls in Settings > Host Alerts are disabled and MonitorService skips
the systeminformation calls and alert dispatch entirely, while clearing
stale suppression state so re-enabling starts fresh.
The dashboard Configuration Status card shows "Off" when host threshold
alerts are disabled. Crash capture, health gate, deploy guardrails,
stack alert rules, and the Docker janitor are all unaffected.
* fix: exit NumberChip edit mode when externally disabled
When the host threshold alerts master toggle is turned OFF while a
NumberChip is in edit mode, force-exit edit mode so the chip renders
the greyed-out button state consistently with the other chips.
* fix(metrics): host memory usage excludes reclaimable page cache
Host RAM was computed as mem.used / mem.total via systeminformation, but mem.used counts
reclaimable buffers/cache as used, so a busy Linux host read ~99%. Switch the dashboard stats,
the fleet node self-report, and the host-RAM alert threshold to mem.active / mem.total
(cache-excluded), and report used: mem.active and free: mem.available so the byte readout stays
consistent with the percentage. Add regression tests for a cache-heavy host (no false alert) and
a genuinely busy host (alert still fires).
* test(metrics): assert system stats memory excludes reclaimable cache
The cached /api/system/stats test mocked si.mem() without active/available, so after
the route switched to the cache-excluded fields it produced a NaN percentage that the
shape-only assertions did not catch. Fill the mock with a realistic shape and assert the
route reports the active working set, not the cache-inclusive used/free.
* fix(fleet): gate node update actions to admins and harden update tracking
Node update affordances now render only for admins, matching the admin-only
routes behind them. Previously a non-admin could open the Fleet view and see the
per-node Update button, Update all, retry, dismiss, and Recheck controls, then
get a 403 on click. Those controls are now hidden for non-admins, who still see
read-only update status.
Both update-status clear routes (per-node and bulk) now require admin, and the
bulk recheck throttles its forced "latest published version" lookup so a caller
cannot loop it to hammer the upstream registries; the response reports whether
the refresh actually ran so the UI can surface a "checked recently" note.
Completion detection no longer reports a node as Updated when it merely blips
offline and returns on the same version with an unchanged process start time.
That case stays in progress and is decided by the existing early-fail and
timeout heuristics, so a momentary network glitch is not mistaken for a
successful update. Failed and timed-out updates now emit an operator-visible
warning, and a periodic safety-net sweep bounds in-flight trackers when no
client is polling for status.
* fix(fleet): harden update completion and recheck failure handling
Refinements from review of the node self-update hardening:
- Completion signal 1 now requires a valid version, not merely a different one.
A node whose /api/meta momentarily omits or mangles its version (online, same
process) reported version=null, which compared unequal to the previous version
and falsely marked the update completed. It now stays in progress and is
decided by the early-fail/timeout heuristics.
- Terminal resolution is atomic: it re-reads the live tracker and transitions
only if it is still in flight with the same start time, so two concurrent
status polls cannot both warn or clobber each other's transition.
- The operator warning for a failed or timed-out update now redacts
secret-shaped text (bearer/basic/token/password, credentialed URLs) from the
underlying error before logging, in addition to stripping control characters.
- The Recheck button now surfaces an error toast when the request throws
(network or auth failure), matching the existing non-ok-response path instead
of only logging to the console.
* feat(security): per-image scroll + retention cap in scan history
Long scan histories for hot images used to monopolise the Scan history
sheet: a single image with dozens of scans pushed every other image off
screen, and the underlying vulnerability_scans table grew without
bound.
Each image group's table now renders inside its own ScrollArea capped
at max-h-64 (~6 rows visible) so a busy image scrolls independently
while the list of images stays navigable. A new global setting
scan_history_per_image_limit (default 50, min 5, max 1000) backs both
a window-function query that caps the response per image_ref and a
prune step that runs on the existing MonitorService cleanup tick. The
response now carries cappedImageRefs + perImageLimit so the UI can
render a "Capped at N · older scans pruned" hint on groups sitting at
the ceiling without a second settings round-trip.
Single-image deep-dive (imageRef query param) bypasses the cap so a
user clicking into one image can still see its full history. The
prune uses self-contained subqueries to avoid SQLITE_MAX_VARIABLE_NUMBER
issues on first-run installs with large backlogs, and explicitly
deletes child rows from vulnerability_details, secret_findings, and
misconfig_findings inside a transaction since FK cascade is not
enabled at the connection level.
Settings → Developer → Data retention gains a "Scan history per image"
field.
* fix(security): skip searchDraft debounce on mount to stop page-reset race
The searchDraft debounce useEffect fires once on initial mount with the
unchanged value and, 300ms later, unconditionally calls setPage(0).
When a user (or a test) paginates inside that 300ms window, the
pending debounce silently undoes the page advance.
CI surfaced this as a flaky 3rd fetch in the "advances offset when the
user pages forward" test once the per-image cap work added enough
state-update overhead to push the click past the 300ms threshold on
the slower Linux jsdom run.
Track searchDraft with a ref and exit the effect when the value has
not actually changed, so the debounce only runs in response to real
user typing.
* fix(stack-activity): per-stack history integrity, attribution, sanitization
Address the Stack Activity audit findings (PR 1 of 2):
- Per-stack history integrity: drop the per-insert 100-row prune in
addNotificationHistory that evicted quieter stacks' history whenever
another stack got chatty. Periodic cleanupOldNotifications now caps
per (node, stack) at 500 rows and per-node unattached system events
at 1000 rows, on top of the existing 30-day retention. Signature
takes an options bag and returns a per-stage summary so MonitorService
can log what actually ran each cycle.
- Actor attribution: thread req.user?.username through every
notifyActionFailure call site and add synthetic actors at service
emit sites (system:autoheal, system:scheduler, system:image-update,
system:docker-events, system:blueprint, system:monitor, system:policy).
The timeline renders system actors as "via <Label>" so an autoheal
redeploy is no longer indistinguishable from a user redeploy.
- Message sanitization: new sanitizeNotificationMessage at
NotificationService.dispatchAlert strips KEY=VALUE pairs whose key
ends in TOKEN/KEY/PASSWORD/SECRET/CREDENTIALS/AUTH, scrubs HTTP basic
auth in URLs and Bearer tokens, collapses COMPOSE_DIR paths, and
truncates to 1000 chars. Applied to the stored history and to every
downstream Discord/Slack/webhook channel. The ImageUpdateService
recovery-path direct DB write also runs through the sanitizer.
- Composite pagination cursor: getStackActivity now accepts a
(timestamp, id) cursor (?before=&beforeId=). The legacy timestamp-only
form silently dropped events when a single compose up emitted many
events sharing one millisecond. Route rejects beforeId without before.
- Frontend hardening: distinct error state with retry button (initial
fetch failure no longer renders as the genuine empty state), strict
positive-integer parsing on cursor params, overrequest-by-1 pagination
so the last page does not leave a dead "Load more" click, runtime
guard on liveEvents merge that validates the level union, per-minute
day-bucket recompute so an open panel does not stay on "Today" past
midnight.
No tier, role, or capability gate touched. Route permission gate
remains stack:read on the named stack.
* fix(stack-activity): sanitizer covers lowercase env vars and per-node compose dir
External review surfaced two leak paths in the message sanitizer:
- The sensitive-key regex was uppercase-only. Compose env names are
conventionally uppercase but lowercase forms (db_password, jwt_secret,
github_token) are valid and do leak through the same Docker and
compose-parse error paths. Make the regex case-insensitive and tighten
it to also catch bare TOKEN= / KEY= / PASSWORD= without a prefix word,
while still leaving BYPASS, COMPASS, and similar non-secret keys alone.
- The compose-dir path collapse only read process.env.COMPOSE_DIR, but
the real resolution chain is node.compose_dir (per-node DB override)
-> process.env.COMPOSE_DIR -> /app/compose. A node with a custom
compose_dir could still leak absolute paths into stored history and
downstream channels. Route both the dispatchAlert call and the
ImageUpdateService recovery-path direct write through
NodeRegistry.getInstance().getComposeDir(localNodeId) so the
collapse covers every resolution outcome.
Tests now assert lowercase keys are redacted and that BYPASS-style
non-secrets stay intact in both cases. notification-routing mock
extended to stub the new getComposeDir call.
* chore(stack-activity): a11y roles, visibility-aware tick, live-disconnect signal
Close three small follow-ups on the per-stack activity timeline:
- A11y: each day-group gets role="list" and each event row gets
role="listitem" so screen readers traverse the timeline as a list
instead of a wall of text. The day-group container also carries an
aria-label naming the bucket.
- Visibility-aware day-bucket tick: the 60s setInterval that re-derives
Today/Yesterday/Earlier now short-circuits when document.hidden, so a
backgrounded panel does not re-render every minute for no visible
effect.
- Live-disconnect signal: useNotifications dispatches a
sencho:notifications-connection custom event on WebSocket open and
close. The timeline listens and, when explicitly disconnected, shows
a one-line "Live updates offline; reconnecting…" hint above the list.
The sidebar ticker already surfaces fleet-wide connection state; this
adds an in-context cue for users who are focused on a single stack.
Stack-name case normalization was considered and rejected: stack names
are case-permissive per the isValidStackName validator, and lowercasing
on read or write would silently rename or hide a user's "MyApp" stack.
* ci(stack-activity): drop unnecessary escape in URL_BASIC_AUTH regex
ESLint no-useless-escape errored on \- inside the character class
[a-zA-Z0-9+.\-] at notificationMessage.ts:14. Move the dash to the
end of the class so it's an unambiguous literal and the escape is no
longer required. Behavior is identical; sanitizer tests still pass.
* revert(stack-activity): drop unvalidated E2E spec from this PR
The spec was committed without ever running against a real Docker
daemon, then failed in CI when it ran for the first time: deploy
returned 200 but no notification appeared on the activity endpoint
within the polling window, suggesting either a deploy-notification
race or a node-id resolution mismatch in the CI environment.
Backend unit tests (route + composite cursor + sanitizer) and
frontend component tests cover the same logic. The E2E spec will
land in a dedicated follow-up once it has been authored against a
working CI environment.
* fix(monitor): collapse repeated host-metric alerts into per-window summary (F-11)
A host metric over threshold previously dispatched one notification every 5
minutes for the duration of the breach, producing 7+ identical messages
in 35 minutes and spamming Discord/Slack routes. Replace the hardcoded
5-minute cooldown for CPU/RAM/disk with a per-metric suppression window
(default 60 min, configurable via host_alert_suppression_mins). The first
breach fires immediately; subsequent cycles within the window are silently
counted; the next dispatch after the window elapses carries a summary
suffix listing how many cycles were suppressed and when the breach first
crossed threshold. Recovery clears the counter so re-breach fires fresh.
The pattern mirrors PolicyEnforcement.notifyTrivyMissingOnce: module-scope
Map, in-memory only, in-cycle dedup, with a test-reset helper. The
existing system_state row keeps post-restart re-fires bounded.
Janitor and per-stack alert rules are unchanged; they already have
adequate cadence and per-rule cooldown respectively.
* fix(ci): restore backend and frontend checks
* fix(e2e): remove create button timing race
* fix(e2e): harden create double-click test
* fix(monitor): clear persisted F-11 timestamp on recovery + clamp suppression window
Independent audit on the previous commit surfaced two issues.
1. clearHostMetricSuppression early-returned on missing in-memory state,
leaving a stale system_state.last_host_*_alert_ts row alive after a
process restart. Scenario: breach fires + persists timestamp, process
restarts, metric recovers before another evaluate cycle re-seeds the
in-memory Map, recovery cleanup early-returns. Next re-breach inside
the original window hits the restart-survivability branch and is
silently suppressed instead of firing fresh. Fix: read persisted
state in clearHostMetricSuppression and reset to '0' independently
of in-memory presence. The read-before-write also skips redundant
writes when the row is already cleared.
2. host_alert_suppression_mins is validated by zod on the bulk PATCH
path but the single-key POST /api/settings path accepts allowlisted
keys without re-validation. A 999999999-minute value would silence
host alerts for centuries. Add MAX_HOST_ALERT_SUPPRESSION_MIN = 1440
mirroring the zod max, and clamp via Math.min in evaluateGlobalSettings.
Two new vitest cases (restart-then-recovery-then-rebreach; the 1440
clamp) confirmed failing before the fix, passing after. The existing
"metric drop" case updated to use a mock-backed persistence pattern
consistent with the new restart-scenario tests. 73/73 monitor-service
tests green; full backend suite 2507/2510 (same pre-existing Windows
EBUSY flake on filesystem-backup.test.ts as baseline).
* fix(monitor): decouple janitor disk-usage check from 30s cycle (F-6)
`docker system df` (called by the MonitorService janitor check) can take
30+ seconds on Docker Desktop with many volumes. Running it on the 30s
evaluate cycle compounded with the per-container stats fan-out and pushed
the cycle to 140s+, blocking subsequent monitoring work.
This change:
- Moves the janitor disk-usage check into its own 15-minute cycle with
a tight 8s timeout. A circuit breaker opens after 3 consecutive
timeouts (60-minute cooldown) so a sick daemon stops pinning Dockerode
sockets every tick. The first janitor tick is deferred 45 seconds past
boot to avoid head-of-line collision with the initial monitor cycle's
stats fan-out.
- Adds a paired 8s `withTimeout` wrap to the admin prune-estimate routes
(`/api/system/prune/estimate` and the dry-run path of
`/api/system/prune/system`) so a slow df does not hang the admin tab.
Both routes respond 503 with code `docker_df_slow` on timeout.
- Factors `withTimeout` and `TimeoutError` into `utils/withTimeout.ts`
so the route layer does not have to import from a service module.
- Adds 10 unit tests covering the decoupling guardrail, breaker
open/close, cooldown, threshold gate, the 100 MB reclaimable floor,
re-entrancy, recovery logging, non-timeout error handling, and the
full timer-cleanup contract of `stop()`.
- Adds 4 integration tests for the prune routes covering the 503
timeout response, the success path, and the non-timeout 5xx path.
* fix(fleet,monitor): extend F-6 timeout to fleet prune routes; close breaker-recovery log gap
Codex audit findings on PR #1164:
Major. The fleet routes that fan out prune-estimate work on local nodes
(`POST /api/fleet/labels/fleet-prune` dry-run path and
`POST /api/fleet/prune/estimate`) called `estimateSystemReclaim` without
a timeout, so a slow local Docker daemon could still hang the fleet
admin tab even though the system-maintenance routes were already
bounded. Wrap both call sites with the shared `withTimeout(..., 8s)`
and surface a "Docker daemon is busy" message via the per-target and
per-node error channels the routes already used for other failures.
The destructive (non-dry-run) prune path stays unwrapped because it
calls `pruneSystem` / `pruneManagedOnly`, not `df`.
Minor. The janitor circuit breaker zeroed `janitorConsecutiveTimeouts`
when it opened, so a successful call after a full breaker-open cooldown
slipped past the `if (counter > 0)` recovery-log branch and never
emitted `[Monitor] Janitor disk-usage check recovered`. The operator
observability signal was missing exactly when it mattered most.
Extend the predicate to also trip on `janitorBreakerUntil > 0` (which
stays set to its past timestamp after cooldown until the next success
clears it), so recovery logs symmetrically for both partial-failure
and post-breaker recovery paths. Added a dedicated test.
Three new integration tests cover the fleet routes (timeout, success,
and the estimate endpoint's per-node unreachable shape).
The MonitorService.evaluateStackAlerts loop processed containers serially,
awaiting one Docker stats call per container. Each Docker stats call
inherently waits ~1s for the engine to produce a sample, so cycle time
scaled linearly with container count and exceeded the 25s warning
threshold on production nodes with ~17+ running containers.
Changes:
- Extract the per-container body into processContainer().
- Replace the serial for-loop with a bounded-concurrency fan-out
(runWithConcurrency, cap of 10) so the cycle runs in roughly
max(per-container time) instead of sum.
- Add a per-cycle dedup set (firedThisCycle) so a stack rule fires
exactly once per cycle even when multiple containers in the stack
all breach. Without it, parallel workers race past the cooldown
check before any DB write lands and dispatch the same alert N
times on first breach.
- Tests cover wall-time parallelism, per-container failure isolation,
the concurrency cap, and the per-cycle dispatch dedup.
- Change network metrics (net_rx/net_tx) from cumulative totals to MB/s rates
so alert thresholds are operationally meaningful
- Wrap all external calls (Docker stats, systeminformation, docker df) in
10-second timeout via Promise.race to prevent hung operations from
blocking the evaluation loop indefinitely
- Parallelize host CPU/RAM/disk queries with Promise.all to bound worst-case
latency at 10 seconds instead of 30
- Use epsilon comparison for == operator so floating-point metric values
can match integer thresholds
- Clean up stale entries in activeBreaches and previousNetworkStats maps
after rules are deleted or containers stop
- Add standard INFO logging for alert firings and WARN logging for slow
cycles and timeouts; add diagnostic cycle timing and breach-count log
MonitorService.evaluate() forked the docker CLI every 30s and
walked the human-readable Reclaimable strings ("1.196GB", etc.)
with a regex to compute the janitor threshold check. The Docker
Engine API returns raw byte counts, and the existing
DockerController.getDiskUsage() already wraps it for images,
containers, and volumes. Extend that helper with reclaimable
build-cache bytes so MonitorService can sum the four categories
in one call.
Drops the child_process / promisify imports from MonitorService and
removes about 30 lines of stdout parsing. Also widens the explicit
return type of getDiskUsageClassified so the new fields aren't
silent runtime additions.
Introduce a NotificationCategory string-literal union (11 values) and
thread it through dispatchAlert as a required second argument. All
callers (DockerEventService, AutoHealService, ImageUpdateService,
MonitorService, PolicyEnforcement, policyGate, SchedulerService,
imageUpdates route) pass an explicit category at every call site,
giving TypeScript compile-time enforcement that no new emit site can
be added without choosing a category.
DatabaseService gains an idempotent migration that adds a nullable
category TEXT column to notification_history; existing rows keep
category=NULL (displayed as Uncategorized in the UI). The
getNotificationHistory method accepts an optional category filter
that is forwarded from the GET /api/notifications/history route via
a ?category= query param.
NotificationPanel gains a category Select dropdown so users can
filter history by category. The frontend types mirror the backend
union so API responses are type-safe end-to-end.
All 75 test files (1410 tests) updated to the new 4-arg dispatchAlert
signature and passing.
The Docker janitor watchdog had three problems on multi-node fleets:
1. The alert text said "Your system has accumulated X GB" with no node
identifier, so on a fleet view the operator could not tell which
node was complaining. Resolve the local node via NodeRegistry and
put the node name in the message.
2. The threshold gate was a single comparison against the user's
configured GB value. A small or accidentally tiny threshold made
the alert fire on hosts with effectively no waste. Add a 100 MB
absolute floor so trivial cruft never triggers a notification.
3. The unit parser only matched uppercase "GB|MB|KB|B" and dropped
"TB" entirely. Modern Docker emits "kB" with a lowercase k, which
silently contributed zero bytes to the running total. Normalise the
unit to uppercase before the comparison and add the TB case.
* fix(notifications): stop Sencho version notifications from silently skipping
Three independent defects combined to make version-update notifications
silently fail even while Fleet overview correctly surfaced an update button:
- The in-memory 6-hour cooldown was advanced before the network fetch,
so a single transient failure at boot could lock the check for the
rest of the container lifetime. Moved the cooldown update inside the
success branch so failures retry on the next eval cycle.
- MonitorService called the raw version fetch directly, bypassing the
CacheService wrapper (TTL, inflight dedup, stale-on-error) that Fleet
uses, so the two paths could diverge. Unified both on a shared
getLatestVersion() helper in utils/version-check.ts.
- The dedup key could carry stale state from a previous build and never
self-clear. It now self-heals when the running version reaches the
previously-notified version, so future releases re-fire as expected.
Added diagnostic logs gated on debug mode for each skip branch, plus
three regression tests covering cooldown-on-failure, cooldown-on-success,
and dedup self-heal.
* docs(notifications): drop legacy-upgrade framing from alerts troubleshooting
Sencho has not shipped publicly, so troubleshooting entries written in
'this used to happen but now does Y' mode reference a past that does not
exist for any reader. Rewrote the version-notification, image-update,
and crash-alert troubleshooting entries to describe current behavior
positively without referring to prior builds, upgrade paths, or legacy
fixes.
* chore(security): accept CVE-2026-33810 in bundled Docker CLI 29.4.0
Trivy now flags CVE-2026-33810 (Go stdlib crypto/x509 DNS constraint
bypass, fixed in Go 1.26.2) in the Docker CLI static binary we ship.
Docker CLI 29.4.0 is the latest upstream release and still links Go
1.26.1; no newer static binary exists yet.
Same exposure profile as the already-accepted CVE-2026-32280: the
Docker CLI and compose plugin only validate certificates from
well-known registry CAs and the local Docker socket, not from
attacker-controlled CAs with crafted DNS name constraints. Revisit
on the next Docker CLI release that rebuilds against Go 1.26.2 or
later.
* fix(notifications): replace polling with Docker event stream for container lifecycle detection
Replaces the 30-second MonitorService crash-detection poll with a causal,
per-node Docker events stream. Eliminates false crash alerts on intentional
stops (docker stop, compose down, stack restart/update), detects OOM kills
as a distinct alert category, and surfaces real crashes in real time.
A new DockerEventManager spawns one DockerEventService per local node. Each
service consumes the filtered container event stream, classifies die events
against recent kill/oom state, and reconciles container state via snapshot
diffing on connect and reconnect. Rate limiting, exponential backoff with
jitter, and parse-error tolerance keep the stream resilient under load and
during daemon interruptions.
MonitorService retains host limits, janitor, version check, and stack metric
alerts; crash and healthcheck detection move out entirely.
* fix(tests): silence require-imports lint in hoisted mock factory
- Read running Sencho version from the packaged manifest via getSenchoVersion() instead of process.env.npm_package_version, which is undefined when launched via node dist/index.js (production Docker). Skip the update check entirely when the version cannot be resolved.
- Add a one-time backfill pass in ImageUpdateService so users who upgraded to a Sencho version with the notification pipeline receive a catch-up entry for stacks already flagged as having updates before the upgrade.
- Surface dispatch failures as error-level entries in the in-app notification bell via a direct notification_history write, so misconfigured webhooks are visible without tailing logs.
- Extend test coverage for both paths: mock getSenchoVersion (including the null-version case) and add dispatch-path tests for transition, no-re-fire, backfill, and error surfacing.
- Expand alerts-notifications docs with per-instance 6-hour check cadence, an example version message, a backfill note, and three new troubleshooting entries.
* fix(alerts): harden with security fixes, design compliance, and test coverage
Add authMiddleware to all alert endpoints, validate notification test
dispatch inputs, fix restart_count metric via Docker inspect, correct
network metric units, replace any types with DockerContainerStats
interface, add webhook timeouts and dispatch error tracking.
Frontend: migrate Select to Combobox, add ScrollArea and delete
confirmation AlertDialog, fix icon strokeWidth to 1.5.
Add update availability notifications for both Sencho version updates
(6-hour check in MonitorService) and stack image updates (state
transition detection in ImageUpdateService). Extract shared version
fetch logic into utils/version-check.ts.
Add diagnostic logging gated behind developer_mode for MonitorService
breach state machine and NotificationService dispatch routing.
Tests: 24 new alert API integration tests, restart_count and version
check unit tests (688 total passing). Docs updated with HTTPS
requirement, update notifications section, and troubleshooting guide.
* fix(alerts): remove unused TEST_USERNAME import in alerts-api tests
* fix(dashboard): harden real-time dashboard with bug fixes and design compliance
- Fix host alert spam: add 5-minute cooldown for CPU/RAM/disk threshold
alerts, preventing duplicate notifications every 30s during sustained
breaches. Extract shared dispatchWithCooldown helper (also used by
Docker janitor alerts).
- Fix memory metric inflation: subtract filesystem cache from stored
memory_mb values, matching the existing calculateMemoryPercent logic.
- Fix crash detection reliability: replace fragile 'seconds ago' string
matching with a tracked Set of alerted container IDs. Containers are
only alerted once per crash event, with automatic cleanup when they
start running again or after a 1-hour TTL.
- Fix health status bar: exited containers now trigger 'degraded' state
independently of unread error notifications.
- Fix CPU chart Y-axis: auto-scale when aggregate container CPU exceeds
100% instead of silently clipping at the hardcoded domain ceiling.
- Fix grammar: 'actives' to 'active' in container count label.
- Add shadow-card-bevel to all dashboard cards per design system.
- Update dashboard docs to reflect revised health status thresholds.
* test(dashboard): update monitor service tests for new alert signatures
- Add container Id fields to crash detection test fixtures
- Update host alert assertions to match dispatchWithCooldown 3-arg call
- Fix unhealthy container test to use State: 'unhealthy' instead of
State: 'running' (running containers are now skipped in crash detect)
The Skipper/Admiral atomic deploy/update path used to create
.sencho-backup/ inside the user's stack folder, which silently failed
with EACCES whenever a container had chowned the bind mount (swag,
tautulli, linuxserver/* images, etc). That broke auto-rollback and the
manual rollback endpoint for those stacks. Stack backups now live under
<DATA_DIR>/backups/<stackName>/ next to sencho.db, which is always
writable by the Sencho user.
While stress-testing the same scenario, MonitorService also flooded the
error log with "Error parsing stats for container ... 404 no such
container" because per-container stats polls (30s tick) raced with
docker compose recreating containers. The 404 case is now skipped
silently; non-404 stats failures still log at error level.
Route stack alerts to specific Discord, Slack, or webhook channels instead
of the single global endpoint. Includes per-rule enable/disable, priority
ordering, and automatic fallback to global agents when no rule matches.
- Add notification_routes table, interface, and CRUD in DatabaseService
- Add routing logic in NotificationService.dispatchAlert with optional stackName
- Pass stack context from MonitorService (crash/health) and SchedulerService
- Add 5 API endpoints gated with requireAdmin + requireAdmiral
- Add NotificationRoutingSection UI with Combobox stack picker, channel tabs
- Parallel webhook dispatch via Promise.allSettled
- 10 unit tests covering routing, fallback, and edge cases
- Documentation with screenshots at docs/features/notification-routing.mdx
Add console.warn/console.error logging to 22 silent catch blocks across
10 files. Errors in cleanup, migrations, SSO, fleet snapshots, shutdown,
and validation are now visible in logs. ENOENT guards added to
file-system catches to distinguish missing files from permission errors.
No control flow changes.
- Configurable retention: audit_retention_days setting (1-365 days, default 90)
replaces hardcoded 90-day retention, exposed in Settings > Data Retention
- Export: one-click CSV/JSON export of filtered audit data via new
GET /api/audit-log/export endpoint (capped at 10,000 entries)
- Auditor role: read-only role with system:audit permission for viewing
and exporting audit logs without admin privileges (Admiral tier)
- Enhanced filtering: full-text search across summaries/paths/usernames,
date range picker, and expandable row details showing request path,
IP address, node ID, and entry ID
* feat: audit logging, secrets at rest, and legacy cleanup
- Add Team Pro audit log: records all mutating API actions with user
attribution, searchable timeline UI with filtering and pagination
- Add AES-256-GCM encryption at rest for node API tokens via CryptoService
- Drop 9 legacy SSH/TLS columns from nodes table (dead since v0.7)
- Remove orphaned MaintenanceModal.tsx (dead code, never imported)
- Add requireTeamPro backend guard for team-tier features
- Add audit log cleanup (90-day retention) to MonitorService
- Add docs page and navigation entry for audit log feature
* fix: remove unused AUTH_TAG_LENGTH constant from CryptoService
Security:
- Strip auth credential keys (auth_username, auth_password_hash,
auth_jwt_secret) from GET /api/settings response
- Add allowlist guard to POST /api/settings — rejects unknown or
auth-namespace keys with a 400
Backend:
- Add PATCH /api/settings bulk endpoint with Zod schema validation
(type coercion, range checks, URL format) and atomic SQLite transaction
- Add system_state table — moves last_janitor_alert_timestamp out of
global_settings; adds getSystemState/setSystemState on DatabaseService
- Add metrics_retention_hours and log_retention_days configurable settings;
MonitorService reads both dynamically each evaluation cycle
- Add cleanupOldNotifications(days) to DatabaseService, called each cycle
Frontend:
- Replace single isLoading flag with per-operation states
(isSavingSystem, isSavingDeveloper, isSavingPassword, isSavingRegistry,
isSavingAgent/isTestingAgent per agent type)
- Add skeleton loader that blocks interaction until fetchSettings resolves
- Explicit key-picking in fetchSettings — auth keys cannot enter state
- Unsaved-changes amber dot on System Limits and Developer sidebar items
- Separate saveSystemSettings / saveDeveloperSettings — no cross-tab clobber
- Developer tab gains Data Retention section (metrics hours, log days)
- All settings saves use new PATCH /api/settings endpoint
Add Node modal — type selector & state reset:
- Restored a Local/Remote <Select> dropdown in renderFormFields so users can
explicitly choose the node type instead of it defaulting silently to 'remote'.
- Switching type clears api_url and api_token so no stale remote credentials
carry over if a user switches from Remote to Local mid-form.
- Replaced the static "Add Remote Node" title with a dynamic one that reflects
the currently selected type ("Add Local Node" / "Add Remote Node").
- onOpenChange now resets formData to defaultFormData whenever the dialog
opens, preventing stale values from a previous session leaking in.
Remote connection details — real metrics:
- testRemoteConnection previously returned hard-coded '-' for containers,
images, and cpus after a successful auth/check ping.
- Now fires three parallel requests (Promise.allSettled) after auth passes:
/api/stats → containers total + running count
/api/system/stats → cpu.cores
/api/system/images → image list length
- Each field falls back to '-' gracefully if an endpoint is unavailable,
so a slow or older remote instance never breaks the connection test.
DEP0060 util._extend suppression:
- http-proxy@1.18.1 calls util._extend when createProxyServer() is first
invoked at runtime (NOT at import time). A process.emitWarning override
placed before the proxy instantiations intercepts only DEP0060 without
suppressing any other warnings. No package version changes needed.
Also includes linter/formatter normalisation across multiple files.
Remote nodes in the Distributed API model are self-monitoring — each
remote Sencho instance runs its own MonitorService against its local
Docker socket. The main instance must not attempt direct DockerController
access for remote nodes, which caused fatal crashes on every 30s tick.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Anso