MonitorService.evaluate() forked the docker CLI every 30s and
walked the human-readable Reclaimable strings ("1.196GB", etc.)
with a regex to compute the janitor threshold check. The Docker
Engine API returns raw byte counts, and the existing
DockerController.getDiskUsage() already wraps it for images,
containers, and volumes. Extend that helper with reclaimable
build-cache bytes so MonitorService can sum the four categories
in one call.
Drops the child_process / promisify imports from MonitorService and
removes about 30 lines of stdout parsing. Also widens the explicit
return type of getDiskUsageClassified so the new fields aren't
silent runtime additions.
Introduce a NotificationCategory string-literal union (11 values) and
thread it through dispatchAlert as a required second argument. All
callers (DockerEventService, AutoHealService, ImageUpdateService,
MonitorService, PolicyEnforcement, policyGate, SchedulerService,
imageUpdates route) pass an explicit category at every call site,
giving TypeScript compile-time enforcement that no new emit site can
be added without choosing a category.
DatabaseService gains an idempotent migration that adds a nullable
category TEXT column to notification_history; existing rows keep
category=NULL (displayed as Uncategorized in the UI). The
getNotificationHistory method accepts an optional category filter
that is forwarded from the GET /api/notifications/history route via
a ?category= query param.
NotificationPanel gains a category Select dropdown so users can
filter history by category. The frontend types mirror the backend
union so API responses are type-safe end-to-end.
All 75 test files (1410 tests) updated to the new 4-arg dispatchAlert
signature and passing.
The Docker janitor watchdog had three problems on multi-node fleets:
1. The alert text said "Your system has accumulated X GB" with no node
identifier, so on a fleet view the operator could not tell which
node was complaining. Resolve the local node via NodeRegistry and
put the node name in the message.
2. The threshold gate was a single comparison against the user's
configured GB value. A small or accidentally tiny threshold made
the alert fire on hosts with effectively no waste. Add a 100 MB
absolute floor so trivial cruft never triggers a notification.
3. The unit parser only matched uppercase "GB|MB|KB|B" and dropped
"TB" entirely. Modern Docker emits "kB" with a lowercase k, which
silently contributed zero bytes to the running total. Normalise the
unit to uppercase before the comparison and add the TB case.
* fix(notifications): stop Sencho version notifications from silently skipping
Three independent defects combined to make version-update notifications
silently fail even while Fleet overview correctly surfaced an update button:
- The in-memory 6-hour cooldown was advanced before the network fetch,
so a single transient failure at boot could lock the check for the
rest of the container lifetime. Moved the cooldown update inside the
success branch so failures retry on the next eval cycle.
- MonitorService called the raw version fetch directly, bypassing the
CacheService wrapper (TTL, inflight dedup, stale-on-error) that Fleet
uses, so the two paths could diverge. Unified both on a shared
getLatestVersion() helper in utils/version-check.ts.
- The dedup key could carry stale state from a previous build and never
self-clear. It now self-heals when the running version reaches the
previously-notified version, so future releases re-fire as expected.
Added diagnostic logs gated on debug mode for each skip branch, plus
three regression tests covering cooldown-on-failure, cooldown-on-success,
and dedup self-heal.
* docs(notifications): drop legacy-upgrade framing from alerts troubleshooting
Sencho has not shipped publicly, so troubleshooting entries written in
'this used to happen but now does Y' mode reference a past that does not
exist for any reader. Rewrote the version-notification, image-update,
and crash-alert troubleshooting entries to describe current behavior
positively without referring to prior builds, upgrade paths, or legacy
fixes.
* chore(security): accept CVE-2026-33810 in bundled Docker CLI 29.4.0
Trivy now flags CVE-2026-33810 (Go stdlib crypto/x509 DNS constraint
bypass, fixed in Go 1.26.2) in the Docker CLI static binary we ship.
Docker CLI 29.4.0 is the latest upstream release and still links Go
1.26.1; no newer static binary exists yet.
Same exposure profile as the already-accepted CVE-2026-32280: the
Docker CLI and compose plugin only validate certificates from
well-known registry CAs and the local Docker socket, not from
attacker-controlled CAs with crafted DNS name constraints. Revisit
on the next Docker CLI release that rebuilds against Go 1.26.2 or
later.
* fix(notifications): replace polling with Docker event stream for container lifecycle detection
Replaces the 30-second MonitorService crash-detection poll with a causal,
per-node Docker events stream. Eliminates false crash alerts on intentional
stops (docker stop, compose down, stack restart/update), detects OOM kills
as a distinct alert category, and surfaces real crashes in real time.
A new DockerEventManager spawns one DockerEventService per local node. Each
service consumes the filtered container event stream, classifies die events
against recent kill/oom state, and reconciles container state via snapshot
diffing on connect and reconnect. Rate limiting, exponential backoff with
jitter, and parse-error tolerance keep the stream resilient under load and
during daemon interruptions.
MonitorService retains host limits, janitor, version check, and stack metric
alerts; crash and healthcheck detection move out entirely.
* fix(tests): silence require-imports lint in hoisted mock factory
- Read running Sencho version from the packaged manifest via getSenchoVersion() instead of process.env.npm_package_version, which is undefined when launched via node dist/index.js (production Docker). Skip the update check entirely when the version cannot be resolved.
- Add a one-time backfill pass in ImageUpdateService so users who upgraded to a Sencho version with the notification pipeline receive a catch-up entry for stacks already flagged as having updates before the upgrade.
- Surface dispatch failures as error-level entries in the in-app notification bell via a direct notification_history write, so misconfigured webhooks are visible without tailing logs.
- Extend test coverage for both paths: mock getSenchoVersion (including the null-version case) and add dispatch-path tests for transition, no-re-fire, backfill, and error surfacing.
- Expand alerts-notifications docs with per-instance 6-hour check cadence, an example version message, a backfill note, and three new troubleshooting entries.
* fix(alerts): harden with security fixes, design compliance, and test coverage
Add authMiddleware to all alert endpoints, validate notification test
dispatch inputs, fix restart_count metric via Docker inspect, correct
network metric units, replace any types with DockerContainerStats
interface, add webhook timeouts and dispatch error tracking.
Frontend: migrate Select to Combobox, add ScrollArea and delete
confirmation AlertDialog, fix icon strokeWidth to 1.5.
Add update availability notifications for both Sencho version updates
(6-hour check in MonitorService) and stack image updates (state
transition detection in ImageUpdateService). Extract shared version
fetch logic into utils/version-check.ts.
Add diagnostic logging gated behind developer_mode for MonitorService
breach state machine and NotificationService dispatch routing.
Tests: 24 new alert API integration tests, restart_count and version
check unit tests (688 total passing). Docs updated with HTTPS
requirement, update notifications section, and troubleshooting guide.
* fix(alerts): remove unused TEST_USERNAME import in alerts-api tests
* fix(dashboard): harden real-time dashboard with bug fixes and design compliance
- Fix host alert spam: add 5-minute cooldown for CPU/RAM/disk threshold
alerts, preventing duplicate notifications every 30s during sustained
breaches. Extract shared dispatchWithCooldown helper (also used by
Docker janitor alerts).
- Fix memory metric inflation: subtract filesystem cache from stored
memory_mb values, matching the existing calculateMemoryPercent logic.
- Fix crash detection reliability: replace fragile 'seconds ago' string
matching with a tracked Set of alerted container IDs. Containers are
only alerted once per crash event, with automatic cleanup when they
start running again or after a 1-hour TTL.
- Fix health status bar: exited containers now trigger 'degraded' state
independently of unread error notifications.
- Fix CPU chart Y-axis: auto-scale when aggregate container CPU exceeds
100% instead of silently clipping at the hardcoded domain ceiling.
- Fix grammar: 'actives' to 'active' in container count label.
- Add shadow-card-bevel to all dashboard cards per design system.
- Update dashboard docs to reflect revised health status thresholds.
* test(dashboard): update monitor service tests for new alert signatures
- Add container Id fields to crash detection test fixtures
- Update host alert assertions to match dispatchWithCooldown 3-arg call
- Fix unhealthy container test to use State: 'unhealthy' instead of
State: 'running' (running containers are now skipped in crash detect)
The Skipper/Admiral atomic deploy/update path used to create
.sencho-backup/ inside the user's stack folder, which silently failed
with EACCES whenever a container had chowned the bind mount (swag,
tautulli, linuxserver/* images, etc). That broke auto-rollback and the
manual rollback endpoint for those stacks. Stack backups now live under
<DATA_DIR>/backups/<stackName>/ next to sencho.db, which is always
writable by the Sencho user.
While stress-testing the same scenario, MonitorService also flooded the
error log with "Error parsing stats for container ... 404 no such
container" because per-container stats polls (30s tick) raced with
docker compose recreating containers. The 404 case is now skipped
silently; non-404 stats failures still log at error level.
Route stack alerts to specific Discord, Slack, or webhook channels instead
of the single global endpoint. Includes per-rule enable/disable, priority
ordering, and automatic fallback to global agents when no rule matches.
- Add notification_routes table, interface, and CRUD in DatabaseService
- Add routing logic in NotificationService.dispatchAlert with optional stackName
- Pass stack context from MonitorService (crash/health) and SchedulerService
- Add 5 API endpoints gated with requireAdmin + requireAdmiral
- Add NotificationRoutingSection UI with Combobox stack picker, channel tabs
- Parallel webhook dispatch via Promise.allSettled
- 10 unit tests covering routing, fallback, and edge cases
- Documentation with screenshots at docs/features/notification-routing.mdx
Add console.warn/console.error logging to 22 silent catch blocks across
10 files. Errors in cleanup, migrations, SSO, fleet snapshots, shutdown,
and validation are now visible in logs. ENOENT guards added to
file-system catches to distinguish missing files from permission errors.
No control flow changes.
- Configurable retention: audit_retention_days setting (1-365 days, default 90)
replaces hardcoded 90-day retention, exposed in Settings > Data Retention
- Export: one-click CSV/JSON export of filtered audit data via new
GET /api/audit-log/export endpoint (capped at 10,000 entries)
- Auditor role: read-only role with system:audit permission for viewing
and exporting audit logs without admin privileges (Admiral tier)
- Enhanced filtering: full-text search across summaries/paths/usernames,
date range picker, and expandable row details showing request path,
IP address, node ID, and entry ID
* feat: audit logging, secrets at rest, and legacy cleanup
- Add Team Pro audit log: records all mutating API actions with user
attribution, searchable timeline UI with filtering and pagination
- Add AES-256-GCM encryption at rest for node API tokens via CryptoService
- Drop 9 legacy SSH/TLS columns from nodes table (dead since v0.7)
- Remove orphaned MaintenanceModal.tsx (dead code, never imported)
- Add requireTeamPro backend guard for team-tier features
- Add audit log cleanup (90-day retention) to MonitorService
- Add docs page and navigation entry for audit log feature
* fix: remove unused AUTH_TAG_LENGTH constant from CryptoService
Security:
- Strip auth credential keys (auth_username, auth_password_hash,
auth_jwt_secret) from GET /api/settings response
- Add allowlist guard to POST /api/settings — rejects unknown or
auth-namespace keys with a 400
Backend:
- Add PATCH /api/settings bulk endpoint with Zod schema validation
(type coercion, range checks, URL format) and atomic SQLite transaction
- Add system_state table — moves last_janitor_alert_timestamp out of
global_settings; adds getSystemState/setSystemState on DatabaseService
- Add metrics_retention_hours and log_retention_days configurable settings;
MonitorService reads both dynamically each evaluation cycle
- Add cleanupOldNotifications(days) to DatabaseService, called each cycle
Frontend:
- Replace single isLoading flag with per-operation states
(isSavingSystem, isSavingDeveloper, isSavingPassword, isSavingRegistry,
isSavingAgent/isTestingAgent per agent type)
- Add skeleton loader that blocks interaction until fetchSettings resolves
- Explicit key-picking in fetchSettings — auth keys cannot enter state
- Unsaved-changes amber dot on System Limits and Developer sidebar items
- Separate saveSystemSettings / saveDeveloperSettings — no cross-tab clobber
- Developer tab gains Data Retention section (metrics hours, log days)
- All settings saves use new PATCH /api/settings endpoint
Add Node modal — type selector & state reset:
- Restored a Local/Remote <Select> dropdown in renderFormFields so users can
explicitly choose the node type instead of it defaulting silently to 'remote'.
- Switching type clears api_url and api_token so no stale remote credentials
carry over if a user switches from Remote to Local mid-form.
- Replaced the static "Add Remote Node" title with a dynamic one that reflects
the currently selected type ("Add Local Node" / "Add Remote Node").
- onOpenChange now resets formData to defaultFormData whenever the dialog
opens, preventing stale values from a previous session leaking in.
Remote connection details — real metrics:
- testRemoteConnection previously returned hard-coded '-' for containers,
images, and cpus after a successful auth/check ping.
- Now fires three parallel requests (Promise.allSettled) after auth passes:
/api/stats → containers total + running count
/api/system/stats → cpu.cores
/api/system/images → image list length
- Each field falls back to '-' gracefully if an endpoint is unavailable,
so a slow or older remote instance never breaks the connection test.
DEP0060 util._extend suppression:
- http-proxy@1.18.1 calls util._extend when createProxyServer() is first
invoked at runtime (NOT at import time). A process.emitWarning override
placed before the proxy instantiations intercepts only DEP0060 without
suppressing any other warnings. No package version changes needed.
Also includes linter/formatter normalisation across multiple files.
Remote nodes in the Distributed API model are self-monitoring — each
remote Sencho instance runs its own MonitorService against its local
Docker socket. The main instance must not attempt direct DockerController
access for remote nodes, which caused fatal crashes on every 30s tick.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Anso