Dashboard and sidebar status indicators previously only refreshed on a
5-30 second polling cadence: a container restart, a degraded -> healthy
transition, or a stack update was invisible until the next tick.
Add a lightweight, non-persisted "state-invalidate" envelope on the
existing /ws/notifications WebSocket:
Backend
- NotificationService.broadcastEvent: sibling of dispatchAlert that
pushes an arbitrary {type, ...} envelope to every subscriber WITHOUT
writing to the alerts history (these are pure ephemeral signals).
- DockerEventService.handleEvent: emit the envelope for state-changing
container actions (start/die/kill/destroy/create/restart/pause/
unpause/health_status/rename/update). Carries node id, stack name
(from the compose project label), container id, action, and
timestamp.
Frontend
- EditorLayout's two notification WebSocket handlers (local plus
per-remote-node) branch on type. On state-invalidate they re-emit a
window CustomEvent and trigger a debounced (250ms) refreshStacks so
a burst of events from compose recreating multiple services
collapses to one refetch. The refresh callback is held in a ref so
the long-lived WS effect never closes over a stale function.
- useDashboardData listens for the same window event and refetches
/stats, /system/stats, and /stacks/statuses on every signal.
Historical metrics stay on their 60s polling cadence (10-minute
trend data, not a live indicator).
Tests
- Three new docker-event-service cases assert broadcastEvent fires on
start and health_status events with the correct envelope shape, and
does not fire on non-state actions like exec_create.
- Existing 28 cases updated with the broadcastEvent mock so the
subscriber stub matches the new shape.
Polling stays as a safety net at the same intervals; the WS path is
the fast path. Multi-node fleets benefit on the local node today;
extending the remote forwarder to relay state-invalidate is a
recommended follow-up.
Previously, fetching the .env file for a stack with no env files at all
returned a 404 with a JSON error body. The frontend's secondary loader
(changeEnvFile) called res.text() without checking res.ok, which caused
the error body to be stuffed directly into the editor as if it were
file content.
Two-part fix:
Backend (routes/stacks.ts):
- For the default GET /stacks/:name/env (no ?file= query) when the
stack has no env files, respond 200 with an empty body and an
X-Env-Exists: false header instead of 404.
- For an explicit ?file= query that resolves to a missing file, keep
the 404 (the caller asked for something specific).
- Catch a TOCTOU ENOENT between access() and readFile() and return the
same friendly empty-body shape, not a generic 500.
Frontend (EditorLayout.tsx::changeEnvFile):
- Check res.ok before reading the body. On a non-OK response, clear the
editor content and surface a friendly toast instead of pasting the
server's JSON error string into the file.
When a registry pushes a new build of an image at the same tag (digest
changes, tag does not), the preview service set next_tag to the same
string as current_tag and the readiness view rendered '10.11 -> 10.11',
which reads as a UI bug.
Add an update_kind field to UpdatePreviewSummary that distinguishes:
- 'tag' - at least one image has a strictly newer tag
- 'digest' - the only updates available are same-tag rebuilds
- 'none' - nothing to apply
The frontend now branches on update_kind and renders 'Rebuild available'
next to the current tag for the digest case, leaving the version-arrow
diff for genuine tag bumps.
Three new buildSummary cases lock in the kind classification.
The Docker janitor watchdog had three problems on multi-node fleets:
1. The alert text said "Your system has accumulated X GB" with no node
identifier, so on a fleet view the operator could not tell which
node was complaining. Resolve the local node via NodeRegistry and
put the node name in the message.
2. The threshold gate was a single comparison against the user's
configured GB value. A small or accidentally tiny threshold made
the alert fire on hosts with effectively no waste. Add a 100 MB
absolute floor so trivial cruft never triggers a notification.
3. The unit parser only matched uppercase "GB|MB|KB|B" and dropped
"TB" entirely. Modern Docker emits "kB" with a lowercase k, which
silently contributed zero bytes to the running total. Normalise the
unit to uppercase before the comparison and add the TB case.
Vulnerability scans cache their policy verdict as a JSON blob in
vulnerability_scans.policy_evaluation. Deleting a scan policy used to
remove only the policies row and leave those blobs intact, so the
scheduler kept emitting violations and stacks remained marked as blocked
against a policy that no longer existed.
deleteScanPolicy now nulls out policy_evaluation on every scan whose
JSON references the deleted policy id, then deletes the policy row, in
one transaction.
Updates the backend listen port, Vite dev proxy target, Docker EXPOSE,
compose port mapping, .env.example default, GitHub Actions smoke-test
default, healthcheck URLs, and every doc/example reference. Test fixtures
that include example URLs were updated for consistency, though their
assertions are port-agnostic.
The rate-limit value of 3000 in middleware/rateLimiters.ts and the
3000 entry in WEB_UI_PORTS (which detects user containers like Grafana)
are intentionally untouched.
Fresh installs land on the Community tier. The 14-day Admiral trial is now
issued by Lemon Squeezy via their hosted checkout: the user enters email +
card, receives a license key by email, and pastes it into the existing
Settings > License activation field.
Backend changes:
- LicenseService.initialize() no longer auto-creates a license_status='trial'
row on first boot. It now only ensures an instance_id exists and starts
periodic validation.
- Drop the TRIAL_DURATION_DAYS constant.
- Drop the status='trial' early-return in getVariant() so LS-issued trials
resolve through the normal variant metadata path (variant_name / product_name).
- Trial branches in getTier() and getLicenseInfo() are retained for future
work that may detect trial state from Lemon Squeezy metadata; they are
currently unreachable via the Sencho code paths.
Frontend changes:
- Settings > License surfaces a new "Try Admiral free for 14 days" CTA block
with Start monthly trial and Start annual trial buttons that open Lemon
Squeezy hosted checkout. The CTA is visible only when the user has no paid
access and is not already on a trial.
- Reserve the Admiral upgrade card for the Skipper-active upgrade path so
unlicensed users see one Admiral path (the trial CTA) instead of two.
- Pull the inline Lemon Squeezy checkout URLs into named module constants so
the Skipper, Admiral monthly, and Admiral annual endpoints are defined in
one place.
Test changes:
- license-service.test.ts covers the no-auto-trial startup path and updates
the trial-variant test to reflect the metadata-driven resolution.
- afterAll in the initialize() describe block calls destroy() so the 72-hour
validation interval does not leak into sibling test files.
Docs:
- Rewrite the Free trial section in features/licensing.mdx to document the
new LS checkout flow (email + card required, auto-converts on day 14 unless
cancelled).
- Add an operations/troubleshooting entry for cases where the trial license
key email does not arrive.
Custom OIDC stays on Community so self-hosters can wire any spec-compliant
OIDC identity provider (Authelia, Keycloak, Authentik, Zitadel, and others).
Google, GitHub, and Okta one-click presets move to Skipper. LDAP / Active
Directory and scoped RBAC are Admiral-only.
Backend enforces the split via a new requireTierForSsoProvider helper in
middleware/tierGates.ts, applied after requireAdmin in all four ssoConfig
mutation handlers. GET /sso/config (list) stays ungated so downgraded
admins can still see previously-configured providers. Invalid provider ids
now 400 before the tier check to avoid leaking tier information.
Frontend adds a compact mode to PaidGate and AdmiralGate for inline
list-item locks, and SSOSection reorders the provider cards as
Custom OIDC > Google > GitHub > Okta > LDAP to reinforce the
free-to-paid progression.
Stale 'SSO is Admiral' copy in AdmiralGate, PaidGate, and the Admiral
upgrade card on the License settings page has been replaced to reflect the
new split. User-facing licensing, SSO, overview, quickstart, and security
docs have been updated with the per-tier provider matrix.
The canonical middleware-order comment in app.ts carried historical
notes from the index.ts refactor ("before Phase 4 finishes", "moves to
routes/* in Phase 4", "moves here in Phase 5") that are no longer
active-voice descriptions of current state. Replace with plain
descriptions matching the final module layout. The 16-step enumeration
and the invariant paragraph about public routers (metaRouter, authRouter,
mfaRouter, ssoRouter) sitting before the auth gate are preserved.
No behavior change.
The index.ts refactor inverted the proxy mount order. The pre-refactor
monolith mounted `app.use('/api/', remoteNodeProxy)` before any inline
route, so remote-nodeId requests short-circuited into the proxy. After
the refactor the proxy was registered after every per-group router, so
Express matched local routers first and remote-nodeId requests were
silently handled with the control instance's local state (e.g.
GET /api/stacks with x-node-id=<remote> returned local stacks rather
than the remote's).
Fix moves createRemoteProxyMiddleware() between enforceApiTokenScope
and the first per-group router, matching middleware-order.md step 13
and restoring pre-refactor behavior. PROXY_EXEMPT_PREFIXES continues to
cover gateway-level paths (auth, nodes, license, fleet, webhooks, meta)
that must stay local even when x-node-id targets a remote.
Add four regression guards that would have caught this:
- json-parser-bypass.test.ts: asserts conditionalJsonParser leaves the
request stream intact on proxy-eligible paths so http-proxy can pipe
the raw body to the upstream; spins up a local echo server and
verifies the bytes arrive.
- proxy-mount-order.test.ts: asserts a remote-nodeId GET short-circuits
into the proxy (502 from unreachable upstream) instead of matching a
local router (200 from local state).
- upgrade-order.test.ts: pins WebSocket dispatch order by observing
handler-specific side effects for notifications, remote forwarder,
logs, and pilot tunnel.
- remote-console-session.test.ts: asserts the HTTP console-token route
mints a JWT with the same claim shape as the shared mintConsoleSession
helper, so gateway and WS forwarder tokens remain interchangeable.
Full suite: 73 files, 1,358 tests, all passing.
Move the startup and shutdown lifecycles out of index.ts:
- bootstrap/startup.ts exports startServer(server) - migration check,
service initialization, background watchdogs, HTTP listen, pilot-agent
loopback bind.
- bootstrap/shutdown.ts exports installShutdownHandlers(server) -
SIGTERM/SIGINT handlers, in-order service stop chain, 10s force-exit
guard, SQLite close.
Restructure MfaService to add an instance + lifecycle so the replay
purge timer no longer lives as a module-scope setInterval in index.ts.
MfaService keeps all existing static methods (generateSecret, verifyTotp,
currentWindow, generateBackupCodes, hashBackupCodes, verifyBackupCode,
formatBackupCodeForDisplay, normalizeBackupCode, buildOtpauthUri) so
every existing caller stays unchanged. The new start() / stop() pair
is idempotent and calls .unref() so test shutdown is not blocked.
bootstrap/startup calls MfaService.getInstance().start().
bootstrap/shutdown calls MfaService.getInstance().stop().
index.ts drops from 305 to 147 lines and now contains only the Express
app composition: createApp, route mounts, remote proxy, createServer,
attachUpgrade, static/SPA fallback, errorHandler, installShutdownHandlers,
and the require.main guard that boots the server when run directly.
Behavior is byte-for-byte identical: shutdown service order, log
strings, force-exit timer, pilot-agent loopback logic, and the MFA
purge cadence and debug logging all preserved verbatim.
Move the nine /api/nodes/* endpoints out of index.ts into routes/nodes.ts
(list, scheduling-summary, get, create, pilot-enroll, update, delete,
test, meta). mintPilotEnrollment and the REMOTE_META_* constants move
with the router as local helpers.
Handlers moved verbatim. Two safe cleanups applied during the move:
- Inline req.apiTokenScope 403 blocks replaced with the shared
rejectApiTokenScope helper; payload shape unchanged.
- catch (error: any) rewritten to catch (error: unknown) with explicit
instanceof Error narrowing to satisfy the no-any strictness rule.
Response body shapes unchanged.
Removes now-unused imports from index.ts: jwt, crypto, authMiddleware,
isValidRemoteUrl, PilotTunnelManager, PilotCloseCode, CAPABILITIES,
getSenchoVersion, fetchRemoteMeta, RemoteMeta, FleetUpdateTrackerService,
plus the module-scope updateTracker alias.
index.ts drops from 1364 to 1021 lines. Only the stacks group remains
inline for the final phase 4c slice.
Move the six /api/containers/* and /api/ports/in-use endpoints out of
index.ts. Handlers moved verbatim. routes/containers.ts exports two
routers:
- containersRouter (mounted at /api/containers): list, stream logs,
start, stop, restart.
- portsRouter (mounted at /api/ports): /in-use host port inventory.
Removes the now-unused requireAdmin import from index.ts. Middleware
chains, response shapes, and error messages are all preserved.
index.ts drops from 1437 to 1364 lines. Remaining inline groups:
stacks and nodes.
Move the entire /api/security/* surface out of index.ts:
- Trivy lifecycle: status, install, uninstall, update-check, update, auto-update toggle
- Scanning: POST /scan (image), POST /scan/stack (compose)
- Scan queries: list, get, vulnerabilities, secrets, misconfigs, image-summaries, SARIF export
- SBOM generation
- Scan policies CRUD (fleet-replicated; replica writes rejected)
- CVE suppressions CRUD (fleet-replicated; replica writes rejected)
- GET /compare: diff two scans
Handlers moved verbatim. Local CVE_ID_RE, parseScannersInput, and
shapeScanForResponse helpers move with the router. The previously-closure
fetchAll inside the SARIF handler is hoisted to module scope as
fetchAllPages so it is not re-allocated per request.
Removes nine now-unused imports from index.ts (parsePolicyEvaluation,
VulnerabilityScan, FleetSyncService, requireAdmiral, trivyInstallLimiter,
SbomFormat, TrivyInstaller, validateImageRef, applySuppressions,
generateSarif).
index.ts drops from 2130 to 1437 lines. Remaining inline groups:
containers, stacks, and nodes.
Move three more Round C route groups out of index.ts:
- /api/registries/* -> routes/registries.ts
- /api/system/{orphans,prune,docker-df,resources,images,volumes,networks,...} -> routes/systemMaintenance.ts
- /api/templates/* -> routes/templates.ts
Handlers moved verbatim. Middleware chains, response shapes, and error
messages are preserved. Registry scope-denial now uses the shared
rejectApiTokenScope helper (same payload shape).
index.ts drops from 2739 to 2130 lines. Remaining inline groups:
containers, stacks, security (Trivy/scans/SBOM/policies), and nodes.
Move four small Round C route groups out of index.ts:
- /api/auto-heal/* -> routes/autoHeal.ts
- /api/notifications/*, /api/notification-routes/* -> routes/notifications.ts
- /api/system/console-token -> routes/console.ts
- /api/sso/config/* -> routes/ssoConfig.ts
Share NOTIFICATION_CHANNEL_TYPES, cleanStackPatterns, and validateHttpsUrl
via a new helpers/notificationChannels.ts so agents.ts and notifications.ts
consume the same allowlist and URL validator.
Handlers moved verbatim. Middleware chains and response shapes preserved.
index.ts drops from 3231 to 2739 lines.
Wraps up Phase 4 Round B by tackling the two deferred groups. 25 new
integration tests land first and run green against the inline monolith,
then each group is extracted byte-for-byte.
index.ts drops from ~3,678 to ~3,231 lines; test count rises 1,320 → 1,345.
New coverage:
- metrics-routes.test.ts (11) — auth + shape checks for /api/stats,
/api/metrics/historical, /api/system/stats, /api/system/cache-stats
(admin-only), and SSE headers for /api/logs/global/stream
- image-updates-routes.test.ts (14) — auth, admin gating, rate-limit
tolerance on /refresh, fleet aggregation, /auto-update/execute input
validation and no-stacks short-circuit
New route files:
- routes/metrics.ts — /stats, /metrics/historical, /logs/global (+ SSE
/stream), /system/stats, /system/cache-stats. Mounted at /api so the
mixed sub-paths line up.
- routes/imageUpdates.ts — /api/image-updates CRUD + fleet aggregation,
plus a separate autoUpdateRouter mounted at /api/auto-update that
owns the /execute handler. Same split pattern as license.ts +
systemUpdateRouter.
index.ts trims unused imports left behind by the extraction:
globalDockerNetwork, si, STATS_CACHE_TTL_MS, SYSTEM_STATS_CACHE_TTL_MS,
GlobalLogEntry + log-parsing helpers.
Round B of Phase 4. Writes integration tests for three under-covered
route groups BEFORE extracting them, then does the extraction once the
new tests pass against the monolith. index.ts drops from ~4,206 to
~3,678 lines.
New test coverage (42 new assertions):
- settings-routes.test.ts (14) — auth, admin gating, private-key stripping,
allowlist, single-key write, bulk PATCH validation + partial update
- scheduled-tasks-routes.test.ts (18) — list/create/get/toggle/delete/runs,
action+target_type matrix, cron validation, tier gating on non-admin
- agents-routes.test.ts (10) — GET/POST, admin gating, channel type +
HTTPS URL validation, boolean enabled check, upsert semantics
Each suite was verified against the inline monolith first, then the
route extraction was performed byte-for-byte and all suites re-run to
ensure no regression.
New route files:
- routes/settings.ts — GET/POST/PATCH with PRIVATE_SETTINGS_KEYS strip,
ALLOWED_SETTING_KEYS allowlist, and SettingsPatchSchema zod bulk schema
- routes/scheduledTasks.ts — 9 endpoints (list, create, get, update,
delete, toggle, run-now, runs history, runs CSV export). File-local
helpers parseTaskId, validateActionTarget, validateOptionalFields
collapse duplication across create+update handlers. Uses shared
escapeCsvField from utils/csv.ts.
- routes/agents.ts — notification-channel GET/POST. Owns
NOTIFICATION_CHANNEL_TYPES and validateHttpsUrl locally because the
notification-routes block still inlines identical copies; the helpers
will converge once those routes extract in a later slice.
Final slice of Phase 4 Round A. Pulls the four remaining well-tested route
groups out of index.ts. index.ts drops from ~5,930 to ~4,206 lines.
New route files:
- routes/webhooks.ts: /api/webhooks CRUD + HMAC-authenticated trigger.
Uses shared webhookTriggerLimiter. Trigger preserves the raw-body path
established by the conditional JSON parser for HMAC validation.
- routes/users.ts: /api/users CRUD + /:id/mfa/reset + /:id/roles
scoped-assignment surface. Uses rejectApiTokenScope across every
handler, validateUsername helper, BCRYPT_SALT_ROUNDS, and
isSqliteUniqueViolation for the role-assignment UNIQUE guard.
- routes/gitSources.ts: /api/git-sources + /api/stacks/:name/git-source/*.
Exports two routers (gitSourcesRouter + stackGitSourceRouter) because
the per-stack paths need to mount at /api/stacks alongside the label
routes extracted in phase 4a-1. String length limits are now named
constants so the 400 responses stay truthful if the bounds change.
- routes/fleet.ts: /api/fleet role, sync, overview, node drill-down,
update-status + trigger (single + fleet-wide), and snapshot CRUD +
restore. Local parseIdParam helper collapses seven copies of the
parseInt/isNaN route-param pattern.
Bugs fixed during review:
- users.ts :id/roles POST — replace the fragile
(err as Error).message?.includes('UNIQUE constraint') check with
isSqliteUniqueViolation from utils/errors.ts.
index.ts carries forward three symbols (updateTracker alias,
CVE_ID_RE, parseScannersInput) until the corresponding security /
nodes / scan routes get extracted in a later slice.
Second slice of Phase 4. Pulls the three auth-family route groups out of
index.ts into focused routers. All handlers move verbatim; index.ts drops
~845 lines.
New route files:
- routes/auth.ts: /api/auth core (status, setup, login, password, logout,
check, generate-node-token)
- routes/mfa.ts: /api/auth/login/mfa + full /api/auth/mfa/* surface
(status, enroll start/confirm, disable, backup-codes/regenerate,
sso-bypass)
- routes/sso.ts: /api/auth/sso/{providers,ldap,oidc/:provider/authorize,
oidc/:provider/callback} + getSSOBaseUrl helper. Module-load calls
SSOService.getInstance().seedFromEnv() so env-seeded providers are
available on the first request.
Shared lifts:
- helpers/constants.ts: MFA_REPLAY_TTL_MS + MFA_REPLAY_PURGE_INTERVAL_MS
(used by mfa.ts and the startup purge timer in index.ts) and
BCRYPT_SALT_ROUNDS (shared between setup and password-change handlers).
- middleware/auth.ts: new reissueSessionAfterTokenBump(req, res, userId)
helper collapses three copies of "bump → fetch user → re-sign cookie"
across auth.ts (password change) and mfa.ts (enrol confirm, disable).
Code review fixes:
- File-local requireEnrolledMfaUser helper in mfa.ts eliminates four
copies of "auth check + rejectApiTokenScope + load enrolled MFA" with
near-identical shape.
- Applied BCRYPT_SALT_ROUNDS to auth.ts setup + password handlers.
Mount order in index.ts: authRouter / mfaRouter / ssoRouter sit before
authGate because login / setup / SSO-callback are public; handlers that
need auth use authMiddleware directly on the route.
First slice of Phase 4 (route extraction). Pulls 8 well-tested, mostly
independent route groups out of index.ts into focused Router files. No
behavior change; every handler body moves verbatim.
New route files under backend/src/routes/:
- meta.ts /api/health, /api/meta (mounted before authGate)
- license.ts /api/license/* + /api/system/update,
exports scheduleLocalUpdate for the fleet route
- permissions.ts /api/permissions/me
- convert.ts POST /api/convert
- alerts.ts /api/alerts/*
- labels.ts /api/labels/* + PUT /api/stacks/:name/labels
(exported as stackLabelsRouter)
- apiTokens.ts /api/api-tokens/*
- auditLog.ts /api/audit-log/*
Shared helper lifts:
- helpers/cacheInvalidation.ts: invalidateNodeCaches()
- middleware/tierGates.ts: requireBody (was inline in index.ts)
- utils/errors.ts: isSqliteUniqueViolation (was inline in index.ts)
- middleware/apiTokenScope.ts: rejectApiTokenScope() helper (new)
- utils/csv.ts: escapeCsvField() (new)
index.ts drops from ~7520 to ~6775 lines and now mounts the routers right
after enforceApiTokenScope. The remote proxy and fleet/auth/webhooks/users
routes remain inline in index.ts pending later Phase 4 slices.
Code review fixes: rejectApiTokenScope helper replaces duplicated
`if (req.apiTokenScope) 403 SCOPE_DENIED` blocks in apiTokens.ts and
license.ts; escapeCsvField replaces the inline CSV escape in auditLog.ts.
Phase 3 of the index.ts refactor. Pulls the remote HTTP/WS proxy plumbing,
the WebSocket upgrade dispatcher, and the http/WSS construction out of the
monolith. index.ts drops roughly 620 lines.
New modules:
- proxy/websocketProxy.ts: shared httpProxy.createProxyServer singleton
(used by both the HTTP proxy middleware and the remote WS forwarder)
- proxy/remoteNodeProxy.ts: createRemoteProxyMiddleware() factory; consumes
the isProxyExemptPath helper instead of open-coding the prefix list
- server.ts: createServer(app) returns { server, wss, pilotTunnelWss }
- services/FleetUpdateTrackerService.ts: singleton wrapping the in-flight
fleet update tracker Map with create()/resolve() helpers
- helpers/consoleSession.ts: mintConsoleSession(), isConsoleSessionScope()
- websocket/upgradeHandler.ts: attachUpgrade(server, deps) dispatcher that
runs the manual cookie/JWT verify and delegates to sub-handlers
- websocket/pilotTunnel.ts: handlePilotTunnel (pilot_enroll consumption and
pilot_tunnel registration)
- websocket/notifications.ts: /ws/notifications local subscriber
- websocket/remoteForwarder.ts: remote-node WS proxy with console_session
token exchange for interactive paths
- websocket/logs.ts: /api/stacks/:name/logs supervisor stream
- websocket/hostConsole.ts: /api/system/host-console PTY, Admiral-gated
- websocket/generic.ts: /ws exec + streamStats action dispatch, owns the
terminalWs single-instance reference
- websocket/reject.ts: shared rejectUpgrade helper (replaces five copies)
Service extension:
- NotificationService: setBroadcaster(fn) replaced by subscribe(ws) that
returns an unsubscriber; broadcastToSubscribers is now internal. Subscriber
set lives on the service rather than in index.ts.
Wiring in index.ts:
- const app = createApp() already in place from Phase 2
- const { server, wss, pilotTunnelWss } = createServer(app)
- attachUpgrade(server, { wss, pilotTunnelWss })
- app.use('/api/', createRemoteProxyMiddleware())
- /api/system/console-token route now uses mintConsoleSession()
- deploy/down/update routes read the streaming target via getTerminalWs()
(return type is WebSocket | undefined so the || undefined fallback is gone)
Code review fixes: five duplicated reject helpers collapsed into
websocket/reject.ts; dropped the createTracker/resolveTracker bind
aliases in index.ts so call sites go through the service directly;
removed em dashes; replaced req.url! with req.url || '/'.
Phase 2 of the index.ts refactor. Pulls the auth middleware and session
cookie issuers into their own module, and introduces the app.ts factory
that owns the first nine steps of the canonical middleware pipeline.
New modules:
- middleware/auth.ts: authMiddleware, issueSessionCookie,
issueMfaPendingCookie, clearMfaPendingCookie
- app.ts: createApp() factory installing trust proxy, helmet, cors,
compression, cookieParser, rate limiters, conditionalJsonParser, and
nodeContextMiddleware. A header comment documents all 16 canonical
middleware steps and where each currently lives.
Changes:
- middleware/authGate.ts: createAuthGate factory removed; authGate now
imports authMiddleware directly (the factory existed only to avoid a
circular import while authMiddleware lived in index.ts).
- services/DatabaseService.ts: added API_TOKEN_SCOPE_TO_ROLE map so the
auth middleware no longer inlines a stringly-typed record.
- index.ts drops ~260 lines; auth routes, authGate, auditLog,
apiTokenScope, remaining routes, static serving, and the error handler
continue to be registered there until their respective phases.
- vitest.config.ts bumps testTimeout to 30s and hookTimeout to 45s so
fork-pool workers have enough headroom to ts-node-transform the
growing module graph under CPU contention (64 workers each import the
full Express stack in beforeAll).
Code review fixes: use getErrorMessage() util in the auth catch block
instead of inline cast; promote the scope-to-role map to a typed
module-level constant.
Phase 1 of the index.ts monolith refactor. Extracts all non-auth middleware
into focused modules. Routes and authMiddleware itself stay in index.ts for
now; Phase 2 introduces createApp() and extracts authMiddleware.
New modules:
- middleware/rateLimiters.ts: globalApiLimiter, pollingLimiter,
webhookTriggerLimiter, authRateLimiter, ssoRateLimiter, trivyInstallLimiter
plus the hybrid rateLimitKeyGenerator and isNodeProxyRequest helper
- middleware/jsonParser.ts: conditionalJsonParser that preserves the raw
stream for remote-proxy forwarding (via helpers/proxyExemptPaths)
- middleware/nodeContext.ts: nodeContextMiddleware
- middleware/apiTokenScope.ts: enforceApiTokenScope + DEPLOY_ALLOWED_PATTERNS
- middleware/authGate.ts: createAuthGate(authMiddleware) factory + auditLog.
Factory takes authMiddleware as a dependency to avoid a circular import
until Phase 2 extracts the auth module.
- middleware/errorHandler.ts: central error handler that preserves
err.status / err.expose from body-parser and other HTTP errors
- helpers/routePatterns.ts: WEBHOOK_TRIGGER_RE shared by rateLimiters and
authGate
index.ts drops ~290 lines. Middleware registration order is unchanged.
All 1278 tests pass.
Code review fixes: typed ApiTokenScope in apiTokenScope.ts; replaced 2
em dashes with colons (Directive 18); added local CachedProxyFlagReq
type alias for the node_proxy memoization cast; extracted deny() helper.
* refactor(backend): extract types, constants, and guards from index.ts (phase 0)
Additive, behavior-preserving first step of the modular backend refactor.
Moves purely static artifacts out of backend/src/index.ts so later phases can
extract routes and middleware without touching shared symbols.
New modules:
- types/express.ts: Express Request augmentation
- helpers/constants.ts: PORT, password policy, label colors, cookie names,
MFA TTLs, hot-path cache TTLs
- helpers/proxyExemptPaths.ts: PROXY_EXEMPT_PREFIXES + isProxyExemptPath
- helpers/cookies.ts: isSecureRequest, getCookieOptions
- helpers/policyGate.ts: buildPolicyGateOptions, runPolicyGate,
triggerPostDeployScan
- middleware/permissions.ts: ROLE_PERMISSIONS, checkPermission,
requirePermission
- middleware/tierGates.ts: requirePaid, requireAdmiral, requireAdmin,
requireNodeProxy, requireScheduledTaskTier + effectiveTier/Variant
index.ts shrinks by ~260 lines; no runtime behavior changes. All 64 vitest
files and 1,278 tests pass.
* refactor(backend): drop unused imports left after phase 0 extraction
LicenseTier, LicenseVariant, DIGEST_CACHE_TTL_MS, and isProxyExemptPath
were imported into index.ts but no longer referenced there after the
phase 0 move; CI lint flagged them as errors.
isProxyExemptPath will be re-imported in phase 1 when the JSON parser
bypass and nodeContext middleware get extracted. Silence the
no-namespace warning on the Express augmentation since the namespace
syntax is required for TypeScript module augmentation.
* feat(security): polish scan sheets, fix CVE links, surface policy violations
Adds cveUrl helper that rewrites Trivy's 404-ing avd.aquasec.com links to
cve.org for CVE-prefixed IDs (GHSA and misconfig URLs pass through unchanged).
Redesigns both scan sheets with shadow-card-bevel chips, tracked-mono kickers,
severity row tinting with a left accent rail, and tabular-nums timestamps.
Surfaces a destructive policy-violation banner on scans whose policy_evaluation
row flags a block, and fixes the compare sheet's delta ribbon so CRITICAL
net-positive deltas render in destructive (not warning) tone. Backend parses
the JSON policy_evaluation column at the API boundary so the UI receives a
structured object.
* chore(security): suppress CVE-2026-32281 and CVE-2026-32283 in Trivy scan
Both CVEs affect Go stdlib crypto/x509 and TLS in Docker CLI 29.4.0
(Go 1.26.1) and Compose v5.1.2 (Go 1.25.8). No upstream static binary
has been released with the patched Go 1.26.2 or 1.25.9 runtimes yet.
Exposure analysis: the Docker CLI and compose plugin connect to the local
Docker socket (Unix socket, no TLS) and to public registries with well-known
CAs. Neither CVE is exploitable in this configuration. Added alongside
sibling entries already in .trivyignore for the same binary versions.
Revisit on next Docker CLI and Compose upstream release.
Policies with block_on_deploy=1 now scan every stack image before
docker compose up runs and reject the deploy with HTTP 409 on violation.
The UI opens a dialog listing offending images; admins can override per
deploy with ?ignorePolicy=true, and every bypass is recorded in the
audit log with the originating route, actor, policy, and image list.
When Trivy is not installed on the target node the gate fails open with
a warning notification, so teams are never locked out by tooling state.
Post-deploy and scheduled scans still evaluate matching policies and
dispatch warnings on violations to surface drift on long-running stacks.
Public API additions: policy and suppression CRUD under /api/security,
plus the documented 409 block-response shape on all deploy paths.
The `trivy config` subcommand does not accept `--no-progress`; the flag
exists only on `trivy image`. Every stack configuration scan therefore
failed with `FATAL Fatal error unknown flag: --no-progress`, and the
"Scan configuration" action on the stack details page has been
non-functional since it shipped.
Removing the flag is the complete fix. `trivy config` is silent by
default, so the flag was redundant even if it had been accepted.
A new Vitest spec pins the exact argument vector
(`['config', '--format', 'json', '--quiet', <stackPath>]`) so a future
edit cannot reintroduce the bug, and exercises the success path end to
end: status transitions to `completed`, misconfig severities tally
correctly, and `highest_severity` rolls up to the worst finding.
Flip Security (Trivy), Notifications (agents + history), and App Store from
global-and-hidden-on-remote to node-scoped so operators can manage them when a
remote node is selected in the node picker. The primary instance proxies the
calls to each remote, which resolves the correct per-instance binary state,
agent config, and template registry.
Backend: key `agents` and `notification_history` by `node_id` with idempotent
column-add migrations and a `(node_id, type)` unique index on agents, matching
the Labels pattern. Thread `req.nodeId` through the /api/agents and
/api/notifications routes. Internal NotificationService and ImageUpdateService
writes resolve the middleware default via `NodeRegistry.getDefaultNodeId()` so
monitor-emitted rows share a bucket with user-facing ones (avoids split-brain
where the UI sees test notifications but not internal alerts).
Frontend: split Security on remote to render only the scanner card and hide
scan policies and CVE suppressions (those remain control-plane-only). Drop the
misleading "Always Local" badge on Developer since retention windows govern
backend jobs, not UI state. Flip the App Store registry to node-scoped.
Docs: add a "What Settings apply per node" table to multi-node, clarify
remote alert setup in alerts-notifications, and note Trivy's per-host install
in vulnerability-scanning.
* fix(app-store): use stack name as compose service key
App Store deployments hardcoded the compose service to "app", so every
container's com.docker.compose.service label collapsed to "app". Global
logs and AutoHeal policies key off that label, making it impossible to
distinguish between deployed apps. Pass the already-validated stack
name through to generateComposeFromTemplate so each app gets a
descriptive service identifier.
* test(template-service): use {2} quantifier to satisfy no-regex-spaces
ESLint's no-regex-spaces rule flagged two consecutive spaces in the
regex literal as an error. Swap for the {2} quantifier to keep the
assertion identical while clearing the lint gate.
---------
Co-authored-by: Claude <noreply@anthropic.com>
Rework the Logs view into the cockpit language: a PageMasthead strip with
cyan rail, pulsing live dot, italic state word, and tracked-mono kicker;
a SignalRail of EVENTS/MIN (with rolling 60s sparkline), ERRORS, WARNINGS,
and CONTAINERS tiles; a segmented filter strip; a day-banded feed with
severity dots and whole-row tint on ERROR/WARN; and a floating glass chip
strip for pause/resume, clear, and download.
Decouple the live log stream from the Developer Mode toggle so SSE runs by
default for everyone, with polling as an invisible fallback when
EventSource is unavailable. Drop the Standard Log Polling Rate setting
and the global_logs_refresh key it persisted; Developer Mode still gates
Real-Time Metrics and Debug Diagnostics and nothing else.
Introduces the shared PageMasthead and SignalRail primitives for reuse
across other cockpit pages.
Add stack_name and container_name columns to notification_history so bell
rows can act as jump points. Producers (AutoHeal, Docker events) pass the
container context through dispatchAlert; the panel renders routable rows
as buttons that load the target stack and, when a container name is
present, open its logs modal. Non-structural notifications stay as
passive display rows.
* feat(stack-view): per-container health strip and structured logs viewer
Replaces the flat container list with a per-container health strip showing
healthcheck state, uptime, port mapping with an open-app link, and live
cpu/memory/network sparklines fed by a 60-sample ring buffer on the stats
WebSocket.
Adds a structured logs viewer that parses docker timestamps (emitted by the
-t flag on the logs stream) and classifies each line by level. Rows render
as a DOM grid with filter pills (all / info / warn / err with count),
following indicator, and plain-text download. A segmented toggle switches
between the structured viewer and the original xterm view; the choice is
persisted in localStorage.
* fix(stack-view): disable no-control-regex for ANSI escape pattern
ANSI escape sequences start with ESC (0x1B), which is a control character.
The regex is intentional and cannot be rewritten without it.
* feat(stack-view): identity header with health state and action hierarchy
Redesign the stack view header around three questions: what is this, is it
healthy, what does it do. Surface Docker healthcheck state, primary image
tag, and image digest; group actions by frequency.
- Backend: extend /api/stacks/:name/containers with healthStatus (healthy /
unhealthy / starting / none), Image, and ImageID by inspecting each
container in parallel.
- Frontend: replace flat CardHeader with breadcrumb, italic serif title,
colored state pill with pulse, and a mono image/digest line with a
one-click copy button for the full digest.
- Frontend: action hierarchy - primary cyan Restart/Start, outline Stop and
Update, and an overflow menu for Rollback, Scan config, and Delete.
- Docs: new Stack header section and updated controlling-a-running-stack
tables showing primary/secondary/overflow grouping.
* test(e2e): open overflow menu to reach stack Delete action
Destructive actions now live under the stack toolbar overflow menu rather
than as a flat top-level button, so the delete flow must click More actions
before selecting the Delete menu item.
Add a Stream view to the Audit Log that leads with a four-tile signal
rail (events, actors, failure rate with inline sparkline, peak hour)
and presents the feed grouped by day with severity dots, relative
times, and inline anomaly callouts. The existing Table view is
preserved behind a toggle for power users.
Anomaly flags are computed at read time against strictly prior history
and returned on demand via ?with_anomalies=1:
- unusual_hour: hour outside the actor's central 7-day window
- new_ip: IP unseen for this actor in the last 30 days
- first_seen_actor: no prior history in the 30-day window
New /audit-log/stats endpoint returns the signal-rail aggregates over
24h/7d/30d windows; stats are derived from a single 30-day scan.
* feat(backend): add stack update-preview endpoint for readiness board
Adds GET /api/stacks/:stackName/update-preview that returns per-image
semver diff, bump classification, and a stack-level summary powering
the Auto-Update readiness board.
- New UpdatePreviewService parses compose images, inspects local
digests, fetches remote digests and tag lists, and finds the
highest compatible semver tag.
- Major bumps are flagged blocked until human review; unknown bumps
rank below real semver so they cannot mask a major.
- Rollback target is reconstructed through parseImageRef to preserve
registry ports and drop the Docker Hub library/ prefix.
- Registry helpers (httpGet, auth token, digest, tag list, ref parse)
are extracted into registry-api.ts and shared with ImageUpdateService.
- 28 Vitest cases cover parse, selection, bump math, digest rebuilds,
blocked policy, and rollback target construction.
* feat(schedules): next-24h timeline, merge auto-update crud, add readiness board
Replace the flat task table with a Timeline view as the default, showing the
next 24 hours of scheduled work across four lanes (Restart, Update, Scan,
Prune) with a live now rail and per-firing pills. The All tasks tab preserves
the existing CRUD surface.
Merge Auto-update Stack into Schedules as a first-class action and replace the
standalone Auto-Update Policies view with a per-stack Readiness board that
surfaces version diffs, risk tags, changelog previews, and rollback targets
sourced from the stack update-preview endpoint.
* feat(app-store): editorial hero, category rail, security scan signal per tile
Rework the App Store view into an editorial layout with a 180px category
rail, a featured-template hero, and compact tiles that surface star counts
and vulnerability scan status at a glance. The deploy sheet splits into
Essentials (one-click deploy) and Advanced (full port/volume/env control)
tabs.
Backend enriches /api/templates with scan_status, scan_cve_count, and a
featured flag computed from the highest-star template with recorded stars.
Scan lookups use a single batched SQL query to avoid N+1 round-trips.
* fix(app-store): split firstSentence helper into its own module
The firstSentence helper lived alongside the TemplateLogo component, which
violates react-refresh/only-export-components — a file that exports a
component must not also export non-component values, or Fast Refresh cannot
establish an HMR boundary. Move the helper into appstore/util.ts and update
the two import sites.
Replaces the stacked bar + legend with a reclaim-first layout: an amber
hero banner surfaces the total reclaimable bytes and breakdown (unused
images, stopped containers, dangling volumes) with a one-click review &
prune CTA; a three-tile treemap replaces the stacked bar with proportional
areas for Sencho-managed, External, and Reclaimable; and the Volumes tab
gets a two-card landing highlighting the largest volumes by size and
recently changed ones.
* feat(dashboard): status masthead, unified gauges, stack health sparklines
Rework the home dashboard around a single cyan-railed status masthead
that carries the health state word, node meta, and reasons inline. The
resource block collapses into one strip with a CPU hero sparkline,
memory and disk gauge bars, and a network tile whose sparkline is built
from per-container byte-counter deltas. Stack health becomes an
8-column mono table with row tinting, an uptime column sourced from
the oldest running container's creation time, and a per-stack 10-minute
CPU sparkline. Historical charts pick up a cyan gradient and an amber
peak marker. A shared Sparkline primitive backs all of the above.
* docs(dashboard): refresh screenshot for phase B layout
* fix(dashboard): anchor sparkline bucketing to latest metric timestamp
The cpuHistory, netHistory, and cpuPeakLabel memos called Date.now()
inside useMemo, which violates react-hooks/purity: the rule fires
because re-renders can produce different bucket boundaries from the
same inputs. Derive a historyEndAt anchor from the newest metric
sample in the polled series and thread it through to ResourceGauges.
* feat(db): add auto_heal_policies and auto_heal_history schema and CRUD
Adds two new SQLite tables (auto_heal_policies, auto_heal_history) to
DatabaseService.initSchema() and exposes CRUD methods: getAutoHealPolicies,
getAutoHealPolicy, addAutoHealPolicy, updateAutoHealPolicy,
deleteAutoHealPolicy, recordAutoHealHistory, getAutoHealHistory,
incrementConsecutiveFailures, resetConsecutiveFailures, setPolicyEnabled.
Also adds AutoHealPolicy and AutoHealHistoryEntry TypeScript interfaces.
* feat(events): track health-status duration and expose state accessors
- Add healthStatus and unhealthySince fields to InternalContainerState
- onHealthStatus now records unhealthySince timestamp on first transition
to unhealthy, and clears it when the container recovers or restarts
- onStart resets both fields so a restarted container begins from 'starting'
- Add listContainerStates() and getContainerState() public accessors for
use by the upcoming AutoHealService evaluator
* fix(auto-heal): key allowlist in updateAutoHealPolicy, cascade delete, extract ContainerHealthSnapshot
* feat: add AutoHealService evaluator singleton
Polls every 30 s, matches containers to enabled policies via Compose
labels, and restarts containers that have been unhealthy beyond the
configured threshold. Enforces cooldown, per-hour rate cap, and
recent-user-action suppression; auto-disables policies after repeated
consecutive failures. Also adds DockerEventManager.getService() accessor
required by the evaluator.
* fix(auto-heal): prune stale restartTimestamps, guard undefined policy id
- Prune restartTimestamps entries for containers no longer running after
each container list fetch, preventing unbounded map growth from dead
container IDs.
- Guard against policies with undefined id at the start of the per-policy
loop; warn and skip rather than proceed with a non-null assertion.
- Extract handleAutoDisable private helper to bring executeHeal under 30
lines and isolate the auto-disable side-effect sequence.
- Move ContainerInfo type to module scope.
* feat: add auto-heal API routes and wire AutoHealService lifecycle
Registers five REST endpoints under /api/auto-heal/policies (list, create,
patch, delete, history) with requirePaid + requireAdmin guards and Zod
validation. Wires AutoHealService.start()/stop() into the server startup
and graceful-shutdown blocks alongside MonitorService.
* test: add AutoHealService and DatabaseService auto-heal unit tests
- 15 unit tests for AutoHealService.shouldHeal covering all decision branches
(healthy state, duration threshold, user-action suppression, cooldown,
rate limiting, and correct skipReason values)
- 13 integration tests for DatabaseService auto-heal CRUD: policy round-trip,
stack-name filter, partial update, cascade delete, history ordering/limit,
consecutive failure counters, and setPolicyEnabled toggle
* fix: log AutoHealService shutdown errors consistently
* fix(api): requireAdmin-first guard order and try/catch on auto-heal routes
* feat(ui): add StackAutoHealSheet component
* feat(ui): add Auto-Heal context menu item to EditorLayout
* fix(ui): StackAutoHealSheet label, token, a11y, and useEffect fixes
- Rename 'All services in stack' to 'All services' in combobox options and placeholder
- Replace text-green-600 with text-success design token in actionColorClass
- Add htmlFor/id pairs to all four numeric form inputs for accessibility
- Inline fetch logic into useEffect, removing stale closure risk and eslint-disable comment
- Remove now-unused fetchPolicies and fetchServices standalone functions
- Update 'Auto-disable after' label to 'Auto-disable after (failures)' for clarity
- Add toast.error in policy fetch failure path; services fetch silently skips as before
* docs: add auto-heal-policies feature documentation
* test(e2e): add auto-heal policies CRUD spec
* fix(docs): correct auto-heal-policies nav position in docs.json
* feat: pilot agent outbound-mode for remote nodes
Adds a second mode for managing remote nodes: the agent dials an outbound
WebSocket tunnel to the primary, so the remote host no longer needs an
inbound port, a reachable URL, or its own TLS certificate. Works behind
NAT, residential routers, and corporate firewalls.
The primary multiplexes HTTP and WebSocket requests over a single tunnel
via a hybrid JSON + binary frame protocol, bridged through a per-tunnel
loopback server so existing proxy and upgrade handlers route pilot-mode
nodes identically to proxy-mode ones.
Enrollment uses a single-use 15-minute pilot_enroll JWT exchanged for a
long-lived pilot_tunnel credential on first connect. Proxy mode continues
to work unchanged and both modes are supported side-by-side.
* test(e2e): switch to proxy mode before asserting api_url field
Remote nodes default to Pilot Agent mode, which hides the api_url input.
The SSRF-validation tests need proxy mode, so the helper now selects
Distributed API Proxy after picking Remote type before asserting the
field is visible.
* fix(e2e): wire Combobox id prop so node-mode selector resolves
The Combobox trigger button had no id, leaving its Label orphaned and
making getByRole name-based lookups fail. Adding id to the primitive,
passing id="node-mode" from NodeManager, and updating the E2E helper to
use #node-mode fixes both the a11y regression and the CI timeout.
Emit a [Compare:diag] entry gated on developer_mode with the scan ids,
effective tier, input and bucket sizes, suppression count, and
truncation flag. Gives operators a trace to correlate with user reports
of unexpected compare results without adding hot-path cost.
Scan history fetched a fixed 200 most-recent rows and paginated them
client-side, so older scans silently fell off mature nodes where
baselining is most valuable. The list now fetches one page at a time
via offset, with status=completed and imageRefLike filters applied
server-side. Search input is debounced to avoid per-keystroke fetches.
Compare endpoint loads up to 1000 findings per scan. When a scan exceeds
this cap, the response now includes truncated=true and row_limit, and
the comparison sheet surfaces a banner so users understand the diff may
be incomplete. Also exposes total_vulnerabilities on scanA/scanB for UI
use and logs a warning when truncation occurs.
Enrich scheduled vulnerability scan completion notifications with
per-severity CVE counts so recipients can triage from the message body
alone. Expose the scan action in the schedule creation UI, require an
explicit node_id, and harden fire-and-forget alert dispatches so a
failing webhook cannot crash the scheduler.
Notification body now reports scanned/skipped/failed counts plus
critical/high/medium totals aggregated across fresh and cached scans,
reflecting the current node posture rather than only what was newly
scanned on this run.
Adds a new SarifExporter service that builds a SARIF 2.1.0 document
from the stored scan findings (vulnerabilities, secrets, misconfigs).
Rule IDs are namespaced to avoid collisions in a flat result list.
Suppressions carry through as SARIF suppressions[] entries so GitHub
code scanning and Defender for Cloud see the same accepted status
shown in the UI.
Exposed via GET /api/security/scans/:id/sarif, admin + paid-tier
gated to match the SBOM export precedent. A SARIF button appears
in the scan sheet next to SBOM and CSV for paid tiers.
Extends Trivy scans with secret detection in image filesystems and
misconfiguration scanning for Compose stacks. Adds tabs to the scan
drawer for vulnerabilities, secrets, and misconfigs. Secret matches
are redacted server-side (first 8 chars + ellipsis) before storage.
- TrivyService: --scanners vuln,secret for images; trivy config for stacks
- DB: scanners_used/secret_count/misconfig_count cols; secret_findings,
misconfig_findings tables; cache key scoped by scanners
- Routes: POST /security/scan accepts scanners array (requirePaid when
secret requested); POST /security/scan/stack; GET .../secrets and
.../misconfigs (paid-tier reads)
- UI: tabs in VulnerabilityScanSheet; scan-options dropdown on images;
Scan config button on stack header
Operators can accept known-benign findings once and have Sencho filter
them out of scan drawers, comparison views, and other read surfaces.
Suppressions replicate from the control instance to every remote node.
* New cve_suppressions table with a COALESCE-based unique index so NULL
scope slots collide the way users expect
* Admin + paid-tier CRUD routes; writes are rejected on replicas
* Read-time filter enriches vulnerability details and compare payloads
without mutating stored counts
* Settings > Security panel for managing rules, per-CVE suppress action
in the scan drawer, dimmed rows with a shield-off indicator
* Vitest unit tests for the filter (glob, expiry, specificity) and
route tests (auth, tier, replica, UNIQUE conflict)
Anso