* feat(stack): per-stack activity timeline with actor attribution
Adds an Activity tab to the Stack Anatomy panel showing a timestamped
event log for each stack: deploys, restarts, starts, stops, and image
updates, attributed to the user who triggered them or 'system' for
automated actions.
Backend:
- Extends notification_history with actor_username column (idempotent
migration) and a partial composite index on (node_id, stack_name,
timestamp DESC) for efficient per-stack lookups.
- NotificationService.dispatchAlert() accepts an optional actor that
is written to the new column.
- Success-side dispatchAlert calls added after deploy, bulkContainerOp
(start/stop/restart), and update handlers in routes/stacks.ts so
user-initiated operations are recorded, not just failures.
- New GET /api/stacks/:stackName/activity?limit&before endpoint with
stack:read permission gate and cursor-based pagination.
Frontend:
- StackAnatomyPanel grows an Anatomy / Activity tab pair using the
existing Tabs primitive.
- StackActivityTimeline fetches the initial 50 events, paginates on
demand, and prepends live events arriving over the existing WS
notifications stream without duplicates.
- NotificationPanel bell dropdown suppresses user-initiated success
events (start/stop/restart/deploy/update triggered by a real user),
keeping the tray focused on alerts and system events.
* docs(stack): add stack activity timeline feature page and internal arch docs
* fix(test): add actor_username to notification-routing history assertions
dispatchAlert now passes actor_username to addNotificationHistory after
the activity timeline PR added the column. Update the two exact-match
assertions that were failing because the expected object shape was missing
this field.
* feat(dashboard): replace 24h charts with Configuration Status and Recent Activity
The 24-hour CPU/Memory area charts summed per-container metrics normalized
to each container's CPU quota, producing numbers that bore no honest
relationship to host load. The live ResourceGauges strip already shows
accurate host-level stats, making the historical charts both inaccurate
and redundant.
This commit replaces that row with two side-by-side cards:
- **Configuration Status**: aggregates every toggleable feature on the
active node (notification agents, alert rules, routing rules, auto-heal,
auto-update, webhooks, scheduled tasks, MFA, SSO, vulnerability scanning,
cloud backup, and alert thresholds) into a single at-a-glance card.
Tier-locked rows display an upgrade indicator instead of a value.
Each row is clickable and navigates to the relevant settings section.
Data refreshes every 60 s and immediately on state-invalidate events.
- **Recent Activity**: lists the ten most recent notification-history events
for the active node (deployments, image updates, auto-heal actions, scan
findings, cloud backup events, system notices) with category icons and
relative timestamps. Refreshes every 30 s.
New backend endpoints:
- GET /api/dashboard/configuration - per-node feature status with locked/
requiredTier markers so the frontend renders upgrade chips without extra
calls. The endpoint sits after authGate and before the remote proxy so
remote-node requests are transparently forwarded.
- GET /api/dashboard/recent-activity?limit=N - thin wrapper over
DatabaseService.getNotificationHistory.
- GET /api/fleet/configuration - fleet-wide fan-out using the same
Promise.allSettled dead-node-tolerant pattern as /fleet/overview.
Exposed as the new "Status" tab on the Fleet page (after Snapshots).
Shared utilities:
- visibilityInterval and formatCount extracted to frontend/src/lib/utils.ts
so the three polling hooks and two components share a single copy.
* docs(dashboard): fix stale alt text referencing removed historical charts
* feat(cloud-backup): mirror fleet snapshots to S3-compatible storage
Add an Admiral-tier Cloud Backup feature that replicates every fleet
snapshot to off-site storage, with two provider modes that share the
same `@aws-sdk/client-s3` code path:
- Sencho Cloud Backup: zero-config, 500 MB allowance backed by
Cloudflare R2, provisioned via the sencho.io worker against the
user's Lemon Squeezy license.
- Custom S3 (BYOB): any S3-compatible bucket (AWS, MinIO, Backblaze
B2, Wasabi, R2 with own keys), with credentials encrypted via
`CryptoService` before storage.
API-triggered snapshots upload fire-and-forget so the UI returns
immediately; scheduled snapshots block on the upload so the task's
success/failure reflects cloud durability. Object keys include the
instance_id segment to prevent collisions when the same Admiral
license is activated on multiple Sencho instances.
* fix(cloud-backup): drop ES2022-only Error cause arg breaking ES2020 build
The backend tsconfig pins lib to ES2020. The two-argument
`Error(message, { cause })` form requires ES2022, so tsc rejected it
with TS2554. Revert to single-argument throw to match the
convention used elsewhere in the backend services.
The index.ts refactor inverted the proxy mount order. The pre-refactor
monolith mounted `app.use('/api/', remoteNodeProxy)` before any inline
route, so remote-nodeId requests short-circuited into the proxy. After
the refactor the proxy was registered after every per-group router, so
Express matched local routers first and remote-nodeId requests were
silently handled with the control instance's local state (e.g.
GET /api/stacks with x-node-id=<remote> returned local stacks rather
than the remote's).
Fix moves createRemoteProxyMiddleware() between enforceApiTokenScope
and the first per-group router, matching middleware-order.md step 13
and restoring pre-refactor behavior. PROXY_EXEMPT_PREFIXES continues to
cover gateway-level paths (auth, nodes, license, fleet, webhooks, meta)
that must stay local even when x-node-id targets a remote.
Add four regression guards that would have caught this:
- json-parser-bypass.test.ts: asserts conditionalJsonParser leaves the
request stream intact on proxy-eligible paths so http-proxy can pipe
the raw body to the upstream; spins up a local echo server and
verifies the bytes arrive.
- proxy-mount-order.test.ts: asserts a remote-nodeId GET short-circuits
into the proxy (502 from unreachable upstream) instead of matching a
local router (200 from local state).
- upgrade-order.test.ts: pins WebSocket dispatch order by observing
handler-specific side effects for notifications, remote forwarder,
logs, and pilot tunnel.
- remote-console-session.test.ts: asserts the HTTP console-token route
mints a JWT with the same claim shape as the shared mintConsoleSession
helper, so gateway and WS forwarder tokens remain interchangeable.
Full suite: 73 files, 1,358 tests, all passing.
Move the startup and shutdown lifecycles out of index.ts:
- bootstrap/startup.ts exports startServer(server) - migration check,
service initialization, background watchdogs, HTTP listen, pilot-agent
loopback bind.
- bootstrap/shutdown.ts exports installShutdownHandlers(server) -
SIGTERM/SIGINT handlers, in-order service stop chain, 10s force-exit
guard, SQLite close.
Restructure MfaService to add an instance + lifecycle so the replay
purge timer no longer lives as a module-scope setInterval in index.ts.
MfaService keeps all existing static methods (generateSecret, verifyTotp,
currentWindow, generateBackupCodes, hashBackupCodes, verifyBackupCode,
formatBackupCodeForDisplay, normalizeBackupCode, buildOtpauthUri) so
every existing caller stays unchanged. The new start() / stop() pair
is idempotent and calls .unref() so test shutdown is not blocked.
bootstrap/startup calls MfaService.getInstance().start().
bootstrap/shutdown calls MfaService.getInstance().stop().
index.ts drops from 305 to 147 lines and now contains only the Express
app composition: createApp, route mounts, remote proxy, createServer,
attachUpgrade, static/SPA fallback, errorHandler, installShutdownHandlers,
and the require.main guard that boots the server when run directly.
Behavior is byte-for-byte identical: shutdown service order, log
strings, force-exit timer, pilot-agent loopback logic, and the MFA
purge cadence and debug logging all preserved verbatim.
Move the nine /api/nodes/* endpoints out of index.ts into routes/nodes.ts
(list, scheduling-summary, get, create, pilot-enroll, update, delete,
test, meta). mintPilotEnrollment and the REMOTE_META_* constants move
with the router as local helpers.
Handlers moved verbatim. Two safe cleanups applied during the move:
- Inline req.apiTokenScope 403 blocks replaced with the shared
rejectApiTokenScope helper; payload shape unchanged.
- catch (error: any) rewritten to catch (error: unknown) with explicit
instanceof Error narrowing to satisfy the no-any strictness rule.
Response body shapes unchanged.
Removes now-unused imports from index.ts: jwt, crypto, authMiddleware,
isValidRemoteUrl, PilotTunnelManager, PilotCloseCode, CAPABILITIES,
getSenchoVersion, fetchRemoteMeta, RemoteMeta, FleetUpdateTrackerService,
plus the module-scope updateTracker alias.
index.ts drops from 1364 to 1021 lines. Only the stacks group remains
inline for the final phase 4c slice.
Move the six /api/containers/* and /api/ports/in-use endpoints out of
index.ts. Handlers moved verbatim. routes/containers.ts exports two
routers:
- containersRouter (mounted at /api/containers): list, stream logs,
start, stop, restart.
- portsRouter (mounted at /api/ports): /in-use host port inventory.
Removes the now-unused requireAdmin import from index.ts. Middleware
chains, response shapes, and error messages are all preserved.
index.ts drops from 1437 to 1364 lines. Remaining inline groups:
stacks and nodes.
Move the entire /api/security/* surface out of index.ts:
- Trivy lifecycle: status, install, uninstall, update-check, update, auto-update toggle
- Scanning: POST /scan (image), POST /scan/stack (compose)
- Scan queries: list, get, vulnerabilities, secrets, misconfigs, image-summaries, SARIF export
- SBOM generation
- Scan policies CRUD (fleet-replicated; replica writes rejected)
- CVE suppressions CRUD (fleet-replicated; replica writes rejected)
- GET /compare: diff two scans
Handlers moved verbatim. Local CVE_ID_RE, parseScannersInput, and
shapeScanForResponse helpers move with the router. The previously-closure
fetchAll inside the SARIF handler is hoisted to module scope as
fetchAllPages so it is not re-allocated per request.
Removes nine now-unused imports from index.ts (parsePolicyEvaluation,
VulnerabilityScan, FleetSyncService, requireAdmiral, trivyInstallLimiter,
SbomFormat, TrivyInstaller, validateImageRef, applySuppressions,
generateSarif).
index.ts drops from 2130 to 1437 lines. Remaining inline groups:
containers, stacks, and nodes.
Move three more Round C route groups out of index.ts:
- /api/registries/* -> routes/registries.ts
- /api/system/{orphans,prune,docker-df,resources,images,volumes,networks,...} -> routes/systemMaintenance.ts
- /api/templates/* -> routes/templates.ts
Handlers moved verbatim. Middleware chains, response shapes, and error
messages are preserved. Registry scope-denial now uses the shared
rejectApiTokenScope helper (same payload shape).
index.ts drops from 2739 to 2130 lines. Remaining inline groups:
containers, stacks, security (Trivy/scans/SBOM/policies), and nodes.
Move four small Round C route groups out of index.ts:
- /api/auto-heal/* -> routes/autoHeal.ts
- /api/notifications/*, /api/notification-routes/* -> routes/notifications.ts
- /api/system/console-token -> routes/console.ts
- /api/sso/config/* -> routes/ssoConfig.ts
Share NOTIFICATION_CHANNEL_TYPES, cleanStackPatterns, and validateHttpsUrl
via a new helpers/notificationChannels.ts so agents.ts and notifications.ts
consume the same allowlist and URL validator.
Handlers moved verbatim. Middleware chains and response shapes preserved.
index.ts drops from 3231 to 2739 lines.
Wraps up Phase 4 Round B by tackling the two deferred groups. 25 new
integration tests land first and run green against the inline monolith,
then each group is extracted byte-for-byte.
index.ts drops from ~3,678 to ~3,231 lines; test count rises 1,320 → 1,345.
New coverage:
- metrics-routes.test.ts (11) — auth + shape checks for /api/stats,
/api/metrics/historical, /api/system/stats, /api/system/cache-stats
(admin-only), and SSE headers for /api/logs/global/stream
- image-updates-routes.test.ts (14) — auth, admin gating, rate-limit
tolerance on /refresh, fleet aggregation, /auto-update/execute input
validation and no-stacks short-circuit
New route files:
- routes/metrics.ts — /stats, /metrics/historical, /logs/global (+ SSE
/stream), /system/stats, /system/cache-stats. Mounted at /api so the
mixed sub-paths line up.
- routes/imageUpdates.ts — /api/image-updates CRUD + fleet aggregation,
plus a separate autoUpdateRouter mounted at /api/auto-update that
owns the /execute handler. Same split pattern as license.ts +
systemUpdateRouter.
index.ts trims unused imports left behind by the extraction:
globalDockerNetwork, si, STATS_CACHE_TTL_MS, SYSTEM_STATS_CACHE_TTL_MS,
GlobalLogEntry + log-parsing helpers.
Round B of Phase 4. Writes integration tests for three under-covered
route groups BEFORE extracting them, then does the extraction once the
new tests pass against the monolith. index.ts drops from ~4,206 to
~3,678 lines.
New test coverage (42 new assertions):
- settings-routes.test.ts (14) — auth, admin gating, private-key stripping,
allowlist, single-key write, bulk PATCH validation + partial update
- scheduled-tasks-routes.test.ts (18) — list/create/get/toggle/delete/runs,
action+target_type matrix, cron validation, tier gating on non-admin
- agents-routes.test.ts (10) — GET/POST, admin gating, channel type +
HTTPS URL validation, boolean enabled check, upsert semantics
Each suite was verified against the inline monolith first, then the
route extraction was performed byte-for-byte and all suites re-run to
ensure no regression.
New route files:
- routes/settings.ts — GET/POST/PATCH with PRIVATE_SETTINGS_KEYS strip,
ALLOWED_SETTING_KEYS allowlist, and SettingsPatchSchema zod bulk schema
- routes/scheduledTasks.ts — 9 endpoints (list, create, get, update,
delete, toggle, run-now, runs history, runs CSV export). File-local
helpers parseTaskId, validateActionTarget, validateOptionalFields
collapse duplication across create+update handlers. Uses shared
escapeCsvField from utils/csv.ts.
- routes/agents.ts — notification-channel GET/POST. Owns
NOTIFICATION_CHANNEL_TYPES and validateHttpsUrl locally because the
notification-routes block still inlines identical copies; the helpers
will converge once those routes extract in a later slice.
Final slice of Phase 4 Round A. Pulls the four remaining well-tested route
groups out of index.ts. index.ts drops from ~5,930 to ~4,206 lines.
New route files:
- routes/webhooks.ts: /api/webhooks CRUD + HMAC-authenticated trigger.
Uses shared webhookTriggerLimiter. Trigger preserves the raw-body path
established by the conditional JSON parser for HMAC validation.
- routes/users.ts: /api/users CRUD + /:id/mfa/reset + /:id/roles
scoped-assignment surface. Uses rejectApiTokenScope across every
handler, validateUsername helper, BCRYPT_SALT_ROUNDS, and
isSqliteUniqueViolation for the role-assignment UNIQUE guard.
- routes/gitSources.ts: /api/git-sources + /api/stacks/:name/git-source/*.
Exports two routers (gitSourcesRouter + stackGitSourceRouter) because
the per-stack paths need to mount at /api/stacks alongside the label
routes extracted in phase 4a-1. String length limits are now named
constants so the 400 responses stay truthful if the bounds change.
- routes/fleet.ts: /api/fleet role, sync, overview, node drill-down,
update-status + trigger (single + fleet-wide), and snapshot CRUD +
restore. Local parseIdParam helper collapses seven copies of the
parseInt/isNaN route-param pattern.
Bugs fixed during review:
- users.ts :id/roles POST — replace the fragile
(err as Error).message?.includes('UNIQUE constraint') check with
isSqliteUniqueViolation from utils/errors.ts.
index.ts carries forward three symbols (updateTracker alias,
CVE_ID_RE, parseScannersInput) until the corresponding security /
nodes / scan routes get extracted in a later slice.
Second slice of Phase 4. Pulls the three auth-family route groups out of
index.ts into focused routers. All handlers move verbatim; index.ts drops
~845 lines.
New route files:
- routes/auth.ts: /api/auth core (status, setup, login, password, logout,
check, generate-node-token)
- routes/mfa.ts: /api/auth/login/mfa + full /api/auth/mfa/* surface
(status, enroll start/confirm, disable, backup-codes/regenerate,
sso-bypass)
- routes/sso.ts: /api/auth/sso/{providers,ldap,oidc/:provider/authorize,
oidc/:provider/callback} + getSSOBaseUrl helper. Module-load calls
SSOService.getInstance().seedFromEnv() so env-seeded providers are
available on the first request.
Shared lifts:
- helpers/constants.ts: MFA_REPLAY_TTL_MS + MFA_REPLAY_PURGE_INTERVAL_MS
(used by mfa.ts and the startup purge timer in index.ts) and
BCRYPT_SALT_ROUNDS (shared between setup and password-change handlers).
- middleware/auth.ts: new reissueSessionAfterTokenBump(req, res, userId)
helper collapses three copies of "bump → fetch user → re-sign cookie"
across auth.ts (password change) and mfa.ts (enrol confirm, disable).
Code review fixes:
- File-local requireEnrolledMfaUser helper in mfa.ts eliminates four
copies of "auth check + rejectApiTokenScope + load enrolled MFA" with
near-identical shape.
- Applied BCRYPT_SALT_ROUNDS to auth.ts setup + password handlers.
Mount order in index.ts: authRouter / mfaRouter / ssoRouter sit before
authGate because login / setup / SSO-callback are public; handlers that
need auth use authMiddleware directly on the route.
First slice of Phase 4 (route extraction). Pulls 8 well-tested, mostly
independent route groups out of index.ts into focused Router files. No
behavior change; every handler body moves verbatim.
New route files under backend/src/routes/:
- meta.ts /api/health, /api/meta (mounted before authGate)
- license.ts /api/license/* + /api/system/update,
exports scheduleLocalUpdate for the fleet route
- permissions.ts /api/permissions/me
- convert.ts POST /api/convert
- alerts.ts /api/alerts/*
- labels.ts /api/labels/* + PUT /api/stacks/:name/labels
(exported as stackLabelsRouter)
- apiTokens.ts /api/api-tokens/*
- auditLog.ts /api/audit-log/*
Shared helper lifts:
- helpers/cacheInvalidation.ts: invalidateNodeCaches()
- middleware/tierGates.ts: requireBody (was inline in index.ts)
- utils/errors.ts: isSqliteUniqueViolation (was inline in index.ts)
- middleware/apiTokenScope.ts: rejectApiTokenScope() helper (new)
- utils/csv.ts: escapeCsvField() (new)
index.ts drops from ~7520 to ~6775 lines and now mounts the routers right
after enforceApiTokenScope. The remote proxy and fleet/auth/webhooks/users
routes remain inline in index.ts pending later Phase 4 slices.
Code review fixes: rejectApiTokenScope helper replaces duplicated
`if (req.apiTokenScope) 403 SCOPE_DENIED` blocks in apiTokens.ts and
license.ts; escapeCsvField replaces the inline CSV escape in auditLog.ts.
Phase 3 of the index.ts refactor. Pulls the remote HTTP/WS proxy plumbing,
the WebSocket upgrade dispatcher, and the http/WSS construction out of the
monolith. index.ts drops roughly 620 lines.
New modules:
- proxy/websocketProxy.ts: shared httpProxy.createProxyServer singleton
(used by both the HTTP proxy middleware and the remote WS forwarder)
- proxy/remoteNodeProxy.ts: createRemoteProxyMiddleware() factory; consumes
the isProxyExemptPath helper instead of open-coding the prefix list
- server.ts: createServer(app) returns { server, wss, pilotTunnelWss }
- services/FleetUpdateTrackerService.ts: singleton wrapping the in-flight
fleet update tracker Map with create()/resolve() helpers
- helpers/consoleSession.ts: mintConsoleSession(), isConsoleSessionScope()
- websocket/upgradeHandler.ts: attachUpgrade(server, deps) dispatcher that
runs the manual cookie/JWT verify and delegates to sub-handlers
- websocket/pilotTunnel.ts: handlePilotTunnel (pilot_enroll consumption and
pilot_tunnel registration)
- websocket/notifications.ts: /ws/notifications local subscriber
- websocket/remoteForwarder.ts: remote-node WS proxy with console_session
token exchange for interactive paths
- websocket/logs.ts: /api/stacks/:name/logs supervisor stream
- websocket/hostConsole.ts: /api/system/host-console PTY, Admiral-gated
- websocket/generic.ts: /ws exec + streamStats action dispatch, owns the
terminalWs single-instance reference
- websocket/reject.ts: shared rejectUpgrade helper (replaces five copies)
Service extension:
- NotificationService: setBroadcaster(fn) replaced by subscribe(ws) that
returns an unsubscriber; broadcastToSubscribers is now internal. Subscriber
set lives on the service rather than in index.ts.
Wiring in index.ts:
- const app = createApp() already in place from Phase 2
- const { server, wss, pilotTunnelWss } = createServer(app)
- attachUpgrade(server, { wss, pilotTunnelWss })
- app.use('/api/', createRemoteProxyMiddleware())
- /api/system/console-token route now uses mintConsoleSession()
- deploy/down/update routes read the streaming target via getTerminalWs()
(return type is WebSocket | undefined so the || undefined fallback is gone)
Code review fixes: five duplicated reject helpers collapsed into
websocket/reject.ts; dropped the createTracker/resolveTracker bind
aliases in index.ts so call sites go through the service directly;
removed em dashes; replaced req.url! with req.url || '/'.
Phase 2 of the index.ts refactor. Pulls the auth middleware and session
cookie issuers into their own module, and introduces the app.ts factory
that owns the first nine steps of the canonical middleware pipeline.
New modules:
- middleware/auth.ts: authMiddleware, issueSessionCookie,
issueMfaPendingCookie, clearMfaPendingCookie
- app.ts: createApp() factory installing trust proxy, helmet, cors,
compression, cookieParser, rate limiters, conditionalJsonParser, and
nodeContextMiddleware. A header comment documents all 16 canonical
middleware steps and where each currently lives.
Changes:
- middleware/authGate.ts: createAuthGate factory removed; authGate now
imports authMiddleware directly (the factory existed only to avoid a
circular import while authMiddleware lived in index.ts).
- services/DatabaseService.ts: added API_TOKEN_SCOPE_TO_ROLE map so the
auth middleware no longer inlines a stringly-typed record.
- index.ts drops ~260 lines; auth routes, authGate, auditLog,
apiTokenScope, remaining routes, static serving, and the error handler
continue to be registered there until their respective phases.
- vitest.config.ts bumps testTimeout to 30s and hookTimeout to 45s so
fork-pool workers have enough headroom to ts-node-transform the
growing module graph under CPU contention (64 workers each import the
full Express stack in beforeAll).
Code review fixes: use getErrorMessage() util in the auth catch block
instead of inline cast; promote the scope-to-role map to a typed
module-level constant.
Phase 1 of the index.ts monolith refactor. Extracts all non-auth middleware
into focused modules. Routes and authMiddleware itself stay in index.ts for
now; Phase 2 introduces createApp() and extracts authMiddleware.
New modules:
- middleware/rateLimiters.ts: globalApiLimiter, pollingLimiter,
webhookTriggerLimiter, authRateLimiter, ssoRateLimiter, trivyInstallLimiter
plus the hybrid rateLimitKeyGenerator and isNodeProxyRequest helper
- middleware/jsonParser.ts: conditionalJsonParser that preserves the raw
stream for remote-proxy forwarding (via helpers/proxyExemptPaths)
- middleware/nodeContext.ts: nodeContextMiddleware
- middleware/apiTokenScope.ts: enforceApiTokenScope + DEPLOY_ALLOWED_PATTERNS
- middleware/authGate.ts: createAuthGate(authMiddleware) factory + auditLog.
Factory takes authMiddleware as a dependency to avoid a circular import
until Phase 2 extracts the auth module.
- middleware/errorHandler.ts: central error handler that preserves
err.status / err.expose from body-parser and other HTTP errors
- helpers/routePatterns.ts: WEBHOOK_TRIGGER_RE shared by rateLimiters and
authGate
index.ts drops ~290 lines. Middleware registration order is unchanged.
All 1278 tests pass.
Code review fixes: typed ApiTokenScope in apiTokenScope.ts; replaced 2
em dashes with colons (Directive 18); added local CachedProxyFlagReq
type alias for the node_proxy memoization cast; extracted deny() helper.
* refactor(backend): extract types, constants, and guards from index.ts (phase 0)
Additive, behavior-preserving first step of the modular backend refactor.
Moves purely static artifacts out of backend/src/index.ts so later phases can
extract routes and middleware without touching shared symbols.
New modules:
- types/express.ts: Express Request augmentation
- helpers/constants.ts: PORT, password policy, label colors, cookie names,
MFA TTLs, hot-path cache TTLs
- helpers/proxyExemptPaths.ts: PROXY_EXEMPT_PREFIXES + isProxyExemptPath
- helpers/cookies.ts: isSecureRequest, getCookieOptions
- helpers/policyGate.ts: buildPolicyGateOptions, runPolicyGate,
triggerPostDeployScan
- middleware/permissions.ts: ROLE_PERMISSIONS, checkPermission,
requirePermission
- middleware/tierGates.ts: requirePaid, requireAdmiral, requireAdmin,
requireNodeProxy, requireScheduledTaskTier + effectiveTier/Variant
index.ts shrinks by ~260 lines; no runtime behavior changes. All 64 vitest
files and 1,278 tests pass.
* refactor(backend): drop unused imports left after phase 0 extraction
LicenseTier, LicenseVariant, DIGEST_CACHE_TTL_MS, and isProxyExemptPath
were imported into index.ts but no longer referenced there after the
phase 0 move; CI lint flagged them as errors.
isProxyExemptPath will be re-imported in phase 1 when the JSON parser
bypass and nodeContext middleware get extracted. Silence the
no-namespace warning on the Express augmentation since the namespace
syntax is required for TypeScript module augmentation.
* feat(security): polish scan sheets, fix CVE links, surface policy violations
Adds cveUrl helper that rewrites Trivy's 404-ing avd.aquasec.com links to
cve.org for CVE-prefixed IDs (GHSA and misconfig URLs pass through unchanged).
Redesigns both scan sheets with shadow-card-bevel chips, tracked-mono kickers,
severity row tinting with a left accent rail, and tabular-nums timestamps.
Surfaces a destructive policy-violation banner on scans whose policy_evaluation
row flags a block, and fixes the compare sheet's delta ribbon so CRITICAL
net-positive deltas render in destructive (not warning) tone. Backend parses
the JSON policy_evaluation column at the API boundary so the UI receives a
structured object.
* chore(security): suppress CVE-2026-32281 and CVE-2026-32283 in Trivy scan
Both CVEs affect Go stdlib crypto/x509 and TLS in Docker CLI 29.4.0
(Go 1.26.1) and Compose v5.1.2 (Go 1.25.8). No upstream static binary
has been released with the patched Go 1.26.2 or 1.25.9 runtimes yet.
Exposure analysis: the Docker CLI and compose plugin connect to the local
Docker socket (Unix socket, no TLS) and to public registries with well-known
CAs. Neither CVE is exploitable in this configuration. Added alongside
sibling entries already in .trivyignore for the same binary versions.
Revisit on next Docker CLI and Compose upstream release.
Policies with block_on_deploy=1 now scan every stack image before
docker compose up runs and reject the deploy with HTTP 409 on violation.
The UI opens a dialog listing offending images; admins can override per
deploy with ?ignorePolicy=true, and every bypass is recorded in the
audit log with the originating route, actor, policy, and image list.
When Trivy is not installed on the target node the gate fails open with
a warning notification, so teams are never locked out by tooling state.
Post-deploy and scheduled scans still evaluate matching policies and
dispatch warnings on violations to surface drift on long-running stacks.
Public API additions: policy and suppression CRUD under /api/security,
plus the documented 409 block-response shape on all deploy paths.
Flip Security (Trivy), Notifications (agents + history), and App Store from
global-and-hidden-on-remote to node-scoped so operators can manage them when a
remote node is selected in the node picker. The primary instance proxies the
calls to each remote, which resolves the correct per-instance binary state,
agent config, and template registry.
Backend: key `agents` and `notification_history` by `node_id` with idempotent
column-add migrations and a `(node_id, type)` unique index on agents, matching
the Labels pattern. Thread `req.nodeId` through the /api/agents and
/api/notifications routes. Internal NotificationService and ImageUpdateService
writes resolve the middleware default via `NodeRegistry.getDefaultNodeId()` so
monitor-emitted rows share a bucket with user-facing ones (avoids split-brain
where the UI sees test notifications but not internal alerts).
Frontend: split Security on remote to render only the scanner card and hide
scan policies and CVE suppressions (those remain control-plane-only). Drop the
misleading "Always Local" badge on Developer since retention windows govern
backend jobs, not UI state. Flip the App Store registry to node-scoped.
Docs: add a "What Settings apply per node" table to multi-node, clarify
remote alert setup in alerts-notifications, and note Trivy's per-host install
in vulnerability-scanning.
* fix(app-store): use stack name as compose service key
App Store deployments hardcoded the compose service to "app", so every
container's com.docker.compose.service label collapsed to "app". Global
logs and AutoHeal policies key off that label, making it impossible to
distinguish between deployed apps. Pass the already-validated stack
name through to generateComposeFromTemplate so each app gets a
descriptive service identifier.
* test(template-service): use {2} quantifier to satisfy no-regex-spaces
ESLint's no-regex-spaces rule flagged two consecutive spaces in the
regex literal as an error. Swap for the {2} quantifier to keep the
assertion identical while clearing the lint gate.
---------
Co-authored-by: Claude <noreply@anthropic.com>
Rework the Logs view into the cockpit language: a PageMasthead strip with
cyan rail, pulsing live dot, italic state word, and tracked-mono kicker;
a SignalRail of EVENTS/MIN (with rolling 60s sparkline), ERRORS, WARNINGS,
and CONTAINERS tiles; a segmented filter strip; a day-banded feed with
severity dots and whole-row tint on ERROR/WARN; and a floating glass chip
strip for pause/resume, clear, and download.
Decouple the live log stream from the Developer Mode toggle so SSE runs by
default for everyone, with polling as an invisible fallback when
EventSource is unavailable. Drop the Standard Log Polling Rate setting
and the global_logs_refresh key it persisted; Developer Mode still gates
Real-Time Metrics and Debug Diagnostics and nothing else.
Introduces the shared PageMasthead and SignalRail primitives for reuse
across other cockpit pages.
Add a Stream view to the Audit Log that leads with a four-tile signal
rail (events, actors, failure rate with inline sparkline, peak hour)
and presents the feed grouped by day with severity dots, relative
times, and inline anomaly callouts. The existing Table view is
preserved behind a toggle for power users.
Anomaly flags are computed at read time against strictly prior history
and returned on demand via ?with_anomalies=1:
- unusual_hour: hour outside the actor's central 7-day window
- new_ip: IP unseen for this actor in the last 30 days
- first_seen_actor: no prior history in the 30-day window
New /audit-log/stats endpoint returns the signal-rail aggregates over
24h/7d/30d windows; stats are derived from a single 30-day scan.
* feat(backend): add stack update-preview endpoint for readiness board
Adds GET /api/stacks/:stackName/update-preview that returns per-image
semver diff, bump classification, and a stack-level summary powering
the Auto-Update readiness board.
- New UpdatePreviewService parses compose images, inspects local
digests, fetches remote digests and tag lists, and finds the
highest compatible semver tag.
- Major bumps are flagged blocked until human review; unknown bumps
rank below real semver so they cannot mask a major.
- Rollback target is reconstructed through parseImageRef to preserve
registry ports and drop the Docker Hub library/ prefix.
- Registry helpers (httpGet, auth token, digest, tag list, ref parse)
are extracted into registry-api.ts and shared with ImageUpdateService.
- 28 Vitest cases cover parse, selection, bump math, digest rebuilds,
blocked policy, and rollback target construction.
* feat(schedules): next-24h timeline, merge auto-update crud, add readiness board
Replace the flat task table with a Timeline view as the default, showing the
next 24 hours of scheduled work across four lanes (Restart, Update, Scan,
Prune) with a live now rail and per-firing pills. The All tasks tab preserves
the existing CRUD surface.
Merge Auto-update Stack into Schedules as a first-class action and replace the
standalone Auto-Update Policies view with a per-stack Readiness board that
surfaces version diffs, risk tags, changelog previews, and rollback targets
sourced from the stack update-preview endpoint.
* feat(app-store): editorial hero, category rail, security scan signal per tile
Rework the App Store view into an editorial layout with a 180px category
rail, a featured-template hero, and compact tiles that surface star counts
and vulnerability scan status at a glance. The deploy sheet splits into
Essentials (one-click deploy) and Advanced (full port/volume/env control)
tabs.
Backend enriches /api/templates with scan_status, scan_cve_count, and a
featured flag computed from the highest-star template with recorded stars.
Scan lookups use a single batched SQL query to avoid N+1 round-trips.
* fix(app-store): split firstSentence helper into its own module
The firstSentence helper lived alongside the TemplateLogo component, which
violates react-refresh/only-export-components — a file that exports a
component must not also export non-component values, or Fast Refresh cannot
establish an HMR boundary. Move the helper into appstore/util.ts and update
the two import sites.
* feat(dashboard): status masthead, unified gauges, stack health sparklines
Rework the home dashboard around a single cyan-railed status masthead
that carries the health state word, node meta, and reasons inline. The
resource block collapses into one strip with a CPU hero sparkline,
memory and disk gauge bars, and a network tile whose sparkline is built
from per-container byte-counter deltas. Stack health becomes an
8-column mono table with row tinting, an uptime column sourced from
the oldest running container's creation time, and a per-stack 10-minute
CPU sparkline. Historical charts pick up a cyan gradient and an amber
peak marker. A shared Sparkline primitive backs all of the above.
* docs(dashboard): refresh screenshot for phase B layout
* fix(dashboard): anchor sparkline bucketing to latest metric timestamp
The cpuHistory, netHistory, and cpuPeakLabel memos called Date.now()
inside useMemo, which violates react-hooks/purity: the rule fires
because re-renders can produce different bucket boundaries from the
same inputs. Derive a historyEndAt anchor from the newest metric
sample in the polled series and thread it through to ResourceGauges.
* feat(db): add auto_heal_policies and auto_heal_history schema and CRUD
Adds two new SQLite tables (auto_heal_policies, auto_heal_history) to
DatabaseService.initSchema() and exposes CRUD methods: getAutoHealPolicies,
getAutoHealPolicy, addAutoHealPolicy, updateAutoHealPolicy,
deleteAutoHealPolicy, recordAutoHealHistory, getAutoHealHistory,
incrementConsecutiveFailures, resetConsecutiveFailures, setPolicyEnabled.
Also adds AutoHealPolicy and AutoHealHistoryEntry TypeScript interfaces.
* feat(events): track health-status duration and expose state accessors
- Add healthStatus and unhealthySince fields to InternalContainerState
- onHealthStatus now records unhealthySince timestamp on first transition
to unhealthy, and clears it when the container recovers or restarts
- onStart resets both fields so a restarted container begins from 'starting'
- Add listContainerStates() and getContainerState() public accessors for
use by the upcoming AutoHealService evaluator
* fix(auto-heal): key allowlist in updateAutoHealPolicy, cascade delete, extract ContainerHealthSnapshot
* feat: add AutoHealService evaluator singleton
Polls every 30 s, matches containers to enabled policies via Compose
labels, and restarts containers that have been unhealthy beyond the
configured threshold. Enforces cooldown, per-hour rate cap, and
recent-user-action suppression; auto-disables policies after repeated
consecutive failures. Also adds DockerEventManager.getService() accessor
required by the evaluator.
* fix(auto-heal): prune stale restartTimestamps, guard undefined policy id
- Prune restartTimestamps entries for containers no longer running after
each container list fetch, preventing unbounded map growth from dead
container IDs.
- Guard against policies with undefined id at the start of the per-policy
loop; warn and skip rather than proceed with a non-null assertion.
- Extract handleAutoDisable private helper to bring executeHeal under 30
lines and isolate the auto-disable side-effect sequence.
- Move ContainerInfo type to module scope.
* feat: add auto-heal API routes and wire AutoHealService lifecycle
Registers five REST endpoints under /api/auto-heal/policies (list, create,
patch, delete, history) with requirePaid + requireAdmin guards and Zod
validation. Wires AutoHealService.start()/stop() into the server startup
and graceful-shutdown blocks alongside MonitorService.
* test: add AutoHealService and DatabaseService auto-heal unit tests
- 15 unit tests for AutoHealService.shouldHeal covering all decision branches
(healthy state, duration threshold, user-action suppression, cooldown,
rate limiting, and correct skipReason values)
- 13 integration tests for DatabaseService auto-heal CRUD: policy round-trip,
stack-name filter, partial update, cascade delete, history ordering/limit,
consecutive failure counters, and setPolicyEnabled toggle
* fix: log AutoHealService shutdown errors consistently
* fix(api): requireAdmin-first guard order and try/catch on auto-heal routes
* feat(ui): add StackAutoHealSheet component
* feat(ui): add Auto-Heal context menu item to EditorLayout
* fix(ui): StackAutoHealSheet label, token, a11y, and useEffect fixes
- Rename 'All services in stack' to 'All services' in combobox options and placeholder
- Replace text-green-600 with text-success design token in actionColorClass
- Add htmlFor/id pairs to all four numeric form inputs for accessibility
- Inline fetch logic into useEffect, removing stale closure risk and eslint-disable comment
- Remove now-unused fetchPolicies and fetchServices standalone functions
- Update 'Auto-disable after' label to 'Auto-disable after (failures)' for clarity
- Add toast.error in policy fetch failure path; services fetch silently skips as before
* docs: add auto-heal-policies feature documentation
* test(e2e): add auto-heal policies CRUD spec
* fix(docs): correct auto-heal-policies nav position in docs.json
* feat: pilot agent outbound-mode for remote nodes
Adds a second mode for managing remote nodes: the agent dials an outbound
WebSocket tunnel to the primary, so the remote host no longer needs an
inbound port, a reachable URL, or its own TLS certificate. Works behind
NAT, residential routers, and corporate firewalls.
The primary multiplexes HTTP and WebSocket requests over a single tunnel
via a hybrid JSON + binary frame protocol, bridged through a per-tunnel
loopback server so existing proxy and upgrade handlers route pilot-mode
nodes identically to proxy-mode ones.
Enrollment uses a single-use 15-minute pilot_enroll JWT exchanged for a
long-lived pilot_tunnel credential on first connect. Proxy mode continues
to work unchanged and both modes are supported side-by-side.
* test(e2e): switch to proxy mode before asserting api_url field
Remote nodes default to Pilot Agent mode, which hides the api_url input.
The SSRF-validation tests need proxy mode, so the helper now selects
Distributed API Proxy after picking Remote type before asserting the
field is visible.
* fix(e2e): wire Combobox id prop so node-mode selector resolves
The Combobox trigger button had no id, leaving its Label orphaned and
making getByRole name-based lookups fail. Adding id to the primitive,
passing id="node-mode" from NodeManager, and updating the E2E helper to
use #node-mode fixes both the a11y regression and the CI timeout.
Emit a [Compare:diag] entry gated on developer_mode with the scan ids,
effective tier, input and bucket sizes, suppression count, and
truncation flag. Gives operators a trace to correlate with user reports
of unexpected compare results without adding hot-path cost.
Scan history fetched a fixed 200 most-recent rows and paginated them
client-side, so older scans silently fell off mature nodes where
baselining is most valuable. The list now fetches one page at a time
via offset, with status=completed and imageRefLike filters applied
server-side. Search input is debounced to avoid per-keystroke fetches.
Compare endpoint loads up to 1000 findings per scan. When a scan exceeds
this cap, the response now includes truncated=true and row_limit, and
the comparison sheet surfaces a banner so users understand the diff may
be incomplete. Also exposes total_vulnerabilities on scanA/scanB for UI
use and logs a warning when truncation occurs.
Enrich scheduled vulnerability scan completion notifications with
per-severity CVE counts so recipients can triage from the message body
alone. Expose the scan action in the schedule creation UI, require an
explicit node_id, and harden fire-and-forget alert dispatches so a
failing webhook cannot crash the scheduler.
Notification body now reports scanned/skipped/failed counts plus
critical/high/medium totals aggregated across fresh and cached scans,
reflecting the current node posture rather than only what was newly
scanned on this run.
Adds a new SarifExporter service that builds a SARIF 2.1.0 document
from the stored scan findings (vulnerabilities, secrets, misconfigs).
Rule IDs are namespaced to avoid collisions in a flat result list.
Suppressions carry through as SARIF suppressions[] entries so GitHub
code scanning and Defender for Cloud see the same accepted status
shown in the UI.
Exposed via GET /api/security/scans/:id/sarif, admin + paid-tier
gated to match the SBOM export precedent. A SARIF button appears
in the scan sheet next to SBOM and CSV for paid tiers.
Extends Trivy scans with secret detection in image filesystems and
misconfiguration scanning for Compose stacks. Adds tabs to the scan
drawer for vulnerabilities, secrets, and misconfigs. Secret matches
are redacted server-side (first 8 chars + ellipsis) before storage.
- TrivyService: --scanners vuln,secret for images; trivy config for stacks
- DB: scanners_used/secret_count/misconfig_count cols; secret_findings,
misconfig_findings tables; cache key scoped by scanners
- Routes: POST /security/scan accepts scanners array (requirePaid when
secret requested); POST /security/scan/stack; GET .../secrets and
.../misconfigs (paid-tier reads)
- UI: tabs in VulnerabilityScanSheet; scan-options dropdown on images;
Scan config button on stack header
Operators can accept known-benign findings once and have Sencho filter
them out of scan drawers, comparison views, and other read surfaces.
Suppressions replicate from the control instance to every remote node.
* New cve_suppressions table with a COALESCE-based unique index so NULL
scope slots collide the way users expect
* Admin + paid-tier CRUD routes; writes are rejected on replicas
* Read-time filter enriches vulnerability details and compare payloads
without mutating stored counts
* Settings > Security panel for managing rules, per-CVE suppress action
in the scan drawer, dimmed rows with a shield-off indicator
* Vitest unit tests for the filter (glob, expiry, specificity) and
route tests (auth, tier, replica, UNIQUE conflict)
Scan policies now propagate from the control Sencho instance to every
registered remote. The control is the source of truth; replicas render
rules read-only with a managed-by-control banner. Pushes fire on every
policy write, record per-node success and failure on a new
fleet_sync_status table, and use node_proxy Bearer tokens exclusively
so only sibling Senchos can apply incoming sync payloads. Policy scope
now travels as a string identity (api_url or a local sentinel) so
node-scoped rules evaluate correctly on each target.
* feat(security): one-click managed Trivy install
Add a Vulnerability Scanner card to Settings, Security with install,
update, uninstall, and auto-update controls (Admiral-only). The installer
downloads a verified Trivy release into the existing data volume at
/app/data/bin/trivy and defaults the cache to /app/data/trivy-cache, so
no host mounts or extra env vars are required. Detection probes the
managed path, a TRIVY_BIN override, and the host PATH, distinguishing
managed vs host installs. A daily scheduled check surfaces available
Trivy updates, installs them automatically when opted in, and dedupes
notifications per version.
* fix(frontend): silence react-hooks/set-state-in-effect in useTrivyStatus
The initial status fetch and managed-source update check both call
setState from the effect body. Match the existing pattern used in
useDashboardData / SSOSection and disable the rule at the call site.
* fix(security): harden Trivy scan lifecycle, logging, and docs
- Call TrivyService.initialize() at startup so capability state is
accurate before first request; add periodic re-detect to the scheduler
so newly installed Trivy binaries light up without a restart.
- Add markStaleScansAsFailed sweep (+ idx_vuln_scans_status index) to
recover any scan row left in_progress after a crash or timeout; sweep
runs before the paid-tier gate so every tier self-heals.
- Split scanImage persistence into beginScan/finishScan so the manual
scan route owns a single code path and can return a scanId synchronously
while work continues asynchronously.
- Validate image refs on /api/security/scan and /sbom via new utility;
defense-in-depth against shell-metacharacter payloads.
- Dispatch a warning-level alert when a post-deploy scan fails so the
operator has a user-visible path to the failure instead of a silent log.
- Share DIGEST_CACHE_TTL_MS and severity ordering across service and
route layers; remove dead invalidateDetection().
- Add [Trivy:diag] logging gated behind developer_mode for support
diagnostics; production logs unchanged.
- Frontend: defensive toast fallback chain, sr-only SheetDescription,
and a truncation badge when the 500-item detail fetch is capped.
- Tests: extend trivy-service and vulnerability-db suites; add
image-ref and severity unit tests.
- Docs: expand vulnerability-scanning troubleshooting with recovery,
re-detect, and diagnostic-log guidance; link Dockerfile comment to
trivy-setup.
* fix(security): drop unnecessary escape in image-ref forbidden-char regex
* feat(images): Trivy-powered vulnerability scanning
Scan container images for known CVEs via Trivy. On-demand scanning and
severity badges are available on every tier; scheduled scans, scan
policies, SBOM generation, and scan history are gated to Skipper+.
- New TrivyService (binary detection, per-image scan, SBOM, digest cache)
- Three new tables: vulnerability_scans, vulnerability_details, scan_policies
- 12 routes under /api/security (scan, results, summaries, SBOM, policies, compare)
- Post-deploy async scans wired into all five deploy paths, with a
per-deploy opt-out toggle in the App Store deploy sheet
- "scan" action type added to SchedulerService for fleet-wide recurring scans
- Frontend: severity badges in Resources Hub with animated cursor detail,
scan results drawer with vulnerability table and filters, and a new
Security section in Settings for scan policy CRUD
- Policy threshold violations dispatch a warning or critical alert based on
the policy's block_on_deploy flag; deploys themselves are never blocked
* fix(security): compute scan age in useEffect to satisfy react-hooks/purity
* feat: add Custom OIDC provider and move SSO to Community tier
Add a generic Custom OIDC provider that works with any spec-compliant
OIDC identity provider (Keycloak, Authentik, Authelia, Zitadel, KanIDM,
Pocket ID, etc.) via standard discovery. Supports configurable claim
mapping for User ID, Username, and Email fields to handle non-standard
providers.
Move all SSO functionality (LDAP and OIDC) from the Admiral tier to the
Community tier so every user has access to identity provider integration.
Backend: add oidc_custom to AuthProvider type, extend SSOService with
claim mapping fields and env-var seeding, add oidc_custom to route
validation, remove requireAdmiral guards from SSO config endpoints.
Frontend: add Custom OIDC card with Display Name, Issuer URL, and claim
mapping fields to SSOSection; add KeyRound icon on login page; remove
AdmiralGate wrapper and lock icon from SSO settings nav.
Tests: update tier guard expectations, add oidc_custom authorize/config/
provisioning tests and claim mapping coverage. All 992 tests pass.
Docs: add Custom OIDC configuration reference, provider-specific setup
examples, troubleshooting section, and updated screenshots.
* fix: settings dialog close button overlap and combobox styling
Reposition the close button in Settings Hub above the scroll area so
it stays fixed when content scrolls. Increase dialog height to
accommodate the growing number of setting sections.
Fix combobox trigger styling to match Input component tokens
(border-glass-border, bg-input) and eliminate the gap between trigger
and dropdown list (top-full -mt-px). Apply the same fixes to
multi-select-combobox for consistency.
Add items-start to the Scopes/Default Role grid so the combobox
aligns with the adjacent input field. Add showClose prop to
DialogContent for consumers that need custom close button placement.
Update SSO doc screenshots at 1920x900.
* feat(convert): harden /api/convert endpoint with auth, validation, and tests
Applies authMiddleware to the docker run to compose endpoint, validates
that the payload is a non-empty string within an 8192 character budget,
rejects inputs containing null bytes, and wraps composerize in a
try/catch that surfaces a 422 with a clear message when the library
cannot produce a services block. Adds a Vitest suite covering the auth
gate, happy path, common flag coverage, boundary and null byte
placement variants, and malformed command handling.
* feat(editor): add From Docker Run tab to create stack dialog
Introduces a third tab in the Create New Stack dialog that accepts a
docker run command, calls the converter endpoint, and previews the
returned compose YAML before writing it to a new stack directory. Uses
the defensive toast pattern, clears the stale preview when the input
changes, and rolls back the empty stack directory if saving the
converted YAML fails so the user never ends up with an orphan stack.
* docs(stack-management): document docker run to compose converter
Adds a Convert from a docker run command section to the stack
management page covering how to use the new tab, the list of supported
flags, and troubleshooting for unparseable inputs. Screenshots show the
empty tab, a successful conversion with the compose preview, the
resulting stack in the editor, and the error toast surfaced when the
input cannot be converted. Appends a matching entry to the
troubleshooting page.
* feat(mfa): auto-submit 6-digit TOTPs and normalize pasted backup codes
Match the UX every major MFA prompt has (GitHub, GitLab, 1Password): the
challenge screen and every code-entry dialog now submit automatically once
the sixth TOTP digit lands, and the backup-code input accepts pastes with
smart-dashes, trailing whitespace, or mixed case without silently
truncating the value. Also caps the backup-code input at the correct
11 characters (10 plus a single separator) instead of 12.
Shared normalization helpers live in frontend/src/lib/mfa.ts so the
challenge and the three account-settings dialogs stay in lockstep.
* feat(mfa): warn users when backup codes run low
The Account & Security card silently showed a dim count of backup codes
remaining, which meant users could drift toward zero without noticing
until their phone was already lost. The card now surfaces a warning tone
with an alert icon when 1 or 2 codes remain, and swaps to a dedicated
destructive warning card with a "Regenerate now" action when the user
has used every code.
* feat(mfa): gate diagnostic logs behind developer mode
Reuses the existing isDebugEnabled() gate so operators investigating a
2FA support ticket can flip Developer Mode on to get per-branch
diagnostics (login path taken, replay check outcome, failure counter
after a verify, replay-table purge counts), and flip it back off when
they are done. Standard lifecycle logs stay on by default: enrolment
completed, 2FA disabled, backup codes regenerated, admin reset, SSO
bypass toggled, lockout engaged. Nothing that could reveal a TOTP code,
base32 secret, backup-code cleartext, or partial-auth JWT is ever
logged.
* test(mfa): cover drift, invalid formats, lockout recovery, and paste normalization
Backend: a TOTP generated for a window that has already slid out is
rejected, malformed backup codes (too short, non-alphanumeric, 11-char
alphanumeric that matches no hash) all increment failed_attempts, a
successful verify clears a below-threshold failure streak, a successful
verify after locked_until has passed clears the lockout, a second
enroll/start overwrites the prior pending secret, and the backup-code
normalizer treats en-dash/em-dash/figure-dash with stray whitespace the
same as the canonical form.
E2E: low-backup-codes warning renders in the warning tone and the
exhausted-codes state flips to the dedicated warning card, a 6-digit
TOTP auto-submits without a button click, and a backup code pasted
without the separator still signs in.
* docs(mfa): auto-submit, paste guidance, and expanded troubleshooting
Document that the challenge screen submits automatically on the sixth
digit, that backup codes accept the separator and any case, and that
the Account & Security card nudges at low code counts. Expands the
troubleshooting section with entries for lost or exhausted backup codes
and adds a short note to the admin guide about surfacing auth
diagnostics via Developer Mode.
* feat(auth): add TOTP two-factor authentication with backup codes
Adds RFC 6238 time-based one-time password support to every tier,
integrated with the existing password and SSO login paths.
Backend:
- New MfaService wrapping otplib with a plus or minus 1 step tolerance,
base32 secret generation, and hashed single-use backup codes (bcrypt).
- user_mfa and mfa_used_tokens tables in DatabaseService. The second
table is a DB-backed replay blacklist, purged on a 60s interval.
- authMiddleware now recognizes an mfa_pending scope. A token carrying
that scope is rejected on every route except the MFA challenge and
logout, so no API surface is reachable before the second factor
clears.
- /api/auth/login issues only a short-lived mfa_pending cookie when the
user has MFA enrolled. /api/auth/login/mfa consumes that cookie,
verifies the code (or backup code), and swaps in a real session.
- /api/auth/mfa/* routes for status, enrol/start, enrol/confirm,
disable, backup-code regenerate, and SSO-bypass opt-in.
- Admin recovery path: POST /api/users/:id/mfa/reset clears the target's
MFA state, bumps token_version, and writes an audit log entry.
- CLI emergency fallback: backend/src/cli/resetMfa.ts is wired via
`npm run reset-mfa <username>` and also exported for tests.
- SSO flows (LDAP and OIDC) gate on user_mfa.sso_enforce_mfa before
issuing a session; default behaviour keeps the SSO path frictionless.
- Per-user lockout after 5 consecutive failed codes (15 min).
Frontend:
- AppStatus gains an mfa-challenge branch driven by /api/auth/status.
- New MfaChallenge screen, MfaEnrollDialog (QR plus manual secret plus
backup codes), MfaDisableDialog, MfaBackupCodesDialog.
- Account section shows a Two-factor authentication card with enrol,
regenerate, disable, and the SSO-enforce toggle (shown only when SSO
providers are configured).
- Users section gains a Reset 2FA action for admins.
Docs:
- New user guide at features/two-factor-authentication.mdx.
- New admin guide at operations/two-factor-admin.mdx.
- SSO page cross-links to the 2FA doc.
* fix(mfa): drop unused TEST_PASSWORD import and stale eslint disable
* fix(mfa): simplify e2e openAccountSettings helper to match working pattern
* fix(mfa): make e2e suite self-contained and always clean up
Test #2 called loginAs() before the MFA challenge step, which waited for
the dashboard indicator that never appears once the previous test enrolled
the user. That timeout skipped the rest of the serial block, including
the disable step, leaving MFA enabled and breaking every later spec.
Two fixes:
- Tests #2 and #3 now navigate directly to the login page instead of
piggybacking on loginAs, which only handles the password-only path.
- A new afterAll hook unconditionally disables MFA via the API using two
unused backup codes, so the DB is reset even if a test fails midway.
* fix(e2e): use backup code for mfa recovery to avoid totp replay race
The final recovery step in the backup-code replay test previously
generated a fresh TOTP to sign back in. When the timing landed inside
the same 30-second window that test #2 consumed, the server's replay
blacklist correctly rejected it, producing a ~50% flake rate. Backup
codes are single-use and sidestep the replay window, so the recovery
becomes deterministic.
* fix(e2e): drive mfa disable test through the challenge screen
Test #4 called loginAs after test #3 left MFA enabled, but loginAs
waits for the dashboard indicator and does not handle the challenge
screen, so it timed out. Drive the login manually, satisfy the
challenge with a backup code, and use a backup code for the disable
step too to avoid any TOTP replay-window race against earlier tests
in the serial block.
* feat(git-sources): surface LFS and submodule warnings on create
Creating a stack from a Git repo now detects two common anomalies and
tells the user about them rather than silently producing broken stacks.
- LFS-pointer compose/env files fail early with a clear error instead
of writing a 130-byte pointer stub to disk as real content.
- Repositories containing .gitmodules produce a non-fatal warning so
the user knows build contexts or volumes inside submodules will be
empty at deploy time.
Also refines the create dialog: sr-only DialogDescription for a11y,
short commit SHA suffix on the success toast, env-path hint under the
"Sync .env" checkbox showing which path will be read, and a route-level
diagnostic log line gated on developer mode for support debugging.
* test(git-sources): cover LFS, submodule, and nested env_path paths
Adds unit coverage for the new LFS-pointer rejection and submodule
warning plumbing, plus a nested compose_path case that exercises the
default env_path resolution ("apps/web/compose.yaml" with sync_env on
and env_path unset writes "apps/web/.env" both to disk and to the DB).
Extends the E2E suite with a happy-path assertion that the full-length
commit SHA is returned in the create response, and a UI flow that
verifies the short-SHA suffix appears in the success toast.
* docs(git-sources): add troubleshooting for LFS, submodules, HTTPS-only
Adds troubleshooting entries for the newly surfaced LFS and submodule
anomalies, expands the clone-timeout entry with the bounded-fetch
explanation, and adds a dedicated HTTPS-only entry. Also consolidates
the known limitations into a single list covering LFS, submodules,
branch-tracking, and HTTPS-only.
* fix(settings): use Route icon for notification routing
The routing section in Settings previously used GitBranch, which now
clashes with the Git Source feature's icon across the editor. Switch
to Route (a branching-flow glyph) so routing rules have a distinct
visual identity and aren't visually conflated with Git-backed stacks.
* fix(git-sources): return 400 for upstream auth failures and disambiguate 404s
Upstream git-host auth failures were mapping to HTTP 401, which the frontend
apiFetch treats as a Sencho session expiry and fires the global logout event.
They now return 400 with code=AUTH_FAILED in the body so the UI can branch on
the discriminator without logging the user out. The status mapping moved into
utils/gitSourceHttp so it can be unit-tested without booting the app.
mapGitError also relied on the HttpError class alone, so any non-2xx response
(including 404) was classified as auth failure. It now inspects the numeric
status on err.data and considers whether a token was supplied, producing more
actionable messages for missing repos, private repos, and wrong-scope tokens.
* refactor(git-sources): extract GitSourceFields from GitSourcePanel
Pure extraction of the repo/branch/path/auth/apply-mode form fields into a
reusable controlled component so the upcoming Create Stack from Git flow can
render the same form in the Create Stack dialog. No behavior change.
* feat(git-sources): create a stack from a Git repository
Add a From Git tab to the Create Stack dialog so users can name a new
stack, point it at a repo + branch + compose path, and have the compose
fetched, validated, written to disk, and linked in one shot. Optional
deploy-after-create runs the initial bring-up when requested.
Backend: new POST /api/stacks/from-git route gated by stack:create.
GitSourceService.createStackFromGit() fetches and validates before
touching disk, then creates the stack, writes the compose (and .env if
sync is enabled), and seeds the git source row with the fetched commit
so future pulls produce a clean diff. Runs under the per-stack lock so
a concurrent webhook cannot race the create. Deploy failure is
non-fatal and surfaced to the caller.
Frontend: the existing Create Stack dialog is now tabbed, with Empty
keeping the original single-field flow unchanged.
* test(git-sources): cover create-from-git endpoint and e2e flow
Service tests verify createStackFromGit seeds the last_applied columns
on success, writes the env file when sync is enabled, refuses an
invalid apply-matrix without fetching, rejects invalid compose
without leaving orphan state, and rolls back the on-disk stack dir
when a post-create step fails.
Route tests cover auth, missing stack_name, invalid stack name,
http:// rejection, oversized repo_url, and the 409 collision guard.
E2E adds a Create-stack-from-Git block covering tab visibility,
client-side HTTPS check, backend .git/config rejection, and a
happy-path fetch against a public demo repo (skipped on network
failure).
* docs(git-sources): document create-stack-from-git tab
Add a new section near the top describing the From Git tab in the
Create Stack dialog: what it does, the Deploy after create checkbox,
and the four failure modes (name collision, unreachable repo,
invalid compose, deploy-after-create failure).
* fix(git-sources): harden validation, RBAC, concurrency, and deploy recovery
Tightens the surface area around the Git source feature:
- Enforce HTTPS-only repo URLs server-side (regex was permissive).
- Add stack:read permission check on git-source reads and filter the
list endpoint by callable permission.
- Validate stack names before permission checks on mutation routes so
scoped lookups never see unvalidated input.
- Cap repo_url / branch / compose_path / env_path / token lengths and
require the stack directory to exist before upsert.
- Wrap pull() in the per-stack mutex to eliminate the pull/delete race
that could orphan pending data.
- Block .git/ path components in compose_path / env_path so a
misconfigured clone cannot leak repo metadata.
- Return {applied, deployed, deployError?} on deploy failure instead of
throwing, and surface deployError as a warning toast so the user can
retry deploy without re-pulling.
- Always clean the stack_git_sources row on stack delete even when the
file deletion step fails.
- Add shadow-card-bevel to the pending alert and metadata card per the
design system.
- Handle the new 403 response on the panel fetch gracefully.
- Add diagnostic logging gated on developer_mode (isDebugEnabled) across
fetch / pull / apply / webhook paths with credential scrubbing.
* test(git-sources): expand coverage for hardening and route validation
- New route-level suite covers HTTPS enforcement, required fields,
max-length caps on repo_url / branch / compose_path / env_path /
token, the stack-existence 404 guard, and GET authz.
- Service tests cover the .git metadata guard on compose and env
paths (including nested and substring-containing "git"), pull and
apply rejections when no source is configured or pending is
cleared, the sha-mismatch branch, and the deploy-failure return
shape that now carries deployError.
- E2E adds three server-side contract assertions: PUT against a
missing stack returns 404, http:// is rejected with 400, and
.git/config is rejected as compose_path.
* docs(git-sources): document deploy-failure recovery path
Adds a Troubleshooting entry explaining that when apply succeeds but
the subsequent deploy fails, the compose content is already on disk
and the user can retry deploy from the stack editor without
re-pulling.
* docs(git-sources): add configuration, diff, pending, and webhook screenshots
Anso