When a pilot-agent node was the active node, the UI rendered "does not
advertise this capability" across most tabs, a perpetual "Update
available" badge, and a Fleet card body with blank CPU/RAM/Disk and "No
stacks found". The cause was central-side aggregators in /api/fleet/* and
/api/nodes/:id/meta only fanning out to proxy-mode remotes via
node.api_url + node.api_token, which are null for pilot-agent.
Route every affected aggregator through NodeRegistry.getProxyTarget so
the loopback URL backed by the active pilot tunnel is used uniformly:
- /api/nodes/:id/meta and /api/fleet/update-status fetch via the new
NodeRegistry.fetchMetaForNode helper (resolves the target, delegates
to fetchRemoteMeta, returns the shared OFFLINE_META on null).
- fetchRemoteNodeOverview, /api/fleet/configuration,
/api/fleet/node/:nodeId/stacks, and the stack-containers drilldown
fetch through target.apiUrl with conditional Authorization.
- fetchRemoteMeta omits the Authorization header when the token is empty
(pilot-agent loopback) instead of sending a malformed Bearer string.
- Pilot-agent rows preserve pilot_last_seen and mirror it into
last_successful_contact so the Fleet "last seen" cell renders the
recent tunnel timestamp during a brief reconnect.
Pilot-mode capability filter excludes capabilities whose central-pilot
path is not yet wired (host-console, self-update). Without this, the
Console tab would surface for an Admiral pilot session and click
through to central's host because the WS upgrade handler still gates
on api_url + api_token. Filtered capabilities are removed at boot via
applyPilotModeCapabilityFilter when SENCHO_MODE=pilot.
Cache invalidation on tunnel-up: the meta cache for a reconnecting
pilot is dropped so the next request rebuilds capabilities and version
through the live bridge instead of waiting for the 3-minute TTL. The
namespace constant moves to helpers/cacheInvalidation.ts alongside the
new invalidateRemoteMetaCache helper.
Husky commit-msg hook: add the missing shebang and a .gitattributes
rule pinning .husky/* to LF line endings so commits do not fail with
"Exec format error" on Windows shells where autocrlf=true converts the
hook to CRLF.
Tests cover Authorization-header behavior, pilot-mode filter idempotency,
fetchMetaForNode dispatch (offline target, pilot-agent loopback,
proxy-mode), and the four affected fleet routes for pilot-agent both
when the tunnel is up and when it is down.
* fix: harden deploy enforcement paths
* fix: update Docker toolchain to Go 1.26.3
* fix: repair Dockerfile tr argument split across lines
* fix: bump protobufjs to clear npm audit high-severity advisories
* fix(test): add execFile to child_process mock in compose-images test
* fix: resolve merge conflicts with main
* fix: resolve merge conflicts with main
* fix: resolve merge conflicts with main
* feat(nodes): hide hub-only views when active node is remote
Fleet, Schedules, Audit, Logs, and Auto-Update operate on hub-owned state
(node registry, fleet schedules, centralized audit, fleet-wide log
aggregation, fleet-wide update preview). When the active node is remote,
proxying those surfaces would show that remote's own disconnected state
instead of the hub's. Hide them from the nav strip and force-redirect to
Home if one was open during the node switch.
Backend hubOnlyGuard middleware sits between nodeContextMiddleware and the
remote proxy and rejects /api/scheduled-tasks, /api/audit-log, and
/api/notification-routes with 403 + HUB_ONLY_ENDPOINT when nodeId resolves
to a remote, closing the script-bypass path the UI gating cannot reach.
Settings sub-sections were already gated via the hiddenOnRemote registry;
this extends the same model to top-level views.
* docs(nodes): note hub-only visibility on Fleet, Schedules, Audit, Logs, Auto-Update
Each of the five hub-only feature pages now points readers to the
canonical "What top-level views show when a remote node is active"
section in multi-node.mdx, so users landing directly on a feature page
understand why the nav item disappears when they switch to a remote node.
* feat(fleet): add Fleet Actions tab for cross-node bulk operations
Introduces a new "Actions" sub-tab in Fleet view with two Skipper+ cards
that fill gaps in the existing surface:
- Stop fleet by label: matches a label name across every node and stops
every stack assigned to it, reporting per-node and per-stack results.
- Bulk label assign: applies the same label set to many stacks on one
node in a single round trip.
Other bulk operations stay in their existing homes (sidebar bulk mode,
Schedules, NodeUpdatesSheet) to avoid duplicate surfaces.
Backend:
- POST /api/fleet/labels/fleet-stop (gateway-orchestrated, multi-node)
- POST /api/fleet-actions/labels/bulk-assign (per-node, capped at 1000)
- Tightens /api/fleet proxy-exempt prefix to /api/fleet/ so
/api/fleet-actions/* is routed through the proxy for per-node calls.
- Exports activeBulkActions from labels.ts so fleet-stop and label-action
share the per-node lock and cannot double-stop the same containers.
- Extracts containerActionForStack helper from stacks.ts for reuse.
* chore(fleet): rename Actions tab to Fleet Actions and reorder Fleet sub-tabs
- Tab label "Actions" -> "Fleet Actions" so the surface is unambiguous
alongside Schedules and the sidebar bulk bar.
- Reorder Fleet sub-tabs as Overview / Snapshots / Status | Deployments /
Traffic / Fleet Actions, with the separator after Status.
- Rename "Traffic · Routing" -> "Traffic" and update Sencho Mesh docs to
match the shorter label.
- Update Fleet Actions docs to the new tab name and placement.
* refactor(backend): sanitize user input before logging to close CRLF injection
Adds a small sanitizeForLog helper that strips CR, LF, tab, and ASCII
control characters (0x00-0x1F, 0x7F) from a value before it is embedded
in a console.log/warn/error/debug call. Wraps every call site where a
user-controlled value (req.params, req.body, req.query, or a value
derived from them) flows into a log message.
Closes the bulk of the open CodeQL alerts in this family:
- 96 js/log-injection
- 28 js/tainted-format-string
The helper is in backend/src/utils/safeLog.ts. Routes still pre-validate
input at the request boundary; this is the second line of defense and
gives static analyzers a sanitizer they can trace through. JSON
responses, Docker filter labels, and other non-log call sites are
intentionally left unwrapped.
* refactor(backend): printf-style format strings for tainted-log call sites
CodeQL's js/tainted-format-string rule flags template literals in the first
arg of console.X when any interpolated value is user-controlled, regardless
of whether each value is sanitized inline. The canonical mitigation is to
use a static format string and pass values as positional args.
Converts the 28 flagged template literals to printf-style ("%s") format
strings, with sanitizeForLog applied to each positional arg. Also fills in
the log-injection wraps on 9 sites where a user-controlled value was
missed in the first sweep (agents, fleet, gitSources, imageUpdates,
GitSourceService).
No behavior change at runtime. Node's util.format substitutes %s tokens
identically to template-literal interpolation.
* fix(backend): wrap nodeId/snapshotId in fleet restore debug log
CodeQL flagged the unwrapped numeric args even though they cannot
contain control chars in practice. Apply the sanitizer for taint-flow
recognition.
Introduce a NotificationCategory string-literal union (11 values) and
thread it through dispatchAlert as a required second argument. All
callers (DockerEventService, AutoHealService, ImageUpdateService,
MonitorService, PolicyEnforcement, policyGate, SchedulerService,
imageUpdates route) pass an explicit category at every call site,
giving TypeScript compile-time enforcement that no new emit site can
be added without choosing a category.
DatabaseService gains an idempotent migration that adds a nullable
category TEXT column to notification_history; existing rows keep
category=NULL (displayed as Uncategorized in the UI). The
getNotificationHistory method accepts an optional category filter
that is forwarded from the GET /api/notifications/history route via
a ?category= query param.
NotificationPanel gains a category Select dropdown so users can
filter history by category. The frontend types mirror the backend
union so API responses are type-safe end-to-end.
All 75 test files (1410 tests) updated to the new 4-arg dispatchAlert
signature and passing.
Updates the backend listen port, Vite dev proxy target, Docker EXPOSE,
compose port mapping, .env.example default, GitHub Actions smoke-test
default, healthcheck URLs, and every doc/example reference. Test fixtures
that include example URLs were updated for consistency, though their
assertions are port-agnostic.
The rate-limit value of 3000 in middleware/rateLimiters.ts and the
3000 entry in WEB_UI_PORTS (which detects user containers like Grafana)
are intentionally untouched.
Move four small Round C route groups out of index.ts:
- /api/auto-heal/* -> routes/autoHeal.ts
- /api/notifications/*, /api/notification-routes/* -> routes/notifications.ts
- /api/system/console-token -> routes/console.ts
- /api/sso/config/* -> routes/ssoConfig.ts
Share NOTIFICATION_CHANNEL_TYPES, cleanStackPatterns, and validateHttpsUrl
via a new helpers/notificationChannels.ts so agents.ts and notifications.ts
consume the same allowlist and URL validator.
Handlers moved verbatim. Middleware chains and response shapes preserved.
index.ts drops from 3231 to 2739 lines.
Second slice of Phase 4. Pulls the three auth-family route groups out of
index.ts into focused routers. All handlers move verbatim; index.ts drops
~845 lines.
New route files:
- routes/auth.ts: /api/auth core (status, setup, login, password, logout,
check, generate-node-token)
- routes/mfa.ts: /api/auth/login/mfa + full /api/auth/mfa/* surface
(status, enroll start/confirm, disable, backup-codes/regenerate,
sso-bypass)
- routes/sso.ts: /api/auth/sso/{providers,ldap,oidc/:provider/authorize,
oidc/:provider/callback} + getSSOBaseUrl helper. Module-load calls
SSOService.getInstance().seedFromEnv() so env-seeded providers are
available on the first request.
Shared lifts:
- helpers/constants.ts: MFA_REPLAY_TTL_MS + MFA_REPLAY_PURGE_INTERVAL_MS
(used by mfa.ts and the startup purge timer in index.ts) and
BCRYPT_SALT_ROUNDS (shared between setup and password-change handlers).
- middleware/auth.ts: new reissueSessionAfterTokenBump(req, res, userId)
helper collapses three copies of "bump → fetch user → re-sign cookie"
across auth.ts (password change) and mfa.ts (enrol confirm, disable).
Code review fixes:
- File-local requireEnrolledMfaUser helper in mfa.ts eliminates four
copies of "auth check + rejectApiTokenScope + load enrolled MFA" with
near-identical shape.
- Applied BCRYPT_SALT_ROUNDS to auth.ts setup + password handlers.
Mount order in index.ts: authRouter / mfaRouter / ssoRouter sit before
authGate because login / setup / SSO-callback are public; handlers that
need auth use authMiddleware directly on the route.
First slice of Phase 4 (route extraction). Pulls 8 well-tested, mostly
independent route groups out of index.ts into focused Router files. No
behavior change; every handler body moves verbatim.
New route files under backend/src/routes/:
- meta.ts /api/health, /api/meta (mounted before authGate)
- license.ts /api/license/* + /api/system/update,
exports scheduleLocalUpdate for the fleet route
- permissions.ts /api/permissions/me
- convert.ts POST /api/convert
- alerts.ts /api/alerts/*
- labels.ts /api/labels/* + PUT /api/stacks/:name/labels
(exported as stackLabelsRouter)
- apiTokens.ts /api/api-tokens/*
- auditLog.ts /api/audit-log/*
Shared helper lifts:
- helpers/cacheInvalidation.ts: invalidateNodeCaches()
- middleware/tierGates.ts: requireBody (was inline in index.ts)
- utils/errors.ts: isSqliteUniqueViolation (was inline in index.ts)
- middleware/apiTokenScope.ts: rejectApiTokenScope() helper (new)
- utils/csv.ts: escapeCsvField() (new)
index.ts drops from ~7520 to ~6775 lines and now mounts the routers right
after enforceApiTokenScope. The remote proxy and fleet/auth/webhooks/users
routes remain inline in index.ts pending later Phase 4 slices.
Code review fixes: rejectApiTokenScope helper replaces duplicated
`if (req.apiTokenScope) 403 SCOPE_DENIED` blocks in apiTokens.ts and
license.ts; escapeCsvField replaces the inline CSV escape in auditLog.ts.
Phase 3 of the index.ts refactor. Pulls the remote HTTP/WS proxy plumbing,
the WebSocket upgrade dispatcher, and the http/WSS construction out of the
monolith. index.ts drops roughly 620 lines.
New modules:
- proxy/websocketProxy.ts: shared httpProxy.createProxyServer singleton
(used by both the HTTP proxy middleware and the remote WS forwarder)
- proxy/remoteNodeProxy.ts: createRemoteProxyMiddleware() factory; consumes
the isProxyExemptPath helper instead of open-coding the prefix list
- server.ts: createServer(app) returns { server, wss, pilotTunnelWss }
- services/FleetUpdateTrackerService.ts: singleton wrapping the in-flight
fleet update tracker Map with create()/resolve() helpers
- helpers/consoleSession.ts: mintConsoleSession(), isConsoleSessionScope()
- websocket/upgradeHandler.ts: attachUpgrade(server, deps) dispatcher that
runs the manual cookie/JWT verify and delegates to sub-handlers
- websocket/pilotTunnel.ts: handlePilotTunnel (pilot_enroll consumption and
pilot_tunnel registration)
- websocket/notifications.ts: /ws/notifications local subscriber
- websocket/remoteForwarder.ts: remote-node WS proxy with console_session
token exchange for interactive paths
- websocket/logs.ts: /api/stacks/:name/logs supervisor stream
- websocket/hostConsole.ts: /api/system/host-console PTY, Admiral-gated
- websocket/generic.ts: /ws exec + streamStats action dispatch, owns the
terminalWs single-instance reference
- websocket/reject.ts: shared rejectUpgrade helper (replaces five copies)
Service extension:
- NotificationService: setBroadcaster(fn) replaced by subscribe(ws) that
returns an unsubscriber; broadcastToSubscribers is now internal. Subscriber
set lives on the service rather than in index.ts.
Wiring in index.ts:
- const app = createApp() already in place from Phase 2
- const { server, wss, pilotTunnelWss } = createServer(app)
- attachUpgrade(server, { wss, pilotTunnelWss })
- app.use('/api/', createRemoteProxyMiddleware())
- /api/system/console-token route now uses mintConsoleSession()
- deploy/down/update routes read the streaming target via getTerminalWs()
(return type is WebSocket | undefined so the || undefined fallback is gone)
Code review fixes: five duplicated reject helpers collapsed into
websocket/reject.ts; dropped the createTracker/resolveTracker bind
aliases in index.ts so call sites go through the service directly;
removed em dashes; replaced req.url! with req.url || '/'.
Phase 1 of the index.ts monolith refactor. Extracts all non-auth middleware
into focused modules. Routes and authMiddleware itself stay in index.ts for
now; Phase 2 introduces createApp() and extracts authMiddleware.
New modules:
- middleware/rateLimiters.ts: globalApiLimiter, pollingLimiter,
webhookTriggerLimiter, authRateLimiter, ssoRateLimiter, trivyInstallLimiter
plus the hybrid rateLimitKeyGenerator and isNodeProxyRequest helper
- middleware/jsonParser.ts: conditionalJsonParser that preserves the raw
stream for remote-proxy forwarding (via helpers/proxyExemptPaths)
- middleware/nodeContext.ts: nodeContextMiddleware
- middleware/apiTokenScope.ts: enforceApiTokenScope + DEPLOY_ALLOWED_PATTERNS
- middleware/authGate.ts: createAuthGate(authMiddleware) factory + auditLog.
Factory takes authMiddleware as a dependency to avoid a circular import
until Phase 2 extracts the auth module.
- middleware/errorHandler.ts: central error handler that preserves
err.status / err.expose from body-parser and other HTTP errors
- helpers/routePatterns.ts: WEBHOOK_TRIGGER_RE shared by rateLimiters and
authGate
index.ts drops ~290 lines. Middleware registration order is unchanged.
All 1278 tests pass.
Code review fixes: typed ApiTokenScope in apiTokenScope.ts; replaced 2
em dashes with colons (Directive 18); added local CachedProxyFlagReq
type alias for the node_proxy memoization cast; extracted deny() helper.
* refactor(backend): extract types, constants, and guards from index.ts (phase 0)
Additive, behavior-preserving first step of the modular backend refactor.
Moves purely static artifacts out of backend/src/index.ts so later phases can
extract routes and middleware without touching shared symbols.
New modules:
- types/express.ts: Express Request augmentation
- helpers/constants.ts: PORT, password policy, label colors, cookie names,
MFA TTLs, hot-path cache TTLs
- helpers/proxyExemptPaths.ts: PROXY_EXEMPT_PREFIXES + isProxyExemptPath
- helpers/cookies.ts: isSecureRequest, getCookieOptions
- helpers/policyGate.ts: buildPolicyGateOptions, runPolicyGate,
triggerPostDeployScan
- middleware/permissions.ts: ROLE_PERMISSIONS, checkPermission,
requirePermission
- middleware/tierGates.ts: requirePaid, requireAdmiral, requireAdmin,
requireNodeProxy, requireScheduledTaskTier + effectiveTier/Variant
index.ts shrinks by ~260 lines; no runtime behavior changes. All 64 vitest
files and 1,278 tests pass.
* refactor(backend): drop unused imports left after phase 0 extraction
LicenseTier, LicenseVariant, DIGEST_CACHE_TTL_MS, and isProxyExemptPath
were imported into index.ts but no longer referenced there after the
phase 0 move; CI lint flagged them as errors.
isProxyExemptPath will be re-imported in phase 1 when the JSON parser
bypass and nodeContext middleware get extracted. Silence the
no-namespace warning on the Express augmentation since the namespace
syntax is required for TypeScript module augmentation.
Anso