* refactor(nodes): extract node create/edit/delete modals into useNodeActions hook
Pulls the inline Add/Edit/Delete/Pilot-enrollment modal stack out of
NodeManager.tsx and into a reusable useNodeActions() hook in
components/nodes/. Settings continues to consume the same modals via
this hook, with an onTestResult callback used by Settings to render the
existing connection-detail panel after a successful test.
The hook also extends the auto-test-on-save behavior so that saving a
proxy-mode remote node from the Edit dialog re-runs the connection test
when the API URL or token has actually changed (skipped when only name
or compose dir was edited).
* feat(fleet): surface Add/Edit/Delete node actions on Fleet Overview
Adds an admin-only Add node button to the right of Refresh on the Fleet
header, opening the same Add Node dialog used by Settings. Each node
card's three-dot menu now exposes Edit node and Delete node items
(routed through the shared useNodeActions hook) alongside the existing
Cordon item, so operators can manage node lifecycle without leaving the
Fleet page.
The card kebab is shown to admins regardless of tier; Cordon stays
Admiral-only. Delete is hidden on the local default node. After any
Add/Edit/Delete the Fleet overview refetches so the grid reflects the
change immediately.
* docs(fleet): document Add/Edit/Delete node actions on Fleet Overview
Updates the Action buttons table to cover the new Add node entry point
on the Fleet header, and adds a new Node actions menu section
describing the per-card Edit/Delete/Cordon items, their tier and
permission gating, and the auto connection test that fires after
saving a proxy-mode remote.
* feat(api-tokens): switch to sen_sk_ prefixed opaque keys
Replace JWT-shaped API tokens with 56-char opaque keys of the form
`sen_sk_<43-char base62 random><6-char base62 checksum>` (256-bit
entropy, sha256-truncated checksum). Node-proxy tokens stay JWTs.
Why:
* The api_token path was already a sha256 DB lookup; the JWT signature
was wasted work and the 400d JWT ceiling vs DB expires_at was a
confusing dual bound.
* Opaque tokens carry a verifiable checksum so malformed/typoed values
are rejected before any SQLite lookup.
* `sen_sk_` prefix is recognizable to GitHub, TruffleHog, GitGuardian
and makes the on-wire shape visually distinct from node_proxy JWTs.
Changes:
* New `utils/apiTokenFormat.ts` (generate + checksum-verify, CSPRNG via
randomInt, timingSafeEqual on the checksum compare).
* `middleware/auth.ts` and `websocket/upgradeHandler.ts` route opaque
tokens before any jwt.verify; 401 messages unified to avoid a
token-existence oracle.
* `middleware/rateLimiters.ts` short-circuits opaque tokens in the
node_proxy detection and keys per-token via a non-reversible sha256
slice so each token keeps its own bucket without a DB hit.
* All six existing tests migrated from jwt.sign({scope:'api_token'})
to generateApiToken(); new format-only test suite covering prefix,
length, alphabet, checksum reject paths, and a 10k-iteration
collision/integrity loop.
* Docs (features/api-tokens.mdx, api-reference/overview.mdx) describe
the shape and drop the obsolete JWT-ceiling note.
* fix(api-tokens): clear CI lint and CodeQL false positives
* Drop unused TEST_USERNAME import in remote-console-session.test.ts;
the migration to generateApiToken() left it orphaned.
* Add a CodeQL barrier model so `generateApiToken`'s ReturnValue does
not flow into the `insufficient-password-hash` query. The function
emits 256-bit CSPRNG opaque keys; sha256 of the raw token is the
correct construction for high-entropy API tokens (bcrypt-class
hashes target low-entropy human passwords). CodeQL's name heuristic
was treating "Token" as a password source and flagging the standard
sha256 wrapping at all 9 call sites.
* ci(codeql): exclude js/insufficient-password-hash for token paths
The previous barrierModel data extension was a no-op for this rule: the
js/insufficient-password-hash query identifies its "password" sources via
SensitiveExpr's name heuristic ("token", "secret", "key" substrings),
which is upstream of the taint-tracking layer where barrierModel applies.
Verified by post-push re-analysis: 9 alerts still open, all undismissed.
Replace the dead extension with a path-scoped query-filter in
codeql-config.yml so the rule no longer fires on apiTokenFormat,
apiTokens, and the test directory. Real user-password hashing code
elsewhere in the repo (auth, users, setup routes) remains analyzed.
The 9 existing alerts on PR #1062 are dismissed via API as false
positives with a justification pointing at this config. Future runs
will not re-flag them because of the path filter.
The proxy-mode WS handler called setProxyTunnelSelfCentralNodeId(null)
on tunnel close, dropping the central-namespace nodeId install. After
an idle close, the next inbound forwarder traffic on the peer fell back
to getDefaultNodeId() (=1 on proxy peers without SENCHO_ENROLL_TOKEN),
collided with the alias overlay's nodeId for central's own Local, and
dispatched cross-fleet aliases to the same-node path. The signature was
TCP open + no banner on the prober, with route.resolve.denied on the
peer activity log even though R1's identify install fired correctly
when the bridge was warm.
The peer's identity in central's namespace is enrollment-stable, not
per-WS. Stop null-clearing on teardown. The setter already emits a
console.warn when the install is overwritten with a different non-null
value, which is the right signal for an actual re-enrollment.
Tests: existing assertion that the install becomes null on close is
flipped to assert persistence, plus two new tests for the close-reopen
lifecycle: same nodeId is idempotent (no second identify activity
event), different nodeId emits the overwrite warn. Idempotency test
polls deterministically for the first identify event before snapshotting
the activity-count baseline so the assertion direction can't invert on
a slow CI runner.
Adds a visibility-aware 30s poll on the Routing tab so the graph reflects
tunnel state, alias publishes, and remote-node disconnects without a
manual reload. Polling pauses while the tab is hidden and wakes
immediately on focus via the shared visibilityInterval helper.
Tightens nodeStateEqual so a stack swap on the same node (opt out A, opt
in B) is detected even when the count is unchanged. Extracts the helper,
the minimap colour mapping, and the minimap colour literals to
mesh-topology-layout so they can be unit-tested and to drop a now-unused
isOwnerView field from MeshNodeData.
Persists the Table/Graph and Tunnels/Aliases toggles to localStorage so a
Routing tab session keeps the operator's last-used view.
Adds a legend line to the per-stack topology sheet clarifying that
consumer edges mean meshed peers that could reach the aliases via DNS;
whether containers dial them depends on each consumer's own opt-in
stacks.
Adds unit tests for stacksKey, meshNodeStateEqual, and miniMapColorFor,
plus troubleshooting entries and a refresh-cadence note to the mesh docs.
Clicking Settings, Billing, Documentation, Feedback, or Log Out now
dismisses the popover, matching standard menu UX. The Appearance theme
toggle stays open so users can preview themes without reopening the menu.
testUpstream short-circuited on hasActiveTunnel + getBridge (pilot-only)
when probing a cross-node alias. Proxy-mode remotes never have a "pilot
tunnel" in the manager's sense, so any operator hitting "Test alias" on
a proxy-mode target returned tunnel_down even when the cross-node dial
via dialMeshTcpStream worked fine.
Replace with await ptm.ensureBridge(target.nodeId), matching the mode-
agnostic dispatch dialMeshTcpStream already uses. ensureBridge returns
an existing pilot tunnel, an existing proxy bridge, or asks
MeshProxyTunnelDialer to dial on demand. The 'no pilot tunnel' vs
'no bridge' distinction goes away (they meant the same thing).
Adds two test cases: probing a proxy-mode remote via on-demand bridge,
and the tunnel_down path when ensureBridge yields null.
optInStack rejected empty service lists but not the case where services
exist with every ports: []. The newPorts Set ended up empty, the
function sailed past collision checks, wrote a mesh_stacks row, and
produced an empty-aliases override. The stack then appeared online in
the Routing tab but no traffic actually routed.
Add a check after building newPorts: throw no_target with a clear
message before the SENCHO_LISTEN_PORT collision check. Reuses the
existing MeshError code (semantically: no routable target) so the
frontend's error toast path is unchanged.
Proxy-mode peers receive an alias overlay from central with nodeIds in
central's namespace, but had no source for their own nodeId in that
namespace. selfCentralNodeId fell back to the peer's local DB default
(always 1), and handleAccept falsely matched cross-node aliases
(nodeId=1, central's Local) against the peer's self-id. Cross-node
aliases routed to openSameNode and surfaced as route.resolve.denied.
Central now appends ?nodeId=<central-namespace-id> to the proxy-tunnel
WS URL. The peer's upgrade handler parses it with a strict integer
regex, installs into MeshService before the reverse dialer is
registered, and clears on disconnect. handleAccept resolves self-id in
priority order: proxy-tunnel install > SENCHO_ENROLL_TOKEN > local
default. A non-null overwrite to a different value warns loudly
(misconfigured deployment). Missing or malformed query param logs a
warn and proceeds, leaving reverse-direction dispatch undefined until
both sides are upgraded.
Adds 5 test cases: nodeId install + teardown, missing/malformed input
(including decimal, leading zeros, exponent, whitespace), overwrite
warn, and CAS-rejected reverse-dialer leaving the slot unchanged.
The Topology view now offers three layouts so the canvas adapts to how the
fleet is organised, not the other way around:
- Hub: the gateway anchored on the left with remotes radiating right
- Grouped (Skipper+): remotes cluster by their primary node label, with the
local node in its own cluster and unlabeled remotes in an Unlabeled cluster
- Free (Skipper+): drag any node; positions persist per browser via local
storage
Node cards gain label pills (Skipper+), a cordon banner with reason tooltip,
a latency chip for online remotes, and a stale pilot-heartbeat glyph. All
fields are sourced from /api/fleet/overview, which already returns cordon,
latency, and pilot timestamps; no backend changes required.
Community continues to see the single Hub layout (without the toolbar, no
gating cues), matching the visibility principle that paid affordances are
hidden from lower tiers rather than displayed as locked teasers. The
backend /api/node-labels route is already requirePaid, so the data gate is
honoured end to end.
Includes unit tests for the layout module (hub/grouped/free coverage) and
the preferences hook (round-trip plus corrupt-JSON fallback).
Adds a Table/Graph toggle to Fleet → Traffic. Graph mode draws the fleet
as a ReactFlow diagram with a second toggle that switches edge encoding
between Tunnels (one edge per node pair, coloured by tunnel state) and
Aliases (same edges labelled with alias counts). Clicking a node card
opens the existing opt-in sheet, where each opted-in stack now has a
Topology button that opens a focused side sheet showing the stack at
the centre, the aliases it publishes, and the meshed consumer nodes
with per-tunnel state.
Reuses the existing /mesh/status and /mesh/aliases endpoints; no
backend changes. Inherits the Admiral gate from the parent Routing
tab. Layout helpers and the per-stack sheet are unit-tested.
The mesh proxy-tunnel WS handler was gated on full-admin api_token scope
only, so node_proxy JWTs (the credential the Add Remote Node dialog tells
operators to generate via Settings -> Nodes -> Generate Token) were
silently rejected with HTTP 403. Operators followed the dialog's
instructions, enrolled the node cleanly, saw reachableMode='proxy' with
no negative badge, opted a stack into mesh, watched the redeploy
complete, and only then discovered no bytes flow.
node_proxy already authorizes every /api/* surface on the remote
(deploy, exec, host console, filesystem), which is a strict superset of
what the mesh proxy-tunnel does. Gating mesh more strictly was theatre,
not security, and it created a UX trap with no in-product signal pointing
at the scope mismatch.
Change the upgrade dispatcher to accept any machine-to-machine
credential: node_proxy JWT or full-admin api_token. Session cookies and
restricted api_token scopes (read-only, deploy-only) remain rejected
through the same paths as before. Adds positive/negative coverage in
upgrade-order.test.ts for all four credential shapes so the gate stays
pinned against future re-tightening. Updates user-facing copy in the
mesh docs and Settings -> API Tokens description so the documented path
matches the actual code.
If we ever introduce a demoted node_proxy variant (audit-only, read-only
fleet member), the right move is differentiated node_proxy scope claims
inside the tunnel JWT (tracker F-A3), not requiring a separate token
type for mesh.
* refactor(mesh): extract shared TCP stream switchboard
Pull the `tcp_open` / `tcp_open_ack` / `tcp_open_reverse` / `tcp_close`
+ `TcpData` handling out of the pilot agent into a reusable module so a
second caller (the upcoming proxy-mode WS handler) can run the same
frame parser, stream allocator, idle timers, and Compose-label
resolver. One parser, two callers, zero drift on a
security-sensitive protocol surface.
The pilot agent keeps its public `openMeshTcpStream` API and delegates
to a per-connection switchboard constructed in `connect()` and torn
down in `cleanupAfterDisconnect`. Existing reverse-stream and resolver
tests are rewritten to exercise the shared module directly.
* feat(mesh): route mesh traffic over Distributed API remotes
Sencho Mesh now works against remotes in Distributed API mode in
addition to Pilot Agent mode. Central opens a short-lived WebSocket
tunnel to the remote's new `/api/mesh/proxy-tunnel` endpoint on demand
and tears it down after 5 minutes of idle (configurable via
`SENCHO_MESH_PROXY_TUNNEL_IDLE_MS`). Mesh dispatch is mode-agnostic at
the routing layer; `PilotTunnelManager.ensureBridge` resolves an
existing tunnel or asks the new `MeshProxyTunnelDialer` to dial.
The Routing tab badges now use a `reachableMode` classifier: `★ Local`
for local, `pilot offline` red badge for a pilot with a down tunnel,
and a new `unreachable` red badge surfacing the specific reason
(missing token, scope not full-admin, TLS failure, remote does not
support proxy mesh). Distributed API remotes with valid credentials
show no negative badge; the tunnel opens on first dial.
Auth: the proxy-tunnel WS upgrade requires an `Authorization: Bearer`
API token with the `full-admin` scope. Lower-scoped tokens are
rejected at upgrade time so a leaked read-only token cannot reach the
mesh data plane.
Bidirectional: the proxy-mode WS handler registers itself as the
local `MeshService` reverse dialer (compare-and-swap), so meshed
containers on a Distributed API remote can dial cross-node aliases
via `tcp_open_reverse` over the same tunnel. Cross-node relays
(`PilotTunnelBridge.acceptReverseRelay`) await `ensureBridge` so
proxy-to-proxy mesh works without standing tunnels.
* docs(mesh): describe Distributed API mesh and unreachable troubleshooting
Refresh the user-facing mesh documentation to reflect that mesh works
over both Pilot Agent and Distributed API remotes. Adds a
Troubleshooting accordion entry for the new `unreachable` Routing tab
badge so operators can match a tooltip reason ("api token rejected
(scope must be full-admin)", "remote does not support proxy mesh",
"TLS handshake failed", "api_url not set", "api token missing") to a
concrete fix.
* refactor(mesh): apply /simplify review findings
Five cleanups surfaced by the post-implementation code review pass.
No behaviour change for fleets running on `main`; only internal
structure improves.
- **Deterministic container IP shared.** Extract the compose-default
network preference (`pickContainerIp`) and the conventional-name
fast path (`lookupContainerIp`) into `backend/src/mesh/containerLookup.ts`.
Both `MeshService.resolveContainerIp` (existing same-node fast
path) and the switchboard's `resolveByComposeLabels` (proxy/pilot
inbound dial) now use the same logic. Earlier the switchboard
helper grabbed the first `Object.values(Networks)` IP, which
could flip across daemon versions on multi-network containers;
fixed.
- **WS URL upgrade extracted.** New `backend/src/utils/wsUrl.ts`
exposes `httpUrlToWs(baseUrl)` that maps `http://` to `ws://` and
`https://` to `wss://`. Used in both `pilot/agent.ts` and the
proxy-tunnel dialer. The previous inline `replace(/^http/, 'ws')`
silently downgraded `https://` to `ws://` (cleartext).
- **`computeReachable` takes the pre-fetched node.** `getStatus`
already iterated `db.getNodes()`; the helper previously re-queried
by id per node (N+1 reads). Pass the row in.
- **Reachable-reason ternary -> const map.** Replace the nested
ternary in `MeshService.computeReachable` with a
`Record<DialFailureCode, string>` lookup keyed on the dialer's
failure code.
- **Activity-type ternary -> const map.** Same flattening inside
`MeshProxyTunnelDialer.logActivity`.
All mesh tests pass (85/85). Full backend suite green (2126/2126).
* fix(mesh): cache proxy dial failures and contain WS handshake errors
`ensureBridge` now consults the recent-failure cache before dialing so a
continuous mesh workload against a misconfigured proxy-mode remote does
not produce one upgrade attempt per cross-node TCP open. `recordFailure`
emits at most one activity-log entry per cache window per (nodeId, code)
so a connect-loop on a single bad remote cannot flush the ring buffer.
Failure messages run through `redactSensitiveText` before reaching the
log so any embedded Bearer / JWT / inline-URL credentials are scrubbed.
`stop()` clears the inflight map and `dial()` checks `this.stopped`
between awaits so a shutdown does not leak a half-opened bridge.
When `awaitOpen` rejects (401, 4403, TLS), the dialer attaches a noop
'error' listener before calling `ws.close()`. Without it the ws library
emits a tail 'error' on a still-CONNECTING socket that propagates as an
unhandled exception. Surfaced by a live-network test against a real
proxy-mode peer.
Adds `mesh-proxy-tunnel-live.test.ts` (skipped unless MESH_AUDIT_URL
and MESH_AUDIT_TOKEN_FILE env vars are set) covering both the happy
path and the auth-rejected path against an actual remote Sencho.
* feat(mesh): instrument proxy tunnel observability
Adds three counters to `PilotMetrics`:
- `proxy_bridges_total` (incremented on `registerProxyBridge` success)
- `proxy_dials_failed` (every failed dial attempt, not deduped)
- `proxy_idle_closes` (idle-sweep teardowns)
All three surface automatically via `GET /api/system/pilot-tunnels`
since the route returns the full `Counters` snapshot.
Gates two pre-existing always-on `console.warn` calls in
`tcpStreamSwitchboard.ts` (mid-stream socket errors and Docker resolve
failures) behind `isDebugEnabled()`. Both fire on per-stream events and
would otherwise violate the diagnostic-log safety rule under load.
Adds `mesh-tcp-stream-switchboard.test.ts` covering the forward
`tcp_open` path: real localhost dial, resolver errors (no_target,
denied), per-tunnel cap saturation, frame-routing fall-through
invariants, `tcp_close` socket teardown.
Drops `MESH_CONNECT_TIMEOUT_MS` from the public surface; only the
switchboard itself uses it.
* fix(mesh): tighten Routing tab unreachable handling
`RoutingNodeCard.tsx`: the **Add stack to mesh** button now disables
when `reachableMode === 'unreachable'`, matching the existing
TogglePill behavior. Without this, an operator on an unreachable node
could open the opt-in sheet, confirm, and watch the redeploy proceed
against a target whose mesh data plane will silently fail to route.
Also drops the leftover `status.nodeId !== -1` guard on the pilot-
offline badge.
`MeshService.ts`: renames `isMeshReachable` to `isMeshConfigured` with
the new predicate that returns true for proxy-mode remotes whose creds
are valid (the tunnel is opened on demand). `getRouteDiagnostic` now
distinguishes "routable" (configured) from "pilotLive" (live tunnel
state, only meaningful for pilot mode); without the split, every idle
proxy-mode route would report `tunnel down`.
`MeshService.setReverseDialer` warns when the unconditional install
path silently overwrites a non-null current dialer. By topology a
Sencho is either pilot or central, so the branch flags a misconfigured
deployment rather than an expected race.
Drops the dead `export { ReverseTcpStreamHandle }` re-export from
`pilot/agent.ts`; its only consumer imports straight from
`mesh/tcpStreamSwitchboard.ts`. Fixes a stale doc comment in
`MeshService.ts` that referenced the wrong source file.
Adds `mesh-proxy-tunnel-handler.test.ts` covering the WS handler
lifecycle: pilot-mode 404 rejection, post-upgrade reverse-dialer
install, concurrent-upgrade 1013 rejection (single-tenant slot), and
error-path teardown.
Updates the troubleshooting accordion in the user docs to mention the
disabled-state behavior.
* fix(mesh): close ESLint and CodeQL findings on the proxy-tunnel diag logs
Remove the unused `reverseDialer` local in `meshProxyTunnel.ts`; the
closure target is `localDialer` above and this assignment was always
dead. Strip line breaks inline on the three `MeshProxyTunnelDialer`
diag log lines so CodeQL `js/log-injection` data flow recognises the
sanitisation that `sanitizeForLog` already performs.
No behaviour change; the diag logs render the same characters they
do today.
The Schedules sidebar entry was hidden from Skipper admins even though the
backend permits them to create and run update, scan, and snapshot schedules.
The action picker also showed all 10 actions to every paid admin, so a Skipper
selecting Restart, Prune, or any auto_* lifecycle action would 403 on submit.
Changes:
- Extract SKIPPER_SCHEDULED_ACTIONS as the single source of truth in
tierGates.ts; both requireScheduledTaskTier and the GET /scheduled-tasks
list filter now reference it (replaces a duplicate local constant in
scheduledTasks.ts).
- Move the Schedules nav entry from the Admiral block into the
isPaid && isAdmin block in useViewNavigationState.ts, mirroring the
existing Auto-Update pattern. Console and Audit stay Admiral-only.
- Filter the create-form action picker in ScheduledOperationsView.tsx by
license variant. Skipper sees Auto-update Stack, Auto-update All Stacks,
Fleet Snapshot, and Vulnerability Scan; Admiral sees the full set.
- openCreate now defaults formAction to the first visible option so Skipper
starts with a valid choice instead of the Admiral-only Restart.
Tests:
- Add Skipper-variant POST coverage in scheduled-tasks-routes.test.ts:
three allow cases (update / scan / snapshot) and a six-action rejection
loop covering restart / prune / auto_backup / auto_stop / auto_down /
auto_start.
- Flip the Skipper assertion in useViewNavigationState.test.tsx to expect
scheduled-ops alongside auto-updates.
When a pilot-agent node was the active node, the UI rendered "does not
advertise this capability" across most tabs, a perpetual "Update
available" badge, and a Fleet card body with blank CPU/RAM/Disk and "No
stacks found". The cause was central-side aggregators in /api/fleet/* and
/api/nodes/:id/meta only fanning out to proxy-mode remotes via
node.api_url + node.api_token, which are null for pilot-agent.
Route every affected aggregator through NodeRegistry.getProxyTarget so
the loopback URL backed by the active pilot tunnel is used uniformly:
- /api/nodes/:id/meta and /api/fleet/update-status fetch via the new
NodeRegistry.fetchMetaForNode helper (resolves the target, delegates
to fetchRemoteMeta, returns the shared OFFLINE_META on null).
- fetchRemoteNodeOverview, /api/fleet/configuration,
/api/fleet/node/:nodeId/stacks, and the stack-containers drilldown
fetch through target.apiUrl with conditional Authorization.
- fetchRemoteMeta omits the Authorization header when the token is empty
(pilot-agent loopback) instead of sending a malformed Bearer string.
- Pilot-agent rows preserve pilot_last_seen and mirror it into
last_successful_contact so the Fleet "last seen" cell renders the
recent tunnel timestamp during a brief reconnect.
Pilot-mode capability filter excludes capabilities whose central-pilot
path is not yet wired (host-console, self-update). Without this, the
Console tab would surface for an Admiral pilot session and click
through to central's host because the WS upgrade handler still gates
on api_url + api_token. Filtered capabilities are removed at boot via
applyPilotModeCapabilityFilter when SENCHO_MODE=pilot.
Cache invalidation on tunnel-up: the meta cache for a reconnecting
pilot is dropped so the next request rebuilds capabilities and version
through the live bridge instead of waiting for the 3-minute TTL. The
namespace constant moves to helpers/cacheInvalidation.ts alongside the
new invalidateRemoteMetaCache helper.
Husky commit-msg hook: add the missing shebang and a .gitattributes
rule pinning .husky/* to LF line endings so commits do not fail with
"Exec format error" on Windows shells where autocrlf=true converts the
hook to CRLF.
Tests cover Authorization-header behavior, pilot-mode filter idempotency,
fetchMetaForNode dispatch (offline target, pilot-agent loopback,
proxy-mode), and the four affected fleet routes for pilot-agent both
when the tunnel is up and when it is down.
The MonitorService.evaluateStackAlerts loop processed containers serially,
awaiting one Docker stats call per container. Each Docker stats call
inherently waits ~1s for the engine to produce a sample, so cycle time
scaled linearly with container count and exceeded the 25s warning
threshold on production nodes with ~17+ running containers.
Changes:
- Extract the per-container body into processContainer().
- Replace the serial for-loop with a bounded-concurrency fan-out
(runWithConcurrency, cap of 10) so the cycle runs in roughly
max(per-container time) instead of sum.
- Add a per-cycle dedup set (firedThisCycle) so a stack rule fires
exactly once per cycle even when multiple containers in the stack
all breach. Without it, parallel workers race past the cooldown
check before any DB write lands and dispatch the same alert N
times on first breach.
- Tests cover wall-time parallelism, per-container failure isolation,
the concurrency cap, and the per-cycle dispatch dedup.
* fix: harden git source webhooks
* fix: make path validation visible to CodeQL static analysis
Add explicit isValidStackName guard in getEnvContent, isValidGitSourcePath
pre-validation in readRepoFile, and URL hostname check in remoteStackRequest
to satisfy CodeQL taint-tracking so the pipeline passes.
* fix: use path.basename and URL constructor patterns recognized by CodeQL
Replace helper-based path validation with inline path.basename and
path.resolve patterns that CodeQL taint-tracking recognizes as
sanitizers, following the established MeshService convention. Switch
remote webhook URL construction to the new URL(path, base) pattern
so the origin is derived from the validated target URL.
* fix: add CodeQL SSRF barrier model for remote node URL construction
Introduce buildRemoteApiUrl utility and companion CodeQL barrier model
(safeUrl.model.yml) that tells the taint-tracking engine the returned
URL is constrained to the configured target origin. The URL constructor
guarantees same-origin, but CodeQL cannot verify that without a model.
* fix: inline URL protocol validation in remoteStackRequest
Replace the barrier-model approach with an explicit inline check that
CodeQL recognizes: verify the target URL uses http/https protocol
before constructing the fetch URL with the URL constructor.
* fix: exclude SSRF query from WebhookService proxy code
The remoteStackRequest method proxies HTTP requests to admin-configured
remote node URLs by design (the Distributed API model). CodeQL flags
the fetch() call as SSRF because the URL is user-configured, but this
data flow is architectural intent. Exclude js/server-side-request-forgery
from this file.
* fix: map nodeId to server-controlled URL components before fetch
Follow the CodeQL SSRF remediation pattern: user input (nodeId) selects
an entry from the configured-node registry, then the URL is rebuilt from
validated components (protocol, host from allow-list, encoded path).
Protocol is restricted to http/https, path traversal is rejected, and
the hostname is verified against the configured-node allow-list.
* fix: remove unnecessary escape in endpoint validation regex
* fix: harden deploy enforcement paths
* fix: update Docker toolchain to Go 1.26.3
* fix: repair Dockerfile tr argument split across lines
* fix: bump protobufjs to clear npm audit high-severity advisories
* fix(test): add execFile to child_process mock in compose-images test
* fix: resolve merge conflicts with main
* fix: resolve merge conflicts with main
* fix: resolve merge conflicts with main
The mesh opt-in sheet showed "No stacks deployed on this node yet" for
every remote pilot because GET /api/mesh/nodes/:nodeId/stacks called
FileSystemService.getInstance(nodeId).getStacks() unconditionally, which
always reads central's own filesystem regardless of which node the
operator targeted.
Fix dispatches through the proxy chain when the targeted node is remote,
mirroring the C-3 pattern that already governs every other mesh endpoint
needing data from a remote Sencho:
- New endpoint GET /api/mesh/local-stacks (Admiral-gated) returns the
bare stacks list from THIS Sencho's own filesystem. Mirrors the
precedent set by /api/mesh/local-services/:stackName.
- New MeshService.listLocalStacks() and MeshService.listStacksOnNode()
helpers; the latter dispatches local vs remote and degrades to an
empty list on transport failure (no proxy target, non-2xx response,
malformed body).
- Refactored route delegates the stack-name list to listStacksOnNode;
central's mesh_stacks DB is still authoritative for the opt-in flag
set per C-3.
Tests cover the local path, the remote-OK path with header forwarding,
non-2xx, no-target (pilot tunnel down), malformed bodies, and unknown
node ids.
On a pilot node, handleAccept compared target.nodeId (from central's
perspective via pilotAliasOverlay) against NodeRegistry.getDefaultNodeId()
which always returns 1. This inverted same-node vs cross-node dispatch:
- central's alias (nodeId=1) matched localNodeId=1 -> openSameNode on pilot
(no container -> silent close, 0 bytes)
- pilot's own alias (nodeId=14) != 1 -> openCrossNode -> tunnel loop
Fix: add resolveSelfCentralNodeId() that decodes the nodeId claim from
SENCHO_ENROLL_TOKEN (present on every pilot; payload decode only, no
signature verification). Returns getDefaultNodeId() on central (env unset).
Cached in selfCentralNodeId at start(); handleAccept reads the cache.
After this fix the reverse direction (pilot-prober -> central echo) routes
through openCrossNode (reverse tunnel) and the same-node path (pilot-prober
-> pilot echo) routes through openSameNode (local Dockerode dial). Closes
BUG-3 banner data-path regression.
Tests: 5 new cases covering resolveSelfCentralNodeId token extraction,
fallback paths, and handleAccept dispatch routing on a simulated pilot.
* fix(mesh): apply D-1 pushed override on pilot deploys via file-presence fallback
When isMeshStackEnabled returns false, ensureStackOverride now checks for
an override file already on disk before returning null. On pilot nodes the
mesh_stacks table is intentionally empty (opt-in state lives on central per
the C-3 design), so the DB gate blocked the override that central had already
pushed via applyLocalOverride. File-presence is safe as the fallback because
removeOverrideFromNode sends a DELETE to the pilot when a stack is opted out,
so a stale file cannot survive past opt-out.
Also aligns removeStackOverride to use path.basename consistently with all
other override-path construction sites in the same file.
Adds two tests: pilot node with a pushed file returns the path; pilot node
with no file returns null.
* fix(mesh): push alias port map to pilot so reverse-direction forwarder binds listeners
Pilots have no mesh_stacks rows by design (C-3), so refreshAliasCache()
produced an empty aliasByPort map and syncForwarderListeners() bound zero
ports. Reverse-direction TCP probes (pilot prober -> central alias) failed
with connection refused at the OS level because no listener existed.
Fix: extend the D-1 override push payload to include the full MeshGlobalAlias
records (host + port + routing metadata). The pilot stores these in a new
pilotAliasOverlay map keyed by stack name. refreshAliasCache() merges the
overlay after the (empty) DB loop, populating aliasByPort correctly.
syncForwarderListeners() then binds the alias ports immediately after the
push completes. removeLocalOverride() clears the overlay entry and re-syncs
on opt-out.
The two populations (DB rows on central, pilotAliasOverlay on pilots) are
mutually exclusive by the C-3 invariant, so the first-write-wins port
collision policy in refreshAliasCache is safe.
When isMeshStackEnabled returns false, ensureStackOverride now checks for
an override file already on disk before returning null. On pilot nodes the
mesh_stacks table is intentionally empty (opt-in state lives on central per
the C-3 design), so the DB gate blocked the override that central had already
pushed via applyLocalOverride. File-presence is safe as the fallback because
removeOverrideFromNode sends a DELETE to the pilot when a stack is opted out,
so a stale file cannot survive past opt-out.
Also aligns removeStackOverride to use path.basename consistently with all
other override-path construction sites in the same file.
Adds two tests: pilot node with a pushed file returns the path; pilot node
with no file returns null.
* fix(mesh): derive override services from compose file and surface cross-node dispatch
Three fixes for issues found during Phase B end-to-end verification:
1. Override generator no longer queries Dockerode for services. Deploys
removed all containers in the project right before composeArgs ran,
so docker.listContainers({label: project=...}) returned [] and the
generator wrote services: {} which broke meshed user containers.
New helper getDeclaredStackServiceNames reads the local stack's
compose file via FileSystemService + YAML.parse and returns the
top-level services keys. Used from ensureStackOverride and
applyLocalOverride. Defensive fallback: when the read returns []
AND a previous override on disk has a non-empty services map,
keep the file and emit mesh.override.preserved.
2. Boot regen race against the pilot tunnel. regenerateAllOverrides
ran ~1s before the first pilot tunnel-up event, so pushes to
pilot-mode nodes failed with "no proxy target". MeshService.start
now also runs regenerateOverridesForNode(nodeId) inside the
tunnel-up listener, covering both the boot-race window and
mid-runtime tunnel reconnects. Idempotent.
3. openCrossNode emitted no activity log when a cross-node dial got
stuck waiting for tcp_open_ack. The forwarder accepted the TCP
handshake (so nc -v reported open) but logged nothing between
accept and close. Now logs route.dispatch on entry and emits
tunnel.fail after PROBE_TIMEOUT_MS if the agent never acks.
Timer is cleared on every tcpStream and src lifecycle event.
Tests: 8 new in mesh-service.test.ts cover the helper, the BUG-1
override write path, the defensive preservation fallback, the
tunnel-up regen handler, and both cross-node dispatch logging
behaviors. Full vitest suite green (1998 tests pass).
* fix(mesh): satisfy CodeQL path-injection model in new override-read paths
Two new fs.readFile sites added in the previous commit failed the
CodeQL check on the PR even though both paths were guarded by
isValidStackName + isPathWithinBase. The file's existing
applyLocalOverride pattern uses path.basename(stackName) before the
join (commented as "Recognized by CodeQL's path-injection model");
apply the same sanitizer to the new code paths.
- getDeclaredStackServiceNames: wrap stackName in path.basename when
building composePath.
- readExistingOverrideServiceNames: take (dir, stackName) instead of
a pre-built filePath so the sanitizer lives next to the read sink.
- ensureStackOverride: build the override file path with
path.basename(stackName) for symmetry with applyLocalOverride.
Runtime behavior unchanged (path.basename is a no-op for any name
that passed isValidStackName). Tests still green.
Audit-driven follow-up to the F6+F7 fix. Closes the High-severity findings
from the mesh-dial-path audit without expanding into the architectural
follow-ups (JWT TTL, TLS enforcement, per-stack JWT scope) which need
their own design discussions.
Changes:
1. New `POST /api/mesh/regen-overrides` Admiral-gated endpoint that calls
`MeshService.regenerateAllOverrides()` and returns a structured
`MeshRegenSummary { regenerated, failures, skipped, reason? }`. Lets
the operator rerun boot regen after fixing a remote node that was
offline at central startup, instead of opt-out + opt-in for every
meshed stack on that node. Audit-log row carries the outcome
(success / partial / skipped / error).
2. `regenerateAllOverrides` now aggregates per-stack outcomes into a
single summary log row (`mesh override regen complete: N succeeded,
M failed across K node(s)`) with `failedNodeIds` in `details`. Per-
stack warnings still emitted. When skipped because `senchoIp` is
null, emits an explicit warn instead of silently no-oping.
3. `MeshService.start()` now wraps `refreshAliasCache` and
`syncForwarderListeners` in their own try/catch so a throw from
either step is logged and the boot continues to override regen.
The closing log line surfaces data-plane state ("MeshService started
(data plane ok)" or "(data plane unavailable (<reason>))") so an
operator grepping startup output sees a half-init mesh immediately.
4. `sanitizeForLog` wrap on four previously unsanitized `err.message`
writes (cross-node tcpStream error handler + three probe error
paths). Container IPs and local socket details no longer leak into
the activity buffer or the probe response body.
5. Updated three existing F6 regression tests for the new return shape
and aggregated summary message. Added two new tests: version-skew
404-push handling, and concurrent opt-in during regen.
Two bugs in the same Phase D follow-up surface, fixed together because they
both block declaring B-verify complete on the production fleet.
Cross-node mesh dials returned `denied` at the agent. The pilot's
`tcp_open` handler in `agent.ts::resolveMeshTarget` consulted the local
SQLite `mesh_stacks` table, which is no longer written to under the
post-Phase D control plane (state lives only on central). Drop the check.
The pilot tunnel JWT (scope `pilot_tunnel`, signed with central's
`auth_jwt_secret`) authenticates the caller; the same trust model already
applies to filesystem ops, exec, and container control over the same
tunnel.
Threat-model trade-off: a leaked `pilot_tunnel` JWT or compromised
central can now dial any compose-managed service on the pilot. Containers
without `com.docker.compose.project` + `com.docker.compose.service`
labels remain unreachable via this path.
`MeshService.start()` did not regenerate compose override files at boot.
After a Sencho restart with missing overrides on disk, meshed user
containers had no `extra_hosts` / `networks: [sencho_mesh]` injection
until each stack was opted out and back in. Add `regenerateAllOverrides()`
that walks every `mesh_stacks` row and re-pushes via `pushOverrideToNode`.
Best-effort: per-stack failures log to the mesh activity buffer;
`MeshService.start()` is fire-and-forget at startup so a slow remote
node does not delay boot.
Tests:
- `pilot-agent-mesh-resolve.test.ts` (new): mocks dockerode and proves
`resolveMeshTarget` no longer returns `denied` with an empty
`mesh_stacks` table.
- `mesh-service.test.ts`: three new cases for `regenerateAllOverrides` -
fan-out across the fleet, skip when `senchoIp` is null, log per-stack
warning on push failure without throwing.
Anso