When the proxy-mode mesh peer already holds an open peer-initiated callback
bridge to central, central's Trigger 2 dial (api_token rotation) arrives at
the peer's `/api/mesh/proxy-tunnel` handler with a fresh `mesh_handshake`
frame. The peer's `attachSwitchboard` refuses the new WS at the reverse-dialer
CAS swap, but it was also closing the connection before its `'message'`
listener had been attached, so the handshake frame went unread and the
peer's cached callback JWT stayed at the pre-rotation fingerprint until a
central restart.
Restructures `attachSwitchboard` to attach the `'message'`, `'close'`, and
`'error'` listeners synchronously after the switchboard is created, before
the async MeshService import and the CAS swap run. On the CAS-fail branch,
holds the WS open for up to FIRST_FRAME_WAIT_MS (30ms, well above localhost
RTT and below operator perception) before sending the 1013 close, so an
in-flight first frame lands in `onMessage` and the bootstrap material is
persisted via `MeshCentralRegistry.upsert` regardless of bridge fate. The
success path is untouched; reverseDialer install timing on accepted dials
is unchanged.
Adds a regression test that pre-installs a stub reverse dialer (simulating
a peer-initiated bridge owning the slot), sends a `mesh_handshake` frame
on the new WS, and asserts the registry receives the rotated material
before the 1013 close fires.
* fix(mesh): peer-initiated callback bridge installs reverse dialer
The R1-A2 symmetric-dial work added a peer-side dialer that opens a
callback WS to central at `/api/mesh/proxy-tunnel-from-peer`. The WS
upgrade, JWT validation, and `mesh_centrals` persistence all worked,
but the peer-side handler put the wrong object on its end of the pipe:
a `PilotTunnelBridge` (the central-role object) rather than a
`TcpStreamSwitchboard` with a registered `reverseDialer`.
The design doc states: "Central retains PilotTunnelBridge ownership;
peer retains TcpStreamSwitchboard + reverseDialer ownership." The
central-initiated handler at `meshProxyTunnel.ts:115-163` honors this.
The peer-initiated handler at `PeerToCentralMeshSessionDialer.attachBridge`
did not. It created a `PilotTunnelBridge(0, ws)` and stored it in a
private `currentSession` field. Result: `MeshService.reverseDialer`
stayed null, `dialMeshTcpStream` fell through to
`PilotTunnelManager.ensureBridge(target.nodeId)`, and `NodeRegistry.
getProxyTarget` on the peer returned null because proxy-mode peers
do not enroll their central. Cross-fleet dispatch from a peer
container failed with `proxy-tunnel.open.fail nodeId=<central>
reason=no_target` even though the callback bridge was up and healthy
(`bridgeOpen: true`, `last_used_at` populated).
Replace the peer-side `PilotTunnelBridge` with the same wiring the
central-initiated handler uses:
- `attachTcpStreamSwitchboard` with `resolveByComposeLabels` (allows
inbound `tcp_open` from central if it ever uses the callback
bridge for a reverse-direction dispatch; matches the central-
initiated pipe's resolver).
- A `SwitchboardReverseDialer` that delegates `openMeshTcpStream`
to `switchboard.openReverseStream`.
- `MeshService.setReverseDialer(localDialer, null)` with the same
CAS guard the central-initiated handler uses, so a concurrent
central-initiated tunnel does not get silently overwritten.
- `ws.on('message', onMessage)` dispatches JSON / binary frames to
the switchboard.
- `ws.once('close'/'error', teardown)` clears the switchboard, the
reverseDialer, and the `currentSession` reference idempotently.
The `currentSession` field type changes from `PilotTunnelBridge | null`
to `TcpStreamSwitchboard | null`. Callers in `MeshService.openCrossNode`
only use the return value for truthiness; the cast is harmless. The
public `hasSession()` signature is unchanged.
A new `currentWs` field holds the WS reference so `resetForTest` can
close the connection on teardown (the switchboard's own `cleanup`
does not close its WS).
Test update mirrors the production wiring: the existing
"marks the row used on successful WS open" case now asserts the
return value is a `TcpStreamSwitchboard`, that `MeshService.reverseDialer`
is non-null after `ensureSession`, and explicitly closes the WS in
the test's finally so `wss.close` can resolve.
Pre-existing test logic that asserted `bridge instanceof PilotTunnelBridge`
on the same path is removed in favor of the new switchboard assertion.
Discovered during the v0.81.1 live verification: the peer-side
`route.dispatch cross-node` event always paired with
`proxy-tunnel.open.fail no_target` and `tunnel.fail`, even though
`centralCallback.bridgeOpen: true` and `mesh_centrals.last_used_at`
were populated. The callback path opened cleanly in isolation but
the dispatch path never consumed it. This fix makes the dispatch
path consume it.
Tests:
- cd backend && npx tsc --noEmit clean
- cd backend && npx vitest run mesh 170/170 green
- cd backend && npx vitest run 2259/2268 green; only
failures are the pre-existing Windows EBUSY flake in
filesystem-backup.test.ts (documented in prior handoff)
* fix(mesh): drop unused imports and align reverse-dialer interface name
CI lint failures on the previous push were two @typescript-eslint/no-unused-vars
errors in the test file:
24:76 'vi' is defined but never used
31:5 'TcpStreamSwitchboardCtor' is assigned a value but only used as a type
Both were collateral from a local-debugging simplification: vi.waitFor was
swapped for a synchronous assertion when the test's wss.close hang was traced
to an unclosed client WS; the toBeInstanceOf assertion that used the runtime
binding was removed at the same time. Restore the instanceof assertion next
to the existing not.toBeNull check so the runtime binding is in use and the
test guards against future regressions where the wrong end-of-pipe object
type is returned on the callback bridge (the bug this PR fixes). Drop the
now-unused `vi` symbol from the vitest import.
Also fold in a /simplify convergent finding: rename the local
CallbackReverseDialer interface to SwitchboardReverseDialer to match the
sibling declaration in meshProxyTunnel.ts:72. Same shape, same name; readers
of either handler now see the same mental model. Cross-file extraction of
the interface to tcpStreamSwitchboard.ts is the cleaner long-term shape but
touches a file outside this PR's scope; filing as a follow-up.
Considered-and-deferred findings from the /simplify pass (file as separate
PRs to keep one-branch-one-concern):
- extract shared attachSwitchboard helper (~40 duplicated lines between
meshProxyTunnel.ts and PeerToCentralMeshSessionDialer.ts)
- public TcpStreamSwitchboard.getWs accessor + drop currentWs field
- public MeshService.hasReverseDialer for the test bracket-cast
- shared close-reason constants (drift risk across the two handlers)
Tests:
- cd backend && npx tsc --noEmit clean
- cd backend && npx eslint <changed> clean (lint_exit=0)
- cd backend && npx vitest run mesh 170/170, dialer 8/8 green
PR #930 opened the backend security routes to Community admin but left
several frontend isPaid gates in place, so the matching UI surfaces
stayed hidden from Community even though the API would accept the
request. This brings the UI in line with the backend matrix.
Flipped:
- ResourcesView Scan history button (was trivy.available && isPaid)
- ResourcesView Full scan (vulnerabilities + secrets) menu item
- ResourcesView VulnerabilityScanSheet canCompare
- ResourcesView VulnerabilityScanSheet canManageSuppressions (now isAdmin)
- EditorView stack-level Scan config button (canScan)
- SecurityHistoryView primaryAction (Compare)
- SecurityHistoryView VulnerabilityScanSheet canManageSuppressions
Unchanged on purpose (still paid, matching backend gates):
- canGenerateSbom (SBOM endpoint requirePaid)
- SARIF download in the drawer overflow
- Scan Policies block in Settings
- Trivy auto-update toggle (requireAdmiral)
Test: inverted the SecurityHistoryView.test.tsx assertion that locked
in the now-incorrect "hides Compare on Community" behavior.
The R1-A2 symmetric-callback work added a peer-side recovery branch to
MeshService.openCrossNode: when reverseDialer is null, kick
PeerToCentralMeshSessionDialer.ensureSession so the peer re-opens its
callback WS to central before the cross-node dispatch.
The guard `!this.reverseDialer` did not distinguish "I am a peer waiting
for central to dial in" from "I am central, and the reverseDialer concept
does not apply to me." On central, reverseDialer is always null (central
is the bridge initiator side and never has another node dial into it), so
central entered the recovery branch on every forward dispatch, found no
session (central has no mesh_centrals row), and destroyed the inbound
socket with route.resolve.fail forward-from-peer no_session.
Gate the branch on `MeshCentralRegistry.getActive() !== null` so it only
runs when this Sencho is acting as a proxy-mode peer that has actually
been bootstrapped. Central falls straight through to dialMeshTcpStream.
The peer cold-start scenario the comment originally described is still
covered: a proxy-mode peer with a cached mesh_centrals row but no live
WS to central will fire the recovery branch as before.
Adds two regression tests:
- central path (no mesh_centrals row, no reverseDialer) skips recovery
and calls dialMeshTcpStream
- proxy-peer path (mesh_centrals row + ensureSession returns null)
still emits route.resolve.fail forward-from-peer no_session
Existing tests in the "openCrossNode (BUG-4)" block already used a stub
reverseDialer to bypass the recovery branch, which is why the central
regression was not caught pre-release. The new tests deliberately leave
reverseDialer null to exercise the gating.
* chore(mesh): foundation for symmetric callback dial
Adds the data-plane scaffolding that the symmetric callback dial fix
builds on:
- mesh_centrals table for peer-side bootstrap material
- MeshCentralRegistry service (upsert/getActive/clear/markUsed/markRejected)
- PilotTunnelManager kind discriminator and replaceOrRegisterProxyBridge
- mesh_proxy_callback_bootstrap capability registration
- MeshProxyTunnelDialer reason-tagged proxy-bridge-down events from a
single tearDownBridge emission point
- Reactive redial scheduler that skips idle and auth_failed reasons
* feat(mesh): add reverse-direction activity log entries (closes R1-B)
acceptReverseLocal now emits route.resolve.ok with direction=reverse on
connect ack and route.resolve.fail with direction=reverse plus
reason=container_not_found / connect_error pre-connect. Post-connect
close/error stays silent. Reuses existing event types via the new
details.direction discriminator so frontend filters are unaffected.
* feat(mesh): add peer-to-central callback dial path (closes R1-A2)
Closes the architectural gap where proxy-mode mesh peers could not
re-establish their tunnel to central after any non-idle bridge teardown
(idle close, network blip, central restart, peer reboot). Central remains
the hub for the data plane; the change is purely about WS initiation.
Symmetric WS initiation, asymmetric protocol roles. Central retains
PilotTunnelBridge ownership; peer retains TcpStreamSwitchboard +
reverseDialer ownership. Central bootstraps callback credentials over
the first authenticated central-initiated mesh tunnel via a one-shot
mesh_handshake JSON frame; peer persists the material in a new
mesh_centrals SQLite table and dials central's new
/api/mesh/proxy-tunnel-from-peer endpoint when local cross-node traffic
needs a bridge and none is live.
Mesh_tunnel JWT (HS256, signed with auth_jwt_secret) carries scope, audience,
issuer (central instance id), peer api_token fingerprint, kid. Validation
on inbound peer dial: algorithm pin, signature, scope, audience, instance,
time bounds, node existence and mode, fingerprint match. Failures return
HTTP 401 with a machine-readable reason; peer routes the response per a
clear-vs-keep cache matrix.
Triggers proactive bootstrap on mesh-enable and api_token rotation; central
startup fans out to mesh-enabled proxy-mode nodes with mesh_stacks rows
(throttled, fire-and-forget). Reactive redial on non-idle bridge loss.
Capability-gated handshake send (mesh_proxy_callback_bootstrap) makes the
upgrade path safe against older peers in mixed-version fleets.
Adds peer-side /api/system/pilot-tunnels centralCallback diag block,
bounded counter metrics for bootstrap and dial events. SENCHO_PRIMARY_URL
preflight warning when unset on a central with mesh-enabled proxy nodes.
Tested with unit suites for the validation chain, registry, manager, and
both dialers; integration tests for bootstrap E2E (asserts protocol-role
invariant), api_token rotation, instance id change, version skew, and
pilot-mode regression.
* fix(mesh): green CI on the symmetric callback branch
Two independent CI failures, both surgical:
1. Backend tests (11 fails): four mesh test files called setupTestDb in
beforeEach. setupTestDb does not reset the DatabaseService singleton,
so the per-test afterEach rm of the previous tmpdir left the singleton
connection pointing at a deleted file. The next beforeEach's line-55
write threw SQLITE_READONLY_DBMOVED on Linux. Windows file-lock
semantics hid this locally. Hoist setupTestDb / cleanupTestDb to
file-scope beforeAll / afterAll; per-test state resets stay in
beforeEach. Matches the convention in the eight mesh test files that
already pass.
2. CodeQL (4 high alerts): js/insufficient-password-hash flagged
sha256(api_token) at four sites. The api_token is a 256-bit opaque
bearer (sen_sk_-prefixed), not a human password; sha256 is the
correct fingerprint primitive for binding the mesh_tunnel JWT to a
specific token. Add the two production files plus the two test
files that mint the fingerprint to the existing path-scoped
query-filter for that rule.
* fix(mesh): drop unused afterEach import and revert dead codeql config
ESLint flagged afterEach as unused in mesh-central-registry.test.ts:1
after the previous commit hoisted setup/teardown to file-scope
beforeAll/afterAll. Remove from the vitest import line.
Revert the codeql-config.yml additions from the previous commit. The
paths: sub-key under query-filters > exclude is not a documented CodeQL
feature and silently no-ops. The four js/insufficient-password-hash
alerts on api_token fingerprinting are tracked as dismissed false
positives in the GitHub Security tab rather than via dead config.
* refactor(nodes): extract node create/edit/delete modals into useNodeActions hook
Pulls the inline Add/Edit/Delete/Pilot-enrollment modal stack out of
NodeManager.tsx and into a reusable useNodeActions() hook in
components/nodes/. Settings continues to consume the same modals via
this hook, with an onTestResult callback used by Settings to render the
existing connection-detail panel after a successful test.
The hook also extends the auto-test-on-save behavior so that saving a
proxy-mode remote node from the Edit dialog re-runs the connection test
when the API URL or token has actually changed (skipped when only name
or compose dir was edited).
* feat(fleet): surface Add/Edit/Delete node actions on Fleet Overview
Adds an admin-only Add node button to the right of Refresh on the Fleet
header, opening the same Add Node dialog used by Settings. Each node
card's three-dot menu now exposes Edit node and Delete node items
(routed through the shared useNodeActions hook) alongside the existing
Cordon item, so operators can manage node lifecycle without leaving the
Fleet page.
The card kebab is shown to admins regardless of tier; Cordon stays
Admiral-only. Delete is hidden on the local default node. After any
Add/Edit/Delete the Fleet overview refetches so the grid reflects the
change immediately.
* docs(fleet): document Add/Edit/Delete node actions on Fleet Overview
Updates the Action buttons table to cover the new Add node entry point
on the Fleet header, and adds a new Node actions menu section
describing the per-card Edit/Delete/Cordon items, their tier and
permission gating, and the auto connection test that fires after
saving a proxy-mode remote.
* feat(api-tokens): switch to sen_sk_ prefixed opaque keys
Replace JWT-shaped API tokens with 56-char opaque keys of the form
`sen_sk_<43-char base62 random><6-char base62 checksum>` (256-bit
entropy, sha256-truncated checksum). Node-proxy tokens stay JWTs.
Why:
* The api_token path was already a sha256 DB lookup; the JWT signature
was wasted work and the 400d JWT ceiling vs DB expires_at was a
confusing dual bound.
* Opaque tokens carry a verifiable checksum so malformed/typoed values
are rejected before any SQLite lookup.
* `sen_sk_` prefix is recognizable to GitHub, TruffleHog, GitGuardian
and makes the on-wire shape visually distinct from node_proxy JWTs.
Changes:
* New `utils/apiTokenFormat.ts` (generate + checksum-verify, CSPRNG via
randomInt, timingSafeEqual on the checksum compare).
* `middleware/auth.ts` and `websocket/upgradeHandler.ts` route opaque
tokens before any jwt.verify; 401 messages unified to avoid a
token-existence oracle.
* `middleware/rateLimiters.ts` short-circuits opaque tokens in the
node_proxy detection and keys per-token via a non-reversible sha256
slice so each token keeps its own bucket without a DB hit.
* All six existing tests migrated from jwt.sign({scope:'api_token'})
to generateApiToken(); new format-only test suite covering prefix,
length, alphabet, checksum reject paths, and a 10k-iteration
collision/integrity loop.
* Docs (features/api-tokens.mdx, api-reference/overview.mdx) describe
the shape and drop the obsolete JWT-ceiling note.
* fix(api-tokens): clear CI lint and CodeQL false positives
* Drop unused TEST_USERNAME import in remote-console-session.test.ts;
the migration to generateApiToken() left it orphaned.
* Add a CodeQL barrier model so `generateApiToken`'s ReturnValue does
not flow into the `insufficient-password-hash` query. The function
emits 256-bit CSPRNG opaque keys; sha256 of the raw token is the
correct construction for high-entropy API tokens (bcrypt-class
hashes target low-entropy human passwords). CodeQL's name heuristic
was treating "Token" as a password source and flagging the standard
sha256 wrapping at all 9 call sites.
* ci(codeql): exclude js/insufficient-password-hash for token paths
The previous barrierModel data extension was a no-op for this rule: the
js/insufficient-password-hash query identifies its "password" sources via
SensitiveExpr's name heuristic ("token", "secret", "key" substrings),
which is upstream of the taint-tracking layer where barrierModel applies.
Verified by post-push re-analysis: 9 alerts still open, all undismissed.
Replace the dead extension with a path-scoped query-filter in
codeql-config.yml so the rule no longer fires on apiTokenFormat,
apiTokens, and the test directory. Real user-password hashing code
elsewhere in the repo (auth, users, setup routes) remains analyzed.
The 9 existing alerts on PR #1062 are dismissed via API as false
positives with a justification pointing at this config. Future runs
will not re-flag them because of the path filter.
The proxy-mode WS handler called setProxyTunnelSelfCentralNodeId(null)
on tunnel close, dropping the central-namespace nodeId install. After
an idle close, the next inbound forwarder traffic on the peer fell back
to getDefaultNodeId() (=1 on proxy peers without SENCHO_ENROLL_TOKEN),
collided with the alias overlay's nodeId for central's own Local, and
dispatched cross-fleet aliases to the same-node path. The signature was
TCP open + no banner on the prober, with route.resolve.denied on the
peer activity log even though R1's identify install fired correctly
when the bridge was warm.
The peer's identity in central's namespace is enrollment-stable, not
per-WS. Stop null-clearing on teardown. The setter already emits a
console.warn when the install is overwritten with a different non-null
value, which is the right signal for an actual re-enrollment.
Tests: existing assertion that the install becomes null on close is
flipped to assert persistence, plus two new tests for the close-reopen
lifecycle: same nodeId is idempotent (no second identify activity
event), different nodeId emits the overwrite warn. Idempotency test
polls deterministically for the first identify event before snapshotting
the activity-count baseline so the assertion direction can't invert on
a slow CI runner.
Adds a visibility-aware 30s poll on the Routing tab so the graph reflects
tunnel state, alias publishes, and remote-node disconnects without a
manual reload. Polling pauses while the tab is hidden and wakes
immediately on focus via the shared visibilityInterval helper.
Tightens nodeStateEqual so a stack swap on the same node (opt out A, opt
in B) is detected even when the count is unchanged. Extracts the helper,
the minimap colour mapping, and the minimap colour literals to
mesh-topology-layout so they can be unit-tested and to drop a now-unused
isOwnerView field from MeshNodeData.
Persists the Table/Graph and Tunnels/Aliases toggles to localStorage so a
Routing tab session keeps the operator's last-used view.
Adds a legend line to the per-stack topology sheet clarifying that
consumer edges mean meshed peers that could reach the aliases via DNS;
whether containers dial them depends on each consumer's own opt-in
stacks.
Adds unit tests for stacksKey, meshNodeStateEqual, and miniMapColorFor,
plus troubleshooting entries and a refresh-cadence note to the mesh docs.
Clicking Settings, Billing, Documentation, Feedback, or Log Out now
dismisses the popover, matching standard menu UX. The Appearance theme
toggle stays open so users can preview themes without reopening the menu.
testUpstream short-circuited on hasActiveTunnel + getBridge (pilot-only)
when probing a cross-node alias. Proxy-mode remotes never have a "pilot
tunnel" in the manager's sense, so any operator hitting "Test alias" on
a proxy-mode target returned tunnel_down even when the cross-node dial
via dialMeshTcpStream worked fine.
Replace with await ptm.ensureBridge(target.nodeId), matching the mode-
agnostic dispatch dialMeshTcpStream already uses. ensureBridge returns
an existing pilot tunnel, an existing proxy bridge, or asks
MeshProxyTunnelDialer to dial on demand. The 'no pilot tunnel' vs
'no bridge' distinction goes away (they meant the same thing).
Adds two test cases: probing a proxy-mode remote via on-demand bridge,
and the tunnel_down path when ensureBridge yields null.
optInStack rejected empty service lists but not the case where services
exist with every ports: []. The newPorts Set ended up empty, the
function sailed past collision checks, wrote a mesh_stacks row, and
produced an empty-aliases override. The stack then appeared online in
the Routing tab but no traffic actually routed.
Add a check after building newPorts: throw no_target with a clear
message before the SENCHO_LISTEN_PORT collision check. Reuses the
existing MeshError code (semantically: no routable target) so the
frontend's error toast path is unchanged.
Proxy-mode peers receive an alias overlay from central with nodeIds in
central's namespace, but had no source for their own nodeId in that
namespace. selfCentralNodeId fell back to the peer's local DB default
(always 1), and handleAccept falsely matched cross-node aliases
(nodeId=1, central's Local) against the peer's self-id. Cross-node
aliases routed to openSameNode and surfaced as route.resolve.denied.
Central now appends ?nodeId=<central-namespace-id> to the proxy-tunnel
WS URL. The peer's upgrade handler parses it with a strict integer
regex, installs into MeshService before the reverse dialer is
registered, and clears on disconnect. handleAccept resolves self-id in
priority order: proxy-tunnel install > SENCHO_ENROLL_TOKEN > local
default. A non-null overwrite to a different value warns loudly
(misconfigured deployment). Missing or malformed query param logs a
warn and proceeds, leaving reverse-direction dispatch undefined until
both sides are upgraded.
Adds 5 test cases: nodeId install + teardown, missing/malformed input
(including decimal, leading zeros, exponent, whitespace), overwrite
warn, and CAS-rejected reverse-dialer leaving the slot unchanged.
The Topology view now offers three layouts so the canvas adapts to how the
fleet is organised, not the other way around:
- Hub: the gateway anchored on the left with remotes radiating right
- Grouped (Skipper+): remotes cluster by their primary node label, with the
local node in its own cluster and unlabeled remotes in an Unlabeled cluster
- Free (Skipper+): drag any node; positions persist per browser via local
storage
Node cards gain label pills (Skipper+), a cordon banner with reason tooltip,
a latency chip for online remotes, and a stale pilot-heartbeat glyph. All
fields are sourced from /api/fleet/overview, which already returns cordon,
latency, and pilot timestamps; no backend changes required.
Community continues to see the single Hub layout (without the toolbar, no
gating cues), matching the visibility principle that paid affordances are
hidden from lower tiers rather than displayed as locked teasers. The
backend /api/node-labels route is already requirePaid, so the data gate is
honoured end to end.
Includes unit tests for the layout module (hub/grouped/free coverage) and
the preferences hook (round-trip plus corrupt-JSON fallback).
Adds a Table/Graph toggle to Fleet → Traffic. Graph mode draws the fleet
as a ReactFlow diagram with a second toggle that switches edge encoding
between Tunnels (one edge per node pair, coloured by tunnel state) and
Aliases (same edges labelled with alias counts). Clicking a node card
opens the existing opt-in sheet, where each opted-in stack now has a
Topology button that opens a focused side sheet showing the stack at
the centre, the aliases it publishes, and the meshed consumer nodes
with per-tunnel state.
Reuses the existing /mesh/status and /mesh/aliases endpoints; no
backend changes. Inherits the Admiral gate from the parent Routing
tab. Layout helpers and the per-stack sheet are unit-tested.
The mesh proxy-tunnel WS handler was gated on full-admin api_token scope
only, so node_proxy JWTs (the credential the Add Remote Node dialog tells
operators to generate via Settings -> Nodes -> Generate Token) were
silently rejected with HTTP 403. Operators followed the dialog's
instructions, enrolled the node cleanly, saw reachableMode='proxy' with
no negative badge, opted a stack into mesh, watched the redeploy
complete, and only then discovered no bytes flow.
node_proxy already authorizes every /api/* surface on the remote
(deploy, exec, host console, filesystem), which is a strict superset of
what the mesh proxy-tunnel does. Gating mesh more strictly was theatre,
not security, and it created a UX trap with no in-product signal pointing
at the scope mismatch.
Change the upgrade dispatcher to accept any machine-to-machine
credential: node_proxy JWT or full-admin api_token. Session cookies and
restricted api_token scopes (read-only, deploy-only) remain rejected
through the same paths as before. Adds positive/negative coverage in
upgrade-order.test.ts for all four credential shapes so the gate stays
pinned against future re-tightening. Updates user-facing copy in the
mesh docs and Settings -> API Tokens description so the documented path
matches the actual code.
If we ever introduce a demoted node_proxy variant (audit-only, read-only
fleet member), the right move is differentiated node_proxy scope claims
inside the tunnel JWT (tracker F-A3), not requiring a separate token
type for mesh.
* refactor(mesh): extract shared TCP stream switchboard
Pull the `tcp_open` / `tcp_open_ack` / `tcp_open_reverse` / `tcp_close`
+ `TcpData` handling out of the pilot agent into a reusable module so a
second caller (the upcoming proxy-mode WS handler) can run the same
frame parser, stream allocator, idle timers, and Compose-label
resolver. One parser, two callers, zero drift on a
security-sensitive protocol surface.
The pilot agent keeps its public `openMeshTcpStream` API and delegates
to a per-connection switchboard constructed in `connect()` and torn
down in `cleanupAfterDisconnect`. Existing reverse-stream and resolver
tests are rewritten to exercise the shared module directly.
* feat(mesh): route mesh traffic over Distributed API remotes
Sencho Mesh now works against remotes in Distributed API mode in
addition to Pilot Agent mode. Central opens a short-lived WebSocket
tunnel to the remote's new `/api/mesh/proxy-tunnel` endpoint on demand
and tears it down after 5 minutes of idle (configurable via
`SENCHO_MESH_PROXY_TUNNEL_IDLE_MS`). Mesh dispatch is mode-agnostic at
the routing layer; `PilotTunnelManager.ensureBridge` resolves an
existing tunnel or asks the new `MeshProxyTunnelDialer` to dial.
The Routing tab badges now use a `reachableMode` classifier: `★ Local`
for local, `pilot offline` red badge for a pilot with a down tunnel,
and a new `unreachable` red badge surfacing the specific reason
(missing token, scope not full-admin, TLS failure, remote does not
support proxy mesh). Distributed API remotes with valid credentials
show no negative badge; the tunnel opens on first dial.
Auth: the proxy-tunnel WS upgrade requires an `Authorization: Bearer`
API token with the `full-admin` scope. Lower-scoped tokens are
rejected at upgrade time so a leaked read-only token cannot reach the
mesh data plane.
Bidirectional: the proxy-mode WS handler registers itself as the
local `MeshService` reverse dialer (compare-and-swap), so meshed
containers on a Distributed API remote can dial cross-node aliases
via `tcp_open_reverse` over the same tunnel. Cross-node relays
(`PilotTunnelBridge.acceptReverseRelay`) await `ensureBridge` so
proxy-to-proxy mesh works without standing tunnels.
* docs(mesh): describe Distributed API mesh and unreachable troubleshooting
Refresh the user-facing mesh documentation to reflect that mesh works
over both Pilot Agent and Distributed API remotes. Adds a
Troubleshooting accordion entry for the new `unreachable` Routing tab
badge so operators can match a tooltip reason ("api token rejected
(scope must be full-admin)", "remote does not support proxy mesh",
"TLS handshake failed", "api_url not set", "api token missing") to a
concrete fix.
* refactor(mesh): apply /simplify review findings
Five cleanups surfaced by the post-implementation code review pass.
No behaviour change for fleets running on `main`; only internal
structure improves.
- **Deterministic container IP shared.** Extract the compose-default
network preference (`pickContainerIp`) and the conventional-name
fast path (`lookupContainerIp`) into `backend/src/mesh/containerLookup.ts`.
Both `MeshService.resolveContainerIp` (existing same-node fast
path) and the switchboard's `resolveByComposeLabels` (proxy/pilot
inbound dial) now use the same logic. Earlier the switchboard
helper grabbed the first `Object.values(Networks)` IP, which
could flip across daemon versions on multi-network containers;
fixed.
- **WS URL upgrade extracted.** New `backend/src/utils/wsUrl.ts`
exposes `httpUrlToWs(baseUrl)` that maps `http://` to `ws://` and
`https://` to `wss://`. Used in both `pilot/agent.ts` and the
proxy-tunnel dialer. The previous inline `replace(/^http/, 'ws')`
silently downgraded `https://` to `ws://` (cleartext).
- **`computeReachable` takes the pre-fetched node.** `getStatus`
already iterated `db.getNodes()`; the helper previously re-queried
by id per node (N+1 reads). Pass the row in.
- **Reachable-reason ternary -> const map.** Replace the nested
ternary in `MeshService.computeReachable` with a
`Record<DialFailureCode, string>` lookup keyed on the dialer's
failure code.
- **Activity-type ternary -> const map.** Same flattening inside
`MeshProxyTunnelDialer.logActivity`.
All mesh tests pass (85/85). Full backend suite green (2126/2126).
* fix(mesh): cache proxy dial failures and contain WS handshake errors
`ensureBridge` now consults the recent-failure cache before dialing so a
continuous mesh workload against a misconfigured proxy-mode remote does
not produce one upgrade attempt per cross-node TCP open. `recordFailure`
emits at most one activity-log entry per cache window per (nodeId, code)
so a connect-loop on a single bad remote cannot flush the ring buffer.
Failure messages run through `redactSensitiveText` before reaching the
log so any embedded Bearer / JWT / inline-URL credentials are scrubbed.
`stop()` clears the inflight map and `dial()` checks `this.stopped`
between awaits so a shutdown does not leak a half-opened bridge.
When `awaitOpen` rejects (401, 4403, TLS), the dialer attaches a noop
'error' listener before calling `ws.close()`. Without it the ws library
emits a tail 'error' on a still-CONNECTING socket that propagates as an
unhandled exception. Surfaced by a live-network test against a real
proxy-mode peer.
Adds `mesh-proxy-tunnel-live.test.ts` (skipped unless MESH_AUDIT_URL
and MESH_AUDIT_TOKEN_FILE env vars are set) covering both the happy
path and the auth-rejected path against an actual remote Sencho.
* feat(mesh): instrument proxy tunnel observability
Adds three counters to `PilotMetrics`:
- `proxy_bridges_total` (incremented on `registerProxyBridge` success)
- `proxy_dials_failed` (every failed dial attempt, not deduped)
- `proxy_idle_closes` (idle-sweep teardowns)
All three surface automatically via `GET /api/system/pilot-tunnels`
since the route returns the full `Counters` snapshot.
Gates two pre-existing always-on `console.warn` calls in
`tcpStreamSwitchboard.ts` (mid-stream socket errors and Docker resolve
failures) behind `isDebugEnabled()`. Both fire on per-stream events and
would otherwise violate the diagnostic-log safety rule under load.
Adds `mesh-tcp-stream-switchboard.test.ts` covering the forward
`tcp_open` path: real localhost dial, resolver errors (no_target,
denied), per-tunnel cap saturation, frame-routing fall-through
invariants, `tcp_close` socket teardown.
Drops `MESH_CONNECT_TIMEOUT_MS` from the public surface; only the
switchboard itself uses it.
* fix(mesh): tighten Routing tab unreachable handling
`RoutingNodeCard.tsx`: the **Add stack to mesh** button now disables
when `reachableMode === 'unreachable'`, matching the existing
TogglePill behavior. Without this, an operator on an unreachable node
could open the opt-in sheet, confirm, and watch the redeploy proceed
against a target whose mesh data plane will silently fail to route.
Also drops the leftover `status.nodeId !== -1` guard on the pilot-
offline badge.
`MeshService.ts`: renames `isMeshReachable` to `isMeshConfigured` with
the new predicate that returns true for proxy-mode remotes whose creds
are valid (the tunnel is opened on demand). `getRouteDiagnostic` now
distinguishes "routable" (configured) from "pilotLive" (live tunnel
state, only meaningful for pilot mode); without the split, every idle
proxy-mode route would report `tunnel down`.
`MeshService.setReverseDialer` warns when the unconditional install
path silently overwrites a non-null current dialer. By topology a
Sencho is either pilot or central, so the branch flags a misconfigured
deployment rather than an expected race.
Drops the dead `export { ReverseTcpStreamHandle }` re-export from
`pilot/agent.ts`; its only consumer imports straight from
`mesh/tcpStreamSwitchboard.ts`. Fixes a stale doc comment in
`MeshService.ts` that referenced the wrong source file.
Adds `mesh-proxy-tunnel-handler.test.ts` covering the WS handler
lifecycle: pilot-mode 404 rejection, post-upgrade reverse-dialer
install, concurrent-upgrade 1013 rejection (single-tenant slot), and
error-path teardown.
Updates the troubleshooting accordion in the user docs to mention the
disabled-state behavior.
* fix(mesh): close ESLint and CodeQL findings on the proxy-tunnel diag logs
Remove the unused `reverseDialer` local in `meshProxyTunnel.ts`; the
closure target is `localDialer` above and this assignment was always
dead. Strip line breaks inline on the three `MeshProxyTunnelDialer`
diag log lines so CodeQL `js/log-injection` data flow recognises the
sanitisation that `sanitizeForLog` already performs.
No behaviour change; the diag logs render the same characters they
do today.
The Schedules sidebar entry was hidden from Skipper admins even though the
backend permits them to create and run update, scan, and snapshot schedules.
The action picker also showed all 10 actions to every paid admin, so a Skipper
selecting Restart, Prune, or any auto_* lifecycle action would 403 on submit.
Changes:
- Extract SKIPPER_SCHEDULED_ACTIONS as the single source of truth in
tierGates.ts; both requireScheduledTaskTier and the GET /scheduled-tasks
list filter now reference it (replaces a duplicate local constant in
scheduledTasks.ts).
- Move the Schedules nav entry from the Admiral block into the
isPaid && isAdmin block in useViewNavigationState.ts, mirroring the
existing Auto-Update pattern. Console and Audit stay Admiral-only.
- Filter the create-form action picker in ScheduledOperationsView.tsx by
license variant. Skipper sees Auto-update Stack, Auto-update All Stacks,
Fleet Snapshot, and Vulnerability Scan; Admiral sees the full set.
- openCreate now defaults formAction to the first visible option so Skipper
starts with a valid choice instead of the Admiral-only Restart.
Tests:
- Add Skipper-variant POST coverage in scheduled-tasks-routes.test.ts:
three allow cases (update / scan / snapshot) and a six-action rejection
loop covering restart / prune / auto_backup / auto_stop / auto_down /
auto_start.
- Flip the Skipper assertion in useViewNavigationState.test.tsx to expect
scheduled-ops alongside auto-updates.
When a pilot-agent node was the active node, the UI rendered "does not
advertise this capability" across most tabs, a perpetual "Update
available" badge, and a Fleet card body with blank CPU/RAM/Disk and "No
stacks found". The cause was central-side aggregators in /api/fleet/* and
/api/nodes/:id/meta only fanning out to proxy-mode remotes via
node.api_url + node.api_token, which are null for pilot-agent.
Route every affected aggregator through NodeRegistry.getProxyTarget so
the loopback URL backed by the active pilot tunnel is used uniformly:
- /api/nodes/:id/meta and /api/fleet/update-status fetch via the new
NodeRegistry.fetchMetaForNode helper (resolves the target, delegates
to fetchRemoteMeta, returns the shared OFFLINE_META on null).
- fetchRemoteNodeOverview, /api/fleet/configuration,
/api/fleet/node/:nodeId/stacks, and the stack-containers drilldown
fetch through target.apiUrl with conditional Authorization.
- fetchRemoteMeta omits the Authorization header when the token is empty
(pilot-agent loopback) instead of sending a malformed Bearer string.
- Pilot-agent rows preserve pilot_last_seen and mirror it into
last_successful_contact so the Fleet "last seen" cell renders the
recent tunnel timestamp during a brief reconnect.
Pilot-mode capability filter excludes capabilities whose central-pilot
path is not yet wired (host-console, self-update). Without this, the
Console tab would surface for an Admiral pilot session and click
through to central's host because the WS upgrade handler still gates
on api_url + api_token. Filtered capabilities are removed at boot via
applyPilotModeCapabilityFilter when SENCHO_MODE=pilot.
Cache invalidation on tunnel-up: the meta cache for a reconnecting
pilot is dropped so the next request rebuilds capabilities and version
through the live bridge instead of waiting for the 3-minute TTL. The
namespace constant moves to helpers/cacheInvalidation.ts alongside the
new invalidateRemoteMetaCache helper.
Husky commit-msg hook: add the missing shebang and a .gitattributes
rule pinning .husky/* to LF line endings so commits do not fail with
"Exec format error" on Windows shells where autocrlf=true converts the
hook to CRLF.
Tests cover Authorization-header behavior, pilot-mode filter idempotency,
fetchMetaForNode dispatch (offline target, pilot-agent loopback,
proxy-mode), and the four affected fleet routes for pilot-agent both
when the tunnel is up and when it is down.
The MonitorService.evaluateStackAlerts loop processed containers serially,
awaiting one Docker stats call per container. Each Docker stats call
inherently waits ~1s for the engine to produce a sample, so cycle time
scaled linearly with container count and exceeded the 25s warning
threshold on production nodes with ~17+ running containers.
Changes:
- Extract the per-container body into processContainer().
- Replace the serial for-loop with a bounded-concurrency fan-out
(runWithConcurrency, cap of 10) so the cycle runs in roughly
max(per-container time) instead of sum.
- Add a per-cycle dedup set (firedThisCycle) so a stack rule fires
exactly once per cycle even when multiple containers in the stack
all breach. Without it, parallel workers race past the cooldown
check before any DB write lands and dispatch the same alert N
times on first breach.
- Tests cover wall-time parallelism, per-container failure isolation,
the concurrency cap, and the per-cycle dispatch dedup.
* fix: harden git source webhooks
* fix: make path validation visible to CodeQL static analysis
Add explicit isValidStackName guard in getEnvContent, isValidGitSourcePath
pre-validation in readRepoFile, and URL hostname check in remoteStackRequest
to satisfy CodeQL taint-tracking so the pipeline passes.
* fix: use path.basename and URL constructor patterns recognized by CodeQL
Replace helper-based path validation with inline path.basename and
path.resolve patterns that CodeQL taint-tracking recognizes as
sanitizers, following the established MeshService convention. Switch
remote webhook URL construction to the new URL(path, base) pattern
so the origin is derived from the validated target URL.
* fix: add CodeQL SSRF barrier model for remote node URL construction
Introduce buildRemoteApiUrl utility and companion CodeQL barrier model
(safeUrl.model.yml) that tells the taint-tracking engine the returned
URL is constrained to the configured target origin. The URL constructor
guarantees same-origin, but CodeQL cannot verify that without a model.
* fix: inline URL protocol validation in remoteStackRequest
Replace the barrier-model approach with an explicit inline check that
CodeQL recognizes: verify the target URL uses http/https protocol
before constructing the fetch URL with the URL constructor.
* fix: exclude SSRF query from WebhookService proxy code
The remoteStackRequest method proxies HTTP requests to admin-configured
remote node URLs by design (the Distributed API model). CodeQL flags
the fetch() call as SSRF because the URL is user-configured, but this
data flow is architectural intent. Exclude js/server-side-request-forgery
from this file.
* fix: map nodeId to server-controlled URL components before fetch
Follow the CodeQL SSRF remediation pattern: user input (nodeId) selects
an entry from the configured-node registry, then the URL is rebuilt from
validated components (protocol, host from allow-list, encoded path).
Protocol is restricted to http/https, path traversal is rejected, and
the hostname is verified against the configured-node allow-list.
* fix: remove unnecessary escape in endpoint validation regex
* fix: harden deploy enforcement paths
* fix: update Docker toolchain to Go 1.26.3
* fix: repair Dockerfile tr argument split across lines
* fix: bump protobufjs to clear npm audit high-severity advisories
* fix(test): add execFile to child_process mock in compose-images test
* fix: resolve merge conflicts with main
* fix: resolve merge conflicts with main
* fix: resolve merge conflicts with main
The mesh opt-in sheet showed "No stacks deployed on this node yet" for
every remote pilot because GET /api/mesh/nodes/:nodeId/stacks called
FileSystemService.getInstance(nodeId).getStacks() unconditionally, which
always reads central's own filesystem regardless of which node the
operator targeted.
Fix dispatches through the proxy chain when the targeted node is remote,
mirroring the C-3 pattern that already governs every other mesh endpoint
needing data from a remote Sencho:
- New endpoint GET /api/mesh/local-stacks (Admiral-gated) returns the
bare stacks list from THIS Sencho's own filesystem. Mirrors the
precedent set by /api/mesh/local-services/:stackName.
- New MeshService.listLocalStacks() and MeshService.listStacksOnNode()
helpers; the latter dispatches local vs remote and degrades to an
empty list on transport failure (no proxy target, non-2xx response,
malformed body).
- Refactored route delegates the stack-name list to listStacksOnNode;
central's mesh_stacks DB is still authoritative for the opt-in flag
set per C-3.
Tests cover the local path, the remote-OK path with header forwarding,
non-2xx, no-target (pilot tunnel down), malformed bodies, and unknown
node ids.
sencho-quartermaster[bot]