mirror of
https://github.com/Studio-Saelix/sencho.git
synced 2026-08-27 18:57:09 +00:00
feat(mesh): route mesh traffic over Distributed API remotes (#1048)
* refactor(mesh): extract shared TCP stream switchboard
Pull the `tcp_open` / `tcp_open_ack` / `tcp_open_reverse` / `tcp_close`
+ `TcpData` handling out of the pilot agent into a reusable module so a
second caller (the upcoming proxy-mode WS handler) can run the same
frame parser, stream allocator, idle timers, and Compose-label
resolver. One parser, two callers, zero drift on a
security-sensitive protocol surface.
The pilot agent keeps its public `openMeshTcpStream` API and delegates
to a per-connection switchboard constructed in `connect()` and torn
down in `cleanupAfterDisconnect`. Existing reverse-stream and resolver
tests are rewritten to exercise the shared module directly.
* feat(mesh): route mesh traffic over Distributed API remotes
Sencho Mesh now works against remotes in Distributed API mode in
addition to Pilot Agent mode. Central opens a short-lived WebSocket
tunnel to the remote's new `/api/mesh/proxy-tunnel` endpoint on demand
and tears it down after 5 minutes of idle (configurable via
`SENCHO_MESH_PROXY_TUNNEL_IDLE_MS`). Mesh dispatch is mode-agnostic at
the routing layer; `PilotTunnelManager.ensureBridge` resolves an
existing tunnel or asks the new `MeshProxyTunnelDialer` to dial.
The Routing tab badges now use a `reachableMode` classifier: `★ Local`
for local, `pilot offline` red badge for a pilot with a down tunnel,
and a new `unreachable` red badge surfacing the specific reason
(missing token, scope not full-admin, TLS failure, remote does not
support proxy mesh). Distributed API remotes with valid credentials
show no negative badge; the tunnel opens on first dial.
Auth: the proxy-tunnel WS upgrade requires an `Authorization: Bearer`
API token with the `full-admin` scope. Lower-scoped tokens are
rejected at upgrade time so a leaked read-only token cannot reach the
mesh data plane.
Bidirectional: the proxy-mode WS handler registers itself as the
local `MeshService` reverse dialer (compare-and-swap), so meshed
containers on a Distributed API remote can dial cross-node aliases
via `tcp_open_reverse` over the same tunnel. Cross-node relays
(`PilotTunnelBridge.acceptReverseRelay`) await `ensureBridge` so
proxy-to-proxy mesh works without standing tunnels.
* docs(mesh): describe Distributed API mesh and unreachable troubleshooting
Refresh the user-facing mesh documentation to reflect that mesh works
over both Pilot Agent and Distributed API remotes. Adds a
Troubleshooting accordion entry for the new `unreachable` Routing tab
badge so operators can match a tooltip reason ("api token rejected
(scope must be full-admin)", "remote does not support proxy mesh",
"TLS handshake failed", "api_url not set", "api token missing") to a
concrete fix.
* refactor(mesh): apply /simplify review findings
Five cleanups surfaced by the post-implementation code review pass.
No behaviour change for fleets running on `main`; only internal
structure improves.
- **Deterministic container IP shared.** Extract the compose-default
network preference (`pickContainerIp`) and the conventional-name
fast path (`lookupContainerIp`) into `backend/src/mesh/containerLookup.ts`.
Both `MeshService.resolveContainerIp` (existing same-node fast
path) and the switchboard's `resolveByComposeLabels` (proxy/pilot
inbound dial) now use the same logic. Earlier the switchboard
helper grabbed the first `Object.values(Networks)` IP, which
could flip across daemon versions on multi-network containers;
fixed.
- **WS URL upgrade extracted.** New `backend/src/utils/wsUrl.ts`
exposes `httpUrlToWs(baseUrl)` that maps `http://` to `ws://` and
`https://` to `wss://`. Used in both `pilot/agent.ts` and the
proxy-tunnel dialer. The previous inline `replace(/^http/, 'ws')`
silently downgraded `https://` to `ws://` (cleartext).
- **`computeReachable` takes the pre-fetched node.** `getStatus`
already iterated `db.getNodes()`; the helper previously re-queried
by id per node (N+1 reads). Pass the row in.
- **Reachable-reason ternary -> const map.** Replace the nested
ternary in `MeshService.computeReachable` with a
`Record<DialFailureCode, string>` lookup keyed on the dialer's
failure code.
- **Activity-type ternary -> const map.** Same flattening inside
`MeshProxyTunnelDialer.logActivity`.
All mesh tests pass (85/85). Full backend suite green (2126/2126).
* fix(mesh): cache proxy dial failures and contain WS handshake errors
`ensureBridge` now consults the recent-failure cache before dialing so a
continuous mesh workload against a misconfigured proxy-mode remote does
not produce one upgrade attempt per cross-node TCP open. `recordFailure`
emits at most one activity-log entry per cache window per (nodeId, code)
so a connect-loop on a single bad remote cannot flush the ring buffer.
Failure messages run through `redactSensitiveText` before reaching the
log so any embedded Bearer / JWT / inline-URL credentials are scrubbed.
`stop()` clears the inflight map and `dial()` checks `this.stopped`
between awaits so a shutdown does not leak a half-opened bridge.
When `awaitOpen` rejects (401, 4403, TLS), the dialer attaches a noop
'error' listener before calling `ws.close()`. Without it the ws library
emits a tail 'error' on a still-CONNECTING socket that propagates as an
unhandled exception. Surfaced by a live-network test against a real
proxy-mode peer.
Adds `mesh-proxy-tunnel-live.test.ts` (skipped unless MESH_AUDIT_URL
and MESH_AUDIT_TOKEN_FILE env vars are set) covering both the happy
path and the auth-rejected path against an actual remote Sencho.
* feat(mesh): instrument proxy tunnel observability
Adds three counters to `PilotMetrics`:
- `proxy_bridges_total` (incremented on `registerProxyBridge` success)
- `proxy_dials_failed` (every failed dial attempt, not deduped)
- `proxy_idle_closes` (idle-sweep teardowns)
All three surface automatically via `GET /api/system/pilot-tunnels`
since the route returns the full `Counters` snapshot.
Gates two pre-existing always-on `console.warn` calls in
`tcpStreamSwitchboard.ts` (mid-stream socket errors and Docker resolve
failures) behind `isDebugEnabled()`. Both fire on per-stream events and
would otherwise violate the diagnostic-log safety rule under load.
Adds `mesh-tcp-stream-switchboard.test.ts` covering the forward
`tcp_open` path: real localhost dial, resolver errors (no_target,
denied), per-tunnel cap saturation, frame-routing fall-through
invariants, `tcp_close` socket teardown.
Drops `MESH_CONNECT_TIMEOUT_MS` from the public surface; only the
switchboard itself uses it.
* fix(mesh): tighten Routing tab unreachable handling
`RoutingNodeCard.tsx`: the **Add stack to mesh** button now disables
when `reachableMode === 'unreachable'`, matching the existing
TogglePill behavior. Without this, an operator on an unreachable node
could open the opt-in sheet, confirm, and watch the redeploy proceed
against a target whose mesh data plane will silently fail to route.
Also drops the leftover `status.nodeId !== -1` guard on the pilot-
offline badge.
`MeshService.ts`: renames `isMeshReachable` to `isMeshConfigured` with
the new predicate that returns true for proxy-mode remotes whose creds
are valid (the tunnel is opened on demand). `getRouteDiagnostic` now
distinguishes "routable" (configured) from "pilotLive" (live tunnel
state, only meaningful for pilot mode); without the split, every idle
proxy-mode route would report `tunnel down`.
`MeshService.setReverseDialer` warns when the unconditional install
path silently overwrites a non-null current dialer. By topology a
Sencho is either pilot or central, so the branch flags a misconfigured
deployment rather than an expected race.
Drops the dead `export { ReverseTcpStreamHandle }` re-export from
`pilot/agent.ts`; its only consumer imports straight from
`mesh/tcpStreamSwitchboard.ts`. Fixes a stale doc comment in
`MeshService.ts` that referenced the wrong source file.
Adds `mesh-proxy-tunnel-handler.test.ts` covering the WS handler
lifecycle: pilot-mode 404 rejection, post-upgrade reverse-dialer
install, concurrent-upgrade 1013 rejection (single-tenant slot), and
error-path teardown.
Updates the troubleshooting accordion in the user docs to mention the
disabled-state behavior.
* fix(mesh): close ESLint and CodeQL findings on the proxy-tunnel diag logs
Remove the unused `reverseDialer` local in `meshProxyTunnel.ts`; the
closure target is `localDialer` above and this assignment was always
dead. Strip line breaks inline on the three `MeshProxyTunnelDialer`
diag log lines so CodeQL `js/log-injection` data flow recognises the
sanitisation that `sanitizeForLog` already performs.
No behaviour change; the diag logs render the same characters they
do today.
This commit is contained in:
Anso
@@ -0,0 +1,137 @@
|
||||
import type { IncomingMessage } from 'http';
|
||||
import type { Duplex } from 'stream';
|
||||
import { WebSocketServer, type WebSocket } from 'ws';
|
||||
import { MAX_FRAME_SIZE_BYTES, decodeBinaryFrame, decodeJsonFrame, wsDataToBuffer, wsDataToString } from '../pilot/protocol';
|
||||
import {
|
||||
TcpStreamSwitchboard,
|
||||
attachTcpStreamSwitchboard,
|
||||
resolveByComposeLabels,
|
||||
type ReverseTcpStreamHandle,
|
||||
} from '../mesh/tcpStreamSwitchboard';
|
||||
import { sanitizeForLog } from '../utils/safeLog';
|
||||
import { isDebugEnabled } from '../utils/debug';
|
||||
import { rejectUpgrade as reject } from './reject';
|
||||
|
||||
/**
|
||||
* Mesh proxy-tunnel ingress.
|
||||
*
|
||||
* The remote side of a Phase C proxy-mode mesh tunnel. Central dials
|
||||
* `WSS <api_url>/api/mesh/proxy-tunnel` using the long-lived `api_token`
|
||||
* as a Bearer credential; this handler upgrades the connection and wires
|
||||
* the shared `TcpStreamSwitchboard` to handle `tcp_open` / `tcp_open_ack`
|
||||
* / `tcp_open_reverse` / `tcp_close` and `TcpData` frames.
|
||||
*
|
||||
* Auth + scope gating happens in `upgradeHandler.ts` before this handler
|
||||
* runs (require `full-admin` api_token scope). The handler itself trusts
|
||||
* the upgrade; the WS credential is the only trust boundary.
|
||||
*
|
||||
* Bidirectional: when the tunnel opens, the handler registers itself as
|
||||
* the reverse-dialer on the local MeshService so meshed containers on
|
||||
* this Sencho can dial cross-node aliases via `tcp_open_reverse` over
|
||||
* the same WS. On disconnect the reverse-dialer registration is
|
||||
* cleared.
|
||||
*/
|
||||
const wss = new WebSocketServer({ noServer: true, maxPayload: MAX_FRAME_SIZE_BYTES });
|
||||
|
||||
interface SwitchboardReverseDialer {
|
||||
openMeshTcpStream(target: { nodeId: number; stack: string; service: string; port: number }): ReverseTcpStreamHandle | null;
|
||||
}
|
||||
|
||||
export async function handleMeshProxyTunnel(req: IncomingMessage, socket: Duplex, head: Buffer): Promise<void> {
|
||||
// Mesh service is only available on the central deployment (SENCHO_MODE
|
||||
// unset or 'central'). Pilot-mode Sencho receives mesh traffic via the
|
||||
// pilot tunnel and has no use for the proxy-mode WS path.
|
||||
if (process.env.SENCHO_MODE === 'pilot') {
|
||||
return reject(socket, 404, 'Not Found');
|
||||
}
|
||||
|
||||
await new Promise<void>((resolve) => {
|
||||
wss.handleUpgrade(req, socket as Parameters<typeof wss.handleUpgrade>[1], head, (ws) => {
|
||||
void attachSwitchboard(ws).finally(resolve);
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
async function attachSwitchboard(ws: WebSocket): Promise<void> {
|
||||
let switchboard: TcpStreamSwitchboard | null = null;
|
||||
let meshServiceCleanup: (() => void) | null = null;
|
||||
|
||||
try {
|
||||
switchboard = attachTcpStreamSwitchboard({
|
||||
ws,
|
||||
resolveTarget: resolveByComposeLabels,
|
||||
logLabel: 'MeshProxy',
|
||||
});
|
||||
|
||||
// Register a reverse dialer so this side's MeshForwarder can dial
|
||||
// cross-node aliases via `tcp_open_reverse` over the same WS. The
|
||||
// CAS swap refuses to overwrite a dialer that another caller
|
||||
// (a concurrent proxy-tunnel upgrade, or a pilot agent in a
|
||||
// misconfigured deployment) has already installed.
|
||||
const { MeshService } = await import('../services/MeshService');
|
||||
const meshService = MeshService.getInstance();
|
||||
const localSwitchboard = switchboard;
|
||||
const localDialer: SwitchboardReverseDialer = {
|
||||
openMeshTcpStream(target) {
|
||||
return localSwitchboard.openReverseStream(target);
|
||||
},
|
||||
};
|
||||
const installed = meshService.setReverseDialer(localDialer, null);
|
||||
if (!installed) {
|
||||
console.warn('[MeshProxy] reverse dialer already installed; rejecting concurrent tunnel');
|
||||
try { ws.close(1013, 'reverse dialer already installed'); } catch { /* ignore */ }
|
||||
switchboard.cleanup('reverse dialer already installed');
|
||||
switchboard = null;
|
||||
return;
|
||||
}
|
||||
meshServiceCleanup = () => {
|
||||
meshService.setReverseDialer(null, localDialer);
|
||||
};
|
||||
} catch (err) {
|
||||
if (isDebugEnabled()) {
|
||||
console.warn('[MeshProxy:diag] failed to attach switchboard:', sanitizeForLog((err as Error).message));
|
||||
}
|
||||
try { ws.close(1011, 'switchboard attach failed'); } catch { /* ignore */ }
|
||||
return;
|
||||
}
|
||||
|
||||
const onMessage = (data: unknown, isBinary: boolean): void => {
|
||||
if (!switchboard) return;
|
||||
try {
|
||||
if (isBinary) {
|
||||
const buf = wsDataToBuffer(data);
|
||||
if (!buf) return;
|
||||
switchboard.handleBinaryFrame(decodeBinaryFrame(buf));
|
||||
} else {
|
||||
const text = wsDataToString(data);
|
||||
if (text == null) return;
|
||||
switchboard.handleJsonFrame(decodeJsonFrame(text));
|
||||
}
|
||||
} catch (err) {
|
||||
if (isDebugEnabled()) {
|
||||
console.warn('[MeshProxy:diag] malformed frame:', sanitizeForLog((err as Error).message));
|
||||
}
|
||||
}
|
||||
};
|
||||
|
||||
const teardown = (): void => {
|
||||
ws.off('message', onMessage);
|
||||
if (switchboard) {
|
||||
switchboard.cleanup('mesh proxy-tunnel closed');
|
||||
switchboard = null;
|
||||
}
|
||||
if (meshServiceCleanup) {
|
||||
meshServiceCleanup();
|
||||
meshServiceCleanup = null;
|
||||
}
|
||||
};
|
||||
|
||||
ws.on('message', onMessage);
|
||||
ws.once('close', teardown);
|
||||
ws.once('error', (err) => {
|
||||
if (isDebugEnabled()) {
|
||||
console.warn('[MeshProxy:diag] ws error:', sanitizeForLog(err.message));
|
||||
}
|
||||
teardown();
|
||||
});
|
||||
}
|
||||
@@ -7,6 +7,7 @@ import { DatabaseService, type UserRole } from '../services/DatabaseService';
|
||||
import { NodeRegistry } from '../services/NodeRegistry';
|
||||
import { COOKIE_NAME } from '../helpers/constants';
|
||||
import { handlePilotTunnel } from './pilotTunnel';
|
||||
import { handleMeshProxyTunnel } from './meshProxyTunnel';
|
||||
import { handleNotificationsWs } from './notifications';
|
||||
import { handleRemoteForwarder } from './remoteForwarder';
|
||||
import { handleLogsWs } from './logs';
|
||||
@@ -32,11 +33,12 @@ function parseCookies(req: IncomingMessage): Record<string, string> {
|
||||
* 1. `/api/pilot/tunnel` -> handlePilotTunnel (own auth, own wss)
|
||||
* 2. shared cookie/Bearer auth + JWT verify (rejects unauthenticated)
|
||||
* 3. API token scope gate (read-only / deploy-only restricted to logs + notifications)
|
||||
* 4. `/ws/notifications` local -> handleNotificationsWs
|
||||
* 5. remote nodeId path -> handleRemoteForwarder
|
||||
* 6. `/api/stacks/:name/logs` -> handleLogsWs
|
||||
* 7. `/api/system/host-console` -> handleHostConsoleWs
|
||||
* 8. fallback -> handleGenericWs (`/ws` exec + stats)
|
||||
* 4. `/api/mesh/proxy-tunnel` -> handleMeshProxyTunnel (requires full-admin api_token scope)
|
||||
* 5. `/ws/notifications` local -> handleNotificationsWs
|
||||
* 6. remote nodeId path -> handleRemoteForwarder
|
||||
* 7. `/api/stacks/:name/logs` -> handleLogsWs
|
||||
* 8. `/api/system/host-console` -> handleHostConsoleWs
|
||||
* 9. fallback -> handleGenericWs (`/ws` exec + stats)
|
||||
*/
|
||||
export function attachUpgrade(
|
||||
server: http.Server,
|
||||
@@ -120,6 +122,18 @@ export function attachUpgrade(
|
||||
}
|
||||
}
|
||||
|
||||
// Mesh proxy-tunnel ingress: a sibling Sencho is dialing this node
|
||||
// to carry mesh TCP traffic. Require an api_token Bearer with the
|
||||
// full-admin scope; mesh manipulates traffic and must not be
|
||||
// reachable under a session cookie or a node_proxy JWT.
|
||||
if (pathname === '/api/mesh/proxy-tunnel') {
|
||||
if (wsApiTokenScope !== 'full-admin') {
|
||||
return reject(socket, 403, 'Forbidden');
|
||||
}
|
||||
await handleMeshProxyTunnel(req, socket, head);
|
||||
return;
|
||||
}
|
||||
|
||||
const nodeIdParam = parsedUrl.searchParams.get('nodeId');
|
||||
const nodeId = nodeIdParam ? parseInt(nodeIdParam, 10) : NodeRegistry.getInstance().getDefaultNodeId();
|
||||
const node = NodeRegistry.getInstance().getNode(nodeId);
|
||||
|
||||
Reference in New Issue
Block a user