gsadmin/sencho
1
0
Fork 0
mirror of https://github.com/Studio-Saelix/sencho.git synced 2026-08-13 04:06:59 +00:00
Code Issues Packages Projects Releases Wiki Activity

fix(mesh): route peer→central traffic over the existing forward WS (#1094)

Browse Source 
* fix(mesh): route peer→central traffic over the existing forward WS

The reverse mesh callback path (`/api/mesh/proxy-tunnel-from-peer`) needed
SENCHO_PRIMARY_URL on central plus a publicly reachable origin from the
peer's perspective. In a typical homelab where central sits behind NAT,
peer→central dispatch silently failed at the dialer's short-circuit and
the headline "call any service on any node by hostname" worked one way
only.

The forward WS at `/api/mesh/proxy-tunnel` is already bidirectional end
to end. Make the bridge a persistent control-plane primitive: dial every
mesh-enabled proxy peer at startup, reconcile every 60 s, never idle-close.
Peer→central traffic multiplexes over the same WS via `tcp_open_reverse`.

Removed:
- `meshProxyTunnelFromPeer.ts` WS handler and dispatch
- `MeshCentralRegistry`, `PeerToCentralMeshSessionDialer`
- `mesh_handshake` first-frame state machine in `meshProxyTunnel.ts`
- `maybeSendBootstrap`, `buildHandshakeFrame` in the dialer
- `mesh_proxy_callback_bootstrap` capability and `maybeWarnUnsetPrimaryUrl`
- `mesh_centrals` table (drop migration; greenfield, no users)
- `PilotTunnelManager.replaceOrRegisterProxyBridge` (dead after handler removal)
- twelve associated unit/integration tests plus the peer-recovery branch
  in `MeshService.openCrossNode`

Added:
- `MeshService.proactiveBridgeFanout` selects every mesh-enabled proxy
  peer (no longer gated on `mesh_stacks` rows)
- `startBridgeReconcileLoop` runs the fanout every 60 s (override via
  `SENCHO_MESH_RECONCILE_INTERVAL_MS`)
- `MeshProxyTunnelDialer` default idle TTL is now `0` and exposes
  `isDialing(nodeId)` for the status surface
- `MeshNodeStatus.reverseCallbackStatus` discriminator
  (`connected | connecting | unavailable | not_applicable`) surfaced via
  `/api/mesh/status` and rendered as a pill in the Routing tab
- `openCrossNode` error message distinguishes "no proxy target" from
  "waiting for central to dial the reverse bridge"
- New tests: `mesh-service-proxy-tunnel-reconcile`,
  `mesh-status-reverse-callback`, `mesh-proxy-tunnel-dialer-no-idle-close`

SENCHO_PRIMARY_URL is no longer required for any mesh function.

* fix(mesh): rewrite proxy-tunnel reconcile test contents

The previous commit renamed the file but the rewritten test bodies stayed
unstaged on top of the rename. This commit lands the actual rewrite: the
fanout assertion now requires every mesh-enabled proxy peer to be dialed,
not just those with `mesh_stacks` rows, and adds a reconcile-tick
repeated-call test.
This commit is contained in:
Anso
2026-05-17 22:00:31 -04:00
committed by GitHub
parent 54c07d4930
commit f6e42535c8
37 changed files with 414 additions and 3107 deletions
Download Patch File Download Diff File Expand all files Collapse all files
backend/src/websocket/meshProxyTunnel.ts
+5 -129
View File
@@ -11,44 +11,11 @@ import {
import { sanitizeForLog } from '../utils/safeLog';
import { isDebugEnabled } from '../utils/debug';
import { rejectUpgrade as reject } from './reject';
import { MeshCentralRegistry, type MeshCentralMaterial } from '../services/MeshCentralRegistry';
/**
* Bootstrap phase for the first-frame state machine.
*
* Central MAY send a `mesh_handshake` JSON frame as the FIRST text frame
* after upgrade. If it arrives we persist the callback material via
* MeshCentralRegistry and transition to `consumed`. Any frame other than
* the handshake (or no frame at all) moves us to `past`, after which a
* later handshake is a protocol error.
*/
type BootstrapPhase = 'awaiting' | 'consumed' | 'past';
interface MeshHandshakeFrame {
t: 'mesh_handshake';
v: number;
peerNodeId: number;
centralInstanceId: string;
centralApiUrl: string;
meshTunnelJwt: string;
jwtExpiresAt: number;
}
function isMeshHandshakeFrame(parsed: unknown): parsed is MeshHandshakeFrame {
return typeof parsed === 'object' && parsed !== null
&& (parsed as { t?: unknown }).t === 'mesh_handshake'
&& typeof (parsed as { v?: unknown }).v === 'number'
&& typeof (parsed as { peerNodeId?: unknown }).peerNodeId === 'number'
&& typeof (parsed as { centralInstanceId?: unknown }).centralInstanceId === 'string'
&& typeof (parsed as { centralApiUrl?: unknown }).centralApiUrl === 'string'
&& typeof (parsed as { meshTunnelJwt?: unknown }).meshTunnelJwt === 'string'
&& typeof (parsed as { jwtExpiresAt?: unknown }).jwtExpiresAt === 'number';
}
/**
* Mesh proxy-tunnel ingress.
*
* The remote side of a Phase C proxy-mode mesh tunnel. Central dials
* The peer side of a proxy-mode mesh tunnel. Central dials
* `WSS <api_url>/api/mesh/proxy-tunnel` using the fleet credential as a
* Bearer header (either the `node_proxy` JWT generated by the remote's
* Settings → Nodes → Generate Token flow, or a `full-admin` api_token
@@ -65,21 +32,12 @@ function isMeshHandshakeFrame(parsed: unknown): parsed is MeshHandshakeFrame {
* the reverse-dialer on the local MeshService so meshed containers on
* this Sencho can dial cross-node aliases via `tcp_open_reverse` over
* the same WS. On disconnect the reverse-dialer registration is
* cleared.
* cleared. Central maintains the bridge persistently (no idle close;
* reconciled on a timer), so peer→central traffic always has a live
* channel in steady state.
*/
const wss = new WebSocketServer({ noServer: true, maxPayload: MAX_FRAME_SIZE_BYTES });
/**
* Upper bound on the pre-CAS first-frame wait inside attachSwitchboard.
* Sized to cover localhost-loopback RTT plus a slim margin for the
* central-side ms-scale gap between `ws.send(mesh_handshake)` and the
* registerProxyBridge throw + ws.close that follows on a refused dial.
* On a dial that carries no bootstrap, this is pure added latency
* before the reverse-dialer install; 30ms is well below operator
* perception and well above realistic localhost RTT.
*/
const FIRST_FRAME_WAIT_MS = 30;
interface SwitchboardReverseDialer {
openMeshTcpStream(target: { nodeId: number; stack: string; service: string; port: number }): ReverseTcpStreamHandle | null;
}
@@ -126,74 +84,18 @@ export async function handleMeshProxyTunnel(req: IncomingMessage, socket: Duplex
async function attachSwitchboard(ws: WebSocket, peerNodeId: number | null): Promise<void> {
let switchboard: TcpStreamSwitchboard | null = null;
let meshServiceCleanup: (() => void) | null = null;
let phase: BootstrapPhase = 'awaiting';
const onMessage = (data: unknown, isBinary: boolean): void => {
if (!switchboard) return;
try {
if (isBinary) {
if (phase === 'awaiting') phase = 'past';
const buf = wsDataToBuffer(data);
if (!buf) return;
switchboard.handleBinaryFrame(decodeBinaryFrame(buf));
return;
}
const text = wsDataToString(data);
if (text == null) {
if (phase === 'awaiting') phase = 'past';
return;
}
let parsedForBootstrap: unknown = null;
let parseSucceeded = false;
try {
parsedForBootstrap = JSON.parse(text);
parseSucceeded = true;
} catch {
// Not valid JSON; defer to decodeJsonFrame's stricter error path below.
}
if (parseSucceeded && typeof parsedForBootstrap === 'object'
&& parsedForBootstrap !== null
&& (parsedForBootstrap as { t?: unknown }).t === 'mesh_handshake') {
if (phase !== 'awaiting') {
ws.close(1008, 'mesh_handshake out of order');
return;
}
if (!isMeshHandshakeFrame(parsedForBootstrap)) {
ws.close(1008, 'malformed mesh_handshake');
return;
}
const material: MeshCentralMaterial = {
centralInstanceId: parsedForBootstrap.centralInstanceId,
centralApiUrl: parsedForBootstrap.centralApiUrl.replace(/\/+$/, ''),
callbackJwt: parsedForBootstrap.meshTunnelJwt,
jwtIssuedAt: Math.floor(Date.now() / 1000),
jwtExpiresAt: parsedForBootstrap.jwtExpiresAt,
};
try {
MeshCentralRegistry.getInstance().upsert(material);
phase = 'consumed';
const centralInstanceId = parsedForBootstrap.centralInstanceId;
const jwtExpiresAt = parsedForBootstrap.jwtExpiresAt;
void (async () => {
try {
const { MeshService: MeshSvc } = await import('../services/MeshService');
MeshSvc.getInstance().logActivity({
source: 'mesh', level: 'info',
type: 'mesh_handshake.received',
message: `mesh callback bootstrap received from central ${centralInstanceId}`,
details: { centralInstanceId, jwtExpiresAt },
});
} catch { /* best-effort log; bootstrap success path must not depend on it */ }
})();
} catch (err) {
console.warn(`[meshProxyTunnel] mesh_handshake persist failed: ${sanitizeForLog((err as Error).message)}`);
try { ws.close(1011, 'bootstrap persist failed'); } catch { /* ignore */ }
}
return;
}
if (phase === 'awaiting') phase = 'past';
if (text == null) return;
switchboard.handleJsonFrame(decodeJsonFrame(text));
} catch (err) {
if (isDebugEnabled()) {
@@ -221,11 +123,6 @@ async function attachSwitchboard(ws: WebSocket, peerNodeId: number | null): Prom
logLabel: 'MeshProxy',
});
// Wire listeners synchronously so any first frame from central
// reaches onMessage even when the reverse-dialer CAS swap below
// refuses this bridge. Bootstrap material identifies central for
// the next peer-initiated callback and must persist independent
// of whether this particular WS becomes the active bridge.
ws.on('message', onMessage);
ws.once('close', teardown);
ws.once('error', (err) => {
@@ -251,23 +148,7 @@ async function attachSwitchboard(ws: WebSocket, peerNodeId: number | null): Prom
const installed = meshService.setReverseDialer(localDialer, null);
if (!installed) {
console.warn('[MeshProxy] reverse dialer already installed; rejecting concurrent tunnel');
// Hold the WS open briefly so any in-flight first frame can
// land in onMessage before we send the close. Central
// optionally sends a mesh_handshake as the first frame on
// Trigger 1 / Trigger 2 dials; that bootstrap material must
// persist via MeshCentralRegistry regardless of whether this
// WS becomes the active bridge. The wait resolves as soon as
// any 'message' arrives (the listener attached above runs
// synchronously on the event and updates the registry); the
// timeout bounds the close-latency overhead for refused
// dials that carry no bootstrap.
await new Promise<void>((resolve) => {
const onFirst = (): void => { clearTimeout(t); resolve(); };
const t = setTimeout(() => { ws.off('message', onFirst); resolve(); }, FIRST_FRAME_WAIT_MS);
ws.once('message', onFirst);
});
try { ws.close(1013, 'reverse dialer already installed'); } catch { /* ignore */ }
// teardown fires on the 'close' event and runs switchboard.cleanup.
return;
}
// Install central's view of this peer's nodeId so handleAccept
@@ -276,11 +157,6 @@ async function attachSwitchboard(ws: WebSocket, peerNodeId: number | null): Prom
//
// The install persists across bridge lifecycles: the peer's identity
// in central's namespace is stable for the enrollment, not per-WS.
// Null-clearing on teardown caused dispatch to fall back to
// getDefaultNodeId() after idle close, which collides with central's
// own nodeId for Local and misdispatches cross-fleet traffic to the
// same-node path. A subsequent install with a different nodeId
// (e.g. re-enrollment) is detected by the setter's overwrite-warn.
if (peerNodeId !== null) {
meshService.setProxyTunnelSelfCentralNodeId(peerNodeId);
}
backend/src/websocket/meshProxyTunnelFromPeer.ts
-142
View File
@@ -1,142 +0,0 @@
/**
* Central-side ingress for peer-initiated proxy-mode mesh tunnels.
*
* After the bootstrap exchange in `meshProxyTunnel.ts` (Task 7/8), a peer
* with the central's `mesh_tunnel` JWT can dial *back* to central at this
* endpoint. The validation chain enforces every claim the bootstrap mint
* produced and confirms the peer's stored fingerprint still matches its
* current api_token. Each failure path returns HTTP 401 with a stable
* machine-readable `reason` code in the JSON body, so the dialer side
* (PeerToCentralMeshSessionDialer, Task 10) can act on the reason without
* scraping prose.
*
* Auth runs MANUALLY in this handler. Express middleware does not run on
* WebSocket upgrades, and the credential here is a Bearer JWT minted by
* the central itself, not a user session.
*/
import type { IncomingMessage } from 'http';
import type { Duplex } from 'stream';
import jwt, { type Algorithm, type JwtPayload } from 'jsonwebtoken';
import { createHash } from 'crypto';
import { WebSocketServer } from 'ws';
import { DatabaseService } from '../services/DatabaseService';
import { PilotTunnelBridge } from '../services/PilotTunnelBridge';
import { PilotTunnelManager } from '../services/PilotTunnelManager';
import { PilotMetrics } from '../services/PilotMetrics';
import { sanitizeForLog } from '../utils/safeLog';
const wss = new WebSocketServer({ noServer: true });
const CLOCK_SKEW_SEC = 60;
type RejectReason =
| 'algorithm_mismatch' | 'signature_invalid'
| 'scope_mismatch' | 'audience_mismatch' | 'instance_mismatch'
| 'stale' | 'clock_skew' | 'node_deleted' | 'mode_mismatch'
| 'token_fingerprint_mismatch' | 'malformed';
function rejectUpgrade(socket: Duplex, reason: RejectReason): void {
const body = JSON.stringify({ reason });
const headers = [
'HTTP/1.1 401 Unauthorized',
'Content-Type: application/json',
`Content-Length: ${Buffer.byteLength(body)}`,
'Connection: close',
'',
body,
].join('\r\n');
try { socket.write(headers); } catch { /* socket already gone */ }
try { socket.destroy(); } catch { /* socket already gone */ }
}
function extractBearer(req: IncomingMessage): string | null {
const h = req.headers['authorization'];
if (typeof h !== 'string' || !h.startsWith('Bearer ')) return null;
return h.slice('Bearer '.length).trim() || null;
}
export function handleMeshProxyTunnelFromPeerUpgrade(
req: IncomingMessage,
socket: Duplex,
head: Buffer,
): void {
const token = extractBearer(req);
if (!token) { rejectUpgrade(socket, 'malformed'); return; }
let header: { alg?: string; kid?: string } = {};
try {
const decoded = jwt.decode(token, { complete: true });
header = (decoded?.header ?? {}) as typeof header;
} catch { rejectUpgrade(socket, 'malformed'); return; }
if (header.alg !== 'HS256') { rejectUpgrade(socket, 'algorithm_mismatch'); return; }
const settings = DatabaseService.getInstance().getGlobalSettings();
const secret = settings.auth_jwt_secret;
if (!secret) { rejectUpgrade(socket, 'malformed'); return; }
let payload: JwtPayload;
try {
payload = jwt.verify(token, secret, { algorithms: ['HS256' as Algorithm] }) as JwtPayload;
} catch { rejectUpgrade(socket, 'signature_invalid'); return; }
if (payload.scope !== 'mesh_tunnel') { rejectUpgrade(socket, 'scope_mismatch'); return; }
// SENCHO_PRIMARY_URL must be set on central for peer-initiated dial-back
// to operate. The Task 8 bootstrap-mint side already skips when it's
// unset; this is the matching fail-safe on the verify side. Reject all
// peer dials with audience_mismatch when central has no canonical
// origin to validate against.
const canonicalOrigin = (process.env.SENCHO_PRIMARY_URL ?? '').replace(/\/+$/, '');
if (!canonicalOrigin) { rejectUpgrade(socket, 'audience_mismatch'); return; }
if (payload.aud !== canonicalOrigin) { rejectUpgrade(socket, 'audience_mismatch'); return; }
// instance_id is operational state, not user-defined config, so it
// lives in system_state rather than global_settings.
const instanceId = DatabaseService.getInstance().getSystemState('instance_id');
if (payload.iss !== instanceId) { rejectUpgrade(socket, 'instance_mismatch'); return; }
const nowSec = Math.floor(Date.now() / 1000);
if (typeof payload.exp !== 'number' || payload.exp <= nowSec) {
rejectUpgrade(socket, 'stale'); return;
}
if (typeof payload.iat !== 'number' || payload.iat > nowSec + CLOCK_SKEW_SEC) {
rejectUpgrade(socket, 'clock_skew'); return;
}
const peerNodeId = Number(payload.sub);
if (!Number.isInteger(peerNodeId) || peerNodeId <= 0) {
rejectUpgrade(socket, 'malformed'); return;
}
const node = DatabaseService.getInstance().getNode(peerNodeId);
if (!node) { rejectUpgrade(socket, 'node_deleted'); return; }
if (node.type !== 'remote' || node.mode !== 'proxy') {
rejectUpgrade(socket, 'mode_mismatch'); return;
}
if (!node.api_token) { rejectUpgrade(socket, 'token_fingerprint_mismatch'); return; }
const expectedFp = createHash('sha256').update(node.api_token).digest('hex').slice(0, 16);
const claimedFp = payload['peer_token_fp'];
if (typeof claimedFp !== 'string' || claimedFp !== expectedFp) {
rejectUpgrade(socket, 'token_fingerprint_mismatch'); return;
}
wss.handleUpgrade(req, socket, head, (ws) => {
try {
const bridge = new PilotTunnelBridge(peerNodeId, ws);
bridge.start().then(() => {
try {
PilotTunnelManager.getInstance().replaceOrRegisterProxyBridge(peerNodeId, bridge);
PilotMetrics.increment('proxy_bridges_peer_initiated_total');
} catch (err) {
try { bridge.close(1013, 'manager rejected'); } catch { /* ignore */ }
console.warn(`[meshProxyTunnelFromPeer] register failed: ${sanitizeForLog((err as Error).message)}`);
}
}).catch((err) => {
try { bridge.close(1011, 'bridge start failed'); } catch { /* ignore */ }
console.warn(`[meshProxyTunnelFromPeer] bridge start failed: ${sanitizeForLog((err as Error).message)}`);
});
} catch (err) {
console.warn(`[meshProxyTunnelFromPeer] bridge construct failed: ${sanitizeForLog((err as Error).message)}`);
try { ws.close(1011, 'bridge init failed'); } catch { /* ignore */ }
}
});
}
backend/src/websocket/upgradeHandler.ts
+12 -20
View File
@@ -8,7 +8,6 @@ import { NodeRegistry } from '../services/NodeRegistry';
import { COOKIE_NAME } from '../helpers/constants';
import { handlePilotTunnel } from './pilotTunnel';
import { handleMeshProxyTunnel } from './meshProxyTunnel';
import { handleMeshProxyTunnelFromPeerUpgrade } from './meshProxyTunnelFromPeer';
import { handleNotificationsWs } from './notifications';
import { handleRemoteForwarder } from './remoteForwarder';
import { handleLogsWs } from './logs';
@@ -33,15 +32,14 @@ function parseCookies(req: IncomingMessage): Record<string, string> {
*
* Dispatch order (first match wins):
* 1. `/api/pilot/tunnel` -> handlePilotTunnel (own auth, own wss)
* 2. `/api/mesh/proxy-tunnel-from-peer` -> handleMeshProxyTunnelFromPeerUpgrade (own JWT chain, central-side dial-back)
* 3. shared cookie/Bearer auth + JWT verify (rejects unauthenticated)
* 4. API token scope gate (read-only / deploy-only restricted to logs + notifications)
* 5. `/api/mesh/proxy-tunnel` -> handleMeshProxyTunnel (machine-to-machine: node_proxy or full-admin api_token)
* 6. `/ws/notifications` local -> handleNotificationsWs
* 7. remote nodeId path -> handleRemoteForwarder
* 8. `/api/stacks/:name/logs` -> handleLogsWs
* 9. `/api/system/host-console` -> handleHostConsoleWs
* 10. fallback -> handleGenericWs (`/ws` exec + stats)
* 2. shared cookie/Bearer auth + JWT verify (rejects unauthenticated)
* 3. API token scope gate (read-only / deploy-only restricted to logs + notifications)
* 4. `/api/mesh/proxy-tunnel` -> handleMeshProxyTunnel (machine-to-machine: node_proxy or full-admin api_token; bidirectional bridge for both forward and reverse mesh traffic)
* 5. `/ws/notifications` local -> handleNotificationsWs
* 6. remote nodeId path -> handleRemoteForwarder
* 7. `/api/stacks/:name/logs` -> handleLogsWs
* 8. `/api/system/host-console` -> handleHostConsoleWs
* 9. fallback -> handleGenericWs (`/ws` exec + stats)
*/
export function attachUpgrade(
server: http.Server,
@@ -52,22 +50,16 @@ export function attachUpgrade(
attachGenericConnectionHandlers(wss);
server.on('upgrade', async (req, socket, head) => {
// Pilot-agent tunnel ingress: machine credentials, no cookies.
// Mesh proxy-tunnel-from-peer ingress: peer-initiated dial-back over
// a `mesh_tunnel`-scoped JWT (minted earlier during the bootstrap
// exchange). Both handlers run their own auth before the shared
// cookie/Bearer pipeline because their credentials are not user
// sessions and would fail the shared user-existence check.
// Pilot-agent tunnel ingress: machine credentials, no cookies. Runs its
// own auth before the shared cookie/Bearer pipeline because the
// credential is not a user session and would fail the shared
// user-existence check.
try {
const reqUrl = new URL(req.url || '/', `http://${req.headers.host || 'localhost'}`);
if (reqUrl.pathname === '/api/pilot/tunnel') {
await handlePilotTunnel(req, socket, head, pilotTunnelWss);
return;
}
if (reqUrl.pathname === '/api/mesh/proxy-tunnel-from-peer') {
handleMeshProxyTunnelFromPeerUpgrade(req, socket, head);
return;
}
} catch {
// URL parse error falls through and will be rejected below.
}