mirror of
https://github.com/Studio-Saelix/sencho.git
synced 2026-08-18 22:36:19 +00:00
feat(mesh): symmetric WS dial for proxy-mode mesh peers (#1066)
* chore(mesh): foundation for symmetric callback dial Adds the data-plane scaffolding that the symmetric callback dial fix builds on: - mesh_centrals table for peer-side bootstrap material - MeshCentralRegistry service (upsert/getActive/clear/markUsed/markRejected) - PilotTunnelManager kind discriminator and replaceOrRegisterProxyBridge - mesh_proxy_callback_bootstrap capability registration - MeshProxyTunnelDialer reason-tagged proxy-bridge-down events from a single tearDownBridge emission point - Reactive redial scheduler that skips idle and auth_failed reasons * feat(mesh): add reverse-direction activity log entries (closes R1-B) acceptReverseLocal now emits route.resolve.ok with direction=reverse on connect ack and route.resolve.fail with direction=reverse plus reason=container_not_found / connect_error pre-connect. Post-connect close/error stays silent. Reuses existing event types via the new details.direction discriminator so frontend filters are unaffected. * feat(mesh): add peer-to-central callback dial path (closes R1-A2) Closes the architectural gap where proxy-mode mesh peers could not re-establish their tunnel to central after any non-idle bridge teardown (idle close, network blip, central restart, peer reboot). Central remains the hub for the data plane; the change is purely about WS initiation. Symmetric WS initiation, asymmetric protocol roles. Central retains PilotTunnelBridge ownership; peer retains TcpStreamSwitchboard + reverseDialer ownership. Central bootstraps callback credentials over the first authenticated central-initiated mesh tunnel via a one-shot mesh_handshake JSON frame; peer persists the material in a new mesh_centrals SQLite table and dials central's new /api/mesh/proxy-tunnel-from-peer endpoint when local cross-node traffic needs a bridge and none is live. Mesh_tunnel JWT (HS256, signed with auth_jwt_secret) carries scope, audience, issuer (central instance id), peer api_token fingerprint, kid. Validation on inbound peer dial: algorithm pin, signature, scope, audience, instance, time bounds, node existence and mode, fingerprint match. Failures return HTTP 401 with a machine-readable reason; peer routes the response per a clear-vs-keep cache matrix. Triggers proactive bootstrap on mesh-enable and api_token rotation; central startup fans out to mesh-enabled proxy-mode nodes with mesh_stacks rows (throttled, fire-and-forget). Reactive redial on non-idle bridge loss. Capability-gated handshake send (mesh_proxy_callback_bootstrap) makes the upgrade path safe against older peers in mixed-version fleets. Adds peer-side /api/system/pilot-tunnels centralCallback diag block, bounded counter metrics for bootstrap and dial events. SENCHO_PRIMARY_URL preflight warning when unset on a central with mesh-enabled proxy nodes. Tested with unit suites for the validation chain, registry, manager, and both dialers; integration tests for bootstrap E2E (asserts protocol-role invariant), api_token rotation, instance id change, version skew, and pilot-mode regression. * fix(mesh): green CI on the symmetric callback branch Two independent CI failures, both surgical: 1. Backend tests (11 fails): four mesh test files called setupTestDb in beforeEach. setupTestDb does not reset the DatabaseService singleton, so the per-test afterEach rm of the previous tmpdir left the singleton connection pointing at a deleted file. The next beforeEach's line-55 write threw SQLITE_READONLY_DBMOVED on Linux. Windows file-lock semantics hid this locally. Hoist setupTestDb / cleanupTestDb to file-scope beforeAll / afterAll; per-test state resets stay in beforeEach. Matches the convention in the eight mesh test files that already pass. 2. CodeQL (4 high alerts): js/insufficient-password-hash flagged sha256(api_token) at four sites. The api_token is a 256-bit opaque bearer (sen_sk_-prefixed), not a human password; sha256 is the correct fingerprint primitive for binding the mesh_tunnel JWT to a specific token. Add the two production files plus the two test files that mint the fingerprint to the existing path-scoped query-filter for that rule. * fix(mesh): drop unused afterEach import and revert dead codeql config ESLint flagged afterEach as unused in mesh-central-registry.test.ts:1 after the previous commit hoisted setup/teardown to file-scope beforeAll/afterAll. Remove from the vitest import line. Revert the codeql-config.yml additions from the previous commit. The paths: sub-key under query-filters > exclude is not a documented CodeQL feature and silently no-ops. The four js/insufficient-password-hash alerts on api_token fingerprinting are tracked as dismissed false positives in the GitHub Security tab rather than via dead config.
This commit is contained in:
Anso
@@ -11,6 +11,39 @@ import {
|
||||
import { sanitizeForLog } from '../utils/safeLog';
|
||||
import { isDebugEnabled } from '../utils/debug';
|
||||
import { rejectUpgrade as reject } from './reject';
|
||||
import { MeshCentralRegistry, type MeshCentralMaterial } from '../services/MeshCentralRegistry';
|
||||
|
||||
/**
|
||||
* Bootstrap phase for the first-frame state machine.
|
||||
*
|
||||
* Central MAY send a `mesh_handshake` JSON frame as the FIRST text frame
|
||||
* after upgrade. If it arrives we persist the callback material via
|
||||
* MeshCentralRegistry and transition to `consumed`. Any frame other than
|
||||
* the handshake (or no frame at all) moves us to `past`, after which a
|
||||
* later handshake is a protocol error.
|
||||
*/
|
||||
type BootstrapPhase = 'awaiting' | 'consumed' | 'past';
|
||||
|
||||
interface MeshHandshakeFrame {
|
||||
t: 'mesh_handshake';
|
||||
v: number;
|
||||
peerNodeId: number;
|
||||
centralInstanceId: string;
|
||||
centralApiUrl: string;
|
||||
meshTunnelJwt: string;
|
||||
jwtExpiresAt: number;
|
||||
}
|
||||
|
||||
function isMeshHandshakeFrame(parsed: unknown): parsed is MeshHandshakeFrame {
|
||||
return typeof parsed === 'object' && parsed !== null
|
||||
&& (parsed as { t?: unknown }).t === 'mesh_handshake'
|
||||
&& typeof (parsed as { v?: unknown }).v === 'number'
|
||||
&& typeof (parsed as { peerNodeId?: unknown }).peerNodeId === 'number'
|
||||
&& typeof (parsed as { centralInstanceId?: unknown }).centralInstanceId === 'string'
|
||||
&& typeof (parsed as { centralApiUrl?: unknown }).centralApiUrl === 'string'
|
||||
&& typeof (parsed as { meshTunnelJwt?: unknown }).meshTunnelJwt === 'string'
|
||||
&& typeof (parsed as { jwtExpiresAt?: unknown }).jwtExpiresAt === 'number';
|
||||
}
|
||||
|
||||
/**
|
||||
* Mesh proxy-tunnel ingress.
|
||||
@@ -136,18 +169,75 @@ async function attachSwitchboard(ws: WebSocket, peerNodeId: number | null): Prom
|
||||
return;
|
||||
}
|
||||
|
||||
let phase: BootstrapPhase = 'awaiting';
|
||||
|
||||
const onMessage = (data: unknown, isBinary: boolean): void => {
|
||||
if (!switchboard) return;
|
||||
try {
|
||||
if (isBinary) {
|
||||
if (phase === 'awaiting') phase = 'past';
|
||||
const buf = wsDataToBuffer(data);
|
||||
if (!buf) return;
|
||||
switchboard.handleBinaryFrame(decodeBinaryFrame(buf));
|
||||
} else {
|
||||
const text = wsDataToString(data);
|
||||
if (text == null) return;
|
||||
switchboard.handleJsonFrame(decodeJsonFrame(text));
|
||||
return;
|
||||
}
|
||||
const text = wsDataToString(data);
|
||||
if (text == null) {
|
||||
if (phase === 'awaiting') phase = 'past';
|
||||
return;
|
||||
}
|
||||
let parsedForBootstrap: unknown = null;
|
||||
let parseSucceeded = false;
|
||||
try {
|
||||
parsedForBootstrap = JSON.parse(text);
|
||||
parseSucceeded = true;
|
||||
} catch {
|
||||
// Not valid JSON; defer to decodeJsonFrame's stricter error path below.
|
||||
}
|
||||
|
||||
if (parseSucceeded && typeof parsedForBootstrap === 'object'
|
||||
&& parsedForBootstrap !== null
|
||||
&& (parsedForBootstrap as { t?: unknown }).t === 'mesh_handshake') {
|
||||
if (phase !== 'awaiting') {
|
||||
ws.close(1008, 'mesh_handshake out of order');
|
||||
return;
|
||||
}
|
||||
if (!isMeshHandshakeFrame(parsedForBootstrap)) {
|
||||
ws.close(1008, 'malformed mesh_handshake');
|
||||
return;
|
||||
}
|
||||
const material: MeshCentralMaterial = {
|
||||
centralInstanceId: parsedForBootstrap.centralInstanceId,
|
||||
centralApiUrl: parsedForBootstrap.centralApiUrl.replace(/\/+$/, ''),
|
||||
callbackJwt: parsedForBootstrap.meshTunnelJwt,
|
||||
jwtIssuedAt: Math.floor(Date.now() / 1000),
|
||||
jwtExpiresAt: parsedForBootstrap.jwtExpiresAt,
|
||||
};
|
||||
try {
|
||||
MeshCentralRegistry.getInstance().upsert(material);
|
||||
phase = 'consumed';
|
||||
const centralInstanceId = parsedForBootstrap.centralInstanceId;
|
||||
const jwtExpiresAt = parsedForBootstrap.jwtExpiresAt;
|
||||
void (async () => {
|
||||
try {
|
||||
const { MeshService: MeshSvc } = await import('../services/MeshService');
|
||||
MeshSvc.getInstance().logActivity({
|
||||
source: 'mesh', level: 'info',
|
||||
type: 'mesh_handshake.received',
|
||||
message: `mesh callback bootstrap received from central ${centralInstanceId}`,
|
||||
details: { centralInstanceId, jwtExpiresAt },
|
||||
});
|
||||
} catch { /* best-effort log; bootstrap success path must not depend on it */ }
|
||||
})();
|
||||
} catch (err) {
|
||||
console.warn(`[meshProxyTunnel] mesh_handshake persist failed: ${sanitizeForLog((err as Error).message)}`);
|
||||
try { ws.close(1011, 'bootstrap persist failed'); } catch { /* ignore */ }
|
||||
}
|
||||
return;
|
||||
}
|
||||
|
||||
if (phase === 'awaiting') phase = 'past';
|
||||
switchboard.handleJsonFrame(decodeJsonFrame(text));
|
||||
} catch (err) {
|
||||
if (isDebugEnabled()) {
|
||||
console.warn('[MeshProxy:diag] malformed frame:', sanitizeForLog((err as Error).message));
|
||||
|
||||
@@ -0,0 +1,142 @@
|
||||
/**
|
||||
* Central-side ingress for peer-initiated proxy-mode mesh tunnels.
|
||||
*
|
||||
* After the bootstrap exchange in `meshProxyTunnel.ts` (Task 7/8), a peer
|
||||
* with the central's `mesh_tunnel` JWT can dial *back* to central at this
|
||||
* endpoint. The validation chain enforces every claim the bootstrap mint
|
||||
* produced and confirms the peer's stored fingerprint still matches its
|
||||
* current api_token. Each failure path returns HTTP 401 with a stable
|
||||
* machine-readable `reason` code in the JSON body, so the dialer side
|
||||
* (PeerToCentralMeshSessionDialer, Task 10) can act on the reason without
|
||||
* scraping prose.
|
||||
*
|
||||
* Auth runs MANUALLY in this handler. Express middleware does not run on
|
||||
* WebSocket upgrades, and the credential here is a Bearer JWT minted by
|
||||
* the central itself, not a user session.
|
||||
*/
|
||||
import type { IncomingMessage } from 'http';
|
||||
import type { Duplex } from 'stream';
|
||||
import jwt, { type Algorithm, type JwtPayload } from 'jsonwebtoken';
|
||||
import { createHash } from 'crypto';
|
||||
import { WebSocketServer } from 'ws';
|
||||
import { DatabaseService } from '../services/DatabaseService';
|
||||
import { PilotTunnelBridge } from '../services/PilotTunnelBridge';
|
||||
import { PilotTunnelManager } from '../services/PilotTunnelManager';
|
||||
import { PilotMetrics } from '../services/PilotMetrics';
|
||||
import { sanitizeForLog } from '../utils/safeLog';
|
||||
|
||||
const wss = new WebSocketServer({ noServer: true });
|
||||
const CLOCK_SKEW_SEC = 60;
|
||||
|
||||
type RejectReason =
|
||||
| 'algorithm_mismatch' | 'signature_invalid'
|
||||
| 'scope_mismatch' | 'audience_mismatch' | 'instance_mismatch'
|
||||
| 'stale' | 'clock_skew' | 'node_deleted' | 'mode_mismatch'
|
||||
| 'token_fingerprint_mismatch' | 'malformed';
|
||||
|
||||
function rejectUpgrade(socket: Duplex, reason: RejectReason): void {
|
||||
const body = JSON.stringify({ reason });
|
||||
const headers = [
|
||||
'HTTP/1.1 401 Unauthorized',
|
||||
'Content-Type: application/json',
|
||||
`Content-Length: ${Buffer.byteLength(body)}`,
|
||||
'Connection: close',
|
||||
'',
|
||||
body,
|
||||
].join('\r\n');
|
||||
try { socket.write(headers); } catch { /* socket already gone */ }
|
||||
try { socket.destroy(); } catch { /* socket already gone */ }
|
||||
}
|
||||
|
||||
function extractBearer(req: IncomingMessage): string | null {
|
||||
const h = req.headers['authorization'];
|
||||
if (typeof h !== 'string' || !h.startsWith('Bearer ')) return null;
|
||||
return h.slice('Bearer '.length).trim() || null;
|
||||
}
|
||||
|
||||
export function handleMeshProxyTunnelFromPeerUpgrade(
|
||||
req: IncomingMessage,
|
||||
socket: Duplex,
|
||||
head: Buffer,
|
||||
): void {
|
||||
const token = extractBearer(req);
|
||||
if (!token) { rejectUpgrade(socket, 'malformed'); return; }
|
||||
|
||||
let header: { alg?: string; kid?: string } = {};
|
||||
try {
|
||||
const decoded = jwt.decode(token, { complete: true });
|
||||
header = (decoded?.header ?? {}) as typeof header;
|
||||
} catch { rejectUpgrade(socket, 'malformed'); return; }
|
||||
if (header.alg !== 'HS256') { rejectUpgrade(socket, 'algorithm_mismatch'); return; }
|
||||
|
||||
const settings = DatabaseService.getInstance().getGlobalSettings();
|
||||
const secret = settings.auth_jwt_secret;
|
||||
if (!secret) { rejectUpgrade(socket, 'malformed'); return; }
|
||||
|
||||
let payload: JwtPayload;
|
||||
try {
|
||||
payload = jwt.verify(token, secret, { algorithms: ['HS256' as Algorithm] }) as JwtPayload;
|
||||
} catch { rejectUpgrade(socket, 'signature_invalid'); return; }
|
||||
|
||||
if (payload.scope !== 'mesh_tunnel') { rejectUpgrade(socket, 'scope_mismatch'); return; }
|
||||
|
||||
// SENCHO_PRIMARY_URL must be set on central for peer-initiated dial-back
|
||||
// to operate. The Task 8 bootstrap-mint side already skips when it's
|
||||
// unset; this is the matching fail-safe on the verify side. Reject all
|
||||
// peer dials with audience_mismatch when central has no canonical
|
||||
// origin to validate against.
|
||||
const canonicalOrigin = (process.env.SENCHO_PRIMARY_URL ?? '').replace(/\/+$/, '');
|
||||
if (!canonicalOrigin) { rejectUpgrade(socket, 'audience_mismatch'); return; }
|
||||
if (payload.aud !== canonicalOrigin) { rejectUpgrade(socket, 'audience_mismatch'); return; }
|
||||
|
||||
// instance_id is operational state, not user-defined config, so it
|
||||
// lives in system_state rather than global_settings.
|
||||
const instanceId = DatabaseService.getInstance().getSystemState('instance_id');
|
||||
if (payload.iss !== instanceId) { rejectUpgrade(socket, 'instance_mismatch'); return; }
|
||||
|
||||
const nowSec = Math.floor(Date.now() / 1000);
|
||||
if (typeof payload.exp !== 'number' || payload.exp <= nowSec) {
|
||||
rejectUpgrade(socket, 'stale'); return;
|
||||
}
|
||||
if (typeof payload.iat !== 'number' || payload.iat > nowSec + CLOCK_SKEW_SEC) {
|
||||
rejectUpgrade(socket, 'clock_skew'); return;
|
||||
}
|
||||
|
||||
const peerNodeId = Number(payload.sub);
|
||||
if (!Number.isInteger(peerNodeId) || peerNodeId <= 0) {
|
||||
rejectUpgrade(socket, 'malformed'); return;
|
||||
}
|
||||
const node = DatabaseService.getInstance().getNode(peerNodeId);
|
||||
if (!node) { rejectUpgrade(socket, 'node_deleted'); return; }
|
||||
if (node.type !== 'remote' || node.mode !== 'proxy') {
|
||||
rejectUpgrade(socket, 'mode_mismatch'); return;
|
||||
}
|
||||
|
||||
if (!node.api_token) { rejectUpgrade(socket, 'token_fingerprint_mismatch'); return; }
|
||||
const expectedFp = createHash('sha256').update(node.api_token).digest('hex').slice(0, 16);
|
||||
const claimedFp = payload['peer_token_fp'];
|
||||
if (typeof claimedFp !== 'string' || claimedFp !== expectedFp) {
|
||||
rejectUpgrade(socket, 'token_fingerprint_mismatch'); return;
|
||||
}
|
||||
|
||||
wss.handleUpgrade(req, socket, head, (ws) => {
|
||||
try {
|
||||
const bridge = new PilotTunnelBridge(peerNodeId, ws);
|
||||
bridge.start().then(() => {
|
||||
try {
|
||||
PilotTunnelManager.getInstance().replaceOrRegisterProxyBridge(peerNodeId, bridge);
|
||||
PilotMetrics.increment('proxy_bridges_peer_initiated_total');
|
||||
} catch (err) {
|
||||
try { bridge.close(1013, 'manager rejected'); } catch { /* ignore */ }
|
||||
console.warn(`[meshProxyTunnelFromPeer] register failed: ${sanitizeForLog((err as Error).message)}`);
|
||||
}
|
||||
}).catch((err) => {
|
||||
try { bridge.close(1011, 'bridge start failed'); } catch { /* ignore */ }
|
||||
console.warn(`[meshProxyTunnelFromPeer] bridge start failed: ${sanitizeForLog((err as Error).message)}`);
|
||||
});
|
||||
} catch (err) {
|
||||
console.warn(`[meshProxyTunnelFromPeer] bridge construct failed: ${sanitizeForLog((err as Error).message)}`);
|
||||
try { ws.close(1011, 'bridge init failed'); } catch { /* ignore */ }
|
||||
}
|
||||
});
|
||||
}
|
||||
@@ -8,6 +8,7 @@ import { NodeRegistry } from '../services/NodeRegistry';
|
||||
import { COOKIE_NAME } from '../helpers/constants';
|
||||
import { handlePilotTunnel } from './pilotTunnel';
|
||||
import { handleMeshProxyTunnel } from './meshProxyTunnel';
|
||||
import { handleMeshProxyTunnelFromPeerUpgrade } from './meshProxyTunnelFromPeer';
|
||||
import { handleNotificationsWs } from './notifications';
|
||||
import { handleRemoteForwarder } from './remoteForwarder';
|
||||
import { handleLogsWs } from './logs';
|
||||
@@ -31,15 +32,16 @@ function parseCookies(req: IncomingMessage): Record<string, string> {
|
||||
* `connection` handler on the main wss.
|
||||
*
|
||||
* Dispatch order (first match wins):
|
||||
* 1. `/api/pilot/tunnel` -> handlePilotTunnel (own auth, own wss)
|
||||
* 2. shared cookie/Bearer auth + JWT verify (rejects unauthenticated)
|
||||
* 3. API token scope gate (read-only / deploy-only restricted to logs + notifications)
|
||||
* 4. `/api/mesh/proxy-tunnel` -> handleMeshProxyTunnel (machine-to-machine: node_proxy or full-admin api_token)
|
||||
* 5. `/ws/notifications` local -> handleNotificationsWs
|
||||
* 6. remote nodeId path -> handleRemoteForwarder
|
||||
* 7. `/api/stacks/:name/logs` -> handleLogsWs
|
||||
* 8. `/api/system/host-console` -> handleHostConsoleWs
|
||||
* 9. fallback -> handleGenericWs (`/ws` exec + stats)
|
||||
* 1. `/api/pilot/tunnel` -> handlePilotTunnel (own auth, own wss)
|
||||
* 2. `/api/mesh/proxy-tunnel-from-peer` -> handleMeshProxyTunnelFromPeerUpgrade (own JWT chain, central-side dial-back)
|
||||
* 3. shared cookie/Bearer auth + JWT verify (rejects unauthenticated)
|
||||
* 4. API token scope gate (read-only / deploy-only restricted to logs + notifications)
|
||||
* 5. `/api/mesh/proxy-tunnel` -> handleMeshProxyTunnel (machine-to-machine: node_proxy or full-admin api_token)
|
||||
* 6. `/ws/notifications` local -> handleNotificationsWs
|
||||
* 7. remote nodeId path -> handleRemoteForwarder
|
||||
* 8. `/api/stacks/:name/logs` -> handleLogsWs
|
||||
* 9. `/api/system/host-console` -> handleHostConsoleWs
|
||||
* 10. fallback -> handleGenericWs (`/ws` exec + stats)
|
||||
*/
|
||||
export function attachUpgrade(
|
||||
server: http.Server,
|
||||
@@ -51,12 +53,21 @@ export function attachUpgrade(
|
||||
|
||||
server.on('upgrade', async (req, socket, head) => {
|
||||
// Pilot-agent tunnel ingress: machine credentials, no cookies.
|
||||
// Mesh proxy-tunnel-from-peer ingress: peer-initiated dial-back over
|
||||
// a `mesh_tunnel`-scoped JWT (minted earlier during the bootstrap
|
||||
// exchange). Both handlers run their own auth before the shared
|
||||
// cookie/Bearer pipeline because their credentials are not user
|
||||
// sessions and would fail the shared user-existence check.
|
||||
try {
|
||||
const reqUrl = new URL(req.url || '/', `http://${req.headers.host || 'localhost'}`);
|
||||
if (reqUrl.pathname === '/api/pilot/tunnel') {
|
||||
await handlePilotTunnel(req, socket, head, pilotTunnelWss);
|
||||
return;
|
||||
}
|
||||
if (reqUrl.pathname === '/api/mesh/proxy-tunnel-from-peer') {
|
||||
handleMeshProxyTunnelFromPeerUpgrade(req, socket, head);
|
||||
return;
|
||||
}
|
||||
} catch {
|
||||
// URL parse error falls through and will be rejected below.
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user