Files
sencho/backend/src/websocket/meshProxyTunnelFromPeer.ts
T
Anso cf618dd866 feat(mesh): symmetric WS dial for proxy-mode mesh peers (#1066)
* chore(mesh): foundation for symmetric callback dial

Adds the data-plane scaffolding that the symmetric callback dial fix
builds on:
- mesh_centrals table for peer-side bootstrap material
- MeshCentralRegistry service (upsert/getActive/clear/markUsed/markRejected)
- PilotTunnelManager kind discriminator and replaceOrRegisterProxyBridge
- mesh_proxy_callback_bootstrap capability registration
- MeshProxyTunnelDialer reason-tagged proxy-bridge-down events from a
  single tearDownBridge emission point
- Reactive redial scheduler that skips idle and auth_failed reasons

* feat(mesh): add reverse-direction activity log entries (closes R1-B)

acceptReverseLocal now emits route.resolve.ok with direction=reverse on
connect ack and route.resolve.fail with direction=reverse plus
reason=container_not_found / connect_error pre-connect. Post-connect
close/error stays silent. Reuses existing event types via the new
details.direction discriminator so frontend filters are unaffected.

* feat(mesh): add peer-to-central callback dial path (closes R1-A2)

Closes the architectural gap where proxy-mode mesh peers could not
re-establish their tunnel to central after any non-idle bridge teardown
(idle close, network blip, central restart, peer reboot). Central remains
the hub for the data plane; the change is purely about WS initiation.

Symmetric WS initiation, asymmetric protocol roles. Central retains
PilotTunnelBridge ownership; peer retains TcpStreamSwitchboard +
reverseDialer ownership. Central bootstraps callback credentials over
the first authenticated central-initiated mesh tunnel via a one-shot
mesh_handshake JSON frame; peer persists the material in a new
mesh_centrals SQLite table and dials central's new
/api/mesh/proxy-tunnel-from-peer endpoint when local cross-node traffic
needs a bridge and none is live.

Mesh_tunnel JWT (HS256, signed with auth_jwt_secret) carries scope, audience,
issuer (central instance id), peer api_token fingerprint, kid. Validation
on inbound peer dial: algorithm pin, signature, scope, audience, instance,
time bounds, node existence and mode, fingerprint match. Failures return
HTTP 401 with a machine-readable reason; peer routes the response per a
clear-vs-keep cache matrix.

Triggers proactive bootstrap on mesh-enable and api_token rotation; central
startup fans out to mesh-enabled proxy-mode nodes with mesh_stacks rows
(throttled, fire-and-forget). Reactive redial on non-idle bridge loss.

Capability-gated handshake send (mesh_proxy_callback_bootstrap) makes the
upgrade path safe against older peers in mixed-version fleets.

Adds peer-side /api/system/pilot-tunnels centralCallback diag block,
bounded counter metrics for bootstrap and dial events. SENCHO_PRIMARY_URL
preflight warning when unset on a central with mesh-enabled proxy nodes.

Tested with unit suites for the validation chain, registry, manager, and
both dialers; integration tests for bootstrap E2E (asserts protocol-role
invariant), api_token rotation, instance id change, version skew, and
pilot-mode regression.

* fix(mesh): green CI on the symmetric callback branch

Two independent CI failures, both surgical:

1. Backend tests (11 fails): four mesh test files called setupTestDb in
   beforeEach. setupTestDb does not reset the DatabaseService singleton,
   so the per-test afterEach rm of the previous tmpdir left the singleton
   connection pointing at a deleted file. The next beforeEach's line-55
   write threw SQLITE_READONLY_DBMOVED on Linux. Windows file-lock
   semantics hid this locally. Hoist setupTestDb / cleanupTestDb to
   file-scope beforeAll / afterAll; per-test state resets stay in
   beforeEach. Matches the convention in the eight mesh test files that
   already pass.

2. CodeQL (4 high alerts): js/insufficient-password-hash flagged
   sha256(api_token) at four sites. The api_token is a 256-bit opaque
   bearer (sen_sk_-prefixed), not a human password; sha256 is the
   correct fingerprint primitive for binding the mesh_tunnel JWT to a
   specific token. Add the two production files plus the two test
   files that mint the fingerprint to the existing path-scoped
   query-filter for that rule.

* fix(mesh): drop unused afterEach import and revert dead codeql config

ESLint flagged afterEach as unused in mesh-central-registry.test.ts:1
after the previous commit hoisted setup/teardown to file-scope
beforeAll/afterAll. Remove from the vitest import line.

Revert the codeql-config.yml additions from the previous commit. The
paths: sub-key under query-filters > exclude is not a documented CodeQL
feature and silently no-ops. The four js/insufficient-password-hash
alerts on api_token fingerprinting are tracked as dismissed false
positives in the GitHub Security tab rather than via dead config.
2026-05-16 14:58:19 -04:00

143 lines
6.5 KiB
TypeScript

/**
* Central-side ingress for peer-initiated proxy-mode mesh tunnels.
*
* After the bootstrap exchange in `meshProxyTunnel.ts` (Task 7/8), a peer
* with the central's `mesh_tunnel` JWT can dial *back* to central at this
* endpoint. The validation chain enforces every claim the bootstrap mint
* produced and confirms the peer's stored fingerprint still matches its
* current api_token. Each failure path returns HTTP 401 with a stable
* machine-readable `reason` code in the JSON body, so the dialer side
* (PeerToCentralMeshSessionDialer, Task 10) can act on the reason without
* scraping prose.
*
* Auth runs MANUALLY in this handler. Express middleware does not run on
* WebSocket upgrades, and the credential here is a Bearer JWT minted by
* the central itself, not a user session.
*/
import type { IncomingMessage } from 'http';
import type { Duplex } from 'stream';
import jwt, { type Algorithm, type JwtPayload } from 'jsonwebtoken';
import { createHash } from 'crypto';
import { WebSocketServer } from 'ws';
import { DatabaseService } from '../services/DatabaseService';
import { PilotTunnelBridge } from '../services/PilotTunnelBridge';
import { PilotTunnelManager } from '../services/PilotTunnelManager';
import { PilotMetrics } from '../services/PilotMetrics';
import { sanitizeForLog } from '../utils/safeLog';
const wss = new WebSocketServer({ noServer: true });
const CLOCK_SKEW_SEC = 60;
type RejectReason =
| 'algorithm_mismatch' | 'signature_invalid'
| 'scope_mismatch' | 'audience_mismatch' | 'instance_mismatch'
| 'stale' | 'clock_skew' | 'node_deleted' | 'mode_mismatch'
| 'token_fingerprint_mismatch' | 'malformed';
function rejectUpgrade(socket: Duplex, reason: RejectReason): void {
const body = JSON.stringify({ reason });
const headers = [
'HTTP/1.1 401 Unauthorized',
'Content-Type: application/json',
`Content-Length: ${Buffer.byteLength(body)}`,
'Connection: close',
'',
body,
].join('\r\n');
try { socket.write(headers); } catch { /* socket already gone */ }
try { socket.destroy(); } catch { /* socket already gone */ }
}
function extractBearer(req: IncomingMessage): string | null {
const h = req.headers['authorization'];
if (typeof h !== 'string' || !h.startsWith('Bearer ')) return null;
return h.slice('Bearer '.length).trim() || null;
}
export function handleMeshProxyTunnelFromPeerUpgrade(
req: IncomingMessage,
socket: Duplex,
head: Buffer,
): void {
const token = extractBearer(req);
if (!token) { rejectUpgrade(socket, 'malformed'); return; }
let header: { alg?: string; kid?: string } = {};
try {
const decoded = jwt.decode(token, { complete: true });
header = (decoded?.header ?? {}) as typeof header;
} catch { rejectUpgrade(socket, 'malformed'); return; }
if (header.alg !== 'HS256') { rejectUpgrade(socket, 'algorithm_mismatch'); return; }
const settings = DatabaseService.getInstance().getGlobalSettings();
const secret = settings.auth_jwt_secret;
if (!secret) { rejectUpgrade(socket, 'malformed'); return; }
let payload: JwtPayload;
try {
payload = jwt.verify(token, secret, { algorithms: ['HS256' as Algorithm] }) as JwtPayload;
} catch { rejectUpgrade(socket, 'signature_invalid'); return; }
if (payload.scope !== 'mesh_tunnel') { rejectUpgrade(socket, 'scope_mismatch'); return; }
// SENCHO_PRIMARY_URL must be set on central for peer-initiated dial-back
// to operate. The Task 8 bootstrap-mint side already skips when it's
// unset; this is the matching fail-safe on the verify side. Reject all
// peer dials with audience_mismatch when central has no canonical
// origin to validate against.
const canonicalOrigin = (process.env.SENCHO_PRIMARY_URL ?? '').replace(/\/+$/, '');
if (!canonicalOrigin) { rejectUpgrade(socket, 'audience_mismatch'); return; }
if (payload.aud !== canonicalOrigin) { rejectUpgrade(socket, 'audience_mismatch'); return; }
// instance_id is operational state, not user-defined config, so it
// lives in system_state rather than global_settings.
const instanceId = DatabaseService.getInstance().getSystemState('instance_id');
if (payload.iss !== instanceId) { rejectUpgrade(socket, 'instance_mismatch'); return; }
const nowSec = Math.floor(Date.now() / 1000);
if (typeof payload.exp !== 'number' || payload.exp <= nowSec) {
rejectUpgrade(socket, 'stale'); return;
}
if (typeof payload.iat !== 'number' || payload.iat > nowSec + CLOCK_SKEW_SEC) {
rejectUpgrade(socket, 'clock_skew'); return;
}
const peerNodeId = Number(payload.sub);
if (!Number.isInteger(peerNodeId) || peerNodeId <= 0) {
rejectUpgrade(socket, 'malformed'); return;
}
const node = DatabaseService.getInstance().getNode(peerNodeId);
if (!node) { rejectUpgrade(socket, 'node_deleted'); return; }
if (node.type !== 'remote' || node.mode !== 'proxy') {
rejectUpgrade(socket, 'mode_mismatch'); return;
}
if (!node.api_token) { rejectUpgrade(socket, 'token_fingerprint_mismatch'); return; }
const expectedFp = createHash('sha256').update(node.api_token).digest('hex').slice(0, 16);
const claimedFp = payload['peer_token_fp'];
if (typeof claimedFp !== 'string' || claimedFp !== expectedFp) {
rejectUpgrade(socket, 'token_fingerprint_mismatch'); return;
}
wss.handleUpgrade(req, socket, head, (ws) => {
try {
const bridge = new PilotTunnelBridge(peerNodeId, ws);
bridge.start().then(() => {
try {
PilotTunnelManager.getInstance().replaceOrRegisterProxyBridge(peerNodeId, bridge);
PilotMetrics.increment('proxy_bridges_peer_initiated_total');
} catch (err) {
try { bridge.close(1013, 'manager rejected'); } catch { /* ignore */ }
console.warn(`[meshProxyTunnelFromPeer] register failed: ${sanitizeForLog((err as Error).message)}`);
}
}).catch((err) => {
try { bridge.close(1011, 'bridge start failed'); } catch { /* ignore */ }
console.warn(`[meshProxyTunnelFromPeer] bridge start failed: ${sanitizeForLog((err as Error).message)}`);
});
} catch (err) {
console.warn(`[meshProxyTunnelFromPeer] bridge construct failed: ${sanitizeForLog((err as Error).message)}`);
try { ws.close(1011, 'bridge init failed'); } catch { /* ignore */ }
}
});
}