mirror of
https://github.com/Studio-Saelix/sencho.git
synced 2026-08-18 06:23:18 +00:00
feat(mesh): symmetric WS dial for proxy-mode mesh peers (#1066)
* chore(mesh): foundation for symmetric callback dial Adds the data-plane scaffolding that the symmetric callback dial fix builds on: - mesh_centrals table for peer-side bootstrap material - MeshCentralRegistry service (upsert/getActive/clear/markUsed/markRejected) - PilotTunnelManager kind discriminator and replaceOrRegisterProxyBridge - mesh_proxy_callback_bootstrap capability registration - MeshProxyTunnelDialer reason-tagged proxy-bridge-down events from a single tearDownBridge emission point - Reactive redial scheduler that skips idle and auth_failed reasons * feat(mesh): add reverse-direction activity log entries (closes R1-B) acceptReverseLocal now emits route.resolve.ok with direction=reverse on connect ack and route.resolve.fail with direction=reverse plus reason=container_not_found / connect_error pre-connect. Post-connect close/error stays silent. Reuses existing event types via the new details.direction discriminator so frontend filters are unaffected. * feat(mesh): add peer-to-central callback dial path (closes R1-A2) Closes the architectural gap where proxy-mode mesh peers could not re-establish their tunnel to central after any non-idle bridge teardown (idle close, network blip, central restart, peer reboot). Central remains the hub for the data plane; the change is purely about WS initiation. Symmetric WS initiation, asymmetric protocol roles. Central retains PilotTunnelBridge ownership; peer retains TcpStreamSwitchboard + reverseDialer ownership. Central bootstraps callback credentials over the first authenticated central-initiated mesh tunnel via a one-shot mesh_handshake JSON frame; peer persists the material in a new mesh_centrals SQLite table and dials central's new /api/mesh/proxy-tunnel-from-peer endpoint when local cross-node traffic needs a bridge and none is live. Mesh_tunnel JWT (HS256, signed with auth_jwt_secret) carries scope, audience, issuer (central instance id), peer api_token fingerprint, kid. Validation on inbound peer dial: algorithm pin, signature, scope, audience, instance, time bounds, node existence and mode, fingerprint match. Failures return HTTP 401 with a machine-readable reason; peer routes the response per a clear-vs-keep cache matrix. Triggers proactive bootstrap on mesh-enable and api_token rotation; central startup fans out to mesh-enabled proxy-mode nodes with mesh_stacks rows (throttled, fire-and-forget). Reactive redial on non-idle bridge loss. Capability-gated handshake send (mesh_proxy_callback_bootstrap) makes the upgrade path safe against older peers in mixed-version fleets. Adds peer-side /api/system/pilot-tunnels centralCallback diag block, bounded counter metrics for bootstrap and dial events. SENCHO_PRIMARY_URL preflight warning when unset on a central with mesh-enabled proxy nodes. Tested with unit suites for the validation chain, registry, manager, and both dialers; integration tests for bootstrap E2E (asserts protocol-role invariant), api_token rotation, instance id change, version skew, and pilot-mode regression. * fix(mesh): green CI on the symmetric callback branch Two independent CI failures, both surgical: 1. Backend tests (11 fails): four mesh test files called setupTestDb in beforeEach. setupTestDb does not reset the DatabaseService singleton, so the per-test afterEach rm of the previous tmpdir left the singleton connection pointing at a deleted file. The next beforeEach's line-55 write threw SQLITE_READONLY_DBMOVED on Linux. Windows file-lock semantics hid this locally. Hoist setupTestDb / cleanupTestDb to file-scope beforeAll / afterAll; per-test state resets stay in beforeEach. Matches the convention in the eight mesh test files that already pass. 2. CodeQL (4 high alerts): js/insufficient-password-hash flagged sha256(api_token) at four sites. The api_token is a 256-bit opaque bearer (sen_sk_-prefixed), not a human password; sha256 is the correct fingerprint primitive for binding the mesh_tunnel JWT to a specific token. Add the two production files plus the two test files that mint the fingerprint to the existing path-scoped query-filter for that rule. * fix(mesh): drop unused afterEach import and revert dead codeql config ESLint flagged afterEach as unused in mesh-central-registry.test.ts:1 after the previous commit hoisted setup/teardown to file-scope beforeAll/afterAll. Remove from the vitest import line. Revert the codeql-config.yml additions from the previous commit. The paths: sub-key under query-filters > exclude is not a documented CodeQL feature and silently no-ops. The four js/insufficient-password-hash alerts on api_token fingerprinting are tracked as dismissed false positives in the GitHub Security tab rather than via dead config.
This commit is contained in:
Anso
@@ -34,6 +34,7 @@ export const CAPABILITIES = [
|
||||
'registries',
|
||||
'self-update',
|
||||
'vulnerability-scanning',
|
||||
'mesh_proxy_callback_bootstrap',
|
||||
] as const;
|
||||
|
||||
export type Capability = (typeof CAPABILITIES)[number];
|
||||
|
||||
@@ -1475,6 +1475,23 @@ export class DatabaseService {
|
||||
} catch (e) {
|
||||
console.warn('[DatabaseService] Could not create mesh_stacks:', (e as Error).message);
|
||||
}
|
||||
try {
|
||||
this.db.prepare(`
|
||||
CREATE TABLE IF NOT EXISTS mesh_centrals (
|
||||
central_instance_id TEXT PRIMARY KEY,
|
||||
central_api_url TEXT NOT NULL,
|
||||
callback_jwt TEXT NOT NULL,
|
||||
jwt_issued_at INTEGER NOT NULL,
|
||||
jwt_expires_at INTEGER NOT NULL,
|
||||
last_bootstrap_at INTEGER NOT NULL,
|
||||
last_used_at INTEGER,
|
||||
last_rejected_at INTEGER,
|
||||
last_reject_reason TEXT
|
||||
)
|
||||
`).run();
|
||||
} catch (e) {
|
||||
console.warn('[DatabaseService] Could not create mesh_centrals:', (e as Error).message);
|
||||
}
|
||||
this.tryAddColumn('nodes', 'mesh_enabled', 'INTEGER NOT NULL DEFAULT 0');
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,119 @@
|
||||
import { EventEmitter } from 'events';
|
||||
import { DatabaseService } from './DatabaseService';
|
||||
|
||||
export interface MeshCentralMaterial {
|
||||
centralInstanceId: string;
|
||||
centralApiUrl: string;
|
||||
callbackJwt: string;
|
||||
jwtIssuedAt: number;
|
||||
jwtExpiresAt: number;
|
||||
}
|
||||
|
||||
export interface MeshCentralRow extends MeshCentralMaterial {
|
||||
lastBootstrapAt: number;
|
||||
lastUsedAt: number | null;
|
||||
lastRejectedAt: number | null;
|
||||
lastRejectReason: string | null;
|
||||
}
|
||||
|
||||
export class MeshCentralRegistry extends EventEmitter {
|
||||
private static instance: MeshCentralRegistry | null = null;
|
||||
private warnedMultiRow = false;
|
||||
|
||||
private constructor() { super(); }
|
||||
|
||||
public static getInstance(): MeshCentralRegistry {
|
||||
if (!this.instance) this.instance = new MeshCentralRegistry();
|
||||
return this.instance;
|
||||
}
|
||||
|
||||
public static resetForTest(): void {
|
||||
this.instance = null;
|
||||
}
|
||||
|
||||
public upsert(material: MeshCentralMaterial): void {
|
||||
const db = DatabaseService.getInstance().getDb();
|
||||
const prior = this.getActive();
|
||||
db.prepare(`
|
||||
INSERT INTO mesh_centrals (
|
||||
central_instance_id, central_api_url, callback_jwt,
|
||||
jwt_issued_at, jwt_expires_at, last_bootstrap_at,
|
||||
last_used_at, last_rejected_at, last_reject_reason
|
||||
) VALUES (?, ?, ?, ?, ?, ?, NULL, NULL, NULL)
|
||||
ON CONFLICT(central_instance_id) DO UPDATE SET
|
||||
central_api_url = excluded.central_api_url,
|
||||
callback_jwt = excluded.callback_jwt,
|
||||
jwt_issued_at = excluded.jwt_issued_at,
|
||||
jwt_expires_at = excluded.jwt_expires_at,
|
||||
last_bootstrap_at = excluded.last_bootstrap_at
|
||||
`).run(
|
||||
material.centralInstanceId,
|
||||
material.centralApiUrl,
|
||||
material.callbackJwt,
|
||||
material.jwtIssuedAt,
|
||||
material.jwtExpiresAt,
|
||||
Date.now(),
|
||||
);
|
||||
|
||||
const isInstanceChange = prior && prior.centralInstanceId !== material.centralInstanceId;
|
||||
if (isInstanceChange) {
|
||||
this.emit('central-instance-changed', {
|
||||
previousInstanceId: prior!.centralInstanceId,
|
||||
newInstanceId: material.centralInstanceId,
|
||||
});
|
||||
}
|
||||
this.emit('central-bootstrap', { centralInstanceId: material.centralInstanceId });
|
||||
}
|
||||
|
||||
public getActive(): MeshCentralRow | null {
|
||||
const db = DatabaseService.getInstance().getDb();
|
||||
const rows = db.prepare(`
|
||||
SELECT * FROM mesh_centrals ORDER BY last_bootstrap_at DESC
|
||||
`).all() as Array<{
|
||||
central_instance_id: string;
|
||||
central_api_url: string;
|
||||
callback_jwt: string;
|
||||
jwt_issued_at: number;
|
||||
jwt_expires_at: number;
|
||||
last_bootstrap_at: number;
|
||||
last_used_at: number | null;
|
||||
last_rejected_at: number | null;
|
||||
last_reject_reason: string | null;
|
||||
}>;
|
||||
if (rows.length === 0) return null;
|
||||
if (rows.length > 1 && !this.warnedMultiRow) {
|
||||
console.warn(`[MeshCentralRegistry] multiple central rows present (${rows.length}), multi-central unsupported in v0.79`);
|
||||
this.warnedMultiRow = true;
|
||||
}
|
||||
const r = rows[0];
|
||||
return {
|
||||
centralInstanceId: r.central_instance_id,
|
||||
centralApiUrl: r.central_api_url,
|
||||
callbackJwt: r.callback_jwt,
|
||||
jwtIssuedAt: r.jwt_issued_at,
|
||||
jwtExpiresAt: r.jwt_expires_at,
|
||||
lastBootstrapAt: r.last_bootstrap_at,
|
||||
lastUsedAt: r.last_used_at,
|
||||
lastRejectedAt: r.last_rejected_at,
|
||||
lastRejectReason: r.last_reject_reason,
|
||||
};
|
||||
}
|
||||
|
||||
public clearForInstance(centralInstanceId: string): void {
|
||||
DatabaseService.getInstance().getDb()
|
||||
.prepare(`DELETE FROM mesh_centrals WHERE central_instance_id = ?`)
|
||||
.run(centralInstanceId);
|
||||
}
|
||||
|
||||
public markUsed(centralInstanceId: string): void {
|
||||
DatabaseService.getInstance().getDb()
|
||||
.prepare(`UPDATE mesh_centrals SET last_used_at = ? WHERE central_instance_id = ?`)
|
||||
.run(Date.now(), centralInstanceId);
|
||||
}
|
||||
|
||||
public markRejected(centralInstanceId: string, reason: string): void {
|
||||
DatabaseService.getInstance().getDb()
|
||||
.prepare(`UPDATE mesh_centrals SET last_rejected_at = ?, last_reject_reason = ? WHERE central_instance_id = ?`)
|
||||
.run(Date.now(), reason, centralInstanceId);
|
||||
}
|
||||
}
|
||||
@@ -1,9 +1,12 @@
|
||||
import { EventEmitter } from 'events';
|
||||
import { createHash } from 'crypto';
|
||||
import jwt from 'jsonwebtoken';
|
||||
import WebSocket from 'ws';
|
||||
import { MAX_FRAME_SIZE_BYTES } from '../pilot/protocol';
|
||||
import { PilotTunnelBridge, type MeshTunnelHandle } from './PilotTunnelBridge';
|
||||
import { PilotTunnelManager } from './PilotTunnelManager';
|
||||
import { NodeRegistry } from './NodeRegistry';
|
||||
import { DatabaseService } from './DatabaseService';
|
||||
import { redactSensitiveText, sanitizeForLog } from '../utils/safeLog';
|
||||
import { httpUrlToWs } from '../utils/wsUrl';
|
||||
import { isDebugEnabled } from '../utils/debug';
|
||||
@@ -46,6 +49,31 @@ export type DialFailureCode =
|
||||
| 'tls_failed'
|
||||
| 'network_error';
|
||||
|
||||
/**
|
||||
* Reason union for `proxy-bridge-down` events. Centralizes the categories
|
||||
* the dialer emits so subscribers (reactive redial, metrics, UI) can branch
|
||||
* deterministically without parsing free-form strings.
|
||||
* - `idle`: idle sweeper closed the bridge after no streams for `idleTtlMs`.
|
||||
* - `remote_closed`: peer closed the WS cleanly (1000/1001) or with an
|
||||
* unclassified non-error code.
|
||||
* - `network_error`: WS dropped abnormally (1006).
|
||||
* - `protocol_error`: WS closed for protocol/policy reasons (1007/1008/1009).
|
||||
* - `auth_failed`: dial-time auth rejection (terminal, no redial).
|
||||
*/
|
||||
export type BridgeDownReason =
|
||||
| 'idle'
|
||||
| 'remote_closed'
|
||||
| 'network_error'
|
||||
| 'protocol_error'
|
||||
| 'auth_failed';
|
||||
|
||||
function classifyCloseCode(code?: number): BridgeDownReason {
|
||||
if (code === 1000 || code === 1001) return 'remote_closed';
|
||||
if (code === 1006) return 'network_error';
|
||||
if (code === 1007 || code === 1008 || code === 1009) return 'protocol_error';
|
||||
return 'remote_closed';
|
||||
}
|
||||
|
||||
/**
|
||||
* Activity-log reason. Wider than `DialFailureCode` because some failures
|
||||
* share a wire-level code (e.g., `network_error`) but deserve a more
|
||||
@@ -76,12 +104,20 @@ export class MeshProxyTunnelDialer extends EventEmitter {
|
||||
private readonly inflight = new Map<number, Promise<MeshTunnelHandle | null>>();
|
||||
private readonly idleSince = new Map<number, number>();
|
||||
private readonly recentFailures = new Map<number, DialFailure>();
|
||||
private readonly redialAttempts = new Map<number, number>();
|
||||
private readonly redialTimers = new Map<number, NodeJS.Timeout>();
|
||||
private readonly idleTtlMs: number;
|
||||
private idleCheckTimer: NodeJS.Timeout | null = null;
|
||||
private stopped = false;
|
||||
|
||||
private constructor(idleTtlOverrideMs?: number) {
|
||||
super();
|
||||
// Reactive redial: any non-terminal teardown triggers a backoff redial.
|
||||
// `idle` is intentional and `auth_failed` is terminal, so both skip.
|
||||
this.on('proxy-bridge-down', (nodeId: number, reason: BridgeDownReason) => {
|
||||
if (reason === 'idle' || reason === 'auth_failed') return;
|
||||
this.scheduleReactiveRedial(nodeId);
|
||||
});
|
||||
if (typeof idleTtlOverrideMs === 'number') {
|
||||
this.idleTtlMs = idleTtlOverrideMs;
|
||||
} else {
|
||||
@@ -163,6 +199,9 @@ export class MeshProxyTunnelDialer extends EventEmitter {
|
||||
this.bridges.delete(nodeId);
|
||||
try { bridge.close(1000, 'dialer shutdown'); } catch { /* ignore */ }
|
||||
}
|
||||
for (const t of this.redialTimers.values()) clearTimeout(t);
|
||||
this.redialTimers.clear();
|
||||
this.redialAttempts.clear();
|
||||
this.idleSince.clear();
|
||||
this.recentFailures.clear();
|
||||
this.inflight.clear();
|
||||
@@ -223,6 +262,18 @@ export class MeshProxyTunnelDialer extends EventEmitter {
|
||||
return null;
|
||||
}
|
||||
|
||||
// Capability-gated mesh_handshake: pushes a signed `mesh_tunnel` JWT
|
||||
// and the central origin so the peer can dial central back for
|
||||
// reverse cross-fleet TCP streams (R2). Skipped silently when the
|
||||
// peer lacks the capability or `SENCHO_PRIMARY_URL` is unset so
|
||||
// legacy peers keep their forward-only TCP path untouched.
|
||||
await this.maybeSendBootstrap(nodeId, ws);
|
||||
|
||||
if (this.stopped) {
|
||||
try { ws.close(1001, 'dialer shutdown'); } catch { /* ignore */ }
|
||||
return null;
|
||||
}
|
||||
|
||||
const bridge = new PilotTunnelBridge(nodeId, ws);
|
||||
try {
|
||||
await bridge.start();
|
||||
@@ -249,17 +300,13 @@ export class MeshProxyTunnelDialer extends EventEmitter {
|
||||
// Mirror the registration in the dialer's own map so the idle
|
||||
// sweeper can call `getActiveStreamCount()` (not on the narrow
|
||||
// MeshTunnelHandle interface) without poking into the manager.
|
||||
bridge.once('closed', () => {
|
||||
if (this.bridges.get(nodeId) === bridge) {
|
||||
this.bridges.delete(nodeId);
|
||||
this.idleSince.delete(nodeId);
|
||||
void this.logActivity(nodeId, 'close', { reason: 'remote-closed' });
|
||||
this.emit('proxy-bridge-down', nodeId);
|
||||
}
|
||||
});
|
||||
this.bridges.set(nodeId, bridge);
|
||||
this.attachBridgeCloseListener(nodeId, bridge);
|
||||
this.idleSince.set(nodeId, Date.now());
|
||||
this.recentFailures.delete(nodeId);
|
||||
// A successful open clears any prior reactive-redial backoff so the
|
||||
// next failure starts fresh from attempt #1.
|
||||
this.redialAttempts.delete(nodeId);
|
||||
void this.logActivity(nodeId, 'open.ok', {});
|
||||
this.emit('proxy-bridge-up', nodeId);
|
||||
if (isDebugEnabled()) {
|
||||
@@ -326,12 +373,7 @@ export class MeshProxyTunnelDialer extends EventEmitter {
|
||||
}
|
||||
const last = this.idleSince.get(nodeId) ?? now;
|
||||
if (now - last >= this.idleTtlMs) {
|
||||
this.bridges.delete(nodeId);
|
||||
this.idleSince.delete(nodeId);
|
||||
try { bridge.close(1000, 'idle timeout'); } catch { /* ignore */ }
|
||||
PilotMetrics.increment('proxy_idle_closes');
|
||||
void this.logActivity(nodeId, 'close', { reason: 'idle' });
|
||||
this.emit('proxy-bridge-down', nodeId);
|
||||
this.tearDownBridge(nodeId, 'idle', { code: 1000, message: 'idle timeout' });
|
||||
if (isDebugEnabled()) {
|
||||
console.log(`[MeshProxyDialer:diag] idle close node=${nodeId} idleMs=${now - last}`);
|
||||
}
|
||||
@@ -339,6 +381,117 @@ export class MeshProxyTunnelDialer extends EventEmitter {
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Single teardown path for any bridge close. Removes the bridge from
|
||||
* the map, optionally invokes `bridge.close()` (skipped when the close
|
||||
* originated from the bridge itself), records the activity log entry,
|
||||
* bumps the idle-close metric when applicable, and emits a reason-tagged
|
||||
* `proxy-bridge-down`. The self-listener installed in the constructor
|
||||
* decides whether to schedule a reactive redial.
|
||||
*/
|
||||
private tearDownBridge(
|
||||
nodeId: number,
|
||||
reason: BridgeDownReason,
|
||||
closeArgs?: { code: number; message: string },
|
||||
): void {
|
||||
const bridge = this.bridges.get(nodeId);
|
||||
if (!bridge) return;
|
||||
this.bridges.delete(nodeId);
|
||||
this.idleSince.delete(nodeId);
|
||||
if (closeArgs) {
|
||||
// Best-effort: closing an already-closed WS is idempotent and
|
||||
// can throw on edge states; we never want teardown to propagate.
|
||||
try { bridge.close(closeArgs.code, closeArgs.message); } catch { /* best-effort */ }
|
||||
}
|
||||
if (reason === 'idle') PilotMetrics.increment('proxy_idle_closes');
|
||||
void this.logActivity(nodeId, 'close', { reason });
|
||||
this.emit('proxy-bridge-down', nodeId, reason);
|
||||
}
|
||||
|
||||
/**
|
||||
* Wire the `closed` listener that classifies the WS close code and
|
||||
* routes through `tearDownBridge` without re-closing the bridge. Called
|
||||
* from `dial()` after registration; also reachable from tests via a
|
||||
* private cast so they can exercise the close-code mapping without
|
||||
* spinning up a real WebSocket.
|
||||
*/
|
||||
private attachBridgeCloseListener(nodeId: number, bridge: EventEmitter): void {
|
||||
bridge.once('closed', (info?: { code?: number }) => {
|
||||
if (this.bridges.get(nodeId) !== bridge) return;
|
||||
const reason = classifyCloseCode(info?.code);
|
||||
this.tearDownBridge(nodeId, reason);
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Schedule a reactive redial with exponential backoff + jitter. Caps at
|
||||
* 8 attempts (~5 minutes between the last few). A successful `dial()`
|
||||
* clears `redialAttempts[nodeId]`, so the counter only grows during a
|
||||
* sustained outage.
|
||||
*/
|
||||
private scheduleReactiveRedial(nodeId: number): void {
|
||||
if (this.stopped) return;
|
||||
const attempt = (this.redialAttempts.get(nodeId) ?? 0) + 1;
|
||||
if (attempt > 8) {
|
||||
this.redialAttempts.delete(nodeId);
|
||||
void this.logActivity(nodeId, 'open.fail', { reason: 'redial_exhausted', attempts: 8 });
|
||||
return;
|
||||
}
|
||||
this.redialAttempts.set(nodeId, attempt);
|
||||
const baseMs = Math.min(5_000 * 2 ** (attempt - 1), 5 * 60_000);
|
||||
const jitter = Math.floor(Math.random() * baseMs * 0.3);
|
||||
const delay = baseMs + jitter;
|
||||
const existing = this.redialTimers.get(nodeId);
|
||||
if (existing) clearTimeout(existing);
|
||||
const timer = setTimeout(() => {
|
||||
this.redialTimers.delete(nodeId);
|
||||
void this.ensureBridge(nodeId);
|
||||
}, delay);
|
||||
timer.unref?.();
|
||||
this.redialTimers.set(nodeId, timer);
|
||||
}
|
||||
|
||||
/**
|
||||
* If the peer advertises `mesh_proxy_callback_bootstrap` and central has
|
||||
* a canonical origin (`SENCHO_PRIMARY_URL`), mint a `mesh_tunnel`-scoped
|
||||
* JWT bound to the peer's `api_token` fingerprint and push it as the
|
||||
* first text frame on the freshly-opened WS. The peer's
|
||||
* `meshProxyTunnel` first-frame state machine persists the bootstrap
|
||||
* material before yielding to TCP traffic. Every other path is a silent
|
||||
* skip so legacy peers continue to receive raw TCP frames immediately.
|
||||
*/
|
||||
private async maybeSendBootstrap(nodeId: number, ws: WebSocket): Promise<void> {
|
||||
const canonicalOrigin = (process.env.SENCHO_PRIMARY_URL ?? '').replace(/\/+$/, '');
|
||||
if (!canonicalOrigin) return;
|
||||
|
||||
let meta;
|
||||
try { meta = await NodeRegistry.getInstance().fetchMetaForNode(nodeId); }
|
||||
catch { return; }
|
||||
if (!meta?.online) return;
|
||||
if (!meta.capabilities?.includes('mesh_proxy_callback_bootstrap')) return;
|
||||
|
||||
const db = DatabaseService.getInstance();
|
||||
const node = db.getNode(nodeId);
|
||||
if (!node || node.type !== 'remote' || node.mode !== 'proxy' || !node.api_token) return;
|
||||
|
||||
const authSecret = db.getGlobalSettings().auth_jwt_secret;
|
||||
const centralInstanceId = db.getSystemState('instance_id');
|
||||
if (!authSecret || !centralInstanceId) return;
|
||||
|
||||
const built = buildHandshakeFrame(nodeId, node.api_token, canonicalOrigin, centralInstanceId, authSecret);
|
||||
try {
|
||||
ws.send(JSON.stringify(built.frame));
|
||||
void this.logActivity(nodeId, 'open.ok', {
|
||||
bootstrapSent: true,
|
||||
centralApiUrl: redactSensitiveText(canonicalOrigin),
|
||||
kid: 'v1',
|
||||
jwtExpiresAt: built.expSec,
|
||||
});
|
||||
} catch (err) {
|
||||
console.warn(`[MeshProxyDialer] mesh_handshake send failed for node ${nodeId}: ${(err as Error).message}`.replace(/[\n\r]/g, ''));
|
||||
}
|
||||
}
|
||||
|
||||
private async logActivity(
|
||||
nodeId: number,
|
||||
event: ProxyTunnelEvent,
|
||||
@@ -364,6 +517,58 @@ export class MeshProxyTunnelDialer extends EventEmitter {
|
||||
}
|
||||
}
|
||||
|
||||
interface HandshakeFrame {
|
||||
t: 'mesh_handshake';
|
||||
v: 1;
|
||||
peerNodeId: number;
|
||||
centralInstanceId: string;
|
||||
centralApiUrl: string;
|
||||
meshTunnelJwt: string;
|
||||
jwtExpiresAt: number;
|
||||
}
|
||||
|
||||
/**
|
||||
* Build the mesh_handshake frame and the associated JWT. Pure helper so the
|
||||
* dialer's `maybeSendBootstrap` stays under the 30-line ceiling.
|
||||
*/
|
||||
function buildHandshakeFrame(
|
||||
nodeId: number,
|
||||
apiToken: string,
|
||||
canonicalOrigin: string,
|
||||
centralInstanceId: string,
|
||||
authSecret: string,
|
||||
): { frame: HandshakeFrame; expSec: number } {
|
||||
const peerTokenFp = createHash('sha256').update(apiToken).digest('hex').slice(0, 16);
|
||||
const nowSec = Math.floor(Date.now() / 1000);
|
||||
const expSec = nowSec + 365 * 24 * 3600;
|
||||
const meshTunnelJwt = jwt.sign(
|
||||
{
|
||||
sub: String(nodeId),
|
||||
iss: centralInstanceId,
|
||||
aud: canonicalOrigin,
|
||||
scope: 'mesh_tunnel',
|
||||
iat: nowSec,
|
||||
exp: expSec,
|
||||
kid: 'v1',
|
||||
peer_token_fp: peerTokenFp,
|
||||
},
|
||||
authSecret,
|
||||
{ algorithm: 'HS256' },
|
||||
);
|
||||
return {
|
||||
frame: {
|
||||
t: 'mesh_handshake',
|
||||
v: 1,
|
||||
peerNodeId: nodeId,
|
||||
centralInstanceId,
|
||||
centralApiUrl: canonicalOrigin,
|
||||
meshTunnelJwt,
|
||||
jwtExpiresAt: expSec,
|
||||
},
|
||||
expSec,
|
||||
};
|
||||
}
|
||||
|
||||
function classifyDialError(err: unknown): { code: DialFailureCode; message: string } {
|
||||
const message = sanitizeForLog((err as Error).message || String(err));
|
||||
const httpStatus = (err as Error & { httpStatus?: number }).httpStatus;
|
||||
|
||||
@@ -64,7 +64,7 @@ export function getSenchoIpFromSubnet(subnet: string): string {
|
||||
export type MeshActivitySource = 'pilot' | 'mesh';
|
||||
export type MeshActivityLevel = 'info' | 'warn' | 'error';
|
||||
export type MeshActivityType =
|
||||
| 'route.dispatch' | 'route.resolve.ok' | 'route.resolve.denied'
|
||||
| 'route.dispatch' | 'route.resolve.ok' | 'route.resolve.denied' | 'route.resolve.fail'
|
||||
| 'tunnel.open' | 'tunnel.fail' | 'tunnel.backpressure'
|
||||
| 'opt_in' | 'opt_out'
|
||||
| 'mesh.enable' | 'mesh.disable'
|
||||
@@ -72,7 +72,8 @@ export type MeshActivityType =
|
||||
| 'probe.ok' | 'probe.fail'
|
||||
| 'forwarder.listen' | 'forwarder.unlisten' | 'forwarder.error'
|
||||
| 'proxy-tunnel.open.ok' | 'proxy-tunnel.open.fail' | 'proxy-tunnel.close'
|
||||
| 'mesh.proxy_tunnel.identify';
|
||||
| 'mesh.proxy_tunnel.identify'
|
||||
| 'mesh_handshake.received';
|
||||
|
||||
export interface MeshActivityEvent {
|
||||
ts: number;
|
||||
@@ -299,6 +300,8 @@ export class MeshService extends EventEmitter implements MeshForwarderHost {
|
||||
|
||||
this.selfCentralNodeId = this.resolveSelfCentralNodeId();
|
||||
|
||||
this.maybeWarnUnsetPrimaryUrl();
|
||||
|
||||
await this.setupMeshNetwork();
|
||||
try {
|
||||
await this.refreshAliasCache();
|
||||
@@ -317,6 +320,10 @@ export class MeshService extends EventEmitter implements MeshForwarderHost {
|
||||
});
|
||||
}
|
||||
await this.regenerateAllOverrides();
|
||||
// Trigger 3: proactively re-establish central->peer bridges so any
|
||||
// peer that is online and capable receives bootstrap material on
|
||||
// startup. Fire-and-forget so start() does not block on remote I/O.
|
||||
void this.proactiveBootstrapFanout();
|
||||
this.aliasRefreshTimer = setInterval(() => {
|
||||
void (async () => {
|
||||
try {
|
||||
@@ -345,6 +352,71 @@ export class MeshService extends EventEmitter implements MeshForwarderHost {
|
||||
await this.forwarder.shutdown();
|
||||
}
|
||||
|
||||
/**
|
||||
* Operator-facing preflight: if `SENCHO_PRIMARY_URL` is unset at
|
||||
* startup and the central has at least one mesh-enabled proxy-mode
|
||||
* node, warn that the Task 8 capability-gated handshake will fall
|
||||
* back to an inferred origin for the callback bootstrap. The matching
|
||||
* fail-safe in `MeshProxyTunnelDialer` silently skips the bootstrap
|
||||
* when the env is unset; this warn makes the consequence visible.
|
||||
*/
|
||||
private maybeWarnUnsetPrimaryUrl(): void {
|
||||
if (process.env.SENCHO_PRIMARY_URL) return;
|
||||
const row = DatabaseService.getInstance().getDb().prepare(`
|
||||
SELECT COUNT(*) as n FROM nodes
|
||||
WHERE type='remote' AND mode='proxy' AND mesh_enabled=1
|
||||
`).get() as { n: number };
|
||||
if (row.n > 0) {
|
||||
console.warn(
|
||||
'[Mesh] SENCHO_PRIMARY_URL is unset; mesh callback bootstrap will use ' +
|
||||
'inferred origin. Set SENCHO_PRIMARY_URL to make peer-initiated mesh ' +
|
||||
'callbacks robust against reverse-proxy edge cases.'
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Trigger 3: at central startup, walk every mesh-enabled proxy-mode
|
||||
* peer that already has at least one `mesh_stacks` row and call
|
||||
* `MeshProxyTunnelDialer.ensureBridge(nodeId)`. The dialer's
|
||||
* capability-gated handshake then mints and ships bootstrap material
|
||||
* to peers that just came online or just got upgraded to a build that
|
||||
* advertises `mesh_proxy_callback_bootstrap`. Bounded concurrency 4
|
||||
* with a 250ms stagger; failures are logged and never abort the
|
||||
* fan-out.
|
||||
*/
|
||||
private async proactiveBootstrapFanout(): Promise<void> {
|
||||
const rows = DatabaseService.getInstance().getDb().prepare(`
|
||||
SELECT DISTINCT n.id AS id
|
||||
FROM nodes n
|
||||
INNER JOIN mesh_stacks ms ON ms.node_id = n.id
|
||||
WHERE n.type = 'remote' AND n.mode = 'proxy' AND n.mesh_enabled = 1
|
||||
ORDER BY n.id
|
||||
`).all() as Array<{ id: number }>;
|
||||
|
||||
const queue = rows.map((r) => r.id);
|
||||
const dialer = MeshProxyTunnelDialer.getInstance();
|
||||
const worker = async (): Promise<void> => {
|
||||
for (;;) {
|
||||
const nodeId = queue.shift();
|
||||
if (nodeId === undefined) return;
|
||||
try {
|
||||
await dialer.ensureBridge(nodeId);
|
||||
} catch (err) {
|
||||
this.logActivity({
|
||||
source: 'mesh', level: 'warn', type: 'proxy-tunnel.open.fail',
|
||||
nodeId,
|
||||
message: `boot proactive bootstrap failed: ${sanitizeForLog((err as Error).message)}`,
|
||||
details: { trigger: 'startup_fanout' },
|
||||
});
|
||||
}
|
||||
await new Promise((r) => setTimeout(r, 250));
|
||||
}
|
||||
};
|
||||
const workerCount = Math.min(4, Math.max(1, queue.length));
|
||||
await Promise.all(Array.from({ length: workerCount }, () => worker()));
|
||||
}
|
||||
|
||||
public getSenchoIp(): string | null {
|
||||
return this.senchoIp;
|
||||
}
|
||||
@@ -653,6 +725,17 @@ export class MeshService extends EventEmitter implements MeshForwarderHost {
|
||||
source: 'mesh', level: 'info', type: 'mesh.enable',
|
||||
nodeId, message: `mesh enabled on node ${nodeId}`,
|
||||
});
|
||||
// Trigger 1: proactive bootstrap. If the peer is a proxy-mode remote,
|
||||
// dial the mesh callback bridge immediately so the capability-gated
|
||||
// handshake can ship `mesh_handshake` material without waiting for
|
||||
// the next forwarder dial. Fire-and-forget; failures are logged by
|
||||
// the dialer.
|
||||
const node = DatabaseService.getInstance().getNode(nodeId);
|
||||
if (node && node.type === 'remote' && node.mode === 'proxy') {
|
||||
void MeshProxyTunnelDialer.getInstance().ensureBridge(nodeId).catch((err) => {
|
||||
console.warn(`[Mesh] proactive bootstrap on mesh-enable failed for node ${nodeId}: ${(err as Error).message}`);
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
public async disableForNode(nodeId: number): Promise<void> {
|
||||
@@ -1506,6 +1589,39 @@ export class MeshService extends EventEmitter implements MeshForwarderHost {
|
||||
message: `cross-node dispatch to ${target.alias} on node ${target.nodeId}`,
|
||||
});
|
||||
|
||||
// Peer-side recovery: when no reverseDialer is installed, kick the
|
||||
// symmetric-dial path so central learns about this peer. The
|
||||
// proactive back-dial from central (registered as a side effect via
|
||||
// the proxy-tunnel WS upgrade) is what actually installs the
|
||||
// reverseDialer on this peer. If the session cannot be established
|
||||
// (cache miss, central down, auth rejected), log route.resolve.fail
|
||||
// and drop the inbound socket.
|
||||
if (!this.reverseDialer) {
|
||||
try {
|
||||
const { PeerToCentralMeshSessionDialer } = await import('./PeerToCentralMeshSessionDialer');
|
||||
const session = await PeerToCentralMeshSessionDialer.getInstance().ensureSession();
|
||||
if (!session) {
|
||||
this.logActivity({
|
||||
source: 'mesh', level: 'warn', type: 'route.resolve.fail',
|
||||
nodeId: target.nodeId, alias: target.alias,
|
||||
message: `peer cross-node dispatch failed: no central callback session available`,
|
||||
details: { direction: 'forward-from-peer', reason: 'no_session' },
|
||||
});
|
||||
try { src.destroy(); } catch { /* ignore */ }
|
||||
return;
|
||||
}
|
||||
} catch (err) {
|
||||
this.logActivity({
|
||||
source: 'mesh', level: 'warn', type: 'route.resolve.fail',
|
||||
nodeId: target.nodeId, alias: target.alias,
|
||||
message: `peer cross-node dispatch failed: bootstrap threw`,
|
||||
details: { direction: 'forward-from-peer', reason: 'bootstrap_threw' },
|
||||
});
|
||||
try { src.destroy(); } catch { /* ignore */ }
|
||||
return;
|
||||
}
|
||||
}
|
||||
|
||||
const tcpStream = await this.dialMeshTcpStream(target);
|
||||
if (!tcpStream) {
|
||||
this.logActivity({
|
||||
|
||||
@@ -0,0 +1,184 @@
|
||||
/**
|
||||
* PeerToCentralMeshSessionDialer: peer-side counterpart to the central
|
||||
* ingress at `/api/mesh/proxy-tunnel-from-peer`. Reads cached central
|
||||
* material from `MeshCentralRegistry`, opens a WebSocket to central
|
||||
* carrying the bootstrapped JWT in the Authorization header, and on a
|
||||
* successful upgrade installs a local `PilotTunnelBridge` whose loopback
|
||||
* URL the local proxy can target as if central had dialed us.
|
||||
*
|
||||
* Failure handling branches on the central's machine-readable reason
|
||||
* code returned in the 401 body: terminal codes (stale,
|
||||
* signature_invalid, fingerprint, node_deleted, instance_mismatch,
|
||||
* audience_mismatch) clear the cache so we stop dialing with a known-bad
|
||||
* credential. Transient codes (clock_skew, mode_mismatch) keep the cache
|
||||
* but record the rejection so an operator can see why we stalled.
|
||||
* HTTP 404 (older central without the peer ingress endpoint) keeps the
|
||||
* cache and arms a longer backoff window so we do not hammer the remote.
|
||||
*
|
||||
* Process-local rate limit: at most 5 dial attempts per 60s window.
|
||||
* Concurrency: a single in-flight promise is shared across callers so
|
||||
* parallel `ensureSession()` calls coalesce into one dial.
|
||||
*/
|
||||
import { EventEmitter } from 'events';
|
||||
import WebSocket from 'ws';
|
||||
import { MAX_FRAME_SIZE_BYTES } from '../pilot/protocol';
|
||||
import { MeshCentralRegistry } from './MeshCentralRegistry';
|
||||
import { PilotTunnelBridge } from './PilotTunnelBridge';
|
||||
import { PilotMetrics } from './PilotMetrics';
|
||||
import { httpUrlToWs } from '../utils/wsUrl';
|
||||
import { sanitizeForLog } from '../utils/safeLog';
|
||||
|
||||
const HANDSHAKE_TIMEOUT_MS = 15_000;
|
||||
const RATE_LIMIT_WINDOW_MS = 60_000;
|
||||
const RATE_LIMIT_MAX = 5;
|
||||
const ENDPOINT_UNAVAILABLE_BACKOFF_MS = 5 * 60_000;
|
||||
const CALLBACK_PATH = '/api/mesh/proxy-tunnel-from-peer';
|
||||
|
||||
/**
|
||||
* Reasons that indicate the cached central material is permanently bad
|
||||
* (wrong secret, wrong fingerprint, node removed on central, etc.) and
|
||||
* should be evicted so the next mesh-enable cycle remints fresh
|
||||
* material. `endpoint_not_found` is explicitly NOT in this set because
|
||||
* a 404 typically means "central is older than the peer ingress",
|
||||
* which is recoverable by waiting.
|
||||
*/
|
||||
const REJECT_REASONS_CLEAR = new Set([
|
||||
'stale',
|
||||
'token_fingerprint_mismatch',
|
||||
'node_deleted',
|
||||
'instance_mismatch',
|
||||
'signature_invalid',
|
||||
'audience_mismatch',
|
||||
'unknown',
|
||||
]);
|
||||
|
||||
interface DialError extends Error {
|
||||
reason?: string;
|
||||
httpStatus?: number;
|
||||
}
|
||||
|
||||
export class PeerToCentralMeshSessionDialer extends EventEmitter {
|
||||
private static instance: PeerToCentralMeshSessionDialer | null = null;
|
||||
private currentSession: PilotTunnelBridge | null = null;
|
||||
private inflight: Promise<PilotTunnelBridge | null> | null = null;
|
||||
private recentDials: number[] = [];
|
||||
private endpointUnavailableUntil = 0;
|
||||
|
||||
private constructor() { super(); }
|
||||
|
||||
public static getInstance(): PeerToCentralMeshSessionDialer {
|
||||
if (!this.instance) this.instance = new PeerToCentralMeshSessionDialer();
|
||||
return this.instance;
|
||||
}
|
||||
|
||||
public static resetForTest(): void {
|
||||
if (this.instance) {
|
||||
try { this.instance.currentSession?.close(1000, 'test reset'); } catch { /* ignore */ }
|
||||
}
|
||||
this.instance = null;
|
||||
}
|
||||
|
||||
public hasSession(): boolean {
|
||||
return this.currentSession !== null;
|
||||
}
|
||||
|
||||
public async ensureSession(): Promise<PilotTunnelBridge | null> {
|
||||
if (this.currentSession) return this.currentSession;
|
||||
if (Date.now() < this.endpointUnavailableUntil) return null;
|
||||
if (this.isRateLimited()) return null;
|
||||
if (this.inflight) return this.inflight;
|
||||
this.inflight = this.dial().finally(() => { this.inflight = null; });
|
||||
return this.inflight;
|
||||
}
|
||||
|
||||
private isRateLimited(): boolean {
|
||||
const now = Date.now();
|
||||
this.recentDials = this.recentDials.filter((ts) => now - ts < RATE_LIMIT_WINDOW_MS);
|
||||
return this.recentDials.length >= RATE_LIMIT_MAX;
|
||||
}
|
||||
|
||||
private async dial(): Promise<PilotTunnelBridge | null> {
|
||||
const material = MeshCentralRegistry.getInstance().getActive();
|
||||
if (!material) return null;
|
||||
this.recentDials.push(Date.now());
|
||||
const wsUrl = httpUrlToWs(material.centralApiUrl) + CALLBACK_PATH;
|
||||
const ws = new WebSocket(wsUrl, {
|
||||
headers: { Authorization: `Bearer ${material.callbackJwt}` },
|
||||
handshakeTimeout: HANDSHAKE_TIMEOUT_MS,
|
||||
maxPayload: MAX_FRAME_SIZE_BYTES,
|
||||
});
|
||||
try {
|
||||
await this.awaitOpen(ws);
|
||||
} catch (err) {
|
||||
ws.on('error', () => { /* swallow tail */ });
|
||||
try { ws.close(); } catch { /* ignore */ }
|
||||
this.handleDialFailure(err, material.centralInstanceId);
|
||||
return null;
|
||||
}
|
||||
return this.attachBridge(ws, material.centralInstanceId);
|
||||
}
|
||||
|
||||
private async attachBridge(ws: WebSocket, instanceId: string): Promise<PilotTunnelBridge | null> {
|
||||
const bridge = new PilotTunnelBridge(0, ws);
|
||||
try {
|
||||
await bridge.start();
|
||||
} catch {
|
||||
try { bridge.close(1011, 'bridge start failed'); } catch { /* ignore */ }
|
||||
PilotMetrics.increment('mesh_callback_dials_failed_total');
|
||||
return null;
|
||||
}
|
||||
bridge.once('closed', () => {
|
||||
if (this.currentSession === bridge) this.currentSession = null;
|
||||
});
|
||||
this.currentSession = bridge;
|
||||
PilotMetrics.increment('mesh_central_bootstraps_total');
|
||||
MeshCentralRegistry.getInstance().markUsed(instanceId);
|
||||
return bridge;
|
||||
}
|
||||
|
||||
private awaitOpen(ws: WebSocket): Promise<void> {
|
||||
return new Promise<void>((resolve, reject) => {
|
||||
const cleanup = (): void => {
|
||||
ws.removeAllListeners('open');
|
||||
ws.removeAllListeners('error');
|
||||
ws.removeAllListeners('unexpected-response');
|
||||
};
|
||||
ws.once('open', () => { cleanup(); resolve(); });
|
||||
ws.once('error', (err) => { cleanup(); reject(err); });
|
||||
ws.once('unexpected-response', (_req, res) => {
|
||||
cleanup();
|
||||
let body = '';
|
||||
res.on('data', (chunk: Buffer) => { body += chunk.toString('utf8'); });
|
||||
res.on('end', () => reject(this.buildUpgradeError(res.statusCode, body)));
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
private buildUpgradeError(statusCode: number | undefined, body: string): DialError {
|
||||
let reason = 'unknown';
|
||||
try {
|
||||
const parsed = JSON.parse(body) as { reason?: unknown };
|
||||
if (typeof parsed.reason === 'string') reason = parsed.reason;
|
||||
} catch { /* keep unknown */ }
|
||||
if (statusCode === 404) reason = 'endpoint_not_found';
|
||||
const err = new Error(`upgrade rejected: HTTP ${statusCode ?? 'unknown'} reason=${reason}`) as DialError;
|
||||
err.reason = reason;
|
||||
err.httpStatus = statusCode;
|
||||
return err;
|
||||
}
|
||||
|
||||
private handleDialFailure(err: unknown, instanceId: string): void {
|
||||
const reason = (err as DialError).reason ?? 'unknown';
|
||||
PilotMetrics.increment('mesh_callback_dials_failed_total');
|
||||
if (REJECT_REASONS_CLEAR.has(reason)) {
|
||||
MeshCentralRegistry.getInstance().clearForInstance(instanceId);
|
||||
PilotMetrics.increment('mesh_callback_auth_failures_total');
|
||||
} else {
|
||||
MeshCentralRegistry.getInstance().markRejected(instanceId, reason);
|
||||
if (reason === 'endpoint_not_found') {
|
||||
this.endpointUnavailableUntil = Date.now() + ENDPOINT_UNAVAILABLE_BACKOFF_MS;
|
||||
}
|
||||
}
|
||||
console.warn(`[PeerToCentralMeshSessionDialer] dial failed: ${sanitizeForLog(`reason=${reason}`)}`);
|
||||
}
|
||||
}
|
||||
@@ -22,6 +22,14 @@ interface Counters {
|
||||
proxy_dials_failed: number;
|
||||
/** Proxy-tunnel teardowns initiated by the dialer's idle sweep (zero active streams for the configured TTL). */
|
||||
proxy_idle_closes: number;
|
||||
/** Proxy bridges registered via the peer-initiated dial-back path (`/api/mesh/proxy-tunnel-from-peer`). Disjoint from `proxy_bridges_total`, which counts central-initiated dials. */
|
||||
proxy_bridges_peer_initiated_total: number;
|
||||
/** Successful peer-to-central dial-back sessions (counted on WS open + bridge start). Disjoint from `proxy_bridges_peer_initiated_total`, which is incremented by the central-side ingress at register time. Useful for operators investigating asymmetric counts (peer thinks it dialed, central never registered). */
|
||||
mesh_central_bootstraps_total: number;
|
||||
/** Peer-to-central dial-back attempts that did not complete a WS open. Covers transport errors, upgrade rejections, and bridge.start failures. */
|
||||
mesh_callback_dials_failed_total: number;
|
||||
/** Subset of `mesh_callback_dials_failed_total` where central responded 401 with a terminal reason code (stale, signature_invalid, ...) that caused the peer to clear its cached central material. */
|
||||
mesh_callback_auth_failures_total: number;
|
||||
}
|
||||
|
||||
class PilotMetricsImpl {
|
||||
@@ -34,6 +42,10 @@ class PilotMetricsImpl {
|
||||
proxy_bridges_total: 0,
|
||||
proxy_dials_failed: 0,
|
||||
proxy_idle_closes: 0,
|
||||
proxy_bridges_peer_initiated_total: 0,
|
||||
mesh_central_bootstraps_total: 0,
|
||||
mesh_callback_dials_failed_total: 0,
|
||||
mesh_callback_auth_failures_total: 0,
|
||||
};
|
||||
|
||||
public increment<K extends keyof Counters>(name: K): void {
|
||||
|
||||
@@ -149,6 +149,7 @@ export interface MeshTunnelHandle {
|
||||
* stripping/injection, and license-tier propagation all work unchanged.
|
||||
*/
|
||||
export class PilotTunnelBridge extends EventEmitter implements MeshTunnelHandle {
|
||||
private readonly nodeId: number;
|
||||
private readonly tunnelWs: WebSocket;
|
||||
private readonly loopback: HttpServer;
|
||||
private readonly wsUpgradeServer: WebSocketServer;
|
||||
@@ -162,8 +163,9 @@ export class PilotTunnelBridge extends EventEmitter implements MeshTunnelHandle
|
||||
private drainTimer?: NodeJS.Timeout;
|
||||
private closed = false;
|
||||
|
||||
constructor(_nodeId: number, tunnelWs: WebSocket) {
|
||||
constructor(nodeId: number, tunnelWs: WebSocket) {
|
||||
super();
|
||||
this.nodeId = nodeId;
|
||||
this.tunnelWs = tunnelWs;
|
||||
this.loopback = http.createServer();
|
||||
this.wsUpgradeServer = new WebSocketServer({ noServer: true });
|
||||
@@ -782,8 +784,26 @@ export class PilotTunnelBridge extends EventEmitter implements MeshTunnelHandle
|
||||
|
||||
private async acceptReverseLocal(s: number, target: { stack: string; service: string; port: number }): Promise<void> {
|
||||
const { MeshService } = await import('./MeshService');
|
||||
const ip = await MeshService.getInstance().resolveContainerIp({ stack: target.stack, service: target.service });
|
||||
const meshSvc = MeshService.getInstance();
|
||||
// Shared discriminator + target metadata so the Routing tab can
|
||||
// separate peer-to-central (reverse) dispatch from central-to-peer
|
||||
// (forward) dispatch when both flow through the same activity feed.
|
||||
const baseDetails = {
|
||||
direction: 'reverse' as const,
|
||||
streamId: s,
|
||||
targetStack: target.stack,
|
||||
targetService: target.service,
|
||||
targetPort: target.port,
|
||||
peerNodeId: this.nodeId,
|
||||
};
|
||||
const ip = await meshSvc.resolveContainerIp({ stack: target.stack, service: target.service });
|
||||
if (!ip) {
|
||||
meshSvc.logActivity({
|
||||
source: 'mesh', level: 'error', type: 'route.resolve.fail',
|
||||
nodeId: this.nodeId,
|
||||
message: `reverse dial failed: container ${target.stack}/${target.service} not found`,
|
||||
details: { ...baseDetails, reason: 'container_not_found' },
|
||||
});
|
||||
this.sendJson({ t: 'tcp_open_ack', s, ok: false, err: 'no_target' });
|
||||
return;
|
||||
}
|
||||
@@ -802,14 +822,26 @@ export class PilotTunnelBridge extends EventEmitter implements MeshTunnelHandle
|
||||
// Pre-connect failure: ack-fail and drop. The handler is removed in
|
||||
// 'connect' below so post-connect errors fall through to the
|
||||
// mid-stream teardown path instead of double-firing.
|
||||
const onPreConnectError = () => {
|
||||
const onPreConnectError = (err?: Error) => {
|
||||
if (!this.streams.has(s)) return;
|
||||
this.streams.delete(s);
|
||||
meshSvc.logActivity({
|
||||
source: 'mesh', level: 'error', type: 'route.resolve.fail',
|
||||
nodeId: this.nodeId,
|
||||
message: `reverse dial failed pre-connect: ${err?.message ?? 'socket error'}`,
|
||||
details: { ...baseDetails, reason: 'connect_error' },
|
||||
});
|
||||
this.sendJson({ t: 'tcp_open_ack', s, ok: false, err: 'unreachable' });
|
||||
};
|
||||
socket.once('error', onPreConnectError);
|
||||
socket.once('connect', () => {
|
||||
socket.off('error', onPreConnectError);
|
||||
meshSvc.logActivity({
|
||||
source: 'mesh', level: 'info', type: 'route.resolve.ok',
|
||||
nodeId: this.nodeId,
|
||||
message: `reverse dial ok: ${target.stack}/${target.service}:${target.port}`,
|
||||
details: baseDetails,
|
||||
});
|
||||
this.sendJson({ t: 'tcp_open_ack', s, ok: true });
|
||||
socket.on('data', (chunk: Buffer) => {
|
||||
const cur = this.streams.get(s);
|
||||
|
||||
@@ -68,6 +68,15 @@ export class PilotTunnelCapacityError extends Error {
|
||||
export class PilotTunnelManager extends EventEmitter {
|
||||
private static instance: PilotTunnelManager;
|
||||
private bridges: Map<number, PilotTunnelBridge> = new Map();
|
||||
/**
|
||||
* Parallel kind index for the `bridges` map. `'pilot'` is set by
|
||||
* `registerTunnel` (agent-initiated long-lived tunnel); `'proxy'` is
|
||||
* set by `registerProxyBridge` and `replaceOrRegisterProxyBridge`
|
||||
* (central-initiated short-lived bridge). Used by
|
||||
* `replaceOrRegisterProxyBridge` so a peer-initiated dial can supersede
|
||||
* a previous proxy bridge but never shadow a live pilot tunnel.
|
||||
*/
|
||||
private bridgeKinds: Map<number, 'pilot' | 'proxy'> = new Map();
|
||||
private softWarned = false;
|
||||
|
||||
private constructor() {
|
||||
@@ -82,6 +91,31 @@ export class PilotTunnelManager extends EventEmitter {
|
||||
return PilotTunnelManager.instance;
|
||||
}
|
||||
|
||||
/**
|
||||
* Test-only: drop the singleton and any held bridges so every test
|
||||
* starts from a clean registry. Closes outstanding bridges best-effort.
|
||||
*/
|
||||
public static resetForTest(): void {
|
||||
if (PilotTunnelManager.instance) {
|
||||
for (const [, b] of PilotTunnelManager.instance.bridges) {
|
||||
try { b.close(1000, 'test reset'); } catch { /* ignore */ }
|
||||
}
|
||||
PilotTunnelManager.instance.bridges.clear();
|
||||
PilotTunnelManager.instance.bridgeKinds.clear();
|
||||
}
|
||||
PilotTunnelManager.instance = undefined as unknown as PilotTunnelManager;
|
||||
}
|
||||
|
||||
/**
|
||||
* Test-only: inject a pre-constructed bridge with an explicit kind.
|
||||
* Bypasses capacity / lifecycle hooks so unit tests can prime the
|
||||
* registry without owning a real WebSocket.
|
||||
*/
|
||||
public injectBridgeForTest(nodeId: number, bridge: PilotTunnelBridge, kind: 'pilot' | 'proxy'): void {
|
||||
this.bridges.set(nodeId, bridge);
|
||||
this.bridgeKinds.set(nodeId, kind);
|
||||
}
|
||||
|
||||
/**
|
||||
* Accept a newly handshaked pilot tunnel. Replaces any prior tunnel for the
|
||||
* same node (split-brain prevention): the previous bridge is closed
|
||||
@@ -95,6 +129,7 @@ export class PilotTunnelManager extends EventEmitter {
|
||||
if (existing) {
|
||||
existing.close(PilotCloseCode.Replaced, 'replaced by newer tunnel');
|
||||
this.bridges.delete(nodeId);
|
||||
this.bridgeKinds.delete(nodeId);
|
||||
}
|
||||
|
||||
// Hard cap: only counts tunnels for *other* nodes since we just
|
||||
@@ -120,6 +155,7 @@ export class PilotTunnelManager extends EventEmitter {
|
||||
bridge.once('closed', () => {
|
||||
if (this.bridges.get(nodeId) === bridge) {
|
||||
this.bridges.delete(nodeId);
|
||||
this.bridgeKinds.delete(nodeId);
|
||||
DatabaseService.getInstance().updateNodeStatus(nodeId, 'offline');
|
||||
this.emit('tunnel-down', nodeId);
|
||||
}
|
||||
@@ -127,6 +163,7 @@ export class PilotTunnelManager extends EventEmitter {
|
||||
await bridge.start();
|
||||
|
||||
this.bridges.set(nodeId, bridge);
|
||||
this.bridgeKinds.set(nodeId, 'pilot');
|
||||
const db = DatabaseService.getInstance();
|
||||
db.updateNodeStatus(nodeId, 'online');
|
||||
db.updateNode(nodeId, {
|
||||
@@ -233,14 +270,48 @@ export class PilotTunnelManager extends EventEmitter {
|
||||
bridge.once('closed', () => {
|
||||
if (this.bridges.get(nodeId) === bridge) {
|
||||
this.bridges.delete(nodeId);
|
||||
this.bridgeKinds.delete(nodeId);
|
||||
this.emit('proxy-bridge-down', nodeId);
|
||||
}
|
||||
});
|
||||
this.bridges.set(nodeId, bridge);
|
||||
this.bridgeKinds.set(nodeId, 'proxy');
|
||||
PilotMetrics.increment('proxy_bridges_total');
|
||||
this.emit('proxy-bridge-up', nodeId);
|
||||
}
|
||||
|
||||
/**
|
||||
* Register a peer-initiated proxy bridge for an existing remote. If a proxy
|
||||
* bridge already exists for this nodeId, close it (the new dial is the source
|
||||
* of truth) and replace. If a pilot tunnel exists, refuse: pilot tunnels
|
||||
* always win over peer-initiated proxy bridges.
|
||||
*/
|
||||
public replaceOrRegisterProxyBridge(nodeId: number, bridge: PilotTunnelBridge): void {
|
||||
const existingKind = this.bridgeKinds.get(nodeId);
|
||||
if (existingKind === 'pilot') {
|
||||
throw new Error(`pilot tunnel already registered for node ${nodeId}; proxy bridge refused`);
|
||||
}
|
||||
if (existingKind === 'proxy') {
|
||||
const old = this.bridges.get(nodeId);
|
||||
this.bridges.delete(nodeId);
|
||||
this.bridgeKinds.delete(nodeId);
|
||||
try { old?.close(1000, 'replaced-by-newer-proxy'); } catch { /* best-effort cleanup */ }
|
||||
}
|
||||
if (this.bridges.size >= PILOT_TUNNEL_HARD_LIMIT) {
|
||||
PilotMetrics.increment('tunnels_rejected_capacity');
|
||||
throw new PilotTunnelCapacityError(PILOT_TUNNEL_HARD_LIMIT);
|
||||
}
|
||||
bridge.once('closed', () => {
|
||||
if (this.bridges.get(nodeId) === bridge) {
|
||||
this.bridges.delete(nodeId);
|
||||
this.bridgeKinds.delete(nodeId);
|
||||
this.emit('proxy-bridge-down', nodeId, 'remote_closed');
|
||||
}
|
||||
});
|
||||
this.bridges.set(nodeId, bridge);
|
||||
this.bridgeKinds.set(nodeId, 'proxy');
|
||||
}
|
||||
|
||||
/**
|
||||
* Force-close a tunnel (e.g., on node deletion).
|
||||
*/
|
||||
@@ -249,6 +320,7 @@ export class PilotTunnelManager extends EventEmitter {
|
||||
if (!bridge) return;
|
||||
bridge.close(code, reason);
|
||||
this.bridges.delete(nodeId);
|
||||
this.bridgeKinds.delete(nodeId);
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user