mirror of
https://github.com/Studio-Saelix/sencho.git
synced 2026-08-25 17:57:06 +00:00
feat(mesh): symmetric WS dial for proxy-mode mesh peers (#1066)
* chore(mesh): foundation for symmetric callback dial Adds the data-plane scaffolding that the symmetric callback dial fix builds on: - mesh_centrals table for peer-side bootstrap material - MeshCentralRegistry service (upsert/getActive/clear/markUsed/markRejected) - PilotTunnelManager kind discriminator and replaceOrRegisterProxyBridge - mesh_proxy_callback_bootstrap capability registration - MeshProxyTunnelDialer reason-tagged proxy-bridge-down events from a single tearDownBridge emission point - Reactive redial scheduler that skips idle and auth_failed reasons * feat(mesh): add reverse-direction activity log entries (closes R1-B) acceptReverseLocal now emits route.resolve.ok with direction=reverse on connect ack and route.resolve.fail with direction=reverse plus reason=container_not_found / connect_error pre-connect. Post-connect close/error stays silent. Reuses existing event types via the new details.direction discriminator so frontend filters are unaffected. * feat(mesh): add peer-to-central callback dial path (closes R1-A2) Closes the architectural gap where proxy-mode mesh peers could not re-establish their tunnel to central after any non-idle bridge teardown (idle close, network blip, central restart, peer reboot). Central remains the hub for the data plane; the change is purely about WS initiation. Symmetric WS initiation, asymmetric protocol roles. Central retains PilotTunnelBridge ownership; peer retains TcpStreamSwitchboard + reverseDialer ownership. Central bootstraps callback credentials over the first authenticated central-initiated mesh tunnel via a one-shot mesh_handshake JSON frame; peer persists the material in a new mesh_centrals SQLite table and dials central's new /api/mesh/proxy-tunnel-from-peer endpoint when local cross-node traffic needs a bridge and none is live. Mesh_tunnel JWT (HS256, signed with auth_jwt_secret) carries scope, audience, issuer (central instance id), peer api_token fingerprint, kid. Validation on inbound peer dial: algorithm pin, signature, scope, audience, instance, time bounds, node existence and mode, fingerprint match. Failures return HTTP 401 with a machine-readable reason; peer routes the response per a clear-vs-keep cache matrix. Triggers proactive bootstrap on mesh-enable and api_token rotation; central startup fans out to mesh-enabled proxy-mode nodes with mesh_stacks rows (throttled, fire-and-forget). Reactive redial on non-idle bridge loss. Capability-gated handshake send (mesh_proxy_callback_bootstrap) makes the upgrade path safe against older peers in mixed-version fleets. Adds peer-side /api/system/pilot-tunnels centralCallback diag block, bounded counter metrics for bootstrap and dial events. SENCHO_PRIMARY_URL preflight warning when unset on a central with mesh-enabled proxy nodes. Tested with unit suites for the validation chain, registry, manager, and both dialers; integration tests for bootstrap E2E (asserts protocol-role invariant), api_token rotation, instance id change, version skew, and pilot-mode regression. * fix(mesh): green CI on the symmetric callback branch Two independent CI failures, both surgical: 1. Backend tests (11 fails): four mesh test files called setupTestDb in beforeEach. setupTestDb does not reset the DatabaseService singleton, so the per-test afterEach rm of the previous tmpdir left the singleton connection pointing at a deleted file. The next beforeEach's line-55 write threw SQLITE_READONLY_DBMOVED on Linux. Windows file-lock semantics hid this locally. Hoist setupTestDb / cleanupTestDb to file-scope beforeAll / afterAll; per-test state resets stay in beforeEach. Matches the convention in the eight mesh test files that already pass. 2. CodeQL (4 high alerts): js/insufficient-password-hash flagged sha256(api_token) at four sites. The api_token is a 256-bit opaque bearer (sen_sk_-prefixed), not a human password; sha256 is the correct fingerprint primitive for binding the mesh_tunnel JWT to a specific token. Add the two production files plus the two test files that mint the fingerprint to the existing path-scoped query-filter for that rule. * fix(mesh): drop unused afterEach import and revert dead codeql config ESLint flagged afterEach as unused in mesh-central-registry.test.ts:1 after the previous commit hoisted setup/teardown to file-scope beforeAll/afterAll. Remove from the vitest import line. Revert the codeql-config.yml additions from the previous commit. The paths: sub-key under query-filters > exclude is not a documented CodeQL feature and silently no-ops. The four js/insufficient-password-hash alerts on api_token fingerprinting are tracked as dismissed false positives in the GitHub Security tab rather than via dead config.
This commit is contained in:
Anso
@@ -64,7 +64,7 @@ export function getSenchoIpFromSubnet(subnet: string): string {
|
||||
export type MeshActivitySource = 'pilot' | 'mesh';
|
||||
export type MeshActivityLevel = 'info' | 'warn' | 'error';
|
||||
export type MeshActivityType =
|
||||
| 'route.dispatch' | 'route.resolve.ok' | 'route.resolve.denied'
|
||||
| 'route.dispatch' | 'route.resolve.ok' | 'route.resolve.denied' | 'route.resolve.fail'
|
||||
| 'tunnel.open' | 'tunnel.fail' | 'tunnel.backpressure'
|
||||
| 'opt_in' | 'opt_out'
|
||||
| 'mesh.enable' | 'mesh.disable'
|
||||
@@ -72,7 +72,8 @@ export type MeshActivityType =
|
||||
| 'probe.ok' | 'probe.fail'
|
||||
| 'forwarder.listen' | 'forwarder.unlisten' | 'forwarder.error'
|
||||
| 'proxy-tunnel.open.ok' | 'proxy-tunnel.open.fail' | 'proxy-tunnel.close'
|
||||
| 'mesh.proxy_tunnel.identify';
|
||||
| 'mesh.proxy_tunnel.identify'
|
||||
| 'mesh_handshake.received';
|
||||
|
||||
export interface MeshActivityEvent {
|
||||
ts: number;
|
||||
@@ -299,6 +300,8 @@ export class MeshService extends EventEmitter implements MeshForwarderHost {
|
||||
|
||||
this.selfCentralNodeId = this.resolveSelfCentralNodeId();
|
||||
|
||||
this.maybeWarnUnsetPrimaryUrl();
|
||||
|
||||
await this.setupMeshNetwork();
|
||||
try {
|
||||
await this.refreshAliasCache();
|
||||
@@ -317,6 +320,10 @@ export class MeshService extends EventEmitter implements MeshForwarderHost {
|
||||
});
|
||||
}
|
||||
await this.regenerateAllOverrides();
|
||||
// Trigger 3: proactively re-establish central->peer bridges so any
|
||||
// peer that is online and capable receives bootstrap material on
|
||||
// startup. Fire-and-forget so start() does not block on remote I/O.
|
||||
void this.proactiveBootstrapFanout();
|
||||
this.aliasRefreshTimer = setInterval(() => {
|
||||
void (async () => {
|
||||
try {
|
||||
@@ -345,6 +352,71 @@ export class MeshService extends EventEmitter implements MeshForwarderHost {
|
||||
await this.forwarder.shutdown();
|
||||
}
|
||||
|
||||
/**
|
||||
* Operator-facing preflight: if `SENCHO_PRIMARY_URL` is unset at
|
||||
* startup and the central has at least one mesh-enabled proxy-mode
|
||||
* node, warn that the Task 8 capability-gated handshake will fall
|
||||
* back to an inferred origin for the callback bootstrap. The matching
|
||||
* fail-safe in `MeshProxyTunnelDialer` silently skips the bootstrap
|
||||
* when the env is unset; this warn makes the consequence visible.
|
||||
*/
|
||||
private maybeWarnUnsetPrimaryUrl(): void {
|
||||
if (process.env.SENCHO_PRIMARY_URL) return;
|
||||
const row = DatabaseService.getInstance().getDb().prepare(`
|
||||
SELECT COUNT(*) as n FROM nodes
|
||||
WHERE type='remote' AND mode='proxy' AND mesh_enabled=1
|
||||
`).get() as { n: number };
|
||||
if (row.n > 0) {
|
||||
console.warn(
|
||||
'[Mesh] SENCHO_PRIMARY_URL is unset; mesh callback bootstrap will use ' +
|
||||
'inferred origin. Set SENCHO_PRIMARY_URL to make peer-initiated mesh ' +
|
||||
'callbacks robust against reverse-proxy edge cases.'
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Trigger 3: at central startup, walk every mesh-enabled proxy-mode
|
||||
* peer that already has at least one `mesh_stacks` row and call
|
||||
* `MeshProxyTunnelDialer.ensureBridge(nodeId)`. The dialer's
|
||||
* capability-gated handshake then mints and ships bootstrap material
|
||||
* to peers that just came online or just got upgraded to a build that
|
||||
* advertises `mesh_proxy_callback_bootstrap`. Bounded concurrency 4
|
||||
* with a 250ms stagger; failures are logged and never abort the
|
||||
* fan-out.
|
||||
*/
|
||||
private async proactiveBootstrapFanout(): Promise<void> {
|
||||
const rows = DatabaseService.getInstance().getDb().prepare(`
|
||||
SELECT DISTINCT n.id AS id
|
||||
FROM nodes n
|
||||
INNER JOIN mesh_stacks ms ON ms.node_id = n.id
|
||||
WHERE n.type = 'remote' AND n.mode = 'proxy' AND n.mesh_enabled = 1
|
||||
ORDER BY n.id
|
||||
`).all() as Array<{ id: number }>;
|
||||
|
||||
const queue = rows.map((r) => r.id);
|
||||
const dialer = MeshProxyTunnelDialer.getInstance();
|
||||
const worker = async (): Promise<void> => {
|
||||
for (;;) {
|
||||
const nodeId = queue.shift();
|
||||
if (nodeId === undefined) return;
|
||||
try {
|
||||
await dialer.ensureBridge(nodeId);
|
||||
} catch (err) {
|
||||
this.logActivity({
|
||||
source: 'mesh', level: 'warn', type: 'proxy-tunnel.open.fail',
|
||||
nodeId,
|
||||
message: `boot proactive bootstrap failed: ${sanitizeForLog((err as Error).message)}`,
|
||||
details: { trigger: 'startup_fanout' },
|
||||
});
|
||||
}
|
||||
await new Promise((r) => setTimeout(r, 250));
|
||||
}
|
||||
};
|
||||
const workerCount = Math.min(4, Math.max(1, queue.length));
|
||||
await Promise.all(Array.from({ length: workerCount }, () => worker()));
|
||||
}
|
||||
|
||||
public getSenchoIp(): string | null {
|
||||
return this.senchoIp;
|
||||
}
|
||||
@@ -653,6 +725,17 @@ export class MeshService extends EventEmitter implements MeshForwarderHost {
|
||||
source: 'mesh', level: 'info', type: 'mesh.enable',
|
||||
nodeId, message: `mesh enabled on node ${nodeId}`,
|
||||
});
|
||||
// Trigger 1: proactive bootstrap. If the peer is a proxy-mode remote,
|
||||
// dial the mesh callback bridge immediately so the capability-gated
|
||||
// handshake can ship `mesh_handshake` material without waiting for
|
||||
// the next forwarder dial. Fire-and-forget; failures are logged by
|
||||
// the dialer.
|
||||
const node = DatabaseService.getInstance().getNode(nodeId);
|
||||
if (node && node.type === 'remote' && node.mode === 'proxy') {
|
||||
void MeshProxyTunnelDialer.getInstance().ensureBridge(nodeId).catch((err) => {
|
||||
console.warn(`[Mesh] proactive bootstrap on mesh-enable failed for node ${nodeId}: ${(err as Error).message}`);
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
public async disableForNode(nodeId: number): Promise<void> {
|
||||
@@ -1506,6 +1589,39 @@ export class MeshService extends EventEmitter implements MeshForwarderHost {
|
||||
message: `cross-node dispatch to ${target.alias} on node ${target.nodeId}`,
|
||||
});
|
||||
|
||||
// Peer-side recovery: when no reverseDialer is installed, kick the
|
||||
// symmetric-dial path so central learns about this peer. The
|
||||
// proactive back-dial from central (registered as a side effect via
|
||||
// the proxy-tunnel WS upgrade) is what actually installs the
|
||||
// reverseDialer on this peer. If the session cannot be established
|
||||
// (cache miss, central down, auth rejected), log route.resolve.fail
|
||||
// and drop the inbound socket.
|
||||
if (!this.reverseDialer) {
|
||||
try {
|
||||
const { PeerToCentralMeshSessionDialer } = await import('./PeerToCentralMeshSessionDialer');
|
||||
const session = await PeerToCentralMeshSessionDialer.getInstance().ensureSession();
|
||||
if (!session) {
|
||||
this.logActivity({
|
||||
source: 'mesh', level: 'warn', type: 'route.resolve.fail',
|
||||
nodeId: target.nodeId, alias: target.alias,
|
||||
message: `peer cross-node dispatch failed: no central callback session available`,
|
||||
details: { direction: 'forward-from-peer', reason: 'no_session' },
|
||||
});
|
||||
try { src.destroy(); } catch { /* ignore */ }
|
||||
return;
|
||||
}
|
||||
} catch (err) {
|
||||
this.logActivity({
|
||||
source: 'mesh', level: 'warn', type: 'route.resolve.fail',
|
||||
nodeId: target.nodeId, alias: target.alias,
|
||||
message: `peer cross-node dispatch failed: bootstrap threw`,
|
||||
details: { direction: 'forward-from-peer', reason: 'bootstrap_threw' },
|
||||
});
|
||||
try { src.destroy(); } catch { /* ignore */ }
|
||||
return;
|
||||
}
|
||||
}
|
||||
|
||||
const tcpStream = await this.dialMeshTcpStream(target);
|
||||
if (!tcpStream) {
|
||||
this.logActivity({
|
||||
|
||||
Reference in New Issue
Block a user