Files
sencho/backend/src/__tests__/pilot-tunnel-bridge-reverse-route-events.test.ts
T
Anso cf618dd866 feat(mesh): symmetric WS dial for proxy-mode mesh peers (#1066)
* chore(mesh): foundation for symmetric callback dial

Adds the data-plane scaffolding that the symmetric callback dial fix
builds on:
- mesh_centrals table for peer-side bootstrap material
- MeshCentralRegistry service (upsert/getActive/clear/markUsed/markRejected)
- PilotTunnelManager kind discriminator and replaceOrRegisterProxyBridge
- mesh_proxy_callback_bootstrap capability registration
- MeshProxyTunnelDialer reason-tagged proxy-bridge-down events from a
  single tearDownBridge emission point
- Reactive redial scheduler that skips idle and auth_failed reasons

* feat(mesh): add reverse-direction activity log entries (closes R1-B)

acceptReverseLocal now emits route.resolve.ok with direction=reverse on
connect ack and route.resolve.fail with direction=reverse plus
reason=container_not_found / connect_error pre-connect. Post-connect
close/error stays silent. Reuses existing event types via the new
details.direction discriminator so frontend filters are unaffected.

* feat(mesh): add peer-to-central callback dial path (closes R1-A2)

Closes the architectural gap where proxy-mode mesh peers could not
re-establish their tunnel to central after any non-idle bridge teardown
(idle close, network blip, central restart, peer reboot). Central remains
the hub for the data plane; the change is purely about WS initiation.

Symmetric WS initiation, asymmetric protocol roles. Central retains
PilotTunnelBridge ownership; peer retains TcpStreamSwitchboard +
reverseDialer ownership. Central bootstraps callback credentials over
the first authenticated central-initiated mesh tunnel via a one-shot
mesh_handshake JSON frame; peer persists the material in a new
mesh_centrals SQLite table and dials central's new
/api/mesh/proxy-tunnel-from-peer endpoint when local cross-node traffic
needs a bridge and none is live.

Mesh_tunnel JWT (HS256, signed with auth_jwt_secret) carries scope, audience,
issuer (central instance id), peer api_token fingerprint, kid. Validation
on inbound peer dial: algorithm pin, signature, scope, audience, instance,
time bounds, node existence and mode, fingerprint match. Failures return
HTTP 401 with a machine-readable reason; peer routes the response per a
clear-vs-keep cache matrix.

Triggers proactive bootstrap on mesh-enable and api_token rotation; central
startup fans out to mesh-enabled proxy-mode nodes with mesh_stacks rows
(throttled, fire-and-forget). Reactive redial on non-idle bridge loss.

Capability-gated handshake send (mesh_proxy_callback_bootstrap) makes the
upgrade path safe against older peers in mixed-version fleets.

Adds peer-side /api/system/pilot-tunnels centralCallback diag block,
bounded counter metrics for bootstrap and dial events. SENCHO_PRIMARY_URL
preflight warning when unset on a central with mesh-enabled proxy nodes.

Tested with unit suites for the validation chain, registry, manager, and
both dialers; integration tests for bootstrap E2E (asserts protocol-role
invariant), api_token rotation, instance id change, version skew, and
pilot-mode regression.

* fix(mesh): green CI on the symmetric callback branch

Two independent CI failures, both surgical:

1. Backend tests (11 fails): four mesh test files called setupTestDb in
   beforeEach. setupTestDb does not reset the DatabaseService singleton,
   so the per-test afterEach rm of the previous tmpdir left the singleton
   connection pointing at a deleted file. The next beforeEach's line-55
   write threw SQLITE_READONLY_DBMOVED on Linux. Windows file-lock
   semantics hid this locally. Hoist setupTestDb / cleanupTestDb to
   file-scope beforeAll / afterAll; per-test state resets stay in
   beforeEach. Matches the convention in the eight mesh test files that
   already pass.

2. CodeQL (4 high alerts): js/insufficient-password-hash flagged
   sha256(api_token) at four sites. The api_token is a 256-bit opaque
   bearer (sen_sk_-prefixed), not a human password; sha256 is the
   correct fingerprint primitive for binding the mesh_tunnel JWT to a
   specific token. Add the two production files plus the two test
   files that mint the fingerprint to the existing path-scoped
   query-filter for that rule.

* fix(mesh): drop unused afterEach import and revert dead codeql config

ESLint flagged afterEach as unused in mesh-central-registry.test.ts:1
after the previous commit hoisted setup/teardown to file-scope
beforeAll/afterAll. Remove from the vitest import line.

Revert the codeql-config.yml additions from the previous commit. The
paths: sub-key under query-filters > exclude is not a documented CodeQL
feature and silently no-ops. The four js/insufficient-password-hash
alerts on api_token fingerprinting are tracked as dismissed false
positives in the GitHub Security tab rather than via dead config.
2026-05-16 14:58:19 -04:00

291 lines
12 KiB
TypeScript

/**
* R1-B: PilotTunnelBridge.acceptReverseLocal emits MeshService activity events
* for the reverse-direction dispatch (peer-to-central). Reuses the existing
* `route.resolve.ok` / `route.resolve.fail` event types and discriminates by
* `details.direction = 'reverse'` so the Routing tab can surface peer-side
* mesh failures alongside central-side dispatch events.
*
* Covers three outcomes plus one negative assertion:
* 1. resolveContainerIp returns null -> route.resolve.fail / container_not_found
* 2. socket emits 'error' pre-connect -> route.resolve.fail / connect_error
* 3. socket emits 'connect' -> route.resolve.ok
* 4. post-connect close/error does NOT emit a second route.resolve.fail
*/
import { describe, it, expect, beforeAll, afterAll, beforeEach, vi } from 'vitest';
import net from 'net';
import { EventEmitter } from 'events';
import { WebSocket } from 'ws';
import { setupTestDb, cleanupTestDb } from './helpers/setupTestDb';
import { AGENT_REVERSE_ID_BASE, encodeJsonFrame, decodeJsonFrame } from '../pilot/protocol';
import type { MeshActivityEvent } from '../services/MeshService';
type LogActivityArgs = [Omit<MeshActivityEvent, 'ts'>];
let tmpDir: string;
let PilotTunnelBridge: typeof import('../services/PilotTunnelBridge').PilotTunnelBridge;
let MeshService: typeof import('../services/MeshService').MeshService;
beforeAll(async () => {
tmpDir = await setupTestDb();
({ PilotTunnelBridge } = await import('../services/PilotTunnelBridge'));
({ MeshService } = await import('../services/MeshService'));
});
afterAll(() => {
vi.restoreAllMocks();
cleanupTestDb(tmpDir);
});
function makeMockTunnelWs(): EventEmitter & {
sent: unknown[];
readyState: number;
bufferedAmount: number;
send: (data: unknown) => void;
ping: () => void;
close: () => void;
} {
const ws = new EventEmitter() as EventEmitter & {
sent: unknown[]; readyState: number; bufferedAmount: number;
send: (data: unknown) => void; ping: () => void; close: () => void;
};
ws.sent = [];
ws.readyState = WebSocket.OPEN;
ws.bufferedAmount = 0;
ws.send = (data: unknown) => { ws.sent.push(data); };
ws.ping = () => { /* no-op */ };
ws.close = () => { ws.readyState = WebSocket.CLOSED; ws.emit('close'); };
return ws;
}
function findAck(ws: { sent: unknown[] }, s: number): { ok: boolean; err?: string } | undefined {
for (const item of ws.sent) {
if (typeof item !== 'string') continue;
try {
const f = decodeJsonFrame(item);
if (f.t === 'tcp_open_ack' && f.s === s) return { ok: f.ok, err: f.err };
} catch { /* ignore */ }
}
return undefined;
}
async function waitFor<T>(check: () => T | undefined): Promise<T> {
const deadline = Date.now() + 1500;
while (Date.now() < deadline) {
const v = check();
if (v !== undefined) return v;
await new Promise((r) => setTimeout(r, 10));
}
throw new Error('timeout');
}
describe('R1-B: PilotTunnelBridge.acceptReverseLocal route events', () => {
let logSpy: ReturnType<typeof vi.fn<(event: Omit<MeshActivityEvent, 'ts'>) => void>>;
beforeEach(() => {
vi.restoreAllMocks();
logSpy = vi.fn<(event: Omit<MeshActivityEvent, 'ts'>) => void>();
vi.spyOn(MeshService.getInstance(), 'logActivity').mockImplementation(logSpy);
});
it('emits route.resolve.fail with direction=reverse + reason=container_not_found when IP is unresolved', async () => {
const mockWs = makeMockTunnelWs();
const peerNodeId = 42;
const bridge = new PilotTunnelBridge(peerNodeId, mockWs as unknown as WebSocket);
await bridge.start();
const localNodeId = (await import('../services/NodeRegistry')).NodeRegistry.getInstance().getDefaultNodeId();
vi.spyOn(MeshService.getInstance(), 'resolveContainerIp').mockResolvedValue(null);
const s = AGENT_REVERSE_ID_BASE + 11;
mockWs.emit('message', encodeJsonFrame({
t: 'tcp_open_reverse', s,
targetNodeId: localNodeId, stack: 'missing', service: 'svc', port: 80,
}), false);
const ack = await waitFor(() => findAck(mockWs, s));
expect(ack.ok).toBe(false);
const failCall = logSpy.mock.calls.find(
(c: LogActivityArgs) => c[0].type === 'route.resolve.fail',
);
expect(failCall).toBeDefined();
const payload = failCall![0] as {
source: string;
level: string;
type: string;
nodeId?: number;
details?: Record<string, unknown>;
};
expect(payload.source).toBe('mesh');
expect(payload.level).toBe('error');
expect(payload.type).toBe('route.resolve.fail');
expect(payload.nodeId).toBe(peerNodeId);
expect(payload.details).toMatchObject({
direction: 'reverse',
reason: 'container_not_found',
targetStack: 'missing',
targetService: 'svc',
targetPort: 80,
});
bridge.close();
});
it('emits route.resolve.fail with direction=reverse + reason=connect_error on socket pre-connect error', async () => {
const mockWs = makeMockTunnelWs();
const peerNodeId = 43;
const bridge = new PilotTunnelBridge(peerNodeId, mockWs as unknown as WebSocket);
await bridge.start();
const localNodeId = (await import('../services/NodeRegistry')).NodeRegistry.getInstance().getDefaultNodeId();
// Resolve to localhost on a port nothing is listening on so the dial
// produces a synchronous ECONNREFUSED via the OS.
vi.spyOn(MeshService.getInstance(), 'resolveContainerIp').mockResolvedValue('127.0.0.1');
// Pick a port we know is closed by binding and immediately releasing.
const probe = net.createServer();
await new Promise<void>((resolve) => probe.listen(0, '127.0.0.1', () => resolve()));
const addr = probe.address();
if (!addr || typeof addr === 'string') throw new Error('no address');
const closedPort = addr.port;
await new Promise<void>((resolve) => probe.close(() => resolve()));
const s = AGENT_REVERSE_ID_BASE + 12;
mockWs.emit('message', encodeJsonFrame({
t: 'tcp_open_reverse', s,
targetNodeId: localNodeId, stack: 'real', service: 'svc', port: closedPort,
}), false);
const ack = await waitFor(() => findAck(mockWs, s));
expect(ack.ok).toBe(false);
expect(ack.err).toBe('unreachable');
const failCall = logSpy.mock.calls.find(
(c: LogActivityArgs) => c[0].type === 'route.resolve.fail',
);
expect(failCall).toBeDefined();
const payload = failCall![0] as {
type: string;
nodeId?: number;
details?: Record<string, unknown>;
};
expect(payload.type).toBe('route.resolve.fail');
expect(payload.nodeId).toBe(peerNodeId);
expect(payload.details).toMatchObject({
direction: 'reverse',
reason: 'connect_error',
targetStack: 'real',
targetService: 'svc',
targetPort: closedPort,
});
bridge.close();
});
it('emits route.resolve.ok with direction=reverse on connect ack', async () => {
const mockWs = makeMockTunnelWs();
const peerNodeId = 44;
const bridge = new PilotTunnelBridge(peerNodeId, mockWs as unknown as WebSocket);
await bridge.start();
// Real local server so the dial succeeds and 'connect' fires.
const upstream = net.createServer((socket) => {
socket.write('hello-upstream');
});
await new Promise<void>((resolve) => upstream.listen(0, '127.0.0.1', () => resolve()));
const addr = upstream.address();
if (!addr || typeof addr === 'string') throw new Error('no address');
const upstreamPort = addr.port;
const localNodeId = (await import('../services/NodeRegistry')).NodeRegistry.getInstance().getDefaultNodeId();
vi.spyOn(MeshService.getInstance(), 'resolveContainerIp').mockResolvedValue('127.0.0.1');
const s = AGENT_REVERSE_ID_BASE + 13;
mockWs.emit('message', encodeJsonFrame({
t: 'tcp_open_reverse', s,
targetNodeId: localNodeId, stack: 'real', service: 'svc', port: upstreamPort,
}), false);
const ack = await waitFor(() => findAck(mockWs, s));
expect(ack.ok).toBe(true);
// Wait for the ok event to land (logActivity is sync but the connect
// callback is async relative to message handling).
await waitFor(() => logSpy.mock.calls.find(
(c: LogActivityArgs) => c[0].type === 'route.resolve.ok',
));
const okCall = logSpy.mock.calls.find(
(c: LogActivityArgs) => c[0].type === 'route.resolve.ok',
);
expect(okCall).toBeDefined();
const payload = okCall![0] as {
source: string;
level: string;
type: string;
nodeId?: number;
details?: Record<string, unknown>;
};
expect(payload.source).toBe('mesh');
expect(payload.level).toBe('info');
expect(payload.type).toBe('route.resolve.ok');
expect(payload.nodeId).toBe(peerNodeId);
expect(payload.details).toMatchObject({
direction: 'reverse',
targetStack: 'real',
targetService: 'svc',
targetPort: upstreamPort,
});
upstream.close();
bridge.close();
});
it('does NOT emit route.resolve.fail when a connected socket closes post-handshake', async () => {
const mockWs = makeMockTunnelWs();
const peerNodeId = 45;
const bridge = new PilotTunnelBridge(peerNodeId, mockWs as unknown as WebSocket);
await bridge.start();
// Real local server. Have it close the connection immediately after
// accept so the bridge socket sees 'close' (and possibly 'error') on
// an already-connected socket.
const upstream = net.createServer((socket) => {
socket.end();
});
await new Promise<void>((resolve) => upstream.listen(0, '127.0.0.1', () => resolve()));
const addr = upstream.address();
if (!addr || typeof addr === 'string') throw new Error('no address');
const upstreamPort = addr.port;
const localNodeId = (await import('../services/NodeRegistry')).NodeRegistry.getInstance().getDefaultNodeId();
vi.spyOn(MeshService.getInstance(), 'resolveContainerIp').mockResolvedValue('127.0.0.1');
const s = AGENT_REVERSE_ID_BASE + 14;
mockWs.emit('message', encodeJsonFrame({
t: 'tcp_open_reverse', s,
targetNodeId: localNodeId, stack: 'real', service: 'svc', port: upstreamPort,
}), false);
// Wait for connect (the success ack confirms onPreConnectError was removed).
const ack = await waitFor(() => findAck(mockWs, s));
expect(ack.ok).toBe(true);
// Wait for the success event to confirm the ok path fired.
await waitFor(() => logSpy.mock.calls.find(
(c: LogActivityArgs) => c[0].type === 'route.resolve.ok',
));
// Give the post-connect close a chance to fire teardown handlers.
await new Promise((r) => setTimeout(r, 100));
const failCalls = logSpy.mock.calls.filter(
(c: LogActivityArgs) => c[0].type === 'route.resolve.fail',
);
expect(failCalls).toHaveLength(0);
upstream.close();
bridge.close();
});
});