mirror of
https://github.com/Studio-Saelix/sencho.git
synced 2026-08-24 00:49:45 +00:00
cf618dd866
* chore(mesh): foundation for symmetric callback dial Adds the data-plane scaffolding that the symmetric callback dial fix builds on: - mesh_centrals table for peer-side bootstrap material - MeshCentralRegistry service (upsert/getActive/clear/markUsed/markRejected) - PilotTunnelManager kind discriminator and replaceOrRegisterProxyBridge - mesh_proxy_callback_bootstrap capability registration - MeshProxyTunnelDialer reason-tagged proxy-bridge-down events from a single tearDownBridge emission point - Reactive redial scheduler that skips idle and auth_failed reasons * feat(mesh): add reverse-direction activity log entries (closes R1-B) acceptReverseLocal now emits route.resolve.ok with direction=reverse on connect ack and route.resolve.fail with direction=reverse plus reason=container_not_found / connect_error pre-connect. Post-connect close/error stays silent. Reuses existing event types via the new details.direction discriminator so frontend filters are unaffected. * feat(mesh): add peer-to-central callback dial path (closes R1-A2) Closes the architectural gap where proxy-mode mesh peers could not re-establish their tunnel to central after any non-idle bridge teardown (idle close, network blip, central restart, peer reboot). Central remains the hub for the data plane; the change is purely about WS initiation. Symmetric WS initiation, asymmetric protocol roles. Central retains PilotTunnelBridge ownership; peer retains TcpStreamSwitchboard + reverseDialer ownership. Central bootstraps callback credentials over the first authenticated central-initiated mesh tunnel via a one-shot mesh_handshake JSON frame; peer persists the material in a new mesh_centrals SQLite table and dials central's new /api/mesh/proxy-tunnel-from-peer endpoint when local cross-node traffic needs a bridge and none is live. Mesh_tunnel JWT (HS256, signed with auth_jwt_secret) carries scope, audience, issuer (central instance id), peer api_token fingerprint, kid. Validation on inbound peer dial: algorithm pin, signature, scope, audience, instance, time bounds, node existence and mode, fingerprint match. Failures return HTTP 401 with a machine-readable reason; peer routes the response per a clear-vs-keep cache matrix. Triggers proactive bootstrap on mesh-enable and api_token rotation; central startup fans out to mesh-enabled proxy-mode nodes with mesh_stacks rows (throttled, fire-and-forget). Reactive redial on non-idle bridge loss. Capability-gated handshake send (mesh_proxy_callback_bootstrap) makes the upgrade path safe against older peers in mixed-version fleets. Adds peer-side /api/system/pilot-tunnels centralCallback diag block, bounded counter metrics for bootstrap and dial events. SENCHO_PRIMARY_URL preflight warning when unset on a central with mesh-enabled proxy nodes. Tested with unit suites for the validation chain, registry, manager, and both dialers; integration tests for bootstrap E2E (asserts protocol-role invariant), api_token rotation, instance id change, version skew, and pilot-mode regression. * fix(mesh): green CI on the symmetric callback branch Two independent CI failures, both surgical: 1. Backend tests (11 fails): four mesh test files called setupTestDb in beforeEach. setupTestDb does not reset the DatabaseService singleton, so the per-test afterEach rm of the previous tmpdir left the singleton connection pointing at a deleted file. The next beforeEach's line-55 write threw SQLITE_READONLY_DBMOVED on Linux. Windows file-lock semantics hid this locally. Hoist setupTestDb / cleanupTestDb to file-scope beforeAll / afterAll; per-test state resets stay in beforeEach. Matches the convention in the eight mesh test files that already pass. 2. CodeQL (4 high alerts): js/insufficient-password-hash flagged sha256(api_token) at four sites. The api_token is a 256-bit opaque bearer (sen_sk_-prefixed), not a human password; sha256 is the correct fingerprint primitive for binding the mesh_tunnel JWT to a specific token. Add the two production files plus the two test files that mint the fingerprint to the existing path-scoped query-filter for that rule. * fix(mesh): drop unused afterEach import and revert dead codeql config ESLint flagged afterEach as unused in mesh-central-registry.test.ts:1 after the previous commit hoisted setup/teardown to file-scope beforeAll/afterAll. Remove from the vitest import line. Revert the codeql-config.yml additions from the previous commit. The paths: sub-key under query-filters > exclude is not a documented CodeQL feature and silently no-ops. The four js/insufficient-password-hash alerts on api_token fingerprinting are tracked as dismissed false positives in the GitHub Security tab rather than via dead config.
291 lines
12 KiB
TypeScript
291 lines
12 KiB
TypeScript
/**
|
|
* R1-B: PilotTunnelBridge.acceptReverseLocal emits MeshService activity events
|
|
* for the reverse-direction dispatch (peer-to-central). Reuses the existing
|
|
* `route.resolve.ok` / `route.resolve.fail` event types and discriminates by
|
|
* `details.direction = 'reverse'` so the Routing tab can surface peer-side
|
|
* mesh failures alongside central-side dispatch events.
|
|
*
|
|
* Covers three outcomes plus one negative assertion:
|
|
* 1. resolveContainerIp returns null -> route.resolve.fail / container_not_found
|
|
* 2. socket emits 'error' pre-connect -> route.resolve.fail / connect_error
|
|
* 3. socket emits 'connect' -> route.resolve.ok
|
|
* 4. post-connect close/error does NOT emit a second route.resolve.fail
|
|
*/
|
|
import { describe, it, expect, beforeAll, afterAll, beforeEach, vi } from 'vitest';
|
|
import net from 'net';
|
|
import { EventEmitter } from 'events';
|
|
import { WebSocket } from 'ws';
|
|
import { setupTestDb, cleanupTestDb } from './helpers/setupTestDb';
|
|
import { AGENT_REVERSE_ID_BASE, encodeJsonFrame, decodeJsonFrame } from '../pilot/protocol';
|
|
import type { MeshActivityEvent } from '../services/MeshService';
|
|
|
|
type LogActivityArgs = [Omit<MeshActivityEvent, 'ts'>];
|
|
|
|
let tmpDir: string;
|
|
let PilotTunnelBridge: typeof import('../services/PilotTunnelBridge').PilotTunnelBridge;
|
|
let MeshService: typeof import('../services/MeshService').MeshService;
|
|
|
|
beforeAll(async () => {
|
|
tmpDir = await setupTestDb();
|
|
({ PilotTunnelBridge } = await import('../services/PilotTunnelBridge'));
|
|
({ MeshService } = await import('../services/MeshService'));
|
|
});
|
|
|
|
afterAll(() => {
|
|
vi.restoreAllMocks();
|
|
cleanupTestDb(tmpDir);
|
|
});
|
|
|
|
function makeMockTunnelWs(): EventEmitter & {
|
|
sent: unknown[];
|
|
readyState: number;
|
|
bufferedAmount: number;
|
|
send: (data: unknown) => void;
|
|
ping: () => void;
|
|
close: () => void;
|
|
} {
|
|
const ws = new EventEmitter() as EventEmitter & {
|
|
sent: unknown[]; readyState: number; bufferedAmount: number;
|
|
send: (data: unknown) => void; ping: () => void; close: () => void;
|
|
};
|
|
ws.sent = [];
|
|
ws.readyState = WebSocket.OPEN;
|
|
ws.bufferedAmount = 0;
|
|
ws.send = (data: unknown) => { ws.sent.push(data); };
|
|
ws.ping = () => { /* no-op */ };
|
|
ws.close = () => { ws.readyState = WebSocket.CLOSED; ws.emit('close'); };
|
|
return ws;
|
|
}
|
|
|
|
function findAck(ws: { sent: unknown[] }, s: number): { ok: boolean; err?: string } | undefined {
|
|
for (const item of ws.sent) {
|
|
if (typeof item !== 'string') continue;
|
|
try {
|
|
const f = decodeJsonFrame(item);
|
|
if (f.t === 'tcp_open_ack' && f.s === s) return { ok: f.ok, err: f.err };
|
|
} catch { /* ignore */ }
|
|
}
|
|
return undefined;
|
|
}
|
|
|
|
async function waitFor<T>(check: () => T | undefined): Promise<T> {
|
|
const deadline = Date.now() + 1500;
|
|
while (Date.now() < deadline) {
|
|
const v = check();
|
|
if (v !== undefined) return v;
|
|
await new Promise((r) => setTimeout(r, 10));
|
|
}
|
|
throw new Error('timeout');
|
|
}
|
|
|
|
describe('R1-B: PilotTunnelBridge.acceptReverseLocal route events', () => {
|
|
let logSpy: ReturnType<typeof vi.fn<(event: Omit<MeshActivityEvent, 'ts'>) => void>>;
|
|
|
|
beforeEach(() => {
|
|
vi.restoreAllMocks();
|
|
logSpy = vi.fn<(event: Omit<MeshActivityEvent, 'ts'>) => void>();
|
|
vi.spyOn(MeshService.getInstance(), 'logActivity').mockImplementation(logSpy);
|
|
});
|
|
|
|
it('emits route.resolve.fail with direction=reverse + reason=container_not_found when IP is unresolved', async () => {
|
|
const mockWs = makeMockTunnelWs();
|
|
const peerNodeId = 42;
|
|
const bridge = new PilotTunnelBridge(peerNodeId, mockWs as unknown as WebSocket);
|
|
await bridge.start();
|
|
|
|
const localNodeId = (await import('../services/NodeRegistry')).NodeRegistry.getInstance().getDefaultNodeId();
|
|
vi.spyOn(MeshService.getInstance(), 'resolveContainerIp').mockResolvedValue(null);
|
|
|
|
const s = AGENT_REVERSE_ID_BASE + 11;
|
|
mockWs.emit('message', encodeJsonFrame({
|
|
t: 'tcp_open_reverse', s,
|
|
targetNodeId: localNodeId, stack: 'missing', service: 'svc', port: 80,
|
|
}), false);
|
|
|
|
const ack = await waitFor(() => findAck(mockWs, s));
|
|
expect(ack.ok).toBe(false);
|
|
|
|
const failCall = logSpy.mock.calls.find(
|
|
(c: LogActivityArgs) => c[0].type === 'route.resolve.fail',
|
|
);
|
|
expect(failCall).toBeDefined();
|
|
const payload = failCall![0] as {
|
|
source: string;
|
|
level: string;
|
|
type: string;
|
|
nodeId?: number;
|
|
details?: Record<string, unknown>;
|
|
};
|
|
expect(payload.source).toBe('mesh');
|
|
expect(payload.level).toBe('error');
|
|
expect(payload.type).toBe('route.resolve.fail');
|
|
expect(payload.nodeId).toBe(peerNodeId);
|
|
expect(payload.details).toMatchObject({
|
|
direction: 'reverse',
|
|
reason: 'container_not_found',
|
|
targetStack: 'missing',
|
|
targetService: 'svc',
|
|
targetPort: 80,
|
|
});
|
|
|
|
bridge.close();
|
|
});
|
|
|
|
it('emits route.resolve.fail with direction=reverse + reason=connect_error on socket pre-connect error', async () => {
|
|
const mockWs = makeMockTunnelWs();
|
|
const peerNodeId = 43;
|
|
const bridge = new PilotTunnelBridge(peerNodeId, mockWs as unknown as WebSocket);
|
|
await bridge.start();
|
|
|
|
const localNodeId = (await import('../services/NodeRegistry')).NodeRegistry.getInstance().getDefaultNodeId();
|
|
// Resolve to localhost on a port nothing is listening on so the dial
|
|
// produces a synchronous ECONNREFUSED via the OS.
|
|
vi.spyOn(MeshService.getInstance(), 'resolveContainerIp').mockResolvedValue('127.0.0.1');
|
|
|
|
// Pick a port we know is closed by binding and immediately releasing.
|
|
const probe = net.createServer();
|
|
await new Promise<void>((resolve) => probe.listen(0, '127.0.0.1', () => resolve()));
|
|
const addr = probe.address();
|
|
if (!addr || typeof addr === 'string') throw new Error('no address');
|
|
const closedPort = addr.port;
|
|
await new Promise<void>((resolve) => probe.close(() => resolve()));
|
|
|
|
const s = AGENT_REVERSE_ID_BASE + 12;
|
|
mockWs.emit('message', encodeJsonFrame({
|
|
t: 'tcp_open_reverse', s,
|
|
targetNodeId: localNodeId, stack: 'real', service: 'svc', port: closedPort,
|
|
}), false);
|
|
|
|
const ack = await waitFor(() => findAck(mockWs, s));
|
|
expect(ack.ok).toBe(false);
|
|
expect(ack.err).toBe('unreachable');
|
|
|
|
const failCall = logSpy.mock.calls.find(
|
|
(c: LogActivityArgs) => c[0].type === 'route.resolve.fail',
|
|
);
|
|
expect(failCall).toBeDefined();
|
|
const payload = failCall![0] as {
|
|
type: string;
|
|
nodeId?: number;
|
|
details?: Record<string, unknown>;
|
|
};
|
|
expect(payload.type).toBe('route.resolve.fail');
|
|
expect(payload.nodeId).toBe(peerNodeId);
|
|
expect(payload.details).toMatchObject({
|
|
direction: 'reverse',
|
|
reason: 'connect_error',
|
|
targetStack: 'real',
|
|
targetService: 'svc',
|
|
targetPort: closedPort,
|
|
});
|
|
|
|
bridge.close();
|
|
});
|
|
|
|
it('emits route.resolve.ok with direction=reverse on connect ack', async () => {
|
|
const mockWs = makeMockTunnelWs();
|
|
const peerNodeId = 44;
|
|
const bridge = new PilotTunnelBridge(peerNodeId, mockWs as unknown as WebSocket);
|
|
await bridge.start();
|
|
|
|
// Real local server so the dial succeeds and 'connect' fires.
|
|
const upstream = net.createServer((socket) => {
|
|
socket.write('hello-upstream');
|
|
});
|
|
await new Promise<void>((resolve) => upstream.listen(0, '127.0.0.1', () => resolve()));
|
|
const addr = upstream.address();
|
|
if (!addr || typeof addr === 'string') throw new Error('no address');
|
|
const upstreamPort = addr.port;
|
|
|
|
const localNodeId = (await import('../services/NodeRegistry')).NodeRegistry.getInstance().getDefaultNodeId();
|
|
vi.spyOn(MeshService.getInstance(), 'resolveContainerIp').mockResolvedValue('127.0.0.1');
|
|
|
|
const s = AGENT_REVERSE_ID_BASE + 13;
|
|
mockWs.emit('message', encodeJsonFrame({
|
|
t: 'tcp_open_reverse', s,
|
|
targetNodeId: localNodeId, stack: 'real', service: 'svc', port: upstreamPort,
|
|
}), false);
|
|
|
|
const ack = await waitFor(() => findAck(mockWs, s));
|
|
expect(ack.ok).toBe(true);
|
|
|
|
// Wait for the ok event to land (logActivity is sync but the connect
|
|
// callback is async relative to message handling).
|
|
await waitFor(() => logSpy.mock.calls.find(
|
|
(c: LogActivityArgs) => c[0].type === 'route.resolve.ok',
|
|
));
|
|
|
|
const okCall = logSpy.mock.calls.find(
|
|
(c: LogActivityArgs) => c[0].type === 'route.resolve.ok',
|
|
);
|
|
expect(okCall).toBeDefined();
|
|
const payload = okCall![0] as {
|
|
source: string;
|
|
level: string;
|
|
type: string;
|
|
nodeId?: number;
|
|
details?: Record<string, unknown>;
|
|
};
|
|
expect(payload.source).toBe('mesh');
|
|
expect(payload.level).toBe('info');
|
|
expect(payload.type).toBe('route.resolve.ok');
|
|
expect(payload.nodeId).toBe(peerNodeId);
|
|
expect(payload.details).toMatchObject({
|
|
direction: 'reverse',
|
|
targetStack: 'real',
|
|
targetService: 'svc',
|
|
targetPort: upstreamPort,
|
|
});
|
|
|
|
upstream.close();
|
|
bridge.close();
|
|
});
|
|
|
|
it('does NOT emit route.resolve.fail when a connected socket closes post-handshake', async () => {
|
|
const mockWs = makeMockTunnelWs();
|
|
const peerNodeId = 45;
|
|
const bridge = new PilotTunnelBridge(peerNodeId, mockWs as unknown as WebSocket);
|
|
await bridge.start();
|
|
|
|
// Real local server. Have it close the connection immediately after
|
|
// accept so the bridge socket sees 'close' (and possibly 'error') on
|
|
// an already-connected socket.
|
|
const upstream = net.createServer((socket) => {
|
|
socket.end();
|
|
});
|
|
await new Promise<void>((resolve) => upstream.listen(0, '127.0.0.1', () => resolve()));
|
|
const addr = upstream.address();
|
|
if (!addr || typeof addr === 'string') throw new Error('no address');
|
|
const upstreamPort = addr.port;
|
|
|
|
const localNodeId = (await import('../services/NodeRegistry')).NodeRegistry.getInstance().getDefaultNodeId();
|
|
vi.spyOn(MeshService.getInstance(), 'resolveContainerIp').mockResolvedValue('127.0.0.1');
|
|
|
|
const s = AGENT_REVERSE_ID_BASE + 14;
|
|
mockWs.emit('message', encodeJsonFrame({
|
|
t: 'tcp_open_reverse', s,
|
|
targetNodeId: localNodeId, stack: 'real', service: 'svc', port: upstreamPort,
|
|
}), false);
|
|
|
|
// Wait for connect (the success ack confirms onPreConnectError was removed).
|
|
const ack = await waitFor(() => findAck(mockWs, s));
|
|
expect(ack.ok).toBe(true);
|
|
|
|
// Wait for the success event to confirm the ok path fired.
|
|
await waitFor(() => logSpy.mock.calls.find(
|
|
(c: LogActivityArgs) => c[0].type === 'route.resolve.ok',
|
|
));
|
|
|
|
// Give the post-connect close a chance to fire teardown handlers.
|
|
await new Promise((r) => setTimeout(r, 100));
|
|
|
|
const failCalls = logSpy.mock.calls.filter(
|
|
(c: LogActivityArgs) => c[0].type === 'route.resolve.fail',
|
|
);
|
|
expect(failCalls).toHaveLength(0);
|
|
|
|
upstream.close();
|
|
bridge.close();
|
|
});
|
|
});
|