mirror of
https://github.com/Studio-Saelix/sencho.git
synced 2026-08-20 15:22:59 +00:00
feat(mesh): symmetric WS dial for proxy-mode mesh peers (#1066)
* chore(mesh): foundation for symmetric callback dial Adds the data-plane scaffolding that the symmetric callback dial fix builds on: - mesh_centrals table for peer-side bootstrap material - MeshCentralRegistry service (upsert/getActive/clear/markUsed/markRejected) - PilotTunnelManager kind discriminator and replaceOrRegisterProxyBridge - mesh_proxy_callback_bootstrap capability registration - MeshProxyTunnelDialer reason-tagged proxy-bridge-down events from a single tearDownBridge emission point - Reactive redial scheduler that skips idle and auth_failed reasons * feat(mesh): add reverse-direction activity log entries (closes R1-B) acceptReverseLocal now emits route.resolve.ok with direction=reverse on connect ack and route.resolve.fail with direction=reverse plus reason=container_not_found / connect_error pre-connect. Post-connect close/error stays silent. Reuses existing event types via the new details.direction discriminator so frontend filters are unaffected. * feat(mesh): add peer-to-central callback dial path (closes R1-A2) Closes the architectural gap where proxy-mode mesh peers could not re-establish their tunnel to central after any non-idle bridge teardown (idle close, network blip, central restart, peer reboot). Central remains the hub for the data plane; the change is purely about WS initiation. Symmetric WS initiation, asymmetric protocol roles. Central retains PilotTunnelBridge ownership; peer retains TcpStreamSwitchboard + reverseDialer ownership. Central bootstraps callback credentials over the first authenticated central-initiated mesh tunnel via a one-shot mesh_handshake JSON frame; peer persists the material in a new mesh_centrals SQLite table and dials central's new /api/mesh/proxy-tunnel-from-peer endpoint when local cross-node traffic needs a bridge and none is live. Mesh_tunnel JWT (HS256, signed with auth_jwt_secret) carries scope, audience, issuer (central instance id), peer api_token fingerprint, kid. Validation on inbound peer dial: algorithm pin, signature, scope, audience, instance, time bounds, node existence and mode, fingerprint match. Failures return HTTP 401 with a machine-readable reason; peer routes the response per a clear-vs-keep cache matrix. Triggers proactive bootstrap on mesh-enable and api_token rotation; central startup fans out to mesh-enabled proxy-mode nodes with mesh_stacks rows (throttled, fire-and-forget). Reactive redial on non-idle bridge loss. Capability-gated handshake send (mesh_proxy_callback_bootstrap) makes the upgrade path safe against older peers in mixed-version fleets. Adds peer-side /api/system/pilot-tunnels centralCallback diag block, bounded counter metrics for bootstrap and dial events. SENCHO_PRIMARY_URL preflight warning when unset on a central with mesh-enabled proxy nodes. Tested with unit suites for the validation chain, registry, manager, and both dialers; integration tests for bootstrap E2E (asserts protocol-role invariant), api_token rotation, instance id change, version skew, and pilot-mode regression. * fix(mesh): green CI on the symmetric callback branch Two independent CI failures, both surgical: 1. Backend tests (11 fails): four mesh test files called setupTestDb in beforeEach. setupTestDb does not reset the DatabaseService singleton, so the per-test afterEach rm of the previous tmpdir left the singleton connection pointing at a deleted file. The next beforeEach's line-55 write threw SQLITE_READONLY_DBMOVED on Linux. Windows file-lock semantics hid this locally. Hoist setupTestDb / cleanupTestDb to file-scope beforeAll / afterAll; per-test state resets stay in beforeEach. Matches the convention in the eight mesh test files that already pass. 2. CodeQL (4 high alerts): js/insufficient-password-hash flagged sha256(api_token) at four sites. The api_token is a 256-bit opaque bearer (sen_sk_-prefixed), not a human password; sha256 is the correct fingerprint primitive for binding the mesh_tunnel JWT to a specific token. Add the two production files plus the two test files that mint the fingerprint to the existing path-scoped query-filter for that rule. * fix(mesh): drop unused afterEach import and revert dead codeql config ESLint flagged afterEach as unused in mesh-central-registry.test.ts:1 after the previous commit hoisted setup/teardown to file-scope beforeAll/afterAll. Remove from the vitest import line. Revert the codeql-config.yml additions from the previous commit. The paths: sub-key under query-filters > exclude is not a documented CodeQL feature and silently no-ops. The four js/insufficient-password-hash alerts on api_token fingerprinting are tracked as dismissed false positives in the GitHub Security tab rather than via dead config.
This commit is contained in:
Anso
@@ -0,0 +1,15 @@
|
||||
import { describe, it, expect } from 'vitest';
|
||||
import { getActiveCapabilities, applyPilotModeCapabilityFilter, enableCapability } from '../services/CapabilityRegistry';
|
||||
|
||||
describe('mesh_proxy_callback_bootstrap capability', () => {
|
||||
it('is registered in the default CAPABILITIES list', () => {
|
||||
expect(getActiveCapabilities()).toContain('mesh_proxy_callback_bootstrap');
|
||||
});
|
||||
|
||||
it('is NOT filtered out in pilot mode (pilots can also be bootstrap targets)', () => {
|
||||
applyPilotModeCapabilityFilter();
|
||||
expect(getActiveCapabilities()).toContain('mesh_proxy_callback_bootstrap');
|
||||
enableCapability('host-console');
|
||||
enableCapability('self-update');
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,187 @@
|
||||
/**
|
||||
* End-to-end protocol-role invariant for the symmetric mesh callback dial.
|
||||
*
|
||||
* The whole point of Task 8 + Task 10 is the asymmetry of registration even
|
||||
* though the wire protocol becomes symmetric:
|
||||
*
|
||||
* - Central is the loopback HTTP host. Every bridge central registers
|
||||
* (pilot-initiated or peer-initiated dial-back) is a
|
||||
* `PilotTunnelBridge` parked in `PilotTunnelManager`.
|
||||
* - The peer (proxy-mode remote) never runs a `PilotTunnelBridge`; on the
|
||||
* peer side a `TcpStreamSwitchboard` rides the same WS and the local
|
||||
* `MeshService.reverseDialer` slot is held while the WS is alive.
|
||||
*
|
||||
* This file exercises the central side of that invariant end-to-end using
|
||||
* the same in-process fixture pattern as `mesh-proxy-tunnel-from-peer.test.ts`:
|
||||
* a fully signed callback JWT lands at `/api/mesh/proxy-tunnel-from-peer`,
|
||||
* and we assert what central produced is a `PilotTunnelBridge` (not a
|
||||
* switchboard) in the manager. The matching mesh_handshake side (central
|
||||
* pushing the JWT) is covered in `mesh-proxy-tunnel-dialer-handshake.test.ts`
|
||||
* and `mesh-proxy-tunnel-first-frame.test.ts`; here we lock in the registry
|
||||
* outcome that ties those two halves together.
|
||||
*/
|
||||
import http from 'http';
|
||||
import { afterAll, beforeAll, beforeEach, describe, expect, it } from 'vitest';
|
||||
import WebSocket from 'ws';
|
||||
import jwt from 'jsonwebtoken';
|
||||
import { createHash } from 'crypto';
|
||||
import type { AddressInfo } from 'net';
|
||||
import { setupTestDb, cleanupTestDb } from './helpers/setupTestDb';
|
||||
import { MeshProxyTunnelDialer } from '../services/MeshProxyTunnelDialer';
|
||||
import { MeshCentralRegistry } from '../services/MeshCentralRegistry';
|
||||
import { PilotTunnelManager } from '../services/PilotTunnelManager';
|
||||
import { PilotTunnelBridge } from '../services/PilotTunnelBridge';
|
||||
|
||||
let tmpDir: string;
|
||||
let handleMeshProxyTunnelFromPeerUpgrade: typeof import('../websocket/meshProxyTunnelFromPeer').handleMeshProxyTunnelFromPeerUpgrade;
|
||||
let DatabaseService: typeof import('../services/DatabaseService').DatabaseService;
|
||||
|
||||
interface ServerHandle {
|
||||
server: http.Server;
|
||||
port: number;
|
||||
close: () => Promise<void>;
|
||||
}
|
||||
|
||||
async function startCentralServer(): Promise<ServerHandle> {
|
||||
const server = http.createServer();
|
||||
server.on('upgrade', (req, socket, head) => {
|
||||
const pathname = new URL(req.url ?? '/', 'http://localhost').pathname;
|
||||
if (pathname === '/api/mesh/proxy-tunnel-from-peer') {
|
||||
handleMeshProxyTunnelFromPeerUpgrade(req, socket, head);
|
||||
} else {
|
||||
socket.destroy();
|
||||
}
|
||||
});
|
||||
await new Promise<void>((resolve) => server.listen(0, '127.0.0.1', () => resolve()));
|
||||
const port = (server.address() as AddressInfo).port;
|
||||
return {
|
||||
server,
|
||||
port,
|
||||
close: () => new Promise<void>((resolve) => server.close(() => resolve())),
|
||||
};
|
||||
}
|
||||
|
||||
const CANONICAL_ORIGIN = 'https://central.example.com';
|
||||
const INSTANCE_ID = 'bootstrap-integration-central';
|
||||
const PEER_API_TOKEN = 'peer-token-bootstrap';
|
||||
|
||||
let secret: string;
|
||||
let srv: ServerHandle;
|
||||
|
||||
beforeAll(async () => {
|
||||
tmpDir = await setupTestDb();
|
||||
({ handleMeshProxyTunnelFromPeerUpgrade } = await import('../websocket/meshProxyTunnelFromPeer'));
|
||||
({ DatabaseService } = await import('../services/DatabaseService'));
|
||||
srv = await startCentralServer();
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
if (srv) await srv.close();
|
||||
delete process.env.SENCHO_PRIMARY_URL;
|
||||
cleanupTestDb(tmpDir);
|
||||
});
|
||||
|
||||
beforeEach(() => {
|
||||
process.env.SENCHO_PRIMARY_URL = CANONICAL_ORIGIN;
|
||||
MeshProxyTunnelDialer.resetForTest();
|
||||
MeshCentralRegistry.resetForTest();
|
||||
PilotTunnelManager.resetForTest();
|
||||
const db = DatabaseService.getInstance();
|
||||
secret = db.getGlobalSettings().auth_jwt_secret;
|
||||
db.setSystemState('instance_id', INSTANCE_ID);
|
||||
});
|
||||
|
||||
function seedPeerNode(): number {
|
||||
const db = DatabaseService.getInstance();
|
||||
const id = db.addNode({
|
||||
name: `peer-bs-${Date.now()}-${Math.random().toString(36).slice(2, 8)}`,
|
||||
type: 'remote',
|
||||
mode: 'proxy',
|
||||
api_url: 'https://peer.example.com',
|
||||
api_token: PEER_API_TOKEN,
|
||||
compose_dir: '/tmp',
|
||||
is_default: false,
|
||||
});
|
||||
db.setNodeMeshEnabled(id, true);
|
||||
return id;
|
||||
}
|
||||
|
||||
function mintCallbackJwt(peerNodeId: number): string {
|
||||
const fp = createHash('sha256').update(PEER_API_TOKEN).digest('hex').slice(0, 16);
|
||||
const nowSec = Math.floor(Date.now() / 1000);
|
||||
return jwt.sign(
|
||||
{
|
||||
sub: String(peerNodeId),
|
||||
iss: INSTANCE_ID,
|
||||
aud: CANONICAL_ORIGIN,
|
||||
scope: 'mesh_tunnel',
|
||||
iat: nowSec,
|
||||
exp: nowSec + 3600,
|
||||
kid: 'v1',
|
||||
peer_token_fp: fp,
|
||||
},
|
||||
secret,
|
||||
{ algorithm: 'HS256' },
|
||||
);
|
||||
}
|
||||
|
||||
describe('Mesh bootstrap E2E (protocol-role invariant)', () => {
|
||||
it('after peer dial-back, central registers a PilotTunnelBridge in PilotTunnelManager', async () => {
|
||||
// This is the symmetric callback half of the protocol: the peer has
|
||||
// already received a bootstrap JWT in some prior session and is now
|
||||
// dialing back into central. On central, the resulting WS must
|
||||
// produce a PilotTunnelBridge (not a TcpStreamSwitchboard) and the
|
||||
// manager must hold it under the peer's nodeId.
|
||||
const peerNodeId = seedPeerNode();
|
||||
const token = mintCallbackJwt(peerNodeId);
|
||||
|
||||
const ws = await new Promise<WebSocket>((resolve, reject) => {
|
||||
const s = new WebSocket(`ws://127.0.0.1:${srv.port}/api/mesh/proxy-tunnel-from-peer`, {
|
||||
headers: { authorization: `Bearer ${token}` },
|
||||
});
|
||||
s.once('open', () => resolve(s));
|
||||
s.once('error', reject);
|
||||
s.once('unexpected-response', (_req, res) => reject(new Error(`upgrade rejected ${res.statusCode}`)));
|
||||
});
|
||||
// Allow the handler's async bridge.start() + manager registration to land.
|
||||
await new Promise((r) => setTimeout(r, 60));
|
||||
|
||||
const bridge = PilotTunnelManager.getInstance().getBridge(peerNodeId);
|
||||
expect(bridge).not.toBeNull();
|
||||
// The manager only narrows to MeshTunnelHandle in its public API,
|
||||
// but the underlying object must be the central-side bridge class.
|
||||
expect(bridge).toBeInstanceOf(PilotTunnelBridge);
|
||||
|
||||
try { ws.close(1000, 'test cleanup'); } catch { /* ignore */ }
|
||||
await new Promise((r) => setTimeout(r, 30));
|
||||
});
|
||||
|
||||
it('a successful dial-back persists the JWT scope on the bridge (mesh_tunnel scope)', async () => {
|
||||
// Cross-check: the JWT carries scope=mesh_tunnel, the central
|
||||
// ingress accepted it, so the bridge that lands in the manager is a
|
||||
// proxy-mode bridge (not a pilot-agent tunnel). We assert via the
|
||||
// dialer's hasBridge surface to confirm the proxy registry side
|
||||
// sees nothing (the dial-back lives in the manager, not in
|
||||
// MeshProxyTunnelDialer's outbound map).
|
||||
const peerNodeId = seedPeerNode();
|
||||
const token = mintCallbackJwt(peerNodeId);
|
||||
const ws = await new Promise<WebSocket>((resolve, reject) => {
|
||||
const s = new WebSocket(`ws://127.0.0.1:${srv.port}/api/mesh/proxy-tunnel-from-peer`, {
|
||||
headers: { authorization: `Bearer ${token}` },
|
||||
});
|
||||
s.once('open', () => resolve(s));
|
||||
s.once('error', reject);
|
||||
s.once('unexpected-response', (_req, res) => reject(new Error(`upgrade rejected ${res.statusCode}`)));
|
||||
});
|
||||
await new Promise((r) => setTimeout(r, 60));
|
||||
|
||||
// The dialer is the central-initiated outbound side. A peer-initiated
|
||||
// dial-back lands in PilotTunnelManager only; the dialer's own
|
||||
// bridges map stays empty for this nodeId.
|
||||
expect(MeshProxyTunnelDialer.getInstance().hasBridge(peerNodeId)).toBe(false);
|
||||
expect(PilotTunnelManager.getInstance().getBridge(peerNodeId)).not.toBeNull();
|
||||
|
||||
try { ws.close(1000, 'test cleanup'); } catch { /* ignore */ }
|
||||
await new Promise((r) => setTimeout(r, 30));
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,122 @@
|
||||
import { describe, it, expect, beforeAll, afterAll, beforeEach, vi } from 'vitest';
|
||||
import { setupTestDb, cleanupTestDb } from './helpers/setupTestDb';
|
||||
import { DatabaseService } from '../services/DatabaseService';
|
||||
import { MeshCentralRegistry } from '../services/MeshCentralRegistry';
|
||||
|
||||
let sharedTmpDir: string;
|
||||
beforeAll(async () => { sharedTmpDir = await setupTestDb(); });
|
||||
afterAll(() => cleanupTestDb(sharedTmpDir));
|
||||
|
||||
describe('mesh_centrals schema', () => {
|
||||
it('creates the mesh_centrals table on initSchema', () => {
|
||||
const db = DatabaseService.getInstance();
|
||||
const row = db.getDb().prepare(
|
||||
"SELECT name FROM sqlite_master WHERE type='table' AND name='mesh_centrals'"
|
||||
).get();
|
||||
expect(row).toBeDefined();
|
||||
});
|
||||
|
||||
it('mesh_centrals has the expected columns', () => {
|
||||
const db = DatabaseService.getInstance();
|
||||
const cols = db.getDb().prepare("PRAGMA table_info('mesh_centrals')").all() as Array<{ name: string }>;
|
||||
const names = cols.map(c => c.name).sort();
|
||||
expect(names).toEqual([
|
||||
'callback_jwt',
|
||||
'central_api_url',
|
||||
'central_instance_id',
|
||||
'jwt_expires_at',
|
||||
'jwt_issued_at',
|
||||
'last_bootstrap_at',
|
||||
'last_reject_reason',
|
||||
'last_rejected_at',
|
||||
'last_used_at',
|
||||
]);
|
||||
});
|
||||
});
|
||||
|
||||
describe('MeshCentralRegistry', () => {
|
||||
beforeEach(() => {
|
||||
MeshCentralRegistry.resetForTest();
|
||||
// Per-test cleanup of the mesh_centrals table so tests are order-independent.
|
||||
// The DB itself is created once per file in the file-scope beforeAll above;
|
||||
// resetting setupTestDb per test would invalidate the DatabaseService
|
||||
// singleton's open connection (SQLITE_READONLY_DBMOVED on Linux CI).
|
||||
DatabaseService.getInstance().getDb().prepare('DELETE FROM mesh_centrals').run();
|
||||
});
|
||||
|
||||
const sampleMaterial = (overrides: Partial<{
|
||||
centralInstanceId: string;
|
||||
centralApiUrl: string;
|
||||
callbackJwt: string;
|
||||
jwtIssuedAt: number;
|
||||
jwtExpiresAt: number;
|
||||
}> = {}) => ({
|
||||
centralInstanceId: 'inst-uuid-1',
|
||||
centralApiUrl: 'https://central.example.com',
|
||||
callbackJwt: 'eyJhbGciOiJIUzI1NiJ9.fake.token',
|
||||
jwtIssuedAt: 1_700_000_000,
|
||||
jwtExpiresAt: 1_700_000_000 + 365 * 24 * 3600,
|
||||
...overrides,
|
||||
});
|
||||
|
||||
it('upsert inserts a new row when none exists', () => {
|
||||
MeshCentralRegistry.getInstance().upsert(sampleMaterial());
|
||||
const row = MeshCentralRegistry.getInstance().getActive();
|
||||
expect(row?.centralInstanceId).toBe('inst-uuid-1');
|
||||
expect(row?.centralApiUrl).toBe('https://central.example.com');
|
||||
expect(row?.lastBootstrapAt).toBeGreaterThan(0);
|
||||
});
|
||||
|
||||
it('upsert overwrites when same instance id is provided', () => {
|
||||
const reg = MeshCentralRegistry.getInstance();
|
||||
reg.upsert(sampleMaterial({ callbackJwt: 'old.jwt' }));
|
||||
reg.upsert(sampleMaterial({ callbackJwt: 'new.jwt' }));
|
||||
expect(reg.getActive()?.callbackJwt).toBe('new.jwt');
|
||||
});
|
||||
|
||||
it('getActive returns most-recently-bootstrapped row when multiple exist and warns once', () => {
|
||||
const reg = MeshCentralRegistry.getInstance();
|
||||
const warn = vi.spyOn(console, 'warn').mockImplementation(() => {});
|
||||
reg.upsert(sampleMaterial({ centralInstanceId: 'old', jwtIssuedAt: 1 }));
|
||||
DatabaseService.getInstance().getDb().prepare(`
|
||||
INSERT INTO mesh_centrals VALUES (?, ?, ?, ?, ?, ?, NULL, NULL, NULL)
|
||||
`).run('new', 'https://other.example.com', 'jwt2', 2, 999, Date.now() + 1000);
|
||||
expect(reg.getActive()?.centralInstanceId).toBe('new');
|
||||
expect(warn).toHaveBeenCalledOnce();
|
||||
expect(warn.mock.calls[0][0]).toMatch(/multiple central rows/);
|
||||
reg.getActive();
|
||||
expect(warn).toHaveBeenCalledOnce();
|
||||
warn.mockRestore();
|
||||
});
|
||||
|
||||
it('getActive returns null when table is empty', () => {
|
||||
expect(MeshCentralRegistry.getInstance().getActive()).toBeNull();
|
||||
});
|
||||
|
||||
it('clearForInstance removes only the named instance', () => {
|
||||
const reg = MeshCentralRegistry.getInstance();
|
||||
reg.upsert(sampleMaterial({ centralInstanceId: 'a' }));
|
||||
reg.upsert(sampleMaterial({ centralInstanceId: 'b' }));
|
||||
reg.clearForInstance('a');
|
||||
const remaining = DatabaseService.getInstance().getDb().prepare(
|
||||
"SELECT central_instance_id FROM mesh_centrals"
|
||||
).all() as Array<{ central_instance_id: string }>;
|
||||
expect(remaining.map(r => r.central_instance_id)).toEqual(['b']);
|
||||
});
|
||||
|
||||
it('markUsed updates last_used_at, only on success', () => {
|
||||
const reg = MeshCentralRegistry.getInstance();
|
||||
reg.upsert(sampleMaterial());
|
||||
reg.markUsed('inst-uuid-1');
|
||||
expect(reg.getActive()?.lastUsedAt).toBeGreaterThan(0);
|
||||
});
|
||||
|
||||
it('markRejected stores reason and timestamp', () => {
|
||||
const reg = MeshCentralRegistry.getInstance();
|
||||
reg.upsert(sampleMaterial());
|
||||
reg.markRejected('inst-uuid-1', 'token_fingerprint_mismatch');
|
||||
const row = reg.getActive();
|
||||
expect(row?.lastRejectedAt).toBeGreaterThan(0);
|
||||
expect(row?.lastRejectReason).toBe('token_fingerprint_mismatch');
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,84 @@
|
||||
/**
|
||||
* Central instance id rotation invariant.
|
||||
*
|
||||
* `MeshCentralRegistry.upsert` is the single source of truth for the
|
||||
* locally-cached callback material. When central regenerates its
|
||||
* `instance_id` (factory reset, DB swap, container rebuild without volume),
|
||||
* the next bootstrap frame arrives with a different `centralInstanceId`. The
|
||||
* registry MUST:
|
||||
* - persist the new row,
|
||||
* - emit a single `central-instance-changed` event so subscribers (loud
|
||||
* log, dialer reset) can react.
|
||||
*
|
||||
* Failing this invariant means the peer would keep dialing back with the
|
||||
* old JWT (rejected with `instance_mismatch`) without anyone noticing.
|
||||
*/
|
||||
import { describe, it, expect, beforeAll, afterAll, beforeEach } from 'vitest';
|
||||
import { setupTestDb, cleanupTestDb } from './helpers/setupTestDb';
|
||||
import { MeshCentralRegistry } from '../services/MeshCentralRegistry';
|
||||
import { DatabaseService } from '../services/DatabaseService';
|
||||
|
||||
let tmpDir: string;
|
||||
beforeAll(async () => { tmpDir = await setupTestDb(); });
|
||||
afterAll(() => cleanupTestDb(tmpDir));
|
||||
|
||||
describe('Central instance id change', () => {
|
||||
beforeEach(() => {
|
||||
MeshCentralRegistry.resetForTest();
|
||||
// Per-test cleanup of the mesh_centrals table. The DB lives once per
|
||||
// file (beforeAll); resetting setupTestDb per test would invalidate
|
||||
// the DatabaseService singleton's connection on Linux.
|
||||
DatabaseService.getInstance().getDb().prepare('DELETE FROM mesh_centrals').run();
|
||||
});
|
||||
|
||||
it('drops the old row and accepts the new one with a central-instance-changed event', () => {
|
||||
const reg = MeshCentralRegistry.getInstance();
|
||||
reg.upsert({
|
||||
centralInstanceId: 'old-uuid',
|
||||
centralApiUrl: 'https://central.example.com',
|
||||
callbackJwt: 'jwt-old',
|
||||
jwtIssuedAt: 1,
|
||||
jwtExpiresAt: 999999,
|
||||
});
|
||||
const events: Array<{ previousInstanceId: string; newInstanceId: string }> = [];
|
||||
reg.on('central-instance-changed', (e: { previousInstanceId: string; newInstanceId: string }) => events.push(e));
|
||||
|
||||
reg.upsert({
|
||||
centralInstanceId: 'new-uuid',
|
||||
centralApiUrl: 'https://central.example.com',
|
||||
callbackJwt: 'jwt-new',
|
||||
jwtIssuedAt: 2,
|
||||
jwtExpiresAt: 999999,
|
||||
});
|
||||
|
||||
expect(events).toHaveLength(1);
|
||||
expect(events[0].previousInstanceId).toBe('old-uuid');
|
||||
expect(events[0].newInstanceId).toBe('new-uuid');
|
||||
// getActive returns the most-recently-bootstrapped row.
|
||||
expect(reg.getActive()?.centralInstanceId).toBe('new-uuid');
|
||||
expect(reg.getActive()?.callbackJwt).toBe('jwt-new');
|
||||
});
|
||||
|
||||
it('an upsert with the same instance id does NOT emit central-instance-changed', () => {
|
||||
const reg = MeshCentralRegistry.getInstance();
|
||||
reg.upsert({
|
||||
centralInstanceId: 'stable-uuid',
|
||||
centralApiUrl: 'https://central.example.com',
|
||||
callbackJwt: 'jwt-1',
|
||||
jwtIssuedAt: 1,
|
||||
jwtExpiresAt: 999999,
|
||||
});
|
||||
const events: Array<unknown> = [];
|
||||
reg.on('central-instance-changed', (e: unknown) => events.push(e));
|
||||
reg.upsert({
|
||||
centralInstanceId: 'stable-uuid',
|
||||
centralApiUrl: 'https://central.example.com',
|
||||
callbackJwt: 'jwt-2-rotated',
|
||||
jwtIssuedAt: 2,
|
||||
jwtExpiresAt: 999999,
|
||||
});
|
||||
expect(events).toHaveLength(0);
|
||||
// Same id, the JWT material is overwritten in place.
|
||||
expect(reg.getActive()?.callbackJwt).toBe('jwt-2-rotated');
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,164 @@
|
||||
/**
|
||||
* `MeshProxyTunnelDialer.maybeSendBootstrap`: capability-gated
|
||||
* `mesh_handshake` send.
|
||||
*
|
||||
* Central mints a signed `mesh_tunnel`-scoped JWT bound to the peer's
|
||||
* `api_token` fingerprint and pushes it as the first text frame on the
|
||||
* fresh WS, but only when the peer advertises the
|
||||
* `mesh_proxy_callback_bootstrap` capability AND a canonical central
|
||||
* origin (`SENCHO_PRIMARY_URL`) is configured. All other paths must
|
||||
* remain silent (no frame) so the legacy peer-side first-frame state
|
||||
* machine can fall straight through to TCP traffic.
|
||||
*/
|
||||
import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from 'vitest';
|
||||
import jwt from 'jsonwebtoken';
|
||||
import { setupTestDb, cleanupTestDb } from './helpers/setupTestDb';
|
||||
import { MeshProxyTunnelDialer } from '../services/MeshProxyTunnelDialer';
|
||||
import { NodeRegistry } from '../services/NodeRegistry';
|
||||
import { OFFLINE_META } from '../services/CapabilityRegistry';
|
||||
import { DatabaseService } from '../services/DatabaseService';
|
||||
|
||||
let tmpDir: string;
|
||||
const CENTRAL_INSTANCE_ID = 'central-instance-for-handshake-test';
|
||||
|
||||
beforeAll(async () => {
|
||||
tmpDir = await setupTestDb();
|
||||
// Seed system_state.instance_id (LicenseService.initialize() is not
|
||||
// invoked in the test harness; mesh handshake requires this value).
|
||||
DatabaseService.getInstance().setSystemState('instance_id', CENTRAL_INSTANCE_ID);
|
||||
});
|
||||
|
||||
afterAll(() => {
|
||||
cleanupTestDb(tmpDir);
|
||||
});
|
||||
|
||||
describe('MeshProxyTunnelDialer capability-gated handshake', () => {
|
||||
let nodeSeq = 0;
|
||||
|
||||
beforeEach(() => {
|
||||
MeshProxyTunnelDialer.resetForTest();
|
||||
process.env.SENCHO_PRIMARY_URL = 'https://central.example.com';
|
||||
nodeSeq += 1;
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
delete process.env.SENCHO_PRIMARY_URL;
|
||||
vi.restoreAllMocks();
|
||||
});
|
||||
|
||||
function fakeWs(): { send: ReturnType<typeof vi.fn>; sentFrames: unknown[] } {
|
||||
const sentFrames: unknown[] = [];
|
||||
return {
|
||||
send: vi.fn((data: string) => { sentFrames.push(JSON.parse(data)); }),
|
||||
sentFrames,
|
||||
};
|
||||
}
|
||||
|
||||
function seedProxyNode(): number {
|
||||
const db = DatabaseService.getInstance();
|
||||
const id = db.addNode({
|
||||
name: `peer-handshake-${nodeSeq}-${Date.now()}-${Math.random().toString(36).slice(2, 8)}`,
|
||||
type: 'remote',
|
||||
mode: 'proxy',
|
||||
api_url: 'https://peer.example.com',
|
||||
api_token: 'peer-token-for-fp',
|
||||
compose_dir: '/tmp',
|
||||
is_default: false,
|
||||
});
|
||||
db.setNodeMeshEnabled(id, true);
|
||||
return id;
|
||||
}
|
||||
|
||||
type DialerPrivate = { maybeSendBootstrap: (nodeId: number, ws: unknown) => Promise<void> };
|
||||
|
||||
it('sends mesh_handshake when peer advertises mesh_proxy_callback_bootstrap', async () => {
|
||||
vi.spyOn(NodeRegistry.getInstance(), 'fetchMetaForNode').mockResolvedValue({
|
||||
...OFFLINE_META,
|
||||
version: '0.79.0',
|
||||
capabilities: ['stacks', 'mesh_proxy_callback_bootstrap'],
|
||||
online: true,
|
||||
startedAt: Date.now(),
|
||||
});
|
||||
const id = seedProxyNode();
|
||||
const ws = fakeWs();
|
||||
await (MeshProxyTunnelDialer.getInstance() as unknown as DialerPrivate).maybeSendBootstrap(id, ws);
|
||||
expect(ws.send).toHaveBeenCalledOnce();
|
||||
expect(ws.sentFrames[0]).toMatchObject({ t: 'mesh_handshake', v: 1 });
|
||||
});
|
||||
|
||||
it('skips mesh_handshake when peer lacks the capability', async () => {
|
||||
vi.spyOn(NodeRegistry.getInstance(), 'fetchMetaForNode').mockResolvedValue({
|
||||
...OFFLINE_META,
|
||||
version: '0.78.0',
|
||||
capabilities: ['stacks'],
|
||||
online: true,
|
||||
startedAt: Date.now(),
|
||||
});
|
||||
const id = seedProxyNode();
|
||||
const ws = fakeWs();
|
||||
await (MeshProxyTunnelDialer.getInstance() as unknown as DialerPrivate).maybeSendBootstrap(id, ws);
|
||||
expect(ws.send).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('skips mesh_handshake when meta fetch returns offline shape', async () => {
|
||||
vi.spyOn(NodeRegistry.getInstance(), 'fetchMetaForNode').mockResolvedValue({
|
||||
...OFFLINE_META,
|
||||
online: false,
|
||||
});
|
||||
const id = seedProxyNode();
|
||||
const ws = fakeWs();
|
||||
await (MeshProxyTunnelDialer.getInstance() as unknown as DialerPrivate).maybeSendBootstrap(id, ws);
|
||||
expect(ws.send).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('skips mesh_handshake when SENCHO_PRIMARY_URL is unset (fail-safe)', async () => {
|
||||
delete process.env.SENCHO_PRIMARY_URL;
|
||||
vi.spyOn(NodeRegistry.getInstance(), 'fetchMetaForNode').mockResolvedValue({
|
||||
...OFFLINE_META,
|
||||
version: '0.79.0',
|
||||
capabilities: ['mesh_proxy_callback_bootstrap'],
|
||||
online: true,
|
||||
startedAt: Date.now(),
|
||||
});
|
||||
const id = seedProxyNode();
|
||||
const ws = fakeWs();
|
||||
await (MeshProxyTunnelDialer.getInstance() as unknown as DialerPrivate).maybeSendBootstrap(id, ws);
|
||||
expect(ws.send).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('signs the JWT with auth_jwt_secret using HS256 and includes the required claims', async () => {
|
||||
vi.spyOn(NodeRegistry.getInstance(), 'fetchMetaForNode').mockResolvedValue({
|
||||
...OFFLINE_META,
|
||||
version: '0.79.0',
|
||||
capabilities: ['mesh_proxy_callback_bootstrap'],
|
||||
online: true,
|
||||
startedAt: Date.now(),
|
||||
});
|
||||
const id = seedProxyNode();
|
||||
const ws = fakeWs();
|
||||
await (MeshProxyTunnelDialer.getInstance() as unknown as DialerPrivate).maybeSendBootstrap(id, ws);
|
||||
const frame = ws.sentFrames[0] as {
|
||||
meshTunnelJwt: string;
|
||||
centralApiUrl: string;
|
||||
centralInstanceId: string;
|
||||
peerNodeId: number;
|
||||
jwtExpiresAt: number;
|
||||
};
|
||||
expect(frame.centralApiUrl).toBe('https://central.example.com');
|
||||
expect(frame.centralInstanceId).toBe(CENTRAL_INSTANCE_ID);
|
||||
expect(frame.peerNodeId).toBe(id);
|
||||
expect(typeof frame.jwtExpiresAt).toBe('number');
|
||||
const decoded = jwt.decode(frame.meshTunnelJwt, { complete: true });
|
||||
expect(decoded?.header.alg).toBe('HS256');
|
||||
const payload = decoded?.payload as Record<string, unknown>;
|
||||
expect(payload).toMatchObject({
|
||||
sub: String(id),
|
||||
iss: CENTRAL_INSTANCE_ID,
|
||||
aud: 'https://central.example.com',
|
||||
scope: 'mesh_tunnel',
|
||||
kid: 'v1',
|
||||
});
|
||||
expect(typeof payload.peer_token_fp).toBe('string');
|
||||
expect((payload.peer_token_fp as string).length).toBe(16);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,64 @@
|
||||
import { describe, it, expect, beforeEach, vi } from 'vitest';
|
||||
import { EventEmitter } from 'events';
|
||||
import { MeshProxyTunnelDialer } from '../services/MeshProxyTunnelDialer';
|
||||
|
||||
type Reason = 'idle' | 'remote_closed' | 'network_error' | 'protocol_error' | 'auth_failed';
|
||||
|
||||
describe('MeshProxyTunnelDialer reason-tagged proxy-bridge-down', () => {
|
||||
beforeEach(() => MeshProxyTunnelDialer.resetForTest());
|
||||
|
||||
it('emits reason="idle" when runIdleCheck closes a bridge', () => {
|
||||
const dialer = MeshProxyTunnelDialer.resetForTest(1);
|
||||
const reasons: Reason[] = [];
|
||||
dialer.on('proxy-bridge-down', (_id: number, reason: Reason) => reasons.push(reason));
|
||||
const fakeBridge = new EventEmitter() as unknown as { close: ReturnType<typeof vi.fn>; getActiveStreamCount: () => number };
|
||||
fakeBridge.close = vi.fn();
|
||||
fakeBridge.getActiveStreamCount = () => 0;
|
||||
(dialer as unknown as { bridges: Map<number, unknown>; idleSince: Map<number, number> }).bridges.set(7, fakeBridge);
|
||||
(dialer as unknown as { bridges: Map<number, unknown>; idleSince: Map<number, number> }).idleSince.set(7, Date.now() - 1000);
|
||||
(dialer as unknown as { runIdleCheck: () => void }).runIdleCheck();
|
||||
expect(reasons).toEqual(['idle']);
|
||||
});
|
||||
|
||||
it('emits reason="remote_closed" when bridge fires "closed" with code 1000', () => {
|
||||
const dialer = MeshProxyTunnelDialer.resetForTest();
|
||||
const reasons: Reason[] = [];
|
||||
dialer.on('proxy-bridge-down', (_id: number, reason: Reason) => reasons.push(reason));
|
||||
const fakeBridge = new EventEmitter() as unknown as EventEmitter & { close: ReturnType<typeof vi.fn> };
|
||||
fakeBridge.close = vi.fn();
|
||||
(dialer as unknown as { bridges: Map<number, unknown> }).bridges.set(7, fakeBridge);
|
||||
// Attach the close listener via the dialer's private helper, mirroring
|
||||
// what `dial()` does when a real bridge is registered. This keeps the
|
||||
// test focused on the close-code classification + tearDownBridge path
|
||||
// without spinning up a real WebSocket.
|
||||
(dialer as unknown as { attachBridgeCloseListener: (id: number, b: EventEmitter) => void })
|
||||
.attachBridgeCloseListener(7, fakeBridge as unknown as EventEmitter);
|
||||
(fakeBridge as unknown as EventEmitter).emit('closed', { code: 1000, reason: 'normal' });
|
||||
expect(reasons).toEqual(['remote_closed']);
|
||||
});
|
||||
});
|
||||
|
||||
describe('MeshProxyTunnelDialer reactive redial filter', () => {
|
||||
beforeEach(() => MeshProxyTunnelDialer.resetForTest());
|
||||
|
||||
it('does NOT schedule redial after reason="idle"', () => {
|
||||
const dialer = MeshProxyTunnelDialer.resetForTest();
|
||||
const scheduleSpy = vi.spyOn(dialer as unknown as { scheduleReactiveRedial: (id: number) => void }, 'scheduleReactiveRedial').mockImplementation(() => {});
|
||||
dialer.emit('proxy-bridge-down', 7, 'idle');
|
||||
expect(scheduleSpy).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('does NOT schedule redial after reason="auth_failed"', () => {
|
||||
const dialer = MeshProxyTunnelDialer.resetForTest();
|
||||
const scheduleSpy = vi.spyOn(dialer as unknown as { scheduleReactiveRedial: (id: number) => void }, 'scheduleReactiveRedial').mockImplementation(() => {});
|
||||
dialer.emit('proxy-bridge-down', 7, 'auth_failed');
|
||||
expect(scheduleSpy).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('schedules redial after reason="remote_closed"', () => {
|
||||
const dialer = MeshProxyTunnelDialer.resetForTest();
|
||||
const scheduleSpy = vi.spyOn(dialer as unknown as { scheduleReactiveRedial: (id: number) => void }, 'scheduleReactiveRedial').mockImplementation(() => {});
|
||||
dialer.emit('proxy-bridge-down', 7, 'remote_closed');
|
||||
expect(scheduleSpy).toHaveBeenCalledWith(7);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,187 @@
|
||||
/**
|
||||
* `meshProxyTunnel.ts` first-frame state machine.
|
||||
*
|
||||
* The PEER side of a proxy-mode mesh tunnel optionally consumes a
|
||||
* `mesh_handshake` JSON frame as the FIRST text frame from central,
|
||||
* persists the bootstrap material via MeshCentralRegistry, then yields
|
||||
* control to the existing TcpStreamSwitchboard. Out-of-order or
|
||||
* malformed handshake frames close the WS with code 1008.
|
||||
*/
|
||||
import http from 'http';
|
||||
import { afterAll, beforeAll, beforeEach, describe, expect, it } from 'vitest';
|
||||
import WebSocket from 'ws';
|
||||
import type { AddressInfo } from 'net';
|
||||
import { setupTestDb, cleanupTestDb } from './helpers/setupTestDb';
|
||||
|
||||
let tmpDir: string;
|
||||
let handleMeshProxyTunnel: typeof import('../websocket/meshProxyTunnel').handleMeshProxyTunnel;
|
||||
let MeshService: typeof import('../services/MeshService').MeshService;
|
||||
let MeshCentralRegistry: typeof import('../services/MeshCentralRegistry').MeshCentralRegistry;
|
||||
let DatabaseService: typeof import('../services/DatabaseService').DatabaseService;
|
||||
|
||||
interface ServerHandle {
|
||||
server: http.Server;
|
||||
port: number;
|
||||
close: () => Promise<void>;
|
||||
}
|
||||
|
||||
async function startServer(): Promise<ServerHandle> {
|
||||
const server = http.createServer();
|
||||
server.on('upgrade', (req, socket, head) => {
|
||||
const pathname = new URL(req.url ?? '/', 'http://localhost').pathname;
|
||||
if (pathname === '/api/mesh/proxy-tunnel') {
|
||||
void handleMeshProxyTunnel(req, socket, head);
|
||||
} else {
|
||||
socket.destroy();
|
||||
}
|
||||
});
|
||||
await new Promise<void>((resolve) => server.listen(0, '127.0.0.1', () => resolve()));
|
||||
const port = (server.address() as AddressInfo).port;
|
||||
return {
|
||||
server,
|
||||
port,
|
||||
close: () => new Promise<void>((resolve) => server.close(() => resolve())),
|
||||
};
|
||||
}
|
||||
|
||||
function dialTunnel(port: number, query: string = ''): Promise<WebSocket> {
|
||||
return new Promise((resolve, reject) => {
|
||||
const ws = new WebSocket(`ws://127.0.0.1:${port}/api/mesh/proxy-tunnel${query}`);
|
||||
ws.once('open', () => resolve(ws));
|
||||
ws.once('error', reject);
|
||||
});
|
||||
}
|
||||
|
||||
function makeHandshakeFrame(overrides: Partial<{
|
||||
centralInstanceId: string;
|
||||
centralApiUrl: string;
|
||||
meshTunnelJwt: string;
|
||||
jwtExpiresAt: number;
|
||||
peerNodeId: number;
|
||||
}> = {}): string {
|
||||
return JSON.stringify({
|
||||
t: 'mesh_handshake',
|
||||
v: 1,
|
||||
peerNodeId: overrides.peerNodeId ?? 7,
|
||||
centralInstanceId: overrides.centralInstanceId ?? 'central-instance-abc',
|
||||
centralApiUrl: overrides.centralApiUrl ?? 'https://central.example.test',
|
||||
meshTunnelJwt: overrides.meshTunnelJwt ?? 'jwt.token.body',
|
||||
jwtExpiresAt: overrides.jwtExpiresAt ?? Math.floor(Date.now() / 1000) + 3600,
|
||||
});
|
||||
}
|
||||
|
||||
beforeAll(async () => {
|
||||
tmpDir = await setupTestDb();
|
||||
({ handleMeshProxyTunnel } = await import('../websocket/meshProxyTunnel'));
|
||||
({ MeshService } = await import('../services/MeshService'));
|
||||
({ MeshCentralRegistry } = await import('../services/MeshCentralRegistry'));
|
||||
({ DatabaseService } = await import('../services/DatabaseService'));
|
||||
});
|
||||
|
||||
afterAll(() => {
|
||||
cleanupTestDb(tmpDir);
|
||||
});
|
||||
|
||||
beforeEach(() => {
|
||||
delete process.env.SENCHO_MODE;
|
||||
MeshService.getInstance().setReverseDialer(null);
|
||||
MeshService.getInstance().setProxyTunnelSelfCentralNodeId(null);
|
||||
MeshCentralRegistry.resetForTest();
|
||||
DatabaseService.getInstance().getDb().prepare('DELETE FROM mesh_centrals').run();
|
||||
});
|
||||
|
||||
describe('meshProxyTunnel first-frame state machine', () => {
|
||||
it('consumes a mesh_handshake first frame and persists material via MeshCentralRegistry', async () => {
|
||||
const srv = await startServer();
|
||||
try {
|
||||
const ws = await dialTunnel(srv.port, '?nodeId=7');
|
||||
await new Promise((r) => setTimeout(r, 20));
|
||||
|
||||
const expiresAt = Math.floor(Date.now() / 1000) + 7200;
|
||||
ws.send(makeHandshakeFrame({
|
||||
centralInstanceId: 'central-instance-xyz',
|
||||
centralApiUrl: 'https://central.example.test/',
|
||||
meshTunnelJwt: 'callback.jwt.value',
|
||||
jwtExpiresAt: expiresAt,
|
||||
}));
|
||||
await new Promise((r) => setTimeout(r, 30));
|
||||
|
||||
const row = MeshCentralRegistry.getInstance().getActive();
|
||||
expect(row).not.toBeNull();
|
||||
expect(row?.centralInstanceId).toBe('central-instance-xyz');
|
||||
// Trailing slash should be stripped to keep the persisted URL canonical.
|
||||
expect(row?.centralApiUrl).toBe('https://central.example.test');
|
||||
expect(row?.callbackJwt).toBe('callback.jwt.value');
|
||||
expect(row?.jwtExpiresAt).toBe(expiresAt);
|
||||
|
||||
ws.close(1000, 'test cleanup');
|
||||
await new Promise((r) => setTimeout(r, 30));
|
||||
} finally {
|
||||
await srv.close();
|
||||
}
|
||||
});
|
||||
|
||||
it('passes through if first frame is not a mesh_handshake (delegates to switchboard)', async () => {
|
||||
const srv = await startServer();
|
||||
try {
|
||||
const ws = await dialTunnel(srv.port, '?nodeId=7');
|
||||
await new Promise((r) => setTimeout(r, 20));
|
||||
|
||||
// Send a known non-handshake JSON frame (tcp_close for a non-existent
|
||||
// stream is a no-op the switchboard accepts without side effects).
|
||||
ws.send(JSON.stringify({ t: 'tcp_close', s: 999999 }));
|
||||
await new Promise((r) => setTimeout(r, 30));
|
||||
|
||||
// No handshake material should have been persisted.
|
||||
expect(MeshCentralRegistry.getInstance().getActive()).toBeNull();
|
||||
// WS should still be open (no protocol error close).
|
||||
expect(ws.readyState).toBe(WebSocket.OPEN);
|
||||
|
||||
ws.close(1000, 'test cleanup');
|
||||
await new Promise((r) => setTimeout(r, 30));
|
||||
} finally {
|
||||
await srv.close();
|
||||
}
|
||||
});
|
||||
|
||||
it('closes the WS as protocol error when mesh_handshake arrives after a non-handshake frame', async () => {
|
||||
const srv = await startServer();
|
||||
try {
|
||||
const ws = await dialTunnel(srv.port, '?nodeId=7');
|
||||
await new Promise((r) => setTimeout(r, 20));
|
||||
|
||||
// First a non-handshake frame, then the handshake.
|
||||
ws.send(JSON.stringify({ t: 'tcp_close', s: 999999 }));
|
||||
await new Promise((r) => setTimeout(r, 10));
|
||||
|
||||
const closeInfo = new Promise<{ code: number; reason: string }>((resolve) => {
|
||||
ws.once('close', (code, reason) => resolve({ code, reason: reason.toString() }));
|
||||
});
|
||||
ws.send(makeHandshakeFrame());
|
||||
const info = await closeInfo;
|
||||
expect(info.code).toBe(1008);
|
||||
expect(MeshCentralRegistry.getInstance().getActive()).toBeNull();
|
||||
} finally {
|
||||
await srv.close();
|
||||
}
|
||||
});
|
||||
|
||||
it('rejects malformed mesh_handshake (missing required fields) and closes WS as protocol error', async () => {
|
||||
const srv = await startServer();
|
||||
try {
|
||||
const ws = await dialTunnel(srv.port, '?nodeId=7');
|
||||
await new Promise((r) => setTimeout(r, 20));
|
||||
|
||||
const closeInfo = new Promise<{ code: number; reason: string }>((resolve) => {
|
||||
ws.once('close', (code, reason) => resolve({ code, reason: reason.toString() }));
|
||||
});
|
||||
// Recognized 't' discriminator but missing required fields.
|
||||
ws.send(JSON.stringify({ t: 'mesh_handshake', v: 1 }));
|
||||
const info = await closeInfo;
|
||||
expect(info.code).toBe(1008);
|
||||
expect(MeshCentralRegistry.getInstance().getActive()).toBeNull();
|
||||
} finally {
|
||||
await srv.close();
|
||||
}
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,282 @@
|
||||
/**
|
||||
* `meshProxyTunnelFromPeer.ts`: central-side ingress for peer-initiated
|
||||
* dial-back tunnels. Validates the chain of JWT claims and node-state
|
||||
* preconditions that the peer's `mesh_tunnel` bootstrap token must satisfy
|
||||
* before the upgrade succeeds and a proxy bridge is registered.
|
||||
*
|
||||
* The chain (in order): algorithm whitelist, signature, scope, audience
|
||||
* (= SENCHO_PRIMARY_URL), issuer (= central instance_id), exp, iat clock
|
||||
* skew (60s), node existence, mode==proxy, api_token fingerprint match.
|
||||
* Every failure point returns HTTP 401 with a JSON `{reason}` body using
|
||||
* a stable machine-readable code; the happy path constructs a
|
||||
* `PilotTunnelBridge` and registers it via
|
||||
* `PilotTunnelManager.replaceOrRegisterProxyBridge`.
|
||||
*/
|
||||
import http from 'http';
|
||||
import { afterAll, beforeAll, beforeEach, describe, expect, it } from 'vitest';
|
||||
import WebSocket from 'ws';
|
||||
import jwt from 'jsonwebtoken';
|
||||
import { createHash } from 'crypto';
|
||||
import type { AddressInfo } from 'net';
|
||||
import { setupTestDb, cleanupTestDb } from './helpers/setupTestDb';
|
||||
|
||||
let tmpDir: string;
|
||||
let handleMeshProxyTunnelFromPeerUpgrade: typeof import('../websocket/meshProxyTunnelFromPeer').handleMeshProxyTunnelFromPeerUpgrade;
|
||||
let DatabaseService: typeof import('../services/DatabaseService').DatabaseService;
|
||||
let PilotTunnelManager: typeof import('../services/PilotTunnelManager').PilotTunnelManager;
|
||||
|
||||
interface ServerHandle {
|
||||
server: http.Server;
|
||||
port: number;
|
||||
close: () => Promise<void>;
|
||||
}
|
||||
|
||||
async function startServer(): Promise<ServerHandle> {
|
||||
const server = http.createServer();
|
||||
server.on('upgrade', (req, socket, head) => {
|
||||
const pathname = new URL(req.url ?? '/', 'http://localhost').pathname;
|
||||
if (pathname === '/api/mesh/proxy-tunnel-from-peer') {
|
||||
handleMeshProxyTunnelFromPeerUpgrade(req, socket, head);
|
||||
} else {
|
||||
socket.destroy();
|
||||
}
|
||||
});
|
||||
await new Promise<void>((resolve) => server.listen(0, '127.0.0.1', () => resolve()));
|
||||
const port = (server.address() as AddressInfo).port;
|
||||
return {
|
||||
server,
|
||||
port,
|
||||
close: () => new Promise<void>((resolve) => server.close(() => resolve())),
|
||||
};
|
||||
}
|
||||
|
||||
interface UpgradeOutcome {
|
||||
kind: 'open' | 'unexpected' | 'error';
|
||||
status?: number;
|
||||
body?: string;
|
||||
ws?: WebSocket;
|
||||
}
|
||||
|
||||
function attemptUpgrade(port: number, token: string): Promise<UpgradeOutcome> {
|
||||
return new Promise((resolve) => {
|
||||
const ws = new WebSocket(`ws://127.0.0.1:${port}/api/mesh/proxy-tunnel-from-peer`, {
|
||||
headers: { authorization: `Bearer ${token}` },
|
||||
});
|
||||
const timer = setTimeout(() => resolve({ kind: 'error' }), 3000);
|
||||
ws.once('open', () => {
|
||||
clearTimeout(timer);
|
||||
resolve({ kind: 'open', ws });
|
||||
});
|
||||
ws.once('unexpected-response', (_req, res) => {
|
||||
const chunks: Buffer[] = [];
|
||||
res.on('data', (c: Buffer) => chunks.push(c));
|
||||
res.on('end', () => {
|
||||
clearTimeout(timer);
|
||||
resolve({ kind: 'unexpected', status: res.statusCode, body: Buffer.concat(chunks).toString('utf8') });
|
||||
});
|
||||
});
|
||||
ws.once('error', () => {
|
||||
// 'unexpected-response' fires first; 'error' here is the
|
||||
// post-handshake failure ws raises after the server destroys
|
||||
// the socket. Already resolved above.
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
function parseReason(body: string | undefined): string | null {
|
||||
if (!body) return null;
|
||||
try {
|
||||
const parsed = JSON.parse(body) as { reason?: unknown };
|
||||
return typeof parsed.reason === 'string' ? parsed.reason : null;
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
const CANONICAL_ORIGIN = 'https://central.example.com';
|
||||
const INSTANCE_ID = 'test-central-instance';
|
||||
const PEER_API_TOKEN = 'peer-token-123';
|
||||
|
||||
let secret: string;
|
||||
let peerNodeId: number;
|
||||
let srv: ServerHandle;
|
||||
|
||||
beforeAll(async () => {
|
||||
tmpDir = await setupTestDb();
|
||||
({ handleMeshProxyTunnelFromPeerUpgrade } = await import('../websocket/meshProxyTunnelFromPeer'));
|
||||
({ DatabaseService } = await import('../services/DatabaseService'));
|
||||
({ PilotTunnelManager } = await import('../services/PilotTunnelManager'));
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
process.env.SENCHO_PRIMARY_URL = CANONICAL_ORIGIN;
|
||||
const db = DatabaseService.getInstance();
|
||||
secret = db.getGlobalSettings().auth_jwt_secret;
|
||||
db.setSystemState('instance_id', INSTANCE_ID);
|
||||
// Unique peer per test to avoid PilotTunnelManager collisions across
|
||||
// the 12 cases (one happy path actually registers a bridge).
|
||||
peerNodeId = db.addNode({
|
||||
name: `peer-${Date.now()}-${Math.random().toString(36).slice(2, 8)}`,
|
||||
type: 'remote',
|
||||
mode: 'proxy',
|
||||
api_url: 'https://peer.example.com',
|
||||
api_token: PEER_API_TOKEN,
|
||||
compose_dir: '/tmp',
|
||||
is_default: false,
|
||||
});
|
||||
db.setNodeMeshEnabled(peerNodeId, true);
|
||||
if (!srv) srv = await startServer();
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
if (srv) await srv.close();
|
||||
delete process.env.SENCHO_PRIMARY_URL;
|
||||
cleanupTestDb(tmpDir);
|
||||
});
|
||||
|
||||
function expectedFp(token: string = PEER_API_TOKEN): string {
|
||||
return createHash('sha256').update(token).digest('hex').slice(0, 16);
|
||||
}
|
||||
|
||||
function makeJwt(overrides: Partial<Record<string, unknown>> = {}, signOpts: { alg?: jwt.Algorithm; secret?: string } = {}): string {
|
||||
const payload: Record<string, unknown> = {
|
||||
sub: String(peerNodeId),
|
||||
iss: INSTANCE_ID,
|
||||
aud: CANONICAL_ORIGIN,
|
||||
scope: 'mesh_tunnel',
|
||||
iat: Math.floor(Date.now() / 1000),
|
||||
exp: Math.floor(Date.now() / 1000) + 3600,
|
||||
kid: 'v1',
|
||||
peer_token_fp: expectedFp(),
|
||||
...overrides,
|
||||
};
|
||||
return jwt.sign(payload, signOpts.secret ?? secret, { algorithm: signOpts.alg ?? 'HS256' });
|
||||
}
|
||||
|
||||
describe('/api/mesh/proxy-tunnel-from-peer validation chain', () => {
|
||||
it('rejects alg=none (algorithm_mismatch)', async () => {
|
||||
// jsonwebtoken refuses to sign with alg=none unless explicitly
|
||||
// enabled and given a null secret; we build the token manually so
|
||||
// the test exercises the central's defence, not the library's.
|
||||
const header = Buffer.from(JSON.stringify({ alg: 'none', typ: 'JWT' })).toString('base64url');
|
||||
const body = Buffer.from(JSON.stringify({
|
||||
sub: String(peerNodeId), iss: INSTANCE_ID, aud: CANONICAL_ORIGIN,
|
||||
scope: 'mesh_tunnel', iat: Math.floor(Date.now() / 1000),
|
||||
exp: Math.floor(Date.now() / 1000) + 3600, peer_token_fp: expectedFp(),
|
||||
})).toString('base64url');
|
||||
const token = `${header}.${body}.`;
|
||||
const outcome = await attemptUpgrade(srv.port, token);
|
||||
expect(outcome.kind).toBe('unexpected');
|
||||
expect(outcome.status).toBe(401);
|
||||
expect(parseReason(outcome.body)).toBe('algorithm_mismatch');
|
||||
});
|
||||
|
||||
it('rejects alg=RS256 (algorithm_mismatch)', async () => {
|
||||
const { generateKeyPairSync } = await import('crypto');
|
||||
const { privateKey } = generateKeyPairSync('rsa', { modulusLength: 2048 });
|
||||
const token = jwt.sign({
|
||||
sub: String(peerNodeId), iss: INSTANCE_ID, aud: CANONICAL_ORIGIN,
|
||||
scope: 'mesh_tunnel', exp: Math.floor(Date.now() / 1000) + 3600,
|
||||
peer_token_fp: expectedFp(),
|
||||
}, privateKey, { algorithm: 'RS256' });
|
||||
const outcome = await attemptUpgrade(srv.port, token);
|
||||
expect(outcome.kind).toBe('unexpected');
|
||||
expect(outcome.status).toBe(401);
|
||||
expect(parseReason(outcome.body)).toBe('algorithm_mismatch');
|
||||
});
|
||||
|
||||
it('rejects bad signature (signature_invalid)', async () => {
|
||||
const token = makeJwt({}, { secret: 'wrong-secret-not-the-real-one' });
|
||||
const outcome = await attemptUpgrade(srv.port, token);
|
||||
expect(outcome.kind).toBe('unexpected');
|
||||
expect(outcome.status).toBe(401);
|
||||
expect(parseReason(outcome.body)).toBe('signature_invalid');
|
||||
});
|
||||
|
||||
it('rejects scope mismatch', async () => {
|
||||
const token = makeJwt({ scope: 'pilot_tunnel' });
|
||||
const outcome = await attemptUpgrade(srv.port, token);
|
||||
expect(outcome.kind).toBe('unexpected');
|
||||
expect(outcome.status).toBe(401);
|
||||
expect(parseReason(outcome.body)).toBe('scope_mismatch');
|
||||
});
|
||||
|
||||
it('rejects audience mismatch', async () => {
|
||||
const token = makeJwt({ aud: 'https://other.example.com' });
|
||||
const outcome = await attemptUpgrade(srv.port, token);
|
||||
expect(outcome.kind).toBe('unexpected');
|
||||
expect(outcome.status).toBe(401);
|
||||
expect(parseReason(outcome.body)).toBe('audience_mismatch');
|
||||
});
|
||||
|
||||
it('rejects issuer mismatch', async () => {
|
||||
const token = makeJwt({ iss: 'wrong-instance-id' });
|
||||
const outcome = await attemptUpgrade(srv.port, token);
|
||||
expect(outcome.kind).toBe('unexpected');
|
||||
expect(outcome.status).toBe(401);
|
||||
expect(parseReason(outcome.body)).toBe('instance_mismatch');
|
||||
});
|
||||
|
||||
it('rejects expired token (stale)', async () => {
|
||||
const token = makeJwt({ exp: Math.floor(Date.now() / 1000) - 10 });
|
||||
const outcome = await attemptUpgrade(srv.port, token);
|
||||
expect(outcome.kind).toBe('unexpected');
|
||||
expect(outcome.status).toBe(401);
|
||||
// jsonwebtoken throws on expired tokens before our exp check sees it,
|
||||
// so the rejection surfaces as signature_invalid via the verify catch.
|
||||
// The contract: stale tokens are rejected with a 401 and some
|
||||
// deterministic reason code; accept either of the two equivalent
|
||||
// failures since both convey "stale credential" to the operator.
|
||||
const reason = parseReason(outcome.body);
|
||||
expect(['stale', 'signature_invalid']).toContain(reason);
|
||||
});
|
||||
|
||||
it('rejects clock-skewed token (clock_skew)', async () => {
|
||||
const token = makeJwt({ iat: Math.floor(Date.now() / 1000) + 600 });
|
||||
const outcome = await attemptUpgrade(srv.port, token);
|
||||
expect(outcome.kind).toBe('unexpected');
|
||||
expect(outcome.status).toBe(401);
|
||||
expect(parseReason(outcome.body)).toBe('clock_skew');
|
||||
});
|
||||
|
||||
it('rejects missing node (node_deleted)', async () => {
|
||||
const token = makeJwt({ sub: '999999' });
|
||||
const outcome = await attemptUpgrade(srv.port, token);
|
||||
expect(outcome.kind).toBe('unexpected');
|
||||
expect(outcome.status).toBe(401);
|
||||
expect(parseReason(outcome.body)).toBe('node_deleted');
|
||||
});
|
||||
|
||||
it('rejects mode mismatch', async () => {
|
||||
DatabaseService.getInstance().updateNode(peerNodeId, { mode: 'pilot_agent' });
|
||||
const token = makeJwt();
|
||||
const outcome = await attemptUpgrade(srv.port, token);
|
||||
expect(outcome.kind).toBe('unexpected');
|
||||
expect(outcome.status).toBe(401);
|
||||
expect(parseReason(outcome.body)).toBe('mode_mismatch');
|
||||
});
|
||||
|
||||
it('rejects token fingerprint mismatch', async () => {
|
||||
const token = makeJwt();
|
||||
// Rotate the api_token after minting; the JWT now carries the
|
||||
// fingerprint of the old token, but getNode returns the new one.
|
||||
DatabaseService.getInstance().updateNode(peerNodeId, { api_token: 'rotated-token-xyz' });
|
||||
const outcome = await attemptUpgrade(srv.port, token);
|
||||
expect(outcome.kind).toBe('unexpected');
|
||||
expect(outcome.status).toBe(401);
|
||||
expect(parseReason(outcome.body)).toBe('token_fingerprint_mismatch');
|
||||
});
|
||||
|
||||
it('accepts a fully valid token and registers a proxy bridge', async () => {
|
||||
const token = makeJwt();
|
||||
const outcome = await attemptUpgrade(srv.port, token);
|
||||
expect(outcome.kind).toBe('open');
|
||||
// Give the bridge.start() microtasks a moment to land and register.
|
||||
await new Promise((r) => setTimeout(r, 50));
|
||||
const bridge = PilotTunnelManager.getInstance().getBridge(peerNodeId);
|
||||
expect(bridge).not.toBeNull();
|
||||
try { outcome.ws?.close(1000, 'test cleanup'); } catch { /* ignore */ }
|
||||
// Allow the manager's 'closed' handler to remove the bridge entry.
|
||||
await new Promise((r) => setTimeout(r, 30));
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,148 @@
|
||||
/**
|
||||
* `MeshService.proactiveBootstrapFanout` (Trigger 3).
|
||||
*
|
||||
* At central startup, after `MeshService.start()` finishes, iterate the
|
||||
* mesh-enabled proxy-mode nodes that have at least one `mesh_stacks` row
|
||||
* and call `MeshProxyTunnelDialer.ensureBridge(nodeId)` on each. This
|
||||
* proactively re-establishes the central->peer bridges so the
|
||||
* capability-gated handshake can mint and ship bootstrap material to any
|
||||
* peer that just came online or just got upgraded to v0.79+.
|
||||
*
|
||||
* Concurrency 4, 250ms stagger, fire-and-forget. Failures do not abort
|
||||
* the fan-out.
|
||||
*/
|
||||
import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from 'vitest';
|
||||
import { setupTestDb, cleanupTestDb } from './helpers/setupTestDb';
|
||||
import { MeshService } from '../services/MeshService';
|
||||
import { MeshProxyTunnelDialer } from '../services/MeshProxyTunnelDialer';
|
||||
import { DatabaseService } from '../services/DatabaseService';
|
||||
|
||||
let tmpDir: string;
|
||||
|
||||
beforeAll(async () => {
|
||||
tmpDir = await setupTestDb();
|
||||
});
|
||||
|
||||
afterAll(() => {
|
||||
cleanupTestDb(tmpDir);
|
||||
});
|
||||
|
||||
function uniqueSuffix(): string {
|
||||
return `${Date.now()}-${Math.random().toString(36).slice(2, 10)}`;
|
||||
}
|
||||
|
||||
function seedProxyNode(meshEnabled: boolean): number {
|
||||
const db = DatabaseService.getInstance();
|
||||
const id = db.addNode({
|
||||
name: `peer-fanout-${uniqueSuffix()}`,
|
||||
type: 'remote',
|
||||
mode: 'proxy',
|
||||
api_url: `https://peer-${uniqueSuffix()}.example.com`,
|
||||
api_token: `tok-${uniqueSuffix()}`,
|
||||
compose_dir: '/tmp',
|
||||
is_default: false,
|
||||
});
|
||||
if (meshEnabled) {
|
||||
db.setNodeMeshEnabled(id, true);
|
||||
}
|
||||
return id;
|
||||
}
|
||||
|
||||
function insertMeshStack(nodeId: number, stackName: string): void {
|
||||
const db = DatabaseService.getInstance();
|
||||
db.getDb().prepare(
|
||||
'INSERT INTO mesh_stacks (node_id, stack_name, created_at, created_by) VALUES (?, ?, ?, ?)',
|
||||
).run(nodeId, stackName, Date.now(), 'test');
|
||||
}
|
||||
|
||||
function clearNodesAndMeshStacks(): void {
|
||||
const db = DatabaseService.getInstance().getDb();
|
||||
db.prepare('DELETE FROM mesh_stacks').run();
|
||||
db.prepare('DELETE FROM nodes WHERE is_default = 0').run();
|
||||
}
|
||||
|
||||
describe('MeshService.proactiveBootstrapFanout (Trigger 3)', () => {
|
||||
beforeEach(() => {
|
||||
clearNodesAndMeshStacks();
|
||||
MeshProxyTunnelDialer.resetForTest();
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
vi.restoreAllMocks();
|
||||
});
|
||||
|
||||
it('iterates only mesh-enabled proxy-mode nodes that have mesh_stacks rows', async () => {
|
||||
const calls: number[] = [];
|
||||
vi.spyOn(MeshProxyTunnelDialer.getInstance(), 'ensureBridge')
|
||||
.mockImplementation(async (id: number) => {
|
||||
calls.push(id);
|
||||
return null;
|
||||
});
|
||||
|
||||
// Eligible: mesh-enabled proxy node with a mesh_stacks row.
|
||||
const eligible = seedProxyNode(true);
|
||||
insertMeshStack(eligible, `stack-eligible-${uniqueSuffix()}`);
|
||||
|
||||
// Ineligible: mesh-enabled proxy node WITHOUT a mesh_stacks row.
|
||||
seedProxyNode(true);
|
||||
|
||||
// Ineligible: mesh-disabled proxy node WITH a mesh_stacks row.
|
||||
const meshOff = seedProxyNode(false);
|
||||
insertMeshStack(meshOff, `stack-meshoff-${uniqueSuffix()}`);
|
||||
|
||||
await (MeshService.getInstance() as unknown as {
|
||||
proactiveBootstrapFanout: () => Promise<void>;
|
||||
}).proactiveBootstrapFanout();
|
||||
|
||||
expect(calls).toEqual([eligible]);
|
||||
});
|
||||
|
||||
it('throttles to concurrency 4', async () => {
|
||||
let inflight = 0;
|
||||
let maxInflight = 0;
|
||||
vi.spyOn(MeshProxyTunnelDialer.getInstance(), 'ensureBridge')
|
||||
.mockImplementation(async () => {
|
||||
inflight++;
|
||||
if (inflight > maxInflight) maxInflight = inflight;
|
||||
await new Promise((r) => setTimeout(r, 10));
|
||||
inflight--;
|
||||
return null;
|
||||
});
|
||||
|
||||
for (let i = 0; i < 12; i++) {
|
||||
const id = seedProxyNode(true);
|
||||
insertMeshStack(id, `stack-${i}-${uniqueSuffix()}`);
|
||||
}
|
||||
|
||||
await (MeshService.getInstance() as unknown as {
|
||||
proactiveBootstrapFanout: () => Promise<void>;
|
||||
}).proactiveBootstrapFanout();
|
||||
|
||||
expect(maxInflight).toBeLessThanOrEqual(4);
|
||||
expect(maxInflight).toBeGreaterThan(0);
|
||||
});
|
||||
|
||||
it('failures do not abort the fanout', async () => {
|
||||
const ids: number[] = [];
|
||||
for (let i = 0; i < 3; i++) {
|
||||
const id = seedProxyNode(true);
|
||||
insertMeshStack(id, `stack-fail-${i}-${uniqueSuffix()}`);
|
||||
ids.push(id);
|
||||
}
|
||||
const failingId = ids[1];
|
||||
|
||||
const calls: number[] = [];
|
||||
vi.spyOn(MeshProxyTunnelDialer.getInstance(), 'ensureBridge')
|
||||
.mockImplementation(async (id: number) => {
|
||||
calls.push(id);
|
||||
if (id === failingId) throw new Error('boom');
|
||||
return null;
|
||||
});
|
||||
|
||||
await (MeshService.getInstance() as unknown as {
|
||||
proactiveBootstrapFanout: () => Promise<void>;
|
||||
}).proactiveBootstrapFanout();
|
||||
|
||||
expect(calls.slice().sort((a, b) => a - b)).toEqual(ids.slice().sort((a, b) => a - b));
|
||||
});
|
||||
});
|
||||
@@ -1,4 +1,4 @@
|
||||
import { afterAll, beforeAll, beforeEach, describe, expect, it, vi } from 'vitest';
|
||||
import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from 'vitest';
|
||||
import { EventEmitter } from 'events';
|
||||
import fsSync from 'fs';
|
||||
import path from 'path';
|
||||
@@ -849,6 +849,14 @@ describe('MeshService.openCrossNode (BUG-4)', () => {
|
||||
};
|
||||
}
|
||||
|
||||
// Install a stub reverseDialer so openCrossNode skips the peer-side
|
||||
// bootstrap path (PeerToCentralMeshSessionDialer.ensureSession). The
|
||||
// dispatch behavior under test relies on dialMeshTcpStream being called
|
||||
// directly; the bootstrap kick would short-circuit before that mock fires.
|
||||
const stubDialer = { openMeshTcpStream: vi.fn() };
|
||||
beforeEach(() => { MeshService.getInstance().setReverseDialer(stubDialer); });
|
||||
afterEach(() => { MeshService.getInstance().setReverseDialer(null); });
|
||||
|
||||
it('emits route.dispatch immediately on cross-node entry', async () => {
|
||||
const svc = MeshService.getInstance();
|
||||
const target: MeshTarget = {
|
||||
|
||||
@@ -0,0 +1,73 @@
|
||||
import { describe, it, expect, beforeAll, afterAll, beforeEach, afterEach, vi } from 'vitest';
|
||||
import { setupTestDb, cleanupTestDb } from './helpers/setupTestDb';
|
||||
import { MeshService } from '../services/MeshService';
|
||||
import { DatabaseService } from '../services/DatabaseService';
|
||||
|
||||
function seedMeshedProxyNode(): void {
|
||||
const db = DatabaseService.getInstance();
|
||||
const id = db.addNode({
|
||||
name: `p-${Date.now()}-${Math.random()}`,
|
||||
type: 'remote',
|
||||
mode: 'proxy',
|
||||
api_url: 'https://p.example.com',
|
||||
api_token: 't',
|
||||
compose_dir: '/tmp',
|
||||
is_default: false,
|
||||
});
|
||||
db.setNodeMeshEnabled(id, true);
|
||||
}
|
||||
|
||||
let tmpDir: string;
|
||||
beforeAll(async () => { tmpDir = await setupTestDb(); });
|
||||
afterAll(() => cleanupTestDb(tmpDir));
|
||||
|
||||
describe('SENCHO_PRIMARY_URL preflight warning', () => {
|
||||
let originalPrimaryUrl: string | undefined;
|
||||
|
||||
beforeEach(() => {
|
||||
originalPrimaryUrl = process.env.SENCHO_PRIMARY_URL;
|
||||
// The DB singleton persists across tests in the same process; wipe any
|
||||
// non-default nodes seeded by a prior test so each case starts clean.
|
||||
// setupTestDb itself runs once per file (beforeAll above) to avoid
|
||||
// invalidating the DatabaseService singleton's open connection.
|
||||
DatabaseService.getInstance().getDb().prepare('DELETE FROM nodes WHERE is_default = 0').run();
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
if (originalPrimaryUrl === undefined) {
|
||||
delete process.env.SENCHO_PRIMARY_URL;
|
||||
} else {
|
||||
process.env.SENCHO_PRIMARY_URL = originalPrimaryUrl;
|
||||
}
|
||||
});
|
||||
|
||||
it('warns when SENCHO_PRIMARY_URL is unset and a mesh-enabled proxy node exists', () => {
|
||||
delete process.env.SENCHO_PRIMARY_URL;
|
||||
seedMeshedProxyNode();
|
||||
const warn = vi.spyOn(console, 'warn').mockImplementation(() => {});
|
||||
const svc = MeshService.getInstance() as unknown as { maybeWarnUnsetPrimaryUrl: () => void };
|
||||
svc.maybeWarnUnsetPrimaryUrl();
|
||||
expect(warn.mock.calls.some(c => /SENCHO_PRIMARY_URL is unset/.test(String(c[0])))).toBe(true);
|
||||
warn.mockRestore();
|
||||
});
|
||||
|
||||
it('does not warn when SENCHO_PRIMARY_URL is set', () => {
|
||||
process.env.SENCHO_PRIMARY_URL = 'https://central.example.com';
|
||||
seedMeshedProxyNode();
|
||||
const warn = vi.spyOn(console, 'warn').mockImplementation(() => {});
|
||||
const svc = MeshService.getInstance() as unknown as { maybeWarnUnsetPrimaryUrl: () => void };
|
||||
svc.maybeWarnUnsetPrimaryUrl();
|
||||
expect(warn.mock.calls.some(c => /SENCHO_PRIMARY_URL is unset/.test(String(c[0])))).toBe(false);
|
||||
warn.mockRestore();
|
||||
});
|
||||
|
||||
it('does not warn when SENCHO_PRIMARY_URL is unset but no mesh-enabled proxy node exists', () => {
|
||||
delete process.env.SENCHO_PRIMARY_URL;
|
||||
// No seeding: zero mesh-enabled proxy nodes (only the default local node).
|
||||
const warn = vi.spyOn(console, 'warn').mockImplementation(() => {});
|
||||
const svc = MeshService.getInstance() as unknown as { maybeWarnUnsetPrimaryUrl: () => void };
|
||||
svc.maybeWarnUnsetPrimaryUrl();
|
||||
expect(warn.mock.calls.some(c => /SENCHO_PRIMARY_URL is unset/.test(String(c[0])))).toBe(false);
|
||||
warn.mockRestore();
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,177 @@
|
||||
/**
|
||||
* Triggers 1 + 2: proactive mesh bridge bootstrap on mesh-enable and on
|
||||
* api_token rotation.
|
||||
*
|
||||
* Trigger 1 (mesh-enable): after `setNodeMeshEnabled(true)`, the service
|
||||
* fire-and-forgets `MeshProxyTunnelDialer.ensureBridge(nodeId)` for any
|
||||
* remote/proxy peer. The capability-gated handshake then ships
|
||||
* `mesh_handshake` material to peers that just came online or just got
|
||||
* upgraded to a build that advertises `mesh_proxy_callback_bootstrap`.
|
||||
*
|
||||
* Trigger 2 (api_token rotation): when the nodes router persists a new
|
||||
* `api_token` for a mesh-enabled proxy peer, it closes the existing bridge
|
||||
* with reason 'peer token rotated' and re-dials. The next ensureBridge mints
|
||||
* a JWT whose fingerprint matches the freshly stored token; without this
|
||||
* trigger the remote would reject the upgrade with token_fingerprint_mismatch
|
||||
* until the next idle re-dial.
|
||||
*/
|
||||
import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from 'vitest';
|
||||
import request from 'supertest';
|
||||
import jwt from 'jsonwebtoken';
|
||||
import { setupTestDb, cleanupTestDb, TEST_USERNAME, TEST_JWT_SECRET } from './helpers/setupTestDb';
|
||||
import { MeshService } from '../services/MeshService';
|
||||
import { MeshProxyTunnelDialer } from '../services/MeshProxyTunnelDialer';
|
||||
import { DatabaseService } from '../services/DatabaseService';
|
||||
|
||||
let tmpDir: string;
|
||||
let app: import('express').Express;
|
||||
let authHeader: string;
|
||||
|
||||
beforeAll(async () => {
|
||||
tmpDir = await setupTestDb();
|
||||
({ app } = await import('../index'));
|
||||
const token = jwt.sign({ username: TEST_USERNAME }, TEST_JWT_SECRET, { expiresIn: '1m' });
|
||||
authHeader = `Bearer ${token}`;
|
||||
});
|
||||
|
||||
afterAll(() => {
|
||||
cleanupTestDb(tmpDir);
|
||||
});
|
||||
|
||||
function uniqueSuffix(): string {
|
||||
return `${Date.now()}-${Math.random().toString(36).slice(2, 10)}`;
|
||||
}
|
||||
|
||||
function seedProxyNode(meshEnabled: boolean): number {
|
||||
const db = DatabaseService.getInstance();
|
||||
const id = db.addNode({
|
||||
name: `peer-rotate-${uniqueSuffix()}`,
|
||||
type: 'remote',
|
||||
mode: 'proxy',
|
||||
api_url: `https://peer-${uniqueSuffix()}.example.com`,
|
||||
api_token: `tok-${uniqueSuffix()}`,
|
||||
compose_dir: '/tmp',
|
||||
is_default: false,
|
||||
});
|
||||
if (meshEnabled) {
|
||||
db.setNodeMeshEnabled(id, true);
|
||||
}
|
||||
return id;
|
||||
}
|
||||
|
||||
function clearTestNodes(): void {
|
||||
const db = DatabaseService.getInstance().getDb();
|
||||
db.prepare('DELETE FROM mesh_stacks').run();
|
||||
db.prepare('DELETE FROM nodes WHERE is_default = 0').run();
|
||||
}
|
||||
|
||||
describe('Trigger 1: enableForNode triggers proactive bridge bootstrap', () => {
|
||||
beforeEach(() => {
|
||||
clearTestNodes();
|
||||
MeshProxyTunnelDialer.resetForTest();
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
vi.restoreAllMocks();
|
||||
});
|
||||
|
||||
it('calls ensureBridge for a remote proxy-mode node', async () => {
|
||||
const ensureSpy = vi.spyOn(MeshProxyTunnelDialer.getInstance(), 'ensureBridge')
|
||||
.mockResolvedValue(null);
|
||||
const nodeId = seedProxyNode(false);
|
||||
|
||||
await MeshService.getInstance().enableForNode(nodeId);
|
||||
|
||||
// Trigger is fire-and-forget; wait a microtask for the void promise.
|
||||
await new Promise((r) => setImmediate(r));
|
||||
expect(ensureSpy).toHaveBeenCalledWith(nodeId);
|
||||
});
|
||||
|
||||
it('skips local nodes', async () => {
|
||||
const ensureSpy = vi.spyOn(MeshProxyTunnelDialer.getInstance(), 'ensureBridge')
|
||||
.mockResolvedValue(null);
|
||||
const localId = DatabaseService.getInstance().getNodes()[0].id;
|
||||
|
||||
await MeshService.getInstance().enableForNode(localId);
|
||||
|
||||
await new Promise((r) => setImmediate(r));
|
||||
expect(ensureSpy).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('swallows ensureBridge rejections without throwing', async () => {
|
||||
vi.spyOn(MeshProxyTunnelDialer.getInstance(), 'ensureBridge')
|
||||
.mockRejectedValue(new Error('peer unreachable'));
|
||||
const nodeId = seedProxyNode(false);
|
||||
|
||||
await expect(MeshService.getInstance().enableForNode(nodeId)).resolves.toBeUndefined();
|
||||
await new Promise((r) => setImmediate(r));
|
||||
});
|
||||
});
|
||||
|
||||
describe('Trigger 2: api_token rotation forces re-bootstrap', () => {
|
||||
beforeEach(() => {
|
||||
clearTestNodes();
|
||||
MeshProxyTunnelDialer.resetForTest();
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
vi.restoreAllMocks();
|
||||
});
|
||||
|
||||
it('calls closeBridge then ensureBridge when rotation handler runs', async () => {
|
||||
const closeSpy = vi.spyOn(MeshProxyTunnelDialer.getInstance(), 'closeBridge');
|
||||
const ensureSpy = vi.spyOn(MeshProxyTunnelDialer.getInstance(), 'ensureBridge')
|
||||
.mockResolvedValue(null);
|
||||
const nodeId = seedProxyNode(true);
|
||||
|
||||
const res = await request(app)
|
||||
.put(`/api/nodes/${nodeId}`)
|
||||
.set('Authorization', authHeader)
|
||||
.send({ api_token: 'rotated-token-abc' });
|
||||
|
||||
expect(res.status).toBe(200);
|
||||
await new Promise((r) => setImmediate(r));
|
||||
|
||||
expect(closeSpy).toHaveBeenCalledWith(nodeId, 'peer token rotated');
|
||||
expect(ensureSpy).toHaveBeenCalledWith(nodeId);
|
||||
const closeOrder = closeSpy.mock.invocationCallOrder[0];
|
||||
const ensureOrder = ensureSpy.mock.invocationCallOrder[0];
|
||||
expect(closeOrder).toBeLessThan(ensureOrder);
|
||||
});
|
||||
|
||||
it('does not fire when api_token is not in the update payload', async () => {
|
||||
const closeSpy = vi.spyOn(MeshProxyTunnelDialer.getInstance(), 'closeBridge');
|
||||
const ensureSpy = vi.spyOn(MeshProxyTunnelDialer.getInstance(), 'ensureBridge')
|
||||
.mockResolvedValue(null);
|
||||
const nodeId = seedProxyNode(true);
|
||||
|
||||
const res = await request(app)
|
||||
.put(`/api/nodes/${nodeId}`)
|
||||
.set('Authorization', authHeader)
|
||||
.send({ name: `renamed-${uniqueSuffix()}` });
|
||||
|
||||
expect(res.status).toBe(200);
|
||||
await new Promise((r) => setImmediate(r));
|
||||
|
||||
expect(closeSpy).not.toHaveBeenCalled();
|
||||
expect(ensureSpy).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('does not fire when mesh is disabled on the node', async () => {
|
||||
const closeSpy = vi.spyOn(MeshProxyTunnelDialer.getInstance(), 'closeBridge');
|
||||
const ensureSpy = vi.spyOn(MeshProxyTunnelDialer.getInstance(), 'ensureBridge')
|
||||
.mockResolvedValue(null);
|
||||
const nodeId = seedProxyNode(false);
|
||||
|
||||
const res = await request(app)
|
||||
.put(`/api/nodes/${nodeId}`)
|
||||
.set('Authorization', authHeader)
|
||||
.send({ api_token: 'rotated-token-xyz' });
|
||||
|
||||
expect(res.status).toBe(200);
|
||||
await new Promise((r) => setImmediate(r));
|
||||
|
||||
expect(closeSpy).not.toHaveBeenCalled();
|
||||
expect(ensureSpy).not.toHaveBeenCalled();
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,112 @@
|
||||
/**
|
||||
* Version skew invariant: central must NOT send a `mesh_handshake` to a peer
|
||||
* that lacks the `mesh_proxy_callback_bootstrap` capability.
|
||||
*
|
||||
* The capability gate keeps central rolling forward without breaking older
|
||||
* peers: a peer that does not understand the bootstrap frame would otherwise
|
||||
* see a stray JSON text frame before the TCP frames it expects. The dialer's
|
||||
* `maybeSendBootstrap` must read the capability list from the remote's meta
|
||||
* endpoint and silently skip the send when the bit is absent.
|
||||
*
|
||||
* This is complementary to `mesh-proxy-tunnel-dialer-handshake.test.ts`,
|
||||
* which exercises the same code path with a finer-grained set of inputs;
|
||||
* this file locks in the same invariant as an integration so a future
|
||||
* refactor that moves the capability check to a different layer does not
|
||||
* silently regress.
|
||||
*/
|
||||
import { describe, it, expect, beforeAll, afterAll, beforeEach, afterEach, vi } from 'vitest';
|
||||
import { setupTestDb, cleanupTestDb } from './helpers/setupTestDb';
|
||||
import { NodeRegistry } from '../services/NodeRegistry';
|
||||
import { MeshProxyTunnelDialer } from '../services/MeshProxyTunnelDialer';
|
||||
import { OFFLINE_META } from '../services/CapabilityRegistry';
|
||||
import { MeshCentralRegistry } from '../services/MeshCentralRegistry';
|
||||
import { DatabaseService } from '../services/DatabaseService';
|
||||
|
||||
let tmpDir: string;
|
||||
beforeAll(async () => { tmpDir = await setupTestDb(); });
|
||||
afterAll(() => cleanupTestDb(tmpDir));
|
||||
|
||||
describe('Version skew: peer without mesh_proxy_callback_bootstrap', () => {
|
||||
beforeEach(() => {
|
||||
// Per-test reset of dialer + registry singletons + env. The DB lives
|
||||
// once per file (beforeAll above); resetting setupTestDb per test
|
||||
// would invalidate the DatabaseService singleton on Linux.
|
||||
MeshProxyTunnelDialer.resetForTest();
|
||||
MeshCentralRegistry.resetForTest();
|
||||
process.env.SENCHO_PRIMARY_URL = 'https://central.example.com';
|
||||
DatabaseService.getInstance().setSystemState('instance_id', 'test-central-instance');
|
||||
});
|
||||
afterEach(() => {
|
||||
delete process.env.SENCHO_PRIMARY_URL;
|
||||
vi.restoreAllMocks();
|
||||
});
|
||||
|
||||
type DialerPrivate = {
|
||||
maybeSendBootstrap: (nodeId: number, ws: unknown) => Promise<void>;
|
||||
};
|
||||
|
||||
it('central does not send mesh_handshake when capability is absent', async () => {
|
||||
vi.spyOn(NodeRegistry.getInstance(), 'fetchMetaForNode').mockResolvedValue({
|
||||
...OFFLINE_META,
|
||||
version: '0.78.0',
|
||||
capabilities: ['stacks'],
|
||||
online: true,
|
||||
startedAt: Date.now(),
|
||||
});
|
||||
|
||||
// Seed a proxy node so maybeSendBootstrap has something to look up.
|
||||
const db = DatabaseService.getInstance();
|
||||
const id = db.addNode({
|
||||
name: `n-${Date.now()}-${Math.random().toString(36).slice(2, 8)}`,
|
||||
type: 'remote',
|
||||
mode: 'proxy',
|
||||
api_url: 'https://n',
|
||||
api_token: 't',
|
||||
compose_dir: '/tmp',
|
||||
is_default: false,
|
||||
});
|
||||
db.setNodeMeshEnabled(id, true);
|
||||
|
||||
const sentFrames: unknown[] = [];
|
||||
const fakeWs = { send: vi.fn((data: string) => sentFrames.push(JSON.parse(data))) };
|
||||
|
||||
await (MeshProxyTunnelDialer.getInstance() as unknown as DialerPrivate)
|
||||
.maybeSendBootstrap(id, fakeWs);
|
||||
|
||||
expect(fakeWs.send).not.toHaveBeenCalled();
|
||||
expect(sentFrames).toHaveLength(0);
|
||||
// No persistence as a side effect either.
|
||||
expect(MeshCentralRegistry.getInstance().getActive()).toBeNull();
|
||||
});
|
||||
|
||||
it('central does send mesh_handshake when the capability flips to advertised', async () => {
|
||||
// Sanity: the same code path with the capability present must take
|
||||
// the affirmative branch, so we know the skip above is gated on the
|
||||
// capability and not on some unrelated precondition.
|
||||
vi.spyOn(NodeRegistry.getInstance(), 'fetchMetaForNode').mockResolvedValue({
|
||||
...OFFLINE_META,
|
||||
version: '0.79.0',
|
||||
capabilities: ['stacks', 'mesh_proxy_callback_bootstrap'],
|
||||
online: true,
|
||||
startedAt: Date.now(),
|
||||
});
|
||||
const db = DatabaseService.getInstance();
|
||||
const id = db.addNode({
|
||||
name: `n-${Date.now()}-${Math.random().toString(36).slice(2, 8)}`,
|
||||
type: 'remote',
|
||||
mode: 'proxy',
|
||||
api_url: 'https://n',
|
||||
api_token: 't',
|
||||
compose_dir: '/tmp',
|
||||
is_default: false,
|
||||
});
|
||||
db.setNodeMeshEnabled(id, true);
|
||||
|
||||
const sentFrames: unknown[] = [];
|
||||
const fakeWs = { send: vi.fn((data: string) => sentFrames.push(JSON.parse(data))) };
|
||||
await (MeshProxyTunnelDialer.getInstance() as unknown as DialerPrivate)
|
||||
.maybeSendBootstrap(id, fakeWs);
|
||||
expect(fakeWs.send).toHaveBeenCalledOnce();
|
||||
expect(sentFrames[0]).toMatchObject({ t: 'mesh_handshake', v: 1 });
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,250 @@
|
||||
/**
|
||||
* `PeerToCentralMeshSessionDialer`: peer-side counterpart to the central
|
||||
* ingress at `/api/mesh/proxy-tunnel-from-peer` (Task 9). Reads cached
|
||||
* central material from `MeshCentralRegistry`, opens a WebSocket to
|
||||
* central carrying the bootstrapped JWT in the Authorization header, and
|
||||
* branches on the upgrade outcome:
|
||||
*
|
||||
* - successful open: marks the cache row "used", arms the bridge.
|
||||
* - 401 with a terminal reason code (stale, signature_invalid, ...):
|
||||
* clears the cache so we stop dialing on a bad credential.
|
||||
* - 401 with a transient reason (clock_skew, mode_mismatch, ...):
|
||||
* keeps the cache, marks rejected so the operator can see why.
|
||||
* - 404 endpoint_not_found: keeps the cache and arms a longer backoff
|
||||
* so we do not hammer an older central that has not yet shipped the
|
||||
* peer ingress endpoint.
|
||||
*
|
||||
* The dial logic is also rate-limited per dialer process (5 attempts /
|
||||
* 60s window) so a misbehaving central or a misconfigured peer cannot
|
||||
* flood either side with handshake traffic.
|
||||
*/
|
||||
import http from 'http';
|
||||
import type { AddressInfo } from 'net';
|
||||
import { WebSocketServer } from 'ws';
|
||||
import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from 'vitest';
|
||||
import { setupTestDb, cleanupTestDb } from './helpers/setupTestDb';
|
||||
|
||||
let tmpDir: string;
|
||||
let PeerToCentralMeshSessionDialer: typeof import('../services/PeerToCentralMeshSessionDialer').PeerToCentralMeshSessionDialer;
|
||||
let MeshCentralRegistry: typeof import('../services/MeshCentralRegistry').MeshCentralRegistry;
|
||||
let DatabaseService: typeof import('../services/DatabaseService').DatabaseService;
|
||||
let PilotTunnelBridge: typeof import('../services/PilotTunnelBridge').PilotTunnelBridge;
|
||||
|
||||
interface RejectingServer {
|
||||
server: http.Server;
|
||||
url: string;
|
||||
lastAuthHeader: string | null;
|
||||
lastPath: string | null;
|
||||
close: () => Promise<void>;
|
||||
}
|
||||
|
||||
/**
|
||||
* Spins up a minimal HTTP server that intercepts WebSocket upgrades and
|
||||
* responds with a fixed status + JSON body, mimicking how the central
|
||||
* ingress rejects bad credentials. Records the auth header and path so
|
||||
* tests can assert the dialer is sending the right material.
|
||||
*/
|
||||
async function startRejectingServer(status: number, reason: string): Promise<RejectingServer> {
|
||||
const handle: RejectingServer = {
|
||||
server: http.createServer(),
|
||||
url: '',
|
||||
lastAuthHeader: null,
|
||||
lastPath: null,
|
||||
close: () => new Promise<void>((resolve) => handle.server.close(() => resolve())),
|
||||
};
|
||||
handle.server.on('upgrade', (req, socket) => {
|
||||
handle.lastAuthHeader = (req.headers['authorization'] as string | undefined) ?? null;
|
||||
handle.lastPath = req.url ?? null;
|
||||
const body = JSON.stringify({ reason });
|
||||
const statusText = status === 404 ? 'Not Found' : 'Unauthorized';
|
||||
const head = [
|
||||
`HTTP/1.1 ${status} ${statusText}`,
|
||||
'Content-Type: application/json',
|
||||
`Content-Length: ${Buffer.byteLength(body)}`,
|
||||
'Connection: close',
|
||||
'',
|
||||
body,
|
||||
].join('\r\n');
|
||||
try { socket.write(head); } catch { /* ignore */ }
|
||||
try { socket.destroy(); } catch { /* ignore */ }
|
||||
});
|
||||
await new Promise<void>((resolve) => handle.server.listen(0, '127.0.0.1', () => resolve()));
|
||||
const port = (handle.server.address() as AddressInfo).port;
|
||||
handle.url = `http://127.0.0.1:${port}`;
|
||||
return handle;
|
||||
}
|
||||
|
||||
beforeAll(async () => {
|
||||
tmpDir = await setupTestDb();
|
||||
({ PeerToCentralMeshSessionDialer } = await import('../services/PeerToCentralMeshSessionDialer'));
|
||||
({ MeshCentralRegistry } = await import('../services/MeshCentralRegistry'));
|
||||
({ DatabaseService } = await import('../services/DatabaseService'));
|
||||
({ PilotTunnelBridge } = await import('../services/PilotTunnelBridge'));
|
||||
});
|
||||
|
||||
afterAll(() => {
|
||||
cleanupTestDb(tmpDir);
|
||||
});
|
||||
|
||||
beforeEach(() => {
|
||||
MeshCentralRegistry.resetForTest();
|
||||
PeerToCentralMeshSessionDialer.resetForTest();
|
||||
DatabaseService.getInstance().getDb().prepare('DELETE FROM mesh_centrals').run();
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
PeerToCentralMeshSessionDialer.resetForTest();
|
||||
MeshCentralRegistry.resetForTest();
|
||||
});
|
||||
|
||||
describe('PeerToCentralMeshSessionDialer', () => {
|
||||
it('returns null when no central material is cached', async () => {
|
||||
const result = await PeerToCentralMeshSessionDialer.getInstance().ensureSession();
|
||||
expect(result).toBeNull();
|
||||
});
|
||||
|
||||
it('dials using the cached URL + JWT bearer (asserted by server-side recording)', async () => {
|
||||
const srv = await startRejectingServer(401, 'clock_skew');
|
||||
try {
|
||||
MeshCentralRegistry.getInstance().upsert({
|
||||
centralInstanceId: 'inst-a',
|
||||
centralApiUrl: srv.url,
|
||||
callbackJwt: 'fake.jwt.token',
|
||||
jwtIssuedAt: 1,
|
||||
jwtExpiresAt: Math.floor(Date.now() / 1000) + 3600,
|
||||
});
|
||||
const result = await PeerToCentralMeshSessionDialer.getInstance().ensureSession();
|
||||
expect(result).toBeNull();
|
||||
expect(srv.lastPath).toBe('/api/mesh/proxy-tunnel-from-peer');
|
||||
expect(srv.lastAuthHeader).toBe('Bearer fake.jwt.token');
|
||||
} finally {
|
||||
await srv.close();
|
||||
}
|
||||
});
|
||||
|
||||
it('clears cache on 401 reason=stale', async () => {
|
||||
const srv = await startRejectingServer(401, 'stale');
|
||||
try {
|
||||
MeshCentralRegistry.getInstance().upsert({
|
||||
centralInstanceId: 'inst-b',
|
||||
centralApiUrl: srv.url,
|
||||
callbackJwt: 'jwt',
|
||||
jwtIssuedAt: 1,
|
||||
jwtExpiresAt: Math.floor(Date.now() / 1000) + 3600,
|
||||
});
|
||||
await PeerToCentralMeshSessionDialer.getInstance().ensureSession();
|
||||
expect(MeshCentralRegistry.getInstance().getActive()).toBeNull();
|
||||
} finally {
|
||||
await srv.close();
|
||||
}
|
||||
});
|
||||
|
||||
it('keeps cache on 401 reason=clock_skew but records the rejection', async () => {
|
||||
const srv = await startRejectingServer(401, 'clock_skew');
|
||||
try {
|
||||
MeshCentralRegistry.getInstance().upsert({
|
||||
centralInstanceId: 'inst-c',
|
||||
centralApiUrl: srv.url,
|
||||
callbackJwt: 'jwt',
|
||||
jwtIssuedAt: 1,
|
||||
jwtExpiresAt: Math.floor(Date.now() / 1000) + 3600,
|
||||
});
|
||||
await PeerToCentralMeshSessionDialer.getInstance().ensureSession();
|
||||
const row = MeshCentralRegistry.getInstance().getActive();
|
||||
expect(row).not.toBeNull();
|
||||
expect(row?.lastRejectReason).toBe('clock_skew');
|
||||
} finally {
|
||||
await srv.close();
|
||||
}
|
||||
});
|
||||
|
||||
it('keeps cache on HTTP 404 endpoint_not_found and rate-limits future dials', async () => {
|
||||
const srv = await startRejectingServer(404, 'whatever');
|
||||
try {
|
||||
MeshCentralRegistry.getInstance().upsert({
|
||||
centralInstanceId: 'inst-d',
|
||||
centralApiUrl: srv.url,
|
||||
callbackJwt: 'jwt',
|
||||
jwtIssuedAt: 1,
|
||||
jwtExpiresAt: Math.floor(Date.now() / 1000) + 3600,
|
||||
});
|
||||
const dialer = PeerToCentralMeshSessionDialer.getInstance();
|
||||
await dialer.ensureSession();
|
||||
// Cache must survive a 404 (older central, transient infra).
|
||||
expect(MeshCentralRegistry.getInstance().getActive()).not.toBeNull();
|
||||
// The second call must not even hit the server: the endpoint
|
||||
// backoff blocks new dials for the configured window.
|
||||
srv.lastAuthHeader = null;
|
||||
const second = await dialer.ensureSession();
|
||||
expect(second).toBeNull();
|
||||
expect(srv.lastAuthHeader).toBeNull();
|
||||
} finally {
|
||||
await srv.close();
|
||||
}
|
||||
});
|
||||
|
||||
it('rate-limits dials after the configured attempt cap', async () => {
|
||||
const srv = await startRejectingServer(401, 'clock_skew');
|
||||
try {
|
||||
MeshCentralRegistry.getInstance().upsert({
|
||||
centralInstanceId: 'inst-e',
|
||||
centralApiUrl: srv.url,
|
||||
callbackJwt: 'jwt',
|
||||
jwtIssuedAt: 1,
|
||||
jwtExpiresAt: Math.floor(Date.now() / 1000) + 3600,
|
||||
});
|
||||
const dialer = PeerToCentralMeshSessionDialer.getInstance();
|
||||
// 5 dials should land; the 6th must be suppressed by the limiter.
|
||||
for (let i = 0; i < 5; i++) {
|
||||
await dialer.ensureSession();
|
||||
}
|
||||
srv.lastAuthHeader = null;
|
||||
const sixth = await dialer.ensureSession();
|
||||
expect(sixth).toBeNull();
|
||||
expect(srv.lastAuthHeader).toBeNull();
|
||||
} finally {
|
||||
await srv.close();
|
||||
}
|
||||
});
|
||||
|
||||
it('marks the row used on successful WS open', async () => {
|
||||
const wss = new WebSocketServer({ noServer: true });
|
||||
const srv = http.createServer();
|
||||
srv.on('upgrade', (req, socket, head) => {
|
||||
if (req.url?.startsWith('/api/mesh/proxy-tunnel-from-peer')) {
|
||||
wss.handleUpgrade(req, socket, head, () => { /* accept */ });
|
||||
} else {
|
||||
socket.destroy();
|
||||
}
|
||||
});
|
||||
await new Promise<void>((resolve) => srv.listen(0, '127.0.0.1', () => resolve()));
|
||||
const port = (srv.address() as AddressInfo).port;
|
||||
const url = `http://127.0.0.1:${port}`;
|
||||
let bridge: InstanceType<typeof PilotTunnelBridge> | null = null;
|
||||
try {
|
||||
MeshCentralRegistry.getInstance().upsert({
|
||||
centralInstanceId: 'inst-success',
|
||||
centralApiUrl: url,
|
||||
callbackJwt: 'fake.jwt',
|
||||
jwtIssuedAt: 1,
|
||||
jwtExpiresAt: 9999999999,
|
||||
});
|
||||
bridge = await PeerToCentralMeshSessionDialer.getInstance().ensureSession();
|
||||
expect(bridge).toBeInstanceOf(PilotTunnelBridge);
|
||||
await vi.waitFor(() => {
|
||||
expect(MeshCentralRegistry.getInstance().getActive()?.lastUsedAt ?? 0).toBeGreaterThan(0);
|
||||
});
|
||||
expect(PeerToCentralMeshSessionDialer.getInstance().hasSession()).toBe(true);
|
||||
} finally {
|
||||
try { bridge?.close(1000, 'test done'); } catch { /* ignore */ }
|
||||
await new Promise<void>((resolve) => wss.close(() => resolve()));
|
||||
await new Promise<void>((resolve) => srv.close(() => resolve()));
|
||||
}
|
||||
});
|
||||
|
||||
it('singleton returns the same instance', () => {
|
||||
const a = PeerToCentralMeshSessionDialer.getInstance();
|
||||
const b = PeerToCentralMeshSessionDialer.getInstance();
|
||||
expect(a).toBe(b);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,74 @@
|
||||
/**
|
||||
* Pilot-mode regression invariant: a pilot-agent tunnel always wins over a
|
||||
* peer-initiated proxy-mode bridge for the same nodeId.
|
||||
*
|
||||
* The two registration paths share the `PilotTunnelManager.bridges` map.
|
||||
* Without the `bridgeKinds` index, a peer-initiated dial-back could quietly
|
||||
* replace a live pilot tunnel and break the agent's reverse-stream relay.
|
||||
* `replaceOrRegisterProxyBridge` is the single point that has to refuse the
|
||||
* replacement; this test locks that contract in.
|
||||
*
|
||||
* Mirrors the `injectBridgeForTest` pattern used elsewhere in the suite so
|
||||
* we exercise the manager invariant without owning a real WebSocket.
|
||||
*/
|
||||
import { describe, it, expect, beforeEach, vi } from 'vitest';
|
||||
import { EventEmitter } from 'events';
|
||||
import { PilotTunnelManager } from '../services/PilotTunnelManager';
|
||||
import type { PilotTunnelBridge } from '../services/PilotTunnelBridge';
|
||||
|
||||
function makeFakeBridge(): EventEmitter & {
|
||||
close: ReturnType<typeof vi.fn>;
|
||||
getActiveStreamCount: () => number;
|
||||
} {
|
||||
const ee = new EventEmitter() as EventEmitter & {
|
||||
close: ReturnType<typeof vi.fn>;
|
||||
getActiveStreamCount: () => number;
|
||||
};
|
||||
ee.close = vi.fn();
|
||||
ee.getActiveStreamCount = () => 0;
|
||||
return ee;
|
||||
}
|
||||
|
||||
describe('Pilot-mode regression: pilot tunnel wins over peer-initiated proxy bridge', () => {
|
||||
beforeEach(() => {
|
||||
PilotTunnelManager.resetForTest();
|
||||
});
|
||||
|
||||
it('refuses replaceOrRegisterProxyBridge when a pilot tunnel exists for the same nodeId', () => {
|
||||
const mgr = PilotTunnelManager.getInstance();
|
||||
const pilot = makeFakeBridge();
|
||||
const proxy = makeFakeBridge();
|
||||
mgr.injectBridgeForTest(7, pilot as unknown as PilotTunnelBridge, 'pilot');
|
||||
|
||||
expect(() => mgr.replaceOrRegisterProxyBridge(7, proxy as unknown as PilotTunnelBridge))
|
||||
.toThrow(/pilot tunnel/);
|
||||
|
||||
// Pilot bridge is still the resident bridge for nodeId 7.
|
||||
expect(mgr.getBridge(7)).toBe(pilot);
|
||||
// The pilot bridge must not be closed by the rejected replacement.
|
||||
expect(pilot.close).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('allows replaceOrRegisterProxyBridge to swap one proxy bridge for another', () => {
|
||||
const mgr = PilotTunnelManager.getInstance();
|
||||
const oldProxy = makeFakeBridge();
|
||||
const newProxy = makeFakeBridge();
|
||||
mgr.injectBridgeForTest(9, oldProxy as unknown as PilotTunnelBridge, 'proxy');
|
||||
|
||||
mgr.replaceOrRegisterProxyBridge(9, newProxy as unknown as PilotTunnelBridge);
|
||||
|
||||
// The new dial is the source of truth; old proxy is closed.
|
||||
expect(oldProxy.close).toHaveBeenCalledOnce();
|
||||
expect(mgr.getBridge(9)).toBe(newProxy);
|
||||
});
|
||||
|
||||
it('replaceOrRegisterProxyBridge on an empty slot just registers (no refuse, no close)', () => {
|
||||
const mgr = PilotTunnelManager.getInstance();
|
||||
const proxy = makeFakeBridge();
|
||||
|
||||
mgr.replaceOrRegisterProxyBridge(11, proxy as unknown as PilotTunnelBridge);
|
||||
|
||||
expect(mgr.getBridge(11)).toBe(proxy);
|
||||
expect(proxy.close).not.toHaveBeenCalled();
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,290 @@
|
||||
/**
|
||||
* R1-B: PilotTunnelBridge.acceptReverseLocal emits MeshService activity events
|
||||
* for the reverse-direction dispatch (peer-to-central). Reuses the existing
|
||||
* `route.resolve.ok` / `route.resolve.fail` event types and discriminates by
|
||||
* `details.direction = 'reverse'` so the Routing tab can surface peer-side
|
||||
* mesh failures alongside central-side dispatch events.
|
||||
*
|
||||
* Covers three outcomes plus one negative assertion:
|
||||
* 1. resolveContainerIp returns null -> route.resolve.fail / container_not_found
|
||||
* 2. socket emits 'error' pre-connect -> route.resolve.fail / connect_error
|
||||
* 3. socket emits 'connect' -> route.resolve.ok
|
||||
* 4. post-connect close/error does NOT emit a second route.resolve.fail
|
||||
*/
|
||||
import { describe, it, expect, beforeAll, afterAll, beforeEach, vi } from 'vitest';
|
||||
import net from 'net';
|
||||
import { EventEmitter } from 'events';
|
||||
import { WebSocket } from 'ws';
|
||||
import { setupTestDb, cleanupTestDb } from './helpers/setupTestDb';
|
||||
import { AGENT_REVERSE_ID_BASE, encodeJsonFrame, decodeJsonFrame } from '../pilot/protocol';
|
||||
import type { MeshActivityEvent } from '../services/MeshService';
|
||||
|
||||
type LogActivityArgs = [Omit<MeshActivityEvent, 'ts'>];
|
||||
|
||||
let tmpDir: string;
|
||||
let PilotTunnelBridge: typeof import('../services/PilotTunnelBridge').PilotTunnelBridge;
|
||||
let MeshService: typeof import('../services/MeshService').MeshService;
|
||||
|
||||
beforeAll(async () => {
|
||||
tmpDir = await setupTestDb();
|
||||
({ PilotTunnelBridge } = await import('../services/PilotTunnelBridge'));
|
||||
({ MeshService } = await import('../services/MeshService'));
|
||||
});
|
||||
|
||||
afterAll(() => {
|
||||
vi.restoreAllMocks();
|
||||
cleanupTestDb(tmpDir);
|
||||
});
|
||||
|
||||
function makeMockTunnelWs(): EventEmitter & {
|
||||
sent: unknown[];
|
||||
readyState: number;
|
||||
bufferedAmount: number;
|
||||
send: (data: unknown) => void;
|
||||
ping: () => void;
|
||||
close: () => void;
|
||||
} {
|
||||
const ws = new EventEmitter() as EventEmitter & {
|
||||
sent: unknown[]; readyState: number; bufferedAmount: number;
|
||||
send: (data: unknown) => void; ping: () => void; close: () => void;
|
||||
};
|
||||
ws.sent = [];
|
||||
ws.readyState = WebSocket.OPEN;
|
||||
ws.bufferedAmount = 0;
|
||||
ws.send = (data: unknown) => { ws.sent.push(data); };
|
||||
ws.ping = () => { /* no-op */ };
|
||||
ws.close = () => { ws.readyState = WebSocket.CLOSED; ws.emit('close'); };
|
||||
return ws;
|
||||
}
|
||||
|
||||
function findAck(ws: { sent: unknown[] }, s: number): { ok: boolean; err?: string } | undefined {
|
||||
for (const item of ws.sent) {
|
||||
if (typeof item !== 'string') continue;
|
||||
try {
|
||||
const f = decodeJsonFrame(item);
|
||||
if (f.t === 'tcp_open_ack' && f.s === s) return { ok: f.ok, err: f.err };
|
||||
} catch { /* ignore */ }
|
||||
}
|
||||
return undefined;
|
||||
}
|
||||
|
||||
async function waitFor<T>(check: () => T | undefined): Promise<T> {
|
||||
const deadline = Date.now() + 1500;
|
||||
while (Date.now() < deadline) {
|
||||
const v = check();
|
||||
if (v !== undefined) return v;
|
||||
await new Promise((r) => setTimeout(r, 10));
|
||||
}
|
||||
throw new Error('timeout');
|
||||
}
|
||||
|
||||
describe('R1-B: PilotTunnelBridge.acceptReverseLocal route events', () => {
|
||||
let logSpy: ReturnType<typeof vi.fn<(event: Omit<MeshActivityEvent, 'ts'>) => void>>;
|
||||
|
||||
beforeEach(() => {
|
||||
vi.restoreAllMocks();
|
||||
logSpy = vi.fn<(event: Omit<MeshActivityEvent, 'ts'>) => void>();
|
||||
vi.spyOn(MeshService.getInstance(), 'logActivity').mockImplementation(logSpy);
|
||||
});
|
||||
|
||||
it('emits route.resolve.fail with direction=reverse + reason=container_not_found when IP is unresolved', async () => {
|
||||
const mockWs = makeMockTunnelWs();
|
||||
const peerNodeId = 42;
|
||||
const bridge = new PilotTunnelBridge(peerNodeId, mockWs as unknown as WebSocket);
|
||||
await bridge.start();
|
||||
|
||||
const localNodeId = (await import('../services/NodeRegistry')).NodeRegistry.getInstance().getDefaultNodeId();
|
||||
vi.spyOn(MeshService.getInstance(), 'resolveContainerIp').mockResolvedValue(null);
|
||||
|
||||
const s = AGENT_REVERSE_ID_BASE + 11;
|
||||
mockWs.emit('message', encodeJsonFrame({
|
||||
t: 'tcp_open_reverse', s,
|
||||
targetNodeId: localNodeId, stack: 'missing', service: 'svc', port: 80,
|
||||
}), false);
|
||||
|
||||
const ack = await waitFor(() => findAck(mockWs, s));
|
||||
expect(ack.ok).toBe(false);
|
||||
|
||||
const failCall = logSpy.mock.calls.find(
|
||||
(c: LogActivityArgs) => c[0].type === 'route.resolve.fail',
|
||||
);
|
||||
expect(failCall).toBeDefined();
|
||||
const payload = failCall![0] as {
|
||||
source: string;
|
||||
level: string;
|
||||
type: string;
|
||||
nodeId?: number;
|
||||
details?: Record<string, unknown>;
|
||||
};
|
||||
expect(payload.source).toBe('mesh');
|
||||
expect(payload.level).toBe('error');
|
||||
expect(payload.type).toBe('route.resolve.fail');
|
||||
expect(payload.nodeId).toBe(peerNodeId);
|
||||
expect(payload.details).toMatchObject({
|
||||
direction: 'reverse',
|
||||
reason: 'container_not_found',
|
||||
targetStack: 'missing',
|
||||
targetService: 'svc',
|
||||
targetPort: 80,
|
||||
});
|
||||
|
||||
bridge.close();
|
||||
});
|
||||
|
||||
it('emits route.resolve.fail with direction=reverse + reason=connect_error on socket pre-connect error', async () => {
|
||||
const mockWs = makeMockTunnelWs();
|
||||
const peerNodeId = 43;
|
||||
const bridge = new PilotTunnelBridge(peerNodeId, mockWs as unknown as WebSocket);
|
||||
await bridge.start();
|
||||
|
||||
const localNodeId = (await import('../services/NodeRegistry')).NodeRegistry.getInstance().getDefaultNodeId();
|
||||
// Resolve to localhost on a port nothing is listening on so the dial
|
||||
// produces a synchronous ECONNREFUSED via the OS.
|
||||
vi.spyOn(MeshService.getInstance(), 'resolveContainerIp').mockResolvedValue('127.0.0.1');
|
||||
|
||||
// Pick a port we know is closed by binding and immediately releasing.
|
||||
const probe = net.createServer();
|
||||
await new Promise<void>((resolve) => probe.listen(0, '127.0.0.1', () => resolve()));
|
||||
const addr = probe.address();
|
||||
if (!addr || typeof addr === 'string') throw new Error('no address');
|
||||
const closedPort = addr.port;
|
||||
await new Promise<void>((resolve) => probe.close(() => resolve()));
|
||||
|
||||
const s = AGENT_REVERSE_ID_BASE + 12;
|
||||
mockWs.emit('message', encodeJsonFrame({
|
||||
t: 'tcp_open_reverse', s,
|
||||
targetNodeId: localNodeId, stack: 'real', service: 'svc', port: closedPort,
|
||||
}), false);
|
||||
|
||||
const ack = await waitFor(() => findAck(mockWs, s));
|
||||
expect(ack.ok).toBe(false);
|
||||
expect(ack.err).toBe('unreachable');
|
||||
|
||||
const failCall = logSpy.mock.calls.find(
|
||||
(c: LogActivityArgs) => c[0].type === 'route.resolve.fail',
|
||||
);
|
||||
expect(failCall).toBeDefined();
|
||||
const payload = failCall![0] as {
|
||||
type: string;
|
||||
nodeId?: number;
|
||||
details?: Record<string, unknown>;
|
||||
};
|
||||
expect(payload.type).toBe('route.resolve.fail');
|
||||
expect(payload.nodeId).toBe(peerNodeId);
|
||||
expect(payload.details).toMatchObject({
|
||||
direction: 'reverse',
|
||||
reason: 'connect_error',
|
||||
targetStack: 'real',
|
||||
targetService: 'svc',
|
||||
targetPort: closedPort,
|
||||
});
|
||||
|
||||
bridge.close();
|
||||
});
|
||||
|
||||
it('emits route.resolve.ok with direction=reverse on connect ack', async () => {
|
||||
const mockWs = makeMockTunnelWs();
|
||||
const peerNodeId = 44;
|
||||
const bridge = new PilotTunnelBridge(peerNodeId, mockWs as unknown as WebSocket);
|
||||
await bridge.start();
|
||||
|
||||
// Real local server so the dial succeeds and 'connect' fires.
|
||||
const upstream = net.createServer((socket) => {
|
||||
socket.write('hello-upstream');
|
||||
});
|
||||
await new Promise<void>((resolve) => upstream.listen(0, '127.0.0.1', () => resolve()));
|
||||
const addr = upstream.address();
|
||||
if (!addr || typeof addr === 'string') throw new Error('no address');
|
||||
const upstreamPort = addr.port;
|
||||
|
||||
const localNodeId = (await import('../services/NodeRegistry')).NodeRegistry.getInstance().getDefaultNodeId();
|
||||
vi.spyOn(MeshService.getInstance(), 'resolveContainerIp').mockResolvedValue('127.0.0.1');
|
||||
|
||||
const s = AGENT_REVERSE_ID_BASE + 13;
|
||||
mockWs.emit('message', encodeJsonFrame({
|
||||
t: 'tcp_open_reverse', s,
|
||||
targetNodeId: localNodeId, stack: 'real', service: 'svc', port: upstreamPort,
|
||||
}), false);
|
||||
|
||||
const ack = await waitFor(() => findAck(mockWs, s));
|
||||
expect(ack.ok).toBe(true);
|
||||
|
||||
// Wait for the ok event to land (logActivity is sync but the connect
|
||||
// callback is async relative to message handling).
|
||||
await waitFor(() => logSpy.mock.calls.find(
|
||||
(c: LogActivityArgs) => c[0].type === 'route.resolve.ok',
|
||||
));
|
||||
|
||||
const okCall = logSpy.mock.calls.find(
|
||||
(c: LogActivityArgs) => c[0].type === 'route.resolve.ok',
|
||||
);
|
||||
expect(okCall).toBeDefined();
|
||||
const payload = okCall![0] as {
|
||||
source: string;
|
||||
level: string;
|
||||
type: string;
|
||||
nodeId?: number;
|
||||
details?: Record<string, unknown>;
|
||||
};
|
||||
expect(payload.source).toBe('mesh');
|
||||
expect(payload.level).toBe('info');
|
||||
expect(payload.type).toBe('route.resolve.ok');
|
||||
expect(payload.nodeId).toBe(peerNodeId);
|
||||
expect(payload.details).toMatchObject({
|
||||
direction: 'reverse',
|
||||
targetStack: 'real',
|
||||
targetService: 'svc',
|
||||
targetPort: upstreamPort,
|
||||
});
|
||||
|
||||
upstream.close();
|
||||
bridge.close();
|
||||
});
|
||||
|
||||
it('does NOT emit route.resolve.fail when a connected socket closes post-handshake', async () => {
|
||||
const mockWs = makeMockTunnelWs();
|
||||
const peerNodeId = 45;
|
||||
const bridge = new PilotTunnelBridge(peerNodeId, mockWs as unknown as WebSocket);
|
||||
await bridge.start();
|
||||
|
||||
// Real local server. Have it close the connection immediately after
|
||||
// accept so the bridge socket sees 'close' (and possibly 'error') on
|
||||
// an already-connected socket.
|
||||
const upstream = net.createServer((socket) => {
|
||||
socket.end();
|
||||
});
|
||||
await new Promise<void>((resolve) => upstream.listen(0, '127.0.0.1', () => resolve()));
|
||||
const addr = upstream.address();
|
||||
if (!addr || typeof addr === 'string') throw new Error('no address');
|
||||
const upstreamPort = addr.port;
|
||||
|
||||
const localNodeId = (await import('../services/NodeRegistry')).NodeRegistry.getInstance().getDefaultNodeId();
|
||||
vi.spyOn(MeshService.getInstance(), 'resolveContainerIp').mockResolvedValue('127.0.0.1');
|
||||
|
||||
const s = AGENT_REVERSE_ID_BASE + 14;
|
||||
mockWs.emit('message', encodeJsonFrame({
|
||||
t: 'tcp_open_reverse', s,
|
||||
targetNodeId: localNodeId, stack: 'real', service: 'svc', port: upstreamPort,
|
||||
}), false);
|
||||
|
||||
// Wait for connect (the success ack confirms onPreConnectError was removed).
|
||||
const ack = await waitFor(() => findAck(mockWs, s));
|
||||
expect(ack.ok).toBe(true);
|
||||
|
||||
// Wait for the success event to confirm the ok path fired.
|
||||
await waitFor(() => logSpy.mock.calls.find(
|
||||
(c: LogActivityArgs) => c[0].type === 'route.resolve.ok',
|
||||
));
|
||||
|
||||
// Give the post-connect close a chance to fire teardown handlers.
|
||||
await new Promise((r) => setTimeout(r, 100));
|
||||
|
||||
const failCalls = logSpy.mock.calls.filter(
|
||||
(c: LogActivityArgs) => c[0].type === 'route.resolve.fail',
|
||||
);
|
||||
expect(failCalls).toHaveLength(0);
|
||||
|
||||
upstream.close();
|
||||
bridge.close();
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,54 @@
|
||||
/**
|
||||
* PilotTunnelManager.replaceOrRegisterProxyBridge: peer-initiated proxy
|
||||
* bridges (Phase R1) need to be able to supersede a previous proxy bridge
|
||||
* for the same nodeId, but must never shadow a live pilot tunnel (which
|
||||
* always wins the bridge slot).
|
||||
*/
|
||||
import { describe, it, expect, beforeEach, vi } from 'vitest';
|
||||
import { EventEmitter } from 'events';
|
||||
import { PilotTunnelManager } from '../services/PilotTunnelManager';
|
||||
import type { PilotTunnelBridge } from '../services/PilotTunnelBridge';
|
||||
|
||||
type FakeBridge = PilotTunnelBridge & {
|
||||
close: ReturnType<typeof vi.fn<(code?: number, reason?: string) => void>>;
|
||||
getActiveStreamCount: ReturnType<typeof vi.fn<() => number>>;
|
||||
};
|
||||
|
||||
function makeFakeBridge(): FakeBridge {
|
||||
const ee = new EventEmitter() as unknown as FakeBridge;
|
||||
ee.close = vi.fn<(code?: number, reason?: string) => void>();
|
||||
ee.getActiveStreamCount = vi.fn<() => number>().mockReturnValue(0);
|
||||
return ee;
|
||||
}
|
||||
|
||||
describe('PilotTunnelManager.replaceOrRegisterProxyBridge', () => {
|
||||
beforeEach(() => PilotTunnelManager.resetForTest?.());
|
||||
|
||||
it('registers when no existing bridge', () => {
|
||||
const mgr = PilotTunnelManager.getInstance();
|
||||
const bridge = makeFakeBridge();
|
||||
mgr.replaceOrRegisterProxyBridge(42, bridge);
|
||||
expect(mgr.getBridge(42)).toBe(bridge);
|
||||
});
|
||||
|
||||
it('replaces when existing bridge is a proxy bridge (closes the old one)', () => {
|
||||
const mgr = PilotTunnelManager.getInstance();
|
||||
const oldBridge = makeFakeBridge();
|
||||
const newBridge = makeFakeBridge();
|
||||
mgr.injectBridgeForTest(42, oldBridge, 'proxy');
|
||||
mgr.replaceOrRegisterProxyBridge(42, newBridge);
|
||||
expect(oldBridge.close).toHaveBeenCalledWith(1000, 'replaced-by-newer-proxy');
|
||||
expect(mgr.getBridge(42)).toBe(newBridge);
|
||||
});
|
||||
|
||||
it('throws when existing bridge is a pilot tunnel (preserves the pilot tunnel)', () => {
|
||||
const mgr = PilotTunnelManager.getInstance();
|
||||
const pilotBridge = makeFakeBridge();
|
||||
const proxyBridge = makeFakeBridge();
|
||||
mgr.injectBridgeForTest(42, pilotBridge, 'pilot');
|
||||
expect(() => mgr.replaceOrRegisterProxyBridge(42, proxyBridge))
|
||||
.toThrow(/pilot tunnel.*proxy bridge refused/);
|
||||
expect(mgr.getBridge(42)).toBe(pilotBridge);
|
||||
expect(pilotBridge.close).not.toHaveBeenCalled();
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,83 @@
|
||||
/**
|
||||
* Tests for the `centralCallback` diag block returned by
|
||||
* GET /api/system/pilot-tunnels. The block reads from MeshCentralRegistry
|
||||
* (cached central material + last-used/last-rejected timestamps) and from
|
||||
* PeerToCentralMeshSessionDialer (live bridge presence). Counters for the
|
||||
* peer-callback path live in PilotMetrics and are exposed via the existing
|
||||
* `counters` field of the same response, so this block only adds the
|
||||
* cached-row diagnostics that PilotMetrics does not track.
|
||||
*/
|
||||
import { describe, it, expect, beforeAll, beforeEach, afterAll } from 'vitest';
|
||||
import request from 'supertest';
|
||||
import type { Express } from 'express';
|
||||
import { setupTestDb, cleanupTestDb, loginAsTestAdmin } from './helpers/setupTestDb';
|
||||
import { MeshCentralRegistry } from '../services/MeshCentralRegistry';
|
||||
import { PeerToCentralMeshSessionDialer } from '../services/PeerToCentralMeshSessionDialer';
|
||||
import { DatabaseService } from '../services/DatabaseService';
|
||||
|
||||
let tmpDir: string;
|
||||
let app: Express;
|
||||
let cookie: string;
|
||||
|
||||
beforeAll(async () => {
|
||||
tmpDir = await setupTestDb();
|
||||
({ app } = await import('../index'));
|
||||
cookie = await loginAsTestAdmin(app);
|
||||
});
|
||||
|
||||
afterAll(() => cleanupTestDb(tmpDir));
|
||||
|
||||
beforeEach(() => {
|
||||
MeshCentralRegistry.resetForTest();
|
||||
PeerToCentralMeshSessionDialer.resetForTest();
|
||||
DatabaseService.getInstance().getDb().prepare('DELETE FROM mesh_centrals').run();
|
||||
});
|
||||
|
||||
describe('/api/system/pilot-tunnels centralCallback diag', () => {
|
||||
it('returns null centralCallback fields when no central material is cached', async () => {
|
||||
const res = await request(app).get('/api/system/pilot-tunnels').set('Cookie', cookie);
|
||||
expect(res.status).toBe(200);
|
||||
expect(res.body.centralCallback).toEqual({
|
||||
bridgeOpen: false,
|
||||
lastBootstrapAt: null,
|
||||
lastDialOkAt: null,
|
||||
lastDialFailAt: null,
|
||||
lastDialFailReason: null,
|
||||
});
|
||||
});
|
||||
|
||||
it('returns populated centralCallback fields when material is present', async () => {
|
||||
MeshCentralRegistry.getInstance().upsert({
|
||||
centralInstanceId: 'inst-1',
|
||||
centralApiUrl: 'https://central.example.com',
|
||||
callbackJwt: 'jwt-value',
|
||||
jwtIssuedAt: 1,
|
||||
jwtExpiresAt: 9999999999,
|
||||
});
|
||||
MeshCentralRegistry.getInstance().markUsed('inst-1');
|
||||
|
||||
const res = await request(app).get('/api/system/pilot-tunnels').set('Cookie', cookie);
|
||||
expect(res.status).toBe(200);
|
||||
expect(res.body.centralCallback.bridgeOpen).toBe(false);
|
||||
expect(res.body.centralCallback.lastBootstrapAt).toBeGreaterThan(0);
|
||||
expect(res.body.centralCallback.lastDialOkAt).toBeGreaterThan(0);
|
||||
expect(res.body.centralCallback.lastDialFailAt).toBeNull();
|
||||
expect(res.body.centralCallback.lastDialFailReason).toBeNull();
|
||||
});
|
||||
|
||||
it('surfaces last reject reason when central marks a dial failed', async () => {
|
||||
MeshCentralRegistry.getInstance().upsert({
|
||||
centralInstanceId: 'inst-2',
|
||||
centralApiUrl: 'https://central.example.com',
|
||||
callbackJwt: 'jwt-value',
|
||||
jwtIssuedAt: 1,
|
||||
jwtExpiresAt: 9999999999,
|
||||
});
|
||||
MeshCentralRegistry.getInstance().markRejected('inst-2', 'signature_invalid');
|
||||
|
||||
const res = await request(app).get('/api/system/pilot-tunnels').set('Cookie', cookie);
|
||||
expect(res.status).toBe(200);
|
||||
expect(res.body.centralCallback.lastDialFailAt).toBeGreaterThan(0);
|
||||
expect(res.body.centralCallback.lastDialFailReason).toBe('signature_invalid');
|
||||
});
|
||||
});
|
||||
Reference in New Issue
Block a user