mirror of
https://github.com/Studio-Saelix/sencho.git
synced 2026-08-28 19:27:41 +00:00
cf618dd866
* chore(mesh): foundation for symmetric callback dial Adds the data-plane scaffolding that the symmetric callback dial fix builds on: - mesh_centrals table for peer-side bootstrap material - MeshCentralRegistry service (upsert/getActive/clear/markUsed/markRejected) - PilotTunnelManager kind discriminator and replaceOrRegisterProxyBridge - mesh_proxy_callback_bootstrap capability registration - MeshProxyTunnelDialer reason-tagged proxy-bridge-down events from a single tearDownBridge emission point - Reactive redial scheduler that skips idle and auth_failed reasons * feat(mesh): add reverse-direction activity log entries (closes R1-B) acceptReverseLocal now emits route.resolve.ok with direction=reverse on connect ack and route.resolve.fail with direction=reverse plus reason=container_not_found / connect_error pre-connect. Post-connect close/error stays silent. Reuses existing event types via the new details.direction discriminator so frontend filters are unaffected. * feat(mesh): add peer-to-central callback dial path (closes R1-A2) Closes the architectural gap where proxy-mode mesh peers could not re-establish their tunnel to central after any non-idle bridge teardown (idle close, network blip, central restart, peer reboot). Central remains the hub for the data plane; the change is purely about WS initiation. Symmetric WS initiation, asymmetric protocol roles. Central retains PilotTunnelBridge ownership; peer retains TcpStreamSwitchboard + reverseDialer ownership. Central bootstraps callback credentials over the first authenticated central-initiated mesh tunnel via a one-shot mesh_handshake JSON frame; peer persists the material in a new mesh_centrals SQLite table and dials central's new /api/mesh/proxy-tunnel-from-peer endpoint when local cross-node traffic needs a bridge and none is live. Mesh_tunnel JWT (HS256, signed with auth_jwt_secret) carries scope, audience, issuer (central instance id), peer api_token fingerprint, kid. Validation on inbound peer dial: algorithm pin, signature, scope, audience, instance, time bounds, node existence and mode, fingerprint match. Failures return HTTP 401 with a machine-readable reason; peer routes the response per a clear-vs-keep cache matrix. Triggers proactive bootstrap on mesh-enable and api_token rotation; central startup fans out to mesh-enabled proxy-mode nodes with mesh_stacks rows (throttled, fire-and-forget). Reactive redial on non-idle bridge loss. Capability-gated handshake send (mesh_proxy_callback_bootstrap) makes the upgrade path safe against older peers in mixed-version fleets. Adds peer-side /api/system/pilot-tunnels centralCallback diag block, bounded counter metrics for bootstrap and dial events. SENCHO_PRIMARY_URL preflight warning when unset on a central with mesh-enabled proxy nodes. Tested with unit suites for the validation chain, registry, manager, and both dialers; integration tests for bootstrap E2E (asserts protocol-role invariant), api_token rotation, instance id change, version skew, and pilot-mode regression. * fix(mesh): green CI on the symmetric callback branch Two independent CI failures, both surgical: 1. Backend tests (11 fails): four mesh test files called setupTestDb in beforeEach. setupTestDb does not reset the DatabaseService singleton, so the per-test afterEach rm of the previous tmpdir left the singleton connection pointing at a deleted file. The next beforeEach's line-55 write threw SQLITE_READONLY_DBMOVED on Linux. Windows file-lock semantics hid this locally. Hoist setupTestDb / cleanupTestDb to file-scope beforeAll / afterAll; per-test state resets stay in beforeEach. Matches the convention in the eight mesh test files that already pass. 2. CodeQL (4 high alerts): js/insufficient-password-hash flagged sha256(api_token) at four sites. The api_token is a 256-bit opaque bearer (sen_sk_-prefixed), not a human password; sha256 is the correct fingerprint primitive for binding the mesh_tunnel JWT to a specific token. Add the two production files plus the two test files that mint the fingerprint to the existing path-scoped query-filter for that rule. * fix(mesh): drop unused afterEach import and revert dead codeql config ESLint flagged afterEach as unused in mesh-central-registry.test.ts:1 after the previous commit hoisted setup/teardown to file-scope beforeAll/afterAll. Remove from the vitest import line. Revert the codeql-config.yml additions from the previous commit. The paths: sub-key under query-filters > exclude is not a documented CodeQL feature and silently no-ops. The four js/insufficient-password-hash alerts on api_token fingerprinting are tracked as dismissed false positives in the GitHub Security tab rather than via dead config.
283 lines
12 KiB
TypeScript
283 lines
12 KiB
TypeScript
/**
|
|
* `meshProxyTunnelFromPeer.ts`: central-side ingress for peer-initiated
|
|
* dial-back tunnels. Validates the chain of JWT claims and node-state
|
|
* preconditions that the peer's `mesh_tunnel` bootstrap token must satisfy
|
|
* before the upgrade succeeds and a proxy bridge is registered.
|
|
*
|
|
* The chain (in order): algorithm whitelist, signature, scope, audience
|
|
* (= SENCHO_PRIMARY_URL), issuer (= central instance_id), exp, iat clock
|
|
* skew (60s), node existence, mode==proxy, api_token fingerprint match.
|
|
* Every failure point returns HTTP 401 with a JSON `{reason}` body using
|
|
* a stable machine-readable code; the happy path constructs a
|
|
* `PilotTunnelBridge` and registers it via
|
|
* `PilotTunnelManager.replaceOrRegisterProxyBridge`.
|
|
*/
|
|
import http from 'http';
|
|
import { afterAll, beforeAll, beforeEach, describe, expect, it } from 'vitest';
|
|
import WebSocket from 'ws';
|
|
import jwt from 'jsonwebtoken';
|
|
import { createHash } from 'crypto';
|
|
import type { AddressInfo } from 'net';
|
|
import { setupTestDb, cleanupTestDb } from './helpers/setupTestDb';
|
|
|
|
let tmpDir: string;
|
|
let handleMeshProxyTunnelFromPeerUpgrade: typeof import('../websocket/meshProxyTunnelFromPeer').handleMeshProxyTunnelFromPeerUpgrade;
|
|
let DatabaseService: typeof import('../services/DatabaseService').DatabaseService;
|
|
let PilotTunnelManager: typeof import('../services/PilotTunnelManager').PilotTunnelManager;
|
|
|
|
interface ServerHandle {
|
|
server: http.Server;
|
|
port: number;
|
|
close: () => Promise<void>;
|
|
}
|
|
|
|
async function startServer(): Promise<ServerHandle> {
|
|
const server = http.createServer();
|
|
server.on('upgrade', (req, socket, head) => {
|
|
const pathname = new URL(req.url ?? '/', 'http://localhost').pathname;
|
|
if (pathname === '/api/mesh/proxy-tunnel-from-peer') {
|
|
handleMeshProxyTunnelFromPeerUpgrade(req, socket, head);
|
|
} else {
|
|
socket.destroy();
|
|
}
|
|
});
|
|
await new Promise<void>((resolve) => server.listen(0, '127.0.0.1', () => resolve()));
|
|
const port = (server.address() as AddressInfo).port;
|
|
return {
|
|
server,
|
|
port,
|
|
close: () => new Promise<void>((resolve) => server.close(() => resolve())),
|
|
};
|
|
}
|
|
|
|
interface UpgradeOutcome {
|
|
kind: 'open' | 'unexpected' | 'error';
|
|
status?: number;
|
|
body?: string;
|
|
ws?: WebSocket;
|
|
}
|
|
|
|
function attemptUpgrade(port: number, token: string): Promise<UpgradeOutcome> {
|
|
return new Promise((resolve) => {
|
|
const ws = new WebSocket(`ws://127.0.0.1:${port}/api/mesh/proxy-tunnel-from-peer`, {
|
|
headers: { authorization: `Bearer ${token}` },
|
|
});
|
|
const timer = setTimeout(() => resolve({ kind: 'error' }), 3000);
|
|
ws.once('open', () => {
|
|
clearTimeout(timer);
|
|
resolve({ kind: 'open', ws });
|
|
});
|
|
ws.once('unexpected-response', (_req, res) => {
|
|
const chunks: Buffer[] = [];
|
|
res.on('data', (c: Buffer) => chunks.push(c));
|
|
res.on('end', () => {
|
|
clearTimeout(timer);
|
|
resolve({ kind: 'unexpected', status: res.statusCode, body: Buffer.concat(chunks).toString('utf8') });
|
|
});
|
|
});
|
|
ws.once('error', () => {
|
|
// 'unexpected-response' fires first; 'error' here is the
|
|
// post-handshake failure ws raises after the server destroys
|
|
// the socket. Already resolved above.
|
|
});
|
|
});
|
|
}
|
|
|
|
function parseReason(body: string | undefined): string | null {
|
|
if (!body) return null;
|
|
try {
|
|
const parsed = JSON.parse(body) as { reason?: unknown };
|
|
return typeof parsed.reason === 'string' ? parsed.reason : null;
|
|
} catch {
|
|
return null;
|
|
}
|
|
}
|
|
|
|
const CANONICAL_ORIGIN = 'https://central.example.com';
|
|
const INSTANCE_ID = 'test-central-instance';
|
|
const PEER_API_TOKEN = 'peer-token-123';
|
|
|
|
let secret: string;
|
|
let peerNodeId: number;
|
|
let srv: ServerHandle;
|
|
|
|
beforeAll(async () => {
|
|
tmpDir = await setupTestDb();
|
|
({ handleMeshProxyTunnelFromPeerUpgrade } = await import('../websocket/meshProxyTunnelFromPeer'));
|
|
({ DatabaseService } = await import('../services/DatabaseService'));
|
|
({ PilotTunnelManager } = await import('../services/PilotTunnelManager'));
|
|
});
|
|
|
|
beforeEach(async () => {
|
|
process.env.SENCHO_PRIMARY_URL = CANONICAL_ORIGIN;
|
|
const db = DatabaseService.getInstance();
|
|
secret = db.getGlobalSettings().auth_jwt_secret;
|
|
db.setSystemState('instance_id', INSTANCE_ID);
|
|
// Unique peer per test to avoid PilotTunnelManager collisions across
|
|
// the 12 cases (one happy path actually registers a bridge).
|
|
peerNodeId = db.addNode({
|
|
name: `peer-${Date.now()}-${Math.random().toString(36).slice(2, 8)}`,
|
|
type: 'remote',
|
|
mode: 'proxy',
|
|
api_url: 'https://peer.example.com',
|
|
api_token: PEER_API_TOKEN,
|
|
compose_dir: '/tmp',
|
|
is_default: false,
|
|
});
|
|
db.setNodeMeshEnabled(peerNodeId, true);
|
|
if (!srv) srv = await startServer();
|
|
});
|
|
|
|
afterAll(async () => {
|
|
if (srv) await srv.close();
|
|
delete process.env.SENCHO_PRIMARY_URL;
|
|
cleanupTestDb(tmpDir);
|
|
});
|
|
|
|
function expectedFp(token: string = PEER_API_TOKEN): string {
|
|
return createHash('sha256').update(token).digest('hex').slice(0, 16);
|
|
}
|
|
|
|
function makeJwt(overrides: Partial<Record<string, unknown>> = {}, signOpts: { alg?: jwt.Algorithm; secret?: string } = {}): string {
|
|
const payload: Record<string, unknown> = {
|
|
sub: String(peerNodeId),
|
|
iss: INSTANCE_ID,
|
|
aud: CANONICAL_ORIGIN,
|
|
scope: 'mesh_tunnel',
|
|
iat: Math.floor(Date.now() / 1000),
|
|
exp: Math.floor(Date.now() / 1000) + 3600,
|
|
kid: 'v1',
|
|
peer_token_fp: expectedFp(),
|
|
...overrides,
|
|
};
|
|
return jwt.sign(payload, signOpts.secret ?? secret, { algorithm: signOpts.alg ?? 'HS256' });
|
|
}
|
|
|
|
describe('/api/mesh/proxy-tunnel-from-peer validation chain', () => {
|
|
it('rejects alg=none (algorithm_mismatch)', async () => {
|
|
// jsonwebtoken refuses to sign with alg=none unless explicitly
|
|
// enabled and given a null secret; we build the token manually so
|
|
// the test exercises the central's defence, not the library's.
|
|
const header = Buffer.from(JSON.stringify({ alg: 'none', typ: 'JWT' })).toString('base64url');
|
|
const body = Buffer.from(JSON.stringify({
|
|
sub: String(peerNodeId), iss: INSTANCE_ID, aud: CANONICAL_ORIGIN,
|
|
scope: 'mesh_tunnel', iat: Math.floor(Date.now() / 1000),
|
|
exp: Math.floor(Date.now() / 1000) + 3600, peer_token_fp: expectedFp(),
|
|
})).toString('base64url');
|
|
const token = `${header}.${body}.`;
|
|
const outcome = await attemptUpgrade(srv.port, token);
|
|
expect(outcome.kind).toBe('unexpected');
|
|
expect(outcome.status).toBe(401);
|
|
expect(parseReason(outcome.body)).toBe('algorithm_mismatch');
|
|
});
|
|
|
|
it('rejects alg=RS256 (algorithm_mismatch)', async () => {
|
|
const { generateKeyPairSync } = await import('crypto');
|
|
const { privateKey } = generateKeyPairSync('rsa', { modulusLength: 2048 });
|
|
const token = jwt.sign({
|
|
sub: String(peerNodeId), iss: INSTANCE_ID, aud: CANONICAL_ORIGIN,
|
|
scope: 'mesh_tunnel', exp: Math.floor(Date.now() / 1000) + 3600,
|
|
peer_token_fp: expectedFp(),
|
|
}, privateKey, { algorithm: 'RS256' });
|
|
const outcome = await attemptUpgrade(srv.port, token);
|
|
expect(outcome.kind).toBe('unexpected');
|
|
expect(outcome.status).toBe(401);
|
|
expect(parseReason(outcome.body)).toBe('algorithm_mismatch');
|
|
});
|
|
|
|
it('rejects bad signature (signature_invalid)', async () => {
|
|
const token = makeJwt({}, { secret: 'wrong-secret-not-the-real-one' });
|
|
const outcome = await attemptUpgrade(srv.port, token);
|
|
expect(outcome.kind).toBe('unexpected');
|
|
expect(outcome.status).toBe(401);
|
|
expect(parseReason(outcome.body)).toBe('signature_invalid');
|
|
});
|
|
|
|
it('rejects scope mismatch', async () => {
|
|
const token = makeJwt({ scope: 'pilot_tunnel' });
|
|
const outcome = await attemptUpgrade(srv.port, token);
|
|
expect(outcome.kind).toBe('unexpected');
|
|
expect(outcome.status).toBe(401);
|
|
expect(parseReason(outcome.body)).toBe('scope_mismatch');
|
|
});
|
|
|
|
it('rejects audience mismatch', async () => {
|
|
const token = makeJwt({ aud: 'https://other.example.com' });
|
|
const outcome = await attemptUpgrade(srv.port, token);
|
|
expect(outcome.kind).toBe('unexpected');
|
|
expect(outcome.status).toBe(401);
|
|
expect(parseReason(outcome.body)).toBe('audience_mismatch');
|
|
});
|
|
|
|
it('rejects issuer mismatch', async () => {
|
|
const token = makeJwt({ iss: 'wrong-instance-id' });
|
|
const outcome = await attemptUpgrade(srv.port, token);
|
|
expect(outcome.kind).toBe('unexpected');
|
|
expect(outcome.status).toBe(401);
|
|
expect(parseReason(outcome.body)).toBe('instance_mismatch');
|
|
});
|
|
|
|
it('rejects expired token (stale)', async () => {
|
|
const token = makeJwt({ exp: Math.floor(Date.now() / 1000) - 10 });
|
|
const outcome = await attemptUpgrade(srv.port, token);
|
|
expect(outcome.kind).toBe('unexpected');
|
|
expect(outcome.status).toBe(401);
|
|
// jsonwebtoken throws on expired tokens before our exp check sees it,
|
|
// so the rejection surfaces as signature_invalid via the verify catch.
|
|
// The contract: stale tokens are rejected with a 401 and some
|
|
// deterministic reason code; accept either of the two equivalent
|
|
// failures since both convey "stale credential" to the operator.
|
|
const reason = parseReason(outcome.body);
|
|
expect(['stale', 'signature_invalid']).toContain(reason);
|
|
});
|
|
|
|
it('rejects clock-skewed token (clock_skew)', async () => {
|
|
const token = makeJwt({ iat: Math.floor(Date.now() / 1000) + 600 });
|
|
const outcome = await attemptUpgrade(srv.port, token);
|
|
expect(outcome.kind).toBe('unexpected');
|
|
expect(outcome.status).toBe(401);
|
|
expect(parseReason(outcome.body)).toBe('clock_skew');
|
|
});
|
|
|
|
it('rejects missing node (node_deleted)', async () => {
|
|
const token = makeJwt({ sub: '999999' });
|
|
const outcome = await attemptUpgrade(srv.port, token);
|
|
expect(outcome.kind).toBe('unexpected');
|
|
expect(outcome.status).toBe(401);
|
|
expect(parseReason(outcome.body)).toBe('node_deleted');
|
|
});
|
|
|
|
it('rejects mode mismatch', async () => {
|
|
DatabaseService.getInstance().updateNode(peerNodeId, { mode: 'pilot_agent' });
|
|
const token = makeJwt();
|
|
const outcome = await attemptUpgrade(srv.port, token);
|
|
expect(outcome.kind).toBe('unexpected');
|
|
expect(outcome.status).toBe(401);
|
|
expect(parseReason(outcome.body)).toBe('mode_mismatch');
|
|
});
|
|
|
|
it('rejects token fingerprint mismatch', async () => {
|
|
const token = makeJwt();
|
|
// Rotate the api_token after minting; the JWT now carries the
|
|
// fingerprint of the old token, but getNode returns the new one.
|
|
DatabaseService.getInstance().updateNode(peerNodeId, { api_token: 'rotated-token-xyz' });
|
|
const outcome = await attemptUpgrade(srv.port, token);
|
|
expect(outcome.kind).toBe('unexpected');
|
|
expect(outcome.status).toBe(401);
|
|
expect(parseReason(outcome.body)).toBe('token_fingerprint_mismatch');
|
|
});
|
|
|
|
it('accepts a fully valid token and registers a proxy bridge', async () => {
|
|
const token = makeJwt();
|
|
const outcome = await attemptUpgrade(srv.port, token);
|
|
expect(outcome.kind).toBe('open');
|
|
// Give the bridge.start() microtasks a moment to land and register.
|
|
await new Promise((r) => setTimeout(r, 50));
|
|
const bridge = PilotTunnelManager.getInstance().getBridge(peerNodeId);
|
|
expect(bridge).not.toBeNull();
|
|
try { outcome.ws?.close(1000, 'test cleanup'); } catch { /* ignore */ }
|
|
// Allow the manager's 'closed' handler to remove the bridge entry.
|
|
await new Promise((r) => setTimeout(r, 30));
|
|
});
|
|
});
|