mirror of
https://github.com/Studio-Saelix/sencho.git
synced 2026-08-25 09:46:47 +00:00
feat(mesh): symmetric WS dial for proxy-mode mesh peers (#1066)
* chore(mesh): foundation for symmetric callback dial Adds the data-plane scaffolding that the symmetric callback dial fix builds on: - mesh_centrals table for peer-side bootstrap material - MeshCentralRegistry service (upsert/getActive/clear/markUsed/markRejected) - PilotTunnelManager kind discriminator and replaceOrRegisterProxyBridge - mesh_proxy_callback_bootstrap capability registration - MeshProxyTunnelDialer reason-tagged proxy-bridge-down events from a single tearDownBridge emission point - Reactive redial scheduler that skips idle and auth_failed reasons * feat(mesh): add reverse-direction activity log entries (closes R1-B) acceptReverseLocal now emits route.resolve.ok with direction=reverse on connect ack and route.resolve.fail with direction=reverse plus reason=container_not_found / connect_error pre-connect. Post-connect close/error stays silent. Reuses existing event types via the new details.direction discriminator so frontend filters are unaffected. * feat(mesh): add peer-to-central callback dial path (closes R1-A2) Closes the architectural gap where proxy-mode mesh peers could not re-establish their tunnel to central after any non-idle bridge teardown (idle close, network blip, central restart, peer reboot). Central remains the hub for the data plane; the change is purely about WS initiation. Symmetric WS initiation, asymmetric protocol roles. Central retains PilotTunnelBridge ownership; peer retains TcpStreamSwitchboard + reverseDialer ownership. Central bootstraps callback credentials over the first authenticated central-initiated mesh tunnel via a one-shot mesh_handshake JSON frame; peer persists the material in a new mesh_centrals SQLite table and dials central's new /api/mesh/proxy-tunnel-from-peer endpoint when local cross-node traffic needs a bridge and none is live. Mesh_tunnel JWT (HS256, signed with auth_jwt_secret) carries scope, audience, issuer (central instance id), peer api_token fingerprint, kid. Validation on inbound peer dial: algorithm pin, signature, scope, audience, instance, time bounds, node existence and mode, fingerprint match. Failures return HTTP 401 with a machine-readable reason; peer routes the response per a clear-vs-keep cache matrix. Triggers proactive bootstrap on mesh-enable and api_token rotation; central startup fans out to mesh-enabled proxy-mode nodes with mesh_stacks rows (throttled, fire-and-forget). Reactive redial on non-idle bridge loss. Capability-gated handshake send (mesh_proxy_callback_bootstrap) makes the upgrade path safe against older peers in mixed-version fleets. Adds peer-side /api/system/pilot-tunnels centralCallback diag block, bounded counter metrics for bootstrap and dial events. SENCHO_PRIMARY_URL preflight warning when unset on a central with mesh-enabled proxy nodes. Tested with unit suites for the validation chain, registry, manager, and both dialers; integration tests for bootstrap E2E (asserts protocol-role invariant), api_token rotation, instance id change, version skew, and pilot-mode regression. * fix(mesh): green CI on the symmetric callback branch Two independent CI failures, both surgical: 1. Backend tests (11 fails): four mesh test files called setupTestDb in beforeEach. setupTestDb does not reset the DatabaseService singleton, so the per-test afterEach rm of the previous tmpdir left the singleton connection pointing at a deleted file. The next beforeEach's line-55 write threw SQLITE_READONLY_DBMOVED on Linux. Windows file-lock semantics hid this locally. Hoist setupTestDb / cleanupTestDb to file-scope beforeAll / afterAll; per-test state resets stay in beforeEach. Matches the convention in the eight mesh test files that already pass. 2. CodeQL (4 high alerts): js/insufficient-password-hash flagged sha256(api_token) at four sites. The api_token is a 256-bit opaque bearer (sen_sk_-prefixed), not a human password; sha256 is the correct fingerprint primitive for binding the mesh_tunnel JWT to a specific token. Add the two production files plus the two test files that mint the fingerprint to the existing path-scoped query-filter for that rule. * fix(mesh): drop unused afterEach import and revert dead codeql config ESLint flagged afterEach as unused in mesh-central-registry.test.ts:1 after the previous commit hoisted setup/teardown to file-scope beforeAll/afterAll. Remove from the vitest import line. Revert the codeql-config.yml additions from the previous commit. The paths: sub-key under query-filters > exclude is not a documented CodeQL feature and silently no-ops. The four js/insufficient-password-hash alerts on api_token fingerprinting are tracked as dismissed false positives in the GitHub Security tab rather than via dead config.
This commit is contained in:
@@ -6,6 +6,8 @@ import { DatabaseService } from '../services/DatabaseService';
|
||||
import { CacheService } from '../services/CacheService';
|
||||
import { NodeRegistry } from '../services/NodeRegistry';
|
||||
import { PilotTunnelManager } from '../services/PilotTunnelManager';
|
||||
import { MeshCentralRegistry } from '../services/MeshCentralRegistry';
|
||||
import { PeerToCentralMeshSessionDialer } from '../services/PeerToCentralMeshSessionDialer';
|
||||
import { authMiddleware } from '../middleware/auth';
|
||||
import { requireAdmin } from '../middleware/tierGates';
|
||||
import { STATS_CACHE_TTL_MS, SYSTEM_STATS_CACHE_TTL_MS } from '../helpers/constants';
|
||||
@@ -260,7 +262,22 @@ metricsRouter.get('/system/cache-stats', authMiddleware, async (req: Request, re
|
||||
metricsRouter.get('/system/pilot-tunnels', authMiddleware, async (req: Request, res: Response): Promise<void> => {
|
||||
if (!requireAdmin(req, res)) return;
|
||||
try {
|
||||
res.json(PilotTunnelManager.getInstance().getMetricsSnapshot());
|
||||
const snapshot = PilotTunnelManager.getInstance().getMetricsSnapshot();
|
||||
// centralCallback exposes the peer-side view of the symmetric mesh
|
||||
// callback path (cached central material plus the live bridge presence).
|
||||
// PilotMetrics already tracks the attempt/success counters via
|
||||
// `counters.mesh_central_bootstraps_total` and friends; this block adds
|
||||
// the per-instance last-success / last-failure diagnostics that live in
|
||||
// MeshCentralRegistry rather than PilotMetrics.
|
||||
const cached = MeshCentralRegistry.getInstance().getActive();
|
||||
const centralCallback = {
|
||||
bridgeOpen: PeerToCentralMeshSessionDialer.getInstance().hasSession(),
|
||||
lastBootstrapAt: cached?.lastBootstrapAt ?? null,
|
||||
lastDialOkAt: cached?.lastUsedAt ?? null,
|
||||
lastDialFailAt: cached?.lastRejectedAt ?? null,
|
||||
lastDialFailReason: cached?.lastRejectReason ?? null,
|
||||
};
|
||||
res.json({ ...snapshot, centralCallback });
|
||||
} catch (error) {
|
||||
console.error('Failed to fetch pilot tunnel metrics:', error);
|
||||
res.status(500).json({ error: 'Failed to fetch pilot tunnel metrics' });
|
||||
|
||||
@@ -13,6 +13,7 @@ import { REMOTE_META_NAMESPACE } from '../helpers/cacheInvalidation';
|
||||
import { CAPABILITIES, getSenchoVersion, type RemoteMeta } from '../services/CapabilityRegistry';
|
||||
import { PilotTunnelManager } from '../services/PilotTunnelManager';
|
||||
import { PilotCloseCode } from '../pilot/protocol';
|
||||
import { MeshProxyTunnelDialer } from '../services/MeshProxyTunnelDialer';
|
||||
import { FleetUpdateTrackerService } from '../services/FleetUpdateTrackerService';
|
||||
import { FleetSyncService } from '../services/FleetSyncService';
|
||||
import { isValidRemoteUrl } from '../utils/validation';
|
||||
@@ -235,6 +236,21 @@ nodesRouter.put('/:id', async (req: Request, res: Response) => {
|
||||
NodeRegistry.getInstance().evictConnection(id);
|
||||
NodeRegistry.getInstance().notifyNodeUpdated(id);
|
||||
|
||||
// Trigger 2: if the api_token was rotated on a mesh-enabled proxy-mode
|
||||
// remote, close the existing callback bridge and re-dial. The next
|
||||
// ensureBridge mints a JWT with the fresh token fingerprint so the
|
||||
// remote's tunnel auth gate accepts the upgrade.
|
||||
if (typeof updates.api_token === 'string') {
|
||||
const node = DatabaseService.getInstance().getNode(id);
|
||||
const meshEnabled = DatabaseService.getInstance().getNodeMeshEnabled(id);
|
||||
if (node && node.type === 'remote' && node.mode === 'proxy' && meshEnabled) {
|
||||
MeshProxyTunnelDialer.getInstance().closeBridge(id, 'peer token rotated');
|
||||
void MeshProxyTunnelDialer.getInstance().ensureBridge(id).catch((err) => {
|
||||
console.warn(`[Mesh] proactive re-bootstrap on token rotation failed for node ${id}: ${(err as Error).message}`);
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
const isPlainHttp = updates.api_url && updates.api_url.startsWith('http://');
|
||||
res.json({
|
||||
success: true,
|
||||
|
||||
Reference in New Issue
Block a user