gsadmin/sencho
1
0
Fork 0
mirror of https://github.com/Studio-Saelix/sencho.git synced 2026-08-27 18:57:09 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat(mesh): bidirectional routing via tcp_open_reverse and central relay (#1003)

Browse Source 
* feat(mesh): bidirectional routing via tcp_open_reverse and central relay

Phase B of the mesh redesign. Adds the reverse-direction protocol so a
pilot's MeshForwarder can route cross-node traffic to central or to
another pilot. Central relays pilot-to-pilot streams transparently;
pilots keep their existing single outbound WS to central.

Protocol additions (backend/src/pilot/protocol.ts):
- TcpOpenReverseFrame { s, targetNodeId, stack, service, port } sent
  agent to primary. Reuses existing tcp_open_ack, tcp_close, and
  TcpData binary frames for the response and byte plane.
- AGENT_REVERSE_ID_BASE = 0x40000001 splits the 32-bit id space so
  agent-allocated reverse stream ids never collide with primary-
  allocated forward ids on the same tunnel.
- StreamIdAllocator now wraps to its configured start (not always 1)
  so an allocator parameterized with the agent base stays in the
  agent half across the wrap.

Agent (backend/src/pilot/agent.ts):
- New ReverseTcpStreamHandle exported class, EventEmitter facade
  matching PilotTunnelBridge.TcpStream's surface (write, end,
  destroy plus open, data, error, close events).
- Public openMeshTcpStream(target) allocates a reverse id, sends
  tcp_open_reverse, returns the handle.
- onTcpOpenAckReverse handles inbound ack, dispatches open or
  error+close to the matching handle.
- TcpData and tcp_close binary/JSON paths route ids in the reverse
  range to reverseTcpStreams; existing primary-allocated paths
  unchanged.
- streamCount, cleanupAfterDisconnect, onStreamIdle include reverse
  streams. The allocator is reset to a fresh base on disconnect so
  long-lived agents that reconnect many times do not drift up the id
  space.
- startPilotAgent registers the agent as MeshService's reverse
  dialer via lazy import.

Bridge (backend/src/services/PilotTunnelBridge.ts):
- Two new StreamState kinds: reverse_local (target = central, dial
  Dockerode container IP) and reverse_relay (target = another pilot,
  open a forward TcpStream on the target pilot's bridge).
- handleTcpOpenReverse validates the agent-id range and stream cap,
  then dispatches to acceptReverseLocal or acceptReverseRelay via
  lazy imports of MeshService, NodeRegistry, and PilotTunnelManager
  (avoids module cycles).
- acceptReverseLocal calls MeshService.resolveContainerIp (now
  public), opens net.createConnection, splices bytes through the
  tunnel. Pre-connect error handler is removed inside connect to
  avoid double-firing with the mid-stream error path.
- acceptReverseRelay opens a forward TcpStream on the target
  bridge, splices bytes between the two tunnels.
- Existing tcp_close and TcpData handlers extended for the new
  state kinds. teardownStream extended.

MeshService (backend/src/services/MeshService.ts):
- resolveContainerIp made public so the bridge's local-target path
  can dial the same shape.
- New reverseDialer field plus setReverseDialer setter. Pilot mode
  registers the agent at boot; central mode leaves it null.
- New private dialMeshTcpStream dispatcher: pilot side uses the
  reverse dialer, central side uses PilotTunnelManager.getBridge.
- openCrossNode refactored to call the dispatcher and use the
  active-stream record's id for activity logging.
- New exported MeshTcpStreamLike interface that both
  PilotTunnelBridge.TcpStream and ReverseTcpStreamHandle satisfy
  structurally; ReverseMeshDialer is the contract for the setter.

Tests (3 files, 16 new cases, 172 total pass):
- pilot-protocol-tcp.test.ts: tcp_open_reverse round-trip; agent
  base invariant.
- pilot-agent-reverse-stream.test.ts: openMeshTcpStream allocation,
  ws-not-open, frame shape, write encoding, end emits tcp_close,
  ack-success, ack-failure, low-id ack ignored, inbound TcpData
  routing.
- pilot-bridge-reverse.test.ts: id-range validation, no-target
  failure, successful local dial against a real upstream server.

Together with PR #1000 (Phase A) and PR #1001 (deps), Phase B
completes the central-pilot-pilot mesh routing matrix. Phase C
(mesh over proxy-mode remotes) is the planned follow-up.

* test(pilot): drop unused encodeJsonFrame import

Lint failed on pilot-agent-reverse-stream.test.ts after the test
changed from constructing tcp_open_reverse frames inline to driving
the agent's frame-dispatch path with synthetic objects. The import
is no longer referenced; ESLint's no-unused-vars rule rejected it.
This commit is contained in:
Anso
2026-05-08 16:21:34 -04:00
committed by GitHub
parent f599110386
commit 567e524450
7 changed files with 884 additions and 35 deletions
Download Patch File Download Diff File Expand all files Collapse all files
backend/src/pilot/agent.ts
+183 -1
View File
@@ -2,18 +2,21 @@ import fs from 'fs';
import net from 'net';
import path from 'path';
import http from 'http';
import { EventEmitter } from 'events';
import jwt from 'jsonwebtoken';
import WebSocket from 'ws';
import { getSenchoVersion } from '../services/CapabilityRegistry';
import { DatabaseService } from '../services/DatabaseService';
import { NodeRegistry } from '../services/NodeRegistry';
import {
AGENT_REVERSE_ID_BASE,
BinaryFrameType,
MAX_FRAME_SIZE_BYTES,
MAX_STREAMS_PER_TUNNEL,
MeshErrCode,
PROTOCOL_VERSION,
STREAM_IDLE_TIMEOUT_MS,
StreamIdAllocator,
decodeBinaryFrame,
decodeJsonFrame,
encodeBinaryFrame,
@@ -58,6 +61,16 @@ export function startPilotAgent(loopbackPort: number): void {
initialToken: persistedToken || enrollToken!,
enrolling: !persistedToken,
});
// Register the agent as MeshService's reverse dialer so outbound
// cross-node mesh traffic from this pilot's MeshForwarder routes via
// `tcp_open_reverse` over the existing pilot tunnel instead of trying
// to use the central-only `PilotTunnelManager.getBridge` path. Lazy
// import keeps `MeshService` outside the cold-boot critical path.
void import('../services/MeshService').then(({ MeshService }) => {
MeshService.getInstance().setReverseDialer(agent);
}).catch((err) => {
console.warn('[Pilot] reverse dialer registration failed:', sanitizeForLog((err as Error).message));
});
agent.start();
}
@@ -78,6 +91,10 @@ export class PilotAgent {
private readonly httpStreams = new Map<number, { req: http.ClientRequest }>();
private readonly wsStreams = new Map<number, WebSocket>();
private readonly tcpStreams = new Map<number, MeshTcpStream>();
/** Reverse mesh streams the agent itself initiated via `tcp_open_reverse`. Keyed on the agent-allocated id. Disjoint from `tcpStreams` because those are primary-allocated and live in the lower id half. */
private readonly reverseTcpStreams = new Map<number, ReverseTcpStreamHandle>();
/** Allocator is recreated on every disconnect (`cleanupAfterDisconnect`) so a long-lived agent that reconnects many times doesn't drift up the id range. */
private reverseStreamIds = new StreamIdAllocator(AGENT_REVERSE_ID_BASE);
private readonly idleTimers = new Map<number, NodeJS.Timeout>();
private shuttingDown = false;
private readonly agentVersion: string;
@@ -221,12 +238,23 @@ export class PilotAgent {
try { stream.socket.destroy(); } catch { /* ignore */ }
}
this.tcpStreams.clear();
for (const [, handle] of this.reverseTcpStreams) {
try {
handle._dispatchError(new Error('pilot tunnel closed'));
handle._dispatchClose();
} catch { /* ignore */ }
}
this.reverseTcpStreams.clear();
// Reset the reverse allocator so a long-lived agent that
// reconnects many times doesn't drift up the id range and
// approach the wrap point unnecessarily.
this.reverseStreamIds = new StreamIdAllocator(AGENT_REVERSE_ID_BASE);
for (const [, timer] of this.idleTimers) clearTimeout(timer);
this.idleTimers.clear();
}
private streamCount(): number {
return this.httpStreams.size + this.wsStreams.size + this.tcpStreams.size;
return this.httpStreams.size + this.wsStreams.size + this.tcpStreams.size + this.reverseTcpStreams.size;
}
private refreshIdleTimer(streamId: number): void {
@@ -272,6 +300,16 @@ export class PilotAgent {
if (ws) {
try { ws.send(encodeJsonFrame({ t: 'tcp_close', s: streamId })); } catch { /* ignore */ }
}
return;
}
const reverseEntry = this.reverseTcpStreams.get(streamId);
if (reverseEntry) {
this.reverseTcpStreams.delete(streamId);
reverseEntry._dispatchError(new Error('agent idle timeout'));
reverseEntry._dispatchClose();
if (ws) {
try { ws.send(encodeJsonFrame({ t: 'tcp_close', s: streamId })); } catch { /* ignore */ }
}
}
}
@@ -337,6 +375,7 @@ export class PilotAgent {
case 'ws_msg_text': this.onWsMsgText(frame.s, frame.data); break;
case 'ws_close': this.onWsClose(frame.s, frame.code, frame.reason); break;
case 'tcp_open': this.onTcpOpen(frame); break;
case 'tcp_open_ack': this.onTcpOpenAckReverse(frame); break;
case 'tcp_close': this.onTcpClose(frame.s); break;
default:
// Other frame types are primary-bound only; agent ignores.
@@ -361,6 +400,13 @@ export class PilotAgent {
break;
}
case BinaryFrameType.TcpData: {
if (frame.streamId >= AGENT_REVERSE_ID_BASE) {
const reverse = this.reverseTcpStreams.get(frame.streamId);
if (!reverse) return;
reverse._dispatchData(frame.payload);
this.refreshIdleTimer(frame.streamId);
return;
}
const stream = this.tcpStreams.get(frame.streamId);
if (!stream) return;
try { stream.socket.write(frame.payload); } catch { /* ignore */ }
@@ -591,6 +637,17 @@ export class PilotAgent {
}
private onTcpClose(streamId: number): void {
// Reverse stream (agent-initiated): primary is closing the
// upstream half. Tear down the local socket the agent's
// MeshForwarder handed us via openMeshTcpStream.
if (streamId >= AGENT_REVERSE_ID_BASE) {
const handle = this.reverseTcpStreams.get(streamId);
if (!handle) return;
this.reverseTcpStreams.delete(streamId);
this.clearIdleTimer(streamId);
handle._dispatchClose();
return;
}
const entry = this.tcpStreams.get(streamId);
if (!entry) return;
this.tcpStreams.delete(streamId);
@@ -598,6 +655,74 @@ export class PilotAgent {
try { entry.socket.destroy(); } catch { /* ignore */ }
}
/**
* Inbound `tcp_open_ack` for an agent-initiated reverse stream. The
* primary acknowledges (or rejects) the dial; emit 'open' or 'error' on
* the local handle so MeshService's splice setup can either start
* piping bytes or tear down the source socket.
*/
private onTcpOpenAckReverse(frame: Extract<ReturnType<typeof decodeJsonFrame>, { t: 'tcp_open_ack' }>): void {
if (frame.s < AGENT_REVERSE_ID_BASE) return; // forward-direction acks are primary-bound; ignore.
const handle = this.reverseTcpStreams.get(frame.s);
if (!handle) return;
if (frame.ok) {
handle._dispatchOpen();
this.refreshIdleTimer(frame.s);
} else {
this.reverseTcpStreams.delete(frame.s);
this.clearIdleTimer(frame.s);
handle._dispatchError(new Error(frame.err ?? 'tcp_open_reverse rejected'));
handle._dispatchClose();
}
}
/**
* Public entry point for the agent's mesh forwarder. Allocates a
* reverse stream id, sends `tcp_open_reverse`, and returns a handle
* MeshService can splice bytes through. Returns null if the tunnel is
* not currently open or the per-tunnel stream cap is reached.
*/
public openMeshTcpStream(target: { nodeId: number; stack: string; service: string; port: number }): ReverseTcpStreamHandle | null {
const ws = this.ws;
if (!ws || ws.readyState !== WebSocket.OPEN) return null;
if (this.streamCount() >= MAX_STREAMS_PER_TUNNEL) return null;
const streamId = this.reverseStreamIds.allocate();
const handle = new ReverseTcpStreamHandle(
streamId,
(sid, payload) => {
if (this.ws?.readyState !== WebSocket.OPEN) return;
try { this.ws.send(encodeBinaryFrame(BinaryFrameType.TcpData, sid, payload), { binary: true }); } catch { /* ignore */ }
this.refreshIdleTimer(sid);
},
(sid) => {
if (!this.reverseTcpStreams.has(sid)) return;
this.reverseTcpStreams.delete(sid);
this.clearIdleTimer(sid);
if (this.ws?.readyState !== WebSocket.OPEN) return;
try { this.ws.send(encodeJsonFrame({ t: 'tcp_close', s: sid })); } catch { /* ignore */ }
},
);
this.reverseTcpStreams.set(streamId, handle);
this.refreshIdleTimer(streamId);
try {
ws.send(encodeJsonFrame({
t: 'tcp_open_reverse',
s: streamId,
targetNodeId: target.nodeId,
stack: target.stack,
service: target.service,
port: target.port,
}));
} catch (err) {
this.reverseTcpStreams.delete(streamId);
this.clearIdleTimer(streamId);
handle._dispatchError(err as Error);
handle._dispatchClose();
return null;
}
return handle;
}
/**
* Resolves a mesh target by consulting the local mesh_stacks opt-in table
* and Compose container labels. Refuses if the target stack is not opted
@@ -645,6 +770,63 @@ interface MeshTcpStream {
accepted: boolean;
}
/**
* Handle returned by `PilotAgent.openMeshTcpStream` to MeshService. Mirrors
* the surface of `PilotTunnelBridge.TcpStream` (write/end/destroy +
* 'open'/'data'/'error'/'close' events) so MeshService.openCrossNode can
* splice bytes against it without caring whether it's running on central
* or on a pilot. The agent owns the per-stream WS plumbing through the
* `sendData` and `sendClose` callbacks; this class is a thin EventEmitter
* facade.
*/
export class ReverseTcpStreamHandle extends EventEmitter {
public readonly streamId: number;
private readonly sendData: (streamId: number, payload: Buffer) => void;
private readonly sendClose: (streamId: number) => void;
private closed = false;
constructor(
streamId: number,
sendData: (streamId: number, payload: Buffer) => void,
sendClose: (streamId: number) => void,
) {
super();
this.streamId = streamId;
this.sendData = sendData;
this.sendClose = sendClose;
}
public write(chunk: Buffer): boolean {
if (this.closed) return false;
this.sendData(this.streamId, chunk);
return true;
}
public end(): void {
if (this.closed) return;
this.closed = true;
this.sendClose(this.streamId);
}
public destroy(): void { this.end(); }
/** @internal Called by PilotAgent on inbound `tcp_open_ack { ok: true }`. */
public _dispatchOpen(): void { this.emit('open'); }
/** @internal Called by PilotAgent on inbound `TcpData` for this stream. */
public _dispatchData(chunk: Buffer): void { this.emit('data', chunk); }
/** @internal Called by PilotAgent on tunnel-side error or rejection. */
public _dispatchError(err: Error): void { this.emit('error', err); }
/** @internal Called by PilotAgent on tunnel-side close. */
public _dispatchClose(): void {
if (this.closed) return;
this.closed = true;
this.emit('close');
}
}
type MeshResolveResult =
| { ok: true; host: string; port: number }
| { ok: false; err: MeshErrCode };
backend/src/pilot/protocol.ts
+49 -10
View File
@@ -75,6 +75,7 @@ export type JsonFrame =
| WsCloseFrame
| ControlFrame
| TcpOpenFrame
| TcpOpenReverseFrame
| TcpOpenAckFrame
| TcpCloseFrame;
@@ -157,9 +158,25 @@ export interface ControlFrame {
}
/**
* Sencho Mesh TCP frames. The primary asks the agent to open a TCP connection
* to a Compose service on the agent's local Docker host. Bytes flow as
* BinaryFrameType.TcpData. Mid-stream failures send tcp_close.
* Sencho Mesh TCP frames. Two directions:
*
* - `tcp_open` (primary -> agent): primary asks the agent to open a TCP
* connection to a Compose service on the agent's local Docker host.
* Stream id is allocated by the primary's `StreamIdAllocator` (low
* range, starts at 1).
*
* - `tcp_open_reverse` (agent -> primary): the agent's mesh forwarder
* accepted a connection destined for another node and asks the primary
* to dial that node. If `targetNodeId` matches the primary's own node,
* the primary dials a local container directly. Otherwise the primary
* relays via its bridge to the target pilot. Stream id is allocated by
* the agent and uses the upper half of the 32-bit space
* (>= 0x40000001) so it cannot collide with primary-allocated ids on
* the same tunnel; the wrap distance (~2^30 vs the 1024 stream cap)
* makes collisions statistically unreachable.
*
* In both directions, `tcp_open_ack` confirms acceptance, bytes flow as
* `BinaryFrameType.TcpData`, and `tcp_close` ends the stream.
*/
export interface TcpOpenFrame {
t: 'tcp_open';
@@ -169,6 +186,15 @@ export interface TcpOpenFrame {
port: number;
}
export interface TcpOpenReverseFrame {
t: 'tcp_open_reverse';
s: number;
targetNodeId: number;
stack: string;
service: string;
port: number;
}
export type MeshErrCode = 'mesh_not_enabled' | 'denied' | 'no_target' | 'unreachable' | 'agent_error';
export interface TcpOpenAckFrame {
@@ -183,6 +209,14 @@ export interface TcpCloseFrame {
s: number;
}
/**
* Stream id space split for `tcp_open` (primary-allocated) vs
* `tcp_open_reverse` (agent-allocated). Used by the agent's reverse
* allocator and by the primary's `tcp_open_reverse` handler to verify
* incoming ids are in the agent half.
*/
export const AGENT_REVERSE_ID_BASE = 0x40000001;
// --- Serialize / parse ---
export function encodeJsonFrame(frame: JsonFrame): string {
@@ -271,24 +305,29 @@ export const PilotCloseCode = {
// --- Stream id allocation ---
/**
* Monotonic stream id generator. Primary is the sole allocator.
* Monotonic stream id generator.
*
* Wraps at 2^31. With MAX_STREAMS_PER_TUNNEL = 1024 the allocator
* cannot collide with a still-live stream during a single tunnel
* lifetime: the wrap distance (~2.1 billion) is more than six orders
* of magnitude larger than the cap. A new tunnel restarts the
* sequence at 1, so cross-tunnel reuse is also harmless.
* Wraps at 2^31 back to the configured `start` value (not back to `1`),
* so allocators initialized with `AGENT_REVERSE_ID_BASE` stay in the
* agent half of the id space across the wrap. With
* MAX_STREAMS_PER_TUNNEL = 1024 the allocator cannot collide with a
* still-live stream during a single tunnel lifetime: the wrap distance
* (~2.1 billion) is more than six orders of magnitude larger than the
* cap. A new tunnel restarts the sequence at `start`, so cross-tunnel
* reuse is also harmless.
*/
export class StreamIdAllocator {
private readonly start: number;
private next: number;
constructor(start = 1) {
this.start = start;
this.next = start;
}
allocate(): number {
const id = this.next;
this.next = this.next >= 0x7fffffff ? 1 : this.next + 1;
this.next = this.next >= 0x7fffffff ? this.start : this.next + 1;
return id;
}
}