mirror of
https://github.com/Studio-Saelix/sencho.git
synced 2026-08-27 10:46:51 +00:00
feat(mesh): route mesh traffic over Distributed API remotes (#1048)
* refactor(mesh): extract shared TCP stream switchboard
Pull the `tcp_open` / `tcp_open_ack` / `tcp_open_reverse` / `tcp_close`
+ `TcpData` handling out of the pilot agent into a reusable module so a
second caller (the upcoming proxy-mode WS handler) can run the same
frame parser, stream allocator, idle timers, and Compose-label
resolver. One parser, two callers, zero drift on a
security-sensitive protocol surface.
The pilot agent keeps its public `openMeshTcpStream` API and delegates
to a per-connection switchboard constructed in `connect()` and torn
down in `cleanupAfterDisconnect`. Existing reverse-stream and resolver
tests are rewritten to exercise the shared module directly.
* feat(mesh): route mesh traffic over Distributed API remotes
Sencho Mesh now works against remotes in Distributed API mode in
addition to Pilot Agent mode. Central opens a short-lived WebSocket
tunnel to the remote's new `/api/mesh/proxy-tunnel` endpoint on demand
and tears it down after 5 minutes of idle (configurable via
`SENCHO_MESH_PROXY_TUNNEL_IDLE_MS`). Mesh dispatch is mode-agnostic at
the routing layer; `PilotTunnelManager.ensureBridge` resolves an
existing tunnel or asks the new `MeshProxyTunnelDialer` to dial.
The Routing tab badges now use a `reachableMode` classifier: `★ Local`
for local, `pilot offline` red badge for a pilot with a down tunnel,
and a new `unreachable` red badge surfacing the specific reason
(missing token, scope not full-admin, TLS failure, remote does not
support proxy mesh). Distributed API remotes with valid credentials
show no negative badge; the tunnel opens on first dial.
Auth: the proxy-tunnel WS upgrade requires an `Authorization: Bearer`
API token with the `full-admin` scope. Lower-scoped tokens are
rejected at upgrade time so a leaked read-only token cannot reach the
mesh data plane.
Bidirectional: the proxy-mode WS handler registers itself as the
local `MeshService` reverse dialer (compare-and-swap), so meshed
containers on a Distributed API remote can dial cross-node aliases
via `tcp_open_reverse` over the same tunnel. Cross-node relays
(`PilotTunnelBridge.acceptReverseRelay`) await `ensureBridge` so
proxy-to-proxy mesh works without standing tunnels.
* docs(mesh): describe Distributed API mesh and unreachable troubleshooting
Refresh the user-facing mesh documentation to reflect that mesh works
over both Pilot Agent and Distributed API remotes. Adds a
Troubleshooting accordion entry for the new `unreachable` Routing tab
badge so operators can match a tooltip reason ("api token rejected
(scope must be full-admin)", "remote does not support proxy mesh",
"TLS handshake failed", "api_url not set", "api token missing") to a
concrete fix.
* refactor(mesh): apply /simplify review findings
Five cleanups surfaced by the post-implementation code review pass.
No behaviour change for fleets running on `main`; only internal
structure improves.
- **Deterministic container IP shared.** Extract the compose-default
network preference (`pickContainerIp`) and the conventional-name
fast path (`lookupContainerIp`) into `backend/src/mesh/containerLookup.ts`.
Both `MeshService.resolveContainerIp` (existing same-node fast
path) and the switchboard's `resolveByComposeLabels` (proxy/pilot
inbound dial) now use the same logic. Earlier the switchboard
helper grabbed the first `Object.values(Networks)` IP, which
could flip across daemon versions on multi-network containers;
fixed.
- **WS URL upgrade extracted.** New `backend/src/utils/wsUrl.ts`
exposes `httpUrlToWs(baseUrl)` that maps `http://` to `ws://` and
`https://` to `wss://`. Used in both `pilot/agent.ts` and the
proxy-tunnel dialer. The previous inline `replace(/^http/, 'ws')`
silently downgraded `https://` to `ws://` (cleartext).
- **`computeReachable` takes the pre-fetched node.** `getStatus`
already iterated `db.getNodes()`; the helper previously re-queried
by id per node (N+1 reads). Pass the row in.
- **Reachable-reason ternary -> const map.** Replace the nested
ternary in `MeshService.computeReachable` with a
`Record<DialFailureCode, string>` lookup keyed on the dialer's
failure code.
- **Activity-type ternary -> const map.** Same flattening inside
`MeshProxyTunnelDialer.logActivity`.
All mesh tests pass (85/85). Full backend suite green (2126/2126).
* fix(mesh): cache proxy dial failures and contain WS handshake errors
`ensureBridge` now consults the recent-failure cache before dialing so a
continuous mesh workload against a misconfigured proxy-mode remote does
not produce one upgrade attempt per cross-node TCP open. `recordFailure`
emits at most one activity-log entry per cache window per (nodeId, code)
so a connect-loop on a single bad remote cannot flush the ring buffer.
Failure messages run through `redactSensitiveText` before reaching the
log so any embedded Bearer / JWT / inline-URL credentials are scrubbed.
`stop()` clears the inflight map and `dial()` checks `this.stopped`
between awaits so a shutdown does not leak a half-opened bridge.
When `awaitOpen` rejects (401, 4403, TLS), the dialer attaches a noop
'error' listener before calling `ws.close()`. Without it the ws library
emits a tail 'error' on a still-CONNECTING socket that propagates as an
unhandled exception. Surfaced by a live-network test against a real
proxy-mode peer.
Adds `mesh-proxy-tunnel-live.test.ts` (skipped unless MESH_AUDIT_URL
and MESH_AUDIT_TOKEN_FILE env vars are set) covering both the happy
path and the auth-rejected path against an actual remote Sencho.
* feat(mesh): instrument proxy tunnel observability
Adds three counters to `PilotMetrics`:
- `proxy_bridges_total` (incremented on `registerProxyBridge` success)
- `proxy_dials_failed` (every failed dial attempt, not deduped)
- `proxy_idle_closes` (idle-sweep teardowns)
All three surface automatically via `GET /api/system/pilot-tunnels`
since the route returns the full `Counters` snapshot.
Gates two pre-existing always-on `console.warn` calls in
`tcpStreamSwitchboard.ts` (mid-stream socket errors and Docker resolve
failures) behind `isDebugEnabled()`. Both fire on per-stream events and
would otherwise violate the diagnostic-log safety rule under load.
Adds `mesh-tcp-stream-switchboard.test.ts` covering the forward
`tcp_open` path: real localhost dial, resolver errors (no_target,
denied), per-tunnel cap saturation, frame-routing fall-through
invariants, `tcp_close` socket teardown.
Drops `MESH_CONNECT_TIMEOUT_MS` from the public surface; only the
switchboard itself uses it.
* fix(mesh): tighten Routing tab unreachable handling
`RoutingNodeCard.tsx`: the **Add stack to mesh** button now disables
when `reachableMode === 'unreachable'`, matching the existing
TogglePill behavior. Without this, an operator on an unreachable node
could open the opt-in sheet, confirm, and watch the redeploy proceed
against a target whose mesh data plane will silently fail to route.
Also drops the leftover `status.nodeId !== -1` guard on the pilot-
offline badge.
`MeshService.ts`: renames `isMeshReachable` to `isMeshConfigured` with
the new predicate that returns true for proxy-mode remotes whose creds
are valid (the tunnel is opened on demand). `getRouteDiagnostic` now
distinguishes "routable" (configured) from "pilotLive" (live tunnel
state, only meaningful for pilot mode); without the split, every idle
proxy-mode route would report `tunnel down`.
`MeshService.setReverseDialer` warns when the unconditional install
path silently overwrites a non-null current dialer. By topology a
Sencho is either pilot or central, so the branch flags a misconfigured
deployment rather than an expected race.
Drops the dead `export { ReverseTcpStreamHandle }` re-export from
`pilot/agent.ts`; its only consumer imports straight from
`mesh/tcpStreamSwitchboard.ts`. Fixes a stale doc comment in
`MeshService.ts` that referenced the wrong source file.
Adds `mesh-proxy-tunnel-handler.test.ts` covering the WS handler
lifecycle: pilot-mode 404 rejection, post-upgrade reverse-dialer
install, concurrent-upgrade 1013 rejection (single-tenant slot), and
error-path teardown.
Updates the troubleshooting accordion in the user docs to mention the
disabled-state behavior.
* fix(mesh): close ESLint and CodeQL findings on the proxy-tunnel diag logs
Remove the unused `reverseDialer` local in `meshProxyTunnel.ts`; the
closure target is `localDialer` above and this assignment was always
dead. Strip line breaks inline on the three `MeshProxyTunnelDialer`
diag log lines so CodeQL `js/log-injection` data flow recognises the
sanitisation that `sanitizeForLog` already performs.
No behaviour change; the diag logs render the same characters they
do today.
This commit is contained in:
@@ -0,0 +1,371 @@
|
||||
import { EventEmitter } from 'events';
|
||||
import WebSocket from 'ws';
|
||||
import { MAX_FRAME_SIZE_BYTES } from '../pilot/protocol';
|
||||
import { PilotTunnelBridge, type MeshTunnelHandle } from './PilotTunnelBridge';
|
||||
import { PilotTunnelManager } from './PilotTunnelManager';
|
||||
import { NodeRegistry } from './NodeRegistry';
|
||||
import { redactSensitiveText, sanitizeForLog } from '../utils/safeLog';
|
||||
import { httpUrlToWs } from '../utils/wsUrl';
|
||||
import { isDebugEnabled } from '../utils/debug';
|
||||
import { PilotMetrics } from './PilotMetrics';
|
||||
import type { MeshActivityType } from './MeshService';
|
||||
|
||||
/**
|
||||
* Central-side dialer for proxy-mode mesh tunnels.
|
||||
*
|
||||
* Today's pilot tunnels are agent-initiated: the pilot dials central and
|
||||
* the resulting WS is registered in `PilotTunnelManager`. Proxy-mode
|
||||
* remotes do not dial; central reaches them via the existing HTTP proxy
|
||||
* using the long-lived `api_token`. To carry streaming TCP mesh traffic to
|
||||
* a proxy-mode remote, central opens a WebSocket to the remote's new
|
||||
* `/api/mesh/proxy-tunnel` endpoint on demand and registers the resulting
|
||||
* `PilotTunnelBridge` in the manager under the same nodeId. The rest of
|
||||
* the mesh code path (alias dispatch, openTcpStream, reverse-stream relay)
|
||||
* is mode-agnostic from there.
|
||||
*
|
||||
* Lifecycle:
|
||||
* - `ensureBridge(nodeId)` dials if not connected; concurrent callers
|
||||
* dedupe through an in-flight promise map.
|
||||
* - A periodic check tears down bridges that have seen no active streams
|
||||
* for `SENCHO_MESH_PROXY_TUNNEL_IDLE_MS` (default 5 minutes). Setting
|
||||
* the env var to `0` disables idle close (tunnel persists until the
|
||||
* remote drops it or central shuts down).
|
||||
* - Recent failures are cached for 60 seconds so a misconfigured remote
|
||||
* does not cause a continuous redial storm; `MeshService.getStatus`
|
||||
* consults the cache to surface a `reachableReason` to the UI.
|
||||
*/
|
||||
const DEFAULT_IDLE_TTL_MS = 5 * 60 * 1000;
|
||||
const IDLE_CHECK_INTERVAL_MS = 60 * 1000;
|
||||
const HANDSHAKE_TIMEOUT_MS = 15_000;
|
||||
const FAILURE_CACHE_TTL_MS = 60 * 1000;
|
||||
|
||||
export type DialFailureCode =
|
||||
| 'no_target'
|
||||
| 'endpoint_not_found'
|
||||
| 'auth_failed'
|
||||
| 'tls_failed'
|
||||
| 'network_error';
|
||||
|
||||
/**
|
||||
* Activity-log reason. Wider than `DialFailureCode` because some failures
|
||||
* share a wire-level code (e.g., `network_error`) but deserve a more
|
||||
* specific label in the operator-facing log to distinguish a network-
|
||||
* layer failure from a post-handshake bridge failure or a manager
|
||||
* rejection.
|
||||
*/
|
||||
type DialFailureReason = DialFailureCode | 'bridge_start_failed' | 'manager_rejected';
|
||||
|
||||
type ProxyTunnelEvent = 'open.ok' | 'open.fail' | 'close';
|
||||
|
||||
const ACTIVITY_TYPE: Record<ProxyTunnelEvent, MeshActivityType> = {
|
||||
'open.ok': 'proxy-tunnel.open.ok',
|
||||
'open.fail': 'proxy-tunnel.open.fail',
|
||||
'close': 'proxy-tunnel.close',
|
||||
};
|
||||
|
||||
export interface DialFailure {
|
||||
code: DialFailureCode;
|
||||
message?: string;
|
||||
ts: number;
|
||||
}
|
||||
|
||||
export class MeshProxyTunnelDialer extends EventEmitter {
|
||||
private static instance: MeshProxyTunnelDialer | null = null;
|
||||
|
||||
private readonly bridges = new Map<number, PilotTunnelBridge>();
|
||||
private readonly inflight = new Map<number, Promise<MeshTunnelHandle | null>>();
|
||||
private readonly idleSince = new Map<number, number>();
|
||||
private readonly recentFailures = new Map<number, DialFailure>();
|
||||
private readonly idleTtlMs: number;
|
||||
private idleCheckTimer: NodeJS.Timeout | null = null;
|
||||
private stopped = false;
|
||||
|
||||
private constructor(idleTtlOverrideMs?: number) {
|
||||
super();
|
||||
if (typeof idleTtlOverrideMs === 'number') {
|
||||
this.idleTtlMs = idleTtlOverrideMs;
|
||||
} else {
|
||||
const raw = process.env.SENCHO_MESH_PROXY_TUNNEL_IDLE_MS;
|
||||
const parsed = raw === undefined ? Number.NaN : Number(raw);
|
||||
this.idleTtlMs = Number.isFinite(parsed) && parsed >= 0 ? parsed : DEFAULT_IDLE_TTL_MS;
|
||||
}
|
||||
this.startIdleCheck();
|
||||
}
|
||||
|
||||
public static getInstance(): MeshProxyTunnelDialer {
|
||||
if (!this.instance) this.instance = new MeshProxyTunnelDialer();
|
||||
return this.instance;
|
||||
}
|
||||
|
||||
/** Test hook: reset the singleton with an optional idle-TTL override. */
|
||||
public static resetForTest(idleTtlOverrideMs?: number): MeshProxyTunnelDialer {
|
||||
if (this.instance) this.instance.stop();
|
||||
this.instance = new MeshProxyTunnelDialer(idleTtlOverrideMs);
|
||||
return this.instance;
|
||||
}
|
||||
|
||||
public hasBridge(nodeId: number): boolean {
|
||||
return this.bridges.has(nodeId);
|
||||
}
|
||||
|
||||
public getBridge(nodeId: number): MeshTunnelHandle | null {
|
||||
return this.bridges.get(nodeId) ?? null;
|
||||
}
|
||||
|
||||
public getRecentFailure(nodeId: number): DialFailure | null {
|
||||
const entry = this.recentFailures.get(nodeId);
|
||||
if (!entry) return null;
|
||||
if (Date.now() - entry.ts > FAILURE_CACHE_TTL_MS) {
|
||||
this.recentFailures.delete(nodeId);
|
||||
return null;
|
||||
}
|
||||
return entry;
|
||||
}
|
||||
|
||||
/**
|
||||
* Dial-if-needed. Concurrent callers dedupe through `inflight`; a
|
||||
* cached recent failure short-circuits the dial so a misconfigured
|
||||
* remote does not see one upgrade attempt per cross-node TCP open.
|
||||
*/
|
||||
public async ensureBridge(nodeId: number): Promise<MeshTunnelHandle | null> {
|
||||
const existing = this.bridges.get(nodeId);
|
||||
if (existing) {
|
||||
this.idleSince.set(nodeId, Date.now());
|
||||
return existing;
|
||||
}
|
||||
if (this.getRecentFailure(nodeId)) return null;
|
||||
const inflight = this.inflight.get(nodeId);
|
||||
if (inflight) return inflight;
|
||||
const dial = this.dial(nodeId).finally(() => {
|
||||
this.inflight.delete(nodeId);
|
||||
});
|
||||
this.inflight.set(nodeId, dial);
|
||||
return dial;
|
||||
}
|
||||
|
||||
/** Force-close a bridge (e.g., on node deletion or scope change). */
|
||||
public closeBridge(nodeId: number, reason = 'closed by dialer'): void {
|
||||
const bridge = this.bridges.get(nodeId);
|
||||
if (!bridge) return;
|
||||
this.bridges.delete(nodeId);
|
||||
this.idleSince.delete(nodeId);
|
||||
try { bridge.close(1000, reason); } catch { /* ignore */ }
|
||||
}
|
||||
|
||||
/** Stop the idle-check timer and tear down every open bridge. */
|
||||
public stop(): void {
|
||||
this.stopped = true;
|
||||
if (this.idleCheckTimer) {
|
||||
clearInterval(this.idleCheckTimer);
|
||||
this.idleCheckTimer = null;
|
||||
}
|
||||
for (const [nodeId, bridge] of this.bridges) {
|
||||
this.bridges.delete(nodeId);
|
||||
try { bridge.close(1000, 'dialer shutdown'); } catch { /* ignore */ }
|
||||
}
|
||||
this.idleSince.clear();
|
||||
this.recentFailures.clear();
|
||||
this.inflight.clear();
|
||||
}
|
||||
|
||||
/** Test hook: count active bridges. */
|
||||
public get activeBridgeCount(): number {
|
||||
return this.bridges.size;
|
||||
}
|
||||
|
||||
private async dial(nodeId: number): Promise<MeshTunnelHandle | null> {
|
||||
const target = NodeRegistry.getInstance().getProxyTarget(nodeId);
|
||||
if (!target || !target.apiToken) {
|
||||
this.recordFailure(nodeId, 'no_target', 'no proxy target configured');
|
||||
return null;
|
||||
}
|
||||
|
||||
if (this.stopped) return null;
|
||||
const dialStartedAt = Date.now();
|
||||
if (isDebugEnabled()) {
|
||||
console.log(`[MeshProxyDialer:diag] dialing node=${nodeId} url=${sanitizeForLog(target.apiUrl)}`.replace(/[\n\r]/g, ''));
|
||||
}
|
||||
|
||||
const wsUrl = httpUrlToWs(target.apiUrl) + '/api/mesh/proxy-tunnel';
|
||||
let ws: WebSocket;
|
||||
try {
|
||||
ws = new WebSocket(wsUrl, {
|
||||
headers: { Authorization: `Bearer ${target.apiToken}` },
|
||||
handshakeTimeout: HANDSHAKE_TIMEOUT_MS,
|
||||
maxPayload: MAX_FRAME_SIZE_BYTES,
|
||||
});
|
||||
} catch (err) {
|
||||
this.recordFailure(nodeId, 'network_error', (err as Error).message);
|
||||
return null;
|
||||
}
|
||||
|
||||
try {
|
||||
await this.awaitOpen(ws);
|
||||
} catch (err) {
|
||||
// `awaitOpen` removes the 'error' listener on reject; calling
|
||||
// `close()` on a still-CONNECTING socket emits a tail 'error'
|
||||
// ('WebSocket was closed before the connection was established')
|
||||
// that would otherwise propagate as an unhandled exception.
|
||||
ws.on('error', () => { /* swallow tail error */ });
|
||||
try { ws.close(); } catch { /* ignore */ }
|
||||
const failure = classifyDialError(err);
|
||||
this.recordFailure(nodeId, failure.code, failure.message);
|
||||
return null;
|
||||
}
|
||||
|
||||
if (this.stopped) {
|
||||
try { ws.close(1001, 'dialer shutdown'); } catch { /* ignore */ }
|
||||
return null;
|
||||
}
|
||||
|
||||
const bridge = new PilotTunnelBridge(nodeId, ws);
|
||||
try {
|
||||
await bridge.start();
|
||||
} catch (err) {
|
||||
this.recordFailure(nodeId, 'network_error', (err as Error).message, 'bridge_start_failed');
|
||||
try { bridge.close(1011, 'bridge start failed'); } catch { /* ignore */ }
|
||||
return null;
|
||||
}
|
||||
|
||||
if (this.stopped) {
|
||||
try { bridge.close(1001, 'dialer shutdown'); } catch { /* ignore */ }
|
||||
return null;
|
||||
}
|
||||
|
||||
try {
|
||||
PilotTunnelManager.getInstance().registerProxyBridge(nodeId, bridge);
|
||||
} catch (err) {
|
||||
// Cap hit or a pilot tunnel concurrently claimed this nodeId.
|
||||
this.recordFailure(nodeId, 'network_error', (err as Error).message, 'manager_rejected');
|
||||
try { bridge.close(1013, 'manager rejected'); } catch { /* ignore */ }
|
||||
return null;
|
||||
}
|
||||
|
||||
// Mirror the registration in the dialer's own map so the idle
|
||||
// sweeper can call `getActiveStreamCount()` (not on the narrow
|
||||
// MeshTunnelHandle interface) without poking into the manager.
|
||||
bridge.once('closed', () => {
|
||||
if (this.bridges.get(nodeId) === bridge) {
|
||||
this.bridges.delete(nodeId);
|
||||
this.idleSince.delete(nodeId);
|
||||
void this.logActivity(nodeId, 'close', { reason: 'remote-closed' });
|
||||
this.emit('proxy-bridge-down', nodeId);
|
||||
}
|
||||
});
|
||||
this.bridges.set(nodeId, bridge);
|
||||
this.idleSince.set(nodeId, Date.now());
|
||||
this.recentFailures.delete(nodeId);
|
||||
void this.logActivity(nodeId, 'open.ok', {});
|
||||
this.emit('proxy-bridge-up', nodeId);
|
||||
if (isDebugEnabled()) {
|
||||
console.log(`[MeshProxyDialer:diag] dial ok node=${nodeId} elapsedMs=${Date.now() - dialStartedAt}`.replace(/[\n\r]/g, ''));
|
||||
}
|
||||
return bridge;
|
||||
}
|
||||
|
||||
private awaitOpen(ws: WebSocket): Promise<void> {
|
||||
return new Promise<void>((resolve, reject) => {
|
||||
const cleanup = () => {
|
||||
ws.removeAllListeners('open');
|
||||
ws.removeAllListeners('error');
|
||||
ws.removeAllListeners('unexpected-response');
|
||||
};
|
||||
ws.once('open', () => { cleanup(); resolve(); });
|
||||
ws.once('error', (err) => { cleanup(); reject(err); });
|
||||
ws.once('unexpected-response', (_req, res) => {
|
||||
cleanup();
|
||||
const err = new Error(`upgrade failed: HTTP ${res.statusCode}`) as Error & { httpStatus?: number };
|
||||
err.httpStatus = res.statusCode ?? 0;
|
||||
try { res.resume(); } catch { /* ignore */ }
|
||||
reject(err);
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Cache a dial failure and emit at most one activity-log entry per
|
||||
* cache window per `(nodeId, code)`. The dedupe bounds operator log
|
||||
* noise when a meshed container retries cross-node TCP opens against
|
||||
* a misconfigured remote.
|
||||
*/
|
||||
private recordFailure(nodeId: number, code: DialFailureCode, rawMessage?: string, reasonOverride?: DialFailureReason): void {
|
||||
const message = rawMessage ? sanitizeForLog(redactSensitiveText(rawMessage)) : undefined;
|
||||
const previous = this.recentFailures.get(nodeId);
|
||||
const reason = reasonOverride ?? code;
|
||||
const isFresh = !previous
|
||||
|| Date.now() - previous.ts > FAILURE_CACHE_TTL_MS
|
||||
|| previous.code !== code;
|
||||
this.recentFailures.set(nodeId, { code, message, ts: Date.now() });
|
||||
PilotMetrics.increment('proxy_dials_failed');
|
||||
if (isFresh) {
|
||||
void this.logActivity(nodeId, 'open.fail', message ? { reason, message } : { reason });
|
||||
}
|
||||
if (isDebugEnabled()) {
|
||||
console.warn(`[MeshProxyDialer:diag] dial failure node=${nodeId} code=${code} reason=${reason}${message ? ` message=${message}` : ''}`.replace(/[\n\r]/g, ''));
|
||||
}
|
||||
}
|
||||
|
||||
private startIdleCheck(): void {
|
||||
if (this.idleCheckTimer || this.stopped) return;
|
||||
if (this.idleTtlMs <= 0) return; // 0 disables idle close
|
||||
this.idleCheckTimer = setInterval(() => this.runIdleCheck(), IDLE_CHECK_INTERVAL_MS);
|
||||
this.idleCheckTimer.unref?.();
|
||||
}
|
||||
|
||||
private runIdleCheck(): void {
|
||||
const now = Date.now();
|
||||
for (const [nodeId, bridge] of this.bridges) {
|
||||
if (bridge.getActiveStreamCount() > 0) {
|
||||
this.idleSince.set(nodeId, now);
|
||||
continue;
|
||||
}
|
||||
const last = this.idleSince.get(nodeId) ?? now;
|
||||
if (now - last >= this.idleTtlMs) {
|
||||
this.bridges.delete(nodeId);
|
||||
this.idleSince.delete(nodeId);
|
||||
try { bridge.close(1000, 'idle timeout'); } catch { /* ignore */ }
|
||||
PilotMetrics.increment('proxy_idle_closes');
|
||||
void this.logActivity(nodeId, 'close', { reason: 'idle' });
|
||||
this.emit('proxy-bridge-down', nodeId);
|
||||
if (isDebugEnabled()) {
|
||||
console.log(`[MeshProxyDialer:diag] idle close node=${nodeId} idleMs=${now - last}`);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private async logActivity(
|
||||
nodeId: number,
|
||||
event: ProxyTunnelEvent,
|
||||
details: Record<string, unknown>,
|
||||
): Promise<void> {
|
||||
try {
|
||||
// Lazy import to avoid a circular dependency with MeshService.
|
||||
const { MeshService } = await import('./MeshService');
|
||||
const message = typeof details.message === 'string'
|
||||
? details.message
|
||||
: `proxy tunnel ${event} for node ${nodeId}`;
|
||||
MeshService.getInstance().logActivity({
|
||||
source: 'mesh',
|
||||
level: event === 'open.fail' ? 'error' : 'info',
|
||||
type: ACTIVITY_TYPE[event],
|
||||
nodeId,
|
||||
message,
|
||||
details,
|
||||
});
|
||||
} catch {
|
||||
// Activity logging is best-effort; never let it propagate.
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
function classifyDialError(err: unknown): { code: DialFailureCode; message: string } {
|
||||
const message = sanitizeForLog((err as Error).message || String(err));
|
||||
const httpStatus = (err as Error & { httpStatus?: number }).httpStatus;
|
||||
if (httpStatus === 404) return { code: 'endpoint_not_found', message: 'remote does not expose /api/mesh/proxy-tunnel' };
|
||||
if (httpStatus === 401 || httpStatus === 403) return { code: 'auth_failed', message: 'api token rejected by remote' };
|
||||
const tlsCodes = new Set(['CERT_HAS_EXPIRED', 'UNABLE_TO_VERIFY_LEAF_SIGNATURE', 'DEPTH_ZERO_SELF_SIGNED_CERT', 'SELF_SIGNED_CERT_IN_CHAIN']);
|
||||
const errno = (err as NodeJS.ErrnoException).code;
|
||||
if (errno && tlsCodes.has(errno)) return { code: 'tls_failed', message };
|
||||
return { code: 'network_error', message };
|
||||
}
|
||||
@@ -12,7 +12,9 @@ import { PROXY_TIER_HEADER, PROXY_VARIANT_HEADER } from './license-headers';
|
||||
import { MeshForwarder, type MeshForwarderHost } from './MeshForwarder';
|
||||
import { NodeRegistry } from './NodeRegistry';
|
||||
import { PilotTunnelManager } from './PilotTunnelManager';
|
||||
import { MeshProxyTunnelDialer, type DialFailureCode } from './MeshProxyTunnelDialer';
|
||||
import { generateOverrideYaml, MeshAlias, SENCHO_MESH_NETWORK } from './MeshComposeOverride';
|
||||
import { lookupContainerIp } from '../mesh/containerLookup';
|
||||
import { sanitizeForLog } from '../utils/safeLog';
|
||||
import { isPathWithinBase, isValidStackName } from '../utils/validation';
|
||||
import { PORT as SENCHO_LISTEN_PORT } from '../helpers/constants';
|
||||
@@ -24,6 +26,14 @@ const PROBE_TIMEOUT_MS = 5_000;
|
||||
const SLOW_PROBE_THRESHOLD_MS = 500;
|
||||
const DEFAULT_MESH_SUBNET = '172.30.0.0/24';
|
||||
|
||||
const REACHABLE_REASON: Record<DialFailureCode, string> = {
|
||||
auth_failed: 'api token rejected (scope must be full-admin)',
|
||||
endpoint_not_found: 'remote does not support proxy mesh',
|
||||
tls_failed: 'TLS handshake failed',
|
||||
no_target: 'proxy target missing',
|
||||
network_error: 'remote unreachable',
|
||||
};
|
||||
|
||||
/**
|
||||
* Returns the static IPv4 address Sencho will pin itself to on the mesh
|
||||
* Docker network: `<network address> + 2`. The Docker daemon assigns
|
||||
@@ -60,7 +70,8 @@ export type MeshActivityType =
|
||||
| 'mesh.enable' | 'mesh.disable'
|
||||
| 'mesh.override.preserved'
|
||||
| 'probe.ok' | 'probe.fail'
|
||||
| 'forwarder.listen' | 'forwarder.unlisten' | 'forwarder.error';
|
||||
| 'forwarder.listen' | 'forwarder.unlisten' | 'forwarder.error'
|
||||
| 'proxy-tunnel.open.ok' | 'proxy-tunnel.open.fail' | 'proxy-tunnel.close';
|
||||
|
||||
export interface MeshActivityEvent {
|
||||
ts: number;
|
||||
@@ -105,13 +116,38 @@ export interface MeshRegenSummary {
|
||||
reason?: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* How a node participates in mesh routing right now:
|
||||
* - `local`: the Sencho serving this request.
|
||||
* - `pilot`: a remote with a pilot agent. Live-tunnel state is captured
|
||||
* separately in `pilotConnected`.
|
||||
* - `proxy`: a remote that central reaches via the long-lived api_token.
|
||||
* Mesh tunnels are opened on demand by `MeshProxyTunnelDialer`; the
|
||||
* operator sees no badge while the configuration is sound.
|
||||
* - `unreachable`: configuration or runtime problem keeps mesh traffic
|
||||
* from flowing. `reachableReason` carries an actionable hint.
|
||||
*/
|
||||
export type MeshReachableMode = 'local' | 'pilot' | 'proxy' | 'unreachable';
|
||||
|
||||
export interface MeshNodeStatus {
|
||||
nodeId: number;
|
||||
nodeName: string;
|
||||
enabled: boolean;
|
||||
/** Forwarder state for the LOCAL node (the Sencho instance answering this request). Always `null` for any non-local node — fetching the remote forwarder state requires a cross-node call which lands in Phase B. */
|
||||
localForwarderListening: boolean | null;
|
||||
/**
|
||||
* True iff a pilot tunnel is currently registered for this node. Only
|
||||
* meaningful when `reachableMode === 'pilot'`. Kept for diagnostic
|
||||
* surfaces; the Routing tab badge logic consumes `reachableMode` and
|
||||
* ignores this field for proxy / local nodes.
|
||||
* TODO: collapse into `reachableMode` (introduce `pilot_offline` value)
|
||||
* once no remaining caller reads `pilotConnected` directly.
|
||||
*/
|
||||
pilotConnected: boolean;
|
||||
/** Canonical reachability classification consumed by the Routing tab. */
|
||||
reachableMode: MeshReachableMode;
|
||||
/** Short, operator-facing reason when `reachableMode === 'unreachable'`. Null otherwise. */
|
||||
reachableReason: string | null;
|
||||
optedInStacks: string[];
|
||||
activeStreamCount: number;
|
||||
}
|
||||
@@ -1284,7 +1320,7 @@ export class MeshService extends EventEmitter implements MeshForwarderHost {
|
||||
if (target.nodeId === selfNodeId) {
|
||||
await this.openSameNode(target, src);
|
||||
} else {
|
||||
this.openCrossNode(target, src);
|
||||
await this.openCrossNode(target, src);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1340,28 +1376,9 @@ export class MeshService extends EventEmitter implements MeshForwarderHost {
|
||||
* central.
|
||||
*/
|
||||
public async resolveContainerIp(target: { stack: string; service: string }): Promise<string | null> {
|
||||
const docker = DockerController.getInstance().getDocker() as unknown as Parameters<typeof lookupContainerIp>[0];
|
||||
try {
|
||||
const docker = DockerController.getInstance().getDocker();
|
||||
// Compose default container name pattern; -1 is the first replica.
|
||||
const conventionalName = `${target.stack}-${target.service}-1`;
|
||||
const info = await docker.getContainer(conventionalName).inspect().catch(() => null);
|
||||
const fromInspect = info ? this.extractContainerIp(target.stack, info) : null;
|
||||
if (fromInspect) return fromInspect;
|
||||
// Fallback: filter by compose labels in case of a non-conventional
|
||||
// container name (operator overrode `container_name` or compose
|
||||
// project).
|
||||
const containers = await docker.listContainers({
|
||||
all: true,
|
||||
filters: {
|
||||
label: [
|
||||
`com.docker.compose.project=${target.stack}`,
|
||||
`com.docker.compose.service=${target.service}`,
|
||||
],
|
||||
},
|
||||
});
|
||||
if (containers.length === 0) return null;
|
||||
const fallbackInfo = await docker.getContainer(containers[0].Id).inspect().catch(() => null);
|
||||
return fallbackInfo ? this.extractContainerIp(target.stack, fallbackInfo) : null;
|
||||
return await lookupContainerIp(docker, target.stack, target.service);
|
||||
} catch (err) {
|
||||
console.warn('[MeshService] container IP lookup failed:', sanitizeForLog((err as Error).message));
|
||||
return null;
|
||||
@@ -1369,43 +1386,39 @@ export class MeshService extends EventEmitter implements MeshForwarderHost {
|
||||
}
|
||||
|
||||
/**
|
||||
* Pick a deterministic IP. Prefer the compose default network
|
||||
* (`<stack>_default` or any network whose name starts with `<stack>_`),
|
||||
* then any other declared network, then the legacy bridge `IPAddress`.
|
||||
* Without this preference order, `Object.values(Networks)` ordering on
|
||||
* containers attached to multiple networks varies across daemon
|
||||
* versions and can make same-node forwarding flaky on a redeploy.
|
||||
*/
|
||||
private extractContainerIp(
|
||||
stackName: string,
|
||||
info: { NetworkSettings?: { Networks?: Record<string, { IPAddress?: string }>; IPAddress?: string } },
|
||||
): string | null {
|
||||
const networks = info.NetworkSettings?.Networks ?? {};
|
||||
const composeDefault = networks[`${stackName}_default`];
|
||||
if (composeDefault?.IPAddress) return composeDefault.IPAddress;
|
||||
for (const [name, net] of Object.entries(networks)) {
|
||||
if (name.startsWith(`${stackName}_`) && net?.IPAddress) return net.IPAddress;
|
||||
}
|
||||
for (const net of Object.values(networks)) {
|
||||
if (net?.IPAddress) return net.IPAddress;
|
||||
}
|
||||
return info.NetworkSettings?.IPAddress || null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Pluggable reverse dialer. Set by `PilotAgent` on a pilot host; left
|
||||
* null on a central host. When set, `openCrossNode` routes outbound
|
||||
* mesh dials through the agent's `tcp_open_reverse` path; when unset,
|
||||
* `openCrossNode` uses the central-side `PilotTunnelManager.getBridge`
|
||||
* directly. Lets the same MeshService code work on both sides.
|
||||
* Pluggable reverse dialer. Set by `PilotAgent` on a pilot host or by
|
||||
* the proxy-mode WS handler when a `/api/mesh/proxy-tunnel` upgrade
|
||||
* comes in; left null on a central host with no inbound mesh tunnel.
|
||||
* When set, `openCrossNode` routes outbound mesh dials through the
|
||||
* dialer's `tcp_open_reverse` path; when unset, `openCrossNode` uses
|
||||
* the central-side `PilotTunnelManager.getBridge` directly. Lets the
|
||||
* same MeshService code work on both sides.
|
||||
*/
|
||||
private reverseDialer: ReverseMeshDialer | null = null;
|
||||
|
||||
public setReverseDialer(dialer: ReverseMeshDialer | null): void {
|
||||
/**
|
||||
* Install or clear the reverse dialer. Supports compare-and-swap via
|
||||
* the optional `expected` argument so a caller (e.g., the proxy-mode
|
||||
* WS handler) can install on a null slot and uninstall only if its
|
||||
* own dialer is still the active one. Without the `expected` arg the
|
||||
* operation is unconditional (used by the pilot agent at boot).
|
||||
*
|
||||
* Returns true when the swap happened, false when the CAS rejected
|
||||
* because `expected` did not match the current dialer.
|
||||
*/
|
||||
public setReverseDialer(dialer: ReverseMeshDialer | null, expected?: ReverseMeshDialer | null): boolean {
|
||||
if (expected !== undefined && this.reverseDialer !== expected) return false;
|
||||
// Loud warn instead of silent overwrite: pilot and proxy modes are
|
||||
// mutually exclusive by topology, so this branch indicates a
|
||||
// misconfigured deployment rather than an expected race.
|
||||
if (expected === undefined && dialer !== null && this.reverseDialer !== null && this.reverseDialer !== dialer) {
|
||||
console.warn('[MeshService] reverse dialer overwritten without CAS; pilot/proxy mode race or duplicate install');
|
||||
}
|
||||
this.reverseDialer = dialer;
|
||||
return true;
|
||||
}
|
||||
|
||||
private dialMeshTcpStream(target: MeshTarget): MeshTcpStreamLike | null {
|
||||
private async dialMeshTcpStream(target: MeshTarget): Promise<MeshTcpStreamLike | null> {
|
||||
if (this.reverseDialer) {
|
||||
return this.reverseDialer.openMeshTcpStream({
|
||||
nodeId: target.nodeId,
|
||||
@@ -1415,13 +1428,16 @@ export class MeshService extends EventEmitter implements MeshForwarderHost {
|
||||
});
|
||||
}
|
||||
const ptm = PilotTunnelManager.getInstance();
|
||||
if (!ptm.hasActiveTunnel(target.nodeId)) return null;
|
||||
const bridge = ptm.getBridge(target.nodeId);
|
||||
// ensureBridge resolves an existing pilot tunnel, an existing
|
||||
// proxy-mode tunnel, or dials a fresh proxy-mode tunnel on demand.
|
||||
// Returns null for unreachable nodes (no api_url/api_token, scope
|
||||
// insufficient, remote pre-Phase-C, network error).
|
||||
const bridge = await ptm.ensureBridge(target.nodeId);
|
||||
if (!bridge) return null;
|
||||
return bridge.openTcpStream({ stack: target.stack, service: target.service, port: target.port });
|
||||
}
|
||||
|
||||
private openCrossNode(target: MeshTarget, src: net.Socket): void {
|
||||
private async openCrossNode(target: MeshTarget, src: net.Socket): Promise<void> {
|
||||
// Log every dispatch entry. Same-node logs route.resolve.ok on
|
||||
// its TCP `connect` event; cross-node only logs route.resolve.ok
|
||||
// once tcp_open_ack arrives from the agent. Without this entry
|
||||
@@ -1433,14 +1449,14 @@ export class MeshService extends EventEmitter implements MeshForwarderHost {
|
||||
message: `cross-node dispatch to ${target.alias} on node ${target.nodeId}`,
|
||||
});
|
||||
|
||||
const tcpStream = this.dialMeshTcpStream(target);
|
||||
const tcpStream = await this.dialMeshTcpStream(target);
|
||||
if (!tcpStream) {
|
||||
this.logActivity({
|
||||
source: 'pilot', level: 'error', type: 'tunnel.fail',
|
||||
nodeId: target.nodeId, alias: target.alias,
|
||||
message: this.reverseDialer
|
||||
? `cannot open reverse mesh stream to node ${target.nodeId}`
|
||||
: `no active pilot tunnel to node ${target.nodeId}`,
|
||||
: `no mesh tunnel reachable for node ${target.nodeId}`,
|
||||
});
|
||||
try { src.destroy(); } catch { /* ignore */ }
|
||||
return;
|
||||
@@ -1597,19 +1613,45 @@ export class MeshService extends EventEmitter implements MeshForwarderHost {
|
||||
// --- Diagnostics ---
|
||||
|
||||
/**
|
||||
* Whether mesh traffic to this node can flow. Local nodes are always
|
||||
* reachable because mesh uses the same-node fast path on localhost.
|
||||
* Remote nodes are reachable only when a pilot tunnel is registered. The
|
||||
* literal `hasActiveTunnel(localNodeId)` would always be false (local
|
||||
* nodes do not establish tunnels to themselves), so a direct call would
|
||||
* render every local alias as `tunnel down` in the UI even on a working
|
||||
* route.
|
||||
* Whether mesh CAN route to a node based on current configuration.
|
||||
* Distinct from "live tunnel up right now": for proxy-mode remotes
|
||||
* the tunnel is opened on demand, so a caller that demanded a
|
||||
* current tunnel would misreport every idle proxy-mode route as
|
||||
* `tunnel down`. Live pilot-tunnel state is surfaced separately.
|
||||
*/
|
||||
private isMeshReachable(nodeId: number): boolean {
|
||||
const node = DatabaseService.getInstance().getNode(nodeId);
|
||||
private isNodeMeshConfigured(node: ReturnType<typeof DatabaseService.prototype.getNode>): boolean {
|
||||
if (!node) return false;
|
||||
if (node.type !== 'remote') return true;
|
||||
return PilotTunnelManager.getInstance().hasActiveTunnel(nodeId);
|
||||
if (node.mode === 'pilot_agent') return true;
|
||||
if (node.mode === 'proxy') return !!node.api_url && !!node.api_token;
|
||||
return false;
|
||||
}
|
||||
|
||||
/**
|
||||
* Compute the reachability classification surfaced in `MeshNodeStatus`
|
||||
* and consumed by the Routing tab. See `MeshReachableMode` for the
|
||||
* meaning of each value. Callers pass in the already-fetched node row
|
||||
* (and the local nodeId) so this helper is free of DB I/O when
|
||||
* `getStatus` iterates the fleet.
|
||||
*/
|
||||
private computeReachable(node: ReturnType<typeof DatabaseService.prototype.getNode>, localNodeId: number): { mode: MeshReachableMode; reason: string | null } {
|
||||
if (!node) return { mode: 'unreachable', reason: 'unknown node' };
|
||||
if (node.id === localNodeId || node.type !== 'remote') return { mode: 'local', reason: null };
|
||||
if (node.mode === 'pilot_agent') return { mode: 'pilot', reason: null };
|
||||
if (node.mode === 'proxy') {
|
||||
if (!node.api_url) return { mode: 'unreachable', reason: 'api_url not set' };
|
||||
if (!node.api_token) return { mode: 'unreachable', reason: 'api token missing' };
|
||||
// Recent-failure cache surfaces the last failed dial so the
|
||||
// operator sees a clear reason without triggering a redial
|
||||
// storm.
|
||||
const failure = MeshProxyTunnelDialer.getInstance().getRecentFailure(node.id);
|
||||
if (failure) {
|
||||
const reason = REACHABLE_REASON[failure.code] ?? failure.message ?? 'remote unreachable';
|
||||
return { mode: 'unreachable', reason };
|
||||
}
|
||||
return { mode: 'proxy', reason: null };
|
||||
}
|
||||
return { mode: 'unreachable', reason: 'unknown node mode' };
|
||||
}
|
||||
|
||||
public async getRouteDiagnostic(alias: string): Promise<MeshRouteDiagnostic> {
|
||||
@@ -1621,14 +1663,20 @@ export class MeshService extends EventEmitter implements MeshForwarderHost {
|
||||
return { alias, target: null, pilot: { connected: false, lastSeen: null }, lastError, lastProbeMs, state: 'not authorized' };
|
||||
}
|
||||
|
||||
const pilotConnected = this.isMeshReachable(target.nodeId);
|
||||
const node = DatabaseService.getInstance().getNode(target.nodeId);
|
||||
const routable = this.isNodeMeshConfigured(node);
|
||||
// Pilot-mode routes surface live tunnel state; proxy-mode routes
|
||||
// fall back to `routable` because the tunnel is opened on demand
|
||||
// and a quiescent state is normal.
|
||||
const pilotLive = node?.type === 'remote' && node.mode === 'pilot_agent'
|
||||
? PilotTunnelManager.getInstance().hasActiveTunnel(target.nodeId)
|
||||
: routable;
|
||||
const lastSeen = node?.pilot_last_seen ?? null;
|
||||
const optedIn = DatabaseService.getInstance().isMeshStackEnabled(target.nodeId, target.stackName);
|
||||
|
||||
let state: MeshRouteDiagnostic['state'];
|
||||
if (!optedIn) state = 'not authorized';
|
||||
else if (!pilotConnected) state = 'tunnel down';
|
||||
else if (!routable || !pilotLive) state = 'tunnel down';
|
||||
else if (lastError && Date.now() - lastError.ts < 60_000) state = 'unreachable';
|
||||
else if (lastProbeMs !== null && lastProbeMs > SLOW_PROBE_THRESHOLD_MS) state = 'degraded';
|
||||
else state = 'healthy';
|
||||
@@ -1642,7 +1690,7 @@ export class MeshService extends EventEmitter implements MeshForwarderHost {
|
||||
port: target.port,
|
||||
alias,
|
||||
},
|
||||
pilot: { connected: pilotConnected, lastSeen },
|
||||
pilot: { connected: pilotLive, lastSeen },
|
||||
lastError,
|
||||
lastProbeMs,
|
||||
state,
|
||||
@@ -1689,15 +1737,26 @@ export class MeshService extends EventEmitter implements MeshForwarderHost {
|
||||
const nodes = db.getNodes();
|
||||
const localNodeId = NodeRegistry.getInstance().getDefaultNodeId();
|
||||
const localListening = this.isLocalForwarderActive();
|
||||
return nodes.map((node) => ({
|
||||
nodeId: node.id,
|
||||
nodeName: node.name,
|
||||
enabled: db.getNodeMeshEnabled(node.id),
|
||||
localForwarderListening: node.id === localNodeId ? localListening : null,
|
||||
pilotConnected: this.isMeshReachable(node.id),
|
||||
optedInStacks: db.listMeshStacks(node.id).map((s) => s.stack_name),
|
||||
activeStreamCount: this.activeStreams.size,
|
||||
}));
|
||||
const ptm = PilotTunnelManager.getInstance();
|
||||
return nodes.map((node) => {
|
||||
const reach = this.computeReachable(node, localNodeId);
|
||||
return {
|
||||
nodeId: node.id,
|
||||
nodeName: node.name,
|
||||
enabled: db.getNodeMeshEnabled(node.id),
|
||||
localForwarderListening: node.id === localNodeId ? localListening : null,
|
||||
// `pilotConnected` stays at its original meaning: a pilot
|
||||
// tunnel is currently registered for this node. The
|
||||
// Routing tab now derives badge state from `reachableMode`
|
||||
// and reads `pilotConnected` only for the pilot-offline
|
||||
// sub-state.
|
||||
pilotConnected: node.type !== 'remote' || ptm.hasActiveTunnel(node.id),
|
||||
reachableMode: reach.mode,
|
||||
reachableReason: reach.reason,
|
||||
optedInStacks: db.listMeshStacks(node.id).map((s) => s.stack_name),
|
||||
activeStreamCount: this.activeStreams.size,
|
||||
};
|
||||
});
|
||||
}
|
||||
|
||||
/** True when the local Sencho's forwarder is started and bound to at least one alias port. */
|
||||
@@ -1731,7 +1790,7 @@ export class MeshError extends Error {
|
||||
/**
|
||||
* Common surface of an outbound mesh TCP stream as MeshService consumes
|
||||
* it. Both the central-side `PilotTunnelBridge.TcpStream` and the
|
||||
* pilot-side `ReverseTcpStreamHandle` (from `pilot/agent.ts`) implement
|
||||
* pilot-side `ReverseTcpStreamHandle` (from `mesh/tcpStreamSwitchboard.ts`) implement
|
||||
* this shape structurally so MeshService.openCrossNode can splice bytes
|
||||
* against either without caring which side initiated the stream.
|
||||
*/
|
||||
|
||||
@@ -16,6 +16,12 @@ interface Counters {
|
||||
tunnels_rejected_capacity: number;
|
||||
enroll_acks: number;
|
||||
frame_decode_errors: number;
|
||||
/** Successful Distributed API mesh proxy-tunnel registrations. Counterpart to `tunnels_total` for pilot-mode tunnels; the two together cover every mesh-capable bridge the manager ever accepted. */
|
||||
proxy_bridges_total: number;
|
||||
/** Failed proxy-tunnel dial attempts (all reasons). Pair with `proxy_bridges_total` for an attempt/success ratio; pair with `proxy_idle_closes` for retention. */
|
||||
proxy_dials_failed: number;
|
||||
/** Proxy-tunnel teardowns initiated by the dialer's idle sweep (zero active streams for the configured TTL). */
|
||||
proxy_idle_closes: number;
|
||||
}
|
||||
|
||||
class PilotMetricsImpl {
|
||||
@@ -25,6 +31,9 @@ class PilotMetricsImpl {
|
||||
tunnels_rejected_capacity: 0,
|
||||
enroll_acks: 0,
|
||||
frame_decode_errors: 0,
|
||||
proxy_bridges_total: 0,
|
||||
proxy_dials_failed: 0,
|
||||
proxy_idle_closes: 0,
|
||||
};
|
||||
|
||||
public increment<K extends keyof Counters>(name: K): void {
|
||||
|
||||
@@ -203,6 +203,8 @@ export class PilotTunnelBridge extends EventEmitter implements MeshTunnelHandle
|
||||
public getLoopbackUrl(): string { return this.loopbackUrl; }
|
||||
public getConnectedAt(): number { return this.connectedAt; }
|
||||
public getBufferedAmount(): number { return this.tunnelWs.bufferedAmount; }
|
||||
/** Total active stream count across HTTP, WebSocket, forward TCP, and reverse TCP. Used by the proxy-tunnel dialer to detect idle bridges eligible for teardown. */
|
||||
public getActiveStreamCount(): number { return this.streams.size; }
|
||||
public isOpen(): boolean { return !this.closed && this.tunnelWs.readyState === WebSocket.OPEN; }
|
||||
|
||||
/**
|
||||
@@ -823,7 +825,10 @@ export class PilotTunnelBridge extends EventEmitter implements MeshTunnelHandle
|
||||
|
||||
private async acceptReverseRelay(s: number, targetNodeId: number, target: { stack: string; service: string; port: number }): Promise<void> {
|
||||
const { PilotTunnelManager } = await import('./PilotTunnelManager');
|
||||
const targetBridge = PilotTunnelManager.getInstance().getBridge(targetNodeId);
|
||||
// ensureBridge resolves either an existing pilot tunnel or a fresh
|
||||
// proxy-mode tunnel dialed on demand, so proxy <-> proxy relays
|
||||
// succeed even when neither remote has a pilot agent.
|
||||
const targetBridge = await PilotTunnelManager.getInstance().ensureBridge(targetNodeId);
|
||||
if (!targetBridge) {
|
||||
this.sendJson({ t: 'tcp_open_ack', s, ok: false, err: 'unreachable' });
|
||||
return;
|
||||
|
||||
@@ -34,17 +34,36 @@ export class PilotTunnelCapacityError extends Error {
|
||||
}
|
||||
|
||||
/**
|
||||
* PilotTunnelManager: singleton registry of active pilot tunnels.
|
||||
* PilotTunnelManager: singleton registry of active mesh-capable bridges.
|
||||
*
|
||||
* Each enrolled pilot-agent node holds one outbound WebSocket to the primary.
|
||||
* For every such tunnel we spin up a local loopback HTTP server that demuxes
|
||||
* requests into frames. Remote-proxy code paths (http-proxy-middleware and the
|
||||
* WebSocket upgrade handler) can then treat pilot nodes identically to standard
|
||||
* proxy nodes by pointing at the loopback URL.
|
||||
* Two flavors of bridge live in the same `bridges` map, keyed by nodeId:
|
||||
*
|
||||
* - **Pilot-agent tunnels** (the original use case): long-lived,
|
||||
* agent-initiated. The pilot dials central; `registerTunnel` accepts
|
||||
* the WS, starts a loopback HTTP server, and emits `tunnel-up` so
|
||||
* downstream observers (capability cache, status badges) refresh. A
|
||||
* pilot-agent bridge stays open for the agent's lifetime and supports
|
||||
* HTTP, WebSocket, and TCP multiplexing.
|
||||
*
|
||||
* - **Proxy-mode tunnels** (Phase C): short-lived, central-initiated.
|
||||
* `ensureBridge` delegates to `MeshProxyTunnelDialer`, which opens a
|
||||
* WebSocket to the remote's `/api/mesh/proxy-tunnel` endpoint using
|
||||
* the long-lived `api_token`. Carries only TCP mesh frames; the
|
||||
* loopback HTTP server is left running on the bridge but unused
|
||||
* because proxy-mode HTTP traffic flows through the existing
|
||||
* `remoteNodeProxy`. Idle close after a configurable TTL.
|
||||
*
|
||||
* Mesh dispatch is mode-agnostic: `MeshService.dialMeshTcpStream` awaits
|
||||
* `ensureBridge(nodeId)` and consumes the resulting `MeshTunnelHandle`.
|
||||
*
|
||||
* Events:
|
||||
* - 'tunnel-up' (nodeId: number) after a tunnel is accepted
|
||||
* - 'tunnel-down' (nodeId: number) after a tunnel closes (for any reason)
|
||||
* - 'tunnel-up' (nodeId) when a pilot-agent tunnel is accepted (NOT
|
||||
* emitted for proxy-mode bridges, which are opened on demand and
|
||||
* should not trigger pilot-specific listeners like the F9 capability
|
||||
* cache invalidation).
|
||||
* - 'tunnel-down' (nodeId) when a pilot-agent tunnel closes.
|
||||
* - 'proxy-bridge-up' / 'proxy-bridge-down' (nodeId) for observability
|
||||
* on proxy-mode bridge lifecycle. No current consumer.
|
||||
*/
|
||||
export class PilotTunnelManager extends EventEmitter {
|
||||
private static instance: PilotTunnelManager;
|
||||
@@ -168,6 +187,60 @@ export class PilotTunnelManager extends EventEmitter {
|
||||
return this.bridges.get(nodeId) ?? null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Dial-if-needed: return the existing pilot or proxy bridge, or open
|
||||
* a new proxy-mode bridge on demand. Used by `MeshService` so cross-
|
||||
* node TCP dispatch works for both pilot-agent remotes (long-lived
|
||||
* tunnel) and proxy-mode remotes (on-demand tunnel) without any
|
||||
* mode-specific branching at the call site.
|
||||
*
|
||||
* Returns null if the node has no active pilot tunnel AND cannot be
|
||||
* dialed as a proxy-mode remote (missing api_url / api_token, scope
|
||||
* insufficient, remote offline, or remote pre-Phase-C).
|
||||
*/
|
||||
public async ensureBridge(nodeId: number): Promise<MeshTunnelHandle | null> {
|
||||
const existing = this.bridges.get(nodeId);
|
||||
if (existing) return existing;
|
||||
// Lazy import to avoid a cycle: MeshProxyTunnelDialer imports
|
||||
// PilotTunnelBridge, which imports PilotTunnelManager via the
|
||||
// existing tcp_open_reverse relay path.
|
||||
const { MeshProxyTunnelDialer } = await import('./MeshProxyTunnelDialer');
|
||||
return MeshProxyTunnelDialer.getInstance().ensureBridge(nodeId);
|
||||
}
|
||||
|
||||
/**
|
||||
* Register a central-initiated proxy-mode bridge for an existing
|
||||
* remote. Distinct from `registerTunnel`: skips the pilot-only side
|
||||
* effects (DB node-status update, `pilot_last_seen` write,
|
||||
* `tunnel-up` event, replacement of any prior pilot tunnel). Still
|
||||
* honors the hard tunnel cap so a dial storm cannot exhaust gateway
|
||||
* memory.
|
||||
*
|
||||
* Throws `PilotTunnelCapacityError` when the cap is reached.
|
||||
*/
|
||||
public registerProxyBridge(nodeId: number, bridge: PilotTunnelBridge): void {
|
||||
const existing = this.bridges.get(nodeId);
|
||||
if (existing) {
|
||||
// A pilot tunnel for this node already exists. Proxy bridges
|
||||
// should not silently shadow them; refuse the registration so
|
||||
// the dialer can surface a clear error.
|
||||
throw new Error(`pilot tunnel already registered for node ${nodeId}; proxy bridge refused`);
|
||||
}
|
||||
if (this.bridges.size >= PILOT_TUNNEL_HARD_LIMIT) {
|
||||
PilotMetrics.increment('tunnels_rejected_capacity');
|
||||
throw new PilotTunnelCapacityError(PILOT_TUNNEL_HARD_LIMIT);
|
||||
}
|
||||
bridge.once('closed', () => {
|
||||
if (this.bridges.get(nodeId) === bridge) {
|
||||
this.bridges.delete(nodeId);
|
||||
this.emit('proxy-bridge-down', nodeId);
|
||||
}
|
||||
});
|
||||
this.bridges.set(nodeId, bridge);
|
||||
PilotMetrics.increment('proxy_bridges_total');
|
||||
this.emit('proxy-bridge-up', nodeId);
|
||||
}
|
||||
|
||||
/**
|
||||
* Force-close a tunnel (e.g., on node deletion).
|
||||
*/
|
||||
|
||||
Reference in New Issue
Block a user