feat(mesh): route mesh traffic over Distributed API remotes (#1048)

* refactor(mesh): extract shared TCP stream switchboard

Pull the `tcp_open` / `tcp_open_ack` / `tcp_open_reverse` / `tcp_close`
+ `TcpData` handling out of the pilot agent into a reusable module so a
second caller (the upcoming proxy-mode WS handler) can run the same
frame parser, stream allocator, idle timers, and Compose-label
resolver. One parser, two callers, zero drift on a
security-sensitive protocol surface.

The pilot agent keeps its public `openMeshTcpStream` API and delegates
to a per-connection switchboard constructed in `connect()` and torn
down in `cleanupAfterDisconnect`. Existing reverse-stream and resolver
tests are rewritten to exercise the shared module directly.

* feat(mesh): route mesh traffic over Distributed API remotes

Sencho Mesh now works against remotes in Distributed API mode in
addition to Pilot Agent mode. Central opens a short-lived WebSocket
tunnel to the remote's new `/api/mesh/proxy-tunnel` endpoint on demand
and tears it down after 5 minutes of idle (configurable via
`SENCHO_MESH_PROXY_TUNNEL_IDLE_MS`). Mesh dispatch is mode-agnostic at
the routing layer; `PilotTunnelManager.ensureBridge` resolves an
existing tunnel or asks the new `MeshProxyTunnelDialer` to dial.

The Routing tab badges now use a `reachableMode` classifier: `★ Local`
for local, `pilot offline` red badge for a pilot with a down tunnel,
and a new `unreachable` red badge surfacing the specific reason
(missing token, scope not full-admin, TLS failure, remote does not
support proxy mesh). Distributed API remotes with valid credentials
show no negative badge; the tunnel opens on first dial.

Auth: the proxy-tunnel WS upgrade requires an `Authorization: Bearer`
API token with the `full-admin` scope. Lower-scoped tokens are
rejected at upgrade time so a leaked read-only token cannot reach the
mesh data plane.

Bidirectional: the proxy-mode WS handler registers itself as the
local `MeshService` reverse dialer (compare-and-swap), so meshed
containers on a Distributed API remote can dial cross-node aliases
via `tcp_open_reverse` over the same tunnel. Cross-node relays
(`PilotTunnelBridge.acceptReverseRelay`) await `ensureBridge` so
proxy-to-proxy mesh works without standing tunnels.

* docs(mesh): describe Distributed API mesh and unreachable troubleshooting

Refresh the user-facing mesh documentation to reflect that mesh works
over both Pilot Agent and Distributed API remotes. Adds a
Troubleshooting accordion entry for the new `unreachable` Routing tab
badge so operators can match a tooltip reason ("api token rejected
(scope must be full-admin)", "remote does not support proxy mesh",
"TLS handshake failed", "api_url not set", "api token missing") to a
concrete fix.

* refactor(mesh): apply /simplify review findings

Five cleanups surfaced by the post-implementation code review pass.
No behaviour change for fleets running on `main`; only internal
structure improves.

- **Deterministic container IP shared.** Extract the compose-default
  network preference (`pickContainerIp`) and the conventional-name
  fast path (`lookupContainerIp`) into `backend/src/mesh/containerLookup.ts`.
  Both `MeshService.resolveContainerIp` (existing same-node fast
  path) and the switchboard's `resolveByComposeLabels` (proxy/pilot
  inbound dial) now use the same logic. Earlier the switchboard
  helper grabbed the first `Object.values(Networks)` IP, which
  could flip across daemon versions on multi-network containers;
  fixed.
- **WS URL upgrade extracted.** New `backend/src/utils/wsUrl.ts`
  exposes `httpUrlToWs(baseUrl)` that maps `http://` to `ws://` and
  `https://` to `wss://`. Used in both `pilot/agent.ts` and the
  proxy-tunnel dialer. The previous inline `replace(/^http/, 'ws')`
  silently downgraded `https://` to `ws://` (cleartext).
- **`computeReachable` takes the pre-fetched node.** `getStatus`
  already iterated `db.getNodes()`; the helper previously re-queried
  by id per node (N+1 reads). Pass the row in.
- **Reachable-reason ternary -> const map.** Replace the nested
  ternary in `MeshService.computeReachable` with a
  `Record<DialFailureCode, string>` lookup keyed on the dialer's
  failure code.
- **Activity-type ternary -> const map.** Same flattening inside
  `MeshProxyTunnelDialer.logActivity`.

All mesh tests pass (85/85). Full backend suite green (2126/2126).

* fix(mesh): cache proxy dial failures and contain WS handshake errors

`ensureBridge` now consults the recent-failure cache before dialing so a
continuous mesh workload against a misconfigured proxy-mode remote does
not produce one upgrade attempt per cross-node TCP open. `recordFailure`
emits at most one activity-log entry per cache window per (nodeId, code)
so a connect-loop on a single bad remote cannot flush the ring buffer.
Failure messages run through `redactSensitiveText` before reaching the
log so any embedded Bearer / JWT / inline-URL credentials are scrubbed.

`stop()` clears the inflight map and `dial()` checks `this.stopped`
between awaits so a shutdown does not leak a half-opened bridge.

When `awaitOpen` rejects (401, 4403, TLS), the dialer attaches a noop
'error' listener before calling `ws.close()`. Without it the ws library
emits a tail 'error' on a still-CONNECTING socket that propagates as an
unhandled exception. Surfaced by a live-network test against a real
proxy-mode peer.

Adds `mesh-proxy-tunnel-live.test.ts` (skipped unless MESH_AUDIT_URL
and MESH_AUDIT_TOKEN_FILE env vars are set) covering both the happy
path and the auth-rejected path against an actual remote Sencho.

* feat(mesh): instrument proxy tunnel observability

Adds three counters to `PilotMetrics`:
- `proxy_bridges_total` (incremented on `registerProxyBridge` success)
- `proxy_dials_failed` (every failed dial attempt, not deduped)
- `proxy_idle_closes` (idle-sweep teardowns)

All three surface automatically via `GET /api/system/pilot-tunnels`
since the route returns the full `Counters` snapshot.

Gates two pre-existing always-on `console.warn` calls in
`tcpStreamSwitchboard.ts` (mid-stream socket errors and Docker resolve
failures) behind `isDebugEnabled()`. Both fire on per-stream events and
would otherwise violate the diagnostic-log safety rule under load.

Adds `mesh-tcp-stream-switchboard.test.ts` covering the forward
`tcp_open` path: real localhost dial, resolver errors (no_target,
denied), per-tunnel cap saturation, frame-routing fall-through
invariants, `tcp_close` socket teardown.

Drops `MESH_CONNECT_TIMEOUT_MS` from the public surface; only the
switchboard itself uses it.

* fix(mesh): tighten Routing tab unreachable handling

`RoutingNodeCard.tsx`: the **Add stack to mesh** button now disables
when `reachableMode === 'unreachable'`, matching the existing
TogglePill behavior. Without this, an operator on an unreachable node
could open the opt-in sheet, confirm, and watch the redeploy proceed
against a target whose mesh data plane will silently fail to route.
Also drops the leftover `status.nodeId !== -1` guard on the pilot-
offline badge.

`MeshService.ts`: renames `isMeshReachable` to `isMeshConfigured` with
the new predicate that returns true for proxy-mode remotes whose creds
are valid (the tunnel is opened on demand). `getRouteDiagnostic` now
distinguishes "routable" (configured) from "pilotLive" (live tunnel
state, only meaningful for pilot mode); without the split, every idle
proxy-mode route would report `tunnel down`.

`MeshService.setReverseDialer` warns when the unconditional install
path silently overwrites a non-null current dialer. By topology a
Sencho is either pilot or central, so the branch flags a misconfigured
deployment rather than an expected race.

Drops the dead `export { ReverseTcpStreamHandle }` re-export from
`pilot/agent.ts`; its only consumer imports straight from
`mesh/tcpStreamSwitchboard.ts`. Fixes a stale doc comment in
`MeshService.ts` that referenced the wrong source file.

Adds `mesh-proxy-tunnel-handler.test.ts` covering the WS handler
lifecycle: pilot-mode 404 rejection, post-upgrade reverse-dialer
install, concurrent-upgrade 1013 rejection (single-tenant slot), and
error-path teardown.

Updates the troubleshooting accordion in the user docs to mention the
disabled-state behavior.

* fix(mesh): close ESLint and CodeQL findings on the proxy-tunnel diag logs

Remove the unused `reverseDialer` local in `meshProxyTunnel.ts`; the
closure target is `localDialer` above and this assignment was always
dead. Strip line breaks inline on the three `MeshProxyTunnelDialer`
diag log lines so CodeQL `js/log-injection` data flow recognises the
sanitisation that `sanitizeForLog` already performs.

No behaviour change; the diag logs render the same characters they
do today.
This commit is contained in:
Anso
2026-05-14 20:51:42 -04:00
committed by GitHub
parent 7c985238e1
commit a38a3e0226
24 changed files with 2225 additions and 561 deletions
@@ -0,0 +1,371 @@
import { EventEmitter } from 'events';
import WebSocket from 'ws';
import { MAX_FRAME_SIZE_BYTES } from '../pilot/protocol';
import { PilotTunnelBridge, type MeshTunnelHandle } from './PilotTunnelBridge';
import { PilotTunnelManager } from './PilotTunnelManager';
import { NodeRegistry } from './NodeRegistry';
import { redactSensitiveText, sanitizeForLog } from '../utils/safeLog';
import { httpUrlToWs } from '../utils/wsUrl';
import { isDebugEnabled } from '../utils/debug';
import { PilotMetrics } from './PilotMetrics';
import type { MeshActivityType } from './MeshService';
/**
* Central-side dialer for proxy-mode mesh tunnels.
*
* Today's pilot tunnels are agent-initiated: the pilot dials central and
* the resulting WS is registered in `PilotTunnelManager`. Proxy-mode
* remotes do not dial; central reaches them via the existing HTTP proxy
* using the long-lived `api_token`. To carry streaming TCP mesh traffic to
* a proxy-mode remote, central opens a WebSocket to the remote's new
* `/api/mesh/proxy-tunnel` endpoint on demand and registers the resulting
* `PilotTunnelBridge` in the manager under the same nodeId. The rest of
* the mesh code path (alias dispatch, openTcpStream, reverse-stream relay)
* is mode-agnostic from there.
*
* Lifecycle:
* - `ensureBridge(nodeId)` dials if not connected; concurrent callers
* dedupe through an in-flight promise map.
* - A periodic check tears down bridges that have seen no active streams
* for `SENCHO_MESH_PROXY_TUNNEL_IDLE_MS` (default 5 minutes). Setting
* the env var to `0` disables idle close (tunnel persists until the
* remote drops it or central shuts down).
* - Recent failures are cached for 60 seconds so a misconfigured remote
* does not cause a continuous redial storm; `MeshService.getStatus`
* consults the cache to surface a `reachableReason` to the UI.
*/
const DEFAULT_IDLE_TTL_MS = 5 * 60 * 1000;
const IDLE_CHECK_INTERVAL_MS = 60 * 1000;
const HANDSHAKE_TIMEOUT_MS = 15_000;
const FAILURE_CACHE_TTL_MS = 60 * 1000;
export type DialFailureCode =
| 'no_target'
| 'endpoint_not_found'
| 'auth_failed'
| 'tls_failed'
| 'network_error';
/**
* Activity-log reason. Wider than `DialFailureCode` because some failures
* share a wire-level code (e.g., `network_error`) but deserve a more
* specific label in the operator-facing log to distinguish a network-
* layer failure from a post-handshake bridge failure or a manager
* rejection.
*/
type DialFailureReason = DialFailureCode | 'bridge_start_failed' | 'manager_rejected';
type ProxyTunnelEvent = 'open.ok' | 'open.fail' | 'close';
const ACTIVITY_TYPE: Record<ProxyTunnelEvent, MeshActivityType> = {
'open.ok': 'proxy-tunnel.open.ok',
'open.fail': 'proxy-tunnel.open.fail',
'close': 'proxy-tunnel.close',
};
export interface DialFailure {
code: DialFailureCode;
message?: string;
ts: number;
}
export class MeshProxyTunnelDialer extends EventEmitter {
private static instance: MeshProxyTunnelDialer | null = null;
private readonly bridges = new Map<number, PilotTunnelBridge>();
private readonly inflight = new Map<number, Promise<MeshTunnelHandle | null>>();
private readonly idleSince = new Map<number, number>();
private readonly recentFailures = new Map<number, DialFailure>();
private readonly idleTtlMs: number;
private idleCheckTimer: NodeJS.Timeout | null = null;
private stopped = false;
private constructor(idleTtlOverrideMs?: number) {
super();
if (typeof idleTtlOverrideMs === 'number') {
this.idleTtlMs = idleTtlOverrideMs;
} else {
const raw = process.env.SENCHO_MESH_PROXY_TUNNEL_IDLE_MS;
const parsed = raw === undefined ? Number.NaN : Number(raw);
this.idleTtlMs = Number.isFinite(parsed) && parsed >= 0 ? parsed : DEFAULT_IDLE_TTL_MS;
}
this.startIdleCheck();
}
public static getInstance(): MeshProxyTunnelDialer {
if (!this.instance) this.instance = new MeshProxyTunnelDialer();
return this.instance;
}
/** Test hook: reset the singleton with an optional idle-TTL override. */
public static resetForTest(idleTtlOverrideMs?: number): MeshProxyTunnelDialer {
if (this.instance) this.instance.stop();
this.instance = new MeshProxyTunnelDialer(idleTtlOverrideMs);
return this.instance;
}
public hasBridge(nodeId: number): boolean {
return this.bridges.has(nodeId);
}
public getBridge(nodeId: number): MeshTunnelHandle | null {
return this.bridges.get(nodeId) ?? null;
}
public getRecentFailure(nodeId: number): DialFailure | null {
const entry = this.recentFailures.get(nodeId);
if (!entry) return null;
if (Date.now() - entry.ts > FAILURE_CACHE_TTL_MS) {
this.recentFailures.delete(nodeId);
return null;
}
return entry;
}
/**
* Dial-if-needed. Concurrent callers dedupe through `inflight`; a
* cached recent failure short-circuits the dial so a misconfigured
* remote does not see one upgrade attempt per cross-node TCP open.
*/
public async ensureBridge(nodeId: number): Promise<MeshTunnelHandle | null> {
const existing = this.bridges.get(nodeId);
if (existing) {
this.idleSince.set(nodeId, Date.now());
return existing;
}
if (this.getRecentFailure(nodeId)) return null;
const inflight = this.inflight.get(nodeId);
if (inflight) return inflight;
const dial = this.dial(nodeId).finally(() => {
this.inflight.delete(nodeId);
});
this.inflight.set(nodeId, dial);
return dial;
}
/** Force-close a bridge (e.g., on node deletion or scope change). */
public closeBridge(nodeId: number, reason = 'closed by dialer'): void {
const bridge = this.bridges.get(nodeId);
if (!bridge) return;
this.bridges.delete(nodeId);
this.idleSince.delete(nodeId);
try { bridge.close(1000, reason); } catch { /* ignore */ }
}
/** Stop the idle-check timer and tear down every open bridge. */
public stop(): void {
this.stopped = true;
if (this.idleCheckTimer) {
clearInterval(this.idleCheckTimer);
this.idleCheckTimer = null;
}
for (const [nodeId, bridge] of this.bridges) {
this.bridges.delete(nodeId);
try { bridge.close(1000, 'dialer shutdown'); } catch { /* ignore */ }
}
this.idleSince.clear();
this.recentFailures.clear();
this.inflight.clear();
}
/** Test hook: count active bridges. */
public get activeBridgeCount(): number {
return this.bridges.size;
}
private async dial(nodeId: number): Promise<MeshTunnelHandle | null> {
const target = NodeRegistry.getInstance().getProxyTarget(nodeId);
if (!target || !target.apiToken) {
this.recordFailure(nodeId, 'no_target', 'no proxy target configured');
return null;
}
if (this.stopped) return null;
const dialStartedAt = Date.now();
if (isDebugEnabled()) {
console.log(`[MeshProxyDialer:diag] dialing node=${nodeId} url=${sanitizeForLog(target.apiUrl)}`.replace(/[\n\r]/g, ''));
}
const wsUrl = httpUrlToWs(target.apiUrl) + '/api/mesh/proxy-tunnel';
let ws: WebSocket;
try {
ws = new WebSocket(wsUrl, {
headers: { Authorization: `Bearer ${target.apiToken}` },
handshakeTimeout: HANDSHAKE_TIMEOUT_MS,
maxPayload: MAX_FRAME_SIZE_BYTES,
});
} catch (err) {
this.recordFailure(nodeId, 'network_error', (err as Error).message);
return null;
}
try {
await this.awaitOpen(ws);
} catch (err) {
// `awaitOpen` removes the 'error' listener on reject; calling
// `close()` on a still-CONNECTING socket emits a tail 'error'
// ('WebSocket was closed before the connection was established')
// that would otherwise propagate as an unhandled exception.
ws.on('error', () => { /* swallow tail error */ });
try { ws.close(); } catch { /* ignore */ }
const failure = classifyDialError(err);
this.recordFailure(nodeId, failure.code, failure.message);
return null;
}
if (this.stopped) {
try { ws.close(1001, 'dialer shutdown'); } catch { /* ignore */ }
return null;
}
const bridge = new PilotTunnelBridge(nodeId, ws);
try {
await bridge.start();
} catch (err) {
this.recordFailure(nodeId, 'network_error', (err as Error).message, 'bridge_start_failed');
try { bridge.close(1011, 'bridge start failed'); } catch { /* ignore */ }
return null;
}
if (this.stopped) {
try { bridge.close(1001, 'dialer shutdown'); } catch { /* ignore */ }
return null;
}
try {
PilotTunnelManager.getInstance().registerProxyBridge(nodeId, bridge);
} catch (err) {
// Cap hit or a pilot tunnel concurrently claimed this nodeId.
this.recordFailure(nodeId, 'network_error', (err as Error).message, 'manager_rejected');
try { bridge.close(1013, 'manager rejected'); } catch { /* ignore */ }
return null;
}
// Mirror the registration in the dialer's own map so the idle
// sweeper can call `getActiveStreamCount()` (not on the narrow
// MeshTunnelHandle interface) without poking into the manager.
bridge.once('closed', () => {
if (this.bridges.get(nodeId) === bridge) {
this.bridges.delete(nodeId);
this.idleSince.delete(nodeId);
void this.logActivity(nodeId, 'close', { reason: 'remote-closed' });
this.emit('proxy-bridge-down', nodeId);
}
});
this.bridges.set(nodeId, bridge);
this.idleSince.set(nodeId, Date.now());
this.recentFailures.delete(nodeId);
void this.logActivity(nodeId, 'open.ok', {});
this.emit('proxy-bridge-up', nodeId);
if (isDebugEnabled()) {
console.log(`[MeshProxyDialer:diag] dial ok node=${nodeId} elapsedMs=${Date.now() - dialStartedAt}`.replace(/[\n\r]/g, ''));
}
return bridge;
}
private awaitOpen(ws: WebSocket): Promise<void> {
return new Promise<void>((resolve, reject) => {
const cleanup = () => {
ws.removeAllListeners('open');
ws.removeAllListeners('error');
ws.removeAllListeners('unexpected-response');
};
ws.once('open', () => { cleanup(); resolve(); });
ws.once('error', (err) => { cleanup(); reject(err); });
ws.once('unexpected-response', (_req, res) => {
cleanup();
const err = new Error(`upgrade failed: HTTP ${res.statusCode}`) as Error & { httpStatus?: number };
err.httpStatus = res.statusCode ?? 0;
try { res.resume(); } catch { /* ignore */ }
reject(err);
});
});
}
/**
* Cache a dial failure and emit at most one activity-log entry per
* cache window per `(nodeId, code)`. The dedupe bounds operator log
* noise when a meshed container retries cross-node TCP opens against
* a misconfigured remote.
*/
private recordFailure(nodeId: number, code: DialFailureCode, rawMessage?: string, reasonOverride?: DialFailureReason): void {
const message = rawMessage ? sanitizeForLog(redactSensitiveText(rawMessage)) : undefined;
const previous = this.recentFailures.get(nodeId);
const reason = reasonOverride ?? code;
const isFresh = !previous
|| Date.now() - previous.ts > FAILURE_CACHE_TTL_MS
|| previous.code !== code;
this.recentFailures.set(nodeId, { code, message, ts: Date.now() });
PilotMetrics.increment('proxy_dials_failed');
if (isFresh) {
void this.logActivity(nodeId, 'open.fail', message ? { reason, message } : { reason });
}
if (isDebugEnabled()) {
console.warn(`[MeshProxyDialer:diag] dial failure node=${nodeId} code=${code} reason=${reason}${message ? ` message=${message}` : ''}`.replace(/[\n\r]/g, ''));
}
}
private startIdleCheck(): void {
if (this.idleCheckTimer || this.stopped) return;
if (this.idleTtlMs <= 0) return; // 0 disables idle close
this.idleCheckTimer = setInterval(() => this.runIdleCheck(), IDLE_CHECK_INTERVAL_MS);
this.idleCheckTimer.unref?.();
}
private runIdleCheck(): void {
const now = Date.now();
for (const [nodeId, bridge] of this.bridges) {
if (bridge.getActiveStreamCount() > 0) {
this.idleSince.set(nodeId, now);
continue;
}
const last = this.idleSince.get(nodeId) ?? now;
if (now - last >= this.idleTtlMs) {
this.bridges.delete(nodeId);
this.idleSince.delete(nodeId);
try { bridge.close(1000, 'idle timeout'); } catch { /* ignore */ }
PilotMetrics.increment('proxy_idle_closes');
void this.logActivity(nodeId, 'close', { reason: 'idle' });
this.emit('proxy-bridge-down', nodeId);
if (isDebugEnabled()) {
console.log(`[MeshProxyDialer:diag] idle close node=${nodeId} idleMs=${now - last}`);
}
}
}
}
private async logActivity(
nodeId: number,
event: ProxyTunnelEvent,
details: Record<string, unknown>,
): Promise<void> {
try {
// Lazy import to avoid a circular dependency with MeshService.
const { MeshService } = await import('./MeshService');
const message = typeof details.message === 'string'
? details.message
: `proxy tunnel ${event} for node ${nodeId}`;
MeshService.getInstance().logActivity({
source: 'mesh',
level: event === 'open.fail' ? 'error' : 'info',
type: ACTIVITY_TYPE[event],
nodeId,
message,
details,
});
} catch {
// Activity logging is best-effort; never let it propagate.
}
}
}
function classifyDialError(err: unknown): { code: DialFailureCode; message: string } {
const message = sanitizeForLog((err as Error).message || String(err));
const httpStatus = (err as Error & { httpStatus?: number }).httpStatus;
if (httpStatus === 404) return { code: 'endpoint_not_found', message: 'remote does not expose /api/mesh/proxy-tunnel' };
if (httpStatus === 401 || httpStatus === 403) return { code: 'auth_failed', message: 'api token rejected by remote' };
const tlsCodes = new Set(['CERT_HAS_EXPIRED', 'UNABLE_TO_VERIFY_LEAF_SIGNATURE', 'DEPTH_ZERO_SELF_SIGNED_CERT', 'SELF_SIGNED_CERT_IN_CHAIN']);
const errno = (err as NodeJS.ErrnoException).code;
if (errno && tlsCodes.has(errno)) return { code: 'tls_failed', message };
return { code: 'network_error', message };
}
+141 -82
View File
@@ -12,7 +12,9 @@ import { PROXY_TIER_HEADER, PROXY_VARIANT_HEADER } from './license-headers';
import { MeshForwarder, type MeshForwarderHost } from './MeshForwarder';
import { NodeRegistry } from './NodeRegistry';
import { PilotTunnelManager } from './PilotTunnelManager';
import { MeshProxyTunnelDialer, type DialFailureCode } from './MeshProxyTunnelDialer';
import { generateOverrideYaml, MeshAlias, SENCHO_MESH_NETWORK } from './MeshComposeOverride';
import { lookupContainerIp } from '../mesh/containerLookup';
import { sanitizeForLog } from '../utils/safeLog';
import { isPathWithinBase, isValidStackName } from '../utils/validation';
import { PORT as SENCHO_LISTEN_PORT } from '../helpers/constants';
@@ -24,6 +26,14 @@ const PROBE_TIMEOUT_MS = 5_000;
const SLOW_PROBE_THRESHOLD_MS = 500;
const DEFAULT_MESH_SUBNET = '172.30.0.0/24';
const REACHABLE_REASON: Record<DialFailureCode, string> = {
auth_failed: 'api token rejected (scope must be full-admin)',
endpoint_not_found: 'remote does not support proxy mesh',
tls_failed: 'TLS handshake failed',
no_target: 'proxy target missing',
network_error: 'remote unreachable',
};
/**
* Returns the static IPv4 address Sencho will pin itself to on the mesh
* Docker network: `<network address> + 2`. The Docker daemon assigns
@@ -60,7 +70,8 @@ export type MeshActivityType =
| 'mesh.enable' | 'mesh.disable'
| 'mesh.override.preserved'
| 'probe.ok' | 'probe.fail'
| 'forwarder.listen' | 'forwarder.unlisten' | 'forwarder.error';
| 'forwarder.listen' | 'forwarder.unlisten' | 'forwarder.error'
| 'proxy-tunnel.open.ok' | 'proxy-tunnel.open.fail' | 'proxy-tunnel.close';
export interface MeshActivityEvent {
ts: number;
@@ -105,13 +116,38 @@ export interface MeshRegenSummary {
reason?: string;
}
/**
* How a node participates in mesh routing right now:
* - `local`: the Sencho serving this request.
* - `pilot`: a remote with a pilot agent. Live-tunnel state is captured
* separately in `pilotConnected`.
* - `proxy`: a remote that central reaches via the long-lived api_token.
* Mesh tunnels are opened on demand by `MeshProxyTunnelDialer`; the
* operator sees no badge while the configuration is sound.
* - `unreachable`: configuration or runtime problem keeps mesh traffic
* from flowing. `reachableReason` carries an actionable hint.
*/
export type MeshReachableMode = 'local' | 'pilot' | 'proxy' | 'unreachable';
export interface MeshNodeStatus {
nodeId: number;
nodeName: string;
enabled: boolean;
/** Forwarder state for the LOCAL node (the Sencho instance answering this request). Always `null` for any non-local node — fetching the remote forwarder state requires a cross-node call which lands in Phase B. */
localForwarderListening: boolean | null;
/**
* True iff a pilot tunnel is currently registered for this node. Only
* meaningful when `reachableMode === 'pilot'`. Kept for diagnostic
* surfaces; the Routing tab badge logic consumes `reachableMode` and
* ignores this field for proxy / local nodes.
* TODO: collapse into `reachableMode` (introduce `pilot_offline` value)
* once no remaining caller reads `pilotConnected` directly.
*/
pilotConnected: boolean;
/** Canonical reachability classification consumed by the Routing tab. */
reachableMode: MeshReachableMode;
/** Short, operator-facing reason when `reachableMode === 'unreachable'`. Null otherwise. */
reachableReason: string | null;
optedInStacks: string[];
activeStreamCount: number;
}
@@ -1284,7 +1320,7 @@ export class MeshService extends EventEmitter implements MeshForwarderHost {
if (target.nodeId === selfNodeId) {
await this.openSameNode(target, src);
} else {
this.openCrossNode(target, src);
await this.openCrossNode(target, src);
}
}
@@ -1340,28 +1376,9 @@ export class MeshService extends EventEmitter implements MeshForwarderHost {
* central.
*/
public async resolveContainerIp(target: { stack: string; service: string }): Promise<string | null> {
const docker = DockerController.getInstance().getDocker() as unknown as Parameters<typeof lookupContainerIp>[0];
try {
const docker = DockerController.getInstance().getDocker();
// Compose default container name pattern; -1 is the first replica.
const conventionalName = `${target.stack}-${target.service}-1`;
const info = await docker.getContainer(conventionalName).inspect().catch(() => null);
const fromInspect = info ? this.extractContainerIp(target.stack, info) : null;
if (fromInspect) return fromInspect;
// Fallback: filter by compose labels in case of a non-conventional
// container name (operator overrode `container_name` or compose
// project).
const containers = await docker.listContainers({
all: true,
filters: {
label: [
`com.docker.compose.project=${target.stack}`,
`com.docker.compose.service=${target.service}`,
],
},
});
if (containers.length === 0) return null;
const fallbackInfo = await docker.getContainer(containers[0].Id).inspect().catch(() => null);
return fallbackInfo ? this.extractContainerIp(target.stack, fallbackInfo) : null;
return await lookupContainerIp(docker, target.stack, target.service);
} catch (err) {
console.warn('[MeshService] container IP lookup failed:', sanitizeForLog((err as Error).message));
return null;
@@ -1369,43 +1386,39 @@ export class MeshService extends EventEmitter implements MeshForwarderHost {
}
/**
* Pick a deterministic IP. Prefer the compose default network
* (`<stack>_default` or any network whose name starts with `<stack>_`),
* then any other declared network, then the legacy bridge `IPAddress`.
* Without this preference order, `Object.values(Networks)` ordering on
* containers attached to multiple networks varies across daemon
* versions and can make same-node forwarding flaky on a redeploy.
*/
private extractContainerIp(
stackName: string,
info: { NetworkSettings?: { Networks?: Record<string, { IPAddress?: string }>; IPAddress?: string } },
): string | null {
const networks = info.NetworkSettings?.Networks ?? {};
const composeDefault = networks[`${stackName}_default`];
if (composeDefault?.IPAddress) return composeDefault.IPAddress;
for (const [name, net] of Object.entries(networks)) {
if (name.startsWith(`${stackName}_`) && net?.IPAddress) return net.IPAddress;
}
for (const net of Object.values(networks)) {
if (net?.IPAddress) return net.IPAddress;
}
return info.NetworkSettings?.IPAddress || null;
}
/**
* Pluggable reverse dialer. Set by `PilotAgent` on a pilot host; left
* null on a central host. When set, `openCrossNode` routes outbound
* mesh dials through the agent's `tcp_open_reverse` path; when unset,
* `openCrossNode` uses the central-side `PilotTunnelManager.getBridge`
* directly. Lets the same MeshService code work on both sides.
* Pluggable reverse dialer. Set by `PilotAgent` on a pilot host or by
* the proxy-mode WS handler when a `/api/mesh/proxy-tunnel` upgrade
* comes in; left null on a central host with no inbound mesh tunnel.
* When set, `openCrossNode` routes outbound mesh dials through the
* dialer's `tcp_open_reverse` path; when unset, `openCrossNode` uses
* the central-side `PilotTunnelManager.getBridge` directly. Lets the
* same MeshService code work on both sides.
*/
private reverseDialer: ReverseMeshDialer | null = null;
public setReverseDialer(dialer: ReverseMeshDialer | null): void {
/**
* Install or clear the reverse dialer. Supports compare-and-swap via
* the optional `expected` argument so a caller (e.g., the proxy-mode
* WS handler) can install on a null slot and uninstall only if its
* own dialer is still the active one. Without the `expected` arg the
* operation is unconditional (used by the pilot agent at boot).
*
* Returns true when the swap happened, false when the CAS rejected
* because `expected` did not match the current dialer.
*/
public setReverseDialer(dialer: ReverseMeshDialer | null, expected?: ReverseMeshDialer | null): boolean {
if (expected !== undefined && this.reverseDialer !== expected) return false;
// Loud warn instead of silent overwrite: pilot and proxy modes are
// mutually exclusive by topology, so this branch indicates a
// misconfigured deployment rather than an expected race.
if (expected === undefined && dialer !== null && this.reverseDialer !== null && this.reverseDialer !== dialer) {
console.warn('[MeshService] reverse dialer overwritten without CAS; pilot/proxy mode race or duplicate install');
}
this.reverseDialer = dialer;
return true;
}
private dialMeshTcpStream(target: MeshTarget): MeshTcpStreamLike | null {
private async dialMeshTcpStream(target: MeshTarget): Promise<MeshTcpStreamLike | null> {
if (this.reverseDialer) {
return this.reverseDialer.openMeshTcpStream({
nodeId: target.nodeId,
@@ -1415,13 +1428,16 @@ export class MeshService extends EventEmitter implements MeshForwarderHost {
});
}
const ptm = PilotTunnelManager.getInstance();
if (!ptm.hasActiveTunnel(target.nodeId)) return null;
const bridge = ptm.getBridge(target.nodeId);
// ensureBridge resolves an existing pilot tunnel, an existing
// proxy-mode tunnel, or dials a fresh proxy-mode tunnel on demand.
// Returns null for unreachable nodes (no api_url/api_token, scope
// insufficient, remote pre-Phase-C, network error).
const bridge = await ptm.ensureBridge(target.nodeId);
if (!bridge) return null;
return bridge.openTcpStream({ stack: target.stack, service: target.service, port: target.port });
}
private openCrossNode(target: MeshTarget, src: net.Socket): void {
private async openCrossNode(target: MeshTarget, src: net.Socket): Promise<void> {
// Log every dispatch entry. Same-node logs route.resolve.ok on
// its TCP `connect` event; cross-node only logs route.resolve.ok
// once tcp_open_ack arrives from the agent. Without this entry
@@ -1433,14 +1449,14 @@ export class MeshService extends EventEmitter implements MeshForwarderHost {
message: `cross-node dispatch to ${target.alias} on node ${target.nodeId}`,
});
const tcpStream = this.dialMeshTcpStream(target);
const tcpStream = await this.dialMeshTcpStream(target);
if (!tcpStream) {
this.logActivity({
source: 'pilot', level: 'error', type: 'tunnel.fail',
nodeId: target.nodeId, alias: target.alias,
message: this.reverseDialer
? `cannot open reverse mesh stream to node ${target.nodeId}`
: `no active pilot tunnel to node ${target.nodeId}`,
: `no mesh tunnel reachable for node ${target.nodeId}`,
});
try { src.destroy(); } catch { /* ignore */ }
return;
@@ -1597,19 +1613,45 @@ export class MeshService extends EventEmitter implements MeshForwarderHost {
// --- Diagnostics ---
/**
* Whether mesh traffic to this node can flow. Local nodes are always
* reachable because mesh uses the same-node fast path on localhost.
* Remote nodes are reachable only when a pilot tunnel is registered. The
* literal `hasActiveTunnel(localNodeId)` would always be false (local
* nodes do not establish tunnels to themselves), so a direct call would
* render every local alias as `tunnel down` in the UI even on a working
* route.
* Whether mesh CAN route to a node based on current configuration.
* Distinct from "live tunnel up right now": for proxy-mode remotes
* the tunnel is opened on demand, so a caller that demanded a
* current tunnel would misreport every idle proxy-mode route as
* `tunnel down`. Live pilot-tunnel state is surfaced separately.
*/
private isMeshReachable(nodeId: number): boolean {
const node = DatabaseService.getInstance().getNode(nodeId);
private isNodeMeshConfigured(node: ReturnType<typeof DatabaseService.prototype.getNode>): boolean {
if (!node) return false;
if (node.type !== 'remote') return true;
return PilotTunnelManager.getInstance().hasActiveTunnel(nodeId);
if (node.mode === 'pilot_agent') return true;
if (node.mode === 'proxy') return !!node.api_url && !!node.api_token;
return false;
}
/**
* Compute the reachability classification surfaced in `MeshNodeStatus`
* and consumed by the Routing tab. See `MeshReachableMode` for the
* meaning of each value. Callers pass in the already-fetched node row
* (and the local nodeId) so this helper is free of DB I/O when
* `getStatus` iterates the fleet.
*/
private computeReachable(node: ReturnType<typeof DatabaseService.prototype.getNode>, localNodeId: number): { mode: MeshReachableMode; reason: string | null } {
if (!node) return { mode: 'unreachable', reason: 'unknown node' };
if (node.id === localNodeId || node.type !== 'remote') return { mode: 'local', reason: null };
if (node.mode === 'pilot_agent') return { mode: 'pilot', reason: null };
if (node.mode === 'proxy') {
if (!node.api_url) return { mode: 'unreachable', reason: 'api_url not set' };
if (!node.api_token) return { mode: 'unreachable', reason: 'api token missing' };
// Recent-failure cache surfaces the last failed dial so the
// operator sees a clear reason without triggering a redial
// storm.
const failure = MeshProxyTunnelDialer.getInstance().getRecentFailure(node.id);
if (failure) {
const reason = REACHABLE_REASON[failure.code] ?? failure.message ?? 'remote unreachable';
return { mode: 'unreachable', reason };
}
return { mode: 'proxy', reason: null };
}
return { mode: 'unreachable', reason: 'unknown node mode' };
}
public async getRouteDiagnostic(alias: string): Promise<MeshRouteDiagnostic> {
@@ -1621,14 +1663,20 @@ export class MeshService extends EventEmitter implements MeshForwarderHost {
return { alias, target: null, pilot: { connected: false, lastSeen: null }, lastError, lastProbeMs, state: 'not authorized' };
}
const pilotConnected = this.isMeshReachable(target.nodeId);
const node = DatabaseService.getInstance().getNode(target.nodeId);
const routable = this.isNodeMeshConfigured(node);
// Pilot-mode routes surface live tunnel state; proxy-mode routes
// fall back to `routable` because the tunnel is opened on demand
// and a quiescent state is normal.
const pilotLive = node?.type === 'remote' && node.mode === 'pilot_agent'
? PilotTunnelManager.getInstance().hasActiveTunnel(target.nodeId)
: routable;
const lastSeen = node?.pilot_last_seen ?? null;
const optedIn = DatabaseService.getInstance().isMeshStackEnabled(target.nodeId, target.stackName);
let state: MeshRouteDiagnostic['state'];
if (!optedIn) state = 'not authorized';
else if (!pilotConnected) state = 'tunnel down';
else if (!routable || !pilotLive) state = 'tunnel down';
else if (lastError && Date.now() - lastError.ts < 60_000) state = 'unreachable';
else if (lastProbeMs !== null && lastProbeMs > SLOW_PROBE_THRESHOLD_MS) state = 'degraded';
else state = 'healthy';
@@ -1642,7 +1690,7 @@ export class MeshService extends EventEmitter implements MeshForwarderHost {
port: target.port,
alias,
},
pilot: { connected: pilotConnected, lastSeen },
pilot: { connected: pilotLive, lastSeen },
lastError,
lastProbeMs,
state,
@@ -1689,15 +1737,26 @@ export class MeshService extends EventEmitter implements MeshForwarderHost {
const nodes = db.getNodes();
const localNodeId = NodeRegistry.getInstance().getDefaultNodeId();
const localListening = this.isLocalForwarderActive();
return nodes.map((node) => ({
nodeId: node.id,
nodeName: node.name,
enabled: db.getNodeMeshEnabled(node.id),
localForwarderListening: node.id === localNodeId ? localListening : null,
pilotConnected: this.isMeshReachable(node.id),
optedInStacks: db.listMeshStacks(node.id).map((s) => s.stack_name),
activeStreamCount: this.activeStreams.size,
}));
const ptm = PilotTunnelManager.getInstance();
return nodes.map((node) => {
const reach = this.computeReachable(node, localNodeId);
return {
nodeId: node.id,
nodeName: node.name,
enabled: db.getNodeMeshEnabled(node.id),
localForwarderListening: node.id === localNodeId ? localListening : null,
// `pilotConnected` stays at its original meaning: a pilot
// tunnel is currently registered for this node. The
// Routing tab now derives badge state from `reachableMode`
// and reads `pilotConnected` only for the pilot-offline
// sub-state.
pilotConnected: node.type !== 'remote' || ptm.hasActiveTunnel(node.id),
reachableMode: reach.mode,
reachableReason: reach.reason,
optedInStacks: db.listMeshStacks(node.id).map((s) => s.stack_name),
activeStreamCount: this.activeStreams.size,
};
});
}
/** True when the local Sencho's forwarder is started and bound to at least one alias port. */
@@ -1731,7 +1790,7 @@ export class MeshError extends Error {
/**
* Common surface of an outbound mesh TCP stream as MeshService consumes
* it. Both the central-side `PilotTunnelBridge.TcpStream` and the
* pilot-side `ReverseTcpStreamHandle` (from `pilot/agent.ts`) implement
* pilot-side `ReverseTcpStreamHandle` (from `mesh/tcpStreamSwitchboard.ts`) implement
* this shape structurally so MeshService.openCrossNode can splice bytes
* against either without caring which side initiated the stream.
*/
+9
View File
@@ -16,6 +16,12 @@ interface Counters {
tunnels_rejected_capacity: number;
enroll_acks: number;
frame_decode_errors: number;
/** Successful Distributed API mesh proxy-tunnel registrations. Counterpart to `tunnels_total` for pilot-mode tunnels; the two together cover every mesh-capable bridge the manager ever accepted. */
proxy_bridges_total: number;
/** Failed proxy-tunnel dial attempts (all reasons). Pair with `proxy_bridges_total` for an attempt/success ratio; pair with `proxy_idle_closes` for retention. */
proxy_dials_failed: number;
/** Proxy-tunnel teardowns initiated by the dialer's idle sweep (zero active streams for the configured TTL). */
proxy_idle_closes: number;
}
class PilotMetricsImpl {
@@ -25,6 +31,9 @@ class PilotMetricsImpl {
tunnels_rejected_capacity: 0,
enroll_acks: 0,
frame_decode_errors: 0,
proxy_bridges_total: 0,
proxy_dials_failed: 0,
proxy_idle_closes: 0,
};
public increment<K extends keyof Counters>(name: K): void {
+6 -1
View File
@@ -203,6 +203,8 @@ export class PilotTunnelBridge extends EventEmitter implements MeshTunnelHandle
public getLoopbackUrl(): string { return this.loopbackUrl; }
public getConnectedAt(): number { return this.connectedAt; }
public getBufferedAmount(): number { return this.tunnelWs.bufferedAmount; }
/** Total active stream count across HTTP, WebSocket, forward TCP, and reverse TCP. Used by the proxy-tunnel dialer to detect idle bridges eligible for teardown. */
public getActiveStreamCount(): number { return this.streams.size; }
public isOpen(): boolean { return !this.closed && this.tunnelWs.readyState === WebSocket.OPEN; }
/**
@@ -823,7 +825,10 @@ export class PilotTunnelBridge extends EventEmitter implements MeshTunnelHandle
private async acceptReverseRelay(s: number, targetNodeId: number, target: { stack: string; service: string; port: number }): Promise<void> {
const { PilotTunnelManager } = await import('./PilotTunnelManager');
const targetBridge = PilotTunnelManager.getInstance().getBridge(targetNodeId);
// ensureBridge resolves either an existing pilot tunnel or a fresh
// proxy-mode tunnel dialed on demand, so proxy <-> proxy relays
// succeed even when neither remote has a pilot agent.
const targetBridge = await PilotTunnelManager.getInstance().ensureBridge(targetNodeId);
if (!targetBridge) {
this.sendJson({ t: 'tcp_open_ack', s, ok: false, err: 'unreachable' });
return;
+81 -8
View File
@@ -34,17 +34,36 @@ export class PilotTunnelCapacityError extends Error {
}
/**
* PilotTunnelManager: singleton registry of active pilot tunnels.
* PilotTunnelManager: singleton registry of active mesh-capable bridges.
*
* Each enrolled pilot-agent node holds one outbound WebSocket to the primary.
* For every such tunnel we spin up a local loopback HTTP server that demuxes
* requests into frames. Remote-proxy code paths (http-proxy-middleware and the
* WebSocket upgrade handler) can then treat pilot nodes identically to standard
* proxy nodes by pointing at the loopback URL.
* Two flavors of bridge live in the same `bridges` map, keyed by nodeId:
*
* - **Pilot-agent tunnels** (the original use case): long-lived,
* agent-initiated. The pilot dials central; `registerTunnel` accepts
* the WS, starts a loopback HTTP server, and emits `tunnel-up` so
* downstream observers (capability cache, status badges) refresh. A
* pilot-agent bridge stays open for the agent's lifetime and supports
* HTTP, WebSocket, and TCP multiplexing.
*
* - **Proxy-mode tunnels** (Phase C): short-lived, central-initiated.
* `ensureBridge` delegates to `MeshProxyTunnelDialer`, which opens a
* WebSocket to the remote's `/api/mesh/proxy-tunnel` endpoint using
* the long-lived `api_token`. Carries only TCP mesh frames; the
* loopback HTTP server is left running on the bridge but unused
* because proxy-mode HTTP traffic flows through the existing
* `remoteNodeProxy`. Idle close after a configurable TTL.
*
* Mesh dispatch is mode-agnostic: `MeshService.dialMeshTcpStream` awaits
* `ensureBridge(nodeId)` and consumes the resulting `MeshTunnelHandle`.
*
* Events:
* - 'tunnel-up' (nodeId: number) after a tunnel is accepted
* - 'tunnel-down' (nodeId: number) after a tunnel closes (for any reason)
* - 'tunnel-up' (nodeId) when a pilot-agent tunnel is accepted (NOT
* emitted for proxy-mode bridges, which are opened on demand and
* should not trigger pilot-specific listeners like the F9 capability
* cache invalidation).
* - 'tunnel-down' (nodeId) when a pilot-agent tunnel closes.
* - 'proxy-bridge-up' / 'proxy-bridge-down' (nodeId) for observability
* on proxy-mode bridge lifecycle. No current consumer.
*/
export class PilotTunnelManager extends EventEmitter {
private static instance: PilotTunnelManager;
@@ -168,6 +187,60 @@ export class PilotTunnelManager extends EventEmitter {
return this.bridges.get(nodeId) ?? null;
}
/**
* Dial-if-needed: return the existing pilot or proxy bridge, or open
* a new proxy-mode bridge on demand. Used by `MeshService` so cross-
* node TCP dispatch works for both pilot-agent remotes (long-lived
* tunnel) and proxy-mode remotes (on-demand tunnel) without any
* mode-specific branching at the call site.
*
* Returns null if the node has no active pilot tunnel AND cannot be
* dialed as a proxy-mode remote (missing api_url / api_token, scope
* insufficient, remote offline, or remote pre-Phase-C).
*/
public async ensureBridge(nodeId: number): Promise<MeshTunnelHandle | null> {
const existing = this.bridges.get(nodeId);
if (existing) return existing;
// Lazy import to avoid a cycle: MeshProxyTunnelDialer imports
// PilotTunnelBridge, which imports PilotTunnelManager via the
// existing tcp_open_reverse relay path.
const { MeshProxyTunnelDialer } = await import('./MeshProxyTunnelDialer');
return MeshProxyTunnelDialer.getInstance().ensureBridge(nodeId);
}
/**
* Register a central-initiated proxy-mode bridge for an existing
* remote. Distinct from `registerTunnel`: skips the pilot-only side
* effects (DB node-status update, `pilot_last_seen` write,
* `tunnel-up` event, replacement of any prior pilot tunnel). Still
* honors the hard tunnel cap so a dial storm cannot exhaust gateway
* memory.
*
* Throws `PilotTunnelCapacityError` when the cap is reached.
*/
public registerProxyBridge(nodeId: number, bridge: PilotTunnelBridge): void {
const existing = this.bridges.get(nodeId);
if (existing) {
// A pilot tunnel for this node already exists. Proxy bridges
// should not silently shadow them; refuse the registration so
// the dialer can surface a clear error.
throw new Error(`pilot tunnel already registered for node ${nodeId}; proxy bridge refused`);
}
if (this.bridges.size >= PILOT_TUNNEL_HARD_LIMIT) {
PilotMetrics.increment('tunnels_rejected_capacity');
throw new PilotTunnelCapacityError(PILOT_TUNNEL_HARD_LIMIT);
}
bridge.once('closed', () => {
if (this.bridges.get(nodeId) === bridge) {
this.bridges.delete(nodeId);
this.emit('proxy-bridge-down', nodeId);
}
});
this.bridges.set(nodeId, bridge);
PilotMetrics.increment('proxy_bridges_total');
this.emit('proxy-bridge-up', nodeId);
}
/**
* Force-close a tunnel (e.g., on node deletion).
*/