Files
sencho/backend/src/__tests__/pilot-agent-reverse-stream.test.ts
T
Anso a38a3e0226 feat(mesh): route mesh traffic over Distributed API remotes (#1048)
* refactor(mesh): extract shared TCP stream switchboard

Pull the `tcp_open` / `tcp_open_ack` / `tcp_open_reverse` / `tcp_close`
+ `TcpData` handling out of the pilot agent into a reusable module so a
second caller (the upcoming proxy-mode WS handler) can run the same
frame parser, stream allocator, idle timers, and Compose-label
resolver. One parser, two callers, zero drift on a
security-sensitive protocol surface.

The pilot agent keeps its public `openMeshTcpStream` API and delegates
to a per-connection switchboard constructed in `connect()` and torn
down in `cleanupAfterDisconnect`. Existing reverse-stream and resolver
tests are rewritten to exercise the shared module directly.

* feat(mesh): route mesh traffic over Distributed API remotes

Sencho Mesh now works against remotes in Distributed API mode in
addition to Pilot Agent mode. Central opens a short-lived WebSocket
tunnel to the remote's new `/api/mesh/proxy-tunnel` endpoint on demand
and tears it down after 5 minutes of idle (configurable via
`SENCHO_MESH_PROXY_TUNNEL_IDLE_MS`). Mesh dispatch is mode-agnostic at
the routing layer; `PilotTunnelManager.ensureBridge` resolves an
existing tunnel or asks the new `MeshProxyTunnelDialer` to dial.

The Routing tab badges now use a `reachableMode` classifier: `★ Local`
for local, `pilot offline` red badge for a pilot with a down tunnel,
and a new `unreachable` red badge surfacing the specific reason
(missing token, scope not full-admin, TLS failure, remote does not
support proxy mesh). Distributed API remotes with valid credentials
show no negative badge; the tunnel opens on first dial.

Auth: the proxy-tunnel WS upgrade requires an `Authorization: Bearer`
API token with the `full-admin` scope. Lower-scoped tokens are
rejected at upgrade time so a leaked read-only token cannot reach the
mesh data plane.

Bidirectional: the proxy-mode WS handler registers itself as the
local `MeshService` reverse dialer (compare-and-swap), so meshed
containers on a Distributed API remote can dial cross-node aliases
via `tcp_open_reverse` over the same tunnel. Cross-node relays
(`PilotTunnelBridge.acceptReverseRelay`) await `ensureBridge` so
proxy-to-proxy mesh works without standing tunnels.

* docs(mesh): describe Distributed API mesh and unreachable troubleshooting

Refresh the user-facing mesh documentation to reflect that mesh works
over both Pilot Agent and Distributed API remotes. Adds a
Troubleshooting accordion entry for the new `unreachable` Routing tab
badge so operators can match a tooltip reason ("api token rejected
(scope must be full-admin)", "remote does not support proxy mesh",
"TLS handshake failed", "api_url not set", "api token missing") to a
concrete fix.

* refactor(mesh): apply /simplify review findings

Five cleanups surfaced by the post-implementation code review pass.
No behaviour change for fleets running on `main`; only internal
structure improves.

- **Deterministic container IP shared.** Extract the compose-default
  network preference (`pickContainerIp`) and the conventional-name
  fast path (`lookupContainerIp`) into `backend/src/mesh/containerLookup.ts`.
  Both `MeshService.resolveContainerIp` (existing same-node fast
  path) and the switchboard's `resolveByComposeLabels` (proxy/pilot
  inbound dial) now use the same logic. Earlier the switchboard
  helper grabbed the first `Object.values(Networks)` IP, which
  could flip across daemon versions on multi-network containers;
  fixed.
- **WS URL upgrade extracted.** New `backend/src/utils/wsUrl.ts`
  exposes `httpUrlToWs(baseUrl)` that maps `http://` to `ws://` and
  `https://` to `wss://`. Used in both `pilot/agent.ts` and the
  proxy-tunnel dialer. The previous inline `replace(/^http/, 'ws')`
  silently downgraded `https://` to `ws://` (cleartext).
- **`computeReachable` takes the pre-fetched node.** `getStatus`
  already iterated `db.getNodes()`; the helper previously re-queried
  by id per node (N+1 reads). Pass the row in.
- **Reachable-reason ternary -> const map.** Replace the nested
  ternary in `MeshService.computeReachable` with a
  `Record<DialFailureCode, string>` lookup keyed on the dialer's
  failure code.
- **Activity-type ternary -> const map.** Same flattening inside
  `MeshProxyTunnelDialer.logActivity`.

All mesh tests pass (85/85). Full backend suite green (2126/2126).

* fix(mesh): cache proxy dial failures and contain WS handshake errors

`ensureBridge` now consults the recent-failure cache before dialing so a
continuous mesh workload against a misconfigured proxy-mode remote does
not produce one upgrade attempt per cross-node TCP open. `recordFailure`
emits at most one activity-log entry per cache window per (nodeId, code)
so a connect-loop on a single bad remote cannot flush the ring buffer.
Failure messages run through `redactSensitiveText` before reaching the
log so any embedded Bearer / JWT / inline-URL credentials are scrubbed.

`stop()` clears the inflight map and `dial()` checks `this.stopped`
between awaits so a shutdown does not leak a half-opened bridge.

When `awaitOpen` rejects (401, 4403, TLS), the dialer attaches a noop
'error' listener before calling `ws.close()`. Without it the ws library
emits a tail 'error' on a still-CONNECTING socket that propagates as an
unhandled exception. Surfaced by a live-network test against a real
proxy-mode peer.

Adds `mesh-proxy-tunnel-live.test.ts` (skipped unless MESH_AUDIT_URL
and MESH_AUDIT_TOKEN_FILE env vars are set) covering both the happy
path and the auth-rejected path against an actual remote Sencho.

* feat(mesh): instrument proxy tunnel observability

Adds three counters to `PilotMetrics`:
- `proxy_bridges_total` (incremented on `registerProxyBridge` success)
- `proxy_dials_failed` (every failed dial attempt, not deduped)
- `proxy_idle_closes` (idle-sweep teardowns)

All three surface automatically via `GET /api/system/pilot-tunnels`
since the route returns the full `Counters` snapshot.

Gates two pre-existing always-on `console.warn` calls in
`tcpStreamSwitchboard.ts` (mid-stream socket errors and Docker resolve
failures) behind `isDebugEnabled()`. Both fire on per-stream events and
would otherwise violate the diagnostic-log safety rule under load.

Adds `mesh-tcp-stream-switchboard.test.ts` covering the forward
`tcp_open` path: real localhost dial, resolver errors (no_target,
denied), per-tunnel cap saturation, frame-routing fall-through
invariants, `tcp_close` socket teardown.

Drops `MESH_CONNECT_TIMEOUT_MS` from the public surface; only the
switchboard itself uses it.

* fix(mesh): tighten Routing tab unreachable handling

`RoutingNodeCard.tsx`: the **Add stack to mesh** button now disables
when `reachableMode === 'unreachable'`, matching the existing
TogglePill behavior. Without this, an operator on an unreachable node
could open the opt-in sheet, confirm, and watch the redeploy proceed
against a target whose mesh data plane will silently fail to route.
Also drops the leftover `status.nodeId !== -1` guard on the pilot-
offline badge.

`MeshService.ts`: renames `isMeshReachable` to `isMeshConfigured` with
the new predicate that returns true for proxy-mode remotes whose creds
are valid (the tunnel is opened on demand). `getRouteDiagnostic` now
distinguishes "routable" (configured) from "pilotLive" (live tunnel
state, only meaningful for pilot mode); without the split, every idle
proxy-mode route would report `tunnel down`.

`MeshService.setReverseDialer` warns when the unconditional install
path silently overwrites a non-null current dialer. By topology a
Sencho is either pilot or central, so the branch flags a misconfigured
deployment rather than an expected race.

Drops the dead `export { ReverseTcpStreamHandle }` re-export from
`pilot/agent.ts`; its only consumer imports straight from
`mesh/tcpStreamSwitchboard.ts`. Fixes a stale doc comment in
`MeshService.ts` that referenced the wrong source file.

Adds `mesh-proxy-tunnel-handler.test.ts` covering the WS handler
lifecycle: pilot-mode 404 rejection, post-upgrade reverse-dialer
install, concurrent-upgrade 1013 rejection (single-tenant slot), and
error-path teardown.

Updates the troubleshooting accordion in the user docs to mention the
disabled-state behavior.

* fix(mesh): close ESLint and CodeQL findings on the proxy-tunnel diag logs

Remove the unused `reverseDialer` local in `meshProxyTunnel.ts`; the
closure target is `localDialer` above and this assignment was always
dead. Strip line breaks inline on the three `MeshProxyTunnelDialer`
diag log lines so CodeQL `js/log-injection` data flow recognises the
sanitisation that `sanitizeForLog` already performs.

No behaviour change; the diag logs render the same characters they
do today.
2026-05-14 20:51:42 -04:00

211 lines
8.9 KiB
TypeScript

/**
* Reverse mesh streams: the switchboard's `openReverseStream` is the
* outbound side of the protocol. Exercises wire-shape (the JSON
* `tcp_open_reverse` frame and the binary `TcpData` envelope) and the
* stream-handle lifecycle dispatchers (open / data / close).
*
* Both the pilot agent and the proxy-mode WS handler delegate to the
* switchboard's `openReverseStream`; the lifecycle assertions here cover
* the shared logic both callers depend on.
*/
import { afterAll, beforeAll, describe, expect, it } from 'vitest';
import { setupTestDb, cleanupTestDb } from './helpers/setupTestDb';
import {
AGENT_REVERSE_ID_BASE,
BinaryFrameType,
decodeBinaryFrame,
decodeJsonFrame,
encodeBinaryFrame,
} from '../pilot/protocol';
let tmpDir: string;
let attachTcpStreamSwitchboard: typeof import('../mesh/tcpStreamSwitchboard').attachTcpStreamSwitchboard;
let ReverseTcpStreamHandle: typeof import('../mesh/tcpStreamSwitchboard').ReverseTcpStreamHandle;
interface CapturedSend {
raw: string | Buffer;
binary: boolean;
}
function makeSwitchboard(): {
switchboard: ReturnType<typeof attachTcpStreamSwitchboard>;
sent: CapturedSend[];
mockWs: { readyState: number; send: (data: unknown, opts?: { binary?: boolean }) => void };
} {
const sent: CapturedSend[] = [];
const mockWs = {
readyState: 1, // WebSocket.OPEN
send(data: unknown, opts?: { binary?: boolean }) {
const isBinary = opts?.binary === true;
sent.push({
raw: isBinary ? (Buffer.isBuffer(data) ? data : Buffer.from(data as Uint8Array)) : String(data),
binary: isBinary,
});
},
};
// The switchboard accepts the `ws` type structurally; the cast is
// narrow and only used in this test harness.
const switchboard = attachTcpStreamSwitchboard({
ws: mockWs as unknown as import('ws').WebSocket,
resolveTarget: async () => ({ ok: false, err: 'no_target' }),
logLabel: 'Test',
});
return { switchboard, sent, mockWs };
}
beforeAll(async () => {
tmpDir = await setupTestDb();
({ attachTcpStreamSwitchboard, ReverseTcpStreamHandle } = await import('../mesh/tcpStreamSwitchboard'));
});
afterAll(() => {
cleanupTestDb(tmpDir);
});
describe('TcpStreamSwitchboard.openReverseStream', () => {
it('allocates an id in the agent-reverse range and emits a tcp_open_reverse frame with the target', () => {
const { switchboard, sent } = makeSwitchboard();
const handle = switchboard.openReverseStream({
nodeId: 12,
stack: 'api',
service: 'db',
port: 5432,
});
expect(handle).toBeInstanceOf(ReverseTcpStreamHandle);
expect(handle!.streamId).toBeGreaterThanOrEqual(AGENT_REVERSE_ID_BASE);
const textFrames = sent.filter((s) => !s.binary);
expect(textFrames.length).toBe(1);
const decoded = decodeJsonFrame(textFrames[0].raw as string);
expect(decoded.t).toBe('tcp_open_reverse');
if (decoded.t !== 'tcp_open_reverse') throw new Error('narrowing');
expect(decoded.s).toBe(handle!.streamId);
expect(decoded.targetNodeId).toBe(12);
expect(decoded.stack).toBe('api');
expect(decoded.service).toBe('db');
expect(decoded.port).toBe(5432);
});
it('returns null when the tunnel is not OPEN', () => {
const { switchboard, mockWs } = makeSwitchboard();
mockWs.readyState = 0; // CONNECTING
const handle = switchboard.openReverseStream({ nodeId: 1, stack: 'a', service: 'b', port: 1 });
expect(handle).toBeNull();
});
it('handle.write encodes a TcpData binary frame with the allocated streamId', () => {
const { switchboard, sent } = makeSwitchboard();
const handle = switchboard.openReverseStream({ nodeId: 2, stack: 's', service: 'svc', port: 80 });
if (!handle) throw new Error('handle should exist');
sent.length = 0; // clear the open frame
const ok = handle.write(Buffer.from('hello'));
expect(ok).toBe(true);
expect(sent.length).toBe(1);
expect(sent[0].binary).toBe(true);
const decoded = decodeBinaryFrame(sent[0].raw as Buffer);
expect(decoded.type).toBe(BinaryFrameType.TcpData);
expect(decoded.streamId).toBe(handle.streamId);
expect(decoded.payload.toString()).toBe('hello');
});
it('handle.end sends a tcp_close JSON frame and drops the stream count', () => {
const { switchboard, sent } = makeSwitchboard();
const handle = switchboard.openReverseStream({ nodeId: 3, stack: 's', service: 'svc', port: 80 });
if (!handle) throw new Error('handle should exist');
expect(switchboard.tcpStreamCount()).toBe(1);
sent.length = 0;
handle.end();
const text = sent.find((s) => !s.binary);
expect(text).toBeDefined();
const decoded = decodeJsonFrame(text!.raw as string);
expect(decoded.t).toBe('tcp_close');
if (decoded.t !== 'tcp_close') throw new Error('narrowing');
expect(decoded.s).toBe(handle.streamId);
expect(switchboard.tcpStreamCount()).toBe(0);
});
it('inbound tcp_open_ack {ok: true} fires the open event on the matching handle', () => {
const { switchboard } = makeSwitchboard();
const handle = switchboard.openReverseStream({ nodeId: 4, stack: 's', service: 'svc', port: 80 });
if (!handle) throw new Error('handle should exist');
let opened = false;
handle.on('open', () => { opened = true; });
const consumed = switchboard.handleJsonFrame({ t: 'tcp_open_ack', s: handle.streamId, ok: true });
expect(consumed).toBe(true);
expect(opened).toBe(true);
});
it('inbound tcp_open_ack {ok: false} emits error and close, then drops the handle', () => {
const { switchboard } = makeSwitchboard();
const handle = switchboard.openReverseStream({ nodeId: 5, stack: 's', service: 'svc', port: 80 });
if (!handle) throw new Error('handle should exist');
let errMessage: string | undefined;
let closed = false;
handle.on('error', (err: Error) => { errMessage = err.message; });
handle.on('close', () => { closed = true; });
switchboard.handleJsonFrame({ t: 'tcp_open_ack', s: handle.streamId, ok: false, err: 'unreachable' });
expect(errMessage).toBe('unreachable');
expect(closed).toBe(true);
expect(switchboard.tcpStreamCount()).toBe(0);
});
it('forward-direction tcp_open_ack ids (low half) are not consumed by the switchboard', () => {
const { switchboard } = makeSwitchboard();
const handle = switchboard.openReverseStream({ nodeId: 6, stack: 's', service: 'svc', port: 80 });
if (!handle) throw new Error('handle should exist');
let opened = false;
handle.on('open', () => { opened = true; });
const consumed = switchboard.handleJsonFrame({ t: 'tcp_open_ack', s: 5, ok: true });
// Low-half ack is left for the caller to route (it's the
// primary-allocated forward-stream ack, irrelevant to the
// switchboard's reverse map).
expect(consumed).toBe(false);
expect(opened).toBe(false);
});
it('inbound TcpData binary frame (encoded against the wire) emits data on the handle', () => {
const { switchboard } = makeSwitchboard();
const handle = switchboard.openReverseStream({ nodeId: 7, stack: 's', service: 'svc', port: 80 });
if (!handle) throw new Error('handle should exist');
const received: Buffer[] = [];
handle.on('data', (chunk: Buffer) => received.push(chunk));
const buf = encodeBinaryFrame(BinaryFrameType.TcpData, handle.streamId, Buffer.from('echo!'));
const decoded = decodeBinaryFrame(buf);
const consumed = switchboard.handleBinaryFrame(decoded);
expect(consumed).toBe(true);
expect(Buffer.concat(received).toString()).toBe('echo!');
});
it('cleanup() emits error + close on every outstanding reverse handle and resets the count', () => {
const { switchboard } = makeSwitchboard();
const h1 = switchboard.openReverseStream({ nodeId: 8, stack: 's', service: 'a', port: 1 })!;
const h2 = switchboard.openReverseStream({ nodeId: 9, stack: 's', service: 'b', port: 2 })!;
const closed: number[] = [];
const errors: string[] = [];
h1.on('close', () => closed.push(h1.streamId));
h2.on('close', () => closed.push(h2.streamId));
h1.on('error', (e: Error) => errors.push(e.message));
h2.on('error', (e: Error) => errors.push(e.message));
switchboard.cleanup('mock disconnect');
expect(closed).toContain(h1.streamId);
expect(closed).toContain(h2.streamId);
expect(errors).toEqual(expect.arrayContaining(['mock disconnect', 'mock disconnect']));
expect(switchboard.tcpStreamCount()).toBe(0);
});
});