fix(mesh): address audit follow-ups (early-data, recompose, admiral gate, error codes, docs) (#1126)

* fix(mesh): buffer early TcpData on reverse-relay path

The reverse-relay code dropped the local-shaped reservation and any
TcpData frames buffered in it before acceptReverseRelay had wired up
the target stream. A peer that sent a request body immediately after
tcp_open_reverse lost those bytes when the target lived on a third
node, since reverse_local buffers them but reverse_relay did not.

Carry the reservation through acceptReverseRelay: transplant its
pendingData and pendingBytes into the new reverse_relay state, gate
TcpData writes on targetOpen, and flush buffered frames into the
target stream before sending tcp_open_ack. Mirrors the reverse_local
pattern. Regression test fires tcp_open_reverse plus an immediate
TcpData while ensureBridge is in flight and verifies the bytes
arrive intact, in order, before the ack.

* fix(mesh): recompose affected stacks when node-level mesh is disabled

disableForNode used to clear DB rows and override files but leave the
running containers attached to sencho_mesh with stale /etc/hosts
alias entries until an operator redeployed every stack by hand.

Mirror optOutStack: after the existing alias/forwarder cleanup, call
regenerateOverridesAcrossFleet, cascadeRecomposeAcrossFleet, and
triggerRedeploy for each previously meshed stack on the disabled
node so containers detach from sencho_mesh and shed the alias
entries they owned. The disabled node's mesh_stacks rows are deleted
before the cascade so listMeshStacks returns the right set with no
skip tuple required. Route threads the actor through actorFor(req)
for parity with optInStack/optOutStack. Tests cover the redeploy
fan-out, the cascade no-skip-tuple invariant, and the default actor
fallback for non-route callers.

* fix(mesh): require Admiral on the WS proxy-tunnel upgrade

HTTP mesh routes in routes/mesh.ts all enforce requireAdmiral, but
the /api/mesh/proxy-tunnel WS upgrade accepted any node_proxy or
full-admin api_token regardless of the receiver's license. A node
downgraded from Admiral kept serving mesh data-plane traffic to a
sibling central while refusing every mesh management call.

Read the receiver's local LicenseService at the upgrade and 403 when
the tier is not paid+admiral. The check sits after the existing
credential gate and uses LicenseService directly rather than
effectiveTier (which trusts forwarded proxy headers); a remote peer
dialing in cannot be trusted to assert our entitlement. Dialer and
node_proxy token format are unchanged. Three regression tests cover
community-tier node_proxy, skipper-tier node_proxy, and
community-tier full-admin api_token all rejected with 403.

* fix(mesh): handle no_target alongside push_failed in inspectStackServices

proxyFetch throws MeshError('no_target') when getProxyTarget returns
null (pilot tunnel offline, proxy bridge unreachable), but the
inspectStackServices catch branch only matched push_failed. Offline
remotes fell through to the generic 'remote unreachable' error log,
which the Routing tab surfaces as an unexpected fault.

Match both error codes and emit the operator-friendly warn message
that names the unreachable node and the error code. Regression test
spies on console.warn/console.error to pin the branch.

* docs(mesh): align env defaults and forwarder comments with current architecture

SENCHO_MESH_PROXY_TUNNEL_IDLE_MS in .env.example carried the old
five-minute idle-close value (=300000), but the code default is
DEFAULT_IDLE_TTL_MS=0 (persistent tunnel). Copying the example
silently reintroduced the idle-close behavior the dialer removed.

MeshForwarder.ts's leading docblock and inline listen comment still
described host-network mode plus extra_hosts: host-gateway as
required for forwarder reachability. Sencho runs in standard bridge
mode and attaches to the shared sencho_mesh network at a stable IP;
meshed user containers reach the forwarder by that IP directly.

Flip the env default to 0, rewrite the env comment to describe the
persistent behavior and the opt-in for idle teardown, and rewrite
both forwarder comments to match the bridge-network reality.

* fix(mesh): trust forwarded tier on proxy-tunnel WS and remove remote overrides on disable

Admiral entitlement on the WS data plane now follows the same trust model
as the HTTP mesh routes: the central asserts its tier via x-sencho-tier
and x-sencho-variant on the WS handshake, and the receiver trusts those
headers only when the upgrade carries a node_proxy credential. When no
headers are present or the credential is a full-admin api_token, the
receiver falls back to its own local license. Without this an Admiral
central could be rejected by a Community remote and a Community central
could dial a locally-Admiral remote.

disableForNode now routes through removeOverrideFromNode so override
files pushed earlier via applyLocalOverride are removed on remote nodes
via DELETE /api/mesh/local-override/:stack. Sequential awaits are wrapped
in Promise.allSettled to match regenerateOverridesForNode's parallel
push pattern.

Other changes:
- Cover the post-state-swap buffering window in the reverse-relay test
  (TcpData arriving after openTcpStream returns but before target open).
- Refresh stale MeshService comments that still referenced the removed
  sidecar layer and host-network listener model.

* test(mesh): pin removeOverrideFromNode remote HTTP shape

The disableForNode regression test mocks removeOverrideFromNode itself,
so a regression inside the helper would not be caught. Add a narrow
contract test that spies on global fetch and asserts the request shape:
DELETE /api/mesh/local-override/:stack against the resolved proxy
target, with Authorization Bearer plus the x-sencho-tier and
x-sencho-variant headers. Also covers the encodeURIComponent path and
the swallowed-network-error behavior the disable cascade relies on.

* test(mesh): use createTestApiToken helper in proxy-tunnel api_token gate test

The full-admin api_token branch of the WS Admiral gate test inlined the
canonical generateApiToken + sha256 + addApiToken triple that already
lives in the createTestApiToken helper. Switching to the helper removes
the duplicated insertion logic and aligns this test with the helper used
by the other api_token call sites.
This commit is contained in:
Anso
2026-05-20 12:53:29 -04:00
committed by GitHub
parent 08caa914ce
commit 982c7830b1
12 changed files with 658 additions and 57 deletions
+40 -25
View File
@@ -242,9 +242,8 @@ interface ActiveStreamRecord {
/**
* Sencho Mesh orchestrator. Owns:
* - in-process TCP forwarder (`MeshForwarder`) that binds host-network
* listeners on alias ports. Replaces the prior separate sidecar
* container; one container per node now.
* - in-process TCP forwarder (`MeshForwarder`) that binds per-alias
* listeners on Sencho's `sencho_mesh` bridge-network IP.
* - opt-in / opt-out persistence and cascading override regeneration
* - global alias aggregation (across the fleet via the existing HTTP
* proxy chain, see `inspectStackServices`)
@@ -258,8 +257,6 @@ interface ActiveStreamRecord {
* Docker bridge network. Meshed user services join `sencho_mesh` so
* that IP is reachable from inside their containers without any
* host-firewall coordination.
* - cross-node mesh routing is central → pilot in this phase. Pilot →
* central and pilot ↔ pilot via central relay land in Phase B.
*/
export class MeshService extends EventEmitter implements MeshForwarderHost {
private static instance: MeshService;
@@ -904,15 +901,38 @@ export class MeshService extends EventEmitter implements MeshForwarderHost {
}
}
public async disableForNode(nodeId: number): Promise<void> {
public async disableForNode(
nodeId: number,
actor: string = 'system:mesh.disable',
): Promise<void> {
DatabaseService.getInstance().setNodeMeshEnabled(nodeId, false);
const stacks = DatabaseService.getInstance().listMeshStacks(nodeId);
for (const s of stacks) {
DatabaseService.getInstance().deleteMeshStack(nodeId, s.stack_name);
await this.removeStackOverride(nodeId, s.stack_name);
}
// Dispatch DELETE /api/mesh/local-override/:stack for remote nodes
// (pilot or proxy) so the override file pushed earlier via
// applyLocalOverride is removed; falls back to local deletion for
// local nodes. Parallelize per the regenerateOverridesForNode
// rationale: each remote call is its own HTTP round-trip, so
// awaiting sequentially turns N stacks into N serialised DELETEs.
// `allSettled` so a single failure does not abort the others
// (removeOverrideFromNode already swallows errors internally).
await Promise.allSettled(
stacks.map((s) => this.removeOverrideFromNode(nodeId, s.stack_name)),
);
await this.refreshAliasCache();
await this.syncForwarderListeners();
// Mirror optOutStack: regenerate every remaining node's override
// without the dropped aliases, recompose the rest of the fleet so
// their containers shed the stale extra_hosts, and redeploy the
// disabled node's own stacks so their containers detach from the
// sencho_mesh network and lose the alias entries they owned.
await this.regenerateOverridesAcrossFleet();
this.cascadeRecomposeAcrossFleet(undefined, undefined, actor);
for (const s of stacks) {
this.triggerRedeploy(nodeId, s.stack_name, actor);
}
this.logActivity({
source: 'mesh', level: 'info', type: 'mesh.disable',
nodeId, message: `mesh disabled on node ${nodeId}`,
@@ -1415,11 +1435,13 @@ export class MeshService extends EventEmitter implements MeshForwarderHost {
const body = await res.json() as { services?: Array<{ service: string; ports: number[] }> };
return body.services ?? [];
} catch (err) {
// proxyFetch throws MeshError('push_failed') when getProxyTarget
// returns null (e.g. pilot tunnel offline). Treat the same as a
// non-OK response: empty list.
if (err instanceof MeshError && err.code === 'push_failed') {
console.warn(`[MeshService] inspectStackServices: no proxy target for node ${nodeId} (${sanitizeForLog(node.name)})`);
// proxyFetch throws MeshError('no_target') when getProxyTarget
// returns null (pilot tunnel offline, proxy bridge unreachable)
// and MeshError('push_failed') on a non-OK HTTP response. Treat
// both as a soft "no services to report" so the Routing tab does
// not surface a stack trace for a node that is simply offline.
if (err instanceof MeshError && (err.code === 'no_target' || err.code === 'push_failed')) {
console.warn(`[MeshService] inspectStackServices: unreachable node ${nodeId} (${sanitizeForLog(node.name)}): ${err.code}`);
return [];
}
console.error('[MeshService] inspectStackServices remote unreachable:', sanitizeForLog((err as Error).message));
@@ -1690,12 +1712,12 @@ export class MeshService extends EventEmitter implements MeshForwarderHost {
/**
* Same-node forward: dial the target container's bridge IP directly.
* Sencho runs in `network_mode: host` so it sees the docker bridge
* networks and can reach container IPs without going through any
* host-port publish. Looks up the container by Compose's
* `<project>-<service>-<index>` naming convention; falls back to a
* label-filtered listContainers if the conventional name is absent
* (e.g. when the operator overrode the project name).
* Sencho joins the `sencho_mesh` Docker bridge network alongside the
* meshed user containers, so it can reach their bridge IPs without
* going through any host-port publish. Looks up the container by
* Compose's `<project>-<service>-<index>` naming convention; falls
* back to a label-filtered listContainers if the conventional name
* is absent (e.g. when the operator overrode the project name).
*/
private async openSameNode(target: MeshTarget, src: net.Socket): Promise<void> {
const ip = await this.resolveContainerIp(target);
@@ -2244,13 +2266,6 @@ export class MeshService extends EventEmitter implements MeshForwarderHost {
private isLocalForwarderActive(): boolean {
return this.started && this.forwarder.getListenerPorts().length > 0;
}
// mintSidecarToken / verifySidecarToken / spawnSidecar / stopSidecar /
// isSidecarRunning / handleSidecarResolve / sendSidecar /
// attachSidecarSocket are gone: the in-process MeshForwarder replaces
// the entire sidecar layer. Routing decisions happen via direct
// MeshService calls — no JWT minting, no separate container, no control
// WebSocket. See `docs/internal/architecture/mesh.md` for the new flow.
}
export type MeshErrorCode =