mirror of
https://github.com/Studio-Saelix/sencho.git
synced 2026-08-31 12:48:10 +00:00
fix(mesh): address audit follow-ups (early-data, recompose, admiral gate, error codes, docs) (#1126)
* fix(mesh): buffer early TcpData on reverse-relay path
The reverse-relay code dropped the local-shaped reservation and any
TcpData frames buffered in it before acceptReverseRelay had wired up
the target stream. A peer that sent a request body immediately after
tcp_open_reverse lost those bytes when the target lived on a third
node, since reverse_local buffers them but reverse_relay did not.
Carry the reservation through acceptReverseRelay: transplant its
pendingData and pendingBytes into the new reverse_relay state, gate
TcpData writes on targetOpen, and flush buffered frames into the
target stream before sending tcp_open_ack. Mirrors the reverse_local
pattern. Regression test fires tcp_open_reverse plus an immediate
TcpData while ensureBridge is in flight and verifies the bytes
arrive intact, in order, before the ack.
* fix(mesh): recompose affected stacks when node-level mesh is disabled
disableForNode used to clear DB rows and override files but leave the
running containers attached to sencho_mesh with stale /etc/hosts
alias entries until an operator redeployed every stack by hand.
Mirror optOutStack: after the existing alias/forwarder cleanup, call
regenerateOverridesAcrossFleet, cascadeRecomposeAcrossFleet, and
triggerRedeploy for each previously meshed stack on the disabled
node so containers detach from sencho_mesh and shed the alias
entries they owned. The disabled node's mesh_stacks rows are deleted
before the cascade so listMeshStacks returns the right set with no
skip tuple required. Route threads the actor through actorFor(req)
for parity with optInStack/optOutStack. Tests cover the redeploy
fan-out, the cascade no-skip-tuple invariant, and the default actor
fallback for non-route callers.
* fix(mesh): require Admiral on the WS proxy-tunnel upgrade
HTTP mesh routes in routes/mesh.ts all enforce requireAdmiral, but
the /api/mesh/proxy-tunnel WS upgrade accepted any node_proxy or
full-admin api_token regardless of the receiver's license. A node
downgraded from Admiral kept serving mesh data-plane traffic to a
sibling central while refusing every mesh management call.
Read the receiver's local LicenseService at the upgrade and 403 when
the tier is not paid+admiral. The check sits after the existing
credential gate and uses LicenseService directly rather than
effectiveTier (which trusts forwarded proxy headers); a remote peer
dialing in cannot be trusted to assert our entitlement. Dialer and
node_proxy token format are unchanged. Three regression tests cover
community-tier node_proxy, skipper-tier node_proxy, and
community-tier full-admin api_token all rejected with 403.
* fix(mesh): handle no_target alongside push_failed in inspectStackServices
proxyFetch throws MeshError('no_target') when getProxyTarget returns
null (pilot tunnel offline, proxy bridge unreachable), but the
inspectStackServices catch branch only matched push_failed. Offline
remotes fell through to the generic 'remote unreachable' error log,
which the Routing tab surfaces as an unexpected fault.
Match both error codes and emit the operator-friendly warn message
that names the unreachable node and the error code. Regression test
spies on console.warn/console.error to pin the branch.
* docs(mesh): align env defaults and forwarder comments with current architecture
SENCHO_MESH_PROXY_TUNNEL_IDLE_MS in .env.example carried the old
five-minute idle-close value (=300000), but the code default is
DEFAULT_IDLE_TTL_MS=0 (persistent tunnel). Copying the example
silently reintroduced the idle-close behavior the dialer removed.
MeshForwarder.ts's leading docblock and inline listen comment still
described host-network mode plus extra_hosts: host-gateway as
required for forwarder reachability. Sencho runs in standard bridge
mode and attaches to the shared sencho_mesh network at a stable IP;
meshed user containers reach the forwarder by that IP directly.
Flip the env default to 0, rewrite the env comment to describe the
persistent behavior and the opt-in for idle teardown, and rewrite
both forwarder comments to match the bridge-network reality.
* fix(mesh): trust forwarded tier on proxy-tunnel WS and remove remote overrides on disable
Admiral entitlement on the WS data plane now follows the same trust model
as the HTTP mesh routes: the central asserts its tier via x-sencho-tier
and x-sencho-variant on the WS handshake, and the receiver trusts those
headers only when the upgrade carries a node_proxy credential. When no
headers are present or the credential is a full-admin api_token, the
receiver falls back to its own local license. Without this an Admiral
central could be rejected by a Community remote and a Community central
could dial a locally-Admiral remote.
disableForNode now routes through removeOverrideFromNode so override
files pushed earlier via applyLocalOverride are removed on remote nodes
via DELETE /api/mesh/local-override/:stack. Sequential awaits are wrapped
in Promise.allSettled to match regenerateOverridesForNode's parallel
push pattern.
Other changes:
- Cover the post-state-swap buffering window in the reverse-relay test
(TcpData arriving after openTcpStream returns but before target open).
- Refresh stale MeshService comments that still referenced the removed
sidecar layer and host-network listener model.
* test(mesh): pin removeOverrideFromNode remote HTTP shape
The disableForNode regression test mocks removeOverrideFromNode itself,
so a regression inside the helper would not be caught. Add a narrow
contract test that spies on global fetch and asserts the request shape:
DELETE /api/mesh/local-override/:stack against the resolved proxy
target, with Authorization Bearer plus the x-sencho-tier and
x-sencho-variant headers. Also covers the encodeURIComponent path and
the swallowed-network-error behavior the disable cascade relies on.
* test(mesh): use createTestApiToken helper in proxy-tunnel api_token gate test
The full-admin api_token branch of the WS Admiral gate test inlined the
canonical generateApiToken + sha256 + addApiToken triple that already
lives in the createTestApiToken helper. Switching to the helper removes
the duplicated insertion logic and aligns this test with the helper used
by the other api_token call sites.
This commit is contained in:
@@ -242,9 +242,8 @@ interface ActiveStreamRecord {
|
||||
|
||||
/**
|
||||
* Sencho Mesh orchestrator. Owns:
|
||||
* - in-process TCP forwarder (`MeshForwarder`) that binds host-network
|
||||
* listeners on alias ports. Replaces the prior separate sidecar
|
||||
* container; one container per node now.
|
||||
* - in-process TCP forwarder (`MeshForwarder`) that binds per-alias
|
||||
* listeners on Sencho's `sencho_mesh` bridge-network IP.
|
||||
* - opt-in / opt-out persistence and cascading override regeneration
|
||||
* - global alias aggregation (across the fleet via the existing HTTP
|
||||
* proxy chain, see `inspectStackServices`)
|
||||
@@ -258,8 +257,6 @@ interface ActiveStreamRecord {
|
||||
* Docker bridge network. Meshed user services join `sencho_mesh` so
|
||||
* that IP is reachable from inside their containers without any
|
||||
* host-firewall coordination.
|
||||
* - cross-node mesh routing is central → pilot in this phase. Pilot →
|
||||
* central and pilot ↔ pilot via central relay land in Phase B.
|
||||
*/
|
||||
export class MeshService extends EventEmitter implements MeshForwarderHost {
|
||||
private static instance: MeshService;
|
||||
@@ -904,15 +901,38 @@ export class MeshService extends EventEmitter implements MeshForwarderHost {
|
||||
}
|
||||
}
|
||||
|
||||
public async disableForNode(nodeId: number): Promise<void> {
|
||||
public async disableForNode(
|
||||
nodeId: number,
|
||||
actor: string = 'system:mesh.disable',
|
||||
): Promise<void> {
|
||||
DatabaseService.getInstance().setNodeMeshEnabled(nodeId, false);
|
||||
const stacks = DatabaseService.getInstance().listMeshStacks(nodeId);
|
||||
for (const s of stacks) {
|
||||
DatabaseService.getInstance().deleteMeshStack(nodeId, s.stack_name);
|
||||
await this.removeStackOverride(nodeId, s.stack_name);
|
||||
}
|
||||
// Dispatch DELETE /api/mesh/local-override/:stack for remote nodes
|
||||
// (pilot or proxy) so the override file pushed earlier via
|
||||
// applyLocalOverride is removed; falls back to local deletion for
|
||||
// local nodes. Parallelize per the regenerateOverridesForNode
|
||||
// rationale: each remote call is its own HTTP round-trip, so
|
||||
// awaiting sequentially turns N stacks into N serialised DELETEs.
|
||||
// `allSettled` so a single failure does not abort the others
|
||||
// (removeOverrideFromNode already swallows errors internally).
|
||||
await Promise.allSettled(
|
||||
stacks.map((s) => this.removeOverrideFromNode(nodeId, s.stack_name)),
|
||||
);
|
||||
await this.refreshAliasCache();
|
||||
await this.syncForwarderListeners();
|
||||
// Mirror optOutStack: regenerate every remaining node's override
|
||||
// without the dropped aliases, recompose the rest of the fleet so
|
||||
// their containers shed the stale extra_hosts, and redeploy the
|
||||
// disabled node's own stacks so their containers detach from the
|
||||
// sencho_mesh network and lose the alias entries they owned.
|
||||
await this.regenerateOverridesAcrossFleet();
|
||||
this.cascadeRecomposeAcrossFleet(undefined, undefined, actor);
|
||||
for (const s of stacks) {
|
||||
this.triggerRedeploy(nodeId, s.stack_name, actor);
|
||||
}
|
||||
this.logActivity({
|
||||
source: 'mesh', level: 'info', type: 'mesh.disable',
|
||||
nodeId, message: `mesh disabled on node ${nodeId}`,
|
||||
@@ -1415,11 +1435,13 @@ export class MeshService extends EventEmitter implements MeshForwarderHost {
|
||||
const body = await res.json() as { services?: Array<{ service: string; ports: number[] }> };
|
||||
return body.services ?? [];
|
||||
} catch (err) {
|
||||
// proxyFetch throws MeshError('push_failed') when getProxyTarget
|
||||
// returns null (e.g. pilot tunnel offline). Treat the same as a
|
||||
// non-OK response: empty list.
|
||||
if (err instanceof MeshError && err.code === 'push_failed') {
|
||||
console.warn(`[MeshService] inspectStackServices: no proxy target for node ${nodeId} (${sanitizeForLog(node.name)})`);
|
||||
// proxyFetch throws MeshError('no_target') when getProxyTarget
|
||||
// returns null (pilot tunnel offline, proxy bridge unreachable)
|
||||
// and MeshError('push_failed') on a non-OK HTTP response. Treat
|
||||
// both as a soft "no services to report" so the Routing tab does
|
||||
// not surface a stack trace for a node that is simply offline.
|
||||
if (err instanceof MeshError && (err.code === 'no_target' || err.code === 'push_failed')) {
|
||||
console.warn(`[MeshService] inspectStackServices: unreachable node ${nodeId} (${sanitizeForLog(node.name)}): ${err.code}`);
|
||||
return [];
|
||||
}
|
||||
console.error('[MeshService] inspectStackServices remote unreachable:', sanitizeForLog((err as Error).message));
|
||||
@@ -1690,12 +1712,12 @@ export class MeshService extends EventEmitter implements MeshForwarderHost {
|
||||
|
||||
/**
|
||||
* Same-node forward: dial the target container's bridge IP directly.
|
||||
* Sencho runs in `network_mode: host` so it sees the docker bridge
|
||||
* networks and can reach container IPs without going through any
|
||||
* host-port publish. Looks up the container by Compose's
|
||||
* `<project>-<service>-<index>` naming convention; falls back to a
|
||||
* label-filtered listContainers if the conventional name is absent
|
||||
* (e.g. when the operator overrode the project name).
|
||||
* Sencho joins the `sencho_mesh` Docker bridge network alongside the
|
||||
* meshed user containers, so it can reach their bridge IPs without
|
||||
* going through any host-port publish. Looks up the container by
|
||||
* Compose's `<project>-<service>-<index>` naming convention; falls
|
||||
* back to a label-filtered listContainers if the conventional name
|
||||
* is absent (e.g. when the operator overrode the project name).
|
||||
*/
|
||||
private async openSameNode(target: MeshTarget, src: net.Socket): Promise<void> {
|
||||
const ip = await this.resolveContainerIp(target);
|
||||
@@ -2244,13 +2266,6 @@ export class MeshService extends EventEmitter implements MeshForwarderHost {
|
||||
private isLocalForwarderActive(): boolean {
|
||||
return this.started && this.forwarder.getListenerPorts().length > 0;
|
||||
}
|
||||
|
||||
// mintSidecarToken / verifySidecarToken / spawnSidecar / stopSidecar /
|
||||
// isSidecarRunning / handleSidecarResolve / sendSidecar /
|
||||
// attachSidecarSocket are gone: the in-process MeshForwarder replaces
|
||||
// the entire sidecar layer. Routing decisions happen via direct
|
||||
// MeshService calls — no JWT minting, no separate container, no control
|
||||
// WebSocket. See `docs/internal/architecture/mesh.md` for the new flow.
|
||||
}
|
||||
|
||||
export type MeshErrorCode =
|
||||
|
||||
Reference in New Issue
Block a user