Files
sencho/backend/src/routes/mesh.ts
T
Anso 982c7830b1 fix(mesh): address audit follow-ups (early-data, recompose, admiral gate, error codes, docs) (#1126)
* fix(mesh): buffer early TcpData on reverse-relay path

The reverse-relay code dropped the local-shaped reservation and any
TcpData frames buffered in it before acceptReverseRelay had wired up
the target stream. A peer that sent a request body immediately after
tcp_open_reverse lost those bytes when the target lived on a third
node, since reverse_local buffers them but reverse_relay did not.

Carry the reservation through acceptReverseRelay: transplant its
pendingData and pendingBytes into the new reverse_relay state, gate
TcpData writes on targetOpen, and flush buffered frames into the
target stream before sending tcp_open_ack. Mirrors the reverse_local
pattern. Regression test fires tcp_open_reverse plus an immediate
TcpData while ensureBridge is in flight and verifies the bytes
arrive intact, in order, before the ack.

* fix(mesh): recompose affected stacks when node-level mesh is disabled

disableForNode used to clear DB rows and override files but leave the
running containers attached to sencho_mesh with stale /etc/hosts
alias entries until an operator redeployed every stack by hand.

Mirror optOutStack: after the existing alias/forwarder cleanup, call
regenerateOverridesAcrossFleet, cascadeRecomposeAcrossFleet, and
triggerRedeploy for each previously meshed stack on the disabled
node so containers detach from sencho_mesh and shed the alias
entries they owned. The disabled node's mesh_stacks rows are deleted
before the cascade so listMeshStacks returns the right set with no
skip tuple required. Route threads the actor through actorFor(req)
for parity with optInStack/optOutStack. Tests cover the redeploy
fan-out, the cascade no-skip-tuple invariant, and the default actor
fallback for non-route callers.

* fix(mesh): require Admiral on the WS proxy-tunnel upgrade

HTTP mesh routes in routes/mesh.ts all enforce requireAdmiral, but
the /api/mesh/proxy-tunnel WS upgrade accepted any node_proxy or
full-admin api_token regardless of the receiver's license. A node
downgraded from Admiral kept serving mesh data-plane traffic to a
sibling central while refusing every mesh management call.

Read the receiver's local LicenseService at the upgrade and 403 when
the tier is not paid+admiral. The check sits after the existing
credential gate and uses LicenseService directly rather than
effectiveTier (which trusts forwarded proxy headers); a remote peer
dialing in cannot be trusted to assert our entitlement. Dialer and
node_proxy token format are unchanged. Three regression tests cover
community-tier node_proxy, skipper-tier node_proxy, and
community-tier full-admin api_token all rejected with 403.

* fix(mesh): handle no_target alongside push_failed in inspectStackServices

proxyFetch throws MeshError('no_target') when getProxyTarget returns
null (pilot tunnel offline, proxy bridge unreachable), but the
inspectStackServices catch branch only matched push_failed. Offline
remotes fell through to the generic 'remote unreachable' error log,
which the Routing tab surfaces as an unexpected fault.

Match both error codes and emit the operator-friendly warn message
that names the unreachable node and the error code. Regression test
spies on console.warn/console.error to pin the branch.

* docs(mesh): align env defaults and forwarder comments with current architecture

SENCHO_MESH_PROXY_TUNNEL_IDLE_MS in .env.example carried the old
five-minute idle-close value (=300000), but the code default is
DEFAULT_IDLE_TTL_MS=0 (persistent tunnel). Copying the example
silently reintroduced the idle-close behavior the dialer removed.

MeshForwarder.ts's leading docblock and inline listen comment still
described host-network mode plus extra_hosts: host-gateway as
required for forwarder reachability. Sencho runs in standard bridge
mode and attaches to the shared sencho_mesh network at a stable IP;
meshed user containers reach the forwarder by that IP directly.

Flip the env default to 0, rewrite the env comment to describe the
persistent behavior and the opt-in for idle teardown, and rewrite
both forwarder comments to match the bridge-network reality.

* fix(mesh): trust forwarded tier on proxy-tunnel WS and remove remote overrides on disable

Admiral entitlement on the WS data plane now follows the same trust model
as the HTTP mesh routes: the central asserts its tier via x-sencho-tier
and x-sencho-variant on the WS handshake, and the receiver trusts those
headers only when the upgrade carries a node_proxy credential. When no
headers are present or the credential is a full-admin api_token, the
receiver falls back to its own local license. Without this an Admiral
central could be rejected by a Community remote and a Community central
could dial a locally-Admiral remote.

disableForNode now routes through removeOverrideFromNode so override
files pushed earlier via applyLocalOverride are removed on remote nodes
via DELETE /api/mesh/local-override/:stack. Sequential awaits are wrapped
in Promise.allSettled to match regenerateOverridesForNode's parallel
push pattern.

Other changes:
- Cover the post-state-swap buffering window in the reverse-relay test
  (TcpData arriving after openTcpStream returns but before target open).
- Refresh stale MeshService comments that still referenced the removed
  sidecar layer and host-network listener model.

* test(mesh): pin removeOverrideFromNode remote HTTP shape

The disableForNode regression test mocks removeOverrideFromNode itself,
so a regression inside the helper would not be caught. Add a narrow
contract test that spies on global fetch and asserts the request shape:
DELETE /api/mesh/local-override/:stack against the resolved proxy
target, with Authorization Bearer plus the x-sencho-tier and
x-sencho-variant headers. Also covers the encodeURIComponent path and
the swallowed-network-error behavior the disable cascade relies on.

* test(mesh): use createTestApiToken helper in proxy-tunnel api_token gate test

The full-admin api_token branch of the WS Admiral gate test inlined the
canonical generateApiToken + sha256 + addApiToken triple that already
lives in the createTestApiToken helper. Switching to the helper removes
the duplicated insertion logic and aligns this test with the helper used
by the other api_token call sites.
2026-05-20 12:53:29 -04:00

351 lines
16 KiB
TypeScript

import { Router, type Request, type Response } from 'express';
import { DatabaseService } from '../services/DatabaseService';
import { NodeRegistry } from '../services/NodeRegistry';
import { MeshError, MeshService, type MeshGlobalAlias, type MeshRegenSummary } from '../services/MeshService';
import { requireAdmin, requireAdmiral } from '../middleware/tierGates';
import { sanitizeForLog } from '../utils/safeLog';
import { isValidStackName } from '../utils/validation';
export const meshRouter = Router();
function actorFor(req: Request): string {
const user = (req as Request & { user?: { username?: string } }).user;
return user?.username || 'system';
}
meshRouter.get('/status', async (_req: Request, res: Response): Promise<void> => {
if (!requireAdmiral(_req, res)) return;
try {
const mesh = MeshService.getInstance();
const status = await mesh.getStatus();
res.json({ nodes: status, localDataPlane: mesh.getDataPlaneStatus() });
} catch (err) {
console.warn('[mesh] /status failed:', sanitizeForLog((err as Error).message));
res.status(500).json({ error: 'Failed to load mesh status' });
}
});
/**
* Operator-triggered rerun of the boot-time override regeneration. Walks
* every `mesh_stacks` row across the fleet and re-pushes each override to
* its owning node. Useful when a remote node was offline at central boot
* and the override files there are stale; previously the only recovery
* path was opt-out + opt-in for every meshed stack on that node.
*/
meshRouter.post('/regen-overrides', async (req: Request, res: Response): Promise<void> => {
if (!requireAdmiral(req, res)) return;
if (!requireAdmin(req, res)) return;
const actor = actorFor(req);
let summary: MeshRegenSummary | null = null;
let outcome: 'success' | 'skipped' | 'partial' | 'error' = 'error';
try {
summary = await MeshService.getInstance().regenerateAllOverrides();
outcome = summary.skipped ? 'skipped' : (summary.failures.length === 0 ? 'success' : 'partial');
res.json(summary);
} catch (err) {
outcome = 'error';
console.warn('[mesh] /regen-overrides failed:', sanitizeForLog((err as Error).message));
res.status(500).json({ error: 'Failed to regenerate mesh overrides' });
} finally {
try {
DatabaseService.getInstance().insertAuditLog({
timestamp: Date.now(),
username: actor,
method: 'POST',
path: req.path,
status_code: res.statusCode,
node_id: null,
ip_address: req.ip ?? 'unknown',
summary: summary
? `Mesh override regen ${outcome}: ${summary.regenerated} regenerated, ${summary.failures.length} failed`
: `Mesh override regen ${outcome}`,
});
} catch (auditErr) {
console.error('[mesh] Audit log insert failed:', auditErr);
}
}
});
meshRouter.post('/nodes/:nodeId/enable', async (req: Request, res: Response): Promise<void> => {
if (!requireAdmiral(req, res)) return;
if (!requireAdmin(req, res)) return;
const nodeId = Number.parseInt(req.params.nodeId as string, 10);
if (!Number.isFinite(nodeId)) { res.status(400).json({ error: 'Invalid node id' }); return; }
try {
await MeshService.getInstance().enableForNode(nodeId);
res.json({ ok: true });
} catch (err) {
res.status(500).json({ error: (err as Error).message });
}
});
meshRouter.post('/nodes/:nodeId/disable', async (req: Request, res: Response): Promise<void> => {
if (!requireAdmiral(req, res)) return;
if (!requireAdmin(req, res)) return;
const nodeId = Number.parseInt(req.params.nodeId as string, 10);
if (!Number.isFinite(nodeId)) { res.status(400).json({ error: 'Invalid node id' }); return; }
try {
await MeshService.getInstance().disableForNode(nodeId, actorFor(req));
res.json({ ok: true });
} catch (err) {
res.status(500).json({ error: (err as Error).message });
}
});
/**
* Returns the LOCAL Docker daemon's services for a stack with their listening
* ports. Always queries this Sencho instance's own Dockerode regardless of
* `x-node-id`. Central calls this endpoint against each remote node via the
* existing proxy chain (`NodeRegistry.getProxyTarget`) so it can build the
* cross-fleet alias cache without violating the local-only Dockerode rule.
*/
meshRouter.get('/local-services/:stackName', async (req: Request, res: Response): Promise<void> => {
if (!requireAdmiral(req, res)) return;
const stackName = req.params.stackName as string;
if (!isValidStackName(stackName)) { res.status(400).json({ error: 'Invalid stack name' }); return; }
try {
const services = await MeshService.getInstance().inspectLocalStackServices(stackName);
res.json({ services });
} catch (err) {
console.warn('[mesh] /local-services failed:', sanitizeForLog((err as Error).message));
res.status(500).json({ error: 'Failed to list local services' });
}
});
/**
* Returns the LOCAL Sencho's compose stacks. Always queries this
* instance's own filesystem regardless of `x-node-id`. Central calls this
* endpoint against each remote node via the existing proxy chain
* (`NodeRegistry.getProxyTarget`) so the mesh opt-in sheet can show the
* stacks deployed on the remote pilot rather than central's own list.
*/
meshRouter.get('/local-stacks', async (req: Request, res: Response): Promise<void> => {
if (!requireAdmiral(req, res)) return;
try {
const stacks = await MeshService.getInstance().listLocalStacks();
res.json({ stacks });
} catch (err) {
console.warn('[mesh] /local-stacks failed:', sanitizeForLog((err as Error).message));
res.status(500).json({ error: 'Failed to list local stacks' });
}
});
const MAX_ALIASES_PER_PUSH = 1024;
function parsePortAlias(entry: unknown): MeshGlobalAlias | null {
const e = entry as Record<string, unknown>;
const { host, nodeId, nodeName, stackName, serviceName, port } = e ?? {};
if (
typeof host !== 'string' || host.length === 0 || host.length > 253 ||
typeof nodeId !== 'number' ||
typeof nodeName !== 'string' || nodeName.length === 0 ||
typeof stackName !== 'string' || stackName.length === 0 ||
typeof serviceName !== 'string' || serviceName.length === 0 ||
typeof port !== 'number' || !Number.isInteger(port) || port < 1 || port > 65535
) return null;
return { host, nodeId, nodeName, stackName, serviceName, port };
}
/**
* Accepts a fleet-wide alias list from central and writes a mesh override
* for the named stack onto THIS Sencho's local DATA_DIR. The pilot looks
* up its own service names and uses its own static IP on `sencho_mesh`,
* so alias hostnames in user containers always resolve to the LOCAL
* Sencho IP on the deploying node. Always writes against the LOCAL
* Sencho's default node id.
*/
meshRouter.put('/local-override/:stackName', async (req: Request, res: Response): Promise<void> => {
if (!requireAdmiral(req, res)) return;
const stackName = req.params.stackName as string;
if (!isValidStackName(stackName)) { res.status(400).json({ error: 'Invalid stack name' }); return; }
const body = req.body as { aliases?: unknown; portAliases?: unknown };
if (!Array.isArray(body?.aliases)) { res.status(400).json({ error: 'Missing aliases array in body' }); return; }
if (body.aliases.length > MAX_ALIASES_PER_PUSH) {
res.status(413).json({ error: `Alias list exceeds ${MAX_ALIASES_PER_PUSH} entries` });
return;
}
const aliases: { host: string }[] = [];
for (const entry of body.aliases) {
const host = (entry as { host?: unknown } | null | undefined)?.host;
if (typeof host !== 'string' || host.length === 0 || host.length > 253) {
// 253 octets is the DNS hostname ceiling. Defensive against a
// malicious or buggy central sending a multi-KB host string.
res.status(400).json({ error: 'Invalid alias entry' });
return;
}
aliases.push({ host });
}
const portAliases: MeshGlobalAlias[] = [];
if (Array.isArray(body?.portAliases)) {
if (body.portAliases.length > MAX_ALIASES_PER_PUSH) {
res.status(413).json({ error: `portAliases list exceeds ${MAX_ALIASES_PER_PUSH} entries` });
return;
}
for (const entry of body.portAliases) {
const parsed = parsePortAlias(entry);
if (!parsed) { res.status(400).json({ error: 'Invalid portAliases entry' }); return; }
portAliases.push(parsed);
}
}
try {
const written = await MeshService.getInstance().applyLocalOverride(stackName, aliases, portAliases);
if (!written) { res.status(400).json({ error: 'Refused to write override (path validation failed)' }); return; }
res.json({ ok: true, path: written });
} catch (err) {
if (err instanceof MeshError && err.code === 'push_failed') {
res.status(503).json({ error: err.message, code: err.code });
return;
}
console.warn('[mesh] /local-override failed:', sanitizeForLog((err as Error).message));
res.status(500).json({ error: 'Failed to write local override' });
}
});
/**
* Delete a previously written local override. Mirror of the PUT endpoint;
* called by central when a stack is opted out so stale overrides do not
* linger on the deploying node.
*/
meshRouter.delete('/local-override/:stackName', async (req: Request, res: Response): Promise<void> => {
if (!requireAdmiral(req, res)) return;
const stackName = req.params.stackName as string;
if (!isValidStackName(stackName)) { res.status(400).json({ error: 'Invalid stack name' }); return; }
try {
await MeshService.getInstance().removeLocalOverride(stackName);
res.json({ ok: true });
} catch (err) {
console.warn('[mesh] DELETE /local-override failed:', sanitizeForLog((err as Error).message));
res.status(500).json({ error: 'Failed to remove local override' });
}
});
meshRouter.get('/nodes/:nodeId/stacks', async (req: Request, res: Response): Promise<void> => {
if (!requireAdmiral(req, res)) return;
const nodeId = Number.parseInt(req.params.nodeId as string, 10);
if (!Number.isFinite(nodeId)) { res.status(400).json({ error: 'Invalid node id' }); return; }
try {
const db = DatabaseService.getInstance();
const optedIn = new Set(db.listMeshStacks(nodeId).map((s) => s.stack_name));
const stacks = await MeshService.getInstance().listStacksOnNode(nodeId);
res.json({
stacks: stacks.map((stackName) => ({
name: stackName,
optedIn: optedIn.has(stackName),
})),
});
} catch (err) {
console.warn('[mesh] list stacks failed:', sanitizeForLog((err as Error).message));
res.status(500).json({ error: 'Failed to list stacks' });
}
});
meshRouter.post('/nodes/:nodeId/stacks/:stackName/opt-in', async (req: Request, res: Response): Promise<void> => {
if (!requireAdmiral(req, res)) return;
if (!requireAdmin(req, res)) return;
const nodeId = Number.parseInt(req.params.nodeId as string, 10);
const stackName = req.params.stackName as string;
if (!Number.isFinite(nodeId) || !stackName) { res.status(400).json({ error: 'Invalid params' }); return; }
try {
await MeshService.getInstance().optInStack(nodeId, stackName, actorFor(req));
res.json({ ok: true });
} catch (err) {
if (err instanceof MeshError && err.code === 'port_collision') {
res.status(409).json({ error: err.message, code: err.code });
return;
}
if (err instanceof MeshError && err.code === 'push_failed') {
res.status(503).json({ error: err.message, code: err.code });
return;
}
if (err instanceof MeshError) {
res.status(400).json({ error: err.message, code: err.code });
return;
}
console.warn('[mesh] opt-in failed:', sanitizeForLog((err as Error).message));
res.status(500).json({ error: 'Opt-in failed' });
}
});
meshRouter.post('/nodes/:nodeId/stacks/:stackName/opt-out', async (req: Request, res: Response): Promise<void> => {
if (!requireAdmiral(req, res)) return;
if (!requireAdmin(req, res)) return;
const nodeId = Number.parseInt(req.params.nodeId as string, 10);
const stackName = req.params.stackName as string;
if (!Number.isFinite(nodeId) || !stackName) { res.status(400).json({ error: 'Invalid params' }); return; }
try {
await MeshService.getInstance().optOutStack(nodeId, stackName, actorFor(req));
res.json({ ok: true });
} catch (err) {
console.warn('[mesh] opt-out failed:', sanitizeForLog((err as Error).message));
res.status(500).json({ error: 'Opt-out failed' });
}
});
meshRouter.get('/aliases', async (req: Request, res: Response): Promise<void> => {
if (!requireAdmiral(req, res)) return;
try {
const aliases = await MeshService.getInstance().listAliases();
res.json({ aliases });
} catch (err) {
console.warn('[mesh] /aliases failed:', sanitizeForLog((err as Error).message));
res.status(500).json({ error: 'Failed to list aliases' });
}
});
meshRouter.get('/aliases/:alias/diagnostic', async (req: Request, res: Response): Promise<void> => {
if (!requireAdmiral(req, res)) return;
try {
const diag = await MeshService.getInstance().getRouteDiagnostic(req.params.alias as string);
res.json(diag);
} catch (err) {
res.status(500).json({ error: (err as Error).message });
}
});
meshRouter.post('/aliases/:alias/test', async (req: Request, res: Response): Promise<void> => {
if (!requireAdmiral(req, res)) return;
try {
const sourceNodeId = NodeRegistry.getInstance().getDefaultNodeId();
const result = await MeshService.getInstance().testUpstream(req.params.alias as string, sourceNodeId);
res.json(result);
} catch (err) {
res.status(500).json({ error: (err as Error).message });
}
});
meshRouter.get('/nodes/:nodeId/diagnostic', async (req: Request, res: Response): Promise<void> => {
if (!requireAdmiral(req, res)) return;
const nodeId = Number.parseInt(req.params.nodeId as string, 10);
if (!Number.isFinite(nodeId)) { res.status(400).json({ error: 'Invalid node id' }); return; }
try {
const diag = await MeshService.getInstance().getNodeDiagnostic(nodeId);
res.json(diag);
} catch (err) {
res.status(500).json({ error: (err as Error).message });
}
});
meshRouter.get('/activity', (req: Request, res: Response): void => {
if (!requireAdmiral(req, res)) return;
const alias = typeof req.query.alias === 'string' ? req.query.alias : undefined;
const source = typeof req.query.source === 'string' ? (req.query.source as 'pilot' | 'mesh') : undefined;
const level = typeof req.query.level === 'string' ? (req.query.level as 'info' | 'warn' | 'error') : undefined;
const limit = typeof req.query.limit === 'string' ? Number.parseInt(req.query.limit, 10) : 200;
const events = MeshService.getInstance().getActivity({ alias, source, level, limit });
res.json({ events });
});
meshRouter.get('/activity/stream', (req: Request, res: Response): void => {
if (!requireAdmiral(req, res)) return;
res.setHeader('Content-Type', 'text/event-stream');
res.setHeader('Cache-Control', 'no-cache, no-transform');
res.setHeader('Connection', 'keep-alive');
res.flushHeaders?.();
const send = (event: object) => {
try { res.write(`data: ${JSON.stringify(event)}\n\n`); } catch { /* ignore */ }
};
const unsubscribe = MeshService.getInstance().subscribeActivity(send);
req.on('close', () => unsubscribe());
});