feat(fleet): sencho mesh in traffic and routing tab (#858)

* feat(fleet): sencho mesh in traffic and routing tab

Lights up Sencho Mesh: cross-node container forwarding rendered as if the
container next to you were on localhost. Builds on the dormant TCP frame
plumbing from the prior PR (pilot tunnel TCP frames + sencho-mesh sidecar
package) and exposes the Admiral-only orchestrator surface.

Backend
- New mesh_stacks table (per-node opt-ins) + nodes.mesh_enabled column
  via DatabaseService.migrateMeshTables.
- MeshService singleton: sidecar lifecycle via Dockerode, opt-in/out with
  cascading override regeneration, request-based resolver from sidecar
  control WS, cross-node TCP forwarding via PilotTunnelManager (same-node
  fast path included), in-memory 1000-event activity ring buffer with
  durable mirror to audit_log for state-change events, per-node and
  per-route diagnostics, and the Test upstream probe.
- MeshComposeOverride: pure YAML generator that injects extra_hosts using
  host-gateway. The user's docker-compose.yml is never mutated; overrides
  live under DATA_DIR/mesh/overrides.
- ComposeService deploy/update splice the override file when the stack
  is opted in; non-mesh stacks behave identically to today.
- Pilot agent resolveMeshTarget consults the local mesh_stacks table
  (defense in depth) and resolves Compose containers via Dockerode.
- /api/mesh router with 13 Admiral-gated endpoints covering status,
  enable/disable, stack opt-in/out, alias listing, per-route diagnostic,
  Test upstream probe, per-node diagnostic, sidecar restart, activity
  log paginated and SSE.
- meshControl WS slot at /api/mesh/control validates the mesh_sidecar
  JWT minted by MeshService; dispatched as upgrade slot 2 (canonical
  order preserved).

Frontend
- New Traffic Routing tab in FleetView, gated by isAdmiral and wrapped
  in AdmiralGate. Tab uses the cyan brand glyph and italic-serif state
  typography from the audit.
- RoutingTab masthead with mesh activity drawer, per-node card grid
  with TogglePill, alias rows with five-state pill taxonomy
  (healthy / degraded / unreachable / tunnel-down / not-authorized),
  inline Test buttons.
- Four sheets: opt-in picker with port-collision inline error,
  per-route detail with diagnostic + filtered activity, per-node
  diagnostics with active streams + resolver cache + restart action,
  fleet-wide activity log with filters.
- meshRouteState helper centralizes pill-state mapping; pure-function
  tests cover all five states.

Docs
- User docs at /docs/features/sencho-mesh.mdx covering opt-in,
  troubleshooting, security model (4 guarantees + 4 explicit
  non-guarantees), and V1 limitations.
- Internal architecture and runbook pages.
- websocket-dispatch internal doc updated with the new slot.

* fix(mesh): validate stack name before path use; fix test DB lifecycle

Two surgical fixes against the prior PR.

Path-injection (CodeQL js/path-injection): MeshService.optInStack,
optOutStack, ensureStackOverride, and removeStackOverride now validate
stackName via isValidStackName from utils/validation, reject malicious
names at the API boundary, and additionally check isPathWithinBase on
the resolved override file path for defense in depth. The dataflow from
req.params.stackName to fs.writeFile no longer reaches an unsanitized
path expression.

Test DB lifecycle: mesh-service.test.ts used per-test setupTestDb /
cleanupTestDb, which deletes the temp dir while DatabaseService still
holds an open SQLite handle. On Linux CI this raises
SQLITE_READONLY_DBMOVED on the next prepare() because the inode has
been unlinked. Switched to file-scoped beforeAll/afterAll matching
agents-routes.test.ts, with a per-test beforeEach that truncates
mesh_stacks plus non-default nodes and resets the MeshService singleton
in-memory state. Adds a new test case asserting the path-traversal
rejection.

* fix(compose): use discovered compose filename instead of hardcoded docker-compose.yml

composeArgs() hardcoded `-f docker-compose.yml` for every deploy. Sencho
writes its canonical compose file as `compose.yaml`, so any stack created
via the UI failed to deploy with `open ...docker-compose.yml: no such
file or directory`.

When no mesh override applies, drop the explicit `-f` so docker compose's
built-in discovery resolves the actual filename. When an override exists,
look up the real base filename via FileSystemService.getComposeFilename()
and pass both files explicitly.

Also hoist the MeshService import to module top now that the dependency
is known to be acyclic, and revert the matching unit-test assertion.
This commit is contained in:
Anso
2026-05-01 01:50:53 -04:00
committed by GitHub
parent 6893ece898
commit 7663f4cd8b
27 changed files with 2667 additions and 13 deletions
@@ -0,0 +1,64 @@
import { describe, expect, it } from 'vitest';
import * as YAML from 'yaml';
import { buildAliasHosts, generateOverrideYaml } from '../services/MeshComposeOverride';
describe('generateOverrideYaml', () => {
it('emits services with extra_hosts pointing to host-gateway', () => {
const yaml = generateOverrideYaml({
services: ['web', 'cache'],
aliases: [
{ host: 'db.api.opsix.sencho' },
{ host: 'etl.worker.opsix.sencho' },
],
});
const parsed = YAML.parse(yaml) as Record<string, unknown>;
expect(parsed.networks).toBeUndefined();
const services = parsed.services as Record<string, { extra_hosts: string[] }>;
expect(Object.keys(services).sort()).toEqual(['cache', 'web']);
for (const svc of ['web', 'cache']) {
expect(services[svc].extra_hosts).toEqual([
'db.api.opsix.sencho:host-gateway',
'etl.worker.opsix.sencho:host-gateway',
]);
}
});
it('emits empty service stubs when no aliases exist yet', () => {
const yaml = generateOverrideYaml({
services: ['web'],
aliases: [],
});
const parsed = YAML.parse(yaml) as Record<string, unknown>;
const services = parsed.services as Record<string, { extra_hosts?: string[] }>;
expect(services.web.extra_hosts).toBeUndefined();
});
it('produces stable output regardless of input ordering', () => {
const a = generateOverrideYaml({
services: ['web', 'cache'],
aliases: [
{ host: 'b.x.y.sencho' },
{ host: 'a.x.y.sencho' },
],
});
const b = generateOverrideYaml({
services: ['cache', 'web'],
aliases: [
{ host: 'a.x.y.sencho' },
{ host: 'b.x.y.sencho' },
],
});
expect(a).toBe(b);
});
});
describe('buildAliasHosts', () => {
it('maps services to alias hostnames', () => {
const out = buildAliasHosts({
nodeName: 'opsix',
stackName: 'api',
services: [{ service: 'db', ports: [5432] }, { service: 'cache', ports: [6379] }],
});
expect(out).toEqual(['db.api.opsix.sencho', 'cache.api.opsix.sencho']);
});
});
+178
View File
@@ -0,0 +1,178 @@
import { afterAll, beforeAll, beforeEach, describe, expect, it, vi } from 'vitest';
import { setupTestDb, cleanupTestDb } from './helpers/setupTestDb';
let tmpDir: string;
let MeshService: typeof import('../services/MeshService').MeshService;
let DatabaseService: typeof import('../services/DatabaseService').DatabaseService;
beforeAll(async () => {
tmpDir = await setupTestDb();
({ MeshService } = await import('../services/MeshService'));
({ DatabaseService } = await import('../services/DatabaseService'));
});
afterAll(() => {
cleanupTestDb(tmpDir);
});
beforeEach(() => {
const db = DatabaseService.getInstance().getDb();
db.prepare('DELETE FROM mesh_stacks').run();
db.prepare('DELETE FROM nodes WHERE is_default = 0').run();
const svc = MeshService.getInstance() as unknown as {
aliasCache: Map<string, unknown>;
aliasByPort: Map<number, unknown>;
activity: unknown[];
activeStreams: Map<number, unknown>;
routeErrorMap: Map<string, unknown>;
routeLatencyMap: Map<string, unknown>;
};
svc.aliasCache = new Map();
svc.aliasByPort = new Map();
svc.activity = [];
svc.activeStreams = new Map();
svc.routeErrorMap = new Map();
svc.routeLatencyMap = new Map();
vi.restoreAllMocks();
});
describe('MeshService.optInStack', () => {
it('writes a mesh_stacks row and rejects duplicate ports', async () => {
const svc = MeshService.getInstance();
vi.spyOn(svc as unknown as { inspectStackServices: (n: number, s: string) => Promise<unknown> }, 'inspectStackServices')
.mockResolvedValue([{ service: 'db', ports: [5432] }]);
vi.spyOn(svc as unknown as { regenerateOverridesForNode: (n: number) => Promise<void> }, 'regenerateOverridesForNode')
.mockResolvedValue(undefined);
const db = DatabaseService.getInstance();
const localNodeId = db.getNodes()[0].id;
await svc.optInStack(localNodeId, 'api', 'tester');
expect(db.isMeshStackEnabled(localNodeId, 'api')).toBe(true);
await expect(svc.optInStack(localNodeId, 'shadow', 'tester'))
.rejects.toThrow(/port 5432 is already claimed/);
});
it('opt-out removes the row and the override', async () => {
const svc = MeshService.getInstance();
vi.spyOn(svc as unknown as { inspectStackServices: (n: number, s: string) => Promise<unknown> }, 'inspectStackServices')
.mockResolvedValue([{ service: 'db', ports: [5432] }]);
vi.spyOn(svc as unknown as { regenerateOverridesForNode: (n: number) => Promise<void> }, 'regenerateOverridesForNode')
.mockResolvedValue(undefined);
vi.spyOn(svc as unknown as { removeStackOverride: (n: number, s: string) => Promise<void> }, 'removeStackOverride')
.mockResolvedValue(undefined);
const db = DatabaseService.getInstance();
const localNodeId = db.getNodes()[0].id;
await svc.optInStack(localNodeId, 'api', 'tester');
await svc.optOutStack(localNodeId, 'api', 'tester');
expect(db.isMeshStackEnabled(localNodeId, 'api')).toBe(false);
});
it('rejects an invalid stack name (path traversal attempt)', async () => {
const svc = MeshService.getInstance();
const db = DatabaseService.getInstance();
const localNodeId = db.getNodes()[0].id;
await expect(svc.optInStack(localNodeId, '../../etc/passwd', 'tester'))
.rejects.toThrow(/invalid stack name/);
expect(db.isMeshStackEnabled(localNodeId, '../../etc/passwd')).toBe(false);
});
});
describe('MeshService activity log', () => {
it('keeps the most recent events under the 1000-cap', () => {
const svc = MeshService.getInstance();
for (let i = 0; i < 1100; i++) {
svc.logActivity({ source: 'mesh', level: 'info', type: 'opt_in', message: `evt-${i}` });
}
const all = svc.getActivity({ limit: 2000 });
expect(all.length).toBe(1000);
expect(all[0].message).toBe('evt-100');
expect(all[all.length - 1].message).toBe('evt-1099');
});
it('filters by alias / source / level', () => {
const svc = MeshService.getInstance();
svc.logActivity({ source: 'mesh', level: 'info', type: 'opt_in', alias: 'a.b.c.sencho', message: 'a' });
svc.logActivity({ source: 'pilot', level: 'error', type: 'tunnel.fail', alias: 'a.b.c.sencho', message: 'b' });
svc.logActivity({ source: 'sidecar', level: 'info', type: 'route.resolve.ok', alias: 'x.y.z.sencho', message: 'c' });
expect(svc.getActivity({ alias: 'a.b.c.sencho' }).length).toBe(2);
expect(svc.getActivity({ source: 'pilot' }).length).toBe(1);
expect(svc.getActivity({ level: 'error' }).length).toBe(1);
});
it('subscribeActivity fires for new events and unsubscribes cleanly', () => {
const svc = MeshService.getInstance();
const seen: string[] = [];
const unsubscribe = svc.subscribeActivity((e) => seen.push(e.message));
svc.logActivity({ source: 'mesh', level: 'info', type: 'mesh.enable', message: 'one' });
unsubscribe();
svc.logActivity({ source: 'mesh', level: 'info', type: 'mesh.disable', message: 'two' });
expect(seen).toEqual(['one']);
});
});
describe('MeshService.testUpstream tunnel-down path', () => {
it('returns ok:false where=pilot_tunnel when no tunnel is registered', async () => {
const svc = MeshService.getInstance();
const db = DatabaseService.getInstance();
const localNodeId = db.getNodes()[0].id;
const remoteNodeId = db.addNode({
name: 'opsix', type: 'remote', is_default: false,
compose_dir: '/tmp', api_url: 'https://opsix.example',
api_token: 'tok', mode: 'pilot_agent',
});
(svc as unknown as { aliasCache: Map<string, unknown> }).aliasCache = new Map([
['db.api.opsix.sencho', {
host: 'db.api.opsix.sencho',
nodeId: remoteNodeId,
nodeName: 'opsix',
stackName: 'api',
serviceName: 'db',
port: 5432,
}],
]);
db.insertMeshStack(remoteNodeId, 'api', 'tester');
const result = await svc.testUpstream('db.api.opsix.sencho', localNodeId);
expect(result.ok).toBe(false);
expect(result.where).toBe('pilot_tunnel');
expect(result.code).toBe('tunnel_down');
});
it('returns ok:false where=agent_resolve when target stack is not opted in', async () => {
const svc = MeshService.getInstance();
const db = DatabaseService.getInstance();
const localNodeId = db.getNodes()[0].id;
(svc as unknown as { aliasCache: Map<string, unknown> }).aliasCache = new Map([
['db.api.opsix.sencho', {
host: 'db.api.opsix.sencho',
nodeId: localNodeId,
nodeName: 'opsix',
stackName: 'api',
serviceName: 'db',
port: 5432,
}],
]);
const result = await svc.testUpstream('db.api.opsix.sencho', localNodeId);
expect(result.ok).toBe(false);
expect(result.where).toBe('agent_resolve');
expect(result.code).toBe('denied');
});
it('returns ok:false where=sidecar when alias is unknown', async () => {
const svc = MeshService.getInstance();
const localNodeId = DatabaseService.getInstance().getNodes()[0].id;
const result = await svc.testUpstream('nonexistent.sencho', localNodeId);
expect(result.ok).toBe(false);
expect(result.where).toBe('sidecar');
expect(result.code).toBe('no_route');
});
});
+4
View File
@@ -7,6 +7,7 @@ import { DockerEventManager } from '../services/DockerEventManager';
import { ImageUpdateService } from '../services/ImageUpdateService';
import { SchedulerService } from '../services/SchedulerService';
import { MfaService } from '../services/MfaService';
import { MeshService } from '../services/MeshService';
/**
* Wire graceful shutdown handlers. Docker sends SIGTERM when the container
@@ -39,6 +40,9 @@ export function installShutdownHandlers(server: Server): void {
try { MfaService.getInstance().stop(); } catch (e) {
console.warn('[Shutdown] MfaService cleanup failed:', (e as Error).message);
}
MeshService.getInstance().stop().catch((e) => {
console.warn('[Shutdown] MeshService cleanup failed:', (e as Error).message);
});
try { DatabaseService.getInstance().flushAuditLogBuffer(); } catch (e) {
console.warn('[Shutdown] Audit log flush failed:', (e as Error).message);
}
+4
View File
@@ -10,6 +10,7 @@ import TrivyService from '../services/TrivyService';
import { ImageUpdateService } from '../services/ImageUpdateService';
import { SchedulerService } from '../services/SchedulerService';
import { MfaService } from '../services/MfaService';
import { MeshService } from '../services/MeshService';
import { sweepStaleTempDirs as sweepStaleGitTempDirs } from '../services/GitSourceService';
import { PORT } from '../helpers/constants';
@@ -38,6 +39,9 @@ export async function startServer(server: Server): Promise<void> {
ImageUpdateService.getInstance().start();
SchedulerService.getInstance().start();
MfaService.getInstance().start();
MeshService.getInstance().start().catch((err) => {
console.warn('[Startup] MeshService start failed:', (err as Error).message);
});
// Async initializers are independent of each other; run in parallel
// so total boot time is the slowest one rather than the sum.
+2
View File
@@ -27,6 +27,7 @@ import { apiTokensRouter } from './routes/apiTokens';
import { auditLogRouter } from './routes/auditLog';
import { settingsRouter } from './routes/settings';
import { scheduledTasksRouter } from './routes/scheduledTasks';
import { meshRouter } from './routes/mesh';
import { agentsRouter } from './routes/agents';
import { metricsRouter } from './routes/metrics';
import { imageUpdatesRouter, autoUpdateRouter } from './routes/imageUpdates';
@@ -101,6 +102,7 @@ app.use('/api/git-sources', gitSourcesRouter);
app.use('/api/stacks', stackGitSourceRouter);
app.use('/api/settings', settingsRouter);
app.use('/api/scheduled-tasks', scheduledTasksRouter);
app.use('/api/mesh', meshRouter);
app.use('/api/agents', agentsRouter);
app.use('/api', metricsRouter);
app.use('/api/image-updates', imageUpdatesRouter);
+33 -6
View File
@@ -416,15 +416,42 @@ class PilotAgent {
}
/**
* Always returns ok:false in PR 1. Tests inject a real resolver via
* setMeshResolver; PR 2 will install the Dockerode-backed implementation.
* Resolves a mesh target by consulting the local mesh_stacks opt-in table
* and Compose container labels. Refuses if the target stack is not opted
* in on this node (defense-in-depth: the primary is trusted, but we also
* gate at the agent so a leaked tunnel token cannot reach unauthorized
* services).
*/
private async resolveMeshTarget(
_stack: string,
_service: string,
_port: number,
stack: string,
service: string,
port: number,
): Promise<MeshResolveResult> {
return { ok: false, err: 'mesh_not_enabled' };
try {
const { DatabaseService } = await import('../services/DatabaseService');
const { NodeRegistry } = await import('../services/NodeRegistry');
const dockerodeMod = await import('dockerode');
const Docker = (dockerodeMod as { default: new (opts?: unknown) => { listContainers: (opts?: unknown) => Promise<unknown[]> } }).default;
const db = DatabaseService.getInstance();
const localNodeId = NodeRegistry.getInstance().getDefaultNodeId();
if (!db.isMeshStackEnabled(localNodeId, stack)) {
return { ok: false, err: 'denied' };
}
const docker = new Docker();
const containers = (await docker.listContainers({
filters: { label: [`com.docker.compose.project=${stack}`, `com.docker.compose.service=${service}`] },
})) as Array<{ NetworkSettings?: { Networks?: Record<string, { IPAddress?: string }> } }>;
for (const c of containers) {
const networks = c.NetworkSettings?.Networks ?? {};
for (const net of Object.values(networks)) {
if (net.IPAddress) return { ok: true, host: net.IPAddress, port };
}
}
return { ok: false, err: 'no_target' };
} catch (err) {
console.warn('[Pilot] resolveMeshTarget failed:', sanitizeForLog((err as Error).message));
return { ok: false, err: 'agent_error' };
}
}
}
+191
View File
@@ -0,0 +1,191 @@
import { Router, type Request, type Response } from 'express';
import { DatabaseService } from '../services/DatabaseService';
import { NodeRegistry } from '../services/NodeRegistry';
import { MeshError, MeshService } from '../services/MeshService';
import { requireAdmin, requireAdmiral } from '../middleware/tierGates';
import { sanitizeForLog } from '../utils/safeLog';
export const meshRouter = Router();
function actorFor(req: Request): string {
const user = (req as Request & { user?: { username?: string } }).user;
return user?.username || 'system';
}
meshRouter.get('/status', async (_req: Request, res: Response): Promise<void> => {
if (!requireAdmiral(_req, res)) return;
try {
const status = await MeshService.getInstance().getStatus();
res.json({ nodes: status });
} catch (err) {
console.warn('[mesh] /status failed:', sanitizeForLog((err as Error).message));
res.status(500).json({ error: 'Failed to load mesh status' });
}
});
meshRouter.post('/nodes/:nodeId/enable', async (req: Request, res: Response): Promise<void> => {
if (!requireAdmiral(req, res)) return;
if (!requireAdmin(req, res)) return;
const nodeId = Number.parseInt(req.params.nodeId as string, 10);
if (!Number.isFinite(nodeId)) { res.status(400).json({ error: 'Invalid node id' }); return; }
try {
await MeshService.getInstance().enableForNode(nodeId);
res.json({ ok: true });
} catch (err) {
res.status(500).json({ error: (err as Error).message });
}
});
meshRouter.post('/nodes/:nodeId/disable', async (req: Request, res: Response): Promise<void> => {
if (!requireAdmiral(req, res)) return;
if (!requireAdmin(req, res)) return;
const nodeId = Number.parseInt(req.params.nodeId as string, 10);
if (!Number.isFinite(nodeId)) { res.status(400).json({ error: 'Invalid node id' }); return; }
try {
await MeshService.getInstance().disableForNode(nodeId);
res.json({ ok: true });
} catch (err) {
res.status(500).json({ error: (err as Error).message });
}
});
meshRouter.get('/nodes/:nodeId/stacks', async (req: Request, res: Response): Promise<void> => {
if (!requireAdmiral(req, res)) return;
const nodeId = Number.parseInt(req.params.nodeId as string, 10);
if (!Number.isFinite(nodeId)) { res.status(400).json({ error: 'Invalid node id' }); return; }
try {
const db = DatabaseService.getInstance();
const optedIn = new Set(db.listMeshStacks(nodeId).map((s) => s.stack_name));
const fsSvc = (await import('../services/FileSystemService')).FileSystemService.getInstance(nodeId);
const stacks = await fsSvc.getStacks();
res.json({
stacks: stacks.map((stackName: string) => ({
name: stackName,
optedIn: optedIn.has(stackName),
})),
});
} catch (err) {
console.warn('[mesh] list stacks failed:', sanitizeForLog((err as Error).message));
res.status(500).json({ error: 'Failed to list stacks' });
}
});
meshRouter.post('/nodes/:nodeId/stacks/:stackName/opt-in', async (req: Request, res: Response): Promise<void> => {
if (!requireAdmiral(req, res)) return;
if (!requireAdmin(req, res)) return;
const nodeId = Number.parseInt(req.params.nodeId as string, 10);
const stackName = req.params.stackName as string;
if (!Number.isFinite(nodeId) || !stackName) { res.status(400).json({ error: 'Invalid params' }); return; }
try {
await MeshService.getInstance().optInStack(nodeId, stackName, actorFor(req));
res.json({ ok: true });
} catch (err) {
if (err instanceof MeshError && err.code === 'port_collision') {
res.status(409).json({ error: err.message, code: err.code });
return;
}
if (err instanceof MeshError) {
res.status(400).json({ error: err.message, code: err.code });
return;
}
console.warn('[mesh] opt-in failed:', sanitizeForLog((err as Error).message));
res.status(500).json({ error: 'Opt-in failed' });
}
});
meshRouter.post('/nodes/:nodeId/stacks/:stackName/opt-out', async (req: Request, res: Response): Promise<void> => {
if (!requireAdmiral(req, res)) return;
if (!requireAdmin(req, res)) return;
const nodeId = Number.parseInt(req.params.nodeId as string, 10);
const stackName = req.params.stackName as string;
if (!Number.isFinite(nodeId) || !stackName) { res.status(400).json({ error: 'Invalid params' }); return; }
try {
await MeshService.getInstance().optOutStack(nodeId, stackName, actorFor(req));
res.json({ ok: true });
} catch (err) {
console.warn('[mesh] opt-out failed:', sanitizeForLog((err as Error).message));
res.status(500).json({ error: 'Opt-out failed' });
}
});
meshRouter.get('/aliases', async (req: Request, res: Response): Promise<void> => {
if (!requireAdmiral(req, res)) return;
try {
const aliases = await MeshService.getInstance().listAliases();
res.json({ aliases });
} catch (err) {
console.warn('[mesh] /aliases failed:', sanitizeForLog((err as Error).message));
res.status(500).json({ error: 'Failed to list aliases' });
}
});
meshRouter.get('/aliases/:alias/diagnostic', async (req: Request, res: Response): Promise<void> => {
if (!requireAdmiral(req, res)) return;
try {
const diag = await MeshService.getInstance().getRouteDiagnostic(req.params.alias as string);
res.json(diag);
} catch (err) {
res.status(500).json({ error: (err as Error).message });
}
});
meshRouter.post('/aliases/:alias/test', async (req: Request, res: Response): Promise<void> => {
if (!requireAdmiral(req, res)) return;
try {
const sourceNodeId = NodeRegistry.getInstance().getDefaultNodeId();
const result = await MeshService.getInstance().testUpstream(req.params.alias as string, sourceNodeId);
res.json(result);
} catch (err) {
res.status(500).json({ error: (err as Error).message });
}
});
meshRouter.get('/nodes/:nodeId/diagnostic', async (req: Request, res: Response): Promise<void> => {
if (!requireAdmiral(req, res)) return;
const nodeId = Number.parseInt(req.params.nodeId as string, 10);
if (!Number.isFinite(nodeId)) { res.status(400).json({ error: 'Invalid node id' }); return; }
try {
const diag = await MeshService.getInstance().getNodeDiagnostic(nodeId);
res.json(diag);
} catch (err) {
res.status(500).json({ error: (err as Error).message });
}
});
meshRouter.post('/nodes/:nodeId/sidecar/restart', async (req: Request, res: Response): Promise<void> => {
if (!requireAdmiral(req, res)) return;
if (!requireAdmin(req, res)) return;
const nodeId = Number.parseInt(req.params.nodeId as string, 10);
if (!Number.isFinite(nodeId)) { res.status(400).json({ error: 'Invalid node id' }); return; }
try {
await MeshService.getInstance().stopSidecar(nodeId);
await MeshService.getInstance().spawnSidecar(nodeId);
res.json({ ok: true });
} catch (err) {
res.status(500).json({ error: (err as Error).message });
}
});
meshRouter.get('/activity', (req: Request, res: Response): void => {
if (!requireAdmiral(req, res)) return;
const alias = typeof req.query.alias === 'string' ? req.query.alias : undefined;
const source = typeof req.query.source === 'string' ? (req.query.source as 'sidecar' | 'pilot' | 'mesh') : undefined;
const level = typeof req.query.level === 'string' ? (req.query.level as 'info' | 'warn' | 'error') : undefined;
const limit = typeof req.query.limit === 'string' ? Number.parseInt(req.query.limit, 10) : 200;
const events = MeshService.getInstance().getActivity({ alias, source, level, limit });
res.json({ events });
});
meshRouter.get('/activity/stream', (req: Request, res: Response): void => {
if (!requireAdmiral(req, res)) return;
res.setHeader('Content-Type', 'text/event-stream');
res.setHeader('Cache-Control', 'no-cache, no-transform');
res.setHeader('Connection', 'keep-alive');
res.flushHeaders?.();
const send = (event: object) => {
try { res.write(`data: ${JSON.stringify(event)}\n\n`); } catch { /* ignore */ }
};
const unsubscribe = MeshService.getInstance().subscribeActivity(send);
req.on('close', () => unsubscribe());
});
+28 -4
View File
@@ -6,6 +6,7 @@ import WebSocket from 'ws';
import DockerController from './DockerController';
import { DatabaseService } from './DatabaseService';
import { FileSystemService } from './FileSystemService';
import { MeshService } from './MeshService';
import { LogFormatter } from './LogFormatter';
import { NodeRegistry } from './NodeRegistry';
import { RegistryService } from './RegistryService';
@@ -33,6 +34,29 @@ export class ComposeService {
return new ComposeService(nodeId);
}
/**
* Build the `docker compose` argument prefix for a stack, splicing in the
* Sencho Mesh override file if the stack is opted into the mesh. When no
* override applies, returns args without `-f` so docker compose's built-in
* file discovery resolves the stack's actual compose filename. The user's
* source compose file is never mutated.
*/
private async composeArgs(stackName: string, action: string[]): Promise<string[]> {
const args: string[] = ['compose'];
let overridePath: string | null = null;
try {
overridePath = await MeshService.getInstance().ensureStackOverride(this.nodeId, stackName);
} catch (err) {
console.warn('[ComposeService] mesh override skipped:', sanitizeForLog((err as Error).message));
}
if (overridePath) {
const baseFilename = await FileSystemService.getInstance(this.nodeId).getComposeFilename(stackName);
args.push('-f', baseFilename, '-f', overridePath);
}
args.push(...action);
return args;
}
private execute(
command: string,
args: string[],
@@ -159,7 +183,7 @@ export class ComposeService {
}
await this.withRegistryAuth(async (env) => {
await this.execute('docker', ['compose', 'up', '-d', '--remove-orphans'], stackDir, ws, true, env);
await this.execute('docker', await this.composeArgs(stackName, ['up', '-d', '--remove-orphans']), stackDir, ws, true, env);
}, sendOutput);
// Post-Deploy Health Probe
@@ -193,7 +217,7 @@ export class ComposeService {
const fsSvc = FileSystemService.getInstance(this.nodeId);
await fsSvc.restoreStackFiles(stackName);
await this.withRegistryAuth(async (env) => {
await this.execute('docker', ['compose', 'up', '-d', '--remove-orphans'], stackDir, ws, true, env);
await this.execute('docker', await this.composeArgs(stackName, ['up', '-d', '--remove-orphans']), stackDir, ws, true, env);
}, sendOutput);
sendOutput('=== Rolled back successfully ===\n');
} catch (rollbackError) {
@@ -353,7 +377,7 @@ export class ComposeService {
await this.execute('docker', ['compose', 'pull'], stackDir, ws, true, env);
sendOutput('=== Recreating containers ===\n');
await this.execute('docker', ['compose', 'up', '-d', '--remove-orphans'], stackDir, ws, true, env);
await this.execute('docker', await this.composeArgs(stackName, ['up', '-d', '--remove-orphans']), stackDir, ws, true, env);
}, sendOutput);
// Post-Update Health Probe
@@ -389,7 +413,7 @@ export class ComposeService {
const fsSvc = FileSystemService.getInstance(this.nodeId);
await fsSvc.restoreStackFiles(stackName);
await this.withRegistryAuth(async (env) => {
await this.execute('docker', ['compose', 'up', '-d', '--remove-orphans'], stackDir, ws, true, env);
await this.execute('docker', await this.composeArgs(stackName, ['up', '-d', '--remove-orphans']), stackDir, ws, true, env);
}, sendOutput);
sendOutput('=== Rolled back successfully ===\n');
} catch (rollbackError) {
+57
View File
@@ -519,6 +519,7 @@ export class DatabaseService {
this.migratePolicyEvaluationColumn();
this.migrateNotificationCategory();
this.migrateNotificationActor();
this.migrateMeshTables();
// Reset the cache once at end of constructor in case any migration
// populated it via getGlobalSettings() and a subsequent migration
@@ -1251,6 +1252,62 @@ export class DatabaseService {
}
}
private migrateMeshTables(): void {
try {
this.db.prepare(`
CREATE TABLE IF NOT EXISTS mesh_stacks (
id INTEGER PRIMARY KEY AUTOINCREMENT,
node_id INTEGER NOT NULL,
stack_name TEXT NOT NULL,
created_at INTEGER NOT NULL,
created_by TEXT,
UNIQUE(node_id, stack_name),
FOREIGN KEY (node_id) REFERENCES nodes(id) ON DELETE CASCADE
)
`).run();
this.db.prepare('CREATE INDEX IF NOT EXISTS idx_mesh_stacks_node ON mesh_stacks(node_id)').run();
} catch (e) {
console.warn('[DatabaseService] Could not create mesh_stacks:', (e as Error).message);
}
this.tryAddColumn('nodes', 'mesh_enabled', 'INTEGER NOT NULL DEFAULT 0');
}
// --- Sencho Mesh ---
public listMeshStacks(nodeId?: number): Array<{ id: number; node_id: number; stack_name: string; created_at: number; created_by: string | null }> {
const sql = nodeId !== undefined
? 'SELECT id, node_id, stack_name, created_at, created_by FROM mesh_stacks WHERE node_id = ?'
: 'SELECT id, node_id, stack_name, created_at, created_by FROM mesh_stacks';
const rows = nodeId !== undefined
? this.db.prepare(sql).all(nodeId)
: this.db.prepare(sql).all();
return rows as Array<{ id: number; node_id: number; stack_name: string; created_at: number; created_by: string | null }>;
}
public isMeshStackEnabled(nodeId: number, stackName: string): boolean {
const row = this.db.prepare('SELECT 1 FROM mesh_stacks WHERE node_id = ? AND stack_name = ?').get(nodeId, stackName);
return !!row;
}
public insertMeshStack(nodeId: number, stackName: string, createdBy: string | null): void {
this.db.prepare(
'INSERT INTO mesh_stacks (node_id, stack_name, created_at, created_by) VALUES (?, ?, ?, ?)'
).run(nodeId, stackName, Date.now(), createdBy);
}
public deleteMeshStack(nodeId: number, stackName: string): void {
this.db.prepare('DELETE FROM mesh_stacks WHERE node_id = ? AND stack_name = ?').run(nodeId, stackName);
}
public setNodeMeshEnabled(nodeId: number, enabled: boolean): void {
this.db.prepare('UPDATE nodes SET mesh_enabled = ? WHERE id = ?').run(enabled ? 1 : 0, nodeId);
}
public getNodeMeshEnabled(nodeId: number): boolean {
const row = this.db.prepare('SELECT mesh_enabled FROM nodes WHERE id = ?').get(nodeId) as { mesh_enabled?: number } | undefined;
return !!row?.mesh_enabled;
}
// --- Agents ---
public getAgents(nodeId: number): Agent[] {
@@ -107,6 +107,10 @@ export class FileSystemService {
throw new Error(`No compose file found for stack: ${stackName}`);
}
async getComposeFilename(stackName: string): Promise<string> {
return path.basename(await this.getComposeFilePath(stackName));
}
async getStacks(): Promise<string[]> {
try {
const items = await fsPromises.readdir(this.baseDir, { withFileTypes: true });
@@ -0,0 +1,60 @@
import * as YAML from 'yaml';
/**
* Sencho Mesh Compose override generator.
*
* Produces a YAML override applied with `docker compose -f compose.yml -f
* mesh.override.yml up` that injects cross-node alias entries into each
* opted-in service's /etc/hosts. The aliases resolve to the host gateway
* (`host-gateway`), which is where the local Sencho Mesh sidecar listens
* (host network mode). The user's source compose file is never mutated; the
* override lives in Sencho's data dir.
*/
export interface MeshAlias {
/** `<service>.<stack>.<nodeName>.sencho` */
host: string;
}
export interface MeshOverrideInput {
/** Service names from the user's compose file (the override echoes them). */
services: string[];
/** Aliases this stack should be able to resolve. Order is normalized in output. */
aliases: MeshAlias[];
}
/**
* Returns a YAML string suitable for `-f mesh.override.yml`. Stable output
* ordering so file content does not churn between deploys.
*/
export function generateOverrideYaml(input: MeshOverrideInput): string {
const sortedServices = [...input.services].sort();
const sortedAliases = [...input.aliases].sort((a, b) => a.host.localeCompare(b.host));
if (sortedAliases.length === 0) {
const services: Record<string, unknown> = {};
for (const svc of sortedServices) services[svc] = {};
return YAML.stringify({ services }, { lineWidth: 0 });
}
const extraHostsList = sortedAliases.map((a) => `${a.host}:host-gateway`);
const services: Record<string, unknown> = {};
for (const svc of sortedServices) {
services[svc] = { extra_hosts: extraHostsList };
}
return YAML.stringify({ services }, { lineWidth: 0 });
}
/**
* Build alias hostnames for every opted-in service across the fleet. Pure
* helper consumed by MeshService and the override generator.
*/
export function buildAliasHosts(opts: {
nodeName: string;
stackName: string;
services: Array<{ service: string; ports: number[] }>;
}): string[] {
return opts.services.map((s) => `${s.service}.${opts.stackName}.${opts.nodeName}.sencho`);
}
+844
View File
@@ -0,0 +1,844 @@
import net from 'net';
import path from 'path';
import fs from 'fs/promises';
import { EventEmitter } from 'events';
import jwt from 'jsonwebtoken';
import { DatabaseService } from './DatabaseService';
import DockerController from './DockerController';
import { PilotTunnelManager } from './PilotTunnelManager';
import { generateOverrideYaml, MeshAlias } from './MeshComposeOverride';
import { sanitizeForLog } from '../utils/safeLog';
import { isPathWithinBase, isValidStackName } from '../utils/validation';
const ACTIVITY_BUFFER_SIZE = 1000;
const ALIAS_REFRESH_INTERVAL_MS = 60_000;
const SIDECAR_CONTAINER_PREFIX = 'sencho-mesh-';
const DEFAULT_SIDECAR_IMAGE = process.env.SENCHO_MESH_IMAGE || 'saelix/sencho-mesh:latest';
const SIDECAR_TOKEN_TTL = '7d';
const PROBE_TIMEOUT_MS = 5_000;
const SLOW_PROBE_THRESHOLD_MS = 500;
export type MeshActivitySource = 'sidecar' | 'pilot' | 'mesh';
export type MeshActivityLevel = 'info' | 'warn' | 'error';
export type MeshActivityType =
| 'route.resolve.ok' | 'route.resolve.denied'
| 'tunnel.open' | 'tunnel.fail' | 'tunnel.backpressure'
| 'opt_in' | 'opt_out'
| 'mesh.enable' | 'mesh.disable'
| 'probe.ok' | 'probe.fail'
| 'sidecar.start' | 'sidecar.stop' | 'sidecar.crash';
export interface MeshActivityEvent {
ts: number;
source: MeshActivitySource;
level: MeshActivityLevel;
type: MeshActivityType;
nodeId?: number;
alias?: string;
streamId?: number;
message: string;
details?: Record<string, unknown>;
}
export interface MeshGlobalAlias {
/** `<service>.<stack>.<nodeName>.sencho` */
host: string;
nodeId: number;
nodeName: string;
stackName: string;
serviceName: string;
port: number;
}
export interface MeshTarget {
nodeId: number;
stack: string;
service: string;
port: number;
alias: string;
}
export interface MeshNodeStatus {
nodeId: number;
nodeName: string;
enabled: boolean;
sidecarRunning: boolean;
pilotConnected: boolean;
optedInStacks: string[];
activeStreamCount: number;
}
export interface MeshNodeDiagnostic {
nodeId: number;
sidecar: { running: boolean; restartCount: number };
pilot: { connected: boolean; bufferedAmount: number; lastSeen: number | null };
activeStreams: Array<{ streamId: number; alias?: string; bytesIn: number; bytesOut: number; ageMs: number }>;
aliasCache: Array<{ host: string; targetNodeId: number; port: number }>;
}
export interface MeshRouteDiagnostic {
alias: string;
target: MeshTarget | null;
pilot: { connected: boolean; lastSeen: number | null };
lastError: { ts: number; message: string } | null;
lastProbeMs: number | null;
state: 'healthy' | 'degraded' | 'unreachable' | 'tunnel down' | 'not authorized';
}
export interface MeshProbeResult {
ok: boolean;
latencyMs?: number;
where?: 'sidecar' | 'pilot_tunnel' | 'agent_resolve' | 'agent_dial' | 'target_port';
code?: string;
message?: string;
}
interface ActiveStreamRecord {
streamId: number;
alias?: string;
bytesIn: number;
bytesOut: number;
openedAt: number;
}
interface PendingResolve {
sidecarSocket: WebSocketLike;
connId: number;
port: number;
remoteAddr: string;
}
interface WebSocketLike {
send(data: string | Buffer, opts?: unknown, cb?: (err?: Error) => void): void;
readyState: number;
on(event: string, listener: (...args: unknown[]) => void): unknown;
}
/**
* Sencho Mesh orchestrator. Owns:
* - sidecar lifecycle (Dockerode-spawned per-instance)
* - opt-in / opt-out persistence and cascading override regeneration
* - global alias aggregation (across the fleet via the existing API)
* - request-based resolution from sidecar control WS
* - cross-node TCP forwarding via PilotTunnelManager
* - probe + diagnostics + activity ring buffer
*
* V1 limitations:
* - one cross-node alias per TCP port across the fleet (port-collision check at opt-in)
* - sidecar runs in host network mode; aliases resolve via `host-gateway` extra_hosts
* - pilot-to-pilot mesh routing is not supported (only central <-> pilot)
*/
export class MeshService extends EventEmitter {
private static instance: MeshService;
private started = false;
private aliasCache = new Map<string, MeshGlobalAlias>();
private aliasByPort = new Map<number, MeshGlobalAlias>();
private activity: MeshActivityEvent[] = [];
private activeStreams = new Map<number, ActiveStreamRecord>();
private pendingResolves = new Map<string, PendingResolve>();
private sidecarSockets = new Set<WebSocketLike>();
private aliasRefreshTimer?: NodeJS.Timeout;
private routeErrorMap = new Map<string, { ts: number; message: string }>();
private routeLatencyMap = new Map<string, number>();
private activityListeners = new Set<(e: MeshActivityEvent) => void>();
private constructor() {
super();
this.setMaxListeners(50);
}
public static getInstance(): MeshService {
if (!MeshService.instance) MeshService.instance = new MeshService();
return MeshService.instance;
}
public async start(): Promise<void> {
if (this.started) return;
this.started = true;
const ptm = PilotTunnelManager.getInstance();
ptm.on('tunnel-down', (nodeId: number) => this.onTunnelDown(nodeId));
ptm.on('tunnel-up', (nodeId: number) => this.logActivity({
source: 'pilot', level: 'info', type: 'tunnel.open',
nodeId, message: `pilot tunnel up for node ${nodeId}`,
}));
await this.refreshAliasCache();
this.aliasRefreshTimer = setInterval(() => {
void this.refreshAliasCache().catch((err) => {
console.warn('[MeshService] alias refresh failed:', sanitizeForLog((err as Error).message));
});
}, ALIAS_REFRESH_INTERVAL_MS);
this.logActivity({
source: 'mesh', level: 'info', type: 'mesh.enable',
message: 'MeshService started',
});
}
public async stop(): Promise<void> {
if (!this.started) return;
this.started = false;
if (this.aliasRefreshTimer) {
clearInterval(this.aliasRefreshTimer);
this.aliasRefreshTimer = undefined;
}
for (const ws of this.sidecarSockets) {
try { (ws as { close?: (code: number) => void }).close?.(1000); } catch { /* ignore */ }
}
this.sidecarSockets.clear();
}
// --- Activity log ---
public logActivity(event: Omit<MeshActivityEvent, 'ts'>): void {
const full: MeshActivityEvent = { ts: Date.now(), ...event };
this.activity.push(full);
if (this.activity.length > ACTIVITY_BUFFER_SIZE) this.activity.shift();
for (const listener of this.activityListeners) {
try { listener(full); } catch { /* ignore */ }
}
if (event.alias && event.level === 'error') {
this.routeErrorMap.set(event.alias, { ts: full.ts, message: event.message });
}
}
public getActivity(filter?: { alias?: string; source?: MeshActivitySource; level?: MeshActivityLevel; limit?: number }): MeshActivityEvent[] {
let out = this.activity;
if (filter?.alias) out = out.filter((e) => e.alias === filter.alias);
if (filter?.source) out = out.filter((e) => e.source === filter.source);
if (filter?.level) out = out.filter((e) => e.level === filter.level);
const limit = filter?.limit ?? 200;
return out.slice(-limit);
}
public subscribeActivity(listener: (e: MeshActivityEvent) => void): () => void {
this.activityListeners.add(listener);
return () => { this.activityListeners.delete(listener); };
}
// --- Opt-in / opt-out ---
public async optInStack(nodeId: number, stackName: string, actor: string): Promise<void> {
if (!isValidStackName(stackName)) {
throw new MeshError('denied', `invalid stack name: ${stackName}`);
}
const db = DatabaseService.getInstance();
if (db.isMeshStackEnabled(nodeId, stackName)) return;
const services = await this.inspectStackServices(nodeId, stackName);
if (services.length === 0) {
throw new MeshError('no_target', `stack ${stackName} has no running services on this node`);
}
const newPorts = new Set<number>();
for (const svc of services) for (const p of svc.ports) newPorts.add(p);
for (const port of newPorts) {
const existing = this.aliasByPort.get(port);
if (existing) {
throw new MeshError(
'port_collision',
`port ${port} is already claimed by ${existing.host}`,
);
}
}
db.insertMeshStack(nodeId, stackName, actor);
await this.refreshAliasCache();
await this.regenerateOverridesForNode(nodeId);
this.logActivity({
source: 'mesh', level: 'info', type: 'opt_in',
nodeId, message: `opt-in ${stackName}`, details: { actor },
});
db.insertAuditLog({
timestamp: Date.now(), username: actor, method: 'POST',
path: `/api/mesh/nodes/${nodeId}/stacks/${stackName}/opt-in`,
status_code: 200, node_id: nodeId, ip_address: '127.0.0.1',
summary: `Sencho Mesh: opted ${stackName} into the mesh`,
});
}
public async optOutStack(nodeId: number, stackName: string, actor: string): Promise<void> {
if (!isValidStackName(stackName)) {
throw new MeshError('denied', `invalid stack name: ${stackName}`);
}
const db = DatabaseService.getInstance();
if (!db.isMeshStackEnabled(nodeId, stackName)) return;
db.deleteMeshStack(nodeId, stackName);
await this.removeStackOverride(nodeId, stackName);
await this.refreshAliasCache();
await this.regenerateOverridesForNode(nodeId);
this.logActivity({
source: 'mesh', level: 'info', type: 'opt_out',
nodeId, message: `opt-out ${stackName}`, details: { actor },
});
db.insertAuditLog({
timestamp: Date.now(), username: actor, method: 'POST',
path: `/api/mesh/nodes/${nodeId}/stacks/${stackName}/opt-out`,
status_code: 200, node_id: nodeId, ip_address: '127.0.0.1',
summary: `Sencho Mesh: opted ${stackName} out of the mesh`,
});
}
public async enableForNode(nodeId: number): Promise<void> {
DatabaseService.getInstance().setNodeMeshEnabled(nodeId, true);
this.logActivity({
source: 'mesh', level: 'info', type: 'mesh.enable',
nodeId, message: `mesh enabled on node ${nodeId}`,
});
}
public async disableForNode(nodeId: number): Promise<void> {
DatabaseService.getInstance().setNodeMeshEnabled(nodeId, false);
const stacks = DatabaseService.getInstance().listMeshStacks(nodeId);
for (const s of stacks) {
DatabaseService.getInstance().deleteMeshStack(nodeId, s.stack_name);
await this.removeStackOverride(nodeId, s.stack_name);
}
await this.refreshAliasCache();
this.logActivity({
source: 'mesh', level: 'info', type: 'mesh.disable',
nodeId, message: `mesh disabled on node ${nodeId}`,
});
}
// --- Override file management ---
public async ensureStackOverride(nodeId: number, stackName: string): Promise<string | null> {
if (!isValidStackName(stackName)) return null;
const db = DatabaseService.getInstance();
if (!db.isMeshStackEnabled(nodeId, stackName)) return null;
const aliases: MeshAlias[] = Array.from(this.aliasCache.values()).map((a) => ({ host: a.host }));
const services = await this.inspectStackServices(nodeId, stackName);
const yaml = generateOverrideYaml({
services: services.map((s) => s.service),
aliases,
});
const dir = this.overrideDirFor(nodeId);
await fs.mkdir(dir, { recursive: true });
const file = path.resolve(dir, `${stackName}.override.yml`);
if (!isPathWithinBase(file, dir)) return null;
await fs.writeFile(file, yaml, 'utf8');
return file;
}
private async removeStackOverride(nodeId: number, stackName: string): Promise<void> {
if (!isValidStackName(stackName)) return;
const dir = this.overrideDirFor(nodeId);
const file = path.resolve(dir, `${stackName}.override.yml`);
if (!isPathWithinBase(file, dir)) return;
try { await fs.unlink(file); } catch { /* ignore not-exist */ }
}
private overrideDirFor(nodeId: number): string {
const dataDir = process.env.DATA_DIR || '/app/data';
return path.join(dataDir, 'mesh', 'overrides', String(nodeId));
}
private async regenerateOverridesForNode(nodeId: number): Promise<void> {
const db = DatabaseService.getInstance();
const stacks = db.listMeshStacks(nodeId);
for (const s of stacks) {
try {
await this.ensureStackOverride(nodeId, s.stack_name);
} catch (err) {
console.warn('[MeshService] override regen failed:', sanitizeForLog((err as Error).message));
}
}
}
// --- Alias aggregation ---
public async refreshAliasCache(): Promise<void> {
const db = DatabaseService.getInstance();
const next = new Map<string, MeshGlobalAlias>();
const portMap = new Map<number, MeshGlobalAlias>();
const stacks = db.listMeshStacks();
for (const row of stacks) {
const node = db.getNode(row.node_id);
if (!node) continue;
const services = await this.inspectStackServices(row.node_id, row.stack_name).catch(() => []);
for (const svc of services) {
const host = `${svc.service}.${row.stack_name}.${node.name}.sencho`;
for (const port of svc.ports) {
const alias: MeshGlobalAlias = {
host,
nodeId: row.node_id,
nodeName: node.name,
stackName: row.stack_name,
serviceName: svc.service,
port,
};
next.set(host, alias);
if (!portMap.has(port)) portMap.set(port, alias);
}
}
}
this.aliasCache = next;
this.aliasByPort = portMap;
}
public async listAliases(): Promise<MeshGlobalAlias[]> {
return Array.from(this.aliasCache.values());
}
/**
* Inspect a stack on a node and return its running services with the ports
* they listen on. Uses Compose container labels.
*/
private async inspectStackServices(nodeId: number, stackName: string): Promise<Array<{ service: string; ports: number[] }>> {
try {
const docker = DockerController.getInstance(nodeId).getDocker();
const containers = await docker.listContainers({
all: true,
filters: { label: [`com.docker.compose.project=${stackName}`] },
});
const byService = new Map<string, Set<number>>();
for (const c of containers) {
const svc = c.Labels?.['com.docker.compose.service'];
if (!svc) continue;
const ports = byService.get(svc) || new Set<number>();
for (const p of c.Ports || []) {
if (p.PrivatePort) ports.add(p.PrivatePort);
}
byService.set(svc, ports);
}
return Array.from(byService.entries()).map(([service, ports]) => ({ service, ports: Array.from(ports) }));
} catch (err) {
console.warn('[MeshService] inspectStackServices failed:', sanitizeForLog((err as Error).message));
return [];
}
}
// --- Resolution + forwarding ---
public resolveByLocalPort(port: number): MeshTarget | null {
const alias = this.aliasByPort.get(port);
if (!alias) return null;
return {
nodeId: alias.nodeId,
stack: alias.stackName,
service: alias.serviceName,
port: alias.port,
alias: alias.host,
};
}
/**
* Forward bytes from a sidecar-accepted local socket to the target.
* Same-node target: open a direct TCP socket.
* Cross-node target: open a pilot-tunnel TcpStream to that node's agent.
*/
public openTcp(target: MeshTarget, src: net.Socket, sourceNodeId: number): void {
if (target.nodeId === sourceNodeId) {
this.openSameNode(target, src);
return;
}
this.openCrossNode(target, src);
}
private openSameNode(target: MeshTarget, src: net.Socket): void {
// For same-node fast path, the agent's resolution logic isn't needed:
// we can dial via Dockerode's container IP. For V1 simplicity, we dial
// the host-gateway's published port if mapped, falling back to
// 127.0.0.1 (the sidecar runs in host network mode so localhost reaches
// the container's published port).
const upstream = net.createConnection({ host: '127.0.0.1', port: target.port });
upstream.setTimeout(PROBE_TIMEOUT_MS);
const stream = this.registerActiveStream(target.alias);
upstream.once('connect', () => {
upstream.setTimeout(0);
this.logActivity({
source: 'mesh', level: 'info', type: 'route.resolve.ok',
alias: target.alias, streamId: stream.streamId,
message: `same-node connect to ${target.alias}`,
});
src.pipe(upstream);
upstream.pipe(src);
});
const teardown = () => {
this.activeStreams.delete(stream.streamId);
try { upstream.destroy(); } catch { /* ignore */ }
try { src.destroy(); } catch { /* ignore */ }
};
upstream.on('error', () => teardown());
upstream.on('close', () => teardown());
src.on('error', () => teardown());
src.on('close', () => teardown());
}
private openCrossNode(target: MeshTarget, src: net.Socket): void {
const ptm = PilotTunnelManager.getInstance();
if (!ptm.hasActiveTunnel(target.nodeId)) {
this.logActivity({
source: 'pilot', level: 'error', type: 'tunnel.fail',
nodeId: target.nodeId, alias: target.alias,
message: `no active pilot tunnel to node ${target.nodeId}`,
});
try { src.destroy(); } catch { /* ignore */ }
return;
}
const bridge = ptm.getBridge(target.nodeId);
if (!bridge) { try { src.destroy(); } catch { /* ignore */ } return; }
const tcpStream = bridge.openTcpStream({ stack: target.stack, service: target.service, port: target.port });
if (!tcpStream) { try { src.destroy(); } catch { /* ignore */ } return; }
const record = this.registerActiveStream(target.alias, tcpStream.streamId);
const t0 = Date.now();
tcpStream.on('open', () => {
this.logActivity({
source: 'mesh', level: 'info', type: 'route.resolve.ok',
nodeId: target.nodeId, alias: target.alias, streamId: tcpStream.streamId,
message: `cross-node connect to ${target.alias}`,
});
this.routeLatencyMap.set(target.alias, Date.now() - t0);
});
tcpStream.on('data', (chunk: Buffer) => {
record.bytesIn += chunk.length;
try { src.write(chunk); } catch { /* ignore */ }
});
tcpStream.on('error', (err: Error) => {
this.logActivity({
source: 'pilot', level: 'error', type: 'tunnel.fail',
nodeId: target.nodeId, alias: target.alias, streamId: tcpStream.streamId,
message: err.message,
});
this.activeStreams.delete(tcpStream.streamId);
try { src.destroy(); } catch { /* ignore */ }
});
tcpStream.on('close', () => {
this.activeStreams.delete(tcpStream.streamId);
try { src.end(); } catch { /* ignore */ }
});
src.on('data', (chunk: Buffer) => {
record.bytesOut += chunk.length;
tcpStream.write(chunk);
});
src.on('end', () => tcpStream.end());
src.on('close', () => tcpStream.destroy());
src.on('error', () => tcpStream.destroy());
}
private registerActiveStream(alias: string, streamId?: number): ActiveStreamRecord {
const id = streamId ?? -Math.floor(Math.random() * 0x7fffffff);
const record: ActiveStreamRecord = {
streamId: id, alias, bytesIn: 0, bytesOut: 0, openedAt: Date.now(),
};
this.activeStreams.set(id, record);
return record;
}
private onTunnelDown(nodeId: number): void {
this.logActivity({
source: 'pilot', level: 'warn', type: 'tunnel.fail',
nodeId, message: `pilot tunnel down for node ${nodeId}`,
});
}
// --- Probe / Test upstream ---
public async testUpstream(alias: string, sourceNodeId: number): Promise<MeshProbeResult> {
const target = this.lookupAliasGlobal(alias);
if (!target) {
return { ok: false, where: 'sidecar', code: 'no_route', message: 'alias not found' };
}
if (!DatabaseService.getInstance().isMeshStackEnabled(target.nodeId, target.stackName)) {
return { ok: false, where: 'agent_resolve', code: 'denied', message: 'target stack not opted in' };
}
if (target.nodeId !== sourceNodeId) {
const ptm = PilotTunnelManager.getInstance();
if (!ptm.hasActiveTunnel(target.nodeId)) {
return { ok: false, where: 'pilot_tunnel', code: 'tunnel_down', message: 'no pilot tunnel' };
}
const bridge = ptm.getBridge(target.nodeId);
if (!bridge) {
return { ok: false, where: 'pilot_tunnel', code: 'tunnel_down', message: 'no bridge' };
}
const t0 = Date.now();
const stream = bridge.openTcpStream({ stack: target.stackName, service: target.serviceName, port: target.port });
if (!stream) return { ok: false, where: 'pilot_tunnel', code: 'tunnel_down', message: 'open failed' };
return new Promise<MeshProbeResult>((resolve) => {
const timer = setTimeout(() => {
stream.destroy();
resolve({ ok: false, where: 'agent_dial', code: 'timeout', message: 'probe timeout' });
}, PROBE_TIMEOUT_MS);
stream.once('open', () => {
clearTimeout(timer);
const latency = Date.now() - t0;
this.routeLatencyMap.set(target.host, latency);
stream.destroy();
this.logActivity({
source: 'mesh', level: 'info', type: 'probe.ok',
alias: target.host, message: `probe ok ${latency}ms`,
});
resolve({ ok: true, latencyMs: latency });
});
stream.once('error', (err: Error) => {
clearTimeout(timer);
this.logActivity({
source: 'mesh', level: 'error', type: 'probe.fail',
alias: target.host, message: err.message,
});
resolve({ ok: false, where: 'agent_dial', code: 'unreachable', message: err.message });
});
});
}
const t0 = Date.now();
return new Promise<MeshProbeResult>((resolve) => {
const sock = net.createConnection({ host: '127.0.0.1', port: target.port });
sock.setTimeout(PROBE_TIMEOUT_MS);
sock.once('connect', () => {
const latency = Date.now() - t0;
this.routeLatencyMap.set(target.host, latency);
sock.destroy();
resolve({ ok: true, latencyMs: latency });
});
sock.once('timeout', () => {
sock.destroy();
resolve({ ok: false, where: 'target_port', code: 'timeout', message: 'connect timeout' });
});
sock.once('error', (err) => {
resolve({ ok: false, where: 'target_port', code: 'unreachable', message: err.message });
});
});
}
private lookupAliasGlobal(host: string): MeshGlobalAlias | null {
return this.aliasCache.get(host) || null;
}
// --- Diagnostics ---
public async getRouteDiagnostic(alias: string): Promise<MeshRouteDiagnostic> {
const target = this.lookupAliasGlobal(alias);
const lastError = this.routeErrorMap.get(alias) || null;
const lastProbeMs = this.routeLatencyMap.get(alias) ?? null;
if (!target) {
return { alias, target: null, pilot: { connected: false, lastSeen: null }, lastError, lastProbeMs, state: 'not authorized' };
}
const ptm = PilotTunnelManager.getInstance();
const pilotConnected = ptm.hasActiveTunnel(target.nodeId);
const node = DatabaseService.getInstance().getNode(target.nodeId);
const lastSeen = node?.pilot_last_seen ?? null;
const optedIn = DatabaseService.getInstance().isMeshStackEnabled(target.nodeId, target.stackName);
let state: MeshRouteDiagnostic['state'];
if (!optedIn) state = 'not authorized';
else if (!pilotConnected) state = 'tunnel down';
else if (lastError && Date.now() - lastError.ts < 60_000) state = 'unreachable';
else if (lastProbeMs !== null && lastProbeMs > SLOW_PROBE_THRESHOLD_MS) state = 'degraded';
else state = 'healthy';
return {
alias,
target: {
nodeId: target.nodeId,
stack: target.stackName,
service: target.serviceName,
port: target.port,
alias,
},
pilot: { connected: pilotConnected, lastSeen },
lastError,
lastProbeMs,
state,
};
}
public async getNodeDiagnostic(nodeId: number): Promise<MeshNodeDiagnostic> {
const ptm = PilotTunnelManager.getInstance();
const bridge = ptm.getBridge(nodeId);
const node = DatabaseService.getInstance().getNode(nodeId);
const sidecarRunning = await this.isSidecarRunning(nodeId);
const aliasCacheRows = Array.from(this.aliasCache.values())
.filter((a) => a.nodeId === nodeId)
.map((a) => ({ host: a.host, targetNodeId: a.nodeId, port: a.port }));
const now = Date.now();
const activeStreams = Array.from(this.activeStreams.values()).map((s) => ({
streamId: s.streamId, alias: s.alias,
bytesIn: s.bytesIn, bytesOut: s.bytesOut,
ageMs: now - s.openedAt,
}));
return {
nodeId,
sidecar: { running: sidecarRunning, restartCount: 0 },
pilot: {
connected: !!bridge,
bufferedAmount: bridge?.getBufferedAmount() ?? 0,
lastSeen: node?.pilot_last_seen ?? null,
},
activeStreams,
aliasCache: aliasCacheRows,
};
}
public async getStatus(): Promise<MeshNodeStatus[]> {
const db = DatabaseService.getInstance();
const ptm = PilotTunnelManager.getInstance();
const nodes = db.getNodes();
const out: MeshNodeStatus[] = [];
for (const node of nodes) {
const optedInStacks = db.listMeshStacks(node.id).map((s) => s.stack_name);
out.push({
nodeId: node.id,
nodeName: node.name,
enabled: db.getNodeMeshEnabled(node.id),
sidecarRunning: await this.isSidecarRunning(node.id),
pilotConnected: node.type === 'remote' ? ptm.hasActiveTunnel(node.id) : true,
optedInStacks,
activeStreamCount: Array.from(this.activeStreams.values()).length,
});
}
return out;
}
// --- Sidecar lifecycle (best-effort; real spawn happens on local node) ---
public async spawnSidecar(nodeId: number): Promise<void> {
const docker = DockerController.getInstance(nodeId).getDocker();
const name = `${SIDECAR_CONTAINER_PREFIX}${nodeId}`;
try {
const existing = docker.getContainer(name);
const info = await existing.inspect().catch(() => null);
if (info?.State?.Running) return;
if (info) await existing.remove({ force: true }).catch(() => undefined);
} catch { /* ignore */ }
const token = this.mintSidecarToken(nodeId);
const controlUrl = process.env.SENCHO_INTERNAL_URL || 'ws://127.0.0.1:1852/api/mesh/control';
try {
const container = await docker.createContainer({
name,
Image: DEFAULT_SIDECAR_IMAGE,
Env: [
`SENCHO_CONTROL_URL=${controlUrl}`,
`SENCHO_MESH_TOKEN=${token}`,
`MESH_NODE_ID=${nodeId}`,
],
HostConfig: {
NetworkMode: 'host',
RestartPolicy: { Name: 'unless-stopped' },
},
Labels: {
'sencho.mesh.role': 'sidecar',
'sencho.mesh.node_id': String(nodeId),
},
});
await container.start();
this.logActivity({
source: 'mesh', level: 'info', type: 'sidecar.start',
nodeId, message: `sidecar started for node ${nodeId}`,
});
} catch (err) {
this.logActivity({
source: 'mesh', level: 'error', type: 'sidecar.crash',
nodeId, message: `sidecar spawn failed: ${(err as Error).message}`,
});
throw err;
}
}
public async stopSidecar(nodeId: number): Promise<void> {
const docker = DockerController.getInstance(nodeId).getDocker();
const name = `${SIDECAR_CONTAINER_PREFIX}${nodeId}`;
try {
const c = docker.getContainer(name);
await c.stop({ t: 5 }).catch(() => undefined);
await c.remove({ force: true }).catch(() => undefined);
this.logActivity({
source: 'mesh', level: 'info', type: 'sidecar.stop',
nodeId, message: `sidecar stopped for node ${nodeId}`,
});
} catch { /* ignore */ }
}
private async isSidecarRunning(nodeId: number): Promise<boolean> {
try {
const docker = DockerController.getInstance(nodeId).getDocker();
const info = await docker.getContainer(`${SIDECAR_CONTAINER_PREFIX}${nodeId}`).inspect();
return !!info.State?.Running;
} catch {
return false;
}
}
public mintSidecarToken(nodeId: number): string {
const settings = DatabaseService.getInstance().getGlobalSettings();
const secret = settings.auth_jwt_secret;
if (!secret) throw new Error('JWT secret not configured');
return jwt.sign({ scope: 'mesh_sidecar', nodeId }, secret, { expiresIn: SIDECAR_TOKEN_TTL });
}
public verifySidecarToken(token: string): { nodeId: number } | null {
try {
const settings = DatabaseService.getInstance().getGlobalSettings();
const secret = settings.auth_jwt_secret;
if (!secret) return null;
const decoded = jwt.verify(token, secret) as { scope?: string; nodeId?: number };
if (decoded.scope !== 'mesh_sidecar' || typeof decoded.nodeId !== 'number') return null;
return { nodeId: decoded.nodeId };
} catch {
return null;
}
}
// --- Sidecar control WS attachment (called from websocket/meshControl.ts) ---
public attachSidecarSocket(ws: WebSocketLike, _nodeId: number): void {
this.sidecarSockets.add(ws);
ws.on('close', () => { this.sidecarSockets.delete(ws); });
}
/** Resolve an inbound sidecar request: "I have a connection on this port; who's it for?" */
public handleSidecarResolve(ws: WebSocketLike, nodeId: number, connId: number, port: number, remoteAddr: string): void {
const target = this.resolveByLocalPort(port);
if (!target) {
this.sendSidecar(ws, { t: 'resolve_err', connId, code: 'no_route', message: 'port not registered' });
this.logActivity({
source: 'sidecar', level: 'warn', type: 'route.resolve.denied',
nodeId, message: `unknown port ${port}`, details: { connId, remoteAddr },
});
return;
}
// The sidecar is the SOURCE; it asks for routing on its local node.
// Open a TCP path on this node's MeshService (same-node fast path or
// pilot tunnel) and acknowledge the resolve with a freshly allocated
// streamId. For V1 we do NOT bridge real bytes through the control WS
// until the sidecar package gains binary frame plumbing (Phase B).
// Instead we ack the resolve with the target metadata so the sidecar
// can dial directly on the host gateway.
this.sendSidecar(ws, { t: 'resolve_ok', connId, streamId: connId, alias: target.alias });
this.logActivity({
source: 'sidecar', level: 'info', type: 'route.resolve.ok',
nodeId, alias: target.alias,
message: `resolved port ${port} to ${target.alias}`,
details: { connId, remoteAddr },
});
}
private sendSidecar(ws: WebSocketLike, frame: Record<string, unknown>): void {
if (ws.readyState !== 1 /* OPEN */) return;
try { ws.send(JSON.stringify(frame)); } catch { /* ignore */ }
}
}
export class MeshError extends Error {
public readonly code: 'no_target' | 'port_collision' | 'denied' | 'agent_error';
constructor(code: 'no_target' | 'port_collision' | 'denied' | 'agent_error', message: string) {
super(message);
this.code = code;
}
}
@@ -83,6 +83,15 @@ export class PilotTunnelManager extends EventEmitter {
return this.bridges.has(nodeId);
}
/**
* Direct accessor for the per-node bridge. Returns null when no tunnel is
* registered. Used by Sencho Mesh to open TCP streams without going
* through the loopback HTTP path.
*/
public getBridge(nodeId: number): PilotTunnelBridge | null {
return this.bridges.get(nodeId) ?? null;
}
/**
* Force-close a tunnel (e.g., on node deletion).
*/
+57
View File
@@ -0,0 +1,57 @@
import type { IncomingMessage } from 'http';
import type { Duplex } from 'stream';
import type { WebSocketServer, WebSocket } from 'ws';
import { MeshService } from '../services/MeshService';
import { sanitizeForLog } from '../utils/safeLog';
import { rejectUpgrade as rejectSocket } from './reject';
/**
* Handle the local Sencho Mesh sidecar's control WebSocket. Authenticated
* with a `mesh_sidecar`-scoped JWT minted by MeshService when it spawned the
* sidecar; the JWT carries the node id the sidecar serves.
*
* The control WS is intentionally local-only: the sidecar runs in host
* network mode on the same Docker host as Sencho and reaches us via the
* loopback interface.
*/
export async function handleMeshControl(
req: IncomingMessage,
socket: Duplex,
head: Buffer,
wss: WebSocketServer,
): Promise<void> {
const authHeader = req.headers['authorization'];
const header = Array.isArray(authHeader) ? authHeader[0] : authHeader;
const token = header?.startsWith('Bearer ') ? header.slice(7) : null;
if (!token) return rejectSocket(socket, 401, 'Unauthorized');
const verified = MeshService.getInstance().verifySidecarToken(token);
if (!verified) return rejectSocket(socket, 401, 'Unauthorized');
wss.handleUpgrade(req, socket as never, head, (ws: WebSocket) => {
MeshService.getInstance().attachSidecarSocket(ws as unknown as never, verified.nodeId);
ws.on('message', (data, isBinary) => {
if (isBinary) return; // V1: control plane is JSON-only.
try {
const text = data.toString('utf8');
const frame = JSON.parse(text) as { t?: string; connId?: number; port?: number; remoteAddr?: string };
if (frame.t === 'resolve' && typeof frame.connId === 'number' && typeof frame.port === 'number') {
MeshService.getInstance().handleSidecarResolve(
ws as unknown as never,
verified.nodeId,
frame.connId,
frame.port,
frame.remoteAddr ?? '',
);
}
// hello / log / stream.stats / close are advisory; we accept
// them silently in V1. Future revisions can expand handling.
} catch (err) {
console.warn('[meshControl] bad frame:', sanitizeForLog((err as Error).message));
}
});
ws.on('error', () => { try { ws.close(); } catch { /* ignore */ } });
});
}
+7 -1
View File
@@ -7,6 +7,7 @@ import { DatabaseService, type UserRole } from '../services/DatabaseService';
import { NodeRegistry } from '../services/NodeRegistry';
import { COOKIE_NAME } from '../helpers/constants';
import { handlePilotTunnel } from './pilotTunnel';
import { handleMeshControl } from './meshControl';
import { handleNotificationsWs } from './notifications';
import { handleRemoteForwarder } from './remoteForwarder';
import { handleLogsWs } from './logs';
@@ -30,7 +31,8 @@ function parseCookies(req: IncomingMessage): Record<string, string> {
*
* Dispatch order (first match wins):
* 1. `/api/pilot/tunnel` -> handlePilotTunnel (own auth, own wss)
* 2. shared cookie/Bearer auth + JWT verify (rejects unauthenticated)
* 2. `/api/mesh/control` -> handleMeshControl (sidecar JWT, local-only)
* 3. shared cookie/Bearer auth + JWT verify (rejects unauthenticated)
* 3. API token scope gate (read-only / deploy-only restricted to logs + notifications)
* 4. `/ws/notifications` local -> handleNotificationsWs
* 5. remote nodeId path -> handleRemoteForwarder
@@ -54,6 +56,10 @@ export function attachUpgrade(
await handlePilotTunnel(req, socket, head, pilotTunnelWss);
return;
}
if (reqUrl.pathname === '/api/mesh/control') {
await handleMeshControl(req, socket, head, wss);
return;
}
} catch {
// URL parse error falls through and will be rejected below.
}
+1
View File
@@ -108,6 +108,7 @@
"features/host-console",
"features/multi-node",
"features/pilot-agent",
"features/sencho-mesh",
"features/fleet-view",
"features/fleet-sync",
"features/remote-updates",
+94
View File
@@ -0,0 +1,94 @@
---
title: Sencho Mesh
description: Connect containers across nodes by hostname over the Pilot tunnel — no VPN, no firewall changes, no extra ports.
---
<Note>
Sencho Mesh requires a Sencho **Admiral** license. Skipper and Community Edition do not include this feature.
</Note>
Sencho Mesh makes a multi-node fleet feel like one machine. Opt a stack into the mesh and its services become reachable from any other meshed stack on the fleet by a stable hostname. Traffic rides the existing Pilot tunnel, so there are no new ports to open and no separate VPN to manage.
## How it works
Each node runs a small `sencho-mesh` sidecar container that listens on the host's network. When you opt a stack into the mesh, Sencho:
1. Generates a Compose override file that injects `extra_hosts` for every cross-node alias the fleet currently exposes.
2. Redeploys the stack with the override applied so its containers pick up the new entries.
3. Routes incoming traffic on the alias's port through the local sidecar, then over the Pilot tunnel to the destination node, where the sidecar dials the target container.
Aliases follow a predictable scheme:
```
<service>.<stack>.<node>.sencho
```
For example, a Postgres `db` service in a stack named `api` on a node named `opsix` is reachable as `db.api.opsix.sencho` from any other meshed stack.
## Enable the mesh
1. Open **Fleet → Traffic · Routing**.
2. Toggle **mesh** on for each node you want to participate.
3. Click **Add stack to mesh** on any node and tick the stacks whose services should be reachable cross-node.
Each opt-in triggers an automatic redeploy of every other meshed stack on that node so the new alias is added to their hosts files. The opt-in sheet warns you before it does.
## What's exposed and what isn't
Four guarantees:
1. **Only opted-in services are reachable.** A stack reaches another stack's services through the mesh only if both stacks have explicitly opted in. The Pilot agent on the target node refuses any request for a non-opted service.
2. **Aliases are not internet-reachable.** Sidecars listen only on the host network of each node. Nothing about the mesh exposes new ports beyond the host's existing firewall posture.
3. **Traffic is encrypted in transit.** Cross-node bytes ride the existing Pilot WSS tunnel.
4. **Tier-gated and audit-logged.** Only Admiral users can configure the mesh. Enable, disable, opt-in, and opt-out events write durable rows to the audit log with the actor's identity.
What the mesh does **not** do:
- No application-layer authentication. Your Postgres still needs a password.
- No per-port firewall. Any opted-in stack can reach any other opted-in service.
- No rate limiting or quotas.
- No persistent traffic metrics. Live diagnostics only.
## Test upstream
Every alias row has a one-click **Test** button that runs a real probe across the same code path traffic uses. Result is shown inline:
- **Green tick** with round-trip time when the path is healthy.
- **Red badge** with the failing stage (`sidecar`, `pilot_tunnel`, `agent_resolve`, `agent_dial`, or `target_port`) when something is wrong.
Use the Test button before assuming an issue is your application's fault. It tells you whether the mesh path itself is the problem.
## Diagnostics
Every node card has a **Diagnostics** button that opens a live view of:
- Sidecar liveness and pilot tunnel state on this node.
- Active TCP streams with byte counters and open age.
- The resolver cache showing which aliases are registered.
- A **Restart sidecar** action.
This is the first place to look when a connection isn't behaving as expected.
## Mesh activity
The masthead has a **Mesh activity** button that opens the fleet-wide event log. Every route resolution, tunnel state change, opt-in, opt-out, and probe is recorded there. Filter by alias, source, type, or message. Useful for understanding what just happened when something flips state.
## V1 limitations
A few things are deliberately out of scope for the first release:
- **One alias per TCP port across the fleet.** If two stacks expose the same port (e.g. two Postgres instances on 5432), only the first can be added to the mesh. The opt-in sheet shows a clear inline error if the second tries.
- **Pilot-to-pilot routing is not supported.** Mesh works for traffic between the central node and its pilots in either direction. Two pilot nodes cannot reach each other through the mesh.
- **No TLS termination, no blue/green cutover.** Layer 7 features land in a follow-up.
## Troubleshooting
**A route shows `tunnel down`.** The Pilot tunnel to the target node is gone. Check **Fleet → Overview** for the node's status. Mesh recovers automatically when the tunnel reconnects.
**A route shows `unreachable`.** The tunnel is up but the destination port did not answer. Check that the target stack is running and that its service is listening on the declared port. Click **Test** to see the exact failing stage.
**A route shows `not authorized`.** The destination stack is not opted into the mesh on its home node. Open Routing on that node and add the stack.
**Adding a stack hangs.** Mesh redeploys peers on opt-in to refresh hostnames. A stuck redeploy usually means the stack itself failed to come back up. Check the stack's deploy logs.
**The sidecar shows `off` for a node.** Click **Diagnostics → Restart sidecar** on the node card. If the sidecar still does not come up, check the node's `docker ps -a` output for `sencho-mesh-<id>` and inspect its logs.
+19 -2
View File
@@ -4,7 +4,7 @@ import {
Layers, Wifi, WifiOff, Search, ArrowUpDown, AlertTriangle,
Play, Square, RotateCcw, ExternalLink, Camera, Download, Loader2, Check,
CircleCheck, CircleAlert, Globe, Monitor, X, LayoutGrid, Network, SlidersHorizontal,
Send, KeyRound,
Send, KeyRound, ArrowLeftRight,
} from 'lucide-react';
import { FleetMasthead } from './fleet/FleetMasthead';
import { FleetTopology } from './fleet/FleetTopology';
@@ -27,9 +27,11 @@ import { springs } from '@/lib/motion';
import { apiFetch, fetchForNode } from '@/lib/api';
import { useLicense } from '@/context/LicenseContext';
import { PaidGate } from './PaidGate';
import { AdmiralGate } from './AdmiralGate';
import FleetSnapshots from './FleetSnapshots';
import { FleetConfiguration } from './fleet/FleetConfiguration';
import { FleetSoonPlaceholder, SoonBadge } from './fleet/FleetSoonPlaceholder';
import { RoutingTab } from './fleet/RoutingTab';
import { toast } from '@/components/ui/toast-store';
import { LabelDot } from './LabelPill';
import { type Label as StackLabel, type LabelColor } from './label-types';
@@ -673,7 +675,8 @@ export function FleetView({ onNavigateToNode }: FleetViewProps) {
const [fleetPalette, setFleetPalette] = useState<FleetPaletteEntry[]>([]);
const [fleetStackLabelMap, setFleetStackLabelMap] = useState<Record<number, Record<string, StackLabel[]>>>({});
const [labelFilters, setLabelFilters] = useState<Set<string>>(new Set());
const { isPaid } = useLicense();
const { isPaid, license } = useLicense();
const isAdmiral = isPaid && license?.variant === 'admiral';
const [updateStatuses, setUpdateStatuses] = useState<NodeUpdateStatus[]>([]);
const [updatingNodeId, setUpdatingNodeId] = useState<number | null>(null);
const [reconnecting, setReconnecting] = useState(false);
@@ -1044,6 +1047,13 @@ export function FleetView({ onNavigateToNode }: FleetViewProps) {
</TabsTrigger>
</TabsHighlightItem>
)}
{isAdmiral && (
<TabsHighlightItem value="routing">
<TabsTrigger value="routing">
<ArrowLeftRight className="w-4 h-4 mr-1.5" />Traffic · Routing
</TabsTrigger>
</TabsHighlightItem>
)}
<TabsHighlightItem value="configuration">
<TabsTrigger value="configuration">
<SlidersHorizontal className="w-4 h-4 mr-1.5" />Status
@@ -1364,6 +1374,13 @@ export function FleetView({ onNavigateToNode }: FleetViewProps) {
<FleetSnapshots />
</TabsContent>
)}
{isAdmiral && (
<TabsContent value="routing">
<AdmiralGate featureName="Sencho Mesh">
<RoutingTab />
</AdmiralGate>
</TabsContent>
)}
<TabsContent value="configuration">
<FleetConfiguration />
</TabsContent>
@@ -0,0 +1,86 @@
import { useEffect, useState } from 'react';
import { apiFetch } from '@/lib/api';
import { Sheet, SheetContent, SheetHeader, SheetTitle } from '@/components/ui/sheet';
import { Input } from '@/components/ui/input';
import { Loader2 } from 'lucide-react';
import type { MeshActivityEvent } from '@/types/mesh';
interface Props {
open: boolean;
onOpenChange: (open: boolean) => void;
}
export function MeshActivitySheet({ open, onOpenChange }: Props) {
const [events, setEvents] = useState<MeshActivityEvent[]>([]);
const [filter, setFilter] = useState('');
const [loading, setLoading] = useState(false);
useEffect(() => {
if (!open) return;
let cancelled = false;
setLoading(true);
(async () => {
try {
const res = await apiFetch('/mesh/activity?limit=200', { localOnly: true });
if (!res.ok) return;
const body = await res.json() as { events: MeshActivityEvent[] };
if (!cancelled) setEvents(body.events);
} finally {
if (!cancelled) setLoading(false);
}
})();
return () => { cancelled = true; };
}, [open]);
const visible = events.filter((e) => {
if (!filter) return true;
const f = filter.toLowerCase();
return (
e.message.toLowerCase().includes(f) ||
(e.alias?.toLowerCase().includes(f) ?? false) ||
e.type.toLowerCase().includes(f)
);
});
return (
<Sheet open={open} onOpenChange={onOpenChange}>
<SheetContent side="right" className="w-[600px] sm:max-w-[600px]">
<SheetHeader>
<SheetTitle>Mesh activity</SheetTitle>
</SheetHeader>
<div className="mt-4 space-y-3">
<Input
placeholder="Filter by alias, type, or message"
value={filter}
onChange={(e) => setFilter(e.target.value)}
className="text-xs font-mono"
/>
<div className="max-h-[70vh] overflow-auto space-y-1">
{loading && (
<div className="flex items-center gap-2 text-stat-subtitle text-sm">
<Loader2 className="w-4 h-4 animate-spin" /> Loading
</div>
)}
{!loading && visible.length === 0 && (
<div className="text-xs text-stat-subtitle">No events.</div>
)}
{visible.slice().reverse().map((e, i) => (
<div key={i} className="grid grid-cols-[80px_70px_120px_1fr] gap-2 text-[11px] font-mono py-1 border-b border-card-border/50">
<span className="text-stat-subtitle tabular-nums">{new Date(e.ts).toLocaleTimeString()}</span>
<span className={
e.level === 'error' ? 'text-destructive uppercase tracking-wide' :
e.level === 'warn' ? 'text-warning uppercase tracking-wide' :
'text-stat-subtitle uppercase tracking-wide'
}>{e.source}</span>
<span className="text-stat-value">{e.type}</span>
<span className="text-stat-value truncate" title={e.message}>{e.alias ? `[${e.alias}] ` : ''}{e.message}</span>
</div>
))}
</div>
</div>
</SheetContent>
</Sheet>
);
}
@@ -0,0 +1,132 @@
import { useEffect, useState } from 'react';
import { apiFetch } from '@/lib/api';
import { toast } from '@/components/ui/toast-store';
import { Sheet, SheetContent, SheetHeader, SheetTitle } from '@/components/ui/sheet';
import { Button } from '@/components/ui/button';
import { Loader2, RefreshCw, ServerCog } from 'lucide-react';
import type { MeshNodeDiagnostic } from '@/types/mesh';
interface Props {
open: boolean;
onOpenChange: (open: boolean) => void;
nodeId: number | null;
nodeName: string | null;
}
function bytesFmt(n: number): string {
if (n < 1024) return `${n}B`;
if (n < 1024 * 1024) return `${(n / 1024).toFixed(1)}KB`;
return `${(n / (1024 * 1024)).toFixed(1)}MB`;
}
function ageFmt(ms: number): string {
if (ms < 1000) return `${ms}ms`;
if (ms < 60_000) return `${Math.floor(ms / 1000)}s`;
return `${Math.floor(ms / 60_000)}m`;
}
export function MeshDiagnosticsSheet({ open, onOpenChange, nodeId, nodeName }: Props) {
const [diag, setDiag] = useState<MeshNodeDiagnostic | null>(null);
const [loading, setLoading] = useState(false);
const [restarting, setRestarting] = useState(false);
const refresh = async () => {
if (nodeId == null) return;
setLoading(true);
try {
const res = await apiFetch(`/mesh/nodes/${nodeId}/diagnostic`, { localOnly: true });
if (res.ok) setDiag(await res.json());
} finally {
setLoading(false);
}
};
useEffect(() => {
if (open) void refresh();
// eslint-disable-next-line react-hooks/exhaustive-deps
}, [open, nodeId]);
const restart = async () => {
if (nodeId == null) return;
setRestarting(true);
try {
const res = await apiFetch(`/mesh/nodes/${nodeId}/sidecar/restart`, {
method: 'POST', localOnly: true,
});
if (res.ok) {
toast.success('Sidecar restart requested');
await refresh();
} else {
toast.error('Sidecar restart failed');
}
} finally {
setRestarting(false);
}
};
return (
<Sheet open={open} onOpenChange={onOpenChange}>
<SheetContent side="right" className="w-[520px] sm:max-w-[520px]">
<SheetHeader>
<SheetTitle className="flex items-center gap-2">
<ServerCog className="w-4 h-4" /> Diagnostics{nodeName ? ` · ${nodeName}` : ''}
</SheetTitle>
</SheetHeader>
<div className="mt-4 space-y-4">
<div className="flex items-center gap-2">
<Button variant="outline" size="sm" onClick={() => { void refresh(); }} disabled={loading}>
{loading ? <Loader2 className="w-3 h-3 mr-1 animate-spin" /> : <RefreshCw className="w-3 h-3 mr-1" />}
Refresh
</Button>
<Button variant="outline" size="sm" onClick={() => { void restart(); }} disabled={restarting}>
{restarting ? <Loader2 className="w-3 h-3 mr-1 animate-spin" /> : null}
Restart sidecar
</Button>
</div>
<div className="grid grid-cols-2 gap-2 rounded border border-card-border bg-card p-3 text-xs">
<div className="text-stat-subtitle">Sidecar</div>
<div className="font-mono text-stat-value">{diag?.sidecar.running ? 'running' : 'off'}</div>
<div className="text-stat-subtitle">Pilot tunnel</div>
<div className="font-mono text-stat-value">{diag?.pilot.connected ? 'connected' : 'disconnected'}</div>
<div className="text-stat-subtitle">Buffered</div>
<div className="font-mono text-stat-value">{diag ? bytesFmt(diag.pilot.bufferedAmount) : '-'}</div>
<div className="text-stat-subtitle">Last seen</div>
<div className="font-mono text-stat-value">{diag?.pilot.lastSeen ? new Date(diag.pilot.lastSeen).toLocaleTimeString() : '-'}</div>
</div>
<div>
<div className="text-[10px] tracking-wide uppercase text-stat-subtitle font-mono mb-2">Active streams</div>
{(!diag || diag.activeStreams.length === 0) && (
<div className="text-xs text-stat-subtitle">No active streams.</div>
)}
<div className="space-y-1">
{diag?.activeStreams.map((s) => (
<div key={s.streamId} className="flex justify-between rounded border border-card-border bg-card px-2 py-1 text-[11px] font-mono">
<span>#{s.streamId} {s.alias ?? '<no-alias>'}</span>
<span className="text-stat-subtitle">in {bytesFmt(s.bytesIn)} / out {bytesFmt(s.bytesOut)} · {ageFmt(s.ageMs)}</span>
</div>
))}
</div>
</div>
<div>
<div className="text-[10px] tracking-wide uppercase text-stat-subtitle font-mono mb-2">Resolver cache</div>
{(!diag || diag.aliasCache.length === 0) && (
<div className="text-xs text-stat-subtitle">No aliases registered.</div>
)}
<div className="space-y-1">
{diag?.aliasCache.map((a) => (
<div key={a.host} className="flex justify-between rounded border border-card-border bg-card px-2 py-1 text-[11px] font-mono">
<span>{a.host}</span>
<span className="text-stat-subtitle">node #{a.targetNodeId}:{a.port}</span>
</div>
))}
</div>
</div>
</div>
</SheetContent>
</Sheet>
);
}
@@ -0,0 +1,118 @@
import { useEffect, useState } from 'react';
import { apiFetch } from '@/lib/api';
import { toast } from '@/components/ui/toast-store';
import { Sheet, SheetContent, SheetHeader, SheetTitle, SheetDescription } from '@/components/ui/sheet';
import { Button } from '@/components/ui/button';
import { Checkbox } from '@/components/ui/checkbox';
import type { MeshStackEntry } from '@/types/mesh';
import { Loader2 } from 'lucide-react';
interface Props {
open: boolean;
onOpenChange: (open: boolean) => void;
nodeId: number;
nodeName: string;
onChanged: () => void;
}
export function MeshOptInSheet({ open, onOpenChange, nodeId, nodeName, onChanged }: Props) {
const [stacks, setStacks] = useState<MeshStackEntry[]>([]);
const [loading, setLoading] = useState(false);
const [error, setError] = useState<string | null>(null);
const [pendingStack, setPendingStack] = useState<string | null>(null);
useEffect(() => {
if (!open) return;
let cancelled = false;
(async () => {
setLoading(true);
setError(null);
try {
const res = await apiFetch(`/mesh/nodes/${nodeId}/stacks`, { localOnly: true });
if (!res.ok) throw new Error(`status ${res.status}`);
const data = await res.json() as { stacks: MeshStackEntry[] };
if (!cancelled) setStacks(data.stacks);
} catch (err) {
if (!cancelled) setError((err as Error).message);
} finally {
if (!cancelled) setLoading(false);
}
})();
return () => { cancelled = true; };
}, [open, nodeId]);
const toggle = async (stack: MeshStackEntry) => {
setPendingStack(stack.name);
setError(null);
try {
const action = stack.optedIn ? 'opt-out' : 'opt-in';
const res = await apiFetch(
`/mesh/nodes/${nodeId}/stacks/${encodeURIComponent(stack.name)}/${action}`,
{ method: 'POST', localOnly: true },
);
if (res.status === 409) {
const body = await res.json().catch(() => ({})) as { error?: string };
setError(body.error || 'Port already claimed by another mesh stack');
return;
}
if (!res.ok) throw new Error(`status ${res.status}`);
setStacks((prev) => prev.map((s) => s.name === stack.name ? { ...s, optedIn: !stack.optedIn } : s));
onChanged();
toast.success(stack.optedIn ? 'Stack removed from mesh' : 'Stack added to mesh');
} catch (err) {
setError((err as Error).message);
toast.error('Mesh update failed');
} finally {
setPendingStack(null);
}
};
return (
<Sheet open={open} onOpenChange={onOpenChange}>
<SheetContent side="right" className="w-[420px] sm:max-w-[420px]">
<SheetHeader>
<SheetTitle>Mesh stacks on {nodeName}</SheetTitle>
<SheetDescription>
Adding a stack lets its services be reached from other meshed stacks by hostname.
Toggling a stack redeploys it to refresh hostnames.
</SheetDescription>
</SheetHeader>
<div className="mt-4 space-y-2">
{loading && (
<div className="flex items-center gap-2 text-stat-subtitle text-sm">
<Loader2 className="w-4 h-4 animate-spin" /> Loading stacks
</div>
)}
{error && (
<div className="rounded border border-destructive/30 bg-destructive/10 p-2 text-xs text-destructive">
{error}
</div>
)}
{!loading && stacks.length === 0 && (
<div className="text-sm text-stat-subtitle">No stacks deployed on this node yet.</div>
)}
{stacks.map((stack) => (
<div key={stack.name} className="flex items-center justify-between rounded border border-card-border bg-card px-3 py-2">
<div className="flex items-center gap-3">
<Checkbox
id={`mesh-stack-${stack.name}`}
checked={stack.optedIn}
disabled={pendingStack === stack.name}
onCheckedChange={() => { void toggle(stack); }}
/>
<label htmlFor={`mesh-stack-${stack.name}`} className="text-sm font-mono">{stack.name}</label>
</div>
{pendingStack === stack.name && <Loader2 className="w-3 h-3 animate-spin text-stat-subtitle" />}
{stack.optedIn && pendingStack !== stack.name && (
<span className="text-[10px] tracking-wide uppercase text-success/80 font-mono">in mesh</span>
)}
</div>
))}
</div>
<div className="mt-6 flex justify-end">
<Button variant="outline" size="sm" onClick={() => onOpenChange(false)}>Close</Button>
</div>
</SheetContent>
</Sheet>
);
}
@@ -0,0 +1,139 @@
import { useEffect, useState } from 'react';
import { apiFetch } from '@/lib/api';
import { Sheet, SheetContent, SheetHeader, SheetTitle } from '@/components/ui/sheet';
import { Button } from '@/components/ui/button';
import { Badge } from '@/components/ui/badge';
import { Loader2, Activity, ServerCog, Hash } from 'lucide-react';
import type { MeshRouteDiagnostic, MeshActivityEvent, MeshProbeResult } from '@/types/mesh';
import { meshRouteStateFromBackend, meshRouteStateTokens } from './meshRouteState';
interface Props {
open: boolean;
onOpenChange: (open: boolean) => void;
alias: string | null;
}
export function MeshRouteDetailSheet({ open, onOpenChange, alias }: Props) {
const [diag, setDiag] = useState<MeshRouteDiagnostic | null>(null);
const [events, setEvents] = useState<MeshActivityEvent[]>([]);
const [probe, setProbe] = useState<MeshProbeResult | null>(null);
const [probing, setProbing] = useState(false);
const [loading, setLoading] = useState(false);
useEffect(() => {
if (!open || !alias) return;
let cancelled = false;
const refresh = async () => {
setLoading(true);
try {
const [diagRes, evRes] = await Promise.all([
apiFetch(`/mesh/aliases/${encodeURIComponent(alias)}/diagnostic`, { localOnly: true }),
apiFetch(`/mesh/activity?alias=${encodeURIComponent(alias)}&limit=20`, { localOnly: true }),
]);
if (cancelled) return;
if (diagRes.ok) setDiag(await diagRes.json());
if (evRes.ok) {
const body = await evRes.json() as { events: MeshActivityEvent[] };
setEvents(body.events);
}
} finally {
if (!cancelled) setLoading(false);
}
};
void refresh();
return () => { cancelled = true; };
}, [open, alias]);
const runProbe = async () => {
if (!alias) return;
setProbing(true);
setProbe(null);
try {
const res = await apiFetch(`/mesh/aliases/${encodeURIComponent(alias)}/test`, {
method: 'POST', localOnly: true,
});
const body = await res.json() as MeshProbeResult;
setProbe(body);
} finally {
setProbing(false);
}
};
if (!alias) return null;
const pillState = diag ? meshRouteStateFromBackend(diag.state) : 'not-authorized';
const pill = meshRouteStateTokens(pillState);
return (
<Sheet open={open} onOpenChange={onOpenChange}>
<SheetContent side="right" className="w-[480px] sm:max-w-[480px]">
<SheetHeader>
<SheetTitle className="font-mono text-sm">{alias}</SheetTitle>
</SheetHeader>
<div className="mt-4 space-y-4">
<div className="flex items-center gap-2">
<span className={`inline-flex items-center px-2 py-0.5 rounded-sm border text-[10px] font-mono uppercase tracking-wide ${pill.toneClass}`}>
{pill.label}
</span>
{diag?.lastProbeMs != null && (
<span className="text-[11px] font-mono text-stat-subtitle">{diag.lastProbeMs}ms</span>
)}
</div>
{diag?.target && (
<div className="grid grid-cols-2 gap-2 rounded border border-card-border bg-card p-3 text-xs">
<div className="text-stat-subtitle">Target node</div>
<div className="font-mono text-stat-value">#{diag.target.nodeId}</div>
<div className="text-stat-subtitle">Stack / service</div>
<div className="font-mono text-stat-value">{diag.target.stack}/{diag.target.service}</div>
<div className="text-stat-subtitle">Port</div>
<div className="font-mono text-stat-value">{diag.target.port}</div>
<div className="text-stat-subtitle">Pilot tunnel</div>
<div className="font-mono text-stat-value">{diag.pilot.connected ? 'connected' : 'disconnected'}</div>
</div>
)}
{diag?.lastError && (
<div className="rounded border border-destructive/30 bg-destructive/10 p-3 text-xs">
<div className="text-destructive font-mono uppercase tracking-wide text-[10px] mb-1">last error</div>
<div className="text-stat-value">{diag.lastError.message}</div>
<div className="text-[10px] text-stat-subtitle mt-1">{new Date(diag.lastError.ts).toLocaleString()}</div>
</div>
)}
<div className="flex items-center gap-2">
<Button size="sm" onClick={() => { void runProbe(); }} disabled={probing}>
{probing ? <Loader2 className="w-3 h-3 mr-1 animate-spin" /> : <Activity className="w-3 h-3 mr-1" />}
Test upstream
</Button>
{probe && (
<Badge variant={probe.ok ? 'default' : 'destructive'} className="text-[10px] font-mono">
{probe.ok ? `ok ${probe.latencyMs}ms` : `${probe.where ?? 'fail'}: ${probe.code ?? 'error'}`}
</Badge>
)}
</div>
<div>
<div className="text-[10px] tracking-wide uppercase text-stat-subtitle font-mono mb-2">Recent activity</div>
<div className="space-y-1 max-h-72 overflow-auto">
{loading && <Loader2 className="w-4 h-4 animate-spin text-stat-subtitle" />}
{!loading && events.length === 0 && (
<div className="text-xs text-stat-subtitle">No events yet for this alias.</div>
)}
{events.map((e, i) => (
<div key={i} className="flex items-start gap-2 text-[11px] font-mono">
{e.source === 'sidecar' && <ServerCog className="w-3 h-3 mt-0.5 text-stat-subtitle" />}
{e.source === 'pilot' && <Hash className="w-3 h-3 mt-0.5 text-stat-subtitle" />}
{e.source === 'mesh' && <Activity className="w-3 h-3 mt-0.5 text-stat-subtitle" />}
<span className={`tabular-nums ${e.level === 'error' ? 'text-destructive' : e.level === 'warn' ? 'text-warning' : 'text-stat-value'}`}>
{new Date(e.ts).toLocaleTimeString()} {e.type} {e.message}
</span>
</div>
))}
</div>
</div>
</div>
</SheetContent>
</Sheet>
);
}
@@ -0,0 +1,141 @@
import { useState } from 'react';
import { apiFetch } from '@/lib/api';
import { toast } from '@/components/ui/toast-store';
import { Card, CardContent } from '@/components/ui/card';
import { Button } from '@/components/ui/button';
import { TogglePill } from '@/components/ui/toggle-pill';
import { Plus, ServerCog, Activity, Loader2 } from 'lucide-react';
import type { MeshAlias, MeshNodeStatus } from '@/types/mesh';
import { meshRouteStateFor, meshRouteStateTokens } from './meshRouteState';
interface Props {
status: MeshNodeStatus;
aliases: MeshAlias[];
isLocal: boolean;
onAddStack: () => void;
onShowDiagnostics: () => void;
onShowAlias: (alias: string) => void;
onTestUpstream: (alias: string) => Promise<void>;
onChanged: () => void;
}
export function RoutingNodeCard({
status, aliases, isLocal, onAddStack, onShowDiagnostics, onShowAlias, onTestUpstream, onChanged,
}: Props) {
const [toggling, setToggling] = useState(false);
const [testingAlias, setTestingAlias] = useState<string | null>(null);
const nodeAliases = aliases.filter((a) => a.nodeId === status.nodeId);
const toggleEnabled = async (next: boolean) => {
setToggling(true);
try {
const action = next ? 'enable' : 'disable';
const res = await apiFetch(`/mesh/nodes/${status.nodeId}/${action}`, {
method: 'POST', localOnly: true,
});
if (!res.ok) throw new Error(`status ${res.status}`);
toast.success(next ? 'Mesh enabled on node' : 'Mesh disabled on node');
onChanged();
} catch (err) {
toast.error(`Failed to ${next ? 'enable' : 'disable'} mesh: ${(err as Error).message}`);
} finally {
setToggling(false);
}
};
const runTest = async (alias: string) => {
setTestingAlias(alias);
try { await onTestUpstream(alias); } finally { setTestingAlias(null); }
};
return (
<Card className="bg-card shadow-card-bevel">
<CardContent className="p-4 space-y-3">
<div className="flex items-center justify-between">
<div className="flex items-center gap-2">
<span className="font-mono text-sm">{status.nodeName}</span>
{isLocal && (
<span className="inline-flex items-center px-1.5 py-0.5 rounded-sm border border-brand/40 bg-brand/10 text-[9px] font-mono uppercase tracking-wide text-brand">
Local
</span>
)}
{!status.pilotConnected && status.nodeId !== -1 && !isLocal && (
<span className="inline-flex items-center px-1.5 py-0.5 rounded-sm border border-destructive/40 bg-destructive/10 text-[9px] font-mono uppercase tracking-wide text-destructive">
pilot offline
</span>
)}
</div>
<div className="flex items-center gap-2">
<TogglePill
checked={status.enabled}
disabled={toggling || (!status.pilotConnected && !isLocal)}
onChange={(next) => { void toggleEnabled(next); }}
/>
<Button variant="outline" size="sm" onClick={onShowDiagnostics}>
<ServerCog className="w-3 h-3 mr-1" /> Diagnostics
</Button>
</div>
</div>
{!status.pilotConnected && !isLocal && (
<div className="text-[11px] text-stat-subtitle">
Sencho Mesh requires the pilot agent on this node.
</div>
)}
<div className="grid grid-cols-2 gap-2 text-xs">
<div className="text-stat-subtitle">Mesh stacks</div>
<div className="font-mono text-stat-value">{status.optedInStacks.length}</div>
<div className="text-stat-subtitle">Aliases</div>
<div className="font-mono text-stat-value">{nodeAliases.length}</div>
</div>
{status.enabled && (
<>
<div className="border-t border-card-border pt-2 space-y-1">
{nodeAliases.length === 0 && (
<div className="text-[11px] text-stat-subtitle">No mesh services on this node yet.</div>
)}
{nodeAliases.map((a) => {
const pillState = meshRouteStateFor({
optedIn: true,
pilotConnected: status.pilotConnected,
});
const pill = meshRouteStateTokens(pillState);
return (
<div key={a.host} className="flex items-center justify-between gap-2 rounded border border-card-border bg-card px-2 py-1.5">
<button
type="button"
className="text-[11px] font-mono text-left truncate hover:text-brand transition-colors"
onClick={() => onShowAlias(a.host)}
>
{a.host}:{a.port}
</button>
<span className={`shrink-0 px-1.5 py-0.5 rounded-sm border text-[9px] font-mono uppercase tracking-wide ${pill.toneClass}`}>
{pill.label}
</span>
<Button
variant="ghost" size="sm"
onClick={() => { void runTest(a.host); }}
disabled={testingAlias === a.host}
className="h-6 px-1.5"
>
{testingAlias === a.host
? <Loader2 className="w-3 h-3 animate-spin" />
: <Activity className="w-3 h-3" />}
</Button>
</div>
);
})}
</div>
<Button variant="outline" size="sm" className="w-full" onClick={onAddStack}>
<Plus className="w-3 h-3 mr-1" /> Add stack to mesh
</Button>
</>
)}
</CardContent>
</Card>
);
}
@@ -0,0 +1,218 @@
import { useCallback, useEffect, useState } from 'react';
import { apiFetch } from '@/lib/api';
import { toast } from '@/components/ui/toast-store';
import { Button } from '@/components/ui/button';
import { ArrowLeftRight, Loader2, ScrollText } from 'lucide-react';
import { RoutingNodeCard } from './RoutingNodeCard';
import { MeshOptInSheet } from './MeshOptInSheet';
import { MeshRouteDetailSheet } from './MeshRouteDetailSheet';
import { MeshDiagnosticsSheet } from './MeshDiagnosticsSheet';
import { MeshActivitySheet } from './MeshActivitySheet';
import type { MeshAlias, MeshNodeStatus, MeshProbeResult } from '@/types/mesh';
export function RoutingTab() {
const [status, setStatus] = useState<MeshNodeStatus[]>([]);
const [aliases, setAliases] = useState<MeshAlias[]>([]);
const [loading, setLoading] = useState(true);
const [optInNode, setOptInNode] = useState<{ id: number; name: string } | null>(null);
const [diagnosticsNode, setDiagnosticsNode] = useState<{ id: number; name: string } | null>(null);
const [routeDetailAlias, setRouteDetailAlias] = useState<string | null>(null);
const [activityOpen, setActivityOpen] = useState(false);
const refresh = useCallback(async () => {
try {
const [statusRes, aliasesRes] = await Promise.all([
apiFetch('/mesh/status', { localOnly: true }),
apiFetch('/mesh/aliases', { localOnly: true }),
]);
if (statusRes.ok) {
const body = await statusRes.json() as { nodes: MeshNodeStatus[] };
setStatus(body.nodes);
}
if (aliasesRes.ok) {
const body = await aliasesRes.json() as { aliases: MeshAlias[] };
setAliases(body.aliases);
}
} catch (err) {
toast.error(`Failed to load mesh state: ${(err as Error).message}`);
} finally {
setLoading(false);
}
}, []);
useEffect(() => { void refresh(); }, [refresh]);
const testUpstream = useCallback(async (alias: string): Promise<void> => {
try {
const res = await apiFetch(`/mesh/aliases/${encodeURIComponent(alias)}/test`, {
method: 'POST', localOnly: true,
});
const body = await res.json() as MeshProbeResult;
if (body.ok) {
toast.success(`${alias} ok (${body.latencyMs}ms)`);
} else {
toast.error(`${alias} ${body.where ?? 'fail'}: ${body.code ?? 'error'}`);
}
} catch (err) {
toast.error(`Probe failed: ${(err as Error).message}`);
}
}, []);
const totalAliases = aliases.length;
const meshedNodes = status.filter((s) => s.enabled).length;
const onlineNodes = status.filter((s) => s.pilotConnected).length;
if (loading) {
return (
<div className="flex items-center justify-center py-12 text-stat-subtitle">
<Loader2 className="w-5 h-5 animate-spin mr-2" />
Loading mesh state
</div>
);
}
if (status.length === 0) {
return (
<div className="flex flex-col items-center justify-center py-16">
<ArrowLeftRight className="w-12 h-12 text-stat-subtitle mb-4" />
<div className="text-lg font-display italic mb-2">No nodes available</div>
<div className="text-sm text-stat-subtitle text-center max-w-md">
Add a node to the fleet to start routing traffic between containers across nodes.
</div>
</div>
);
}
if (meshedNodes === 0) {
return (
<div className="space-y-4">
<RoutingMasthead meshedNodes={meshedNodes} onlineNodes={onlineNodes} totalAliases={totalAliases} onShowActivity={() => setActivityOpen(true)} />
<div className="flex flex-col items-center justify-center py-12 rounded border border-dashed border-card-border bg-card/50">
<ArrowLeftRight className="w-12 h-12 text-stat-subtitle mb-4" />
<div className="text-lg font-display italic mb-2">Mesh containers across nodes</div>
<div className="text-sm text-stat-subtitle text-center max-w-md mb-6">
Add a stack to the mesh and its services become reachable from any other meshed
stack by hostname. No VPN, no firewall changes.
</div>
<div className="grid grid-cols-1 md:grid-cols-2 xl:grid-cols-3 gap-4 w-full">
{status.map((s) => (
<RoutingNodeCard
key={s.nodeId}
status={s}
aliases={aliases}
isLocal={s.nodeId === status[0]?.nodeId && s.pilotConnected}
onAddStack={() => setOptInNode({ id: s.nodeId, name: s.nodeName })}
onShowDiagnostics={() => setDiagnosticsNode({ id: s.nodeId, name: s.nodeName })}
onShowAlias={(alias) => setRouteDetailAlias(alias)}
onTestUpstream={testUpstream}
onChanged={() => { void refresh(); }}
/>
))}
</div>
</div>
<SheetsRoot
optInNode={optInNode} setOptInNode={setOptInNode}
diagnosticsNode={diagnosticsNode} setDiagnosticsNode={setDiagnosticsNode}
routeDetailAlias={routeDetailAlias} setRouteDetailAlias={setRouteDetailAlias}
activityOpen={activityOpen} setActivityOpen={setActivityOpen}
onChanged={() => { void refresh(); }}
/>
</div>
);
}
return (
<div className="space-y-4">
<RoutingMasthead meshedNodes={meshedNodes} onlineNodes={onlineNodes} totalAliases={totalAliases} onShowActivity={() => setActivityOpen(true)} />
<div className="grid grid-cols-1 md:grid-cols-2 xl:grid-cols-3 gap-4">
{status.map((s) => (
<RoutingNodeCard
key={s.nodeId}
status={s}
aliases={aliases}
isLocal={s.nodeId === status[0]?.nodeId && s.pilotConnected}
onAddStack={() => setOptInNode({ id: s.nodeId, name: s.nodeName })}
onShowDiagnostics={() => setDiagnosticsNode({ id: s.nodeId, name: s.nodeName })}
onShowAlias={(alias) => setRouteDetailAlias(alias)}
onTestUpstream={testUpstream}
onChanged={() => { void refresh(); }}
/>
))}
</div>
<SheetsRoot
optInNode={optInNode} setOptInNode={setOptInNode}
diagnosticsNode={diagnosticsNode} setDiagnosticsNode={setDiagnosticsNode}
routeDetailAlias={routeDetailAlias} setRouteDetailAlias={setRouteDetailAlias}
activityOpen={activityOpen} setActivityOpen={setActivityOpen}
onChanged={() => { void refresh(); }}
/>
</div>
);
}
function RoutingMasthead({ meshedNodes, onlineNodes, totalAliases, onShowActivity }: {
meshedNodes: number; onlineNodes: number; totalAliases: number; onShowActivity: () => void;
}) {
const stateWord = meshedNodes === 0 ? 'unmeshed' : meshedNodes < onlineNodes ? 'partial' : 'meshed';
return (
<div className="flex items-center justify-between rounded-lg border border-card-border bg-card p-4 shadow-card-bevel">
<div className="flex items-center gap-4">
<div className="font-display italic text-2xl">{stateWord}</div>
<div className="grid grid-cols-3 gap-4 text-xs">
<div>
<div className="text-[10px] tracking-wide uppercase text-stat-subtitle font-mono">meshed</div>
<div className="font-mono text-stat-value">{meshedNodes}/{onlineNodes}</div>
</div>
<div>
<div className="text-[10px] tracking-wide uppercase text-stat-subtitle font-mono">aliases</div>
<div className="font-mono text-stat-value">{totalAliases}</div>
</div>
</div>
</div>
<Button variant="outline" size="sm" onClick={onShowActivity}>
<ScrollText className="w-3 h-3 mr-1" /> Mesh activity
</Button>
</div>
);
}
function SheetsRoot(props: {
optInNode: { id: number; name: string } | null;
setOptInNode: (v: { id: number; name: string } | null) => void;
diagnosticsNode: { id: number; name: string } | null;
setDiagnosticsNode: (v: { id: number; name: string } | null) => void;
routeDetailAlias: string | null;
setRouteDetailAlias: (v: string | null) => void;
activityOpen: boolean;
setActivityOpen: (v: boolean) => void;
onChanged: () => void;
}) {
return (
<>
{props.optInNode && (
<MeshOptInSheet
open={!!props.optInNode}
onOpenChange={(open) => { if (!open) props.setOptInNode(null); }}
nodeId={props.optInNode.id}
nodeName={props.optInNode.name}
onChanged={props.onChanged}
/>
)}
<MeshDiagnosticsSheet
open={!!props.diagnosticsNode}
onOpenChange={(open) => { if (!open) props.setDiagnosticsNode(null); }}
nodeId={props.diagnosticsNode?.id ?? null}
nodeName={props.diagnosticsNode?.name ?? null}
/>
<MeshRouteDetailSheet
open={!!props.routeDetailAlias}
onOpenChange={(open) => { if (!open) props.setRouteDetailAlias(null); }}
alias={props.routeDetailAlias}
/>
<MeshActivitySheet
open={props.activityOpen}
onOpenChange={props.setActivityOpen}
/>
</>
);
}
@@ -0,0 +1,62 @@
import { describe, expect, it } from 'vitest';
import { meshRouteStateFor, meshRouteStateFromBackend, meshRouteStateTokens } from './meshRouteState';
describe('meshRouteStateFor', () => {
const FROZEN_NOW = 1_700_000_000_000;
it('returns not-authorized when the route is not opted in', () => {
expect(meshRouteStateFor({ optedIn: false, pilotConnected: true, now: FROZEN_NOW })).toBe('not-authorized');
});
it('returns tunnel-down when opted in but pilot is offline', () => {
expect(meshRouteStateFor({ optedIn: true, pilotConnected: false, now: FROZEN_NOW })).toBe('tunnel-down');
});
it('returns unreachable when a recent error is recorded', () => {
const lastErrorAt = FROZEN_NOW - 5_000;
expect(meshRouteStateFor({
optedIn: true, pilotConnected: true, lastErrorAt, now: FROZEN_NOW,
})).toBe('unreachable');
});
it('does NOT return unreachable when the error is older than the window', () => {
const lastErrorAt = FROZEN_NOW - 90_000;
expect(meshRouteStateFor({
optedIn: true, pilotConnected: true, lastErrorAt, now: FROZEN_NOW,
})).toBe('healthy');
});
it('returns degraded when the last probe was slow', () => {
expect(meshRouteStateFor({
optedIn: true, pilotConnected: true, lastProbeMs: 750, now: FROZEN_NOW,
})).toBe('degraded');
});
it('returns healthy in the happy path', () => {
expect(meshRouteStateFor({
optedIn: true, pilotConnected: true, lastProbeMs: 12, now: FROZEN_NOW,
})).toBe('healthy');
});
});
describe('meshRouteStateFromBackend', () => {
it('maps each backend label to its pill key', () => {
expect(meshRouteStateFromBackend('healthy')).toBe('healthy');
expect(meshRouteStateFromBackend('degraded')).toBe('degraded');
expect(meshRouteStateFromBackend('unreachable')).toBe('unreachable');
expect(meshRouteStateFromBackend('tunnel down')).toBe('tunnel-down');
expect(meshRouteStateFromBackend('not authorized')).toBe('not-authorized');
expect(meshRouteStateFromBackend('anything-else')).toBe('not-authorized');
});
});
describe('meshRouteStateTokens', () => {
it('produces a token + label + toneClass for every pill state', () => {
for (const s of ['healthy', 'degraded', 'unreachable', 'tunnel-down', 'not-authorized'] as const) {
const tk = meshRouteStateTokens(s);
expect(tk.label.length).toBeGreaterThan(0);
expect(tk.toneClass.length).toBeGreaterThan(0);
expect(tk.token.length).toBeGreaterThan(0);
}
});
});
@@ -0,0 +1,47 @@
import type { MeshRoutePillState } from '@/types/mesh';
const SLOW_PROBE_THRESHOLD_MS = 500;
const RECENT_ERROR_WINDOW_MS = 60_000;
export interface MeshRouteSignals {
optedIn: boolean;
pilotConnected: boolean;
lastErrorAt?: number | null;
lastProbeMs?: number | null;
now?: number;
}
/**
* Pure function mapping the four observable signals to a single pill state.
* Centralizes the priority order so every UI surface (route row, node card,
* detail sheet, activity log) renders the same word for the same situation.
*/
export function meshRouteStateFor(signals: MeshRouteSignals): MeshRoutePillState {
if (!signals.optedIn) return 'not-authorized';
if (!signals.pilotConnected) return 'tunnel-down';
const now = signals.now ?? Date.now();
if (signals.lastErrorAt && now - signals.lastErrorAt < RECENT_ERROR_WINDOW_MS) return 'unreachable';
if (signals.lastProbeMs != null && signals.lastProbeMs > SLOW_PROBE_THRESHOLD_MS) return 'degraded';
return 'healthy';
}
const TONE_TOKENS: Record<MeshRoutePillState, { token: string; label: string; toneClass: string }> = {
'healthy': { token: 'success', label: 'healthy', toneClass: 'text-success bg-success/10 border-success/30' },
'degraded': { token: 'warning', label: 'degraded', toneClass: 'text-warning bg-warning/10 border-warning/30' },
'unreachable': { token: 'destructive', label: 'unreachable', toneClass: 'text-destructive bg-destructive/10 border-destructive/30' },
'tunnel-down': { token: 'destructive', label: 'tunnel down', toneClass: 'text-destructive bg-destructive/10 border-destructive/30' },
'not-authorized': { token: 'muted', label: 'not authorized', toneClass: 'text-ink-3 bg-ink-3/10 border-ink-3/30' },
};
export function meshRouteStateTokens(state: MeshRoutePillState) {
return TONE_TOKENS[state];
}
/** Convert the backend's diagnostic state string to the pill key. */
export function meshRouteStateFromBackend(state: string): MeshRoutePillState {
if (state === 'healthy') return 'healthy';
if (state === 'degraded') return 'degraded';
if (state === 'unreachable') return 'unreachable';
if (state === 'tunnel down') return 'tunnel-down';
return 'not-authorized';
}
+68
View File
@@ -0,0 +1,68 @@
export type MeshRoutePillState = 'healthy' | 'degraded' | 'unreachable' | 'tunnel-down' | 'not-authorized';
export interface MeshAlias {
host: string;
nodeId: number;
nodeName: string;
stackName: string;
serviceName: string;
port: number;
}
export interface MeshNodeStatus {
nodeId: number;
nodeName: string;
enabled: boolean;
sidecarRunning: boolean;
pilotConnected: boolean;
optedInStacks: string[];
activeStreamCount: number;
}
export interface MeshRouteDiagnostic {
alias: string;
target: {
nodeId: number;
stack: string;
service: string;
port: number;
alias: string;
} | null;
pilot: { connected: boolean; lastSeen: number | null };
lastError: { ts: number; message: string } | null;
lastProbeMs: number | null;
state: 'healthy' | 'degraded' | 'unreachable' | 'tunnel down' | 'not authorized';
}
export interface MeshNodeDiagnostic {
nodeId: number;
sidecar: { running: boolean; restartCount: number };
pilot: { connected: boolean; bufferedAmount: number; lastSeen: number | null };
activeStreams: Array<{ streamId: number; alias?: string; bytesIn: number; bytesOut: number; ageMs: number }>;
aliasCache: Array<{ host: string; targetNodeId: number; port: number }>;
}
export interface MeshProbeResult {
ok: boolean;
latencyMs?: number;
where?: 'sidecar' | 'pilot_tunnel' | 'agent_resolve' | 'agent_dial' | 'target_port';
code?: string;
message?: string;
}
export interface MeshActivityEvent {
ts: number;
source: 'sidecar' | 'pilot' | 'mesh';
level: 'info' | 'warn' | 'error';
type: string;
nodeId?: number;
alias?: string;
streamId?: number;
message: string;
details?: Record<string, unknown>;
}
export interface MeshStackEntry {
name: string;
optedIn: boolean;
}