feat(mesh): replace host-mode with shared sencho_mesh Docker network (#1009)

* feat(mesh): replace host-mode with shared sencho_mesh Docker network

Phase D of the mesh redesign: drop the operator's `network_mode: host`
requirement and the `host-gateway` extra_hosts pattern that did not work
on cloud iptables-restrictive distros (OCI, etc.) or Docker Desktop.

Each Sencho creates a shared `sencho_mesh` Docker bridge network on
boot (default subnet 172.30.0.0/24, override via SENCHO_MESH_SUBNET),
pins itself at `<network>+2`, and attaches every meshed user service to
the same bridge. Compose overrides now emit IP-based `extra_hosts` plus
a top-level `networks` block declaring `sencho_mesh` external.

Override delivery: central renders for local stacks; for remote stacks
it sends the fleet alias list to the remote's new `PUT /api/mesh/local-
override/:stackName` endpoint, which renders against the remote's OWN
local senchoIp and writes under its OWN DATA_DIR. Each node may use a
different subnet without coordination beyond the env var.

Opt-in / opt-out now trigger an automatic redeploy of the affected
stack via the existing deploy code path (local: ComposeService; remote:
HTTP POST through proxyFetch). The frontend opt-in sheet shows a
confirmation modal (ConfirmModal) before the mutation. Failed
redeploys emit both a mesh activity event and a durable audit-log row.

Hardening:
- Reserve port 1852 at opt-in (prevents user containers from racing
  the Sencho API listener).
- ensureMeshNetwork refuses to continue if `sencho_mesh` exists with a
  mismatched subnet rather than silently routing to the wrong IP.
- Idempotent network connect/disconnect helpers in DockerController.
- optInStack rolls back the DB row if the just-inserted stack's
  override push fails (no half-states surviving across calls).
- regenerateOverridesForNode runs in parallel and skips the just-
  pushed stack on opt-in.

Operator template: drop `network_mode: host`, restore
`ports: ["1852:1852"]`. Mesh now works identically on Linux LAN, OCI,
and Docker Desktop without firewall changes.

Docs: rewrite docs/features/sencho-mesh.mdx around the shared bridge
network, document SENCHO_MESH_SUBNET, surface the host-network-service
opt-in restriction, and cross-link with the Pilot Agent docs.

BREAKING CHANGE: the operator's `docker-compose.yml` no longer uses
`network_mode: host`. After upgrading, redeploy any meshed stacks once
so they pick up the new IP-based override and join `sencho_mesh`.

* fix(mesh): wrap stackName with path.basename in local-override fs ops

CodeQL flagged js/path-injection on the new applyLocalOverride and
removeLocalOverride methods because they are publicly reachable and
its data-flow model does not recognize isValidStackName /
isPathWithinBase as sanitizers. The validation IS sufficient (the
allowlist regex blocks path separators, the path-prefix check blocks
escape), but path.basename is a model CodeQL recognizes and is purely
defensive: for any input that already passes isValidStackName,
basename is the identity.

backend/src/__tests__/docker-controller.test.ts

@@ -663,3 +663,90 @@ describe('removeContainers', () => {
     expect(results).toEqual([]);
   });
 });
+
+// ── Network connect / disconnect helpers ───────────────────────────────
+
+describe('DockerController - connectContainerToNetwork', () => {
+  it('attaches a container to a network with no static IP', async () => {
+    const connect = vi.fn().mockResolvedValue(undefined);
+    mockDocker.getNetwork.mockReturnValue({ connect });
+
+    const dc = DockerController.getInstance(1);
+    await dc.connectContainerToNetwork('sencho_mesh', 'sencho-host-1234');
+
+    expect(mockDocker.getNetwork).toHaveBeenCalledWith('sencho_mesh');
+    expect(connect).toHaveBeenCalledWith({ Container: 'sencho-host-1234' });
+  });
+
+  it('attaches with a static IPv4 address when provided', async () => {
+    const connect = vi.fn().mockResolvedValue(undefined);
+    mockDocker.getNetwork.mockReturnValue({ connect });
+
+    const dc = DockerController.getInstance(1);
+    await dc.connectContainerToNetwork('sencho_mesh', 'sencho-host-1234', { ipv4Address: '172.30.0.2' });
+
+    expect(connect).toHaveBeenCalledWith({
+      Container: 'sencho-host-1234',
+      EndpointConfig: { IPAMConfig: { IPv4Address: '172.30.0.2' } },
+    });
+  });
+
+  it('treats 403 already-connected as success (idempotent)', async () => {
+    const connect = vi.fn().mockRejectedValue({ statusCode: 403, message: 'endpoint already exists' });
+    mockDocker.getNetwork.mockReturnValue({ connect });
+
+    const dc = DockerController.getInstance(1);
+    await expect(dc.connectContainerToNetwork('sencho_mesh', 'sencho-host-1234')).resolves.toBeUndefined();
+  });
+
+  it('rethrows non-idempotent errors', async () => {
+    const connect = vi.fn().mockRejectedValue({ statusCode: 500, message: 'server error' });
+    mockDocker.getNetwork.mockReturnValue({ connect });
+
+    const dc = DockerController.getInstance(1);
+    await expect(dc.connectContainerToNetwork('sencho_mesh', 'sencho-host-1234')).rejects.toMatchObject({
+      statusCode: 500,
+    });
+  });
+
+  it('rethrows a 403 whose message is unrelated to already-attached', async () => {
+    const connect = vi.fn().mockRejectedValue({ statusCode: 403, message: 'host-mode container cannot join network' });
+    mockDocker.getNetwork.mockReturnValue({ connect });
+
+    const dc = DockerController.getInstance(1);
+    await expect(dc.connectContainerToNetwork('sencho_mesh', 'sencho-host-1234')).rejects.toMatchObject({
+      statusCode: 403,
+    });
+  });
+});
+
+describe('DockerController - disconnectContainerFromNetwork', () => {
+  it('detaches a container from a network with force=true', async () => {
+    const disconnect = vi.fn().mockResolvedValue(undefined);
+    mockDocker.getNetwork.mockReturnValue({ disconnect });
+
+    const dc = DockerController.getInstance(1);
+    await dc.disconnectContainerFromNetwork('sencho_mesh', 'sencho-host-1234');
+
+    expect(mockDocker.getNetwork).toHaveBeenCalledWith('sencho_mesh');
+    expect(disconnect).toHaveBeenCalledWith({ Container: 'sencho-host-1234', Force: true });
+  });
+
+  it('treats 404 not-connected as success (idempotent)', async () => {
+    const disconnect = vi.fn().mockRejectedValue({ statusCode: 404, message: 'no such network endpoint' });
+    mockDocker.getNetwork.mockReturnValue({ disconnect });
+
+    const dc = DockerController.getInstance(1);
+    await expect(dc.disconnectContainerFromNetwork('sencho_mesh', 'sencho-host-1234')).resolves.toBeUndefined();
+  });
+
+  it('rethrows non-idempotent errors', async () => {
+    const disconnect = vi.fn().mockRejectedValue({ statusCode: 500, message: 'server error' });
+    mockDocker.getNetwork.mockReturnValue({ disconnect });
+
+    const dc = DockerController.getInstance(1);
+    await expect(dc.disconnectContainerFromNetwork('sencho_mesh', 'sencho-host-1234')).rejects.toMatchObject({
+      statusCode: 500,
+    });
+  });
+});

backend/src/__tests__/mesh-compose-override.test.ts

@@ -1,36 +1,64 @@
 import { describe, expect, it } from 'vitest';
 import * as YAML from 'yaml';
-import { buildAliasHosts, generateOverrideYaml } from '../services/MeshComposeOverride';
+import { buildAliasHosts, generateOverrideYaml, SENCHO_MESH_NETWORK } from '../services/MeshComposeOverride';
+
+const SENCHO_IP = '172.30.0.2';
 
 describe('generateOverrideYaml', () => {
-    it('emits services with extra_hosts pointing to host-gateway', () => {
+    it('emits services with extra_hosts pointing to the Sencho mesh IP', () => {
         const yaml = generateOverrideYaml({
             services: ['web', 'cache'],
             aliases: [
                 { host: 'db.api.opsix.sencho' },
                 { host: 'etl.worker.opsix.sencho' },
             ],
+            senchoIp: SENCHO_IP,
         });
         const parsed = YAML.parse(yaml) as Record<string, unknown>;
-        expect(parsed.networks).toBeUndefined();
-        const services = parsed.services as Record<string, { extra_hosts: string[] }>;
+        const services = parsed.services as Record<string, { extra_hosts: string[]; networks: string[] }>;
         expect(Object.keys(services).sort()).toEqual(['cache', 'web']);
         for (const svc of ['web', 'cache']) {
             expect(services[svc].extra_hosts).toEqual([
-                'db.api.opsix.sencho:host-gateway',
-                'etl.worker.opsix.sencho:host-gateway',
+                `db.api.opsix.sencho:${SENCHO_IP}`,
+                `etl.worker.opsix.sencho:${SENCHO_IP}`,
             ]);
         }
     });
 
-    it('emits empty service stubs when no aliases exist yet', () => {
+    it('attaches every service to the sencho_mesh network', () => {
+        const yaml = generateOverrideYaml({
+            services: ['web', 'cache', 'worker'],
+            aliases: [{ host: 'db.api.opsix.sencho' }],
+            senchoIp: SENCHO_IP,
+        });
+        const parsed = YAML.parse(yaml) as Record<string, unknown>;
+        const services = parsed.services as Record<string, { networks: string[] }>;
+        for (const svc of ['web', 'cache', 'worker']) {
+            expect(services[svc].networks).toEqual([SENCHO_MESH_NETWORK]);
+        }
+    });
+
+    it('declares sencho_mesh as an external network at the top level', () => {
         const yaml = generateOverrideYaml({
             services: ['web'],
             aliases: [],
+            senchoIp: SENCHO_IP,
         });
         const parsed = YAML.parse(yaml) as Record<string, unknown>;
-        const services = parsed.services as Record<string, { extra_hosts?: string[] }>;
+        const networks = parsed.networks as Record<string, { external: boolean }>;
+        expect(networks[SENCHO_MESH_NETWORK]).toEqual({ external: true });
+    });
+
+    it('still attaches services to the network when no aliases exist yet', () => {
+        const yaml = generateOverrideYaml({
+            services: ['web'],
+            aliases: [],
+            senchoIp: SENCHO_IP,
+        });
+        const parsed = YAML.parse(yaml) as Record<string, unknown>;
+        const services = parsed.services as Record<string, { extra_hosts?: string[]; networks: string[] }>;
         expect(services.web.extra_hosts).toBeUndefined();
+        expect(services.web.networks).toEqual([SENCHO_MESH_NETWORK]);
     });
 
     it('produces stable output regardless of input ordering', () => {
@@ -40,6 +68,7 @@ describe('generateOverrideYaml', () => {
                 { host: 'b.x.y.sencho' },
                 { host: 'a.x.y.sencho' },
             ],
+            senchoIp: SENCHO_IP,
         });
         const b = generateOverrideYaml({
             services: ['cache', 'web'],
@@ -47,9 +76,20 @@ describe('generateOverrideYaml', () => {
                 { host: 'a.x.y.sencho' },
                 { host: 'b.x.y.sencho' },
             ],
+            senchoIp: SENCHO_IP,
         });
         expect(a).toBe(b);
     });
+
+    it('uses the senchoIp argument verbatim in extra_hosts entries', () => {
+        const customIp = '10.42.7.99';
+        const yaml = generateOverrideYaml({
+            services: ['web'],
+            aliases: [{ host: 'svc.stack.node.sencho' }],
+            senchoIp: customIp,
+        });
+        expect(yaml).toContain(`svc.stack.node.sencho:${customIp}`);
+    });
 });
 
 describe('buildAliasHosts', () => {

backend/src/__tests__/mesh-service.test.ts

@@ -1,5 +1,6 @@
 import { afterAll, beforeAll, beforeEach, describe, expect, it, vi } from 'vitest';
 import { setupTestDb, cleanupTestDb } from './helpers/setupTestDb';
+import { getSenchoIpFromSubnet } from '../services/MeshService';
 
 let tmpDir: string;
 let MeshService: typeof import('../services/MeshService').MeshService;
@@ -26,6 +27,9 @@ beforeEach(() => {
         activeStreams: Map<number, unknown>;
         routeErrorMap: Map<string, unknown>;
         routeLatencyMap: Map<string, unknown>;
+        senchoIp: string | null;
+        meshSubnet: string;
+        networkSetupError: string | null;
     };
     svc.aliasCache = new Map();
     svc.aliasByPort = new Map();
@@ -33,6 +37,9 @@ beforeEach(() => {
     svc.activeStreams = new Map();
     svc.routeErrorMap = new Map();
     svc.routeLatencyMap = new Map();
+    svc.senchoIp = '172.30.0.2';
+    svc.meshSubnet = '172.30.0.0/24';
+    svc.networkSetupError = null;
     vi.restoreAllMocks();
 });
 
@@ -227,3 +234,114 @@ describe('MeshService.testUpstream tunnel-down path', () => {
         expect(result.code).toBe('no_route');
     });
 });
+
+describe('getSenchoIpFromSubnet', () => {
+    it('returns network+2 for the default /24', () => {
+        expect(getSenchoIpFromSubnet('172.30.0.0/24')).toBe('172.30.0.2');
+    });
+
+    it('handles a custom /24 in a different range', () => {
+        expect(getSenchoIpFromSubnet('10.42.7.0/24')).toBe('10.42.7.2');
+    });
+
+    it('handles a /16', () => {
+        expect(getSenchoIpFromSubnet('172.30.0.0/16')).toBe('172.30.0.2');
+    });
+
+    it('masks the input IP to the network address before adding 2', () => {
+        // 172.30.0.50/24 → network 172.30.0.0 → +2 = 172.30.0.2
+        expect(getSenchoIpFromSubnet('172.30.0.50/24')).toBe('172.30.0.2');
+    });
+
+    it('rejects a malformed CIDR', () => {
+        expect(() => getSenchoIpFromSubnet('not-a-cidr')).toThrow(/Invalid mesh subnet/);
+        expect(() => getSenchoIpFromSubnet('172.30.0.0')).toThrow(/Invalid mesh subnet/);
+    });
+
+    it('rejects prefixes too narrow to host two addresses', () => {
+        expect(() => getSenchoIpFromSubnet('172.30.0.0/31')).toThrow(/Invalid mesh subnet/);
+    });
+
+    it('rejects out-of-range octets', () => {
+        expect(() => getSenchoIpFromSubnet('172.30.0.999/24')).toThrow(/Invalid mesh subnet/);
+    });
+});
+
+describe('MeshService.ensureMeshNetwork', () => {
+    it('refuses to continue when sencho_mesh exists with a different subnet', async () => {
+        const svc = MeshService.getInstance();
+        const dcModule = await import('../services/DockerController');
+        const fakeController = {
+            createNetwork: vi.fn().mockRejectedValue({ statusCode: 409, message: 'network already exists' }),
+            inspectNetwork: vi.fn().mockResolvedValue({ IPAM: { Config: [{ Subnet: '10.99.0.0/24' }] } }),
+        };
+        vi.spyOn(dcModule.default, 'getInstance').mockReturnValue(fakeController as unknown as ReturnType<typeof dcModule.default.getInstance>);
+
+        await expect(
+            (svc as unknown as { ensureMeshNetwork: (s: string) => Promise<void> }).ensureMeshNetwork('172.30.0.0/24'),
+        ).rejects.toThrow(/exists with subnet 10\.99\.0\.0\/24/);
+    });
+
+    it('treats 409 with matching subnet as idempotent success', async () => {
+        const svc = MeshService.getInstance();
+        const dcModule = await import('../services/DockerController');
+        const fakeController = {
+            createNetwork: vi.fn().mockRejectedValue({ statusCode: 409, message: 'network already exists' }),
+            inspectNetwork: vi.fn().mockResolvedValue({ IPAM: { Config: [{ Subnet: '172.30.0.0/24' }] } }),
+        };
+        vi.spyOn(dcModule.default, 'getInstance').mockReturnValue(fakeController as unknown as ReturnType<typeof dcModule.default.getInstance>);
+
+        await expect(
+            (svc as unknown as { ensureMeshNetwork: (s: string) => Promise<void> }).ensureMeshNetwork('172.30.0.0/24'),
+        ).resolves.toBeUndefined();
+    });
+});
+
+describe('MeshService.optInStack rollback', () => {
+    it('rolls back the DB row when the just-inserted stack fails to push its override', async () => {
+        const svc = MeshService.getInstance();
+        vi.spyOn(svc as unknown as { inspectStackServices: (n: number, s: string) => Promise<unknown> }, 'inspectStackServices')
+            .mockResolvedValue([{ service: 'db', ports: [5432] }]);
+        vi.spyOn(svc, 'pushOverrideToNode')
+            .mockRejectedValue(new Error('simulated remote pilot offline'));
+        vi.spyOn(svc as unknown as { triggerRedeploy: (n: number, s: string, a: string) => void }, 'triggerRedeploy')
+            .mockImplementation(() => { /* noop */ });
+
+        const db = DatabaseService.getInstance();
+        const localNodeId = db.getNodes()[0].id;
+
+        await expect(svc.optInStack(localNodeId, 'api', 'tester'))
+            .rejects.toThrow(/simulated remote pilot offline/);
+        expect(db.isMeshStackEnabled(localNodeId, 'api')).toBe(false);
+    });
+});
+
+describe('MeshService.optInStack guard rails (network setup)', () => {
+    it('rejects opt-in when senchoIp is null (mesh data plane unavailable)', async () => {
+        const svc = MeshService.getInstance();
+        (svc as unknown as { senchoIp: string | null }).senchoIp = null;
+        (svc as unknown as { networkSetupError: string | null }).networkSetupError = 'sencho_mesh subnet mismatch';
+
+        const db = DatabaseService.getInstance();
+        const localNodeId = db.getNodes()[0].id;
+
+        await expect(svc.optInStack(localNodeId, 'api', 'tester'))
+            .rejects.toThrow(/sencho_mesh subnet mismatch/);
+        expect(db.isMeshStackEnabled(localNodeId, 'api')).toBe(false);
+    });
+
+    it('rejects opt-in when a service exposes the reserved Sencho API port', async () => {
+        const svc = MeshService.getInstance();
+        vi.spyOn(svc as unknown as { inspectStackServices: (n: number, s: string) => Promise<unknown> }, 'inspectStackServices')
+            .mockResolvedValue([{ service: 'web', ports: [1852] }]);
+        vi.spyOn(svc as unknown as { regenerateOverridesForNode: (n: number) => Promise<void> }, 'regenerateOverridesForNode')
+            .mockResolvedValue(undefined);
+
+        const db = DatabaseService.getInstance();
+        const localNodeId = db.getNodes()[0].id;
+
+        await expect(svc.optInStack(localNodeId, 'api', 'tester'))
+            .rejects.toThrow(/port 1852 is reserved/);
+        expect(db.isMeshStackEnabled(localNodeId, 'api')).toBe(false);
+    });
+});