gsadmin/sencho
1
0
Fork 0
mirror of https://github.com/Studio-Saelix/sencho.git synced 2026-08-07 01:14:14 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
9aaa8573c0ecceade58a613c209effeab52aeec0
sencho/docs/features/sencho-mesh.mdx
T
Anso cf618dd866 feat(mesh): symmetric WS dial for proxy-mode mesh peers (#1066) 
* chore(mesh): foundation for symmetric callback dial

Adds the data-plane scaffolding that the symmetric callback dial fix
builds on:
- mesh_centrals table for peer-side bootstrap material
- MeshCentralRegistry service (upsert/getActive/clear/markUsed/markRejected)
- PilotTunnelManager kind discriminator and replaceOrRegisterProxyBridge
- mesh_proxy_callback_bootstrap capability registration
- MeshProxyTunnelDialer reason-tagged proxy-bridge-down events from a
  single tearDownBridge emission point
- Reactive redial scheduler that skips idle and auth_failed reasons

* feat(mesh): add reverse-direction activity log entries (closes R1-B)

acceptReverseLocal now emits route.resolve.ok with direction=reverse on
connect ack and route.resolve.fail with direction=reverse plus
reason=container_not_found / connect_error pre-connect. Post-connect
close/error stays silent. Reuses existing event types via the new
details.direction discriminator so frontend filters are unaffected.

* feat(mesh): add peer-to-central callback dial path (closes R1-A2)

Closes the architectural gap where proxy-mode mesh peers could not
re-establish their tunnel to central after any non-idle bridge teardown
(idle close, network blip, central restart, peer reboot). Central remains
the hub for the data plane; the change is purely about WS initiation.

Symmetric WS initiation, asymmetric protocol roles. Central retains
PilotTunnelBridge ownership; peer retains TcpStreamSwitchboard +
reverseDialer ownership. Central bootstraps callback credentials over
the first authenticated central-initiated mesh tunnel via a one-shot
mesh_handshake JSON frame; peer persists the material in a new
mesh_centrals SQLite table and dials central's new
/api/mesh/proxy-tunnel-from-peer endpoint when local cross-node traffic
needs a bridge and none is live.

Mesh_tunnel JWT (HS256, signed with auth_jwt_secret) carries scope, audience,
issuer (central instance id), peer api_token fingerprint, kid. Validation
on inbound peer dial: algorithm pin, signature, scope, audience, instance,
time bounds, node existence and mode, fingerprint match. Failures return
HTTP 401 with a machine-readable reason; peer routes the response per a
clear-vs-keep cache matrix.

Triggers proactive bootstrap on mesh-enable and api_token rotation; central
startup fans out to mesh-enabled proxy-mode nodes with mesh_stacks rows
(throttled, fire-and-forget). Reactive redial on non-idle bridge loss.

Capability-gated handshake send (mesh_proxy_callback_bootstrap) makes the
upgrade path safe against older peers in mixed-version fleets.

Adds peer-side /api/system/pilot-tunnels centralCallback diag block,
bounded counter metrics for bootstrap and dial events. SENCHO_PRIMARY_URL
preflight warning when unset on a central with mesh-enabled proxy nodes.

Tested with unit suites for the validation chain, registry, manager, and
both dialers; integration tests for bootstrap E2E (asserts protocol-role
invariant), api_token rotation, instance id change, version skew, and
pilot-mode regression.

* fix(mesh): green CI on the symmetric callback branch

Two independent CI failures, both surgical:

1. Backend tests (11 fails): four mesh test files called setupTestDb in
   beforeEach. setupTestDb does not reset the DatabaseService singleton,
   so the per-test afterEach rm of the previous tmpdir left the singleton
   connection pointing at a deleted file. The next beforeEach's line-55
   write threw SQLITE_READONLY_DBMOVED on Linux. Windows file-lock
   semantics hid this locally. Hoist setupTestDb / cleanupTestDb to
   file-scope beforeAll / afterAll; per-test state resets stay in
   beforeEach. Matches the convention in the eight mesh test files that
   already pass.

2. CodeQL (4 high alerts): js/insufficient-password-hash flagged
   sha256(api_token) at four sites. The api_token is a 256-bit opaque
   bearer (sen_sk_-prefixed), not a human password; sha256 is the
   correct fingerprint primitive for binding the mesh_tunnel JWT to a
   specific token. Add the two production files plus the two test
   files that mint the fingerprint to the existing path-scoped
   query-filter for that rule.

* fix(mesh): drop unused afterEach import and revert dead codeql config

ESLint flagged afterEach as unused in mesh-central-registry.test.ts:1
after the previous commit hoisted setup/teardown to file-scope
beforeAll/afterAll. Remove from the vitest import line.

Revert the codeql-config.yml additions from the previous commit. The
paths: sub-key under query-filters > exclude is not a documented CodeQL
feature and silently no-ops. The four js/insufficient-password-hash
alerts on api_token fingerprinting are tracked as dismissed false
positives in the GitHub Security tab rather than via dead config.
2026-05-16 14:58:19 -04:00

194 lines
14 KiB
Plaintext
Raw Blame History

---
title: Sencho Mesh
description: Connect containers across nodes by hostname over an authenticated WebSocket tunnel, no VPN, no firewall changes, no extra ports.
---
<Note>
Sencho Mesh requires a Sencho **Admiral** license. Skipper and Community Edition do not include this feature.
</Note>
Sencho Mesh makes a multi-node fleet feel like one machine. Opt a stack into the mesh and its services become reachable from any other meshed stack on the fleet by a stable hostname. Traffic rides an authenticated WebSocket tunnel between Sencho instances, so there are no new ports to open and no separate VPN to manage.
Mesh works with any remote mode:
- **Pilot Agent** nodes carry mesh traffic over the long-lived agent tunnel.
- **Distributed API** nodes carry mesh traffic over a short-lived tunnel that central opens on demand using the node's API token. Central tears the tunnel down after five minutes of idle, so a node that sees no mesh traffic costs nothing extra to keep configured.
<Card title="Pilot Agent" icon="link" href="/features/pilot-agent">
Pilot Agent is the easiest remote mode for nodes that cannot expose an inbound port. See Pilot Agent for how to enroll a node into your fleet.
</Card>
## How it works
Each Sencho instance creates an internal Docker bridge network called `sencho_mesh` (default subnet `172.30.0.0/24`) on first boot and pins itself to a static IP on that network. When you opt a stack into the mesh, Sencho:
1. Generates a Compose override file that injects `extra_hosts` for every cross-node alias the fleet currently exposes, pointing each one at the local Sencho's static IP.
2. Attaches every service in the stack to the `sencho_mesh` network so the alias IP is reachable from inside the user's containers.
3. Redeploys the stack with the override applied so its containers pick up the new entries and the network attachment.
Aliases follow a predictable scheme:
```
<service>.<stack>.<node>.sencho
```
A Postgres `db` service in a stack named `api` on a node named `opsix` is reachable as `db.api.opsix.sencho` from any other meshed stack.
## Enable the mesh
1. Open **Fleet → Routing**.
2. Toggle **mesh** on for each node you want to participate.
3. Click **Add stack to mesh** on any node and tick the stacks whose services should be reachable cross-node.
Each opt-in or opt-out triggers an automatic redeploy of the affected stack so its `/etc/hosts` and network attachments refresh. The opt-in sheet shows a confirmation prompt before it starts the redeploy.
## What's exposed and what isn't
Four guarantees:
1. **Only opted-in services are reachable.** A stack reaches another stack's services through the mesh only if both stacks have explicitly opted in. The Pilot agent on the target node refuses any request for a non-opted service.
2. **Aliases are not internet-reachable.** Sencho listens on an internal Docker network; nothing about the mesh exposes new ports beyond the host's existing firewall posture.
3. **Traffic is encrypted in transit.** Cross-node bytes ride an authenticated WSS tunnel between Sencho instances. The Node Token generated during fleet enrollment is the credential that authorizes the mesh tunnel; restricted API token scopes (read-only, deploy-only) cannot carry mesh traffic.
4. **Tier-gated and audit-logged.** Only Admiral users can configure the mesh. Enable, disable, opt-in, and opt-out events write durable rows to the audit log with the actor's identity.
What the mesh does **not** do:
- No application-layer authentication. Your Postgres still needs a password.
- No per-port firewall. Any opted-in stack can reach any other opted-in service.
- No rate limiting or quotas.
- No persistent traffic metrics. Live diagnostics only.
## Customizing the mesh subnet
Each Sencho creates `sencho_mesh` as a `/24` bridge network at `172.30.0.0/24` by default. If that range collides with an existing network on a host, override it with the `SENCHO_MESH_SUBNET` environment variable on that node:
```yaml
services:
sencho:
environment:
- SENCHO_MESH_SUBNET=10.42.0.0/24
```
Sencho's static IP on the network is `<network address> + 2` (so `10.42.0.2` for the example above). Operators can configure each node independently; the override generator pushes alias hostnames that resolve to whichever IP the deploying node uses locally.
## Bidirectional secure callback
Proxy-mode mesh nodes can re-establish their secure mesh callback to central
when traffic starts from that node. Central is the hub; the callback is signed
with a `mesh_tunnel`-scoped credential bound to the node's API token.
<Note>
**Prerequisite:** set `SENCHO_PRIMARY_URL` on central to its canonical public
origin. This value is used as the audience for the mesh callback credential
and as the callback URL on the node side. Without it, central infers an origin
from inbound request headers; the inference works in standard installs but is
not the intended deploy shape for fleets where reverse-proxy headers may differ.
</Note>
## Test upstream
Every alias row has a one-click **Test** button that runs a real probe across the same code path traffic uses. Result is shown inline:
- **Green tick** with round-trip time when the path is healthy.
- **Red badge** with the failing stage (`pilot_tunnel`, `agent_resolve`, `agent_dial`, or `target_port`) when something is wrong.
Use the Test button before assuming an issue is your application's fault. It tells you whether the mesh path itself is the problem.
## Diagnostics
Every node card has a **Diagnostics** button that opens a live view of:
- Forwarder liveness and pilot tunnel state on this node.
- Active TCP streams with byte counters and open age.
- The resolver cache showing which aliases are registered.
This is the first place to look when a connection isn't behaving as expected.
## Mesh activity
The masthead has a **Mesh activity** button that opens the fleet-wide event log. Every route resolution, tunnel state change, opt-in, opt-out, and probe is recorded there. Filter by alias, source, type, or message. Useful for understanding what just happened when something flips state.
## Topology view
The Routing tab has a **Table** / **Graph** toggle in its header. Graph mode draws the fleet as a node-and-edge diagram so it is clear at a glance which nodes are meshed, which tunnels are live, and where aliases are published.
A second toggle picks what the edges encode:
- **Tunnels** shows one edge per remote node, coloured by tunnel state. A solid brand edge labelled `pilot · ok` or `proxy` means traffic is ready to flow. A dashed muted edge labelled `pilot · idle` means the Pilot agent is offline. A dashed red edge labelled `unreachable` means the credentials or remote build cannot carry mesh traffic; the node card shows the specific reason.
- **Aliases** keeps the same node layout and labels each edge with the number of aliases the remote node publishes. A remote that publishes nothing reads `no aliases`.
Click any node card to open the opt-in sheet for that node. On each opted-in stack row the sheet shows a **Topology** button that opens a focused diagram for that one stack: the stack at the centre, every alias it publishes branching out, and a column of meshed consumer nodes with their tunnel state. Use it to confirm what a stack exposes and which peers can reach it.
In the stack diagram, *consumer nodes* are meshed peers that could reach this stack's aliases via DNS. Whether a container on a consumer actually dials an alias depends on the consumer's own opt-in stacks.
The graph reads the same `/mesh/status` and `/mesh/aliases` data the Table view does, so any opt-in or opt-out refreshes both views. The Routing tab also refreshes both feeds every 30 seconds while the browser tab is in the foreground, so tunnel state changes and alias additions appear without a manual reload. Polling pauses automatically when the tab is hidden.
The graph is designed for fleet sizes typical of self-hosted Compose setups (up to roughly 50 nodes). Larger fleets render but become visually dense; the Table view is the more readable surface for inventory at scale.
## V1 limitations
A few things are deliberately out of scope for the first release:
- **One alias per TCP port across the fleet.** If two stacks expose the same port (for example two Postgres instances on 5432), only the first can be added to the mesh. The opt-in sheet shows a clear inline error if the second tries.
- **Sencho's API port is reserved.** A meshed service exposing port 1852 is rejected at opt-in to prevent collision with the Sencho UI / API listener.
- **Remote-to-remote routing rides through central.** Mesh works for traffic between central and any remote in either direction, and between two remotes via central relay. Direct peer-to-peer tunnels between remotes are not supported.
- **Stream pool shared with the Pilot tunnel.** Each Pilot tunnel multiplexes up to 1024 concurrent streams covering HTTP, WebSocket, and mesh TCP traffic. A heavy mesh workload counts against the same ceiling as ordinary fleet API traffic. See the [Pilot Agent](/features/pilot-agent) resource-limit notes for the full picture.
- **No TLS termination, no blue/green cutover.** Layer 7 features land in a follow-up.
## Troubleshooting
<AccordionGroup>
<Accordion title="Adding a stack rejects with 'mesh data plane unavailable'">
The Sencho instance failed to set up its `sencho_mesh` Docker network at boot. Check the Sencho container's logs for `[Mesh]` messages. Common causes: Sencho is running outside Docker (dev mode, mesh routing skipped); the Docker socket is not mounted; `sencho_mesh` already exists with a different subnet from a previous install. The mesh status surface on the Routing tab shows the specific reason.
</Accordion>
<Accordion title="`sencho_mesh` exists with a different subnet">
Sencho refuses to start the mesh data plane if a network named `sencho_mesh` exists with a subnet different from the one this Sencho is configured for. Either remove the existing network (`docker network rm sencho_mesh` on the affected host, after detaching any containers) or set `SENCHO_MESH_SUBNET` on this node to match the existing subnet. Restart the Sencho container so the setup runs again.
</Accordion>
<Accordion title="Opt-in rejects with 'host-network service'">
Stacks whose services declare `network_mode: host` cannot join `sencho_mesh` and therefore cannot participate in the mesh. Switch the affected service to bridge networking and redeploy, or accept that the stack stays out of the mesh.
</Accordion>
<Accordion title="A route shows `tunnel down`">
The Pilot tunnel to the target node is gone. Check **Fleet → Overview** for the node's status. Mesh recovers automatically when the tunnel reconnects.
</Accordion>
<Accordion title="A route shows `unreachable`">
The tunnel is up but the destination port did not answer. Check that the target stack is running and that its service is listening on the declared port. Click **Test** to see the exact failing stage.
</Accordion>
<Accordion title="A route shows `not authorized`">
The destination stack is not opted into the mesh on its home node. Open Routing on that node and add the stack.
</Accordion>
<Accordion title="Adding a stack hangs on redeploy">
Mesh redeploys the stack on opt-in to refresh hostnames and network attachments. A stuck redeploy usually means the stack itself failed to come back up. Check the stack's deploy logs.
</Accordion>
<Accordion title="Subnet conflict with another network on the host">
`172.30.0.0/24` is the default range for the `sencho_mesh` network and is rare in most setups. If it collides with an existing VPN, VLAN, or Docker network, set `SENCHO_MESH_SUBNET` on the Sencho service to a free `/24` and restart the container. Each node can be configured independently.
</Accordion>
<Accordion title="A Distributed API node shows `unreachable` on the Routing tab">
Central could not open a mesh tunnel to this node. The badge tooltip shows the specific reason. While a node is in this state the **mesh toggle and Add stack to mesh action on the node card are disabled** so a redeploy is not triggered against an unreachable target. Common causes:
- `api token rejected by remote` — the credential central uses to dial this node is not accepted. Open the remote Sencho, generate a fresh Node Token via **Settings → Nodes → Generate Token**, and paste it back into the node's credentials in central's **Settings → Nodes**.
- `remote does not support proxy mesh` — the remote Sencho is on a version that predates proxy-mode mesh. Update the remote and the badge clears on the next refresh.
- `TLS handshake failed` — the remote serves a certificate Node's default trust store does not accept. Use a certificate issued by a trusted authority on the remote.
- `api_url not set` or `api token missing` — the node was added without credentials. Edit the node in **Settings → Nodes** and supply the URL and token.
</Accordion>
<Accordion title="Stack topology sheet says 'No published mesh services'">
The stack is opted into the mesh but exposes no service ports that became aliases. The stack joins `sencho_mesh` (other meshed containers can talk to it directly by container name) but no fleet-wide hostname is published. To publish an alias, declare a port on a service in the stack's compose file and redeploy.
</Accordion>
<Accordion title="Graph reflects a stale node state">
The Routing tab polls `/mesh/status` and `/mesh/aliases` every 30 seconds while the browser tab is focused. To force an immediate refresh, leave and return to the Routing tab, or toggle any stack's mesh state to trigger an action-driven refresh. Polling pauses when the tab is hidden, so a long-dormant tab catches up on the first poll after it regains focus.
</Accordion>
<Accordion title="Graph mode is hard to read with a large fleet">
The diagram suits typical fleet sizes of up to roughly 50 nodes. Larger fleets render but the layout becomes dense. Use the Table view for inventory at scale and reach for Graph mode for spot checks of tunnel state and alias publication.
</Accordion>
</AccordionGroup>
Reference in New Issue View Git Blame Copy Permalink