mirror of
https://github.com/Studio-Saelix/sencho.git
synced 2026-08-08 01:43:55 +00:00
2e82eb44fec48c33c73cd10e030c4b27010b4824
17 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Anso
|
946db9df52 |
fix(mesh): accept node_proxy at the proxy-tunnel WS upgrade (#1050)
The mesh proxy-tunnel WS handler was gated on full-admin api_token scope only, so node_proxy JWTs (the credential the Add Remote Node dialog tells operators to generate via Settings -> Nodes -> Generate Token) were silently rejected with HTTP 403. Operators followed the dialog's instructions, enrolled the node cleanly, saw reachableMode='proxy' with no negative badge, opted a stack into mesh, watched the redeploy complete, and only then discovered no bytes flow. node_proxy already authorizes every /api/* surface on the remote (deploy, exec, host console, filesystem), which is a strict superset of what the mesh proxy-tunnel does. Gating mesh more strictly was theatre, not security, and it created a UX trap with no in-product signal pointing at the scope mismatch. Change the upgrade dispatcher to accept any machine-to-machine credential: node_proxy JWT or full-admin api_token. Session cookies and restricted api_token scopes (read-only, deploy-only) remain rejected through the same paths as before. Adds positive/negative coverage in upgrade-order.test.ts for all four credential shapes so the gate stays pinned against future re-tightening. Updates user-facing copy in the mesh docs and Settings -> API Tokens description so the documented path matches the actual code. If we ever introduce a demoted node_proxy variant (audit-only, read-only fleet member), the right move is differentiated node_proxy scope claims inside the tunnel JWT (tracker F-A3), not requiring a separate token type for mesh. |
||
|
Anso
|
a38a3e0226 |
feat(mesh): route mesh traffic over Distributed API remotes (#1048)
* refactor(mesh): extract shared TCP stream switchboard
Pull the `tcp_open` / `tcp_open_ack` / `tcp_open_reverse` / `tcp_close`
+ `TcpData` handling out of the pilot agent into a reusable module so a
second caller (the upcoming proxy-mode WS handler) can run the same
frame parser, stream allocator, idle timers, and Compose-label
resolver. One parser, two callers, zero drift on a
security-sensitive protocol surface.
The pilot agent keeps its public `openMeshTcpStream` API and delegates
to a per-connection switchboard constructed in `connect()` and torn
down in `cleanupAfterDisconnect`. Existing reverse-stream and resolver
tests are rewritten to exercise the shared module directly.
* feat(mesh): route mesh traffic over Distributed API remotes
Sencho Mesh now works against remotes in Distributed API mode in
addition to Pilot Agent mode. Central opens a short-lived WebSocket
tunnel to the remote's new `/api/mesh/proxy-tunnel` endpoint on demand
and tears it down after 5 minutes of idle (configurable via
`SENCHO_MESH_PROXY_TUNNEL_IDLE_MS`). Mesh dispatch is mode-agnostic at
the routing layer; `PilotTunnelManager.ensureBridge` resolves an
existing tunnel or asks the new `MeshProxyTunnelDialer` to dial.
The Routing tab badges now use a `reachableMode` classifier: `★ Local`
for local, `pilot offline` red badge for a pilot with a down tunnel,
and a new `unreachable` red badge surfacing the specific reason
(missing token, scope not full-admin, TLS failure, remote does not
support proxy mesh). Distributed API remotes with valid credentials
show no negative badge; the tunnel opens on first dial.
Auth: the proxy-tunnel WS upgrade requires an `Authorization: Bearer`
API token with the `full-admin` scope. Lower-scoped tokens are
rejected at upgrade time so a leaked read-only token cannot reach the
mesh data plane.
Bidirectional: the proxy-mode WS handler registers itself as the
local `MeshService` reverse dialer (compare-and-swap), so meshed
containers on a Distributed API remote can dial cross-node aliases
via `tcp_open_reverse` over the same tunnel. Cross-node relays
(`PilotTunnelBridge.acceptReverseRelay`) await `ensureBridge` so
proxy-to-proxy mesh works without standing tunnels.
* docs(mesh): describe Distributed API mesh and unreachable troubleshooting
Refresh the user-facing mesh documentation to reflect that mesh works
over both Pilot Agent and Distributed API remotes. Adds a
Troubleshooting accordion entry for the new `unreachable` Routing tab
badge so operators can match a tooltip reason ("api token rejected
(scope must be full-admin)", "remote does not support proxy mesh",
"TLS handshake failed", "api_url not set", "api token missing") to a
concrete fix.
* refactor(mesh): apply /simplify review findings
Five cleanups surfaced by the post-implementation code review pass.
No behaviour change for fleets running on `main`; only internal
structure improves.
- **Deterministic container IP shared.** Extract the compose-default
network preference (`pickContainerIp`) and the conventional-name
fast path (`lookupContainerIp`) into `backend/src/mesh/containerLookup.ts`.
Both `MeshService.resolveContainerIp` (existing same-node fast
path) and the switchboard's `resolveByComposeLabels` (proxy/pilot
inbound dial) now use the same logic. Earlier the switchboard
helper grabbed the first `Object.values(Networks)` IP, which
could flip across daemon versions on multi-network containers;
fixed.
- **WS URL upgrade extracted.** New `backend/src/utils/wsUrl.ts`
exposes `httpUrlToWs(baseUrl)` that maps `http://` to `ws://` and
`https://` to `wss://`. Used in both `pilot/agent.ts` and the
proxy-tunnel dialer. The previous inline `replace(/^http/, 'ws')`
silently downgraded `https://` to `ws://` (cleartext).
- **`computeReachable` takes the pre-fetched node.** `getStatus`
already iterated `db.getNodes()`; the helper previously re-queried
by id per node (N+1 reads). Pass the row in.
- **Reachable-reason ternary -> const map.** Replace the nested
ternary in `MeshService.computeReachable` with a
`Record<DialFailureCode, string>` lookup keyed on the dialer's
failure code.
- **Activity-type ternary -> const map.** Same flattening inside
`MeshProxyTunnelDialer.logActivity`.
All mesh tests pass (85/85). Full backend suite green (2126/2126).
* fix(mesh): cache proxy dial failures and contain WS handshake errors
`ensureBridge` now consults the recent-failure cache before dialing so a
continuous mesh workload against a misconfigured proxy-mode remote does
not produce one upgrade attempt per cross-node TCP open. `recordFailure`
emits at most one activity-log entry per cache window per (nodeId, code)
so a connect-loop on a single bad remote cannot flush the ring buffer.
Failure messages run through `redactSensitiveText` before reaching the
log so any embedded Bearer / JWT / inline-URL credentials are scrubbed.
`stop()` clears the inflight map and `dial()` checks `this.stopped`
between awaits so a shutdown does not leak a half-opened bridge.
When `awaitOpen` rejects (401, 4403, TLS), the dialer attaches a noop
'error' listener before calling `ws.close()`. Without it the ws library
emits a tail 'error' on a still-CONNECTING socket that propagates as an
unhandled exception. Surfaced by a live-network test against a real
proxy-mode peer.
Adds `mesh-proxy-tunnel-live.test.ts` (skipped unless MESH_AUDIT_URL
and MESH_AUDIT_TOKEN_FILE env vars are set) covering both the happy
path and the auth-rejected path against an actual remote Sencho.
* feat(mesh): instrument proxy tunnel observability
Adds three counters to `PilotMetrics`:
- `proxy_bridges_total` (incremented on `registerProxyBridge` success)
- `proxy_dials_failed` (every failed dial attempt, not deduped)
- `proxy_idle_closes` (idle-sweep teardowns)
All three surface automatically via `GET /api/system/pilot-tunnels`
since the route returns the full `Counters` snapshot.
Gates two pre-existing always-on `console.warn` calls in
`tcpStreamSwitchboard.ts` (mid-stream socket errors and Docker resolve
failures) behind `isDebugEnabled()`. Both fire on per-stream events and
would otherwise violate the diagnostic-log safety rule under load.
Adds `mesh-tcp-stream-switchboard.test.ts` covering the forward
`tcp_open` path: real localhost dial, resolver errors (no_target,
denied), per-tunnel cap saturation, frame-routing fall-through
invariants, `tcp_close` socket teardown.
Drops `MESH_CONNECT_TIMEOUT_MS` from the public surface; only the
switchboard itself uses it.
* fix(mesh): tighten Routing tab unreachable handling
`RoutingNodeCard.tsx`: the **Add stack to mesh** button now disables
when `reachableMode === 'unreachable'`, matching the existing
TogglePill behavior. Without this, an operator on an unreachable node
could open the opt-in sheet, confirm, and watch the redeploy proceed
against a target whose mesh data plane will silently fail to route.
Also drops the leftover `status.nodeId !== -1` guard on the pilot-
offline badge.
`MeshService.ts`: renames `isMeshReachable` to `isMeshConfigured` with
the new predicate that returns true for proxy-mode remotes whose creds
are valid (the tunnel is opened on demand). `getRouteDiagnostic` now
distinguishes "routable" (configured) from "pilotLive" (live tunnel
state, only meaningful for pilot mode); without the split, every idle
proxy-mode route would report `tunnel down`.
`MeshService.setReverseDialer` warns when the unconditional install
path silently overwrites a non-null current dialer. By topology a
Sencho is either pilot or central, so the branch flags a misconfigured
deployment rather than an expected race.
Drops the dead `export { ReverseTcpStreamHandle }` re-export from
`pilot/agent.ts`; its only consumer imports straight from
`mesh/tcpStreamSwitchboard.ts`. Fixes a stale doc comment in
`MeshService.ts` that referenced the wrong source file.
Adds `mesh-proxy-tunnel-handler.test.ts` covering the WS handler
lifecycle: pilot-mode 404 rejection, post-upgrade reverse-dialer
install, concurrent-upgrade 1013 rejection (single-tenant slot), and
error-path teardown.
Updates the troubleshooting accordion in the user docs to mention the
disabled-state behavior.
* fix(mesh): close ESLint and CodeQL findings on the proxy-tunnel diag logs
Remove the unused `reverseDialer` local in `meshProxyTunnel.ts`; the
closure target is `localDialer` above and this assignment was always
dead. Strip line breaks inline on the three `MeshProxyTunnelDialer`
diag log lines so CodeQL `js/log-injection` data flow recognises the
sanitisation that `sanitizeForLog` already performs.
No behaviour change; the diag logs render the same characters they
do today.
|
||
|
Anso
|
b1c5fe8391 |
fix: harden deploy enforcement paths (#1030)
* fix: harden deploy enforcement paths * fix: update Docker toolchain to Go 1.26.3 * fix: repair Dockerfile tr argument split across lines * fix: bump protobufjs to clear npm audit high-severity advisories * fix(test): add execFile to child_process mock in compose-images test * fix: resolve merge conflicts with main * fix: resolve merge conflicts with main * fix: resolve merge conflicts with main |
||
|
Anso
|
b8af40d2b4 |
fix(mesh): enumerate stacks on remote pilot nodes for opt-in sheet (#1025)
The mesh opt-in sheet showed "No stacks deployed on this node yet" for every remote pilot because GET /api/mesh/nodes/:nodeId/stacks called FileSystemService.getInstance(nodeId).getStacks() unconditionally, which always reads central's own filesystem regardless of which node the operator targeted. Fix dispatches through the proxy chain when the targeted node is remote, mirroring the C-3 pattern that already governs every other mesh endpoint needing data from a remote Sencho: - New endpoint GET /api/mesh/local-stacks (Admiral-gated) returns the bare stacks list from THIS Sencho's own filesystem. Mirrors the precedent set by /api/mesh/local-services/:stackName. - New MeshService.listLocalStacks() and MeshService.listStacksOnNode() helpers; the latter dispatches local vs remote and degrades to an empty list on transport failure (no proxy target, non-2xx response, malformed body). - Refactored route delegates the stack-name list to listStacksOnNode; central's mesh_stacks DB is still authoritative for the opt-in flag set per C-3. Tests cover the local path, the remote-OK path with header forwarding, non-2xx, no-target (pilot tunnel down), malformed bodies, and unknown node ids. |
||
|
Anso
|
41a1df279d |
fix(mesh): fix pilot handleAccept dispatch by deriving self nodeId from enrollment token (#1023)
On a pilot node, handleAccept compared target.nodeId (from central's perspective via pilotAliasOverlay) against NodeRegistry.getDefaultNodeId() which always returns 1. This inverted same-node vs cross-node dispatch: - central's alias (nodeId=1) matched localNodeId=1 -> openSameNode on pilot (no container -> silent close, 0 bytes) - pilot's own alias (nodeId=14) != 1 -> openCrossNode -> tunnel loop Fix: add resolveSelfCentralNodeId() that decodes the nodeId claim from SENCHO_ENROLL_TOKEN (present on every pilot; payload decode only, no signature verification). Returns getDefaultNodeId() on central (env unset). Cached in selfCentralNodeId at start(); handleAccept reads the cache. After this fix the reverse direction (pilot-prober -> central echo) routes through openCrossNode (reverse tunnel) and the same-node path (pilot-prober -> pilot echo) routes through openSameNode (local Dockerode dial). Closes BUG-3 banner data-path regression. Tests: 5 new cases covering resolveSelfCentralNodeId token extraction, fallback paths, and handleAccept dispatch routing on a simulated pilot. |
||
|
Anso
|
6ac7da978c |
fix(mesh): push alias port map to pilot so reverse-direction forwarder binds listeners (#1021)
* fix(mesh): apply D-1 pushed override on pilot deploys via file-presence fallback When isMeshStackEnabled returns false, ensureStackOverride now checks for an override file already on disk before returning null. On pilot nodes the mesh_stacks table is intentionally empty (opt-in state lives on central per the C-3 design), so the DB gate blocked the override that central had already pushed via applyLocalOverride. File-presence is safe as the fallback because removeOverrideFromNode sends a DELETE to the pilot when a stack is opted out, so a stale file cannot survive past opt-out. Also aligns removeStackOverride to use path.basename consistently with all other override-path construction sites in the same file. Adds two tests: pilot node with a pushed file returns the path; pilot node with no file returns null. * fix(mesh): push alias port map to pilot so reverse-direction forwarder binds listeners Pilots have no mesh_stacks rows by design (C-3), so refreshAliasCache() produced an empty aliasByPort map and syncForwarderListeners() bound zero ports. Reverse-direction TCP probes (pilot prober -> central alias) failed with connection refused at the OS level because no listener existed. Fix: extend the D-1 override push payload to include the full MeshGlobalAlias records (host + port + routing metadata). The pilot stores these in a new pilotAliasOverlay map keyed by stack name. refreshAliasCache() merges the overlay after the (empty) DB loop, populating aliasByPort correctly. syncForwarderListeners() then binds the alias ports immediately after the push completes. removeLocalOverride() clears the overlay entry and re-syncs on opt-out. The two populations (DB rows on central, pilotAliasOverlay on pilots) are mutually exclusive by the C-3 invariant, so the first-write-wins port collision policy in refreshAliasCache is safe. |
||
|
Anso
|
c12b3ba995 |
fix(mesh): apply D-1 pushed override on pilot deploys via file-presence fallback (#1019)
When isMeshStackEnabled returns false, ensureStackOverride now checks for an override file already on disk before returning null. On pilot nodes the mesh_stacks table is intentionally empty (opt-in state lives on central per the C-3 design), so the DB gate blocked the override that central had already pushed via applyLocalOverride. File-presence is safe as the fallback because removeOverrideFromNode sends a DELETE to the pilot when a stack is opted out, so a stale file cannot survive past opt-out. Also aligns removeStackOverride to use path.basename consistently with all other override-path construction sites in the same file. Adds two tests: pilot node with a pushed file returns the path; pilot node with no file returns null. |
||
|
Anso
|
2d9e4d0480 |
fix(mesh): derive override services from compose file and surface cross-node dispatch (#1017)
* fix(mesh): derive override services from compose file and surface cross-node dispatch
Three fixes for issues found during Phase B end-to-end verification:
1. Override generator no longer queries Dockerode for services. Deploys
removed all containers in the project right before composeArgs ran,
so docker.listContainers({label: project=...}) returned [] and the
generator wrote services: {} which broke meshed user containers.
New helper getDeclaredStackServiceNames reads the local stack's
compose file via FileSystemService + YAML.parse and returns the
top-level services keys. Used from ensureStackOverride and
applyLocalOverride. Defensive fallback: when the read returns []
AND a previous override on disk has a non-empty services map,
keep the file and emit mesh.override.preserved.
2. Boot regen race against the pilot tunnel. regenerateAllOverrides
ran ~1s before the first pilot tunnel-up event, so pushes to
pilot-mode nodes failed with "no proxy target". MeshService.start
now also runs regenerateOverridesForNode(nodeId) inside the
tunnel-up listener, covering both the boot-race window and
mid-runtime tunnel reconnects. Idempotent.
3. openCrossNode emitted no activity log when a cross-node dial got
stuck waiting for tcp_open_ack. The forwarder accepted the TCP
handshake (so nc -v reported open) but logged nothing between
accept and close. Now logs route.dispatch on entry and emits
tunnel.fail after PROBE_TIMEOUT_MS if the agent never acks.
Timer is cleared on every tcpStream and src lifecycle event.
Tests: 8 new in mesh-service.test.ts cover the helper, the BUG-1
override write path, the defensive preservation fallback, the
tunnel-up regen handler, and both cross-node dispatch logging
behaviors. Full vitest suite green (1998 tests pass).
* fix(mesh): satisfy CodeQL path-injection model in new override-read paths
Two new fs.readFile sites added in the previous commit failed the
CodeQL check on the PR even though both paths were guarded by
isValidStackName + isPathWithinBase. The file's existing
applyLocalOverride pattern uses path.basename(stackName) before the
join (commented as "Recognized by CodeQL's path-injection model");
apply the same sanitizer to the new code paths.
- getDeclaredStackServiceNames: wrap stackName in path.basename when
building composePath.
- readExistingOverrideServiceNames: take (dir, stackName) instead of
a pre-built filePath so the sanitizer lives next to the read sink.
- ensureStackOverride: build the override file path with
path.basename(stackName) for symmetry with applyLocalOverride.
Runtime behavior unchanged (path.basename is a no-op for any name
that passed isValidStackName). Tests still green.
|
||
|
Anso
|
7cca68f84a |
fix(mesh): operator-triggered override regen and boot lifecycle hardening (#1016)
Audit-driven follow-up to the F6+F7 fix. Closes the High-severity findings
from the mesh-dial-path audit without expanding into the architectural
follow-ups (JWT TTL, TLS enforcement, per-stack JWT scope) which need
their own design discussions.
Changes:
1. New `POST /api/mesh/regen-overrides` Admiral-gated endpoint that calls
`MeshService.regenerateAllOverrides()` and returns a structured
`MeshRegenSummary { regenerated, failures, skipped, reason? }`. Lets
the operator rerun boot regen after fixing a remote node that was
offline at central startup, instead of opt-out + opt-in for every
meshed stack on that node. Audit-log row carries the outcome
(success / partial / skipped / error).
2. `regenerateAllOverrides` now aggregates per-stack outcomes into a
single summary log row (`mesh override regen complete: N succeeded,
M failed across K node(s)`) with `failedNodeIds` in `details`. Per-
stack warnings still emitted. When skipped because `senchoIp` is
null, emits an explicit warn instead of silently no-oping.
3. `MeshService.start()` now wraps `refreshAliasCache` and
`syncForwarderListeners` in their own try/catch so a throw from
either step is logged and the boot continues to override regen.
The closing log line surfaces data-plane state ("MeshService started
(data plane ok)" or "(data plane unavailable (<reason>))") so an
operator grepping startup output sees a half-init mesh immediately.
4. `sanitizeForLog` wrap on four previously unsanitized `err.message`
writes (cross-node tcpStream error handler + three probe error
paths). Container IPs and local socket details no longer leak into
the activity buffer or the probe response body.
5. Updated three existing F6 regression tests for the new return shape
and aggregated summary message. Added two new tests: version-skew
404-push handling, and concurrent opt-in during regen.
|
||
|
Anso
|
0947a80cda |
fix(mesh): trust central for cross-node dial auth and regenerate overrides at boot (#1014)
Two bugs in the same Phase D follow-up surface, fixed together because they both block declaring B-verify complete on the production fleet. Cross-node mesh dials returned `denied` at the agent. The pilot's `tcp_open` handler in `agent.ts::resolveMeshTarget` consulted the local SQLite `mesh_stacks` table, which is no longer written to under the post-Phase D control plane (state lives only on central). Drop the check. The pilot tunnel JWT (scope `pilot_tunnel`, signed with central's `auth_jwt_secret`) authenticates the caller; the same trust model already applies to filesystem ops, exec, and container control over the same tunnel. Threat-model trade-off: a leaked `pilot_tunnel` JWT or compromised central can now dial any compose-managed service on the pilot. Containers without `com.docker.compose.project` + `com.docker.compose.service` labels remain unreachable via this path. `MeshService.start()` did not regenerate compose override files at boot. After a Sencho restart with missing overrides on disk, meshed user containers had no `extra_hosts` / `networks: [sencho_mesh]` injection until each stack was opted out and back in. Add `regenerateAllOverrides()` that walks every `mesh_stacks` row and re-pushes via `pushOverrideToNode`. Best-effort: per-stack failures log to the mesh activity buffer; `MeshService.start()` is fire-and-forget at startup so a slow remote node does not delay boot. Tests: - `pilot-agent-mesh-resolve.test.ts` (new): mocks dockerode and proves `resolveMeshTarget` no longer returns `denied` with an empty `mesh_stacks` table. - `mesh-service.test.ts`: three new cases for `regenerateAllOverrides` - fan-out across the fleet, skip when `senchoIp` is null, log per-stack warning on push failure without throwing. |
||
|
Anso
|
23bbee4f45 |
feat(mesh): replace host-mode with shared sencho_mesh Docker network (#1009)
* feat(mesh): replace host-mode with shared sencho_mesh Docker network Phase D of the mesh redesign: drop the operator's `network_mode: host` requirement and the `host-gateway` extra_hosts pattern that did not work on cloud iptables-restrictive distros (OCI, etc.) or Docker Desktop. Each Sencho creates a shared `sencho_mesh` Docker bridge network on boot (default subnet 172.30.0.0/24, override via SENCHO_MESH_SUBNET), pins itself at `<network>+2`, and attaches every meshed user service to the same bridge. Compose overrides now emit IP-based `extra_hosts` plus a top-level `networks` block declaring `sencho_mesh` external. Override delivery: central renders for local stacks; for remote stacks it sends the fleet alias list to the remote's new `PUT /api/mesh/local- override/:stackName` endpoint, which renders against the remote's OWN local senchoIp and writes under its OWN DATA_DIR. Each node may use a different subnet without coordination beyond the env var. Opt-in / opt-out now trigger an automatic redeploy of the affected stack via the existing deploy code path (local: ComposeService; remote: HTTP POST through proxyFetch). The frontend opt-in sheet shows a confirmation modal (ConfirmModal) before the mutation. Failed redeploys emit both a mesh activity event and a durable audit-log row. Hardening: - Reserve port 1852 at opt-in (prevents user containers from racing the Sencho API listener). - ensureMeshNetwork refuses to continue if `sencho_mesh` exists with a mismatched subnet rather than silently routing to the wrong IP. - Idempotent network connect/disconnect helpers in DockerController. - optInStack rolls back the DB row if the just-inserted stack's override push fails (no half-states surviving across calls). - regenerateOverridesForNode runs in parallel and skips the just- pushed stack on opt-in. Operator template: drop `network_mode: host`, restore `ports: ["1852:1852"]`. Mesh now works identically on Linux LAN, OCI, and Docker Desktop without firewall changes. Docs: rewrite docs/features/sencho-mesh.mdx around the shared bridge network, document SENCHO_MESH_SUBNET, surface the host-network-service opt-in restriction, and cross-link with the Pilot Agent docs. BREAKING CHANGE: the operator's `docker-compose.yml` no longer uses `network_mode: host`. After upgrading, redeploy any meshed stacks once so they pick up the new IP-based override and join `sencho_mesh`. * fix(mesh): wrap stackName with path.basename in local-override fs ops CodeQL flagged js/path-injection on the new applyLocalOverride and removeLocalOverride methods because they are publicly reachable and its data-flow model does not recognize isValidStackName / isPathWithinBase as sanitizers. The validation IS sufficient (the allowlist regex blocks path separators, the path-prefix check blocks escape), but path.basename is a model CodeQL recognizes and is purely defensive: for any input that already passes isValidStackName, basename is the identity. |
||
|
Anso
|
10a469ecb4 |
fix(mesh): bind every alias port on every node, not just local-owned (#1004)
syncForwarderListeners filtered the bind set to ports owned by the local node. That broke cross-node mesh routing end to end: meshed containers' extra_hosts: <alias>:host-gateway entries resolve to the SOURCE node's gateway, so the source node is where the inbound TCP connection lands. With the filter, the source node never bound the remote alias's port and the connection went nowhere. The architecture model documented in docs/internal/architecture/ mesh.md (data flow step 3) is explicit: monolith's Sencho process accepts the connection on its MeshForwarder listener bound to port 5432 — where monolith is the source and opsix is the target. Every meshed node binds every alias port. handleAccept then resolves the alias and dispatches to openSameNode (when target.nodeId equals the local node) or openCrossNode (otherwise). Both branches were already correct; only the bind filter was wrong. Fleet-wide port collisions remain blocked at opt-in time (optInStack checks aliasByPort), so binding every alias port is unambiguous. New vitest case asserts the want-set includes both local-owned and remote-owned ports. Discovered while running PR 0 verification on v0.75.0 against Local + sencho-pilot-test: connections from a Local prober to the pilot's alias port hit Local's host with no listener. |
||
|
Anso
|
567e524450 |
feat(mesh): bidirectional routing via tcp_open_reverse and central relay (#1003)
* feat(mesh): bidirectional routing via tcp_open_reverse and central relay
Phase B of the mesh redesign. Adds the reverse-direction protocol so a
pilot's MeshForwarder can route cross-node traffic to central or to
another pilot. Central relays pilot-to-pilot streams transparently;
pilots keep their existing single outbound WS to central.
Protocol additions (backend/src/pilot/protocol.ts):
- TcpOpenReverseFrame { s, targetNodeId, stack, service, port } sent
agent to primary. Reuses existing tcp_open_ack, tcp_close, and
TcpData binary frames for the response and byte plane.
- AGENT_REVERSE_ID_BASE = 0x40000001 splits the 32-bit id space so
agent-allocated reverse stream ids never collide with primary-
allocated forward ids on the same tunnel.
- StreamIdAllocator now wraps to its configured start (not always 1)
so an allocator parameterized with the agent base stays in the
agent half across the wrap.
Agent (backend/src/pilot/agent.ts):
- New ReverseTcpStreamHandle exported class, EventEmitter facade
matching PilotTunnelBridge.TcpStream's surface (write, end,
destroy plus open, data, error, close events).
- Public openMeshTcpStream(target) allocates a reverse id, sends
tcp_open_reverse, returns the handle.
- onTcpOpenAckReverse handles inbound ack, dispatches open or
error+close to the matching handle.
- TcpData and tcp_close binary/JSON paths route ids in the reverse
range to reverseTcpStreams; existing primary-allocated paths
unchanged.
- streamCount, cleanupAfterDisconnect, onStreamIdle include reverse
streams. The allocator is reset to a fresh base on disconnect so
long-lived agents that reconnect many times do not drift up the id
space.
- startPilotAgent registers the agent as MeshService's reverse
dialer via lazy import.
Bridge (backend/src/services/PilotTunnelBridge.ts):
- Two new StreamState kinds: reverse_local (target = central, dial
Dockerode container IP) and reverse_relay (target = another pilot,
open a forward TcpStream on the target pilot's bridge).
- handleTcpOpenReverse validates the agent-id range and stream cap,
then dispatches to acceptReverseLocal or acceptReverseRelay via
lazy imports of MeshService, NodeRegistry, and PilotTunnelManager
(avoids module cycles).
- acceptReverseLocal calls MeshService.resolveContainerIp (now
public), opens net.createConnection, splices bytes through the
tunnel. Pre-connect error handler is removed inside connect to
avoid double-firing with the mid-stream error path.
- acceptReverseRelay opens a forward TcpStream on the target
bridge, splices bytes between the two tunnels.
- Existing tcp_close and TcpData handlers extended for the new
state kinds. teardownStream extended.
MeshService (backend/src/services/MeshService.ts):
- resolveContainerIp made public so the bridge's local-target path
can dial the same shape.
- New reverseDialer field plus setReverseDialer setter. Pilot mode
registers the agent at boot; central mode leaves it null.
- New private dialMeshTcpStream dispatcher: pilot side uses the
reverse dialer, central side uses PilotTunnelManager.getBridge.
- openCrossNode refactored to call the dispatcher and use the
active-stream record's id for activity logging.
- New exported MeshTcpStreamLike interface that both
PilotTunnelBridge.TcpStream and ReverseTcpStreamHandle satisfy
structurally; ReverseMeshDialer is the contract for the setter.
Tests (3 files, 16 new cases, 172 total pass):
- pilot-protocol-tcp.test.ts: tcp_open_reverse round-trip; agent
base invariant.
- pilot-agent-reverse-stream.test.ts: openMeshTcpStream allocation,
ws-not-open, frame shape, write encoding, end emits tcp_close,
ack-success, ack-failure, low-id ack ignored, inbound TcpData
routing.
- pilot-bridge-reverse.test.ts: id-range validation, no-target
failure, successful local dial against a real upstream server.
Together with PR #1000 (Phase A) and PR #1001 (deps), Phase B
completes the central-pilot-pilot mesh routing matrix. Phase C
(mesh over proxy-mode remotes) is the planned follow-up.
* test(pilot): drop unused encodeJsonFrame import
Lint failed on pilot-agent-reverse-stream.test.ts after the test
changed from constructing tcp_open_reverse frames inline to driving
the agent's frame-dispatch path with synthetic objects. The import
is no longer referenced; ESLint's no-unused-vars rule rejected it.
|
||
|
Anso
|
f599110386 |
feat(mesh): collapse sidecar into Sencho process via in-process forwarder (#1000)
The separate saelix/sencho-mesh sidecar container is gone. The
forwarder logic that previously lived in mesh-sidecar/src/forwarder.ts
moves into the Sencho process as backend/src/services/MeshForwarder.ts,
a thin per-port net.Server lifecycle wrapper. MeshService implements
the host interface and owns resolve plus splice; MeshForwarder owns
listener boilerplate. One container per node, no separate image to
publish, no control WebSocket.
Operator-facing change: the Sencho container now runs in
network_mode: host so the forwarder can bind alias ports on the host
network where meshed containers' extra_hosts host-gateway entries
point. Without host network mode the listeners would land in the
container's namespace and inbound traffic from peers would never
reach them. The 1852:1852 port publish becomes a no-op under host
mode and is commented out in the operator template.
Same-node forward path now dials the target container's bridge IP
via Dockerode (preferring the compose default network for
deterministic selection across daemon versions) instead of 127.0.0.1.
The legacy 127.0.0.1 path only worked when the target service
published its port to the host; the IP path works regardless.
Cross-node mesh routing in this phase is central -> pilot direction
only via PilotTunnelManager.openTcpStream. Pilot -> central and
pilot <-> pilot via central relay land in Phase B with the
tcp_open_reverse frame.
Deletions:
- mesh-sidecar/ package entirely (Dockerfile, package, sources, tests)
- backend/src/websocket/meshControl.ts
- MeshService sidecar lifecycle: spawnSidecar, stopSidecar,
isSidecarRunning, mintSidecarToken, verifySidecarToken,
attachSidecarSocket, handleSidecarResolve, sendSidecar
- POST /api/mesh/nodes/:id/sidecar/restart route
- /api/mesh/control WS dispatch in upgradeHandler
Type cleanup: 'sidecar' literal removed from MeshActivitySource and
MeshProbeResult.where (also the frontend mirror). MeshNodeStatus
sidecarRunning becomes localForwarderListening (boolean | null) so
non-local nodes get a null instead of an unconditional false; the
honest semantic is "this view only knows the local forwarder state;
remote forwarder status lands in Phase B." MeshNodeDiagnostic
sidecar object becomes forwarder { listening, listenerCount }.
Frontend MeshDiagnosticsSheet drops the restart-sidecar action and
sidecar liveness card; surfaces forwarder state plus a "runs
in-process; no separate container" caption.
Resolves audit findings C-1 (data plane non-functional), C-2 (sidecar
control WS not loopback-enforced), C-4 (sidecar lifecycle Dockerode-
on-remote, PR #999 closed), and C-5 (saelix/sencho-mesh:latest
unreachable). C-3 (PR #992) is unchanged. M-12 (PR #994) is
unchanged.
|
||
|
Anso
|
34581e55e7 |
fix(mesh): treat local-node aliases as always reachable in diagnostics (#994)
MeshService.getRouteDiagnostic called PilotTunnelManager.hasActiveTunnel unconditionally for the alias's target node. Local nodes never establish pilot tunnels because they do not need them (mesh same-node traffic uses the localhost fast path), so the call always returned false and the diagnostic flipped to state: 'tunnel down' even on a working local route. The route detail sheet and node card consequently rendered every local mesh alias as a destructive 'tunnel down' pill. Added a private isMeshReachable helper that returns true for any local-type node and otherwise delegates to hasActiveTunnel. Used it in both getRouteDiagnostic and getStatus, replacing the inline ternary in getStatus that already had the right shape but lived as duplicated logic. The helper falls back to false when the node row is missing (orphaned alias from a partial cascade), matching the pre-existing conservative behavior. Verified live: GET /api/mesh/aliases/echo.audit-mesh-prod.Local.sencho/diagnostic previously returned state='tunnel down' on a healthy local route. |
||
|
Anso
|
e176ae9f17 |
fix(mesh): inspect remote stacks via HTTP proxy not Dockerode (#992)
MeshService.inspectStackServices used DockerController.getInstance(nodeId)
to enumerate compose-labeled containers, but NodeRegistry.getDocker
explicitly throws for any node with type='remote' by design. The throw
was silently caught and an empty array returned, so opt-in for any
remote node failed with a misleading "no running services" error and
refreshAliasCache only ever populated LOCAL aliases. Cross-node mesh
routing was therefore impossible end-to-end regardless of pilot-tunnel
state.
Split the inspector. inspectLocalStackServices keeps the Dockerode
listContainers path and always queries the local Docker daemon; it is
public so the new route can call it. inspectStackServices is now a
dispatcher: local nodes fall through to the Dockerode path, remote
nodes (proxy mode and pilot-agent) HTTP-fetch
/api/mesh/local-services/:stackName against the URL resolved by
NodeRegistry.getProxyTarget, with the persisted node_proxy Bearer
token and license tier headers attached. The remote's MeshService
enumerates its own LOCAL Docker daemon and returns the
{service, ports[]} envelope.
refreshAliasCache now inspects every opted-in stack in parallel via
Promise.allSettled so a slow or unreachable remote does not stall the
60-second loop. The new /api/mesh/local-services/:stackName route is
gated by requireAdmiral plus isValidStackName and always queries the
caller's own Sencho instance.
Together with PR #989 (proxy-side bridge dispatch) and PR #990
(agent-side loopback auth), this makes the central, bridge, agent,
local-Sencho mesh control plane functional end-to-end for both
proxy-mode and pilot-agent remotes.
|
||
|
Anso
|
7663f4cd8b |
feat(fleet): sencho mesh in traffic and routing tab (#858)
* feat(fleet): sencho mesh in traffic and routing tab Lights up Sencho Mesh: cross-node container forwarding rendered as if the container next to you were on localhost. Builds on the dormant TCP frame plumbing from the prior PR (pilot tunnel TCP frames + sencho-mesh sidecar package) and exposes the Admiral-only orchestrator surface. Backend - New mesh_stacks table (per-node opt-ins) + nodes.mesh_enabled column via DatabaseService.migrateMeshTables. - MeshService singleton: sidecar lifecycle via Dockerode, opt-in/out with cascading override regeneration, request-based resolver from sidecar control WS, cross-node TCP forwarding via PilotTunnelManager (same-node fast path included), in-memory 1000-event activity ring buffer with durable mirror to audit_log for state-change events, per-node and per-route diagnostics, and the Test upstream probe. - MeshComposeOverride: pure YAML generator that injects extra_hosts using host-gateway. The user's docker-compose.yml is never mutated; overrides live under DATA_DIR/mesh/overrides. - ComposeService deploy/update splice the override file when the stack is opted in; non-mesh stacks behave identically to today. - Pilot agent resolveMeshTarget consults the local mesh_stacks table (defense in depth) and resolves Compose containers via Dockerode. - /api/mesh router with 13 Admiral-gated endpoints covering status, enable/disable, stack opt-in/out, alias listing, per-route diagnostic, Test upstream probe, per-node diagnostic, sidecar restart, activity log paginated and SSE. - meshControl WS slot at /api/mesh/control validates the mesh_sidecar JWT minted by MeshService; dispatched as upgrade slot 2 (canonical order preserved). Frontend - New Traffic Routing tab in FleetView, gated by isAdmiral and wrapped in AdmiralGate. Tab uses the cyan brand glyph and italic-serif state typography from the audit. - RoutingTab masthead with mesh activity drawer, per-node card grid with TogglePill, alias rows with five-state pill taxonomy (healthy / degraded / unreachable / tunnel-down / not-authorized), inline Test buttons. - Four sheets: opt-in picker with port-collision inline error, per-route detail with diagnostic + filtered activity, per-node diagnostics with active streams + resolver cache + restart action, fleet-wide activity log with filters. - meshRouteState helper centralizes pill-state mapping; pure-function tests cover all five states. Docs - User docs at /docs/features/sencho-mesh.mdx covering opt-in, troubleshooting, security model (4 guarantees + 4 explicit non-guarantees), and V1 limitations. - Internal architecture and runbook pages. - websocket-dispatch internal doc updated with the new slot. * fix(mesh): validate stack name before path use; fix test DB lifecycle Two surgical fixes against the prior PR. Path-injection (CodeQL js/path-injection): MeshService.optInStack, optOutStack, ensureStackOverride, and removeStackOverride now validate stackName via isValidStackName from utils/validation, reject malicious names at the API boundary, and additionally check isPathWithinBase on the resolved override file path for defense in depth. The dataflow from req.params.stackName to fs.writeFile no longer reaches an unsanitized path expression. Test DB lifecycle: mesh-service.test.ts used per-test setupTestDb / cleanupTestDb, which deletes the temp dir while DatabaseService still holds an open SQLite handle. On Linux CI this raises SQLITE_READONLY_DBMOVED on the next prepare() because the inode has been unlinked. Switched to file-scoped beforeAll/afterAll matching agents-routes.test.ts, with a per-test beforeEach that truncates mesh_stacks plus non-default nodes and resets the MeshService singleton in-memory state. Adds a new test case asserting the path-traversal rejection. * fix(compose): use discovered compose filename instead of hardcoded docker-compose.yml composeArgs() hardcoded `-f docker-compose.yml` for every deploy. Sencho writes its canonical compose file as `compose.yaml`, so any stack created via the UI failed to deploy with `open ...docker-compose.yml: no such file or directory`. When no mesh override applies, drop the explicit `-f` so docker compose's built-in discovery resolves the actual filename. When an override exists, look up the real base filename via FileSystemService.getComposeFilename() and pass both files explicitly. Also hoist the MeshService import to module top now that the dependency is known to be acyclic, and revert the matching unit-test assertion. |