mirror of
https://github.com/Studio-Saelix/sencho.git
synced 2026-08-21 07:36:40 +00:00
17a8dc8a94b18b396492735de8cd89bc2cf45c97
21 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Anso
|
1a03cf82af |
fix(mesh): auto-fallback through candidate subnets when default overlaps (#1156)
The default mesh subnet 172.30.0.0/24 is fully contained in linuxserver/* default networks (sonarr_default 172.30.0.0/16, etc.), so libnetwork rejects the IPAM allocation with "Pool overlaps with other one on this address space" on a typical homelab Docker host. The single hard-coded default left first-run operators with a silently broken mesh. MeshService.setupMeshNetwork now resolves the subnet via three paths: 1. Operator-explicit (SENCHO_MESH_SUBNET set): use that subnet, strict. Pre-existing sencho_mesh with a different subnet still raises subnet_mismatch. 2. Adopt-existing (env unset, sencho_mesh already on the daemon): adopt the existing subnet. Docker is the source of truth across restarts. 3. Candidate iteration (env unset, no existing network): walk 172.30.0.0/24, 172.31.0.0/24, 10.42.0.0/24, 10.43.0.0/24 in order. First subnet Docker accepts wins. If every candidate overlaps, record subnet_overlap with a message naming every attempt. The dashboard's Fleet Heartbeat card now surfaces the down state via a compact banner above the per-node rows, plus a "mesh down" counter suffix on the right of the title. The existing Routing-tab banner is extracted into a shared MeshDataPlaneBanner component with tab and card variants. Dashboard polling is gated on Admiral tier so non-paid users do not fire the Admiral-only /mesh/status endpoint. Six new tests in mesh-setup-error-classification cover: iterates past first overlap, all candidates overlap, adopts existing network, inspectNetwork non-404 failure classified as attach_failed, env-matches- existing skip-create, and operator-explicit strict (no fallback). Fixes F-1 in the pre-1.0 audit. Closes the silent-failure mode that left the mesh down on the most common homelab Docker layout. |
||
|
Anso
|
e65c5e8551 |
fix(pilot): post-merge audit followups (WS via getProxyTarget, closeTunnel lifecycle, mesh source buffer, docs) (#1128)
* fix(pilot): route remote WS upgrades through NodeRegistry.getProxyTarget Pilot-mode nodes carry empty api_url and api_token by design and expose their API on a per-tunnel loopback bridge. The upgrade handler gated the remote-forwarder branch on `node.api_url && node.api_token`, so WS requests targeting pilot nodes silently fell through to the local handlers (live logs, exec, generic) instead of tunneling to the agent. Resolve the target via NodeRegistry.getProxyTarget so pilot and proxy modes share one dispatch path, mirroring the HTTP proxy. handleRemoteForwarder now takes the resolved target and, when the target is the pilot loopback (empty token), skips the console-token exchange and the Authorization injection so the tunnel-side auth is the only source of truth on that path. Unresolvable targets reject the upgrade with HTTP 503 instead of being served gateway-local data. * fix(pilot): emit tunnel-down and mark node offline on closeTunnel PilotTunnelManager.closeTunnel closed the underlying WebSocket but skipped the cleanup the natural-disconnect path runs, so explicit closures (enrollment regenerate, node deletion) left the node row at status='online' until the next reconnect. The dashboard kept showing the stale state for the entire interval. closeTunnel now writes nodes.status='offline' and emits tunnel-down for pilot bridges, and emits proxy-bridge-down for central-initiated proxy bridges. The maps are cleared before bridge.close() so the natural 'closed' handler's bridge-identity guard short-circuits and we do not double-emit. * fix(mesh): buffer cross-node source data until tcp_open_ack arrives openCrossNode piped src socket data straight to tcpStream.write before the forward TcpStream emitted 'open'. The first packet on a fresh cross-node stream raced ahead of the agent's tcp_open_ack on the wire, which broke protocols that send immediately after connect (HTTP, TLS, Redis, Postgres) on Pilot and proxy mesh paths. Buffer src chunks in a local array capped at STREAM_PENDING_DATA_MAX_BYTES until tcpStream emits 'open', then flush them in order before any post-open writes. Tear down both sockets if the buffer overflows so a misbehaving source cannot exhaust gateway memory while waiting for the ack. * docs(pilot): clarify host-console non-parity and narrow the parity claim Pilot mode disables the host-console capability at the capability registry (the agent container has no useful host shell to surface), but the public docs listed host console among the WebSockets that ride through the tunnel and described pilot as behaving identically to proxy mode. State the shared-capability claim more carefully and call out the intentional non-parity in a dedicated subsection. |
||
|
Anso
|
982c7830b1 |
fix(mesh): address audit follow-ups (early-data, recompose, admiral gate, error codes, docs) (#1126)
* fix(mesh): buffer early TcpData on reverse-relay path
The reverse-relay code dropped the local-shaped reservation and any
TcpData frames buffered in it before acceptReverseRelay had wired up
the target stream. A peer that sent a request body immediately after
tcp_open_reverse lost those bytes when the target lived on a third
node, since reverse_local buffers them but reverse_relay did not.
Carry the reservation through acceptReverseRelay: transplant its
pendingData and pendingBytes into the new reverse_relay state, gate
TcpData writes on targetOpen, and flush buffered frames into the
target stream before sending tcp_open_ack. Mirrors the reverse_local
pattern. Regression test fires tcp_open_reverse plus an immediate
TcpData while ensureBridge is in flight and verifies the bytes
arrive intact, in order, before the ack.
* fix(mesh): recompose affected stacks when node-level mesh is disabled
disableForNode used to clear DB rows and override files but leave the
running containers attached to sencho_mesh with stale /etc/hosts
alias entries until an operator redeployed every stack by hand.
Mirror optOutStack: after the existing alias/forwarder cleanup, call
regenerateOverridesAcrossFleet, cascadeRecomposeAcrossFleet, and
triggerRedeploy for each previously meshed stack on the disabled
node so containers detach from sencho_mesh and shed the alias
entries they owned. The disabled node's mesh_stacks rows are deleted
before the cascade so listMeshStacks returns the right set with no
skip tuple required. Route threads the actor through actorFor(req)
for parity with optInStack/optOutStack. Tests cover the redeploy
fan-out, the cascade no-skip-tuple invariant, and the default actor
fallback for non-route callers.
* fix(mesh): require Admiral on the WS proxy-tunnel upgrade
HTTP mesh routes in routes/mesh.ts all enforce requireAdmiral, but
the /api/mesh/proxy-tunnel WS upgrade accepted any node_proxy or
full-admin api_token regardless of the receiver's license. A node
downgraded from Admiral kept serving mesh data-plane traffic to a
sibling central while refusing every mesh management call.
Read the receiver's local LicenseService at the upgrade and 403 when
the tier is not paid+admiral. The check sits after the existing
credential gate and uses LicenseService directly rather than
effectiveTier (which trusts forwarded proxy headers); a remote peer
dialing in cannot be trusted to assert our entitlement. Dialer and
node_proxy token format are unchanged. Three regression tests cover
community-tier node_proxy, skipper-tier node_proxy, and
community-tier full-admin api_token all rejected with 403.
* fix(mesh): handle no_target alongside push_failed in inspectStackServices
proxyFetch throws MeshError('no_target') when getProxyTarget returns
null (pilot tunnel offline, proxy bridge unreachable), but the
inspectStackServices catch branch only matched push_failed. Offline
remotes fell through to the generic 'remote unreachable' error log,
which the Routing tab surfaces as an unexpected fault.
Match both error codes and emit the operator-friendly warn message
that names the unreachable node and the error code. Regression test
spies on console.warn/console.error to pin the branch.
* docs(mesh): align env defaults and forwarder comments with current architecture
SENCHO_MESH_PROXY_TUNNEL_IDLE_MS in .env.example carried the old
five-minute idle-close value (=300000), but the code default is
DEFAULT_IDLE_TTL_MS=0 (persistent tunnel). Copying the example
silently reintroduced the idle-close behavior the dialer removed.
MeshForwarder.ts's leading docblock and inline listen comment still
described host-network mode plus extra_hosts: host-gateway as
required for forwarder reachability. Sencho runs in standard bridge
mode and attaches to the shared sencho_mesh network at a stable IP;
meshed user containers reach the forwarder by that IP directly.
Flip the env default to 0, rewrite the env comment to describe the
persistent behavior and the opt-in for idle teardown, and rewrite
both forwarder comments to match the bridge-network reality.
* fix(mesh): trust forwarded tier on proxy-tunnel WS and remove remote overrides on disable
Admiral entitlement on the WS data plane now follows the same trust model
as the HTTP mesh routes: the central asserts its tier via x-sencho-tier
and x-sencho-variant on the WS handshake, and the receiver trusts those
headers only when the upgrade carries a node_proxy credential. When no
headers are present or the credential is a full-admin api_token, the
receiver falls back to its own local license. Without this an Admiral
central could be rejected by a Community remote and a Community central
could dial a locally-Admiral remote.
disableForNode now routes through removeOverrideFromNode so override
files pushed earlier via applyLocalOverride are removed on remote nodes
via DELETE /api/mesh/local-override/:stack. Sequential awaits are wrapped
in Promise.allSettled to match regenerateOverridesForNode's parallel
push pattern.
Other changes:
- Cover the post-state-swap buffering window in the reverse-relay test
(TcpData arriving after openTcpStream returns but before target open).
- Refresh stale MeshService comments that still referenced the removed
sidecar layer and host-network listener model.
* test(mesh): pin removeOverrideFromNode remote HTTP shape
The disableForNode regression test mocks removeOverrideFromNode itself,
so a regression inside the helper would not be caught. Add a narrow
contract test that spies on global fetch and asserts the request shape:
DELETE /api/mesh/local-override/:stack against the resolved proxy
target, with Authorization Bearer plus the x-sencho-tier and
x-sencho-variant headers. Also covers the encodeURIComponent path and
the swallowed-network-error behavior the disable cascade relies on.
* test(mesh): use createTestApiToken helper in proxy-tunnel api_token gate test
The full-admin api_token branch of the WS Admiral gate test inlined the
canonical generateApiToken + sha256 + addApiToken triple that already
lives in the createTestApiToken helper. Switching to the helper removes
the duplicated insertion logic and aligns this test with the helper used
by the other api_token call sites.
|
||
|
Anso
|
0cffd60173 |
fix(mesh): clean up activeStreams on src socket close/error (F-10) (#1105)
The cross-node MeshService.openCrossNode handler attached src.on('close')
and src.on('error') listeners that only called tcpStream.destroy() and
never deleted the activeStreams Map entry. MeshTcpStreamLike.destroy()
sends a tcp_close frame to the remote but does not synchronously emit
the local close event; that fires only when the remote sends back a
tcp_close ack via _dispatchClose (or when the entire tunnel tears down
and the bridge force-emits close on every stream).
Failed dials whose remote ack was lost (peer gone, network drop, or the
F-9-era timeouts) therefore leaked records into activeStreams until the
tunnel itself idle-closed, producing the monotonically-climbing
activeStreamCount documented in the mesh E2E audit (F-10).
Add a cleanupRecord() closure that clears the open timer and deletes
the record idempotently, then route all four lifecycle handlers
(tcpStream.on('error'), tcpStream.on('close'), src.on('close'),
src.on('error')) through it. Map.delete is naturally idempotent so the
double-delete that fires when both sides close normally is a free
no-op. The same-node openSameNode path already had this shape via its
existing teardown() closure; this brings the cross-node path to parity.
Three new Vitest cases in mesh-service.test.ts use an EventEmitter-
backed fake src so emit('close') actually delivers to the listener
(the existing makeFakeSocket uses vi.fn for .on which records calls
but never fires them). They cover src.close, src.error, and the
idempotency case where both src and tcpStream emit close in sequence.
|
||
|
Anso
|
f6e42535c8 |
fix(mesh): route peer→central traffic over the existing forward WS (#1094)
* fix(mesh): route peer→central traffic over the existing forward WS The reverse mesh callback path (`/api/mesh/proxy-tunnel-from-peer`) needed SENCHO_PRIMARY_URL on central plus a publicly reachable origin from the peer's perspective. In a typical homelab where central sits behind NAT, peer→central dispatch silently failed at the dialer's short-circuit and the headline "call any service on any node by hostname" worked one way only. The forward WS at `/api/mesh/proxy-tunnel` is already bidirectional end to end. Make the bridge a persistent control-plane primitive: dial every mesh-enabled proxy peer at startup, reconcile every 60 s, never idle-close. Peer→central traffic multiplexes over the same WS via `tcp_open_reverse`. Removed: - `meshProxyTunnelFromPeer.ts` WS handler and dispatch - `MeshCentralRegistry`, `PeerToCentralMeshSessionDialer` - `mesh_handshake` first-frame state machine in `meshProxyTunnel.ts` - `maybeSendBootstrap`, `buildHandshakeFrame` in the dialer - `mesh_proxy_callback_bootstrap` capability and `maybeWarnUnsetPrimaryUrl` - `mesh_centrals` table (drop migration; greenfield, no users) - `PilotTunnelManager.replaceOrRegisterProxyBridge` (dead after handler removal) - twelve associated unit/integration tests plus the peer-recovery branch in `MeshService.openCrossNode` Added: - `MeshService.proactiveBridgeFanout` selects every mesh-enabled proxy peer (no longer gated on `mesh_stacks` rows) - `startBridgeReconcileLoop` runs the fanout every 60 s (override via `SENCHO_MESH_RECONCILE_INTERVAL_MS`) - `MeshProxyTunnelDialer` default idle TTL is now `0` and exposes `isDialing(nodeId)` for the status surface - `MeshNodeStatus.reverseCallbackStatus` discriminator (`connected | connecting | unavailable | not_applicable`) surfaced via `/api/mesh/status` and rendered as a pill in the Routing tab - `openCrossNode` error message distinguishes "no proxy target" from "waiting for central to dial the reverse bridge" - New tests: `mesh-service-proxy-tunnel-reconcile`, `mesh-status-reverse-callback`, `mesh-proxy-tunnel-dialer-no-idle-close` SENCHO_PRIMARY_URL is no longer required for any mesh function. * fix(mesh): rewrite proxy-tunnel reconcile test contents The previous commit renamed the file but the rewritten test bodies stayed unstaged on top of the rename. This commit lands the actual rewrite: the fanout assertion now requires every mesh-enabled proxy peer to be dialed, not just those with `mesh_stacks` rows, and adds a reconcile-tick repeated-call test. |
||
|
Anso
|
1d7418a233 |
fix(mesh): recompose previously-meshed containers when alias set changes (#1090)
When a stack is opted into the mesh, every previously-meshed stack's override.yml on disk gains the new alias, but the running containers keep their stale /etc/hosts because extra_hosts is read at container creation time. Cross-stack DNS for the new alias silently fails until an operator manually redeploys each prior stack. Complete the cascade: after regenerateOverridesAcrossFleet finishes, fan out triggerRedeploy across every (node_id, stack_name) tuple in mesh_stacks, skipping the just-opted-in pair (the explicit triggerRedeploy at the end of optInStack covers it). Mirror the same cascade in optOutStack so the dropped alias exits every container's /etc/hosts. triggerRedeploy already dispatches local vs remote, already wraps runRedeploy in fire-and-forget error logging to the mesh activity ring and audit log, and already enforces compose policy gates. The cascade just reuses it. regenerateAllOverrides (boot and manual /regen-overrides) is intentionally NOT covered: a Sencho restart must not force a fleet-wide recompose; override files alone are sufficient there. Adds four unit tests in mesh-service.test.ts: cascade fires for every prior meshed stack on opt-in, no double-redeploy of the just-opted-in tuple, cascade is a no-op when no peers exist, cascade fires for every survivor on opt-out. |
||
|
Anso
|
a32183d198 |
fix(mesh): cascade override regen across all meshed nodes on opt-in/opt-out (#1079)
Opting a stack into mesh on node A now propagates the new alias to every meshed node's override file, not just stacks on node A. Same on opt-out. Pre-fix, optInStack and optOutStack called regenerateOverridesForNode which only walks db.listMeshStacks(nodeId). Other meshed stacks on other nodes did not learn about the new alias until something else re-pushed their overrides; operators worked around it with a manual POST /api/mesh/regen-overrides plus a redeploy of each affected stack. New regenerateOverridesAcrossFleet helper iterates db.listMeshStacks() (no arg, fleet-wide) under Promise.allSettled, skipping the (nodeId, stackName) tuple opt-in just pushed loudly. Per-stack failures surface as forwarder.error activity events, matching the pattern used by regenerateAllOverrides. Offline remote nodes leave stale overrides until the next opt-in/opt-out, the next tunnel reconnect, or a manual rerun of the regen-overrides endpoint. regenerateOverridesForNode stays for the tunnel-up retry listener: a reconnecting pilot only needs its own stacks re-pushed, not the fleet. Tests: redirect five existing spies in mesh-service.test.ts to the new method; add two new cases asserting cross-node cascade for opt-in and opt-out. 52 tests in mesh-service.test.ts pass. |
||
|
Anso
|
54d84a7edb |
fix(mesh): skip peer-side recovery in openCrossNode on central instances (#1068)
The R1-A2 symmetric-callback work added a peer-side recovery branch to
MeshService.openCrossNode: when reverseDialer is null, kick
PeerToCentralMeshSessionDialer.ensureSession so the peer re-opens its
callback WS to central before the cross-node dispatch.
The guard `!this.reverseDialer` did not distinguish "I am a peer waiting
for central to dial in" from "I am central, and the reverseDialer concept
does not apply to me." On central, reverseDialer is always null (central
is the bridge initiator side and never has another node dial into it), so
central entered the recovery branch on every forward dispatch, found no
session (central has no mesh_centrals row), and destroyed the inbound
socket with route.resolve.fail forward-from-peer no_session.
Gate the branch on `MeshCentralRegistry.getActive() !== null` so it only
runs when this Sencho is acting as a proxy-mode peer that has actually
been bootstrapped. Central falls straight through to dialMeshTcpStream.
The peer cold-start scenario the comment originally described is still
covered: a proxy-mode peer with a cached mesh_centrals row but no live
WS to central will fire the recovery branch as before.
Adds two regression tests:
- central path (no mesh_centrals row, no reverseDialer) skips recovery
and calls dialMeshTcpStream
- proxy-peer path (mesh_centrals row + ensureSession returns null)
still emits route.resolve.fail forward-from-peer no_session
Existing tests in the "openCrossNode (BUG-4)" block already used a stub
reverseDialer to bypass the recovery branch, which is why the central
regression was not caught pre-release. The new tests deliberately leave
reverseDialer null to exercise the gating.
|
||
|
Anso
|
cf618dd866 |
feat(mesh): symmetric WS dial for proxy-mode mesh peers (#1066)
* chore(mesh): foundation for symmetric callback dial Adds the data-plane scaffolding that the symmetric callback dial fix builds on: - mesh_centrals table for peer-side bootstrap material - MeshCentralRegistry service (upsert/getActive/clear/markUsed/markRejected) - PilotTunnelManager kind discriminator and replaceOrRegisterProxyBridge - mesh_proxy_callback_bootstrap capability registration - MeshProxyTunnelDialer reason-tagged proxy-bridge-down events from a single tearDownBridge emission point - Reactive redial scheduler that skips idle and auth_failed reasons * feat(mesh): add reverse-direction activity log entries (closes R1-B) acceptReverseLocal now emits route.resolve.ok with direction=reverse on connect ack and route.resolve.fail with direction=reverse plus reason=container_not_found / connect_error pre-connect. Post-connect close/error stays silent. Reuses existing event types via the new details.direction discriminator so frontend filters are unaffected. * feat(mesh): add peer-to-central callback dial path (closes R1-A2) Closes the architectural gap where proxy-mode mesh peers could not re-establish their tunnel to central after any non-idle bridge teardown (idle close, network blip, central restart, peer reboot). Central remains the hub for the data plane; the change is purely about WS initiation. Symmetric WS initiation, asymmetric protocol roles. Central retains PilotTunnelBridge ownership; peer retains TcpStreamSwitchboard + reverseDialer ownership. Central bootstraps callback credentials over the first authenticated central-initiated mesh tunnel via a one-shot mesh_handshake JSON frame; peer persists the material in a new mesh_centrals SQLite table and dials central's new /api/mesh/proxy-tunnel-from-peer endpoint when local cross-node traffic needs a bridge and none is live. Mesh_tunnel JWT (HS256, signed with auth_jwt_secret) carries scope, audience, issuer (central instance id), peer api_token fingerprint, kid. Validation on inbound peer dial: algorithm pin, signature, scope, audience, instance, time bounds, node existence and mode, fingerprint match. Failures return HTTP 401 with a machine-readable reason; peer routes the response per a clear-vs-keep cache matrix. Triggers proactive bootstrap on mesh-enable and api_token rotation; central startup fans out to mesh-enabled proxy-mode nodes with mesh_stacks rows (throttled, fire-and-forget). Reactive redial on non-idle bridge loss. Capability-gated handshake send (mesh_proxy_callback_bootstrap) makes the upgrade path safe against older peers in mixed-version fleets. Adds peer-side /api/system/pilot-tunnels centralCallback diag block, bounded counter metrics for bootstrap and dial events. SENCHO_PRIMARY_URL preflight warning when unset on a central with mesh-enabled proxy nodes. Tested with unit suites for the validation chain, registry, manager, and both dialers; integration tests for bootstrap E2E (asserts protocol-role invariant), api_token rotation, instance id change, version skew, and pilot-mode regression. * fix(mesh): green CI on the symmetric callback branch Two independent CI failures, both surgical: 1. Backend tests (11 fails): four mesh test files called setupTestDb in beforeEach. setupTestDb does not reset the DatabaseService singleton, so the per-test afterEach rm of the previous tmpdir left the singleton connection pointing at a deleted file. The next beforeEach's line-55 write threw SQLITE_READONLY_DBMOVED on Linux. Windows file-lock semantics hid this locally. Hoist setupTestDb / cleanupTestDb to file-scope beforeAll / afterAll; per-test state resets stay in beforeEach. Matches the convention in the eight mesh test files that already pass. 2. CodeQL (4 high alerts): js/insufficient-password-hash flagged sha256(api_token) at four sites. The api_token is a 256-bit opaque bearer (sen_sk_-prefixed), not a human password; sha256 is the correct fingerprint primitive for binding the mesh_tunnel JWT to a specific token. Add the two production files plus the two test files that mint the fingerprint to the existing path-scoped query-filter for that rule. * fix(mesh): drop unused afterEach import and revert dead codeql config ESLint flagged afterEach as unused in mesh-central-registry.test.ts:1 after the previous commit hoisted setup/teardown to file-scope beforeAll/afterAll. Remove from the vitest import line. Revert the codeql-config.yml additions from the previous commit. The paths: sub-key under query-filters > exclude is not a documented CodeQL feature and silently no-ops. The four js/insufficient-password-hash alerts on api_token fingerprinting are tracked as dismissed false positives in the GitHub Security tab rather than via dead config. |
||
|
Anso
|
489aab4516 |
fix(mesh): testUpstream dials via ensureBridge so proxy-mode targets are probeable (#1057)
testUpstream short-circuited on hasActiveTunnel + getBridge (pilot-only) when probing a cross-node alias. Proxy-mode remotes never have a "pilot tunnel" in the manager's sense, so any operator hitting "Test alias" on a proxy-mode target returned tunnel_down even when the cross-node dial via dialMeshTcpStream worked fine. Replace with await ptm.ensureBridge(target.nodeId), matching the mode- agnostic dispatch dialMeshTcpStream already uses. ensureBridge returns an existing pilot tunnel, an existing proxy bridge, or asks MeshProxyTunnelDialer to dial on demand. The 'no pilot tunnel' vs 'no bridge' distinction goes away (they meant the same thing). Adds two test cases: probing a proxy-mode remote via on-demand bridge, and the tunnel_down path when ensureBridge yields null. |
||
|
Anso
|
3d5f0ffacd |
fix(mesh): reject opt-in when every service has empty ports (#1056)
optInStack rejected empty service lists but not the case where services exist with every ports: []. The newPorts Set ended up empty, the function sailed past collision checks, wrote a mesh_stacks row, and produced an empty-aliases override. The stack then appeared online in the Routing tab but no traffic actually routed. Add a check after building newPorts: throw no_target with a clear message before the SENCHO_LISTEN_PORT collision check. Reuses the existing MeshError code (semantically: no routable target) so the frontend's error toast path is unchanged. |
||
|
Anso
|
6185206323 |
fix(mesh): proxy peer learns its central-namespace nodeId at tunnel upgrade (#1055)
Proxy-mode peers receive an alias overlay from central with nodeIds in central's namespace, but had no source for their own nodeId in that namespace. selfCentralNodeId fell back to the peer's local DB default (always 1), and handleAccept falsely matched cross-node aliases (nodeId=1, central's Local) against the peer's self-id. Cross-node aliases routed to openSameNode and surfaced as route.resolve.denied. Central now appends ?nodeId=<central-namespace-id> to the proxy-tunnel WS URL. The peer's upgrade handler parses it with a strict integer regex, installs into MeshService before the reverse dialer is registered, and clears on disconnect. handleAccept resolves self-id in priority order: proxy-tunnel install > SENCHO_ENROLL_TOKEN > local default. A non-null overwrite to a different value warns loudly (misconfigured deployment). Missing or malformed query param logs a warn and proceeds, leaving reverse-direction dispatch undefined until both sides are upgraded. Adds 5 test cases: nodeId install + teardown, missing/malformed input (including decimal, leading zeros, exponent, whitespace), overwrite warn, and CAS-rejected reverse-dialer leaving the slot unchanged. |
||
|
Anso
|
41a1df279d |
fix(mesh): fix pilot handleAccept dispatch by deriving self nodeId from enrollment token (#1023)
On a pilot node, handleAccept compared target.nodeId (from central's perspective via pilotAliasOverlay) against NodeRegistry.getDefaultNodeId() which always returns 1. This inverted same-node vs cross-node dispatch: - central's alias (nodeId=1) matched localNodeId=1 -> openSameNode on pilot (no container -> silent close, 0 bytes) - pilot's own alias (nodeId=14) != 1 -> openCrossNode -> tunnel loop Fix: add resolveSelfCentralNodeId() that decodes the nodeId claim from SENCHO_ENROLL_TOKEN (present on every pilot; payload decode only, no signature verification). Returns getDefaultNodeId() on central (env unset). Cached in selfCentralNodeId at start(); handleAccept reads the cache. After this fix the reverse direction (pilot-prober -> central echo) routes through openCrossNode (reverse tunnel) and the same-node path (pilot-prober -> pilot echo) routes through openSameNode (local Dockerode dial). Closes BUG-3 banner data-path regression. Tests: 5 new cases covering resolveSelfCentralNodeId token extraction, fallback paths, and handleAccept dispatch routing on a simulated pilot. |
||
|
Anso
|
c12b3ba995 |
fix(mesh): apply D-1 pushed override on pilot deploys via file-presence fallback (#1019)
When isMeshStackEnabled returns false, ensureStackOverride now checks for an override file already on disk before returning null. On pilot nodes the mesh_stacks table is intentionally empty (opt-in state lives on central per the C-3 design), so the DB gate blocked the override that central had already pushed via applyLocalOverride. File-presence is safe as the fallback because removeOverrideFromNode sends a DELETE to the pilot when a stack is opted out, so a stale file cannot survive past opt-out. Also aligns removeStackOverride to use path.basename consistently with all other override-path construction sites in the same file. Adds two tests: pilot node with a pushed file returns the path; pilot node with no file returns null. |
||
|
Anso
|
2d9e4d0480 |
fix(mesh): derive override services from compose file and surface cross-node dispatch (#1017)
* fix(mesh): derive override services from compose file and surface cross-node dispatch
Three fixes for issues found during Phase B end-to-end verification:
1. Override generator no longer queries Dockerode for services. Deploys
removed all containers in the project right before composeArgs ran,
so docker.listContainers({label: project=...}) returned [] and the
generator wrote services: {} which broke meshed user containers.
New helper getDeclaredStackServiceNames reads the local stack's
compose file via FileSystemService + YAML.parse and returns the
top-level services keys. Used from ensureStackOverride and
applyLocalOverride. Defensive fallback: when the read returns []
AND a previous override on disk has a non-empty services map,
keep the file and emit mesh.override.preserved.
2. Boot regen race against the pilot tunnel. regenerateAllOverrides
ran ~1s before the first pilot tunnel-up event, so pushes to
pilot-mode nodes failed with "no proxy target". MeshService.start
now also runs regenerateOverridesForNode(nodeId) inside the
tunnel-up listener, covering both the boot-race window and
mid-runtime tunnel reconnects. Idempotent.
3. openCrossNode emitted no activity log when a cross-node dial got
stuck waiting for tcp_open_ack. The forwarder accepted the TCP
handshake (so nc -v reported open) but logged nothing between
accept and close. Now logs route.dispatch on entry and emits
tunnel.fail after PROBE_TIMEOUT_MS if the agent never acks.
Timer is cleared on every tcpStream and src lifecycle event.
Tests: 8 new in mesh-service.test.ts cover the helper, the BUG-1
override write path, the defensive preservation fallback, the
tunnel-up regen handler, and both cross-node dispatch logging
behaviors. Full vitest suite green (1998 tests pass).
* fix(mesh): satisfy CodeQL path-injection model in new override-read paths
Two new fs.readFile sites added in the previous commit failed the
CodeQL check on the PR even though both paths were guarded by
isValidStackName + isPathWithinBase. The file's existing
applyLocalOverride pattern uses path.basename(stackName) before the
join (commented as "Recognized by CodeQL's path-injection model");
apply the same sanitizer to the new code paths.
- getDeclaredStackServiceNames: wrap stackName in path.basename when
building composePath.
- readExistingOverrideServiceNames: take (dir, stackName) instead of
a pre-built filePath so the sanitizer lives next to the read sink.
- ensureStackOverride: build the override file path with
path.basename(stackName) for symmetry with applyLocalOverride.
Runtime behavior unchanged (path.basename is a no-op for any name
that passed isValidStackName). Tests still green.
|
||
|
Anso
|
7cca68f84a |
fix(mesh): operator-triggered override regen and boot lifecycle hardening (#1016)
Audit-driven follow-up to the F6+F7 fix. Closes the High-severity findings
from the mesh-dial-path audit without expanding into the architectural
follow-ups (JWT TTL, TLS enforcement, per-stack JWT scope) which need
their own design discussions.
Changes:
1. New `POST /api/mesh/regen-overrides` Admiral-gated endpoint that calls
`MeshService.regenerateAllOverrides()` and returns a structured
`MeshRegenSummary { regenerated, failures, skipped, reason? }`. Lets
the operator rerun boot regen after fixing a remote node that was
offline at central startup, instead of opt-out + opt-in for every
meshed stack on that node. Audit-log row carries the outcome
(success / partial / skipped / error).
2. `regenerateAllOverrides` now aggregates per-stack outcomes into a
single summary log row (`mesh override regen complete: N succeeded,
M failed across K node(s)`) with `failedNodeIds` in `details`. Per-
stack warnings still emitted. When skipped because `senchoIp` is
null, emits an explicit warn instead of silently no-oping.
3. `MeshService.start()` now wraps `refreshAliasCache` and
`syncForwarderListeners` in their own try/catch so a throw from
either step is logged and the boot continues to override regen.
The closing log line surfaces data-plane state ("MeshService started
(data plane ok)" or "(data plane unavailable (<reason>))") so an
operator grepping startup output sees a half-init mesh immediately.
4. `sanitizeForLog` wrap on four previously unsanitized `err.message`
writes (cross-node tcpStream error handler + three probe error
paths). Container IPs and local socket details no longer leak into
the activity buffer or the probe response body.
5. Updated three existing F6 regression tests for the new return shape
and aggregated summary message. Added two new tests: version-skew
404-push handling, and concurrent opt-in during regen.
|
||
|
Anso
|
0947a80cda |
fix(mesh): trust central for cross-node dial auth and regenerate overrides at boot (#1014)
Two bugs in the same Phase D follow-up surface, fixed together because they both block declaring B-verify complete on the production fleet. Cross-node mesh dials returned `denied` at the agent. The pilot's `tcp_open` handler in `agent.ts::resolveMeshTarget` consulted the local SQLite `mesh_stacks` table, which is no longer written to under the post-Phase D control plane (state lives only on central). Drop the check. The pilot tunnel JWT (scope `pilot_tunnel`, signed with central's `auth_jwt_secret`) authenticates the caller; the same trust model already applies to filesystem ops, exec, and container control over the same tunnel. Threat-model trade-off: a leaked `pilot_tunnel` JWT or compromised central can now dial any compose-managed service on the pilot. Containers without `com.docker.compose.project` + `com.docker.compose.service` labels remain unreachable via this path. `MeshService.start()` did not regenerate compose override files at boot. After a Sencho restart with missing overrides on disk, meshed user containers had no `extra_hosts` / `networks: [sencho_mesh]` injection until each stack was opted out and back in. Add `regenerateAllOverrides()` that walks every `mesh_stacks` row and re-pushes via `pushOverrideToNode`. Best-effort: per-stack failures log to the mesh activity buffer; `MeshService.start()` is fire-and-forget at startup so a slow remote node does not delay boot. Tests: - `pilot-agent-mesh-resolve.test.ts` (new): mocks dockerode and proves `resolveMeshTarget` no longer returns `denied` with an empty `mesh_stacks` table. - `mesh-service.test.ts`: three new cases for `regenerateAllOverrides` - fan-out across the fleet, skip when `senchoIp` is null, log per-stack warning on push failure without throwing. |
||
|
Anso
|
23bbee4f45 |
feat(mesh): replace host-mode with shared sencho_mesh Docker network (#1009)
* feat(mesh): replace host-mode with shared sencho_mesh Docker network Phase D of the mesh redesign: drop the operator's `network_mode: host` requirement and the `host-gateway` extra_hosts pattern that did not work on cloud iptables-restrictive distros (OCI, etc.) or Docker Desktop. Each Sencho creates a shared `sencho_mesh` Docker bridge network on boot (default subnet 172.30.0.0/24, override via SENCHO_MESH_SUBNET), pins itself at `<network>+2`, and attaches every meshed user service to the same bridge. Compose overrides now emit IP-based `extra_hosts` plus a top-level `networks` block declaring `sencho_mesh` external. Override delivery: central renders for local stacks; for remote stacks it sends the fleet alias list to the remote's new `PUT /api/mesh/local- override/:stackName` endpoint, which renders against the remote's OWN local senchoIp and writes under its OWN DATA_DIR. Each node may use a different subnet without coordination beyond the env var. Opt-in / opt-out now trigger an automatic redeploy of the affected stack via the existing deploy code path (local: ComposeService; remote: HTTP POST through proxyFetch). The frontend opt-in sheet shows a confirmation modal (ConfirmModal) before the mutation. Failed redeploys emit both a mesh activity event and a durable audit-log row. Hardening: - Reserve port 1852 at opt-in (prevents user containers from racing the Sencho API listener). - ensureMeshNetwork refuses to continue if `sencho_mesh` exists with a mismatched subnet rather than silently routing to the wrong IP. - Idempotent network connect/disconnect helpers in DockerController. - optInStack rolls back the DB row if the just-inserted stack's override push fails (no half-states surviving across calls). - regenerateOverridesForNode runs in parallel and skips the just- pushed stack on opt-in. Operator template: drop `network_mode: host`, restore `ports: ["1852:1852"]`. Mesh now works identically on Linux LAN, OCI, and Docker Desktop without firewall changes. Docs: rewrite docs/features/sencho-mesh.mdx around the shared bridge network, document SENCHO_MESH_SUBNET, surface the host-network-service opt-in restriction, and cross-link with the Pilot Agent docs. BREAKING CHANGE: the operator's `docker-compose.yml` no longer uses `network_mode: host`. After upgrading, redeploy any meshed stacks once so they pick up the new IP-based override and join `sencho_mesh`. * fix(mesh): wrap stackName with path.basename in local-override fs ops CodeQL flagged js/path-injection on the new applyLocalOverride and removeLocalOverride methods because they are publicly reachable and its data-flow model does not recognize isValidStackName / isPathWithinBase as sanitizers. The validation IS sufficient (the allowlist regex blocks path separators, the path-prefix check blocks escape), but path.basename is a model CodeQL recognizes and is purely defensive: for any input that already passes isValidStackName, basename is the identity. |
||
|
Anso
|
10a469ecb4 |
fix(mesh): bind every alias port on every node, not just local-owned (#1004)
syncForwarderListeners filtered the bind set to ports owned by the local node. That broke cross-node mesh routing end to end: meshed containers' extra_hosts: <alias>:host-gateway entries resolve to the SOURCE node's gateway, so the source node is where the inbound TCP connection lands. With the filter, the source node never bound the remote alias's port and the connection went nowhere. The architecture model documented in docs/internal/architecture/ mesh.md (data flow step 3) is explicit: monolith's Sencho process accepts the connection on its MeshForwarder listener bound to port 5432 — where monolith is the source and opsix is the target. Every meshed node binds every alias port. handleAccept then resolves the alias and dispatches to openSameNode (when target.nodeId equals the local node) or openCrossNode (otherwise). Both branches were already correct; only the bind filter was wrong. Fleet-wide port collisions remain blocked at opt-in time (optInStack checks aliasByPort), so binding every alias port is unambiguous. New vitest case asserts the want-set includes both local-owned and remote-owned ports. Discovered while running PR 0 verification on v0.75.0 against Local + sencho-pilot-test: connections from a Local prober to the pilot's alias port hit Local's host with no listener. |
||
|
Anso
|
f599110386 |
feat(mesh): collapse sidecar into Sencho process via in-process forwarder (#1000)
The separate saelix/sencho-mesh sidecar container is gone. The
forwarder logic that previously lived in mesh-sidecar/src/forwarder.ts
moves into the Sencho process as backend/src/services/MeshForwarder.ts,
a thin per-port net.Server lifecycle wrapper. MeshService implements
the host interface and owns resolve plus splice; MeshForwarder owns
listener boilerplate. One container per node, no separate image to
publish, no control WebSocket.
Operator-facing change: the Sencho container now runs in
network_mode: host so the forwarder can bind alias ports on the host
network where meshed containers' extra_hosts host-gateway entries
point. Without host network mode the listeners would land in the
container's namespace and inbound traffic from peers would never
reach them. The 1852:1852 port publish becomes a no-op under host
mode and is commented out in the operator template.
Same-node forward path now dials the target container's bridge IP
via Dockerode (preferring the compose default network for
deterministic selection across daemon versions) instead of 127.0.0.1.
The legacy 127.0.0.1 path only worked when the target service
published its port to the host; the IP path works regardless.
Cross-node mesh routing in this phase is central -> pilot direction
only via PilotTunnelManager.openTcpStream. Pilot -> central and
pilot <-> pilot via central relay land in Phase B with the
tcp_open_reverse frame.
Deletions:
- mesh-sidecar/ package entirely (Dockerfile, package, sources, tests)
- backend/src/websocket/meshControl.ts
- MeshService sidecar lifecycle: spawnSidecar, stopSidecar,
isSidecarRunning, mintSidecarToken, verifySidecarToken,
attachSidecarSocket, handleSidecarResolve, sendSidecar
- POST /api/mesh/nodes/:id/sidecar/restart route
- /api/mesh/control WS dispatch in upgradeHandler
Type cleanup: 'sidecar' literal removed from MeshActivitySource and
MeshProbeResult.where (also the frontend mirror). MeshNodeStatus
sidecarRunning becomes localForwarderListening (boolean | null) so
non-local nodes get a null instead of an unconditional false; the
honest semantic is "this view only knows the local forwarder state;
remote forwarder status lands in Phase B." MeshNodeDiagnostic
sidecar object becomes forwarder { listening, listenerCount }.
Frontend MeshDiagnosticsSheet drops the restart-sidecar action and
sidecar liveness card; surfaces forwarder state plus a "runs
in-process; no separate container" caption.
Resolves audit findings C-1 (data plane non-functional), C-2 (sidecar
control WS not loopback-enforced), C-4 (sidecar lifecycle Dockerode-
on-remote, PR #999 closed), and C-5 (saelix/sencho-mesh:latest
unreachable). C-3 (PR #992) is unchanged. M-12 (PR #994) is
unchanged.
|
||
|
Anso
|
7663f4cd8b |
feat(fleet): sencho mesh in traffic and routing tab (#858)
* feat(fleet): sencho mesh in traffic and routing tab Lights up Sencho Mesh: cross-node container forwarding rendered as if the container next to you were on localhost. Builds on the dormant TCP frame plumbing from the prior PR (pilot tunnel TCP frames + sencho-mesh sidecar package) and exposes the Admiral-only orchestrator surface. Backend - New mesh_stacks table (per-node opt-ins) + nodes.mesh_enabled column via DatabaseService.migrateMeshTables. - MeshService singleton: sidecar lifecycle via Dockerode, opt-in/out with cascading override regeneration, request-based resolver from sidecar control WS, cross-node TCP forwarding via PilotTunnelManager (same-node fast path included), in-memory 1000-event activity ring buffer with durable mirror to audit_log for state-change events, per-node and per-route diagnostics, and the Test upstream probe. - MeshComposeOverride: pure YAML generator that injects extra_hosts using host-gateway. The user's docker-compose.yml is never mutated; overrides live under DATA_DIR/mesh/overrides. - ComposeService deploy/update splice the override file when the stack is opted in; non-mesh stacks behave identically to today. - Pilot agent resolveMeshTarget consults the local mesh_stacks table (defense in depth) and resolves Compose containers via Dockerode. - /api/mesh router with 13 Admiral-gated endpoints covering status, enable/disable, stack opt-in/out, alias listing, per-route diagnostic, Test upstream probe, per-node diagnostic, sidecar restart, activity log paginated and SSE. - meshControl WS slot at /api/mesh/control validates the mesh_sidecar JWT minted by MeshService; dispatched as upgrade slot 2 (canonical order preserved). Frontend - New Traffic Routing tab in FleetView, gated by isAdmiral and wrapped in AdmiralGate. Tab uses the cyan brand glyph and italic-serif state typography from the audit. - RoutingTab masthead with mesh activity drawer, per-node card grid with TogglePill, alias rows with five-state pill taxonomy (healthy / degraded / unreachable / tunnel-down / not-authorized), inline Test buttons. - Four sheets: opt-in picker with port-collision inline error, per-route detail with diagnostic + filtered activity, per-node diagnostics with active streams + resolver cache + restart action, fleet-wide activity log with filters. - meshRouteState helper centralizes pill-state mapping; pure-function tests cover all five states. Docs - User docs at /docs/features/sencho-mesh.mdx covering opt-in, troubleshooting, security model (4 guarantees + 4 explicit non-guarantees), and V1 limitations. - Internal architecture and runbook pages. - websocket-dispatch internal doc updated with the new slot. * fix(mesh): validate stack name before path use; fix test DB lifecycle Two surgical fixes against the prior PR. Path-injection (CodeQL js/path-injection): MeshService.optInStack, optOutStack, ensureStackOverride, and removeStackOverride now validate stackName via isValidStackName from utils/validation, reject malicious names at the API boundary, and additionally check isPathWithinBase on the resolved override file path for defense in depth. The dataflow from req.params.stackName to fs.writeFile no longer reaches an unsanitized path expression. Test DB lifecycle: mesh-service.test.ts used per-test setupTestDb / cleanupTestDb, which deletes the temp dir while DatabaseService still holds an open SQLite handle. On Linux CI this raises SQLITE_READONLY_DBMOVED on the next prepare() because the inode has been unlinked. Switched to file-scoped beforeAll/afterAll matching agents-routes.test.ts, with a per-test beforeEach that truncates mesh_stacks plus non-default nodes and resets the MeshService singleton in-memory state. Adds a new test case asserting the path-traversal rejection. * fix(compose): use discovered compose filename instead of hardcoded docker-compose.yml composeArgs() hardcoded `-f docker-compose.yml` for every deploy. Sencho writes its canonical compose file as `compose.yaml`, so any stack created via the UI failed to deploy with `open ...docker-compose.yml: no such file or directory`. When no mesh override applies, drop the explicit `-f` so docker compose's built-in discovery resolves the actual filename. When an override exists, look up the real base filename via FileSystemService.getComposeFilename() and pass both files explicitly. Also hoist the MeshService import to module top now that the dependency is known to be acyclic, and revert the matching unit-test assertion. |