9 Commits

Author	SHA1	Message	Date
Anso	982c7830b1	fix(mesh): address audit follow-ups (early-data, recompose, admiral gate, error codes, docs) (#1126) ... * fix(mesh): buffer early TcpData on reverse-relay path The reverse-relay code dropped the local-shaped reservation and any TcpData frames buffered in it before acceptReverseRelay had wired up the target stream. A peer that sent a request body immediately after tcp_open_reverse lost those bytes when the target lived on a third node, since reverse_local buffers them but reverse_relay did not. Carry the reservation through acceptReverseRelay: transplant its pendingData and pendingBytes into the new reverse_relay state, gate TcpData writes on targetOpen, and flush buffered frames into the target stream before sending tcp_open_ack. Mirrors the reverse_local pattern. Regression test fires tcp_open_reverse plus an immediate TcpData while ensureBridge is in flight and verifies the bytes arrive intact, in order, before the ack. * fix(mesh): recompose affected stacks when node-level mesh is disabled disableForNode used to clear DB rows and override files but leave the running containers attached to sencho_mesh with stale /etc/hosts alias entries until an operator redeployed every stack by hand. Mirror optOutStack: after the existing alias/forwarder cleanup, call regenerateOverridesAcrossFleet, cascadeRecomposeAcrossFleet, and triggerRedeploy for each previously meshed stack on the disabled node so containers detach from sencho_mesh and shed the alias entries they owned. The disabled node's mesh_stacks rows are deleted before the cascade so listMeshStacks returns the right set with no skip tuple required. Route threads the actor through actorFor(req) for parity with optInStack/optOutStack. Tests cover the redeploy fan-out, the cascade no-skip-tuple invariant, and the default actor fallback for non-route callers. * fix(mesh): require Admiral on the WS proxy-tunnel upgrade HTTP mesh routes in routes/mesh.ts all enforce requireAdmiral, but the /api/mesh/proxy-tunnel WS upgrade accepted any node_proxy or full-admin api_token regardless of the receiver's license. A node downgraded from Admiral kept serving mesh data-plane traffic to a sibling central while refusing every mesh management call. Read the receiver's local LicenseService at the upgrade and 403 when the tier is not paid+admiral. The check sits after the existing credential gate and uses LicenseService directly rather than effectiveTier (which trusts forwarded proxy headers); a remote peer dialing in cannot be trusted to assert our entitlement. Dialer and node_proxy token format are unchanged. Three regression tests cover community-tier node_proxy, skipper-tier node_proxy, and community-tier full-admin api_token all rejected with 403. * fix(mesh): handle no_target alongside push_failed in inspectStackServices proxyFetch throws MeshError('no_target') when getProxyTarget returns null (pilot tunnel offline, proxy bridge unreachable), but the inspectStackServices catch branch only matched push_failed. Offline remotes fell through to the generic 'remote unreachable' error log, which the Routing tab surfaces as an unexpected fault. Match both error codes and emit the operator-friendly warn message that names the unreachable node and the error code. Regression test spies on console.warn/console.error to pin the branch. * docs(mesh): align env defaults and forwarder comments with current architecture SENCHO_MESH_PROXY_TUNNEL_IDLE_MS in .env.example carried the old five-minute idle-close value (=300000), but the code default is DEFAULT_IDLE_TTL_MS=0 (persistent tunnel). Copying the example silently reintroduced the idle-close behavior the dialer removed. MeshForwarder.ts's leading docblock and inline listen comment still described host-network mode plus extra_hosts: host-gateway as required for forwarder reachability. Sencho runs in standard bridge mode and attaches to the shared sencho_mesh network at a stable IP; meshed user containers reach the forwarder by that IP directly. Flip the env default to 0, rewrite the env comment to describe the persistent behavior and the opt-in for idle teardown, and rewrite both forwarder comments to match the bridge-network reality. * fix(mesh): trust forwarded tier on proxy-tunnel WS and remove remote overrides on disable Admiral entitlement on the WS data plane now follows the same trust model as the HTTP mesh routes: the central asserts its tier via x-sencho-tier and x-sencho-variant on the WS handshake, and the receiver trusts those headers only when the upgrade carries a node_proxy credential. When no headers are present or the credential is a full-admin api_token, the receiver falls back to its own local license. Without this an Admiral central could be rejected by a Community remote and a Community central could dial a locally-Admiral remote. disableForNode now routes through removeOverrideFromNode so override files pushed earlier via applyLocalOverride are removed on remote nodes via DELETE /api/mesh/local-override/:stack. Sequential awaits are wrapped in Promise.allSettled to match regenerateOverridesForNode's parallel push pattern. Other changes: - Cover the post-state-swap buffering window in the reverse-relay test (TcpData arriving after openTcpStream returns but before target open). - Refresh stale MeshService comments that still referenced the removed sidecar layer and host-network listener model. * test(mesh): pin removeOverrideFromNode remote HTTP shape The disableForNode regression test mocks removeOverrideFromNode itself, so a regression inside the helper would not be caught. Add a narrow contract test that spies on global fetch and asserts the request shape: DELETE /api/mesh/local-override/:stack against the resolved proxy target, with Authorization Bearer plus the x-sencho-tier and x-sencho-variant headers. Also covers the encodeURIComponent path and the swallowed-network-error behavior the disable cascade relies on. * test(mesh): use createTestApiToken helper in proxy-tunnel api_token gate test The full-admin api_token branch of the WS Admiral gate test inlined the canonical generateApiToken + sha256 + addApiToken triple that already lives in the createTestApiToken helper. Switching to the helper removes the duplicated insertion logic and aligns this test with the helper used by the other api_token call sites.	2026-05-20 12:53:29 -04:00
Anso	a318e6b3c1	fix(mesh): close data-plane race that dropped early TcpData frames (#1086) ... The tcp_open and tcp_open_reverse receivers registered their stream entry only after awaiting resolveTarget / resolveContainerIp. The peer, which is free to send TcpData immediately after the open frame, hit the lookup-miss path in handleBinaryFrame and the bytes were silently dropped. Probes passed because they only exercise the handshake; real HTTP hung with 0 bytes received. Reserve the stream entry synchronously, buffer early TcpData in a per-stream pendingData queue capped at 1 MiB, and flush onto the local socket inside the connect handler before sending tcp_open_ack. The payload is copied because decodeBinaryFrame returns a subarray view of the WS receive buffer; holding the view would pin the parent buffer past its lifecycle. Both sides of the data plane are patched: - tcpStreamSwitchboard.onTcpOpen (forward, agent + proxy-mode peer) - PilotTunnelBridge.handleTcpOpenReverse / acceptReverseLocal (reverse, primary acting as the local dial target) Adds unit coverage for the race window and for the overflow path. The cap constant lives in pilot/protocol.ts alongside the other per-stream limits.	2026-05-17 14:01:39 -04:00
Anso	cf618dd866	feat(mesh): symmetric WS dial for proxy-mode mesh peers (#1066) ... * chore(mesh): foundation for symmetric callback dial Adds the data-plane scaffolding that the symmetric callback dial fix builds on: - mesh_centrals table for peer-side bootstrap material - MeshCentralRegistry service (upsert/getActive/clear/markUsed/markRejected) - PilotTunnelManager kind discriminator and replaceOrRegisterProxyBridge - mesh_proxy_callback_bootstrap capability registration - MeshProxyTunnelDialer reason-tagged proxy-bridge-down events from a single tearDownBridge emission point - Reactive redial scheduler that skips idle and auth_failed reasons * feat(mesh): add reverse-direction activity log entries (closes R1-B) acceptReverseLocal now emits route.resolve.ok with direction=reverse on connect ack and route.resolve.fail with direction=reverse plus reason=container_not_found / connect_error pre-connect. Post-connect close/error stays silent. Reuses existing event types via the new details.direction discriminator so frontend filters are unaffected. * feat(mesh): add peer-to-central callback dial path (closes R1-A2) Closes the architectural gap where proxy-mode mesh peers could not re-establish their tunnel to central after any non-idle bridge teardown (idle close, network blip, central restart, peer reboot). Central remains the hub for the data plane; the change is purely about WS initiation. Symmetric WS initiation, asymmetric protocol roles. Central retains PilotTunnelBridge ownership; peer retains TcpStreamSwitchboard + reverseDialer ownership. Central bootstraps callback credentials over the first authenticated central-initiated mesh tunnel via a one-shot mesh_handshake JSON frame; peer persists the material in a new mesh_centrals SQLite table and dials central's new /api/mesh/proxy-tunnel-from-peer endpoint when local cross-node traffic needs a bridge and none is live. Mesh_tunnel JWT (HS256, signed with auth_jwt_secret) carries scope, audience, issuer (central instance id), peer api_token fingerprint, kid. Validation on inbound peer dial: algorithm pin, signature, scope, audience, instance, time bounds, node existence and mode, fingerprint match. Failures return HTTP 401 with a machine-readable reason; peer routes the response per a clear-vs-keep cache matrix. Triggers proactive bootstrap on mesh-enable and api_token rotation; central startup fans out to mesh-enabled proxy-mode nodes with mesh_stacks rows (throttled, fire-and-forget). Reactive redial on non-idle bridge loss. Capability-gated handshake send (mesh_proxy_callback_bootstrap) makes the upgrade path safe against older peers in mixed-version fleets. Adds peer-side /api/system/pilot-tunnels centralCallback diag block, bounded counter metrics for bootstrap and dial events. SENCHO_PRIMARY_URL preflight warning when unset on a central with mesh-enabled proxy nodes. Tested with unit suites for the validation chain, registry, manager, and both dialers; integration tests for bootstrap E2E (asserts protocol-role invariant), api_token rotation, instance id change, version skew, and pilot-mode regression. * fix(mesh): green CI on the symmetric callback branch Two independent CI failures, both surgical: 1. Backend tests (11 fails): four mesh test files called setupTestDb in beforeEach. setupTestDb does not reset the DatabaseService singleton, so the per-test afterEach rm of the previous tmpdir left the singleton connection pointing at a deleted file. The next beforeEach's line-55 write threw SQLITE_READONLY_DBMOVED on Linux. Windows file-lock semantics hid this locally. Hoist setupTestDb / cleanupTestDb to file-scope beforeAll / afterAll; per-test state resets stay in beforeEach. Matches the convention in the eight mesh test files that already pass. 2. CodeQL (4 high alerts): js/insufficient-password-hash flagged sha256(api_token) at four sites. The api_token is a 256-bit opaque bearer (sen_sk_-prefixed), not a human password; sha256 is the correct fingerprint primitive for binding the mesh_tunnel JWT to a specific token. Add the two production files plus the two test files that mint the fingerprint to the existing path-scoped query-filter for that rule. * fix(mesh): drop unused afterEach import and revert dead codeql config ESLint flagged afterEach as unused in mesh-central-registry.test.ts:1 after the previous commit hoisted setup/teardown to file-scope beforeAll/afterAll. Remove from the vitest import line. Revert the codeql-config.yml additions from the previous commit. The paths: sub-key under query-filters > exclude is not a documented CodeQL feature and silently no-ops. The four js/insufficient-password-hash alerts on api_token fingerprinting are tracked as dismissed false positives in the GitHub Security tab rather than via dead config.	2026-05-16 14:58:19 -04:00
Anso	a38a3e0226	feat(mesh): route mesh traffic over Distributed API remotes (#1048) ... * refactor(mesh): extract shared TCP stream switchboard Pull the `tcp_open` / `tcp_open_ack` / `tcp_open_reverse` / `tcp_close` + `TcpData` handling out of the pilot agent into a reusable module so a second caller (the upcoming proxy-mode WS handler) can run the same frame parser, stream allocator, idle timers, and Compose-label resolver. One parser, two callers, zero drift on a security-sensitive protocol surface. The pilot agent keeps its public `openMeshTcpStream` API and delegates to a per-connection switchboard constructed in `connect()` and torn down in `cleanupAfterDisconnect`. Existing reverse-stream and resolver tests are rewritten to exercise the shared module directly. * feat(mesh): route mesh traffic over Distributed API remotes Sencho Mesh now works against remotes in Distributed API mode in addition to Pilot Agent mode. Central opens a short-lived WebSocket tunnel to the remote's new `/api/mesh/proxy-tunnel` endpoint on demand and tears it down after 5 minutes of idle (configurable via `SENCHO_MESH_PROXY_TUNNEL_IDLE_MS`). Mesh dispatch is mode-agnostic at the routing layer; `PilotTunnelManager.ensureBridge` resolves an existing tunnel or asks the new `MeshProxyTunnelDialer` to dial. The Routing tab badges now use a `reachableMode` classifier: `★ Local` for local, `pilot offline` red badge for a pilot with a down tunnel, and a new `unreachable` red badge surfacing the specific reason (missing token, scope not full-admin, TLS failure, remote does not support proxy mesh). Distributed API remotes with valid credentials show no negative badge; the tunnel opens on first dial. Auth: the proxy-tunnel WS upgrade requires an `Authorization: Bearer` API token with the `full-admin` scope. Lower-scoped tokens are rejected at upgrade time so a leaked read-only token cannot reach the mesh data plane. Bidirectional: the proxy-mode WS handler registers itself as the local `MeshService` reverse dialer (compare-and-swap), so meshed containers on a Distributed API remote can dial cross-node aliases via `tcp_open_reverse` over the same tunnel. Cross-node relays (`PilotTunnelBridge.acceptReverseRelay`) await `ensureBridge` so proxy-to-proxy mesh works without standing tunnels. * docs(mesh): describe Distributed API mesh and unreachable troubleshooting Refresh the user-facing mesh documentation to reflect that mesh works over both Pilot Agent and Distributed API remotes. Adds a Troubleshooting accordion entry for the new `unreachable` Routing tab badge so operators can match a tooltip reason ("api token rejected (scope must be full-admin)", "remote does not support proxy mesh", "TLS handshake failed", "api_url not set", "api token missing") to a concrete fix. * refactor(mesh): apply /simplify review findings Five cleanups surfaced by the post-implementation code review pass. No behaviour change for fleets running on `main`; only internal structure improves. - **Deterministic container IP shared.** Extract the compose-default network preference (`pickContainerIp`) and the conventional-name fast path (`lookupContainerIp`) into `backend/src/mesh/containerLookup.ts`. Both `MeshService.resolveContainerIp` (existing same-node fast path) and the switchboard's `resolveByComposeLabels` (proxy/pilot inbound dial) now use the same logic. Earlier the switchboard helper grabbed the first `Object.values(Networks)` IP, which could flip across daemon versions on multi-network containers; fixed. - **WS URL upgrade extracted.** New `backend/src/utils/wsUrl.ts` exposes `httpUrlToWs(baseUrl)` that maps `http://` to `ws://` and `https://` to `wss://`. Used in both `pilot/agent.ts` and the proxy-tunnel dialer. The previous inline `replace(/^http/, 'ws')` silently downgraded `https://` to `ws://` (cleartext). - **`computeReachable` takes the pre-fetched node.** `getStatus` already iterated `db.getNodes()`; the helper previously re-queried by id per node (N+1 reads). Pass the row in. - **Reachable-reason ternary -> const map.** Replace the nested ternary in `MeshService.computeReachable` with a `Record<DialFailureCode, string>` lookup keyed on the dialer's failure code. - **Activity-type ternary -> const map.** Same flattening inside `MeshProxyTunnelDialer.logActivity`. All mesh tests pass (85/85). Full backend suite green (2126/2126). * fix(mesh): cache proxy dial failures and contain WS handshake errors `ensureBridge` now consults the recent-failure cache before dialing so a continuous mesh workload against a misconfigured proxy-mode remote does not produce one upgrade attempt per cross-node TCP open. `recordFailure` emits at most one activity-log entry per cache window per (nodeId, code) so a connect-loop on a single bad remote cannot flush the ring buffer. Failure messages run through `redactSensitiveText` before reaching the log so any embedded Bearer / JWT / inline-URL credentials are scrubbed. `stop()` clears the inflight map and `dial()` checks `this.stopped` between awaits so a shutdown does not leak a half-opened bridge. When `awaitOpen` rejects (401, 4403, TLS), the dialer attaches a noop 'error' listener before calling `ws.close()`. Without it the ws library emits a tail 'error' on a still-CONNECTING socket that propagates as an unhandled exception. Surfaced by a live-network test against a real proxy-mode peer. Adds `mesh-proxy-tunnel-live.test.ts` (skipped unless MESH_AUDIT_URL and MESH_AUDIT_TOKEN_FILE env vars are set) covering both the happy path and the auth-rejected path against an actual remote Sencho. * feat(mesh): instrument proxy tunnel observability Adds three counters to `PilotMetrics`: - `proxy_bridges_total` (incremented on `registerProxyBridge` success) - `proxy_dials_failed` (every failed dial attempt, not deduped) - `proxy_idle_closes` (idle-sweep teardowns) All three surface automatically via `GET /api/system/pilot-tunnels` since the route returns the full `Counters` snapshot. Gates two pre-existing always-on `console.warn` calls in `tcpStreamSwitchboard.ts` (mid-stream socket errors and Docker resolve failures) behind `isDebugEnabled()`. Both fire on per-stream events and would otherwise violate the diagnostic-log safety rule under load. Adds `mesh-tcp-stream-switchboard.test.ts` covering the forward `tcp_open` path: real localhost dial, resolver errors (no_target, denied), per-tunnel cap saturation, frame-routing fall-through invariants, `tcp_close` socket teardown. Drops `MESH_CONNECT_TIMEOUT_MS` from the public surface; only the switchboard itself uses it. * fix(mesh): tighten Routing tab unreachable handling `RoutingNodeCard.tsx`: the **Add stack to mesh** button now disables when `reachableMode === 'unreachable'`, matching the existing TogglePill behavior. Without this, an operator on an unreachable node could open the opt-in sheet, confirm, and watch the redeploy proceed against a target whose mesh data plane will silently fail to route. Also drops the leftover `status.nodeId !== -1` guard on the pilot- offline badge. `MeshService.ts`: renames `isMeshReachable` to `isMeshConfigured` with the new predicate that returns true for proxy-mode remotes whose creds are valid (the tunnel is opened on demand). `getRouteDiagnostic` now distinguishes "routable" (configured) from "pilotLive" (live tunnel state, only meaningful for pilot mode); without the split, every idle proxy-mode route would report `tunnel down`. `MeshService.setReverseDialer` warns when the unconditional install path silently overwrites a non-null current dialer. By topology a Sencho is either pilot or central, so the branch flags a misconfigured deployment rather than an expected race. Drops the dead `export { ReverseTcpStreamHandle }` re-export from `pilot/agent.ts`; its only consumer imports straight from `mesh/tcpStreamSwitchboard.ts`. Fixes a stale doc comment in `MeshService.ts` that referenced the wrong source file. Adds `mesh-proxy-tunnel-handler.test.ts` covering the WS handler lifecycle: pilot-mode 404 rejection, post-upgrade reverse-dialer install, concurrent-upgrade 1013 rejection (single-tenant slot), and error-path teardown. Updates the troubleshooting accordion in the user docs to mention the disabled-state behavior. * fix(mesh): close ESLint and CodeQL findings on the proxy-tunnel diag logs Remove the unused `reverseDialer` local in `meshProxyTunnel.ts`; the closure target is `localDialer` above and this assignment was always dead. Strip line breaks inline on the three `MeshProxyTunnelDialer` diag log lines so CodeQL `js/log-injection` data flow recognises the sanitisation that `sanitizeForLog` already performs. No behaviour change; the diag logs render the same characters they do today.	2026-05-14 20:51:42 -04:00
Anso	567e524450	feat(mesh): bidirectional routing via tcp_open_reverse and central relay (#1003) ... * feat(mesh): bidirectional routing via tcp_open_reverse and central relay Phase B of the mesh redesign. Adds the reverse-direction protocol so a pilot's MeshForwarder can route cross-node traffic to central or to another pilot. Central relays pilot-to-pilot streams transparently; pilots keep their existing single outbound WS to central. Protocol additions (backend/src/pilot/protocol.ts): - TcpOpenReverseFrame { s, targetNodeId, stack, service, port } sent agent to primary. Reuses existing tcp_open_ack, tcp_close, and TcpData binary frames for the response and byte plane. - AGENT_REVERSE_ID_BASE = 0x40000001 splits the 32-bit id space so agent-allocated reverse stream ids never collide with primary- allocated forward ids on the same tunnel. - StreamIdAllocator now wraps to its configured start (not always 1) so an allocator parameterized with the agent base stays in the agent half across the wrap. Agent (backend/src/pilot/agent.ts): - New ReverseTcpStreamHandle exported class, EventEmitter facade matching PilotTunnelBridge.TcpStream's surface (write, end, destroy plus open, data, error, close events). - Public openMeshTcpStream(target) allocates a reverse id, sends tcp_open_reverse, returns the handle. - onTcpOpenAckReverse handles inbound ack, dispatches open or error+close to the matching handle. - TcpData and tcp_close binary/JSON paths route ids in the reverse range to reverseTcpStreams; existing primary-allocated paths unchanged. - streamCount, cleanupAfterDisconnect, onStreamIdle include reverse streams. The allocator is reset to a fresh base on disconnect so long-lived agents that reconnect many times do not drift up the id space. - startPilotAgent registers the agent as MeshService's reverse dialer via lazy import. Bridge (backend/src/services/PilotTunnelBridge.ts): - Two new StreamState kinds: reverse_local (target = central, dial Dockerode container IP) and reverse_relay (target = another pilot, open a forward TcpStream on the target pilot's bridge). - handleTcpOpenReverse validates the agent-id range and stream cap, then dispatches to acceptReverseLocal or acceptReverseRelay via lazy imports of MeshService, NodeRegistry, and PilotTunnelManager (avoids module cycles). - acceptReverseLocal calls MeshService.resolveContainerIp (now public), opens net.createConnection, splices bytes through the tunnel. Pre-connect error handler is removed inside connect to avoid double-firing with the mid-stream error path. - acceptReverseRelay opens a forward TcpStream on the target bridge, splices bytes between the two tunnels. - Existing tcp_close and TcpData handlers extended for the new state kinds. teardownStream extended. MeshService (backend/src/services/MeshService.ts): - resolveContainerIp made public so the bridge's local-target path can dial the same shape. - New reverseDialer field plus setReverseDialer setter. Pilot mode registers the agent at boot; central mode leaves it null. - New private dialMeshTcpStream dispatcher: pilot side uses the reverse dialer, central side uses PilotTunnelManager.getBridge. - openCrossNode refactored to call the dispatcher and use the active-stream record's id for activity logging. - New exported MeshTcpStreamLike interface that both PilotTunnelBridge.TcpStream and ReverseTcpStreamHandle satisfy structurally; ReverseMeshDialer is the contract for the setter. Tests (3 files, 16 new cases, 172 total pass): - pilot-protocol-tcp.test.ts: tcp_open_reverse round-trip; agent base invariant. - pilot-agent-reverse-stream.test.ts: openMeshTcpStream allocation, ws-not-open, frame shape, write encoding, end emits tcp_close, ack-success, ack-failure, low-id ack ignored, inbound TcpData routing. - pilot-bridge-reverse.test.ts: id-range validation, no-target failure, successful local dial against a real upstream server. Together with PR #1000 (Phase A) and PR #1001 (deps), Phase B completes the central-pilot-pilot mesh routing matrix. Phase C (mesh over proxy-mode remotes) is the planned follow-up. * test(pilot): drop unused encodeJsonFrame import Lint failed on pilot-agent-reverse-stream.test.ts after the test changed from constructing tcp_open_reverse frames inline to driving the agent's frame-dispatch path with synthetic objects. The import is no longer referenced; ESLint's no-unused-vars rule rejected it.	2026-05-08 16:21:34 -04:00
Anso	f5a52e44dc	refactor(pilot): narrow Mesh handle, prune dead code, add replay-route test (#982) ... * refactor(pilot): narrow PilotTunnelManager.getBridge to MeshTunnelHandle The manager handed out the entire PilotTunnelBridge to its only consumer, MeshService, which let any current or future caller reach into transport internals: loopback URL, per-stream maps, the close API, the underscored _writeTcpData / _closeTcpStream, the diagnostic helpers. None of that was load-bearing for Mesh. Introduce a MeshTunnelHandle interface alongside TcpStream in PilotTunnelBridge.ts that exposes only the two methods Mesh actually calls (openTcpStream and getBufferedAmount), have PilotTunnelBridge declare implements MeshTunnelHandle, and change getBridge to return the interface. MeshService continues to compile unchanged because its existing call sites only touch the narrowed surface. A future alternative transport (a stub for tests, a different routing strategy) now has a one-method-and-a-getter contract to satisfy instead of the full bridge. * refactor(pilot): drop unused tunnel manager and bridge surface Three public methods predating the hardening pass had zero callers across the entire codebase (verified via grep across backend, frontend, e2e): - PilotTunnelManager.touch(nodeId): never invoked. The pilot_last_seen timestamp is updated by the manager itself on registerTunnel and by the persistence layer on heartbeat events. - PilotTunnelManager.listActive(): never invoked. The metrics endpoint added in PR #979 returns its own per-node breakdown via getMetricsSnapshot, which is the canonical observability surface. - PilotTunnelBridge.listTcpStreams() and the supporting TcpStreamSummary interface: never invoked. The Mesh diagnostics sheet that would have consumed it is not wired and would use getMetricsSnapshot if/when it ships. Removing them tightens the public surface and prevents accidental new dependencies on speculative future-proofing. * test(pilot): cover enrollment replay rejection at the route layer The DB-level test in pilot-enrollment.test.ts already verifies that consumePilotEnrollment is one-shot. That guards the persistence layer but not the route handler: a refactor of handlePilotTunnel that swaps the order of consume vs upgrade, or that grants the WebSocket upgrade before checking the consume result, would silently break the security invariant while DB-layer tests stay green. Drive handlePilotTunnel directly with a stub IncomingMessage and Duplex socket. Six cases: - Already-consumed enrollment row -> 401. - Token whose hash matches no row -> 401. - Row whose expires_at has passed -> 401. - Missing Authorization header -> 401. - JWT signed with a wrong secret -> 401. - pilot_tunnel JWT for an unknown node -> 404. The stub captures HTTP/1.1 status writes so the test asserts the exact rejection lands on the wire, not just that the function returned without throwing. * docs(debug): warn future committers off per-frame isDebugEnabled calls Code-review feedback on PR #979 noted that isDebugEnabled is fine in the cadences it has today (per-tunnel, per-request, error paths) but is fragile against a future commit that drops it into a per-frame WebSocket loop. The function does a try/catch + Node require cache lookup + a method call into DatabaseService on every invocation; acceptable at hundreds of calls per second, expensive at thousands. Add a comment block above the function spelling out the acceptable and unacceptable cadences and the snapshot-outside-the-loop mitigation, so the next person to add a diag log there sees the constraint. * fix(pilot): address PR A code-review findings Code review on this branch surfaced four items: - Em-dash directive (CLAUDE.md Directive 18) violations in four comment sites; replaced with colons or restructured. - debug.ts perf comment claimed try/catch frame setup as the cost driver. V8 inlines those; the load-bearing cost is the require lookup and singleton dispatch. Reword. - Test file header described coverage as 'replay rejection at the route layer' but the file now also covers missing-header, wrong-secret, expired-row, never-stored, and unknown-node rejection paths. Widen the JSDoc and the describe label to match. - Stub-cast comment in the replay test now explicitly lists the IncomingMessage and Duplex surface the stub satisfies, so a future commit that grows handlePilotTunnel's surface (rate limiting, socket options) updates both stubs instead of silently no-opping against them. No behavior change.	2026-05-07 23:45:16 -04:00
Anso	3d9489648e	fix(pilot): harden outbound reverse-tunnel against resource exhaustion (#979) ... * fix(pilot): cap tunnel frame size, concurrent streams, and stream idle time Pilot tunnels carry every HTTP, WS, and Mesh-TCP byte for a remote node through a single multiplexed WebSocket. A buggy or compromised peer that sent oversized frames, opened streams in a tight loop, or left streams parked indefinitely could exhaust gateway memory. Three protocol-level limits applied symmetrically on both ends: - MAX_FRAME_SIZE_BYTES (8 MB): set as the ws maxPayload on the gateway-side WebSocketServer and the agent-side WebSocket client, plus a defense-in-depth length check in decodeBinaryFrame and decodeJsonFrame. - MAX_STREAMS_PER_TUNNEL (1024): bridge refuses new loopback HTTP / upgrade / TCP allocations with 503 once at the cap; the agent rejects new incoming http_req / ws_open / tcp_open with the appropriate error frame. - STREAM_IDLE_TIMEOUT_MS (10 min): every stream gets a per-stream timer refreshed on each inbound or outbound activity. Expiry tears down the local half and notifies the peer. The constants are colocated in pilot/protocol.ts so any future agent build picks them up via the protocol module. * fix(pilot): rate-limit pilot enrollment endpoints to 10 per minute Pilot enrollment mints a JWT and writes a pilot_enrollments row, both privileged operations that should not share the global API budget (200/min). Add a dedicated express-rate-limit instance keyed by user or IP at 10/min in production (100/min in dev so the local enrollment test loop is not throttled). The limiter is applied directly on POST /api/nodes/:id/pilot/enroll. On POST /api/nodes the limiter's skip function reads the parsed body and exempts proxy-mode creates; only requests that resolve to mode=pilot_agent count against the enrollment budget. * fix(pilot): cap concurrent pilot tunnels system-wide The PilotTunnelManager held an unbounded Map of bridges. A reconnect storm, runaway enrollment, or operator misconfiguration could grow the map without limit, even though every tunnel still carries a valid JWT. Cap at 256 concurrent tunnels per primary instance: - Soft warning at 128 (logged once per crossing). - Hard refusal at 256: registerTunnel throws PilotTunnelCapacityError and the upgrade handler closes the WebSocket with 1013 (Try Again Later) so the agent backs off rather than tight-looping. A node that already has a registered tunnel does not consume a new slot when it reconnects; the existing bridge is closed first. * fix(pilot): release backpressure as soon as the tunnel buffer drains Previously the bridge paused HTTP request bodies when the tunnel WebSocket's bufferedAmount climbed above 4 MB but never resumed them explicitly; the next data event re-checked the threshold, and TCP streams only saw a 'drain' fan-out on the 30 s ping cycle. Slow-consumer peers held buffered bytes for tens of seconds longer than needed. Replace both with an on-demand drain check: when at least one stream is paused, sample bufferedAmount every 100 ms; once it drops below the high-water mark, resume every paused request and emit 'drain' to every accepted TCP stream, then stop the timer. Dormant when no stream is paused, so steady-state cost is zero. * fix(pilot): trust internal CAs via SENCHO_PILOT_CA_FILE Self-hosted deployments often terminate TLS with an internal CA. The only previous escape hatch was NODE_TLS_REJECT_UNAUTHORIZED=0, which disables verification across every outbound connection in the agent process and is the wrong shape of fix. Add SENCHO_PILOT_CA_FILE: when set, the agent reads the file as a PEM bundle and passes it to ws as the `ca` option for the tunnel WebSocket. rejectUnauthorized remains true. Failure to read the file exits with a clear error so the operator does not silently fall back to the default trust store. There is no flag to disable TLS verification entirely; that would defeat the credential trust model. * fix(pilot): gate frame-rate-adjacent logs behind developer_mode The agent's malformed-frame warning fires per inbound frame and could flood logs under attack or against a buggy primary. The bridge swallowed parse errors silently, leaving operators blind to protocol drift, and the manager had no signal at all on tunnel registration. Route per-frame and per-tunnel diagnostic logs through the existing isDebugEnabled() helper (same pattern used by middleware/auth, RBAC, and websocket/logs). Production stays quiet by default; the toggle is the existing developer_mode setting in global_settings — no new env var, no new dependency. * feat(pilot): expose per-tunnel metrics via /api/system/pilot-tunnels Operator support had no signal beyond raw logs to answer "is this tunnel flapping?" or "is one bad node hiding behind the aggregate?". Add an in-process counter set covering tunnels_total, tunnels_replaced, tunnels_rejected_capacity, enroll_acks, frame_decode_errors, plus a per-node array carrying the connectedAt and bufferedAmount so a single tunnel sitting on a stuck buffer remains visible. Counters live in services/PilotMetrics.ts; the gateway has no shared in-process metrics facility today, so this is a per-feature pattern documented as such for future consolidation. Counters are strictly process-local — no telemetry, no export, no phone-home — consistent with Sencho's privacy posture. Surfaced read-only via GET /api/system/pilot-tunnels behind admin auth, mirroring the existing cache-stats endpoint. * fix(pilot): tighten reconnect backoff and log handling Three small follow-ups to the hardening pass: - Sanitize the primary URL on the agent's connect log so a malicious SENCHO_PRIMARY_URL with embedded control characters cannot inject fake log lines. - Move the reconnect-backoff reset from the 'open' callback to the 'hello' frame handler. A peer that always rejects the handshake (incompatible version, consumed enrollment token) used to reset the backoff on every TCP-level connect and tight-loop reconnects; now the reset waits for a clean protocol round-trip. - Document the StreamIdAllocator wrap behavior so future readers can see the relationship between the 2^31 wrap and the MAX_STREAMS_PER_TUNNEL cap without re-deriving it. * test(pilot): add coverage for enrollment, rate limiter, and tunnel caps Adds two test files exercising the hardening pass surfaces that were previously uncovered: - pilot-enrollment.test.ts: end-to-end through POST /api/nodes (pilot mode), POST /api/nodes/:id/pilot/enroll, the SHA256 token-hash persistence, replay protection, expired enrollments, and the rate-limit header wiring (enrollment limiter on pilot paths, global limiter elsewhere). - pilot-bridge-limits.test.ts: oversize-frame rejection at both binary and JSON decoders, and the per-tunnel concurrent stream cap as observed via openTcpStream returning null and the loopback HTTP server returning 503 once the bridge is at capacity. Bridge cap test fills the slot map with TCP stream handles rather than real HTTP requests so the OS socket pool stays out of the picture. * test(pilot): cover manager metrics snapshot and capacity-error shape Adds direct coverage for: - PilotTunnelCapacityError exposing the limit so the upgrade handler can format a useful close-frame reason. - PilotMetrics.snapshot returning a defensive copy (callers must not mutate the live counter set). - getMetricsSnapshot returning the open count, per-node breakdown, and counter set in a stable shape, with tunnels_total bumping on a successful registerTunnel. The manager test seeds a real pilot-mode node row first because updateNode throws on missing rows; this models the production path where a node is created before its enrollment is consumed. * docs(pilot): document tunnel limits, custom CA, and new failure modes Refresh the Pilot Agent feature doc to cover the hardening surfaces: - New 'Self-signed primary TLS certs' section walking through the SENCHO_PILOT_CA_FILE env var with an example docker run command. - New 'Resource limits' section listing the per-tunnel and system-wide ceilings so operators know what the wire enforces. - Four new troubleshooting entries: WebSocket close 1013 (system cap), loopback 503 (stream cap), close 1002 with protocol error (frame size / malformed), and HTTP 429 on enrollment. Also expand .env.example with a new pilot-agent block covering SENCHO_MODE, SENCHO_PRIMARY_URL, SENCHO_ENROLL_TOKEN, and SENCHO_PILOT_CA_FILE so operators do not have to read code or the feature doc to discover the agent-side config surface. * fix(pilot): address code-review findings on the hardening pass Critical: - Bridge: TCP backpressure now starts the drain timer and tracks streams awaiting drain, so a TcpStream caller waiting on 'drain' no longer hangs when no HTTP request is in flight. - Bridge: every direct streams.delete call now goes through the removeStream helper so per-stream idle timers are cleared consistently and pausedReqs / tcpAwaitingDrain stay aligned. - Bridge close(): resume any paused IncomingMessage before clearing the map so a parser does not stay stuck across teardown. High: - Protocol: decodeJsonFrame now compares Buffer.byteLength(raw, 'utf8') against the cap; raw.length (UTF-16 code units) let multi-byte payloads sneak ~3x the byte budget through. - Agent: the optional CA bundle is read once at construction and cached on the instance, so a missing or rotated SENCHO_PILOT_CA_FILE no longer process-exits on every reconnect attempt. Medium / Low: - Bridge limits test: assert mockWs.sent.length to confirm slot allocations actually serialized a tcp_open frame and that a rejected (cap+1) attempt did not consume a stream id. - Manager: defer the tunnels_replaced increment until after the cap check passes, so a rejected reconnect does not double-count as both replacement and capacity rejection. - Manager: clarify the protocol comment about MAX_FRAME_SIZE_BYTES enforcement layering (ws maxPayload is authoritative; decoder check is for tests and defense-in-depth). - Limiter: defensive `if (!req.body) return false` so a future refactor that delays body parsing cannot silently skip the enrollment limiter. Deferred (documented as follow-ups, not in this branch): - M2: narrowing public surface of PilotTunnelManager.getBridge to a MeshTunnelHandle interface (cross-cutting refactor). - H4: end-to-end replay test driving the upgrade handler twice with the same enrollment JWT (needs WS test harness).	2026-05-07 21:28:35 -04:00
Anso	6893ece898	feat(pilot): add tcp tunnel frames + mesh sidecar package (#857) ... Lays the dormant data-plane foundation for Sencho Mesh. The pilot tunnel gains TCP forwarding frames (tcp_open / tcp_open_ack / tcp_close JSON plus a 0x04 TcpData binary type) and a TcpStream surface on the bridge so a future MeshService can ride the existing WSS tunnel for cross-node container traffic. The agent rejects every tcp_open with mesh_not_enabled until a follow-up PR wires the Dockerode resolver gated by a mesh_stacks opt-in table; ships dormant. A new top-level mesh-sidecar/ package provides the per-node container that will host the L4 forwarder + control WS in production. Built as a small Node 22 alpine image and published in lockstep with the main sencho image via a parallel docker-publish workflow job. Tests cover protocol roundtrips on both packages and the sidecar forwarder end-to-end including resolve, splice, close, and stats.	2026-05-01 00:28:18 -04:00
Anso	8e7a567f69	feat: pilot agent outbound-mode for remote nodes (#667) ... * feat: pilot agent outbound-mode for remote nodes Adds a second mode for managing remote nodes: the agent dials an outbound WebSocket tunnel to the primary, so the remote host no longer needs an inbound port, a reachable URL, or its own TLS certificate. Works behind NAT, residential routers, and corporate firewalls. The primary multiplexes HTTP and WebSocket requests over a single tunnel via a hybrid JSON + binary frame protocol, bridged through a per-tunnel loopback server so existing proxy and upgrade handlers route pilot-mode nodes identically to proxy-mode ones. Enrollment uses a single-use 15-minute pilot_enroll JWT exchanged for a long-lived pilot_tunnel credential on first connect. Proxy mode continues to work unchanged and both modes are supported side-by-side. * test(e2e): switch to proxy mode before asserting api_url field Remote nodes default to Pilot Agent mode, which hides the api_url input. The SSRF-validation tests need proxy mode, so the helper now selects Distributed API Proxy after picking Remote type before asserting the field is visible. * fix(e2e): wire Combobox id prop so node-mode selector resolves The Combobox trigger button had no id, leaving its Label orphaned and making getByRole name-based lookups fail. Adding id to the primitive, passing id="node-mode" from NodeManager, and updating the E2E helper to use #node-mode fixes both the a11y regression and the CI timeout.	2026-04-17 20:31:43 -04:00