3 Commits

Author	SHA1	Message	Date
Anso	23bbee4f45	feat(mesh): replace host-mode with shared sencho_mesh Docker network (#1009) ... * feat(mesh): replace host-mode with shared sencho_mesh Docker network Phase D of the mesh redesign: drop the operator's `network_mode: host` requirement and the `host-gateway` extra_hosts pattern that did not work on cloud iptables-restrictive distros (OCI, etc.) or Docker Desktop. Each Sencho creates a shared `sencho_mesh` Docker bridge network on boot (default subnet 172.30.0.0/24, override via SENCHO_MESH_SUBNET), pins itself at `<network>+2`, and attaches every meshed user service to the same bridge. Compose overrides now emit IP-based `extra_hosts` plus a top-level `networks` block declaring `sencho_mesh` external. Override delivery: central renders for local stacks; for remote stacks it sends the fleet alias list to the remote's new `PUT /api/mesh/local- override/:stackName` endpoint, which renders against the remote's OWN local senchoIp and writes under its OWN DATA_DIR. Each node may use a different subnet without coordination beyond the env var. Opt-in / opt-out now trigger an automatic redeploy of the affected stack via the existing deploy code path (local: ComposeService; remote: HTTP POST through proxyFetch). The frontend opt-in sheet shows a confirmation modal (ConfirmModal) before the mutation. Failed redeploys emit both a mesh activity event and a durable audit-log row. Hardening: - Reserve port 1852 at opt-in (prevents user containers from racing the Sencho API listener). - ensureMeshNetwork refuses to continue if `sencho_mesh` exists with a mismatched subnet rather than silently routing to the wrong IP. - Idempotent network connect/disconnect helpers in DockerController. - optInStack rolls back the DB row if the just-inserted stack's override push fails (no half-states surviving across calls). - regenerateOverridesForNode runs in parallel and skips the just- pushed stack on opt-in. Operator template: drop `network_mode: host`, restore `ports: ["1852:1852"]`. Mesh now works identically on Linux LAN, OCI, and Docker Desktop without firewall changes. Docs: rewrite docs/features/sencho-mesh.mdx around the shared bridge network, document SENCHO_MESH_SUBNET, surface the host-network-service opt-in restriction, and cross-link with the Pilot Agent docs. BREAKING CHANGE: the operator's `docker-compose.yml` no longer uses `network_mode: host`. After upgrading, redeploy any meshed stacks once so they pick up the new IP-based override and join `sencho_mesh`. * fix(mesh): wrap stackName with path.basename in local-override fs ops CodeQL flagged js/path-injection on the new applyLocalOverride and removeLocalOverride methods because they are publicly reachable and its data-flow model does not recognize isValidStackName / isPathWithinBase as sanitizers. The validation IS sufficient (the allowlist regex blocks path separators, the path-prefix check blocks escape), but path.basename is a model CodeQL recognizes and is purely defensive: for any input that already passes isValidStackName, basename is the identity.	2026-05-09 00:11:09 -04:00
Anso	3d9489648e	fix(pilot): harden outbound reverse-tunnel against resource exhaustion (#979) ... * fix(pilot): cap tunnel frame size, concurrent streams, and stream idle time Pilot tunnels carry every HTTP, WS, and Mesh-TCP byte for a remote node through a single multiplexed WebSocket. A buggy or compromised peer that sent oversized frames, opened streams in a tight loop, or left streams parked indefinitely could exhaust gateway memory. Three protocol-level limits applied symmetrically on both ends: - MAX_FRAME_SIZE_BYTES (8 MB): set as the ws maxPayload on the gateway-side WebSocketServer and the agent-side WebSocket client, plus a defense-in-depth length check in decodeBinaryFrame and decodeJsonFrame. - MAX_STREAMS_PER_TUNNEL (1024): bridge refuses new loopback HTTP / upgrade / TCP allocations with 503 once at the cap; the agent rejects new incoming http_req / ws_open / tcp_open with the appropriate error frame. - STREAM_IDLE_TIMEOUT_MS (10 min): every stream gets a per-stream timer refreshed on each inbound or outbound activity. Expiry tears down the local half and notifies the peer. The constants are colocated in pilot/protocol.ts so any future agent build picks them up via the protocol module. * fix(pilot): rate-limit pilot enrollment endpoints to 10 per minute Pilot enrollment mints a JWT and writes a pilot_enrollments row, both privileged operations that should not share the global API budget (200/min). Add a dedicated express-rate-limit instance keyed by user or IP at 10/min in production (100/min in dev so the local enrollment test loop is not throttled). The limiter is applied directly on POST /api/nodes/:id/pilot/enroll. On POST /api/nodes the limiter's skip function reads the parsed body and exempts proxy-mode creates; only requests that resolve to mode=pilot_agent count against the enrollment budget. * fix(pilot): cap concurrent pilot tunnels system-wide The PilotTunnelManager held an unbounded Map of bridges. A reconnect storm, runaway enrollment, or operator misconfiguration could grow the map without limit, even though every tunnel still carries a valid JWT. Cap at 256 concurrent tunnels per primary instance: - Soft warning at 128 (logged once per crossing). - Hard refusal at 256: registerTunnel throws PilotTunnelCapacityError and the upgrade handler closes the WebSocket with 1013 (Try Again Later) so the agent backs off rather than tight-looping. A node that already has a registered tunnel does not consume a new slot when it reconnects; the existing bridge is closed first. * fix(pilot): release backpressure as soon as the tunnel buffer drains Previously the bridge paused HTTP request bodies when the tunnel WebSocket's bufferedAmount climbed above 4 MB but never resumed them explicitly; the next data event re-checked the threshold, and TCP streams only saw a 'drain' fan-out on the 30 s ping cycle. Slow-consumer peers held buffered bytes for tens of seconds longer than needed. Replace both with an on-demand drain check: when at least one stream is paused, sample bufferedAmount every 100 ms; once it drops below the high-water mark, resume every paused request and emit 'drain' to every accepted TCP stream, then stop the timer. Dormant when no stream is paused, so steady-state cost is zero. * fix(pilot): trust internal CAs via SENCHO_PILOT_CA_FILE Self-hosted deployments often terminate TLS with an internal CA. The only previous escape hatch was NODE_TLS_REJECT_UNAUTHORIZED=0, which disables verification across every outbound connection in the agent process and is the wrong shape of fix. Add SENCHO_PILOT_CA_FILE: when set, the agent reads the file as a PEM bundle and passes it to ws as the `ca` option for the tunnel WebSocket. rejectUnauthorized remains true. Failure to read the file exits with a clear error so the operator does not silently fall back to the default trust store. There is no flag to disable TLS verification entirely; that would defeat the credential trust model. * fix(pilot): gate frame-rate-adjacent logs behind developer_mode The agent's malformed-frame warning fires per inbound frame and could flood logs under attack or against a buggy primary. The bridge swallowed parse errors silently, leaving operators blind to protocol drift, and the manager had no signal at all on tunnel registration. Route per-frame and per-tunnel diagnostic logs through the existing isDebugEnabled() helper (same pattern used by middleware/auth, RBAC, and websocket/logs). Production stays quiet by default; the toggle is the existing developer_mode setting in global_settings — no new env var, no new dependency. * feat(pilot): expose per-tunnel metrics via /api/system/pilot-tunnels Operator support had no signal beyond raw logs to answer "is this tunnel flapping?" or "is one bad node hiding behind the aggregate?". Add an in-process counter set covering tunnels_total, tunnels_replaced, tunnels_rejected_capacity, enroll_acks, frame_decode_errors, plus a per-node array carrying the connectedAt and bufferedAmount so a single tunnel sitting on a stuck buffer remains visible. Counters live in services/PilotMetrics.ts; the gateway has no shared in-process metrics facility today, so this is a per-feature pattern documented as such for future consolidation. Counters are strictly process-local — no telemetry, no export, no phone-home — consistent with Sencho's privacy posture. Surfaced read-only via GET /api/system/pilot-tunnels behind admin auth, mirroring the existing cache-stats endpoint. * fix(pilot): tighten reconnect backoff and log handling Three small follow-ups to the hardening pass: - Sanitize the primary URL on the agent's connect log so a malicious SENCHO_PRIMARY_URL with embedded control characters cannot inject fake log lines. - Move the reconnect-backoff reset from the 'open' callback to the 'hello' frame handler. A peer that always rejects the handshake (incompatible version, consumed enrollment token) used to reset the backoff on every TCP-level connect and tight-loop reconnects; now the reset waits for a clean protocol round-trip. - Document the StreamIdAllocator wrap behavior so future readers can see the relationship between the 2^31 wrap and the MAX_STREAMS_PER_TUNNEL cap without re-deriving it. * test(pilot): add coverage for enrollment, rate limiter, and tunnel caps Adds two test files exercising the hardening pass surfaces that were previously uncovered: - pilot-enrollment.test.ts: end-to-end through POST /api/nodes (pilot mode), POST /api/nodes/:id/pilot/enroll, the SHA256 token-hash persistence, replay protection, expired enrollments, and the rate-limit header wiring (enrollment limiter on pilot paths, global limiter elsewhere). - pilot-bridge-limits.test.ts: oversize-frame rejection at both binary and JSON decoders, and the per-tunnel concurrent stream cap as observed via openTcpStream returning null and the loopback HTTP server returning 503 once the bridge is at capacity. Bridge cap test fills the slot map with TCP stream handles rather than real HTTP requests so the OS socket pool stays out of the picture. * test(pilot): cover manager metrics snapshot and capacity-error shape Adds direct coverage for: - PilotTunnelCapacityError exposing the limit so the upgrade handler can format a useful close-frame reason. - PilotMetrics.snapshot returning a defensive copy (callers must not mutate the live counter set). - getMetricsSnapshot returning the open count, per-node breakdown, and counter set in a stable shape, with tunnels_total bumping on a successful registerTunnel. The manager test seeds a real pilot-mode node row first because updateNode throws on missing rows; this models the production path where a node is created before its enrollment is consumed. * docs(pilot): document tunnel limits, custom CA, and new failure modes Refresh the Pilot Agent feature doc to cover the hardening surfaces: - New 'Self-signed primary TLS certs' section walking through the SENCHO_PILOT_CA_FILE env var with an example docker run command. - New 'Resource limits' section listing the per-tunnel and system-wide ceilings so operators know what the wire enforces. - Four new troubleshooting entries: WebSocket close 1013 (system cap), loopback 503 (stream cap), close 1002 with protocol error (frame size / malformed), and HTTP 429 on enrollment. Also expand .env.example with a new pilot-agent block covering SENCHO_MODE, SENCHO_PRIMARY_URL, SENCHO_ENROLL_TOKEN, and SENCHO_PILOT_CA_FILE so operators do not have to read code or the feature doc to discover the agent-side config surface. * fix(pilot): address code-review findings on the hardening pass Critical: - Bridge: TCP backpressure now starts the drain timer and tracks streams awaiting drain, so a TcpStream caller waiting on 'drain' no longer hangs when no HTTP request is in flight. - Bridge: every direct streams.delete call now goes through the removeStream helper so per-stream idle timers are cleared consistently and pausedReqs / tcpAwaitingDrain stay aligned. - Bridge close(): resume any paused IncomingMessage before clearing the map so a parser does not stay stuck across teardown. High: - Protocol: decodeJsonFrame now compares Buffer.byteLength(raw, 'utf8') against the cap; raw.length (UTF-16 code units) let multi-byte payloads sneak ~3x the byte budget through. - Agent: the optional CA bundle is read once at construction and cached on the instance, so a missing or rotated SENCHO_PILOT_CA_FILE no longer process-exits on every reconnect attempt. Medium / Low: - Bridge limits test: assert mockWs.sent.length to confirm slot allocations actually serialized a tcp_open frame and that a rejected (cap+1) attempt did not consume a stream id. - Manager: defer the tunnels_replaced increment until after the cap check passes, so a rejected reconnect does not double-count as both replacement and capacity rejection. - Manager: clarify the protocol comment about MAX_FRAME_SIZE_BYTES enforcement layering (ws maxPayload is authoritative; decoder check is for tests and defense-in-depth). - Limiter: defensive `if (!req.body) return false` so a future refactor that delays body parsing cannot silently skip the enrollment limiter. Deferred (documented as follow-ups, not in this branch): - M2: narrowing public surface of PilotTunnelManager.getBridge to a MeshTunnelHandle interface (cross-cutting refactor). - H4: end-to-end replay test driving the upgrade handler twice with the same enrollment JWT (needs WS test harness).	2026-05-07 21:28:35 -04:00
Anso	8e7a567f69	feat: pilot agent outbound-mode for remote nodes (#667) ... * feat: pilot agent outbound-mode for remote nodes Adds a second mode for managing remote nodes: the agent dials an outbound WebSocket tunnel to the primary, so the remote host no longer needs an inbound port, a reachable URL, or its own TLS certificate. Works behind NAT, residential routers, and corporate firewalls. The primary multiplexes HTTP and WebSocket requests over a single tunnel via a hybrid JSON + binary frame protocol, bridged through a per-tunnel loopback server so existing proxy and upgrade handlers route pilot-mode nodes identically to proxy-mode ones. Enrollment uses a single-use 15-minute pilot_enroll JWT exchanged for a long-lived pilot_tunnel credential on first connect. Proxy mode continues to work unchanged and both modes are supported side-by-side. * test(e2e): switch to proxy mode before asserting api_url field Remote nodes default to Pilot Agent mode, which hides the api_url input. The SSRF-validation tests need proxy mode, so the helper now selects Distributed API Proxy after picking Remote type before asserting the field is visible. * fix(e2e): wire Combobox id prop so node-mode selector resolves The Combobox trigger button had no id, leaving its Label orphaned and making getByRole name-based lookups fail. Adding id to the primitive, passing id="node-mode" from NodeManager, and updating the E2E helper to use #node-mode fixes both the a11y regression and the CI timeout.	2026-04-17 20:31:43 -04:00