428 Commits

Author	SHA1	Message	Date
Anso	9aaa8573c0	fix(mesh): gate Trigger 2 re-bootstrap on actual api_token change (#1076) ... The PUT /api/nodes/:id handler closed and re-dialed the mesh callback bridge on every save that included api_token in the body, even when the token was unchanged. The frontend always sends the full formData on Save, so renames and compose_dir edits against a mesh-enabled proxy remote produced a wasted closeBridge + ensureBridge round-trip and a spurious manager_rejected entry in the activity log. Gate the re-bootstrap on a real value diff against the persisted token. Adds an existing-node lookup (returns 404 on missing id, which the handler previously lacked) so the comparison has the pre-update value. Reorders the guards so resource-not-found beats payload validation. Adds one integration test for the same-token path; the existing three trigger-2 cases continue to assert close + ensure firing on a real rotation, no-token-in-payload, and mesh-disabled.	2026-05-17 01:48:02 -04:00
Anso	7e3e22138b	fix(mesh): name actual bridge kind in proxy-bridge rejection (#1075) ... * fix(mesh): distinguish proxy bridge from pilot tunnel in registerProxyBridge rejection When central's MeshProxyTunnelDialer dials a node whose slot is already held by a peer-initiated proxy bridge, the registration is correctly refused, but the activity-log message read "pilot tunnel already registered for node N; proxy bridge refused" regardless of which kind of bridge was actually in the slot. No pilot tunnel exists in proxy-mode remotes by definition, so the wording pointed operators at the wrong subsystem. Use the existing bridgeKinds index (already consulted correctly by the sister method replaceOrRegisterProxyBridge) to emit a kind-accurate message: "proxy bridge already registered for node N; concurrent dial refused" when the existing bridge is a peer-initiated proxy bridge, keeping the original "pilot tunnel" wording for the pilot-agent case so existing log filters and the sister test still match. Pure cosmetic; the rejection behavior is unchanged. New regression test asserts the discriminator in both directions. Also tightens the test-fixture beforeEach to clear bridgeKinds alongside bridges so future tests cannot read a stale kind. * chore: ignore local trailer directory in git repository * refactor(mesh): centralize bridge-kind type and conflict-message formatter Followup tidy on PilotTunnelManager after /simplify flagged two related items in the F-R-2 change: - Extract `BridgeKind = 'pilot' | 'proxy'` as a module-scoped type alias so the kind taxonomy lives in one place. `bridgeKinds` and `injectBridgeForTest` now consume it; the inline union literal is gone. - Extract `formatBridgeConflict(nodeId, existingKind)` as a private helper. Both `registerProxyBridge` and `replaceOrRegisterProxyBridge` call it from their "slot already held" rejection paths, removing the byte-identical "pilot tunnel already registered for node N; proxy bridge refused" string copy across the two sites. - Trim the 6-line WHAT comment above the rejection throw to a 2-line WHY so the call site stays scannable. No behavior change; the three relevant test suites stay green.	2026-05-17 01:16:21 -04:00
Anso	3a20105a33	fix(mesh): persist rotated handshake when peer bridge owns reverse-dialer slot (#1073) ... When the proxy-mode mesh peer already holds an open peer-initiated callback bridge to central, central's Trigger 2 dial (api_token rotation) arrives at the peer's `/api/mesh/proxy-tunnel` handler with a fresh `mesh_handshake` frame. The peer's `attachSwitchboard` refuses the new WS at the reverse-dialer CAS swap, but it was also closing the connection before its `'message'` listener had been attached, so the handshake frame went unread and the peer's cached callback JWT stayed at the pre-rotation fingerprint until a central restart. Restructures `attachSwitchboard` to attach the `'message'`, `'close'`, and `'error'` listeners synchronously after the switchboard is created, before the async MeshService import and the CAS swap run. On the CAS-fail branch, holds the WS open for up to FIRST_FRAME_WAIT_MS (30ms, well above localhost RTT and below operator perception) before sending the 1013 close, so an in-flight first frame lands in `onMessage` and the bootstrap material is persisted via `MeshCentralRegistry.upsert` regardless of bridge fate. The success path is untouched; reverseDialer install timing on accepted dials is unchanged. Adds a regression test that pre-installs a stub reverse dialer (simulating a peer-initiated bridge owning the slot), sends a `mesh_handshake` frame on the new WS, and asserts the registry receives the rotated material before the 1013 close fires.	2026-05-16 22:32:35 -04:00
Anso	0b4cf90bf1	fix(mesh): peer-initiated callback bridge installs reverse dialer (#1071) ... * fix(mesh): peer-initiated callback bridge installs reverse dialer The R1-A2 symmetric-dial work added a peer-side dialer that opens a callback WS to central at `/api/mesh/proxy-tunnel-from-peer`. The WS upgrade, JWT validation, and `mesh_centrals` persistence all worked, but the peer-side handler put the wrong object on its end of the pipe: a `PilotTunnelBridge` (the central-role object) rather than a `TcpStreamSwitchboard` with a registered `reverseDialer`. The design doc states: "Central retains PilotTunnelBridge ownership; peer retains TcpStreamSwitchboard + reverseDialer ownership." The central-initiated handler at `meshProxyTunnel.ts:115-163` honors this. The peer-initiated handler at `PeerToCentralMeshSessionDialer.attachBridge` did not. It created a `PilotTunnelBridge(0, ws)` and stored it in a private `currentSession` field. Result: `MeshService.reverseDialer` stayed null, `dialMeshTcpStream` fell through to `PilotTunnelManager.ensureBridge(target.nodeId)`, and `NodeRegistry. getProxyTarget` on the peer returned null because proxy-mode peers do not enroll their central. Cross-fleet dispatch from a peer container failed with `proxy-tunnel.open.fail nodeId=<central> reason=no_target` even though the callback bridge was up and healthy (`bridgeOpen: true`, `last_used_at` populated). Replace the peer-side `PilotTunnelBridge` with the same wiring the central-initiated handler uses: - `attachTcpStreamSwitchboard` with `resolveByComposeLabels` (allows inbound `tcp_open` from central if it ever uses the callback bridge for a reverse-direction dispatch; matches the central- initiated pipe's resolver). - A `SwitchboardReverseDialer` that delegates `openMeshTcpStream` to `switchboard.openReverseStream`. - `MeshService.setReverseDialer(localDialer, null)` with the same CAS guard the central-initiated handler uses, so a concurrent central-initiated tunnel does not get silently overwritten. - `ws.on('message', onMessage)` dispatches JSON / binary frames to the switchboard. - `ws.once('close'/'error', teardown)` clears the switchboard, the reverseDialer, and the `currentSession` reference idempotently. The `currentSession` field type changes from `PilotTunnelBridge | null` to `TcpStreamSwitchboard | null`. Callers in `MeshService.openCrossNode` only use the return value for truthiness; the cast is harmless. The public `hasSession()` signature is unchanged. A new `currentWs` field holds the WS reference so `resetForTest` can close the connection on teardown (the switchboard's own `cleanup` does not close its WS). Test update mirrors the production wiring: the existing "marks the row used on successful WS open" case now asserts the return value is a `TcpStreamSwitchboard`, that `MeshService.reverseDialer` is non-null after `ensureSession`, and explicitly closes the WS in the test's finally so `wss.close` can resolve. Pre-existing test logic that asserted `bridge instanceof PilotTunnelBridge` on the same path is removed in favor of the new switchboard assertion. Discovered during the v0.81.1 live verification: the peer-side `route.dispatch cross-node` event always paired with `proxy-tunnel.open.fail no_target` and `tunnel.fail`, even though `centralCallback.bridgeOpen: true` and `mesh_centrals.last_used_at` were populated. The callback path opened cleanly in isolation but the dispatch path never consumed it. This fix makes the dispatch path consume it. Tests: - cd backend && npx tsc --noEmit clean - cd backend && npx vitest run mesh 170/170 green - cd backend && npx vitest run 2259/2268 green; only failures are the pre-existing Windows EBUSY flake in filesystem-backup.test.ts (documented in prior handoff) * fix(mesh): drop unused imports and align reverse-dialer interface name CI lint failures on the previous push were two @typescript-eslint/no-unused-vars errors in the test file: 24:76 'vi' is defined but never used 31:5 'TcpStreamSwitchboardCtor' is assigned a value but only used as a type Both were collateral from a local-debugging simplification: vi.waitFor was swapped for a synchronous assertion when the test's wss.close hang was traced to an unclosed client WS; the toBeInstanceOf assertion that used the runtime binding was removed at the same time. Restore the instanceof assertion next to the existing not.toBeNull check so the runtime binding is in use and the test guards against future regressions where the wrong end-of-pipe object type is returned on the callback bridge (the bug this PR fixes). Drop the now-unused `vi` symbol from the vitest import. Also fold in a /simplify convergent finding: rename the local CallbackReverseDialer interface to SwitchboardReverseDialer to match the sibling declaration in meshProxyTunnel.ts:72. Same shape, same name; readers of either handler now see the same mental model. Cross-file extraction of the interface to tcpStreamSwitchboard.ts is the cleaner long-term shape but touches a file outside this PR's scope; filing as a follow-up. Considered-and-deferred findings from the /simplify pass (file as separate PRs to keep one-branch-one-concern): - extract shared attachSwitchboard helper (~40 duplicated lines between meshProxyTunnel.ts and PeerToCentralMeshSessionDialer.ts) - public TcpStreamSwitchboard.getWs accessor + drop currentWs field - public MeshService.hasReverseDialer for the test bracket-cast - shared close-reason constants (drift risk across the two handlers) Tests: - cd backend && npx tsc --noEmit clean - cd backend && npx eslint <changed> clean (lint_exit=0) - cd backend && npx vitest run mesh 170/170, dialer 8/8 green	2026-05-16 20:11:31 -04:00
Anso	54d84a7edb	fix(mesh): skip peer-side recovery in openCrossNode on central instances (#1068) ... The R1-A2 symmetric-callback work added a peer-side recovery branch to MeshService.openCrossNode: when reverseDialer is null, kick PeerToCentralMeshSessionDialer.ensureSession so the peer re-opens its callback WS to central before the cross-node dispatch. The guard `!this.reverseDialer` did not distinguish "I am a peer waiting for central to dial in" from "I am central, and the reverseDialer concept does not apply to me." On central, reverseDialer is always null (central is the bridge initiator side and never has another node dial into it), so central entered the recovery branch on every forward dispatch, found no session (central has no mesh_centrals row), and destroyed the inbound socket with route.resolve.fail forward-from-peer no_session. Gate the branch on `MeshCentralRegistry.getActive() !== null` so it only runs when this Sencho is acting as a proxy-mode peer that has actually been bootstrapped. Central falls straight through to dialMeshTcpStream. The peer cold-start scenario the comment originally described is still covered: a proxy-mode peer with a cached mesh_centrals row but no live WS to central will fire the recovery branch as before. Adds two regression tests: - central path (no mesh_centrals row, no reverseDialer) skips recovery and calls dialMeshTcpStream - proxy-peer path (mesh_centrals row + ensureSession returns null) still emits route.resolve.fail forward-from-peer no_session Existing tests in the "openCrossNode (BUG-4)" block already used a stub reverseDialer to bypass the recovery branch, which is why the central regression was not caught pre-release. The new tests deliberately leave reverseDialer null to exercise the gating.	2026-05-16 18:28:16 -04:00
Anso	cf618dd866	feat(mesh): symmetric WS dial for proxy-mode mesh peers (#1066) ... * chore(mesh): foundation for symmetric callback dial Adds the data-plane scaffolding that the symmetric callback dial fix builds on: - mesh_centrals table for peer-side bootstrap material - MeshCentralRegistry service (upsert/getActive/clear/markUsed/markRejected) - PilotTunnelManager kind discriminator and replaceOrRegisterProxyBridge - mesh_proxy_callback_bootstrap capability registration - MeshProxyTunnelDialer reason-tagged proxy-bridge-down events from a single tearDownBridge emission point - Reactive redial scheduler that skips idle and auth_failed reasons * feat(mesh): add reverse-direction activity log entries (closes R1-B) acceptReverseLocal now emits route.resolve.ok with direction=reverse on connect ack and route.resolve.fail with direction=reverse plus reason=container_not_found / connect_error pre-connect. Post-connect close/error stays silent. Reuses existing event types via the new details.direction discriminator so frontend filters are unaffected. * feat(mesh): add peer-to-central callback dial path (closes R1-A2) Closes the architectural gap where proxy-mode mesh peers could not re-establish their tunnel to central after any non-idle bridge teardown (idle close, network blip, central restart, peer reboot). Central remains the hub for the data plane; the change is purely about WS initiation. Symmetric WS initiation, asymmetric protocol roles. Central retains PilotTunnelBridge ownership; peer retains TcpStreamSwitchboard + reverseDialer ownership. Central bootstraps callback credentials over the first authenticated central-initiated mesh tunnel via a one-shot mesh_handshake JSON frame; peer persists the material in a new mesh_centrals SQLite table and dials central's new /api/mesh/proxy-tunnel-from-peer endpoint when local cross-node traffic needs a bridge and none is live. Mesh_tunnel JWT (HS256, signed with auth_jwt_secret) carries scope, audience, issuer (central instance id), peer api_token fingerprint, kid. Validation on inbound peer dial: algorithm pin, signature, scope, audience, instance, time bounds, node existence and mode, fingerprint match. Failures return HTTP 401 with a machine-readable reason; peer routes the response per a clear-vs-keep cache matrix. Triggers proactive bootstrap on mesh-enable and api_token rotation; central startup fans out to mesh-enabled proxy-mode nodes with mesh_stacks rows (throttled, fire-and-forget). Reactive redial on non-idle bridge loss. Capability-gated handshake send (mesh_proxy_callback_bootstrap) makes the upgrade path safe against older peers in mixed-version fleets. Adds peer-side /api/system/pilot-tunnels centralCallback diag block, bounded counter metrics for bootstrap and dial events. SENCHO_PRIMARY_URL preflight warning when unset on a central with mesh-enabled proxy nodes. Tested with unit suites for the validation chain, registry, manager, and both dialers; integration tests for bootstrap E2E (asserts protocol-role invariant), api_token rotation, instance id change, version skew, and pilot-mode regression. * fix(mesh): green CI on the symmetric callback branch Two independent CI failures, both surgical: 1. Backend tests (11 fails): four mesh test files called setupTestDb in beforeEach. setupTestDb does not reset the DatabaseService singleton, so the per-test afterEach rm of the previous tmpdir left the singleton connection pointing at a deleted file. The next beforeEach's line-55 write threw SQLITE_READONLY_DBMOVED on Linux. Windows file-lock semantics hid this locally. Hoist setupTestDb / cleanupTestDb to file-scope beforeAll / afterAll; per-test state resets stay in beforeEach. Matches the convention in the eight mesh test files that already pass. 2. CodeQL (4 high alerts): js/insufficient-password-hash flagged sha256(api_token) at four sites. The api_token is a 256-bit opaque bearer (sen_sk_-prefixed), not a human password; sha256 is the correct fingerprint primitive for binding the mesh_tunnel JWT to a specific token. Add the two production files plus the two test files that mint the fingerprint to the existing path-scoped query-filter for that rule. * fix(mesh): drop unused afterEach import and revert dead codeql config ESLint flagged afterEach as unused in mesh-central-registry.test.ts:1 after the previous commit hoisted setup/teardown to file-scope beforeAll/afterAll. Remove from the vitest import line. Revert the codeql-config.yml additions from the previous commit. The paths: sub-key under query-filters > exclude is not a documented CodeQL feature and silently no-ops. The four js/insufficient-password-hash alerts on api_token fingerprinting are tracked as dismissed false positives in the GitHub Security tab rather than via dead config.	2026-05-16 14:58:19 -04:00
Anso	d882f223f4	feat(api-tokens): switch to sen_sk_ prefixed opaque keys (#1062) ... * feat(api-tokens): switch to sen_sk_ prefixed opaque keys Replace JWT-shaped API tokens with 56-char opaque keys of the form `sen_sk_<43-char base62 random><6-char base62 checksum>` (256-bit entropy, sha256-truncated checksum). Node-proxy tokens stay JWTs. Why: * The api_token path was already a sha256 DB lookup; the JWT signature was wasted work and the 400d JWT ceiling vs DB expires_at was a confusing dual bound. * Opaque tokens carry a verifiable checksum so malformed/typoed values are rejected before any SQLite lookup. * `sen_sk_` prefix is recognizable to GitHub, TruffleHog, GitGuardian and makes the on-wire shape visually distinct from node_proxy JWTs. Changes: * New `utils/apiTokenFormat.ts` (generate + checksum-verify, CSPRNG via randomInt, timingSafeEqual on the checksum compare). * `middleware/auth.ts` and `websocket/upgradeHandler.ts` route opaque tokens before any jwt.verify; 401 messages unified to avoid a token-existence oracle. * `middleware/rateLimiters.ts` short-circuits opaque tokens in the node_proxy detection and keys per-token via a non-reversible sha256 slice so each token keeps its own bucket without a DB hit. * All six existing tests migrated from jwt.sign({scope:'api_token'}) to generateApiToken(); new format-only test suite covering prefix, length, alphabet, checksum reject paths, and a 10k-iteration collision/integrity loop. * Docs (features/api-tokens.mdx, api-reference/overview.mdx) describe the shape and drop the obsolete JWT-ceiling note. * fix(api-tokens): clear CI lint and CodeQL false positives * Drop unused TEST_USERNAME import in remote-console-session.test.ts; the migration to generateApiToken() left it orphaned. * Add a CodeQL barrier model so `generateApiToken`'s ReturnValue does not flow into the `insufficient-password-hash` query. The function emits 256-bit CSPRNG opaque keys; sha256 of the raw token is the correct construction for high-entropy API tokens (bcrypt-class hashes target low-entropy human passwords). CodeQL's name heuristic was treating "Token" as a password source and flagging the standard sha256 wrapping at all 9 call sites. * ci(codeql): exclude js/insufficient-password-hash for token paths The previous barrierModel data extension was a no-op for this rule: the js/insufficient-password-hash query identifies its "password" sources via SensitiveExpr's name heuristic ("token", "secret", "key" substrings), which is upstream of the taint-tracking layer where barrierModel applies. Verified by post-push re-analysis: 9 alerts still open, all undismissed. Replace the dead extension with a path-scoped query-filter in codeql-config.yml so the rule no longer fires on apiTokenFormat, apiTokens, and the test directory. Real user-password hashing code elsewhere in the repo (auth, users, setup routes) remains analyzed. The 9 existing alerts on PR #1062 are dismissed via API as false positives with a justification pointing at this config. Future runs will not re-flag them because of the path filter.	2026-05-15 18:21:28 -04:00
Anso	f63ffdd098	fix(mesh): persist proxy-tunnel self-id across bridge teardown (#1060) ... The proxy-mode WS handler called setProxyTunnelSelfCentralNodeId(null) on tunnel close, dropping the central-namespace nodeId install. After an idle close, the next inbound forwarder traffic on the peer fell back to getDefaultNodeId() (=1 on proxy peers without SENCHO_ENROLL_TOKEN), collided with the alias overlay's nodeId for central's own Local, and dispatched cross-fleet aliases to the same-node path. The signature was TCP open + no banner on the prober, with route.resolve.denied on the peer activity log even though R1's identify install fired correctly when the bridge was warm. The peer's identity in central's namespace is enrollment-stable, not per-WS. Stop null-clearing on teardown. The setter already emits a console.warn when the install is overwritten with a different non-null value, which is the right signal for an actual re-enrollment. Tests: existing assertion that the install becomes null on close is flipped to assert persistence, plus two new tests for the close-reopen lifecycle: same nodeId is idempotent (no second identify activity event), different nodeId emits the overwrite warn. Idempotency test polls deterministically for the first identify event before snapshotting the activity-count baseline so the assertion direction can't invert on a slow CI runner.	2026-05-15 14:33:57 -04:00
Anso	489aab4516	fix(mesh): testUpstream dials via ensureBridge so proxy-mode targets are probeable (#1057) ... testUpstream short-circuited on hasActiveTunnel + getBridge (pilot-only) when probing a cross-node alias. Proxy-mode remotes never have a "pilot tunnel" in the manager's sense, so any operator hitting "Test alias" on a proxy-mode target returned tunnel_down even when the cross-node dial via dialMeshTcpStream worked fine. Replace with await ptm.ensureBridge(target.nodeId), matching the mode- agnostic dispatch dialMeshTcpStream already uses. ensureBridge returns an existing pilot tunnel, an existing proxy bridge, or asks MeshProxyTunnelDialer to dial on demand. The 'no pilot tunnel' vs 'no bridge' distinction goes away (they meant the same thing). Adds two test cases: probing a proxy-mode remote via on-demand bridge, and the tunnel_down path when ensureBridge yields null.	2026-05-15 09:00:39 -04:00
Anso	3d5f0ffacd	fix(mesh): reject opt-in when every service has empty ports (#1056) ... optInStack rejected empty service lists but not the case where services exist with every ports: []. The newPorts Set ended up empty, the function sailed past collision checks, wrote a mesh_stacks row, and produced an empty-aliases override. The stack then appeared online in the Routing tab but no traffic actually routed. Add a check after building newPorts: throw no_target with a clear message before the SENCHO_LISTEN_PORT collision check. Reuses the existing MeshError code (semantically: no routable target) so the frontend's error toast path is unchanged.	2026-05-15 09:00:24 -04:00
Anso	6185206323	fix(mesh): proxy peer learns its central-namespace nodeId at tunnel upgrade (#1055) ... Proxy-mode peers receive an alias overlay from central with nodeIds in central's namespace, but had no source for their own nodeId in that namespace. selfCentralNodeId fell back to the peer's local DB default (always 1), and handleAccept falsely matched cross-node aliases (nodeId=1, central's Local) against the peer's self-id. Cross-node aliases routed to openSameNode and surfaced as route.resolve.denied. Central now appends ?nodeId=<central-namespace-id> to the proxy-tunnel WS URL. The peer's upgrade handler parses it with a strict integer regex, installs into MeshService before the reverse dialer is registered, and clears on disconnect. handleAccept resolves self-id in priority order: proxy-tunnel install > SENCHO_ENROLL_TOKEN > local default. A non-null overwrite to a different value warns loudly (misconfigured deployment). Missing or malformed query param logs a warn and proceeds, leaving reverse-direction dispatch undefined until both sides are upgraded. Adds 5 test cases: nodeId install + teardown, missing/malformed input (including decimal, leading zeros, exponent, whitespace), overwrite warn, and CAS-rejected reverse-dialer leaving the slot unchanged.	2026-05-15 09:00:05 -04:00
Anso	946db9df52	fix(mesh): accept node_proxy at the proxy-tunnel WS upgrade (#1050) ... The mesh proxy-tunnel WS handler was gated on full-admin api_token scope only, so node_proxy JWTs (the credential the Add Remote Node dialog tells operators to generate via Settings -> Nodes -> Generate Token) were silently rejected with HTTP 403. Operators followed the dialog's instructions, enrolled the node cleanly, saw reachableMode='proxy' with no negative badge, opted a stack into mesh, watched the redeploy complete, and only then discovered no bytes flow. node_proxy already authorizes every /api/* surface on the remote (deploy, exec, host console, filesystem), which is a strict superset of what the mesh proxy-tunnel does. Gating mesh more strictly was theatre, not security, and it created a UX trap with no in-product signal pointing at the scope mismatch. Change the upgrade dispatcher to accept any machine-to-machine credential: node_proxy JWT or full-admin api_token. Session cookies and restricted api_token scopes (read-only, deploy-only) remain rejected through the same paths as before. Adds positive/negative coverage in upgrade-order.test.ts for all four credential shapes so the gate stays pinned against future re-tightening. Updates user-facing copy in the mesh docs and Settings -> API Tokens description so the documented path matches the actual code. If we ever introduce a demoted node_proxy variant (audit-only, read-only fleet member), the right move is differentiated node_proxy scope claims inside the tunnel JWT (tracker F-A3), not requiring a separate token type for mesh.	2026-05-14 23:26:41 -04:00
Anso	a38a3e0226	feat(mesh): route mesh traffic over Distributed API remotes (#1048) ... * refactor(mesh): extract shared TCP stream switchboard Pull the `tcp_open` / `tcp_open_ack` / `tcp_open_reverse` / `tcp_close` + `TcpData` handling out of the pilot agent into a reusable module so a second caller (the upcoming proxy-mode WS handler) can run the same frame parser, stream allocator, idle timers, and Compose-label resolver. One parser, two callers, zero drift on a security-sensitive protocol surface. The pilot agent keeps its public `openMeshTcpStream` API and delegates to a per-connection switchboard constructed in `connect()` and torn down in `cleanupAfterDisconnect`. Existing reverse-stream and resolver tests are rewritten to exercise the shared module directly. * feat(mesh): route mesh traffic over Distributed API remotes Sencho Mesh now works against remotes in Distributed API mode in addition to Pilot Agent mode. Central opens a short-lived WebSocket tunnel to the remote's new `/api/mesh/proxy-tunnel` endpoint on demand and tears it down after 5 minutes of idle (configurable via `SENCHO_MESH_PROXY_TUNNEL_IDLE_MS`). Mesh dispatch is mode-agnostic at the routing layer; `PilotTunnelManager.ensureBridge` resolves an existing tunnel or asks the new `MeshProxyTunnelDialer` to dial. The Routing tab badges now use a `reachableMode` classifier: `★ Local` for local, `pilot offline` red badge for a pilot with a down tunnel, and a new `unreachable` red badge surfacing the specific reason (missing token, scope not full-admin, TLS failure, remote does not support proxy mesh). Distributed API remotes with valid credentials show no negative badge; the tunnel opens on first dial. Auth: the proxy-tunnel WS upgrade requires an `Authorization: Bearer` API token with the `full-admin` scope. Lower-scoped tokens are rejected at upgrade time so a leaked read-only token cannot reach the mesh data plane. Bidirectional: the proxy-mode WS handler registers itself as the local `MeshService` reverse dialer (compare-and-swap), so meshed containers on a Distributed API remote can dial cross-node aliases via `tcp_open_reverse` over the same tunnel. Cross-node relays (`PilotTunnelBridge.acceptReverseRelay`) await `ensureBridge` so proxy-to-proxy mesh works without standing tunnels. * docs(mesh): describe Distributed API mesh and unreachable troubleshooting Refresh the user-facing mesh documentation to reflect that mesh works over both Pilot Agent and Distributed API remotes. Adds a Troubleshooting accordion entry for the new `unreachable` Routing tab badge so operators can match a tooltip reason ("api token rejected (scope must be full-admin)", "remote does not support proxy mesh", "TLS handshake failed", "api_url not set", "api token missing") to a concrete fix. * refactor(mesh): apply /simplify review findings Five cleanups surfaced by the post-implementation code review pass. No behaviour change for fleets running on `main`; only internal structure improves. - **Deterministic container IP shared.** Extract the compose-default network preference (`pickContainerIp`) and the conventional-name fast path (`lookupContainerIp`) into `backend/src/mesh/containerLookup.ts`. Both `MeshService.resolveContainerIp` (existing same-node fast path) and the switchboard's `resolveByComposeLabels` (proxy/pilot inbound dial) now use the same logic. Earlier the switchboard helper grabbed the first `Object.values(Networks)` IP, which could flip across daemon versions on multi-network containers; fixed. - **WS URL upgrade extracted.** New `backend/src/utils/wsUrl.ts` exposes `httpUrlToWs(baseUrl)` that maps `http://` to `ws://` and `https://` to `wss://`. Used in both `pilot/agent.ts` and the proxy-tunnel dialer. The previous inline `replace(/^http/, 'ws')` silently downgraded `https://` to `ws://` (cleartext). - **`computeReachable` takes the pre-fetched node.** `getStatus` already iterated `db.getNodes()`; the helper previously re-queried by id per node (N+1 reads). Pass the row in. - **Reachable-reason ternary -> const map.** Replace the nested ternary in `MeshService.computeReachable` with a `Record<DialFailureCode, string>` lookup keyed on the dialer's failure code. - **Activity-type ternary -> const map.** Same flattening inside `MeshProxyTunnelDialer.logActivity`. All mesh tests pass (85/85). Full backend suite green (2126/2126). * fix(mesh): cache proxy dial failures and contain WS handshake errors `ensureBridge` now consults the recent-failure cache before dialing so a continuous mesh workload against a misconfigured proxy-mode remote does not produce one upgrade attempt per cross-node TCP open. `recordFailure` emits at most one activity-log entry per cache window per (nodeId, code) so a connect-loop on a single bad remote cannot flush the ring buffer. Failure messages run through `redactSensitiveText` before reaching the log so any embedded Bearer / JWT / inline-URL credentials are scrubbed. `stop()` clears the inflight map and `dial()` checks `this.stopped` between awaits so a shutdown does not leak a half-opened bridge. When `awaitOpen` rejects (401, 4403, TLS), the dialer attaches a noop 'error' listener before calling `ws.close()`. Without it the ws library emits a tail 'error' on a still-CONNECTING socket that propagates as an unhandled exception. Surfaced by a live-network test against a real proxy-mode peer. Adds `mesh-proxy-tunnel-live.test.ts` (skipped unless MESH_AUDIT_URL and MESH_AUDIT_TOKEN_FILE env vars are set) covering both the happy path and the auth-rejected path against an actual remote Sencho. * feat(mesh): instrument proxy tunnel observability Adds three counters to `PilotMetrics`: - `proxy_bridges_total` (incremented on `registerProxyBridge` success) - `proxy_dials_failed` (every failed dial attempt, not deduped) - `proxy_idle_closes` (idle-sweep teardowns) All three surface automatically via `GET /api/system/pilot-tunnels` since the route returns the full `Counters` snapshot. Gates two pre-existing always-on `console.warn` calls in `tcpStreamSwitchboard.ts` (mid-stream socket errors and Docker resolve failures) behind `isDebugEnabled()`. Both fire on per-stream events and would otherwise violate the diagnostic-log safety rule under load. Adds `mesh-tcp-stream-switchboard.test.ts` covering the forward `tcp_open` path: real localhost dial, resolver errors (no_target, denied), per-tunnel cap saturation, frame-routing fall-through invariants, `tcp_close` socket teardown. Drops `MESH_CONNECT_TIMEOUT_MS` from the public surface; only the switchboard itself uses it. * fix(mesh): tighten Routing tab unreachable handling `RoutingNodeCard.tsx`: the **Add stack to mesh** button now disables when `reachableMode === 'unreachable'`, matching the existing TogglePill behavior. Without this, an operator on an unreachable node could open the opt-in sheet, confirm, and watch the redeploy proceed against a target whose mesh data plane will silently fail to route. Also drops the leftover `status.nodeId !== -1` guard on the pilot- offline badge. `MeshService.ts`: renames `isMeshReachable` to `isMeshConfigured` with the new predicate that returns true for proxy-mode remotes whose creds are valid (the tunnel is opened on demand). `getRouteDiagnostic` now distinguishes "routable" (configured) from "pilotLive" (live tunnel state, only meaningful for pilot mode); without the split, every idle proxy-mode route would report `tunnel down`. `MeshService.setReverseDialer` warns when the unconditional install path silently overwrites a non-null current dialer. By topology a Sencho is either pilot or central, so the branch flags a misconfigured deployment rather than an expected race. Drops the dead `export { ReverseTcpStreamHandle }` re-export from `pilot/agent.ts`; its only consumer imports straight from `mesh/tcpStreamSwitchboard.ts`. Fixes a stale doc comment in `MeshService.ts` that referenced the wrong source file. Adds `mesh-proxy-tunnel-handler.test.ts` covering the WS handler lifecycle: pilot-mode 404 rejection, post-upgrade reverse-dialer install, concurrent-upgrade 1013 rejection (single-tenant slot), and error-path teardown. Updates the troubleshooting accordion in the user docs to mention the disabled-state behavior. * fix(mesh): close ESLint and CodeQL findings on the proxy-tunnel diag logs Remove the unused `reverseDialer` local in `meshProxyTunnel.ts`; the closure target is `localDialer` above and this assignment was always dead. Strip line breaks inline on the three `MeshProxyTunnelDialer` diag log lines so CodeQL `js/log-injection` data flow recognises the sanitisation that `sanitizeForLog` already performs. No behaviour change; the diag logs render the same characters they do today.	2026-05-14 20:51:42 -04:00
Anso	44e40afb62	fix(schedules): align Schedules surface with backend tier gate for Skipper admins (#1047) ... The Schedules sidebar entry was hidden from Skipper admins even though the backend permits them to create and run update, scan, and snapshot schedules. The action picker also showed all 10 actions to every paid admin, so a Skipper selecting Restart, Prune, or any auto_* lifecycle action would 403 on submit. Changes: - Extract SKIPPER_SCHEDULED_ACTIONS as the single source of truth in tierGates.ts; both requireScheduledTaskTier and the GET /scheduled-tasks list filter now reference it (replaces a duplicate local constant in scheduledTasks.ts). - Move the Schedules nav entry from the Admiral block into the isPaid && isAdmin block in useViewNavigationState.ts, mirroring the existing Auto-Update pattern. Console and Audit stay Admiral-only. - Filter the create-form action picker in ScheduledOperationsView.tsx by license variant. Skipper sees Auto-update Stack, Auto-update All Stacks, Fleet Snapshot, and Vulnerability Scan; Admiral sees the full set. - openCreate now defaults formAction to the first visible option so Skipper starts with a valid choice instead of the Admiral-only Restart. Tests: - Add Skipper-variant POST coverage in scheduled-tasks-routes.test.ts: three allow cases (update / scan / snapshot) and a six-action rejection loop covering restart / prune / auto_backup / auto_stop / auto_down / auto_start. - Flip the Skipper assertion in useViewNavigationState.test.tsx to expect scheduled-ops alongside auto-updates.	2026-05-14 10:31:20 -04:00
Anso	5461bc316b	fix: harden stack management operations (#1046)	2026-05-14 10:21:18 -04:00
Anso	8dd0fce621	fix(fleet): show capabilities, version, metrics, and stacks for pilot-agent nodes (#1044) ... When a pilot-agent node was the active node, the UI rendered "does not advertise this capability" across most tabs, a perpetual "Update available" badge, and a Fleet card body with blank CPU/RAM/Disk and "No stacks found". The cause was central-side aggregators in /api/fleet/* and /api/nodes/:id/meta only fanning out to proxy-mode remotes via node.api_url + node.api_token, which are null for pilot-agent. Route every affected aggregator through NodeRegistry.getProxyTarget so the loopback URL backed by the active pilot tunnel is used uniformly: - /api/nodes/:id/meta and /api/fleet/update-status fetch via the new NodeRegistry.fetchMetaForNode helper (resolves the target, delegates to fetchRemoteMeta, returns the shared OFFLINE_META on null). - fetchRemoteNodeOverview, /api/fleet/configuration, /api/fleet/node/:nodeId/stacks, and the stack-containers drilldown fetch through target.apiUrl with conditional Authorization. - fetchRemoteMeta omits the Authorization header when the token is empty (pilot-agent loopback) instead of sending a malformed Bearer string. - Pilot-agent rows preserve pilot_last_seen and mirror it into last_successful_contact so the Fleet "last seen" cell renders the recent tunnel timestamp during a brief reconnect. Pilot-mode capability filter excludes capabilities whose central-pilot path is not yet wired (host-console, self-update). Without this, the Console tab would surface for an Admiral pilot session and click through to central's host because the WS upgrade handler still gates on api_url + api_token. Filtered capabilities are removed at boot via applyPilotModeCapabilityFilter when SENCHO_MODE=pilot. Cache invalidation on tunnel-up: the meta cache for a reconnecting pilot is dropped so the next request rebuilds capabilities and version through the live bridge instead of waiting for the 3-minute TTL. The namespace constant moves to helpers/cacheInvalidation.ts alongside the new invalidateRemoteMetaCache helper. Husky commit-msg hook: add the missing shebang and a .gitattributes rule pinning .husky/* to LF line endings so commits do not fail with "Exec format error" on Windows shells where autocrlf=true converts the hook to CRLF. Tests cover Authorization-header behavior, pilot-mode filter idempotency, fetchMetaForNode dispatch (offline target, pilot-agent loopback, proxy-mode), and the four affected fleet routes for pilot-agent both when the tunnel is up and when it is down.	2026-05-14 10:21:08 -04:00
Anso	e7a3b544c0	fix: harden auto-heal policies (#1042) ... * fix: harden auto-heal policies * fix: resolve auto-heal lint failure	2026-05-14 10:20:57 -04:00
Anso	2dda1bee1f	perf(monitor): parallelize per-container stats fetch in evaluation cycle (#1045) ... The MonitorService.evaluateStackAlerts loop processed containers serially, awaiting one Docker stats call per container. Each Docker stats call inherently waits ~1s for the engine to produce a sample, so cycle time scaled linearly with container count and exceeded the 25s warning threshold on production nodes with ~17+ running containers. Changes: - Extract the per-container body into processContainer(). - Replace the serial for-loop with a bounded-concurrency fan-out (runWithConcurrency, cap of 10) so the cycle runs in roughly max(per-container time) instead of sum. - Add a per-cycle dedup set (firedThisCycle) so a stack rule fires exactly once per cycle even when multiple containers in the stack all breach. Without it, parallel workers race past the cooldown check before any DB write lands and dispatch the same alert N times on first breach. - Tests cover wall-time parallelism, per-container failure isolation, the concurrency cap, and the per-cycle dispatch dedup.	2026-05-14 10:20:49 -04:00
dependabot[bot]	1262385d63	chore(deps): bump the all-npm-backend group in /backend with 11 updates (#1041) ... Bumps the all-npm-backend group in /backend with 11 updates: | Package | From | To | | --- | --- | --- | | [axios](https://github.com/axios/axios) | `1.16.0` | `1.16.1` | | [better-sqlite3](https://github.com/WiseLibs/better-sqlite3) | `12.9.0` | `12.10.0` | | [semver](https://github.com/npm/node-semver) | `7.7.4` | `7.8.0` | | [systeminformation](https://github.com/sebhildebrandt/systeminformation) | `5.31.5` | `5.31.6` | | [ws](https://github.com/websockets/ws) | `8.20.0` | `8.20.1` | | [yaml](https://github.com/eemeli/yaml) | `2.8.4` | `2.9.0` | | [@types/node](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/node) | `25.6.0` | `25.7.0` | | [typescript-eslint](https://github.com/typescript-eslint/typescript-eslint/tree/HEAD/packages/typescript-eslint) | `8.59.2` | `8.59.3` | | [vitest](https://github.com/vitest-dev/vitest/tree/HEAD/packages/vitest) | `4.1.5` | `4.1.6` | | [@aws-sdk/client-ecr](https://github.com/aws/aws-sdk-js-v3/tree/HEAD/clients/client-ecr) | `3.1043.0` | `3.1045.0` | | [@aws-sdk/client-s3](https://github.com/aws/aws-sdk-js-v3/tree/HEAD/clients/client-s3) | `3.1043.0` | `3.1045.0` | Updates `axios` from 1.16.0 to 1.16.1 - [Release notes](https://github.com/axios/axios/releases) - [Changelog](https://github.com/axios/axios/blob/v1.x/CHANGELOG.md) - [Commits](https://github.com/axios/axios/compare/v1.16.0...v1.16.1) Updates `better-sqlite3` from 12.9.0 to 12.10.0 - [Release notes](https://github.com/WiseLibs/better-sqlite3/releases) - [Commits](https://github.com/WiseLibs/better-sqlite3/compare/v12.9.0...v12.10.0) Updates `semver` from 7.7.4 to 7.8.0 - [Release notes](https://github.com/npm/node-semver/releases) - [Changelog](https://github.com/npm/node-semver/blob/main/CHANGELOG.md) - [Commits](https://github.com/npm/node-semver/compare/v7.7.4...v7.8.0) Updates `systeminformation` from 5.31.5 to 5.31.6 - [Release notes](https://github.com/sebhildebrandt/systeminformation/releases) - [Changelog](https://github.com/sebhildebrandt/systeminformation/blob/master/CHANGELOG.md) - [Commits](https://github.com/sebhildebrandt/systeminformation/compare/v5.31.5...v5.31.6) Updates `ws` from 8.20.0 to 8.20.1 - [Release notes](https://github.com/websockets/ws/releases) - [Commits](https://github.com/websockets/ws/compare/8.20.0...8.20.1) Updates `yaml` from 2.8.4 to 2.9.0 - [Release notes](https://github.com/eemeli/yaml/releases) - [Commits](https://github.com/eemeli/yaml/compare/v2.8.4...v2.9.0) Updates `@types/node` from 25.6.0 to 25.7.0 - [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases) - [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/node) Updates `typescript-eslint` from 8.59.2 to 8.59.3 - [Release notes](https://github.com/typescript-eslint/typescript-eslint/releases) - [Changelog](https://github.com/typescript-eslint/typescript-eslint/blob/main/packages/typescript-eslint/CHANGELOG.md) - [Commits](https://github.com/typescript-eslint/typescript-eslint/commits/v8.59.3/packages/typescript-eslint) Updates `vitest` from 4.1.5 to 4.1.6 - [Release notes](https://github.com/vitest-dev/vitest/releases) - [Commits](https://github.com/vitest-dev/vitest/commits/v4.1.6/packages/vitest) Updates `@aws-sdk/client-ecr` from 3.1043.0 to 3.1045.0 - [Release notes](https://github.com/aws/aws-sdk-js-v3/releases) - [Changelog](https://github.com/aws/aws-sdk-js-v3/blob/main/clients/client-ecr/CHANGELOG.md) - [Commits](https://github.com/aws/aws-sdk-js-v3/commits/v3.1045.0/clients/client-ecr) Updates `@aws-sdk/client-s3` from 3.1043.0 to 3.1045.0 - [Release notes](https://github.com/aws/aws-sdk-js-v3/releases) - [Changelog](https://github.com/aws/aws-sdk-js-v3/blob/main/clients/client-s3/CHANGELOG.md) - [Commits](https://github.com/aws/aws-sdk-js-v3/commits/v3.1045.0/clients/client-s3) --- updated-dependencies: - dependency-name: axios dependency-version: 1.16.1 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: all-npm-backend - dependency-name: better-sqlite3 dependency-version: 12.10.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: all-npm-backend - dependency-name: semver dependency-version: 7.8.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: all-npm-backend - dependency-name: systeminformation dependency-version: 5.31.6 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: all-npm-backend - dependency-name: ws dependency-version: 8.20.1 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: all-npm-backend - dependency-name: yaml dependency-version: 2.9.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: all-npm-backend - dependency-name: "@types/node" dependency-version: 25.7.0 dependency-type: direct:development update-type: version-update:semver-minor dependency-group: all-npm-backend - dependency-name: typescript-eslint dependency-version: 8.59.3 dependency-type: direct:development update-type: version-update:semver-patch dependency-group: all-npm-backend - dependency-name: vitest dependency-version: 4.1.6 dependency-type: direct:development update-type: version-update:semver-patch dependency-group: all-npm-backend - dependency-name: "@aws-sdk/client-ecr" dependency-version: 3.1045.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: all-npm-backend - dependency-name: "@aws-sdk/client-s3" dependency-version: 3.1045.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: all-npm-backend ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2026-05-14 09:22:40 -04:00
Anso	b52323036b	fix: harden stack label permissions (#1036) ... * fix: harden stack label permissions * fix: avoid test db init from debug logging	2026-05-13 11:56:22 -04:00
Anso	328a98439d	fix: harden vulnerability scan scheduling (#1035)	2026-05-13 09:40:45 -04:00
Anso	c31d48b933	fix: harden git source webhooks (#1033) ... * fix: harden git source webhooks * fix: make path validation visible to CodeQL static analysis Add explicit isValidStackName guard in getEnvContent, isValidGitSourcePath pre-validation in readRepoFile, and URL hostname check in remoteStackRequest to satisfy CodeQL taint-tracking so the pipeline passes. * fix: use path.basename and URL constructor patterns recognized by CodeQL Replace helper-based path validation with inline path.basename and path.resolve patterns that CodeQL taint-tracking recognizes as sanitizers, following the established MeshService convention. Switch remote webhook URL construction to the new URL(path, base) pattern so the origin is derived from the validated target URL. * fix: add CodeQL SSRF barrier model for remote node URL construction Introduce buildRemoteApiUrl utility and companion CodeQL barrier model (safeUrl.model.yml) that tells the taint-tracking engine the returned URL is constrained to the configured target origin. The URL constructor guarantees same-origin, but CodeQL cannot verify that without a model. * fix: inline URL protocol validation in remoteStackRequest Replace the barrier-model approach with an explicit inline check that CodeQL recognizes: verify the target URL uses http/https protocol before constructing the fetch URL with the URL constructor. * fix: exclude SSRF query from WebhookService proxy code The remoteStackRequest method proxies HTTP requests to admin-configured remote node URLs by design (the Distributed API model). CodeQL flags the fetch() call as SSRF because the URL is user-configured, but this data flow is architectural intent. Exclude js/server-side-request-forgery from this file. * fix: map nodeId to server-controlled URL components before fetch Follow the CodeQL SSRF remediation pattern: user input (nodeId) selects an entry from the configured-node registry, then the URL is rebuilt from validated components (protocol, host from allow-list, encoded path). Protocol is restricted to http/https, path traversal is rejected, and the hostname is verified against the configured-node allow-list. * fix: remove unnecessary escape in endpoint validation regex	2026-05-13 03:02:21 -04:00
Anso	b1c5fe8391	fix: harden deploy enforcement paths (#1030) ... * fix: harden deploy enforcement paths * fix: update Docker toolchain to Go 1.26.3 * fix: repair Dockerfile tr argument split across lines * fix: bump protobufjs to clear npm audit high-severity advisories * fix(test): add execFile to child_process mock in compose-images test * fix: resolve merge conflicts with main * fix: resolve merge conflicts with main * fix: resolve merge conflicts with main	2026-05-12 19:30:49 -04:00
Anso	74ae2ce0c6	fix: harden atomic deployment rollback (#1029) ... * fix: harden atomic deployment rollback * fix: update Docker toolchain to Go 1.26.3 * fix: repair Dockerfile tr argument split across lines * fix: bump protobufjs to clear npm audit high-severity advisories * fix: sanitize error objects in console.error to prevent log injection	2026-05-12 15:58:30 -04:00
Anso	69b6ac1f3b	fix: harden stack file explorer operations (#1028) ... * fix: harden stack file explorer operations * fix: update Docker toolchain to Go 1.26.3 * fix: repair Dockerfile tr argument split across lines * fix: bump protobufjs to clear npm audit high-severity advisories	2026-05-12 15:49:51 -04:00
Anso	19cdb3681d	fix: harden blueprint deployment guardrails (#1027) ... * fix: harden blueprint deployment guardrails * fix: update Docker toolchain to Go 1.26.3 * fix: repair Dockerfile tr argument split across lines * fix: bump protobufjs to clear npm audit high-severity advisories	2026-05-12 15:49:19 -04:00
Anso	b8af40d2b4	fix(mesh): enumerate stacks on remote pilot nodes for opt-in sheet (#1025) ... The mesh opt-in sheet showed "No stacks deployed on this node yet" for every remote pilot because GET /api/mesh/nodes/:nodeId/stacks called FileSystemService.getInstance(nodeId).getStacks() unconditionally, which always reads central's own filesystem regardless of which node the operator targeted. Fix dispatches through the proxy chain when the targeted node is remote, mirroring the C-3 pattern that already governs every other mesh endpoint needing data from a remote Sencho: - New endpoint GET /api/mesh/local-stacks (Admiral-gated) returns the bare stacks list from THIS Sencho's own filesystem. Mirrors the precedent set by /api/mesh/local-services/:stackName. - New MeshService.listLocalStacks() and MeshService.listStacksOnNode() helpers; the latter dispatches local vs remote and degrades to an empty list on transport failure (no proxy target, non-2xx response, malformed body). - Refactored route delegates the stack-name list to listStacksOnNode; central's mesh_stacks DB is still authoritative for the opt-in flag set per C-3. Tests cover the local path, the remote-OK path with header forwarding, non-2xx, no-target (pilot tunnel down), malformed bodies, and unknown node ids.	2026-05-10 04:01:39 -04:00
Anso	41a1df279d	fix(mesh): fix pilot handleAccept dispatch by deriving self nodeId from enrollment token (#1023) ... On a pilot node, handleAccept compared target.nodeId (from central's perspective via pilotAliasOverlay) against NodeRegistry.getDefaultNodeId() which always returns 1. This inverted same-node vs cross-node dispatch: - central's alias (nodeId=1) matched localNodeId=1 -> openSameNode on pilot (no container -> silent close, 0 bytes) - pilot's own alias (nodeId=14) != 1 -> openCrossNode -> tunnel loop Fix: add resolveSelfCentralNodeId() that decodes the nodeId claim from SENCHO_ENROLL_TOKEN (present on every pilot; payload decode only, no signature verification). Returns getDefaultNodeId() on central (env unset). Cached in selfCentralNodeId at start(); handleAccept reads the cache. After this fix the reverse direction (pilot-prober -> central echo) routes through openCrossNode (reverse tunnel) and the same-node path (pilot-prober -> pilot echo) routes through openSameNode (local Dockerode dial). Closes BUG-3 banner data-path regression. Tests: 5 new cases covering resolveSelfCentralNodeId token extraction, fallback paths, and handleAccept dispatch routing on a simulated pilot.	2026-05-10 03:09:53 -04:00
Anso	6ac7da978c	fix(mesh): push alias port map to pilot so reverse-direction forwarder binds listeners (#1021) ... * fix(mesh): apply D-1 pushed override on pilot deploys via file-presence fallback When isMeshStackEnabled returns false, ensureStackOverride now checks for an override file already on disk before returning null. On pilot nodes the mesh_stacks table is intentionally empty (opt-in state lives on central per the C-3 design), so the DB gate blocked the override that central had already pushed via applyLocalOverride. File-presence is safe as the fallback because removeOverrideFromNode sends a DELETE to the pilot when a stack is opted out, so a stale file cannot survive past opt-out. Also aligns removeStackOverride to use path.basename consistently with all other override-path construction sites in the same file. Adds two tests: pilot node with a pushed file returns the path; pilot node with no file returns null. * fix(mesh): push alias port map to pilot so reverse-direction forwarder binds listeners Pilots have no mesh_stacks rows by design (C-3), so refreshAliasCache() produced an empty aliasByPort map and syncForwarderListeners() bound zero ports. Reverse-direction TCP probes (pilot prober -> central alias) failed with connection refused at the OS level because no listener existed. Fix: extend the D-1 override push payload to include the full MeshGlobalAlias records (host + port + routing metadata). The pilot stores these in a new pilotAliasOverlay map keyed by stack name. refreshAliasCache() merges the overlay after the (empty) DB loop, populating aliasByPort correctly. syncForwarderListeners() then binds the alias ports immediately after the push completes. removeLocalOverride() clears the overlay entry and re-syncs on opt-out. The two populations (DB rows on central, pilotAliasOverlay on pilots) are mutually exclusive by the C-3 invariant, so the first-write-wins port collision policy in refreshAliasCache is safe.	2026-05-10 02:04:52 -04:00
Anso	c12b3ba995	fix(mesh): apply D-1 pushed override on pilot deploys via file-presence fallback (#1019) ... When isMeshStackEnabled returns false, ensureStackOverride now checks for an override file already on disk before returning null. On pilot nodes the mesh_stacks table is intentionally empty (opt-in state lives on central per the C-3 design), so the DB gate blocked the override that central had already pushed via applyLocalOverride. File-presence is safe as the fallback because removeOverrideFromNode sends a DELETE to the pilot when a stack is opted out, so a stale file cannot survive past opt-out. Also aligns removeStackOverride to use path.basename consistently with all other override-path construction sites in the same file. Adds two tests: pilot node with a pushed file returns the path; pilot node with no file returns null.	2026-05-10 01:29:21 -04:00
Anso	2d9e4d0480	fix(mesh): derive override services from compose file and surface cross-node dispatch (#1017) ... * fix(mesh): derive override services from compose file and surface cross-node dispatch Three fixes for issues found during Phase B end-to-end verification: 1. Override generator no longer queries Dockerode for services. Deploys removed all containers in the project right before composeArgs ran, so docker.listContainers({label: project=...}) returned [] and the generator wrote services: {} which broke meshed user containers. New helper getDeclaredStackServiceNames reads the local stack's compose file via FileSystemService + YAML.parse and returns the top-level services keys. Used from ensureStackOverride and applyLocalOverride. Defensive fallback: when the read returns [] AND a previous override on disk has a non-empty services map, keep the file and emit mesh.override.preserved. 2. Boot regen race against the pilot tunnel. regenerateAllOverrides ran ~1s before the first pilot tunnel-up event, so pushes to pilot-mode nodes failed with "no proxy target". MeshService.start now also runs regenerateOverridesForNode(nodeId) inside the tunnel-up listener, covering both the boot-race window and mid-runtime tunnel reconnects. Idempotent. 3. openCrossNode emitted no activity log when a cross-node dial got stuck waiting for tcp_open_ack. The forwarder accepted the TCP handshake (so nc -v reported open) but logged nothing between accept and close. Now logs route.dispatch on entry and emits tunnel.fail after PROBE_TIMEOUT_MS if the agent never acks. Timer is cleared on every tcpStream and src lifecycle event. Tests: 8 new in mesh-service.test.ts cover the helper, the BUG-1 override write path, the defensive preservation fallback, the tunnel-up regen handler, and both cross-node dispatch logging behaviors. Full vitest suite green (1998 tests pass). * fix(mesh): satisfy CodeQL path-injection model in new override-read paths Two new fs.readFile sites added in the previous commit failed the CodeQL check on the PR even though both paths were guarded by isValidStackName + isPathWithinBase. The file's existing applyLocalOverride pattern uses path.basename(stackName) before the join (commented as "Recognized by CodeQL's path-injection model"); apply the same sanitizer to the new code paths. - getDeclaredStackServiceNames: wrap stackName in path.basename when building composePath. - readExistingOverrideServiceNames: take (dir, stackName) instead of a pre-built filePath so the sanitizer lives next to the read sink. - ensureStackOverride: build the override file path with path.basename(stackName) for symmetry with applyLocalOverride. Runtime behavior unchanged (path.basename is a no-op for any name that passed isValidStackName). Tests still green.	2026-05-09 05:39:33 -04:00
Anso	7cca68f84a	fix(mesh): operator-triggered override regen and boot lifecycle hardening (#1016) ... Audit-driven follow-up to the F6+F7 fix. Closes the High-severity findings from the mesh-dial-path audit without expanding into the architectural follow-ups (JWT TTL, TLS enforcement, per-stack JWT scope) which need their own design discussions. Changes: 1. New `POST /api/mesh/regen-overrides` Admiral-gated endpoint that calls `MeshService.regenerateAllOverrides()` and returns a structured `MeshRegenSummary { regenerated, failures, skipped, reason? }`. Lets the operator rerun boot regen after fixing a remote node that was offline at central startup, instead of opt-out + opt-in for every meshed stack on that node. Audit-log row carries the outcome (success / partial / skipped / error). 2. `regenerateAllOverrides` now aggregates per-stack outcomes into a single summary log row (`mesh override regen complete: N succeeded, M failed across K node(s)`) with `failedNodeIds` in `details`. Per- stack warnings still emitted. When skipped because `senchoIp` is null, emits an explicit warn instead of silently no-oping. 3. `MeshService.start()` now wraps `refreshAliasCache` and `syncForwarderListeners` in their own try/catch so a throw from either step is logged and the boot continues to override regen. The closing log line surfaces data-plane state ("MeshService started (data plane ok)" or "(data plane unavailable (<reason>))") so an operator grepping startup output sees a half-init mesh immediately. 4. `sanitizeForLog` wrap on four previously unsanitized `err.message` writes (cross-node tcpStream error handler + three probe error paths). Container IPs and local socket details no longer leak into the activity buffer or the probe response body. 5. Updated three existing F6 regression tests for the new return shape and aggregated summary message. Added two new tests: version-skew 404-push handling, and concurrent opt-in during regen.	2026-05-09 03:45:04 -04:00
Anso	0947a80cda	fix(mesh): trust central for cross-node dial auth and regenerate overrides at boot (#1014) ... Two bugs in the same Phase D follow-up surface, fixed together because they both block declaring B-verify complete on the production fleet. Cross-node mesh dials returned `denied` at the agent. The pilot's `tcp_open` handler in `agent.ts::resolveMeshTarget` consulted the local SQLite `mesh_stacks` table, which is no longer written to under the post-Phase D control plane (state lives only on central). Drop the check. The pilot tunnel JWT (scope `pilot_tunnel`, signed with central's `auth_jwt_secret`) authenticates the caller; the same trust model already applies to filesystem ops, exec, and container control over the same tunnel. Threat-model trade-off: a leaked `pilot_tunnel` JWT or compromised central can now dial any compose-managed service on the pilot. Containers without `com.docker.compose.project` + `com.docker.compose.service` labels remain unreachable via this path. `MeshService.start()` did not regenerate compose override files at boot. After a Sencho restart with missing overrides on disk, meshed user containers had no `extra_hosts` / `networks: [sencho_mesh]` injection until each stack was opted out and back in. Add `regenerateAllOverrides()` that walks every `mesh_stacks` row and re-pushes via `pushOverrideToNode`. Best-effort: per-stack failures log to the mesh activity buffer; `MeshService.start()` is fire-and-forget at startup so a slow remote node does not delay boot. Tests: - `pilot-agent-mesh-resolve.test.ts` (new): mocks dockerode and proves `resolveMeshTarget` no longer returns `denied` with an empty `mesh_stacks` table. - `mesh-service.test.ts`: three new cases for `regenerateAllOverrides` - fan-out across the fleet, skip when `senchoIp` is null, log per-stack warning on push failure without throwing.	2026-05-09 03:27:39 -04:00
Anso	693f9b4495	fix(fleet): drop tier gate from stack and container list endpoints (#1012) ... Community tier opened the Fleet Overview drilldown (node card → stack list → container list) but two read-only metadata endpoints in fleet.ts kept their requirePaid gate. Stack names still rendered because they ride on the un-gated /api/fleet/overview payload, but expanding a stack always re-fetched containers and hit the orphan gate, surfacing as a "Failed to load containers for X" toast on every node card on Community. Drop requirePaid from GET /node/:nodeId/stacks and GET /node/:nodeId/stacks/:stackName/containers. Both handlers stay behind authMiddleware, validate inputs, and proxy to the per-node api_token for remote nodes. Paid mutations (bulk update, scheduled snapshots, label bulk actions, Fleet Actions cards) keep their gates in fleetActions.ts. Add positive Community-tier tests in fleet.test.ts and clear the stale "Requires Skipper or Admiral license" line from the OpenAPI descriptions.	2026-05-09 01:55:56 -04:00
Anso	23bbee4f45	feat(mesh): replace host-mode with shared sencho_mesh Docker network (#1009) ... * feat(mesh): replace host-mode with shared sencho_mesh Docker network Phase D of the mesh redesign: drop the operator's `network_mode: host` requirement and the `host-gateway` extra_hosts pattern that did not work on cloud iptables-restrictive distros (OCI, etc.) or Docker Desktop. Each Sencho creates a shared `sencho_mesh` Docker bridge network on boot (default subnet 172.30.0.0/24, override via SENCHO_MESH_SUBNET), pins itself at `<network>+2`, and attaches every meshed user service to the same bridge. Compose overrides now emit IP-based `extra_hosts` plus a top-level `networks` block declaring `sencho_mesh` external. Override delivery: central renders for local stacks; for remote stacks it sends the fleet alias list to the remote's new `PUT /api/mesh/local- override/:stackName` endpoint, which renders against the remote's OWN local senchoIp and writes under its OWN DATA_DIR. Each node may use a different subnet without coordination beyond the env var. Opt-in / opt-out now trigger an automatic redeploy of the affected stack via the existing deploy code path (local: ComposeService; remote: HTTP POST through proxyFetch). The frontend opt-in sheet shows a confirmation modal (ConfirmModal) before the mutation. Failed redeploys emit both a mesh activity event and a durable audit-log row. Hardening: - Reserve port 1852 at opt-in (prevents user containers from racing the Sencho API listener). - ensureMeshNetwork refuses to continue if `sencho_mesh` exists with a mismatched subnet rather than silently routing to the wrong IP. - Idempotent network connect/disconnect helpers in DockerController. - optInStack rolls back the DB row if the just-inserted stack's override push fails (no half-states surviving across calls). - regenerateOverridesForNode runs in parallel and skips the just- pushed stack on opt-in. Operator template: drop `network_mode: host`, restore `ports: ["1852:1852"]`. Mesh now works identically on Linux LAN, OCI, and Docker Desktop without firewall changes. Docs: rewrite docs/features/sencho-mesh.mdx around the shared bridge network, document SENCHO_MESH_SUBNET, surface the host-network-service opt-in restriction, and cross-link with the Pilot Agent docs. BREAKING CHANGE: the operator's `docker-compose.yml` no longer uses `network_mode: host`. After upgrading, redeploy any meshed stacks once so they pick up the new IP-based override and join `sencho_mesh`. * fix(mesh): wrap stackName with path.basename in local-override fs ops CodeQL flagged js/path-injection on the new applyLocalOverride and removeLocalOverride methods because they are publicly reachable and its data-flow model does not recognize isValidStackName / isPathWithinBase as sanitizers. The validation IS sufficient (the allowlist regex blocks path separators, the path-prefix check blocks escape), but path.basename is a model CodeQL recognizes and is purely defensive: for any input that already passes isValidStackName, basename is the identity.	2026-05-09 00:11:09 -04:00
Anso	ccad5c925b	feat(nodes): hide hub-only views when active node is remote (#1007) ... * feat(nodes): hide hub-only views when active node is remote Fleet, Schedules, Audit, Logs, and Auto-Update operate on hub-owned state (node registry, fleet schedules, centralized audit, fleet-wide log aggregation, fleet-wide update preview). When the active node is remote, proxying those surfaces would show that remote's own disconnected state instead of the hub's. Hide them from the nav strip and force-redirect to Home if one was open during the node switch. Backend hubOnlyGuard middleware sits between nodeContextMiddleware and the remote proxy and rejects /api/scheduled-tasks, /api/audit-log, and /api/notification-routes with 403 + HUB_ONLY_ENDPOINT when nodeId resolves to a remote, closing the script-bypass path the UI gating cannot reach. Settings sub-sections were already gated via the hiddenOnRemote registry; this extends the same model to top-level views. * docs(nodes): note hub-only visibility on Fleet, Schedules, Audit, Logs, Auto-Update Each of the five hub-only feature pages now points readers to the canonical "What top-level views show when a remote node is active" section in multi-node.mdx, so users landing directly on a feature page understand why the nav item disappears when they switch to a remote node.	2026-05-08 22:54:58 -04:00
Anso	10a469ecb4	fix(mesh): bind every alias port on every node, not just local-owned (#1004) ... syncForwarderListeners filtered the bind set to ports owned by the local node. That broke cross-node mesh routing end to end: meshed containers' extra_hosts: <alias>:host-gateway entries resolve to the SOURCE node's gateway, so the source node is where the inbound TCP connection lands. With the filter, the source node never bound the remote alias's port and the connection went nowhere. The architecture model documented in docs/internal/architecture/ mesh.md (data flow step 3) is explicit: monolith's Sencho process accepts the connection on its MeshForwarder listener bound to port 5432 — where monolith is the source and opsix is the target. Every meshed node binds every alias port. handleAccept then resolves the alias and dispatches to openSameNode (when target.nodeId equals the local node) or openCrossNode (otherwise). Both branches were already correct; only the bind filter was wrong. Fleet-wide port collisions remain blocked at opt-in time (optInStack checks aliasByPort), so binding every alias port is unambiguous. New vitest case asserts the want-set includes both local-owned and remote-owned ports. Discovered while running PR 0 verification on v0.75.0 against Local + sencho-pilot-test: connections from a Local prober to the pilot's alias port hit Local's host with no listener.	2026-05-08 16:57:13 -04:00
Anso	567e524450	feat(mesh): bidirectional routing via tcp_open_reverse and central relay (#1003) ... * feat(mesh): bidirectional routing via tcp_open_reverse and central relay Phase B of the mesh redesign. Adds the reverse-direction protocol so a pilot's MeshForwarder can route cross-node traffic to central or to another pilot. Central relays pilot-to-pilot streams transparently; pilots keep their existing single outbound WS to central. Protocol additions (backend/src/pilot/protocol.ts): - TcpOpenReverseFrame { s, targetNodeId, stack, service, port } sent agent to primary. Reuses existing tcp_open_ack, tcp_close, and TcpData binary frames for the response and byte plane. - AGENT_REVERSE_ID_BASE = 0x40000001 splits the 32-bit id space so agent-allocated reverse stream ids never collide with primary- allocated forward ids on the same tunnel. - StreamIdAllocator now wraps to its configured start (not always 1) so an allocator parameterized with the agent base stays in the agent half across the wrap. Agent (backend/src/pilot/agent.ts): - New ReverseTcpStreamHandle exported class, EventEmitter facade matching PilotTunnelBridge.TcpStream's surface (write, end, destroy plus open, data, error, close events). - Public openMeshTcpStream(target) allocates a reverse id, sends tcp_open_reverse, returns the handle. - onTcpOpenAckReverse handles inbound ack, dispatches open or error+close to the matching handle. - TcpData and tcp_close binary/JSON paths route ids in the reverse range to reverseTcpStreams; existing primary-allocated paths unchanged. - streamCount, cleanupAfterDisconnect, onStreamIdle include reverse streams. The allocator is reset to a fresh base on disconnect so long-lived agents that reconnect many times do not drift up the id space. - startPilotAgent registers the agent as MeshService's reverse dialer via lazy import. Bridge (backend/src/services/PilotTunnelBridge.ts): - Two new StreamState kinds: reverse_local (target = central, dial Dockerode container IP) and reverse_relay (target = another pilot, open a forward TcpStream on the target pilot's bridge). - handleTcpOpenReverse validates the agent-id range and stream cap, then dispatches to acceptReverseLocal or acceptReverseRelay via lazy imports of MeshService, NodeRegistry, and PilotTunnelManager (avoids module cycles). - acceptReverseLocal calls MeshService.resolveContainerIp (now public), opens net.createConnection, splices bytes through the tunnel. Pre-connect error handler is removed inside connect to avoid double-firing with the mid-stream error path. - acceptReverseRelay opens a forward TcpStream on the target bridge, splices bytes between the two tunnels. - Existing tcp_close and TcpData handlers extended for the new state kinds. teardownStream extended. MeshService (backend/src/services/MeshService.ts): - resolveContainerIp made public so the bridge's local-target path can dial the same shape. - New reverseDialer field plus setReverseDialer setter. Pilot mode registers the agent at boot; central mode leaves it null. - New private dialMeshTcpStream dispatcher: pilot side uses the reverse dialer, central side uses PilotTunnelManager.getBridge. - openCrossNode refactored to call the dispatcher and use the active-stream record's id for activity logging. - New exported MeshTcpStreamLike interface that both PilotTunnelBridge.TcpStream and ReverseTcpStreamHandle satisfy structurally; ReverseMeshDialer is the contract for the setter. Tests (3 files, 16 new cases, 172 total pass): - pilot-protocol-tcp.test.ts: tcp_open_reverse round-trip; agent base invariant. - pilot-agent-reverse-stream.test.ts: openMeshTcpStream allocation, ws-not-open, frame shape, write encoding, end emits tcp_close, ack-success, ack-failure, low-id ack ignored, inbound TcpData routing. - pilot-bridge-reverse.test.ts: id-range validation, no-target failure, successful local dial against a real upstream server. Together with PR #1000 (Phase A) and PR #1001 (deps), Phase B completes the central-pilot-pilot mesh routing matrix. Phase C (mesh over proxy-mode remotes) is the planned follow-up. * test(pilot): drop unused encodeJsonFrame import Lint failed on pilot-agent-reverse-stream.test.ts after the test changed from constructing tcp_open_reverse frames inline to driving the agent's frame-dispatch path with synthetic objects. The import is no longer referenced; ESLint's no-unused-vars rule rejected it.	2026-05-08 16:21:34 -04:00
Anso	f599110386	feat(mesh): collapse sidecar into Sencho process via in-process forwarder (#1000) ... The separate saelix/sencho-mesh sidecar container is gone. The forwarder logic that previously lived in mesh-sidecar/src/forwarder.ts moves into the Sencho process as backend/src/services/MeshForwarder.ts, a thin per-port net.Server lifecycle wrapper. MeshService implements the host interface and owns resolve plus splice; MeshForwarder owns listener boilerplate. One container per node, no separate image to publish, no control WebSocket. Operator-facing change: the Sencho container now runs in network_mode: host so the forwarder can bind alias ports on the host network where meshed containers' extra_hosts host-gateway entries point. Without host network mode the listeners would land in the container's namespace and inbound traffic from peers would never reach them. The 1852:1852 port publish becomes a no-op under host mode and is commented out in the operator template. Same-node forward path now dials the target container's bridge IP via Dockerode (preferring the compose default network for deterministic selection across daemon versions) instead of 127.0.0.1. The legacy 127.0.0.1 path only worked when the target service published its port to the host; the IP path works regardless. Cross-node mesh routing in this phase is central -> pilot direction only via PilotTunnelManager.openTcpStream. Pilot -> central and pilot <-> pilot via central relay land in Phase B with the tcp_open_reverse frame. Deletions: - mesh-sidecar/ package entirely (Dockerfile, package, sources, tests) - backend/src/websocket/meshControl.ts - MeshService sidecar lifecycle: spawnSidecar, stopSidecar, isSidecarRunning, mintSidecarToken, verifySidecarToken, attachSidecarSocket, handleSidecarResolve, sendSidecar - POST /api/mesh/nodes/:id/sidecar/restart route - /api/mesh/control WS dispatch in upgradeHandler Type cleanup: 'sidecar' literal removed from MeshActivitySource and MeshProbeResult.where (also the frontend mirror). MeshNodeStatus sidecarRunning becomes localForwarderListening (boolean | null) so non-local nodes get a null instead of an unconditional false; the honest semantic is "this view only knows the local forwarder state; remote forwarder status lands in Phase B." MeshNodeDiagnostic sidecar object becomes forwarder { listening, listenerCount }. Frontend MeshDiagnosticsSheet drops the restart-sidecar action and sidecar liveness card; surfaces forwarder state plus a "runs in-process; no separate container" caption. Resolves audit findings C-1 (data plane non-functional), C-2 (sidecar control WS not loopback-enforced), C-4 (sidecar lifecycle Dockerode- on-remote, PR #999 closed), and C-5 (saelix/sencho-mesh:latest unreachable). C-3 (PR #992) is unchanged. M-12 (PR #994) is unchanged.	2026-05-08 15:51:39 -04:00
Anso	b5463e1771	chore(deps): bump fast-uri to 3.1.1+ to clear GHSA-q3j6-qgpj-74h6 (#1001) ... GitHub published a high-severity advisory against fast-uri <= 3.1.0 (path traversal via percent-encoded dot segments, GHSA-q3j6-qgpj-74h6) which CI's npm audit step now flags. The vulnerability is not exploitable in Sencho's usage: The dependency chain is composerize@1.7.5 -> composeverter@1.7.5 -> ajv@8.18.0 -> fast-uri@3.1.0. Composerize is consumed only by POST /api/convert/ in routes/convert.ts, which parses CLI-style docker run flags (not URLs) and returns YAML. ajv's use of fast-uri is for internal JSON-Schema $ref/$id resolution, not user input. The CVE requires attacker-controlled URLs flowing into fast-uri.normalize() or equal() for path-based policy bypass; that flow does not exist in Sencho. Even though the vulnerability is not exploitable, npm audit blocks CI regardless of exploitability assessment (it does not read VEX). This commit is a lockfile-only bump produced by npm audit fix. ajv@8.18.0's range already accepts fast-uri >= 3.1.1 so no package.json change is needed. Verification: npm audit reports 0 vulnerabilities post-fix; backend test suite (1921 tests, 5 skipped) is green; tsc --noEmit is clean.	2026-05-08 15:45:32 -04:00
Anso	1803512f70	feat: drop stack labels and network topology to Community tier (#995) ... * feat(labels): drop tier gate to Community for organization endpoints Stack Labels CRUD and per-stack assignment are now Community-tier features: list, create, update, delete labels, and assign labels to a single stack. The two automation surfaces stay Skipper+: per-label bulk deploy / stop / restart (POST /api/labels/:id/action) and the Fleet Actions tab's bulk-assign card (POST /api/fleet-actions/labels/bulk-assign). Tier story is now organize free, automate paid. Add a route-level test that proves the CRUD endpoints succeed on a Community license while the bulk-action endpoint still returns 403. Update overview, licensing, and stack-labels docs to reflect the new tier placement and to note that bulk actions on a label still require Skipper or Admiral. * feat(topology): drop tier gate to Community for network topology view The Resources tab's Networks > Topology view is now available on every tier. Drops requirePaid from GET /api/system/networks/topology, removes the isPaid wrapper around the List | Topology toggle in ResourcesView, and removes the PaidGate around the topology graph. CapabilityGate stays in place so a node running on a build without the network-topology capability still renders its lock card instead of the graph. Add a route-level test that proves the endpoint returns 200 on a Community license. Update licensing and resources docs to reflect the new tier placement. * fix(labels): expose Settings > Labels tab on Community tier The settings registry entry for the Labels tab still carried tier: 'skipper', which kept the tab hidden in the Settings sidebar even though the underlying CRUD endpoints now serve Community. Drop the tier flag so Community users can discover and reach the section that backs the already-Community-tier label endpoints.	2026-05-08 15:24:25 -04:00
Anso	3d1cf0f8f1	fix(pilot): generate auth_jwt_secret on pilot-agent boot (#997) ... Pilot-agent hosts never run the first-run setup wizard, so the wizard path that normally generates auth_jwt_secret (routes/auth.ts) never fires. Without that secret, the agent-side loopback auth helper added in PR #990 (pilot/agent.ts::getLoopbackAuthHeader) returned null at mint time, no Authorization header was attached on the forwarded loopback request, and the local Sencho's authMiddleware rejected every proxied call with 401 "Authentication required". This was reproducible end-to-end on v0.74.1, .2, and .3. Add ensurePilotJwtSecret() to bootstrap/startup, called early in startServer (before LicenseService.initialize, which only reads system_state and does not touch global_settings, so the order is safe). When SENCHO_MODE=pilot and global_settings.auth_jwt_secret is empty, generate 64 random bytes (hex-encoded, matching the wizard's pattern in routes/auth.ts) and persist. Subsequent boots see the persisted value and no-op. Outside pilot mode the function is a no-op since the wizard owns the lifecycle. The helper is exported so the regression test can exercise the real function rather than maintaining a hand-mirrored copy. Three vitest cases cover: first-boot generation, no-op on persisted secret, no-op in non-pilot mode. Together with PR #989 (proxy-side bridge dispatch), PR #990 (agent-side loopback auth injection), PR #992 (cross-node control plane via HTTP proxy), and PR #994 (local-node mesh-reachable diagnostic), this completes the central, bridge, agent, local-Sencho HTTP path for pilot-agent-mode nodes.	2026-05-08 11:59:02 -04:00
Anso	34581e55e7	fix(mesh): treat local-node aliases as always reachable in diagnostics (#994) ... MeshService.getRouteDiagnostic called PilotTunnelManager.hasActiveTunnel unconditionally for the alias's target node. Local nodes never establish pilot tunnels because they do not need them (mesh same-node traffic uses the localhost fast path), so the call always returned false and the diagnostic flipped to state: 'tunnel down' even on a working local route. The route detail sheet and node card consequently rendered every local mesh alias as a destructive 'tunnel down' pill. Added a private isMeshReachable helper that returns true for any local-type node and otherwise delegates to hasActiveTunnel. Used it in both getRouteDiagnostic and getStatus, replacing the inline ternary in getStatus that already had the right shape but lived as duplicated logic. The helper falls back to false when the node row is missing (orphaned alias from a partial cascade), matching the pre-existing conservative behavior. Verified live: GET /api/mesh/aliases/echo.audit-mesh-prod.Local.sencho/diagnostic previously returned state='tunnel down' on a healthy local route.	2026-05-08 11:38:35 -04:00
Anso	e176ae9f17	fix(mesh): inspect remote stacks via HTTP proxy not Dockerode (#992) ... MeshService.inspectStackServices used DockerController.getInstance(nodeId) to enumerate compose-labeled containers, but NodeRegistry.getDocker explicitly throws for any node with type='remote' by design. The throw was silently caught and an empty array returned, so opt-in for any remote node failed with a misleading "no running services" error and refreshAliasCache only ever populated LOCAL aliases. Cross-node mesh routing was therefore impossible end-to-end regardless of pilot-tunnel state. Split the inspector. inspectLocalStackServices keeps the Dockerode listContainers path and always queries the local Docker daemon; it is public so the new route can call it. inspectStackServices is now a dispatcher: local nodes fall through to the Dockerode path, remote nodes (proxy mode and pilot-agent) HTTP-fetch /api/mesh/local-services/:stackName against the URL resolved by NodeRegistry.getProxyTarget, with the persisted node_proxy Bearer token and license tier headers attached. The remote's MeshService enumerates its own LOCAL Docker daemon and returns the {service, ports[]} envelope. refreshAliasCache now inspects every opted-in stack in parallel via Promise.allSettled so a slow or unreachable remote does not stall the 60-second loop. The new /api/mesh/local-services/:stackName route is gated by requireAdmiral plus isValidStackName and always queries the caller's own Sencho instance. Together with PR #989 (proxy-side bridge dispatch) and PR #990 (agent-side loopback auth), this makes the central, bridge, agent, local-Sencho mesh control plane functional end-to-end for both proxy-mode and pilot-agent remotes.	2026-05-08 11:12:29 -04:00
Anso	a6d3e5d052	fix(pilot): inject loopback auth on agent-side HTTP/WS forwarding (#990) ... After the central proxy forwards a request through the pilot tunnel, the agent receives an http_req or ws_open frame and opens a local loopback request to its own Sencho server. The forwarder previously copied the inbound headers verbatim with no Authorization header, so the local authMiddleware rejected every proxied call with 401 Authentication required. The central cannot sign the missing token because each Sencho instance has its own auth_jwt_secret generated at first-run setup. The agent process runs in the same Node process as the local Sencho server and shares the local DatabaseService, so it can mint a fresh pilot_tunnel-scoped JWT signed with the local secret. Token TTL is 5 minutes with a refresh-on-use 4-minute cache to avoid jwt.sign on every request. The header is built once per request via buildLoopbackHeaders and applied to both onHttpReq and onWsOpen so the two forwarder paths stay in lockstep. The local authMiddleware accepts the scope through its existing pilot_tunnel branch with no special-case bypass. Together with the proxy-side change in the previous commit, this completes the central, bridge, agent, local-Sencho HTTP path for pilot-agent-mode nodes.	2026-05-08 10:00:15 -04:00
Anso	94ce7c71d2	fix(proxy): route pilot-agent HTTP via PilotTunnelBridge loopback (#989) ... The remote-node HTTP proxy resolved targets by reading nodes.api_url and nodes.api_token directly from the database. Both fields are empty for pilot-agent nodes by design, which produced a misleading 503 ("no API URL or token configured. Update it in Settings, Nodes.") for any API call targeting a healthy pilot-agent: stack creation, log retrieval, and every other resource a pilot-agent should serve. NodeRegistry.getProxyTarget already encapsulates the correct dispatch. For proxy mode it returns the persisted api_url and api_token. For pilot-agent it returns the loopback URL of the active PilotTunnelBridge with an empty token, since the bridge re-authenticates implicitly via the pre-verified tunnel socket. Switch all three lookup sites in remoteNodeProxy to this helper, cache the resolved target on req.proxyTarget so the http-proxy router and proxyReq callbacks do not re-resolve, and split the 503 message so pilot-agent operators see "Pilot tunnel to X is disconnected" instead of the proxy-mode hint.	2026-05-08 09:03:22 -04:00
Anso	c9452337e1	test(pilot): in-process simulation of mid-tunnel disconnects (#987) ... * test(pilot): in-process simulation of mid-tunnel disconnects The PR #979 hardening added per-stream idle timers, drain handling, paused-request maps, and a tcpAwaitingDrain set on the bridge. None of those cleanup paths had an end-to-end test driving them through a real disconnect. A future refactor that inverts the order of clearIdleTimer + streams.delete (or that misses one of the aux maps) would leak per-tunnel memory in a way unit tests cannot catch. Spin up a real http.Server with attachUpgrade and a real ws.WebSocket client (same self-contained pattern as pilot-tunnel-integration.test.ts). Four cases: - HTTP request mid-flight: open a real loopback HTTP request, wait for the bridge to forward http_req to the agent, terminate the agent WS, assert the loopback request resolves with 502 (the teardownStream path) and the manager marks the node Offline. - TCP stream with bytes outstanding: openTcpStream from the bridge, ack open from the test agent, write some bytes, terminate the tunnel, assert the TcpStream emits 'close' and the bridge is gone from the manager. - Reconnect with the long-lived token after a clean close does NOT bump tunnels_replaced (no live tunnel to replace). - Reconnect WHILE the prior tunnel is still live DOES bump tunnels_replaced. Inverse case proves the counter wiring is correct in both directions. Self-contained: no shared helper modules, no dependencies on other in-flight pilot test files. Can land independently of PRs #982, #983, or #985. * test(pilot): address PR E code-review on disconnect tests Two small findings: - Defensive: attach a no-op 'error' listener on the TcpStream in test #2 after the 'open' event has fired. teardownStream only emits 'error' for unaccepted streams today, so the listener is not exercised; the listener exists so a future refactor that delays accepted=true past 'open' cannot crash the worker on an unhandled 'error' event. - Removed a stale cross-file reference in the header comment that pointed at pilot-tunnel-integration.test.ts; that file lives in a separate in-flight branch and does not exist on main. No behavior change. All 4 disconnect tests pass. * chore(test): drop unused PROTOCOL_VERSION import The import was leftover from an earlier draft and ESLint flagged it as no-unused-vars (error level for symbols not prefixed with _), failing the lint job in CI.	2026-05-08 09:02:59 -04:00
Anso	3112f58a88	fix(templates): align App Store deploy gate with stack:create permission (#986) ... The POST /api/templates/deploy handler gated on requireAdmin, while every other create-stack endpoint in routes/stacks.ts uses requirePermission('stack:create'). Per the role table in middleware/permissions.ts, node-admin holds stack:create — so a node-admin could create stacks the regular way but got 403 ADMIN_REQUIRED from the App Store. The cockpit's Deploy button is gated on can('stack:create'), so the button looked enabled and the click silently failed. Swap the gate to requirePermission(req, res, 'stack:create'). Admin still passes through the global bypass; node-admin now passes via the role permissions table; deployer, viewer, and auditor stay denied. Cache-refresh on the same router keeps requireAdmin since cache invalidation has no per-resource scope. Adds backend/src/__tests__/templates-deploy-rbac.test.ts with six parameterised supertest cases (one per role plus an unauthenticated case) so the matrix is locked in. The two passing-role cases assert the request clears the gate (status !== 403, code !== PERMISSION_DENIED) without depending on Docker being available in the test environment.	2026-05-08 03:50:03 -04:00
Anso	8f13a7faf3	fix(pilot): stop silently swallowing fs errors in agent token helpers (#985) ... * fix(pilot): stop silently swallowing fs errors in agent token helpers The pilot-agent audit found that both filesystem-touching helpers in the agent process discarded every fs error class: - readPersistedToken caught all errors and returned null. ENOENT (normal first boot) and EACCES / EIO / EROFS (a real disk or volume-permission failure) were indistinguishable; the latter silently fell back to "no persisted token", and on a node where the enrollment token had already been consumed the agent would enter a re-enrollment loop with no log signal pointing at the disk. - persistToken caught all errors and logged at console.warn. The operator saw the next-boot loop with the same diagnostic gap. Replace both with errno-aware handling: - readPersistedToken calls fs.readFileSync directly (no TOCTOU race against existsSync), treats ENOENT as silent, and logs every other errno at ERROR with the path. - persistToken logs at ERROR (not WARN) with the failing errno and an explicit "next agent restart will require re-enrollment until the volume is writable" message. The agent still continues with the in-memory token so the current session is unaffected. Both helpers are now exported (marked @internal) so the new test file can mock fs and assert the error-class-vs-log-level matrix. 12 unit cases in pilot-agent-fs-errors.test.ts cover ENOENT, EACCES, EIO, EROFS, ENOSPC, missing errno, empty file, and the happy path. Mock pattern follows backend/src/__tests__/filesystem.test.ts. * fix(pilot): address code-review on the fs-error branch Three findings from the review pass: - Em dash in a test description (Directive 18). Rewritten as "does not throw, so the in-memory token stays usable for the current session". - persistToken still had an existsSync + mkdirSync pair around the token-write. mkdirSync({ recursive: true }) is idempotent on existing directories, so the existsSync was redundant and added a TOCTOU window where the directory could be removed between the probe and the write. Dropped the existsSync; the test that previously primed mockExistsSync now asserts mockExistsSync is NOT called, locking the TOCTOU removal. - The two new exports used the @internal JSDoc tag, but this repo's existing pattern for "public-by-convention-for-tests" helpers (e.g. RegistryService.ts:481) is plain prose "Exposed for unit tests." Switched to that style. No behavior change beyond the TOCTOU removal.	2026-05-08 03:48:35 -04:00
Anso	e380ee7b16	test(pilot): in-process integration test and Playwright E2E (#983) ... * test(pilot): in-process integration test for the tunnel handshake Layered unit tests cover the protocol decoder, the bridge cap, the manager, and the DB-layer enrollment lifecycle. None exercise the glue: the WebSocket dispatch order in upgradeHandler.ts, the actual hello / enroll_ack round-trip, the long-lived token swap. A regression that moves /api/pilot/tunnel below the auth gate, breaks the hello / enroll_ack ordering, or changes the token shape would not be caught today. Spin up a real http.Server with attachUpgrade wired and a real ws.WebSocket client. Three tests: - Enroll-ack happy path: connect with an enrollment token, receive hello + ctrl enroll_ack, assert the manager has registered the tunnel. - Replay rejection on the wire: connect, complete enroll-ack, close, reconnect with the same enrollment token, assert HTTP 401 from the upgrade handshake. - Long-lived token reconnect: capture the enroll_ack token, close, reconnect with the long-lived pilot_tunnel JWT, assert hello but no enroll_ack and the tunnel is registered. The test does not mock the agent side beyond the framing protocol. The goal is to confirm the wires connect end-to-end, not to drive a full request round-trip (which would require a synthetic agent bridge and adds little marginal coverage over the existing bridge unit tests). This file also establishes the in-process WS-pair test pattern for any future WebSocket integration work; no such pattern existed in the suite before. * test(e2e): cover operator-side pilot-agent enrollment in Playwright Backend integration is covered by pilot-tunnel-integration.test.ts and the vitest enrollment suites. The browser-only surfaces (mode selector default, enrollment dialog, docker run code block, regenerate affordance on an existing pilot-mode node) had no automated coverage; a typo in the mode label, a styling regression that hid the docker-run code, or a backend response shape change that broke the rendered command would all ship unchecked today. Two specs reusing the existing loginAs helper and the same Settings -> Nodes navigation pattern from nodes.spec.ts: - Create flow: open Add Node, switch type to Remote, confirm pilot mode is the default (api_url field stays absent), submit, assert the enrollment dialog renders a docker run command containing SENCHO_MODE=pilot, SENCHO_PRIMARY_URL=, and a JWT-shaped SENCHO_ENROLL_TOKEN=. Cleanup deletes the row. - Regenerate flow: create a pilot node, capture the first token, open the row's edit dialog, click Regenerate enrollment token, assert the second token differs from the first. Cleanup deletes the row. Out of scope for this E2E: simulating an agent connecting to flip the row to Online. The integration test covers the wire side; this spec keeps focus on the operator-visible UI. Each test name-suffixes with Date.now() so re-runs do not collide on the UNIQUE name constraint and so leftover rows from a partial run get distinct names instead of stacking on the same row. * fix(pilot): address PR B code-review findings Three code-review findings: - High: NodeManager row action buttons were icon-only with no accessible name. Added aria-label="Edit node", aria-label="Delete node", aria-label="Test connection" so the Playwright E2E can target them by role/name (currently the accessible name is rendered into a Radix tooltip portal that Playwright cannot resolve as the button's name). - High: Replaced UI-driven cleanup in the E2E with API-based deletion via page.request, run in both beforeEach (sweep leftovers) and afterEach (sweep this test's row even on failure). Targets every node whose name starts with the test prefix so a crashed previous run cannot affect the next. - Medium: Replaced two fixed 50ms sleeps in the integration test with vi.waitFor polling on hasActiveTunnel(nodeId) === false, eliminating the race between the WS close hop chain and the next test step on slow CI runners. Plus two cleanup items: - The integration test's afterAll now removes 'tunnel-up' / 'tunnel-down' listeners from the manager singleton so this file's runs do not leak listeners into other test files in the same Vitest worker. - Tightened the WS error swallowing in the rejection-path test: only swallow the expected "Unexpected server response" error shape; surface anything else (ECONNREFUSED etc.) instead of silently masking it. No em dashes added (Directive 18 verified clean).	2026-05-07 23:51:52 -04:00