mirror of
https://github.com/Studio-Saelix/sencho.git
synced 2026-07-29 21:29:18 +00:00
v0.83.1
406 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Anso
|
e05099f2a1 |
fix(fleet-sync): make control-identity-mismatch sticky and surface in UI (#1117)
Treat 409 CONTROL_IDENTITY_MISMATCH from a replica as a non-retriable
failure instead of looping the same 409 through the 5-minute retry
service forever and silently writing identical failure rows.
Backend
- DatabaseService: add `sticky_error_code`, `sticky_error_expected`,
`sticky_error_got` columns to `fleet_sync_status` via an idempotent
migration. New methods setFleetSyncSticky, getFleetSyncStickyCode,
clearFleetSyncStickyForNode. recordFleetSyncSuccess clears the sticky
flag on a clean push. getFailedSyncTargets SQL adds
`AND sticky_error_code IS NULL` so the retry loop skips sticky rows.
- FleetSyncService.executePushToNode: short-circuits at the top when
sticky is set (covers event-driven pushResourceAsync calls). On a 409
with code CONTROL_IDENTITY_MISMATCH, records the failure once and
pins sticky with the expected/got fingerprints carried in the 409 body.
- routes/nodes.ts: new POST /api/nodes/:id/fleet-sync/reset-anchor.
Admin + paid + node:manage. Proxies POST /api/fleet/role/reanchor to
the peer with `{override:true}` using the stored Bearer node_proxy
token. On peer 200, clears every sticky row for the node so the next
push re-anchors and resumes replication. Distinct 502 / 504 responses
for peer-rejected / peer-unreachable so the UI can show a useful toast.
Frontend
- New lib/fleetSyncApi.ts + hooks/useFleetSyncStatus.ts. Polling hook
(30s visibilityInterval) skips fetch when !isPaid.
- NodeManager.tsx: destructive banner per affected node listing both
fingerprints, with `Reset anchor on peer` and `Remove node` buttons.
Hidden for community-tier users via empty hook data.
- FleetConfiguration.tsx (Fleet -> Status): read-only `Policy sync`
SummaryRow per remote node card. In sync / degraded / paused with
a tooltip; no action buttons (the action lives in NodeManager).
Tests
- fleet-sync-service.test.ts: 4 new cases for sticky-set on first
mismatch, short-circuit on subsequent pushes, null fingerprints,
and non-mismatch failures not setting sticky.
- database-fleet-sync-sticky.test.ts (new): 6 cases pinning the DB
contract incl. retry-loop SQL filter and migration idempotency.
- nodes-fleet-sync-reset-anchor.test.ts (new): 6 cases covering
happy path, peer 401 -> 502, peer unreachable -> 504, local-node
rejection, unknown node id, and community-tier 403.
Gate parity (Directive 30): the new POST .../reset-anchor enforces
requireAdmin + requirePaid + node:manage (matches the existing read at
GET /api/fleet/sync-status). UI banner + SummaryRow only render when the
hook returns data, which it only does for paid-tier authed users. No
existing tier-gate file moved; this is greenfield parity.
Auth audit: the peer's POST /api/fleet/role/reanchor route already uses
requireAdmin, which accepts the central's stored node_proxy Bearer
token because authMiddleware maps `scope === 'node_proxy'` to
`req.user = { username: 'node-proxy', role: 'admin', userId: 0 }`.
No widening required.
Backend tsc clean. Frontend tsc -b clean. 59 fleet-sync tests pass; full
backend suite green minus the pre-existing Windows-only file-lock flake
on filesystem-backup.test.ts that reproduces unchanged on main.
|
||
|
Anso
|
6722335a79 |
fix(stack-update): refresh frontend state automatically after a stack update (#1113)
After applying a stack update the sidebar's "update available" dot stayed visible and the stack's status indicator was stuck on the optimistic value until the page was manually refreshed. Two root causes: 1. Image-updates state refresh was a fire-and-forget call in some paths and entirely missing from the bulk-update, auto-update, and state-invalidate WebSocket-handler paths. 2. stackActionsRef.current was resynced only at render time, so the post- update refreshStacks(true) running in the action's finally block read a stale "busy" map and preserved the optimistic mask via prev[file] ?? status. Backend now broadcasts a state-invalidate event with scope='image-updates' and action='stack-updated' after every successful update (single-stack route and auto-update loop). The frontend useNotifications hook routes this to a new onImageUpdatesChange callback wired to fetchImageUpdates in EditorLayout, so every connected client refreshes the dot through the same code path. Bulk update also calls fetchImageUpdates directly for fast local feedback, and setStackAction/clearStackAction now keep stackActionsRef synchronously in sync with state so the busy-stack check inside refreshStacks observes the cleared map immediately. Adds 3 unit tests covering the new WS branch (positive, scope-mismatch negative, auto-update-settings-changed negative). |
||
|
Anso
|
9dbce9c3c7 |
fix(spawn): attribute ENOMEM and ENOENT-under-memory-pressure spawn failures to host OOM (#1111)
Operators previously saw "spawn docker ENOENT" or "spawn /bin/sh ENOENT" when the host was under memory pressure, which sent them down a missing-binary debugging path. Linux libuv's posix_spawn can fail to allocate its argv / path-search arena under low free memory and surface the underlying ENOMEM as ENOENT. Centralizes spawn-error mapping in a new utils/spawnErrors.ts helper: - Explicit ENOMEM is rewritten to "Out of memory while launching <command> (host free memory: X MiB of Y MiB)". - ENOENT under the 128 MiB free-memory floor is rewritten with the same wording plus a "reported as ENOENT under memory pressure" hint. - ENOENT for docker on a healthy host preserves the existing "Docker CLI unavailable on this node" mapping. - Other errors pass through unchanged. Applied at the four named offenders: ComposeService.execute(), ComposeService.captureCompose(), DockerController.getContainersByStack(), and FileSystemService.getStacks() (which gets an ENOMEM-aware log line for the scandir failure). Startup also logs host free/total MiB once and warns when free memory is below the 128 MiB floor, so the diagnostic surfaces before the first spawn attempt rather than after it fails. 37 tests cover the mapping function directly and the ComposeService / FileSystemService integration paths. |
||
|
Anso
|
523ba5854c |
fix(stacks): return 404 for nonexistent stacks on deploy/down/update (F-7) (#1108)
POST /api/stacks/:name/{deploy,down,update} previously returned HTTP 500
with body {"error":"spawn docker ENOENT"} when invoked against a stack
whose compose directory was missing. The status code was wrong (the
named resource did not exist, so 404 is the right answer) and the
message misled operators into thinking the docker CLI was unavailable.
Add a small requireStackExists(nodeId, stackName, res) helper in
routes/stacks.ts that validates the stack name and confirms a compose
file is present via FileSystemService.hasComposeFile before any of the
three handlers spawn docker compose. The helper is called immediately
after requirePermission and before runPolicyGate so unauthorized
callers still get 403 first and the policy gate never runs against a
phantom stack.
In ComposeService.execute(), narrow the child.on('error') handler so
the genuine docker-binary-missing case (ENOENT on the spawn itself)
rejects with "Docker CLI unavailable on this node" instead of the raw
"spawn docker ENOENT". This is defense in depth for the rare case the
pre-check cannot cover, and it fixes the misleading-message half of
the bug as well.
Cover the new contract with stack-actions-missing-stack.test.ts (four
cases: deploy/down/update return 404, invalid name returns 400). Mock
ComposeService as a tripwire so a future code path that bypasses the
guard would fail loudly. Fix stacks-failure-notifications.test.ts by
adding hasComposeFile to its FileSystemService partial mock so the
existing happy-path-error-handling cases continue to flow into
ComposeService.
|
||
|
Anso
|
1f673073ca |
feat(fleet): add fleet-wide Docker prune to Fleet Actions (#1104)
Adds a third card to the Fleet Actions tab that fans out Docker prune (images, volumes, networks) across every node in one submit. Local nodes call DockerController under a bulk-prune lock; remote nodes receive one POST /api/system/prune/system per target. Per-node + per-target results with reclaimed bytes are surfaced inline via ResultsList. Tier: Skipper / Admiral (requirePaid + requireAdmin), matching the rest of Fleet Actions. The frontend card is mounted inside the existing isPaid branch at FleetActionsTab; no new frontend gate is required. The card uses an amber accent rail and the Eraser icon so it reads as 'cleanup' rather than 'destructive stop'. Scope toggle defaults to Managed only (Sencho-tagged resources) with an All unused option that escalates the destructive-confirm copy. |
||
|
Anso
|
0cffd60173 |
fix(mesh): clean up activeStreams on src socket close/error (F-10) (#1105)
The cross-node MeshService.openCrossNode handler attached src.on('close')
and src.on('error') listeners that only called tcpStream.destroy() and
never deleted the activeStreams Map entry. MeshTcpStreamLike.destroy()
sends a tcp_close frame to the remote but does not synchronously emit
the local close event; that fires only when the remote sends back a
tcp_close ack via _dispatchClose (or when the entire tunnel tears down
and the bridge force-emits close on every stream).
Failed dials whose remote ack was lost (peer gone, network drop, or the
F-9-era timeouts) therefore leaked records into activeStreams until the
tunnel itself idle-closed, producing the monotonically-climbing
activeStreamCount documented in the mesh E2E audit (F-10).
Add a cleanupRecord() closure that clears the open timer and deletes
the record idempotently, then route all four lifecycle handlers
(tcpStream.on('error'), tcpStream.on('close'), src.on('close'),
src.on('error')) through it. Map.delete is naturally idempotent so the
double-delete that fires when both sides close normally is a free
no-op. The same-node openSameNode path already had this shape via its
existing teardown() closure; this brings the cross-node path to parity.
Three new Vitest cases in mesh-service.test.ts use an EventEmitter-
backed fake src so emit('close') actually delivers to the listener
(the existing makeFakeSocket uses vi.fn for .on which records calls
but never fires them). They cover src.close, src.error, and the
idempotency case where both src and tcpStream emit close in sequence.
|
||
|
Anso
|
c460bb87a8 |
fix(mesh): probe upstream synchronously in route diagnostic (F-11) (#1100)
GET /api/mesh/aliases/:alias/diagnostic returned a cached state derived from the last latency/error maps. Those maps only mutated when someone called POST .../test or when a cross-node connect logged an event, so once an upstream stopped the diagnostic kept reporting "healthy" until the 60 s alias-cache refresh pruned the alias entirely. The GET now calls testUpstream synchronously after the alias-resolved, opted-in, tunnel-up short-circuits. Probe failures land in routeErrorMap via logActivity (cross-node already did this; same-node timeout/error paths now log probe.fail in the same shape), so the state computation flips to "unreachable" on the same request that exposed the stopped upstream. A new routeProbeAtMap stamps freshness and surfaces in the response as lastProbeAt; the route detail sheet renders "Last probe <age> · <ms>" using the existing formatTimeAgo helper. |
||
|
Anso
|
554f662563 |
fix(mesh): surface stopped-stack opt-ins on routing node cards (#1098)
Add `currentlyResolvable: boolean` per entry in `MeshNodeStatus.optedInStacks`, derived from the existing alias cache so the new field stays consistent with `/api/mesh/aliases` without any extra Dockerode or cross-node inspect on the status path. The Routing tab renders an amber `suspended` pill for entries whose stack is opted in but currently has no running services, plus a single explanatory caption below the suspended list. Resolves the contradictory `Mesh stacks: 1 / Aliases: 0 / No mesh services on this node yet` copy on the node card when a meshed stack's container has been stopped; the misleading line is now only shown when the node truly has no opt-ins. The opt-in itself remains sticky: when the stack starts again, its aliases reappear automatically on the next refresh. Defensive de-dup in the UI filters suspended entries against the live alias snapshot to handle the transient gap where `/mesh/status` and `/mesh/aliases` return slightly inconsistent views from their separate fetches. Tests: - new `mesh-status-resolvability.test.ts` (6 cases) locks the resolvable / suspended / mixed / empty / per-node-scoping / stale-alias-no-phantom invariants for `getStatus`. - `mesh-topology-layout.test.ts` gains a `stacksKey` resolvability-flip case and a `meshNodeStateEqual` case asserting a resolvability flip on an otherwise identical stack registers as a state change so the topology layout re-runs. Operator docs gain one new troubleshooting accordion in `/docs/features/sencho-mesh.mdx` explaining the suspended state. |
||
|
Anso
|
9f238e187c |
fix(mesh): cascade opt-out when a stack is deleted (F-1 / F-14) (#1096)
DELETE /api/stacks/:name now calls MeshService.optOutStack after the DB cleanup so the mesh_stacks row, the override file under <DATA_DIR>/mesh/overrides/<nodeId>/, and any derived aliases do not outlive the deleted stack. Pre-fix, those artifacts leaked and the reconcile loop logged "No compose file found for stack" every tick. optOutStack is idempotent (early return when the stack was never opted in) and already cascades override-regen plus recompose across the rest of the fleet, so peers' /etc/hosts drop the dropped alias. The new cascade call is best-effort relative to the delete itself: mesh cleanup failures warn but never regress the delete contract. New regression test asserts three contracts: cascade on delete, no-op on never-meshed, and 200 + warn when the cascade rejects. |
||
|
Anso
|
f6e42535c8 |
fix(mesh): route peer→central traffic over the existing forward WS (#1094)
* fix(mesh): route peer→central traffic over the existing forward WS The reverse mesh callback path (`/api/mesh/proxy-tunnel-from-peer`) needed SENCHO_PRIMARY_URL on central plus a publicly reachable origin from the peer's perspective. In a typical homelab where central sits behind NAT, peer→central dispatch silently failed at the dialer's short-circuit and the headline "call any service on any node by hostname" worked one way only. The forward WS at `/api/mesh/proxy-tunnel` is already bidirectional end to end. Make the bridge a persistent control-plane primitive: dial every mesh-enabled proxy peer at startup, reconcile every 60 s, never idle-close. Peer→central traffic multiplexes over the same WS via `tcp_open_reverse`. Removed: - `meshProxyTunnelFromPeer.ts` WS handler and dispatch - `MeshCentralRegistry`, `PeerToCentralMeshSessionDialer` - `mesh_handshake` first-frame state machine in `meshProxyTunnel.ts` - `maybeSendBootstrap`, `buildHandshakeFrame` in the dialer - `mesh_proxy_callback_bootstrap` capability and `maybeWarnUnsetPrimaryUrl` - `mesh_centrals` table (drop migration; greenfield, no users) - `PilotTunnelManager.replaceOrRegisterProxyBridge` (dead after handler removal) - twelve associated unit/integration tests plus the peer-recovery branch in `MeshService.openCrossNode` Added: - `MeshService.proactiveBridgeFanout` selects every mesh-enabled proxy peer (no longer gated on `mesh_stacks` rows) - `startBridgeReconcileLoop` runs the fanout every 60 s (override via `SENCHO_MESH_RECONCILE_INTERVAL_MS`) - `MeshProxyTunnelDialer` default idle TTL is now `0` and exposes `isDialing(nodeId)` for the status surface - `MeshNodeStatus.reverseCallbackStatus` discriminator (`connected | connecting | unavailable | not_applicable`) surfaced via `/api/mesh/status` and rendered as a pill in the Routing tab - `openCrossNode` error message distinguishes "no proxy target" from "waiting for central to dial the reverse bridge" - New tests: `mesh-service-proxy-tunnel-reconcile`, `mesh-status-reverse-callback`, `mesh-proxy-tunnel-dialer-no-idle-close` SENCHO_PRIMARY_URL is no longer required for any mesh function. * fix(mesh): rewrite proxy-tunnel reconcile test contents The previous commit renamed the file but the rewritten test bodies stayed unstaged on top of the rename. This commit lands the actual rewrite: the fanout assertion now requires every mesh-enabled proxy peer to be dialed, not just those with `mesh_stacks` rows, and adds a reconcile-tick repeated-call test. |
||
|
Anso
|
ed49ed6165 |
fix(http): strip zstd from Accept-Encoding so compression sets Content-Encoding (#1092)
compression@1.8.1's negotiator silently fails to set the Content-Encoding response header when Accept-Encoding carries an unknown token like zstd, even though the body is still compressed with brotli. Chromium 123+ sends "Accept-Encoding: zstd, gzip, br" by default, including Playwright's bundled Chromium over plain HTTP, so any homelab Sencho viewed without a TLS terminator triggered ERR_CONTENT_DECODING_FAILED in the browser and rendered as a blank page. Insert a tiny middleware (step 4 of the canonical pipeline) that drops zstd tokens from Accept-Encoding before compression's negotiator runs. The negotiator then picks br or gzip and writes the matching header. The fix is no-op when zstd is absent. If only zstd was offered, the header falls back to identity so the response is served uncompressed. Renumber the canonical-order JSDoc in app.ts from 17 to 18 steps and update the step-number references in hub-only-guard and proxy-mount-order test docstrings to match. 11 new tests (8 unit + 3 supertest integration against real compression with a 3000-byte body) lock in the behavior: zstd-stripping across casing and q-values, surviving tokens preserved verbatim, identity fallback when only zstd was offered, Content-Encoding header always written when at least one supported codec remains. |
||
|
Anso
|
1d7418a233 |
fix(mesh): recompose previously-meshed containers when alias set changes (#1090)
When a stack is opted into the mesh, every previously-meshed stack's override.yml on disk gains the new alias, but the running containers keep their stale /etc/hosts because extra_hosts is read at container creation time. Cross-stack DNS for the new alias silently fails until an operator manually redeploys each prior stack. Complete the cascade: after regenerateOverridesAcrossFleet finishes, fan out triggerRedeploy across every (node_id, stack_name) tuple in mesh_stacks, skipping the just-opted-in pair (the explicit triggerRedeploy at the end of optInStack covers it). Mirror the same cascade in optOutStack so the dropped alias exits every container's /etc/hosts. triggerRedeploy already dispatches local vs remote, already wraps runRedeploy in fire-and-forget error logging to the mesh activity ring and audit log, and already enforces compose policy gates. The cascade just reuses it. regenerateAllOverrides (boot and manual /regen-overrides) is intentionally NOT covered: a Sencho restart must not force a fleet-wide recompose; override files alone are sufficient there. Adds four unit tests in mesh-service.test.ts: cascade fires for every prior meshed stack on opt-in, no double-redeploy of the just-opted-in tuple, cascade is a no-op when no peers exist, cascade fires for every survivor on opt-out. |
||
|
Anso
|
578cac89da |
fix(mesh): surface data-plane failures in health, meta, and Routing tab (#1088)
The three previously-silent console.warn paths in MeshService.setupMeshNetwork now route through a typed recordSetupFailure helper that classifies the failure (subnet_invalid, subnet_overlap, subnet_mismatch, ip_in_use, attach_failed, not_in_docker), emits a mesh.disable activity entry at the matching level (error for real failures, warn for the expected dev-mode not_in_docker case), and strips mesh_proxy_callback_bootstrap from advertised capabilities via CapabilityRegistry. /api/health gains a mesh.dataPlane block carrying the typed status. /api/mesh/status carries localDataPlane at the top level so the Routing tab renders a red banner with an operator-actionable recovery hint (set SENCHO_MESH_SUBNET to a free /24 and recreate the container) when the data plane is down. The success path re-enables the capability and flips the status to ok. Generalizes the previously-documented F-0 failure mode (IP-in-subnet collision) to also cover the subnet-pool-overlap case where another Docker bridge on the host already owns the requested CIDR. |
||
|
Anso
|
a318e6b3c1 |
fix(mesh): close data-plane race that dropped early TcpData frames (#1086)
The tcp_open and tcp_open_reverse receivers registered their stream entry only after awaiting resolveTarget / resolveContainerIp. The peer, which is free to send TcpData immediately after the open frame, hit the lookup-miss path in handleBinaryFrame and the bytes were silently dropped. Probes passed because they only exercise the handshake; real HTTP hung with 0 bytes received. Reserve the stream entry synchronously, buffer early TcpData in a per-stream pendingData queue capped at 1 MiB, and flush onto the local socket inside the connect handler before sending tcp_open_ack. The payload is copied because decodeBinaryFrame returns a subarray view of the WS receive buffer; holding the view would pin the parent buffer past its lifecycle. Both sides of the data plane are patched: - tcpStreamSwitchboard.onTcpOpen (forward, agent + proxy-mode peer) - PilotTunnelBridge.handleTcpOpenReverse / acceptReverseLocal (reverse, primary acting as the local dial target) Adds unit coverage for the race window and for the overflow path. The cap constant lives in pilot/protocol.ts alongside the other per-stream limits. |
||
|
Anso
|
50b89db3b8 |
chore(mesh): drop mesh_stacks table on pilot-mode DBs (#1085)
Per the C-3 design, mesh state lives on central. Pilots learn aliases via the D-1 override push and hold them in `pilotAliasOverlay`. The local `mesh_stacks` table on pilots has been dead since C-3 shipped. This is a greenfield cleanup (Directive 20): - `migrateMeshTables()` runs `DROP TABLE IF EXISTS mesh_stacks` when `SENCHO_MODE === 'pilot'` and skips the CREATE. Central-mode behavior is unchanged. - The four mesh CRUD methods (`listMeshStacks`, `isMeshStackEnabled`, `insertMeshStack`, `deleteMeshStack`) short-circuit on the same check. Reads early-return empty/false; writes early-return; `insertMeshStack` also warns so an accidental pilot-side write is loud. - New `isPilotMode()` helper mirrors the existing one in `bootstrap/startup.ts`. Layering rules forbid the shared import; a third call-site would justify extraction to `helpers/`. Operator-visible behavior is unchanged on either side. On central, the table and its rows are untouched. On pilots, the table is removed (it held no rows in steady state) and any caller of the CRUD methods continues to see empty list / false / no-op without touching SQLite. Test plan - `npx tsc --noEmit`: clean. - New `database-service-pilot-mesh-stacks.test.ts`: 5/5 green (table absence + four short-circuit assertions). - `mesh-service.test.ts`, `mesh-diagnostic-local.test.ts`, and `mesh-service-proactive-bootstrap-fanout.test.ts`: 58/58 green. - Full backend suite: 2280/2281; the single failure is a pre-existing Windows EBUSY flake in `filesystem-backup.test.ts` that reproduces on a clean tree. - Code reviewed; findings applied. |
||
|
Anso
|
5bc4add099 |
refactor(mesh): retarget proxyFetch null-target throw to no_target (#1083)
`MeshService.proxyFetch` previously threw `MeshError('push_failed', ...)`
when `NodeRegistry.getProxyTarget(nodeId)` returned null, but semantically
no push was attempted: the target simply did not exist. Callers had to
special-case the code and explain the overload in a comment.
Retarget the throw to the existing `'no_target'` variant in
`MeshErrorCode`. `listStacksOnNode` matches the new code directly; the
explanatory comment is gone since the code reads for itself.
`MeshError('push_failed', ...)` continues to signal real outbound push
failures: `applyLocalOverride` (mesh data plane not yet ready on the
receiving node) and `pushOverrideToNode` (HTTP 404 from an outdated
remote, non-OK HTTP response). Those sites are unchanged.
Operator-visible behavior is unchanged in the steady state. The rare
"remote disappeared between opt-in start and override push" case now
returns HTTP 400 with `code: 'no_target'` instead of 503; the response
body carries a clear message ("no proxy target for node N") so clients
can disambiguate.
Test plan
- `npx tsc --noEmit`: clean.
- `npx vitest run src/__tests__/mesh-list-stacks-remote.test.ts`: 8/8 green
(6 existing + 2 new cases pinning the catch-arm semantics).
- `npx vitest run src/__tests__/mesh-service.test.ts src/__tests__/mesh-inspect-remote.test.ts`:
56/56 green.
- Code reviewed; findings applied.
Notes
- PR #1082 (refactor: route inspectStackServices through proxyFetch) needs
a rebase + one-line update to its catch arm before merging on top of
this change.
|
||
|
Anso
|
dcc43205e0 |
refactor(mesh): route inspectStackServices through proxyFetch (#1082)
inspectStackServices was constructing its Authorization, x-sencho-tier,
and x-sencho-variant headers inline. proxyFetch is the centralized
helper used by listStacksOnNode, pushOverrideToNode, triggerRemeshRedeploy,
and removeOverrideFromNode. Equivalent today for a GET-no-body call, but
any future header added to proxyFetch (audit context, license fingerprint,
request id) would silently miss inspectStackServices.
Wire output is byte-identical: same URL, same method, same three headers,
same 5 s timeout. The new catch arm converts MeshError('push_failed') back
to the existing warn + return [] contract callers (optInStack,
refreshAliasCache) rely on. Pattern mirrors listStacksOnNode.
|
||
|
Anso
|
a32183d198 |
fix(mesh): cascade override regen across all meshed nodes on opt-in/opt-out (#1079)
Opting a stack into mesh on node A now propagates the new alias to every meshed node's override file, not just stacks on node A. Same on opt-out. Pre-fix, optInStack and optOutStack called regenerateOverridesForNode which only walks db.listMeshStacks(nodeId). Other meshed stacks on other nodes did not learn about the new alias until something else re-pushed their overrides; operators worked around it with a manual POST /api/mesh/regen-overrides plus a redeploy of each affected stack. New regenerateOverridesAcrossFleet helper iterates db.listMeshStacks() (no arg, fleet-wide) under Promise.allSettled, skipping the (nodeId, stackName) tuple opt-in just pushed loudly. Per-stack failures surface as forwarder.error activity events, matching the pattern used by regenerateAllOverrides. Offline remote nodes leave stale overrides until the next opt-in/opt-out, the next tunnel reconnect, or a manual rerun of the regen-overrides endpoint. regenerateOverridesForNode stays for the tunnel-up retry listener: a reconnecting pilot only needs its own stacks re-pushed, not the fleet. Tests: redirect five existing spies in mesh-service.test.ts to the new method; add two new cases asserting cross-node cascade for opt-in and opt-out. 52 tests in mesh-service.test.ts pass. |
||
|
Anso
|
6b728599c4 |
fix(mesh): persist PilotMetrics counters across central restart (#1078)
The 12 PilotMetrics counters (proxy_dials_failed, proxy_bridges_total, proxy_idle_closes, the pilot-tunnel counters, and the peer-to-central callback counters) live in a singleton holding scalar fields, so process restart wipes them. Operators investigating rotation- or dial-failure trends across a central restart lost the aggregate metric. Persist the snapshot under a pilot_metrics_counters JSON row on the existing system_state KV table. Hydrate via PilotMetrics.load() in bootstrap/startup.ts before any service that increments runs; final-flush in bootstrap/shutdown.ts before the SQLite handle closes. Between those boundaries a buffered flush mirrors the audit-log pattern: every 1 s or every 100 increments, whichever fires first. Hot increment path stays a single property write plus a pending-counter check. Defensive parse on the row: object check + numeric filter, so an operator hand-edit or schema drift across releases degrades to zero state with a warn rather than crashing boot. Schema additions back-fill missing keys with zero so a release that adds a counter does not need a migration. Closes F-R-4 from the mesh tracker. |
||
|
Anso
|
9aaa8573c0 |
fix(mesh): gate Trigger 2 re-bootstrap on actual api_token change (#1076)
The PUT /api/nodes/:id handler closed and re-dialed the mesh callback bridge on every save that included api_token in the body, even when the token was unchanged. The frontend always sends the full formData on Save, so renames and compose_dir edits against a mesh-enabled proxy remote produced a wasted closeBridge + ensureBridge round-trip and a spurious manager_rejected entry in the activity log. Gate the re-bootstrap on a real value diff against the persisted token. Adds an existing-node lookup (returns 404 on missing id, which the handler previously lacked) so the comparison has the pre-update value. Reorders the guards so resource-not-found beats payload validation. Adds one integration test for the same-token path; the existing three trigger-2 cases continue to assert close + ensure firing on a real rotation, no-token-in-payload, and mesh-disabled. |
||
|
Anso
|
7e3e22138b |
fix(mesh): name actual bridge kind in proxy-bridge rejection (#1075)
* fix(mesh): distinguish proxy bridge from pilot tunnel in registerProxyBridge rejection When central's MeshProxyTunnelDialer dials a node whose slot is already held by a peer-initiated proxy bridge, the registration is correctly refused, but the activity-log message read "pilot tunnel already registered for node N; proxy bridge refused" regardless of which kind of bridge was actually in the slot. No pilot tunnel exists in proxy-mode remotes by definition, so the wording pointed operators at the wrong subsystem. Use the existing bridgeKinds index (already consulted correctly by the sister method replaceOrRegisterProxyBridge) to emit a kind-accurate message: "proxy bridge already registered for node N; concurrent dial refused" when the existing bridge is a peer-initiated proxy bridge, keeping the original "pilot tunnel" wording for the pilot-agent case so existing log filters and the sister test still match. Pure cosmetic; the rejection behavior is unchanged. New regression test asserts the discriminator in both directions. Also tightens the test-fixture beforeEach to clear bridgeKinds alongside bridges so future tests cannot read a stale kind. * chore: ignore local trailer directory in git repository * refactor(mesh): centralize bridge-kind type and conflict-message formatter Followup tidy on PilotTunnelManager after /simplify flagged two related items in the F-R-2 change: - Extract `BridgeKind = 'pilot' | 'proxy'` as a module-scoped type alias so the kind taxonomy lives in one place. `bridgeKinds` and `injectBridgeForTest` now consume it; the inline union literal is gone. - Extract `formatBridgeConflict(nodeId, existingKind)` as a private helper. Both `registerProxyBridge` and `replaceOrRegisterProxyBridge` call it from their "slot already held" rejection paths, removing the byte-identical "pilot tunnel already registered for node N; proxy bridge refused" string copy across the two sites. - Trim the 6-line WHAT comment above the rejection throw to a 2-line WHY so the call site stays scannable. No behavior change; the three relevant test suites stay green. |
||
|
Anso
|
3a20105a33 |
fix(mesh): persist rotated handshake when peer bridge owns reverse-dialer slot (#1073)
When the proxy-mode mesh peer already holds an open peer-initiated callback bridge to central, central's Trigger 2 dial (api_token rotation) arrives at the peer's `/api/mesh/proxy-tunnel` handler with a fresh `mesh_handshake` frame. The peer's `attachSwitchboard` refuses the new WS at the reverse-dialer CAS swap, but it was also closing the connection before its `'message'` listener had been attached, so the handshake frame went unread and the peer's cached callback JWT stayed at the pre-rotation fingerprint until a central restart. Restructures `attachSwitchboard` to attach the `'message'`, `'close'`, and `'error'` listeners synchronously after the switchboard is created, before the async MeshService import and the CAS swap run. On the CAS-fail branch, holds the WS open for up to FIRST_FRAME_WAIT_MS (30ms, well above localhost RTT and below operator perception) before sending the 1013 close, so an in-flight first frame lands in `onMessage` and the bootstrap material is persisted via `MeshCentralRegistry.upsert` regardless of bridge fate. The success path is untouched; reverseDialer install timing on accepted dials is unchanged. Adds a regression test that pre-installs a stub reverse dialer (simulating a peer-initiated bridge owning the slot), sends a `mesh_handshake` frame on the new WS, and asserts the registry receives the rotated material before the 1013 close fires. |
||
|
Anso
|
0b4cf90bf1 |
fix(mesh): peer-initiated callback bridge installs reverse dialer (#1071)
* fix(mesh): peer-initiated callback bridge installs reverse dialer
The R1-A2 symmetric-dial work added a peer-side dialer that opens a
callback WS to central at `/api/mesh/proxy-tunnel-from-peer`. The WS
upgrade, JWT validation, and `mesh_centrals` persistence all worked,
but the peer-side handler put the wrong object on its end of the pipe:
a `PilotTunnelBridge` (the central-role object) rather than a
`TcpStreamSwitchboard` with a registered `reverseDialer`.
The design doc states: "Central retains PilotTunnelBridge ownership;
peer retains TcpStreamSwitchboard + reverseDialer ownership." The
central-initiated handler at `meshProxyTunnel.ts:115-163` honors this.
The peer-initiated handler at `PeerToCentralMeshSessionDialer.attachBridge`
did not. It created a `PilotTunnelBridge(0, ws)` and stored it in a
private `currentSession` field. Result: `MeshService.reverseDialer`
stayed null, `dialMeshTcpStream` fell through to
`PilotTunnelManager.ensureBridge(target.nodeId)`, and `NodeRegistry.
getProxyTarget` on the peer returned null because proxy-mode peers
do not enroll their central. Cross-fleet dispatch from a peer
container failed with `proxy-tunnel.open.fail nodeId=<central>
reason=no_target` even though the callback bridge was up and healthy
(`bridgeOpen: true`, `last_used_at` populated).
Replace the peer-side `PilotTunnelBridge` with the same wiring the
central-initiated handler uses:
- `attachTcpStreamSwitchboard` with `resolveByComposeLabels` (allows
inbound `tcp_open` from central if it ever uses the callback
bridge for a reverse-direction dispatch; matches the central-
initiated pipe's resolver).
- A `SwitchboardReverseDialer` that delegates `openMeshTcpStream`
to `switchboard.openReverseStream`.
- `MeshService.setReverseDialer(localDialer, null)` with the same
CAS guard the central-initiated handler uses, so a concurrent
central-initiated tunnel does not get silently overwritten.
- `ws.on('message', onMessage)` dispatches JSON / binary frames to
the switchboard.
- `ws.once('close'/'error', teardown)` clears the switchboard, the
reverseDialer, and the `currentSession` reference idempotently.
The `currentSession` field type changes from `PilotTunnelBridge | null`
to `TcpStreamSwitchboard | null`. Callers in `MeshService.openCrossNode`
only use the return value for truthiness; the cast is harmless. The
public `hasSession()` signature is unchanged.
A new `currentWs` field holds the WS reference so `resetForTest` can
close the connection on teardown (the switchboard's own `cleanup`
does not close its WS).
Test update mirrors the production wiring: the existing
"marks the row used on successful WS open" case now asserts the
return value is a `TcpStreamSwitchboard`, that `MeshService.reverseDialer`
is non-null after `ensureSession`, and explicitly closes the WS in
the test's finally so `wss.close` can resolve.
Pre-existing test logic that asserted `bridge instanceof PilotTunnelBridge`
on the same path is removed in favor of the new switchboard assertion.
Discovered during the v0.81.1 live verification: the peer-side
`route.dispatch cross-node` event always paired with
`proxy-tunnel.open.fail no_target` and `tunnel.fail`, even though
`centralCallback.bridgeOpen: true` and `mesh_centrals.last_used_at`
were populated. The callback path opened cleanly in isolation but
the dispatch path never consumed it. This fix makes the dispatch
path consume it.
Tests:
- cd backend && npx tsc --noEmit clean
- cd backend && npx vitest run mesh 170/170 green
- cd backend && npx vitest run 2259/2268 green; only
failures are the pre-existing Windows EBUSY flake in
filesystem-backup.test.ts (documented in prior handoff)
* fix(mesh): drop unused imports and align reverse-dialer interface name
CI lint failures on the previous push were two @typescript-eslint/no-unused-vars
errors in the test file:
24:76 'vi' is defined but never used
31:5 'TcpStreamSwitchboardCtor' is assigned a value but only used as a type
Both were collateral from a local-debugging simplification: vi.waitFor was
swapped for a synchronous assertion when the test's wss.close hang was traced
to an unclosed client WS; the toBeInstanceOf assertion that used the runtime
binding was removed at the same time. Restore the instanceof assertion next
to the existing not.toBeNull check so the runtime binding is in use and the
test guards against future regressions where the wrong end-of-pipe object
type is returned on the callback bridge (the bug this PR fixes). Drop the
now-unused `vi` symbol from the vitest import.
Also fold in a /simplify convergent finding: rename the local
CallbackReverseDialer interface to SwitchboardReverseDialer to match the
sibling declaration in meshProxyTunnel.ts:72. Same shape, same name; readers
of either handler now see the same mental model. Cross-file extraction of
the interface to tcpStreamSwitchboard.ts is the cleaner long-term shape but
touches a file outside this PR's scope; filing as a follow-up.
Considered-and-deferred findings from the /simplify pass (file as separate
PRs to keep one-branch-one-concern):
- extract shared attachSwitchboard helper (~40 duplicated lines between
meshProxyTunnel.ts and PeerToCentralMeshSessionDialer.ts)
- public TcpStreamSwitchboard.getWs accessor + drop currentWs field
- public MeshService.hasReverseDialer for the test bracket-cast
- shared close-reason constants (drift risk across the two handlers)
Tests:
- cd backend && npx tsc --noEmit clean
- cd backend && npx eslint <changed> clean (lint_exit=0)
- cd backend && npx vitest run mesh 170/170, dialer 8/8 green
|
||
|
Anso
|
54d84a7edb |
fix(mesh): skip peer-side recovery in openCrossNode on central instances (#1068)
The R1-A2 symmetric-callback work added a peer-side recovery branch to
MeshService.openCrossNode: when reverseDialer is null, kick
PeerToCentralMeshSessionDialer.ensureSession so the peer re-opens its
callback WS to central before the cross-node dispatch.
The guard `!this.reverseDialer` did not distinguish "I am a peer waiting
for central to dial in" from "I am central, and the reverseDialer concept
does not apply to me." On central, reverseDialer is always null (central
is the bridge initiator side and never has another node dial into it), so
central entered the recovery branch on every forward dispatch, found no
session (central has no mesh_centrals row), and destroyed the inbound
socket with route.resolve.fail forward-from-peer no_session.
Gate the branch on `MeshCentralRegistry.getActive() !== null` so it only
runs when this Sencho is acting as a proxy-mode peer that has actually
been bootstrapped. Central falls straight through to dialMeshTcpStream.
The peer cold-start scenario the comment originally described is still
covered: a proxy-mode peer with a cached mesh_centrals row but no live
WS to central will fire the recovery branch as before.
Adds two regression tests:
- central path (no mesh_centrals row, no reverseDialer) skips recovery
and calls dialMeshTcpStream
- proxy-peer path (mesh_centrals row + ensureSession returns null)
still emits route.resolve.fail forward-from-peer no_session
Existing tests in the "openCrossNode (BUG-4)" block already used a stub
reverseDialer to bypass the recovery branch, which is why the central
regression was not caught pre-release. The new tests deliberately leave
reverseDialer null to exercise the gating.
|
||
|
Anso
|
cf618dd866 |
feat(mesh): symmetric WS dial for proxy-mode mesh peers (#1066)
* chore(mesh): foundation for symmetric callback dial Adds the data-plane scaffolding that the symmetric callback dial fix builds on: - mesh_centrals table for peer-side bootstrap material - MeshCentralRegistry service (upsert/getActive/clear/markUsed/markRejected) - PilotTunnelManager kind discriminator and replaceOrRegisterProxyBridge - mesh_proxy_callback_bootstrap capability registration - MeshProxyTunnelDialer reason-tagged proxy-bridge-down events from a single tearDownBridge emission point - Reactive redial scheduler that skips idle and auth_failed reasons * feat(mesh): add reverse-direction activity log entries (closes R1-B) acceptReverseLocal now emits route.resolve.ok with direction=reverse on connect ack and route.resolve.fail with direction=reverse plus reason=container_not_found / connect_error pre-connect. Post-connect close/error stays silent. Reuses existing event types via the new details.direction discriminator so frontend filters are unaffected. * feat(mesh): add peer-to-central callback dial path (closes R1-A2) Closes the architectural gap where proxy-mode mesh peers could not re-establish their tunnel to central after any non-idle bridge teardown (idle close, network blip, central restart, peer reboot). Central remains the hub for the data plane; the change is purely about WS initiation. Symmetric WS initiation, asymmetric protocol roles. Central retains PilotTunnelBridge ownership; peer retains TcpStreamSwitchboard + reverseDialer ownership. Central bootstraps callback credentials over the first authenticated central-initiated mesh tunnel via a one-shot mesh_handshake JSON frame; peer persists the material in a new mesh_centrals SQLite table and dials central's new /api/mesh/proxy-tunnel-from-peer endpoint when local cross-node traffic needs a bridge and none is live. Mesh_tunnel JWT (HS256, signed with auth_jwt_secret) carries scope, audience, issuer (central instance id), peer api_token fingerprint, kid. Validation on inbound peer dial: algorithm pin, signature, scope, audience, instance, time bounds, node existence and mode, fingerprint match. Failures return HTTP 401 with a machine-readable reason; peer routes the response per a clear-vs-keep cache matrix. Triggers proactive bootstrap on mesh-enable and api_token rotation; central startup fans out to mesh-enabled proxy-mode nodes with mesh_stacks rows (throttled, fire-and-forget). Reactive redial on non-idle bridge loss. Capability-gated handshake send (mesh_proxy_callback_bootstrap) makes the upgrade path safe against older peers in mixed-version fleets. Adds peer-side /api/system/pilot-tunnels centralCallback diag block, bounded counter metrics for bootstrap and dial events. SENCHO_PRIMARY_URL preflight warning when unset on a central with mesh-enabled proxy nodes. Tested with unit suites for the validation chain, registry, manager, and both dialers; integration tests for bootstrap E2E (asserts protocol-role invariant), api_token rotation, instance id change, version skew, and pilot-mode regression. * fix(mesh): green CI on the symmetric callback branch Two independent CI failures, both surgical: 1. Backend tests (11 fails): four mesh test files called setupTestDb in beforeEach. setupTestDb does not reset the DatabaseService singleton, so the per-test afterEach rm of the previous tmpdir left the singleton connection pointing at a deleted file. The next beforeEach's line-55 write threw SQLITE_READONLY_DBMOVED on Linux. Windows file-lock semantics hid this locally. Hoist setupTestDb / cleanupTestDb to file-scope beforeAll / afterAll; per-test state resets stay in beforeEach. Matches the convention in the eight mesh test files that already pass. 2. CodeQL (4 high alerts): js/insufficient-password-hash flagged sha256(api_token) at four sites. The api_token is a 256-bit opaque bearer (sen_sk_-prefixed), not a human password; sha256 is the correct fingerprint primitive for binding the mesh_tunnel JWT to a specific token. Add the two production files plus the two test files that mint the fingerprint to the existing path-scoped query-filter for that rule. * fix(mesh): drop unused afterEach import and revert dead codeql config ESLint flagged afterEach as unused in mesh-central-registry.test.ts:1 after the previous commit hoisted setup/teardown to file-scope beforeAll/afterAll. Remove from the vitest import line. Revert the codeql-config.yml additions from the previous commit. The paths: sub-key under query-filters > exclude is not a documented CodeQL feature and silently no-ops. The four js/insufficient-password-hash alerts on api_token fingerprinting are tracked as dismissed false positives in the GitHub Security tab rather than via dead config. |
||
|
Anso
|
d882f223f4 |
feat(api-tokens): switch to sen_sk_ prefixed opaque keys (#1062)
* feat(api-tokens): switch to sen_sk_ prefixed opaque keys
Replace JWT-shaped API tokens with 56-char opaque keys of the form
`sen_sk_<43-char base62 random><6-char base62 checksum>` (256-bit
entropy, sha256-truncated checksum). Node-proxy tokens stay JWTs.
Why:
* The api_token path was already a sha256 DB lookup; the JWT signature
was wasted work and the 400d JWT ceiling vs DB expires_at was a
confusing dual bound.
* Opaque tokens carry a verifiable checksum so malformed/typoed values
are rejected before any SQLite lookup.
* `sen_sk_` prefix is recognizable to GitHub, TruffleHog, GitGuardian
and makes the on-wire shape visually distinct from node_proxy JWTs.
Changes:
* New `utils/apiTokenFormat.ts` (generate + checksum-verify, CSPRNG via
randomInt, timingSafeEqual on the checksum compare).
* `middleware/auth.ts` and `websocket/upgradeHandler.ts` route opaque
tokens before any jwt.verify; 401 messages unified to avoid a
token-existence oracle.
* `middleware/rateLimiters.ts` short-circuits opaque tokens in the
node_proxy detection and keys per-token via a non-reversible sha256
slice so each token keeps its own bucket without a DB hit.
* All six existing tests migrated from jwt.sign({scope:'api_token'})
to generateApiToken(); new format-only test suite covering prefix,
length, alphabet, checksum reject paths, and a 10k-iteration
collision/integrity loop.
* Docs (features/api-tokens.mdx, api-reference/overview.mdx) describe
the shape and drop the obsolete JWT-ceiling note.
* fix(api-tokens): clear CI lint and CodeQL false positives
* Drop unused TEST_USERNAME import in remote-console-session.test.ts;
the migration to generateApiToken() left it orphaned.
* Add a CodeQL barrier model so `generateApiToken`'s ReturnValue does
not flow into the `insufficient-password-hash` query. The function
emits 256-bit CSPRNG opaque keys; sha256 of the raw token is the
correct construction for high-entropy API tokens (bcrypt-class
hashes target low-entropy human passwords). CodeQL's name heuristic
was treating "Token" as a password source and flagging the standard
sha256 wrapping at all 9 call sites.
* ci(codeql): exclude js/insufficient-password-hash for token paths
The previous barrierModel data extension was a no-op for this rule: the
js/insufficient-password-hash query identifies its "password" sources via
SensitiveExpr's name heuristic ("token", "secret", "key" substrings),
which is upstream of the taint-tracking layer where barrierModel applies.
Verified by post-push re-analysis: 9 alerts still open, all undismissed.
Replace the dead extension with a path-scoped query-filter in
codeql-config.yml so the rule no longer fires on apiTokenFormat,
apiTokens, and the test directory. Real user-password hashing code
elsewhere in the repo (auth, users, setup routes) remains analyzed.
The 9 existing alerts on PR #1062 are dismissed via API as false
positives with a justification pointing at this config. Future runs
will not re-flag them because of the path filter.
|
||
|
Anso
|
f63ffdd098 |
fix(mesh): persist proxy-tunnel self-id across bridge teardown (#1060)
The proxy-mode WS handler called setProxyTunnelSelfCentralNodeId(null) on tunnel close, dropping the central-namespace nodeId install. After an idle close, the next inbound forwarder traffic on the peer fell back to getDefaultNodeId() (=1 on proxy peers without SENCHO_ENROLL_TOKEN), collided with the alias overlay's nodeId for central's own Local, and dispatched cross-fleet aliases to the same-node path. The signature was TCP open + no banner on the prober, with route.resolve.denied on the peer activity log even though R1's identify install fired correctly when the bridge was warm. The peer's identity in central's namespace is enrollment-stable, not per-WS. Stop null-clearing on teardown. The setter already emits a console.warn when the install is overwritten with a different non-null value, which is the right signal for an actual re-enrollment. Tests: existing assertion that the install becomes null on close is flipped to assert persistence, plus two new tests for the close-reopen lifecycle: same nodeId is idempotent (no second identify activity event), different nodeId emits the overwrite warn. Idempotency test polls deterministically for the first identify event before snapshotting the activity-count baseline so the assertion direction can't invert on a slow CI runner. |
||
|
Anso
|
489aab4516 |
fix(mesh): testUpstream dials via ensureBridge so proxy-mode targets are probeable (#1057)
testUpstream short-circuited on hasActiveTunnel + getBridge (pilot-only) when probing a cross-node alias. Proxy-mode remotes never have a "pilot tunnel" in the manager's sense, so any operator hitting "Test alias" on a proxy-mode target returned tunnel_down even when the cross-node dial via dialMeshTcpStream worked fine. Replace with await ptm.ensureBridge(target.nodeId), matching the mode- agnostic dispatch dialMeshTcpStream already uses. ensureBridge returns an existing pilot tunnel, an existing proxy bridge, or asks MeshProxyTunnelDialer to dial on demand. The 'no pilot tunnel' vs 'no bridge' distinction goes away (they meant the same thing). Adds two test cases: probing a proxy-mode remote via on-demand bridge, and the tunnel_down path when ensureBridge yields null. |
||
|
Anso
|
3d5f0ffacd |
fix(mesh): reject opt-in when every service has empty ports (#1056)
optInStack rejected empty service lists but not the case where services exist with every ports: []. The newPorts Set ended up empty, the function sailed past collision checks, wrote a mesh_stacks row, and produced an empty-aliases override. The stack then appeared online in the Routing tab but no traffic actually routed. Add a check after building newPorts: throw no_target with a clear message before the SENCHO_LISTEN_PORT collision check. Reuses the existing MeshError code (semantically: no routable target) so the frontend's error toast path is unchanged. |
||
|
Anso
|
6185206323 |
fix(mesh): proxy peer learns its central-namespace nodeId at tunnel upgrade (#1055)
Proxy-mode peers receive an alias overlay from central with nodeIds in central's namespace, but had no source for their own nodeId in that namespace. selfCentralNodeId fell back to the peer's local DB default (always 1), and handleAccept falsely matched cross-node aliases (nodeId=1, central's Local) against the peer's self-id. Cross-node aliases routed to openSameNode and surfaced as route.resolve.denied. Central now appends ?nodeId=<central-namespace-id> to the proxy-tunnel WS URL. The peer's upgrade handler parses it with a strict integer regex, installs into MeshService before the reverse dialer is registered, and clears on disconnect. handleAccept resolves self-id in priority order: proxy-tunnel install > SENCHO_ENROLL_TOKEN > local default. A non-null overwrite to a different value warns loudly (misconfigured deployment). Missing or malformed query param logs a warn and proceeds, leaving reverse-direction dispatch undefined until both sides are upgraded. Adds 5 test cases: nodeId install + teardown, missing/malformed input (including decimal, leading zeros, exponent, whitespace), overwrite warn, and CAS-rejected reverse-dialer leaving the slot unchanged. |
||
|
Anso
|
946db9df52 |
fix(mesh): accept node_proxy at the proxy-tunnel WS upgrade (#1050)
The mesh proxy-tunnel WS handler was gated on full-admin api_token scope only, so node_proxy JWTs (the credential the Add Remote Node dialog tells operators to generate via Settings -> Nodes -> Generate Token) were silently rejected with HTTP 403. Operators followed the dialog's instructions, enrolled the node cleanly, saw reachableMode='proxy' with no negative badge, opted a stack into mesh, watched the redeploy complete, and only then discovered no bytes flow. node_proxy already authorizes every /api/* surface on the remote (deploy, exec, host console, filesystem), which is a strict superset of what the mesh proxy-tunnel does. Gating mesh more strictly was theatre, not security, and it created a UX trap with no in-product signal pointing at the scope mismatch. Change the upgrade dispatcher to accept any machine-to-machine credential: node_proxy JWT or full-admin api_token. Session cookies and restricted api_token scopes (read-only, deploy-only) remain rejected through the same paths as before. Adds positive/negative coverage in upgrade-order.test.ts for all four credential shapes so the gate stays pinned against future re-tightening. Updates user-facing copy in the mesh docs and Settings -> API Tokens description so the documented path matches the actual code. If we ever introduce a demoted node_proxy variant (audit-only, read-only fleet member), the right move is differentiated node_proxy scope claims inside the tunnel JWT (tracker F-A3), not requiring a separate token type for mesh. |
||
|
Anso
|
a38a3e0226 |
feat(mesh): route mesh traffic over Distributed API remotes (#1048)
* refactor(mesh): extract shared TCP stream switchboard
Pull the `tcp_open` / `tcp_open_ack` / `tcp_open_reverse` / `tcp_close`
+ `TcpData` handling out of the pilot agent into a reusable module so a
second caller (the upcoming proxy-mode WS handler) can run the same
frame parser, stream allocator, idle timers, and Compose-label
resolver. One parser, two callers, zero drift on a
security-sensitive protocol surface.
The pilot agent keeps its public `openMeshTcpStream` API and delegates
to a per-connection switchboard constructed in `connect()` and torn
down in `cleanupAfterDisconnect`. Existing reverse-stream and resolver
tests are rewritten to exercise the shared module directly.
* feat(mesh): route mesh traffic over Distributed API remotes
Sencho Mesh now works against remotes in Distributed API mode in
addition to Pilot Agent mode. Central opens a short-lived WebSocket
tunnel to the remote's new `/api/mesh/proxy-tunnel` endpoint on demand
and tears it down after 5 minutes of idle (configurable via
`SENCHO_MESH_PROXY_TUNNEL_IDLE_MS`). Mesh dispatch is mode-agnostic at
the routing layer; `PilotTunnelManager.ensureBridge` resolves an
existing tunnel or asks the new `MeshProxyTunnelDialer` to dial.
The Routing tab badges now use a `reachableMode` classifier: `★ Local`
for local, `pilot offline` red badge for a pilot with a down tunnel,
and a new `unreachable` red badge surfacing the specific reason
(missing token, scope not full-admin, TLS failure, remote does not
support proxy mesh). Distributed API remotes with valid credentials
show no negative badge; the tunnel opens on first dial.
Auth: the proxy-tunnel WS upgrade requires an `Authorization: Bearer`
API token with the `full-admin` scope. Lower-scoped tokens are
rejected at upgrade time so a leaked read-only token cannot reach the
mesh data plane.
Bidirectional: the proxy-mode WS handler registers itself as the
local `MeshService` reverse dialer (compare-and-swap), so meshed
containers on a Distributed API remote can dial cross-node aliases
via `tcp_open_reverse` over the same tunnel. Cross-node relays
(`PilotTunnelBridge.acceptReverseRelay`) await `ensureBridge` so
proxy-to-proxy mesh works without standing tunnels.
* docs(mesh): describe Distributed API mesh and unreachable troubleshooting
Refresh the user-facing mesh documentation to reflect that mesh works
over both Pilot Agent and Distributed API remotes. Adds a
Troubleshooting accordion entry for the new `unreachable` Routing tab
badge so operators can match a tooltip reason ("api token rejected
(scope must be full-admin)", "remote does not support proxy mesh",
"TLS handshake failed", "api_url not set", "api token missing") to a
concrete fix.
* refactor(mesh): apply /simplify review findings
Five cleanups surfaced by the post-implementation code review pass.
No behaviour change for fleets running on `main`; only internal
structure improves.
- **Deterministic container IP shared.** Extract the compose-default
network preference (`pickContainerIp`) and the conventional-name
fast path (`lookupContainerIp`) into `backend/src/mesh/containerLookup.ts`.
Both `MeshService.resolveContainerIp` (existing same-node fast
path) and the switchboard's `resolveByComposeLabels` (proxy/pilot
inbound dial) now use the same logic. Earlier the switchboard
helper grabbed the first `Object.values(Networks)` IP, which
could flip across daemon versions on multi-network containers;
fixed.
- **WS URL upgrade extracted.** New `backend/src/utils/wsUrl.ts`
exposes `httpUrlToWs(baseUrl)` that maps `http://` to `ws://` and
`https://` to `wss://`. Used in both `pilot/agent.ts` and the
proxy-tunnel dialer. The previous inline `replace(/^http/, 'ws')`
silently downgraded `https://` to `ws://` (cleartext).
- **`computeReachable` takes the pre-fetched node.** `getStatus`
already iterated `db.getNodes()`; the helper previously re-queried
by id per node (N+1 reads). Pass the row in.
- **Reachable-reason ternary -> const map.** Replace the nested
ternary in `MeshService.computeReachable` with a
`Record<DialFailureCode, string>` lookup keyed on the dialer's
failure code.
- **Activity-type ternary -> const map.** Same flattening inside
`MeshProxyTunnelDialer.logActivity`.
All mesh tests pass (85/85). Full backend suite green (2126/2126).
* fix(mesh): cache proxy dial failures and contain WS handshake errors
`ensureBridge` now consults the recent-failure cache before dialing so a
continuous mesh workload against a misconfigured proxy-mode remote does
not produce one upgrade attempt per cross-node TCP open. `recordFailure`
emits at most one activity-log entry per cache window per (nodeId, code)
so a connect-loop on a single bad remote cannot flush the ring buffer.
Failure messages run through `redactSensitiveText` before reaching the
log so any embedded Bearer / JWT / inline-URL credentials are scrubbed.
`stop()` clears the inflight map and `dial()` checks `this.stopped`
between awaits so a shutdown does not leak a half-opened bridge.
When `awaitOpen` rejects (401, 4403, TLS), the dialer attaches a noop
'error' listener before calling `ws.close()`. Without it the ws library
emits a tail 'error' on a still-CONNECTING socket that propagates as an
unhandled exception. Surfaced by a live-network test against a real
proxy-mode peer.
Adds `mesh-proxy-tunnel-live.test.ts` (skipped unless MESH_AUDIT_URL
and MESH_AUDIT_TOKEN_FILE env vars are set) covering both the happy
path and the auth-rejected path against an actual remote Sencho.
* feat(mesh): instrument proxy tunnel observability
Adds three counters to `PilotMetrics`:
- `proxy_bridges_total` (incremented on `registerProxyBridge` success)
- `proxy_dials_failed` (every failed dial attempt, not deduped)
- `proxy_idle_closes` (idle-sweep teardowns)
All three surface automatically via `GET /api/system/pilot-tunnels`
since the route returns the full `Counters` snapshot.
Gates two pre-existing always-on `console.warn` calls in
`tcpStreamSwitchboard.ts` (mid-stream socket errors and Docker resolve
failures) behind `isDebugEnabled()`. Both fire on per-stream events and
would otherwise violate the diagnostic-log safety rule under load.
Adds `mesh-tcp-stream-switchboard.test.ts` covering the forward
`tcp_open` path: real localhost dial, resolver errors (no_target,
denied), per-tunnel cap saturation, frame-routing fall-through
invariants, `tcp_close` socket teardown.
Drops `MESH_CONNECT_TIMEOUT_MS` from the public surface; only the
switchboard itself uses it.
* fix(mesh): tighten Routing tab unreachable handling
`RoutingNodeCard.tsx`: the **Add stack to mesh** button now disables
when `reachableMode === 'unreachable'`, matching the existing
TogglePill behavior. Without this, an operator on an unreachable node
could open the opt-in sheet, confirm, and watch the redeploy proceed
against a target whose mesh data plane will silently fail to route.
Also drops the leftover `status.nodeId !== -1` guard on the pilot-
offline badge.
`MeshService.ts`: renames `isMeshReachable` to `isMeshConfigured` with
the new predicate that returns true for proxy-mode remotes whose creds
are valid (the tunnel is opened on demand). `getRouteDiagnostic` now
distinguishes "routable" (configured) from "pilotLive" (live tunnel
state, only meaningful for pilot mode); without the split, every idle
proxy-mode route would report `tunnel down`.
`MeshService.setReverseDialer` warns when the unconditional install
path silently overwrites a non-null current dialer. By topology a
Sencho is either pilot or central, so the branch flags a misconfigured
deployment rather than an expected race.
Drops the dead `export { ReverseTcpStreamHandle }` re-export from
`pilot/agent.ts`; its only consumer imports straight from
`mesh/tcpStreamSwitchboard.ts`. Fixes a stale doc comment in
`MeshService.ts` that referenced the wrong source file.
Adds `mesh-proxy-tunnel-handler.test.ts` covering the WS handler
lifecycle: pilot-mode 404 rejection, post-upgrade reverse-dialer
install, concurrent-upgrade 1013 rejection (single-tenant slot), and
error-path teardown.
Updates the troubleshooting accordion in the user docs to mention the
disabled-state behavior.
* fix(mesh): close ESLint and CodeQL findings on the proxy-tunnel diag logs
Remove the unused `reverseDialer` local in `meshProxyTunnel.ts`; the
closure target is `localDialer` above and this assignment was always
dead. Strip line breaks inline on the three `MeshProxyTunnelDialer`
diag log lines so CodeQL `js/log-injection` data flow recognises the
sanitisation that `sanitizeForLog` already performs.
No behaviour change; the diag logs render the same characters they
do today.
|
||
|
Anso
|
44e40afb62 |
fix(schedules): align Schedules surface with backend tier gate for Skipper admins (#1047)
The Schedules sidebar entry was hidden from Skipper admins even though the backend permits them to create and run update, scan, and snapshot schedules. The action picker also showed all 10 actions to every paid admin, so a Skipper selecting Restart, Prune, or any auto_* lifecycle action would 403 on submit. Changes: - Extract SKIPPER_SCHEDULED_ACTIONS as the single source of truth in tierGates.ts; both requireScheduledTaskTier and the GET /scheduled-tasks list filter now reference it (replaces a duplicate local constant in scheduledTasks.ts). - Move the Schedules nav entry from the Admiral block into the isPaid && isAdmin block in useViewNavigationState.ts, mirroring the existing Auto-Update pattern. Console and Audit stay Admiral-only. - Filter the create-form action picker in ScheduledOperationsView.tsx by license variant. Skipper sees Auto-update Stack, Auto-update All Stacks, Fleet Snapshot, and Vulnerability Scan; Admiral sees the full set. - openCreate now defaults formAction to the first visible option so Skipper starts with a valid choice instead of the Admiral-only Restart. Tests: - Add Skipper-variant POST coverage in scheduled-tasks-routes.test.ts: three allow cases (update / scan / snapshot) and a six-action rejection loop covering restart / prune / auto_backup / auto_stop / auto_down / auto_start. - Flip the Skipper assertion in useViewNavigationState.test.tsx to expect scheduled-ops alongside auto-updates. |
||
|
Anso
|
5461bc316b | fix: harden stack management operations (#1046) | ||
|
Anso
|
8dd0fce621 |
fix(fleet): show capabilities, version, metrics, and stacks for pilot-agent nodes (#1044)
When a pilot-agent node was the active node, the UI rendered "does not advertise this capability" across most tabs, a perpetual "Update available" badge, and a Fleet card body with blank CPU/RAM/Disk and "No stacks found". The cause was central-side aggregators in /api/fleet/* and /api/nodes/:id/meta only fanning out to proxy-mode remotes via node.api_url + node.api_token, which are null for pilot-agent. Route every affected aggregator through NodeRegistry.getProxyTarget so the loopback URL backed by the active pilot tunnel is used uniformly: - /api/nodes/:id/meta and /api/fleet/update-status fetch via the new NodeRegistry.fetchMetaForNode helper (resolves the target, delegates to fetchRemoteMeta, returns the shared OFFLINE_META on null). - fetchRemoteNodeOverview, /api/fleet/configuration, /api/fleet/node/:nodeId/stacks, and the stack-containers drilldown fetch through target.apiUrl with conditional Authorization. - fetchRemoteMeta omits the Authorization header when the token is empty (pilot-agent loopback) instead of sending a malformed Bearer string. - Pilot-agent rows preserve pilot_last_seen and mirror it into last_successful_contact so the Fleet "last seen" cell renders the recent tunnel timestamp during a brief reconnect. Pilot-mode capability filter excludes capabilities whose central-pilot path is not yet wired (host-console, self-update). Without this, the Console tab would surface for an Admiral pilot session and click through to central's host because the WS upgrade handler still gates on api_url + api_token. Filtered capabilities are removed at boot via applyPilotModeCapabilityFilter when SENCHO_MODE=pilot. Cache invalidation on tunnel-up: the meta cache for a reconnecting pilot is dropped so the next request rebuilds capabilities and version through the live bridge instead of waiting for the 3-minute TTL. The namespace constant moves to helpers/cacheInvalidation.ts alongside the new invalidateRemoteMetaCache helper. Husky commit-msg hook: add the missing shebang and a .gitattributes rule pinning .husky/* to LF line endings so commits do not fail with "Exec format error" on Windows shells where autocrlf=true converts the hook to CRLF. Tests cover Authorization-header behavior, pilot-mode filter idempotency, fetchMetaForNode dispatch (offline target, pilot-agent loopback, proxy-mode), and the four affected fleet routes for pilot-agent both when the tunnel is up and when it is down. |
||
|
Anso
|
e7a3b544c0 |
fix: harden auto-heal policies (#1042)
* fix: harden auto-heal policies * fix: resolve auto-heal lint failure |
||
|
Anso
|
2dda1bee1f |
perf(monitor): parallelize per-container stats fetch in evaluation cycle (#1045)
The MonitorService.evaluateStackAlerts loop processed containers serially, awaiting one Docker stats call per container. Each Docker stats call inherently waits ~1s for the engine to produce a sample, so cycle time scaled linearly with container count and exceeded the 25s warning threshold on production nodes with ~17+ running containers. Changes: - Extract the per-container body into processContainer(). - Replace the serial for-loop with a bounded-concurrency fan-out (runWithConcurrency, cap of 10) so the cycle runs in roughly max(per-container time) instead of sum. - Add a per-cycle dedup set (firedThisCycle) so a stack rule fires exactly once per cycle even when multiple containers in the stack all breach. Without it, parallel workers race past the cooldown check before any DB write lands and dispatch the same alert N times on first breach. - Tests cover wall-time parallelism, per-container failure isolation, the concurrency cap, and the per-cycle dispatch dedup. |
||
|
Anso
|
b52323036b |
fix: harden stack label permissions (#1036)
* fix: harden stack label permissions * fix: avoid test db init from debug logging |
||
|
Anso
|
328a98439d | fix: harden vulnerability scan scheduling (#1035) | ||
|
Anso
|
c31d48b933 |
fix: harden git source webhooks (#1033)
* fix: harden git source webhooks * fix: make path validation visible to CodeQL static analysis Add explicit isValidStackName guard in getEnvContent, isValidGitSourcePath pre-validation in readRepoFile, and URL hostname check in remoteStackRequest to satisfy CodeQL taint-tracking so the pipeline passes. * fix: use path.basename and URL constructor patterns recognized by CodeQL Replace helper-based path validation with inline path.basename and path.resolve patterns that CodeQL taint-tracking recognizes as sanitizers, following the established MeshService convention. Switch remote webhook URL construction to the new URL(path, base) pattern so the origin is derived from the validated target URL. * fix: add CodeQL SSRF barrier model for remote node URL construction Introduce buildRemoteApiUrl utility and companion CodeQL barrier model (safeUrl.model.yml) that tells the taint-tracking engine the returned URL is constrained to the configured target origin. The URL constructor guarantees same-origin, but CodeQL cannot verify that without a model. * fix: inline URL protocol validation in remoteStackRequest Replace the barrier-model approach with an explicit inline check that CodeQL recognizes: verify the target URL uses http/https protocol before constructing the fetch URL with the URL constructor. * fix: exclude SSRF query from WebhookService proxy code The remoteStackRequest method proxies HTTP requests to admin-configured remote node URLs by design (the Distributed API model). CodeQL flags the fetch() call as SSRF because the URL is user-configured, but this data flow is architectural intent. Exclude js/server-side-request-forgery from this file. * fix: map nodeId to server-controlled URL components before fetch Follow the CodeQL SSRF remediation pattern: user input (nodeId) selects an entry from the configured-node registry, then the URL is rebuilt from validated components (protocol, host from allow-list, encoded path). Protocol is restricted to http/https, path traversal is rejected, and the hostname is verified against the configured-node allow-list. * fix: remove unnecessary escape in endpoint validation regex |
||
|
Anso
|
b1c5fe8391 |
fix: harden deploy enforcement paths (#1030)
* fix: harden deploy enforcement paths * fix: update Docker toolchain to Go 1.26.3 * fix: repair Dockerfile tr argument split across lines * fix: bump protobufjs to clear npm audit high-severity advisories * fix(test): add execFile to child_process mock in compose-images test * fix: resolve merge conflicts with main * fix: resolve merge conflicts with main * fix: resolve merge conflicts with main |
||
|
Anso
|
74ae2ce0c6 |
fix: harden atomic deployment rollback (#1029)
* fix: harden atomic deployment rollback * fix: update Docker toolchain to Go 1.26.3 * fix: repair Dockerfile tr argument split across lines * fix: bump protobufjs to clear npm audit high-severity advisories * fix: sanitize error objects in console.error to prevent log injection |
||
|
Anso
|
69b6ac1f3b |
fix: harden stack file explorer operations (#1028)
* fix: harden stack file explorer operations * fix: update Docker toolchain to Go 1.26.3 * fix: repair Dockerfile tr argument split across lines * fix: bump protobufjs to clear npm audit high-severity advisories |
||
|
Anso
|
19cdb3681d |
fix: harden blueprint deployment guardrails (#1027)
* fix: harden blueprint deployment guardrails * fix: update Docker toolchain to Go 1.26.3 * fix: repair Dockerfile tr argument split across lines * fix: bump protobufjs to clear npm audit high-severity advisories |
||
|
Anso
|
b8af40d2b4 |
fix(mesh): enumerate stacks on remote pilot nodes for opt-in sheet (#1025)
The mesh opt-in sheet showed "No stacks deployed on this node yet" for every remote pilot because GET /api/mesh/nodes/:nodeId/stacks called FileSystemService.getInstance(nodeId).getStacks() unconditionally, which always reads central's own filesystem regardless of which node the operator targeted. Fix dispatches through the proxy chain when the targeted node is remote, mirroring the C-3 pattern that already governs every other mesh endpoint needing data from a remote Sencho: - New endpoint GET /api/mesh/local-stacks (Admiral-gated) returns the bare stacks list from THIS Sencho's own filesystem. Mirrors the precedent set by /api/mesh/local-services/:stackName. - New MeshService.listLocalStacks() and MeshService.listStacksOnNode() helpers; the latter dispatches local vs remote and degrades to an empty list on transport failure (no proxy target, non-2xx response, malformed body). - Refactored route delegates the stack-name list to listStacksOnNode; central's mesh_stacks DB is still authoritative for the opt-in flag set per C-3. Tests cover the local path, the remote-OK path with header forwarding, non-2xx, no-target (pilot tunnel down), malformed bodies, and unknown node ids. |
||
|
Anso
|
41a1df279d |
fix(mesh): fix pilot handleAccept dispatch by deriving self nodeId from enrollment token (#1023)
On a pilot node, handleAccept compared target.nodeId (from central's perspective via pilotAliasOverlay) against NodeRegistry.getDefaultNodeId() which always returns 1. This inverted same-node vs cross-node dispatch: - central's alias (nodeId=1) matched localNodeId=1 -> openSameNode on pilot (no container -> silent close, 0 bytes) - pilot's own alias (nodeId=14) != 1 -> openCrossNode -> tunnel loop Fix: add resolveSelfCentralNodeId() that decodes the nodeId claim from SENCHO_ENROLL_TOKEN (present on every pilot; payload decode only, no signature verification). Returns getDefaultNodeId() on central (env unset). Cached in selfCentralNodeId at start(); handleAccept reads the cache. After this fix the reverse direction (pilot-prober -> central echo) routes through openCrossNode (reverse tunnel) and the same-node path (pilot-prober -> pilot echo) routes through openSameNode (local Dockerode dial). Closes BUG-3 banner data-path regression. Tests: 5 new cases covering resolveSelfCentralNodeId token extraction, fallback paths, and handleAccept dispatch routing on a simulated pilot. |
||
|
Anso
|
6ac7da978c |
fix(mesh): push alias port map to pilot so reverse-direction forwarder binds listeners (#1021)
* fix(mesh): apply D-1 pushed override on pilot deploys via file-presence fallback When isMeshStackEnabled returns false, ensureStackOverride now checks for an override file already on disk before returning null. On pilot nodes the mesh_stacks table is intentionally empty (opt-in state lives on central per the C-3 design), so the DB gate blocked the override that central had already pushed via applyLocalOverride. File-presence is safe as the fallback because removeOverrideFromNode sends a DELETE to the pilot when a stack is opted out, so a stale file cannot survive past opt-out. Also aligns removeStackOverride to use path.basename consistently with all other override-path construction sites in the same file. Adds two tests: pilot node with a pushed file returns the path; pilot node with no file returns null. * fix(mesh): push alias port map to pilot so reverse-direction forwarder binds listeners Pilots have no mesh_stacks rows by design (C-3), so refreshAliasCache() produced an empty aliasByPort map and syncForwarderListeners() bound zero ports. Reverse-direction TCP probes (pilot prober -> central alias) failed with connection refused at the OS level because no listener existed. Fix: extend the D-1 override push payload to include the full MeshGlobalAlias records (host + port + routing metadata). The pilot stores these in a new pilotAliasOverlay map keyed by stack name. refreshAliasCache() merges the overlay after the (empty) DB loop, populating aliasByPort correctly. syncForwarderListeners() then binds the alias ports immediately after the push completes. removeLocalOverride() clears the overlay entry and re-syncs on opt-out. The two populations (DB rows on central, pilotAliasOverlay on pilots) are mutually exclusive by the C-3 invariant, so the first-write-wins port collision policy in refreshAliasCache is safe. |
||
|
Anso
|
c12b3ba995 |
fix(mesh): apply D-1 pushed override on pilot deploys via file-presence fallback (#1019)
When isMeshStackEnabled returns false, ensureStackOverride now checks for an override file already on disk before returning null. On pilot nodes the mesh_stacks table is intentionally empty (opt-in state lives on central per the C-3 design), so the DB gate blocked the override that central had already pushed via applyLocalOverride. File-presence is safe as the fallback because removeOverrideFromNode sends a DELETE to the pilot when a stack is opted out, so a stale file cannot survive past opt-out. Also aligns removeStackOverride to use path.basename consistently with all other override-path construction sites in the same file. Adds two tests: pilot node with a pushed file returns the path; pilot node with no file returns null. |
||
|
Anso
|
2d9e4d0480 |
fix(mesh): derive override services from compose file and surface cross-node dispatch (#1017)
* fix(mesh): derive override services from compose file and surface cross-node dispatch
Three fixes for issues found during Phase B end-to-end verification:
1. Override generator no longer queries Dockerode for services. Deploys
removed all containers in the project right before composeArgs ran,
so docker.listContainers({label: project=...}) returned [] and the
generator wrote services: {} which broke meshed user containers.
New helper getDeclaredStackServiceNames reads the local stack's
compose file via FileSystemService + YAML.parse and returns the
top-level services keys. Used from ensureStackOverride and
applyLocalOverride. Defensive fallback: when the read returns []
AND a previous override on disk has a non-empty services map,
keep the file and emit mesh.override.preserved.
2. Boot regen race against the pilot tunnel. regenerateAllOverrides
ran ~1s before the first pilot tunnel-up event, so pushes to
pilot-mode nodes failed with "no proxy target". MeshService.start
now also runs regenerateOverridesForNode(nodeId) inside the
tunnel-up listener, covering both the boot-race window and
mid-runtime tunnel reconnects. Idempotent.
3. openCrossNode emitted no activity log when a cross-node dial got
stuck waiting for tcp_open_ack. The forwarder accepted the TCP
handshake (so nc -v reported open) but logged nothing between
accept and close. Now logs route.dispatch on entry and emits
tunnel.fail after PROBE_TIMEOUT_MS if the agent never acks.
Timer is cleared on every tcpStream and src lifecycle event.
Tests: 8 new in mesh-service.test.ts cover the helper, the BUG-1
override write path, the defensive preservation fallback, the
tunnel-up regen handler, and both cross-node dispatch logging
behaviors. Full vitest suite green (1998 tests pass).
* fix(mesh): satisfy CodeQL path-injection model in new override-read paths
Two new fs.readFile sites added in the previous commit failed the
CodeQL check on the PR even though both paths were guarded by
isValidStackName + isPathWithinBase. The file's existing
applyLocalOverride pattern uses path.basename(stackName) before the
join (commented as "Recognized by CodeQL's path-injection model");
apply the same sanitizer to the new code paths.
- getDeclaredStackServiceNames: wrap stackName in path.basename when
building composePath.
- readExistingOverrideServiceNames: take (dir, stackName) instead of
a pre-built filePath so the sanitizer lives next to the read sink.
- ensureStackOverride: build the override file path with
path.basename(stackName) for symmetry with applyLocalOverride.
Runtime behavior unchanged (path.basename is a no-op for any name
that passed isValidStackName). Tests still green.
|
||
|
Anso
|
7cca68f84a |
fix(mesh): operator-triggered override regen and boot lifecycle hardening (#1016)
Audit-driven follow-up to the F6+F7 fix. Closes the High-severity findings
from the mesh-dial-path audit without expanding into the architectural
follow-ups (JWT TTL, TLS enforcement, per-stack JWT scope) which need
their own design discussions.
Changes:
1. New `POST /api/mesh/regen-overrides` Admiral-gated endpoint that calls
`MeshService.regenerateAllOverrides()` and returns a structured
`MeshRegenSummary { regenerated, failures, skipped, reason? }`. Lets
the operator rerun boot regen after fixing a remote node that was
offline at central startup, instead of opt-out + opt-in for every
meshed stack on that node. Audit-log row carries the outcome
(success / partial / skipped / error).
2. `regenerateAllOverrides` now aggregates per-stack outcomes into a
single summary log row (`mesh override regen complete: N succeeded,
M failed across K node(s)`) with `failedNodeIds` in `details`. Per-
stack warnings still emitted. When skipped because `senchoIp` is
null, emits an explicit warn instead of silently no-oping.
3. `MeshService.start()` now wraps `refreshAliasCache` and
`syncForwarderListeners` in their own try/catch so a throw from
either step is logged and the boot continues to override regen.
The closing log line surfaces data-plane state ("MeshService started
(data plane ok)" or "(data plane unavailable (<reason>))") so an
operator grepping startup output sees a half-init mesh immediately.
4. `sanitizeForLog` wrap on four previously unsanitized `err.message`
writes (cross-node tcpStream error handler + three probe error
paths). Container IPs and local socket details no longer leak into
the activity buffer or the probe response body.
5. Updated three existing F6 regression tests for the new return shape
and aggregated summary message. Added two new tests: version-skew
404-push handling, and concurrent opt-in during regen.
|