mirror of
https://github.com/Studio-Saelix/sencho.git
synced 2026-08-11 03:06:54 +00:00
3c934066f3d49d5850a42475821a2d47dbf52b2e
44 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Anso
|
3c934066f3 | fix(networking): ignore verified Mesh attachments in drift (#1729) | ||
|
Anso
|
35bb74425b |
feat: guide missing external network creation during deploy (#1645)
* feat: guide missing external network creation during deploy Detect missing external networks before Compose runs, prompt or auto-create safe bridge networks, and keep unsupported declarations blocked with trusted deploy provenance. * test: align deploy context and settings fixtures with missing-network gate Update caller spies, EffResource expectations, StacksSection save keys, and git-source spy cleanup so CI matches the new deployStack context and auto-create setting. * fix: drop unused renderError binding in missing-network resolver Satisfies no-unused-vars so backend ESLint CI passes; callers already key only on model presence. * fix: use HTTP-safe clipboard helper in missing-network dialog navigator.clipboard fails on plain HTTP LAN hosts; route copy actions through copyToClipboard so Docker and Compose copy buttons work on self-hosted instances. * fix: simplify missing-network dialog actions and copy label Drop the Compose snippet escape hatch, move secondary actions under More, and rename the terminal copy action to Copy create command so the footer is a clear Cancel / Create decision. |
||
|
Anso
|
5f1baa7522 |
fix: harden deploy/update concurrency and node-targeting safety (#1390)
* fix: harden deploy/update concurrency and node-targeting safety Release stabilization for deploy/update operational safety. Per-stack operation locking is now global. Background lifecycle paths (scheduler auto stop/down/start/backup/update, webhook execute, Git source auto-deploy, image auto-update, label bulk actions, fleet snapshot redeploy, and mesh redeploy) acquire the per-node, per-stack lock through a new StackOpLockService.runExclusive helper and skip rather than race a manual deploy/update/rollback/backup on the same stack and node. Skips surface honestly (a failed scheduled run, a recorded webhook failure, a per-stack batch result, or a thrown error) instead of a silent no-op. Update readiness and policy-bypass now run against the node captured when the dialog opened, not the live active node, so switching nodes while a dialog is open cannot retarget the update or the bypass retry. Rollback readiness no longer presents a moving-tag or unpinned image as a ready image revert. Restoring files does not revert a moving tag, so those stacks read as partial, and the rollback success message states that the compose and env files were restored. * fix: lock blueprint reconcile against manual ops and correct rollback wording Follow-up to the deploy/update safety hardening, closing two more gaps from a verification pass. BlueprintService.deployLocal and withdrawLocal called ComposeService directly, so blueprint reconciliation could race a manual deploy/update/rollback/backup on an owned stack. Both now run their compose lifecycle call through StackOpLockService.runExclusive and skip (recorded as a failed reconcile, retried on the next cycle) on conflict. The withdraw holds the lock across both the compose down and the directory delete so neither races a manual operation. The runtime rollback messages overstated recovery: a rollback restores the compose and env files and recreates containers, but does not revert an image behind a moving tag. The auto-rollback deploy-progress output, the recovery panel and chip, the failure toasts, and the manual rollback route message now state that the compose and env files were restored, with the matching OpenAPI example and atomic-deployments doc updated. * fix: acquire stack lock before blueprint deploy mutates compose and marker files Local blueprint deploy wrote the compose and marker files and ran the policy assert before acquiring the per-stack lock; the lock only wrapped the deploy itself. A reconcile could therefore rewrite an owned stack's files while a manual deploy/update/rollback/backup was running. The lock now wraps the whole critical section (create, write compose, write marker, policy assert, deploy), so on conflict nothing is written and the reconcile records a failed outcome. Adds a test asserting a deploy under a held lock records failed, writes no marker file, and leaves the manual lock untouched. * fix: make remote blueprint apply atomic under the receiving node's stack lock Remote blueprint deploy wrote the compose and marker files to the target node via separate HTTP calls and only locked on the final deploy, so the file writes could race a manual operation on that node. A node's operation lock is process-local and cannot be held by the hub across HTTP calls, so the locked create/write/deploy now runs on the receiving node. The locked critical section is extracted into BlueprintService.applyLocalUnderLock and exposed via POST /api/blueprints/apply-local. The hub posts the blueprint to that endpoint in one call; the receiving node runs create + write compose+marker + deploy under its own per-stack lock. Older nodes without the route answer 404 and fall back to the legacy multi-call flow. The endpoint is gated by paid tier and the same per-stack stack:edit and stack:deploy permissions as the PUT-compose + deploy it bundles, validates the stack name, compose size, and marker structure, and returns 409 on a lock conflict without writing anything. Adds tests for the atomic single-call path, the 404 legacy fallback, the 409 lock-conflict mapping, the route validation and permission paths, and the write-compose-then-marker-then-deploy ordering of the shared locked apply. * fix(deps): bump undici to 7.28.0 to clear high-severity advisory The frontend CI npm audit gate (--audit-level=high) failed on a transitive undici 7.25.0 (a dev-only dependency via jsdom): TLS certificate validation bypass (GHSA-vmh5-mc38-953g) and cross-user cache information disclosure (GHSA-pr7r-676h-xcf6). Bumping undici within jsdom's existing ^7.25.0 range to 7.28.0 clears the high-severity advisory and unblocks the frontend job. Lockfile only; no direct dependency or source change. |
||
|
Anso
|
f23b7e1bac |
feat: ordered multi-file Compose for Git sources (#1380)
* feat: ordered multi-file Compose for Git sources
Extend Git sources to deploy an ordered list of compose files merged with
docker compose -f base.yaml -f override.yaml ..., plus an optional project
directory.
- Pick and reorder compose files from the repository tree (drag to reorder on
desktop, up/down arrows on phones); manual path entry is also supported.
- The ordered set drives every stack-scoped compose command (deploy, update,
start/stop/restart/down, image scans, Compose Doctor) and the container
lookup, so a service or image declared only in an override is handled too.
- Runtime keys off the materialized set, not the saved configuration: saving a
source does not change deploy args until the pull is applied, and apply
materializes from the pending snapshot rather than live config.
- The project directory is passed as --project-directory, with -p <stack>
pinning the Compose project so container labels stay stable.
- The Mesh override is layered last; single-file sources are byte-identical to
before, and existing rows keep working via the single-path fallback.
Docs cover the picker, ordering, project directory, and the new troubleshooting
and limitations (referenced files are not materialized; the dependency graph,
drift, and networking views read the primary file).
* fix: harden multi-file Git source (hash, unlink, collisions, node id)
- hashContent folds ordered file CONTENTS (not paths) so a clean multi-file
stack is not flagged as locally edited: create/apply hash the fetched files
(repo paths) while pull hashes the on-disk files (materialized paths), which
previously disagreed and showed a false "local edits detected".
- Block unlinking a multi-file or project-directory Git source (409): the deploy
spec lives on the source row, so removing it would silently revert deploys to
root compose.yaml. Single-file sources still unlink.
- Reject materialized-path collisions in the selection validator: an additional
file equal to or nested under compose.yaml, an ancestor/descendant overlap
between selected files, and a project directory nested under a compose file
(previously a 500 at materialization).
- DockerController.getContainersByStack uses the controller's node compose dir
and passes its node id to the authored prefix, instead of the process default.
* fix: CI failures on multi-file Git source (test crash, aria query, path barrier)
- GitSourceFields no longer crashes when repoUrl/branch are falsy: the canBrowse
trim() is optional-chained, so a reusable field component tolerates partial
props. Fixes the apply-binding panel test, which feeds a minimal source object.
- GitSourcePanel tests query the footer Remove button by its exact name, so the
picker's per-file "Remove <path>" buttons no longer collide with the broad
/remove/i match (the test intent, footer Remove present/absent, is unchanged).
- validateCompose uses an inline resolve + startsWith barrier at the context-dir
mkdir sink (CodeQL does not credit the wrapped isPathWithinBase helper),
clearing the js/path-injection alert. The containment check is equivalent and
contextDir is also validated upstream.
* test: update Git source E2E spec for the multi-file compose picker
The compose-file picker replaced the single #git-source-path input and added
per-file Remove buttons, so the E2E spec drove selectors that no longer exist:
- Drop the redundant compose.yaml fills (the picker defaults to compose.yaml).
- Select the footer Remove button by exact name so the picker's per-file
"Remove <path>" buttons no longer make the locator ambiguous.
- Set a custom compose path through the picker (add via the manual input, press
Enter, then remove the default compose.yaml).
* test: match the footer Remove button with an exact Playwright name
Playwright's getByRole name option is a substring match by default, so
{ name: 'Remove' } also matched the picker's "Remove <path>" buttons. Require an
exact match so only the footer Remove button is selected.
|
||
|
Anso
|
865d792874 |
feat(pricing): collapse to two tiers (#1309)
* feat(pricing): collapse to two tiers (Community + Admiral) Collapse Sencho's pricing from three tiers (Community / Skipper / Admiral) to two: a generous free Community tier and a single paid Admiral tier. The Skipper tier is removed. Now free in Community: auto-heal, auto-update, scheduled operations, webhooks, notification routing, Fleet Actions and bulk operations, SSO preset providers (Google / GitHub / Okta), unlimited users with admin and viewer roles, and deploy safety (atomic deploys, auto-rollback, and one-click rollback). Admiral (paid) is focused on running and governing a fleet: blueprints, Fleet Secrets, deploy enforcement, vulnerability report export, audit log, host console, private registries, mesh networking, node cordon, managed cloud backup, LDAP / Active Directory SSO, and the advanced RBAC roles (deployer, node-admin, auditor) with per-resource scoped assignments. Internally the license variant distinction is removed so tier is binary (community / paid). License validation still verifies the Lemon Squeezy store and product before granting paid status. Docs and the contributor guide are updated to the two-tier model. * docs(pricing): correct licensing page to two-tier pricing and tidy stale tier wording The licensing docs page kept the old Admiral pricing plus a Founder Lifetime column and an Enterprise paragraph after the two-tier collapse. Update it to $12/month or $99/year, drop the lifetime and Enterprise content, and link to the pricing page for current pricing. Also fix stale "Skipper" wording in CLA.md, SUPPORT.md, one test title, and three test comments. Historical CHANGELOG entries and the retired-Skipper license-guard test are intentionally left as-is. * docs: align licensing and SSO pages with the two-tier model Correct the SSO overview so the Google, GitHub, and Okta presets read as available on every tier, matching the provider table; only LDAP and Active Directory require Sencho Admiral. Remove the lifetime-plan references from the licensing, settings, and troubleshooting pages so they reflect subscription-only Admiral pricing. * fix(rbac): omit scoped permissions from /me on the Community tier Scoped role assignments only take effect on the paid tier, but GET /api/permissions/me returned them unconditionally, so a downgraded instance with leftover assignments rendered per-resource affordances the API then rejected with 403. The endpoint now mirrors the permission middleware and includes scoped permissions only on the paid tier. Adds a regression test covering the downgrade case. * docs: use custom-pricing wording on the contact page The two-tier model has no Enterprise tier; reword the contact page's enterprise pricing/deals to custom pricing/deals so it does not imply a tier that no longer exists. |
||
|
Anso
|
5e6e96e405 |
feat(mesh): add developer-mode diagnostics to the mesh data plane (#1287)
* feat(mesh): add developer-mode diagnostics to the mesh data plane Add a developer-mode-gated [Mesh:diag] diagnostic log to MeshService, following the existing diagnostic-logging pattern: gated on the shared developer_mode setting (off in production by default) and every value run through sanitizeForLog. The diagnostics cover the forwarder dispatch decision (which self-node resolution won, and whether the route is same-node or cross-node), the entry of the opt-in, opt-out, enable, and disable operations, and completion timing for opt-in, node disable, and the alias-cache refresh. This lets an operator trace a slow or stuck mesh operation from the logs without attaching a debugger. The calls sit only at per-operation, per-accept, and per-refresh cadences, never inside the per-frame byte-relay loops. Add tests covering the developer_mode on/off gating and a guard that a node's api token never reaches a diagnostic log or the activity buffer. * refactor(mesh): redact secret-shaped values in developer-mode diagnostics Run each [Mesh:diag] detail through redactSensitiveText before sanitizeForLog so a future caller cannot leak a Bearer token, JWT, or credentialed URL through a diagnostic line, regardless of which call site emits it. No current call site passes a secret; this is defense in depth. Use a JWT-shaped canary in the no-leak test so it guards the real credential class. |
||
|
Anso
|
aa3d99a594 |
fix(mesh): re-evaluate data plane every 10s and add opt-in auto-recreate (#1184)
* fix(mesh): re-evaluate data plane every 10s and add opt-in auto-recreate MeshService.dataPlaneStatus was written exactly once at boot in setupMeshNetwork() and never re-evaluated. After the operator removed sencho_mesh at runtime (or it was recreated externally, or Sencho was disconnected from it), /api/health and the dashboard banner kept returning the stale boot-time discriminator until the next process restart. Adds a 10s revalidator that inspects the current Docker truth in one network-inspect call and transitions dataPlaneStatus to reflect it. Short-circuits in not_started / not_in_docker / subnet_invalid (states that cannot change within this process) and in concurrent ticks. Transitions are idempotent on stable state, so the timer can tick indefinitely on a healthy mesh without log noise. New 'not_found' reason value for the network-was-removed-at-runtime case. Existing reasons (subnet_mismatch, subnet_overlap, attach_failed) also surface from the revalidator when their underlying conditions arise post-boot. transitionDataPlane keeps message and subnet fields fresh across consecutive observations even when reason is unchanged, so /api/health never reports stale numbers (e.g. two consecutive subnet_mismatch observations against different external subnets). Adds an opt-in mesh_auto_recreate global setting (default off). When on, the revalidator additionally calls attemptInPlaceRecreate() after surfacing not_found. The helper hard-prefers the boot-chosen subnet (this.meshSubnet) and never iterates candidates, because changing the subnet here would invalidate every existing extra_hosts override on disk. A real conflict on the original subnet is reported as subnet_overlap and preserved during the 60s recreate throttle window so the operator-actionable reason is not flapped back to not_found between attempts. Self-attachment is checked via Name match (operator --hostname X matches container Name /X) or full container-ID prefix for hex HOSTNAMEs >= 12 chars (Docker default short ID). Non-hex HOSTNAMEs cannot collide with container IDs at all so a Name miss is conclusive; short hex HOSTNAMEs preserve the prior status as 'unknown' rather than risking a false-positive prefix match. Frontend surfaces: - types/mesh.ts: 'not_found' added to MeshDataPlaneReason. - MeshDataPlaneBanner: 'not_found' headline copy. - Settings > System > Mesh data plane: TogglePill bound to mesh_auto_recreate, default off, helper text explains the tradeoff. Backend coverage in backend/src/__tests__/mesh-data-plane-revalidate.test.ts (25 cases): short-circuits, idempotent stable-state, recovery from subnet_mismatch / subnet_overlap, transition to not_found / subnet_mismatch / attach_failed, transient-Docker anti-flap, re-entrancy guard, name match path, ID-prefix path, short hex hostname ambiguity, non-hex hostname certainty, transition message refresh on observation drift, auto-recreate off (default), auto-recreate success with senchoIp preservation, auto-recreate overlap classification with no subnet drift, throttle window preserves classified reason, throttle release. Lifecycle test covers timer wiring in start()/stop(). Existing mesh-setup-error-classification suite (27 cases) still green. Resolves: F-4 in the v1.0 audit tracker. * fix(mesh): address Codex review of PR #1184 Three findings from the independent review: BLOCKER: attemptInPlaceRecreate() called recordSetupFailure() on create / attach failures, which clears this.senchoIp. The next revalidator tick's attachment check is guarded on senchoIp, so with it null the check is skipped and the snapshot path can silently flip the status back to ok against a network where Sencho is in fact not attached. Also: a later successful recreate would call ensureSelfAttached() with senchoIp null, which short-circuits, so the network gets recreated without binding Sencho. Replaced the recordSetupFailure() calls in attemptInPlaceRecreate with a new recordRecreateFailure() that uses transitionDataPlane and preserves senchoIp. Added two tests: create-fails-then-succeeds (verifies senchoIp survives the failure and the later retry binds Sencho correctly) and create-succeeds-attach-fails-then-next-tick (verifies the snapshot path surfaces attach_failed on the next tick instead of falsely reporting ok). SHOULD-FIX 1: single-key POST /api/settings wrote String(value) without re-validating against the per-key schema, so an allowlisted enum-shaped key like mesh_auto_recreate could persist arbitrary strings ('banana', 'true') that the bulk PATCH would later refuse. Routed the single-key path through SettingsPatchSchema.safeParse so both write paths validate identically. Added regression tests for an invalid mesh_auto_recreate value, a valid mesh_auto_recreate write, and an out-of-range numeric value. SHOULD-FIX 2: the new Mesh data plane subsection lived inside a section the registry exposes to non-admins, who would see the toggle and only learn it was admin-only after the save 403'd. Gated the subsection on `isAdmin` from useAuth so non-admins do not see the control. The other system controls keep their existing visibility pattern (read-only for non-admins). 71/71 backend tests green (revalidate + mesh-setup + settings-routes). 276/276 frontend tests green. tsc clean on backend + frontend. |
||
|
Anso
|
64bf6344a3 |
fix(mesh): bias Sencho static IP via IPAM IPRange (F-13) (#1162)
* fix(mesh): reserve Sencho static IP via IPAM auxiliary address (F-13)
Sencho pins itself to <network>+2 on sencho_mesh, but the IPAM block only
declared Subnet, so Docker freely handed that address to any meshed
workload that restarted while Sencho was offline. A real-world hit on
arrapps-prod during a Sencho upgrade had tautulli grab 172.30.0.2, which
blocked the new Sencho container's mesh attach with "Address already in
use" and left compose in a half-state needing manual disconnect/recreate.
The fix reserves <network>+2 via AuxiliaryAddresses on the IPAM Config
when creating sencho_mesh. Aux-listed addresses are removed from the
auto-allocatable pool, so Docker refuses to hand the IP to any container
that does not explicitly request it. Sencho's own attach via
connectContainerToNetwork({ ipv4Address }) is unaffected because
explicit pins still bind aux-reserved addresses. Workload containers
without a pinned IP get .3 and up.
Wire format verified against the Docker Engine REST v1.33 OpenAPI spec:
the JSON key on POST /networks/create (and on the inspect response) is
AuxiliaryAddresses inside each IPAM.Config item, value { sencho: <ip> }.
Adopt-existing path: when Sencho boots against a sencho_mesh that
pre-dates this reservation, the data plane still comes up but a one-time
warn fires (mesh.enable activity at level: 'warn' plus a [Mesh] console
line so docker logs surfaces it). The advisory explains the squat risk
and gives the recreate recipe.
Tests: 5 cases added to mesh-setup-error-classification.test.ts covering
the explicit-env create payload, the candidate-iteration winner payload,
the adopt-legacy warn (env-unset), the adopt-already-reserved silent
path, and the TOCTOU 409 race-winner adopt-legacy warn.
Docs: one sentence added to docs/features/sencho-mesh.mdx under
"Customising the mesh subnet" describing the reservation positively.
No tier/role/capability/flag gates touched (no frontend changes).
* fix(mesh): use IPRange upper-half bias instead of aux-address reservation
The initial F-13 fix used IPAM AuxiliaryAddresses to reserve <network>+2
on sencho_mesh. Empirical probe against Docker 29.4.3 confirmed this
also blocks explicit pins via EndpointConfig.IPAMConfig.IPv4Address:
libnetwork's RequestAddress() rejects a preferred-address request when
the bit is already set by the aux reservation. Result: the freshly-
reserved network refuses Sencho's own ensureSelfAttached, and the data
plane never comes up.
The pivot uses IPRange instead. IPRange constrains Docker's auto-
allocation to the configured CIDR; preferred-address requests via
RequestAddress(prefAddress) skip the IPRange check and only consult
the subnet-wide bitmap. So setting IPRange to the upper half of the
subnet (e.g. 172.30.0.128/25 for 172.30.0.0/24) biases workloads
without an explicit IP to <network>+128 and up, while Sencho's
explicit pin to <network>+2 still succeeds.
Verified on Docker 29.4.3 with the same workstation that produced the
original audit:
- `docker run --rm --network N --ip 10.99.99.2` against a network
with `--aux-address sencho=10.99.99.2` → "Address already in use"
(rejects explicit pin).
- Same `--ip` against a network with `--ip-range 10.99.99.128/25` →
succeeds (10.99.99.2 is outside the range but inside the subnet).
- Auto-allocated workload on the IPRange network lands at .129.
Adopt-existing legacy detection now compares IPRange instead of
AuxiliaryAddresses. inspectExistingMeshSubnet returns { subnet,
ipRange } and the warn fires when ipRange differs from the expected
upper-half CIDR. Same once-per-process semantics as before.
createMeshNetwork now derives the IPRange via a new
getMeshIpRangeFromSubnet helper. The five regression tests assert
IPRange = <network>+128/<prefix+1> in the create payload and the
expected/actual IPRange in the legacy-warn details.
|
||
|
Anso
|
474290081d |
fix(mesh): log boot state to console so docker logs surfaces mesh status (#1159)
MeshService records its boot summary and setup failures only through logActivity (in-memory ring buffer + WS listeners), which the Routing tab consumes. The docker-logs surface was silent for the mesh subsystem, so operators running Sencho-in-Docker had no boot-time visibility into whether the data plane came up cleanly. Mirror the existing logActivity entries to console without replacing them: - MeshService.start() success summary: console.log / console.warn / console.error gated on the summary level. Format: [Mesh] data plane ok, self attached at <ip>, subnet <X> [Mesh] data plane unavailable (<reason>: <message>) - recordSetupFailure: console.warn for the expected dev-mode not_in_docker case, console.error for real failures. Format: [Mesh] data plane unavailable (<reason>, subnet <X>): <sanitized> The activity entries that already drive the Routing tab banner stay intact; the console lines are purely additive for the docker logs workflow. Two new unit tests assert the failure and not_in_docker console mirrors fire with the [Mesh] prefix. Fixes F-5. |
||
|
Anso
|
e4fe4cfced |
fix(mesh): address Codex audit findings on F-1 PR (#1158)
- docs(sencho-mesh): split subnet_overlap troubleshooting into env-set vs env-unset paths; rewrite the "Customising the mesh subnet" intro to describe the candidate list and the adopt-existing behavior. - backend(MeshService): preserve idempotent 409 handling in the explicit-env path. On createNetwork 409 (TOCTOU race against another process), re-inspect and treat the race-winner as success when its subnet matches the operator's request; subnet_mismatch otherwise. - frontend(MeshDataPlaneBanner): trim the card variant to a true one-line strip (headline only, truncate min-w-0). Full recovery hint stays on the Routing tab variant and in docs. - tests(mesh): add five cases covering the previously untested branches — candidate-loop non-overlap bail, adopt-existing with unparseable subnet, explicit-env generic createNetwork failure, TOCTOU 409 race-winner match, TOCTOU 409 race-winner mismatch. Architecture map (gitignored per Directive 11) updated locally with the new useMeshDataPlane hook node and the mesh.dashboardBanner flow so the local interactive viewer stays accurate. |
||
|
Anso
|
1a03cf82af |
fix(mesh): auto-fallback through candidate subnets when default overlaps (#1156)
The default mesh subnet 172.30.0.0/24 is fully contained in linuxserver/* default networks (sonarr_default 172.30.0.0/16, etc.), so libnetwork rejects the IPAM allocation with "Pool overlaps with other one on this address space" on a typical homelab Docker host. The single hard-coded default left first-run operators with a silently broken mesh. MeshService.setupMeshNetwork now resolves the subnet via three paths: 1. Operator-explicit (SENCHO_MESH_SUBNET set): use that subnet, strict. Pre-existing sencho_mesh with a different subnet still raises subnet_mismatch. 2. Adopt-existing (env unset, sencho_mesh already on the daemon): adopt the existing subnet. Docker is the source of truth across restarts. 3. Candidate iteration (env unset, no existing network): walk 172.30.0.0/24, 172.31.0.0/24, 10.42.0.0/24, 10.43.0.0/24 in order. First subnet Docker accepts wins. If every candidate overlaps, record subnet_overlap with a message naming every attempt. The dashboard's Fleet Heartbeat card now surfaces the down state via a compact banner above the per-node rows, plus a "mesh down" counter suffix on the right of the title. The existing Routing-tab banner is extracted into a shared MeshDataPlaneBanner component with tab and card variants. Dashboard polling is gated on Admiral tier so non-paid users do not fire the Admiral-only /mesh/status endpoint. Six new tests in mesh-setup-error-classification cover: iterates past first overlap, all candidates overlap, adopts existing network, inspectNetwork non-404 failure classified as attach_failed, env-matches- existing skip-create, and operator-explicit strict (no fallback). Fixes F-1 in the pre-1.0 audit. Closes the silent-failure mode that left the mesh down on the most common homelab Docker layout. |
||
|
Anso
|
e65c5e8551 |
fix(pilot): post-merge audit followups (WS via getProxyTarget, closeTunnel lifecycle, mesh source buffer, docs) (#1128)
* fix(pilot): route remote WS upgrades through NodeRegistry.getProxyTarget Pilot-mode nodes carry empty api_url and api_token by design and expose their API on a per-tunnel loopback bridge. The upgrade handler gated the remote-forwarder branch on `node.api_url && node.api_token`, so WS requests targeting pilot nodes silently fell through to the local handlers (live logs, exec, generic) instead of tunneling to the agent. Resolve the target via NodeRegistry.getProxyTarget so pilot and proxy modes share one dispatch path, mirroring the HTTP proxy. handleRemoteForwarder now takes the resolved target and, when the target is the pilot loopback (empty token), skips the console-token exchange and the Authorization injection so the tunnel-side auth is the only source of truth on that path. Unresolvable targets reject the upgrade with HTTP 503 instead of being served gateway-local data. * fix(pilot): emit tunnel-down and mark node offline on closeTunnel PilotTunnelManager.closeTunnel closed the underlying WebSocket but skipped the cleanup the natural-disconnect path runs, so explicit closures (enrollment regenerate, node deletion) left the node row at status='online' until the next reconnect. The dashboard kept showing the stale state for the entire interval. closeTunnel now writes nodes.status='offline' and emits tunnel-down for pilot bridges, and emits proxy-bridge-down for central-initiated proxy bridges. The maps are cleared before bridge.close() so the natural 'closed' handler's bridge-identity guard short-circuits and we do not double-emit. * fix(mesh): buffer cross-node source data until tcp_open_ack arrives openCrossNode piped src socket data straight to tcpStream.write before the forward TcpStream emitted 'open'. The first packet on a fresh cross-node stream raced ahead of the agent's tcp_open_ack on the wire, which broke protocols that send immediately after connect (HTTP, TLS, Redis, Postgres) on Pilot and proxy mesh paths. Buffer src chunks in a local array capped at STREAM_PENDING_DATA_MAX_BYTES until tcpStream emits 'open', then flush them in order before any post-open writes. Tear down both sockets if the buffer overflows so a misbehaving source cannot exhaust gateway memory while waiting for the ack. * docs(pilot): clarify host-console non-parity and narrow the parity claim Pilot mode disables the host-console capability at the capability registry (the agent container has no useful host shell to surface), but the public docs listed host console among the WebSockets that ride through the tunnel and described pilot as behaving identically to proxy mode. State the shared-capability claim more carefully and call out the intentional non-parity in a dedicated subsection. |
||
|
Anso
|
982c7830b1 |
fix(mesh): address audit follow-ups (early-data, recompose, admiral gate, error codes, docs) (#1126)
* fix(mesh): buffer early TcpData on reverse-relay path
The reverse-relay code dropped the local-shaped reservation and any
TcpData frames buffered in it before acceptReverseRelay had wired up
the target stream. A peer that sent a request body immediately after
tcp_open_reverse lost those bytes when the target lived on a third
node, since reverse_local buffers them but reverse_relay did not.
Carry the reservation through acceptReverseRelay: transplant its
pendingData and pendingBytes into the new reverse_relay state, gate
TcpData writes on targetOpen, and flush buffered frames into the
target stream before sending tcp_open_ack. Mirrors the reverse_local
pattern. Regression test fires tcp_open_reverse plus an immediate
TcpData while ensureBridge is in flight and verifies the bytes
arrive intact, in order, before the ack.
* fix(mesh): recompose affected stacks when node-level mesh is disabled
disableForNode used to clear DB rows and override files but leave the
running containers attached to sencho_mesh with stale /etc/hosts
alias entries until an operator redeployed every stack by hand.
Mirror optOutStack: after the existing alias/forwarder cleanup, call
regenerateOverridesAcrossFleet, cascadeRecomposeAcrossFleet, and
triggerRedeploy for each previously meshed stack on the disabled
node so containers detach from sencho_mesh and shed the alias
entries they owned. The disabled node's mesh_stacks rows are deleted
before the cascade so listMeshStacks returns the right set with no
skip tuple required. Route threads the actor through actorFor(req)
for parity with optInStack/optOutStack. Tests cover the redeploy
fan-out, the cascade no-skip-tuple invariant, and the default actor
fallback for non-route callers.
* fix(mesh): require Admiral on the WS proxy-tunnel upgrade
HTTP mesh routes in routes/mesh.ts all enforce requireAdmiral, but
the /api/mesh/proxy-tunnel WS upgrade accepted any node_proxy or
full-admin api_token regardless of the receiver's license. A node
downgraded from Admiral kept serving mesh data-plane traffic to a
sibling central while refusing every mesh management call.
Read the receiver's local LicenseService at the upgrade and 403 when
the tier is not paid+admiral. The check sits after the existing
credential gate and uses LicenseService directly rather than
effectiveTier (which trusts forwarded proxy headers); a remote peer
dialing in cannot be trusted to assert our entitlement. Dialer and
node_proxy token format are unchanged. Three regression tests cover
community-tier node_proxy, skipper-tier node_proxy, and
community-tier full-admin api_token all rejected with 403.
* fix(mesh): handle no_target alongside push_failed in inspectStackServices
proxyFetch throws MeshError('no_target') when getProxyTarget returns
null (pilot tunnel offline, proxy bridge unreachable), but the
inspectStackServices catch branch only matched push_failed. Offline
remotes fell through to the generic 'remote unreachable' error log,
which the Routing tab surfaces as an unexpected fault.
Match both error codes and emit the operator-friendly warn message
that names the unreachable node and the error code. Regression test
spies on console.warn/console.error to pin the branch.
* docs(mesh): align env defaults and forwarder comments with current architecture
SENCHO_MESH_PROXY_TUNNEL_IDLE_MS in .env.example carried the old
five-minute idle-close value (=300000), but the code default is
DEFAULT_IDLE_TTL_MS=0 (persistent tunnel). Copying the example
silently reintroduced the idle-close behavior the dialer removed.
MeshForwarder.ts's leading docblock and inline listen comment still
described host-network mode plus extra_hosts: host-gateway as
required for forwarder reachability. Sencho runs in standard bridge
mode and attaches to the shared sencho_mesh network at a stable IP;
meshed user containers reach the forwarder by that IP directly.
Flip the env default to 0, rewrite the env comment to describe the
persistent behavior and the opt-in for idle teardown, and rewrite
both forwarder comments to match the bridge-network reality.
* fix(mesh): trust forwarded tier on proxy-tunnel WS and remove remote overrides on disable
Admiral entitlement on the WS data plane now follows the same trust model
as the HTTP mesh routes: the central asserts its tier via x-sencho-tier
and x-sencho-variant on the WS handshake, and the receiver trusts those
headers only when the upgrade carries a node_proxy credential. When no
headers are present or the credential is a full-admin api_token, the
receiver falls back to its own local license. Without this an Admiral
central could be rejected by a Community remote and a Community central
could dial a locally-Admiral remote.
disableForNode now routes through removeOverrideFromNode so override
files pushed earlier via applyLocalOverride are removed on remote nodes
via DELETE /api/mesh/local-override/:stack. Sequential awaits are wrapped
in Promise.allSettled to match regenerateOverridesForNode's parallel
push pattern.
Other changes:
- Cover the post-state-swap buffering window in the reverse-relay test
(TcpData arriving after openTcpStream returns but before target open).
- Refresh stale MeshService comments that still referenced the removed
sidecar layer and host-network listener model.
* test(mesh): pin removeOverrideFromNode remote HTTP shape
The disableForNode regression test mocks removeOverrideFromNode itself,
so a regression inside the helper would not be caught. Add a narrow
contract test that spies on global fetch and asserts the request shape:
DELETE /api/mesh/local-override/:stack against the resolved proxy
target, with Authorization Bearer plus the x-sencho-tier and
x-sencho-variant headers. Also covers the encodeURIComponent path and
the swallowed-network-error behavior the disable cascade relies on.
* test(mesh): use createTestApiToken helper in proxy-tunnel api_token gate test
The full-admin api_token branch of the WS Admiral gate test inlined the
canonical generateApiToken + sha256 + addApiToken triple that already
lives in the createTestApiToken helper. Switching to the helper removes
the duplicated insertion logic and aligns this test with the helper used
by the other api_token call sites.
|
||
|
Anso
|
0cffd60173 |
fix(mesh): clean up activeStreams on src socket close/error (F-10) (#1105)
The cross-node MeshService.openCrossNode handler attached src.on('close')
and src.on('error') listeners that only called tcpStream.destroy() and
never deleted the activeStreams Map entry. MeshTcpStreamLike.destroy()
sends a tcp_close frame to the remote but does not synchronously emit
the local close event; that fires only when the remote sends back a
tcp_close ack via _dispatchClose (or when the entire tunnel tears down
and the bridge force-emits close on every stream).
Failed dials whose remote ack was lost (peer gone, network drop, or the
F-9-era timeouts) therefore leaked records into activeStreams until the
tunnel itself idle-closed, producing the monotonically-climbing
activeStreamCount documented in the mesh E2E audit (F-10).
Add a cleanupRecord() closure that clears the open timer and deletes
the record idempotently, then route all four lifecycle handlers
(tcpStream.on('error'), tcpStream.on('close'), src.on('close'),
src.on('error')) through it. Map.delete is naturally idempotent so the
double-delete that fires when both sides close normally is a free
no-op. The same-node openSameNode path already had this shape via its
existing teardown() closure; this brings the cross-node path to parity.
Three new Vitest cases in mesh-service.test.ts use an EventEmitter-
backed fake src so emit('close') actually delivers to the listener
(the existing makeFakeSocket uses vi.fn for .on which records calls
but never fires them). They cover src.close, src.error, and the
idempotency case where both src and tcpStream emit close in sequence.
|
||
|
Anso
|
c460bb87a8 |
fix(mesh): probe upstream synchronously in route diagnostic (F-11) (#1100)
GET /api/mesh/aliases/:alias/diagnostic returned a cached state derived from the last latency/error maps. Those maps only mutated when someone called POST .../test or when a cross-node connect logged an event, so once an upstream stopped the diagnostic kept reporting "healthy" until the 60 s alias-cache refresh pruned the alias entirely. The GET now calls testUpstream synchronously after the alias-resolved, opted-in, tunnel-up short-circuits. Probe failures land in routeErrorMap via logActivity (cross-node already did this; same-node timeout/error paths now log probe.fail in the same shape), so the state computation flips to "unreachable" on the same request that exposed the stopped upstream. A new routeProbeAtMap stamps freshness and surfaces in the response as lastProbeAt; the route detail sheet renders "Last probe <age> · <ms>" using the existing formatTimeAgo helper. |
||
|
Anso
|
554f662563 |
fix(mesh): surface stopped-stack opt-ins on routing node cards (#1098)
Add `currentlyResolvable: boolean` per entry in `MeshNodeStatus.optedInStacks`, derived from the existing alias cache so the new field stays consistent with `/api/mesh/aliases` without any extra Dockerode or cross-node inspect on the status path. The Routing tab renders an amber `suspended` pill for entries whose stack is opted in but currently has no running services, plus a single explanatory caption below the suspended list. Resolves the contradictory `Mesh stacks: 1 / Aliases: 0 / No mesh services on this node yet` copy on the node card when a meshed stack's container has been stopped; the misleading line is now only shown when the node truly has no opt-ins. The opt-in itself remains sticky: when the stack starts again, its aliases reappear automatically on the next refresh. Defensive de-dup in the UI filters suspended entries against the live alias snapshot to handle the transient gap where `/mesh/status` and `/mesh/aliases` return slightly inconsistent views from their separate fetches. Tests: - new `mesh-status-resolvability.test.ts` (6 cases) locks the resolvable / suspended / mixed / empty / per-node-scoping / stale-alias-no-phantom invariants for `getStatus`. - `mesh-topology-layout.test.ts` gains a `stacksKey` resolvability-flip case and a `meshNodeStateEqual` case asserting a resolvability flip on an otherwise identical stack registers as a state change so the topology layout re-runs. Operator docs gain one new troubleshooting accordion in `/docs/features/sencho-mesh.mdx` explaining the suspended state. |
||
|
Anso
|
f6e42535c8 |
fix(mesh): route peer→central traffic over the existing forward WS (#1094)
* fix(mesh): route peer→central traffic over the existing forward WS The reverse mesh callback path (`/api/mesh/proxy-tunnel-from-peer`) needed SENCHO_PRIMARY_URL on central plus a publicly reachable origin from the peer's perspective. In a typical homelab where central sits behind NAT, peer→central dispatch silently failed at the dialer's short-circuit and the headline "call any service on any node by hostname" worked one way only. The forward WS at `/api/mesh/proxy-tunnel` is already bidirectional end to end. Make the bridge a persistent control-plane primitive: dial every mesh-enabled proxy peer at startup, reconcile every 60 s, never idle-close. Peer→central traffic multiplexes over the same WS via `tcp_open_reverse`. Removed: - `meshProxyTunnelFromPeer.ts` WS handler and dispatch - `MeshCentralRegistry`, `PeerToCentralMeshSessionDialer` - `mesh_handshake` first-frame state machine in `meshProxyTunnel.ts` - `maybeSendBootstrap`, `buildHandshakeFrame` in the dialer - `mesh_proxy_callback_bootstrap` capability and `maybeWarnUnsetPrimaryUrl` - `mesh_centrals` table (drop migration; greenfield, no users) - `PilotTunnelManager.replaceOrRegisterProxyBridge` (dead after handler removal) - twelve associated unit/integration tests plus the peer-recovery branch in `MeshService.openCrossNode` Added: - `MeshService.proactiveBridgeFanout` selects every mesh-enabled proxy peer (no longer gated on `mesh_stacks` rows) - `startBridgeReconcileLoop` runs the fanout every 60 s (override via `SENCHO_MESH_RECONCILE_INTERVAL_MS`) - `MeshProxyTunnelDialer` default idle TTL is now `0` and exposes `isDialing(nodeId)` for the status surface - `MeshNodeStatus.reverseCallbackStatus` discriminator (`connected | connecting | unavailable | not_applicable`) surfaced via `/api/mesh/status` and rendered as a pill in the Routing tab - `openCrossNode` error message distinguishes "no proxy target" from "waiting for central to dial the reverse bridge" - New tests: `mesh-service-proxy-tunnel-reconcile`, `mesh-status-reverse-callback`, `mesh-proxy-tunnel-dialer-no-idle-close` SENCHO_PRIMARY_URL is no longer required for any mesh function. * fix(mesh): rewrite proxy-tunnel reconcile test contents The previous commit renamed the file but the rewritten test bodies stayed unstaged on top of the rename. This commit lands the actual rewrite: the fanout assertion now requires every mesh-enabled proxy peer to be dialed, not just those with `mesh_stacks` rows, and adds a reconcile-tick repeated-call test. |
||
|
Anso
|
1d7418a233 |
fix(mesh): recompose previously-meshed containers when alias set changes (#1090)
When a stack is opted into the mesh, every previously-meshed stack's override.yml on disk gains the new alias, but the running containers keep their stale /etc/hosts because extra_hosts is read at container creation time. Cross-stack DNS for the new alias silently fails until an operator manually redeploys each prior stack. Complete the cascade: after regenerateOverridesAcrossFleet finishes, fan out triggerRedeploy across every (node_id, stack_name) tuple in mesh_stacks, skipping the just-opted-in pair (the explicit triggerRedeploy at the end of optInStack covers it). Mirror the same cascade in optOutStack so the dropped alias exits every container's /etc/hosts. triggerRedeploy already dispatches local vs remote, already wraps runRedeploy in fire-and-forget error logging to the mesh activity ring and audit log, and already enforces compose policy gates. The cascade just reuses it. regenerateAllOverrides (boot and manual /regen-overrides) is intentionally NOT covered: a Sencho restart must not force a fleet-wide recompose; override files alone are sufficient there. Adds four unit tests in mesh-service.test.ts: cascade fires for every prior meshed stack on opt-in, no double-redeploy of the just-opted-in tuple, cascade is a no-op when no peers exist, cascade fires for every survivor on opt-out. |
||
|
Anso
|
578cac89da |
fix(mesh): surface data-plane failures in health, meta, and Routing tab (#1088)
The three previously-silent console.warn paths in MeshService.setupMeshNetwork now route through a typed recordSetupFailure helper that classifies the failure (subnet_invalid, subnet_overlap, subnet_mismatch, ip_in_use, attach_failed, not_in_docker), emits a mesh.disable activity entry at the matching level (error for real failures, warn for the expected dev-mode not_in_docker case), and strips mesh_proxy_callback_bootstrap from advertised capabilities via CapabilityRegistry. /api/health gains a mesh.dataPlane block carrying the typed status. /api/mesh/status carries localDataPlane at the top level so the Routing tab renders a red banner with an operator-actionable recovery hint (set SENCHO_MESH_SUBNET to a free /24 and recreate the container) when the data plane is down. The success path re-enables the capability and flips the status to ok. Generalizes the previously-documented F-0 failure mode (IP-in-subnet collision) to also cover the subnet-pool-overlap case where another Docker bridge on the host already owns the requested CIDR. |
||
|
Anso
|
5bc4add099 |
refactor(mesh): retarget proxyFetch null-target throw to no_target (#1083)
`MeshService.proxyFetch` previously threw `MeshError('push_failed', ...)`
when `NodeRegistry.getProxyTarget(nodeId)` returned null, but semantically
no push was attempted: the target simply did not exist. Callers had to
special-case the code and explain the overload in a comment.
Retarget the throw to the existing `'no_target'` variant in
`MeshErrorCode`. `listStacksOnNode` matches the new code directly; the
explanatory comment is gone since the code reads for itself.
`MeshError('push_failed', ...)` continues to signal real outbound push
failures: `applyLocalOverride` (mesh data plane not yet ready on the
receiving node) and `pushOverrideToNode` (HTTP 404 from an outdated
remote, non-OK HTTP response). Those sites are unchanged.
Operator-visible behavior is unchanged in the steady state. The rare
"remote disappeared between opt-in start and override push" case now
returns HTTP 400 with `code: 'no_target'` instead of 503; the response
body carries a clear message ("no proxy target for node N") so clients
can disambiguate.
Test plan
- `npx tsc --noEmit`: clean.
- `npx vitest run src/__tests__/mesh-list-stacks-remote.test.ts`: 8/8 green
(6 existing + 2 new cases pinning the catch-arm semantics).
- `npx vitest run src/__tests__/mesh-service.test.ts src/__tests__/mesh-inspect-remote.test.ts`:
56/56 green.
- Code reviewed; findings applied.
Notes
- PR #1082 (refactor: route inspectStackServices through proxyFetch) needs
a rebase + one-line update to its catch arm before merging on top of
this change.
|
||
|
Anso
|
dcc43205e0 |
refactor(mesh): route inspectStackServices through proxyFetch (#1082)
inspectStackServices was constructing its Authorization, x-sencho-tier,
and x-sencho-variant headers inline. proxyFetch is the centralized
helper used by listStacksOnNode, pushOverrideToNode, triggerRemeshRedeploy,
and removeOverrideFromNode. Equivalent today for a GET-no-body call, but
any future header added to proxyFetch (audit context, license fingerprint,
request id) would silently miss inspectStackServices.
Wire output is byte-identical: same URL, same method, same three headers,
same 5 s timeout. The new catch arm converts MeshError('push_failed') back
to the existing warn + return [] contract callers (optInStack,
refreshAliasCache) rely on. Pattern mirrors listStacksOnNode.
|
||
|
Anso
|
a32183d198 |
fix(mesh): cascade override regen across all meshed nodes on opt-in/opt-out (#1079)
Opting a stack into mesh on node A now propagates the new alias to every meshed node's override file, not just stacks on node A. Same on opt-out. Pre-fix, optInStack and optOutStack called regenerateOverridesForNode which only walks db.listMeshStacks(nodeId). Other meshed stacks on other nodes did not learn about the new alias until something else re-pushed their overrides; operators worked around it with a manual POST /api/mesh/regen-overrides plus a redeploy of each affected stack. New regenerateOverridesAcrossFleet helper iterates db.listMeshStacks() (no arg, fleet-wide) under Promise.allSettled, skipping the (nodeId, stackName) tuple opt-in just pushed loudly. Per-stack failures surface as forwarder.error activity events, matching the pattern used by regenerateAllOverrides. Offline remote nodes leave stale overrides until the next opt-in/opt-out, the next tunnel reconnect, or a manual rerun of the regen-overrides endpoint. regenerateOverridesForNode stays for the tunnel-up retry listener: a reconnecting pilot only needs its own stacks re-pushed, not the fleet. Tests: redirect five existing spies in mesh-service.test.ts to the new method; add two new cases asserting cross-node cascade for opt-in and opt-out. 52 tests in mesh-service.test.ts pass. |
||
|
Anso
|
54d84a7edb |
fix(mesh): skip peer-side recovery in openCrossNode on central instances (#1068)
The R1-A2 symmetric-callback work added a peer-side recovery branch to
MeshService.openCrossNode: when reverseDialer is null, kick
PeerToCentralMeshSessionDialer.ensureSession so the peer re-opens its
callback WS to central before the cross-node dispatch.
The guard `!this.reverseDialer` did not distinguish "I am a peer waiting
for central to dial in" from "I am central, and the reverseDialer concept
does not apply to me." On central, reverseDialer is always null (central
is the bridge initiator side and never has another node dial into it), so
central entered the recovery branch on every forward dispatch, found no
session (central has no mesh_centrals row), and destroyed the inbound
socket with route.resolve.fail forward-from-peer no_session.
Gate the branch on `MeshCentralRegistry.getActive() !== null` so it only
runs when this Sencho is acting as a proxy-mode peer that has actually
been bootstrapped. Central falls straight through to dialMeshTcpStream.
The peer cold-start scenario the comment originally described is still
covered: a proxy-mode peer with a cached mesh_centrals row but no live
WS to central will fire the recovery branch as before.
Adds two regression tests:
- central path (no mesh_centrals row, no reverseDialer) skips recovery
and calls dialMeshTcpStream
- proxy-peer path (mesh_centrals row + ensureSession returns null)
still emits route.resolve.fail forward-from-peer no_session
Existing tests in the "openCrossNode (BUG-4)" block already used a stub
reverseDialer to bypass the recovery branch, which is why the central
regression was not caught pre-release. The new tests deliberately leave
reverseDialer null to exercise the gating.
|
||
|
Anso
|
cf618dd866 |
feat(mesh): symmetric WS dial for proxy-mode mesh peers (#1066)
* chore(mesh): foundation for symmetric callback dial Adds the data-plane scaffolding that the symmetric callback dial fix builds on: - mesh_centrals table for peer-side bootstrap material - MeshCentralRegistry service (upsert/getActive/clear/markUsed/markRejected) - PilotTunnelManager kind discriminator and replaceOrRegisterProxyBridge - mesh_proxy_callback_bootstrap capability registration - MeshProxyTunnelDialer reason-tagged proxy-bridge-down events from a single tearDownBridge emission point - Reactive redial scheduler that skips idle and auth_failed reasons * feat(mesh): add reverse-direction activity log entries (closes R1-B) acceptReverseLocal now emits route.resolve.ok with direction=reverse on connect ack and route.resolve.fail with direction=reverse plus reason=container_not_found / connect_error pre-connect. Post-connect close/error stays silent. Reuses existing event types via the new details.direction discriminator so frontend filters are unaffected. * feat(mesh): add peer-to-central callback dial path (closes R1-A2) Closes the architectural gap where proxy-mode mesh peers could not re-establish their tunnel to central after any non-idle bridge teardown (idle close, network blip, central restart, peer reboot). Central remains the hub for the data plane; the change is purely about WS initiation. Symmetric WS initiation, asymmetric protocol roles. Central retains PilotTunnelBridge ownership; peer retains TcpStreamSwitchboard + reverseDialer ownership. Central bootstraps callback credentials over the first authenticated central-initiated mesh tunnel via a one-shot mesh_handshake JSON frame; peer persists the material in a new mesh_centrals SQLite table and dials central's new /api/mesh/proxy-tunnel-from-peer endpoint when local cross-node traffic needs a bridge and none is live. Mesh_tunnel JWT (HS256, signed with auth_jwt_secret) carries scope, audience, issuer (central instance id), peer api_token fingerprint, kid. Validation on inbound peer dial: algorithm pin, signature, scope, audience, instance, time bounds, node existence and mode, fingerprint match. Failures return HTTP 401 with a machine-readable reason; peer routes the response per a clear-vs-keep cache matrix. Triggers proactive bootstrap on mesh-enable and api_token rotation; central startup fans out to mesh-enabled proxy-mode nodes with mesh_stacks rows (throttled, fire-and-forget). Reactive redial on non-idle bridge loss. Capability-gated handshake send (mesh_proxy_callback_bootstrap) makes the upgrade path safe against older peers in mixed-version fleets. Adds peer-side /api/system/pilot-tunnels centralCallback diag block, bounded counter metrics for bootstrap and dial events. SENCHO_PRIMARY_URL preflight warning when unset on a central with mesh-enabled proxy nodes. Tested with unit suites for the validation chain, registry, manager, and both dialers; integration tests for bootstrap E2E (asserts protocol-role invariant), api_token rotation, instance id change, version skew, and pilot-mode regression. * fix(mesh): green CI on the symmetric callback branch Two independent CI failures, both surgical: 1. Backend tests (11 fails): four mesh test files called setupTestDb in beforeEach. setupTestDb does not reset the DatabaseService singleton, so the per-test afterEach rm of the previous tmpdir left the singleton connection pointing at a deleted file. The next beforeEach's line-55 write threw SQLITE_READONLY_DBMOVED on Linux. Windows file-lock semantics hid this locally. Hoist setupTestDb / cleanupTestDb to file-scope beforeAll / afterAll; per-test state resets stay in beforeEach. Matches the convention in the eight mesh test files that already pass. 2. CodeQL (4 high alerts): js/insufficient-password-hash flagged sha256(api_token) at four sites. The api_token is a 256-bit opaque bearer (sen_sk_-prefixed), not a human password; sha256 is the correct fingerprint primitive for binding the mesh_tunnel JWT to a specific token. Add the two production files plus the two test files that mint the fingerprint to the existing path-scoped query-filter for that rule. * fix(mesh): drop unused afterEach import and revert dead codeql config ESLint flagged afterEach as unused in mesh-central-registry.test.ts:1 after the previous commit hoisted setup/teardown to file-scope beforeAll/afterAll. Remove from the vitest import line. Revert the codeql-config.yml additions from the previous commit. The paths: sub-key under query-filters > exclude is not a documented CodeQL feature and silently no-ops. The four js/insufficient-password-hash alerts on api_token fingerprinting are tracked as dismissed false positives in the GitHub Security tab rather than via dead config. |
||
|
Anso
|
489aab4516 |
fix(mesh): testUpstream dials via ensureBridge so proxy-mode targets are probeable (#1057)
testUpstream short-circuited on hasActiveTunnel + getBridge (pilot-only) when probing a cross-node alias. Proxy-mode remotes never have a "pilot tunnel" in the manager's sense, so any operator hitting "Test alias" on a proxy-mode target returned tunnel_down even when the cross-node dial via dialMeshTcpStream worked fine. Replace with await ptm.ensureBridge(target.nodeId), matching the mode- agnostic dispatch dialMeshTcpStream already uses. ensureBridge returns an existing pilot tunnel, an existing proxy bridge, or asks MeshProxyTunnelDialer to dial on demand. The 'no pilot tunnel' vs 'no bridge' distinction goes away (they meant the same thing). Adds two test cases: probing a proxy-mode remote via on-demand bridge, and the tunnel_down path when ensureBridge yields null. |
||
|
Anso
|
3d5f0ffacd |
fix(mesh): reject opt-in when every service has empty ports (#1056)
optInStack rejected empty service lists but not the case where services exist with every ports: []. The newPorts Set ended up empty, the function sailed past collision checks, wrote a mesh_stacks row, and produced an empty-aliases override. The stack then appeared online in the Routing tab but no traffic actually routed. Add a check after building newPorts: throw no_target with a clear message before the SENCHO_LISTEN_PORT collision check. Reuses the existing MeshError code (semantically: no routable target) so the frontend's error toast path is unchanged. |
||
|
Anso
|
6185206323 |
fix(mesh): proxy peer learns its central-namespace nodeId at tunnel upgrade (#1055)
Proxy-mode peers receive an alias overlay from central with nodeIds in central's namespace, but had no source for their own nodeId in that namespace. selfCentralNodeId fell back to the peer's local DB default (always 1), and handleAccept falsely matched cross-node aliases (nodeId=1, central's Local) against the peer's self-id. Cross-node aliases routed to openSameNode and surfaced as route.resolve.denied. Central now appends ?nodeId=<central-namespace-id> to the proxy-tunnel WS URL. The peer's upgrade handler parses it with a strict integer regex, installs into MeshService before the reverse dialer is registered, and clears on disconnect. handleAccept resolves self-id in priority order: proxy-tunnel install > SENCHO_ENROLL_TOKEN > local default. A non-null overwrite to a different value warns loudly (misconfigured deployment). Missing or malformed query param logs a warn and proceeds, leaving reverse-direction dispatch undefined until both sides are upgraded. Adds 5 test cases: nodeId install + teardown, missing/malformed input (including decimal, leading zeros, exponent, whitespace), overwrite warn, and CAS-rejected reverse-dialer leaving the slot unchanged. |
||
|
Anso
|
946db9df52 |
fix(mesh): accept node_proxy at the proxy-tunnel WS upgrade (#1050)
The mesh proxy-tunnel WS handler was gated on full-admin api_token scope only, so node_proxy JWTs (the credential the Add Remote Node dialog tells operators to generate via Settings -> Nodes -> Generate Token) were silently rejected with HTTP 403. Operators followed the dialog's instructions, enrolled the node cleanly, saw reachableMode='proxy' with no negative badge, opted a stack into mesh, watched the redeploy complete, and only then discovered no bytes flow. node_proxy already authorizes every /api/* surface on the remote (deploy, exec, host console, filesystem), which is a strict superset of what the mesh proxy-tunnel does. Gating mesh more strictly was theatre, not security, and it created a UX trap with no in-product signal pointing at the scope mismatch. Change the upgrade dispatcher to accept any machine-to-machine credential: node_proxy JWT or full-admin api_token. Session cookies and restricted api_token scopes (read-only, deploy-only) remain rejected through the same paths as before. Adds positive/negative coverage in upgrade-order.test.ts for all four credential shapes so the gate stays pinned against future re-tightening. Updates user-facing copy in the mesh docs and Settings -> API Tokens description so the documented path matches the actual code. If we ever introduce a demoted node_proxy variant (audit-only, read-only fleet member), the right move is differentiated node_proxy scope claims inside the tunnel JWT (tracker F-A3), not requiring a separate token type for mesh. |
||
|
Anso
|
a38a3e0226 |
feat(mesh): route mesh traffic over Distributed API remotes (#1048)
* refactor(mesh): extract shared TCP stream switchboard
Pull the `tcp_open` / `tcp_open_ack` / `tcp_open_reverse` / `tcp_close`
+ `TcpData` handling out of the pilot agent into a reusable module so a
second caller (the upcoming proxy-mode WS handler) can run the same
frame parser, stream allocator, idle timers, and Compose-label
resolver. One parser, two callers, zero drift on a
security-sensitive protocol surface.
The pilot agent keeps its public `openMeshTcpStream` API and delegates
to a per-connection switchboard constructed in `connect()` and torn
down in `cleanupAfterDisconnect`. Existing reverse-stream and resolver
tests are rewritten to exercise the shared module directly.
* feat(mesh): route mesh traffic over Distributed API remotes
Sencho Mesh now works against remotes in Distributed API mode in
addition to Pilot Agent mode. Central opens a short-lived WebSocket
tunnel to the remote's new `/api/mesh/proxy-tunnel` endpoint on demand
and tears it down after 5 minutes of idle (configurable via
`SENCHO_MESH_PROXY_TUNNEL_IDLE_MS`). Mesh dispatch is mode-agnostic at
the routing layer; `PilotTunnelManager.ensureBridge` resolves an
existing tunnel or asks the new `MeshProxyTunnelDialer` to dial.
The Routing tab badges now use a `reachableMode` classifier: `★ Local`
for local, `pilot offline` red badge for a pilot with a down tunnel,
and a new `unreachable` red badge surfacing the specific reason
(missing token, scope not full-admin, TLS failure, remote does not
support proxy mesh). Distributed API remotes with valid credentials
show no negative badge; the tunnel opens on first dial.
Auth: the proxy-tunnel WS upgrade requires an `Authorization: Bearer`
API token with the `full-admin` scope. Lower-scoped tokens are
rejected at upgrade time so a leaked read-only token cannot reach the
mesh data plane.
Bidirectional: the proxy-mode WS handler registers itself as the
local `MeshService` reverse dialer (compare-and-swap), so meshed
containers on a Distributed API remote can dial cross-node aliases
via `tcp_open_reverse` over the same tunnel. Cross-node relays
(`PilotTunnelBridge.acceptReverseRelay`) await `ensureBridge` so
proxy-to-proxy mesh works without standing tunnels.
* docs(mesh): describe Distributed API mesh and unreachable troubleshooting
Refresh the user-facing mesh documentation to reflect that mesh works
over both Pilot Agent and Distributed API remotes. Adds a
Troubleshooting accordion entry for the new `unreachable` Routing tab
badge so operators can match a tooltip reason ("api token rejected
(scope must be full-admin)", "remote does not support proxy mesh",
"TLS handshake failed", "api_url not set", "api token missing") to a
concrete fix.
* refactor(mesh): apply /simplify review findings
Five cleanups surfaced by the post-implementation code review pass.
No behaviour change for fleets running on `main`; only internal
structure improves.
- **Deterministic container IP shared.** Extract the compose-default
network preference (`pickContainerIp`) and the conventional-name
fast path (`lookupContainerIp`) into `backend/src/mesh/containerLookup.ts`.
Both `MeshService.resolveContainerIp` (existing same-node fast
path) and the switchboard's `resolveByComposeLabels` (proxy/pilot
inbound dial) now use the same logic. Earlier the switchboard
helper grabbed the first `Object.values(Networks)` IP, which
could flip across daemon versions on multi-network containers;
fixed.
- **WS URL upgrade extracted.** New `backend/src/utils/wsUrl.ts`
exposes `httpUrlToWs(baseUrl)` that maps `http://` to `ws://` and
`https://` to `wss://`. Used in both `pilot/agent.ts` and the
proxy-tunnel dialer. The previous inline `replace(/^http/, 'ws')`
silently downgraded `https://` to `ws://` (cleartext).
- **`computeReachable` takes the pre-fetched node.** `getStatus`
already iterated `db.getNodes()`; the helper previously re-queried
by id per node (N+1 reads). Pass the row in.
- **Reachable-reason ternary -> const map.** Replace the nested
ternary in `MeshService.computeReachable` with a
`Record<DialFailureCode, string>` lookup keyed on the dialer's
failure code.
- **Activity-type ternary -> const map.** Same flattening inside
`MeshProxyTunnelDialer.logActivity`.
All mesh tests pass (85/85). Full backend suite green (2126/2126).
* fix(mesh): cache proxy dial failures and contain WS handshake errors
`ensureBridge` now consults the recent-failure cache before dialing so a
continuous mesh workload against a misconfigured proxy-mode remote does
not produce one upgrade attempt per cross-node TCP open. `recordFailure`
emits at most one activity-log entry per cache window per (nodeId, code)
so a connect-loop on a single bad remote cannot flush the ring buffer.
Failure messages run through `redactSensitiveText` before reaching the
log so any embedded Bearer / JWT / inline-URL credentials are scrubbed.
`stop()` clears the inflight map and `dial()` checks `this.stopped`
between awaits so a shutdown does not leak a half-opened bridge.
When `awaitOpen` rejects (401, 4403, TLS), the dialer attaches a noop
'error' listener before calling `ws.close()`. Without it the ws library
emits a tail 'error' on a still-CONNECTING socket that propagates as an
unhandled exception. Surfaced by a live-network test against a real
proxy-mode peer.
Adds `mesh-proxy-tunnel-live.test.ts` (skipped unless MESH_AUDIT_URL
and MESH_AUDIT_TOKEN_FILE env vars are set) covering both the happy
path and the auth-rejected path against an actual remote Sencho.
* feat(mesh): instrument proxy tunnel observability
Adds three counters to `PilotMetrics`:
- `proxy_bridges_total` (incremented on `registerProxyBridge` success)
- `proxy_dials_failed` (every failed dial attempt, not deduped)
- `proxy_idle_closes` (idle-sweep teardowns)
All three surface automatically via `GET /api/system/pilot-tunnels`
since the route returns the full `Counters` snapshot.
Gates two pre-existing always-on `console.warn` calls in
`tcpStreamSwitchboard.ts` (mid-stream socket errors and Docker resolve
failures) behind `isDebugEnabled()`. Both fire on per-stream events and
would otherwise violate the diagnostic-log safety rule under load.
Adds `mesh-tcp-stream-switchboard.test.ts` covering the forward
`tcp_open` path: real localhost dial, resolver errors (no_target,
denied), per-tunnel cap saturation, frame-routing fall-through
invariants, `tcp_close` socket teardown.
Drops `MESH_CONNECT_TIMEOUT_MS` from the public surface; only the
switchboard itself uses it.
* fix(mesh): tighten Routing tab unreachable handling
`RoutingNodeCard.tsx`: the **Add stack to mesh** button now disables
when `reachableMode === 'unreachable'`, matching the existing
TogglePill behavior. Without this, an operator on an unreachable node
could open the opt-in sheet, confirm, and watch the redeploy proceed
against a target whose mesh data plane will silently fail to route.
Also drops the leftover `status.nodeId !== -1` guard on the pilot-
offline badge.
`MeshService.ts`: renames `isMeshReachable` to `isMeshConfigured` with
the new predicate that returns true for proxy-mode remotes whose creds
are valid (the tunnel is opened on demand). `getRouteDiagnostic` now
distinguishes "routable" (configured) from "pilotLive" (live tunnel
state, only meaningful for pilot mode); without the split, every idle
proxy-mode route would report `tunnel down`.
`MeshService.setReverseDialer` warns when the unconditional install
path silently overwrites a non-null current dialer. By topology a
Sencho is either pilot or central, so the branch flags a misconfigured
deployment rather than an expected race.
Drops the dead `export { ReverseTcpStreamHandle }` re-export from
`pilot/agent.ts`; its only consumer imports straight from
`mesh/tcpStreamSwitchboard.ts`. Fixes a stale doc comment in
`MeshService.ts` that referenced the wrong source file.
Adds `mesh-proxy-tunnel-handler.test.ts` covering the WS handler
lifecycle: pilot-mode 404 rejection, post-upgrade reverse-dialer
install, concurrent-upgrade 1013 rejection (single-tenant slot), and
error-path teardown.
Updates the troubleshooting accordion in the user docs to mention the
disabled-state behavior.
* fix(mesh): close ESLint and CodeQL findings on the proxy-tunnel diag logs
Remove the unused `reverseDialer` local in `meshProxyTunnel.ts`; the
closure target is `localDialer` above and this assignment was always
dead. Strip line breaks inline on the three `MeshProxyTunnelDialer`
diag log lines so CodeQL `js/log-injection` data flow recognises the
sanitisation that `sanitizeForLog` already performs.
No behaviour change; the diag logs render the same characters they
do today.
|
||
|
Anso
|
b1c5fe8391 |
fix: harden deploy enforcement paths (#1030)
* fix: harden deploy enforcement paths * fix: update Docker toolchain to Go 1.26.3 * fix: repair Dockerfile tr argument split across lines * fix: bump protobufjs to clear npm audit high-severity advisories * fix(test): add execFile to child_process mock in compose-images test * fix: resolve merge conflicts with main * fix: resolve merge conflicts with main * fix: resolve merge conflicts with main |
||
|
Anso
|
b8af40d2b4 |
fix(mesh): enumerate stacks on remote pilot nodes for opt-in sheet (#1025)
The mesh opt-in sheet showed "No stacks deployed on this node yet" for every remote pilot because GET /api/mesh/nodes/:nodeId/stacks called FileSystemService.getInstance(nodeId).getStacks() unconditionally, which always reads central's own filesystem regardless of which node the operator targeted. Fix dispatches through the proxy chain when the targeted node is remote, mirroring the C-3 pattern that already governs every other mesh endpoint needing data from a remote Sencho: - New endpoint GET /api/mesh/local-stacks (Admiral-gated) returns the bare stacks list from THIS Sencho's own filesystem. Mirrors the precedent set by /api/mesh/local-services/:stackName. - New MeshService.listLocalStacks() and MeshService.listStacksOnNode() helpers; the latter dispatches local vs remote and degrades to an empty list on transport failure (no proxy target, non-2xx response, malformed body). - Refactored route delegates the stack-name list to listStacksOnNode; central's mesh_stacks DB is still authoritative for the opt-in flag set per C-3. Tests cover the local path, the remote-OK path with header forwarding, non-2xx, no-target (pilot tunnel down), malformed bodies, and unknown node ids. |
||
|
Anso
|
41a1df279d |
fix(mesh): fix pilot handleAccept dispatch by deriving self nodeId from enrollment token (#1023)
On a pilot node, handleAccept compared target.nodeId (from central's perspective via pilotAliasOverlay) against NodeRegistry.getDefaultNodeId() which always returns 1. This inverted same-node vs cross-node dispatch: - central's alias (nodeId=1) matched localNodeId=1 -> openSameNode on pilot (no container -> silent close, 0 bytes) - pilot's own alias (nodeId=14) != 1 -> openCrossNode -> tunnel loop Fix: add resolveSelfCentralNodeId() that decodes the nodeId claim from SENCHO_ENROLL_TOKEN (present on every pilot; payload decode only, no signature verification). Returns getDefaultNodeId() on central (env unset). Cached in selfCentralNodeId at start(); handleAccept reads the cache. After this fix the reverse direction (pilot-prober -> central echo) routes through openCrossNode (reverse tunnel) and the same-node path (pilot-prober -> pilot echo) routes through openSameNode (local Dockerode dial). Closes BUG-3 banner data-path regression. Tests: 5 new cases covering resolveSelfCentralNodeId token extraction, fallback paths, and handleAccept dispatch routing on a simulated pilot. |
||
|
Anso
|
6ac7da978c |
fix(mesh): push alias port map to pilot so reverse-direction forwarder binds listeners (#1021)
* fix(mesh): apply D-1 pushed override on pilot deploys via file-presence fallback When isMeshStackEnabled returns false, ensureStackOverride now checks for an override file already on disk before returning null. On pilot nodes the mesh_stacks table is intentionally empty (opt-in state lives on central per the C-3 design), so the DB gate blocked the override that central had already pushed via applyLocalOverride. File-presence is safe as the fallback because removeOverrideFromNode sends a DELETE to the pilot when a stack is opted out, so a stale file cannot survive past opt-out. Also aligns removeStackOverride to use path.basename consistently with all other override-path construction sites in the same file. Adds two tests: pilot node with a pushed file returns the path; pilot node with no file returns null. * fix(mesh): push alias port map to pilot so reverse-direction forwarder binds listeners Pilots have no mesh_stacks rows by design (C-3), so refreshAliasCache() produced an empty aliasByPort map and syncForwarderListeners() bound zero ports. Reverse-direction TCP probes (pilot prober -> central alias) failed with connection refused at the OS level because no listener existed. Fix: extend the D-1 override push payload to include the full MeshGlobalAlias records (host + port + routing metadata). The pilot stores these in a new pilotAliasOverlay map keyed by stack name. refreshAliasCache() merges the overlay after the (empty) DB loop, populating aliasByPort correctly. syncForwarderListeners() then binds the alias ports immediately after the push completes. removeLocalOverride() clears the overlay entry and re-syncs on opt-out. The two populations (DB rows on central, pilotAliasOverlay on pilots) are mutually exclusive by the C-3 invariant, so the first-write-wins port collision policy in refreshAliasCache is safe. |
||
|
Anso
|
c12b3ba995 |
fix(mesh): apply D-1 pushed override on pilot deploys via file-presence fallback (#1019)
When isMeshStackEnabled returns false, ensureStackOverride now checks for an override file already on disk before returning null. On pilot nodes the mesh_stacks table is intentionally empty (opt-in state lives on central per the C-3 design), so the DB gate blocked the override that central had already pushed via applyLocalOverride. File-presence is safe as the fallback because removeOverrideFromNode sends a DELETE to the pilot when a stack is opted out, so a stale file cannot survive past opt-out. Also aligns removeStackOverride to use path.basename consistently with all other override-path construction sites in the same file. Adds two tests: pilot node with a pushed file returns the path; pilot node with no file returns null. |
||
|
Anso
|
2d9e4d0480 |
fix(mesh): derive override services from compose file and surface cross-node dispatch (#1017)
* fix(mesh): derive override services from compose file and surface cross-node dispatch
Three fixes for issues found during Phase B end-to-end verification:
1. Override generator no longer queries Dockerode for services. Deploys
removed all containers in the project right before composeArgs ran,
so docker.listContainers({label: project=...}) returned [] and the
generator wrote services: {} which broke meshed user containers.
New helper getDeclaredStackServiceNames reads the local stack's
compose file via FileSystemService + YAML.parse and returns the
top-level services keys. Used from ensureStackOverride and
applyLocalOverride. Defensive fallback: when the read returns []
AND a previous override on disk has a non-empty services map,
keep the file and emit mesh.override.preserved.
2. Boot regen race against the pilot tunnel. regenerateAllOverrides
ran ~1s before the first pilot tunnel-up event, so pushes to
pilot-mode nodes failed with "no proxy target". MeshService.start
now also runs regenerateOverridesForNode(nodeId) inside the
tunnel-up listener, covering both the boot-race window and
mid-runtime tunnel reconnects. Idempotent.
3. openCrossNode emitted no activity log when a cross-node dial got
stuck waiting for tcp_open_ack. The forwarder accepted the TCP
handshake (so nc -v reported open) but logged nothing between
accept and close. Now logs route.dispatch on entry and emits
tunnel.fail after PROBE_TIMEOUT_MS if the agent never acks.
Timer is cleared on every tcpStream and src lifecycle event.
Tests: 8 new in mesh-service.test.ts cover the helper, the BUG-1
override write path, the defensive preservation fallback, the
tunnel-up regen handler, and both cross-node dispatch logging
behaviors. Full vitest suite green (1998 tests pass).
* fix(mesh): satisfy CodeQL path-injection model in new override-read paths
Two new fs.readFile sites added in the previous commit failed the
CodeQL check on the PR even though both paths were guarded by
isValidStackName + isPathWithinBase. The file's existing
applyLocalOverride pattern uses path.basename(stackName) before the
join (commented as "Recognized by CodeQL's path-injection model");
apply the same sanitizer to the new code paths.
- getDeclaredStackServiceNames: wrap stackName in path.basename when
building composePath.
- readExistingOverrideServiceNames: take (dir, stackName) instead of
a pre-built filePath so the sanitizer lives next to the read sink.
- ensureStackOverride: build the override file path with
path.basename(stackName) for symmetry with applyLocalOverride.
Runtime behavior unchanged (path.basename is a no-op for any name
that passed isValidStackName). Tests still green.
|
||
|
Anso
|
7cca68f84a |
fix(mesh): operator-triggered override regen and boot lifecycle hardening (#1016)
Audit-driven follow-up to the F6+F7 fix. Closes the High-severity findings
from the mesh-dial-path audit without expanding into the architectural
follow-ups (JWT TTL, TLS enforcement, per-stack JWT scope) which need
their own design discussions.
Changes:
1. New `POST /api/mesh/regen-overrides` Admiral-gated endpoint that calls
`MeshService.regenerateAllOverrides()` and returns a structured
`MeshRegenSummary { regenerated, failures, skipped, reason? }`. Lets
the operator rerun boot regen after fixing a remote node that was
offline at central startup, instead of opt-out + opt-in for every
meshed stack on that node. Audit-log row carries the outcome
(success / partial / skipped / error).
2. `regenerateAllOverrides` now aggregates per-stack outcomes into a
single summary log row (`mesh override regen complete: N succeeded,
M failed across K node(s)`) with `failedNodeIds` in `details`. Per-
stack warnings still emitted. When skipped because `senchoIp` is
null, emits an explicit warn instead of silently no-oping.
3. `MeshService.start()` now wraps `refreshAliasCache` and
`syncForwarderListeners` in their own try/catch so a throw from
either step is logged and the boot continues to override regen.
The closing log line surfaces data-plane state ("MeshService started
(data plane ok)" or "(data plane unavailable (<reason>))") so an
operator grepping startup output sees a half-init mesh immediately.
4. `sanitizeForLog` wrap on four previously unsanitized `err.message`
writes (cross-node tcpStream error handler + three probe error
paths). Container IPs and local socket details no longer leak into
the activity buffer or the probe response body.
5. Updated three existing F6 regression tests for the new return shape
and aggregated summary message. Added two new tests: version-skew
404-push handling, and concurrent opt-in during regen.
|
||
|
Anso
|
0947a80cda |
fix(mesh): trust central for cross-node dial auth and regenerate overrides at boot (#1014)
Two bugs in the same Phase D follow-up surface, fixed together because they both block declaring B-verify complete on the production fleet. Cross-node mesh dials returned `denied` at the agent. The pilot's `tcp_open` handler in `agent.ts::resolveMeshTarget` consulted the local SQLite `mesh_stacks` table, which is no longer written to under the post-Phase D control plane (state lives only on central). Drop the check. The pilot tunnel JWT (scope `pilot_tunnel`, signed with central's `auth_jwt_secret`) authenticates the caller; the same trust model already applies to filesystem ops, exec, and container control over the same tunnel. Threat-model trade-off: a leaked `pilot_tunnel` JWT or compromised central can now dial any compose-managed service on the pilot. Containers without `com.docker.compose.project` + `com.docker.compose.service` labels remain unreachable via this path. `MeshService.start()` did not regenerate compose override files at boot. After a Sencho restart with missing overrides on disk, meshed user containers had no `extra_hosts` / `networks: [sencho_mesh]` injection until each stack was opted out and back in. Add `regenerateAllOverrides()` that walks every `mesh_stacks` row and re-pushes via `pushOverrideToNode`. Best-effort: per-stack failures log to the mesh activity buffer; `MeshService.start()` is fire-and-forget at startup so a slow remote node does not delay boot. Tests: - `pilot-agent-mesh-resolve.test.ts` (new): mocks dockerode and proves `resolveMeshTarget` no longer returns `denied` with an empty `mesh_stacks` table. - `mesh-service.test.ts`: three new cases for `regenerateAllOverrides` - fan-out across the fleet, skip when `senchoIp` is null, log per-stack warning on push failure without throwing. |
||
|
Anso
|
23bbee4f45 |
feat(mesh): replace host-mode with shared sencho_mesh Docker network (#1009)
* feat(mesh): replace host-mode with shared sencho_mesh Docker network Phase D of the mesh redesign: drop the operator's `network_mode: host` requirement and the `host-gateway` extra_hosts pattern that did not work on cloud iptables-restrictive distros (OCI, etc.) or Docker Desktop. Each Sencho creates a shared `sencho_mesh` Docker bridge network on boot (default subnet 172.30.0.0/24, override via SENCHO_MESH_SUBNET), pins itself at `<network>+2`, and attaches every meshed user service to the same bridge. Compose overrides now emit IP-based `extra_hosts` plus a top-level `networks` block declaring `sencho_mesh` external. Override delivery: central renders for local stacks; for remote stacks it sends the fleet alias list to the remote's new `PUT /api/mesh/local- override/:stackName` endpoint, which renders against the remote's OWN local senchoIp and writes under its OWN DATA_DIR. Each node may use a different subnet without coordination beyond the env var. Opt-in / opt-out now trigger an automatic redeploy of the affected stack via the existing deploy code path (local: ComposeService; remote: HTTP POST through proxyFetch). The frontend opt-in sheet shows a confirmation modal (ConfirmModal) before the mutation. Failed redeploys emit both a mesh activity event and a durable audit-log row. Hardening: - Reserve port 1852 at opt-in (prevents user containers from racing the Sencho API listener). - ensureMeshNetwork refuses to continue if `sencho_mesh` exists with a mismatched subnet rather than silently routing to the wrong IP. - Idempotent network connect/disconnect helpers in DockerController. - optInStack rolls back the DB row if the just-inserted stack's override push fails (no half-states surviving across calls). - regenerateOverridesForNode runs in parallel and skips the just- pushed stack on opt-in. Operator template: drop `network_mode: host`, restore `ports: ["1852:1852"]`. Mesh now works identically on Linux LAN, OCI, and Docker Desktop without firewall changes. Docs: rewrite docs/features/sencho-mesh.mdx around the shared bridge network, document SENCHO_MESH_SUBNET, surface the host-network-service opt-in restriction, and cross-link with the Pilot Agent docs. BREAKING CHANGE: the operator's `docker-compose.yml` no longer uses `network_mode: host`. After upgrading, redeploy any meshed stacks once so they pick up the new IP-based override and join `sencho_mesh`. * fix(mesh): wrap stackName with path.basename in local-override fs ops CodeQL flagged js/path-injection on the new applyLocalOverride and removeLocalOverride methods because they are publicly reachable and its data-flow model does not recognize isValidStackName / isPathWithinBase as sanitizers. The validation IS sufficient (the allowlist regex blocks path separators, the path-prefix check blocks escape), but path.basename is a model CodeQL recognizes and is purely defensive: for any input that already passes isValidStackName, basename is the identity. |
||
|
Anso
|
10a469ecb4 |
fix(mesh): bind every alias port on every node, not just local-owned (#1004)
syncForwarderListeners filtered the bind set to ports owned by the local node. That broke cross-node mesh routing end to end: meshed containers' extra_hosts: <alias>:host-gateway entries resolve to the SOURCE node's gateway, so the source node is where the inbound TCP connection lands. With the filter, the source node never bound the remote alias's port and the connection went nowhere. The architecture model documented in docs/internal/architecture/ mesh.md (data flow step 3) is explicit: monolith's Sencho process accepts the connection on its MeshForwarder listener bound to port 5432 — where monolith is the source and opsix is the target. Every meshed node binds every alias port. handleAccept then resolves the alias and dispatches to openSameNode (when target.nodeId equals the local node) or openCrossNode (otherwise). Both branches were already correct; only the bind filter was wrong. Fleet-wide port collisions remain blocked at opt-in time (optInStack checks aliasByPort), so binding every alias port is unambiguous. New vitest case asserts the want-set includes both local-owned and remote-owned ports. Discovered while running PR 0 verification on v0.75.0 against Local + sencho-pilot-test: connections from a Local prober to the pilot's alias port hit Local's host with no listener. |
||
|
Anso
|
567e524450 |
feat(mesh): bidirectional routing via tcp_open_reverse and central relay (#1003)
* feat(mesh): bidirectional routing via tcp_open_reverse and central relay
Phase B of the mesh redesign. Adds the reverse-direction protocol so a
pilot's MeshForwarder can route cross-node traffic to central or to
another pilot. Central relays pilot-to-pilot streams transparently;
pilots keep their existing single outbound WS to central.
Protocol additions (backend/src/pilot/protocol.ts):
- TcpOpenReverseFrame { s, targetNodeId, stack, service, port } sent
agent to primary. Reuses existing tcp_open_ack, tcp_close, and
TcpData binary frames for the response and byte plane.
- AGENT_REVERSE_ID_BASE = 0x40000001 splits the 32-bit id space so
agent-allocated reverse stream ids never collide with primary-
allocated forward ids on the same tunnel.
- StreamIdAllocator now wraps to its configured start (not always 1)
so an allocator parameterized with the agent base stays in the
agent half across the wrap.
Agent (backend/src/pilot/agent.ts):
- New ReverseTcpStreamHandle exported class, EventEmitter facade
matching PilotTunnelBridge.TcpStream's surface (write, end,
destroy plus open, data, error, close events).
- Public openMeshTcpStream(target) allocates a reverse id, sends
tcp_open_reverse, returns the handle.
- onTcpOpenAckReverse handles inbound ack, dispatches open or
error+close to the matching handle.
- TcpData and tcp_close binary/JSON paths route ids in the reverse
range to reverseTcpStreams; existing primary-allocated paths
unchanged.
- streamCount, cleanupAfterDisconnect, onStreamIdle include reverse
streams. The allocator is reset to a fresh base on disconnect so
long-lived agents that reconnect many times do not drift up the id
space.
- startPilotAgent registers the agent as MeshService's reverse
dialer via lazy import.
Bridge (backend/src/services/PilotTunnelBridge.ts):
- Two new StreamState kinds: reverse_local (target = central, dial
Dockerode container IP) and reverse_relay (target = another pilot,
open a forward TcpStream on the target pilot's bridge).
- handleTcpOpenReverse validates the agent-id range and stream cap,
then dispatches to acceptReverseLocal or acceptReverseRelay via
lazy imports of MeshService, NodeRegistry, and PilotTunnelManager
(avoids module cycles).
- acceptReverseLocal calls MeshService.resolveContainerIp (now
public), opens net.createConnection, splices bytes through the
tunnel. Pre-connect error handler is removed inside connect to
avoid double-firing with the mid-stream error path.
- acceptReverseRelay opens a forward TcpStream on the target
bridge, splices bytes between the two tunnels.
- Existing tcp_close and TcpData handlers extended for the new
state kinds. teardownStream extended.
MeshService (backend/src/services/MeshService.ts):
- resolveContainerIp made public so the bridge's local-target path
can dial the same shape.
- New reverseDialer field plus setReverseDialer setter. Pilot mode
registers the agent at boot; central mode leaves it null.
- New private dialMeshTcpStream dispatcher: pilot side uses the
reverse dialer, central side uses PilotTunnelManager.getBridge.
- openCrossNode refactored to call the dispatcher and use the
active-stream record's id for activity logging.
- New exported MeshTcpStreamLike interface that both
PilotTunnelBridge.TcpStream and ReverseTcpStreamHandle satisfy
structurally; ReverseMeshDialer is the contract for the setter.
Tests (3 files, 16 new cases, 172 total pass):
- pilot-protocol-tcp.test.ts: tcp_open_reverse round-trip; agent
base invariant.
- pilot-agent-reverse-stream.test.ts: openMeshTcpStream allocation,
ws-not-open, frame shape, write encoding, end emits tcp_close,
ack-success, ack-failure, low-id ack ignored, inbound TcpData
routing.
- pilot-bridge-reverse.test.ts: id-range validation, no-target
failure, successful local dial against a real upstream server.
Together with PR #1000 (Phase A) and PR #1001 (deps), Phase B
completes the central-pilot-pilot mesh routing matrix. Phase C
(mesh over proxy-mode remotes) is the planned follow-up.
* test(pilot): drop unused encodeJsonFrame import
Lint failed on pilot-agent-reverse-stream.test.ts after the test
changed from constructing tcp_open_reverse frames inline to driving
the agent's frame-dispatch path with synthetic objects. The import
is no longer referenced; ESLint's no-unused-vars rule rejected it.
|
||
|
Anso
|
f599110386 |
feat(mesh): collapse sidecar into Sencho process via in-process forwarder (#1000)
The separate saelix/sencho-mesh sidecar container is gone. The
forwarder logic that previously lived in mesh-sidecar/src/forwarder.ts
moves into the Sencho process as backend/src/services/MeshForwarder.ts,
a thin per-port net.Server lifecycle wrapper. MeshService implements
the host interface and owns resolve plus splice; MeshForwarder owns
listener boilerplate. One container per node, no separate image to
publish, no control WebSocket.
Operator-facing change: the Sencho container now runs in
network_mode: host so the forwarder can bind alias ports on the host
network where meshed containers' extra_hosts host-gateway entries
point. Without host network mode the listeners would land in the
container's namespace and inbound traffic from peers would never
reach them. The 1852:1852 port publish becomes a no-op under host
mode and is commented out in the operator template.
Same-node forward path now dials the target container's bridge IP
via Dockerode (preferring the compose default network for
deterministic selection across daemon versions) instead of 127.0.0.1.
The legacy 127.0.0.1 path only worked when the target service
published its port to the host; the IP path works regardless.
Cross-node mesh routing in this phase is central -> pilot direction
only via PilotTunnelManager.openTcpStream. Pilot -> central and
pilot <-> pilot via central relay land in Phase B with the
tcp_open_reverse frame.
Deletions:
- mesh-sidecar/ package entirely (Dockerfile, package, sources, tests)
- backend/src/websocket/meshControl.ts
- MeshService sidecar lifecycle: spawnSidecar, stopSidecar,
isSidecarRunning, mintSidecarToken, verifySidecarToken,
attachSidecarSocket, handleSidecarResolve, sendSidecar
- POST /api/mesh/nodes/:id/sidecar/restart route
- /api/mesh/control WS dispatch in upgradeHandler
Type cleanup: 'sidecar' literal removed from MeshActivitySource and
MeshProbeResult.where (also the frontend mirror). MeshNodeStatus
sidecarRunning becomes localForwarderListening (boolean | null) so
non-local nodes get a null instead of an unconditional false; the
honest semantic is "this view only knows the local forwarder state;
remote forwarder status lands in Phase B." MeshNodeDiagnostic
sidecar object becomes forwarder { listening, listenerCount }.
Frontend MeshDiagnosticsSheet drops the restart-sidecar action and
sidecar liveness card; surfaces forwarder state plus a "runs
in-process; no separate container" caption.
Resolves audit findings C-1 (data plane non-functional), C-2 (sidecar
control WS not loopback-enforced), C-4 (sidecar lifecycle Dockerode-
on-remote, PR #999 closed), and C-5 (saelix/sencho-mesh:latest
unreachable). C-3 (PR #992) is unchanged. M-12 (PR #994) is
unchanged.
|
||
|
Anso
|
34581e55e7 |
fix(mesh): treat local-node aliases as always reachable in diagnostics (#994)
MeshService.getRouteDiagnostic called PilotTunnelManager.hasActiveTunnel unconditionally for the alias's target node. Local nodes never establish pilot tunnels because they do not need them (mesh same-node traffic uses the localhost fast path), so the call always returned false and the diagnostic flipped to state: 'tunnel down' even on a working local route. The route detail sheet and node card consequently rendered every local mesh alias as a destructive 'tunnel down' pill. Added a private isMeshReachable helper that returns true for any local-type node and otherwise delegates to hasActiveTunnel. Used it in both getRouteDiagnostic and getStatus, replacing the inline ternary in getStatus that already had the right shape but lived as duplicated logic. The helper falls back to false when the node row is missing (orphaned alias from a partial cascade), matching the pre-existing conservative behavior. Verified live: GET /api/mesh/aliases/echo.audit-mesh-prod.Local.sencho/diagnostic previously returned state='tunnel down' on a healthy local route. |
||
|
Anso
|
e176ae9f17 |
fix(mesh): inspect remote stacks via HTTP proxy not Dockerode (#992)
MeshService.inspectStackServices used DockerController.getInstance(nodeId)
to enumerate compose-labeled containers, but NodeRegistry.getDocker
explicitly throws for any node with type='remote' by design. The throw
was silently caught and an empty array returned, so opt-in for any
remote node failed with a misleading "no running services" error and
refreshAliasCache only ever populated LOCAL aliases. Cross-node mesh
routing was therefore impossible end-to-end regardless of pilot-tunnel
state.
Split the inspector. inspectLocalStackServices keeps the Dockerode
listContainers path and always queries the local Docker daemon; it is
public so the new route can call it. inspectStackServices is now a
dispatcher: local nodes fall through to the Dockerode path, remote
nodes (proxy mode and pilot-agent) HTTP-fetch
/api/mesh/local-services/:stackName against the URL resolved by
NodeRegistry.getProxyTarget, with the persisted node_proxy Bearer
token and license tier headers attached. The remote's MeshService
enumerates its own LOCAL Docker daemon and returns the
{service, ports[]} envelope.
refreshAliasCache now inspects every opted-in stack in parallel via
Promise.allSettled so a slow or unreachable remote does not stall the
60-second loop. The new /api/mesh/local-services/:stackName route is
gated by requireAdmiral plus isValidStackName and always queries the
caller's own Sencho instance.
Together with PR #989 (proxy-side bridge dispatch) and PR #990
(agent-side loopback auth), this makes the central, bridge, agent,
local-Sencho mesh control plane functional end-to-end for both
proxy-mode and pilot-agent remotes.
|
||
|
Anso
|
7663f4cd8b |
feat(fleet): sencho mesh in traffic and routing tab (#858)
* feat(fleet): sencho mesh in traffic and routing tab Lights up Sencho Mesh: cross-node container forwarding rendered as if the container next to you were on localhost. Builds on the dormant TCP frame plumbing from the prior PR (pilot tunnel TCP frames + sencho-mesh sidecar package) and exposes the Admiral-only orchestrator surface. Backend - New mesh_stacks table (per-node opt-ins) + nodes.mesh_enabled column via DatabaseService.migrateMeshTables. - MeshService singleton: sidecar lifecycle via Dockerode, opt-in/out with cascading override regeneration, request-based resolver from sidecar control WS, cross-node TCP forwarding via PilotTunnelManager (same-node fast path included), in-memory 1000-event activity ring buffer with durable mirror to audit_log for state-change events, per-node and per-route diagnostics, and the Test upstream probe. - MeshComposeOverride: pure YAML generator that injects extra_hosts using host-gateway. The user's docker-compose.yml is never mutated; overrides live under DATA_DIR/mesh/overrides. - ComposeService deploy/update splice the override file when the stack is opted in; non-mesh stacks behave identically to today. - Pilot agent resolveMeshTarget consults the local mesh_stacks table (defense in depth) and resolves Compose containers via Dockerode. - /api/mesh router with 13 Admiral-gated endpoints covering status, enable/disable, stack opt-in/out, alias listing, per-route diagnostic, Test upstream probe, per-node diagnostic, sidecar restart, activity log paginated and SSE. - meshControl WS slot at /api/mesh/control validates the mesh_sidecar JWT minted by MeshService; dispatched as upgrade slot 2 (canonical order preserved). Frontend - New Traffic Routing tab in FleetView, gated by isAdmiral and wrapped in AdmiralGate. Tab uses the cyan brand glyph and italic-serif state typography from the audit. - RoutingTab masthead with mesh activity drawer, per-node card grid with TogglePill, alias rows with five-state pill taxonomy (healthy / degraded / unreachable / tunnel-down / not-authorized), inline Test buttons. - Four sheets: opt-in picker with port-collision inline error, per-route detail with diagnostic + filtered activity, per-node diagnostics with active streams + resolver cache + restart action, fleet-wide activity log with filters. - meshRouteState helper centralizes pill-state mapping; pure-function tests cover all five states. Docs - User docs at /docs/features/sencho-mesh.mdx covering opt-in, troubleshooting, security model (4 guarantees + 4 explicit non-guarantees), and V1 limitations. - Internal architecture and runbook pages. - websocket-dispatch internal doc updated with the new slot. * fix(mesh): validate stack name before path use; fix test DB lifecycle Two surgical fixes against the prior PR. Path-injection (CodeQL js/path-injection): MeshService.optInStack, optOutStack, ensureStackOverride, and removeStackOverride now validate stackName via isValidStackName from utils/validation, reject malicious names at the API boundary, and additionally check isPathWithinBase on the resolved override file path for defense in depth. The dataflow from req.params.stackName to fs.writeFile no longer reaches an unsanitized path expression. Test DB lifecycle: mesh-service.test.ts used per-test setupTestDb / cleanupTestDb, which deletes the temp dir while DatabaseService still holds an open SQLite handle. On Linux CI this raises SQLITE_READONLY_DBMOVED on the next prepare() because the inode has been unlinked. Switched to file-scoped beforeAll/afterAll matching agents-routes.test.ts, with a per-test beforeEach that truncates mesh_stacks plus non-default nodes and resets the MeshService singleton in-memory state. Adds a new test case asserting the path-traversal rejection. * fix(compose): use discovered compose filename instead of hardcoded docker-compose.yml composeArgs() hardcoded `-f docker-compose.yml` for every deploy. Sencho writes its canonical compose file as `compose.yaml`, so any stack created via the UI failed to deploy with `open ...docker-compose.yml: no such file or directory`. When no mesh override applies, drop the explicit `-f` so docker compose's built-in discovery resolves the actual filename. When an override exists, look up the real base filename via FileSystemService.getComposeFilename() and pass both files explicitly. Also hoist the MeshService import to module top now that the dependency is known to be acyclic, and revert the matching unit-test assertion. |