mirror of
https://github.com/Studio-Saelix/sencho.git
synced 2026-07-28 04:38:59 +00:00
be22e3ded1f8bf8c4ecc66b77c12db96152b6dd2
461 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Anso
|
66b84932e0 |
feat(notifications): move Notification Routing to Skipper tier (#1145)
* feat(notifications): move Notification Routing to Skipper tier Notification routing is automation (route alerts to channels by rules), not enterprise compliance. Aligning the gate with Skipper makes the tier boundary read consistently with the rest of the automation surface (webhooks, auto-update, auto-heal, scheduled tasks). Backend: requireAdmiral -> requirePaid on the five /api/notification-routes endpoints. Dashboard configuration-status now exposes the routing-rules row to any paid tier. Frontend: settings registry tier flipped to skipper; the Admiral wrapper around NotificationRoutingSection is removed (the inner CapabilityGate stays, preserving forward-compat with older remote nodes). Tests: added a tier-enforcement describe block covering Skipper (200) and Community (403 PAID_REQUIRED on all five endpoints). Docs: refreshed alerts-notifications, licensing, overview, dashboard, troubleshooting, and reference/settings; cleaned one fence-spec line per Directive 31. * fix(notifications): address audit findings on tier-move PR Docs: rewrite three lines that survived the initial sweep. The dashboard "you do not see a locked placeholder" clause and the settings.mdx "hidden on Community and Skipper" phrase were Directive 31 fence-spec. The alerts-notifications troubleshooting note still said "an Admiral routing rule" and contradicted the tier move. Tests: the Community-negative cases on POST/PUT/DELETE/POST :id/test could not distinguish requirePaid from a stray requireAdmiral, because Community fails on the tier check before variant is read. Adding Skipper-positive coverage per endpoint locks the gate identity in. Replace the leaky mockReturnValueOnce with a per-test mockReturnValue plus an afterEach restore so spies cannot bleed across tests. |
||
|
Anso
|
535023b350 |
feat(files): open stack file explorer to every tier (#1144)
* feat(files): open stack file explorer to every tier
Drop the `requirePaid` guard from the seven stack-file write routes
(download, upload, write-content, delete, mkdir, rename, chmod) and
remove every matching `isPaid` check from the file-explorer frontend.
Stack edit permission (RBAC) continues to gate every write end-to-end.
The file explorer is the primary way a user touches a stack's on-disk
surface; gating it behind a paid tier conflicted with the principle
that Community covers single user-initiated actions while paid tiers
add automation and governance.
* docs(files): treat download as a read action, not a write
Download has no `requirePermission('stack:edit')` on the route and no
`canEdit` gate in the UI, so viewer accounts can download. Update the
top paragraph to list download under reads, and rewrite the
troubleshooting accordion to describe the actual gating (a file must be
selected) instead of asserting a role gate that does not exist.
* test(e2e): align stack-files spec with the new tier rule
The community-tier describe block asserted that the Upload control is
absent and the editor shows a `Read-only` chip; the admin-tier block
skipped on Community via `test.skip(tier !== 'paid')`. Both rules
reflected the previous gate, where writes required a paid tier.
Writes are now gated on the `stack:edit` role, not on the license tier.
Repurpose the community describe to assert that a Community admin
under a mocked community license still sees the Upload control and an
editable Save button. Drop the obsolete tier-skip in the admin describe
so upload, edit, delete, and download exercise on every tier. Update
stale comments to reference the role gate.
|
||
|
Anso
|
380ed6fd50 |
feat(cloud-backup): make Custom S3-compatible target available on every tier (#1143)
* feat(cloud-backup): make Custom S3-compatible target available on every tier Sencho Cloud Backup remains an Admiral feature; the bring-your-own-bucket Custom S3 target is now reachable on Community and Skipper as well. Backend splits the per-route Admiral gate into two helpers: operations that touch the saved provider use gateForCurrentProvider, PUT /config uses gateForRequestedProvider against the body. /provision and /usage stay requireAdmiral because they are Sencho-only by definition; GET /config is ungated so any tier can read its own stored configuration. Frontend drops the AdmiralGate wrapper on the Cloud Backup section, filters the Sencho provider option out of the dropdown for non-Admiral users, and gates the per-snapshot cloud-upload affordance on "cloud-backup configured" instead of Admiral tier. Dashboard Configuration row is no longer locked on lower tiers. Sidebar registry tier on cloud-backup goes from 'admiral' to null. Docs and licensing breakdown restate the rule once per page without fence-spec. * fix(cloud-backup): keep downgraded sencho config off the upload surface If an Admiral configured Sencho Cloud Backup and the license later drops to Skipper or Community, the saved provider is still 'sencho'. The FleetSnapshots cloud-upload affordance now requires either provider= custom (every tier) or provider=sencho with an active Admiral license, so a downgraded admin never sees an upload button that the backend would 403 on click. Also tidies the Fleet Backups doc, which still claimed the cloud-upload icon was Admiral only; the icon now renders whenever a Cloud Backup target is configured. |
||
|
Anso
|
c491d309c1 |
feat(schedules): make every scheduled action available on Skipper (#1141)
Collapse the Admiral carveout that restricted restart, prune, auto_backup, auto_stop, auto_down, and auto_start schedules to the Admiral variant. Scheduled Operations stays at Skipper+ (paid). The action picker now lists every supported operation for any paid admin, and the scheduler runner executes every action on either variant. |
||
|
Anso
|
8d1304b0df |
fix(hub-only-guard): reject hub-only collection paths without trailing slash (#1142)
HUB_ONLY_PREFIXES entries are stored with a trailing slash, but isHubOnlyPath() used a bare startsWith(). The collection paths /api/scheduled-tasks, /api/audit-log, and /api/notification-routes (no trailing slash) silently fell through the guard and could be forwarded by the remote proxy, bypassing the hub-only boundary on three path families. The matcher now accepts the exact prefix (slash stripped) in addition to the slash-suffixed prefix. Adds three regression tests covering the bare-collection-path 403 for each hub-only family. Confirmed the new tests fail without the matcher change and pass with it. |
||
|
Anso
|
6f301e005a |
feat(fleet): redesign Fleet Action cards to the System Sheet recipe (#1137)
Lift <FleetActionCard> primitive from audit §20 / DESIGN §9.12. Migrate the three v1 cards (stop-by-label, bulk-label-assign, prune-fleet-wide) to consume it: cyan rail on every card, action class as a chip, blast radius as a live readout in the toolbar, preview section replaces the warning banner, footer carries reversibility plus freshness. Drop the per-card tone rail, the icon prop, and cards/tone.ts. Add /fleet/labels/match-preview and /fleet/prune/estimate for the live blast readouts (chrome falls back to "preview unavailable" if either 404s). Add dryRun: true to the existing fleet-stop and fleet-prune endpoints so the Dry run button rehearses the full code path (locks, per-node fan-out, remote propagation) without firing the destructive leaf call. Result flows into the existing ResultsList. Extend <SheetSection> with optional meta. Add --action-transformative semantic token. |
||
|
Anso
|
d0e140444a |
feat(api-tokens): make API tokens available on every tier (#1136)
API tokens are credential management, not a tier-gated capability. Remove the Admiral gate from the POST/GET/DELETE handlers, drop the AdmiralGate wrapper from the settings UI, set the registry entry's tier to null so the tab renders on every tier, and update the docs Note to state availability plainly. The three permission scopes (read-only, deploy-only, full-admin), the 25-token-per-user cap, the per-token 200 req/min rate limit, the sen_sk_ prefix format, and the SHA-256 hashed storage are all unchanged. The Vitest suite now runs at Community tier to prove every code path works without a paid license. A new "API token tier accessibility" describe block mints all three scopes via POST /api/api-tokens to lock the behavior. |
||
|
dependabot[bot]
|
3eafd1ab89 |
chore(deps): bump the all-npm-backend group in /backend with 10 updates (#1131)
Bumps the all-npm-backend group in /backend with 10 updates: | Package | From | To | | --- | --- | --- | | composerize | `1.7.5` | `1.7.6` | | [express-rate-limit](https://github.com/express-rate-limit/express-rate-limit) | `8.5.1` | `8.5.2` | | [isomorphic-git](https://github.com/isomorphic-git/isomorphic-git) | `1.37.6` | `1.38.1` | | [ldapts](https://github.com/ldapts/ldapts) | `8.1.7` | `8.1.8` | | [@types/node](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/node) | `25.7.0` | `25.9.1` | | [eslint](https://github.com/eslint/eslint) | `10.3.0` | `10.4.0` | | [typescript-eslint](https://github.com/typescript-eslint/typescript-eslint/tree/HEAD/packages/typescript-eslint) | `8.59.3` | `8.59.4` | | [vitest](https://github.com/vitest-dev/vitest/tree/HEAD/packages/vitest) | `4.1.6` | `4.1.7` | | [@aws-sdk/client-ecr](https://github.com/aws/aws-sdk-js-v3/tree/HEAD/clients/client-ecr) | `3.1045.0` | `3.1051.0` | | [@aws-sdk/client-s3](https://github.com/aws/aws-sdk-js-v3/tree/HEAD/clients/client-s3) | `3.1045.0` | `3.1051.0` | Updates `composerize` from 1.7.5 to 1.7.6 Updates `express-rate-limit` from 8.5.1 to 8.5.2 - [Release notes](https://github.com/express-rate-limit/express-rate-limit/releases) - [Commits](https://github.com/express-rate-limit/express-rate-limit/compare/v8.5.1...v8.5.2) Updates `isomorphic-git` from 1.37.6 to 1.38.1 - [Release notes](https://github.com/isomorphic-git/isomorphic-git/releases) - [Commits](https://github.com/isomorphic-git/isomorphic-git/compare/v1.37.6...v1.38.1) Updates `ldapts` from 8.1.7 to 8.1.8 - [Release notes](https://github.com/ldapts/ldapts/releases) - [Changelog](https://github.com/ldapts/ldapts/blob/main/CHANGELOG.md) - [Commits](https://github.com/ldapts/ldapts/compare/v8.1.7...v8.1.8) Updates `@types/node` from 25.7.0 to 25.9.1 - [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases) - [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/node) Updates `eslint` from 10.3.0 to 10.4.0 - [Release notes](https://github.com/eslint/eslint/releases) - [Commits](https://github.com/eslint/eslint/compare/v10.3.0...v10.4.0) Updates `typescript-eslint` from 8.59.3 to 8.59.4 - [Release notes](https://github.com/typescript-eslint/typescript-eslint/releases) - [Changelog](https://github.com/typescript-eslint/typescript-eslint/blob/main/packages/typescript-eslint/CHANGELOG.md) - [Commits](https://github.com/typescript-eslint/typescript-eslint/commits/v8.59.4/packages/typescript-eslint) Updates `vitest` from 4.1.6 to 4.1.7 - [Release notes](https://github.com/vitest-dev/vitest/releases) - [Commits](https://github.com/vitest-dev/vitest/commits/v4.1.7/packages/vitest) Updates `@aws-sdk/client-ecr` from 3.1045.0 to 3.1051.0 - [Release notes](https://github.com/aws/aws-sdk-js-v3/releases) - [Changelog](https://github.com/aws/aws-sdk-js-v3/blob/main/clients/client-ecr/CHANGELOG.md) - [Commits](https://github.com/aws/aws-sdk-js-v3/commits/v3.1051.0/clients/client-ecr) Updates `@aws-sdk/client-s3` from 3.1045.0 to 3.1051.0 - [Release notes](https://github.com/aws/aws-sdk-js-v3/releases) - [Changelog](https://github.com/aws/aws-sdk-js-v3/blob/main/clients/client-s3/CHANGELOG.md) - [Commits](https://github.com/aws/aws-sdk-js-v3/commits/v3.1051.0/clients/client-s3) --- updated-dependencies: - dependency-name: composerize dependency-version: 1.7.6 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: all-npm-backend - dependency-name: express-rate-limit dependency-version: 8.5.2 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: all-npm-backend - dependency-name: isomorphic-git dependency-version: 1.38.1 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: all-npm-backend - dependency-name: ldapts dependency-version: 8.1.8 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: all-npm-backend - dependency-name: "@types/node" dependency-version: 25.9.1 dependency-type: direct:development update-type: version-update:semver-minor dependency-group: all-npm-backend - dependency-name: eslint dependency-version: 10.4.0 dependency-type: direct:development update-type: version-update:semver-minor dependency-group: all-npm-backend - dependency-name: typescript-eslint dependency-version: 8.59.4 dependency-type: direct:development update-type: version-update:semver-patch dependency-group: all-npm-backend - dependency-name: vitest dependency-version: 4.1.7 dependency-type: direct:development update-type: version-update:semver-patch dependency-group: all-npm-backend - dependency-name: "@aws-sdk/client-ecr" dependency-version: 3.1051.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: all-npm-backend - dependency-name: "@aws-sdk/client-s3" dependency-version: 3.1051.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: all-npm-backend ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
Anso
|
e65c5e8551 |
fix(pilot): post-merge audit followups (WS via getProxyTarget, closeTunnel lifecycle, mesh source buffer, docs) (#1128)
* fix(pilot): route remote WS upgrades through NodeRegistry.getProxyTarget Pilot-mode nodes carry empty api_url and api_token by design and expose their API on a per-tunnel loopback bridge. The upgrade handler gated the remote-forwarder branch on `node.api_url && node.api_token`, so WS requests targeting pilot nodes silently fell through to the local handlers (live logs, exec, generic) instead of tunneling to the agent. Resolve the target via NodeRegistry.getProxyTarget so pilot and proxy modes share one dispatch path, mirroring the HTTP proxy. handleRemoteForwarder now takes the resolved target and, when the target is the pilot loopback (empty token), skips the console-token exchange and the Authorization injection so the tunnel-side auth is the only source of truth on that path. Unresolvable targets reject the upgrade with HTTP 503 instead of being served gateway-local data. * fix(pilot): emit tunnel-down and mark node offline on closeTunnel PilotTunnelManager.closeTunnel closed the underlying WebSocket but skipped the cleanup the natural-disconnect path runs, so explicit closures (enrollment regenerate, node deletion) left the node row at status='online' until the next reconnect. The dashboard kept showing the stale state for the entire interval. closeTunnel now writes nodes.status='offline' and emits tunnel-down for pilot bridges, and emits proxy-bridge-down for central-initiated proxy bridges. The maps are cleared before bridge.close() so the natural 'closed' handler's bridge-identity guard short-circuits and we do not double-emit. * fix(mesh): buffer cross-node source data until tcp_open_ack arrives openCrossNode piped src socket data straight to tcpStream.write before the forward TcpStream emitted 'open'. The first packet on a fresh cross-node stream raced ahead of the agent's tcp_open_ack on the wire, which broke protocols that send immediately after connect (HTTP, TLS, Redis, Postgres) on Pilot and proxy mesh paths. Buffer src chunks in a local array capped at STREAM_PENDING_DATA_MAX_BYTES until tcpStream emits 'open', then flush them in order before any post-open writes. Tear down both sockets if the buffer overflows so a misbehaving source cannot exhaust gateway memory while waiting for the ack. * docs(pilot): clarify host-console non-parity and narrow the parity claim Pilot mode disables the host-console capability at the capability registry (the agent container has no useful host shell to surface), but the public docs listed host console among the WebSockets that ride through the tunnel and described pilot as behaving identically to proxy mode. State the shared-capability claim more carefully and call out the intentional non-parity in a dedicated subsection. |
||
|
Anso
|
982c7830b1 |
fix(mesh): address audit follow-ups (early-data, recompose, admiral gate, error codes, docs) (#1126)
* fix(mesh): buffer early TcpData on reverse-relay path
The reverse-relay code dropped the local-shaped reservation and any
TcpData frames buffered in it before acceptReverseRelay had wired up
the target stream. A peer that sent a request body immediately after
tcp_open_reverse lost those bytes when the target lived on a third
node, since reverse_local buffers them but reverse_relay did not.
Carry the reservation through acceptReverseRelay: transplant its
pendingData and pendingBytes into the new reverse_relay state, gate
TcpData writes on targetOpen, and flush buffered frames into the
target stream before sending tcp_open_ack. Mirrors the reverse_local
pattern. Regression test fires tcp_open_reverse plus an immediate
TcpData while ensureBridge is in flight and verifies the bytes
arrive intact, in order, before the ack.
* fix(mesh): recompose affected stacks when node-level mesh is disabled
disableForNode used to clear DB rows and override files but leave the
running containers attached to sencho_mesh with stale /etc/hosts
alias entries until an operator redeployed every stack by hand.
Mirror optOutStack: after the existing alias/forwarder cleanup, call
regenerateOverridesAcrossFleet, cascadeRecomposeAcrossFleet, and
triggerRedeploy for each previously meshed stack on the disabled
node so containers detach from sencho_mesh and shed the alias
entries they owned. The disabled node's mesh_stacks rows are deleted
before the cascade so listMeshStacks returns the right set with no
skip tuple required. Route threads the actor through actorFor(req)
for parity with optInStack/optOutStack. Tests cover the redeploy
fan-out, the cascade no-skip-tuple invariant, and the default actor
fallback for non-route callers.
* fix(mesh): require Admiral on the WS proxy-tunnel upgrade
HTTP mesh routes in routes/mesh.ts all enforce requireAdmiral, but
the /api/mesh/proxy-tunnel WS upgrade accepted any node_proxy or
full-admin api_token regardless of the receiver's license. A node
downgraded from Admiral kept serving mesh data-plane traffic to a
sibling central while refusing every mesh management call.
Read the receiver's local LicenseService at the upgrade and 403 when
the tier is not paid+admiral. The check sits after the existing
credential gate and uses LicenseService directly rather than
effectiveTier (which trusts forwarded proxy headers); a remote peer
dialing in cannot be trusted to assert our entitlement. Dialer and
node_proxy token format are unchanged. Three regression tests cover
community-tier node_proxy, skipper-tier node_proxy, and
community-tier full-admin api_token all rejected with 403.
* fix(mesh): handle no_target alongside push_failed in inspectStackServices
proxyFetch throws MeshError('no_target') when getProxyTarget returns
null (pilot tunnel offline, proxy bridge unreachable), but the
inspectStackServices catch branch only matched push_failed. Offline
remotes fell through to the generic 'remote unreachable' error log,
which the Routing tab surfaces as an unexpected fault.
Match both error codes and emit the operator-friendly warn message
that names the unreachable node and the error code. Regression test
spies on console.warn/console.error to pin the branch.
* docs(mesh): align env defaults and forwarder comments with current architecture
SENCHO_MESH_PROXY_TUNNEL_IDLE_MS in .env.example carried the old
five-minute idle-close value (=300000), but the code default is
DEFAULT_IDLE_TTL_MS=0 (persistent tunnel). Copying the example
silently reintroduced the idle-close behavior the dialer removed.
MeshForwarder.ts's leading docblock and inline listen comment still
described host-network mode plus extra_hosts: host-gateway as
required for forwarder reachability. Sencho runs in standard bridge
mode and attaches to the shared sencho_mesh network at a stable IP;
meshed user containers reach the forwarder by that IP directly.
Flip the env default to 0, rewrite the env comment to describe the
persistent behavior and the opt-in for idle teardown, and rewrite
both forwarder comments to match the bridge-network reality.
* fix(mesh): trust forwarded tier on proxy-tunnel WS and remove remote overrides on disable
Admiral entitlement on the WS data plane now follows the same trust model
as the HTTP mesh routes: the central asserts its tier via x-sencho-tier
and x-sencho-variant on the WS handshake, and the receiver trusts those
headers only when the upgrade carries a node_proxy credential. When no
headers are present or the credential is a full-admin api_token, the
receiver falls back to its own local license. Without this an Admiral
central could be rejected by a Community remote and a Community central
could dial a locally-Admiral remote.
disableForNode now routes through removeOverrideFromNode so override
files pushed earlier via applyLocalOverride are removed on remote nodes
via DELETE /api/mesh/local-override/:stack. Sequential awaits are wrapped
in Promise.allSettled to match regenerateOverridesForNode's parallel
push pattern.
Other changes:
- Cover the post-state-swap buffering window in the reverse-relay test
(TcpData arriving after openTcpStream returns but before target open).
- Refresh stale MeshService comments that still referenced the removed
sidecar layer and host-network listener model.
* test(mesh): pin removeOverrideFromNode remote HTTP shape
The disableForNode regression test mocks removeOverrideFromNode itself,
so a regression inside the helper would not be caught. Add a narrow
contract test that spies on global fetch and asserts the request shape:
DELETE /api/mesh/local-override/:stack against the resolved proxy
target, with Authorization Bearer plus the x-sencho-tier and
x-sencho-variant headers. Also covers the encodeURIComponent path and
the swallowed-network-error behavior the disable cascade relies on.
* test(mesh): use createTestApiToken helper in proxy-tunnel api_token gate test
The full-admin api_token branch of the WS Admiral gate test inlined the
canonical generateApiToken + sha256 + addApiToken triple that already
lives in the createTestApiToken helper. Switching to the helper removes
the duplicated insertion logic and aligns this test with the helper used
by the other api_token call sites.
|
||
|
Anso
|
ee891b093b |
fix(self-update): resolve /app/data through bind or named volume (#1124)
SelfUpdateService scanned the container's Mounts list for Type='bind' only when picking the host path to forward to the helper container. A pilot enrolled with the recommended Docker Compose snippet uses a named volume (sencho-agent-data:/app/data, Type='volume'), so the lookup returned null and the boot log read "/app/data mount not found - update error recovery will be unavailable". The helper still ran on update but could not persist UPDATE_ERROR_FILE on a failed pull, so the next gateway process had nothing to surface. Extract findDataDirHost(mounts) and accept Type='bind' or Type='volume'. Docker populates Source with the on-disk volume directory for either type, so the existing :rw bind in the helper spawn works unchanged. The independent hostBindMounts filter stays strict to Type='bind' (forwarded operator-declared compose paths only; named volumes are not in scope there). Tests: pure unit coverage for the helper across 6 cases (bind hit, volume hit, no /app/data, mixed list with siblings, empty Source, tmpfs ignored). |
||
|
Anso
|
0b50c88eb3 |
fix(fleet): route remote update trigger through getProxyTarget (#1123)
POST /api/fleet/nodes/:id/update and POST /api/fleet/update-all read node.api_url + node.api_token directly, so a pilot-agent row (which carries neither) returned "Remote node not configured." and was filtered out of bulk update. Both routes now resolve the target via NodeRegistry.getProxyTarget(), which returns the loopback URL for an active pilot tunnel and the configured api_url for proxy-mode remotes. fetchMetaForNode replaces the direct fetchRemoteMeta call for the self-update capability check. self-update is removed from PILOT_DISABLED_CAPABILITIES so a Compose-deployed pilot can advertise it; the host-console filter stays. SelfUpdateService still gates the local-end capability on the container actually carrying docker-compose labels, so a docker-run pilot self-disables and the Fleet UI sees a clean 503 instead of an ambiguous failure. Tests: new fleet-pilot-update covers the four single-node branches (success via loopback, null target, no self-update capability, meta offline) and two update-all cases (mixed-fleet dispatch, all-targets- null skip). capability-registry-pilot rewired to assert the new filter set. |
||
|
Anso
|
282ab8d844 |
fix(pilot): let SENCHO_PUBLIC_URL override the request Host in enrollment (#1122)
The enrollment minter inferred SENCHO_PRIMARY_URL from the request Host header, which baked loopback or LAN addresses into the compose YAML when the admin opened Add Node on the central's own machine. Pilots on a different network (a public cloud VPS, for example) cannot dial that. SENCHO_PUBLIC_URL on the primary now wins when set and well-formed (http(s)://, no loopback). Trailing slashes are stripped. Falls back to the request Host when unset or invalid. |
||
|
Anso
|
3ad6ea9c5d |
feat(pilot): make Docker Compose the canonical pilot enrollment payload (#1121)
* feat(pilot): make Docker Compose the canonical pilot enrollment payload The Add Node dialog for a pilot-agent now returns a Compose snippet instead of a single-line docker run command, and the enrollment dialog walks the operator through a save-and-up flow. The compose project name and container name align with what SelfUpdateService looks up at boot, so a Compose-deployed pilot can be updated remotely through the Fleet view without intervention on the remote host. Docs (pilot-agent, remote-updates) were rewritten to match. * test(e2e): align pilot enrollment spec with Compose payload The spec was written against the docker-run payload; it now asserts the Compose YAML the dialog renders. |
||
|
Anso
|
e05099f2a1 |
fix(fleet-sync): make control-identity-mismatch sticky and surface in UI (#1117)
Treat 409 CONTROL_IDENTITY_MISMATCH from a replica as a non-retriable
failure instead of looping the same 409 through the 5-minute retry
service forever and silently writing identical failure rows.
Backend
- DatabaseService: add `sticky_error_code`, `sticky_error_expected`,
`sticky_error_got` columns to `fleet_sync_status` via an idempotent
migration. New methods setFleetSyncSticky, getFleetSyncStickyCode,
clearFleetSyncStickyForNode. recordFleetSyncSuccess clears the sticky
flag on a clean push. getFailedSyncTargets SQL adds
`AND sticky_error_code IS NULL` so the retry loop skips sticky rows.
- FleetSyncService.executePushToNode: short-circuits at the top when
sticky is set (covers event-driven pushResourceAsync calls). On a 409
with code CONTROL_IDENTITY_MISMATCH, records the failure once and
pins sticky with the expected/got fingerprints carried in the 409 body.
- routes/nodes.ts: new POST /api/nodes/:id/fleet-sync/reset-anchor.
Admin + paid + node:manage. Proxies POST /api/fleet/role/reanchor to
the peer with `{override:true}` using the stored Bearer node_proxy
token. On peer 200, clears every sticky row for the node so the next
push re-anchors and resumes replication. Distinct 502 / 504 responses
for peer-rejected / peer-unreachable so the UI can show a useful toast.
Frontend
- New lib/fleetSyncApi.ts + hooks/useFleetSyncStatus.ts. Polling hook
(30s visibilityInterval) skips fetch when !isPaid.
- NodeManager.tsx: destructive banner per affected node listing both
fingerprints, with `Reset anchor on peer` and `Remove node` buttons.
Hidden for community-tier users via empty hook data.
- FleetConfiguration.tsx (Fleet -> Status): read-only `Policy sync`
SummaryRow per remote node card. In sync / degraded / paused with
a tooltip; no action buttons (the action lives in NodeManager).
Tests
- fleet-sync-service.test.ts: 4 new cases for sticky-set on first
mismatch, short-circuit on subsequent pushes, null fingerprints,
and non-mismatch failures not setting sticky.
- database-fleet-sync-sticky.test.ts (new): 6 cases pinning the DB
contract incl. retry-loop SQL filter and migration idempotency.
- nodes-fleet-sync-reset-anchor.test.ts (new): 6 cases covering
happy path, peer 401 -> 502, peer unreachable -> 504, local-node
rejection, unknown node id, and community-tier 403.
Gate parity (Directive 30): the new POST .../reset-anchor enforces
requireAdmin + requirePaid + node:manage (matches the existing read at
GET /api/fleet/sync-status). UI banner + SummaryRow only render when the
hook returns data, which it only does for paid-tier authed users. No
existing tier-gate file moved; this is greenfield parity.
Auth audit: the peer's POST /api/fleet/role/reanchor route already uses
requireAdmin, which accepts the central's stored node_proxy Bearer
token because authMiddleware maps `scope === 'node_proxy'` to
`req.user = { username: 'node-proxy', role: 'admin', userId: 0 }`.
No widening required.
Backend tsc clean. Frontend tsc -b clean. 59 fleet-sync tests pass; full
backend suite green minus the pre-existing Windows-only file-lock flake
on filesystem-backup.test.ts that reproduces unchanged on main.
|
||
|
Anso
|
6722335a79 |
fix(stack-update): refresh frontend state automatically after a stack update (#1113)
After applying a stack update the sidebar's "update available" dot stayed visible and the stack's status indicator was stuck on the optimistic value until the page was manually refreshed. Two root causes: 1. Image-updates state refresh was a fire-and-forget call in some paths and entirely missing from the bulk-update, auto-update, and state-invalidate WebSocket-handler paths. 2. stackActionsRef.current was resynced only at render time, so the post- update refreshStacks(true) running in the action's finally block read a stale "busy" map and preserved the optimistic mask via prev[file] ?? status. Backend now broadcasts a state-invalidate event with scope='image-updates' and action='stack-updated' after every successful update (single-stack route and auto-update loop). The frontend useNotifications hook routes this to a new onImageUpdatesChange callback wired to fetchImageUpdates in EditorLayout, so every connected client refreshes the dot through the same code path. Bulk update also calls fetchImageUpdates directly for fast local feedback, and setStackAction/clearStackAction now keep stackActionsRef synchronously in sync with state so the busy-stack check inside refreshStacks observes the cleared map immediately. Adds 3 unit tests covering the new WS branch (positive, scope-mismatch negative, auto-update-settings-changed negative). |
||
|
Anso
|
9dbce9c3c7 |
fix(spawn): attribute ENOMEM and ENOENT-under-memory-pressure spawn failures to host OOM (#1111)
Operators previously saw "spawn docker ENOENT" or "spawn /bin/sh ENOENT" when the host was under memory pressure, which sent them down a missing-binary debugging path. Linux libuv's posix_spawn can fail to allocate its argv / path-search arena under low free memory and surface the underlying ENOMEM as ENOENT. Centralizes spawn-error mapping in a new utils/spawnErrors.ts helper: - Explicit ENOMEM is rewritten to "Out of memory while launching <command> (host free memory: X MiB of Y MiB)". - ENOENT under the 128 MiB free-memory floor is rewritten with the same wording plus a "reported as ENOENT under memory pressure" hint. - ENOENT for docker on a healthy host preserves the existing "Docker CLI unavailable on this node" mapping. - Other errors pass through unchanged. Applied at the four named offenders: ComposeService.execute(), ComposeService.captureCompose(), DockerController.getContainersByStack(), and FileSystemService.getStacks() (which gets an ENOMEM-aware log line for the scandir failure). Startup also logs host free/total MiB once and warns when free memory is below the 128 MiB floor, so the diagnostic surfaces before the first spawn attempt rather than after it fails. 37 tests cover the mapping function directly and the ComposeService / FileSystemService integration paths. |
||
|
Anso
|
523ba5854c |
fix(stacks): return 404 for nonexistent stacks on deploy/down/update (F-7) (#1108)
POST /api/stacks/:name/{deploy,down,update} previously returned HTTP 500
with body {"error":"spawn docker ENOENT"} when invoked against a stack
whose compose directory was missing. The status code was wrong (the
named resource did not exist, so 404 is the right answer) and the
message misled operators into thinking the docker CLI was unavailable.
Add a small requireStackExists(nodeId, stackName, res) helper in
routes/stacks.ts that validates the stack name and confirms a compose
file is present via FileSystemService.hasComposeFile before any of the
three handlers spawn docker compose. The helper is called immediately
after requirePermission and before runPolicyGate so unauthorized
callers still get 403 first and the policy gate never runs against a
phantom stack.
In ComposeService.execute(), narrow the child.on('error') handler so
the genuine docker-binary-missing case (ENOENT on the spawn itself)
rejects with "Docker CLI unavailable on this node" instead of the raw
"spawn docker ENOENT". This is defense in depth for the rare case the
pre-check cannot cover, and it fixes the misleading-message half of
the bug as well.
Cover the new contract with stack-actions-missing-stack.test.ts (four
cases: deploy/down/update return 404, invalid name returns 400). Mock
ComposeService as a tripwire so a future code path that bypasses the
guard would fail loudly. Fix stacks-failure-notifications.test.ts by
adding hasComposeFile to its FileSystemService partial mock so the
existing happy-path-error-handling cases continue to flow into
ComposeService.
|
||
|
Anso
|
1f673073ca |
feat(fleet): add fleet-wide Docker prune to Fleet Actions (#1104)
Adds a third card to the Fleet Actions tab that fans out Docker prune (images, volumes, networks) across every node in one submit. Local nodes call DockerController under a bulk-prune lock; remote nodes receive one POST /api/system/prune/system per target. Per-node + per-target results with reclaimed bytes are surfaced inline via ResultsList. Tier: Skipper / Admiral (requirePaid + requireAdmin), matching the rest of Fleet Actions. The frontend card is mounted inside the existing isPaid branch at FleetActionsTab; no new frontend gate is required. The card uses an amber accent rail and the Eraser icon so it reads as 'cleanup' rather than 'destructive stop'. Scope toggle defaults to Managed only (Sencho-tagged resources) with an All unused option that escalates the destructive-confirm copy. |
||
|
Anso
|
0cffd60173 |
fix(mesh): clean up activeStreams on src socket close/error (F-10) (#1105)
The cross-node MeshService.openCrossNode handler attached src.on('close')
and src.on('error') listeners that only called tcpStream.destroy() and
never deleted the activeStreams Map entry. MeshTcpStreamLike.destroy()
sends a tcp_close frame to the remote but does not synchronously emit
the local close event; that fires only when the remote sends back a
tcp_close ack via _dispatchClose (or when the entire tunnel tears down
and the bridge force-emits close on every stream).
Failed dials whose remote ack was lost (peer gone, network drop, or the
F-9-era timeouts) therefore leaked records into activeStreams until the
tunnel itself idle-closed, producing the monotonically-climbing
activeStreamCount documented in the mesh E2E audit (F-10).
Add a cleanupRecord() closure that clears the open timer and deletes
the record idempotently, then route all four lifecycle handlers
(tcpStream.on('error'), tcpStream.on('close'), src.on('close'),
src.on('error')) through it. Map.delete is naturally idempotent so the
double-delete that fires when both sides close normally is a free
no-op. The same-node openSameNode path already had this shape via its
existing teardown() closure; this brings the cross-node path to parity.
Three new Vitest cases in mesh-service.test.ts use an EventEmitter-
backed fake src so emit('close') actually delivers to the listener
(the existing makeFakeSocket uses vi.fn for .on which records calls
but never fires them). They cover src.close, src.error, and the
idempotency case where both src and tcpStream emit close in sequence.
|
||
|
Anso
|
c460bb87a8 |
fix(mesh): probe upstream synchronously in route diagnostic (F-11) (#1100)
GET /api/mesh/aliases/:alias/diagnostic returned a cached state derived from the last latency/error maps. Those maps only mutated when someone called POST .../test or when a cross-node connect logged an event, so once an upstream stopped the diagnostic kept reporting "healthy" until the 60 s alias-cache refresh pruned the alias entirely. The GET now calls testUpstream synchronously after the alias-resolved, opted-in, tunnel-up short-circuits. Probe failures land in routeErrorMap via logActivity (cross-node already did this; same-node timeout/error paths now log probe.fail in the same shape), so the state computation flips to "unreachable" on the same request that exposed the stopped upstream. A new routeProbeAtMap stamps freshness and surfaces in the response as lastProbeAt; the route detail sheet renders "Last probe <age> · <ms>" using the existing formatTimeAgo helper. |
||
|
Anso
|
554f662563 |
fix(mesh): surface stopped-stack opt-ins on routing node cards (#1098)
Add `currentlyResolvable: boolean` per entry in `MeshNodeStatus.optedInStacks`, derived from the existing alias cache so the new field stays consistent with `/api/mesh/aliases` without any extra Dockerode or cross-node inspect on the status path. The Routing tab renders an amber `suspended` pill for entries whose stack is opted in but currently has no running services, plus a single explanatory caption below the suspended list. Resolves the contradictory `Mesh stacks: 1 / Aliases: 0 / No mesh services on this node yet` copy on the node card when a meshed stack's container has been stopped; the misleading line is now only shown when the node truly has no opt-ins. The opt-in itself remains sticky: when the stack starts again, its aliases reappear automatically on the next refresh. Defensive de-dup in the UI filters suspended entries against the live alias snapshot to handle the transient gap where `/mesh/status` and `/mesh/aliases` return slightly inconsistent views from their separate fetches. Tests: - new `mesh-status-resolvability.test.ts` (6 cases) locks the resolvable / suspended / mixed / empty / per-node-scoping / stale-alias-no-phantom invariants for `getStatus`. - `mesh-topology-layout.test.ts` gains a `stacksKey` resolvability-flip case and a `meshNodeStateEqual` case asserting a resolvability flip on an otherwise identical stack registers as a state change so the topology layout re-runs. Operator docs gain one new troubleshooting accordion in `/docs/features/sencho-mesh.mdx` explaining the suspended state. |
||
|
Anso
|
9f238e187c |
fix(mesh): cascade opt-out when a stack is deleted (F-1 / F-14) (#1096)
DELETE /api/stacks/:name now calls MeshService.optOutStack after the DB cleanup so the mesh_stacks row, the override file under <DATA_DIR>/mesh/overrides/<nodeId>/, and any derived aliases do not outlive the deleted stack. Pre-fix, those artifacts leaked and the reconcile loop logged "No compose file found for stack" every tick. optOutStack is idempotent (early return when the stack was never opted in) and already cascades override-regen plus recompose across the rest of the fleet, so peers' /etc/hosts drop the dropped alias. The new cascade call is best-effort relative to the delete itself: mesh cleanup failures warn but never regress the delete contract. New regression test asserts three contracts: cascade on delete, no-op on never-meshed, and 200 + warn when the cascade rejects. |
||
|
Anso
|
f6e42535c8 |
fix(mesh): route peer→central traffic over the existing forward WS (#1094)
* fix(mesh): route peer→central traffic over the existing forward WS The reverse mesh callback path (`/api/mesh/proxy-tunnel-from-peer`) needed SENCHO_PRIMARY_URL on central plus a publicly reachable origin from the peer's perspective. In a typical homelab where central sits behind NAT, peer→central dispatch silently failed at the dialer's short-circuit and the headline "call any service on any node by hostname" worked one way only. The forward WS at `/api/mesh/proxy-tunnel` is already bidirectional end to end. Make the bridge a persistent control-plane primitive: dial every mesh-enabled proxy peer at startup, reconcile every 60 s, never idle-close. Peer→central traffic multiplexes over the same WS via `tcp_open_reverse`. Removed: - `meshProxyTunnelFromPeer.ts` WS handler and dispatch - `MeshCentralRegistry`, `PeerToCentralMeshSessionDialer` - `mesh_handshake` first-frame state machine in `meshProxyTunnel.ts` - `maybeSendBootstrap`, `buildHandshakeFrame` in the dialer - `mesh_proxy_callback_bootstrap` capability and `maybeWarnUnsetPrimaryUrl` - `mesh_centrals` table (drop migration; greenfield, no users) - `PilotTunnelManager.replaceOrRegisterProxyBridge` (dead after handler removal) - twelve associated unit/integration tests plus the peer-recovery branch in `MeshService.openCrossNode` Added: - `MeshService.proactiveBridgeFanout` selects every mesh-enabled proxy peer (no longer gated on `mesh_stacks` rows) - `startBridgeReconcileLoop` runs the fanout every 60 s (override via `SENCHO_MESH_RECONCILE_INTERVAL_MS`) - `MeshProxyTunnelDialer` default idle TTL is now `0` and exposes `isDialing(nodeId)` for the status surface - `MeshNodeStatus.reverseCallbackStatus` discriminator (`connected | connecting | unavailable | not_applicable`) surfaced via `/api/mesh/status` and rendered as a pill in the Routing tab - `openCrossNode` error message distinguishes "no proxy target" from "waiting for central to dial the reverse bridge" - New tests: `mesh-service-proxy-tunnel-reconcile`, `mesh-status-reverse-callback`, `mesh-proxy-tunnel-dialer-no-idle-close` SENCHO_PRIMARY_URL is no longer required for any mesh function. * fix(mesh): rewrite proxy-tunnel reconcile test contents The previous commit renamed the file but the rewritten test bodies stayed unstaged on top of the rename. This commit lands the actual rewrite: the fanout assertion now requires every mesh-enabled proxy peer to be dialed, not just those with `mesh_stacks` rows, and adds a reconcile-tick repeated-call test. |
||
|
Anso
|
ed49ed6165 |
fix(http): strip zstd from Accept-Encoding so compression sets Content-Encoding (#1092)
compression@1.8.1's negotiator silently fails to set the Content-Encoding response header when Accept-Encoding carries an unknown token like zstd, even though the body is still compressed with brotli. Chromium 123+ sends "Accept-Encoding: zstd, gzip, br" by default, including Playwright's bundled Chromium over plain HTTP, so any homelab Sencho viewed without a TLS terminator triggered ERR_CONTENT_DECODING_FAILED in the browser and rendered as a blank page. Insert a tiny middleware (step 4 of the canonical pipeline) that drops zstd tokens from Accept-Encoding before compression's negotiator runs. The negotiator then picks br or gzip and writes the matching header. The fix is no-op when zstd is absent. If only zstd was offered, the header falls back to identity so the response is served uncompressed. Renumber the canonical-order JSDoc in app.ts from 17 to 18 steps and update the step-number references in hub-only-guard and proxy-mount-order test docstrings to match. 11 new tests (8 unit + 3 supertest integration against real compression with a 3000-byte body) lock in the behavior: zstd-stripping across casing and q-values, surviving tokens preserved verbatim, identity fallback when only zstd was offered, Content-Encoding header always written when at least one supported codec remains. |
||
|
Anso
|
1d7418a233 |
fix(mesh): recompose previously-meshed containers when alias set changes (#1090)
When a stack is opted into the mesh, every previously-meshed stack's override.yml on disk gains the new alias, but the running containers keep their stale /etc/hosts because extra_hosts is read at container creation time. Cross-stack DNS for the new alias silently fails until an operator manually redeploys each prior stack. Complete the cascade: after regenerateOverridesAcrossFleet finishes, fan out triggerRedeploy across every (node_id, stack_name) tuple in mesh_stacks, skipping the just-opted-in pair (the explicit triggerRedeploy at the end of optInStack covers it). Mirror the same cascade in optOutStack so the dropped alias exits every container's /etc/hosts. triggerRedeploy already dispatches local vs remote, already wraps runRedeploy in fire-and-forget error logging to the mesh activity ring and audit log, and already enforces compose policy gates. The cascade just reuses it. regenerateAllOverrides (boot and manual /regen-overrides) is intentionally NOT covered: a Sencho restart must not force a fleet-wide recompose; override files alone are sufficient there. Adds four unit tests in mesh-service.test.ts: cascade fires for every prior meshed stack on opt-in, no double-redeploy of the just-opted-in tuple, cascade is a no-op when no peers exist, cascade fires for every survivor on opt-out. |
||
|
Anso
|
578cac89da |
fix(mesh): surface data-plane failures in health, meta, and Routing tab (#1088)
The three previously-silent console.warn paths in MeshService.setupMeshNetwork now route through a typed recordSetupFailure helper that classifies the failure (subnet_invalid, subnet_overlap, subnet_mismatch, ip_in_use, attach_failed, not_in_docker), emits a mesh.disable activity entry at the matching level (error for real failures, warn for the expected dev-mode not_in_docker case), and strips mesh_proxy_callback_bootstrap from advertised capabilities via CapabilityRegistry. /api/health gains a mesh.dataPlane block carrying the typed status. /api/mesh/status carries localDataPlane at the top level so the Routing tab renders a red banner with an operator-actionable recovery hint (set SENCHO_MESH_SUBNET to a free /24 and recreate the container) when the data plane is down. The success path re-enables the capability and flips the status to ok. Generalizes the previously-documented F-0 failure mode (IP-in-subnet collision) to also cover the subnet-pool-overlap case where another Docker bridge on the host already owns the requested CIDR. |
||
|
Anso
|
a318e6b3c1 |
fix(mesh): close data-plane race that dropped early TcpData frames (#1086)
The tcp_open and tcp_open_reverse receivers registered their stream entry only after awaiting resolveTarget / resolveContainerIp. The peer, which is free to send TcpData immediately after the open frame, hit the lookup-miss path in handleBinaryFrame and the bytes were silently dropped. Probes passed because they only exercise the handshake; real HTTP hung with 0 bytes received. Reserve the stream entry synchronously, buffer early TcpData in a per-stream pendingData queue capped at 1 MiB, and flush onto the local socket inside the connect handler before sending tcp_open_ack. The payload is copied because decodeBinaryFrame returns a subarray view of the WS receive buffer; holding the view would pin the parent buffer past its lifecycle. Both sides of the data plane are patched: - tcpStreamSwitchboard.onTcpOpen (forward, agent + proxy-mode peer) - PilotTunnelBridge.handleTcpOpenReverse / acceptReverseLocal (reverse, primary acting as the local dial target) Adds unit coverage for the race window and for the overflow path. The cap constant lives in pilot/protocol.ts alongside the other per-stream limits. |
||
|
Anso
|
50b89db3b8 |
chore(mesh): drop mesh_stacks table on pilot-mode DBs (#1085)
Per the C-3 design, mesh state lives on central. Pilots learn aliases via the D-1 override push and hold them in `pilotAliasOverlay`. The local `mesh_stacks` table on pilots has been dead since C-3 shipped. This is a greenfield cleanup (Directive 20): - `migrateMeshTables()` runs `DROP TABLE IF EXISTS mesh_stacks` when `SENCHO_MODE === 'pilot'` and skips the CREATE. Central-mode behavior is unchanged. - The four mesh CRUD methods (`listMeshStacks`, `isMeshStackEnabled`, `insertMeshStack`, `deleteMeshStack`) short-circuit on the same check. Reads early-return empty/false; writes early-return; `insertMeshStack` also warns so an accidental pilot-side write is loud. - New `isPilotMode()` helper mirrors the existing one in `bootstrap/startup.ts`. Layering rules forbid the shared import; a third call-site would justify extraction to `helpers/`. Operator-visible behavior is unchanged on either side. On central, the table and its rows are untouched. On pilots, the table is removed (it held no rows in steady state) and any caller of the CRUD methods continues to see empty list / false / no-op without touching SQLite. Test plan - `npx tsc --noEmit`: clean. - New `database-service-pilot-mesh-stacks.test.ts`: 5/5 green (table absence + four short-circuit assertions). - `mesh-service.test.ts`, `mesh-diagnostic-local.test.ts`, and `mesh-service-proactive-bootstrap-fanout.test.ts`: 58/58 green. - Full backend suite: 2280/2281; the single failure is a pre-existing Windows EBUSY flake in `filesystem-backup.test.ts` that reproduces on a clean tree. - Code reviewed; findings applied. |
||
|
Anso
|
5bc4add099 |
refactor(mesh): retarget proxyFetch null-target throw to no_target (#1083)
`MeshService.proxyFetch` previously threw `MeshError('push_failed', ...)`
when `NodeRegistry.getProxyTarget(nodeId)` returned null, but semantically
no push was attempted: the target simply did not exist. Callers had to
special-case the code and explain the overload in a comment.
Retarget the throw to the existing `'no_target'` variant in
`MeshErrorCode`. `listStacksOnNode` matches the new code directly; the
explanatory comment is gone since the code reads for itself.
`MeshError('push_failed', ...)` continues to signal real outbound push
failures: `applyLocalOverride` (mesh data plane not yet ready on the
receiving node) and `pushOverrideToNode` (HTTP 404 from an outdated
remote, non-OK HTTP response). Those sites are unchanged.
Operator-visible behavior is unchanged in the steady state. The rare
"remote disappeared between opt-in start and override push" case now
returns HTTP 400 with `code: 'no_target'` instead of 503; the response
body carries a clear message ("no proxy target for node N") so clients
can disambiguate.
Test plan
- `npx tsc --noEmit`: clean.
- `npx vitest run src/__tests__/mesh-list-stacks-remote.test.ts`: 8/8 green
(6 existing + 2 new cases pinning the catch-arm semantics).
- `npx vitest run src/__tests__/mesh-service.test.ts src/__tests__/mesh-inspect-remote.test.ts`:
56/56 green.
- Code reviewed; findings applied.
Notes
- PR #1082 (refactor: route inspectStackServices through proxyFetch) needs
a rebase + one-line update to its catch arm before merging on top of
this change.
|
||
|
Anso
|
dcc43205e0 |
refactor(mesh): route inspectStackServices through proxyFetch (#1082)
inspectStackServices was constructing its Authorization, x-sencho-tier,
and x-sencho-variant headers inline. proxyFetch is the centralized
helper used by listStacksOnNode, pushOverrideToNode, triggerRemeshRedeploy,
and removeOverrideFromNode. Equivalent today for a GET-no-body call, but
any future header added to proxyFetch (audit context, license fingerprint,
request id) would silently miss inspectStackServices.
Wire output is byte-identical: same URL, same method, same three headers,
same 5 s timeout. The new catch arm converts MeshError('push_failed') back
to the existing warn + return [] contract callers (optInStack,
refreshAliasCache) rely on. Pattern mirrors listStacksOnNode.
|
||
|
Anso
|
a32183d198 |
fix(mesh): cascade override regen across all meshed nodes on opt-in/opt-out (#1079)
Opting a stack into mesh on node A now propagates the new alias to every meshed node's override file, not just stacks on node A. Same on opt-out. Pre-fix, optInStack and optOutStack called regenerateOverridesForNode which only walks db.listMeshStacks(nodeId). Other meshed stacks on other nodes did not learn about the new alias until something else re-pushed their overrides; operators worked around it with a manual POST /api/mesh/regen-overrides plus a redeploy of each affected stack. New regenerateOverridesAcrossFleet helper iterates db.listMeshStacks() (no arg, fleet-wide) under Promise.allSettled, skipping the (nodeId, stackName) tuple opt-in just pushed loudly. Per-stack failures surface as forwarder.error activity events, matching the pattern used by regenerateAllOverrides. Offline remote nodes leave stale overrides until the next opt-in/opt-out, the next tunnel reconnect, or a manual rerun of the regen-overrides endpoint. regenerateOverridesForNode stays for the tunnel-up retry listener: a reconnecting pilot only needs its own stacks re-pushed, not the fleet. Tests: redirect five existing spies in mesh-service.test.ts to the new method; add two new cases asserting cross-node cascade for opt-in and opt-out. 52 tests in mesh-service.test.ts pass. |
||
|
Anso
|
6b728599c4 |
fix(mesh): persist PilotMetrics counters across central restart (#1078)
The 12 PilotMetrics counters (proxy_dials_failed, proxy_bridges_total, proxy_idle_closes, the pilot-tunnel counters, and the peer-to-central callback counters) live in a singleton holding scalar fields, so process restart wipes them. Operators investigating rotation- or dial-failure trends across a central restart lost the aggregate metric. Persist the snapshot under a pilot_metrics_counters JSON row on the existing system_state KV table. Hydrate via PilotMetrics.load() in bootstrap/startup.ts before any service that increments runs; final-flush in bootstrap/shutdown.ts before the SQLite handle closes. Between those boundaries a buffered flush mirrors the audit-log pattern: every 1 s or every 100 increments, whichever fires first. Hot increment path stays a single property write plus a pending-counter check. Defensive parse on the row: object check + numeric filter, so an operator hand-edit or schema drift across releases degrades to zero state with a warn rather than crashing boot. Schema additions back-fill missing keys with zero so a release that adds a counter does not need a migration. Closes F-R-4 from the mesh tracker. |
||
|
Anso
|
9aaa8573c0 |
fix(mesh): gate Trigger 2 re-bootstrap on actual api_token change (#1076)
The PUT /api/nodes/:id handler closed and re-dialed the mesh callback bridge on every save that included api_token in the body, even when the token was unchanged. The frontend always sends the full formData on Save, so renames and compose_dir edits against a mesh-enabled proxy remote produced a wasted closeBridge + ensureBridge round-trip and a spurious manager_rejected entry in the activity log. Gate the re-bootstrap on a real value diff against the persisted token. Adds an existing-node lookup (returns 404 on missing id, which the handler previously lacked) so the comparison has the pre-update value. Reorders the guards so resource-not-found beats payload validation. Adds one integration test for the same-token path; the existing three trigger-2 cases continue to assert close + ensure firing on a real rotation, no-token-in-payload, and mesh-disabled. |
||
|
Anso
|
7e3e22138b |
fix(mesh): name actual bridge kind in proxy-bridge rejection (#1075)
* fix(mesh): distinguish proxy bridge from pilot tunnel in registerProxyBridge rejection When central's MeshProxyTunnelDialer dials a node whose slot is already held by a peer-initiated proxy bridge, the registration is correctly refused, but the activity-log message read "pilot tunnel already registered for node N; proxy bridge refused" regardless of which kind of bridge was actually in the slot. No pilot tunnel exists in proxy-mode remotes by definition, so the wording pointed operators at the wrong subsystem. Use the existing bridgeKinds index (already consulted correctly by the sister method replaceOrRegisterProxyBridge) to emit a kind-accurate message: "proxy bridge already registered for node N; concurrent dial refused" when the existing bridge is a peer-initiated proxy bridge, keeping the original "pilot tunnel" wording for the pilot-agent case so existing log filters and the sister test still match. Pure cosmetic; the rejection behavior is unchanged. New regression test asserts the discriminator in both directions. Also tightens the test-fixture beforeEach to clear bridgeKinds alongside bridges so future tests cannot read a stale kind. * chore: ignore local trailer directory in git repository * refactor(mesh): centralize bridge-kind type and conflict-message formatter Followup tidy on PilotTunnelManager after /simplify flagged two related items in the F-R-2 change: - Extract `BridgeKind = 'pilot' | 'proxy'` as a module-scoped type alias so the kind taxonomy lives in one place. `bridgeKinds` and `injectBridgeForTest` now consume it; the inline union literal is gone. - Extract `formatBridgeConflict(nodeId, existingKind)` as a private helper. Both `registerProxyBridge` and `replaceOrRegisterProxyBridge` call it from their "slot already held" rejection paths, removing the byte-identical "pilot tunnel already registered for node N; proxy bridge refused" string copy across the two sites. - Trim the 6-line WHAT comment above the rejection throw to a 2-line WHY so the call site stays scannable. No behavior change; the three relevant test suites stay green. |
||
|
Anso
|
3a20105a33 |
fix(mesh): persist rotated handshake when peer bridge owns reverse-dialer slot (#1073)
When the proxy-mode mesh peer already holds an open peer-initiated callback bridge to central, central's Trigger 2 dial (api_token rotation) arrives at the peer's `/api/mesh/proxy-tunnel` handler with a fresh `mesh_handshake` frame. The peer's `attachSwitchboard` refuses the new WS at the reverse-dialer CAS swap, but it was also closing the connection before its `'message'` listener had been attached, so the handshake frame went unread and the peer's cached callback JWT stayed at the pre-rotation fingerprint until a central restart. Restructures `attachSwitchboard` to attach the `'message'`, `'close'`, and `'error'` listeners synchronously after the switchboard is created, before the async MeshService import and the CAS swap run. On the CAS-fail branch, holds the WS open for up to FIRST_FRAME_WAIT_MS (30ms, well above localhost RTT and below operator perception) before sending the 1013 close, so an in-flight first frame lands in `onMessage` and the bootstrap material is persisted via `MeshCentralRegistry.upsert` regardless of bridge fate. The success path is untouched; reverseDialer install timing on accepted dials is unchanged. Adds a regression test that pre-installs a stub reverse dialer (simulating a peer-initiated bridge owning the slot), sends a `mesh_handshake` frame on the new WS, and asserts the registry receives the rotated material before the 1013 close fires. |
||
|
Anso
|
0b4cf90bf1 |
fix(mesh): peer-initiated callback bridge installs reverse dialer (#1071)
* fix(mesh): peer-initiated callback bridge installs reverse dialer
The R1-A2 symmetric-dial work added a peer-side dialer that opens a
callback WS to central at `/api/mesh/proxy-tunnel-from-peer`. The WS
upgrade, JWT validation, and `mesh_centrals` persistence all worked,
but the peer-side handler put the wrong object on its end of the pipe:
a `PilotTunnelBridge` (the central-role object) rather than a
`TcpStreamSwitchboard` with a registered `reverseDialer`.
The design doc states: "Central retains PilotTunnelBridge ownership;
peer retains TcpStreamSwitchboard + reverseDialer ownership." The
central-initiated handler at `meshProxyTunnel.ts:115-163` honors this.
The peer-initiated handler at `PeerToCentralMeshSessionDialer.attachBridge`
did not. It created a `PilotTunnelBridge(0, ws)` and stored it in a
private `currentSession` field. Result: `MeshService.reverseDialer`
stayed null, `dialMeshTcpStream` fell through to
`PilotTunnelManager.ensureBridge(target.nodeId)`, and `NodeRegistry.
getProxyTarget` on the peer returned null because proxy-mode peers
do not enroll their central. Cross-fleet dispatch from a peer
container failed with `proxy-tunnel.open.fail nodeId=<central>
reason=no_target` even though the callback bridge was up and healthy
(`bridgeOpen: true`, `last_used_at` populated).
Replace the peer-side `PilotTunnelBridge` with the same wiring the
central-initiated handler uses:
- `attachTcpStreamSwitchboard` with `resolveByComposeLabels` (allows
inbound `tcp_open` from central if it ever uses the callback
bridge for a reverse-direction dispatch; matches the central-
initiated pipe's resolver).
- A `SwitchboardReverseDialer` that delegates `openMeshTcpStream`
to `switchboard.openReverseStream`.
- `MeshService.setReverseDialer(localDialer, null)` with the same
CAS guard the central-initiated handler uses, so a concurrent
central-initiated tunnel does not get silently overwritten.
- `ws.on('message', onMessage)` dispatches JSON / binary frames to
the switchboard.
- `ws.once('close'/'error', teardown)` clears the switchboard, the
reverseDialer, and the `currentSession` reference idempotently.
The `currentSession` field type changes from `PilotTunnelBridge | null`
to `TcpStreamSwitchboard | null`. Callers in `MeshService.openCrossNode`
only use the return value for truthiness; the cast is harmless. The
public `hasSession()` signature is unchanged.
A new `currentWs` field holds the WS reference so `resetForTest` can
close the connection on teardown (the switchboard's own `cleanup`
does not close its WS).
Test update mirrors the production wiring: the existing
"marks the row used on successful WS open" case now asserts the
return value is a `TcpStreamSwitchboard`, that `MeshService.reverseDialer`
is non-null after `ensureSession`, and explicitly closes the WS in
the test's finally so `wss.close` can resolve.
Pre-existing test logic that asserted `bridge instanceof PilotTunnelBridge`
on the same path is removed in favor of the new switchboard assertion.
Discovered during the v0.81.1 live verification: the peer-side
`route.dispatch cross-node` event always paired with
`proxy-tunnel.open.fail no_target` and `tunnel.fail`, even though
`centralCallback.bridgeOpen: true` and `mesh_centrals.last_used_at`
were populated. The callback path opened cleanly in isolation but
the dispatch path never consumed it. This fix makes the dispatch
path consume it.
Tests:
- cd backend && npx tsc --noEmit clean
- cd backend && npx vitest run mesh 170/170 green
- cd backend && npx vitest run 2259/2268 green; only
failures are the pre-existing Windows EBUSY flake in
filesystem-backup.test.ts (documented in prior handoff)
* fix(mesh): drop unused imports and align reverse-dialer interface name
CI lint failures on the previous push were two @typescript-eslint/no-unused-vars
errors in the test file:
24:76 'vi' is defined but never used
31:5 'TcpStreamSwitchboardCtor' is assigned a value but only used as a type
Both were collateral from a local-debugging simplification: vi.waitFor was
swapped for a synchronous assertion when the test's wss.close hang was traced
to an unclosed client WS; the toBeInstanceOf assertion that used the runtime
binding was removed at the same time. Restore the instanceof assertion next
to the existing not.toBeNull check so the runtime binding is in use and the
test guards against future regressions where the wrong end-of-pipe object
type is returned on the callback bridge (the bug this PR fixes). Drop the
now-unused `vi` symbol from the vitest import.
Also fold in a /simplify convergent finding: rename the local
CallbackReverseDialer interface to SwitchboardReverseDialer to match the
sibling declaration in meshProxyTunnel.ts:72. Same shape, same name; readers
of either handler now see the same mental model. Cross-file extraction of
the interface to tcpStreamSwitchboard.ts is the cleaner long-term shape but
touches a file outside this PR's scope; filing as a follow-up.
Considered-and-deferred findings from the /simplify pass (file as separate
PRs to keep one-branch-one-concern):
- extract shared attachSwitchboard helper (~40 duplicated lines between
meshProxyTunnel.ts and PeerToCentralMeshSessionDialer.ts)
- public TcpStreamSwitchboard.getWs accessor + drop currentWs field
- public MeshService.hasReverseDialer for the test bracket-cast
- shared close-reason constants (drift risk across the two handlers)
Tests:
- cd backend && npx tsc --noEmit clean
- cd backend && npx eslint <changed> clean (lint_exit=0)
- cd backend && npx vitest run mesh 170/170, dialer 8/8 green
|
||
|
Anso
|
54d84a7edb |
fix(mesh): skip peer-side recovery in openCrossNode on central instances (#1068)
The R1-A2 symmetric-callback work added a peer-side recovery branch to
MeshService.openCrossNode: when reverseDialer is null, kick
PeerToCentralMeshSessionDialer.ensureSession so the peer re-opens its
callback WS to central before the cross-node dispatch.
The guard `!this.reverseDialer` did not distinguish "I am a peer waiting
for central to dial in" from "I am central, and the reverseDialer concept
does not apply to me." On central, reverseDialer is always null (central
is the bridge initiator side and never has another node dial into it), so
central entered the recovery branch on every forward dispatch, found no
session (central has no mesh_centrals row), and destroyed the inbound
socket with route.resolve.fail forward-from-peer no_session.
Gate the branch on `MeshCentralRegistry.getActive() !== null` so it only
runs when this Sencho is acting as a proxy-mode peer that has actually
been bootstrapped. Central falls straight through to dialMeshTcpStream.
The peer cold-start scenario the comment originally described is still
covered: a proxy-mode peer with a cached mesh_centrals row but no live
WS to central will fire the recovery branch as before.
Adds two regression tests:
- central path (no mesh_centrals row, no reverseDialer) skips recovery
and calls dialMeshTcpStream
- proxy-peer path (mesh_centrals row + ensureSession returns null)
still emits route.resolve.fail forward-from-peer no_session
Existing tests in the "openCrossNode (BUG-4)" block already used a stub
reverseDialer to bypass the recovery branch, which is why the central
regression was not caught pre-release. The new tests deliberately leave
reverseDialer null to exercise the gating.
|
||
|
Anso
|
cf618dd866 |
feat(mesh): symmetric WS dial for proxy-mode mesh peers (#1066)
* chore(mesh): foundation for symmetric callback dial Adds the data-plane scaffolding that the symmetric callback dial fix builds on: - mesh_centrals table for peer-side bootstrap material - MeshCentralRegistry service (upsert/getActive/clear/markUsed/markRejected) - PilotTunnelManager kind discriminator and replaceOrRegisterProxyBridge - mesh_proxy_callback_bootstrap capability registration - MeshProxyTunnelDialer reason-tagged proxy-bridge-down events from a single tearDownBridge emission point - Reactive redial scheduler that skips idle and auth_failed reasons * feat(mesh): add reverse-direction activity log entries (closes R1-B) acceptReverseLocal now emits route.resolve.ok with direction=reverse on connect ack and route.resolve.fail with direction=reverse plus reason=container_not_found / connect_error pre-connect. Post-connect close/error stays silent. Reuses existing event types via the new details.direction discriminator so frontend filters are unaffected. * feat(mesh): add peer-to-central callback dial path (closes R1-A2) Closes the architectural gap where proxy-mode mesh peers could not re-establish their tunnel to central after any non-idle bridge teardown (idle close, network blip, central restart, peer reboot). Central remains the hub for the data plane; the change is purely about WS initiation. Symmetric WS initiation, asymmetric protocol roles. Central retains PilotTunnelBridge ownership; peer retains TcpStreamSwitchboard + reverseDialer ownership. Central bootstraps callback credentials over the first authenticated central-initiated mesh tunnel via a one-shot mesh_handshake JSON frame; peer persists the material in a new mesh_centrals SQLite table and dials central's new /api/mesh/proxy-tunnel-from-peer endpoint when local cross-node traffic needs a bridge and none is live. Mesh_tunnel JWT (HS256, signed with auth_jwt_secret) carries scope, audience, issuer (central instance id), peer api_token fingerprint, kid. Validation on inbound peer dial: algorithm pin, signature, scope, audience, instance, time bounds, node existence and mode, fingerprint match. Failures return HTTP 401 with a machine-readable reason; peer routes the response per a clear-vs-keep cache matrix. Triggers proactive bootstrap on mesh-enable and api_token rotation; central startup fans out to mesh-enabled proxy-mode nodes with mesh_stacks rows (throttled, fire-and-forget). Reactive redial on non-idle bridge loss. Capability-gated handshake send (mesh_proxy_callback_bootstrap) makes the upgrade path safe against older peers in mixed-version fleets. Adds peer-side /api/system/pilot-tunnels centralCallback diag block, bounded counter metrics for bootstrap and dial events. SENCHO_PRIMARY_URL preflight warning when unset on a central with mesh-enabled proxy nodes. Tested with unit suites for the validation chain, registry, manager, and both dialers; integration tests for bootstrap E2E (asserts protocol-role invariant), api_token rotation, instance id change, version skew, and pilot-mode regression. * fix(mesh): green CI on the symmetric callback branch Two independent CI failures, both surgical: 1. Backend tests (11 fails): four mesh test files called setupTestDb in beforeEach. setupTestDb does not reset the DatabaseService singleton, so the per-test afterEach rm of the previous tmpdir left the singleton connection pointing at a deleted file. The next beforeEach's line-55 write threw SQLITE_READONLY_DBMOVED on Linux. Windows file-lock semantics hid this locally. Hoist setupTestDb / cleanupTestDb to file-scope beforeAll / afterAll; per-test state resets stay in beforeEach. Matches the convention in the eight mesh test files that already pass. 2. CodeQL (4 high alerts): js/insufficient-password-hash flagged sha256(api_token) at four sites. The api_token is a 256-bit opaque bearer (sen_sk_-prefixed), not a human password; sha256 is the correct fingerprint primitive for binding the mesh_tunnel JWT to a specific token. Add the two production files plus the two test files that mint the fingerprint to the existing path-scoped query-filter for that rule. * fix(mesh): drop unused afterEach import and revert dead codeql config ESLint flagged afterEach as unused in mesh-central-registry.test.ts:1 after the previous commit hoisted setup/teardown to file-scope beforeAll/afterAll. Remove from the vitest import line. Revert the codeql-config.yml additions from the previous commit. The paths: sub-key under query-filters > exclude is not a documented CodeQL feature and silently no-ops. The four js/insufficient-password-hash alerts on api_token fingerprinting are tracked as dismissed false positives in the GitHub Security tab rather than via dead config. |
||
|
Anso
|
d882f223f4 |
feat(api-tokens): switch to sen_sk_ prefixed opaque keys (#1062)
* feat(api-tokens): switch to sen_sk_ prefixed opaque keys
Replace JWT-shaped API tokens with 56-char opaque keys of the form
`sen_sk_<43-char base62 random><6-char base62 checksum>` (256-bit
entropy, sha256-truncated checksum). Node-proxy tokens stay JWTs.
Why:
* The api_token path was already a sha256 DB lookup; the JWT signature
was wasted work and the 400d JWT ceiling vs DB expires_at was a
confusing dual bound.
* Opaque tokens carry a verifiable checksum so malformed/typoed values
are rejected before any SQLite lookup.
* `sen_sk_` prefix is recognizable to GitHub, TruffleHog, GitGuardian
and makes the on-wire shape visually distinct from node_proxy JWTs.
Changes:
* New `utils/apiTokenFormat.ts` (generate + checksum-verify, CSPRNG via
randomInt, timingSafeEqual on the checksum compare).
* `middleware/auth.ts` and `websocket/upgradeHandler.ts` route opaque
tokens before any jwt.verify; 401 messages unified to avoid a
token-existence oracle.
* `middleware/rateLimiters.ts` short-circuits opaque tokens in the
node_proxy detection and keys per-token via a non-reversible sha256
slice so each token keeps its own bucket without a DB hit.
* All six existing tests migrated from jwt.sign({scope:'api_token'})
to generateApiToken(); new format-only test suite covering prefix,
length, alphabet, checksum reject paths, and a 10k-iteration
collision/integrity loop.
* Docs (features/api-tokens.mdx, api-reference/overview.mdx) describe
the shape and drop the obsolete JWT-ceiling note.
* fix(api-tokens): clear CI lint and CodeQL false positives
* Drop unused TEST_USERNAME import in remote-console-session.test.ts;
the migration to generateApiToken() left it orphaned.
* Add a CodeQL barrier model so `generateApiToken`'s ReturnValue does
not flow into the `insufficient-password-hash` query. The function
emits 256-bit CSPRNG opaque keys; sha256 of the raw token is the
correct construction for high-entropy API tokens (bcrypt-class
hashes target low-entropy human passwords). CodeQL's name heuristic
was treating "Token" as a password source and flagging the standard
sha256 wrapping at all 9 call sites.
* ci(codeql): exclude js/insufficient-password-hash for token paths
The previous barrierModel data extension was a no-op for this rule: the
js/insufficient-password-hash query identifies its "password" sources via
SensitiveExpr's name heuristic ("token", "secret", "key" substrings),
which is upstream of the taint-tracking layer where barrierModel applies.
Verified by post-push re-analysis: 9 alerts still open, all undismissed.
Replace the dead extension with a path-scoped query-filter in
codeql-config.yml so the rule no longer fires on apiTokenFormat,
apiTokens, and the test directory. Real user-password hashing code
elsewhere in the repo (auth, users, setup routes) remains analyzed.
The 9 existing alerts on PR #1062 are dismissed via API as false
positives with a justification pointing at this config. Future runs
will not re-flag them because of the path filter.
|
||
|
Anso
|
f63ffdd098 |
fix(mesh): persist proxy-tunnel self-id across bridge teardown (#1060)
The proxy-mode WS handler called setProxyTunnelSelfCentralNodeId(null) on tunnel close, dropping the central-namespace nodeId install. After an idle close, the next inbound forwarder traffic on the peer fell back to getDefaultNodeId() (=1 on proxy peers without SENCHO_ENROLL_TOKEN), collided with the alias overlay's nodeId for central's own Local, and dispatched cross-fleet aliases to the same-node path. The signature was TCP open + no banner on the prober, with route.resolve.denied on the peer activity log even though R1's identify install fired correctly when the bridge was warm. The peer's identity in central's namespace is enrollment-stable, not per-WS. Stop null-clearing on teardown. The setter already emits a console.warn when the install is overwritten with a different non-null value, which is the right signal for an actual re-enrollment. Tests: existing assertion that the install becomes null on close is flipped to assert persistence, plus two new tests for the close-reopen lifecycle: same nodeId is idempotent (no second identify activity event), different nodeId emits the overwrite warn. Idempotency test polls deterministically for the first identify event before snapshotting the activity-count baseline so the assertion direction can't invert on a slow CI runner. |
||
|
Anso
|
489aab4516 |
fix(mesh): testUpstream dials via ensureBridge so proxy-mode targets are probeable (#1057)
testUpstream short-circuited on hasActiveTunnel + getBridge (pilot-only) when probing a cross-node alias. Proxy-mode remotes never have a "pilot tunnel" in the manager's sense, so any operator hitting "Test alias" on a proxy-mode target returned tunnel_down even when the cross-node dial via dialMeshTcpStream worked fine. Replace with await ptm.ensureBridge(target.nodeId), matching the mode- agnostic dispatch dialMeshTcpStream already uses. ensureBridge returns an existing pilot tunnel, an existing proxy bridge, or asks MeshProxyTunnelDialer to dial on demand. The 'no pilot tunnel' vs 'no bridge' distinction goes away (they meant the same thing). Adds two test cases: probing a proxy-mode remote via on-demand bridge, and the tunnel_down path when ensureBridge yields null. |
||
|
Anso
|
3d5f0ffacd |
fix(mesh): reject opt-in when every service has empty ports (#1056)
optInStack rejected empty service lists but not the case where services exist with every ports: []. The newPorts Set ended up empty, the function sailed past collision checks, wrote a mesh_stacks row, and produced an empty-aliases override. The stack then appeared online in the Routing tab but no traffic actually routed. Add a check after building newPorts: throw no_target with a clear message before the SENCHO_LISTEN_PORT collision check. Reuses the existing MeshError code (semantically: no routable target) so the frontend's error toast path is unchanged. |
||
|
Anso
|
6185206323 |
fix(mesh): proxy peer learns its central-namespace nodeId at tunnel upgrade (#1055)
Proxy-mode peers receive an alias overlay from central with nodeIds in central's namespace, but had no source for their own nodeId in that namespace. selfCentralNodeId fell back to the peer's local DB default (always 1), and handleAccept falsely matched cross-node aliases (nodeId=1, central's Local) against the peer's self-id. Cross-node aliases routed to openSameNode and surfaced as route.resolve.denied. Central now appends ?nodeId=<central-namespace-id> to the proxy-tunnel WS URL. The peer's upgrade handler parses it with a strict integer regex, installs into MeshService before the reverse dialer is registered, and clears on disconnect. handleAccept resolves self-id in priority order: proxy-tunnel install > SENCHO_ENROLL_TOKEN > local default. A non-null overwrite to a different value warns loudly (misconfigured deployment). Missing or malformed query param logs a warn and proceeds, leaving reverse-direction dispatch undefined until both sides are upgraded. Adds 5 test cases: nodeId install + teardown, missing/malformed input (including decimal, leading zeros, exponent, whitespace), overwrite warn, and CAS-rejected reverse-dialer leaving the slot unchanged. |
||
|
Anso
|
946db9df52 |
fix(mesh): accept node_proxy at the proxy-tunnel WS upgrade (#1050)
The mesh proxy-tunnel WS handler was gated on full-admin api_token scope only, so node_proxy JWTs (the credential the Add Remote Node dialog tells operators to generate via Settings -> Nodes -> Generate Token) were silently rejected with HTTP 403. Operators followed the dialog's instructions, enrolled the node cleanly, saw reachableMode='proxy' with no negative badge, opted a stack into mesh, watched the redeploy complete, and only then discovered no bytes flow. node_proxy already authorizes every /api/* surface on the remote (deploy, exec, host console, filesystem), which is a strict superset of what the mesh proxy-tunnel does. Gating mesh more strictly was theatre, not security, and it created a UX trap with no in-product signal pointing at the scope mismatch. Change the upgrade dispatcher to accept any machine-to-machine credential: node_proxy JWT or full-admin api_token. Session cookies and restricted api_token scopes (read-only, deploy-only) remain rejected through the same paths as before. Adds positive/negative coverage in upgrade-order.test.ts for all four credential shapes so the gate stays pinned against future re-tightening. Updates user-facing copy in the mesh docs and Settings -> API Tokens description so the documented path matches the actual code. If we ever introduce a demoted node_proxy variant (audit-only, read-only fleet member), the right move is differentiated node_proxy scope claims inside the tunnel JWT (tracker F-A3), not requiring a separate token type for mesh. |
||
|
Anso
|
a38a3e0226 |
feat(mesh): route mesh traffic over Distributed API remotes (#1048)
* refactor(mesh): extract shared TCP stream switchboard
Pull the `tcp_open` / `tcp_open_ack` / `tcp_open_reverse` / `tcp_close`
+ `TcpData` handling out of the pilot agent into a reusable module so a
second caller (the upcoming proxy-mode WS handler) can run the same
frame parser, stream allocator, idle timers, and Compose-label
resolver. One parser, two callers, zero drift on a
security-sensitive protocol surface.
The pilot agent keeps its public `openMeshTcpStream` API and delegates
to a per-connection switchboard constructed in `connect()` and torn
down in `cleanupAfterDisconnect`. Existing reverse-stream and resolver
tests are rewritten to exercise the shared module directly.
* feat(mesh): route mesh traffic over Distributed API remotes
Sencho Mesh now works against remotes in Distributed API mode in
addition to Pilot Agent mode. Central opens a short-lived WebSocket
tunnel to the remote's new `/api/mesh/proxy-tunnel` endpoint on demand
and tears it down after 5 minutes of idle (configurable via
`SENCHO_MESH_PROXY_TUNNEL_IDLE_MS`). Mesh dispatch is mode-agnostic at
the routing layer; `PilotTunnelManager.ensureBridge` resolves an
existing tunnel or asks the new `MeshProxyTunnelDialer` to dial.
The Routing tab badges now use a `reachableMode` classifier: `★ Local`
for local, `pilot offline` red badge for a pilot with a down tunnel,
and a new `unreachable` red badge surfacing the specific reason
(missing token, scope not full-admin, TLS failure, remote does not
support proxy mesh). Distributed API remotes with valid credentials
show no negative badge; the tunnel opens on first dial.
Auth: the proxy-tunnel WS upgrade requires an `Authorization: Bearer`
API token with the `full-admin` scope. Lower-scoped tokens are
rejected at upgrade time so a leaked read-only token cannot reach the
mesh data plane.
Bidirectional: the proxy-mode WS handler registers itself as the
local `MeshService` reverse dialer (compare-and-swap), so meshed
containers on a Distributed API remote can dial cross-node aliases
via `tcp_open_reverse` over the same tunnel. Cross-node relays
(`PilotTunnelBridge.acceptReverseRelay`) await `ensureBridge` so
proxy-to-proxy mesh works without standing tunnels.
* docs(mesh): describe Distributed API mesh and unreachable troubleshooting
Refresh the user-facing mesh documentation to reflect that mesh works
over both Pilot Agent and Distributed API remotes. Adds a
Troubleshooting accordion entry for the new `unreachable` Routing tab
badge so operators can match a tooltip reason ("api token rejected
(scope must be full-admin)", "remote does not support proxy mesh",
"TLS handshake failed", "api_url not set", "api token missing") to a
concrete fix.
* refactor(mesh): apply /simplify review findings
Five cleanups surfaced by the post-implementation code review pass.
No behaviour change for fleets running on `main`; only internal
structure improves.
- **Deterministic container IP shared.** Extract the compose-default
network preference (`pickContainerIp`) and the conventional-name
fast path (`lookupContainerIp`) into `backend/src/mesh/containerLookup.ts`.
Both `MeshService.resolveContainerIp` (existing same-node fast
path) and the switchboard's `resolveByComposeLabels` (proxy/pilot
inbound dial) now use the same logic. Earlier the switchboard
helper grabbed the first `Object.values(Networks)` IP, which
could flip across daemon versions on multi-network containers;
fixed.
- **WS URL upgrade extracted.** New `backend/src/utils/wsUrl.ts`
exposes `httpUrlToWs(baseUrl)` that maps `http://` to `ws://` and
`https://` to `wss://`. Used in both `pilot/agent.ts` and the
proxy-tunnel dialer. The previous inline `replace(/^http/, 'ws')`
silently downgraded `https://` to `ws://` (cleartext).
- **`computeReachable` takes the pre-fetched node.** `getStatus`
already iterated `db.getNodes()`; the helper previously re-queried
by id per node (N+1 reads). Pass the row in.
- **Reachable-reason ternary -> const map.** Replace the nested
ternary in `MeshService.computeReachable` with a
`Record<DialFailureCode, string>` lookup keyed on the dialer's
failure code.
- **Activity-type ternary -> const map.** Same flattening inside
`MeshProxyTunnelDialer.logActivity`.
All mesh tests pass (85/85). Full backend suite green (2126/2126).
* fix(mesh): cache proxy dial failures and contain WS handshake errors
`ensureBridge` now consults the recent-failure cache before dialing so a
continuous mesh workload against a misconfigured proxy-mode remote does
not produce one upgrade attempt per cross-node TCP open. `recordFailure`
emits at most one activity-log entry per cache window per (nodeId, code)
so a connect-loop on a single bad remote cannot flush the ring buffer.
Failure messages run through `redactSensitiveText` before reaching the
log so any embedded Bearer / JWT / inline-URL credentials are scrubbed.
`stop()` clears the inflight map and `dial()` checks `this.stopped`
between awaits so a shutdown does not leak a half-opened bridge.
When `awaitOpen` rejects (401, 4403, TLS), the dialer attaches a noop
'error' listener before calling `ws.close()`. Without it the ws library
emits a tail 'error' on a still-CONNECTING socket that propagates as an
unhandled exception. Surfaced by a live-network test against a real
proxy-mode peer.
Adds `mesh-proxy-tunnel-live.test.ts` (skipped unless MESH_AUDIT_URL
and MESH_AUDIT_TOKEN_FILE env vars are set) covering both the happy
path and the auth-rejected path against an actual remote Sencho.
* feat(mesh): instrument proxy tunnel observability
Adds three counters to `PilotMetrics`:
- `proxy_bridges_total` (incremented on `registerProxyBridge` success)
- `proxy_dials_failed` (every failed dial attempt, not deduped)
- `proxy_idle_closes` (idle-sweep teardowns)
All three surface automatically via `GET /api/system/pilot-tunnels`
since the route returns the full `Counters` snapshot.
Gates two pre-existing always-on `console.warn` calls in
`tcpStreamSwitchboard.ts` (mid-stream socket errors and Docker resolve
failures) behind `isDebugEnabled()`. Both fire on per-stream events and
would otherwise violate the diagnostic-log safety rule under load.
Adds `mesh-tcp-stream-switchboard.test.ts` covering the forward
`tcp_open` path: real localhost dial, resolver errors (no_target,
denied), per-tunnel cap saturation, frame-routing fall-through
invariants, `tcp_close` socket teardown.
Drops `MESH_CONNECT_TIMEOUT_MS` from the public surface; only the
switchboard itself uses it.
* fix(mesh): tighten Routing tab unreachable handling
`RoutingNodeCard.tsx`: the **Add stack to mesh** button now disables
when `reachableMode === 'unreachable'`, matching the existing
TogglePill behavior. Without this, an operator on an unreachable node
could open the opt-in sheet, confirm, and watch the redeploy proceed
against a target whose mesh data plane will silently fail to route.
Also drops the leftover `status.nodeId !== -1` guard on the pilot-
offline badge.
`MeshService.ts`: renames `isMeshReachable` to `isMeshConfigured` with
the new predicate that returns true for proxy-mode remotes whose creds
are valid (the tunnel is opened on demand). `getRouteDiagnostic` now
distinguishes "routable" (configured) from "pilotLive" (live tunnel
state, only meaningful for pilot mode); without the split, every idle
proxy-mode route would report `tunnel down`.
`MeshService.setReverseDialer` warns when the unconditional install
path silently overwrites a non-null current dialer. By topology a
Sencho is either pilot or central, so the branch flags a misconfigured
deployment rather than an expected race.
Drops the dead `export { ReverseTcpStreamHandle }` re-export from
`pilot/agent.ts`; its only consumer imports straight from
`mesh/tcpStreamSwitchboard.ts`. Fixes a stale doc comment in
`MeshService.ts` that referenced the wrong source file.
Adds `mesh-proxy-tunnel-handler.test.ts` covering the WS handler
lifecycle: pilot-mode 404 rejection, post-upgrade reverse-dialer
install, concurrent-upgrade 1013 rejection (single-tenant slot), and
error-path teardown.
Updates the troubleshooting accordion in the user docs to mention the
disabled-state behavior.
* fix(mesh): close ESLint and CodeQL findings on the proxy-tunnel diag logs
Remove the unused `reverseDialer` local in `meshProxyTunnel.ts`; the
closure target is `localDialer` above and this assignment was always
dead. Strip line breaks inline on the three `MeshProxyTunnelDialer`
diag log lines so CodeQL `js/log-injection` data flow recognises the
sanitisation that `sanitizeForLog` already performs.
No behaviour change; the diag logs render the same characters they
do today.
|
||
|
Anso
|
44e40afb62 |
fix(schedules): align Schedules surface with backend tier gate for Skipper admins (#1047)
The Schedules sidebar entry was hidden from Skipper admins even though the backend permits them to create and run update, scan, and snapshot schedules. The action picker also showed all 10 actions to every paid admin, so a Skipper selecting Restart, Prune, or any auto_* lifecycle action would 403 on submit. Changes: - Extract SKIPPER_SCHEDULED_ACTIONS as the single source of truth in tierGates.ts; both requireScheduledTaskTier and the GET /scheduled-tasks list filter now reference it (replaces a duplicate local constant in scheduledTasks.ts). - Move the Schedules nav entry from the Admiral block into the isPaid && isAdmin block in useViewNavigationState.ts, mirroring the existing Auto-Update pattern. Console and Audit stay Admiral-only. - Filter the create-form action picker in ScheduledOperationsView.tsx by license variant. Skipper sees Auto-update Stack, Auto-update All Stacks, Fleet Snapshot, and Vulnerability Scan; Admiral sees the full set. - openCreate now defaults formAction to the first visible option so Skipper starts with a valid choice instead of the Admiral-only Restart. Tests: - Add Skipper-variant POST coverage in scheduled-tasks-routes.test.ts: three allow cases (update / scan / snapshot) and a six-action rejection loop covering restart / prune / auto_backup / auto_stop / auto_down / auto_start. - Flip the Skipper assertion in useViewNavigationState.test.tsx to expect scheduled-ops alongside auto-updates. |
||
|
Anso
|
5461bc316b | fix: harden stack management operations (#1046) | ||
|
Anso
|
8dd0fce621 |
fix(fleet): show capabilities, version, metrics, and stacks for pilot-agent nodes (#1044)
When a pilot-agent node was the active node, the UI rendered "does not advertise this capability" across most tabs, a perpetual "Update available" badge, and a Fleet card body with blank CPU/RAM/Disk and "No stacks found". The cause was central-side aggregators in /api/fleet/* and /api/nodes/:id/meta only fanning out to proxy-mode remotes via node.api_url + node.api_token, which are null for pilot-agent. Route every affected aggregator through NodeRegistry.getProxyTarget so the loopback URL backed by the active pilot tunnel is used uniformly: - /api/nodes/:id/meta and /api/fleet/update-status fetch via the new NodeRegistry.fetchMetaForNode helper (resolves the target, delegates to fetchRemoteMeta, returns the shared OFFLINE_META on null). - fetchRemoteNodeOverview, /api/fleet/configuration, /api/fleet/node/:nodeId/stacks, and the stack-containers drilldown fetch through target.apiUrl with conditional Authorization. - fetchRemoteMeta omits the Authorization header when the token is empty (pilot-agent loopback) instead of sending a malformed Bearer string. - Pilot-agent rows preserve pilot_last_seen and mirror it into last_successful_contact so the Fleet "last seen" cell renders the recent tunnel timestamp during a brief reconnect. Pilot-mode capability filter excludes capabilities whose central-pilot path is not yet wired (host-console, self-update). Without this, the Console tab would surface for an Admiral pilot session and click through to central's host because the WS upgrade handler still gates on api_url + api_token. Filtered capabilities are removed at boot via applyPilotModeCapabilityFilter when SENCHO_MODE=pilot. Cache invalidation on tunnel-up: the meta cache for a reconnecting pilot is dropped so the next request rebuilds capabilities and version through the live bridge instead of waiting for the 3-minute TTL. The namespace constant moves to helpers/cacheInvalidation.ts alongside the new invalidateRemoteMetaCache helper. Husky commit-msg hook: add the missing shebang and a .gitattributes rule pinning .husky/* to LF line endings so commits do not fail with "Exec format error" on Windows shells where autocrlf=true converts the hook to CRLF. Tests cover Authorization-header behavior, pilot-mode filter idempotency, fetchMetaForNode dispatch (offline target, pilot-agent loopback, proxy-mode), and the four affected fleet routes for pilot-agent both when the tunnel is up and when it is down. |
||
|
Anso
|
e7a3b544c0 |
fix: harden auto-heal policies (#1042)
* fix: harden auto-heal policies * fix: resolve auto-heal lint failure |