mirror of
https://github.com/Studio-Saelix/sencho.git
synced 2026-08-19 06:46:23 +00:00
2f2401df68e9a50dbd6d724dce0a0a181dc21f44
109 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Anso
|
2f2401df68 |
fix(fleet): route remaining fleet dispatches through getProxyTarget for pilot-agent nodes (#1152)
* fix(fleet): route remaining fleet dispatches through getProxyTarget for pilot-agent nodes PR #1123 migrated POST /api/fleet/nodes/:id/update to use NodeRegistry.getProxyTarget so pilot-agent rows (no api_url / api_token) participate in remote update via the tunnel loopback. The same bug shape lived on at ten sibling fleet-dispatch sites: each read node.api_url and node.api_token directly, returning "Remote node not configured." against pilots, or silently filtered pilot rows out of a fan-out loop. Migrate every remaining fleet-wide remote dispatch to the same pattern: - routes/fleet.ts: fleet-stop, fleet-prune, prune/estimate, snapshot restore (4 sites) -> getProxyTarget + mode-aware error copy + conditional Authorization header - routes/imageUpdates.ts: fleet status + fleet refresh (2 sites) -> swap n.api_url filter for getProxyTarget != null and use target.apiUrl, so pilot rows appear in the aggregated image-updates view instead of being silently excluded - utils/snapshot-capture.ts: captureRemoteNodeFiles (1 site) -> same pattern; CaptureNode interface gains required mode field so the thrown error message picks the pilot-tunnel copy automatically - services/SecretsService.ts: resolveEnvFileRemote, readEnvRemote, writeEnvRemote (3 sites) -> same pattern; thrown errors now use the shared formatNoTargetError helper instead of leaking api_url/api_token field names Extract the previously-private noTargetMessage helper from fleet.ts into utils/remoteTarget.ts as formatNoTargetError so SecretsService, snapshot-capture, and the fleet routes share one copy of the mode-aware error string. Add 10 regression tests in fleet-pilot-dispatch-parity.test.ts covering each migrated route + the snapshot-capture utility: dispatch through the loopback target with no Authorization header for pilots, and a mode-aware error when the tunnel is disconnected. FleetSyncService (4 additional sites) carries an api_url-anchored targetIdentity in the wire protocol; pilot support there needs a protocol-level identity decision and stays as a separate follow-up. * fix(fleet): throw tunnel-disconnected error from resolveEnvFileRemote Codex audit flagged that resolveEnvFileRemote returned null when getProxyTarget was null. That predates the parity migration but the migration was the right place to fix it: the null flowed through readExistingEnv into previewPushDiff / executePush as "env file not found", which is wrong (the env exists, the node is unreachable). Throwing formatNoTargetError(node) here lets the existing catch arms in previewPushDiff (lines 450-453) surface reachable=false with the tunnel-disconnected message on the right axis, and executePush picks up the same shape via its outer catch. Also drop overstated coverage claims from the parity test header (snapshot restore + SecretsService were never actually exercised in this file, only structurally identical via tsc), and fix two describe labels that read /api/labels/* instead of the mounted /api/fleet/labels/*. |
||
|
Anso
|
60f893a81f |
feat(fleet): move bulk Remote OTA updates to Community tier (#1151)
Drops `requirePaid` from `POST /api/fleet/update-all` so Community admins can dispatch bulk node updates. Per-node OTA was already Community- reachable (admin-only); this completes the move so the full Remote OTA surface ships at Community. Frontend mirrors the backend: removes `canBulkUpdate` from NodeUpdatesSheet so the "Update all (N)" affordance is purely data- driven on `updatableRemoteCount > 0`. Docs realigned to drop fence-spec and Skipper-only phrasing on the Update all bulk action: - features/licensing.mdx: Community line now lists Remote OTA (per-node and Update all); Skipper Fleet Actions parenthetical drops "bulk update all" - features/remote-updates.mdx: Note rewritten to role-only requirement - features/fleet-view.mdx: Update all (n) bullet drops the tier clause - features/overview.mdx: Fleet View and Remote updates blurbs drop the Skipper/Admiral fences - operations/upgrade.mdx: Note rephrased without naming tiers Test coverage: - fleet.test.ts: tier-gating spec flipped to assert Community access - fleet-pilot-update.test.ts: bulk-OTA dispatch suite now spies tier to Community so it doubles as a regression guard |
||
|
Anso
|
8a3889dc67 |
feat(security): move managed Trivy auto-update to Skipper tier (#1150)
* feat(security): move managed Trivy auto-update to Skipper tier Drop the gate on the managed Trivy auto-update toggle from Admiral to Skipper so it lives alongside the rest of Sencho's automation features (auto-heal, scheduled ops, per-stack image auto-update, scan policies) instead of behind the enterprise-control tier. Backend: `PUT /api/security/trivy-auto-update` switches from `requireAdmiral` to `requirePaid`. The 24h scheduler tick in SchedulerService that reads the setting is tier-neutral and picks the new gate up automatically. Frontend: SecuritySection.tsx swaps the inline `isAdmiral` conditional on the toggle render for `isPaid`. The local `isAdmiral` derivation and the `useLicense` import become unused and are removed. Docs: licensing, overview, vulnerability-scanning matrix, trivy-setup, and the settings reference now read Skipper consistently for this feature. * fix(security): require admin role on trivy-auto-update toggle Independent audit of the prior commit flagged that PUT /api/security/trivy-auto-update had no admin-role guard. The route was authenticated and tier-gated, but the global /api authGate only authenticates and `requirePaid` only checks tier. Any paid viewer could flip the global trivy_auto_update setting via a direct API call. Add `requireAdmin` ahead of `requirePaid`, matching the pattern used by every other mutating route in this file (sbom, policies, suppressions, misconfig-acks). Add route tests covering paid admin allowed, paid viewer rejected, community admin rejected, and unauthenticated rejected. |
||
|
Anso
|
b740dd1078 |
feat(resources): protect Sencho's own image, network, volumes from deletion (#1149)
* feat(resources): protect Sencho's own image, network, volumes from deletion Adds SelfIdentityService that reads HOSTNAME at startup and inspects the running Sencho container via Dockerode to record its image ID, attached networks, named volumes, and container ID. The classification API marks these with isSencho:true, destructive delete routes return 423 Locked when the target matches self, the orphan-containers API filters the Sencho container out so it cannot be selected and purged from the Unmanaged tab, and the managed-prune path adds an explicit self filter for defense-in-depth on top of Docker's in-use semantics. The Resources view renders a Sencho pill alongside the managed badge on matching rows and disables the trash control with a hover tooltip. When Sencho runs outside Docker (dev mode), inspect returns 404, the service stays empty, and every isOwn* returns false so today's behaviour is preserved. * fix(resources): handle sha256-prefixed image IDs and custom hostnames Addresses independent-review findings on PR #1149: - Strip sha256: prefix in POST /api/system/images/delete before validating the ID, matching the inspect route's handling. Without this, /system/images responses round-trip through the UI as sha256:<hex> and got 400 Invalid image ID format before rejectIfSelf could run. - Add /proc/self/cgroup fallback to SelfIdentityService so custom --hostname, Compose hostname:, or --uts=host setups still self-identify. HOSTNAME inspect runs first; on 404 the service parses the cgroup file for a 64-hex container ID (cgroupv1 docker, cgroupv2 docker, podman libpod formats all covered) and retries inspect with that ID. - Restrict prefix matching in isOwnNetwork / matchesId to hex-shaped candidates (12 to 64 hex chars), so a non-Sencho network whose name happens to start with a hex prefix of Sencho's network ID is no longer flagged as self. - Trim the resources.mdx Note to customer-visible behaviour without enumerating every tab. - New tests: prefixed-image-ID 200 path, three cgroup file format parses (v1, v2, podman) plus the no-match and missing-file cases, HOSTNAME-404-then-cgroup-success fallback path, name-collision regression for the hex-only prefix rule, and an empty-cache no-regression check. Test hygiene: mockReset on the inspect stub and restoreAllMocks in afterEach so spies do not leak across tests. * chore(security): VEX not_affected for CVE-2026-46680 (containerd in docker-compose) Trivy now flags CVE-2026-46680 HIGH on usr/local/lib/docker/cli-plugins/docker-compose, which statically embeds github.com/containerd/containerd/v2 v2.2.3 (compose v5.1.3's resolved module graph). The CVE is a runtime-executor flaw: containerd's runc invocation can be tricked into running a Kubernetes pod marked runAsNonRoot as root via crafted user ID handling. The vulnerable code path is reached only by containerd-shim executing a container with a populated OCI runtime spec on the daemon side. docker-compose vendors the containerd Go module purely as a client (gRPC stubs, API types, shared utilities); it never executes containers and never enforces runAsNonRoot. Sencho's compose usage (up / down / ps against user-authored files) cannot construct a Kubernetes pod security context. The vulnerable path is unreachable. Adds a not_affected entry to security/vex/sencho.openvex.json with justification vulnerable_code_not_in_execute_path, bumps version 5 to 6, and updates last_updated to 2026-05-22 per Directive 23. |
||
|
Anso
|
66b84932e0 |
feat(notifications): move Notification Routing to Skipper tier (#1145)
* feat(notifications): move Notification Routing to Skipper tier Notification routing is automation (route alerts to channels by rules), not enterprise compliance. Aligning the gate with Skipper makes the tier boundary read consistently with the rest of the automation surface (webhooks, auto-update, auto-heal, scheduled tasks). Backend: requireAdmiral -> requirePaid on the five /api/notification-routes endpoints. Dashboard configuration-status now exposes the routing-rules row to any paid tier. Frontend: settings registry tier flipped to skipper; the Admiral wrapper around NotificationRoutingSection is removed (the inner CapabilityGate stays, preserving forward-compat with older remote nodes). Tests: added a tier-enforcement describe block covering Skipper (200) and Community (403 PAID_REQUIRED on all five endpoints). Docs: refreshed alerts-notifications, licensing, overview, dashboard, troubleshooting, and reference/settings; cleaned one fence-spec line per Directive 31. * fix(notifications): address audit findings on tier-move PR Docs: rewrite three lines that survived the initial sweep. The dashboard "you do not see a locked placeholder" clause and the settings.mdx "hidden on Community and Skipper" phrase were Directive 31 fence-spec. The alerts-notifications troubleshooting note still said "an Admiral routing rule" and contradicted the tier move. Tests: the Community-negative cases on POST/PUT/DELETE/POST :id/test could not distinguish requirePaid from a stray requireAdmiral, because Community fails on the tier check before variant is read. Adding Skipper-positive coverage per endpoint locks the gate identity in. Replace the leaky mockReturnValueOnce with a per-test mockReturnValue plus an afterEach restore so spies cannot bleed across tests. |
||
|
Anso
|
535023b350 |
feat(files): open stack file explorer to every tier (#1144)
* feat(files): open stack file explorer to every tier
Drop the `requirePaid` guard from the seven stack-file write routes
(download, upload, write-content, delete, mkdir, rename, chmod) and
remove every matching `isPaid` check from the file-explorer frontend.
Stack edit permission (RBAC) continues to gate every write end-to-end.
The file explorer is the primary way a user touches a stack's on-disk
surface; gating it behind a paid tier conflicted with the principle
that Community covers single user-initiated actions while paid tiers
add automation and governance.
* docs(files): treat download as a read action, not a write
Download has no `requirePermission('stack:edit')` on the route and no
`canEdit` gate in the UI, so viewer accounts can download. Update the
top paragraph to list download under reads, and rewrite the
troubleshooting accordion to describe the actual gating (a file must be
selected) instead of asserting a role gate that does not exist.
* test(e2e): align stack-files spec with the new tier rule
The community-tier describe block asserted that the Upload control is
absent and the editor shows a `Read-only` chip; the admin-tier block
skipped on Community via `test.skip(tier !== 'paid')`. Both rules
reflected the previous gate, where writes required a paid tier.
Writes are now gated on the `stack:edit` role, not on the license tier.
Repurpose the community describe to assert that a Community admin
under a mocked community license still sees the Upload control and an
editable Save button. Drop the obsolete tier-skip in the admin describe
so upload, edit, delete, and download exercise on every tier. Update
stale comments to reference the role gate.
|
||
|
Anso
|
380ed6fd50 |
feat(cloud-backup): make Custom S3-compatible target available on every tier (#1143)
* feat(cloud-backup): make Custom S3-compatible target available on every tier Sencho Cloud Backup remains an Admiral feature; the bring-your-own-bucket Custom S3 target is now reachable on Community and Skipper as well. Backend splits the per-route Admiral gate into two helpers: operations that touch the saved provider use gateForCurrentProvider, PUT /config uses gateForRequestedProvider against the body. /provision and /usage stay requireAdmiral because they are Sencho-only by definition; GET /config is ungated so any tier can read its own stored configuration. Frontend drops the AdmiralGate wrapper on the Cloud Backup section, filters the Sencho provider option out of the dropdown for non-Admiral users, and gates the per-snapshot cloud-upload affordance on "cloud-backup configured" instead of Admiral tier. Dashboard Configuration row is no longer locked on lower tiers. Sidebar registry tier on cloud-backup goes from 'admiral' to null. Docs and licensing breakdown restate the rule once per page without fence-spec. * fix(cloud-backup): keep downgraded sencho config off the upload surface If an Admiral configured Sencho Cloud Backup and the license later drops to Skipper or Community, the saved provider is still 'sencho'. The FleetSnapshots cloud-upload affordance now requires either provider= custom (every tier) or provider=sencho with an active Admiral license, so a downgraded admin never sees an upload button that the backend would 403 on click. Also tidies the Fleet Backups doc, which still claimed the cloud-upload icon was Admiral only; the icon now renders whenever a Cloud Backup target is configured. |
||
|
Anso
|
c491d309c1 |
feat(schedules): make every scheduled action available on Skipper (#1141)
Collapse the Admiral carveout that restricted restart, prune, auto_backup, auto_stop, auto_down, and auto_start schedules to the Admiral variant. Scheduled Operations stays at Skipper+ (paid). The action picker now lists every supported operation for any paid admin, and the scheduler runner executes every action on either variant. |
||
|
Anso
|
6f301e005a |
feat(fleet): redesign Fleet Action cards to the System Sheet recipe (#1137)
Lift <FleetActionCard> primitive from audit §20 / DESIGN §9.12. Migrate the three v1 cards (stop-by-label, bulk-label-assign, prune-fleet-wide) to consume it: cyan rail on every card, action class as a chip, blast radius as a live readout in the toolbar, preview section replaces the warning banner, footer carries reversibility plus freshness. Drop the per-card tone rail, the icon prop, and cards/tone.ts. Add /fleet/labels/match-preview and /fleet/prune/estimate for the live blast readouts (chrome falls back to "preview unavailable" if either 404s). Add dryRun: true to the existing fleet-stop and fleet-prune endpoints so the Dry run button rehearses the full code path (locks, per-node fan-out, remote propagation) without firing the destructive leaf call. Result flows into the existing ResultsList. Extend <SheetSection> with optional meta. Add --action-transformative semantic token. |
||
|
Anso
|
d0e140444a |
feat(api-tokens): make API tokens available on every tier (#1136)
API tokens are credential management, not a tier-gated capability. Remove the Admiral gate from the POST/GET/DELETE handlers, drop the AdmiralGate wrapper from the settings UI, set the registry entry's tier to null so the tab renders on every tier, and update the docs Note to state availability plainly. The three permission scopes (read-only, deploy-only, full-admin), the 25-token-per-user cap, the per-token 200 req/min rate limit, the sen_sk_ prefix format, and the SHA-256 hashed storage are all unchanged. The Vitest suite now runs at Community tier to prove every code path works without a paid license. A new "API token tier accessibility" describe block mints all three scopes via POST /api/api-tokens to lock the behavior. |
||
|
Anso
|
982c7830b1 |
fix(mesh): address audit follow-ups (early-data, recompose, admiral gate, error codes, docs) (#1126)
* fix(mesh): buffer early TcpData on reverse-relay path
The reverse-relay code dropped the local-shaped reservation and any
TcpData frames buffered in it before acceptReverseRelay had wired up
the target stream. A peer that sent a request body immediately after
tcp_open_reverse lost those bytes when the target lived on a third
node, since reverse_local buffers them but reverse_relay did not.
Carry the reservation through acceptReverseRelay: transplant its
pendingData and pendingBytes into the new reverse_relay state, gate
TcpData writes on targetOpen, and flush buffered frames into the
target stream before sending tcp_open_ack. Mirrors the reverse_local
pattern. Regression test fires tcp_open_reverse plus an immediate
TcpData while ensureBridge is in flight and verifies the bytes
arrive intact, in order, before the ack.
* fix(mesh): recompose affected stacks when node-level mesh is disabled
disableForNode used to clear DB rows and override files but leave the
running containers attached to sencho_mesh with stale /etc/hosts
alias entries until an operator redeployed every stack by hand.
Mirror optOutStack: after the existing alias/forwarder cleanup, call
regenerateOverridesAcrossFleet, cascadeRecomposeAcrossFleet, and
triggerRedeploy for each previously meshed stack on the disabled
node so containers detach from sencho_mesh and shed the alias
entries they owned. The disabled node's mesh_stacks rows are deleted
before the cascade so listMeshStacks returns the right set with no
skip tuple required. Route threads the actor through actorFor(req)
for parity with optInStack/optOutStack. Tests cover the redeploy
fan-out, the cascade no-skip-tuple invariant, and the default actor
fallback for non-route callers.
* fix(mesh): require Admiral on the WS proxy-tunnel upgrade
HTTP mesh routes in routes/mesh.ts all enforce requireAdmiral, but
the /api/mesh/proxy-tunnel WS upgrade accepted any node_proxy or
full-admin api_token regardless of the receiver's license. A node
downgraded from Admiral kept serving mesh data-plane traffic to a
sibling central while refusing every mesh management call.
Read the receiver's local LicenseService at the upgrade and 403 when
the tier is not paid+admiral. The check sits after the existing
credential gate and uses LicenseService directly rather than
effectiveTier (which trusts forwarded proxy headers); a remote peer
dialing in cannot be trusted to assert our entitlement. Dialer and
node_proxy token format are unchanged. Three regression tests cover
community-tier node_proxy, skipper-tier node_proxy, and
community-tier full-admin api_token all rejected with 403.
* fix(mesh): handle no_target alongside push_failed in inspectStackServices
proxyFetch throws MeshError('no_target') when getProxyTarget returns
null (pilot tunnel offline, proxy bridge unreachable), but the
inspectStackServices catch branch only matched push_failed. Offline
remotes fell through to the generic 'remote unreachable' error log,
which the Routing tab surfaces as an unexpected fault.
Match both error codes and emit the operator-friendly warn message
that names the unreachable node and the error code. Regression test
spies on console.warn/console.error to pin the branch.
* docs(mesh): align env defaults and forwarder comments with current architecture
SENCHO_MESH_PROXY_TUNNEL_IDLE_MS in .env.example carried the old
five-minute idle-close value (=300000), but the code default is
DEFAULT_IDLE_TTL_MS=0 (persistent tunnel). Copying the example
silently reintroduced the idle-close behavior the dialer removed.
MeshForwarder.ts's leading docblock and inline listen comment still
described host-network mode plus extra_hosts: host-gateway as
required for forwarder reachability. Sencho runs in standard bridge
mode and attaches to the shared sencho_mesh network at a stable IP;
meshed user containers reach the forwarder by that IP directly.
Flip the env default to 0, rewrite the env comment to describe the
persistent behavior and the opt-in for idle teardown, and rewrite
both forwarder comments to match the bridge-network reality.
* fix(mesh): trust forwarded tier on proxy-tunnel WS and remove remote overrides on disable
Admiral entitlement on the WS data plane now follows the same trust model
as the HTTP mesh routes: the central asserts its tier via x-sencho-tier
and x-sencho-variant on the WS handshake, and the receiver trusts those
headers only when the upgrade carries a node_proxy credential. When no
headers are present or the credential is a full-admin api_token, the
receiver falls back to its own local license. Without this an Admiral
central could be rejected by a Community remote and a Community central
could dial a locally-Admiral remote.
disableForNode now routes through removeOverrideFromNode so override
files pushed earlier via applyLocalOverride are removed on remote nodes
via DELETE /api/mesh/local-override/:stack. Sequential awaits are wrapped
in Promise.allSettled to match regenerateOverridesForNode's parallel
push pattern.
Other changes:
- Cover the post-state-swap buffering window in the reverse-relay test
(TcpData arriving after openTcpStream returns but before target open).
- Refresh stale MeshService comments that still referenced the removed
sidecar layer and host-network listener model.
* test(mesh): pin removeOverrideFromNode remote HTTP shape
The disableForNode regression test mocks removeOverrideFromNode itself,
so a regression inside the helper would not be caught. Add a narrow
contract test that spies on global fetch and asserts the request shape:
DELETE /api/mesh/local-override/:stack against the resolved proxy
target, with Authorization Bearer plus the x-sencho-tier and
x-sencho-variant headers. Also covers the encodeURIComponent path and
the swallowed-network-error behavior the disable cascade relies on.
* test(mesh): use createTestApiToken helper in proxy-tunnel api_token gate test
The full-admin api_token branch of the WS Admiral gate test inlined the
canonical generateApiToken + sha256 + addApiToken triple that already
lives in the createTestApiToken helper. Switching to the helper removes
the duplicated insertion logic and aligns this test with the helper used
by the other api_token call sites.
|
||
|
Anso
|
0b50c88eb3 |
fix(fleet): route remote update trigger through getProxyTarget (#1123)
POST /api/fleet/nodes/:id/update and POST /api/fleet/update-all read node.api_url + node.api_token directly, so a pilot-agent row (which carries neither) returned "Remote node not configured." and was filtered out of bulk update. Both routes now resolve the target via NodeRegistry.getProxyTarget(), which returns the loopback URL for an active pilot tunnel and the configured api_url for proxy-mode remotes. fetchMetaForNode replaces the direct fetchRemoteMeta call for the self-update capability check. self-update is removed from PILOT_DISABLED_CAPABILITIES so a Compose-deployed pilot can advertise it; the host-console filter stays. SelfUpdateService still gates the local-end capability on the container actually carrying docker-compose labels, so a docker-run pilot self-disables and the Fleet UI sees a clean 503 instead of an ambiguous failure. Tests: new fleet-pilot-update covers the four single-node branches (success via loopback, null target, no self-update capability, meta offline) and two update-all cases (mixed-fleet dispatch, all-targets- null skip). capability-registry-pilot rewired to assert the new filter set. |
||
|
Anso
|
282ab8d844 |
fix(pilot): let SENCHO_PUBLIC_URL override the request Host in enrollment (#1122)
The enrollment minter inferred SENCHO_PRIMARY_URL from the request Host header, which baked loopback or LAN addresses into the compose YAML when the admin opened Add Node on the central's own machine. Pilots on a different network (a public cloud VPS, for example) cannot dial that. SENCHO_PUBLIC_URL on the primary now wins when set and well-formed (http(s)://, no loopback). Trailing slashes are stripped. Falls back to the request Host when unset or invalid. |
||
|
Anso
|
3ad6ea9c5d |
feat(pilot): make Docker Compose the canonical pilot enrollment payload (#1121)
* feat(pilot): make Docker Compose the canonical pilot enrollment payload The Add Node dialog for a pilot-agent now returns a Compose snippet instead of a single-line docker run command, and the enrollment dialog walks the operator through a save-and-up flow. The compose project name and container name align with what SelfUpdateService looks up at boot, so a Compose-deployed pilot can be updated remotely through the Fleet view without intervention on the remote host. Docs (pilot-agent, remote-updates) were rewritten to match. * test(e2e): align pilot enrollment spec with Compose payload The spec was written against the docker-run payload; it now asserts the Compose YAML the dialog renders. |
||
|
Anso
|
e05099f2a1 |
fix(fleet-sync): make control-identity-mismatch sticky and surface in UI (#1117)
Treat 409 CONTROL_IDENTITY_MISMATCH from a replica as a non-retriable
failure instead of looping the same 409 through the 5-minute retry
service forever and silently writing identical failure rows.
Backend
- DatabaseService: add `sticky_error_code`, `sticky_error_expected`,
`sticky_error_got` columns to `fleet_sync_status` via an idempotent
migration. New methods setFleetSyncSticky, getFleetSyncStickyCode,
clearFleetSyncStickyForNode. recordFleetSyncSuccess clears the sticky
flag on a clean push. getFailedSyncTargets SQL adds
`AND sticky_error_code IS NULL` so the retry loop skips sticky rows.
- FleetSyncService.executePushToNode: short-circuits at the top when
sticky is set (covers event-driven pushResourceAsync calls). On a 409
with code CONTROL_IDENTITY_MISMATCH, records the failure once and
pins sticky with the expected/got fingerprints carried in the 409 body.
- routes/nodes.ts: new POST /api/nodes/:id/fleet-sync/reset-anchor.
Admin + paid + node:manage. Proxies POST /api/fleet/role/reanchor to
the peer with `{override:true}` using the stored Bearer node_proxy
token. On peer 200, clears every sticky row for the node so the next
push re-anchors and resumes replication. Distinct 502 / 504 responses
for peer-rejected / peer-unreachable so the UI can show a useful toast.
Frontend
- New lib/fleetSyncApi.ts + hooks/useFleetSyncStatus.ts. Polling hook
(30s visibilityInterval) skips fetch when !isPaid.
- NodeManager.tsx: destructive banner per affected node listing both
fingerprints, with `Reset anchor on peer` and `Remove node` buttons.
Hidden for community-tier users via empty hook data.
- FleetConfiguration.tsx (Fleet -> Status): read-only `Policy sync`
SummaryRow per remote node card. In sync / degraded / paused with
a tooltip; no action buttons (the action lives in NodeManager).
Tests
- fleet-sync-service.test.ts: 4 new cases for sticky-set on first
mismatch, short-circuit on subsequent pushes, null fingerprints,
and non-mismatch failures not setting sticky.
- database-fleet-sync-sticky.test.ts (new): 6 cases pinning the DB
contract incl. retry-loop SQL filter and migration idempotency.
- nodes-fleet-sync-reset-anchor.test.ts (new): 6 cases covering
happy path, peer 401 -> 502, peer unreachable -> 504, local-node
rejection, unknown node id, and community-tier 403.
Gate parity (Directive 30): the new POST .../reset-anchor enforces
requireAdmin + requirePaid + node:manage (matches the existing read at
GET /api/fleet/sync-status). UI banner + SummaryRow only render when the
hook returns data, which it only does for paid-tier authed users. No
existing tier-gate file moved; this is greenfield parity.
Auth audit: the peer's POST /api/fleet/role/reanchor route already uses
requireAdmin, which accepts the central's stored node_proxy Bearer
token because authMiddleware maps `scope === 'node_proxy'` to
`req.user = { username: 'node-proxy', role: 'admin', userId: 0 }`.
No widening required.
Backend tsc clean. Frontend tsc -b clean. 59 fleet-sync tests pass; full
backend suite green minus the pre-existing Windows-only file-lock flake
on filesystem-backup.test.ts that reproduces unchanged on main.
|
||
|
Anso
|
6722335a79 |
fix(stack-update): refresh frontend state automatically after a stack update (#1113)
After applying a stack update the sidebar's "update available" dot stayed visible and the stack's status indicator was stuck on the optimistic value until the page was manually refreshed. Two root causes: 1. Image-updates state refresh was a fire-and-forget call in some paths and entirely missing from the bulk-update, auto-update, and state-invalidate WebSocket-handler paths. 2. stackActionsRef.current was resynced only at render time, so the post- update refreshStacks(true) running in the action's finally block read a stale "busy" map and preserved the optimistic mask via prev[file] ?? status. Backend now broadcasts a state-invalidate event with scope='image-updates' and action='stack-updated' after every successful update (single-stack route and auto-update loop). The frontend useNotifications hook routes this to a new onImageUpdatesChange callback wired to fetchImageUpdates in EditorLayout, so every connected client refreshes the dot through the same code path. Bulk update also calls fetchImageUpdates directly for fast local feedback, and setStackAction/clearStackAction now keep stackActionsRef synchronously in sync with state so the busy-stack check inside refreshStacks observes the cleared map immediately. Adds 3 unit tests covering the new WS branch (positive, scope-mismatch negative, auto-update-settings-changed negative). |
||
|
Anso
|
523ba5854c |
fix(stacks): return 404 for nonexistent stacks on deploy/down/update (F-7) (#1108)
POST /api/stacks/:name/{deploy,down,update} previously returned HTTP 500
with body {"error":"spawn docker ENOENT"} when invoked against a stack
whose compose directory was missing. The status code was wrong (the
named resource did not exist, so 404 is the right answer) and the
message misled operators into thinking the docker CLI was unavailable.
Add a small requireStackExists(nodeId, stackName, res) helper in
routes/stacks.ts that validates the stack name and confirms a compose
file is present via FileSystemService.hasComposeFile before any of the
three handlers spawn docker compose. The helper is called immediately
after requirePermission and before runPolicyGate so unauthorized
callers still get 403 first and the policy gate never runs against a
phantom stack.
In ComposeService.execute(), narrow the child.on('error') handler so
the genuine docker-binary-missing case (ENOENT on the spawn itself)
rejects with "Docker CLI unavailable on this node" instead of the raw
"spawn docker ENOENT". This is defense in depth for the rare case the
pre-check cannot cover, and it fixes the misleading-message half of
the bug as well.
Cover the new contract with stack-actions-missing-stack.test.ts (four
cases: deploy/down/update return 404, invalid name returns 400). Mock
ComposeService as a tripwire so a future code path that bypasses the
guard would fail loudly. Fix stacks-failure-notifications.test.ts by
adding hasComposeFile to its FileSystemService partial mock so the
existing happy-path-error-handling cases continue to flow into
ComposeService.
|
||
|
Anso
|
1f673073ca |
feat(fleet): add fleet-wide Docker prune to Fleet Actions (#1104)
Adds a third card to the Fleet Actions tab that fans out Docker prune (images, volumes, networks) across every node in one submit. Local nodes call DockerController under a bulk-prune lock; remote nodes receive one POST /api/system/prune/system per target. Per-node + per-target results with reclaimed bytes are surfaced inline via ResultsList. Tier: Skipper / Admiral (requirePaid + requireAdmin), matching the rest of Fleet Actions. The frontend card is mounted inside the existing isPaid branch at FleetActionsTab; no new frontend gate is required. The card uses an amber accent rail and the Eraser icon so it reads as 'cleanup' rather than 'destructive stop'. Scope toggle defaults to Managed only (Sencho-tagged resources) with an All unused option that escalates the destructive-confirm copy. |
||
|
Anso
|
9f238e187c |
fix(mesh): cascade opt-out when a stack is deleted (F-1 / F-14) (#1096)
DELETE /api/stacks/:name now calls MeshService.optOutStack after the DB cleanup so the mesh_stacks row, the override file under <DATA_DIR>/mesh/overrides/<nodeId>/, and any derived aliases do not outlive the deleted stack. Pre-fix, those artifacts leaked and the reconcile loop logged "No compose file found for stack" every tick. optOutStack is idempotent (early return when the stack was never opted in) and already cascades override-regen plus recompose across the rest of the fleet, so peers' /etc/hosts drop the dropped alias. The new cascade call is best-effort relative to the delete itself: mesh cleanup failures warn but never regress the delete contract. New regression test asserts three contracts: cascade on delete, no-op on never-meshed, and 200 + warn when the cascade rejects. |
||
|
Anso
|
f6e42535c8 |
fix(mesh): route peer→central traffic over the existing forward WS (#1094)
* fix(mesh): route peer→central traffic over the existing forward WS The reverse mesh callback path (`/api/mesh/proxy-tunnel-from-peer`) needed SENCHO_PRIMARY_URL on central plus a publicly reachable origin from the peer's perspective. In a typical homelab where central sits behind NAT, peer→central dispatch silently failed at the dialer's short-circuit and the headline "call any service on any node by hostname" worked one way only. The forward WS at `/api/mesh/proxy-tunnel` is already bidirectional end to end. Make the bridge a persistent control-plane primitive: dial every mesh-enabled proxy peer at startup, reconcile every 60 s, never idle-close. Peer→central traffic multiplexes over the same WS via `tcp_open_reverse`. Removed: - `meshProxyTunnelFromPeer.ts` WS handler and dispatch - `MeshCentralRegistry`, `PeerToCentralMeshSessionDialer` - `mesh_handshake` first-frame state machine in `meshProxyTunnel.ts` - `maybeSendBootstrap`, `buildHandshakeFrame` in the dialer - `mesh_proxy_callback_bootstrap` capability and `maybeWarnUnsetPrimaryUrl` - `mesh_centrals` table (drop migration; greenfield, no users) - `PilotTunnelManager.replaceOrRegisterProxyBridge` (dead after handler removal) - twelve associated unit/integration tests plus the peer-recovery branch in `MeshService.openCrossNode` Added: - `MeshService.proactiveBridgeFanout` selects every mesh-enabled proxy peer (no longer gated on `mesh_stacks` rows) - `startBridgeReconcileLoop` runs the fanout every 60 s (override via `SENCHO_MESH_RECONCILE_INTERVAL_MS`) - `MeshProxyTunnelDialer` default idle TTL is now `0` and exposes `isDialing(nodeId)` for the status surface - `MeshNodeStatus.reverseCallbackStatus` discriminator (`connected | connecting | unavailable | not_applicable`) surfaced via `/api/mesh/status` and rendered as a pill in the Routing tab - `openCrossNode` error message distinguishes "no proxy target" from "waiting for central to dial the reverse bridge" - New tests: `mesh-service-proxy-tunnel-reconcile`, `mesh-status-reverse-callback`, `mesh-proxy-tunnel-dialer-no-idle-close` SENCHO_PRIMARY_URL is no longer required for any mesh function. * fix(mesh): rewrite proxy-tunnel reconcile test contents The previous commit renamed the file but the rewritten test bodies stayed unstaged on top of the rename. This commit lands the actual rewrite: the fanout assertion now requires every mesh-enabled proxy peer to be dialed, not just those with `mesh_stacks` rows, and adds a reconcile-tick repeated-call test. |
||
|
Anso
|
578cac89da |
fix(mesh): surface data-plane failures in health, meta, and Routing tab (#1088)
The three previously-silent console.warn paths in MeshService.setupMeshNetwork now route through a typed recordSetupFailure helper that classifies the failure (subnet_invalid, subnet_overlap, subnet_mismatch, ip_in_use, attach_failed, not_in_docker), emits a mesh.disable activity entry at the matching level (error for real failures, warn for the expected dev-mode not_in_docker case), and strips mesh_proxy_callback_bootstrap from advertised capabilities via CapabilityRegistry. /api/health gains a mesh.dataPlane block carrying the typed status. /api/mesh/status carries localDataPlane at the top level so the Routing tab renders a red banner with an operator-actionable recovery hint (set SENCHO_MESH_SUBNET to a free /24 and recreate the container) when the data plane is down. The success path re-enables the capability and flips the status to ok. Generalizes the previously-documented F-0 failure mode (IP-in-subnet collision) to also cover the subnet-pool-overlap case where another Docker bridge on the host already owns the requested CIDR. |
||
|
Anso
|
9aaa8573c0 |
fix(mesh): gate Trigger 2 re-bootstrap on actual api_token change (#1076)
The PUT /api/nodes/:id handler closed and re-dialed the mesh callback bridge on every save that included api_token in the body, even when the token was unchanged. The frontend always sends the full formData on Save, so renames and compose_dir edits against a mesh-enabled proxy remote produced a wasted closeBridge + ensureBridge round-trip and a spurious manager_rejected entry in the activity log. Gate the re-bootstrap on a real value diff against the persisted token. Adds an existing-node lookup (returns 404 on missing id, which the handler previously lacked) so the comparison has the pre-update value. Reorders the guards so resource-not-found beats payload validation. Adds one integration test for the same-token path; the existing three trigger-2 cases continue to assert close + ensure firing on a real rotation, no-token-in-payload, and mesh-disabled. |
||
|
Anso
|
cf618dd866 |
feat(mesh): symmetric WS dial for proxy-mode mesh peers (#1066)
* chore(mesh): foundation for symmetric callback dial Adds the data-plane scaffolding that the symmetric callback dial fix builds on: - mesh_centrals table for peer-side bootstrap material - MeshCentralRegistry service (upsert/getActive/clear/markUsed/markRejected) - PilotTunnelManager kind discriminator and replaceOrRegisterProxyBridge - mesh_proxy_callback_bootstrap capability registration - MeshProxyTunnelDialer reason-tagged proxy-bridge-down events from a single tearDownBridge emission point - Reactive redial scheduler that skips idle and auth_failed reasons * feat(mesh): add reverse-direction activity log entries (closes R1-B) acceptReverseLocal now emits route.resolve.ok with direction=reverse on connect ack and route.resolve.fail with direction=reverse plus reason=container_not_found / connect_error pre-connect. Post-connect close/error stays silent. Reuses existing event types via the new details.direction discriminator so frontend filters are unaffected. * feat(mesh): add peer-to-central callback dial path (closes R1-A2) Closes the architectural gap where proxy-mode mesh peers could not re-establish their tunnel to central after any non-idle bridge teardown (idle close, network blip, central restart, peer reboot). Central remains the hub for the data plane; the change is purely about WS initiation. Symmetric WS initiation, asymmetric protocol roles. Central retains PilotTunnelBridge ownership; peer retains TcpStreamSwitchboard + reverseDialer ownership. Central bootstraps callback credentials over the first authenticated central-initiated mesh tunnel via a one-shot mesh_handshake JSON frame; peer persists the material in a new mesh_centrals SQLite table and dials central's new /api/mesh/proxy-tunnel-from-peer endpoint when local cross-node traffic needs a bridge and none is live. Mesh_tunnel JWT (HS256, signed with auth_jwt_secret) carries scope, audience, issuer (central instance id), peer api_token fingerprint, kid. Validation on inbound peer dial: algorithm pin, signature, scope, audience, instance, time bounds, node existence and mode, fingerprint match. Failures return HTTP 401 with a machine-readable reason; peer routes the response per a clear-vs-keep cache matrix. Triggers proactive bootstrap on mesh-enable and api_token rotation; central startup fans out to mesh-enabled proxy-mode nodes with mesh_stacks rows (throttled, fire-and-forget). Reactive redial on non-idle bridge loss. Capability-gated handshake send (mesh_proxy_callback_bootstrap) makes the upgrade path safe against older peers in mixed-version fleets. Adds peer-side /api/system/pilot-tunnels centralCallback diag block, bounded counter metrics for bootstrap and dial events. SENCHO_PRIMARY_URL preflight warning when unset on a central with mesh-enabled proxy nodes. Tested with unit suites for the validation chain, registry, manager, and both dialers; integration tests for bootstrap E2E (asserts protocol-role invariant), api_token rotation, instance id change, version skew, and pilot-mode regression. * fix(mesh): green CI on the symmetric callback branch Two independent CI failures, both surgical: 1. Backend tests (11 fails): four mesh test files called setupTestDb in beforeEach. setupTestDb does not reset the DatabaseService singleton, so the per-test afterEach rm of the previous tmpdir left the singleton connection pointing at a deleted file. The next beforeEach's line-55 write threw SQLITE_READONLY_DBMOVED on Linux. Windows file-lock semantics hid this locally. Hoist setupTestDb / cleanupTestDb to file-scope beforeAll / afterAll; per-test state resets stay in beforeEach. Matches the convention in the eight mesh test files that already pass. 2. CodeQL (4 high alerts): js/insufficient-password-hash flagged sha256(api_token) at four sites. The api_token is a 256-bit opaque bearer (sen_sk_-prefixed), not a human password; sha256 is the correct fingerprint primitive for binding the mesh_tunnel JWT to a specific token. Add the two production files plus the two test files that mint the fingerprint to the existing path-scoped query-filter for that rule. * fix(mesh): drop unused afterEach import and revert dead codeql config ESLint flagged afterEach as unused in mesh-central-registry.test.ts:1 after the previous commit hoisted setup/teardown to file-scope beforeAll/afterAll. Remove from the vitest import line. Revert the codeql-config.yml additions from the previous commit. The paths: sub-key under query-filters > exclude is not a documented CodeQL feature and silently no-ops. The four js/insufficient-password-hash alerts on api_token fingerprinting are tracked as dismissed false positives in the GitHub Security tab rather than via dead config. |
||
|
Anso
|
d882f223f4 |
feat(api-tokens): switch to sen_sk_ prefixed opaque keys (#1062)
* feat(api-tokens): switch to sen_sk_ prefixed opaque keys
Replace JWT-shaped API tokens with 56-char opaque keys of the form
`sen_sk_<43-char base62 random><6-char base62 checksum>` (256-bit
entropy, sha256-truncated checksum). Node-proxy tokens stay JWTs.
Why:
* The api_token path was already a sha256 DB lookup; the JWT signature
was wasted work and the 400d JWT ceiling vs DB expires_at was a
confusing dual bound.
* Opaque tokens carry a verifiable checksum so malformed/typoed values
are rejected before any SQLite lookup.
* `sen_sk_` prefix is recognizable to GitHub, TruffleHog, GitGuardian
and makes the on-wire shape visually distinct from node_proxy JWTs.
Changes:
* New `utils/apiTokenFormat.ts` (generate + checksum-verify, CSPRNG via
randomInt, timingSafeEqual on the checksum compare).
* `middleware/auth.ts` and `websocket/upgradeHandler.ts` route opaque
tokens before any jwt.verify; 401 messages unified to avoid a
token-existence oracle.
* `middleware/rateLimiters.ts` short-circuits opaque tokens in the
node_proxy detection and keys per-token via a non-reversible sha256
slice so each token keeps its own bucket without a DB hit.
* All six existing tests migrated from jwt.sign({scope:'api_token'})
to generateApiToken(); new format-only test suite covering prefix,
length, alphabet, checksum reject paths, and a 10k-iteration
collision/integrity loop.
* Docs (features/api-tokens.mdx, api-reference/overview.mdx) describe
the shape and drop the obsolete JWT-ceiling note.
* fix(api-tokens): clear CI lint and CodeQL false positives
* Drop unused TEST_USERNAME import in remote-console-session.test.ts;
the migration to generateApiToken() left it orphaned.
* Add a CodeQL barrier model so `generateApiToken`'s ReturnValue does
not flow into the `insufficient-password-hash` query. The function
emits 256-bit CSPRNG opaque keys; sha256 of the raw token is the
correct construction for high-entropy API tokens (bcrypt-class
hashes target low-entropy human passwords). CodeQL's name heuristic
was treating "Token" as a password source and flagging the standard
sha256 wrapping at all 9 call sites.
* ci(codeql): exclude js/insufficient-password-hash for token paths
The previous barrierModel data extension was a no-op for this rule: the
js/insufficient-password-hash query identifies its "password" sources via
SensitiveExpr's name heuristic ("token", "secret", "key" substrings),
which is upstream of the taint-tracking layer where barrierModel applies.
Verified by post-push re-analysis: 9 alerts still open, all undismissed.
Replace the dead extension with a path-scoped query-filter in
codeql-config.yml so the rule no longer fires on apiTokenFormat,
apiTokens, and the test directory. Real user-password hashing code
elsewhere in the repo (auth, users, setup routes) remains analyzed.
The 9 existing alerts on PR #1062 are dismissed via API as false
positives with a justification pointing at this config. Future runs
will not re-flag them because of the path filter.
|
||
|
Anso
|
44e40afb62 |
fix(schedules): align Schedules surface with backend tier gate for Skipper admins (#1047)
The Schedules sidebar entry was hidden from Skipper admins even though the backend permits them to create and run update, scan, and snapshot schedules. The action picker also showed all 10 actions to every paid admin, so a Skipper selecting Restart, Prune, or any auto_* lifecycle action would 403 on submit. Changes: - Extract SKIPPER_SCHEDULED_ACTIONS as the single source of truth in tierGates.ts; both requireScheduledTaskTier and the GET /scheduled-tasks list filter now reference it (replaces a duplicate local constant in scheduledTasks.ts). - Move the Schedules nav entry from the Admiral block into the isPaid && isAdmin block in useViewNavigationState.ts, mirroring the existing Auto-Update pattern. Console and Audit stay Admiral-only. - Filter the create-form action picker in ScheduledOperationsView.tsx by license variant. Skipper sees Auto-update Stack, Auto-update All Stacks, Fleet Snapshot, and Vulnerability Scan; Admiral sees the full set. - openCreate now defaults formAction to the first visible option so Skipper starts with a valid choice instead of the Admiral-only Restart. Tests: - Add Skipper-variant POST coverage in scheduled-tasks-routes.test.ts: three allow cases (update / scan / snapshot) and a six-action rejection loop covering restart / prune / auto_backup / auto_stop / auto_down / auto_start. - Flip the Skipper assertion in useViewNavigationState.test.tsx to expect scheduled-ops alongside auto-updates. |
||
|
Anso
|
5461bc316b | fix: harden stack management operations (#1046) | ||
|
Anso
|
8dd0fce621 |
fix(fleet): show capabilities, version, metrics, and stacks for pilot-agent nodes (#1044)
When a pilot-agent node was the active node, the UI rendered "does not advertise this capability" across most tabs, a perpetual "Update available" badge, and a Fleet card body with blank CPU/RAM/Disk and "No stacks found". The cause was central-side aggregators in /api/fleet/* and /api/nodes/:id/meta only fanning out to proxy-mode remotes via node.api_url + node.api_token, which are null for pilot-agent. Route every affected aggregator through NodeRegistry.getProxyTarget so the loopback URL backed by the active pilot tunnel is used uniformly: - /api/nodes/:id/meta and /api/fleet/update-status fetch via the new NodeRegistry.fetchMetaForNode helper (resolves the target, delegates to fetchRemoteMeta, returns the shared OFFLINE_META on null). - fetchRemoteNodeOverview, /api/fleet/configuration, /api/fleet/node/:nodeId/stacks, and the stack-containers drilldown fetch through target.apiUrl with conditional Authorization. - fetchRemoteMeta omits the Authorization header when the token is empty (pilot-agent loopback) instead of sending a malformed Bearer string. - Pilot-agent rows preserve pilot_last_seen and mirror it into last_successful_contact so the Fleet "last seen" cell renders the recent tunnel timestamp during a brief reconnect. Pilot-mode capability filter excludes capabilities whose central-pilot path is not yet wired (host-console, self-update). Without this, the Console tab would surface for an Admiral pilot session and click through to central's host because the WS upgrade handler still gates on api_url + api_token. Filtered capabilities are removed at boot via applyPilotModeCapabilityFilter when SENCHO_MODE=pilot. Cache invalidation on tunnel-up: the meta cache for a reconnecting pilot is dropped so the next request rebuilds capabilities and version through the live bridge instead of waiting for the 3-minute TTL. The namespace constant moves to helpers/cacheInvalidation.ts alongside the new invalidateRemoteMetaCache helper. Husky commit-msg hook: add the missing shebang and a .gitattributes rule pinning .husky/* to LF line endings so commits do not fail with "Exec format error" on Windows shells where autocrlf=true converts the hook to CRLF. Tests cover Authorization-header behavior, pilot-mode filter idempotency, fetchMetaForNode dispatch (offline target, pilot-agent loopback, proxy-mode), and the four affected fleet routes for pilot-agent both when the tunnel is up and when it is down. |
||
|
Anso
|
e7a3b544c0 |
fix: harden auto-heal policies (#1042)
* fix: harden auto-heal policies * fix: resolve auto-heal lint failure |
||
|
Anso
|
b52323036b |
fix: harden stack label permissions (#1036)
* fix: harden stack label permissions * fix: avoid test db init from debug logging |
||
|
Anso
|
328a98439d | fix: harden vulnerability scan scheduling (#1035) | ||
|
Anso
|
c31d48b933 |
fix: harden git source webhooks (#1033)
* fix: harden git source webhooks * fix: make path validation visible to CodeQL static analysis Add explicit isValidStackName guard in getEnvContent, isValidGitSourcePath pre-validation in readRepoFile, and URL hostname check in remoteStackRequest to satisfy CodeQL taint-tracking so the pipeline passes. * fix: use path.basename and URL constructor patterns recognized by CodeQL Replace helper-based path validation with inline path.basename and path.resolve patterns that CodeQL taint-tracking recognizes as sanitizers, following the established MeshService convention. Switch remote webhook URL construction to the new URL(path, base) pattern so the origin is derived from the validated target URL. * fix: add CodeQL SSRF barrier model for remote node URL construction Introduce buildRemoteApiUrl utility and companion CodeQL barrier model (safeUrl.model.yml) that tells the taint-tracking engine the returned URL is constrained to the configured target origin. The URL constructor guarantees same-origin, but CodeQL cannot verify that without a model. * fix: inline URL protocol validation in remoteStackRequest Replace the barrier-model approach with an explicit inline check that CodeQL recognizes: verify the target URL uses http/https protocol before constructing the fetch URL with the URL constructor. * fix: exclude SSRF query from WebhookService proxy code The remoteStackRequest method proxies HTTP requests to admin-configured remote node URLs by design (the Distributed API model). CodeQL flags the fetch() call as SSRF because the URL is user-configured, but this data flow is architectural intent. Exclude js/server-side-request-forgery from this file. * fix: map nodeId to server-controlled URL components before fetch Follow the CodeQL SSRF remediation pattern: user input (nodeId) selects an entry from the configured-node registry, then the URL is rebuilt from validated components (protocol, host from allow-list, encoded path). Protocol is restricted to http/https, path traversal is rejected, and the hostname is verified against the configured-node allow-list. * fix: remove unnecessary escape in endpoint validation regex |
||
|
Anso
|
b1c5fe8391 |
fix: harden deploy enforcement paths (#1030)
* fix: harden deploy enforcement paths * fix: update Docker toolchain to Go 1.26.3 * fix: repair Dockerfile tr argument split across lines * fix: bump protobufjs to clear npm audit high-severity advisories * fix(test): add execFile to child_process mock in compose-images test * fix: resolve merge conflicts with main * fix: resolve merge conflicts with main * fix: resolve merge conflicts with main |
||
|
Anso
|
74ae2ce0c6 |
fix: harden atomic deployment rollback (#1029)
* fix: harden atomic deployment rollback * fix: update Docker toolchain to Go 1.26.3 * fix: repair Dockerfile tr argument split across lines * fix: bump protobufjs to clear npm audit high-severity advisories * fix: sanitize error objects in console.error to prevent log injection |
||
|
Anso
|
69b6ac1f3b |
fix: harden stack file explorer operations (#1028)
* fix: harden stack file explorer operations * fix: update Docker toolchain to Go 1.26.3 * fix: repair Dockerfile tr argument split across lines * fix: bump protobufjs to clear npm audit high-severity advisories |
||
|
Anso
|
19cdb3681d |
fix: harden blueprint deployment guardrails (#1027)
* fix: harden blueprint deployment guardrails * fix: update Docker toolchain to Go 1.26.3 * fix: repair Dockerfile tr argument split across lines * fix: bump protobufjs to clear npm audit high-severity advisories |
||
|
Anso
|
b8af40d2b4 |
fix(mesh): enumerate stacks on remote pilot nodes for opt-in sheet (#1025)
The mesh opt-in sheet showed "No stacks deployed on this node yet" for every remote pilot because GET /api/mesh/nodes/:nodeId/stacks called FileSystemService.getInstance(nodeId).getStacks() unconditionally, which always reads central's own filesystem regardless of which node the operator targeted. Fix dispatches through the proxy chain when the targeted node is remote, mirroring the C-3 pattern that already governs every other mesh endpoint needing data from a remote Sencho: - New endpoint GET /api/mesh/local-stacks (Admiral-gated) returns the bare stacks list from THIS Sencho's own filesystem. Mirrors the precedent set by /api/mesh/local-services/:stackName. - New MeshService.listLocalStacks() and MeshService.listStacksOnNode() helpers; the latter dispatches local vs remote and degrades to an empty list on transport failure (no proxy target, non-2xx response, malformed body). - Refactored route delegates the stack-name list to listStacksOnNode; central's mesh_stacks DB is still authoritative for the opt-in flag set per C-3. Tests cover the local path, the remote-OK path with header forwarding, non-2xx, no-target (pilot tunnel down), malformed bodies, and unknown node ids. |
||
|
Anso
|
6ac7da978c |
fix(mesh): push alias port map to pilot so reverse-direction forwarder binds listeners (#1021)
* fix(mesh): apply D-1 pushed override on pilot deploys via file-presence fallback When isMeshStackEnabled returns false, ensureStackOverride now checks for an override file already on disk before returning null. On pilot nodes the mesh_stacks table is intentionally empty (opt-in state lives on central per the C-3 design), so the DB gate blocked the override that central had already pushed via applyLocalOverride. File-presence is safe as the fallback because removeOverrideFromNode sends a DELETE to the pilot when a stack is opted out, so a stale file cannot survive past opt-out. Also aligns removeStackOverride to use path.basename consistently with all other override-path construction sites in the same file. Adds two tests: pilot node with a pushed file returns the path; pilot node with no file returns null. * fix(mesh): push alias port map to pilot so reverse-direction forwarder binds listeners Pilots have no mesh_stacks rows by design (C-3), so refreshAliasCache() produced an empty aliasByPort map and syncForwarderListeners() bound zero ports. Reverse-direction TCP probes (pilot prober -> central alias) failed with connection refused at the OS level because no listener existed. Fix: extend the D-1 override push payload to include the full MeshGlobalAlias records (host + port + routing metadata). The pilot stores these in a new pilotAliasOverlay map keyed by stack name. refreshAliasCache() merges the overlay after the (empty) DB loop, populating aliasByPort correctly. syncForwarderListeners() then binds the alias ports immediately after the push completes. removeLocalOverride() clears the overlay entry and re-syncs on opt-out. The two populations (DB rows on central, pilotAliasOverlay on pilots) are mutually exclusive by the C-3 invariant, so the first-write-wins port collision policy in refreshAliasCache is safe. |
||
|
Anso
|
7cca68f84a |
fix(mesh): operator-triggered override regen and boot lifecycle hardening (#1016)
Audit-driven follow-up to the F6+F7 fix. Closes the High-severity findings
from the mesh-dial-path audit without expanding into the architectural
follow-ups (JWT TTL, TLS enforcement, per-stack JWT scope) which need
their own design discussions.
Changes:
1. New `POST /api/mesh/regen-overrides` Admiral-gated endpoint that calls
`MeshService.regenerateAllOverrides()` and returns a structured
`MeshRegenSummary { regenerated, failures, skipped, reason? }`. Lets
the operator rerun boot regen after fixing a remote node that was
offline at central startup, instead of opt-out + opt-in for every
meshed stack on that node. Audit-log row carries the outcome
(success / partial / skipped / error).
2. `regenerateAllOverrides` now aggregates per-stack outcomes into a
single summary log row (`mesh override regen complete: N succeeded,
M failed across K node(s)`) with `failedNodeIds` in `details`. Per-
stack warnings still emitted. When skipped because `senchoIp` is
null, emits an explicit warn instead of silently no-oping.
3. `MeshService.start()` now wraps `refreshAliasCache` and
`syncForwarderListeners` in their own try/catch so a throw from
either step is logged and the boot continues to override regen.
The closing log line surfaces data-plane state ("MeshService started
(data plane ok)" or "(data plane unavailable (<reason>))") so an
operator grepping startup output sees a half-init mesh immediately.
4. `sanitizeForLog` wrap on four previously unsanitized `err.message`
writes (cross-node tcpStream error handler + three probe error
paths). Container IPs and local socket details no longer leak into
the activity buffer or the probe response body.
5. Updated three existing F6 regression tests for the new return shape
and aggregated summary message. Added two new tests: version-skew
404-push handling, and concurrent opt-in during regen.
|
||
|
Anso
|
693f9b4495 |
fix(fleet): drop tier gate from stack and container list endpoints (#1012)
Community tier opened the Fleet Overview drilldown (node card → stack list → container list) but two read-only metadata endpoints in fleet.ts kept their requirePaid gate. Stack names still rendered because they ride on the un-gated /api/fleet/overview payload, but expanding a stack always re-fetched containers and hit the orphan gate, surfacing as a "Failed to load containers for X" toast on every node card on Community. Drop requirePaid from GET /node/:nodeId/stacks and GET /node/:nodeId/stacks/:stackName/containers. Both handlers stay behind authMiddleware, validate inputs, and proxy to the per-node api_token for remote nodes. Paid mutations (bulk update, scheduled snapshots, label bulk actions, Fleet Actions cards) keep their gates in fleetActions.ts. Add positive Community-tier tests in fleet.test.ts and clear the stale "Requires Skipper or Admiral license" line from the OpenAPI descriptions. |
||
|
Anso
|
23bbee4f45 |
feat(mesh): replace host-mode with shared sencho_mesh Docker network (#1009)
* feat(mesh): replace host-mode with shared sencho_mesh Docker network Phase D of the mesh redesign: drop the operator's `network_mode: host` requirement and the `host-gateway` extra_hosts pattern that did not work on cloud iptables-restrictive distros (OCI, etc.) or Docker Desktop. Each Sencho creates a shared `sencho_mesh` Docker bridge network on boot (default subnet 172.30.0.0/24, override via SENCHO_MESH_SUBNET), pins itself at `<network>+2`, and attaches every meshed user service to the same bridge. Compose overrides now emit IP-based `extra_hosts` plus a top-level `networks` block declaring `sencho_mesh` external. Override delivery: central renders for local stacks; for remote stacks it sends the fleet alias list to the remote's new `PUT /api/mesh/local- override/:stackName` endpoint, which renders against the remote's OWN local senchoIp and writes under its OWN DATA_DIR. Each node may use a different subnet without coordination beyond the env var. Opt-in / opt-out now trigger an automatic redeploy of the affected stack via the existing deploy code path (local: ComposeService; remote: HTTP POST through proxyFetch). The frontend opt-in sheet shows a confirmation modal (ConfirmModal) before the mutation. Failed redeploys emit both a mesh activity event and a durable audit-log row. Hardening: - Reserve port 1852 at opt-in (prevents user containers from racing the Sencho API listener). - ensureMeshNetwork refuses to continue if `sencho_mesh` exists with a mismatched subnet rather than silently routing to the wrong IP. - Idempotent network connect/disconnect helpers in DockerController. - optInStack rolls back the DB row if the just-inserted stack's override push fails (no half-states surviving across calls). - regenerateOverridesForNode runs in parallel and skips the just- pushed stack on opt-in. Operator template: drop `network_mode: host`, restore `ports: ["1852:1852"]`. Mesh now works identically on Linux LAN, OCI, and Docker Desktop without firewall changes. Docs: rewrite docs/features/sencho-mesh.mdx around the shared bridge network, document SENCHO_MESH_SUBNET, surface the host-network-service opt-in restriction, and cross-link with the Pilot Agent docs. BREAKING CHANGE: the operator's `docker-compose.yml` no longer uses `network_mode: host`. After upgrading, redeploy any meshed stacks once so they pick up the new IP-based override and join `sencho_mesh`. * fix(mesh): wrap stackName with path.basename in local-override fs ops CodeQL flagged js/path-injection on the new applyLocalOverride and removeLocalOverride methods because they are publicly reachable and its data-flow model does not recognize isValidStackName / isPathWithinBase as sanitizers. The validation IS sufficient (the allowlist regex blocks path separators, the path-prefix check blocks escape), but path.basename is a model CodeQL recognizes and is purely defensive: for any input that already passes isValidStackName, basename is the identity. |
||
|
Anso
|
f599110386 |
feat(mesh): collapse sidecar into Sencho process via in-process forwarder (#1000)
The separate saelix/sencho-mesh sidecar container is gone. The
forwarder logic that previously lived in mesh-sidecar/src/forwarder.ts
moves into the Sencho process as backend/src/services/MeshForwarder.ts,
a thin per-port net.Server lifecycle wrapper. MeshService implements
the host interface and owns resolve plus splice; MeshForwarder owns
listener boilerplate. One container per node, no separate image to
publish, no control WebSocket.
Operator-facing change: the Sencho container now runs in
network_mode: host so the forwarder can bind alias ports on the host
network where meshed containers' extra_hosts host-gateway entries
point. Without host network mode the listeners would land in the
container's namespace and inbound traffic from peers would never
reach them. The 1852:1852 port publish becomes a no-op under host
mode and is commented out in the operator template.
Same-node forward path now dials the target container's bridge IP
via Dockerode (preferring the compose default network for
deterministic selection across daemon versions) instead of 127.0.0.1.
The legacy 127.0.0.1 path only worked when the target service
published its port to the host; the IP path works regardless.
Cross-node mesh routing in this phase is central -> pilot direction
only via PilotTunnelManager.openTcpStream. Pilot -> central and
pilot <-> pilot via central relay land in Phase B with the
tcp_open_reverse frame.
Deletions:
- mesh-sidecar/ package entirely (Dockerfile, package, sources, tests)
- backend/src/websocket/meshControl.ts
- MeshService sidecar lifecycle: spawnSidecar, stopSidecar,
isSidecarRunning, mintSidecarToken, verifySidecarToken,
attachSidecarSocket, handleSidecarResolve, sendSidecar
- POST /api/mesh/nodes/:id/sidecar/restart route
- /api/mesh/control WS dispatch in upgradeHandler
Type cleanup: 'sidecar' literal removed from MeshActivitySource and
MeshProbeResult.where (also the frontend mirror). MeshNodeStatus
sidecarRunning becomes localForwarderListening (boolean | null) so
non-local nodes get a null instead of an unconditional false; the
honest semantic is "this view only knows the local forwarder state;
remote forwarder status lands in Phase B." MeshNodeDiagnostic
sidecar object becomes forwarder { listening, listenerCount }.
Frontend MeshDiagnosticsSheet drops the restart-sidecar action and
sidecar liveness card; surfaces forwarder state plus a "runs
in-process; no separate container" caption.
Resolves audit findings C-1 (data plane non-functional), C-2 (sidecar
control WS not loopback-enforced), C-4 (sidecar lifecycle Dockerode-
on-remote, PR #999 closed), and C-5 (saelix/sencho-mesh:latest
unreachable). C-3 (PR #992) is unchanged. M-12 (PR #994) is
unchanged.
|
||
|
Anso
|
1803512f70 |
feat: drop stack labels and network topology to Community tier (#995)
* feat(labels): drop tier gate to Community for organization endpoints Stack Labels CRUD and per-stack assignment are now Community-tier features: list, create, update, delete labels, and assign labels to a single stack. The two automation surfaces stay Skipper+: per-label bulk deploy / stop / restart (POST /api/labels/:id/action) and the Fleet Actions tab's bulk-assign card (POST /api/fleet-actions/labels/bulk-assign). Tier story is now organize free, automate paid. Add a route-level test that proves the CRUD endpoints succeed on a Community license while the bulk-action endpoint still returns 403. Update overview, licensing, and stack-labels docs to reflect the new tier placement and to note that bulk actions on a label still require Skipper or Admiral. * feat(topology): drop tier gate to Community for network topology view The Resources tab's Networks > Topology view is now available on every tier. Drops requirePaid from GET /api/system/networks/topology, removes the isPaid wrapper around the List | Topology toggle in ResourcesView, and removes the PaidGate around the topology graph. CapabilityGate stays in place so a node running on a build without the network-topology capability still renders its lock card instead of the graph. Add a route-level test that proves the endpoint returns 200 on a Community license. Update licensing and resources docs to reflect the new tier placement. * fix(labels): expose Settings > Labels tab on Community tier The settings registry entry for the Labels tab still carried tier: 'skipper', which kept the tab hidden in the Settings sidebar even though the underlying CRUD endpoints now serve Community. Drop the tier flag so Community users can discover and reach the section that backs the already-Community-tier label endpoints. |
||
|
Anso
|
e176ae9f17 |
fix(mesh): inspect remote stacks via HTTP proxy not Dockerode (#992)
MeshService.inspectStackServices used DockerController.getInstance(nodeId)
to enumerate compose-labeled containers, but NodeRegistry.getDocker
explicitly throws for any node with type='remote' by design. The throw
was silently caught and an empty array returned, so opt-in for any
remote node failed with a misleading "no running services" error and
refreshAliasCache only ever populated LOCAL aliases. Cross-node mesh
routing was therefore impossible end-to-end regardless of pilot-tunnel
state.
Split the inspector. inspectLocalStackServices keeps the Dockerode
listContainers path and always queries the local Docker daemon; it is
public so the new route can call it. inspectStackServices is now a
dispatcher: local nodes fall through to the Dockerode path, remote
nodes (proxy mode and pilot-agent) HTTP-fetch
/api/mesh/local-services/:stackName against the URL resolved by
NodeRegistry.getProxyTarget, with the persisted node_proxy Bearer
token and license tier headers attached. The remote's MeshService
enumerates its own LOCAL Docker daemon and returns the
{service, ports[]} envelope.
refreshAliasCache now inspects every opted-in stack in parallel via
Promise.allSettled so a slow or unreachable remote does not stall the
60-second loop. The new /api/mesh/local-services/:stackName route is
gated by requireAdmiral plus isValidStackName and always queries the
caller's own Sencho instance.
Together with PR #989 (proxy-side bridge dispatch) and PR #990
(agent-side loopback auth), this makes the central, bridge, agent,
local-Sencho mesh control plane functional end-to-end for both
proxy-mode and pilot-agent remotes.
|
||
|
Anso
|
3112f58a88 |
fix(templates): align App Store deploy gate with stack:create permission (#986)
The POST /api/templates/deploy handler gated on requireAdmin, while every
other create-stack endpoint in routes/stacks.ts uses
requirePermission('stack:create'). Per the role table in
middleware/permissions.ts, node-admin holds stack:create — so a node-admin
could create stacks the regular way but got 403 ADMIN_REQUIRED from the
App Store. The cockpit's Deploy button is gated on can('stack:create'),
so the button looked enabled and the click silently failed.
Swap the gate to requirePermission(req, res, 'stack:create'). Admin still
passes through the global bypass; node-admin now passes via the role
permissions table; deployer, viewer, and auditor stay denied. Cache-refresh
on the same router keeps requireAdmin since cache invalidation has no
per-resource scope.
Adds backend/src/__tests__/templates-deploy-rbac.test.ts with six
parameterised supertest cases (one per role plus an unauthenticated
case) so the matrix is locked in. The two passing-role cases assert
the request clears the gate (status !== 403, code !== PERMISSION_DENIED)
without depending on Docker being available in the test environment.
|
||
|
Anso
|
3d9489648e |
fix(pilot): harden outbound reverse-tunnel against resource exhaustion (#979)
* fix(pilot): cap tunnel frame size, concurrent streams, and stream idle time
Pilot tunnels carry every HTTP, WS, and Mesh-TCP byte for a remote node
through a single multiplexed WebSocket. A buggy or compromised peer that
sent oversized frames, opened streams in a tight loop, or left streams
parked indefinitely could exhaust gateway memory.
Three protocol-level limits applied symmetrically on both ends:
- MAX_FRAME_SIZE_BYTES (8 MB): set as the ws maxPayload on the
gateway-side WebSocketServer and the agent-side WebSocket client,
plus a defense-in-depth length check in decodeBinaryFrame and
decodeJsonFrame.
- MAX_STREAMS_PER_TUNNEL (1024): bridge refuses new loopback HTTP /
upgrade / TCP allocations with 503 once at the cap; the agent
rejects new incoming http_req / ws_open / tcp_open with the
appropriate error frame.
- STREAM_IDLE_TIMEOUT_MS (10 min): every stream gets a per-stream
timer refreshed on each inbound or outbound activity. Expiry tears
down the local half and notifies the peer.
The constants are colocated in pilot/protocol.ts so any future agent
build picks them up via the protocol module.
* fix(pilot): rate-limit pilot enrollment endpoints to 10 per minute
Pilot enrollment mints a JWT and writes a pilot_enrollments row, both
privileged operations that should not share the global API budget
(200/min). Add a dedicated express-rate-limit instance keyed by user or
IP at 10/min in production (100/min in dev so the local enrollment
test loop is not throttled).
The limiter is applied directly on POST /api/nodes/:id/pilot/enroll.
On POST /api/nodes the limiter's skip function reads the parsed body
and exempts proxy-mode creates; only requests that resolve to
mode=pilot_agent count against the enrollment budget.
* fix(pilot): cap concurrent pilot tunnels system-wide
The PilotTunnelManager held an unbounded Map of bridges. A reconnect
storm, runaway enrollment, or operator misconfiguration could grow the
map without limit, even though every tunnel still carries a valid JWT.
Cap at 256 concurrent tunnels per primary instance:
- Soft warning at 128 (logged once per crossing).
- Hard refusal at 256: registerTunnel throws PilotTunnelCapacityError
and the upgrade handler closes the WebSocket with 1013 (Try Again
Later) so the agent backs off rather than tight-looping.
A node that already has a registered tunnel does not consume a new
slot when it reconnects; the existing bridge is closed first.
* fix(pilot): release backpressure as soon as the tunnel buffer drains
Previously the bridge paused HTTP request bodies when the tunnel
WebSocket's bufferedAmount climbed above 4 MB but never resumed them
explicitly; the next data event re-checked the threshold, and TCP
streams only saw a 'drain' fan-out on the 30 s ping cycle. Slow-consumer
peers held buffered bytes for tens of seconds longer than needed.
Replace both with an on-demand drain check: when at least one stream
is paused, sample bufferedAmount every 100 ms; once it drops below the
high-water mark, resume every paused request and emit 'drain' to every
accepted TCP stream, then stop the timer. Dormant when no stream is
paused, so steady-state cost is zero.
* fix(pilot): trust internal CAs via SENCHO_PILOT_CA_FILE
Self-hosted deployments often terminate TLS with an internal CA. The
only previous escape hatch was NODE_TLS_REJECT_UNAUTHORIZED=0, which
disables verification across every outbound connection in the agent
process and is the wrong shape of fix.
Add SENCHO_PILOT_CA_FILE: when set, the agent reads the file as a PEM
bundle and passes it to ws as the `ca` option for the tunnel
WebSocket. rejectUnauthorized remains true. Failure to read the file
exits with a clear error so the operator does not silently fall back
to the default trust store.
There is no flag to disable TLS verification entirely; that would
defeat the credential trust model.
* fix(pilot): gate frame-rate-adjacent logs behind developer_mode
The agent's malformed-frame warning fires per inbound frame and could
flood logs under attack or against a buggy primary. The bridge swallowed
parse errors silently, leaving operators blind to protocol drift, and
the manager had no signal at all on tunnel registration.
Route per-frame and per-tunnel diagnostic logs through the existing
isDebugEnabled() helper (same pattern used by middleware/auth, RBAC,
and websocket/logs). Production stays quiet by default; the toggle is
the existing developer_mode setting in global_settings — no new env
var, no new dependency.
* feat(pilot): expose per-tunnel metrics via /api/system/pilot-tunnels
Operator support had no signal beyond raw logs to answer "is this
tunnel flapping?" or "is one bad node hiding behind the aggregate?".
Add an in-process counter set covering tunnels_total, tunnels_replaced,
tunnels_rejected_capacity, enroll_acks, frame_decode_errors, plus a
per-node array carrying the connectedAt and bufferedAmount so a single
tunnel sitting on a stuck buffer remains visible.
Counters live in services/PilotMetrics.ts; the gateway has no shared
in-process metrics facility today, so this is a per-feature pattern
documented as such for future consolidation. Counters are strictly
process-local — no telemetry, no export, no phone-home — consistent
with Sencho's privacy posture.
Surfaced read-only via GET /api/system/pilot-tunnels behind admin auth,
mirroring the existing cache-stats endpoint.
* fix(pilot): tighten reconnect backoff and log handling
Three small follow-ups to the hardening pass:
- Sanitize the primary URL on the agent's connect log so a malicious
SENCHO_PRIMARY_URL with embedded control characters cannot inject
fake log lines.
- Move the reconnect-backoff reset from the 'open' callback to the
'hello' frame handler. A peer that always rejects the handshake
(incompatible version, consumed enrollment token) used to reset
the backoff on every TCP-level connect and tight-loop reconnects;
now the reset waits for a clean protocol round-trip.
- Document the StreamIdAllocator wrap behavior so future readers
can see the relationship between the 2^31 wrap and the
MAX_STREAMS_PER_TUNNEL cap without re-deriving it.
* test(pilot): add coverage for enrollment, rate limiter, and tunnel caps
Adds two test files exercising the hardening pass surfaces that were
previously uncovered:
- pilot-enrollment.test.ts: end-to-end through POST /api/nodes
(pilot mode), POST /api/nodes/:id/pilot/enroll, the SHA256
token-hash persistence, replay protection, expired enrollments,
and the rate-limit header wiring (enrollment limiter on pilot
paths, global limiter elsewhere).
- pilot-bridge-limits.test.ts: oversize-frame rejection at both
binary and JSON decoders, and the per-tunnel concurrent stream
cap as observed via openTcpStream returning null and the loopback
HTTP server returning 503 once the bridge is at capacity.
Bridge cap test fills the slot map with TCP stream handles rather than
real HTTP requests so the OS socket pool stays out of the picture.
* test(pilot): cover manager metrics snapshot and capacity-error shape
Adds direct coverage for:
- PilotTunnelCapacityError exposing the limit so the upgrade
handler can format a useful close-frame reason.
- PilotMetrics.snapshot returning a defensive copy (callers must not
mutate the live counter set).
- getMetricsSnapshot returning the open count, per-node breakdown,
and counter set in a stable shape, with tunnels_total bumping on
a successful registerTunnel.
The manager test seeds a real pilot-mode node row first because
updateNode throws on missing rows; this models the production path
where a node is created before its enrollment is consumed.
* docs(pilot): document tunnel limits, custom CA, and new failure modes
Refresh the Pilot Agent feature doc to cover the hardening surfaces:
- New 'Self-signed primary TLS certs' section walking through the
SENCHO_PILOT_CA_FILE env var with an example docker run command.
- New 'Resource limits' section listing the per-tunnel and
system-wide ceilings so operators know what the wire enforces.
- Four new troubleshooting entries: WebSocket close 1013 (system
cap), loopback 503 (stream cap), close 1002 with protocol error
(frame size / malformed), and HTTP 429 on enrollment.
Also expand .env.example with a new pilot-agent block covering
SENCHO_MODE, SENCHO_PRIMARY_URL, SENCHO_ENROLL_TOKEN, and
SENCHO_PILOT_CA_FILE so operators do not have to read code or the
feature doc to discover the agent-side config surface.
* fix(pilot): address code-review findings on the hardening pass
Critical:
- Bridge: TCP backpressure now starts the drain timer and tracks
streams awaiting drain, so a TcpStream caller waiting on 'drain'
no longer hangs when no HTTP request is in flight.
- Bridge: every direct streams.delete call now goes through the
removeStream helper so per-stream idle timers are cleared
consistently and pausedReqs / tcpAwaitingDrain stay aligned.
- Bridge close(): resume any paused IncomingMessage before clearing
the map so a parser does not stay stuck across teardown.
High:
- Protocol: decodeJsonFrame now compares Buffer.byteLength(raw,
'utf8') against the cap; raw.length (UTF-16 code units) let
multi-byte payloads sneak ~3x the byte budget through.
- Agent: the optional CA bundle is read once at construction and
cached on the instance, so a missing or rotated SENCHO_PILOT_CA_FILE
no longer process-exits on every reconnect attempt.
Medium / Low:
- Bridge limits test: assert mockWs.sent.length to confirm slot
allocations actually serialized a tcp_open frame and that a
rejected (cap+1) attempt did not consume a stream id.
- Manager: defer the tunnels_replaced increment until after the
cap check passes, so a rejected reconnect does not double-count
as both replacement and capacity rejection.
- Manager: clarify the protocol comment about MAX_FRAME_SIZE_BYTES
enforcement layering (ws maxPayload is authoritative; decoder
check is for tests and defense-in-depth).
- Limiter: defensive `if (!req.body) return false` so a future
refactor that delays body parsing cannot silently skip the
enrollment limiter.
Deferred (documented as follow-ups, not in this branch):
- M2: narrowing public surface of PilotTunnelManager.getBridge to a
MeshTunnelHandle interface (cross-cutting refactor).
- H4: end-to-end replay test driving the upgrade handler twice
with the same enrollment JWT (needs WS test harness).
|
||
|
Anso
|
3b650523c1 |
Audit-hardening pass for secret and misconfiguration scanning (#977)
* fix(security): dedupe concurrent compose-stack scans
Track stack scans in scanningImages keyed stack:<nodeId>:<stackName>.
The /scan/stack route returns 409 when an in-flight scan exists, and
the service-side check is the real correctness barrier (the route
pre-check is a fast-path optimization that mirrors scanImage). The
dedup key release lives in a try/finally so failed scans free the
slot for retry.
Why: scanComposeStack had no equivalent of scanImage's scanningImages
guard, so two simultaneous calls for the same stack would both run
trivy config, both insert a vulnerability_scans row, and double-
process the result.
* feat(security): acknowledge misconfig findings
Adds a parallel acknowledgement system for Trivy misconfig findings
that mirrors cve_suppressions: a new misconfig_acknowledgements table,
read-time enrichment via the new misconfig-ack-filter utility, REST
CRUD endpoints, fleet-sync replication from control to replicas, a
Settings panel, and an Acknowledge button on the Misconfigs tab.
Schema and behavior parity with cve_suppressions:
- UNIQUE(rule_id, COALESCE(stack_pattern, '')) so fleet-wide acks
collide as expected
- blockIfReplica on every write
- Audit-log entries name the scope (rule_id, stack_pattern) but
never the reason text
- replicated_from_control flag controls UI delete affordance and
drives clearReplicatedRows on demote/reanchor
- Validators reused: validateStackPatternForRedos for glob safety,
sanitizeForLog for log fragments
SARIF export emits an external/accepted suppression entry per
acknowledged misconfig, matching the CVE pattern.
Per-row Acknowledge dialog prefills stack_pattern with the scan's
stack_context so the default scope is "rule + this stack only" and an
operator must broaden explicitly.
Tests: misconfig-ack-filter (15) and misconfig-ack-routes (23)
including the duplicate-409 case for both pinned and fleet-wide acks.
* fix(security): reap orphaned trivy tmp dirs at startup
When the buildEnv path writes a per-scan DOCKER_CONFIG dir under
os.tmpdir() and the process crashes before the finally block runs,
the dir leaks. Mirrors GitSourceService.sweepStaleTempDirs:
exported sweepStaleTrivyTempDirs is fire-and-forget at boot,
removes prefix-matching dirs older than 1 hour, swallows
permission/race failures, logs a single line if any were reaped.
* perf(security): emit per-batch summary for scanAllNodeImages
Adds one diag() line at the end of scanAllNodeImages summarising
unique image count, scanned, skipped, failed, violation count, and
elapsed time. Per-image diag inside scanImage stays useful for
debugging individual scans; the summary gives operators a single
fleet-level checkpoint when developer_mode is on.
* perf(security): cap SARIF export at 5000 findings per type
Replace the unbounded fetchAllPages walk on /scans/:id/sarif with a
hard limit of 5000 findings per type. When any type trips the cap,
emit run-level properties.truncated=true plus row_limit and per-type
totals so downstream tooling can flag the export as partial.
Console-warns for ops visibility.
A scan with 50k vulns previously streamed every row into memory
before serialising; the cap bounds memory and serialisation time at
the cost of completeness on pathological scans.
* docs(env): document TRIVY_BIN host-binary override
The env var is honored by TrivyService.detectTrivy as a fallback when
no managed install is present, but it was undocumented in
.env.example. Adds the var with a comment explaining precedence
(managed > TRIVY_BIN > PATH).
* test(security): cover scanComposeStack failure modes
Two new cases drive the existing try/catch through real failure
paths:
- Malformed Trivy stdout: row flips to status='failed' with the
parser error preserved on `error`.
- execFile rejection: row flips to status='failed' with a string
error message.
Pairs with the existing dedup tests so the failure path now also
verifies the scan row state, not just the thrown exception.
* test(e2e): security scanner + misconfig acknowledgement flow
Seven Playwright tests covering the scanner UI and the new
acknowledgement system end-to-end:
- Trivy availability gate (skips suite when binary absent so CI
without Trivy can opt out via E2E_SKIP_TRIVY=1)
- Stack config scan completes and records misconfig findings
- Concurrent stack scan returns 409 from the dedup gate
- Misconfig ack POST creates and lists on Settings
- Duplicate (rule_id, stack_pattern) returns 409
- Malformed rule_id (shell metacharacters) returns 400
- Misconfigs tab renders against a real stack scan
Tests drive the API for behaviour assertions and the UI only for
shell-rendering checks; the visual snapshot suite owns screenshots.
* docs(features): add misconfig acknowledgement workflow and SARIF cap
Refreshes vulnerability-scanning.mdx with:
- Misconfig acknowledgements section covering the per-row dialog,
Settings panel, scope/matching rules, and SARIF emission
- Tier table row for the new feature
- SARIF section note on the 5000 row-per-type cap and the
properties.truncated marker for partial exports
- Troubleshooting entries: SARIF cap, hidden Acknowledge button,
findings resurfacing after delete, Trivy DB phone-home, and
409 on concurrent compose-stack scans
* fix(ci): clear backend lint and CodeQL alerts
- Remove the dead fetchAllPages helper in routes/security.ts. It lost
its callers when the SARIF endpoint switched to direct paged reads
for the truncation cap. ESLint flagged it as unused.
- Switch the trivy-tmp-cleanup test helper to fs.mkdtempSync. Building
paths under os.tmpdir() with predictable names tripped CodeQL's
js/insecure-temporary-file rule (high severity), which warns about
symlink-pre-creation attacks even in test code. mkdtempSync appends
a process-random suffix and creates the dir atomically; the
sencho-trivy- prefix is preserved so the production sweep still
matches the test fixtures.
|
||
|
Anso
|
4b1de35dda |
Audit-hardening pass for fleet-replicated CVE suppressions (#976)
* perf(security): bucket CVE suppressions by id at read time
applySuppressions now builds a Map<cve_id, suppression[]> once before the
per-finding loop, dropping per-finding work to O(matching-cve-suppressions)
rather than O(all-suppressions). At a fleet-wide cap of 10000 rows against
a multi-thousand-finding scan, the prior linear-per-finding shape drifted
into tens of millions of comparisons per render.
Public API of findSuppression and applySuppressions is unchanged.
Specificity scoring, expiry handling, and image-glob matching are
preserved bit-for-bit. Adds a regression guard that pins a 10000x2000
workload under 1.5s and asserts at least one match was actually returned.
* feat(security): record audit-log entries on control-side CVE suppression CRUD
The replica receive path already wrote an audit-log entry on apply; the
control-side POST/PUT/DELETE handlers did not. Operators reading the
audit panel saw mirrored security-rule changes from the replica view but
could not see who originated them on the control. Symmetric logging
closes that gap.
The summary records the CVE id and pinned scope (pkg, image) but never
the suppression's reason text. Reasons are free-form admin input that
replicate fleet-wide and may carry incident-tracker IDs or vendor
context the operator did not intend to broadcast.
The summary is also sanitised before emission so an operator-supplied
package name or image pattern carrying a smuggled newline plus a forged
"cve_suppression.delete:" prefix cannot inject a fake row into the audit
panel. Control characters become "?" and the field is capped to its
validator length.
Adds a Control-side audit log block to suppression-routes.test.ts that
asserts the privacy contract on each verb (scope present, reason absent),
plus a log-injection guard test, plus a regression that GET still works
when the local instance is a replica.
* test(security): cover cve_suppressions fleet-sync receive path
The existing fleet-sync route tests covered the protocol mechanics
(auth, anchor, stale push, reanchor, demote) for cve_suppressions only
with empty-rows payloads. The suppression-specific concerns were
unverified end-to-end:
- actual rows replace prior replicated rows on a fresh push
- the receive path writes an audit-log entry naming the source
fingerprint and row count
- a malformed suppression row is rejected at the validator before any
DB write
Adds a focused block exercising those three properties against the real
Express app, real SQLite, and the real apply transaction.
* docs(features): refresh CVE suppressions troubleshooting and field guidance
Converts the troubleshooting section to Mintlify Accordion blocks so
each entry is foldable and the page stays scannable. Adds entries for:
- control-identity mismatch on a replica (anchor-aware reanchor flow)
- mirrored rules persisting after a control demote
- the 10000-row truncation cap on a fleet sync push
Adds a privacy note to the Reason field guidance: do not paste
credentials, tokens, or vendor secrets there, since the field replicates
fleet-wide and surfaces on every node's suppressions panel.
|
||
|
Anso
|
a284732a95 |
feat(fleet-sync): hide other replicas' identity-scoped policies on a replica (#973)
GET /api/security/policies on a replica now returns only the policies
that apply to THIS replica. Replicated rows with a node_identity
targeting a sibling replica are filtered out so an operator cannot
enumerate the names and rules of policies meant for another node in
the fleet. Defense in depth: the security panel is admin-only, but a
backend filter is bypass-proof and matches Sencho's privacy posture.
Internal evaluators (getMatchingPolicy, evaluateScanAgainstPolicies)
keep using the unfiltered list because they already enforce identity
matching at evaluation time.
CVE suppressions are fleet-wide on every replica (no node_identity
column) so no analogous filter is needed.
Public surface:
- DatabaseService.getScanPoliciesForUi(role, selfIdentity): the
filtered variant, called from securityRouter.get('/policies').
Tests:
- 3 new vitest cases: control sees full set; replica hides
other-replica scoped rows; replica always includes locally created
rows.
- Full backend suite: 1795 pass / 5 skipped.
|
||
|
Anso
|
4007709590 |
fix(fleet-sync): hygiene pass on receiver behavior and cleanup (#972)
A bundle of small file-local fixes to the receiver path and node-deletion flow. Changes: - F4 receiver audit log: applyIncomingSync now writes a system audit entry on every applied push so mirrored security-rule changes show up in the replica's audit panel with a clear control-side origin. - F7 pilot-agent skip: pushResource explicitly excludes pilot-agent nodes (they have no api_url for HTTP push) and warns once per node id so the operator sees they will not receive replicated policies. - B4 identity-drift notification: when targetIdentity differs from the cached fleet_self_identity, dispatch a warning so the operator can audit any identity-scoped policies that may need re-targeting. - B6 stack_pattern ReDoS guard: reject patterns with 4+ consecutive wildcards or more than 8 wildcards total. Both control-side validators (POST/PUT scan policies) and the receiver-side row validator share the helper. - B9 deleteNode cascade: clear fleet_sync_status rows for the node inside the existing transaction so the sync-status panel does not render ghost entries after a node is removed. - S6 last_error redaction: formatError strips Bearer tokens and JWT-shaped values from error messages and caps at 500 chars before storing in fleet_sync_status.last_error or logging. Tests: - 8 new vitest cases covering audit-log entry, identity-drift alert, pilot-agent warn-once, formatError redaction (Bearer + JWT), ReDoS validator rejection, and a backtracking-time smoke test. - New database-fleet-sync-cascade.test.ts: deleteNode removes fleet_sync_status rows for the deleted node and leaves siblings untouched. - Full backend suite: 1792 pass / 5 skipped. |
||
|
Anso
|
33b15d6cba |
feat(fleet-sync): retry failed pushes and backfill on add-node (#970)
A control instance now retries fleet-sync pushes that hit a transient failure and backfills the security state on a freshly registered remote without waiting for the next policy edit. New service: - FleetSyncRetryService (singleton, start/stop) wakes 30s after boot and ticks every 5min. For each fleet resource, queries getFailedSyncTargets within a 24h window and re-pushes via FleetSyncService.pushResourceToNode through the same per-node mutex, so a normal fanout in flight serializes naturally with a retry. - After STALE_THRESHOLD_MS (1h) of continuous failure for a previously-working node, dispatches a single warning notification per cooldown window. Brand-new nodes that have never succeeded do not alert via this path; misconfigured remotes are caught by the test-connection affordance at registration time. - Wired into bootstrap startup/shutdown next to AutoHealService. Public surface: - FleetSyncService.pushResourceToNode(node, resource): targeted push to one node that re-uses the per-node mutex. Used by the retry service and any future targeted-resync flow. - routes/nodes.ts POST /api/nodes fires pushResourceAsync for both resources after a remote-proxy node row commits. Tuning constants centralized in fleetSyncConstants.ts: - RETRY_MAX_AGE_MS = 24h - STALE_THRESHOLD_MS = 1h Tests: - 8 vitest cases covering replica skip, retry dispatch, missing-node skip, alert-once-per-cooldown across the threshold window, no-alert for recent failures, no-alert for brand-new never-succeeded nodes, no-alert when the retry itself succeeds, start/stop idempotency. - Full backend suite: 1781 pass / 5 skipped. |