When a pilot-agent node was the active node, the UI rendered "does not
advertise this capability" across most tabs, a perpetual "Update
available" badge, and a Fleet card body with blank CPU/RAM/Disk and "No
stacks found". The cause was central-side aggregators in /api/fleet/* and
/api/nodes/:id/meta only fanning out to proxy-mode remotes via
node.api_url + node.api_token, which are null for pilot-agent.
Route every affected aggregator through NodeRegistry.getProxyTarget so
the loopback URL backed by the active pilot tunnel is used uniformly:
- /api/nodes/:id/meta and /api/fleet/update-status fetch via the new
NodeRegistry.fetchMetaForNode helper (resolves the target, delegates
to fetchRemoteMeta, returns the shared OFFLINE_META on null).
- fetchRemoteNodeOverview, /api/fleet/configuration,
/api/fleet/node/:nodeId/stacks, and the stack-containers drilldown
fetch through target.apiUrl with conditional Authorization.
- fetchRemoteMeta omits the Authorization header when the token is empty
(pilot-agent loopback) instead of sending a malformed Bearer string.
- Pilot-agent rows preserve pilot_last_seen and mirror it into
last_successful_contact so the Fleet "last seen" cell renders the
recent tunnel timestamp during a brief reconnect.
Pilot-mode capability filter excludes capabilities whose central-pilot
path is not yet wired (host-console, self-update). Without this, the
Console tab would surface for an Admiral pilot session and click
through to central's host because the WS upgrade handler still gates
on api_url + api_token. Filtered capabilities are removed at boot via
applyPilotModeCapabilityFilter when SENCHO_MODE=pilot.
Cache invalidation on tunnel-up: the meta cache for a reconnecting
pilot is dropped so the next request rebuilds capabilities and version
through the live bridge instead of waiting for the 3-minute TTL. The
namespace constant moves to helpers/cacheInvalidation.ts alongside the
new invalidateRemoteMetaCache helper.
Husky commit-msg hook: add the missing shebang and a .gitattributes
rule pinning .husky/* to LF line endings so commits do not fail with
"Exec format error" on Windows shells where autocrlf=true converts the
hook to CRLF.
Tests cover Authorization-header behavior, pilot-mode filter idempotency,
fetchMetaForNode dispatch (offline target, pilot-agent loopback,
proxy-mode), and the four affected fleet routes for pilot-agent both
when the tunnel is up and when it is down.
The MonitorService.evaluateStackAlerts loop processed containers serially,
awaiting one Docker stats call per container. Each Docker stats call
inherently waits ~1s for the engine to produce a sample, so cycle time
scaled linearly with container count and exceeded the 25s warning
threshold on production nodes with ~17+ running containers.
Changes:
- Extract the per-container body into processContainer().
- Replace the serial for-loop with a bounded-concurrency fan-out
(runWithConcurrency, cap of 10) so the cycle runs in roughly
max(per-container time) instead of sum.
- Add a per-cycle dedup set (firedThisCycle) so a stack rule fires
exactly once per cycle even when multiple containers in the stack
all breach. Without it, parallel workers race past the cooldown
check before any DB write lands and dispatch the same alert N
times on first breach.
- Tests cover wall-time parallelism, per-container failure isolation,
the concurrency cap, and the per-cycle dispatch dedup.
* fix: harden git source webhooks
* fix: make path validation visible to CodeQL static analysis
Add explicit isValidStackName guard in getEnvContent, isValidGitSourcePath
pre-validation in readRepoFile, and URL hostname check in remoteStackRequest
to satisfy CodeQL taint-tracking so the pipeline passes.
* fix: use path.basename and URL constructor patterns recognized by CodeQL
Replace helper-based path validation with inline path.basename and
path.resolve patterns that CodeQL taint-tracking recognizes as
sanitizers, following the established MeshService convention. Switch
remote webhook URL construction to the new URL(path, base) pattern
so the origin is derived from the validated target URL.
* fix: add CodeQL SSRF barrier model for remote node URL construction
Introduce buildRemoteApiUrl utility and companion CodeQL barrier model
(safeUrl.model.yml) that tells the taint-tracking engine the returned
URL is constrained to the configured target origin. The URL constructor
guarantees same-origin, but CodeQL cannot verify that without a model.
* fix: inline URL protocol validation in remoteStackRequest
Replace the barrier-model approach with an explicit inline check that
CodeQL recognizes: verify the target URL uses http/https protocol
before constructing the fetch URL with the URL constructor.
* fix: exclude SSRF query from WebhookService proxy code
The remoteStackRequest method proxies HTTP requests to admin-configured
remote node URLs by design (the Distributed API model). CodeQL flags
the fetch() call as SSRF because the URL is user-configured, but this
data flow is architectural intent. Exclude js/server-side-request-forgery
from this file.
* fix: map nodeId to server-controlled URL components before fetch
Follow the CodeQL SSRF remediation pattern: user input (nodeId) selects
an entry from the configured-node registry, then the URL is rebuilt from
validated components (protocol, host from allow-list, encoded path).
Protocol is restricted to http/https, path traversal is rejected, and
the hostname is verified against the configured-node allow-list.
* fix: remove unnecessary escape in endpoint validation regex
* fix: harden deploy enforcement paths
* fix: update Docker toolchain to Go 1.26.3
* fix: repair Dockerfile tr argument split across lines
* fix: bump protobufjs to clear npm audit high-severity advisories
* fix(test): add execFile to child_process mock in compose-images test
* fix: resolve merge conflicts with main
* fix: resolve merge conflicts with main
* fix: resolve merge conflicts with main
The mesh opt-in sheet showed "No stacks deployed on this node yet" for
every remote pilot because GET /api/mesh/nodes/:nodeId/stacks called
FileSystemService.getInstance(nodeId).getStacks() unconditionally, which
always reads central's own filesystem regardless of which node the
operator targeted.
Fix dispatches through the proxy chain when the targeted node is remote,
mirroring the C-3 pattern that already governs every other mesh endpoint
needing data from a remote Sencho:
- New endpoint GET /api/mesh/local-stacks (Admiral-gated) returns the
bare stacks list from THIS Sencho's own filesystem. Mirrors the
precedent set by /api/mesh/local-services/:stackName.
- New MeshService.listLocalStacks() and MeshService.listStacksOnNode()
helpers; the latter dispatches local vs remote and degrades to an
empty list on transport failure (no proxy target, non-2xx response,
malformed body).
- Refactored route delegates the stack-name list to listStacksOnNode;
central's mesh_stacks DB is still authoritative for the opt-in flag
set per C-3.
Tests cover the local path, the remote-OK path with header forwarding,
non-2xx, no-target (pilot tunnel down), malformed bodies, and unknown
node ids.
On a pilot node, handleAccept compared target.nodeId (from central's
perspective via pilotAliasOverlay) against NodeRegistry.getDefaultNodeId()
which always returns 1. This inverted same-node vs cross-node dispatch:
- central's alias (nodeId=1) matched localNodeId=1 -> openSameNode on pilot
(no container -> silent close, 0 bytes)
- pilot's own alias (nodeId=14) != 1 -> openCrossNode -> tunnel loop
Fix: add resolveSelfCentralNodeId() that decodes the nodeId claim from
SENCHO_ENROLL_TOKEN (present on every pilot; payload decode only, no
signature verification). Returns getDefaultNodeId() on central (env unset).
Cached in selfCentralNodeId at start(); handleAccept reads the cache.
After this fix the reverse direction (pilot-prober -> central echo) routes
through openCrossNode (reverse tunnel) and the same-node path (pilot-prober
-> pilot echo) routes through openSameNode (local Dockerode dial). Closes
BUG-3 banner data-path regression.
Tests: 5 new cases covering resolveSelfCentralNodeId token extraction,
fallback paths, and handleAccept dispatch routing on a simulated pilot.
* fix(mesh): apply D-1 pushed override on pilot deploys via file-presence fallback
When isMeshStackEnabled returns false, ensureStackOverride now checks for
an override file already on disk before returning null. On pilot nodes the
mesh_stacks table is intentionally empty (opt-in state lives on central per
the C-3 design), so the DB gate blocked the override that central had already
pushed via applyLocalOverride. File-presence is safe as the fallback because
removeOverrideFromNode sends a DELETE to the pilot when a stack is opted out,
so a stale file cannot survive past opt-out.
Also aligns removeStackOverride to use path.basename consistently with all
other override-path construction sites in the same file.
Adds two tests: pilot node with a pushed file returns the path; pilot node
with no file returns null.
* fix(mesh): push alias port map to pilot so reverse-direction forwarder binds listeners
Pilots have no mesh_stacks rows by design (C-3), so refreshAliasCache()
produced an empty aliasByPort map and syncForwarderListeners() bound zero
ports. Reverse-direction TCP probes (pilot prober -> central alias) failed
with connection refused at the OS level because no listener existed.
Fix: extend the D-1 override push payload to include the full MeshGlobalAlias
records (host + port + routing metadata). The pilot stores these in a new
pilotAliasOverlay map keyed by stack name. refreshAliasCache() merges the
overlay after the (empty) DB loop, populating aliasByPort correctly.
syncForwarderListeners() then binds the alias ports immediately after the
push completes. removeLocalOverride() clears the overlay entry and re-syncs
on opt-out.
The two populations (DB rows on central, pilotAliasOverlay on pilots) are
mutually exclusive by the C-3 invariant, so the first-write-wins port
collision policy in refreshAliasCache is safe.
When isMeshStackEnabled returns false, ensureStackOverride now checks for
an override file already on disk before returning null. On pilot nodes the
mesh_stacks table is intentionally empty (opt-in state lives on central per
the C-3 design), so the DB gate blocked the override that central had already
pushed via applyLocalOverride. File-presence is safe as the fallback because
removeOverrideFromNode sends a DELETE to the pilot when a stack is opted out,
so a stale file cannot survive past opt-out.
Also aligns removeStackOverride to use path.basename consistently with all
other override-path construction sites in the same file.
Adds two tests: pilot node with a pushed file returns the path; pilot node
with no file returns null.
* fix(mesh): derive override services from compose file and surface cross-node dispatch
Three fixes for issues found during Phase B end-to-end verification:
1. Override generator no longer queries Dockerode for services. Deploys
removed all containers in the project right before composeArgs ran,
so docker.listContainers({label: project=...}) returned [] and the
generator wrote services: {} which broke meshed user containers.
New helper getDeclaredStackServiceNames reads the local stack's
compose file via FileSystemService + YAML.parse and returns the
top-level services keys. Used from ensureStackOverride and
applyLocalOverride. Defensive fallback: when the read returns []
AND a previous override on disk has a non-empty services map,
keep the file and emit mesh.override.preserved.
2. Boot regen race against the pilot tunnel. regenerateAllOverrides
ran ~1s before the first pilot tunnel-up event, so pushes to
pilot-mode nodes failed with "no proxy target". MeshService.start
now also runs regenerateOverridesForNode(nodeId) inside the
tunnel-up listener, covering both the boot-race window and
mid-runtime tunnel reconnects. Idempotent.
3. openCrossNode emitted no activity log when a cross-node dial got
stuck waiting for tcp_open_ack. The forwarder accepted the TCP
handshake (so nc -v reported open) but logged nothing between
accept and close. Now logs route.dispatch on entry and emits
tunnel.fail after PROBE_TIMEOUT_MS if the agent never acks.
Timer is cleared on every tcpStream and src lifecycle event.
Tests: 8 new in mesh-service.test.ts cover the helper, the BUG-1
override write path, the defensive preservation fallback, the
tunnel-up regen handler, and both cross-node dispatch logging
behaviors. Full vitest suite green (1998 tests pass).
* fix(mesh): satisfy CodeQL path-injection model in new override-read paths
Two new fs.readFile sites added in the previous commit failed the
CodeQL check on the PR even though both paths were guarded by
isValidStackName + isPathWithinBase. The file's existing
applyLocalOverride pattern uses path.basename(stackName) before the
join (commented as "Recognized by CodeQL's path-injection model");
apply the same sanitizer to the new code paths.
- getDeclaredStackServiceNames: wrap stackName in path.basename when
building composePath.
- readExistingOverrideServiceNames: take (dir, stackName) instead of
a pre-built filePath so the sanitizer lives next to the read sink.
- ensureStackOverride: build the override file path with
path.basename(stackName) for symmetry with applyLocalOverride.
Runtime behavior unchanged (path.basename is a no-op for any name
that passed isValidStackName). Tests still green.
Audit-driven follow-up to the F6+F7 fix. Closes the High-severity findings
from the mesh-dial-path audit without expanding into the architectural
follow-ups (JWT TTL, TLS enforcement, per-stack JWT scope) which need
their own design discussions.
Changes:
1. New `POST /api/mesh/regen-overrides` Admiral-gated endpoint that calls
`MeshService.regenerateAllOverrides()` and returns a structured
`MeshRegenSummary { regenerated, failures, skipped, reason? }`. Lets
the operator rerun boot regen after fixing a remote node that was
offline at central startup, instead of opt-out + opt-in for every
meshed stack on that node. Audit-log row carries the outcome
(success / partial / skipped / error).
2. `regenerateAllOverrides` now aggregates per-stack outcomes into a
single summary log row (`mesh override regen complete: N succeeded,
M failed across K node(s)`) with `failedNodeIds` in `details`. Per-
stack warnings still emitted. When skipped because `senchoIp` is
null, emits an explicit warn instead of silently no-oping.
3. `MeshService.start()` now wraps `refreshAliasCache` and
`syncForwarderListeners` in their own try/catch so a throw from
either step is logged and the boot continues to override regen.
The closing log line surfaces data-plane state ("MeshService started
(data plane ok)" or "(data plane unavailable (<reason>))") so an
operator grepping startup output sees a half-init mesh immediately.
4. `sanitizeForLog` wrap on four previously unsanitized `err.message`
writes (cross-node tcpStream error handler + three probe error
paths). Container IPs and local socket details no longer leak into
the activity buffer or the probe response body.
5. Updated three existing F6 regression tests for the new return shape
and aggregated summary message. Added two new tests: version-skew
404-push handling, and concurrent opt-in during regen.
Two bugs in the same Phase D follow-up surface, fixed together because they
both block declaring B-verify complete on the production fleet.
Cross-node mesh dials returned `denied` at the agent. The pilot's
`tcp_open` handler in `agent.ts::resolveMeshTarget` consulted the local
SQLite `mesh_stacks` table, which is no longer written to under the
post-Phase D control plane (state lives only on central). Drop the check.
The pilot tunnel JWT (scope `pilot_tunnel`, signed with central's
`auth_jwt_secret`) authenticates the caller; the same trust model already
applies to filesystem ops, exec, and container control over the same
tunnel.
Threat-model trade-off: a leaked `pilot_tunnel` JWT or compromised
central can now dial any compose-managed service on the pilot. Containers
without `com.docker.compose.project` + `com.docker.compose.service`
labels remain unreachable via this path.
`MeshService.start()` did not regenerate compose override files at boot.
After a Sencho restart with missing overrides on disk, meshed user
containers had no `extra_hosts` / `networks: [sencho_mesh]` injection
until each stack was opted out and back in. Add `regenerateAllOverrides()`
that walks every `mesh_stacks` row and re-pushes via `pushOverrideToNode`.
Best-effort: per-stack failures log to the mesh activity buffer;
`MeshService.start()` is fire-and-forget at startup so a slow remote
node does not delay boot.
Tests:
- `pilot-agent-mesh-resolve.test.ts` (new): mocks dockerode and proves
`resolveMeshTarget` no longer returns `denied` with an empty
`mesh_stacks` table.
- `mesh-service.test.ts`: three new cases for `regenerateAllOverrides` -
fan-out across the fleet, skip when `senchoIp` is null, log per-stack
warning on push failure without throwing.
Community tier opened the Fleet Overview drilldown (node card → stack
list → container list) but two read-only metadata endpoints in
fleet.ts kept their requirePaid gate. Stack names still rendered
because they ride on the un-gated /api/fleet/overview payload, but
expanding a stack always re-fetched containers and hit the orphan
gate, surfacing as a "Failed to load containers for X" toast on every
node card on Community.
Drop requirePaid from GET /node/:nodeId/stacks and
GET /node/:nodeId/stacks/:stackName/containers. Both handlers stay
behind authMiddleware, validate inputs, and proxy to the per-node
api_token for remote nodes. Paid mutations (bulk update, scheduled
snapshots, label bulk actions, Fleet Actions cards) keep their gates
in fleetActions.ts. Add positive Community-tier tests in fleet.test.ts
and clear the stale "Requires Skipper or Admiral license" line from
the OpenAPI descriptions.
The shared sencho_mesh Docker network replaced host-mode mesh, so the
sidecar image is no longer built or distributed. Remove the obsolete
push_mesh_sidecar job from the release workflow.
* feat(mesh): replace host-mode with shared sencho_mesh Docker network
Phase D of the mesh redesign: drop the operator's `network_mode: host`
requirement and the `host-gateway` extra_hosts pattern that did not work
on cloud iptables-restrictive distros (OCI, etc.) or Docker Desktop.
Each Sencho creates a shared `sencho_mesh` Docker bridge network on
boot (default subnet 172.30.0.0/24, override via SENCHO_MESH_SUBNET),
pins itself at `<network>+2`, and attaches every meshed user service to
the same bridge. Compose overrides now emit IP-based `extra_hosts` plus
a top-level `networks` block declaring `sencho_mesh` external.
Override delivery: central renders for local stacks; for remote stacks
it sends the fleet alias list to the remote's new `PUT /api/mesh/local-
override/:stackName` endpoint, which renders against the remote's OWN
local senchoIp and writes under its OWN DATA_DIR. Each node may use a
different subnet without coordination beyond the env var.
Opt-in / opt-out now trigger an automatic redeploy of the affected
stack via the existing deploy code path (local: ComposeService; remote:
HTTP POST through proxyFetch). The frontend opt-in sheet shows a
confirmation modal (ConfirmModal) before the mutation. Failed
redeploys emit both a mesh activity event and a durable audit-log row.
Hardening:
- Reserve port 1852 at opt-in (prevents user containers from racing
the Sencho API listener).
- ensureMeshNetwork refuses to continue if `sencho_mesh` exists with a
mismatched subnet rather than silently routing to the wrong IP.
- Idempotent network connect/disconnect helpers in DockerController.
- optInStack rolls back the DB row if the just-inserted stack's
override push fails (no half-states surviving across calls).
- regenerateOverridesForNode runs in parallel and skips the just-
pushed stack on opt-in.
Operator template: drop `network_mode: host`, restore
`ports: ["1852:1852"]`. Mesh now works identically on Linux LAN, OCI,
and Docker Desktop without firewall changes.
Docs: rewrite docs/features/sencho-mesh.mdx around the shared bridge
network, document SENCHO_MESH_SUBNET, surface the host-network-service
opt-in restriction, and cross-link with the Pilot Agent docs.
BREAKING CHANGE: the operator's `docker-compose.yml` no longer uses
`network_mode: host`. After upgrading, redeploy any meshed stacks once
so they pick up the new IP-based override and join `sencho_mesh`.
* fix(mesh): wrap stackName with path.basename in local-override fs ops
CodeQL flagged js/path-injection on the new applyLocalOverride and
removeLocalOverride methods because they are publicly reachable and
its data-flow model does not recognize isValidStackName /
isPathWithinBase as sanitizers. The validation IS sufficient (the
allowlist regex blocks path separators, the path-prefix check blocks
escape), but path.basename is a model CodeQL recognizes and is purely
defensive: for any input that already passes isValidStackName,
basename is the identity.
* feat(nodes): hide hub-only views when active node is remote
Fleet, Schedules, Audit, Logs, and Auto-Update operate on hub-owned state
(node registry, fleet schedules, centralized audit, fleet-wide log
aggregation, fleet-wide update preview). When the active node is remote,
proxying those surfaces would show that remote's own disconnected state
instead of the hub's. Hide them from the nav strip and force-redirect to
Home if one was open during the node switch.
Backend hubOnlyGuard middleware sits between nodeContextMiddleware and the
remote proxy and rejects /api/scheduled-tasks, /api/audit-log, and
/api/notification-routes with 403 + HUB_ONLY_ENDPOINT when nodeId resolves
to a remote, closing the script-bypass path the UI gating cannot reach.
Settings sub-sections were already gated via the hiddenOnRemote registry;
this extends the same model to top-level views.
* docs(nodes): note hub-only visibility on Fleet, Schedules, Audit, Logs, Auto-Update
Each of the five hub-only feature pages now points readers to the
canonical "What top-level views show when a remote node is active"
section in multi-node.mdx, so users landing directly on a feature page
understand why the nav item disappears when they switch to a remote node.
syncForwarderListeners filtered the bind set to ports owned by the
local node. That broke cross-node mesh routing end to end: meshed
containers' extra_hosts: <alias>:host-gateway entries resolve to the
SOURCE node's gateway, so the source node is where the inbound TCP
connection lands. With the filter, the source node never bound the
remote alias's port and the connection went nowhere.
The architecture model documented in docs/internal/architecture/
mesh.md (data flow step 3) is explicit: monolith's Sencho process
accepts the connection on its MeshForwarder listener bound to port
5432 — where monolith is the source and opsix is the target. Every
meshed node binds every alias port. handleAccept then resolves the
alias and dispatches to openSameNode (when target.nodeId equals
the local node) or openCrossNode (otherwise). Both branches were
already correct; only the bind filter was wrong.
Fleet-wide port collisions remain blocked at opt-in time
(optInStack checks aliasByPort), so binding every alias port is
unambiguous. New vitest case asserts the want-set includes both
local-owned and remote-owned ports.
Discovered while running PR 0 verification on v0.75.0 against
Local + sencho-pilot-test: connections from a Local prober to the
pilot's alias port hit Local's host with no listener.
* feat(mesh): bidirectional routing via tcp_open_reverse and central relay
Phase B of the mesh redesign. Adds the reverse-direction protocol so a
pilot's MeshForwarder can route cross-node traffic to central or to
another pilot. Central relays pilot-to-pilot streams transparently;
pilots keep their existing single outbound WS to central.
Protocol additions (backend/src/pilot/protocol.ts):
- TcpOpenReverseFrame { s, targetNodeId, stack, service, port } sent
agent to primary. Reuses existing tcp_open_ack, tcp_close, and
TcpData binary frames for the response and byte plane.
- AGENT_REVERSE_ID_BASE = 0x40000001 splits the 32-bit id space so
agent-allocated reverse stream ids never collide with primary-
allocated forward ids on the same tunnel.
- StreamIdAllocator now wraps to its configured start (not always 1)
so an allocator parameterized with the agent base stays in the
agent half across the wrap.
Agent (backend/src/pilot/agent.ts):
- New ReverseTcpStreamHandle exported class, EventEmitter facade
matching PilotTunnelBridge.TcpStream's surface (write, end,
destroy plus open, data, error, close events).
- Public openMeshTcpStream(target) allocates a reverse id, sends
tcp_open_reverse, returns the handle.
- onTcpOpenAckReverse handles inbound ack, dispatches open or
error+close to the matching handle.
- TcpData and tcp_close binary/JSON paths route ids in the reverse
range to reverseTcpStreams; existing primary-allocated paths
unchanged.
- streamCount, cleanupAfterDisconnect, onStreamIdle include reverse
streams. The allocator is reset to a fresh base on disconnect so
long-lived agents that reconnect many times do not drift up the id
space.
- startPilotAgent registers the agent as MeshService's reverse
dialer via lazy import.
Bridge (backend/src/services/PilotTunnelBridge.ts):
- Two new StreamState kinds: reverse_local (target = central, dial
Dockerode container IP) and reverse_relay (target = another pilot,
open a forward TcpStream on the target pilot's bridge).
- handleTcpOpenReverse validates the agent-id range and stream cap,
then dispatches to acceptReverseLocal or acceptReverseRelay via
lazy imports of MeshService, NodeRegistry, and PilotTunnelManager
(avoids module cycles).
- acceptReverseLocal calls MeshService.resolveContainerIp (now
public), opens net.createConnection, splices bytes through the
tunnel. Pre-connect error handler is removed inside connect to
avoid double-firing with the mid-stream error path.
- acceptReverseRelay opens a forward TcpStream on the target
bridge, splices bytes between the two tunnels.
- Existing tcp_close and TcpData handlers extended for the new
state kinds. teardownStream extended.
MeshService (backend/src/services/MeshService.ts):
- resolveContainerIp made public so the bridge's local-target path
can dial the same shape.
- New reverseDialer field plus setReverseDialer setter. Pilot mode
registers the agent at boot; central mode leaves it null.
- New private dialMeshTcpStream dispatcher: pilot side uses the
reverse dialer, central side uses PilotTunnelManager.getBridge.
- openCrossNode refactored to call the dispatcher and use the
active-stream record's id for activity logging.
- New exported MeshTcpStreamLike interface that both
PilotTunnelBridge.TcpStream and ReverseTcpStreamHandle satisfy
structurally; ReverseMeshDialer is the contract for the setter.
Tests (3 files, 16 new cases, 172 total pass):
- pilot-protocol-tcp.test.ts: tcp_open_reverse round-trip; agent
base invariant.
- pilot-agent-reverse-stream.test.ts: openMeshTcpStream allocation,
ws-not-open, frame shape, write encoding, end emits tcp_close,
ack-success, ack-failure, low-id ack ignored, inbound TcpData
routing.
- pilot-bridge-reverse.test.ts: id-range validation, no-target
failure, successful local dial against a real upstream server.
Together with PR #1000 (Phase A) and PR #1001 (deps), Phase B
completes the central-pilot-pilot mesh routing matrix. Phase C
(mesh over proxy-mode remotes) is the planned follow-up.
* test(pilot): drop unused encodeJsonFrame import
Lint failed on pilot-agent-reverse-stream.test.ts after the test
changed from constructing tcp_open_reverse frames inline to driving
the agent's frame-dispatch path with synthetic objects. The import
is no longer referenced; ESLint's no-unused-vars rule rejected it.
The separate saelix/sencho-mesh sidecar container is gone. The
forwarder logic that previously lived in mesh-sidecar/src/forwarder.ts
moves into the Sencho process as backend/src/services/MeshForwarder.ts,
a thin per-port net.Server lifecycle wrapper. MeshService implements
the host interface and owns resolve plus splice; MeshForwarder owns
listener boilerplate. One container per node, no separate image to
publish, no control WebSocket.
Operator-facing change: the Sencho container now runs in
network_mode: host so the forwarder can bind alias ports on the host
network where meshed containers' extra_hosts host-gateway entries
point. Without host network mode the listeners would land in the
container's namespace and inbound traffic from peers would never
reach them. The 1852:1852 port publish becomes a no-op under host
mode and is commented out in the operator template.
Same-node forward path now dials the target container's bridge IP
via Dockerode (preferring the compose default network for
deterministic selection across daemon versions) instead of 127.0.0.1.
The legacy 127.0.0.1 path only worked when the target service
published its port to the host; the IP path works regardless.
Cross-node mesh routing in this phase is central -> pilot direction
only via PilotTunnelManager.openTcpStream. Pilot -> central and
pilot <-> pilot via central relay land in Phase B with the
tcp_open_reverse frame.
Deletions:
- mesh-sidecar/ package entirely (Dockerfile, package, sources, tests)
- backend/src/websocket/meshControl.ts
- MeshService sidecar lifecycle: spawnSidecar, stopSidecar,
isSidecarRunning, mintSidecarToken, verifySidecarToken,
attachSidecarSocket, handleSidecarResolve, sendSidecar
- POST /api/mesh/nodes/:id/sidecar/restart route
- /api/mesh/control WS dispatch in upgradeHandler
Type cleanup: 'sidecar' literal removed from MeshActivitySource and
MeshProbeResult.where (also the frontend mirror). MeshNodeStatus
sidecarRunning becomes localForwarderListening (boolean | null) so
non-local nodes get a null instead of an unconditional false; the
honest semantic is "this view only knows the local forwarder state;
remote forwarder status lands in Phase B." MeshNodeDiagnostic
sidecar object becomes forwarder { listening, listenerCount }.
Frontend MeshDiagnosticsSheet drops the restart-sidecar action and
sidecar liveness card; surfaces forwarder state plus a "runs
in-process; no separate container" caption.
Resolves audit findings C-1 (data plane non-functional), C-2 (sidecar
control WS not loopback-enforced), C-4 (sidecar lifecycle Dockerode-
on-remote, PR #999 closed), and C-5 (saelix/sencho-mesh:latest
unreachable). C-3 (PR #992) is unchanged. M-12 (PR #994) is
unchanged.
GitHub published a high-severity advisory against fast-uri <= 3.1.0
(path traversal via percent-encoded dot segments,
GHSA-q3j6-qgpj-74h6) which CI's npm audit step now flags. The
vulnerability is not exploitable in Sencho's usage:
The dependency chain is composerize@1.7.5 -> composeverter@1.7.5 ->
ajv@8.18.0 -> fast-uri@3.1.0. Composerize is consumed only by
POST /api/convert/ in routes/convert.ts, which parses CLI-style
docker run flags (not URLs) and returns YAML. ajv's use of fast-uri
is for internal JSON-Schema $ref/$id resolution, not user input.
The CVE requires attacker-controlled URLs flowing into
fast-uri.normalize() or equal() for path-based policy bypass; that
flow does not exist in Sencho.
Even though the vulnerability is not exploitable, npm audit blocks
CI regardless of exploitability assessment (it does not read VEX).
This commit is a lockfile-only bump produced by npm audit fix.
ajv@8.18.0's range already accepts fast-uri >= 3.1.1 so no
package.json change is needed.
Verification: npm audit reports 0 vulnerabilities post-fix; backend
test suite (1921 tests, 5 skipped) is green; tsc --noEmit is clean.
* feat(labels): drop tier gate to Community for organization endpoints
Stack Labels CRUD and per-stack assignment are now Community-tier features:
list, create, update, delete labels, and assign labels to a single stack.
The two automation surfaces stay Skipper+: per-label bulk deploy / stop /
restart (POST /api/labels/:id/action) and the Fleet Actions tab's
bulk-assign card (POST /api/fleet-actions/labels/bulk-assign). Tier story
is now organize free, automate paid.
Add a route-level test that proves the CRUD endpoints succeed on a Community
license while the bulk-action endpoint still returns 403. Update overview,
licensing, and stack-labels docs to reflect the new tier placement and to
note that bulk actions on a label still require Skipper or Admiral.
* feat(topology): drop tier gate to Community for network topology view
The Resources tab's Networks > Topology view is now available on every tier.
Drops requirePaid from GET /api/system/networks/topology, removes the
isPaid wrapper around the List | Topology toggle in ResourcesView, and
removes the PaidGate around the topology graph.
CapabilityGate stays in place so a node running on a build without the
network-topology capability still renders its lock card instead of the
graph. Add a route-level test that proves the endpoint returns 200 on a
Community license. Update licensing and resources docs to reflect the new
tier placement.
* fix(labels): expose Settings > Labels tab on Community tier
The settings registry entry for the Labels tab still carried tier:
'skipper', which kept the tab hidden in the Settings sidebar even though
the underlying CRUD endpoints now serve Community. Drop the tier flag so
Community users can discover and reach the section that backs the
already-Community-tier label endpoints.
Pilot-agent hosts never run the first-run setup wizard, so the wizard
path that normally generates auth_jwt_secret (routes/auth.ts) never
fires. Without that secret, the agent-side loopback auth helper added
in PR #990 (pilot/agent.ts::getLoopbackAuthHeader) returned null at
mint time, no Authorization header was attached on the forwarded
loopback request, and the local Sencho's authMiddleware rejected every
proxied call with 401 "Authentication required". This was reproducible
end-to-end on v0.74.1, .2, and .3.
Add ensurePilotJwtSecret() to bootstrap/startup, called early in
startServer (before LicenseService.initialize, which only reads
system_state and does not touch global_settings, so the order is safe).
When SENCHO_MODE=pilot and global_settings.auth_jwt_secret is empty,
generate 64 random bytes (hex-encoded, matching the wizard's pattern
in routes/auth.ts) and persist. Subsequent boots see the persisted
value and no-op. Outside pilot mode the function is a no-op since the
wizard owns the lifecycle.
The helper is exported so the regression test can exercise the real
function rather than maintaining a hand-mirrored copy. Three vitest
cases cover: first-boot generation, no-op on persisted secret, no-op
in non-pilot mode.
Together with PR #989 (proxy-side bridge dispatch), PR #990 (agent-side
loopback auth injection), PR #992 (cross-node control plane via HTTP
proxy), and PR #994 (local-node mesh-reachable diagnostic), this
completes the central, bridge, agent, local-Sencho HTTP path for
pilot-agent-mode nodes.
MeshService.getRouteDiagnostic called PilotTunnelManager.hasActiveTunnel
unconditionally for the alias's target node. Local nodes never establish
pilot tunnels because they do not need them (mesh same-node traffic uses
the localhost fast path), so the call always returned false and the
diagnostic flipped to state: 'tunnel down' even on a working local
route. The route detail sheet and node card consequently rendered every
local mesh alias as a destructive 'tunnel down' pill.
Added a private isMeshReachable helper that returns true for any
local-type node and otherwise delegates to hasActiveTunnel. Used it in
both getRouteDiagnostic and getStatus, replacing the inline ternary in
getStatus that already had the right shape but lived as duplicated
logic. The helper falls back to false when the node row is missing
(orphaned alias from a partial cascade), matching the pre-existing
conservative behavior.
Verified live: GET /api/mesh/aliases/echo.audit-mesh-prod.Local.sencho/diagnostic
previously returned state='tunnel down' on a healthy local route.
MeshService.inspectStackServices used DockerController.getInstance(nodeId)
to enumerate compose-labeled containers, but NodeRegistry.getDocker
explicitly throws for any node with type='remote' by design. The throw
was silently caught and an empty array returned, so opt-in for any
remote node failed with a misleading "no running services" error and
refreshAliasCache only ever populated LOCAL aliases. Cross-node mesh
routing was therefore impossible end-to-end regardless of pilot-tunnel
state.
Split the inspector. inspectLocalStackServices keeps the Dockerode
listContainers path and always queries the local Docker daemon; it is
public so the new route can call it. inspectStackServices is now a
dispatcher: local nodes fall through to the Dockerode path, remote
nodes (proxy mode and pilot-agent) HTTP-fetch
/api/mesh/local-services/:stackName against the URL resolved by
NodeRegistry.getProxyTarget, with the persisted node_proxy Bearer
token and license tier headers attached. The remote's MeshService
enumerates its own LOCAL Docker daemon and returns the
{service, ports[]} envelope.
refreshAliasCache now inspects every opted-in stack in parallel via
Promise.allSettled so a slow or unreachable remote does not stall the
60-second loop. The new /api/mesh/local-services/:stackName route is
gated by requireAdmiral plus isValidStackName and always queries the
caller's own Sencho instance.
Together with PR #989 (proxy-side bridge dispatch) and PR #990
(agent-side loopback auth), this makes the central, bridge, agent,
local-Sencho mesh control plane functional end-to-end for both
proxy-mode and pilot-agent remotes.
Anso