Splits the Cloud Snapshots panel in Settings > Cloud Backup into pages of
10 items with prev/next chevrons and a page counter, matching the existing
pattern shipped in Fleet Snapshots. Long-running deployments with many
snapshots no longer overflow the settings panel.
The empty state, refresh, download, and delete flows are unchanged. The
safePage clamp handles the last-page-delete case without extra reset
logic. Chevron buttons carry aria-label values for screen reader users.
The bundle-row action uses the lucide Send icon but the button's
aria-label was "Push", which surfaced inconsistently to screen-reader
users vs the visible/iconic intent. The same action is referred to
as "Send" in the feature docs, so standardise on Send everywhere.
Adds an in-sheet "Import from stack" affordance to the Secret bundle
editor (edit mode only). Picks a node + stack + env file basename,
calls the existing /secrets/:id/import-from-stack endpoint, and
overlays the returned key/value pairs onto the editor: existing keys
have their values updated in place (preserving row order and any
duplicate-keyed rows for save-time validation), new keys append at
the bottom, empty placeholder rows are kept.
The endpoint and audit log row already shipped; only the UI trigger
was missing.
After applying a stack update the sidebar's "update available" dot stayed
visible and the stack's status indicator was stuck on the optimistic value
until the page was manually refreshed. Two root causes:
1. Image-updates state refresh was a fire-and-forget call in some paths and
entirely missing from the bulk-update, auto-update, and state-invalidate
WebSocket-handler paths.
2. stackActionsRef.current was resynced only at render time, so the post-
update refreshStacks(true) running in the action's finally block read a
stale "busy" map and preserved the optimistic mask via prev[file] ?? status.
Backend now broadcasts a state-invalidate event with scope='image-updates'
and action='stack-updated' after every successful update (single-stack route
and auto-update loop). The frontend useNotifications hook routes this to a
new onImageUpdatesChange callback wired to fetchImageUpdates in EditorLayout,
so every connected client refreshes the dot through the same code path.
Bulk update also calls fetchImageUpdates directly for fast local feedback,
and setStackAction/clearStackAction now keep stackActionsRef synchronously
in sync with state so the busy-stack check inside refreshStacks observes
the cleared map immediately.
Adds 3 unit tests covering the new WS branch (positive, scope-mismatch
negative, auto-update-settings-changed negative).
Operators previously saw "spawn docker ENOENT" or "spawn /bin/sh ENOENT" when
the host was under memory pressure, which sent them down a missing-binary
debugging path. Linux libuv's posix_spawn can fail to allocate its argv /
path-search arena under low free memory and surface the underlying ENOMEM
as ENOENT.
Centralizes spawn-error mapping in a new utils/spawnErrors.ts helper:
- Explicit ENOMEM is rewritten to "Out of memory while launching <command>
(host free memory: X MiB of Y MiB)".
- ENOENT under the 128 MiB free-memory floor is rewritten with the same
wording plus a "reported as ENOENT under memory pressure" hint.
- ENOENT for docker on a healthy host preserves the existing
"Docker CLI unavailable on this node" mapping.
- Other errors pass through unchanged.
Applied at the four named offenders: ComposeService.execute(),
ComposeService.captureCompose(), DockerController.getContainersByStack(),
and FileSystemService.getStacks() (which gets an ENOMEM-aware log line
for the scandir failure).
Startup also logs host free/total MiB once and warns when free memory is
below the 128 MiB floor, so the diagnostic surfaces before the first
spawn attempt rather than after it fails.
37 tests cover the mapping function directly and the ComposeService /
FileSystemService integration paths.
POST /api/stacks/:name/{deploy,down,update} previously returned HTTP 500
with body {"error":"spawn docker ENOENT"} when invoked against a stack
whose compose directory was missing. The status code was wrong (the
named resource did not exist, so 404 is the right answer) and the
message misled operators into thinking the docker CLI was unavailable.
Add a small requireStackExists(nodeId, stackName, res) helper in
routes/stacks.ts that validates the stack name and confirms a compose
file is present via FileSystemService.hasComposeFile before any of the
three handlers spawn docker compose. The helper is called immediately
after requirePermission and before runPolicyGate so unauthorized
callers still get 403 first and the policy gate never runs against a
phantom stack.
In ComposeService.execute(), narrow the child.on('error') handler so
the genuine docker-binary-missing case (ENOENT on the spawn itself)
rejects with "Docker CLI unavailable on this node" instead of the raw
"spawn docker ENOENT". This is defense in depth for the rare case the
pre-check cannot cover, and it fixes the misleading-message half of
the bug as well.
Cover the new contract with stack-actions-missing-stack.test.ts (four
cases: deploy/down/update return 404, invalid name returns 400). Mock
ComposeService as a tripwire so a future code path that bypasses the
guard would fail loudly. Fix stacks-failure-notifications.test.ts by
adding hasComposeFile to its FileSystemService partial mock so the
existing happy-path-error-handling cases continue to flow into
ComposeService.
The Edit affordance on the stack anatomy panel previously only flipped
the editor visibility flag and left activeTab whatever it was. After a
user clicked Files (which set activeTab to 'files'), closed the editor,
then clicked Edit, the editor reopened still on the Files tab instead of
showing the compose.yaml editor.
Make onEditCompose mirror the sibling onOpenFiles handler by also
setting activeTab to 'compose', so the Edit button always lands on the
compose editor regardless of which tab was last viewed.
Adds a third card to the Fleet Actions tab that fans out Docker prune
(images, volumes, networks) across every node in one submit. Local nodes
call DockerController under a bulk-prune lock; remote nodes receive one
POST /api/system/prune/system per target. Per-node + per-target results
with reclaimed bytes are surfaced inline via ResultsList.
Tier: Skipper / Admiral (requirePaid + requireAdmin), matching the rest
of Fleet Actions. The frontend card is mounted inside the existing
isPaid branch at FleetActionsTab; no new frontend gate is required.
The card uses an amber accent rail and the Eraser icon so it reads as
'cleanup' rather than 'destructive stop'. Scope toggle defaults to
Managed only (Sencho-tagged resources) with an All unused option that
escalates the destructive-confirm copy.
The cross-node MeshService.openCrossNode handler attached src.on('close')
and src.on('error') listeners that only called tcpStream.destroy() and
never deleted the activeStreams Map entry. MeshTcpStreamLike.destroy()
sends a tcp_close frame to the remote but does not synchronously emit
the local close event; that fires only when the remote sends back a
tcp_close ack via _dispatchClose (or when the entire tunnel tears down
and the bridge force-emits close on every stream).
Failed dials whose remote ack was lost (peer gone, network drop, or the
F-9-era timeouts) therefore leaked records into activeStreams until the
tunnel itself idle-closed, producing the monotonically-climbing
activeStreamCount documented in the mesh E2E audit (F-10).
Add a cleanupRecord() closure that clears the open timer and deletes
the record idempotently, then route all four lifecycle handlers
(tcpStream.on('error'), tcpStream.on('close'), src.on('close'),
src.on('error')) through it. Map.delete is naturally idempotent so the
double-delete that fires when both sides close normally is a free
no-op. The same-node openSameNode path already had this shape via its
existing teardown() closure; this brings the cross-node path to parity.
Three new Vitest cases in mesh-service.test.ts use an EventEmitter-
backed fake src so emit('close') actually delivers to the listener
(the existing makeFakeSocket uses vi.fn for .on which records calls
but never fires them). They cover src.close, src.error, and the
idempotency case where both src and tcpStream emit close in sequence.
The v0.81.13 Docker publish workflow failed on the post-build Trivy
re-scan because the moby/Docker vulnerability database picked up two new
HIGH CVEs against `github.com/docker/docker v28.5.2+incompatible`, which
docker-compose v5.1.3 vendors statically as a client-side API/codec
library. The CVEs were not present in the PR-side Trivy DB three hours
earlier, so PR CI passed but the release re-scan failed.
Both CVEs are daemon-side: CVE-2026-41567 targets the daemon's
PUT /containers/{id}/archive handler, CVE-2026-42306 is a race in the
daemon-side bind-mount resolution that backs `docker cp`. docker-compose
never acts as the daemon, never serves these routes, and Sencho only
invokes compose for up/down/ps. The vulnerable code paths are
unreachable. The fix lives on the github.com/moby/moby/v2 module path;
until upstream compose migrates, v28.5.2+incompatible remains the only
Go-module-resolvable version compose can reference. Same triage shape as
the existing CVE-2026-34040 entry, so the new statements mirror its top-
level products purl form.
Bumps OpenVEX document version 4 -> 5 and updates last_updated /
timestamp to 2026-05-18.
GET /api/mesh/aliases/:alias/diagnostic returned a cached state derived
from the last latency/error maps. Those maps only mutated when someone
called POST .../test or when a cross-node connect logged an event, so
once an upstream stopped the diagnostic kept reporting "healthy" until
the 60 s alias-cache refresh pruned the alias entirely.
The GET now calls testUpstream synchronously after the alias-resolved,
opted-in, tunnel-up short-circuits. Probe failures land in routeErrorMap
via logActivity (cross-node already did this; same-node timeout/error
paths now log probe.fail in the same shape), so the state computation
flips to "unreachable" on the same request that exposed the stopped
upstream. A new routeProbeAtMap stamps freshness and surfaces in the
response as lastProbeAt; the route detail sheet renders "Last probe
<age> · <ms>" using the existing formatTimeAgo helper.
Add `currentlyResolvable: boolean` per entry in `MeshNodeStatus.optedInStacks`,
derived from the existing alias cache so the new field stays consistent with
`/api/mesh/aliases` without any extra Dockerode or cross-node inspect on the
status path. The Routing tab renders an amber `suspended` pill for entries
whose stack is opted in but currently has no running services, plus a single
explanatory caption below the suspended list.
Resolves the contradictory `Mesh stacks: 1 / Aliases: 0 / No mesh services
on this node yet` copy on the node card when a meshed stack's container has
been stopped; the misleading line is now only shown when the node truly has
no opt-ins. The opt-in itself remains sticky: when the stack starts again,
its aliases reappear automatically on the next refresh.
Defensive de-dup in the UI filters suspended entries against the live alias
snapshot to handle the transient gap where `/mesh/status` and `/mesh/aliases`
return slightly inconsistent views from their separate fetches.
Tests:
- new `mesh-status-resolvability.test.ts` (6 cases) locks the resolvable /
suspended / mixed / empty / per-node-scoping / stale-alias-no-phantom
invariants for `getStatus`.
- `mesh-topology-layout.test.ts` gains a `stacksKey` resolvability-flip case
and a `meshNodeStateEqual` case asserting a resolvability flip on an
otherwise identical stack registers as a state change so the topology
layout re-runs.
Operator docs gain one new troubleshooting accordion in
`/docs/features/sencho-mesh.mdx` explaining the suspended state.
DELETE /api/stacks/:name now calls MeshService.optOutStack after the
DB cleanup so the mesh_stacks row, the override file under
<DATA_DIR>/mesh/overrides/<nodeId>/, and any derived aliases do not
outlive the deleted stack. Pre-fix, those artifacts leaked and the
reconcile loop logged "No compose file found for stack" every tick.
optOutStack is idempotent (early return when the stack was never
opted in) and already cascades override-regen plus recompose across
the rest of the fleet, so peers' /etc/hosts drop the dropped alias.
The new cascade call is best-effort relative to the delete itself:
mesh cleanup failures warn but never regress the delete contract.
New regression test asserts three contracts: cascade on delete,
no-op on never-meshed, and 200 + warn when the cascade rejects.
* fix(mesh): route peer→central traffic over the existing forward WS
The reverse mesh callback path (`/api/mesh/proxy-tunnel-from-peer`) needed
SENCHO_PRIMARY_URL on central plus a publicly reachable origin from the
peer's perspective. In a typical homelab where central sits behind NAT,
peer→central dispatch silently failed at the dialer's short-circuit and
the headline "call any service on any node by hostname" worked one way
only.
The forward WS at `/api/mesh/proxy-tunnel` is already bidirectional end
to end. Make the bridge a persistent control-plane primitive: dial every
mesh-enabled proxy peer at startup, reconcile every 60 s, never idle-close.
Peer→central traffic multiplexes over the same WS via `tcp_open_reverse`.
Removed:
- `meshProxyTunnelFromPeer.ts` WS handler and dispatch
- `MeshCentralRegistry`, `PeerToCentralMeshSessionDialer`
- `mesh_handshake` first-frame state machine in `meshProxyTunnel.ts`
- `maybeSendBootstrap`, `buildHandshakeFrame` in the dialer
- `mesh_proxy_callback_bootstrap` capability and `maybeWarnUnsetPrimaryUrl`
- `mesh_centrals` table (drop migration; greenfield, no users)
- `PilotTunnelManager.replaceOrRegisterProxyBridge` (dead after handler removal)
- twelve associated unit/integration tests plus the peer-recovery branch
in `MeshService.openCrossNode`
Added:
- `MeshService.proactiveBridgeFanout` selects every mesh-enabled proxy
peer (no longer gated on `mesh_stacks` rows)
- `startBridgeReconcileLoop` runs the fanout every 60 s (override via
`SENCHO_MESH_RECONCILE_INTERVAL_MS`)
- `MeshProxyTunnelDialer` default idle TTL is now `0` and exposes
`isDialing(nodeId)` for the status surface
- `MeshNodeStatus.reverseCallbackStatus` discriminator
(`connected | connecting | unavailable | not_applicable`) surfaced via
`/api/mesh/status` and rendered as a pill in the Routing tab
- `openCrossNode` error message distinguishes "no proxy target" from
"waiting for central to dial the reverse bridge"
- New tests: `mesh-service-proxy-tunnel-reconcile`,
`mesh-status-reverse-callback`, `mesh-proxy-tunnel-dialer-no-idle-close`
SENCHO_PRIMARY_URL is no longer required for any mesh function.
* fix(mesh): rewrite proxy-tunnel reconcile test contents
The previous commit renamed the file but the rewritten test bodies stayed
unstaged on top of the rename. This commit lands the actual rewrite: the
fanout assertion now requires every mesh-enabled proxy peer to be dialed,
not just those with `mesh_stacks` rows, and adds a reconcile-tick
repeated-call test.
compression@1.8.1's negotiator silently fails to set the Content-Encoding
response header when Accept-Encoding carries an unknown token like zstd,
even though the body is still compressed with brotli. Chromium 123+ sends
"Accept-Encoding: zstd, gzip, br" by default, including Playwright's
bundled Chromium over plain HTTP, so any homelab Sencho viewed without a
TLS terminator triggered ERR_CONTENT_DECODING_FAILED in the browser and
rendered as a blank page.
Insert a tiny middleware (step 4 of the canonical pipeline) that drops
zstd tokens from Accept-Encoding before compression's negotiator runs.
The negotiator then picks br or gzip and writes the matching header. The
fix is no-op when zstd is absent. If only zstd was offered, the header
falls back to identity so the response is served uncompressed.
Renumber the canonical-order JSDoc in app.ts from 17 to 18 steps and
update the step-number references in hub-only-guard and proxy-mount-order
test docstrings to match.
11 new tests (8 unit + 3 supertest integration against real compression
with a 3000-byte body) lock in the behavior: zstd-stripping across casing
and q-values, surviving tokens preserved verbatim, identity fallback when
only zstd was offered, Content-Encoding header always written when at
least one supported codec remains.
When a stack is opted into the mesh, every previously-meshed stack's
override.yml on disk gains the new alias, but the running containers
keep their stale /etc/hosts because extra_hosts is read at container
creation time. Cross-stack DNS for the new alias silently fails until
an operator manually redeploys each prior stack.
Complete the cascade: after regenerateOverridesAcrossFleet finishes,
fan out triggerRedeploy across every (node_id, stack_name) tuple in
mesh_stacks, skipping the just-opted-in pair (the explicit
triggerRedeploy at the end of optInStack covers it). Mirror the same
cascade in optOutStack so the dropped alias exits every container's
/etc/hosts.
triggerRedeploy already dispatches local vs remote, already wraps
runRedeploy in fire-and-forget error logging to the mesh activity
ring and audit log, and already enforces compose policy gates. The
cascade just reuses it.
regenerateAllOverrides (boot and manual /regen-overrides) is
intentionally NOT covered: a Sencho restart must not force a
fleet-wide recompose; override files alone are sufficient there.
Adds four unit tests in mesh-service.test.ts: cascade fires for every
prior meshed stack on opt-in, no double-redeploy of the just-opted-in
tuple, cascade is a no-op when no peers exist, cascade fires for
every survivor on opt-out.
The three previously-silent console.warn paths in MeshService.setupMeshNetwork
now route through a typed recordSetupFailure helper that classifies the
failure (subnet_invalid, subnet_overlap, subnet_mismatch, ip_in_use,
attach_failed, not_in_docker), emits a mesh.disable activity entry at the
matching level (error for real failures, warn for the expected dev-mode
not_in_docker case), and strips mesh_proxy_callback_bootstrap from
advertised capabilities via CapabilityRegistry.
/api/health gains a mesh.dataPlane block carrying the typed status.
/api/mesh/status carries localDataPlane at the top level so the Routing tab
renders a red banner with an operator-actionable recovery hint (set
SENCHO_MESH_SUBNET to a free /24 and recreate the container) when the data
plane is down. The success path re-enables the capability and flips the
status to ok.
Generalizes the previously-documented F-0 failure mode (IP-in-subnet
collision) to also cover the subnet-pool-overlap case where another Docker
bridge on the host already owns the requested CIDR.
The tcp_open and tcp_open_reverse receivers registered their stream
entry only after awaiting resolveTarget / resolveContainerIp. The
peer, which is free to send TcpData immediately after the open frame,
hit the lookup-miss path in handleBinaryFrame and the bytes were
silently dropped. Probes passed because they only exercise the
handshake; real HTTP hung with 0 bytes received.
Reserve the stream entry synchronously, buffer early TcpData in a
per-stream pendingData queue capped at 1 MiB, and flush onto the
local socket inside the connect handler before sending tcp_open_ack.
The payload is copied because decodeBinaryFrame returns a subarray
view of the WS receive buffer; holding the view would pin the parent
buffer past its lifecycle.
Both sides of the data plane are patched:
- tcpStreamSwitchboard.onTcpOpen (forward, agent + proxy-mode peer)
- PilotTunnelBridge.handleTcpOpenReverse / acceptReverseLocal
(reverse, primary acting as the local dial target)
Adds unit coverage for the race window and for the overflow path.
The cap constant lives in pilot/protocol.ts alongside the other
per-stream limits.
Per the C-3 design, mesh state lives on central. Pilots learn aliases
via the D-1 override push and hold them in `pilotAliasOverlay`. The
local `mesh_stacks` table on pilots has been dead since C-3 shipped.
This is a greenfield cleanup (Directive 20):
- `migrateMeshTables()` runs `DROP TABLE IF EXISTS mesh_stacks` when
`SENCHO_MODE === 'pilot'` and skips the CREATE. Central-mode
behavior is unchanged.
- The four mesh CRUD methods (`listMeshStacks`, `isMeshStackEnabled`,
`insertMeshStack`, `deleteMeshStack`) short-circuit on the same
check. Reads early-return empty/false; writes early-return;
`insertMeshStack` also warns so an accidental pilot-side write is
loud.
- New `isPilotMode()` helper mirrors the existing one in
`bootstrap/startup.ts`. Layering rules forbid the shared import; a
third call-site would justify extraction to `helpers/`.
Operator-visible behavior is unchanged on either side. On central, the
table and its rows are untouched. On pilots, the table is removed (it
held no rows in steady state) and any caller of the CRUD methods
continues to see empty list / false / no-op without touching SQLite.
Test plan
- `npx tsc --noEmit`: clean.
- New `database-service-pilot-mesh-stacks.test.ts`: 5/5 green (table
absence + four short-circuit assertions).
- `mesh-service.test.ts`, `mesh-diagnostic-local.test.ts`, and
`mesh-service-proactive-bootstrap-fanout.test.ts`: 58/58 green.
- Full backend suite: 2280/2281; the single failure is a pre-existing
Windows EBUSY flake in `filesystem-backup.test.ts` that reproduces
on a clean tree.
- Code reviewed; findings applied.
Removes the auto-publish workflow that fired on every v* tag and
generated a release post in the sencho-website repo on every 5th
release. Release posts are no longer the desired blog content for
the project, so the workflow, its helper script, and its template
are all removed together.
Deletes:
- .github/workflows/release-blog-scaffold.yml
- .github/scripts/scaffold-release-post.mjs
- .github/release-blog-template.tsx.tmpl
The 7 historical release posts are removed from the sencho-website
repo in a separate PR.
`MeshService.proxyFetch` previously threw `MeshError('push_failed', ...)`
when `NodeRegistry.getProxyTarget(nodeId)` returned null, but semantically
no push was attempted: the target simply did not exist. Callers had to
special-case the code and explain the overload in a comment.
Retarget the throw to the existing `'no_target'` variant in
`MeshErrorCode`. `listStacksOnNode` matches the new code directly; the
explanatory comment is gone since the code reads for itself.
`MeshError('push_failed', ...)` continues to signal real outbound push
failures: `applyLocalOverride` (mesh data plane not yet ready on the
receiving node) and `pushOverrideToNode` (HTTP 404 from an outdated
remote, non-OK HTTP response). Those sites are unchanged.
Operator-visible behavior is unchanged in the steady state. The rare
"remote disappeared between opt-in start and override push" case now
returns HTTP 400 with `code: 'no_target'` instead of 503; the response
body carries a clear message ("no proxy target for node N") so clients
can disambiguate.
Test plan
- `npx tsc --noEmit`: clean.
- `npx vitest run src/__tests__/mesh-list-stacks-remote.test.ts`: 8/8 green
(6 existing + 2 new cases pinning the catch-arm semantics).
- `npx vitest run src/__tests__/mesh-service.test.ts src/__tests__/mesh-inspect-remote.test.ts`:
56/56 green.
- Code reviewed; findings applied.
Notes
- PR #1082 (refactor: route inspectStackServices through proxyFetch) needs
a rebase + one-line update to its catch arm before merging on top of
this change.
inspectStackServices was constructing its Authorization, x-sencho-tier,
and x-sencho-variant headers inline. proxyFetch is the centralized
helper used by listStacksOnNode, pushOverrideToNode, triggerRemeshRedeploy,
and removeOverrideFromNode. Equivalent today for a GET-no-body call, but
any future header added to proxyFetch (audit context, license fingerprint,
request id) would silently miss inspectStackServices.
Wire output is byte-identical: same URL, same method, same three headers,
same 5 s timeout. The new catch arm converts MeshError('push_failed') back
to the existing warn + return [] contract callers (optInStack,
refreshAliasCache) rely on. Pattern mirrors listStacksOnNode.
The "From Docker Run" tab's converted compose.yaml preview uses
<pre whitespace-pre> inside a ScrollArea that defaulted to
overflow-x: hidden. Long YAML lines (long env values, long volume
paths) were silently clipped on the right with no horizontal scroll
affordance, and Radix's display: table viewport child propagated the
overflow back up the chain, pushing the form's ModalBody past the
dialog's 576px max-width.
Opt into the ScrollArea component's existing `block` prop on the
YAML preview and on both tabpanel form ScrollAreas. `block` switches
the viewport child to display: block; min-w-0 and renders a
horizontal ScrollBar (Radix mounts it only when content actually
overflows). DOM trace confirms the outer `block` is load-bearing
when the inner pre is wider than the viewport.
Matches the established pattern used by VulnerabilityScanSheet,
ScanComparisonSheet, and SecurityHistoryView.
Opting a stack into mesh on node A now propagates the new alias to every
meshed node's override file, not just stacks on node A. Same on opt-out.
Pre-fix, optInStack and optOutStack called regenerateOverridesForNode
which only walks db.listMeshStacks(nodeId). Other meshed stacks on other
nodes did not learn about the new alias until something else re-pushed
their overrides; operators worked around it with a manual
POST /api/mesh/regen-overrides plus a redeploy of each affected stack.
New regenerateOverridesAcrossFleet helper iterates db.listMeshStacks()
(no arg, fleet-wide) under Promise.allSettled, skipping the
(nodeId, stackName) tuple opt-in just pushed loudly. Per-stack failures
surface as forwarder.error activity events, matching the pattern used by
regenerateAllOverrides. Offline remote nodes leave stale overrides until
the next opt-in/opt-out, the next tunnel reconnect, or a manual rerun of
the regen-overrides endpoint.
regenerateOverridesForNode stays for the tunnel-up retry listener: a
reconnecting pilot only needs its own stacks re-pushed, not the fleet.
Tests: redirect five existing spies in mesh-service.test.ts to the new
method; add two new cases asserting cross-node cascade for opt-in and
opt-out. 52 tests in mesh-service.test.ts pass.
The 12 PilotMetrics counters (proxy_dials_failed, proxy_bridges_total,
proxy_idle_closes, the pilot-tunnel counters, and the peer-to-central
callback counters) live in a singleton holding scalar fields, so process
restart wipes them. Operators investigating rotation- or dial-failure
trends across a central restart lost the aggregate metric.
Persist the snapshot under a pilot_metrics_counters JSON row on the
existing system_state KV table. Hydrate via PilotMetrics.load() in
bootstrap/startup.ts before any service that increments runs; final-flush
in bootstrap/shutdown.ts before the SQLite handle closes. Between those
boundaries a buffered flush mirrors the audit-log pattern: every 1 s or
every 100 increments, whichever fires first. Hot increment path stays a
single property write plus a pending-counter check.
Defensive parse on the row: object check + numeric filter, so an operator
hand-edit or schema drift across releases degrades to zero state with a
warn rather than crashing boot. Schema additions back-fill missing keys
with zero so a release that adds a counter does not need a migration.
Closes F-R-4 from the mesh tracker.
The PUT /api/nodes/:id handler closed and re-dialed the mesh callback
bridge on every save that included api_token in the body, even when the
token was unchanged. The frontend always sends the full formData on
Save, so renames and compose_dir edits against a mesh-enabled proxy
remote produced a wasted closeBridge + ensureBridge round-trip and a
spurious manager_rejected entry in the activity log.
Gate the re-bootstrap on a real value diff against the persisted token.
Adds an existing-node lookup (returns 404 on missing id, which the
handler previously lacked) so the comparison has the pre-update value.
Reorders the guards so resource-not-found beats payload validation.
Adds one integration test for the same-token path; the existing three
trigger-2 cases continue to assert close + ensure firing on a real
rotation, no-token-in-payload, and mesh-disabled.
* fix(mesh): distinguish proxy bridge from pilot tunnel in registerProxyBridge rejection
When central's MeshProxyTunnelDialer dials a node whose slot is already
held by a peer-initiated proxy bridge, the registration is correctly
refused, but the activity-log message read "pilot tunnel already
registered for node N; proxy bridge refused" regardless of which kind
of bridge was actually in the slot. No pilot tunnel exists in
proxy-mode remotes by definition, so the wording pointed operators at
the wrong subsystem.
Use the existing bridgeKinds index (already consulted correctly by the
sister method replaceOrRegisterProxyBridge) to emit a kind-accurate
message: "proxy bridge already registered for node N; concurrent dial
refused" when the existing bridge is a peer-initiated proxy bridge,
keeping the original "pilot tunnel" wording for the pilot-agent case
so existing log filters and the sister test still match.
Pure cosmetic; the rejection behavior is unchanged.
New regression test asserts the discriminator in both directions. Also
tightens the test-fixture beforeEach to clear bridgeKinds alongside
bridges so future tests cannot read a stale kind.
* chore: ignore local trailer directory in git repository
* refactor(mesh): centralize bridge-kind type and conflict-message formatter
Followup tidy on PilotTunnelManager after /simplify flagged two related
items in the F-R-2 change:
- Extract `BridgeKind = 'pilot' | 'proxy'` as a module-scoped type alias
so the kind taxonomy lives in one place. `bridgeKinds` and
`injectBridgeForTest` now consume it; the inline union literal is
gone.
- Extract `formatBridgeConflict(nodeId, existingKind)` as a private
helper. Both `registerProxyBridge` and `replaceOrRegisterProxyBridge`
call it from their "slot already held" rejection paths, removing the
byte-identical "pilot tunnel already registered for node N; proxy
bridge refused" string copy across the two sites.
- Trim the 6-line WHAT comment above the rejection throw to a 2-line
WHY so the call site stays scannable.
No behavior change; the three relevant test suites stay green.
When the proxy-mode mesh peer already holds an open peer-initiated callback
bridge to central, central's Trigger 2 dial (api_token rotation) arrives at
the peer's `/api/mesh/proxy-tunnel` handler with a fresh `mesh_handshake`
frame. The peer's `attachSwitchboard` refuses the new WS at the reverse-dialer
CAS swap, but it was also closing the connection before its `'message'`
listener had been attached, so the handshake frame went unread and the
peer's cached callback JWT stayed at the pre-rotation fingerprint until a
central restart.
Restructures `attachSwitchboard` to attach the `'message'`, `'close'`, and
`'error'` listeners synchronously after the switchboard is created, before
the async MeshService import and the CAS swap run. On the CAS-fail branch,
holds the WS open for up to FIRST_FRAME_WAIT_MS (30ms, well above localhost
RTT and below operator perception) before sending the 1013 close, so an
in-flight first frame lands in `onMessage` and the bootstrap material is
persisted via `MeshCentralRegistry.upsert` regardless of bridge fate. The
success path is untouched; reverseDialer install timing on accepted dials
is unchanged.
Adds a regression test that pre-installs a stub reverse dialer (simulating
a peer-initiated bridge owning the slot), sends a `mesh_handshake` frame
on the new WS, and asserts the registry receives the rotated material
before the 1013 close fires.
* fix(mesh): peer-initiated callback bridge installs reverse dialer
The R1-A2 symmetric-dial work added a peer-side dialer that opens a
callback WS to central at `/api/mesh/proxy-tunnel-from-peer`. The WS
upgrade, JWT validation, and `mesh_centrals` persistence all worked,
but the peer-side handler put the wrong object on its end of the pipe:
a `PilotTunnelBridge` (the central-role object) rather than a
`TcpStreamSwitchboard` with a registered `reverseDialer`.
The design doc states: "Central retains PilotTunnelBridge ownership;
peer retains TcpStreamSwitchboard + reverseDialer ownership." The
central-initiated handler at `meshProxyTunnel.ts:115-163` honors this.
The peer-initiated handler at `PeerToCentralMeshSessionDialer.attachBridge`
did not. It created a `PilotTunnelBridge(0, ws)` and stored it in a
private `currentSession` field. Result: `MeshService.reverseDialer`
stayed null, `dialMeshTcpStream` fell through to
`PilotTunnelManager.ensureBridge(target.nodeId)`, and `NodeRegistry.
getProxyTarget` on the peer returned null because proxy-mode peers
do not enroll their central. Cross-fleet dispatch from a peer
container failed with `proxy-tunnel.open.fail nodeId=<central>
reason=no_target` even though the callback bridge was up and healthy
(`bridgeOpen: true`, `last_used_at` populated).
Replace the peer-side `PilotTunnelBridge` with the same wiring the
central-initiated handler uses:
- `attachTcpStreamSwitchboard` with `resolveByComposeLabels` (allows
inbound `tcp_open` from central if it ever uses the callback
bridge for a reverse-direction dispatch; matches the central-
initiated pipe's resolver).
- A `SwitchboardReverseDialer` that delegates `openMeshTcpStream`
to `switchboard.openReverseStream`.
- `MeshService.setReverseDialer(localDialer, null)` with the same
CAS guard the central-initiated handler uses, so a concurrent
central-initiated tunnel does not get silently overwritten.
- `ws.on('message', onMessage)` dispatches JSON / binary frames to
the switchboard.
- `ws.once('close'/'error', teardown)` clears the switchboard, the
reverseDialer, and the `currentSession` reference idempotently.
The `currentSession` field type changes from `PilotTunnelBridge | null`
to `TcpStreamSwitchboard | null`. Callers in `MeshService.openCrossNode`
only use the return value for truthiness; the cast is harmless. The
public `hasSession()` signature is unchanged.
A new `currentWs` field holds the WS reference so `resetForTest` can
close the connection on teardown (the switchboard's own `cleanup`
does not close its WS).
Test update mirrors the production wiring: the existing
"marks the row used on successful WS open" case now asserts the
return value is a `TcpStreamSwitchboard`, that `MeshService.reverseDialer`
is non-null after `ensureSession`, and explicitly closes the WS in
the test's finally so `wss.close` can resolve.
Pre-existing test logic that asserted `bridge instanceof PilotTunnelBridge`
on the same path is removed in favor of the new switchboard assertion.
Discovered during the v0.81.1 live verification: the peer-side
`route.dispatch cross-node` event always paired with
`proxy-tunnel.open.fail no_target` and `tunnel.fail`, even though
`centralCallback.bridgeOpen: true` and `mesh_centrals.last_used_at`
were populated. The callback path opened cleanly in isolation but
the dispatch path never consumed it. This fix makes the dispatch
path consume it.
Tests:
- cd backend && npx tsc --noEmit clean
- cd backend && npx vitest run mesh 170/170 green
- cd backend && npx vitest run 2259/2268 green; only
failures are the pre-existing Windows EBUSY flake in
filesystem-backup.test.ts (documented in prior handoff)
* fix(mesh): drop unused imports and align reverse-dialer interface name
CI lint failures on the previous push were two @typescript-eslint/no-unused-vars
errors in the test file:
24:76 'vi' is defined but never used
31:5 'TcpStreamSwitchboardCtor' is assigned a value but only used as a type
Both were collateral from a local-debugging simplification: vi.waitFor was
swapped for a synchronous assertion when the test's wss.close hang was traced
to an unclosed client WS; the toBeInstanceOf assertion that used the runtime
binding was removed at the same time. Restore the instanceof assertion next
to the existing not.toBeNull check so the runtime binding is in use and the
test guards against future regressions where the wrong end-of-pipe object
type is returned on the callback bridge (the bug this PR fixes). Drop the
now-unused `vi` symbol from the vitest import.
Also fold in a /simplify convergent finding: rename the local
CallbackReverseDialer interface to SwitchboardReverseDialer to match the
sibling declaration in meshProxyTunnel.ts:72. Same shape, same name; readers
of either handler now see the same mental model. Cross-file extraction of
the interface to tcpStreamSwitchboard.ts is the cleaner long-term shape but
touches a file outside this PR's scope; filing as a follow-up.
Considered-and-deferred findings from the /simplify pass (file as separate
PRs to keep one-branch-one-concern):
- extract shared attachSwitchboard helper (~40 duplicated lines between
meshProxyTunnel.ts and PeerToCentralMeshSessionDialer.ts)
- public TcpStreamSwitchboard.getWs accessor + drop currentWs field
- public MeshService.hasReverseDialer for the test bracket-cast
- shared close-reason constants (drift risk across the two handlers)
Tests:
- cd backend && npx tsc --noEmit clean
- cd backend && npx eslint <changed> clean (lint_exit=0)
- cd backend && npx vitest run mesh 170/170, dialer 8/8 green
PR #930 opened the backend security routes to Community admin but left
several frontend isPaid gates in place, so the matching UI surfaces
stayed hidden from Community even though the API would accept the
request. This brings the UI in line with the backend matrix.
Flipped:
- ResourcesView Scan history button (was trivy.available && isPaid)
- ResourcesView Full scan (vulnerabilities + secrets) menu item
- ResourcesView VulnerabilityScanSheet canCompare
- ResourcesView VulnerabilityScanSheet canManageSuppressions (now isAdmin)
- EditorView stack-level Scan config button (canScan)
- SecurityHistoryView primaryAction (Compare)
- SecurityHistoryView VulnerabilityScanSheet canManageSuppressions
Unchanged on purpose (still paid, matching backend gates):
- canGenerateSbom (SBOM endpoint requirePaid)
- SARIF download in the drawer overflow
- Scan Policies block in Settings
- Trivy auto-update toggle (requireAdmiral)
Test: inverted the SecurityHistoryView.test.tsx assertion that locked
in the now-incorrect "hides Compare on Community" behavior.
The R1-A2 symmetric-callback work added a peer-side recovery branch to
MeshService.openCrossNode: when reverseDialer is null, kick
PeerToCentralMeshSessionDialer.ensureSession so the peer re-opens its
callback WS to central before the cross-node dispatch.
The guard `!this.reverseDialer` did not distinguish "I am a peer waiting
for central to dial in" from "I am central, and the reverseDialer concept
does not apply to me." On central, reverseDialer is always null (central
is the bridge initiator side and never has another node dial into it), so
central entered the recovery branch on every forward dispatch, found no
session (central has no mesh_centrals row), and destroyed the inbound
socket with route.resolve.fail forward-from-peer no_session.
Gate the branch on `MeshCentralRegistry.getActive() !== null` so it only
runs when this Sencho is acting as a proxy-mode peer that has actually
been bootstrapped. Central falls straight through to dialMeshTcpStream.
The peer cold-start scenario the comment originally described is still
covered: a proxy-mode peer with a cached mesh_centrals row but no live
WS to central will fire the recovery branch as before.
Adds two regression tests:
- central path (no mesh_centrals row, no reverseDialer) skips recovery
and calls dialMeshTcpStream
- proxy-peer path (mesh_centrals row + ensureSession returns null)
still emits route.resolve.fail forward-from-peer no_session
Existing tests in the "openCrossNode (BUG-4)" block already used a stub
reverseDialer to bypass the recovery branch, which is why the central
regression was not caught pre-release. The new tests deliberately leave
reverseDialer null to exercise the gating.
sencho-quartermaster[bot]