Commit Graph

7 Commits

Author SHA1 Message Date
Anso f599110386 feat(mesh): collapse sidecar into Sencho process via in-process forwarder (#1000)
The separate saelix/sencho-mesh sidecar container is gone. The
forwarder logic that previously lived in mesh-sidecar/src/forwarder.ts
moves into the Sencho process as backend/src/services/MeshForwarder.ts,
a thin per-port net.Server lifecycle wrapper. MeshService implements
the host interface and owns resolve plus splice; MeshForwarder owns
listener boilerplate. One container per node, no separate image to
publish, no control WebSocket.

Operator-facing change: the Sencho container now runs in
network_mode: host so the forwarder can bind alias ports on the host
network where meshed containers' extra_hosts host-gateway entries
point. Without host network mode the listeners would land in the
container's namespace and inbound traffic from peers would never
reach them. The 1852:1852 port publish becomes a no-op under host
mode and is commented out in the operator template.

Same-node forward path now dials the target container's bridge IP
via Dockerode (preferring the compose default network for
deterministic selection across daemon versions) instead of 127.0.0.1.
The legacy 127.0.0.1 path only worked when the target service
published its port to the host; the IP path works regardless.

Cross-node mesh routing in this phase is central -> pilot direction
only via PilotTunnelManager.openTcpStream. Pilot -> central and
pilot <-> pilot via central relay land in Phase B with the
tcp_open_reverse frame.

Deletions:
- mesh-sidecar/ package entirely (Dockerfile, package, sources, tests)
- backend/src/websocket/meshControl.ts
- MeshService sidecar lifecycle: spawnSidecar, stopSidecar,
  isSidecarRunning, mintSidecarToken, verifySidecarToken,
  attachSidecarSocket, handleSidecarResolve, sendSidecar
- POST /api/mesh/nodes/:id/sidecar/restart route
- /api/mesh/control WS dispatch in upgradeHandler

Type cleanup: 'sidecar' literal removed from MeshActivitySource and
MeshProbeResult.where (also the frontend mirror). MeshNodeStatus
sidecarRunning becomes localForwarderListening (boolean | null) so
non-local nodes get a null instead of an unconditional false; the
honest semantic is "this view only knows the local forwarder state;
remote forwarder status lands in Phase B." MeshNodeDiagnostic
sidecar object becomes forwarder { listening, listenerCount }.

Frontend MeshDiagnosticsSheet drops the restart-sidecar action and
sidecar liveness card; surfaces forwarder state plus a "runs
in-process; no separate container" caption.

Resolves audit findings C-1 (data plane non-functional), C-2 (sidecar
control WS not loopback-enforced), C-4 (sidecar lifecycle Dockerode-
on-remote, PR #999 closed), and C-5 (saelix/sencho-mesh:latest
unreachable). C-3 (PR #992) is unchanged. M-12 (PR #994) is
unchanged.
2026-05-08 15:51:39 -04:00
Anso 3d9489648e fix(pilot): harden outbound reverse-tunnel against resource exhaustion (#979)
* fix(pilot): cap tunnel frame size, concurrent streams, and stream idle time

Pilot tunnels carry every HTTP, WS, and Mesh-TCP byte for a remote node
through a single multiplexed WebSocket. A buggy or compromised peer that
sent oversized frames, opened streams in a tight loop, or left streams
parked indefinitely could exhaust gateway memory.

Three protocol-level limits applied symmetrically on both ends:

  - MAX_FRAME_SIZE_BYTES (8 MB): set as the ws maxPayload on the
    gateway-side WebSocketServer and the agent-side WebSocket client,
    plus a defense-in-depth length check in decodeBinaryFrame and
    decodeJsonFrame.
  - MAX_STREAMS_PER_TUNNEL (1024): bridge refuses new loopback HTTP /
    upgrade / TCP allocations with 503 once at the cap; the agent
    rejects new incoming http_req / ws_open / tcp_open with the
    appropriate error frame.
  - STREAM_IDLE_TIMEOUT_MS (10 min): every stream gets a per-stream
    timer refreshed on each inbound or outbound activity. Expiry tears
    down the local half and notifies the peer.

The constants are colocated in pilot/protocol.ts so any future agent
build picks them up via the protocol module.

* fix(pilot): rate-limit pilot enrollment endpoints to 10 per minute

Pilot enrollment mints a JWT and writes a pilot_enrollments row, both
privileged operations that should not share the global API budget
(200/min). Add a dedicated express-rate-limit instance keyed by user or
IP at 10/min in production (100/min in dev so the local enrollment
test loop is not throttled).

The limiter is applied directly on POST /api/nodes/:id/pilot/enroll.
On POST /api/nodes the limiter's skip function reads the parsed body
and exempts proxy-mode creates; only requests that resolve to
mode=pilot_agent count against the enrollment budget.

* fix(pilot): cap concurrent pilot tunnels system-wide

The PilotTunnelManager held an unbounded Map of bridges. A reconnect
storm, runaway enrollment, or operator misconfiguration could grow the
map without limit, even though every tunnel still carries a valid JWT.

Cap at 256 concurrent tunnels per primary instance:

  - Soft warning at 128 (logged once per crossing).
  - Hard refusal at 256: registerTunnel throws PilotTunnelCapacityError
    and the upgrade handler closes the WebSocket with 1013 (Try Again
    Later) so the agent backs off rather than tight-looping.

A node that already has a registered tunnel does not consume a new
slot when it reconnects; the existing bridge is closed first.

* fix(pilot): release backpressure as soon as the tunnel buffer drains

Previously the bridge paused HTTP request bodies when the tunnel
WebSocket's bufferedAmount climbed above 4 MB but never resumed them
explicitly; the next data event re-checked the threshold, and TCP
streams only saw a 'drain' fan-out on the 30 s ping cycle. Slow-consumer
peers held buffered bytes for tens of seconds longer than needed.

Replace both with an on-demand drain check: when at least one stream
is paused, sample bufferedAmount every 100 ms; once it drops below the
high-water mark, resume every paused request and emit 'drain' to every
accepted TCP stream, then stop the timer. Dormant when no stream is
paused, so steady-state cost is zero.

* fix(pilot): trust internal CAs via SENCHO_PILOT_CA_FILE

Self-hosted deployments often terminate TLS with an internal CA. The
only previous escape hatch was NODE_TLS_REJECT_UNAUTHORIZED=0, which
disables verification across every outbound connection in the agent
process and is the wrong shape of fix.

Add SENCHO_PILOT_CA_FILE: when set, the agent reads the file as a PEM
bundle and passes it to ws as the `ca` option for the tunnel
WebSocket. rejectUnauthorized remains true. Failure to read the file
exits with a clear error so the operator does not silently fall back
to the default trust store.

There is no flag to disable TLS verification entirely; that would
defeat the credential trust model.

* fix(pilot): gate frame-rate-adjacent logs behind developer_mode

The agent's malformed-frame warning fires per inbound frame and could
flood logs under attack or against a buggy primary. The bridge swallowed
parse errors silently, leaving operators blind to protocol drift, and
the manager had no signal at all on tunnel registration.

Route per-frame and per-tunnel diagnostic logs through the existing
isDebugEnabled() helper (same pattern used by middleware/auth, RBAC,
and websocket/logs). Production stays quiet by default; the toggle is
the existing developer_mode setting in global_settings — no new env
var, no new dependency.

* feat(pilot): expose per-tunnel metrics via /api/system/pilot-tunnels

Operator support had no signal beyond raw logs to answer "is this
tunnel flapping?" or "is one bad node hiding behind the aggregate?".
Add an in-process counter set covering tunnels_total, tunnels_replaced,
tunnels_rejected_capacity, enroll_acks, frame_decode_errors, plus a
per-node array carrying the connectedAt and bufferedAmount so a single
tunnel sitting on a stuck buffer remains visible.

Counters live in services/PilotMetrics.ts; the gateway has no shared
in-process metrics facility today, so this is a per-feature pattern
documented as such for future consolidation. Counters are strictly
process-local — no telemetry, no export, no phone-home — consistent
with Sencho's privacy posture.

Surfaced read-only via GET /api/system/pilot-tunnels behind admin auth,
mirroring the existing cache-stats endpoint.

* fix(pilot): tighten reconnect backoff and log handling

Three small follow-ups to the hardening pass:

  - Sanitize the primary URL on the agent's connect log so a malicious
    SENCHO_PRIMARY_URL with embedded control characters cannot inject
    fake log lines.
  - Move the reconnect-backoff reset from the 'open' callback to the
    'hello' frame handler. A peer that always rejects the handshake
    (incompatible version, consumed enrollment token) used to reset
    the backoff on every TCP-level connect and tight-loop reconnects;
    now the reset waits for a clean protocol round-trip.
  - Document the StreamIdAllocator wrap behavior so future readers
    can see the relationship between the 2^31 wrap and the
    MAX_STREAMS_PER_TUNNEL cap without re-deriving it.

* test(pilot): add coverage for enrollment, rate limiter, and tunnel caps

Adds two test files exercising the hardening pass surfaces that were
previously uncovered:

  - pilot-enrollment.test.ts: end-to-end through POST /api/nodes
    (pilot mode), POST /api/nodes/:id/pilot/enroll, the SHA256
    token-hash persistence, replay protection, expired enrollments,
    and the rate-limit header wiring (enrollment limiter on pilot
    paths, global limiter elsewhere).
  - pilot-bridge-limits.test.ts: oversize-frame rejection at both
    binary and JSON decoders, and the per-tunnel concurrent stream
    cap as observed via openTcpStream returning null and the loopback
    HTTP server returning 503 once the bridge is at capacity.

Bridge cap test fills the slot map with TCP stream handles rather than
real HTTP requests so the OS socket pool stays out of the picture.

* test(pilot): cover manager metrics snapshot and capacity-error shape

Adds direct coverage for:

  - PilotTunnelCapacityError exposing the limit so the upgrade
    handler can format a useful close-frame reason.
  - PilotMetrics.snapshot returning a defensive copy (callers must not
    mutate the live counter set).
  - getMetricsSnapshot returning the open count, per-node breakdown,
    and counter set in a stable shape, with tunnels_total bumping on
    a successful registerTunnel.

The manager test seeds a real pilot-mode node row first because
updateNode throws on missing rows; this models the production path
where a node is created before its enrollment is consumed.

* docs(pilot): document tunnel limits, custom CA, and new failure modes

Refresh the Pilot Agent feature doc to cover the hardening surfaces:

  - New 'Self-signed primary TLS certs' section walking through the
    SENCHO_PILOT_CA_FILE env var with an example docker run command.
  - New 'Resource limits' section listing the per-tunnel and
    system-wide ceilings so operators know what the wire enforces.
  - Four new troubleshooting entries: WebSocket close 1013 (system
    cap), loopback 503 (stream cap), close 1002 with protocol error
    (frame size / malformed), and HTTP 429 on enrollment.

Also expand .env.example with a new pilot-agent block covering
SENCHO_MODE, SENCHO_PRIMARY_URL, SENCHO_ENROLL_TOKEN, and
SENCHO_PILOT_CA_FILE so operators do not have to read code or the
feature doc to discover the agent-side config surface.

* fix(pilot): address code-review findings on the hardening pass

Critical:
  - Bridge: TCP backpressure now starts the drain timer and tracks
    streams awaiting drain, so a TcpStream caller waiting on 'drain'
    no longer hangs when no HTTP request is in flight.
  - Bridge: every direct streams.delete call now goes through the
    removeStream helper so per-stream idle timers are cleared
    consistently and pausedReqs / tcpAwaitingDrain stay aligned.
  - Bridge close(): resume any paused IncomingMessage before clearing
    the map so a parser does not stay stuck across teardown.

High:
  - Protocol: decodeJsonFrame now compares Buffer.byteLength(raw,
    'utf8') against the cap; raw.length (UTF-16 code units) let
    multi-byte payloads sneak ~3x the byte budget through.
  - Agent: the optional CA bundle is read once at construction and
    cached on the instance, so a missing or rotated SENCHO_PILOT_CA_FILE
    no longer process-exits on every reconnect attempt.

Medium / Low:
  - Bridge limits test: assert mockWs.sent.length to confirm slot
    allocations actually serialized a tcp_open frame and that a
    rejected (cap+1) attempt did not consume a stream id.
  - Manager: defer the tunnels_replaced increment until after the
    cap check passes, so a rejected reconnect does not double-count
    as both replacement and capacity rejection.
  - Manager: clarify the protocol comment about MAX_FRAME_SIZE_BYTES
    enforcement layering (ws maxPayload is authoritative; decoder
    check is for tests and defense-in-depth).
  - Limiter: defensive `if (!req.body) return false` so a future
    refactor that delays body parsing cannot silently skip the
    enrollment limiter.

Deferred (documented as follow-ups, not in this branch):
  - M2: narrowing public surface of PilotTunnelManager.getBridge to a
    MeshTunnelHandle interface (cross-cutting refactor).
  - H4: end-to-end replay test driving the upgrade handler twice
    with the same enrollment JWT (needs WS test harness).
2026-05-07 21:28:35 -04:00
Anso e5b1c7b22b refactor(backend): collapse entitlement provider abstraction back to LicenseService (#889)
Removes backend/src/entitlements/ (registry, loadProvider,
CommunityEntitlementProvider, types, headers, normalize) and the two
abstraction-only tests. Relocates headers/normalize/types to
services/license-*.ts. Swaps 22 consumer call sites from
getEntitlementProvider() to LicenseService.getInstance(). Drops the
Dockerfile install step plus PRO_PACKAGE_VERSION build-arg and
github_token BuildKit secret in docker-publish.yml. Removes the now
stale no-restricted-imports rule in backend/eslint.config.mjs.

Net: 37 files changed, ~700 lines removed, no behavior change. Local
dev no longer requires GitHub Packages auth to start the backend.

Rationale and revisit conditions in
docs/internal/adrs/2026-05-02-collapse-entitlement-provider.md.
2026-05-02 23:45:44 -04:00
Anso 3324616e59 refactor(backend): extract EntitlementProvider abstraction (Phase 1) (#878)
* refactor(backend): extract EntitlementProvider abstraction (Phase 1)

Phase 1 of the open-core hybrid extraction described in
docs/internal/adrs/2026-05-02-open-core-hybrid-strategy.md. Introduces
the abstraction without moving any code out of the public repo; Phase
2 will actually move services/LicenseService.ts to a private
@studio-saelix/sencho-pro package.

The new backend/src/entitlements/ module contains:

- types.ts. The EntitlementProvider interface plus all tier/license
  types (LicenseTier, LicenseVariant, LicenseInfo, SeatLimits,
  ActivationResult, etc.). The interface mirrors the existing
  LicenseService public surface so the migration was mechanical.

- registry.ts. Module-scope holder for the active provider with
  setEntitlementProvider, getEntitlementProvider, and a test-only
  reset helper. getEntitlementProvider throws if called before
  bootstrap registers a provider; the throw is intentional fail-fast
  on a bootstrap-order bug rather than a silent degradation.

- CommunityEntitlementProvider.ts. Phase 2 fallback that returns
  community tier and rejects activate(). NOT instantiated in
  production today; a smoke test keeps it covered against bitrot.

- loadProvider.ts. Async resolver. Phase 1 returns
  LicenseService.getInstance() directly. The async signature matches
  what Phase 2 needs (dynamic import of @studio-saelix/sencho-pro
  with a "module not found" vs "construction threw" narrowing); the
  call site does not change between phases.

- headers.ts. PROXY_TIER_HEADER and PROXY_VARIANT_HEADER constants.
  These are part of the wire contract between Sencho instances and
  belong in the public core regardless of which entitlement provider
  is bound.

- normalize.ts. isLicenseTier, isLicenseVariant, normalizeTier,
  normalizeVariant. Domain knowledge about Sencho's tier model
  (legacy name maps from pre-0.38.1 versions), not LemonSqueezy
  internals. Phase 2 keeps these in the public core.

services/LicenseService.ts now imports its types from
entitlements/types and adds an "implements EntitlementProvider"
clause. Re-exports the types for back-compat with ~20 type-only
consumers; a follow-up PR will sweep those imports to entitlements/
types directly before Phase 2 deletes the file.

bootstrap/startup.ts awaits loadEntitlementProvider, registers the
result, then calls initialize. shutdown.ts calls
getEntitlementProvider().destroy() instead of the LicenseService
singleton.

middleware/tierGates.ts, the chokepoint for ~154 tier-check call
sites, now reads through getEntitlementProvider. Sixteen other
production files (routes/{fleet,imageUpdates,license,permissions,
scheduledTasks,security,stacks,templates,users,webhooks},
services/{BlueprintService,CloudBackupService,SchedulerService,
SSOService}, proxy/remoteNodeProxy, websocket/{hostConsole,
remoteForwarder}, middleware/auth) had their LicenseService.getInstance
calls and utility-export imports redirected to the entitlements
module. The only remaining LicenseService.getInstance in production
code is in entitlements/loadProvider.ts itself, which is the
intentional Phase-1 binding site.

Test infrastructure: setupTestDb registers
LicenseService.getInstance() as the active provider so existing
test files using the helper need no changes. The mocking pattern
many tests use, vi.spyOn(LicenseService.getInstance(), 'getTier'),
keeps working because LicenseService.getInstance() and
getEntitlementProvider() return the same singleton in Phase 1.
scheduler-service.test.ts is the only test that does not use
setupTestDb but exercises tier-gating; it now mocks
entitlements/registry alongside its existing LicenseService mock.

Adds a smoke test for CommunityEntitlementProvider so the Phase 2
fallback class stays covered.

Adds an architecture doc at
docs/internal/architecture/entitlement-provider.md covering the
runtime registry, bootstrap order invariants, and the Phase 1 vs
Phase 2 binding table.

Test results: 89/89 backend test files pass, 1657 passing tests, 5
pre-existing skips. The pre-existing database-metrics > handles
1000+ metrics stress test continues to flake under parallel load
and pass when re-run solo, same flake observed in PRs #862, #863.

* chore(backend): drop unused entitlement type imports from LicenseService

Phase 1 of the EntitlementProvider extraction left five type imports
(ActivationResult, BillingPortalError, BillingPortalResult,
DeactivationResult, ValidationResult) unreferenced after the runtime
methods that produced them began inferring their result shapes via the
EntitlementProvider interface contract. ESLint's no-unused-vars rule
flagged them as errors and failed the lint step in CI.
2026-05-02 05:07:00 -04:00
Anso 7663f4cd8b feat(fleet): sencho mesh in traffic and routing tab (#858)
* feat(fleet): sencho mesh in traffic and routing tab

Lights up Sencho Mesh: cross-node container forwarding rendered as if the
container next to you were on localhost. Builds on the dormant TCP frame
plumbing from the prior PR (pilot tunnel TCP frames + sencho-mesh sidecar
package) and exposes the Admiral-only orchestrator surface.

Backend
- New mesh_stacks table (per-node opt-ins) + nodes.mesh_enabled column
  via DatabaseService.migrateMeshTables.
- MeshService singleton: sidecar lifecycle via Dockerode, opt-in/out with
  cascading override regeneration, request-based resolver from sidecar
  control WS, cross-node TCP forwarding via PilotTunnelManager (same-node
  fast path included), in-memory 1000-event activity ring buffer with
  durable mirror to audit_log for state-change events, per-node and
  per-route diagnostics, and the Test upstream probe.
- MeshComposeOverride: pure YAML generator that injects extra_hosts using
  host-gateway. The user's docker-compose.yml is never mutated; overrides
  live under DATA_DIR/mesh/overrides.
- ComposeService deploy/update splice the override file when the stack
  is opted in; non-mesh stacks behave identically to today.
- Pilot agent resolveMeshTarget consults the local mesh_stacks table
  (defense in depth) and resolves Compose containers via Dockerode.
- /api/mesh router with 13 Admiral-gated endpoints covering status,
  enable/disable, stack opt-in/out, alias listing, per-route diagnostic,
  Test upstream probe, per-node diagnostic, sidecar restart, activity
  log paginated and SSE.
- meshControl WS slot at /api/mesh/control validates the mesh_sidecar
  JWT minted by MeshService; dispatched as upgrade slot 2 (canonical
  order preserved).

Frontend
- New Traffic Routing tab in FleetView, gated by isAdmiral and wrapped
  in AdmiralGate. Tab uses the cyan brand glyph and italic-serif state
  typography from the audit.
- RoutingTab masthead with mesh activity drawer, per-node card grid
  with TogglePill, alias rows with five-state pill taxonomy
  (healthy / degraded / unreachable / tunnel-down / not-authorized),
  inline Test buttons.
- Four sheets: opt-in picker with port-collision inline error,
  per-route detail with diagnostic + filtered activity, per-node
  diagnostics with active streams + resolver cache + restart action,
  fleet-wide activity log with filters.
- meshRouteState helper centralizes pill-state mapping; pure-function
  tests cover all five states.

Docs
- User docs at /docs/features/sencho-mesh.mdx covering opt-in,
  troubleshooting, security model (4 guarantees + 4 explicit
  non-guarantees), and V1 limitations.
- Internal architecture and runbook pages.
- websocket-dispatch internal doc updated with the new slot.

* fix(mesh): validate stack name before path use; fix test DB lifecycle

Two surgical fixes against the prior PR.

Path-injection (CodeQL js/path-injection): MeshService.optInStack,
optOutStack, ensureStackOverride, and removeStackOverride now validate
stackName via isValidStackName from utils/validation, reject malicious
names at the API boundary, and additionally check isPathWithinBase on
the resolved override file path for defense in depth. The dataflow from
req.params.stackName to fs.writeFile no longer reaches an unsanitized
path expression.

Test DB lifecycle: mesh-service.test.ts used per-test setupTestDb /
cleanupTestDb, which deletes the temp dir while DatabaseService still
holds an open SQLite handle. On Linux CI this raises
SQLITE_READONLY_DBMOVED on the next prepare() because the inode has
been unlinked. Switched to file-scoped beforeAll/afterAll matching
agents-routes.test.ts, with a per-test beforeEach that truncates
mesh_stacks plus non-default nodes and resets the MeshService singleton
in-memory state. Adds a new test case asserting the path-traversal
rejection.

* fix(compose): use discovered compose filename instead of hardcoded docker-compose.yml

composeArgs() hardcoded `-f docker-compose.yml` for every deploy. Sencho
writes its canonical compose file as `compose.yaml`, so any stack created
via the UI failed to deploy with `open ...docker-compose.yml: no such
file or directory`.

When no mesh override applies, drop the explicit `-f` so docker compose's
built-in discovery resolves the actual filename. When an override exists,
look up the real base filename via FileSystemService.getComposeFilename()
and pass both files explicitly.

Also hoist the MeshService import to module top now that the dependency
is known to be acyclic, and revert the matching unit-test assertion.
2026-05-01 01:50:53 -04:00
Anso 61a7e43d82 perf(proxy): cache LicenseService tier headers for the proxy hot path (#815)
The remote-node HTTP proxy and WebSocket forwarder read getTier() +
getVariant() on every forwarded request to set the Distributed License
Enforcement headers. Each call hits system_state 5+ times. Add a
30-second cached snapshot inside LicenseService and route every
license_status write through a new private setLicenseStatus() helper
so activate, deactivate, validate, and the auto-demote paths inside
getTier() all invalidate the cache.

Routing all license_status writes through one chokepoint also closes
a latent drift window: the self-heal paths in getTier() (trial
expired, offline grace exceeded, subscription expired) used to mutate
state silently and now invalidate the cache the same way explicit
license events do.

The TTL becomes a safety net against any future write that bypasses
the helper, not a load-bearing freshness bound. Existing 44 license
and distributed-license tests pass unchanged.
2026-04-28 00:13:07 -04:00
Anso dc3699189d refactor(backend): extract remote proxy, WebSocket upgrade handler, and server factory (phase 3) (#733)
Phase 3 of the index.ts refactor. Pulls the remote HTTP/WS proxy plumbing,
the WebSocket upgrade dispatcher, and the http/WSS construction out of the
monolith. index.ts drops roughly 620 lines.

New modules:
- proxy/websocketProxy.ts: shared httpProxy.createProxyServer singleton
  (used by both the HTTP proxy middleware and the remote WS forwarder)
- proxy/remoteNodeProxy.ts: createRemoteProxyMiddleware() factory; consumes
  the isProxyExemptPath helper instead of open-coding the prefix list
- server.ts: createServer(app) returns { server, wss, pilotTunnelWss }
- services/FleetUpdateTrackerService.ts: singleton wrapping the in-flight
  fleet update tracker Map with create()/resolve() helpers
- helpers/consoleSession.ts: mintConsoleSession(), isConsoleSessionScope()
- websocket/upgradeHandler.ts: attachUpgrade(server, deps) dispatcher that
  runs the manual cookie/JWT verify and delegates to sub-handlers
- websocket/pilotTunnel.ts: handlePilotTunnel (pilot_enroll consumption and
  pilot_tunnel registration)
- websocket/notifications.ts: /ws/notifications local subscriber
- websocket/remoteForwarder.ts: remote-node WS proxy with console_session
  token exchange for interactive paths
- websocket/logs.ts: /api/stacks/:name/logs supervisor stream
- websocket/hostConsole.ts: /api/system/host-console PTY, Admiral-gated
- websocket/generic.ts: /ws exec + streamStats action dispatch, owns the
  terminalWs single-instance reference
- websocket/reject.ts: shared rejectUpgrade helper (replaces five copies)

Service extension:
- NotificationService: setBroadcaster(fn) replaced by subscribe(ws) that
  returns an unsubscriber; broadcastToSubscribers is now internal. Subscriber
  set lives on the service rather than in index.ts.

Wiring in index.ts:
- const app = createApp() already in place from Phase 2
- const { server, wss, pilotTunnelWss } = createServer(app)
- attachUpgrade(server, { wss, pilotTunnelWss })
- app.use('/api/', createRemoteProxyMiddleware())
- /api/system/console-token route now uses mintConsoleSession()
- deploy/down/update routes read the streaming target via getTerminalWs()
  (return type is WebSocket | undefined so the || undefined fallback is gone)

Code review fixes: five duplicated reject helpers collapsed into
websocket/reject.ts; dropped the createTracker/resolveTracker bind
aliases in index.ts so call sites go through the service directly;
removed em dashes; replaced req.url! with req.url || '/'.
2026-04-23 19:31:16 -04:00