mirror of
https://github.com/Studio-Saelix/sencho.git
synced 2026-07-26 11:49:16 +00:00
Anso
3d9489648e
* fix(pilot): cap tunnel frame size, concurrent streams, and stream idle time
Pilot tunnels carry every HTTP, WS, and Mesh-TCP byte for a remote node
through a single multiplexed WebSocket. A buggy or compromised peer that
sent oversized frames, opened streams in a tight loop, or left streams
parked indefinitely could exhaust gateway memory.
Three protocol-level limits applied symmetrically on both ends:
- MAX_FRAME_SIZE_BYTES (8 MB): set as the ws maxPayload on the
gateway-side WebSocketServer and the agent-side WebSocket client,
plus a defense-in-depth length check in decodeBinaryFrame and
decodeJsonFrame.
- MAX_STREAMS_PER_TUNNEL (1024): bridge refuses new loopback HTTP /
upgrade / TCP allocations with 503 once at the cap; the agent
rejects new incoming http_req / ws_open / tcp_open with the
appropriate error frame.
- STREAM_IDLE_TIMEOUT_MS (10 min): every stream gets a per-stream
timer refreshed on each inbound or outbound activity. Expiry tears
down the local half and notifies the peer.
The constants are colocated in pilot/protocol.ts so any future agent
build picks them up via the protocol module.
* fix(pilot): rate-limit pilot enrollment endpoints to 10 per minute
Pilot enrollment mints a JWT and writes a pilot_enrollments row, both
privileged operations that should not share the global API budget
(200/min). Add a dedicated express-rate-limit instance keyed by user or
IP at 10/min in production (100/min in dev so the local enrollment
test loop is not throttled).
The limiter is applied directly on POST /api/nodes/:id/pilot/enroll.
On POST /api/nodes the limiter's skip function reads the parsed body
and exempts proxy-mode creates; only requests that resolve to
mode=pilot_agent count against the enrollment budget.
* fix(pilot): cap concurrent pilot tunnels system-wide
The PilotTunnelManager held an unbounded Map of bridges. A reconnect
storm, runaway enrollment, or operator misconfiguration could grow the
map without limit, even though every tunnel still carries a valid JWT.
Cap at 256 concurrent tunnels per primary instance:
- Soft warning at 128 (logged once per crossing).
- Hard refusal at 256: registerTunnel throws PilotTunnelCapacityError
and the upgrade handler closes the WebSocket with 1013 (Try Again
Later) so the agent backs off rather than tight-looping.
A node that already has a registered tunnel does not consume a new
slot when it reconnects; the existing bridge is closed first.
* fix(pilot): release backpressure as soon as the tunnel buffer drains
Previously the bridge paused HTTP request bodies when the tunnel
WebSocket's bufferedAmount climbed above 4 MB but never resumed them
explicitly; the next data event re-checked the threshold, and TCP
streams only saw a 'drain' fan-out on the 30 s ping cycle. Slow-consumer
peers held buffered bytes for tens of seconds longer than needed.
Replace both with an on-demand drain check: when at least one stream
is paused, sample bufferedAmount every 100 ms; once it drops below the
high-water mark, resume every paused request and emit 'drain' to every
accepted TCP stream, then stop the timer. Dormant when no stream is
paused, so steady-state cost is zero.
* fix(pilot): trust internal CAs via SENCHO_PILOT_CA_FILE
Self-hosted deployments often terminate TLS with an internal CA. The
only previous escape hatch was NODE_TLS_REJECT_UNAUTHORIZED=0, which
disables verification across every outbound connection in the agent
process and is the wrong shape of fix.
Add SENCHO_PILOT_CA_FILE: when set, the agent reads the file as a PEM
bundle and passes it to ws as the `ca` option for the tunnel
WebSocket. rejectUnauthorized remains true. Failure to read the file
exits with a clear error so the operator does not silently fall back
to the default trust store.
There is no flag to disable TLS verification entirely; that would
defeat the credential trust model.
* fix(pilot): gate frame-rate-adjacent logs behind developer_mode
The agent's malformed-frame warning fires per inbound frame and could
flood logs under attack or against a buggy primary. The bridge swallowed
parse errors silently, leaving operators blind to protocol drift, and
the manager had no signal at all on tunnel registration.
Route per-frame and per-tunnel diagnostic logs through the existing
isDebugEnabled() helper (same pattern used by middleware/auth, RBAC,
and websocket/logs). Production stays quiet by default; the toggle is
the existing developer_mode setting in global_settings — no new env
var, no new dependency.
* feat(pilot): expose per-tunnel metrics via /api/system/pilot-tunnels
Operator support had no signal beyond raw logs to answer "is this
tunnel flapping?" or "is one bad node hiding behind the aggregate?".
Add an in-process counter set covering tunnels_total, tunnels_replaced,
tunnels_rejected_capacity, enroll_acks, frame_decode_errors, plus a
per-node array carrying the connectedAt and bufferedAmount so a single
tunnel sitting on a stuck buffer remains visible.
Counters live in services/PilotMetrics.ts; the gateway has no shared
in-process metrics facility today, so this is a per-feature pattern
documented as such for future consolidation. Counters are strictly
process-local — no telemetry, no export, no phone-home — consistent
with Sencho's privacy posture.
Surfaced read-only via GET /api/system/pilot-tunnels behind admin auth,
mirroring the existing cache-stats endpoint.
* fix(pilot): tighten reconnect backoff and log handling
Three small follow-ups to the hardening pass:
- Sanitize the primary URL on the agent's connect log so a malicious
SENCHO_PRIMARY_URL with embedded control characters cannot inject
fake log lines.
- Move the reconnect-backoff reset from the 'open' callback to the
'hello' frame handler. A peer that always rejects the handshake
(incompatible version, consumed enrollment token) used to reset
the backoff on every TCP-level connect and tight-loop reconnects;
now the reset waits for a clean protocol round-trip.
- Document the StreamIdAllocator wrap behavior so future readers
can see the relationship between the 2^31 wrap and the
MAX_STREAMS_PER_TUNNEL cap without re-deriving it.
* test(pilot): add coverage for enrollment, rate limiter, and tunnel caps
Adds two test files exercising the hardening pass surfaces that were
previously uncovered:
- pilot-enrollment.test.ts: end-to-end through POST /api/nodes
(pilot mode), POST /api/nodes/:id/pilot/enroll, the SHA256
token-hash persistence, replay protection, expired enrollments,
and the rate-limit header wiring (enrollment limiter on pilot
paths, global limiter elsewhere).
- pilot-bridge-limits.test.ts: oversize-frame rejection at both
binary and JSON decoders, and the per-tunnel concurrent stream
cap as observed via openTcpStream returning null and the loopback
HTTP server returning 503 once the bridge is at capacity.
Bridge cap test fills the slot map with TCP stream handles rather than
real HTTP requests so the OS socket pool stays out of the picture.
* test(pilot): cover manager metrics snapshot and capacity-error shape
Adds direct coverage for:
- PilotTunnelCapacityError exposing the limit so the upgrade
handler can format a useful close-frame reason.
- PilotMetrics.snapshot returning a defensive copy (callers must not
mutate the live counter set).
- getMetricsSnapshot returning the open count, per-node breakdown,
and counter set in a stable shape, with tunnels_total bumping on
a successful registerTunnel.
The manager test seeds a real pilot-mode node row first because
updateNode throws on missing rows; this models the production path
where a node is created before its enrollment is consumed.
* docs(pilot): document tunnel limits, custom CA, and new failure modes
Refresh the Pilot Agent feature doc to cover the hardening surfaces:
- New 'Self-signed primary TLS certs' section walking through the
SENCHO_PILOT_CA_FILE env var with an example docker run command.
- New 'Resource limits' section listing the per-tunnel and
system-wide ceilings so operators know what the wire enforces.
- Four new troubleshooting entries: WebSocket close 1013 (system
cap), loopback 503 (stream cap), close 1002 with protocol error
(frame size / malformed), and HTTP 429 on enrollment.
Also expand .env.example with a new pilot-agent block covering
SENCHO_MODE, SENCHO_PRIMARY_URL, SENCHO_ENROLL_TOKEN, and
SENCHO_PILOT_CA_FILE so operators do not have to read code or the
feature doc to discover the agent-side config surface.
* fix(pilot): address code-review findings on the hardening pass
Critical:
- Bridge: TCP backpressure now starts the drain timer and tracks
streams awaiting drain, so a TcpStream caller waiting on 'drain'
no longer hangs when no HTTP request is in flight.
- Bridge: every direct streams.delete call now goes through the
removeStream helper so per-stream idle timers are cleared
consistently and pausedReqs / tcpAwaitingDrain stay aligned.
- Bridge close(): resume any paused IncomingMessage before clearing
the map so a parser does not stay stuck across teardown.
High:
- Protocol: decodeJsonFrame now compares Buffer.byteLength(raw,
'utf8') against the cap; raw.length (UTF-16 code units) let
multi-byte payloads sneak ~3x the byte budget through.
- Agent: the optional CA bundle is read once at construction and
cached on the instance, so a missing or rotated SENCHO_PILOT_CA_FILE
no longer process-exits on every reconnect attempt.
Medium / Low:
- Bridge limits test: assert mockWs.sent.length to confirm slot
allocations actually serialized a tcp_open frame and that a
rejected (cap+1) attempt did not consume a stream id.
- Manager: defer the tunnels_replaced increment until after the
cap check passes, so a rejected reconnect does not double-count
as both replacement and capacity rejection.
- Manager: clarify the protocol comment about MAX_FRAME_SIZE_BYTES
enforcement layering (ws maxPayload is authoritative; decoder
check is for tests and defense-in-depth).
- Limiter: defensive `if (!req.body) return false` so a future
refactor that delays body parsing cannot silently skip the
enrollment limiter.
Deferred (documented as follow-ups, not in this branch):
- M2: narrowing public surface of PilotTunnelManager.getBridge to a
MeshTunnelHandle interface (cross-cutting refactor).
- H4: end-to-end replay test driving the upgrade handler twice
with the same enrollment JWT (needs WS test harness).
120 lines
4.4 KiB
Bash
120 lines
4.4 KiB
Bash
# Sencho Configuration
|
|
# Copy this file to .env and update the values for production
|
|
|
|
# JWT secret - generate a secure random string for production
|
|
JWT_SECRET=your-secure-jwt-secret-here
|
|
|
|
# Directory containing docker-compose files
|
|
COMPOSE_DIR=/path/to/your/compose/files
|
|
|
|
# HTTP server port (default: 1852)
|
|
PORT=1852
|
|
|
|
# Database and state directory inside the container (default: /app/data)
|
|
DATA_DIR=/app/data
|
|
|
|
# Optional path to a host-installed Trivy binary. Sencho first looks for a
|
|
# managed install under DATA_DIR/bin/trivy; if absent and TRIVY_BIN is set,
|
|
# Sencho uses that path; otherwise it falls back to `trivy` on PATH. Once a
|
|
# managed install is present, the managed copy takes precedence over this.
|
|
TRIVY_BIN=
|
|
|
|
# Node environment (set automatically in Docker image; only change for local dev)
|
|
NODE_ENV=production
|
|
|
|
# Frontend URL for CORS in production (leave empty for same-origin setups)
|
|
FRONTEND_URL=
|
|
|
|
# Global API rate limit (requests per minute per user session, production only)
|
|
# Authenticated requests are keyed by user ID; unauthenticated by IP.
|
|
# Internal node-to-node traffic (node_proxy tokens) bypasses this limit entirely.
|
|
API_RATE_LIMIT=200
|
|
|
|
# Polling endpoint rate limit (requests per minute, production only)
|
|
# Applies to dashboard/status polling endpoints that are exempt from the global limit.
|
|
# Increase for environments with many concurrent browser sessions behind shared NAT.
|
|
API_POLLING_RATE_LIMIT=300
|
|
|
|
# ─── Pilot agent (remote host only) ──────────────────────────────
|
|
# These three vars are required ONLY on a remote host running as a
|
|
# pilot-agent reverse-tunnel container. The primary instance does not
|
|
# read them. See docs/features/pilot-agent.mdx for the full setup.
|
|
|
|
# Switches the container into agent mode. The primary instance leaves
|
|
# this unset.
|
|
SENCHO_MODE=
|
|
|
|
# WebSocket-capable URL of the controlling Sencho instance. Use https://
|
|
# scheme; the agent rewrites it to wss:// for the tunnel upgrade.
|
|
SENCHO_PRIMARY_URL=
|
|
|
|
# Single-use, 15-minute enrollment token issued by the primary when the
|
|
# pilot-agent node is created. After the first successful connect the
|
|
# agent persists a long-lived tunnel credential at /app/data/pilot.jwt
|
|
# and ignores this var on subsequent restarts.
|
|
SENCHO_ENROLL_TOKEN=
|
|
|
|
# Optional: path inside the agent container to a PEM CA bundle the
|
|
# agent should trust when validating the primary's TLS cert. Use this
|
|
# for self-hosted deployments terminating TLS with an internal CA.
|
|
# Leave unset to fall back to the system trust store. There is no
|
|
# escape hatch to disable TLS verification.
|
|
SENCHO_PILOT_CA_FILE=
|
|
|
|
# ─── SSO / LDAP Configuration ────────────────────────────────────
|
|
|
|
# LDAP / Active Directory
|
|
SSO_LDAP_ENABLED=false
|
|
SSO_LDAP_URL=ldap://ldap.example.com:389
|
|
SSO_LDAP_BIND_DN=cn=readonly,dc=example,dc=com
|
|
SSO_LDAP_BIND_PASSWORD=
|
|
SSO_LDAP_SEARCH_BASE=ou=users,dc=example,dc=com
|
|
SSO_LDAP_SEARCH_FILTER=(uid={{username}})
|
|
SSO_LDAP_ADMIN_GROUP_DN=
|
|
SSO_LDAP_DEFAULT_ROLE=viewer
|
|
SSO_LDAP_DISPLAY_NAME=LDAP
|
|
SSO_LDAP_TLS_REJECT_UNAUTHORIZED=true
|
|
|
|
# Google OIDC
|
|
SSO_OIDC_GOOGLE_ENABLED=false
|
|
SSO_OIDC_GOOGLE_CLIENT_ID=
|
|
SSO_OIDC_GOOGLE_CLIENT_SECRET=
|
|
|
|
# GitHub OAuth
|
|
SSO_OIDC_GITHUB_ENABLED=false
|
|
SSO_OIDC_GITHUB_CLIENT_ID=
|
|
SSO_OIDC_GITHUB_CLIENT_SECRET=
|
|
|
|
# Okta OIDC
|
|
SSO_OIDC_OKTA_ENABLED=false
|
|
SSO_OIDC_OKTA_ISSUER_URL=
|
|
SSO_OIDC_OKTA_CLIENT_ID=
|
|
SSO_OIDC_OKTA_CLIENT_SECRET=
|
|
|
|
# Custom OIDC (Keycloak, Authentik, Authelia, Zitadel, etc.)
|
|
SSO_OIDC_CUSTOM_ENABLED=false
|
|
SSO_OIDC_CUSTOM_DISPLAY_NAME=Custom OIDC
|
|
SSO_OIDC_CUSTOM_ISSUER_URL=
|
|
SSO_OIDC_CUSTOM_CLIENT_ID=
|
|
SSO_OIDC_CUSTOM_CLIENT_SECRET=
|
|
SSO_OIDC_CUSTOM_SCOPES=openid email profile
|
|
SSO_OIDC_CUSTOM_ID_CLAIM=
|
|
SSO_OIDC_CUSTOM_USERNAME_CLAIM=
|
|
SSO_OIDC_CUSTOM_EMAIL_CLAIM=
|
|
|
|
# Role mapping (shared across OIDC providers)
|
|
SSO_OIDC_ADMIN_CLAIM=groups
|
|
SSO_OIDC_ADMIN_CLAIM_VALUE=sencho-admins
|
|
SSO_DEFAULT_ROLE=viewer
|
|
|
|
# External base URL for OAuth callback URLs (required behind reverse proxy)
|
|
SSO_CALLBACK_URL=
|
|
|
|
# Experimental UI surfaces. When unset or any value other than "true",
|
|
# the UI hides surfaces that are kept in the repo but not yet promoted
|
|
# to the default 1.0 build (Fleet Traffic · Routing tab, Fleet
|
|
# Deployments / Blueprints tab, Fleet Federation tab, Fleet Secrets
|
|
# sync tab, Fleet Actions tab). Backend routes for these surfaces stay
|
|
# live regardless. This flag controls UI discovery only.
|
|
SENCHO_EXPERIMENTAL=false
|