mirror of
https://github.com/Studio-Saelix/sencho.git
synced 2026-08-16 21:48:45 +00:00
d369b03a38e19d1e2fa7ee98f4e6738fce5da3ef
16 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Anso
|
865d792874 |
feat(pricing): collapse to two tiers (#1309)
* feat(pricing): collapse to two tiers (Community + Admiral) Collapse Sencho's pricing from three tiers (Community / Skipper / Admiral) to two: a generous free Community tier and a single paid Admiral tier. The Skipper tier is removed. Now free in Community: auto-heal, auto-update, scheduled operations, webhooks, notification routing, Fleet Actions and bulk operations, SSO preset providers (Google / GitHub / Okta), unlimited users with admin and viewer roles, and deploy safety (atomic deploys, auto-rollback, and one-click rollback). Admiral (paid) is focused on running and governing a fleet: blueprints, Fleet Secrets, deploy enforcement, vulnerability report export, audit log, host console, private registries, mesh networking, node cordon, managed cloud backup, LDAP / Active Directory SSO, and the advanced RBAC roles (deployer, node-admin, auditor) with per-resource scoped assignments. Internally the license variant distinction is removed so tier is binary (community / paid). License validation still verifies the Lemon Squeezy store and product before granting paid status. Docs and the contributor guide are updated to the two-tier model. * docs(pricing): correct licensing page to two-tier pricing and tidy stale tier wording The licensing docs page kept the old Admiral pricing plus a Founder Lifetime column and an Enterprise paragraph after the two-tier collapse. Update it to $12/month or $99/year, drop the lifetime and Enterprise content, and link to the pricing page for current pricing. Also fix stale "Skipper" wording in CLA.md, SUPPORT.md, one test title, and three test comments. Historical CHANGELOG entries and the retired-Skipper license-guard test are intentionally left as-is. * docs: align licensing and SSO pages with the two-tier model Correct the SSO overview so the Google, GitHub, and Okta presets read as available on every tier, matching the provider table; only LDAP and Active Directory require Sencho Admiral. Remove the lifetime-plan references from the licensing, settings, and troubleshooting pages so they reflect subscription-only Admiral pricing. * fix(rbac): omit scoped permissions from /me on the Community tier Scoped role assignments only take effect on the paid tier, but GET /api/permissions/me returned them unconditionally, so a downgraded instance with leftover assignments rendered per-resource affordances the API then rejected with 403. The endpoint now mirrors the permission middleware and includes scoped permissions only on the paid tier. Adds a regression test covering the downgrade case. * docs: use custom-pricing wording on the contact page The two-tier model has no Enterprise tier; reword the contact page's enterprise pricing/deals to custom pricing/deals so it does not imply a tier that no longer exists. |
||
|
Anso
|
65a69d9ecc |
fix(nodes): never send stored node tokens to clients (#1281)
Node read endpoints now return a client-safe projection that omits the stored api_token and exposes a has_token boolean instead, so a node's long-lived proxy credential is never serialized to a browser or API token client. The token stays encrypted at rest and is read server-side only by the components that need it (the remote proxy, the connection test, and the mesh dialer). The edit form opens the API Token field blank, and a blank value keeps the existing credential, so saving an edit without retyping the token no longer clears it; a non-empty value rotates it. The backend enforces the same rule defensively. Node management actions (add, edit, delete, test connection, generate node token) are gated in the UI to match their server-side permission checks, so operators no longer see an action the API would reject. The test-connection route also gains the missing server-side permission and token-scope guards. Also validate the x-node-id header and fall back to the default node for malformed values instead of an obscure 404, and return 400 (not 500) when deleting the default node. |
||
|
Anso
|
35a1182890 |
fix(nodes): gate node-management actions by role and release pilot tunnels on delete (#1280)
* fix(nodes): gate node-management actions by role and release pilot tunnels on delete The node-management write actions in the Nodes panel (add, edit, delete, generate node token, reset fleet-sync anchor) rendered for every signed-in role, but the API enforces the manage-nodes permission on them, so lower-privilege roles saw buttons that returned 403. The panel now renders each action against the same permission its route enforces; the read-only node table stays visible to every role. Deleting a node now also tears down its live pilot-agent tunnel (and any mesh bridge) immediately, releasing the loopback server, heartbeat timer, and open streams instead of leaving them until the agent next disconnects, matching the cleanup the re-enrollment path already performed. Adds backend route tests for the permission boundaries and tunnel teardown, and a Nodes panel render test covering the viewer, admin, and node-admin views. * fix(nodes): close proxy mesh bridges via the dialer on node delete to skip a redial Deleting a node closed any active mesh bridge through PilotTunnelManager, but a proxy-mode bridge is owned by the mesh dialer, whose close listener then treated the close as unexpected and scheduled a reactive redial against the node being removed. The delete handler now closes a proxy bridge through the dialer's intentional-close path first (which suppresses the redial), then closes a pilot-agent tunnel as before. Adds a backend test that primes a live proxy bridge and asserts deletion closes it without scheduling a redial. |
||
|
Anso
|
dd2c2b22ec |
fix(federation): gate node cordon control on node:manage to match backend (#1277)
* fix(federation): gate node cordon control on node:manage to match backend
The cordon and uncordon control on the node card rendered whenever the
instance held an Admiral license, but the backend route also requires the
node:manage permission. Non-admin users without that permission (deployer,
viewer, auditor) saw a Cordon action the API rejected with 403. Gate the
control on the same permission the route enforces, so it renders only for
users who can use it.
Add developer-mode diagnostic logging to the cordon and pin handlers, and
route-level tests covering the cordon and uncordon permission, tier,
API-token, and input-validation boundaries.
* fix(federation): reject non-numeric node ids in cordon/uncordon
The cordon and uncordon routes validated the node id with parseInt, which
accepts numeric-prefix strings (parseInt('1abc', 10) === 1), so a request to
/api/nodes/1abc/cordon would operate on node 1. Require a strict positive
integer before parsing, and add tests for both routes.
* fix(federation): sanitize user-derived values in cordon and pin diagnostics
Route the node id, blueprint id, target node id, and reason length through
the shared log sanitizer before they reach the developer-mode diagnostic
log lines, and switch the format specifiers to %s to match. The values are
already validated integers, so this is defense in depth at the log sink and
keeps the diagnostics on the same sanitize-every-interpolated-value pattern
used elsewhere. No runtime behavior change: the lines stay gated behind the
developer_mode setting, off by default.
|
||
|
Anso
|
d03d97d964 |
fix(nodes): close capability-gating gaps in node compatibility (#1261)
* fix(nodes): close capability-gating gaps in node compatibility Vulnerability scanning is now gated correctly on whether the active node advertises support for it: - A node without the Trivy binary stops advertising the scanning capability. Previously the capability was toggled only on a state change, so a node that booted without Trivy kept advertising scanning it could not perform. - The control node's own capability list now reflects features disabled at runtime, matching what it advertises to peers. - The scan history surface shows a clear "not available on this node" card, with its header actions hidden, instead of attempting a request that fails. A node's version and capability metadata now refreshes immediately after a connection test or a completed update, rather than waiting out the cache. Capability gates fail closed to the unavailable card when a node's metadata request errors, instead of staying open until the next fetch. Adds a test that fails if the frontend and backend capability lists drift, plus coverage for the metadata error path, the runtime-disabled local meta, the scanning capability sync, and the metadata cache invalidation paths. * fix(nodes): refresh node metadata client-side after a connection test A connection test dropped the server-side metadata cache, but the dashboard kept its own cached copy until the client TTL expired, so version and capability gates could stay stale in the browser. The test now forces a client-side metadata refresh for that node, so the version pill and gates reflect the node's current state immediately. Also strips any URL userinfo before logging the metadata fetch target, and makes the scanning-capability detection test deterministically exercise the no-binary disable path rather than depending on whether the runner has Trivy. |
||
|
Anso
|
282ab8d844 |
fix(pilot): let SENCHO_PUBLIC_URL override the request Host in enrollment (#1122)
The enrollment minter inferred SENCHO_PRIMARY_URL from the request Host header, which baked loopback or LAN addresses into the compose YAML when the admin opened Add Node on the central's own machine. Pilots on a different network (a public cloud VPS, for example) cannot dial that. SENCHO_PUBLIC_URL on the primary now wins when set and well-formed (http(s)://, no loopback). Trailing slashes are stripped. Falls back to the request Host when unset or invalid. |
||
|
Anso
|
3ad6ea9c5d |
feat(pilot): make Docker Compose the canonical pilot enrollment payload (#1121)
* feat(pilot): make Docker Compose the canonical pilot enrollment payload The Add Node dialog for a pilot-agent now returns a Compose snippet instead of a single-line docker run command, and the enrollment dialog walks the operator through a save-and-up flow. The compose project name and container name align with what SelfUpdateService looks up at boot, so a Compose-deployed pilot can be updated remotely through the Fleet view without intervention on the remote host. Docs (pilot-agent, remote-updates) were rewritten to match. * test(e2e): align pilot enrollment spec with Compose payload The spec was written against the docker-run payload; it now asserts the Compose YAML the dialog renders. |
||
|
Anso
|
e05099f2a1 |
fix(fleet-sync): make control-identity-mismatch sticky and surface in UI (#1117)
Treat 409 CONTROL_IDENTITY_MISMATCH from a replica as a non-retriable
failure instead of looping the same 409 through the 5-minute retry
service forever and silently writing identical failure rows.
Backend
- DatabaseService: add `sticky_error_code`, `sticky_error_expected`,
`sticky_error_got` columns to `fleet_sync_status` via an idempotent
migration. New methods setFleetSyncSticky, getFleetSyncStickyCode,
clearFleetSyncStickyForNode. recordFleetSyncSuccess clears the sticky
flag on a clean push. getFailedSyncTargets SQL adds
`AND sticky_error_code IS NULL` so the retry loop skips sticky rows.
- FleetSyncService.executePushToNode: short-circuits at the top when
sticky is set (covers event-driven pushResourceAsync calls). On a 409
with code CONTROL_IDENTITY_MISMATCH, records the failure once and
pins sticky with the expected/got fingerprints carried in the 409 body.
- routes/nodes.ts: new POST /api/nodes/:id/fleet-sync/reset-anchor.
Admin + paid + node:manage. Proxies POST /api/fleet/role/reanchor to
the peer with `{override:true}` using the stored Bearer node_proxy
token. On peer 200, clears every sticky row for the node so the next
push re-anchors and resumes replication. Distinct 502 / 504 responses
for peer-rejected / peer-unreachable so the UI can show a useful toast.
Frontend
- New lib/fleetSyncApi.ts + hooks/useFleetSyncStatus.ts. Polling hook
(30s visibilityInterval) skips fetch when !isPaid.
- NodeManager.tsx: destructive banner per affected node listing both
fingerprints, with `Reset anchor on peer` and `Remove node` buttons.
Hidden for community-tier users via empty hook data.
- FleetConfiguration.tsx (Fleet -> Status): read-only `Policy sync`
SummaryRow per remote node card. In sync / degraded / paused with
a tooltip; no action buttons (the action lives in NodeManager).
Tests
- fleet-sync-service.test.ts: 4 new cases for sticky-set on first
mismatch, short-circuit on subsequent pushes, null fingerprints,
and non-mismatch failures not setting sticky.
- database-fleet-sync-sticky.test.ts (new): 6 cases pinning the DB
contract incl. retry-loop SQL filter and migration idempotency.
- nodes-fleet-sync-reset-anchor.test.ts (new): 6 cases covering
happy path, peer 401 -> 502, peer unreachable -> 504, local-node
rejection, unknown node id, and community-tier 403.
Gate parity (Directive 30): the new POST .../reset-anchor enforces
requireAdmin + requirePaid + node:manage (matches the existing read at
GET /api/fleet/sync-status). UI banner + SummaryRow only render when the
hook returns data, which it only does for paid-tier authed users. No
existing tier-gate file moved; this is greenfield parity.
Auth audit: the peer's POST /api/fleet/role/reanchor route already uses
requireAdmin, which accepts the central's stored node_proxy Bearer
token because authMiddleware maps `scope === 'node_proxy'` to
`req.user = { username: 'node-proxy', role: 'admin', userId: 0 }`.
No widening required.
Backend tsc clean. Frontend tsc -b clean. 59 fleet-sync tests pass; full
backend suite green minus the pre-existing Windows-only file-lock flake
on filesystem-backup.test.ts that reproduces unchanged on main.
|
||
|
Anso
|
9aaa8573c0 |
fix(mesh): gate Trigger 2 re-bootstrap on actual api_token change (#1076)
The PUT /api/nodes/:id handler closed and re-dialed the mesh callback bridge on every save that included api_token in the body, even when the token was unchanged. The frontend always sends the full formData on Save, so renames and compose_dir edits against a mesh-enabled proxy remote produced a wasted closeBridge + ensureBridge round-trip and a spurious manager_rejected entry in the activity log. Gate the re-bootstrap on a real value diff against the persisted token. Adds an existing-node lookup (returns 404 on missing id, which the handler previously lacked) so the comparison has the pre-update value. Reorders the guards so resource-not-found beats payload validation. Adds one integration test for the same-token path; the existing three trigger-2 cases continue to assert close + ensure firing on a real rotation, no-token-in-payload, and mesh-disabled. |
||
|
Anso
|
cf618dd866 |
feat(mesh): symmetric WS dial for proxy-mode mesh peers (#1066)
* chore(mesh): foundation for symmetric callback dial Adds the data-plane scaffolding that the symmetric callback dial fix builds on: - mesh_centrals table for peer-side bootstrap material - MeshCentralRegistry service (upsert/getActive/clear/markUsed/markRejected) - PilotTunnelManager kind discriminator and replaceOrRegisterProxyBridge - mesh_proxy_callback_bootstrap capability registration - MeshProxyTunnelDialer reason-tagged proxy-bridge-down events from a single tearDownBridge emission point - Reactive redial scheduler that skips idle and auth_failed reasons * feat(mesh): add reverse-direction activity log entries (closes R1-B) acceptReverseLocal now emits route.resolve.ok with direction=reverse on connect ack and route.resolve.fail with direction=reverse plus reason=container_not_found / connect_error pre-connect. Post-connect close/error stays silent. Reuses existing event types via the new details.direction discriminator so frontend filters are unaffected. * feat(mesh): add peer-to-central callback dial path (closes R1-A2) Closes the architectural gap where proxy-mode mesh peers could not re-establish their tunnel to central after any non-idle bridge teardown (idle close, network blip, central restart, peer reboot). Central remains the hub for the data plane; the change is purely about WS initiation. Symmetric WS initiation, asymmetric protocol roles. Central retains PilotTunnelBridge ownership; peer retains TcpStreamSwitchboard + reverseDialer ownership. Central bootstraps callback credentials over the first authenticated central-initiated mesh tunnel via a one-shot mesh_handshake JSON frame; peer persists the material in a new mesh_centrals SQLite table and dials central's new /api/mesh/proxy-tunnel-from-peer endpoint when local cross-node traffic needs a bridge and none is live. Mesh_tunnel JWT (HS256, signed with auth_jwt_secret) carries scope, audience, issuer (central instance id), peer api_token fingerprint, kid. Validation on inbound peer dial: algorithm pin, signature, scope, audience, instance, time bounds, node existence and mode, fingerprint match. Failures return HTTP 401 with a machine-readable reason; peer routes the response per a clear-vs-keep cache matrix. Triggers proactive bootstrap on mesh-enable and api_token rotation; central startup fans out to mesh-enabled proxy-mode nodes with mesh_stacks rows (throttled, fire-and-forget). Reactive redial on non-idle bridge loss. Capability-gated handshake send (mesh_proxy_callback_bootstrap) makes the upgrade path safe against older peers in mixed-version fleets. Adds peer-side /api/system/pilot-tunnels centralCallback diag block, bounded counter metrics for bootstrap and dial events. SENCHO_PRIMARY_URL preflight warning when unset on a central with mesh-enabled proxy nodes. Tested with unit suites for the validation chain, registry, manager, and both dialers; integration tests for bootstrap E2E (asserts protocol-role invariant), api_token rotation, instance id change, version skew, and pilot-mode regression. * fix(mesh): green CI on the symmetric callback branch Two independent CI failures, both surgical: 1. Backend tests (11 fails): four mesh test files called setupTestDb in beforeEach. setupTestDb does not reset the DatabaseService singleton, so the per-test afterEach rm of the previous tmpdir left the singleton connection pointing at a deleted file. The next beforeEach's line-55 write threw SQLITE_READONLY_DBMOVED on Linux. Windows file-lock semantics hid this locally. Hoist setupTestDb / cleanupTestDb to file-scope beforeAll / afterAll; per-test state resets stay in beforeEach. Matches the convention in the eight mesh test files that already pass. 2. CodeQL (4 high alerts): js/insufficient-password-hash flagged sha256(api_token) at four sites. The api_token is a 256-bit opaque bearer (sen_sk_-prefixed), not a human password; sha256 is the correct fingerprint primitive for binding the mesh_tunnel JWT to a specific token. Add the two production files plus the two test files that mint the fingerprint to the existing path-scoped query-filter for that rule. * fix(mesh): drop unused afterEach import and revert dead codeql config ESLint flagged afterEach as unused in mesh-central-registry.test.ts:1 after the previous commit hoisted setup/teardown to file-scope beforeAll/afterAll. Remove from the vitest import line. Revert the codeql-config.yml additions from the previous commit. The paths: sub-key under query-filters > exclude is not a documented CodeQL feature and silently no-ops. The four js/insufficient-password-hash alerts on api_token fingerprinting are tracked as dismissed false positives in the GitHub Security tab rather than via dead config. |
||
|
Anso
|
8dd0fce621 |
fix(fleet): show capabilities, version, metrics, and stacks for pilot-agent nodes (#1044)
When a pilot-agent node was the active node, the UI rendered "does not advertise this capability" across most tabs, a perpetual "Update available" badge, and a Fleet card body with blank CPU/RAM/Disk and "No stacks found". The cause was central-side aggregators in /api/fleet/* and /api/nodes/:id/meta only fanning out to proxy-mode remotes via node.api_url + node.api_token, which are null for pilot-agent. Route every affected aggregator through NodeRegistry.getProxyTarget so the loopback URL backed by the active pilot tunnel is used uniformly: - /api/nodes/:id/meta and /api/fleet/update-status fetch via the new NodeRegistry.fetchMetaForNode helper (resolves the target, delegates to fetchRemoteMeta, returns the shared OFFLINE_META on null). - fetchRemoteNodeOverview, /api/fleet/configuration, /api/fleet/node/:nodeId/stacks, and the stack-containers drilldown fetch through target.apiUrl with conditional Authorization. - fetchRemoteMeta omits the Authorization header when the token is empty (pilot-agent loopback) instead of sending a malformed Bearer string. - Pilot-agent rows preserve pilot_last_seen and mirror it into last_successful_contact so the Fleet "last seen" cell renders the recent tunnel timestamp during a brief reconnect. Pilot-mode capability filter excludes capabilities whose central-pilot path is not yet wired (host-console, self-update). Without this, the Console tab would surface for an Admiral pilot session and click through to central's host because the WS upgrade handler still gates on api_url + api_token. Filtered capabilities are removed at boot via applyPilotModeCapabilityFilter when SENCHO_MODE=pilot. Cache invalidation on tunnel-up: the meta cache for a reconnecting pilot is dropped so the next request rebuilds capabilities and version through the live bridge instead of waiting for the 3-minute TTL. The namespace constant moves to helpers/cacheInvalidation.ts alongside the new invalidateRemoteMetaCache helper. Husky commit-msg hook: add the missing shebang and a .gitattributes rule pinning .husky/* to LF line endings so commits do not fail with "Exec format error" on Windows shells where autocrlf=true converts the hook to CRLF. Tests cover Authorization-header behavior, pilot-mode filter idempotency, fetchMetaForNode dispatch (offline target, pilot-agent loopback, proxy-mode), and the four affected fleet routes for pilot-agent both when the tunnel is up and when it is down. |
||
|
Anso
|
3d9489648e |
fix(pilot): harden outbound reverse-tunnel against resource exhaustion (#979)
* fix(pilot): cap tunnel frame size, concurrent streams, and stream idle time
Pilot tunnels carry every HTTP, WS, and Mesh-TCP byte for a remote node
through a single multiplexed WebSocket. A buggy or compromised peer that
sent oversized frames, opened streams in a tight loop, or left streams
parked indefinitely could exhaust gateway memory.
Three protocol-level limits applied symmetrically on both ends:
- MAX_FRAME_SIZE_BYTES (8 MB): set as the ws maxPayload on the
gateway-side WebSocketServer and the agent-side WebSocket client,
plus a defense-in-depth length check in decodeBinaryFrame and
decodeJsonFrame.
- MAX_STREAMS_PER_TUNNEL (1024): bridge refuses new loopback HTTP /
upgrade / TCP allocations with 503 once at the cap; the agent
rejects new incoming http_req / ws_open / tcp_open with the
appropriate error frame.
- STREAM_IDLE_TIMEOUT_MS (10 min): every stream gets a per-stream
timer refreshed on each inbound or outbound activity. Expiry tears
down the local half and notifies the peer.
The constants are colocated in pilot/protocol.ts so any future agent
build picks them up via the protocol module.
* fix(pilot): rate-limit pilot enrollment endpoints to 10 per minute
Pilot enrollment mints a JWT and writes a pilot_enrollments row, both
privileged operations that should not share the global API budget
(200/min). Add a dedicated express-rate-limit instance keyed by user or
IP at 10/min in production (100/min in dev so the local enrollment
test loop is not throttled).
The limiter is applied directly on POST /api/nodes/:id/pilot/enroll.
On POST /api/nodes the limiter's skip function reads the parsed body
and exempts proxy-mode creates; only requests that resolve to
mode=pilot_agent count against the enrollment budget.
* fix(pilot): cap concurrent pilot tunnels system-wide
The PilotTunnelManager held an unbounded Map of bridges. A reconnect
storm, runaway enrollment, or operator misconfiguration could grow the
map without limit, even though every tunnel still carries a valid JWT.
Cap at 256 concurrent tunnels per primary instance:
- Soft warning at 128 (logged once per crossing).
- Hard refusal at 256: registerTunnel throws PilotTunnelCapacityError
and the upgrade handler closes the WebSocket with 1013 (Try Again
Later) so the agent backs off rather than tight-looping.
A node that already has a registered tunnel does not consume a new
slot when it reconnects; the existing bridge is closed first.
* fix(pilot): release backpressure as soon as the tunnel buffer drains
Previously the bridge paused HTTP request bodies when the tunnel
WebSocket's bufferedAmount climbed above 4 MB but never resumed them
explicitly; the next data event re-checked the threshold, and TCP
streams only saw a 'drain' fan-out on the 30 s ping cycle. Slow-consumer
peers held buffered bytes for tens of seconds longer than needed.
Replace both with an on-demand drain check: when at least one stream
is paused, sample bufferedAmount every 100 ms; once it drops below the
high-water mark, resume every paused request and emit 'drain' to every
accepted TCP stream, then stop the timer. Dormant when no stream is
paused, so steady-state cost is zero.
* fix(pilot): trust internal CAs via SENCHO_PILOT_CA_FILE
Self-hosted deployments often terminate TLS with an internal CA. The
only previous escape hatch was NODE_TLS_REJECT_UNAUTHORIZED=0, which
disables verification across every outbound connection in the agent
process and is the wrong shape of fix.
Add SENCHO_PILOT_CA_FILE: when set, the agent reads the file as a PEM
bundle and passes it to ws as the `ca` option for the tunnel
WebSocket. rejectUnauthorized remains true. Failure to read the file
exits with a clear error so the operator does not silently fall back
to the default trust store.
There is no flag to disable TLS verification entirely; that would
defeat the credential trust model.
* fix(pilot): gate frame-rate-adjacent logs behind developer_mode
The agent's malformed-frame warning fires per inbound frame and could
flood logs under attack or against a buggy primary. The bridge swallowed
parse errors silently, leaving operators blind to protocol drift, and
the manager had no signal at all on tunnel registration.
Route per-frame and per-tunnel diagnostic logs through the existing
isDebugEnabled() helper (same pattern used by middleware/auth, RBAC,
and websocket/logs). Production stays quiet by default; the toggle is
the existing developer_mode setting in global_settings — no new env
var, no new dependency.
* feat(pilot): expose per-tunnel metrics via /api/system/pilot-tunnels
Operator support had no signal beyond raw logs to answer "is this
tunnel flapping?" or "is one bad node hiding behind the aggregate?".
Add an in-process counter set covering tunnels_total, tunnels_replaced,
tunnels_rejected_capacity, enroll_acks, frame_decode_errors, plus a
per-node array carrying the connectedAt and bufferedAmount so a single
tunnel sitting on a stuck buffer remains visible.
Counters live in services/PilotMetrics.ts; the gateway has no shared
in-process metrics facility today, so this is a per-feature pattern
documented as such for future consolidation. Counters are strictly
process-local — no telemetry, no export, no phone-home — consistent
with Sencho's privacy posture.
Surfaced read-only via GET /api/system/pilot-tunnels behind admin auth,
mirroring the existing cache-stats endpoint.
* fix(pilot): tighten reconnect backoff and log handling
Three small follow-ups to the hardening pass:
- Sanitize the primary URL on the agent's connect log so a malicious
SENCHO_PRIMARY_URL with embedded control characters cannot inject
fake log lines.
- Move the reconnect-backoff reset from the 'open' callback to the
'hello' frame handler. A peer that always rejects the handshake
(incompatible version, consumed enrollment token) used to reset
the backoff on every TCP-level connect and tight-loop reconnects;
now the reset waits for a clean protocol round-trip.
- Document the StreamIdAllocator wrap behavior so future readers
can see the relationship between the 2^31 wrap and the
MAX_STREAMS_PER_TUNNEL cap without re-deriving it.
* test(pilot): add coverage for enrollment, rate limiter, and tunnel caps
Adds two test files exercising the hardening pass surfaces that were
previously uncovered:
- pilot-enrollment.test.ts: end-to-end through POST /api/nodes
(pilot mode), POST /api/nodes/:id/pilot/enroll, the SHA256
token-hash persistence, replay protection, expired enrollments,
and the rate-limit header wiring (enrollment limiter on pilot
paths, global limiter elsewhere).
- pilot-bridge-limits.test.ts: oversize-frame rejection at both
binary and JSON decoders, and the per-tunnel concurrent stream
cap as observed via openTcpStream returning null and the loopback
HTTP server returning 503 once the bridge is at capacity.
Bridge cap test fills the slot map with TCP stream handles rather than
real HTTP requests so the OS socket pool stays out of the picture.
* test(pilot): cover manager metrics snapshot and capacity-error shape
Adds direct coverage for:
- PilotTunnelCapacityError exposing the limit so the upgrade
handler can format a useful close-frame reason.
- PilotMetrics.snapshot returning a defensive copy (callers must not
mutate the live counter set).
- getMetricsSnapshot returning the open count, per-node breakdown,
and counter set in a stable shape, with tunnels_total bumping on
a successful registerTunnel.
The manager test seeds a real pilot-mode node row first because
updateNode throws on missing rows; this models the production path
where a node is created before its enrollment is consumed.
* docs(pilot): document tunnel limits, custom CA, and new failure modes
Refresh the Pilot Agent feature doc to cover the hardening surfaces:
- New 'Self-signed primary TLS certs' section walking through the
SENCHO_PILOT_CA_FILE env var with an example docker run command.
- New 'Resource limits' section listing the per-tunnel and
system-wide ceilings so operators know what the wire enforces.
- Four new troubleshooting entries: WebSocket close 1013 (system
cap), loopback 503 (stream cap), close 1002 with protocol error
(frame size / malformed), and HTTP 429 on enrollment.
Also expand .env.example with a new pilot-agent block covering
SENCHO_MODE, SENCHO_PRIMARY_URL, SENCHO_ENROLL_TOKEN, and
SENCHO_PILOT_CA_FILE so operators do not have to read code or the
feature doc to discover the agent-side config surface.
* fix(pilot): address code-review findings on the hardening pass
Critical:
- Bridge: TCP backpressure now starts the drain timer and tracks
streams awaiting drain, so a TcpStream caller waiting on 'drain'
no longer hangs when no HTTP request is in flight.
- Bridge: every direct streams.delete call now goes through the
removeStream helper so per-stream idle timers are cleared
consistently and pausedReqs / tcpAwaitingDrain stay aligned.
- Bridge close(): resume any paused IncomingMessage before clearing
the map so a parser does not stay stuck across teardown.
High:
- Protocol: decodeJsonFrame now compares Buffer.byteLength(raw,
'utf8') against the cap; raw.length (UTF-16 code units) let
multi-byte payloads sneak ~3x the byte budget through.
- Agent: the optional CA bundle is read once at construction and
cached on the instance, so a missing or rotated SENCHO_PILOT_CA_FILE
no longer process-exits on every reconnect attempt.
Medium / Low:
- Bridge limits test: assert mockWs.sent.length to confirm slot
allocations actually serialized a tcp_open frame and that a
rejected (cap+1) attempt did not consume a stream id.
- Manager: defer the tunnels_replaced increment until after the
cap check passes, so a rejected reconnect does not double-count
as both replacement and capacity rejection.
- Manager: clarify the protocol comment about MAX_FRAME_SIZE_BYTES
enforcement layering (ws maxPayload is authoritative; decoder
check is for tests and defense-in-depth).
- Limiter: defensive `if (!req.body) return false` so a future
refactor that delays body parsing cannot silently skip the
enrollment limiter.
Deferred (documented as follow-ups, not in this branch):
- M2: narrowing public surface of PilotTunnelManager.getBridge to a
MeshTunnelHandle interface (cross-cutting refactor).
- H4: end-to-end replay test driving the upgrade handler twice
with the same enrollment JWT (needs WS test harness).
|
||
|
Anso
|
33b15d6cba |
feat(fleet-sync): retry failed pushes and backfill on add-node (#970)
A control instance now retries fleet-sync pushes that hit a transient failure and backfills the security state on a freshly registered remote without waiting for the next policy edit. New service: - FleetSyncRetryService (singleton, start/stop) wakes 30s after boot and ticks every 5min. For each fleet resource, queries getFailedSyncTargets within a 24h window and re-pushes via FleetSyncService.pushResourceToNode through the same per-node mutex, so a normal fanout in flight serializes naturally with a retry. - After STALE_THRESHOLD_MS (1h) of continuous failure for a previously-working node, dispatches a single warning notification per cooldown window. Brand-new nodes that have never succeeded do not alert via this path; misconfigured remotes are caught by the test-connection affordance at registration time. - Wired into bootstrap startup/shutdown next to AutoHealService. Public surface: - FleetSyncService.pushResourceToNode(node, resource): targeted push to one node that re-uses the per-node mutex. Used by the retry service and any future targeted-resync flow. - routes/nodes.ts POST /api/nodes fires pushResourceAsync for both resources after a remote-proxy node row commits. Tuning constants centralized in fleetSyncConstants.ts: - RETRY_MAX_AGE_MS = 24h - STALE_THRESHOLD_MS = 1h Tests: - 8 vitest cases covering replica skip, retry dispatch, missing-node skip, alert-once-per-cooldown across the threshold window, no-alert for recent failures, no-alert for brand-new never-succeeded nodes, no-alert when the retry itself succeeds, start/stop idempotency. - Full backend suite: 1781 pass / 5 skipped. |
||
|
Anso
|
52b46753af |
feat(fleet): add Federation tab with cordon and pin policy (Admiral) (#964)
Ships the v1 MVP for the Federation tab as placement control, not
placement automation:
- Cordon a node: marks the node unschedulable so the BlueprintReconciler
skips it for new placements only. Existing deployments continue to
drift-check and redeploy on revision changes; cordon never triggers
withdraw or eviction. Toggle on the NodeCard kebab (Admiral, admin
role); Cordoned pill renders for all tiers.
- Pin a blueprint to a node: stores blueprints.pinned_node_id, replacing
the desired set with the pinned node regardless of selector. Pin
overrides cordon by design. Action lives only in the Federation tab;
BlueprintDetail and the deployment table show read-only Pinned
indicators.
Backend: idempotent migrations add nodes.cordoned/cordoned_at/cordoned_reason
and blueprints.pinned_node_id. New routes POST /api/nodes/:id/cordon,
POST /api/nodes/:id/uncordon, PUT /api/blueprints/:id/pin, all gated by
requireAdmiral plus requireAdmin. Audit summaries added so the existing
auditLog middleware records every operator action. deleteNode clears
dangling pins.
Reconciler: pin override evaluated before selector match; cordon filter
applied only to the new-placement branch (deploy/stateReview without an
existing deployment). 11 new Vitest cases cover cordon filter, pin
override, pin-overrides-cordon, missing pin target, pin shrinks
desired set (stateless withdraw + stateful evict_blocked), and pin
clearing on node delete.
Frontend: new FederationTab.tsx with cordoned-nodes summary and
pin-policy table. Federation moved out of the experimental flag into
{isAdmiral && (...)} + AdmiralGate, mirroring the Routing tab pattern.
Secrets stays under experimental.
Tests pass: backend tsc, full Vitest suite (1704 passed), frontend
tsc -b, ESLint (0 errors). Manual verification via the local dev
instance confirmed the tab is hidden at Community, the kebab and pill
render at Admiral, and cordon and pin endpoints round-trip end to end.
Refs cut-line-1.0.md Federation v1 MVP.
|
||
|
Anso
|
ed553f1f19 |
feat: change default listen port from 3000 to 1852 (#756)
Updates the backend listen port, Vite dev proxy target, Docker EXPOSE, compose port mapping, .env.example default, GitHub Actions smoke-test default, healthcheck URLs, and every doc/example reference. Test fixtures that include example URLs were updated for consistency, though their assertions are port-agnostic. The rate-limit value of 3000 in middleware/rateLimiters.ts and the 3000 entry in WEB_UI_PORTS (which detects user containers like Grafana) are intentionally untouched. |
||
|
Anso
|
3995086872 |
refactor(backend): extract nodes router (phase 4c-5) (#743)
Move the nine /api/nodes/* endpoints out of index.ts into routes/nodes.ts (list, scheduling-summary, get, create, pilot-enroll, update, delete, test, meta). mintPilotEnrollment and the REMOTE_META_* constants move with the router as local helpers. Handlers moved verbatim. Two safe cleanups applied during the move: - Inline req.apiTokenScope 403 blocks replaced with the shared rejectApiTokenScope helper; payload shape unchanged. - catch (error: any) rewritten to catch (error: unknown) with explicit instanceof Error narrowing to satisfy the no-any strictness rule. Response body shapes unchanged. Removes now-unused imports from index.ts: jwt, crypto, authMiddleware, isValidRemoteUrl, PilotTunnelManager, PilotCloseCode, CAPABILITIES, getSenchoVersion, fetchRemoteMeta, RemoteMeta, FleetUpdateTrackerService, plus the module-scope updateTracker alias. index.ts drops from 1364 to 1021 lines. Only the stacks group remains inline for the final phase 4c slice. |