mirror of
https://github.com/Studio-Saelix/sencho.git
synced 2026-07-27 20:29:10 +00:00
Anso
23bbee4f45
* feat(mesh): replace host-mode with shared sencho_mesh Docker network Phase D of the mesh redesign: drop the operator's `network_mode: host` requirement and the `host-gateway` extra_hosts pattern that did not work on cloud iptables-restrictive distros (OCI, etc.) or Docker Desktop. Each Sencho creates a shared `sencho_mesh` Docker bridge network on boot (default subnet 172.30.0.0/24, override via SENCHO_MESH_SUBNET), pins itself at `<network>+2`, and attaches every meshed user service to the same bridge. Compose overrides now emit IP-based `extra_hosts` plus a top-level `networks` block declaring `sencho_mesh` external. Override delivery: central renders for local stacks; for remote stacks it sends the fleet alias list to the remote's new `PUT /api/mesh/local- override/:stackName` endpoint, which renders against the remote's OWN local senchoIp and writes under its OWN DATA_DIR. Each node may use a different subnet without coordination beyond the env var. Opt-in / opt-out now trigger an automatic redeploy of the affected stack via the existing deploy code path (local: ComposeService; remote: HTTP POST through proxyFetch). The frontend opt-in sheet shows a confirmation modal (ConfirmModal) before the mutation. Failed redeploys emit both a mesh activity event and a durable audit-log row. Hardening: - Reserve port 1852 at opt-in (prevents user containers from racing the Sencho API listener). - ensureMeshNetwork refuses to continue if `sencho_mesh` exists with a mismatched subnet rather than silently routing to the wrong IP. - Idempotent network connect/disconnect helpers in DockerController. - optInStack rolls back the DB row if the just-inserted stack's override push fails (no half-states surviving across calls). - regenerateOverridesForNode runs in parallel and skips the just- pushed stack on opt-in. Operator template: drop `network_mode: host`, restore `ports: ["1852:1852"]`. Mesh now works identically on Linux LAN, OCI, and Docker Desktop without firewall changes. Docs: rewrite docs/features/sencho-mesh.mdx around the shared bridge network, document SENCHO_MESH_SUBNET, surface the host-network-service opt-in restriction, and cross-link with the Pilot Agent docs. BREAKING CHANGE: the operator's `docker-compose.yml` no longer uses `network_mode: host`. After upgrading, redeploy any meshed stacks once so they pick up the new IP-based override and join `sencho_mesh`. * fix(mesh): wrap stackName with path.basename in local-override fs ops CodeQL flagged js/path-injection on the new applyLocalOverride and removeLocalOverride methods because they are publicly reachable and its data-flow model does not recognize isValidStackName / isPathWithinBase as sanitizers. The validation IS sufficient (the allowlist regex blocks path separators, the path-prefix check blocks escape), but path.basename is a model CodeQL recognizes and is purely defensive: for any input that already passes isValidStackName, basename is the identity.
145 lines
10 KiB
Plaintext
145 lines
10 KiB
Plaintext
---
|
|
title: Pilot Agent
|
|
description: Add remote nodes behind NAT, residential networks, or corporate firewalls without exposing any inbound port.
|
|
---
|
|
|
|
Pilot Agent mode connects a remote server to your primary Sencho instance through a single outbound WebSocket tunnel. Every request the primary sends to that node, HTTP or WebSocket, rides through this tunnel. The remote host never opens an inbound port and never needs a TLS certificate.
|
|
|
|
<Card title="Sencho Mesh" icon="link" href="/features/sencho-mesh">
|
|
Once a node is enrolled, you can wire stacks across nodes by hostname using Sencho Mesh, which rides on this same tunnel.
|
|
</Card>
|
|
|
|
## When to use Pilot Agent
|
|
|
|
Pick Pilot Agent when the remote host:
|
|
- Sits behind NAT or a residential router.
|
|
- Lives on a corporate network that blocks inbound connections.
|
|
- Roams between networks (laptops, mobile hotspots, rotating cloud IPs).
|
|
- Would otherwise need a reverse proxy, a dynamic DNS entry, or a self-signed TLS certificate just to be reachable.
|
|
|
|
Pick **Distributed API Proxy** (documented in [Multi-Node Management](/features/multi-node)) when the remote host already has a stable, reachable URL, for example a VPS with a public IP or a LAN server on a home network. Both modes are supported side-by-side, one per node.
|
|
|
|
## How it works
|
|
|
|
The agent runs inside a second container on the remote host, using the same `saelix/sencho:latest` image. Setting `SENCHO_MODE=pilot` plus a primary URL and a one-time enrollment token puts the container into agent mode. On boot it dials the primary at `wss://<primary>/api/pilot/tunnel` and holds that connection open. For every tunneled request the primary demultiplexes frames to an internal loopback server, which re-issues the request locally on the agent host against its Docker socket. License tier, role checks, and all other authorization continue to flow from the primary, exactly like proxy mode.
|
|
|
|
Only outbound HTTPS from the remote to the primary is required. Nothing else is exposed.
|
|
|
|
## Enrollment walkthrough
|
|
|
|
### 1. Add the node on the primary
|
|
|
|
On your primary instance open **Settings → Nodes** and click **Add Node**.
|
|
|
|
- **Type:** Remote
|
|
- **Mode:** Pilot Agent (the default for remote nodes)
|
|
- **Name:** any label, for example `homelab-nuc`
|
|
- **Compose Directory:** the folder on the remote host where stack folders will live
|
|
|
|
Click **Add Node**. The form is replaced by an enrollment dialog with a one-line `docker run` command.
|
|
|
|
<Frame>
|
|
<img src="/images/pilot-agent/enrollment-dialog.png" alt="Pilot Agent enrollment dialog with docker run command" />
|
|
</Frame>
|
|
|
|
### 2. Run the command on the remote host
|
|
|
|
Copy the command and paste it on the remote host. It looks like:
|
|
|
|
```bash
|
|
docker run -d --restart=unless-stopped --name sencho-agent \
|
|
-v /var/run/docker.sock:/var/run/docker.sock \
|
|
-v sencho-agent-data:/app/data \
|
|
-v /opt/docker/sencho:/app/compose \
|
|
-e SENCHO_MODE=pilot \
|
|
-e SENCHO_PRIMARY_URL=https://sencho.example.com \
|
|
-e SENCHO_ENROLL_TOKEN=<short-lived token> \
|
|
saelix/sencho:latest
|
|
```
|
|
|
|
The enrollment token is single-use and expires after 15 minutes. On first connect the agent exchanges it for a long-lived tunnel credential, persisted inside the container volume at `/app/data/pilot.jwt`. Subsequent restarts reconnect automatically without needing a new token.
|
|
|
|
### 3. Confirm the node is online
|
|
|
|
The node flips to **Online** in the primary within a few seconds of the agent container starting. The **Endpoint** column shows `tunnel` alongside the time the primary last saw a frame from the agent.
|
|
|
|
Open the node in the sidebar switcher and use it exactly like any other node: deploy stacks, tail logs, open container terminals, watch host stats.
|
|
|
|
## Day-to-day operation
|
|
|
|
Once enrolled, a Pilot Agent node is indistinguishable from a proxy-mode node in the UI. The same switcher, the same stacks page, the same editor, the same host console, the same dashboard.
|
|
|
|
The agent reconnects automatically if the tunnel drops (transient network blip, primary restart, remote host reboot). It applies an exponential backoff between retries starting at 1 second and capping at 60 seconds, so reconnect traffic stays bounded even during long outages.
|
|
|
|
## Regenerating enrollment
|
|
|
|
If the agent container is destroyed before its first successful connect, or the enrollment token expires before you paste it on the remote, open the node's edit dialog on the primary and click **Regenerate enrollment token**. A fresh 15-minute token is issued and the previous tunnel (if any) is disconnected so the new agent replaces it cleanly.
|
|
|
|
## Self-signed primary TLS certs
|
|
|
|
If your primary terminates TLS with an internal CA and the agent cannot validate the certificate against the system trust store, point the agent at the CA bundle with `SENCHO_PILOT_CA_FILE`. Mount the bundle into the agent container and add the env var:
|
|
|
|
```bash
|
|
docker run -d --restart=unless-stopped --name sencho-agent \
|
|
-v /var/run/docker.sock:/var/run/docker.sock \
|
|
-v sencho-agent-data:/app/data \
|
|
-v /opt/docker/sencho:/app/compose \
|
|
-v /etc/ssl/internal-ca.pem:/etc/ssl/internal-ca.pem:ro \
|
|
-e SENCHO_MODE=pilot \
|
|
-e SENCHO_PRIMARY_URL=https://sencho.internal.example.com \
|
|
-e SENCHO_ENROLL_TOKEN=<short-lived token> \
|
|
-e SENCHO_PILOT_CA_FILE=/etc/ssl/internal-ca.pem \
|
|
saelix/sencho:latest
|
|
```
|
|
|
|
The agent uses the bundle as the only trust anchor for the tunnel WebSocket. TLS verification stays on; there is no env var to disable verification globally because that would defeat the credential trust model.
|
|
|
|
## Resource limits
|
|
|
|
Each tunnel has fixed protocol-level ceilings to keep one misbehaving agent from impacting the primary:
|
|
|
|
- **Frame size:** individual WebSocket frames are capped at 8 MB. Single requests and responses larger than that are rejected and the tunnel reconnects.
|
|
- **Concurrent streams:** at most 1024 multiplexed HTTP and WebSocket streams per tunnel (mesh TCP streams share the same pool). Above the cap the bridge returns 503 and the agent rejects new incoming streams with a typed error frame.
|
|
- **Stream idle:** any stream with no activity for 10 minutes is closed and removed.
|
|
- **System-wide tunnels:** a single primary accepts up to 256 concurrent pilot tunnels. Past the soft warning at 128 the primary logs a WARN; past the hard cap of 256 new tunnels are refused with WebSocket close code 1013 (Try Again Later) so the agent backs off rather than tight-looping.
|
|
|
|
## Troubleshooting
|
|
|
|
<AccordionGroup>
|
|
<Accordion title="Node stays Offline after starting the agent container">
|
|
Check the agent container's logs for the first `[Pilot]` line. It prints the primary URL it is dialing and the reason for any failed connect. Common causes: the primary URL is wrong or unreachable from the remote host, the enrollment token expired, or the primary is behind a reverse proxy that strips WebSocket upgrades.
|
|
</Accordion>
|
|
|
|
<Accordion title="Tunnel keeps disconnecting and reconnecting">
|
|
Usually caused by an HTTP proxy or load balancer in front of the primary with a short idle timeout. The tunnel sends a ping every 30 seconds, so raise the proxy's WebSocket idle timeout above that (90 seconds is a safe floor). If you terminate TLS on the proxy, make sure WebSocket upgrade passthrough is enabled.
|
|
</Accordion>
|
|
|
|
<Accordion title="Enrollment token expired before I ran the docker command">
|
|
Open **Settings → Nodes** on the primary, click the pencil icon on the pending node, then **Regenerate enrollment token**. The dialog shows a fresh command with a new token.
|
|
</Accordion>
|
|
|
|
<Accordion title="I ran the enrollment command on the wrong host">
|
|
Stop and remove the agent container on the wrong host (`docker stop sencho-agent && docker rm sencho-agent`), then regenerate the enrollment token on the primary and run the command on the correct host.
|
|
</Accordion>
|
|
|
|
<Accordion title="Primary was restored from backup and the agent will not reconnect">
|
|
The agent's persisted tunnel credential is signed with the primary's JWT secret. If that secret is rotated or the primary is rebuilt from scratch, existing tunnels stop verifying. Regenerate enrollment for the affected node and restart the agent container so it consumes the fresh token.
|
|
</Accordion>
|
|
|
|
<Accordion title="Tunnel rejected with 'pilot tunnel cap reached' (close code 1013)">
|
|
The primary is at the system-wide concurrent tunnel cap of 256. A reconnect storm or runaway enrollment is the usual cause. Inspect the primary's logs for the `[Pilot] Active tunnel count at soft limit` warning that fires at 128 to find the trend, and check `GET /api/system/pilot-tunnels` (admin-only) for the per-node breakdown to identify a flapping agent. The agent backs off and retries automatically once headroom returns.
|
|
</Accordion>
|
|
|
|
<Accordion title="Bridge returns 503 'pilot tunnel: stream cap reached'">
|
|
A single tunnel is at the 1024 concurrent stream cap. The most common trigger is a UI session that opened many long-poll streams and never closed them, or a script firing parallel API calls without bound. Reload the affected browser tab so its WebSocket and SSE streams reset. If the cap is being hit by automation, throttle the caller; the cap protects the primary's memory from a runaway agent or proxy.
|
|
</Accordion>
|
|
|
|
<Accordion title="Tunnel closes immediately with 'protocol error' (close code 1002)">
|
|
The agent sent a frame larger than the 8 MB ceiling, or the wire decoder rejected a malformed frame. Run the agent with developer mode enabled on the primary (Settings → Developer) to surface the diagnostic decode log, then check the primary's logs for `[PilotBridge:diag] Malformed frame from agent`. The agent reconnects automatically; persistent failures usually mean a version mismatch between primary and agent images.
|
|
</Accordion>
|
|
|
|
<Accordion title="HTTP 429 'Too many enrollment requests'">
|
|
Enrollment endpoints are rate-limited to 10 requests per minute per user (or per IP, when unauthenticated) so a script accidentally minting tokens in a loop cannot exhaust the database. Wait sixty seconds and retry. If you genuinely need to bulk-enroll many nodes, space the requests out or contact the operator who runs the primary about raising the limit.
|
|
</Accordion>
|
|
</AccordionGroup>
|