mirror of
https://github.com/Studio-Saelix/sencho.git
synced 2026-07-26 11:49:16 +00:00
b1decbb32a
* docs(introduction): refresh screenshots and correct stale nav/tab coverage
Replace all 5 screenshots with fresh 1920x1080 production captures.
Document the shipped Take down stack action, the Compose Labels stack
tab, and the Docker Labels Fleet tab, none of which were mentioned.
Correct the Console nav item to note it is a limited-availability
surface rather than a plain role/tier-gated view.
* docs(quickstart): refresh screenshots and correct preflight, dashboard, and menu drift
Replace all three quickstart screenshots with fresh captures and correct
several claims that drifted from the current UI:
- Document the environment preflight's 7th check (Sencho compose
location), previously missing entirely from both the text and the
screenshot alt text.
- Add the Stack health table's Source and Port columns, and note that
columns are sortable and default to load order.
- Note the masthead's running-container count, live CPU/memory readout,
and alert count.
- Fix the profile menu list: replace the nonexistent "Feedback" entry
with "Open New Issue", drop "Appearance" (it lives in Settings, not
the profile menu), and add the conditional "Billing" entry.
- Note that the sidebar groups stacks by Docker Compose label once
labels are assigned, with pinned stacks first.
* docs(configuration): document missing env vars and fix JWT secret wording
Add SENCHO_UPLOAD_DIR, TRIVY_CACHE_DIR, SENCHO_MESH_RECONCILE_INTERVAL_MS,
SENCHO_MESH_PROXY_TUNNEL_IDLE_MS, SENCHO_COMPOSE_COMMAND_TIMEOUT_MS, and
SSO_OIDC_CUSTOM_ENABLED, all real environment variables that were missing
from the reference tables. Clarify that the JWT signing secret has no
environment variable at all, generated and stored in the database only,
rather than implying an unused JWT_SECRET var exists.
* docs(stack-management): refresh against current live app and source
Rewrites the Stack Management page against the production node and
frontend source: adds the Compose Labels anatomy tab (new since the
last refresh), the Mute action on the stack header and sidebar
context menu, corrects the update-available banner wording and the
sidebar context menu's lifecycle ordering, notes the Doctor tab's
severity dot and the scan-status banner, and replaces all 14
screenshots with fresh production captures.
* docs(editor): refresh against current live app and source
Replace all 8 screenshots with fresh 1920x1080 production captures (mobile
shot at phone viewport). Document the self-stack protection dialog that
blocks deploy/delete actions on the stack running the current Sencho
instance, the multi-container summary strip and Compact/Detailed density
toggle, the mutually exclusive Expand containers / Expand logs controls,
the Files tab full-screen toggle, and the post-deploy scan-status banner
on the Anatomy panel, none of which were previously covered.
* docs(editor): recapture mobile compose screenshot without redaction
The previous mobile screenshot used the plex stack, whose compose volume
mount includes a real host path that had to be blacked out and overlaid
with placeholder text, leaving a visible seam. Recapture against the
dozzle stack instead, whose only volume is the Docker socket, so the
screenshot needs no editing.
* docs(stack-file-explorer): refresh page against current app and code
Renames "Files & Volumes tab" references to the current "Files" tab
label, documents the full-screen toggle and word-wrap control, expands
the non-browsable-volume reasons to match the current containment
logic (single-file binds, the full protected host-path list, and the
Docker-unreachable named-volume case), documents the 100-item bulk
selection cap and the 5000-file/1 GiB archive limits, corrects the
non-existent DISK_FULL error code, notes that override compose files
are unprotected, and scopes the atomic-write claim to fs-backed roots.
Replaces all 9 screenshots with fresh production captures taken in the
Files tab's full-screen mode, so no other stack UI (health metrics,
logs) appears in the background.
* docs(resources): refresh against current live app and source
* docs(dashboard): refresh against current live app and source
* docs(app-store): refresh against current live app and source
Replaces all 6 screenshots with fresh production captures and corrects
the Environment Variables and About-panel metadata claims to match what
the bundled LinuxServer.io registry actually returns today.
* fix(docs): recapture app-store advanced-tab screenshot without scroll cut-off
The prior capture was taken mid-scroll to reveal the Security checkbox,
leaving the template logo and About text cropped awkwardly at the top.
* docs(global-search): refresh against current live app and source
* docs(deep-links): refresh against current live app and source
- Add App Store, Logs, Update, Console, and Audit to the URL table (previously undocumented)
- Document hub-only URL redirect behavior and cross-link to Multi-Node Fleet
- Note Fleet tab URL segments do not always match their on-screen label (Status/configuration, Map/dependencies)
- Document the no-env-files edge case (Env tab absent, falls back to compose)
- Clarify Settings section-list state is phone-only; desktop always normalizes to /settings/appearance
- Note which top-level views share identical URLs between desktop and phone
- Remove greenfield-violating temporal phrasing ("now has", "as today")
- Replace em dashes in touched bullet list
* docs(appearance): refresh against current live app and source
* docs(stack-activity): refresh against current live app and source
Add the missing "Stack taken down" event category, cross-link the
Suppressed badge to notification mute rules, list the new Compose
Labels tab in the Anatomy panel strip, and replace both screenshots
with current production captures.
* docs(dossier): fix generated-facts wording, add rollback readiness section
Corrects three Dossier tab Generated Facts rows against current frontend
behavior: the Ports count is not filtered to host-published mappings, the
Network row's "bridge" label is fixed text rather than a derived driver,
and missing env var names appear only in the Markdown export, not the
live tab. Adds a Rollback Readiness subsection under Connected Features
(previously only an orphaned Limitations bullet), links to Compose
Networking and Storage Portability, and clarifies that a missing export
section can mean either an older node build or a failed on-demand fetch,
not only a version gap. Recaptures all three screenshots against
production.
* docs(stack-drift): refresh against current live app and source
Fixes the stack detail tab bar description (Compose Labels tab was
missing, Files/Edit compose were wrongly listed as tabs), documents
that re-check only requires read access to the stack (so viewer and
auditor roles can trigger a ledger write), and cross-links the network
findings to the Networking tab's runtime drift section and the Fleet
Overview's Drift filter. Replaces all four screenshots with fresh
1920x1080 production captures.
* docs(compose-doctor): refresh against current live app and source
Corrects the rule count from 31 to 32 and documents the previously
undocumented self-managed-stack guard rule and enforcement, the
per-browser dismiss control for the summary card and Doctor tab dot,
the acknowledged summary state, the full Health-Gated Updates verdict
mapping, and the temporary exposure intent option. Replaces all four
screenshots with production captures.
* docs(compose-networking): refresh against current live app and source
Adds cross-links to the new node-wide Networking operator page, documents
the "view node networking" link and the unset/inherit exposure-intent pill
labels, corrects the stale Resources Hub network-tab reference, and
replaces all screenshots against the production node.
* fix(docs): unbreak stack-activity page render
The screenshot alt text used backslash-escaped quotes, which is
invalid MDX/HTML attribute syntax. Mintlify failed to parse the file
and silently dropped it from routing, so the page 404'd despite being
listed in docs.json navigation.
* docs(environment-guardrails): refresh against current live app and source
Fix the ENV FILES section description: it lists env_file: entries and
project files with an existence problem, not every declared env file.
A cleanly-resolving project file is already named in the Project
Environment File panel above it. Replace all three screenshots with
current production captures.
* docs(docker-label-audit): refresh against current live app and source
Add production screenshots (page had none), a Capability gating
section, a Limitations section, and a Troubleshooting accordion.
Tighten the secret-redaction heuristic description and cross-link
Stack Labels, Compose Doctor, Environment Guardrails, and Node
Compatibility.
* docs(compose-storage): refresh against current live app and source
Captured fresh production screenshots (both prior images were stale)
and fixed a leftover pre-rename card title pointing at the Files tab.
* docs(stack-labels): refresh against current live app and source
Removed the stale trailing-dot sidebar claim (that rendering was
removed from the sidebar in a prior UI pass), documented the new
label-scoped notification muting available from the sidebar, stack
context menu, and Settings panel, corrected the Fleet Actions card
copy to match the redesigned cards, added the previously undocumented
bulk-assign size cap and the node-scoped label bulk-action API, and
replaced all 8 screenshots with fresh production captures.
* docs(sidebar): refresh against current live app and source
Removes the stale per-row label dot claim (removed from StackRow), corrects the update indicator and Updates chip color from orange to fuchsia, and documents Mute/Take down in the context menu, the label group mute kebab, Ctrl+A/Esc bulk-mode shortcuts, pin-eviction toast, and the offline-node skip behavior in cross-node search. Replaces all 8 screenshots with fresh production captures.
* docs(deploy-progress): refresh against current live app and source
Corrected the Scan entry point (node-wide Scan this node from Security
Overview, not a per-stack config scan), added the missing Take down
verb and entry point, broadened recovery actions to cover restart and
rollback failures, fixed the modal status text and raw-output color
claims to match the live UI, and replaced all screenshots with fresh
captures from a throwaway demo stack on the production node.
* docs(atomic-deployments): refresh against current live app and source
* docs(health-gated-updates): refresh against current live app and source
* docs(deploy-enforcement): refresh against current live app and source
Correct the tier note to every-tier (verified against the current
policy gate, which no longer has a paid-only blocking switch), add the
Security Overview deploy-enforcement summary card, tighten the honor-
suppressions default wording, cross-link the separate pre-deploy scan
advisory dialog to avoid confusion with the block dialog, and replace
all three screenshots against the current production UI.
* docs(blueprint-model): refresh against current live app and source
Replaced all 11 screenshots with fresh production captures. Corrected
the volume-destroying-drift Enforce-downgrade claim (appeared three
times): that code path has no call sites in the runtime reconciler, so
any compose edit on a stateful or state-unknown blueprint always
re-enters Awaiting confirmation regardless of drift mode. Corrected
the Create workflow (creation does not trigger an immediate
reconciliation tick) and the Delete workflow (now requires typing the
blueprint name to confirm).
* docs(scheduled-operations): refresh against current live app and source
* docs(blueprint-model): drop tier framing, treat as Community-native
Blueprints carry no tier gate, so "available on every tier" implied a
comparison that doesn't exist. Removed the tier framing from the intro
note, the Prerequisites table, and the Security section; the role
requirement (admin to write, every role to read) already says
everything that matters.
* docs(auto-update-policies): refresh against current live app and source
Documents that every update trigger on this page, Apply now and
scheduled Auto-update tasks alike, runs through the same atomic backup,
build-aware rebuild, and post-update health gate as a manual update
from the stack editor, and is subject to the same deploy enforcement
policy gate. Corrects the readiness-computation description to scope it
to registry-image services and cross-links to Health-Gated Updates,
Atomic Deployments, Deploy Enforcement, and Stack Management. Replaces
the readiness board screenshot with a fresh production capture and
removes three orphaned screenshots left over from the page's prior
CRUD-policy layout.
* docs(auto-heal-policies): refresh against current live app and source
Correct the admin-only prerequisite: viewing policies and history is
open to every signed-in role, only creating/toggling/deleting requires
admin. Rewrite the matching troubleshooting entry to match, note the
overlap behavior between an All-services and a named-service policy on
the same container, and replace all three screenshots with a real
production policy.
* docs(webhooks): refresh against current live app and source
Replaced all three screenshots with fresh production captures (single
webhook create/reveal flow, two-card configured list), verified every
claim against WebhookService/StackOpLockService/registry.ts, and added
the previously undocumented per-stack lock skip behavior for
non-Git-source-sync actions with a matching troubleshooting entry.
* docs(global-observability): refresh against current live app and source
* docs(audit-log): refresh against current live app and source
- Fix nav terminology: Audit is a top navigation tab, not a sidebar tab.
- Correct retention claim: Admiral shows full retained history (default
90 days), not a fixed 14-day window; the 14-day clamp only applies to
the Community API, which has no navigation entry point at all.
- Fix settings path: Settings - Operations - Data Retention (previously
pointed at the now-split Developer Diagnostics section).
- Restore current Recovery Vault naming (was stale Sencho Cloud Backup).
- Expand the tracked-actions list: stack take-down, label management,
MFA reset, and notification suppression rules were missing.
- Note that assigning the Auditor role itself requires Admiral.
- Replace all four screenshots with fresh production captures (Stream,
Table, expanded row, Data Retention) reflecting the current five-row
retention card.
* fix(audit-log): reframe data-retention screenshot to card content only
The prior crop included the Settings sub-navigation panel, which isn't
part of what the surrounding text describes. Retake tightly cropped to
match the page's other screenshots (card content only, no chrome).
* docs(multi-node): refresh against current live app and source
Rewrites the Pilot Agent enrollment flow (now a generated compose.yaml
plus docker compose up -d, not a single docker run command) and the
Settings scope table (current registry: Stacks, Container Alerts,
Docker & Storage, Data Retention, Mute Rules, Recovery, and the
Admiral Account / Recovery Vault renames). Adds a Sencho Mesh
cross-link matching the live add-node form copy. Replaces all eight
screenshots with sanitized production captures.
* docs(pilot-agent): refresh against current live app and source
* docs(readme): refresh GitHub README against current live app
Replace all 9 screenshots (stale top nav showing a removed Console
item and missing the new Networking page); document Take down,
Drift Detection, Environment and Secrets Guardrails, Storage
Portability, Docker Label Audit, Remote Updates, and the node-wide
Networking dashboard; fix a broken non-root-user anchor link; correct
the notification channels list (no email channel exists yet) and the
global search scope (pages, nodes, and stacks, not containers or
services); rename "scan policy packs" to the current "scan policies"
terminology; broaden the App Store bullet to cover custom registries.
* docs(readme): reposition intro copy and refresh badge row
Lead with DevOps/platform/sysadmin framing instead of homelab-first,
drop pre-1.0 "production" language, state plainly that Sencho
provides a UI instead of promotional "does the work you do" phrasing,
and call out that multi-node was part of the architecture from the
start. Swap the Discussions badge for CodeQL, last-commit, open
issues, and live website/docs status badges, and add a blog link.
* docs: retake Dashboard and Global Search screenshots for accuracy
Configuration Status now shows Recovery Vault instead of the stale
Cloud Backup label, and the search palette capture includes the
Networking page entry added after the prior screenshot batch.
* docs: refresh Fleet View page against current app
Fleet View has drifted since its last rewrite: the Docker Labels tab,
Export Dossier action, Networking filter/badge, node label pills and
latency in topology, the Policy sync status row, and the node card
Mute submenu were all shipped but undocumented. The Check Updates and
Add Node actions also moved into the Overview toolbar (renamed Node
Update / Manage Nodes) and no longer sit in the shared action row.
Replaces all five screenshots with fresh production captures and
corrects the masthead's motion description (a calm shimmer on Healthy,
a steady glow on Degraded/Critical, not a pulsing dot). Also fixes a
Fleet Sync limitation that claimed no first-party sync-status panel
exists, now that the Status tab surfaces one.
* docs: refresh Fleet Dossier page against current app
Adds the page's first screenshot, documents the network exposure summary now included per stack, documents the storage-summary omission versus the Stack Dossier export, and adds two troubleshooting entries.
* docs(fleet-federation): deep rewrite for the rollout-approval gate
Federation's cordon and pin controls no longer take effect by
themselves: the reconciler requires a confirmed rollout preview
(Apply now -> Confirm Apply) before it mutates the fleet, and pinning
or unpinning always clears a blueprint's approval. Rewrote the mental
model, step-by-step, lifecycle table, security section, limitations,
practical workflows, troubleshooting, and cross-links to reflect that
gate. Also corrected the audit-log claims: cordon/uncordon/pin are not
filterable by a node.cordon/blueprint.pin action taxonomy, only by the
free-text search box against the method, path, and summary.
Replaced 4 screenshots and added 3 new ones (production fleet plus an
isolated instance to capture the rollout preview dialog in both a
safe and a blocked state), and fixed one stale pin-workflow sentence
in blueprint-model.mdx that the same audit surfaced.
* docs(fleet-federation): drop tier-availability framing
The feature isn't tier-gated, so calling out "every tier" reads as an
unnecessary advertisement. Removed the tier note and the Community/
Admiral prerequisite row, and reworded two role-visibility sentences
that had conflated tier with role.
* docs(fleet-actions): deep rewrite for the v2 action-card redesign
Rewrites the page against the shipped Fleet Action Card redesign: a
single cyan rail with per-card action-class chips instead of the old
per-card rose/purple/amber rails, plus the live blast-radius preview,
dry-run, and confirmed-target binding mechanics that replaced the old
static warning banners. Documents two previously-undocumented
endpoints (match-preview, prune/estimate), the preview-confirm-execute
model shared by all three cards, per-node locking, timeouts, and the
version-gated confirmed-target contract for mixed-version fleets. All
five screenshots recaptured against the live production UI.
* docs(fleet-actions): fix nested quote in screenshot alt text
* docs(fleet-sync): refresh against current app and document proxy gap
Retake all screenshots against the live fleet (control authoring view,
a genuine replica showing the read-only banner, and replicated
suppression rows) and document a previously-unwritten behavior: scan
policies, CVE suppressions, and misconfig acknowledgements are fetched
localOnly and are not proxied through the node switcher like most
other Security page tabs, so viewing a remote's Fleet Sync state
requires signing into that instance directly.
* docs(fleet-backups): refresh against current app and note new integration points
Replaces all 6 screenshots with fresh 1920x1080 production captures and
corrects several drifted details: the Cloud Snapshots panel's pagination
and per-item delete action, the exact Recovery Vault storage-mode label,
and the real per-file size-cap behavior (an oversized compose.yaml skips
the whole stack; an oversized .env is dropped but the stack still
captures). Documents two entry points that were missing from the page:
the pre-update snapshot checkbox in Health-Gated Updates and the snapshot
coverage nudge on the Storage Portability tab. Expands the single warning
banner description in the detail view to cover all four banner types,
notes the Snapshots tab and Recovery Vault are hub-only, and adds a
Where Fleet Backups fits table cross-linking the six adjacent features.
* docs(remote-updates): refresh against current app and add hardened-channel notes
Replace all screenshots with fresh 1920x1080 production captures and fix the
Node Updates trigger label (Check Updates -> Node Update). Add the Changelog
tab, correct the reconnect overlay's stale 'Update timed out'/Try Reloading
copy to the current Taking longer than expected/Reload to check text, and
document the Admiral Hardened Build channel's carve-outs (pinned-but-not-blocked,
entitlement-gated local update path). Disambiguate this page from the
unrelated top-level Update (Health-Gated Updates) nav tab.
* docs(node-compatibility): refresh against current app and expand capability list
Replace all screenshots with fresh production captures (node switcher,
capability lock card, connection test panel) and swap the lock-card example
from a now-hidden Host Console to Audit Log, which is directly reachable.
Bring the capability table from 26 to 35 entries to match the current
registry, add a Mute Rules row, and split out fine-grained fallback
capabilities (update-guard, service-scoped-update, cross-node-rbac,
stack-down-remove-volumes) with their non-lock-card fallback behavior.
Fix the compose-networking row, which incorrectly claimed to also gate the
node-level Networking overview. Note that Pilot Agent nodes never advertise
host-console regardless of version.
* docs(security): refresh against current app and document scan-node launcher
Replaces all screenshots with fresh production captures, documents the
Scan this node launcher and the third Scanner setup toggle (pre-deploy
scan advisory), tightens the Compose risks example list and the Policies
block-condition description to match the live app, and adds an On a
phone section.
* docs(vulnerability-scanning): refresh for exploit intel and risk-based policies
Documents exploit intelligence (KEV/EPSS evidence tags), the redesigned
risk-based scan policies (severity/known-exploited/fixable block
conditions replacing the single max-severity field), and the new
Findings badge state. Replaces all screenshots with fresh production
captures and adds a Secrets tab example.
* docs: refresh CVE suppressions page
Retook all screenshots against production (prior ones predated the
triage-status/OpenVEX UI and the edit capability). Documented the
suppression edit flow, which the page previously described as
delete-and-recreate only. Corrected the remote-node banner copy on the
Suppressions tab and added a screenshot of it. Cross-linked the
Misconfig acknowledgements panel that now sits on the same tab.
* docs(two-factor-authentication): refresh screenshots and document rate-limit layers
Replace all 11 non-count-variant screenshots with fresh production captures.
Document the throttled sign-in state precisely (kicker/hero/caption change,
not just the input), that a replayed TOTP counts toward the lockout counter,
and the separate per-network-address sign-in rate limit that sits in front
of the per-account MFA lockout.
* docs(api-tokens): refresh screenshots and document service-scoped deploy actions
Deploy Only now authorizes eight lifecycle POSTs (was six): the per-service
update and restore endpoints were missing from the scope table. Universal
restrictions gained a row for image channel management, which was already
rejecting API tokens in code but undocumented. Replaced all five screenshots
with fresh captures showing the current three-scope create flow and a
populated list with one token of each scope.
* docs(upgrade): refresh against current app and remove legacy-version framing
Fixes migration-list inaccuracies (registry credentials were never
plaintext, unlike node API tokens), rephrases the SSH/TLS and JSON-config
migration bullets to drop version-numbered legacy framing, adds a GHCR
mirror note matching the quickstart pattern, and cross-links the Hardened
Build entitlement-gated update path so digest-pinned installs aren't sent
down the manual docker pull steps.
* docs: refresh Backup & Restore against current backend
Broadens the encryption.key warning to cover everything CryptoService
actually encrypts (node tokens, Git source and Fleet Secrets, SSO/MFA,
Recovery Vault, fleet snapshot contents), not just registry credentials.
Adds the built-in backupData CLI as a no-host-tooling alternative to the
sqlite3 .backup command, cross-linked to Emergency command-line recovery.
Corrects node-token migration guidance: tokens are signed with a secret
stored in sencho.db and remain valid after a host move, so no
regeneration is required.
* docs: refresh Recovery guide against current Settings page and CLI
Documents the Settings · Recovery page's full System health / Environment /
Safe actions / Command-line hub instead of only its environment-preflight
section, adds the missing "Sencho compose location" preflight check, adds the
SSO/OIDC/LDAP lockout scenario now that disableSso.js exists, and corrects the
rollback description (exact UI label, backup-required and image-layer caveats).
* docs(emergency-cli): sync in-app page description, add CLI operational details
Aligns the "in-app Recovery page" summary with the freshly refreshed Recovery
guide (System health / Environment / Safe actions / Command-line recovery, SSO
providers, one-click command download). Adds constraints verified against the
CLI source: password/username minimums, the valid SSO provider identifiers with
an example, and a note that diagnostics.js always reports Docker as unreachable
(it runs without a live Docker connection, unlike the in-app page).
* docs(troubleshooting): refresh against current backend and nav
Corrects several claims that had drifted from the implementation:
network name validation (underscore is not a valid leading character),
the remote-update delayed-failure window (3 minutes, not 90 seconds),
and an unsubstantiated version-compatibility check on the Admiral 403
entry. Removes two entries describing legacy pre-v0.39 node behavior
that no longer applies to any currently shipped build. Replaces stale
"Profile > Settings > X" navigation with the current Settings hub
paths, and "wifi icon" with the current Test Connection label. Adds
Pilot Agent awareness to the remote-offline entry with a cross-link to
its dedicated troubleshooting section, and trims the docker-run
converter entry to point at the fuller, already-current version on the
Stack Management page instead of duplicating it with a broken anchor.
* docs(verifying-images): document the :dev GHCR integration tag
Verified the existing cosign/SBOM/VEX/tag-policy content against
docker-publish.yml and docker-preview.yml; all accurate. Added the
previously undocumented :dev/:dev-<sha> integration tag published on
every push to main via docker-dev.yml.
* docs(trivy-setup): sync install guide with node-scoped scanner and current Trivy upstream docs
Documents that Trivy installs independently per node, adds the TRIVY_BIN
override and non-PATH mounting option, fixes the deprecated apt-key install
flow and RHEL repo gpgkey placement, notes the Exploit intelligence toggle
for air-gapped hosts, and replaces the stale Scanner setup screenshot.
* docs(two-factor-admin): document lockout recovery and self-reset gotcha
Verified the reset flow, CLI recovery, and SSO toggle claims against
current source and replaced the stale admin-reset modal screenshot.
Added the failed-attempt lockout behavior (undocumented despite being
promised in the page description) and a warning about resetting your
own 2FA from the Users list, which signs you out immediately unlike
the self-service disable flow.
* fix(docs): make Settings and Self-hosting discoverable in the sidebar
The Reference and Operations groups used root pages that were only
reachable by clicking the ambiguous group label, and that click also
triggered an unwanted double action (expand plus navigate). List both
pages as explicit sidebar entries and disable global drilldown so group
headers only expand or collapse.
* docs(settings): sync reference page with current Settings Hub
Renamed License to Admiral Account throughout (Hardened Build channel
switch, image channel display, DURATION pill), corrected the password
policy, documented the new Appearance navigation and log-chip-color
controls plus font options, noted Mesh data plane is not on every
installation, fixed the label cap (50, not 100), and replaced every
screenshot with a fresh capture from the production node.
* fix(docs): apply the sidebar toggle-only fix to Start here and Product guide
These two groups had the same click-to-navigate-and-expand pattern as
Reference and Operations. Flattening them is safe here too: the
Product guide root page has no directory listing depending on it, and
the Start here root page's own hand-authored Next steps CardGroup
already covers the same links the auto-generated listing duplicated.
* docs(licensing): refresh licensing page for Hardened Build and sales-led Admiral pricing
Admiral pricing moved from self-serve checkout to a contact-sales model, and
a new Hardened Build image-channel switcher shipped in the Admiral Account
settings page; neither was reflected in the docs. Also documents the
lifetime-license edge case (no Manage subscription button, no Billing row)
and refreshes all four screenshots against the current UI.
* docs(security): sync reference page with current API-token and encryption scope
Verified every claim against the live implementation: the API-token universal
restrictions list was missing MFA management, Recovery Vault, and image
channel management; the encryption-at-rest field list was missing Recovery
Vault credentials; added a note on the password strength indicator's
recommended-vs-enforced distinction. Refreshed the SSO settings, API tokens,
and audit log screenshots against the production node.
* docs(settings): fix password policy and session-invalidation claims
Cross-checked against auth.ts while verifying the same claims on the
security reference page: the enforced minimum is 8 characters (the
12+/mixed-case/number text is a frontend strength hint, not a validated
rule), and a password change invalidates every other session for the
account rather than leaving them valid.
* docs(contact): sync contact channels with sales-led pricing and current support gating
Replace the retired contact@sencho.io with hello@sencho.io (the address actually
used in the website footer and the pricing page's Get in touch CTA), narrow the
licensing@sencho.io scope to existing-license activation/billing/refunds now that
new Admiral conversations route through hello@sencho.io, drop the stale LICENSE-file
and in-app upgrade-prompt claims (the AGPLv3 relicense removed both), correct the
Support settings path and add the published response-time targets, and remove the
unsubstantiated bug bounty mention.
365 lines
29 KiB
Plaintext
365 lines
29 KiB
Plaintext
---
|
||
title: Pilot Agent
|
||
description: Connect a remote Sencho node to your control instance through a single outbound WebSocket tunnel. No inbound ports, no TLS certificates, no public URL on the remote host.
|
||
---
|
||
|
||
Pilot Agent is one of Sencho's two remote-node modes. A small agent container runs on the remote host, dials your control instance over a single outbound WebSocket, and holds that connection open. Every request the control instance sends to that node rides through the same tunnel (HTTP, WebSocket, and, when that limited-availability surface is present, mesh TCP). The remote host never opens an inbound port and never needs a TLS certificate of its own.
|
||
|
||
This page is the architecture-and-operations reference for Pilot Agent. For the step-by-step UI walkthrough that adds a remote node, see [Multi-Node Management](/features/multi-node).
|
||
|
||
## How Pilot Agent fits Sencho
|
||
|
||
Sencho splits responsibilities between a **control instance** (the browser-facing Sencho your operators log into) and **nodes** (Docker hosts you manage from that control instance). The local node is always the control instance's own Docker socket. Remote nodes are other Sencho installations that the control instance manages on your behalf.
|
||
|
||
There are two remote modes, chosen per node when you register it:
|
||
|
||
<Frame>
|
||
<img src="/images/pilot-agent/01-add-node-pilot.png" alt="Add remote node modal with Type set to Remote and the Mode dropdown open, showing the Pilot Agent option above the Distributed API Proxy option." />
|
||
</Frame>
|
||
|
||
- **Pilot Agent** (this page). The remote dials the control instance. Outbound HTTPS from the remote is the only network requirement.
|
||
- **Distributed API Proxy** ([documented under Multi-Node Management](/features/multi-node)). The control instance dials the remote. The remote must expose a reachable URL.
|
||
|
||
Both modes are first-class. They can coexist on the same control instance, one per node. Pilot Agent is part of Sencho's Community-tier core surface; there is no license gate on using it.
|
||
|
||
## When to choose Pilot Agent
|
||
|
||
Pilot Agent is the right choice when the remote host:
|
||
|
||
- Sits behind NAT or a residential router with no port forwarding.
|
||
- Lives on a corporate network that blocks inbound connections.
|
||
- Has a rotating cloud IP, a CG-NAT'd mobile uplink, or roams between networks.
|
||
- Has only outbound internet access (a typical homelab or office workstation).
|
||
- Would otherwise need a reverse proxy, dynamic DNS, or a self-signed TLS certificate purely to be reachable.
|
||
|
||
Choose **Distributed API Proxy** instead when the remote already has a stable, reachable URL (a VPS with a public IP, a LAN server with a static address, or a Tailscale node you have already terminated TLS on). The [Multi-Node Management](/features/multi-node) page compares both modes side by side.
|
||
|
||
You do not have to pick one mode for the whole fleet. Mix and match per node.
|
||
|
||
## How it works
|
||
|
||
Conceptually, the agent reverses the usual client/server direction.
|
||
|
||
1. The agent container starts on the remote host with its mode, control instance URL, short-lived enrollment token, and compose directory configured.
|
||
2. It dials `wss://<control-instance>/api/pilot/tunnel` and holds the WebSocket open for as long as the container runs.
|
||
3. For every request the control instance needs to make against that node (listing containers, deploying a stack, streaming logs, opening a console, forwarding mesh traffic), frames are multiplexed over that single connection.
|
||
4. On the agent side, those frames are demultiplexed and replayed locally against the host's Docker socket and filesystem.
|
||
|
||
Inside the control instance, the tunnel terminates at a **loopback bridge**: a tiny HTTP server on `127.0.0.1:<random-port>` per active tunnel. Every other Sencho feature (the existing HTTP proxy, WebSocket forwarder, mesh dialer, license-tier propagation) treats the bridge as just another remote URL. Once enrolled, the pilot node shows up in the same Nodes table, the same node switcher, and the same dashboard as a proxy-mode node, with the same shipped capabilities except where pilot mode intentionally narrows the surface (see below).
|
||
|
||
<Frame>
|
||
<img src="/images/pilot-agent/03-nodes-table-tunnel.png" alt="Settings · Infrastructure · Nodes table showing a Local row and a Pilot Agent row. The pilot row has Mode badge 'Pilot Agent', Endpoint 'tunnel (seen 45m ago)', and Status badge 'Online'." />
|
||
</Frame>
|
||
|
||
What rides through the tunnel:
|
||
|
||
- HTTP requests (every API call and stream).
|
||
- WebSockets (live logs, container exec, notifications).
|
||
- Sencho Mesh TCP streams (cross-node service-to-service traffic).
|
||
|
||
What does NOT ride through the tunnel:
|
||
|
||
- Nothing inbound. The agent opens no listening port.
|
||
- The agent serves no UI of its own. Operators always work from the control instance.
|
||
- The agent's internal `/api/health` endpoint is reachable only from the tunnel's loopback bridge, not from outside the agent container.
|
||
|
||
### Intentional non-parity with proxy mode
|
||
|
||
Pilot mode narrows the operator surface in a couple of places where parity with proxy mode would be misleading:
|
||
|
||
- **Host console is not available on pilot nodes.** A pilot agent runs as a container with no useful shell into the underlying host, so the in-app host console is disabled at the capability level. If you need a host shell on a remote, use proxy mode for that node, or SSH in directly.
|
||
- **Self-update on pilot nodes recreates the agent container in place via the compose file the operator wrote during enrollment.** This is the same code path proxy-mode nodes use; it works as long as the host runs the compose file from the same directory across updates.
|
||
|
||
## Prerequisites
|
||
|
||
| Requirement | Detail |
|
||
|---|---|
|
||
| **Remote host** | Docker installed and a user that can run containers. About 1 GB free on the volume that backs `/app/data` and the selected compose directory. |
|
||
| **Network from remote → control instance** | Outbound TCP to the control instance's HTTPS port (whatever your reverse proxy or Sencho exposes). Nothing inbound. |
|
||
| **Network at the control instance** | The path from the remote to the control instance must allow WebSocket upgrades end-to-end. If you terminate TLS at a reverse proxy in front of Sencho, that proxy needs `Upgrade: websocket` passthrough. |
|
||
| **Operator role** | Admin on the control instance. The Settings → Nodes panel is admin-only. |
|
||
| **License tier** | None. Pilot Agent is Community-tier core surface. |
|
||
|
||
### The control instance's public URL
|
||
|
||
When the enrollment dialog generates the agent's compose file, it bakes the control instance's URL into `SENCHO_PRIMARY_URL` by reading the Host header on the admin's request. That works on a LAN, but it breaks when the pilot lives on a different network (a public cloud VPS, for example) and the admin happened to open the dialog at `http://127.0.0.1:1852` or a LAN address: the baked-in URL is then unreachable from the pilot.
|
||
|
||
Set `SENCHO_PUBLIC_URL` on the control instance to lock in a publicly-routable URL the dialog should always use:
|
||
|
||
```bash
|
||
-e SENCHO_PUBLIC_URL=https://sencho.example.com
|
||
```
|
||
|
||
Any HTTPS hostname reachable from the pilot works: a Cloudflare Tunnel, a reverse proxy, a port-forwarded home network with DDNS. The control instance validates the value (must be `http(s)://`, no loopback) and falls back to the request Host when unset.
|
||
|
||
## Enrollment and lifecycle
|
||
|
||
The end-to-end UI walkthrough for adding a Pilot Agent node (click paths, screenshots of the Add Node form, the Verify-the-tunnel step) lives in [Multi-Node Management](/features/multi-node#add-a-remote-node-pilot-agent). This section covers the underlying credential lifecycle, what happens during the connection, and how reconnects behave.
|
||
|
||
### Enrollment
|
||
|
||
When you submit the Add Node form with Mode set to Pilot Agent, the control instance:
|
||
|
||
1. Inserts the node into its registry with status `unknown` and no endpoint.
|
||
2. Mints a one-time enrollment JWT scoped to that node. The token's hash is stored on the control instance so it cannot be replayed; the token itself is returned to the operator and embedded in a generated Docker Compose file.
|
||
3. Shows the compose file in an enrollment dialog with a 15-minute countdown.
|
||
|
||
<Frame>
|
||
<img src="/images/pilot-agent/02-enrollment-dialog.png" alt="Enroll the pilot agent modal with kicker 'NODES · PILOT ENROLLMENT'. Step 1 saves a Docker Compose file as 'compose.yaml' with the env block showing SENCHO_MODE: pilot, SENCHO_PRIMARY_URL: https://sencho.example.com, and SENCHO_ENROLL_TOKEN: <short-lived token>. Step 2 shows the 'docker compose up -d' command. Footer reads 'Expires 14m from now' alongside a 'Copy compose file' button." />
|
||
</Frame>
|
||
|
||
The enrollment token is one-time and short-lived by design. If a token is intercepted or leaked, the window of risk is small and the slot is consumed on first use.
|
||
|
||
The generated compose file looks like this, with the Pilot connection and compose path already configured:
|
||
|
||
```yaml
|
||
name: sencho-agent
|
||
services:
|
||
agent:
|
||
image: saelix/sencho:latest
|
||
container_name: sencho-agent
|
||
restart: unless-stopped
|
||
volumes:
|
||
- /var/run/docker.sock:/var/run/docker.sock
|
||
- sencho-agent-data:/app/data
|
||
- type: bind
|
||
source: /opt/docker/sencho
|
||
target: /opt/docker/sencho
|
||
environment:
|
||
SENCHO_MODE: pilot
|
||
SENCHO_PRIMARY_URL: https://sencho.example.com
|
||
SENCHO_ENROLL_TOKEN: <short-lived token>
|
||
COMPOSE_DIR: /opt/docker/sencho
|
||
|
||
volumes:
|
||
sencho-agent-data:
|
||
```
|
||
|
||
Save it as `compose.yaml` on the remote host and bring the agent up from the same directory:
|
||
|
||
```bash
|
||
docker compose up -d
|
||
```
|
||
|
||
Deploying through Compose also lets the control instance push over-the-air updates to the agent later from the Fleet view ([Remote Updates](/features/remote-updates)), without manual intervention on the remote host. Compose-managed containers carry the labels Sencho needs to recreate them in place.
|
||
|
||
### First connect
|
||
|
||
The agent boots, reads its configuration, and dials `wss://<control-instance>/api/pilot/tunnel` carrying the enrollment token in an `Authorization: Bearer` header.
|
||
|
||
The control instance verifies the token, marks the enrollment slot consumed, and replies with a `hello` frame followed by a control frame that carries a **long-lived tunnel JWT** (365-day expiry). The agent writes that token to its data volume at `/app/data/pilot.jwt`. The control instance will not accept the same enrollment token again, but leave `SENCHO_ENROLL_TOKEN` in the compose file: the agent uses it as a fallback if the persisted tunnel JWT is ever rejected at upgrade (for example after a secret rotation), clearing the stale file and re-enrolling automatically.
|
||
|
||
The tunnel is now active. The Endpoint column in the Nodes table flips from `tunnel (waiting)` to `tunnel (seen Xs ago)` and the node's status badge turns Online.
|
||
|
||
### Subsequent reconnects
|
||
|
||
On every later boot, restart, or transient network drop, the agent reads `pilot.jwt` from disk and dials with that. No new operator action is needed. The persisted token is good for a year from the original enrollment.
|
||
|
||
The reconnect loop uses exponential backoff from 1 second to 60 seconds with up to 500 ms of jitter. The backoff resets only after a successful `hello` round-trip, so an agent that always fails the handshake (wrong URL, revoked token, version mismatch) cannot tight-loop and burn resources.
|
||
|
||
The control instance pings every connected agent every 30 seconds. Any stream with no activity for 10 minutes is closed and reclaimed.
|
||
|
||
### Primary restart, agent restart
|
||
|
||
- **Control instance restarts**: every agent's tunnel drops cleanly. Each agent reconnects within seconds. In-flight HTTP requests fail and the browser retries; long-lived WebSockets (logs, console) reconnect.
|
||
- **Agent restarts**: the persisted token re-authenticates without operator intervention. Same reconnect-and-recover behavior.
|
||
- **Operator deletes the node on the control instance**: the agent's existing tunnel is closed. The agent will keep trying to reconnect with its persisted token, which the control instance no longer accepts; the agent stays in a reconnect-backoff loop. To stop it, tear the agent down on the remote with `docker compose down -v`.
|
||
|
||
### Re-enrollment
|
||
|
||
If the agent container is destroyed before its first successful connect, the enrollment token expires unused, the control instance is restored from a backup with a different JWT secret, or you simply want to rotate the credential, open the node's Edit dialog and click **Regenerate enrollment token**.
|
||
|
||
<Frame>
|
||
<img src="/images/pilot-agent/04-regenerate-token.png" alt="Edit node modal for a pilot-agent node showing a Re-enroll the agent helper text and a Regenerate enrollment token button." />
|
||
</Frame>
|
||
|
||
A fresh 15-minute token is issued and the previous tunnel (if any) is closed immediately so the new agent replaces it cleanly.
|
||
|
||
## Security and trust model
|
||
|
||
### Tokens
|
||
|
||
Two JWT scopes, both signed with the control instance's `auth_jwt_secret`:
|
||
|
||
| Token | Lifetime | Where it lives | Use |
|
||
|---|---|---|---|
|
||
| **Enrollment** | 15 min, one-time | Returned to the operator; SHA-256 hash stored on the control instance | Bootstraps the first connect |
|
||
| **Tunnel** | 365 days | Agent's container volume at `/app/data/pilot.jwt` | Every reconnect after the first |
|
||
|
||
Enrollment is rate-limited to 10 requests per minute per operator (per source IP if unauthenticated) so a runaway script cannot exhaust the database with pending tokens.
|
||
|
||
### TLS
|
||
|
||
The agent dials `wss://`. Certificate validation is on by default against the system trust store. There is no flag to disable verification; that would defeat the credential model. For internal CAs, mount your bundle into the agent container and set `SENCHO_PILOT_CA_FILE` to the in-container path. See [Self-signed control-instance TLS certificates](#self-signed-control-instance-tls-certificates) below for a worked example.
|
||
|
||
### Trust boundaries
|
||
|
||
- **The control instance trusts the agent.** A connected agent's frames are believed. A compromised remote can flood frames; the per-tunnel resource limits documented below bound the blast radius.
|
||
- **The agent trusts the control instance.** The control instance proves its identity by producing a valid JWT under the secret it shares with itself. A compromised control instance can ask the agent to do anything the agent could do directly on its host, and the agent holds the Docker socket, so that is effectively root on the remote.
|
||
- **Tier-trust headers (`x-sencho-tier`, `x-sencho-variant`) are only honored for bearer tokens with `node_proxy` or `pilot_tunnel` scope.** Browser session cookies cannot spoof them.
|
||
- **The agent exposes nothing inbound.** No port. No UI. No directly reachable health endpoint.
|
||
|
||
### What rotating the JWT secret does
|
||
|
||
The tunnel JWTs are signed with the control instance's `auth_jwt_secret`. If that secret rotates (because the control instance was rebuilt from scratch, restored into a different environment, or manually rotated), every existing tunnel JWT stops verifying. Regenerate enrollment for each affected node, update the agent compose file with the fresh token, and restart the agent containers. When `SENCHO_ENROLL_TOKEN` is present in the running container, the agent clears the stale `pilot.jwt` and re-enrolls on its own; you do not need to delete the data volume by hand.
|
||
|
||
## Self-signed control-instance TLS certificates
|
||
|
||
If your control instance terminates TLS with an internal CA that the agent's system trust store does not recognise, mount the CA bundle into the agent container and point `SENCHO_PILOT_CA_FILE` at it:
|
||
|
||
```yaml
|
||
name: sencho-agent
|
||
services:
|
||
agent:
|
||
image: saelix/sencho:latest
|
||
container_name: sencho-agent
|
||
restart: unless-stopped
|
||
volumes:
|
||
- /var/run/docker.sock:/var/run/docker.sock
|
||
- sencho-agent-data:/app/data
|
||
- type: bind
|
||
source: /opt/docker/sencho
|
||
target: /opt/docker/sencho
|
||
- /etc/ssl/internal-ca.pem:/etc/ssl/internal-ca.pem:ro
|
||
environment:
|
||
SENCHO_MODE: pilot
|
||
SENCHO_PRIMARY_URL: https://sencho.internal.example.com
|
||
SENCHO_ENROLL_TOKEN: <short-lived token>
|
||
SENCHO_PILOT_CA_FILE: /etc/ssl/internal-ca.pem
|
||
COMPOSE_DIR: /opt/docker/sencho
|
||
|
||
volumes:
|
||
sencho-agent-data:
|
||
```
|
||
|
||
The agent uses the bundle as the only trust anchor for the tunnel WebSocket. TLS verification stays on.
|
||
|
||
## Resource limits
|
||
|
||
Each pilot tunnel has fixed protocol-level ceilings so one misbehaving agent cannot impact the control instance or its peers.
|
||
|
||
| Limit | Value | What it caps | Why it matters |
|
||
|---|---|---|---|
|
||
| **Frame size** | 8 MB | Individual WebSocket frame, including a single HTTP request or response body chunk | A request or response body larger than 8 MB is rejected and the tunnel reconnects. Stream HTTP responses are chunked, so normal traffic stays well under this. |
|
||
| **Concurrent streams per tunnel** | 1024 | HTTP, WebSocket, and Sencho Mesh TCP streams share the same pool | New streams above the cap are rejected with 503 from the bridge and a typed error frame from the agent. A runaway script or a long-poll session that leaks streams can exhaust this; reload the affected tab or throttle the caller. |
|
||
| **Stream idle timeout** | 10 minutes | Per-stream inactivity before teardown | Idle WebSockets and forgotten HTTP streams are reclaimed automatically. |
|
||
| **System-wide tunnels** | 256, soft-warn at 128 | Concurrent pilot tunnels on a single control instance | Past the soft warning the control instance logs WARN. Past the hard cap, new tunnels are refused with WebSocket close code 1013 ("Try Again Later"); the agent backs off rather than tight-looping. |
|
||
| **Tunnel write buffer pause** | 4 MB | High-water mark for the agent-bound write queue | When the control instance's queue is over the mark, inbound HTTP request bodies are paused and resumed on drain. Backpressure rather than memory growth. |
|
||
| **Enrollment rate** | 10 per minute | Operator-issued enrollment tokens | Protects the database against a script accidentally minting tokens in a loop. |
|
||
|
||
An admin-only diagnostic endpoint, `GET /api/system/pilot-tunnels`, returns counters and a per-node breakdown of active tunnels (connect time and `bufferedAmount` for each). Useful for triaging a single sticky tunnel inside an otherwise healthy fleet.
|
||
|
||
## Environment variables
|
||
|
||
These environment variables are read by the **agent** container at boot.
|
||
|
||
| Variable | Required? | Default | Notes |
|
||
|---|---|---|---|
|
||
| `SENCHO_MODE` | Yes | none | Must be `pilot`. Putting any other value here causes the container to start as a normal Sencho instance, not as an agent. |
|
||
| `SENCHO_PRIMARY_URL` | Yes | none | The base URL of your control instance (e.g. `https://sencho.example.com`). The agent appends `/api/pilot/tunnel` and dials `wss://`. |
|
||
| `SENCHO_ENROLL_TOKEN` | First boot; keep for recovery | none | The 15-minute enrollment token issued by the control instance. During normal operation the agent dials with `pilot.jwt` instead. If that persisted token is rejected at upgrade (for example after a secret rotation or node re-registration), the agent falls back to this value automatically when it is still present in the container environment, clears the stale file, and re-enrolls. Update it in compose after regenerating enrollment on the control instance. |
|
||
| `SENCHO_PILOT_CA_FILE` | Optional | unset | Absolute path inside the container to a PEM bundle. Use when your control instance's TLS chain is rooted in a private CA. |
|
||
| `DATA_DIR` | Optional | `/app/data` | Where the persisted `pilot.jwt` is stored. Override only if you are mounting a different volume layout. |
|
||
| `COMPOSE_DIR` | Optional | `/app/compose` | Root directory where compose stack folders live. Enrollment sets it to the absolute path selected for the node and mounts that path identically on the host and in the agent. |
|
||
|
||
One variable is read on the **control instance** side, not the agent:
|
||
|
||
| Variable | Required? | Default | Notes |
|
||
|---|---|---|---|
|
||
| `SENCHO_PUBLIC_URL` | Optional | request Host | Locks in the URL the enrollment dialog bakes into `SENCHO_PRIMARY_URL`. See [The control instance's public URL](#the-control-instances-public-url) above. Must be `http(s)://`, no loopback. |
|
||
|
||
The agent does not currently expose tunables for the ping interval (30 s), the reconnect backoff (1–60 s), or the per-stream idle timeout (10 min). These are deliberate fixed values; if your environment requires different ones, open an issue.
|
||
|
||
## Limitations and non-goals
|
||
|
||
These are the boundaries operators should know about before designing a fleet around Pilot Agent.
|
||
|
||
- **No mode conversion.** The Edit dialog shows a Mode field for an enrolled node, but switching a node between Pilot Agent and Distributed API Proxy after enrollment leaves the credentials and connection state inconsistent. To change modes, delete the node and re-create it in the desired mode.
|
||
- **No audit log entries for enrollment lifecycle.** Node creation, enrollment regeneration, and node deletion do not write to the audit log today. This is on the roadmap.
|
||
- **`pilot.jwt` is not cleaned up on node deletion.** When you delete a node from the control instance, the agent's persisted token stays on the remote's data volume. The agent will fail to reconnect on next restart, but the file persists. If you are repurposing the host, tear the agent down with `docker compose down -v` to remove the `sencho-agent-data` volume.
|
||
- **JWT-secret rotation invalidates existing tunnel JWTs.** Rebuilding the control instance from scratch or rotating `auth_jwt_secret` requires a fresh enrollment token on each agent. When that token is still in the container environment, the agent re-enrolls automatically without manual deletion of `pilot.jwt`.
|
||
- **One tunnel per node.** Splitting a node's load across multiple control instances or running multiple agent containers against the same control instance for the same node is not supported.
|
||
- **Mesh and pilot share the per-tunnel stream pool.** A node that runs heavy Sencho Mesh traffic counts those streams against the same 1024-stream cap as HTTP and WebSocket traffic.
|
||
- **The agent has no UI of its own.** All operation flows through the control instance. The agent's container logs (`docker logs sencho-agent`) are the only direct visibility into agent-side behaviour.
|
||
|
||
## Sencho Mesh on Pilot Agent
|
||
|
||
When the limited-availability Mesh surface is present, mesh TCP can ride the existing pilot tunnel instead of opening a separate connection. Stack opt-in is per stack per node and is not affected by the transport choice. See [Sencho Mesh](/features/sencho-mesh) for the full mesh model.
|
||
|
||
## Troubleshooting
|
||
|
||
The generic node-connectivity issues (a node showing Offline, a pilot agent stuck on `tunnel (waiting)`, 401 Unauthorized on Test Connection) are covered in the [Multi-Node Management troubleshooting section](/features/multi-node#troubleshooting). The entries below cover failure modes that are specific to Pilot Agent.
|
||
|
||
<AccordionGroup>
|
||
<Accordion title="Tunnel keeps disconnecting every ~30 seconds">
|
||
The control instance pings the agent every 30 seconds; a reconnect that lines up with the ping interval almost always means an HTTP proxy or load balancer between the agent and the control instance is closing idle WebSockets faster than the ping cadence. Raise the proxy's WebSocket idle timeout above 30 seconds (a 90-second floor is safe). If you terminate TLS at a reverse proxy, also confirm that the proxy forwards the `Upgrade: websocket` header end-to-end.
|
||
</Accordion>
|
||
|
||
<Accordion title="Tunnel rejected with WebSocket close code 1013 (Try Again Later)">
|
||
The control instance has reached the system-wide hard cap of 256 concurrent pilot tunnels. A reconnect storm or a runaway enrollment is the usual cause. Inspect the control instance's logs for the `[Pilot] Active tunnel count at soft limit` warning that fires at 128 to spot the trend, and check `GET /api/system/pilot-tunnels` (admin-only) for the per-node breakdown to identify the flapping agent. The agent backs off and retries automatically once headroom returns.
|
||
</Accordion>
|
||
|
||
<Accordion title="Loopback bridge returns 503 'pilot tunnel: stream cap reached'">
|
||
A single tunnel has hit the per-tunnel 1024-stream cap. The most common trigger is a browser session that opened many long-poll streams without closing them, or a script firing parallel API calls without bound. Reload the affected tab so its WebSocket and SSE streams reset. If the cap is being hit by automation, throttle the caller. Sencho Mesh streams share the same pool, so a runaway mesh workload can also push the count up.
|
||
</Accordion>
|
||
|
||
<Accordion title="Tunnel closes with WebSocket close code 1002 (protocol error)">
|
||
The agent sent a frame larger than the 8 MB ceiling, or the wire decoder rejected a malformed frame. Run the control instance with developer mode enabled (Settings → Operations → Developer Diagnostics) to surface the diagnostic decode log, then check the logs for `[PilotBridge:diag] Malformed frame from agent`. The agent reconnects automatically; persistent failures usually mean a version mismatch between the control instance and the agent image.
|
||
</Accordion>
|
||
|
||
<Accordion title="I ran the enrollment command on the wrong host">
|
||
Tear the agent down on the wrong host (`docker compose down -v` from the directory you saved the compose file in), then regenerate the enrollment token on the control instance and deploy the new compose file on the correct host.
|
||
</Accordion>
|
||
|
||
<Accordion title="HTTP 429 'Too many enrollment requests'">
|
||
The enrollment endpoint is rate-limited to 10 requests per minute per operator (or per IP when unauthenticated). A script accidentally minting tokens in a loop trips this. Wait 60 seconds and retry. To bulk-enroll many nodes, space the requests out.
|
||
</Accordion>
|
||
|
||
<Accordion title="Control instance was restored from backup and the agent will not reconnect">
|
||
The persisted tunnel credential is signed with the control instance's `auth_jwt_secret`. If that secret was not part of the backup (or has been rotated for any other reason), existing tunnels stop verifying. Regenerate enrollment for each affected node from Settings → Nodes, update the agent compose file with the fresh `SENCHO_ENROLL_TOKEN`, and restart the agent containers. The agent removes the stale `pilot.jwt` and completes enrollment automatically when the new token is in the container environment.
|
||
</Accordion>
|
||
|
||
<Accordion title="I deleted the node on the control instance but the agent keeps trying to reconnect">
|
||
Deleting the node removes the routing entry on the control instance but does not touch the remote. The agent still holds `/app/data/pilot.jwt` and will keep dialing. Tear the agent down on the remote (`docker compose down -v` from the directory holding the compose file) so the `sencho-agent-data` named volume that stores the persisted token is removed.
|
||
</Accordion>
|
||
|
||
<Accordion title="Stacks tab on a pilot node shows nothing or fails to list">
|
||
The agent's compose directory is missing or not writable. Verify the selected absolute path exists, is owned by a user the agent container can write as, and is mounted at the same path on the host and inside the agent. The Compose Directory in Settings → Nodes controls the generated enrollment file.
|
||
</Accordion>
|
||
|
||
<Accordion title="I changed the Mode on a pilot node in Edit and now it is broken">
|
||
The Edit dialog shows the Mode field, but switching from Pilot Agent to Distributed API Proxy (or back) after enrollment is not a clean flow; the persisted credentials and the new mode end up inconsistent. Delete the node, re-create it in the mode you want, and re-enroll or re-paste the API token.
|
||
</Accordion>
|
||
</AccordionGroup>
|
||
|
||
## Common questions
|
||
|
||
<AccordionGroup>
|
||
<Accordion title="Do I need port forwarding or a public IP on the remote?">
|
||
No. The agent dials outbound; the remote needs no inbound port, no public IP, no DNS entry, and no TLS certificate of its own.
|
||
</Accordion>
|
||
|
||
<Accordion title="Is Pilot Agent the same as a VPN or Tailscale?">
|
||
No. A VPN gives every host on the network a routable address for arbitrary peer-to-peer traffic. Pilot Agent is a single-purpose tunnel between one agent and one control instance, carrying only Sencho's own protocol. The two are complementary; you can run Sencho over a VPN (using Distributed API Proxy mode against the VPN address) if you prefer.
|
||
</Accordion>
|
||
|
||
<Accordion title="Does my data go through Sencho's servers?">
|
||
No. The tunnel is a direct WebSocket between your remote host and your control instance. There is no Sencho-operated relay.
|
||
</Accordion>
|
||
|
||
<Accordion title="Can I run multiple agents on the same host?">
|
||
Yes. Each agent needs its own container name and its own data volume so the persisted `pilot.jwt` files do not collide. Enroll each one against a distinct node entry on the control instance.
|
||
</Accordion>
|
||
|
||
<Accordion title="Does the control instance need a public domain?">
|
||
No. The agent only needs to reach the control instance over whatever path your network provides: LAN, VPC, VPN, public internet, or a private tunnel. Set `SENCHO_PRIMARY_URL` to whatever URL works from the agent's vantage point.
|
||
</Accordion>
|
||
</AccordionGroup>
|
||
|
||
## Related
|
||
|
||
- [Multi-Node Management](/features/multi-node): the operator walkthrough for adding remote nodes in either mode, the Nodes table reference, and the cross-mode comparison.
|
||
- [Sencho Mesh](/features/sencho-mesh): cross-node networking that rides on the pilot tunnel.
|
||
- [Fleet view](/features/fleet-view): fleet-wide aggregation and node health across both modes.
|
||
- [Security architecture](/reference/security): credential model and trust posture across the rest of Sencho.
|