mirror of
https://github.com/Studio-Saelix/sencho.git
synced 2026-09-01 13:18:08 +00:00
Anso
275c654407
* feat(fleet): add self dev-build detection primitives Split compareLocalToRemoteTag into compareLocalToRemoteTagDetailed (returns the probe's primary digest alongside the match/update/error verdict) with compareLocalToRemoteTag now a thin wrapper, so a caller that needs both the verdict and the digest no longer has to probe the same mutable tag twice. Add detectSelfDevBuildUpdate, which compares the running container's own image against the rolling ghcr.io/studio-saelix/sencho-dev:dev tag using the new detailed comparison, laying the groundwork for surfacing dev-build updates in Fleet. * feat: add isSenchoDevRepository and isSenchoDevFloatingTag predicates Add two pure predicate functions to helpers/selfUpdateCompose.ts for identifying Sencho dev repository references and floating tag variants: - isSenchoDevRepository: checks if a reference is to the ghcr.io/studio-saelix/sencho-dev repository, including digest-pinned and dev-<sha> tag variants - isSenchoDevFloatingTag: checks if a reference is specifically the floating :dev tag on the Sencho dev repository (not digest-pinned, not immutable dev-<sha>) Both functions reuse existing parsing patterns (normalizeImageRepository for repository extraction, classifyImagePin idiom for digest and tag detection) to maintain consistency. Add comprehensive test coverage in self-update-compose.test.ts covering all specified test cases including edge cases (malformed refs, unrelated repos, digest pins, etc.). * feat(gitops): wire dev-build detection into MonitorService Adds a dev_build_update_available notification category and a new checkSenchoDevBuild() cycle in MonitorService that detects when the running container has fallen behind the rolling ghcr.io/studio-saelix/sencho-dev:dev build it is pinned to, using detectSelfDevBuildUpdate() and isSenchoDevFloatingTag(). Availability state is written unconditionally so the Fleet update affordance never depends on notification delivery succeeding, while a separate dedup key prevents re-notifying for a digest already announced. Also guards checkSenchoVersion() so a dev-repo pin no longer produces a false positive stable-release update notification. * feat(fleet): surface dev-image status and build availability Fleet's GET /update-status now reports isDevImage (any reference to the sencho-dev repository, including digest pins) and devBuildUpdateAvailable (the exact floating :dev tag with a newer build observed, read from the system-state key MonitorService already maintains). A dev-pinned local node forces updateAvailable to false and clears any stale stable-release skip, since that skip was computed before image-pin classification and would otherwise leak a bogus "Skipped" state onto a dev row. Made MonitorService's SENCHO_DEV_BUILD_AVAILABLE_KEY constant public so both call sites share one string instead of duplicating it. * fix(fleet): omit targetVersion for a dev-image update trigger updateRequestInit() always forwarded latestVersion (the latest stable release) as targetVersion whenever it was valid semver, even for a dev-pinned node. The backend already ignores targetVersion safely for a floating pin, so this never caused an actual repin, but it produced a misleading "Update to X.Y.Z" button label and confirm-dialog copy for an update that installs the dev image, not that stable release. * feat(fleet): add integration-image badge and dev build update button NodeCard now shows a persistent "Integration image" badge whenever a node's compose image is any sencho-dev reference, independent of update availability, visible to every role. When a newer dev build is available, a solid brand-colored "Update dev build" button appears alongside it, admin-only, reusing the existing update trigger and requireAdmin route. Styled distinctly from the neutral stable "Update to X.Y.Z" button so an operator always knows which channel they're acting on. * feat(fleet): add dev-image copy to the local update confirm dialog LocalUpdateConfirmDialog now recognizes isDevImage and shows a distinct LOCAL - DEV UPDATE kicker plus copy stating the sencho-dev:dev reference will be pulled without rewriting the compose image, and that integration images are unsigned and carry no release attestations. Without this, a dev-pinned node's update confirmation fell through to the generic "Pulls Sencho the latest release" copy. FleetView.tsx threads isDevImage from the node's update status through to the dialog, same source as its other pin fields. * feat(fleet): separate dev and stable availability in the Node Updates sheet The sheet counted stable and dev availability together via the same updateAvailable field, so a dev-pinned node with a build available fell into neither the summary counts nor any row action, and would have misleadingly rendered as "Up to date" once devBuildUpdateAvailable existed. stableAvailable and devAvailable are now tracked separately: the changelog dot lights only from stableAvailable (a dev build has no release changelog), the summary and meta text report the combined total, a dev row shows "Integration build" instead of a stable version in the Latest column, and the existing Update button/badge now also fires for devBuildUpdateAvailable. Update all and Skip stay stable-only, since both already gate on fields a dev row never satisfies. * feat(fleet): bring dev-build detection and update to Mobile Fleet Mobile Fleet previously had no update capability at all: it only polled /fleet/overview and never called useFleetUpdateStatus, so it could not show the stable update flow either. It now fetches update status alongside the overview poll, shows the same "integration" marker as desktop on any dev-pinned node's card (visible to every role), and gives admins a dev-build update action. The action renders as a sibling of the card's own button rather than nested inside it, since the card is itself a <button> and a nested button is invalid HTML with broken touch semantics. It reuses the exact same triggerNodeUpdate/confirmLocalUpdate flow and LocalUpdateConfirmDialog /ReconnectingOverlay components desktop already renders, so there is no parallel API implementation to keep in sync. * feat(notifications): wire dev_build_update_available through the frontend Adds the category to the frontend NotificationCategory union, its bell label, the per-node "mute update notifications" bundle, and the bell's friendly dot-color memo. The changelog navigation and "View changelog" button stay scoped to node_update_available only: a dev build has no release changelog entry to navigate to. * docs: document dev-build detection and update on Fleet Adds the dev_build_update_available notification category, the persistent Integration image marker, and the dev-build update action (desktop and mobile) to the alerts-notifications, verifying-images, fleet-view, remote-updates, and upgrade pages. States the detection cadence explicitly: it polls on a fixed interval and reflects the newest build observed, not necessarily every individual build. * fix(gitops): sanitize the inconclusive-reason debug log for log injection CodeQL flagged the dev-build check's debug log as depending on a user-influenced value (a registry probe failure reason can trace back to external input). Wraps it with sanitizeForLog(), the existing repo-wide remediation for this class of finding, matching how registry-api.ts already handles the same pattern. * test(gitops): cover the no-repin invariant on a dev-build self-update Proves triggerUpdate(), called with neither targetVersion nor targetImageRef (the exact dev-build update call), pulls the current compose-declared ref unchanged and never stages a compose rewrite. * fix(fleet): use the shared busy-button pattern on Mobile Fleet's dev update action Replaces the local Loader2 plus boolean pending logic with BusyButton so busy behavior and interaction locking stay in sync with the rest of the app's async click surfaces. * test(gitops): exercise the production call shape in the no-repin regression Fleet substitutes the stable compare target when the request body omits one, so SelfUpdateService receives a targetVersion even for a dev-build update. The guard that protects a :dev install is therefore the semver check inside the repin branch, not the absence of a target. Drives triggerUpdate with a forwarded target against a floating :dev pin and asserts the reference is pulled unchanged with no staged patch, and pairs it with a semver case so the negative assertions cannot pass vacuously.
166 lines
18 KiB
Plaintext
166 lines
18 KiB
Plaintext
---
|
|
title: Remote Updates
|
|
description: Pull the latest Sencho image and recreate the container on any node in the fleet, including the gateway, from the Fleet view.
|
|
---
|
|
|
|
Sencho can update any node in the fleet to the latest published release without ever opening an SSH session. The control instance opens the **Node updates** sheet, dispatches the update to one or more nodes, and watches each one come back online with the new version. Update availability is confirmed against a pullable Docker image on the public registry mirrors, not merely the GitHub Release timestamp, so the **Update** button does not appear until the release image has finished publishing.
|
|
|
|
This page covers the mechanism: prerequisites, what happens on a node during an update, how completion and failure are detected, and how to recover. The full UI tour for the Node updates sheet itself lives in [Fleet View](/features/fleet-view#node-updates).
|
|
|
|
<Note>
|
|
Triggering an update (per-row or **Update all**) requires the **admin** role. Viewer and operator roles can read update status but cannot dispatch updates.
|
|
</Note>
|
|
|
|
<Note>
|
|
This is the Sencho self-update feature, reached from the Fleet tab. The top-level **Update** nav item is a different feature: it tracks pending Docker image updates for your own stacks (Plex, Radarr, and the rest), not the Sencho binary itself. See [Health-Gated Updates](/features/health-gated-updates) for that page.
|
|
</Note>
|
|
|
|
## Prerequisites
|
|
|
|
A node can self-update when all of the following are true:
|
|
|
|
- It was deployed via **Docker Compose** with the published `saelix/sencho` image (the canonical install for both control instances and pilot agents).
|
|
- The Docker socket (`/var/run/docker.sock`) is mounted into the container so Sencho can drive Compose from inside.
|
|
- It runs a Sencho version that advertises the `self-update` capability. See [Node Compatibility](/features/node-compatibility) for the full capability list.
|
|
|
|
Nodes deployed with `docker run`, with hand-rolled systemd units, or with orchestrators like Kubernetes do not advertise `self-update`. Their **Update** button returns an error and they are skipped by **Update all**.
|
|
|
|
## Anatomy at a glance
|
|
|
|
Open the **Fleet** tab and click **Node Update** in the Overview toolbar to open the **Node updates** sheet. The sheet lists every registered node with its current and latest Sencho version, the cluster-wide summary, and the action buttons, split across two tabs: **Nodes** (the table and summary) and **Changelog** (release notes for the latest published version, pulled from GitHub Releases).
|
|
|
|
<Frame>
|
|
<img src="/images/fleet-view/fleet-node-updates.png" alt="Node updates sheet titled 'Node updates · 4 nodes · 0 updates available'. A Recheck button sits under the header, above Nodes and Changelog tabs. Four summary cards read '4 Up to date', '0 Available', '0 Updating', '0 Failed'. A 'Filter nodes…' search box sits above a table with columns Node / Type / Current / Latest / Status; all four rows (Local, Opsix, Pitt-Moba, SLX-Mars) show a green 'Up to date' badge at v0.95.0. The footer reads 'LATEST VERSION v0.95.0'." />
|
|
</Frame>
|
|
|
|
For the per-control breakdown of the sheet (header actions, summary cards, table columns, skip/unskip, changelog tab), see [Fleet View · Node Updates](/features/fleet-view#node-updates).
|
|
|
|
## Triggering a remote update
|
|
|
|
There are two surfaces that initiate an update on a remote node:
|
|
|
|
- The **Update** button on a row inside the Node updates sheet.
|
|
- The **Update to vX.Y.Z** outline button along the bottom of any online node card on the Fleet grid.
|
|
|
|
The **Skip** button next to Update hides the update prompt for that node and excludes it from **Update all** until a newer version is released; the row shows a **Skipped vX.Y.Z** badge with an **Unskip** button to bring the prompt back early. The skip is per-node and per-version: when the next Sencho release comes out, the skip clears automatically and the update surfaces again. Skipping and unskipping both require the admin role.
|
|
|
|
<Frame>
|
|
<img src="/images/fleet-view/node-card-opsix-online.png" alt="Node card for the Opsix remote showing an Online badge, the version chip 'v0.95.0', the Running / Stopped / Stacks counters (4 / 0 / 3), and CPU / RAM / Disk usage bars." />
|
|
</Frame>
|
|
|
|
When an update is available, this same card gains a warning **Update available** pill next to the version chip and an outline **Update to vX.Y.Z** button across the bottom (not pictured above, since every node in this fleet is currently up to date). Either surface dispatches the same backend call. While the call is in flight, the button reads **Triggering...** with a spinning icon, then settles into the **Updating** badge once the gateway has accepted the request.
|
|
|
|
The gateway switches to a fast 5-second polling loop while any node is in the `Updating` state, so the badge advances in near real time without waiting for the next 30-second fleet refresh.
|
|
|
|
## Reapply configuration
|
|
|
|
When a node is already on the current Sencho release, **Update** is hidden, but Compose-managed nodes still expose a **Reapply configuration** icon control (tooltip on hover) in the Node updates sheet. Use this after you change the node's on-disk Compose project (environment variables, mounts, ports, labels, limits, healthcheck, networks, or socket-proxy settings) and need Sencho to recreate itself from that project without selecting a newer release.
|
|
|
|
Reapply:
|
|
|
|
- Uses the same Compose project context as a version update (multiple Compose files, `env_file`, configs, secrets, and bind mounts).
|
|
- Preserves the image reference declared in Compose. Sencho does not resolve a newer release or rewrite the pin.
|
|
- Works for digest-pinned installs, because no repin step runs.
|
|
- Requires the admin role.
|
|
|
|
Confirming a local reapply opens a dialog that states the node will recreate from its current Compose configuration, the dashboard may briefly disconnect, no newer version is selected, and the configured image reference is not rewritten. Confirming a remote reapply uses the same required acknowledgement (recreate from current Compose, no newer version, no image rewrite) before the request is sent; Fleet then shows a **Reapplying** badge until the node restarts or the operation fails.
|
|
|
|
Eligible admins can also run the same procedure from the Compose editor: on Sencho's own stack the primary save action becomes **Save & Reapply**, which saves the file first, then opens the same confirmation and fleet reapply path.
|
|
|
|
## Updating the local (gateway) node
|
|
|
|
Updating the gateway is special because the dashboard is hosted by the very container that is about to restart. Clicking **Update** on the local row, or **Update to vX.Y.Z** on the Local card, opens a confirmation dialog (kicker **LOCAL · UPDATE**, title **Update local node**, with **Cancel** and **Update & restart** buttons) before anything happens on disk. The body text depends on how the compose file pins the image: for a semver pin it names the exact rewrite (for example, "This install pins `saelix/sencho:0.94.1`. Updating rewrites it to `saelix/sencho:0.95.0`..."); for a floating tag it reads more generally ("Pulls Sencho v0.95.0 and restarts the server..."). Both variants end with the same note that the dashboard briefly disconnects and reconnects automatically.
|
|
|
|
A node pinned to the `sencho-dev:dev` integration image gets a third variant instead: kicker **LOCAL · DEV UPDATE**, stating the `:dev` reference will be pulled and recreated without rewriting the pinned tag, plus a note that integration images are unsigned and carry no release attestations. Dev-build detection and its update action are local-node only: a remote node's dev-image status is not surfaced in Fleet, since a full compose image reference is not exposed for remotes (see [Pinned image tags](#pinned-image-tags)).
|
|
|
|
<Note>
|
|
If the local node is running the Admiral **Hardened Build** image instead of a Community image, clicking **Update** here runs a different, entitlement-gated switch flow instead of the compose-repin steps below: it requires an authenticated browser session (not an API token) and can fail with its own codes (`entitlement_denied`, `preflight_mismatch`, `compose_unavailable`, `registry_access_unavailable`) surfaced on the same **Failed** badge. See [Licensing · Feature breakdown](/features/licensing#feature-breakdown) for what Hardened Build is.
|
|
</Note>
|
|
|
|
After **Update & restart** is confirmed on a Community-channel install:
|
|
|
|
1. The browser captures the gateway's current boot timestamp from `/api/health`.
|
|
2. The server classifies the compose image pin. For semver pins it pulls the target release, stages a rewritten compose file, and spawns a short-lived helper container that copies the file onto the host and runs `docker compose up -d --force-recreate`.
|
|
3. A full-screen **Updating Sencho...** overlay takes over the browser tab. The overlay polls `/api/health` every 3 seconds and keeps the page from reloading until a *new* boot timestamp comes back, even if the API briefly responds during the pull.
|
|
4. Once a fresh boot timestamp is reported, the overlay reloads the page on the new version.
|
|
|
|
If the pull or compose rewrite fails before the container restarts, the overlay dismisses within a few seconds and surfaces the error in a toast instead of waiting for the full reconnect timeout.
|
|
|
|
If the new container does not come up within 5 minutes, the overlay surfaces a **Taking longer than expected** message with a **Reload to check** button so the page is never left waiting indefinitely; this is not necessarily a failure, since a large image pull can legitimately run past that window. If the gateway can detect that the update did not even start (for example, the image pull failed before the helper container could spawn) it surfaces a **Failed** badge with the underlying error on the Local card. The badge appears as soon as the helper writes its error file, or by the 3-minute mark at the latest, instead of waiting for the full 5-minute timeout.
|
|
|
|
<Note>
|
|
The self-update helper container inherits all bind mounts from the main Sencho container 1:1. If your `docker-compose.yml` references `env_file`, `configs`, or `secrets` outside the compose working directory, those host paths must be mounted into the Sencho container at the *same container path* as on the host. See [Troubleshooting](/operations/troubleshooting#local-self-update-fails-with-env-file-not-found) if you encounter `env file not found` errors during a local update.
|
|
</Note>
|
|
|
|
<Note>
|
|
Self-update helpers always mount the host `/var/run/docker.sock` into the helper container, independent of `DOCKER_HOST`. On a TCP-only socket proxy, update and reapply helpers do not reach the proxy, so use a deployment where the helpers still receive a working host socket path. The generated Pilot Agent compose also mounts the raw socket path, so a TCP-only proxy on an agent node requires editing the generated compose. See [Self-Hosting: Docker socket proxy](/operations/self-hosting#self-update-behind-a-proxy).
|
|
</Note>
|
|
|
|
|
|
## Pinned image tags
|
|
|
|
Fleet self-update respects how each node compose file declares the Sencho image. Semver pins are rewritten to the target release before recreate. Floating tags such as `latest` and `sencho-dev:dev` are pulled without changing the compose file. Digest pins and unresolved interpolated values block automatic updates; those rows show a **Pinned** badge instead of an **Update** button. A reference to the `sencho-dev` repository is always treated as an integration image, whether pinned to `:dev`, the immutable `:dev-<sha>`, or a digest, and never compared against the stable release train.
|
|
|
|
The image pull always runs before any compose rewrite, so a failed pull never leaves the compose file half-updated.
|
|
|
|
A node running the Admiral **Hardened Build** image is a deliberate exception: it is digest-pinned like any other locked reference, but it does not show the **Pinned** badge and its **Update** button stays available, because that path routes through the Hardened Build's own entitlement-gated switch instead of a compose repin. See the note in [Updating the local (gateway) node](#updating-the-local-gateway-node) above.
|
|
|
|
## What happens during an update
|
|
|
|
For both local and remote nodes, an update goes through the same three steps:
|
|
|
|
1. **Pull** the target Sencho image from the registry (the compose-declared reference for floating tags, or the repinned semver tag for version pins).
|
|
2. **Recreate** the container with the new image via `docker compose up -d --force-recreate`.
|
|
3. **Restart** the Sencho process. The node is briefly offline during the swap.
|
|
|
|
The gateway tracks each in-flight update in memory and watches the target node for one of four completion signals, in this order of precedence:
|
|
|
|
- The remote reports a *new* `version` value over its `/api/meta` endpoint.
|
|
- The remote's process `startedAt` timestamp moves forward, indicating a fresh container start.
|
|
- The remote went offline during the watch window and has come back online, used as a fallback only for nodes that do not report a process start timestamp. A node that simply blips offline and returns on the *same* version with an unchanged start time is treated as still updating, not done, so a momentary network glitch is never mistaken for a successful update.
|
|
- More than 15 seconds have elapsed and the remote's reported version is at or above the comparison target (the gateway's own version, or the published latest, whichever is appropriate).
|
|
|
|
When any signal fires, the row flips from **Updating** to a green **Updated** badge. The **Updated** state stays visible for 60 seconds, then auto-clears so the row settles back to **Up to date**.
|
|
|
|
## Failure detection and recovery
|
|
|
|
The gateway uses two thresholds to decide when an update has failed:
|
|
|
|
| Threshold | What it means |
|
|
|-----------|---------------|
|
|
| **About 3 minutes** | If the remote is still reachable but no completion signal has fired, the gateway flips the row to **Failed**. This is the most common outcome of a registry pull error or a misconfigured Compose file: the remote answered the dispatch, started the pull, but never restarted. |
|
|
| **5 minutes** | If the remote has been unreachable and has not come back, the gateway flips the row to **Timed out**. The image was almost certainly pulled, but the new container did not stay up. |
|
|
|
|
Both **Failed** and **Timed out** badges expose two inline actions:
|
|
|
|
- A circular-arrow **Retry** button that clears the failed state and re-dispatches the update against the same node.
|
|
- An **X** **Dismiss** button that clears the failed state without retrying, returning the row to **Update available** so the operator can investigate before trying again.
|
|
|
|
Hovering either badge reveals the underlying error message reported by the remote (or by the helper container, for the gateway itself).
|
|
|
|
The header of the sheet exposes a **Recheck** button that does three things in one click: it flushes the cached "latest published version" lookup (resolved against the GitHub Releases API, with a Docker Hub fallback), it clears every terminal `Failed` and `Timed out` badge across the fleet, and it re-fetches the version metadata from every node. The published-version lookup is rate-limited to one refresh every couple of minutes, so clicking **Recheck** again right away reuses the recent result and shows a brief "checked recently" note rather than hitting the registry on every click.
|
|
|
|
<Note>
|
|
Update tracking is held entirely in the gateway's memory. Restarting the gateway clears all in-flight, failed, and timed-out states automatically, so a stuck row always resolves itself on the next gateway restart even without a manual **Dismiss**.
|
|
</Note>
|
|
|
|
## Troubleshooting
|
|
|
|
<AccordionGroup>
|
|
<Accordion title="Local update fails with 'env file not found'">
|
|
The self-update helper container is launched with the same bind-mount set as the main Sencho container, but only paths that are explicitly mounted are visible inside the helper. If your `docker-compose.yml` references an `env_file` (or `configs`, or `secrets`) at a host path that is *not* mounted into the Sencho container, the helper cannot read it and the recreate step fails. Mount the referenced host directory into the Sencho container at the same container path (the 1:1 rule), redeploy Sencho once, and retry the update. The full walk-through with example mounts is in [Troubleshooting](/operations/troubleshooting#local-self-update-fails-with-env-file-not-found).
|
|
</Accordion>
|
|
<Accordion title="The reconnecting overlay never clears">
|
|
The overlay polls `/api/health` every 3 seconds for up to 5 minutes. If it has not cleared by then, the new container is either still pulling a very large image, refusing to start, or crashing on boot. Open a host shell on the gateway and run `docker logs <sencho-container>` to inspect the boot output, then `docker ps` to confirm a new container ID is running. The overlay's **Reload to check** button bypasses the timer once the container is healthy again.
|
|
</Accordion>
|
|
<Accordion title="A remote shows 'Failed' after about 3 minutes">
|
|
The remote answered the dispatch but did not restart in time. The two most common causes are a registry pull error (rate limit, private-registry auth missing, network egress blocked) or a Compose file that fails validation under the new version. Open a shell on the remote and run `docker compose pull` followed by `docker compose up -d` against the Sencho working directory; the error printed there is the same one Sencho captured. Click **Retry** on the row once the cause is fixed.
|
|
</Accordion>
|
|
<Accordion title="A remote shows 'Timed out' after 5 minutes">
|
|
The remote went offline during the swap and has not come back within the 5-minute watch window. The image almost certainly pulled (otherwise the row would have surfaced **Failed** earlier), but the new container is crashing on boot or is bound to a port that another process is now holding. Inspect `docker ps -a` on the remote for a recently exited Sencho container and `docker logs` it to see the crash. Once the remote answers `/api/health` again, click **Retry** to re-dispatch.
|
|
</Accordion>
|
|
<Accordion title="A node reports 'unknown' for its current version">
|
|
The remote is reachable but its `/api/meta` response either does not include a `version` field or it cannot be parsed. The node is treated as outdated for the per-row **Update** button, so you can still trigger an update on it, but it is excluded from **Update all** because the gateway has no safe way to compare versions. After the update completes the version field comes back populated and the row joins **Update all** on the next dispatch.
|
|
</Accordion>
|
|
</AccordionGroup>
|