mirror of
https://github.com/Studio-Saelix/sencho.git
synced 2026-08-09 18:32:52 +00:00
fix(fleet): resolve stuck update states and improve detection (#405)
* fix(fleet): resolve stuck update states and improve update UX The fleet node update flow had several bugs: the in-memory update tracker never cleared terminal states (timeout, failed, completed), leaving nodes permanently stuck with no way to retry or dismiss. The Recheck button only re-fetched stale state without clearing it, and the POST trigger rejected retries with 409 even after timeout. Backend fixes: - Add DELETE endpoints (single node + batch) to clear tracker entries - Fix 409 race: detect expired timeouts and clear terminal states before re-triggering - Populate error messages in the tracker for timeouts and failures - Include error field in the update-status API response - Auto-expire completed entries after 60 seconds Frontend fixes: - Add retry (RotateCcw) and dismiss (X) buttons on failed/timed-out badges - Show error details via animated cursor hover (CursorFollow pattern) - Recheck button now batch-clears all terminal states before fetching - Recheck shows loading spinner and disables while checking - Extract NodeCardProps interface for readability * fix(fleet): detect update completion via process start time Remote nodes that cannot report their version (e.g. older builds) caused updates to always time out because completion detection relied solely on version comparison. The gateway now tracks the remote node's process start time from /api/meta and detects container restarts by comparing it across polls. Also extracts a createTracker() factory to eliminate repeated object construction across 5 call sites. * docs: add troubleshooting for first-update timeout on old nodes Adds a new troubleshooting entry explaining why the first remote update on nodes running pre-v0.40.0 always times out (neither version nor process start time can be detected). Documents the fix: dismiss, recheck, and confirm the node updated. Also adds a screenshot of the timed-out state with retry/dismiss buttons to the remote updates feature page. * fix(fleet): detect update completion via offline detection and error reporting The update completion detection relied on version change and process start time, both of which fail on nodes running older Sencho versions that report "unknown" and lack the startedAt field. This caused every update to time out after 5 minutes. Add three-signal detection: version change, process restart (startedAt), and offline/online detection (node went unreachable during update and came back). Also add a 90-second early failure heuristic for when the remote image pull fails silently, and surface pull errors from SelfUpdateService via /api/meta so the gateway can report them immediately. * fix(deps): bump vite to 8.0.5 to resolve high severity vulnerabilities Fixes GHSA-4w7w-66w2-5vf9, GHSA-v2wj-q39q-566r, GHSA-p9ff-h696-f583. * fix(deps): bump vite in backend lockfile to resolve audit failures Vitest pulls in vite as a transitive dependency. Bumps to 8.0.5.
This commit is contained in:
@@ -43,10 +43,18 @@ Each node's status column shows one of:
|
||||
| **Updating** badge | The node is pulling the new image and restarting |
|
||||
| **Updated** badge | The node came back online with the new version |
|
||||
| **Timed out** badge | The node did not come back within 5 minutes |
|
||||
| **Failed** badge | The update command failed |
|
||||
| **Failed** badge | The update was rejected or the image pull failed on the remote host |
|
||||
|
||||
Nodes that are too old to report their version show "unknown" in the current version column. These nodes are treated as outdated.
|
||||
|
||||
### Updating state
|
||||
|
||||
When an update is in progress, the dialog shows a spinning "Updating" badge and the summary card count changes in real time. The Fleet View polls every 5 seconds while an update is active.
|
||||
|
||||
<Frame>
|
||||
<img src="/images/fleet-view/fleet-node-updating.png" alt="Node Updates dialog showing a node in the updating state" />
|
||||
</Frame>
|
||||
|
||||
## Updating a single node
|
||||
|
||||
You can trigger an update in two ways:
|
||||
@@ -58,7 +66,7 @@ For remote nodes, the update happens in the background. The Fleet View automatic
|
||||
|
||||
## Updating all nodes
|
||||
|
||||
Click **Update All (N)** in the footer of the Node Updates dialog to trigger updates on all outdated remote nodes simultaneously. The local node is excluded from bulk updates to avoid losing dashboard connectivity.
|
||||
Click **Update All (N)** in the footer of the Node Updates dialog to trigger updates on all outdated remote nodes simultaneously. The local node is excluded from bulk updates to avoid losing dashboard connectivity. Nodes that previously failed or timed out are automatically retried in a bulk update.
|
||||
|
||||
## Local node updates
|
||||
|
||||
@@ -79,6 +87,27 @@ When an update is triggered on a node, Sencho:
|
||||
2. Recreates the container with the new image
|
||||
3. The node goes briefly offline during the restart
|
||||
|
||||
The gateway monitors the remote node until it comes back online with the new version, then marks it as **Updated**.
|
||||
The gateway monitors the remote node until it comes back online, then marks it as **Updated**. Completion is detected by three signals: a version change, a change in the node's process start time, or detecting that the node went briefly offline and came back (indicating a container restart). The "Updated" badge clears automatically after about 60 seconds and the node returns to "Up to date" status.
|
||||
|
||||
If the node is still reachable and unchanged after about 90 seconds, the gateway marks the update as **Failed**. This usually means the image pull failed on the remote host. Check the Docker logs on the remote node for details.
|
||||
|
||||
## Handling failures
|
||||
|
||||
If an update times out or fails, the badge shows **Timed out** or **Failed** with two action buttons:
|
||||
|
||||
- **Retry** (circular arrow icon) clears the failed state and re-triggers the update
|
||||
- **Dismiss** (X icon) clears the failed state without retrying, returning the node to "Update available"
|
||||
|
||||
Hovering over a failed or timed-out badge reveals the error message with details about what went wrong.
|
||||
|
||||
<Frame>
|
||||
<img src="/images/fleet-view/fleet-node-failed.png" alt="Node Updates dialog showing a failed node with retry and dismiss buttons" />
|
||||
</Frame>
|
||||
|
||||
You can also click **Recheck** in the dialog footer to clear all failed and timed-out states at once and fetch fresh version information from every node.
|
||||
|
||||
<Note>
|
||||
Update tracking is stored in memory on the gateway. Restarting the gateway clears all update states, so any stuck "Timed out" or "Failed" badges will resolve on their own after a restart.
|
||||
</Note>
|
||||
|
||||
If you run into issues with remote updates, see the [Troubleshooting](/operations/troubleshooting#remote-update-button-does-not-appear) page.
|
||||
|
||||
Reference in New Issue
Block a user