mirror of
https://github.com/Studio-Saelix/sencho.git
synced 2026-08-12 19:57:37 +00:00
fix(fleet): resolve stuck update states and improve detection (#405)
* fix(fleet): resolve stuck update states and improve update UX The fleet node update flow had several bugs: the in-memory update tracker never cleared terminal states (timeout, failed, completed), leaving nodes permanently stuck with no way to retry or dismiss. The Recheck button only re-fetched stale state without clearing it, and the POST trigger rejected retries with 409 even after timeout. Backend fixes: - Add DELETE endpoints (single node + batch) to clear tracker entries - Fix 409 race: detect expired timeouts and clear terminal states before re-triggering - Populate error messages in the tracker for timeouts and failures - Include error field in the update-status API response - Auto-expire completed entries after 60 seconds Frontend fixes: - Add retry (RotateCcw) and dismiss (X) buttons on failed/timed-out badges - Show error details via animated cursor hover (CursorFollow pattern) - Recheck button now batch-clears all terminal states before fetching - Recheck shows loading spinner and disables while checking - Extract NodeCardProps interface for readability * fix(fleet): detect update completion via process start time Remote nodes that cannot report their version (e.g. older builds) caused updates to always time out because completion detection relied solely on version comparison. The gateway now tracks the remote node's process start time from /api/meta and detects container restarts by comparing it across polls. Also extracts a createTracker() factory to eliminate repeated object construction across 5 call sites. * docs: add troubleshooting for first-update timeout on old nodes Adds a new troubleshooting entry explaining why the first remote update on nodes running pre-v0.40.0 always times out (neither version nor process start time can be detected). Documents the fix: dismiss, recheck, and confirm the node updated. Also adds a screenshot of the timed-out state with retry/dismiss buttons to the remote updates feature page. * fix(fleet): detect update completion via offline detection and error reporting The update completion detection relied on version change and process start time, both of which fail on nodes running older Sencho versions that report "unknown" and lack the startedAt field. This caused every update to time out after 5 minutes. Add three-signal detection: version change, process restart (startedAt), and offline/online detection (node went unreachable during update and came back). Also add a 90-second early failure heuristic for when the remote image pull fails silently, and surface pull errors from SelfUpdateService via /api/meta so the gateway can report them immediately. * fix(deps): bump vite to 8.0.5 to resolve high severity vulnerabilities Fixes GHSA-4w7w-66w2-5vf9, GHSA-v2wj-q39q-566r, GHSA-p9ff-h696-f583. * fix(deps): bump vite in backend lockfile to resolve audit failures Vitest pulls in vite as a transitive dependency. Bumps to 8.0.5.
This commit is contained in:
@@ -328,9 +328,60 @@ Nodes running very old Sencho versions that predate the capability system need t
|
||||
|
||||
**Symptom:** After clicking **Update** on a remote node, the status changes to **Timed out** instead of **Updated**.
|
||||
|
||||
**Cause:** The image pull may be slow on the remote host, or the container failed to restart.
|
||||
**Cause:** The gateway waits up to 5 minutes for the remote node to come back online after triggering the update. A timeout can happen if:
|
||||
|
||||
**Fix:** Check the Docker logs on the remote host for errors. You can also update manually by running:
|
||||
- The image pull is very slow on the remote host (large image, slow connection)
|
||||
- The container failed to restart after pulling
|
||||
- Network connectivity was lost between the gateway and remote node during the update
|
||||
|
||||
**Fix:**
|
||||
|
||||
1. Hover over the **Timed out** badge to see the error message for more detail.
|
||||
2. Click the **Retry** button (circular arrow) next to the badge to try again.
|
||||
3. If the issue persists, click **Dismiss** (X button) to clear the state, then check the Docker logs on the remote host directly.
|
||||
4. You can also click **Recheck** in the dialog footer to clear all failed states and re-scan every node.
|
||||
5. As a last resort, update manually on the remote host:
|
||||
|
||||
```bash
|
||||
docker compose pull && docker compose up -d
|
||||
```
|
||||
|
||||
<Note>
|
||||
Update state is tracked in memory on the gateway. Restarting the gateway clears all stuck states automatically.
|
||||
</Note>
|
||||
|
||||
---
|
||||
|
||||
## Remote update shows "Failed"
|
||||
|
||||
**Symptom:** After clicking **Update**, the status shows **Failed** with an error message.
|
||||
|
||||
**Cause:** There are two common scenarios:
|
||||
|
||||
1. **Immediate failure:** The gateway could not reach the remote node's update endpoint, or the remote node rejected the request. Common reasons include network issues, expired API tokens, or the remote node not supporting self-update.
|
||||
|
||||
2. **Delayed failure (after ~90 seconds):** The update was accepted by the remote node, but the node never restarted. This usually means the image pull failed on the remote host. The error message will read: *"Update may have failed. The node is still running and its version has not changed."*
|
||||
|
||||
**Fix:**
|
||||
|
||||
1. Hover over the **Failed** badge to see the specific error message.
|
||||
2. For immediate failures: verify the remote node is online and the API token is valid.
|
||||
3. For delayed failures: check the Docker logs on the remote host. The pull may have failed due to network issues, registry authentication, or disk space.
|
||||
4. Click **Retry** to attempt the update again after resolving the issue.
|
||||
|
||||
---
|
||||
|
||||
## First remote update always times out on old nodes
|
||||
|
||||
**Symptom:** After triggering a remote update on a node running a very old Sencho version (pre-v0.39.3), the node successfully restarts with the new version, but the dashboard shows **Timed out** or **Failed** instead of **Updated**.
|
||||
|
||||
**Cause:** The gateway detects update completion using three signals: a version change, a process restart (via `startedAt`), or temporary offline detection. Nodes running Sencho versions older than v0.39.3 report their version as `"unknown"` and do not include `startedAt` in their metadata. If the container restart is brief enough that the gateway does not detect the node going offline, none of the three signals fire.
|
||||
|
||||
**Fix:** This only affects the **first** update from a very old version. After the update completes (even though the dashboard may show "Failed" or "Timed out"), the remote node will be running the latest version with proper version reporting and restart detection. Future updates triggered from the dashboard will complete normally.
|
||||
|
||||
1. Click **Dismiss** on the badge to clear the state.
|
||||
2. Click **Recheck** to verify the node is now running the latest version.
|
||||
3. If the node still shows "unknown", update it manually on the remote host:
|
||||
|
||||
```bash
|
||||
docker compose pull && docker compose up -d
|
||||
|
||||
Reference in New Issue
Block a user