mirror of
https://github.com/Studio-Saelix/sencho.git
synced 2026-07-27 20:29:10 +00:00
fix(fleet): resolve stuck update states and improve detection (#405)
* fix(fleet): resolve stuck update states and improve update UX The fleet node update flow had several bugs: the in-memory update tracker never cleared terminal states (timeout, failed, completed), leaving nodes permanently stuck with no way to retry or dismiss. The Recheck button only re-fetched stale state without clearing it, and the POST trigger rejected retries with 409 even after timeout. Backend fixes: - Add DELETE endpoints (single node + batch) to clear tracker entries - Fix 409 race: detect expired timeouts and clear terminal states before re-triggering - Populate error messages in the tracker for timeouts and failures - Include error field in the update-status API response - Auto-expire completed entries after 60 seconds Frontend fixes: - Add retry (RotateCcw) and dismiss (X) buttons on failed/timed-out badges - Show error details via animated cursor hover (CursorFollow pattern) - Recheck button now batch-clears all terminal states before fetching - Recheck shows loading spinner and disables while checking - Extract NodeCardProps interface for readability * fix(fleet): detect update completion via process start time Remote nodes that cannot report their version (e.g. older builds) caused updates to always time out because completion detection relied solely on version comparison. The gateway now tracks the remote node's process start time from /api/meta and detects container restarts by comparing it across polls. Also extracts a createTracker() factory to eliminate repeated object construction across 5 call sites. * docs: add troubleshooting for first-update timeout on old nodes Adds a new troubleshooting entry explaining why the first remote update on nodes running pre-v0.40.0 always times out (neither version nor process start time can be detected). Documents the fix: dismiss, recheck, and confirm the node updated. Also adds a screenshot of the timed-out state with retry/dismiss buttons to the remote updates feature page. * fix(fleet): detect update completion via offline detection and error reporting The update completion detection relied on version change and process start time, both of which fail on nodes running older Sencho versions that report "unknown" and lack the startedAt field. This caused every update to time out after 5 minutes. Add three-signal detection: version change, process restart (startedAt), and offline/online detection (node went unreachable during update and came back). Also add a 90-second early failure heuristic for when the remote image pull fails silently, and surface pull errors from SelfUpdateService via /api/meta so the gateway can report them immediately. * fix(deps): bump vite to 8.0.5 to resolve high severity vulnerabilities Fixes GHSA-4w7w-66w2-5vf9, GHSA-v2wj-q39q-566r, GHSA-p9ff-h696-f583. * fix(deps): bump vite in backend lockfile to resolve audit failures Vitest pulls in vite as a transitive dependency. Bumps to 8.0.5.
This commit is contained in:
Anso
+18
-6
@@ -4,6 +4,24 @@ All notable changes to this project will be documented in this file.
|
||||
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
|
||||
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
### Fixed
|
||||
|
||||
* **fleet:** fix permanently stuck "Timed out" / "Failed" badges after node update attempts. The in-memory update tracker now supports clearing via a new DELETE endpoint, and terminal states are automatically clearable through the Recheck button.
|
||||
* **fleet:** fix update completion detection for remote nodes that cannot report their version. The gateway now uses three completion signals: version change, process restart detection (`startedAt`), and offline/online detection (node went unreachable during update and came back). This eliminates false timeouts on nodes running older Sencho versions.
|
||||
* **fleet:** fix 409 race condition where retrying a timed-out update was rejected because the tracker still showed "updating". The POST trigger now detects expired timeouts and allows re-triggering.
|
||||
* **fleet:** populate error messages in the update tracker so users can see why an update failed or timed out.
|
||||
* **fleet:** detect probable image pull failures within 90 seconds instead of waiting the full 5-minute timeout. If the node is still running unchanged after 90 seconds, the update is marked as "Failed" with a descriptive message.
|
||||
* **fleet:** surface remote self-update pull errors via `/api/meta` so the gateway can immediately report failures instead of timing out.
|
||||
|
||||
### Added
|
||||
|
||||
* **fleet:** retry and dismiss buttons on failed/timed-out update badges, with animated cursor hover detail showing the error message
|
||||
* **fleet:** "Recheck" button now clears all failed and timed-out states before fetching fresh status, with a loading spinner
|
||||
* **fleet:** auto-expire "completed" update status after 60 seconds so nodes return to "Up to date" automatically
|
||||
* **fleet:** "Update All" now retries nodes that previously timed out or failed
|
||||
|
||||
## [0.39.3](https://github.com/AnsoCode/Sencho/compare/v0.39.2...v0.39.3) (2026-04-06)
|
||||
|
||||
|
||||
@@ -18,12 +36,6 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
||||
|
||||
* **fleet:** filter invalid version strings from UI display ([#399](https://github.com/AnsoCode/Sencho/issues/399)) ([2089e75](https://github.com/AnsoCode/Sencho/commit/2089e75ef1e973f3e04aa6cc448211db35d95848))
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
### Fixed
|
||||
|
||||
* **fleet:** resolve root cause of "vunknown" version display across all UI surfaces. The Dockerfile was missing a `COPY package.json` in the backend build stage, causing version resolution to fail at runtime. Backend now returns `null` instead of the string `"unknown"` for unresolvable versions. Frontend guards all version display points (Fleet Overview cards, update buttons, Check for Updates modal). Fleet update logic correctly flags remote nodes with unresolvable versions as potentially outdated instead of silently marking them up to date. Consolidated version validation into shared utilities (`isValidVersion`, `formatVersion`) across both frontend and backend.
|
||||
|
||||
## [0.39.1](https://github.com/AnsoCode/Sencho/compare/v0.39.0...v0.39.1) (2026-04-06)
|
||||
|
||||
|
||||
|
||||
Generated
+85
-116
@@ -7,6 +7,7 @@
|
||||
"": {
|
||||
"name": "backend",
|
||||
"version": "1.0.0",
|
||||
"hasInstallScript": true,
|
||||
"license": "SEE LICENSE IN LICENSE",
|
||||
"dependencies": {
|
||||
"@aws-sdk/client-ecr": "^3.1019.0",
|
||||
@@ -683,40 +684,6 @@
|
||||
"node": ">=12"
|
||||
}
|
||||
},
|
||||
"node_modules/@emnapi/core": {
|
||||
"version": "1.9.1",
|
||||
"resolved": "https://registry.npmjs.org/@emnapi/core/-/core-1.9.1.tgz",
|
||||
"integrity": "sha512-mukuNALVsoix/w1BJwFzwXBN/dHeejQtuVzcDsfOEsdpCumXb/E9j8w11h5S54tT1xhifGfbbSm/ICrObRb3KA==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"dependencies": {
|
||||
"@emnapi/wasi-threads": "1.2.0",
|
||||
"tslib": "^2.4.0"
|
||||
}
|
||||
},
|
||||
"node_modules/@emnapi/runtime": {
|
||||
"version": "1.9.1",
|
||||
"resolved": "https://registry.npmjs.org/@emnapi/runtime/-/runtime-1.9.1.tgz",
|
||||
"integrity": "sha512-VYi5+ZVLhpgK4hQ0TAjiQiZ6ol0oe4mBx7mVv7IflsiEp0OWoVsp/+f9Vc1hOhE0TtkORVrI1GvzyreqpgWtkA==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"dependencies": {
|
||||
"tslib": "^2.4.0"
|
||||
}
|
||||
},
|
||||
"node_modules/@emnapi/wasi-threads": {
|
||||
"version": "1.2.0",
|
||||
"resolved": "https://registry.npmjs.org/@emnapi/wasi-threads/-/wasi-threads-1.2.0.tgz",
|
||||
"integrity": "sha512-N10dEJNSsUx41Z6pZsXU8FjPjpBEplgH24sfkmITrBED1/U2Esum9F3lfLrMjKHHjmi557zQn7kR9R+XWXu5Rg==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"dependencies": {
|
||||
"tslib": "^2.4.0"
|
||||
}
|
||||
},
|
||||
"node_modules/@eslint-community/eslint-utils": {
|
||||
"version": "4.9.1",
|
||||
"resolved": "https://registry.npmjs.org/@eslint-community/eslint-utils/-/eslint-utils-4.9.1.tgz",
|
||||
@@ -985,20 +952,22 @@
|
||||
}
|
||||
},
|
||||
"node_modules/@napi-rs/wasm-runtime": {
|
||||
"version": "1.1.1",
|
||||
"resolved": "https://registry.npmjs.org/@napi-rs/wasm-runtime/-/wasm-runtime-1.1.1.tgz",
|
||||
"integrity": "sha512-p64ah1M1ld8xjWv3qbvFwHiFVWrq1yFvV4f7w+mzaqiR4IlSgkqhcRdHwsGgomwzBH51sRY4NEowLxnaBjcW/A==",
|
||||
"version": "1.1.2",
|
||||
"resolved": "https://registry.npmjs.org/@napi-rs/wasm-runtime/-/wasm-runtime-1.1.2.tgz",
|
||||
"integrity": "sha512-sNXv5oLJ7ob93xkZ1XnxisYhGYXfaG9f65/ZgYuAu3qt7b3NadcOEhLvx28hv31PgX8SZJRYrAIPQilQmFpLVw==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"dependencies": {
|
||||
"@emnapi/core": "^1.7.1",
|
||||
"@emnapi/runtime": "^1.7.1",
|
||||
"@tybys/wasm-util": "^0.10.1"
|
||||
},
|
||||
"funding": {
|
||||
"type": "github",
|
||||
"url": "https://github.com/sponsors/Brooooooklyn"
|
||||
},
|
||||
"peerDependencies": {
|
||||
"@emnapi/core": "^1.7.1",
|
||||
"@emnapi/runtime": "^1.7.1"
|
||||
}
|
||||
},
|
||||
"node_modules/@noble/hashes": {
|
||||
@@ -1015,9 +984,9 @@
|
||||
}
|
||||
},
|
||||
"node_modules/@oxc-project/types": {
|
||||
"version": "0.120.0",
|
||||
"resolved": "https://registry.npmjs.org/@oxc-project/types/-/types-0.120.0.tgz",
|
||||
"integrity": "sha512-k1YNu55DuvAip/MGE1FTsIuU3FUCn6v/ujG9V7Nq5Df/kX2CWb13hhwD0lmJGMGqE+bE1MXvv9SZVnMzEXlWcg==",
|
||||
"version": "0.122.0",
|
||||
"resolved": "https://registry.npmjs.org/@oxc-project/types/-/types-0.122.0.tgz",
|
||||
"integrity": "sha512-oLAl5kBpV4w69UtFZ9xqcmTi+GENWOcPF7FCrczTiBbmC0ibXxCwyvZGbO39rCVEuLGAZM84DH0pUIyyv/YJzA==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"funding": {
|
||||
@@ -1099,9 +1068,9 @@
|
||||
"license": "BSD-3-Clause"
|
||||
},
|
||||
"node_modules/@rolldown/binding-android-arm64": {
|
||||
"version": "1.0.0-rc.10",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-android-arm64/-/binding-android-arm64-1.0.0-rc.10.tgz",
|
||||
"integrity": "sha512-jOHxwXhxmFKuXztiu1ORieJeTbx5vrTkcOkkkn2d35726+iwhrY1w/+nYY/AGgF12thg33qC3R1LMBF5tHTZHg==",
|
||||
"version": "1.0.0-rc.12",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-android-arm64/-/binding-android-arm64-1.0.0-rc.12.tgz",
|
||||
"integrity": "sha512-pv1y2Fv0JybcykuiiD3qBOBdz6RteYojRFY1d+b95WVuzx211CRh+ytI/+9iVyWQ6koTh5dawe4S/yRfOFjgaA==",
|
||||
"cpu": [
|
||||
"arm64"
|
||||
],
|
||||
@@ -1116,9 +1085,9 @@
|
||||
}
|
||||
},
|
||||
"node_modules/@rolldown/binding-darwin-arm64": {
|
||||
"version": "1.0.0-rc.10",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-darwin-arm64/-/binding-darwin-arm64-1.0.0-rc.10.tgz",
|
||||
"integrity": "sha512-gED05Teg/vtTZbIJBc4VNMAxAFDUPkuO/rAIyyxZjTj1a1/s6z5TII/5yMGZ0uLRCifEtwUQn8OlYzuYc0m70w==",
|
||||
"version": "1.0.0-rc.12",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-darwin-arm64/-/binding-darwin-arm64-1.0.0-rc.12.tgz",
|
||||
"integrity": "sha512-cFYr6zTG/3PXXF3pUO+umXxt1wkRK/0AYT8lDwuqvRC+LuKYWSAQAQZjCWDQpAH172ZV6ieYrNnFzVVcnSflAg==",
|
||||
"cpu": [
|
||||
"arm64"
|
||||
],
|
||||
@@ -1133,9 +1102,9 @@
|
||||
}
|
||||
},
|
||||
"node_modules/@rolldown/binding-darwin-x64": {
|
||||
"version": "1.0.0-rc.10",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-darwin-x64/-/binding-darwin-x64-1.0.0-rc.10.tgz",
|
||||
"integrity": "sha512-rI15NcM1mA48lqrIxVkHfAqcyFLcQwyXWThy+BQ5+mkKKPvSO26ir+ZDp36AgYoYVkqvMcdS8zOE6SeBsR9e8A==",
|
||||
"version": "1.0.0-rc.12",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-darwin-x64/-/binding-darwin-x64-1.0.0-rc.12.tgz",
|
||||
"integrity": "sha512-ZCsYknnHzeXYps0lGBz8JrF37GpE9bFVefrlmDrAQhOEi4IOIlcoU1+FwHEtyXGx2VkYAvhu7dyBf75EJQffBw==",
|
||||
"cpu": [
|
||||
"x64"
|
||||
],
|
||||
@@ -1150,9 +1119,9 @@
|
||||
}
|
||||
},
|
||||
"node_modules/@rolldown/binding-freebsd-x64": {
|
||||
"version": "1.0.0-rc.10",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-freebsd-x64/-/binding-freebsd-x64-1.0.0-rc.10.tgz",
|
||||
"integrity": "sha512-XZRXHdTa+4ME1MuDVp021+doQ+z6Ei4CCFmNc5/sKbqb8YmkiJdj8QKlV3rCI0AJtAeSB5n0WGPuJWNL9p/L2w==",
|
||||
"version": "1.0.0-rc.12",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-freebsd-x64/-/binding-freebsd-x64-1.0.0-rc.12.tgz",
|
||||
"integrity": "sha512-dMLeprcVsyJsKolRXyoTH3NL6qtsT0Y2xeuEA8WQJquWFXkEC4bcu1rLZZSnZRMtAqwtrF/Ib9Ddtpa/Gkge9Q==",
|
||||
"cpu": [
|
||||
"x64"
|
||||
],
|
||||
@@ -1167,9 +1136,9 @@
|
||||
}
|
||||
},
|
||||
"node_modules/@rolldown/binding-linux-arm-gnueabihf": {
|
||||
"version": "1.0.0-rc.10",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-arm-gnueabihf/-/binding-linux-arm-gnueabihf-1.0.0-rc.10.tgz",
|
||||
"integrity": "sha512-R0SQMRluISSLzFE20sPWYHVmJdDQnRyc/FzSCN72BqQmh2SOZUFG+N3/vBZpR4C6WpEUVYJLrYUXaj43sJsNLA==",
|
||||
"version": "1.0.0-rc.12",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-arm-gnueabihf/-/binding-linux-arm-gnueabihf-1.0.0-rc.12.tgz",
|
||||
"integrity": "sha512-YqWjAgGC/9M1lz3GR1r1rP79nMgo3mQiiA+Hfo+pvKFK1fAJ1bCi0ZQVh8noOqNacuY1qIcfyVfP6HoyBRZ85Q==",
|
||||
"cpu": [
|
||||
"arm"
|
||||
],
|
||||
@@ -1184,9 +1153,9 @@
|
||||
}
|
||||
},
|
||||
"node_modules/@rolldown/binding-linux-arm64-gnu": {
|
||||
"version": "1.0.0-rc.10",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-arm64-gnu/-/binding-linux-arm64-gnu-1.0.0-rc.10.tgz",
|
||||
"integrity": "sha512-Y1reMrV/o+cwpduYhJuOE3OMKx32RMYCidf14y+HssARRmhDuWXJ4yVguDg2R/8SyyGNo+auzz64LnPK9Hq6jg==",
|
||||
"version": "1.0.0-rc.12",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-arm64-gnu/-/binding-linux-arm64-gnu-1.0.0-rc.12.tgz",
|
||||
"integrity": "sha512-/I5AS4cIroLpslsmzXfwbe5OmWvSsrFuEw3mwvbQ1kDxJ822hFHIx+vsN/TAzNVyepI/j/GSzrtCIwQPeKCLIg==",
|
||||
"cpu": [
|
||||
"arm64"
|
||||
],
|
||||
@@ -1201,9 +1170,9 @@
|
||||
}
|
||||
},
|
||||
"node_modules/@rolldown/binding-linux-arm64-musl": {
|
||||
"version": "1.0.0-rc.10",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-arm64-musl/-/binding-linux-arm64-musl-1.0.0-rc.10.tgz",
|
||||
"integrity": "sha512-vELN+HNb2IzuzSBUOD4NHmP9yrGwl1DVM29wlQvx1OLSclL0NgVWnVDKl/8tEks79EFek/kebQKnNJkIAA4W2g==",
|
||||
"version": "1.0.0-rc.12",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-arm64-musl/-/binding-linux-arm64-musl-1.0.0-rc.12.tgz",
|
||||
"integrity": "sha512-V6/wZztnBqlx5hJQqNWwFdxIKN0m38p8Jas+VoSfgH54HSj9tKTt1dZvG6JRHcjh6D7TvrJPWFGaY9UBVOaWPw==",
|
||||
"cpu": [
|
||||
"arm64"
|
||||
],
|
||||
@@ -1218,9 +1187,9 @@
|
||||
}
|
||||
},
|
||||
"node_modules/@rolldown/binding-linux-ppc64-gnu": {
|
||||
"version": "1.0.0-rc.10",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-ppc64-gnu/-/binding-linux-ppc64-gnu-1.0.0-rc.10.tgz",
|
||||
"integrity": "sha512-ZqrufYTgzxbHwpqOjzSsb0UV/aV2TFIY5rP8HdsiPTv/CuAgCRjM6s9cYFwQ4CNH+hf9Y4erHW1GjZuZ7WoI7w==",
|
||||
"version": "1.0.0-rc.12",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-ppc64-gnu/-/binding-linux-ppc64-gnu-1.0.0-rc.12.tgz",
|
||||
"integrity": "sha512-AP3E9BpcUYliZCxa3w5Kwj9OtEVDYK6sVoUzy4vTOJsjPOgdaJZKFmN4oOlX0Wp0RPV2ETfmIra9x1xuayFB7g==",
|
||||
"cpu": [
|
||||
"ppc64"
|
||||
],
|
||||
@@ -1235,9 +1204,9 @@
|
||||
}
|
||||
},
|
||||
"node_modules/@rolldown/binding-linux-s390x-gnu": {
|
||||
"version": "1.0.0-rc.10",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-s390x-gnu/-/binding-linux-s390x-gnu-1.0.0-rc.10.tgz",
|
||||
"integrity": "sha512-gSlmVS1FZJSRicA6IyjoRoKAFK7IIHBs7xJuHRSmjImqk3mPPWbR7RhbnfH2G6bcmMEllCt2vQ/7u9e6bBnByg==",
|
||||
"version": "1.0.0-rc.12",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-s390x-gnu/-/binding-linux-s390x-gnu-1.0.0-rc.12.tgz",
|
||||
"integrity": "sha512-nWwpvUSPkoFmZo0kQazZYOrT7J5DGOJ/+QHHzjvNlooDZED8oH82Yg67HvehPPLAg5fUff7TfWFHQS8IV1n3og==",
|
||||
"cpu": [
|
||||
"s390x"
|
||||
],
|
||||
@@ -1252,9 +1221,9 @@
|
||||
}
|
||||
},
|
||||
"node_modules/@rolldown/binding-linux-x64-gnu": {
|
||||
"version": "1.0.0-rc.10",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-x64-gnu/-/binding-linux-x64-gnu-1.0.0-rc.10.tgz",
|
||||
"integrity": "sha512-eOCKUpluKgfObT2pHjztnaWEIbUabWzk3qPZ5PuacuPmr4+JtQG4k2vGTY0H15edaTnicgU428XW/IH6AimcQw==",
|
||||
"version": "1.0.0-rc.12",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-x64-gnu/-/binding-linux-x64-gnu-1.0.0-rc.12.tgz",
|
||||
"integrity": "sha512-RNrafz5bcwRy+O9e6P8Z/OCAJW/A+qtBczIqVYwTs14pf4iV1/+eKEjdOUta93q2TsT/FI0XYDP3TCky38LMAg==",
|
||||
"cpu": [
|
||||
"x64"
|
||||
],
|
||||
@@ -1269,9 +1238,9 @@
|
||||
}
|
||||
},
|
||||
"node_modules/@rolldown/binding-linux-x64-musl": {
|
||||
"version": "1.0.0-rc.10",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-x64-musl/-/binding-linux-x64-musl-1.0.0-rc.10.tgz",
|
||||
"integrity": "sha512-Xdf2jQbfQowJnLcgYfD/m0Uu0Qj5OdxKallD78/IPPfzaiaI4KRAwZzHcKQ4ig1gtg1SuzC7jovNiM2TzQsBXA==",
|
||||
"version": "1.0.0-rc.12",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-linux-x64-musl/-/binding-linux-x64-musl-1.0.0-rc.12.tgz",
|
||||
"integrity": "sha512-Jpw/0iwoKWx3LJ2rc1yjFrj+T7iHZn2JDg1Yny1ma0luviFS4mhAIcd1LFNxK3EYu3DHWCps0ydXQ5i/rrJ2ig==",
|
||||
"cpu": [
|
||||
"x64"
|
||||
],
|
||||
@@ -1286,9 +1255,9 @@
|
||||
}
|
||||
},
|
||||
"node_modules/@rolldown/binding-openharmony-arm64": {
|
||||
"version": "1.0.0-rc.10",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-openharmony-arm64/-/binding-openharmony-arm64-1.0.0-rc.10.tgz",
|
||||
"integrity": "sha512-o1hYe8hLi1EY6jgPFyxQgQ1wcycX+qz8eEbVmot2hFkgUzPxy9+kF0u0NIQBeDq+Mko47AkaFFaChcvZa9UX9Q==",
|
||||
"version": "1.0.0-rc.12",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-openharmony-arm64/-/binding-openharmony-arm64-1.0.0-rc.12.tgz",
|
||||
"integrity": "sha512-vRugONE4yMfVn0+7lUKdKvN4D5YusEiPilaoO2sgUWpCvrncvWgPMzK00ZFFJuiPgLwgFNP5eSiUlv2tfc+lpA==",
|
||||
"cpu": [
|
||||
"arm64"
|
||||
],
|
||||
@@ -1303,9 +1272,9 @@
|
||||
}
|
||||
},
|
||||
"node_modules/@rolldown/binding-wasm32-wasi": {
|
||||
"version": "1.0.0-rc.10",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-wasm32-wasi/-/binding-wasm32-wasi-1.0.0-rc.10.tgz",
|
||||
"integrity": "sha512-Ugv9o7qYJudqQO5Y5y2N2SOo6S4WiqiNOpuQyoPInnhVzCY+wi/GHltcLHypG9DEUYMB0iTB/huJrpadiAcNcA==",
|
||||
"version": "1.0.0-rc.12",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-wasm32-wasi/-/binding-wasm32-wasi-1.0.0-rc.12.tgz",
|
||||
"integrity": "sha512-ykGiLr/6kkiHc0XnBfmFJuCjr5ZYKKofkx+chJWDjitX+KsJuAmrzWhwyOMSHzPhzOHOy7u9HlFoa5MoAOJ/Zg==",
|
||||
"cpu": [
|
||||
"wasm32"
|
||||
],
|
||||
@@ -1320,9 +1289,9 @@
|
||||
}
|
||||
},
|
||||
"node_modules/@rolldown/binding-win32-arm64-msvc": {
|
||||
"version": "1.0.0-rc.10",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-win32-arm64-msvc/-/binding-win32-arm64-msvc-1.0.0-rc.10.tgz",
|
||||
"integrity": "sha512-7UODQb4fQUNT/vmgDZBl3XOBAIOutP5R3O/rkxg0aLfEGQ4opbCgU5vOw/scPe4xOqBwL9fw7/RP1vAMZ6QlAQ==",
|
||||
"version": "1.0.0-rc.12",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-win32-arm64-msvc/-/binding-win32-arm64-msvc-1.0.0-rc.12.tgz",
|
||||
"integrity": "sha512-5eOND4duWkwx1AzCxadcOrNeighiLwMInEADT0YM7xeEOOFcovWZCq8dadXgcRHSf3Ulh1kFo/qvzoFiCLOL1Q==",
|
||||
"cpu": [
|
||||
"arm64"
|
||||
],
|
||||
@@ -1337,9 +1306,9 @@
|
||||
}
|
||||
},
|
||||
"node_modules/@rolldown/binding-win32-x64-msvc": {
|
||||
"version": "1.0.0-rc.10",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-win32-x64-msvc/-/binding-win32-x64-msvc-1.0.0-rc.10.tgz",
|
||||
"integrity": "sha512-PYxKHMVHOb5NJuDL53vBUl1VwUjymDcYI6rzpIni0C9+9mTiJedvUxSk7/RPp7OOAm3v+EjgMu9bIy3N6b408w==",
|
||||
"version": "1.0.0-rc.12",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/binding-win32-x64-msvc/-/binding-win32-x64-msvc-1.0.0-rc.12.tgz",
|
||||
"integrity": "sha512-PyqoipaswDLAZtot351MLhrlrh6lcZPo2LSYE+VDxbVk24LVKAGOuE4hb8xZQmrPAuEtTZW8E6D2zc5EUZX4Lw==",
|
||||
"cpu": [
|
||||
"x64"
|
||||
],
|
||||
@@ -1354,9 +1323,9 @@
|
||||
}
|
||||
},
|
||||
"node_modules/@rolldown/pluginutils": {
|
||||
"version": "1.0.0-rc.10",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/pluginutils/-/pluginutils-1.0.0-rc.10.tgz",
|
||||
"integrity": "sha512-UkVDEFk1w3mveXeKgaTuYfKWtPbvgck1dT8TUG3bnccrH0XtLTuAyfCoks4Q/M5ZGToSVJTIQYCzy2g/atAOeg==",
|
||||
"version": "1.0.0-rc.12",
|
||||
"resolved": "https://registry.npmjs.org/@rolldown/pluginutils/-/pluginutils-1.0.0-rc.12.tgz",
|
||||
"integrity": "sha512-HHMwmarRKvoFsJorqYlFeFRzXZqCt2ETQlEDOb9aqssrnVBB1/+xgTGtuTrIk5vzLNX1MjMtTf7W9z3tsSbrxw==",
|
||||
"dev": true,
|
||||
"license": "MIT"
|
||||
},
|
||||
@@ -5782,14 +5751,14 @@
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/rolldown": {
|
||||
"version": "1.0.0-rc.10",
|
||||
"resolved": "https://registry.npmjs.org/rolldown/-/rolldown-1.0.0-rc.10.tgz",
|
||||
"integrity": "sha512-q7j6vvarRFmKpgJUT8HCAUljkgzEp4LAhPlJUvQhA5LA1SUL36s5QCysMutErzL3EbNOZOkoziSx9iZC4FddKA==",
|
||||
"version": "1.0.0-rc.12",
|
||||
"resolved": "https://registry.npmjs.org/rolldown/-/rolldown-1.0.0-rc.12.tgz",
|
||||
"integrity": "sha512-yP4USLIMYrwpPHEFB5JGH1uxhcslv6/hL0OyvTuY+3qlOSJvZ7ntYnoWpehBxufkgN0cvXxppuTu5hHa/zPh+A==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"@oxc-project/types": "=0.120.0",
|
||||
"@rolldown/pluginutils": "1.0.0-rc.10"
|
||||
"@oxc-project/types": "=0.122.0",
|
||||
"@rolldown/pluginutils": "1.0.0-rc.12"
|
||||
},
|
||||
"bin": {
|
||||
"rolldown": "bin/cli.mjs"
|
||||
@@ -5798,21 +5767,21 @@
|
||||
"node": "^20.19.0 || >=22.12.0"
|
||||
},
|
||||
"optionalDependencies": {
|
||||
"@rolldown/binding-android-arm64": "1.0.0-rc.10",
|
||||
"@rolldown/binding-darwin-arm64": "1.0.0-rc.10",
|
||||
"@rolldown/binding-darwin-x64": "1.0.0-rc.10",
|
||||
"@rolldown/binding-freebsd-x64": "1.0.0-rc.10",
|
||||
"@rolldown/binding-linux-arm-gnueabihf": "1.0.0-rc.10",
|
||||
"@rolldown/binding-linux-arm64-gnu": "1.0.0-rc.10",
|
||||
"@rolldown/binding-linux-arm64-musl": "1.0.0-rc.10",
|
||||
"@rolldown/binding-linux-ppc64-gnu": "1.0.0-rc.10",
|
||||
"@rolldown/binding-linux-s390x-gnu": "1.0.0-rc.10",
|
||||
"@rolldown/binding-linux-x64-gnu": "1.0.0-rc.10",
|
||||
"@rolldown/binding-linux-x64-musl": "1.0.0-rc.10",
|
||||
"@rolldown/binding-openharmony-arm64": "1.0.0-rc.10",
|
||||
"@rolldown/binding-wasm32-wasi": "1.0.0-rc.10",
|
||||
"@rolldown/binding-win32-arm64-msvc": "1.0.0-rc.10",
|
||||
"@rolldown/binding-win32-x64-msvc": "1.0.0-rc.10"
|
||||
"@rolldown/binding-android-arm64": "1.0.0-rc.12",
|
||||
"@rolldown/binding-darwin-arm64": "1.0.0-rc.12",
|
||||
"@rolldown/binding-darwin-x64": "1.0.0-rc.12",
|
||||
"@rolldown/binding-freebsd-x64": "1.0.0-rc.12",
|
||||
"@rolldown/binding-linux-arm-gnueabihf": "1.0.0-rc.12",
|
||||
"@rolldown/binding-linux-arm64-gnu": "1.0.0-rc.12",
|
||||
"@rolldown/binding-linux-arm64-musl": "1.0.0-rc.12",
|
||||
"@rolldown/binding-linux-ppc64-gnu": "1.0.0-rc.12",
|
||||
"@rolldown/binding-linux-s390x-gnu": "1.0.0-rc.12",
|
||||
"@rolldown/binding-linux-x64-gnu": "1.0.0-rc.12",
|
||||
"@rolldown/binding-linux-x64-musl": "1.0.0-rc.12",
|
||||
"@rolldown/binding-openharmony-arm64": "1.0.0-rc.12",
|
||||
"@rolldown/binding-wasm32-wasi": "1.0.0-rc.12",
|
||||
"@rolldown/binding-win32-arm64-msvc": "1.0.0-rc.12",
|
||||
"@rolldown/binding-win32-x64-msvc": "1.0.0-rc.12"
|
||||
}
|
||||
},
|
||||
"node_modules/router": {
|
||||
@@ -6621,16 +6590,16 @@
|
||||
}
|
||||
},
|
||||
"node_modules/vite": {
|
||||
"version": "8.0.1",
|
||||
"resolved": "https://registry.npmjs.org/vite/-/vite-8.0.1.tgz",
|
||||
"integrity": "sha512-wt+Z2qIhfFt85uiyRt5LPU4oVEJBXj8hZNWKeqFG4gRG/0RaRGJ7njQCwzFVjO+v4+Ipmf5CY7VdmZRAYYBPHw==",
|
||||
"version": "8.0.5",
|
||||
"resolved": "https://registry.npmjs.org/vite/-/vite-8.0.5.tgz",
|
||||
"integrity": "sha512-nmu43Qvq9UopTRfMx2jOYW5l16pb3iDC1JH6yMuPkpVbzK0k+L7dfsEDH4jRgYFmsg0sTAqkojoZgzLMlwHsCQ==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"lightningcss": "^1.32.0",
|
||||
"picomatch": "^4.0.3",
|
||||
"picomatch": "^4.0.4",
|
||||
"postcss": "^8.5.8",
|
||||
"rolldown": "1.0.0-rc.10",
|
||||
"rolldown": "1.0.0-rc.12",
|
||||
"tinyglobby": "^0.2.15"
|
||||
},
|
||||
"bin": {
|
||||
@@ -6648,7 +6617,7 @@
|
||||
"peerDependencies": {
|
||||
"@types/node": "^20.19.0 || >=22.12.0",
|
||||
"@vitejs/devtools": "^0.1.0",
|
||||
"esbuild": "^0.27.0",
|
||||
"esbuild": "^0.27.0 || ^0.28.0",
|
||||
"jiti": ">=1.21.0",
|
||||
"less": "^4.0.0",
|
||||
"sass": "^1.70.0",
|
||||
|
||||
+135
-13
@@ -346,8 +346,16 @@ app.get('/api/health', (_req: Request, res: Response): void => {
|
||||
|
||||
// Public meta endpoint - returns this instance's version and supported capabilities.
|
||||
// No auth required (like /health). Used by remote nodes during connection tests.
|
||||
const processStartedAt = Date.now();
|
||||
|
||||
app.get('/api/meta', (_req: Request, res: Response): void => {
|
||||
res.json({ version: getSenchoVersion(), capabilities: getActiveCapabilities() });
|
||||
const updateError = SelfUpdateService.getInstance().getLastError();
|
||||
res.json({
|
||||
version: getSenchoVersion(),
|
||||
capabilities: getActiveCapabilities(),
|
||||
startedAt: processStartedAt,
|
||||
...(updateError ? { updateError } : {}),
|
||||
});
|
||||
});
|
||||
|
||||
// Auth Routes (no authentication required)
|
||||
@@ -1103,9 +1111,24 @@ interface UpdateTracker {
|
||||
startedAt: number;
|
||||
previousVersion: string | null;
|
||||
error?: string;
|
||||
/** Process start time of the remote node before the update was triggered. */
|
||||
previousProcessStart: number | null;
|
||||
/** True when the node became unreachable at least once during the update window. */
|
||||
wasOffline: boolean;
|
||||
}
|
||||
const updateTracker = new Map<number, UpdateTracker>();
|
||||
const UPDATE_TIMEOUT_MS = 5 * 60 * 1000; // 5 minutes
|
||||
const UPDATE_TIMEOUT_MSG = 'Node did not come back online within 5 minutes.';
|
||||
const EARLY_FAIL_MS = 90 * 1000; // 90 seconds before declaring a probable pull failure
|
||||
|
||||
function createTracker(
|
||||
status: UpdateTracker['status'],
|
||||
previousVersion: string | null,
|
||||
previousProcessStart: number | null,
|
||||
error?: string,
|
||||
): UpdateTracker {
|
||||
return { status, startedAt: Date.now(), previousVersion, previousProcessStart, wasOffline: false, error };
|
||||
}
|
||||
|
||||
interface FleetNodeOverview {
|
||||
id: number;
|
||||
@@ -1258,22 +1281,71 @@ app.get('/api/fleet/update-status', async (_req: Request, res: Response): Promis
|
||||
const tracker = updateTracker.get(node.id);
|
||||
|
||||
let version: string | null = null;
|
||||
let remoteStartedAt: number | null = null;
|
||||
let remoteUpdateError: string | null = null;
|
||||
let remoteOnline = false;
|
||||
if (node.type === 'local') {
|
||||
version = gatewayVersion;
|
||||
} else if (node.api_url && node.api_token) {
|
||||
const meta = await fetchRemoteMeta(node.api_url, node.api_token);
|
||||
version = meta.version;
|
||||
remoteStartedAt = meta.startedAt;
|
||||
remoteUpdateError = meta.updateError;
|
||||
remoteOnline = meta.online;
|
||||
}
|
||||
|
||||
// For nodes actively updating, check if they've come back with a new version
|
||||
// For nodes actively updating, check if they've come back
|
||||
if (tracker?.status === 'updating') {
|
||||
if (Date.now() - tracker.startedAt > UPDATE_TIMEOUT_MS) {
|
||||
updateTracker.set(node.id, { ...tracker, status: 'timeout' });
|
||||
} else if (node.type === 'remote' && version && version !== tracker.previousVersion) {
|
||||
updateTracker.set(node.id, { ...tracker, status: 'completed' });
|
||||
const elapsed = Date.now() - tracker.startedAt;
|
||||
|
||||
if (elapsed > UPDATE_TIMEOUT_MS) {
|
||||
// Final timeout (5 min)
|
||||
updateTracker.set(node.id, { ...tracker, status: 'timeout', error: UPDATE_TIMEOUT_MSG });
|
||||
} else if (node.type === 'remote') {
|
||||
if (remoteUpdateError) {
|
||||
// Remote reported a pull failure via /api/meta
|
||||
updateTracker.set(node.id, { ...tracker, status: 'failed', error: remoteUpdateError });
|
||||
} else if (!remoteOnline) {
|
||||
// Node is unreachable (restarting); record that it went offline
|
||||
if (!tracker.wasOffline) {
|
||||
updateTracker.set(node.id, { ...tracker, wasOffline: true });
|
||||
}
|
||||
} else if (version !== tracker.previousVersion) {
|
||||
// Signal 1: Version changed (or version now resolvable after being unknown)
|
||||
updateTracker.set(node.id, { ...tracker, status: 'completed' });
|
||||
} else if (
|
||||
remoteStartedAt !== null &&
|
||||
tracker.previousProcessStart !== null &&
|
||||
remoteStartedAt !== tracker.previousProcessStart
|
||||
) {
|
||||
// Signal 2: Process restarted (startedAt changed)
|
||||
updateTracker.set(node.id, { ...tracker, status: 'completed' });
|
||||
} else if (tracker.wasOffline && remoteOnline) {
|
||||
// Signal 3: Node went offline and is back online (container was recreated)
|
||||
updateTracker.set(node.id, { ...tracker, status: 'completed' });
|
||||
} else if (elapsed > EARLY_FAIL_MS) {
|
||||
// Heuristic: node never went offline and nothing changed after 90s
|
||||
updateTracker.set(node.id, {
|
||||
...tracker,
|
||||
status: 'failed',
|
||||
error: 'Update may have failed. The node is still running and its version has not changed.',
|
||||
});
|
||||
}
|
||||
} else if (node.type === 'local') {
|
||||
// Local node: check if SelfUpdateService reported a pull failure
|
||||
const localError = SelfUpdateService.getInstance().getLastError();
|
||||
if (localError) {
|
||||
updateTracker.set(node.id, { ...tracker, status: 'failed', error: localError });
|
||||
SelfUpdateService.getInstance().clearLastError();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Auto-expire completed entries after 60 seconds so nodes return to "Up to date"
|
||||
if (tracker?.status === 'completed' && Date.now() - tracker.startedAt > 60_000) {
|
||||
updateTracker.delete(node.id);
|
||||
}
|
||||
|
||||
// Assume remote nodes are outdated when their version is unresolvable
|
||||
let updateAvailable = false;
|
||||
if (!isValidVersion(version)) {
|
||||
@@ -1291,6 +1363,7 @@ app.get('/api/fleet/update-status', async (_req: Request, res: Response): Promis
|
||||
latestVersion: gatewayVersion,
|
||||
updateAvailable,
|
||||
updateStatus: currentTracker?.status ?? null,
|
||||
error: currentTracker?.error ?? null,
|
||||
};
|
||||
})
|
||||
);
|
||||
@@ -1329,8 +1402,16 @@ app.post('/api/fleet/nodes/:nodeId/update', async (req: Request, res: Response):
|
||||
|
||||
const existing = updateTracker.get(nodeId);
|
||||
if (existing?.status === 'updating') {
|
||||
res.status(409).json({ error: 'Update already in progress for this node.' });
|
||||
return;
|
||||
if (Date.now() - existing.startedAt > UPDATE_TIMEOUT_MS) {
|
||||
updateTracker.set(nodeId, { ...existing, status: 'timeout', error: UPDATE_TIMEOUT_MSG });
|
||||
} else {
|
||||
res.status(409).json({ error: 'Update already in progress for this node.' });
|
||||
return;
|
||||
}
|
||||
}
|
||||
// Clear terminal states to allow retry
|
||||
if (existing && (existing.status === 'timeout' || existing.status === 'failed' || existing.status === 'completed')) {
|
||||
updateTracker.delete(nodeId);
|
||||
}
|
||||
|
||||
if (node.type === 'local') {
|
||||
@@ -1338,7 +1419,7 @@ app.post('/api/fleet/nodes/:nodeId/update', async (req: Request, res: Response):
|
||||
res.status(503).json({ error: 'Self-update unavailable on the local node.' });
|
||||
return;
|
||||
}
|
||||
updateTracker.set(nodeId, { status: 'updating', startedAt: Date.now(), previousVersion: getSenchoVersion() });
|
||||
updateTracker.set(nodeId, createTracker('updating', getSenchoVersion(), null));
|
||||
scheduleLocalUpdate(res, 'Update initiated on local node. The server will restart shortly.');
|
||||
return;
|
||||
}
|
||||
@@ -1368,14 +1449,21 @@ app.post('/api/fleet/nodes/:nodeId/update', async (req: Request, res: Response):
|
||||
|
||||
if (!response.ok) {
|
||||
const err = await response.json().catch(() => ({}));
|
||||
res.status(502).json({ error: (err as Record<string, string>)?.error || 'Remote node rejected update request.' });
|
||||
const errorMsg = (err as Record<string, string>)?.error || 'Remote node rejected update request.';
|
||||
updateTracker.set(nodeId, createTracker('failed', meta.version, meta.startedAt, errorMsg));
|
||||
res.status(502).json({ error: errorMsg });
|
||||
return;
|
||||
}
|
||||
|
||||
updateTracker.set(nodeId, { status: 'updating', startedAt: Date.now(), previousVersion: meta.version });
|
||||
updateTracker.set(nodeId, createTracker('updating', meta.version, meta.startedAt));
|
||||
res.status(202).json({ message: `Update initiated on ${node.name}.` });
|
||||
} catch (error) {
|
||||
console.error('[Fleet] Node update error:', error);
|
||||
const errorMsg = (error as Error)?.message || 'Failed to trigger node update.';
|
||||
const failedNodeId = parseInt(req.params.nodeId as string, 10);
|
||||
if (!isNaN(failedNodeId)) {
|
||||
updateTracker.set(failedNodeId, createTracker('failed', null, null, errorMsg));
|
||||
}
|
||||
res.status(500).json({ error: 'Failed to trigger node update.' });
|
||||
}
|
||||
});
|
||||
@@ -1391,8 +1479,13 @@ app.post('/api/fleet/update-all', async (req: Request, res: Response): Promise<v
|
||||
// Filter to eligible candidates, then trigger all in parallel
|
||||
const candidates = nodes.filter(node => {
|
||||
if (node.type === 'local') return false;
|
||||
if (updateTracker.get(node.id)?.status === 'updating') return false;
|
||||
const tracker = updateTracker.get(node.id);
|
||||
if (tracker?.status === 'updating') return false;
|
||||
if (!node.api_url || !node.api_token) return false;
|
||||
// Clear terminal states so they can be re-triggered
|
||||
if (tracker && (tracker.status === 'timeout' || tracker.status === 'failed' || tracker.status === 'completed')) {
|
||||
updateTracker.delete(node.id);
|
||||
}
|
||||
return true;
|
||||
});
|
||||
|
||||
@@ -1410,7 +1503,7 @@ app.post('/api/fleet/update-all', async (req: Request, res: Response): Promise<v
|
||||
signal: AbortSignal.timeout(10000),
|
||||
});
|
||||
if (response.ok) {
|
||||
updateTracker.set(node.id, { status: 'updating', startedAt: Date.now(), previousVersion: meta.version });
|
||||
updateTracker.set(node.id, createTracker('updating', meta.version, meta.startedAt));
|
||||
return { name: node.name, triggered: true };
|
||||
}
|
||||
return { name: node.name, triggered: false };
|
||||
@@ -1430,6 +1523,35 @@ app.post('/api/fleet/update-all', async (req: Request, res: Response): Promise<v
|
||||
}
|
||||
});
|
||||
|
||||
// Clear update tracker entry for a specific node (dismiss or before retry)
|
||||
app.delete('/api/fleet/nodes/:nodeId/update-status', async (req: Request, res: Response): Promise<void> => {
|
||||
if (!requirePaid(req, res)) return;
|
||||
try {
|
||||
const nodeId = parseInt(req.params.nodeId as string, 10);
|
||||
const node = DatabaseService.getInstance().getNode(nodeId);
|
||||
if (!node) {
|
||||
res.status(404).json({ error: 'Node not found' });
|
||||
return;
|
||||
}
|
||||
updateTracker.delete(nodeId);
|
||||
res.status(204).send();
|
||||
} catch (error) {
|
||||
console.error('[Fleet] Clear update status error:', error);
|
||||
res.status(500).json({ error: 'Failed to clear update status.' });
|
||||
}
|
||||
});
|
||||
|
||||
// Clear all terminal (timed-out, failed, completed) tracker entries at once
|
||||
app.delete('/api/fleet/update-status', async (req: Request, res: Response): Promise<void> => {
|
||||
if (!requirePaid(req, res)) return;
|
||||
for (const [nodeId, tracker] of updateTracker) {
|
||||
if (tracker.status === 'timeout' || tracker.status === 'failed' || tracker.status === 'completed') {
|
||||
updateTracker.delete(nodeId);
|
||||
}
|
||||
}
|
||||
res.status(204).send();
|
||||
});
|
||||
|
||||
async function fetchLocalNodeOverview(node: Node): Promise<FleetNodeOverview> {
|
||||
try {
|
||||
const composeDir = path.resolve(NodeRegistry.getInstance().getComposeDir(node.id));
|
||||
|
||||
@@ -68,6 +68,11 @@ export function getSenchoVersion(): string | null {
|
||||
export interface RemoteMeta {
|
||||
version: string | null;
|
||||
capabilities: string[];
|
||||
startedAt: number | null;
|
||||
/** Error message from a failed self-update attempt on the remote node. */
|
||||
updateError: string | null;
|
||||
/** True when the /api/meta request succeeded (node is reachable). */
|
||||
online: boolean;
|
||||
}
|
||||
|
||||
// Runtime capability overrides — services call disableCapability() during init
|
||||
@@ -94,9 +99,12 @@ export async function fetchRemoteMeta(baseUrl: string, apiToken: string): Promis
|
||||
return {
|
||||
version: isValidVersion(rawVersion) ? rawVersion : null,
|
||||
capabilities: Array.isArray(res.data.capabilities) ? res.data.capabilities : [],
|
||||
startedAt: typeof res.data.startedAt === 'number' ? res.data.startedAt : null,
|
||||
updateError: typeof res.data.updateError === 'string' ? res.data.updateError : null,
|
||||
online: true,
|
||||
};
|
||||
} catch (err) {
|
||||
console.warn(`[CapabilityRegistry] Failed to fetch meta from ${baseUrl}:`, (err as Error).message);
|
||||
return { version: null, capabilities: [] };
|
||||
return { version: null, capabilities: [], startedAt: null, updateError: null, online: false };
|
||||
}
|
||||
}
|
||||
|
||||
@@ -12,6 +12,7 @@ class SelfUpdateService {
|
||||
private static instance: SelfUpdateService;
|
||||
private canSelfUpdate = false;
|
||||
private composeContext: ComposeContext | null = null;
|
||||
private lastUpdateError: string | null = null;
|
||||
|
||||
public static getInstance(): SelfUpdateService {
|
||||
if (!SelfUpdateService.instance) {
|
||||
@@ -57,10 +58,21 @@ class SelfUpdateService {
|
||||
return this.canSelfUpdate;
|
||||
}
|
||||
|
||||
/** Returns the error message from the last failed update attempt, or null. */
|
||||
getLastError(): string | null {
|
||||
return this.lastUpdateError;
|
||||
}
|
||||
|
||||
/** Clears the stored update error (call after reading it). */
|
||||
clearLastError(): void {
|
||||
this.lastUpdateError = null;
|
||||
}
|
||||
|
||||
triggerUpdate(): void {
|
||||
if (!this.composeContext) return;
|
||||
const { workingDir, configFiles, serviceName } = this.composeContext;
|
||||
const env = { ...process.env, PATH: process.env.PATH || '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin' };
|
||||
this.lastUpdateError = null;
|
||||
|
||||
console.log(`[SelfUpdate] Pulling latest image for ${serviceName}...`);
|
||||
try {
|
||||
@@ -71,7 +83,8 @@ class SelfUpdateService {
|
||||
timeout: 300_000, // 5 min max for pull
|
||||
});
|
||||
} catch (error) {
|
||||
console.error('[SelfUpdate] Pull failed:', (error as Error).message);
|
||||
this.lastUpdateError = (error as Error).message;
|
||||
console.error('[SelfUpdate] Pull failed:', this.lastUpdateError);
|
||||
return;
|
||||
}
|
||||
|
||||
|
||||
@@ -43,10 +43,18 @@ Each node's status column shows one of:
|
||||
| **Updating** badge | The node is pulling the new image and restarting |
|
||||
| **Updated** badge | The node came back online with the new version |
|
||||
| **Timed out** badge | The node did not come back within 5 minutes |
|
||||
| **Failed** badge | The update command failed |
|
||||
| **Failed** badge | The update was rejected or the image pull failed on the remote host |
|
||||
|
||||
Nodes that are too old to report their version show "unknown" in the current version column. These nodes are treated as outdated.
|
||||
|
||||
### Updating state
|
||||
|
||||
When an update is in progress, the dialog shows a spinning "Updating" badge and the summary card count changes in real time. The Fleet View polls every 5 seconds while an update is active.
|
||||
|
||||
<Frame>
|
||||
<img src="/images/fleet-view/fleet-node-updating.png" alt="Node Updates dialog showing a node in the updating state" />
|
||||
</Frame>
|
||||
|
||||
## Updating a single node
|
||||
|
||||
You can trigger an update in two ways:
|
||||
@@ -58,7 +66,7 @@ For remote nodes, the update happens in the background. The Fleet View automatic
|
||||
|
||||
## Updating all nodes
|
||||
|
||||
Click **Update All (N)** in the footer of the Node Updates dialog to trigger updates on all outdated remote nodes simultaneously. The local node is excluded from bulk updates to avoid losing dashboard connectivity.
|
||||
Click **Update All (N)** in the footer of the Node Updates dialog to trigger updates on all outdated remote nodes simultaneously. The local node is excluded from bulk updates to avoid losing dashboard connectivity. Nodes that previously failed or timed out are automatically retried in a bulk update.
|
||||
|
||||
## Local node updates
|
||||
|
||||
@@ -79,6 +87,27 @@ When an update is triggered on a node, Sencho:
|
||||
2. Recreates the container with the new image
|
||||
3. The node goes briefly offline during the restart
|
||||
|
||||
The gateway monitors the remote node until it comes back online with the new version, then marks it as **Updated**.
|
||||
The gateway monitors the remote node until it comes back online, then marks it as **Updated**. Completion is detected by three signals: a version change, a change in the node's process start time, or detecting that the node went briefly offline and came back (indicating a container restart). The "Updated" badge clears automatically after about 60 seconds and the node returns to "Up to date" status.
|
||||
|
||||
If the node is still reachable and unchanged after about 90 seconds, the gateway marks the update as **Failed**. This usually means the image pull failed on the remote host. Check the Docker logs on the remote node for details.
|
||||
|
||||
## Handling failures
|
||||
|
||||
If an update times out or fails, the badge shows **Timed out** or **Failed** with two action buttons:
|
||||
|
||||
- **Retry** (circular arrow icon) clears the failed state and re-triggers the update
|
||||
- **Dismiss** (X icon) clears the failed state without retrying, returning the node to "Update available"
|
||||
|
||||
Hovering over a failed or timed-out badge reveals the error message with details about what went wrong.
|
||||
|
||||
<Frame>
|
||||
<img src="/images/fleet-view/fleet-node-failed.png" alt="Node Updates dialog showing a failed node with retry and dismiss buttons" />
|
||||
</Frame>
|
||||
|
||||
You can also click **Recheck** in the dialog footer to clear all failed and timed-out states at once and fetch fresh version information from every node.
|
||||
|
||||
<Note>
|
||||
Update tracking is stored in memory on the gateway. Restarting the gateway clears all update states, so any stuck "Timed out" or "Failed" badges will resolve on their own after a restart.
|
||||
</Note>
|
||||
|
||||
If you run into issues with remote updates, see the [Troubleshooting](/operations/troubleshooting#remote-update-button-does-not-appear) page.
|
||||
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 32 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 32 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 28 KiB After Width: | Height: | Size: 33 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 32 KiB |
@@ -328,9 +328,60 @@ Nodes running very old Sencho versions that predate the capability system need t
|
||||
|
||||
**Symptom:** After clicking **Update** on a remote node, the status changes to **Timed out** instead of **Updated**.
|
||||
|
||||
**Cause:** The image pull may be slow on the remote host, or the container failed to restart.
|
||||
**Cause:** The gateway waits up to 5 minutes for the remote node to come back online after triggering the update. A timeout can happen if:
|
||||
|
||||
**Fix:** Check the Docker logs on the remote host for errors. You can also update manually by running:
|
||||
- The image pull is very slow on the remote host (large image, slow connection)
|
||||
- The container failed to restart after pulling
|
||||
- Network connectivity was lost between the gateway and remote node during the update
|
||||
|
||||
**Fix:**
|
||||
|
||||
1. Hover over the **Timed out** badge to see the error message for more detail.
|
||||
2. Click the **Retry** button (circular arrow) next to the badge to try again.
|
||||
3. If the issue persists, click **Dismiss** (X button) to clear the state, then check the Docker logs on the remote host directly.
|
||||
4. You can also click **Recheck** in the dialog footer to clear all failed states and re-scan every node.
|
||||
5. As a last resort, update manually on the remote host:
|
||||
|
||||
```bash
|
||||
docker compose pull && docker compose up -d
|
||||
```
|
||||
|
||||
<Note>
|
||||
Update state is tracked in memory on the gateway. Restarting the gateway clears all stuck states automatically.
|
||||
</Note>
|
||||
|
||||
---
|
||||
|
||||
## Remote update shows "Failed"
|
||||
|
||||
**Symptom:** After clicking **Update**, the status shows **Failed** with an error message.
|
||||
|
||||
**Cause:** There are two common scenarios:
|
||||
|
||||
1. **Immediate failure:** The gateway could not reach the remote node's update endpoint, or the remote node rejected the request. Common reasons include network issues, expired API tokens, or the remote node not supporting self-update.
|
||||
|
||||
2. **Delayed failure (after ~90 seconds):** The update was accepted by the remote node, but the node never restarted. This usually means the image pull failed on the remote host. The error message will read: *"Update may have failed. The node is still running and its version has not changed."*
|
||||
|
||||
**Fix:**
|
||||
|
||||
1. Hover over the **Failed** badge to see the specific error message.
|
||||
2. For immediate failures: verify the remote node is online and the API token is valid.
|
||||
3. For delayed failures: check the Docker logs on the remote host. The pull may have failed due to network issues, registry authentication, or disk space.
|
||||
4. Click **Retry** to attempt the update again after resolving the issue.
|
||||
|
||||
---
|
||||
|
||||
## First remote update always times out on old nodes
|
||||
|
||||
**Symptom:** After triggering a remote update on a node running a very old Sencho version (pre-v0.39.3), the node successfully restarts with the new version, but the dashboard shows **Timed out** or **Failed** instead of **Updated**.
|
||||
|
||||
**Cause:** The gateway detects update completion using three signals: a version change, a process restart (via `startedAt`), or temporary offline detection. Nodes running Sencho versions older than v0.39.3 report their version as `"unknown"` and do not include `startedAt` in their metadata. If the container restart is brief enough that the gateway does not detect the node going offline, none of the three signals fire.
|
||||
|
||||
**Fix:** This only affects the **first** update from a very old version. After the update completes (even though the dashboard may show "Failed" or "Timed out"), the remote node will be running the latest version with proper version reporting and restart detection. Future updates triggered from the dashboard will complete normally.
|
||||
|
||||
1. Click **Dismiss** on the badge to clear the state.
|
||||
2. Click **Recheck** to verify the node is now running the latest version.
|
||||
3. If the node still shows "unknown", update it manually on the remote host:
|
||||
|
||||
```bash
|
||||
docker compose pull && docker compose up -d
|
||||
|
||||
Generated
+4
-4
@@ -5825,9 +5825,9 @@
|
||||
}
|
||||
},
|
||||
"node_modules/vite": {
|
||||
"version": "8.0.3",
|
||||
"resolved": "https://registry.npmjs.org/vite/-/vite-8.0.3.tgz",
|
||||
"integrity": "sha512-B9ifbFudT1TFhfltfaIPgjo9Z3mDynBTJSUYxTjOQruf/zHH+ezCQKcoqO+h7a9Pw9Nm/OtlXAiGT1axBgwqrQ==",
|
||||
"version": "8.0.5",
|
||||
"resolved": "https://registry.npmjs.org/vite/-/vite-8.0.5.tgz",
|
||||
"integrity": "sha512-nmu43Qvq9UopTRfMx2jOYW5l16pb3iDC1JH6yMuPkpVbzK0k+L7dfsEDH4jRgYFmsg0sTAqkojoZgzLMlwHsCQ==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
@@ -5852,7 +5852,7 @@
|
||||
"peerDependencies": {
|
||||
"@types/node": "^20.19.0 || >=22.12.0",
|
||||
"@vitejs/devtools": "^0.1.0",
|
||||
"esbuild": "^0.27.0",
|
||||
"esbuild": "^0.27.0 || ^0.28.0",
|
||||
"jiti": ">=1.21.0",
|
||||
"less": "^4.0.0",
|
||||
"sass": "^1.70.0",
|
||||
|
||||
@@ -3,7 +3,7 @@ import {
|
||||
Server, Cpu, MemoryStick, HardDrive, RefreshCw, ChevronDown, ChevronRight,
|
||||
Layers, Wifi, WifiOff, Search, ArrowUpDown, AlertTriangle, Box, Activity,
|
||||
Play, Square, RotateCcw, ExternalLink, Camera, Download, Loader2, Check,
|
||||
CircleCheck, CircleAlert, Globe, Monitor,
|
||||
CircleCheck, CircleAlert, Globe, Monitor, X,
|
||||
} from 'lucide-react';
|
||||
import { Button } from '@/components/ui/button';
|
||||
import { Badge } from '@/components/ui/badge';
|
||||
@@ -29,6 +29,7 @@ import { toast } from '@/components/ui/toast-store';
|
||||
import { LabelDot, type Label as StackLabel } from './LabelPill';
|
||||
import { MultiSelectCombobox } from '@/components/ui/multi-select-combobox';
|
||||
import { formatVersion } from '@/lib/version';
|
||||
import { CursorProvider, Cursor, CursorFollow, CursorContainer } from '@/components/animate-ui/primitives/animate/cursor';
|
||||
|
||||
// --- Types ---
|
||||
|
||||
@@ -72,6 +73,7 @@ interface NodeUpdateStatus {
|
||||
latestVersion: string | null;
|
||||
updateAvailable: boolean;
|
||||
updateStatus: 'updating' | 'completed' | 'timeout' | 'failed' | null;
|
||||
error?: string | null;
|
||||
}
|
||||
|
||||
type SortField = 'name' | 'cpu' | 'memory' | 'containers' | 'status';
|
||||
@@ -288,7 +290,12 @@ function StackSection({ stackName, nodeId, onNavigate, labelMap }: {
|
||||
);
|
||||
}
|
||||
|
||||
function UpdateStatusBadge({ status }: { status: NodeUpdateStatus['updateStatus'] }) {
|
||||
function UpdateStatusBadge({ status, error, onRetry, onDismiss }: {
|
||||
status: NodeUpdateStatus['updateStatus'];
|
||||
error?: string | null;
|
||||
onRetry?: () => void;
|
||||
onDismiss?: () => void;
|
||||
}) {
|
||||
if (status === 'updating') return (
|
||||
<Badge className="text-[10px] px-1.5 py-0 h-4 bg-info/15 text-info border-info/30 shrink-0">
|
||||
<Loader2 className="w-2.5 h-2.5 mr-0.5 animate-spin" /> Updating
|
||||
@@ -299,12 +306,46 @@ function UpdateStatusBadge({ status }: { status: NodeUpdateStatus['updateStatus'
|
||||
<Check className="w-2.5 h-2.5 mr-0.5" /> Updated
|
||||
</Badge>
|
||||
);
|
||||
if (status === 'timeout') return (
|
||||
<Badge variant="destructive" className="text-[10px] px-1.5 py-0 h-4 shrink-0">Timed out</Badge>
|
||||
);
|
||||
if (status === 'failed') return (
|
||||
<Badge variant="destructive" className="text-[10px] px-1.5 py-0 h-4 shrink-0">Failed</Badge>
|
||||
);
|
||||
if (status === 'timeout' || status === 'failed') {
|
||||
const label = status === 'timeout' ? 'Timed out' : 'Failed';
|
||||
return (
|
||||
<CursorProvider>
|
||||
<CursorContainer className="flex items-center gap-1">
|
||||
<Badge variant="destructive" className="text-[10px] px-1.5 py-0 h-4 shrink-0">{label}</Badge>
|
||||
{onRetry && (
|
||||
<button
|
||||
onClick={(e) => { e.stopPropagation(); onRetry(); }}
|
||||
className="h-5 w-5 flex items-center justify-center rounded text-muted-foreground hover:text-foreground hover:bg-muted transition-colors"
|
||||
title="Retry update"
|
||||
>
|
||||
<RotateCcw className="w-3 h-3" strokeWidth={1.5} />
|
||||
</button>
|
||||
)}
|
||||
{onDismiss && (
|
||||
<button
|
||||
onClick={(e) => { e.stopPropagation(); onDismiss(); }}
|
||||
className="h-5 w-5 flex items-center justify-center rounded text-muted-foreground hover:text-foreground hover:bg-muted transition-colors"
|
||||
title="Dismiss"
|
||||
>
|
||||
<X className="w-3 h-3" strokeWidth={1.5} />
|
||||
</button>
|
||||
)}
|
||||
{error && (
|
||||
<>
|
||||
<Cursor>
|
||||
<div className="h-2 w-2 rounded-full bg-destructive/60" />
|
||||
</Cursor>
|
||||
<CursorFollow side="bottom" sideOffset={8} align="end" alignOffset={0}>
|
||||
<div className="bg-popover/95 backdrop-blur-[10px] backdrop-saturate-[1.15] border border-card-border shadow-md rounded-lg px-3 py-2 max-w-xs">
|
||||
<p className="font-mono tabular-nums text-xs text-stat-subtitle">{error}</p>
|
||||
</div>
|
||||
</CursorFollow>
|
||||
</>
|
||||
)}
|
||||
</CursorContainer>
|
||||
</CursorProvider>
|
||||
);
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
@@ -357,7 +398,18 @@ function ReconnectingOverlay() {
|
||||
);
|
||||
}
|
||||
|
||||
function NodeCard({ node, onNavigate, labelMap, updateStatus, onUpdate, updatingNodeId }: { node: FleetNode; onNavigate: (nodeId: number, stackName: string) => void; labelMap?: Record<string, StackLabel[]>; updateStatus?: NodeUpdateStatus; onUpdate?: (nodeId: number) => void; updatingNodeId?: number | null }) {
|
||||
interface NodeCardProps {
|
||||
node: FleetNode;
|
||||
onNavigate: (nodeId: number, stackName: string) => void;
|
||||
labelMap?: Record<string, StackLabel[]>;
|
||||
updateStatus?: NodeUpdateStatus;
|
||||
onUpdate?: (nodeId: number) => void;
|
||||
updatingNodeId?: number | null;
|
||||
onRetryUpdate?: (nodeId: number) => void;
|
||||
onDismissUpdate?: (nodeId: number) => void;
|
||||
}
|
||||
|
||||
function NodeCard({ node, onNavigate, labelMap, updateStatus, onUpdate, updatingNodeId, onRetryUpdate, onDismissUpdate }: NodeCardProps) {
|
||||
const { isPaid } = useLicense();
|
||||
const [expanded, setExpanded] = useState(false);
|
||||
const [stacks, setStacks] = useState<string[] | null>(node.stacks);
|
||||
@@ -419,7 +471,14 @@ function NodeCard({ node, onNavigate, labelMap, updateStatus, onUpdate, updating
|
||||
{formattedVersion}
|
||||
</Badge>
|
||||
)}
|
||||
{updateStatus?.updateStatus && <UpdateStatusBadge status={updateStatus.updateStatus} />}
|
||||
{updateStatus?.updateStatus && (
|
||||
<UpdateStatusBadge
|
||||
status={updateStatus.updateStatus}
|
||||
error={updateStatus.error}
|
||||
onRetry={onRetryUpdate ? () => onRetryUpdate(node.id) : undefined}
|
||||
onDismiss={onDismissUpdate ? () => onDismissUpdate(node.id) : undefined}
|
||||
/>
|
||||
)}
|
||||
{updateStatus?.updateAvailable && !updateStatus.updateStatus && (
|
||||
<Badge className="text-[10px] px-1.5 py-0 h-4 bg-warning/15 text-warning border-warning/30 shrink-0">
|
||||
Update available
|
||||
@@ -701,6 +760,19 @@ export function FleetView({ onNavigateToNode }: FleetViewProps) {
|
||||
}
|
||||
}, [fetchUpdateStatus]);
|
||||
|
||||
const dismissNodeUpdate = useCallback(async (nodeId: number) => {
|
||||
try {
|
||||
await apiFetch(`/fleet/nodes/${nodeId}/update-status`, { method: 'DELETE', localOnly: true });
|
||||
fetchUpdateStatus();
|
||||
} catch (error) {
|
||||
console.error('[Fleet] Failed to dismiss update status:', error);
|
||||
}
|
||||
}, [fetchUpdateStatus]);
|
||||
|
||||
const retryNodeUpdate = useCallback(async (nodeId: number) => {
|
||||
triggerNodeUpdate(nodeId);
|
||||
}, [triggerNodeUpdate]);
|
||||
|
||||
useEffect(() => {
|
||||
fetchOverview();
|
||||
fetchLabels();
|
||||
@@ -1056,6 +1128,8 @@ export function FleetView({ onNavigateToNode }: FleetViewProps) {
|
||||
updateStatus={updateStatusMap.get(node.id)}
|
||||
onUpdate={isPaid ? triggerNodeUpdate : undefined}
|
||||
updatingNodeId={updatingNodeId}
|
||||
onRetryUpdate={isPaid ? retryNodeUpdate : undefined}
|
||||
onDismissUpdate={isPaid ? dismissNodeUpdate : undefined}
|
||||
/>
|
||||
))}
|
||||
</div>
|
||||
@@ -1234,7 +1308,14 @@ export function FleetView({ onNavigateToNode }: FleetViewProps) {
|
||||
|
||||
{/* Status / Action */}
|
||||
<div className="flex justify-end">
|
||||
{s.updateStatus && <UpdateStatusBadge status={s.updateStatus} />}
|
||||
{s.updateStatus && (
|
||||
<UpdateStatusBadge
|
||||
status={s.updateStatus}
|
||||
error={s.error}
|
||||
onRetry={() => retryNodeUpdate(s.nodeId)}
|
||||
onDismiss={() => dismissNodeUpdate(s.nodeId)}
|
||||
/>
|
||||
)}
|
||||
{!s.updateStatus && !s.updateAvailable && (
|
||||
<Badge className="text-[10px] px-1.5 py-0 h-5 bg-success-muted text-success border-success/30">
|
||||
<Check className="w-2.5 h-2.5 mr-0.5" /> Up to date
|
||||
@@ -1271,9 +1352,15 @@ export function FleetView({ onNavigateToNode }: FleetViewProps) {
|
||||
variant="ghost"
|
||||
size="sm"
|
||||
className="h-7 text-xs text-muted-foreground"
|
||||
onClick={async () => { setCheckingUpdates(true); await fetchUpdateStatus(); setCheckingUpdates(false); }}
|
||||
disabled={checkingUpdates}
|
||||
onClick={async () => {
|
||||
setCheckingUpdates(true);
|
||||
await apiFetch('/fleet/update-status', { method: 'DELETE', localOnly: true });
|
||||
await fetchUpdateStatus();
|
||||
setCheckingUpdates(false);
|
||||
}}
|
||||
>
|
||||
<RefreshCw className="w-3 h-3 mr-1.5" strokeWidth={1.5} />
|
||||
<RefreshCw className={`w-3 h-3 mr-1.5 ${checkingUpdates ? 'animate-spin' : ''}`} strokeWidth={1.5} />
|
||||
Recheck
|
||||
</Button>
|
||||
{updatableRemoteCount > 0 && (
|
||||
|
||||
Reference in New Issue
Block a user