fix(fleet): make local self-update flow reliable end-to-end (#472)

The "Updating Sencho..." overlay used to dismiss prematurely while the
image pull was still running, after which the local node card would get
stuck in "updating" and eventually surface a generic "Timed Out" error
while the container remained on the old version.

Three root causes are addressed:

1. The image pull was synchronous (`execFileSync`), which blocked the
   Node event loop. The overlay's health probe saw the server come back
   the moment the pull finished and reloaded the page, even though the
   container had not restarted yet. The pull is now async via
   `promisify(execFile)`, so /api/health and /api/fleet/update-status
   keep serving throughout.

2. The overlay reloaded on the first 200 from /api/health regardless of
   whether the underlying process had actually restarted. /api/health
   now exposes the gateway boot timestamp, and the overlay captures it
   pre-update and only reloads when it observes a different value. A
   wasOffline-then-online fallback handles the case where the pre-update
   fetch failed.

3. Helper container spawn errors from `docker run` were silently
   discarded, so a failed compose recreate never surfaced anywhere.
   Errors are now captured into `lastUpdateError` via the execFile
   callback and surfaced through the existing /api/fleet/update-status
   error path.

A 3-minute early-fail heuristic on the local node block surfaces a clear
failure message when the helper fails silently, instead of waiting the
full 5-minute timeout for an unknown failure.
This commit is contained in:
Anso
2026-04-10 12:00:44 -04:00
committed by GitHub
parent c09673d8bb
commit 3d69746eee
5 changed files with 103 additions and 22 deletions
+26 -7
View File
@@ -491,15 +491,17 @@ const authRateLimiter = rateLimit({
message: { error: 'Too many attempts. Please try again in 15 minutes.' },
});
// Captured at boot. Exposed via /api/health and /api/meta so the Fleet update overlay
// can distinguish a brand-new process from the old one still mid-pull.
const processStartedAt = Date.now();
// Public health endpoint - no auth required (used by Docker HEALTHCHECK and uptime monitors)
app.get('/api/health', (_req: Request, res: Response): void => {
res.json({ status: 'ok', uptime: process.uptime() });
res.json({ status: 'ok', uptime: process.uptime(), startedAt: processStartedAt });
});
// Public meta endpoint - returns this instance's version and supported capabilities.
// No auth required (like /health). Used by remote nodes during connection tests.
const processStartedAt = Date.now();
app.get('/api/meta', (_req: Request, res: Response): void => {
const updateError = SelfUpdateService.getInstance().getLastError();
res.json({
@@ -1243,7 +1245,13 @@ app.get('/api/license/billing-portal', async (_req: Request, res: Response): Pro
function scheduleLocalUpdate(res: Response, message: string): void {
res.status(202).json({ message });
res.on('finish', () => {
setTimeout(() => SelfUpdateService.getInstance().triggerUpdate(), 500);
setTimeout(() => {
// Defense in depth: triggerUpdate records its own errors into lastUpdateError,
// but guard against an unexpected throw becoming an unhandled rejection.
SelfUpdateService.getInstance().triggerUpdate().catch((err) => {
console.error('[SelfUpdate] Unexpected error during triggerUpdate:', err);
});
}, 500);
});
}
@@ -1575,11 +1583,22 @@ app.get('/api/fleet/update-status', async (_req: Request, res: Response): Promis
});
}
} else if (node.type === 'local') {
// Local node: check if SelfUpdateService reported a pull failure
const localError = SelfUpdateService.getInstance().getLastError();
// Local node has only two failure signals: an explicit pull/spawn error,
// or the early-fail heuristic. Success is observed by the frontend overlay
// (it reloads the page when /api/health reports a new startedAt), at which
// point the new process starts with an empty tracker map.
const selfUpdate = SelfUpdateService.getInstance();
const localError = selfUpdate.getLastError();
if (localError) {
updateTracker.set(node.id, { ...tracker, status: 'failed', error: localError });
SelfUpdateService.getInstance().clearLastError();
selfUpdate.clearLastError();
} else if (elapsed > EARLY_FAIL_MS) {
// Helper container likely failed silently. Surface failure before the 5 min timeout.
updateTracker.set(node.id, {
...tracker,
status: 'failed',
error: 'Local update did not complete. The container may not have restarted; check Docker logs on the host.',
});
}
}
}