gsadmin/sencho
1
0
Fork 0
mirror of https://github.com/Studio-Saelix/sencho.git synced 2026-08-11 03:06:54 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat: detect stalled stack updates and add in-app recovery actions (#1347)

Browse Source 
* feat: detect stalled stack updates and add in-app recovery actions

Add a backend idle-output backstop that stops a deploy/update compose step
that has gone silent (SENCHO_COMPOSE_STALL_TIMEOUT_MS, default 10m), so a
hung image pull surfaces a fast failure instead of spinning indefinitely.

Surface failed, timed-out, and stalled operations with recovery actions on
the stack page: a desktop chip plus popover menu and an inline mobile card
offering retry, restart, roll back (when a backup exists), refresh state,
and copy diagnostics, all gated by deploy permission. The streaming
deploy/update progress modal is now on by default and warns when output
goes quiet. Container state is refreshed after a failed or stalled
operation, and the UI never sits in an indefinite spinner.

* fix: harden rollback against policy-blocked file mutation and refine recovery

Address review findings on the stalled-update recovery work:

- The rollback route restored backup files before running the policy gate, so
  a policy-blocked rollback could leave the on-disk config rolled back while the
  deployed containers were unchanged. Snapshot the current files first and
  revert them when the gate blocks; if that revert itself fails, escalate it on
  the persistent alert feed since the 409 is already sent.
- Refresh container state after a successful manual rollback (rollback
  redeploys), without mis-recording a refetch failure as a rollback failure.
- Suppress the stalled-output warning once live progress is unavailable.

* test: mock snapshotStackFiles in the atomic-deploy rollback route tests

The rollback route now snapshots stack files before restoring a backup, so its
FileSystemService mock needs snapshotStackFiles. Without it the mocked call
threw and the route returned 500, failing the success-path rollback assertions.
This commit is contained in:
Anso
2026-06-10 10:12:24 -04:00
committed by GitHub
parent a3033a848e
commit d369b03a38
31 changed files with 1580 additions and 76 deletions
Download Patch File Download Diff File Expand all files Collapse all files
backend/src/routes/stacks.ts
+17 -1
View File
@@ -1421,8 +1421,24 @@ stacksRouter.post('/:stackName/rollback', async (req: Request, res: Response) =>
return;
}
dlog(`[Stacks] Rollback initiated: ${sanitizeForLog(stackName)}`);
// Snapshot the current files before restoring so a policy gate that blocks
// the restored target can be undone: restoreStackFiles commits to disk, and
// without this a blocked rollback would leave disk rolled back while the
// deployed state is unchanged.
const revertRestore = await fsSvc.snapshotStackFiles(stackName);
await fsSvc.restoreStackFiles(stackName);
if (!(await runPolicyGate(req, res, stackName, req.nodeId))) return;
if (!(await runPolicyGate(req, res, stackName, req.nodeId))) {
try {
await revertRestore();
} catch (revertError) {
console.error('[Stacks] Failed to revert files after a policy-blocked rollback: %s', sanitizeForLog(stackName), revertError);
// The 409 is already sent and the on-disk config now diverges from the
// running stack; surface it on the persistent alert feed so the operator
// can repair it rather than discovering it on the next deploy.
notifyActionFailure('rollback', stackName, revertError, req.user?.username ?? 'system');
}
return;
}
await ComposeService.getInstance(req.nodeId).deployStack(stackName, getTerminalWs(req.get(DEPLOY_SESSION_HEADER)), false);
invalidateNodeCaches(req.nodeId);
dlog(`[Stacks] Rollback completed: ${sanitizeForLog(stackName)}`);