feat: detect stalled stack updates and add in-app recovery actions (#1347)

* feat: detect stalled stack updates and add in-app recovery actions

Add a backend idle-output backstop that stops a deploy/update compose step
that has gone silent (SENCHO_COMPOSE_STALL_TIMEOUT_MS, default 10m), so a
hung image pull surfaces a fast failure instead of spinning indefinitely.

Surface failed, timed-out, and stalled operations with recovery actions on
the stack page: a desktop chip plus popover menu and an inline mobile card
offering retry, restart, roll back (when a backup exists), refresh state,
and copy diagnostics, all gated by deploy permission. The streaming
deploy/update progress modal is now on by default and warns when output
goes quiet. Container state is refreshed after a failed or stalled
operation, and the UI never sits in an indefinite spinner.

* fix: harden rollback against policy-blocked file mutation and refine recovery

Address review findings on the stalled-update recovery work:

- The rollback route restored backup files before running the policy gate, so
  a policy-blocked rollback could leave the on-disk config rolled back while the
  deployed containers were unchanged. Snapshot the current files first and
  revert them when the gate blocks; if that revert itself fails, escalate it on
  the persistent alert feed since the 409 is already sent.
- Refresh container state after a successful manual rollback (rollback
  redeploys), without mis-recording a refetch failure as a rollback failure.
- Suppress the stalled-output warning once live progress is unavailable.

* test: mock snapshotStackFiles in the atomic-deploy rollback route tests

The rollback route now snapshots stack files before restoring a backup, so its
FileSystemService mock needs snapshotStackFiles. Without it the mocked call
threw and the route returned 500, failing the success-path rollback assertions.
This commit is contained in:
Anso
2026-06-10 10:12:24 -04:00
committed by GitHub
parent a3033a848e
commit d369b03a38
31 changed files with 1580 additions and 76 deletions
@@ -4,6 +4,8 @@ import { cn } from '@/lib/utils';
import ErrorBoundary from '../ErrorBoundary';
import StackAnatomyPanel from '../StackAnatomyPanel';
import { StackIdentityHeader, ContainersHealth, StackLogsSection } from './editor-view-blocks';
import { RecoveryPanel } from './RecoveryPanel';
import { retryHandlerFor } from './recovery-retry';
import type { EditorViewProps } from './EditorView';
const SEGMENTS = [
@@ -55,6 +57,9 @@ export function MobileStackDetail(props: EditorViewProps) {
requestDeleteStack,
onMobileBack,
headerActions,
recoveryResult,
onRefreshState,
onDismissRecovery,
} = props;
const [segment, setSegment] = useState<Segment>('logs');
@@ -104,6 +109,23 @@ export function MobileStackDetail(props: EditorViewProps) {
/>
</div>
{recoveryResult && loadingAction == null && (
<div className="shrink-0 px-4 pt-3">
<RecoveryPanel
stackName={stackName}
result={recoveryResult}
activeNode={activeNode}
backupInfo={backupInfo}
canDeploy={can('stack:deploy', 'stack', stackName)}
onRetry={retryHandlerFor(recoveryResult.action, { deployStack, restartStack, updateStack, rollbackStack })}
onRestart={restartStack}
onRollback={rollbackStack}
onRefreshState={onRefreshState}
onDismiss={onDismissRecovery}
/>
</div>
)}
{/* Segmented control: Health · Logs · Compose */}
<div className="shrink-0 px-4 pt-3">
<div