feat: detect stalled stack updates and add in-app recovery actions (#1347)

* feat: detect stalled stack updates and add in-app recovery actions

Add a backend idle-output backstop that stops a deploy/update compose step
that has gone silent (SENCHO_COMPOSE_STALL_TIMEOUT_MS, default 10m), so a
hung image pull surfaces a fast failure instead of spinning indefinitely.

Surface failed, timed-out, and stalled operations with recovery actions on
the stack page: a desktop chip plus popover menu and an inline mobile card
offering retry, restart, roll back (when a backup exists), refresh state,
and copy diagnostics, all gated by deploy permission. The streaming
deploy/update progress modal is now on by default and warns when output
goes quiet. Container state is refreshed after a failed or stalled
operation, and the UI never sits in an indefinite spinner.

* fix: harden rollback against policy-blocked file mutation and refine recovery

Address review findings on the stalled-update recovery work:

- The rollback route restored backup files before running the policy gate, so
  a policy-blocked rollback could leave the on-disk config rolled back while the
  deployed containers were unchanged. Snapshot the current files first and
  revert them when the gate blocks; if that revert itself fails, escalate it on
  the persistent alert feed since the 409 is already sent.
- Refresh container state after a successful manual rollback (rollback
  redeploys), without mis-recording a refetch failure as a rollback failure.
- Suppress the stalled-output warning once live progress is unavailable.

* test: mock snapshotStackFiles in the atomic-deploy rollback route tests

The rollback route now snapshots stack files before restoring a backup, so its
FileSystemService mock needs snapshotStackFiles. Without it the mocked call
threw and the route returned 500, failing the success-path rollback assertions.

frontend/src/components/EditorLayout.tsx

@@ -52,7 +52,19 @@ import type { SectionId } from './settings/types';
 export default function EditorLayout() {
   const { isAdmin, can } = useAuth();
   const { status: trivy } = useTrivyStatus();
-  const { runWithLog, panelState } = useDeployFeedback();
+  const { runWithLog, panelState, logRows } = useDeployFeedback();
+
+  // The last live output line captured for a stack while its deploy-feedback
+  // session is still streaming, used to enrich a failure record's diagnostics.
+  // Guards on both the streaming status and an exact stack-name match, so
+  // neither another stack's output nor a finished session's stale line leaks in.
+  const getLastDeployOutputLine = useCallback(
+    (forStack: string): string | undefined =>
+      panelState.status === 'streaming' && panelState.stackName === forStack
+        ? logRows.at(-1)?.message
+        : undefined,
+    [panelState.status, panelState.stackName, logRows],
+  );
 
   const editorState = useEditorViewState();
   const {
@@ -101,6 +113,9 @@ export default function EditorLayout() {
     isCollapsed, toggleCollapse,
     remoteSearchLoading,
     remoteSearchFailedNodes,
+    lastActionResult,
+    clearActionRecords,
+    dismissActionResult,
   } = stackListState;
 
   const { nodes, activeNode, setActiveNode } = useNodes();
@@ -176,6 +191,7 @@ export default function EditorLayout() {
     setActiveNode,
     nodes,
     runWithLog,
+    getLastDeployOutputLine,
     diffPreviewEnabled,
   });
 
@@ -368,6 +384,16 @@ export default function EditorLayout() {
       setGitSourceOpen={setGitSourceOpen}
       setCopiedDigest={setCopiedDigest}
       requestDeleteStack={stackActions.requestDeleteStack}
+      recoveryResult={selectedFile ? lastActionResult[selectedFile] : undefined}
+      onRefreshState={async () => {
+        if (!selectedFile) return;
+        const name = selectedFile.replace(/\.(yml|yaml)$/, '');
+        const ok = await stackActions.refreshSelectedContainers(name, selectedFile);
+        await refreshStacks(true);
+        if (ok) toast.success('Refreshed container state.');
+        else toast.error('Could not refresh container state.');
+      }}
+      onDismissRecovery={() => { if (selectedFile) dismissActionResult(selectedFile); }}
       onMobileBack={goToMobileList}
     />
   );
@@ -429,6 +455,9 @@ export default function EditorLayout() {
     pendingStackLoadRef.current = null;
 
     stackActions.resetEditorState();
+    // Stack filenames can repeat across nodes; drop the previous node's failure
+    // records so a stale recovery panel cannot surface on the new node.
+    clearActionRecords();
 
     if (pendingStack) {
       void stackActions.loadFile(pendingStack);