feat: detect stalled stack updates and add in-app recovery actions (#1347)

* feat: detect stalled stack updates and add in-app recovery actions

Add a backend idle-output backstop that stops a deploy/update compose step
that has gone silent (SENCHO_COMPOSE_STALL_TIMEOUT_MS, default 10m), so a
hung image pull surfaces a fast failure instead of spinning indefinitely.

Surface failed, timed-out, and stalled operations with recovery actions on
the stack page: a desktop chip plus popover menu and an inline mobile card
offering retry, restart, roll back (when a backup exists), refresh state,
and copy diagnostics, all gated by deploy permission. The streaming
deploy/update progress modal is now on by default and warns when output
goes quiet. Container state is refreshed after a failed or stalled
operation, and the UI never sits in an indefinite spinner.

* fix: harden rollback against policy-blocked file mutation and refine recovery

Address review findings on the stalled-update recovery work:

- The rollback route restored backup files before running the policy gate, so
  a policy-blocked rollback could leave the on-disk config rolled back while the
  deployed containers were unchanged. Snapshot the current files first and
  revert them when the gate blocks; if that revert itself fails, escalate it on
  the persistent alert feed since the 409 is already sent.
- Refresh container state after a successful manual rollback (rollback
  redeploys), without mis-recording a refetch failure as a rollback failure.
- Suppress the stalled-output warning once live progress is unavailable.

* test: mock snapshotStackFiles in the atomic-deploy rollback route tests

The rollback route now snapshots stack files before restoring a backup, so its
FileSystemService mock needs snapshotStackFiles. Without it the mocked call
threw and the route returned 500, failing the success-path rollback assertions.

frontend/src/components/EditorLayout/hooks/useStackListState.ts

@@ -9,7 +9,7 @@ import { useBulkStackActions, type BulkAction } from '@/hooks/useBulkStackAction
 import { useCrossNodeStackSearch } from '@/hooks/useCrossNodeStackSearch';
 import { SENCHO_LABELS_CHANGED } from '@/lib/events';
 import { isInputFocused, isPaletteOpen } from '@/lib/keyboard-guards';
-import type { StackAction, ContainerInfo } from '../EditorView';
+import type { StackAction, StackActionResult, ContainerInfo } from '../EditorView';
 import type { Label as StackLabel } from '../../label-types';
 import type { FilterChip } from '../../sidebar/sidebar-types';
 import type { StackRowStatus } from '../../sidebar/stack-status-utils';
@@ -38,6 +38,12 @@ export function useStackListState() {
   const [stackActions, setStackActions] = useState<Record<string, StackAction>>({});
   const stackActionsRef = useRef<Record<string, StackAction>>({});
 
+  // Per-stack terminal failure records driving the in-detail recovery panel.
+  // In-memory only. Node scoping is enforced by the caller, which clears these
+  // on active-node change (see EditorLayout's node-switch effect) so a repeated
+  // stack filename cannot carry a failure across nodes.
+  const [lastActionResult, setLastActionResult] = useState<Record<string, StackActionResult>>({});
+
   const [isScanning, setIsScanning] = useState(false);
   const [searchQuery, setSearchQuery] = useState('');
   const [stackStatuses, setStackStatuses] = useState<StackStatus>({});
@@ -85,6 +91,27 @@ export function useStackListState() {
     setStackStatuses(prev => ({ ...prev, [stackFile]: status }));
   };
 
+  // Recovery record lifecycle. recordActionFailure stores a terminal failure;
+  // recordActionSuccess / dismissActionResult drop it; clearActionRecords wipes
+  // all (node switch). The recovery panel itself renders only when the stack is
+  // not mid-operation, so a stale record never shows during a retry.
+  const clearStackResult = useCallback((stackFile: string) => {
+    setLastActionResult(prev => {
+      if (!(stackFile in prev)) return prev;
+      const next = { ...prev };
+      delete next[stackFile];
+      return next;
+    });
+  }, []);
+  const recordActionFailure = useCallback((stackFile: string, result: StackActionResult) => {
+    setLastActionResult(prev => ({ ...prev, [stackFile]: result }));
+  }, []);
+  const recordActionSuccess = clearStackResult;
+  const dismissActionResult = clearStackResult;
+  const clearActionRecords = useCallback(() => {
+    setLastActionResult({});
+  }, []);
+
   const refreshLabels = useCallback(async () => {
     try {
       const [labelsRes, assignmentsRes] = await Promise.all([
@@ -333,6 +360,8 @@ export function useStackListState() {
     remoteResults,
     setStackAction, clearStackAction, isStackBusy,
     setOptimisticStatus,
+    lastActionResult,
+    recordActionFailure, recordActionSuccess, clearActionRecords, dismissActionResult,
     refreshLabels,
     refreshStacks,
     handleScanStacks,