feat: detect stalled stack updates and add in-app recovery actions (#1347)

* feat: detect stalled stack updates and add in-app recovery actions

Add a backend idle-output backstop that stops a deploy/update compose step
that has gone silent (SENCHO_COMPOSE_STALL_TIMEOUT_MS, default 10m), so a
hung image pull surfaces a fast failure instead of spinning indefinitely.

Surface failed, timed-out, and stalled operations with recovery actions on
the stack page: a desktop chip plus popover menu and an inline mobile card
offering retry, restart, roll back (when a backup exists), refresh state,
and copy diagnostics, all gated by deploy permission. The streaming
deploy/update progress modal is now on by default and warns when output
goes quiet. Container state is refreshed after a failed or stalled
operation, and the UI never sits in an indefinite spinner.

* fix: harden rollback against policy-blocked file mutation and refine recovery

Address review findings on the stalled-update recovery work:

- The rollback route restored backup files before running the policy gate, so
  a policy-blocked rollback could leave the on-disk config rolled back while the
  deployed containers were unchanged. Snapshot the current files first and
  revert them when the gate blocks; if that revert itself fails, escalate it on
  the persistent alert feed since the 409 is already sent.
- Refresh container state after a successful manual rollback (rollback
  redeploys), without mis-recording a refetch failure as a rollback failure.
- Suppress the stalled-output warning once live progress is unavailable.

* test: mock snapshotStackFiles in the atomic-deploy rollback route tests

The rollback route now snapshots stack files before restoring a backup, so its
FileSystemService mock needs snapshotStackFiles. Without it the mocked call
threw and the route returned 500, failing the success-path rollback assertions.

frontend/src/components/DeployFeedbackModal.tsx

@@ -3,6 +3,7 @@ import {
   Loader2,
   CheckCircle2,
   AlertCircle,
+  AlertTriangle,
   X,
   Minimize2,
   Terminal as TerminalIcon,
@@ -16,6 +17,11 @@ import { useDeployFeedback, VERB_LABELS } from '@/context/DeployFeedbackContext'
 
 const AUTO_CLOSE_SECONDS = 4;
 
+// Warn that an in-flight operation has gone quiet after this much silence. The
+// backend idle-output timeout terminates a truly hung step later (default
+// ~10min); this earlier heads-up keeps the modal from looking falsely busy.
+const STALL_WARN_MS = 75_000;
+
 interface DeployFeedbackModalProps {
   isMinimized: boolean;
   onMinimize: () => void;
@@ -31,7 +37,7 @@ function formatElapsed(seconds: number): string {
 }
 
 export function DeployFeedbackModal({ isMinimized, onMinimize }: DeployFeedbackModalProps) {
-  const { panelState, logRows, onTerminalReady, onTerminalError, onMessage, onPanelClose } = useDeployFeedback();
+  const { panelState, logRows, lastOutputAt, onTerminalReady, onTerminalError, onMessage, onPanelClose } = useDeployFeedback();
 
   const [showRaw, setShowRaw] = useState(false);
   const [elapsedSeconds, setElapsedSeconds] = useState(0);
@@ -152,6 +158,15 @@ export function DeployFeedbackModal({ isMinimized, onMinimize }: DeployFeedbackM
   const isDialogOpen = isOpen && !isMinimized;
   const verbLabel = VERB_LABELS[action];
 
+  // Re-evaluated each second by the elapsed-time interval's re-render. Warns
+  // while the operation is still streaming but has produced no output for a
+  // while, including the case where no first line ever arrived. Suppressed once
+  // the progress stream is unavailable: no more output can arrive, so the modal
+  // already shows "Live progress unavailable" and a stall warning would be noise.
+  const lastLine = logRows.length > 0 ? logRows[logRows.length - 1].message : null;
+  const secondsSinceOutput = lastOutputAt > 0 ? Math.floor((Date.now() - lastOutputAt) / 1000) : 0;
+  const stalled = status === 'streaming' && !progressUnavailable && lastOutputAt > 0 && Date.now() - lastOutputAt > STALL_WARN_MS;
+
   return (
     <Modal
       open={isDialogOpen}
@@ -214,6 +229,28 @@ export function DeployFeedbackModal({ isMinimized, onMinimize }: DeployFeedbackM
           </div>
         </div>
 
+        {/* Stalled-output warning: in-flight but quiet */}
+        {stalled && (
+          <div
+            data-testid="deploy-feedback-stalled"
+            className="flex items-start gap-2 px-4 py-2 border-b border-warning/30 bg-warning/5 shrink-0"
+          >
+            <AlertTriangle className="h-3.5 w-3.5 mt-0.5 shrink-0 text-warning" />
+            <div className="min-w-0 text-xs text-warning">
+              <p>
+                {lastLine
+                  ? `No new output for ${formatElapsed(secondsSinceOutput)}. The operation may be stalled.`
+                  : 'No output received yet. The operation may be stalled.'}
+              </p>
+              {lastLine && (
+                <p className="mt-0.5 truncate font-mono text-[11px] text-warning/80" title={lastLine}>
+                  {lastLine}
+                </p>
+              )}
+            </div>
+          </div>
+        )}
+
         {/* Body */}
         <div
           ref={scrollRef}