feat: detect stalled stack updates and add in-app recovery actions (#1347)

* feat: detect stalled stack updates and add in-app recovery actions

Add a backend idle-output backstop that stops a deploy/update compose step
that has gone silent (SENCHO_COMPOSE_STALL_TIMEOUT_MS, default 10m), so a
hung image pull surfaces a fast failure instead of spinning indefinitely.

Surface failed, timed-out, and stalled operations with recovery actions on
the stack page: a desktop chip plus popover menu and an inline mobile card
offering retry, restart, roll back (when a backup exists), refresh state,
and copy diagnostics, all gated by deploy permission. The streaming
deploy/update progress modal is now on by default and warns when output
goes quiet. Container state is refreshed after a failed or stalled
operation, and the UI never sits in an indefinite spinner.

* fix: harden rollback against policy-blocked file mutation and refine recovery

Address review findings on the stalled-update recovery work:

- The rollback route restored backup files before running the policy gate, so
  a policy-blocked rollback could leave the on-disk config rolled back while the
  deployed containers were unchanged. Snapshot the current files first and
  revert them when the gate blocks; if that revert itself fails, escalate it on
  the persistent alert feed since the 409 is already sent.
- Refresh container state after a successful manual rollback (rollback
  redeploys), without mis-recording a refetch failure as a rollback failure.
- Suppress the stalled-output warning once live progress is unavailable.

* test: mock snapshotStackFiles in the atomic-deploy rollback route tests

The rollback route now snapshots stack files before restoring a backup, so its
FileSystemService mock needs snapshotStackFiles. Without it the mocked call
threw and the route returned 500, failing the success-path rollback assertions.

backend/src/services/FileSystemService.ts

@@ -874,6 +874,52 @@ export class FileSystemService {
     if (debug) console.debug(`[FileSystemService:debug] Restore completed in ${Date.now() - t0}ms`, { stackName, restored: items.filter(i => i !== '.timestamp').length, removedOrphans });
   }
 
+  /**
+   * Capture the current managed stack files (PROTECTED_STACK_FILES) in memory and
+   * return a function that puts them back, faithfully (writing the captured
+   * contents and removing any managed file that did not exist when captured).
+   *
+   * Used by the rollback route to undo a restored backup when the policy gate
+   * blocks before the deploy commits: restoreStackFiles has already overwritten
+   * the on-disk files, so without this a blocked rollback would leave disk holding
+   * the rolled-back configuration while the deployed containers are unchanged.
+   */
+  async snapshotStackFiles(stackName: string): Promise<() => Promise<void>> {
+    const stackDir = this.resolveStackDir(stackName);
+    // Canonical js/path-injection barrier inline with the read/write sinks, the
+    // same pattern restoreStackFiles uses: resolve against the base and confirm
+    // containment so static analysis credits the barrier.
+    const baseResolved = path.resolve(this.baseDir);
+    const snapshot = new Map<string, Buffer>();
+    for (const file of PROTECTED_STACK_FILES) {
+      const target = path.resolve(baseResolved, path.join(stackDir, file));
+      if (!target.startsWith(baseResolved + path.sep)) {
+        throw Object.assign(new Error('Path escapes compose directory'), { code: 'INVALID_PATH' });
+      }
+      try {
+        snapshot.set(file, await fsPromises.readFile(target));
+      } catch (e: unknown) {
+        if ((e as NodeJS.ErrnoException)?.code !== 'ENOENT') throw e;
+      }
+    }
+    return async () => {
+      for (const file of PROTECTED_STACK_FILES) {
+        const target = path.resolve(baseResolved, path.join(stackDir, file));
+        if (!target.startsWith(baseResolved + path.sep)) continue;
+        const saved = snapshot.get(file);
+        if (saved !== undefined) {
+          await fsPromises.writeFile(target, saved);
+        } else {
+          try {
+            await fsPromises.unlink(target);
+          } catch (e: unknown) {
+            if ((e as NodeJS.ErrnoException)?.code !== 'ENOENT') throw e;
+          }
+        }
+      }
+    };
+  }
+
   async getBackupInfo(stackName: string): Promise<{ exists: boolean; timestamp: number | null }> {
     const backupDir = this.getBackupDir(stackName);
     try {