feat: detect stalled stack updates and add in-app recovery actions (#1347)

* feat: detect stalled stack updates and add in-app recovery actions

Add a backend idle-output backstop that stops a deploy/update compose step
that has gone silent (SENCHO_COMPOSE_STALL_TIMEOUT_MS, default 10m), so a
hung image pull surfaces a fast failure instead of spinning indefinitely.

Surface failed, timed-out, and stalled operations with recovery actions on
the stack page: a desktop chip plus popover menu and an inline mobile card
offering retry, restart, roll back (when a backup exists), refresh state,
and copy diagnostics, all gated by deploy permission. The streaming
deploy/update progress modal is now on by default and warns when output
goes quiet. Container state is refreshed after a failed or stalled
operation, and the UI never sits in an indefinite spinner.

* fix: harden rollback against policy-blocked file mutation and refine recovery

Address review findings on the stalled-update recovery work:

- The rollback route restored backup files before running the policy gate, so
  a policy-blocked rollback could leave the on-disk config rolled back while the
  deployed containers were unchanged. Snapshot the current files first and
  revert them when the gate blocks; if that revert itself fails, escalate it on
  the persistent alert feed since the 409 is already sent.
- Refresh container state after a successful manual rollback (rollback
  redeploys), without mis-recording a refetch failure as a rollback failure.
- Suppress the stalled-output warning once live progress is unavailable.

* test: mock snapshotStackFiles in the atomic-deploy rollback route tests

The rollback route now snapshots stack files before restoring a backup, so its
FileSystemService mock needs snapshotStackFiles. Without it the mocked call
threw and the route returned 500, failing the success-path rollback assertions.

backend/src/__tests__/filesystem-backup.test.ts

@@ -228,6 +228,58 @@ describe('FileSystemService backup location', () => {
     await expect(service.restoreStackFiles(stackName)).rejects.toThrow(/Rollback aborted/i);
   });
 
+  it('snapshotStackFiles reverts a restore so a policy-blocked rollback leaves current files in place', async () => {
+    const stackName = 'revert';
+    const stackDir = path.join(composeDir, stackName);
+    await fsPromises.mkdir(stackDir, { recursive: true });
+    // The rollback target (older config) is captured into the backup slot.
+    await fsPromises.writeFile(path.join(stackDir, 'compose.yaml'), 'name: old\n', 'utf-8');
+
+    const service = FileSystemService.getInstance();
+    await service.backupStackFiles(stackName);
+
+    // The current in-use config differs from the backup and adds a managed file
+    // (.env) the backup does not have.
+    await fsPromises.writeFile(path.join(stackDir, 'compose.yaml'), 'name: current\n', 'utf-8');
+    await fsPromises.writeFile(path.join(stackDir, '.env'), 'TOKEN=current\n', 'utf-8');
+
+    // Snapshot current, then restore the backup, mirroring the rollback route.
+    const revert = await service.snapshotStackFiles(stackName);
+    await service.restoreStackFiles(stackName);
+    expect(await fsPromises.readFile(path.join(stackDir, 'compose.yaml'), 'utf-8')).toBe('name: old\n');
+    await expect(fsPromises.access(path.join(stackDir, '.env'))).rejects.toMatchObject({ code: 'ENOENT' });
+
+    // The policy gate blocks the restored target: revert must put the current
+    // files back exactly (content restored, the removed .env recreated).
+    await revert();
+    expect(await fsPromises.readFile(path.join(stackDir, 'compose.yaml'), 'utf-8')).toBe('name: current\n');
+    expect(await fsPromises.readFile(path.join(stackDir, '.env'), 'utf-8')).toBe('TOKEN=current\n');
+  });
+
+  it('snapshotStackFiles revert removes a managed file the snapshot did not have', async () => {
+    const stackName = 'revert-orphan';
+    const stackDir = path.join(composeDir, stackName);
+    await fsPromises.mkdir(stackDir, { recursive: true });
+    // Backup target has compose + .env; current has only compose.
+    await fsPromises.writeFile(path.join(stackDir, 'compose.yaml'), 'name: target\n', 'utf-8');
+    await fsPromises.writeFile(path.join(stackDir, '.env'), 'TOKEN=target\n', 'utf-8');
+
+    const service = FileSystemService.getInstance();
+    await service.backupStackFiles(stackName);
+
+    await fsPromises.rm(path.join(stackDir, '.env'));
+    await fsPromises.writeFile(path.join(stackDir, 'compose.yaml'), 'name: current\n', 'utf-8');
+
+    const revert = await service.snapshotStackFiles(stackName);
+    await service.restoreStackFiles(stackName); // brings back the .env from the backup
+    await expect(fsPromises.access(path.join(stackDir, '.env'))).resolves.toBeUndefined();
+
+    await revert();
+    // The current state had no .env, so revert must remove the restored one.
+    await expect(fsPromises.access(path.join(stackDir, '.env'))).rejects.toMatchObject({ code: 'ENOENT' });
+    expect(await fsPromises.readFile(path.join(stackDir, 'compose.yaml'), 'utf-8')).toBe('name: current\n');
+  });
+
   it('restoreStackFiles leaves non-managed files untouched', async () => {
     const stackName = 'userdata';
     const stackDir = path.join(composeDir, stackName);