feat: detect stalled stack updates and add in-app recovery actions (#1347)

* feat: detect stalled stack updates and add in-app recovery actions

Add a backend idle-output backstop that stops a deploy/update compose step
that has gone silent (SENCHO_COMPOSE_STALL_TIMEOUT_MS, default 10m), so a
hung image pull surfaces a fast failure instead of spinning indefinitely.

Surface failed, timed-out, and stalled operations with recovery actions on
the stack page: a desktop chip plus popover menu and an inline mobile card
offering retry, restart, roll back (when a backup exists), refresh state,
and copy diagnostics, all gated by deploy permission. The streaming
deploy/update progress modal is now on by default and warns when output
goes quiet. Container state is refreshed after a failed or stalled
operation, and the UI never sits in an indefinite spinner.

* fix: harden rollback against policy-blocked file mutation and refine recovery

Address review findings on the stalled-update recovery work:

- The rollback route restored backup files before running the policy gate, so
  a policy-blocked rollback could leave the on-disk config rolled back while the
  deployed containers were unchanged. Snapshot the current files first and
  revert them when the gate blocks; if that revert itself fails, escalate it on
  the persistent alert feed since the 409 is already sent.
- Refresh container state after a successful manual rollback (rollback
  redeploys), without mis-recording a refetch failure as a rollback failure.
- Suppress the stalled-output warning once live progress is unavailable.

* test: mock snapshotStackFiles in the atomic-deploy rollback route tests

The rollback route now snapshots stack files before restoring a backup, so its
FileSystemService mock needs snapshotStackFiles. Without it the mocked call
threw and the route returned 500, failing the success-path rollback assertions.

backend/src/__tests__/stacks-failure-notifications.test.ts

@@ -12,6 +12,7 @@ import request from 'supertest';
 import jwt from 'jsonwebtoken';
 import { setupTestDb, cleanupTestDb, loginAsTestAdmin, TEST_JWT_SECRET } from './helpers/setupTestDb';
 import { ComposeRollbackError } from '../services/ComposeService';
+import * as policyGate from '../helpers/policyGate';
 
 // ── Hoisted mocks (must come before importing the app) ──────────────────────
 
@@ -26,6 +27,9 @@ const {
   mockIsTrivyAvailable,
   mockGetImageDigest,
   mockRunScanAndPersist,
+  mockGetBackupInfo,
+  mockRestoreStackFiles,
+  mockSnapshotStackFiles,
 } = vi.hoisted(() => ({
   mockDeployStack: vi.fn(),
   mockRunCommand: vi.fn(),
@@ -37,6 +41,9 @@ const {
   mockIsTrivyAvailable: vi.fn(),
   mockGetImageDigest: vi.fn(),
   mockRunScanAndPersist: vi.fn(),
+  mockGetBackupInfo: vi.fn(),
+  mockRestoreStackFiles: vi.fn(),
+  mockSnapshotStackFiles: vi.fn(),
 }));
 
 vi.mock('../services/ComposeService', async () => {
@@ -100,6 +107,9 @@ vi.mock('../services/FileSystemService', () => ({
       getBaseDir: () => '/tmp/compose',
       readComposeFile: vi.fn().mockResolvedValue(''),
       hasComposeFile: vi.fn().mockResolvedValue(true),
+      getBackupInfo: mockGetBackupInfo,
+      restoreStackFiles: mockRestoreStackFiles,
+      snapshotStackFiles: mockSnapshotStackFiles,
     }),
   },
 }));
@@ -145,6 +155,12 @@ beforeEach(() => {
     critical_count: 0,
     high_count: 0,
   });
+  mockGetBackupInfo.mockReset();
+  mockRestoreStackFiles.mockReset();
+  mockSnapshotStackFiles.mockReset();
+  mockGetBackupInfo.mockResolvedValue({ exists: true, timestamp: 1 });
+  mockRestoreStackFiles.mockResolvedValue(undefined);
+  mockSnapshotStackFiles.mockResolvedValue(async () => {});
   dispatchAlertSpy.mockClear();
 });
 
@@ -354,3 +370,63 @@ describe('deploy_failure notification on /update error', () => {
     expect(mockUpdateStack.mock.calls[0][2]).toBe(true);
   });
 });
+
+describe('rollback file-revert safety on a policy-blocked rollback', () => {
+  it('does not deploy and alerts the operator when the post-block file revert fails', async () => {
+    // The restored backup is blocked by policy after files were already restored.
+    const gateSpy = vi
+      .spyOn(policyGate, 'runPolicyGate')
+      .mockImplementation(async (_req, res) => {
+        res.status(409).json({ error: 'Rollback blocked by policy' });
+        return false;
+      });
+    // The revert that should undo the restore itself fails (e.g. EACCES on a
+    // chowned bind mount), leaving disk inconsistent with the deployed stack.
+    mockSnapshotStackFiles.mockResolvedValue(async () => {
+      throw Object.assign(new Error('EACCES: permission denied'), { code: 'EACCES' });
+    });
+    try {
+      const res = await request(app)
+        .post('/api/stacks/myapp/rollback')
+        .set('Cookie', authCookie);
+
+      expect(res.status).toBe(409);
+      // The rollback must not have deployed the blocked target.
+      expect(mockDeployStack).not.toHaveBeenCalled();
+      await new Promise(resolve => setImmediate(resolve));
+      // The revert failure is escalated on the persistent alert feed.
+      expect(dispatchAlertSpy).toHaveBeenCalledWith(
+        'error',
+        'deploy_failure',
+        expect.stringContaining('EACCES'),
+        { stackName: 'myapp', actor: 'testadmin' },
+      );
+    } finally {
+      gateSpy.mockRestore();
+    }
+  });
+
+  it('reverts cleanly and stays quiet when the policy block revert succeeds', async () => {
+    const revert = vi.fn().mockResolvedValue(undefined);
+    const gateSpy = vi
+      .spyOn(policyGate, 'runPolicyGate')
+      .mockImplementation(async (_req, res) => {
+        res.status(409).json({ error: 'Rollback blocked by policy' });
+        return false;
+      });
+    mockSnapshotStackFiles.mockResolvedValue(revert);
+    try {
+      const res = await request(app)
+        .post('/api/stacks/myapp/rollback')
+        .set('Cookie', authCookie);
+
+      expect(res.status).toBe(409);
+      expect(revert).toHaveBeenCalledTimes(1);
+      expect(mockDeployStack).not.toHaveBeenCalled();
+      await new Promise(resolve => setImmediate(resolve));
+      expect(dispatchAlertSpy).not.toHaveBeenCalled();
+    } finally {
+      gateSpy.mockRestore();
+    }
+  });
+});