fix: keep running containers until stack pull/build succeeds (#1657)

* fix: keep running containers until stack pull/build succeeds

Acquire images before reconcile, capture a recovery generation for
compensation, and only remove classified orphans after handoff.

* fix: address recovery audit blockers for safe stack updates

Retire abandoned and expired recovery artifacts, probe compensated
runtimes before reporting rollback success, preserve local Docker when
deleting a node, validate the exact Compose invocation before capture,
and repair updateStack return-contract fixtures.

* fix: resolve ESLint errors blocking CI on this branch

Unused-import and unused-variable errors left over from the stack
deletion refactor: MeshService in stacks.ts (its opt-out cascade moved
into DeployedStackDeletionService), a redundant pruneVolumes
destructure in deleteDeployedStack (the real one is re-derived from
the same input object inside runDeletionBody), and an unused beforeAll
import in a Docker-integration test stub. Also scopes the webhook
pull-action case body in a block to satisfy no-case-declarations;
purely syntactic, no behavior change.

* fix: harden recovery probe, cleanup retry, and failed-pull Docker test

Reject absent or unhealthy expected replicas before reporting rollback
success, keep cleanup records until artifacts are actually removed, fail
closed when a mesh override cannot be generated, and assert a real
failed pull leaves the original container running.

* fix: verify recovery probe image identity and stack-scoped override paths

Reject recovered runtimes that use the wrong image or leave scale-zero
services running, and confine tombstone override deletion to the intent
stack directory so forged cross-stack paths cannot be swept.

* test: batch notification cap fixtures in a SQLite transaction

Unbatched 1200-row inserts were timing out at the default 30s under
CI load even though the same assertions pass in under 2s when green.

backend/src/services/DockerController.ts

@@ -24,6 +24,7 @@ import {
 import type { NetworkingNetworkBase } from './network/networkingTypes';
 import { isPathWithinBase } from '../utils/validation';
 import { isDebugEnabled } from '../utils/debug';
+import { getErrorMessage } from '../utils/errors';
 import { sanitizeForLog } from '../utils/safeLog';
 import { describeSpawnError } from '../utils/spawnErrors';
 import { authoredComposeFileArgs, authoredComposeEnvFileArgs } from '../utils/authoredComposeArgs';
@@ -2258,6 +2259,57 @@ class DockerController {
     }
   }
 
+  /**
+   * Strict Result API for full-stack updates. Never converts Compose-ps + fallback
+   * failure into empty success. Compose-managed containers are never orphan IDs.
+   */
+  public async classifyLegacyOrphansForUpdate(
+    stackName: string,
+  ): Promise<
+    | { status: 'none' }
+    | { status: 'orphans'; ids: string[] }
+    | { status: 'classification_failed'; error: string }
+  > {
+    const stackDir = path.join(NodeRegistry.getInstance().getComposeDir(this.nodeId), stackName);
+    const toIds = (list: Array<{ Id?: string }>) =>
+      list.filter((c): c is { Id: string } => typeof c.Id === 'string' && c.Id.length > 0)
+        .map((c) => c.Id);
+
+    const fallbackOrphans = async (): Promise<
+      | { status: 'none' }
+      | { status: 'orphans'; ids: string[] }
+      | { status: 'classification_failed'; error: string }
+    > => {
+      try {
+        const ids = toIds(await this.smartFallback(stackName, stackDir));
+        return ids.length === 0 ? { status: 'none' } : { status: 'orphans', ids };
+      } catch (fallbackError) {
+        return {
+          status: 'classification_failed',
+          error: getErrorMessage(fallbackError, 'Legacy orphan classification failed'),
+        };
+      }
+    };
+
+    try {
+      const composeContainers = await this.fetchComposePsContainers(stackName, stackDir);
+      // Compose already manages this stack: no legacy orphan cleanup (same as deploy).
+      if (composeContainers.length > 0) return { status: 'none' };
+      return await fallbackOrphans();
+    } catch (error) {
+      const execError = error as NodeJS.ErrnoException & { stderr?: string };
+      const mapped = describeSpawnError(execError, { command: 'docker compose ps' });
+      const detail = execError.stderr || mapped.message || getErrorMessage(error, 'docker compose ps failed');
+      console.error('Docker Compose Error for %s:', sanitizeForLog(stackName), sanitizeForLog(detail));
+      // Unlike getLegacyOrphanContainersByStack, never convert dual failure into empty success.
+      const fallback = await fallbackOrphans();
+      if (fallback.status === 'classification_failed') {
+        return { status: 'classification_failed', error: String(detail) };
+      }
+      return fallback;
+    }
+  }
+
   public async getContainersByStack(stackName: string) {
     // Resolve the compose dir and the authored prefix for THIS controller's node,
     // not the process default, so a non-default local node sees its own stack dir