fix: keep running containers until stack pull/build succeeds (#1657)

* fix: keep running containers until stack pull/build succeeds

Acquire images before reconcile, capture a recovery generation for
compensation, and only remove classified orphans after handoff.

* fix: address recovery audit blockers for safe stack updates

Retire abandoned and expired recovery artifacts, probe compensated
runtimes before reporting rollback success, preserve local Docker when
deleting a node, validate the exact Compose invocation before capture,
and repair updateStack return-contract fixtures.

* fix: resolve ESLint errors blocking CI on this branch

Unused-import and unused-variable errors left over from the stack
deletion refactor: MeshService in stacks.ts (its opt-out cascade moved
into DeployedStackDeletionService), a redundant pruneVolumes
destructure in deleteDeployedStack (the real one is re-derived from
the same input object inside runDeletionBody), and an unused beforeAll
import in a Docker-integration test stub. Also scopes the webhook
pull-action case body in a block to satisfy no-case-declarations;
purely syntactic, no behavior change.

* fix: harden recovery probe, cleanup retry, and failed-pull Docker test

Reject absent or unhealthy expected replicas before reporting rollback
success, keep cleanup records until artifacts are actually removed, fail
closed when a mesh override cannot be generated, and assert a real
failed pull leaves the original container running.

* fix: verify recovery probe image identity and stack-scoped override paths

Reject recovered runtimes that use the wrong image or leave scale-zero
services running, and confine tombstone override deletion to the intent
stack directory so forged cross-stack paths cannot be swept.

* test: batch notification cap fixtures in a SQLite transaction

Unbatched 1200-row inserts were timing out at the default 30s under
CI load even though the same assertions pass in under 2s when green.

backend/src/services/StackOpLockService.ts

@@ -1,3 +1,4 @@
+import { DatabaseService } from './DatabaseService';
 /**
  * Tracks in-flight stack lifecycle operations (deploy, down, restart, stop,
  * start, update, rollback, backup) per (nodeId, stackName). A second request to
@@ -9,7 +10,7 @@
  * which matches the lifecycle of any in-flight `docker compose` child process.
  */
 
-export type StackOpAction = 'deploy' | 'down' | 'restart' | 'stop' | 'start' | 'update' | 'rollback' | 'backup';
+export type StackOpAction = 'deploy' | 'down' | 'restart' | 'stop' | 'start' | 'update' | 'rollback' | 'backup' | 'delete';
 
 /**
  * Note returned by a background path that skipped its operation because a manual
@@ -70,6 +71,31 @@ export class StackOpLockService {
     this.locks.delete(this.key(nodeId, stackName));
   }
 
+  /**
+   * Matching-intent continuation for crash-resume of a prepared deletion.
+   * Succeeds only when the persisted intent matches and no other in-memory lock holds.
+   */
+  public tryAcquireDeletionContinuation(args: {
+    intentId: string;
+    nodeId: number;
+    stackName: string;
+  }): AcquireResult {
+    const intent = DatabaseService.getInstance().getDeletionIntentById(args.intentId);
+    if (
+      !intent
+      || intent.status !== 'prepared'
+      || intent.node_id !== args.nodeId
+      || intent.stack_name !== args.stackName
+    ) {
+      const existing = this.locks.get(this.key(args.nodeId, args.stackName));
+      return {
+        acquired: false,
+        existing: existing ?? { action: 'delete', startedAt: Date.now(), user: 'system' },
+      };
+    }
+    return this.tryAcquire(args.nodeId, args.stackName, 'delete', 'system:deletion-continuation');
+  }
+
   /**
    * Acquire the per-(nodeId, stackName) lock for the duration of `fn`, then
    * release it. Returns `{ ran: true, result }` when the lock was free, or