feat(recovery): complete authored-project atomic rollback generations (#1819)

* feat(recovery): capture complete authored Compose project for atomic rollback

Replace the root-compose-only backup slot with staged recovery generations that
record the managed inventory, exact Compose invocation, and prior image identity,
and wire the same engine through deploy, update, manual rollback, and Git apply.

* fix(recovery): satisfy CodeQL path barriers and update-guard mock

Inline resolve+startsWith checks at generation/inventory fs sinks and stub getCurrentStackUpdateRecovery in UpdateGuardService tests.

* fix(recovery): drop unused FileSystemService import in generation store test

* fix(recovery): harden authored-project rollback for upgrade and restore safety

Preserve legacy UUID backup rows, restore Git deploy state with files, make multi-file restore recoverable, evaluate policy on the restored target, and fail closed when Git capture cannot cover an apply.

* fix(recovery): unblock Git apply unit tests and CodeQL pre-restore TOCTOU

Mock recovery capture in git-source-service tests after fail-closed apply capture, and re-resolve live paths immediately before pre-restore snapshot reads.

* fix(recovery): fall back to authored inventory when Git manifesto is missing

First Git apply captures before promote, so a missing managed-project manifesto must not block rollback capture when the live stack already has authored files.

* fix(recovery): make authored-project rollback atomic across Git state

Restore the managed-project manifesto with files, keep nullable Git identity on first-apply captures, persist Git side-state in restore intents for startup reconcile, compensate legacy materialize failures, and refuse directory collisions before mutation.

* fix(recovery): satisfy CodeQL path and TOCTOU barriers on manifesto restore

Add inline resolve barriers for manifesto read/clear sinks and remove the access-then-read race when restoring a generation manifesto snapshot.

* fix(recovery): close third-audit rollback generation blockers

Fail closed on incomplete Git inventory fallbacks, execute captured Compose
invocation during recovery, refuse startup and mutations while restore intents
remain unresolved, propagate legacy stale-delete failures, and add Docker-level
exact prior-image coverage plus regression tests.

* fix(recovery): mark acquired before handoff in prior-image Docker test

Match the production updateStack CAS sequence so the exact prior-image
integration test does not fail handoff from the captured phase.

* fix(recovery): close fourth-audit rollback safety blockers

Evaluate policy against held images, use index-based pre-restore snapshots, hold the shared stack lock across Git apply, replay Mesh and empty captured invocations exactly, restore POSIX modes with fail-closed sensitive permissions, keep case-sensitive paths, and link Git auto-deploy health gates. Add regression coverage for these cases.

* test(recovery): fix mocks for health-gate link and authored compose args

Add linkGateOrRetain to the Git apply recovery mock, and mock authoredComposeArgs so the case-collision inventory test is not masked by a missing getComposeDir stub.

* fix(recovery): close fifth-audit rollback safety blockers

Share git_apply locking for webhook auto-apply, fail closed on malformed recovery service records, refuse mixed-image capture, and require exact probe counts with hold-tag eligibility checks.

* fix(recovery): close sixth-audit rollback safety blockers

Preserve the legacy backup slot during generation capture, encrypt sensitive pre-restore snapshots, revert files on a failed health probe without committing Git, fail closed when an absent-file revert would delete a directory, skip Compose one-offs, route manual and scheduled backup through the current generation, and persist runtime image platform identity.

* fix(recovery): close seventh-audit rollback safety blockers

Fleet snapshot restore and restore-all now capture a recovery generation under the stack lock before any authored file write, including on remote nodes.

* fix(recovery): keep pre-deploy generations during health-gate observe

Link deploy recovery generations to the observing gate so backup cannot replace them mid-observe. Distinguish missing hold tags from probe failures, refuse generation release when services metadata is corrupt, classify mixed-replica and coverage refusals, and toast the backend rollback message.

* fix(recovery): wrap webhook deploy case for eslint

const bindings in an unbraced switch case trip no-case-declarations. Match the pull case block.

backend/src/services/composeProjectContext.ts

@@ -1,9 +1,9 @@
 /**
- * Thin Compose project context for safe full-stack updates.
+ * Shared Compose project context for safe full-stack mutations.
  *
- * Wraps the current authored compose argument path and atomic file backup/restore.
- * When a richer shared Compose project context lands, migrate callers to that type;
- * this module must not become a competing full-manifest resolver.
+ * Resolves the authored inventory (Git managed-project manifest or live stack
+ * discovery), captures/restores generation content through RollbackGenerationStore,
+ * and builds the exact Compose invocation used for deploy/update/rollback.
  */
 import path from 'path';
 import { randomUUID } from 'crypto';
@@ -11,9 +11,18 @@ import { ComposeService } from './ComposeService';
 import { FileSystemService } from './FileSystemService';
 import { buildEffectiveServiceModel } from './effectiveServiceModel';
 import { getErrorMessage } from '../utils/errors';
+import { resolveRollbackInventory } from './rollbackInventory';
+import { RollbackGenerationStore } from './RollbackGenerationStore';
+import type {
+  RollbackGenerationManifest,
+  RollbackOperationKind,
+  RollbackRestoreTransactionMeta,
+} from '../types/rollbackGeneration';
 
 export type ImageReferenceKind = 'moving_tag' | 'digest_pinned' | 'none';
 
+export type BackupOperation = RollbackOperationKind;
+
 const DIGEST_PIN_PATTERN = /@sha256:[a-f0-9]{64}$/i;
 
 export function classifyReferenceKind(declaredImageRef: string | null): ImageReferenceKind {
@@ -34,11 +43,18 @@ export interface ComposeProjectContext {
   readonly nodeId: number;
   readonly stackName: string;
   readonly stackDir: string;
+  /** Generation id used as content-store key and backup_slot_id on the DB row. */
   backupSlotId: string | null;
   toComposeArgs(action: string[]): Promise<string[]>;
   validateForMutation(): Promise<void>;
-  backupFromContext(operation: 'update' | 'deployment'): Promise<string>;
-  restoreFromContext(): Promise<void>;
+  /**
+   * Capture a staged generation. When exactCoverage is required and inventory
+   * refuses it, throws before writing any generation content.
+   */
+  backupFromContext(operation: BackupOperation): Promise<string>;
+  restoreFromContext(
+    transactionMeta?: RollbackRestoreTransactionMeta,
+  ): Promise<RollbackGenerationManifest | void>;
   resolveServiceImageMap(): Promise<Map<string, string | null>>;
 }
 
@@ -60,15 +76,63 @@ class AuthoredComposeProjectContext implements ComposeProjectContext {
     await requireRenderableModel(this.nodeId, this.stackName);
   }
 
-  async backupFromContext(_operation: 'update' | 'deployment'): Promise<string> {
-    await FileSystemService.getInstance(this.nodeId).backupStackFiles(this.stackName);
-    const slotId = randomUUID();
-    this.backupSlotId = slotId;
-    return slotId;
+  async backupFromContext(operation: BackupOperation): Promise<string> {
+    const inventory = await resolveRollbackInventory(this.nodeId, this.stackName);
+    if (!inventory.exactCoverage) {
+      throw Object.assign(
+        new Error(
+          inventory.coverageRefusal
+            || 'Exact authored-project rollback coverage is unavailable for this stack',
+        ),
+        { code: 'ROLLBACK_COVERAGE_UNAVAILABLE' },
+      );
+    }
+
+    // Do not refresh the legacy single-slot backup. Clearing that reused slot
+    // would destroy a pre-migration recovery point if a later capture step fails.
+
+    const generationId = randomUUID();
+    await RollbackGenerationStore.captureGeneration({
+      nodeId: this.nodeId,
+      stackName: this.stackName,
+      generationId,
+      inventory,
+      operationKind: operation,
+    });
+    this.backupSlotId = generationId;
+    return generationId;
   }
 
-  async restoreFromContext(): Promise<void> {
-    await FileSystemService.getInstance(this.nodeId).restoreStackFiles(this.stackName);
+  async restoreFromContext(
+    transactionMeta?: RollbackRestoreTransactionMeta,
+  ): Promise<RollbackGenerationManifest | void> {
+    const generationId = this.backupSlotId;
+    if (!generationId) {
+      // Legacy pre-migration restore: only when no generation id is bound.
+      await FileSystemService.getInstance(this.nodeId).restoreStackFiles(this.stackName);
+      return;
+    }
+
+    const present = await RollbackGenerationStore.verifyGenerationContent(
+      this.nodeId,
+      this.stackName,
+      generationId,
+    );
+    if (!present) {
+      throw Object.assign(
+        new Error('Recovery generation content is missing or incomplete'),
+        { code: 'GENERATION_CONTENT_MISSING' },
+      );
+    }
+
+    const inventory = await resolveRollbackInventory(this.nodeId, this.stackName);
+    return RollbackGenerationStore.restoreGeneration(
+      this.nodeId,
+      this.stackName,
+      generationId,
+      inventory.entries.map((e) => e.relativePath),
+      transactionMeta,
+    );
   }
 
   async resolveServiceImageMap(): Promise<Map<string, string | null>> {
@@ -89,6 +153,17 @@ export async function resolveComposeProjectContext(
   return new AuthoredComposeProjectContext(nodeId, stackName, stackDir);
 }
 
+/** Bind an existing generation id onto a fresh context for restore. */
+export async function resolveComposeProjectContextForGeneration(
+  nodeId: number,
+  stackName: string,
+  generationId: string,
+): Promise<ComposeProjectContext> {
+  const ctx = await resolveComposeProjectContext(nodeId, stackName);
+  ctx.backupSlotId = generationId;
+  return ctx;
+}
+
 export function describeContextError(error: unknown): string {
   return getErrorMessage(error, 'Compose project context failed');
 }