fix: keep running containers until stack pull/build succeeds (#1657)

* fix: keep running containers until stack pull/build succeeds

Acquire images before reconcile, capture a recovery generation for
compensation, and only remove classified orphans after handoff.

* fix: address recovery audit blockers for safe stack updates

Retire abandoned and expired recovery artifacts, probe compensated
runtimes before reporting rollback success, preserve local Docker when
deleting a node, validate the exact Compose invocation before capture,
and repair updateStack return-contract fixtures.

* fix: resolve ESLint errors blocking CI on this branch

Unused-import and unused-variable errors left over from the stack
deletion refactor: MeshService in stacks.ts (its opt-out cascade moved
into DeployedStackDeletionService), a redundant pruneVolumes
destructure in deleteDeployedStack (the real one is re-derived from
the same input object inside runDeletionBody), and an unused beforeAll
import in a Docker-integration test stub. Also scopes the webhook
pull-action case body in a block to satisfy no-case-declarations;
purely syntactic, no behavior change.

* fix: harden recovery probe, cleanup retry, and failed-pull Docker test

Reject absent or unhealthy expected replicas before reporting rollback
success, keep cleanup records until artifacts are actually removed, fail
closed when a mesh override cannot be generated, and assert a real
failed pull leaves the original container running.

* fix: verify recovery probe image identity and stack-scoped override paths

Reject recovered runtimes that use the wrong image or leave scale-zero
services running, and confine tombstone override deletion to the intent
stack directory so forged cross-stack paths cannot be swept.

* test: batch notification cap fixtures in a SQLite transaction

Unbatched 1200-row inserts were timing out at the default 30s under
CI load even though the same assertions pass in under 2s when green.
This commit is contained in:
Anso
2026-07-21 12:18:01 -04:00
committed by GitHub
parent b1decbb32a
commit 3f1f15a6f4
41 changed files with 3087 additions and 244 deletions
+12
View File
@@ -12,6 +12,8 @@ import { MonitorService } from '../services/MonitorService';
import { AutoHealService } from '../services/AutoHealService';
import { HealthGateService } from '../services/HealthGateService';
import { ServiceUpdateRecoveryService } from '../services/ServiceUpdateRecoveryService';
import { DeployedStackDeletionService } from '../services/DeployedStackDeletionService';
import { StackUpdateRecoveryService } from '../services/StackUpdateRecoveryService';
import { FleetSyncRetryService } from '../services/FleetSyncRetryService';
import { DockerEventManager } from '../services/DockerEventManager';
import TrivyService, { sweepStaleTrivyTempDirs } from '../services/TrivyService';
@@ -134,6 +136,16 @@ export async function startServer(server: Server): Promise<void> {
// Initialize the license service before any tier-gated code can run.
LicenseService.getInstance().initialize();
// Deletion-intent reconciliation must finish before mutation-capable
// background services or HTTP accept traffic that could recreate a stack
// name still covered by a prepared/ready tombstone.
try {
await DeployedStackDeletionService.getInstance().reconcileAtStartup();
} catch (err) {
console.error('[Startup] Deployed stack deletion reconcile failed:', (err as Error).message);
}
StackUpdateRecoveryService.getInstance().start();
// Synchronous starts: schedule background timers and continue. None of
// these fire their first tick for at least a few seconds, so they
// safely run alongside the async initializers below.