fix: keep running containers until stack pull/build succeeds (#1657)

* fix: keep running containers until stack pull/build succeeds

Acquire images before reconcile, capture a recovery generation for
compensation, and only remove classified orphans after handoff.

* fix: address recovery audit blockers for safe stack updates

Retire abandoned and expired recovery artifacts, probe compensated
runtimes before reporting rollback success, preserve local Docker when
deleting a node, validate the exact Compose invocation before capture,
and repair updateStack return-contract fixtures.

* fix: resolve ESLint errors blocking CI on this branch

Unused-import and unused-variable errors left over from the stack
deletion refactor: MeshService in stacks.ts (its opt-out cascade moved
into DeployedStackDeletionService), a redundant pruneVolumes
destructure in deleteDeployedStack (the real one is re-derived from
the same input object inside runDeletionBody), and an unused beforeAll
import in a Docker-integration test stub. Also scopes the webhook
pull-action case body in a block to satisfy no-case-declarations;
purely syntactic, no behavior change.

* fix: harden recovery probe, cleanup retry, and failed-pull Docker test

Reject absent or unhealthy expected replicas before reporting rollback
success, keep cleanup records until artifacts are actually removed, fail
closed when a mesh override cannot be generated, and assert a real
failed pull leaves the original container running.

* fix: verify recovery probe image identity and stack-scoped override paths

Reject recovered runtimes that use the wrong image or leave scale-zero
services running, and confine tombstone override deletion to the intent
stack directory so forged cross-stack paths cannot be swept.

* test: batch notification cap fixtures in a SQLite transaction

Unbatched 1200-row inserts were timing out at the default 30s under
CI load even though the same assertions pass in under 2s when green.
This commit is contained in:
Anso
2026-07-21 12:18:01 -04:00
committed by GitHub
parent b1decbb32a
commit 3f1f15a6f4
41 changed files with 3087 additions and 244 deletions
+9 -1
View File
@@ -7,6 +7,7 @@ import { requirePermission } from '../middleware/permissions';
import { rejectApiTokenScope } from '../middleware/apiTokenScope';
import { requireAdmin, requirePaid } from '../middleware/tierGates';
import { enrollmentLimiter } from '../middleware/rateLimiters';
import { DeployedStackDeletionService } from '../services/DeployedStackDeletionService';
import { DatabaseService } from '../services/DatabaseService';
import { NodeRegistry } from '../services/NodeRegistry';
import { CacheService } from '../services/CacheService';
@@ -401,7 +402,14 @@ nodesRouter.delete('/:id', async (req: Request, res: Response) => {
// local node, or one with no active connection). Mirrors the re-enroll path.
MeshProxyTunnelDialer.getInstance().closeBridge(id, 'node deleted');
PilotTunnelManager.getInstance().closeTunnel(id, PilotCloseCode.NormalClosure, 'node deleted');
DatabaseService.getInstance().deleteNode(id);
// Local-socket nodes: ready tombstone + recovery-row retirement in the same
// transaction as the node delete, then sweep tags/paths. Remote hub records
// create no Docker cleanup tombstone.
if (existing.type === 'local') {
await DeployedStackDeletionService.getInstance().deleteLocalNode(id);
} else {
DatabaseService.getInstance().deleteNode(id);
}
NodeRegistry.getInstance().evictConnection(id);
NodeRegistry.getInstance().notifyNodeRemoved(id);
CacheService.getInstance().invalidate(`${REMOTE_META_NAMESPACE}:${id}`);