fix: harden deploy/update concurrency and node-targeting safety (#1390)

* fix: harden deploy/update concurrency and node-targeting safety

Release stabilization for deploy/update operational safety.

Per-stack operation locking is now global. Background lifecycle paths
(scheduler auto stop/down/start/backup/update, webhook execute, Git source
auto-deploy, image auto-update, label bulk actions, fleet snapshot redeploy,
and mesh redeploy) acquire the per-node, per-stack lock through a new
StackOpLockService.runExclusive helper and skip rather than race a manual
deploy/update/rollback/backup on the same stack and node. Skips surface
honestly (a failed scheduled run, a recorded webhook failure, a per-stack
batch result, or a thrown error) instead of a silent no-op.

Update readiness and policy-bypass now run against the node captured when the
dialog opened, not the live active node, so switching nodes while a dialog is
open cannot retarget the update or the bypass retry.

Rollback readiness no longer presents a moving-tag or unpinned image as a ready
image revert. Restoring files does not revert a moving tag, so those stacks
read as partial, and the rollback success message states that the compose and
env files were restored.

* fix: lock blueprint reconcile against manual ops and correct rollback wording

Follow-up to the deploy/update safety hardening, closing two more gaps from a
verification pass.

BlueprintService.deployLocal and withdrawLocal called ComposeService directly,
so blueprint reconciliation could race a manual deploy/update/rollback/backup on
an owned stack. Both now run their compose lifecycle call through
StackOpLockService.runExclusive and skip (recorded as a failed reconcile,
retried on the next cycle) on conflict. The withdraw holds the lock across both
the compose down and the directory delete so neither races a manual operation.

The runtime rollback messages overstated recovery: a rollback restores the
compose and env files and recreates containers, but does not revert an image
behind a moving tag. The auto-rollback deploy-progress output, the recovery
panel and chip, the failure toasts, and the manual rollback route message now
state that the compose and env files were restored, with the matching OpenAPI
example and atomic-deployments doc updated.

* fix: acquire stack lock before blueprint deploy mutates compose and marker files

Local blueprint deploy wrote the compose and marker files and ran the policy
assert before acquiring the per-stack lock; the lock only wrapped the deploy
itself. A reconcile could therefore rewrite an owned stack's files while a
manual deploy/update/rollback/backup was running. The lock now wraps the whole
critical section (create, write compose, write marker, policy assert, deploy),
so on conflict nothing is written and the reconcile records a failed outcome.

Adds a test asserting a deploy under a held lock records failed, writes no
marker file, and leaves the manual lock untouched.

* fix: make remote blueprint apply atomic under the receiving node's stack lock

Remote blueprint deploy wrote the compose and marker files to the target node
via separate HTTP calls and only locked on the final deploy, so the file writes
could race a manual operation on that node. A node's operation lock is
process-local and cannot be held by the hub across HTTP calls, so the locked
create/write/deploy now runs on the receiving node.

The locked critical section is extracted into BlueprintService.applyLocalUnderLock
and exposed via POST /api/blueprints/apply-local. The hub posts the blueprint to
that endpoint in one call; the receiving node runs create + write compose+marker
+ deploy under its own per-stack lock. Older nodes without the route answer 404
and fall back to the legacy multi-call flow. The endpoint is gated by paid tier
and the same per-stack stack:edit and stack:deploy permissions as the
PUT-compose + deploy it bundles, validates the stack name, compose size, and
marker structure, and returns 409 on a lock conflict without writing anything.

Adds tests for the atomic single-call path, the 404 legacy fallback, the 409
lock-conflict mapping, the route validation and permission paths, and the
write-compose-then-marker-then-deploy ordering of the shared locked apply.

* fix(deps): bump undici to 7.28.0 to clear high-severity advisory

The frontend CI npm audit gate (--audit-level=high) failed on a transitive
undici 7.25.0 (a dev-only dependency via jsdom): TLS certificate validation
bypass (GHSA-vmh5-mc38-953g) and cross-user cache information disclosure
(GHSA-pr7r-676h-xcf6). Bumping undici within jsdom's existing ^7.25.0 range to
7.28.0 clears the high-severity advisory and unblocks the frontend job. Lockfile
only; no direct dependency or source change.

backend/src/__tests__/blueprints-remote-deploy.test.ts

@@ -73,7 +73,7 @@ beforeEach(() => {
 afterEach(() => vi.restoreAllMocks());
 
 describe('BlueprintService remote deploy', () => {
-    it('creates the stack, writes compose then marker, then deploys, in order', async () => {
+    it('applies atomically via the remote apply-local endpoint in a single call', async () => {
         const node = seedRemoteNode();
         const bp = seedBlueprint([node.id]);
         const nodeObj = DatabaseService.getInstance().getNode(node.id)!;
@@ -81,32 +81,27 @@ describe('BlueprintService remote deploy', () => {
 
         vi.spyOn(axios, 'get').mockResolvedValue({ status: 200, data: [] }); // hasNameConflict: no stacks
         const putSpy = vi.spyOn(axios, 'put').mockResolvedValue({ status: 200, data: {} });
-        const postSpy = vi.spyOn(axios, 'post')
-            .mockResolvedValueOnce({ status: 201, data: {} }) // create stack
-            .mockResolvedValueOnce({ status: 200, data: {} }); // deploy
+        const postSpy = vi.spyOn(axios, 'post').mockResolvedValue({ status: 200, data: { deployed: true } });
 
         const result = await BlueprintService.getInstance().deployToNode(bpObj, nodeObj);
 
         expect(result.status).toBe('active');
-        expect(postSpy.mock.calls[0][0]).toMatch(/\/api\/stacks$/);
-        expect(putSpy.mock.calls[0][0]).toContain('docker-compose.yml');
-        expect(putSpy.mock.calls[1][0]).toContain('.blueprint.json');
-        expect(postSpy.mock.calls[1][0]).toMatch(/\/deploy$/);
-        // Assert global interleaving across spies, not just per-method order:
-        // create < compose < marker < deploy. (mock.calls indices alone would not
-        // catch the deploy POST firing before the file PUTs.)
-        const [createOrder, deployOrder] = postSpy.mock.invocationCallOrder;
-        const [composeOrder, markerOrder] = putSpy.mock.invocationCallOrder;
-        expect(createOrder).toBeLessThan(composeOrder);
-        expect(composeOrder).toBeLessThan(markerOrder);
-        expect(markerOrder).toBeLessThan(deployOrder);
+        // One atomic call to the remote (create + write + deploy run under the
+        // remote's lock); no separate file PUTs from the hub.
+        expect(postSpy).toHaveBeenCalledTimes(1);
+        expect(postSpy.mock.calls[0][0]).toMatch(/\/api\/blueprints\/apply-local$/);
+        expect(putSpy).not.toHaveBeenCalled();
+        const payload = postSpy.mock.calls[0][1] as { stackName: string; composeContent: string; markerContent: string };
+        expect(payload.stackName).toBe(bpObj.name);
+        expect(typeof payload.composeContent).toBe('string');
+        expect(typeof payload.markerContent).toBe('string');
 
         const dep = DatabaseService.getInstance().getDeployment(bp.id, node.id);
         expect(dep?.status).toBe('active');
         expect(dep?.applied_revision).toBe(bpObj.revision);
     });
 
-    it('treats a 409 on stack create as already-exists and proceeds', async () => {
+    it('falls back to the legacy create/write/deploy flow when the remote lacks apply-local (404)', async () => {
         const node = seedRemoteNode();
         const bp = seedBlueprint([node.id]);
         const nodeObj = DatabaseService.getInstance().getNode(node.id)!;
@@ -115,17 +110,40 @@ describe('BlueprintService remote deploy', () => {
         vi.spyOn(axios, 'get').mockResolvedValue({ status: 200, data: [] });
         const putSpy = vi.spyOn(axios, 'put').mockResolvedValue({ status: 200, data: {} });
         const postSpy = vi.spyOn(axios, 'post')
-            .mockResolvedValueOnce({ status: 409, data: { error: 'already exists' } })
-            .mockResolvedValueOnce({ status: 200, data: {} });
+            .mockResolvedValueOnce({ status: 404, data: {} })   // apply-local missing on older node
+            .mockResolvedValueOnce({ status: 201, data: {} })   // legacy create stack
+            .mockResolvedValueOnce({ status: 200, data: {} });  // legacy deploy
 
         const result = await BlueprintService.getInstance().deployToNode(bpObj, nodeObj);
 
         expect(result.status).toBe('active');
-        expect(putSpy).toHaveBeenCalledTimes(2);
-        expect(postSpy).toHaveBeenCalledTimes(2);
+        expect(postSpy.mock.calls[0][0]).toMatch(/\/api\/blueprints\/apply-local$/);
+        expect(postSpy.mock.calls[1][0]).toMatch(/\/api\/stacks$/);
+        expect(putSpy).toHaveBeenCalledTimes(2); // compose + marker
+        expect(postSpy.mock.calls[2][0]).toMatch(/\/deploy$/);
     });
 
-    it('maps a remote deploy failure to status=failed with the HTTP error', async () => {
+    it('maps a remote apply lock-conflict (409) to status=failed', async () => {
+        const node = seedRemoteNode();
+        const bp = seedBlueprint([node.id]);
+        const nodeObj = DatabaseService.getInstance().getNode(node.id)!;
+        const bpObj = DatabaseService.getInstance().getBlueprint(bp.id)!;
+
+        vi.spyOn(axios, 'get').mockResolvedValue({ status: 200, data: [] });
+        const putSpy = vi.spyOn(axios, 'put').mockResolvedValue({ status: 200, data: {} });
+        vi.spyOn(axios, 'post').mockResolvedValue({
+            status: 409,
+            data: { error: 'web is busy: another operation (update) is already in progress' },
+        });
+
+        const result = await BlueprintService.getInstance().deployToNode(bpObj, nodeObj);
+
+        expect(result.status).toBe('failed');
+        expect(result.error).toContain('already in progress');
+        expect(putSpy).not.toHaveBeenCalled(); // no legacy file writes on conflict
+    });
+
+    it('maps a remote apply failure to status=failed with the HTTP error', async () => {
         const node = seedRemoteNode();
         const bp = seedBlueprint([node.id]);
         const nodeObj = DatabaseService.getInstance().getNode(node.id)!;
@@ -133,9 +151,7 @@ describe('BlueprintService remote deploy', () => {
 
         vi.spyOn(axios, 'get').mockResolvedValue({ status: 200, data: [] });
         vi.spyOn(axios, 'put').mockResolvedValue({ status: 200, data: {} });
-        vi.spyOn(axios, 'post')
-            .mockResolvedValueOnce({ status: 201, data: {} })
-            .mockResolvedValueOnce({ status: 500, data: { error: 'boom' } });
+        vi.spyOn(axios, 'post').mockResolvedValue({ status: 500, data: { error: 'boom' } });
 
         const result = await BlueprintService.getInstance().deployToNode(bpObj, nodeObj);
 

backend/src/__tests__/blueprints-route-validation.test.ts

@@ -7,6 +7,7 @@ let tmpDir: string;
 let app: import('express').Express;
 let DatabaseService: typeof import('../services/DatabaseService').DatabaseService;
 let LicenseService: typeof import('../services/LicenseService').LicenseService;
+let StackOpLockService: typeof import('../services/StackOpLockService').StackOpLockService;
 let adminCookie: string;
 let counter = 0;
 
@@ -14,6 +15,7 @@ beforeAll(async () => {
     tmpDir = await setupTestDb();
     ({ DatabaseService } = await import('../services/DatabaseService'));
     ({ LicenseService } = await import('../services/LicenseService'));
+    ({ StackOpLockService } = await import('../services/StackOpLockService'));
 
     vi.spyOn(LicenseService.getInstance(), 'getTier').mockReturnValue('paid');
 
@@ -26,6 +28,7 @@ afterAll(() => cleanupTestDb(tmpDir));
 beforeEach(() => {
     vi.restoreAllMocks();
     vi.spyOn(LicenseService.getInstance(), 'getTier').mockReturnValue('paid');
+    StackOpLockService.resetForTests();
     const db = DatabaseService.getInstance().getDb();
     db.prepare('DELETE FROM blueprint_deployments').run();
     db.prepare('DELETE FROM blueprints').run();
@@ -80,3 +83,49 @@ describe('Blueprint route compose validation', () => {
         expect(res.body.error).toContain(`${MAX_BLUEPRINT_COMPOSE_BYTES} bytes or fewer`);
     });
 });
+
+describe('POST /api/blueprints/apply-local (node-to-node atomic apply)', () => {
+    it('rejects an invalid stack name', async () => {
+        const res = await request(app)
+            .post('/api/blueprints/apply-local')
+            .set('Cookie', adminCookie)
+            .send({ stackName: '../escape', composeContent: 'services: {}', markerContent: '{}' });
+        expect(res.status).toBe(400);
+        expect(res.body.error).toContain('Invalid stack name');
+    });
+
+    it('rejects a missing compose/marker payload', async () => {
+        const res = await request(app)
+            .post('/api/blueprints/apply-local')
+            .set('Cookie', adminCookie)
+            .send({ stackName: 'apply-local-stack' });
+        expect(res.status).toBe(400);
+    });
+
+    it('rejects a structurally invalid marker', async () => {
+        const res = await request(app)
+            .post('/api/blueprints/apply-local')
+            .set('Cookie', adminCookie)
+            .send({ stackName: 'apply-local-stack', composeContent: 'services:\n  app:\n    image: nginx\n', markerContent: '{}' });
+        expect(res.status).toBe(400);
+        expect(res.body.error).toContain('marker');
+    });
+
+    it('returns 409 without deploying when the per-stack lock is held', async () => {
+        // The local node (id 1) holds the lock for this stack; the apply must be
+        // rejected before it can create or write any files.
+        StackOpLockService.getInstance().tryAcquire(1, 'apply-local-busy', 'update', 'admin');
+        const res = await request(app)
+            .post('/api/blueprints/apply-local')
+            .set('Cookie', adminCookie)
+            .send({
+                stackName: 'apply-local-busy',
+                composeContent: 'services:\n  app:\n    image: nginx\n',
+                markerContent: JSON.stringify({ blueprintId: 1, revision: 1, lastApplied: 123 }),
+            });
+        expect(res.status).toBe(409);
+        expect(res.body.code).toBe('stack_op_in_progress');
+        // The manual op still owns the lock; the apply never acquired it.
+        expect(StackOpLockService.getInstance().get(1, 'apply-local-busy')?.action).toBe('update');
+    });
+});

backend/src/__tests__/blueprints.test.ts

@@ -24,6 +24,7 @@ let DatabaseService: typeof import('../services/DatabaseService').DatabaseServic
 let BlueprintReconciler: typeof import('../services/BlueprintReconciler').BlueprintReconciler;
 let NodeLabelService: typeof import('../services/NodeLabelService').NodeLabelService;
 let BlueprintService: typeof import('../services/BlueprintService').BlueprintService;
+let StackOpLockService: typeof import('../services/StackOpLockService').StackOpLockService;
 let counter = 0;
 
 beforeAll(async () => {
@@ -32,6 +33,7 @@ beforeAll(async () => {
     ({ BlueprintReconciler } = await import('../services/BlueprintReconciler'));
     ({ NodeLabelService } = await import('../services/NodeLabelService'));
     ({ BlueprintService } = await import('../services/BlueprintService'));
+    ({ StackOpLockService } = await import('../services/StackOpLockService'));
 });
 
 afterAll(() => cleanupTestDb(tmpDir));
@@ -43,6 +45,7 @@ beforeEach(() => {
     db.prepare('DELETE FROM node_labels').run();
     db.prepare("DELETE FROM nodes WHERE is_default = 0").run();
     db.prepare("UPDATE global_settings SET value = '0' WHERE key = 'developer_mode'").run();
+    StackOpLockService.resetForTests();
     vi.restoreAllMocks();
 });
 
@@ -393,6 +396,67 @@ describe('BlueprintReconciler developer-mode diagnostics', () => {
     });
 });
 
+describe('BlueprintService per-stack lock', () => {
+    it('deploy under a free lock writes compose then marker, then deploys', async () => {
+        const nodeId = seedNode();
+        const bp = seedBlueprint({ classification: 'stateless', nodeIds: [nodeId] });
+        const node = DatabaseService.getInstance().getNode(nodeId)!;
+        const { FileSystemService } = await import('../services/FileSystemService');
+        const { ComposeService } = await import('../services/ComposeService');
+        // Spy the file/deploy primitives so the locked critical section runs
+        // without touching the real filesystem or Docker.
+        vi.spyOn(FileSystemService.prototype, 'createStack').mockResolvedValue(undefined);
+        const writeSpy = vi.spyOn(FileSystemService.prototype, 'writeStackFile').mockResolvedValue(undefined);
+        const deploySpy = vi.spyOn(ComposeService.prototype, 'deployStack').mockResolvedValue(undefined);
+
+        const outcome = await BlueprintService.getInstance().deployToNode(bp, node);
+
+        expect(outcome.status).toBe('active');
+        expect(deploySpy).toHaveBeenCalledWith(bp.name, undefined, false);
+        // Compose is written first, then the marker, both before the deploy.
+        expect(writeSpy).toHaveBeenCalledTimes(2);
+        expect(writeSpy.mock.calls[0][2]).toBe(bp.compose_content);
+        expect(writeSpy.mock.calls[1][2]).toContain('"blueprintId"');
+        const [composeOrder, markerOrder] = writeSpy.mock.invocationCallOrder;
+        const [deployOrder] = deploySpy.mock.invocationCallOrder;
+        expect(composeOrder).toBeLessThan(markerOrder);
+        expect(markerOrder).toBeLessThan(deployOrder);
+    });
+
+    it('deploy skips, writes no stack files, and records failed when the stack lock is held', async () => {
+        const nodeId = seedNode();
+        const bp = seedBlueprint({ classification: 'stateless', nodeIds: [nodeId] });
+        const node = DatabaseService.getInstance().getNode(nodeId)!;
+        // A manual operation holds the lock; the reconcile deploy must not race
+        // it, and must not mutate compose/marker files before owning the lock.
+        StackOpLockService.getInstance().tryAcquire(nodeId, bp.name, 'update', 'admin');
+
+        const outcome = await BlueprintService.getInstance().deployToNode(bp, node);
+
+        expect(outcome.status).toBe('failed');
+        expect(outcome.error).toContain('already in progress');
+        // No marker file was written (the lock guards the file writes too).
+        expect(await BlueprintService.getInstance().readMarker(bp.name, node)).toBeNull();
+        // The manual op still holds the lock; the deploy never acquired it.
+        expect(StackOpLockService.getInstance().get(nodeId, bp.name)?.action).toBe('update');
+    });
+
+    it('withdraw skips and records failed when a manual operation holds the stack lock', async () => {
+        const nodeId = seedNode();
+        const bp = seedBlueprint({ classification: 'stateless', nodeIds: [nodeId] });
+        const node = DatabaseService.getInstance().getNode(nodeId)!;
+        // A manual operation holds the lock; the withdraw must not race it.
+        StackOpLockService.getInstance().tryAcquire(nodeId, bp.name, 'update', 'admin');
+
+        const outcome = await BlueprintService.getInstance().withdrawFromNode(bp, node);
+
+        expect(outcome.status).toBe('failed');
+        expect(outcome.error).toContain('already in progress');
+        // The lock is still held by the manual op (the withdraw never acquired it).
+        expect(StackOpLockService.getInstance().get(nodeId, bp.name)?.action).toBe('update');
+    });
+});
+
 describe('BlueprintService marker parsing + name-conflict guard', () => {
     it('parseMarker accepts a well-formed marker', () => {
         const marker = BlueprintService.parseMarker(JSON.stringify({ blueprintId: 7, revision: 3, lastApplied: 12345 }));

backend/src/__tests__/labels-bulk-actions.test.ts

@@ -49,6 +49,7 @@ let authHeader: string;
 let db: import('../services/DatabaseService').DatabaseService;
 let LicenseService: typeof import('../services/LicenseService').LicenseService;
 let activeBulkActions: typeof import('../routes/labels').activeBulkActions;
+let StackOpLockService: typeof import('../services/StackOpLockService').StackOpLockService;
 let labelCounter = 0;
 
 beforeAll(async () => {
@@ -56,6 +57,7 @@ beforeAll(async () => {
   ({ app } = await import('../index'));
   ({ LicenseService } = await import('../services/LicenseService'));
   ({ activeBulkActions } = await import('../routes/labels'));
+  ({ StackOpLockService } = await import('../services/StackOpLockService'));
   const { DatabaseService } = await import('../services/DatabaseService');
   db = DatabaseService.getInstance();
   authHeader = `Bearer ${jwt.sign({ username: TEST_USERNAME }, TEST_JWT_SECRET, { expiresIn: '1m' })}`;
@@ -77,6 +79,7 @@ beforeEach(() => {
   restartContainer.mockResolvedValue(undefined);
   enforcePolicyPreDeploy.mockResolvedValue({ ok: true });
   activeBulkActions.clear();
+  StackOpLockService.resetForTests();
   db.getDb().prepare('DELETE FROM stack_label_assignments').run();
   db.getDb().prepare('DELETE FROM stack_labels').run();
 });
@@ -150,6 +153,27 @@ describe('Stack Labels bulk actions', () => {
     expect(restartContainer).not.toHaveBeenCalled();
   });
 
+  it('skips a stack whose per-stack lock is held by a manual operation', async () => {
+    const label = await createAssignedLabel(['alpha', 'beta']);
+    // A manual operation holds 'alpha'; the bulk deploy must not race it.
+    StackOpLockService.getInstance().tryAcquire(label.node_id, 'alpha', 'update', 'admin');
+
+    const res = await request(app)
+      .post(`/api/labels/${label.id}/action`)
+      .set('Authorization', authHeader)
+      .send({ action: 'deploy' });
+
+    expect(res.status).toBe(200);
+    const alpha = res.body.results.find((r: { stackName: string }) => r.stackName === 'alpha');
+    const beta = res.body.results.find((r: { stackName: string }) => r.stackName === 'beta');
+    expect(alpha).toMatchObject({ stackName: 'alpha', success: false });
+    expect(alpha.error).toContain('another operation (update) is already in progress');
+    expect(beta).toEqual({ stackName: 'beta', success: true });
+    // 'alpha' was skipped; only 'beta' reached ComposeService.
+    expect(deployStack).toHaveBeenCalledTimes(1);
+    expect(deployStack).toHaveBeenCalledWith('beta', undefined, false);
+  });
+
   it('dry-run deploy runs the policy gate and reports blocked stacks honestly', async () => {
     const label = await createAssignedLabel(['alpha']);
     enforcePolicyPreDeploy.mockResolvedValue({

backend/src/__tests__/scheduler-service.test.ts

@@ -195,9 +195,13 @@ vi.mock('../services/PolicyEnforcement', () => ({
 }));
 
 import { SchedulerService } from '../services/SchedulerService';
+import { StackOpLockService } from '../services/StackOpLockService';
 
 beforeEach(() => {
   vi.clearAllMocks();
+  // Lifecycle/update handlers run through the real StackOpLockService; reset it
+  // so a lock left held by one test cannot make a later test skip its op.
+  StackOpLockService.resetForTests();
   // clearAllMocks only clears call history, not implementations, so restore the
   // mocks that individual tests mutate (tier, node lookup, proxy target) to
   // their documented defaults. Without this a test that points getNode at a
@@ -1643,6 +1647,17 @@ describe('SchedulerService - lifecycle actions', () => {
     expect(mockUpdateScheduledTaskRun).toHaveBeenCalledWith(1, expect.objectContaining({ status: 'failure' }));
   });
 
+  it('records failure (not success) and skips the op when the stack lock is held', async () => {
+    // A manual operation holds the lock; the scheduled lifecycle op must skip
+    // rather than race, and surface as a failed run instead of a silent success.
+    StackOpLockService.getInstance().tryAcquire(1, 'my-stack', 'deploy', 'admin');
+    mockGetScheduledTask.mockReturnValue(makeLifecycleTask('auto_stop'));
+    await SchedulerService.getInstance().triggerTask(300);
+    expect(mockRunCommand).not.toHaveBeenCalled();
+    expect(mockUpdateScheduledTaskRun).toHaveBeenCalledWith(1, expect.objectContaining({ status: 'failure' }));
+    expect(mockUpdateScheduledTask).toHaveBeenCalledWith(300, expect.objectContaining({ last_status: 'failure' }));
+  });
+
   it('paid tier executes lifecycle actions', async () => {
     mockGetTier.mockReturnValue('paid');
     mockGetScheduledTask.mockReturnValue(makeLifecycleTask('auto_stop'));

backend/src/__tests__/stack-op-lock-service.test.ts

@@ -6,7 +6,7 @@
  * `resetForTests()` so state doesn't leak between cases.
  */
 import { describe, it, expect, beforeEach } from 'vitest';
-import { StackOpLockService } from '../services/StackOpLockService';
+import { StackOpLockService, stackOpSkipMessage } from '../services/StackOpLockService';
 
 beforeEach(() => {
   StackOpLockService.resetForTests();
@@ -94,3 +94,56 @@ describe('StackOpLockService', () => {
     expect(lock!.startedAt).toBeLessThanOrEqual(Date.now());
   });
 });
+
+describe('StackOpLockService.runExclusive', () => {
+  it('runs fn and releases the lock when the slot is free', async () => {
+    const svc = StackOpLockService.getInstance();
+    const outcome = await svc.runExclusive(1, 'web', 'deploy', 'system', async () => 'done');
+    expect(outcome).toEqual({ ran: true, result: 'done' });
+    // Released after fn resolves, so a later op acquires.
+    expect(svc.size()).toBe(0);
+  });
+
+  it('holds the lock for the duration of fn, blocking a concurrent acquire', async () => {
+    const svc = StackOpLockService.getInstance();
+    let observed: ReturnType<typeof svc.tryAcquire> | null = null;
+    const outcome = await svc.runExclusive(1, 'web', 'update', 'system', async () => {
+      // A manual op attempting to acquire mid-operation must be rejected.
+      observed = svc.tryAcquire(1, 'web', 'deploy', 'admin');
+      return 42;
+    });
+    expect(outcome).toEqual({ ran: true, result: 42 });
+    expect(observed!.acquired).toBe(false);
+  });
+
+  it('skips (ran=false) and returns the holder when the lock is already held', async () => {
+    const svc = StackOpLockService.getInstance();
+    svc.tryAcquire(1, 'web', 'rollback', 'admin');
+    let called = false;
+    const outcome = await svc.runExclusive(1, 'web', 'deploy', 'system', async () => {
+      called = true;
+      return 'should not run';
+    });
+    expect(called).toBe(false);
+    expect(outcome.ran).toBe(false);
+    if (!outcome.ran) expect(outcome.existing.action).toBe('rollback');
+  });
+
+  it('releases the lock even when fn throws, then propagates', async () => {
+    const svc = StackOpLockService.getInstance();
+    await expect(
+      svc.runExclusive(1, 'web', 'deploy', 'system', async () => {
+        throw new Error('boom');
+      }),
+    ).rejects.toThrow('boom');
+    expect(svc.size()).toBe(0);
+  });
+});
+
+describe('stackOpSkipMessage', () => {
+  it('names the stack and the conflicting action', () => {
+    expect(stackOpSkipMessage('web', 'update')).toBe(
+      'Skipped "web": another operation (update) is already in progress.',
+    );
+  });
+});

backend/src/__tests__/update-guard-rollback.test.ts

@@ -30,7 +30,7 @@ const baseInputs = (over: Partial<RollbackInputs> = {}): RollbackInputs => ({
   backup: { exists: true, timestamp: NOW - 3_600_000 },
   envSummary: { exists: true, envPresent: true, keys: ['DB_HOST', 'DB_PASS'] },
   stackHasEnv: true,
-  rollbackTarget: { target: 'nginx:1.27.1' },
+  rollbackTarget: { target: 'nginx:1.27.1', moving: false },
   lastDeployAt: NOW - 3_600_000,
   containers: [{
     name: 'app-web-1', state: 'running', health: 'healthy', exitCode: null,
@@ -89,15 +89,22 @@ describe('buildRollbackItems', () => {
   });
 
   it('marks the previous image unknown when no rollback target is known', () => {
-    expect(itemById(baseInputs({ rollbackTarget: { target: null } }), 'previous_images').state).toBe('unknown');
+    expect(itemById(baseInputs({ rollbackTarget: { target: null, moving: false } }), 'previous_images').state).toBe('unknown');
     expect(itemById(baseInputs({ rollbackTarget: 'error' }), 'previous_images').state).toBe('unknown');
     const known = itemById(baseInputs(), 'previous_images');
     expect(known.state).toBe('ready');
     expect(known.detail).toContain('nginx:1.27.1');
   });
 
+  it('downgrades the previous image to not_covered for a moving tag', () => {
+    const item = itemById(baseInputs({ rollbackTarget: { target: 'nginx:latest', moving: true } }), 'previous_images');
+    expect(item.state).toBe('not_covered');
+    expect(item.detail).toContain('moving image tag');
+    expect(item.detail).toContain('nginx:latest');
+  });
+
   it('does not mistake an image literally named error for a failed preview', () => {
-    const item = itemById(baseInputs({ rollbackTarget: { target: 'error' } }), 'previous_images');
+    const item = itemById(baseInputs({ rollbackTarget: { target: 'error', moving: false } }), 'previous_images');
     expect(item.state).toBe('ready');
     expect(item.detail).toContain('error');
   });
@@ -128,7 +135,12 @@ describe('aggregateRollbackOverall', () => {
   });
 
   it('is partial when the previous image tag is unknown', () => {
-    const items = buildRollbackItems(baseInputs({ rollbackTarget: { target: null } }), NOW);
+    const items = buildRollbackItems(baseInputs({ rollbackTarget: { target: null, moving: false } }), NOW);
+    expect(aggregateRollbackOverall(items)).toBe('partial');
+  });
+
+  it('is partial when the rollback target is a moving tag', () => {
+    const items = buildRollbackItems(baseInputs({ rollbackTarget: { target: 'nginx:latest', moving: true } }), NOW);
     expect(aggregateRollbackOverall(items)).toBe('partial');
   });
 

backend/src/__tests__/update-guard-service.test.ts

@@ -11,6 +11,8 @@ const {
   mockGetLatest,
   mockGetPreview,
   mockGetBackupInfo,
+  mockGetBackupEnvSummary,
+  mockEnvExists,
   mockGetOpenDriftFindings,
   mockGetGlobalSettings,
   mockFsSize,
@@ -20,6 +22,8 @@ const {
   mockGetLatest: vi.fn(),
   mockGetPreview: vi.fn(),
   mockGetBackupInfo: vi.fn(),
+  mockGetBackupEnvSummary: vi.fn(),
+  mockEnvExists: vi.fn(),
   mockGetOpenDriftFindings: vi.fn(),
   mockGetGlobalSettings: vi.fn(),
   mockFsSize: vi.fn(),
@@ -40,7 +44,10 @@ vi.mock('../services/ComposeDoctorService', () => ({
   ComposeDoctorService: { getInstance: () => ({ getLatest: mockGetLatest }) },
 }));
 
-vi.mock('../services/UpdatePreviewService', () => ({
+vi.mock('../services/UpdatePreviewService', async (importOriginal) => ({
+  // Keep the real pure helpers (isMovingTag, parseSemverTag) that
+  // UpdateGuardService imports; only stub the service singleton.
+  ...(await importOriginal<typeof import('../services/UpdatePreviewService')>()),
   UpdatePreviewService: { getInstance: () => ({ getPreview: mockGetPreview }) },
 }));
 
@@ -48,8 +55,8 @@ vi.mock('../services/FileSystemService', () => ({
   FileSystemService: {
     getInstance: () => ({
       getBackupInfo: mockGetBackupInfo,
-      getBackupEnvSummary: vi.fn().mockRejectedValue(new Error('not used here')),
-      envExists: vi.fn().mockRejectedValue(new Error('not used here')),
+      getBackupEnvSummary: mockGetBackupEnvSummary,
+      envExists: mockEnvExists,
     }),
   },
 }));
@@ -81,6 +88,9 @@ const inspectResult = (over: Record<string, unknown> = {}) => ({
 beforeEach(() => {
   vi.clearAllMocks();
   mockGetGlobalSettings.mockReturnValue({ host_disk_limit: '90' });
+  // Sensible defaults for the rollback-readiness inputs (only computeRollbackReadiness reads these).
+  mockGetBackupEnvSummary.mockResolvedValue({ exists: true, envPresent: true, keys: ['DB_HOST'] });
+  mockEnvExists.mockResolvedValue(true);
 });
 
 describe('UpdateGuardService.probeContainers', () => {
@@ -153,3 +163,36 @@ describe('UpdateGuardService.computeUpdateReadiness wiring', () => {
     expect(report.verdict).toBe('ready');
   });
 });
+
+describe('UpdateGuardService.computeRollbackReadiness moving-tag wiring', () => {
+  const preview = (images: Array<{ current_tag: string }>) => ({
+    stack_name: 'app',
+    images,
+    summary: {
+      has_update: false, primary_image: 'app', current_tag: images[0]?.current_tag ?? null,
+      next_tag: null, semver_bump: 'none', update_kind: 'none', blocked: false, blocked_reason: null,
+    },
+    rollback_target: 'app:1.2.3',
+    changelog: null,
+  });
+
+  beforeEach(() => {
+    mockGetBackupInfo.mockResolvedValue({ exists: true, timestamp: Date.now() });
+    mockListContainers.mockResolvedValue([]);
+  });
+
+  it('marks previous_images not_covered (overall partial) when any image uses a moving tag', async () => {
+    // Primary pinned, sidecar on a moving tag: a file rollback cannot revert it.
+    mockGetPreview.mockResolvedValue(preview([{ current_tag: '1.2.3' }, { current_tag: 'latest' }]));
+    const report = await UpdateGuardService.getInstance().computeRollbackReadiness(0, 'app');
+    expect(report.items.find(i => i.id === 'previous_images')?.state).toBe('not_covered');
+    expect(report.overall).toBe('partial');
+  });
+
+  it('marks previous_images ready (overall ready) when every image is pinned', async () => {
+    mockGetPreview.mockResolvedValue(preview([{ current_tag: '1.2.3' }, { current_tag: 'v2.0.1' }]));
+    const report = await UpdateGuardService.getInstance().computeRollbackReadiness(0, 'app');
+    expect(report.items.find(i => i.id === 'previous_images')?.state).toBe('ready');
+    expect(report.overall).toBe('ready');
+  });
+});

backend/src/__tests__/update-preview-service.test.ts

@@ -5,6 +5,7 @@ import {
     computeSemverBump,
     computeImagePreview,
     buildSummary,
+    isMovingTag,
     type ComputePreviewDeps,
 } from '../services/UpdatePreviewService';
 
@@ -25,6 +26,21 @@ describe('parseSemverTag', () => {
     });
 });
 
+describe('isMovingTag', () => {
+    it('treats fully-pinned semver as immutable', () => {
+        expect(isMovingTag('1.2.3')).toBe(false);
+        expect(isMovingTag('v1.2.3')).toBe(false);
+        expect(isMovingTag('27.1.4-alpine')).toBe(false);
+    });
+    it('treats latest, branches, and unpinned major/minor as moving', () => {
+        expect(isMovingTag('latest')).toBe(true);
+        expect(isMovingTag('main')).toBe(true);
+        expect(isMovingTag('stable')).toBe(true);
+        expect(isMovingTag('1.25')).toBe(true);
+        expect(isMovingTag('unknown')).toBe(true);
+    });
+});
+
 describe('findNextTag', () => {
     it('picks highest semver greater than current', () => {
         const tags = ['27.1.3', '27.1.4', '27.1.5', '27.2.0', '27.1.5-alpine'];