fix: harden deploy/update concurrency and node-targeting safety (#1390)

* fix: harden deploy/update concurrency and node-targeting safety

Release stabilization for deploy/update operational safety.

Per-stack operation locking is now global. Background lifecycle paths
(scheduler auto stop/down/start/backup/update, webhook execute, Git source
auto-deploy, image auto-update, label bulk actions, fleet snapshot redeploy,
and mesh redeploy) acquire the per-node, per-stack lock through a new
StackOpLockService.runExclusive helper and skip rather than race a manual
deploy/update/rollback/backup on the same stack and node. Skips surface
honestly (a failed scheduled run, a recorded webhook failure, a per-stack
batch result, or a thrown error) instead of a silent no-op.

Update readiness and policy-bypass now run against the node captured when the
dialog opened, not the live active node, so switching nodes while a dialog is
open cannot retarget the update or the bypass retry.

Rollback readiness no longer presents a moving-tag or unpinned image as a ready
image revert. Restoring files does not revert a moving tag, so those stacks
read as partial, and the rollback success message states that the compose and
env files were restored.

* fix: lock blueprint reconcile against manual ops and correct rollback wording

Follow-up to the deploy/update safety hardening, closing two more gaps from a
verification pass.

BlueprintService.deployLocal and withdrawLocal called ComposeService directly,
so blueprint reconciliation could race a manual deploy/update/rollback/backup on
an owned stack. Both now run their compose lifecycle call through
StackOpLockService.runExclusive and skip (recorded as a failed reconcile,
retried on the next cycle) on conflict. The withdraw holds the lock across both
the compose down and the directory delete so neither races a manual operation.

The runtime rollback messages overstated recovery: a rollback restores the
compose and env files and recreates containers, but does not revert an image
behind a moving tag. The auto-rollback deploy-progress output, the recovery
panel and chip, the failure toasts, and the manual rollback route message now
state that the compose and env files were restored, with the matching OpenAPI
example and atomic-deployments doc updated.

* fix: acquire stack lock before blueprint deploy mutates compose and marker files

Local blueprint deploy wrote the compose and marker files and ran the policy
assert before acquiring the per-stack lock; the lock only wrapped the deploy
itself. A reconcile could therefore rewrite an owned stack's files while a
manual deploy/update/rollback/backup was running. The lock now wraps the whole
critical section (create, write compose, write marker, policy assert, deploy),
so on conflict nothing is written and the reconcile records a failed outcome.

Adds a test asserting a deploy under a held lock records failed, writes no
marker file, and leaves the manual lock untouched.

* fix: make remote blueprint apply atomic under the receiving node's stack lock

Remote blueprint deploy wrote the compose and marker files to the target node
via separate HTTP calls and only locked on the final deploy, so the file writes
could race a manual operation on that node. A node's operation lock is
process-local and cannot be held by the hub across HTTP calls, so the locked
create/write/deploy now runs on the receiving node.

The locked critical section is extracted into BlueprintService.applyLocalUnderLock
and exposed via POST /api/blueprints/apply-local. The hub posts the blueprint to
that endpoint in one call; the receiving node runs create + write compose+marker
+ deploy under its own per-stack lock. Older nodes without the route answer 404
and fall back to the legacy multi-call flow. The endpoint is gated by paid tier
and the same per-stack stack:edit and stack:deploy permissions as the
PUT-compose + deploy it bundles, validates the stack name, compose size, and
marker structure, and returns 409 on a lock conflict without writing anything.

Adds tests for the atomic single-call path, the 404 legacy fallback, the 409
lock-conflict mapping, the route validation and permission paths, and the
write-compose-then-marker-then-deploy ordering of the shared locked apply.

* fix(deps): bump undici to 7.28.0 to clear high-severity advisory

The frontend CI npm audit gate (--audit-level=high) failed on a transitive
undici 7.25.0 (a dev-only dependency via jsdom): TLS certificate validation
bypass (GHSA-vmh5-mc38-953g) and cross-user cache information disclosure
(GHSA-pr7r-676h-xcf6). Bumping undici within jsdom's existing ^7.25.0 range to
7.28.0 clears the high-severity advisory and unblocks the frontend job. Lockfile
only; no direct dependency or source change.

backend/src/services/BlueprintService.ts

@@ -9,6 +9,7 @@ import {
     type Node,
 } from './DatabaseService';
 import { ComposeService } from './ComposeService';
+import { StackOpLockService, stackOpSkipMessage, type StackOpAction } from './StackOpLockService';
 import { FileSystemService } from './FileSystemService';
 import { NodeRegistry } from './NodeRegistry';
 import { PROXY_TIER_HEADER } from './license-headers';
@@ -379,8 +380,8 @@ export class BlueprintService {
 
     // ---- local primitives ----
 
-    private async stackDirExists(node: Node, blueprintName: string): Promise<boolean> {
-        const baseDir = NodeRegistry.getInstance().getComposeDir(node.id);
+    private async stackDirExists(nodeId: number, blueprintName: string): Promise<boolean> {
+        const baseDir = NodeRegistry.getInstance().getComposeDir(nodeId);
         const stackDir = path.resolve(baseDir, blueprintName);
         if (!stackDir.startsWith(path.resolve(baseDir))) return false;
         try {
@@ -403,35 +404,79 @@ export class BlueprintService {
             throw new Error(`Policy "${gate.policy?.name}" blocked deploy: ${gate.violations.length} image(s) exceed ${gate.policy?.max_severity}`);
         }
 
-        const fs = FileSystemService.getInstance(node.id);
-        if (!(await this.stackDirExists(node, blueprint.name))) {
-            await fs.createStack(blueprint.name);
-        }
-        await fs.writeStackFile(blueprint.name, COMPOSE_FILENAME, blueprint.compose_content);
-        await fs.writeStackFile(blueprint.name, MARKER_FILENAME, JSON.stringify(marker, null, 2));
-        await assertPolicyGateAllows(
-            blueprint.name,
+        const outcome = await this.applyLocalUnderLock(
             node.id,
-            buildSystemPolicyGateOptions('blueprint', {
-                auditPath: `/api/blueprints/${blueprint.id}/deployments/${node.id}`,
-            }),
+            blueprint.name,
+            blueprint.compose_content,
+            JSON.stringify(marker, null, 2),
+            `/api/blueprints/${blueprint.id}/deployments/${node.id}`,
         );
-        await ComposeService.getInstance(node.id).deployStack(blueprint.name, undefined, false);
+        if (!outcome.ran) {
+            throw new Error(stackOpSkipMessage(blueprint.name, outcome.existingAction));
+        }
         triggerPostDeployScan(blueprint.name, node.id).catch(err => {
             console.error('[BlueprintService] post-deploy scan failed for "%s" on node %s: %s',
                 sanitizeForLog(blueprint.name), node.id, sanitizeForLog(BlueprintService.formatError(err)));
         });
     }
 
+    /**
+     * Create the stack if needed, write the compose and marker files, run the
+     * deploy policy gate, and deploy, all under the per-stack operation lock so
+     * none of it can race a manual deploy/update/rollback/backup on the same
+     * stack and node. Runs on the node that owns the stack: deployLocal calls it
+     * for the hub's own node, and the /api/blueprints/apply-local route calls it
+     * on a remote node receiving a blueprint apply from its hub (so the file
+     * writes hold the remote's lock, not just the deploy). On lock conflict
+     * nothing is written and { ran: false } is returned.
+     */
+    async applyLocalUnderLock(
+        nodeId: number,
+        stackName: string,
+        composeContent: string,
+        markerContent: string,
+        auditPath: string,
+    ): Promise<{ ran: true } | { ran: false; existingAction: StackOpAction }> {
+        const fs = FileSystemService.getInstance(nodeId);
+        const lock = await StackOpLockService.getInstance().runExclusive(
+            nodeId, stackName, 'deploy', 'system',
+            async () => {
+                if (!(await this.stackDirExists(nodeId, stackName))) {
+                    await fs.createStack(stackName);
+                }
+                await fs.writeStackFile(stackName, COMPOSE_FILENAME, composeContent);
+                await fs.writeStackFile(stackName, MARKER_FILENAME, markerContent);
+                await assertPolicyGateAllows(
+                    stackName,
+                    nodeId,
+                    buildSystemPolicyGateOptions('blueprint', { auditPath }),
+                );
+                await ComposeService.getInstance(nodeId).deployStack(stackName, undefined, false);
+            },
+        );
+        return lock.ran ? { ran: true } : { ran: false, existingAction: lock.existing.action };
+    }
+
     private async withdrawLocal(blueprint: Blueprint, node: Node): Promise<void> {
-        try {
-            await ComposeService.getInstance(node.id).downStack(blueprint.name);
-        } catch (err) {
-            // best-effort: continue to delete the directory even if down fails
-            console.warn(`[BlueprintService] downStack failed for "${blueprint.name}" on node ${node.id}: ${BlueprintService.formatError(err)}`);
-        }
-        if (await this.stackDirExists(node, blueprint.name)) {
-            await FileSystemService.getInstance(node.id).deleteStack(blueprint.name);
+        // Hold the per-stack lock across both the compose down and the directory
+        // delete so a withdraw cannot race a manual operation, nor tear the
+        // files out from under one that starts mid-withdraw.
+        const lock = await StackOpLockService.getInstance().runExclusive(
+            node.id, blueprint.name, 'down', 'system',
+            async () => {
+                try {
+                    await ComposeService.getInstance(node.id).downStack(blueprint.name);
+                } catch (err) {
+                    // best-effort: continue to delete the directory even if down fails
+                    console.warn(`[BlueprintService] downStack failed for "${blueprint.name}" on node ${node.id}: ${BlueprintService.formatError(err)}`);
+                }
+                if (await this.stackDirExists(node.id, blueprint.name)) {
+                    await FileSystemService.getInstance(node.id).deleteStack(blueprint.name);
+                }
+            },
+        );
+        if (!lock.ran) {
+            throw new Error(stackOpSkipMessage(blueprint.name, lock.existing.action));
         }
     }
 
@@ -452,6 +497,45 @@ export class BlueprintService {
         const baseUrl = target.apiUrl.replace(/\/$/, '');
         const headers = this.remoteHeaders(target.apiToken);
 
+        // Atomic apply: the remote runs create + write compose/marker + deploy
+        // under its own per-stack lock, so the file writes cannot race a manual
+        // operation on that node. Older nodes without this route answer 404; we
+        // fall back to the legacy multi-call flow there (not lock-atomic).
+        const res = await axios.post(
+            `${baseUrl}/api/blueprints/apply-local`,
+            {
+                stackName: blueprint.name,
+                composeContent: blueprint.compose_content,
+                markerContent: JSON.stringify(marker, null, 2),
+            },
+            { headers, timeout: REMOTE_HTTP_TIMEOUT_MS, validateStatus: () => true },
+        );
+        if (res.status === 404) {
+            console.warn(`[BlueprintService] remote node ${node.id} lacks /api/blueprints/apply-local; using legacy non-atomic apply`);
+            await this.deployRemoteLegacy(blueprint, node, marker);
+            return;
+        }
+        if (res.status === 409) {
+            throw new Error(`blueprint apply skipped: ${BlueprintService.extractApiError(res.data) || 'another operation is already in progress'}`);
+        }
+        if (res.status >= 400) {
+            throw new Error(`blueprint apply: HTTP ${res.status} ${BlueprintService.extractApiError(res.data)}`);
+        }
+    }
+
+    /**
+     * Legacy remote apply for nodes that predate /api/blueprints/apply-local:
+     * create, write compose, write marker, deploy as separate calls. The remote
+     * deploy locks, but the preceding file writes do not, so this is not atomic
+     * against a concurrent manual operation on that node. Kept only as a
+     * compatibility fallback.
+     */
+    private async deployRemoteLegacy(blueprint: Blueprint, node: Node, marker: BlueprintMarker): Promise<void> {
+        const target = NodeRegistry.getInstance().getProxyTarget(node.id);
+        if (!target) throw new Error(`Remote node "${node.name}" has no proxy target configured`);
+        const baseUrl = target.apiUrl.replace(/\/$/, '');
+        const headers = this.remoteHeaders(target.apiToken);
+
         // 1. Ensure stack exists. POST returns 409 when already exists; we treat that as success.
         const createRes = await axios.post(`${baseUrl}/api/stacks`,
             { stackName: blueprint.name },

backend/src/services/ComposeService.ts

@@ -341,7 +341,7 @@ export class ComposeService {
       await this.withRegistryAuth(async (env) => {
         await this.execute('docker', await this.authoredComposeArgs(stackName, ['up', '-d', '--remove-orphans']), stackDir, ws, true, env);
       }, sendOutput);
-      sendOutput('=== Rolled back successfully ===\n');
+      sendOutput('=== Restored previous compose and env files ===\n');
       return true;
     } catch (rollbackError) {
       console.error('Rollback failed for %s:', sanitizeForLog(stackName), getErrorMessage(rollbackError, 'unknown error'));
@@ -413,7 +413,7 @@ export class ComposeService {
       if (debug) console.debug(`[ComposeService:debug] deployStack completed in ${Date.now() - t0}ms`, { stackName });
     } catch (deployError) {
       if (atomic) {
-        sendOutput('\n=== Deployment failed - rolling back to previous version ===\n');
+        sendOutput('\n=== Deployment failed - restoring previous compose and env files ===\n');
         const rolledBack = await this.restoreAtomicBackup(stackName, stackDir, ws, sendOutput);
         throw new ComposeRollbackError(deployError, true, rolledBack);
       }
@@ -613,7 +613,7 @@ export class ComposeService {
       if (debug) console.debug(`[ComposeService:debug] updateStack completed in ${Date.now() - t0}ms`, { stackName });
     } catch (updateError) {
       if (atomic) {
-        sendOutput('\n=== Update failed - rolling back to previous version ===\n');
+        sendOutput('\n=== Update failed - restoring previous compose and env files ===\n');
         const rolledBack = await this.restoreAtomicBackup(stackName, stackDir, ws, sendOutput);
         throw new ComposeRollbackError(updateError, true, rolledBack);
       }

backend/src/services/GitSourceService.ts

@@ -8,6 +8,7 @@ import { CryptoService } from './CryptoService';
 import { DatabaseService, type StackGitSource, type GitSourceAuthType, type GitSourceAppliedSpec } from './DatabaseService';
 import { FileSystemService } from './FileSystemService';
 import { ComposeService } from './ComposeService';
+import { StackOpLockService } from './StackOpLockService';
 import { HealthGateService } from './HealthGateService';
 import { NodeRegistry } from './NodeRegistry';
 import { assertPolicyGateAllows, buildSystemPolicyGateOptions } from '../helpers/policyGate';
@@ -1315,7 +1316,15 @@ export class GitSourceService {
                         auditPath: `/api/stacks/${stackName}/git-source/apply`,
                     }),
                 );
-                await ComposeService.getInstance().deployStack(stackName);
+                const lock = await StackOpLockService.getInstance().runExclusive(
+                    nodeId, stackName, 'deploy', 'system',
+                    () => ComposeService.getInstance(nodeId).deployStack(stackName),
+                );
+                if (!lock.ran) {
+                    const busy = `Auto-deploy skipped: another operation (${lock.existing.action}) is already in progress for ${stackName}.`;
+                    console.warn(`[GitSource] ${busy}`);
+                    return { applied: true, deployed: false, deployError: busy };
+                }
                 HealthGateService.getInstance().begin(nodeId, stackName, 'deploy', 'system:git-source');
                 console.log(`[GitSource] Applied and deployed ${stackName} at ${commitSha.slice(0, 7)}`);
                 return { applied: true, deployed: true };

backend/src/services/MeshService.ts

@@ -4,6 +4,7 @@ import fs from 'fs/promises';
 import { EventEmitter } from 'events';
 import * as YAML from 'yaml';
 import { ComposeService } from './ComposeService';
+import { StackOpLockService } from './StackOpLockService';
 import { DatabaseService, type NodeMode } from './DatabaseService';
 import DockerController from './DockerController';
 import { FileSystemService } from './FileSystemService';
@@ -2380,7 +2381,13 @@ export class MeshService extends EventEmitter implements MeshForwarderHost {
                     auditPath: `/api/mesh/nodes/${nodeId}/stacks/${stackName}/redeploy`,
                 }),
             );
-            await ComposeService.getInstance(nodeId).deployStack(stackName);
+            const lock = await StackOpLockService.getInstance().runExclusive(
+                nodeId, stackName, 'deploy', 'system',
+                () => ComposeService.getInstance(nodeId).deployStack(stackName),
+            );
+            if (!lock.ran) {
+                throw new Error(`Cannot redeploy "${stackName}": another operation (${lock.existing.action}) is already in progress.`);
+            }
             this.logActivity({
                 source: 'mesh', level: 'info', type: 'mesh.enable',
                 nodeId,

backend/src/services/SchedulerService.ts

@@ -5,6 +5,7 @@ import { LicenseService } from './LicenseService';
 import { PROXY_TIER_HEADER } from './license-headers';
 import DockerController from './DockerController';
 import { ComposeService } from './ComposeService';
+import { StackOpLockService, stackOpSkipMessage as skipMessage } from './StackOpLockService';
 import { FileSystemService } from './FileSystemService';
 import { HealthGateService } from './HealthGateService';
 import { ImageUpdateService } from './ImageUpdateService';
@@ -488,7 +489,14 @@ export class SchedulerService {
             await this.postToRemoteStack(task.node_id, `${encodeURIComponent(task.target_id)}/backup`);
             return `Backed up stack "${task.target_id}" files on remote node`;
         }
-        await FileSystemService.getInstance(task.node_id).backupStackFiles(task.target_id);
+        const localNodeId = task.node_id ?? NodeRegistry.getInstance().getDefaultNodeId();
+        const lock = await StackOpLockService.getInstance().runExclusive(
+            localNodeId, task.target_id, 'backup', 'system',
+            () => FileSystemService.getInstance(localNodeId).backupStackFiles(task.target_id),
+        );
+        // Throw (not return) so the skip records as a failed run instead of a
+        // silent success; the next scheduled tick retries once the lock frees.
+        if (!lock.ran) throw new Error(skipMessage(task.target_id, lock.existing.action));
         return `Backed up stack "${task.target_id}" files`;
     }
 
@@ -498,7 +506,12 @@ export class SchedulerService {
             await this.postToRemoteStack(task.node_id, `${encodeURIComponent(task.target_id)}/stop`);
             return `Stopped stack "${task.target_id}" (containers preserved) on remote node`;
         }
-        await ComposeService.getInstance(task.node_id).runCommand(task.target_id, 'stop');
+        const localNodeId = task.node_id ?? NodeRegistry.getInstance().getDefaultNodeId();
+        const lock = await StackOpLockService.getInstance().runExclusive(
+            localNodeId, task.target_id, 'stop', 'system',
+            () => ComposeService.getInstance(localNodeId).runCommand(task.target_id, 'stop'),
+        );
+        if (!lock.ran) throw new Error(skipMessage(task.target_id, lock.existing.action));
         return `Stopped stack "${task.target_id}" (containers preserved)`;
     }
 
@@ -508,7 +521,12 @@ export class SchedulerService {
             await this.postToRemoteStack(task.node_id, `${encodeURIComponent(task.target_id)}/down`);
             return `Took down stack "${task.target_id}" (containers removed) on remote node`;
         }
-        await ComposeService.getInstance(task.node_id).runCommand(task.target_id, 'down');
+        const localNodeId = task.node_id ?? NodeRegistry.getInstance().getDefaultNodeId();
+        const lock = await StackOpLockService.getInstance().runExclusive(
+            localNodeId, task.target_id, 'down', 'system',
+            () => ComposeService.getInstance(localNodeId).runCommand(task.target_id, 'down'),
+        );
+        if (!lock.ran) throw new Error(skipMessage(task.target_id, lock.existing.action));
         return `Took down stack "${task.target_id}" (containers removed)`;
     }
 
@@ -527,7 +545,12 @@ export class SchedulerService {
             'Auto-start',
             `/api/scheduled-tasks/${task.id}/run`,
         );
-        await ComposeService.getInstance(task.node_id).deployStack(task.target_id);
+        const localNodeId = task.node_id ?? NodeRegistry.getInstance().getDefaultNodeId();
+        const lock = await StackOpLockService.getInstance().runExclusive(
+            localNodeId, task.target_id, 'deploy', 'system',
+            () => ComposeService.getInstance(localNodeId).deployStack(task.target_id),
+        );
+        if (!lock.ran) throw new Error(skipMessage(task.target_id, lock.existing.action));
         return `Started stack "${task.target_id}"`;
     }
 
@@ -860,7 +883,11 @@ export class SchedulerService {
         // Atomic backup/rollback is the default deploy mode: take a pre-op
         // backup and roll back on failure for every scheduled auto-update.
         const atomic = true;
-        await compose.updateStack(stackName, undefined, atomic);
+        const lock = await StackOpLockService.getInstance().runExclusive(
+            nodeId, stackName, 'update', 'system',
+            () => compose.updateStack(stackName, undefined, atomic),
+        );
+        if (!lock.ran) return skipMessage(stackName, lock.existing.action);
         db.clearStackUpdateStatus(nodeId, stackName);
         HealthGateService.getInstance().begin(nodeId, stackName, 'update', 'system:scheduler');
 

backend/src/services/StackOpLockService.ts

@@ -11,6 +11,14 @@
 
 export type StackOpAction = 'deploy' | 'down' | 'restart' | 'stop' | 'start' | 'update' | 'rollback' | 'backup';
 
+/**
+ * Note returned by a background path that skipped its operation because a manual
+ * or concurrent operation already held the stack's lock.
+ */
+export function stackOpSkipMessage(stackName: string, existingAction: StackOpAction): string {
+  return `Skipped "${stackName}": another operation (${existingAction}) is already in progress.`;
+}
+
 export interface StackOpLock {
   action: StackOpAction;
   startedAt: number;
@@ -62,6 +70,34 @@ export class StackOpLockService {
     this.locks.delete(this.key(nodeId, stackName));
   }
 
+  /**
+   * Acquire the per-(nodeId, stackName) lock for the duration of `fn`, then
+   * release it. Returns `{ ran: true, result }` when the lock was free, or
+   * `{ ran: false, existing }` when another operation already holds it, so the
+   * caller can skip rather than race. Background/system paths (scheduler,
+   * webhook, Git source, image auto-update, label bulk actions, fleet snapshot
+   * redeploy, mesh redeploy) run their lifecycle calls through this so they
+   * cannot interleave with a manual deploy/update/rollback/backup on the same
+   * stack and node. An error thrown by `fn` still releases the lock, then
+   * propagates to the caller.
+   */
+  public async runExclusive<T>(
+    nodeId: number,
+    stackName: string,
+    action: StackOpAction,
+    user: string,
+    fn: () => Promise<T>,
+  ): Promise<{ ran: true; result: T } | { ran: false; existing: StackOpLock }> {
+    const acquired = this.tryAcquire(nodeId, stackName, action, user);
+    if (!acquired.acquired) return { ran: false, existing: acquired.existing };
+    try {
+      const result = await fn();
+      return { ran: true, result };
+    } finally {
+      this.release(nodeId, stackName);
+    }
+  }
+
   public get(nodeId: number, stackName: string): StackOpLock | undefined {
     return this.locks.get(this.key(nodeId, stackName));
   }

backend/src/services/UpdateGuardService.ts

@@ -3,7 +3,7 @@ import DockerController from './DockerController';
 import { DatabaseService } from './DatabaseService';
 import { FileSystemService } from './FileSystemService';
 import { ComposeDoctorService } from './ComposeDoctorService';
-import { UpdatePreviewService } from './UpdatePreviewService';
+import { UpdatePreviewService, isMovingTag } from './UpdatePreviewService';
 import { withTimeout } from '../utils/withTimeout';
 import { getErrorMessage } from '../utils/errors';
 import { sanitizeForLog } from '../utils/safeLog';
@@ -140,7 +140,14 @@ export class UpdateGuardService {
       backup,
       envSummary,
       stackHasEnv,
-      rollbackTarget: preview === 'error' ? 'error' : { target: preview.rollback_target },
+      rollbackTarget: preview === 'error'
+        ? 'error'
+        : {
+            target: preview.rollback_target,
+            // Any image on a moving tag means restoring files cannot guarantee
+            // the image reverts, so the rollback target is not a true revert.
+            moving: preview.images.some(img => isMovingTag(img.current_tag)),
+          },
       lastDeployAt,
       containers,
     }, now);

backend/src/services/UpdatePreviewService.ts

@@ -76,6 +76,17 @@ export function parseSemverTag(tag: string): SemverParts | null {
     };
 }
 
+/**
+ * A tag is "moving" when restoring the compose file would not revert the image
+ * behind it: `latest`, a branch name, or an unpinned major/minor like `1.25`.
+ * Only a fully-pinned semver tag (X.Y.Z, optionally `v`-prefixed and/or with a
+ * `-prerelease` suffix) is treated as immutable, matching how a file rollback
+ * restores the exact tag.
+ */
+export function isMovingTag(tag: string): boolean {
+    return parseSemverTag(tag) === null;
+}
+
 function compareSemver(a: SemverParts, b: SemverParts): number {
     if (a.major !== b.major) return a.major - b.major;
     if (a.minor !== b.minor) return a.minor - b.minor;

backend/src/services/WebhookService.ts

@@ -1,5 +1,6 @@
 import crypto from 'crypto';
 import { ComposeService } from './ComposeService';
+import { StackOpLockService, stackOpSkipMessage, type StackOpAction } from './StackOpLockService';
 import { DatabaseService, type Webhook } from './DatabaseService';
 import { FileSystemService } from './FileSystemService';
 import { GitSourceService } from './GitSourceService';
@@ -17,6 +18,16 @@ type ExecutionStatus = 'success' | 'failure';
 
 const REMOTE_WEBHOOK_REQUEST_TIMEOUT_MS = 30_000;
 
+// Maps a webhook lifecycle action to the per-stack lock action. 'pull' updates,
+// so it locks as 'update'; 'git-pull' is excluded (it locks inside GitSourceService).
+const WEBHOOK_LOCK_ACTION: Record<string, StackOpAction | undefined> = {
+    deploy: 'deploy',
+    restart: 'restart',
+    stop: 'stop',
+    start: 'start',
+    pull: 'update',
+};
+
 export class WebhookService {
     private static instance: WebhookService;
     // Stable per-process decoy secret used to keep HMAC work non-skippable on
@@ -125,42 +136,57 @@ export class WebhookService {
 
         const startTime = Date.now();
         try {
-            const compose = ComposeService.getInstance(nodeId);
-            switch (action) {
-                case 'deploy':
-                    await assertPolicyGateAllows(
-                        stackName,
-                        nodeId,
-                        buildSystemPolicyGateOptions('webhook', { auditPath: `/api/webhooks/${webhookId}/execute` }),
-                    );
-                    await compose.deployStack(stackName, undefined, atomic);
-                    HealthGateService.getInstance().begin(nodeId, stackName, 'deploy', 'system:webhook');
-                    break;
-                case 'restart':
-                    await compose.runCommand(stackName, 'restart');
-                    break;
-                case 'stop':
-                    await compose.runCommand(stackName, 'stop');
-                    break;
-                case 'start':
-                    await compose.runCommand(stackName, 'start');
-                    break;
-                case 'pull':
-                    await assertPolicyGateAllows(
-                        stackName,
-                        nodeId,
-                        buildSystemPolicyGateOptions('webhook', { auditPath: `/api/webhooks/${webhookId}/execute` }),
-                    );
-                    await compose.updateStack(stackName, undefined, atomic);
-                    HealthGateService.getInstance().begin(nodeId, stackName, 'update', 'system:webhook');
-                    break;
-                case 'git-pull':
-                    return this.executeLocalGitPull(webhookId, stackName, action, triggerSource, startTime);
-                default:
-                    throw new Error(`Unknown action: ${action}`);
+            // git-pull pulls then deploys through GitSourceService, which holds
+            // the per-stack lock itself; locking here too would self-conflict.
+            if (action === 'git-pull') {
+                return this.executeLocalGitPull(webhookId, stackName, action, triggerSource, startTime);
             }
+            const lockAction = WEBHOOK_LOCK_ACTION[action];
+            if (!lockAction) throw new Error(`Unknown action: ${action}`);
+            const compose = ComposeService.getInstance(nodeId);
+            // Run the lifecycle op under the per-stack lock so a webhook cannot
+            // race a manual deploy/update/rollback/backup on the same stack.
+            const lock = await StackOpLockService.getInstance().runExclusive(
+                nodeId, stackName, lockAction, 'system',
+                async () => {
+                    switch (action) {
+                        case 'deploy':
+                            await assertPolicyGateAllows(
+                                stackName,
+                                nodeId,
+                                buildSystemPolicyGateOptions('webhook', { auditPath: `/api/webhooks/${webhookId}/execute` }),
+                            );
+                            await compose.deployStack(stackName, undefined, atomic);
+                            HealthGateService.getInstance().begin(nodeId, stackName, 'deploy', 'system:webhook');
+                            break;
+                        case 'restart':
+                            await compose.runCommand(stackName, 'restart');
+                            break;
+                        case 'stop':
+                            await compose.runCommand(stackName, 'stop');
+                            break;
+                        case 'start':
+                            await compose.runCommand(stackName, 'start');
+                            break;
+                        case 'pull':
+                            await assertPolicyGateAllows(
+                                stackName,
+                                nodeId,
+                                buildSystemPolicyGateOptions('webhook', { auditPath: `/api/webhooks/${webhookId}/execute` }),
+                            );
+                            await compose.updateStack(stackName, undefined, atomic);
+                            HealthGateService.getInstance().begin(nodeId, stackName, 'update', 'system:webhook');
+                            break;
+                    }
+                },
+            );
 
             const durationMs = Date.now() - startTime;
+            if (!lock.ran) {
+                const error = stackOpSkipMessage(stackName, lock.existing.action);
+                this.recordExecution(webhookId, action, 'failure', triggerSource, durationMs, error);
+                return { success: false, error, duration_ms: durationMs };
+            }
             this.recordExecution(webhookId, action, 'success', triggerSource, durationMs, null);
             return { success: true, duration_ms: durationMs };
         } catch (err) {

backend/src/services/updateGuard/readiness.ts

@@ -188,9 +188,12 @@ export interface RollbackInputs {
   /**
    * UpdatePreview.rollback_target wrapped in an object so the Errored sentinel
    * cannot be absorbed into the string domain (an image literally named
-   * "error" must not read as a failed preview).
+   * "error" must not read as a failed preview). `moving` is true when any image
+   * in the stack uses a moving tag (latest, a branch, an unpinned major/minor),
+   * in which case restoring files does not revert the image because the local
+   * tag already resolves to the newer digest.
    */
-  rollbackTarget: { target: string | null } | Errored;
+  rollbackTarget: { target: string | null; moving: boolean } | Errored;
   /** Timestamp of the most recent deploy_success activity event, if any. */
   lastDeployAt: number | null | Errored;
   containers: ContainerProbe[] | Errored;
@@ -224,8 +227,10 @@ export function buildRollbackItems(inputs: RollbackInputs, now: number): Rollbac
 
   if (inputs.rollbackTarget === 'error') {
     items.push({ id: 'previous_images', state: 'unknown', label: 'Previous image tag', detail: 'The update preview is unavailable.' });
+  } else if (inputs.rollbackTarget.target && inputs.rollbackTarget.moving) {
+    items.push({ id: 'previous_images', state: 'not_covered', label: 'Previous image tag', detail: `Rollback target ${inputs.rollbackTarget.target}. This stack uses a moving image tag, so restoring the compose and env files does not revert the image: the local tag still resolves to the newer digest. Pin every image to an immutable version tag for a true image rollback.` });
   } else if (inputs.rollbackTarget.target) {
-    items.push({ id: 'previous_images', state: 'ready', label: 'Previous image tag', detail: `Known rollback target: ${inputs.rollbackTarget.target}. If the compose file uses a moving tag, restoring files alone does not revert the image; pin this tag to be exact.` });
+    items.push({ id: 'previous_images', state: 'ready', label: 'Previous image tag', detail: `Known rollback target: ${inputs.rollbackTarget.target}. The compose file pins an immutable tag, so restoring files also restores the image.` });
   } else {
     items.push({ id: 'previous_images', state: 'unknown', label: 'Previous image tag', detail: 'The previous image tag could not be determined. A rollback restores compose and env files; a moving tag may keep the newer image.' });
   }