fix(scheduled-ops): run stack lifecycle schedules on remote nodes and harden run visibility (#1260)

* fix(scheduled-ops): run stack lifecycle schedules on remote nodes and harden run visibility

Stack lifecycle schedules (Restart, Stop, Take Down, Start, Backup Stack
Files) now run against whichever node the schedule targets, local or
remote. Each remote run proxies to that node's own stack-operation
endpoint, so a hub-managed schedule reaches the node that actually holds
the stack. Restart with a service subset restarts each selected service
and, if one fails, names the services already restarted so run history
reflects the stack's partial state. Auto-start on a remote node runs that
node's own pre-deploy scan-policy check against the images it holds.

Add POST /api/stacks/:name/backup to trigger an on-demand backup of a
stack's compose and env files (the same rollback snapshot a deploy
takes); it backs the remote backup schedule and is available to operators
on its own.

A scheduled task that reaches execution on an unpaid licence is now
skipped and written to run history as a failed run, so a manual trigger
that returned a queued response never silently disappears.

Test plan:
- Backend unit + integration: scheduler-service (remote proxy per action,
  per-service fan-out, auto-start policy delegation, remote-failure and
  no-credentials paths, unpaid-tier skip), stack-backup-route
  (auth/role/paid/404/400/500), scheduled-tasks-routes.
- Frontend component test for the schedules view (list, prefill, node
  filter, create payload).
- tsc and lint clean on both packages.

* fix(scheduled-ops): lock the stack-files backup route against concurrent stack ops

The stack-files backup writes the same slot the pre-deploy rollback
snapshot uses, so running it while a deploy, update, or rollback is in
flight on the same stack could overwrite the rollback point. The backup
route now takes the per-stack operation lock (as deploy/down/restart do)
and returns 409 when the stack is busy, keeping the rollback snapshot
intact. Adds the 'backup' action to the stack-op lock type and a busy
participle for the 409 message.

* fix(scheduled-ops): enforce backup-path containment inline at the filesystem sink

The on-demand backup route passes the stack name straight into
backupStackFiles, so resolve the backup directory against the backup root
and confirm containment with an inline startsWith check before the
mkdir/copy/write sinks, matching the barrier restoreStackFiles already
uses. The stack name is validated at the route and again by
resolveStackDir, so this is defense in depth that also closes a
static path-injection finding on the new call path.

backend/src/services/FileSystemService.ts

@@ -485,7 +485,16 @@ export class FileSystemService {
     const debug = isDebugEnabled();
     const t0 = Date.now();
     const stackDir = this.resolveStackDir(stackName);
-    const backupDir = this.getBackupDir(stackName);
+    // Canonical js/path-injection barrier (mirrors restoreStackFiles): resolve the
+    // backup path against the backup root and confirm containment inline, so the
+    // mkdir/copy/write sinks below operate on a validated path. stackName is
+    // already validated by resolveStackDir above; this re-establishes containment
+    // at the backup sinks themselves so static analysis sees the barrier.
+    const backupRoot = path.resolve(getBackupBaseDir());
+    const backupDir = path.resolve(backupRoot, String(this.nodeId), stackName);
+    if (!backupDir.startsWith(backupRoot + path.sep)) {
+      throw Object.assign(new Error('Path escapes backup directory'), { code: 'INVALID_PATH' });
+    }
     await fsPromises.mkdir(backupDir, { recursive: true });
 
     // Clear stale managed files from the backup slot before writing the current
@@ -493,11 +502,9 @@ export class FileSystemService {
     // stack since the last backup (e.g. a deleted .env or a switched compose
     // variant) would otherwise linger here and a later restore would resurrect
     // it, breaking the faithful-revert guarantee. Scope is the protected set
-    // Sencho writes; .timestamp is rewritten below. Containment is re-checked at
-    // the sink, rooted at the backup base, for the same reason restoreStackFiles
-    // does it. A clear failure is logged but not fatal: it only risks a stale
-    // future rollback, so it should not block an otherwise valid deploy.
-    const backupRoot = path.resolve(getBackupBaseDir());
+    // Sencho writes; .timestamp is rewritten below. A clear failure is logged but
+    // not fatal: it only risks a stale future rollback, so it should not block an
+    // otherwise valid deploy.
     for (const file of PROTECTED_STACK_FILES) {
       const stale = path.resolve(backupRoot, path.join(backupDir, file));
       if (!stale.startsWith(backupRoot + path.sep)) continue;

backend/src/services/SchedulerService.ts

@@ -291,6 +291,21 @@ export class SchedulerService {
             triggered_by: triggeredBy,
         });
 
+        // Defense in depth: every entry point that reaches here is already paid-gated
+        // (the route's requirePaid and the tick's tier check), but guard again so a
+        // task can never run on an unpaid licence regardless of the caller. Record the
+        // skip as a failed run so a manual trigger (which already returned 202 to the
+        // operator) shows in run history rather than vanishing silently.
+        if (LicenseService.getInstance().getTier() !== 'paid') {
+            console.warn(`[SchedulerService] Skipping task "${task.name}" (id=${task.id}): licence is not paid`);
+            db.updateScheduledTaskRun(runId, {
+                completed_at: Date.now(),
+                status: 'failure',
+                error: 'Scheduled tasks require a paid licence; task was not run.',
+            });
+            return;
+        }
+
         try {
             // Pre-check: ensure target node exists and is reachable
             if (task.node_id != null && task.action !== 'snapshot') {
@@ -423,6 +438,9 @@ export class SchedulerService {
         if (!task.target_id || task.node_id == null) {
             throw new Error('Stack restart requires target_id and node_id');
         }
+        if (this.isRemoteNode(task.node_id)) {
+            return this.executeRestartRemote(task.node_id, task.target_id, task.target_services);
+        }
         const docker = DockerController.getInstance(task.node_id);
         const containers = await docker.getContainersByStack(task.target_id);
         if (!containers || containers.length === 0) {
@@ -445,32 +463,82 @@ export class SchedulerService {
         return `Restarted ${filtered.length} container(s) in stack "${task.target_id}"${servicesSuffix}`;
     }
 
+    /**
+     * Remote restart. The remote bulk-restart endpoint restarts every container
+     * in the stack, so when the task targets specific services we fan out to the
+     * per-service restart route to preserve the filter.
+     */
+    private async executeRestartRemote(nodeId: number, stackName: string, targetServices: string | null): Promise<string> {
+        const stackSeg = encodeURIComponent(stackName);
+        if (targetServices) {
+            const serviceNames: string[] = JSON.parse(targetServices);
+            // Fail fast, but name the services already restarted so a mid-loop failure
+            // records the partial state of the remote stack in run history.
+            const restarted: string[] = [];
+            for (const svc of serviceNames) {
+                try {
+                    await this.postToRemoteStack(nodeId, `${stackSeg}/services/${encodeURIComponent(svc)}/restart`);
+                    restarted.push(svc);
+                } catch (e) {
+                    const done = restarted.length ? ` (already restarted: ${restarted.join(', ')})` : '';
+                    throw new Error(`Restart of service "${svc}" failed${done}: ${getErrorMessage(e, String(e))}`);
+                }
+            }
+            return `Restarted services [${serviceNames.join(', ')}] in stack "${stackName}" on remote node`;
+        }
+        await this.postToRemoteStack(nodeId, `${stackSeg}/restart`);
+        return `Restarted stack "${stackName}" on remote node`;
+    }
+
     private assertStackTarget(task: ScheduledTask, label: string): asserts task is ScheduledTask & { target_id: string; node_id: number } {
         if (!task.target_id || task.node_id == null) {
             throw new Error(`${label} requires target_id and node_id`);
         }
     }
 
+    private isRemoteNode(nodeId: number): boolean {
+        return NodeRegistry.getInstance().getNode(nodeId)?.type === 'remote';
+    }
+
     private async executeAutoBackup(task: ScheduledTask): Promise<string> {
         this.assertStackTarget(task, 'Auto-backup');
+        if (this.isRemoteNode(task.node_id)) {
+            await this.postToRemoteStack(task.node_id, `${encodeURIComponent(task.target_id)}/backup`);
+            return `Backed up stack "${task.target_id}" files on remote node`;
+        }
         await FileSystemService.getInstance(task.node_id).backupStackFiles(task.target_id);
         return `Backed up stack "${task.target_id}" files`;
     }
 
     private async executeAutoStop(task: ScheduledTask): Promise<string> {
         this.assertStackTarget(task, 'Auto-stop');
+        if (this.isRemoteNode(task.node_id)) {
+            await this.postToRemoteStack(task.node_id, `${encodeURIComponent(task.target_id)}/stop`);
+            return `Stopped stack "${task.target_id}" (containers preserved) on remote node`;
+        }
         await ComposeService.getInstance(task.node_id).runCommand(task.target_id, 'stop');
         return `Stopped stack "${task.target_id}" (containers preserved)`;
     }
 
     private async executeAutoDown(task: ScheduledTask): Promise<string> {
         this.assertStackTarget(task, 'Auto-down');
+        if (this.isRemoteNode(task.node_id)) {
+            await this.postToRemoteStack(task.node_id, `${encodeURIComponent(task.target_id)}/down`);
+            return `Took down stack "${task.target_id}" (containers removed) on remote node`;
+        }
         await ComposeService.getInstance(task.node_id).runCommand(task.target_id, 'down');
         return `Took down stack "${task.target_id}" (containers removed)`;
     }
 
     private async executeAutoStart(task: ScheduledTask): Promise<string> {
         this.assertStackTarget(task, 'Auto-start');
+        // Remote auto-start proxies to the remote's own deploy route, which runs
+        // that node's scan-policy gate against the images it actually holds. The
+        // hub-side enforceSchedulerPolicyGate below is for local nodes only.
+        if (this.isRemoteNode(task.node_id)) {
+            await this.postToRemoteStack(task.node_id, `${encodeURIComponent(task.target_id)}/deploy`);
+            return `Started stack "${task.target_id}" on remote node`;
+        }
         await this.enforceSchedulerPolicyGate(
             task.target_id,
             task.node_id,
@@ -684,6 +752,40 @@ export class SchedulerService {
         return body.result || 'Remote auto-update completed (no details returned).';
     }
 
+    /**
+     * Proxy a stack lifecycle action to a remote Sencho instance. ComposeService,
+     * DockerController, and FileSystemService are local-only, so for a remote node
+     * we POST to the remote's own stack-operation endpoint with the node Bearer
+     * token and the licence proxy headers, exactly as executeUpdateRemote does.
+     * `routeSuffix` is the path under `/api/stacks/`; the caller URL-encodes each
+     * segment.
+     */
+    private async postToRemoteStack(nodeId: number, routeSuffix: string): Promise<void> {
+        const proxyTarget = NodeRegistry.getInstance().getProxyTarget(nodeId);
+        if (!proxyTarget) {
+            throw new Error('Remote node is not configured or missing API credentials');
+        }
+        const baseUrl = proxyTarget.apiUrl.replace(/\/$/, '');
+        const proxyHeaders = LicenseService.getInstance().getProxyHeaders();
+        if (isDebugEnabled()) {
+            console.log(`[SchedulerService:debug] postToRemoteStack: node=${nodeId} route=${routeSuffix}`);
+        }
+        const response = await fetch(`${baseUrl}/api/stacks/${routeSuffix}`, {
+            method: 'POST',
+            headers: {
+                'Content-Type': 'application/json',
+                'Authorization': `Bearer ${proxyTarget.apiToken}`,
+                [PROXY_TIER_HEADER]: proxyHeaders.tier,
+                [PROXY_VARIANT_HEADER]: proxyHeaders.variant ?? '',
+            },
+            signal: AbortSignal.timeout(300_000),
+        });
+        if (!response.ok) {
+            const body = await response.json().catch(() => ({ error: `HTTP ${response.status}` }));
+            throw new Error((body as { error?: string }).error || `Remote node returned ${response.status}`);
+        }
+    }
+
     private async executeUpdateForStack(
         stackName: string,
         nodeId: number,

backend/src/services/StackOpLockService.ts

@@ -1,14 +1,15 @@
 /**
  * Tracks in-flight stack lifecycle operations (deploy, down, restart, stop,
- * start, update, rollback) per (nodeId, stackName). A second request to the
- * same stack while the first is still running returns 409 instead of racing
- * the first.
+ * start, update, rollback, backup) per (nodeId, stackName). A second request to
+ * the same stack while the first is still running returns 409 instead of racing
+ * the first. Backup is included because it rewrites the shared rollback slot, so
+ * it must not interleave with a deploy/update/rollback on the same stack.
  *
  * State is intentionally process-local: a Sencho restart clears all locks,
  * which matches the lifecycle of any in-flight `docker compose` child process.
  */
 
-export type StackOpAction = 'deploy' | 'down' | 'restart' | 'stop' | 'start' | 'update' | 'rollback';
+export type StackOpAction = 'deploy' | 'down' | 'restart' | 'stop' | 'start' | 'update' | 'rollback' | 'backup';
 
 export interface StackOpLock {
   action: StackOpAction;