fix(scheduled-ops): run stack lifecycle schedules on remote nodes and harden run visibility (#1260)

* fix(scheduled-ops): run stack lifecycle schedules on remote nodes and harden run visibility

Stack lifecycle schedules (Restart, Stop, Take Down, Start, Backup Stack
Files) now run against whichever node the schedule targets, local or
remote. Each remote run proxies to that node's own stack-operation
endpoint, so a hub-managed schedule reaches the node that actually holds
the stack. Restart with a service subset restarts each selected service
and, if one fails, names the services already restarted so run history
reflects the stack's partial state. Auto-start on a remote node runs that
node's own pre-deploy scan-policy check against the images it holds.

Add POST /api/stacks/:name/backup to trigger an on-demand backup of a
stack's compose and env files (the same rollback snapshot a deploy
takes); it backs the remote backup schedule and is available to operators
on its own.

A scheduled task that reaches execution on an unpaid licence is now
skipped and written to run history as a failed run, so a manual trigger
that returned a queued response never silently disappears.

Test plan:
- Backend unit + integration: scheduler-service (remote proxy per action,
  per-service fan-out, auto-start policy delegation, remote-failure and
  no-credentials paths, unpaid-tier skip), stack-backup-route
  (auth/role/paid/404/400/500), scheduled-tasks-routes.
- Frontend component test for the schedules view (list, prefill, node
  filter, create payload).
- tsc and lint clean on both packages.

* fix(scheduled-ops): lock the stack-files backup route against concurrent stack ops

The stack-files backup writes the same slot the pre-deploy rollback
snapshot uses, so running it while a deploy, update, or rollback is in
flight on the same stack could overwrite the rollback point. The backup
route now takes the per-stack operation lock (as deploy/down/restart do)
and returns 409 when the stack is busy, keeping the rollback snapshot
intact. Adds the 'backup' action to the stack-op lock type and a busy
participle for the 409 message.

* fix(scheduled-ops): enforce backup-path containment inline at the filesystem sink

The on-demand backup route passes the stack name straight into
backupStackFiles, so resolve the backup directory against the backup root
and confirm containment with an inline startsWith check before the
mkdir/copy/write sinks, matching the barrier restoreStackFiles already
uses. The stack name is validated at the route and again by
resolveStackDir, so this is defense in depth that also closes a
static path-injection finding on the new call path.

backend/src/__tests__/scheduler-service.test.ts

@@ -195,6 +195,15 @@ import { SchedulerService } from '../services/SchedulerService';
 
 beforeEach(() => {
   vi.clearAllMocks();
+  // clearAllMocks only clears call history, not implementations, so restore the
+  // mocks that individual tests mutate (tier, variant, node lookup, proxy
+  // target) to their documented defaults. Without this a test that points
+  // getNode at a remote node or drops the tier leaks that state into every
+  // later test in the file.
+  mockGetTier.mockReturnValue('paid');
+  mockGetVariant.mockReturnValue('admiral');
+  mockGetNode.mockReturnValue({ id: 1, name: 'local', type: 'local', status: 'online' });
+  mockGetProxyTarget.mockReturnValue(null);
   // Default: the scan-policy gate allows. Individual tests override to a block.
   mockEnforcePolicyPreDeploy.mockResolvedValue({ ok: true, bypassed: false, violations: [] });
   (SchedulerService as any).instance = undefined;
@@ -620,7 +629,12 @@ describe('SchedulerService - executeUpdate', () => {
     expect(mockClearStackUpdateStatus).toHaveBeenCalledWith(1, 'web-app');
   });
 
-  it('runs the update without the atomic wrapper on the community tier', async () => {
+  it('does not run a scheduled update on the community tier', async () => {
+    // Scheduled tasks are paid-only at every entry point (the tick tier check and
+    // the manual-run route both require paid), and executeTask guards again, so a
+    // community licence never runs the update. Hub-driven updates to a community
+    // remote worker take a different path (the /auto-update/execute route, which
+    // derives atomicity from the proxy tier header) and are unaffected.
     mockGetTier.mockReturnValue('community');
     mockGetScheduledTask.mockReturnValue({
       id: 82,
@@ -633,21 +647,15 @@ describe('SchedulerService - executeUpdate', () => {
       created_by: 'admin',
       last_status: null,
     });
-    mockGetContainersByStack.mockResolvedValue([
-      { Id: 'c1', Image: 'nginx:latest' },
-    ]);
-    mockCheckImage.mockResolvedValue({ hasUpdate: true });
 
     const svc = SchedulerService.getInstance();
     await svc.triggerTask(82);
 
-    // Atomic backup/rollback is a paid capability: the auto-update path derives
-    // the flag from the licence, so a community instance updates without it.
-    expect(mockUpdateStack).toHaveBeenCalledWith('web-app', undefined, false);
-
-    // clearAllMocks does not reset return values; restore the suite default so
-    // later tier-agnostic tests keep the paid behavior they assume.
-    mockGetTier.mockReturnValue('paid');
+    expect(mockUpdateStack).not.toHaveBeenCalled();
+    expect(mockUpdateScheduledTaskRun).toHaveBeenCalledWith(
+      1,
+      expect.objectContaining({ status: 'failure' }),
+    );
   });
 
   it('skips when all images up to date', async () => {
@@ -1611,6 +1619,184 @@ describe('SchedulerService - lifecycle actions', () => {
   });
 });
 
+// ── Lifecycle remote proxy ──────────────────────────────────────────────
+
+describe('SchedulerService - lifecycle remote proxy', () => {
+  afterEach(() => {
+    vi.unstubAllGlobals();
+  });
+
+  const remoteHeaders = expect.objectContaining({
+    'Authorization': 'Bearer tkn',
+    'x-sencho-tier': 'paid',
+    'x-sencho-variant': 'admiral',
+  });
+
+  function stubRemote(okBody: unknown = { success: true }) {
+    mockGetNode.mockReturnValue({ id: 2, name: 'remote', type: 'remote', status: 'online' });
+    mockGetProxyTarget.mockReturnValue({ apiUrl: 'http://remote:1852', apiToken: 'tkn' });
+    const mockFetch = vi.fn().mockResolvedValue({ ok: true, json: async () => okBody });
+    vi.stubGlobal('fetch', mockFetch);
+    return mockFetch;
+  }
+
+  it('auto_stop proxies to the remote stop endpoint instead of running locally', async () => {
+    const fetchMock = stubRemote();
+    mockGetScheduledTask.mockReturnValue(makeLifecycleTask('auto_stop', { node_id: 2 }));
+    await SchedulerService.getInstance().triggerTask(300);
+    expect(fetchMock).toHaveBeenCalledWith(
+      'http://remote:1852/api/stacks/my-stack/stop',
+      expect.objectContaining({ method: 'POST', headers: remoteHeaders }),
+    );
+    expect(mockRunCommand).not.toHaveBeenCalled();
+    expect(mockUpdateScheduledTaskRun).toHaveBeenCalledWith(1, expect.objectContaining({ status: 'success' }));
+  });
+
+  it('auto_down proxies to the remote down endpoint', async () => {
+    const fetchMock = stubRemote();
+    mockGetScheduledTask.mockReturnValue(makeLifecycleTask('auto_down', { node_id: 2 }));
+    await SchedulerService.getInstance().triggerTask(300);
+    expect(fetchMock).toHaveBeenCalledWith(
+      'http://remote:1852/api/stacks/my-stack/down',
+      expect.objectContaining({ method: 'POST' }),
+    );
+    expect(mockRunCommand).not.toHaveBeenCalled();
+  });
+
+  it('auto_start proxies to the remote deploy endpoint and skips the hub policy gate', async () => {
+    const fetchMock = stubRemote();
+    mockGetScheduledTask.mockReturnValue(makeLifecycleTask('auto_start', { node_id: 2 }));
+    await SchedulerService.getInstance().triggerTask(300);
+    expect(fetchMock).toHaveBeenCalledWith(
+      'http://remote:1852/api/stacks/my-stack/deploy',
+      expect.objectContaining({ method: 'POST' }),
+    );
+    expect(mockDeployStack).not.toHaveBeenCalled();
+    expect(mockEnforcePolicyPreDeploy).not.toHaveBeenCalled();
+  });
+
+  it('auto_backup proxies to the remote backup endpoint', async () => {
+    const fetchMock = stubRemote();
+    mockGetScheduledTask.mockReturnValue(makeLifecycleTask('auto_backup', { node_id: 2 }));
+    await SchedulerService.getInstance().triggerTask(300);
+    expect(fetchMock).toHaveBeenCalledWith(
+      'http://remote:1852/api/stacks/my-stack/backup',
+      expect.objectContaining({ method: 'POST' }),
+    );
+    expect(mockBackupStackFiles).not.toHaveBeenCalled();
+  });
+
+  it('restart (all services) proxies to the remote restart endpoint', async () => {
+    const fetchMock = stubRemote();
+    mockGetScheduledTask.mockReturnValue(makeLifecycleTask('restart', { node_id: 2 }));
+    await SchedulerService.getInstance().triggerTask(300);
+    expect(fetchMock).toHaveBeenCalledWith(
+      'http://remote:1852/api/stacks/my-stack/restart',
+      expect.objectContaining({ method: 'POST' }),
+    );
+    expect(mockGetContainersByStack).not.toHaveBeenCalled();
+  });
+
+  it('restart with target_services fans out to per-service restart endpoints', async () => {
+    const fetchMock = stubRemote();
+    mockGetScheduledTask.mockReturnValue(
+      makeLifecycleTask('restart', { node_id: 2, target_services: JSON.stringify(['api', 'worker']) }),
+    );
+    await SchedulerService.getInstance().triggerTask(300);
+    expect(fetchMock).toHaveBeenCalledWith(
+      'http://remote:1852/api/stacks/my-stack/services/api/restart',
+      expect.objectContaining({ method: 'POST' }),
+    );
+    expect(fetchMock).toHaveBeenCalledWith(
+      'http://remote:1852/api/stacks/my-stack/services/worker/restart',
+      expect.objectContaining({ method: 'POST' }),
+    );
+    expect(fetchMock).toHaveBeenCalledTimes(2);
+  });
+
+  it('records failure when the remote node returns an error', async () => {
+    mockGetNode.mockReturnValue({ id: 2, name: 'remote', type: 'remote', status: 'online' });
+    mockGetProxyTarget.mockReturnValue({ apiUrl: 'http://remote:1852', apiToken: 'tkn' });
+    vi.stubGlobal('fetch', vi.fn().mockResolvedValue({
+      ok: false,
+      status: 503,
+      json: async () => ({ error: 'Docker daemon is unreachable' }),
+    }));
+    mockGetScheduledTask.mockReturnValue(makeLifecycleTask('auto_stop', { node_id: 2 }));
+    await SchedulerService.getInstance().triggerTask(300);
+    expect(mockUpdateScheduledTaskRun).toHaveBeenCalledWith(
+      1,
+      expect.objectContaining({ status: 'failure', error: expect.stringContaining('Docker daemon is unreachable') }),
+    );
+  });
+
+  it('falls back to the HTTP status when the remote error body is not JSON', async () => {
+    mockGetNode.mockReturnValue({ id: 2, name: 'remote', type: 'remote', status: 'online' });
+    mockGetProxyTarget.mockReturnValue({ apiUrl: 'http://remote:1852', apiToken: 'tkn' });
+    vi.stubGlobal('fetch', vi.fn().mockResolvedValue({
+      ok: false,
+      status: 502,
+      json: async () => { throw new Error('not json'); },
+    }));
+    mockGetScheduledTask.mockReturnValue(makeLifecycleTask('auto_down', { node_id: 2 }));
+    await SchedulerService.getInstance().triggerTask(300);
+    expect(mockUpdateScheduledTaskRun).toHaveBeenCalledWith(
+      1,
+      expect.objectContaining({ status: 'failure', error: expect.stringContaining('HTTP 502') }),
+    );
+  });
+
+  it('records failure when the remote node has no proxy credentials', async () => {
+    mockGetNode.mockReturnValue({ id: 2, name: 'remote', type: 'remote', status: 'online' });
+    // getProxyTarget defaults to null (no credentials configured).
+    const fetchMock = vi.fn();
+    vi.stubGlobal('fetch', fetchMock);
+    mockGetScheduledTask.mockReturnValue(makeLifecycleTask('auto_stop', { node_id: 2 }));
+    await SchedulerService.getInstance().triggerTask(300);
+    expect(fetchMock).not.toHaveBeenCalled();
+    expect(mockUpdateScheduledTaskRun).toHaveBeenCalledWith(
+      1,
+      expect.objectContaining({ status: 'failure', error: expect.stringContaining('not configured or missing API credentials') }),
+    );
+  });
+
+  it('restart fan-out fails fast and names already-restarted services', async () => {
+    mockGetNode.mockReturnValue({ id: 2, name: 'remote', type: 'remote', status: 'online' });
+    mockGetProxyTarget.mockReturnValue({ apiUrl: 'http://remote:1852', apiToken: 'tkn' });
+    const fetchMock = vi.fn()
+      .mockResolvedValueOnce({ ok: true, json: async () => ({ success: true }) })
+      .mockResolvedValueOnce({ ok: false, status: 500, json: async () => ({ error: 'boom' }) });
+    vi.stubGlobal('fetch', fetchMock);
+    mockGetScheduledTask.mockReturnValue(
+      makeLifecycleTask('restart', { node_id: 2, target_services: JSON.stringify(['api', 'worker', 'cache']) }),
+    );
+    await SchedulerService.getInstance().triggerTask(300);
+    // Third service is never reached after the second fails.
+    expect(fetchMock).toHaveBeenCalledTimes(2);
+    expect(mockUpdateScheduledTaskRun).toHaveBeenCalledWith(
+      1,
+      expect.objectContaining({ status: 'failure', error: expect.stringContaining('already restarted: api') }),
+    );
+  });
+});
+
+// ── Unpaid-tier guard in executeTask ────────────────────────────────────
+
+describe('SchedulerService - unpaid tier guard', () => {
+  it('records a failed run and does not execute when the licence is not paid', async () => {
+    mockGetTier.mockReturnValue('community');
+    mockGetScheduledTask.mockReturnValue(makeLifecycleTask('auto_stop'));
+    await SchedulerService.getInstance().triggerTask(300);
+    expect(mockRunCommand).not.toHaveBeenCalled();
+    // The skip is visible in run history rather than silently dropped.
+    expect(mockCreateScheduledTaskRun).toHaveBeenCalled();
+    expect(mockUpdateScheduledTaskRun).toHaveBeenCalledWith(
+      1,
+      expect.objectContaining({ status: 'failure', error: expect.stringContaining('paid licence') }),
+    );
+  });
+});
+
 // ── delete_after_run ────────────────────────────────────────────────────
 
 describe('SchedulerService - delete_after_run', () => {

backend/src/__tests__/stack-backup-route.test.ts

@@ -0,0 +1,127 @@
+/**
+ * Integration tests for POST /api/stacks/:stackName/backup, the on-demand
+ * stack-files backup trigger. Covers auth, role, paid gating, the success
+ * path, the missing-stack 404, name validation, and error propagation. The
+ * route exists so a scheduled auto_backup can run on a remote node through the
+ * proxy path, and so an operator can take a snapshot on demand.
+ */
+import { describe, it, expect, beforeAll, afterAll, beforeEach, vi } from 'vitest';
+import request from 'supertest';
+import bcrypt from 'bcrypt';
+import { setupTestDb, cleanupTestDb, loginAsTestAdmin } from './helpers/setupTestDb';
+
+const { mockBackupStackFiles, mockHasComposeFile } = vi.hoisted(() => ({
+  mockBackupStackFiles: vi.fn(),
+  mockHasComposeFile: vi.fn(),
+}));
+
+vi.mock('../services/FileSystemService', () => ({
+  FileSystemService: {
+    getInstance: () => ({
+      getBaseDir: () => '/tmp/compose',
+      hasComposeFile: mockHasComposeFile,
+      backupStackFiles: mockBackupStackFiles,
+      getStacks: vi.fn().mockResolvedValue([]),
+    }),
+  },
+}));
+
+let tmpDir: string;
+let app: import('express').Express;
+let DatabaseService: typeof import('../services/DatabaseService').DatabaseService;
+let adminCookie: string;
+let viewerCookie: string;
+let tierSpy: ReturnType<typeof vi.spyOn>;
+
+beforeAll(async () => {
+  tmpDir = await setupTestDb();
+  ({ DatabaseService } = await import('../services/DatabaseService'));
+
+  const { LicenseService } = await import('../services/LicenseService');
+  tierSpy = vi.spyOn(LicenseService.getInstance(), 'getTier').mockReturnValue('paid');
+  vi.spyOn(LicenseService.getInstance(), 'getVariant').mockReturnValue('admiral');
+
+  ({ app } = await import('../index'));
+  adminCookie = await loginAsTestAdmin(app);
+
+  const viewerHash = await bcrypt.hash('viewerpass', 1);
+  DatabaseService.getInstance().addUser({ username: 'backup-viewer', password_hash: viewerHash, role: 'viewer' });
+  const viewerRes = await request(app).post('/api/auth/login').send({ username: 'backup-viewer', password: 'viewerpass' });
+  const cookies = viewerRes.headers['set-cookie'] as string | string[];
+  viewerCookie = Array.isArray(cookies) ? cookies[0] : cookies;
+});
+
+afterAll(() => {
+  vi.restoreAllMocks();
+  cleanupTestDb(tmpDir);
+});
+
+beforeEach(() => {
+  mockBackupStackFiles.mockReset().mockResolvedValue(undefined);
+  mockHasComposeFile.mockReset().mockResolvedValue(true);
+  tierSpy.mockReturnValue('paid');
+});
+
+describe('POST /api/stacks/:stackName/backup', () => {
+  it('backs up the stack files and returns success', async () => {
+    const res = await request(app).post('/api/stacks/web/backup').set('Cookie', adminCookie);
+    expect(res.status).toBe(200);
+    expect(res.body.success).toBe(true);
+    expect(mockBackupStackFiles).toHaveBeenCalledWith('web');
+  });
+
+  it('returns 401 without an auth cookie', async () => {
+    const res = await request(app).post('/api/stacks/web/backup');
+    expect(res.status).toBe(401);
+    expect(mockBackupStackFiles).not.toHaveBeenCalled();
+  });
+
+  it('returns 403 for a viewer (no deploy permission)', async () => {
+    const res = await request(app).post('/api/stacks/web/backup').set('Cookie', viewerCookie);
+    expect(res.status).toBe(403);
+    expect(mockBackupStackFiles).not.toHaveBeenCalled();
+  });
+
+  it('returns 403 on the community tier', async () => {
+    tierSpy.mockReturnValue('community');
+    const res = await request(app).post('/api/stacks/web/backup').set('Cookie', adminCookie);
+    expect(res.status).toBe(403);
+    expect(mockBackupStackFiles).not.toHaveBeenCalled();
+  });
+
+  it('returns 404 when the stack does not exist', async () => {
+    mockHasComposeFile.mockResolvedValue(false);
+    const res = await request(app).post('/api/stacks/ghost/backup').set('Cookie', adminCookie);
+    expect(res.status).toBe(404);
+    expect(mockBackupStackFiles).not.toHaveBeenCalled();
+  });
+
+  it('returns 400 for an invalid stack name', async () => {
+    const res = await request(app).post('/api/stacks/..bad../backup').set('Cookie', adminCookie);
+    expect(res.status).toBe(400);
+    expect(mockBackupStackFiles).not.toHaveBeenCalled();
+  });
+
+  it('returns 500 when the backup operation fails', async () => {
+    mockBackupStackFiles.mockRejectedValue(new Error('disk full'));
+    const res = await request(app).post('/api/stacks/web/backup').set('Cookie', adminCookie);
+    expect(res.status).toBe(500);
+    expect(res.body.error).toContain('disk full');
+  });
+
+  it('returns 409 when the stack is busy with another operation', async () => {
+    // The backup shares the rollback slot, so it must not run while a deploy
+    // holds the stack-op lock for the same stack.
+    const { StackOpLockService } = await import('../services/StackOpLockService');
+    const localNodeId = DatabaseService.getInstance().getNodes().find(n => n.type === 'local')!.id;
+    StackOpLockService.getInstance().tryAcquire(localNodeId, 'web', 'deploy', 'someone');
+    try {
+      const res = await request(app).post('/api/stacks/web/backup').set('Cookie', adminCookie);
+      expect(res.status).toBe(409);
+      expect(res.body.code).toBe('stack_op_in_progress');
+      expect(mockBackupStackFiles).not.toHaveBeenCalled();
+    } finally {
+      StackOpLockService.getInstance().release(localNodeId, 'web');
+    }
+  });
+});