fix(scheduled-ops): run stack lifecycle schedules on remote nodes and harden run visibility (#1260)

* fix(scheduled-ops): run stack lifecycle schedules on remote nodes and harden run visibility

Stack lifecycle schedules (Restart, Stop, Take Down, Start, Backup Stack
Files) now run against whichever node the schedule targets, local or
remote. Each remote run proxies to that node's own stack-operation
endpoint, so a hub-managed schedule reaches the node that actually holds
the stack. Restart with a service subset restarts each selected service
and, if one fails, names the services already restarted so run history
reflects the stack's partial state. Auto-start on a remote node runs that
node's own pre-deploy scan-policy check against the images it holds.

Add POST /api/stacks/:name/backup to trigger an on-demand backup of a
stack's compose and env files (the same rollback snapshot a deploy
takes); it backs the remote backup schedule and is available to operators
on its own.

A scheduled task that reaches execution on an unpaid licence is now
skipped and written to run history as a failed run, so a manual trigger
that returned a queued response never silently disappears.

Test plan:
- Backend unit + integration: scheduler-service (remote proxy per action,
  per-service fan-out, auto-start policy delegation, remote-failure and
  no-credentials paths, unpaid-tier skip), stack-backup-route
  (auth/role/paid/404/400/500), scheduled-tasks-routes.
- Frontend component test for the schedules view (list, prefill, node
  filter, create payload).
- tsc and lint clean on both packages.

* fix(scheduled-ops): lock the stack-files backup route against concurrent stack ops

The stack-files backup writes the same slot the pre-deploy rollback
snapshot uses, so running it while a deploy, update, or rollback is in
flight on the same stack could overwrite the rollback point. The backup
route now takes the per-stack operation lock (as deploy/down/restart do)
and returns 409 when the stack is busy, keeping the rollback snapshot
intact. Adds the 'backup' action to the stack-op lock type and a busy
participle for the 409 message.

* fix(scheduled-ops): enforce backup-path containment inline at the filesystem sink

The on-demand backup route passes the stack name straight into
backupStackFiles, so resolve the backup directory against the backup root
and confirm containment with an inline startsWith check before the
mkdir/copy/write sinks, matching the barrier restoreStackFiles already
uses. The stack name is validated at the route and again by
resolveStackDir, so this is defense in depth that also closes a
static path-injection finding on the new call path.
This commit is contained in:
Anso
2026-05-31 17:47:34 -04:00
committed by GitHub
parent 5e66b54153
commit 6fc7f200a6
9 changed files with 618 additions and 24 deletions
+4 -1
View File
@@ -123,7 +123,10 @@ scheduledTasksRouter.get('/', (req: Request, res: Response): void => {
if (!requirePaid(req, res)) return;
try {
let tasks = DatabaseService.getInstance().getScheduledTasks();
// Split Auto-Update and Scheduled Operations into distinct views.
// The Scheduled Operations view manages every task type, so it lists all of
// them. `action` / `exclude_action` exist for the read-only consumers that
// want a slice: the Auto-Update readiness card and the sidebar next-run
// indicator both request `?action=update`.
const actionFilter = typeof req.query.action === 'string' ? req.query.action : undefined;
const excludeAction = typeof req.query.exclude_action === 'string' ? req.query.exclude_action : undefined;
if (actionFilter) {
+27
View File
@@ -62,6 +62,7 @@ const STACK_OP_PRESENT_PARTICIPLE: Record<StackOpAction, string> = {
start: 'starting',
update: 'updating',
rollback: 'rolling back',
backup: 'backing up',
};
function tryAcquireStackOpLock(
@@ -1251,6 +1252,32 @@ stacksRouter.get('/:stackName/backup', async (req: Request, res: Response) => {
}
});
stacksRouter.post('/:stackName/backup', async (req: Request, res: Response) => {
// Triggers a server-side backup of the stack's managed files: the same
// rollback snapshot a deploy takes. Exposed so a scheduled backup can run on
// a remote node through the proxy path, and so an operator can capture an
// on-demand snapshot. Paid-gated to match the rollback feature it feeds.
const stackName = req.params.stackName as string;
if (!requirePermission(req, res, 'stack:deploy', 'stack', stackName)) return;
if (!requirePaid(req, res)) return;
if (!(await requireStackExists(req.nodeId, stackName, res))) return;
// The backup slot is shared with the pre-deploy rollback snapshot, so hold the
// stack-op lock to keep a backup from interleaving with a concurrent
// deploy/update/rollback on the same stack. All early-returns stay inside the
// try so finally always releases.
if (!tryAcquireStackOpLock(req, res, stackName, 'backup')) return;
try {
await FileSystemService.getInstance(req.nodeId).backupStackFiles(stackName);
dlog(`[Stacks] Backup completed: ${sanitizeForLog(stackName)}`);
res.json({ success: true });
} catch (error: unknown) {
console.error('[Stacks] Backup failed: %s', sanitizeForLog(stackName), error);
res.status(500).json({ error: getErrorMessage(error, 'Failed to back up stack files') });
} finally {
releaseStackOpLock(req, stackName);
}
});
/**
* Returns the latest post-deploy scan attempt for this stack, or null if
* no scan has been attempted yet. Used by the editor UI to flag stacks