mirror of
https://github.com/Studio-Saelix/sencho.git
synced 2026-07-26 11:49:16 +00:00
fix(atomic-deploy): harden rollback locking, restore fidelity, and tier gating (#1247)
* fix(atomic-deploy): harden rollback locking, restore fidelity, and tier gating Hardens the Atomic Deployments feature found during a full audit: - Rollback now holds the per-stack lifecycle lock (deploy/update already do), so a rollback can no longer race a concurrent deploy on the same compose files. Adds a 'rollback' lifecycle action and releases the lock in finally. - restoreStackFiles is now a faithful revert: it removes managed compose/.env files added after the backup before copying, so a rollback no longer leaves a hybrid of old and new configuration. Scope is the protected file set only; user data is untouched. Aborts (rather than reporting success) if a stale managed file cannot be removed. - The scheduled image-update path derives the atomic flag from the licence tier instead of hardcoding it on, keeping the paid capability explicit at the call site (the scheduler is already paid-gated; this prevents silent drift). - The backup-metadata read (GET /stacks/:name/backup) now requires a paid licence, matching the rollback flow that is the only caller. - Manual rollback dispatches a success/failure notification, alongside the existing audit-log entry. Adds route integration tests (lock acquisition/release, tier 403, notifications, no-backup 404), filesystem tests for the faithful restore (orphan removal, variant switch, abort path, non-managed files preserved), a community-tier scheduler test, and a developer-mode logging matrix. Documents the restore semantics and reconciles the scheduled-update wording in the feature guide. * fix(atomic-deploy): assert restore target stays within the compose dir before unlink The orphan-removal step in restoreStackFiles joins the stack directory with a managed filename and unlinks it. The stack directory is already validated and contained by resolveStackDir (allowlist stack name + within-base assertion), but the containment guard was not reapplied to the joined target at the delete sink, so static analysis flagged the path as derived from user input. Reassert containment on the final path before unlinking, matching the barrier the other write/read helpers in this service already apply. No behavior change for valid stacks; defense-in-depth at the sink. * fix(atomic-deploy): inline the path-containment barrier at the restore unlink sink The wrapped within-base assertion was not recognized as a sanitizer by the static path-injection analysis, which still traced the stack name to the unlink sink. Replace it with the inline path.resolve + startsWith containment check the other write helpers in this service already use (the recognized barrier), kept in the same scope as the sink. Behavior is unchanged for valid stack names. * fix(atomic-deploy): clear stale managed files from the backup slot before writing The backup directory is reused across runs and was only ever added to, never cleared. A managed file removed from the stack since the last backup (e.g. a deleted .env or a switched compose variant) lingered in the slot, so a later rollback restored a file that did not exist immediately before the failed run, contradicting the faithful-revert guarantee. Clear the protected file set from the slot before copying the current files, with the same inline containment barrier the restore path uses. A clear failure is logged, not fatal, since it only risks a stale future rollback and should not block a valid deploy.
This commit is contained in:
Anso
@@ -0,0 +1,264 @@
|
||||
/**
|
||||
* Route-level tests for the Atomic Deployments hardening pass.
|
||||
*
|
||||
* Covers:
|
||||
* - The manual rollback route holds the per-stack lifecycle lock, so it cannot
|
||||
* race a concurrent deploy/update on the same stack (and vice versa).
|
||||
* - Rollback releases the lock after both success and failure.
|
||||
* - Rollback dispatches a notification on success/failure.
|
||||
* - The backup-metadata read and the rollback action are paid-gated, matching
|
||||
* the frontend, which only fetches backup state on a licensed instance.
|
||||
*/
|
||||
import { describe, it, expect, beforeAll, afterAll, beforeEach, afterEach, vi } from 'vitest';
|
||||
import request from 'supertest';
|
||||
import { setupTestDb, cleanupTestDb, loginAsTestAdmin } from './helpers/setupTestDb';
|
||||
|
||||
const {
|
||||
mockDeployStack,
|
||||
mockGetBackupInfo,
|
||||
mockRestoreStackFiles,
|
||||
} = vi.hoisted(() => ({
|
||||
mockDeployStack: vi.fn(),
|
||||
mockGetBackupInfo: vi.fn(),
|
||||
mockRestoreStackFiles: vi.fn(),
|
||||
}));
|
||||
|
||||
vi.mock('../services/ComposeService', async () => {
|
||||
const actual = await vi.importActual<typeof import('../services/ComposeService')>(
|
||||
'../services/ComposeService',
|
||||
);
|
||||
return {
|
||||
...actual,
|
||||
ComposeService: {
|
||||
...actual.ComposeService,
|
||||
getInstance: () => ({ deployStack: mockDeployStack }),
|
||||
},
|
||||
};
|
||||
});
|
||||
|
||||
vi.mock('../services/FileSystemService', () => ({
|
||||
FileSystemService: {
|
||||
getInstance: () => ({
|
||||
getBaseDir: () => '/tmp/compose',
|
||||
hasComposeFile: vi.fn().mockResolvedValue(true),
|
||||
getBackupInfo: mockGetBackupInfo,
|
||||
restoreStackFiles: mockRestoreStackFiles,
|
||||
}),
|
||||
},
|
||||
}));
|
||||
|
||||
let tmpDir: string;
|
||||
let app: import('express').Express;
|
||||
let authCookie: string;
|
||||
let LicenseService: typeof import('../services/LicenseService').LicenseService;
|
||||
|
||||
function mockTier(tier: 'paid' | 'community') {
|
||||
vi.spyOn(LicenseService.getInstance(), 'getTier').mockReturnValue(tier);
|
||||
}
|
||||
|
||||
interface Deferred<T> {
|
||||
promise: Promise<T>;
|
||||
resolve: (value: T) => void;
|
||||
}
|
||||
|
||||
function deferred<T>(): Deferred<T> {
|
||||
let resolve!: (value: T) => void;
|
||||
const promise = new Promise<T>((res) => { resolve = res; });
|
||||
return { promise, resolve };
|
||||
}
|
||||
|
||||
beforeAll(async () => {
|
||||
tmpDir = await setupTestDb();
|
||||
({ app } = await import('../index'));
|
||||
({ LicenseService } = await import('../services/LicenseService'));
|
||||
authCookie = await loginAsTestAdmin(app);
|
||||
});
|
||||
|
||||
afterAll(() => {
|
||||
vi.restoreAllMocks();
|
||||
cleanupTestDb(tmpDir);
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
mockDeployStack.mockReset();
|
||||
mockGetBackupInfo.mockReset().mockResolvedValue({ exists: true, timestamp: Date.now() });
|
||||
mockRestoreStackFiles.mockReset().mockResolvedValue(undefined);
|
||||
const { StackOpLockService } = await import('../services/StackOpLockService');
|
||||
StackOpLockService.resetForTests();
|
||||
});
|
||||
|
||||
afterEach(() => vi.restoreAllMocks());
|
||||
|
||||
describe('Rollback holds the stack lifecycle lock (H-1)', () => {
|
||||
it('blocks deploy while a rollback is in flight on the same stack', async () => {
|
||||
mockTier('paid');
|
||||
const gate = deferred<void>();
|
||||
mockDeployStack.mockImplementationOnce(() => gate.promise);
|
||||
|
||||
const rollback = request(app)
|
||||
.post('/api/stacks/web/rollback')
|
||||
.set('Cookie', authCookie)
|
||||
.then(r => r);
|
||||
await vi.waitFor(() => expect(mockDeployStack).toHaveBeenCalled());
|
||||
|
||||
const deploy = await request(app)
|
||||
.post('/api/stacks/web/deploy')
|
||||
.set('Cookie', authCookie)
|
||||
.send({ skip_scan: true });
|
||||
expect(deploy.status).toBe(409);
|
||||
expect(deploy.body.code).toBe('stack_op_in_progress');
|
||||
expect(deploy.body.inProgress.action).toBe('rollback');
|
||||
|
||||
gate.resolve();
|
||||
const rollbackRes = await rollback;
|
||||
expect(rollbackRes.status).toBe(200);
|
||||
});
|
||||
|
||||
it('returns 409 when a rollback lands while a deploy is in flight', async () => {
|
||||
mockTier('paid');
|
||||
const gate = deferred<void>();
|
||||
mockDeployStack.mockImplementationOnce(() => gate.promise);
|
||||
|
||||
const deploy = request(app)
|
||||
.post('/api/stacks/web/deploy')
|
||||
.set('Cookie', authCookie)
|
||||
.send({ skip_scan: true })
|
||||
.then(r => r);
|
||||
await vi.waitFor(() => expect(mockDeployStack).toHaveBeenCalled());
|
||||
|
||||
const rollback = await request(app)
|
||||
.post('/api/stacks/web/rollback')
|
||||
.set('Cookie', authCookie);
|
||||
expect(rollback.status).toBe(409);
|
||||
expect(rollback.body.inProgress.action).toBe('deploy');
|
||||
|
||||
gate.resolve();
|
||||
await deploy;
|
||||
});
|
||||
|
||||
it('releases the lock after a successful rollback', async () => {
|
||||
mockTier('paid');
|
||||
mockDeployStack.mockResolvedValue(undefined);
|
||||
|
||||
const first = await request(app).post('/api/stacks/web/rollback').set('Cookie', authCookie);
|
||||
expect(first.status).toBe(200);
|
||||
|
||||
const second = await request(app).post('/api/stacks/web/rollback').set('Cookie', authCookie);
|
||||
expect(second.status).toBe(200);
|
||||
});
|
||||
|
||||
it('releases the lock after a failed rollback', async () => {
|
||||
mockTier('paid');
|
||||
mockDeployStack.mockRejectedValueOnce(new Error('image pull failed'));
|
||||
const first = await request(app).post('/api/stacks/web/rollback').set('Cookie', authCookie);
|
||||
expect(first.status).toBe(500);
|
||||
|
||||
mockDeployStack.mockResolvedValueOnce(undefined);
|
||||
const second = await request(app).post('/api/stacks/web/rollback').set('Cookie', authCookie);
|
||||
expect(second.status).toBe(200);
|
||||
});
|
||||
});
|
||||
|
||||
describe('Rollback notifications (M-2)', () => {
|
||||
it('dispatches a success notification when a rollback completes', async () => {
|
||||
mockTier('paid');
|
||||
mockDeployStack.mockResolvedValue(undefined);
|
||||
const { NotificationService } = await import('../services/NotificationService');
|
||||
const spy = vi.spyOn(NotificationService.getInstance(), 'dispatchAlert').mockResolvedValue(undefined);
|
||||
|
||||
const res = await request(app).post('/api/stacks/web/rollback').set('Cookie', authCookie);
|
||||
expect(res.status).toBe(200);
|
||||
await vi.waitFor(() =>
|
||||
expect(spy).toHaveBeenCalledWith(
|
||||
'info',
|
||||
'deploy_success',
|
||||
expect.stringMatching(/rolled back/i),
|
||||
expect.objectContaining({ stackName: 'web', actor: expect.any(String) }),
|
||||
),
|
||||
);
|
||||
});
|
||||
|
||||
it('dispatches a failure notification when a rollback fails', async () => {
|
||||
mockTier('paid');
|
||||
mockDeployStack.mockRejectedValueOnce(new Error('restore failed'));
|
||||
const { NotificationService } = await import('../services/NotificationService');
|
||||
const spy = vi.spyOn(NotificationService.getInstance(), 'dispatchAlert').mockResolvedValue(undefined);
|
||||
|
||||
const res = await request(app).post('/api/stacks/web/rollback').set('Cookie', authCookie);
|
||||
expect(res.status).toBe(500);
|
||||
await vi.waitFor(() =>
|
||||
expect(spy).toHaveBeenCalledWith(
|
||||
'error',
|
||||
'deploy_failure',
|
||||
expect.any(String),
|
||||
expect.objectContaining({ stackName: 'web', actor: expect.any(String) }),
|
||||
),
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
describe('Rollback returns 404 when no backup exists', () => {
|
||||
it('does not touch compose when there is nothing to restore, and releases the lock', async () => {
|
||||
mockTier('paid');
|
||||
mockGetBackupInfo.mockResolvedValue({ exists: false, timestamp: null });
|
||||
|
||||
const res = await request(app).post('/api/stacks/web/rollback').set('Cookie', authCookie);
|
||||
expect(res.status).toBe(404);
|
||||
expect(mockRestoreStackFiles).not.toHaveBeenCalled();
|
||||
expect(mockDeployStack).not.toHaveBeenCalled();
|
||||
|
||||
// The 404 is an early return inside the try; the finally must still release
|
||||
// the lock so the stack is not wedged at 409 afterwards.
|
||||
mockGetBackupInfo.mockResolvedValue({ exists: true, timestamp: Date.now() });
|
||||
mockDeployStack.mockResolvedValue(undefined);
|
||||
const next = await request(app).post('/api/stacks/web/rollback').set('Cookie', authCookie);
|
||||
expect(next.status).toBe(200);
|
||||
});
|
||||
});
|
||||
|
||||
describe('Developer Mode logging matrix', () => {
|
||||
it('only emits rollback diagnostic logs when Developer Mode is enabled', async () => {
|
||||
mockTier('paid');
|
||||
mockDeployStack.mockResolvedValue(undefined);
|
||||
const { DatabaseService } = await import('../services/DatabaseService');
|
||||
const db = DatabaseService.getInstance();
|
||||
const logSpy = vi.spyOn(console, 'log').mockImplementation(() => {});
|
||||
|
||||
db.updateGlobalSetting('developer_mode', '0');
|
||||
const quiet = await request(app).post('/api/stacks/web/rollback').set('Cookie', authCookie);
|
||||
expect(quiet.status).toBe(200);
|
||||
expect(logSpy.mock.calls.flat().some(a => typeof a === 'string' && /Rollback initiated/i.test(a))).toBe(false);
|
||||
|
||||
db.updateGlobalSetting('developer_mode', '1');
|
||||
const noisy = await request(app).post('/api/stacks/web/rollback').set('Cookie', authCookie);
|
||||
expect(noisy.status).toBe(200);
|
||||
expect(logSpy.mock.calls.flat().some(a => typeof a === 'string' && /Rollback initiated/i.test(a))).toBe(true);
|
||||
|
||||
db.updateGlobalSetting('developer_mode', '0');
|
||||
});
|
||||
});
|
||||
|
||||
describe('Tier gating parity (M-3)', () => {
|
||||
it('rejects rollback on community with PAID_REQUIRED', async () => {
|
||||
mockTier('community');
|
||||
const res = await request(app).post('/api/stacks/web/rollback').set('Cookie', authCookie);
|
||||
expect(res.status).toBe(403);
|
||||
expect(res.body.code).toBe('PAID_REQUIRED');
|
||||
expect(mockDeployStack).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('rejects GET /backup on community with PAID_REQUIRED', async () => {
|
||||
mockTier('community');
|
||||
const res = await request(app).get('/api/stacks/web/backup').set('Cookie', authCookie);
|
||||
expect(res.status).toBe(403);
|
||||
expect(res.body.code).toBe('PAID_REQUIRED');
|
||||
});
|
||||
|
||||
it('returns backup metadata on paid', async () => {
|
||||
mockTier('paid');
|
||||
mockGetBackupInfo.mockResolvedValue({ exists: true, timestamp: 1700000000000 });
|
||||
const res = await request(app).get('/api/stacks/web/backup').set('Cookie', authCookie);
|
||||
expect(res.status).toBe(200);
|
||||
expect(res.body).toEqual({ exists: true, timestamp: 1700000000000 });
|
||||
});
|
||||
});
|
||||
@@ -126,4 +126,123 @@ describe('FileSystemService backup location', () => {
|
||||
const restored = await fsPromises.readFile(path.join(stackDir, 'compose.yaml'), 'utf-8');
|
||||
expect(restored).toBe('version: original\n');
|
||||
});
|
||||
|
||||
it('restoreStackFiles removes a managed file the backup does not contain', async () => {
|
||||
const stackName = 'noenv';
|
||||
const stackDir = path.join(composeDir, stackName);
|
||||
await fsPromises.mkdir(stackDir, { recursive: true });
|
||||
// Backup captures a stack that has no .env.
|
||||
await fsPromises.writeFile(path.join(stackDir, 'compose.yaml'), 'services: {}\n', 'utf-8');
|
||||
|
||||
const service = FileSystemService.getInstance();
|
||||
await service.backupStackFiles(stackName);
|
||||
|
||||
// A later deploy adds a .env the backup predates. A faithful revert must
|
||||
// remove it so the restored stack is not a hybrid of old + new config.
|
||||
await fsPromises.writeFile(path.join(stackDir, '.env'), 'SECRET=added-after-backup\n', 'utf-8');
|
||||
await service.restoreStackFiles(stackName);
|
||||
|
||||
await expect(fsPromises.access(path.join(stackDir, '.env'))).rejects.toMatchObject({ code: 'ENOENT' });
|
||||
await expect(fsPromises.access(path.join(stackDir, 'compose.yaml'))).resolves.toBeUndefined();
|
||||
});
|
||||
|
||||
it('restoreStackFiles reverts a compose-variant switch', async () => {
|
||||
const stackName = 'variant';
|
||||
const stackDir = path.join(composeDir, stackName);
|
||||
await fsPromises.mkdir(stackDir, { recursive: true });
|
||||
await fsPromises.writeFile(path.join(stackDir, 'compose.yaml'), 'name: original\n', 'utf-8');
|
||||
|
||||
const service = FileSystemService.getInstance();
|
||||
await service.backupStackFiles(stackName);
|
||||
|
||||
// The failed deploy renamed the compose variant: it removed compose.yaml and
|
||||
// wrote docker-compose.yml instead. Restore must undo both halves.
|
||||
await fsPromises.rm(path.join(stackDir, 'compose.yaml'));
|
||||
await fsPromises.writeFile(path.join(stackDir, 'docker-compose.yml'), 'name: broken\n', 'utf-8');
|
||||
await service.restoreStackFiles(stackName);
|
||||
|
||||
await expect(fsPromises.access(path.join(stackDir, 'docker-compose.yml'))).rejects.toMatchObject({ code: 'ENOENT' });
|
||||
const restored = await fsPromises.readFile(path.join(stackDir, 'compose.yaml'), 'utf-8');
|
||||
expect(restored).toBe('name: original\n');
|
||||
});
|
||||
|
||||
it('does not retain a managed file in the backup slot once the stack drops it', async () => {
|
||||
const stackName = 'slot';
|
||||
const stackDir = path.join(composeDir, stackName);
|
||||
await fsPromises.mkdir(stackDir, { recursive: true });
|
||||
await fsPromises.writeFile(path.join(stackDir, 'compose.yaml'), 'name: v1\n', 'utf-8');
|
||||
await fsPromises.writeFile(path.join(stackDir, '.env'), 'SECRET=v1\n', 'utf-8');
|
||||
|
||||
const service = FileSystemService.getInstance();
|
||||
// Backup #1: stack has compose.yaml + .env.
|
||||
await service.backupStackFiles(stackName);
|
||||
|
||||
// The stack drops the .env, then is backed up again. The reused slot must
|
||||
// not keep the stale .env from backup #1, or a later restore resurrects it.
|
||||
await fsPromises.rm(path.join(stackDir, '.env'));
|
||||
await service.backupStackFiles(stackName);
|
||||
|
||||
await fsPromises.writeFile(path.join(stackDir, 'compose.yaml'), 'name: mutated\n', 'utf-8');
|
||||
await service.restoreStackFiles(stackName);
|
||||
|
||||
await expect(fsPromises.access(path.join(stackDir, '.env'))).rejects.toMatchObject({ code: 'ENOENT' });
|
||||
const restored = await fsPromises.readFile(path.join(stackDir, 'compose.yaml'), 'utf-8');
|
||||
expect(restored).toBe('name: v1\n');
|
||||
});
|
||||
|
||||
it('restoreStackFiles removes multiple orphans in one restore', async () => {
|
||||
const stackName = 'multi';
|
||||
const stackDir = path.join(composeDir, stackName);
|
||||
await fsPromises.mkdir(stackDir, { recursive: true });
|
||||
await fsPromises.writeFile(path.join(stackDir, 'compose.yaml'), 'name: original\n', 'utf-8');
|
||||
|
||||
const service = FileSystemService.getInstance();
|
||||
await service.backupStackFiles(stackName);
|
||||
|
||||
// The failed deploy switched the compose variant AND added a .env at once.
|
||||
await fsPromises.rm(path.join(stackDir, 'compose.yaml'));
|
||||
await fsPromises.writeFile(path.join(stackDir, 'docker-compose.yml'), 'name: broken\n', 'utf-8');
|
||||
await fsPromises.writeFile(path.join(stackDir, '.env'), 'SECRET=x\n', 'utf-8');
|
||||
await service.restoreStackFiles(stackName);
|
||||
|
||||
await expect(fsPromises.access(path.join(stackDir, 'docker-compose.yml'))).rejects.toMatchObject({ code: 'ENOENT' });
|
||||
await expect(fsPromises.access(path.join(stackDir, '.env'))).rejects.toMatchObject({ code: 'ENOENT' });
|
||||
const restored = await fsPromises.readFile(path.join(stackDir, 'compose.yaml'), 'utf-8');
|
||||
expect(restored).toBe('name: original\n');
|
||||
});
|
||||
|
||||
it('restoreStackFiles aborts instead of leaving a hybrid when an orphan cannot be removed', async () => {
|
||||
const stackName = 'blocked';
|
||||
const stackDir = path.join(composeDir, stackName);
|
||||
await fsPromises.mkdir(stackDir, { recursive: true });
|
||||
await fsPromises.writeFile(path.join(stackDir, 'compose.yaml'), 'name: original\n', 'utf-8');
|
||||
|
||||
const service = FileSystemService.getInstance();
|
||||
await service.backupStackFiles(stackName);
|
||||
|
||||
// An orphan managed name that is a directory: unlink fails with a non-ENOENT
|
||||
// code (EPERM/EISDIR), standing in for the EACCES/EBUSY cases on a real
|
||||
// chowned bind mount. The restore must reject rather than report success
|
||||
// with a stale file still present.
|
||||
await fsPromises.mkdir(path.join(stackDir, 'docker-compose.yml'));
|
||||
await expect(service.restoreStackFiles(stackName)).rejects.toThrow(/Rollback aborted/i);
|
||||
});
|
||||
|
||||
it('restoreStackFiles leaves non-managed files untouched', async () => {
|
||||
const stackName = 'userdata';
|
||||
const stackDir = path.join(composeDir, stackName);
|
||||
await fsPromises.mkdir(stackDir, { recursive: true });
|
||||
await fsPromises.writeFile(path.join(stackDir, 'compose.yaml'), 'services: {}\n', 'utf-8');
|
||||
|
||||
const service = FileSystemService.getInstance();
|
||||
await service.backupStackFiles(stackName);
|
||||
|
||||
// A file Sencho does not manage (user notes, mounted config) must survive a
|
||||
// rollback: the cleanup is scoped to the protected compose/.env set only.
|
||||
await fsPromises.writeFile(path.join(stackDir, 'notes.txt'), 'keep me\n', 'utf-8');
|
||||
await service.restoreStackFiles(stackName);
|
||||
|
||||
const kept = await fsPromises.readFile(path.join(stackDir, 'notes.txt'), 'utf-8');
|
||||
expect(kept).toBe('keep me\n');
|
||||
});
|
||||
});
|
||||
|
||||
@@ -612,6 +612,36 @@ describe('SchedulerService - executeUpdate', () => {
|
||||
expect(mockClearStackUpdateStatus).toHaveBeenCalledWith(1, 'web-app');
|
||||
});
|
||||
|
||||
it('runs the update without the atomic wrapper on the community tier', async () => {
|
||||
mockGetTier.mockReturnValue('community');
|
||||
mockGetScheduledTask.mockReturnValue({
|
||||
id: 82,
|
||||
name: 'update-community',
|
||||
action: 'update',
|
||||
cron_expression: '0 4 * * *',
|
||||
enabled: true,
|
||||
target_id: 'web-app',
|
||||
node_id: 1,
|
||||
created_by: 'admin',
|
||||
last_status: null,
|
||||
});
|
||||
mockGetContainersByStack.mockResolvedValue([
|
||||
{ Id: 'c1', Image: 'nginx:latest' },
|
||||
]);
|
||||
mockCheckImage.mockResolvedValue({ hasUpdate: true });
|
||||
|
||||
const svc = SchedulerService.getInstance();
|
||||
await svc.triggerTask(82);
|
||||
|
||||
// Atomic backup/rollback is a paid capability: the auto-update path derives
|
||||
// the flag from the licence, so a community instance updates without it.
|
||||
expect(mockUpdateStack).toHaveBeenCalledWith('web-app', undefined, false);
|
||||
|
||||
// clearAllMocks does not reset return values; restore the suite default so
|
||||
// later tier-agnostic tests keep the paid behavior they assume.
|
||||
mockGetTier.mockReturnValue('paid');
|
||||
});
|
||||
|
||||
it('skips when all images up to date', async () => {
|
||||
mockGetScheduledTask.mockReturnValue({
|
||||
id: 81,
|
||||
|
||||
@@ -61,6 +61,7 @@ const STACK_OP_PRESENT_PARTICIPLE: Record<StackOpAction, string> = {
|
||||
stop: 'stopping',
|
||||
start: 'starting',
|
||||
update: 'updating',
|
||||
rollback: 'rolling back',
|
||||
};
|
||||
|
||||
function tryAcquireStackOpLock(
|
||||
@@ -1200,11 +1201,18 @@ stacksRouter.post('/:stackName/rollback', async (req: Request, res: Response) =>
|
||||
const stackName = req.params.stackName as string;
|
||||
if (!requirePermission(req, res, 'stack:deploy', 'stack', stackName)) return;
|
||||
if (!requirePaid(req, res)) return;
|
||||
// Rollback restores files and re-deploys, so it must hold the same per-stack
|
||||
// lock deploy/update use. Without it a rollback racing an in-flight deploy
|
||||
// would mutate the compose files and run a second `docker compose up` against
|
||||
// the same project. Lock held below: all early-returns stay inside the try so
|
||||
// finally fires.
|
||||
if (!tryAcquireStackOpLock(req, res, stackName, 'rollback')) return;
|
||||
try {
|
||||
const fsSvc = FileSystemService.getInstance(req.nodeId);
|
||||
const backupInfo = await fsSvc.getBackupInfo(stackName);
|
||||
if (!backupInfo.exists) {
|
||||
return res.status(404).json({ error: 'No backup available for this stack.' });
|
||||
res.status(404).json({ error: 'No backup available for this stack.' });
|
||||
return;
|
||||
}
|
||||
dlog(`[Stacks] Rollback initiated: ${sanitizeForLog(stackName)}`);
|
||||
await fsSvc.restoreStackFiles(stackName);
|
||||
@@ -1213,14 +1221,24 @@ stacksRouter.post('/:stackName/rollback', async (req: Request, res: Response) =>
|
||||
invalidateNodeCaches(req.nodeId);
|
||||
dlog(`[Stacks] Rollback completed: ${sanitizeForLog(stackName)}`);
|
||||
res.json({ message: 'Stack rolled back successfully.' });
|
||||
notifyActionSuccess('deploy_success', `${stackName} rolled back`, stackName, req.user?.username ?? 'system');
|
||||
} catch (error: unknown) {
|
||||
console.error('[Stacks] Rollback failed: %s', sanitizeForLog(stackName), error);
|
||||
const message = getErrorMessage(error, 'Rollback failed.');
|
||||
res.status(500).json({ error: message });
|
||||
notifyActionFailure('rollback', stackName, error, req.user?.username ?? 'system');
|
||||
if (!res.headersSent) {
|
||||
res.status(500).json({ error: message });
|
||||
}
|
||||
} finally {
|
||||
releaseStackOpLock(req, stackName);
|
||||
}
|
||||
});
|
||||
|
||||
stacksRouter.get('/:stackName/backup', async (req: Request, res: Response) => {
|
||||
// Backup metadata exists only to drive the paid-only Rollback affordance, so
|
||||
// the read is gated to paid to match the frontend, which only fetches it when
|
||||
// the instance is licensed.
|
||||
if (!requirePaid(req, res)) return;
|
||||
try {
|
||||
const stackName = req.params.stackName as string;
|
||||
const fsSvc = FileSystemService.getInstance(req.nodeId);
|
||||
|
||||
@@ -488,6 +488,28 @@ export class FileSystemService {
|
||||
const backupDir = this.getBackupDir(stackName);
|
||||
await fsPromises.mkdir(backupDir, { recursive: true });
|
||||
|
||||
// Clear stale managed files from the backup slot before writing the current
|
||||
// ones. The slot is reused across runs, so a managed file removed from the
|
||||
// stack since the last backup (e.g. a deleted .env or a switched compose
|
||||
// variant) would otherwise linger here and a later restore would resurrect
|
||||
// it, breaking the faithful-revert guarantee. Scope is the protected set
|
||||
// Sencho writes; .timestamp is rewritten below. Containment is re-checked at
|
||||
// the sink, rooted at the backup base, for the same reason restoreStackFiles
|
||||
// does it. A clear failure is logged but not fatal: it only risks a stale
|
||||
// future rollback, so it should not block an otherwise valid deploy.
|
||||
const backupRoot = path.resolve(getBackupBaseDir());
|
||||
for (const file of PROTECTED_STACK_FILES) {
|
||||
const stale = path.resolve(backupRoot, path.join(backupDir, file));
|
||||
if (!stale.startsWith(backupRoot + path.sep)) continue;
|
||||
try {
|
||||
await fsPromises.unlink(stale);
|
||||
} catch (e: unknown) {
|
||||
if ((e as NodeJS.ErrnoException)?.code !== 'ENOENT') {
|
||||
console.warn(`[FileSystemService] Could not clear stale backup ${file}:`, (e as Error).message);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Copy compose file
|
||||
const composeFiles = ['compose.yaml', 'compose.yml', 'docker-compose.yaml', 'docker-compose.yml'];
|
||||
for (const file of composeFiles) {
|
||||
@@ -527,11 +549,49 @@ export class FileSystemService {
|
||||
const backupDir = this.getBackupDir(stackName);
|
||||
|
||||
const items = await fsPromises.readdir(backupDir);
|
||||
const backedUp = new Set(items);
|
||||
|
||||
// Remove managed files the backup does not contain before copying, so a
|
||||
// rollback is a faithful revert rather than an additive overlay. If the
|
||||
// failed deploy switched compose variants (e.g. compose.yaml ->
|
||||
// docker-compose.yml) or added a .env the backup predates, leaving the new
|
||||
// file in place would re-deploy a hybrid of old and new configuration.
|
||||
// Scope is strictly PROTECTED_STACK_FILES (the same set Sencho backs up);
|
||||
// user data and bind-mounted content in the stack directory are untouched.
|
||||
// Canonical js/path-injection barrier: path.resolve(SAFE_ROOT, untrusted)
|
||||
// followed by a single startsWith check, both inline with the sink. stackDir
|
||||
// is already validated by resolveStackDir; this re-establishes containment at
|
||||
// the delete sink itself so static analysis sees the barrier.
|
||||
const baseResolved = path.resolve(this.baseDir);
|
||||
let removedOrphans = 0;
|
||||
for (const file of PROTECTED_STACK_FILES) {
|
||||
if (backedUp.has(file)) continue;
|
||||
const target = path.resolve(baseResolved, path.join(stackDir, file));
|
||||
if (!target.startsWith(baseResolved + path.sep)) {
|
||||
throw Object.assign(new Error('Path escapes compose directory'), { code: 'INVALID_PATH' });
|
||||
}
|
||||
try {
|
||||
await fsPromises.unlink(target);
|
||||
removedOrphans++;
|
||||
} catch (e: unknown) {
|
||||
const code = (e as NodeJS.ErrnoException)?.code;
|
||||
// ENOENT means the file is already absent, which is the desired end
|
||||
// state. Any other code (EACCES on a chowned bind mount, EBUSY on a
|
||||
// held file) means a managed file Sencho meant to remove is still on
|
||||
// disk: completing the copy below would leave a hybrid config while
|
||||
// reporting success. Abort so the caller surfaces a real failure and
|
||||
// preserves the backup for manual recovery.
|
||||
if (code !== 'ENOENT') {
|
||||
throw new Error(`Rollback aborted: could not remove stale ${file} (${code ?? 'unknown error'}); the restore would leave a mix of old and new configuration.`);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
for (const item of items) {
|
||||
if (item === '.timestamp') continue;
|
||||
await fsPromises.copyFile(path.join(backupDir, item), path.join(stackDir, item));
|
||||
}
|
||||
if (debug) console.debug(`[FileSystemService:debug] Restore completed in ${Date.now() - t0}ms`, { stackName, files: items.filter(i => i !== '.timestamp') });
|
||||
if (debug) console.debug(`[FileSystemService:debug] Restore completed in ${Date.now() - t0}ms`, { stackName, restored: items.filter(i => i !== '.timestamp').length, removedOrphans });
|
||||
}
|
||||
|
||||
async getBackupInfo(stackName: string): Promise<{ exists: boolean; timestamp: number | null }> {
|
||||
|
||||
@@ -719,7 +719,13 @@ export class SchedulerService {
|
||||
auditPath: `/api/scheduled-tasks/auto-update/${stackName}`,
|
||||
}),
|
||||
);
|
||||
await compose.updateStack(stackName, undefined, true);
|
||||
// Atomic backup/rollback is a paid capability. Every path that reaches
|
||||
// this method is already paid-gated (the scheduler tick and the manual
|
||||
// run route both require a paid licence), but the flag is resolved from
|
||||
// the licence here so the tier intent is explicit at the call site and
|
||||
// survives any future refactor that introduces another caller.
|
||||
const atomic = LicenseService.getInstance().getTier() === 'paid';
|
||||
await compose.updateStack(stackName, undefined, atomic);
|
||||
db.clearStackUpdateStatus(nodeId, stackName);
|
||||
|
||||
this.safeDispatch(
|
||||
|
||||
@@ -1,13 +1,14 @@
|
||||
/**
|
||||
* Tracks in-flight stack lifecycle operations (deploy, down, restart, stop,
|
||||
* start, update) per (nodeId, stackName). A second request to the same stack
|
||||
* while the first is still running returns 409 instead of racing the first.
|
||||
* start, update, rollback) per (nodeId, stackName). A second request to the
|
||||
* same stack while the first is still running returns 409 instead of racing
|
||||
* the first.
|
||||
*
|
||||
* State is intentionally process-local: a Sencho restart clears all locks,
|
||||
* which matches the lifecycle of any in-flight `docker compose` child process.
|
||||
*/
|
||||
|
||||
export type StackOpAction = 'deploy' | 'down' | 'restart' | 'stop' | 'start' | 'update';
|
||||
export type StackOpAction = 'deploy' | 'down' | 'restart' | 'stop' | 'start' | 'update' | 'rollback';
|
||||
|
||||
export interface StackOpLock {
|
||||
action: StackOpAction;
|
||||
|
||||
@@ -29,7 +29,7 @@ Atomic deployments wrap:
|
||||
- **Webhook** triggers for deploy and pull actions.
|
||||
- **Image auto-updates** triggered by an auto-update policy.
|
||||
|
||||
Scheduled tasks invoke `docker compose` without the atomic wrapper, so a deploy or update launched from a schedule runs without a backup or auto-rollback. If you need atomic safety on a recurring deploy, trigger it through a webhook on a cron rather than through Scheduled Tasks.
|
||||
A scheduled image-update task uses the same atomic wrapper as a manual update, so a recurring update still takes a backup and rolls back automatically when a container crashes. Scheduled lifecycle actions (start, stop, restart) change no stack configuration and run `docker compose` directly without a backup.
|
||||
|
||||
## Manual rollback
|
||||
|
||||
@@ -47,6 +47,8 @@ Backups live under `<DATA_DIR>/backups/<stack>/`, in the same writable volume Se
|
||||
|
||||
Each backup is a flat copy of the compose file Sencho found, plus `.env` if it exists, plus a `.timestamp` marker that records when the backup was taken. There is one backup slot per stack: every protected deploy or update overwrites the previous backup, so the **Rollback** menu always reverts to the configuration that was on disk immediately before the most recent run.
|
||||
|
||||
A restore is a faithful revert, not an overlay. Sencho replaces the compose file and `.env` with the backed-up copies and removes any compose variant or `.env` that was added after the backup was taken, so the stack returns to exactly the file set it had before the run. For example, if a deploy switched the stack from `compose.yaml` to `docker-compose.yml` or introduced a new `.env`, a rollback undoes both. Files Sencho does not manage are left untouched.
|
||||
|
||||
## Community Edition behavior
|
||||
|
||||
On Community Edition, Sencho runs the same `docker compose` commands without the atomic wrapper. There is no backup, no health probe, and no automatic rollback, and the **Rollback** menu entry is hidden. On a Skipper or Admiral license, atomic deployments are active immediately for every protected action; no configuration is required.
|
||||
|
||||
Reference in New Issue
Block a user