mirror of
https://github.com/Studio-Saelix/sencho.git
synced 2026-08-16 13:38:33 +00:00
feat(recovery): complete authored-project atomic rollback generations (#1819)
* feat(recovery): capture complete authored Compose project for atomic rollback Replace the root-compose-only backup slot with staged recovery generations that record the managed inventory, exact Compose invocation, and prior image identity, and wire the same engine through deploy, update, manual rollback, and Git apply. * fix(recovery): satisfy CodeQL path barriers and update-guard mock Inline resolve+startsWith checks at generation/inventory fs sinks and stub getCurrentStackUpdateRecovery in UpdateGuardService tests. * fix(recovery): drop unused FileSystemService import in generation store test * fix(recovery): harden authored-project rollback for upgrade and restore safety Preserve legacy UUID backup rows, restore Git deploy state with files, make multi-file restore recoverable, evaluate policy on the restored target, and fail closed when Git capture cannot cover an apply. * fix(recovery): unblock Git apply unit tests and CodeQL pre-restore TOCTOU Mock recovery capture in git-source-service tests after fail-closed apply capture, and re-resolve live paths immediately before pre-restore snapshot reads. * fix(recovery): fall back to authored inventory when Git manifesto is missing First Git apply captures before promote, so a missing managed-project manifesto must not block rollback capture when the live stack already has authored files. * fix(recovery): make authored-project rollback atomic across Git state Restore the managed-project manifesto with files, keep nullable Git identity on first-apply captures, persist Git side-state in restore intents for startup reconcile, compensate legacy materialize failures, and refuse directory collisions before mutation. * fix(recovery): satisfy CodeQL path and TOCTOU barriers on manifesto restore Add inline resolve barriers for manifesto read/clear sinks and remove the access-then-read race when restoring a generation manifesto snapshot. * fix(recovery): close third-audit rollback generation blockers Fail closed on incomplete Git inventory fallbacks, execute captured Compose invocation during recovery, refuse startup and mutations while restore intents remain unresolved, propagate legacy stale-delete failures, and add Docker-level exact prior-image coverage plus regression tests. * fix(recovery): mark acquired before handoff in prior-image Docker test Match the production updateStack CAS sequence so the exact prior-image integration test does not fail handoff from the captured phase. * fix(recovery): close fourth-audit rollback safety blockers Evaluate policy against held images, use index-based pre-restore snapshots, hold the shared stack lock across Git apply, replay Mesh and empty captured invocations exactly, restore POSIX modes with fail-closed sensitive permissions, keep case-sensitive paths, and link Git auto-deploy health gates. Add regression coverage for these cases. * test(recovery): fix mocks for health-gate link and authored compose args Add linkGateOrRetain to the Git apply recovery mock, and mock authoredComposeArgs so the case-collision inventory test is not masked by a missing getComposeDir stub. * fix(recovery): close fifth-audit rollback safety blockers Share git_apply locking for webhook auto-apply, fail closed on malformed recovery service records, refuse mixed-image capture, and require exact probe counts with hold-tag eligibility checks. * fix(recovery): close sixth-audit rollback safety blockers Preserve the legacy backup slot during generation capture, encrypt sensitive pre-restore snapshots, revert files on a failed health probe without committing Git, fail closed when an absent-file revert would delete a directory, skip Compose one-offs, route manual and scheduled backup through the current generation, and persist runtime image platform identity. * fix(recovery): close seventh-audit rollback safety blockers Fleet snapshot restore and restore-all now capture a recovery generation under the stack lock before any authored file write, including on remote nodes. * fix(recovery): keep pre-deploy generations during health-gate observe Link deploy recovery generations to the observing gate so backup cannot replace them mid-observe. Distinguish missing hold tags from probe failures, refuse generation release when services metadata is corrupt, classify mixed-replica and coverage refusals, and toast the backend rollback message. * fix(recovery): wrap webhook deploy case for eslint const bindings in an unbraced switch case trip no-case-declarations. Match the pull case block.
This commit is contained in:
Anso
@@ -0,0 +1,159 @@
|
||||
import { describe, it, expect, beforeEach, afterEach, vi } from 'vitest';
|
||||
import { applyFleetSnapshotFiles, selectFleetSnapshotApplyFiles, type FleetSnapshotApplyFile } from '../helpers/applyFleetSnapshotFiles';
|
||||
import { StackOpLockService } from '../services/StackOpLockService';
|
||||
|
||||
const mocks = vi.hoisted(() => ({
|
||||
hasComposeFile: vi.fn(),
|
||||
saveStackContent: vi.fn(),
|
||||
saveEnvContent: vi.fn(),
|
||||
getBaseDir: vi.fn(() => '/tmp/compose'),
|
||||
captureCurrentBackup: vi.fn(),
|
||||
invalidateNodeCaches: vi.fn(),
|
||||
}));
|
||||
|
||||
vi.mock('../services/FileSystemService', () => ({
|
||||
FileSystemService: {
|
||||
getInstance: () => ({
|
||||
getBaseDir: mocks.getBaseDir,
|
||||
hasComposeFile: mocks.hasComposeFile,
|
||||
saveStackContent: mocks.saveStackContent,
|
||||
saveEnvContent: mocks.saveEnvContent,
|
||||
}),
|
||||
},
|
||||
}));
|
||||
|
||||
vi.mock('../helpers/cacheInvalidation', () => ({
|
||||
invalidateNodeCaches: mocks.invalidateNodeCaches,
|
||||
}));
|
||||
|
||||
vi.mock('../services/StackUpdateRecoveryService', () => ({
|
||||
StackUpdateRecoveryService: {
|
||||
getInstance: () => ({
|
||||
captureCurrentBackup: mocks.captureCurrentBackup,
|
||||
}),
|
||||
},
|
||||
}));
|
||||
|
||||
const FILES: FleetSnapshotApplyFile[] = [
|
||||
{ filename: 'compose.yaml', content: 'services:\n snap: {}\n' },
|
||||
{ filename: '.env', content: 'SNAP=1\n' },
|
||||
];
|
||||
|
||||
function applyWeb(overrides: Partial<{
|
||||
stackName: string;
|
||||
files: FleetSnapshotApplyFile[];
|
||||
actor: string;
|
||||
}> = {}) {
|
||||
return applyFleetSnapshotFiles({
|
||||
nodeId: 1,
|
||||
stackName: 'web',
|
||||
files: FILES,
|
||||
actor: 'system:fleet-snapshot',
|
||||
...overrides,
|
||||
});
|
||||
}
|
||||
|
||||
describe('applyFleetSnapshotFiles', () => {
|
||||
beforeEach(() => {
|
||||
StackOpLockService.resetForTests();
|
||||
mocks.hasComposeFile.mockReset().mockResolvedValue(true);
|
||||
mocks.saveStackContent.mockReset().mockResolvedValue(undefined);
|
||||
mocks.saveEnvContent.mockReset().mockResolvedValue(undefined);
|
||||
mocks.captureCurrentBackup.mockReset().mockResolvedValue({ id: 'gen-pre' });
|
||||
mocks.invalidateNodeCaches.mockReset();
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
StackOpLockService.resetForTests();
|
||||
});
|
||||
|
||||
it('captures a recovery generation before writing when the stack already exists', async () => {
|
||||
const order: string[] = [];
|
||||
mocks.captureCurrentBackup.mockImplementation(async () => {
|
||||
order.push('capture');
|
||||
return { id: 'gen-pre' };
|
||||
});
|
||||
mocks.saveStackContent.mockImplementation(async () => {
|
||||
order.push('compose');
|
||||
});
|
||||
mocks.saveEnvContent.mockImplementation(async () => {
|
||||
order.push('env');
|
||||
});
|
||||
|
||||
const result = await applyWeb();
|
||||
|
||||
expect(result.capturedGenerationId).toBe('gen-pre');
|
||||
expect(order).toEqual(['capture', 'compose', 'env']);
|
||||
expect(mocks.invalidateNodeCaches).toHaveBeenCalledWith(1);
|
||||
expect(mocks.captureCurrentBackup).toHaveBeenCalledWith({
|
||||
nodeId: 1,
|
||||
stackName: 'web',
|
||||
createdBy: 'system:fleet-snapshot',
|
||||
});
|
||||
});
|
||||
|
||||
it('skips capture for a new stack with no compose file', async () => {
|
||||
mocks.hasComposeFile.mockResolvedValue(false);
|
||||
|
||||
const result = await applyWeb({ stackName: 'fresh' });
|
||||
|
||||
expect(result.capturedGenerationId).toBeNull();
|
||||
expect(mocks.captureCurrentBackup).not.toHaveBeenCalled();
|
||||
expect(mocks.saveStackContent).toHaveBeenCalledWith('fresh', FILES[0].content);
|
||||
expect(mocks.saveEnvContent).toHaveBeenCalledWith('fresh', FILES[1].content);
|
||||
});
|
||||
|
||||
it('aborts without writing when capture fails', async () => {
|
||||
mocks.captureCurrentBackup.mockRejectedValue(Object.assign(new Error('capture failed'), { code: 'CAPTURE_FAILED' }));
|
||||
|
||||
await expect(applyWeb()).rejects.toThrow('capture failed');
|
||||
|
||||
expect(mocks.saveStackContent).not.toHaveBeenCalled();
|
||||
expect(mocks.saveEnvContent).not.toHaveBeenCalled();
|
||||
expect(mocks.invalidateNodeCaches).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('refuses to mutate when another stack operation holds the lock', async () => {
|
||||
StackOpLockService.getInstance().tryAcquire(1, 'web', 'deploy', 'other');
|
||||
|
||||
await expect(applyWeb()).rejects.toMatchObject({ code: 'stack_op_in_progress' });
|
||||
|
||||
expect(mocks.captureCurrentBackup).not.toHaveBeenCalled();
|
||||
expect(mocks.saveStackContent).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('leaves the captured generation in place when a later file write fails', async () => {
|
||||
mocks.saveEnvContent.mockRejectedValue(new Error('disk full'));
|
||||
|
||||
await expect(applyWeb()).rejects.toThrow('disk full');
|
||||
|
||||
expect(mocks.captureCurrentBackup).toHaveBeenCalledTimes(1);
|
||||
expect(mocks.saveStackContent).toHaveBeenCalledWith('web', FILES[0].content);
|
||||
expect(mocks.invalidateNodeCaches).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('rejects an empty apply file list before locking', async () => {
|
||||
await expect(applyWeb({ files: [] })).rejects.toMatchObject({ code: 'INVALID_SNAPSHOT_FILES' });
|
||||
|
||||
expect(mocks.captureCurrentBackup).not.toHaveBeenCalled();
|
||||
expect(mocks.saveStackContent).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('rejects an invalid stack name before locking or writing', async () => {
|
||||
await expect(applyWeb({ stackName: '../escape' })).rejects.toMatchObject({ code: 'INVALID_STACK_NAME' });
|
||||
|
||||
expect(mocks.captureCurrentBackup).not.toHaveBeenCalled();
|
||||
expect(mocks.saveStackContent).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('drops snapshot filenames other than compose.yaml and .env', () => {
|
||||
expect(selectFleetSnapshotApplyFiles([
|
||||
{ filename: 'compose.yaml', content: 'a' },
|
||||
{ filename: 'notes.txt', content: 'nope' },
|
||||
{ filename: '.env', content: 'b' },
|
||||
])).toEqual([
|
||||
{ filename: 'compose.yaml', content: 'a' },
|
||||
{ filename: '.env', content: 'b' },
|
||||
]);
|
||||
});
|
||||
});
|
||||
@@ -83,7 +83,7 @@ afterAll(() => {
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
mockDeployStack.mockReset();
|
||||
mockDeployStack.mockReset().mockResolvedValue({ recoveryId: null });
|
||||
mockGetBackupInfo.mockReset().mockResolvedValue({ exists: true, timestamp: Date.now() });
|
||||
mockRestoreStackFiles.mockReset().mockResolvedValue(undefined);
|
||||
mockSnapshotStackFiles.mockReset().mockResolvedValue(async () => {});
|
||||
@@ -96,7 +96,7 @@ afterEach(() => vi.restoreAllMocks());
|
||||
describe('Rollback holds the stack lifecycle lock (H-1)', () => {
|
||||
it('blocks deploy while a rollback is in flight on the same stack', async () => {
|
||||
mockTier('paid');
|
||||
const gate = deferred<void>();
|
||||
const gate = deferred<{ recoveryId: string | null }>();
|
||||
mockDeployStack.mockImplementationOnce(() => gate.promise);
|
||||
|
||||
const rollback = request(app)
|
||||
@@ -113,14 +113,14 @@ describe('Rollback holds the stack lifecycle lock (H-1)', () => {
|
||||
expect(deploy.body.code).toBe('stack_op_in_progress');
|
||||
expect(deploy.body.inProgress.action).toBe('rollback');
|
||||
|
||||
gate.resolve();
|
||||
gate.resolve({ recoveryId: null });
|
||||
const rollbackRes = await rollback;
|
||||
expect(rollbackRes.status).toBe(200);
|
||||
});
|
||||
|
||||
it('returns 409 when a rollback lands while a deploy is in flight', async () => {
|
||||
mockTier('paid');
|
||||
const gate = deferred<void>();
|
||||
const gate = deferred<{ recoveryId: string | null }>();
|
||||
mockDeployStack.mockImplementationOnce(() => gate.promise);
|
||||
|
||||
const deploy = request(app)
|
||||
@@ -136,13 +136,13 @@ describe('Rollback holds the stack lifecycle lock (H-1)', () => {
|
||||
expect(rollback.status).toBe(409);
|
||||
expect(rollback.body.inProgress.action).toBe('deploy');
|
||||
|
||||
gate.resolve();
|
||||
gate.resolve({ recoveryId: null });
|
||||
await deploy;
|
||||
});
|
||||
|
||||
it('releases the lock after a successful rollback', async () => {
|
||||
mockTier('paid');
|
||||
mockDeployStack.mockResolvedValue(undefined);
|
||||
mockDeployStack.mockResolvedValue({ recoveryId: null });
|
||||
|
||||
const first = await request(app).post('/api/stacks/web/rollback').set('Cookie', authCookie);
|
||||
expect(first.status).toBe(200);
|
||||
@@ -157,7 +157,7 @@ describe('Rollback holds the stack lifecycle lock (H-1)', () => {
|
||||
const first = await request(app).post('/api/stacks/web/rollback').set('Cookie', authCookie);
|
||||
expect(first.status).toBe(500);
|
||||
|
||||
mockDeployStack.mockResolvedValueOnce(undefined);
|
||||
mockDeployStack.mockResolvedValueOnce({ recoveryId: null });
|
||||
const second = await request(app).post('/api/stacks/web/rollback').set('Cookie', authCookie);
|
||||
expect(second.status).toBe(200);
|
||||
});
|
||||
@@ -166,7 +166,7 @@ describe('Rollback holds the stack lifecycle lock (H-1)', () => {
|
||||
describe('Rollback notifications (M-2)', () => {
|
||||
it('dispatches a success notification when a rollback completes', async () => {
|
||||
mockTier('paid');
|
||||
mockDeployStack.mockResolvedValue(undefined);
|
||||
mockDeployStack.mockResolvedValue({ recoveryId: null });
|
||||
const { NotificationService } = await import('../services/NotificationService');
|
||||
const spy = vi.spyOn(NotificationService.getInstance(), 'dispatchAlert').mockResolvedValue({ persisted: true });
|
||||
|
||||
@@ -214,7 +214,7 @@ describe('Rollback returns 404 when no backup exists', () => {
|
||||
// The 404 is an early return inside the try; the finally must still release
|
||||
// the lock so the stack is not wedged at 409 afterwards.
|
||||
mockGetBackupInfo.mockResolvedValue({ exists: true, timestamp: Date.now() });
|
||||
mockDeployStack.mockResolvedValue(undefined);
|
||||
mockDeployStack.mockResolvedValue({ recoveryId: null });
|
||||
const next = await request(app).post('/api/stacks/web/rollback').set('Cookie', authCookie);
|
||||
expect(next.status).toBe(200);
|
||||
});
|
||||
@@ -223,7 +223,7 @@ describe('Rollback returns 404 when no backup exists', () => {
|
||||
describe('Developer Mode logging matrix', () => {
|
||||
it('only emits rollback diagnostic logs when Developer Mode is enabled', async () => {
|
||||
mockTier('paid');
|
||||
mockDeployStack.mockResolvedValue(undefined);
|
||||
mockDeployStack.mockResolvedValue({ recoveryId: null });
|
||||
const { DatabaseService } = await import('../services/DatabaseService');
|
||||
const db = DatabaseService.getInstance();
|
||||
const logSpy = vi.spyOn(console, 'log').mockImplementation(() => {});
|
||||
@@ -246,7 +246,7 @@ describe('Deploy safety is available on every tier', () => {
|
||||
it('allows rollback on community', async () => {
|
||||
mockTier('community');
|
||||
mockGetBackupInfo.mockResolvedValue({ exists: true, timestamp: 1700000000000 });
|
||||
mockDeployStack.mockResolvedValue(undefined);
|
||||
mockDeployStack.mockResolvedValue({ recoveryId: null });
|
||||
const res = await request(app).post('/api/stacks/web/rollback').set('Cookie', authCookie);
|
||||
expect(res.status).toBe(200);
|
||||
expect(mockDeployStack).toHaveBeenCalled();
|
||||
@@ -257,7 +257,7 @@ describe('Deploy safety is available on every tier', () => {
|
||||
mockGetBackupInfo.mockResolvedValue({ exists: true, timestamp: 1700000000000 });
|
||||
const res = await request(app).get('/api/stacks/web/backup').set('Cookie', authCookie);
|
||||
expect(res.status).toBe(200);
|
||||
expect(res.body).toEqual({ exists: true, timestamp: 1700000000000 });
|
||||
expect(res.body).toMatchObject({ exists: true, timestamp: 1700000000000 });
|
||||
});
|
||||
|
||||
it('returns backup metadata on paid', async () => {
|
||||
@@ -265,6 +265,6 @@ describe('Deploy safety is available on every tier', () => {
|
||||
mockGetBackupInfo.mockResolvedValue({ exists: true, timestamp: 1700000000000 });
|
||||
const res = await request(app).get('/api/stacks/web/backup').set('Cookie', authCookie);
|
||||
expect(res.status).toBe(200);
|
||||
expect(res.body).toEqual({ exists: true, timestamp: 1700000000000 });
|
||||
expect(res.body).toMatchObject({ exists: true, timestamp: 1700000000000 });
|
||||
});
|
||||
});
|
||||
|
||||
@@ -75,7 +75,7 @@ describe('Blueprint compose apply (real filesystem)', () => {
|
||||
const composeContent = 'services:\n web:\n image: traefik:v3\n';
|
||||
const markerContent = JSON.stringify({ blueprintId: 1, revision: 1, lastApplied: Date.now() }, null, 2);
|
||||
|
||||
const deploySpy = vi.spyOn(ComposeService.prototype, 'deployStack').mockResolvedValue(undefined);
|
||||
const deploySpy = vi.spyOn(ComposeService.prototype, 'deployStack').mockResolvedValue({ recoveryId: null });
|
||||
|
||||
const outcome = await BlueprintService.getInstance().applyLocalUnderLock(
|
||||
nodeId,
|
||||
@@ -120,6 +120,7 @@ describe('Blueprint compose apply (real filesystem)', () => {
|
||||
path.join(stackDir, 'docker-compose.yaml'),
|
||||
path.join(stackDir, 'docker-compose.yml'),
|
||||
);
|
||||
return { recoveryId: null };
|
||||
});
|
||||
|
||||
const outcome = await BlueprintService.getInstance().applyLocalUnderLock(
|
||||
@@ -197,7 +198,7 @@ describe('Blueprint compose apply (real filesystem)', () => {
|
||||
const original = 'services:\n mine:\n image: nginx:alpine\n';
|
||||
await fsPromises.writeFile(path.join(stackDir, 'compose.yaml'), original);
|
||||
|
||||
const deploySpy = vi.spyOn(ComposeService.prototype, 'deployStack').mockResolvedValue(undefined);
|
||||
const deploySpy = vi.spyOn(ComposeService.prototype, 'deployStack').mockResolvedValue({ recoveryId: null });
|
||||
|
||||
await expect(
|
||||
BlueprintService.getInstance().applyLocalUnderLock(
|
||||
|
||||
@@ -410,7 +410,7 @@ describe('BlueprintService per-stack lock', () => {
|
||||
vi.spyOn(FileSystemService.prototype, 'createStack').mockResolvedValue(undefined);
|
||||
const writeSpy = vi.spyOn(FileSystemService.prototype, 'writeStackFile').mockResolvedValue(undefined);
|
||||
const cleanupSpy = vi.spyOn(FileSystemService.prototype, 'removeAlternateRootComposeFiles').mockResolvedValue(undefined);
|
||||
const deploySpy = vi.spyOn(ComposeService.prototype, 'deployStack').mockResolvedValue(undefined);
|
||||
const deploySpy = vi.spyOn(ComposeService.prototype, 'deployStack').mockResolvedValue({ recoveryId: null });
|
||||
|
||||
const outcome = await BlueprintService.getInstance().deployToNode(bp, node);
|
||||
|
||||
|
||||
@@ -49,7 +49,7 @@ beforeAll(async () => {
|
||||
|
||||
const { ComposeService } = await import('../services/ComposeService');
|
||||
listImagesSpy = vi.spyOn(ComposeService.prototype, 'listStackImages').mockResolvedValue(['nginx:bad']);
|
||||
deploySpy = vi.spyOn(ComposeService.prototype, 'deployStack').mockResolvedValue(undefined);
|
||||
deploySpy = vi.spyOn(ComposeService.prototype, 'deployStack').mockResolvedValue({ recoveryId: null });
|
||||
|
||||
const TrivyService = (await import('../services/TrivyService')).default;
|
||||
const trivy = TrivyService.getInstance();
|
||||
|
||||
@@ -0,0 +1,112 @@
|
||||
/**
|
||||
* Generation capture must not rewrite the legacy single-slot backup. A later
|
||||
* capture failure has to leave a pre-migration recovery point byte-identical.
|
||||
*/
|
||||
import { describe, it, expect, beforeEach, afterEach, vi } from 'vitest';
|
||||
import path from 'path';
|
||||
import os from 'os';
|
||||
import { promises as fsPromises } from 'fs';
|
||||
|
||||
const mockState = { composeDir: '', composeDirs: new Map<number, string>() };
|
||||
|
||||
vi.mock('../services/NodeRegistry', () => ({
|
||||
NodeRegistry: {
|
||||
getInstance: () => ({
|
||||
getComposeDir: (nodeId?: number) => mockState.composeDirs.get(nodeId ?? 1) ?? mockState.composeDir,
|
||||
getDefaultNodeId: () => 1,
|
||||
}),
|
||||
},
|
||||
}));
|
||||
|
||||
vi.mock('../utils/debug', () => ({
|
||||
isDebugEnabled: () => false,
|
||||
}));
|
||||
|
||||
vi.mock('../services/rollbackInventory', () => ({
|
||||
resolveRollbackInventory: vi.fn(),
|
||||
}));
|
||||
|
||||
vi.mock('../services/DatabaseService', () => ({
|
||||
DatabaseService: {
|
||||
getInstance: () => ({
|
||||
getStackProjectEnvFiles: () => [],
|
||||
}),
|
||||
},
|
||||
}));
|
||||
|
||||
import { FileSystemService } from '../services/FileSystemService';
|
||||
import { RollbackGenerationStore } from '../services/RollbackGenerationStore';
|
||||
import { resolveRollbackInventory } from '../services/rollbackInventory';
|
||||
import { resolveComposeProjectContext } from '../services/composeProjectContext';
|
||||
|
||||
describe('composeProjectContext backupFromContext', () => {
|
||||
let composeDir: string;
|
||||
let dataDir: string;
|
||||
let originalDataDir: string | undefined;
|
||||
|
||||
beforeEach(async () => {
|
||||
composeDir = await fsPromises.mkdtemp(path.join(os.tmpdir(), 'sencho-compose-'));
|
||||
dataDir = await fsPromises.mkdtemp(path.join(os.tmpdir(), 'sencho-data-'));
|
||||
mockState.composeDir = composeDir;
|
||||
mockState.composeDirs = new Map([[1, composeDir]]);
|
||||
originalDataDir = process.env.DATA_DIR;
|
||||
process.env.DATA_DIR = dataDir;
|
||||
});
|
||||
|
||||
afterEach(async () => {
|
||||
if (originalDataDir === undefined) delete process.env.DATA_DIR;
|
||||
else process.env.DATA_DIR = originalDataDir;
|
||||
await fsPromises.rm(composeDir, { recursive: true, force: true });
|
||||
await fsPromises.rm(dataDir, { recursive: true, force: true });
|
||||
vi.restoreAllMocks();
|
||||
});
|
||||
|
||||
it('preserves a seeded legacy backup when generation capture fails', async () => {
|
||||
const stackName = 'legacy-slot';
|
||||
const stackDir = path.join(composeDir, stackName);
|
||||
await fsPromises.mkdir(stackDir, { recursive: true });
|
||||
await fsPromises.writeFile(path.join(stackDir, 'compose.yaml'), 'ORIGINAL\n', 'utf8');
|
||||
|
||||
const fsSvc = FileSystemService.getInstance(1);
|
||||
await fsSvc.backupStackFiles(stackName);
|
||||
const backupPath = path.join(dataDir, 'backups', '1', stackName, 'compose.yaml');
|
||||
const originalBackup = await fsPromises.readFile(backupPath);
|
||||
|
||||
await fsPromises.writeFile(path.join(stackDir, 'compose.yaml'), 'MUTATED\n', 'utf8');
|
||||
|
||||
vi.mocked(resolveRollbackInventory).mockResolvedValue({
|
||||
entries: [{
|
||||
relativePath: 'compose.yaml',
|
||||
dependencyKind: 'compose-root',
|
||||
provenance: 'authored',
|
||||
sensitivity: 'low',
|
||||
absolutePath: path.join(stackDir, 'compose.yaml'),
|
||||
}],
|
||||
invocation: {
|
||||
composeArgsPrefix: [],
|
||||
projectDirectory: null,
|
||||
projectName: stackName,
|
||||
explicitComposeFiles: ['compose.yaml'],
|
||||
meshEnabled: false,
|
||||
meshOverrideRelativePath: null,
|
||||
},
|
||||
git: null,
|
||||
appliedDeploySpec: null,
|
||||
lastAppliedContentHash: null,
|
||||
manifestState: null,
|
||||
manifestGeneration: null,
|
||||
exactCoverage: true,
|
||||
coverageRefusal: null,
|
||||
});
|
||||
vi.spyOn(RollbackGenerationStore, 'captureGeneration').mockRejectedValue(
|
||||
new Error('injected capture failure'),
|
||||
);
|
||||
|
||||
const ctx = await resolveComposeProjectContext(1, stackName);
|
||||
await expect(ctx.backupFromContext('update')).rejects.toThrow(/injected capture failure/);
|
||||
|
||||
expect(await fsPromises.readFile(backupPath)).toEqual(originalBackup);
|
||||
await fsSvc.restoreStackFiles(stackName);
|
||||
expect(await fsPromises.readFile(path.join(stackDir, 'compose.yaml'), 'utf8')).toBe('ORIGINAL\n');
|
||||
});
|
||||
});
|
||||
@@ -735,7 +735,7 @@ describe('ComposeService - deployStack', () => {
|
||||
const promise = ComposeService.getInstance(1).deployStack('my-stack');
|
||||
await vi.advanceTimersByTimeAsync(3100);
|
||||
|
||||
await expect(promise).resolves.toBeUndefined();
|
||||
await expect(promise).resolves.toEqual({ recoveryId: null });
|
||||
expect(mockGetLegacyOrphanContainersByStack).toHaveBeenCalledWith('my-stack');
|
||||
});
|
||||
|
||||
@@ -829,7 +829,7 @@ describe('ComposeService - deployStack', () => {
|
||||
);
|
||||
});
|
||||
|
||||
it('creates backup when atomic=true', async () => {
|
||||
it('captures recovery generation when atomic=true', async () => {
|
||||
setupAutoCloseSpawn();
|
||||
mockListContainers.mockResolvedValue([]);
|
||||
|
||||
@@ -837,9 +837,16 @@ describe('ComposeService - deployStack', () => {
|
||||
const promise = svc.deployStack('my-stack', undefined, true);
|
||||
|
||||
await vi.advanceTimersByTimeAsync(3100);
|
||||
await promise;
|
||||
const result = await promise;
|
||||
|
||||
expect(mockBackupStackFiles).toHaveBeenCalledWith('my-stack');
|
||||
expect(result).toEqual({ recoveryId: 'recovery-1' });
|
||||
expect(mockCaptureCandidate).toHaveBeenCalledWith(expect.objectContaining({
|
||||
stackName: 'my-stack',
|
||||
operationKind: 'deployment',
|
||||
createdBy: 'atomic-deploy',
|
||||
}));
|
||||
expect(mockHandoff).toHaveBeenCalled();
|
||||
expect(mockMarkImmediateVerified).toHaveBeenCalledWith('recovery-1');
|
||||
});
|
||||
|
||||
it('blocks deploy before backup when missing external networks need a prompt', async () => {
|
||||
@@ -861,7 +868,7 @@ describe('ComposeService - deployStack', () => {
|
||||
|
||||
const svc = ComposeService.getInstance(1);
|
||||
await expect(svc.deployStack('my-stack', undefined, true)).rejects.toBeInstanceOf(MissingExternalNetworksError);
|
||||
expect(mockBackupStackFiles).not.toHaveBeenCalled();
|
||||
expect(mockCaptureCandidate).not.toHaveBeenCalled();
|
||||
expect(mockSpawn).not.toHaveBeenCalled();
|
||||
expect(mockAddNotificationHistory).not.toHaveBeenCalled();
|
||||
});
|
||||
@@ -913,7 +920,7 @@ describe('ComposeService - deployStack', () => {
|
||||
stack_name: 'my-stack',
|
||||
}),
|
||||
);
|
||||
expect(mockBackupStackFiles).toHaveBeenCalled();
|
||||
expect(mockCaptureCandidate).toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('does not wrap a missing-external prompt in ComposeRollbackError when atomic', async () => {
|
||||
@@ -938,14 +945,12 @@ describe('ComposeService - deployStack', () => {
|
||||
expect(error?.name).toBe('MissingExternalNetworksError');
|
||||
});
|
||||
|
||||
it('aborts atomic deploy before docker side effects when backup fails', async () => {
|
||||
mockBackupStackFiles.mockRejectedValueOnce(new Error('disk full'));
|
||||
it('aborts atomic deploy before docker side effects when capture fails', async () => {
|
||||
mockCaptureCandidate.mockRejectedValueOnce(new Error('disk full'));
|
||||
|
||||
const svc = ComposeService.getInstance(1);
|
||||
|
||||
await expect(svc.deployStack('my-stack', undefined, true)).rejects.toThrow(
|
||||
'Atomic deployment backup failed',
|
||||
);
|
||||
await expect(svc.deployStack('my-stack', undefined, true)).rejects.toThrow('disk full');
|
||||
expect(mockSpawn).not.toHaveBeenCalled();
|
||||
expect(mockGetLegacyOrphanContainersByStack).not.toHaveBeenCalled();
|
||||
});
|
||||
@@ -990,7 +995,7 @@ describe('ComposeService - deployStack', () => {
|
||||
expect(error).not.toBeNull();
|
||||
expect(error!.message).toContain('CONTAINER_CRASHED');
|
||||
expect(getComposeRollbackInfo(error)).toEqual({ attempted: true, rolledBack: true });
|
||||
expect(mockRestoreStackFiles).toHaveBeenCalledWith('my-stack');
|
||||
expect(mockCompensateWithCandidate).toHaveBeenCalledWith('recovery-1', expect.any(Function));
|
||||
});
|
||||
|
||||
it('reports rollback failure when atomic restore fails', async () => {
|
||||
@@ -1001,7 +1006,7 @@ describe('ComposeService - deployStack', () => {
|
||||
Labels: { 'com.docker.compose.project': 'my-stack' },
|
||||
}]);
|
||||
mockContainerInspect.mockResolvedValue({ State: { ExitCode: 1 } });
|
||||
mockRestoreStackFiles.mockRejectedValueOnce(new Error('restore denied'));
|
||||
mockCompensateWithCandidate.mockResolvedValueOnce(false);
|
||||
|
||||
const svc = ComposeService.getInstance(1);
|
||||
const result = svc.deployStack('my-stack', undefined, true).then(() => null, (e: Error) => e);
|
||||
@@ -1012,6 +1017,28 @@ describe('ComposeService - deployStack', () => {
|
||||
expect(getComposeRollbackInfo(error)).toEqual({ attempted: true, rolledBack: false });
|
||||
});
|
||||
|
||||
it('wraps a missing hold tag as rolledBack=false without replacing the original error', async () => {
|
||||
setupAutoCloseSpawn();
|
||||
mockListContainers.mockResolvedValue([{
|
||||
Id: 'crashed-c1',
|
||||
State: 'exited',
|
||||
Labels: { 'com.docker.compose.project': 'my-stack' },
|
||||
}]);
|
||||
mockContainerInspect.mockResolvedValue({ State: { ExitCode: 1 } });
|
||||
mockCompensateWithCandidate.mockRejectedValueOnce(
|
||||
Object.assign(new Error('Held recovery image is missing'), { code: 'HELD_IMAGE_MISSING' }),
|
||||
);
|
||||
|
||||
const svc = ComposeService.getInstance(1);
|
||||
const result = svc.deployStack('my-stack', undefined, true).then(() => null, (e: Error) => e);
|
||||
|
||||
await vi.runAllTimersAsync();
|
||||
const error = await result;
|
||||
expect(error).not.toBeNull();
|
||||
expect(error!.message).toContain('CONTAINER_CRASHED');
|
||||
expect(getComposeRollbackInfo(error)).toEqual({ attempted: true, rolledBack: false });
|
||||
});
|
||||
|
||||
it('does not roll back when atomic=false', async () => {
|
||||
setupAutoCloseSpawn();
|
||||
mockListContainers.mockResolvedValue([{
|
||||
|
||||
@@ -50,6 +50,8 @@ describe('DeployedStackDeletionService ready transaction', () => {
|
||||
phase: 'immediate_verified',
|
||||
is_current: 1,
|
||||
backup_slot_id: null,
|
||||
content_path: null,
|
||||
operation_kind: null,
|
||||
override_path: null,
|
||||
services_json: '[]',
|
||||
health_gate_id: null,
|
||||
|
||||
@@ -0,0 +1,150 @@
|
||||
/**
|
||||
* Docker-backed: recovery override must recreate containers whose inspected
|
||||
* Image ID equals the captured prior image ID (moving-tag case).
|
||||
* Skipped when Docker is unavailable.
|
||||
*/
|
||||
import { describe, it, expect, beforeAll, afterAll } from 'vitest';
|
||||
import { execFileSync } from 'child_process';
|
||||
import fs from 'fs';
|
||||
import path from 'path';
|
||||
import { setupTestDb, cleanupTestDb } from '../helpers/setupTestDb';
|
||||
|
||||
function dockerAvailable(): boolean {
|
||||
try {
|
||||
execFileSync('docker', ['info'], {
|
||||
stdio: 'ignore',
|
||||
timeout: 8_000,
|
||||
windowsHide: true,
|
||||
});
|
||||
return true;
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
const hasDocker = dockerAvailable();
|
||||
const STACK = 'exactimg';
|
||||
|
||||
function compose(args: string[], cwd: string): string {
|
||||
return execFileSync('docker', ['compose', '-p', STACK, ...args], {
|
||||
cwd,
|
||||
encoding: 'utf8',
|
||||
stdio: ['ignore', 'pipe', 'pipe'],
|
||||
});
|
||||
}
|
||||
|
||||
function inspectImageId(containerId: string): string {
|
||||
const raw = execFileSync('docker', ['inspect', containerId, '--format', '{{.Image}}'], {
|
||||
encoding: 'utf8',
|
||||
}).trim();
|
||||
expect(raw.length).toBeGreaterThan(0);
|
||||
return raw;
|
||||
}
|
||||
|
||||
describe.skipIf(!hasDocker)('exact prior-image rollback after post-handoff failure', () => {
|
||||
let tmpDir: string;
|
||||
let composeDir: string;
|
||||
let stackDir: string;
|
||||
let nodeId: number;
|
||||
|
||||
beforeAll(async () => {
|
||||
tmpDir = await setupTestDb();
|
||||
composeDir = process.env.COMPOSE_DIR!;
|
||||
stackDir = path.join(composeDir, STACK);
|
||||
fs.mkdirSync(stackDir, { recursive: true });
|
||||
|
||||
const { DatabaseService } = await import('../../services/DatabaseService');
|
||||
const db = DatabaseService.getInstance();
|
||||
const local = db.getDefaultNode();
|
||||
if (!local?.id) throw new Error('Test DB has no default local node');
|
||||
nodeId = local.id;
|
||||
db.updateGlobalSetting('prune_on_update', '0');
|
||||
|
||||
execFileSync('docker', ['pull', 'busybox:1.36.1'], { stdio: 'ignore' });
|
||||
execFileSync('docker', ['pull', 'busybox:1.36.0'], { stdio: 'ignore' });
|
||||
|
||||
fs.writeFileSync(
|
||||
path.join(stackDir, 'compose.yaml'),
|
||||
[
|
||||
'services:',
|
||||
' web:',
|
||||
' image: busybox:1.36.1',
|
||||
' command: ["sleep", "3600"]',
|
||||
'',
|
||||
].join('\n'),
|
||||
'utf8',
|
||||
);
|
||||
|
||||
compose(['up', '-d', '--pull', 'never'], stackDir);
|
||||
}, 300_000);
|
||||
|
||||
afterAll(async () => {
|
||||
try {
|
||||
if (stackDir && fs.existsSync(stackDir)) {
|
||||
compose(['down', '--remove-orphans'], stackDir);
|
||||
}
|
||||
} catch {
|
||||
// Best-effort cleanup.
|
||||
}
|
||||
if (tmpDir) cleanupTestDb(tmpDir);
|
||||
}, 120_000);
|
||||
|
||||
it('restores a container whose inspected Image equals the captured prior image ID', async () => {
|
||||
const beforeId = compose(['ps', '-q'], stackDir).trim();
|
||||
expect(beforeId.length).toBeGreaterThan(0);
|
||||
const priorImageId = inspectImageId(beforeId);
|
||||
|
||||
const { StackUpdateRecoveryService } = await import('../../services/StackUpdateRecoveryService');
|
||||
StackUpdateRecoveryService.resetForTests();
|
||||
const recoverySvc = StackUpdateRecoveryService.getInstance();
|
||||
|
||||
// Capture while the prior runtime is still healthy.
|
||||
const candidate = await recoverySvc.captureCandidate({
|
||||
nodeId,
|
||||
stackName: STACK,
|
||||
createdBy: null,
|
||||
operationKind: 'update',
|
||||
});
|
||||
expect(candidate.override_path).toBeTruthy();
|
||||
expect(recoverySvc.markAcquired(candidate.id)).toBe(true);
|
||||
expect(recoverySvc.handoff(candidate.id, nodeId, STACK)).toBe(true);
|
||||
|
||||
// Simulate a post-handoff mutation that moves the tag / image identity.
|
||||
fs.writeFileSync(
|
||||
path.join(stackDir, 'compose.yaml'),
|
||||
[
|
||||
'services:',
|
||||
' web:',
|
||||
' image: busybox:1.36.0',
|
||||
' command: ["sleep", "3600"]',
|
||||
'',
|
||||
].join('\n'),
|
||||
'utf8',
|
||||
);
|
||||
compose(['up', '-d', '--pull', 'never', '--force-recreate'], stackDir);
|
||||
const midId = compose(['ps', '-q'], stackDir).trim();
|
||||
const midImageId = inspectImageId(midId);
|
||||
expect(midImageId).not.toBe(priorImageId);
|
||||
|
||||
const { ComposeService } = await import('../../services/ComposeService');
|
||||
const rolledBack = await recoverySvc.compensateWithCandidate(
|
||||
candidate.id,
|
||||
(overridePath, invocation) => ComposeService.getInstance(nodeId).composeUpWithRecoveryOverride(
|
||||
STACK,
|
||||
overridePath,
|
||||
undefined,
|
||||
invocation,
|
||||
),
|
||||
);
|
||||
expect(rolledBack).toBe(true);
|
||||
|
||||
const afterId = compose(['ps', '-q'], stackDir).trim();
|
||||
expect(afterId.length).toBeGreaterThan(0);
|
||||
expect(inspectImageId(afterId)).toBe(priorImageId);
|
||||
|
||||
const afterInspect = JSON.parse(
|
||||
execFileSync('docker', ['inspect', afterId], { encoding: 'utf8' }),
|
||||
) as Array<{ State: { Running: boolean } }>;
|
||||
expect(afterInspect[0].State.Running).toBe(true);
|
||||
}, 300_000);
|
||||
});
|
||||
@@ -10,7 +10,11 @@ import { setupTestDb, cleanupTestDb } from '../helpers/setupTestDb';
|
||||
|
||||
function dockerAvailable(): boolean {
|
||||
try {
|
||||
execFileSync('docker', ['info'], { stdio: 'ignore' });
|
||||
execFileSync('docker', ['info'], {
|
||||
stdio: 'ignore',
|
||||
timeout: 8_000,
|
||||
windowsHide: true,
|
||||
});
|
||||
return true;
|
||||
} catch {
|
||||
return false;
|
||||
|
||||
@@ -109,6 +109,21 @@ describe('classifyFailure', () => {
|
||||
message: 'something completely unexpected happened',
|
||||
reason: 'unknown',
|
||||
},
|
||||
{
|
||||
name: 'mixed replica images capture refusal',
|
||||
message: 'Service "web" has mixed replica images; refusing recovery capture that cannot restore exact prior identity',
|
||||
reason: 'mixed_replica_images',
|
||||
},
|
||||
{
|
||||
name: 'host-absolute include coverage refusal',
|
||||
message: 'Host-absolute include path cannot be captured for exact rollback',
|
||||
reason: 'rollback_coverage_unavailable',
|
||||
},
|
||||
{
|
||||
name: 'generic rollback coverage unavailable',
|
||||
message: 'Exact authored-project rollback coverage is unavailable for this stack',
|
||||
reason: 'rollback_coverage_unavailable',
|
||||
},
|
||||
];
|
||||
|
||||
it.each(cases)('classifies $name as $reason', ({ message, reason }) => {
|
||||
|
||||
@@ -8,8 +8,11 @@ import { describe, it, expect, beforeAll, afterAll, afterEach, vi } from 'vitest
|
||||
import request from 'supertest';
|
||||
import fs from 'fs';
|
||||
import path from 'path';
|
||||
import { createHash, randomUUID } from 'crypto';
|
||||
import { setupTestDb, cleanupTestDb, loginAsTestAdmin } from './helpers/setupTestDb';
|
||||
import * as policyGate from '../helpers/policyGate';
|
||||
import { RollbackGenerationStore } from '../services/RollbackGenerationStore';
|
||||
import type { ResolvedRollbackInventory, RollbackGenerationManifest } from '../types/rollbackGeneration';
|
||||
|
||||
let tmpDir: string;
|
||||
let app: import('express').Express;
|
||||
@@ -18,6 +21,9 @@ let CryptoService: typeof import('../services/CryptoService').CryptoService;
|
||||
let ComposeService: typeof import('../services/ComposeService').ComposeService;
|
||||
let LicenseService: typeof import('../services/LicenseService').LicenseService;
|
||||
let NodeRegistry: typeof import('../services/NodeRegistry').NodeRegistry;
|
||||
let FileSystemService: typeof import('../services/FileSystemService').FileSystemService;
|
||||
let StackUpdateRecoveryService: typeof import('../services/StackUpdateRecoveryService').StackUpdateRecoveryService;
|
||||
let StackOpLockService: typeof import('../services/StackOpLockService').StackOpLockService;
|
||||
let adminCookie: string;
|
||||
let viewerCookie: string;
|
||||
let snapshotId: number;
|
||||
@@ -39,6 +45,9 @@ beforeAll(async () => {
|
||||
({ ComposeService } = await import('../services/ComposeService'));
|
||||
({ LicenseService } = await import('../services/LicenseService'));
|
||||
({ NodeRegistry } = await import('../services/NodeRegistry'));
|
||||
({ FileSystemService } = await import('../services/FileSystemService'));
|
||||
({ StackUpdateRecoveryService } = await import('../services/StackUpdateRecoveryService'));
|
||||
({ StackOpLockService } = await import('../services/StackOpLockService'));
|
||||
({ app } = await import('../index'));
|
||||
adminCookie = await loginAsTestAdmin(app);
|
||||
|
||||
@@ -289,7 +298,7 @@ describe('Single-stack snapshot restore (behavior lock)', () => {
|
||||
fs.writeFileSync(composePath('corrupt-web'), beforeCompose);
|
||||
fs.writeFileSync(envPath('corrupt-web'), beforeEnv);
|
||||
|
||||
const deploySpy = vi.spyOn(ComposeService.prototype, 'deployStack').mockResolvedValue();
|
||||
const deploySpy = vi.spyOn(ComposeService.prototype, 'deployStack').mockResolvedValue({ recoveryId: null });
|
||||
const res = await request(app)
|
||||
.post(`/api/fleet/snapshots/${id}/restore`)
|
||||
.set('Cookie', adminCookie)
|
||||
@@ -320,7 +329,7 @@ describe('Single-stack snapshot restore (behavior lock)', () => {
|
||||
fs.writeFileSync(composePath('mixed-web'), beforeCompose);
|
||||
fs.writeFileSync(envPath('mixed-web'), beforeEnv);
|
||||
|
||||
const deploySpy = vi.spyOn(ComposeService.prototype, 'deployStack').mockResolvedValue();
|
||||
const deploySpy = vi.spyOn(ComposeService.prototype, 'deployStack').mockResolvedValue({ recoveryId: null });
|
||||
const res = await request(app)
|
||||
.post(`/api/fleet/snapshots/${id}/restore`)
|
||||
.set('Cookie', adminCookie)
|
||||
@@ -346,7 +355,7 @@ describe('Single-stack snapshot restore (behavior lock)', () => {
|
||||
const beforeCompose = 'services:\n keep: {}\n';
|
||||
fs.writeFileSync(composePath('delim-web'), beforeCompose);
|
||||
|
||||
const deploySpy = vi.spyOn(ComposeService.prototype, 'deployStack').mockResolvedValue();
|
||||
const deploySpy = vi.spyOn(ComposeService.prototype, 'deployStack').mockResolvedValue({ recoveryId: null });
|
||||
const res = await request(app)
|
||||
.post(`/api/fleet/snapshots/${id}/restore`)
|
||||
.set('Cookie', adminCookie)
|
||||
@@ -387,7 +396,7 @@ describe('Single-stack snapshot restore (behavior lock)', () => {
|
||||
|
||||
it('redeploys after restore when requested', async () => {
|
||||
vi.spyOn(LicenseService.getInstance(), 'getTier').mockReturnValue('community');
|
||||
const deploySpy = vi.spyOn(ComposeService.prototype, 'deployStack').mockResolvedValue();
|
||||
const deploySpy = vi.spyOn(ComposeService.prototype, 'deployStack').mockResolvedValue({ recoveryId: null });
|
||||
const db = DatabaseService.getInstance();
|
||||
const id = db.createSnapshot('restore-redeploy', 'admin', 1, 1, '[]', '[]');
|
||||
db.insertSnapshotFiles(id, [
|
||||
@@ -749,7 +758,7 @@ describe('Restore-all', () => {
|
||||
|
||||
it('isolates corrupt decrypt stacks before any mutation with notes and redeploy requested', async () => {
|
||||
vi.spyOn(LicenseService.getInstance(), 'getTier').mockReturnValue('community');
|
||||
const deploySpy = vi.spyOn(ComposeService.prototype, 'deployStack').mockResolvedValue();
|
||||
const deploySpy = vi.spyOn(ComposeService.prototype, 'deployStack').mockResolvedValue({ recoveryId: null });
|
||||
const db = DatabaseService.getInstance();
|
||||
const id = db.createSnapshot('restore-all-corrupt', 'admin', 1, 2, '[]', '[]');
|
||||
const good = CryptoService.getInstance().encrypt('services:\n app: {}\n');
|
||||
@@ -791,7 +800,7 @@ describe('Restore-all', () => {
|
||||
|
||||
it('isolates delimiter-byte corruption before restore-all mutation', async () => {
|
||||
vi.spyOn(LicenseService.getInstance(), 'getTier').mockReturnValue('community');
|
||||
const deploySpy = vi.spyOn(ComposeService.prototype, 'deployStack').mockResolvedValue();
|
||||
const deploySpy = vi.spyOn(ComposeService.prototype, 'deployStack').mockResolvedValue({ recoveryId: null });
|
||||
const db = DatabaseService.getInstance();
|
||||
const id = db.createSnapshot('restore-all-delim', 'admin', 1, 2, '[]', '[]');
|
||||
const good = CryptoService.getInstance().encrypt('services:\n app: {}\n');
|
||||
@@ -801,13 +810,13 @@ describe('Restore-all', () => {
|
||||
const damaged = `enc:${iv} ${tag}:${ct}`;
|
||||
db.getDb().prepare(
|
||||
'INSERT INTO fleet_snapshot_files (snapshot_id, node_id, node_name, stack_name, filename, content) VALUES (?, ?, ?, ?, ?, ?)',
|
||||
).run(id, LOCAL_NODE_ID, 'local', 'healthy', 'compose.yaml', good);
|
||||
).run(id, LOCAL_NODE_ID, 'local', 'delim-healthy', 'compose.yaml', good);
|
||||
db.getDb().prepare(
|
||||
'INSERT INTO fleet_snapshot_files (snapshot_id, node_id, node_name, stack_name, filename, content) VALUES (?, ?, ?, ?, ?, ?)',
|
||||
).run(id, LOCAL_NODE_ID, 'local', 'corrupt', 'compose.yaml', damaged);
|
||||
seedStackDir('healthy');
|
||||
seedStackDir('corrupt');
|
||||
fs.writeFileSync(composePath('corrupt'), 'services:\n keep: {}\n');
|
||||
).run(id, LOCAL_NODE_ID, 'local', 'delim-corrupt', 'compose.yaml', damaged);
|
||||
seedStackDir('delim-healthy');
|
||||
seedStackDir('delim-corrupt');
|
||||
fs.writeFileSync(composePath('delim-corrupt'), 'services:\n keep: {}\n');
|
||||
|
||||
const res = await request(app)
|
||||
.post(`/api/fleet/snapshots/${id}/restore-all`)
|
||||
@@ -817,16 +826,16 @@ describe('Restore-all', () => {
|
||||
expect(res.body.restored).toBe(1);
|
||||
expect(res.body.failed).toBe(1);
|
||||
const corrupt = (res.body.results as Array<{ stackName: string; success: boolean; error?: string }>)
|
||||
.find(r => r.stackName === 'corrupt');
|
||||
.find(r => r.stackName === 'delim-corrupt');
|
||||
expect(corrupt?.success).toBe(false);
|
||||
expect(fs.readFileSync(composePath('corrupt'), 'utf-8')).toContain('keep: {}');
|
||||
expect(fs.readFileSync(composePath('delim-corrupt'), 'utf-8')).toContain('keep: {}');
|
||||
expect(deploySpy).toHaveBeenCalledTimes(1);
|
||||
expect(deploySpy.mock.calls.every(call => call[0] !== 'corrupt')).toBe(true);
|
||||
expect(deploySpy.mock.calls.every(call => call[0] !== 'delim-corrupt')).toBe(true);
|
||||
});
|
||||
|
||||
it('redeploys each restored stack when requested', async () => {
|
||||
vi.spyOn(LicenseService.getInstance(), 'getTier').mockReturnValue('community');
|
||||
const deploySpy = vi.spyOn(ComposeService.prototype, 'deployStack').mockResolvedValue();
|
||||
const deploySpy = vi.spyOn(ComposeService.prototype, 'deployStack').mockResolvedValue({ recoveryId: null });
|
||||
const db = DatabaseService.getInstance();
|
||||
const id = db.createSnapshot('restore-all-redeploy', 'admin', 1, 1, '[]', '[]');
|
||||
db.insertSnapshotFiles(id, [
|
||||
@@ -847,7 +856,7 @@ describe('Restore-all', () => {
|
||||
});
|
||||
|
||||
it('records a policy-blocked redeploy as a per-stack failure and still restores the rest', async () => {
|
||||
const deploySpy = vi.spyOn(ComposeService.prototype, 'deployStack').mockResolvedValue();
|
||||
const deploySpy = vi.spyOn(ComposeService.prototype, 'deployStack').mockResolvedValue({ recoveryId: null });
|
||||
vi.spyOn(policyGate, 'assertPolicyGateAllows').mockImplementation(async (stackName: string) => {
|
||||
if (stackName === 'blocked-web') throw new Error('Policy "block-criticals" blocked deploy: 1 image(s) exceed high');
|
||||
});
|
||||
@@ -876,3 +885,456 @@ describe('Restore-all', () => {
|
||||
expect(deploySpy.mock.calls.map((c) => c[0])).not.toContain('blocked-web');
|
||||
});
|
||||
});
|
||||
|
||||
describe('Snapshot restore: recovery generation contract', () => {
|
||||
afterEach(() => {
|
||||
vi.restoreAllMocks();
|
||||
vi.unstubAllGlobals();
|
||||
StackOpLockService.resetForTests();
|
||||
DatabaseService.getInstance().getDb().prepare('DELETE FROM stack_update_recovery_generations').run();
|
||||
});
|
||||
|
||||
function seedExistingStack(stack: string, compose: string, extraName?: string, extraContent?: string): void {
|
||||
seedStackDir(stack);
|
||||
fs.writeFileSync(composePath(stack), compose);
|
||||
if (extraName && extraContent !== undefined) {
|
||||
fs.writeFileSync(path.join(process.env.COMPOSE_DIR as string, stack, extraName), extraContent);
|
||||
}
|
||||
}
|
||||
|
||||
function insertSnapshot(
|
||||
label: string,
|
||||
files: Array<{ nodeId: number; nodeName: string; stackName: string; filename: string; content: string }>,
|
||||
stackCount = 1,
|
||||
): number {
|
||||
const db = DatabaseService.getInstance();
|
||||
const id = db.createSnapshot(label, 'admin', 1, stackCount, '[]', '[]');
|
||||
db.insertSnapshotFiles(id, files);
|
||||
return id;
|
||||
}
|
||||
|
||||
function addRemoteNode(name: string): number {
|
||||
return DatabaseService.getInstance().addNode({
|
||||
name,
|
||||
type: 'remote',
|
||||
api_url: 'http://remote:1852',
|
||||
api_token: 'tok',
|
||||
compose_dir: '/app/compose',
|
||||
is_default: false,
|
||||
});
|
||||
}
|
||||
|
||||
function stubRemoteProxy(): void {
|
||||
vi.spyOn(NodeRegistry.getInstance(), 'getProxyTarget').mockReturnValue({
|
||||
apiUrl: 'http://remote:1852',
|
||||
apiToken: 'tok',
|
||||
});
|
||||
}
|
||||
|
||||
function sha256Text(text: string): string {
|
||||
return createHash('sha256').update(text, 'utf8').digest('hex');
|
||||
}
|
||||
|
||||
function liveInventory(stack: string): ResolvedRollbackInventory {
|
||||
const stackDir = path.join(process.env.COMPOSE_DIR as string, stack);
|
||||
const entries: ResolvedRollbackInventory['entries'] = [{
|
||||
relativePath: 'compose.yaml',
|
||||
dependencyKind: 'compose-root',
|
||||
provenance: 'authored',
|
||||
sensitivity: 'low',
|
||||
absolutePath: path.join(stackDir, 'compose.yaml'),
|
||||
}];
|
||||
if (fs.existsSync(envPath(stack))) {
|
||||
entries.push({
|
||||
relativePath: '.env',
|
||||
dependencyKind: 'project-env',
|
||||
provenance: 'authored',
|
||||
sensitivity: 'low',
|
||||
absolutePath: envPath(stack),
|
||||
});
|
||||
}
|
||||
return {
|
||||
entries,
|
||||
invocation: {
|
||||
composeArgsPrefix: [],
|
||||
projectDirectory: null,
|
||||
projectName: stack,
|
||||
explicitComposeFiles: ['compose.yaml'],
|
||||
meshEnabled: false,
|
||||
meshOverrideRelativePath: null,
|
||||
},
|
||||
git: null,
|
||||
appliedDeploySpec: null,
|
||||
lastAppliedContentHash: null,
|
||||
manifestState: null,
|
||||
manifestGeneration: null,
|
||||
exactCoverage: true,
|
||||
coverageRefusal: null,
|
||||
};
|
||||
}
|
||||
|
||||
async function persistPreRestoreGeneration(stack: string): Promise<string> {
|
||||
const id = randomUUID();
|
||||
await RollbackGenerationStore.captureGeneration({
|
||||
nodeId: LOCAL_NODE_ID,
|
||||
stackName: stack,
|
||||
generationId: id,
|
||||
inventory: liveInventory(stack),
|
||||
operationKind: 'manual_backup',
|
||||
});
|
||||
const now = Date.now();
|
||||
DatabaseService.getInstance().insertStackUpdateRecoveryGeneration({
|
||||
id,
|
||||
node_id: LOCAL_NODE_ID,
|
||||
stack_name: stack,
|
||||
status: 'active',
|
||||
phase: 'immediate_verified',
|
||||
is_current: 1,
|
||||
backup_slot_id: id,
|
||||
content_path: id,
|
||||
operation_kind: 'manual_backup',
|
||||
override_path: null,
|
||||
services_json: '[]',
|
||||
health_gate_id: null,
|
||||
gate_retain_until: null,
|
||||
artifact_expires_at: null,
|
||||
operation_lease_expires_at: null,
|
||||
created_at: now,
|
||||
updated_at: now,
|
||||
created_by: 'system:fleet-snapshot',
|
||||
artifacts_retired: 0,
|
||||
released_at: null,
|
||||
released_by: null,
|
||||
});
|
||||
return id;
|
||||
}
|
||||
|
||||
function spyCaptureRecordingLive(stack: string): { composeAtCapture: string; envAtCapture: string } {
|
||||
const captured = { composeAtCapture: '', envAtCapture: '' };
|
||||
vi.spyOn(StackUpdateRecoveryService.getInstance(), 'captureCurrentBackup').mockImplementation(async () => {
|
||||
captured.composeAtCapture = fs.readFileSync(composePath(stack), 'utf-8');
|
||||
captured.envAtCapture = fs.existsSync(envPath(stack)) ? fs.readFileSync(envPath(stack), 'utf-8') : '';
|
||||
const id = await persistPreRestoreGeneration(stack);
|
||||
return { id } as never;
|
||||
});
|
||||
return captured;
|
||||
}
|
||||
|
||||
function expectCurrentGenerationMatches(stack: string, compose: string, env: string): void {
|
||||
const current = StackUpdateRecoveryService.getInstance().getCurrent(LOCAL_NODE_ID, stack);
|
||||
expect(current?.id).toBeTruthy();
|
||||
const genDir = RollbackGenerationStore.getGenerationDir(LOCAL_NODE_ID, stack, current!.id);
|
||||
const manifest = JSON.parse(
|
||||
fs.readFileSync(path.join(genDir, 'generation.json'), 'utf-8'),
|
||||
) as RollbackGenerationManifest;
|
||||
expect(manifest.entries.find((e) => e.relativePath === 'compose.yaml')?.contentSha256).toBe(sha256Text(compose));
|
||||
expect(manifest.entries.find((e) => e.relativePath === '.env')?.contentSha256).toBe(sha256Text(env));
|
||||
}
|
||||
|
||||
it('captures the pre-restore multi-file project before overwriting compose.yaml', async () => {
|
||||
const id = insertSnapshot('restore-gen-local', [
|
||||
{ nodeId: LOCAL_NODE_ID, nodeName: 'local', stackName: 'preimage-web', filename: 'compose.yaml', content: 'services:\n snap: {}\n' },
|
||||
{ nodeId: LOCAL_NODE_ID, nodeName: 'local', stackName: 'preimage-web', filename: '.env', content: 'SNAP=1\n' },
|
||||
]);
|
||||
const preCompose = 'services:\n old: {}\n';
|
||||
const preEnv = 'OLD=1\n';
|
||||
seedExistingStack('preimage-web', preCompose, 'extra.yml', 'x: 1\n');
|
||||
fs.writeFileSync(envPath('preimage-web'), preEnv);
|
||||
const captured = spyCaptureRecordingLive('preimage-web');
|
||||
|
||||
const res = await request(app)
|
||||
.post(`/api/fleet/snapshots/${id}/restore`)
|
||||
.set('Cookie', adminCookie)
|
||||
.send({ nodeId: LOCAL_NODE_ID, stackName: 'preimage-web' });
|
||||
|
||||
expect(res.status).toBe(200);
|
||||
expect(captured.composeAtCapture).toBe(preCompose);
|
||||
expect(captured.envAtCapture).toBe(preEnv);
|
||||
expect(fs.readFileSync(composePath('preimage-web'), 'utf-8')).toContain('snap: {}');
|
||||
expect(fs.readFileSync(envPath('preimage-web'), 'utf-8')).toContain('SNAP=1');
|
||||
expect(fs.readFileSync(path.join(process.env.COMPOSE_DIR as string, 'preimage-web', 'extra.yml'), 'utf-8')).toBe('x: 1\n');
|
||||
expectCurrentGenerationMatches('preimage-web', preCompose, preEnv);
|
||||
});
|
||||
|
||||
it('does not mutate live files when capture fails', async () => {
|
||||
const id = insertSnapshot('restore-gen-fail', [
|
||||
{ nodeId: LOCAL_NODE_ID, nodeName: 'local', stackName: 'fail-web', filename: 'compose.yaml', content: 'services:\n snap: {}\n' },
|
||||
]);
|
||||
const preCompose = 'services:\n keep: {}\n';
|
||||
seedExistingStack('fail-web', preCompose);
|
||||
vi.spyOn(StackUpdateRecoveryService.getInstance(), 'captureCurrentBackup')
|
||||
.mockRejectedValue(new Error('generation capture failed'));
|
||||
|
||||
const res = await request(app)
|
||||
.post(`/api/fleet/snapshots/${id}/restore`)
|
||||
.set('Cookie', adminCookie)
|
||||
.send({ nodeId: LOCAL_NODE_ID, stackName: 'fail-web' });
|
||||
|
||||
expect(res.status).toBe(500);
|
||||
expect(fs.readFileSync(composePath('fail-web'), 'utf-8')).toBe(preCompose);
|
||||
});
|
||||
|
||||
it('returns 409 and does not write when the stack operation lock is held', async () => {
|
||||
const id = insertSnapshot('restore-gen-lock', [
|
||||
{ nodeId: LOCAL_NODE_ID, nodeName: 'local', stackName: 'lock-web', filename: 'compose.yaml', content: 'services:\n snap: {}\n' },
|
||||
]);
|
||||
const preCompose = 'services:\n keep: {}\n';
|
||||
seedExistingStack('lock-web', preCompose);
|
||||
const captureSpy = vi.spyOn(StackUpdateRecoveryService.getInstance(), 'captureCurrentBackup')
|
||||
.mockResolvedValue({ id: 'gen-lock-web' } as never);
|
||||
StackOpLockService.getInstance().tryAcquire(LOCAL_NODE_ID, 'lock-web', 'deploy', 'other');
|
||||
|
||||
const res = await request(app)
|
||||
.post(`/api/fleet/snapshots/${id}/restore`)
|
||||
.set('Cookie', adminCookie)
|
||||
.send({ nodeId: LOCAL_NODE_ID, stackName: 'lock-web' });
|
||||
|
||||
expect(res.status).toBe(409);
|
||||
expect(res.body.code).toBe('stack_op_in_progress');
|
||||
expect(captureSpy).not.toHaveBeenCalled();
|
||||
expect(fs.readFileSync(composePath('lock-web'), 'utf-8')).toBe(preCompose);
|
||||
});
|
||||
|
||||
it('keeps the captured generation when a later snapshot file write fails', async () => {
|
||||
const id = insertSnapshot('restore-gen-partial', [
|
||||
{ nodeId: LOCAL_NODE_ID, nodeName: 'local', stackName: 'partial-web', filename: 'compose.yaml', content: 'services:\n snap: {}\n' },
|
||||
{ nodeId: LOCAL_NODE_ID, nodeName: 'local', stackName: 'partial-web', filename: '.env', content: 'SNAP=1\n' },
|
||||
]);
|
||||
const preCompose = 'services:\n old: {}\n';
|
||||
const preEnv = 'OLD=1\n';
|
||||
seedExistingStack('partial-web', preCompose);
|
||||
fs.writeFileSync(envPath('partial-web'), preEnv);
|
||||
const captured = spyCaptureRecordingLive('partial-web');
|
||||
vi.spyOn(FileSystemService.prototype, 'saveEnvContent').mockRejectedValue(new Error('disk full'));
|
||||
|
||||
const res = await request(app)
|
||||
.post(`/api/fleet/snapshots/${id}/restore`)
|
||||
.set('Cookie', adminCookie)
|
||||
.send({ nodeId: LOCAL_NODE_ID, stackName: 'partial-web' });
|
||||
|
||||
expect(res.status).toBe(500);
|
||||
expect(captured.composeAtCapture).toBe(preCompose);
|
||||
expect(captured.envAtCapture).toBe(preEnv);
|
||||
expect(fs.readFileSync(composePath('partial-web'), 'utf-8')).toContain('snap: {}');
|
||||
expect(fs.readFileSync(envPath('partial-web'), 'utf-8')).toBe(preEnv);
|
||||
expectCurrentGenerationMatches('partial-web', preCompose, preEnv);
|
||||
});
|
||||
|
||||
it('does not capture a generation when restoring a stack that does not exist yet', async () => {
|
||||
const id = insertSnapshot('restore-gen-new', [
|
||||
{ nodeId: LOCAL_NODE_ID, nodeName: 'local', stackName: 'brand-new', filename: 'compose.yaml', content: 'services:\n snap: {}\n' },
|
||||
]);
|
||||
seedStackDir('brand-new');
|
||||
const captureSpy = vi.spyOn(StackUpdateRecoveryService.getInstance(), 'captureCurrentBackup')
|
||||
.mockResolvedValue({ id: 'should-not-run' } as never);
|
||||
|
||||
const res = await request(app)
|
||||
.post(`/api/fleet/snapshots/${id}/restore`)
|
||||
.set('Cookie', adminCookie)
|
||||
.send({ nodeId: LOCAL_NODE_ID, stackName: 'brand-new' });
|
||||
|
||||
expect(res.status).toBe(200);
|
||||
expect(captureSpy).not.toHaveBeenCalled();
|
||||
expect(fs.readFileSync(composePath('brand-new'), 'utf-8')).toContain('snap: {}');
|
||||
});
|
||||
|
||||
it('restore-all captures existing stacks and still restores siblings when one capture fails', async () => {
|
||||
const id = insertSnapshot('restore-all-gen', [
|
||||
{ nodeId: LOCAL_NODE_ID, nodeName: 'local', stackName: 'all-keep', filename: 'compose.yaml', content: 'services:\n keep: {}\n' },
|
||||
{ nodeId: LOCAL_NODE_ID, nodeName: 'local', stackName: 'all-fail', filename: 'compose.yaml', content: 'services:\n fail: {}\n' },
|
||||
], 2);
|
||||
seedExistingStack('all-keep', 'services:\n old-keep: {}\n');
|
||||
fs.writeFileSync(envPath('all-keep'), 'KEEP=1\n');
|
||||
seedExistingStack('all-fail', 'services:\n old-fail: {}\n');
|
||||
fs.writeFileSync(envPath('all-fail'), 'FAIL=1\n');
|
||||
vi.spyOn(StackUpdateRecoveryService.getInstance(), 'captureCurrentBackup')
|
||||
.mockImplementation(async (input) => {
|
||||
if (input.stackName === 'all-fail') throw new Error('generation capture failed');
|
||||
const id = await persistPreRestoreGeneration(input.stackName);
|
||||
return { id } as never;
|
||||
});
|
||||
|
||||
const res = await request(app)
|
||||
.post(`/api/fleet/snapshots/${id}/restore-all`)
|
||||
.set('Cookie', adminCookie)
|
||||
.send({});
|
||||
|
||||
expect(res.status).toBe(200);
|
||||
expect(res.body.restored).toBe(1);
|
||||
expect(res.body.failed).toBe(1);
|
||||
const failed = (res.body.results as Array<{ stackName: string; success: boolean }>).find(r => r.stackName === 'all-fail');
|
||||
expect(failed?.success).toBe(false);
|
||||
expect(fs.readFileSync(composePath('all-keep'), 'utf-8')).toContain('keep: {}');
|
||||
expect(fs.readFileSync(composePath('all-fail'), 'utf-8')).toContain('old-fail: {}');
|
||||
expect(fs.readFileSync(envPath('all-fail'), 'utf-8')).toBe('FAIL=1\n');
|
||||
expectCurrentGenerationMatches('all-keep', 'services:\n old-keep: {}\n', 'KEEP=1\n');
|
||||
expect(StackUpdateRecoveryService.getInstance().getCurrent(LOCAL_NODE_ID, 'all-fail')).toBeUndefined();
|
||||
});
|
||||
|
||||
it('restores a remote stack through one node-local apply request', async () => {
|
||||
const remoteId = addRemoteNode('remote-gen');
|
||||
const id = insertSnapshot('restore-gen-remote', [
|
||||
{ nodeId: remoteId, nodeName: 'remote-gen', stackName: 'rweb', filename: 'compose.yaml', content: 'services: {}\n' },
|
||||
{ nodeId: remoteId, nodeName: 'remote-gen', stackName: 'rweb', filename: '.env', content: 'SNAP=1\n' },
|
||||
]);
|
||||
stubRemoteProxy();
|
||||
const calls: Array<{ url: string; method?: string; body?: string }> = [];
|
||||
vi.stubGlobal('fetch', vi.fn(async (url: string, opts?: { method?: string; body?: string }) => {
|
||||
calls.push({ url, method: opts?.method, body: opts?.body });
|
||||
return { ok: true, status: 200, text: async () => '' } as unknown as Response;
|
||||
}));
|
||||
|
||||
const res = await request(app)
|
||||
.post(`/api/fleet/snapshots/${id}/restore`)
|
||||
.set('Cookie', adminCookie)
|
||||
.send({ nodeId: remoteId, stackName: 'rweb' });
|
||||
|
||||
expect(res.status).toBe(200);
|
||||
const applyCalls = calls.filter(c => c.url.includes('/fleet-snapshot-apply'));
|
||||
expect(applyCalls).toHaveLength(1);
|
||||
expect(applyCalls[0].method).toBe('POST');
|
||||
expect(applyCalls[0].body).toContain('compose.yaml');
|
||||
expect(applyCalls[0].body).toContain('.env');
|
||||
expect(calls.some(c => c.method === 'PUT' && /\/api\/stacks\/[^/]+$/.test(c.url))).toBe(false);
|
||||
expect(calls.some(c => c.method === 'PUT' && /\/env$/.test(c.url))).toBe(false);
|
||||
});
|
||||
|
||||
it('fails a remote restore without writing when the node-local apply returns 409', async () => {
|
||||
const remoteId = addRemoteNode('remote-lock');
|
||||
const id = insertSnapshot('restore-gen-remote-lock', [
|
||||
{ nodeId: remoteId, nodeName: 'remote-lock', stackName: 'rlock', filename: 'compose.yaml', content: 'services: {}\n' },
|
||||
]);
|
||||
stubRemoteProxy();
|
||||
vi.stubGlobal('fetch', vi.fn(async () => ({
|
||||
ok: false,
|
||||
status: 409,
|
||||
text: async () => JSON.stringify({ error: 'locked', code: 'stack_op_in_progress' }),
|
||||
} as unknown as Response)));
|
||||
|
||||
const res = await request(app)
|
||||
.post(`/api/fleet/snapshots/${id}/restore`)
|
||||
.set('Cookie', adminCookie)
|
||||
.send({ nodeId: remoteId, stackName: 'rlock' });
|
||||
|
||||
expect(res.status).toBe(409);
|
||||
expect(res.body.code).toBe('stack_op_in_progress');
|
||||
});
|
||||
|
||||
it('does not invent stack_op_in_progress from a bare remote 409', async () => {
|
||||
const remoteId = addRemoteNode('remote-bare-409');
|
||||
const id = insertSnapshot('restore-gen-remote-bare-409', [
|
||||
{ nodeId: remoteId, nodeName: 'remote-bare-409', stackName: 'rbare', filename: 'compose.yaml', content: 'services: {}\n' },
|
||||
]);
|
||||
stubRemoteProxy();
|
||||
vi.stubGlobal('fetch', vi.fn(async () => ({
|
||||
ok: false,
|
||||
status: 409,
|
||||
text: async () => JSON.stringify({ error: 'conflict' }),
|
||||
} as unknown as Response)));
|
||||
|
||||
const res = await request(app)
|
||||
.post(`/api/fleet/snapshots/${id}/restore`)
|
||||
.set('Cookie', adminCookie)
|
||||
.send({ nodeId: remoteId, stackName: 'rbare' });
|
||||
|
||||
expect(res.status).toBe(500);
|
||||
expect(res.body.code).toBeUndefined();
|
||||
});
|
||||
|
||||
it('restore-all sends one node-local apply per remote stack', async () => {
|
||||
const remoteId = addRemoteNode('remote-all');
|
||||
const id = insertSnapshot('restore-all-gen-remote', [
|
||||
{ nodeId: remoteId, nodeName: 'remote-all', stackName: 'ra', filename: 'compose.yaml', content: 'services:\n a: {}\n' },
|
||||
{ nodeId: remoteId, nodeName: 'remote-all', stackName: 'rb', filename: 'compose.yaml', content: 'services:\n b: {}\n' },
|
||||
], 2);
|
||||
stubRemoteProxy();
|
||||
const urls: string[] = [];
|
||||
vi.stubGlobal('fetch', vi.fn(async (url: string) => {
|
||||
urls.push(url);
|
||||
return { ok: true, status: 200, text: async () => '' } as unknown as Response;
|
||||
}));
|
||||
|
||||
const res = await request(app)
|
||||
.post(`/api/fleet/snapshots/${id}/restore-all`)
|
||||
.set('Cookie', adminCookie)
|
||||
.send({});
|
||||
|
||||
expect(res.status).toBe(200);
|
||||
expect(res.body.restored).toBe(2);
|
||||
expect(urls.filter(u => u.includes('/fleet-snapshot-apply'))).toHaveLength(2);
|
||||
expect(urls.some(u => /\/api\/stacks\/[^/]+$/.test(u))).toBe(false);
|
||||
});
|
||||
|
||||
it('POST /api/stacks/:stackName/fleet-snapshot-apply returns 403 for a viewer', async () => {
|
||||
const res = await request(app)
|
||||
.post('/api/stacks/preimage-web/fleet-snapshot-apply')
|
||||
.set('Cookie', viewerCookie)
|
||||
.send({ files: [{ filename: 'compose.yaml', content: 'services: {}\n' }] });
|
||||
expect(res.status).toBe(403);
|
||||
});
|
||||
|
||||
it('POST /api/stacks/:stackName/fleet-snapshot-apply captures an existing stack before writing', async () => {
|
||||
const preCompose = 'services:\n old: {}\n';
|
||||
const preEnv = 'OLD=1\n';
|
||||
seedExistingStack('apply-existing', preCompose);
|
||||
fs.writeFileSync(envPath('apply-existing'), preEnv);
|
||||
spyCaptureRecordingLive('apply-existing');
|
||||
|
||||
const res = await request(app)
|
||||
.post('/api/stacks/apply-existing/fleet-snapshot-apply')
|
||||
.set('Cookie', adminCookie)
|
||||
.send({
|
||||
files: [
|
||||
{ filename: 'compose.yaml', content: 'services:\n snap: {}\n' },
|
||||
{ filename: '.env', content: 'SNAP=1\n' },
|
||||
],
|
||||
});
|
||||
|
||||
expect(res.status).toBe(200);
|
||||
expect(res.body.capturedGenerationId).toBe(
|
||||
StackUpdateRecoveryService.getInstance().getCurrent(LOCAL_NODE_ID, 'apply-existing')?.id,
|
||||
);
|
||||
expect(fs.readFileSync(composePath('apply-existing'), 'utf-8')).toContain('snap: {}');
|
||||
expect(fs.readFileSync(envPath('apply-existing'), 'utf-8')).toBe('SNAP=1\n');
|
||||
expectCurrentGenerationMatches('apply-existing', preCompose, preEnv);
|
||||
});
|
||||
|
||||
it('POST /api/stacks/:stackName/fleet-snapshot-apply returns 400 when no restoreable files remain', async () => {
|
||||
seedStackDir('apply-empty');
|
||||
const res = await request(app)
|
||||
.post('/api/stacks/apply-empty/fleet-snapshot-apply')
|
||||
.set('Cookie', adminCookie)
|
||||
.send({ files: [{ filename: 'notes.txt', content: 'nope\n' }] });
|
||||
expect(res.status).toBe(400);
|
||||
expect(fs.existsSync(composePath('apply-empty'))).toBe(false);
|
||||
});
|
||||
|
||||
it('POST /api/stacks/:stackName/fleet-snapshot-apply ignores extra filenames', async () => {
|
||||
seedStackDir('apply-extra');
|
||||
const res = await request(app)
|
||||
.post('/api/stacks/apply-extra/fleet-snapshot-apply')
|
||||
.set('Cookie', adminCookie)
|
||||
.send({
|
||||
files: [
|
||||
{ filename: 'compose.yaml', content: 'services:\n snap: {}\n' },
|
||||
{ filename: 'notes.txt', content: 'should-not-write\n' },
|
||||
{ filename: '.env', content: 'SNAP=1\n' },
|
||||
],
|
||||
});
|
||||
expect(res.status).toBe(200);
|
||||
expect(fs.readFileSync(composePath('apply-extra'), 'utf-8')).toContain('snap: {}');
|
||||
expect(fs.readFileSync(envPath('apply-extra'), 'utf-8')).toBe('SNAP=1\n');
|
||||
expect(fs.existsSync(path.join(process.env.COMPOSE_DIR as string, 'apply-extra', 'notes.txt'))).toBe(false);
|
||||
});
|
||||
|
||||
it('POST /api/stacks/:stackName/fleet-snapshot-apply accepts a combined body over 100 KB', async () => {
|
||||
seedStackDir('apply-large');
|
||||
const compose = `services:\n snap:\n image: nginx\n labels:\n note: "${'x'.repeat(150 * 1024)}"\n`;
|
||||
const res = await request(app)
|
||||
.post('/api/stacks/apply-large/fleet-snapshot-apply')
|
||||
.set('Cookie', adminCookie)
|
||||
.send({ files: [{ filename: 'compose.yaml', content: compose }] });
|
||||
expect(res.status).toBe(200);
|
||||
expect(fs.readFileSync(composePath('apply-large'), 'utf-8')).toBe(compose);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -0,0 +1,283 @@
|
||||
/**
|
||||
* R1: Git apply promote succeeds, deploy fails → applied true, generation current,
|
||||
* compensateWithCandidate is not called.
|
||||
*/
|
||||
import { beforeEach, describe, expect, it, vi } from 'vitest';
|
||||
|
||||
const mockCaptureCandidate = vi.fn();
|
||||
const mockAbandon = vi.fn();
|
||||
const mockMarkAcquired = vi.fn().mockReturnValue(true);
|
||||
const mockHandoff = vi.fn().mockReturnValue(true);
|
||||
const mockMarkReconciling = vi.fn().mockReturnValue(true);
|
||||
const mockMarkImmediateVerified = vi.fn().mockReturnValue(true);
|
||||
const mockGet = vi.fn();
|
||||
const mockCompensate = vi.fn();
|
||||
|
||||
vi.mock('../services/StackUpdateRecoveryService', () => ({
|
||||
StackUpdateRecoveryService: {
|
||||
getInstance: () => ({
|
||||
captureCandidate: mockCaptureCandidate,
|
||||
abandon: mockAbandon,
|
||||
markAcquired: mockMarkAcquired,
|
||||
handoff: mockHandoff,
|
||||
markReconciling: mockMarkReconciling,
|
||||
markImmediateVerified: mockMarkImmediateVerified,
|
||||
get: mockGet,
|
||||
compensateWithCandidate: mockCompensate,
|
||||
}),
|
||||
},
|
||||
}));
|
||||
|
||||
const mockDeployStack = vi.fn();
|
||||
vi.mock('../services/ComposeService', () => ({
|
||||
ComposeService: {
|
||||
getInstance: () => ({
|
||||
deployStack: mockDeployStack,
|
||||
}),
|
||||
},
|
||||
}));
|
||||
|
||||
vi.mock('../services/StackOpLockService', () => ({
|
||||
StackOpLockService: {
|
||||
getInstance: () => ({
|
||||
runExclusive: async (
|
||||
_n: number,
|
||||
_s: string,
|
||||
_a: string,
|
||||
_who: string,
|
||||
fn: () => Promise<unknown>,
|
||||
) => {
|
||||
const result = await fn();
|
||||
return { ran: true, result };
|
||||
},
|
||||
}),
|
||||
},
|
||||
}));
|
||||
|
||||
vi.mock('../helpers/policyGate', () => ({
|
||||
assertPolicyGateAllows: vi.fn().mockResolvedValue(undefined),
|
||||
buildSystemPolicyGateOptions: vi.fn().mockReturnValue({}),
|
||||
}));
|
||||
|
||||
vi.mock('../services/HealthGateService', () => ({
|
||||
HealthGateService: {
|
||||
getInstance: () => ({
|
||||
beginStack: vi.fn(),
|
||||
}),
|
||||
},
|
||||
}));
|
||||
|
||||
vi.mock('../services/NodeRegistry', () => ({
|
||||
NodeRegistry: {
|
||||
getInstance: () => ({
|
||||
getDefaultNodeId: () => 1,
|
||||
}),
|
||||
},
|
||||
}));
|
||||
|
||||
const mockPromoteGeneration = vi.fn().mockResolvedValue(undefined);
|
||||
vi.mock('../services/GitProjectManifestService', () => ({
|
||||
GitProjectManifestService: {
|
||||
getInstance: () => ({
|
||||
readManifest: vi.fn().mockResolvedValue(null),
|
||||
buildManifest: vi.fn().mockReturnValue({
|
||||
manifestVersion: 1,
|
||||
state: 'active',
|
||||
inputs: [],
|
||||
refusals: [],
|
||||
generation: { candidateDir: 'c', appliedDir: 'a', previousDir: null },
|
||||
}),
|
||||
promoteGeneration: mockPromoteGeneration,
|
||||
boundsConfig: vi.fn().mockReturnValue({}),
|
||||
hashStackFile: vi.fn(),
|
||||
verifyContextOnDisk: vi.fn().mockResolvedValue([]),
|
||||
writeManifest: vi.fn(),
|
||||
buildMigratedManifest: vi.fn(),
|
||||
}),
|
||||
},
|
||||
}));
|
||||
|
||||
vi.mock('../utils/authoredComposeArgs', () => ({
|
||||
authoredComposeFileArgs: vi.fn().mockResolvedValue(['-f', 'compose.yaml']),
|
||||
authoredComposeEnvFileArgs: vi.fn().mockResolvedValue([]),
|
||||
}));
|
||||
|
||||
const mockGetGitSource = vi.fn();
|
||||
const mockMarkGitSourceApplied = vi.fn();
|
||||
const mockSetGitSourceAppliedSpec = vi.fn();
|
||||
const mockSetGitSourceManifestState = vi.fn();
|
||||
|
||||
vi.mock('../services/DatabaseService', () => ({
|
||||
DatabaseService: {
|
||||
getInstance: () => ({
|
||||
getGitSource: mockGetGitSource,
|
||||
markGitSourceApplied: mockMarkGitSourceApplied,
|
||||
setGitSourceAppliedSpec: mockSetGitSourceAppliedSpec,
|
||||
setGitSourceManifestState: mockSetGitSourceManifestState,
|
||||
}),
|
||||
},
|
||||
}));
|
||||
|
||||
vi.mock('../services/CryptoService', () => ({
|
||||
CryptoService: {
|
||||
getInstance: () => ({
|
||||
decrypt: (v: string) => v,
|
||||
encrypt: (v: string) => v,
|
||||
}),
|
||||
},
|
||||
}));
|
||||
|
||||
vi.mock('fs', async () => {
|
||||
const actual = await vi.importActual<typeof import('fs')>('fs');
|
||||
return {
|
||||
...actual,
|
||||
promises: {
|
||||
...actual.promises,
|
||||
access: vi.fn().mockResolvedValue(undefined),
|
||||
},
|
||||
};
|
||||
});
|
||||
|
||||
describe('git-source apply recovery (R1)', () => {
|
||||
beforeEach(() => {
|
||||
vi.clearAllMocks();
|
||||
mockCaptureCandidate.mockResolvedValue({
|
||||
id: 'rec-1',
|
||||
node_id: 1,
|
||||
stack_name: 'app',
|
||||
status: 'candidate',
|
||||
phase: 'captured',
|
||||
is_current: 0,
|
||||
});
|
||||
mockGet.mockReturnValue({
|
||||
id: 'rec-1',
|
||||
is_current: 1,
|
||||
status: 'active',
|
||||
phase: 'reconciling',
|
||||
});
|
||||
mockDeployStack.mockRejectedValue(new Error('compose up failed'));
|
||||
mockGetGitSource.mockReturnValue({
|
||||
stack_name: 'app',
|
||||
repo_url: 'https://example.com/repo.git',
|
||||
branch: 'main',
|
||||
pending_commit_sha: 'abc1234deadbeef',
|
||||
pending_compose_content: JSON.stringify({
|
||||
v: 3,
|
||||
files: { 'compose.yaml': 'services:\n web:\n image: nginx\n' },
|
||||
contextDir: null,
|
||||
candidateRelPath: 'generations/cand',
|
||||
inventory: {
|
||||
inputs: [],
|
||||
refusals: [],
|
||||
buildContexts: [],
|
||||
},
|
||||
}),
|
||||
pending_env_content: null,
|
||||
sync_env: false,
|
||||
compose_paths: ['compose.yaml'],
|
||||
context_dir: null,
|
||||
auto_deploy_on_apply: true,
|
||||
applied_deploy_spec: null,
|
||||
});
|
||||
});
|
||||
|
||||
it('keeps applied=true and generation current without compensate when deploy fails', async () => {
|
||||
const { GitSourceService } = await import('../services/GitSourceService');
|
||||
// Avoid withStackLock contention by calling applyLocked through apply
|
||||
// after stubbing the lock if present.
|
||||
const svc = GitSourceService.getInstance();
|
||||
const withLock = vi.spyOn(
|
||||
svc as unknown as { withStackLock: (name: string, fn: () => Promise<unknown>) => Promise<unknown> },
|
||||
'withStackLock',
|
||||
);
|
||||
withLock.mockImplementation(async (_name, fn) => fn());
|
||||
|
||||
// validateCandidate is used on the v3 path; stub it open.
|
||||
vi.spyOn(
|
||||
svc as unknown as {
|
||||
validateCandidate: (...args: unknown[]) => Promise<{ ok: boolean }>;
|
||||
},
|
||||
'validateCandidate',
|
||||
).mockResolvedValue({ ok: true });
|
||||
|
||||
vi.spyOn(
|
||||
svc as unknown as {
|
||||
decodePendingCompose: (raw: string) => unknown;
|
||||
},
|
||||
'decodePendingCompose',
|
||||
).mockReturnValue({
|
||||
files: { 'compose.yaml': 'services:\n web:\n image: nginx\n' },
|
||||
contextDir: null,
|
||||
candidateRelPath: 'generations/cand',
|
||||
inventory: { inputs: [], refusals: [], buildContexts: [] },
|
||||
});
|
||||
|
||||
vi.spyOn(
|
||||
svc as unknown as {
|
||||
deriveAppliedSpec: (...args: unknown[]) => unknown;
|
||||
},
|
||||
'deriveAppliedSpec',
|
||||
).mockReturnValue({ files: ['compose.yaml'], contextDir: null });
|
||||
|
||||
vi.spyOn(
|
||||
svc as unknown as {
|
||||
hashContent: (...args: unknown[]) => string;
|
||||
},
|
||||
'hashContent',
|
||||
).mockReturnValue('hash');
|
||||
|
||||
const result = await svc.apply('app', 'abc1234deadbeef', { deploy: true, actor: 'tester' });
|
||||
|
||||
expect(result.applied).toBe(true);
|
||||
expect(result.deployed).toBe(false);
|
||||
expect(result.deployError).toBeTruthy();
|
||||
expect(result.recoveryId).toBe('rec-1');
|
||||
expect(mockPromoteGeneration).toHaveBeenCalled();
|
||||
expect(mockCaptureCandidate).toHaveBeenCalledWith(
|
||||
expect.objectContaining({ operationKind: 'git_apply', stackName: 'app' }),
|
||||
);
|
||||
expect(mockHandoff).toHaveBeenCalled();
|
||||
expect(mockCompensate).not.toHaveBeenCalled();
|
||||
expect(mockAbandon).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('refuses to promote when recovery capture fails', async () => {
|
||||
mockCaptureCandidate.mockRejectedValue(new Error('Exact authored-project rollback coverage is unavailable'));
|
||||
mockDeployStack.mockResolvedValue({ recoveryId: null });
|
||||
|
||||
const { GitSourceService, GitSourceError } = await import('../services/GitSourceService');
|
||||
const svc = GitSourceService.getInstance();
|
||||
vi.spyOn(
|
||||
svc as unknown as { withStackLock: (name: string, fn: () => Promise<unknown>) => Promise<unknown> },
|
||||
'withStackLock',
|
||||
).mockImplementation(async (_name, fn) => fn());
|
||||
vi.spyOn(
|
||||
svc as unknown as { validateCandidate: (...args: unknown[]) => Promise<{ ok: boolean }> },
|
||||
'validateCandidate',
|
||||
).mockResolvedValue({ ok: true });
|
||||
vi.spyOn(
|
||||
svc as unknown as { decodePendingCompose: (raw: string) => unknown },
|
||||
'decodePendingCompose',
|
||||
).mockReturnValue({
|
||||
files: { 'compose.yaml': 'services:\n web:\n image: nginx\n' },
|
||||
contextDir: null,
|
||||
candidateRelPath: 'generations/cand',
|
||||
inventory: { inputs: [], refusals: [], buildContexts: [] },
|
||||
});
|
||||
vi.spyOn(
|
||||
svc as unknown as { deriveAppliedSpec: (...args: unknown[]) => unknown },
|
||||
'deriveAppliedSpec',
|
||||
).mockReturnValue({ files: ['compose.yaml'], contextDir: null });
|
||||
vi.spyOn(
|
||||
svc as unknown as { hashContent: (...args: unknown[]) => string },
|
||||
'hashContent',
|
||||
).mockReturnValue('hash');
|
||||
|
||||
await expect(svc.apply('app', 'abc1234deadbeef', { deploy: true, actor: 'tester' })).rejects.toBeInstanceOf(GitSourceError);
|
||||
expect(mockPromoteGeneration).not.toHaveBeenCalled();
|
||||
expect(mockCaptureCandidate).toHaveBeenCalled();
|
||||
expect(mockMarkGitSourceApplied).not.toHaveBeenCalled();
|
||||
expect(mockHandoff).not.toHaveBeenCalled();
|
||||
expect(mockAbandon).not.toHaveBeenCalled();
|
||||
});
|
||||
});
|
||||
@@ -32,6 +32,44 @@ vi.mock('isomorphic-git', () => {
|
||||
|
||||
vi.mock('isomorphic-git/http/node', () => ({ default: {} }));
|
||||
|
||||
|
||||
const {
|
||||
mockCaptureCandidate,
|
||||
mockRecoveryAbandon,
|
||||
mockRecoveryMarkAcquired,
|
||||
mockRecoveryHandoff,
|
||||
mockRecoveryMarkReconciling,
|
||||
mockRecoveryMarkImmediateVerified,
|
||||
mockRecoveryGet,
|
||||
mockRecoveryLinkGateOrRetain,
|
||||
} = vi.hoisted(() => ({
|
||||
mockCaptureCandidate: vi.fn(async () => ({ id: 'rec-test-1' })),
|
||||
mockRecoveryAbandon: vi.fn(async () => true),
|
||||
mockRecoveryMarkAcquired: vi.fn(() => true),
|
||||
mockRecoveryHandoff: vi.fn(() => true),
|
||||
mockRecoveryMarkReconciling: vi.fn(() => true),
|
||||
mockRecoveryMarkImmediateVerified: vi.fn(() => true),
|
||||
mockRecoveryGet: vi.fn(() => ({ id: 'rec-test-1', is_current: 1 })),
|
||||
mockRecoveryLinkGateOrRetain: vi.fn(),
|
||||
}));
|
||||
|
||||
vi.mock('../services/StackUpdateRecoveryService', () => ({
|
||||
StackUpdateRecoveryService: {
|
||||
getInstance: () => ({
|
||||
captureCandidate: mockCaptureCandidate,
|
||||
abandon: mockRecoveryAbandon,
|
||||
markAcquired: mockRecoveryMarkAcquired,
|
||||
handoff: mockRecoveryHandoff,
|
||||
markReconciling: mockRecoveryMarkReconciling,
|
||||
markImmediateVerified: mockRecoveryMarkImmediateVerified,
|
||||
get: mockRecoveryGet,
|
||||
linkGateOrRetain: mockRecoveryLinkGateOrRetain,
|
||||
compensateWithCandidate: vi.fn(async () => true),
|
||||
}),
|
||||
},
|
||||
}));
|
||||
|
||||
|
||||
let tmpDir: string;
|
||||
let GitSourceService: typeof import('../services/GitSourceService').GitSourceService;
|
||||
let GitSourceError: typeof import('../services/GitSourceService').GitSourceError;
|
||||
@@ -50,6 +88,21 @@ afterAll(() => {
|
||||
beforeEach(() => {
|
||||
mockGitClone.mockReset();
|
||||
mockGitLog.mockReset();
|
||||
mockCaptureCandidate.mockReset();
|
||||
mockCaptureCandidate.mockImplementation(async () => ({ id: 'rec-test-1' }));
|
||||
mockRecoveryAbandon.mockReset();
|
||||
mockRecoveryAbandon.mockResolvedValue(true);
|
||||
mockRecoveryMarkAcquired.mockReset();
|
||||
mockRecoveryMarkAcquired.mockReturnValue(true);
|
||||
mockRecoveryHandoff.mockReset();
|
||||
mockRecoveryHandoff.mockReturnValue(true);
|
||||
mockRecoveryMarkReconciling.mockReset();
|
||||
mockRecoveryMarkReconciling.mockReturnValue(true);
|
||||
mockRecoveryMarkImmediateVerified.mockReset();
|
||||
mockRecoveryMarkImmediateVerified.mockReturnValue(true);
|
||||
mockRecoveryGet.mockReset();
|
||||
mockRecoveryLinkGateOrRetain.mockReset();
|
||||
mockRecoveryGet.mockReturnValue({ id: 'rec-test-1', is_current: 1 });
|
||||
|
||||
// Wipe persisted git sources between tests
|
||||
const db = DatabaseService.getInstance();
|
||||
@@ -894,6 +947,47 @@ describe('GitSourceService.handleWebhookPull debounce', () => {
|
||||
expect(result.message).toMatch(/validation failed/i);
|
||||
runSpy.mockRestore();
|
||||
});
|
||||
|
||||
it('routes webhook auto-apply through the shared stack-operation lock', async () => {
|
||||
const sha = 'ffff666ffff666ffff666ffff666ffff666ffff6';
|
||||
mockSuccessfulClone({ sha });
|
||||
const svc = GitSourceService.getInstance();
|
||||
const validateSpy = vi.spyOn(svc, 'validateCompose').mockResolvedValue({ ok: true });
|
||||
await svc.upsert({
|
||||
stackName: 'webhook-shared-lock',
|
||||
repoUrl: 'https://github.com/example/repo.git',
|
||||
branch: 'main',
|
||||
composePaths: ['compose.yaml'],
|
||||
contextDir: null,
|
||||
syncEnv: false,
|
||||
envPath: null,
|
||||
authType: 'none',
|
||||
autoApplyOnWebhook: true,
|
||||
autoDeployOnApply: false,
|
||||
});
|
||||
mockGitClone.mockClear();
|
||||
|
||||
const { StackOpLockService } = await import('../services/StackOpLockService');
|
||||
const runExclusive = vi.spyOn(StackOpLockService.getInstance(), 'runExclusive')
|
||||
.mockResolvedValue({
|
||||
ran: false,
|
||||
existing: { action: 'update', actor: 'user:admin', startedAt: Date.now() },
|
||||
} as never);
|
||||
|
||||
const result = await svc.handleWebhookPull('webhook-shared-lock');
|
||||
expect(result.status).toBe('error');
|
||||
expect(result.message).toMatch(/already in progress/i);
|
||||
expect(runExclusive).toHaveBeenCalledWith(
|
||||
expect.any(Number),
|
||||
'webhook-shared-lock',
|
||||
'git_apply',
|
||||
'system:webhook',
|
||||
expect.any(Function),
|
||||
);
|
||||
|
||||
runExclusive.mockRestore();
|
||||
validateSpy.mockRestore();
|
||||
});
|
||||
});
|
||||
|
||||
describe('GitSourceService per-stack mutex', () => {
|
||||
@@ -1359,7 +1453,7 @@ describe('GitSourceService.apply', () => {
|
||||
const { ComposeService } = await import('../services/ComposeService');
|
||||
const { HealthGateService } = await import('../services/HealthGateService');
|
||||
const saveSpy = vi.spyOn(FileSystemService.prototype, 'saveStackContent').mockResolvedValue();
|
||||
const deploySpy = vi.spyOn(ComposeService.prototype, 'deployStack').mockResolvedValue();
|
||||
const deploySpy = vi.spyOn(ComposeService.prototype, 'deployStack').mockResolvedValue({ recoveryId: null });
|
||||
const beginSpy = vi.spyOn(HealthGateService.getInstance(), 'beginStack').mockReturnValue('gate-git');
|
||||
const nodeId = DatabaseService.getInstance().getDefaultNode()!.id!;
|
||||
|
||||
@@ -1371,6 +1465,7 @@ describe('GitSourceService.apply', () => {
|
||||
actor: 'system:git-source',
|
||||
});
|
||||
expect(beginSpy).toHaveBeenCalledWith(nodeId, 'apply-deploy-gate', 'deploy', 'system:git-source');
|
||||
expect(mockRecoveryLinkGateOrRetain).toHaveBeenCalledWith('rec-test-1', 'gate-git');
|
||||
} finally {
|
||||
validateSpy.mockRestore();
|
||||
saveSpy.mockRestore();
|
||||
@@ -1470,7 +1565,7 @@ describe('GitSourceService.apply', () => {
|
||||
const TrivyService = (await import('../services/TrivyService')).default;
|
||||
const saveSpy = vi.spyOn(FileSystemService.prototype, 'saveStackContent').mockResolvedValue();
|
||||
const listImagesSpy = vi.spyOn(ComposeService.prototype, 'listStackImages').mockResolvedValue(['nginx:bad']);
|
||||
const deploySpy = vi.spyOn(ComposeService.prototype, 'deployStack').mockResolvedValue();
|
||||
const deploySpy = vi.spyOn(ComposeService.prototype, 'deployStack').mockResolvedValue({ recoveryId: null });
|
||||
const trivy = TrivyService.getInstance();
|
||||
const trivyAvailableSpy = vi.spyOn(trivy, 'isTrivyAvailable').mockReturnValue(true);
|
||||
const scanSpy = vi.spyOn(trivy, 'scanImagePreflight').mockResolvedValue({
|
||||
@@ -2178,6 +2273,50 @@ describe('GitSourceService legacy pending apply (migration path)', () => {
|
||||
await cleanupStackDir('legacy-apply');
|
||||
}
|
||||
});
|
||||
|
||||
it('rejects materialize when deleting a stale override fails', async () => {
|
||||
const svc = GitSourceService.getInstance();
|
||||
const { FileSystemService } = await import('../services/FileSystemService');
|
||||
const fsSvc = FileSystemService.getInstance();
|
||||
const stackName = 'legacy-stale-del';
|
||||
await fsSvc.createStack(stackName);
|
||||
await fsSvc.saveStackContent(stackName, 'services:\n web:\n image: nginx\n');
|
||||
await fsSvc.writeStackFile(stackName, 'compose.override.yaml', 'services:\n web:\n environment: [X=1]\n');
|
||||
|
||||
const deleteSpy = vi.spyOn(FileSystemService.prototype, 'deleteStackPath')
|
||||
.mockRejectedValue(Object.assign(new Error('permission denied'), { code: 'EACCES' }));
|
||||
|
||||
const materialize = (svc as unknown as {
|
||||
materialize: (
|
||||
stackName: string,
|
||||
files: Array<{ path: string; content: string }>,
|
||||
contextDir: string | null,
|
||||
syncEnv: boolean,
|
||||
envContent: string | null,
|
||||
prevSpec: { files: string[]; contextDir: string | null } | null,
|
||||
) => Promise<unknown>;
|
||||
}).materialize.bind(svc);
|
||||
|
||||
try {
|
||||
await expect(
|
||||
materialize(
|
||||
stackName,
|
||||
[{ path: 'compose.yaml', content: 'services:\n web:\n image: alpine\n' }],
|
||||
null,
|
||||
false,
|
||||
null,
|
||||
{ files: ['compose.yaml', 'compose.override.yaml'], contextDir: null },
|
||||
),
|
||||
).rejects.toThrow(/permission denied/);
|
||||
expect(deleteSpy).toHaveBeenCalledWith(stackName, 'compose.override.yaml');
|
||||
// Stale override must still be present; apply must not report success over a hybrid.
|
||||
const override = await fsSvc.readStackFile(stackName, 'compose.override.yaml');
|
||||
expect(override.content).toContain('X=1');
|
||||
} finally {
|
||||
deleteSpy.mockRestore();
|
||||
await cleanupStackDir(stackName);
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe('GitSourceService sync-env stacks with a repo .env (audit C-2)', () => {
|
||||
|
||||
@@ -43,6 +43,7 @@ describe('named stack route permission inventory', () => {
|
||||
['PUT', '/stacks/web/env', 'stack:edit'],
|
||||
['PUT', '/stacks/web/dossier', 'stack:edit'],
|
||||
['PUT', '/stacks/web/labels', 'stack:edit'],
|
||||
['POST', '/stacks/web/fleet-snapshot-apply', 'stack:edit'],
|
||||
['POST', '/stacks/web/deploy', 'stack:deploy'],
|
||||
['POST', '/stacks/web/stop', 'stack:deploy'],
|
||||
['POST', '/stacks/web/services/api/update', 'stack:deploy'],
|
||||
|
||||
@@ -0,0 +1,102 @@
|
||||
/**
|
||||
* Recovery must execute the generation-captured Compose invocation, not the
|
||||
* live database-derived file / env selection after capture.
|
||||
*/
|
||||
import { afterEach, beforeEach, describe, expect, it } from 'vitest';
|
||||
import path from 'path';
|
||||
import { setupTestDb, cleanupTestDb } from './helpers/setupTestDb';
|
||||
|
||||
describe('captured invocation on recovery Compose args', () => {
|
||||
let tmpDir: string;
|
||||
let composeDir: string;
|
||||
let nodeId: number;
|
||||
|
||||
beforeEach(async () => {
|
||||
tmpDir = await setupTestDb();
|
||||
composeDir = process.env.COMPOSE_DIR!;
|
||||
const { DatabaseService } = await import('../services/DatabaseService');
|
||||
const db = DatabaseService.getInstance();
|
||||
const local = db.getDefaultNode();
|
||||
if (!local?.id) throw new Error('missing default node');
|
||||
nodeId = local.id;
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
if (tmpDir) cleanupTestDb(tmpDir);
|
||||
});
|
||||
|
||||
it('uses captured -f order and env-file when live settings diverge', async () => {
|
||||
const stackName = 'inv-capture';
|
||||
const stackDir = path.join(composeDir, stackName);
|
||||
const fsPromises = await import('fs/promises');
|
||||
await fsPromises.mkdir(stackDir, { recursive: true });
|
||||
await fsPromises.writeFile(path.join(stackDir, 'compose.yaml'), 'services:\n a: {}\n', 'utf8');
|
||||
await fsPromises.writeFile(path.join(stackDir, 'compose.prod.yaml'), 'services:\n b: {}\n', 'utf8');
|
||||
await fsPromises.writeFile(path.join(stackDir, 'prod.env'), 'A=1\n', 'utf8');
|
||||
await fsPromises.writeFile(path.join(stackDir, 'dev.env'), 'A=2\n', 'utf8');
|
||||
|
||||
const { DatabaseService } = await import('../services/DatabaseService');
|
||||
const db = DatabaseService.getInstance();
|
||||
db.upsertGitSource({
|
||||
stack_name: stackName,
|
||||
repo_url: 'https://example.com/repo.git',
|
||||
branch: 'main',
|
||||
compose_path: 'compose.yaml',
|
||||
compose_paths: ['compose.yaml', 'compose.prod.yaml'],
|
||||
context_dir: null,
|
||||
sync_env: false,
|
||||
env_path: null,
|
||||
auth_type: 'none',
|
||||
encrypted_token: null,
|
||||
auto_apply_on_webhook: false,
|
||||
auto_deploy_on_apply: false,
|
||||
last_applied_commit_sha: 'abc',
|
||||
last_applied_content_hash: 'h',
|
||||
pending_commit_sha: null,
|
||||
pending_compose_content: null,
|
||||
pending_env_content: null,
|
||||
pending_fetched_at: null,
|
||||
last_debounce_at: null,
|
||||
});
|
||||
db.getDb().prepare(
|
||||
`UPDATE stack_git_sources SET applied_deploy_spec = ? WHERE stack_name = ?`,
|
||||
).run(
|
||||
JSON.stringify({ files: ['compose.yaml'], contextDir: null }),
|
||||
stackName,
|
||||
);
|
||||
db.setStackProjectEnvFiles(nodeId, stackName, ['dev.env']);
|
||||
|
||||
const { ComposeService } = await import('../services/ComposeService');
|
||||
const svc = ComposeService.getInstance(nodeId);
|
||||
const captured = {
|
||||
composeArgsPrefix: [
|
||||
'-f', 'compose.yaml',
|
||||
'-f', 'compose.prod.yaml',
|
||||
'-p', stackName,
|
||||
'--env-file', path.join(stackDir, 'prod.env'),
|
||||
],
|
||||
projectDirectory: null,
|
||||
projectName: stackName,
|
||||
explicitComposeFiles: ['compose.yaml', 'compose.prod.yaml'],
|
||||
};
|
||||
|
||||
const args = await svc.buildComposeArgsWithRecoveryOverride(
|
||||
stackName,
|
||||
['up', '-d'],
|
||||
path.join(stackDir, '.sencho-recovery-aaaaaaaaaaaa.yml'),
|
||||
captured,
|
||||
);
|
||||
|
||||
expect(args).toEqual([
|
||||
'compose',
|
||||
'-f', 'compose.yaml',
|
||||
'-f', 'compose.prod.yaml',
|
||||
'-p', stackName,
|
||||
'--env-file', path.resolve(stackDir, 'prod.env'),
|
||||
'-f', path.join(stackDir, '.sencho-recovery-aaaaaaaaaaaa.yml'),
|
||||
'up', '-d',
|
||||
]);
|
||||
// Live settings would prefer only compose.yaml + dev.env; captured must win.
|
||||
expect(args.join('\0')).not.toContain('dev.env');
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,70 @@
|
||||
/**
|
||||
* Fifth-audit regressions: structural services_json validation.
|
||||
*/
|
||||
import { describe, expect, it } from 'vitest';
|
||||
import {
|
||||
parseServicesJsonStrict,
|
||||
scrapeRollbackTagsLenient,
|
||||
type StackRecoveryServiceCapture,
|
||||
} from '../services/recoveryServicesJson';
|
||||
|
||||
function validService(over: Partial<StackRecoveryServiceCapture> = {}): StackRecoveryServiceCapture {
|
||||
return {
|
||||
serviceName: 'web',
|
||||
scale: 1,
|
||||
hasBuild: false,
|
||||
declaredImageRef: 'nginx:latest',
|
||||
referenceKind: 'moving_tag',
|
||||
replicas: [{
|
||||
containerId: 'c1',
|
||||
imageId: 'sha256:aaa',
|
||||
repoDigest: null,
|
||||
state: 'running',
|
||||
rollbackTag: 'sencho-rb/aaaaaaaaaaaa/web:hold',
|
||||
}],
|
||||
...over,
|
||||
};
|
||||
}
|
||||
|
||||
describe('parseServicesJsonStrict', () => {
|
||||
it('accepts a well-formed services array', () => {
|
||||
const parsed = parseServicesJsonStrict(JSON.stringify([validService()]));
|
||||
expect(parsed.ok).toBe(true);
|
||||
if (parsed.ok) expect(parsed.services[0].serviceName).toBe('web');
|
||||
});
|
||||
|
||||
it('rejects empty object elements like [{}]', () => {
|
||||
expect(parseServicesJsonStrict('[{}]').ok).toBe(false);
|
||||
});
|
||||
|
||||
it('rejects malformed replica arrays', () => {
|
||||
expect(parseServicesJsonStrict(JSON.stringify([{
|
||||
...validService(),
|
||||
replicas: [{ state: 'running' }],
|
||||
}])).ok).toBe(false);
|
||||
|
||||
expect(parseServicesJsonStrict(JSON.stringify([{
|
||||
...validService(),
|
||||
replicas: 'nope',
|
||||
}])).ok).toBe(false);
|
||||
});
|
||||
|
||||
it('scrapeRollbackTagsLenient recovers tags from near-valid JSON', () => {
|
||||
expect(scrapeRollbackTagsLenient(JSON.stringify([{
|
||||
serviceName: 'web',
|
||||
replicas: [{ rollbackTag: 'sencho-rb/aaaaaaaaaaaa/web:hold' }],
|
||||
}]))).toEqual(['sencho-rb/aaaaaaaaaaaa/web:hold']);
|
||||
});
|
||||
|
||||
it('rejects invalid referenceKind or scale', () => {
|
||||
expect(parseServicesJsonStrict(JSON.stringify([{
|
||||
...validService(),
|
||||
referenceKind: 'mystery',
|
||||
}])).ok).toBe(false);
|
||||
|
||||
expect(parseServicesJsonStrict(JSON.stringify([{
|
||||
...validService(),
|
||||
scale: -1,
|
||||
}])).ok).toBe(false);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,103 @@
|
||||
/**
|
||||
* Integration coverage for assessGenerationEligibility: structural services_json
|
||||
* refusal and opaque hold-tag presence checks.
|
||||
*/
|
||||
import { beforeEach, describe, expect, it, vi } from 'vitest';
|
||||
|
||||
const mockInspectByRef = vi.fn(async (_ref: string) => ({ Id: 'ok' }));
|
||||
|
||||
vi.mock('../services/DockerController', () => ({
|
||||
default: {
|
||||
getInstance: () => ({
|
||||
getDocker: () => ({
|
||||
getImage: (ref: string) => ({
|
||||
inspect: () => mockInspectByRef(ref),
|
||||
}),
|
||||
}),
|
||||
}),
|
||||
},
|
||||
}));
|
||||
|
||||
vi.mock('../services/RollbackGenerationStore', () => ({
|
||||
RollbackGenerationStore: {
|
||||
verifyGenerationContent: vi.fn().mockResolvedValue(true),
|
||||
},
|
||||
}));
|
||||
|
||||
vi.mock('../services/PolicyEnforcement', () => ({
|
||||
enforcePolicyForImageRefs: vi.fn().mockResolvedValue({ ok: true, bypassed: false, violations: [] }),
|
||||
}));
|
||||
|
||||
import { assessGenerationEligibility } from '../services/rollbackEligibility';
|
||||
import type { StackUpdateRecoveryGenerationRow } from '../services/DatabaseService';
|
||||
|
||||
const HOLD_TAG = 'sencho-rb/aaaaaaaaaaaa/web:hold';
|
||||
const IMAGE_ID = 'sha256:aaa';
|
||||
|
||||
function baseRow(over: Partial<StackUpdateRecoveryGenerationRow> = {}): StackUpdateRecoveryGenerationRow {
|
||||
return {
|
||||
id: '11111111-1111-4111-8111-111111111111',
|
||||
node_id: 1,
|
||||
stack_name: 'my-stack',
|
||||
status: 'active',
|
||||
phase: 'reconciling',
|
||||
is_current: 1,
|
||||
backup_slot_id: '11111111-1111-4111-8111-111111111111',
|
||||
content_path: '11111111-1111-4111-8111-111111111111',
|
||||
operation_kind: 'update',
|
||||
override_path: null,
|
||||
services_json: JSON.stringify([{
|
||||
serviceName: 'web',
|
||||
scale: 1,
|
||||
hasBuild: false,
|
||||
declaredImageRef: 'nginx:latest',
|
||||
referenceKind: 'moving_tag',
|
||||
replicas: [{
|
||||
containerId: 'c1',
|
||||
imageId: IMAGE_ID,
|
||||
repoDigest: null,
|
||||
state: 'running',
|
||||
rollbackTag: HOLD_TAG,
|
||||
}],
|
||||
}]),
|
||||
health_gate_id: null,
|
||||
gate_retain_until: null,
|
||||
artifact_expires_at: null,
|
||||
operation_lease_expires_at: null,
|
||||
created_at: Date.now(),
|
||||
updated_at: Date.now(),
|
||||
created_by: null,
|
||||
artifacts_retired: 0,
|
||||
released_at: null,
|
||||
released_by: null,
|
||||
...over,
|
||||
};
|
||||
}
|
||||
|
||||
describe('assessGenerationEligibility', () => {
|
||||
beforeEach(() => {
|
||||
vi.clearAllMocks();
|
||||
mockInspectByRef.mockResolvedValue({ Id: 'ok' });
|
||||
});
|
||||
|
||||
it('prohibits structurally malformed services_json such as [{}]', async () => {
|
||||
await expect(assessGenerationEligibility(baseRow({ services_json: '[{}]' }))).resolves.toBe('prohibited');
|
||||
});
|
||||
|
||||
it('returns eligible_with_warning when an opaque hold tag is missing', async () => {
|
||||
mockInspectByRef.mockImplementation(async (ref: string) => {
|
||||
if (ref === HOLD_TAG) {
|
||||
throw Object.assign(new Error('No such image'), { statusCode: 404 });
|
||||
}
|
||||
return { Id: 'ok' };
|
||||
});
|
||||
|
||||
await expect(assessGenerationEligibility(baseRow())).resolves.toBe('eligible_with_warning');
|
||||
expect(mockInspectByRef).toHaveBeenCalledWith(IMAGE_ID);
|
||||
expect(mockInspectByRef).toHaveBeenCalledWith(HOLD_TAG);
|
||||
});
|
||||
|
||||
it('returns eligible when image ids and hold tags both resolve', async () => {
|
||||
await expect(assessGenerationEligibility(baseRow())).resolves.toBe('eligible');
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,55 @@
|
||||
/**
|
||||
* Unit tests for evaluateRollbackEligibility (pure verdict mapping).
|
||||
*/
|
||||
import { describe, expect, it } from 'vitest';
|
||||
import {
|
||||
evaluateRollbackEligibility,
|
||||
type RollbackEligibilityInput,
|
||||
} from '../services/rollbackEligibility';
|
||||
|
||||
const base = (over: Partial<RollbackEligibilityInput> = {}): RollbackEligibilityInput => ({
|
||||
generationIntegrityOk: true,
|
||||
heldImagesPresent: true,
|
||||
securityPostureBlocked: false,
|
||||
...over,
|
||||
});
|
||||
|
||||
describe('evaluateRollbackEligibility', () => {
|
||||
it('returns eligible when every signal is known-good', () => {
|
||||
expect(evaluateRollbackEligibility(base())).toBe('eligible');
|
||||
});
|
||||
|
||||
it('returns prohibited when security posture is blocked', () => {
|
||||
expect(evaluateRollbackEligibility(base({ securityPostureBlocked: true }))).toBe('prohibited');
|
||||
});
|
||||
|
||||
it('returns prohibited when generation integrity is known bad', () => {
|
||||
expect(evaluateRollbackEligibility(base({ generationIntegrityOk: false }))).toBe('prohibited');
|
||||
});
|
||||
|
||||
it('prefers prohibited over other signals when security is blocked', () => {
|
||||
expect(evaluateRollbackEligibility(base({
|
||||
securityPostureBlocked: true,
|
||||
generationIntegrityOk: null,
|
||||
heldImagesPresent: false,
|
||||
}))).toBe('prohibited');
|
||||
});
|
||||
|
||||
it('returns eligible_with_warning when held images are missing', () => {
|
||||
expect(evaluateRollbackEligibility(base({ heldImagesPresent: false }))).toBe('eligible_with_warning');
|
||||
});
|
||||
|
||||
it('returns unknown when any remaining signal is null', () => {
|
||||
expect(evaluateRollbackEligibility(base({ generationIntegrityOk: null }))).toBe('unknown');
|
||||
expect(evaluateRollbackEligibility(base({ heldImagesPresent: null }))).toBe('unknown');
|
||||
expect(evaluateRollbackEligibility(base({ securityPostureBlocked: null }))).toBe('unknown');
|
||||
});
|
||||
|
||||
it('returns unknown when all signals are null', () => {
|
||||
expect(evaluateRollbackEligibility({
|
||||
generationIntegrityOk: null,
|
||||
heldImagesPresent: null,
|
||||
securityPostureBlocked: null,
|
||||
})).toBe('unknown');
|
||||
});
|
||||
});
|
||||
@@ -82,6 +82,8 @@ function makeRow(overrides: Partial<StackUpdateRecoveryGenerationRow> = {}): Sta
|
||||
phase: 'immediate_verified',
|
||||
is_current: 1,
|
||||
backup_slot_id: null,
|
||||
content_path: null,
|
||||
operation_kind: null,
|
||||
override_path: null,
|
||||
services_json: JSON.stringify([{
|
||||
serviceName: 'web',
|
||||
@@ -349,6 +351,22 @@ describe('StackUpdateRecoveryService.releaseGeneration', () => {
|
||||
if (!second.ok) expect(second.reason).toBe('already_released');
|
||||
});
|
||||
|
||||
it('refuses release when services_json is malformed and leaves hold tags intact', async () => {
|
||||
const row = insertRow({ status: 'active', is_current: 1, services_json: '{not-json' });
|
||||
const svc = StackUpdateRecoveryService.getInstance();
|
||||
|
||||
expect(svc.isReleaseEligible(row)).toBe(false);
|
||||
const result = await svc.releaseGeneration(row.id, 'tester');
|
||||
expect(result.ok).toBe(false);
|
||||
if (!result.ok) expect(result.reason).toBe('malformed_services');
|
||||
|
||||
const after = DatabaseService.getInstance().getStackUpdateRecoveryGeneration(row.id)!;
|
||||
expect(after.released_at).toBeNull();
|
||||
expect(after.is_current).toBe(1);
|
||||
expect(after.artifacts_retired).toBe(0);
|
||||
expect(mockRemove).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('leaves artifacts_retired at 0 (retryable) when Docker tag removal fails', async () => {
|
||||
mockRemove.mockRejectedValueOnce(Object.assign(new Error('docker busy'), { statusCode: 500 }));
|
||||
const row = insertRow({ status: 'active', is_current: 1 });
|
||||
@@ -415,6 +433,18 @@ describe('GET/POST /api/system/rollback/generations', () => {
|
||||
expect(res.status).toBe(409);
|
||||
expect(res.body.code).toBe('NOT_ELIGIBLE');
|
||||
});
|
||||
|
||||
it('409s releasing a generation with malformed services_json', async () => {
|
||||
const row = insertRow({ status: 'superseded', is_current: 0, services_json: '{not-json' });
|
||||
const res = await request(app)
|
||||
.post(`/api/system/rollback/generations/${row.id}/release`)
|
||||
.set('Authorization', authHeader);
|
||||
|
||||
expect(res.status).toBe(409);
|
||||
expect(res.body.code).toBe('MALFORMED_SERVICES');
|
||||
const after = DatabaseService.getInstance().getStackUpdateRecoveryGeneration(row.id)!;
|
||||
expect(after.released_at).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
describe('RBAC on the rollback-generation routes', () => {
|
||||
|
||||
@@ -0,0 +1,788 @@
|
||||
/**
|
||||
* Unit tests for RollbackGenerationStore: capture staging, checksum verify,
|
||||
* restore round-trip, mid-capture failure isolation, and managed-set tombstones.
|
||||
*/
|
||||
import { describe, it, expect, beforeEach, afterEach, vi } from 'vitest';
|
||||
import path from 'path';
|
||||
import os from 'os';
|
||||
import { promises as fsPromises } from 'fs';
|
||||
import { randomUUID } from 'crypto';
|
||||
|
||||
const mockState = { composeDir: '', composeDirs: new Map<number, string>() };
|
||||
|
||||
vi.mock('../services/NodeRegistry', () => ({
|
||||
NodeRegistry: {
|
||||
getInstance: () => ({
|
||||
getComposeDir: (nodeId?: number) => mockState.composeDirs.get(nodeId ?? 1) ?? mockState.composeDir,
|
||||
getDefaultNodeId: () => 1,
|
||||
}),
|
||||
},
|
||||
}));
|
||||
|
||||
vi.mock('../utils/debug', () => ({
|
||||
isDebugEnabled: () => false,
|
||||
}));
|
||||
|
||||
import { RollbackGenerationStore } from '../services/RollbackGenerationStore';
|
||||
import type { ResolvedRollbackInventory } from '../types/rollbackGeneration';
|
||||
|
||||
const NODE = 1;
|
||||
|
||||
function inventoryFor(
|
||||
stackName: string,
|
||||
files: Array<{
|
||||
relativePath: string;
|
||||
absolutePath: string | null;
|
||||
kind?: ResolvedRollbackInventory['entries'][number]['dependencyKind'];
|
||||
sensitivity?: ResolvedRollbackInventory['entries'][number]['sensitivity'];
|
||||
}>,
|
||||
): ResolvedRollbackInventory {
|
||||
return {
|
||||
entries: files.map((f) => ({
|
||||
relativePath: f.relativePath,
|
||||
dependencyKind: f.kind ?? 'compose-root',
|
||||
provenance: 'authored' as const,
|
||||
sensitivity: f.sensitivity ?? 'low',
|
||||
absolutePath: f.absolutePath,
|
||||
})),
|
||||
invocation: {
|
||||
composeArgsPrefix: [],
|
||||
projectDirectory: null,
|
||||
projectName: stackName,
|
||||
explicitComposeFiles: files.map((f) => f.relativePath).filter((p) => !p.includes('/')),
|
||||
meshEnabled: false,
|
||||
meshOverrideRelativePath: null,
|
||||
},
|
||||
git: null,
|
||||
appliedDeploySpec: null,
|
||||
lastAppliedContentHash: null,
|
||||
manifestState: null,
|
||||
manifestGeneration: null,
|
||||
exactCoverage: true,
|
||||
coverageRefusal: null,
|
||||
};
|
||||
}
|
||||
|
||||
describe('RollbackGenerationStore', () => {
|
||||
let composeDir: string;
|
||||
let dataDir: string;
|
||||
let originalDataDir: string | undefined;
|
||||
|
||||
beforeEach(async () => {
|
||||
composeDir = await fsPromises.mkdtemp(path.join(os.tmpdir(), 'sencho-compose-'));
|
||||
dataDir = await fsPromises.mkdtemp(path.join(os.tmpdir(), 'sencho-data-'));
|
||||
mockState.composeDir = composeDir;
|
||||
mockState.composeDirs = new Map([[1, composeDir]]);
|
||||
originalDataDir = process.env.DATA_DIR;
|
||||
process.env.DATA_DIR = dataDir;
|
||||
});
|
||||
|
||||
afterEach(async () => {
|
||||
if (originalDataDir === undefined) delete process.env.DATA_DIR;
|
||||
else process.env.DATA_DIR = originalDataDir;
|
||||
await fsPromises.rm(composeDir, { recursive: true, force: true });
|
||||
await fsPromises.rm(dataDir, { recursive: true, force: true });
|
||||
vi.restoreAllMocks();
|
||||
});
|
||||
|
||||
it('captures and restores a nested path round-trip', async () => {
|
||||
const stackName = 'web';
|
||||
const stackDir = path.join(composeDir, stackName);
|
||||
const nestedRel = 'includes/base.yaml';
|
||||
await fsPromises.mkdir(path.join(stackDir, 'includes'), { recursive: true });
|
||||
await fsPromises.writeFile(path.join(stackDir, 'compose.yaml'), 'services:\n app:\n image: a\n', 'utf8');
|
||||
await fsPromises.writeFile(path.join(stackDir, nestedRel), 'services: {}\n', 'utf8');
|
||||
|
||||
const generationId = randomUUID();
|
||||
const inv = inventoryFor(stackName, [
|
||||
{ relativePath: 'compose.yaml', absolutePath: path.join(stackDir, 'compose.yaml') },
|
||||
{ relativePath: nestedRel, absolutePath: path.join(stackDir, nestedRel), kind: 'include' },
|
||||
]);
|
||||
|
||||
const manifest = await RollbackGenerationStore.captureGeneration({
|
||||
nodeId: NODE,
|
||||
stackName,
|
||||
generationId,
|
||||
inventory: inv,
|
||||
operationKind: 'manual_backup',
|
||||
});
|
||||
|
||||
expect(manifest.managedRelativePaths).toEqual(['compose.yaml', nestedRel].sort((a, b) => a.localeCompare(b)));
|
||||
const genDir = RollbackGenerationStore.getGenerationDir(NODE, stackName, generationId);
|
||||
await expect(fsPromises.access(path.join(genDir, 'generation.json'))).resolves.toBeUndefined();
|
||||
await expect(fsPromises.access(path.join(genDir, 'files', nestedRel))).resolves.toBeUndefined();
|
||||
|
||||
// Mutate live files, then restore
|
||||
await fsPromises.writeFile(path.join(stackDir, 'compose.yaml'), 'services: { mutated: {} }\n', 'utf8');
|
||||
await fsPromises.writeFile(path.join(stackDir, nestedRel), 'services: { mutated: true }\n', 'utf8');
|
||||
|
||||
await RollbackGenerationStore.restoreGeneration(NODE, stackName, generationId, [
|
||||
'compose.yaml',
|
||||
nestedRel,
|
||||
]);
|
||||
|
||||
expect(await fsPromises.readFile(path.join(stackDir, 'compose.yaml'), 'utf8')).toBe(
|
||||
'services:\n app:\n image: a\n',
|
||||
);
|
||||
expect(await fsPromises.readFile(path.join(stackDir, nestedRel), 'utf8')).toBe('services: {}\n');
|
||||
});
|
||||
|
||||
it('leaves no final generation dir when capture fails mid-write', async () => {
|
||||
const stackName = 'failcap';
|
||||
const stackDir = path.join(composeDir, stackName);
|
||||
await fsPromises.mkdir(stackDir, { recursive: true });
|
||||
await fsPromises.writeFile(path.join(stackDir, 'compose.yaml'), 'services: {}\n', 'utf8');
|
||||
await fsPromises.mkdir(path.join(stackDir, 'configs'), { recursive: true });
|
||||
await fsPromises.writeFile(path.join(stackDir, 'configs', 'app.conf'), 'ok\n', 'utf8');
|
||||
|
||||
const generationId = randomUUID();
|
||||
const inv = inventoryFor(stackName, [
|
||||
{ relativePath: 'compose.yaml', absolutePath: path.join(stackDir, 'compose.yaml') },
|
||||
{
|
||||
relativePath: 'configs/app.conf',
|
||||
absolutePath: path.join(stackDir, 'configs', 'app.conf'),
|
||||
kind: 'config',
|
||||
sensitivity: 'high',
|
||||
},
|
||||
]);
|
||||
|
||||
const realWriteFile = fsPromises.writeFile.bind(fsPromises);
|
||||
const writeSpy = vi.spyOn(fsPromises, 'writeFile').mockImplementation(async (file, data, options) => {
|
||||
const normalized = path.normalize(String(file));
|
||||
if (normalized.includes(`${path.sep}configs${path.sep}app.conf`) && normalized.includes('staging-')) {
|
||||
throw new Error('disk full during capture');
|
||||
}
|
||||
return realWriteFile(file, data, options);
|
||||
});
|
||||
|
||||
try {
|
||||
await expect(
|
||||
RollbackGenerationStore.captureGeneration({
|
||||
nodeId: NODE,
|
||||
stackName,
|
||||
generationId,
|
||||
inventory: inv,
|
||||
}),
|
||||
).rejects.toThrow(/disk full during capture/);
|
||||
|
||||
const finalDir = RollbackGenerationStore.getGenerationDir(NODE, stackName, generationId);
|
||||
await expect(fsPromises.access(finalDir)).rejects.toMatchObject({ code: 'ENOENT' });
|
||||
|
||||
const gensRoot = RollbackGenerationStore.getGenerationsRoot(NODE, stackName);
|
||||
let leftoverStaging = false;
|
||||
try {
|
||||
const entries = await fsPromises.readdir(gensRoot);
|
||||
leftoverStaging = entries.some((e) => e.startsWith('staging-'));
|
||||
} catch (e) {
|
||||
if ((e as NodeJS.ErrnoException).code !== 'ENOENT') throw e;
|
||||
}
|
||||
expect(leftoverStaging).toBe(false);
|
||||
} finally {
|
||||
writeSpy.mockRestore();
|
||||
}
|
||||
});
|
||||
|
||||
it('refuses restore when a checksum is corrupt before mutating live files', async () => {
|
||||
const stackName = 'corrupt';
|
||||
const stackDir = path.join(composeDir, stackName);
|
||||
await fsPromises.mkdir(stackDir, { recursive: true });
|
||||
const original = 'services:\n app:\n image: good\n';
|
||||
await fsPromises.writeFile(path.join(stackDir, 'compose.yaml'), original, 'utf8');
|
||||
|
||||
const generationId = randomUUID();
|
||||
await RollbackGenerationStore.captureGeneration({
|
||||
nodeId: NODE,
|
||||
stackName,
|
||||
generationId,
|
||||
inventory: inventoryFor(stackName, [
|
||||
{ relativePath: 'compose.yaml', absolutePath: path.join(stackDir, 'compose.yaml') },
|
||||
]),
|
||||
});
|
||||
|
||||
const genDir = RollbackGenerationStore.getGenerationDir(NODE, stackName, generationId);
|
||||
await fsPromises.writeFile(path.join(genDir, 'files', 'compose.yaml'), 'services: { tampered: true }\n', 'utf8');
|
||||
|
||||
const liveMutated = 'services: { live: true }\n';
|
||||
await fsPromises.writeFile(path.join(stackDir, 'compose.yaml'), liveMutated, 'utf8');
|
||||
|
||||
await expect(
|
||||
RollbackGenerationStore.restoreGeneration(NODE, stackName, generationId, ['compose.yaml']),
|
||||
).rejects.toThrow(/Checksum mismatch/);
|
||||
|
||||
expect(await fsPromises.readFile(path.join(stackDir, 'compose.yaml'), 'utf8')).toBe(liveMutated);
|
||||
});
|
||||
|
||||
it('tombstones post-capture managed additions and leaves unrelated files', async () => {
|
||||
const stackName = 'tomb';
|
||||
const stackDir = path.join(composeDir, stackName);
|
||||
await fsPromises.mkdir(path.join(stackDir, 'configs'), { recursive: true });
|
||||
await fsPromises.writeFile(path.join(stackDir, 'compose.yaml'), 'services: {}\n', 'utf8');
|
||||
await fsPromises.writeFile(path.join(stackDir, 'configs', 'app.conf'), 'v1\n', 'utf8');
|
||||
|
||||
const generationId = randomUUID();
|
||||
await RollbackGenerationStore.captureGeneration({
|
||||
nodeId: NODE,
|
||||
stackName,
|
||||
generationId,
|
||||
inventory: inventoryFor(stackName, [
|
||||
{ relativePath: 'compose.yaml', absolutePath: path.join(stackDir, 'compose.yaml') },
|
||||
{
|
||||
relativePath: 'configs/app.conf',
|
||||
absolutePath: path.join(stackDir, 'configs', 'app.conf'),
|
||||
kind: 'config',
|
||||
sensitivity: 'high',
|
||||
},
|
||||
]),
|
||||
});
|
||||
|
||||
// Post-capture addition inside the managed discovery set
|
||||
await fsPromises.writeFile(path.join(stackDir, 'configs', 'extra.conf'), 'new\n', 'utf8');
|
||||
// Unrelated file outside managed discovery
|
||||
await fsPromises.writeFile(path.join(stackDir, 'README.md'), 'keep me\n', 'utf8');
|
||||
// Mutate a present managed file
|
||||
await fsPromises.writeFile(path.join(stackDir, 'configs', 'app.conf'), 'v2\n', 'utf8');
|
||||
|
||||
await RollbackGenerationStore.restoreGeneration(NODE, stackName, generationId, [
|
||||
'compose.yaml',
|
||||
'configs/app.conf',
|
||||
'configs/extra.conf',
|
||||
]);
|
||||
|
||||
expect(await fsPromises.readFile(path.join(stackDir, 'configs', 'app.conf'), 'utf8')).toBe('v1\n');
|
||||
await expect(fsPromises.access(path.join(stackDir, 'configs', 'extra.conf'))).rejects.toMatchObject({
|
||||
code: 'ENOENT',
|
||||
});
|
||||
expect(await fsPromises.readFile(path.join(stackDir, 'README.md'), 'utf8')).toBe('keep me\n');
|
||||
});
|
||||
|
||||
it('encrypts medium sensitivity entries and refuses symlink escapes', async () => {
|
||||
const stackName = 'secure';
|
||||
const stackDir = path.join(composeDir, stackName);
|
||||
await fsPromises.mkdir(stackDir, { recursive: true });
|
||||
await fsPromises.writeFile(path.join(stackDir, 'compose.yaml'), 'services: {}\n', 'utf8');
|
||||
await fsPromises.writeFile(path.join(stackDir, '.env'), 'SECRET=1\n', 'utf8');
|
||||
|
||||
const outside = path.join(composeDir, 'outside-secret.txt');
|
||||
await fsPromises.writeFile(outside, 'escaped\n', 'utf8');
|
||||
const linkPath = path.join(stackDir, 'escape.link');
|
||||
let symlinkOk = true;
|
||||
try {
|
||||
await fsPromises.symlink(outside, linkPath);
|
||||
} catch {
|
||||
symlinkOk = false;
|
||||
}
|
||||
|
||||
if (symlinkOk) {
|
||||
await expect(
|
||||
RollbackGenerationStore.captureGeneration({
|
||||
nodeId: NODE,
|
||||
stackName,
|
||||
generationId: randomUUID(),
|
||||
inventory: inventoryFor(stackName, [
|
||||
{ relativePath: 'compose.yaml', absolutePath: path.join(stackDir, 'compose.yaml') },
|
||||
{
|
||||
relativePath: 'escape.link',
|
||||
absolutePath: linkPath,
|
||||
kind: 'other',
|
||||
sensitivity: 'low',
|
||||
},
|
||||
]),
|
||||
}),
|
||||
).rejects.toMatchObject({ code: 'SYMLINK_ESCAPE' });
|
||||
}
|
||||
|
||||
const generationId = randomUUID();
|
||||
const manifest = await RollbackGenerationStore.captureGeneration({
|
||||
nodeId: NODE,
|
||||
stackName,
|
||||
generationId,
|
||||
inventory: inventoryFor(stackName, [
|
||||
{ relativePath: 'compose.yaml', absolutePath: path.join(stackDir, 'compose.yaml') },
|
||||
{
|
||||
relativePath: '.env',
|
||||
absolutePath: path.join(stackDir, '.env'),
|
||||
kind: 'interpolation-env',
|
||||
sensitivity: 'medium',
|
||||
},
|
||||
]),
|
||||
});
|
||||
const envEntry = manifest.entries.find((e) => e.relativePath === '.env');
|
||||
expect(envEntry?.encrypted).toBe(true);
|
||||
expect(envEntry?.state).toBe('present');
|
||||
});
|
||||
it('reverts an interrupted multi-file restore from pre-restore snapshot', async () => {
|
||||
const stackName = 'tx';
|
||||
const stackDir = path.join(composeDir, stackName);
|
||||
await fsPromises.mkdir(path.join(stackDir, 'includes'), { recursive: true });
|
||||
await fsPromises.writeFile(path.join(stackDir, 'compose.yaml'), 'OLD_BASE\n', 'utf8');
|
||||
await fsPromises.writeFile(path.join(stackDir, 'includes/base.yaml'), 'OLD_INC\n', 'utf8');
|
||||
|
||||
const generationId = randomUUID();
|
||||
const inv = inventoryFor(stackName, [
|
||||
{ relativePath: 'compose.yaml', absolutePath: path.join(stackDir, 'compose.yaml') },
|
||||
{ relativePath: 'includes/base.yaml', absolutePath: path.join(stackDir, 'includes/base.yaml'), kind: 'include' },
|
||||
]);
|
||||
await RollbackGenerationStore.captureGeneration({
|
||||
nodeId: NODE,
|
||||
stackName,
|
||||
generationId,
|
||||
inventory: inv,
|
||||
operationKind: 'manual_backup',
|
||||
});
|
||||
|
||||
await fsPromises.writeFile(path.join(stackDir, 'compose.yaml'), 'NEW_BASE\n', 'utf8');
|
||||
await fsPromises.writeFile(path.join(stackDir, 'includes/base.yaml'), 'NEW_INC\n', 'utf8');
|
||||
|
||||
const { FileSystemService } = await import('../services/FileSystemService');
|
||||
const originalWrite = FileSystemService.prototype.writeStackFile;
|
||||
let n = 0;
|
||||
vi.spyOn(FileSystemService.prototype, 'writeStackFile').mockImplementation(async function (this: InstanceType<typeof FileSystemService>, stack, rel, content) {
|
||||
n += 1;
|
||||
if (n === 2) throw new Error('injected write failure');
|
||||
return originalWrite.call(this, stack, rel, content);
|
||||
});
|
||||
|
||||
await expect(
|
||||
RollbackGenerationStore.restoreGeneration(NODE, stackName, generationId, ['compose.yaml', 'includes/base.yaml']),
|
||||
).rejects.toThrow(/injected write failure/);
|
||||
|
||||
expect(await fsPromises.readFile(path.join(stackDir, 'compose.yaml'), 'utf8')).toBe('NEW_BASE\n');
|
||||
expect(await fsPromises.readFile(path.join(stackDir, 'includes/base.yaml'), 'utf8')).toBe('NEW_INC\n');
|
||||
|
||||
const genDir = RollbackGenerationStore.getGenerationDir(NODE, stackName, generationId);
|
||||
await expect(fsPromises.access(path.join(genDir, 'restore-intent.json'))).rejects.toMatchObject({ code: 'ENOENT' });
|
||||
});
|
||||
|
||||
it('restores a snapshotted Git manifesto on compensate path helper', async () => {
|
||||
const stackName = 'gitman';
|
||||
const stackDir = path.join(composeDir, stackName);
|
||||
await fsPromises.mkdir(stackDir, { recursive: true });
|
||||
await fsPromises.writeFile(path.join(stackDir, 'compose.yaml'), 'services: {}\n', 'utf8');
|
||||
|
||||
const managedDir = path.join(dataDir, 'git-managed', '1', stackName);
|
||||
await fsPromises.mkdir(managedDir, { recursive: true });
|
||||
const priorManifest = {
|
||||
manifestVersion: 3,
|
||||
state: 'active',
|
||||
repo: { url: 'https://example.com/r.git', branch: 'main' },
|
||||
resolvedRevision: { commitSha: 'abc1234' },
|
||||
project: {
|
||||
root: '.',
|
||||
composeFiles: ['compose.yaml'],
|
||||
projectName: stackName,
|
||||
effectiveProjectDir: null,
|
||||
invocation: ['-f', 'compose.yaml'],
|
||||
},
|
||||
inputs: [],
|
||||
buildContexts: [],
|
||||
counts: { total: 0, refused: 0 },
|
||||
refusals: [],
|
||||
generation: { appliedDir: 'generations/prior', previousDir: null },
|
||||
};
|
||||
await fsPromises.writeFile(
|
||||
path.join(managedDir, 'manifest.v1.json'),
|
||||
JSON.stringify(priorManifest),
|
||||
'utf8',
|
||||
);
|
||||
|
||||
const generationId = randomUUID();
|
||||
const inv = inventoryFor(stackName, [
|
||||
{ relativePath: 'compose.yaml', absolutePath: path.join(stackDir, 'compose.yaml') },
|
||||
]);
|
||||
inv.git = {
|
||||
repoUrl: 'https://example.com/r.git',
|
||||
branch: 'main',
|
||||
commitSha: 'abc1234',
|
||||
manifestVersion: 3,
|
||||
};
|
||||
const { GitProjectManifestService } = await import('../services/GitProjectManifestService');
|
||||
const readSpy = vi.spyOn(GitProjectManifestService.getInstance(), 'readManifest')
|
||||
.mockResolvedValue(priorManifest as never);
|
||||
let genDir = '';
|
||||
try {
|
||||
await RollbackGenerationStore.captureGeneration({
|
||||
nodeId: NODE,
|
||||
stackName,
|
||||
generationId,
|
||||
inventory: inv,
|
||||
});
|
||||
|
||||
genDir = RollbackGenerationStore.getGenerationDir(NODE, stackName, generationId);
|
||||
const snap = await fsPromises.readFile(path.join(genDir, 'git-manifest.v1.json'), 'utf8');
|
||||
expect(JSON.parse(snap).resolvedRevision.commitSha).toBe('abc1234');
|
||||
} finally {
|
||||
readSpy.mockRestore();
|
||||
}
|
||||
const newer = { ...priorManifest, resolvedRevision: { commitSha: 'ffffff' } };
|
||||
await fsPromises.writeFile(path.join(managedDir, 'manifest.v1.json'), JSON.stringify(newer), 'utf8');
|
||||
|
||||
const captured = JSON.parse(await fsPromises.readFile(path.join(genDir, 'generation.json'), 'utf8'));
|
||||
await RollbackGenerationStore.restoreCapturedGitManifest(stackName, genDir, captured);
|
||||
const restored = JSON.parse(await fsPromises.readFile(path.join(managedDir, 'manifest.v1.json'), 'utf8'));
|
||||
expect(restored.resolvedRevision.commitSha).toBe('abc1234');
|
||||
});
|
||||
|
||||
it('does not snapshot a corrupt Git manifesto into the generation', async () => {
|
||||
const stackName = 'nocorrupt';
|
||||
const stackDir = path.join(composeDir, stackName);
|
||||
await fsPromises.mkdir(stackDir, { recursive: true });
|
||||
await fsPromises.writeFile(path.join(stackDir, 'compose.yaml'), 'services: {}\n', 'utf8');
|
||||
|
||||
const generationId = randomUUID();
|
||||
const inv = inventoryFor(stackName, [
|
||||
{ relativePath: 'compose.yaml', absolutePath: path.join(stackDir, 'compose.yaml') },
|
||||
]);
|
||||
inv.git = {
|
||||
repoUrl: 'https://example.com/r.git',
|
||||
branch: 'main',
|
||||
commitSha: 'abc',
|
||||
manifestVersion: 3,
|
||||
};
|
||||
const { GitProjectManifestService } = await import('../services/GitProjectManifestService');
|
||||
const readSpy = vi.spyOn(GitProjectManifestService.getInstance(), 'readManifest')
|
||||
.mockResolvedValue({ corrupt: 'identity mismatch' });
|
||||
try {
|
||||
await RollbackGenerationStore.captureGeneration({
|
||||
nodeId: NODE,
|
||||
stackName,
|
||||
generationId,
|
||||
inventory: inv,
|
||||
});
|
||||
const genDir = RollbackGenerationStore.getGenerationDir(NODE, stackName, generationId);
|
||||
await expect(fsPromises.access(path.join(genDir, 'git-manifest.v1.json'))).rejects.toMatchObject({
|
||||
code: 'ENOENT',
|
||||
});
|
||||
const captured = JSON.parse(await fsPromises.readFile(path.join(genDir, 'generation.json'), 'utf8'));
|
||||
expect(captured.priorRecords.gitManifestCaptured).toBe(false);
|
||||
} finally {
|
||||
readSpy.mockRestore();
|
||||
}
|
||||
});
|
||||
|
||||
it('clears post-capture manifesto for first-apply preimage', async () => {
|
||||
const stackName = 'firstapply';
|
||||
const stackDir = path.join(composeDir, stackName);
|
||||
await fsPromises.mkdir(stackDir, { recursive: true });
|
||||
await fsPromises.writeFile(path.join(stackDir, 'compose.yaml'), 'services: {}\n', 'utf8');
|
||||
|
||||
const generationId = randomUUID();
|
||||
const inv = inventoryFor(stackName, [
|
||||
{ relativePath: 'compose.yaml', absolutePath: path.join(stackDir, 'compose.yaml') },
|
||||
]);
|
||||
inv.git = {
|
||||
repoUrl: 'https://example.com/r.git',
|
||||
branch: 'main',
|
||||
commitSha: '',
|
||||
manifestVersion: null,
|
||||
};
|
||||
await RollbackGenerationStore.captureGeneration({
|
||||
nodeId: NODE,
|
||||
stackName,
|
||||
generationId,
|
||||
inventory: inv,
|
||||
});
|
||||
|
||||
const genDir = RollbackGenerationStore.getGenerationDir(NODE, stackName, generationId);
|
||||
const captured = JSON.parse(await fsPromises.readFile(path.join(genDir, 'generation.json'), 'utf8'));
|
||||
expect(captured.priorRecords.gitManifestCaptured).toBe(false);
|
||||
|
||||
const managedDir = path.join(dataDir, 'git-managed', '1', stackName);
|
||||
await fsPromises.mkdir(managedDir, { recursive: true });
|
||||
await fsPromises.writeFile(path.join(managedDir, 'manifest.v1.json'), '{"manifestVersion":3}\n', 'utf8');
|
||||
|
||||
await RollbackGenerationStore.restoreCapturedGitManifest(stackName, genDir, captured);
|
||||
await expect(fsPromises.access(path.join(managedDir, 'manifest.v1.json'))).rejects.toMatchObject({ code: 'ENOENT' });
|
||||
});
|
||||
|
||||
it('does not clear live manifesto for legacy generations without a snapshot', async () => {
|
||||
const stackName = 'legacygen';
|
||||
const stackDir = path.join(composeDir, stackName);
|
||||
await fsPromises.mkdir(stackDir, { recursive: true });
|
||||
await fsPromises.writeFile(path.join(stackDir, 'compose.yaml'), 'services: {}\n', 'utf8');
|
||||
|
||||
const generationId = randomUUID();
|
||||
await RollbackGenerationStore.captureGeneration({
|
||||
nodeId: NODE,
|
||||
stackName,
|
||||
generationId,
|
||||
inventory: inventoryFor(stackName, [
|
||||
{ relativePath: 'compose.yaml', absolutePath: path.join(stackDir, 'compose.yaml') },
|
||||
]),
|
||||
});
|
||||
|
||||
const genDir = RollbackGenerationStore.getGenerationDir(NODE, stackName, generationId);
|
||||
const captured = JSON.parse(await fsPromises.readFile(path.join(genDir, 'generation.json'), 'utf8'));
|
||||
delete captured.priorRecords.gitManifestCaptured;
|
||||
captured.git = {
|
||||
repoUrl: 'https://example.com/r.git',
|
||||
branch: 'main',
|
||||
commitSha: 'abc',
|
||||
manifestVersion: 3,
|
||||
};
|
||||
await fsPromises.writeFile(path.join(genDir, 'generation.json'), JSON.stringify(captured), 'utf8');
|
||||
|
||||
const managedDir = path.join(dataDir, 'git-managed', '1', stackName);
|
||||
await fsPromises.mkdir(managedDir, { recursive: true });
|
||||
await fsPromises.writeFile(path.join(managedDir, 'manifest.v1.json'), '{"keep":true}\n', 'utf8');
|
||||
|
||||
await RollbackGenerationStore.restoreCapturedGitManifest(stackName, genDir, captured);
|
||||
expect(await fsPromises.readFile(path.join(managedDir, 'manifest.v1.json'), 'utf8')).toBe('{"keep":true}\n');
|
||||
});
|
||||
|
||||
it('refuses restore when a managed file path is currently a directory', async () => {
|
||||
const stackName = 'dircollide';
|
||||
const stackDir = path.join(composeDir, stackName);
|
||||
await fsPromises.mkdir(stackDir, { recursive: true });
|
||||
await fsPromises.writeFile(path.join(stackDir, 'compose.yaml'), 'OLD\n', 'utf8');
|
||||
|
||||
const generationId = randomUUID();
|
||||
await RollbackGenerationStore.captureGeneration({
|
||||
nodeId: NODE,
|
||||
stackName,
|
||||
generationId,
|
||||
inventory: inventoryFor(stackName, [
|
||||
{ relativePath: 'compose.yaml', absolutePath: path.join(stackDir, 'compose.yaml') },
|
||||
]),
|
||||
});
|
||||
|
||||
await fsPromises.rm(path.join(stackDir, 'compose.yaml'));
|
||||
await fsPromises.mkdir(path.join(stackDir, 'compose.yaml'), { recursive: true });
|
||||
await fsPromises.writeFile(path.join(stackDir, 'compose.yaml', 'sentinel.txt'), 'keep-me\n', 'utf8');
|
||||
|
||||
await expect(
|
||||
RollbackGenerationStore.restoreGeneration(NODE, stackName, generationId, ['compose.yaml']),
|
||||
).rejects.toMatchObject({ code: 'DIRECTORY_COLLISION' });
|
||||
|
||||
expect(await fsPromises.readFile(path.join(stackDir, 'compose.yaml', 'sentinel.txt'), 'utf8')).toBe('keep-me\n');
|
||||
});
|
||||
|
||||
it('restores a legitimate file whose name ends with .sencho-tombstone (index-based snapshot)', async () => {
|
||||
const stackName = 'tombname';
|
||||
const stackDir = path.join(composeDir, stackName);
|
||||
await fsPromises.mkdir(stackDir, { recursive: true });
|
||||
await fsPromises.writeFile(path.join(stackDir, 'compose.yaml'), 'services: {}\n', 'utf8');
|
||||
const special = 'config.sencho-tombstone';
|
||||
await fsPromises.writeFile(path.join(stackDir, special), 'KEEP_ME\n', 'utf8');
|
||||
|
||||
const generationId = randomUUID();
|
||||
await RollbackGenerationStore.captureGeneration({
|
||||
nodeId: NODE,
|
||||
stackName,
|
||||
generationId,
|
||||
inventory: inventoryFor(stackName, [
|
||||
{ relativePath: 'compose.yaml', absolutePath: path.join(stackDir, 'compose.yaml') },
|
||||
{ relativePath: special, absolutePath: path.join(stackDir, special), kind: 'other' },
|
||||
]),
|
||||
});
|
||||
|
||||
await fsPromises.writeFile(path.join(stackDir, special), 'CHANGED\n', 'utf8');
|
||||
await RollbackGenerationStore.restoreGeneration(
|
||||
NODE,
|
||||
stackName,
|
||||
generationId,
|
||||
['compose.yaml', special],
|
||||
);
|
||||
|
||||
expect(await fsPromises.readFile(path.join(stackDir, special), 'utf8')).toBe('KEEP_ME\n');
|
||||
});
|
||||
|
||||
it('captures POSIX mode and restores it on generation restore', async () => {
|
||||
if (process.platform === 'win32') return;
|
||||
|
||||
const stackName = 'modes';
|
||||
const stackDir = path.join(composeDir, stackName);
|
||||
await fsPromises.mkdir(stackDir, { recursive: true });
|
||||
const scriptPath = path.join(stackDir, 'entrypoint.sh');
|
||||
await fsPromises.writeFile(scriptPath, '#!/bin/sh\necho hi\n', 'utf8');
|
||||
await fsPromises.chmod(scriptPath, 0o755);
|
||||
await fsPromises.writeFile(path.join(stackDir, 'compose.yaml'), 'services: {}\n', 'utf8');
|
||||
|
||||
const generationId = randomUUID();
|
||||
const manifest = await RollbackGenerationStore.captureGeneration({
|
||||
nodeId: NODE,
|
||||
stackName,
|
||||
generationId,
|
||||
inventory: inventoryFor(stackName, [
|
||||
{ relativePath: 'compose.yaml', absolutePath: path.join(stackDir, 'compose.yaml') },
|
||||
{ relativePath: 'entrypoint.sh', absolutePath: scriptPath, kind: 'other' },
|
||||
]),
|
||||
});
|
||||
const entry = manifest.entries.find((e) => e.relativePath === 'entrypoint.sh');
|
||||
expect(entry?.mode).toBe(0o755);
|
||||
|
||||
await fsPromises.chmod(scriptPath, 0o644);
|
||||
await RollbackGenerationStore.restoreGeneration(
|
||||
NODE,
|
||||
stackName,
|
||||
generationId,
|
||||
['compose.yaml', 'entrypoint.sh'],
|
||||
);
|
||||
const st = await fsPromises.stat(scriptPath);
|
||||
expect(st.mode & 0o777).toBe(0o755);
|
||||
});
|
||||
|
||||
it('reverts an interrupted restore that deleted a post-capture file via index absent entries', async () => {
|
||||
const stackName = 'absenttx';
|
||||
const stackDir = path.join(composeDir, stackName);
|
||||
await fsPromises.mkdir(stackDir, { recursive: true });
|
||||
await fsPromises.writeFile(path.join(stackDir, 'compose.yaml'), 'OLD\n', 'utf8');
|
||||
|
||||
const generationId = randomUUID();
|
||||
await RollbackGenerationStore.captureGeneration({
|
||||
nodeId: NODE,
|
||||
stackName,
|
||||
generationId,
|
||||
inventory: inventoryFor(stackName, [
|
||||
{ relativePath: 'compose.yaml', absolutePath: path.join(stackDir, 'compose.yaml') },
|
||||
]),
|
||||
});
|
||||
|
||||
await fsPromises.writeFile(path.join(stackDir, 'compose.yaml'), 'NEW\n', 'utf8');
|
||||
await fsPromises.writeFile(path.join(stackDir, 'extra.yml'), 'EXTRA\n', 'utf8');
|
||||
|
||||
const { FileSystemService } = await import('../services/FileSystemService');
|
||||
const originalWrite = FileSystemService.prototype.writeStackFile;
|
||||
vi.spyOn(FileSystemService.prototype, 'writeStackFile').mockImplementation(async function (
|
||||
this: InstanceType<typeof FileSystemService>,
|
||||
stack,
|
||||
rel,
|
||||
content,
|
||||
) {
|
||||
if (rel === 'compose.yaml') throw new Error('injected write failure');
|
||||
return originalWrite.call(this, stack, rel, content);
|
||||
});
|
||||
|
||||
await expect(
|
||||
RollbackGenerationStore.restoreGeneration(NODE, stackName, generationId, ['compose.yaml', 'extra.yml']),
|
||||
).rejects.toThrow(/injected write failure/);
|
||||
|
||||
expect(await fsPromises.readFile(path.join(stackDir, 'compose.yaml'), 'utf8')).toBe('NEW\n');
|
||||
expect(await fsPromises.readFile(path.join(stackDir, 'extra.yml'), 'utf8')).toBe('EXTRA\n');
|
||||
});
|
||||
|
||||
it('encrypts sensitive pre-restore snapshots and restores them from ciphertext', async () => {
|
||||
const stackName = 'presensitive';
|
||||
const stackDir = path.join(composeDir, stackName);
|
||||
await fsPromises.mkdir(stackDir, { recursive: true });
|
||||
await fsPromises.writeFile(path.join(stackDir, 'compose.yaml'), 'services: {}\n', 'utf8');
|
||||
await fsPromises.writeFile(path.join(stackDir, '.env'), 'CAPTURED=1\n', 'utf8');
|
||||
|
||||
const generationId = randomUUID();
|
||||
await RollbackGenerationStore.captureGeneration({
|
||||
nodeId: NODE,
|
||||
stackName,
|
||||
generationId,
|
||||
inventory: inventoryFor(stackName, [
|
||||
{ relativePath: 'compose.yaml', absolutePath: path.join(stackDir, 'compose.yaml') },
|
||||
{
|
||||
relativePath: '.env',
|
||||
absolutePath: path.join(stackDir, '.env'),
|
||||
kind: 'interpolation-env',
|
||||
sensitivity: 'medium',
|
||||
},
|
||||
]),
|
||||
});
|
||||
|
||||
await fsPromises.writeFile(path.join(stackDir, '.env'), 'SUPERSECRET=live\n', 'utf8');
|
||||
await RollbackGenerationStore.restoreGeneration(
|
||||
NODE,
|
||||
stackName,
|
||||
generationId,
|
||||
['compose.yaml', '.env'],
|
||||
);
|
||||
|
||||
const genDir = RollbackGenerationStore.getGenerationDir(NODE, stackName, generationId);
|
||||
const blobsDir = path.join(genDir, 'pre-restore', 'blobs');
|
||||
const blobNames = await fsPromises.readdir(blobsDir);
|
||||
expect(blobNames.length).toBeGreaterThan(0);
|
||||
for (const name of blobNames) {
|
||||
const blob = await fsPromises.readFile(path.join(blobsDir, name));
|
||||
expect(blob.toString('utf8')).not.toContain('SUPERSECRET');
|
||||
expect(blob.toString('utf8')).not.toContain('CAPTURED=1');
|
||||
}
|
||||
const index = JSON.parse(
|
||||
await fsPromises.readFile(path.join(genDir, 'pre-restore', 'index.json'), 'utf8'),
|
||||
) as { entries: Array<{ relativePath: string; encrypted?: boolean; blobId?: string }> };
|
||||
const envEntry = index.entries.find((e) => e.relativePath === '.env');
|
||||
expect(envEntry?.encrypted).toBe(true);
|
||||
expect(envEntry?.blobId).toBeTruthy();
|
||||
const envBlob = await fsPromises.readFile(path.join(blobsDir, envEntry!.blobId!), 'utf8');
|
||||
expect(envBlob.startsWith('enc:')).toBe(true);
|
||||
expect(envBlob).not.toContain('SUPERSECRET');
|
||||
|
||||
await RollbackGenerationStore.reconcileInterruptedRestore(NODE, stackName, generationId);
|
||||
expect(await fsPromises.readFile(path.join(stackDir, '.env'), 'utf8')).toBe('SUPERSECRET=live\n');
|
||||
});
|
||||
|
||||
it('persists runtime image identity on an existing generation', async () => {
|
||||
const stackName = 'imgid';
|
||||
const stackDir = path.join(composeDir, stackName);
|
||||
await fsPromises.mkdir(stackDir, { recursive: true });
|
||||
await fsPromises.writeFile(path.join(stackDir, 'compose.yaml'), 'services: {}\n', 'utf8');
|
||||
const generationId = randomUUID();
|
||||
await RollbackGenerationStore.captureGeneration({
|
||||
nodeId: NODE,
|
||||
stackName,
|
||||
generationId,
|
||||
inventory: inventoryFor(stackName, [
|
||||
{ relativePath: 'compose.yaml', absolutePath: path.join(stackDir, 'compose.yaml') },
|
||||
]),
|
||||
});
|
||||
|
||||
await RollbackGenerationStore.attachImages(NODE, stackName, generationId, [{
|
||||
serviceName: 'web',
|
||||
imageId: 'sha256:abc',
|
||||
repoDigest: 'nginx@sha256:digest',
|
||||
platform: 'linux/amd64',
|
||||
declaredImageRef: 'nginx:latest',
|
||||
}]);
|
||||
|
||||
const genDir = RollbackGenerationStore.getGenerationDir(NODE, stackName, generationId);
|
||||
const manifest = JSON.parse(
|
||||
await fsPromises.readFile(path.join(genDir, 'generation.json'), 'utf8'),
|
||||
) as { images: Array<{ serviceName: string; imageId: string; repoDigest: string; platform: string }> };
|
||||
expect(manifest.images).toEqual([{
|
||||
serviceName: 'web',
|
||||
imageId: 'sha256:abc',
|
||||
repoDigest: 'nginx@sha256:digest',
|
||||
platform: 'linux/amd64',
|
||||
declaredImageRef: 'nginx:latest',
|
||||
}]);
|
||||
});
|
||||
|
||||
it('does not recursively delete a directory created after an absent-file snapshot', async () => {
|
||||
const stackName = 'dirrace';
|
||||
const stackDir = path.join(composeDir, stackName);
|
||||
await fsPromises.mkdir(stackDir, { recursive: true });
|
||||
await fsPromises.writeFile(path.join(stackDir, 'compose.yaml'), 'OLD\n', 'utf8');
|
||||
|
||||
const generationId = randomUUID();
|
||||
await RollbackGenerationStore.captureGeneration({
|
||||
nodeId: NODE,
|
||||
stackName,
|
||||
generationId,
|
||||
inventory: inventoryFor(stackName, [
|
||||
{ relativePath: 'compose.yaml', absolutePath: path.join(stackDir, 'compose.yaml') },
|
||||
]),
|
||||
});
|
||||
|
||||
await RollbackGenerationStore.restoreGeneration(
|
||||
NODE,
|
||||
stackName,
|
||||
generationId,
|
||||
['compose.yaml', 'scratch'],
|
||||
);
|
||||
|
||||
const scratchDir = path.join(stackDir, 'scratch');
|
||||
await fsPromises.mkdir(scratchDir, { recursive: true });
|
||||
await fsPromises.writeFile(path.join(scratchDir, 'keep.txt'), 'unrelated\n', 'utf8');
|
||||
|
||||
await expect(
|
||||
RollbackGenerationStore.reconcileInterruptedRestore(NODE, stackName, generationId),
|
||||
).rejects.toMatchObject({ code: 'DIRECTORY_COLLISION' });
|
||||
|
||||
expect(await fsPromises.readFile(path.join(scratchDir, 'keep.txt'), 'utf8')).toBe('unrelated\n');
|
||||
const genDir = RollbackGenerationStore.getGenerationDir(NODE, stackName, generationId);
|
||||
await expect(fsPromises.access(path.join(genDir, 'restore-intent.json'))).resolves.toBeUndefined();
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,311 @@
|
||||
/**
|
||||
* Git manifesto inventory: first-apply merge vs established fail-closed.
|
||||
*/
|
||||
import { beforeEach, describe, expect, it, vi } from 'vitest';
|
||||
import path from 'path';
|
||||
import os from 'os';
|
||||
import { promises as fsPromises } from 'fs';
|
||||
|
||||
const mockGetGitSource = vi.fn();
|
||||
const mockReadManifest = vi.fn();
|
||||
const mockGetOverrideFilename = vi.fn().mockResolvedValue(null);
|
||||
const mockGetStackProjectEnvFiles = vi.fn().mockReturnValue([]);
|
||||
const mockBuildAuthoredComposeArgs = vi.fn().mockResolvedValue(['compose', '-f', 'compose.yaml', 'config', '--quiet']);
|
||||
const mockAuthoredComposeFileArgs = vi.fn().mockReturnValue(['-f', 'compose.yaml']);
|
||||
const mockAuthoredComposeEnvFileArgs = vi.fn().mockResolvedValue([]);
|
||||
|
||||
vi.mock('../services/DatabaseService', () => ({
|
||||
DatabaseService: {
|
||||
getInstance: () => ({
|
||||
getGitSource: mockGetGitSource,
|
||||
getStackProjectEnvFiles: mockGetStackProjectEnvFiles,
|
||||
isMeshStackEnabled: () => false,
|
||||
}),
|
||||
},
|
||||
}));
|
||||
|
||||
vi.mock('../services/NodeRegistry', () => ({
|
||||
NodeRegistry: {
|
||||
getInstance: () => ({
|
||||
getDefaultNodeId: () => 1,
|
||||
}),
|
||||
},
|
||||
}));
|
||||
|
||||
vi.mock('../services/GitProjectManifestService', () => ({
|
||||
GitProjectManifestService: {
|
||||
getInstance: () => ({
|
||||
readManifest: mockReadManifest,
|
||||
}),
|
||||
},
|
||||
}));
|
||||
|
||||
const mockState = { composeDir: '' };
|
||||
|
||||
vi.mock('../services/FileSystemService', () => ({
|
||||
FileSystemService: {
|
||||
getInstance: () => ({
|
||||
getBaseDir: () => mockState.composeDir,
|
||||
getOverrideFilename: mockGetOverrideFilename,
|
||||
}),
|
||||
},
|
||||
}));
|
||||
|
||||
vi.mock('../services/ComposeService', () => ({
|
||||
ComposeService: {
|
||||
getInstance: () => ({
|
||||
buildAuthoredComposeArgs: mockBuildAuthoredComposeArgs,
|
||||
}),
|
||||
},
|
||||
}));
|
||||
|
||||
vi.mock('../utils/authoredComposeArgs', () => ({
|
||||
authoredComposeFileArgs: (...args: unknown[]) => mockAuthoredComposeFileArgs(...args),
|
||||
authoredComposeEnvFileArgs: (...args: unknown[]) => mockAuthoredComposeEnvFileArgs(...args),
|
||||
}));
|
||||
|
||||
import { resolveRollbackInventory } from '../services/rollbackInventory';
|
||||
|
||||
describe('resolveRollbackInventory', () => {
|
||||
let composeDir: string;
|
||||
|
||||
beforeEach(async () => {
|
||||
vi.clearAllMocks();
|
||||
composeDir = await fsPromises.mkdtemp(path.join(os.tmpdir(), 'sencho-inv-'));
|
||||
mockState.composeDir = composeDir;
|
||||
mockGetOverrideFilename.mockResolvedValue(null);
|
||||
mockGetStackProjectEnvFiles.mockReturnValue([]);
|
||||
mockBuildAuthoredComposeArgs.mockResolvedValue(['compose', '-f', 'compose.yaml', 'config', '--quiet']);
|
||||
mockAuthoredComposeFileArgs.mockReturnValue(['-f', 'compose.yaml']);
|
||||
mockAuthoredComposeEnvFileArgs.mockResolvedValue([]);
|
||||
});
|
||||
|
||||
it('merges authored files with Git identity on first apply when manifesto is missing', async () => {
|
||||
const stackName = 'gitapp';
|
||||
const stackDir = path.join(composeDir, stackName);
|
||||
await fsPromises.mkdir(stackDir, { recursive: true });
|
||||
await fsPromises.writeFile(
|
||||
path.join(stackDir, 'compose.yaml'),
|
||||
'services:\n web:\n image: nginx\n',
|
||||
'utf8',
|
||||
);
|
||||
|
||||
mockGetGitSource.mockReturnValue({
|
||||
stack_name: stackName,
|
||||
repo_url: 'https://example.com/repo.git',
|
||||
branch: 'main',
|
||||
last_applied_commit_sha: null,
|
||||
last_applied_content_hash: null,
|
||||
applied_deploy_spec: null,
|
||||
sync_env: false,
|
||||
manifest_version: null,
|
||||
manifest_state: null,
|
||||
manifest_generation: null,
|
||||
});
|
||||
mockReadManifest.mockResolvedValue(null);
|
||||
|
||||
const inventory = await resolveRollbackInventory(1, stackName);
|
||||
|
||||
expect(inventory.exactCoverage).toBe(true);
|
||||
expect(inventory.coverageRefusal).toBeNull();
|
||||
expect(inventory.entries.map((e) => e.relativePath)).toContain('compose.yaml');
|
||||
expect(inventory.git).toEqual({
|
||||
repoUrl: 'https://example.com/repo.git',
|
||||
branch: 'main',
|
||||
commitSha: '',
|
||||
manifestVersion: null,
|
||||
});
|
||||
});
|
||||
|
||||
it('fails closed when an established Git stack is missing its manifesto and cannot rebuild from applied spec', async () => {
|
||||
const stackName = 'established';
|
||||
const stackDir = path.join(composeDir, stackName);
|
||||
await fsPromises.mkdir(stackDir, { recursive: true });
|
||||
await fsPromises.writeFile(path.join(stackDir, 'compose.yaml'), 'services: {}\n', 'utf8');
|
||||
|
||||
mockGetGitSource.mockReturnValue({
|
||||
stack_name: stackName,
|
||||
repo_url: 'https://example.com/repo.git',
|
||||
branch: 'main',
|
||||
last_applied_commit_sha: 'abc1234',
|
||||
last_applied_content_hash: 'hash',
|
||||
applied_deploy_spec: null,
|
||||
sync_env: false,
|
||||
manifest_version: 3,
|
||||
manifest_state: 'active',
|
||||
manifest_generation: 'generations/prior',
|
||||
});
|
||||
mockReadManifest.mockResolvedValue(null);
|
||||
|
||||
const inventory = await resolveRollbackInventory(1, stackName);
|
||||
|
||||
expect(inventory.exactCoverage).toBe(false);
|
||||
expect(inventory.coverageRefusal).toMatch(/missing/i);
|
||||
expect(inventory.git?.commitSha).toBe('abc1234');
|
||||
});
|
||||
|
||||
it('fails closed for established missing manifesto instead of applied_deploy_spec exact coverage', async () => {
|
||||
const stackName = 'rebuild';
|
||||
const stackDir = path.join(composeDir, stackName);
|
||||
await fsPromises.mkdir(stackDir, { recursive: true });
|
||||
await fsPromises.writeFile(path.join(stackDir, 'compose.yaml'), 'services:\n a: {}\n', 'utf8');
|
||||
await fsPromises.writeFile(path.join(stackDir, 'compose.prod.yaml'), 'services:\n b: {}\n', 'utf8');
|
||||
await fsPromises.writeFile(path.join(stackDir, 'app.env'), 'FOO=1\n', 'utf8');
|
||||
|
||||
mockGetGitSource.mockReturnValue({
|
||||
stack_name: stackName,
|
||||
repo_url: 'https://example.com/repo.git',
|
||||
branch: 'main',
|
||||
last_applied_commit_sha: 'deadbeef',
|
||||
last_applied_content_hash: 'h1',
|
||||
applied_deploy_spec: { files: ['compose.yaml', 'compose.prod.yaml'], contextDir: null },
|
||||
sync_env: false,
|
||||
manifest_version: 3,
|
||||
manifest_state: 'active',
|
||||
manifest_generation: 'generations/x',
|
||||
});
|
||||
mockReadManifest.mockResolvedValue(null);
|
||||
|
||||
const inventory = await resolveRollbackInventory(1, stackName);
|
||||
|
||||
expect(inventory.exactCoverage).toBe(false);
|
||||
expect(inventory.coverageRefusal).toMatch(/missing/i);
|
||||
expect(inventory.git?.commitSha).toBe('deadbeef');
|
||||
});
|
||||
|
||||
it('fails closed when the manifesto is corrupt even if applied_deploy_spec files exist', async () => {
|
||||
const stackName = 'corruptgit';
|
||||
const stackDir = path.join(composeDir, stackName);
|
||||
await fsPromises.mkdir(stackDir, { recursive: true });
|
||||
await fsPromises.writeFile(path.join(stackDir, 'compose.yaml'), 'services: {}\n', 'utf8');
|
||||
await fsPromises.writeFile(path.join(stackDir, 'app.env'), 'PASS=x\n', 'utf8');
|
||||
|
||||
mockGetGitSource.mockReturnValue({
|
||||
stack_name: stackName,
|
||||
repo_url: 'https://example.com/repo.git',
|
||||
branch: 'main',
|
||||
last_applied_commit_sha: 'abc',
|
||||
last_applied_content_hash: 'h',
|
||||
applied_deploy_spec: { files: ['compose.yaml'], contextDir: null },
|
||||
sync_env: false,
|
||||
manifest_version: 3,
|
||||
manifest_state: 'active',
|
||||
manifest_generation: 'generations/x',
|
||||
});
|
||||
mockReadManifest.mockResolvedValue({ corrupt: 'identity mismatch' });
|
||||
|
||||
const inventory = await resolveRollbackInventory(1, stackName);
|
||||
|
||||
expect(inventory.exactCoverage).toBe(false);
|
||||
expect(inventory.coverageRefusal).toMatch(/unreadable/i);
|
||||
expect(inventory.git?.commitSha).toBe('abc');
|
||||
});
|
||||
|
||||
it('fails closed when the manifesto is corrupt and applied_deploy_spec cannot be rebuilt', async () => {
|
||||
const stackName = 'corrupt-norebuild';
|
||||
const stackDir = path.join(composeDir, stackName);
|
||||
await fsPromises.mkdir(stackDir, { recursive: true });
|
||||
await fsPromises.writeFile(path.join(stackDir, 'compose.yaml'), 'services: {}\n', 'utf8');
|
||||
|
||||
mockGetGitSource.mockReturnValue({
|
||||
stack_name: stackName,
|
||||
repo_url: 'https://example.com/repo.git',
|
||||
branch: 'main',
|
||||
last_applied_commit_sha: 'abc',
|
||||
last_applied_content_hash: 'h',
|
||||
applied_deploy_spec: { files: ['compose.yaml', 'missing.override.yaml'], contextDir: null },
|
||||
sync_env: false,
|
||||
manifest_version: 3,
|
||||
manifest_state: 'active',
|
||||
manifest_generation: 'generations/x',
|
||||
});
|
||||
mockReadManifest.mockResolvedValue({ corrupt: 'identity mismatch' });
|
||||
|
||||
const inventory = await resolveRollbackInventory(1, stackName);
|
||||
|
||||
expect(inventory.exactCoverage).toBe(false);
|
||||
expect(inventory.coverageRefusal).toMatch(/unreadable/i);
|
||||
});
|
||||
|
||||
it('fails closed for missing manifesto with include/env/config dependency files on disk', async () => {
|
||||
const stackName = 'deps-matrix';
|
||||
const stackDir = path.join(composeDir, stackName);
|
||||
await fsPromises.mkdir(path.join(stackDir, 'configs'), { recursive: true });
|
||||
await fsPromises.writeFile(path.join(stackDir, 'compose.yaml'), 'services:\n a:\n env_file: [app.env]\n', 'utf8');
|
||||
await fsPromises.writeFile(path.join(stackDir, 'compose.prod.yaml'), 'include:\n - path: compose.yaml\n', 'utf8');
|
||||
await fsPromises.writeFile(path.join(stackDir, 'app.env'), 'A=1\n', 'utf8');
|
||||
await fsPromises.writeFile(path.join(stackDir, 'configs/app.conf'), 'x=1\n', 'utf8');
|
||||
|
||||
mockGetGitSource.mockReturnValue({
|
||||
stack_name: stackName,
|
||||
repo_url: 'https://example.com/repo.git',
|
||||
branch: 'main',
|
||||
last_applied_commit_sha: 'cafebabe',
|
||||
last_applied_content_hash: 'h2',
|
||||
applied_deploy_spec: { files: ['compose.yaml', 'compose.prod.yaml'], contextDir: null },
|
||||
sync_env: false,
|
||||
manifest_version: 3,
|
||||
manifest_state: 'active',
|
||||
manifest_generation: 'generations/y',
|
||||
});
|
||||
mockReadManifest.mockResolvedValue(null);
|
||||
|
||||
const inventory = await resolveRollbackInventory(1, stackName);
|
||||
|
||||
expect(inventory.exactCoverage).toBe(false);
|
||||
expect(inventory.coverageRefusal).toMatch(/missing/i);
|
||||
});
|
||||
|
||||
it('fails closed when the Git manifesto is corrupt even before first apply', async () => {
|
||||
const stackName = 'corrupt-first';
|
||||
const stackDir = path.join(composeDir, stackName);
|
||||
await fsPromises.mkdir(stackDir, { recursive: true });
|
||||
await fsPromises.writeFile(path.join(stackDir, 'compose.yaml'), 'services: {}\n', 'utf8');
|
||||
|
||||
mockGetGitSource.mockReturnValue({
|
||||
stack_name: stackName,
|
||||
repo_url: 'https://example.com/repo.git',
|
||||
branch: 'main',
|
||||
last_applied_commit_sha: null,
|
||||
last_applied_content_hash: null,
|
||||
applied_deploy_spec: null,
|
||||
sync_env: false,
|
||||
manifest_version: null,
|
||||
manifest_state: null,
|
||||
manifest_generation: null,
|
||||
});
|
||||
mockReadManifest.mockResolvedValue({ corrupt: 'identity mismatch' });
|
||||
|
||||
const inventory = await resolveRollbackInventory(1, stackName);
|
||||
|
||||
expect(inventory.exactCoverage).toBe(false);
|
||||
expect(inventory.coverageRefusal).toMatch(/unreadable/i);
|
||||
});
|
||||
|
||||
it('refuses exact coverage when case-colliding managed paths exist', async () => {
|
||||
const stackName = 'casefold';
|
||||
const stackDir = path.join(composeDir, stackName);
|
||||
await fsPromises.mkdir(stackDir, { recursive: true });
|
||||
await fsPromises.writeFile(path.join(stackDir, 'compose.yaml'), 'services: {}\n', 'utf8');
|
||||
await fsPromises.writeFile(path.join(stackDir, 'App.conf'), 'A\n', 'utf8');
|
||||
await fsPromises.writeFile(path.join(stackDir, 'app.conf'), 'B\n', 'utf8');
|
||||
|
||||
mockGetGitSource.mockReturnValue(undefined);
|
||||
|
||||
// Force discovery to see both via include parse would be heavy; instead seed
|
||||
// by making them both compose roots is impossible. Use override + compose:
|
||||
mockGetOverrideFilename.mockResolvedValue('App.conf');
|
||||
// Also plant app.conf as project env so both enter the map.
|
||||
mockGetStackProjectEnvFiles.mockReturnValue(['app.conf']);
|
||||
|
||||
const inventory = await resolveRollbackInventory(1, stackName);
|
||||
// On case-sensitive FS both files exist; inventory should refuse collision.
|
||||
// On Windows case-folding FS the second write may overwrite the first.
|
||||
if (process.platform === 'win32') {
|
||||
expect(inventory).toBeTruthy();
|
||||
return;
|
||||
}
|
||||
expect(inventory.exactCoverage).toBe(false);
|
||||
expect(inventory.coverageRefusal).toMatch(/Case-colliding/i);
|
||||
});
|
||||
});
|
||||
@@ -28,6 +28,7 @@ const {
|
||||
mockRunCommand,
|
||||
mockDeployStack,
|
||||
mockBackupStackFiles,
|
||||
mockCaptureCurrentBackup,
|
||||
mockEnforcePolicyPreDeploy,
|
||||
} = vi.hoisted(() => ({
|
||||
mockGetDueScheduledTasks: vi.fn().mockReturnValue([]),
|
||||
@@ -76,6 +77,7 @@ const {
|
||||
mockRunCommand: vi.fn().mockResolvedValue(undefined),
|
||||
mockDeployStack: vi.fn().mockResolvedValue(undefined),
|
||||
mockBackupStackFiles: vi.fn().mockResolvedValue(undefined),
|
||||
mockCaptureCurrentBackup: vi.fn().mockResolvedValue({ id: 'gen-1' }),
|
||||
mockEnforcePolicyPreDeploy: vi.fn(),
|
||||
}));
|
||||
|
||||
@@ -165,6 +167,14 @@ vi.mock('../services/FileSystemService', () => ({
|
||||
},
|
||||
}));
|
||||
|
||||
vi.mock('../services/StackUpdateRecoveryService', () => ({
|
||||
StackUpdateRecoveryService: {
|
||||
getInstance: () => ({
|
||||
captureCurrentBackup: mockCaptureCurrentBackup,
|
||||
}),
|
||||
},
|
||||
}));
|
||||
|
||||
vi.mock('../services/ImageUpdateService', () => ({
|
||||
ImageUpdateService: {
|
||||
// Default on so existing executeUpdate tests keep prior behavior.
|
||||
@@ -1907,10 +1917,15 @@ describe('SchedulerService - lifecycle actions', () => {
|
||||
expect(mockUpdateScheduledTaskRun).toHaveBeenCalledWith(1, expect.objectContaining({ status: 'success' }));
|
||||
});
|
||||
|
||||
it('auto_backup calls backupStackFiles', async () => {
|
||||
it('auto_backup captures a current recovery generation', async () => {
|
||||
mockGetScheduledTask.mockReturnValue(makeLifecycleTask('auto_backup'));
|
||||
await SchedulerService.getInstance().triggerTask(300);
|
||||
expect(mockBackupStackFiles).toHaveBeenCalledWith('my-stack');
|
||||
expect(mockCaptureCurrentBackup).toHaveBeenCalledWith({
|
||||
nodeId: 1,
|
||||
stackName: 'my-stack',
|
||||
createdBy: 'system:scheduler',
|
||||
});
|
||||
expect(mockBackupStackFiles).not.toHaveBeenCalled();
|
||||
expect(mockUpdateScheduledTaskRun).toHaveBeenCalledWith(1, expect.objectContaining({ status: 'success' }));
|
||||
});
|
||||
|
||||
@@ -1924,6 +1939,7 @@ describe('SchedulerService - lifecycle actions', () => {
|
||||
it('auto_backup records failure when node_id is missing', async () => {
|
||||
mockGetScheduledTask.mockReturnValue(makeLifecycleTask('auto_backup', { node_id: null }));
|
||||
await SchedulerService.getInstance().triggerTask(300);
|
||||
expect(mockCaptureCurrentBackup).not.toHaveBeenCalled();
|
||||
expect(mockBackupStackFiles).not.toHaveBeenCalled();
|
||||
expect(mockUpdateScheduledTaskRun).toHaveBeenCalledWith(1, expect.objectContaining({ status: 'failure' }));
|
||||
});
|
||||
@@ -2015,6 +2031,7 @@ describe('SchedulerService - lifecycle remote proxy', () => {
|
||||
'http://remote:1852/api/stacks/my-stack/backup',
|
||||
expect.objectContaining({ method: 'POST' }),
|
||||
);
|
||||
expect(mockCaptureCurrentBackup).not.toHaveBeenCalled();
|
||||
expect(mockBackupStackFiles).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
@@ -2246,7 +2263,7 @@ describe('SchedulerService - delete_after_run', () => {
|
||||
});
|
||||
|
||||
it('does not delete task when run fails even if delete_after_run is 1', async () => {
|
||||
mockBackupStackFiles.mockRejectedValueOnce(new Error('disk full'));
|
||||
mockCaptureCurrentBackup.mockRejectedValueOnce(new Error('disk full'));
|
||||
mockGetScheduledTask.mockReturnValue(makeLifecycleTask('auto_backup', { delete_after_run: 1 }));
|
||||
await SchedulerService.getInstance().triggerTask(300);
|
||||
expect(mockDeleteScheduledTask).not.toHaveBeenCalled();
|
||||
|
||||
@@ -1,9 +1,9 @@
|
||||
/**
|
||||
* Integration tests for POST /api/stacks/:stackName/backup, the on-demand
|
||||
* stack-files backup trigger. Covers auth, role, the success path, the
|
||||
* recovery-generation capture. Covers auth, role, the success path, the
|
||||
* missing-stack 404, name validation, and error propagation. The route exists
|
||||
* so a scheduled auto_backup can run on a remote node through the proxy path,
|
||||
* and so an operator can take a snapshot on demand. The backup is available on
|
||||
* and so an operator can capture a current generation on demand. Available on
|
||||
* every tier (no paid gate).
|
||||
*/
|
||||
import { describe, it, expect, beforeAll, afterAll, beforeEach, vi } from 'vitest';
|
||||
@@ -11,9 +11,11 @@ import request from 'supertest';
|
||||
import bcrypt from 'bcrypt';
|
||||
import { setupTestDb, cleanupTestDb, loginAsTestAdmin } from './helpers/setupTestDb';
|
||||
|
||||
const { mockBackupStackFiles, mockHasComposeFile } = vi.hoisted(() => ({
|
||||
const { mockBackupStackFiles, mockHasComposeFile, mockCaptureCurrentBackup, mockGetCurrent } = vi.hoisted(() => ({
|
||||
mockBackupStackFiles: vi.fn(),
|
||||
mockHasComposeFile: vi.fn(),
|
||||
mockCaptureCurrentBackup: vi.fn(),
|
||||
mockGetCurrent: vi.fn(),
|
||||
}));
|
||||
|
||||
vi.mock('../services/FileSystemService', () => ({
|
||||
@@ -27,6 +29,15 @@ vi.mock('../services/FileSystemService', () => ({
|
||||
},
|
||||
}));
|
||||
|
||||
vi.mock('../services/StackUpdateRecoveryService', () => ({
|
||||
StackUpdateRecoveryService: {
|
||||
getInstance: () => ({
|
||||
getCurrent: mockGetCurrent,
|
||||
captureCurrentBackup: mockCaptureCurrentBackup,
|
||||
}),
|
||||
},
|
||||
}));
|
||||
|
||||
let tmpDir: string;
|
||||
let app: import('express').Express;
|
||||
let DatabaseService: typeof import('../services/DatabaseService').DatabaseService;
|
||||
@@ -59,6 +70,8 @@ afterAll(() => {
|
||||
beforeEach(() => {
|
||||
mockBackupStackFiles.mockReset().mockResolvedValue(undefined);
|
||||
mockHasComposeFile.mockReset().mockResolvedValue(true);
|
||||
mockCaptureCurrentBackup.mockReset().mockResolvedValue({ id: 'gen-1' });
|
||||
mockGetCurrent.mockReset().mockReturnValue(null);
|
||||
tierSpy.mockReturnValue('paid');
|
||||
});
|
||||
|
||||
@@ -67,18 +80,25 @@ describe('POST /api/stacks/:stackName/backup', () => {
|
||||
const res = await request(app).post('/api/stacks/web/backup').set('Cookie', adminCookie);
|
||||
expect(res.status).toBe(200);
|
||||
expect(res.body.success).toBe(true);
|
||||
expect(mockBackupStackFiles).toHaveBeenCalledWith('web');
|
||||
expect(mockCaptureCurrentBackup).toHaveBeenCalledWith(expect.objectContaining({
|
||||
nodeId: expect.any(Number),
|
||||
stackName: 'web',
|
||||
createdBy: expect.any(String),
|
||||
}));
|
||||
expect(mockBackupStackFiles).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('returns 401 without an auth cookie', async () => {
|
||||
const res = await request(app).post('/api/stacks/web/backup');
|
||||
expect(res.status).toBe(401);
|
||||
expect(mockCaptureCurrentBackup).not.toHaveBeenCalled();
|
||||
expect(mockBackupStackFiles).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('returns 403 for a viewer (no deploy permission)', async () => {
|
||||
const res = await request(app).post('/api/stacks/web/backup').set('Cookie', viewerCookie);
|
||||
expect(res.status).toBe(403);
|
||||
expect(mockCaptureCurrentBackup).not.toHaveBeenCalled();
|
||||
expect(mockBackupStackFiles).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
@@ -87,32 +107,39 @@ describe('POST /api/stacks/:stackName/backup', () => {
|
||||
const res = await request(app).post('/api/stacks/web/backup').set('Cookie', adminCookie);
|
||||
expect(res.status).toBe(200);
|
||||
expect(res.body.success).toBe(true);
|
||||
expect(mockBackupStackFiles).toHaveBeenCalledWith('web');
|
||||
expect(mockCaptureCurrentBackup).toHaveBeenCalledWith(expect.objectContaining({
|
||||
nodeId: expect.any(Number),
|
||||
stackName: 'web',
|
||||
createdBy: expect.any(String),
|
||||
}));
|
||||
expect(mockBackupStackFiles).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('returns 404 when the stack does not exist', async () => {
|
||||
mockHasComposeFile.mockResolvedValue(false);
|
||||
const res = await request(app).post('/api/stacks/ghost/backup').set('Cookie', adminCookie);
|
||||
expect(res.status).toBe(404);
|
||||
expect(mockCaptureCurrentBackup).not.toHaveBeenCalled();
|
||||
expect(mockBackupStackFiles).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('returns 400 for an invalid stack name', async () => {
|
||||
const res = await request(app).post('/api/stacks/..bad../backup').set('Cookie', adminCookie);
|
||||
expect(res.status).toBe(400);
|
||||
expect(mockCaptureCurrentBackup).not.toHaveBeenCalled();
|
||||
expect(mockBackupStackFiles).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('returns 500 when the backup operation fails', async () => {
|
||||
mockBackupStackFiles.mockRejectedValue(new Error('disk full'));
|
||||
mockCaptureCurrentBackup.mockRejectedValue(new Error('disk full'));
|
||||
const res = await request(app).post('/api/stacks/web/backup').set('Cookie', adminCookie);
|
||||
expect(res.status).toBe(500);
|
||||
expect(res.body.error).toContain('disk full');
|
||||
});
|
||||
|
||||
it('returns 409 when the stack is busy with another operation', async () => {
|
||||
// The backup shares the rollback slot, so it must not run while a deploy
|
||||
// holds the stack-op lock for the same stack.
|
||||
// Handoff mutates the current generation, so backup must not run while a
|
||||
// deploy holds the stack-op lock for the same stack.
|
||||
const { StackOpLockService } = await import('../services/StackOpLockService');
|
||||
const localNodeId = DatabaseService.getInstance().getNodes().find(n => n.type === 'local')!.id;
|
||||
StackOpLockService.getInstance().tryAcquire(localNodeId, 'web', 'deploy', 'someone');
|
||||
@@ -120,6 +147,7 @@ describe('POST /api/stacks/:stackName/backup', () => {
|
||||
const res = await request(app).post('/api/stacks/web/backup').set('Cookie', adminCookie);
|
||||
expect(res.status).toBe(409);
|
||||
expect(res.body.code).toBe('stack_op_in_progress');
|
||||
expect(mockCaptureCurrentBackup).not.toHaveBeenCalled();
|
||||
expect(mockBackupStackFiles).not.toHaveBeenCalled();
|
||||
} finally {
|
||||
StackOpLockService.getInstance().release(localNodeId, 'web');
|
||||
|
||||
@@ -98,7 +98,7 @@ afterAll(() => {
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
mockDeployStack.mockReset();
|
||||
mockDeployStack.mockReset().mockResolvedValue({ recoveryId: null });
|
||||
mockRunCommand.mockReset();
|
||||
mockRunDown.mockReset();
|
||||
mockUpdateStack.mockReset();
|
||||
@@ -128,7 +128,7 @@ function deferred<T>(): Deferred<T> {
|
||||
|
||||
describe('Stack lifecycle mutex', () => {
|
||||
it('returns 409 with stack_op_in_progress when a deploy is already running', async () => {
|
||||
const gate = deferred<void>();
|
||||
const gate = deferred<{ recoveryId: string | null }>();
|
||||
mockDeployStack.mockImplementationOnce(() => gate.promise);
|
||||
|
||||
const first = request(app)
|
||||
@@ -152,20 +152,20 @@ describe('Stack lifecycle mutex', () => {
|
||||
expect(second.body.error).toMatch(/already deploying/i);
|
||||
expect(typeof second.body.inProgress.startedAt).toBe('number');
|
||||
|
||||
gate.resolve();
|
||||
gate.resolve({ recoveryId: null });
|
||||
const firstRes = await first;
|
||||
expect(firstRes.status).toBe(200);
|
||||
});
|
||||
|
||||
it('releases the lock after a successful deploy so the next request acquires', async () => {
|
||||
mockDeployStack.mockResolvedValueOnce(undefined);
|
||||
mockDeployStack.mockResolvedValueOnce({ recoveryId: null });
|
||||
const first = await request(app)
|
||||
.post('/api/stacks/web/deploy')
|
||||
.set('Cookie', authCookie)
|
||||
.send({ skip_scan: true });
|
||||
expect(first.status).toBe(200);
|
||||
|
||||
mockDeployStack.mockResolvedValueOnce(undefined);
|
||||
mockDeployStack.mockResolvedValueOnce({ recoveryId: null });
|
||||
const second = await request(app)
|
||||
.post('/api/stacks/web/deploy')
|
||||
.set('Cookie', authCookie)
|
||||
@@ -181,7 +181,7 @@ describe('Stack lifecycle mutex', () => {
|
||||
.send({ skip_scan: true });
|
||||
expect(first.status).toBe(500);
|
||||
|
||||
mockDeployStack.mockResolvedValueOnce(undefined);
|
||||
mockDeployStack.mockResolvedValueOnce({ recoveryId: null });
|
||||
const second = await request(app)
|
||||
.post('/api/stacks/web/deploy')
|
||||
.set('Cookie', authCookie)
|
||||
@@ -190,7 +190,7 @@ describe('Stack lifecycle mutex', () => {
|
||||
});
|
||||
|
||||
it('blocks restart while a deploy is in flight on the same stack', async () => {
|
||||
const gate = deferred<void>();
|
||||
const gate = deferred<{ recoveryId: string | null }>();
|
||||
mockDeployStack.mockImplementationOnce(() => gate.promise);
|
||||
|
||||
const deploy = request(app)
|
||||
@@ -207,12 +207,12 @@ describe('Stack lifecycle mutex', () => {
|
||||
expect(restart.body.code).toBe('stack_op_in_progress');
|
||||
expect(restart.body.inProgress.action).toBe('deploy');
|
||||
|
||||
gate.resolve();
|
||||
gate.resolve({ recoveryId: null });
|
||||
await deploy;
|
||||
});
|
||||
|
||||
it('allows concurrent ops on different stacks', async () => {
|
||||
const gate = deferred<void>();
|
||||
const gate = deferred<{ recoveryId: string | null }>();
|
||||
mockDeployStack.mockImplementation(() => gate.promise);
|
||||
|
||||
const webDeploy = request(app)
|
||||
@@ -229,7 +229,7 @@ describe('Stack lifecycle mutex', () => {
|
||||
.then(r => r);
|
||||
await vi.waitFor(() => expect(mockDeployStack).toHaveBeenCalledTimes(2));
|
||||
|
||||
gate.resolve();
|
||||
gate.resolve({ recoveryId: null });
|
||||
const [webRes, apiRes] = await Promise.all([webDeploy, apiDeploy]);
|
||||
expect(webRes.status).toBe(200);
|
||||
expect(apiRes.status).toBe(200);
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -39,6 +39,9 @@ describe('classifyStackApiPath', () => {
|
||||
expect(classifyStackApiPath('DELETE', '/stacks/web/git-source')).toEqual({
|
||||
kind: 'named-stack', stackName: 'web', action: 'stack:edit',
|
||||
});
|
||||
expect(classifyStackApiPath('POST', '/stacks/web/fleet-snapshot-apply')).toEqual({
|
||||
kind: 'named-stack', stackName: 'web', action: 'stack:edit',
|
||||
});
|
||||
});
|
||||
|
||||
it('maps deploy routes and service lifecycle ops to stack:deploy', () => {
|
||||
|
||||
@@ -13,6 +13,7 @@ import jwt from 'jsonwebtoken';
|
||||
import { setupTestDb, cleanupTestDb, loginAsTestAdmin, TEST_JWT_SECRET } from './helpers/setupTestDb';
|
||||
import { ComposeRollbackError } from '../services/ComposeService';
|
||||
import * as policyGate from '../helpers/policyGate';
|
||||
import type { StackUpdateRecoveryGenerationRow } from '../services/DatabaseService';
|
||||
|
||||
// ── Hoisted mocks (must come before importing the app) ──────────────────────
|
||||
|
||||
@@ -141,10 +142,10 @@ afterAll(() => {
|
||||
});
|
||||
|
||||
beforeEach(() => {
|
||||
mockDeployStack.mockReset();
|
||||
mockDeployStack.mockReset().mockResolvedValue({ recoveryId: null });
|
||||
mockRunCommand.mockReset();
|
||||
mockRunDown.mockReset();
|
||||
mockUpdateStack.mockReset();
|
||||
mockUpdateStack.mockReset().mockResolvedValue({ recoveryId: null });
|
||||
mockGetContainersByStack.mockReset();
|
||||
mockRestartContainer.mockReset();
|
||||
mockStopContainer.mockReset();
|
||||
@@ -233,7 +234,7 @@ describe('deploy_failure notification on /deploy error', () => {
|
||||
});
|
||||
|
||||
it('uses trusted proxy tier headers for remote atomic deploys', async () => {
|
||||
mockDeployStack.mockResolvedValue(undefined);
|
||||
mockDeployStack.mockResolvedValue({ recoveryId: null });
|
||||
const token = jwt.sign({ scope: 'node_proxy' }, TEST_JWT_SECRET, { expiresIn: '1m' });
|
||||
|
||||
const res = await request(app)
|
||||
@@ -259,7 +260,7 @@ describe('health gate begin call sites', () => {
|
||||
});
|
||||
|
||||
it('begins a gate after a manual deploy and returns its id', async () => {
|
||||
mockDeployStack.mockResolvedValue(undefined);
|
||||
mockDeployStack.mockResolvedValue({ recoveryId: null });
|
||||
const res = await request(app)
|
||||
.post('/api/stacks/myapp/deploy')
|
||||
.set('Cookie', authCookie)
|
||||
@@ -269,6 +270,19 @@ describe('health gate begin call sites', () => {
|
||||
expect(res.body.healthGateId).toBe('gate-123');
|
||||
});
|
||||
|
||||
it('links the deploy recovery generation to the observing gate', async () => {
|
||||
mockDeployStack.mockResolvedValue({ recoveryId: 'rec-deploy' });
|
||||
const { StackUpdateRecoveryService } = await import('../services/StackUpdateRecoveryService');
|
||||
const linkSpy = vi.spyOn(StackUpdateRecoveryService.getInstance(), 'linkGateOrRetain');
|
||||
const res = await request(app)
|
||||
.post('/api/stacks/myapp/deploy')
|
||||
.set('Cookie', authCookie)
|
||||
.send({ skip_scan: true });
|
||||
expect(res.status).toBe(200);
|
||||
expect(linkSpy).toHaveBeenCalledWith('rec-deploy', 'gate-123');
|
||||
linkSpy.mockRestore();
|
||||
});
|
||||
|
||||
it('begins a gate after a manual update and returns its id', async () => {
|
||||
mockUpdateStack.mockResolvedValue({ recoveryId: null });
|
||||
const res = await request(app)
|
||||
@@ -304,7 +318,7 @@ describe('health gate begin call sites', () => {
|
||||
});
|
||||
|
||||
it('never begins a gate for the rollback recovery path', async () => {
|
||||
mockDeployStack.mockResolvedValue(undefined);
|
||||
mockDeployStack.mockResolvedValue({ recoveryId: null });
|
||||
const res = await request(app)
|
||||
.post('/api/stacks/myapp/rollback')
|
||||
.set('Cookie', authCookie);
|
||||
@@ -370,13 +384,39 @@ describe('failure classification on deploy/update error responses', () => {
|
||||
.set('Cookie', authCookie);
|
||||
|
||||
expect(res.status).toBe(500);
|
||||
expect(res.body.failure.reason).toBe('unknown');
|
||||
expect(res.body.failure).toMatchObject({ reason: 'unknown' });
|
||||
});
|
||||
|
||||
it('classifies mixed replica image capture refusals', async () => {
|
||||
mockDeployStack.mockRejectedValue(
|
||||
new Error('Service "web" has mixed replica images; refusing recovery capture that cannot restore exact prior identity'),
|
||||
);
|
||||
|
||||
const res = await request(app)
|
||||
.post('/api/stacks/myapp/deploy')
|
||||
.set('Cookie', authCookie);
|
||||
|
||||
expect(res.status).toBe(500);
|
||||
expect(res.body.failure).toMatchObject({ reason: 'mixed_replica_images' });
|
||||
});
|
||||
|
||||
it('classifies rollback coverage refusals', async () => {
|
||||
mockDeployStack.mockRejectedValue(
|
||||
new Error('Host-absolute include path cannot be captured for exact rollback'),
|
||||
);
|
||||
|
||||
const res = await request(app)
|
||||
.post('/api/stacks/myapp/deploy')
|
||||
.set('Cookie', authCookie);
|
||||
|
||||
expect(res.status).toBe(500);
|
||||
expect(res.body.failure).toMatchObject({ reason: 'rollback_coverage_unavailable' });
|
||||
});
|
||||
});
|
||||
|
||||
describe('post-deploy scan opt-out', () => {
|
||||
it('does not trigger a post-deploy scan when skip_scan is true', async () => {
|
||||
mockDeployStack.mockResolvedValue(undefined);
|
||||
mockDeployStack.mockResolvedValue({ recoveryId: null });
|
||||
|
||||
const res = await request(app)
|
||||
.post('/api/stacks/myapp/deploy')
|
||||
@@ -503,6 +543,76 @@ describe('deploy_failure notification on /update error', () => {
|
||||
});
|
||||
});
|
||||
|
||||
describe('generation rollback error mapping', () => {
|
||||
function stubCurrentGeneration(id: string): StackUpdateRecoveryGenerationRow {
|
||||
return {
|
||||
id,
|
||||
node_id: 1,
|
||||
stack_name: 'myapp',
|
||||
status: 'active',
|
||||
phase: 'immediate_verified',
|
||||
is_current: 1,
|
||||
backup_slot_id: id,
|
||||
content_path: null,
|
||||
operation_kind: null,
|
||||
override_path: '/tmp/override.yml',
|
||||
services_json: '[]',
|
||||
health_gate_id: null,
|
||||
gate_retain_until: null,
|
||||
artifact_expires_at: null,
|
||||
operation_lease_expires_at: null,
|
||||
created_at: 0,
|
||||
updated_at: 0,
|
||||
created_by: null,
|
||||
artifacts_retired: 0,
|
||||
released_at: null,
|
||||
released_by: null,
|
||||
};
|
||||
}
|
||||
|
||||
async function withGenerationRollback(compensate: () => Promise<boolean>) {
|
||||
const { StackUpdateRecoveryService } = await import('../services/StackUpdateRecoveryService');
|
||||
const svc = StackUpdateRecoveryService.getInstance();
|
||||
const getSpy = vi.spyOn(svc, 'getCurrent').mockReturnValue(stubCurrentGeneration('gen-1'));
|
||||
const compensateSpy = vi.spyOn(svc, 'compensateWithCandidate').mockImplementation(compensate);
|
||||
try {
|
||||
return await request(app).post('/api/stacks/myapp/rollback').set('Cookie', authCookie);
|
||||
} finally {
|
||||
getSpy.mockRestore();
|
||||
compensateSpy.mockRestore();
|
||||
}
|
||||
}
|
||||
|
||||
it('returns HELD_IMAGE_MISSING when the hold tag is gone', async () => {
|
||||
const res = await withGenerationRollback(async () => {
|
||||
throw Object.assign(new Error('Held recovery image is missing'), { code: 'HELD_IMAGE_MISSING' });
|
||||
});
|
||||
expect(res.status).toBe(500);
|
||||
expect(res.body).toMatchObject({
|
||||
code: 'HELD_IMAGE_MISSING',
|
||||
error: 'Held recovery image is missing',
|
||||
});
|
||||
});
|
||||
|
||||
it('returns RECOVERY_PROBE_FAILED when restore completed but the probe failed', async () => {
|
||||
const res = await withGenerationRollback(async () => {
|
||||
throw Object.assign(new Error('Recovery health probe failed'), { code: 'RECOVERY_PROBE_FAILED' });
|
||||
});
|
||||
expect(res.status).toBe(500);
|
||||
expect(res.body).toMatchObject({
|
||||
code: 'RECOVERY_PROBE_FAILED',
|
||||
error: 'Rollback restore completed but recovery probe failed.',
|
||||
});
|
||||
});
|
||||
|
||||
it('returns a generic restore failure when compensation returns false', async () => {
|
||||
const res = await withGenerationRollback(async () => false);
|
||||
expect(res.status).toBe(500);
|
||||
expect(res.body).toMatchObject({ error: 'Rollback restore did not complete.' });
|
||||
expect(res.body.code).toBeUndefined();
|
||||
});
|
||||
});
|
||||
|
||||
describe('rollback file-revert safety on a policy-blocked rollback', () => {
|
||||
it('does not deploy and alerts the operator when the post-block file revert fails', async () => {
|
||||
// The restored backup is blocked by policy after files were already restored.
|
||||
|
||||
@@ -123,7 +123,7 @@ beforeEach(() => {
|
||||
envWritten: false,
|
||||
warnings: [],
|
||||
});
|
||||
mockDeployStack.mockResolvedValue(undefined);
|
||||
mockDeployStack.mockResolvedValue({ recoveryId: null });
|
||||
mockIsTrivyAvailable.mockReturnValue(true);
|
||||
mockListContainers.mockResolvedValue([{ Image: 'nginx:latest' }]);
|
||||
mockGetImageDigest.mockResolvedValue(null);
|
||||
|
||||
@@ -36,6 +36,9 @@ const baseInputs = (over: Partial<RollbackInputs> = {}): RollbackInputs => ({
|
||||
name: 'app-web-1', state: 'running', health: 'healthy', exitCode: null,
|
||||
hasHealthcheck: true, restartPolicy: 'unless-stopped', mounts: ['volume app_data'],
|
||||
}],
|
||||
recoveryGeneration: { exists: false },
|
||||
policyEligibility: null,
|
||||
managedInputs: { covered: true, detail: 'Exact authored-project coverage includes 1 managed path(s).' },
|
||||
...over,
|
||||
});
|
||||
|
||||
@@ -43,19 +46,20 @@ const itemById = (inputs: RollbackInputs, id: string) =>
|
||||
buildRollbackItems(inputs, NOW).find(i => i.id === id)!;
|
||||
|
||||
describe('buildRollbackItems', () => {
|
||||
it('reports a full set of six items', () => {
|
||||
it('reports the full set of rollback readiness items', () => {
|
||||
const items = buildRollbackItems(baseInputs(), NOW);
|
||||
expect(items.map(i => i.id)).toEqual([
|
||||
'compose_source', 'env_keys', 'previous_images', 'last_deploy', 'healthchecks', 'volume_data',
|
||||
'recovery_generation', 'compose_source', 'env_keys', 'previous_images', 'last_deploy', 'healthchecks',
|
||||
'policy_eligibility', 'managed_inputs', 'volume_data',
|
||||
]);
|
||||
});
|
||||
|
||||
it('marks the volume row not_covered unconditionally and names the mounts', () => {
|
||||
const item = itemById(baseInputs(), 'volume_data');
|
||||
expect(item.state).toBe('not_covered');
|
||||
expect(item.detail).toContain('not included in file backups');
|
||||
expect(item.detail).toContain('not included in recovery generations');
|
||||
expect(item.detail).toContain('volume app_data');
|
||||
|
||||
expect(item.detail).toMatch(/managed authored project/i);
|
||||
const noMounts = itemById(baseInputs({ containers: [] }), 'volume_data');
|
||||
expect(noMounts.state).toBe('not_covered');
|
||||
|
||||
@@ -122,6 +126,28 @@ describe('buildRollbackItems', () => {
|
||||
});
|
||||
});
|
||||
|
||||
|
||||
it('marks policy eligibility blocked and warning states', () => {
|
||||
expect(itemById(baseInputs({
|
||||
recoveryGeneration: { exists: true, shortId: 'abc' },
|
||||
policyEligibility: 'prohibited',
|
||||
}), 'policy_eligibility').state).toBe('blocked');
|
||||
expect(itemById(baseInputs({
|
||||
recoveryGeneration: { exists: true, shortId: 'abc' },
|
||||
policyEligibility: 'eligible_with_warning',
|
||||
}), 'policy_eligibility').state).toBe('warning');
|
||||
});
|
||||
|
||||
it('lets recovery_generation supersede the legacy backup slot for compose_source', () => {
|
||||
const items = buildRollbackItems(baseInputs({
|
||||
backup: { exists: false, timestamp: null },
|
||||
recoveryGeneration: { exists: true, shortId: 'deadbeefcaf0' },
|
||||
policyEligibility: 'eligible',
|
||||
}), NOW);
|
||||
expect(items.find(i => i.id === 'compose_source')!.state).toBe('ready');
|
||||
expect(items.find(i => i.id === 'recovery_generation')!.state).toBe('ready');
|
||||
});
|
||||
|
||||
describe('aggregateRollbackOverall', () => {
|
||||
it('is ready when compose, env, and previous image are all covered', () => {
|
||||
expect(aggregateRollbackOverall(buildRollbackItems(baseInputs(), NOW))).toBe('ready');
|
||||
@@ -157,6 +183,27 @@ describe('aggregateRollbackOverall', () => {
|
||||
const items = buildRollbackItems(baseInputs({ containers: 'error' }), NOW);
|
||||
expect(aggregateRollbackOverall(items)).toBe('ready');
|
||||
});
|
||||
it('is not_ready when policy eligibility is blocked', () => {
|
||||
const items = buildRollbackItems(baseInputs({
|
||||
recoveryGeneration: { exists: true, shortId: 'abc' },
|
||||
policyEligibility: 'prohibited',
|
||||
}), NOW);
|
||||
expect(aggregateRollbackOverall(items)).toBe('not_ready');
|
||||
});
|
||||
|
||||
it('is at best partial when policy eligibility is warning or unknown', () => {
|
||||
const warn = buildRollbackItems(baseInputs({
|
||||
recoveryGeneration: { exists: true, shortId: 'abc' },
|
||||
policyEligibility: 'eligible_with_warning',
|
||||
}), NOW);
|
||||
expect(aggregateRollbackOverall(warn)).toBe('partial');
|
||||
const unk = buildRollbackItems(baseInputs({
|
||||
recoveryGeneration: { exists: true, shortId: 'abc' },
|
||||
policyEligibility: 'unknown',
|
||||
}), NOW);
|
||||
expect(aggregateRollbackOverall(unk)).toBe('partial');
|
||||
});
|
||||
|
||||
});
|
||||
|
||||
describe('FileSystemService.getBackupEnvSummary', () => {
|
||||
|
||||
@@ -74,6 +74,8 @@ vi.mock('../services/DatabaseService', () => ({
|
||||
// Rollback-readiness partial-revert disclosure reads the git source row;
|
||||
// no git-managed stacks in these fixtures by default.
|
||||
getGitSource: mockGetGitSource,
|
||||
// Generation supersede path; no current recovery generation in these fixtures.
|
||||
getCurrentStackUpdateRecovery: () => undefined,
|
||||
}),
|
||||
},
|
||||
}));
|
||||
|
||||
@@ -461,12 +461,15 @@ describe('WebhookService.execute: health gate begin call sites', () => {
|
||||
const { HealthGateService } = await import('../services/HealthGateService');
|
||||
vi.spyOn(policyGate, 'assertPolicyGateAllows').mockResolvedValue(undefined);
|
||||
vi.spyOn(fs.FileSystemService.prototype, 'getStacks').mockResolvedValue([stack]);
|
||||
vi.spyOn(compose.ComposeService.prototype, 'deployStack').mockResolvedValue(undefined);
|
||||
vi.spyOn(compose.ComposeService.prototype, 'deployStack').mockResolvedValue({ recoveryId: 'rec-hook' });
|
||||
const beginSpy = vi.spyOn(HealthGateService.getInstance(), 'beginStack').mockReturnValue('gate-hook');
|
||||
const { StackUpdateRecoveryService } = await import('../services/StackUpdateRecoveryService');
|
||||
const linkSpy = vi.spyOn(StackUpdateRecoveryService.getInstance(), 'linkGateOrRetain');
|
||||
|
||||
const result = await WebhookService.getInstance().execute(webhook, 'deploy', 'test', true);
|
||||
expect(result.success).toBe(true);
|
||||
expect(beginSpy).toHaveBeenCalledWith(nodeId, stack, 'deploy', 'system:webhook');
|
||||
expect(linkSpy).toHaveBeenCalledWith('rec-hook', 'gate-hook');
|
||||
});
|
||||
|
||||
it('begins an update gate after a webhook pull succeeds', async () => {
|
||||
|
||||
Reference in New Issue
Block a user