mirror of
https://github.com/Studio-Saelix/sencho.git
synced 2026-08-20 23:32:19 +00:00
5f1baa7522
* fix: harden deploy/update concurrency and node-targeting safety Release stabilization for deploy/update operational safety. Per-stack operation locking is now global. Background lifecycle paths (scheduler auto stop/down/start/backup/update, webhook execute, Git source auto-deploy, image auto-update, label bulk actions, fleet snapshot redeploy, and mesh redeploy) acquire the per-node, per-stack lock through a new StackOpLockService.runExclusive helper and skip rather than race a manual deploy/update/rollback/backup on the same stack and node. Skips surface honestly (a failed scheduled run, a recorded webhook failure, a per-stack batch result, or a thrown error) instead of a silent no-op. Update readiness and policy-bypass now run against the node captured when the dialog opened, not the live active node, so switching nodes while a dialog is open cannot retarget the update or the bypass retry. Rollback readiness no longer presents a moving-tag or unpinned image as a ready image revert. Restoring files does not revert a moving tag, so those stacks read as partial, and the rollback success message states that the compose and env files were restored. * fix: lock blueprint reconcile against manual ops and correct rollback wording Follow-up to the deploy/update safety hardening, closing two more gaps from a verification pass. BlueprintService.deployLocal and withdrawLocal called ComposeService directly, so blueprint reconciliation could race a manual deploy/update/rollback/backup on an owned stack. Both now run their compose lifecycle call through StackOpLockService.runExclusive and skip (recorded as a failed reconcile, retried on the next cycle) on conflict. The withdraw holds the lock across both the compose down and the directory delete so neither races a manual operation. The runtime rollback messages overstated recovery: a rollback restores the compose and env files and recreates containers, but does not revert an image behind a moving tag. The auto-rollback deploy-progress output, the recovery panel and chip, the failure toasts, and the manual rollback route message now state that the compose and env files were restored, with the matching OpenAPI example and atomic-deployments doc updated. * fix: acquire stack lock before blueprint deploy mutates compose and marker files Local blueprint deploy wrote the compose and marker files and ran the policy assert before acquiring the per-stack lock; the lock only wrapped the deploy itself. A reconcile could therefore rewrite an owned stack's files while a manual deploy/update/rollback/backup was running. The lock now wraps the whole critical section (create, write compose, write marker, policy assert, deploy), so on conflict nothing is written and the reconcile records a failed outcome. Adds a test asserting a deploy under a held lock records failed, writes no marker file, and leaves the manual lock untouched. * fix: make remote blueprint apply atomic under the receiving node's stack lock Remote blueprint deploy wrote the compose and marker files to the target node via separate HTTP calls and only locked on the final deploy, so the file writes could race a manual operation on that node. A node's operation lock is process-local and cannot be held by the hub across HTTP calls, so the locked create/write/deploy now runs on the receiving node. The locked critical section is extracted into BlueprintService.applyLocalUnderLock and exposed via POST /api/blueprints/apply-local. The hub posts the blueprint to that endpoint in one call; the receiving node runs create + write compose+marker + deploy under its own per-stack lock. Older nodes without the route answer 404 and fall back to the legacy multi-call flow. The endpoint is gated by paid tier and the same per-stack stack:edit and stack:deploy permissions as the PUT-compose + deploy it bundles, validates the stack name, compose size, and marker structure, and returns 409 on a lock conflict without writing anything. Adds tests for the atomic single-call path, the 404 legacy fallback, the 409 lock-conflict mapping, the route validation and permission paths, and the write-compose-then-marker-then-deploy ordering of the shared locked apply. * fix(deps): bump undici to 7.28.0 to clear high-severity advisory The frontend CI npm audit gate (--audit-level=high) failed on a transitive undici 7.25.0 (a dev-only dependency via jsdom): TLS certificate validation bypass (GHSA-vmh5-mc38-953g) and cross-user cache information disclosure (GHSA-pr7r-676h-xcf6). Bumping undici within jsdom's existing ^7.25.0 range to 7.28.0 clears the high-severity advisory and unblocks the frontend job. Lockfile only; no direct dependency or source change.
210 lines
9.6 KiB
TypeScript
210 lines
9.6 KiB
TypeScript
/**
|
|
* Unit tests for the remote (proxy) branch of BlueprintService deploy/withdraw.
|
|
*
|
|
* The remote path talks to a sibling Sencho's /api/stacks surface over HTTP
|
|
* (create stack, write compose, write marker, deploy). These tests mock that
|
|
* surface via axios so we can assert the call ordering, the 409-on-create
|
|
* "already exists" tolerance, the failure mapping to status='failed', the
|
|
* name-conflict guard, and the withdraw delete path, without a live remote.
|
|
*/
|
|
import { describe, it, expect, beforeAll, afterAll, beforeEach, afterEach, vi } from 'vitest';
|
|
import axios from 'axios';
|
|
|
|
let tmpDir: string;
|
|
let DatabaseService: typeof import('../services/DatabaseService').DatabaseService;
|
|
let BlueprintService: typeof import('../services/BlueprintService').BlueprintService;
|
|
let NodeRegistry: typeof import('../services/NodeRegistry').NodeRegistry;
|
|
let setupTestDb: typeof import('./helpers/setupTestDb').setupTestDb;
|
|
let cleanupTestDb: typeof import('./helpers/setupTestDb').cleanupTestDb;
|
|
let counter = 0;
|
|
|
|
function seedRemoteNode(): { id: number; name: string } {
|
|
counter += 1;
|
|
const name = `bp-remote-${counter}`;
|
|
const id = DatabaseService.getInstance().addNode({
|
|
name,
|
|
type: 'remote',
|
|
mode: 'proxy',
|
|
compose_dir: '/tmp/compose',
|
|
is_default: false,
|
|
api_url: 'https://remote.example.com:1852',
|
|
api_token: 'remote-tok',
|
|
});
|
|
return { id, name };
|
|
}
|
|
|
|
function seedBlueprint(nodeIds: number[]) {
|
|
counter += 1;
|
|
return DatabaseService.getInstance().createBlueprint({
|
|
name: `bp-remote-bp-${counter}`,
|
|
description: null,
|
|
compose_content: 'services:\n app:\n image: nginx\n',
|
|
selector: { type: 'nodes', ids: nodeIds },
|
|
drift_mode: 'suggest',
|
|
classification: 'stateless',
|
|
classification_reasons: [],
|
|
enabled: true,
|
|
created_by: 'admin',
|
|
});
|
|
}
|
|
|
|
beforeAll(async () => {
|
|
({ setupTestDb, cleanupTestDb } = await import('./helpers/setupTestDb'));
|
|
tmpDir = await setupTestDb();
|
|
({ DatabaseService } = await import('../services/DatabaseService'));
|
|
({ BlueprintService } = await import('../services/BlueprintService'));
|
|
({ NodeRegistry } = await import('../services/NodeRegistry'));
|
|
});
|
|
|
|
afterAll(() => cleanupTestDb(tmpDir));
|
|
|
|
beforeEach(() => {
|
|
vi.restoreAllMocks();
|
|
vi.spyOn(NodeRegistry.getInstance(), 'getProxyTarget').mockReturnValue({
|
|
apiUrl: 'https://remote.example.com:1852',
|
|
apiToken: 'remote-tok',
|
|
});
|
|
const db = DatabaseService.getInstance().getDb();
|
|
db.prepare('DELETE FROM blueprint_deployments').run();
|
|
db.prepare('DELETE FROM blueprints').run();
|
|
db.prepare('DELETE FROM nodes WHERE is_default = 0').run();
|
|
});
|
|
|
|
afterEach(() => vi.restoreAllMocks());
|
|
|
|
describe('BlueprintService remote deploy', () => {
|
|
it('applies atomically via the remote apply-local endpoint in a single call', async () => {
|
|
const node = seedRemoteNode();
|
|
const bp = seedBlueprint([node.id]);
|
|
const nodeObj = DatabaseService.getInstance().getNode(node.id)!;
|
|
const bpObj = DatabaseService.getInstance().getBlueprint(bp.id)!;
|
|
|
|
vi.spyOn(axios, 'get').mockResolvedValue({ status: 200, data: [] }); // hasNameConflict: no stacks
|
|
const putSpy = vi.spyOn(axios, 'put').mockResolvedValue({ status: 200, data: {} });
|
|
const postSpy = vi.spyOn(axios, 'post').mockResolvedValue({ status: 200, data: { deployed: true } });
|
|
|
|
const result = await BlueprintService.getInstance().deployToNode(bpObj, nodeObj);
|
|
|
|
expect(result.status).toBe('active');
|
|
// One atomic call to the remote (create + write + deploy run under the
|
|
// remote's lock); no separate file PUTs from the hub.
|
|
expect(postSpy).toHaveBeenCalledTimes(1);
|
|
expect(postSpy.mock.calls[0][0]).toMatch(/\/api\/blueprints\/apply-local$/);
|
|
expect(putSpy).not.toHaveBeenCalled();
|
|
const payload = postSpy.mock.calls[0][1] as { stackName: string; composeContent: string; markerContent: string };
|
|
expect(payload.stackName).toBe(bpObj.name);
|
|
expect(typeof payload.composeContent).toBe('string');
|
|
expect(typeof payload.markerContent).toBe('string');
|
|
|
|
const dep = DatabaseService.getInstance().getDeployment(bp.id, node.id);
|
|
expect(dep?.status).toBe('active');
|
|
expect(dep?.applied_revision).toBe(bpObj.revision);
|
|
});
|
|
|
|
it('falls back to the legacy create/write/deploy flow when the remote lacks apply-local (404)', async () => {
|
|
const node = seedRemoteNode();
|
|
const bp = seedBlueprint([node.id]);
|
|
const nodeObj = DatabaseService.getInstance().getNode(node.id)!;
|
|
const bpObj = DatabaseService.getInstance().getBlueprint(bp.id)!;
|
|
|
|
vi.spyOn(axios, 'get').mockResolvedValue({ status: 200, data: [] });
|
|
const putSpy = vi.spyOn(axios, 'put').mockResolvedValue({ status: 200, data: {} });
|
|
const postSpy = vi.spyOn(axios, 'post')
|
|
.mockResolvedValueOnce({ status: 404, data: {} }) // apply-local missing on older node
|
|
.mockResolvedValueOnce({ status: 201, data: {} }) // legacy create stack
|
|
.mockResolvedValueOnce({ status: 200, data: {} }); // legacy deploy
|
|
|
|
const result = await BlueprintService.getInstance().deployToNode(bpObj, nodeObj);
|
|
|
|
expect(result.status).toBe('active');
|
|
expect(postSpy.mock.calls[0][0]).toMatch(/\/api\/blueprints\/apply-local$/);
|
|
expect(postSpy.mock.calls[1][0]).toMatch(/\/api\/stacks$/);
|
|
expect(putSpy).toHaveBeenCalledTimes(2); // compose + marker
|
|
expect(postSpy.mock.calls[2][0]).toMatch(/\/deploy$/);
|
|
});
|
|
|
|
it('maps a remote apply lock-conflict (409) to status=failed', async () => {
|
|
const node = seedRemoteNode();
|
|
const bp = seedBlueprint([node.id]);
|
|
const nodeObj = DatabaseService.getInstance().getNode(node.id)!;
|
|
const bpObj = DatabaseService.getInstance().getBlueprint(bp.id)!;
|
|
|
|
vi.spyOn(axios, 'get').mockResolvedValue({ status: 200, data: [] });
|
|
const putSpy = vi.spyOn(axios, 'put').mockResolvedValue({ status: 200, data: {} });
|
|
vi.spyOn(axios, 'post').mockResolvedValue({
|
|
status: 409,
|
|
data: { error: 'web is busy: another operation (update) is already in progress' },
|
|
});
|
|
|
|
const result = await BlueprintService.getInstance().deployToNode(bpObj, nodeObj);
|
|
|
|
expect(result.status).toBe('failed');
|
|
expect(result.error).toContain('already in progress');
|
|
expect(putSpy).not.toHaveBeenCalled(); // no legacy file writes on conflict
|
|
});
|
|
|
|
it('maps a remote apply failure to status=failed with the HTTP error', async () => {
|
|
const node = seedRemoteNode();
|
|
const bp = seedBlueprint([node.id]);
|
|
const nodeObj = DatabaseService.getInstance().getNode(node.id)!;
|
|
const bpObj = DatabaseService.getInstance().getBlueprint(bp.id)!;
|
|
|
|
vi.spyOn(axios, 'get').mockResolvedValue({ status: 200, data: [] });
|
|
vi.spyOn(axios, 'put').mockResolvedValue({ status: 200, data: {} });
|
|
vi.spyOn(axios, 'post').mockResolvedValue({ status: 500, data: { error: 'boom' } });
|
|
|
|
const result = await BlueprintService.getInstance().deployToNode(bpObj, nodeObj);
|
|
|
|
expect(result.status).toBe('failed');
|
|
expect(result.error).toContain('HTTP 500');
|
|
const dep = DatabaseService.getInstance().getDeployment(bp.id, node.id);
|
|
expect(dep?.status).toBe('failed');
|
|
expect(dep?.last_error).toContain('HTTP 500');
|
|
});
|
|
|
|
it('refuses to deploy when an unmanaged stack of the same name exists on the remote', async () => {
|
|
const node = seedRemoteNode();
|
|
const bp = seedBlueprint([node.id]);
|
|
const nodeObj = DatabaseService.getInstance().getNode(node.id)!;
|
|
const bpObj = DatabaseService.getInstance().getBlueprint(bp.id)!;
|
|
|
|
// hasNameConflict lists /api/stacks first, then reads the marker. A 404 marker on an
|
|
// existing stack means it is unmanaged, so the deploy must refuse.
|
|
vi.spyOn(axios, 'get')
|
|
.mockResolvedValueOnce({ status: 200, data: [{ name: bpObj.name }] })
|
|
.mockResolvedValueOnce({ status: 404, data: {} });
|
|
const postSpy = vi.spyOn(axios, 'post');
|
|
|
|
const result = await BlueprintService.getInstance().deployToNode(bpObj, nodeObj);
|
|
|
|
expect(result.status).toBe('name_conflict');
|
|
expect(postSpy).not.toHaveBeenCalled();
|
|
const dep = DatabaseService.getInstance().getDeployment(bp.id, node.id);
|
|
expect(dep).toBeDefined();
|
|
expect(dep?.status).toBe('name_conflict');
|
|
});
|
|
|
|
it('withdraws a remote deployment by deleting the stack and removing the row', async () => {
|
|
const node = seedRemoteNode();
|
|
const bp = seedBlueprint([node.id]);
|
|
const nodeObj = DatabaseService.getInstance().getNode(node.id)!;
|
|
const bpObj = DatabaseService.getInstance().getBlueprint(bp.id)!;
|
|
DatabaseService.getInstance().upsertDeployment({
|
|
blueprint_id: bp.id,
|
|
node_id: node.id,
|
|
status: 'active',
|
|
applied_revision: bpObj.revision,
|
|
});
|
|
|
|
vi.spyOn(axios, 'get').mockResolvedValue({ status: 404, data: {} }); // readMarker → null → proceed
|
|
vi.spyOn(axios, 'post').mockResolvedValue({ status: 200, data: {} }); // remote down (best-effort)
|
|
const delSpy = vi.spyOn(axios, 'delete').mockResolvedValue({ status: 200, data: {} });
|
|
|
|
const result = await BlueprintService.getInstance().withdrawFromNode(bpObj, nodeObj);
|
|
|
|
expect(result.status).toBe('withdrawn');
|
|
expect(delSpy.mock.calls[0][0]).toMatch(/\/api\/stacks\//);
|
|
expect(DatabaseService.getInstance().getDeployment(bp.id, node.id)).toBeUndefined();
|
|
});
|
|
});
|