mirror of
https://github.com/Studio-Saelix/sencho.git
synced 2026-08-29 11:47:01 +00:00
feat: health-gated updates and rollback readiness (#1354)
* feat: classify stack deploy and update failures with suggested next actions Failed deploy and update responses now carry a failure classification (cause category, headline, and suggested next step) derived from the compose error output. The recovery panel and chip render the classification and include it in copied diagnostics, and gateway-style failures surface as a node-unreachable cause. * feat: add update and rollback readiness reports for stacks Before a manual update, Sencho now shows an advisory readiness verdict computed from the stored preflight result, open drift findings, live container health, the pending image change, the rollback backup slot, and node disk headroom. The Stack Dossier gains a rollback readiness section that states what a rollback can restore and explicitly discloses that volume and bind-mounted data are not covered. Toolbar and sidebar updates now share one update path, and admins can create a fleet snapshot from the readiness dialog before updating. Nodes that do not advertise the capability keep the direct update flow. * feat: observe stack health after updates with a post-deploy health gate After a deploy or update succeeds, Sencho now watches the stack for a configurable observation window and records a passed, failed, or unknown verdict: containers must stay running, healthchecks must report healthy, and restart loops or disappearing containers fail the gate. The deploy panel shows the observation live and holds off auto-closing until the verdict lands, a failed gate surfaces the existing recovery actions including rollback, and the stack timeline records update started and gate verdict events. Scheduled, webhook, bulk, and git-source updates are gated the same way; rollbacks and installs are deliberately not. The gate is observational only and can be tuned or disabled per node under host alert settings. * docs: document health-gated updates and rollback readiness New operator page covering the update readiness dialog, the post-update health gate and its settings, the rollback readiness disclosure, and classified failures, with cross-links from the atomic deployments and deploy progress pages. The API reference gains the readiness and health-gate endpoints, the healthGateId success field, and the failure classification schema on deploy and update error responses. * feat: withhold the success verdict while the health gate observes An update used to show a green Succeeded that a failed health gate then contradicted moments later. The deploy modal now reports Verifying health while the gate observes, shows success only when the gate passes, and makes a failed or unknown gate the headline result; success toasts soften to a verifying message while a gate runs. The mobile recovery card groups its actions behind one bottom-right Take action menu so it stays compact on a phone, with the classified cause still visible on the card. A successful image update now also counts as the last known-good marker in rollback readiness, and the docs gain screenshots of the readiness dialog, gate states, dossier section, and settings. * fix: harden log format strings and the env existence path check Log calls that interpolated the stack name into the console format string now use constant format strings with placeholder arguments, and envExists validates path containment inline at its filesystem access, matching the established patterns used elsewhere in the same files. * test: adapt deploy modal success specs to the post-deploy health gate The deploy feedback modal now withholds its success verdict while the health gate observes the new containers, showing "Verifying health" until the gate passes. The two success-path E2E tests waited for "Succeeded" within the gate's 90s default window and timed out. Shorten the observation window to the 15s minimum for these tests via the settings API, assert the verify-then-succeed sequence the modal actually renders, and restore the default window afterward so the test value does not leak into later runs. * fix: serialize health gate polling and harden gate observation Address race conditions in the post-update health gate found in review. Backend: the gate poller used setInterval, so a Docker observe slower than the 5s tick could overlap the next poll and corrupt the restart and missing-container accounting, and a wedged socket could leave a poll pending forever. Polling is now single-flight: each cycle self-schedules the next only after it settles, and the observe is bounded by an 8s timeout so a hung probe counts as a poll error and resolves the gate unknown after three in a row. Frontend: the gate poller could overlap requests, letting a slow earlier "observing" response overwrite an already-applied terminal verdict. It is now single-flight with a terminal latch, so a late response can never roll the UI back from passed or failed. Also reject a non-digit nodeId on the snapshot coverage route instead of letting parseInt coerce it, document that turning off the deploy progress panel opts out of the live gate UI while the gate still runs server-side, and add gate-coverage tests for the webhook, git source, and auto-update apply paths plus the new single-flight, observe-timeout, and recovery cases.
This commit is contained in:
Anso
@@ -0,0 +1,154 @@
|
||||
import { describe, it, expect } from 'vitest';
|
||||
import { classifyFailure } from '../services/updateGuard/failureClassifier';
|
||||
import type { FailureReason } from '../services/updateGuard/types';
|
||||
|
||||
describe('classifyFailure', () => {
|
||||
const cases: Array<{ name: string; message: string; reason: FailureReason }> = [
|
||||
{
|
||||
name: 'container crash sentinel',
|
||||
message: 'CONTAINER_CRASHED\nExit Code: 137\nContainer exited after deployment. Check container logs for details.',
|
||||
reason: 'container_exited',
|
||||
},
|
||||
{
|
||||
name: 'idle stall sentinel',
|
||||
message: 'STACK_STALLED_OUTPUT: no output for 600s',
|
||||
reason: 'unknown',
|
||||
},
|
||||
{
|
||||
name: 'docker daemon down (message)',
|
||||
message: 'Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?',
|
||||
reason: 'node_unreachable',
|
||||
},
|
||||
{
|
||||
name: 'docker desktop connect error',
|
||||
message: 'error during connect: this error may indicate that the docker daemon is not running',
|
||||
reason: 'node_unreachable',
|
||||
},
|
||||
{
|
||||
name: 'required variable missing',
|
||||
message: 'error while interpolating services.web.environment.TOKEN: required variable REQ_TOKEN is missing a value: must be provided',
|
||||
reason: 'env_missing',
|
||||
},
|
||||
{
|
||||
name: 'variable not set (logfmt warning escalated)',
|
||||
message: 'level=warning msg="The \\"DB_HOST\\" variable is not set. Defaulting to a blank string."',
|
||||
reason: 'env_missing',
|
||||
},
|
||||
{
|
||||
name: 'env file not found',
|
||||
message: "env file /compose/app/.env not found: stat /compose/app/.env: no such file or directory",
|
||||
reason: 'env_missing',
|
||||
},
|
||||
{
|
||||
name: 'pull access denied',
|
||||
message: 'Error response from daemon: pull access denied for ghost/missing, repository does not exist or may require docker login',
|
||||
reason: 'image_pull_failed',
|
||||
},
|
||||
{
|
||||
name: 'manifest unknown',
|
||||
message: 'manifest unknown: manifest unknown',
|
||||
reason: 'image_pull_failed',
|
||||
},
|
||||
{
|
||||
name: 'registry rate limited',
|
||||
message: 'toomanyrequests: You have reached your pull rate limit.',
|
||||
reason: 'image_pull_failed',
|
||||
},
|
||||
{
|
||||
name: 'port already allocated',
|
||||
message: 'Error response from daemon: driver failed programming external connectivity: Bind for 0.0.0.0:8080 failed: port is already allocated',
|
||||
reason: 'port_conflict',
|
||||
},
|
||||
{
|
||||
name: 'address already in use',
|
||||
message: 'listen tcp4 0.0.0.0:443: bind: address already in use',
|
||||
reason: 'port_conflict',
|
||||
},
|
||||
{
|
||||
name: 'windows ports not available',
|
||||
message: 'ports are not available: exposing port TCP 0.0.0.0:5432 -> 0.0.0.0:0',
|
||||
reason: 'port_conflict',
|
||||
},
|
||||
{
|
||||
name: 'bind source missing',
|
||||
message: 'Error response from daemon: invalid mount config for type "bind": bind source path does not exist: /srv/missing',
|
||||
reason: 'bind_path_missing',
|
||||
},
|
||||
{
|
||||
name: 'permission denied',
|
||||
message: 'open /compose/app/data: permission denied',
|
||||
reason: 'permission_denied',
|
||||
},
|
||||
{
|
||||
name: 'unhealthy dependency',
|
||||
message: 'dependency failed to start: container app-db-1 is unhealthy',
|
||||
reason: 'healthcheck_failed',
|
||||
},
|
||||
{
|
||||
name: 'dependency exited',
|
||||
message: 'dependency failed to start: container app-db-1 exited (1)',
|
||||
reason: 'dependency_unavailable',
|
||||
},
|
||||
{
|
||||
name: 'external network missing',
|
||||
message: 'network proxy_net declared as external, but could not be found',
|
||||
reason: 'dependency_unavailable',
|
||||
},
|
||||
{
|
||||
name: 'yaml syntax error',
|
||||
message: 'yaml: line 14: mapping values are not allowed in this context',
|
||||
reason: 'compose_render_failed',
|
||||
},
|
||||
{
|
||||
name: 'undefined volume',
|
||||
message: 'service "web" refers to undefined volume data: invalid compose project',
|
||||
reason: 'compose_render_failed',
|
||||
},
|
||||
{
|
||||
name: 'unmatched output falls back to unknown',
|
||||
message: 'something completely unexpected happened',
|
||||
reason: 'unknown',
|
||||
},
|
||||
];
|
||||
|
||||
it.each(cases)('classifies $name as $reason', ({ message, reason }) => {
|
||||
const result = classifyFailure(message);
|
||||
expect(result.reason).toBe(reason);
|
||||
expect(result.label.length).toBeGreaterThan(0);
|
||||
expect(result.suggestion.length).toBeGreaterThan(0);
|
||||
});
|
||||
|
||||
it('short-circuits to node_unreachable when the route flags a dead daemon', () => {
|
||||
expect(classifyFailure('arbitrary text', { dockerUnavailable: true }).reason).toBe('node_unreachable');
|
||||
});
|
||||
|
||||
it('prefers env_missing over compose_render_failed when a render fails on a missing variable', () => {
|
||||
const result = classifyFailure(
|
||||
'invalid compose project: required variable DB_PASS is missing a value',
|
||||
);
|
||||
expect(result.reason).toBe('env_missing');
|
||||
});
|
||||
|
||||
it('prefers the crash sentinel over any other pattern in the same output', () => {
|
||||
const result = classifyFailure(
|
||||
'CONTAINER_CRASHED\nExit Code: 1\nport is already allocated',
|
||||
);
|
||||
expect(result.reason).toBe('container_exited');
|
||||
});
|
||||
|
||||
it('classifies the underlying cause of a rolled-back update from its message', () => {
|
||||
// ComposeRollbackError copies the original error message, so the route can
|
||||
// pass getErrorMessage(error) unchanged for wrapped failures.
|
||||
const causeMessage = 'pull access denied for private/app';
|
||||
expect(classifyFailure(causeMessage).reason).toBe('image_pull_failed');
|
||||
});
|
||||
|
||||
it('always returns a classification (total over arbitrary input)', () => {
|
||||
for (const message of ['', ' ', 'x'.repeat(10_000)]) {
|
||||
const result = classifyFailure(message);
|
||||
expect(result.reason).toBeDefined();
|
||||
expect(result.label).toBeDefined();
|
||||
expect(result.suggestion).toBeDefined();
|
||||
}
|
||||
});
|
||||
});
|
||||
@@ -1052,6 +1052,29 @@ describe('GitSourceService.apply', () => {
|
||||
.rejects.toMatchObject({ code: 'GIT_ERROR', message: expect.stringMatching(/pending commit has changed/i) });
|
||||
});
|
||||
|
||||
it('begins a deploy health gate after a successful apply-and-deploy', async () => {
|
||||
const sha = 'eeee555eeee555eeee555eeee555eeee555eeee5';
|
||||
const svc = await seedPending('apply-deploy-gate', 'services:\n x:\n image: alpine\n', sha);
|
||||
const validateSpy = vi.spyOn(svc, 'validateCompose').mockResolvedValue({ ok: true });
|
||||
const { FileSystemService } = await import('../services/FileSystemService');
|
||||
const { ComposeService } = await import('../services/ComposeService');
|
||||
const { HealthGateService } = await import('../services/HealthGateService');
|
||||
const saveSpy = vi.spyOn(FileSystemService.prototype, 'saveStackContent').mockResolvedValue();
|
||||
const deploySpy = vi.spyOn(ComposeService.prototype, 'deployStack').mockResolvedValue();
|
||||
const beginSpy = vi.spyOn(HealthGateService.getInstance(), 'begin').mockReturnValue('gate-git');
|
||||
const nodeId = DatabaseService.getInstance().getDefaultNode()!.id!;
|
||||
|
||||
const result = await svc.apply('apply-deploy-gate', sha, { deploy: true });
|
||||
expect(result.deployed).toBe(true);
|
||||
expect(deploySpy).toHaveBeenCalledWith('apply-deploy-gate');
|
||||
expect(beginSpy).toHaveBeenCalledWith(nodeId, 'apply-deploy-gate', 'deploy', 'system:git-source');
|
||||
|
||||
validateSpy.mockRestore();
|
||||
saveSpy.mockRestore();
|
||||
deploySpy.mockRestore();
|
||||
beginSpy.mockRestore();
|
||||
});
|
||||
|
||||
it('returns deployError when the deploy step fails after writing to disk', async () => {
|
||||
const sha = 'cccc333cccc333cccc333cccc333cccc333cccc3';
|
||||
const svc = await seedPending('apply-deploy-fail', 'services:\n x:\n image: alpine\n', sha);
|
||||
|
||||
@@ -0,0 +1,402 @@
|
||||
/**
|
||||
* State-machine tests for HealthGateService with fake timers and an in-memory
|
||||
* DatabaseService mock: verdicts, restart detection, supersede semantics,
|
||||
* startup sweep, the disabled setting, the concurrency cap, and timer hygiene.
|
||||
*/
|
||||
import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest';
|
||||
|
||||
interface StoredRun {
|
||||
id: string;
|
||||
node_id: number;
|
||||
stack_name: string;
|
||||
trigger_action: 'update' | 'deploy';
|
||||
status: 'observing' | 'passed' | 'failed' | 'unknown';
|
||||
reason: string | null;
|
||||
window_seconds: number;
|
||||
containers_json: string;
|
||||
started_at: number;
|
||||
ended_at: number | null;
|
||||
created_by: string | null;
|
||||
}
|
||||
|
||||
const { state } = vi.hoisted(() => ({
|
||||
state: {
|
||||
runs: new Map<string, StoredRun>(),
|
||||
activity: [] as Array<{ category?: string; message: string; level: string }>,
|
||||
settings: {} as Record<string, string>,
|
||||
listContainers: vi.fn(),
|
||||
inspect: vi.fn(),
|
||||
},
|
||||
}));
|
||||
|
||||
vi.mock('../services/DatabaseService', () => ({
|
||||
DatabaseService: {
|
||||
getInstance: () => ({
|
||||
getGlobalSettings: () => state.settings,
|
||||
insertHealthGateRun: (run: StoredRun) => { state.runs.set(run.id, { ...run }); },
|
||||
finalizeHealthGateRun: (id: string, status: StoredRun['status'], reason: string | null, endedAt: number, containersJson: string) => {
|
||||
const run = state.runs.get(id);
|
||||
if (run) Object.assign(run, { status, reason, ended_at: endedAt, containers_json: containersJson });
|
||||
},
|
||||
getHealthGateRun: (nodeId: number, stackName: string, id: string) => {
|
||||
const run = state.runs.get(id);
|
||||
return run && run.node_id === nodeId && run.stack_name === stackName ? { ...run } : undefined;
|
||||
},
|
||||
getLatestHealthGateRun: (nodeId: number, stackName: string) => {
|
||||
const matches = [...state.runs.values()]
|
||||
.filter(r => r.node_id === nodeId && r.stack_name === stackName)
|
||||
.sort((a, b) => b.started_at - a.started_at);
|
||||
return matches[0] ? { ...matches[0] } : undefined;
|
||||
},
|
||||
markInterruptedHealthGateRuns: (reason: string, endedAt: number) => {
|
||||
let n = 0;
|
||||
for (const run of state.runs.values()) {
|
||||
if (run.status === 'observing') {
|
||||
Object.assign(run, { status: 'unknown', reason, ended_at: endedAt });
|
||||
n++;
|
||||
}
|
||||
}
|
||||
return n;
|
||||
},
|
||||
addNotificationHistory: (_nodeId: number, item: { category?: string; message: string; level: string }) => {
|
||||
state.activity.push(item);
|
||||
return { ...item, id: state.activity.length, is_read: false };
|
||||
},
|
||||
}),
|
||||
},
|
||||
}));
|
||||
|
||||
vi.mock('../services/DockerController', () => ({
|
||||
default: {
|
||||
getInstance: () => ({
|
||||
getDocker: () => ({
|
||||
listContainers: state.listContainers,
|
||||
getContainer: (id: string) => ({ inspect: () => state.inspect(id) }),
|
||||
}),
|
||||
}),
|
||||
},
|
||||
}));
|
||||
|
||||
import { HealthGateService } from '../services/HealthGateService';
|
||||
|
||||
type ContainerFixture = {
|
||||
id: string;
|
||||
name: string;
|
||||
state?: string;
|
||||
health?: string | null;
|
||||
restartCount?: number;
|
||||
startedAt?: string;
|
||||
};
|
||||
|
||||
/** Configure the docker mocks from a simple fixture list. */
|
||||
function setContainers(fixtures: ContainerFixture[]): void {
|
||||
state.listContainers.mockResolvedValue(fixtures.map(f => ({ Id: f.id, Names: [`/${f.name}`], State: f.state ?? 'running' })));
|
||||
state.inspect.mockImplementation((id: string) => {
|
||||
const f = fixtures.find(c => c.id === id);
|
||||
if (!f) return Promise.reject(Object.assign(new Error('no such container'), { statusCode: 404 }));
|
||||
return Promise.resolve({
|
||||
State: {
|
||||
Status: f.state ?? 'running',
|
||||
Health: f.health !== undefined && f.health !== null ? { Status: f.health } : undefined,
|
||||
StartedAt: f.startedAt ?? '2026-06-10T00:00:00Z',
|
||||
},
|
||||
RestartCount: f.restartCount ?? 0,
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
const svc = () => HealthGateService.getInstance();
|
||||
|
||||
const latest = (stack = 'web') => svc().getReport(0, stack);
|
||||
|
||||
async function ticks(n: number): Promise<void> {
|
||||
for (let i = 0; i < n; i++) {
|
||||
await vi.advanceTimersByTimeAsync(5_000);
|
||||
}
|
||||
}
|
||||
|
||||
beforeEach(() => {
|
||||
vi.useFakeTimers();
|
||||
state.runs.clear();
|
||||
state.activity.length = 0;
|
||||
state.settings = { health_gate_enabled: '1', health_gate_window_seconds: '30' };
|
||||
state.listContainers.mockReset();
|
||||
state.inspect.mockReset();
|
||||
setContainers([{ id: 'aaa', name: 'web-app-1' }]);
|
||||
svc().start();
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
svc().stop();
|
||||
expect(vi.getTimerCount()).toBe(0);
|
||||
vi.useRealTimers();
|
||||
});
|
||||
|
||||
describe('HealthGateService verdicts', () => {
|
||||
it('passes at the window end when containers stay running', async () => {
|
||||
const id = svc().begin(0, 'web', 'update', 'tester');
|
||||
expect(id).toBeTruthy();
|
||||
await ticks(3); // 15s: still observing
|
||||
expect(latest().status).toBe('observing');
|
||||
await ticks(4); // past the 30s window
|
||||
expect(latest().status).toBe('passed');
|
||||
expect(state.activity.some(a => a.category === 'health_gate_passed')).toBe(true);
|
||||
});
|
||||
|
||||
it('fails fast when a container exits', async () => {
|
||||
svc().begin(0, 'web', 'update', 'tester');
|
||||
await ticks(1); // baseline
|
||||
setContainers([{ id: 'aaa', name: 'web-app-1', state: 'exited' }]);
|
||||
await ticks(1);
|
||||
const report = latest();
|
||||
expect(report.status).toBe('failed');
|
||||
expect(report.reason).toContain('exited');
|
||||
expect(state.activity.some(a => a.category === 'health_gate_failed')).toBe(true);
|
||||
});
|
||||
|
||||
it('fails fast when a healthcheck reports unhealthy', async () => {
|
||||
svc().begin(0, 'web', 'update', 'tester');
|
||||
await ticks(1);
|
||||
setContainers([{ id: 'aaa', name: 'web-app-1', health: 'unhealthy' }]);
|
||||
await ticks(1);
|
||||
expect(latest().status).toBe('failed');
|
||||
expect(latest().reason).toContain('unhealthy');
|
||||
});
|
||||
|
||||
it('detects a restart loop via container replacement (new id)', async () => {
|
||||
svc().begin(0, 'web', 'update', 'tester');
|
||||
await ticks(1);
|
||||
setContainers([{ id: 'bbb', name: 'web-app-1' }]);
|
||||
await ticks(1); // restart 1 observed; carried as new baseline
|
||||
setContainers([{ id: 'ccc', name: 'web-app-1' }]);
|
||||
await ticks(1); // restart 2: loop
|
||||
const report = latest();
|
||||
expect(report.status).toBe('failed');
|
||||
expect(report.reason).toContain('restart looping');
|
||||
// The persisted summary reflects the tally the verdict acted on.
|
||||
expect(report.containers).toEqual([
|
||||
expect.objectContaining({ name: 'web-app-1', restarts: 2 }),
|
||||
]);
|
||||
});
|
||||
|
||||
it('detects a restart loop via RestartCount and StartedAt movement', async () => {
|
||||
svc().begin(0, 'web', 'update', 'tester');
|
||||
await ticks(1);
|
||||
setContainers([{ id: 'aaa', name: 'web-app-1', restartCount: 1 }]);
|
||||
await ticks(1);
|
||||
setContainers([{ id: 'aaa', name: 'web-app-1', restartCount: 1, startedAt: '2026-06-10T00:05:00Z' }]);
|
||||
await ticks(1);
|
||||
const report = latest();
|
||||
expect(report.status).toBe('failed');
|
||||
expect(report.reason).toContain('restart looping');
|
||||
expect(report.containers).toEqual([
|
||||
expect.objectContaining({ name: 'web-app-1', restarts: 2 }),
|
||||
]);
|
||||
});
|
||||
|
||||
it('tolerates a one-poll disappearance but fails on two consecutive misses', async () => {
|
||||
svc().begin(0, 'web', 'update', 'tester');
|
||||
await ticks(1); // baseline
|
||||
setContainers([]); // one missed poll: tolerated
|
||||
await ticks(1);
|
||||
setContainers([{ id: 'aaa', name: 'web-app-1' }]); // back before the second miss
|
||||
await ticks(5); // through the 30s window
|
||||
expect(latest().status).toBe('passed');
|
||||
|
||||
const second = svc().begin(0, 'web', 'update', 'tester')!;
|
||||
await ticks(1);
|
||||
setContainers([]);
|
||||
await ticks(2); // two consecutive misses: disappeared
|
||||
const report = svc().getReport(0, 'web', second);
|
||||
expect(report.status).toBe('failed');
|
||||
expect(report.reason).toContain('disappeared');
|
||||
});
|
||||
|
||||
it('fails when a container is stuck restarting across consecutive polls', async () => {
|
||||
svc().begin(0, 'web', 'update', 'tester');
|
||||
await ticks(1);
|
||||
setContainers([{ id: 'aaa', name: 'web-app-1', state: 'restarting' }]);
|
||||
await ticks(2);
|
||||
expect(latest().status).toBe('failed');
|
||||
expect(latest().reason).toContain('restarting');
|
||||
});
|
||||
|
||||
it('goes unknown after three consecutive docker errors', async () => {
|
||||
svc().begin(0, 'web', 'update', 'tester');
|
||||
await ticks(1);
|
||||
state.listContainers.mockRejectedValue(new Error('socket gone'));
|
||||
await ticks(3);
|
||||
expect(latest().status).toBe('unknown');
|
||||
expect(latest().reason).toContain('unreachable');
|
||||
});
|
||||
|
||||
it('resolves unknown when every docker observe hangs', async () => {
|
||||
svc().begin(0, 'web', 'update', 'tester');
|
||||
// A wedged socket never settles. The per-observe timeout turns each poll
|
||||
// into an error, and three in a row finalize the gate unknown instead of
|
||||
// observing forever on a pending promise.
|
||||
state.listContainers.mockImplementation(() => new Promise<never>(() => {}));
|
||||
// Each cycle is the 5s interval plus the 8s observe timeout; 45s covers
|
||||
// three of them.
|
||||
await vi.advanceTimersByTimeAsync(45_000);
|
||||
expect(latest().status).toBe('unknown');
|
||||
expect(latest().reason).toContain('unreachable');
|
||||
});
|
||||
|
||||
it('recovers from a transient observe timeout instead of finalizing', async () => {
|
||||
svc().begin(0, 'web', 'update', 'tester');
|
||||
// One observe wedges and times out (a single strike), then the socket
|
||||
// recovers; the gate must keep observing, not give up at one error.
|
||||
state.listContainers.mockImplementationOnce(() => new Promise<never>(() => {}));
|
||||
// 14s covers the first cycle's 5s wait plus 8s timeout.
|
||||
await vi.advanceTimersByTimeAsync(14_000);
|
||||
expect(latest().status).toBe('observing');
|
||||
// Later polls succeed and carry the gate to a pass at the window end.
|
||||
await vi.advanceTimersByTimeAsync(50_000);
|
||||
expect(latest().status).toBe('passed');
|
||||
});
|
||||
|
||||
it('runs polls single-flight: no second observe until the first settles', async () => {
|
||||
svc().begin(0, 'web', 'update', 'tester');
|
||||
let release: (value: Array<{ Id: string; Names: string[]; State: string }>) => void = () => {};
|
||||
state.listContainers.mockImplementationOnce(() => new Promise(resolve => { release = resolve; }));
|
||||
// Advance past a second poll interval while the first observe is still
|
||||
// pending. Self-scheduling means the next poll is armed only after the
|
||||
// current cycle settles, so listContainers is entered exactly once.
|
||||
await vi.advanceTimersByTimeAsync(7_000);
|
||||
expect(state.listContainers).toHaveBeenCalledTimes(1);
|
||||
// Let the first cycle finish; the next poll then runs and observes again.
|
||||
release([{ Id: 'aaa', Names: ['/web-app-1'], State: 'running' }]);
|
||||
await vi.advanceTimersByTimeAsync(5_000);
|
||||
expect(state.listContainers.mock.calls.length).toBeGreaterThanOrEqual(2);
|
||||
});
|
||||
|
||||
it('ends unknown when a healthcheck is still starting at the window end', async () => {
|
||||
svc().begin(0, 'web', 'update', 'tester');
|
||||
setContainers([{ id: 'aaa', name: 'web-app-1', health: 'starting' }]);
|
||||
await ticks(7);
|
||||
expect(latest().status).toBe('unknown');
|
||||
expect(latest().reason).toContain('still starting');
|
||||
});
|
||||
|
||||
it('goes unknown when no containers ever appear', async () => {
|
||||
setContainers([]);
|
||||
svc().begin(0, 'web', 'update', 'tester');
|
||||
await ticks(4);
|
||||
expect(latest().status).toBe('unknown');
|
||||
expect(latest().reason).toContain('no containers');
|
||||
});
|
||||
});
|
||||
|
||||
describe('HealthGateService lifecycle', () => {
|
||||
it('never lets a poll that straddled a supersede overwrite the terminal verdict', async () => {
|
||||
// A poll is mid-await on Docker when a newer update supersedes the gate;
|
||||
// when the await resolves with healthy containers, the superseded run
|
||||
// must keep its terminal unknown verdict.
|
||||
const first = svc().begin(0, 'web', 'update', 'tester')!;
|
||||
await ticks(2); // baseline established, healthy
|
||||
|
||||
let releasePoll: (value: Array<{ Id: string; Names: string[]; State: string }>) => void = () => {};
|
||||
state.listContainers.mockImplementationOnce(
|
||||
() => new Promise(resolve => { releasePoll = resolve; }),
|
||||
);
|
||||
const straddlingPoll = vi.advanceTimersByTimeAsync(5_000); // poll now awaiting Docker
|
||||
|
||||
const second = svc().begin(0, 'web', 'update', 'tester')!;
|
||||
expect(svc().getReport(0, 'web', first).status).toBe('unknown');
|
||||
|
||||
releasePoll([{ Id: 'aaa', Names: ['/web-app-1'], State: 'running' }]);
|
||||
await straddlingPoll;
|
||||
|
||||
const superseded = svc().getReport(0, 'web', first);
|
||||
expect(superseded.status).toBe('unknown');
|
||||
expect(superseded.reason).toContain('superseded');
|
||||
|
||||
await ticks(7);
|
||||
expect(svc().getReport(0, 'web', second).status).toBe('passed');
|
||||
});
|
||||
|
||||
it('supersede finalizes the old run as unknown, clears its timer, and getRun still resolves it', async () => {
|
||||
const first = svc().begin(0, 'web', 'update', 'tester')!;
|
||||
await ticks(1);
|
||||
const timersBefore = vi.getTimerCount();
|
||||
const second = svc().begin(0, 'web', 'update', 'tester')!;
|
||||
expect(vi.getTimerCount()).toBe(timersBefore); // old interval cleared, new one added
|
||||
|
||||
const superseded = svc().getReport(0, 'web', first);
|
||||
expect(superseded.status).toBe('unknown');
|
||||
expect(superseded.reason).toContain('superseded');
|
||||
|
||||
await ticks(7);
|
||||
expect(svc().getReport(0, 'web', second).status).toBe('passed');
|
||||
// The by-id read still returns the superseded run unchanged.
|
||||
expect(svc().getReport(0, 'web', first).status).toBe('unknown');
|
||||
});
|
||||
|
||||
it('start() sweeps runs left observing by a previous process', () => {
|
||||
state.runs.set('stale', {
|
||||
id: 'stale', node_id: 0, stack_name: 'web', trigger_action: 'update', status: 'observing',
|
||||
reason: null, window_seconds: 30, containers_json: '[]', started_at: 1, ended_at: null, created_by: null,
|
||||
});
|
||||
svc().start();
|
||||
expect(state.runs.get('stale')!.status).toBe('unknown');
|
||||
expect(state.runs.get('stale')!.reason).toContain('restarted');
|
||||
});
|
||||
|
||||
it('no-ops when disabled but still records the update_started event', () => {
|
||||
state.settings.health_gate_enabled = '0';
|
||||
const id = svc().begin(0, 'web', 'update', 'tester');
|
||||
expect(id).toBeNull();
|
||||
expect(state.runs.size).toBe(0);
|
||||
expect(state.activity.some(a => a.category === 'update_started')).toBe(true);
|
||||
expect(vi.getTimerCount()).toBe(0);
|
||||
});
|
||||
|
||||
it('records update_started for update triggers but not deploy triggers', () => {
|
||||
svc().begin(0, 'web', 'deploy', 'tester');
|
||||
expect(state.activity.some(a => a.category === 'update_started')).toBe(false);
|
||||
svc().begin(0, 'web', 'update', 'tester');
|
||||
expect(state.activity.some(a => a.category === 'update_started')).toBe(true);
|
||||
});
|
||||
|
||||
it('refuses to begin before start() so shutdown cannot leak timers', () => {
|
||||
svc().stop();
|
||||
expect(svc().begin(0, 'web', 'update', 'tester')).toBeNull();
|
||||
expect(vi.getTimerCount()).toBe(0);
|
||||
svc().start();
|
||||
});
|
||||
|
||||
it('persists an immediate unknown past the concurrency cap', () => {
|
||||
for (let i = 0; i < 25; i++) {
|
||||
svc().begin(0, `stack-${i}`, 'update', 'tester');
|
||||
}
|
||||
const overCap = svc().begin(0, 'one-too-many', 'update', 'tester')!;
|
||||
const report = svc().getReport(0, 'one-too-many', overCap);
|
||||
expect(report.status).toBe('unknown');
|
||||
expect(report.reason).toContain('concurrent');
|
||||
});
|
||||
|
||||
it('clamps the configured window into its valid range and falls back on garbage', () => {
|
||||
state.settings.health_gate_window_seconds = '99999';
|
||||
const a = svc().begin(0, 'web', 'update', 'tester')!;
|
||||
expect(svc().getReport(0, 'web', a).windowSeconds).toBe(600);
|
||||
state.settings.health_gate_window_seconds = 'banana';
|
||||
const b = svc().begin(0, 'web', 'update', 'tester')!;
|
||||
expect(svc().getReport(0, 'web', b).windowSeconds).toBe(90);
|
||||
});
|
||||
|
||||
it('returns the never-run sentinel for a stack with no runs', () => {
|
||||
const report = svc().getReport(0, 'nothing-here');
|
||||
expect(report.status).toBe('never-run');
|
||||
expect(report.id).toBeNull();
|
||||
});
|
||||
|
||||
it('stop() finalizes in-flight gates as unknown with zero timers left', async () => {
|
||||
const id = svc().begin(0, 'web', 'update', 'tester')!;
|
||||
await ticks(1);
|
||||
svc().stop();
|
||||
expect(vi.getTimerCount()).toBe(0);
|
||||
expect(svc().getReport(0, 'web', id).status).toBe('unknown');
|
||||
svc().start();
|
||||
});
|
||||
});
|
||||
@@ -207,4 +207,37 @@ describe('POST /api/auto-update/execute', () => {
|
||||
expect(res.status).toBe(200);
|
||||
expect(typeof res.body.result).toBe('string');
|
||||
});
|
||||
|
||||
it('begins an update health gate after an auto-update applies', async () => {
|
||||
// Target a single stack; the route works off the running containers, so
|
||||
// stub the container probe, the update check, and the compose update, then
|
||||
// assert the gate begins for the applied stack.
|
||||
const { TEST_USERNAME } = await import('./helpers/setupTestDb');
|
||||
const DockerController = (await import('../services/DockerController')).default;
|
||||
const { ImageUpdateService } = await import('../services/ImageUpdateService');
|
||||
const { ComposeService } = await import('../services/ComposeService');
|
||||
const { HealthGateService } = await import('../services/HealthGateService');
|
||||
const nodeId = DatabaseService.getInstance().getDefaultNode()!.id!;
|
||||
|
||||
const containersSpy = vi.spyOn(DockerController.prototype, 'getContainersByStack')
|
||||
.mockResolvedValue([{ Id: 'c1', Image: 'nginx:latest' }] as never);
|
||||
const checkSpy = vi.spyOn(ImageUpdateService.getInstance(), 'checkImage')
|
||||
.mockResolvedValue({ hasUpdate: true } as never);
|
||||
const updateSpy = vi.spyOn(ComposeService.prototype, 'updateStack').mockResolvedValue();
|
||||
const beginSpy = vi.spyOn(HealthGateService.getInstance(), 'begin').mockReturnValue('gate-au');
|
||||
try {
|
||||
const res = await request(app)
|
||||
.post('/api/auto-update/execute')
|
||||
.set('Cookie', adminCookie)
|
||||
.send({ target: 'auto-upd-gate' });
|
||||
expect(res.status).toBe(200);
|
||||
expect(updateSpy).toHaveBeenCalledWith('auto-upd-gate', undefined, true);
|
||||
expect(beginSpy).toHaveBeenCalledWith(nodeId, 'auto-upd-gate', 'update', `auto-update:${TEST_USERNAME}`);
|
||||
} finally {
|
||||
containersSpy.mockRestore();
|
||||
checkSpy.mockRestore();
|
||||
updateSpy.mockRestore();
|
||||
beginSpy.mockRestore();
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
@@ -647,6 +647,32 @@ describe('SchedulerService - executeUpdate', () => {
|
||||
);
|
||||
});
|
||||
|
||||
it('begins a health gate after a scheduled update succeeds', async () => {
|
||||
const { HealthGateService } = await import('../services/HealthGateService');
|
||||
const beginSpy = vi.spyOn(HealthGateService.getInstance(), 'begin').mockReturnValue('gate-1');
|
||||
try {
|
||||
mockGetScheduledTask.mockReturnValue({
|
||||
id: 83,
|
||||
name: 'update-gated',
|
||||
action: 'update',
|
||||
cron_expression: '0 4 * * *',
|
||||
enabled: true,
|
||||
target_id: 'web-app',
|
||||
node_id: 1,
|
||||
created_by: 'admin',
|
||||
last_status: null,
|
||||
});
|
||||
mockGetContainersByStack.mockResolvedValue([{ Id: 'c1', Image: 'nginx:latest' }]);
|
||||
mockCheckImage.mockResolvedValue({ hasUpdate: true });
|
||||
|
||||
await SchedulerService.getInstance().triggerTask(83);
|
||||
|
||||
expect(beginSpy).toHaveBeenCalledWith(1, 'web-app', 'update', 'system:scheduler');
|
||||
} finally {
|
||||
beginSpy.mockRestore();
|
||||
}
|
||||
});
|
||||
|
||||
it('skips when all images up to date', async () => {
|
||||
mockGetScheduledTask.mockReturnValue({
|
||||
id: 81,
|
||||
|
||||
@@ -236,6 +236,63 @@ describe('prune_on_update (auto-prune after updates)', () => {
|
||||
});
|
||||
});
|
||||
|
||||
describe('health gate settings', () => {
|
||||
it('seeds enabled with a 90 second window in a fresh database', () => {
|
||||
const settings = DatabaseService.getInstance().getGlobalSettings();
|
||||
expect(settings.health_gate_enabled).toBe('1');
|
||||
expect(settings.health_gate_window_seconds).toBe('90');
|
||||
});
|
||||
|
||||
it('is exposed through the settings GET projection', async () => {
|
||||
const res = await request(app).get('/api/settings').set('Cookie', adminCookie);
|
||||
expect(res.status).toBe(200);
|
||||
expect(res.body.health_gate_enabled).toBeDefined();
|
||||
expect(res.body.health_gate_window_seconds).toBeDefined();
|
||||
});
|
||||
|
||||
it('accepts a single-key toggle write and persists it', async () => {
|
||||
const res = await request(app)
|
||||
.post('/api/settings')
|
||||
.set('Cookie', adminCookie)
|
||||
.send({ key: 'health_gate_enabled', value: '0' });
|
||||
expect(res.status).toBe(200);
|
||||
expect(DatabaseService.getInstance().getGlobalSettings().health_gate_enabled).toBe('0');
|
||||
DatabaseService.getInstance().updateGlobalSetting('health_gate_enabled', '1');
|
||||
});
|
||||
|
||||
it('accepts an in-range window via bulk PATCH alongside another key', async () => {
|
||||
const res = await request(app)
|
||||
.patch('/api/settings')
|
||||
.set('Cookie', adminCookie)
|
||||
.send({ health_gate_window_seconds: 120, health_gate_enabled: '1' });
|
||||
expect(res.status).toBe(200);
|
||||
expect(DatabaseService.getInstance().getGlobalSettings().health_gate_window_seconds).toBe('120');
|
||||
DatabaseService.getInstance().updateGlobalSetting('health_gate_window_seconds', '90');
|
||||
});
|
||||
|
||||
it('rejects out-of-range windows and non-enum toggles', async () => {
|
||||
const tooShort = await request(app)
|
||||
.post('/api/settings')
|
||||
.set('Cookie', adminCookie)
|
||||
.send({ key: 'health_gate_window_seconds', value: '5' });
|
||||
expect(tooShort.status).toBe(400);
|
||||
|
||||
const tooLong = await request(app)
|
||||
.post('/api/settings')
|
||||
.set('Cookie', adminCookie)
|
||||
.send({ key: 'health_gate_window_seconds', value: '9000' });
|
||||
expect(tooLong.status).toBe(400);
|
||||
|
||||
const badToggle = await request(app)
|
||||
.post('/api/settings')
|
||||
.set('Cookie', adminCookie)
|
||||
.send({ key: 'health_gate_enabled', value: 'yes' });
|
||||
expect(badToggle.status).toBe(400);
|
||||
|
||||
expect(DatabaseService.getInstance().getGlobalSettings().health_gate_window_seconds).toBe('90');
|
||||
});
|
||||
});
|
||||
|
||||
describe('PATCH /api/settings (bulk update)', () => {
|
||||
it('rejects unauthenticated requests with 401', async () => {
|
||||
const res = await request(app).patch('/api/settings').send({ host_cpu_limit: 50 });
|
||||
|
||||
@@ -7,7 +7,7 @@
|
||||
* ComposeService and DockerController are mocked so no real Docker daemon is
|
||||
* required. NotificationService.dispatchAlert is spied on to assert dispatch.
|
||||
*/
|
||||
import { describe, it, expect, beforeAll, afterAll, vi, beforeEach } from 'vitest';
|
||||
import { describe, it, expect, beforeAll, afterAll, vi, beforeEach, afterEach } from 'vitest';
|
||||
import request from 'supertest';
|
||||
import jwt from 'jsonwebtoken';
|
||||
import { setupTestDb, cleanupTestDb, loginAsTestAdmin, TEST_JWT_SECRET } from './helpers/setupTestDb';
|
||||
@@ -242,6 +242,134 @@ describe('deploy_failure notification on /deploy error', () => {
|
||||
});
|
||||
});
|
||||
|
||||
describe('health gate begin call sites', () => {
|
||||
let beginSpy: ReturnType<typeof vi.spyOn>;
|
||||
|
||||
beforeEach(async () => {
|
||||
const { HealthGateService } = await import('../services/HealthGateService');
|
||||
beginSpy = vi.spyOn(HealthGateService.getInstance(), 'begin').mockReturnValue('gate-123') as ReturnType<typeof vi.spyOn>;
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
beginSpy.mockRestore();
|
||||
});
|
||||
|
||||
it('begins a gate after a manual deploy and returns its id', async () => {
|
||||
mockDeployStack.mockResolvedValue(undefined);
|
||||
const res = await request(app)
|
||||
.post('/api/stacks/myapp/deploy')
|
||||
.set('Cookie', authCookie)
|
||||
.send({ skip_scan: true });
|
||||
expect(res.status).toBe(200);
|
||||
expect(beginSpy).toHaveBeenCalledWith(expect.any(Number), 'myapp', 'deploy', 'testadmin');
|
||||
expect(res.body.healthGateId).toBe('gate-123');
|
||||
});
|
||||
|
||||
it('begins a gate after a manual update and returns its id', async () => {
|
||||
mockUpdateStack.mockResolvedValue(undefined);
|
||||
const res = await request(app)
|
||||
.post('/api/stacks/myapp/update')
|
||||
.set('Cookie', authCookie)
|
||||
.send({ skip_scan: true });
|
||||
expect(res.status).toBe(200);
|
||||
expect(beginSpy).toHaveBeenCalledWith(expect.any(Number), 'myapp', 'update', 'testadmin');
|
||||
expect(res.body.healthGateId).toBe('gate-123');
|
||||
});
|
||||
|
||||
it('begins a gate per stack in a bulk update and carries ids in the results', async () => {
|
||||
mockUpdateStack.mockResolvedValue(undefined);
|
||||
const res = await request(app)
|
||||
.post('/api/stacks/bulk')
|
||||
.set('Cookie', authCookie)
|
||||
.send({ action: 'update', stackNames: ['myapp', 'webapp'] });
|
||||
expect(res.status).toBe(200);
|
||||
expect(beginSpy).toHaveBeenCalledWith(expect.any(Number), 'myapp', 'update', 'testadmin');
|
||||
expect(beginSpy).toHaveBeenCalledWith(expect.any(Number), 'webapp', 'update', 'testadmin');
|
||||
const items = res.body.results as Array<{ stackName: string; ok: boolean; healthGateId?: string | null }>;
|
||||
expect(items).toHaveLength(2);
|
||||
for (const item of items) {
|
||||
expect(item.ok).toBe(true);
|
||||
expect(item.healthGateId).toBe('gate-123');
|
||||
}
|
||||
});
|
||||
|
||||
it('does not begin a gate on a failed deploy', async () => {
|
||||
mockDeployStack.mockRejectedValue(new Error('boom'));
|
||||
await request(app).post('/api/stacks/myapp/deploy').set('Cookie', authCookie);
|
||||
expect(beginSpy).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('never begins a gate for the rollback recovery path', async () => {
|
||||
mockDeployStack.mockResolvedValue(undefined);
|
||||
const res = await request(app)
|
||||
.post('/api/stacks/myapp/rollback')
|
||||
.set('Cookie', authCookie);
|
||||
expect(res.status).toBe(200);
|
||||
expect(beginSpy).not.toHaveBeenCalled();
|
||||
});
|
||||
});
|
||||
|
||||
describe('failure classification on deploy/update error responses', () => {
|
||||
it('classifies a failed deploy and includes failure in the body', async () => {
|
||||
mockDeployStack.mockRejectedValue(
|
||||
new Error('Bind for 0.0.0.0:8080 failed: port is already allocated'),
|
||||
);
|
||||
|
||||
const res = await request(app)
|
||||
.post('/api/stacks/myapp/deploy')
|
||||
.set('Cookie', authCookie);
|
||||
|
||||
expect(res.status).toBe(500);
|
||||
expect(res.body.failure).toMatchObject({ reason: 'port_conflict' });
|
||||
expect(typeof res.body.failure.label).toBe('string');
|
||||
expect(typeof res.body.failure.suggestion).toBe('string');
|
||||
});
|
||||
|
||||
it('classifies a failed update and includes failure in the body', async () => {
|
||||
mockUpdateStack.mockRejectedValue(
|
||||
new Error('pull access denied for private/app, repository does not exist'),
|
||||
);
|
||||
|
||||
const res = await request(app)
|
||||
.post('/api/stacks/myapp/update')
|
||||
.set('Cookie', authCookie);
|
||||
|
||||
expect(res.status).toBe(500);
|
||||
expect(res.body.failure).toMatchObject({ reason: 'image_pull_failed' });
|
||||
});
|
||||
|
||||
it('classifies the underlying cause when the update was rolled back', async () => {
|
||||
mockUpdateStack.mockRejectedValue(
|
||||
new ComposeRollbackError(
|
||||
new Error('dependency failed to start: container app-db-1 is unhealthy'),
|
||||
true,
|
||||
true,
|
||||
),
|
||||
);
|
||||
|
||||
const res = await request(app)
|
||||
.post('/api/stacks/myapp/update')
|
||||
.set('Cookie', authCookie);
|
||||
|
||||
expect(res.status).toBe(500);
|
||||
expect(res.body).toMatchObject({
|
||||
rolledBack: true,
|
||||
failure: { reason: 'healthcheck_failed' },
|
||||
});
|
||||
});
|
||||
|
||||
it('falls back to unknown for unrecognized failures', async () => {
|
||||
mockDeployStack.mockRejectedValue(new Error('weird one-off explosion'));
|
||||
|
||||
const res = await request(app)
|
||||
.post('/api/stacks/myapp/deploy')
|
||||
.set('Cookie', authCookie);
|
||||
|
||||
expect(res.status).toBe(500);
|
||||
expect(res.body.failure.reason).toBe('unknown');
|
||||
});
|
||||
});
|
||||
|
||||
describe('post-deploy scan opt-out', () => {
|
||||
it('does not trigger a post-deploy scan when skip_scan is true', async () => {
|
||||
mockDeployStack.mockResolvedValue(undefined);
|
||||
|
||||
@@ -0,0 +1,191 @@
|
||||
import { describe, it, expect } from 'vitest';
|
||||
import {
|
||||
aggregateVerdict,
|
||||
backupSlotSignal,
|
||||
containersSignal,
|
||||
diskSignal,
|
||||
driftSignal,
|
||||
healthchecksSignal,
|
||||
preflightSignal,
|
||||
updatePreviewSignal,
|
||||
} from '../services/updateGuard/readiness';
|
||||
import type { ContainerProbe, ReadinessSignal } from '../services/updateGuard/types';
|
||||
import type { UpdatePreviewSummary } from '../services/UpdatePreviewService';
|
||||
|
||||
const NOW = 1_750_000_000_000;
|
||||
|
||||
const probe = (over: Partial<ContainerProbe> = {}): ContainerProbe => ({
|
||||
name: 'app-web-1',
|
||||
state: 'running',
|
||||
health: null,
|
||||
exitCode: null,
|
||||
hasHealthcheck: false,
|
||||
restartPolicy: 'unless-stopped',
|
||||
mounts: [],
|
||||
...over,
|
||||
});
|
||||
|
||||
const summary = (over: Partial<UpdatePreviewSummary> = {}): UpdatePreviewSummary => ({
|
||||
has_update: false,
|
||||
primary_image: null,
|
||||
current_tag: null,
|
||||
next_tag: null,
|
||||
semver_bump: 'none',
|
||||
update_kind: 'none',
|
||||
blocked: false,
|
||||
blocked_reason: null,
|
||||
...over,
|
||||
});
|
||||
|
||||
describe('preflightSignal', () => {
|
||||
const cases = [
|
||||
{ status: 'never-run', expected: 'unknown', affects: false },
|
||||
{ status: 'blocker', expected: 'blocked', affects: true },
|
||||
{ status: 'unrenderable', expected: 'attention', affects: true },
|
||||
{ status: 'high', expected: 'attention', affects: true },
|
||||
{ status: 'warning', expected: 'warning', affects: true },
|
||||
{ status: 'pass', expected: 'ok', affects: true },
|
||||
{ status: 'info', expected: 'ok', affects: true },
|
||||
] as const;
|
||||
|
||||
it.each(cases)('maps preflight status $status to $expected', ({ status, expected, affects }) => {
|
||||
const signal = preflightSignal({ status });
|
||||
expect(signal.status).toBe(expected);
|
||||
expect(signal.affectsVerdict).toBe(affects);
|
||||
});
|
||||
|
||||
it('degrades a read failure to a non-verdict-affecting unknown', () => {
|
||||
const signal = preflightSignal('error');
|
||||
expect(signal.status).toBe('unknown');
|
||||
expect(signal.affectsVerdict).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe('driftSignal', () => {
|
||||
it('warns on open findings and is ok at zero', () => {
|
||||
expect(driftSignal(2).status).toBe('warning');
|
||||
expect(driftSignal(0).status).toBe('ok');
|
||||
expect(driftSignal('error')).toMatchObject({ status: 'unknown', affectsVerdict: false });
|
||||
});
|
||||
});
|
||||
|
||||
describe('containersSignal', () => {
|
||||
it('is ok when all containers run normally', () => {
|
||||
expect(containersSignal([probe(), probe({ name: 'app-db-1' })]).status).toBe('ok');
|
||||
});
|
||||
|
||||
it('warns when the stack is not running', () => {
|
||||
const signal = containersSignal([]);
|
||||
expect(signal.status).toBe('warning');
|
||||
expect(signal.detail).toContain('not running');
|
||||
});
|
||||
|
||||
it('flags unhealthy, restarting, and crashed containers for review', () => {
|
||||
expect(containersSignal([probe({ health: 'unhealthy' })]).status).toBe('attention');
|
||||
expect(containersSignal([probe({ state: 'restarting' })]).status).toBe('attention');
|
||||
expect(containersSignal([probe({ state: 'exited', exitCode: 1 })]).status).toBe('attention');
|
||||
});
|
||||
|
||||
it('does not flag a cleanly exited container', () => {
|
||||
expect(containersSignal([probe(), probe({ state: 'exited', exitCode: 0 })]).status).toBe('ok');
|
||||
});
|
||||
|
||||
it('treats a docker error as a verdict-affecting unknown', () => {
|
||||
expect(containersSignal('error')).toMatchObject({ status: 'unknown', affectsVerdict: true });
|
||||
});
|
||||
});
|
||||
|
||||
describe('healthchecksSignal', () => {
|
||||
it('is informational and never affects the verdict', () => {
|
||||
for (const input of [[probe()], [probe({ hasHealthcheck: true })], [], 'error'] as const) {
|
||||
const signal = healthchecksSignal(input as ContainerProbe[] | 'error');
|
||||
expect(signal.status).toBe('ok');
|
||||
expect(signal.affectsVerdict).toBe(false);
|
||||
}
|
||||
});
|
||||
|
||||
it('states coverage and missing restart policies', () => {
|
||||
const signal = healthchecksSignal([
|
||||
probe({ hasHealthcheck: true }),
|
||||
probe({ name: 'app-db-1', restartPolicy: null }),
|
||||
]);
|
||||
expect(signal.detail).toContain('1 of 2');
|
||||
expect(signal.detail).toContain('no restart policy');
|
||||
});
|
||||
});
|
||||
|
||||
describe('updatePreviewSignal', () => {
|
||||
it('reflects a policy block as blocked', () => {
|
||||
const signal = updatePreviewSignal(summary({ blocked: true, blocked_reason: 'Policy "prod" blocks critical CVEs' }));
|
||||
expect(signal.status).toBe('blocked');
|
||||
expect(signal.detail).toContain('prod');
|
||||
});
|
||||
|
||||
it('flags a major bump for review', () => {
|
||||
expect(updatePreviewSignal(summary({ has_update: true, semver_bump: 'major', current_tag: '1.9.0', next_tag: '2.0.0' })).status).toBe('attention');
|
||||
});
|
||||
|
||||
it('warns on an unclassifiable pending update', () => {
|
||||
expect(updatePreviewSignal(summary({ has_update: true, semver_bump: 'unknown' })).status).toBe('warning');
|
||||
});
|
||||
|
||||
it('is ok for patch and digest updates and for no update', () => {
|
||||
expect(updatePreviewSignal(summary({ has_update: true, semver_bump: 'patch', update_kind: 'tag' })).status).toBe('ok');
|
||||
expect(updatePreviewSignal(summary({ has_update: true, update_kind: 'digest' })).status).toBe('ok');
|
||||
expect(updatePreviewSignal(summary()).status).toBe('ok');
|
||||
});
|
||||
|
||||
it('degrades a preview failure to a non-verdict-affecting unknown', () => {
|
||||
expect(updatePreviewSignal('error')).toMatchObject({ status: 'unknown', affectsVerdict: false });
|
||||
});
|
||||
});
|
||||
|
||||
describe('backupSlotSignal', () => {
|
||||
it('is ok with an existing backup and warns without one', () => {
|
||||
expect(backupSlotSignal({ exists: true, timestamp: NOW - 60_000 }, NOW).status).toBe('ok');
|
||||
expect(backupSlotSignal({ exists: false, timestamp: null }, NOW).status).toBe('warning');
|
||||
expect(backupSlotSignal('error', NOW)).toMatchObject({ status: 'unknown', affectsVerdict: false });
|
||||
});
|
||||
});
|
||||
|
||||
describe('diskSignal', () => {
|
||||
it('grades disk pressure against the alert threshold', () => {
|
||||
expect(diskSignal({ usePercent: 50, limitPercent: 90 }).status).toBe('ok');
|
||||
expect(diskSignal({ usePercent: 86, limitPercent: 90 }).status).toBe('warning');
|
||||
expect(diskSignal({ usePercent: 90, limitPercent: 90 }).status).toBe('attention');
|
||||
expect(diskSignal({ usePercent: 97, limitPercent: 90 }).status).toBe('attention');
|
||||
expect(diskSignal(null)).toMatchObject({ status: 'unknown', affectsVerdict: false });
|
||||
expect(diskSignal('error')).toMatchObject({ status: 'unknown', affectsVerdict: false });
|
||||
});
|
||||
});
|
||||
|
||||
describe('aggregateVerdict', () => {
|
||||
const signal = (status: ReadinessSignal['status'], affectsVerdict = true): ReadinessSignal => ({
|
||||
id: 'drift',
|
||||
status,
|
||||
title: 't',
|
||||
detail: 'd',
|
||||
affectsVerdict,
|
||||
});
|
||||
|
||||
it('orders blocked > attention > unknown > warning > ready', () => {
|
||||
expect(aggregateVerdict([signal('ok'), signal('blocked'), signal('attention'), signal('unknown'), signal('warning')])).toBe('blocked');
|
||||
expect(aggregateVerdict([signal('ok'), signal('attention'), signal('unknown'), signal('warning')])).toBe('review_required');
|
||||
expect(aggregateVerdict([signal('ok'), signal('unknown'), signal('warning')])).toBe('unknown');
|
||||
expect(aggregateVerdict([signal('ok'), signal('warning')])).toBe('ready_with_warnings');
|
||||
expect(aggregateVerdict([signal('ok'), signal('ok')])).toBe('ready');
|
||||
});
|
||||
|
||||
it('ignores informational unknowns', () => {
|
||||
expect(aggregateVerdict([signal('ok'), signal('unknown', false)])).toBe('ready');
|
||||
expect(aggregateVerdict([signal('warning'), signal('unknown', false)])).toBe('ready_with_warnings');
|
||||
});
|
||||
|
||||
it('reaches every verdict from realistic signal sets', () => {
|
||||
expect(aggregateVerdict([preflightSignal({ status: 'blocker' }), driftSignal(0)])).toBe('blocked');
|
||||
expect(aggregateVerdict([preflightSignal({ status: 'high' }), driftSignal(0)])).toBe('review_required');
|
||||
expect(aggregateVerdict([containersSignal('error'), driftSignal(0)])).toBe('unknown');
|
||||
expect(aggregateVerdict([driftSignal(1), preflightSignal({ status: 'pass' })])).toBe('ready_with_warnings');
|
||||
expect(aggregateVerdict([driftSignal(0), preflightSignal({ status: 'pass' }), healthchecksSignal([probe()])])).toBe('ready');
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,219 @@
|
||||
import { describe, it, expect, beforeEach, afterEach, vi } from 'vitest';
|
||||
import fs from 'fs';
|
||||
import os from 'os';
|
||||
import path from 'path';
|
||||
|
||||
// Mutable state the mocked NodeRegistry reads (same harness as
|
||||
// filesystem-backup.test.ts) so FileSystemService can be constructed against
|
||||
// temp directories.
|
||||
const mockState = { composeDir: '' };
|
||||
|
||||
vi.mock('../services/NodeRegistry', () => ({
|
||||
NodeRegistry: {
|
||||
getInstance: () => ({
|
||||
getComposeDir: () => mockState.composeDir,
|
||||
getDefaultNodeId: () => 1,
|
||||
}),
|
||||
},
|
||||
}));
|
||||
|
||||
import {
|
||||
aggregateRollbackOverall,
|
||||
buildRollbackItems,
|
||||
type RollbackInputs,
|
||||
} from '../services/updateGuard/readiness';
|
||||
import type { ContainerProbe } from '../services/updateGuard/types';
|
||||
|
||||
const NOW = 1_750_000_000_000;
|
||||
|
||||
const baseInputs = (over: Partial<RollbackInputs> = {}): RollbackInputs => ({
|
||||
backup: { exists: true, timestamp: NOW - 3_600_000 },
|
||||
envSummary: { exists: true, envPresent: true, keys: ['DB_HOST', 'DB_PASS'] },
|
||||
stackHasEnv: true,
|
||||
rollbackTarget: { target: 'nginx:1.27.1' },
|
||||
lastDeployAt: NOW - 3_600_000,
|
||||
containers: [{
|
||||
name: 'app-web-1', state: 'running', health: 'healthy', exitCode: null,
|
||||
hasHealthcheck: true, restartPolicy: 'unless-stopped', mounts: ['volume app_data'],
|
||||
}],
|
||||
...over,
|
||||
});
|
||||
|
||||
const itemById = (inputs: RollbackInputs, id: string) =>
|
||||
buildRollbackItems(inputs, NOW).find(i => i.id === id)!;
|
||||
|
||||
describe('buildRollbackItems', () => {
|
||||
it('reports a full set of six items', () => {
|
||||
const items = buildRollbackItems(baseInputs(), NOW);
|
||||
expect(items.map(i => i.id)).toEqual([
|
||||
'compose_source', 'env_keys', 'previous_images', 'last_deploy', 'healthchecks', 'volume_data',
|
||||
]);
|
||||
});
|
||||
|
||||
it('marks the volume row not_covered unconditionally and names the mounts', () => {
|
||||
const item = itemById(baseInputs(), 'volume_data');
|
||||
expect(item.state).toBe('not_covered');
|
||||
expect(item.detail).toContain('not included in file backups');
|
||||
expect(item.detail).toContain('volume app_data');
|
||||
|
||||
const noMounts = itemById(baseInputs({ containers: [] }), 'volume_data');
|
||||
expect(noMounts.state).toBe('not_covered');
|
||||
|
||||
const dockerDown = itemById(baseInputs({ containers: 'error' }), 'volume_data');
|
||||
expect(dockerDown.state).toBe('not_covered');
|
||||
});
|
||||
|
||||
it('exposes env coverage as names only', () => {
|
||||
const item = itemById(baseInputs(), 'env_keys');
|
||||
expect(item.state).toBe('ready');
|
||||
expect(item.detail).toContain('2 variable names');
|
||||
expect(item.detail).not.toContain('DB_HOST');
|
||||
expect(item.detail).not.toContain('DB_PASS');
|
||||
});
|
||||
|
||||
it('treats a stack without an env file as covered', () => {
|
||||
const item = itemById(baseInputs({
|
||||
envSummary: { exists: true, envPresent: false, keys: [] },
|
||||
stackHasEnv: false,
|
||||
}), 'env_keys');
|
||||
expect(item.state).toBe('ready');
|
||||
expect(item.detail).toContain('no env file');
|
||||
});
|
||||
|
||||
it('flags an env file the backup predates', () => {
|
||||
const item = itemById(baseInputs({
|
||||
envSummary: { exists: true, envPresent: false, keys: [] },
|
||||
stackHasEnv: true,
|
||||
}), 'env_keys');
|
||||
expect(item.state).toBe('missing');
|
||||
});
|
||||
|
||||
it('marks the previous image unknown when no rollback target is known', () => {
|
||||
expect(itemById(baseInputs({ rollbackTarget: { target: null } }), 'previous_images').state).toBe('unknown');
|
||||
expect(itemById(baseInputs({ rollbackTarget: 'error' }), 'previous_images').state).toBe('unknown');
|
||||
const known = itemById(baseInputs(), 'previous_images');
|
||||
expect(known.state).toBe('ready');
|
||||
expect(known.detail).toContain('nginx:1.27.1');
|
||||
});
|
||||
|
||||
it('does not mistake an image literally named error for a failed preview', () => {
|
||||
const item = itemById(baseInputs({ rollbackTarget: { target: 'error' } }), 'previous_images');
|
||||
expect(item.state).toBe('ready');
|
||||
expect(item.detail).toContain('error');
|
||||
});
|
||||
|
||||
it('reports last deploy and healthcheck coverage', () => {
|
||||
expect(itemById(baseInputs(), 'last_deploy').state).toBe('ready');
|
||||
expect(itemById(baseInputs({ lastDeployAt: null }), 'last_deploy').state).toBe('missing');
|
||||
expect(itemById(baseInputs(), 'healthchecks').state).toBe('ready');
|
||||
const none: ContainerProbe[] = [{
|
||||
name: 'a', state: 'running', health: null, exitCode: null,
|
||||
hasHealthcheck: false, restartPolicy: null, mounts: [],
|
||||
}];
|
||||
expect(itemById(baseInputs({ containers: none }), 'healthchecks').state).toBe('missing');
|
||||
});
|
||||
});
|
||||
|
||||
describe('aggregateRollbackOverall', () => {
|
||||
it('is ready when compose, env, and previous image are all covered', () => {
|
||||
expect(aggregateRollbackOverall(buildRollbackItems(baseInputs(), NOW))).toBe('ready');
|
||||
});
|
||||
|
||||
it('is not_ready without a backup slot', () => {
|
||||
const items = buildRollbackItems(baseInputs({
|
||||
backup: { exists: false, timestamp: null },
|
||||
envSummary: { exists: false, envPresent: false, keys: [] },
|
||||
}), NOW);
|
||||
expect(aggregateRollbackOverall(items)).toBe('not_ready');
|
||||
});
|
||||
|
||||
it('is partial when the previous image tag is unknown', () => {
|
||||
const items = buildRollbackItems(baseInputs({ rollbackTarget: { target: null } }), NOW);
|
||||
expect(aggregateRollbackOverall(items)).toBe('partial');
|
||||
});
|
||||
|
||||
it('is partial when env coverage is missing', () => {
|
||||
const items = buildRollbackItems(baseInputs({
|
||||
envSummary: { exists: true, envPresent: false, keys: [] },
|
||||
stackHasEnv: true,
|
||||
}), NOW);
|
||||
expect(aggregateRollbackOverall(items)).toBe('partial');
|
||||
});
|
||||
|
||||
it('never gates on the volume or healthcheck disclosures', () => {
|
||||
const items = buildRollbackItems(baseInputs({ containers: 'error' }), NOW);
|
||||
expect(aggregateRollbackOverall(items)).toBe('ready');
|
||||
});
|
||||
});
|
||||
|
||||
describe('FileSystemService.getBackupEnvSummary', () => {
|
||||
let tmpDir: string;
|
||||
let composeDir: string;
|
||||
let originalDataDir: string | undefined;
|
||||
|
||||
beforeEach(() => {
|
||||
tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), 'sencho-envsum-'));
|
||||
composeDir = fs.mkdtempSync(path.join(os.tmpdir(), 'sencho-envsum-compose-'));
|
||||
mockState.composeDir = composeDir;
|
||||
originalDataDir = process.env.DATA_DIR;
|
||||
process.env.DATA_DIR = tmpDir;
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
if (originalDataDir === undefined) delete process.env.DATA_DIR;
|
||||
else process.env.DATA_DIR = originalDataDir;
|
||||
fs.rmSync(tmpDir, { recursive: true, force: true });
|
||||
fs.rmSync(composeDir, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
async function getService() {
|
||||
const { FileSystemService } = await import('../services/FileSystemService');
|
||||
return FileSystemService.getInstance();
|
||||
}
|
||||
|
||||
function writeBackupEnv(stackName: string, content: string | null) {
|
||||
const dir = path.join(tmpDir, 'backups', '1', stackName);
|
||||
fs.mkdirSync(dir, { recursive: true });
|
||||
if (content !== null) fs.writeFileSync(path.join(dir, '.env'), content, 'utf-8');
|
||||
}
|
||||
|
||||
it('returns key names only, never values', async () => {
|
||||
writeBackupEnv('web', 'DB_HOST=db.internal\nDB_PASS=s3cret-value\n# comment\nEMPTY=\n INDENTED=ok\nnot a var line\n');
|
||||
const svc = await getService();
|
||||
const summary = await svc.getBackupEnvSummary('web');
|
||||
expect(summary).toEqual({
|
||||
exists: true,
|
||||
envPresent: true,
|
||||
keys: ['DB_HOST', 'DB_PASS', 'EMPTY', 'INDENTED'],
|
||||
});
|
||||
expect(JSON.stringify(summary)).not.toContain('s3cret-value');
|
||||
expect(JSON.stringify(summary)).not.toContain('db.internal');
|
||||
});
|
||||
|
||||
it('reports a backup without an env file', async () => {
|
||||
writeBackupEnv('web', null);
|
||||
const svc = await getService();
|
||||
expect(await svc.getBackupEnvSummary('web')).toEqual({ exists: true, envPresent: false, keys: [] });
|
||||
});
|
||||
|
||||
it('reports a missing backup slot', async () => {
|
||||
const svc = await getService();
|
||||
expect(await svc.getBackupEnvSummary('web')).toEqual({ exists: false, envPresent: false, keys: [] });
|
||||
});
|
||||
|
||||
it('rejects traversal-shaped stack names without touching the filesystem', async () => {
|
||||
const svc = await getService();
|
||||
expect(await svc.getBackupEnvSummary('../../etc')).toEqual({ exists: false, envPresent: false, keys: [] });
|
||||
expect(await svc.getBackupEnvSummary('..')).toEqual({ exists: false, envPresent: false, keys: [] });
|
||||
});
|
||||
|
||||
it('propagates a non-ENOENT env read failure instead of reporting "no env in backup"', async () => {
|
||||
// An unreadable .env (here: a directory, EISDIR) must throw so callers
|
||||
// degrade the item to unknown rather than falsely claiming the backup
|
||||
// contains no env file.
|
||||
const dir = path.join(tmpDir, 'backups', '1', 'web');
|
||||
fs.mkdirSync(path.join(dir, '.env'), { recursive: true });
|
||||
const svc = await getService();
|
||||
await expect(svc.getBackupEnvSummary('web')).rejects.toThrow();
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,223 @@
|
||||
/**
|
||||
* Route-level tests for the update guard endpoints: the readiness GETs on the
|
||||
* stacks router and the fleet snapshot coverage lookup. UpdateGuardService and
|
||||
* FileSystemService are mocked; the focus is auth, validation, route
|
||||
* placement, and response shape.
|
||||
*/
|
||||
import { describe, it, expect, beforeAll, afterAll, vi, beforeEach } from 'vitest';
|
||||
import request from 'supertest';
|
||||
import bcrypt from 'bcrypt';
|
||||
import { setupTestDb, cleanupTestDb, loginAsTestAdmin } from './helpers/setupTestDb';
|
||||
|
||||
const {
|
||||
mockComputeUpdateReadiness,
|
||||
mockComputeRollbackReadiness,
|
||||
} = vi.hoisted(() => ({
|
||||
mockComputeUpdateReadiness: vi.fn(),
|
||||
mockComputeRollbackReadiness: vi.fn(),
|
||||
}));
|
||||
|
||||
vi.mock('../services/UpdateGuardService', () => ({
|
||||
UpdateGuardService: {
|
||||
getInstance: () => ({
|
||||
computeUpdateReadiness: mockComputeUpdateReadiness,
|
||||
computeRollbackReadiness: mockComputeRollbackReadiness,
|
||||
}),
|
||||
},
|
||||
}));
|
||||
|
||||
vi.mock('../services/FileSystemService', () => ({
|
||||
FileSystemService: {
|
||||
getInstance: () => ({
|
||||
getStacks: vi.fn().mockResolvedValue([]),
|
||||
getBaseDir: () => '/tmp/compose',
|
||||
hasComposeFile: vi.fn().mockResolvedValue(true),
|
||||
}),
|
||||
},
|
||||
}));
|
||||
|
||||
let tmpDir: string;
|
||||
let app: import('express').Express;
|
||||
let authCookie: string;
|
||||
let viewerCookie: string;
|
||||
|
||||
beforeAll(async () => {
|
||||
tmpDir = await setupTestDb();
|
||||
({ app } = await import('../index'));
|
||||
authCookie = await loginAsTestAdmin(app);
|
||||
|
||||
const { DatabaseService } = await import('../services/DatabaseService');
|
||||
const viewerHash = await bcrypt.hash('viewerpass', 1);
|
||||
DatabaseService.getInstance().addUser({ username: 'guard-viewer', password_hash: viewerHash, role: 'viewer' });
|
||||
const viewerRes = await request(app).post('/api/auth/login').send({ username: 'guard-viewer', password: 'viewerpass' });
|
||||
const cookies = viewerRes.headers['set-cookie'] as string | string[];
|
||||
viewerCookie = Array.isArray(cookies) ? cookies[0] : cookies;
|
||||
});
|
||||
|
||||
afterAll(() => {
|
||||
vi.restoreAllMocks();
|
||||
cleanupTestDb(tmpDir);
|
||||
});
|
||||
|
||||
beforeEach(() => {
|
||||
mockComputeUpdateReadiness.mockReset();
|
||||
mockComputeRollbackReadiness.mockReset();
|
||||
});
|
||||
|
||||
describe('GET /api/stacks/:stackName/update-readiness', () => {
|
||||
it('requires authentication', async () => {
|
||||
const res = await request(app).get('/api/stacks/web/update-readiness');
|
||||
expect(res.status).toBe(401);
|
||||
expect(mockComputeUpdateReadiness).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('returns the computed report', async () => {
|
||||
const report = { stack: 'web', computedAt: 1, verdict: 'ready', signals: [] };
|
||||
mockComputeUpdateReadiness.mockResolvedValue(report);
|
||||
const res = await request(app).get('/api/stacks/web/update-readiness').set('Cookie', authCookie);
|
||||
expect(res.status).toBe(200);
|
||||
expect(res.body).toEqual(report);
|
||||
});
|
||||
|
||||
it('returns a clean 500 when the computation fails', async () => {
|
||||
mockComputeUpdateReadiness.mockRejectedValue(new Error('docker exploded'));
|
||||
const res = await request(app).get('/api/stacks/web/update-readiness').set('Cookie', authCookie);
|
||||
expect(res.status).toBe(500);
|
||||
expect(res.body).toEqual({ error: 'Failed to compute update readiness' });
|
||||
});
|
||||
});
|
||||
|
||||
describe('GET /api/stacks/:stackName/rollback-readiness', () => {
|
||||
it('requires authentication', async () => {
|
||||
const res = await request(app).get('/api/stacks/web/rollback-readiness');
|
||||
expect(res.status).toBe(401);
|
||||
expect(mockComputeRollbackReadiness).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('returns the computed report', async () => {
|
||||
const report = { stack: 'web', computedAt: 1, overall: 'partial', items: [] };
|
||||
mockComputeRollbackReadiness.mockResolvedValue(report);
|
||||
const res = await request(app).get('/api/stacks/web/rollback-readiness').set('Cookie', authCookie);
|
||||
expect(res.status).toBe(200);
|
||||
expect(res.body).toEqual(report);
|
||||
});
|
||||
|
||||
it('returns a clean 500 when the computation fails', async () => {
|
||||
mockComputeRollbackReadiness.mockRejectedValue(new Error('boom'));
|
||||
const res = await request(app).get('/api/stacks/web/rollback-readiness').set('Cookie', authCookie);
|
||||
expect(res.status).toBe(500);
|
||||
expect(res.body).toEqual({ error: 'Failed to compute rollback readiness' });
|
||||
});
|
||||
});
|
||||
|
||||
describe('GET /api/stacks/:stackName/health-gate', () => {
|
||||
const seedRun = async (id: string, status: 'observing' | 'passed' | 'failed' | 'unknown', startedAt: number, reason: string | null = null) => {
|
||||
const { DatabaseService } = await import('../services/DatabaseService');
|
||||
const db = DatabaseService.getInstance();
|
||||
// The route resolves req.nodeId to the seeded default node, so the rows
|
||||
// must carry that id, not a literal.
|
||||
const defaultNodeId = db.getNodes().find(n => n.is_default)!.id;
|
||||
db.insertHealthGateRun({
|
||||
id, node_id: defaultNodeId, stack_name: 'web', trigger_action: 'update', status, reason,
|
||||
window_seconds: 90, containers_json: '[]', started_at: startedAt, ended_at: status === 'observing' ? null : startedAt + 1000, created_by: 'tester',
|
||||
});
|
||||
};
|
||||
|
||||
it('requires authentication', async () => {
|
||||
const res = await request(app).get('/api/stacks/web/health-gate');
|
||||
expect(res.status).toBe(401);
|
||||
});
|
||||
|
||||
it('returns the never-run sentinel before any run exists', async () => {
|
||||
const res = await request(app).get('/api/stacks/web/health-gate').set('Cookie', authCookie);
|
||||
expect(res.status).toBe(200);
|
||||
expect(res.body).toMatchObject({ stack: 'web', id: null, status: 'never-run' });
|
||||
});
|
||||
|
||||
it('returns the latest run without a gateId and a specific (superseded) run with one', async () => {
|
||||
await seedRun('gate-old', 'unknown', 1_000, 'superseded by a newer update');
|
||||
await seedRun('gate-new', 'passed', 2_000);
|
||||
|
||||
const latest = await request(app).get('/api/stacks/web/health-gate').set('Cookie', authCookie);
|
||||
expect(latest.status).toBe(200);
|
||||
expect(latest.body).toMatchObject({ id: 'gate-new', status: 'passed' });
|
||||
|
||||
const byId = await request(app).get('/api/stacks/web/health-gate?gateId=gate-old').set('Cookie', authCookie);
|
||||
expect(byId.status).toBe(200);
|
||||
expect(byId.body).toMatchObject({ id: 'gate-old', status: 'unknown', reason: 'superseded by a newer update' });
|
||||
});
|
||||
|
||||
it('returns never-run for an unknown gateId', async () => {
|
||||
const res = await request(app).get('/api/stacks/web/health-gate?gateId=no-such-id').set('Cookie', authCookie);
|
||||
expect(res.status).toBe(200);
|
||||
expect(res.body).toMatchObject({ status: 'never-run' });
|
||||
});
|
||||
});
|
||||
|
||||
describe('GET /api/fleet/snapshots/coverage', () => {
|
||||
it('requires authentication', async () => {
|
||||
const res = await request(app).get('/api/fleet/snapshots/coverage?nodeId=0&stackName=web');
|
||||
expect(res.status).toBe(401);
|
||||
});
|
||||
|
||||
it('hits the coverage handler, not the /snapshots/:id route', async () => {
|
||||
const res = await request(app)
|
||||
.get('/api/fleet/snapshots/coverage?nodeId=0&stackName=web')
|
||||
.set('Cookie', authCookie);
|
||||
// The /:id handler would have rejected "coverage" as a bad snapshot ID;
|
||||
// the coverage handler returns the latestAt shape instead.
|
||||
expect(res.status).toBe(200);
|
||||
expect(res.body).toEqual({ latestAt: null });
|
||||
});
|
||||
|
||||
it('scopes coverage to the requested node and stack', async () => {
|
||||
const { DatabaseService } = await import('../services/DatabaseService');
|
||||
const db = DatabaseService.getInstance();
|
||||
const matching = db.createSnapshot('covers web on node 0', 'tester', 1, 1, '[]');
|
||||
db.insertSnapshotFiles(matching, [
|
||||
{ nodeId: 0, nodeName: 'local', stackName: 'web', filename: 'compose.yaml', content: 'services: {}' },
|
||||
]);
|
||||
const decoy = db.createSnapshot('covers other things', 'tester', 2, 2, '[]');
|
||||
db.insertSnapshotFiles(decoy, [
|
||||
{ nodeId: 1, nodeName: 'remote', stackName: 'web', filename: 'compose.yaml', content: 'services: {}' },
|
||||
{ nodeId: 0, nodeName: 'local', stackName: 'other', filename: 'compose.yaml', content: 'services: {}' },
|
||||
]);
|
||||
const matchingCreatedAt = db.getSnapshots().find(s => s.id === matching)!.created_at;
|
||||
|
||||
const res = await request(app)
|
||||
.get('/api/fleet/snapshots/coverage?nodeId=0&stackName=web')
|
||||
.set('Cookie', authCookie);
|
||||
expect(res.status).toBe(200);
|
||||
expect(res.body).toEqual({ latestAt: matchingCreatedAt });
|
||||
});
|
||||
|
||||
it('rejects non-admin users', async () => {
|
||||
const res = await request(app)
|
||||
.get('/api/fleet/snapshots/coverage?nodeId=0&stackName=web')
|
||||
.set('Cookie', viewerCookie);
|
||||
expect(res.status).toBe(403);
|
||||
});
|
||||
|
||||
it('validates nodeId and stackName', async () => {
|
||||
const missingNode = await request(app)
|
||||
.get('/api/fleet/snapshots/coverage?stackName=web')
|
||||
.set('Cookie', authCookie);
|
||||
expect(missingNode.status).toBe(400);
|
||||
|
||||
const badNode = await request(app)
|
||||
.get('/api/fleet/snapshots/coverage?nodeId=-2&stackName=web')
|
||||
.set('Cookie', authCookie);
|
||||
expect(badNode.status).toBe(400);
|
||||
|
||||
// A trailing-garbage nodeId must be rejected, not coerced by parseInt.
|
||||
const garbageNode = await request(app)
|
||||
.get('/api/fleet/snapshots/coverage?nodeId=1abc&stackName=web')
|
||||
.set('Cookie', authCookie);
|
||||
expect(garbageNode.status).toBe(400);
|
||||
|
||||
const badStack = await request(app)
|
||||
.get('/api/fleet/snapshots/coverage?nodeId=0&stackName=..%2Fetc')
|
||||
.set('Cookie', authCookie);
|
||||
expect(badStack.status).toBe(400);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,155 @@
|
||||
/**
|
||||
* Wiring tests for UpdateGuardService: container probing resilience and the
|
||||
* degrade-everything-to-unknown contract when every collaborator fails. The
|
||||
* grading rules themselves are covered by the pure readiness tests.
|
||||
*/
|
||||
import { describe, it, expect, vi, beforeEach } from 'vitest';
|
||||
|
||||
const {
|
||||
mockListContainers,
|
||||
mockGetContainer,
|
||||
mockGetLatest,
|
||||
mockGetPreview,
|
||||
mockGetBackupInfo,
|
||||
mockGetOpenDriftFindings,
|
||||
mockGetGlobalSettings,
|
||||
mockFsSize,
|
||||
} = vi.hoisted(() => ({
|
||||
mockListContainers: vi.fn(),
|
||||
mockGetContainer: vi.fn(),
|
||||
mockGetLatest: vi.fn(),
|
||||
mockGetPreview: vi.fn(),
|
||||
mockGetBackupInfo: vi.fn(),
|
||||
mockGetOpenDriftFindings: vi.fn(),
|
||||
mockGetGlobalSettings: vi.fn(),
|
||||
mockFsSize: vi.fn(),
|
||||
}));
|
||||
|
||||
vi.mock('../services/DockerController', () => ({
|
||||
default: {
|
||||
getInstance: () => ({
|
||||
getDocker: () => ({
|
||||
listContainers: mockListContainers,
|
||||
getContainer: mockGetContainer,
|
||||
}),
|
||||
}),
|
||||
},
|
||||
}));
|
||||
|
||||
vi.mock('../services/ComposeDoctorService', () => ({
|
||||
ComposeDoctorService: { getInstance: () => ({ getLatest: mockGetLatest }) },
|
||||
}));
|
||||
|
||||
vi.mock('../services/UpdatePreviewService', () => ({
|
||||
UpdatePreviewService: { getInstance: () => ({ getPreview: mockGetPreview }) },
|
||||
}));
|
||||
|
||||
vi.mock('../services/FileSystemService', () => ({
|
||||
FileSystemService: {
|
||||
getInstance: () => ({
|
||||
getBackupInfo: mockGetBackupInfo,
|
||||
getBackupEnvSummary: vi.fn().mockRejectedValue(new Error('not used here')),
|
||||
envExists: vi.fn().mockRejectedValue(new Error('not used here')),
|
||||
}),
|
||||
},
|
||||
}));
|
||||
|
||||
vi.mock('../services/DatabaseService', () => ({
|
||||
DatabaseService: {
|
||||
getInstance: () => ({
|
||||
getOpenDriftFindings: mockGetOpenDriftFindings,
|
||||
getGlobalSettings: mockGetGlobalSettings,
|
||||
getStackActivity: vi.fn().mockReturnValue([]),
|
||||
}),
|
||||
},
|
||||
}));
|
||||
|
||||
vi.mock('systeminformation', () => ({
|
||||
default: { fsSize: mockFsSize },
|
||||
}));
|
||||
|
||||
import { UpdateGuardService } from '../services/UpdateGuardService';
|
||||
|
||||
const inspectResult = (over: Record<string, unknown> = {}) => ({
|
||||
State: { Status: 'running', ExitCode: 0 },
|
||||
Config: { Healthcheck: { Test: ['CMD', 'true'] } },
|
||||
HostConfig: { RestartPolicy: { Name: 'unless-stopped' } },
|
||||
Mounts: [],
|
||||
...over,
|
||||
});
|
||||
|
||||
beforeEach(() => {
|
||||
vi.clearAllMocks();
|
||||
mockGetGlobalSettings.mockReturnValue({ host_disk_limit: '90' });
|
||||
});
|
||||
|
||||
describe('UpdateGuardService.probeContainers', () => {
|
||||
it('skips a container that vanished between list and inspect (404)', async () => {
|
||||
mockListContainers.mockResolvedValue([
|
||||
{ Id: 'aaa', Names: ['/app-web-1'], State: 'running' },
|
||||
{ Id: 'bbb', Names: ['/app-db-1'], State: 'running' },
|
||||
]);
|
||||
mockGetContainer.mockImplementation((id: string) => ({
|
||||
inspect: id === 'bbb'
|
||||
? vi.fn().mockRejectedValue(Object.assign(new Error('no such container'), { statusCode: 404 }))
|
||||
: vi.fn().mockResolvedValue(inspectResult()),
|
||||
}));
|
||||
|
||||
const probes = await UpdateGuardService.getInstance().probeContainers(0, 'app');
|
||||
expect(probes).toHaveLength(1);
|
||||
expect(probes[0].name).toBe('app-web-1');
|
||||
});
|
||||
|
||||
it('propagates non-404 inspect failures so the whole signal degrades honestly', async () => {
|
||||
mockListContainers.mockResolvedValue([
|
||||
{ Id: 'aaa', Names: ['/app-web-1'], State: 'running' },
|
||||
]);
|
||||
mockGetContainer.mockReturnValue({
|
||||
inspect: vi.fn().mockRejectedValue(Object.assign(new Error('daemon hiccup'), { statusCode: 500 })),
|
||||
});
|
||||
|
||||
await expect(UpdateGuardService.getInstance().probeContainers(0, 'app')).rejects.toThrow('daemon hiccup');
|
||||
});
|
||||
});
|
||||
|
||||
describe('UpdateGuardService.computeUpdateReadiness wiring', () => {
|
||||
it('returns a complete unknown-verdict report when every collaborator fails', async () => {
|
||||
mockGetLatest.mockImplementation(() => { throw new Error('db gone'); });
|
||||
mockGetOpenDriftFindings.mockImplementation(() => { throw new Error('db gone'); });
|
||||
mockListContainers.mockRejectedValue(new Error('docker gone'));
|
||||
mockGetPreview.mockRejectedValue(new Error('registry gone'));
|
||||
mockGetBackupInfo.mockRejectedValue(new Error('fs gone'));
|
||||
mockFsSize.mockRejectedValue(new Error('si gone'));
|
||||
|
||||
const report = await UpdateGuardService.getInstance().computeUpdateReadiness(0, 'app');
|
||||
|
||||
expect(report.stack).toBe('app');
|
||||
expect(report.signals.map(s => s.id)).toEqual([
|
||||
'preflight', 'drift', 'containers', 'healthchecks', 'update_preview', 'backup_slot', 'disk',
|
||||
]);
|
||||
// The container probe failure is the verdict-affecting unknown.
|
||||
expect(report.verdict).toBe('unknown');
|
||||
});
|
||||
|
||||
it('produces a ready verdict from healthy collaborator outputs', async () => {
|
||||
mockGetLatest.mockReturnValue({ status: 'pass' });
|
||||
mockGetOpenDriftFindings.mockReturnValue([]);
|
||||
mockListContainers.mockResolvedValue([{ Id: 'aaa', Names: ['/app-web-1'], State: 'running' }]);
|
||||
mockGetContainer.mockReturnValue({ inspect: vi.fn().mockResolvedValue(inspectResult()) });
|
||||
mockGetPreview.mockResolvedValue({
|
||||
stack_name: 'app',
|
||||
images: [],
|
||||
summary: {
|
||||
has_update: true, primary_image: 'nginx', current_tag: '1.27.0', next_tag: '1.27.1',
|
||||
semver_bump: 'patch', update_kind: 'tag', blocked: false, blocked_reason: null,
|
||||
},
|
||||
rollback_target: 'nginx:1.27.0',
|
||||
changelog: null,
|
||||
});
|
||||
mockGetBackupInfo.mockResolvedValue({ exists: true, timestamp: Date.now() });
|
||||
mockFsSize.mockResolvedValue([{ mount: '/', use: 42 }]);
|
||||
|
||||
const report = await UpdateGuardService.getInstance().computeUpdateReadiness(0, 'app');
|
||||
expect(report.verdict).toBe('ready');
|
||||
});
|
||||
});
|
||||
@@ -443,3 +443,49 @@ describe('webhook_executions.error redaction (M6)', () => {
|
||||
expect(history[0].error).toContain('/home/<user>');
|
||||
});
|
||||
});
|
||||
|
||||
describe('WebhookService.execute: health gate begin call sites', () => {
|
||||
// FileSystemService and ComposeService return a fresh instance per nodeId,
|
||||
// so those spies attach to the prototypes (matching the redaction tests
|
||||
// above); HealthGateService is a singleton spied directly. The policy gate
|
||||
// is stubbed to allow so the test isolates the begin wiring, not policy.
|
||||
it('begins a deploy gate after a webhook deploy succeeds', async () => {
|
||||
const stack = 'hook-deploy-gate';
|
||||
const { id } = createWebhook({ action: 'deploy', stack });
|
||||
const webhook = DatabaseService.getInstance().getWebhook(id)!;
|
||||
const nodeId = DatabaseService.getInstance().getDefaultNode()!.id!;
|
||||
|
||||
const fs = await import('../services/FileSystemService');
|
||||
const compose = await import('../services/ComposeService');
|
||||
const policyGate = await import('../helpers/policyGate');
|
||||
const { HealthGateService } = await import('../services/HealthGateService');
|
||||
vi.spyOn(policyGate, 'assertPolicyGateAllows').mockResolvedValue(undefined);
|
||||
vi.spyOn(fs.FileSystemService.prototype, 'getStacks').mockResolvedValue([stack]);
|
||||
vi.spyOn(compose.ComposeService.prototype, 'deployStack').mockResolvedValue(undefined);
|
||||
const beginSpy = vi.spyOn(HealthGateService.getInstance(), 'begin').mockReturnValue('gate-hook');
|
||||
|
||||
const result = await WebhookService.getInstance().execute(webhook, 'deploy', 'test', true);
|
||||
expect(result.success).toBe(true);
|
||||
expect(beginSpy).toHaveBeenCalledWith(nodeId, stack, 'deploy', 'system:webhook');
|
||||
});
|
||||
|
||||
it('begins an update gate after a webhook pull succeeds', async () => {
|
||||
const stack = 'hook-pull-gate';
|
||||
const { id } = createWebhook({ action: 'pull', stack });
|
||||
const webhook = DatabaseService.getInstance().getWebhook(id)!;
|
||||
const nodeId = DatabaseService.getInstance().getDefaultNode()!.id!;
|
||||
|
||||
const fs = await import('../services/FileSystemService');
|
||||
const compose = await import('../services/ComposeService');
|
||||
const policyGate = await import('../helpers/policyGate');
|
||||
const { HealthGateService } = await import('../services/HealthGateService');
|
||||
vi.spyOn(policyGate, 'assertPolicyGateAllows').mockResolvedValue(undefined);
|
||||
vi.spyOn(fs.FileSystemService.prototype, 'getStacks').mockResolvedValue([stack]);
|
||||
vi.spyOn(compose.ComposeService.prototype, 'updateStack').mockResolvedValue(undefined);
|
||||
const beginSpy = vi.spyOn(HealthGateService.getInstance(), 'begin').mockReturnValue('gate-hook');
|
||||
|
||||
const result = await WebhookService.getInstance().execute(webhook, 'pull', 'test', true);
|
||||
expect(result.success).toBe(true);
|
||||
expect(beginSpy).toHaveBeenCalledWith(nodeId, stack, 'update', 'system:webhook');
|
||||
});
|
||||
});
|
||||
|
||||
Reference in New Issue
Block a user