Files
sencho/backend/src/__tests__/webhooks-trigger.test.ts
T
Anso 38aabe7064 feat: health-gated updates and rollback readiness (#1354)
* feat: classify stack deploy and update failures with suggested next actions

Failed deploy and update responses now carry a failure classification
(cause category, headline, and suggested next step) derived from the
compose error output. The recovery panel and chip render the
classification and include it in copied diagnostics, and gateway-style
failures surface as a node-unreachable cause.

* feat: add update and rollback readiness reports for stacks

Before a manual update, Sencho now shows an advisory readiness verdict
computed from the stored preflight result, open drift findings, live
container health, the pending image change, the rollback backup slot,
and node disk headroom. The Stack Dossier gains a rollback readiness
section that states what a rollback can restore and explicitly
discloses that volume and bind-mounted data are not covered. Toolbar
and sidebar updates now share one update path, and admins can create a
fleet snapshot from the readiness dialog before updating. Nodes that do
not advertise the capability keep the direct update flow.

* feat: observe stack health after updates with a post-deploy health gate

After a deploy or update succeeds, Sencho now watches the stack for a
configurable observation window and records a passed, failed, or
unknown verdict: containers must stay running, healthchecks must report
healthy, and restart loops or disappearing containers fail the gate.
The deploy panel shows the observation live and holds off auto-closing
until the verdict lands, a failed gate surfaces the existing recovery
actions including rollback, and the stack timeline records update
started and gate verdict events. Scheduled, webhook, bulk, and
git-source updates are gated the same way; rollbacks and installs are
deliberately not. The gate is observational only and can be tuned or
disabled per node under host alert settings.

* docs: document health-gated updates and rollback readiness

New operator page covering the update readiness dialog, the post-update
health gate and its settings, the rollback readiness disclosure, and
classified failures, with cross-links from the atomic deployments and
deploy progress pages. The API reference gains the readiness and
health-gate endpoints, the healthGateId success field, and the failure
classification schema on deploy and update error responses.

* feat: withhold the success verdict while the health gate observes

An update used to show a green Succeeded that a failed health gate then
contradicted moments later. The deploy modal now reports Verifying
health while the gate observes, shows success only when the gate
passes, and makes a failed or unknown gate the headline result; success
toasts soften to a verifying message while a gate runs. The mobile
recovery card groups its actions behind one bottom-right Take action
menu so it stays compact on a phone, with the classified cause still
visible on the card. A successful image update now also counts as the
last known-good marker in rollback readiness, and the docs gain
screenshots of the readiness dialog, gate states, dossier section, and
settings.

* fix: harden log format strings and the env existence path check

Log calls that interpolated the stack name into the console format
string now use constant format strings with placeholder arguments, and
envExists validates path containment inline at its filesystem access,
matching the established patterns used elsewhere in the same files.

* test: adapt deploy modal success specs to the post-deploy health gate

The deploy feedback modal now withholds its success verdict while the
health gate observes the new containers, showing "Verifying health"
until the gate passes. The two success-path E2E tests waited for
"Succeeded" within the gate's 90s default window and timed out.

Shorten the observation window to the 15s minimum for these tests via
the settings API, assert the verify-then-succeed sequence the modal
actually renders, and restore the default window afterward so the test
value does not leak into later runs.

* fix: serialize health gate polling and harden gate observation

Address race conditions in the post-update health gate found in review.

Backend: the gate poller used setInterval, so a Docker observe slower
than the 5s tick could overlap the next poll and corrupt the restart and
missing-container accounting, and a wedged socket could leave a poll
pending forever. Polling is now single-flight: each cycle self-schedules
the next only after it settles, and the observe is bounded by an 8s
timeout so a hung probe counts as a poll error and resolves the gate
unknown after three in a row.

Frontend: the gate poller could overlap requests, letting a slow earlier
"observing" response overwrite an already-applied terminal verdict. It is
now single-flight with a terminal latch, so a late response can never
roll the UI back from passed or failed.

Also reject a non-digit nodeId on the snapshot coverage route instead of
letting parseInt coerce it, document that turning off the deploy progress
panel opts out of the live gate UI while the gate still runs server-side,
and add gate-coverage tests for the webhook, git source, and auto-update
apply paths plus the new single-flight, observe-timeout, and recovery
cases.
2026-06-11 00:26:26 -04:00

492 lines
21 KiB
TypeScript

import { describe, it, expect, beforeAll, afterAll, beforeEach, vi } from 'vitest';
import request from 'supertest';
import crypto from 'crypto';
import jwt from 'jsonwebtoken';
import { setupTestDb, cleanupTestDb, TEST_USERNAME, TEST_JWT_SECRET } from './helpers/setupTestDb';
let tmpDir: string;
let app: import('express').Express;
let DatabaseService: typeof import('../services/DatabaseService').DatabaseService;
let LicenseService: typeof import('../services/LicenseService').LicenseService;
let WebhookService: typeof import('../services/WebhookService').WebhookService;
function adminToken(): string {
return jwt.sign({ username: TEST_USERNAME, role: 'admin' }, TEST_JWT_SECRET, { expiresIn: '1m' });
}
function sign(rawBody: string, secret: string): string {
return 'sha256=' + crypto.createHmac('sha256', secret).update(rawBody).digest('hex');
}
interface WebhookFixture {
id: number;
secret: string;
}
function createWebhook(opts: { action?: string; enabled?: boolean; name?: string; stack?: string } = {}): WebhookFixture {
const db = DatabaseService.getInstance();
const nodeId = db.getDefaultNode()!.id!;
const secret = WebhookService.getInstance().generateSecret();
const id = db.addWebhook({
node_id: nodeId,
name: opts.name ?? 'trigger-test',
stack_name: opts.stack ?? 'missing-stack',
action: (opts.action ?? 'restart') as never,
secret,
enabled: opts.enabled ?? true,
});
return { id, secret };
}
beforeAll(async () => {
tmpDir = await setupTestDb();
({ app } = await import('../index'));
({ DatabaseService } = await import('../services/DatabaseService'));
({ LicenseService } = await import('../services/LicenseService'));
({ WebhookService } = await import('../services/WebhookService'));
});
afterAll(() => {
cleanupTestDb(tmpDir);
});
beforeEach(() => {
vi.restoreAllMocks();
// Webhooks are free; run at the Community tier to prove the trigger and the
// management routes work without a paid license.
vi.spyOn(LicenseService.getInstance(), 'getTier').mockReturnValue('community');
});
describe('POST /api/webhooks/:id/trigger: uniform unauthenticated 404 (M1, H3)', () => {
const expected = { error: 'Webhook not found or signature invalid' };
it('returns 404 when the webhook id is unknown', async () => {
const body = '{}';
const res = await request(app)
.post('/api/webhooks/9999999/trigger')
.set('Content-Type', 'application/json')
.set('X-Webhook-Signature', sign(body, 'whatever'))
.send(body);
expect(res.status).toBe(404);
expect(res.body).toEqual(expected);
});
it('returns the same 404 when the webhook exists but is disabled', async () => {
const { id, secret } = createWebhook({ enabled: false });
const body = '{}';
const res = await request(app)
.post(`/api/webhooks/${id}/trigger`)
.set('Content-Type', 'application/json')
.set('X-Webhook-Signature', sign(body, secret))
.send(body);
expect(res.status).toBe(404);
expect(res.body).toEqual(expected);
});
it('returns the same 404 when the X-Webhook-Signature header is missing', async () => {
const { id } = createWebhook();
const body = '{}';
const res = await request(app)
.post(`/api/webhooks/${id}/trigger`)
.set('Content-Type', 'application/json')
.send(body);
expect(res.status).toBe(404);
expect(res.body).toEqual(expected);
});
it('returns the same 404 when the request has no body (H3 fail-closed)', async () => {
const { id, secret } = createWebhook();
const res = await request(app)
.post(`/api/webhooks/${id}/trigger`)
// No Content-Type → express.json() does not run verify, so
// req.rawBody is never populated. The handler must fail closed
// instead of re-stringifying req.body to compute the HMAC.
.set('X-Webhook-Signature', sign('', secret));
expect(res.status).toBe(404);
expect(res.body).toEqual(expected);
});
it('returns the same 404 for a signature with the wrong prefix', async () => {
const { id, secret } = createWebhook();
const body = '{}';
const hex = crypto.createHmac('sha256', secret).update(body).digest('hex');
const res = await request(app)
.post(`/api/webhooks/${id}/trigger`)
.set('Content-Type', 'application/json')
.set('X-Webhook-Signature', `sha1=${hex}`)
.send(body);
expect(res.status).toBe(404);
expect(res.body).toEqual(expected);
});
it('returns the same 404 for a malformed hex signature', async () => {
const { id } = createWebhook();
const body = '{}';
const res = await request(app)
.post(`/api/webhooks/${id}/trigger`)
.set('Content-Type', 'application/json')
.set('X-Webhook-Signature', 'sha256=notavalidhex-zzzz')
.send(body);
expect(res.status).toBe(404);
expect(res.body).toEqual(expected);
});
it('returns the same 404 when the signature does not match the body', async () => {
const { id } = createWebhook();
const body = '{"foo":"bar"}';
const wrongSig = sign(body, 'wrong-secret-of-equal-length-as-the-real-one-1234');
const res = await request(app)
.post(`/api/webhooks/${id}/trigger`)
.set('Content-Type', 'application/json')
.set('X-Webhook-Signature', wrongSig)
.send(body);
expect(res.status).toBe(404);
expect(res.body).toEqual(expected);
});
it('runs validateSignature even when the webhook id is unknown (timing oracle)', async () => {
const sigSpy = vi.spyOn(WebhookService.getInstance(), 'validateSignature');
const body = '{"probe":true}';
const res = await request(app)
.post('/api/webhooks/9999999/trigger')
.set('Content-Type', 'application/json')
.set('X-Webhook-Signature', `sha256=${'a'.repeat(64)}`)
.send(body);
expect(res.status).toBe(404);
expect(sigSpy).toHaveBeenCalledTimes(1);
const [usedPayload, usedSecret, usedSignature] = sigSpy.mock.calls[0];
// HMAC ran against the attacker-supplied body, not a short-circuited
// empty string; the unknown-id path must do the same work as the
// wrong-signature path.
expect(usedPayload).toBe(body);
// Decoy secret got plumbed through so the HMAC compute was real.
expect(typeof usedSecret).toBe('string');
expect((usedSecret as string).length).toBeGreaterThan(0);
expect(usedSignature).toContain('sha256=');
});
it('runs validateSignature even when the signature header is missing', async () => {
const sigSpy = vi.spyOn(WebhookService.getInstance(), 'validateSignature');
const { id } = createWebhook();
const body = '{}';
const res = await request(app)
.post(`/api/webhooks/${id}/trigger`)
.set('Content-Type', 'application/json')
.send(body);
expect(res.status).toBe(404);
expect(sigSpy).toHaveBeenCalledTimes(1);
const [usedPayload, , usedSignature] = sigSpy.mock.calls[0];
expect(usedPayload).toBe(body);
// Empty-string signature flowed into validateSignature instead of
// short-circuiting before the HMAC compute.
expect(usedSignature).toBe('');
});
});
describe('WebhookService.validateSignature: constant-time over input shape', () => {
it('returns false but does not throw on an empty signature string', () => {
const result = WebhookService.getInstance().validateSignature('payload', 'secret', '');
expect(result).toBe(false);
});
it('returns false but does not throw on a wrong-prefix signature', () => {
const result = WebhookService.getInstance().validateSignature('payload', 'secret', 'sha1=abc');
expect(result).toBe(false);
});
it('returns false but does not throw on a malformed hex signature', () => {
const result = WebhookService.getInstance().validateSignature('payload', 'secret', 'sha256=not-hex-data-shorter-than-64-chars');
expect(result).toBe(false);
});
it('returns false but does not throw on a hex signature of the wrong length', () => {
const result = WebhookService.getInstance().validateSignature('payload', 'secret', `sha256=${'a'.repeat(32)}`);
expect(result).toBe(false);
});
});
describe('POST /api/webhooks/:id/trigger: authenticated happy path', () => {
it('returns 202 and echoes the configured action', async () => {
const { id, secret } = createWebhook({ action: 'stop' });
const body = '{}';
const res = await request(app)
.post(`/api/webhooks/${id}/trigger`)
.set('Content-Type', 'application/json')
.set('X-Webhook-Signature', sign(body, secret))
.send(body);
expect(res.status).toBe(202);
expect(res.body).toMatchObject({ message: 'Webhook accepted', action: 'stop' });
});
it('accepts a valid action override and echoes it', async () => {
const { id, secret } = createWebhook({ action: 'restart' });
const body = '{"action":"start"}';
const res = await request(app)
.post(`/api/webhooks/${id}/trigger`)
.set('Content-Type', 'application/json')
.set('X-Webhook-Signature', sign(body, secret))
.send(body);
expect(res.status).toBe(202);
expect(res.body).toMatchObject({ action: 'start' });
});
it('rejects an unknown action override with 400 after the signature passes (L2)', async () => {
const { id, secret } = createWebhook();
const body = '{"action":"nuke-the-cluster"}';
const res = await request(app)
.post(`/api/webhooks/${id}/trigger`)
.set('Content-Type', 'application/json')
.set('X-Webhook-Signature', sign(body, secret))
.send(body);
// Auth succeeded, so the caller learns the action was rejected.
// Pre-auth callers would still get the uniform 404 instead.
expect(res.status).toBe(400);
expect(res.body.error).toMatch(/action must be one of/);
});
it('rejects a non-string action override with 400', async () => {
const { id, secret } = createWebhook();
const body = '{"action":42}';
const res = await request(app)
.post(`/api/webhooks/${id}/trigger`)
.set('Content-Type', 'application/json')
.set('X-Webhook-Signature', sign(body, secret))
.send(body);
expect(res.status).toBe(400);
expect(res.body.error).toMatch(/action must be one of/);
});
});
describe('POST /api/webhooks: name length cap (L1)', () => {
it('rejects a name longer than 100 characters', async () => {
const res = await request(app)
.post('/api/webhooks')
.set('Authorization', `Bearer ${adminToken()}`)
.send({
name: 'a'.repeat(101),
stack_name: 'irrelevant-stack',
action: 'restart',
});
expect(res.status).toBe(400);
expect(res.body.error).toMatch(/100 characters or fewer/);
});
it('rejects a non-string name', async () => {
const res = await request(app)
.post('/api/webhooks')
.set('Authorization', `Bearer ${adminToken()}`)
.send({
name: { obj: 'not-a-string' },
stack_name: 'irrelevant-stack',
action: 'restart',
});
expect(res.status).toBe(400);
// 'name, stack_name, and action are required' catches this when name is
// truthy-but-not-a-string before the length check; either error message
// is acceptable for non-string input.
expect(res.body.error).toBeTruthy();
});
it('accepts a name at the 100-character boundary', async () => {
const res = await request(app)
.post('/api/webhooks')
.set('Authorization', `Bearer ${adminToken()}`)
.send({
name: 'a'.repeat(100),
stack_name: 'boundary-stack',
action: 'restart',
});
expect(res.status).toBe(201);
expect(typeof res.body.secret).toBe('string');
});
});
describe('PUT /api/webhooks/:id: name length cap (L1)', () => {
it('rejects updating name to over 100 characters', async () => {
const { id } = createWebhook();
const res = await request(app)
.put(`/api/webhooks/${id}`)
.set('Authorization', `Bearer ${adminToken()}`)
.send({ name: 'b'.repeat(101) });
expect(res.status).toBe(400);
expect(res.body.error).toMatch(/100 characters or fewer/);
});
it('allows partial updates that omit name', async () => {
const { id } = createWebhook({ enabled: true });
const res = await request(app)
.put(`/api/webhooks/${id}`)
.set('Authorization', `Bearer ${adminToken()}`)
.send({ enabled: false });
expect(res.status).toBe(200);
expect(DatabaseService.getInstance().getWebhook(id)?.enabled).toBe(false);
});
});
describe('WebhookService.execute: delete-during-execution race (M5)', () => {
it('does not crash when the parent webhook is deleted before recordExecution runs', async () => {
// The webhook targets a stack that does not exist on disk, so
// executeLocal fails fast at the FileSystemService.getStacks() check
// and tries to write a failure row to webhook_executions. By the time
// that insert fires the parent row is gone, so the FK CASCADE makes
// the insert fail. The fix is for recordExecution to swallow that
// error with a console.warn instead of crashing the async dispatch.
const { id } = createWebhook({ action: 'restart', stack: 'definitely-not-on-disk' });
const webhook = DatabaseService.getInstance().getWebhook(id)!;
// Mid-flight delete.
DatabaseService.getInstance().deleteWebhook(id);
const warnSpy = vi.spyOn(console, 'warn').mockImplementation(() => undefined);
await expect(
WebhookService.getInstance().execute(webhook, 'restart', 'test', true),
).resolves.toMatchObject({ success: false });
// recordExecution caught the FK error and logged a single warning.
const calls = warnSpy.mock.calls.map(args => args.join(' '));
expect(calls.some(line => line.includes(`webhook ${id}`))).toBe(true);
});
it('records the execution row when the webhook persists through execution', async () => {
const { id } = createWebhook({ action: 'restart', stack: 'definitely-not-on-disk' });
const webhook = DatabaseService.getInstance().getWebhook(id)!;
const result = await WebhookService.getInstance().execute(webhook, 'restart', 'test', true);
expect(result.success).toBe(false);
// Filter by webhook_id rather than asserting toHaveLength on the
// entire history: getWebhookExecutions already scopes to this row's
// id, but tests in this file share a baseline DB and an earlier
// run could re-use an id range. Read it positionally instead.
const history = DatabaseService.getInstance().getWebhookExecutions(id);
expect(history.length).toBeGreaterThanOrEqual(1);
expect(history[0].status).toBe('failure');
expect(history[0].error).toMatch(/not found/i);
});
});
describe('webhook_executions.error redaction (M6)', () => {
// Both tests force executeLocal's switch-statement try/catch path so the
// raw upstream error flows through getErrorMessage -> recordExecution ->
// redactSensitiveText. FileSystemService.getStacks is stubbed to claim the
// stack exists, and ComposeService.runCommand is stubbed to throw the
// sensitive content. Spies attach to the prototypes because both
// singletons hand out fresh instances per nodeId.
it('strips bearer tokens before persisting the execution error', async () => {
const stack = 'redact-stack-bearer';
const { id } = createWebhook({ action: 'restart', stack });
const webhook = DatabaseService.getInstance().getWebhook(id)!;
const fs = await import('../services/FileSystemService');
const compose = await import('../services/ComposeService');
vi.spyOn(fs.FileSystemService.prototype, 'getStacks').mockResolvedValue([stack]);
vi.spyOn(compose.ComposeService.prototype, 'runCommand').mockRejectedValue(
new Error('upstream rejected: Authorization: Bearer abcdef1234567890tokenvalue'),
);
const result = await WebhookService.getInstance().execute(webhook, 'restart', 'test', true);
expect(result.success).toBe(false);
const history = DatabaseService.getInstance().getWebhookExecutions(id);
expect(history[0].error).toBeTruthy();
expect(history[0].error).not.toContain('abcdef1234567890tokenvalue');
expect(history[0].error).toContain('[redacted]');
});
it('strips homedir paths before persisting the execution error', async () => {
const stack = 'redact-stack-home';
const { id } = createWebhook({ action: 'restart', stack });
const webhook = DatabaseService.getInstance().getWebhook(id)!;
const fs = await import('../services/FileSystemService');
const compose = await import('../services/ComposeService');
vi.spyOn(fs.FileSystemService.prototype, 'getStacks').mockResolvedValue([stack]);
vi.spyOn(compose.ComposeService.prototype, 'runCommand').mockRejectedValue(
new Error('compose error reading /home/user-redact-target/docker/compose.yaml'),
);
const result = await WebhookService.getInstance().execute(webhook, 'restart', 'test', true);
expect(result.success).toBe(false);
const history = DatabaseService.getInstance().getWebhookExecutions(id);
expect(history[0].error).toBeTruthy();
expect(history[0].error).not.toContain('user-redact-target');
expect(history[0].error).toContain('/home/<user>');
});
});
describe('WebhookService.execute: health gate begin call sites', () => {
// FileSystemService and ComposeService return a fresh instance per nodeId,
// so those spies attach to the prototypes (matching the redaction tests
// above); HealthGateService is a singleton spied directly. The policy gate
// is stubbed to allow so the test isolates the begin wiring, not policy.
it('begins a deploy gate after a webhook deploy succeeds', async () => {
const stack = 'hook-deploy-gate';
const { id } = createWebhook({ action: 'deploy', stack });
const webhook = DatabaseService.getInstance().getWebhook(id)!;
const nodeId = DatabaseService.getInstance().getDefaultNode()!.id!;
const fs = await import('../services/FileSystemService');
const compose = await import('../services/ComposeService');
const policyGate = await import('../helpers/policyGate');
const { HealthGateService } = await import('../services/HealthGateService');
vi.spyOn(policyGate, 'assertPolicyGateAllows').mockResolvedValue(undefined);
vi.spyOn(fs.FileSystemService.prototype, 'getStacks').mockResolvedValue([stack]);
vi.spyOn(compose.ComposeService.prototype, 'deployStack').mockResolvedValue(undefined);
const beginSpy = vi.spyOn(HealthGateService.getInstance(), 'begin').mockReturnValue('gate-hook');
const result = await WebhookService.getInstance().execute(webhook, 'deploy', 'test', true);
expect(result.success).toBe(true);
expect(beginSpy).toHaveBeenCalledWith(nodeId, stack, 'deploy', 'system:webhook');
});
it('begins an update gate after a webhook pull succeeds', async () => {
const stack = 'hook-pull-gate';
const { id } = createWebhook({ action: 'pull', stack });
const webhook = DatabaseService.getInstance().getWebhook(id)!;
const nodeId = DatabaseService.getInstance().getDefaultNode()!.id!;
const fs = await import('../services/FileSystemService');
const compose = await import('../services/ComposeService');
const policyGate = await import('../helpers/policyGate');
const { HealthGateService } = await import('../services/HealthGateService');
vi.spyOn(policyGate, 'assertPolicyGateAllows').mockResolvedValue(undefined);
vi.spyOn(fs.FileSystemService.prototype, 'getStacks').mockResolvedValue([stack]);
vi.spyOn(compose.ComposeService.prototype, 'updateStack').mockResolvedValue(undefined);
const beginSpy = vi.spyOn(HealthGateService.getInstance(), 'begin').mockReturnValue('gate-hook');
const result = await WebhookService.getInstance().execute(webhook, 'pull', 'test', true);
expect(result.success).toBe(true);
expect(beginSpy).toHaveBeenCalledWith(nodeId, stack, 'update', 'system:webhook');
});
});