Files
sencho/backend/src/__tests__/stacks-failure-notifications.test.ts
T
Anso 2d56ea958a fix(stack-activity): per-stack history integrity, attribution, sanitization (#1228)
* fix(stack-activity): per-stack history integrity, attribution, sanitization

Address the Stack Activity audit findings (PR 1 of 2):

- Per-stack history integrity: drop the per-insert 100-row prune in
  addNotificationHistory that evicted quieter stacks' history whenever
  another stack got chatty. Periodic cleanupOldNotifications now caps
  per (node, stack) at 500 rows and per-node unattached system events
  at 1000 rows, on top of the existing 30-day retention. Signature
  takes an options bag and returns a per-stage summary so MonitorService
  can log what actually ran each cycle.

- Actor attribution: thread req.user?.username through every
  notifyActionFailure call site and add synthetic actors at service
  emit sites (system:autoheal, system:scheduler, system:image-update,
  system:docker-events, system:blueprint, system:monitor, system:policy).
  The timeline renders system actors as "via <Label>" so an autoheal
  redeploy is no longer indistinguishable from a user redeploy.

- Message sanitization: new sanitizeNotificationMessage at
  NotificationService.dispatchAlert strips KEY=VALUE pairs whose key
  ends in TOKEN/KEY/PASSWORD/SECRET/CREDENTIALS/AUTH, scrubs HTTP basic
  auth in URLs and Bearer tokens, collapses COMPOSE_DIR paths, and
  truncates to 1000 chars. Applied to the stored history and to every
  downstream Discord/Slack/webhook channel. The ImageUpdateService
  recovery-path direct DB write also runs through the sanitizer.

- Composite pagination cursor: getStackActivity now accepts a
  (timestamp, id) cursor (?before=&beforeId=). The legacy timestamp-only
  form silently dropped events when a single compose up emitted many
  events sharing one millisecond. Route rejects beforeId without before.

- Frontend hardening: distinct error state with retry button (initial
  fetch failure no longer renders as the genuine empty state), strict
  positive-integer parsing on cursor params, overrequest-by-1 pagination
  so the last page does not leave a dead "Load more" click, runtime
  guard on liveEvents merge that validates the level union, per-minute
  day-bucket recompute so an open panel does not stay on "Today" past
  midnight.

No tier, role, or capability gate touched. Route permission gate
remains stack:read on the named stack.

* fix(stack-activity): sanitizer covers lowercase env vars and per-node compose dir

External review surfaced two leak paths in the message sanitizer:

- The sensitive-key regex was uppercase-only. Compose env names are
  conventionally uppercase but lowercase forms (db_password, jwt_secret,
  github_token) are valid and do leak through the same Docker and
  compose-parse error paths. Make the regex case-insensitive and tighten
  it to also catch bare TOKEN= / KEY= / PASSWORD= without a prefix word,
  while still leaving BYPASS, COMPASS, and similar non-secret keys alone.

- The compose-dir path collapse only read process.env.COMPOSE_DIR, but
  the real resolution chain is node.compose_dir (per-node DB override)
  -> process.env.COMPOSE_DIR -> /app/compose. A node with a custom
  compose_dir could still leak absolute paths into stored history and
  downstream channels. Route both the dispatchAlert call and the
  ImageUpdateService recovery-path direct write through
  NodeRegistry.getInstance().getComposeDir(localNodeId) so the
  collapse covers every resolution outcome.

Tests now assert lowercase keys are redacted and that BYPASS-style
non-secrets stay intact in both cases. notification-routing mock
extended to stub the new getComposeDir call.

* chore(stack-activity): a11y roles, visibility-aware tick, live-disconnect signal

Close three small follow-ups on the per-stack activity timeline:

- A11y: each day-group gets role="list" and each event row gets
  role="listitem" so screen readers traverse the timeline as a list
  instead of a wall of text. The day-group container also carries an
  aria-label naming the bucket.

- Visibility-aware day-bucket tick: the 60s setInterval that re-derives
  Today/Yesterday/Earlier now short-circuits when document.hidden, so a
  backgrounded panel does not re-render every minute for no visible
  effect.

- Live-disconnect signal: useNotifications dispatches a
  sencho:notifications-connection custom event on WebSocket open and
  close. The timeline listens and, when explicitly disconnected, shows
  a one-line "Live updates offline; reconnecting…" hint above the list.
  The sidebar ticker already surfaces fleet-wide connection state; this
  adds an in-context cue for users who are focused on a single stack.

Stack-name case normalization was considered and rejected: stack names
are case-permissive per the isValidStackName validator, and lowercasing
on read or write would silently rename or hide a user's "MyApp" stack.

* ci(stack-activity): drop unnecessary escape in URL_BASIC_AUTH regex

ESLint no-useless-escape errored on \- inside the character class
[a-zA-Z0-9+.\-] at notificationMessage.ts:14. Move the dash to the
end of the class so it's an unambiguous literal and the escape is no
longer required. Behavior is identical; sanitizer tests still pass.

* revert(stack-activity): drop unvalidated E2E spec from this PR

The spec was committed without ever running against a real Docker
daemon, then failed in CI when it ran for the first time: deploy
returned 200 but no notification appeared on the activity endpoint
within the polling window, suggesting either a deploy-notification
race or a node-id resolution mismatch in the CI environment.

Backend unit tests (route + composite cursor + sanitizer) and
frontend component tests cover the same logic. The E2E spec will
land in a dedicated follow-up once it has been authored against a
working CI environment.
2026-05-25 21:09:00 -04:00

359 lines
11 KiB
TypeScript

/**
* Integration tests verifying that deploy_failure notifications are dispatched
* when stack action routes encounter errors.
*
* Covers: deploy, down, restart, stop, update
*
* ComposeService and DockerController are mocked so no real Docker daemon is
* required. NotificationService.dispatchAlert is spied on to assert dispatch.
*/
import { describe, it, expect, beforeAll, afterAll, vi, beforeEach } from 'vitest';
import request from 'supertest';
import jwt from 'jsonwebtoken';
import { setupTestDb, cleanupTestDb, loginAsTestAdmin, TEST_JWT_SECRET } from './helpers/setupTestDb';
import { ComposeRollbackError } from '../services/ComposeService';
// ── Hoisted mocks (must come before importing the app) ──────────────────────
const {
mockDeployStack,
mockRunCommand,
mockUpdateStack,
mockGetContainersByStack,
mockRestartContainer,
mockStopContainer,
mockListContainers,
mockIsTrivyAvailable,
mockGetImageDigest,
mockRunScanAndPersist,
} = vi.hoisted(() => ({
mockDeployStack: vi.fn(),
mockRunCommand: vi.fn(),
mockUpdateStack: vi.fn(),
mockGetContainersByStack: vi.fn(),
mockRestartContainer: vi.fn(),
mockStopContainer: vi.fn(),
mockListContainers: vi.fn(),
mockIsTrivyAvailable: vi.fn(),
mockGetImageDigest: vi.fn(),
mockRunScanAndPersist: vi.fn(),
}));
vi.mock('../services/ComposeService', async () => {
const actual = await vi.importActual<typeof import('../services/ComposeService')>(
'../services/ComposeService',
);
return {
...actual,
ComposeService: {
...actual.ComposeService,
getInstance: () => ({
deployStack: mockDeployStack,
runCommand: mockRunCommand,
updateStack: mockUpdateStack,
}),
},
};
});
vi.mock('../services/DockerController', async () => {
const actual = await vi.importActual<typeof import('../services/DockerController')>(
'../services/DockerController',
);
return {
...actual,
default: {
...actual.default,
getInstance: () => ({
getContainersByStack: mockGetContainersByStack,
restartContainer: mockRestartContainer,
stopContainer: mockStopContainer,
getDocker: () => ({
listContainers: mockListContainers,
}),
}),
},
};
});
vi.mock('../services/TrivyService', async () => {
const actual = await vi.importActual<typeof import('../services/TrivyService')>(
'../services/TrivyService',
);
return {
...actual,
default: {
...actual.default,
getInstance: () => ({
isTrivyAvailable: mockIsTrivyAvailable,
getImageDigest: mockGetImageDigest,
runScanAndPersist: mockRunScanAndPersist,
}),
},
};
});
vi.mock('../services/FileSystemService', () => ({
FileSystemService: {
getInstance: () => ({
getStacks: vi.fn().mockResolvedValue([]),
getBaseDir: () => '/tmp/compose',
readComposeFile: vi.fn().mockResolvedValue(''),
hasComposeFile: vi.fn().mockResolvedValue(true),
}),
},
}));
// ── Setup ───────────────────────────────────────────────────────────────────
let tmpDir: string;
let app: import('express').Express;
let authCookie: string;
let dispatchAlertSpy: ReturnType<typeof vi.spyOn>;
beforeAll(async () => {
tmpDir = await setupTestDb();
({ app } = await import('../index'));
authCookie = await loginAsTestAdmin(app);
const { NotificationService } = await import('../services/NotificationService');
dispatchAlertSpy = vi
.spyOn(NotificationService.getInstance(), 'dispatchAlert')
.mockResolvedValue(undefined);
});
afterAll(() => {
vi.restoreAllMocks();
cleanupTestDb(tmpDir);
});
beforeEach(() => {
mockDeployStack.mockReset();
mockRunCommand.mockReset();
mockUpdateStack.mockReset();
mockGetContainersByStack.mockReset();
mockRestartContainer.mockReset();
mockStopContainer.mockReset();
mockListContainers.mockReset();
mockIsTrivyAvailable.mockReset();
mockGetImageDigest.mockReset();
mockRunScanAndPersist.mockReset();
mockIsTrivyAvailable.mockReturnValue(true);
mockListContainers.mockResolvedValue([{ Image: 'nginx:latest' }]);
mockGetImageDigest.mockResolvedValue(null);
mockRunScanAndPersist.mockResolvedValue({
critical_count: 0,
high_count: 0,
});
dispatchAlertSpy.mockClear();
});
// ── Tests ───────────────────────────────────────────────────────────────────
describe('deploy_failure notification on /deploy error', () => {
it('dispatches deploy_failure alert with correct stackName when deployStack throws', async () => {
mockDeployStack.mockRejectedValue(new Error('image pull failed'));
const res = await request(app)
.post('/api/stacks/myapp/deploy')
.set('Cookie', authCookie);
expect(res.status).toBe(500);
await new Promise(resolve => setImmediate(resolve));
expect(dispatchAlertSpy).toHaveBeenCalledWith(
'error',
'deploy_failure',
expect.stringContaining('image pull failed'),
{ stackName: 'myapp', actor: 'testadmin' },
);
});
it('includes the error message in the dispatched alert', async () => {
mockDeployStack.mockRejectedValue(new Error('network timeout'));
await request(app)
.post('/api/stacks/webapp/deploy')
.set('Cookie', authCookie);
await new Promise(resolve => setImmediate(resolve));
const call = dispatchAlertSpy.mock.calls[0];
expect(call[0]).toBe('error');
expect(call[1]).toBe('deploy_failure');
expect(call[2]).toContain('network timeout');
expect(call[3]).toEqual({ stackName: 'webapp', actor: 'testadmin' });
});
it('returns rolledBack=true only when compose rollback completed', async () => {
mockDeployStack.mockRejectedValue(
new ComposeRollbackError(new Error('image pull failed'), true, true),
);
const res = await request(app)
.post('/api/stacks/myapp/deploy')
.set('Cookie', authCookie);
expect(res.status).toBe(500);
expect(res.body).toMatchObject({ rolledBack: true });
});
it('returns rolledBack=false when compose rollback failed', async () => {
mockDeployStack.mockRejectedValue(
new ComposeRollbackError(new Error('image pull failed'), true, false),
);
const res = await request(app)
.post('/api/stacks/myapp/deploy')
.set('Cookie', authCookie);
expect(res.status).toBe(500);
expect(res.body).toMatchObject({ rolledBack: false });
});
it('uses trusted proxy tier headers for remote atomic deploys', async () => {
mockDeployStack.mockResolvedValue(undefined);
const token = jwt.sign({ scope: 'node_proxy' }, TEST_JWT_SECRET, { expiresIn: '1m' });
const res = await request(app)
.post('/api/stacks/myapp/deploy')
.set('Authorization', `Bearer ${token}`)
.set('x-sencho-tier', 'paid')
.set('x-sencho-variant', 'skipper');
expect(res.status).toBe(200);
expect(mockDeployStack.mock.calls[0][2]).toBe(true);
});
});
describe('post-deploy scan opt-out', () => {
it('does not trigger a post-deploy scan when skip_scan is true', async () => {
mockDeployStack.mockResolvedValue(undefined);
const res = await request(app)
.post('/api/stacks/myapp/deploy')
.set('Cookie', authCookie)
.send({ skip_scan: true });
expect(res.status).toBe(200);
await new Promise(resolve => setImmediate(resolve));
expect(mockListContainers).not.toHaveBeenCalled();
expect(mockRunScanAndPersist).not.toHaveBeenCalled();
});
});
describe('deploy_failure notification on /down error', () => {
it('dispatches deploy_failure alert when runCommand (down) throws', async () => {
mockRunCommand.mockRejectedValue(new Error('container removal error'));
const res = await request(app)
.post('/api/stacks/myapp/down')
.set('Cookie', authCookie);
expect(res.status).toBe(500);
await new Promise(resolve => setImmediate(resolve));
expect(dispatchAlertSpy).toHaveBeenCalledWith(
'error',
'deploy_failure',
expect.any(String),
{ stackName: 'myapp', actor: 'testadmin' },
);
});
});
describe('deploy_failure notification on /restart error', () => {
it('dispatches deploy_failure alert when restartContainer throws', async () => {
mockGetContainersByStack.mockResolvedValue([{ Id: 'c1', Service: 'web' }]);
mockRestartContainer.mockRejectedValue(new Error('restart daemon error'));
const res = await request(app)
.post('/api/stacks/myapp/restart')
.set('Cookie', authCookie);
expect(res.status).toBe(500);
await new Promise(resolve => setImmediate(resolve));
expect(dispatchAlertSpy).toHaveBeenCalledWith(
'error',
'deploy_failure',
expect.stringContaining('restart daemon error'),
{ stackName: 'myapp', actor: 'testadmin' },
);
});
});
describe('deploy_failure notification on /stop error', () => {
it('dispatches deploy_failure alert when stopContainer throws', async () => {
mockGetContainersByStack.mockResolvedValue([{ Id: 'c1', Service: 'web' }]);
mockStopContainer.mockRejectedValue(new Error('stop daemon error'));
const res = await request(app)
.post('/api/stacks/myapp/stop')
.set('Cookie', authCookie);
expect(res.status).toBe(500);
await new Promise(resolve => setImmediate(resolve));
expect(dispatchAlertSpy).toHaveBeenCalledWith(
'error',
'deploy_failure',
expect.stringContaining('stop daemon error'),
{ stackName: 'myapp', actor: 'testadmin' },
);
});
});
describe('deploy_failure notification on /update error', () => {
it('dispatches deploy_failure alert with correct stackName when updateStack throws', async () => {
mockUpdateStack.mockRejectedValue(new Error('image not found'));
const res = await request(app)
.post('/api/stacks/myapp/update')
.set('Cookie', authCookie);
expect(res.status).toBe(500);
await new Promise(resolve => setImmediate(resolve));
expect(dispatchAlertSpy).toHaveBeenCalledWith(
'error',
'deploy_failure',
expect.stringContaining('image not found'),
{ stackName: 'myapp', actor: 'testadmin' },
);
});
it('returns rollback completion status when updateStack throws rollback metadata', async () => {
mockUpdateStack.mockRejectedValue(
new ComposeRollbackError(new Error('image not found'), true, false),
);
const res = await request(app)
.post('/api/stacks/myapp/update')
.set('Cookie', authCookie);
expect(res.status).toBe(500);
expect(res.body).toMatchObject({ rolledBack: false });
});
it('uses trusted proxy tier headers for remote atomic updates', async () => {
mockUpdateStack.mockResolvedValue(undefined);
const token = jwt.sign({ scope: 'node_proxy' }, TEST_JWT_SECRET, { expiresIn: '1m' });
const res = await request(app)
.post('/api/stacks/myapp/update')
.set('Authorization', `Bearer ${token}`)
.set('x-sencho-tier', 'paid')
.set('x-sencho-variant', 'skipper');
expect(res.status).toBe(200);
expect(mockUpdateStack.mock.calls[0][2]).toBe(true);
});
});