mirror of
https://github.com/Studio-Saelix/sencho.git
synced 2026-08-07 09:24:09 +00:00
fix(stack-activity): per-stack history integrity, attribution, sanitization (#1228)
* fix(stack-activity): per-stack history integrity, attribution, sanitization Address the Stack Activity audit findings (PR 1 of 2): - Per-stack history integrity: drop the per-insert 100-row prune in addNotificationHistory that evicted quieter stacks' history whenever another stack got chatty. Periodic cleanupOldNotifications now caps per (node, stack) at 500 rows and per-node unattached system events at 1000 rows, on top of the existing 30-day retention. Signature takes an options bag and returns a per-stage summary so MonitorService can log what actually ran each cycle. - Actor attribution: thread req.user?.username through every notifyActionFailure call site and add synthetic actors at service emit sites (system:autoheal, system:scheduler, system:image-update, system:docker-events, system:blueprint, system:monitor, system:policy). The timeline renders system actors as "via <Label>" so an autoheal redeploy is no longer indistinguishable from a user redeploy. - Message sanitization: new sanitizeNotificationMessage at NotificationService.dispatchAlert strips KEY=VALUE pairs whose key ends in TOKEN/KEY/PASSWORD/SECRET/CREDENTIALS/AUTH, scrubs HTTP basic auth in URLs and Bearer tokens, collapses COMPOSE_DIR paths, and truncates to 1000 chars. Applied to the stored history and to every downstream Discord/Slack/webhook channel. The ImageUpdateService recovery-path direct DB write also runs through the sanitizer. - Composite pagination cursor: getStackActivity now accepts a (timestamp, id) cursor (?before=&beforeId=). The legacy timestamp-only form silently dropped events when a single compose up emitted many events sharing one millisecond. Route rejects beforeId without before. - Frontend hardening: distinct error state with retry button (initial fetch failure no longer renders as the genuine empty state), strict positive-integer parsing on cursor params, overrequest-by-1 pagination so the last page does not leave a dead "Load more" click, runtime guard on liveEvents merge that validates the level union, per-minute day-bucket recompute so an open panel does not stay on "Today" past midnight. No tier, role, or capability gate touched. Route permission gate remains stack:read on the named stack. * fix(stack-activity): sanitizer covers lowercase env vars and per-node compose dir External review surfaced two leak paths in the message sanitizer: - The sensitive-key regex was uppercase-only. Compose env names are conventionally uppercase but lowercase forms (db_password, jwt_secret, github_token) are valid and do leak through the same Docker and compose-parse error paths. Make the regex case-insensitive and tighten it to also catch bare TOKEN= / KEY= / PASSWORD= without a prefix word, while still leaving BYPASS, COMPASS, and similar non-secret keys alone. - The compose-dir path collapse only read process.env.COMPOSE_DIR, but the real resolution chain is node.compose_dir (per-node DB override) -> process.env.COMPOSE_DIR -> /app/compose. A node with a custom compose_dir could still leak absolute paths into stored history and downstream channels. Route both the dispatchAlert call and the ImageUpdateService recovery-path direct write through NodeRegistry.getInstance().getComposeDir(localNodeId) so the collapse covers every resolution outcome. Tests now assert lowercase keys are redacted and that BYPASS-style non-secrets stay intact in both cases. notification-routing mock extended to stub the new getComposeDir call. * chore(stack-activity): a11y roles, visibility-aware tick, live-disconnect signal Close three small follow-ups on the per-stack activity timeline: - A11y: each day-group gets role="list" and each event row gets role="listitem" so screen readers traverse the timeline as a list instead of a wall of text. The day-group container also carries an aria-label naming the bucket. - Visibility-aware day-bucket tick: the 60s setInterval that re-derives Today/Yesterday/Earlier now short-circuits when document.hidden, so a backgrounded panel does not re-render every minute for no visible effect. - Live-disconnect signal: useNotifications dispatches a sencho:notifications-connection custom event on WebSocket open and close. The timeline listens and, when explicitly disconnected, shows a one-line "Live updates offline; reconnecting…" hint above the list. The sidebar ticker already surfaces fleet-wide connection state; this adds an in-context cue for users who are focused on a single stack. Stack-name case normalization was considered and rejected: stack names are case-permissive per the isValidStackName validator, and lowercasing on read or write would silently rename or hide a user's "MyApp" stack. * ci(stack-activity): drop unnecessary escape in URL_BASIC_AUTH regex ESLint no-useless-escape errored on \- inside the character class [a-zA-Z0-9+.\-] at notificationMessage.ts:14. Move the dash to the end of the class so it's an unambiguous literal and the escape is no longer required. Behavior is identical; sanitizer tests still pass. * revert(stack-activity): drop unvalidated E2E spec from this PR The spec was committed without ever running against a real Docker daemon, then failed in CI when it ran for the first time: deploy returned 200 but no notification appeared on the activity endpoint within the polling window, suggesting either a deploy-notification race or a node-id resolution mismatch in the CI environment. Backend unit tests (route + composite cursor + sanitizer) and frontend component tests cover the same logic. The E2E spec will land in a dedicated follow-up once it has been authored against a working CI environment.
This commit is contained in:
@@ -219,40 +219,96 @@ describe('DatabaseService - cleanupOldAuditLogs', () => {
|
||||
});
|
||||
});
|
||||
|
||||
describe('DatabaseService - notification history cap', () => {
|
||||
it('auto-prunes to 100 entries when adding notifications', () => {
|
||||
// Insert 105 notifications
|
||||
for (let i = 0; i < 105; i++) {
|
||||
db.addNotificationHistory(0, {
|
||||
level: 'info',
|
||||
message: `cap-test-${i}`,
|
||||
timestamp: Date.now() + i,
|
||||
});
|
||||
}
|
||||
|
||||
// The table should have at most 100 rows
|
||||
const all = db.getNotificationHistory(0, 200);
|
||||
expect(all.length).toBeLessThanOrEqual(100);
|
||||
});
|
||||
|
||||
it('keeps the most recent entries after pruning', () => {
|
||||
// Clear all first
|
||||
describe('DatabaseService - notification history cap (periodic)', () => {
|
||||
it('does not prune on insert; periodic cleanup caps per (node, stack)', () => {
|
||||
db.deleteAllNotifications(0);
|
||||
|
||||
for (let i = 0; i < 105; i++) {
|
||||
// A chatty stack writes 600 events.
|
||||
const base = Date.now();
|
||||
for (let i = 0; i < 600; i++) {
|
||||
db.addNotificationHistory(0, {
|
||||
level: 'info',
|
||||
message: `order-test-${i}`,
|
||||
timestamp: Date.now() + i * 10,
|
||||
message: `chatty-${i}`,
|
||||
timestamp: base + i,
|
||||
stack_name: 'chatty',
|
||||
});
|
||||
}
|
||||
// A quiet stack writes 3 events long before the chatty burst.
|
||||
for (let i = 0; i < 3; i++) {
|
||||
db.addNotificationHistory(0, {
|
||||
level: 'info',
|
||||
message: `quiet-${i}`,
|
||||
timestamp: base - 10_000 + i,
|
||||
stack_name: 'quiet',
|
||||
});
|
||||
}
|
||||
|
||||
const all = db.getNotificationHistory(0, 200);
|
||||
// The newest entries should survive (ordered DESC by timestamp)
|
||||
expect(all[0].message).toContain('order-test-');
|
||||
// The oldest entries (0-4) should have been pruned
|
||||
const oldest = all.find((n: any) => n.message === 'order-test-0');
|
||||
expect(oldest).toBeUndefined();
|
||||
// No per-insert prune: every row is present.
|
||||
const beforeCleanup = db.getNotificationHistory(0, 2000);
|
||||
expect(beforeCleanup.length).toBe(603);
|
||||
|
||||
db.cleanupOldNotifications(30, { perStackCap: 500, perNodeUnattachedCap: 1000 });
|
||||
|
||||
const after = db.getNotificationHistory(0, 2000);
|
||||
const chatty = after.filter((n: any) => n.stack_name === 'chatty');
|
||||
const quiet = after.filter((n: any) => n.stack_name === 'quiet');
|
||||
expect(chatty.length).toBe(500);
|
||||
// Quiet stack is untouched even though chatty is far noisier.
|
||||
expect(quiet.length).toBe(3);
|
||||
});
|
||||
|
||||
it('caps per-node events without a stack_name', () => {
|
||||
db.deleteAllNotifications(0);
|
||||
|
||||
const base = Date.now();
|
||||
for (let i = 0; i < 1200; i++) {
|
||||
db.addNotificationHistory(0, {
|
||||
level: 'info',
|
||||
message: `system-${i}`,
|
||||
timestamp: base + i,
|
||||
});
|
||||
}
|
||||
|
||||
db.cleanupOldNotifications(30, { perStackCap: 500, perNodeUnattachedCap: 1000 });
|
||||
|
||||
const all = db.getNotificationHistory(0, 2000);
|
||||
const unattached = all.filter((n: any) => !n.stack_name);
|
||||
expect(unattached.length).toBe(1000);
|
||||
});
|
||||
|
||||
it('keeps the newest entries per (node, stack) after periodic cap', () => {
|
||||
db.deleteAllNotifications(0);
|
||||
const base = Date.now();
|
||||
for (let i = 0; i < 600; i++) {
|
||||
db.addNotificationHistory(0, {
|
||||
level: 'info',
|
||||
message: `ordered-${i}`,
|
||||
timestamp: base + i * 10,
|
||||
stack_name: 'ordered',
|
||||
});
|
||||
}
|
||||
|
||||
db.cleanupOldNotifications(30, { perStackCap: 500, perNodeUnattachedCap: 1000 });
|
||||
|
||||
const after = db.getNotificationHistory(0, 2000);
|
||||
const ordered = after.filter((n: any) => n.stack_name === 'ordered');
|
||||
expect(ordered.length).toBe(500);
|
||||
// Newest 500 survive; oldest 100 are gone.
|
||||
expect(ordered.find((n: any) => n.message === 'ordered-0')).toBeUndefined();
|
||||
expect(ordered.find((n: any) => n.message === 'ordered-599')).toBeDefined();
|
||||
});
|
||||
|
||||
it('uses safe defaults when called with only the retention argument', () => {
|
||||
db.deleteAllNotifications(0);
|
||||
const base = Date.now();
|
||||
for (let i = 0; i < 600; i++) {
|
||||
db.addNotificationHistory(0, { level: 'info', message: `d-${i}`, timestamp: base + i, stack_name: 'default' });
|
||||
}
|
||||
// Production caller (MonitorService) only passes daysToKeep; the cap defaults must enforce the per-stack 500 limit.
|
||||
const summary = db.cleanupOldNotifications(30);
|
||||
const after = db.getNotificationHistory(0, 2000).filter((n: any) => n.stack_name === 'default');
|
||||
expect(after.length).toBe(500);
|
||||
expect(summary.perStack).toBe(100);
|
||||
});
|
||||
});
|
||||
|
||||
|
||||
@@ -389,7 +389,7 @@ services:
|
||||
'info',
|
||||
'image_update_available',
|
||||
expect.stringContaining('stackA'),
|
||||
{ stackName: 'stackA' },
|
||||
{ stackName: 'stackA', actor: 'system:image-update' },
|
||||
);
|
||||
expect(mockUpsertStackUpdateStatus).toHaveBeenCalledWith(1, 'stackA', true, expect.any(Number));
|
||||
});
|
||||
|
||||
@@ -716,7 +716,7 @@ describe('MonitorService - breach state machine', () => {
|
||||
const svc = MonitorService.getInstance();
|
||||
await (svc as any).evaluate();
|
||||
|
||||
expect(mockDispatchAlert).toHaveBeenCalledWith('warning', 'monitor_alert', expect.stringContaining('CPU'), { stackName: 'my-stack' });
|
||||
expect(mockDispatchAlert).toHaveBeenCalledWith('warning', 'monitor_alert', expect.stringContaining('CPU'), { stackName: 'my-stack', actor: 'system:monitor' });
|
||||
expect(mockUpdateStackAlertLastFired).toHaveBeenCalledWith(1, expect.any(Number));
|
||||
});
|
||||
|
||||
@@ -890,7 +890,7 @@ describe('MonitorService - restart_count metric', () => {
|
||||
|
||||
expect(mockGetContainerRestartCount).toHaveBeenCalledWith('c1');
|
||||
// restart_count=5 > threshold=3, should fire
|
||||
expect(mockDispatchAlert).toHaveBeenCalledWith('warning', 'monitor_alert', expect.stringContaining('Restart count'), { stackName: 'my-stack' });
|
||||
expect(mockDispatchAlert).toHaveBeenCalledWith('warning', 'monitor_alert', expect.stringContaining('Restart count'), { stackName: 'my-stack', actor: 'system:monitor' });
|
||||
});
|
||||
|
||||
it('skips Docker inspect when no restart_count rules exist', async () => {
|
||||
@@ -1283,7 +1283,7 @@ describe('MonitorService - janitor cycle and circuit breaker', () => {
|
||||
await (svc as any).evaluateJanitor();
|
||||
|
||||
expect(mockDispatchAlert).toHaveBeenCalledWith(
|
||||
'info', 'system', expect.stringContaining('3.0 GB'), { stackName: undefined },
|
||||
'info', 'system', expect.stringContaining('3.0 GB'), { stackName: undefined, actor: 'system:monitor' },
|
||||
);
|
||||
});
|
||||
|
||||
|
||||
@@ -45,6 +45,7 @@ vi.mock('../services/NodeRegistry', () => ({
|
||||
NodeRegistry: {
|
||||
getInstance: () => ({
|
||||
getDefaultNodeId: () => 1,
|
||||
getComposeDir: () => '/app/compose',
|
||||
}),
|
||||
},
|
||||
}));
|
||||
|
||||
@@ -919,7 +919,7 @@ describe('SchedulerService - error handling', () => {
|
||||
const svc = SchedulerService.getInstance();
|
||||
await svc.triggerTask(91);
|
||||
|
||||
expect(mockDispatchAlert).toHaveBeenCalledWith('error', 'system', expect.stringContaining('failed'), { stackName: undefined });
|
||||
expect(mockDispatchAlert).toHaveBeenCalledWith('error', 'system', expect.stringContaining('failed'), { stackName: undefined, actor: 'system:scheduler' });
|
||||
});
|
||||
|
||||
it('dispatches recovery notification when previous status was failure', async () => {
|
||||
@@ -939,7 +939,7 @@ describe('SchedulerService - error handling', () => {
|
||||
const svc = SchedulerService.getInstance();
|
||||
await svc.triggerTask(92);
|
||||
|
||||
expect(mockDispatchAlert).toHaveBeenCalledWith('info', 'system', expect.stringContaining('recovered'), { stackName: 'my-stack' });
|
||||
expect(mockDispatchAlert).toHaveBeenCalledWith('info', 'system', expect.stringContaining('recovered'), { stackName: 'my-stack', actor: 'system:scheduler' });
|
||||
});
|
||||
});
|
||||
|
||||
@@ -1005,13 +1005,13 @@ describe('SchedulerService - scheduled scan notifications', () => {
|
||||
'info',
|
||||
'scan_finding',
|
||||
expect.stringContaining('nightly-scan'),
|
||||
{ stackName: undefined },
|
||||
{ stackName: undefined, actor: 'system:scheduler' },
|
||||
);
|
||||
expect(mockDispatchAlert).toHaveBeenCalledWith(
|
||||
'info',
|
||||
'scan_finding',
|
||||
expect.stringContaining('Scanned 3 image(s)'),
|
||||
{ stackName: undefined },
|
||||
{ stackName: undefined, actor: 'system:scheduler' },
|
||||
);
|
||||
});
|
||||
|
||||
@@ -1026,7 +1026,7 @@ describe('SchedulerService - scheduled scan notifications', () => {
|
||||
'warning',
|
||||
'scan_finding',
|
||||
expect.stringContaining('2 failed'),
|
||||
{ stackName: undefined },
|
||||
{ stackName: undefined, actor: 'system:scheduler' },
|
||||
);
|
||||
});
|
||||
|
||||
@@ -1041,7 +1041,7 @@ describe('SchedulerService - scheduled scan notifications', () => {
|
||||
'info',
|
||||
'scan_finding',
|
||||
expect.stringContaining('completed'),
|
||||
{ stackName: 'web-stack' },
|
||||
{ stackName: 'web-stack', actor: 'system:scheduler' },
|
||||
);
|
||||
});
|
||||
|
||||
@@ -1061,7 +1061,7 @@ describe('SchedulerService - scheduled scan notifications', () => {
|
||||
'info',
|
||||
'scan_finding',
|
||||
expect.stringContaining('recovered-scan'),
|
||||
{ stackName: undefined },
|
||||
{ stackName: undefined, actor: 'system:scheduler' },
|
||||
);
|
||||
});
|
||||
|
||||
@@ -1096,7 +1096,7 @@ describe('SchedulerService - scheduled scan notifications', () => {
|
||||
'error',
|
||||
'system',
|
||||
expect.stringMatching(/failed.*Trivy/i),
|
||||
{ stackName: 'payment-stack' },
|
||||
{ stackName: 'payment-stack', actor: 'system:scheduler' },
|
||||
);
|
||||
});
|
||||
|
||||
@@ -1145,7 +1145,7 @@ describe('SchedulerService - scheduled scan notifications', () => {
|
||||
'info',
|
||||
'scan_finding',
|
||||
expect.stringContaining('No images to scan'),
|
||||
{ stackName: undefined },
|
||||
{ stackName: undefined, actor: 'system:scheduler' },
|
||||
);
|
||||
});
|
||||
|
||||
@@ -1160,7 +1160,7 @@ describe('SchedulerService - scheduled scan notifications', () => {
|
||||
'info',
|
||||
'scan_finding',
|
||||
expect.stringContaining('All 12 image(s) already scanned recently'),
|
||||
{ stackName: undefined },
|
||||
{ stackName: undefined, actor: 'system:scheduler' },
|
||||
);
|
||||
});
|
||||
|
||||
@@ -1300,7 +1300,7 @@ describe('SchedulerService - invalid cron at execution time', () => {
|
||||
'error',
|
||||
'system',
|
||||
expect.stringContaining('failed'),
|
||||
{ stackName: undefined },
|
||||
{ stackName: undefined, actor: 'system:scheduler' },
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -0,0 +1,169 @@
|
||||
import { describe, it, expect, beforeAll, afterAll, beforeEach } from 'vitest';
|
||||
import { setupTestDb, cleanupTestDb } from './helpers/setupTestDb';
|
||||
import { sanitizeNotificationMessage } from '../utils/notificationMessage';
|
||||
|
||||
let tmpDir: string;
|
||||
let db: any;
|
||||
|
||||
beforeAll(async () => {
|
||||
tmpDir = await setupTestDb();
|
||||
const { DatabaseService } = await import('../services/DatabaseService');
|
||||
db = DatabaseService.getInstance();
|
||||
});
|
||||
|
||||
afterAll(() => {
|
||||
cleanupTestDb(tmpDir);
|
||||
});
|
||||
|
||||
beforeEach(() => {
|
||||
db.deleteAllNotifications(0);
|
||||
});
|
||||
|
||||
describe('sanitizeNotificationMessage', () => {
|
||||
it('passes plain messages through unchanged', () => {
|
||||
expect(sanitizeNotificationMessage('Stack deployed in 4.2s')).toBe('Stack deployed in 4.2s');
|
||||
});
|
||||
|
||||
it('redacts KEY=VALUE pairs whose key matches a sensitive suffix', () => {
|
||||
const raw = 'compose parse failed: DB_PASSWORD=hunter2 missing closing quote';
|
||||
expect(sanitizeNotificationMessage(raw)).toBe('compose parse failed: DB_PASSWORD=<redacted> missing closing quote');
|
||||
});
|
||||
|
||||
it('redacts API_KEY, SECRET, TOKEN, CREDENTIAL variants', () => {
|
||||
expect(sanitizeNotificationMessage('STRIPE_API_KEY=sk_live_abc')).toContain('STRIPE_API_KEY=<redacted>');
|
||||
expect(sanitizeNotificationMessage('JWT_SECRET=eyJabc')).toContain('JWT_SECRET=<redacted>');
|
||||
expect(sanitizeNotificationMessage('GITHUB_TOKEN=ghp_abc')).toContain('GITHUB_TOKEN=<redacted>');
|
||||
expect(sanitizeNotificationMessage('AWS_CREDENTIALS=foo')).toContain('AWS_CREDENTIALS=<redacted>');
|
||||
});
|
||||
|
||||
it('redacts every sensitive KEY=VALUE pair in a single message', () => {
|
||||
const raw = 'compose env: DB_PASSWORD=hunter2 API_KEY=abc OTHER=keep';
|
||||
const out = sanitizeNotificationMessage(raw);
|
||||
expect(out).toContain('DB_PASSWORD=<redacted>');
|
||||
expect(out).toContain('API_KEY=<redacted>');
|
||||
expect(out).toContain('OTHER=keep');
|
||||
});
|
||||
|
||||
it('leaves bare PASS-suffix keys like BYPASS and COMPASS unredacted', () => {
|
||||
expect(sanitizeNotificationMessage('BYPASS=true')).toBe('BYPASS=true');
|
||||
expect(sanitizeNotificationMessage('COMPASS_URL=https://example.com')).toBe('COMPASS_URL=https://example.com');
|
||||
});
|
||||
|
||||
it('leaves non-sensitive uppercase KEY=VALUE untouched', () => {
|
||||
expect(sanitizeNotificationMessage('NODE_ENV=production')).toBe('NODE_ENV=production');
|
||||
expect(sanitizeNotificationMessage('PORT=1852')).toBe('PORT=1852');
|
||||
});
|
||||
|
||||
it('redacts lowercase sensitive keys (compose env vars are commonly lowercase)', () => {
|
||||
expect(sanitizeNotificationMessage('db_password=foo')).toBe('db_password=<redacted>');
|
||||
expect(sanitizeNotificationMessage('jwt_secret=abc')).toBe('jwt_secret=<redacted>');
|
||||
expect(sanitizeNotificationMessage('github_token=ghp_xyz')).toBe('github_token=<redacted>');
|
||||
});
|
||||
|
||||
it('still leaves bare PASS-suffix lowercase keys unredacted', () => {
|
||||
expect(sanitizeNotificationMessage('bypass=true')).toBe('bypass=true');
|
||||
expect(sanitizeNotificationMessage('compass_url=https://example.com')).toBe('compass_url=https://example.com');
|
||||
});
|
||||
|
||||
it('redacts HTTP basic auth in URLs', () => {
|
||||
const raw = 'pull failed for https://user:pa55@registry.example.com/img';
|
||||
expect(sanitizeNotificationMessage(raw)).toBe('pull failed for https://user:<redacted>@registry.example.com/img');
|
||||
});
|
||||
|
||||
it('redacts bearer tokens', () => {
|
||||
const raw = 'auth header was Bearer abc123xyzdef456';
|
||||
expect(sanitizeNotificationMessage(raw)).toBe('auth header was Bearer <redacted>');
|
||||
});
|
||||
|
||||
it('truncates messages longer than 1000 characters', () => {
|
||||
const raw = 'x'.repeat(2000);
|
||||
const out = sanitizeNotificationMessage(raw);
|
||||
expect(out.length).toBeLessThanOrEqual(1000);
|
||||
expect(out.endsWith('… [truncated]')).toBe(true);
|
||||
});
|
||||
|
||||
it('collapses COMPOSE_DIR path prefixes', () => {
|
||||
const out = sanitizeNotificationMessage(
|
||||
'file not found: /opt/docker/sencho/compose/myapp/.env',
|
||||
{ composeDir: '/opt/docker/sencho/compose' },
|
||||
);
|
||||
expect(out).toBe('file not found: <compose-dir>/myapp/.env');
|
||||
});
|
||||
|
||||
});
|
||||
|
||||
describe('DatabaseService.getStackActivity', () => {
|
||||
it('returns only events for the requested (node, stack)', () => {
|
||||
const base = Date.now();
|
||||
db.addNotificationHistory(0, { level: 'info', message: 'a-evt', timestamp: base, stack_name: 'a' });
|
||||
db.addNotificationHistory(0, { level: 'info', message: 'b-evt', timestamp: base + 1, stack_name: 'b' });
|
||||
db.addNotificationHistory(0, { level: 'info', message: 'a-evt-2', timestamp: base + 2, stack_name: 'a' });
|
||||
|
||||
const aOnly = db.getStackActivity(0, 'a', { limit: 50 });
|
||||
expect(aOnly.map((e: any) => e.message)).toEqual(['a-evt-2', 'a-evt']);
|
||||
});
|
||||
|
||||
it('honors limit and orders newest first', () => {
|
||||
const base = Date.now();
|
||||
for (let i = 0; i < 5; i++) {
|
||||
db.addNotificationHistory(0, { level: 'info', message: `e-${i}`, timestamp: base + i, stack_name: 's' });
|
||||
}
|
||||
const out = db.getStackActivity(0, 's', { limit: 3 });
|
||||
expect(out.length).toBe(3);
|
||||
expect(out.map((e: any) => e.message)).toEqual(['e-4', 'e-3', 'e-2']);
|
||||
});
|
||||
|
||||
it('legacy timestamp-only cursor excludes equal-or-newer rows', () => {
|
||||
const base = Date.now();
|
||||
for (let i = 0; i < 5; i++) {
|
||||
db.addNotificationHistory(0, { level: 'info', message: `e-${i}`, timestamp: base + i * 10, stack_name: 's' });
|
||||
}
|
||||
// Cursor at base+20 (= e-2). before=base+20 means timestamp < base+20, so only e-0/e-1 returned.
|
||||
const out = db.getStackActivity(0, 's', { limit: 50, before: base + 20 });
|
||||
expect(out.map((e: any) => e.message)).toEqual(['e-1', 'e-0']);
|
||||
});
|
||||
|
||||
it('composite (timestamp, id) cursor drops only the cursor row and older when many share a ms', () => {
|
||||
const ts = Date.now();
|
||||
const ids: number[] = [];
|
||||
for (let i = 0; i < 5; i++) {
|
||||
const row = db.addNotificationHistory(0, { level: 'info', message: `e-${i}`, timestamp: ts, stack_name: 's' });
|
||||
ids.push(row.id);
|
||||
}
|
||||
// ids[0..4] all share ts. Cursor at the third row's id should return ids[0] and ids[1].
|
||||
const out = db.getStackActivity(0, 's', { limit: 50, before: ts, beforeId: ids[2] });
|
||||
const returnedIds = out.map((e: any) => e.id);
|
||||
expect(returnedIds).toEqual([ids[1], ids[0]]);
|
||||
});
|
||||
|
||||
it('documents the legacy timestamp-only cursor drops same-ms rows (kept for backward compat)', () => {
|
||||
const ts = Date.now();
|
||||
for (let i = 0; i < 5; i++) {
|
||||
db.addNotificationHistory(0, { level: 'info', message: `e-${i}`, timestamp: ts, stack_name: 's' });
|
||||
}
|
||||
// The composite-cursor test above proves the fix; this test pins the legacy form's
|
||||
// behavior so a client that omits beforeId gets a predictable (if lossy) result.
|
||||
const out = db.getStackActivity(0, 's', { limit: 50, before: ts });
|
||||
expect(out).toEqual([]);
|
||||
});
|
||||
|
||||
it('returns empty array when stack has no events', () => {
|
||||
db.addNotificationHistory(0, { level: 'info', message: 'other', timestamp: Date.now(), stack_name: 'other' });
|
||||
const out = db.getStackActivity(0, 'missing', { limit: 50 });
|
||||
expect(out).toEqual([]);
|
||||
});
|
||||
});
|
||||
|
||||
describe('DatabaseService.addNotificationHistory (no per-insert prune)', () => {
|
||||
it('keeps a quiet stack visible even after a chatty stack writes past the old 100-row per-node cap', () => {
|
||||
const ts = Date.now();
|
||||
db.addNotificationHistory(0, { level: 'info', message: 'first', timestamp: ts, stack_name: 'a' });
|
||||
// The old per-insert prune kept only the newest 100 rows per node, regardless of stack.
|
||||
// Writing 150 rows for stack b would have evicted 'first' from stack a under the old rule.
|
||||
for (let i = 0; i < 150; i++) {
|
||||
db.addNotificationHistory(0, { level: 'info', message: `chatty-${i}`, timestamp: ts + i + 1, stack_name: 'b' });
|
||||
}
|
||||
const aActivity = db.getStackActivity(0, 'a', { limit: 50 });
|
||||
expect(aActivity.map((e: any) => e.message)).toEqual(['first']);
|
||||
});
|
||||
});
|
||||
@@ -166,7 +166,7 @@ describe('deploy_failure notification on /deploy error', () => {
|
||||
'error',
|
||||
'deploy_failure',
|
||||
expect.stringContaining('image pull failed'),
|
||||
{ stackName: 'myapp' },
|
||||
{ stackName: 'myapp', actor: 'testadmin' },
|
||||
);
|
||||
});
|
||||
|
||||
@@ -183,7 +183,7 @@ describe('deploy_failure notification on /deploy error', () => {
|
||||
expect(call[0]).toBe('error');
|
||||
expect(call[1]).toBe('deploy_failure');
|
||||
expect(call[2]).toContain('network timeout');
|
||||
expect(call[3]).toEqual({ stackName: 'webapp' });
|
||||
expect(call[3]).toEqual({ stackName: 'webapp', actor: 'testadmin' });
|
||||
});
|
||||
|
||||
it('returns rolledBack=true only when compose rollback completed', async () => {
|
||||
@@ -260,7 +260,7 @@ describe('deploy_failure notification on /down error', () => {
|
||||
'error',
|
||||
'deploy_failure',
|
||||
expect.any(String),
|
||||
{ stackName: 'myapp' },
|
||||
{ stackName: 'myapp', actor: 'testadmin' },
|
||||
);
|
||||
});
|
||||
});
|
||||
@@ -282,7 +282,7 @@ describe('deploy_failure notification on /restart error', () => {
|
||||
'error',
|
||||
'deploy_failure',
|
||||
expect.stringContaining('restart daemon error'),
|
||||
{ stackName: 'myapp' },
|
||||
{ stackName: 'myapp', actor: 'testadmin' },
|
||||
);
|
||||
});
|
||||
});
|
||||
@@ -304,7 +304,7 @@ describe('deploy_failure notification on /stop error', () => {
|
||||
'error',
|
||||
'deploy_failure',
|
||||
expect.stringContaining('stop daemon error'),
|
||||
{ stackName: 'myapp' },
|
||||
{ stackName: 'myapp', actor: 'testadmin' },
|
||||
);
|
||||
});
|
||||
});
|
||||
@@ -325,7 +325,7 @@ describe('deploy_failure notification on /update error', () => {
|
||||
'error',
|
||||
'deploy_failure',
|
||||
expect.stringContaining('image not found'),
|
||||
{ stackName: 'myapp' },
|
||||
{ stackName: 'myapp', actor: 'testadmin' },
|
||||
);
|
||||
});
|
||||
|
||||
|
||||
@@ -122,7 +122,7 @@ export async function triggerPostDeployScan(
|
||||
scan.critical_count > 0 ? 'error' : 'warning',
|
||||
'scan_finding',
|
||||
`Vulnerability scan for ${imageRef}: ${scan.critical_count} critical, ${scan.high_count} high`,
|
||||
{ stackName },
|
||||
{ stackName, actor: 'system:policy' },
|
||||
);
|
||||
}
|
||||
} catch (err) {
|
||||
@@ -133,7 +133,7 @@ export async function triggerPostDeployScan(
|
||||
'warning',
|
||||
'scan_finding',
|
||||
`Post-deploy scan failed for ${imageRef} (${stackName}): ${message}`,
|
||||
{ stackName },
|
||||
{ stackName, actor: 'system:policy' },
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -294,7 +294,7 @@ autoUpdateRouter.post('/execute', authMiddleware, async (req: Request, res: Resp
|
||||
);
|
||||
if (!autoUpdateGate.ok) {
|
||||
const blockedMsg = `Policy "${autoUpdateGate.policy?.name}" blocked auto-update: ${autoUpdateGate.violations.length} image(s) exceed ${autoUpdateGate.policy?.max_severity}`;
|
||||
NotificationService.getInstance().dispatchAlert('warning', 'scan_finding', blockedMsg, { stackName });
|
||||
NotificationService.getInstance().dispatchAlert('warning', 'scan_finding', blockedMsg, { stackName, actor: 'system:image-update' });
|
||||
results.push(`Stack "${stackName}": ${blockedMsg}`);
|
||||
continue;
|
||||
}
|
||||
@@ -315,7 +315,7 @@ autoUpdateRouter.post('/execute', authMiddleware, async (req: Request, res: Resp
|
||||
'info',
|
||||
'image_update_applied',
|
||||
`Auto-update: stack "${stackName}" updated with new images`,
|
||||
{ stackName },
|
||||
{ stackName, actor: 'system:image-update' },
|
||||
);
|
||||
|
||||
results.push(`Stack "${stackName}": updated (${updatedImages.join(', ')}).`);
|
||||
|
||||
@@ -5,6 +5,14 @@ import { isValidStackName } from '../utils/validation';
|
||||
|
||||
export const stackActivityRouter = Router();
|
||||
|
||||
function parseStrictPositiveInt(raw: unknown): number | null {
|
||||
if (raw === undefined || raw === null) return null;
|
||||
const s = String(raw).trim();
|
||||
if (s === '' || !/^\d+$/.test(s)) return null;
|
||||
const n = Number(s);
|
||||
return Number.isFinite(n) && n >= 1 ? n : null;
|
||||
}
|
||||
|
||||
stackActivityRouter.get('/:stackName/activity', (req: Request, res: Response): void => {
|
||||
const stackName = req.params.stackName as string;
|
||||
if (!isValidStackName(stackName)) {
|
||||
@@ -12,12 +20,43 @@ stackActivityRouter.get('/:stackName/activity', (req: Request, res: Response): v
|
||||
return;
|
||||
}
|
||||
if (!requirePermission(req, res, 'stack:read', 'stack', stackName)) return;
|
||||
const limit = Math.min(parseInt(String(req.query.limit ?? '50'), 10) || 50, 200);
|
||||
const before = req.query.before ? parseInt(String(req.query.before), 10) : undefined;
|
||||
if (before !== undefined && isNaN(before)) {
|
||||
res.status(400).json({ error: 'Invalid before parameter' });
|
||||
|
||||
const parsedLimit = parseStrictPositiveInt(req.query.limit ?? '50');
|
||||
if (parsedLimit === null) {
|
||||
res.status(400).json({ error: 'Invalid limit parameter' });
|
||||
return;
|
||||
}
|
||||
const events = DatabaseService.getInstance().getStackActivity(req.nodeId, stackName, { limit, before });
|
||||
const limit = Math.min(parsedLimit, 200);
|
||||
|
||||
const hasBefore = req.query.before !== undefined;
|
||||
const hasBeforeId = req.query.beforeId !== undefined;
|
||||
// beforeId without before would silently fall back to "page 1" in the DB
|
||||
// layer; reject so a paginating client cannot loop on the same page.
|
||||
if (hasBeforeId && !hasBefore) {
|
||||
res.status(400).json({ error: 'beforeId requires before' });
|
||||
return;
|
||||
}
|
||||
|
||||
let before: number | undefined;
|
||||
if (hasBefore) {
|
||||
const parsed = parseStrictPositiveInt(req.query.before);
|
||||
if (parsed === null) {
|
||||
res.status(400).json({ error: 'Invalid before parameter' });
|
||||
return;
|
||||
}
|
||||
before = parsed;
|
||||
}
|
||||
|
||||
let beforeId: number | undefined;
|
||||
if (hasBeforeId) {
|
||||
const parsed = parseStrictPositiveInt(req.query.beforeId);
|
||||
if (parsed === null) {
|
||||
res.status(400).json({ error: 'Invalid beforeId parameter' });
|
||||
return;
|
||||
}
|
||||
beforeId = parsed;
|
||||
}
|
||||
|
||||
const events = DatabaseService.getInstance().getStackActivity(req.nodeId, stackName, { limit, before, beforeId });
|
||||
res.json({ events });
|
||||
});
|
||||
|
||||
@@ -41,10 +41,10 @@ const MAX_COMPOSE_PARSE_BYTES = 1_048_576; // 1 MiB
|
||||
function dlog(...args: Parameters<typeof console.log>): void {
|
||||
if (isDebugEnabled()) console.log(...args);
|
||||
}
|
||||
function notifyActionFailure(action: string, stackName: string, error: unknown): void {
|
||||
function notifyActionFailure(action: string, stackName: string, error: unknown, actor: string): void {
|
||||
const message = getErrorMessage(error, `Failed to ${action} stack`);
|
||||
NotificationService.getInstance()
|
||||
.dispatchAlert('error', 'deploy_failure', message, { stackName })
|
||||
.dispatchAlert('error', 'deploy_failure', message, { stackName, actor })
|
||||
.catch(err => console.error('[Stacks] Failed to dispatch failure notification for %s:', sanitizeForLog(stackName), err));
|
||||
}
|
||||
|
||||
@@ -341,7 +341,7 @@ async function runStackBulkOp(
|
||||
return { stackName, ok: false, error: 'No containers found for this stack', code: 'no_containers' };
|
||||
}
|
||||
if (outcome.kind === 'error') {
|
||||
if (action !== 'start') notifyActionFailure(action, stackName, new Error(outcome.message));
|
||||
if (action !== 'start') notifyActionFailure(action, stackName, new Error(outcome.message), user);
|
||||
return { stackName, ok: false, error: outcome.message, code: 'op_failed' };
|
||||
}
|
||||
const meta = CONTAINER_ACTION_META[action];
|
||||
@@ -349,7 +349,7 @@ async function runStackBulkOp(
|
||||
}
|
||||
return { stackName, ok: true };
|
||||
} catch (err) {
|
||||
if (action !== 'start') notifyActionFailure(action, stackName, err);
|
||||
if (action !== 'start') notifyActionFailure(action, stackName, err, user);
|
||||
return { stackName, ok: false, error: getErrorMessage(err, `${action} failed`), code: 'op_failed' };
|
||||
} finally {
|
||||
StackOpLockService.getInstance().release(req.nodeId, stackName);
|
||||
@@ -962,7 +962,7 @@ stacksRouter.post('/:stackName/deploy', async (req: Request, res: Response) => {
|
||||
console.warn('[Stacks] Deploy failed, rollback did not complete: %s', sanitizeForLog(stackName));
|
||||
}
|
||||
const message = getErrorMessage(error, 'Failed to deploy stack');
|
||||
notifyActionFailure('deploy', stackName, error);
|
||||
notifyActionFailure('deploy', stackName, error, req.user?.username ?? 'system');
|
||||
if (!res.headersSent) {
|
||||
if (isDockerUnavailableError(error)) {
|
||||
res.status(503).json({ error: message, code: 'docker_unavailable', rolledBack });
|
||||
@@ -993,7 +993,7 @@ stacksRouter.post('/:stackName/down', async (req: Request, res: Response) => {
|
||||
res.json({ status: 'Command started' });
|
||||
} catch (error: unknown) {
|
||||
console.error('[Stacks] Down failed: %s', sanitizeForLog(stackName), error);
|
||||
notifyActionFailure('down', stackName, error);
|
||||
notifyActionFailure('down', stackName, error, req.user?.username ?? 'system');
|
||||
if (!res.headersSent) {
|
||||
if (isDockerUnavailableError(error)) {
|
||||
res.status(503).json({ error: getErrorMessage(error, 'Docker daemon is unreachable'), code: 'docker_unavailable' });
|
||||
@@ -1088,13 +1088,13 @@ async function bulkContainerOp(
|
||||
}
|
||||
if (outcome.kind === 'docker-unavailable') {
|
||||
console.error('[Stacks] %s failed: docker unavailable for %s', sanitizeForLog(titleCase), sanitizeForLog(stackName));
|
||||
if (action !== 'start') notifyActionFailure(action, stackName, new Error(outcome.message));
|
||||
if (action !== 'start') notifyActionFailure(action, stackName, new Error(outcome.message), req.user?.username ?? 'system');
|
||||
res.status(503).json({ error: outcome.message, code: 'docker_unavailable' });
|
||||
return;
|
||||
}
|
||||
if (outcome.kind === 'error') {
|
||||
console.error('[Stacks] %s failed: %s %s', sanitizeForLog(titleCase), sanitizeForLog(stackName), sanitizeForLog(outcome.message));
|
||||
if (action !== 'start') notifyActionFailure(action, stackName, new Error(outcome.message));
|
||||
if (action !== 'start') notifyActionFailure(action, stackName, new Error(outcome.message), req.user?.username ?? 'system');
|
||||
res.status(500).json({ error: outcome.message });
|
||||
return;
|
||||
}
|
||||
@@ -1230,7 +1230,7 @@ stacksRouter.post('/:stackName/update', async (req: Request, res: Response) => {
|
||||
} else if (rollbackInfo?.attempted) {
|
||||
console.warn(`[Stacks] Update failed, rollback did not complete: ${sanitizeForLog(stackName)}`);
|
||||
}
|
||||
notifyActionFailure('update', stackName, error);
|
||||
notifyActionFailure('update', stackName, error, req.user?.username ?? 'system');
|
||||
if (!res.headersSent) {
|
||||
if (isDockerUnavailableError(error)) {
|
||||
res.status(503).json({ error: getErrorMessage(error, 'Docker daemon is unreachable'), code: 'docker_unavailable', rolledBack });
|
||||
|
||||
@@ -327,7 +327,7 @@ export class AutoHealService {
|
||||
'info',
|
||||
'autoheal_triggered',
|
||||
`Auto-Heal: Restarted ${containerName} on stack ${policy.stack_name} after being unhealthy for ${policy.unhealthy_duration_mins} minute(s).`,
|
||||
{ stackName: policy.stack_name, containerName },
|
||||
{ stackName: policy.stack_name, containerName, actor: 'system:autoheal' },
|
||||
)
|
||||
.catch(err => console.error('[AutoHeal] notification dispatch failed:', err));
|
||||
} catch (err) {
|
||||
@@ -351,7 +351,7 @@ export class AutoHealService {
|
||||
'warning',
|
||||
'autoheal_triggered',
|
||||
`Auto-Heal: Failed to restart ${containerName} on stack ${policy.stack_name}. Error: ${errorMsg}`,
|
||||
{ stackName: policy.stack_name, containerName },
|
||||
{ stackName: policy.stack_name, containerName, actor: 'system:autoheal' },
|
||||
)
|
||||
.catch(e => console.error('[AutoHeal] notification dispatch failed:', e));
|
||||
|
||||
@@ -384,7 +384,7 @@ export class AutoHealService {
|
||||
'warning',
|
||||
'autoheal_triggered',
|
||||
`Auto-Heal: Policy for ${policy.stack_name}${policy.service_name ? '/' + policy.service_name : ''} has been auto-disabled after ${failures} consecutive failures.`,
|
||||
{ stackName: policy.stack_name },
|
||||
{ stackName: policy.stack_name, actor: 'system:autoheal' },
|
||||
)
|
||||
.catch(e => console.error('[AutoHeal] notification dispatch failed:', e));
|
||||
}
|
||||
|
||||
@@ -288,7 +288,7 @@ export class BlueprintReconciler {
|
||||
'warning',
|
||||
'blueprint_drift_detected',
|
||||
`Blueprint "${blueprint.name}" drifted on node "${node.name}": ${reason}`,
|
||||
{ stackName: blueprint.name },
|
||||
{ stackName: blueprint.name, actor: 'system:blueprint' },
|
||||
);
|
||||
return;
|
||||
|
||||
@@ -302,7 +302,7 @@ export class BlueprintReconciler {
|
||||
'warning',
|
||||
'blueprint_drift_detected',
|
||||
`Blueprint "${blueprint.name}" lost its marker on node "${node.name}"; auto-fix declined to avoid stomping unowned data. Reason: ${reason}`,
|
||||
{ stackName: blueprint.name },
|
||||
{ stackName: blueprint.name, actor: 'system:blueprint' },
|
||||
);
|
||||
return;
|
||||
}
|
||||
@@ -319,7 +319,7 @@ export class BlueprintReconciler {
|
||||
'error',
|
||||
'blueprint_drift_correction_failed',
|
||||
`Auto-fix for "${blueprint.name}" on node "${node.name}" failed: ${result.error ?? 'unknown error'}`,
|
||||
{ stackName: blueprint.name },
|
||||
{ stackName: blueprint.name, actor: 'system:blueprint' },
|
||||
);
|
||||
}
|
||||
return;
|
||||
|
||||
@@ -2040,13 +2040,6 @@ export class DatabaseService {
|
||||
notification.actor_username ?? null,
|
||||
);
|
||||
|
||||
this.db.prepare(`
|
||||
DELETE FROM notification_history
|
||||
WHERE node_id = ? AND id NOT IN (
|
||||
SELECT id FROM notification_history WHERE node_id = ? ORDER BY timestamp DESC LIMIT 100
|
||||
)
|
||||
`).run(nodeId, nodeId);
|
||||
|
||||
return {
|
||||
id: result.lastInsertRowid as number,
|
||||
level: notification.level,
|
||||
@@ -2060,13 +2053,21 @@ export class DatabaseService {
|
||||
};
|
||||
}
|
||||
|
||||
public getStackActivity(nodeId: number, stackName: string, opts: { limit: number; before?: number }): NotificationHistory[] {
|
||||
const sql = opts.before
|
||||
? 'SELECT * FROM notification_history WHERE node_id = ? AND stack_name = ? AND timestamp < ? ORDER BY timestamp DESC LIMIT ?'
|
||||
: 'SELECT * FROM notification_history WHERE node_id = ? AND stack_name = ? ORDER BY timestamp DESC LIMIT ?';
|
||||
const args: (number | string)[] = opts.before
|
||||
? [nodeId, stackName, opts.before, opts.limit]
|
||||
: [nodeId, stackName, opts.limit];
|
||||
public getStackActivity(nodeId: number, stackName: string, opts: { limit: number; before?: number; beforeId?: number }): NotificationHistory[] {
|
||||
// Composite (timestamp, id) cursor: pure timestamp pagination drops rows
|
||||
// on same-millisecond bursts (Docker events from one compose up).
|
||||
let sql: string;
|
||||
let args: (number | string)[];
|
||||
if (opts.before !== undefined && opts.beforeId !== undefined) {
|
||||
sql = 'SELECT * FROM notification_history WHERE node_id = ? AND stack_name = ? AND (timestamp < ? OR (timestamp = ? AND id < ?)) ORDER BY timestamp DESC, id DESC LIMIT ?';
|
||||
args = [nodeId, stackName, opts.before, opts.before, opts.beforeId, opts.limit];
|
||||
} else if (opts.before !== undefined) {
|
||||
sql = 'SELECT * FROM notification_history WHERE node_id = ? AND stack_name = ? AND timestamp < ? ORDER BY timestamp DESC, id DESC LIMIT ?';
|
||||
args = [nodeId, stackName, opts.before, opts.limit];
|
||||
} else {
|
||||
sql = 'SELECT * FROM notification_history WHERE node_id = ? AND stack_name = ? ORDER BY timestamp DESC, id DESC LIMIT ?';
|
||||
args = [nodeId, stackName, opts.limit];
|
||||
}
|
||||
return (this.db.prepare(sql).all(...args) as unknown[]).map(row => this.mapNotificationRow(row as any));
|
||||
}
|
||||
|
||||
@@ -2157,9 +2158,50 @@ export class DatabaseService {
|
||||
stmt.run(cutoff);
|
||||
}
|
||||
|
||||
public cleanupOldNotifications(daysToKeep = 30): void {
|
||||
public cleanupOldNotifications(daysToKeep = 30, opts: { perStackCap?: number; perNodeUnattachedCap?: number } = {}): { ttl: number; perStack: number; perNode: number } {
|
||||
const perStackCap = opts.perStackCap ?? 500;
|
||||
const perNodeUnattachedCap = opts.perNodeUnattachedCap ?? 1000;
|
||||
const cutoff = Date.now() - (daysToKeep * 24 * 60 * 60 * 1000);
|
||||
this.db.prepare('DELETE FROM notification_history WHERE timestamp < ?').run(cutoff);
|
||||
const ttlInfo = this.db.prepare('DELETE FROM notification_history WHERE timestamp < ?').run(cutoff);
|
||||
|
||||
const deleteById = this.db.prepare('DELETE FROM notification_history WHERE id = ?');
|
||||
const deleteMany = this.db.transaction((ids: number[]) => {
|
||||
for (const id of ids) deleteById.run(id);
|
||||
});
|
||||
|
||||
// Per (node_id, stack_name) cap so a chatty stack cannot evict a quieter stack's history.
|
||||
const stackOverflow = this.db.prepare(`
|
||||
SELECT id FROM (
|
||||
SELECT id, ROW_NUMBER() OVER (
|
||||
PARTITION BY node_id, stack_name
|
||||
ORDER BY timestamp DESC, id DESC
|
||||
) AS rn
|
||||
FROM notification_history
|
||||
WHERE stack_name IS NOT NULL
|
||||
)
|
||||
WHERE rn > ?
|
||||
`).all(perStackCap) as { id: number }[];
|
||||
if (stackOverflow.length > 0) deleteMany(stackOverflow.map(r => r.id));
|
||||
|
||||
// Unattached system events have no stack to scope by, so they cannot share the per-stack quota; cap them per-node separately.
|
||||
const unattachedOverflow = this.db.prepare(`
|
||||
SELECT id FROM (
|
||||
SELECT id, ROW_NUMBER() OVER (
|
||||
PARTITION BY node_id
|
||||
ORDER BY timestamp DESC, id DESC
|
||||
) AS rn
|
||||
FROM notification_history
|
||||
WHERE stack_name IS NULL
|
||||
)
|
||||
WHERE rn > ?
|
||||
`).all(perNodeUnattachedCap) as { id: number }[];
|
||||
if (unattachedOverflow.length > 0) deleteMany(unattachedOverflow.map(r => r.id));
|
||||
|
||||
return {
|
||||
ttl: Number(ttlInfo.changes ?? 0),
|
||||
perStack: stackOverflow.length,
|
||||
perNode: unattachedOverflow.length,
|
||||
};
|
||||
}
|
||||
|
||||
// --- Nodes ---
|
||||
|
||||
@@ -673,15 +673,15 @@ export class DockerEventService {
|
||||
// ========================================================================
|
||||
|
||||
private async emitError(category: NotificationCategory, message: string, stackName?: string, containerName?: string): Promise<void> {
|
||||
return this.notifier.dispatchAlert('error', category, this.prefix(message), { stackName, containerName });
|
||||
return this.notifier.dispatchAlert('error', category, this.prefix(message), { stackName, containerName, actor: 'system:docker-events' });
|
||||
}
|
||||
|
||||
private async emitWarning(category: NotificationCategory, message: string, stackName?: string, containerName?: string): Promise<void> {
|
||||
return this.notifier.dispatchAlert('warning', category, this.prefix(message), { stackName, containerName });
|
||||
return this.notifier.dispatchAlert('warning', category, this.prefix(message), { stackName, containerName, actor: 'system:docker-events' });
|
||||
}
|
||||
|
||||
private async emitInfo(category: NotificationCategory, message: string, stackName?: string, containerName?: string): Promise<void> {
|
||||
return this.notifier.dispatchAlert('info', category, this.prefix(message), { stackName, containerName });
|
||||
return this.notifier.dispatchAlert('info', category, this.prefix(message), { stackName, containerName, actor: 'system:docker-events' });
|
||||
}
|
||||
|
||||
private prefix(message: string): string {
|
||||
|
||||
@@ -6,6 +6,7 @@ import { FileSystemService } from './FileSystemService';
|
||||
import { RegistryService } from './RegistryService';
|
||||
import { NodeRegistry } from './NodeRegistry';
|
||||
import { NotificationService } from './NotificationService';
|
||||
import { sanitizeNotificationMessage } from '../utils/notificationMessage';
|
||||
import { parseImageRef, getRemoteDigest } from './registry-api';
|
||||
import { isDebugEnabled } from '../utils/debug';
|
||||
import { getErrorMessage } from '../utils/errors';
|
||||
@@ -308,7 +309,7 @@ export class ImageUpdateService {
|
||||
'info',
|
||||
'image_update_available',
|
||||
`[Node: ${nodeName}] Stack "${stackName}" has image updates available.`,
|
||||
{ stackName },
|
||||
{ stackName, actor: 'system:image-update' },
|
||||
);
|
||||
} catch (e) {
|
||||
console.error(`[ImageUpdateService] Failed to dispatch update notification for "${stackName}":`, e);
|
||||
@@ -316,11 +317,16 @@ export class ImageUpdateService {
|
||||
// Key on the local default: the iterated `nodeId` may be a remote's id in the
|
||||
// control plane's DB, and the UI never queries that row (it proxies instead).
|
||||
try {
|
||||
db.addNotificationHistory(NodeRegistry.getInstance().getDefaultNodeId(), {
|
||||
const localNodeId = NodeRegistry.getInstance().getDefaultNodeId();
|
||||
db.addNotificationHistory(localNodeId, {
|
||||
level: 'error',
|
||||
category: 'system',
|
||||
message: `[Node: ${nodeName}] Failed to notify about image updates for stack "${stackName}": ${getErrorMessage(e, String(e))}`,
|
||||
message: sanitizeNotificationMessage(
|
||||
`[Node: ${nodeName}] Failed to notify about image updates for stack "${stackName}": ${getErrorMessage(e, String(e))}`,
|
||||
{ composeDir: NodeRegistry.getInstance().getComposeDir(localNodeId) },
|
||||
),
|
||||
timestamp: Date.now(),
|
||||
actor_username: 'system:image-update',
|
||||
});
|
||||
} catch (dbErr) {
|
||||
console.error('[ImageUpdateService] Failed to record dispatch error:', dbErr);
|
||||
|
||||
@@ -548,10 +548,10 @@ export class MonitorService {
|
||||
const retentionHours = parseInt(settings['metrics_retention_hours'] || '24', 10);
|
||||
db.cleanupOldMetrics(isNaN(retentionHours) ? 24 : retentionHours);
|
||||
const retentionDays = parseInt(settings['log_retention_days'] || '30', 10);
|
||||
db.cleanupOldNotifications(isNaN(retentionDays) ? 30 : retentionDays);
|
||||
const notifSummary = db.cleanupOldNotifications(isNaN(retentionDays) ? 30 : retentionDays);
|
||||
const auditRetentionDays = parseInt(settings['audit_retention_days'] || '90', 10);
|
||||
db.cleanupOldAuditLogs(isNaN(auditRetentionDays) ? 90 : auditRetentionDays);
|
||||
if (isDebugEnabled()) console.log(`[Monitor:diag] Cleanup: metrics ${isNaN(retentionHours) ? 24 : retentionHours}h, notifications ${isNaN(retentionDays) ? 30 : retentionDays}d, audit ${isNaN(auditRetentionDays) ? 90 : auditRetentionDays}d`);
|
||||
if (isDebugEnabled()) console.log(`[Monitor:diag] Cleanup: metrics ${isNaN(retentionHours) ? 24 : retentionHours}h, notifications ${isNaN(retentionDays) ? 30 : retentionDays}d (ttl=${notifSummary.ttl} perStack=${notifSummary.perStack} perNode=${notifSummary.perNode}), audit ${isNaN(auditRetentionDays) ? 90 : auditRetentionDays}d`);
|
||||
} catch (e) {
|
||||
console.error('MonitorService: failed to cleanup old data', e);
|
||||
}
|
||||
@@ -670,7 +670,7 @@ export class MonitorService {
|
||||
'warning',
|
||||
'monitor_alert',
|
||||
message,
|
||||
{ stackName: rule.stack_name },
|
||||
{ stackName: rule.stack_name, actor: 'system:monitor' },
|
||||
);
|
||||
|
||||
db.updateStackAlertLastFired(ruleId, Date.now());
|
||||
@@ -723,7 +723,7 @@ export class MonitorService {
|
||||
const db = DatabaseService.getInstance();
|
||||
const last = parseInt(db.getSystemState(stateKey) || '0', 10);
|
||||
if (Date.now() - last > cooldownMs) {
|
||||
await NotificationService.getInstance().dispatchAlert(severity, category, message, { stackName: stack });
|
||||
await NotificationService.getInstance().dispatchAlert(severity, category, message, { stackName: stack, actor: 'system:monitor' });
|
||||
db.setSystemState(stateKey, Date.now().toString());
|
||||
}
|
||||
}
|
||||
|
||||
@@ -4,6 +4,7 @@ import { NodeRegistry } from './NodeRegistry';
|
||||
import { isDebugEnabled } from '../utils/debug';
|
||||
import { getErrorMessage } from '../utils/errors';
|
||||
import { sanitizeForLog } from '../utils/safeLog';
|
||||
import { sanitizeNotificationMessage } from '../utils/notificationMessage';
|
||||
|
||||
export type NotificationCategory =
|
||||
| 'deploy_success'
|
||||
@@ -119,10 +120,15 @@ export class NotificationService {
|
||||
// with user-initiated requests; otherwise the UI and monitors split
|
||||
// between different node_id buckets.
|
||||
const localNodeId = NodeRegistry.getInstance().getDefaultNodeId();
|
||||
// Use the full resolution chain (node.compose_dir → env → default)
|
||||
// so messages mentioning a per-node compose override get collapsed.
|
||||
const sanitized = sanitizeNotificationMessage(message, {
|
||||
composeDir: NodeRegistry.getInstance().getComposeDir(localNodeId),
|
||||
});
|
||||
const notification = this.dbService.addNotificationHistory(localNodeId, {
|
||||
level,
|
||||
category,
|
||||
message,
|
||||
message: sanitized,
|
||||
timestamp: Date.now(),
|
||||
stack_name: stackName,
|
||||
container_name: containerName,
|
||||
@@ -151,7 +157,7 @@ export class NotificationService {
|
||||
if (isDebugEnabled()) console.log(`[Notify:diag] Matched ${matched.length} route(s) for stack "${sanitizeForLog(stackName ?? '(none)')}", category="${sanitizeForLog(category)}"`);
|
||||
await Promise.allSettled(
|
||||
matched.map(route =>
|
||||
this.sendToChannel(route.channel_type, route.channel_url, level, message)
|
||||
this.sendToChannel(route.channel_type, route.channel_url, level, sanitized)
|
||||
.then(() => {
|
||||
if (isDebugEnabled()) console.log(`[Notify:diag] Dispatched ${level} via route "${route.name}" (${route.channel_type})`);
|
||||
})
|
||||
@@ -176,7 +182,7 @@ export class NotificationService {
|
||||
if (isDebugEnabled()) console.log(`[Notify:diag] Falling back to ${agents.length} global agent(s)`);
|
||||
await Promise.allSettled(
|
||||
agents.map(agent =>
|
||||
this.sendToChannel(agent.type, agent.url, level, message)
|
||||
this.sendToChannel(agent.type, agent.url, level, sanitized)
|
||||
.then(() => {
|
||||
if (isDebugEnabled()) console.log(`[Notify:diag] Dispatched ${level} via global agent (${agent.type})`);
|
||||
})
|
||||
|
||||
@@ -66,7 +66,7 @@ function notifyTrivyMissingOnce(nodeId: number, stackName: string): void {
|
||||
'warning',
|
||||
'scan_finding',
|
||||
`Pre-deploy scan for "${stackName}" skipped: Trivy not installed on this node`,
|
||||
{ stackName },
|
||||
{ stackName, actor: 'system:policy' },
|
||||
);
|
||||
}
|
||||
|
||||
|
||||
@@ -167,7 +167,7 @@ export class SchedulerService {
|
||||
*/
|
||||
private safeDispatch(level: 'info' | 'warning' | 'error', category: import('./NotificationService').NotificationCategory, message: string, stackName?: string): void {
|
||||
NotificationService.getInstance()
|
||||
.dispatchAlert(level, category, message, { stackName })
|
||||
.dispatchAlert(level, category, message, { stackName, actor: 'system:scheduler' })
|
||||
.catch(err => console.error('[SchedulerService] Notification dispatch failed:', getErrorMessage(err, 'unknown error')));
|
||||
}
|
||||
|
||||
@@ -778,6 +778,7 @@ export class SchedulerService {
|
||||
'warning',
|
||||
'scan_finding',
|
||||
`Policy "${v.policyName}" violated by ${v.imageRef}: ${v.severity} exceeds ${v.maxSeverity}`,
|
||||
{ actor: 'system:scheduler' },
|
||||
);
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,35 @@
|
||||
const MAX_MESSAGE_CHARS = 1000;
|
||||
// U+2026 ellipsis (one codepoint), not three ASCII dots; tests assert this exact suffix.
|
||||
const TRUNCATION_SUFFIX = '… [truncated]';
|
||||
|
||||
// KEY=VALUE pairs whose KEY ends in a sensitive suffix. Values are stripped
|
||||
// because compose-parse errors and docker run failures surface them verbatim.
|
||||
// Case-insensitive: compose env names are conventionally uppercase but
|
||||
// lowercase forms (db_password, jwt_secret, github_token) are valid and do
|
||||
// leak through the same error paths. PASSWORD covers the bare-PASS case;
|
||||
// standalone PASS would over-redact BYPASS, COMPASS, and similar non-secret
|
||||
// keys.
|
||||
const SENSITIVE_KEY_PATTERN = /\b([A-Za-z0-9_]*(?:TOKEN|KEY|PASSWORD|SECRET|CREDENTIALS?|AUTH))\s*=\s*("[^"]*"|'[^']*'|\S+)/gi;
|
||||
|
||||
const URL_BASIC_AUTH = /\b([a-zA-Z][a-zA-Z0-9+.-]*:\/\/)([^/\s:@]+):([^/\s:@]+)@/g;
|
||||
const BEARER_TOKEN = /\b(Bearer)\s+([A-Za-z0-9._\-+/=]{8,})/g;
|
||||
|
||||
export function sanitizeNotificationMessage(raw: string, opts: { composeDir?: string } = {}): string {
|
||||
if (!raw) return raw;
|
||||
let s = raw;
|
||||
|
||||
s = s.replace(SENSITIVE_KEY_PATTERN, (_m, key) => `${key}=<redacted>`);
|
||||
s = s.replace(URL_BASIC_AUTH, (_m, scheme, user) => `${scheme}${user}:<redacted>@`);
|
||||
s = s.replace(BEARER_TOKEN, (_m, kw) => `${kw} <redacted>`);
|
||||
|
||||
if (opts.composeDir) {
|
||||
const escaped = opts.composeDir.replace(/[.*+?^${}()|[\]\\]/g, '\\$&');
|
||||
s = s.replace(new RegExp(escaped, 'g'), '<compose-dir>');
|
||||
}
|
||||
|
||||
if (s.length > MAX_MESSAGE_CHARS) {
|
||||
s = s.slice(0, MAX_MESSAGE_CHARS - TRUNCATION_SUFFIX.length) + TRUNCATION_SUFFIX;
|
||||
}
|
||||
|
||||
return s;
|
||||
}
|
||||
Reference in New Issue
Block a user