fix(stack-activity): per-stack history integrity, attribution, sanitization (#1228)

* fix(stack-activity): per-stack history integrity, attribution, sanitization

Address the Stack Activity audit findings (PR 1 of 2):

- Per-stack history integrity: drop the per-insert 100-row prune in
  addNotificationHistory that evicted quieter stacks' history whenever
  another stack got chatty. Periodic cleanupOldNotifications now caps
  per (node, stack) at 500 rows and per-node unattached system events
  at 1000 rows, on top of the existing 30-day retention. Signature
  takes an options bag and returns a per-stage summary so MonitorService
  can log what actually ran each cycle.

- Actor attribution: thread req.user?.username through every
  notifyActionFailure call site and add synthetic actors at service
  emit sites (system:autoheal, system:scheduler, system:image-update,
  system:docker-events, system:blueprint, system:monitor, system:policy).
  The timeline renders system actors as "via <Label>" so an autoheal
  redeploy is no longer indistinguishable from a user redeploy.

- Message sanitization: new sanitizeNotificationMessage at
  NotificationService.dispatchAlert strips KEY=VALUE pairs whose key
  ends in TOKEN/KEY/PASSWORD/SECRET/CREDENTIALS/AUTH, scrubs HTTP basic
  auth in URLs and Bearer tokens, collapses COMPOSE_DIR paths, and
  truncates to 1000 chars. Applied to the stored history and to every
  downstream Discord/Slack/webhook channel. The ImageUpdateService
  recovery-path direct DB write also runs through the sanitizer.

- Composite pagination cursor: getStackActivity now accepts a
  (timestamp, id) cursor (?before=&beforeId=). The legacy timestamp-only
  form silently dropped events when a single compose up emitted many
  events sharing one millisecond. Route rejects beforeId without before.

- Frontend hardening: distinct error state with retry button (initial
  fetch failure no longer renders as the genuine empty state), strict
  positive-integer parsing on cursor params, overrequest-by-1 pagination
  so the last page does not leave a dead "Load more" click, runtime
  guard on liveEvents merge that validates the level union, per-minute
  day-bucket recompute so an open panel does not stay on "Today" past
  midnight.

No tier, role, or capability gate touched. Route permission gate
remains stack:read on the named stack.

* fix(stack-activity): sanitizer covers lowercase env vars and per-node compose dir

External review surfaced two leak paths in the message sanitizer:

- The sensitive-key regex was uppercase-only. Compose env names are
  conventionally uppercase but lowercase forms (db_password, jwt_secret,
  github_token) are valid and do leak through the same Docker and
  compose-parse error paths. Make the regex case-insensitive and tighten
  it to also catch bare TOKEN= / KEY= / PASSWORD= without a prefix word,
  while still leaving BYPASS, COMPASS, and similar non-secret keys alone.

- The compose-dir path collapse only read process.env.COMPOSE_DIR, but
  the real resolution chain is node.compose_dir (per-node DB override)
  -> process.env.COMPOSE_DIR -> /app/compose. A node with a custom
  compose_dir could still leak absolute paths into stored history and
  downstream channels. Route both the dispatchAlert call and the
  ImageUpdateService recovery-path direct write through
  NodeRegistry.getInstance().getComposeDir(localNodeId) so the
  collapse covers every resolution outcome.

Tests now assert lowercase keys are redacted and that BYPASS-style
non-secrets stay intact in both cases. notification-routing mock
extended to stub the new getComposeDir call.

* chore(stack-activity): a11y roles, visibility-aware tick, live-disconnect signal

Close three small follow-ups on the per-stack activity timeline:

- A11y: each day-group gets role="list" and each event row gets
  role="listitem" so screen readers traverse the timeline as a list
  instead of a wall of text. The day-group container also carries an
  aria-label naming the bucket.

- Visibility-aware day-bucket tick: the 60s setInterval that re-derives
  Today/Yesterday/Earlier now short-circuits when document.hidden, so a
  backgrounded panel does not re-render every minute for no visible
  effect.

- Live-disconnect signal: useNotifications dispatches a
  sencho:notifications-connection custom event on WebSocket open and
  close. The timeline listens and, when explicitly disconnected, shows
  a one-line "Live updates offline; reconnecting…" hint above the list.
  The sidebar ticker already surfaces fleet-wide connection state; this
  adds an in-context cue for users who are focused on a single stack.

Stack-name case normalization was considered and rejected: stack names
are case-permissive per the isValidStackName validator, and lowercasing
on read or write would silently rename or hide a user's "MyApp" stack.

* ci(stack-activity): drop unnecessary escape in URL_BASIC_AUTH regex

ESLint no-useless-escape errored on \- inside the character class
[a-zA-Z0-9+.\-] at notificationMessage.ts:14. Move the dash to the
end of the class so it's an unambiguous literal and the escape is no
longer required. Behavior is identical; sanitizer tests still pass.

* revert(stack-activity): drop unvalidated E2E spec from this PR

The spec was committed without ever running against a real Docker
daemon, then failed in CI when it ran for the first time: deploy
returned 200 but no notification appeared on the activity endpoint
within the polling window, suggesting either a deploy-notification
race or a node-id resolution mismatch in the CI environment.

Backend unit tests (route + composite cursor + sanitizer) and
frontend component tests cover the same logic. The E2E spec will
land in a dedicated follow-up once it has been authored against a
working CI environment.
This commit is contained in:
Anso
2026-05-25 21:09:00 -04:00
committed by GitHub
parent 117f590332
commit 2d56ea958a
24 changed files with 852 additions and 133 deletions
+83 -27
View File
@@ -219,40 +219,96 @@ describe('DatabaseService - cleanupOldAuditLogs', () => {
});
});
describe('DatabaseService - notification history cap', () => {
it('auto-prunes to 100 entries when adding notifications', () => {
// Insert 105 notifications
for (let i = 0; i < 105; i++) {
db.addNotificationHistory(0, {
level: 'info',
message: `cap-test-${i}`,
timestamp: Date.now() + i,
});
}
// The table should have at most 100 rows
const all = db.getNotificationHistory(0, 200);
expect(all.length).toBeLessThanOrEqual(100);
});
it('keeps the most recent entries after pruning', () => {
// Clear all first
describe('DatabaseService - notification history cap (periodic)', () => {
it('does not prune on insert; periodic cleanup caps per (node, stack)', () => {
db.deleteAllNotifications(0);
for (let i = 0; i < 105; i++) {
// A chatty stack writes 600 events.
const base = Date.now();
for (let i = 0; i < 600; i++) {
db.addNotificationHistory(0, {
level: 'info',
message: `order-test-${i}`,
timestamp: Date.now() + i * 10,
message: `chatty-${i}`,
timestamp: base + i,
stack_name: 'chatty',
});
}
// A quiet stack writes 3 events long before the chatty burst.
for (let i = 0; i < 3; i++) {
db.addNotificationHistory(0, {
level: 'info',
message: `quiet-${i}`,
timestamp: base - 10_000 + i,
stack_name: 'quiet',
});
}
const all = db.getNotificationHistory(0, 200);
// The newest entries should survive (ordered DESC by timestamp)
expect(all[0].message).toContain('order-test-');
// The oldest entries (0-4) should have been pruned
const oldest = all.find((n: any) => n.message === 'order-test-0');
expect(oldest).toBeUndefined();
// No per-insert prune: every row is present.
const beforeCleanup = db.getNotificationHistory(0, 2000);
expect(beforeCleanup.length).toBe(603);
db.cleanupOldNotifications(30, { perStackCap: 500, perNodeUnattachedCap: 1000 });
const after = db.getNotificationHistory(0, 2000);
const chatty = after.filter((n: any) => n.stack_name === 'chatty');
const quiet = after.filter((n: any) => n.stack_name === 'quiet');
expect(chatty.length).toBe(500);
// Quiet stack is untouched even though chatty is far noisier.
expect(quiet.length).toBe(3);
});
it('caps per-node events without a stack_name', () => {
db.deleteAllNotifications(0);
const base = Date.now();
for (let i = 0; i < 1200; i++) {
db.addNotificationHistory(0, {
level: 'info',
message: `system-${i}`,
timestamp: base + i,
});
}
db.cleanupOldNotifications(30, { perStackCap: 500, perNodeUnattachedCap: 1000 });
const all = db.getNotificationHistory(0, 2000);
const unattached = all.filter((n: any) => !n.stack_name);
expect(unattached.length).toBe(1000);
});
it('keeps the newest entries per (node, stack) after periodic cap', () => {
db.deleteAllNotifications(0);
const base = Date.now();
for (let i = 0; i < 600; i++) {
db.addNotificationHistory(0, {
level: 'info',
message: `ordered-${i}`,
timestamp: base + i * 10,
stack_name: 'ordered',
});
}
db.cleanupOldNotifications(30, { perStackCap: 500, perNodeUnattachedCap: 1000 });
const after = db.getNotificationHistory(0, 2000);
const ordered = after.filter((n: any) => n.stack_name === 'ordered');
expect(ordered.length).toBe(500);
// Newest 500 survive; oldest 100 are gone.
expect(ordered.find((n: any) => n.message === 'ordered-0')).toBeUndefined();
expect(ordered.find((n: any) => n.message === 'ordered-599')).toBeDefined();
});
it('uses safe defaults when called with only the retention argument', () => {
db.deleteAllNotifications(0);
const base = Date.now();
for (let i = 0; i < 600; i++) {
db.addNotificationHistory(0, { level: 'info', message: `d-${i}`, timestamp: base + i, stack_name: 'default' });
}
// Production caller (MonitorService) only passes daysToKeep; the cap defaults must enforce the per-stack 500 limit.
const summary = db.cleanupOldNotifications(30);
const after = db.getNotificationHistory(0, 2000).filter((n: any) => n.stack_name === 'default');
expect(after.length).toBe(500);
expect(summary.perStack).toBe(100);
});
});
@@ -389,7 +389,7 @@ services:
'info',
'image_update_available',
expect.stringContaining('stackA'),
{ stackName: 'stackA' },
{ stackName: 'stackA', actor: 'system:image-update' },
);
expect(mockUpsertStackUpdateStatus).toHaveBeenCalledWith(1, 'stackA', true, expect.any(Number));
});
@@ -716,7 +716,7 @@ describe('MonitorService - breach state machine', () => {
const svc = MonitorService.getInstance();
await (svc as any).evaluate();
expect(mockDispatchAlert).toHaveBeenCalledWith('warning', 'monitor_alert', expect.stringContaining('CPU'), { stackName: 'my-stack' });
expect(mockDispatchAlert).toHaveBeenCalledWith('warning', 'monitor_alert', expect.stringContaining('CPU'), { stackName: 'my-stack', actor: 'system:monitor' });
expect(mockUpdateStackAlertLastFired).toHaveBeenCalledWith(1, expect.any(Number));
});
@@ -890,7 +890,7 @@ describe('MonitorService - restart_count metric', () => {
expect(mockGetContainerRestartCount).toHaveBeenCalledWith('c1');
// restart_count=5 > threshold=3, should fire
expect(mockDispatchAlert).toHaveBeenCalledWith('warning', 'monitor_alert', expect.stringContaining('Restart count'), { stackName: 'my-stack' });
expect(mockDispatchAlert).toHaveBeenCalledWith('warning', 'monitor_alert', expect.stringContaining('Restart count'), { stackName: 'my-stack', actor: 'system:monitor' });
});
it('skips Docker inspect when no restart_count rules exist', async () => {
@@ -1283,7 +1283,7 @@ describe('MonitorService - janitor cycle and circuit breaker', () => {
await (svc as any).evaluateJanitor();
expect(mockDispatchAlert).toHaveBeenCalledWith(
'info', 'system', expect.stringContaining('3.0 GB'), { stackName: undefined },
'info', 'system', expect.stringContaining('3.0 GB'), { stackName: undefined, actor: 'system:monitor' },
);
});
@@ -45,6 +45,7 @@ vi.mock('../services/NodeRegistry', () => ({
NodeRegistry: {
getInstance: () => ({
getDefaultNodeId: () => 1,
getComposeDir: () => '/app/compose',
}),
},
}));
+11 -11
View File
@@ -919,7 +919,7 @@ describe('SchedulerService - error handling', () => {
const svc = SchedulerService.getInstance();
await svc.triggerTask(91);
expect(mockDispatchAlert).toHaveBeenCalledWith('error', 'system', expect.stringContaining('failed'), { stackName: undefined });
expect(mockDispatchAlert).toHaveBeenCalledWith('error', 'system', expect.stringContaining('failed'), { stackName: undefined, actor: 'system:scheduler' });
});
it('dispatches recovery notification when previous status was failure', async () => {
@@ -939,7 +939,7 @@ describe('SchedulerService - error handling', () => {
const svc = SchedulerService.getInstance();
await svc.triggerTask(92);
expect(mockDispatchAlert).toHaveBeenCalledWith('info', 'system', expect.stringContaining('recovered'), { stackName: 'my-stack' });
expect(mockDispatchAlert).toHaveBeenCalledWith('info', 'system', expect.stringContaining('recovered'), { stackName: 'my-stack', actor: 'system:scheduler' });
});
});
@@ -1005,13 +1005,13 @@ describe('SchedulerService - scheduled scan notifications', () => {
'info',
'scan_finding',
expect.stringContaining('nightly-scan'),
{ stackName: undefined },
{ stackName: undefined, actor: 'system:scheduler' },
);
expect(mockDispatchAlert).toHaveBeenCalledWith(
'info',
'scan_finding',
expect.stringContaining('Scanned 3 image(s)'),
{ stackName: undefined },
{ stackName: undefined, actor: 'system:scheduler' },
);
});
@@ -1026,7 +1026,7 @@ describe('SchedulerService - scheduled scan notifications', () => {
'warning',
'scan_finding',
expect.stringContaining('2 failed'),
{ stackName: undefined },
{ stackName: undefined, actor: 'system:scheduler' },
);
});
@@ -1041,7 +1041,7 @@ describe('SchedulerService - scheduled scan notifications', () => {
'info',
'scan_finding',
expect.stringContaining('completed'),
{ stackName: 'web-stack' },
{ stackName: 'web-stack', actor: 'system:scheduler' },
);
});
@@ -1061,7 +1061,7 @@ describe('SchedulerService - scheduled scan notifications', () => {
'info',
'scan_finding',
expect.stringContaining('recovered-scan'),
{ stackName: undefined },
{ stackName: undefined, actor: 'system:scheduler' },
);
});
@@ -1096,7 +1096,7 @@ describe('SchedulerService - scheduled scan notifications', () => {
'error',
'system',
expect.stringMatching(/failed.*Trivy/i),
{ stackName: 'payment-stack' },
{ stackName: 'payment-stack', actor: 'system:scheduler' },
);
});
@@ -1145,7 +1145,7 @@ describe('SchedulerService - scheduled scan notifications', () => {
'info',
'scan_finding',
expect.stringContaining('No images to scan'),
{ stackName: undefined },
{ stackName: undefined, actor: 'system:scheduler' },
);
});
@@ -1160,7 +1160,7 @@ describe('SchedulerService - scheduled scan notifications', () => {
'info',
'scan_finding',
expect.stringContaining('All 12 image(s) already scanned recently'),
{ stackName: undefined },
{ stackName: undefined, actor: 'system:scheduler' },
);
});
@@ -1300,7 +1300,7 @@ describe('SchedulerService - invalid cron at execution time', () => {
'error',
'system',
expect.stringContaining('failed'),
{ stackName: undefined },
{ stackName: undefined, actor: 'system:scheduler' },
);
});
});
@@ -0,0 +1,169 @@
import { describe, it, expect, beforeAll, afterAll, beforeEach } from 'vitest';
import { setupTestDb, cleanupTestDb } from './helpers/setupTestDb';
import { sanitizeNotificationMessage } from '../utils/notificationMessage';
let tmpDir: string;
let db: any;
beforeAll(async () => {
tmpDir = await setupTestDb();
const { DatabaseService } = await import('../services/DatabaseService');
db = DatabaseService.getInstance();
});
afterAll(() => {
cleanupTestDb(tmpDir);
});
beforeEach(() => {
db.deleteAllNotifications(0);
});
describe('sanitizeNotificationMessage', () => {
it('passes plain messages through unchanged', () => {
expect(sanitizeNotificationMessage('Stack deployed in 4.2s')).toBe('Stack deployed in 4.2s');
});
it('redacts KEY=VALUE pairs whose key matches a sensitive suffix', () => {
const raw = 'compose parse failed: DB_PASSWORD=hunter2 missing closing quote';
expect(sanitizeNotificationMessage(raw)).toBe('compose parse failed: DB_PASSWORD=<redacted> missing closing quote');
});
it('redacts API_KEY, SECRET, TOKEN, CREDENTIAL variants', () => {
expect(sanitizeNotificationMessage('STRIPE_API_KEY=sk_live_abc')).toContain('STRIPE_API_KEY=<redacted>');
expect(sanitizeNotificationMessage('JWT_SECRET=eyJabc')).toContain('JWT_SECRET=<redacted>');
expect(sanitizeNotificationMessage('GITHUB_TOKEN=ghp_abc')).toContain('GITHUB_TOKEN=<redacted>');
expect(sanitizeNotificationMessage('AWS_CREDENTIALS=foo')).toContain('AWS_CREDENTIALS=<redacted>');
});
it('redacts every sensitive KEY=VALUE pair in a single message', () => {
const raw = 'compose env: DB_PASSWORD=hunter2 API_KEY=abc OTHER=keep';
const out = sanitizeNotificationMessage(raw);
expect(out).toContain('DB_PASSWORD=<redacted>');
expect(out).toContain('API_KEY=<redacted>');
expect(out).toContain('OTHER=keep');
});
it('leaves bare PASS-suffix keys like BYPASS and COMPASS unredacted', () => {
expect(sanitizeNotificationMessage('BYPASS=true')).toBe('BYPASS=true');
expect(sanitizeNotificationMessage('COMPASS_URL=https://example.com')).toBe('COMPASS_URL=https://example.com');
});
it('leaves non-sensitive uppercase KEY=VALUE untouched', () => {
expect(sanitizeNotificationMessage('NODE_ENV=production')).toBe('NODE_ENV=production');
expect(sanitizeNotificationMessage('PORT=1852')).toBe('PORT=1852');
});
it('redacts lowercase sensitive keys (compose env vars are commonly lowercase)', () => {
expect(sanitizeNotificationMessage('db_password=foo')).toBe('db_password=<redacted>');
expect(sanitizeNotificationMessage('jwt_secret=abc')).toBe('jwt_secret=<redacted>');
expect(sanitizeNotificationMessage('github_token=ghp_xyz')).toBe('github_token=<redacted>');
});
it('still leaves bare PASS-suffix lowercase keys unredacted', () => {
expect(sanitizeNotificationMessage('bypass=true')).toBe('bypass=true');
expect(sanitizeNotificationMessage('compass_url=https://example.com')).toBe('compass_url=https://example.com');
});
it('redacts HTTP basic auth in URLs', () => {
const raw = 'pull failed for https://user:pa55@registry.example.com/img';
expect(sanitizeNotificationMessage(raw)).toBe('pull failed for https://user:<redacted>@registry.example.com/img');
});
it('redacts bearer tokens', () => {
const raw = 'auth header was Bearer abc123xyzdef456';
expect(sanitizeNotificationMessage(raw)).toBe('auth header was Bearer <redacted>');
});
it('truncates messages longer than 1000 characters', () => {
const raw = 'x'.repeat(2000);
const out = sanitizeNotificationMessage(raw);
expect(out.length).toBeLessThanOrEqual(1000);
expect(out.endsWith('… [truncated]')).toBe(true);
});
it('collapses COMPOSE_DIR path prefixes', () => {
const out = sanitizeNotificationMessage(
'file not found: /opt/docker/sencho/compose/myapp/.env',
{ composeDir: '/opt/docker/sencho/compose' },
);
expect(out).toBe('file not found: <compose-dir>/myapp/.env');
});
});
describe('DatabaseService.getStackActivity', () => {
it('returns only events for the requested (node, stack)', () => {
const base = Date.now();
db.addNotificationHistory(0, { level: 'info', message: 'a-evt', timestamp: base, stack_name: 'a' });
db.addNotificationHistory(0, { level: 'info', message: 'b-evt', timestamp: base + 1, stack_name: 'b' });
db.addNotificationHistory(0, { level: 'info', message: 'a-evt-2', timestamp: base + 2, stack_name: 'a' });
const aOnly = db.getStackActivity(0, 'a', { limit: 50 });
expect(aOnly.map((e: any) => e.message)).toEqual(['a-evt-2', 'a-evt']);
});
it('honors limit and orders newest first', () => {
const base = Date.now();
for (let i = 0; i < 5; i++) {
db.addNotificationHistory(0, { level: 'info', message: `e-${i}`, timestamp: base + i, stack_name: 's' });
}
const out = db.getStackActivity(0, 's', { limit: 3 });
expect(out.length).toBe(3);
expect(out.map((e: any) => e.message)).toEqual(['e-4', 'e-3', 'e-2']);
});
it('legacy timestamp-only cursor excludes equal-or-newer rows', () => {
const base = Date.now();
for (let i = 0; i < 5; i++) {
db.addNotificationHistory(0, { level: 'info', message: `e-${i}`, timestamp: base + i * 10, stack_name: 's' });
}
// Cursor at base+20 (= e-2). before=base+20 means timestamp < base+20, so only e-0/e-1 returned.
const out = db.getStackActivity(0, 's', { limit: 50, before: base + 20 });
expect(out.map((e: any) => e.message)).toEqual(['e-1', 'e-0']);
});
it('composite (timestamp, id) cursor drops only the cursor row and older when many share a ms', () => {
const ts = Date.now();
const ids: number[] = [];
for (let i = 0; i < 5; i++) {
const row = db.addNotificationHistory(0, { level: 'info', message: `e-${i}`, timestamp: ts, stack_name: 's' });
ids.push(row.id);
}
// ids[0..4] all share ts. Cursor at the third row's id should return ids[0] and ids[1].
const out = db.getStackActivity(0, 's', { limit: 50, before: ts, beforeId: ids[2] });
const returnedIds = out.map((e: any) => e.id);
expect(returnedIds).toEqual([ids[1], ids[0]]);
});
it('documents the legacy timestamp-only cursor drops same-ms rows (kept for backward compat)', () => {
const ts = Date.now();
for (let i = 0; i < 5; i++) {
db.addNotificationHistory(0, { level: 'info', message: `e-${i}`, timestamp: ts, stack_name: 's' });
}
// The composite-cursor test above proves the fix; this test pins the legacy form's
// behavior so a client that omits beforeId gets a predictable (if lossy) result.
const out = db.getStackActivity(0, 's', { limit: 50, before: ts });
expect(out).toEqual([]);
});
it('returns empty array when stack has no events', () => {
db.addNotificationHistory(0, { level: 'info', message: 'other', timestamp: Date.now(), stack_name: 'other' });
const out = db.getStackActivity(0, 'missing', { limit: 50 });
expect(out).toEqual([]);
});
});
describe('DatabaseService.addNotificationHistory (no per-insert prune)', () => {
it('keeps a quiet stack visible even after a chatty stack writes past the old 100-row per-node cap', () => {
const ts = Date.now();
db.addNotificationHistory(0, { level: 'info', message: 'first', timestamp: ts, stack_name: 'a' });
// The old per-insert prune kept only the newest 100 rows per node, regardless of stack.
// Writing 150 rows for stack b would have evicted 'first' from stack a under the old rule.
for (let i = 0; i < 150; i++) {
db.addNotificationHistory(0, { level: 'info', message: `chatty-${i}`, timestamp: ts + i + 1, stack_name: 'b' });
}
const aActivity = db.getStackActivity(0, 'a', { limit: 50 });
expect(aActivity.map((e: any) => e.message)).toEqual(['first']);
});
});
@@ -166,7 +166,7 @@ describe('deploy_failure notification on /deploy error', () => {
'error',
'deploy_failure',
expect.stringContaining('image pull failed'),
{ stackName: 'myapp' },
{ stackName: 'myapp', actor: 'testadmin' },
);
});
@@ -183,7 +183,7 @@ describe('deploy_failure notification on /deploy error', () => {
expect(call[0]).toBe('error');
expect(call[1]).toBe('deploy_failure');
expect(call[2]).toContain('network timeout');
expect(call[3]).toEqual({ stackName: 'webapp' });
expect(call[3]).toEqual({ stackName: 'webapp', actor: 'testadmin' });
});
it('returns rolledBack=true only when compose rollback completed', async () => {
@@ -260,7 +260,7 @@ describe('deploy_failure notification on /down error', () => {
'error',
'deploy_failure',
expect.any(String),
{ stackName: 'myapp' },
{ stackName: 'myapp', actor: 'testadmin' },
);
});
});
@@ -282,7 +282,7 @@ describe('deploy_failure notification on /restart error', () => {
'error',
'deploy_failure',
expect.stringContaining('restart daemon error'),
{ stackName: 'myapp' },
{ stackName: 'myapp', actor: 'testadmin' },
);
});
});
@@ -304,7 +304,7 @@ describe('deploy_failure notification on /stop error', () => {
'error',
'deploy_failure',
expect.stringContaining('stop daemon error'),
{ stackName: 'myapp' },
{ stackName: 'myapp', actor: 'testadmin' },
);
});
});
@@ -325,7 +325,7 @@ describe('deploy_failure notification on /update error', () => {
'error',
'deploy_failure',
expect.stringContaining('image not found'),
{ stackName: 'myapp' },
{ stackName: 'myapp', actor: 'testadmin' },
);
});
+2 -2
View File
@@ -122,7 +122,7 @@ export async function triggerPostDeployScan(
scan.critical_count > 0 ? 'error' : 'warning',
'scan_finding',
`Vulnerability scan for ${imageRef}: ${scan.critical_count} critical, ${scan.high_count} high`,
{ stackName },
{ stackName, actor: 'system:policy' },
);
}
} catch (err) {
@@ -133,7 +133,7 @@ export async function triggerPostDeployScan(
'warning',
'scan_finding',
`Post-deploy scan failed for ${imageRef} (${stackName}): ${message}`,
{ stackName },
{ stackName, actor: 'system:policy' },
);
}
}
+2 -2
View File
@@ -294,7 +294,7 @@ autoUpdateRouter.post('/execute', authMiddleware, async (req: Request, res: Resp
);
if (!autoUpdateGate.ok) {
const blockedMsg = `Policy "${autoUpdateGate.policy?.name}" blocked auto-update: ${autoUpdateGate.violations.length} image(s) exceed ${autoUpdateGate.policy?.max_severity}`;
NotificationService.getInstance().dispatchAlert('warning', 'scan_finding', blockedMsg, { stackName });
NotificationService.getInstance().dispatchAlert('warning', 'scan_finding', blockedMsg, { stackName, actor: 'system:image-update' });
results.push(`Stack "${stackName}": ${blockedMsg}`);
continue;
}
@@ -315,7 +315,7 @@ autoUpdateRouter.post('/execute', authMiddleware, async (req: Request, res: Resp
'info',
'image_update_applied',
`Auto-update: stack "${stackName}" updated with new images`,
{ stackName },
{ stackName, actor: 'system:image-update' },
);
results.push(`Stack "${stackName}": updated (${updatedImages.join(', ')}).`);
+44 -5
View File
@@ -5,6 +5,14 @@ import { isValidStackName } from '../utils/validation';
export const stackActivityRouter = Router();
function parseStrictPositiveInt(raw: unknown): number | null {
if (raw === undefined || raw === null) return null;
const s = String(raw).trim();
if (s === '' || !/^\d+$/.test(s)) return null;
const n = Number(s);
return Number.isFinite(n) && n >= 1 ? n : null;
}
stackActivityRouter.get('/:stackName/activity', (req: Request, res: Response): void => {
const stackName = req.params.stackName as string;
if (!isValidStackName(stackName)) {
@@ -12,12 +20,43 @@ stackActivityRouter.get('/:stackName/activity', (req: Request, res: Response): v
return;
}
if (!requirePermission(req, res, 'stack:read', 'stack', stackName)) return;
const limit = Math.min(parseInt(String(req.query.limit ?? '50'), 10) || 50, 200);
const before = req.query.before ? parseInt(String(req.query.before), 10) : undefined;
if (before !== undefined && isNaN(before)) {
res.status(400).json({ error: 'Invalid before parameter' });
const parsedLimit = parseStrictPositiveInt(req.query.limit ?? '50');
if (parsedLimit === null) {
res.status(400).json({ error: 'Invalid limit parameter' });
return;
}
const events = DatabaseService.getInstance().getStackActivity(req.nodeId, stackName, { limit, before });
const limit = Math.min(parsedLimit, 200);
const hasBefore = req.query.before !== undefined;
const hasBeforeId = req.query.beforeId !== undefined;
// beforeId without before would silently fall back to "page 1" in the DB
// layer; reject so a paginating client cannot loop on the same page.
if (hasBeforeId && !hasBefore) {
res.status(400).json({ error: 'beforeId requires before' });
return;
}
let before: number | undefined;
if (hasBefore) {
const parsed = parseStrictPositiveInt(req.query.before);
if (parsed === null) {
res.status(400).json({ error: 'Invalid before parameter' });
return;
}
before = parsed;
}
let beforeId: number | undefined;
if (hasBeforeId) {
const parsed = parseStrictPositiveInt(req.query.beforeId);
if (parsed === null) {
res.status(400).json({ error: 'Invalid beforeId parameter' });
return;
}
beforeId = parsed;
}
const events = DatabaseService.getInstance().getStackActivity(req.nodeId, stackName, { limit, before, beforeId });
res.json({ events });
});
+9 -9
View File
@@ -41,10 +41,10 @@ const MAX_COMPOSE_PARSE_BYTES = 1_048_576; // 1 MiB
function dlog(...args: Parameters<typeof console.log>): void {
if (isDebugEnabled()) console.log(...args);
}
function notifyActionFailure(action: string, stackName: string, error: unknown): void {
function notifyActionFailure(action: string, stackName: string, error: unknown, actor: string): void {
const message = getErrorMessage(error, `Failed to ${action} stack`);
NotificationService.getInstance()
.dispatchAlert('error', 'deploy_failure', message, { stackName })
.dispatchAlert('error', 'deploy_failure', message, { stackName, actor })
.catch(err => console.error('[Stacks] Failed to dispatch failure notification for %s:', sanitizeForLog(stackName), err));
}
@@ -341,7 +341,7 @@ async function runStackBulkOp(
return { stackName, ok: false, error: 'No containers found for this stack', code: 'no_containers' };
}
if (outcome.kind === 'error') {
if (action !== 'start') notifyActionFailure(action, stackName, new Error(outcome.message));
if (action !== 'start') notifyActionFailure(action, stackName, new Error(outcome.message), user);
return { stackName, ok: false, error: outcome.message, code: 'op_failed' };
}
const meta = CONTAINER_ACTION_META[action];
@@ -349,7 +349,7 @@ async function runStackBulkOp(
}
return { stackName, ok: true };
} catch (err) {
if (action !== 'start') notifyActionFailure(action, stackName, err);
if (action !== 'start') notifyActionFailure(action, stackName, err, user);
return { stackName, ok: false, error: getErrorMessage(err, `${action} failed`), code: 'op_failed' };
} finally {
StackOpLockService.getInstance().release(req.nodeId, stackName);
@@ -962,7 +962,7 @@ stacksRouter.post('/:stackName/deploy', async (req: Request, res: Response) => {
console.warn('[Stacks] Deploy failed, rollback did not complete: %s', sanitizeForLog(stackName));
}
const message = getErrorMessage(error, 'Failed to deploy stack');
notifyActionFailure('deploy', stackName, error);
notifyActionFailure('deploy', stackName, error, req.user?.username ?? 'system');
if (!res.headersSent) {
if (isDockerUnavailableError(error)) {
res.status(503).json({ error: message, code: 'docker_unavailable', rolledBack });
@@ -993,7 +993,7 @@ stacksRouter.post('/:stackName/down', async (req: Request, res: Response) => {
res.json({ status: 'Command started' });
} catch (error: unknown) {
console.error('[Stacks] Down failed: %s', sanitizeForLog(stackName), error);
notifyActionFailure('down', stackName, error);
notifyActionFailure('down', stackName, error, req.user?.username ?? 'system');
if (!res.headersSent) {
if (isDockerUnavailableError(error)) {
res.status(503).json({ error: getErrorMessage(error, 'Docker daemon is unreachable'), code: 'docker_unavailable' });
@@ -1088,13 +1088,13 @@ async function bulkContainerOp(
}
if (outcome.kind === 'docker-unavailable') {
console.error('[Stacks] %s failed: docker unavailable for %s', sanitizeForLog(titleCase), sanitizeForLog(stackName));
if (action !== 'start') notifyActionFailure(action, stackName, new Error(outcome.message));
if (action !== 'start') notifyActionFailure(action, stackName, new Error(outcome.message), req.user?.username ?? 'system');
res.status(503).json({ error: outcome.message, code: 'docker_unavailable' });
return;
}
if (outcome.kind === 'error') {
console.error('[Stacks] %s failed: %s %s', sanitizeForLog(titleCase), sanitizeForLog(stackName), sanitizeForLog(outcome.message));
if (action !== 'start') notifyActionFailure(action, stackName, new Error(outcome.message));
if (action !== 'start') notifyActionFailure(action, stackName, new Error(outcome.message), req.user?.username ?? 'system');
res.status(500).json({ error: outcome.message });
return;
}
@@ -1230,7 +1230,7 @@ stacksRouter.post('/:stackName/update', async (req: Request, res: Response) => {
} else if (rollbackInfo?.attempted) {
console.warn(`[Stacks] Update failed, rollback did not complete: ${sanitizeForLog(stackName)}`);
}
notifyActionFailure('update', stackName, error);
notifyActionFailure('update', stackName, error, req.user?.username ?? 'system');
if (!res.headersSent) {
if (isDockerUnavailableError(error)) {
res.status(503).json({ error: getErrorMessage(error, 'Docker daemon is unreachable'), code: 'docker_unavailable', rolledBack });
+3 -3
View File
@@ -327,7 +327,7 @@ export class AutoHealService {
'info',
'autoheal_triggered',
`Auto-Heal: Restarted ${containerName} on stack ${policy.stack_name} after being unhealthy for ${policy.unhealthy_duration_mins} minute(s).`,
{ stackName: policy.stack_name, containerName },
{ stackName: policy.stack_name, containerName, actor: 'system:autoheal' },
)
.catch(err => console.error('[AutoHeal] notification dispatch failed:', err));
} catch (err) {
@@ -351,7 +351,7 @@ export class AutoHealService {
'warning',
'autoheal_triggered',
`Auto-Heal: Failed to restart ${containerName} on stack ${policy.stack_name}. Error: ${errorMsg}`,
{ stackName: policy.stack_name, containerName },
{ stackName: policy.stack_name, containerName, actor: 'system:autoheal' },
)
.catch(e => console.error('[AutoHeal] notification dispatch failed:', e));
@@ -384,7 +384,7 @@ export class AutoHealService {
'warning',
'autoheal_triggered',
`Auto-Heal: Policy for ${policy.stack_name}${policy.service_name ? '/' + policy.service_name : ''} has been auto-disabled after ${failures} consecutive failures.`,
{ stackName: policy.stack_name },
{ stackName: policy.stack_name, actor: 'system:autoheal' },
)
.catch(e => console.error('[AutoHeal] notification dispatch failed:', e));
}
+3 -3
View File
@@ -288,7 +288,7 @@ export class BlueprintReconciler {
'warning',
'blueprint_drift_detected',
`Blueprint "${blueprint.name}" drifted on node "${node.name}": ${reason}`,
{ stackName: blueprint.name },
{ stackName: blueprint.name, actor: 'system:blueprint' },
);
return;
@@ -302,7 +302,7 @@ export class BlueprintReconciler {
'warning',
'blueprint_drift_detected',
`Blueprint "${blueprint.name}" lost its marker on node "${node.name}"; auto-fix declined to avoid stomping unowned data. Reason: ${reason}`,
{ stackName: blueprint.name },
{ stackName: blueprint.name, actor: 'system:blueprint' },
);
return;
}
@@ -319,7 +319,7 @@ export class BlueprintReconciler {
'error',
'blueprint_drift_correction_failed',
`Auto-fix for "${blueprint.name}" on node "${node.name}" failed: ${result.error ?? 'unknown error'}`,
{ stackName: blueprint.name },
{ stackName: blueprint.name, actor: 'system:blueprint' },
);
}
return;
+58 -16
View File
@@ -2040,13 +2040,6 @@ export class DatabaseService {
notification.actor_username ?? null,
);
this.db.prepare(`
DELETE FROM notification_history
WHERE node_id = ? AND id NOT IN (
SELECT id FROM notification_history WHERE node_id = ? ORDER BY timestamp DESC LIMIT 100
)
`).run(nodeId, nodeId);
return {
id: result.lastInsertRowid as number,
level: notification.level,
@@ -2060,13 +2053,21 @@ export class DatabaseService {
};
}
public getStackActivity(nodeId: number, stackName: string, opts: { limit: number; before?: number }): NotificationHistory[] {
const sql = opts.before
? 'SELECT * FROM notification_history WHERE node_id = ? AND stack_name = ? AND timestamp < ? ORDER BY timestamp DESC LIMIT ?'
: 'SELECT * FROM notification_history WHERE node_id = ? AND stack_name = ? ORDER BY timestamp DESC LIMIT ?';
const args: (number | string)[] = opts.before
? [nodeId, stackName, opts.before, opts.limit]
: [nodeId, stackName, opts.limit];
public getStackActivity(nodeId: number, stackName: string, opts: { limit: number; before?: number; beforeId?: number }): NotificationHistory[] {
// Composite (timestamp, id) cursor: pure timestamp pagination drops rows
// on same-millisecond bursts (Docker events from one compose up).
let sql: string;
let args: (number | string)[];
if (opts.before !== undefined && opts.beforeId !== undefined) {
sql = 'SELECT * FROM notification_history WHERE node_id = ? AND stack_name = ? AND (timestamp < ? OR (timestamp = ? AND id < ?)) ORDER BY timestamp DESC, id DESC LIMIT ?';
args = [nodeId, stackName, opts.before, opts.before, opts.beforeId, opts.limit];
} else if (opts.before !== undefined) {
sql = 'SELECT * FROM notification_history WHERE node_id = ? AND stack_name = ? AND timestamp < ? ORDER BY timestamp DESC, id DESC LIMIT ?';
args = [nodeId, stackName, opts.before, opts.limit];
} else {
sql = 'SELECT * FROM notification_history WHERE node_id = ? AND stack_name = ? ORDER BY timestamp DESC, id DESC LIMIT ?';
args = [nodeId, stackName, opts.limit];
}
return (this.db.prepare(sql).all(...args) as unknown[]).map(row => this.mapNotificationRow(row as any));
}
@@ -2157,9 +2158,50 @@ export class DatabaseService {
stmt.run(cutoff);
}
public cleanupOldNotifications(daysToKeep = 30): void {
public cleanupOldNotifications(daysToKeep = 30, opts: { perStackCap?: number; perNodeUnattachedCap?: number } = {}): { ttl: number; perStack: number; perNode: number } {
const perStackCap = opts.perStackCap ?? 500;
const perNodeUnattachedCap = opts.perNodeUnattachedCap ?? 1000;
const cutoff = Date.now() - (daysToKeep * 24 * 60 * 60 * 1000);
this.db.prepare('DELETE FROM notification_history WHERE timestamp < ?').run(cutoff);
const ttlInfo = this.db.prepare('DELETE FROM notification_history WHERE timestamp < ?').run(cutoff);
const deleteById = this.db.prepare('DELETE FROM notification_history WHERE id = ?');
const deleteMany = this.db.transaction((ids: number[]) => {
for (const id of ids) deleteById.run(id);
});
// Per (node_id, stack_name) cap so a chatty stack cannot evict a quieter stack's history.
const stackOverflow = this.db.prepare(`
SELECT id FROM (
SELECT id, ROW_NUMBER() OVER (
PARTITION BY node_id, stack_name
ORDER BY timestamp DESC, id DESC
) AS rn
FROM notification_history
WHERE stack_name IS NOT NULL
)
WHERE rn > ?
`).all(perStackCap) as { id: number }[];
if (stackOverflow.length > 0) deleteMany(stackOverflow.map(r => r.id));
// Unattached system events have no stack to scope by, so they cannot share the per-stack quota; cap them per-node separately.
const unattachedOverflow = this.db.prepare(`
SELECT id FROM (
SELECT id, ROW_NUMBER() OVER (
PARTITION BY node_id
ORDER BY timestamp DESC, id DESC
) AS rn
FROM notification_history
WHERE stack_name IS NULL
)
WHERE rn > ?
`).all(perNodeUnattachedCap) as { id: number }[];
if (unattachedOverflow.length > 0) deleteMany(unattachedOverflow.map(r => r.id));
return {
ttl: Number(ttlInfo.changes ?? 0),
perStack: stackOverflow.length,
perNode: unattachedOverflow.length,
};
}
// --- Nodes ---
+3 -3
View File
@@ -673,15 +673,15 @@ export class DockerEventService {
// ========================================================================
private async emitError(category: NotificationCategory, message: string, stackName?: string, containerName?: string): Promise<void> {
return this.notifier.dispatchAlert('error', category, this.prefix(message), { stackName, containerName });
return this.notifier.dispatchAlert('error', category, this.prefix(message), { stackName, containerName, actor: 'system:docker-events' });
}
private async emitWarning(category: NotificationCategory, message: string, stackName?: string, containerName?: string): Promise<void> {
return this.notifier.dispatchAlert('warning', category, this.prefix(message), { stackName, containerName });
return this.notifier.dispatchAlert('warning', category, this.prefix(message), { stackName, containerName, actor: 'system:docker-events' });
}
private async emitInfo(category: NotificationCategory, message: string, stackName?: string, containerName?: string): Promise<void> {
return this.notifier.dispatchAlert('info', category, this.prefix(message), { stackName, containerName });
return this.notifier.dispatchAlert('info', category, this.prefix(message), { stackName, containerName, actor: 'system:docker-events' });
}
private prefix(message: string): string {
+9 -3
View File
@@ -6,6 +6,7 @@ import { FileSystemService } from './FileSystemService';
import { RegistryService } from './RegistryService';
import { NodeRegistry } from './NodeRegistry';
import { NotificationService } from './NotificationService';
import { sanitizeNotificationMessage } from '../utils/notificationMessage';
import { parseImageRef, getRemoteDigest } from './registry-api';
import { isDebugEnabled } from '../utils/debug';
import { getErrorMessage } from '../utils/errors';
@@ -308,7 +309,7 @@ export class ImageUpdateService {
'info',
'image_update_available',
`[Node: ${nodeName}] Stack "${stackName}" has image updates available.`,
{ stackName },
{ stackName, actor: 'system:image-update' },
);
} catch (e) {
console.error(`[ImageUpdateService] Failed to dispatch update notification for "${stackName}":`, e);
@@ -316,11 +317,16 @@ export class ImageUpdateService {
// Key on the local default: the iterated `nodeId` may be a remote's id in the
// control plane's DB, and the UI never queries that row (it proxies instead).
try {
db.addNotificationHistory(NodeRegistry.getInstance().getDefaultNodeId(), {
const localNodeId = NodeRegistry.getInstance().getDefaultNodeId();
db.addNotificationHistory(localNodeId, {
level: 'error',
category: 'system',
message: `[Node: ${nodeName}] Failed to notify about image updates for stack "${stackName}": ${getErrorMessage(e, String(e))}`,
message: sanitizeNotificationMessage(
`[Node: ${nodeName}] Failed to notify about image updates for stack "${stackName}": ${getErrorMessage(e, String(e))}`,
{ composeDir: NodeRegistry.getInstance().getComposeDir(localNodeId) },
),
timestamp: Date.now(),
actor_username: 'system:image-update',
});
} catch (dbErr) {
console.error('[ImageUpdateService] Failed to record dispatch error:', dbErr);
+4 -4
View File
@@ -548,10 +548,10 @@ export class MonitorService {
const retentionHours = parseInt(settings['metrics_retention_hours'] || '24', 10);
db.cleanupOldMetrics(isNaN(retentionHours) ? 24 : retentionHours);
const retentionDays = parseInt(settings['log_retention_days'] || '30', 10);
db.cleanupOldNotifications(isNaN(retentionDays) ? 30 : retentionDays);
const notifSummary = db.cleanupOldNotifications(isNaN(retentionDays) ? 30 : retentionDays);
const auditRetentionDays = parseInt(settings['audit_retention_days'] || '90', 10);
db.cleanupOldAuditLogs(isNaN(auditRetentionDays) ? 90 : auditRetentionDays);
if (isDebugEnabled()) console.log(`[Monitor:diag] Cleanup: metrics ${isNaN(retentionHours) ? 24 : retentionHours}h, notifications ${isNaN(retentionDays) ? 30 : retentionDays}d, audit ${isNaN(auditRetentionDays) ? 90 : auditRetentionDays}d`);
if (isDebugEnabled()) console.log(`[Monitor:diag] Cleanup: metrics ${isNaN(retentionHours) ? 24 : retentionHours}h, notifications ${isNaN(retentionDays) ? 30 : retentionDays}d (ttl=${notifSummary.ttl} perStack=${notifSummary.perStack} perNode=${notifSummary.perNode}), audit ${isNaN(auditRetentionDays) ? 90 : auditRetentionDays}d`);
} catch (e) {
console.error('MonitorService: failed to cleanup old data', e);
}
@@ -670,7 +670,7 @@ export class MonitorService {
'warning',
'monitor_alert',
message,
{ stackName: rule.stack_name },
{ stackName: rule.stack_name, actor: 'system:monitor' },
);
db.updateStackAlertLastFired(ruleId, Date.now());
@@ -723,7 +723,7 @@ export class MonitorService {
const db = DatabaseService.getInstance();
const last = parseInt(db.getSystemState(stateKey) || '0', 10);
if (Date.now() - last > cooldownMs) {
await NotificationService.getInstance().dispatchAlert(severity, category, message, { stackName: stack });
await NotificationService.getInstance().dispatchAlert(severity, category, message, { stackName: stack, actor: 'system:monitor' });
db.setSystemState(stateKey, Date.now().toString());
}
}
+9 -3
View File
@@ -4,6 +4,7 @@ import { NodeRegistry } from './NodeRegistry';
import { isDebugEnabled } from '../utils/debug';
import { getErrorMessage } from '../utils/errors';
import { sanitizeForLog } from '../utils/safeLog';
import { sanitizeNotificationMessage } from '../utils/notificationMessage';
export type NotificationCategory =
| 'deploy_success'
@@ -119,10 +120,15 @@ export class NotificationService {
// with user-initiated requests; otherwise the UI and monitors split
// between different node_id buckets.
const localNodeId = NodeRegistry.getInstance().getDefaultNodeId();
// Use the full resolution chain (node.compose_dir → env → default)
// so messages mentioning a per-node compose override get collapsed.
const sanitized = sanitizeNotificationMessage(message, {
composeDir: NodeRegistry.getInstance().getComposeDir(localNodeId),
});
const notification = this.dbService.addNotificationHistory(localNodeId, {
level,
category,
message,
message: sanitized,
timestamp: Date.now(),
stack_name: stackName,
container_name: containerName,
@@ -151,7 +157,7 @@ export class NotificationService {
if (isDebugEnabled()) console.log(`[Notify:diag] Matched ${matched.length} route(s) for stack "${sanitizeForLog(stackName ?? '(none)')}", category="${sanitizeForLog(category)}"`);
await Promise.allSettled(
matched.map(route =>
this.sendToChannel(route.channel_type, route.channel_url, level, message)
this.sendToChannel(route.channel_type, route.channel_url, level, sanitized)
.then(() => {
if (isDebugEnabled()) console.log(`[Notify:diag] Dispatched ${level} via route "${route.name}" (${route.channel_type})`);
})
@@ -176,7 +182,7 @@ export class NotificationService {
if (isDebugEnabled()) console.log(`[Notify:diag] Falling back to ${agents.length} global agent(s)`);
await Promise.allSettled(
agents.map(agent =>
this.sendToChannel(agent.type, agent.url, level, message)
this.sendToChannel(agent.type, agent.url, level, sanitized)
.then(() => {
if (isDebugEnabled()) console.log(`[Notify:diag] Dispatched ${level} via global agent (${agent.type})`);
})
+1 -1
View File
@@ -66,7 +66,7 @@ function notifyTrivyMissingOnce(nodeId: number, stackName: string): void {
'warning',
'scan_finding',
`Pre-deploy scan for "${stackName}" skipped: Trivy not installed on this node`,
{ stackName },
{ stackName, actor: 'system:policy' },
);
}
+2 -1
View File
@@ -167,7 +167,7 @@ export class SchedulerService {
*/
private safeDispatch(level: 'info' | 'warning' | 'error', category: import('./NotificationService').NotificationCategory, message: string, stackName?: string): void {
NotificationService.getInstance()
.dispatchAlert(level, category, message, { stackName })
.dispatchAlert(level, category, message, { stackName, actor: 'system:scheduler' })
.catch(err => console.error('[SchedulerService] Notification dispatch failed:', getErrorMessage(err, 'unknown error')));
}
@@ -778,6 +778,7 @@ export class SchedulerService {
'warning',
'scan_finding',
`Policy "${v.policyName}" violated by ${v.imageRef}: ${v.severity} exceeds ${v.maxSeverity}`,
{ actor: 'system:scheduler' },
);
}
+35
View File
@@ -0,0 +1,35 @@
const MAX_MESSAGE_CHARS = 1000;
// U+2026 ellipsis (one codepoint), not three ASCII dots; tests assert this exact suffix.
const TRUNCATION_SUFFIX = '… [truncated]';
// KEY=VALUE pairs whose KEY ends in a sensitive suffix. Values are stripped
// because compose-parse errors and docker run failures surface them verbatim.
// Case-insensitive: compose env names are conventionally uppercase but
// lowercase forms (db_password, jwt_secret, github_token) are valid and do
// leak through the same error paths. PASSWORD covers the bare-PASS case;
// standalone PASS would over-redact BYPASS, COMPASS, and similar non-secret
// keys.
const SENSITIVE_KEY_PATTERN = /\b([A-Za-z0-9_]*(?:TOKEN|KEY|PASSWORD|SECRET|CREDENTIALS?|AUTH))\s*=\s*("[^"]*"|'[^']*'|\S+)/gi;
const URL_BASIC_AUTH = /\b([a-zA-Z][a-zA-Z0-9+.-]*:\/\/)([^/\s:@]+):([^/\s:@]+)@/g;
const BEARER_TOKEN = /\b(Bearer)\s+([A-Za-z0-9._\-+/=]{8,})/g;
export function sanitizeNotificationMessage(raw: string, opts: { composeDir?: string } = {}): string {
if (!raw) return raw;
let s = raw;
s = s.replace(SENSITIVE_KEY_PATTERN, (_m, key) => `${key}=<redacted>`);
s = s.replace(URL_BASIC_AUTH, (_m, scheme, user) => `${scheme}${user}:<redacted>@`);
s = s.replace(BEARER_TOKEN, (_m, kw) => `${kw} <redacted>`);
if (opts.composeDir) {
const escaped = opts.composeDir.replace(/[.*+?^${}()|[\]\\]/g, '\\$&');
s = s.replace(new RegExp(escaped, 'g'), '<compose-dir>');
}
if (s.length > MAX_MESSAGE_CHARS) {
s = s.slice(0, MAX_MESSAGE_CHARS - TRUNCATION_SUFFIX.length) + TRUNCATION_SUFFIX;
}
return s;
}
@@ -82,6 +82,7 @@ export function useNotifications({ nodes, onStateInvalidate, onAutoUpdateChange,
if (!isMounted) { ws?.close(); return; }
setTickerConnected(true);
retryCount = 0;
window.dispatchEvent(new CustomEvent('sencho:notifications-connection', { detail: { connected: true } }));
};
ws.onmessage = (event) => {
@@ -113,6 +114,7 @@ export function useNotifications({ nodes, onStateInvalidate, onAutoUpdateChange,
ws.onclose = (event) => {
setTickerConnected(false);
window.dispatchEvent(new CustomEvent('sencho:notifications-connection', { detail: { connected: false } }));
if (!isMounted) return;
const delay = Math.min(1000 * Math.pow(2, retryCount), MAX_RETRY_DELAY_MS);
retryCount++;
@@ -1,6 +1,6 @@
import { useCallback, useEffect, useMemo, useRef, useState } from 'react';
import {
Rocket, RefreshCcw, CircleStop, Play, ArrowUp, Activity, Loader2,
Rocket, RefreshCcw, CircleStop, Play, ArrowUp, Activity, Loader2, AlertCircle,
} from 'lucide-react';
import type { LucideIcon } from 'lucide-react';
import { Button } from '@/components/ui/button';
@@ -9,9 +9,12 @@ import { apiFetch } from '@/lib/api';
import { formatTimeAgo } from '@/lib/relativeTime';
import type { NotificationItem } from '@/components/dashboard/types';
type ActivityLevel = 'info' | 'warning' | 'error';
const ACTIVITY_LEVELS: readonly string[] = ['info', 'warning', 'error'];
interface ActivityEvent {
id: number;
level: string;
level: ActivityLevel;
category?: string;
message: string;
timestamp: number;
@@ -33,9 +36,37 @@ const CATEGORY_ICON: Record<string, LucideIcon> = {
};
const DAY_MS = 86_400_000;
const PAGE_SIZE = 50;
const DAY_BUCKET_REFRESH_MS = 60_000;
function dayLabel(ts: number): 'Today' | 'Yesterday' | 'Earlier' {
const todayStart = new Date();
const SYSTEM_ACTOR_LABEL: Record<string, string> = {
'system': 'System',
'system:autoheal': 'Auto-Heal',
'system:scheduler': 'Scheduler',
'system:image-update': 'Image Update',
'system:docker-events': 'Docker',
'system:blueprint': 'Blueprint',
'system:monitor': 'Monitor',
'system:policy': 'Policy',
};
function formatActor(actor: string): { label: string; isSystem: boolean } {
const isSystem = actor === 'system' || actor.startsWith('system:');
return { label: SYSTEM_ACTOR_LABEL[actor] ?? actor, isSystem };
}
function isActivityEvent(value: unknown): value is ActivityEvent {
if (!value || typeof value !== 'object') return false;
const v = value as Record<string, unknown>;
return typeof v.id === 'number'
&& typeof v.timestamp === 'number'
&& typeof v.message === 'string'
&& typeof v.level === 'string'
&& ACTIVITY_LEVELS.includes(v.level);
}
function dayLabel(ts: number, now: number): 'Today' | 'Yesterday' | 'Earlier' {
const todayStart = new Date(now);
todayStart.setHours(0, 0, 0, 0);
const todayMs = todayStart.getTime();
if (ts >= todayMs) return 'Today';
@@ -43,11 +74,11 @@ function dayLabel(ts: number): 'Today' | 'Yesterday' | 'Earlier' {
return 'Earlier';
}
function groupEvents(events: ActivityEvent[]): { label: string; events: ActivityEvent[] }[] {
function groupEvents(events: ActivityEvent[], now: number): { label: string; events: ActivityEvent[] }[] {
const groups: Record<string, ActivityEvent[]> = {};
const order: string[] = [];
for (const e of events) {
const label = dayLabel(e.timestamp);
const label = dayLabel(e.timestamp, now);
if (!groups[label]) { groups[label] = []; order.push(label); }
groups[label].push(e);
}
@@ -59,6 +90,10 @@ export function StackActivityTimeline({ stackName, liveEvents }: StackActivityTi
const [loading, setLoading] = useState(true);
const [loadingMore, setLoadingMore] = useState(false);
const [hasMore, setHasMore] = useState(true);
const [error, setError] = useState(false);
const [reloadKey, setReloadKey] = useState(0);
const [now, setNow] = useState(() => Date.now());
const [liveDisconnected, setLiveDisconnected] = useState(false);
const seenIdsRef = useRef(new Set<number>());
const mergeEvents = useCallback((incoming: ActivityEvent[]) => {
@@ -72,7 +107,7 @@ export function StackActivityTimeline({ stackName, liveEvents }: StackActivityTi
added = true;
}
if (!added) return prev;
next.sort((a, b) => b.timestamp - a.timestamp);
next.sort((a, b) => b.timestamp - a.timestamp || b.id - a.id);
return next;
});
}, []);
@@ -80,48 +115,95 @@ export function StackActivityTimeline({ stackName, liveEvents }: StackActivityTi
useEffect(() => {
let cancelled = false;
setLoading(true);
setError(false);
seenIdsRef.current = new Set();
setEvents([]);
setHasMore(true);
apiFetch(`/stacks/${stackName}/activity?limit=50`)
.then(r => (r.ok ? r.json() : Promise.reject()))
.then((data: { events: ActivityEvent[] }) => {
if (cancelled) return;
setHasMore(data.events.length === 50);
data.events.forEach(e => seenIdsRef.current.add(e.id));
setEvents(data.events);
apiFetch(`/stacks/${stackName}/activity?limit=${PAGE_SIZE + 1}`)
.then(async r => {
if (!r.ok) throw new Error(`HTTP ${r.status}`);
return r.json() as Promise<{ events: ActivityEvent[] }>;
})
.then(data => {
if (cancelled) return;
const more = data.events.length > PAGE_SIZE;
const trimmed = more ? data.events.slice(0, PAGE_SIZE) : data.events;
setHasMore(more);
trimmed.forEach(e => seenIdsRef.current.add(e.id));
setEvents(trimmed);
})
.catch(() => {
if (cancelled) return;
setError(true);
setHasMore(false);
})
.catch(() => { if (!cancelled) setEvents([]); })
.finally(() => { if (!cancelled) setLoading(false); });
return () => { cancelled = true; };
}, [stackName]);
}, [stackName, reloadKey]);
// liveEvents is pre-filtered by stack_name in the parent
useEffect(() => {
if (!liveEvents || liveEvents.length === 0) return;
mergeEvents(liveEvents as ActivityEvent[]);
const safe = liveEvents.filter(isActivityEvent);
if (safe.length === 0) return;
mergeEvents(safe);
}, [liveEvents, mergeEvents]);
useEffect(() => {
// Skip the day-bucket refresh while the tab is hidden so a backgrounded
// panel does not re-render every minute for no visible effect.
const id = window.setInterval(() => {
if (typeof document !== 'undefined' && document.hidden) return;
setNow(Date.now());
}, DAY_BUCKET_REFRESH_MS);
return () => window.clearInterval(id);
}, []);
useEffect(() => {
// Upstream useNotifications dispatches this event when its WebSocket
// flips. A single layout-level listener keeps the timeline honest about
// whether new events would actually arrive.
const handler = (e: Event) => {
const detail = (e as CustomEvent<{ connected: boolean }>).detail;
setLiveDisconnected(detail?.connected === false);
};
window.addEventListener('sencho:notifications-connection', handler);
return () => window.removeEventListener('sencho:notifications-connection', handler);
}, []);
const loadMore = useCallback(async () => {
const oldest = events[events.length - 1]?.timestamp;
if (!oldest) return;
const oldestEvent = events[events.length - 1];
if (!oldestEvent) return;
setLoadingMore(true);
try {
const r = await apiFetch(`/stacks/${stackName}/activity?limit=50&before=${oldest}`);
if (!r.ok) return;
const params = new URLSearchParams({
limit: String(PAGE_SIZE + 1),
before: String(oldestEvent.timestamp),
beforeId: String(oldestEvent.id),
});
const r = await apiFetch(`/stacks/${stackName}/activity?${params.toString()}`);
if (!r.ok) {
toast.error('Failed to load more activity');
// 5xx is likely server-side; stop offering a button that keeps failing.
if (r.status >= 500) setHasMore(false);
return;
}
const data: { events: ActivityEvent[] } = await r.json();
setHasMore(data.events.length === 50);
mergeEvents(data.events);
} catch {
const more = data.events.length > PAGE_SIZE;
const trimmed = more ? data.events.slice(0, PAGE_SIZE) : data.events;
setHasMore(more);
mergeEvents(trimmed);
} catch (err) {
console.error('[StackActivity] loadMore failed:', err);
toast.error('Failed to load more activity');
setHasMore(false);
} finally {
setLoadingMore(false);
}
}, [events, stackName, mergeEvents]);
const groups = useMemo(() => groupEvents(events), [events]);
const groups = useMemo(() => groupEvents(events, now), [events, now]);
if (loading) {
return (
@@ -131,6 +213,23 @@ export function StackActivityTimeline({ stackName, liveEvents }: StackActivityTi
);
}
if (error && events.length === 0) {
return (
<div className="flex flex-col items-center justify-center py-8 gap-2">
<AlertCircle className="w-5 h-5 text-destructive/60" />
<span className="font-mono text-[11px] text-muted-foreground">Activity unavailable</span>
<Button
variant="ghost"
size="sm"
className="h-7 font-mono text-[10px]"
onClick={() => setReloadKey(k => k + 1)}
>
Retry
</Button>
</div>
);
}
if (events.length === 0) {
return (
<div className="flex flex-col items-center justify-center py-8 gap-2">
@@ -142,18 +241,33 @@ export function StackActivityTimeline({ stackName, liveEvents }: StackActivityTi
return (
<div className="flex flex-col gap-3 py-3">
{liveDisconnected && (
<div
role="status"
className="font-mono text-[10px] text-stat-subtitle italic px-1"
>
Live updates offline; reconnecting
</div>
)}
{groups.map(g => (
<div key={g.label}>
<div key={g.label} role="list" aria-label={`Stack activity, ${g.label}`}>
<div className="font-mono text-[9px] uppercase tracking-[0.18em] text-stat-subtitle mb-1.5 px-1">{g.label}</div>
{g.events.map(e => {
const Icon = CATEGORY_ICON[e.category ?? ''] ?? Activity;
const actor = e.actor_username ? formatActor(e.actor_username) : null;
return (
<div key={e.id} className="flex items-start gap-2 py-1.5 px-1 rounded-md hover:bg-glass-highlight/30 transition-colors">
<div
key={e.id}
role="listitem"
className="flex items-start gap-2 py-1.5 px-1 rounded-md hover:bg-glass-highlight/30 transition-colors"
>
<Icon className="w-3 h-3 mt-0.5 shrink-0 text-brand/70" strokeWidth={1.5} />
<div className="flex-1 min-w-0">
<span className="font-mono text-[11px] text-foreground/90">{e.message}</span>
{e.actor_username && e.actor_username !== 'system' && (
<span className="ml-1.5 font-mono text-[10px] text-stat-subtitle">by {e.actor_username}</span>
{actor && (
<span className={`ml-1.5 font-mono text-[10px] text-stat-subtitle${actor.isSystem ? ' italic' : ''}`}>
{actor.isSystem ? `via ${actor.label}` : `by ${actor.label}`}
</span>
)}
</div>
<span className="font-mono text-[10px] text-stat-subtitle shrink-0">{formatTimeAgo(e.timestamp)}</span>
@@ -0,0 +1,248 @@
import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest';
import { render, screen, waitFor } from '@testing-library/react';
import userEvent from '@testing-library/user-event';
vi.mock('@/lib/api', () => ({
apiFetch: vi.fn(),
}));
const { toastFns } = vi.hoisted(() => ({
toastFns: {
error: vi.fn(),
success: vi.fn(),
warning: vi.fn(),
info: vi.fn(),
loading: vi.fn(),
dismiss: vi.fn(),
},
}));
vi.mock('@/components/ui/toast-store', () => ({ toast: toastFns }));
vi.mock('@/lib/relativeTime', () => ({
formatTimeAgo: (ts: number) => `t-${ts}`,
}));
import { apiFetch } from '@/lib/api';
import { StackActivityTimeline } from '../StackActivityTimeline';
const mockFetch = apiFetch as unknown as ReturnType<typeof vi.fn>;
interface FakeEvent {
id: number;
level: string;
category?: string;
message: string;
timestamp: number;
stack_name?: string;
actor_username?: string | null;
}
function evt(overrides: Partial<FakeEvent> = {}): FakeEvent {
return {
id: 1,
level: 'info',
category: 'deploy_success',
message: 'deployed',
timestamp: Date.now(),
stack_name: 'web',
actor_username: null,
...overrides,
};
}
function jsonResponse(events: FakeEvent[]): Promise<Response> {
return Promise.resolve({
ok: true,
status: 200,
json: async () => ({ events }),
} as Response);
}
beforeEach(() => {
mockFetch.mockReset();
Object.values(toastFns).forEach(fn => (fn as ReturnType<typeof vi.fn>).mockReset());
});
afterEach(() => {
vi.useRealTimers();
});
describe('StackActivityTimeline - loading and empty', () => {
it('renders a spinner while the initial fetch is in flight', () => {
mockFetch.mockReturnValueOnce(new Promise(() => { /* never resolves */ }));
render(<StackActivityTimeline stackName="web" />);
expect(document.querySelector('.animate-spin')).toBeTruthy();
});
it('renders empty state when no events are returned', async () => {
mockFetch.mockReturnValueOnce(jsonResponse([]));
render(<StackActivityTimeline stackName="web" />);
await waitFor(() => expect(screen.getByText('No activity recorded yet')).toBeTruthy());
});
it('renders an error affordance with retry when fetch fails', async () => {
mockFetch.mockReturnValueOnce(Promise.resolve({ ok: false, status: 500, json: async () => ({}) } as Response));
render(<StackActivityTimeline stackName="web" />);
await waitFor(() => expect(screen.getByText('Activity unavailable')).toBeTruthy());
expect(screen.getByRole('button', { name: 'Retry' })).toBeTruthy();
});
it('Retry re-issues the fetch', async () => {
mockFetch
.mockReturnValueOnce(Promise.resolve({ ok: false, status: 500, json: async () => ({}) } as Response))
.mockReturnValueOnce(jsonResponse([evt({ id: 1, message: 'recovered' })]));
const user = userEvent.setup();
render(<StackActivityTimeline stackName="web" />);
await waitFor(() => expect(screen.getByText('Activity unavailable')).toBeTruthy());
await user.click(screen.getByRole('button', { name: 'Retry' }));
await waitFor(() => expect(screen.getByText('recovered')).toBeTruthy());
});
});
describe('StackActivityTimeline - pagination', () => {
it('hides "Load more" when initial response is shorter than PAGE_SIZE + 1', async () => {
mockFetch.mockReturnValueOnce(jsonResponse([evt({ id: 1, message: 'only' })]));
render(<StackActivityTimeline stackName="web" />);
await waitFor(() => expect(screen.getByText('only')).toBeTruthy());
expect(screen.queryByText('Load more')).toBeNull();
});
it('shows "Load more" when initial response has PAGE_SIZE+1 events and trims one row', async () => {
const page1 = Array.from({ length: 51 }, (_, i) => evt({ id: 100 - i, message: `e-${i}`, timestamp: 1000 - i }));
mockFetch.mockReturnValueOnce(jsonResponse(page1));
render(<StackActivityTimeline stackName="web" />);
await waitFor(() => expect(screen.getByText('e-0')).toBeTruthy());
expect(screen.queryByText('e-50')).toBeNull(); // 51st event was trimmed
expect(screen.getByText('Load more')).toBeTruthy();
});
it('Load more requests both before and beforeId, dedupes against existing events', async () => {
const page1 = Array.from({ length: 51 }, (_, i) => evt({ id: 100 - i, message: `e-${i}`, timestamp: 1000 - i }));
const page2 = [evt({ id: 50, message: 'older', timestamp: 949 })];
mockFetch
.mockReturnValueOnce(jsonResponse(page1))
.mockReturnValueOnce(jsonResponse(page2));
const user = userEvent.setup();
render(<StackActivityTimeline stackName="web" />);
await waitFor(() => expect(screen.getByText('Load more')).toBeTruthy());
await user.click(screen.getByText('Load more'));
await waitFor(() => expect(screen.getByText('older')).toBeTruthy());
// After trimming the 51st row, the oldest displayed event is { id: 51, ts: 951 },
// so loadMore sends before=951, beforeId=51 as the composite cursor.
const lastCall = mockFetch.mock.calls[1][0] as string;
expect(lastCall).toContain('limit=51');
expect(lastCall).toContain('before=951');
expect(lastCall).toContain('beforeId=51');
});
it('toasts on loadMore failure and keeps existing rows', async () => {
const page1 = Array.from({ length: 51 }, (_, i) => evt({ id: 100 - i, message: `e-${i}`, timestamp: 1000 - i }));
mockFetch
.mockReturnValueOnce(jsonResponse(page1))
.mockReturnValueOnce(Promise.resolve({ ok: false, status: 500, json: async () => ({}) } as Response));
const user = userEvent.setup();
render(<StackActivityTimeline stackName="web" />);
await waitFor(() => expect(screen.getByText('Load more')).toBeTruthy());
await user.click(screen.getByText('Load more'));
await waitFor(() => expect(toastFns.error).toHaveBeenCalledWith('Failed to load more activity'));
expect(screen.getByText('e-0')).toBeTruthy();
});
});
describe('StackActivityTimeline - liveEvents merge', () => {
it('merges new live events into the timeline', async () => {
mockFetch.mockReturnValueOnce(jsonResponse([evt({ id: 1, message: 'first', timestamp: 100 })]));
const { rerender } = render(<StackActivityTimeline stackName="web" />);
await waitFor(() => expect(screen.getByText('first')).toBeTruthy());
rerender(<StackActivityTimeline stackName="web" liveEvents={[evt({ id: 2, message: 'live', timestamp: 200 }) as never]} />);
await waitFor(() => expect(screen.getByText('live')).toBeTruthy());
});
it('dedupes live events that overlap an already-loaded id', async () => {
mockFetch.mockReturnValueOnce(jsonResponse([evt({ id: 1, message: 'first', timestamp: 100 })]));
const { rerender } = render(<StackActivityTimeline stackName="web" />);
await waitFor(() => expect(screen.getByText('first')).toBeTruthy());
rerender(<StackActivityTimeline stackName="web" liveEvents={[evt({ id: 1, message: 'first', timestamp: 100 }) as never]} />);
// Still exactly one row.
expect(screen.getAllByText('first')).toHaveLength(1);
});
it('drops malformed live events through the narrow guard but accepts well-formed siblings', async () => {
mockFetch.mockReturnValueOnce(jsonResponse([evt({ id: 1, message: 'first', timestamp: 100 })]));
const { rerender } = render(<StackActivityTimeline stackName="web" />);
await waitFor(() => expect(screen.getByText('first')).toBeTruthy());
// Mixed batch: one bad (id is a string, wrong level), one good. The guard
// must reject the bad row and let the good one through.
rerender(
<StackActivityTimeline
stackName="web"
liveEvents={[
{ id: 'not-a-number', message: 'BAD', timestamp: 1, level: 'info' } as never,
evt({ id: 99, message: 'GOOD', timestamp: 200 }) as never,
]}
/>,
);
await waitFor(() => expect(screen.getByText('GOOD')).toBeTruthy());
expect(screen.queryByText('BAD')).toBeNull();
});
it('rejects an event whose level is outside the union', async () => {
mockFetch.mockReturnValueOnce(jsonResponse([evt({ id: 1, message: 'first', timestamp: 100 })]));
const { rerender } = render(<StackActivityTimeline stackName="web" />);
await waitFor(() => expect(screen.getByText('first')).toBeTruthy());
rerender(
<StackActivityTimeline
stackName="web"
liveEvents={[{ id: 2, message: 'CRIT', timestamp: 200, level: 'critical' } as never]}
/>,
);
expect(screen.queryByText('CRIT')).toBeNull();
});
});
describe('StackActivityTimeline - stackName change resets state', () => {
it('refetches when stackName changes and clears prior events', async () => {
mockFetch.mockReturnValueOnce(jsonResponse([evt({ id: 1, message: 'web-evt' })]));
const { rerender } = render(<StackActivityTimeline stackName="web" />);
await waitFor(() => expect(screen.getByText('web-evt')).toBeTruthy());
mockFetch.mockReturnValueOnce(jsonResponse([evt({ id: 2, message: 'api-evt', stack_name: 'api' })]));
rerender(<StackActivityTimeline stackName="api" />);
await waitFor(() => expect(screen.getByText('api-evt')).toBeTruthy());
expect(screen.queryByText('web-evt')).toBeNull();
});
});
describe('StackActivityTimeline - actor rendering', () => {
it('renders human actor with "by <name>" prefix', async () => {
mockFetch.mockReturnValueOnce(jsonResponse([evt({ id: 1, message: 'deployed', actor_username: 'alice' })]));
render(<StackActivityTimeline stackName="web" />);
await waitFor(() => expect(screen.getByText(/by alice/)).toBeTruthy());
});
it('renders synthetic system actor with "via <Label>" prefix', async () => {
mockFetch.mockReturnValueOnce(jsonResponse([evt({ id: 1, message: 'restarted', actor_username: 'system:autoheal' })]));
render(<StackActivityTimeline stackName="web" />);
await waitFor(() => expect(screen.getByText(/via Auto-Heal/)).toBeTruthy());
});
it('renders bare "system" actor as "via System"', async () => {
mockFetch.mockReturnValueOnce(jsonResponse([evt({ id: 1, message: 'event', actor_username: 'system' })]));
render(<StackActivityTimeline stackName="web" />);
await waitFor(() => expect(screen.getByText(/via System/)).toBeTruthy());
});
it('omits actor line when actor_username is null', async () => {
mockFetch.mockReturnValueOnce(jsonResponse([evt({ id: 1, message: 'event', actor_username: null })]));
render(<StackActivityTimeline stackName="web" />);
await waitFor(() => expect(screen.getByText('event')).toBeTruthy());
expect(screen.queryByText(/^(by|via)\s/)).toBeNull();
});
});