mirror of
https://github.com/Studio-Saelix/sencho.git
synced 2026-08-09 18:32:52 +00:00
fix(stack-activity): per-stack history integrity, attribution, sanitization (#1228)
* fix(stack-activity): per-stack history integrity, attribution, sanitization Address the Stack Activity audit findings (PR 1 of 2): - Per-stack history integrity: drop the per-insert 100-row prune in addNotificationHistory that evicted quieter stacks' history whenever another stack got chatty. Periodic cleanupOldNotifications now caps per (node, stack) at 500 rows and per-node unattached system events at 1000 rows, on top of the existing 30-day retention. Signature takes an options bag and returns a per-stage summary so MonitorService can log what actually ran each cycle. - Actor attribution: thread req.user?.username through every notifyActionFailure call site and add synthetic actors at service emit sites (system:autoheal, system:scheduler, system:image-update, system:docker-events, system:blueprint, system:monitor, system:policy). The timeline renders system actors as "via <Label>" so an autoheal redeploy is no longer indistinguishable from a user redeploy. - Message sanitization: new sanitizeNotificationMessage at NotificationService.dispatchAlert strips KEY=VALUE pairs whose key ends in TOKEN/KEY/PASSWORD/SECRET/CREDENTIALS/AUTH, scrubs HTTP basic auth in URLs and Bearer tokens, collapses COMPOSE_DIR paths, and truncates to 1000 chars. Applied to the stored history and to every downstream Discord/Slack/webhook channel. The ImageUpdateService recovery-path direct DB write also runs through the sanitizer. - Composite pagination cursor: getStackActivity now accepts a (timestamp, id) cursor (?before=&beforeId=). The legacy timestamp-only form silently dropped events when a single compose up emitted many events sharing one millisecond. Route rejects beforeId without before. - Frontend hardening: distinct error state with retry button (initial fetch failure no longer renders as the genuine empty state), strict positive-integer parsing on cursor params, overrequest-by-1 pagination so the last page does not leave a dead "Load more" click, runtime guard on liveEvents merge that validates the level union, per-minute day-bucket recompute so an open panel does not stay on "Today" past midnight. No tier, role, or capability gate touched. Route permission gate remains stack:read on the named stack. * fix(stack-activity): sanitizer covers lowercase env vars and per-node compose dir External review surfaced two leak paths in the message sanitizer: - The sensitive-key regex was uppercase-only. Compose env names are conventionally uppercase but lowercase forms (db_password, jwt_secret, github_token) are valid and do leak through the same Docker and compose-parse error paths. Make the regex case-insensitive and tighten it to also catch bare TOKEN= / KEY= / PASSWORD= without a prefix word, while still leaving BYPASS, COMPASS, and similar non-secret keys alone. - The compose-dir path collapse only read process.env.COMPOSE_DIR, but the real resolution chain is node.compose_dir (per-node DB override) -> process.env.COMPOSE_DIR -> /app/compose. A node with a custom compose_dir could still leak absolute paths into stored history and downstream channels. Route both the dispatchAlert call and the ImageUpdateService recovery-path direct write through NodeRegistry.getInstance().getComposeDir(localNodeId) so the collapse covers every resolution outcome. Tests now assert lowercase keys are redacted and that BYPASS-style non-secrets stay intact in both cases. notification-routing mock extended to stub the new getComposeDir call. * chore(stack-activity): a11y roles, visibility-aware tick, live-disconnect signal Close three small follow-ups on the per-stack activity timeline: - A11y: each day-group gets role="list" and each event row gets role="listitem" so screen readers traverse the timeline as a list instead of a wall of text. The day-group container also carries an aria-label naming the bucket. - Visibility-aware day-bucket tick: the 60s setInterval that re-derives Today/Yesterday/Earlier now short-circuits when document.hidden, so a backgrounded panel does not re-render every minute for no visible effect. - Live-disconnect signal: useNotifications dispatches a sencho:notifications-connection custom event on WebSocket open and close. The timeline listens and, when explicitly disconnected, shows a one-line "Live updates offline; reconnecting…" hint above the list. The sidebar ticker already surfaces fleet-wide connection state; this adds an in-context cue for users who are focused on a single stack. Stack-name case normalization was considered and rejected: stack names are case-permissive per the isValidStackName validator, and lowercasing on read or write would silently rename or hide a user's "MyApp" stack. * ci(stack-activity): drop unnecessary escape in URL_BASIC_AUTH regex ESLint no-useless-escape errored on \- inside the character class [a-zA-Z0-9+.\-] at notificationMessage.ts:14. Move the dash to the end of the class so it's an unambiguous literal and the escape is no longer required. Behavior is identical; sanitizer tests still pass. * revert(stack-activity): drop unvalidated E2E spec from this PR The spec was committed without ever running against a real Docker daemon, then failed in CI when it ran for the first time: deploy returned 200 but no notification appeared on the activity endpoint within the polling window, suggesting either a deploy-notification race or a node-id resolution mismatch in the CI environment. Backend unit tests (route + composite cursor + sanitizer) and frontend component tests cover the same logic. The E2E spec will land in a dedicated follow-up once it has been authored against a working CI environment.
This commit is contained in:
Anso
@@ -327,7 +327,7 @@ export class AutoHealService {
|
||||
'info',
|
||||
'autoheal_triggered',
|
||||
`Auto-Heal: Restarted ${containerName} on stack ${policy.stack_name} after being unhealthy for ${policy.unhealthy_duration_mins} minute(s).`,
|
||||
{ stackName: policy.stack_name, containerName },
|
||||
{ stackName: policy.stack_name, containerName, actor: 'system:autoheal' },
|
||||
)
|
||||
.catch(err => console.error('[AutoHeal] notification dispatch failed:', err));
|
||||
} catch (err) {
|
||||
@@ -351,7 +351,7 @@ export class AutoHealService {
|
||||
'warning',
|
||||
'autoheal_triggered',
|
||||
`Auto-Heal: Failed to restart ${containerName} on stack ${policy.stack_name}. Error: ${errorMsg}`,
|
||||
{ stackName: policy.stack_name, containerName },
|
||||
{ stackName: policy.stack_name, containerName, actor: 'system:autoheal' },
|
||||
)
|
||||
.catch(e => console.error('[AutoHeal] notification dispatch failed:', e));
|
||||
|
||||
@@ -384,7 +384,7 @@ export class AutoHealService {
|
||||
'warning',
|
||||
'autoheal_triggered',
|
||||
`Auto-Heal: Policy for ${policy.stack_name}${policy.service_name ? '/' + policy.service_name : ''} has been auto-disabled after ${failures} consecutive failures.`,
|
||||
{ stackName: policy.stack_name },
|
||||
{ stackName: policy.stack_name, actor: 'system:autoheal' },
|
||||
)
|
||||
.catch(e => console.error('[AutoHeal] notification dispatch failed:', e));
|
||||
}
|
||||
|
||||
@@ -288,7 +288,7 @@ export class BlueprintReconciler {
|
||||
'warning',
|
||||
'blueprint_drift_detected',
|
||||
`Blueprint "${blueprint.name}" drifted on node "${node.name}": ${reason}`,
|
||||
{ stackName: blueprint.name },
|
||||
{ stackName: blueprint.name, actor: 'system:blueprint' },
|
||||
);
|
||||
return;
|
||||
|
||||
@@ -302,7 +302,7 @@ export class BlueprintReconciler {
|
||||
'warning',
|
||||
'blueprint_drift_detected',
|
||||
`Blueprint "${blueprint.name}" lost its marker on node "${node.name}"; auto-fix declined to avoid stomping unowned data. Reason: ${reason}`,
|
||||
{ stackName: blueprint.name },
|
||||
{ stackName: blueprint.name, actor: 'system:blueprint' },
|
||||
);
|
||||
return;
|
||||
}
|
||||
@@ -319,7 +319,7 @@ export class BlueprintReconciler {
|
||||
'error',
|
||||
'blueprint_drift_correction_failed',
|
||||
`Auto-fix for "${blueprint.name}" on node "${node.name}" failed: ${result.error ?? 'unknown error'}`,
|
||||
{ stackName: blueprint.name },
|
||||
{ stackName: blueprint.name, actor: 'system:blueprint' },
|
||||
);
|
||||
}
|
||||
return;
|
||||
|
||||
@@ -2040,13 +2040,6 @@ export class DatabaseService {
|
||||
notification.actor_username ?? null,
|
||||
);
|
||||
|
||||
this.db.prepare(`
|
||||
DELETE FROM notification_history
|
||||
WHERE node_id = ? AND id NOT IN (
|
||||
SELECT id FROM notification_history WHERE node_id = ? ORDER BY timestamp DESC LIMIT 100
|
||||
)
|
||||
`).run(nodeId, nodeId);
|
||||
|
||||
return {
|
||||
id: result.lastInsertRowid as number,
|
||||
level: notification.level,
|
||||
@@ -2060,13 +2053,21 @@ export class DatabaseService {
|
||||
};
|
||||
}
|
||||
|
||||
public getStackActivity(nodeId: number, stackName: string, opts: { limit: number; before?: number }): NotificationHistory[] {
|
||||
const sql = opts.before
|
||||
? 'SELECT * FROM notification_history WHERE node_id = ? AND stack_name = ? AND timestamp < ? ORDER BY timestamp DESC LIMIT ?'
|
||||
: 'SELECT * FROM notification_history WHERE node_id = ? AND stack_name = ? ORDER BY timestamp DESC LIMIT ?';
|
||||
const args: (number | string)[] = opts.before
|
||||
? [nodeId, stackName, opts.before, opts.limit]
|
||||
: [nodeId, stackName, opts.limit];
|
||||
public getStackActivity(nodeId: number, stackName: string, opts: { limit: number; before?: number; beforeId?: number }): NotificationHistory[] {
|
||||
// Composite (timestamp, id) cursor: pure timestamp pagination drops rows
|
||||
// on same-millisecond bursts (Docker events from one compose up).
|
||||
let sql: string;
|
||||
let args: (number | string)[];
|
||||
if (opts.before !== undefined && opts.beforeId !== undefined) {
|
||||
sql = 'SELECT * FROM notification_history WHERE node_id = ? AND stack_name = ? AND (timestamp < ? OR (timestamp = ? AND id < ?)) ORDER BY timestamp DESC, id DESC LIMIT ?';
|
||||
args = [nodeId, stackName, opts.before, opts.before, opts.beforeId, opts.limit];
|
||||
} else if (opts.before !== undefined) {
|
||||
sql = 'SELECT * FROM notification_history WHERE node_id = ? AND stack_name = ? AND timestamp < ? ORDER BY timestamp DESC, id DESC LIMIT ?';
|
||||
args = [nodeId, stackName, opts.before, opts.limit];
|
||||
} else {
|
||||
sql = 'SELECT * FROM notification_history WHERE node_id = ? AND stack_name = ? ORDER BY timestamp DESC, id DESC LIMIT ?';
|
||||
args = [nodeId, stackName, opts.limit];
|
||||
}
|
||||
return (this.db.prepare(sql).all(...args) as unknown[]).map(row => this.mapNotificationRow(row as any));
|
||||
}
|
||||
|
||||
@@ -2157,9 +2158,50 @@ export class DatabaseService {
|
||||
stmt.run(cutoff);
|
||||
}
|
||||
|
||||
public cleanupOldNotifications(daysToKeep = 30): void {
|
||||
public cleanupOldNotifications(daysToKeep = 30, opts: { perStackCap?: number; perNodeUnattachedCap?: number } = {}): { ttl: number; perStack: number; perNode: number } {
|
||||
const perStackCap = opts.perStackCap ?? 500;
|
||||
const perNodeUnattachedCap = opts.perNodeUnattachedCap ?? 1000;
|
||||
const cutoff = Date.now() - (daysToKeep * 24 * 60 * 60 * 1000);
|
||||
this.db.prepare('DELETE FROM notification_history WHERE timestamp < ?').run(cutoff);
|
||||
const ttlInfo = this.db.prepare('DELETE FROM notification_history WHERE timestamp < ?').run(cutoff);
|
||||
|
||||
const deleteById = this.db.prepare('DELETE FROM notification_history WHERE id = ?');
|
||||
const deleteMany = this.db.transaction((ids: number[]) => {
|
||||
for (const id of ids) deleteById.run(id);
|
||||
});
|
||||
|
||||
// Per (node_id, stack_name) cap so a chatty stack cannot evict a quieter stack's history.
|
||||
const stackOverflow = this.db.prepare(`
|
||||
SELECT id FROM (
|
||||
SELECT id, ROW_NUMBER() OVER (
|
||||
PARTITION BY node_id, stack_name
|
||||
ORDER BY timestamp DESC, id DESC
|
||||
) AS rn
|
||||
FROM notification_history
|
||||
WHERE stack_name IS NOT NULL
|
||||
)
|
||||
WHERE rn > ?
|
||||
`).all(perStackCap) as { id: number }[];
|
||||
if (stackOverflow.length > 0) deleteMany(stackOverflow.map(r => r.id));
|
||||
|
||||
// Unattached system events have no stack to scope by, so they cannot share the per-stack quota; cap them per-node separately.
|
||||
const unattachedOverflow = this.db.prepare(`
|
||||
SELECT id FROM (
|
||||
SELECT id, ROW_NUMBER() OVER (
|
||||
PARTITION BY node_id
|
||||
ORDER BY timestamp DESC, id DESC
|
||||
) AS rn
|
||||
FROM notification_history
|
||||
WHERE stack_name IS NULL
|
||||
)
|
||||
WHERE rn > ?
|
||||
`).all(perNodeUnattachedCap) as { id: number }[];
|
||||
if (unattachedOverflow.length > 0) deleteMany(unattachedOverflow.map(r => r.id));
|
||||
|
||||
return {
|
||||
ttl: Number(ttlInfo.changes ?? 0),
|
||||
perStack: stackOverflow.length,
|
||||
perNode: unattachedOverflow.length,
|
||||
};
|
||||
}
|
||||
|
||||
// --- Nodes ---
|
||||
|
||||
@@ -673,15 +673,15 @@ export class DockerEventService {
|
||||
// ========================================================================
|
||||
|
||||
private async emitError(category: NotificationCategory, message: string, stackName?: string, containerName?: string): Promise<void> {
|
||||
return this.notifier.dispatchAlert('error', category, this.prefix(message), { stackName, containerName });
|
||||
return this.notifier.dispatchAlert('error', category, this.prefix(message), { stackName, containerName, actor: 'system:docker-events' });
|
||||
}
|
||||
|
||||
private async emitWarning(category: NotificationCategory, message: string, stackName?: string, containerName?: string): Promise<void> {
|
||||
return this.notifier.dispatchAlert('warning', category, this.prefix(message), { stackName, containerName });
|
||||
return this.notifier.dispatchAlert('warning', category, this.prefix(message), { stackName, containerName, actor: 'system:docker-events' });
|
||||
}
|
||||
|
||||
private async emitInfo(category: NotificationCategory, message: string, stackName?: string, containerName?: string): Promise<void> {
|
||||
return this.notifier.dispatchAlert('info', category, this.prefix(message), { stackName, containerName });
|
||||
return this.notifier.dispatchAlert('info', category, this.prefix(message), { stackName, containerName, actor: 'system:docker-events' });
|
||||
}
|
||||
|
||||
private prefix(message: string): string {
|
||||
|
||||
@@ -6,6 +6,7 @@ import { FileSystemService } from './FileSystemService';
|
||||
import { RegistryService } from './RegistryService';
|
||||
import { NodeRegistry } from './NodeRegistry';
|
||||
import { NotificationService } from './NotificationService';
|
||||
import { sanitizeNotificationMessage } from '../utils/notificationMessage';
|
||||
import { parseImageRef, getRemoteDigest } from './registry-api';
|
||||
import { isDebugEnabled } from '../utils/debug';
|
||||
import { getErrorMessage } from '../utils/errors';
|
||||
@@ -308,7 +309,7 @@ export class ImageUpdateService {
|
||||
'info',
|
||||
'image_update_available',
|
||||
`[Node: ${nodeName}] Stack "${stackName}" has image updates available.`,
|
||||
{ stackName },
|
||||
{ stackName, actor: 'system:image-update' },
|
||||
);
|
||||
} catch (e) {
|
||||
console.error(`[ImageUpdateService] Failed to dispatch update notification for "${stackName}":`, e);
|
||||
@@ -316,11 +317,16 @@ export class ImageUpdateService {
|
||||
// Key on the local default: the iterated `nodeId` may be a remote's id in the
|
||||
// control plane's DB, and the UI never queries that row (it proxies instead).
|
||||
try {
|
||||
db.addNotificationHistory(NodeRegistry.getInstance().getDefaultNodeId(), {
|
||||
const localNodeId = NodeRegistry.getInstance().getDefaultNodeId();
|
||||
db.addNotificationHistory(localNodeId, {
|
||||
level: 'error',
|
||||
category: 'system',
|
||||
message: `[Node: ${nodeName}] Failed to notify about image updates for stack "${stackName}": ${getErrorMessage(e, String(e))}`,
|
||||
message: sanitizeNotificationMessage(
|
||||
`[Node: ${nodeName}] Failed to notify about image updates for stack "${stackName}": ${getErrorMessage(e, String(e))}`,
|
||||
{ composeDir: NodeRegistry.getInstance().getComposeDir(localNodeId) },
|
||||
),
|
||||
timestamp: Date.now(),
|
||||
actor_username: 'system:image-update',
|
||||
});
|
||||
} catch (dbErr) {
|
||||
console.error('[ImageUpdateService] Failed to record dispatch error:', dbErr);
|
||||
|
||||
@@ -548,10 +548,10 @@ export class MonitorService {
|
||||
const retentionHours = parseInt(settings['metrics_retention_hours'] || '24', 10);
|
||||
db.cleanupOldMetrics(isNaN(retentionHours) ? 24 : retentionHours);
|
||||
const retentionDays = parseInt(settings['log_retention_days'] || '30', 10);
|
||||
db.cleanupOldNotifications(isNaN(retentionDays) ? 30 : retentionDays);
|
||||
const notifSummary = db.cleanupOldNotifications(isNaN(retentionDays) ? 30 : retentionDays);
|
||||
const auditRetentionDays = parseInt(settings['audit_retention_days'] || '90', 10);
|
||||
db.cleanupOldAuditLogs(isNaN(auditRetentionDays) ? 90 : auditRetentionDays);
|
||||
if (isDebugEnabled()) console.log(`[Monitor:diag] Cleanup: metrics ${isNaN(retentionHours) ? 24 : retentionHours}h, notifications ${isNaN(retentionDays) ? 30 : retentionDays}d, audit ${isNaN(auditRetentionDays) ? 90 : auditRetentionDays}d`);
|
||||
if (isDebugEnabled()) console.log(`[Monitor:diag] Cleanup: metrics ${isNaN(retentionHours) ? 24 : retentionHours}h, notifications ${isNaN(retentionDays) ? 30 : retentionDays}d (ttl=${notifSummary.ttl} perStack=${notifSummary.perStack} perNode=${notifSummary.perNode}), audit ${isNaN(auditRetentionDays) ? 90 : auditRetentionDays}d`);
|
||||
} catch (e) {
|
||||
console.error('MonitorService: failed to cleanup old data', e);
|
||||
}
|
||||
@@ -670,7 +670,7 @@ export class MonitorService {
|
||||
'warning',
|
||||
'monitor_alert',
|
||||
message,
|
||||
{ stackName: rule.stack_name },
|
||||
{ stackName: rule.stack_name, actor: 'system:monitor' },
|
||||
);
|
||||
|
||||
db.updateStackAlertLastFired(ruleId, Date.now());
|
||||
@@ -723,7 +723,7 @@ export class MonitorService {
|
||||
const db = DatabaseService.getInstance();
|
||||
const last = parseInt(db.getSystemState(stateKey) || '0', 10);
|
||||
if (Date.now() - last > cooldownMs) {
|
||||
await NotificationService.getInstance().dispatchAlert(severity, category, message, { stackName: stack });
|
||||
await NotificationService.getInstance().dispatchAlert(severity, category, message, { stackName: stack, actor: 'system:monitor' });
|
||||
db.setSystemState(stateKey, Date.now().toString());
|
||||
}
|
||||
}
|
||||
|
||||
@@ -4,6 +4,7 @@ import { NodeRegistry } from './NodeRegistry';
|
||||
import { isDebugEnabled } from '../utils/debug';
|
||||
import { getErrorMessage } from '../utils/errors';
|
||||
import { sanitizeForLog } from '../utils/safeLog';
|
||||
import { sanitizeNotificationMessage } from '../utils/notificationMessage';
|
||||
|
||||
export type NotificationCategory =
|
||||
| 'deploy_success'
|
||||
@@ -119,10 +120,15 @@ export class NotificationService {
|
||||
// with user-initiated requests; otherwise the UI and monitors split
|
||||
// between different node_id buckets.
|
||||
const localNodeId = NodeRegistry.getInstance().getDefaultNodeId();
|
||||
// Use the full resolution chain (node.compose_dir → env → default)
|
||||
// so messages mentioning a per-node compose override get collapsed.
|
||||
const sanitized = sanitizeNotificationMessage(message, {
|
||||
composeDir: NodeRegistry.getInstance().getComposeDir(localNodeId),
|
||||
});
|
||||
const notification = this.dbService.addNotificationHistory(localNodeId, {
|
||||
level,
|
||||
category,
|
||||
message,
|
||||
message: sanitized,
|
||||
timestamp: Date.now(),
|
||||
stack_name: stackName,
|
||||
container_name: containerName,
|
||||
@@ -151,7 +157,7 @@ export class NotificationService {
|
||||
if (isDebugEnabled()) console.log(`[Notify:diag] Matched ${matched.length} route(s) for stack "${sanitizeForLog(stackName ?? '(none)')}", category="${sanitizeForLog(category)}"`);
|
||||
await Promise.allSettled(
|
||||
matched.map(route =>
|
||||
this.sendToChannel(route.channel_type, route.channel_url, level, message)
|
||||
this.sendToChannel(route.channel_type, route.channel_url, level, sanitized)
|
||||
.then(() => {
|
||||
if (isDebugEnabled()) console.log(`[Notify:diag] Dispatched ${level} via route "${route.name}" (${route.channel_type})`);
|
||||
})
|
||||
@@ -176,7 +182,7 @@ export class NotificationService {
|
||||
if (isDebugEnabled()) console.log(`[Notify:diag] Falling back to ${agents.length} global agent(s)`);
|
||||
await Promise.allSettled(
|
||||
agents.map(agent =>
|
||||
this.sendToChannel(agent.type, agent.url, level, message)
|
||||
this.sendToChannel(agent.type, agent.url, level, sanitized)
|
||||
.then(() => {
|
||||
if (isDebugEnabled()) console.log(`[Notify:diag] Dispatched ${level} via global agent (${agent.type})`);
|
||||
})
|
||||
|
||||
@@ -66,7 +66,7 @@ function notifyTrivyMissingOnce(nodeId: number, stackName: string): void {
|
||||
'warning',
|
||||
'scan_finding',
|
||||
`Pre-deploy scan for "${stackName}" skipped: Trivy not installed on this node`,
|
||||
{ stackName },
|
||||
{ stackName, actor: 'system:policy' },
|
||||
);
|
||||
}
|
||||
|
||||
|
||||
@@ -167,7 +167,7 @@ export class SchedulerService {
|
||||
*/
|
||||
private safeDispatch(level: 'info' | 'warning' | 'error', category: import('./NotificationService').NotificationCategory, message: string, stackName?: string): void {
|
||||
NotificationService.getInstance()
|
||||
.dispatchAlert(level, category, message, { stackName })
|
||||
.dispatchAlert(level, category, message, { stackName, actor: 'system:scheduler' })
|
||||
.catch(err => console.error('[SchedulerService] Notification dispatch failed:', getErrorMessage(err, 'unknown error')));
|
||||
}
|
||||
|
||||
@@ -778,6 +778,7 @@ export class SchedulerService {
|
||||
'warning',
|
||||
'scan_finding',
|
||||
`Policy "${v.policyName}" violated by ${v.imageRef}: ${v.severity} exceeds ${v.maxSeverity}`,
|
||||
{ actor: 'system:scheduler' },
|
||||
);
|
||||
}
|
||||
|
||||
|
||||
Reference in New Issue
Block a user