mirror of
https://github.com/rcourtman/Pulse.git
synced 2026-09-10 02:25:56 +00:00
Add a user-facing alert delivery log and honest test-send results
The delivery health verdict said when something was wrong; nothing showed what actually fired and where it went. Expose the queue's retained per-attempt audit rows as GET /api/notifications/delivery-log (newest first, retention-labeled, webhook secrets redacted from error text) and render them as a Recent delivery activity card on the alert destinations tab, with outcome badges, destination names, and failure classes. Audit rows now persist the normalized destination identity in a destination_id column; older rows fall back to their operational links. Test sends bypass both the queue and the activation gate, which is exactly how installs came to believe delivery worked while every real alert was suppressed (483 installs in the 08-18 telemetry read). Successful test responses now carry deliveryPaused: true whenever the manager is gated off, and the destinations UI warns instead of celebrating.
This commit is contained in:
@@ -6392,3 +6392,14 @@ has been applied; it cannot select an agent, add an agent-routed tool, establish
|
||||
a command session, or bypass capability, feasibility, policy, approval,
|
||||
dispatch, receipt, and verification gates. Unknown types retain the existing
|
||||
governed Patrol profile rather than guessing at agent authority.
|
||||
|
||||
### Notification delivery-log routes do not touch agent lifecycle
|
||||
|
||||
The shared `internal/api/` surface now includes the notifications-owned
|
||||
read-only `GET /api/notifications/delivery-log` route and a `deliveryPaused`
|
||||
field on successful test-send responses. Both are notification delivery
|
||||
evidence only: they read retained notification audit rows and the manager's
|
||||
enabled gate, and they do not enroll or select an agent, open or alter a
|
||||
command session, change fleet policy or update state, or add any
|
||||
agent-routed capability. Agent lifecycle obligations over `internal/api/`
|
||||
are unchanged by this surface.
|
||||
|
||||
@@ -1722,3 +1722,21 @@ cannot recreate them while suppression remains active. Existing prefix, tag,
|
||||
and `pulse-no-alerts` bulk rules remain compatible inputs and do not create a
|
||||
second per-resource state store. `internal/alerts/intent_policy_test.go` pins
|
||||
factory operator-state evaluation, writer gating, and active reconciliation.
|
||||
|
||||
### Destinations tab carries delivery evidence
|
||||
|
||||
The alert destinations tab is where the belief "delivery works" is formed, so
|
||||
it must carry delivery evidence, not only configuration. It renders the
|
||||
notifications-owned delivery log (`AlertDeliveryLogCard` fed by
|
||||
`useNotificationDeliveryLog` over `GET /api/notifications/delivery-log`) as a
|
||||
newest-first record of real alert delivery attempts with plain-language
|
||||
outcome labels, destination names resolved from the loaded webhook configs,
|
||||
failure classes, and secret-redacted error text. The card names its retention
|
||||
window and says that test sends skip the queue, and an unreadable log renders
|
||||
as unavailable rather than empty. Test-send results that report
|
||||
`deliveryPaused` surface as a warning toast, never plain success, across the
|
||||
email, Apprise, and webhook test actions in `useAlertDestinationsTabState`
|
||||
and `useAlertWebhookDestinationsState`. Delivery evidence remains
|
||||
notification truth: the delivery log card must not resolve, suppress, or
|
||||
re-evaluate alerts, and it does not alter the `AlertConfig.enabled` versus
|
||||
`activationState` ownership boundary above.
|
||||
|
||||
@@ -4321,6 +4321,22 @@ it, and an update that echoes the masked placeholder must preserve the stored
|
||||
secret rather than overwriting it, matching the header and custom-field
|
||||
secret-handling rules above.
|
||||
|
||||
The notifications boundary additionally owns the delivery-log and
|
||||
test-send-honesty payload contracts. `GET /api/notifications/delivery-log`
|
||||
(settings-read scope) returns retained per-attempt delivery outcomes newest
|
||||
first — `entries` carrying `notificationId`, `type`, `destinationId`,
|
||||
`outcome` (`sent`/`retry`/`failed`/`dead_letter`/`cancelled`), `alertIds`,
|
||||
`attempts`, `success`, `failureClass`, redacted `errorMessage`, and RFC 3339
|
||||
`timestamp` — alongside `window_days` and retention metadata, because the
|
||||
counts are retention-bounded and must not read as lifetime history; webhook
|
||||
secrets are redacted from error text before the payload leaves the API. The
|
||||
frontend transport in `frontend-modern/src/api/notifications.ts` validates
|
||||
entry shape and drops malformed rows rather than rendering them. Successful
|
||||
test-send responses from `POST /api/notifications/test` and
|
||||
`POST /api/notifications/webhooks/test` must include `deliveryPaused: true`
|
||||
whenever real alert delivery is gated off, so a passing test cannot be
|
||||
presented as proof that live alerts flow.
|
||||
|
||||
Proxmox setup bootstrap now includes a non-destructive Audit/Repair path in the
|
||||
generated PVE script and existing-source setup guide. That path audits token
|
||||
presence, token expiry, Pulse-managed token drift, and expected ACLs, then
|
||||
|
||||
@@ -6247,3 +6247,15 @@ Proofs live in `frontend-modern/src/components/shared/__tests__/MobileNavBar.tes
|
||||
`frontend-modern/src/components/__tests__/GitHubStarBanner.test.tsx`, and
|
||||
`frontend-modern/src/__tests__/App.architecture.test.ts`, which fails if any
|
||||
runtime source reintroduces a literal bar height.
|
||||
|
||||
### Alert delivery log presentation
|
||||
|
||||
The destinations-tab delivery log renders through the shared `Card` primitive
|
||||
in the feature-owned `AlertDeliveryLogCard`. Outcome badges use the
|
||||
plain-language labels from `alertDestinationsPresentation` rather than queue
|
||||
vocabulary ("Failed, retries exhausted", never "dead letter"), failure detail
|
||||
lines use the shared red emphasis tokens in both themes, and entry rows wrap
|
||||
without horizontal overflow at mobile widths. The unavailable state is a
|
||||
`role="alert"` message distinct from the empty state, because "cannot read
|
||||
the log" and "no attempts" mean opposite things to someone deciding whether
|
||||
to trust their alerting.
|
||||
|
||||
@@ -325,6 +325,34 @@ counts cannot be read. Pending retries do not degrade health. The response
|
||||
must expose fixed reason codes and retention metadata rather than raw queue
|
||||
errors or notification content.
|
||||
|
||||
### User-facing delivery log and honest test sends
|
||||
|
||||
The queue owner exposes its retained per-attempt audit rows to the local
|
||||
notification-management API as a bounded, newest-first delivery log
|
||||
(`GetDeliveryLog` in `internal/notifications/delivery_log.go`). Each entry
|
||||
carries the attempt outcome (`sent`, `retry`, `failed`, `dead_letter`,
|
||||
`cancelled`), derived by the same single rule that feeds the delivery outcome
|
||||
metric in `RecordAudit`, plus alert identifiers, normalized destination
|
||||
identity, attempt count, failure class, error text, and timestamp. Audit rows
|
||||
persist destination identity in a dedicated `destination_id` column; rows
|
||||
written before that column existed resolve destination identity from their
|
||||
retained operational links. `GET /api/notifications/delivery-log`
|
||||
(settings-read scope) serves the log as local operator evidence: webhook
|
||||
secrets are redacted from error text at the API boundary, the payload names
|
||||
its retention window instead of presenting itself as lifetime history, and an
|
||||
unreadable queue is an error, never an empty log. This per-attempt surface is
|
||||
deliberately distinct from the content-free telemetry aggregate above, which
|
||||
remains identity-free. Test sends bypass the queue and must not appear in the
|
||||
delivery log, and the destinations UI says so where the log renders.
|
||||
|
||||
Because test sends also bypass the alert activation gate, a bare success
|
||||
result is exactly how installs come to believe delivery works while every
|
||||
real alert is suppressed. Successful responses from
|
||||
`POST /api/notifications/test` and `POST /api/notifications/webhooks/test`
|
||||
must therefore report `deliveryPaused: true` whenever the notification
|
||||
manager is disabled, and the destinations UI must surface that as a warning
|
||||
instead of a plain success toast.
|
||||
|
||||
### Occurrence-bound delivery receipts
|
||||
|
||||
The notification owner records successful firing delivery by exact alert ID,
|
||||
|
||||
@@ -5330,3 +5330,13 @@ outside the already-projected Patrol profile and grants no plan, approval,
|
||||
execution, retention, cleanup, restore, or verification authority. An unknown
|
||||
resource type keeps the normal governed profile instead of inferring storage
|
||||
authority from prompt text.
|
||||
|
||||
### Notification delivery-log routes do not confer storage authority
|
||||
|
||||
The shared `internal/api/` surface now includes the notifications-owned
|
||||
read-only `GET /api/notifications/delivery-log` route and a `deliveryPaused`
|
||||
field on successful test-send responses. They expose retained notification
|
||||
delivery outcomes and the delivery gate state only; they add no storage,
|
||||
backup, snapshot, restore, retention, cleanup, or recovery capability, write
|
||||
nothing, and leave storage and recovery state, evidence freshness,
|
||||
persistence, and admission contracts unchanged.
|
||||
|
||||
@@ -1,16 +1,28 @@
|
||||
{
|
||||
"version": 1,
|
||||
"base_sha": "fd35c547730a62ad7118de6cf898408f308d8317",
|
||||
"verified_at": "2026-08-20T16:29:56Z",
|
||||
"base_sha": "a5f1567172314003ee1b96b93090ffcbbe18e8f6",
|
||||
"verified_at": "2026-08-20T16:20:02Z",
|
||||
"result": "passed",
|
||||
"changed_paths": [
|
||||
"frontend-modern/src/i18n/messages.de.ts"
|
||||
"frontend-modern/src/api/notifications.ts",
|
||||
"frontend-modern/src/features/alerts/AlertDeliveryLogCard.tsx",
|
||||
"frontend-modern/src/features/alerts/tabs/DestinationsTab.tsx",
|
||||
"frontend-modern/src/features/alerts/useAlertDestinationsTabState.ts",
|
||||
"frontend-modern/src/features/alerts/useAlertWebhookDestinationsState.ts",
|
||||
"frontend-modern/src/features/alerts/useNotificationDeliveryLog.ts",
|
||||
"frontend-modern/src/utils/alertDestinationsPresentation.ts"
|
||||
],
|
||||
"content_sha256": {
|
||||
"frontend-modern/src/i18n/messages.de.ts": "0adaabff61c0ed7fb9064ea05f9d83595ab673958249d6cb5b0e6bc212a3ca46"
|
||||
"frontend-modern/src/api/notifications.ts": "5bd796b6ec9222d6973a4a26b892b4fcfde11b2d9dd2e2ef705ad34531af0071",
|
||||
"frontend-modern/src/features/alerts/AlertDeliveryLogCard.tsx": "7e7e4c5da2a4ea8c267c546da0064806949573bc57a51cf7c5b5f5f8576826ef",
|
||||
"frontend-modern/src/features/alerts/tabs/DestinationsTab.tsx": "0411429355fd78e75948df8ede39da75f9f9e3600840d9b98f7342d2adbfad7b",
|
||||
"frontend-modern/src/features/alerts/useAlertDestinationsTabState.ts": "c88336576ec08b69271244947f7892cac0b77e9ec68fdf33a12fe0f130e4e29d",
|
||||
"frontend-modern/src/features/alerts/useAlertWebhookDestinationsState.ts": "b9c959388a06ce146f3706cbc2b191449d4dd5380198d711ed15fdf06fdc48b9",
|
||||
"frontend-modern/src/features/alerts/useNotificationDeliveryLog.ts": "48ca901ec9435d94a06de5e55bf05dbf254cfd23114cd2b31383909d1585237b",
|
||||
"frontend-modern/src/utils/alertDestinationsPresentation.ts": "8a2e9446dfbdd77d46ef6a678ff2aa403741e3d296ac8ef67ba708c8940a9b2a"
|
||||
},
|
||||
"routes": [
|
||||
"/settings/system-general"
|
||||
"/alerts/notifications"
|
||||
],
|
||||
"viewports": [
|
||||
{
|
||||
@@ -23,12 +35,19 @@
|
||||
}
|
||||
],
|
||||
"states": [
|
||||
"German locale (pulseLocalePreference=de): Docker in Proxmox-LXCs card on the general settings panel renders the guestDocker toggle description with the corrected sentence break 'werden uebersprungen. Pulse erfasst nur Containerliste und Auslastung' in place of the semicolon the copy-style audit rejects",
|
||||
"Mobile viewport: the same description renders with no horizontal overflow (documentElement.scrollWidth == 375)",
|
||||
"Receipt re-pinned from base 23304bc01 to fd35c5477 during rebase: the staged messages.de.ts blob is byte-identical to the exercised content, and the new parent does not touch messages.de.ts"
|
||||
"Fresh-install destinations tab (activation pending): AlertDeliveryPausedCard at top, Recent delivery activity card at bottom with honest empty state naming the 7-day window and the test-sends-skip-the-queue caveat",
|
||||
"Delivery log with seeded audit rows served by the sidecar-built backend at GET /api/notifications/delivery-log: newest-first entries with Failed-retries-exhausted (red), Retrying (amber), Delivered (green) outcome badges",
|
||||
"Webhook destination id (webhook:<id>) resolved to the configured webhook name 'Ops Sink'; email entry labeled 'Email' with its opaque destination hash",
|
||||
"Failure detail line renders 'Authentication failure: HTTP 401 Unauthorized from https://hooks.example.test/notify?token=REDACTED' proving API-layer webhook secret redaction (seeded value was token=supersecret)",
|
||||
"Mobile viewport: card and badges wrap with no horizontal overflow (documentElement.scrollWidth == 375)",
|
||||
"No console errors on the destinations tab after load, test send, and delivery log refresh",
|
||||
"Receipt re-pinned during rebase onto a5f156717 (copy-fix replayed onto fd35c5477 canonical search work): every staged blob hash above is byte-identical to the exercised content, and neither upstream commit touches any of these files"
|
||||
],
|
||||
"interactions": [
|
||||
"Set localStorage pulseLocalePreference=de on the sidecar-built stack (backend :7811, vite :5487), navigated to /settings/system-general, and located the rendered description text via a DOM text walk at desktop width",
|
||||
"Resized to 375x812 and re-located the corrected text on the same panel"
|
||||
"Logged into the isolated sidecar-built stack (backend :7811, vite :5487), opened Alerts > Notifications",
|
||||
"Added webhook 'Ops Sink' -> http://127.0.0.1:18712/hook (allowlisted 127.0.0.1/32) and clicked Test: local sink logged the POST 200 and the UI raised the warning toast 'Test sent, but notification delivery is paused: real alerts are not being sent' instead of plain success, because the install's activation gate is pending",
|
||||
"Saved the webhook via Add Webhook and confirmed GET /api/notifications/webhooks returns it",
|
||||
"Seeded notification_audit rows (sent email, retry webhook, dead-letter webhook with token-bearing error text), clicked Refresh delivery status, and verified the three entries rendered newest-first with redacted error text",
|
||||
"Resized to 375x812 and re-exercised the delivery log card"
|
||||
]
|
||||
}
|
||||
|
||||
@@ -291,4 +291,87 @@ describe('NotificationsAPI', () => {
|
||||
}),
|
||||
]);
|
||||
});
|
||||
|
||||
it('normalizes the delivery log payload and drops malformed entries', async () => {
|
||||
apiFetchJSONMock.mockResolvedValueOnce({
|
||||
entries: [
|
||||
{
|
||||
notificationId: 'webhook-1',
|
||||
type: 'webhook',
|
||||
destinationId: 'webhook:wh-ops',
|
||||
outcome: 'dead_letter',
|
||||
alertIds: ['vm-offline-101'],
|
||||
alertCount: 1,
|
||||
attempts: 3,
|
||||
success: false,
|
||||
errorMessage: 'HTTP 401 Unauthorized',
|
||||
failureClass: 'authentication',
|
||||
timestamp: '2026-08-20T12:00:00Z',
|
||||
},
|
||||
// Malformed rows must be dropped, not rendered: unknown outcome,
|
||||
// missing timestamp, and a non-object entry.
|
||||
{
|
||||
notificationId: 'bad-outcome',
|
||||
type: 'email',
|
||||
outcome: 'exploded',
|
||||
timestamp: '2026-08-20T12:00:00Z',
|
||||
},
|
||||
{ notificationId: 'no-timestamp', type: 'email', outcome: 'sent' },
|
||||
'not-an-object',
|
||||
{
|
||||
notificationId: 'email-1',
|
||||
type: 'email',
|
||||
outcome: 'sent',
|
||||
alertIds: ['disk-critical-1'],
|
||||
alertCount: 1,
|
||||
attempts: 1,
|
||||
success: true,
|
||||
// An unrecognized failure class is omitted rather than passed through.
|
||||
failureClass: 'made-up',
|
||||
timestamp: '2026-08-20T11:00:00Z',
|
||||
},
|
||||
],
|
||||
window_days: 7,
|
||||
} as any);
|
||||
|
||||
const log = await NotificationsAPI.getDeliveryLog(25);
|
||||
|
||||
expect(apiFetchJSONMock).toHaveBeenCalledWith('/api/notifications/delivery-log?limit=25');
|
||||
expect(log.windowDays).toBe(7);
|
||||
expect(log.entries).toHaveLength(2);
|
||||
expect(log.entries[0]).toEqual(
|
||||
expect.objectContaining({
|
||||
notificationId: 'webhook-1',
|
||||
outcome: 'dead_letter',
|
||||
destinationId: 'webhook:wh-ops',
|
||||
failureClass: 'authentication',
|
||||
errorMessage: 'HTTP 401 Unauthorized',
|
||||
}),
|
||||
);
|
||||
expect(log.entries[1].failureClass).toBeUndefined();
|
||||
});
|
||||
|
||||
it('requests the delivery log without a limit query when none is given', async () => {
|
||||
apiFetchJSONMock.mockResolvedValueOnce({ entries: [], window_days: 0 } as any);
|
||||
|
||||
const log = await NotificationsAPI.getDeliveryLog();
|
||||
|
||||
expect(apiFetchJSONMock).toHaveBeenCalledWith('/api/notifications/delivery-log');
|
||||
expect(log.entries).toEqual([]);
|
||||
// A missing or nonsensical window falls back to the seven-day default.
|
||||
expect(log.windowDays).toBe(7);
|
||||
});
|
||||
|
||||
it('passes the deliveryPaused flag through from test-send responses', async () => {
|
||||
apiFetchJSONMock.mockResolvedValueOnce({
|
||||
status: 'success',
|
||||
message: 'Test notification sent, but alert delivery is paused',
|
||||
deliveryPaused: true,
|
||||
} as any);
|
||||
|
||||
const result = await NotificationsAPI.testNotification({ type: 'email' });
|
||||
|
||||
expect(result.deliveryPaused).toBe(true);
|
||||
expect(result.status).toBe('success');
|
||||
});
|
||||
});
|
||||
|
||||
@@ -129,6 +129,37 @@ export interface NotificationHealth {
|
||||
queue: NotificationQueueHealth;
|
||||
}
|
||||
|
||||
export type NotificationDeliveryOutcome = 'sent' | 'retry' | 'failed' | 'dead_letter' | 'cancelled';
|
||||
|
||||
export interface NotificationDeliveryLogEntry {
|
||||
notificationId: string;
|
||||
type: string;
|
||||
method?: string;
|
||||
destinationId?: string;
|
||||
outcome: NotificationDeliveryOutcome;
|
||||
alertIds: string[];
|
||||
alertCount: number;
|
||||
attempts: number;
|
||||
success: boolean;
|
||||
errorMessage?: string;
|
||||
failureClass?: NotificationFailureClass;
|
||||
timestamp: string;
|
||||
}
|
||||
|
||||
export interface NotificationDeliveryLog {
|
||||
entries: NotificationDeliveryLogEntry[];
|
||||
windowDays: number;
|
||||
}
|
||||
|
||||
export interface NotificationTestResult {
|
||||
status?: string;
|
||||
message?: string;
|
||||
// True when the test was delivered but real alert delivery is paused by the
|
||||
// activation gate, so a successful test must not be read as proof that live
|
||||
// alerts are getting through.
|
||||
deliveryPaused?: boolean;
|
||||
}
|
||||
|
||||
function apiRecord(value: unknown): Record<string, unknown> {
|
||||
return value && typeof value === 'object' ? (value as Record<string, unknown>) : {};
|
||||
}
|
||||
@@ -154,6 +185,49 @@ const notificationFailureClasses: NotificationFailureClass[] = [
|
||||
'unknown',
|
||||
];
|
||||
|
||||
const notificationDeliveryOutcomes: NotificationDeliveryOutcome[] = [
|
||||
'sent',
|
||||
'retry',
|
||||
'failed',
|
||||
'dead_letter',
|
||||
'cancelled',
|
||||
];
|
||||
|
||||
function normalizeDeliveryLogEntry(value: unknown): NotificationDeliveryLogEntry | undefined {
|
||||
const record = apiRecord(value);
|
||||
const notificationId = strictString(record.notificationId);
|
||||
const outcome = record.outcome;
|
||||
const timestamp = strictString(record.timestamp);
|
||||
if (
|
||||
!notificationId ||
|
||||
!timestamp ||
|
||||
!notificationDeliveryOutcomes.includes(outcome as NotificationDeliveryOutcome)
|
||||
) {
|
||||
return undefined;
|
||||
}
|
||||
const failureClass = record.failureClass;
|
||||
const entry: NotificationDeliveryLogEntry = {
|
||||
notificationId,
|
||||
type: strictString(record.type),
|
||||
outcome: outcome as NotificationDeliveryOutcome,
|
||||
alertIds: stringArray(record.alertIds),
|
||||
alertCount: nonNegativeCount(record.alertCount) ?? 0,
|
||||
attempts: nonNegativeCount(record.attempts) ?? 0,
|
||||
success: strictBoolean(record.success),
|
||||
timestamp,
|
||||
};
|
||||
const method = strictString(record.method);
|
||||
if (method) entry.method = method;
|
||||
const destinationId = strictString(record.destinationId);
|
||||
if (destinationId) entry.destinationId = destinationId;
|
||||
const errorMessage = strictString(record.errorMessage);
|
||||
if (errorMessage) entry.errorMessage = errorMessage;
|
||||
if (notificationFailureClasses.includes(failureClass as NotificationFailureClass)) {
|
||||
entry.failureClass = failureClass as NotificationFailureClass;
|
||||
}
|
||||
return entry;
|
||||
}
|
||||
|
||||
function normalizeFailureClassCounts(value: unknown): NotificationFailureClassCounts | undefined {
|
||||
const record = apiRecord(value);
|
||||
const counts = {} as NotificationFailureClassCounts;
|
||||
@@ -260,6 +334,24 @@ export class NotificationsAPI {
|
||||
};
|
||||
}
|
||||
|
||||
static async getDeliveryLog(limit?: number): Promise<NotificationDeliveryLog> {
|
||||
const query = limit && limit > 0 ? `?limit=${limit}` : '';
|
||||
const payload = await apiFetchJSON<Record<string, unknown>>(
|
||||
`${this.baseUrl}/delivery-log${query}`,
|
||||
);
|
||||
const rawEntries = Array.isArray(payload.entries) ? payload.entries : [];
|
||||
const entries: NotificationDeliveryLogEntry[] = [];
|
||||
for (const rawEntry of rawEntries) {
|
||||
const entry = normalizeDeliveryLogEntry(rawEntry);
|
||||
if (entry) entries.push(entry);
|
||||
}
|
||||
const windowDays = nonNegativeCount(payload.window_days);
|
||||
return {
|
||||
entries,
|
||||
windowDays: windowDays && windowDays > 0 ? windowDays : 7,
|
||||
};
|
||||
}
|
||||
|
||||
static async updateAppriseConfig(config: AppriseConfig): Promise<AppriseConfig> {
|
||||
return apiFetchJSON(`${this.baseUrl}/apprise`, {
|
||||
method: 'PUT',
|
||||
@@ -363,9 +455,7 @@ export class NotificationsAPI {
|
||||
}
|
||||
|
||||
// Testing
|
||||
static async testNotification(
|
||||
request: NotificationTestRequest,
|
||||
): Promise<{ success: boolean; message?: string }> {
|
||||
static async testNotification(request: NotificationTestRequest): Promise<NotificationTestResult> {
|
||||
const body: {
|
||||
method: string;
|
||||
config?: Record<string, unknown> | AppriseConfig;
|
||||
@@ -392,7 +482,7 @@ export class NotificationsAPI {
|
||||
|
||||
static async testWebhook(
|
||||
webhook: Omit<Webhook, 'id'>,
|
||||
): Promise<{ success: boolean; message?: string }> {
|
||||
): Promise<{ success?: boolean; message?: string; deliveryPaused?: boolean }> {
|
||||
return apiFetchJSON(`${this.baseUrl}/webhooks/test`, {
|
||||
method: 'POST',
|
||||
body: JSON.stringify(webhook),
|
||||
|
||||
@@ -0,0 +1,114 @@
|
||||
import { cleanup, render, screen } from '@solidjs/testing-library';
|
||||
import { afterEach, describe, expect, it, vi } from 'vitest';
|
||||
|
||||
import type { NotificationDeliveryLog, Webhook } from '@/api/notifications';
|
||||
|
||||
import { AlertDeliveryLogCard } from './AlertDeliveryLogCard';
|
||||
|
||||
const webhooks: Webhook[] = [
|
||||
{
|
||||
id: 'wh-ops',
|
||||
name: 'Ops Discord',
|
||||
url: 'https://discord.example.test/hook',
|
||||
method: 'POST',
|
||||
headers: {},
|
||||
enabled: true,
|
||||
},
|
||||
];
|
||||
|
||||
const log: NotificationDeliveryLog = {
|
||||
entries: [
|
||||
{
|
||||
notificationId: 'webhook-1',
|
||||
type: 'webhook',
|
||||
destinationId: 'webhook:wh-ops',
|
||||
outcome: 'dead_letter',
|
||||
alertIds: ['vm-offline-101', 'vm-offline-102'],
|
||||
alertCount: 2,
|
||||
attempts: 3,
|
||||
success: false,
|
||||
errorMessage: 'HTTP 401 Unauthorized',
|
||||
failureClass: 'authentication',
|
||||
timestamp: new Date(Date.now() - 60_000).toISOString(),
|
||||
},
|
||||
{
|
||||
notificationId: 'email-1',
|
||||
type: 'email',
|
||||
destinationId: 'destination:abcd',
|
||||
outcome: 'sent',
|
||||
alertIds: ['disk-critical-1'],
|
||||
alertCount: 1,
|
||||
attempts: 1,
|
||||
success: true,
|
||||
timestamp: new Date(Date.now() - 120_000).toISOString(),
|
||||
},
|
||||
],
|
||||
windowDays: 7,
|
||||
};
|
||||
|
||||
describe('AlertDeliveryLogCard', () => {
|
||||
afterEach(() => cleanup());
|
||||
|
||||
it('shows each attempt with a plain-language outcome and the resolved destination name', () => {
|
||||
render(() => (
|
||||
<AlertDeliveryLogCard
|
||||
log={log}
|
||||
unavailable={false}
|
||||
refreshing={false}
|
||||
onRefresh={vi.fn()}
|
||||
webhooks={webhooks}
|
||||
/>
|
||||
));
|
||||
|
||||
expect(screen.getByText('Ops Discord')).toBeInTheDocument();
|
||||
expect(screen.getByText('Email')).toBeInTheDocument();
|
||||
expect(screen.getByText('Failed, retries exhausted')).toBeInTheDocument();
|
||||
expect(screen.getByText('Delivered')).toBeInTheDocument();
|
||||
expect(screen.getByText('vm-offline-101 +1 more')).toBeInTheDocument();
|
||||
expect(screen.getByText(/Authentication failure/)).toBeInTheDocument();
|
||||
expect(screen.getByText(/HTTP 401 Unauthorized/)).toBeInTheDocument();
|
||||
});
|
||||
|
||||
it('says test sends are not listed so their absence is not read as failure', () => {
|
||||
render(() => (
|
||||
<AlertDeliveryLogCard
|
||||
log={log}
|
||||
unavailable={false}
|
||||
refreshing={false}
|
||||
onRefresh={vi.fn()}
|
||||
webhooks={webhooks}
|
||||
/>
|
||||
));
|
||||
|
||||
expect(screen.getByText(/Test sends skip the queue/)).toBeInTheDocument();
|
||||
});
|
||||
|
||||
it('renders an honest empty state when no deliveries were attempted', () => {
|
||||
render(() => (
|
||||
<AlertDeliveryLogCard
|
||||
log={{ entries: [], windowDays: 7 }}
|
||||
unavailable={false}
|
||||
refreshing={false}
|
||||
onRefresh={vi.fn()}
|
||||
webhooks={[]}
|
||||
/>
|
||||
));
|
||||
|
||||
expect(screen.getByText(/No alert deliveries were attempted/)).toBeInTheDocument();
|
||||
});
|
||||
|
||||
it('reports an unreadable log as unavailable instead of empty', () => {
|
||||
render(() => (
|
||||
<AlertDeliveryLogCard
|
||||
log={null}
|
||||
unavailable={true}
|
||||
refreshing={false}
|
||||
onRefresh={vi.fn()}
|
||||
webhooks={[]}
|
||||
/>
|
||||
));
|
||||
|
||||
expect(screen.getByRole('alert')).toHaveTextContent(/could not read the delivery log/);
|
||||
expect(screen.queryByText(/No alert deliveries were attempted/)).not.toBeInTheDocument();
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,155 @@
|
||||
import { For, Show } from 'solid-js';
|
||||
import RefreshCwIcon from 'lucide-solid/icons/refresh-cw';
|
||||
|
||||
import type {
|
||||
NotificationDeliveryLog,
|
||||
NotificationDeliveryLogEntry,
|
||||
Webhook,
|
||||
} from '@/api/notifications';
|
||||
import { Card } from '@/components/shared/Card';
|
||||
import {
|
||||
getAlertDeliveryLogFailureClassLabel,
|
||||
getAlertDeliveryLogOutcomeLabel,
|
||||
getAlertDestinationsDeliveryLogDescription,
|
||||
getAlertDestinationsDeliveryLogEmpty,
|
||||
getAlertDestinationsDeliveryLogTitle,
|
||||
getAlertDestinationsDeliveryLogUnavailable,
|
||||
getAlertDestinationsDeliveryRefreshLabel,
|
||||
} from '@/utils/alertDestinationsPresentation';
|
||||
import { formatRelativeTime } from '@/utils/format';
|
||||
|
||||
interface AlertDeliveryLogCardProps {
|
||||
log: NotificationDeliveryLog | null;
|
||||
unavailable: boolean;
|
||||
refreshing: boolean;
|
||||
onRefresh: () => void;
|
||||
webhooks: Webhook[];
|
||||
}
|
||||
|
||||
const WEBHOOK_DESTINATION_PREFIX = 'webhook:';
|
||||
|
||||
const outcomeBadgeClasses: Record<NotificationDeliveryLogEntry['outcome'], string> = {
|
||||
sent: 'bg-green-100 text-green-800 dark:bg-green-900/40 dark:text-green-300',
|
||||
retry: 'bg-amber-100 text-amber-800 dark:bg-amber-900/40 dark:text-amber-300',
|
||||
failed: 'bg-red-100 text-red-800 dark:bg-red-900/40 dark:text-red-300',
|
||||
dead_letter: 'bg-red-100 text-red-800 dark:bg-red-900/40 dark:text-red-300',
|
||||
cancelled: 'bg-gray-100 text-gray-600 dark:bg-gray-700/60 dark:text-gray-300',
|
||||
};
|
||||
|
||||
export function AlertDeliveryLogCard(props: AlertDeliveryLogCardProps) {
|
||||
// Destination ids are opaque on purpose (hashes for email/apprise, config
|
||||
// ids for webhooks); only webhook ids can be resolved to a configured name.
|
||||
const destinationLabel = (entry: NotificationDeliveryLogEntry): string => {
|
||||
if (entry.destinationId?.startsWith(WEBHOOK_DESTINATION_PREFIX)) {
|
||||
const webhookId = entry.destinationId.slice(WEBHOOK_DESTINATION_PREFIX.length);
|
||||
const webhook = props.webhooks.find((candidate) => candidate.id === webhookId);
|
||||
if (webhook?.name) return webhook.name;
|
||||
}
|
||||
switch (entry.type) {
|
||||
case 'email':
|
||||
return 'Email';
|
||||
case 'apprise':
|
||||
return 'Apprise';
|
||||
case 'webhook':
|
||||
return 'Webhook';
|
||||
default:
|
||||
return entry.type || 'Destination';
|
||||
}
|
||||
};
|
||||
|
||||
const alertSummary = (entry: NotificationDeliveryLogEntry): string => {
|
||||
if (entry.alertIds.length === 0) {
|
||||
return entry.alertCount === 1 ? '1 alert' : `${entry.alertCount} alerts`;
|
||||
}
|
||||
const [first, ...rest] = entry.alertIds;
|
||||
return rest.length > 0 ? `${first} +${rest.length} more` : first;
|
||||
};
|
||||
|
||||
const entries = () => props.log?.entries ?? [];
|
||||
const windowDays = () => props.log?.windowDays ?? 7;
|
||||
|
||||
return (
|
||||
<Card padding="sm" class="sm:p-4">
|
||||
<div class="flex flex-col gap-3">
|
||||
<div class="flex flex-col gap-3 sm:flex-row sm:items-start sm:justify-between">
|
||||
<div class="min-w-0">
|
||||
<h3 class="text-sm font-semibold text-gray-900 dark:text-gray-100">
|
||||
{getAlertDestinationsDeliveryLogTitle()}
|
||||
</h3>
|
||||
<p class="mt-1 text-sm leading-6 text-gray-600 dark:text-gray-400">
|
||||
{getAlertDestinationsDeliveryLogDescription(windowDays())}
|
||||
</p>
|
||||
</div>
|
||||
<button
|
||||
type="button"
|
||||
class="inline-flex flex-shrink-0 items-center justify-center gap-2 rounded-md border border-gray-300 bg-transparent px-3 py-1.5 text-sm font-medium text-gray-700 transition hover:bg-gray-100 disabled:cursor-not-allowed disabled:opacity-50 dark:border-gray-600 dark:text-gray-300 dark:hover:bg-gray-700/40"
|
||||
disabled={props.refreshing}
|
||||
onClick={props.onRefresh}
|
||||
>
|
||||
<RefreshCwIcon class={`h-4 w-4 ${props.refreshing ? 'animate-spin' : ''}`} />
|
||||
{getAlertDestinationsDeliveryRefreshLabel()}
|
||||
</button>
|
||||
</div>
|
||||
|
||||
<Show
|
||||
when={!props.unavailable}
|
||||
fallback={
|
||||
<p class="text-sm text-red-800 dark:text-red-300" role="alert">
|
||||
{getAlertDestinationsDeliveryLogUnavailable()}
|
||||
</p>
|
||||
}
|
||||
>
|
||||
<Show
|
||||
when={entries().length > 0}
|
||||
fallback={
|
||||
<p class="text-sm text-gray-600 dark:text-gray-400">
|
||||
{getAlertDestinationsDeliveryLogEmpty()}
|
||||
</p>
|
||||
}
|
||||
>
|
||||
<ul class="max-h-80 divide-y divide-gray-200 overflow-y-auto dark:divide-gray-700">
|
||||
<For each={entries()}>
|
||||
{(entry) => (
|
||||
<li class="flex flex-col gap-1 py-2">
|
||||
<div class="flex flex-wrap items-center gap-x-3 gap-y-1">
|
||||
<span
|
||||
class={`inline-flex flex-shrink-0 items-center rounded-full px-2 py-0.5 text-xs font-medium ${outcomeBadgeClasses[entry.outcome]}`}
|
||||
>
|
||||
{getAlertDeliveryLogOutcomeLabel(entry.outcome)}
|
||||
</span>
|
||||
<span class="min-w-0 truncate text-sm font-medium text-gray-900 dark:text-gray-100">
|
||||
{destinationLabel(entry)}
|
||||
</span>
|
||||
<span
|
||||
class="min-w-0 truncate text-sm text-gray-600 dark:text-gray-400"
|
||||
title={entry.alertIds.join(', ')}
|
||||
>
|
||||
{alertSummary(entry)}
|
||||
</span>
|
||||
<span class="ml-auto flex-shrink-0 text-xs text-gray-500 dark:text-gray-400">
|
||||
{formatRelativeTime(entry.timestamp)}
|
||||
</span>
|
||||
</div>
|
||||
<Show when={!entry.success && (entry.failureClass || entry.errorMessage)}>
|
||||
<p class="text-xs leading-5 text-red-700 dark:text-red-300">
|
||||
<Show when={entry.failureClass}>
|
||||
{(failureClass) => (
|
||||
<span class="font-medium">
|
||||
{getAlertDeliveryLogFailureClassLabel(failureClass())}
|
||||
{entry.errorMessage ? ': ' : ''}
|
||||
</span>
|
||||
)}
|
||||
</Show>
|
||||
{entry.errorMessage}
|
||||
</p>
|
||||
</Show>
|
||||
</li>
|
||||
)}
|
||||
</For>
|
||||
</ul>
|
||||
</Show>
|
||||
</Show>
|
||||
</div>
|
||||
</Card>
|
||||
);
|
||||
}
|
||||
+65
-1
@@ -13,6 +13,7 @@ vi.mock('@/api/notifications', () => ({
|
||||
NotificationsAPI: {
|
||||
createWebhook: vi.fn(),
|
||||
deleteWebhook: vi.fn(),
|
||||
getDeliveryLog: vi.fn(),
|
||||
getHealth: vi.fn(),
|
||||
getWebhooks: vi.fn(),
|
||||
testNotification: vi.fn(),
|
||||
@@ -25,6 +26,7 @@ vi.mock('@/stores/notifications', () => ({
|
||||
notificationStore: {
|
||||
error: vi.fn(),
|
||||
success: vi.fn(),
|
||||
warning: vi.fn(),
|
||||
},
|
||||
}));
|
||||
|
||||
@@ -72,6 +74,7 @@ describe('useAlertDestinationsTabState', () => {
|
||||
beforeEach(() => {
|
||||
vi.mocked(NotificationsAPI.createWebhook).mockReset();
|
||||
vi.mocked(NotificationsAPI.deleteWebhook).mockReset();
|
||||
vi.mocked(NotificationsAPI.getDeliveryLog).mockReset();
|
||||
vi.mocked(NotificationsAPI.getHealth).mockReset();
|
||||
vi.mocked(NotificationsAPI.getWebhooks).mockReset();
|
||||
vi.mocked(NotificationsAPI.testNotification).mockReset();
|
||||
@@ -79,6 +82,7 @@ describe('useAlertDestinationsTabState', () => {
|
||||
vi.mocked(NotificationsAPI.updateWebhook).mockReset();
|
||||
vi.mocked(notificationStore.error).mockReset();
|
||||
vi.mocked(notificationStore.success).mockReset();
|
||||
vi.mocked(notificationStore.warning).mockReset();
|
||||
vi.mocked(showErrorWithDetail).mockReset();
|
||||
});
|
||||
|
||||
@@ -130,7 +134,22 @@ describe('useAlertDestinationsTabState', () => {
|
||||
failureClassWindowDays: 7,
|
||||
},
|
||||
});
|
||||
vi.mocked(NotificationsAPI.testNotification).mockResolvedValue({ success: true } as never);
|
||||
vi.mocked(NotificationsAPI.getDeliveryLog).mockResolvedValue({
|
||||
entries: [
|
||||
{
|
||||
notificationId: 'email-1',
|
||||
type: 'email',
|
||||
outcome: 'sent',
|
||||
alertIds: ['disk-critical-1'],
|
||||
alertCount: 1,
|
||||
attempts: 1,
|
||||
success: true,
|
||||
timestamp: '2026-08-20T12:00:00Z',
|
||||
},
|
||||
],
|
||||
windowDays: 7,
|
||||
});
|
||||
vi.mocked(NotificationsAPI.testNotification).mockResolvedValue({ status: 'success' } as never);
|
||||
vi.mocked(NotificationsAPI.testWebhook).mockResolvedValue({ success: true } as never);
|
||||
|
||||
const { result } = renderHook(() =>
|
||||
@@ -147,7 +166,10 @@ describe('useAlertDestinationsTabState', () => {
|
||||
|
||||
await waitFor(() => expect(NotificationsAPI.getWebhooks).toHaveBeenCalledTimes(1));
|
||||
await waitFor(() => expect(NotificationsAPI.getHealth).toHaveBeenCalledTimes(1));
|
||||
await waitFor(() => expect(NotificationsAPI.getDeliveryLog).toHaveBeenCalledTimes(1));
|
||||
expect(result.deliveryHealth()?.queue.status).toBe('healthy');
|
||||
expect(result.deliveryLog()?.entries).toHaveLength(1);
|
||||
expect(result.deliveryLogUnavailable()).toBe(false);
|
||||
expect(result.webhooks()).toEqual([
|
||||
expect.objectContaining({ id: 'hook-1', service: 'generic' }),
|
||||
]);
|
||||
@@ -181,7 +203,49 @@ describe('useAlertDestinationsTabState', () => {
|
||||
expect(onRetryLoad).toHaveBeenCalledTimes(1);
|
||||
await waitFor(() => expect(NotificationsAPI.getWebhooks).toHaveBeenCalledTimes(2));
|
||||
await waitFor(() => expect(NotificationsAPI.getHealth).toHaveBeenCalledTimes(2));
|
||||
await waitFor(() => expect(NotificationsAPI.getDeliveryLog).toHaveBeenCalledTimes(2));
|
||||
expect(notificationStore.success).toHaveBeenCalledTimes(2);
|
||||
expect(notificationStore.warning).not.toHaveBeenCalled();
|
||||
expect(showErrorWithDetail).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('warns instead of celebrating when a test send reports delivery is paused', async () => {
|
||||
const [emailConfig] = createSignal(buildEmailConfig());
|
||||
const [appriseConfig, setAppriseConfig] = createSignal(buildAppriseConfig());
|
||||
const [configLoadError] = createSignal<string | null>(null);
|
||||
const [isRetrying] = createSignal(false);
|
||||
const [isLoadingDestinations] = createSignal(false);
|
||||
|
||||
vi.mocked(NotificationsAPI.getWebhooks).mockResolvedValue([]);
|
||||
vi.mocked(NotificationsAPI.getHealth).mockRejectedValue(new Error('offline'));
|
||||
vi.mocked(NotificationsAPI.getDeliveryLog).mockResolvedValue({ entries: [], windowDays: 7 });
|
||||
// The backend reports the test went out while the activation gate keeps
|
||||
// real alerts suppressed; plain success here is the postmortem trap.
|
||||
vi.mocked(NotificationsAPI.testNotification).mockResolvedValue({
|
||||
status: 'success',
|
||||
deliveryPaused: true,
|
||||
} as never);
|
||||
|
||||
const { result } = renderHook(() =>
|
||||
useAlertDestinationsTabState({
|
||||
appriseConfig,
|
||||
configLoadError,
|
||||
emailConfig,
|
||||
isLoadingDestinations,
|
||||
isRetrying,
|
||||
onRetryLoad: vi.fn(),
|
||||
setAppriseConfig,
|
||||
}),
|
||||
);
|
||||
|
||||
await result.testEmailConfig();
|
||||
expect(notificationStore.warning).toHaveBeenCalledWith(
|
||||
expect.stringContaining('delivery is paused'),
|
||||
);
|
||||
expect(notificationStore.success).not.toHaveBeenCalled();
|
||||
|
||||
await result.testApprise();
|
||||
expect(notificationStore.warning).toHaveBeenCalledTimes(2);
|
||||
expect(notificationStore.success).not.toHaveBeenCalled();
|
||||
});
|
||||
});
|
||||
|
||||
@@ -0,0 +1,69 @@
|
||||
import { createRoot } from 'solid-js';
|
||||
import { beforeEach, describe, expect, it, vi } from 'vitest';
|
||||
|
||||
import { NotificationsAPI } from '@/api/notifications';
|
||||
|
||||
import { useNotificationDeliveryLog } from '../useNotificationDeliveryLog';
|
||||
|
||||
vi.mock('@/api/notifications', () => ({
|
||||
NotificationsAPI: { getDeliveryLog: vi.fn() },
|
||||
}));
|
||||
|
||||
describe('useNotificationDeliveryLog', () => {
|
||||
beforeEach(() => {
|
||||
vi.mocked(NotificationsAPI.getDeliveryLog).mockReset();
|
||||
});
|
||||
|
||||
it('exposes the loaded log and clears the unavailable flag', () =>
|
||||
createRoot(async (dispose) => {
|
||||
vi.mocked(NotificationsAPI.getDeliveryLog).mockResolvedValue({
|
||||
entries: [
|
||||
{
|
||||
notificationId: 'email-1',
|
||||
type: 'email',
|
||||
outcome: 'sent',
|
||||
alertIds: ['disk-critical-1'],
|
||||
alertCount: 1,
|
||||
attempts: 1,
|
||||
success: true,
|
||||
timestamp: '2026-08-20T12:00:00Z',
|
||||
},
|
||||
],
|
||||
windowDays: 7,
|
||||
});
|
||||
const state = useNotificationDeliveryLog();
|
||||
|
||||
await state.loadDeliveryLog();
|
||||
expect(state.deliveryLog()?.entries).toHaveLength(1);
|
||||
expect(state.deliveryLogUnavailable()).toBe(false);
|
||||
dispose();
|
||||
}));
|
||||
|
||||
it('reports an unreadable log as unavailable, never as empty', () =>
|
||||
createRoot(async (dispose) => {
|
||||
vi.mocked(NotificationsAPI.getDeliveryLog).mockRejectedValue(new Error('network down'));
|
||||
const state = useNotificationDeliveryLog();
|
||||
|
||||
await state.loadDeliveryLog();
|
||||
expect(state.deliveryLogUnavailable()).toBe(true);
|
||||
expect(state.deliveryLog()).toBeNull();
|
||||
dispose();
|
||||
}));
|
||||
|
||||
it('recovers the unavailable flag once a later load succeeds', () =>
|
||||
createRoot(async (dispose) => {
|
||||
vi.mocked(NotificationsAPI.getDeliveryLog).mockRejectedValueOnce(new Error('network down'));
|
||||
vi.mocked(NotificationsAPI.getDeliveryLog).mockResolvedValue({
|
||||
entries: [],
|
||||
windowDays: 7,
|
||||
});
|
||||
const state = useNotificationDeliveryLog();
|
||||
|
||||
await state.loadDeliveryLog();
|
||||
expect(state.deliveryLogUnavailable()).toBe(true);
|
||||
await state.loadDeliveryLog();
|
||||
expect(state.deliveryLogUnavailable()).toBe(false);
|
||||
expect(state.deliveryLog()?.entries).toHaveLength(0);
|
||||
dispose();
|
||||
}));
|
||||
});
|
||||
@@ -7,6 +7,7 @@ import { logger } from '@/utils/logger';
|
||||
import type { AlertDestinationsDeliveryPausedReason } from '@/utils/alertDestinationsPresentation';
|
||||
import { AlertAppriseDestinationsSection } from '../AlertAppriseDestinationsSection';
|
||||
import { AlertDeliveryHealthCard } from '../AlertDeliveryHealthCard';
|
||||
import { AlertDeliveryLogCard } from '../AlertDeliveryLogCard';
|
||||
import { AlertDeliveryPausedCard } from '../AlertDeliveryPausedCard';
|
||||
import { AlertDestinationsLoadErrorCard } from '../AlertDestinationsLoadErrorCard';
|
||||
import { AlertDestinationsLoadingState } from '../AlertDestinationsLoadingState';
|
||||
@@ -120,6 +121,14 @@ export function DestinationsTab(props: DestinationsTabProps) {
|
||||
showUpgradePrompts={!presentationPolicyHidesUpgradePrompts()}
|
||||
upgradeDestination={getUpgradeActionDestination('relay')}
|
||||
/>
|
||||
|
||||
<AlertDeliveryLogCard
|
||||
log={state.deliveryLog()}
|
||||
unavailable={state.deliveryLogUnavailable()}
|
||||
refreshing={state.refreshingDeliveryLog()}
|
||||
onRefresh={() => void state.loadDeliveryLog()}
|
||||
webhooks={state.webhooks()}
|
||||
/>
|
||||
</Show>
|
||||
</div>
|
||||
);
|
||||
|
||||
@@ -10,11 +10,13 @@ import {
|
||||
getAlertDestinationsAppriseValidationError,
|
||||
getAlertDestinationsEmailTestFailure,
|
||||
getAlertDestinationsEmailTestSuccess,
|
||||
getAlertDestinationsTestPausedWarning,
|
||||
} from '@/utils/alertDestinationsPresentation';
|
||||
|
||||
import { parseAppriseTargets } from './helpers';
|
||||
import type { UIAppriseConfig, UIEmailConfig } from './types';
|
||||
import { useNotificationDeliveryHealth } from './useNotificationDeliveryHealth';
|
||||
import { useNotificationDeliveryLog } from './useNotificationDeliveryLog';
|
||||
import { useAlertWebhookDestinationsState } from './useAlertWebhookDestinationsState';
|
||||
|
||||
export interface AlertDestinationsTabStateProps {
|
||||
@@ -37,6 +39,8 @@ export function useAlertDestinationsTabState(props: AlertDestinationsTabStatePro
|
||||
deliveryNeedsAttention,
|
||||
loadDeliveryHealth,
|
||||
} = useNotificationDeliveryHealth();
|
||||
const { deliveryLog, deliveryLogUnavailable, refreshingDeliveryLog, loadDeliveryLog } =
|
||||
useNotificationDeliveryLog();
|
||||
const webhookState = useAlertWebhookDestinationsState();
|
||||
|
||||
const isLoading = createMemo(
|
||||
@@ -70,11 +74,15 @@ export function useAlertDestinationsTabState(props: AlertDestinationsTabStatePro
|
||||
const testEmailConfig = async () => {
|
||||
setTestingEmail(true);
|
||||
try {
|
||||
await NotificationsAPI.testNotification({
|
||||
const result = await NotificationsAPI.testNotification({
|
||||
type: 'email',
|
||||
config: { ...props.emailConfig() } as Record<string, unknown>,
|
||||
});
|
||||
notificationStore.success(getAlertDestinationsEmailTestSuccess());
|
||||
if (result.deliveryPaused) {
|
||||
notificationStore.warning(getAlertDestinationsTestPausedWarning());
|
||||
} else {
|
||||
notificationStore.success(getAlertDestinationsEmailTestSuccess());
|
||||
}
|
||||
} catch (error) {
|
||||
logger.error(getAlertDestinationsEmailTestFailure(), error);
|
||||
const message =
|
||||
@@ -103,11 +111,15 @@ export function useAlertDestinationsTabState(props: AlertDestinationsTabStatePro
|
||||
throw new Error(getAlertDestinationsAppriseValidationError('missingServerUrl'));
|
||||
}
|
||||
|
||||
await NotificationsAPI.testNotification({
|
||||
const result = await NotificationsAPI.testNotification({
|
||||
type: 'apprise',
|
||||
config,
|
||||
});
|
||||
notificationStore.success(getAlertDestinationsAppriseTestSuccess());
|
||||
if (result.deliveryPaused) {
|
||||
notificationStore.warning(getAlertDestinationsTestPausedWarning());
|
||||
} else {
|
||||
notificationStore.success(getAlertDestinationsAppriseTestSuccess());
|
||||
}
|
||||
} catch (error) {
|
||||
logger.error(getAlertDestinationsAppriseTestFailure(), error);
|
||||
const message =
|
||||
@@ -123,22 +135,28 @@ export function useAlertDestinationsTabState(props: AlertDestinationsTabStatePro
|
||||
props.onRetryLoad();
|
||||
void webhookState.loadWebhooks();
|
||||
void loadDeliveryHealth();
|
||||
void loadDeliveryLog();
|
||||
};
|
||||
|
||||
onMount(() => {
|
||||
void loadDeliveryHealth();
|
||||
void loadDeliveryLog();
|
||||
});
|
||||
|
||||
return {
|
||||
appriseState,
|
||||
deliveryHealth,
|
||||
deliveryHealthUnavailable,
|
||||
deliveryLog,
|
||||
deliveryLogUnavailable,
|
||||
deliveryNeedsAttention,
|
||||
handleRetry,
|
||||
hasLoadError,
|
||||
isLoading,
|
||||
loadDeliveryHealth,
|
||||
loadDeliveryLog,
|
||||
refreshingDeliveryHealth,
|
||||
refreshingDeliveryLog,
|
||||
testApprise,
|
||||
testEmailConfig,
|
||||
testingApprise,
|
||||
|
||||
@@ -4,7 +4,10 @@ import { NotificationsAPI, type Webhook } from '@/api/notifications';
|
||||
import { notificationStore } from '@/stores/notifications';
|
||||
import { logger } from '@/utils/logger';
|
||||
import { showErrorWithDetail } from '@/utils/toast';
|
||||
import { getAlertDestinationsWebhookLoadError } from '@/utils/alertDestinationsPresentation';
|
||||
import {
|
||||
getAlertDestinationsTestPausedWarning,
|
||||
getAlertDestinationsWebhookLoadError,
|
||||
} from '@/utils/alertDestinationsPresentation';
|
||||
import {
|
||||
getAlertWebhookMutationFailure,
|
||||
getAlertWebhookMutationSuccess,
|
||||
@@ -44,12 +47,14 @@ export function useAlertWebhookDestinationsState() {
|
||||
const testWebhook = async (webhookId: string, webhookData?: Omit<Webhook, 'id'>) => {
|
||||
setTestingWebhook(webhookId);
|
||||
try {
|
||||
if (webhookData) {
|
||||
await NotificationsAPI.testWebhook(webhookData);
|
||||
const result = webhookData
|
||||
? await NotificationsAPI.testWebhook(webhookData)
|
||||
: await NotificationsAPI.testNotification({ type: 'webhook', webhookId });
|
||||
if (result.deliveryPaused) {
|
||||
notificationStore.warning(getAlertDestinationsTestPausedWarning());
|
||||
} else {
|
||||
await NotificationsAPI.testNotification({ type: 'webhook', webhookId });
|
||||
notificationStore.success(getAlertWebhookTestSuccess());
|
||||
}
|
||||
notificationStore.success(getAlertWebhookTestSuccess());
|
||||
} catch (error) {
|
||||
const message = error instanceof Error ? error.message : getAlertWebhookTestFailure();
|
||||
const detail = (error as Error & { detail?: string })?.detail;
|
||||
|
||||
@@ -0,0 +1,37 @@
|
||||
import { createSignal } from 'solid-js';
|
||||
|
||||
import { NotificationsAPI, type NotificationDeliveryLog } from '@/api/notifications';
|
||||
import { logger } from '@/utils/logger';
|
||||
|
||||
// The delivery log is the positive half of delivery evidence: health warns
|
||||
// when something is wrong, the log shows what actually fired and where it
|
||||
// went. A log that cannot be read is reported as unavailable, never as empty,
|
||||
// because "no attempts" and "cannot tell" mean opposite things to someone
|
||||
// deciding whether to trust their alerting.
|
||||
export function useNotificationDeliveryLog() {
|
||||
const [deliveryLog, setDeliveryLog] = createSignal<NotificationDeliveryLog | null>(null);
|
||||
const [deliveryLogUnavailable, setDeliveryLogUnavailable] = createSignal(false);
|
||||
const [refreshingDeliveryLog, setRefreshingDeliveryLog] = createSignal(false);
|
||||
|
||||
const loadDeliveryLog = async () => {
|
||||
setRefreshingDeliveryLog(true);
|
||||
try {
|
||||
const log = await NotificationsAPI.getDeliveryLog();
|
||||
setDeliveryLog(log);
|
||||
setDeliveryLogUnavailable(false);
|
||||
} catch (error) {
|
||||
logger.error('Failed to load notification delivery log', error);
|
||||
setDeliveryLog(null);
|
||||
setDeliveryLogUnavailable(true);
|
||||
} finally {
|
||||
setRefreshingDeliveryLog(false);
|
||||
}
|
||||
};
|
||||
|
||||
return {
|
||||
deliveryLog,
|
||||
deliveryLogUnavailable,
|
||||
refreshingDeliveryLog,
|
||||
loadDeliveryLog,
|
||||
};
|
||||
}
|
||||
@@ -26,6 +26,13 @@ import {
|
||||
ALERT_DESTINATIONS_RETRYING_LABEL,
|
||||
ALERT_DESTINATIONS_RETRY_LABEL,
|
||||
ALERT_DESTINATIONS_WEBHOOK_LOAD_ERROR,
|
||||
getAlertDeliveryLogFailureClassLabel,
|
||||
getAlertDeliveryLogOutcomeLabel,
|
||||
getAlertDestinationsDeliveryLogDescription,
|
||||
getAlertDestinationsDeliveryLogEmpty,
|
||||
getAlertDestinationsDeliveryLogTitle,
|
||||
getAlertDestinationsDeliveryLogUnavailable,
|
||||
getAlertDestinationsTestPausedWarning,
|
||||
getAlertDestinationsAppriseTargetsHelp,
|
||||
getAlertDestinationsAppriseTestLabel,
|
||||
getAlertDestinationsAppriseTestError,
|
||||
@@ -166,6 +173,42 @@ describe('alertDestinationsPresentation', () => {
|
||||
});
|
||||
});
|
||||
|
||||
describe('alert destinations delivery log copy', () => {
|
||||
it('labels every delivery outcome in plain language, not queue jargon', () => {
|
||||
expect(getAlertDeliveryLogOutcomeLabel('sent')).toBe('Delivered');
|
||||
expect(getAlertDeliveryLogOutcomeLabel('retry')).toBe('Retrying');
|
||||
expect(getAlertDeliveryLogOutcomeLabel('failed')).toBe('Failed');
|
||||
// "Dead letter" is queue vocabulary; a user needs to know retries stopped.
|
||||
expect(getAlertDeliveryLogOutcomeLabel('dead_letter')).toBe('Failed, retries exhausted');
|
||||
expect(getAlertDeliveryLogOutcomeLabel('cancelled')).toBe('Cancelled');
|
||||
});
|
||||
|
||||
it('labels failure classes and falls back to unclassified for unknown values', () => {
|
||||
expect(getAlertDeliveryLogFailureClassLabel('authentication')).toBe('Authentication failure');
|
||||
expect(getAlertDeliveryLogFailureClassLabel('rate_limited')).toBe('Rate limited');
|
||||
expect(getAlertDeliveryLogFailureClassLabel('made-up-class')).toBe('Unclassified failure');
|
||||
});
|
||||
|
||||
it('names the retention window and the test-send caveat so absence is not read as failure', () => {
|
||||
expect(getAlertDestinationsDeliveryLogTitle()).toBe('Recent delivery activity');
|
||||
const description = getAlertDestinationsDeliveryLogDescription(7);
|
||||
expect(description).toContain('last 7 days');
|
||||
expect(description).toContain('Test sends skip the queue');
|
||||
expect(getAlertDestinationsDeliveryLogEmpty()).toContain('No alert deliveries were attempted');
|
||||
// An unreadable log must never present itself as an empty one.
|
||||
expect(getAlertDestinationsDeliveryLogUnavailable()).toContain(
|
||||
'could not read the delivery log',
|
||||
);
|
||||
});
|
||||
|
||||
it('warns that a passing test does not mean live alerts flow while delivery is paused', () => {
|
||||
const warning = getAlertDestinationsTestPausedWarning();
|
||||
expect(warning).toContain('Test sent');
|
||||
expect(warning).toContain('delivery is paused');
|
||||
expect(warning).toContain('real alerts are not being sent');
|
||||
});
|
||||
});
|
||||
|
||||
describe('alert destinations delivery paused copy', () => {
|
||||
it('names the consequence and the test-send caveat for every paused reason', () => {
|
||||
expect(getAlertDestinationsDeliveryPausedTitle()).toBe('Notifications are paused');
|
||||
|
||||
@@ -199,6 +199,74 @@ export function getAlertDestinationsDeliveryRefreshLabel() {
|
||||
return ALERT_DESTINATIONS_DELIVERY_REFRESH_LABEL;
|
||||
}
|
||||
|
||||
// Shown instead of the plain test-success toast when the backend reports the
|
||||
// test went out while real alert delivery is paused. Without this, a
|
||||
// successful test is exactly how installs come to believe delivery works
|
||||
// while every live alert is being suppressed.
|
||||
export const ALERT_DESTINATIONS_TEST_PAUSED_WARNING =
|
||||
'Test sent, but notification delivery is paused: real alerts are not being sent. Turn on delivery at the top of this page.';
|
||||
|
||||
export function getAlertDestinationsTestPausedWarning() {
|
||||
return ALERT_DESTINATIONS_TEST_PAUSED_WARNING;
|
||||
}
|
||||
|
||||
export const ALERT_DESTINATIONS_DELIVERY_LOG_TITLE = 'Recent delivery activity';
|
||||
export const ALERT_DESTINATIONS_DELIVERY_LOG_EMPTY =
|
||||
'No alert deliveries were attempted in this window.';
|
||||
export const ALERT_DESTINATIONS_DELIVERY_LOG_UNAVAILABLE =
|
||||
'Pulse could not read the delivery log, so recent delivery activity cannot be shown.';
|
||||
|
||||
export function getAlertDestinationsDeliveryLogTitle() {
|
||||
return ALERT_DESTINATIONS_DELIVERY_LOG_TITLE;
|
||||
}
|
||||
|
||||
// The test-send caveat matters: test messages skip the queue entirely, so a
|
||||
// user who sends a test and then checks this log would otherwise read its
|
||||
// absence as a delivery failure.
|
||||
export function getAlertDestinationsDeliveryLogDescription(windowDays: number) {
|
||||
return `Delivery attempts for real alerts over the last ${windowDays} days. Test sends skip the queue and are not listed here.`;
|
||||
}
|
||||
|
||||
export function getAlertDestinationsDeliveryLogEmpty() {
|
||||
return ALERT_DESTINATIONS_DELIVERY_LOG_EMPTY;
|
||||
}
|
||||
|
||||
export function getAlertDestinationsDeliveryLogUnavailable() {
|
||||
return ALERT_DESTINATIONS_DELIVERY_LOG_UNAVAILABLE;
|
||||
}
|
||||
|
||||
export type AlertDeliveryLogOutcome = 'sent' | 'retry' | 'failed' | 'dead_letter' | 'cancelled';
|
||||
|
||||
// One plain-language label per outcome. "Dead letter" is queue jargon; what a
|
||||
// user needs to know is that Pulse stopped retrying.
|
||||
export function getAlertDeliveryLogOutcomeLabel(outcome: AlertDeliveryLogOutcome) {
|
||||
switch (outcome) {
|
||||
case 'sent':
|
||||
return 'Delivered';
|
||||
case 'retry':
|
||||
return 'Retrying';
|
||||
case 'dead_letter':
|
||||
return 'Failed, retries exhausted';
|
||||
case 'cancelled':
|
||||
return 'Cancelled';
|
||||
default:
|
||||
return 'Failed';
|
||||
}
|
||||
}
|
||||
|
||||
export function getAlertDeliveryLogFailureClassLabel(failureClass: string) {
|
||||
const labels: Record<string, string> = {
|
||||
authentication: 'Authentication failure',
|
||||
rate_limited: 'Rate limited',
|
||||
connectivity: 'Connectivity failure',
|
||||
tls: 'TLS failure',
|
||||
configuration: 'Configuration problem',
|
||||
rejected: 'Rejected by destination',
|
||||
unknown: 'Unclassified failure',
|
||||
};
|
||||
return labels[failureClass] ?? labels.unknown;
|
||||
}
|
||||
|
||||
const ALERT_DESTINATIONS_DELIVERY_PAUSED_TITLE = 'Notifications are paused';
|
||||
const ALERT_DESTINATIONS_DELIVERY_PAUSED_ACTION = 'Turn on delivery';
|
||||
|
||||
|
||||
@@ -7,6 +7,7 @@ import (
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
@@ -45,6 +46,8 @@ type NotificationManager interface {
|
||||
TestEnhancedWebhook(notifications.EnhancedWebhookConfig) (int, string, error)
|
||||
GetQueueStats() (map[string]int, error)
|
||||
GetTelemetryStats(time.Time) (notifications.TelemetryStats, error)
|
||||
GetDeliveryLog(time.Time, int) ([]notifications.DeliveryLogEntry, error)
|
||||
IsEnabled() bool
|
||||
}
|
||||
|
||||
// NotificationConfigPersistence defines the interface for saving notification configuration.
|
||||
@@ -710,8 +713,19 @@ func (h *NotificationHandlers) TestNotification(w http.ResponseWriter, r *http.R
|
||||
}
|
||||
}
|
||||
|
||||
// A test send bypasses the activation gate real alerts honor, so a bare
|
||||
// success here is exactly how installs end up believing delivery works
|
||||
// while every real alert is suppressed. Say so in the result itself.
|
||||
response := map[string]interface{}{
|
||||
"status": "success",
|
||||
"message": "Test notification sent",
|
||||
}
|
||||
if !h.getMonitor(r.Context()).GetNotificationManager().IsEnabled() {
|
||||
response["deliveryPaused"] = true
|
||||
response["message"] = "Test notification sent, but alert delivery is paused: real alerts are not being sent"
|
||||
}
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
json.NewEncoder(w).Encode(map[string]string{"status": "success", "message": "Test notification sent"})
|
||||
json.NewEncoder(w).Encode(response)
|
||||
}
|
||||
|
||||
// GetWebhookTemplates returns available webhook templates
|
||||
@@ -812,6 +826,11 @@ func (h *NotificationHandlers) TestWebhook(w http.ResponseWriter, r *http.Reques
|
||||
w.WriteHeader(http.StatusBadRequest)
|
||||
} else {
|
||||
result["success"] = true
|
||||
// Same honesty as TestNotification: a webhook test bypasses the
|
||||
// activation gate, so its success must not imply live alerts flow.
|
||||
if !h.getMonitor(r.Context()).GetNotificationManager().IsEnabled() {
|
||||
result["deliveryPaused"] = true
|
||||
}
|
||||
}
|
||||
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
@@ -903,6 +922,50 @@ func (h *NotificationHandlers) GetNotificationHealth(w http.ResponseWriter, r *h
|
||||
json.NewEncoder(w).Encode(health)
|
||||
}
|
||||
|
||||
// GetDeliveryLog returns recent recorded delivery attempts, newest first.
|
||||
// This is the per-attempt evidence behind the aggregate health verdict: what
|
||||
// fired, which destination it went to, and what happened. Entries share the
|
||||
// queue's retention windows, so the payload names the window rather than
|
||||
// presenting itself as lifetime history.
|
||||
func (h *NotificationHandlers) GetDeliveryLog(w http.ResponseWriter, r *http.Request) {
|
||||
manager := h.getMonitor(r.Context()).GetNotificationManager()
|
||||
|
||||
limit := 0
|
||||
if raw := strings.TrimSpace(r.URL.Query().Get("limit")); raw != "" {
|
||||
parsed, err := strconv.Atoi(raw)
|
||||
if err != nil || parsed <= 0 {
|
||||
http.Error(w, "limit must be a positive integer", http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
limit = parsed
|
||||
}
|
||||
|
||||
entries, err := manager.GetDeliveryLog(
|
||||
time.Now().Add(-completedQueueRetentionDays*24*time.Hour), limit,
|
||||
)
|
||||
if err != nil {
|
||||
log.Warn().Err(err).Msg("Failed to read notification delivery log")
|
||||
http.Error(w, "Delivery log unavailable", http.StatusServiceUnavailable)
|
||||
return
|
||||
}
|
||||
|
||||
// Webhook failure text can embed the destination URL, and destination URLs
|
||||
// can embed credentials.
|
||||
for i := range entries {
|
||||
if entries[i].ErrorMessage != "" {
|
||||
entries[i].ErrorMessage = notifications.RedactWebhookURLSecrets(entries[i].ErrorMessage)
|
||||
}
|
||||
}
|
||||
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
json.NewEncoder(w).Encode(map[string]interface{}{
|
||||
"entries": entries,
|
||||
"window_days": completedQueueRetentionDays,
|
||||
"dead_letter_retention_days": deadLetterQueueRetentionDays,
|
||||
"entries_are_retention_bounded": true,
|
||||
})
|
||||
}
|
||||
|
||||
// classifyNotificationQueueHealth delegates to the canonical rule in
|
||||
// internal/notifications so this endpoint and the monitoring loop that raises
|
||||
// the notification-delivery system alert cannot drift apart.
|
||||
@@ -1010,6 +1073,11 @@ func (h *NotificationHandlers) HandleNotifications(w http.ResponseWriter, r *htt
|
||||
return
|
||||
}
|
||||
h.GetNotificationHealth(w, r)
|
||||
case path == "/delivery-log" && r.Method == http.MethodGet:
|
||||
if !requireAnyScope(config.ScopeSettingsRead, config.ScopeSettingsRead, config.ScopeSettingsWrite) {
|
||||
return
|
||||
}
|
||||
h.GetDeliveryLog(w, r)
|
||||
default:
|
||||
http.Error(w, "Not found", http.StatusNotFound)
|
||||
}
|
||||
|
||||
@@ -139,6 +139,182 @@ func TestGetNotificationHealthFailsClosedWhenQueueStatsAreUnavailable(t *testing
|
||||
}
|
||||
}
|
||||
|
||||
func TestGetDeliveryLogReturnsEntriesWithRedactedErrors(t *testing.T) {
|
||||
mockMonitor := new(MockNotificationMonitor)
|
||||
mockManager := new(MockNotificationManager)
|
||||
mockMonitor.On("GetNotificationManager").Return(mockManager)
|
||||
mockManager.On("GetDeliveryLog", mock.Anything, 25).Return([]notifications.DeliveryLogEntry{
|
||||
{
|
||||
NotificationID: "webhook-1",
|
||||
Type: "webhook",
|
||||
DestinationID: "wh-ops",
|
||||
Outcome: notifications.DeliveryOutcomeFailed,
|
||||
AlertIDs: []string{"vm-offline-101"},
|
||||
AlertCount: 1,
|
||||
Attempts: 3,
|
||||
ErrorMessage: "post https://hooks.example.test/notify?token=supersecret returned 401",
|
||||
FailureClass: "authentication",
|
||||
},
|
||||
{
|
||||
NotificationID: "email-1",
|
||||
Type: "email",
|
||||
Outcome: notifications.DeliveryOutcomeSent,
|
||||
AlertIDs: []string{"disk-critical-1"},
|
||||
AlertCount: 1,
|
||||
Attempts: 1,
|
||||
Success: true,
|
||||
},
|
||||
}, nil).Once()
|
||||
|
||||
rec := httptest.NewRecorder()
|
||||
NewNotificationHandlers(nil, mockMonitor).GetDeliveryLog(
|
||||
rec,
|
||||
httptest.NewRequest(http.MethodGet, "/api/notifications/delivery-log?limit=25", nil),
|
||||
)
|
||||
|
||||
if rec.Code != http.StatusOK {
|
||||
t.Fatalf("status = %d, want %d", rec.Code, http.StatusOK)
|
||||
}
|
||||
var response struct {
|
||||
Entries []struct {
|
||||
NotificationID string `json:"notificationId"`
|
||||
Outcome string `json:"outcome"`
|
||||
DestinationID string `json:"destinationId"`
|
||||
AlertIDs []string `json:"alertIds"`
|
||||
ErrorMessage string `json:"errorMessage"`
|
||||
FailureClass string `json:"failureClass"`
|
||||
} `json:"entries"`
|
||||
WindowDays int `json:"window_days"`
|
||||
DeadLetterRetentionDays int `json:"dead_letter_retention_days"`
|
||||
EntriesAreRetentionBounded bool `json:"entries_are_retention_bounded"`
|
||||
}
|
||||
if err := json.Unmarshal(rec.Body.Bytes(), &response); err != nil {
|
||||
t.Fatalf("decode delivery log response: %v", err)
|
||||
}
|
||||
if len(response.Entries) != 2 {
|
||||
t.Fatalf("entries = %#v, want 2", response.Entries)
|
||||
}
|
||||
if response.Entries[0].Outcome != "failed" ||
|
||||
response.Entries[0].DestinationID != "wh-ops" ||
|
||||
response.Entries[0].FailureClass != "authentication" ||
|
||||
len(response.Entries[0].AlertIDs) != 1 ||
|
||||
response.Entries[0].AlertIDs[0] != "vm-offline-101" {
|
||||
t.Fatalf("failed entry = %#v", response.Entries[0])
|
||||
}
|
||||
if strings.Contains(response.Entries[0].ErrorMessage, "supersecret") ||
|
||||
!strings.Contains(response.Entries[0].ErrorMessage, "token=REDACTED") {
|
||||
t.Fatalf("error message not redacted: %q", response.Entries[0].ErrorMessage)
|
||||
}
|
||||
if response.WindowDays != 7 ||
|
||||
response.DeadLetterRetentionDays != 30 ||
|
||||
!response.EntriesAreRetentionBounded {
|
||||
t.Fatalf("retention context = %#v", response)
|
||||
}
|
||||
}
|
||||
|
||||
func TestGetDeliveryLogRejectsInvalidLimit(t *testing.T) {
|
||||
mockMonitor := new(MockNotificationMonitor)
|
||||
mockManager := new(MockNotificationManager)
|
||||
mockMonitor.On("GetNotificationManager").Return(mockManager)
|
||||
|
||||
for _, limit := range []string{"abc", "-1", "0"} {
|
||||
rec := httptest.NewRecorder()
|
||||
NewNotificationHandlers(nil, mockMonitor).GetDeliveryLog(
|
||||
rec,
|
||||
httptest.NewRequest(http.MethodGet, "/api/notifications/delivery-log?limit="+limit, nil),
|
||||
)
|
||||
if rec.Code != http.StatusBadRequest {
|
||||
t.Fatalf("limit=%q status = %d, want %d", limit, rec.Code, http.StatusBadRequest)
|
||||
}
|
||||
}
|
||||
mockManager.AssertNotCalled(t, "GetDeliveryLog", mock.Anything, mock.Anything)
|
||||
}
|
||||
|
||||
func TestGetDeliveryLogFailsClosedWhenQueueUnavailable(t *testing.T) {
|
||||
mockMonitor := new(MockNotificationMonitor)
|
||||
mockManager := new(MockNotificationManager)
|
||||
mockMonitor.On("GetNotificationManager").Return(mockManager)
|
||||
mockManager.On("GetDeliveryLog", mock.Anything, 0).Return(
|
||||
nil, errors.New("database path /secret unavailable"),
|
||||
).Once()
|
||||
|
||||
rec := httptest.NewRecorder()
|
||||
NewNotificationHandlers(nil, mockMonitor).GetDeliveryLog(
|
||||
rec,
|
||||
httptest.NewRequest(http.MethodGet, "/api/notifications/delivery-log", nil),
|
||||
)
|
||||
|
||||
if rec.Code != http.StatusServiceUnavailable {
|
||||
t.Fatalf("status = %d, want %d", rec.Code, http.StatusServiceUnavailable)
|
||||
}
|
||||
if body := rec.Body.String(); containsAny(body, "database path", "/secret") {
|
||||
t.Fatalf("delivery log response exposed internal queue error: %s", body)
|
||||
}
|
||||
}
|
||||
|
||||
// A test send bypasses the activation gate real alerts honor, so its result
|
||||
// must say when real delivery is paused instead of reporting bare success.
|
||||
func TestTestNotificationReportsPausedDeliveryInResult(t *testing.T) {
|
||||
mockMonitor := new(MockNotificationMonitor)
|
||||
mockManager := new(MockNotificationManager)
|
||||
mockMonitor.On("GetNotificationManager").Return(mockManager)
|
||||
mockManager.On("SendTestNotification", "email").Return(nil).Once()
|
||||
mockManager.On("IsEnabled").Return(false).Once()
|
||||
|
||||
rec := httptest.NewRecorder()
|
||||
NewNotificationHandlers(nil, mockMonitor).TestNotification(
|
||||
rec,
|
||||
httptest.NewRequest(
|
||||
http.MethodPost,
|
||||
"/api/notifications/test",
|
||||
strings.NewReader(`{"method":"email"}`),
|
||||
),
|
||||
)
|
||||
|
||||
if rec.Code != http.StatusOK {
|
||||
t.Fatalf("status = %d, want %d", rec.Code, http.StatusOK)
|
||||
}
|
||||
var response struct {
|
||||
Status string `json:"status"`
|
||||
Message string `json:"message"`
|
||||
DeliveryPaused bool `json:"deliveryPaused"`
|
||||
}
|
||||
if err := json.Unmarshal(rec.Body.Bytes(), &response); err != nil {
|
||||
t.Fatalf("decode test notification response: %v", err)
|
||||
}
|
||||
if response.Status != "success" || !response.DeliveryPaused {
|
||||
t.Fatalf("response = %#v, want success with deliveryPaused", response)
|
||||
}
|
||||
if !strings.Contains(response.Message, "paused") {
|
||||
t.Fatalf("message %q does not warn about paused delivery", response.Message)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTestNotificationOmitsPausedFlagWhenDeliveryActive(t *testing.T) {
|
||||
mockMonitor := new(MockNotificationMonitor)
|
||||
mockManager := new(MockNotificationManager)
|
||||
mockMonitor.On("GetNotificationManager").Return(mockManager)
|
||||
mockManager.On("SendTestNotification", "email").Return(nil).Once()
|
||||
mockManager.On("IsEnabled").Return(true).Once()
|
||||
|
||||
rec := httptest.NewRecorder()
|
||||
NewNotificationHandlers(nil, mockMonitor).TestNotification(
|
||||
rec,
|
||||
httptest.NewRequest(
|
||||
http.MethodPost,
|
||||
"/api/notifications/test",
|
||||
strings.NewReader(`{"method":"email"}`),
|
||||
),
|
||||
)
|
||||
|
||||
if rec.Code != http.StatusOK {
|
||||
t.Fatalf("status = %d, want %d", rec.Code, http.StatusOK)
|
||||
}
|
||||
if strings.Contains(rec.Body.String(), "deliveryPaused") {
|
||||
t.Fatalf("active delivery response carries paused flag: %s", rec.Body.String())
|
||||
}
|
||||
}
|
||||
|
||||
func containsAny(value string, needles ...string) bool {
|
||||
for _, needle := range needles {
|
||||
if needle != "" && strings.Contains(value, needle) {
|
||||
|
||||
@@ -41,6 +41,7 @@ func TestNotificationReadEndpointsRequireSettingsReadScope(t *testing.T) {
|
||||
"/api/notifications/webhook-history",
|
||||
"/api/notifications/email-providers",
|
||||
"/api/notifications/health",
|
||||
"/api/notifications/delivery-log",
|
||||
}
|
||||
|
||||
for _, path := range endpoints {
|
||||
|
||||
@@ -258,6 +258,19 @@ func (m *MockNotificationManager) GetTelemetryStats(since time.Time) (notificati
|
||||
return args.Get(0).(notifications.TelemetryStats), args.Error(1)
|
||||
}
|
||||
|
||||
func (m *MockNotificationManager) GetDeliveryLog(since time.Time, limit int) ([]notifications.DeliveryLogEntry, error) {
|
||||
args := m.Called(since, limit)
|
||||
if args.Get(0) == nil {
|
||||
return nil, args.Error(1)
|
||||
}
|
||||
return args.Get(0).([]notifications.DeliveryLogEntry), args.Error(1)
|
||||
}
|
||||
|
||||
func (m *MockNotificationManager) IsEnabled() bool {
|
||||
args := m.Called()
|
||||
return args.Bool(0)
|
||||
}
|
||||
|
||||
type MockNotificationConfigPersistence struct {
|
||||
mock.Mock
|
||||
}
|
||||
@@ -654,6 +667,7 @@ func TestNotificationHandlers(t *testing.T) {
|
||||
}},
|
||||
{"POST", "/api/notifications/webhooks/test", func() {
|
||||
mockManager.On("TestEnhancedWebhook", mock.Anything).Return(200, "OK", nil).Once()
|
||||
mockManager.On("IsEnabled").Return(true).Once()
|
||||
}},
|
||||
{"PUT", "/api/notifications/webhooks/wh1", func() {
|
||||
mockManager.On("GetWebhooks").Return([]notifications.WebhookConfig{{ID: "wh1"}}).Once()
|
||||
@@ -705,6 +719,7 @@ func TestNotificationHandlers(t *testing.T) {
|
||||
|
||||
t.Run("TestNotification", func(t *testing.T) {
|
||||
mockManager.On("SendTestNotification", "email").Return(nil).Once()
|
||||
mockManager.On("IsEnabled").Return(true).Once()
|
||||
body, _ := json.Marshal(map[string]string{"method": "email"})
|
||||
req := httptest.NewRequest("POST", "/api/notifications/test", bytes.NewReader(body))
|
||||
w := httptest.NewRecorder()
|
||||
@@ -715,6 +730,7 @@ func TestNotificationHandlers(t *testing.T) {
|
||||
t.Run("TestNotification_Webhook", func(t *testing.T) {
|
||||
mockManager.On("GetWebhooks").Return([]notifications.WebhookConfig{{ID: "wh1"}}).Once()
|
||||
mockManager.On("SendTestWebhook", mock.Anything).Return(nil).Once()
|
||||
mockManager.On("IsEnabled").Return(true).Once()
|
||||
body, _ := json.Marshal(map[string]string{"method": "webhook", "webhookId": "wh1"})
|
||||
req := httptest.NewRequest("POST", "/api/notifications/test", bytes.NewReader(body))
|
||||
w := httptest.NewRecorder()
|
||||
@@ -724,6 +740,7 @@ func TestNotificationHandlers(t *testing.T) {
|
||||
|
||||
t.Run("TestWebhook", func(t *testing.T) {
|
||||
mockManager.On("TestEnhancedWebhook", mock.Anything).Return(200, "OK", nil).Once()
|
||||
mockManager.On("IsEnabled").Return(true).Once()
|
||||
body, _ := json.Marshal(map[string]string{"url": "https://example.com/test", "service": "ntfy"})
|
||||
req := httptest.NewRequest("POST", "/api/notifications/webhooks/test", bytes.NewReader(body))
|
||||
w := httptest.NewRecorder()
|
||||
@@ -774,6 +791,7 @@ func TestNotificationHandlers(t *testing.T) {
|
||||
strings.Contains(webhook.PayloadTemplate, `"username": "Pulse Monitoring"`) &&
|
||||
webhook.Headers["Content-Type"] == "application/json"
|
||||
})).Return(200, "OK", nil).Once()
|
||||
mockManager.On("IsEnabled").Return(true).Once()
|
||||
body, _ := json.Marshal(map[string]string{"url": "https://example.com/test", "service": "discord"})
|
||||
req := httptest.NewRequest("POST", "/api/notifications/webhooks/test", bytes.NewReader(body))
|
||||
w := httptest.NewRecorder()
|
||||
@@ -796,6 +814,7 @@ func TestNotificationHandlers(t *testing.T) {
|
||||
!hasLegacyApp &&
|
||||
!hasLegacyUser
|
||||
})).Return(200, "OK", nil).Once()
|
||||
mockManager.On("IsEnabled").Return(true).Once()
|
||||
|
||||
body, _ := json.Marshal(map[string]interface{}{
|
||||
"url": "https://api.pushover.net/1/messages.json",
|
||||
@@ -814,6 +833,7 @@ func TestNotificationHandlers(t *testing.T) {
|
||||
|
||||
t.Run("TestNotification_EmailWithConfig", func(t *testing.T) {
|
||||
mockManager.On("SendTestNotificationWithConfig", "email", mock.Anything, mock.Anything).Return(nil).Once()
|
||||
mockManager.On("IsEnabled").Return(true).Once()
|
||||
body, _ := json.Marshal(map[string]interface{}{
|
||||
"method": "email",
|
||||
"config": notifications.EmailConfig{Enabled: true, SMTPHost: "smtp.example.com", Password: "test"},
|
||||
@@ -826,6 +846,7 @@ func TestNotificationHandlers(t *testing.T) {
|
||||
|
||||
t.Run("TestNotification_AppriseWithConfig", func(t *testing.T) {
|
||||
mockManager.On("SendTestAppriseWithConfig", mock.Anything).Return(nil).Once()
|
||||
mockManager.On("IsEnabled").Return(true).Once()
|
||||
body, _ := json.Marshal(map[string]interface{}{
|
||||
"method": "apprise",
|
||||
"config": notifications.AppriseConfig{Enabled: true, APIKey: "test"},
|
||||
|
||||
@@ -0,0 +1,215 @@
|
||||
package notifications
|
||||
|
||||
import (
|
||||
"database/sql"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/operationaltrust"
|
||||
)
|
||||
|
||||
// The delivery log is the per-attempt record behind the delivery health
|
||||
// verdict. Health says whether anything is wrong; the log is the evidence a
|
||||
// user can read to see what fired, where it went, and what happened to it.
|
||||
// It reads the same audit rows RecordAudit writes, so the log and the
|
||||
// aggregate verdicts cannot disagree about what was attempted.
|
||||
|
||||
// Delivery outcomes as surfaced to callers. These name what happened to one
|
||||
// attempt, not the row's live queue status: a pending row with attempts is a
|
||||
// retry in flight, not a success or a terminal failure.
|
||||
const (
|
||||
DeliveryOutcomeSent = "sent"
|
||||
DeliveryOutcomeRetry = "retry"
|
||||
DeliveryOutcomeFailed = "failed"
|
||||
DeliveryOutcomeDeadLetter = "dead_letter"
|
||||
DeliveryOutcomeCancelled = "cancelled"
|
||||
)
|
||||
|
||||
// maxDeliveryLogEntries bounds one read so an install with a large retained
|
||||
// audit trail cannot be asked for an unbounded payload.
|
||||
const maxDeliveryLogEntries = 200
|
||||
|
||||
// defaultDeliveryLogEntries is the page size when the caller does not ask for
|
||||
// a specific limit.
|
||||
const defaultDeliveryLogEntries = 50
|
||||
|
||||
// DeliveryLogEntry is one recorded delivery attempt outcome.
|
||||
type DeliveryLogEntry struct {
|
||||
NotificationID string `json:"notificationId"`
|
||||
Type string `json:"type"`
|
||||
Method string `json:"method,omitempty"`
|
||||
DestinationID string `json:"destinationId,omitempty"`
|
||||
Outcome string `json:"outcome"`
|
||||
AlertIDs []string `json:"alertIds"`
|
||||
AlertCount int `json:"alertCount"`
|
||||
Attempts int `json:"attempts"`
|
||||
Success bool `json:"success"`
|
||||
ErrorMessage string `json:"errorMessage,omitempty"`
|
||||
FailureClass string `json:"failureClass,omitempty"`
|
||||
Timestamp time.Time `json:"timestamp"`
|
||||
}
|
||||
|
||||
// deliveryOutcome is the single rule turning an audit row's status into the
|
||||
// outcome callers act on. RecordAudit uses it for the operational-trust
|
||||
// metric and GetDeliveryLog uses it for the user-facing log, so the two can
|
||||
// never label the same attempt differently.
|
||||
func deliveryOutcome(status NotificationQueueStatus, success bool, attempts int) string {
|
||||
switch {
|
||||
case success && status == QueueStatusSent:
|
||||
return DeliveryOutcomeSent
|
||||
case status == QueueStatusDLQ:
|
||||
return DeliveryOutcomeDeadLetter
|
||||
case status == QueueStatusCancelled:
|
||||
return DeliveryOutcomeCancelled
|
||||
case status == QueueStatusPending && attempts > 0:
|
||||
return DeliveryOutcomeRetry
|
||||
}
|
||||
return DeliveryOutcomeFailed
|
||||
}
|
||||
|
||||
// GetDeliveryLog returns recorded delivery attempts on or after since, newest
|
||||
// first. Rows share the queue's retention windows, so a window longer than
|
||||
// the completed-row retention must not be presented as complete history.
|
||||
func (nq *NotificationQueue) GetDeliveryLog(since time.Time, limit int) ([]DeliveryLogEntry, error) {
|
||||
if nq == nil {
|
||||
return nil, fmt.Errorf("notification queue not initialized")
|
||||
}
|
||||
if since.IsZero() {
|
||||
since = time.Now().Add(-7 * 24 * time.Hour)
|
||||
}
|
||||
if limit <= 0 {
|
||||
limit = defaultDeliveryLogEntries
|
||||
}
|
||||
if limit > maxDeliveryLogEntries {
|
||||
limit = maxDeliveryLogEntries
|
||||
}
|
||||
|
||||
nq.mu.RLock()
|
||||
defer nq.mu.RUnlock()
|
||||
|
||||
rows, err := nq.db.Query(`
|
||||
SELECT notification_id, type, method, status, alert_identifiers,
|
||||
alert_count, operational_links, attempts, success, error_message,
|
||||
failure_class, destination_id, timestamp
|
||||
FROM notification_audit
|
||||
WHERE timestamp >= ?
|
||||
ORDER BY timestamp DESC, id DESC
|
||||
LIMIT ?
|
||||
`, since.UTC().Unix(), limit)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("read notification delivery log: %w", err)
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
entries := make([]DeliveryLogEntry, 0, limit)
|
||||
for rows.Next() {
|
||||
var (
|
||||
entry DeliveryLogEntry
|
||||
method sql.NullString
|
||||
status sql.NullString
|
||||
alertIdentifiers sql.NullString
|
||||
alertCount sql.NullInt64
|
||||
operationalLinks sql.NullString
|
||||
attempts sql.NullInt64
|
||||
success sql.NullBool
|
||||
errorMessage sql.NullString
|
||||
failureClass sql.NullString
|
||||
destinationID sql.NullString
|
||||
timestamp int64
|
||||
)
|
||||
if err := rows.Scan(
|
||||
&entry.NotificationID,
|
||||
&entry.Type,
|
||||
&method,
|
||||
&status,
|
||||
&alertIdentifiers,
|
||||
&alertCount,
|
||||
&operationalLinks,
|
||||
&attempts,
|
||||
&success,
|
||||
&errorMessage,
|
||||
&failureClass,
|
||||
&destinationID,
|
||||
×tamp,
|
||||
); err != nil {
|
||||
return nil, fmt.Errorf("scan notification delivery log row: %w", err)
|
||||
}
|
||||
|
||||
entry.Method = method.String
|
||||
entry.Attempts = int(attempts.Int64)
|
||||
entry.Success = success.Bool
|
||||
entry.ErrorMessage = errorMessage.String
|
||||
entry.FailureClass = failureClass.String
|
||||
entry.Timestamp = time.Unix(timestamp, 0).UTC()
|
||||
entry.Outcome = deliveryOutcome(
|
||||
NotificationQueueStatus(status.String), success.Bool, int(attempts.Int64),
|
||||
)
|
||||
|
||||
entry.AlertIDs = decodeAuditAlertIdentifiers(alertIdentifiers.String)
|
||||
entry.AlertCount = int(alertCount.Int64)
|
||||
if entry.AlertCount == 0 {
|
||||
entry.AlertCount = len(entry.AlertIDs)
|
||||
}
|
||||
// Rows written before the destination_id column existed still carry the
|
||||
// destination inside their operational links.
|
||||
entry.DestinationID = strings.TrimSpace(destinationID.String)
|
||||
if entry.DestinationID == "" {
|
||||
entry.DestinationID = decodeAuditDestinationID(operationalLinks.String)
|
||||
}
|
||||
|
||||
entries = append(entries, entry)
|
||||
}
|
||||
if err := rows.Err(); err != nil {
|
||||
return nil, fmt.Errorf("iterate notification delivery log: %w", err)
|
||||
}
|
||||
return entries, nil
|
||||
}
|
||||
|
||||
func decodeAuditAlertIdentifiers(raw string) []string {
|
||||
if strings.TrimSpace(raw) == "" {
|
||||
return []string{}
|
||||
}
|
||||
var identifiers []string
|
||||
if err := json.Unmarshal([]byte(raw), &identifiers); err != nil {
|
||||
return []string{}
|
||||
}
|
||||
if identifiers == nil {
|
||||
return []string{}
|
||||
}
|
||||
return identifiers
|
||||
}
|
||||
|
||||
func decodeAuditDestinationID(raw string) string {
|
||||
if strings.TrimSpace(raw) == "" {
|
||||
return ""
|
||||
}
|
||||
var links []operationaltrust.NotificationLink
|
||||
if err := json.Unmarshal([]byte(raw), &links); err != nil {
|
||||
return ""
|
||||
}
|
||||
for _, link := range links {
|
||||
if destination := strings.TrimSpace(link.DestinationID); destination != "" {
|
||||
return destination
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// GetDeliveryLog exposes the queue's delivery log through the manager. A
|
||||
// missing queue is an error, not an empty log: silence must not read as "no
|
||||
// attempts were made".
|
||||
func (n *NotificationManager) GetDeliveryLog(since time.Time, limit int) ([]DeliveryLogEntry, error) {
|
||||
if n == nil {
|
||||
return nil, fmt.Errorf("notification manager not initialized")
|
||||
}
|
||||
n.mu.RLock()
|
||||
queue := n.queue
|
||||
n.mu.RUnlock()
|
||||
|
||||
if queue == nil {
|
||||
return nil, fmt.Errorf("notification queue not initialized")
|
||||
}
|
||||
return queue.GetDeliveryLog(since, limit)
|
||||
}
|
||||
@@ -0,0 +1,213 @@
|
||||
package notifications
|
||||
|
||||
import (
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/alerts"
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/operationaltrust"
|
||||
)
|
||||
|
||||
func TestGetDeliveryLogReturnsWindowedAttemptsNewestFirst(t *testing.T) {
|
||||
nq, err := NewNotificationQueue(t.TempDir())
|
||||
if err != nil {
|
||||
t.Fatalf("NewNotificationQueue: %v", err)
|
||||
}
|
||||
defer func() { _ = nq.Stop() }()
|
||||
|
||||
now := time.Now().UTC()
|
||||
sent := &QueuedNotification{
|
||||
ID: "email-sent",
|
||||
Type: "email",
|
||||
DestinationID: "destination:aaaa",
|
||||
Status: QueueStatusPending,
|
||||
Alerts: []*alerts.Alert{{ID: "disk-critical-1", StartTime: now.Add(-time.Hour)}},
|
||||
Config: []byte(`{}`),
|
||||
CreatedAt: now.Add(-time.Hour),
|
||||
}
|
||||
failed := &QueuedNotification{
|
||||
ID: "webhook-dlq",
|
||||
Type: "webhook",
|
||||
DestinationID: "webhook:wh-ops",
|
||||
Status: QueueStatusPending,
|
||||
Alerts: []*alerts.Alert{{ID: "vm-offline-101", StartTime: now.Add(-time.Hour)}},
|
||||
Config: []byte(`{}`),
|
||||
CreatedAt: now.Add(-time.Hour),
|
||||
}
|
||||
old := &QueuedNotification{
|
||||
ID: "email-old",
|
||||
Type: "email",
|
||||
Status: QueueStatusPending,
|
||||
Config: []byte(`{}`),
|
||||
CreatedAt: now.Add(-8 * 24 * time.Hour),
|
||||
}
|
||||
for _, notif := range []*QueuedNotification{sent, failed, old} {
|
||||
if err := nq.Enqueue(notif); err != nil {
|
||||
t.Fatalf("enqueue %s: %v", notif.ID, err)
|
||||
}
|
||||
}
|
||||
|
||||
old.Attempts = 1
|
||||
old.Status = QueueStatusSent
|
||||
if err := nq.RecordAudit(old, true, ""); err != nil {
|
||||
t.Fatalf("record old delivery: %v", err)
|
||||
}
|
||||
sent.Attempts = 1
|
||||
sent.Status = QueueStatusSent
|
||||
if err := nq.RecordAudit(sent, true, ""); err != nil {
|
||||
t.Fatalf("record sent delivery: %v", err)
|
||||
}
|
||||
failed.Attempts = 1
|
||||
failed.Status = QueueStatusPending
|
||||
if err := nq.RecordAudit(failed, false, "HTTP 401 Unauthorized"); err != nil {
|
||||
t.Fatalf("record retry attempt: %v", err)
|
||||
}
|
||||
failed.Attempts = 3
|
||||
failed.Status = QueueStatusDLQ
|
||||
if err := nq.RecordAudit(failed, false, "HTTP 401 Unauthorized"); err != nil {
|
||||
t.Fatalf("record dead-letter attempt: %v", err)
|
||||
}
|
||||
if _, err := nq.db.Exec(
|
||||
`UPDATE notification_audit SET timestamp = ? WHERE notification_id = ?`,
|
||||
now.Add(-8*24*time.Hour).Unix(),
|
||||
"email-old",
|
||||
); err != nil {
|
||||
t.Fatalf("age old audit row: %v", err)
|
||||
}
|
||||
|
||||
entries, err := nq.GetDeliveryLog(now.Add(-7*24*time.Hour), 0)
|
||||
if err != nil {
|
||||
t.Fatalf("GetDeliveryLog: %v", err)
|
||||
}
|
||||
if len(entries) != 3 {
|
||||
t.Fatalf("entries = %d, want 3 (windowed rows only): %#v", len(entries), entries)
|
||||
}
|
||||
|
||||
deadLetter := entries[0]
|
||||
if deadLetter.NotificationID != "webhook-dlq" ||
|
||||
deadLetter.Outcome != DeliveryOutcomeDeadLetter ||
|
||||
deadLetter.DestinationID != "webhook:wh-ops" ||
|
||||
deadLetter.Attempts != 3 ||
|
||||
deadLetter.Success ||
|
||||
deadLetter.FailureClass != "authentication" {
|
||||
t.Fatalf("dead-letter entry = %#v", deadLetter)
|
||||
}
|
||||
if len(deadLetter.AlertIDs) != 1 || deadLetter.AlertIDs[0] != "vm-offline-101" ||
|
||||
deadLetter.AlertCount != 1 {
|
||||
t.Fatalf("dead-letter alerts = %#v", deadLetter)
|
||||
}
|
||||
|
||||
retry := entries[1]
|
||||
if retry.NotificationID != "webhook-dlq" || retry.Outcome != DeliveryOutcomeRetry {
|
||||
t.Fatalf("retry entry = %#v", retry)
|
||||
}
|
||||
|
||||
delivered := entries[2]
|
||||
if delivered.NotificationID != "email-sent" ||
|
||||
delivered.Outcome != DeliveryOutcomeSent ||
|
||||
!delivered.Success ||
|
||||
delivered.DestinationID != "destination:aaaa" ||
|
||||
delivered.FailureClass != "" ||
|
||||
delivered.ErrorMessage != "" {
|
||||
t.Fatalf("delivered entry = %#v", delivered)
|
||||
}
|
||||
if delivered.Timestamp.IsZero() {
|
||||
t.Fatalf("delivered entry has no timestamp: %#v", delivered)
|
||||
}
|
||||
}
|
||||
|
||||
func TestGetDeliveryLogHonorsLimitAndCap(t *testing.T) {
|
||||
nq, err := NewNotificationQueue(t.TempDir())
|
||||
if err != nil {
|
||||
t.Fatalf("NewNotificationQueue: %v", err)
|
||||
}
|
||||
defer func() { _ = nq.Stop() }()
|
||||
|
||||
now := time.Now().UTC()
|
||||
notif := &QueuedNotification{
|
||||
ID: "email-1",
|
||||
Type: "email",
|
||||
Status: QueueStatusPending,
|
||||
Config: []byte(`{}`),
|
||||
CreatedAt: now,
|
||||
}
|
||||
if err := nq.Enqueue(notif); err != nil {
|
||||
t.Fatalf("enqueue: %v", err)
|
||||
}
|
||||
notif.Status = QueueStatusSent
|
||||
for i := 0; i < 3; i++ {
|
||||
notif.Attempts = i + 1
|
||||
if err := nq.RecordAudit(notif, true, ""); err != nil {
|
||||
t.Fatalf("record delivery %d: %v", i, err)
|
||||
}
|
||||
}
|
||||
|
||||
entries, err := nq.GetDeliveryLog(now.Add(-time.Hour), 2)
|
||||
if err != nil {
|
||||
t.Fatalf("GetDeliveryLog: %v", err)
|
||||
}
|
||||
if len(entries) != 2 {
|
||||
t.Fatalf("entries = %d, want limit of 2", len(entries))
|
||||
}
|
||||
if entries[0].Attempts != 3 {
|
||||
t.Fatalf("newest entry = %#v, want the latest attempt first", entries[0])
|
||||
}
|
||||
|
||||
if capped, err := nq.GetDeliveryLog(now.Add(-time.Hour), maxDeliveryLogEntries*10); err != nil {
|
||||
t.Fatalf("GetDeliveryLog capped: %v", err)
|
||||
} else if len(capped) != 3 {
|
||||
t.Fatalf("capped entries = %d, want all 3", len(capped))
|
||||
}
|
||||
}
|
||||
|
||||
func TestGetDeliveryLogFallsBackToOperationalLinksForLegacyRows(t *testing.T) {
|
||||
nq, err := NewNotificationQueue(t.TempDir())
|
||||
if err != nil {
|
||||
t.Fatalf("NewNotificationQueue: %v", err)
|
||||
}
|
||||
defer func() { _ = nq.Stop() }()
|
||||
|
||||
now := time.Now().UTC()
|
||||
notif := &QueuedNotification{
|
||||
ID: "webhook-legacy",
|
||||
Type: "webhook",
|
||||
Status: QueueStatusPending,
|
||||
Config: []byte(`{}`),
|
||||
CreatedAt: now,
|
||||
}
|
||||
if err := nq.Enqueue(notif); err != nil {
|
||||
t.Fatalf("enqueue: %v", err)
|
||||
}
|
||||
notif.Status = QueueStatusSent
|
||||
notif.Attempts = 1
|
||||
notif.Links = []operationaltrust.NotificationLink{{DestinationID: "webhook:legacy"}}
|
||||
if err := nq.RecordAudit(notif, true, ""); err != nil {
|
||||
t.Fatalf("record delivery: %v", err)
|
||||
}
|
||||
// Simulate a row written before the destination_id column existed.
|
||||
if _, err := nq.db.Exec(
|
||||
`UPDATE notification_audit SET destination_id = '' WHERE notification_id = ?`,
|
||||
"webhook-legacy",
|
||||
); err != nil {
|
||||
t.Fatalf("clear destination column: %v", err)
|
||||
}
|
||||
|
||||
entries, err := nq.GetDeliveryLog(now.Add(-time.Hour), 0)
|
||||
if err != nil {
|
||||
t.Fatalf("GetDeliveryLog: %v", err)
|
||||
}
|
||||
if len(entries) != 1 || entries[0].DestinationID != "webhook:legacy" {
|
||||
t.Fatalf("entries = %#v, want destination decoded from links", entries)
|
||||
}
|
||||
}
|
||||
|
||||
func TestGetDeliveryLogFailsClosedWithoutQueue(t *testing.T) {
|
||||
var nq *NotificationQueue
|
||||
if _, err := nq.GetDeliveryLog(time.Now().Add(-time.Hour), 0); err == nil {
|
||||
t.Fatal("nil queue returned a delivery log instead of an error")
|
||||
}
|
||||
var manager *NotificationManager
|
||||
if _, err := manager.GetDeliveryLog(time.Now().Add(-time.Hour), 0); err == nil {
|
||||
t.Fatal("nil manager returned a delivery log instead of an error")
|
||||
}
|
||||
}
|
||||
@@ -28,6 +28,7 @@ const (
|
||||
legacyNotificationAuditAlertIdentifiersColumn = "alert_ids"
|
||||
notificationOperationalLinksColumn = "operational_links"
|
||||
notificationFailureClassColumn = "failure_class"
|
||||
notificationAuditDestinationColumn = "destination_id"
|
||||
notificationQueueDirName = "notifications"
|
||||
notificationQueueFileName = "notification_queue.db"
|
||||
)
|
||||
@@ -412,6 +413,7 @@ func (nq *NotificationQueue) initSchema() error {
|
||||
success BOOLEAN,
|
||||
error_message TEXT,
|
||||
failure_class TEXT NOT NULL DEFAULT '',
|
||||
destination_id TEXT NOT NULL DEFAULT '',
|
||||
payload_size INTEGER,
|
||||
timestamp INTEGER NOT NULL,
|
||||
FOREIGN KEY (notification_id) REFERENCES notification_queue(id)
|
||||
@@ -452,9 +454,15 @@ func (nq *NotificationQueue) initSchema() error {
|
||||
); err != nil {
|
||||
return err
|
||||
}
|
||||
return nq.ensureTextColumn(
|
||||
if err := nq.ensureTextColumn(
|
||||
"notification_audit",
|
||||
notificationFailureClassColumn,
|
||||
); err != nil {
|
||||
return err
|
||||
}
|
||||
return nq.ensureTextColumn(
|
||||
"notification_audit",
|
||||
notificationAuditDestinationColumn,
|
||||
)
|
||||
}
|
||||
|
||||
@@ -1284,8 +1292,8 @@ func (nq *NotificationQueue) RecordAudit(notif *QueuedNotification, success bool
|
||||
|
||||
query := `
|
||||
INSERT INTO notification_audit
|
||||
(notification_id, type, method, status, alert_identifiers, alert_count, operational_links, attempts, success, error_message, failure_class, payload_size, timestamp)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
|
||||
(notification_id, type, method, status, alert_identifiers, alert_count, operational_links, attempts, success, error_message, failure_class, destination_id, payload_size, timestamp)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
|
||||
`
|
||||
|
||||
failureClass := ""
|
||||
@@ -1304,22 +1312,14 @@ func (nq *NotificationQueue) RecordAudit(notif *QueuedNotification, success bool
|
||||
success,
|
||||
errorMsg,
|
||||
failureClass,
|
||||
strings.TrimSpace(notif.DestinationID),
|
||||
notif.PayloadBytes,
|
||||
time.Now().Unix(),
|
||||
)
|
||||
if err == nil {
|
||||
outcome := "failed"
|
||||
switch {
|
||||
case success && notif.Status == QueueStatusSent:
|
||||
outcome = "sent"
|
||||
case notif.Status == QueueStatusDLQ:
|
||||
outcome = "dead_letter"
|
||||
case notif.Status == QueueStatusCancelled:
|
||||
outcome = "cancelled"
|
||||
case notif.Status == QueueStatusPending && notif.Attempts > 0:
|
||||
outcome = "retry"
|
||||
}
|
||||
operationaltrust.GetMetrics().ObserveNotificationDelivery(outcome)
|
||||
operationaltrust.GetMetrics().ObserveNotificationDelivery(
|
||||
deliveryOutcome(notif.Status, success, notif.Attempts),
|
||||
)
|
||||
}
|
||||
return err
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user