gsadmin/pulse
1
0
Fork 0
mirror of https://github.com/rcourtman/Pulse.git synced 2026-10-23 20:25:56 +00:00
Code Issues Packages Projects Releases Wiki Activity

Make alert escalations destination-specific and repeatable

Browse Source
This commit is contained in:
rcourtman
2026-08-27 19:48:56 +01:00
parent 5f6bfefa56
commit e0baecc418
37 changed files with 1015 additions and 85 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/release-control/v6/internal/subsystems/alerts.md
+20
View File
@@ -2184,3 +2184,23 @@ idempotent fingerprints. A restart interruption is raised into history even
when the first successful external report immediately resolves it, so recovery
does not erase the outage record. The watchdog must never invent resource
identity or become a second lifecycle for infrastructure alerts.
### Escalation targets are exact and critical repeats are bounded
An escalation level may address credential-free logical destination IDs:
`email`, `apprise`, or `webhook:<id>`. Once exact IDs are present they are the
authority for that level; a deleted or unknown ID fails closed and must never
widen to a channel or every webhook. The legacy `notify` channel remains a
mixed-version fallback for configurations that do not yet carry exact IDs.
Optional critical-repeat policy applies only after the final escalation level.
It sends at most one repeat per checker pass, uses a normalized 5–180 minute
interval, and stops while the incident is acknowledged, resolved, snoozed, or
global detection or delivery controls are paused. A repeat is durable escalation
evidence with an explicit repeat marker rather than a new incident or a replay
of missed intervals.
The Schedule surface loads the same email, Apprise, and webhook catalog as the
Destinations surface. It shows disabled and deleted selections explicitly,
prevents an escalation level from becoming destinationless, and explains the
critical-repeat stop conditions before save.
docs/release-control/v6/internal/subsystems/frontend-primitives.md
+11
View File
@@ -6686,3 +6686,14 @@ its privacy and current-state boundary—but it must not redefine the policy.
Alert feature state owns persistence and entitlement gating; the primitive
owns presentation only. `alertDestinationsPresentation.test.ts` pins the
shared vocabulary and the distinct mobile guidance.
### Escalation configuration uses destination identity, not channel aliases
The alerts-owned escalation section composes shared settings, toggle, and form
controls while presenting a feature-owned checkbox catalog keyed by logical
destination ID. It sources that catalog from the same loaded destination state
used by the Notifications tab, preserves selected-but-disabled and deleted
entries visibly, and prevents the final selection from being removed. The
critical-repeat control exposes a bounded numeric interval only when enabled
and states every lifecycle condition that stops paging. Desktop and mobile
layouts must retain associated labels and avoid horizontal overflow.
docs/release-control/v6/internal/subsystems/monitoring.md
+9
View File
@@ -3293,3 +3293,12 @@ in-flight request before durable stop or replacement state is published, so a
revoked endpoint receives no trailing heartbeat and a removed destination
becomes disabled immediately. State corruption and write failure are visible
system conditions rather than silent loss of future outage evidence.
### Escalation callbacks preserve exact routing intent
The monitoring callback resolves the configured escalation level and forwards
exact logical destination IDs to notification delivery when present. Legacy
levels continue to route by channel, including Apprise; this compatibility path
must not silently skip a supported destination. Monitoring broadcasts the
escalated alert after dispatch but does not reinterpret destination identity,
retry semantics, acknowledgement, or the critical-repeat cadence.
docs/release-control/v6/internal/subsystems/notifications.md
+14
View File
@@ -472,3 +472,17 @@ have queued delivery cancelled by an uncommitted disable action. The forced
failure paths are pinned in
`internal/api/alerting/notifications_test.go`; the API ordering proof lives in
`internal/api/contract_test.go`.
### Escalation delivery can select exact logical destinations
Escalation dispatch accepts optional logical destination IDs in addition to the
legacy channel target. Exact selection is applied before destination tag and
severity policy: `email` selects the enabled email job, `apprise` selects the
enabled Apprise job, and `webhook:<id>` selects only that enabled webhook.
Unknown or unavailable IDs produce no job and never fall back to a broader
channel. A no-job exact selection also leaves notification cooldown state
untouched so configuration drift cannot masquerade as successful paging.
Destination IDs never contain endpoint URLs, credentials, or Apprise targets.
All retry, queue, receipt, grouping, and recovery semantics remain owned by the
selected notification job after routing.
frontend-modern/browser-verification.json
+39 -14
View File
@@ -1,30 +1,55 @@
{
"version": 1,
"base_sha": "8df9db7ab1159c9384b233d6f1e45a78720d231e",
"verified_at": "2026-08-27T18:21:17Z",
"base_sha": "5f6bfefa5603adc16c349115b8f453c8430ddce3",
"verified_at": "2026-08-27T18:39:55Z",
"result": "passed",
"changed_paths": [
"frontend-modern/src/components/Infrastructure/ResourceInvestigationContextTables.tsx",
"frontend-modern/src/components/shared/DetailSectionTable.tsx",
"frontend-modern/src/components/shared/detailSectionModel.ts"
"frontend-modern/src/features/alerts/AlertEscalationSection.tsx",
"frontend-modern/src/features/alerts/AlertsConfigurationSurface.tsx",
"frontend-modern/src/features/alerts/alertsConfigurationModel.ts",
"frontend-modern/src/features/alerts/helpers.ts",
"frontend-modern/src/features/alerts/tabs/ScheduleTab.tsx",
"frontend-modern/src/features/alerts/types.ts",
"frontend-modern/src/features/alerts/useAlertDestinationsState.ts",
"frontend-modern/src/features/alerts/useAlertDestinationsTabState.ts",
"frontend-modern/src/features/alerts/useAlertScheduleState.ts",
"frontend-modern/src/features/alerts/useAlertWebhookDestinationsState.ts",
"frontend-modern/src/features/alerts/useAlertsConfigurationSnapshotState.ts",
"frontend-modern/src/features/alerts/useAlertsConfigurationState.ts",
"frontend-modern/src/types/alerts.ts",
"frontend-modern/src/utils/alertConfigPresentation.ts"
],
"content_sha256": {
"frontend-modern/src/components/Infrastructure/ResourceInvestigationContextTables.tsx": "86708854264fb59d14aa23766ccceb406301b9606e3af5ec5964ae524170f2d4",
"frontend-modern/src/components/shared/DetailSectionTable.tsx": "135153e1a29dc85afe8867643a31cbb66a6a13468f0f3af8efdd4be049f4d223",
"frontend-modern/src/components/shared/detailSectionModel.ts": "c98b6befd1348282abd990c47886fb5af60e362fdbb6c453d60f36e2fd3ebe1c"
"frontend-modern/src/features/alerts/AlertEscalationSection.tsx": "f4b12a0f7d5da49961ac952b94963d61a1b1f59bc3234ed7e961c9d264139021",
"frontend-modern/src/features/alerts/AlertsConfigurationSurface.tsx": "c514dd4b9f769c58af9af4fa5fcf647163224c85e6c4e640969a34c1d3b24708",
"frontend-modern/src/features/alerts/alertsConfigurationModel.ts": "855189aaf23c66b14ac43a1b7b0101e7fc0d59348a09415971a16a276f083f7f",
"frontend-modern/src/features/alerts/helpers.ts": "6ba556d57241fc7d7bfff7218d0dd924eaaeb9507f73ccda62204def14c3be61",
"frontend-modern/src/features/alerts/tabs/ScheduleTab.tsx": "7e5fa374edf72de0b9216f672cd179c183d348e22f087827c478bb29170ba9cf",
"frontend-modern/src/features/alerts/types.ts": "9b7c4331ac52f06d889fcdd02e75a95b247945cea6036a10e6fbea30e55d1eca",
"frontend-modern/src/features/alerts/useAlertDestinationsState.ts": "abf1a8c0acd32fbc4a1c7177dca213edee817b637af953c0f5a5ec275d41a600",
"frontend-modern/src/features/alerts/useAlertDestinationsTabState.ts": "e9b2cabfaa1720ab430c410fdb269a164710478bcef775a5d163f69cc3965226",
"frontend-modern/src/features/alerts/useAlertScheduleState.ts": "fc439f16c0b79bfa6c01a74a3359ae0cb58e5765c074d75107c1ba8bd763cf31",
"frontend-modern/src/features/alerts/useAlertWebhookDestinationsState.ts": "ee7a5d6c75902ab66a92ca0292b367bc02ba8dd268cab1e011fe719d09251fa9",
"frontend-modern/src/features/alerts/useAlertsConfigurationSnapshotState.ts": "af9014e6d93b0aa22399470efa00f8e048df7fbe54559092d8749b3fcf5bb43a",
"frontend-modern/src/features/alerts/useAlertsConfigurationState.ts": "20d61b9a97180d56aa6cef8445ba9d53e7043a22cc4020e6f72b505ed0c38fe7",
"frontend-modern/src/types/alerts.ts": "24999c6668c262214fcd3b8a92b09e7a68cab2163b8a4876780ce1148d9064cf",
"frontend-modern/src/utils/alertConfigPresentation.ts": "82d04fb6ce0c209fa968b1dabf19333dffae5a7317441a8a2cac5cfda86b831c"
},
"routes": ["/proxmox/overview"],
"routes": ["/alerts/schedule"],
"viewports": [
{ "width": 1920, "height": 800 },
{ "width": 390, "height": 844 }
],
"states": [
"expanded backup-vault drawer after extending the shared responsive section primitive",
"Analysis and Governance responsive section state covered by focused rendered drawer tests"
"escalation enabled with one level and the unsaved-changes guard visible",
"exact Email selected while Apprise is unselected and both disabled states remain explicit",
"repeat critical escalation enabled with a 30 minute interval"
],
"interactions": [
"confirmed shared desktop sections remain aligned equal-width cards with adjacent labels and values",
"confirmed compact mobile table rows remain intact at 390x844 with zero horizontal overflow",
"reloaded the final implementation and confirmed no new browser console errors"
"enabled escalation and added a level without saving the live configuration",
"confirmed exact destination selection updates independently and never broadens to another destination",
"confirmed repeat-until-ack exposes its bounded interval and stop-condition guidance",
"confirmed the 390x844 schedule layout has zero horizontal overflow",
"confirmed no browser console errors on the exercised route"
]
}
frontend-modern/src/features/alerts/AlertEscalationSection.tsx
+109 -30
View File
@@ -1,29 +1,37 @@
import { For, Show, createUniqueId } from 'solid-js';
import { controlClass, formHelpText } from '@/components/shared/Form';
import { FormSelect } from '@/components/shared/FormSelect';
import { SettingsPanel } from '@/components/shared/SettingsPanel';
import { Toggle } from '@/components/shared/Toggle';
import {
ALERT_CONFIG_ESCALATION_ADD_LABEL,
ALERT_CONFIG_ESCALATION_AFTER_LABEL,
ALERT_CONFIG_ESCALATION_DESCRIPTION,
ALERT_CONFIG_ESCALATION_DESTINATIONS_HELP,
ALERT_CONFIG_ESCALATION_DESTINATION_DISABLED,
ALERT_CONFIG_ESCALATION_DESTINATION_UNAVAILABLE,
ALERT_CONFIG_ESCALATION_MINUTES_SUFFIX,
ALERT_CONFIG_ESCALATION_NOTIFY_LABEL,
ALERT_CONFIG_ESCALATION_REMOVE_TITLE,
ALERT_CONFIG_ESCALATION_REPEAT_EVERY_LABEL,
ALERT_CONFIG_ESCALATION_REPEAT_HELP,
ALERT_CONFIG_ESCALATION_REPEAT_LABEL,
ALERT_CONFIG_ESCALATION_TITLE,
getAlertConfigEscalationHelp,
getAlertConfigEscalationNotifyLabel,
getAlertConfigToggleStatusLabel,
} from '@/utils/alertConfigPresentation';
import type { EscalationConfig, EscalationNotifyTarget } from './types';
import type { EscalationConfig, EscalationDestination, EscalationNotifyTarget } from './types';
interface AlertEscalationSectionProps {
escalation: EscalationConfig;
setEscalationEnabled: (value: boolean) => void;
setEscalationAfter: (index: number, value: string) => void;
setEscalationNotify: (index: number, value: EscalationNotifyTarget) => void;
destinations: EscalationDestination[];
setEscalationDestinationIds: (index: number, destinationIds: string[]) => void;
setEscalationRepeatCritical: (value: boolean) => void;
setEscalationRepeatEvery: (value: string) => void;
removeEscalationLevel: (index: number) => void;
addEscalationLevel: () => void;
}
@@ -59,11 +67,34 @@ export function AlertEscalationSection(props: AlertEscalationSectionProps) {
<For each={props.escalation.levels}>
{(level, index) => {
const afterId = () => `${fieldIdPrefix}-after-${index()}`;
const notifyId = () => `${fieldIdPrefix}-notify-${index()}`;
const destinationsId = () => `${fieldIdPrefix}-destinations-${index()}`;
const selectedDestinationIds = () =>
level.destinationIds && level.destinationIds.length > 0
? level.destinationIds
: props.destinations
.filter(
(destination) =>
level.notify === 'all' || destination.kind === level.notify,
)
.map((destination) => destination.id);
const destinationOptions = () => [
...props.destinations,
...selectedDestinationIds()
.filter((id) => !props.destinations.some((destination) => destination.id === id))
.map(
(id) =>
({
id,
label: `${ALERT_CONFIG_ESCALATION_DESTINATION_UNAVAILABLE} (${id})`,
kind: id === 'email' ? 'email' : id === 'apprise' ? 'apprise' : 'webhook',
enabled: false,
}) as EscalationDestination,
),
];
return (
<div class="flex items-center gap-3 rounded-md border border-border bg-surface-hover p-3">
<div class="flex flex-1 flex-col gap-3 sm:grid sm:grid-cols-2 sm:items-center sm:gap-2">
<div class="flex items-start gap-3 rounded-md border border-border bg-surface-hover p-3">
<div class="flex flex-1 flex-col gap-3">
<div class="flex items-center gap-2">
<label for={afterId()} class="text-xs font-medium text-muted">
{ALERT_CONFIG_ESCALATION_AFTER_LABEL}
@@ -83,30 +114,43 @@ export function AlertEscalationSection(props: AlertEscalationSectionProps) {
{ALERT_CONFIG_ESCALATION_MINUTES_SUFFIX}
</span>
</div>
<FormSelect
id={notifyId()}
label={ALERT_CONFIG_ESCALATION_NOTIFY_LABEL}
value={level.notify}
onChange={(event) =>
props.setEscalationNotify(
index(),
event.currentTarget.value as EscalationNotifyTarget,
)
}
fieldBaseClass="flex"
fieldClass="items-center gap-2"
labelClass="text-xs font-medium text-muted"
selectBaseClass={`${controlClass('px-2 py-1 text-sm')} flex-1`}
>
<option value="email">{getAlertConfigEscalationNotifyLabel('email')}</option>
<option value="webhook">
{getAlertConfigEscalationNotifyLabel('webhook')}
</option>
<option value="apprise">
{getAlertConfigEscalationNotifyLabel('apprise')}
</option>
<option value="all">{getAlertConfigEscalationNotifyLabel('all')}</option>
</FormSelect>
<fieldset id={destinationsId()} class="space-y-2">
<legend class="text-xs font-medium text-muted">
{ALERT_CONFIG_ESCALATION_NOTIFY_LABEL}
</legend>
<div class="grid gap-2 sm:grid-cols-2">
<For each={destinationOptions()}>
{(destination) => {
const checked = () => selectedDestinationIds().includes(destination.id);
return (
<label class="flex min-w-0 items-center gap-2 rounded-md border border-border bg-surface px-2.5 py-2 text-sm">
<input
type="checkbox"
checked={checked()}
disabled={checked() && selectedDestinationIds().length === 1}
onChange={(event) => {
const next = event.currentTarget.checked
? [...selectedDestinationIds(), destination.id]
: selectedDestinationIds().filter(
(id) => id !== destination.id,
);
props.setEscalationDestinationIds(index(), next);
}}
class="h-4 w-4 rounded border-border"
/>
<span class="min-w-0 truncate">{destination.label}</span>
<Show when={!destination.enabled}>
<span class="ml-auto text-xs text-muted">
{ALERT_CONFIG_ESCALATION_DESTINATION_DISABLED}
</span>
</Show>
</label>
);
}}
</For>
</div>
<p class={formHelpText}>{ALERT_CONFIG_ESCALATION_DESTINATIONS_HELP}</p>
</fieldset>
</div>
<button
type="button"
@@ -129,6 +173,41 @@ export function AlertEscalationSection(props: AlertEscalationSectionProps) {
}}
</For>
<Show when={props.escalation.levels.length > 0}>
<div class="space-y-3 rounded-md border border-border bg-surface p-3">
<Toggle
checked={props.escalation.repeatCritical}
onChange={(event) => props.setEscalationRepeatCritical(event.currentTarget.checked)}
label={
<span class="text-sm font-medium text-base-content">
{ALERT_CONFIG_ESCALATION_REPEAT_LABEL}
</span>
}
/>
<Show when={props.escalation.repeatCritical}>
<div class="flex items-center gap-2">
<label
for={`${fieldIdPrefix}-repeat-every`}
class="text-xs font-medium text-muted"
>
{ALERT_CONFIG_ESCALATION_REPEAT_EVERY_LABEL}
</label>
<input
id={`${fieldIdPrefix}-repeat-every`}
type="number"
min="5"
max="180"
value={props.escalation.repeatEvery}
onChange={(event) => props.setEscalationRepeatEvery(event.currentTarget.value)}
class={`${controlClass('px-2 py-1 text-sm')} w-20`}
/>
<span class="text-xs text-muted">{ALERT_CONFIG_ESCALATION_MINUTES_SUFFIX}</span>
</div>
</Show>
<p class={formHelpText}>{ALERT_CONFIG_ESCALATION_REPEAT_HELP}</p>
</div>
</Show>
<button
type="button"
onClick={props.addEscalationLevel}
frontend-modern/src/features/alerts/AlertScheduleSections.test.tsx
+17 -3
View File
@@ -119,7 +119,9 @@ describe('Alert schedule sections', () => {
it('associates visible labels with escalation controls', () => {
const escalation: EscalationConfig = {
enabled: true,
levels: [{ after: 15, notify: 'email' }],
levels: [{ after: 15, notify: 'email', destinationIds: ['email'] }],
repeatCritical: true,
repeatEvery: 30,
};
render(() => (
@@ -128,6 +130,13 @@ describe('Alert schedule sections', () => {
setEscalationEnabled={vi.fn()}
setEscalationAfter={vi.fn()}
setEscalationNotify={vi.fn()}
destinations={[
{ id: 'email', label: 'Email', kind: 'email', enabled: true },
{ id: 'apprise', label: 'Apprise', kind: 'apprise', enabled: false },
]}
setEscalationDestinationIds={vi.fn()}
setEscalationRepeatCritical={vi.fn()}
setEscalationRepeatEvery={vi.fn()}
removeEscalationLevel={vi.fn()}
addEscalationLevel={vi.fn()}
/>
@@ -138,8 +147,13 @@ describe('Alert schedule sections', () => {
'true',
);
expect(screen.getByRole('spinbutton', { name: 'After' })).toBeInTheDocument();
expect(screen.getByRole('combobox', { name: 'Notify' })).toBeInTheDocument();
expect(screen.getByRole('option', { name: 'Apprise' })).toBeInTheDocument();
expect(screen.getByRole('group', { name: 'Notify' })).toBeInTheDocument();
expect(screen.getByRole('checkbox', { name: 'Email' })).toBeChecked();
expect(screen.getByRole('checkbox', { name: /Apprise/ })).not.toBeChecked();
expect(
screen.getByRole('button', { name: 'Repeat critical escalation until acknowledged' }),
).toHaveAttribute('aria-pressed', 'true');
expect(screen.getByRole('spinbutton', { name: 'Repeat every' })).toHaveValue(30);
expect(screen.getByRole('button', { name: 'Remove escalation level' })).toBeInTheDocument();
});
frontend-modern/src/features/alerts/AlertsConfigurationSurface.tsx
+24 -1
View File
@@ -1,4 +1,4 @@
import { Show } from 'solid-js';
import { createMemo, Show } from 'solid-js';
import { Card } from '@/components/shared/Card';
import {
@@ -20,6 +20,26 @@ import {
export function AlertsConfigurationSurface(props: AlertsConfigurationSurfaceProps) {
const state = useAlertsConfigurationState(props);
const escalationDestinations = createMemo(() => [
{
id: 'email',
label: 'Email',
kind: 'email' as const,
enabled: state.emailConfig().enabled,
},
{
id: 'apprise',
label: 'Apprise',
kind: 'apprise' as const,
enabled: state.appriseConfig().enabled,
},
...state.webhooks().map((webhook) => ({
id: `webhook:${webhook.id}`,
label: webhook.name || webhook.id,
kind: 'webhook' as const,
enabled: webhook.enabled,
})),
]);
return (
<>
@@ -224,6 +244,8 @@ export function AlertsConfigurationSurface(props: AlertsConfigurationSurfaceProp
isRetrying={state.isReloadingConfig}
isLoadingDestinations={state.isLoadingDestinations}
onRetryLoad={() => void state.loadAlertConfiguration()}
webhooks={state.webhooks}
setWebhooks={state.setWebhooks}
/>
</Show>
@@ -242,6 +264,7 @@ export function AlertsConfigurationSurface(props: AlertsConfigurationSurfaceProp
setNotifyOnResolve={state.setNotifyOnResolve}
escalation={state.scheduleEscalation}
setEscalation={state.setScheduleEscalation}
escalationDestinations={escalationDestinations}
/>
</Show>
</>
frontend-modern/src/features/alerts/__tests__/alertsConfigurationModel.snapshot.test.ts
+33
View File
@@ -1180,6 +1180,39 @@ describe('readAlertsConfigurationSnapshot — schedule.escalation', () => {
]);
});
it('preserves exact destination IDs and critical repeat policy', () => {
const snapshot = readAlertsConfigurationSnapshot(
cfg({
schedule: {
escalation: {
enabled: true,
repeatCritical: true,
repeatEvery: 15,
levels: [
{
after: 30,
notify: 'webhook',
destinationIds: ['webhook:pager', 'apprise'],
},
],
},
},
}),
);
expect(snapshot.scheduleEscalation).toEqual({
enabled: true,
repeatCritical: true,
repeatEvery: 15,
levels: [
{
after: 30,
notify: 'webhook',
destinationIds: ['webhook:pager', 'apprise'],
},
],
});
});
it('defaults level.after to 15 for non-number values', () => {
const snapshot = readAlertsConfigurationSnapshot(
cfg({
frontend-modern/src/features/alerts/__tests__/alertsConfigurationModel.test.ts
+24
View File
@@ -130,6 +130,18 @@ describe('alertsConfigurationModel', () => {
byGuest: false,
};
snapshot.initialNotify = 'email';
snapshot.scheduleEscalation = {
enabled: true,
repeatCritical: true,
repeatEvery: 15,
levels: [
{
after: 30,
notify: 'webhook',
destinationIds: ['webhook:pager'],
},
],
};
snapshot.backupDefaults = {
enabled: true,
warningDays: 30,
@@ -169,6 +181,18 @@ describe('alertsConfigurationModel', () => {
byNode: true,
byGuest: false,
});
expect(result.alertConfig?.schedule?.escalation).toEqual({
enabled: true,
repeatCritical: true,
repeatEvery: 15,
levels: [
{
after: 30,
notify: 'webhook',
destinationIds: ['webhook:pager'],
},
],
});
expect(result.alertConfig?.backupDefaults).toMatchObject({
warningDays: 10,
criticalDays: 10,
frontend-modern/src/features/alerts/__tests__/useAlertDestinationsState.test.tsx
+17
View File
@@ -13,6 +13,7 @@ vi.mock('@/api/notifications', () => ({
NotificationsAPI: {
getAppriseConfig: vi.fn(),
getEmailConfig: vi.fn(),
getWebhooks: vi.fn(),
updateAppriseConfig: vi.fn(),
updateEmailConfig: vi.fn(),
},
@@ -46,6 +47,7 @@ describe('useAlertDestinationsState', () => {
beforeEach(() => {
vi.mocked(NotificationsAPI.getEmailConfig).mockReset();
vi.mocked(NotificationsAPI.getAppriseConfig).mockReset();
vi.mocked(NotificationsAPI.getWebhooks).mockReset();
vi.mocked(NotificationsAPI.updateEmailConfig).mockReset();
vi.mocked(NotificationsAPI.updateAppriseConfig).mockReset();
vi.mocked(AlertsAPI.getDeadManConfig).mockReset();
@@ -92,6 +94,16 @@ describe('useAlertDestinationsState', () => {
timeoutSeconds: 30,
skipTlsVerify: false,
} as any);
vi.mocked(NotificationsAPI.getWebhooks).mockResolvedValue([
{
id: 'pager',
name: 'Pager',
url: 'https://pager.example.test',
method: 'POST',
headers: {},
enabled: true,
},
]);
vi.mocked(AlertsAPI.getDeadManConfig).mockResolvedValue({
pingUrl: '***REDACTED***',
configured: true,
@@ -112,6 +124,7 @@ describe('useAlertDestinationsState', () => {
await result.loadDestinations();
expect(NotificationsAPI.getEmailConfig).toHaveBeenCalledTimes(1);
expect(NotificationsAPI.getAppriseConfig).toHaveBeenCalledTimes(1);
expect(NotificationsAPI.getWebhooks).toHaveBeenCalledTimes(1);
expect(AlertsAPI.getDeadManConfig).toHaveBeenCalledTimes(1);
expect(RelayAPI.getConfig).toHaveBeenCalledTimes(1);
expect(result.emailConfig().server).toBe('smtp.example.com');
@@ -120,12 +133,16 @@ describe('useAlertDestinationsState', () => {
expect(result.appriseConfig().minimumSeverity).toBe('critical');
expect(result.deadManPingUrl()).toBe('***REDACTED***');
expect(result.pushMinimumSeverity()).toBe('critical');
expect(result.webhooks()).toEqual([
expect.objectContaining({ id: 'pager', service: 'generic' }),
]);
setActiveTab('destinations');
await Promise.resolve();
await Promise.resolve();
expect(NotificationsAPI.getEmailConfig).toHaveBeenCalledTimes(2);
expect(NotificationsAPI.getAppriseConfig).toHaveBeenCalledTimes(2);
expect(NotificationsAPI.getWebhooks).toHaveBeenCalledTimes(2);
expect(AlertsAPI.getDeadManConfig).toHaveBeenCalledTimes(2);
expect(RelayAPI.getConfig).toHaveBeenCalledTimes(2);
frontend-modern/src/features/alerts/__tests__/useAlertScheduleState.test.tsx
+16 -1
View File
@@ -38,6 +38,10 @@ describe('useAlertScheduleState', () => {
setNotifyOnResolve,
escalation,
setEscalation,
escalationDestinations: () => [
{ id: 'email', label: 'Email', kind: 'email', enabled: true },
{ id: 'webhook:pager', label: 'Pager', kind: 'webhook', enabled: true },
],
}),
);
@@ -57,6 +61,9 @@ describe('useAlertScheduleState', () => {
result.addEscalationLevel();
result.setEscalationAfter(0, '30');
result.setEscalationNotify(0, 'webhook');
result.setEscalationDestinationIds(0, ['webhook:pager']);
result.setEscalationRepeatCritical(true);
result.setEscalationRepeatEvery('15');
expect(hasUnsavedChanges()).toBe(true);
expect(quietHours()).toMatchObject({
@@ -72,7 +79,15 @@ describe('useAlertScheduleState', () => {
expect(notifyOnResolve()).toBe(true);
expect(escalation()).toMatchObject({
enabled: true,
levels: [expect.objectContaining({ after: 30, notify: 'webhook' })],
repeatCritical: true,
repeatEvery: 15,
levels: [
expect.objectContaining({
after: 30,
notify: 'webhook',
destinationIds: ['webhook:pager'],
}),
],
});
result.removeEscalationLevel(0);
frontend-modern/src/features/alerts/alertsConfigurationModel.ts
+14 -1
View File
@@ -690,9 +690,17 @@ export function readAlertsConfigurationSnapshot(config: AlertConfig): AlertsConf
if (config.schedule.escalation) {
snapshot.scheduleEscalation = {
enabled: Boolean(config.schedule.escalation.enabled),
repeatCritical: Boolean(config.schedule.escalation.repeatCritical),
repeatEvery:
typeof config.schedule.escalation.repeatEvery === 'number'
? config.schedule.escalation.repeatEvery
: 30,
levels: (config.schedule.escalation.levels || []).map((level) => ({
after: typeof level.after === 'number' ? level.after : 15,
notify: normalizeNotificationDeliveryTarget(level.notify) as EscalationNotifyTarget,
...(Array.isArray(level.destinationIds)
? { destinationIds: level.destinationIds.filter((id) => typeof id === 'string') }
: {}),
})),
};
}
@@ -899,7 +907,12 @@ export function buildAlertsConfigurationPayload({
maxAlertsHour: normalizedMaxAlertsHour,
escalation: {
enabled: snapshot.scheduleEscalation.enabled,
levels: snapshot.scheduleEscalation.levels.map((level) => ({ ...level })),
repeatCritical: snapshot.scheduleEscalation.repeatCritical,
repeatEvery: snapshot.scheduleEscalation.repeatEvery,
levels: snapshot.scheduleEscalation.levels.map((level) => ({
...level,
...(level.destinationIds ? { destinationIds: [...level.destinationIds] } : {}),
})),
},
grouping: {
enabled: groupingEnabled,
frontend-modern/src/features/alerts/helpers.ts
+2
View File
@@ -184,6 +184,8 @@ export const normalizeMetricDelayMap = (
export const createDefaultEscalation = (): EscalationConfig => ({
enabled: false,
levels: [],
repeatCritical: false,
repeatEvery: 30,
});
export const getTriggerValue = (
frontend-modern/src/features/alerts/tabs/ScheduleTab.tsx
+6
View File
@@ -16,6 +16,7 @@ import { useAlertScheduleState } from '../useAlertScheduleState';
import type {
CooldownConfig,
EscalationConfig,
EscalationDestination,
GroupingConfig,
NotificationDeliveryTarget,
QuietHoursConfig,
@@ -35,6 +36,7 @@ export interface ScheduleTabProps {
setNotifyOnResolve: (value: boolean) => void;
escalation: () => EscalationConfig;
setEscalation: (value: EscalationConfig) => void;
escalationDestinations: () => EscalationDestination[];
}
export function ScheduleTab(props: ScheduleTabProps) {
@@ -119,6 +121,10 @@ export function ScheduleTab(props: ScheduleTabProps) {
setEscalationEnabled={scheduleState.setEscalationEnabled}
setEscalationAfter={scheduleState.setEscalationAfter}
setEscalationNotify={scheduleState.setEscalationNotify}
destinations={props.escalationDestinations()}
setEscalationDestinationIds={scheduleState.setEscalationDestinationIds}
setEscalationRepeatCritical={scheduleState.setEscalationRepeatCritical}
setEscalationRepeatEvery={scheduleState.setEscalationRepeatEvery}
removeEscalationLevel={scheduleState.removeEscalationLevel}
addEscalationLevel={scheduleState.addEscalationLevel}
/>
frontend-modern/src/features/alerts/types.ts
+11
View File
@@ -202,15 +202,26 @@ export interface GroupingConfig {
export type NotificationDeliveryTarget = 'email' | 'webhook' | 'apprise' | 'all';
export type EscalationNotifyTarget = NotificationDeliveryTarget;
export type EscalationDestinationKind = Exclude<NotificationDeliveryTarget, 'all'>;
export interface EscalationDestination {
id: string;
label: string;
kind: EscalationDestinationKind;
enabled: boolean;
}
export interface EscalationLevel {
after: number;
notify: EscalationNotifyTarget;
destinationIds?: string[];
}
export interface EscalationConfig {
enabled: boolean;
levels: EscalationLevel[];
repeatCritical: boolean;
repeatEvery: number;
}
const COOLDOWN_MIN_MINUTES = 5;
frontend-modern/src/features/alerts/useAlertDestinationsState.ts
+16 -1
View File
@@ -3,6 +3,7 @@ import type { Accessor } from 'solid-js';
import { AlertsAPI } from '@/api/alerts';
import { NotificationsAPI } from '@/api/notifications';
import type { Webhook } from '@/api/notifications';
import { RelayAPI } from '@/api/relay';
import { hasFeature } from '@/stores/license';
import { getAlertDestinationsConfigLoadError } from '@/utils/alertDestinationsPresentation';
@@ -30,6 +31,7 @@ export function useAlertDestinationsState(options: AlertDestinationsStateOptions
);
const [deadManPingUrl, setDeadManPingUrl] = createSignal('');
const [pushMinimumSeverity, setPushMinimumSeverity] = createSignal<'all' | 'critical'>('all');
const [webhooks, setWebhooks] = createSignal<Webhook[]>([]);
let reloadVersion = 0;
let lastActiveTab: AlertTab | null = null;
@@ -40,6 +42,7 @@ export function useAlertDestinationsState(options: AlertDestinationsStateOptions
setAppriseConfig(createDefaultAppriseConfig());
setDeadManPingUrl('');
setPushMinimumSeverity('all');
setWebhooks([]);
};
const loadDestinations = async (options: { indicateLoading?: boolean } = {}) => {
@@ -55,13 +58,14 @@ export function useAlertDestinationsState(options: AlertDestinationsStateOptions
NotificationsAPI.getAppriseConfig(),
AlertsAPI.getDeadManConfig(),
hasFeature('relay') ? RelayAPI.getConfig() : Promise.resolve(null),
NotificationsAPI.getWebhooks(),
]);
if (thisVersion !== reloadVersion) {
return;
}
const [emailResult, appriseResult, deadManResult, relayResult] = results;
const [emailResult, appriseResult, deadManResult, relayResult, webhooksResult] = results;
if (emailResult.status === 'fulfilled') {
setEmailConfig(normalizeEmailConfigFromAPI(emailResult.value));
@@ -81,6 +85,15 @@ export function useAlertDestinationsState(options: AlertDestinationsStateOptions
);
}
if (webhooksResult.status === 'fulfilled') {
setWebhooks(
webhooksResult.value.map((webhook) => ({
...webhook,
service: webhook.service || 'generic',
})),
);
}
const failures = results.filter(
(result): result is PromiseRejectedResult => result.status === 'rejected',
);
@@ -138,6 +151,8 @@ export function useAlertDestinationsState(options: AlertDestinationsStateOptions
setDeadManPingUrl,
pushMinimumSeverity,
setPushMinimumSeverity,
webhooks,
setWebhooks,
resetDestinations,
loadDestinations,
saveDestinations,
frontend-modern/src/features/alerts/useAlertDestinationsTabState.ts
+12 -3
View File
@@ -1,6 +1,7 @@
import { createMemo, createSignal, onMount, type Accessor } from 'solid-js';
import { NotificationsAPI, type AppriseConfig } from '@/api/notifications';
import { NotificationsAPI, type AppriseConfig, type Webhook } from '@/api/notifications';
import type { Setter } from 'solid-js';
import { notificationStore } from '@/stores/notifications';
import { logger } from '@/utils/logger';
import { showErrorWithDetail } from '@/utils/toast';
@@ -29,6 +30,8 @@ export interface AlertDestinationsTabStateProps {
isRetrying: Accessor<boolean>;
isLoadingDestinations: Accessor<boolean>;
onRetryLoad: () => void;
webhooks?: Accessor<Webhook[]>;
setWebhooks?: Setter<Webhook[]>;
}
export function useAlertDestinationsTabState(props: AlertDestinationsTabStateProps) {
@@ -50,7 +53,11 @@ export function useAlertDestinationsTabState(props: AlertDestinationsTabStatePro
heldEvents,
loadDeliveryLog,
} = useNotificationDeliveryLog();
const webhookState = useAlertWebhookDestinationsState();
const webhookState = useAlertWebhookDestinationsState({
webhooks: props.webhooks,
setWebhooks: props.setWebhooks,
autoLoad: props.webhooks === undefined,
});
const isLoading = createMemo(
() => props.isLoadingDestinations() || webhookState.isLoadingWebhooks() || props.isRetrying(),
@@ -142,7 +149,9 @@ export function useAlertDestinationsTabState(props: AlertDestinationsTabStatePro
const handleRetry = () => {
props.onRetryLoad();
void webhookState.loadWebhooks();
if (props.webhooks === undefined) {
void webhookState.loadWebhooks();
}
void loadDeliveryHealth();
void loadDeliveryLog();
};
frontend-modern/src/features/alerts/useAlertScheduleState.ts
+45 -1
View File
@@ -11,6 +11,7 @@ import {
import type {
CooldownConfig,
EscalationConfig,
EscalationDestination,
EscalationLevel,
EscalationNotifyTarget,
GroupingConfig,
@@ -33,6 +34,7 @@ export interface UseAlertScheduleStateProps {
setNotifyOnResolve: (value: boolean) => void;
escalation: () => EscalationConfig;
setEscalation: (value: EscalationConfig) => void;
escalationDestinations: () => EscalationDestination[];
}
export const ALERT_SCHEDULE_TIMEZONES = [
@@ -328,6 +330,40 @@ export function useAlertScheduleState(props: UseAlertScheduleStateProps) {
markUnsaved();
};
const setEscalationDestinationIds = (index: number, destinationIds: string[]) => {
if (destinationIds.length === 0) return;
const destinations = props.escalationDestinations();
const selected = destinations.filter((destination) => destinationIds.includes(destination.id));
const allSelected = destinations.length > 0 && selected.length === destinations.length;
const fallbackNotify: EscalationNotifyTarget = allSelected
? 'all'
: (selected[0]?.kind ?? 'all');
const nextLevels = [...props.escalation().levels];
nextLevels[index] = {
...nextLevels[index],
notify: fallbackNotify,
destinationIds: [...new Set(destinationIds)],
};
props.setEscalation({ ...props.escalation(), levels: nextLevels });
markUnsaved();
};
const setEscalationRepeatCritical = (repeatCritical: boolean) => {
props.setEscalation({ ...props.escalation(), repeatCritical });
markUnsaved();
};
const setEscalationRepeatEvery = (rawValue: string) => {
const parsed = Number.parseInt(rawValue, 10);
props.setEscalation({
...props.escalation(),
repeatEvery: Number.isNaN(parsed)
? props.escalation().repeatEvery
: Math.min(180, Math.max(5, parsed)),
});
markUnsaved();
};
const removeEscalationLevel = (index: number) => {
const nextLevels = props
.escalation()
@@ -342,11 +378,16 @@ export function useAlertScheduleState(props: UseAlertScheduleStateProps) {
const addEscalationLevel = () => {
const lastLevel = props.escalation().levels[props.escalation().levels.length - 1];
const nextAfter = typeof lastLevel?.after === 'number' ? lastLevel.after + 30 : 15;
const destinationIds = props.escalationDestinations().map((destination) => destination.id);
props.setEscalation({
...props.escalation(),
levels: [
...props.escalation().levels,
{ after: nextAfter, notify: 'all' as EscalationNotifyTarget },
{
after: nextAfter,
notify: 'all' as EscalationNotifyTarget,
...(destinationIds.length > 0 ? { destinationIds } : {}),
},
],
});
markUnsaved();
@@ -374,6 +415,9 @@ export function useAlertScheduleState(props: UseAlertScheduleStateProps) {
setEscalationEnabled,
setEscalationAfter,
setEscalationNotify,
setEscalationDestinationIds,
setEscalationRepeatCritical,
setEscalationRepeatEvery,
removeEscalationLevel,
addEscalationLevel,
};
frontend-modern/src/features/alerts/useAlertWebhookDestinationsState.ts
+17 -5
View File
@@ -1,4 +1,4 @@
import { createSignal, onMount } from 'solid-js';
import { createSignal, onMount, type Accessor, type Setter } from 'solid-js';
import { NotificationsAPI, type Webhook } from '@/api/notifications';
import { notificationStore } from '@/stores/notifications';
@@ -20,10 +20,20 @@ const normalizeWebhook = (webhook: Webhook): Webhook => ({
service: webhook.service || 'generic',
});
export function useAlertWebhookDestinationsState() {
const [webhooks, setWebhooks] = createSignal<Webhook[]>([]);
interface AlertWebhookDestinationsStateOptions {
webhooks?: Accessor<Webhook[]>;
setWebhooks?: Setter<Webhook[]>;
autoLoad?: boolean;
}
export function useAlertWebhookDestinationsState(
options: AlertWebhookDestinationsStateOptions = {},
) {
const [localWebhooks, setLocalWebhooks] = createSignal<Webhook[]>([]);
const webhooks = options.webhooks ?? localWebhooks;
const setWebhooks = options.setWebhooks ?? setLocalWebhooks;
const [webhookLoadError, setWebhookLoadError] = createSignal<string | null>(null);
const [isLoadingWebhooks, setIsLoadingWebhooks] = createSignal(true);
const [isLoadingWebhooks, setIsLoadingWebhooks] = createSignal(options.autoLoad !== false);
const [testingWebhook, setTestingWebhook] = createSignal<string | null>(null);
const loadWebhooks = async () => {
@@ -41,7 +51,9 @@ export function useAlertWebhookDestinationsState() {
};
onMount(() => {
void loadWebhooks();
if (options.autoLoad !== false) {
void loadWebhooks();
}
});
const testWebhook = async (webhookId: string, webhookData?: Omit<Webhook, 'id'>) => {
frontend-modern/src/features/alerts/useAlertsConfigurationSnapshotState.ts
+12 -2
View File
@@ -176,7 +176,12 @@ export function useAlertsConfigurationSnapshotState(
setScheduleGrouping({ ...snapshot.scheduleGrouping });
setScheduleEscalation({
enabled: snapshot.scheduleEscalation.enabled,
levels: snapshot.scheduleEscalation.levels.map((level) => ({ ...level })),
repeatCritical: snapshot.scheduleEscalation.repeatCritical,
repeatEvery: snapshot.scheduleEscalation.repeatEvery,
levels: snapshot.scheduleEscalation.levels.map((level) => ({
...level,
...(level.destinationIds ? { destinationIds: [...level.destinationIds] } : {}),
})),
});
setInitialNotify(snapshot.initialNotify);
setNotifyOnResolve(snapshot.notifyOnResolve);
@@ -239,7 +244,12 @@ export function useAlertsConfigurationSnapshotState(
scheduleGrouping: { ...scheduleGrouping() },
scheduleEscalation: {
enabled: scheduleEscalation().enabled,
levels: scheduleEscalation().levels.map((level) => ({ ...level })),
repeatCritical: scheduleEscalation().repeatCritical,
repeatEvery: scheduleEscalation().repeatEvery,
levels: scheduleEscalation().levels.map((level) => ({
...level,
...(level.destinationIds ? { destinationIds: [...level.destinationIds] } : {}),
})),
},
initialNotify: initialNotify(),
notifyOnResolve: notifyOnResolve(),
frontend-modern/src/features/alerts/useAlertsConfigurationState.ts
+2
View File
@@ -141,6 +141,8 @@ export function useAlertsConfigurationState(props: AlertsConfigurationSurfacePro
setDeadManPingUrl: destinationsState.setDeadManPingUrl,
pushMinimumSeverity: destinationsState.pushMinimumSeverity,
setPushMinimumSeverity: destinationsState.setPushMinimumSeverity,
webhooks: destinationsState.webhooks,
setWebhooks: destinationsState.setWebhooks,
...configurationSnapshotState,
allGuests: overridesState.allGuests,
agentResources: overridesState.agentResources,
frontend-modern/src/types/alerts.ts
+3 -1
View File
@@ -238,7 +238,9 @@ export interface AlertConfig {
};
escalation?: {
enabled: boolean;
levels?: Array<{ after: number; notify: string }>;
levels?: Array<{ after: number; notify: string; destinationIds?: string[] }>;
repeatCritical?: boolean;
repeatEvery?: number;
};
};
disableAllNodes?: boolean;
frontend-modern/src/utils/__tests__/alertConfigPresentation.test.ts
+16
View File
@@ -13,10 +13,16 @@ import {
ALERT_CONFIG_DELIVERY_TARGET_LABEL,
ALERT_CONFIG_DELIVERY_TITLE,
ALERT_CONFIG_ESCALATION_DESCRIPTION,
ALERT_CONFIG_ESCALATION_DESTINATIONS_HELP,
ALERT_CONFIG_ESCALATION_DESTINATION_DISABLED,
ALERT_CONFIG_ESCALATION_DESTINATION_UNAVAILABLE,
ALERT_CONFIG_ESCALATION_NOTIFY_ALL,
ALERT_CONFIG_ESCALATION_NOTIFY_APPRISE,
ALERT_CONFIG_ESCALATION_NOTIFY_EMAIL,
ALERT_CONFIG_ESCALATION_NOTIFY_WEBHOOKS,
ALERT_CONFIG_ESCALATION_REPEAT_EVERY_LABEL,
ALERT_CONFIG_ESCALATION_REPEAT_HELP,
ALERT_CONFIG_ESCALATION_REPEAT_LABEL,
ALERT_CONFIG_ESCALATION_TITLE,
ALERT_CONFIG_GROUPING_BY_GUEST,
ALERT_CONFIG_GROUPING_BY_NODE,
@@ -152,6 +158,16 @@ describe('alertConfigPresentation', () => {
expect(ALERT_CONFIG_ESCALATION_NOTIFY_WEBHOOKS).toBe('Webhooks');
expect(ALERT_CONFIG_ESCALATION_NOTIFY_APPRISE).toBe('Apprise');
expect(ALERT_CONFIG_ESCALATION_NOTIFY_ALL).toBe('All channels');
expect(ALERT_CONFIG_ESCALATION_DESTINATIONS_HELP).toContain('exact destinations');
expect(ALERT_CONFIG_ESCALATION_DESTINATION_DISABLED).toBe('disabled');
expect(ALERT_CONFIG_ESCALATION_DESTINATION_UNAVAILABLE).toBe('Unavailable destination');
expect(ALERT_CONFIG_ESCALATION_REPEAT_LABEL).toBe(
'Repeat critical escalation until acknowledged',
);
expect(ALERT_CONFIG_ESCALATION_REPEAT_EVERY_LABEL).toBe('Repeat every');
expect(ALERT_CONFIG_ESCALATION_REPEAT_HELP).toContain(
'acknowledged, resolved, snoozed, or notifications are paused',
);
expect(ALERT_CONFIG_SUMMARY_TITLE).toBe('Configuration summary');
expect(ALERT_CONFIG_SUMMARY_DESCRIPTION).toBe('Preview of the active schedule settings.');
expect(ALERT_CONFIG_SUMMARY_ALL_DISABLED).toBe(
frontend-modern/src/utils/alertConfigPresentation.ts
+8
View File
@@ -72,6 +72,14 @@ export const ALERT_CONFIG_RECOVERY_HELP =
export const ALERT_CONFIG_ESCALATION_HELP = 'Define escalation levels for unresolved alerts:';
export const ALERT_CONFIG_ESCALATION_AFTER_LABEL = 'After';
export const ALERT_CONFIG_ESCALATION_NOTIFY_LABEL = 'Notify';
export const ALERT_CONFIG_ESCALATION_DESTINATIONS_HELP =
'Choose the exact destinations for this level. Disabled destinations remain selected but will not receive alerts until they are enabled.';
export const ALERT_CONFIG_ESCALATION_DESTINATION_DISABLED = 'disabled';
export const ALERT_CONFIG_ESCALATION_DESTINATION_UNAVAILABLE = 'Unavailable destination';
export const ALERT_CONFIG_ESCALATION_REPEAT_LABEL = 'Repeat critical escalation until acknowledged';
export const ALERT_CONFIG_ESCALATION_REPEAT_HELP =
'After the final level, repeat critical alerts at this interval. Repeats stop when the alert is acknowledged, resolved, snoozed, or notifications are paused.';
export const ALERT_CONFIG_ESCALATION_REPEAT_EVERY_LABEL = 'Repeat every';
export const ALERT_CONFIG_ESCALATION_MINUTES_SUFFIX = 'min';
export const ALERT_CONFIG_ESCALATION_NOTIFY_EMAIL = 'Email';
export const ALERT_CONFIG_ESCALATION_NOTIFY_WEBHOOKS = 'Webhooks';
internal/alerts/callbacks.go
+15 -2
View File
@@ -487,8 +487,21 @@ func (m *Manager) safeCallUnacknowledgedCallback(alert *Alert, user string) {
// safeCallEscalateCallback invokes onEscalate with panic recovery and alert cloning.
func (m *Manager) safeCallEscalateCallback(alert *Alert, level int) {
m.recordAlertEvent(eventlog.TypeEscalated, alert, "", "", "Alert escalated.",
map[string]string{"level": strconv.Itoa(level)})
m.safeCallEscalateCallbackWithKind(alert, level, false)
}
func (m *Manager) safeCallEscalationRepeatCallback(alert *Alert, level int) {
m.safeCallEscalateCallbackWithKind(alert, level, true)
}
func (m *Manager) safeCallEscalateCallbackWithKind(alert *Alert, level int, repeat bool) {
details := map[string]string{"level": strconv.Itoa(level)}
message := "Alert escalated."
if repeat {
details["repeat"] = "true"
message = "Critical alert escalation repeated."
}
m.recordAlertEvent(eventlog.TypeEscalated, alert, "", "", message, details)
callback := m.getEscalateCallback()
if callback == nil {
internal/alerts/config/notification_delivery_target_test.go
+35 -1
View File
@@ -1,6 +1,9 @@
package config
import "testing"
import (
"reflect"
"testing"
)
func TestNormalizeNotificationDeliveryTarget(t *testing.T) {
tests := map[string]string{
@@ -21,3 +24,34 @@ func TestNormalizeNotificationDeliveryTarget(t *testing.T) {
})
}
}
func TestNormalizeEscalationConfigPreservesLegacyAndBoundsExactRouting(t *testing.T) {
config := EscalationConfig{
Enabled: true,
RepeatCritical: true,
RepeatEvery: 1,
Levels: []EscalationLevel{{
After: 15,
Notify: " WEBHOOKS ",
DestinationIDs: []string{" webhook:ops ", "email", "webhook:ops", "https://secret.example", "webhook:in\nvalid"},
}},
}
NormalizeEscalationConfig(&config)
if config.Levels[0].Notify != "webhook" {
t.Fatalf("legacy fallback = %q, want webhook", config.Levels[0].Notify)
}
if got, want := config.Levels[0].DestinationIDs, []string{"webhook:ops", "email"}; !reflect.DeepEqual(got, want) {
t.Fatalf("destination IDs = %v, want %v", got, want)
}
if config.RepeatEvery != DefaultEscalationRepeatMinutes {
t.Fatalf("repeat interval = %d, want %d", config.RepeatEvery, DefaultEscalationRepeatMinutes)
}
legacy := EscalationConfig{Enabled: true, Levels: []EscalationLevel{{After: 30, Notify: "apprise"}}}
NormalizeEscalationConfig(&legacy)
if legacy.RepeatCritical || legacy.RepeatEvery != 0 || legacy.Levels[0].DestinationIDs != nil {
t.Fatalf("legacy config gained new behavior: %+v", legacy)
}
}
internal/alerts/config/types.go
+71 -4
View File
@@ -87,14 +87,17 @@ type QuietHoursSuppression struct {
// EscalationLevel represents an escalation rule
type EscalationLevel struct {
After int `json:"after"` // minutes after initial alert
Notify string `json:"notify"` // "email", "webhook", "apprise", or "all"
After int `json:"after"` // minutes after initial alert
Notify string `json:"notify"` // backward-compatible channel fallback
DestinationIDs []string `json:"destinationIds,omitempty"` // exact logical destinations: email, apprise, webhook:<id>
}
// EscalationConfig represents alert escalation configuration
type EscalationConfig struct {
Enabled bool `json:"enabled"`
Levels []EscalationLevel `json:"levels"`
Enabled bool `json:"enabled"`
Levels []EscalationLevel `json:"levels"`
RepeatCritical bool `json:"repeatCritical,omitempty"`
RepeatEvery int `json:"repeatEvery,omitempty"` // minutes after the final level
}
// GroupingConfig represents alert grouping configuration
@@ -131,6 +134,70 @@ func NormalizeNotificationDeliveryTarget(target string) string {
}
}
const (
DefaultEscalationRepeatMinutes = 30
MinEscalationRepeatMinutes = 5
MaxEscalationRepeatMinutes = 180
maxEscalationDestinations = 32
)
// NormalizeEscalationDestinationIDs keeps exact escalation routing bounded and
// credential-free. Unknown identifiers are dropped rather than widening to all
// destinations; an empty result preserves the legacy channel fallback.
func NormalizeEscalationDestinationIDs(destinationIDs []string) []string {
if len(destinationIDs) == 0 {
return nil
}
result := make([]string, 0, min(len(destinationIDs), maxEscalationDestinations))
seen := make(map[string]struct{}, len(destinationIDs))
for _, raw := range destinationIDs {
id := strings.TrimSpace(raw)
valid := id == "email" || id == "apprise"
if strings.HasPrefix(id, "webhook:") {
webhookID := strings.TrimSpace(strings.TrimPrefix(id, "webhook:"))
valid = webhookID != "" && len(webhookID) <= 128 && !strings.ContainsAny(webhookID, "\r\n\x00")
if valid {
id = "webhook:" + webhookID
}
}
if !valid {
continue
}
if _, exists := seen[id]; exists {
continue
}
seen[id] = struct{}{}
result = append(result, id)
if len(result) == maxEscalationDestinations {
break
}
}
if len(result) == 0 {
return nil
}
return result
}
// NormalizeEscalationConfig applies the schedule safety bounds without
// inventing repeat behavior for configurations written by older releases.
func NormalizeEscalationConfig(config *EscalationConfig) {
if config == nil {
return
}
for index := range config.Levels {
config.Levels[index].Notify = NormalizeNotificationDeliveryTarget(config.Levels[index].Notify)
config.Levels[index].DestinationIDs = NormalizeEscalationDestinationIDs(config.Levels[index].DestinationIDs)
}
if !config.RepeatCritical || len(config.Levels) == 0 {
config.RepeatCritical = false
config.RepeatEvery = 0
return
}
if config.RepeatEvery < MinEscalationRepeatMinutes || config.RepeatEvery > MaxEscalationRepeatMinutes {
config.RepeatEvery = DefaultEscalationRepeatMinutes
}
}
// FilterCondition represents a single filter condition
type FilterCondition struct {
Type string `json:"type"` // "metric", "text", or "raw"
internal/alerts/config_facade.go
+7
View File
@@ -42,6 +42,9 @@ const (
AlertIntentSignalDefault = alertconfig.AlertIntentSignalDefault
AlertIntentSignalOffline = alertconfig.AlertIntentSignalOffline
AlertIntentSignalAvailability = alertconfig.AlertIntentSignalAvailability
DefaultEscalationRepeatMinutes = alertconfig.DefaultEscalationRepeatMinutes
MinEscalationRepeatMinutes = alertconfig.MinEscalationRepeatMinutes
MaxEscalationRepeatMinutes = alertconfig.MaxEscalationRepeatMinutes
)
func NewAlertIntentPolicyDocument() AlertIntentPolicyDocument {
@@ -78,6 +81,10 @@ func NormalizeDockerIgnoredPrefixes(prefixes []string) []string {
return alertconfig.NormalizeDockerIgnoredPrefixes(prefixes)
}
func NormalizeEscalationDestinationIDs(destinationIDs []string) []string {
return alertconfig.NormalizeEscalationDestinationIDs(destinationIDs)
}
func CanonicalResourceTypeKeys(resourceType string) []string {
return alertconfig.CanonicalResourceTypeKeys(resourceType)
}
internal/alerts/config_runtime.go
+1 -5
View File
@@ -52,11 +52,7 @@ func (m *Manager) UpdateConfig(config AlertConfig) {
config.NodeDefaults.PoweredOffSeverity = alertconfig.NormalizePoweredOffSeverity(config.NodeDefaults.PoweredOffSeverity)
config.DockerIgnoredContainerPrefixes = alertconfig.NormalizeDockerIgnoredPrefixes(config.DockerIgnoredContainerPrefixes)
config.Schedule.InitialNotify = alertconfig.NormalizeNotificationDeliveryTarget(config.Schedule.InitialNotify)
for index := range config.Schedule.Escalation.Levels {
config.Schedule.Escalation.Levels[index].Notify = alertconfig.NormalizeNotificationDeliveryTarget(
config.Schedule.Escalation.Levels[index].Notify,
)
}
alertconfig.NormalizeEscalationConfig(&config.Schedule.Escalation)
// Migration logic for activation state (backward compatibility)
m.migrateActivationState(&config)
internal/alerts/config_validation_test.go
+34
View File
@@ -82,3 +82,37 @@ func TestUpdateConfigNormalizesFlappingSettings(t *testing.T) {
t.Fatalf("expected FlappingCooldownMinutes to default to 15, got %d", m.config.FlappingCooldownMinutes)
}
}
func TestUpdateConfigNormalizesExactEscalationRoutingAndRepeatPolicy(t *testing.T) {
m := newTestManager(t)
cfg := m.GetConfig()
cfg.Schedule.Escalation = EscalationConfig{
Enabled: true,
RepeatCritical: true,
RepeatEvery: 1,
Levels: []EscalationLevel{{
After: 15,
Notify: "WEBHOOKS",
DestinationIDs: []string{" webhook:pager ", "email", "webhook:pager", "https://secret.example"},
}},
}
m.UpdateConfig(cfg)
got := m.GetConfig().Schedule.Escalation
if got.RepeatEvery != DefaultEscalationRepeatMinutes {
t.Fatalf("repeat interval = %d, want %d", got.RepeatEvery, DefaultEscalationRepeatMinutes)
}
if got.Levels[0].Notify != "webhook" {
t.Fatalf("legacy target = %q, want webhook", got.Levels[0].Notify)
}
wantIDs := []string{"webhook:pager", "email"}
if len(got.Levels[0].DestinationIDs) != len(wantIDs) {
t.Fatalf("destination IDs = %v, want %v", got.Levels[0].DestinationIDs, wantIDs)
}
for index, want := range wantIDs {
if got.Levels[0].DestinationIDs[index] != want {
t.Fatalf("destination IDs = %v, want %v", got.Levels[0].DestinationIDs, wantIDs)
}
}
}
internal/alerts/escalation.go
+26
View File
@@ -86,5 +86,31 @@ func (m *Manager) checkEscalations() {
m.safeCallEscalateCallback(alert, i+1)
}
}
escalation := m.config.Schedule.Escalation
if !escalation.RepeatCritical || alert.Level != AlertLevelCritical || len(escalation.Levels) == 0 || alert.LastEscalation < len(escalation.Levels) {
continue
}
repeatEvery := time.Duration(escalation.RepeatEvery) * time.Minute
if repeatEvery <= 0 {
continue
}
lastLevel := escalation.Levels[len(escalation.Levels)-1]
repeatBaseline := escalationBaseline.Add(time.Duration(lastLevel.After) * time.Minute)
if count := len(alert.EscalationTimes); count > 0 && alert.EscalationTimes[count-1].After(repeatBaseline) {
repeatBaseline = alert.EscalationTimes[count-1]
}
if !now.After(repeatBaseline.Add(repeatEvery)) {
continue
}
alert.EscalationTimes = append(alert.EscalationTimes, now)
log.Info().
Str("alertID", alert.ID).
Str("trackingKey", canonicalTrackingKeyForAlert(alert)).
Int("level", len(escalation.Levels)).
Int("repeatEveryMinutes", escalation.RepeatEvery).
Msg("Critical alert escalation repeated")
m.safeCallEscalationRepeatCallback(alert, len(escalation.Levels))
}
}
internal/alerts/escalation_repeat_test.go
+139
View File
@@ -0,0 +1,139 @@
package alerts
import (
"sync"
"sync/atomic"
"testing"
"time"
"github.com/rcourtman/pulse-go-rewrite/internal/alerts/eventlog"
)
func TestCriticalEscalationRepeatsAtFinalLevelUntilAcknowledged(t *testing.T) {
m := newTestManager(t)
store, err := eventlog.OpenInMemory()
if err != nil {
t.Fatalf("open in-memory event log: %v", err)
}
m.SetEventLog(store)
now := time.Date(2026, time.August, 27, 18, 0, 0, 0, time.UTC)
m.now = func() time.Time { return now }
cfg := m.GetConfig()
cfg.Enabled = true
cfg.ActivationState = ActivationActive
cfg.Schedule.Escalation = EscalationConfig{
Enabled: true,
RepeatCritical: true,
RepeatEvery: 15,
Levels: []EscalationLevel{{
After: 5,
Notify: "webhook",
DestinationIDs: []string{"webhook:pager"},
}},
}
m.UpdateConfig(cfg)
var mu sync.Mutex
callbacks := 0
m.SetEscalateCallback(func(*Alert, int) {
mu.Lock()
callbacks++
mu.Unlock()
})
alert := &Alert{
ID: "critical-repeat",
Level: AlertLevelCritical,
ResourceID: "node-1",
CanonicalSpecID: "connectivity",
StartTime: now.Add(-6 * time.Minute),
LastSeen: now,
}
m.mu.Lock()
m.setActiveAlertNoLock(alert.ID, alert)
m.mu.Unlock()
m.checkEscalations()
waitForEscalationCallbacks(t, &mu, &callbacks, 1)
now = now.Add(14 * time.Minute)
m.checkEscalations()
waitForEscalationCallbacks(t, &mu, &callbacks, 1)
now = now.Add(2 * time.Minute)
m.checkEscalations()
waitForEscalationCallbacks(t, &mu, &callbacks, 2)
m.mu.RLock()
active := testRequireActiveAlert(t, m, "critical-repeat")
if active.LastEscalation != 1 || len(active.EscalationTimes) != 2 {
t.Fatalf("repeat lifecycle = %+v", active)
}
canonicalAlertID := exportedAlertID(active, active.ID)
m.mu.RUnlock()
if err := m.AcknowledgeAlert(canonicalAlertID, "operator"); err != nil {
t.Fatalf("AcknowledgeAlert: %v", err)
}
now = now.Add(20 * time.Minute)
m.checkEscalations()
waitForEscalationCallbacks(t, &mu, &callbacks, 2)
events, err := m.AlertEvents(eventlog.Filter{Types: []string{eventlog.TypeEscalated}, Limit: 10})
if err != nil {
t.Fatalf("list escalation events: %v", err)
}
foundRepeat := false
for _, event := range events {
if event.Details["repeat"] == "true" {
foundRepeat = true
}
}
if !foundRepeat {
t.Fatalf("repeat escalation was not distinguished in event log: %+v", events)
}
}
func TestEscalationRepeatSkipsWarningAlerts(t *testing.T) {
m := newTestManager(t)
now := time.Date(2026, time.August, 27, 18, 0, 0, 0, time.UTC)
m.now = func() time.Time { return now }
m.mu.Lock()
m.config.Enabled = true
m.config.ActivationState = ActivationActive
m.config.Schedule.Escalation = EscalationConfig{
Enabled: true, RepeatCritical: true, RepeatEvery: 5,
Levels: []EscalationLevel{{After: 5, Notify: "email"}},
}
m.activeAlerts["warning"] = &Alert{
ID: "warning", Level: AlertLevelWarning, StartTime: now.Add(-30 * time.Minute),
LastEscalation: 1, EscalationTimes: []time.Time{now.Add(-10 * time.Minute)},
}
m.mu.Unlock()
var callbacks atomic.Int32
m.SetEscalateCallback(func(*Alert, int) { callbacks.Add(1) })
m.checkEscalations()
time.Sleep(20 * time.Millisecond)
if got := callbacks.Load(); got != 0 {
t.Fatalf("warning alert repeated %d times", got)
}
}
func waitForEscalationCallbacks(t *testing.T, mu *sync.Mutex, callbacks *int, want int) {
t.Helper()
deadline := time.Now().Add(2 * time.Second)
for {
mu.Lock()
got := *callbacks
mu.Unlock()
if got == want {
return
}
if got > want || time.Now().After(deadline) {
t.Fatalf("escalation callbacks = %d, want %d", got, want)
}
time.Sleep(5 * time.Millisecond)
}
}
internal/monitoring/monitor_alert_override_migration_test.go
+91
View File
@@ -1,6 +1,8 @@
package monitoring
import (
"net/http"
"net/http/httptest"
"strings"
"testing"
"time"
@@ -8,6 +10,7 @@ import (
"github.com/rcourtman/pulse-go-rewrite/internal/alerts"
"github.com/rcourtman/pulse-go-rewrite/internal/config"
"github.com/rcourtman/pulse-go-rewrite/internal/models"
"github.com/rcourtman/pulse-go-rewrite/internal/notifications"
"github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources"
)
@@ -33,6 +36,94 @@ func trueNASMemoryAlertResource(id, supersededID, hostname string, memory float6
return resource
}
func TestHandleAlertEscalatedPreservesLegacyAppriseAndExactWebhookRouting(t *testing.T) {
t.Run("legacy Apprise", func(t *testing.T) {
t.Setenv("PULSE_DATA_DIR", t.TempDir())
requests := make(chan struct{}, 1)
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
requests <- struct{}{}
w.WriteHeader(http.StatusOK)
}))
defer server.Close()
notifMgr := notifications.NewNotificationManager("https://pulse.local")
defer notifMgr.Stop()
notifMgr.SetGroupingWindow(0)
if err := notifMgr.UpdateAllowedPrivateCIDRs("127.0.0.1/32"); err != nil {
t.Fatalf("UpdateAllowedPrivateCIDRs: %v", err)
}
notifMgr.SetAppriseConfig(notifications.AppriseConfig{
Enabled: true, Mode: notifications.AppriseModeHTTP, ServerURL: server.URL,
})
manager := alerts.NewManager()
cfg := manager.GetConfig()
cfg.Schedule.Escalation.Levels = []alerts.EscalationLevel{{After: 1, Notify: "apprise"}}
cfg.Schedule.QuietHours.Enabled = false
manager.UpdateConfig(cfg)
(&Monitor{notificationMgr: notifMgr, alertManager: manager}).handleAlertEscalated(nil, &alerts.Alert{
ID: "apprise-escalation", Type: "connectivity", Level: alerts.AlertLevelCritical,
ResourceID: "node/pve-1", ResourceName: "pve-1", StartTime: time.Now(),
}, 1)
select {
case <-requests:
case <-time.After(2 * time.Second):
t.Fatal("expected legacy Apprise escalation delivery")
}
})
t.Run("exact webhook", func(t *testing.T) {
t.Setenv("PULSE_DATA_DIR", t.TempDir())
selectedRequests := make(chan struct{}, 1)
selectedServer := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
selectedRequests <- struct{}{}
w.WriteHeader(http.StatusOK)
}))
defer selectedServer.Close()
unselectedRequests := make(chan struct{}, 1)
unselectedServer := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
unselectedRequests <- struct{}{}
w.WriteHeader(http.StatusOK)
}))
defer unselectedServer.Close()
notifMgr := notifications.NewNotificationManager("https://pulse.local")
defer notifMgr.Stop()
notifMgr.SetGroupingWindow(0)
if err := notifMgr.UpdateAllowedPrivateCIDRs("127.0.0.1/32"); err != nil {
t.Fatalf("UpdateAllowedPrivateCIDRs: %v", err)
}
notifMgr.AddWebhook(notifications.WebhookConfig{ID: "ops", Name: "ops", URL: unselectedServer.URL, Enabled: true})
notifMgr.AddWebhook(notifications.WebhookConfig{ID: "pager", Name: "pager", URL: selectedServer.URL, Enabled: true})
manager := alerts.NewManager()
cfg := manager.GetConfig()
cfg.Schedule.Escalation.Levels = []alerts.EscalationLevel{{
After: 1, Notify: "webhook", DestinationIDs: []string{"webhook:pager"},
}}
cfg.Schedule.QuietHours.Enabled = false
manager.UpdateConfig(cfg)
(&Monitor{notificationMgr: notifMgr, alertManager: manager}).handleAlertEscalated(nil, &alerts.Alert{
ID: "exact-webhook-escalation", Type: "connectivity", Level: alerts.AlertLevelCritical,
ResourceID: "node/pve-1", ResourceName: "pve-1", StartTime: time.Now(),
}, 1)
select {
case <-selectedRequests:
case <-time.After(2 * time.Second):
t.Fatal("expected selected webhook escalation delivery")
}
select {
case <-unselectedRequests:
t.Fatal("exact escalation selection widened to an unselected webhook")
case <-time.After(300 * time.Millisecond):
}
})
}
func activeMemoryAlertForResource(manager *alerts.Manager, resourceID string) (alerts.Alert, bool) {
for _, alert := range manager.GetActiveAlerts() {
if alert.ResourceID == resourceID && alert.Type == "memory" {
internal/monitoring/monitor_alerts.go
+6 -1
View File
@@ -295,8 +295,13 @@ func (m *Monitor) handleAlertEscalated(hub *websocket.Hub, alert *alerts.Alert,
if m.notificationMgr != nil {
escalationLevel := config.Schedule.Escalation.Levels[level-1]
if len(escalationLevel.DestinationIDs) > 0 {
m.notificationMgr.SendEscalatedAlertToDestinations(alert, escalationLevel.Notify, escalationLevel.DestinationIDs)
m.broadcastEscalatedAlert(hub, alert)
return
}
switch strings.ToLower(strings.TrimSpace(escalationLevel.Notify)) {
case "", "all", "email", "webhook", "webhooks":
case "", "all", "email", "webhook", "webhooks", "apprise":
m.notificationMgr.SendEscalatedAlert(alert, escalationLevel.Notify)
default:
log.Warn().
internal/notifications/notifications.go
+58 -9
View File
@@ -97,6 +97,7 @@ type alertSendOptions struct {
bypassCooldown bool
immediate bool
target notificationDeliveryTarget
destinationIDs []string
}
// createSecureWebhookClient creates an HTTP client with security controls
@@ -969,7 +970,7 @@ func (n *NotificationManager) SetGroupingConfig(enabled bool, seconds int, byNod
n.mu.Unlock()
for _, alert := range pending {
n.dispatchFiringAlerts(emailConfig, webhooks, appriseConfig, []*alerts.Alert{alert}, initialTarget, queue)
n.dispatchFiringAlerts(emailConfig, webhooks, appriseConfig, []*alerts.Alert{alert}, initialTarget, nil, queue)
}
log.Info().
@@ -1109,10 +1110,18 @@ func (n *NotificationManager) SendAlert(alert *alerts.Alert) {
// cadence is owned by the alert schedule, so delivery cooldown must not suppress
// a level that the alert manager has already decided is due.
func (n *NotificationManager) SendEscalatedAlert(alert *alerts.Alert, notifyTarget string) {
n.SendEscalatedAlertToDestinations(alert, notifyTarget, nil)
}
// SendEscalatedAlertToDestinations sends one due escalation to exact logical
// destinations when provided. Exact selection is intentionally fail-closed:
// stale or unknown IDs never widen into a channel-wide or all-destination send.
func (n *NotificationManager) SendEscalatedAlertToDestinations(alert *alerts.Alert, notifyTarget string, destinationIDs []string) {
n.sendAlert(alert, alertSendOptions{
bypassCooldown: true,
immediate: true,
target: normalizeNotificationDeliveryTarget(notifyTarget),
destinationIDs: normalizeEscalationDestinationIDs(destinationIDs),
})
}
@@ -1172,7 +1181,7 @@ func (n *NotificationManager) sendAlert(alert *alerts.Alert, options alertSendOp
queue := n.queue
n.mu.Unlock()
n.dispatchFiringAlerts(emailConfig, webhooks, appriseConfig, []*alerts.Alert{alert}, options.target, queue)
n.dispatchFiringAlerts(emailConfig, webhooks, appriseConfig, []*alerts.Alert{alert}, options.target, options.destinationIDs, queue)
return
}
@@ -1224,9 +1233,10 @@ func (n *NotificationManager) dispatchFiringAlerts(
appriseConfig AppriseConfig,
alertsToSend []*alerts.Alert,
target notificationDeliveryTarget,
destinationIDs []string,
queue *NotificationQueue,
) {
jobs := buildNotificationDeliveryJobsForTarget(
jobs := buildNotificationDeliveryJobsForSelection(
emailConfig,
webhooks,
appriseConfig,
@@ -1234,9 +1244,12 @@ func (n *NotificationManager) dispatchFiringAlerts(
eventAlert,
time.Time{},
target,
destinationIDs,
)
if len(jobs) == 0 {
n.markAlertsNotified(alertsToSend, time.Now())
if len(destinationIDs) == 0 {
n.markAlertsNotified(alertsToSend, time.Now())
}
return
}
if queue != nil {
@@ -1533,7 +1546,7 @@ func (n *NotificationManager) sendGroupedAlerts() {
queue := n.queue
n.mu.Unlock()
n.dispatchFiringAlerts(emailConfig, webhooks, appriseConfig, alertsToSend, initialTarget, queue)
n.dispatchFiringAlerts(emailConfig, webhooks, appriseConfig, alertsToSend, initialTarget, nil, queue)
}
func buildNotificationDeliveryJobs(emailConfig EmailConfig, webhooks []WebhookConfig, appriseConfig AppriseConfig, alertsToSend []*alerts.Alert, event notificationEvent, resolvedAt time.Time) []notificationDeliveryJob {
@@ -1556,13 +1569,45 @@ func buildNotificationDeliveryJobsForTarget(
event notificationEvent,
resolvedAt time.Time,
target notificationDeliveryTarget,
) []notificationDeliveryJob {
return buildNotificationDeliveryJobsForSelection(
emailConfig,
webhooks,
appriseConfig,
alertsToSend,
event,
resolvedAt,
target,
nil,
)
}
func buildNotificationDeliveryJobsForSelection(
emailConfig EmailConfig,
webhooks []WebhookConfig,
appriseConfig AppriseConfig,
alertsToSend []*alerts.Alert,
event notificationEvent,
resolvedAt time.Time,
target notificationDeliveryTarget,
destinationIDs []string,
) []notificationDeliveryJob {
if target == "" {
target = notificationDeliveryTargetAll
}
destinationIDs = normalizeEscalationDestinationIDs(destinationIDs)
exactSelection := len(destinationIDs) > 0
selected := make(map[string]struct{}, len(destinationIDs))
for _, id := range destinationIDs {
selected[id] = struct{}{}
}
isSelected := func(id string) bool {
_, ok := selected[id]
return ok
}
jobs := make([]notificationDeliveryJob, 0, 2+len(webhooks))
if emailConfig.Enabled && (target == notificationDeliveryTargetAll || target == notificationDeliveryTargetEmail) {
if emailConfig.Enabled && ((exactSelection && isSelected("email")) || (!exactSelection && (target == notificationDeliveryTargetAll || target == notificationDeliveryTargetEmail))) {
routedAlerts := routeNotificationAlerts(alertsToSend, emailConfig.TagFilter, emailConfig.TagMode, emailConfig.MinimumSeverity, event)
if len(routedAlerts) > 0 {
emailCopy := emailConfig
@@ -1575,9 +1620,9 @@ func buildNotificationDeliveryJobsForTarget(
})
}
}
if target == notificationDeliveryTargetAll || target == notificationDeliveryTargetWebhook {
if exactSelection || target == notificationDeliveryTargetAll || target == notificationDeliveryTargetWebhook {
for _, webhook := range webhooks {
if !webhook.Enabled {
if !webhook.Enabled || (exactSelection && !isSelected("webhook:"+strings.TrimSpace(webhook.ID))) {
continue
}
routedAlerts := routeNotificationAlerts(alertsToSend, webhook.TagFilter, webhook.TagMode, webhook.MinimumSeverity, event)
@@ -1593,7 +1638,7 @@ func buildNotificationDeliveryJobsForTarget(
}
}
}
if appriseConfig.Enabled && (target == notificationDeliveryTargetAll || target == notificationDeliveryTargetApprise) {
if appriseConfig.Enabled && ((exactSelection && isSelected("apprise")) || (!exactSelection && (target == notificationDeliveryTargetAll || target == notificationDeliveryTargetApprise))) {
routedAlerts := routeNotificationAlerts(alertsToSend, nil, "", appriseConfig.MinimumSeverity, event)
if len(routedAlerts) > 0 {
appriseCopy := appriseConfig
@@ -1609,6 +1654,10 @@ func buildNotificationDeliveryJobsForTarget(
return jobs
}
func normalizeEscalationDestinationIDs(destinationIDs []string) []string {
return alerts.NormalizeEscalationDestinationIDs(destinationIDs)
}
func queueTypeForNotificationDeliveryJob(job notificationDeliveryJob) string {
queueType := job.Type
if job.Event == eventResolved {
internal/notifications/tag_routing_test.go
+35
View File
@@ -121,6 +121,41 @@ func TestBuildNotificationDeliveryJobsRoutesGroupedAlertsByDestinationSeverity(t
}
}
func TestBuildNotificationDeliveryJobsTargetsExactEscalationDestinations(t *testing.T) {
alert := &alerts.Alert{ID: "critical", Level: alerts.AlertLevelCritical, StartTime: time.Unix(1_700_000_000, 0)}
jobs := buildNotificationDeliveryJobsForSelection(
EmailConfig{Enabled: true},
[]WebhookConfig{
{ID: "ops", Enabled: true},
{ID: "pager", Enabled: true},
},
AppriseConfig{Enabled: true, Mode: AppriseModeHTTP, ServerURL: "https://apprise.example.test"},
[]*alerts.Alert{alert},
eventAlert,
time.Time{},
notificationDeliveryTargetAll,
[]string{"webhook:pager", "apprise"},
)
if len(jobs) != 2 || jobs[0].WebhookConfig == nil || jobs[0].WebhookConfig.ID != "pager" || jobs[1].AppriseConfig == nil {
t.Fatalf("exact escalation jobs = %+v, want pager webhook plus Apprise", jobs)
}
jobs = buildNotificationDeliveryJobsForSelection(
EmailConfig{Enabled: true},
[]WebhookConfig{{ID: "ops", Enabled: true}},
AppriseConfig{Enabled: true},
[]*alerts.Alert{alert},
eventAlert,
time.Time{},
notificationDeliveryTargetAll,
[]string{"webhook:deleted"},
)
if len(jobs) != 0 {
t.Fatalf("unknown exact destination widened delivery: %+v", jobs)
}
}
func TestResolvedSeverityRoutingDefersToOccurrenceReceipts(t *testing.T) {
warning := &alerts.Alert{ID: "warning", Level: alerts.AlertLevelWarning}
routed := routeNotificationAlerts(