Clarify combined destination severity and tag routing

Correct the empty-filter delivery claim and explain non-exclusive minimum severity, exact all/any tag matching and per-member grouped routing. Bind the existing help and copied examples to the real saved settings and ordinary delivery job builder, without changing routing behaviour.

Contract-Neutral: Documentation and verification only; existing notification routing, alert policy, runtime and API contracts are unchanged.
Change-source: pulse-maintainer
This commit is contained in:
pulse-triage[bot]
2026-10-07 06:55:39 +01:00
parent 96be0b33a4
commit 53e1bff13c
8 changed files with 335 additions and 6 deletions
+51
View File
@@ -549,6 +549,57 @@ This tolerance delays activation; it does not disable powered-off monitoring.
Use the existing guest offline-alert toggle when a guest should never produce
powered-off alerts.
### Destination severity and tag routing
Open **Alerts → Notifications** and edit the email or webhook destination.
For ordinary firing alerts, an enabled destination must match **both** its
**Minimum alert severity** and **Resource tag routing**. Matching one does not
bypass the other, or the separate notification schedules and holds.
| Minimum alert severity | Eligible firing levels |
| --- | --- |
| **All alerts** | Informational, warning and critical alerts; other policies still apply. |
| **Warnings and critical alerts** | Warning and critical alerts, not informational alerts. |
| **Critical alerts only** | Critical alerts, not warning or informational alerts. |
These are minimums, not exclusive channels. A critical alert can match both a
**Warnings and critical alerts** destination and a **Critical alerts only**
destination. There is no warning-only severity setting; tag filters select
resources, not a severity exclusion. Each destination is evaluated independently,
so one alert can go to more than one destination. Grouped alerts are filtered
member by member: a critical member does not make the group's warning members
eligible for a critical-only destination.
An empty tag filter removes the **resource-tag restriction only**. It does not
override minimum severity, paused delivery, a disabled destination or other
notification policies. With tags selected:
- **Match all tags** requires every selected tag on the same alert's routing
metadata. **Match any tag** requires at least one. An alert without usable
routing tags does not match a nonempty filter.
- Matching is exact, ignoring case and surrounding whitespace; it does not
interpret wildcards, prefixes or regular expressions. A tag named `critical`
is a resource tag, not an alert-severity rule.
- Proxmox resource tags use their tag text. Docker container and service labels
become `key:value` tags, or `key` when the value is empty: the label
`env=prod` is matched by `env:prod`, not just `prod`.
For example, select `env:prod` and `team:ops`, **Match all tags**, and
**Warnings and critical alerts**. A warning or critical alert carrying both
tags is eligible for that destination. An informational alert with both tags
is not. A critical alert carrying only `env:prod` also fails this filter;
critical severity does not bypass tag routing. Choosing **Match any tag** would
allow either tag, but would still exclude informational alerts.
A successful **Test** does not exercise ordinary alert routing. Check an
already occurring alert and [Recent delivery activity](TROUBLESHOOTING.md#test-succeeds-but-real-alerts-are-missing)
instead of creating an outage, removing filters or changing workload labels
just to test delivery. Missing delivery is not evidence that the workload is
healthy. Recovery uses the successful firing receipt for the same occurrence
and destination, rather than reapplying changed tags or minimum severity;
the separate [recovery controls and holds](#alert-reminders-and-recovery-notifications)
still apply. Retained queued work also [keeps its saved destination settings](TROUBLESHOOTING.md#recover-retained-delivery-failures).
### Alert reminders and recovery notifications
Open **Alerts → Schedule**. These controls govern notification eligibility, not
+4 -2
View File
@@ -736,8 +736,10 @@ alert was generated, routed or delivered to the intended recipient.
- Open **Alerts → Notifications**. If **Notifications are paused** is shown,
configured destinations and a successful test do not enable real delivery.
Turn delivery on there only when you intend to send alerts.
- Check the affected alert, the destination's **Enabled** state, minimum alert
severity and tag filters. Review the [quiet-hours schedule](CONFIGURATION.md#quiet-hours-and-notification-holds)
- Check the affected alert, the destination's **Enabled** state and
[minimum severity and tag filters](CONFIGURATION.md#destination-severity-and-tag-routing).
Both must match: an empty tag filter does not bypass minimum severity, and
critical severity does not bypass a nonempty tag filter. Review the [quiet-hours schedule](CONFIGURATION.md#quiet-hours-and-notification-holds)
and [acknowledgement or snooze](CONFIGURATION.md#acknowledge-and-snooze-existing-alerts),
plus any mute or maintenance policy, before treating an absent attempt as a
transport failure. In **Alerts → Overview**, use **Show acknowledged** to find
+11 -1
View File
@@ -102,7 +102,17 @@ severity and other delivery policies still apply.
**Tenant identity.** In multi-tenant organizations and MSP client runtimes, `{{.TenantID}}` and `{{.TenantName}}` identify which tenant fired the alert. Client runtimes get identity from the `PULSE_TENANT_ID` / `PULSE_TENANT_NAME` environment; shared-process organizations stamp the org ID and display name automatically.
**Resource tag routing.** Email and each alert webhook can be limited to resources with selected tags in **Alerts → Notifications**. An empty filter receives every alert. With multiple tags, choose **Match all tags** or **Match any tag**. Matching ignores case. Proxmox tags are matched as shown; Docker container and service labels are exposed as `key:value` tags (or `key` when the label value is empty). Recovery notifications follow the destinations that received the firing alert, even if a resource's tags change before recovery.
**Destination routing.** Email and each alert webhook apply minimum severity
and resource tags together in **Alerts → Notifications**. An empty tag filter
removes only the tag restriction, not minimum severity or other delivery
policies. **Warnings and critical alerts** includes critical alerts; it is not
a warning-only channel. Multiple destinations can match the same firing alert,
and groups are filtered member by member. See the
[severity and tag routing guide](CONFIGURATION.md#destination-severity-and-tag-routing)
for exact all/any matching and Docker label syntax. Recovery remains qualified
by a successful firing-delivery receipt for the same occurrence and destination,
even if tags or minimum severity change before recovery; other recovery controls
and holds still apply.
**Retries and retained failures.**
@@ -549,6 +549,57 @@ This tolerance delays activation; it does not disable powered-off monitoring.
Use the existing guest offline-alert toggle when a guest should never produce
powered-off alerts.
### Destination severity and tag routing
Open **Alerts → Notifications** and edit the email or webhook destination.
For ordinary firing alerts, an enabled destination must match **both** its
**Minimum alert severity** and **Resource tag routing**. Matching one does not
bypass the other, or the separate notification schedules and holds.
| Minimum alert severity | Eligible firing levels |
| --- | --- |
| **All alerts** | Informational, warning and critical alerts; other policies still apply. |
| **Warnings and critical alerts** | Warning and critical alerts, not informational alerts. |
| **Critical alerts only** | Critical alerts, not warning or informational alerts. |
These are minimums, not exclusive channels. A critical alert can match both a
**Warnings and critical alerts** destination and a **Critical alerts only**
destination. There is no warning-only severity setting; tag filters select
resources, not a severity exclusion. Each destination is evaluated independently,
so one alert can go to more than one destination. Grouped alerts are filtered
member by member: a critical member does not make the group's warning members
eligible for a critical-only destination.
An empty tag filter removes the **resource-tag restriction only**. It does not
override minimum severity, paused delivery, a disabled destination or other
notification policies. With tags selected:
- **Match all tags** requires every selected tag on the same alert's routing
metadata. **Match any tag** requires at least one. An alert without usable
routing tags does not match a nonempty filter.
- Matching is exact, ignoring case and surrounding whitespace; it does not
interpret wildcards, prefixes or regular expressions. A tag named `critical`
is a resource tag, not an alert-severity rule.
- Proxmox resource tags use their tag text. Docker container and service labels
become `key:value` tags, or `key` when the value is empty: the label
`env=prod` is matched by `env:prod`, not just `prod`.
For example, select `env:prod` and `team:ops`, **Match all tags**, and
**Warnings and critical alerts**. A warning or critical alert carrying both
tags is eligible for that destination. An informational alert with both tags
is not. A critical alert carrying only `env:prod` also fails this filter;
critical severity does not bypass tag routing. Choosing **Match any tag** would
allow either tag, but would still exclude informational alerts.
A successful **Test** does not exercise ordinary alert routing. Check an
already occurring alert and [Recent delivery activity](TROUBLESHOOTING.md#test-succeeds-but-real-alerts-are-missing)
instead of creating an outage, removing filters or changing workload labels
just to test delivery. Missing delivery is not evidence that the workload is
healthy. Recovery uses the successful firing receipt for the same occurrence
and destination, rather than reapplying changed tags or minimum severity;
the separate [recovery controls and holds](#alert-reminders-and-recovery-notifications)
still apply. Retained queued work also [keeps its saved destination settings](TROUBLESHOOTING.md#recover-retained-delivery-failures).
### Alert reminders and recovery notifications
Open **Alerts → Schedule**. These controls govern notification eligibility, not
@@ -736,8 +736,10 @@ alert was generated, routed or delivered to the intended recipient.
- Open **Alerts → Notifications**. If **Notifications are paused** is shown,
configured destinations and a successful test do not enable real delivery.
Turn delivery on there only when you intend to send alerts.
- Check the affected alert, the destination's **Enabled** state, minimum alert
severity and tag filters. Review the [quiet-hours schedule](CONFIGURATION.md#quiet-hours-and-notification-holds)
- Check the affected alert, the destination's **Enabled** state and
[minimum severity and tag filters](CONFIGURATION.md#destination-severity-and-tag-routing).
Both must match: an empty tag filter does not bypass minimum severity, and
critical severity does not bypass a nonempty tag filter. Review the [quiet-hours schedule](CONFIGURATION.md#quiet-hours-and-notification-holds)
and [acknowledgement or snooze](CONFIGURATION.md#acknowledge-and-snooze-existing-alerts),
plus any mute or maintenance policy, before treating an absent attempt as a
transport failure. In **Alerts → Overview**, use **Show acknowledged** to find
+11 -1
View File
@@ -102,7 +102,17 @@ severity and other delivery policies still apply.
**Tenant identity.** In multi-tenant organizations and MSP client runtimes, `{{.TenantID}}` and `{{.TenantName}}` identify which tenant fired the alert. Client runtimes get identity from the `PULSE_TENANT_ID` / `PULSE_TENANT_NAME` environment; shared-process organizations stamp the org ID and display name automatically.
**Resource tag routing.** Email and each alert webhook can be limited to resources with selected tags in **Alerts → Notifications**. An empty filter receives every alert. With multiple tags, choose **Match all tags** or **Match any tag**. Matching ignores case. Proxmox tags are matched as shown; Docker container and service labels are exposed as `key:value` tags (or `key` when the label value is empty). Recovery notifications follow the destinations that received the firing alert, even if a resource's tags change before recovery.
**Destination routing.** Email and each alert webhook apply minimum severity
and resource tags together in **Alerts → Notifications**. An empty tag filter
removes only the tag restriction, not minimum severity or other delivery
policies. **Warnings and critical alerts** includes critical alerts; it is not
a warning-only channel. Multiple destinations can match the same firing alert,
and groups are filtered member by member. See the
[severity and tag routing guide](CONFIGURATION.md#destination-severity-and-tag-routing)
for exact all/any matching and Docker label syntax. Recovery remains qualified
by a successful firing-delivery receipt for the same occurrence and destination,
even if tags or minimum severity change before recovery; other recovery controls
and holds still apply.
**Retries and retained failures.**
@@ -0,0 +1,146 @@
import { readFileSync } from 'node:fs';
import path from 'node:path';
import { fileURLToPath } from 'node:url';
import { describe, expect, it } from 'vitest';
import { renderDocMarkdown } from '@/features/docs/docMarkdown';
import {
buildEmailConfigPayload,
normalizeEmailConfigFromAPI,
} from '@/features/alerts/alertDestinationsModel';
import {
ALERT_DESTINATION_ALL_SEVERITIES_LABEL,
ALERT_DESTINATION_CRITICAL_ONLY_LABEL,
ALERT_DESTINATION_MINIMUM_SEVERITY_LABEL,
ALERT_DESTINATION_WARNING_AND_CRITICAL_LABEL,
} from '@/utils/alertDestinationsPresentation';
const root = path.resolve(path.dirname(fileURLToPath(import.meta.url)), '../../../..');
const read = (name: string): string => readFileSync(path.join(root, name), 'utf8');
function section(name: string, heading: string): HTMLElement {
const source = read(`docs/${name}.md`)
.split(heading)[1]
?.split(/\n#{1,6} /)[0];
expect(source, `missing ${heading}`).toBeDefined();
const element = document.createElement('article');
element.innerHTML = renderDocMarkdown(heading + source!, name);
return element;
}
const routingHelp = () => section('CONFIGURATION', '### Destination severity and tag routing');
const text = (element: Element): string => element.textContent!.replace(/\s+/g, ' ').trim();
describe('existing destination severity and tag routing help', () => {
it('uses the current severity labels and does not promise exclusive channels', () => {
const guide = routingHelp();
const rows = [...guide.querySelectorAll('tbody tr')].map((row) =>
[...row.querySelectorAll('td')].map(text),
);
expect(text(guide)).toContain(ALERT_DESTINATION_MINIMUM_SEVERITY_LABEL);
expect(rows).toEqual([
[
ALERT_DESTINATION_ALL_SEVERITIES_LABEL,
'Informational, warning and critical alerts; other policies still apply.',
],
[
ALERT_DESTINATION_WARNING_AND_CRITICAL_LABEL,
'Warning and critical alerts, not informational alerts.',
],
[
ALERT_DESTINATION_CRITICAL_ONLY_LABEL,
'Critical alerts, not warning or informational alerts.',
],
]);
expect(text(guide)).toContain('minimums, not exclusive channels');
expect(text(guide)).toContain('There is no warning-only severity setting');
expect(text(guide)).toContain('one alert can go to more than one destination');
expect(text(guide)).toContain('Grouped alerts are filtered member by member');
expect(text(guide)).toContain(
"a critical member does not make the group's warning members eligible",
);
});
it('explains combined filters, exact matching and the critical-tag exclusion example', () => {
const guide = text(routingHelp());
expect(guide).toContain('must match both its Minimum alert severity and Resource tag routing');
expect(guide).toContain('An empty tag filter removes the resource-tag restriction only');
expect(guide).toContain('every selected tag on the same alert');
expect(guide).toContain('Match any tag requires at least one');
expect(guide).toContain('without usable routing tags does not match a nonempty filter');
expect(guide).toContain('exact, ignoring case and surrounding whitespace');
expect(guide).toContain('does not interpret wildcards, prefixes or regular expressions');
expect(guide).toContain('critical is a resource tag, not an alert-severity rule');
expect(guide).toContain('env=prod is matched by env:prod, not just prod');
expect(guide).toContain('A warning or critical alert carrying both tags is eligible');
expect(guide).toContain('An informational alert with both tags is not');
expect(guide).toContain('critical severity does not bypass tag routing');
});
it('keeps empty resource filters independent of severity in the actual saved email settings', () => {
const current = normalizeEmailConfigFromAPI({
enabled: true,
tagFilter: ['env:prod', 'team:ops'],
tagFilterMode: 'all',
minimumSeverity: 'critical',
});
current.tagFilter = [];
const saved = buildEmailConfigPayload(current);
expect(saved.tagFilter).toEqual([]);
expect(saved.minimumSeverity).toBe('critical');
const reloaded = normalizeEmailConfigFromAPI(saved);
expect(reloaded.tagFilter).toEqual([]);
expect(reloaded.minimumSeverity).toBe('critical');
expect(text(routingHelp())).toContain('It does not override minimum severity');
});
it('keeps Test, recovery receipts and retained settings separate from firing eligibility', () => {
const guide = text(routingHelp());
expect(guide).toContain('A successful Test does not exercise ordinary alert routing');
expect(guide).toContain('instead of creating an outage, removing filters');
expect(guide).toContain('Missing delivery is not evidence that the workload is healthy');
expect(guide).toContain('successful firing receipt for the same occurrence and destination');
expect(guide).toContain('rather than reapplying changed tags or minimum severity');
expect(guide).toContain('Retained queued work also keeps its saved destination settings');
});
it('ships identical guides with navigable diagnosis, receipt and recovery links', () => {
for (const name of ['CONFIGURATION', 'WEBHOOKS', 'TROUBLESHOOTING']) {
expect(read(`frontend-modern/public/docs/${name}.md`)).toBe(read(`docs/${name}.md`));
}
const diagnosis = section('TROUBLESHOOTING', '#### Test succeeds but real alerts are missing');
const contract = section('WEBHOOKS', '## 📦 Delivery Contract');
const guide = routingHelp();
for (const from of [diagnosis, contract]) {
const link = from.querySelector(
'a[href="/docs/CONFIGURATION#destination-severity-and-tag-routing"]',
);
expect(link?.hasAttribute('data-doc-link')).toBe(true);
}
expect(guide.querySelector('#destination-severity-and-tag-routing')).not.toBeNull();
expect(text(contract)).not.toContain('An empty filter receives every alert');
expect(text(contract)).toContain('removes only the tag restriction, not minimum severity');
expect(text(diagnosis)).toContain('Both must match');
expect(text(diagnosis)).toContain('critical severity does not bypass a nonempty tag filter');
for (const [href, target, heading] of [
[
'#alert-reminders-and-recovery-notifications',
'CONFIGURATION',
'### Alert reminders and recovery notifications',
],
[
'/docs/TROUBLESHOOTING#test-succeeds-but-real-alerts-are-missing',
'TROUBLESHOOTING',
'#### Test succeeds but real alerts are missing',
],
[
'/docs/TROUBLESHOOTING#recover-retained-delivery-failures',
'TROUBLESHOOTING',
'#### Recover retained delivery failures',
],
]) {
expect(guide.querySelector(`a[href="${href}"]`)).not.toBeNull();
expect(section(target, heading).querySelector(href.slice(href.indexOf('#')))).not.toBeNull();
}
});
});
@@ -80,6 +80,63 @@ func TestTagRoutingNormalizesCaseSourcesAndModes(t *testing.T) {
}
}
// Exercise the documented combinations through ordinary grouped job creation,
// not a settings Test or a synthetic reimplementation of the routing predicates.
func TestDestinationRoutingHelpExamples(t *testing.T) {
makeAlert := func(id string, level alerts.AlertLevel, tags []string) *alerts.Alert {
alert := taggedRoutingAlert(id, tags)
alert.Level = level
return alert
}
group := []*alerts.Alert{
makeAlert("info-both", alerts.AlertLevelInfo, []string{"env:prod", "team:ops", "critical"}),
makeAlert("warning-both", alerts.AlertLevelWarning, []string{"env:prod", "team:ops"}),
makeAlert("critical-both", alerts.AlertLevelCritical, []string{"env:prod", "team:ops"}),
makeAlert("critical-env", alerts.AlertLevelCritical, []string{"env:prod"}),
makeAlert("critical-untagged", alerts.AlertLevelCritical, nil),
makeAlert("warning-prefix", alerts.AlertLevelWarning, []string{"env:production", "team:ops-extra"}),
}
for _, tc := range []struct {
name string
filter []string
mode string
severity string
want []string
}{
{"all-warning", []string{"env:prod", "team:ops"}, "all", "warning", []string{"warning-both", "critical-both"}},
{"any-warning", []string{"env:prod", "team:ops"}, "any", "warning", []string{"warning-both", "critical-both", "critical-env"}},
{"all-critical", []string{"env:prod", "team:ops"}, "all", "critical", []string{"critical-both"}},
{"empty-critical", nil, "all", "critical", []string{"critical-both", "critical-env", "critical-untagged"}},
{"critical-tag-is-not-severity", []string{"critical"}, "all", "all", []string{"info-both"}},
{"no-wildcard", []string{"env:*"}, "any", "warning", nil},
} {
t.Run(tc.name, func(t *testing.T) {
jobs := buildNotificationDeliveryJobs(
EmailConfig{Enabled: true, TagFilter: tc.filter, TagMode: tc.mode, MinimumSeverity: tc.severity},
[]WebhookConfig{
{ID: "ops", Enabled: true, TagFilter: tc.filter, TagMode: tc.mode, MinimumSeverity: tc.severity},
{ID: "disabled", Enabled: false},
},
AppriseConfig{}, group, eventAlert, time.Time{},
)
if len(tc.want) == 0 {
if len(jobs) != 0 {
t.Fatalf("unmatched tags produced %d delivery jobs", len(jobs))
}
return
}
if len(jobs) != 2 || jobs[0].EmailConfig == nil || jobs[1].WebhookConfig == nil || jobs[1].WebhookConfig.ID != "ops" {
t.Fatalf("expected independent enabled email and webhook jobs, got %+v", jobs)
}
for _, job := range jobs {
if got := alertIDs(job.Alerts); !reflect.DeepEqual(got, tc.want) {
t.Fatalf("%s firing members = %v, want %v", job.Type, got, tc.want)
}
}
})
}
}
func TestResolvedTagRoutingDefersToDeliveryReceipts(t *testing.T) {
alert := taggedRoutingAlert("changed-tags", []string{"customer:beta"})
routed := routeNotificationAlerts(