diff --git a/docs/WEBHOOKS.md b/docs/WEBHOOKS.md index 3acf6649b4..cb0f8424fd 100644 --- a/docs/WEBHOOKS.md +++ b/docs/WEBHOOKS.md @@ -65,11 +65,18 @@ For generic webhooks, use Go templates to format the JSON payload. These fields and behaviors are stable; ticket-routing integrations can rely on them. -**Events.** Every webhook fires on both `alert` and `resolved` events. `{{.Event}}` is `"alert"` or `"resolved"` — there is no separate "info" event class. +**Events.** `{{.Event}}` is `"alert"` or `"resolved"` — there is no separate +"info" event class. Recovery delivery depends on **Notify on resolve** and a +successful firing-delivery receipt for that occurrence and destination. Do not +assume that every configured webhook receives both events. -**Severity.** `{{.Level}}` is `"warning"` or `"critical"`. Pulse has exactly these two alert levels. +**Severity.** Metric alerts commonly use `"warning"` or `"critical"`; +informational conditions can use `"info"`. Event and severity are separate: +an informational firing still has event `"alert"`. Preserve and flag an +unrecognised level rather than silently dropping the notification. Minimum +severity and other delivery policies still apply. -**Alert type.** `{{.Type}}` is the metric or condition that fired: `cpu`, `memory`, `disk`, `diskRead`, `diskWrite`, `networkIn`, `networkOut`, `connectivity`, and similar. The alert ID (`{{.ID}}`) is stable for the lifetime of an alert occurrence, so the `resolved` event carries the same ID as the `alert` event it closes. +**Alert type.** `{{.Type}}` is the metric or condition that fired: `cpu`, `memory`, `disk`, `diskRead`, `diskWrite`, `networkIn`, `networkOut`, `connectivity`, and similar. The alert ID (`{{.ID}}`) correlates firing and recovery, but can be reused when the same condition fires again. It is not a unique incident or delivery ID. **Message key.** `{{.MessageKey}}` is the stable, language-neutral condition key for rebuilding or translating a notification. Canonical alerts use `.` (for example, `metric-threshold.disk`); older alert paths fall back to `{{.Type}}`. Combine it with `{{.Event}}` and `{{.ResourceType}}` when the receiving system needs separate wording for firing/recovery events or different resource classes. Unlike a numeric message index, the symbolic key does not change when another alert type is added. @@ -79,7 +86,13 @@ These fields and behaviors are stable; ticket-routing integrations can rely on t **Retries.** Failed deliveries retry with exponential backoff. The persistent notification queue makes up to 3 delivery attempts per notification; webhooks configured with transport-level retry add up to 3 more HTTP retries per attempt (1s doubling to a 30s cap, honoring `Retry-After` on HTTP 429). A receiver may therefore see the same logical event more than once. -**Idempotency.** Every alert delivery carries an `X-Pulse-Event-ID` header of the form `:` (e.g. `a1b2c3:alert`, `a1b2c3:resolved`). It is identical across all retries of the same logical event — deduplicate on it. +**Correlation header.** Alert webhooks carry `X-Pulse-Event-ID` in the form +`:` (e.g. `a1b2c3:alert`, `a1b2c3:resolved`). Retries retain it, +but later occurrences, severity changes and reminders can share it too. A group +uses its primary alert's ID, not a unique ID for every member or batch. **Do not +deduplicate permanently on this header or on alert ID and event alone:** doing +so can silently discard a later incident or a changed group. See +[receiver correlation and deduplication](#receiver-correlation-and-deduplication). **Signed deliveries.** Set a `signingSecret` on the webhook config to enable HMAC signing. Signed requests carry: @@ -126,14 +139,12 @@ def verify(secret: str, timestamp: str, body: bytes, signature: str) -> bool: A time window rejects old captures, **not every replay**: the same signed request can arrive more than once within that window, and Pulse signs each retry with its current send time. After verification, deduplicate atomically -before creating a ticket or taking another action. Use a durable key scoped to -the configured Pulse sender and destination, built from the authenticated -payload's alert ID and event (`{{.ID}}` and `{{.Event}}` in your template); keep -it across retries and receiver restarts. An `alert` and its later `resolved` -are different events, not duplicates. The `X-Pulse-Event-ID` header is a useful +using the [receiver rules below](#receiver-correlation-and-deduplication), +not the signature, send timestamp or alert ID alone. An `alert` and its later +`resolved` are different events, not duplicates. The `X-Pulse-Event-ID` header is a useful correlation hint, but is **not covered by this HMAC**: do not let a changed header bypass deduplication of an otherwise identical signed event. The sample -PSA payloads below include `alertId` and `event` for this purpose. A valid HMAC +PSA payload below includes each member's occurrence identity for this purpose. A valid HMAC establishes integrity, not that processing succeeded; retain your receiver's normal durable processing and retry handling. Never log the secret or full credential-bearing request while diagnosing a rejection. @@ -153,6 +164,44 @@ Content-Type: application/json } ``` +### Receiver correlation and deduplication + +Keep ticket correlation separate from suppressing repeated processing: + +- Scope durable receiver records to the **configured sender and destination** + and tenant, not just a resource name or alert ID. Read identity from the + authenticated payload's fields after verifying the request; retain it across + receiver restarts. The tenant field is context, not permission to act in an + arbitrary account. +- For normal queued firing and recovery notifications, use each member of + `{{.Alerts}}`, not only the primary `{{.ID}}`. The [full PSA template below](#sample-psa-payloads) + includes each member's `alertId` and full-precision UTC `startedAt`. These + identify its observed occurrence; a later start with the same ID must be + processed as a new occurrence. Firing and recovery use the same occurrence + record, so an old delayed recovery must not close a newer incident. +- Within that record, handle `event` **and severity**. A warning becoming + critical is an update, not a duplicate warning. Decide explicitly whether + reminders update the existing ticket; do not create another ticket for each + retry or discard a severity increase. Group membership can change without + changing the header: process every member, even when the primary is unchanged. +- The primary `{{.StartTime}}` string has only whole-second precision. Member + `StartTime` values in `{{.Alerts}}` are Go times and can be formatted with + fractional seconds as below. Do not assume the shorter string is universally + unique. A test or legacy payload can lack member identity; keep incomplete or + ambiguous events for reconciliation rather than silently treating them as + duplicates. **Send test** alone does not validate this lifecycle. +- Commit receiver state and the ticket action atomically, or use your ticket + system's durable idempotent operation. Do not mark an event processed before + its action succeeds. Exact signed-byte replays can be rejected separately, + but a queue retry may render a new body or send timestamp: hashing those alone + is not logical-event deduplication. There is no universal exactly-once key in + this legacy header. + +Validate in an authorised test environment with firing, retry, warning-to-critical, +recovery, recurrence of the same condition and a changed multi-alert group. +Check actual tickets and receiver restart behaviour before enabling automatic +actions. Do not cause a production outage or notification storm for this test. + ## 🛡️ Security - **Private IPs**: By default, webhooks to private IPs are blocked. Allow them in **Settings → System → Network → Webhook Security**. @@ -204,7 +253,14 @@ Suspended or pending-deletion organizations return `403`, and an unknown org ID There are two integration models. The push model is usually the right fit when tickets or incidents should open and close automatically. -**Push (recommended): one outbound webhook per organization.** Create a **Generic** webhook for each organization and point it at your external system's inbound endpoint (an ITSM/PSA inbound webhook, an email connector, or middleware that opens service tickets). Shape the JSON with a [custom template](#-custom-templates) so it matches the receiving system's expected schema: every template variable listed above is available. Pulse fires on both `alert` and `resolved` events (`{{.Event}}` is `"alert"` or `"resolved"`), so the receiving system can open a ticket on alert and auto-resolve it on recovery. Add authentication as a custom header (e.g. `Authorization: Bearer ...`). +**Push (recommended): one outbound webhook per organization.** Create a +**Generic** webhook for each organization and point it at your external system's +inbound endpoint (an ITSM/PSA inbound webhook, an email connector, or middleware +that opens service tickets). Shape the JSON with a [custom template](#-custom-templates) +to match the receiving system's schema. The bridge can open or update a ticket +on `alert` and resolve that occurrence on `resolved`, subject to the [delivery +conditions above](#-delivery-contract). Add authentication as a custom header +in the settings form; keep its value private. Configure it from the UI (**Alerts → Notifications → Add Webhook**) per org, or programmatically with an org-bound admin token: @@ -221,20 +277,25 @@ Content-Type: application/json "service": "generic", "enabled": true, "headers": { "Authorization": "Bearer " }, - "template": "{\"summary\":\"{{.Level}}: {{.ResourceName}} {{.Message | jsonString}}\",\"event\":\"{{.Event}}\",\"alertId\":\"{{.ID}}\"}" + "template": "{\"summary\":\"{{.Level | jsonString}}: {{.ResourceName | jsonString}} {{.Message | jsonString}}\",\"event\":\"{{.Event | jsonString}}\",\"alertId\":\"{{.ID | jsonString}}\",\"startedAt\":\"{{.StartTime | jsonString}}\"}" } ``` -The exact ticket fields differ by platform (ConnectWise, Autotask, Halo, and others each expect their own inbound shape), so map the template to your platform's contract. The `alertId` round-trips through `{{.ID}}`, which lets the receiving system correlate the later `resolved` event to the ticket it opened. +The exact ticket fields differ by platform (ConnectWise, Autotask, Halo, and +others each expect their own inbound shape), so map the template to your +platform's contract. This short example carries only the primary alert and a +whole-second start. For a bridge handling real grouped notifications, use the +member-aware template below; do not use the short example as an exactly-once key. ### Sample PSA payloads -A fuller template suited to ticket routing, including tenant identity and the -stable severity/type fields from the [delivery contract](#-delivery-contract): +A fuller template for normal queued firing and recovery notifications includes +tenant context and every member's occurrence, severity and condition. The +primary fields remain convenient summary context, **not the whole batch**: ```json { - "event": "{{.Event}}", + "event": "{{.Event | jsonString}}", "alertId": "{{.ID | jsonString}}", "messageKey": "{{.MessageKey | jsonString}}", "severity": "{{.Level | jsonString}}", @@ -248,11 +309,24 @@ stable severity/type fields from the [delivery contract](#-delivery-contract): "value": {{.Value}}, "threshold": {{.Threshold}}, "startedAt": "{{.StartTime | jsonString}}", - "duration": "{{.Duration | jsonString}}" + "duration": "{{.Duration | jsonString}}", + "alertCount": {{.AlertCount}}, + "alerts": [{{$comma := ""}}{{range .Alerts}}{{if .}}{{$comma}} + { + "alertId": "{{.ID | jsonString}}", + "startedAt": "{{.StartTime.UTC.Format "2006-01-02T15:04:05.999999999Z07:00" | jsonString}}", + "severity": "{{.Level | jsonString}}", + "alertType": "{{.Type | jsonString}}", + "resourceId": "{{.ResourceID | jsonString}}", + "resource": "{{.ResourceName | jsonString}}", + "summary": "{{.Message | jsonString}}" + }{{$comma = ","}}{{end}}{{end}} + ] } ``` -What the receiver sees for a **critical** alert: +An abridged primary summary for a **critical** alert (the template also emits +the `alerts` member array): ```json { @@ -272,9 +346,12 @@ What the receiver sees for a **critical** alert: } ``` -A **warning** alert is identical except `"severity": "warning"` — warning and critical are the only two severities Pulse emits, so a two-priority PSA mapping covers the full range. +A **warning** alert is identical except `"severity": "warning"`. Map +informational levels too; a two-priority mapping does not cover every condition. -The **resolved** event reuses the same `alertId`, letting the bridge close the ticket it opened: +The **resolved** event retains the original member `alertId` and `startedAt`, +letting the bridge close that occurrence's ticket rather than a later incident +with the same ID. Its primary summary is: ```json { @@ -294,7 +371,12 @@ The **resolved** event reuses the same `alertId`, letting the bridge close the t } ``` -For ConnectWise specifically, point the webhook at a ConnectWise inbound API callback (or middleware that calls the ConnectWise REST API) and map `severity` to ticket priority, `tenantName` to the company, and `alertId` to your correlation field. Combine with a [`signingSecret`](#-delivery-contract) and the `X-Pulse-Event-ID` dedup header for a production-grade bridge. +For ConnectWise specifically, point the webhook at a ConnectWise inbound API +callback (or middleware that calls the ConnectWise REST API). Map each member's +severity to ticket priority and the tenant to your configured company. Combine +[`signingSecret`](#-delivery-contract) with the [receiver correlation rules](#receiver-correlation-and-deduplication), +not header-only deduplication. Verify your platform's processing and recovery +behaviour before treating the bridge as production-ready. **Pull (poll): org-scoped read API.** Issue a `monitoring:read` token bound to each organization and poll that org's alerts. Send `X-Pulse-Org-ID` (or rely on the org-bound token) so you get only that organization's data: diff --git a/docs/release-control/v6/internal/subsystems/notifications.md b/docs/release-control/v6/internal/subsystems/notifications.md index 1e63866860..9081f935bf 100644 --- a/docs/release-control/v6/internal/subsystems/notifications.md +++ b/docs/release-control/v6/internal/subsystems/notifications.md @@ -493,9 +493,20 @@ through the canonical webhook transport must send `X-Pulse-Timestamp` and `X-Pulse-Signature` (`v1=` + hex HMAC-SHA256 over `timestamp + "." + body`), computed at the single request-construction choke point and set after custom headers so user-provided header maps cannot shadow them. Alert deliveries also -send `X-Pulse-Event-ID` (`alertID:event`) as the idempotency token; it must be +send `X-Pulse-Event-ID` (`alertID:event`) as a legacy correlation token; it must be stable across both transport-layer and queue-layer retries of the same alert -occurrence. The management API must mask a configured signing secret on read +occurrence. It omits occurrence start, severity and grouped membership and is +not a unique incident or delivery identity; it is not covered by the HMAC. +Receiver guidance must not prescribe permanent header-only deduplication. +Normal queued templates expose every member through `.Alerts`, whose full- +precision `StartTime` can distinguish later occurrences even when the primary +RFC3339 start string has the same whole second. Correlate recovery with that +occurrence and preserve severity updates and changed members. The existing +wire format and retry stability remain unchanged. `webhook_guide_test.go` +exercises copied templates through the actual signed sender over loopback for +retry, escalation, recovery, recurrence, group changes and JSON escaping; it +does not establish durable ticket actions or installed notification delivery. +The management API must mask a configured signing secret on read and preserve the stored secret when an update echoes the masked placeholder, the same ownership rule it applies to header and custom-field secrets. That same transport boundary also owns webhook request normalization. Rendered diff --git a/frontend-modern/browser-tests/docs-fragment-navigation.tsx b/frontend-modern/browser-tests/docs-fragment-navigation.tsx index 373a4dd878..628e095444 100644 --- a/frontend-modern/browser-tests/docs-fragment-navigation.tsx +++ b/frontend-modern/browser-tests/docs-fragment-navigation.tsx @@ -25,6 +25,7 @@ const targets: Record = { truenas: '/docs/TRUENAS', installation: '/docs/INSTALL', configuration: '/docs/CONFIGURATION#-api-tokens', + webhooks: '/docs/WEBHOOKS', }; window.history.replaceState({}, '', targets[scenario] ?? targets.plain); diff --git a/frontend-modern/browser-tests/webhook-guidance.cjs b/frontend-modern/browser-tests/webhook-guidance.cjs new file mode 100644 index 0000000000..5a948f73d7 --- /dev/null +++ b/frontend-modern/browser-tests/webhook-guidance.cjs @@ -0,0 +1,116 @@ +// Inspect the real shipped Docs page and copyable webhook template. This is +// offline rendering, not a ticket receiver, notification or external delivery. +const assert = require('node:assert/strict'); +const fs = require('node:fs'); +const path = require('node:path'); +const crypto = require('node:crypto'); +const { chromium, webkit } = require('playwright'); + +(async () => { + const root = '/workspace/frontend-modern'; + const artifacts = path.join(root, 'node_modules', 'webhook-guidance-browser'); + fs.mkdirSync(artifacts, { recursive: true }); + const { createServer } = await import(path.join(root, 'node_modules/vite/dist/node/index.js')); + const server = await createServer({ root, configFile: path.join(root, 'vite.config.ts'), + cacheDir: path.join(root, 'node_modules', '.vite-webhook-guidance'), + server: { host: '127.0.0.1', port: 5253, strictPort: true }, + optimizeDeps: { noDiscovery: true, include: [] } }); + const docs = fs.readFileSync(path.join(root, 'public/docs/WEBHOOKS.md'), 'utf8'); + const expectedTemplate = docs.split('### Sample PSA payloads')[1].match(/```json\n([\s\S]*?)\n```/)[1]; + let browser; + const results = []; + try { + await server.listen(); + for (const [engine, width] of [['chromium', 1280], ['webkit', 390]]) { + browser = await { chromium, webkit }[engine].launch(engine === 'chromium' + ? { headless: true, channel: 'chromium', args: ['--no-sandbox'] } : { headless: true }); + const page = await browser.newPage({ viewport: { width, height: 1000 }, + ...(engine === 'webkit' ? { isMobile: true, hasTouch: true } : {}) }); + const errors = []; + const unexpectedAPIRequests = []; + page.on('pageerror', (error) => errors.push(error.message)); + page.on('console', (message) => { if (message.type() === 'error') errors.push(message.text()); }); + page.on('request', (request) => { + const pathname = new URL(request.url()).pathname; + if (pathname.startsWith('/api/') && pathname !== '/api/security/status') { + unexpectedAPIRequests.push(pathname); + } + }); + let securityRequests = 0; + // Docs imports runtime context, whose initial auth check is unrelated + // to Markdown rendering. Model first-run state explicitly; there is no + // authenticated runtime, backend or provider in this offline fixture. + await page.route('**/api/security/status', async (route) => { + securityRequests++; + await route.fulfill({ json: { hasAuthentication: false, requiresAuth: true } }); + }); + await page.goto('http://127.0.0.1:5253/browser-tests/docs-fragment-navigation.html?scenario=webhooks', + { waitUntil: 'domcontentloaded', timeout: 60_000 }); + const heading = page.getByRole('heading', { name: 'Receiver correlation and deduplication', exact: true }); + await heading.waitFor(); + if (engine === 'webkit') await page.evaluate(() => document.documentElement.classList.add('dark')); + const body = (await page.locator('article').innerText()).replace(/\s+/g, ' '); + for (const text of ['not a unique incident or delivery ID', 'Do not deduplicate permanently', + 'old delayed recovery must not close a newer incident', 'event and severity', + 'whole-second precision', 'incomplete or ambiguous events', + 'successful firing-delivery receipt', 'informational conditions can use "info"']) { + assert.ok(body.includes(text), text); + } + assert.ok(!body.includes('deduplicate on it')); + const follow = async (name, href, target) => { + const link = page.getByRole('link', { name, exact: true }); + assert.equal(await link.getAttribute('href'), href); + await link.focus(); + await page.keyboard.press('Enter'); + await page.waitForURL(`**${href}`); + await page.waitForFunction((id) => document.activeElement?.id === id, target); + }; + await follow('receiver correlation and deduplication', '#receiver-correlation-and-deduplication', + 'receiver-correlation-and-deduplication'); + const screenshot = async (name, headingName) => { + await page.getByRole('heading', { name: headingName, exact: true }) + .evaluate((element) => element.scrollIntoView({ block: 'start' })); + await page.evaluate(async () => { + await new Promise((resolve) => requestAnimationFrame(() => requestAnimationFrame(resolve))); + await Promise.all(document.getAnimations().map((animation) => animation.finished.catch(() => {}))); + }); + assert.ok(await page.evaluate(() => document.documentElement.scrollWidth <= innerWidth + 1)); + await page.screenshot({ path: path.join(artifacts, `${engine}-${name}.png`) }); + }; + await screenshot('receiver', 'Receiver correlation and deduplication'); + await follow('full PSA template below', '#sample-psa-payloads', 'sample-psa-payloads'); + const template = page.locator('pre code').filter({ hasText: '"alertCount":' }); + assert.equal(await template.count(), 1); + assert.equal((await template.innerText()).trim(), expectedTemplate.trim()); + assert.ok(await template.evaluate((element) => { + const pre = element.closest('pre'); + return getComputedStyle(pre).overflowX === 'auto' || pre.scrollWidth <= pre.clientWidth + 1; + })); + await screenshot('template', 'Sample PSA payloads'); + await page.reload({ waitUntil: 'domcontentloaded' }); + await page.getByRole('heading', { name: 'Sample PSA payloads', exact: true }).waitFor(); + await page.waitForFunction(() => document.activeElement?.id === 'sample-psa-payloads'); + // An incidental import is not obliged to make an auth request on every + // engine/reload. Observe those requests, but assert only the Docs result + // and absence of unintended backend activity or errors. + await page.waitForLoadState('networkidle'); + assert.deepEqual(errors, []); + assert.deepEqual(unexpectedAPIRequests, []); + results.push({ engine, browserVersion: browser.version(), width, + scope: engine === 'webkit' ? 'phone-emulated, not native' : 'desktop', + warningsRendered: true, keyboardReceiverLink: true, keyboardTemplateLink: true, + templateTextExact: true, reloadFragmentFocus: true, noDocumentOverflow: true, + syntheticFirstRunSecurityRequests: securityRequests, unexpectedAPIRequests, errors }); + await browser.close(); + browser = undefined; + } + const files = ['public/docs/WEBHOOKS.md', 'browser-tests/docs-fragment-navigation.tsx', + 'browser-tests/webhook-guidance.cjs', 'src/features/docs/docMarkdown.ts', 'src/pages/Docs.tsx'] + .map((file) => ({ file, sha256: crypto.createHash('sha256').update(fs.readFileSync(path.join(root, file))).digest('hex') })); + fs.writeFileSync(path.join(artifacts, 'result.json'), JSON.stringify({ playwright: require('playwright/package.json').version, + files, results, limits: 'No real receiver, ticket, provider, native device or installed notification acceptance.' }, null, 2)); + } finally { + if (browser) await browser.close(); + await server.close(); + } +})().catch((error) => { console.error(error); process.exitCode = 1; }); diff --git a/frontend-modern/public/docs/WEBHOOKS.md b/frontend-modern/public/docs/WEBHOOKS.md index 3acf6649b4..cb0f8424fd 100644 --- a/frontend-modern/public/docs/WEBHOOKS.md +++ b/frontend-modern/public/docs/WEBHOOKS.md @@ -65,11 +65,18 @@ For generic webhooks, use Go templates to format the JSON payload. These fields and behaviors are stable; ticket-routing integrations can rely on them. -**Events.** Every webhook fires on both `alert` and `resolved` events. `{{.Event}}` is `"alert"` or `"resolved"` — there is no separate "info" event class. +**Events.** `{{.Event}}` is `"alert"` or `"resolved"` — there is no separate +"info" event class. Recovery delivery depends on **Notify on resolve** and a +successful firing-delivery receipt for that occurrence and destination. Do not +assume that every configured webhook receives both events. -**Severity.** `{{.Level}}` is `"warning"` or `"critical"`. Pulse has exactly these two alert levels. +**Severity.** Metric alerts commonly use `"warning"` or `"critical"`; +informational conditions can use `"info"`. Event and severity are separate: +an informational firing still has event `"alert"`. Preserve and flag an +unrecognised level rather than silently dropping the notification. Minimum +severity and other delivery policies still apply. -**Alert type.** `{{.Type}}` is the metric or condition that fired: `cpu`, `memory`, `disk`, `diskRead`, `diskWrite`, `networkIn`, `networkOut`, `connectivity`, and similar. The alert ID (`{{.ID}}`) is stable for the lifetime of an alert occurrence, so the `resolved` event carries the same ID as the `alert` event it closes. +**Alert type.** `{{.Type}}` is the metric or condition that fired: `cpu`, `memory`, `disk`, `diskRead`, `diskWrite`, `networkIn`, `networkOut`, `connectivity`, and similar. The alert ID (`{{.ID}}`) correlates firing and recovery, but can be reused when the same condition fires again. It is not a unique incident or delivery ID. **Message key.** `{{.MessageKey}}` is the stable, language-neutral condition key for rebuilding or translating a notification. Canonical alerts use `.` (for example, `metric-threshold.disk`); older alert paths fall back to `{{.Type}}`. Combine it with `{{.Event}}` and `{{.ResourceType}}` when the receiving system needs separate wording for firing/recovery events or different resource classes. Unlike a numeric message index, the symbolic key does not change when another alert type is added. @@ -79,7 +86,13 @@ These fields and behaviors are stable; ticket-routing integrations can rely on t **Retries.** Failed deliveries retry with exponential backoff. The persistent notification queue makes up to 3 delivery attempts per notification; webhooks configured with transport-level retry add up to 3 more HTTP retries per attempt (1s doubling to a 30s cap, honoring `Retry-After` on HTTP 429). A receiver may therefore see the same logical event more than once. -**Idempotency.** Every alert delivery carries an `X-Pulse-Event-ID` header of the form `:` (e.g. `a1b2c3:alert`, `a1b2c3:resolved`). It is identical across all retries of the same logical event — deduplicate on it. +**Correlation header.** Alert webhooks carry `X-Pulse-Event-ID` in the form +`:` (e.g. `a1b2c3:alert`, `a1b2c3:resolved`). Retries retain it, +but later occurrences, severity changes and reminders can share it too. A group +uses its primary alert's ID, not a unique ID for every member or batch. **Do not +deduplicate permanently on this header or on alert ID and event alone:** doing +so can silently discard a later incident or a changed group. See +[receiver correlation and deduplication](#receiver-correlation-and-deduplication). **Signed deliveries.** Set a `signingSecret` on the webhook config to enable HMAC signing. Signed requests carry: @@ -126,14 +139,12 @@ def verify(secret: str, timestamp: str, body: bytes, signature: str) -> bool: A time window rejects old captures, **not every replay**: the same signed request can arrive more than once within that window, and Pulse signs each retry with its current send time. After verification, deduplicate atomically -before creating a ticket or taking another action. Use a durable key scoped to -the configured Pulse sender and destination, built from the authenticated -payload's alert ID and event (`{{.ID}}` and `{{.Event}}` in your template); keep -it across retries and receiver restarts. An `alert` and its later `resolved` -are different events, not duplicates. The `X-Pulse-Event-ID` header is a useful +using the [receiver rules below](#receiver-correlation-and-deduplication), +not the signature, send timestamp or alert ID alone. An `alert` and its later +`resolved` are different events, not duplicates. The `X-Pulse-Event-ID` header is a useful correlation hint, but is **not covered by this HMAC**: do not let a changed header bypass deduplication of an otherwise identical signed event. The sample -PSA payloads below include `alertId` and `event` for this purpose. A valid HMAC +PSA payload below includes each member's occurrence identity for this purpose. A valid HMAC establishes integrity, not that processing succeeded; retain your receiver's normal durable processing and retry handling. Never log the secret or full credential-bearing request while diagnosing a rejection. @@ -153,6 +164,44 @@ Content-Type: application/json } ``` +### Receiver correlation and deduplication + +Keep ticket correlation separate from suppressing repeated processing: + +- Scope durable receiver records to the **configured sender and destination** + and tenant, not just a resource name or alert ID. Read identity from the + authenticated payload's fields after verifying the request; retain it across + receiver restarts. The tenant field is context, not permission to act in an + arbitrary account. +- For normal queued firing and recovery notifications, use each member of + `{{.Alerts}}`, not only the primary `{{.ID}}`. The [full PSA template below](#sample-psa-payloads) + includes each member's `alertId` and full-precision UTC `startedAt`. These + identify its observed occurrence; a later start with the same ID must be + processed as a new occurrence. Firing and recovery use the same occurrence + record, so an old delayed recovery must not close a newer incident. +- Within that record, handle `event` **and severity**. A warning becoming + critical is an update, not a duplicate warning. Decide explicitly whether + reminders update the existing ticket; do not create another ticket for each + retry or discard a severity increase. Group membership can change without + changing the header: process every member, even when the primary is unchanged. +- The primary `{{.StartTime}}` string has only whole-second precision. Member + `StartTime` values in `{{.Alerts}}` are Go times and can be formatted with + fractional seconds as below. Do not assume the shorter string is universally + unique. A test or legacy payload can lack member identity; keep incomplete or + ambiguous events for reconciliation rather than silently treating them as + duplicates. **Send test** alone does not validate this lifecycle. +- Commit receiver state and the ticket action atomically, or use your ticket + system's durable idempotent operation. Do not mark an event processed before + its action succeeds. Exact signed-byte replays can be rejected separately, + but a queue retry may render a new body or send timestamp: hashing those alone + is not logical-event deduplication. There is no universal exactly-once key in + this legacy header. + +Validate in an authorised test environment with firing, retry, warning-to-critical, +recovery, recurrence of the same condition and a changed multi-alert group. +Check actual tickets and receiver restart behaviour before enabling automatic +actions. Do not cause a production outage or notification storm for this test. + ## 🛡️ Security - **Private IPs**: By default, webhooks to private IPs are blocked. Allow them in **Settings → System → Network → Webhook Security**. @@ -204,7 +253,14 @@ Suspended or pending-deletion organizations return `403`, and an unknown org ID There are two integration models. The push model is usually the right fit when tickets or incidents should open and close automatically. -**Push (recommended): one outbound webhook per organization.** Create a **Generic** webhook for each organization and point it at your external system's inbound endpoint (an ITSM/PSA inbound webhook, an email connector, or middleware that opens service tickets). Shape the JSON with a [custom template](#-custom-templates) so it matches the receiving system's expected schema: every template variable listed above is available. Pulse fires on both `alert` and `resolved` events (`{{.Event}}` is `"alert"` or `"resolved"`), so the receiving system can open a ticket on alert and auto-resolve it on recovery. Add authentication as a custom header (e.g. `Authorization: Bearer ...`). +**Push (recommended): one outbound webhook per organization.** Create a +**Generic** webhook for each organization and point it at your external system's +inbound endpoint (an ITSM/PSA inbound webhook, an email connector, or middleware +that opens service tickets). Shape the JSON with a [custom template](#-custom-templates) +to match the receiving system's schema. The bridge can open or update a ticket +on `alert` and resolve that occurrence on `resolved`, subject to the [delivery +conditions above](#-delivery-contract). Add authentication as a custom header +in the settings form; keep its value private. Configure it from the UI (**Alerts → Notifications → Add Webhook**) per org, or programmatically with an org-bound admin token: @@ -221,20 +277,25 @@ Content-Type: application/json "service": "generic", "enabled": true, "headers": { "Authorization": "Bearer " }, - "template": "{\"summary\":\"{{.Level}}: {{.ResourceName}} {{.Message | jsonString}}\",\"event\":\"{{.Event}}\",\"alertId\":\"{{.ID}}\"}" + "template": "{\"summary\":\"{{.Level | jsonString}}: {{.ResourceName | jsonString}} {{.Message | jsonString}}\",\"event\":\"{{.Event | jsonString}}\",\"alertId\":\"{{.ID | jsonString}}\",\"startedAt\":\"{{.StartTime | jsonString}}\"}" } ``` -The exact ticket fields differ by platform (ConnectWise, Autotask, Halo, and others each expect their own inbound shape), so map the template to your platform's contract. The `alertId` round-trips through `{{.ID}}`, which lets the receiving system correlate the later `resolved` event to the ticket it opened. +The exact ticket fields differ by platform (ConnectWise, Autotask, Halo, and +others each expect their own inbound shape), so map the template to your +platform's contract. This short example carries only the primary alert and a +whole-second start. For a bridge handling real grouped notifications, use the +member-aware template below; do not use the short example as an exactly-once key. ### Sample PSA payloads -A fuller template suited to ticket routing, including tenant identity and the -stable severity/type fields from the [delivery contract](#-delivery-contract): +A fuller template for normal queued firing and recovery notifications includes +tenant context and every member's occurrence, severity and condition. The +primary fields remain convenient summary context, **not the whole batch**: ```json { - "event": "{{.Event}}", + "event": "{{.Event | jsonString}}", "alertId": "{{.ID | jsonString}}", "messageKey": "{{.MessageKey | jsonString}}", "severity": "{{.Level | jsonString}}", @@ -248,11 +309,24 @@ stable severity/type fields from the [delivery contract](#-delivery-contract): "value": {{.Value}}, "threshold": {{.Threshold}}, "startedAt": "{{.StartTime | jsonString}}", - "duration": "{{.Duration | jsonString}}" + "duration": "{{.Duration | jsonString}}", + "alertCount": {{.AlertCount}}, + "alerts": [{{$comma := ""}}{{range .Alerts}}{{if .}}{{$comma}} + { + "alertId": "{{.ID | jsonString}}", + "startedAt": "{{.StartTime.UTC.Format "2006-01-02T15:04:05.999999999Z07:00" | jsonString}}", + "severity": "{{.Level | jsonString}}", + "alertType": "{{.Type | jsonString}}", + "resourceId": "{{.ResourceID | jsonString}}", + "resource": "{{.ResourceName | jsonString}}", + "summary": "{{.Message | jsonString}}" + }{{$comma = ","}}{{end}}{{end}} + ] } ``` -What the receiver sees for a **critical** alert: +An abridged primary summary for a **critical** alert (the template also emits +the `alerts` member array): ```json { @@ -272,9 +346,12 @@ What the receiver sees for a **critical** alert: } ``` -A **warning** alert is identical except `"severity": "warning"` — warning and critical are the only two severities Pulse emits, so a two-priority PSA mapping covers the full range. +A **warning** alert is identical except `"severity": "warning"`. Map +informational levels too; a two-priority mapping does not cover every condition. -The **resolved** event reuses the same `alertId`, letting the bridge close the ticket it opened: +The **resolved** event retains the original member `alertId` and `startedAt`, +letting the bridge close that occurrence's ticket rather than a later incident +with the same ID. Its primary summary is: ```json { @@ -294,7 +371,12 @@ The **resolved** event reuses the same `alertId`, letting the bridge close the t } ``` -For ConnectWise specifically, point the webhook at a ConnectWise inbound API callback (or middleware that calls the ConnectWise REST API) and map `severity` to ticket priority, `tenantName` to the company, and `alertId` to your correlation field. Combine with a [`signingSecret`](#-delivery-contract) and the `X-Pulse-Event-ID` dedup header for a production-grade bridge. +For ConnectWise specifically, point the webhook at a ConnectWise inbound API +callback (or middleware that calls the ConnectWise REST API). Map each member's +severity to ticket priority and the tenant to your configured company. Combine +[`signingSecret`](#-delivery-contract) with the [receiver correlation rules](#receiver-correlation-and-deduplication), +not header-only deduplication. Verify your platform's processing and recovery +behaviour before treating the bridge as production-ready. **Pull (poll): org-scoped read API.** Issue a `monitoring:read` token bound to each organization and poll that org's alerts. Send `X-Pulse-Org-ID` (or rely on the org-bound token) so you get only that organization's data: diff --git a/internal/notifications/notifications.go b/internal/notifications/notifications.go index 18d01ea67a..61277858b0 100644 --- a/internal/notifications/notifications.go +++ b/internal/notifications/notifications.go @@ -280,7 +280,7 @@ type webhookHTTPResult struct { type webhookRequestOptions struct { alertType string - eventID string // Idempotency token sent as X-Pulse-Event-ID (stable across retries) + eventID string // Legacy X-Pulse-Event-ID correlation header, not a unique delivery ID timeout time.Duration userAgent string responseLogging bool @@ -3344,9 +3344,9 @@ func signWebhookPayload(secret, timestamp string, payload []byte) string { return hex.EncodeToString(mac.Sum(nil)) } -// webhookEventID builds the idempotency token for a delivery: stable for the -// same alert occurrence and event so receivers can deduplicate retries at -// either retry layer (transport or queue). +// webhookEventID builds the legacy correlation header, stable across retries. +// It omits occurrence start, severity and grouped membership, so it is not a +// unique incident/delivery identity or a sufficient durable deduplication key. func webhookEventID(alertID, event string) string { if alertID == "" { return "" diff --git a/internal/notifications/webhook_guide_test.go b/internal/notifications/webhook_guide_test.go new file mode 100644 index 0000000000..37a27878f8 --- /dev/null +++ b/internal/notifications/webhook_guide_test.go @@ -0,0 +1,193 @@ +package notifications + +import ( + "crypto/hmac" + "crypto/sha256" + "encoding/hex" + "encoding/json" + "io" + "net/http" + "os" + "regexp" + "strings" + "testing" + "time" + + "github.com/rcourtman/pulse-go-rewrite/internal/alerts" + "github.com/stretchr/testify/require" +) + +// Exercise the copyable templates through the real sender, not a second +// renderer or a model receiver. The input override permits an exact-parent +// documentation negative control without modifying production source. +func webhookGuide(t *testing.T) string { + t.Helper() + path := os.Getenv("PULSE_WEBHOOK_GUIDE_TEST_INPUT") + if path == "" { + path = "../../docs/WEBHOOKS.md" + } + data, err := os.ReadFile(path) + require.NoError(t, err) + return string(data) +} + +func webhookGuidePSATemplate(t *testing.T) string { + t.Helper() + sections := strings.SplitN(webhookGuide(t), "### Sample PSA payloads", 2) + require.Len(t, sections, 2) + block := regexp.MustCompile("(?s)```json\\n(.*?)\\n```").FindStringSubmatch(sections[1]) + require.Len(t, block, 2) + return block[1] +} + +type guideMember struct { + ID string `json:"alertId"` + StartedAt string `json:"startedAt"` + Severity string `json:"severity"` + ResourceID string `json:"resourceId"` + Resource string `json:"resource"` + Summary string `json:"summary"` +} + +type guidePayload struct { + Event string `json:"event"` + TenantID string `json:"tenantId"` + StartedAt string `json:"startedAt"` + AlertCount int `json:"alertCount"` + Members []guideMember `json:"alerts"` +} + +type guideRequest struct { + body, eventID, timestamp, signature string +} + +func webhookGuideSender(t *testing.T, payloadTemplate string) (*NotificationManager, WebhookConfig, <-chan guideRequest) { + t.Helper() + manager := NewNotificationManager("https://pulse.example.test") + require.NoError(t, manager.UpdateAllowedPrivateCIDRs("127.0.0.1")) + manager.SetTenantIdentityResolver(func() (string, string) { return "synthetic-tenant", "Synthetic tenant" }) + requests := make(chan guideRequest, 16) + server := newIPv4HTTPServer(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + body, err := io.ReadAll(r.Body) + if err != nil { + w.WriteHeader(http.StatusBadRequest) + return + } + requests <- guideRequest{string(body), r.Header.Get("X-Pulse-Event-ID"), + r.Header.Get("X-Pulse-Timestamp"), r.Header.Get("X-Pulse-Signature")} + w.WriteHeader(http.StatusOK) + })) + t.Cleanup(server.Close) + webhook := WebhookConfig{ID: "synthetic-guide", Name: "Synthetic guide receiver", URL: server.URL, + Enabled: true, Service: "generic", SigningSecret: "synthetic-guide-test-secret", Template: payloadTemplate} + return manager, webhook, requests +} + +func webhookGuideRead(t *testing.T, requests <-chan guideRequest) (guideRequest, guidePayload) { + t.Helper() + select { + case request := <-requests: + var payload guidePayload + require.NoError(t, json.Unmarshal([]byte(request.body), &payload), "copied template must render valid JSON") + mac := hmac.New(sha256.New, []byte("synthetic-guide-test-secret")) + _, err := mac.Write([]byte(request.timestamp + "." + request.body)) + require.NoError(t, err) + require.Equal(t, "v1="+hex.EncodeToString(mac.Sum(nil)), request.signature) + return request, payload + case <-time.After(5 * time.Second): + t.Fatal("no local sender request received") + return guideRequest{}, guidePayload{} + } +} + +func TestWebhookGuidePSALifecycle(t *testing.T) { + manager, webhook, requests := webhookGuideSender(t, webhookGuidePSATemplate(t)) + start := time.Date(2026, 10, 4, 7, 0, 0, 123456789, time.UTC) + alert := &alerts.Alert{ID: "host::metric-threshold:cpu", StartTime: start, + Level: alerts.AlertLevelWarning, Type: "cpu", ResourceID: "synthetic-host", + ResourceName: "quoted \"host\"\nwith é", Message: "synthetic \\ message\n\"quoted\"", Value: 95, Threshold: 90} + + require.NoError(t, manager.sendGroupedWebhook(webhook, []*alerts.Alert{alert})) + firstRequest, first := webhookGuideRead(t, requests) + require.Len(t, first.Members, 1) + require.Equal(t, "alert", first.Event) + require.Equal(t, "synthetic-tenant", first.TenantID) + require.Equal(t, 1, first.AlertCount) + require.Equal(t, guideMember{alert.ID, start.Format(time.RFC3339Nano), "warning", + alert.ResourceID, alert.ResourceName, alert.Message}, first.Members[0]) + + // A synthetic retry retains the member identity even when rendering anew. + require.NoError(t, manager.sendGroupedWebhook(webhook, []*alerts.Alert{alert})) + retryRequest, retry := webhookGuideRead(t, requests) + require.Equal(t, first.Members, retry.Members) + require.Equal(t, firstRequest.eventID, retryRequest.eventID) + + critical := alert.Clone() + critical.Level = alerts.AlertLevelCritical + require.NoError(t, manager.sendGroupedWebhook(webhook, []*alerts.Alert{critical})) + criticalRequest, escalated := webhookGuideRead(t, requests) + require.Equal(t, firstRequest.eventID, criticalRequest.eventID, "header cannot identify a severity change") + require.Equal(t, first.Members[0].StartedAt, escalated.Members[0].StartedAt) + require.Equal(t, "critical", escalated.Members[0].Severity) + + require.NoError(t, manager.sendResolvedWebhook(webhook, []*alerts.Alert{critical}, start.Add(time.Minute))) + _, resolved := webhookGuideRead(t, requests) + require.Equal(t, "resolved", resolved.Event) + require.Equal(t, escalated.Members, resolved.Members, "recovery must correlate to the original occurrence") + + // A second occurrence in the SAME SECOND defeats both header-only and + // whole-second primary-start deduplication. Member precision preserves it. + recurrent := critical.Clone() + recurrent.StartTime = start.Add(100 * time.Nanosecond) + require.NoError(t, manager.sendGroupedWebhook(webhook, []*alerts.Alert{recurrent})) + recurrentRequest, recurrence := webhookGuideRead(t, requests) + require.Equal(t, firstRequest.eventID, recurrentRequest.eventID) + require.Equal(t, first.StartedAt, recurrence.StartedAt) + require.NotEqual(t, first.Members[0].StartedAt, recurrence.Members[0].StartedAt) + require.Equal(t, recurrent.StartTime.Format(time.RFC3339Nano), recurrence.Members[0].StartedAt) +} + +func TestWebhookGuidePSAGroupMembershipAndInfo(t *testing.T) { + manager, webhook, requests := webhookGuideSender(t, webhookGuidePSATemplate(t)) + start := time.Date(2026, 10, 4, 9, 0, 0, 987654321, time.FixedZone("synthetic", 2*60*60)) + primary := &alerts.Alert{ID: "primary", StartTime: start, Level: alerts.AlertLevelWarning} + info := &alerts.Alert{ID: "info-member", StartTime: start.Add(time.Second), Level: alerts.AlertLevelInfo} + critical := &alerts.Alert{ID: "critical-member", StartTime: start.Add(2 * time.Second), Level: alerts.AlertLevelCritical} + for _, members := range [][]*alerts.Alert{{primary, info}, {primary, critical}, {primary, critical, info}} { + require.NoError(t, manager.sendGroupedWebhook(webhook, members)) + request, payload := webhookGuideRead(t, requests) + require.Equal(t, "primary:alert", request.eventID, "changed groups can have the same correlation header") + require.Equal(t, len(members), payload.AlertCount) + require.Len(t, payload.Members, len(members)) + for index, alert := range members { + require.Equal(t, alert.ID, payload.Members[index].ID) + require.Equal(t, string(alert.Level), payload.Members[index].Severity) + require.Equal(t, alert.StartTime.UTC().Format(time.RFC3339Nano), payload.Members[index].StartedAt) + } + } + require.NoError(t, manager.sendResolvedWebhook(webhook, []*alerts.Alert{primary, critical, info}, start.Add(time.Minute))) + _, resolved := webhookGuideRead(t, requests) + require.Equal(t, "resolved", resolved.Event) + require.Len(t, resolved.Members, 3) + require.Equal(t, "info", resolved.Members[2].Severity) +} + +func TestWebhookGuideShortTemplateEscapesStrings(t *testing.T) { + section := strings.SplitN(webhookGuide(t), "### Sample PSA payloads", 2)[0] + block := regexp.MustCompile("(?s)```http\\nPOST /api/notifications/webhooks\\nX-Pulse-Org-ID:.*?\\n\\n(.*?)\\n```").FindStringSubmatch(section) + require.Len(t, block, 2) + var config struct { + Template string `json:"template"` + } + require.NoError(t, json.Unmarshal([]byte(block[1]), &config)) + manager, webhook, requests := webhookGuideSender(t, config.Template) + alert := &alerts.Alert{ID: "quoted\"id", ResourceName: "host\"\\\nname", Message: "message\"\\\n", + StartTime: time.Date(2026, 10, 4, 7, 0, 0, 123, time.UTC), Level: alerts.AlertLevelWarning} + require.NoError(t, manager.sendGroupedWebhook(webhook, []*alerts.Alert{alert})) + request, _ := webhookGuideRead(t, requests) + var payload map[string]interface{} + require.NoError(t, json.Unmarshal([]byte(request.body), &payload)) + require.Equal(t, alert.ID, payload["alertId"]) + require.Equal(t, "warning: "+alert.ResourceName+" "+alert.Message, payload["summary"]) + require.Equal(t, alert.StartTime.Format(time.RFC3339), payload["startedAt"]) +} diff --git a/scripts/tests/test_webhook_verification_docs.py b/scripts/tests/test_webhook_verification_docs.py index 6092c377b5..32b366e264 100644 --- a/scripts/tests/test_webhook_verification_docs.py +++ b/scripts/tests/test_webhook_verification_docs.py @@ -146,6 +146,25 @@ class WebhookVerificationDocsTest(unittest.TestCase): "not from the request", "Never log the secret"): self.assertIn(phrase, text) + def test_receiver_guidance_preserves_recurrences_and_all_group_members(self): + # The Go sender tests render the copied PSA templates over actual + # loopback HTTP. Keep the safety explanation beside those examples. + text = " ".join(DOC.read_text().split()) + for phrase in ("not a unique incident or delivery ID", "later occurrences", + "Do not deduplicate permanently", "every member", + "old delayed recovery must not close a newer incident", + "event and severity", "whole-second precision", + "incomplete or ambiguous events", "atomically", + "Do not mark an event processed before its action succeeds"): + self.assertIn(phrase, text.replace("**", "").replace("`", "")) + self.assertNotIn("deduplicate on it", text) + self.assertNotIn("two-priority PSA mapping covers the full range", text) + self.assertIn('"info"', text) + self.assertIn("successful firing-delivery receipt", text) + contract = (ROOT / "docs/release-control/v6/internal/subsystems/notifications.md").read_text() + self.assertIn("as a legacy correlation token", contract) + self.assertIn("must not prescribe permanent header-only deduplication", contract) + if __name__ == "__main__": unittest.main()