gsadmin/pulse
1
0
Fork 0
mirror of https://github.com/rcourtman/Pulse.git synced 2026-10-24 04:33:17 +00:00
Code Issues Packages Projects Releases Wiki Activity

Keep webhook bridge guidance from discarding later alerts

Browse Source 
The legacy correlation header omits occurrence start, severity and group members. Show receivers the existing full-precision member data, preserve later incidents and severity updates, and test the copyable templates through the actual signed sender. Keep the freshness verifier and runtime wire format unchanged.

Contract-Neutral: Help, executable examples, tests and sender comments only; notification behaviour, schemas and webhook headers are unchanged.
Change-source: pulse-maintainer
This commit is contained in:
pulse-triage[bot]
2026-10-04 08:16:23 +01:00
parent 7ca8945f0d
commit 94a14c78c7
7 changed files with 515 additions and 46 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/WEBHOOKS.md
+103 -21
View File
@@ -65,11 +65,18 @@ For generic webhooks, use Go templates to format the JSON payload.
These fields and behaviors are stable; ticket-routing integrations can rely on them.
**Events.** Every webhook fires on both `alert` and `resolved` events. `{{.Event}}` is `"alert"` or `"resolved"` — there is no separate "info" event class.
**Events.** `{{.Event}}` is `"alert"` or `"resolved"` — there is no separate
"info" event class. Recovery delivery depends on **Notify on resolve** and a
successful firing-delivery receipt for that occurrence and destination. Do not
assume that every configured webhook receives both events.
**Severity.** `{{.Level}}` is `"warning"` or `"critical"`. Pulse has exactly these two alert levels.
**Severity.** Metric alerts commonly use `"warning"` or `"critical"`;
informational conditions can use `"info"`. Event and severity are separate:
an informational firing still has event `"alert"`. Preserve and flag an
unrecognised level rather than silently dropping the notification. Minimum
severity and other delivery policies still apply.
**Alert type.** `{{.Type}}` is the metric or condition that fired: `cpu`, `memory`, `disk`, `diskRead`, `diskWrite`, `networkIn`, `networkOut`, `connectivity`, and similar. The alert ID (`{{.ID}}`) is stable for the lifetime of an alert occurrence, so the `resolved` event carries the same ID as the `alert` event it closes.
**Alert type.** `{{.Type}}` is the metric or condition that fired: `cpu`, `memory`, `disk`, `diskRead`, `diskWrite`, `networkIn`, `networkOut`, `connectivity`, and similar. The alert ID (`{{.ID}}`) correlates firing and recovery, but can be reused when the same condition fires again. It is not a unique incident or delivery ID.
**Message key.** `{{.MessageKey}}` is the stable, language-neutral condition key for rebuilding or translating a notification. Canonical alerts use `<kind>.<type>` (for example, `metric-threshold.disk`); older alert paths fall back to `{{.Type}}`. Combine it with `{{.Event}}` and `{{.ResourceType}}` when the receiving system needs separate wording for firing/recovery events or different resource classes. Unlike a numeric message index, the symbolic key does not change when another alert type is added.
@@ -79,7 +86,13 @@ These fields and behaviors are stable; ticket-routing integrations can rely on t
**Retries.** Failed deliveries retry with exponential backoff. The persistent notification queue makes up to 3 delivery attempts per notification; webhooks configured with transport-level retry add up to 3 more HTTP retries per attempt (1s doubling to a 30s cap, honoring `Retry-After` on HTTP 429). A receiver may therefore see the same logical event more than once.
**Idempotency.** Every alert delivery carries an `X-Pulse-Event-ID` header of the form `<alertID>:<event>` (e.g. `a1b2c3:alert`, `a1b2c3:resolved`). It is identical across all retries of the same logical event — deduplicate on it.
**Correlation header.** Alert webhooks carry `X-Pulse-Event-ID` in the form
`<alertID>:<event>` (e.g. `a1b2c3:alert`, `a1b2c3:resolved`). Retries retain it,
but later occurrences, severity changes and reminders can share it too. A group
uses its primary alert's ID, not a unique ID for every member or batch. **Do not
deduplicate permanently on this header or on alert ID and event alone:** doing
so can silently discard a later incident or a changed group. See
[receiver correlation and deduplication](#receiver-correlation-and-deduplication).
**Signed deliveries.** Set a `signingSecret` on the webhook config to enable HMAC signing. Signed requests carry:
@@ -126,14 +139,12 @@ def verify(secret: str, timestamp: str, body: bytes, signature: str) -> bool:
A time window rejects old captures, **not every replay**: the same signed
request can arrive more than once within that window, and Pulse signs each
retry with its current send time. After verification, deduplicate atomically
before creating a ticket or taking another action. Use a durable key scoped to
the configured Pulse sender and destination, built from the authenticated
payload's alert ID and event (`{{.ID}}` and `{{.Event}}` in your template); keep
it across retries and receiver restarts. An `alert` and its later `resolved`
are different events, not duplicates. The `X-Pulse-Event-ID` header is a useful
using the [receiver rules below](#receiver-correlation-and-deduplication),
not the signature, send timestamp or alert ID alone. An `alert` and its later
`resolved` are different events, not duplicates. The `X-Pulse-Event-ID` header is a useful
correlation hint, but is **not covered by this HMAC**: do not let a changed
header bypass deduplication of an otherwise identical signed event. The sample
PSA payloads below include `alertId` and `event` for this purpose. A valid HMAC
PSA payload below includes each member's occurrence identity for this purpose. A valid HMAC
establishes integrity, not that processing succeeded; retain your receiver's
normal durable processing and retry handling. Never log the secret or full
credential-bearing request while diagnosing a rejection.
@@ -153,6 +164,44 @@ Content-Type: application/json
}
```
### Receiver correlation and deduplication
Keep ticket correlation separate from suppressing repeated processing:
- Scope durable receiver records to the **configured sender and destination**
and tenant, not just a resource name or alert ID. Read identity from the
authenticated payload's fields after verifying the request; retain it across
receiver restarts. The tenant field is context, not permission to act in an
arbitrary account.
- For normal queued firing and recovery notifications, use each member of
`{{.Alerts}}`, not only the primary `{{.ID}}`. The [full PSA template below](#sample-psa-payloads)
includes each member's `alertId` and full-precision UTC `startedAt`. These
identify its observed occurrence; a later start with the same ID must be
processed as a new occurrence. Firing and recovery use the same occurrence
record, so an old delayed recovery must not close a newer incident.
- Within that record, handle `event` **and severity**. A warning becoming
critical is an update, not a duplicate warning. Decide explicitly whether
reminders update the existing ticket; do not create another ticket for each
retry or discard a severity increase. Group membership can change without
changing the header: process every member, even when the primary is unchanged.
- The primary `{{.StartTime}}` string has only whole-second precision. Member
`StartTime` values in `{{.Alerts}}` are Go times and can be formatted with
fractional seconds as below. Do not assume the shorter string is universally
unique. A test or legacy payload can lack member identity; keep incomplete or
ambiguous events for reconciliation rather than silently treating them as
duplicates. **Send test** alone does not validate this lifecycle.
- Commit receiver state and the ticket action atomically, or use your ticket
system's durable idempotent operation. Do not mark an event processed before
its action succeeds. Exact signed-byte replays can be rejected separately,
but a queue retry may render a new body or send timestamp: hashing those alone
is not logical-event deduplication. There is no universal exactly-once key in
this legacy header.
Validate in an authorised test environment with firing, retry, warning-to-critical,
recovery, recurrence of the same condition and a changed multi-alert group.
Check actual tickets and receiver restart behaviour before enabling automatic
actions. Do not cause a production outage or notification storm for this test.
## 🛡️ Security
- **Private IPs**: By default, webhooks to private IPs are blocked. Allow them in **Settings → System → Network → Webhook Security**.
@@ -204,7 +253,14 @@ Suspended or pending-deletion organizations return `403`, and an unknown org ID
There are two integration models. The push model is usually the right fit when tickets or incidents should open and close automatically.
**Push (recommended): one outbound webhook per organization.** Create a **Generic** webhook for each organization and point it at your external system's inbound endpoint (an ITSM/PSA inbound webhook, an email connector, or middleware that opens service tickets). Shape the JSON with a [custom template](#-custom-templates) so it matches the receiving system's expected schema: every template variable listed above is available. Pulse fires on both `alert` and `resolved` events (`{{.Event}}` is `"alert"` or `"resolved"`), so the receiving system can open a ticket on alert and auto-resolve it on recovery. Add authentication as a custom header (e.g. `Authorization: Bearer ...`).
**Push (recommended): one outbound webhook per organization.** Create a
**Generic** webhook for each organization and point it at your external system's
inbound endpoint (an ITSM/PSA inbound webhook, an email connector, or middleware
that opens service tickets). Shape the JSON with a [custom template](#-custom-templates)
to match the receiving system's schema. The bridge can open or update a ticket
on `alert` and resolve that occurrence on `resolved`, subject to the [delivery
conditions above](#-delivery-contract). Add authentication as a custom header
in the settings form; keep its value private.
Configure it from the UI (**Alerts → Notifications → Add Webhook**) per org, or programmatically with an org-bound admin token:
@@ -221,20 +277,25 @@ Content-Type: application/json
"service": "generic",
"enabled": true,
"headers": { "Authorization": "Bearer <psa-token>" },
"template": "{\"summary\":\"{{.Level}}: {{.ResourceName}} {{.Message | jsonString}}\",\"event\":\"{{.Event}}\",\"alertId\":\"{{.ID}}\"}"
"template": "{\"summary\":\"{{.Level | jsonString}}: {{.ResourceName | jsonString}} {{.Message | jsonString}}\",\"event\":\"{{.Event | jsonString}}\",\"alertId\":\"{{.ID | jsonString}}\",\"startedAt\":\"{{.StartTime | jsonString}}\"}"
}
```
The exact ticket fields differ by platform (ConnectWise, Autotask, Halo, and others each expect their own inbound shape), so map the template to your platform's contract. The `alertId` round-trips through `{{.ID}}`, which lets the receiving system correlate the later `resolved` event to the ticket it opened.
The exact ticket fields differ by platform (ConnectWise, Autotask, Halo, and
others each expect their own inbound shape), so map the template to your
platform's contract. This short example carries only the primary alert and a
whole-second start. For a bridge handling real grouped notifications, use the
member-aware template below; do not use the short example as an exactly-once key.
### Sample PSA payloads
A fuller template suited to ticket routing, including tenant identity and the
stable severity/type fields from the [delivery contract](#-delivery-contract):
A fuller template for normal queued firing and recovery notifications includes
tenant context and every member's occurrence, severity and condition. The
primary fields remain convenient summary context, **not the whole batch**:
```json
{
"event": "{{.Event}}",
"event": "{{.Event | jsonString}}",
"alertId": "{{.ID | jsonString}}",
"messageKey": "{{.MessageKey | jsonString}}",
"severity": "{{.Level | jsonString}}",
@@ -248,11 +309,24 @@ stable severity/type fields from the [delivery contract](#-delivery-contract):
"value": {{.Value}},
"threshold": {{.Threshold}},
"startedAt": "{{.StartTime | jsonString}}",
"duration": "{{.Duration | jsonString}}"
"duration": "{{.Duration | jsonString}}",
"alertCount": {{.AlertCount}},
"alerts": [{{$comma := ""}}{{range .Alerts}}{{if .}}{{$comma}}
{
"alertId": "{{.ID | jsonString}}",
"startedAt": "{{.StartTime.UTC.Format "2006-01-02T15:04:05.999999999Z07:00" | jsonString}}",
"severity": "{{.Level | jsonString}}",
"alertType": "{{.Type | jsonString}}",
"resourceId": "{{.ResourceID | jsonString}}",
"resource": "{{.ResourceName | jsonString}}",
"summary": "{{.Message | jsonString}}"
}{{$comma = ","}}{{end}}{{end}}
]
}
```
What the receiver sees for a **critical** alert:
An abridged primary summary for a **critical** alert (the template also emits
the `alerts` member array):
```json
{
@@ -272,9 +346,12 @@ What the receiver sees for a **critical** alert:
}
```
A **warning** alert is identical except `"severity": "warning"` — warning and critical are the only two severities Pulse emits, so a two-priority PSA mapping covers the full range.
A **warning** alert is identical except `"severity": "warning"`. Map
informational levels too; a two-priority mapping does not cover every condition.
The **resolved** event reuses the same `alertId`, letting the bridge close the ticket it opened:
The **resolved** event retains the original member `alertId` and `startedAt`,
letting the bridge close that occurrence's ticket rather than a later incident
with the same ID. Its primary summary is:
```json
{
@@ -294,7 +371,12 @@ The **resolved** event reuses the same `alertId`, letting the bridge close the t
}
```
For ConnectWise specifically, point the webhook at a ConnectWise inbound API callback (or middleware that calls the ConnectWise REST API) and map `severity` to ticket priority, `tenantName` to the company, and `alertId` to your correlation field. Combine with a [`signingSecret`](#-delivery-contract) and the `X-Pulse-Event-ID` dedup header for a production-grade bridge.
For ConnectWise specifically, point the webhook at a ConnectWise inbound API
callback (or middleware that calls the ConnectWise REST API). Map each member's
severity to ticket priority and the tenant to your configured company. Combine
[`signingSecret`](#-delivery-contract) with the [receiver correlation rules](#receiver-correlation-and-deduplication),
not header-only deduplication. Verify your platform's processing and recovery
behaviour before treating the bridge as production-ready.
**Pull (poll): org-scoped read API.** Issue a `monitoring:read` token bound to each organization and poll that org's alerts. Send `X-Pulse-Org-ID` (or rely on the org-bound token) so you get only that organization's data:
frontend-modern/browser-tests/docs-fragment-navigation.tsx
+1
View File
@@ -25,6 +25,7 @@ const targets: Record<string, string> = {
truenas: '/docs/TRUENAS',
installation: '/docs/INSTALL',
configuration: '/docs/CONFIGURATION#-api-tokens',
webhooks: '/docs/WEBHOOKS',
};
window.history.replaceState({}, '', targets[scenario] ?? targets.plain);
frontend-modern/browser-tests/webhook-guidance.cjs
+95
View File
@@ -0,0 +1,95 @@
// Inspect the real shipped Docs page and copyable webhook template. This is
// offline rendering, not a ticket receiver, notification or external delivery.
const assert = require('node:assert/strict');
const fs = require('node:fs');
const path = require('node:path');
const crypto = require('node:crypto');
const { chromium, webkit } = require('playwright');
(async () => {
const root = '/workspace/frontend-modern';
const artifacts = path.join(root, 'node_modules', 'webhook-guidance-browser');
fs.mkdirSync(artifacts, { recursive: true });
const { createServer } = await import(path.join(root, 'node_modules/vite/dist/node/index.js'));
const server = await createServer({ root, configFile: path.join(root, 'vite.config.ts'),
cacheDir: path.join(root, 'node_modules', '.vite-webhook-guidance'),
server: { host: '127.0.0.1', port: 5253, strictPort: true },
optimizeDeps: { noDiscovery: true, include: [] } });
const docs = fs.readFileSync(path.join(root, 'public/docs/WEBHOOKS.md'), 'utf8');
const expectedTemplate = docs.split('### Sample PSA payloads')[1].match(/```json\n([\s\S]*?)\n```/)[1];
let browser;
const results = [];
try {
await server.listen();
for (const [engine, width] of [['chromium', 1280], ['webkit', 390]]) {
browser = await { chromium, webkit }[engine].launch(engine === 'chromium'
? { headless: true, channel: 'chromium', args: ['--no-sandbox'] } : { headless: true });
const page = await browser.newPage({ viewport: { width, height: 1000 },
...(engine === 'webkit' ? { isMobile: true, hasTouch: true } : {}) });
const errors = [];
page.on('pageerror', (error) => errors.push(error.message));
page.on('console', (message) => { if (message.type() === 'error') errors.push(message.text()); });
await page.goto('http://127.0.0.1:5253/browser-tests/docs-fragment-navigation.html?scenario=webhooks',
{ waitUntil: 'domcontentloaded', timeout: 60_000 });
const heading = page.getByRole('heading', { name: 'Receiver correlation and deduplication', exact: true });
await heading.waitFor();
if (engine === 'webkit') await page.evaluate(() => document.documentElement.classList.add('dark'));
const body = (await page.locator('article').innerText()).replace(/\s+/g, ' ');
for (const text of ['not a unique incident or delivery ID', 'Do not deduplicate permanently',
'old delayed recovery must not close a newer incident', 'event and severity',
'whole-second precision', 'incomplete or ambiguous events',
'successful firing-delivery receipt', 'informational conditions can use "info"']) {
assert.ok(body.includes(text), text);
}
assert.ok(!body.includes('deduplicate on it'));
const follow = async (name, href, target) => {
const link = page.getByRole('link', { name, exact: true });
assert.equal(await link.getAttribute('href'), href);
await link.focus();
await page.keyboard.press('Enter');
await page.waitForURL(`**${href}`);
await page.waitForFunction((id) => document.activeElement?.id === id, target);
};
await follow('receiver correlation and deduplication', '#receiver-correlation-and-deduplication',
'receiver-correlation-and-deduplication');
const screenshot = async (name, headingName) => {
await page.getByRole('heading', { name: headingName, exact: true })
.evaluate((element) => element.scrollIntoView({ block: 'start' }));
await page.evaluate(async () => {
await new Promise((resolve) => requestAnimationFrame(() => requestAnimationFrame(resolve)));
await Promise.all(document.getAnimations().map((animation) => animation.finished.catch(() => {})));
});
assert.ok(await page.evaluate(() => document.documentElement.scrollWidth <= innerWidth + 1));
await page.screenshot({ path: path.join(artifacts, `${engine}-${name}.png`) });
};
await screenshot('receiver', 'Receiver correlation and deduplication');
await follow('full PSA template below', '#sample-psa-payloads', 'sample-psa-payloads');
const template = page.locator('pre code').filter({ hasText: '"alertCount":' });
assert.equal(await template.count(), 1);
assert.equal((await template.innerText()).trim(), expectedTemplate.trim());
assert.ok(await template.evaluate((element) => {
const pre = element.closest('pre');
return getComputedStyle(pre).overflowX === 'auto' || pre.scrollWidth <= pre.clientWidth + 1;
}));
await screenshot('template', 'Sample PSA payloads');
await page.reload({ waitUntil: 'domcontentloaded' });
await page.getByRole('heading', { name: 'Sample PSA payloads', exact: true }).waitFor();
await page.waitForFunction(() => document.activeElement?.id === 'sample-psa-payloads');
assert.deepEqual(errors, []);
results.push({ engine, browserVersion: browser.version(), width,
scope: engine === 'webkit' ? 'phone-emulated, not native' : 'desktop',
warningsRendered: true, keyboardReceiverLink: true, keyboardTemplateLink: true,
templateTextExact: true, reloadFragmentFocus: true, noDocumentOverflow: true, errors });
await browser.close();
browser = undefined;
}
const files = ['public/docs/WEBHOOKS.md', 'browser-tests/docs-fragment-navigation.tsx',
'browser-tests/webhook-guidance.cjs', 'src/features/docs/docMarkdown.ts', 'src/pages/Docs.tsx']
.map((file) => ({ file, sha256: crypto.createHash('sha256').update(fs.readFileSync(path.join(root, file))).digest('hex') }));
fs.writeFileSync(path.join(artifacts, 'result.json'), JSON.stringify({ playwright: require('playwright/package.json').version,
files, results, limits: 'No real receiver, ticket, provider, native device or installed notification acceptance.' }, null, 2));
} finally {
if (browser) await browser.close();
await server.close();
}
})().catch((error) => { console.error(error); process.exitCode = 1; });
frontend-modern/public/docs/WEBHOOKS.md
+103 -21
View File
@@ -65,11 +65,18 @@ For generic webhooks, use Go templates to format the JSON payload.
These fields and behaviors are stable; ticket-routing integrations can rely on them.
**Events.** Every webhook fires on both `alert` and `resolved` events. `{{.Event}}` is `"alert"` or `"resolved"` — there is no separate "info" event class.
**Events.** `{{.Event}}` is `"alert"` or `"resolved"` — there is no separate
"info" event class. Recovery delivery depends on **Notify on resolve** and a
successful firing-delivery receipt for that occurrence and destination. Do not
assume that every configured webhook receives both events.
**Severity.** `{{.Level}}` is `"warning"` or `"critical"`. Pulse has exactly these two alert levels.
**Severity.** Metric alerts commonly use `"warning"` or `"critical"`;
informational conditions can use `"info"`. Event and severity are separate:
an informational firing still has event `"alert"`. Preserve and flag an
unrecognised level rather than silently dropping the notification. Minimum
severity and other delivery policies still apply.
**Alert type.** `{{.Type}}` is the metric or condition that fired: `cpu`, `memory`, `disk`, `diskRead`, `diskWrite`, `networkIn`, `networkOut`, `connectivity`, and similar. The alert ID (`{{.ID}}`) is stable for the lifetime of an alert occurrence, so the `resolved` event carries the same ID as the `alert` event it closes.
**Alert type.** `{{.Type}}` is the metric or condition that fired: `cpu`, `memory`, `disk`, `diskRead`, `diskWrite`, `networkIn`, `networkOut`, `connectivity`, and similar. The alert ID (`{{.ID}}`) correlates firing and recovery, but can be reused when the same condition fires again. It is not a unique incident or delivery ID.
**Message key.** `{{.MessageKey}}` is the stable, language-neutral condition key for rebuilding or translating a notification. Canonical alerts use `<kind>.<type>` (for example, `metric-threshold.disk`); older alert paths fall back to `{{.Type}}`. Combine it with `{{.Event}}` and `{{.ResourceType}}` when the receiving system needs separate wording for firing/recovery events or different resource classes. Unlike a numeric message index, the symbolic key does not change when another alert type is added.
@@ -79,7 +86,13 @@ These fields and behaviors are stable; ticket-routing integrations can rely on t
**Retries.** Failed deliveries retry with exponential backoff. The persistent notification queue makes up to 3 delivery attempts per notification; webhooks configured with transport-level retry add up to 3 more HTTP retries per attempt (1s doubling to a 30s cap, honoring `Retry-After` on HTTP 429). A receiver may therefore see the same logical event more than once.
**Idempotency.** Every alert delivery carries an `X-Pulse-Event-ID` header of the form `<alertID>:<event>` (e.g. `a1b2c3:alert`, `a1b2c3:resolved`). It is identical across all retries of the same logical event — deduplicate on it.
**Correlation header.** Alert webhooks carry `X-Pulse-Event-ID` in the form
`<alertID>:<event>` (e.g. `a1b2c3:alert`, `a1b2c3:resolved`). Retries retain it,
but later occurrences, severity changes and reminders can share it too. A group
uses its primary alert's ID, not a unique ID for every member or batch. **Do not
deduplicate permanently on this header or on alert ID and event alone:** doing
so can silently discard a later incident or a changed group. See
[receiver correlation and deduplication](#receiver-correlation-and-deduplication).
**Signed deliveries.** Set a `signingSecret` on the webhook config to enable HMAC signing. Signed requests carry:
@@ -126,14 +139,12 @@ def verify(secret: str, timestamp: str, body: bytes, signature: str) -> bool:
A time window rejects old captures, **not every replay**: the same signed
request can arrive more than once within that window, and Pulse signs each
retry with its current send time. After verification, deduplicate atomically
before creating a ticket or taking another action. Use a durable key scoped to
the configured Pulse sender and destination, built from the authenticated
payload's alert ID and event (`{{.ID}}` and `{{.Event}}` in your template); keep
it across retries and receiver restarts. An `alert` and its later `resolved`
are different events, not duplicates. The `X-Pulse-Event-ID` header is a useful
using the [receiver rules below](#receiver-correlation-and-deduplication),
not the signature, send timestamp or alert ID alone. An `alert` and its later
`resolved` are different events, not duplicates. The `X-Pulse-Event-ID` header is a useful
correlation hint, but is **not covered by this HMAC**: do not let a changed
header bypass deduplication of an otherwise identical signed event. The sample
PSA payloads below include `alertId` and `event` for this purpose. A valid HMAC
PSA payload below includes each member's occurrence identity for this purpose. A valid HMAC
establishes integrity, not that processing succeeded; retain your receiver's
normal durable processing and retry handling. Never log the secret or full
credential-bearing request while diagnosing a rejection.
@@ -153,6 +164,44 @@ Content-Type: application/json
}
```
### Receiver correlation and deduplication
Keep ticket correlation separate from suppressing repeated processing:
- Scope durable receiver records to the **configured sender and destination**
and tenant, not just a resource name or alert ID. Read identity from the
authenticated payload's fields after verifying the request; retain it across
receiver restarts. The tenant field is context, not permission to act in an
arbitrary account.
- For normal queued firing and recovery notifications, use each member of
`{{.Alerts}}`, not only the primary `{{.ID}}`. The [full PSA template below](#sample-psa-payloads)
includes each member's `alertId` and full-precision UTC `startedAt`. These
identify its observed occurrence; a later start with the same ID must be
processed as a new occurrence. Firing and recovery use the same occurrence
record, so an old delayed recovery must not close a newer incident.
- Within that record, handle `event` **and severity**. A warning becoming
critical is an update, not a duplicate warning. Decide explicitly whether
reminders update the existing ticket; do not create another ticket for each
retry or discard a severity increase. Group membership can change without
changing the header: process every member, even when the primary is unchanged.
- The primary `{{.StartTime}}` string has only whole-second precision. Member
`StartTime` values in `{{.Alerts}}` are Go times and can be formatted with
fractional seconds as below. Do not assume the shorter string is universally
unique. A test or legacy payload can lack member identity; keep incomplete or
ambiguous events for reconciliation rather than silently treating them as
duplicates. **Send test** alone does not validate this lifecycle.
- Commit receiver state and the ticket action atomically, or use your ticket
system's durable idempotent operation. Do not mark an event processed before
its action succeeds. Exact signed-byte replays can be rejected separately,
but a queue retry may render a new body or send timestamp: hashing those alone
is not logical-event deduplication. There is no universal exactly-once key in
this legacy header.
Validate in an authorised test environment with firing, retry, warning-to-critical,
recovery, recurrence of the same condition and a changed multi-alert group.
Check actual tickets and receiver restart behaviour before enabling automatic
actions. Do not cause a production outage or notification storm for this test.
## 🛡️ Security
- **Private IPs**: By default, webhooks to private IPs are blocked. Allow them in **Settings → System → Network → Webhook Security**.
@@ -204,7 +253,14 @@ Suspended or pending-deletion organizations return `403`, and an unknown org ID
There are two integration models. The push model is usually the right fit when tickets or incidents should open and close automatically.
**Push (recommended): one outbound webhook per organization.** Create a **Generic** webhook for each organization and point it at your external system's inbound endpoint (an ITSM/PSA inbound webhook, an email connector, or middleware that opens service tickets). Shape the JSON with a [custom template](#-custom-templates) so it matches the receiving system's expected schema: every template variable listed above is available. Pulse fires on both `alert` and `resolved` events (`{{.Event}}` is `"alert"` or `"resolved"`), so the receiving system can open a ticket on alert and auto-resolve it on recovery. Add authentication as a custom header (e.g. `Authorization: Bearer ...`).
**Push (recommended): one outbound webhook per organization.** Create a
**Generic** webhook for each organization and point it at your external system's
inbound endpoint (an ITSM/PSA inbound webhook, an email connector, or middleware
that opens service tickets). Shape the JSON with a [custom template](#-custom-templates)
to match the receiving system's schema. The bridge can open or update a ticket
on `alert` and resolve that occurrence on `resolved`, subject to the [delivery
conditions above](#-delivery-contract). Add authentication as a custom header
in the settings form; keep its value private.
Configure it from the UI (**Alerts → Notifications → Add Webhook**) per org, or programmatically with an org-bound admin token:
@@ -221,20 +277,25 @@ Content-Type: application/json
"service": "generic",
"enabled": true,
"headers": { "Authorization": "Bearer <psa-token>" },
"template": "{\"summary\":\"{{.Level}}: {{.ResourceName}} {{.Message | jsonString}}\",\"event\":\"{{.Event}}\",\"alertId\":\"{{.ID}}\"}"
"template": "{\"summary\":\"{{.Level | jsonString}}: {{.ResourceName | jsonString}} {{.Message | jsonString}}\",\"event\":\"{{.Event | jsonString}}\",\"alertId\":\"{{.ID | jsonString}}\",\"startedAt\":\"{{.StartTime | jsonString}}\"}"
}
```
The exact ticket fields differ by platform (ConnectWise, Autotask, Halo, and others each expect their own inbound shape), so map the template to your platform's contract. The `alertId` round-trips through `{{.ID}}`, which lets the receiving system correlate the later `resolved` event to the ticket it opened.
The exact ticket fields differ by platform (ConnectWise, Autotask, Halo, and
others each expect their own inbound shape), so map the template to your
platform's contract. This short example carries only the primary alert and a
whole-second start. For a bridge handling real grouped notifications, use the
member-aware template below; do not use the short example as an exactly-once key.
### Sample PSA payloads
A fuller template suited to ticket routing, including tenant identity and the
stable severity/type fields from the [delivery contract](#-delivery-contract):
A fuller template for normal queued firing and recovery notifications includes
tenant context and every member's occurrence, severity and condition. The
primary fields remain convenient summary context, **not the whole batch**:
```json
{
"event": "{{.Event}}",
"event": "{{.Event | jsonString}}",
"alertId": "{{.ID | jsonString}}",
"messageKey": "{{.MessageKey | jsonString}}",
"severity": "{{.Level | jsonString}}",
@@ -248,11 +309,24 @@ stable severity/type fields from the [delivery contract](#-delivery-contract):
"value": {{.Value}},
"threshold": {{.Threshold}},
"startedAt": "{{.StartTime | jsonString}}",
"duration": "{{.Duration | jsonString}}"
"duration": "{{.Duration | jsonString}}",
"alertCount": {{.AlertCount}},
"alerts": [{{$comma := ""}}{{range .Alerts}}{{if .}}{{$comma}}
{
"alertId": "{{.ID | jsonString}}",
"startedAt": "{{.StartTime.UTC.Format "2006-01-02T15:04:05.999999999Z07:00" | jsonString}}",
"severity": "{{.Level | jsonString}}",
"alertType": "{{.Type | jsonString}}",
"resourceId": "{{.ResourceID | jsonString}}",
"resource": "{{.ResourceName | jsonString}}",
"summary": "{{.Message | jsonString}}"
}{{$comma = ","}}{{end}}{{end}}
]
}
```
What the receiver sees for a **critical** alert:
An abridged primary summary for a **critical** alert (the template also emits
the `alerts` member array):
```json
{
@@ -272,9 +346,12 @@ What the receiver sees for a **critical** alert:
}
```
A **warning** alert is identical except `"severity": "warning"` — warning and critical are the only two severities Pulse emits, so a two-priority PSA mapping covers the full range.
A **warning** alert is identical except `"severity": "warning"`. Map
informational levels too; a two-priority mapping does not cover every condition.
The **resolved** event reuses the same `alertId`, letting the bridge close the ticket it opened:
The **resolved** event retains the original member `alertId` and `startedAt`,
letting the bridge close that occurrence's ticket rather than a later incident
with the same ID. Its primary summary is:
```json
{
@@ -294,7 +371,12 @@ The **resolved** event reuses the same `alertId`, letting the bridge close the t
}
```
For ConnectWise specifically, point the webhook at a ConnectWise inbound API callback (or middleware that calls the ConnectWise REST API) and map `severity` to ticket priority, `tenantName` to the company, and `alertId` to your correlation field. Combine with a [`signingSecret`](#-delivery-contract) and the `X-Pulse-Event-ID` dedup header for a production-grade bridge.
For ConnectWise specifically, point the webhook at a ConnectWise inbound API
callback (or middleware that calls the ConnectWise REST API). Map each member's
severity to ticket priority and the tenant to your configured company. Combine
[`signingSecret`](#-delivery-contract) with the [receiver correlation rules](#receiver-correlation-and-deduplication),
not header-only deduplication. Verify your platform's processing and recovery
behaviour before treating the bridge as production-ready.
**Pull (poll): org-scoped read API.** Issue a `monitoring:read` token bound to each organization and poll that org's alerts. Send `X-Pulse-Org-ID` (or rely on the org-bound token) so you get only that organization's data:
internal/notifications/notifications.go
+4 -4
View File
@@ -280,7 +280,7 @@ type webhookHTTPResult struct {
type webhookRequestOptions struct {
alertType string
eventID string // Idempotency token sent as X-Pulse-Event-ID (stable across retries)
eventID string // Legacy X-Pulse-Event-ID correlation header, not a unique delivery ID
timeout time.Duration
userAgent string
responseLogging bool
@@ -3344,9 +3344,9 @@ func signWebhookPayload(secret, timestamp string, payload []byte) string {
return hex.EncodeToString(mac.Sum(nil))
}
// webhookEventID builds the idempotency token for a delivery: stable for the
// same alert occurrence and event so receivers can deduplicate retries at
// either retry layer (transport or queue).
// webhookEventID builds the legacy correlation header, stable across retries.
// It omits occurrence start, severity and grouped membership, so it is not a
// unique incident/delivery identity or a sufficient durable deduplication key.
func webhookEventID(alertID, event string) string {
if alertID == "" {
return ""
internal/notifications/webhook_guide_test.go
+193
View File
@@ -0,0 +1,193 @@
package notifications
import (
"crypto/hmac"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"io"
"net/http"
"os"
"regexp"
"strings"
"testing"
"time"
"github.com/rcourtman/pulse-go-rewrite/internal/alerts"
"github.com/stretchr/testify/require"
)
// Exercise the copyable templates through the real sender, not a second
// renderer or a model receiver. The input override permits an exact-parent
// documentation negative control without modifying production source.
func webhookGuide(t *testing.T) string {
t.Helper()
path := os.Getenv("PULSE_WEBHOOK_GUIDE_TEST_INPUT")
if path == "" {
path = "../../docs/WEBHOOKS.md"
}
data, err := os.ReadFile(path)
require.NoError(t, err)
return string(data)
}
func webhookGuidePSATemplate(t *testing.T) string {
t.Helper()
sections := strings.SplitN(webhookGuide(t), "### Sample PSA payloads", 2)
require.Len(t, sections, 2)
block := regexp.MustCompile("(?s)```json\\n(.*?)\\n```").FindStringSubmatch(sections[1])
require.Len(t, block, 2)
return block[1]
}
type guideMember struct {
ID string `json:"alertId"`
StartedAt string `json:"startedAt"`
Severity string `json:"severity"`
ResourceID string `json:"resourceId"`
Resource string `json:"resource"`
Summary string `json:"summary"`
}
type guidePayload struct {
Event string `json:"event"`
TenantID string `json:"tenantId"`
StartedAt string `json:"startedAt"`
AlertCount int `json:"alertCount"`
Members []guideMember `json:"alerts"`
}
type guideRequest struct {
body, eventID, timestamp, signature string
}
func webhookGuideSender(t *testing.T, payloadTemplate string) (*NotificationManager, WebhookConfig, <-chan guideRequest) {
t.Helper()
manager := NewNotificationManager("https://pulse.example.test")
require.NoError(t, manager.UpdateAllowedPrivateCIDRs("127.0.0.1"))
manager.SetTenantIdentityResolver(func() (string, string) { return "synthetic-tenant", "Synthetic tenant" })
requests := make(chan guideRequest, 16)
server := newIPv4HTTPServer(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
body, err := io.ReadAll(r.Body)
if err != nil {
w.WriteHeader(http.StatusBadRequest)
return
}
requests <- guideRequest{string(body), r.Header.Get("X-Pulse-Event-ID"),
r.Header.Get("X-Pulse-Timestamp"), r.Header.Get("X-Pulse-Signature")}
w.WriteHeader(http.StatusOK)
}))
t.Cleanup(server.Close)
webhook := WebhookConfig{ID: "synthetic-guide", Name: "Synthetic guide receiver", URL: server.URL,
Enabled: true, Service: "generic", SigningSecret: "synthetic-guide-test-secret", Template: payloadTemplate}
return manager, webhook, requests
}
func webhookGuideRead(t *testing.T, requests <-chan guideRequest) (guideRequest, guidePayload) {
t.Helper()
select {
case request := <-requests:
var payload guidePayload
require.NoError(t, json.Unmarshal([]byte(request.body), &payload), "copied template must render valid JSON")
mac := hmac.New(sha256.New, []byte("synthetic-guide-test-secret"))
_, err := mac.Write([]byte(request.timestamp + "." + request.body))
require.NoError(t, err)
require.Equal(t, "v1="+hex.EncodeToString(mac.Sum(nil)), request.signature)
return request, payload
case <-time.After(5 * time.Second):
t.Fatal("no local sender request received")
return guideRequest{}, guidePayload{}
}
}
func TestWebhookGuidePSALifecycle(t *testing.T) {
manager, webhook, requests := webhookGuideSender(t, webhookGuidePSATemplate(t))
start := time.Date(2026, 10, 4, 7, 0, 0, 123456789, time.UTC)
alert := &alerts.Alert{ID: "host::metric-threshold:cpu", StartTime: start,
Level: alerts.AlertLevelWarning, Type: "cpu", ResourceID: "synthetic-host",
ResourceName: "quoted \"host\"\nwith é", Message: "synthetic \\ message\n\"quoted\"", Value: 95, Threshold: 90}
require.NoError(t, manager.sendGroupedWebhook(webhook, []*alerts.Alert{alert}))
firstRequest, first := webhookGuideRead(t, requests)
require.Len(t, first.Members, 1)
require.Equal(t, "alert", first.Event)
require.Equal(t, "synthetic-tenant", first.TenantID)
require.Equal(t, 1, first.AlertCount)
require.Equal(t, guideMember{alert.ID, start.Format(time.RFC3339Nano), "warning",
alert.ResourceID, alert.ResourceName, alert.Message}, first.Members[0])
// A synthetic retry retains the member identity even when rendering anew.
require.NoError(t, manager.sendGroupedWebhook(webhook, []*alerts.Alert{alert}))
retryRequest, retry := webhookGuideRead(t, requests)
require.Equal(t, first.Members, retry.Members)
require.Equal(t, firstRequest.eventID, retryRequest.eventID)
critical := alert.Clone()
critical.Level = alerts.AlertLevelCritical
require.NoError(t, manager.sendGroupedWebhook(webhook, []*alerts.Alert{critical}))
criticalRequest, escalated := webhookGuideRead(t, requests)
require.Equal(t, firstRequest.eventID, criticalRequest.eventID, "header cannot identify a severity change")
require.Equal(t, first.Members[0].StartedAt, escalated.Members[0].StartedAt)
require.Equal(t, "critical", escalated.Members[0].Severity)
require.NoError(t, manager.sendResolvedWebhook(webhook, []*alerts.Alert{critical}, start.Add(time.Minute)))
_, resolved := webhookGuideRead(t, requests)
require.Equal(t, "resolved", resolved.Event)
require.Equal(t, escalated.Members, resolved.Members, "recovery must correlate to the original occurrence")
// A second occurrence in the SAME SECOND defeats both header-only and
// whole-second primary-start deduplication. Member precision preserves it.
recurrent := critical.Clone()
recurrent.StartTime = start.Add(100 * time.Nanosecond)
require.NoError(t, manager.sendGroupedWebhook(webhook, []*alerts.Alert{recurrent}))
recurrentRequest, recurrence := webhookGuideRead(t, requests)
require.Equal(t, firstRequest.eventID, recurrentRequest.eventID)
require.Equal(t, first.StartedAt, recurrence.StartedAt)
require.NotEqual(t, first.Members[0].StartedAt, recurrence.Members[0].StartedAt)
require.Equal(t, recurrent.StartTime.Format(time.RFC3339Nano), recurrence.Members[0].StartedAt)
}
func TestWebhookGuidePSAGroupMembershipAndInfo(t *testing.T) {
manager, webhook, requests := webhookGuideSender(t, webhookGuidePSATemplate(t))
start := time.Date(2026, 10, 4, 9, 0, 0, 987654321, time.FixedZone("synthetic", 2*60*60))
primary := &alerts.Alert{ID: "primary", StartTime: start, Level: alerts.AlertLevelWarning}
info := &alerts.Alert{ID: "info-member", StartTime: start.Add(time.Second), Level: alerts.AlertLevelInfo}
critical := &alerts.Alert{ID: "critical-member", StartTime: start.Add(2 * time.Second), Level: alerts.AlertLevelCritical}
for _, members := range [][]*alerts.Alert{{primary, info}, {primary, critical}, {primary, critical, info}} {
require.NoError(t, manager.sendGroupedWebhook(webhook, members))
request, payload := webhookGuideRead(t, requests)
require.Equal(t, "primary:alert", request.eventID, "changed groups can have the same correlation header")
require.Equal(t, len(members), payload.AlertCount)
require.Len(t, payload.Members, len(members))
for index, alert := range members {
require.Equal(t, alert.ID, payload.Members[index].ID)
require.Equal(t, string(alert.Level), payload.Members[index].Severity)
require.Equal(t, alert.StartTime.UTC().Format(time.RFC3339Nano), payload.Members[index].StartedAt)
}
}
require.NoError(t, manager.sendResolvedWebhook(webhook, []*alerts.Alert{primary, critical, info}, start.Add(time.Minute)))
_, resolved := webhookGuideRead(t, requests)
require.Equal(t, "resolved", resolved.Event)
require.Len(t, resolved.Members, 3)
require.Equal(t, "info", resolved.Members[2].Severity)
}
func TestWebhookGuideShortTemplateEscapesStrings(t *testing.T) {
section := strings.SplitN(webhookGuide(t), "### Sample PSA payloads", 2)[0]
block := regexp.MustCompile("(?s)```http\\nPOST /api/notifications/webhooks\\nX-Pulse-Org-ID:.*?\\n\\n(.*?)\\n```").FindStringSubmatch(section)
require.Len(t, block, 2)
var config struct {
Template string `json:"template"`
}
require.NoError(t, json.Unmarshal([]byte(block[1]), &config))
manager, webhook, requests := webhookGuideSender(t, config.Template)
alert := &alerts.Alert{ID: "quoted\"id", ResourceName: "host\"\\\nname", Message: "message\"\\\n",
StartTime: time.Date(2026, 10, 4, 7, 0, 0, 123, time.UTC), Level: alerts.AlertLevelWarning}
require.NoError(t, manager.sendGroupedWebhook(webhook, []*alerts.Alert{alert}))
request, _ := webhookGuideRead(t, requests)
var payload map[string]interface{}
require.NoError(t, json.Unmarshal([]byte(request.body), &payload))
require.Equal(t, alert.ID, payload["alertId"])
require.Equal(t, "warning: "+alert.ResourceName+" "+alert.Message, payload["summary"])
require.Equal(t, alert.StartTime.Format(time.RFC3339), payload["startedAt"])
}
scripts/tests/test_webhook_verification_docs.py
+16
View File
@@ -146,6 +146,22 @@ class WebhookVerificationDocsTest(unittest.TestCase):
"not from the request", "Never log the secret"):
self.assertIn(phrase, text)
def test_receiver_guidance_preserves_recurrences_and_all_group_members(self):
# The Go sender tests render the copied PSA templates over actual
# loopback HTTP. Keep the safety explanation beside those examples.
text = " ".join(DOC.read_text().split())
for phrase in ("not a unique incident or delivery ID", "later occurrences",
"Do not deduplicate permanently", "every member",
"old delayed recovery must not close a newer incident",
"event and severity", "whole-second precision",
"incomplete or ambiguous events", "atomically",
"Do not mark an event processed before its action succeeds"):
self.assertIn(phrase, text.replace("**", ""))
self.assertNotIn("deduplicate on it", text)
self.assertNotIn("two-priority PSA mapping covers the full range", text)
self.assertIn('"info"', text)
self.assertIn("successful firing-delivery receipt", text)
if __name__ == "__main__":
unittest.main()