gsadmin/pulse
1
0
Fork 0
mirror of https://github.com/rcourtman/Pulse.git synced 2026-10-23 20:25:56 +00:00
Code Issues Packages Projects Releases Wiki Activity

Make notification troubleshooting bounded and delivery-aware

Browse Source 
Use the existing delivery activity and failure classes before collecting raw logs. Keep both Docker streams and log-reader exits visible, warn about provider echoes and duplicate retries, and distinguish test success from real alert delivery. Exercise the copied bounded recipes and their unsafe predecessor with synthetic readers.

Change-source: pulse-maintainer
This commit is contained in:
pulse-triage[bot]
2026-10-02 00:28:59 +01:00
parent fc4deb6c17
commit 76e1aea061
3 changed files with 349 additions and 46 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/TROUBLESHOOTING.md
+97 -23
View File
@@ -250,32 +250,72 @@ A healthy heartbeat indicates Pulse monitoring-loop progress, not successful
delivery of every resource alert or external reachability of your services.
Continue checking delivery activity for destination failures.
#### Test succeeds but real alerts are missing
A test sends directly to its destination: it skips the persistent delivery queue
and is not listed in **Recent delivery activity**. It does not prove that a real
alert was generated, routed or delivered to the intended recipient.
- Open **Alerts → Notifications**. If **Notifications are paused** is shown,
configured destinations and a successful test do not enable real delivery.
Turn delivery on there only when you intend to send alerts.
- Check the affected alert, the destination's **Enabled** state, minimum alert
severity and tag filters. Review quiet hours and any mute, acknowledgement or
maintenance policy before treating an absent attempt as a transport failure.
- Use **Recent delivery activity** to correlate the original alert, destination
and absolute timestamp, including held-notification reasons. An empty window
is not proof of healthy delivery; an **unavailable** read is not an empty log.
Do not create an outage or repeat a notification storm to populate it.
#### Recover retained delivery failures
Pulse shows a delivery warning for failed or dead-lettered notifications in its
persistent queue, not for every recoverable retry. **Recent delivery activity**
includes safely redacted provider errors; completed attempts remain for 7 days
and dead-letter attempts for 30 days. Start with the failure class and timestamp:
| Failure | Check before retrying |
| --- | --- |
| Authentication | Destination credentials and account permissions, locally; never post them. |
| Rate limited | Provider limits and delivery volume; repeated tests or retries can make this worse. |
| Connectivity | DNS, firewall, proxy and reachability from the Pulse server, not just your browser. |
| TLS | Certificate trust, expiry and hostname matching; do not disable verification to diagnose it. |
| Configuration / rejected | Enabled destination, required fields and the provider's endpoint or payload requirements. |
| Server error / unknown | Destination service status and a relevant, bounded local error excerpt. |
Save the corrected destination settings and send one test; check receipt at the
intended destination. **Retry retained deliveries** gives terminal failures a
fresh retry budget, but a destination that accepted an earlier attempt may
receive a duplicate. Review the confirmation's delivery count and provider
limits before retrying. A successful test does not itself retry retained items.
Use **Dismiss retained failures** only when those deliveries should not be sent.
Dismissal clears the warning without retrying them; delivery history remains.
Neither action deletes the audit trail. Do not delete `notification_queue.db`
or audit data to clear the warning.
#### Emails not sending
- Open **Alerts → Notifications** first. Pulse shows a delivery warning when
failed or dead-lettered notifications remain in the persistent queue; a
missing queue-health read is shown as unavailable rather than healthy.
- **Recent delivery activity** appears directly below that warning. It names
the destination and affected alert, shows an absolute timestamp for timeline
correlation, and includes safely redacted provider errors. Completed attempts
remain for 7 days and dead-letter attempts remain for 30 days.
- After correcting the destination, use **Retry retained deliveries**. Use
**Dismiss retained failures** only when those deliveries should not be sent.
Both actions preserve delivery history; do not delete `notification_queue.db`
to clear the warning.
- Check SMTP settings in **Alerts → Notifications**.
- Check logs: `docker logs pulse | grep email`.
- Ensure your SMTP provider allows the connection (e.g., Gmail App Passwords).
Follow [retained-failure recovery](#recover-retained-delivery-failures) first.
Check SMTP host, port, sender, recipients, authentication and TLS settings in
**Alerts → Notifications** against your provider's requirements (some providers
require an app password). Keep passwords in the settings form, not a diagnostic
command or report. If the delivery error is insufficient, inspect
[bounded notification logs](#inspect-notification-logs) locally.
#### Webhooks failing
- Check the delivery warning in **Alerts → Notifications** and use **Send test**
after correcting the destination. Recoverable retries do not trigger the
warning; retained terminal failures do.
- If the test succeeds, use **Retry retained deliveries** to give the retained
items a fresh retry budget. Dismiss them only when delivery is no longer
wanted; neither action deletes the audit trail.
- Verify the URL is reachable from the Pulse server.
- If targeting private IPs, allow them in **Settings → System → Network → Webhook Security**.
- Check Pulse logs for HTTP status codes and response bodies.
Follow [retained-failure recovery](#recover-retained-delivery-failures) first.
Use the failure class and HTTP status to check the provider's endpoint and
payload requirements. Verify reachability from the Pulse server. For an intended
private destination, review **Settings → System → Network → Webhook Security**;
do not broadly weaken network or TLS controls just to make a test pass.
Prefer the redacted delivery error over raw provider response bodies. A provider
can echo credentials or private content in its response; do not post it wholesale
or enable debug logging just to collect it. If needed, inspect
[bounded notification logs](#inspect-notification-logs) and share only the
consequential, manually redacted error.
### TrueNAS
@@ -309,6 +349,40 @@ Continue checking delivery activity for destination failures.
## 🛠️ Advanced Diagnostics
### Inspect Notification Logs
Prefer **Recent delivery activity** in **Alerts → Notifications**. If a local log
is needed, run only the command for your deployment, on the Pulse host with an
account authorised to read its logs. For Proxmox LXC, run the systemd command
inside the Pulse container, not on the Proxmox host. Adjust the time window to
the original incident and substitute your actual service or container name
(`pulse-backend` on some older systemd installs). These examples read at most
200 records from the last 15 minutes; they do not follow the log or send a test.
```bash
# systemd / Proxmox LXC
journalctl -u pulse --since '15 minutes ago' --lines 200 --no-pager
```
```bash
# Docker
docker logs --since 15m --tail 200 pulse
```
Docker can write application logs to either stdout or stderr; inspect both.
Do not pipe the reader into `grep email`: it can miss SMTP or webhook errors
and hide a failed read behind a matching partial line. A nonzero reader exit,
access error or missing service/container is a failed read, not “no delivery
errors”. Even a successful empty read is inconclusive: the window, retained
logs or selected instance may differ.
These local excerpts are **not sanitised**. Do not post them wholesale. Share
only the relevant timestamp, method, HTTP status or SMTP error code and a
manually redacted error. Remove credentials, cookies, secret URLs, addresses
and private host or personal information, including anything echoed by the
provider. Never upload full environments, configuration, a queue database or
audit data. See [Getting Help](#-getting-help).
### Correlate Logs with Requests
For a failed HTTP API request, inspect its response in your authenticated
frontend-modern/public/docs/TROUBLESHOOTING.md
+97 -23
View File
@@ -250,32 +250,72 @@ A healthy heartbeat indicates Pulse monitoring-loop progress, not successful
delivery of every resource alert or external reachability of your services.
Continue checking delivery activity for destination failures.
#### Test succeeds but real alerts are missing
A test sends directly to its destination: it skips the persistent delivery queue
and is not listed in **Recent delivery activity**. It does not prove that a real
alert was generated, routed or delivered to the intended recipient.
- Open **Alerts → Notifications**. If **Notifications are paused** is shown,
configured destinations and a successful test do not enable real delivery.
Turn delivery on there only when you intend to send alerts.
- Check the affected alert, the destination's **Enabled** state, minimum alert
severity and tag filters. Review quiet hours and any mute, acknowledgement or
maintenance policy before treating an absent attempt as a transport failure.
- Use **Recent delivery activity** to correlate the original alert, destination
and absolute timestamp, including held-notification reasons. An empty window
is not proof of healthy delivery; an **unavailable** read is not an empty log.
Do not create an outage or repeat a notification storm to populate it.
#### Recover retained delivery failures
Pulse shows a delivery warning for failed or dead-lettered notifications in its
persistent queue, not for every recoverable retry. **Recent delivery activity**
includes safely redacted provider errors; completed attempts remain for 7 days
and dead-letter attempts for 30 days. Start with the failure class and timestamp:
| Failure | Check before retrying |
| --- | --- |
| Authentication | Destination credentials and account permissions, locally; never post them. |
| Rate limited | Provider limits and delivery volume; repeated tests or retries can make this worse. |
| Connectivity | DNS, firewall, proxy and reachability from the Pulse server, not just your browser. |
| TLS | Certificate trust, expiry and hostname matching; do not disable verification to diagnose it. |
| Configuration / rejected | Enabled destination, required fields and the provider's endpoint or payload requirements. |
| Server error / unknown | Destination service status and a relevant, bounded local error excerpt. |
Save the corrected destination settings and send one test; check receipt at the
intended destination. **Retry retained deliveries** gives terminal failures a
fresh retry budget, but a destination that accepted an earlier attempt may
receive a duplicate. Review the confirmation's delivery count and provider
limits before retrying. A successful test does not itself retry retained items.
Use **Dismiss retained failures** only when those deliveries should not be sent.
Dismissal clears the warning without retrying them; delivery history remains.
Neither action deletes the audit trail. Do not delete `notification_queue.db`
or audit data to clear the warning.
#### Emails not sending
- Open **Alerts → Notifications** first. Pulse shows a delivery warning when
failed or dead-lettered notifications remain in the persistent queue; a
missing queue-health read is shown as unavailable rather than healthy.
- **Recent delivery activity** appears directly below that warning. It names
the destination and affected alert, shows an absolute timestamp for timeline
correlation, and includes safely redacted provider errors. Completed attempts
remain for 7 days and dead-letter attempts remain for 30 days.
- After correcting the destination, use **Retry retained deliveries**. Use
**Dismiss retained failures** only when those deliveries should not be sent.
Both actions preserve delivery history; do not delete `notification_queue.db`
to clear the warning.
- Check SMTP settings in **Alerts → Notifications**.
- Check logs: `docker logs pulse | grep email`.
- Ensure your SMTP provider allows the connection (e.g., Gmail App Passwords).
Follow [retained-failure recovery](#recover-retained-delivery-failures) first.
Check SMTP host, port, sender, recipients, authentication and TLS settings in
**Alerts → Notifications** against your provider's requirements (some providers
require an app password). Keep passwords in the settings form, not a diagnostic
command or report. If the delivery error is insufficient, inspect
[bounded notification logs](#inspect-notification-logs) locally.
#### Webhooks failing
- Check the delivery warning in **Alerts → Notifications** and use **Send test**
after correcting the destination. Recoverable retries do not trigger the
warning; retained terminal failures do.
- If the test succeeds, use **Retry retained deliveries** to give the retained
items a fresh retry budget. Dismiss them only when delivery is no longer
wanted; neither action deletes the audit trail.
- Verify the URL is reachable from the Pulse server.
- If targeting private IPs, allow them in **Settings → System → Network → Webhook Security**.
- Check Pulse logs for HTTP status codes and response bodies.
Follow [retained-failure recovery](#recover-retained-delivery-failures) first.
Use the failure class and HTTP status to check the provider's endpoint and
payload requirements. Verify reachability from the Pulse server. For an intended
private destination, review **Settings → System → Network → Webhook Security**;
do not broadly weaken network or TLS controls just to make a test pass.
Prefer the redacted delivery error over raw provider response bodies. A provider
can echo credentials or private content in its response; do not post it wholesale
or enable debug logging just to collect it. If needed, inspect
[bounded notification logs](#inspect-notification-logs) and share only the
consequential, manually redacted error.
### TrueNAS
@@ -309,6 +349,40 @@ Continue checking delivery activity for destination failures.
## 🛠️ Advanced Diagnostics
### Inspect Notification Logs
Prefer **Recent delivery activity** in **Alerts → Notifications**. If a local log
is needed, run only the command for your deployment, on the Pulse host with an
account authorised to read its logs. For Proxmox LXC, run the systemd command
inside the Pulse container, not on the Proxmox host. Adjust the time window to
the original incident and substitute your actual service or container name
(`pulse-backend` on some older systemd installs). These examples read at most
200 records from the last 15 minutes; they do not follow the log or send a test.
```bash
# systemd / Proxmox LXC
journalctl -u pulse --since '15 minutes ago' --lines 200 --no-pager
```
```bash
# Docker
docker logs --since 15m --tail 200 pulse
```
Docker can write application logs to either stdout or stderr; inspect both.
Do not pipe the reader into `grep email`: it can miss SMTP or webhook errors
and hide a failed read behind a matching partial line. A nonzero reader exit,
access error or missing service/container is a failed read, not “no delivery
errors”. Even a successful empty read is inconclusive: the window, retained
logs or selected instance may differ.
These local excerpts are **not sanitised**. Do not post them wholesale. Share
only the relevant timestamp, method, HTTP status or SMTP error code and a
manually redacted error. Remove credentials, cookies, secret URLs, addresses
and private host or personal information, including anything echoed by the
provider. Never upload full environments, configuration, a queue database or
audit data. See [Getting Help](#-getting-help).
### Correlate Logs with Requests
For a failed HTTP API request, inspect its response in your authenticated
scripts/tests/test_notification_troubleshooting_docs.py
+155
View File
@@ -0,0 +1,155 @@
#!/usr/bin/env python3
"""Exercise copied notification-log recipes without a service or destination.
Synthetic readers prove bounds, stream visibility and exit handling. They do
not prove native installation, notification delivery or automatic redaction.
"""
from __future__ import annotations
import json
import os
from pathlib import Path
import re
import subprocess
import tempfile
import unittest
ROOT = Path(__file__).resolve().parents[2]
DOC = ROOT / "docs/TROUBLESHOOTING.md"
SMTP_ERROR = '{"level":"error","message":"SMTP authentication failed"}\n'
WEBHOOK_ERROR = '{"level":"error","message":"webhook returned HTTP 429"}\n'
PARTIAL_EMAIL_LOG = '{"message":"attempting to send email"}\n'
SYNTHETIC_SECRET = "synthetic-provider-echo-secret"
def section(heading: str, level: int = 3) -> str:
marker = "#" * level + " " + heading + "\n"
text = DOC.read_text(encoding="utf-8")
if marker not in text:
raise AssertionError(f"missing diagnostic section: {heading}")
# These guide sections have no later H1. Ignore the single-# shell comments
# inside recipes while stopping at the next same/higher-level section.
return re.split(r"\n#{2," + str(level) + r"} ", text.split(marker, 1)[1], 1)[0]
def recipes() -> dict[str, str]:
blocks = re.findall(r"```bash\n(.*?)```", section("Inspect Notification Logs"), re.DOTALL)
if len(blocks) != 2:
raise AssertionError("expected one recipe per log reader")
return {"docker" if "# Docker" in block else "journalctl": block for block in blocks}
READER = '''#!/usr/bin/env python3
import json, os, pathlib, sys
pathlib.Path(os.environ['READER_ARGV']).write_text(json.dumps(sys.argv[1:]))
sys.stdout.write(os.environ.get('LOG_STDOUT', ''))
sys.stderr.write(os.environ.get('LOG_STDERR', ''))
sys.exit(int(os.environ.get('READER_EXIT', '0')))
'''
class NotificationTroubleshootingDocsTest(unittest.TestCase):
def exercise(self, reader: str, *, stdout: str = "", stderr: str = "",
exit_code: int = 0, recipe: str | None = None):
with tempfile.TemporaryDirectory() as temporary:
directory = Path(temporary)
command = directory / reader
command.write_text(READER, encoding="utf-8")
command.chmod(0o700)
argv = directory / "argv.json"
env = dict(os.environ, PATH=f"{directory}:{os.environ['PATH']}",
READER_ARGV=str(argv), LOG_STDOUT=stdout, LOG_STDERR=stderr,
READER_EXIT=str(exit_code))
copied = recipe if recipe is not None else recipes()[reader]
result = subprocess.run(["bash", "-c", copied], env=env,
capture_output=True, text=True, timeout=5)
self.assertTrue(argv.exists(), "copied command did not invoke its log reader")
return result, json.loads(argv.read_text())
def test_shipped_document_matches_the_tested_source(self):
self.assertEqual(DOC.read_bytes(),
(ROOT / "frontend-modern/public/docs/TROUBLESHOOTING.md").read_bytes())
def test_copied_commands_bound_the_read_without_filtering_or_mutation(self):
expected = {
"journalctl": ["-u", "pulse", "--since", "15 minutes ago", "--lines", "200", "--no-pager"],
"docker": ["logs", "--since", "15m", "--tail", "200", "pulse"],
}
self.assertEqual(set(recipes()), set(expected))
for reader, copied in recipes().items():
with self.subTest(reader=reader):
result, argv = self.exercise(reader, stdout=SMTP_ERROR + WEBHOOK_ERROR)
self.assertEqual(result.returncode, 0, result.stderr)
self.assertEqual(argv, expected[reader])
self.assertEqual(result.stdout, SMTP_ERROR + WEBHOOK_ERROR)
self.assertNotRegex(copied, r"[|<>]|\b(?:grep|curl|inspect|restart|printenv)\b|--follow|--token")
def test_docker_application_logs_on_both_streams_remain_visible(self):
result, _ = self.exercise("docker", stdout=WEBHOOK_ERROR, stderr=SMTP_ERROR)
self.assertEqual(result.returncode, 0, result.stderr)
self.assertEqual(result.stdout, WEBHOOK_ERROR)
self.assertEqual(result.stderr, SMTP_ERROR)
def test_reader_failure_keeps_its_exit_even_after_a_partial_email_log(self):
for reader in recipes():
for output in ("", PARTIAL_EMAIL_LOG):
with self.subTest(reader=reader, partial=bool(output)):
result, _ = self.exercise(reader, stdout=output,
stderr="synthetic reader access failure\n", exit_code=2)
self.assertEqual(result.returncode, 2)
self.assertEqual(result.stdout, output)
self.assertEqual(result.stderr, "synthetic reader access failure\n")
def test_successful_empty_read_is_not_documented_as_healthy_delivery(self):
for reader in recipes():
result, _ = self.exercise(reader)
self.assertEqual(result.returncode, 0, result.stderr)
self.assertEqual(result.stdout + result.stderr, "")
self.assertIn("successful empty read is inconclusive", section("Inspect Notification Logs"))
def test_raw_provider_echo_is_not_claimed_to_be_sanitised(self):
# The command does not magically sanitise either stream. The nearby
# disclosure boundary must survive edits to the copied recipe.
for reader in recipes():
result, _ = self.exercise(reader, stderr=SYNTHETIC_SECRET + "\n")
self.assertIn(SYNTHETIC_SECRET, result.stderr)
text = " ".join(section("Inspect Notification Logs").split())
for boundary in ("not sanitised", "Do not post them wholesale", "manually redacted",
"anything echoed by the provider", "Never upload full environments",
"queue database", "inside the Pulse container", "actual service or container name"):
self.assertIn(boundary, text)
def test_legacy_email_pipeline_is_an_adverse_control(self):
legacy = "docker logs pulse | grep email"
result, argv = self.exercise("docker", stderr=SMTP_ERROR, recipe=legacy)
self.assertEqual(argv, ["logs", "pulse"])
self.assertEqual(result.returncode, 1)
self.assertEqual(result.stdout, "")
# A human can see stderr in a terminal, but the old search itself loses
# the match, encouraging collection of more history or debug output.
self.assertEqual(result.stderr, SMTP_ERROR)
result, _ = self.exercise("docker", stdout=PARTIAL_EMAIL_LOG,
stderr="synthetic reader access failure\n", exit_code=2, recipe=legacy)
self.assertEqual(result.returncode, 0, "legacy pipe must demonstrate the masked reader failure")
def test_recovery_guidance_uses_existing_ui_without_promising_delivery(self):
presentation = (ROOT / "frontend-modern/src/utils/alertDestinationsPresentation.ts").read_text()
notifications = " ".join(section("Notifications").split())
for label in ("Recent delivery activity", "Retry retained deliveries", "Dismiss retained failures"):
self.assertIn(label, presentation)
self.assertIn(label, notifications)
for boundary in ("skips the persistent delivery queue", "not listed", "Notifications are paused",
"minimum alert severity and tag filters", "held-notification reasons",
"unavailable", "may receive a duplicate", "successful test does not itself retry",
"do not disable verification", "Do not delete `notification_queue.db`",
"do not post it wholesale", "enable debug logging just to collect it"):
self.assertIn(boundary, notifications)
for heading in ("Emails not sending", "Webhooks failing"):
self.assertIn("#recover-retained-delivery-failures", section(heading, 4))
self.assertIn("#inspect-notification-logs", section(heading, 4))
if __name__ == "__main__":
unittest.main()