diff --git a/docs/TROUBLESHOOTING.md b/docs/TROUBLESHOOTING.md index d6a966caa6..55badfaa03 100644 --- a/docs/TROUBLESHOOTING.md +++ b/docs/TROUBLESHOOTING.md @@ -250,32 +250,72 @@ A healthy heartbeat indicates Pulse monitoring-loop progress, not successful delivery of every resource alert or external reachability of your services. Continue checking delivery activity for destination failures. +#### Test succeeds but real alerts are missing + +A test sends directly to its destination: it skips the persistent delivery queue +and is not listed in **Recent delivery activity**. It does not prove that a real +alert was generated, routed or delivered to the intended recipient. + +- Open **Alerts → Notifications**. If **Notifications are paused** is shown, + configured destinations and a successful test do not enable real delivery. + Turn delivery on there only when you intend to send alerts. +- Check the affected alert, the destination's **Enabled** state, minimum alert + severity and tag filters. Review quiet hours and any mute, acknowledgement or + maintenance policy before treating an absent attempt as a transport failure. +- Use **Recent delivery activity** to correlate the original alert, destination + and absolute timestamp, including held-notification reasons. An empty window + is not proof of healthy delivery; an **unavailable** read is not an empty log. + Do not create an outage or repeat a notification storm to populate it. + +#### Recover retained delivery failures + +Pulse shows a delivery warning for failed or dead-lettered notifications in its +persistent queue, not for every recoverable retry. **Recent delivery activity** +includes safely redacted provider errors; completed attempts remain for 7 days +and dead-letter attempts for 30 days. Start with the failure class and timestamp: + +| Failure | Check before retrying | +| --- | --- | +| Authentication | Destination credentials and account permissions, locally; never post them. | +| Rate limited | Provider limits and delivery volume; repeated tests or retries can make this worse. | +| Connectivity | DNS, firewall, proxy and reachability from the Pulse server, not just your browser. | +| TLS | Certificate trust, expiry and hostname matching; do not disable verification to diagnose it. | +| Configuration / rejected | Enabled destination, required fields and the provider's endpoint or payload requirements. | +| Server error / unknown | Destination service status and a relevant, bounded local error excerpt. | + +Save the corrected destination settings and send one test; check receipt at the +intended destination. **Retry retained deliveries** gives terminal failures a +fresh retry budget, but a destination that accepted an earlier attempt may +receive a duplicate. Review the confirmation's delivery count and provider +limits before retrying. A successful test does not itself retry retained items. + +Use **Dismiss retained failures** only when those deliveries should not be sent. +Dismissal clears the warning without retrying them; delivery history remains. +Neither action deletes the audit trail. Do not delete `notification_queue.db` +or audit data to clear the warning. + #### Emails not sending -- Open **Alerts → Notifications** first. Pulse shows a delivery warning when - failed or dead-lettered notifications remain in the persistent queue; a - missing queue-health read is shown as unavailable rather than healthy. -- **Recent delivery activity** appears directly below that warning. It names - the destination and affected alert, shows an absolute timestamp for timeline - correlation, and includes safely redacted provider errors. Completed attempts - remain for 7 days and dead-letter attempts remain for 30 days. -- After correcting the destination, use **Retry retained deliveries**. Use - **Dismiss retained failures** only when those deliveries should not be sent. - Both actions preserve delivery history; do not delete `notification_queue.db` - to clear the warning. -- Check SMTP settings in **Alerts → Notifications**. -- Check logs: `docker logs pulse | grep email`. -- Ensure your SMTP provider allows the connection (e.g., Gmail App Passwords). + +Follow [retained-failure recovery](#recover-retained-delivery-failures) first. +Check SMTP host, port, sender, recipients, authentication and TLS settings in +**Alerts → Notifications** against your provider's requirements (some providers +require an app password). Keep passwords in the settings form, not a diagnostic +command or report. If the delivery error is insufficient, inspect +[bounded notification logs](#inspect-notification-logs) locally. #### Webhooks failing -- Check the delivery warning in **Alerts → Notifications** and use **Send test** - after correcting the destination. Recoverable retries do not trigger the - warning; retained terminal failures do. -- If the test succeeds, use **Retry retained deliveries** to give the retained - items a fresh retry budget. Dismiss them only when delivery is no longer - wanted; neither action deletes the audit trail. -- Verify the URL is reachable from the Pulse server. -- If targeting private IPs, allow them in **Settings → System → Network → Webhook Security**. -- Check Pulse logs for HTTP status codes and response bodies. + +Follow [retained-failure recovery](#recover-retained-delivery-failures) first. +Use the failure class and HTTP status to check the provider's endpoint and +payload requirements. Verify reachability from the Pulse server. For an intended +private destination, review **Settings → System → Network → Webhook Security**; +do not broadly weaken network or TLS controls just to make a test pass. + +Prefer the redacted delivery error over raw provider response bodies. A provider +can echo credentials or private content in its response; do not post it wholesale +or enable debug logging just to collect it. If needed, inspect +[bounded notification logs](#inspect-notification-logs) and share only the +consequential, manually redacted error. ### TrueNAS @@ -309,6 +349,40 @@ Continue checking delivery activity for destination failures. ## 🛠️ Advanced Diagnostics +### Inspect Notification Logs + +Prefer **Recent delivery activity** in **Alerts → Notifications**. If a local log +is needed, run only the command for your deployment, on the Pulse host with an +account authorised to read its logs. For Proxmox LXC, run the systemd command +inside the Pulse container, not on the Proxmox host. Adjust the time window to +the original incident and substitute your actual service or container name +(`pulse-backend` on some older systemd installs). These examples read at most +200 records from the last 15 minutes; they do not follow the log or send a test. + +```bash +# systemd / Proxmox LXC +journalctl -u pulse --since '15 minutes ago' --lines 200 --no-pager +``` + +```bash +# Docker +docker logs --since 15m --tail 200 pulse +``` + +Docker can write application logs to either stdout or stderr; inspect both. +Do not pipe the reader into `grep email`: it can miss SMTP or webhook errors +and hide a failed read behind a matching partial line. A nonzero reader exit, +access error or missing service/container is a failed read, not “no delivery +errors”. Even a successful empty read is inconclusive: the window, retained +logs or selected instance may differ. + +These local excerpts are **not sanitised**. Do not post them wholesale. Share +only the relevant timestamp, method, HTTP status or SMTP error code and a +manually redacted error. Remove credentials, cookies, secret URLs, addresses +and private host or personal information, including anything echoed by the +provider. Never upload full environments, configuration, a queue database or +audit data. See [Getting Help](#-getting-help). + ### Correlate Logs with Requests For a failed HTTP API request, inspect its response in your authenticated diff --git a/frontend-modern/public/docs/TROUBLESHOOTING.md b/frontend-modern/public/docs/TROUBLESHOOTING.md index d6a966caa6..55badfaa03 100644 --- a/frontend-modern/public/docs/TROUBLESHOOTING.md +++ b/frontend-modern/public/docs/TROUBLESHOOTING.md @@ -250,32 +250,72 @@ A healthy heartbeat indicates Pulse monitoring-loop progress, not successful delivery of every resource alert or external reachability of your services. Continue checking delivery activity for destination failures. +#### Test succeeds but real alerts are missing + +A test sends directly to its destination: it skips the persistent delivery queue +and is not listed in **Recent delivery activity**. It does not prove that a real +alert was generated, routed or delivered to the intended recipient. + +- Open **Alerts → Notifications**. If **Notifications are paused** is shown, + configured destinations and a successful test do not enable real delivery. + Turn delivery on there only when you intend to send alerts. +- Check the affected alert, the destination's **Enabled** state, minimum alert + severity and tag filters. Review quiet hours and any mute, acknowledgement or + maintenance policy before treating an absent attempt as a transport failure. +- Use **Recent delivery activity** to correlate the original alert, destination + and absolute timestamp, including held-notification reasons. An empty window + is not proof of healthy delivery; an **unavailable** read is not an empty log. + Do not create an outage or repeat a notification storm to populate it. + +#### Recover retained delivery failures + +Pulse shows a delivery warning for failed or dead-lettered notifications in its +persistent queue, not for every recoverable retry. **Recent delivery activity** +includes safely redacted provider errors; completed attempts remain for 7 days +and dead-letter attempts for 30 days. Start with the failure class and timestamp: + +| Failure | Check before retrying | +| --- | --- | +| Authentication | Destination credentials and account permissions, locally; never post them. | +| Rate limited | Provider limits and delivery volume; repeated tests or retries can make this worse. | +| Connectivity | DNS, firewall, proxy and reachability from the Pulse server, not just your browser. | +| TLS | Certificate trust, expiry and hostname matching; do not disable verification to diagnose it. | +| Configuration / rejected | Enabled destination, required fields and the provider's endpoint or payload requirements. | +| Server error / unknown | Destination service status and a relevant, bounded local error excerpt. | + +Save the corrected destination settings and send one test; check receipt at the +intended destination. **Retry retained deliveries** gives terminal failures a +fresh retry budget, but a destination that accepted an earlier attempt may +receive a duplicate. Review the confirmation's delivery count and provider +limits before retrying. A successful test does not itself retry retained items. + +Use **Dismiss retained failures** only when those deliveries should not be sent. +Dismissal clears the warning without retrying them; delivery history remains. +Neither action deletes the audit trail. Do not delete `notification_queue.db` +or audit data to clear the warning. + #### Emails not sending -- Open **Alerts → Notifications** first. Pulse shows a delivery warning when - failed or dead-lettered notifications remain in the persistent queue; a - missing queue-health read is shown as unavailable rather than healthy. -- **Recent delivery activity** appears directly below that warning. It names - the destination and affected alert, shows an absolute timestamp for timeline - correlation, and includes safely redacted provider errors. Completed attempts - remain for 7 days and dead-letter attempts remain for 30 days. -- After correcting the destination, use **Retry retained deliveries**. Use - **Dismiss retained failures** only when those deliveries should not be sent. - Both actions preserve delivery history; do not delete `notification_queue.db` - to clear the warning. -- Check SMTP settings in **Alerts → Notifications**. -- Check logs: `docker logs pulse | grep email`. -- Ensure your SMTP provider allows the connection (e.g., Gmail App Passwords). + +Follow [retained-failure recovery](#recover-retained-delivery-failures) first. +Check SMTP host, port, sender, recipients, authentication and TLS settings in +**Alerts → Notifications** against your provider's requirements (some providers +require an app password). Keep passwords in the settings form, not a diagnostic +command or report. If the delivery error is insufficient, inspect +[bounded notification logs](#inspect-notification-logs) locally. #### Webhooks failing -- Check the delivery warning in **Alerts → Notifications** and use **Send test** - after correcting the destination. Recoverable retries do not trigger the - warning; retained terminal failures do. -- If the test succeeds, use **Retry retained deliveries** to give the retained - items a fresh retry budget. Dismiss them only when delivery is no longer - wanted; neither action deletes the audit trail. -- Verify the URL is reachable from the Pulse server. -- If targeting private IPs, allow them in **Settings → System → Network → Webhook Security**. -- Check Pulse logs for HTTP status codes and response bodies. + +Follow [retained-failure recovery](#recover-retained-delivery-failures) first. +Use the failure class and HTTP status to check the provider's endpoint and +payload requirements. Verify reachability from the Pulse server. For an intended +private destination, review **Settings → System → Network → Webhook Security**; +do not broadly weaken network or TLS controls just to make a test pass. + +Prefer the redacted delivery error over raw provider response bodies. A provider +can echo credentials or private content in its response; do not post it wholesale +or enable debug logging just to collect it. If needed, inspect +[bounded notification logs](#inspect-notification-logs) and share only the +consequential, manually redacted error. ### TrueNAS @@ -309,6 +349,40 @@ Continue checking delivery activity for destination failures. ## 🛠️ Advanced Diagnostics +### Inspect Notification Logs + +Prefer **Recent delivery activity** in **Alerts → Notifications**. If a local log +is needed, run only the command for your deployment, on the Pulse host with an +account authorised to read its logs. For Proxmox LXC, run the systemd command +inside the Pulse container, not on the Proxmox host. Adjust the time window to +the original incident and substitute your actual service or container name +(`pulse-backend` on some older systemd installs). These examples read at most +200 records from the last 15 minutes; they do not follow the log or send a test. + +```bash +# systemd / Proxmox LXC +journalctl -u pulse --since '15 minutes ago' --lines 200 --no-pager +``` + +```bash +# Docker +docker logs --since 15m --tail 200 pulse +``` + +Docker can write application logs to either stdout or stderr; inspect both. +Do not pipe the reader into `grep email`: it can miss SMTP or webhook errors +and hide a failed read behind a matching partial line. A nonzero reader exit, +access error or missing service/container is a failed read, not “no delivery +errors”. Even a successful empty read is inconclusive: the window, retained +logs or selected instance may differ. + +These local excerpts are **not sanitised**. Do not post them wholesale. Share +only the relevant timestamp, method, HTTP status or SMTP error code and a +manually redacted error. Remove credentials, cookies, secret URLs, addresses +and private host or personal information, including anything echoed by the +provider. Never upload full environments, configuration, a queue database or +audit data. See [Getting Help](#-getting-help). + ### Correlate Logs with Requests For a failed HTTP API request, inspect its response in your authenticated diff --git a/scripts/tests/test_notification_troubleshooting_docs.py b/scripts/tests/test_notification_troubleshooting_docs.py new file mode 100644 index 0000000000..1531b86116 --- /dev/null +++ b/scripts/tests/test_notification_troubleshooting_docs.py @@ -0,0 +1,155 @@ +#!/usr/bin/env python3 +"""Exercise copied notification-log recipes without a service or destination. + +Synthetic readers prove bounds, stream visibility and exit handling. They do +not prove native installation, notification delivery or automatic redaction. +""" + +from __future__ import annotations + +import json +import os +from pathlib import Path +import re +import subprocess +import tempfile +import unittest + + +ROOT = Path(__file__).resolve().parents[2] +DOC = ROOT / "docs/TROUBLESHOOTING.md" +SMTP_ERROR = '{"level":"error","message":"SMTP authentication failed"}\n' +WEBHOOK_ERROR = '{"level":"error","message":"webhook returned HTTP 429"}\n' +PARTIAL_EMAIL_LOG = '{"message":"attempting to send email"}\n' +SYNTHETIC_SECRET = "synthetic-provider-echo-secret" + + +def section(heading: str, level: int = 3) -> str: + marker = "#" * level + " " + heading + "\n" + text = DOC.read_text(encoding="utf-8") + if marker not in text: + raise AssertionError(f"missing diagnostic section: {heading}") + # These guide sections have no later H1. Ignore the single-# shell comments + # inside recipes while stopping at the next same/higher-level section. + return re.split(r"\n#{2," + str(level) + r"} ", text.split(marker, 1)[1], 1)[0] + + +def recipes() -> dict[str, str]: + blocks = re.findall(r"```bash\n(.*?)```", section("Inspect Notification Logs"), re.DOTALL) + if len(blocks) != 2: + raise AssertionError("expected one recipe per log reader") + return {"docker" if "# Docker" in block else "journalctl": block for block in blocks} + + +READER = '''#!/usr/bin/env python3 +import json, os, pathlib, sys +pathlib.Path(os.environ['READER_ARGV']).write_text(json.dumps(sys.argv[1:])) +sys.stdout.write(os.environ.get('LOG_STDOUT', '')) +sys.stderr.write(os.environ.get('LOG_STDERR', '')) +sys.exit(int(os.environ.get('READER_EXIT', '0'))) +''' + + +class NotificationTroubleshootingDocsTest(unittest.TestCase): + def exercise(self, reader: str, *, stdout: str = "", stderr: str = "", + exit_code: int = 0, recipe: str | None = None): + with tempfile.TemporaryDirectory() as temporary: + directory = Path(temporary) + command = directory / reader + command.write_text(READER, encoding="utf-8") + command.chmod(0o700) + argv = directory / "argv.json" + env = dict(os.environ, PATH=f"{directory}:{os.environ['PATH']}", + READER_ARGV=str(argv), LOG_STDOUT=stdout, LOG_STDERR=stderr, + READER_EXIT=str(exit_code)) + copied = recipe if recipe is not None else recipes()[reader] + result = subprocess.run(["bash", "-c", copied], env=env, + capture_output=True, text=True, timeout=5) + self.assertTrue(argv.exists(), "copied command did not invoke its log reader") + return result, json.loads(argv.read_text()) + + def test_shipped_document_matches_the_tested_source(self): + self.assertEqual(DOC.read_bytes(), + (ROOT / "frontend-modern/public/docs/TROUBLESHOOTING.md").read_bytes()) + + def test_copied_commands_bound_the_read_without_filtering_or_mutation(self): + expected = { + "journalctl": ["-u", "pulse", "--since", "15 minutes ago", "--lines", "200", "--no-pager"], + "docker": ["logs", "--since", "15m", "--tail", "200", "pulse"], + } + self.assertEqual(set(recipes()), set(expected)) + for reader, copied in recipes().items(): + with self.subTest(reader=reader): + result, argv = self.exercise(reader, stdout=SMTP_ERROR + WEBHOOK_ERROR) + self.assertEqual(result.returncode, 0, result.stderr) + self.assertEqual(argv, expected[reader]) + self.assertEqual(result.stdout, SMTP_ERROR + WEBHOOK_ERROR) + self.assertNotRegex(copied, r"[|<>]|\b(?:grep|curl|inspect|restart|printenv)\b|--follow|--token") + + def test_docker_application_logs_on_both_streams_remain_visible(self): + result, _ = self.exercise("docker", stdout=WEBHOOK_ERROR, stderr=SMTP_ERROR) + self.assertEqual(result.returncode, 0, result.stderr) + self.assertEqual(result.stdout, WEBHOOK_ERROR) + self.assertEqual(result.stderr, SMTP_ERROR) + + def test_reader_failure_keeps_its_exit_even_after_a_partial_email_log(self): + for reader in recipes(): + for output in ("", PARTIAL_EMAIL_LOG): + with self.subTest(reader=reader, partial=bool(output)): + result, _ = self.exercise(reader, stdout=output, + stderr="synthetic reader access failure\n", exit_code=2) + self.assertEqual(result.returncode, 2) + self.assertEqual(result.stdout, output) + self.assertEqual(result.stderr, "synthetic reader access failure\n") + + def test_successful_empty_read_is_not_documented_as_healthy_delivery(self): + for reader in recipes(): + result, _ = self.exercise(reader) + self.assertEqual(result.returncode, 0, result.stderr) + self.assertEqual(result.stdout + result.stderr, "") + self.assertIn("successful empty read is inconclusive", section("Inspect Notification Logs")) + + def test_raw_provider_echo_is_not_claimed_to_be_sanitised(self): + # The command does not magically sanitise either stream. The nearby + # disclosure boundary must survive edits to the copied recipe. + for reader in recipes(): + result, _ = self.exercise(reader, stderr=SYNTHETIC_SECRET + "\n") + self.assertIn(SYNTHETIC_SECRET, result.stderr) + text = " ".join(section("Inspect Notification Logs").split()) + for boundary in ("not sanitised", "Do not post them wholesale", "manually redacted", + "anything echoed by the provider", "Never upload full environments", + "queue database", "inside the Pulse container", "actual service or container name"): + self.assertIn(boundary, text) + + def test_legacy_email_pipeline_is_an_adverse_control(self): + legacy = "docker logs pulse | grep email" + result, argv = self.exercise("docker", stderr=SMTP_ERROR, recipe=legacy) + self.assertEqual(argv, ["logs", "pulse"]) + self.assertEqual(result.returncode, 1) + self.assertEqual(result.stdout, "") + # A human can see stderr in a terminal, but the old search itself loses + # the match, encouraging collection of more history or debug output. + self.assertEqual(result.stderr, SMTP_ERROR) + result, _ = self.exercise("docker", stdout=PARTIAL_EMAIL_LOG, + stderr="synthetic reader access failure\n", exit_code=2, recipe=legacy) + self.assertEqual(result.returncode, 0, "legacy pipe must demonstrate the masked reader failure") + + def test_recovery_guidance_uses_existing_ui_without_promising_delivery(self): + presentation = (ROOT / "frontend-modern/src/utils/alertDestinationsPresentation.ts").read_text() + notifications = " ".join(section("Notifications").split()) + for label in ("Recent delivery activity", "Retry retained deliveries", "Dismiss retained failures"): + self.assertIn(label, presentation) + self.assertIn(label, notifications) + for boundary in ("skips the persistent delivery queue", "not listed", "Notifications are paused", + "minimum alert severity and tag filters", "held-notification reasons", + "unavailable", "may receive a duplicate", "successful test does not itself retry", + "do not disable verification", "Do not delete `notification_queue.db`", + "do not post it wholesale", "enable debug logging just to collect it"): + self.assertIn(boundary, notifications) + for heading in ("Emails not sending", "Webhooks failing"): + self.assertIn("#recover-retained-delivery-failures", section(heading, 4)) + self.assertIn("#inspect-notification-logs", section(heading, 4)) + + +if __name__ == "__main__": + unittest.main()