Merge reviewed evidence-preserving performance help

Preserve exact candidate 286c8a1f51. Retained copied-recipe, mirror, Docs viewer and desktop/phone browser evidence covers this existing guide improvement; no runtime or installed performance claim.

Change-source: pulse-maintainer

docs/TROUBLESHOOTING.md

@@ -216,6 +216,99 @@ leave the original host unchanged. Follow
 [Clone identity recovery](UNIFIED_AGENT.md#duplicate-agents) for the configuration
 precedence, systemd example and checks after restart.
 
+#### Excessive CPU, writes or database growth
+
+First distinguish **Pulse server activity**, agent activity and total host or
+storage-device activity. A database's size is retained space, not the number of
+bytes written: repeated updates can produce high writes without growing the file.
+Keep the original running version, uptime, measurement time and workload before
+changing anything. A different binary can migrate persistent data; switching
+back to an older version does not make that data a clean older-version baseline.
+
+- For CPU, record the measuring tool, allocated CPUs or container quota and
+  whether its percentage represents one core or the whole allocation. A `ps`
+  `%CPU` value is an average since process start, not a recent sampling window.
+- Note fleet size, polling interval and whether Pulse dashboards are open. If
+  safe, close all Pulse tabs and compare another equal-length window, without
+  restarting the server or changing polling. A decrease narrows the workload;
+  it does not establish the cause or prove monitoring is healthy.
+- If the host is unresponsive or storage is nearly full, do not prolong the
+  failure to collect a benchmark. Keep existing observations and report the
+  interruption instead. Do not enable Debug or repeatedly export diagnostics
+  merely to measure performance; those actions add work and can change the result.
+
+For a responsive **Linux systemd / Proxmox LXC** install, the following reads
+two process-I/O samples, waiting 60 seconds between them. Run it inside the
+Pulse container for LXC, not on the Proxmox host. Substitute the actual service
+name (`pulse-backend` on some older installs). Use an account authorised to
+read the process counters; no service restart or database access is needed.
+
+```bash
+# systemd / Proxmox LXC: bounded process-write samples
+(
+  set -e
+  for sample in 1 2; do
+    date -u +'%Y-%m-%dT%H:%M:%SZ'
+    pid=$(systemctl show pulse --property=MainPID --value)
+    case "$pid" in
+      ''|0|*[!0-9]*) printf 'No running Pulse PID; sample unavailable.\n' >&2; exit 1 ;;
+    esac
+    TZ=UTC ps -p "$pid" -o pid=,lstart=
+    sudo awk '
+      $1 == "write_bytes:" || $1 == "cancelled_write_bytes:" { print; fields++ }
+      END { if (fields != 2) exit 1 }
+    ' "/proc/$pid/io"
+    if [ "$sample" -eq 1 ]; then sleep 60; fi
+  done
+)
+```
+
+Compare `write_bytes` only when both samples have the same PID and process
+start time, no restart occurred, and the counter did not decrease. Divide the
+byte difference by the **actual elapsed seconds**. This is storage-accounted
+process I/O, not filesystem growth or physical SSD wear; cancelled writes and
+background writeback can differ from device measurements. Keep
+`cancelled_write_bytes` alongside it, not as proof of bytes reaching the drive.
+An extrapolated GB/day rate is a projection of that short window, not a measured
+day's total. Preserve the window and units with the result.
+
+For **Docker / Compose**, run this on the Docker host, replacing `pulse` with
+the running container name. It reads only the start time and selected statistics,
+not the container environment or configuration.
+
+```bash
+# Docker: bounded container statistics
+(
+  set -e
+  for sample in 1 2; do
+    date -u +'%Y-%m-%dT%H:%M:%SZ'
+    docker inspect --format 'Started={{.State.StartedAt}}' pulse
+    stats=$(docker stats --no-stream --format \
+      'CPU={{.CPUPerc}} Memory={{.MemUsage}} BlockIO={{.BlockIO}}' pulse)
+    if [ -z "$stats" ]; then
+      printf 'Container statistics unavailable; no zero inferred.\n' >&2
+      exit 1
+    fi
+    printf '%s\n' "$stats"
+    if [ "$sample" -eq 1 ]; then sleep 60; fi
+  done
+)
+```
+
+Docker **BlockIO** is cumulative read / write activity, not bytes per second;
+compare its write side only across the same uninterrupted container run. Its
+displayed units are rounded. Container, process and whole-device counters have
+different scopes and must not be added together or compared as interchangeable
+measurements. A failed, denied or empty read is unavailable, not zero activity.
+
+Keep persistent data on durable storage. Do not delete or truncate history,
+incident, queue or audit files, remove database indexes, or move the data
+directory to tmpfs as a diagnostic workaround. These actions can lose evidence
+or protections without fixing the writer. Do not post databases, profiles, full
+`/proc` dumps, container configuration or raw environments. Share only the bounded
+measurements, workload, versions and a relevant manually redacted error; follow
+[Getting Help](#-getting-help) for any additional evidence.
+
 ### Notifications
 
 #### No alert when Pulse, power or internet goes down

frontend-modern/public/docs/TROUBLESHOOTING.md

@@ -216,6 +216,99 @@ leave the original host unchanged. Follow
 [Clone identity recovery](UNIFIED_AGENT.md#duplicate-agents) for the configuration
 precedence, systemd example and checks after restart.
 
+#### Excessive CPU, writes or database growth
+
+First distinguish **Pulse server activity**, agent activity and total host or
+storage-device activity. A database's size is retained space, not the number of
+bytes written: repeated updates can produce high writes without growing the file.
+Keep the original running version, uptime, measurement time and workload before
+changing anything. A different binary can migrate persistent data; switching
+back to an older version does not make that data a clean older-version baseline.
+
+- For CPU, record the measuring tool, allocated CPUs or container quota and
+  whether its percentage represents one core or the whole allocation. A `ps`
+  `%CPU` value is an average since process start, not a recent sampling window.
+- Note fleet size, polling interval and whether Pulse dashboards are open. If
+  safe, close all Pulse tabs and compare another equal-length window, without
+  restarting the server or changing polling. A decrease narrows the workload;
+  it does not establish the cause or prove monitoring is healthy.
+- If the host is unresponsive or storage is nearly full, do not prolong the
+  failure to collect a benchmark. Keep existing observations and report the
+  interruption instead. Do not enable Debug or repeatedly export diagnostics
+  merely to measure performance; those actions add work and can change the result.
+
+For a responsive **Linux systemd / Proxmox LXC** install, the following reads
+two process-I/O samples, waiting 60 seconds between them. Run it inside the
+Pulse container for LXC, not on the Proxmox host. Substitute the actual service
+name (`pulse-backend` on some older installs). Use an account authorised to
+read the process counters; no service restart or database access is needed.
+
+```bash
+# systemd / Proxmox LXC: bounded process-write samples
+(
+  set -e
+  for sample in 1 2; do
+    date -u +'%Y-%m-%dT%H:%M:%SZ'
+    pid=$(systemctl show pulse --property=MainPID --value)
+    case "$pid" in
+      ''|0|*[!0-9]*) printf 'No running Pulse PID; sample unavailable.\n' >&2; exit 1 ;;
+    esac
+    TZ=UTC ps -p "$pid" -o pid=,lstart=
+    sudo awk '
+      $1 == "write_bytes:" || $1 == "cancelled_write_bytes:" { print; fields++ }
+      END { if (fields != 2) exit 1 }
+    ' "/proc/$pid/io"
+    if [ "$sample" -eq 1 ]; then sleep 60; fi
+  done
+)
+```
+
+Compare `write_bytes` only when both samples have the same PID and process
+start time, no restart occurred, and the counter did not decrease. Divide the
+byte difference by the **actual elapsed seconds**. This is storage-accounted
+process I/O, not filesystem growth or physical SSD wear; cancelled writes and
+background writeback can differ from device measurements. Keep
+`cancelled_write_bytes` alongside it, not as proof of bytes reaching the drive.
+An extrapolated GB/day rate is a projection of that short window, not a measured
+day's total. Preserve the window and units with the result.
+
+For **Docker / Compose**, run this on the Docker host, replacing `pulse` with
+the running container name. It reads only the start time and selected statistics,
+not the container environment or configuration.
+
+```bash
+# Docker: bounded container statistics
+(
+  set -e
+  for sample in 1 2; do
+    date -u +'%Y-%m-%dT%H:%M:%SZ'
+    docker inspect --format 'Started={{.State.StartedAt}}' pulse
+    stats=$(docker stats --no-stream --format \
+      'CPU={{.CPUPerc}} Memory={{.MemUsage}} BlockIO={{.BlockIO}}' pulse)
+    if [ -z "$stats" ]; then
+      printf 'Container statistics unavailable; no zero inferred.\n' >&2
+      exit 1
+    fi
+    printf '%s\n' "$stats"
+    if [ "$sample" -eq 1 ]; then sleep 60; fi
+  done
+)
+```
+
+Docker **BlockIO** is cumulative read / write activity, not bytes per second;
+compare its write side only across the same uninterrupted container run. Its
+displayed units are rounded. Container, process and whole-device counters have
+different scopes and must not be added together or compared as interchangeable
+measurements. A failed, denied or empty read is unavailable, not zero activity.
+
+Keep persistent data on durable storage. Do not delete or truncate history,
+incident, queue or audit files, remove database indexes, or move the data
+directory to tmpfs as a diagnostic workaround. These actions can lose evidence
+or protections without fixing the writer. Do not post databases, profiles, full
+`/proc` dumps, container configuration or raw environments. Share only the bounded
+measurements, workload, versions and a relevant manually redacted error; follow
+[Getting Help](#-getting-help) for any additional evidence.
+
 ### Notifications
 
 #### No alert when Pulse, power or internet goes down

scripts/tests/test_performance_troubleshooting_docs.py

@@ -0,0 +1,209 @@
+#!/usr/bin/env python3
+"""Exercise the guide's bounded I/O recipes with synthetic OS/container readers.
+
+The real awk filters run against fixtures. No Pulse process, database, Docker
+daemon or host service is inspected. These checks establish recipe behaviour,
+not native write rates, CPU relief or an operational repair.
+"""
+
+from __future__ import annotations
+
+import json
+import os
+from pathlib import Path
+import re
+import subprocess
+import tempfile
+import unittest
+
+
+ROOT = Path(__file__).resolve().parents[2]
+DOC = ROOT / "docs/TROUBLESHOOTING.md"
+
+
+def section() -> str:
+    return DOC.read_text(encoding="utf-8").split(
+        "#### Excessive CPU, writes or database growth\n", 1
+    )[1].split("\n### Notifications\n", 1)[0]
+
+
+def recipes() -> dict[str, str]:
+    blocks = re.findall(r"```bash\n(.*?)```", section(), re.DOTALL)
+    if len(blocks) != 2:
+        raise AssertionError("expected two bounded deployment recipes")
+    return {"docker" if "# Docker:" in block else "systemd": block for block in blocks}
+
+
+# All external commands record exact arguments. sudo is a test-only adapter:
+# it replaces only the documented /proc/PID/io input and executes real awk.
+READER = '''#!/usr/bin/env python3
+import json, os, pathlib, subprocess, sys
+name = pathlib.Path(sys.argv[0]).name
+args = sys.argv[1:]
+root = pathlib.Path(os.environ['RECIPE_FIXTURE'])
+log = root / 'calls.jsonl'
+previous = [json.loads(line) for line in log.read_text().splitlines()] if log.exists() else []
+count = sum(call['name'] == name for call in previous)
+with log.open('a') as stream:
+    stream.write(json.dumps({'name': name, 'args': args}) + '\\n')
+if name == 'date':
+    print('2026-10-03T00:00:00Z' if count == 0 else '2026-10-03T00:01:00Z')
+elif name == 'sleep':
+    assert args == ['60']
+elif name == 'systemctl':
+    print(os.environ.get('FIXTURE_PID', '4321'))
+    sys.exit(int(os.environ.get('SYSTEMCTL_EXIT', '0')))
+elif name == 'ps':
+    assert os.environ['TZ'] == 'UTC'
+    print('4321 Sat Oct 3 00:00:00 2026')
+    sys.exit(int(os.environ.get('PS_EXIT', '0')))
+elif name == 'sudo':
+    assert args[0] == 'awk'
+    assert args[-1] == '/proc/4321/io'
+    target = root / ('missing' if os.environ.get('IO_MISSING') else f'io-{count}')
+    result = subprocess.run([os.environ['REAL_AWK'], *args[1:-1], str(target)])
+    sys.exit(result.returncode)
+elif name == 'docker':
+    if args[0] == 'inspect':
+        print('Started=2026-10-02T20:00:00Z')
+    elif args[0] == 'stats':
+        text = os.environ.get('DOCKER_STATS', 'CPU=12.50% Memory=100MiB / 1GiB BlockIO=1MB / 12MB')
+        if text:
+            print(text)
+    else:
+        raise AssertionError('unsupported docker command')
+    setting = 'DOCKER_INSPECT_EXIT' if args[0] == 'inspect' else 'DOCKER_STATS_EXIT'
+    sys.exit(int(os.environ.get(setting, '0')))
+else:
+    raise AssertionError('unexpected command')
+'''
+
+
+class PerformanceTroubleshootingDocsTest(unittest.TestCase):
+    def exercise(self, deployment: str, *, counters: tuple[str, str] | None = None,
+                 **settings: str):
+        import shutil
+
+        with tempfile.TemporaryDirectory() as temporary:
+            directory = Path(temporary)
+            for name in ("date", "sleep", "systemctl", "ps", "sudo", "docker"):
+                command = directory / name
+                command.write_text(READER, encoding="utf-8")
+                command.chmod(0o700)
+            counters = counters or (
+                "rchar: 987654\nwchar: 999999\nwrite_bytes: 1000\ncancelled_write_bytes: 20\n",
+                "rchar: 999999\nwchar: 999999\nwrite_bytes: 13000\ncancelled_write_bytes: 30\n",
+            )
+            for index, text in enumerate(counters):
+                (directory / f"io-{index}").write_text(text, encoding="utf-8")
+            env = dict(os.environ, PATH=f"{directory}:{os.environ['PATH']}",
+                       RECIPE_FIXTURE=str(directory), REAL_AWK=shutil.which("awk"), **settings)
+            result = subprocess.run(["bash", "-c", recipes()[deployment]], env=env,
+                                    text=True, capture_output=True, timeout=10)
+            calls = [json.loads(line) for line in (directory / "calls.jsonl").read_text().splitlines()]
+            return result, calls
+
+    def test_shipped_document_matches_the_exercised_source(self):
+        self.assertEqual(DOC.read_bytes(),
+                         (ROOT / "frontend-modern/public/docs/TROUBLESHOOTING.md").read_bytes())
+
+    def test_systemd_samples_only_write_counters_and_process_identity(self):
+        result, calls = self.exercise("systemd")
+        self.assertEqual(result.returncode, 0, result.stderr)
+        self.assertEqual([call["name"] for call in calls],
+                         ["date", "systemctl", "ps", "sudo", "sleep",
+                          "date", "systemctl", "ps", "sudo"])
+        for call in calls:
+            if call["name"] == "systemctl":
+                self.assertEqual(call["args"], ["show", "pulse", "--property=MainPID", "--value"])
+            elif call["name"] == "ps":
+                self.assertEqual(call["args"], ["-p", "4321", "-o", "pid=,lstart="])
+        self.assertIn("write_bytes: 1000", result.stdout)
+        self.assertIn("write_bytes: 13000", result.stdout)
+        self.assertIn("cancelled_write_bytes: 20", result.stdout)
+        self.assertNotIn("rchar", result.stdout)
+        self.assertNotIn("wchar", result.stdout)
+
+    def test_unavailable_pid_never_reads_a_process_or_sleeps(self):
+        for pid in ("", "0", "not-a-pid", "4321 extra"):
+            with self.subTest(pid=pid):
+                result, calls = self.exercise("systemd", FIXTURE_PID=pid)
+                self.assertNotEqual(result.returncode, 0)
+                self.assertIn("sample unavailable", result.stderr)
+                self.assertEqual([c["name"] for c in calls], ["date", "systemctl"])
+
+    def test_service_and_process_read_failures_stop_the_collection(self):
+        for settings, expected in (({"SYSTEMCTL_EXIT": "3"}, ["date", "systemctl"]),
+                                   ({"PS_EXIT": "4"}, ["date", "systemctl", "ps"])):
+            with self.subTest(settings=settings):
+                result, calls = self.exercise("systemd", **settings)
+                self.assertNotEqual(result.returncode, 0)
+                self.assertEqual([c["name"] for c in calls], expected)
+
+    def test_missing_process_io_is_failure_not_zero_writes(self):
+        result, calls = self.exercise("systemd", IO_MISSING="1")
+        self.assertNotEqual(result.returncode, 0)
+        self.assertNotIn("write_bytes:", result.stdout)
+        self.assertNotIn("sleep", [c["name"] for c in calls])
+
+    def test_incomplete_process_counters_do_not_become_a_success(self):
+        for text in ("", "rchar: 999\n", "write_bytes: 1000\n"):
+            with self.subTest(text=text):
+                result, calls = self.exercise("systemd", counters=(text, text))
+                self.assertNotEqual(result.returncode, 0)
+                self.assertNotIn("sleep", [c["name"] for c in calls])
+
+    def test_decreasing_counters_are_retained_not_turned_into_a_rate(self):
+        result, _ = self.exercise("systemd", counters=(
+            "write_bytes: 13000\ncancelled_write_bytes: 30\n",
+            "write_bytes: 1000\ncancelled_write_bytes: 20\n",
+        ))
+        self.assertEqual(result.returncode, 0, result.stderr)
+        self.assertIn("write_bytes: 13000", result.stdout)
+        self.assertIn("write_bytes: 1000", result.stdout)
+        self.assertNotRegex(result.stdout, r"GB/day|bytes/s|bytes: -[0-9]+")
+
+    def test_docker_reads_only_start_time_and_selected_statistics_twice(self):
+        result, calls = self.exercise("docker")
+        self.assertEqual(result.returncode, 0, result.stderr)
+        expected = [
+            ["inspect", "--format", "Started={{.State.StartedAt}}", "pulse"],
+            ["stats", "--no-stream", "--format",
+             "CPU={{.CPUPerc}} Memory={{.MemUsage}} BlockIO={{.BlockIO}}", "pulse"],
+        ]
+        self.assertEqual([c["args"] for c in calls if c["name"] == "docker"], expected * 2)
+        self.assertEqual([c["name"] for c in calls],
+                         ["date", "docker", "docker", "sleep", "date", "docker", "docker"])
+        self.assertEqual(result.stdout.count("BlockIO=1MB / 12MB"), 2)
+
+    def test_empty_docker_statistics_are_unavailable_not_zero(self):
+        result, calls = self.exercise("docker", DOCKER_STATS="")
+        self.assertNotEqual(result.returncode, 0)
+        self.assertIn("no zero inferred", result.stderr)
+        self.assertNotIn("sleep", [c["name"] for c in calls])
+
+    def test_docker_reader_failure_stops_before_another_sample(self):
+        for settings, expected in (
+            ({"DOCKER_INSPECT_EXIT": "2"}, ["date", "docker"]),
+            ({"DOCKER_STATS_EXIT": "3"}, ["date", "docker", "docker"]),
+        ):
+            with self.subTest(settings=settings):
+                result, calls = self.exercise("docker", **settings)
+                self.assertEqual(result.returncode, int(next(iter(settings.values()))))
+                self.assertEqual([c["name"] for c in calls], expected)
+
+    def test_guidance_keeps_scope_units_baselines_and_data_safety(self):
+        guide = " ".join(section().split())
+        for phrase in ("same PID and process start time", "actual elapsed seconds",
+                       "not a measured day's total", "not bytes per second",
+                       "not a recent sampling window", "different scopes",
+                       "inside the Pulse container", "does not make that data a clean",
+                       "Do not enable Debug", "remove database indexes", "tmpfs",
+                       "Do not post databases", "manually redacted error"):
+            self.assertIn(phrase, guide)
+        for recipe in recipes().values():
+            self.assertNotRegex(recipe, r"\b(?:restart|kill|rm|truncate|sqlite3|curl|printenv)\b|--follow|--token")
+
+
+if __name__ == "__main__":
+    unittest.main()