gsadmin/pulse
1
0
Fork 0
mirror of https://github.com/rcourtman/Pulse.git synced 2026-10-23 20:25:56 +00:00
Code Issues Packages Projects Releases Wiki Activity

Make write and CPU troubleshooting bounded and evidence-preserving

Browse Source 
Give existing help readers scoped process/container samples, honest units and unchanged-workload baselines. Exercise the exact copied recipes, including stopped services, missing counters and failed or empty container reads, without touching a database or service.

Contract-Neutral: Existing-guide documentation and recipe tests only; no runtime, API, persistence, access or retention change.
Change-source: pulse-maintainer
This commit is contained in:
pulse-triage[bot]
2026-10-03 00:45:25 +01:00
parent e929442038
commit 286c8a1f51
3 changed files with 395 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/TROUBLESHOOTING.md
+93
View File
@@ -216,6 +216,99 @@ leave the original host unchanged. Follow
[Clone identity recovery](UNIFIED_AGENT.md#duplicate-agents) for the configuration
precedence, systemd example and checks after restart.
#### Excessive CPU, writes or database growth
First distinguish **Pulse server activity**, agent activity and total host or
storage-device activity. A database's size is retained space, not the number of
bytes written: repeated updates can produce high writes without growing the file.
Keep the original running version, uptime, measurement time and workload before
changing anything. A different binary can migrate persistent data; switching
back to an older version does not make that data a clean older-version baseline.
- For CPU, record the measuring tool, allocated CPUs or container quota and
whether its percentage represents one core or the whole allocation. A `ps`
`%CPU` value is an average since process start, not a recent sampling window.
- Note fleet size, polling interval and whether Pulse dashboards are open. If
safe, close all Pulse tabs and compare another equal-length window, without
restarting the server or changing polling. A decrease narrows the workload;
it does not establish the cause or prove monitoring is healthy.
- If the host is unresponsive or storage is nearly full, do not prolong the
failure to collect a benchmark. Keep existing observations and report the
interruption instead. Do not enable Debug or repeatedly export diagnostics
merely to measure performance; those actions add work and can change the result.
For a responsive **Linux systemd / Proxmox LXC** install, the following reads
two process-I/O samples, waiting 60 seconds between them. Run it inside the
Pulse container for LXC, not on the Proxmox host. Substitute the actual service
name (`pulse-backend` on some older installs). Use an account authorised to
read the process counters; no service restart or database access is needed.
```bash
# systemd / Proxmox LXC: bounded process-write samples
(
set -e
for sample in 1 2; do
date -u +'%Y-%m-%dT%H:%M:%SZ'
pid=$(systemctl show pulse --property=MainPID --value)
case "$pid" in
''|0|*[!0-9]*) printf 'No running Pulse PID; sample unavailable.\n' >&2; exit 1 ;;
esac
TZ=UTC ps -p "$pid" -o pid=,lstart=
sudo awk '
$1 == "write_bytes:" || $1 == "cancelled_write_bytes:" { print; fields++ }
END { if (fields != 2) exit 1 }
' "/proc/$pid/io"
if [ "$sample" -eq 1 ]; then sleep 60; fi
done
)
```
Compare `write_bytes` only when both samples have the same PID and process
start time, no restart occurred, and the counter did not decrease. Divide the
byte difference by the **actual elapsed seconds**. This is storage-accounted
process I/O, not filesystem growth or physical SSD wear; cancelled writes and
background writeback can differ from device measurements. Keep
`cancelled_write_bytes` alongside it, not as proof of bytes reaching the drive.
An extrapolated GB/day rate is a projection of that short window, not a measured
day's total. Preserve the window and units with the result.
For **Docker / Compose**, run this on the Docker host, replacing `pulse` with
the running container name. It reads only the start time and selected statistics,
not the container environment or configuration.
```bash
# Docker: bounded container statistics
(
set -e
for sample in 1 2; do
date -u +'%Y-%m-%dT%H:%M:%SZ'
docker inspect --format 'Started={{.State.StartedAt}}' pulse
stats=$(docker stats --no-stream --format \
'CPU={{.CPUPerc}} Memory={{.MemUsage}} BlockIO={{.BlockIO}}' pulse)
if [ -z "$stats" ]; then
printf 'Container statistics unavailable; no zero inferred.\n' >&2
exit 1
fi
printf '%s\n' "$stats"
if [ "$sample" -eq 1 ]; then sleep 60; fi
done
)
```
Docker **BlockIO** is cumulative read / write activity, not bytes per second;
compare its write side only across the same uninterrupted container run. Its
displayed units are rounded. Container, process and whole-device counters have
different scopes and must not be added together or compared as interchangeable
measurements. A failed, denied or empty read is unavailable, not zero activity.
Keep persistent data on durable storage. Do not delete or truncate history,
incident, queue or audit files, remove database indexes, or move the data
directory to tmpfs as a diagnostic workaround. These actions can lose evidence
or protections without fixing the writer. Do not post databases, profiles, full
`/proc` dumps, container configuration or raw environments. Share only the bounded
measurements, workload, versions and a relevant manually redacted error; follow
[Getting Help](#-getting-help) for any additional evidence.
### Notifications
#### No alert when Pulse, power or internet goes down
frontend-modern/public/docs/TROUBLESHOOTING.md
+93
View File
@@ -216,6 +216,99 @@ leave the original host unchanged. Follow
[Clone identity recovery](UNIFIED_AGENT.md#duplicate-agents) for the configuration
precedence, systemd example and checks after restart.
#### Excessive CPU, writes or database growth
First distinguish **Pulse server activity**, agent activity and total host or
storage-device activity. A database's size is retained space, not the number of
bytes written: repeated updates can produce high writes without growing the file.
Keep the original running version, uptime, measurement time and workload before
changing anything. A different binary can migrate persistent data; switching
back to an older version does not make that data a clean older-version baseline.
- For CPU, record the measuring tool, allocated CPUs or container quota and
whether its percentage represents one core or the whole allocation. A `ps`
`%CPU` value is an average since process start, not a recent sampling window.
- Note fleet size, polling interval and whether Pulse dashboards are open. If
safe, close all Pulse tabs and compare another equal-length window, without
restarting the server or changing polling. A decrease narrows the workload;
it does not establish the cause or prove monitoring is healthy.
- If the host is unresponsive or storage is nearly full, do not prolong the
failure to collect a benchmark. Keep existing observations and report the
interruption instead. Do not enable Debug or repeatedly export diagnostics
merely to measure performance; those actions add work and can change the result.
For a responsive **Linux systemd / Proxmox LXC** install, the following reads
two process-I/O samples, waiting 60 seconds between them. Run it inside the
Pulse container for LXC, not on the Proxmox host. Substitute the actual service
name (`pulse-backend` on some older installs). Use an account authorised to
read the process counters; no service restart or database access is needed.
```bash
# systemd / Proxmox LXC: bounded process-write samples
(
set -e
for sample in 1 2; do
date -u +'%Y-%m-%dT%H:%M:%SZ'
pid=$(systemctl show pulse --property=MainPID --value)
case "$pid" in
''|0|*[!0-9]*) printf 'No running Pulse PID; sample unavailable.\n' >&2; exit 1 ;;
esac
TZ=UTC ps -p "$pid" -o pid=,lstart=
sudo awk '
$1 == "write_bytes:" || $1 == "cancelled_write_bytes:" { print; fields++ }
END { if (fields != 2) exit 1 }
' "/proc/$pid/io"
if [ "$sample" -eq 1 ]; then sleep 60; fi
done
)
```
Compare `write_bytes` only when both samples have the same PID and process
start time, no restart occurred, and the counter did not decrease. Divide the
byte difference by the **actual elapsed seconds**. This is storage-accounted
process I/O, not filesystem growth or physical SSD wear; cancelled writes and
background writeback can differ from device measurements. Keep
`cancelled_write_bytes` alongside it, not as proof of bytes reaching the drive.
An extrapolated GB/day rate is a projection of that short window, not a measured
day's total. Preserve the window and units with the result.
For **Docker / Compose**, run this on the Docker host, replacing `pulse` with
the running container name. It reads only the start time and selected statistics,
not the container environment or configuration.
```bash
# Docker: bounded container statistics
(
set -e
for sample in 1 2; do
date -u +'%Y-%m-%dT%H:%M:%SZ'
docker inspect --format 'Started={{.State.StartedAt}}' pulse
stats=$(docker stats --no-stream --format \
'CPU={{.CPUPerc}} Memory={{.MemUsage}} BlockIO={{.BlockIO}}' pulse)
if [ -z "$stats" ]; then
printf 'Container statistics unavailable; no zero inferred.\n' >&2
exit 1
fi
printf '%s\n' "$stats"
if [ "$sample" -eq 1 ]; then sleep 60; fi
done
)
```
Docker **BlockIO** is cumulative read / write activity, not bytes per second;
compare its write side only across the same uninterrupted container run. Its
displayed units are rounded. Container, process and whole-device counters have
different scopes and must not be added together or compared as interchangeable
measurements. A failed, denied or empty read is unavailable, not zero activity.
Keep persistent data on durable storage. Do not delete or truncate history,
incident, queue or audit files, remove database indexes, or move the data
directory to tmpfs as a diagnostic workaround. These actions can lose evidence
or protections without fixing the writer. Do not post databases, profiles, full
`/proc` dumps, container configuration or raw environments. Share only the bounded
measurements, workload, versions and a relevant manually redacted error; follow
[Getting Help](#-getting-help) for any additional evidence.
### Notifications
#### No alert when Pulse, power or internet goes down
scripts/tests/test_performance_troubleshooting_docs.py
+209
View File
@@ -0,0 +1,209 @@
#!/usr/bin/env python3
"""Exercise the guide's bounded I/O recipes with synthetic OS/container readers.
The real awk filters run against fixtures. No Pulse process, database, Docker
daemon or host service is inspected. These checks establish recipe behaviour,
not native write rates, CPU relief or an operational repair.
"""
from __future__ import annotations
import json
import os
from pathlib import Path
import re
import subprocess
import tempfile
import unittest
ROOT = Path(__file__).resolve().parents[2]
DOC = ROOT / "docs/TROUBLESHOOTING.md"
def section() -> str:
return DOC.read_text(encoding="utf-8").split(
"#### Excessive CPU, writes or database growth\n", 1
)[1].split("\n### Notifications\n", 1)[0]
def recipes() -> dict[str, str]:
blocks = re.findall(r"```bash\n(.*?)```", section(), re.DOTALL)
if len(blocks) != 2:
raise AssertionError("expected two bounded deployment recipes")
return {"docker" if "# Docker:" in block else "systemd": block for block in blocks}
# All external commands record exact arguments. sudo is a test-only adapter:
# it replaces only the documented /proc/PID/io input and executes real awk.
READER = '''#!/usr/bin/env python3
import json, os, pathlib, subprocess, sys
name = pathlib.Path(sys.argv[0]).name
args = sys.argv[1:]
root = pathlib.Path(os.environ['RECIPE_FIXTURE'])
log = root / 'calls.jsonl'
previous = [json.loads(line) for line in log.read_text().splitlines()] if log.exists() else []
count = sum(call['name'] == name for call in previous)
with log.open('a') as stream:
stream.write(json.dumps({'name': name, 'args': args}) + '\\n')
if name == 'date':
print('2026-10-03T00:00:00Z' if count == 0 else '2026-10-03T00:01:00Z')
elif name == 'sleep':
assert args == ['60']
elif name == 'systemctl':
print(os.environ.get('FIXTURE_PID', '4321'))
sys.exit(int(os.environ.get('SYSTEMCTL_EXIT', '0')))
elif name == 'ps':
assert os.environ['TZ'] == 'UTC'
print('4321 Sat Oct 3 00:00:00 2026')
sys.exit(int(os.environ.get('PS_EXIT', '0')))
elif name == 'sudo':
assert args[0] == 'awk'
assert args[-1] == '/proc/4321/io'
target = root / ('missing' if os.environ.get('IO_MISSING') else f'io-{count}')
result = subprocess.run([os.environ['REAL_AWK'], *args[1:-1], str(target)])
sys.exit(result.returncode)
elif name == 'docker':
if args[0] == 'inspect':
print('Started=2026-10-02T20:00:00Z')
elif args[0] == 'stats':
text = os.environ.get('DOCKER_STATS', 'CPU=12.50% Memory=100MiB / 1GiB BlockIO=1MB / 12MB')
if text:
print(text)
else:
raise AssertionError('unsupported docker command')
setting = 'DOCKER_INSPECT_EXIT' if args[0] == 'inspect' else 'DOCKER_STATS_EXIT'
sys.exit(int(os.environ.get(setting, '0')))
else:
raise AssertionError('unexpected command')
'''
class PerformanceTroubleshootingDocsTest(unittest.TestCase):
def exercise(self, deployment: str, *, counters: tuple[str, str] | None = None,
**settings: str):
import shutil
with tempfile.TemporaryDirectory() as temporary:
directory = Path(temporary)
for name in ("date", "sleep", "systemctl", "ps", "sudo", "docker"):
command = directory / name
command.write_text(READER, encoding="utf-8")
command.chmod(0o700)
counters = counters or (
"rchar: 987654\nwchar: 999999\nwrite_bytes: 1000\ncancelled_write_bytes: 20\n",
"rchar: 999999\nwchar: 999999\nwrite_bytes: 13000\ncancelled_write_bytes: 30\n",
)
for index, text in enumerate(counters):
(directory / f"io-{index}").write_text(text, encoding="utf-8")
env = dict(os.environ, PATH=f"{directory}:{os.environ['PATH']}",
RECIPE_FIXTURE=str(directory), REAL_AWK=shutil.which("awk"), **settings)
result = subprocess.run(["bash", "-c", recipes()[deployment]], env=env,
text=True, capture_output=True, timeout=10)
calls = [json.loads(line) for line in (directory / "calls.jsonl").read_text().splitlines()]
return result, calls
def test_shipped_document_matches_the_exercised_source(self):
self.assertEqual(DOC.read_bytes(),
(ROOT / "frontend-modern/public/docs/TROUBLESHOOTING.md").read_bytes())
def test_systemd_samples_only_write_counters_and_process_identity(self):
result, calls = self.exercise("systemd")
self.assertEqual(result.returncode, 0, result.stderr)
self.assertEqual([call["name"] for call in calls],
["date", "systemctl", "ps", "sudo", "sleep",
"date", "systemctl", "ps", "sudo"])
for call in calls:
if call["name"] == "systemctl":
self.assertEqual(call["args"], ["show", "pulse", "--property=MainPID", "--value"])
elif call["name"] == "ps":
self.assertEqual(call["args"], ["-p", "4321", "-o", "pid=,lstart="])
self.assertIn("write_bytes: 1000", result.stdout)
self.assertIn("write_bytes: 13000", result.stdout)
self.assertIn("cancelled_write_bytes: 20", result.stdout)
self.assertNotIn("rchar", result.stdout)
self.assertNotIn("wchar", result.stdout)
def test_unavailable_pid_never_reads_a_process_or_sleeps(self):
for pid in ("", "0", "not-a-pid", "4321 extra"):
with self.subTest(pid=pid):
result, calls = self.exercise("systemd", FIXTURE_PID=pid)
self.assertNotEqual(result.returncode, 0)
self.assertIn("sample unavailable", result.stderr)
self.assertEqual([c["name"] for c in calls], ["date", "systemctl"])
def test_service_and_process_read_failures_stop_the_collection(self):
for settings, expected in (({"SYSTEMCTL_EXIT": "3"}, ["date", "systemctl"]),
({"PS_EXIT": "4"}, ["date", "systemctl", "ps"])):
with self.subTest(settings=settings):
result, calls = self.exercise("systemd", **settings)
self.assertNotEqual(result.returncode, 0)
self.assertEqual([c["name"] for c in calls], expected)
def test_missing_process_io_is_failure_not_zero_writes(self):
result, calls = self.exercise("systemd", IO_MISSING="1")
self.assertNotEqual(result.returncode, 0)
self.assertNotIn("write_bytes:", result.stdout)
self.assertNotIn("sleep", [c["name"] for c in calls])
def test_incomplete_process_counters_do_not_become_a_success(self):
for text in ("", "rchar: 999\n", "write_bytes: 1000\n"):
with self.subTest(text=text):
result, calls = self.exercise("systemd", counters=(text, text))
self.assertNotEqual(result.returncode, 0)
self.assertNotIn("sleep", [c["name"] for c in calls])
def test_decreasing_counters_are_retained_not_turned_into_a_rate(self):
result, _ = self.exercise("systemd", counters=(
"write_bytes: 13000\ncancelled_write_bytes: 30\n",
"write_bytes: 1000\ncancelled_write_bytes: 20\n",
))
self.assertEqual(result.returncode, 0, result.stderr)
self.assertIn("write_bytes: 13000", result.stdout)
self.assertIn("write_bytes: 1000", result.stdout)
self.assertNotRegex(result.stdout, r"GB/day|bytes/s|bytes: -[0-9]+")
def test_docker_reads_only_start_time_and_selected_statistics_twice(self):
result, calls = self.exercise("docker")
self.assertEqual(result.returncode, 0, result.stderr)
expected = [
["inspect", "--format", "Started={{.State.StartedAt}}", "pulse"],
["stats", "--no-stream", "--format",
"CPU={{.CPUPerc}} Memory={{.MemUsage}} BlockIO={{.BlockIO}}", "pulse"],
]
self.assertEqual([c["args"] for c in calls if c["name"] == "docker"], expected * 2)
self.assertEqual([c["name"] for c in calls],
["date", "docker", "docker", "sleep", "date", "docker", "docker"])
self.assertEqual(result.stdout.count("BlockIO=1MB / 12MB"), 2)
def test_empty_docker_statistics_are_unavailable_not_zero(self):
result, calls = self.exercise("docker", DOCKER_STATS="")
self.assertNotEqual(result.returncode, 0)
self.assertIn("no zero inferred", result.stderr)
self.assertNotIn("sleep", [c["name"] for c in calls])
def test_docker_reader_failure_stops_before_another_sample(self):
for settings, expected in (
({"DOCKER_INSPECT_EXIT": "2"}, ["date", "docker"]),
({"DOCKER_STATS_EXIT": "3"}, ["date", "docker", "docker"]),
):
with self.subTest(settings=settings):
result, calls = self.exercise("docker", **settings)
self.assertEqual(result.returncode, int(next(iter(settings.values()))))
self.assertEqual([c["name"] for c in calls], expected)
def test_guidance_keeps_scope_units_baselines_and_data_safety(self):
guide = " ".join(section().split())
for phrase in ("same PID and process start time", "actual elapsed seconds",
"not a measured day's total", "not bytes per second",
"not a recent sampling window", "different scopes",
"inside the Pulse container", "does not make that data a clean",
"Do not enable Debug", "remove database indexes", "tmpfs",
"Do not post databases", "manually redacted error"):
self.assertIn(phrase, guide)
for recipe in recipes().values():
self.assertNotRegex(recipe, r"\b(?:restart|kill|rm|truncate|sqlite3|curl|printenv)\b|--follow|--token")
if __name__ == "__main__":
unittest.main()