gsadmin/pulse
1
0
Fork 0
mirror of https://github.com/rcourtman/Pulse.git synced 2026-10-23 20:25:56 +00:00
Code Issues Packages Projects Releases Wiki Activity

Keep history diagnostics private and match current query types

Browse Source 
Reuse the tested API helper, explain point/source and access distinctions, and preserve data before storage tuning. Copied examples now exercise private success/error responses, redirects, incomplete transport and encoded IDs.

Contract-Neutral: Documentation and recipe checks align with existing history API behaviour; no runtime, storage, entitlement or authentication contract changes.
Change-source: pulse-maintainer
This commit is contained in:
pulse-triage[bot]
2026-10-05 06:47:09 +01:00
parent c0f0474423
commit c420758068
5 changed files with 222 additions and 65 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/API.md
+7 -2
View File
@@ -1548,14 +1548,19 @@ Returns stats for the persistent metrics store (SQLite-backed).
Returns historical metric series for a resource and time range.
Query params:
- `resourceType` (required): `node`, `vm`, `container`, `storage`, `dockerHost`, `dockerContainer`
- `resourceType` (required): `node`, `storage`, `agent`, `disk`, `k8s`, `vm`, `system-container`, `oci-container`, `app-container`, `docker-host`
- `resourceId` (required)
- `metric` (optional): `cpu`, `memory`, `disk`, etc. Omit for all metrics
- `range` (optional): `1h`, `6h`, `12h`, `24h`, `1d`, `7d`, `30d`, `90d` (default `24h`; duration strings also accepted)
- `maxPoints` (optional): Downsample to a target number of points
> **License**: Requests beyond Community's `7d` floor require the paid `long_term_metrics` entitlement. Relay unlocks `14d`, Pro and legacy Pro+ unlock `90d`, and requests beyond the active tier's limit return `402 Payment Required`.
> **Aliases**: `guest` (VM/LXC) and `docker` (Docker container) are accepted, but persistent store data uses the canonical types above.
An explicit `metric` returns a `points` array; omitting it returns a `metrics`
object keyed by metric name. Use the source ID and type from the affected
chart's request, not its display name or an internal store type. Older
`container`, `dockerHost`, `dockerContainer`, `guest` and `docker` query values
are unsupported. See [Metrics History](METRICS_HISTORY.md#api-access) for
private one-shot reads, response sources and empty-history troubleshooting.
---
docs/METRICS_HISTORY.md
+73 -26
View File
@@ -29,6 +29,11 @@ Default retention values (subject to change) are:
## Advanced: Retention Tuning
Do not change retention or polling merely because a chart is empty or writes
are high. First follow [Troubleshooting](#troubleshooting) and preserve the
original observations. Shorter retention removes history; slower polling also
changes how quickly monitoring can notice a problem.
Tiered retention is stored in `system.json` in the Pulse data directory:
- **systemd/LXC installs**: typically `/etc/pulse/system.json`
@@ -49,6 +54,12 @@ After changing these values, restart Pulse.
## Advanced: Disk Write Tuning
This is a deliberate storage trade-off, not a repair for missing history or
unexplained writes. Changing the path opens a different history store; it does
not migrate the existing history. Keep the original durable data and establish
the writer using [bounded performance checks](TROUBLESHOOTING.md#excessive-cpu-writes-or-database-growth)
before changing storage.
Pulse keeps metrics history on disk by default. SSD-sensitive installs can move
only the metrics SQLite database without moving secrets or general config:
@@ -95,9 +106,10 @@ services:
- /metrics-tmpfs:size=512m,uid=1000,gid=1000,mode=0700
```
Using tmpfs makes metrics history ephemeral across restarts. It should not be
used for `/data`, because `/data` also contains config, encrypted credentials,
tokens, and other state that must remain durable.
Using tmpfs loses that history when the RAM-backed mount is discarded, for
example on a host reboot or container recreation. A service restart alone is
not guaranteed to clear it. Do not use tmpfs for `/data`: it also contains
config, encrypted credentials, tokens, and other state that must remain durable.
The aggregation cadence can also be lengthened when an install prefers fewer,
larger rollup writes over more frequent smaller writes:
@@ -123,39 +135,74 @@ These endpoints require authentication with the `monitoring:read` scope.
`GET /api/metrics-store/history` supports:
- `resourceType` (required): `node`, `vm`, `container`, `storage`, `dockerHost`, `dockerContainer`
- `resourceId` (required): resource identifier (for guests use `instance:node:vmid`)
- `resourceType` (required): `node`, `storage`, `agent`, `disk`, `k8s`, `vm`,
`system-container`, `oci-container`, `app-container`, or `docker-host`
- `resourceId` (required): the source identifier used by that chart, not its
display name (for Proxmox guests this is normally `instance:node:vmid`)
- `metric` (optional): `cpu`, `memory`, `disk`, etc. Omit to return all metrics for the resource.
- `range` (optional): `1h`, `6h`, `12h`, `24h`, `1d`, `7d`, `30d`, `90d` (default `24h`; duration strings also accepted)
- `maxPoints` (optional): Downsample to a target number of points
For a one-off read, open `/api/metrics-store/stats` in your signed-in Pulse
browser; do not extract its session cookie. For curl, prepare the private
header file described in [API authentication](API.md#-authentication), using a
token with only `monitoring:read`. The example requires curl 7.76 or later and
runs **on the Pulse host**:
For a one-off check, open the affected resource's History panel in your signed-in
Pulse browser. If an API read is needed, use the path and query from that
panel's request on the same instance; do not extract its session cookie, use
**Copy as cURL**, or share a full network export. `/api/metrics-store/stats`
describes the store overall, not whether this resource has the requested metric.
For curl, follow [API authentication](API.md#-authentication) to prepare the
private header file with only `monitoring:read` and define its `pulse_api`
helper **in the same Bash session**. It is example code, not an installed Pulse
command. The default helper runs **on the Pulse host**; for remote access,
follow that guide's HTTPS-origin and verified-CA instructions. Then make one
read-only request (curl 7.76 or later):
```bash
curl --disable --fail-with-body --connect-timeout 10 --max-time 30 \
--header "@$HOME/.config/pulse/api-header" --get \
--data-urlencode 'resourceType=vm' \
--data-urlencode 'resourceId=pve1:node1:100' \
--data-urlencode 'range=7d' --data-urlencode 'metric=cpu' \
http://127.0.0.1:7655/api/metrics-store/history
pulse_api GET '/api/metrics-store/history?resourceType=vm&resourceId=pve1%3Anode1%3A100&range=7d&metric=cpu'
```
Replace the example resource ID with the ID reported by your instance. For a
remote request, use your Pulse HTTPS address and keep certificate verification
enabled. Keep `--disable` first to ignore local curl defaults that could enable
credential-bearing traces. HTTP errors, including 401, 402 and 403, return a
non-zero exit; do not treat the error body as history. Never put a token in a
command, URL or report, and share only the relevant redacted error rather than
the whole infrastructure response.
Replace the example type and ID with the chart's values, URL-encoding each
query value (`:` becomes `%3A`; an ID's `&` must be `%26`, not another query
parameter). Do not substitute a hostname or strip a provider prefix to guess
an ID. The helper saves each response, including an error body, in a new
owner-only file and prints only the HTTP status and file location. Open that
file privately; do not print or post it. HTTP 401, 402, 403 and server errors,
redirects, or incomplete transport are not an empty history or zero usage.
There is no automatic retry. Never put a token in a command, URL or report;
share only the relevant manually redacted error and bounded observations.
With `metric=cpu` (or another explicit metric), the response has a `points`
array. Omitting `metric` instead returns a `metrics` object keyed by metric
name, not the same single-series shape. Inspect timestamps and `source` too:
`store` is persisted history; `memory` or `live` fallback does not establish
durable historical coverage. An HTTP 200 with no points proves neither zero
usage nor successful collection, and a non-zero current value does not prove
every historical interval is present.
> **License**: Requests beyond Community's `7d` floor require the paid `long_term_metrics` entitlement. Relay unlocks `14d`, Pro and legacy Pro+ unlock `90d`, and requests beyond the active tier's limit return `402 Payment Required`.
> **Aliases**: `guest` (VM/LXC) and `docker` (Docker container) are accepted, but persistent store data uses the canonical types above.
> **Older examples**: `container`, `dockerHost`, `dockerContainer`, `guest` and
> `docker` are not current `resourceType` query values. Use the chart's current
> type rather than a display label or internal storage name.
## Troubleshooting
- **No sparklines / empty history**: confirm the instance can write to the data directory and that `metrics.db` exists.
- **Large disk usage**: reduce polling frequency first. If you need tighter retention, adjust the tiered retention settings in `system.json` (advanced) and restart Pulse.
- **Missing or stale history**: retain the affected panel, metric, range,
collection time and current server/agent versions. Check the connection's
last normal poll or agent report in **Settings → Infrastructure**, without
running Diagnostics, a guest-agent probe or a restart during backup freeze/thaw.
Compare the panel's actual request, HTTP result, point timestamps and source.
A valid connection test, another resource's graph or the existence of
`metrics.db` does not establish this metric's collection or display.
- **Access or query error**: keep it separate from an empty result. Check the
selected organisation, `monitoring:read` access, exact query type/ID and the
requested range. A 402 is a history-range entitlement failure, not evidence
that storage is empty; try an allowed range without buying another licence
or resetting data.
- **Storage warning**: inspect a bounded existing startup/runtime error and
the deployment's persistent mount and service-account access privately.
Follow the [safe troubleshooting guide](TROUBLESHOOTING.md#excessive-cpu-writes-or-database-growth)
for high writes or space pressure. Do not delete history, edit a database,
shorten retention, move data to tmpfs or re-enrol a host as a diagnostic fix.
- **Report still needed**: share the affected metric/range, versions,
whether normal collection advances, and a manually redacted error; keep
resource identities, response files and databases private. Follow
[Getting Help](TROUBLESHOOTING.md#-getting-help), not a full response dump.
frontend-modern/public/docs/API.md
+7 -2
View File
@@ -1548,14 +1548,19 @@ Returns stats for the persistent metrics store (SQLite-backed).
Returns historical metric series for a resource and time range.
Query params:
- `resourceType` (required): `node`, `vm`, `container`, `storage`, `dockerHost`, `dockerContainer`
- `resourceType` (required): `node`, `storage`, `agent`, `disk`, `k8s`, `vm`, `system-container`, `oci-container`, `app-container`, `docker-host`
- `resourceId` (required)
- `metric` (optional): `cpu`, `memory`, `disk`, etc. Omit for all metrics
- `range` (optional): `1h`, `6h`, `12h`, `24h`, `1d`, `7d`, `30d`, `90d` (default `24h`; duration strings also accepted)
- `maxPoints` (optional): Downsample to a target number of points
> **License**: Requests beyond Community's `7d` floor require the paid `long_term_metrics` entitlement. Relay unlocks `14d`, Pro and legacy Pro+ unlock `90d`, and requests beyond the active tier's limit return `402 Payment Required`.
> **Aliases**: `guest` (VM/LXC) and `docker` (Docker container) are accepted, but persistent store data uses the canonical types above.
An explicit `metric` returns a `points` array; omitting it returns a `metrics`
object keyed by metric name. Use the source ID and type from the affected
chart's request, not its display name or an internal store type. Older
`container`, `dockerHost`, `dockerContainer`, `guest` and `docker` query values
are unsupported. See [Metrics History](METRICS_HISTORY.md#api-access) for
private one-shot reads, response sources and empty-history troubleshooting.
---
frontend-modern/public/docs/METRICS_HISTORY.md
+73 -26
View File
@@ -29,6 +29,11 @@ Default retention values (subject to change) are:
## Advanced: Retention Tuning
Do not change retention or polling merely because a chart is empty or writes
are high. First follow [Troubleshooting](#troubleshooting) and preserve the
original observations. Shorter retention removes history; slower polling also
changes how quickly monitoring can notice a problem.
Tiered retention is stored in `system.json` in the Pulse data directory:
- **systemd/LXC installs**: typically `/etc/pulse/system.json`
@@ -49,6 +54,12 @@ After changing these values, restart Pulse.
## Advanced: Disk Write Tuning
This is a deliberate storage trade-off, not a repair for missing history or
unexplained writes. Changing the path opens a different history store; it does
not migrate the existing history. Keep the original durable data and establish
the writer using [bounded performance checks](TROUBLESHOOTING.md#excessive-cpu-writes-or-database-growth)
before changing storage.
Pulse keeps metrics history on disk by default. SSD-sensitive installs can move
only the metrics SQLite database without moving secrets or general config:
@@ -95,9 +106,10 @@ services:
- /metrics-tmpfs:size=512m,uid=1000,gid=1000,mode=0700
```
Using tmpfs makes metrics history ephemeral across restarts. It should not be
used for `/data`, because `/data` also contains config, encrypted credentials,
tokens, and other state that must remain durable.
Using tmpfs loses that history when the RAM-backed mount is discarded, for
example on a host reboot or container recreation. A service restart alone is
not guaranteed to clear it. Do not use tmpfs for `/data`: it also contains
config, encrypted credentials, tokens, and other state that must remain durable.
The aggregation cadence can also be lengthened when an install prefers fewer,
larger rollup writes over more frequent smaller writes:
@@ -123,39 +135,74 @@ These endpoints require authentication with the `monitoring:read` scope.
`GET /api/metrics-store/history` supports:
- `resourceType` (required): `node`, `vm`, `container`, `storage`, `dockerHost`, `dockerContainer`
- `resourceId` (required): resource identifier (for guests use `instance:node:vmid`)
- `resourceType` (required): `node`, `storage`, `agent`, `disk`, `k8s`, `vm`,
`system-container`, `oci-container`, `app-container`, or `docker-host`
- `resourceId` (required): the source identifier used by that chart, not its
display name (for Proxmox guests this is normally `instance:node:vmid`)
- `metric` (optional): `cpu`, `memory`, `disk`, etc. Omit to return all metrics for the resource.
- `range` (optional): `1h`, `6h`, `12h`, `24h`, `1d`, `7d`, `30d`, `90d` (default `24h`; duration strings also accepted)
- `maxPoints` (optional): Downsample to a target number of points
For a one-off read, open `/api/metrics-store/stats` in your signed-in Pulse
browser; do not extract its session cookie. For curl, prepare the private
header file described in [API authentication](API.md#-authentication), using a
token with only `monitoring:read`. The example requires curl 7.76 or later and
runs **on the Pulse host**:
For a one-off check, open the affected resource's History panel in your signed-in
Pulse browser. If an API read is needed, use the path and query from that
panel's request on the same instance; do not extract its session cookie, use
**Copy as cURL**, or share a full network export. `/api/metrics-store/stats`
describes the store overall, not whether this resource has the requested metric.
For curl, follow [API authentication](API.md#-authentication) to prepare the
private header file with only `monitoring:read` and define its `pulse_api`
helper **in the same Bash session**. It is example code, not an installed Pulse
command. The default helper runs **on the Pulse host**; for remote access,
follow that guide's HTTPS-origin and verified-CA instructions. Then make one
read-only request (curl 7.76 or later):
```bash
curl --disable --fail-with-body --connect-timeout 10 --max-time 30 \
--header "@$HOME/.config/pulse/api-header" --get \
--data-urlencode 'resourceType=vm' \
--data-urlencode 'resourceId=pve1:node1:100' \
--data-urlencode 'range=7d' --data-urlencode 'metric=cpu' \
http://127.0.0.1:7655/api/metrics-store/history
pulse_api GET '/api/metrics-store/history?resourceType=vm&resourceId=pve1%3Anode1%3A100&range=7d&metric=cpu'
```
Replace the example resource ID with the ID reported by your instance. For a
remote request, use your Pulse HTTPS address and keep certificate verification
enabled. Keep `--disable` first to ignore local curl defaults that could enable
credential-bearing traces. HTTP errors, including 401, 402 and 403, return a
non-zero exit; do not treat the error body as history. Never put a token in a
command, URL or report, and share only the relevant redacted error rather than
the whole infrastructure response.
Replace the example type and ID with the chart's values, URL-encoding each
query value (`:` becomes `%3A`; an ID's `&` must be `%26`, not another query
parameter). Do not substitute a hostname or strip a provider prefix to guess
an ID. The helper saves each response, including an error body, in a new
owner-only file and prints only the HTTP status and file location. Open that
file privately; do not print or post it. HTTP 401, 402, 403 and server errors,
redirects, or incomplete transport are not an empty history or zero usage.
There is no automatic retry. Never put a token in a command, URL or report;
share only the relevant manually redacted error and bounded observations.
With `metric=cpu` (or another explicit metric), the response has a `points`
array. Omitting `metric` instead returns a `metrics` object keyed by metric
name, not the same single-series shape. Inspect timestamps and `source` too:
`store` is persisted history; `memory` or `live` fallback does not establish
durable historical coverage. An HTTP 200 with no points proves neither zero
usage nor successful collection, and a non-zero current value does not prove
every historical interval is present.
> **License**: Requests beyond Community's `7d` floor require the paid `long_term_metrics` entitlement. Relay unlocks `14d`, Pro and legacy Pro+ unlock `90d`, and requests beyond the active tier's limit return `402 Payment Required`.
> **Aliases**: `guest` (VM/LXC) and `docker` (Docker container) are accepted, but persistent store data uses the canonical types above.
> **Older examples**: `container`, `dockerHost`, `dockerContainer`, `guest` and
> `docker` are not current `resourceType` query values. Use the chart's current
> type rather than a display label or internal storage name.
## Troubleshooting
- **No sparklines / empty history**: confirm the instance can write to the data directory and that `metrics.db` exists.
- **Large disk usage**: reduce polling frequency first. If you need tighter retention, adjust the tiered retention settings in `system.json` (advanced) and restart Pulse.
- **Missing or stale history**: retain the affected panel, metric, range,
collection time and current server/agent versions. Check the connection's
last normal poll or agent report in **Settings → Infrastructure**, without
running Diagnostics, a guest-agent probe or a restart during backup freeze/thaw.
Compare the panel's actual request, HTTP result, point timestamps and source.
A valid connection test, another resource's graph or the existence of
`metrics.db` does not establish this metric's collection or display.
- **Access or query error**: keep it separate from an empty result. Check the
selected organisation, `monitoring:read` access, exact query type/ID and the
requested range. A 402 is a history-range entitlement failure, not evidence
that storage is empty; try an allowed range without buying another licence
or resetting data.
- **Storage warning**: inspect a bounded existing startup/runtime error and
the deployment's persistent mount and service-account access privately.
Follow the [safe troubleshooting guide](TROUBLESHOOTING.md#excessive-cpu-writes-or-database-growth)
for high writes or space pressure. Do not delete history, edit a database,
shorten retention, move data to tmpfs or re-enrol a host as a diagnostic fix.
- **Report still needed**: share the affected metric/range, versions,
whether normal collection advances, and a manually redacted error; keep
resource identities, response files and databases private. Follow
[Getting Help](TROUBLESHOOTING.md#-getting-help), not a full response dump.
scripts/tests/test_setup_access_docs.py
+62 -9
View File
@@ -17,9 +17,11 @@ import stat
import subprocess
import tempfile
import unittest
from urllib.parse import parse_qs, urlsplit
from urllib.parse import parse_qs, quote, urlsplit
from test_api_auth_docs import ROOT, TEST_TOKEN, exercise_curl, recording_server
from test_api_auth_docs import (
ROOT, TEST_TOKEN, exercise_curl, private_recording_server, request_helper,
)
GUIDES = ("KUBERNETES", "METRICS_HISTORY", "CLOUD")
@@ -186,14 +188,20 @@ class SetupAccessDocsTest(unittest.TestCase):
class MetricsAccessDocsHTTPTest(unittest.TestCase):
def history_request(self) -> str:
# Use the same helper readers are told to define, not a fixture-only
# implementation that could conceal a direct-curl response leak.
return request_helper() + "\n" + recipe("METRICS_HISTORY", "pulse_api GET ")
def test_history_query_with_both_headers_and_hostile_curl_defaults(self):
request = recipe("METRICS_HISTORY", "curl ")
request = self.history_request()
expected_id = "pve 1:node/1:100&other=1"
request = request.replace("pve1:node1:100", expected_id)
with recording_server() as (port, records):
request = request.replace("pve1%3Anode1%3A100", quote(expected_id, safe=""))
with private_recording_server() as (port, records):
for key, value in (("X-API-Token", TEST_TOKEN), ("Authorization", "Bearer " + TEST_TOKEN)):
with self.subTest(header=key), tempfile.TemporaryDirectory() as temporary:
result = exercise_curl(self, Path(temporary), f"{key}: {value}", port, request)
result = exercise_curl(self, Path(temporary), f"{key}: {value}", port,
request, private_response=True)
self.assertEqual(result.returncode, 0, result.stderr.decode())
path, headers, method, body = records[-1]
self.assertEqual(urlsplit(path).path, "/api/metrics-store/history")
@@ -205,12 +213,57 @@ class MetricsAccessDocsHTTPTest(unittest.TestCase):
def test_http_auth_license_and_server_errors_fail(self):
for status in (401, 402, 403, 500):
with self.subTest(status=status), recording_server(status) as (port, records), tempfile.TemporaryDirectory() as temporary:
with self.subTest(status=status), private_recording_server(status) as (port, records), tempfile.TemporaryDirectory() as temporary:
result = exercise_curl(self, Path(temporary), f"X-API-Token: {TEST_TOKEN}", port,
recipe("METRICS_HISTORY", "curl "))
self.history_request(), private_response=True)
self.assertEqual(result.returncode, 22, result.stderr.decode())
self.assertEqual(len(records), 1)
self.assertIn(b'{"fixture":true}', result.stdout)
self.assertIn(f"HTTP {status}".encode(), result.stdout)
def test_redirect_and_incomplete_transport_are_not_empty_success(self):
for status, partial, expected_exit in ((302, False, 1), (200, True, 18)):
with self.subTest(status=status, partial=partial), private_recording_server(status, partial) as (port, records), tempfile.TemporaryDirectory() as temporary:
result = exercise_curl(self, Path(temporary), f"X-API-Token: {TEST_TOKEN}", port,
self.history_request(), private_response=True)
self.assertEqual(result.returncode, expected_exit, result.stderr.decode())
self.assertEqual(len(records), 1)
def test_repeated_reads_preserve_earlier_private_responses(self):
with private_recording_server() as (port, records), tempfile.TemporaryDirectory() as temporary:
home = Path(temporary)
for _ in range(2):
result = exercise_curl(self, home, f"X-API-Token: {TEST_TOKEN}", port,
self.history_request(), private_response=True)
self.assertEqual(result.returncode, 0, result.stderr.decode())
self.assertEqual(len(records), 2)
self.assertEqual(len(list((home / ".config/pulse").glob("api-response.*"))), 3)
def test_query_types_match_current_handler_and_both_references(self):
expected = {"node", "storage", "agent", "disk", "k8s", "vm", "system-container",
"oci-container", "app-container", "docker-host"}
handler = (ROOT / "internal/api/router.go").read_text().split(
"func normalizeMetricsHistoryResourceType(", 1)[1].split("\n}\n", 1)[0]
self.assertEqual(set(re.findall(r'case "([^"]+)":', handler)), expected)
for name in ("API", "METRICS_HISTORY"):
with self.subTest(guide=name):
parameters = guide(name).split("### History" if name == "API" else
"### History Query Parameters", 1)[1]
types = parameters.split("- `resourceType` (required):", 1)[1].split(
"- `resourceId`", 1)[0]
self.assertEqual(set(re.findall(r"`([^`]+)`", types)), expected)
def test_guidance_distinguishes_coverage_access_and_safe_collection(self):
content = " ".join(guide("METRICS_HISTORY").split())
for boundary in ("same Bash session", "not an installed Pulse command", "owner-only file",
"not print or post it", "URL-encoding each query value", "no automatic retry",
"a `points` array", "a `metrics` object", "timestamps and `source`",
"does not establish durable historical coverage", "no points proves neither zero",
"existence of `metrics.db` does not establish", "selected organisation",
"not a repair for missing history", "does not migrate the existing history",
"Do not delete history", "backup freeze/thaw", "manually redacted error"):
self.assertIn(boundary, content)
self.assertFalse(any("curl " in block for block in blocks("METRICS_HISTORY")),
"history examples must use the existing private-response helper")
if __name__ == "__main__":