From c4207580689166eb04233a050706416da4a8c362 Mon Sep 17 00:00:00 2001 From: "pulse-triage[bot]" <249995291+pulse-triage[bot]@users.noreply.github.com> Date: Mon, 5 Oct 2026 06:47:09 +0100 Subject: [PATCH] Keep history diagnostics private and match current query types Reuse the tested API helper, explain point/source and access distinctions, and preserve data before storage tuning. Copied examples now exercise private success/error responses, redirects, incomplete transport and encoded IDs. Contract-Neutral: Documentation and recipe checks align with existing history API behaviour; no runtime, storage, entitlement or authentication contract changes. Change-source: pulse-maintainer --- docs/API.md | 9 +- docs/METRICS_HISTORY.md | 99 ++++++++++++++----- frontend-modern/public/docs/API.md | 9 +- .../public/docs/METRICS_HISTORY.md | 99 ++++++++++++++----- scripts/tests/test_setup_access_docs.py | 71 +++++++++++-- 5 files changed, 222 insertions(+), 65 deletions(-) diff --git a/docs/API.md b/docs/API.md index 7c5aea6f7a..296d4cc116 100644 --- a/docs/API.md +++ b/docs/API.md @@ -1548,14 +1548,19 @@ Returns stats for the persistent metrics store (SQLite-backed). Returns historical metric series for a resource and time range. Query params: -- `resourceType` (required): `node`, `vm`, `container`, `storage`, `dockerHost`, `dockerContainer` +- `resourceType` (required): `node`, `storage`, `agent`, `disk`, `k8s`, `vm`, `system-container`, `oci-container`, `app-container`, `docker-host` - `resourceId` (required) - `metric` (optional): `cpu`, `memory`, `disk`, etc. Omit for all metrics - `range` (optional): `1h`, `6h`, `12h`, `24h`, `1d`, `7d`, `30d`, `90d` (default `24h`; duration strings also accepted) - `maxPoints` (optional): Downsample to a target number of points > **License**: Requests beyond Community's `7d` floor require the paid `long_term_metrics` entitlement. Relay unlocks `14d`, Pro and legacy Pro+ unlock `90d`, and requests beyond the active tier's limit return `402 Payment Required`. -> **Aliases**: `guest` (VM/LXC) and `docker` (Docker container) are accepted, but persistent store data uses the canonical types above. +An explicit `metric` returns a `points` array; omitting it returns a `metrics` +object keyed by metric name. Use the source ID and type from the affected +chart's request, not its display name or an internal store type. Older +`container`, `dockerHost`, `dockerContainer`, `guest` and `docker` query values +are unsupported. See [Metrics History](METRICS_HISTORY.md#api-access) for +private one-shot reads, response sources and empty-history troubleshooting. --- diff --git a/docs/METRICS_HISTORY.md b/docs/METRICS_HISTORY.md index 43ee4e4c35..9165500d6d 100644 --- a/docs/METRICS_HISTORY.md +++ b/docs/METRICS_HISTORY.md @@ -29,6 +29,11 @@ Default retention values (subject to change) are: ## Advanced: Retention Tuning +Do not change retention or polling merely because a chart is empty or writes +are high. First follow [Troubleshooting](#troubleshooting) and preserve the +original observations. Shorter retention removes history; slower polling also +changes how quickly monitoring can notice a problem. + Tiered retention is stored in `system.json` in the Pulse data directory: - **systemd/LXC installs**: typically `/etc/pulse/system.json` @@ -49,6 +54,12 @@ After changing these values, restart Pulse. ## Advanced: Disk Write Tuning +This is a deliberate storage trade-off, not a repair for missing history or +unexplained writes. Changing the path opens a different history store; it does +not migrate the existing history. Keep the original durable data and establish +the writer using [bounded performance checks](TROUBLESHOOTING.md#excessive-cpu-writes-or-database-growth) +before changing storage. + Pulse keeps metrics history on disk by default. SSD-sensitive installs can move only the metrics SQLite database without moving secrets or general config: @@ -95,9 +106,10 @@ services: - /metrics-tmpfs:size=512m,uid=1000,gid=1000,mode=0700 ``` -Using tmpfs makes metrics history ephemeral across restarts. It should not be -used for `/data`, because `/data` also contains config, encrypted credentials, -tokens, and other state that must remain durable. +Using tmpfs loses that history when the RAM-backed mount is discarded, for +example on a host reboot or container recreation. A service restart alone is +not guaranteed to clear it. Do not use tmpfs for `/data`: it also contains +config, encrypted credentials, tokens, and other state that must remain durable. The aggregation cadence can also be lengthened when an install prefers fewer, larger rollup writes over more frequent smaller writes: @@ -123,39 +135,74 @@ These endpoints require authentication with the `monitoring:read` scope. `GET /api/metrics-store/history` supports: -- `resourceType` (required): `node`, `vm`, `container`, `storage`, `dockerHost`, `dockerContainer` -- `resourceId` (required): resource identifier (for guests use `instance:node:vmid`) +- `resourceType` (required): `node`, `storage`, `agent`, `disk`, `k8s`, `vm`, + `system-container`, `oci-container`, `app-container`, or `docker-host` +- `resourceId` (required): the source identifier used by that chart, not its + display name (for Proxmox guests this is normally `instance:node:vmid`) - `metric` (optional): `cpu`, `memory`, `disk`, etc. Omit to return all metrics for the resource. - `range` (optional): `1h`, `6h`, `12h`, `24h`, `1d`, `7d`, `30d`, `90d` (default `24h`; duration strings also accepted) - `maxPoints` (optional): Downsample to a target number of points -For a one-off read, open `/api/metrics-store/stats` in your signed-in Pulse -browser; do not extract its session cookie. For curl, prepare the private -header file described in [API authentication](API.md#-authentication), using a -token with only `monitoring:read`. The example requires curl 7.76 or later and -runs **on the Pulse host**: +For a one-off check, open the affected resource's History panel in your signed-in +Pulse browser. If an API read is needed, use the path and query from that +panel's request on the same instance; do not extract its session cookie, use +**Copy as cURL**, or share a full network export. `/api/metrics-store/stats` +describes the store overall, not whether this resource has the requested metric. + +For curl, follow [API authentication](API.md#-authentication) to prepare the +private header file with only `monitoring:read` and define its `pulse_api` +helper **in the same Bash session**. It is example code, not an installed Pulse +command. The default helper runs **on the Pulse host**; for remote access, +follow that guide's HTTPS-origin and verified-CA instructions. Then make one +read-only request (curl 7.76 or later): ```bash -curl --disable --fail-with-body --connect-timeout 10 --max-time 30 \ - --header "@$HOME/.config/pulse/api-header" --get \ - --data-urlencode 'resourceType=vm' \ - --data-urlencode 'resourceId=pve1:node1:100' \ - --data-urlencode 'range=7d' --data-urlencode 'metric=cpu' \ - http://127.0.0.1:7655/api/metrics-store/history +pulse_api GET '/api/metrics-store/history?resourceType=vm&resourceId=pve1%3Anode1%3A100&range=7d&metric=cpu' ``` -Replace the example resource ID with the ID reported by your instance. For a -remote request, use your Pulse HTTPS address and keep certificate verification -enabled. Keep `--disable` first to ignore local curl defaults that could enable -credential-bearing traces. HTTP errors, including 401, 402 and 403, return a -non-zero exit; do not treat the error body as history. Never put a token in a -command, URL or report, and share only the relevant redacted error rather than -the whole infrastructure response. +Replace the example type and ID with the chart's values, URL-encoding each +query value (`:` becomes `%3A`; an ID's `&` must be `%26`, not another query +parameter). Do not substitute a hostname or strip a provider prefix to guess +an ID. The helper saves each response, including an error body, in a new +owner-only file and prints only the HTTP status and file location. Open that +file privately; do not print or post it. HTTP 401, 402, 403 and server errors, +redirects, or incomplete transport are not an empty history or zero usage. +There is no automatic retry. Never put a token in a command, URL or report; +share only the relevant manually redacted error and bounded observations. + +With `metric=cpu` (or another explicit metric), the response has a `points` +array. Omitting `metric` instead returns a `metrics` object keyed by metric +name, not the same single-series shape. Inspect timestamps and `source` too: +`store` is persisted history; `memory` or `live` fallback does not establish +durable historical coverage. An HTTP 200 with no points proves neither zero +usage nor successful collection, and a non-zero current value does not prove +every historical interval is present. > **License**: Requests beyond Community's `7d` floor require the paid `long_term_metrics` entitlement. Relay unlocks `14d`, Pro and legacy Pro+ unlock `90d`, and requests beyond the active tier's limit return `402 Payment Required`. -> **Aliases**: `guest` (VM/LXC) and `docker` (Docker container) are accepted, but persistent store data uses the canonical types above. +> **Older examples**: `container`, `dockerHost`, `dockerContainer`, `guest` and +> `docker` are not current `resourceType` query values. Use the chart's current +> type rather than a display label or internal storage name. ## Troubleshooting -- **No sparklines / empty history**: confirm the instance can write to the data directory and that `metrics.db` exists. -- **Large disk usage**: reduce polling frequency first. If you need tighter retention, adjust the tiered retention settings in `system.json` (advanced) and restart Pulse. +- **Missing or stale history**: retain the affected panel, metric, range, + collection time and current server/agent versions. Check the connection's + last normal poll or agent report in **Settings → Infrastructure**, without + running Diagnostics, a guest-agent probe or a restart during backup freeze/thaw. + Compare the panel's actual request, HTTP result, point timestamps and source. + A valid connection test, another resource's graph or the existence of + `metrics.db` does not establish this metric's collection or display. +- **Access or query error**: keep it separate from an empty result. Check the + selected organisation, `monitoring:read` access, exact query type/ID and the + requested range. A 402 is a history-range entitlement failure, not evidence + that storage is empty; try an allowed range without buying another licence + or resetting data. +- **Storage warning**: inspect a bounded existing startup/runtime error and + the deployment's persistent mount and service-account access privately. + Follow the [safe troubleshooting guide](TROUBLESHOOTING.md#excessive-cpu-writes-or-database-growth) + for high writes or space pressure. Do not delete history, edit a database, + shorten retention, move data to tmpfs or re-enrol a host as a diagnostic fix. +- **Report still needed**: share the affected metric/range, versions, + whether normal collection advances, and a manually redacted error; keep + resource identities, response files and databases private. Follow + [Getting Help](TROUBLESHOOTING.md#-getting-help), not a full response dump. diff --git a/frontend-modern/public/docs/API.md b/frontend-modern/public/docs/API.md index 7c5aea6f7a..296d4cc116 100644 --- a/frontend-modern/public/docs/API.md +++ b/frontend-modern/public/docs/API.md @@ -1548,14 +1548,19 @@ Returns stats for the persistent metrics store (SQLite-backed). Returns historical metric series for a resource and time range. Query params: -- `resourceType` (required): `node`, `vm`, `container`, `storage`, `dockerHost`, `dockerContainer` +- `resourceType` (required): `node`, `storage`, `agent`, `disk`, `k8s`, `vm`, `system-container`, `oci-container`, `app-container`, `docker-host` - `resourceId` (required) - `metric` (optional): `cpu`, `memory`, `disk`, etc. Omit for all metrics - `range` (optional): `1h`, `6h`, `12h`, `24h`, `1d`, `7d`, `30d`, `90d` (default `24h`; duration strings also accepted) - `maxPoints` (optional): Downsample to a target number of points > **License**: Requests beyond Community's `7d` floor require the paid `long_term_metrics` entitlement. Relay unlocks `14d`, Pro and legacy Pro+ unlock `90d`, and requests beyond the active tier's limit return `402 Payment Required`. -> **Aliases**: `guest` (VM/LXC) and `docker` (Docker container) are accepted, but persistent store data uses the canonical types above. +An explicit `metric` returns a `points` array; omitting it returns a `metrics` +object keyed by metric name. Use the source ID and type from the affected +chart's request, not its display name or an internal store type. Older +`container`, `dockerHost`, `dockerContainer`, `guest` and `docker` query values +are unsupported. See [Metrics History](METRICS_HISTORY.md#api-access) for +private one-shot reads, response sources and empty-history troubleshooting. --- diff --git a/frontend-modern/public/docs/METRICS_HISTORY.md b/frontend-modern/public/docs/METRICS_HISTORY.md index 43ee4e4c35..9165500d6d 100644 --- a/frontend-modern/public/docs/METRICS_HISTORY.md +++ b/frontend-modern/public/docs/METRICS_HISTORY.md @@ -29,6 +29,11 @@ Default retention values (subject to change) are: ## Advanced: Retention Tuning +Do not change retention or polling merely because a chart is empty or writes +are high. First follow [Troubleshooting](#troubleshooting) and preserve the +original observations. Shorter retention removes history; slower polling also +changes how quickly monitoring can notice a problem. + Tiered retention is stored in `system.json` in the Pulse data directory: - **systemd/LXC installs**: typically `/etc/pulse/system.json` @@ -49,6 +54,12 @@ After changing these values, restart Pulse. ## Advanced: Disk Write Tuning +This is a deliberate storage trade-off, not a repair for missing history or +unexplained writes. Changing the path opens a different history store; it does +not migrate the existing history. Keep the original durable data and establish +the writer using [bounded performance checks](TROUBLESHOOTING.md#excessive-cpu-writes-or-database-growth) +before changing storage. + Pulse keeps metrics history on disk by default. SSD-sensitive installs can move only the metrics SQLite database without moving secrets or general config: @@ -95,9 +106,10 @@ services: - /metrics-tmpfs:size=512m,uid=1000,gid=1000,mode=0700 ``` -Using tmpfs makes metrics history ephemeral across restarts. It should not be -used for `/data`, because `/data` also contains config, encrypted credentials, -tokens, and other state that must remain durable. +Using tmpfs loses that history when the RAM-backed mount is discarded, for +example on a host reboot or container recreation. A service restart alone is +not guaranteed to clear it. Do not use tmpfs for `/data`: it also contains +config, encrypted credentials, tokens, and other state that must remain durable. The aggregation cadence can also be lengthened when an install prefers fewer, larger rollup writes over more frequent smaller writes: @@ -123,39 +135,74 @@ These endpoints require authentication with the `monitoring:read` scope. `GET /api/metrics-store/history` supports: -- `resourceType` (required): `node`, `vm`, `container`, `storage`, `dockerHost`, `dockerContainer` -- `resourceId` (required): resource identifier (for guests use `instance:node:vmid`) +- `resourceType` (required): `node`, `storage`, `agent`, `disk`, `k8s`, `vm`, + `system-container`, `oci-container`, `app-container`, or `docker-host` +- `resourceId` (required): the source identifier used by that chart, not its + display name (for Proxmox guests this is normally `instance:node:vmid`) - `metric` (optional): `cpu`, `memory`, `disk`, etc. Omit to return all metrics for the resource. - `range` (optional): `1h`, `6h`, `12h`, `24h`, `1d`, `7d`, `30d`, `90d` (default `24h`; duration strings also accepted) - `maxPoints` (optional): Downsample to a target number of points -For a one-off read, open `/api/metrics-store/stats` in your signed-in Pulse -browser; do not extract its session cookie. For curl, prepare the private -header file described in [API authentication](API.md#-authentication), using a -token with only `monitoring:read`. The example requires curl 7.76 or later and -runs **on the Pulse host**: +For a one-off check, open the affected resource's History panel in your signed-in +Pulse browser. If an API read is needed, use the path and query from that +panel's request on the same instance; do not extract its session cookie, use +**Copy as cURL**, or share a full network export. `/api/metrics-store/stats` +describes the store overall, not whether this resource has the requested metric. + +For curl, follow [API authentication](API.md#-authentication) to prepare the +private header file with only `monitoring:read` and define its `pulse_api` +helper **in the same Bash session**. It is example code, not an installed Pulse +command. The default helper runs **on the Pulse host**; for remote access, +follow that guide's HTTPS-origin and verified-CA instructions. Then make one +read-only request (curl 7.76 or later): ```bash -curl --disable --fail-with-body --connect-timeout 10 --max-time 30 \ - --header "@$HOME/.config/pulse/api-header" --get \ - --data-urlencode 'resourceType=vm' \ - --data-urlencode 'resourceId=pve1:node1:100' \ - --data-urlencode 'range=7d' --data-urlencode 'metric=cpu' \ - http://127.0.0.1:7655/api/metrics-store/history +pulse_api GET '/api/metrics-store/history?resourceType=vm&resourceId=pve1%3Anode1%3A100&range=7d&metric=cpu' ``` -Replace the example resource ID with the ID reported by your instance. For a -remote request, use your Pulse HTTPS address and keep certificate verification -enabled. Keep `--disable` first to ignore local curl defaults that could enable -credential-bearing traces. HTTP errors, including 401, 402 and 403, return a -non-zero exit; do not treat the error body as history. Never put a token in a -command, URL or report, and share only the relevant redacted error rather than -the whole infrastructure response. +Replace the example type and ID with the chart's values, URL-encoding each +query value (`:` becomes `%3A`; an ID's `&` must be `%26`, not another query +parameter). Do not substitute a hostname or strip a provider prefix to guess +an ID. The helper saves each response, including an error body, in a new +owner-only file and prints only the HTTP status and file location. Open that +file privately; do not print or post it. HTTP 401, 402, 403 and server errors, +redirects, or incomplete transport are not an empty history or zero usage. +There is no automatic retry. Never put a token in a command, URL or report; +share only the relevant manually redacted error and bounded observations. + +With `metric=cpu` (or another explicit metric), the response has a `points` +array. Omitting `metric` instead returns a `metrics` object keyed by metric +name, not the same single-series shape. Inspect timestamps and `source` too: +`store` is persisted history; `memory` or `live` fallback does not establish +durable historical coverage. An HTTP 200 with no points proves neither zero +usage nor successful collection, and a non-zero current value does not prove +every historical interval is present. > **License**: Requests beyond Community's `7d` floor require the paid `long_term_metrics` entitlement. Relay unlocks `14d`, Pro and legacy Pro+ unlock `90d`, and requests beyond the active tier's limit return `402 Payment Required`. -> **Aliases**: `guest` (VM/LXC) and `docker` (Docker container) are accepted, but persistent store data uses the canonical types above. +> **Older examples**: `container`, `dockerHost`, `dockerContainer`, `guest` and +> `docker` are not current `resourceType` query values. Use the chart's current +> type rather than a display label or internal storage name. ## Troubleshooting -- **No sparklines / empty history**: confirm the instance can write to the data directory and that `metrics.db` exists. -- **Large disk usage**: reduce polling frequency first. If you need tighter retention, adjust the tiered retention settings in `system.json` (advanced) and restart Pulse. +- **Missing or stale history**: retain the affected panel, metric, range, + collection time and current server/agent versions. Check the connection's + last normal poll or agent report in **Settings → Infrastructure**, without + running Diagnostics, a guest-agent probe or a restart during backup freeze/thaw. + Compare the panel's actual request, HTTP result, point timestamps and source. + A valid connection test, another resource's graph or the existence of + `metrics.db` does not establish this metric's collection or display. +- **Access or query error**: keep it separate from an empty result. Check the + selected organisation, `monitoring:read` access, exact query type/ID and the + requested range. A 402 is a history-range entitlement failure, not evidence + that storage is empty; try an allowed range without buying another licence + or resetting data. +- **Storage warning**: inspect a bounded existing startup/runtime error and + the deployment's persistent mount and service-account access privately. + Follow the [safe troubleshooting guide](TROUBLESHOOTING.md#excessive-cpu-writes-or-database-growth) + for high writes or space pressure. Do not delete history, edit a database, + shorten retention, move data to tmpfs or re-enrol a host as a diagnostic fix. +- **Report still needed**: share the affected metric/range, versions, + whether normal collection advances, and a manually redacted error; keep + resource identities, response files and databases private. Follow + [Getting Help](TROUBLESHOOTING.md#-getting-help), not a full response dump. diff --git a/scripts/tests/test_setup_access_docs.py b/scripts/tests/test_setup_access_docs.py index d43f2dba75..834d49a1df 100644 --- a/scripts/tests/test_setup_access_docs.py +++ b/scripts/tests/test_setup_access_docs.py @@ -17,9 +17,11 @@ import stat import subprocess import tempfile import unittest -from urllib.parse import parse_qs, urlsplit +from urllib.parse import parse_qs, quote, urlsplit -from test_api_auth_docs import ROOT, TEST_TOKEN, exercise_curl, recording_server +from test_api_auth_docs import ( + ROOT, TEST_TOKEN, exercise_curl, private_recording_server, request_helper, +) GUIDES = ("KUBERNETES", "METRICS_HISTORY", "CLOUD") @@ -186,14 +188,20 @@ class SetupAccessDocsTest(unittest.TestCase): class MetricsAccessDocsHTTPTest(unittest.TestCase): + def history_request(self) -> str: + # Use the same helper readers are told to define, not a fixture-only + # implementation that could conceal a direct-curl response leak. + return request_helper() + "\n" + recipe("METRICS_HISTORY", "pulse_api GET ") + def test_history_query_with_both_headers_and_hostile_curl_defaults(self): - request = recipe("METRICS_HISTORY", "curl ") + request = self.history_request() expected_id = "pve 1:node/1:100&other=1" - request = request.replace("pve1:node1:100", expected_id) - with recording_server() as (port, records): + request = request.replace("pve1%3Anode1%3A100", quote(expected_id, safe="")) + with private_recording_server() as (port, records): for key, value in (("X-API-Token", TEST_TOKEN), ("Authorization", "Bearer " + TEST_TOKEN)): with self.subTest(header=key), tempfile.TemporaryDirectory() as temporary: - result = exercise_curl(self, Path(temporary), f"{key}: {value}", port, request) + result = exercise_curl(self, Path(temporary), f"{key}: {value}", port, + request, private_response=True) self.assertEqual(result.returncode, 0, result.stderr.decode()) path, headers, method, body = records[-1] self.assertEqual(urlsplit(path).path, "/api/metrics-store/history") @@ -205,12 +213,57 @@ class MetricsAccessDocsHTTPTest(unittest.TestCase): def test_http_auth_license_and_server_errors_fail(self): for status in (401, 402, 403, 500): - with self.subTest(status=status), recording_server(status) as (port, records), tempfile.TemporaryDirectory() as temporary: + with self.subTest(status=status), private_recording_server(status) as (port, records), tempfile.TemporaryDirectory() as temporary: result = exercise_curl(self, Path(temporary), f"X-API-Token: {TEST_TOKEN}", port, - recipe("METRICS_HISTORY", "curl ")) + self.history_request(), private_response=True) self.assertEqual(result.returncode, 22, result.stderr.decode()) self.assertEqual(len(records), 1) - self.assertIn(b'{"fixture":true}', result.stdout) + self.assertIn(f"HTTP {status}".encode(), result.stdout) + + def test_redirect_and_incomplete_transport_are_not_empty_success(self): + for status, partial, expected_exit in ((302, False, 1), (200, True, 18)): + with self.subTest(status=status, partial=partial), private_recording_server(status, partial) as (port, records), tempfile.TemporaryDirectory() as temporary: + result = exercise_curl(self, Path(temporary), f"X-API-Token: {TEST_TOKEN}", port, + self.history_request(), private_response=True) + self.assertEqual(result.returncode, expected_exit, result.stderr.decode()) + self.assertEqual(len(records), 1) + + def test_repeated_reads_preserve_earlier_private_responses(self): + with private_recording_server() as (port, records), tempfile.TemporaryDirectory() as temporary: + home = Path(temporary) + for _ in range(2): + result = exercise_curl(self, home, f"X-API-Token: {TEST_TOKEN}", port, + self.history_request(), private_response=True) + self.assertEqual(result.returncode, 0, result.stderr.decode()) + self.assertEqual(len(records), 2) + self.assertEqual(len(list((home / ".config/pulse").glob("api-response.*"))), 3) + + def test_query_types_match_current_handler_and_both_references(self): + expected = {"node", "storage", "agent", "disk", "k8s", "vm", "system-container", + "oci-container", "app-container", "docker-host"} + handler = (ROOT / "internal/api/router.go").read_text().split( + "func normalizeMetricsHistoryResourceType(", 1)[1].split("\n}\n", 1)[0] + self.assertEqual(set(re.findall(r'case "([^"]+)":', handler)), expected) + for name in ("API", "METRICS_HISTORY"): + with self.subTest(guide=name): + parameters = guide(name).split("### History" if name == "API" else + "### History Query Parameters", 1)[1] + types = parameters.split("- `resourceType` (required):", 1)[1].split( + "- `resourceId`", 1)[0] + self.assertEqual(set(re.findall(r"`([^`]+)`", types)), expected) + + def test_guidance_distinguishes_coverage_access_and_safe_collection(self): + content = " ".join(guide("METRICS_HISTORY").split()) + for boundary in ("same Bash session", "not an installed Pulse command", "owner-only file", + "not print or post it", "URL-encoding each query value", "no automatic retry", + "a `points` array", "a `metrics` object", "timestamps and `source`", + "does not establish durable historical coverage", "no points proves neither zero", + "existence of `metrics.db` does not establish", "selected organisation", + "not a repair for missing history", "does not migrate the existing history", + "Do not delete history", "backup freeze/thaw", "manually redacted error"): + self.assertIn(boundary, content) + self.assertFalse(any("curl " in block for block in blocks("METRICS_HISTORY")), + "history examples must use the existing private-response helper") if __name__ == "__main__":