pulse/docs/METRICS_HISTORY.md

# Metrics History (Persistent)

Pulse persists metrics history to disk so trend views and sparklines survive restarts.

## Read a History panel

In current drawers, open the affected resource's **Overview** and use **History
range** beside its charts. Proxmox guests and nodes, Docker hosts and resource
drawers (including PBS) show stored charts there instead of a separate History
tab. If your installed version still has a **History** tab, use that tab; do not
upgrade or restart just to follow this guide. A resource without a supported
history target may have no chart.

**On a phone**, drawers can show only **Utilization** and panels with active
alerts. Use the **Show …** button below the charts to reveal the others, for
example **Show Network I/O, Disk I/O, Thermals**. Its labels name the currently
collapsed panels, so they vary by resource and alert. **Show fewer charts**
collapses them again. Wider layouts show all panels and may have no button;
older installed layouts may differ.

This button changes only the layout, preserving **History range**; it does not
refresh history or collect new measurements. A collapsed panel is not missing
data or measured zero. After revealing it, **No stored history in this range**
means that panel has no plotted stored samples for the selected range, not zero
usage or proof that collection stopped. Loading, an access error, a locked range
and **History unavailable** are different states: retain the existing message
and range. Do not refresh, run Diagnostics or restart just to reveal a chart.

These charts are separate from **Alerts → History**, table sparklines and
current summary cards. A current card or a short trend preview does not
establish the observations in the selected stored range. For a PBS host, use
the [Backup Server chart](PBS.md#pbs-is-connected-but-history-stays-empty),
not its datastore or the VM that happens to host it.

Start with the existing chart, not a restart or a storage change. A current
reading does not prove that it was stored, and a selected time range does not
prove continuous coverage. **Window start** and **Window end** bound the
requested interval; they are not the first and last successful observations.

VM and LXC drawer panels can show these labels. Other charts and older Pulse
versions may differ; if a time or source is not shown, keep it unknown rather
than borrowing one from the current overview.

| Display | How to read it | What it does not establish |
| --- | --- | --- |
| **Latest stored** | The newest stored observation time in this panel. Each value belongs to that time; a dash means no stored observation for that metric at that time. | Not necessarily now, every metric's newest value, or another panel's time. |
| **Stored** while inspecting | Values at the selected stored observation time. | Not a live reading or an interpolated observation between points. |
| **current**, **last known** or **freshness unknown** with no stored history | A separately qualified fallback reading. Keep its label and explanation. | Not proof that the value was persisted or belongs to a past incident. |
| **Single observation. No trend yet.** | One stored time in that panel. | Not a flat trend or sustained recovery. |
| **Missing observations** | Some series lack readings at other stored times in that panel; lines stop across those gaps. | Not the cause or duration of a monitoring outage. Missing is not measured zero. |
| **Shaded low–high band** | The recorded minimum and maximum within an aggregated bucket, when supplied; the line shows its average. | Not continuous coverage, every original sample, or the duration of a spike. |
| **peak** while inspecting | That metric's maximum in the inspected bucket, when the panel shows it beside the average. | Not necessarily the highest value in the whole selected range or the value at an alert's exact time. |

For example, CPU **22% at 12:02** and memory **95% at 12:01** are not one
observation. At **Latest stored · 12:02**, memory is a dash if there is no
memory point at 12:02; the earlier 95% remains inspectable. Network and disk
I/O panels have their own times, not a shared snapshot of the guest.

An aggregated CPU bucket can show **22% average, 10% to 95%**. The line at 22%
does not rule out the recorded 95% peak, and the band does not say how long
that peak lasted. No visible band may mean a single reading, equal bounds or
unavailable bounds; it does not prove that usage stayed constant. Older panels
may not show these bounds. Keep that limitation rather than inferring a peak
from the line alone.

1. Keep the same resource, metric and range when comparing existing readings.
   **Inspect history**, when offered, steps through stored times: focus the
   slider and use the arrow keys, **Home** or **End**, or move it by touch.
   Hover is another way to inspect; it is not required. Leaving the slider
   returns to the latest stored display.
2. Keep the displayed observation time and qualification with the value.
   Stored points can be aggregated or downsampled; they are not a measurement
   of every instant in the selected range. A page refresh time is not the
   observation time. If an absolute time is unavailable, retain that limit.
3. Compare an alert or Patrol finding with the original episode, not just
   today's overview. Today's source cannot identify the source of a past
   warning. Follow [guest memory interpretation](TROUBLESHOOTING.md#guest-memory-is-high-but-available-memory-is-plentiful)
   when unlike readings suggest memory pressure.

Use already available observations; do not run Diagnostics or guest-agent
probes, restart a guest, repeat a backup, clear History or change retention to
make a comparison agree. An unresponsive guest or an uncertain freeze/thaw
operation needs [Backup safety](VM_DISK_MONITORING.md#backup-safety), not chart
inspection. When reporting a display disagreement, keep the panel, time,
range, labels and relevant values with consistent private aliases; do not
share raw responses, credentials or a full network export.

## Retention and accessible range

Storage retention and the range your running instance can serve are separate.
Community supports up to **7 days**; Pro and legacy Pro+ entitlements support
up to **90 days**. The panel uses the active runtime's history limit, not the
length of time the database has existed. A longer retention setting does not
grant access, and a licence cannot recover observations that were never stored.

**Relay is no longer sold.** Continuing Relay subscribers receive Pro features,
including 90-day history, at their existing price. Older Relay payloads can
still show a 14-day compatibility limit; that is not a plan to buy or proof
that a continuing subscription has received its current entitlement. See
[plans and legacy subscriptions](PULSE_PRO.md#self-hosted-commercial-model).

An API request beyond the active limit returns **402 Payment Required**, not
an empty history or a failed collector. Use an allowed range and the existing
licence/range display first; do not buy another licence, reset history or
reinstall merely to diagnose it. Storage defaults below do not guarantee that
every resource has data for every allowed range.

## Storage Location

Metrics history is stored in a SQLite database named `metrics.db` under the Pulse data directory:

- **systemd/LXC installs**: typically `/etc/pulse/metrics.db`
- **Docker/Kubernetes installs**: typically `/data/metrics.db`

## Retention Model (Tiered)

Pulse keeps multiple resolutions of the same data, which allows longer history without storing raw samples forever:

- **Raw** (high-resolution, short window)
- **Minute aggregates**
- **Hourly aggregates**
- **Daily aggregates**

Default retention values (subject to change) are:

- Raw: 2 hours
- Minute: 24 hours
- Hourly: 7 days
- Daily: 90 days
- Rollups: every 15 minutes by default, bounded so rollups still run well
  before raw samples expire

## Advanced: Retention Tuning

Do not change retention or polling merely because a chart is empty or writes
are high. First follow [Troubleshooting](#troubleshooting) and preserve the
original observations. Shorter retention removes history; slower polling also
changes how quickly monitoring can notice a problem.

Tiered retention is stored in `system.json` in the Pulse data directory:

- **systemd/LXC installs**: typically `/etc/pulse/system.json`
- **Docker/Kubernetes installs**: typically `/data/system.json`

Keys:

```json
{
  "metricsRetentionRawHours": 2,
  "metricsRetentionMinuteHours": 24,
  "metricsRetentionHourlyDays": 7,
  "metricsRetentionDailyDays": 90
}
```

After changing these values, restart Pulse.

## Advanced: Disk Write Tuning

This is a deliberate storage trade-off, not a repair for missing history or
unexplained writes. Changing the path opens a different history store; it does
not migrate the existing history. Keep the original durable data and establish
the writer using [bounded performance checks](TROUBLESHOOTING.md#excessive-cpu-writes-or-database-growth)
before changing storage.

Pulse keeps metrics history on disk by default. SSD-sensitive installs can move
only the metrics SQLite database without moving secrets or general config:

```bash
PULSE_METRICS_DB_PATH=/dev/shm/pulse/metrics.db
```

Use a dedicated directory owned by the account running Pulse. Do not place the
database directly in a shared parent such as `/tmp/metrics.db`: Pulse secures
the database directory to mode `0700` and will reject a directory owned by
another user. On systemd installs, `PrivateTmp=true` also means a path below
`/tmp` is private to the service and will not appear in the host's `/tmp`.

The default systemd sandbox permits `/dev/shm/pulse` and Pulse can create that
subdirectory itself. A dedicated mount elsewhere, such as `/mnt/ramdisk`, also
needs an explicit writable-path grant:

```bash
sudo install -d -o pulse -g pulse -m 0700 /mnt/ramdisk/pulse
sudo systemctl edit pulse
```

Add this drop-in (replace `pulse` with the installed unit name if different):

```ini
[Service]
ReadWritePaths=/mnt/ramdisk/pulse
```

Then set `PULSE_METRICS_DB_PATH=/mnt/ramdisk/pulse/metrics.db` in
`/etc/pulse/.env` and restart Pulse. `ProtectSystem=strict` deliberately keeps
paths outside the unit's writable allowlist read-only even when Unix ownership
would otherwise permit writes.

For Docker, mount a tmpfs at the selected directory and keep `/data` on a
persistent volume:

```yaml
services:
  pulse:
    environment:
      PULSE_METRICS_DB_PATH: /metrics-tmpfs/metrics.db
    tmpfs:
      - /metrics-tmpfs:size=512m,uid=1000,gid=1000,mode=0700
```

Using tmpfs loses that history when the RAM-backed mount is discarded, for
example on a host reboot or container recreation. A service restart alone is
not guaranteed to clear it. Do not use tmpfs for `/data`: it also contains
config, encrypted credentials, tokens, and other state that must remain durable.

The aggregation cadence can also be lengthened when an install prefers fewer,
larger rollup writes over more frequent smaller writes:

```bash
PULSE_METRICS_ROLLUP_INTERVAL=30m
```

Values below 5 minutes are ignored. Values longer than half of the raw-retention
window are capped by the metrics store so raw samples are still rolled up before
retention pruning can remove them.

## API Access

Pulse exposes the persistent metrics store via:

- `GET /api/metrics-store/stats`
- `GET /api/metrics-store/history`

These endpoints require authentication with the `monitoring:read` scope.

### History Query Parameters

`GET /api/metrics-store/history` supports:

- `resourceType` (required): `node`, `storage`, `agent`, `disk`, `k8s`, `vm`,
  `system-container`, `oci-container`, `app-container`, or `docker-host`
- `resourceId` (required): the source identifier used by that chart, not its
  display name (for Proxmox guests this is normally `instance:node:vmid`)
- `metric` (optional): `cpu`, `memory`, `disk`, etc. Omit to return all metrics for the resource.
- `range` (optional): `1h`, `6h`, `12h`, `24h`, `1d`, `7d`, `30d`, `90d` (default `24h`; duration strings also accepted)
- `maxPoints` (optional): Downsample to a target number of points

For a one-off check, open the affected resource's History panel in your signed-in
Pulse browser. If an API read is needed, use the path and query from that
panel's request on the same instance; do not extract its session cookie, use
**Copy as cURL**, or share a full network export. `/api/metrics-store/stats`
describes the store overall, not whether this resource has the requested metric.

For curl, follow [API authentication](API.md#-authentication) to prepare the
private header file with only `monitoring:read` and define its `pulse_api`
helper **in the same Bash session**. It is example code, not an installed Pulse
command. The default helper runs **on the Pulse host**; for remote access,
follow that guide's HTTPS-origin and verified-CA instructions. Then make one
read-only request (curl 7.76 or later):

```bash
pulse_api GET '/api/metrics-store/history?resourceType=vm&resourceId=pve1%3Anode1%3A100&range=7d&metric=cpu'
```

Replace the example type and ID with the chart's values, URL-encoding each
query value (`:` becomes `%3A`; an ID's `&` must be `%26`, not another query
parameter). Do not substitute a hostname or strip a provider prefix to guess
an ID. The helper saves each response, including an error body, in a new
owner-only file and prints only the HTTP status and file location. Open that
file privately; do not print or post it. HTTP 401, 402, 403 and server errors,
redirects, or incomplete transport are not an empty history or zero usage.
There is no automatic retry. Never put a token in a command, URL or report;
share only the relevant manually redacted error and bounded observations.

With `metric=cpu` (or another explicit metric), the response has a `points`
array. Omitting `metric` instead returns a `metrics` object keyed by metric
name, not the same single-series shape. Inspect timestamps and `source` too:
`store` is persisted history; `memory` or `live` fallback does not establish
durable historical coverage. An HTTP 200 with no points proves neither zero
usage nor successful collection, and a non-zero current value does not prove
every historical interval is present.

> **History limit**: Requests beyond the active runtime's allowed range return
> `402 Payment Required`. See [retention and accessible range](#retention-and-accessible-range);
> stored retention is not access or proof of collected observations.
> **Older examples**: `container`, `dockerHost`, `dockerContainer`, `guest` and
> `docker` are not current `resourceType` query values. Use the chart's current
> type rather than a display label or internal storage name.

## Troubleshooting

- **Missing or stale history**: retain the affected panel, metric, range,
  collection time and current server/agent versions. Check the connection's
  last normal poll or agent report in **Settings → Infrastructure**, without
  running Diagnostics, a guest-agent probe or a restart during backup freeze/thaw.
  Compare the panel's actual request, HTTP result, point timestamps and source.
  A valid connection test, another resource's graph or the existence of
  `metrics.db` does not establish this metric's collection or display.
- **Access or query error**: keep it separate from an empty result. Check the
  selected organisation, `monitoring:read` access, exact query type/ID and the
  requested range. A 402 is a history-range entitlement failure, not evidence
  that storage is empty; try an allowed range without buying another licence
  or resetting data.
- **Storage warning**: inspect a bounded existing startup/runtime error and
  the deployment's persistent mount and service-account access privately.
  Follow the [safe troubleshooting guide](TROUBLESHOOTING.md#excessive-cpu-writes-or-database-growth)
  for high writes or space pressure. Do not delete history, edit a database,
  shorten retention, move data to tmpfs or re-enrol a host as a diagnostic fix.
- **Report still needed**: share the affected metric/range, versions,
  whether normal collection advances, and a manually redacted error; keep
  resource identities, response files and databases private. Follow
  [Getting Help](TROUBLESHOOTING.md#-getting-help), not a full response dump.