pulse/docs/TROUBLESHOOTING.md

# 🔧 Troubleshooting Guide

## Choose the affected workflow

Start with the symptom below, using observations you already have. A restart,
reinstall or new diagnostic run is not a prerequisite for getting help.

**If a guest is unresponsive, a backup is in freeze/thaw, or an earlier guest
request's state is uncertain, read [Backup safety](VM_DISK_MONITORING.md#backup-safety)
first.** Do not run Diagnostics, force guest-agent probes or repeat a backup to
fill a report. A successful backup task or responsive console alone does not
establish safe recovery.

- **Cannot sign in:** use [local password recovery](TROUBLESHOOTING.md#i-forgot-my-password)
  or [401 and login failures](TROUBLESHOOTING.md#cannot-login--401-unauthorized),
  according to the existing authentication method. Do not delete configuration
  or repeat first-time setup.
- **Update failed or its result is uncertain:** start with
  [Pulse server update checks](AUTO_UPDATE.md#update-failed). An update of a
  monitored container is a different job; use
  [Docker workload update checks](DOCKER.md#check-a-failed-or-pending-workload-update).
  Preserve the current state and rollback evidence; do not repeat the attempt.
- **Agent host is missing or reporting is rejected:** check the existing
  [agent authentication and reporting evidence](UNIFIED_AGENT.md#agent-lookup-returns-http-401-or-403)
  before replacing a token or reinstalling. Installer success is not an accepted
  report.
- **Copied agent install or removal command failed:** distinguish its
  [preparation, private prompt and lifecycle action](UNIFIED_AGENT.md#copied-agent-command-fails-before-installation).
  A stopped new command does not diagnose the existing agent. Keep state and
  credentials; do not repeat removal or installation to obtain an error.
- **Guest readings are missing or disagree:** distinguish
  [current, retained and unavailable disks](VM_DISK_MONITORING.md#current-retained-and-unavailable-disk-readings)
  from [guest memory pressure](TROUBLESHOOTING.md#guest-memory-is-high-but-available-memory-is-plentiful).
  Unknown is not zero, and a working Pulse Agent view does not prove API recovery.
- **Proxmox data belongs to the wrong host, or healthy jobs look Pending:** use
  [cross-installation identity checks](TROUBLESHOOTING.md#monitoring-is-mixed-between-proxmox-installations)
  or [replication status checks](TROUBLESHOOTING.md#replication-jobs-are-pending-stale-or-missing).
  Do not rename, remove or re-enrol resources to create evidence.
- **TrueNAS readings are stale or absent:** use
  [TrueNAS data checks](TRUENAS.md#stale-truenas-data), keeping the platform API
  and Pulse Agent paths separate.
- **PBS is Active but missing from monitoring:** distinguish the
  [server, backup inventory and History](PBS.md#pbs-is-active-but-missing-from-monitoring).
  An Active API connection does not establish a visible PBS resource; installing
  an agent does not prove API recovery.
- **Pulse itself uses more memory, CPU or disk writes:** use
  [memory baseline and growth checks](TROUBLESHOOTING.md#memory-use-keeps-growing)
  or [CPU and write-rate checks](TROUBLESHOOTING.md#excessive-cpu-writes-or-database-growth).
  Guest usage and database size are different measurements.
- **Notification Test works but alerts do not arrive:** follow
  [ordinary delivery checks](TROUBLESHOOTING.md#test-succeeds-but-real-alerts-are-missing).
  Do not trigger an alert or replay the queue just to complete a report.
- **Private information was posted:** follow
  [disclosure containment](TROUBLESHOOTING.md#if-you-already-posted-private-information).
  Deleting the public copy does not revoke a credential.

If none fits, use [Getting Help](TROUBLESHOOTING.md#-getting-help).
Review evidence locally before sharing it; unknown or unavailable details are
valid, and a full configuration, journal or network export is not required.

## ⚡ Quick Fixes

### I forgot my password

Pulse does not provide an email password-reset flow. Choose the path that
matches how this self-hosted instance authenticates:

- **Local Pulse administrator username and password**: recovery requires
  authorised access to the host or deployment's credential source. Follow the
  steps below; keep the existing username.
- **OIDC, SAML, or proxy authentication**: contact the identity-provider or
  Pulse administrator. If the administrator deliberately kept local login as a
  fallback, they can open the Pulse URL with `?show_local=true`; this only
  reveals the existing local form and does not reset credentials.
- **Temporary lockout**: wait for the displayed lockout to expire. An authorised
  administrator can use the [manual recovery API](../SECURITY.md#manual-recovery-admin);
  this does not reset a password or bypass SSO.

#### Recover the existing local administrator login

Do not delete `.env` or repeat first-time setup to recover a password. The file
can also hold deployment settings, and setup replaces the primary API token.
Replace only the password in its active credential source instead, preserving
the username, API tokens, encryption keys, monitoring configuration and data.
This is login recovery, not a complete response to a compromised account;
existing sessions and API tokens need separate review if credentials leaked.

1. **Find the active credential source locally.** Deployment-supplied
   `PULSE_AUTH_USER` and `PULSE_AUTH_PASS` take precedence over Pulse's `.env`.
   Do not post full service units, container inspections or resolved Compose
   configuration; they can contain secrets.
   - **Docker**: the generated file is in the mounted data directory, normally
     `/data/.env` inside the container. If Compose, `--env-file`, Kubernetes or
     another manager supplies the password, edit that managed source instead.
   - **Systemd**: check the active service's unit, environment files and
     drop-ins privately. Root-run setup can also save credentials in
     `/etc/systemd/system/pulse.service.d/override.conf` (or the corresponding
     `pulse-backend.service.d` path on legacy installs). Editing only
     `/etc/pulse/.env` will not override those values. A custom
     `PULSE_DATA_DIR` changes the generated file's location.
   - **Proxmox LXC**: perform the systemd steps inside the Pulse container, not
     on the Proxmox host. Use its console or `pct enter <ctid>`.
2. **Prepare a new password hash privately.** Use a trusted local bcrypt tool
   that prompts for a password of at least 12 characters, never one that needs
   the password in its command arguments. For example, if Apache's `htpasswd`
   is installed on your trusted administration machine:

   ```bash
   (
   set -eu
   umask 077
   recovery_dir="$(mktemp -d "${TMPDIR:-/tmp}/pulse-password.XXXXXX")"
   htpasswd -nB -C 12 pulse-recovery > "$recovery_dir/password-record"
   printf 'Private password record saved in %s/password-record\n' "$recovery_dir"
   )
   ```

   This prints only the file location. Open the record in a private editor and
   copy the complete 60-character hash after `pulse-recovery:`; that label is
   not a new Pulse username. Do not post the record or hash. If the tool fails,
   stop rather than saving a partial hash or putting the password in a shell
   command.
3. **Back up and edit only the active password setting.** Keep an owner-only
   backup of each file you change, then replace `PULSE_AUTH_PASS` in a private
   editor. Keep `PULSE_AUTH_USER` and unrelated settings unchanged. In a Pulse
   `.env`, put the hash in single quotes with literal `$` characters. Compose
   YAML has different interpolation rules; follow the
   [authentication guide](CONFIGURATION.md#private-docker-authentication-file)
   for the source you actually use. If a systemd drop-in supplies the password,
   update it and the generated authentication file consistently so a later
   restart or reload does not restore the old value. Preserve file permissions
   and ownership; hashes and backups are sensitive too.
4. **Apply through the existing deployment.** For a generated Docker data-file
   change, restart the same container. Changing Docker's managed environment
   requires its recreate/redeploy operation, not just `docker restart`; preserve
   the same image, mounted data and other settings. For systemd, reload the
   service manager after a unit or drop-in change, then restart the active Pulse
   service during a suitable maintenance window. Do not remove a data volume,
   reinstall Pulse or re-enrol agents.
5. **Verify the login.** Use a fresh browser session to sign in with the same
   local administrator username and new password, then check that monitoring
   and agent connections remain intact. If it still fails, stop and reconcile
   the effective source and any lockout; do not delete more state. Restore the
   private backup through the same deployment path if you need to undo the
   change. Remove the temporary password record after verification,
   and retain or dispose of the backup under your normal credential policy.

#### A fresh install's bootstrap token was missed

If no local password has ever been set, there is nothing to reset. Follow
[first login](INSTALL.md#step-1-get-the-token) to read the existing bootstrap
token for that instance, using its actual data directory. Enter it only in the
Pulse setup screen, not in command arguments, URLs or issue reports.

### Port change didn't take effect

The web UI and API listen on `FRONTEND_PORT` (default `7655`). The deprecated
`PORT` alias applies only when `FRONTEND_PORT` is unset; changing
`frontendPort` in `system.json` has no effect. `PULSE_AGENT_INGEST_PORT` is a
separate agent listener, not the web UI port. See
[Port configuration](CONFIGURATION.md#common-overrides-environment-variables).

- **Systemd / Proxmox LXC**: identify the active service with
  `systemctl is-active pulse` (legacy installs may use `pulse-backend`). Inspect
  only the port setting in its managed configuration locally. If you change a
  unit or drop-in, run `sudo systemctl daemon-reload`, then restart the affected
  service during a suitable maintenance window. For LXC, run these checks inside
  the Pulse container, not on the Proxmox host.
- **Docker / Compose**: distinguish the published host port from the listener
  inside the container. With the default listener, `8080:7655` exposes the UI on
  host port `8080`; changing the host port does not require `FRONTEND_PORT`.
  In the repository's Compose file, `PULSE_PORT` controls this host-side mapping.
  Save the mapping in your existing Compose project and apply it with
  `docker compose up -d pulse`, keeping the same image and mounted data volume.
  Restarting an existing container does not apply a new port mapping. For other
  container managers, use their recreate/redeploy action while preserving the
  data mount; do not delete the volume.
- **Reverse proxy**: check its upstream port and the firewall separately. A
  working agent connection on a split-port deployment does not prove that the
  web UI is reachable.

Do not post full service environments, `docker inspect` output or resolved
`docker compose config` output: they can include passwords and tokens. Share
only the relevant port numbers and a redacted error if help is needed.

### "Connection Refused"

Identify which connection failed before changing ports or credentials:

- **Your browser cannot open Pulse:** check that the intended Pulse service or
  container is running, then its actual listening port, container port mapping
  and reverse proxy target. Pulse's default port is **7655**, but an existing
  deployment can use a different one. Use the [port-change checks](#port-change-didnt-take-effect)
  and [reverse proxy guide](REVERSE_PROXY.md) for that path; do not broadly open
  firewall access or expose Pulse directly to bypass a proxy.
- **Pulse opens, but a monitored platform request is refused:** check the
  affected connection's saved hostname, scheme and port in **Settings →
  Infrastructure**, and the network path **from the Pulse server**. Browser
  access to the platform is a different path. Proxmox VE normally uses HTTPS
  on **8006**; PBS uses HTTPS on **8007**. Other platforms use their saved
  endpoint, not an assumed Proxmox port. Changing Pulse's listening port does
  not repair this connection.

A connection refusal is not an authentication response (**401/403**) or a
certificate-validation error. Keep the original time and redacted error;
do not replace credentials or disable TLS verification on a refusal alone.
For a page that opens but has missing readings, use the
[missing-data checks](FAQ.md#no-data-showing) instead of resetting the connection.

### CORS errors

CORS controls whether a browser can read a response from a different origin.
It is not Pulse's connection to Proxmox, PBS or TrueNAS, and it does not replace
authentication, CSRF protection or TLS.

In your browser's developer tools, compare the page's origin with the failed
API request's origin and HTTP status. An origin consists of **scheme, host and
port**: HTTP and HTTPS, or two different ports, are different origins. Review
the original console error locally; a certificate failure, login redirect or
401/403 response needs its own correction, not a broader CORS allowlist.

- **Pulse UI and API share a public origin:** a same-origin reverse proxy needs
  no CORS exception. Keep the API behind that proxy and correct its routing or
  [trusted forwarded headers](REVERSE_PROXY.md#before-configuring-the-proxy);
  do not expose the backend or add wildcard response headers to bypass it.
- **A separate trusted browser app needs API access:** in **Settings → System →
  Network → CORS Allowed Origins**, list only that app's exact origin, for
  example `https://app.example.com:8443`. Do not include a path, trailing slash
  or a hostname pattern. Multiple exact origins are comma-separated.
  `ALLOWED_ORIGINS` overrides the saved setting; removing that environment
  override does not erase an existing saved allowlist. Re-open settings after
  saving and verify the effective value and the ordinary browser request.

An empty policy grants no cross-origin browser permission. `*` allows any
origin **without credentialed browser access**; it cannot fix a request that
needs a browser session cookie. Do not disable authentication, CSRF protection
or TLS verification to make such a request work. Iframe embedding and proxy
authentication have separate settings; CORS is not their trust boundary.

Keep cookies, authorization headers, API tokens and full network exports
private. A report needs only the redacted error, HTTP status and relevant
origins, not a credential-bearing request or HAR file.

---

## 🔍 Common Issues

### Authentication

#### "Invalid username or password" after setup

When the local login form reports this error, stop repeated password guesses:
failed attempts count against both the username and client IP. Privately check
the existing username and the active credential source. Deployment-supplied
`PULSE_AUTH_USER` and `PULSE_AUTH_PASS` take precedence over Pulse's generated
`.env`, so changing only that file may leave the running password unchanged.

A bcrypt hash must be complete (60 characters). Compose YAML interpolation,
Docker `--env-file` and Pulse's generated `.env` use different quoting rules;
do not blindly replace every `$` with `$$`. Follow the
[authentication-file guidance](CONFIGURATION.md#private-docker-authentication-file)
for your actual source, keeping hashes and resolved deployment configuration
private. If the password is genuinely lost, use
[deployment-specific recovery](#i-forgot-my-password), not setup or re-enrolment.

#### Cannot login / 401 Unauthorized

Identify which request failed before clearing cookies or changing credentials:

- **Browser session:** a 401 on an ordinary UI request can mean a missing or
  expired session, not a wrong password. Open the same public Pulse URL in a
  fresh browser session and sign in through the configured method. Preserve any
  working administrator session; do not clear unrelated sites' cookies.
- **SSO or proxy login:** use the identity-provider or Pulse administrator's
  recovery path. Check the [proxy authentication guide](PROXY_AUTH.md) when the
  proxy signs you in but Pulse rejects the request. A local lockout reset does
  not repair an identity-provider account or missing trusted proxy headers.
- **Local login form:** an **Account locked** response gives the remaining wait.
  Lockout applies separately to username and client IP, normally for 15 minutes;
  clearing cookies does not reset it. A **Too many requests** response (429) is
  rate limiting, not proof that the password is wrong. Wait rather than retrying
  rapidly, restarting Pulse or disabling authentication.
- **API client:** a missing or invalid API token can cause 401 even when browser
  login works. Check the token's status and intended permissions privately.
  A 403 can instead mean insufficient permissions or failed CSRF protection;
  do not disable those checks or reset the password to repair API access. Use
  the [private header-file examples](../SECURITY.md#usage), never a token or
  session cookie pasted into a command or URL.

Manual lockout reset requires an authenticated administrator with
`settings:write` authority; session requests also require CSRF protection. The
[manual recovery API](../SECURITY.md#manual-recovery-admin) resets one username
or IP identifier at a time, not a password, SSO account or every lockout. Do not
assume resetting the username also clears a separately locked IP.

For a report, retain the time, sign-in method, redacted error, HTTP status and
request path (without query strings). Keep passwords, hashes, cookies, tokens,
full headers, **Copy as cURL** commands and network exports private. A 401 alone
does not justify deleting configuration, recreating the data volume or
re-enrolling agents.

#### Audit Log verification shows unsigned events

**Unsigned** means no signature was stored for that event; it is not the same
as **Failed** verification or a request **Error**. Signing can be unavailable
when Pulse cannot initialise its encryption manager. Check a bounded startup
log excerpt and the persistent data mount and access for the service account,
without printing key contents. Do not delete or regenerate `.encryption.key`
or an audit signing key to make the warning disappear. Restoring signing for
new events cannot authenticate an old unsigned event.

See [Audit verification and safe recovery](AUDIT_LOGGING.md#verification-failures-and-safe-recovery)
for the different results and evidence to retain.

#### Audit Log is empty

Clear the Event, user search, Outcome and Verification filters (or use
**Clear all filters** in the empty-state message), and check the selected
organisation first. A query error is not an empty history. **Pulse Pro runtime
required** means an active licence is running on the public community runtime;
follow the panel's **Download Pulse Pro** link rather than buying another
licence or resetting storage. Without the audit capability, reads and exports
are gated, but Pulse still attempts to capture events persistently on all
plans. **Console Logging Only** can also reflect unavailable persistent
storage: inspect the bounded startup logs for audit initialisation errors.
Do not change passwords or create tokens merely to populate the panel.

#### Audit Log verification fails for older events

A failed signature check does not by itself prove tampering. An event signed
with a different key can fail even when its contents are unchanged; missing
signatures and an unsupported or damaged signature format also cannot verify.
Keep the failure as evidence. Do not swap an old key into the live instance,
edit audit rows or re-sign old events. Preserve the current data and keys
privately before any recovery; compare a matching backup only in an isolated
restore, not by overwriting today's history. Follow
[safe audit recovery](AUDIT_LOGGING.md#verification-failures-and-safe-recovery).

### Monitoring Data

#### Primary IP shows a Podman or Docker bridge address

On a Proxmox VM or LXC, **Primary IP** is Pulse's selected address, not proof of
the guest's default route or reachability from your browser. An agentless LXC
uses the Proxmox collection path. When interface associations are supplied,
Pulse prefers other guest interfaces to recognised local-container/overlay
interfaces such as `podman0`; this is a display hint, not a physical-NIC test.
Useful secondary addresses are retained, including a virtual-only guest's
addresses. An unknown interface association remains unknown; address order
does not establish the intended route. The Proxmox API and Pulse agent use
different collection paths; an
agent's correct address does not establish that the API-only view is correct.

For access, use the intended guest address from your existing Proxmox network
configuration rather than assuming the displayed first address is suitable.
In **Workloads**, hover over the guest's network icon to inspect **Network
Interfaces**: when supplied, interface names appear beside their addresses.
The guest's details also show **Other IPs** when more than one address is
available. An address list without interface names does not establish which
interface it belongs to; keep that association unknown rather than guessing
from its prefix or position. These views do not test routing or connectivity.

Do not remove a working bridge, renumber the guest, reinstall or add an agent,
recreate the monitored connection, or run a guest-agent probe just to change
this display. For a report, retain the collection path and the interface/address
relationship already observed. Keep actual addresses, MACs and hostnames
private, using consistent aliases to preserve which address belongs to which
interface; a raw network or configuration dump is not needed.

#### Agent fleet update or identity issue

- If readings became mixed after adding another Proxmox installation, start
  with the [cross-installation identity checks](#monitoring-is-mixed-between-proxmox-installations),
  not an agent update or re-enrolment command.
- Open an outdated-agent notice or
  `/settings/infrastructure?agentDoctor=1` to open **Agent Doctor** and
  copy the platform-specific command for each reported host. This is a manual
  handoff; Pulse does not remotely execute the command.
- Administrators can call the read-only Agent Fleet Doctor endpoint,
  `GET /api/agents/diagnostics`, to inspect liveness, version drift, profile
  deployment drift, expected telemetry gaps, and identity-split evidence. It
  does not change agent configuration or enqueue a repair.
- A current Pulse server does not prove fleet convergence. Eligible v6 agents
  update asynchronously; v5, PVE, disabled, and failed updates require manual
  handling.

#### Removed Pulse server but `pulse-agent` still logs connection failures

Removing the Pulse server does not remove agent services installed on monitored
hosts. On a systemd host, stop and disable the orphaned service to halt retries:

```bash
sudo systemctl disable --now pulse-agent.service
```

If the Pulse server is still reachable, use its generated uninstall command so
the agent can deregister cleanly. If it is unreachable or the saved server URL
is missing, **stopping retries is not completed removal or credential
revocation**. Keep the agent's saved identity, credentials, connection files and
needed logs private and intact; do not reinstall, create a new token or delete
state to bypass a removal refusal. Temporary loss of connectivity does not
establish that the server is gone. Follow the
[saved-state removal guidance](UNIFIED_AGENT.md#uninstall) before local cleanup;
Unix credential-bearing removal needs confirmation from the same trusted Pulse
instance. The stop command above is for systemd, not Windows or other service
managers.

#### Filter Pulse's systemd journal by severity

Current systemd installs preserve Pulse's structured log level as the journal
priority. For warnings and more important records, use this bounded Bash reader
on the Pulse host, with an account authorised to read its journal. In Proxmox
LXC, run it inside the Pulse container, not on the Proxmox host. Substitute the
actual service name (`pulse-backend` on some older installs) and original
incident's time window; no new action or request ID is needed.

```bash
# systemd / Proxmox LXC — warning and higher priority
(
umask 077
for tool in timeout bash head mktemp wc rm cat; do
  command -v "$tool" >/dev/null 2>&1 || {
    printf 'Log read unavailable: required tool %s is missing; no unbounded fallback.\n' "$tool" >&2
    exit 1
  }
done
if ! pulse_log_file=$(mktemp "${TMPDIR:-/tmp}/pulse-log.XXXXXX" 2>/dev/null); then
  printf 'Log read unavailable: private temporary file could not be created.\n' >&2
  exit 1
fi
trap 'rm -- "$pulse_log_file"' EXIT
if timeout --signal=KILL 8s bash -o pipefail > "$pulse_log_file" 2>/dev/null <<'PULSE_LOG_READ'
journalctl -u pulse -p warning --since '15 minutes ago' --lines 200 --no-pager 2>&1 | head -c 262145
PULSE_LOG_READ
then
  if ! pulse_log_bytes=$(wc -c < "$pulse_log_file" 2>/dev/null); then
    printf 'Log read unavailable: excerpt size could not be checked.\n' >&2
    exit 1
  fi
  if [ "$pulse_log_bytes" -gt 262144 ]; then
    printf 'Log read unavailable: excerpt exceeds 256 KiB; no partial excerpt shown.\n' >&2
    exit 1
  fi
  cat -- "$pulse_log_file"
else
  pulse_log_status=$?
  printf 'Log read unavailable (exit %s); no partial excerpt shown.\n' "$pulse_log_status" >&2
  exit "$pulse_log_status"
fi
)
```

This requires **GNU `timeout`** and the listed local utilities, reads at most
200 records from the last 15 minutes, and limits the excerpt to **256 KiB**.
The eight-second hard deadline covers the reader and byte limiter together.
A failed, oversized or timed-out read is unavailable, not an empty warning
search; no partial excerpt is shown. Stop if a utility or journal access is unavailable; do not
use an unbounded substitute. These local excerpts are **not sanitised**; follow
[log-sharing precautions](#inspect-notification-logs) before sharing any line.

The stored message remains JSON (`"level":"warn"`, for example), while
`PRIORITY` is available to `journalctl` and syslog forwarding. An empty filtered
read does not prove that no warning occurred: an older generated unit may store
all records at info priority. Use the [bounded unfiltered systemd reader](#inspect-notification-logs)
to retain the original incident evidence instead. An administrator can check
locally whether the service configuration contains `SyslogLevelPrefix=true` and
`PULSE_LOG_JOURNAL_LEVEL_PREFIX=true`; do not post the full unit or environment,
which can contain credentials. Do not reinstall or restart Pulse just to obtain
logs. Repair an older generated unit through the current signed installer at a
planned maintenance window, preserving the deployment's settings and data.

#### VMs show "-" for disk usage
- Start with [current, retained and unavailable disk readings](VM_DISK_MONITORING.md#current-retained-and-unavailable-disk-readings).
  A dash is not zero; a **Prior** number is not current free space. Clear numeric
  disk conditions to inspect missing readings, not to change collection or access.
  A filtered list or disk sort is not proof of complete monitoring.
- Check the guest-local service, current VM Options and the configured API
  token's read permissions. Schedule any setup change or restart outside backups.
- Do not run guest-agent probes during backup freeze/thaw. An OK backup task
  or an absent lock does not confirm thaw.
- See [VM Disk Monitoring](VM_DISK_MONITORING.md) for the passive host preflight;
  it does not verify a fresh disk poll. For an affected installation, follow
  the [manual backup precaution](VM_DISK_MONITORING.md#backup-safety) for your
  actual server deployment: **systemd or Docker/Compose**. A planned pause is
  **not incident recovery**, and stopping Pulse does not cancel a guest-agent
  request already issued. If an existing operation's state is unknown, do not
  start a backup on the strength of a stopped server or an elapsed wait.
- Keep Pulse stopped until the backup has ended and independent post-backup
  checks confirm **thaw, fresh successful workload writes to every filesystem
  covered by the backup, and workload liveness**. Use established safe checks,
  independent of Pulse and the QEMU Guest Agent; a console connection or a
  successful read alone is not enough. If any check fails or is unavailable,
  leave Pulse and its automatic updater paused and use the guest/platform's
  recovery procedure, not new probes, forced writes or another backup.
- After all checks pass, restore **only services and timers that were active
  before the pause**, following the deployment-specific precaution. Unknown
  pre-pause states are not permission to start them. **Pulse monitoring and
  alerts are unavailable while stopped**; arrange independent outage coverage.

#### Backup-age alerts were not evaluated

The **Pulse** warning “Backup-age alerts were not evaluated because recovery
data could not be read” means that Pulse could not complete that backup-age
check. It is a monitoring failure, **not evidence that a backup failed**.
Existing backup-age alerts are kept during the failed read; they are not a
fresh assessment, and the absence of a new age alert does not establish that
backups are current. A working connection or a visible backup row does not
prove that age evaluation completed.

Use the owning Proxmox/PBS installation's existing backup records and your
established backup checks while evaluation is unavailable. Compare the same
installation, guest type/ID, datastore, namespace and backup time; a matching
VMID in another installation is not the same guest. For a separate Coverage
disagreement, use the [backup health checks](PBS.md#backups-are-visible-but-coverage-says-unprotected).

Keep the warning's original time and message. On the actual Pulse server,
use the existing [bounded log reader](#inspect-notification-logs) for the
incident window, with the matching systemd or Docker deployment. The log
message **Failed to list recovery rollups for backup alerts** can identify a
failed evaluation read; a timeout does not by itself establish its cause.
A failed log read is unavailable, not an empty successful search. Share only
the running Pulse version, affected time window, warning and relevant manually
redacted error. Keep credentials, private paths/hostnames, full logs, API
responses and database files out of public reports.

**A disappearing warning alone is not recovery.** Disabling alerts or backup-age
checks, or setting both age thresholds to zero, can clear it without a
successful evaluation. Keep the intended policy unchanged and observe ordinary
polling; do not disable protection just to clear the warning. A successful age
evaluation still does not establish backup completeness, verification, restore
readiness or guest thaw.

Do not restart or reinstall Pulse, clear History, delete/prune backup records,
change database schema or increase timeouts just to diagnose this warning. Do
not run a new backup, verification, restore, live diagnostics or guest-agent
probe as a test. An OK backup task does not prove guest thaw; a frozen or
unresponsive guest needs the separate [backup safety procedure](VM_DISK_MONITORING.md#backup-safety),
including independent post-backup thaw, successful writes to every covered
filesystem and workload liveness before restoring only previously active
monitoring.

#### Backup health disagrees with PBS

A visible or Verified PBS backup is not the same reading as a workload's
Coverage posture. Use the [backup health checks](PBS.md#backups-are-visible-but-coverage-says-unprotected)
to compare one affected row's explanation, Job, History and Access with the
matching native PBS record. Do not run a new backup, restart or clear history
just to diagnose the disagreement.

#### Replication jobs are Pending, stale or missing

**Proxmox → Replication** monitors storage replication between PVE nodes,
not PBS backups. A replication result does not establish backup coverage,
restore readiness or guest thaw. Use the owning Proxmox installation's
existing replication status while Pulse's reading is uncertain.

Start with **All** and clear the table's search. **No replication jobs match
current filters** is different from an empty inventory, a missing tab or
**Could not load replication jobs**. Record the actual message; a missing
row is not proof that Proxmox deleted the job.

| Observation in Pulse | What to compare | What it does not establish |
| --- | --- | --- |
| **Pending**, with current sync times | The same job's outcome, failures and error in Proxmox. Pulse can retain sync times without a usable outcome for its Healthy/Pending classification. | Pending alone does not prove a job is waiting or failed. A recent time, zero displayed failures or no displayed error alone does not prove success. |
| Old sync times or an overdue next sync | The same job's existing native last/next sync and outcome. Record whether actual sync times advance during ordinary polling. | A changing “ago” or “overdue” label is the browser clock advancing, not evidence of a new poll. A Healthy pill does not establish current inventory or a running scheduler. |
| Empty inventory, missing tab or load error | Whether the expected jobs still exist in Proxmox, which Pulse connection owns them, and any displayed read error. | Missing or failed collection is unknown, not an empty healthy result. Correcting a status pill does not establish lifecycle or polling recovery. |

For one affected job, expand its row using the control beside **Guest**;
the details retain **Job**, **Route**, **Last sync**, **Next sync**, **Duration**
and **Failures** even when narrow layouts hide table columns. Compare the
owning installation, job/guest ID and source → target route with the existing
native Proxmox replication view. Similar names or VMIDs in separate
installations do not identify the same job. An unavailable native observation
can be reported as unavailable; do not force a sync to obtain one.

Treat **classification** and **stale or absent inventory after a lifecycle
change** as separate symptoms. If a previous restart restored data, keep that
observation and the now-working setup; do not repeat a restart, reboot or
update to reproduce it. One current row does not prove every job recovered.

For help, give the running Pulse/PVE versions, container image tag if relevant,
the original sequence and time, which symptoms remain, and one matching job's
Pulse versus native status, sync times, failures and redacted error. Include
any already-observed recovery, without recreating missing original evidence.
Use consistent placeholders for private installations, nodes and guest/job
identities so the route remains comparable. Do not share tokens, full API
responses, configuration, HAR exports or unredacted screenshots.

Do not restart or upgrade Pulse, reboot nodes, change schedules, disable/re-enable
jobs, recreate connections, widen token permissions or disable privilege
separation just to diagnose the table. Do not run a new sync, backup, restore,
live diagnostics or guest-agent probe as a test. A frozen or unresponsive
guest needs the separate [backup safety procedure](VM_DISK_MONITORING.md#backup-safety),
not these display checks.

#### Monitoring is mixed between Proxmox installations

Pulse is intended to monitor [multiple Proxmox installations](CONFIGURATION.md#multiple-proxmox-installations).
Separate installations can reuse native node names and VMIDs. A distinct
Pulse connection label or display name is not a repair for wrong attribution.

If adding a connection led to node errors on the wrong node, unavailable
backup status for overlapping VMIDs, or lost agent-backed Docker monitoring,
treat each as an unresolved symptom. One restored view does not establish
that the others recovered. Use **existing observations only**; if you already
repaired the setup, keep that working setup and the original sequence rather
than undoing the repair or repeating the addition.

| Affected reading | Safe comparison | What it does not prove |
| --- | --- | --- |
| Node errors | Compare the affected node and observation time with that installation's own Proxmox view. | A similar node name or a different Pulse label does not establish which installation owns the error. |
| Guest backup status | Compare one affected guest with the matching native Proxmox/PBS record: owning installation, guest type/VMID, datastore, namespace and backup time. Follow the [PBS evidence checks](PBS.md#backups-are-visible-but-coverage-says-unprotected). | The same VMID or a visible backup from another installation does not establish protection for this guest. |
| Agent-backed Docker monitoring | In **Settings → Infrastructure → Agent Doctor**, inspect the affected host's existing **Last seen** and **Identity evidence**, including **Connection** and **Hostname** where available. Compare with the known Docker host, not a similar guest name. | Recent agent contact or a Healthy badge does not prove that the correct host's containers are visible or correctly attributed. Missing evidence is unknown, not proof that the agent stopped. |

While attribution is uncertain, use each installation's own Proxmox/PBS
views for node errors and backup status, and the Docker host's existing runtime
view for container status. Do not make backup, restore or workload changes
from a potentially misattributed Pulse reading.

For a report, give the running server/agent versions, the original sequence,
which of the three symptoms remain, and one affected pair's expected versus
displayed origin and observation times where available. Use **consistent
placeholders** such as `site-A`, `site-B`, `node-X` and `guest-100` across every
view; say explicitly when the native names or VMIDs are equal. Keep actual
addresses, hostnames, connection IDs and machine identities private. Do not
share a full Agent Doctor report, configuration, API response, HAR export,
token or unredacted screenshot. Unavailable original evidence can be reported
as unavailable; do not recreate the incident to obtain it.

Do not rename production nodes, change VMIDs or machine IDs, delete/re-add
connections, re-enrol agents, rotate tokens, restart services or clear History
just to diagnose this disagreement. Do not run live diagnostics, guest-agent
probes, a new backup or a restore as an identity test. An OK backup does not
prove thaw; a frozen or unresponsive guest needs the separate
[backup safety procedure](VM_DISK_MONITORING.md#backup-safety), not these display checks.

#### Temperature data missing
- Compare the affected host's active agent version, last report, sensor and
  observation time. A current server or another sensor's value is not evidence
  that this reading is available.
- Use the [bounded local reading check](TEMPERATURE_MONITORING.md#check-existing-linux-readings-safely)
  on the monitored host, not the Pulse container. Linux agents can use existing
  recognised CPU/SoC sysfs readings without `lm-sensors`; unavailable is not zero.
- Do not run automatic hardware detection, load drivers, reboot or loosen SSH
  restrictions merely to fill a temperature row. Retain output privately and
  follow the [platform-specific guide](TEMPERATURE_MONITORING.md) for any needed
  setup change during a maintenance window.

#### Container CPU differs from Docker or Podman stats

Pulse uses the runtime host's total CPU capacity; `docker stats` and
`podman stats` normally use 100% per logical CPU. Divide a per-CPU stats reading
by the **reported runtime host CPU count**, not the physical Proxmox host's
count when the engine runs inside a VM. Compare the same container and time
interval in **CPU History**, not just a rounded table label. A dash or missing
History is unavailable, not zero; an Active connection alone is not fresh CPU
evidence.

See [container CPU readings](DOCKER.md#container-cpu-readings) for worked
examples, rounding and alert limits, and a passive comparison. Do not restart
workloads, create CPU load or lower alert thresholds to test this.

#### Docker hosts appearing/disappearing

Cloned hosts can share a **saved Pulse agent ID**, not just an OS machine ID.
The agent uses an explicit ID first, then its saved `agent-id` file, and derives
one from the machine only when neither is available. Changing a hostname, IP or
`/etc/machine-id` therefore does not necessarily change its Pulse identity.

Compare the affected hosts in **Agent Doctor** and inspect only their configured
`agent-id` files locally. Do not delete the OS machine ID, agent state or Pulse
history as a troubleshooting step. If a duplicate is confirmed, give only the
clone a stable, unique ID in its managed service or container configuration;
leave the original host unchanged. Follow
[Clone identity recovery](UNIFIED_AGENT.md#duplicate-agents) for the configuration
precedence, systemd example and checks after restart.

#### Guest memory is high but available memory is plentiful

A VM's hypervisor memory footprint is not the same as usable memory inside the
guest. Some Proxmox API readings include reclaimable cache or allocated memory;
a near-100% value alone does not establish guest pressure or a leak. Conversely,
a cache-heavy reading does not prove the guest is healthy. Keep an unresponsive
workload or genuine allocation failure separate from a misleading percentage.

Compare **existing observations of the same guest, kernel and time**, with the
same units and memory allocation. The Proxmox node's available memory is not the
VM's available memory, and a single application's RSS is not total guest usage.
Do not sum process RSS to infer guest pressure: processes can share pages.

For History comparisons, keep each panel's stored time and qualification with
the value; a range endpoint, page refresh or today's live source does not date
a past reading. Use [History panel interpretation](METRICS_HISTORY.md#read-a-history-panel)
to distinguish a missing observation from zero, and stored data from fallback.

| Reading | What it describes | What it does not establish |
| --- | --- | --- |
| Linux `available` / `MemAvailable` | An estimate of memory usable for new allocations without swapping, including some reclaimable cache. | It is not just `free`, a guarantee of every allocation succeeding, or a reading for another guest. |
| Linux `free` and `buff/cache` | Unused pages and memory used for buffers/cache. Cache can be useful and partly reclaimable. | Low `free` alone is not pressure; not all cache is immediately reclaimable. |
| Proxmox status memory | The platform's selected footprint or estimate. The collection path matters. | A cache-inclusive or allocation-based percentage is not equivalent to cache-aware guest usage. |
| Application RSS | Resident pages for that process at that time. | It is not total guest usage, Pulse's heap size, or proof that no other process is under pressure. |
| Missing, retained or undated reading | Incomplete evidence, or an earlier observation. | Missing is not zero; a prior low value or a fresh page refresh is not current recovery. |

For a **same-guest, same-time** Linux example with **20 GiB total** and
**15.2 GiB available**, cache-aware usage is `(20 - 15.2) / 20 × 100 = 24%`.
Use the observed total, not a nominal configured limit, and do not mix GB with
GiB or kernel `kB` (1,024 bytes). This illustrates the distinction; it is not a
correction to apply blindly to another metric or evidence that an existing
Patrol finding has cleared. Do not subtract all `buff/cache` from total, or
substitute `free` for `available`.

Keep the running Pulse/agent and platform versions, collection path (platform
API, linked Pulse agent, or both), source and original observation time where
shown. A working agent reading does not prove that the API-only path recovered.
Unknown source, time or counters can be reported as unknown; existing screenshots
and the original sequence are enough to begin a report. Do not publish raw API
responses, full `/proc` output, configuration or private host names; review and
share only the relevant counters with consistent aliases.

[Memory metric Off](CONFIGURATION.md#metric-thresholds-off-and-inheritance)
disables that threshold rule only: it does not hide the reading, dismiss a
separate Patrol finding or establish recovery. Do not disable all monitoring or
change thresholds merely to make a comparison pass.

Use the guest's ordinary status if it is already available; do not run
Diagnostics, guest-agent probes or another Patrol run, install an agent, restart
the guest, drop caches or change memory limits just to obtain evidence or lower
the number. If the guest is frozen or unresponsive, stop these comparisons and
follow [backup safety](VM_DISK_MONITORING.md#backup-safety), not a memory probe.

#### Memory use keeps growing

For a monitored VM, start with [guest memory interpretation](#guest-memory-is-high-but-available-memory-is-plentiful);
the recipe below measures the **Pulse server process**, not a monitored guest.

A Proxmox LXC memory chart covers the container, not just Pulse. It can include
other processes and filesystem cache; Docker's displayed memory also uses its
own cache accounting. Neither is interchangeable with Pulse's resident memory
(RSS), a Go heap size, or virtual address space (`VSZ` / `VmSize`). A large
container reading alone does not establish a Pulse memory leak.

**A higher steady baseline is also worth reporting; continuing growth is not
required.** First distinguish what your existing readings show:

| Observation | What it establishes |
| --- | --- |
| Comparable Pulse process RSS settles at a higher level than before with an unchanged workload | A possible baseline regression, even if usage then stays flat. Keep the old and new versions, uptime and measurement scope; a plateau does not dismiss the increased requirement. |
| Comparable samples rise during one uninterrupted process run | Growth during that interval, not by itself a leak or its cause. Keep the same PID/start time, units, workload and collection settings. |
| A new process uses less memory shortly after an upgrade than an older process did after days | Different runs at different ages, not verified recovery or growth within one run. Startup and long-lived usage can differ. |

For example, **510 MiB after a week** and **420 MiB ten minutes into a new
process** cannot establish that steady-state memory improved. Use already
available readings at comparable uptimes and workloads if you have them;
otherwise keep the mismatch explicit. Do not restart, downgrade or change
retention, polling or enabled features to manufacture a comparison. A startup
CPU improvement is not evidence that the resident-memory baseline fell.

For a responsive **Linux systemd / Proxmox LXC** install, take this small,
read-only sample at a normally occurring high point. Run it **inside the Pulse
container** for LXC, not on the Proxmox host. Substitute the actual service name
(`pulse-backend` on older installs) in both `systemctl` calls. Use an account
already authorised to read these counters. The collector does not elevate
privileges or prompt; do not add `sudo` inside it. A failed read is unavailable,
not zero.

```bash
# systemd / Proxmox LXC: Pulse resident-memory sample
(
  set -eu
  command -v timeout >/dev/null 2>&1 || {
    printf 'Resident-memory sample unavailable: GNU timeout is required.\n' >&2; exit 1
  }
  if sample=$(timeout --signal=KILL 8s bash <<'PULSE_MEMORY_SAMPLE' 2>/dev/null
set -eu
date -u +'%Y-%m-%dT%H:%M:%SZ'
pid=$(systemctl show pulse --property=MainPID --value)
case "$pid" in
  ''|0|*[!0-9]*) printf 'No running Pulse PID; sample unavailable.\n' >&2; exit 1 ;;
esac
identity=$(TZ=UTC ps -p "$pid" -o pid=,lstart=)
if [ -z "$identity" ]; then
  printf 'Process identity unavailable.\n' >&2; exit 1
fi
counters=$(awk '
  $1 == "VmRSS:" || $1 == "RssAnon:" || $1 == "RssFile:" ||
  $1 == "RssShmem:" || $1 == "VmSwap:" {
    if (NF != 3 || $2 !~ /^[0-9]+$/ || $3 != "kB" || seen[$1]++) exit 1
    print; fields++
  }
  END { if (fields != 5) exit 1 }
' "/proc/$pid/status")
current_pid=$(systemctl show pulse --property=MainPID --value)
current_identity=$(TZ=UTC ps -p "$pid" -o pid=,lstart=)
if [ "$pid" != "$current_pid" ] || [ "$identity" != "$current_identity" ]; then
  printf 'Pulse changed during collection; discard this sample.\n' >&2; exit 1
fi
printf 'Pulse process (PID and UTC start): %s\n%s\n' "$identity" "$counters"
PULSE_MEMORY_SAMPLE
  ); then
    printf '%s\n' "$sample"
  else
    status=$?
    printf 'Resident-memory sample unavailable (exit %s); stop sampling.\n' "$status" >&2
    exit "$status"
  fi
)
```

The recipe requires GNU `timeout`: it gives the read-only sampling commands
eight seconds, then kills its own collection group, including readers that
ignore TERM. Waiting for the collection shell alone can leave a reader running
after it exits. This hard deadline does not stop or restart Pulse. A missing
utility, timeout or reader error means the sample is unavailable; keep the
failure, and do not remove the deadline or rerun it against an unresponsive
installation. No timestamp, process identity, counter or raw reader error is
printed unless every check succeeds; the failure reports only its exit status.

Linux labels these values `kB`, meaning 1,024 bytes; divide by 1,024 for MiB.
`VmRSS` is resident process memory, split into anonymous (`RssAnon`),
file-backed (`RssFile`) and shared-memory (`RssShmem`) pages. `VmSwap` is swapped
private anonymous memory, not additional resident memory or all container swap.
These counters are approximate and not an atomic snapshot; a nonzero
`RssFile` is not proof of a leak, and `RssAnon` is not specifically the Go heap.
Missing fields on an older kernel make this recipe fail rather than invent
values. If it fails or Pulse stops, retain that fact; do not loosen access
controls or keep retrying against an unresponsive installation.

Keep the sample time, PID/start time, running version, container memory limit,
fleet size, polling interval, enabled collectors/features and whether dashboards
were open. If safe, compare
another naturally occurring point with the **same PID and process start time**;
record any restart or upgrade as a different run, not evidence that the cause
was fixed. For Docker, the [bounded container statistics below](#excessive-cpu-writes-or-database-growth)
retain the container start time and memory usage but do not measure Pulse RSS;
do not assume its PID 1 is Pulse.

Do not restart Pulse, drop caches, force garbage collection, change memory limits
or delete history just to obtain a lower reading. If the container is near its
limit or the host is unresponsive, stop sampling and prioritise safe recovery.
Do not post a heap dump, profile, full `/proc` status or process command line:
they can contain private details. Share only these counters and the relevant
context after local review; existing screenshots remain useful if collection
is unsafe.

#### Excessive CPU, writes or database growth

First distinguish **Pulse server activity**, agent activity and total host or
storage-device activity. A database's size is retained space, not the number of
bytes written: repeated updates can produce high writes without growing the file.
Keep the original running version, uptime, measurement time and workload before
changing anything. A different binary can migrate persistent data; switching
back to an older version does not make that data a clean older-version baseline.

- For CPU, record the measuring tool, allocated CPUs or container quota and
  whether its percentage represents one core or the whole allocation. A `ps`
  `%CPU` value is an average since process start, not a recent sampling window.
- Note fleet size, polling interval and whether Pulse dashboards are open. If
  safe, close all Pulse tabs and compare another equal-length window, without
  restarting the server or changing polling. A decrease narrows the workload;
  it does not establish the cause or prove monitoring is healthy.
- If the host is unresponsive or storage is nearly full, do not prolong the
  failure to collect a benchmark. Keep existing observations and report the
  interruption instead. Do not enable Debug or repeatedly export diagnostics
  merely to measure performance; those actions add work and can change the result.

For a responsive **Linux systemd / Proxmox LXC** install, the following reads
two process-I/O samples, waiting 60 seconds between them. Run it inside the
Pulse container for LXC, not on the Proxmox host. Substitute the actual service
name (`pulse-backend` on some older installs) in both `systemctl` calls. Use an
account already permitted to read the process counters. The collector does not
elevate privileges or prompt: an unprivileged deadline cannot reliably stop a
privileged reader. If access is denied, stop rather than changing privileges
to make it pass; no service restart or database access is needed.

```bash
# systemd / Proxmox LXC: bounded process-write samples
(
  set -eu
  command -v timeout >/dev/null 2>&1 || {
    printf 'Write samples unavailable: GNU timeout is required.\n' >&2; exit 1
  }
  if samples=$(timeout --signal=KILL 80s bash <<'PULSE_WRITE_SAMPLES' 2>/dev/null
set -eu
read_sample() {
  set -eu
  pid=$(systemctl show pulse --property=MainPID --value)
  case "$pid" in
    ''|0|*[!0-9]*) exit 1 ;;
  esac
  identity=$(TZ=UTC ps -p "$pid" -o pid=,lstart=)
  [ -n "$identity" ] || exit 1
  start_ticks=$(awk '
    { sub(/^.*\) /, ""); if ($20 !~ /^[0-9]+$/) exit 1; print $20; found++ }
    END { if (found != 1) exit 1 }
  ' "/proc/$pid/stat")
  timestamp=$(date -u +'%Y-%m-%dT%H:%M:%SZ')
  counters=$(awk '
    $1 == "write_bytes:" || $1 == "cancelled_write_bytes:" {
      if (NF != 2 || $2 !~ /^[0-9]+$/ || seen[$1]++) exit 1
      print; fields++
    }
    END { if (fields != 2) exit 1 }
  ' "/proc/$pid/io")
  current_pid=$(systemctl show pulse --property=MainPID --value)
  current_identity=$(TZ=UTC ps -p "$pid" -o pid=,lstart=)
  current_ticks=$(awk '
    { sub(/^.*\) /, ""); if ($20 !~ /^[0-9]+$/) exit 1; print $20; found++ }
    END { if (found != 1) exit 1 }
  ' "/proc/$pid/stat")
  [ "$pid" = "$current_pid" ] && [ "$identity" = "$current_identity" ] &&
    [ "$start_ticks" = "$current_ticks" ] || exit 1
  printf 'Identity: %s; start ticks=%s\nSample UTC: %s\n%s\n' \
    "$identity" "$start_ticks" "$timestamp" "$counters"
}
first=$(read_sample)
sleep 60
second=$(read_sample)
[ "${first%%$'\n'*}" = "${second%%$'\n'*}" ] || exit 1
printf '%s\n\n%s\n' "$first" "$second"
PULSE_WRITE_SAMPLES
  ); then
    printf '%s\n' "$samples"
  else
    status=$?
    printf 'Write samples unavailable (exit %s); no complete pair. Stop sampling.\n' "$status" >&2
    exit "$status"
  fi
)
```

Each recipe requires **GNU `timeout`** and limits the complete two-sample
collection, including the 60-second wait, to 80 seconds. At the deadline it
kills its own read-only collection group, including readers that ignore TERM,
not Pulse or Docker. This deliberate hard stop prevents a reader surviving after
the collection shell exits. A missing utility, permission failure, timeout or changed identity
makes the pair unavailable; no partial identity, counter or raw reader error is
printed, and there is no unbounded fallback. Stop sampling on failure rather
than removing the deadline or retrying against an unresponsive installation.

The process recipe checks the same PID and process start time before and after
each read and across both samples. Linux start ticks also distinguish PID reuse
within one displayed second. These checks do not make the counters atomic.
Compare `write_bytes` only when no restart occurred and the counter did not decrease. Divide the
byte difference by the **actual elapsed seconds**. This is storage-accounted
process I/O, not filesystem growth or physical SSD wear; cancelled writes and
background writeback can differ from device measurements. Keep
`cancelled_write_bytes` alongside it, not as proof of bytes reaching the drive.
An extrapolated GB/day rate is a projection of that short window, not a measured
day's total. Preserve the window and units with the result.

For **Docker / Compose**, run this on the Docker host, replacing `pulse` with
the running container name. It reads only the start time and selected statistics,
not the container environment or configuration. Each stats call targets the
full container ID, not a reusable name. The name must still resolve to the same
running ID, start time and restart count after the read and across both samples;
a replacement, restart or stopped container makes the pair unavailable.

```bash
# Docker: bounded container statistics
(
  set -eu
  command -v timeout >/dev/null 2>&1 || {
    printf 'Container samples unavailable: GNU timeout is required.\n' >&2; exit 1
  }
  if samples=$(timeout --signal=KILL 80s bash <<'PULSE_CONTAINER_SAMPLES' 2>/dev/null
set -eu
read_sample() {
  set -eu
  identity=$(docker inspect --type container --format \
    '{{.Id}} {{.State.StartedAt}} {{.State.Running}} {{.RestartCount}}' pulse)
  case "$identity" in *$'\n'*) exit 1 ;; esac
  read -r id started running restarts extra <<< "$identity"
  [ "${#id}" -eq 64 ] && [ "$running" = true ] && [ -z "$extra" ] || exit 1
  case "$id" in *[!0-9a-f]*) exit 1 ;; esac
  case "$started" in 0001-*) exit 1 ;; ????-??-??T??:??:??*Z) ;; *) exit 1 ;; esac
  case "$restarts" in ''|*[!0-9]*) exit 1 ;; esac
  timestamp=$(date -u +'%Y-%m-%dT%H:%M:%SZ')
  stats=$(docker stats --no-stream --format \
    'CPU={{.CPUPerc}} Memory={{.MemUsage}} BlockIO={{.BlockIO}}' "$id")
  case "$stats" in
    *$'\n'*) exit 1 ;;
    CPU=*%\ Memory=*\ /\ *\ BlockIO=*\ /\ *) ;;
    *) exit 1 ;;
  esac
  cpu=${stats%% *}
  [[ "${cpu#CPU=}" =~ ^[0-9]+([.][0-9]+)?%$ ]] || exit 1
  case "$stats" in *--*) exit 1 ;; esac
  current_identity=$(docker inspect --type container --format \
    '{{.Id}} {{.State.StartedAt}} {{.State.Running}} {{.RestartCount}}' pulse)
  [ "$identity" = "$current_identity" ] || exit 1
  printf 'Identity: %s\nSample UTC: %s\n%s\n' "$identity" "$timestamp" "$stats"
}
first=$(read_sample)
sleep 60
second=$(read_sample)
[ "${first%%$'\n'*}" = "${second%%$'\n'*}" ] || exit 1
printf '%s\n\n%s\n' "$first" "$second"
PULSE_CONTAINER_SAMPLES
  ); then
    printf '%s\n' "$samples"
  else
    status=$?
    printf 'Container samples unavailable (exit %s); no complete pair. Stop sampling.\n' "$status" >&2
    exit "$status"
  fi
)
```

Docker **BlockIO** is cumulative read / write activity, not bytes per second;
compare its write side only across the same uninterrupted container run. Its
displayed units are rounded. Container, process and whole-device counters have
different scopes and must not be added together or compared as interchangeable
measurements. A failed, denied or empty read is unavailable, not zero activity.

Keep persistent data on durable storage. Do not delete or truncate history,
incident, queue or audit files, remove database indexes, or move the data
directory to tmpfs as a diagnostic workaround. These actions can lose evidence
or protections without fixing the writer. Do not post databases, profiles, full
`/proc` dumps, container configuration or raw environments. Share only the bounded
measurements, workload, versions and a relevant manually redacted error; follow
[Getting Help](#-getting-help) for any additional evidence.

### Notifications

#### No alert when Pulse, power or internet goes down

Pulse cannot send a notification while its host is stopped or its outbound
network is unavailable. A local delivery-health warning is not an external
outage detector, and a successful **Send test** does not prove outage coverage.

- In **Alerts → Notifications → External watchdog**, configure a
  Healthchecks-compatible **success ping URL** and save the configuration.
  Keep the URL secret; do not include it in screenshots or support reports.
- Run the watchdog outside the failure you want to detect. Another machine on
  the same power supply or internet connection does not cover a whole-site
  outage. Its notification destination must also remain reachable independently
  of that site.
- At the watchdog, configure a one-minute period and a three-minute grace
  period, and enable its notification integration. Pulse sends a heartbeat
  every minute and a `/fail` signal if its monitoring loop stalls. For
  Healthchecks simple schedules, a missing heartbeat becomes down after
  **period plus grace**: approximately four minutes after the last success,
  not three. Recipient delivery can take longer. See
  [Healthchecks timing and notification concepts](https://healthchecks.io/docs/).
- Verify in an authorised test environment: confirm incoming heartbeats at
  the watchdog, stop the test Pulse instance, and check that the intended
  recipient actually receives the external alert. Restore Pulse and verify
  heartbeat recovery. Test loss of internet separately if you need that
  coverage; a stopped-process test does not prove it.
- To retire the check, pause or remove it at the watchdog too. Clearing the
  URL in Pulse stops its signals but does not pause the remote check.

A healthy heartbeat indicates Pulse monitoring-loop progress, not successful
delivery of every resource alert or external reachability of your services.
Continue checking delivery activity for destination failures.

#### Test succeeds but real alerts are missing

A test sends directly to its destination: it skips the persistent delivery queue
and is not listed in **Recent delivery activity**. It does not prove that a real
alert was generated, routed or delivered to the intended recipient.

- Open **Alerts → Notifications**. If **Notifications are paused** is shown,
  configured destinations and a successful test do not enable real delivery.
  Turn delivery on there only when you intend to send alerts.
- Check the affected alert, the destination's **Enabled** state and
  [minimum severity and tag filters](CONFIGURATION.md#destination-severity-and-tag-routing).
  Both must match: an empty tag filter does not bypass minimum severity, and
  critical severity does not bypass a nonempty tag filter. Review the [quiet-hours schedule](CONFIGURATION.md#quiet-hours-and-notification-holds)
  and [acknowledgement or snooze](CONFIGURATION.md#acknowledge-and-snooze-existing-alerts),
  plus any mute or maintenance policy, before treating an absent attempt as a
  transport failure. In **Alerts → Overview**, use **Show acknowledged** to find
  hidden incidents; **Resume** ends a snooze but does not remove acknowledgement.
- For a missing repeat or all-clear, review [alert reminders and recovery notifications](CONFIGURATION.md#alert-reminders-and-recovery-notifications).
  Cooldown off stops ordinary reminders for the same occurrence; it does not
  mean unlimited repeats. Recovery messages require eligible prior firing
  delivery to that destination. Their absence does not establish whether the
  workload recovered: check fresh readings and the incident state separately.
- Use **Recent delivery activity** to correlate the original alert, destination
  and absolute timestamp, including held-notification reasons. An empty window
  is not proof of healthy delivery; an **unavailable** read is not an empty log.
  Do not create an outage or repeat a notification storm to populate it.

#### Recover retained delivery failures

Pulse shows a delivery warning for failed or dead-lettered notifications in its
persistent queue, not for every recoverable retry. Authentication,
configuration and rejected failures stop automatic retries as soon as they are
classified, even with attempts left. A terminal failure can therefore appear
without exhausting the retry budget; waiting alone will not resend it. See
[webhook retry behaviour](WEBHOOKS.md#-delivery-contract) for HTTP exceptions,
transient failures and the normal queued attempt limit.

**Recent delivery activity** masks recognised credentials in URLs, **not all
private information**. Older records and other error sources can still include
email addresses, private destinations or credentials echoed outside a URL;
alert identifiers and resource names can also identify your infrastructure.
Withheld webhook response text is intentional, not a reason to collect it again;
see [withheld-response guidance](#webhook-response-body-is-withheld).
Keep full entries, copied responses and screenshots private.
A `REDACTED` marker does not make the rest
safe to share. For a public report, extract only the relevant timestamp,
delivery method, failure class, HTTP status or SMTP error code and a manually
redacted error. See [notification log precautions](#inspect-notification-logs).

Completed attempts remain for 7 days and dead-letter attempts for 30 days.
Start with the failure class and timestamp:

| Failure | Check before retrying |
| --- | --- |
| Authentication | Destination credentials and account permissions, locally; never post them. |
| Rate limited | Provider limits and delivery volume; repeated tests or retries can make this worse. |
| Connectivity | DNS, firewall, proxy and reachability from the Pulse server, not just your browser. |
| TLS | Certificate trust, expiry and hostname matching; do not disable verification to diagnose it. |
| Configuration / rejected | Enabled destination, required fields and the provider's endpoint or payload requirements. |
| Server error / unknown | Destination service status and a relevant, bounded local error excerpt. |

Save the corrected destination settings and send one test; check receipt at the
intended destination. **Retained deliveries keep the destination settings saved
when they were queued.** Editing a URL, recipient, credential, header or template
does not replace that saved configuration. A test uses the edited settings;
retrying an old delivery can still use the old endpoint or credential and fail
again. If the old destination must no longer receive data, disable it rather
than relying on a URL edit to redirect queued work. This is not a recall of
requests already sent; see the [pause and cancellation limits](#pause-and-cancellation-limits).

**Retry retained deliveries** gives all retained terminal failures a fresh retry
budget, not just the destination you tested. Use it only when sending those
original deliveries is still intended and their original settings remain
appropriate, for example after a temporary provider outage. A destination that
accepted an earlier attempt may receive a duplicate. Review the confirmation's
delivery count and provider limits before retrying. A successful test does not
itself retry retained items or prove that their saved settings now work. If
uncertain, leave the failures retained and check the next normally occurring
alert instead; do not use a batch retry to test a settings edit.

Use **Dismiss retained failures** only when those deliveries should not be sent.
Dismissal clears the warning without retrying them; delivery history remains.
Neither action deletes the audit trail. Do not delete `notification_queue.db`
or audit data to clear the warning.

#### Pause and cancellation limits

In **Alerts → Notifications**, global **Notifications paused** is not a
store-and-forward hold. Monitoring continues, but pausing clears buffered alert
groups and cancels pending queue deliveries for email, webhooks and Apprise,
including recovery deliveries. **Turning delivery back on does not replay those
cancelled deliveries.** Retained terminal failures are separate: pausing does
not dismiss them or repair their saved destination settings.

Disabling or removing an individual destination prevents a queued delivery that
sees that policy from sending. A policy-skipped item is cancelled, not a
successful provider delivery, and **Retry retained deliveries** does not replay
cancelled items. Re-enabling a destination is not proof of catch-up or receipt.
For a scheduled quiet period, use [Quiet hours](CONFIGURATION.md#quiet-hours-and-notification-holds)
instead, reviewing its critical-alert exceptions and later re-evaluation; it
still does not guarantee that every held message will be sent.

**Pause, disable and removal do not guarantee cancellation of a request already
in flight.** The receiver may accept it after the settings change. Neither a
paused banner nor a cancelled queue row proves that the destination received no
data, and Pulse cannot recall a message or delete the receiver's copy. Reconcile
the original time with **Recent delivery activity** and the receiver's existing
record, keeping both private; do not send another Test or replay a queue to
check containment. If a destination credential was exposed, revoke it with
the provider too: editing its URL or turning delivery off is not credential
revocation and cannot undo an earlier disclosure.

#### Emails not sending

Follow [retained-failure recovery](#recover-retained-delivery-failures) first.
Check SMTP host, port, sender, recipients, authentication and TLS settings in
**Alerts → Notifications** against your provider's requirements (some providers
require an app password). Keep passwords in the settings form, not a diagnostic
command or report. If the delivery error is insufficient, inspect
[bounded notification logs](#inspect-notification-logs) locally.

#### Webhooks failing

Follow [retained-failure recovery](#recover-retained-delivery-failures) first.
Use the failure class and HTTP status to check the provider's endpoint and
payload requirements. Verify reachability from the Pulse server. For an intended
private destination, review **Settings → System → Network → Webhook Security**;
do not broadly weaken network or TLS controls just to make a test pass.

Prefer the redacted delivery error over raw provider response bodies. A provider
can echo credentials or private content in its response; do not post it wholesale
or enable debug logging just to collect it. If needed, inspect
[bounded notification logs](#inspect-notification-logs) and share only the
consequential, manually redacted error.

#### Webhook response body is withheld

`response body withheld`, `details withheld` and **Response body withheld**
are deliberate privacy messages, not the receiver's explanation of a failure.
Alert-webhook receivers can echo tokens, alert content or private provider data
in their replies. Pulse discards that response text instead of copying it into
delivery errors, logs or a Test result. Do not enable Debug, capture raw traffic,
weaken a safeguard or send another Test or queue retry to recover the text.

Use the original timestamp, destination type, failure class and HTTP status
from **Recent delivery activity**. An HTTP rejection still matters when its
body is withheld; withholding alone does not mean that delivery failed.
For example, HTTP 429 is still rate limiting, not missing diagnostic data.
If the error says **failed to read response body**, reading the reply failed:
the receiver may already have accepted the request, so retrying can duplicate
it. Neither HTTP 2xx nor a completed Test proves receipt at the intended
destination or the success of queued alert delivery.

A Test summary such as `Response body withheld (128 bytes read)` records only
how much Pulse read, not the response's contents or total size. **read incomplete**
means the read failed; **read limit reached** means it reached the **1 MiB** cap,
not proof that the response exceeded that cap. A complete empty reply may have
no summary. These are not byte counts of the alert Pulse sent.

If provider-specific detail is consequential, review the receiver's existing
record locally using its normal authorised access. Share only the relevant
error code or manually redacted explanation, not response bodies, headers,
payloads or credentials. Unknown receipt remains unknown; do not replay the
request to create evidence. Older logs and retained delivery errors are **not
retroactively scrubbed**, and this safeguard does not sanitise every other error
or export. Keep those private and follow the
[log and export precautions](#inspect-notification-logs).

#### Telegram Test works but real alerts say "message text is empty"

With the built-in Telegram template (no custom template), a blank or incorrect
saved `Content-Type` can make Telegram ignore the JSON body. The HTTP 400 error
does not by itself prove that Pulse omitted the `text` field. Test can supply
the correct header even when real delivery uses the saved one.

In **Alerts → Notifications**, edit that Telegram destination's **Custom
headers**: set one static `Content-Type` header to `application/json` and remove
any conflicting duplicate, including differently capitalised names. Save,
leaving the bot URL, `chat_id`, template and grouping settings unchanged. Check
whether the next normally occurring alert reaches the intended chat and compare
its timestamp with **Recent delivery activity**; do not induce an alert or retry
the old rejected batch to test this edit. Retained jobs keep their original
settings as explained [above](#recover-retained-delivery-failures).

For a custom template, check its JSON and required Telegram fields locally
instead of assuming this built-in-template workaround applies. If reporting a
continuing failure, share only the version, redacted delivery error, timestamp
and whether a custom template is used. Keep the bot URL/token, chat ID and full
notification configuration private.

### TrueNAS

#### "TrueNAS service unavailable"

The exact `truenas_unavailable` error (HTTP **500** or **503**) concerns Pulse's
connection-management handler or configuration persistence, not proof of an
appliance outage or an invalid TrueNAS API key. Keep the existing request's
status/code and time, then inspect a bounded Pulse log excerpt locally. Do not
rotate the key, recreate the connection or weaken TLS verification to clear it.

Follow [TrueNAS error diagnosis](TRUENAS.md#truenas-service-unavailable) to
separate this from an explicitly disabled integration or a failed live test.
For missing or stale readings, use the [polling checks](TRUENAS.md#stale-truenas-data);
a successful **Test Connection** does not prove collection has recovered.

#### TrueNAS pools/datasets not appearing
- TrueNAS data appears in the unified resource model and may take one configured
  polling cycle (60 seconds by default) to appear.
- Open **TrueNAS → Overview** for the appliance, **TrueNAS → Storage**
  (`/truenas/storage`) for pools, datasets and disks, and **TrueNAS → Protection**
  (`/truenas/protection`) for snapshots and replication. These are tabs within
  TrueNAS, not separate top-level Storage or Recovery pages.
- For data that stops refreshing, use the [TrueNAS polling checks](TRUENAS.md#stale-truenas-data)
  before testing or restarting. A stale badge is not proof of an invalid key,
  and a successful connection test is not proof that collection has recovered.

### Navigation (v6)

#### Old bookmarks don't work

Current Pulse uses platform navigation. `/proxmox`, `/docker` and `/kubernetes`
are supported; do not replace them with the retired task-based routes.
Open Pulse at its base URL and use the menu to find the relevant page:

| Menu | Entry route |
| --- | --- |
| Proxmox | `/proxmox/overview` |
| Docker | `/docker/overview` |
| Kubernetes | `/kubernetes/overview` |
| TrueNAS | `/truenas/overview` |
| vSphere | `/vmware/overview` |
| Machines | `/standalone/machines` |

The short-lived top-level `/workloads`, `/storage` and `/recovery` layout is
retired. `/infrastructure` now opens the default workspace, not the former
unified host page. For old `/hosts` or `/services` bookmarks, select the current
platform or Machines page instead of assuming an automatic redirect. PBS
backups are under **Proxmox → Backups**, and TrueNAS snapshots and replication
are under **TrueNAS → Protection**.

Platform-connected hosts can appear under their platform rather than Machines.
If a menu or expected resource is missing, check its saved connection and last
successful collection in **Settings → Infrastructure**; a missing page alone
does not prove a host was deleted. Do not delete connections or re-enrol agents
just to repair a bookmark.

If menu navigation works but reloading the same URL returns a proxy 404, check
the proxy's route handling using [Reverse Proxy Configuration](REVERSE_PROXY.md).
For the current layout, see [FAQ](FAQ.md#how-is-navigation-organised-in-pulse-v6).
The [unified-navigation migration](MIGRATION_UNIFIED_NAV.md) is historical,
not a guide to the current menu.

### Relay / Mobile

#### Relay showing "Disconnected"

Pulse Mobile and Relay retire on **31 March 2027**; existing paired phones keep
working until then. **Relay is no longer sold.** Existing Relay subscribers keep
Pro features at their current price for as long as their subscription continues,
including after the app retires. Do not buy or switch plans to diagnose a
connection failure.

For an existing paired phone, use the [Relay connection checks](RELAY.md#relay-showing-disconnected).
Keep the pairing and current configuration; a relay connection is not proof
that the phone has connected. Relay connects the app, **not the web UI**.
For phone alerts after retirement, configure [ntfy, Gotify or Pushover](WEBHOOKS.md#-quick-setup)
under **Alerts → Notifications** and open Pulse in your phone's browser, using
your own VPN or tunnel when away from home.

---

## 🛠️ Advanced Diagnostics

### Collect diagnostics safely

Open **Settings → Support → Diagnostics & Health** in your existing authorised
Pulse session (`/settings/support/diagnostics` on your Pulse URL). Opening this
panel does not run diagnostics. If no result is displayed, the download buttons
are absent: use the original errors or observations rather than running checks
just to obtain an attachment. Diagnostics are optional, not a condition of reporting.

**Run Diagnostics** (shown as **Run** on phones) is not just a passive export.
It can make live Proxmox/PBS API and guest-agent requests; results may also come
from a short-lived cache. Do not run it during a backup, freeze/thaw or an
unresponsive-host incident merely to obtain a report. Keep the original errors
and existing observations instead. A successful one-off check does not prove
that normal monitoring has recovered or that a guest has thawed.

After a result is displayed, the download buttons reuse that result without
running the checks again. They save a local JSON file, **not an upload**:

- **Full (private)** retains identifying diagnostic details. Keep it private;
  do not attach it to a public issue or discussion.
- **GitHub (review first)** replaces selected infrastructure and token
  identifiers, private filesystem paths and raw disk-response fields. Counts,
  measurements, collection times and diagnostic states remain useful for
  triage. It is not a guarantee that every free-text error or future field is
  free of private information. Open the file locally and review it before sharing.

Remove credentials, cookies, secret URLs, private host/network or personal
information, including details echoed in errors or notes. Do not paste a
**Copy as cURL** command, full network export, configuration or data directory.
Share only evidence relevant to the symptom; a screenshot or exact redacted
error may be enough. See [Getting Help](#-getting-help).

The **Support Bundle** download in **Settings → Support → System Logs** is a
different export: it can
contain a complete log file or server buffer plus configuration and environment
information. It is not the **GitHub (review first)** diagnostics export and is
not limited to the lines visible in the panel. Keep the archive private and
share only relevant, manually reviewed, redacted excerpts. See the
[log-control and export limits](CONFIGURATION.md#log-levels).

### If you already posted private information

Treat a publicly posted credential as exposed, even if you later edit or delete
the post. Do not paste it again to identify it or test whether it still works.

- **Revoke access at its source.** Use a trusted administrator session to revoke
  the affected credential with its issuer. Manage Pulse API tokens in
  **Settings → API Access**; Proxmox/PBS tokens and notification-provider secrets
  need revocation at their respective providers. Removing a Pulse connection or
  stopping an agent is not credential revocation. A session cookie needs the
  authentication system's session-revocation procedure; clearing local browser
  cookies is not server-side revocation. Check the issuer's confirmation, not
  just a dismissed dialog. Revocation can interrupt monitoring or notifications;
  arrange independent coverage and configure legitimate consumers privately
  with a replacement carrying no wider permissions.
- **Remove the public copy too.** Edit the affected text and remove the attached
  file or screenshot. Editing the text alone does not remove an attachment.
  Ask GitHub for help with content you cannot remove yourself, but do not wait
  for removal before revoking access. Copies, notification emails and attachment
  links may survive; deleting a post is not proof that the information is erased.
- **Keep the remaining evidence private.** Preserve the original locally, not
  in another public upload. Public follow-up needs only the time, affected
  version, type of exposed information and relevant redacted symptoms. For
  sensitive security details, use the [private disclosure route](../SECURITY.md),
  not another issue or discussion. Do not send the credential itself or upload
  an unreviewed replacement bundle. If only private identifying details were
  exposed, seek GitHub's help with removal of copies you cannot remove yourself;
  that alone is not a reason to rotate unrelated credentials.

### Inspect Notification Logs

For notification failures, prefer **Recent delivery activity** in **Alerts →
Notifications**. These bounded Pulse log readers also apply to other server
errors; a request ID is not required. If a local log is needed, run only the
command for your deployment, on the Pulse host with an account authorised to
read its logs. For Proxmox LXC, run the systemd command
inside the Pulse container, not on the Proxmox host. Adjust the time window to
the original incident and substitute your actual service or container name
(`pulse-backend` on some older systemd installs). These examples read at most
200 records from the last 15 minutes; they do not follow the log or send a test.
Each requires **GNU `timeout`** and the listed local utilities. A **256 KiB**
byte limit complements the record limit: a single log record can be large.
The eight-second hard deadline kills only the read-only collection group,
including the reader and byte limiter, not Pulse or Docker. If a required
utility is unavailable, stop rather than running an unbounded substitute.
These are Bash examples; the enclosing subshell keeps an unavailable read from
exiting your interactive shell.

```bash
# systemd / Proxmox LXC
(
umask 077
for tool in timeout bash head mktemp wc rm cat; do
  command -v "$tool" >/dev/null 2>&1 || {
    printf 'Log read unavailable: required tool %s is missing; no unbounded fallback.\n' "$tool" >&2
    exit 1
  }
done
if ! pulse_log_file=$(mktemp "${TMPDIR:-/tmp}/pulse-log.XXXXXX" 2>/dev/null); then
  printf 'Log read unavailable: private temporary file could not be created.\n' >&2
  exit 1
fi
trap 'rm -- "$pulse_log_file"' EXIT
if timeout --signal=KILL 8s bash -o pipefail > "$pulse_log_file" 2>/dev/null <<'PULSE_LOG_READ'
journalctl -u pulse --since '15 minutes ago' --lines 200 --no-pager 2>&1 | head -c 262145
PULSE_LOG_READ
then
  if ! pulse_log_bytes=$(wc -c < "$pulse_log_file" 2>/dev/null); then
    printf 'Log read unavailable: excerpt size could not be checked.\n' >&2
    exit 1
  fi
  if [ "$pulse_log_bytes" -gt 262144 ]; then
    printf 'Log read unavailable: excerpt exceeds 256 KiB; no partial excerpt shown.\n' >&2
    exit 1
  fi
  cat -- "$pulse_log_file"
else
  pulse_log_status=$?
  printf 'Log read unavailable (exit %s); no partial excerpt shown.\n' "$pulse_log_status" >&2
  exit "$pulse_log_status"
fi
)
```

```bash
# Docker
(
umask 077
for tool in timeout bash head mktemp wc rm cat; do
  command -v "$tool" >/dev/null 2>&1 || {
    printf 'Log read unavailable: required tool %s is missing; no unbounded fallback.\n' "$tool" >&2
    exit 1
  }
done
if ! pulse_log_file=$(mktemp "${TMPDIR:-/tmp}/pulse-log.XXXXXX" 2>/dev/null); then
  printf 'Log read unavailable: private temporary file could not be created.\n' >&2
  exit 1
fi
trap 'rm -- "$pulse_log_file"' EXIT
if timeout --signal=KILL 8s bash -o pipefail > "$pulse_log_file" 2>/dev/null <<'PULSE_LOG_READ'
docker logs --since 15m --tail 200 pulse 2>&1 | head -c 262145
PULSE_LOG_READ
then
  if ! pulse_log_bytes=$(wc -c < "$pulse_log_file" 2>/dev/null); then
    printf 'Log read unavailable: excerpt size could not be checked.\n' >&2
    exit 1
  fi
  if [ "$pulse_log_bytes" -gt 262144 ]; then
    printf 'Log read unavailable: excerpt exceeds 256 KiB; no partial excerpt shown.\n' >&2
    exit 1
  fi
  cat -- "$pulse_log_file"
else
  pulse_log_status=$?
  printf 'Log read unavailable (exit %s); no partial excerpt shown.\n' "$pulse_log_status" >&2
  exit "$pulse_log_status"
fi
)
```

Application logs can be written to stdout or stderr; these commands collect
both into one owner-only temporary file, without preserving stdout/stderr
attribution, and display it **only after the reader completes successfully**
and the byte limit is checked. The file holds at most 256 KiB plus one byte
used to detect overflow; it is removed when the subshell exits. A failed,
oversized or timed-out read shows no partial excerpt that could be mistaken
for complete evidence. If the shell is forcibly interrupted, an owner-only
`pulse-log.*` file may remain in its temporary directory; remove that exact
file privately, not unrelated logs or deployment data.
Do not pipe the reader into `grep email`: it can miss SMTP or webhook errors
and hide a failed read behind a matching partial line. A nonzero reader exit,
access error or missing service/container is a failed read, not “no delivery
errors”. Even a successful empty read is inconclusive: the window, retained
logs or selected instance may differ.

These local excerpts are **not sanitised**. Do not post them wholesale. Share
only the relevant timestamp, method, HTTP status or SMTP error code and a
manually redacted error. Remove credentials, cookies, secret URLs, addresses
and private host or personal information, including anything echoed by the
provider. Never upload full environments, configuration, a queue database or
audit data. See [Getting Help](#-getting-help).

### Correlate Logs with Requests

For a failed HTTP API request, inspect its response in your authenticated
browser's **Developer tools → Network** panel. Copy only the `X-Request-ID`
response header, if present, and keep the HTTP status and time. Do not copy a
session cookie, **Copy as cURL** command or full network export into a report.
WebSocket upgrades do not pass through this request-ID middleware.

Service logs normally use JSON (`"request_id":"abc123"`); console logs may use
`request_id=abc123`. Search for the literal ID value so both formats work.
Replace `abc123` below with the response's ID. Run only the command for your
deployment, on the Pulse host using an account authorised to read its logs.
These examples limit collection to the last 15 minutes and 1,000 lines; adjust
the time window to the original incident rather than repeating the failed action.
Like the notification-log examples above, these require **GNU `timeout`**
and the listed local utilities, enforce an eight-second hard deadline and a
**256 KiB** byte limit, and search **only a successfully completed read**.
They use an owner-only temporary file with the same cleanup precautions. A
deadline, overflow or failure shows no partial match; a missing utility does
not fall back to an unbounded reader. Keep the original response status, time
and ID if the chosen window is too large; do not raise the limit or repeat the
failed action to fill that gap.

```bash
# systemd / Proxmox LXC
(
umask 077
for tool in timeout bash head mktemp wc rm grep; do
  command -v "$tool" >/dev/null 2>&1 || {
    printf 'Log read unavailable: required tool %s is missing; no unbounded fallback.\n' "$tool" >&2
    exit 1
  }
done
if ! pulse_log_file=$(mktemp "${TMPDIR:-/tmp}/pulse-log.XXXXXX" 2>/dev/null); then
  printf 'Log read unavailable: private temporary file could not be created.\n' >&2
  exit 1
fi
trap 'rm -- "$pulse_log_file"' EXIT
REQUEST_ID='abc123'
if timeout --signal=KILL 8s bash -o pipefail > "$pulse_log_file" 2>/dev/null <<'PULSE_LOG_READ'
journalctl -u pulse --since '15 minutes ago' --lines 1000 --no-pager 2>&1 | head -c 262145
PULSE_LOG_READ
then
  if ! pulse_log_bytes=$(wc -c < "$pulse_log_file" 2>/dev/null); then
    printf 'Log read unavailable: excerpt size could not be checked.\n' >&2
    exit 1
  fi
  if [ "$pulse_log_bytes" -gt 262144 ]; then
    printf 'Log read unavailable: excerpt exceeds 256 KiB; no partial excerpt shown.\n' >&2
    exit 1
  fi
  grep -aF -- "$REQUEST_ID" "$pulse_log_file"
else
  pulse_log_status=$?
  printf 'Log read unavailable (exit %s); no partial excerpt shown.\n' "$pulse_log_status" >&2
  exit "$pulse_log_status"
fi
)
```

```bash
# Docker
(
umask 077
for tool in timeout bash head mktemp wc rm grep; do
  command -v "$tool" >/dev/null 2>&1 || {
    printf 'Log read unavailable: required tool %s is missing; no unbounded fallback.\n' "$tool" >&2
    exit 1
  }
done
if ! pulse_log_file=$(mktemp "${TMPDIR:-/tmp}/pulse-log.XXXXXX" 2>/dev/null); then
  printf 'Log read unavailable: private temporary file could not be created.\n' >&2
  exit 1
fi
trap 'rm -- "$pulse_log_file"' EXIT
REQUEST_ID='abc123'
if timeout --signal=KILL 8s bash -o pipefail > "$pulse_log_file" 2>/dev/null <<'PULSE_LOG_READ'
docker logs --since 15m --tail 1000 pulse 2>&1 | head -c 262145
PULSE_LOG_READ
then
  if ! pulse_log_bytes=$(wc -c < "$pulse_log_file" 2>/dev/null); then
    printf 'Log read unavailable: excerpt size could not be checked.\n' >&2
    exit 1
  fi
  if [ "$pulse_log_bytes" -gt 262144 ]; then
    printf 'Log read unavailable: excerpt exceeds 256 KiB; no partial excerpt shown.\n' >&2
    exit 1
  fi
  grep -aF -- "$REQUEST_ID" "$pulse_log_file"
else
  pulse_log_status=$?
  printf 'Log read unavailable (exit %s); no partial excerpt shown.\n' "$pulse_log_status" >&2
  exit "$pulse_log_status"
fi
)
```

A log-reader failure is not an empty search result: resolve any access or
container/service error locally first. Exit **137** can mean the hard deadline
killed the collection group; an oversized excerpt is also unavailable. The
byte limiter can close the pipe before the reader finishes, producing a
nonzero reader exit such as **141**. These are collection results, not proof
that Pulse itself is hung. Do not restart Pulse or Docker, enable
Debug, widen collection or repeat the failed action merely to obtain logs.
Even a successful read with no match does not prove the request succeeded. At the default log level, this middleware
logs HTTP 5xx failures but not successful requests; HTTP 4xx failures are logged
at debug level. The selected window, retained logs or deployment may also differ.
Keep the original response status, time and ID even when there is no matching log;
do not enable debug logging or retry a state-changing request just to fill that gap.

These local excerpts are **not sanitised**. Before sharing a relevant line,
remove credentials, cookies, secret URLs and private host, network or personal
information. See [Getting Help](#-getting-help) for safe evidence collection.

### Check Permissions (Proxmox)

If Pulse cannot see VMs, storage or guest readings, start with the **user,
realm and token ID configured for the affected Pulse connection**, not a
guessed account or your administrator login. Use the existing connection
settings; do not dump the server configuration or expose the token secret.

With privilege separation enabled, the token's effective access is the
**intersection of the user and token permissions**: both must allow the
required privilege on the affected resource path. Ask the Proxmox
administrator to inspect both sides, including inherited ACLs and their
propagation, for that VM, storage or node. An administrator session or
user-only permission listing does not prove the token has access.

First distinguish a missing privilege on an intended target from an
**intentional access boundary**. Pulse 6.5.0 has no per-node include/exclude
setting for a Proxmox API connection: cluster discovery can list nodes outside
the token's authorised scope, and Pulse can still attempt to read them. Keep
a department-scoped token restricted to its intended nodes; **do not grant
access to other departments just to silence permission errors**. Table filters
and alert suppression do not stop API polling. Connecting to another member
of the same cluster does not limit discovery to that member. Denied readings
are unavailable, not evidence that those nodes are down or unhealthy.

Inspect the existing denial and compare it with the relevant read privileges:

- Inventory and metrics: `Sys.Audit` and `Datastore.Audit`, on the relevant
  node/storage scope.
- VM filesystem usage and guest information on PVE 9+: `VM.GuestAgent.Audit`.
- Guest memory through `/proc/meminfo` on PVE 9+: also
  `VM.GuestAgent.FileRead`.
- On PVE 8, `VM.Monitor` is the legacy guest-agent fallback, not an extra
  requirement for PVE 9+.

The built-in `PVEAuditor` role cannot be modified. Where a read privilege is
actually missing, use the existing narrow custom role and matching scoped
user/token ACLs. **Do not disable privilege separation, grant Administrator or
add guest execution/write privileges** to diagnose missing data. Do not rerun
setup or replace a token as a permissions test: setup can rotate an existing
credential. Make any necessary access repair in your normal maintenance
window, outside backups, then observe normal polling without manual
guest-agent probes. `Sys.Modify` is not a read-only monitoring privilege; see
[pending-update access](#proxmox-pending-update-access) before considering it.

Keep token secrets and full permission listings private. A report needs only
the relevant missing privilege and a redacted denial, not private identities,
resource paths or the complete ACL tree. Successful inventory does not prove
that guest readings are fresh, or that a guest has thawed.

**Rocky Linux / RHEL VMs**: File-read restrictions in `/etc/sysconfig/qemu-ga`
can explain missing guest memory; they do not by themselves establish why disk
usage is absent. Review the guest's policy before changing it. Schedule any
allowlist change or agent restart outside backups, following the guest OS's
documentation. See [VM Disk Monitoring](VM_DISK_MONITORING.md) for the distinct
permissions and backup safety boundary.

### Proxmox pending-update access

A successful inventory check does not establish access to the package-update list.
Pulse requests `GET /nodes/{node}/apt/update` with its configured Proxmox credential.
[Proxmox's endpoint implementation](https://github.com/proxmox/pve-manager/blob/614bede5d65599c67e068cbf18d49717ea8ab33b/PVE/API2/APT.pm)
requires `Sys.Modify` on `/nodes/{node}` for that GET; `Sys.Audit` alone is not sufficient
in that implementation. Check the API requirements for your installed Proxmox version.

`Sys.Modify` is broader than read-only monitoring: the same privilege also authorises
the separate POST that refreshes package indexes. Do not add it automatically to a
monitoring role just to obtain an update badge. Keeping the narrower role and an
unavailable update check is a valid choice; unavailable is not a confirmed zero.
Older Pulse wording that says “Sys.Audit permission required” does not identify the
actual missing privilege.

For diagnosis, compare the affected node and endpoint using the exact credential
configured in Pulse, not an administrator's browser session. Keep TLS verification
enabled and credentials private; report only the HTTP status and a redacted denial
or whether `data` is empty/nonempty. Do not use POST, Refresh or Upgrade as a test.
If the matching GET succeeds but Pulse remains unavailable after its next update
check, report the Pulse version and displayed check time/status separately; API
success alone does not confirm the Pulse display has recovered.

### Recovery Mode

For a forgotten local administrator password, follow
[I forgot my password](#i-forgot-my-password) above to update its active
credential source without deleting configuration or repeating first-time setup.
For OIDC, SAML or proxy login, use the identity-provider or administrator path
described there instead.

The advanced `/api/security/recovery` API creates a **browser-bound recovery
session**, not a server-wide authentication bypass. Despite its legacy name,
`disable_auth` does not disable authentication for other clients. Recovery
sessions work only over direct loopback requests; remote and reverse-proxy
requests cannot use them. A successful curl response does not unlock a separate
browser: the session cookie belongs to the client that made the request.
`enable_auth` clears that recovery session; it does not reset a password.

Do not transfer recovery cookies between clients or paste recovery tokens into
command arguments, URLs, screenshots or GitHub threads. The password-reset steps
above avoid that credential-handling detour.

---

## 🆘 Getting Help

If you're still stuck:

1. **Keep the original evidence**: note what you did, when it happened and the
   exact error. Do not repeat an update, outage or notification storm merely to
   reproduce it. A failed update banner does not prove the action left the
   target unchanged; check its current state before another attempt.
2. **Identify the affected version**: give the running Pulse and relevant agent
   versions, not just the version before an upgrade. For Docker, include the
   running image tag or digest. If installation never started Pulse, give the
   attempted release and public installer/helper source, or say "unknown".
3. **Choose relevant, safe evidence**: use existing logs, errors or observations
   first. If a result is already displayed in **Settings → Support → Diagnostics
   & Health**, **GitHub (review first)** downloads it without running checks again;
   follow [safe diagnostics collection](#collect-diagnostics-safely).
   For a visual problem, a screenshot or the exact
   error may be enough. If logs are needed, use the
   [bounded Pulse log readers](#inspect-notification-logs), choosing the command
   for the Pulse server's deployment, not the monitored target. A record limit
   alone does not bound a hung reader. If the read fails or times out, retain
   that result; do not use an unbounded substitute or treat a partial excerpt
   as complete. No request ID is required; do not repeat the failed action to
   obtain one. Never upload a full configuration or data directory.
4. **Review before posting**: even a sanitized export or screenshot can contain
   identifying details. Remove credentials, session cookies, webhook URLs and
   private host, network or personal information. Never post bootstrap/recovery
   tokens, `.env` files, private keys or an unsanitized export.
5. **Use the appropriate thread**: [GitHub Issues](https://github.com/rcourtman/Pulse/issues)
   for a bug, or [Discussions](https://github.com/rcourtman/Pulse/discussions) for
   a setup question. Add new evidence to an existing matching report rather
   than opening a duplicate. Do not refile information you have already supplied.