From 138901e422c18dac7afd44cac95196100ab95110 Mon Sep 17 00:00:00 2001 From: "pulse-triage[bot]" <249995291+pulse-triage[bot]@users.noreply.github.com> Date: Wed, 7 Oct 2026 01:03:57 +0100 Subject: [PATCH 1/2] Keep missing PVE backup diagnostics non-destructive Replace node deletion, registration-state removal and blanket ACL instructions with source-aware checks of existing native evidence, ordinary API collection and the actual rejected request. Preserve credential and backup safety boundaries in the shipped guide, with regression checks against the collector and setup behaviour. Contract-Neutral: Documentation-only correction of existing PVE backup diagnostics; runtime collection and access contracts are unchanged. Change-source: pulse-maintainer --- docs/UNIFIED_AGENT.md | 55 +++++++++---- frontend-modern/public/docs/UNIFIED_AGENT.md | 55 +++++++++---- .../test_pve_backup_troubleshooting_docs.py | 82 +++++++++++++++++++ 3 files changed, 164 insertions(+), 28 deletions(-) create mode 100644 scripts/tests/test_pve_backup_troubleshooting_docs.py diff --git a/docs/UNIFIED_AGENT.md b/docs/UNIFIED_AGENT.md index 5bd325603e..0b891d5ead 100644 --- a/docs/UNIFIED_AGENT.md +++ b/docs/UNIFIED_AGENT.md @@ -1165,20 +1165,47 @@ If your Docker Swarm cluster isn't being detected: ### PVE Backups Not Showing (Recovery) -If local PVE backups aren't appearing in Pulse after setting up via `--enable-proxmox`: +Pulse's **server collects PVE backup inventory through the saved Proxmox API +connection**, including storage and storage-content requests. A working host +agent or successful `--enable-proxmox` setup does not prove that this separate +API collection is current. Guest-agent disk readings are not backup inventory. -1. **Check permissions**: The API token needs `PVEDatastoreAdmin` on `/storage`: - ```bash - pveum aclmod /storage -user pulse-monitor@pve -role PVEDatastoreAdmin - pveum aclmod /storage -token 'pulse-monitor@pve!' -role PVEDatastoreAdmin - ``` - Replace `pulse-monitor@pve!` with the full token ID shown in Pulse. - Privilege-separated PVE tokens need the storage ACL on the token as well as the service user. +1. **Check existing evidence in Proxmox first.** In the native PVE interface, + identify the existing archive's node, storage, guest type/ID and time. Do not + run another backup or restore to create evidence. A missing Pulse row is not + proof that the archive is absent, and an OK task does not prove guest thaw. +2. **Compare the same source in Pulse.** Open **Proxmox → Backups → By date** + and check the selected connection and filters against that archive. Coverage + posture is a different reading: expand its explanation rather than treating + an unprotected badge as a missing backup. Direct PBS collection and backups + seen through PVE are distinct sources; see [PBS data sources](PBS.md#data-source-indicator) + and [Recovery](RECOVERY.md#missing-or-inconsistent-evidence). +3. **Inspect ordinary collection, not a new probe.** In **Settings → + Infrastructure**, check the affected PVE connection's existing status and + last successful collection, together with the original redacted error and + time. The server checks online nodes and queryable storage configured for + backup content. A failed, unavailable or partial read is not an empty + inventory; a successful connection test is not proof of sustained collection. +4. **Repair access only when the evidence supports it.** Ask the Proxmox + administrator to compare the actual rejected endpoint and installed PVE + version with the configured service user and token's effective storage + permissions. With privilege separation, both user and token ACLs must allow + the request, including the relevant scope and inheritance. Audit-only + inventory access does not establish access to every backup-content request. + Do not apply blanket storage-administrator ACLs merely because a table is + empty, disable privilege separation or use an administrator token as a test. + See [Proxmox permission checks](TROUBLESHOOTING.md#check-permissions-proxmox). -2. **Re-run setup**: Delete the node in Pulse Settings and re-run the agent with `--enable-proxmox`. Recent versions grant this permission automatically. +**Do not delete the monitored node, remove registration state or rerun setup as +a diagnostic.** Setup can rotate an existing API token and change permissions; +it is not a harmless refresh. Preserve the connection, credentials, registration +state and backup/history data. Make any evidenced configuration repair through +normal maintenance outside backups, then observe ordinary polling without +manual guest-agent probes, restarts or repeated **Test Connection** requests. -3. **Check state file**: If re-running doesn't trigger setup, remove the state file: - ```bash - rm /var/lib/pulse-agent/proxmox-pve-registered - ``` - Then restart the agent. +For freeze-enabled backups, follow the [backup safety precaution](VM_DISK_MONITORING.md#backup-safety). +It is not an incident recovery procedure or proof of a repaired thaw failure. +If reporting a continuing mismatch, retain the running server version, source, +original time, relevant missing privilege or redacted error and the differing +native/Pulse observations. Keep token secrets, full ACL listings and private +infrastructure details out of public reports; follow [Getting Help](TROUBLESHOOTING.md#-getting-help). diff --git a/frontend-modern/public/docs/UNIFIED_AGENT.md b/frontend-modern/public/docs/UNIFIED_AGENT.md index 5bd325603e..0b891d5ead 100644 --- a/frontend-modern/public/docs/UNIFIED_AGENT.md +++ b/frontend-modern/public/docs/UNIFIED_AGENT.md @@ -1165,20 +1165,47 @@ If your Docker Swarm cluster isn't being detected: ### PVE Backups Not Showing (Recovery) -If local PVE backups aren't appearing in Pulse after setting up via `--enable-proxmox`: +Pulse's **server collects PVE backup inventory through the saved Proxmox API +connection**, including storage and storage-content requests. A working host +agent or successful `--enable-proxmox` setup does not prove that this separate +API collection is current. Guest-agent disk readings are not backup inventory. -1. **Check permissions**: The API token needs `PVEDatastoreAdmin` on `/storage`: - ```bash - pveum aclmod /storage -user pulse-monitor@pve -role PVEDatastoreAdmin - pveum aclmod /storage -token 'pulse-monitor@pve!' -role PVEDatastoreAdmin - ``` - Replace `pulse-monitor@pve!` with the full token ID shown in Pulse. - Privilege-separated PVE tokens need the storage ACL on the token as well as the service user. +1. **Check existing evidence in Proxmox first.** In the native PVE interface, + identify the existing archive's node, storage, guest type/ID and time. Do not + run another backup or restore to create evidence. A missing Pulse row is not + proof that the archive is absent, and an OK task does not prove guest thaw. +2. **Compare the same source in Pulse.** Open **Proxmox → Backups → By date** + and check the selected connection and filters against that archive. Coverage + posture is a different reading: expand its explanation rather than treating + an unprotected badge as a missing backup. Direct PBS collection and backups + seen through PVE are distinct sources; see [PBS data sources](PBS.md#data-source-indicator) + and [Recovery](RECOVERY.md#missing-or-inconsistent-evidence). +3. **Inspect ordinary collection, not a new probe.** In **Settings → + Infrastructure**, check the affected PVE connection's existing status and + last successful collection, together with the original redacted error and + time. The server checks online nodes and queryable storage configured for + backup content. A failed, unavailable or partial read is not an empty + inventory; a successful connection test is not proof of sustained collection. +4. **Repair access only when the evidence supports it.** Ask the Proxmox + administrator to compare the actual rejected endpoint and installed PVE + version with the configured service user and token's effective storage + permissions. With privilege separation, both user and token ACLs must allow + the request, including the relevant scope and inheritance. Audit-only + inventory access does not establish access to every backup-content request. + Do not apply blanket storage-administrator ACLs merely because a table is + empty, disable privilege separation or use an administrator token as a test. + See [Proxmox permission checks](TROUBLESHOOTING.md#check-permissions-proxmox). -2. **Re-run setup**: Delete the node in Pulse Settings and re-run the agent with `--enable-proxmox`. Recent versions grant this permission automatically. +**Do not delete the monitored node, remove registration state or rerun setup as +a diagnostic.** Setup can rotate an existing API token and change permissions; +it is not a harmless refresh. Preserve the connection, credentials, registration +state and backup/history data. Make any evidenced configuration repair through +normal maintenance outside backups, then observe ordinary polling without +manual guest-agent probes, restarts or repeated **Test Connection** requests. -3. **Check state file**: If re-running doesn't trigger setup, remove the state file: - ```bash - rm /var/lib/pulse-agent/proxmox-pve-registered - ``` - Then restart the agent. +For freeze-enabled backups, follow the [backup safety precaution](VM_DISK_MONITORING.md#backup-safety). +It is not an incident recovery procedure or proof of a repaired thaw failure. +If reporting a continuing mismatch, retain the running server version, source, +original time, relevant missing privilege or redacted error and the differing +native/Pulse observations. Keep token secrets, full ACL listings and private +infrastructure details out of public reports; follow [Getting Help](TROUBLESHOOTING.md#-getting-help). diff --git a/scripts/tests/test_pve_backup_troubleshooting_docs.py b/scripts/tests/test_pve_backup_troubleshooting_docs.py new file mode 100644 index 0000000000..44215d7f0a --- /dev/null +++ b/scripts/tests/test_pve_backup_troubleshooting_docs.py @@ -0,0 +1,82 @@ +#!/usr/bin/env python3 +"""Keep missing-backup guidance on the real collector and non-destructive path. + +No provider calls, setup commands, backups or guest-agent probes are executed. +""" + +from pathlib import Path +import re +import unittest + + +ROOT = Path(__file__).resolve().parents[2] +GUIDE = "UNIFIED_AGENT.md" +HEADING = "### PVE Backups Not Showing (Recovery)" + + +def backup_section(): + text = (ROOT / "docs" / GUIDE).read_text(encoding="utf-8") + return text.split(HEADING, 1)[1].split("\n### ", 1)[0] + + +class PVEBackupTroubleshootingDocsTest(unittest.TestCase): + def test_shipped_help_matches_source(self): + self.assertEqual((ROOT / "docs" / GUIDE).read_bytes(), + (ROOT / "frontend-modern/public/docs" / GUIDE).read_bytes()) + + def test_explains_actual_server_collection_not_guest_agent_readings(self): + section = backup_section() + for phrase in ("server collects PVE backup inventory", "saved Proxmox API", + "storage-content requests", "Guest-agent disk readings are not backup inventory", + "online nodes", "queryable storage", "backup content"): + self.assertIn(phrase, section) + collector = (ROOT / "internal/monitoring/monitor_backups.go").read_text() + for operation in ("client.GetStorage(ctx, node.Node)", + "client.GetStorageContent(ctx, node.Node, storage.Storage)", + 'nodeEffectiveStatus[node.Node] != "online"', + 'strings.Contains(storage.Content, "backup")', + "storageContentQueryable(storage)"): + self.assertIn(operation, collector) + + def test_no_destructive_reenrolment_or_blanket_acl_recipe(self): + section = backup_section() + shell = "\n".join(re.findall(r"```(?:bash|sh)\n(.*?)```", section, re.DOTALL)) + self.assertNotRegex(shell, r"\brm\b|aclmod|--enable-proxmox|systemctl\s+restart") + normalised = " ".join(section.split()) + self.assertIn("Do not delete the monitored node, remove registration state or rerun setup", normalised) + self.assertIn("Setup can rotate an existing API token", normalised) + for item in ("connection", "credentials", "registration state", "backup/history data"): + self.assertIn(item, normalised) + setup = (ROOT / "internal/hostagent/proxmox_setup.go").read_text() + self.assertIn("rotating token in place", setup) + self.assertIn('"token", "remove", proxmoxUserPVE, tokenName', setup) + + def test_permissions_are_endpoint_specific_and_keep_privilege_separation(self): + section = " ".join(backup_section().split()) + for phrase in ("actual rejected endpoint", "installed PVE version", + "both user and token ACLs", "scope and inheritance", + "Audit-only inventory access does not establish access", + "Do not apply blanket storage-administrator ACLs", + "disable privilege separation", "administrator token as a test"): + self.assertIn(phrase, section) + self.assertIn("TROUBLESHOOTING.md#check-permissions-proxmox", section) + + def test_preserves_source_freshness_and_backup_safety_distinctions(self): + section = " ".join(backup_section().split()) + for phrase in ("node, storage, guest type/ID and time", "Proxmox → Backups → By date", + "Coverage posture is a different reading", "distinct sources", + "failed, unavailable or partial read is not an empty inventory", + "OK task does not prove guest thaw", "Do not run another backup or restore", + "outside backups", "without manual guest-agent probes", + "Keep token secrets, full ACL listings and private infrastructure details"): + self.assertIn(phrase, section) + for target in ("PBS.md#data-source-indicator", "RECOVERY.md#missing-or-inconsistent-evidence", + "VM_DISK_MONITORING.md#backup-safety", "TROUBLESHOOTING.md#-getting-help"): + self.assertIn(target, section) + safety = (ROOT / "docs/VM_DISK_MONITORING.md").read_text() + self.assertIn("every filesystem covered by the backup", safety) + self.assertIn("Pulse monitoring and alerts are unavailable", safety) + + +if __name__ == "__main__": + unittest.main() From da747553d9adf6a3cd50eae72aa5d20f00043337 Mon Sep 17 00:00:00 2001 From: "pulse-triage[bot]" <249995291+pulse-triage[bot]@users.noreply.github.com> Date: Wed, 7 Oct 2026 01:05:17 +0100 Subject: [PATCH 2/2] Keep connection liveness separate from backup freshness Source inspection shows connection/agent status is not a dedicated PVE backup last-success control. Use existing collection timestamps where available without inventing a settings field or treating a live agent as fresh backup inventory. Contract-Neutral: Documentation-only correction of existing PVE backup diagnostics; runtime collection and access contracts are unchanged. Change-source: pulse-maintainer --- docs/UNIFIED_AGENT.md | 7 ++++--- frontend-modern/public/docs/UNIFIED_AGENT.md | 7 ++++--- scripts/tests/test_pve_backup_troubleshooting_docs.py | 2 ++ 3 files changed, 10 insertions(+), 6 deletions(-) diff --git a/docs/UNIFIED_AGENT.md b/docs/UNIFIED_AGENT.md index 0b891d5ead..e5b698559e 100644 --- a/docs/UNIFIED_AGENT.md +++ b/docs/UNIFIED_AGENT.md @@ -1181,9 +1181,10 @@ API collection is current. Guest-agent disk readings are not backup inventory. seen through PVE are distinct sources; see [PBS data sources](PBS.md#data-source-indicator) and [Recovery](RECOVERY.md#missing-or-inconsistent-evidence). 3. **Inspect ordinary collection, not a new probe.** In **Settings → - Infrastructure**, check the affected PVE connection's existing status and - last successful collection, together with the original redacted error and - time. The server checks online nodes and queryable storage configured for + Infrastructure**, check the affected saved PVE connection and existing + status. Keep connection/agent liveness separate from backup collection + freshness; use existing collection timestamps where available, and retain + the original redacted error and time. The server checks online nodes and queryable storage configured for backup content. A failed, unavailable or partial read is not an empty inventory; a successful connection test is not proof of sustained collection. 4. **Repair access only when the evidence supports it.** Ask the Proxmox diff --git a/frontend-modern/public/docs/UNIFIED_AGENT.md b/frontend-modern/public/docs/UNIFIED_AGENT.md index 0b891d5ead..e5b698559e 100644 --- a/frontend-modern/public/docs/UNIFIED_AGENT.md +++ b/frontend-modern/public/docs/UNIFIED_AGENT.md @@ -1181,9 +1181,10 @@ API collection is current. Guest-agent disk readings are not backup inventory. seen through PVE are distinct sources; see [PBS data sources](PBS.md#data-source-indicator) and [Recovery](RECOVERY.md#missing-or-inconsistent-evidence). 3. **Inspect ordinary collection, not a new probe.** In **Settings → - Infrastructure**, check the affected PVE connection's existing status and - last successful collection, together with the original redacted error and - time. The server checks online nodes and queryable storage configured for + Infrastructure**, check the affected saved PVE connection and existing + status. Keep connection/agent liveness separate from backup collection + freshness; use existing collection timestamps where available, and retain + the original redacted error and time. The server checks online nodes and queryable storage configured for backup content. A failed, unavailable or partial read is not an empty inventory; a successful connection test is not proof of sustained collection. 4. **Repair access only when the evidence supports it.** Ask the Proxmox diff --git a/scripts/tests/test_pve_backup_troubleshooting_docs.py b/scripts/tests/test_pve_backup_troubleshooting_docs.py index 44215d7f0a..b871737f8f 100644 --- a/scripts/tests/test_pve_backup_troubleshooting_docs.py +++ b/scripts/tests/test_pve_backup_troubleshooting_docs.py @@ -66,6 +66,8 @@ class PVEBackupTroubleshootingDocsTest(unittest.TestCase): for phrase in ("node, storage, guest type/ID and time", "Proxmox → Backups → By date", "Coverage posture is a different reading", "distinct sources", "failed, unavailable or partial read is not an empty inventory", + "Keep connection/agent liveness separate from backup collection freshness", + "existing collection timestamps where available", "OK task does not prove guest thaw", "Do not run another backup or restore", "outside backups", "without manual guest-agent probes", "Keep token secrets, full ACL listings and private infrastructure details"):