diff --git a/docs/UNIFIED_AGENT.md b/docs/UNIFIED_AGENT.md index 5bd325603e..e5b698559e 100644 --- a/docs/UNIFIED_AGENT.md +++ b/docs/UNIFIED_AGENT.md @@ -1165,20 +1165,48 @@ If your Docker Swarm cluster isn't being detected: ### PVE Backups Not Showing (Recovery) -If local PVE backups aren't appearing in Pulse after setting up via `--enable-proxmox`: +Pulse's **server collects PVE backup inventory through the saved Proxmox API +connection**, including storage and storage-content requests. A working host +agent or successful `--enable-proxmox` setup does not prove that this separate +API collection is current. Guest-agent disk readings are not backup inventory. -1. **Check permissions**: The API token needs `PVEDatastoreAdmin` on `/storage`: - ```bash - pveum aclmod /storage -user pulse-monitor@pve -role PVEDatastoreAdmin - pveum aclmod /storage -token 'pulse-monitor@pve!' -role PVEDatastoreAdmin - ``` - Replace `pulse-monitor@pve!` with the full token ID shown in Pulse. - Privilege-separated PVE tokens need the storage ACL on the token as well as the service user. +1. **Check existing evidence in Proxmox first.** In the native PVE interface, + identify the existing archive's node, storage, guest type/ID and time. Do not + run another backup or restore to create evidence. A missing Pulse row is not + proof that the archive is absent, and an OK task does not prove guest thaw. +2. **Compare the same source in Pulse.** Open **Proxmox → Backups → By date** + and check the selected connection and filters against that archive. Coverage + posture is a different reading: expand its explanation rather than treating + an unprotected badge as a missing backup. Direct PBS collection and backups + seen through PVE are distinct sources; see [PBS data sources](PBS.md#data-source-indicator) + and [Recovery](RECOVERY.md#missing-or-inconsistent-evidence). +3. **Inspect ordinary collection, not a new probe.** In **Settings → + Infrastructure**, check the affected saved PVE connection and existing + status. Keep connection/agent liveness separate from backup collection + freshness; use existing collection timestamps where available, and retain + the original redacted error and time. The server checks online nodes and queryable storage configured for + backup content. A failed, unavailable or partial read is not an empty + inventory; a successful connection test is not proof of sustained collection. +4. **Repair access only when the evidence supports it.** Ask the Proxmox + administrator to compare the actual rejected endpoint and installed PVE + version with the configured service user and token's effective storage + permissions. With privilege separation, both user and token ACLs must allow + the request, including the relevant scope and inheritance. Audit-only + inventory access does not establish access to every backup-content request. + Do not apply blanket storage-administrator ACLs merely because a table is + empty, disable privilege separation or use an administrator token as a test. + See [Proxmox permission checks](TROUBLESHOOTING.md#check-permissions-proxmox). -2. **Re-run setup**: Delete the node in Pulse Settings and re-run the agent with `--enable-proxmox`. Recent versions grant this permission automatically. +**Do not delete the monitored node, remove registration state or rerun setup as +a diagnostic.** Setup can rotate an existing API token and change permissions; +it is not a harmless refresh. Preserve the connection, credentials, registration +state and backup/history data. Make any evidenced configuration repair through +normal maintenance outside backups, then observe ordinary polling without +manual guest-agent probes, restarts or repeated **Test Connection** requests. -3. **Check state file**: If re-running doesn't trigger setup, remove the state file: - ```bash - rm /var/lib/pulse-agent/proxmox-pve-registered - ``` - Then restart the agent. +For freeze-enabled backups, follow the [backup safety precaution](VM_DISK_MONITORING.md#backup-safety). +It is not an incident recovery procedure or proof of a repaired thaw failure. +If reporting a continuing mismatch, retain the running server version, source, +original time, relevant missing privilege or redacted error and the differing +native/Pulse observations. Keep token secrets, full ACL listings and private +infrastructure details out of public reports; follow [Getting Help](TROUBLESHOOTING.md#-getting-help). diff --git a/frontend-modern/public/docs/UNIFIED_AGENT.md b/frontend-modern/public/docs/UNIFIED_AGENT.md index 5bd325603e..e5b698559e 100644 --- a/frontend-modern/public/docs/UNIFIED_AGENT.md +++ b/frontend-modern/public/docs/UNIFIED_AGENT.md @@ -1165,20 +1165,48 @@ If your Docker Swarm cluster isn't being detected: ### PVE Backups Not Showing (Recovery) -If local PVE backups aren't appearing in Pulse after setting up via `--enable-proxmox`: +Pulse's **server collects PVE backup inventory through the saved Proxmox API +connection**, including storage and storage-content requests. A working host +agent or successful `--enable-proxmox` setup does not prove that this separate +API collection is current. Guest-agent disk readings are not backup inventory. -1. **Check permissions**: The API token needs `PVEDatastoreAdmin` on `/storage`: - ```bash - pveum aclmod /storage -user pulse-monitor@pve -role PVEDatastoreAdmin - pveum aclmod /storage -token 'pulse-monitor@pve!' -role PVEDatastoreAdmin - ``` - Replace `pulse-monitor@pve!` with the full token ID shown in Pulse. - Privilege-separated PVE tokens need the storage ACL on the token as well as the service user. +1. **Check existing evidence in Proxmox first.** In the native PVE interface, + identify the existing archive's node, storage, guest type/ID and time. Do not + run another backup or restore to create evidence. A missing Pulse row is not + proof that the archive is absent, and an OK task does not prove guest thaw. +2. **Compare the same source in Pulse.** Open **Proxmox → Backups → By date** + and check the selected connection and filters against that archive. Coverage + posture is a different reading: expand its explanation rather than treating + an unprotected badge as a missing backup. Direct PBS collection and backups + seen through PVE are distinct sources; see [PBS data sources](PBS.md#data-source-indicator) + and [Recovery](RECOVERY.md#missing-or-inconsistent-evidence). +3. **Inspect ordinary collection, not a new probe.** In **Settings → + Infrastructure**, check the affected saved PVE connection and existing + status. Keep connection/agent liveness separate from backup collection + freshness; use existing collection timestamps where available, and retain + the original redacted error and time. The server checks online nodes and queryable storage configured for + backup content. A failed, unavailable or partial read is not an empty + inventory; a successful connection test is not proof of sustained collection. +4. **Repair access only when the evidence supports it.** Ask the Proxmox + administrator to compare the actual rejected endpoint and installed PVE + version with the configured service user and token's effective storage + permissions. With privilege separation, both user and token ACLs must allow + the request, including the relevant scope and inheritance. Audit-only + inventory access does not establish access to every backup-content request. + Do not apply blanket storage-administrator ACLs merely because a table is + empty, disable privilege separation or use an administrator token as a test. + See [Proxmox permission checks](TROUBLESHOOTING.md#check-permissions-proxmox). -2. **Re-run setup**: Delete the node in Pulse Settings and re-run the agent with `--enable-proxmox`. Recent versions grant this permission automatically. +**Do not delete the monitored node, remove registration state or rerun setup as +a diagnostic.** Setup can rotate an existing API token and change permissions; +it is not a harmless refresh. Preserve the connection, credentials, registration +state and backup/history data. Make any evidenced configuration repair through +normal maintenance outside backups, then observe ordinary polling without +manual guest-agent probes, restarts or repeated **Test Connection** requests. -3. **Check state file**: If re-running doesn't trigger setup, remove the state file: - ```bash - rm /var/lib/pulse-agent/proxmox-pve-registered - ``` - Then restart the agent. +For freeze-enabled backups, follow the [backup safety precaution](VM_DISK_MONITORING.md#backup-safety). +It is not an incident recovery procedure or proof of a repaired thaw failure. +If reporting a continuing mismatch, retain the running server version, source, +original time, relevant missing privilege or redacted error and the differing +native/Pulse observations. Keep token secrets, full ACL listings and private +infrastructure details out of public reports; follow [Getting Help](TROUBLESHOOTING.md#-getting-help). diff --git a/scripts/tests/test_pve_backup_troubleshooting_docs.py b/scripts/tests/test_pve_backup_troubleshooting_docs.py new file mode 100644 index 0000000000..b871737f8f --- /dev/null +++ b/scripts/tests/test_pve_backup_troubleshooting_docs.py @@ -0,0 +1,84 @@ +#!/usr/bin/env python3 +"""Keep missing-backup guidance on the real collector and non-destructive path. + +No provider calls, setup commands, backups or guest-agent probes are executed. +""" + +from pathlib import Path +import re +import unittest + + +ROOT = Path(__file__).resolve().parents[2] +GUIDE = "UNIFIED_AGENT.md" +HEADING = "### PVE Backups Not Showing (Recovery)" + + +def backup_section(): + text = (ROOT / "docs" / GUIDE).read_text(encoding="utf-8") + return text.split(HEADING, 1)[1].split("\n### ", 1)[0] + + +class PVEBackupTroubleshootingDocsTest(unittest.TestCase): + def test_shipped_help_matches_source(self): + self.assertEqual((ROOT / "docs" / GUIDE).read_bytes(), + (ROOT / "frontend-modern/public/docs" / GUIDE).read_bytes()) + + def test_explains_actual_server_collection_not_guest_agent_readings(self): + section = backup_section() + for phrase in ("server collects PVE backup inventory", "saved Proxmox API", + "storage-content requests", "Guest-agent disk readings are not backup inventory", + "online nodes", "queryable storage", "backup content"): + self.assertIn(phrase, section) + collector = (ROOT / "internal/monitoring/monitor_backups.go").read_text() + for operation in ("client.GetStorage(ctx, node.Node)", + "client.GetStorageContent(ctx, node.Node, storage.Storage)", + 'nodeEffectiveStatus[node.Node] != "online"', + 'strings.Contains(storage.Content, "backup")', + "storageContentQueryable(storage)"): + self.assertIn(operation, collector) + + def test_no_destructive_reenrolment_or_blanket_acl_recipe(self): + section = backup_section() + shell = "\n".join(re.findall(r"```(?:bash|sh)\n(.*?)```", section, re.DOTALL)) + self.assertNotRegex(shell, r"\brm\b|aclmod|--enable-proxmox|systemctl\s+restart") + normalised = " ".join(section.split()) + self.assertIn("Do not delete the monitored node, remove registration state or rerun setup", normalised) + self.assertIn("Setup can rotate an existing API token", normalised) + for item in ("connection", "credentials", "registration state", "backup/history data"): + self.assertIn(item, normalised) + setup = (ROOT / "internal/hostagent/proxmox_setup.go").read_text() + self.assertIn("rotating token in place", setup) + self.assertIn('"token", "remove", proxmoxUserPVE, tokenName', setup) + + def test_permissions_are_endpoint_specific_and_keep_privilege_separation(self): + section = " ".join(backup_section().split()) + for phrase in ("actual rejected endpoint", "installed PVE version", + "both user and token ACLs", "scope and inheritance", + "Audit-only inventory access does not establish access", + "Do not apply blanket storage-administrator ACLs", + "disable privilege separation", "administrator token as a test"): + self.assertIn(phrase, section) + self.assertIn("TROUBLESHOOTING.md#check-permissions-proxmox", section) + + def test_preserves_source_freshness_and_backup_safety_distinctions(self): + section = " ".join(backup_section().split()) + for phrase in ("node, storage, guest type/ID and time", "Proxmox → Backups → By date", + "Coverage posture is a different reading", "distinct sources", + "failed, unavailable or partial read is not an empty inventory", + "Keep connection/agent liveness separate from backup collection freshness", + "existing collection timestamps where available", + "OK task does not prove guest thaw", "Do not run another backup or restore", + "outside backups", "without manual guest-agent probes", + "Keep token secrets, full ACL listings and private infrastructure details"): + self.assertIn(phrase, section) + for target in ("PBS.md#data-source-indicator", "RECOVERY.md#missing-or-inconsistent-evidence", + "VM_DISK_MONITORING.md#backup-safety", "TROUBLESHOOTING.md#-getting-help"): + self.assertIn(target, section) + safety = (ROOT / "docs/VM_DISK_MONITORING.md").read_text() + self.assertIn("every filesystem covered by the backup", safety) + self.assertIn("Pulse monitoring and alerts are unavailable", safety) + + +if __name__ == "__main__": + unittest.main()