mirror of
https://github.com/rcourtman/Pulse.git
synced 2026-10-26 21:47:20 +00:00
Integrate reviewed non-destructive PVE backup troubleshooting
Preserve monitored connections, credentials, registration state and history during diagnosis. Retain the exact documentation and regression-proof commits; source guidance is not released availability or native recovery. Change-source: pulse-maintainer
This commit is contained in:
pulse-triage[bot]
+42
-14
@@ -1165,20 +1165,48 @@ If your Docker Swarm cluster isn't being detected:
|
||||
|
||||
### PVE Backups Not Showing (Recovery)
|
||||
|
||||
If local PVE backups aren't appearing in Pulse after setting up via `--enable-proxmox`:
|
||||
Pulse's **server collects PVE backup inventory through the saved Proxmox API
|
||||
connection**, including storage and storage-content requests. A working host
|
||||
agent or successful `--enable-proxmox` setup does not prove that this separate
|
||||
API collection is current. Guest-agent disk readings are not backup inventory.
|
||||
|
||||
1. **Check permissions**: The API token needs `PVEDatastoreAdmin` on `/storage`:
|
||||
```bash
|
||||
pveum aclmod /storage -user pulse-monitor@pve -role PVEDatastoreAdmin
|
||||
pveum aclmod /storage -token 'pulse-monitor@pve!<token-name>' -role PVEDatastoreAdmin
|
||||
```
|
||||
Replace `pulse-monitor@pve!<token-name>` with the full token ID shown in Pulse.
|
||||
Privilege-separated PVE tokens need the storage ACL on the token as well as the service user.
|
||||
1. **Check existing evidence in Proxmox first.** In the native PVE interface,
|
||||
identify the existing archive's node, storage, guest type/ID and time. Do not
|
||||
run another backup or restore to create evidence. A missing Pulse row is not
|
||||
proof that the archive is absent, and an OK task does not prove guest thaw.
|
||||
2. **Compare the same source in Pulse.** Open **Proxmox → Backups → By date**
|
||||
and check the selected connection and filters against that archive. Coverage
|
||||
posture is a different reading: expand its explanation rather than treating
|
||||
an unprotected badge as a missing backup. Direct PBS collection and backups
|
||||
seen through PVE are distinct sources; see [PBS data sources](PBS.md#data-source-indicator)
|
||||
and [Recovery](RECOVERY.md#missing-or-inconsistent-evidence).
|
||||
3. **Inspect ordinary collection, not a new probe.** In **Settings →
|
||||
Infrastructure**, check the affected saved PVE connection and existing
|
||||
status. Keep connection/agent liveness separate from backup collection
|
||||
freshness; use existing collection timestamps where available, and retain
|
||||
the original redacted error and time. The server checks online nodes and queryable storage configured for
|
||||
backup content. A failed, unavailable or partial read is not an empty
|
||||
inventory; a successful connection test is not proof of sustained collection.
|
||||
4. **Repair access only when the evidence supports it.** Ask the Proxmox
|
||||
administrator to compare the actual rejected endpoint and installed PVE
|
||||
version with the configured service user and token's effective storage
|
||||
permissions. With privilege separation, both user and token ACLs must allow
|
||||
the request, including the relevant scope and inheritance. Audit-only
|
||||
inventory access does not establish access to every backup-content request.
|
||||
Do not apply blanket storage-administrator ACLs merely because a table is
|
||||
empty, disable privilege separation or use an administrator token as a test.
|
||||
See [Proxmox permission checks](TROUBLESHOOTING.md#check-permissions-proxmox).
|
||||
|
||||
2. **Re-run setup**: Delete the node in Pulse Settings and re-run the agent with `--enable-proxmox`. Recent versions grant this permission automatically.
|
||||
**Do not delete the monitored node, remove registration state or rerun setup as
|
||||
a diagnostic.** Setup can rotate an existing API token and change permissions;
|
||||
it is not a harmless refresh. Preserve the connection, credentials, registration
|
||||
state and backup/history data. Make any evidenced configuration repair through
|
||||
normal maintenance outside backups, then observe ordinary polling without
|
||||
manual guest-agent probes, restarts or repeated **Test Connection** requests.
|
||||
|
||||
3. **Check state file**: If re-running doesn't trigger setup, remove the state file:
|
||||
```bash
|
||||
rm /var/lib/pulse-agent/proxmox-pve-registered
|
||||
```
|
||||
Then restart the agent.
|
||||
For freeze-enabled backups, follow the [backup safety precaution](VM_DISK_MONITORING.md#backup-safety).
|
||||
It is not an incident recovery procedure or proof of a repaired thaw failure.
|
||||
If reporting a continuing mismatch, retain the running server version, source,
|
||||
original time, relevant missing privilege or redacted error and the differing
|
||||
native/Pulse observations. Keep token secrets, full ACL listings and private
|
||||
infrastructure details out of public reports; follow [Getting Help](TROUBLESHOOTING.md#-getting-help).
|
||||
|
||||
@@ -1165,20 +1165,48 @@ If your Docker Swarm cluster isn't being detected:
|
||||
|
||||
### PVE Backups Not Showing (Recovery)
|
||||
|
||||
If local PVE backups aren't appearing in Pulse after setting up via `--enable-proxmox`:
|
||||
Pulse's **server collects PVE backup inventory through the saved Proxmox API
|
||||
connection**, including storage and storage-content requests. A working host
|
||||
agent or successful `--enable-proxmox` setup does not prove that this separate
|
||||
API collection is current. Guest-agent disk readings are not backup inventory.
|
||||
|
||||
1. **Check permissions**: The API token needs `PVEDatastoreAdmin` on `/storage`:
|
||||
```bash
|
||||
pveum aclmod /storage -user pulse-monitor@pve -role PVEDatastoreAdmin
|
||||
pveum aclmod /storage -token 'pulse-monitor@pve!<token-name>' -role PVEDatastoreAdmin
|
||||
```
|
||||
Replace `pulse-monitor@pve!<token-name>` with the full token ID shown in Pulse.
|
||||
Privilege-separated PVE tokens need the storage ACL on the token as well as the service user.
|
||||
1. **Check existing evidence in Proxmox first.** In the native PVE interface,
|
||||
identify the existing archive's node, storage, guest type/ID and time. Do not
|
||||
run another backup or restore to create evidence. A missing Pulse row is not
|
||||
proof that the archive is absent, and an OK task does not prove guest thaw.
|
||||
2. **Compare the same source in Pulse.** Open **Proxmox → Backups → By date**
|
||||
and check the selected connection and filters against that archive. Coverage
|
||||
posture is a different reading: expand its explanation rather than treating
|
||||
an unprotected badge as a missing backup. Direct PBS collection and backups
|
||||
seen through PVE are distinct sources; see [PBS data sources](PBS.md#data-source-indicator)
|
||||
and [Recovery](RECOVERY.md#missing-or-inconsistent-evidence).
|
||||
3. **Inspect ordinary collection, not a new probe.** In **Settings →
|
||||
Infrastructure**, check the affected saved PVE connection and existing
|
||||
status. Keep connection/agent liveness separate from backup collection
|
||||
freshness; use existing collection timestamps where available, and retain
|
||||
the original redacted error and time. The server checks online nodes and queryable storage configured for
|
||||
backup content. A failed, unavailable or partial read is not an empty
|
||||
inventory; a successful connection test is not proof of sustained collection.
|
||||
4. **Repair access only when the evidence supports it.** Ask the Proxmox
|
||||
administrator to compare the actual rejected endpoint and installed PVE
|
||||
version with the configured service user and token's effective storage
|
||||
permissions. With privilege separation, both user and token ACLs must allow
|
||||
the request, including the relevant scope and inheritance. Audit-only
|
||||
inventory access does not establish access to every backup-content request.
|
||||
Do not apply blanket storage-administrator ACLs merely because a table is
|
||||
empty, disable privilege separation or use an administrator token as a test.
|
||||
See [Proxmox permission checks](TROUBLESHOOTING.md#check-permissions-proxmox).
|
||||
|
||||
2. **Re-run setup**: Delete the node in Pulse Settings and re-run the agent with `--enable-proxmox`. Recent versions grant this permission automatically.
|
||||
**Do not delete the monitored node, remove registration state or rerun setup as
|
||||
a diagnostic.** Setup can rotate an existing API token and change permissions;
|
||||
it is not a harmless refresh. Preserve the connection, credentials, registration
|
||||
state and backup/history data. Make any evidenced configuration repair through
|
||||
normal maintenance outside backups, then observe ordinary polling without
|
||||
manual guest-agent probes, restarts or repeated **Test Connection** requests.
|
||||
|
||||
3. **Check state file**: If re-running doesn't trigger setup, remove the state file:
|
||||
```bash
|
||||
rm /var/lib/pulse-agent/proxmox-pve-registered
|
||||
```
|
||||
Then restart the agent.
|
||||
For freeze-enabled backups, follow the [backup safety precaution](VM_DISK_MONITORING.md#backup-safety).
|
||||
It is not an incident recovery procedure or proof of a repaired thaw failure.
|
||||
If reporting a continuing mismatch, retain the running server version, source,
|
||||
original time, relevant missing privilege or redacted error and the differing
|
||||
native/Pulse observations. Keep token secrets, full ACL listings and private
|
||||
infrastructure details out of public reports; follow [Getting Help](TROUBLESHOOTING.md#-getting-help).
|
||||
|
||||
@@ -0,0 +1,84 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Keep missing-backup guidance on the real collector and non-destructive path.
|
||||
|
||||
No provider calls, setup commands, backups or guest-agent probes are executed.
|
||||
"""
|
||||
|
||||
from pathlib import Path
|
||||
import re
|
||||
import unittest
|
||||
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[2]
|
||||
GUIDE = "UNIFIED_AGENT.md"
|
||||
HEADING = "### PVE Backups Not Showing (Recovery)"
|
||||
|
||||
|
||||
def backup_section():
|
||||
text = (ROOT / "docs" / GUIDE).read_text(encoding="utf-8")
|
||||
return text.split(HEADING, 1)[1].split("\n### ", 1)[0]
|
||||
|
||||
|
||||
class PVEBackupTroubleshootingDocsTest(unittest.TestCase):
|
||||
def test_shipped_help_matches_source(self):
|
||||
self.assertEqual((ROOT / "docs" / GUIDE).read_bytes(),
|
||||
(ROOT / "frontend-modern/public/docs" / GUIDE).read_bytes())
|
||||
|
||||
def test_explains_actual_server_collection_not_guest_agent_readings(self):
|
||||
section = backup_section()
|
||||
for phrase in ("server collects PVE backup inventory", "saved Proxmox API",
|
||||
"storage-content requests", "Guest-agent disk readings are not backup inventory",
|
||||
"online nodes", "queryable storage", "backup content"):
|
||||
self.assertIn(phrase, section)
|
||||
collector = (ROOT / "internal/monitoring/monitor_backups.go").read_text()
|
||||
for operation in ("client.GetStorage(ctx, node.Node)",
|
||||
"client.GetStorageContent(ctx, node.Node, storage.Storage)",
|
||||
'nodeEffectiveStatus[node.Node] != "online"',
|
||||
'strings.Contains(storage.Content, "backup")',
|
||||
"storageContentQueryable(storage)"):
|
||||
self.assertIn(operation, collector)
|
||||
|
||||
def test_no_destructive_reenrolment_or_blanket_acl_recipe(self):
|
||||
section = backup_section()
|
||||
shell = "\n".join(re.findall(r"```(?:bash|sh)\n(.*?)```", section, re.DOTALL))
|
||||
self.assertNotRegex(shell, r"\brm\b|aclmod|--enable-proxmox|systemctl\s+restart")
|
||||
normalised = " ".join(section.split())
|
||||
self.assertIn("Do not delete the monitored node, remove registration state or rerun setup", normalised)
|
||||
self.assertIn("Setup can rotate an existing API token", normalised)
|
||||
for item in ("connection", "credentials", "registration state", "backup/history data"):
|
||||
self.assertIn(item, normalised)
|
||||
setup = (ROOT / "internal/hostagent/proxmox_setup.go").read_text()
|
||||
self.assertIn("rotating token in place", setup)
|
||||
self.assertIn('"token", "remove", proxmoxUserPVE, tokenName', setup)
|
||||
|
||||
def test_permissions_are_endpoint_specific_and_keep_privilege_separation(self):
|
||||
section = " ".join(backup_section().split())
|
||||
for phrase in ("actual rejected endpoint", "installed PVE version",
|
||||
"both user and token ACLs", "scope and inheritance",
|
||||
"Audit-only inventory access does not establish access",
|
||||
"Do not apply blanket storage-administrator ACLs",
|
||||
"disable privilege separation", "administrator token as a test"):
|
||||
self.assertIn(phrase, section)
|
||||
self.assertIn("TROUBLESHOOTING.md#check-permissions-proxmox", section)
|
||||
|
||||
def test_preserves_source_freshness_and_backup_safety_distinctions(self):
|
||||
section = " ".join(backup_section().split())
|
||||
for phrase in ("node, storage, guest type/ID and time", "Proxmox → Backups → By date",
|
||||
"Coverage posture is a different reading", "distinct sources",
|
||||
"failed, unavailable or partial read is not an empty inventory",
|
||||
"Keep connection/agent liveness separate from backup collection freshness",
|
||||
"existing collection timestamps where available",
|
||||
"OK task does not prove guest thaw", "Do not run another backup or restore",
|
||||
"outside backups", "without manual guest-agent probes",
|
||||
"Keep token secrets, full ACL listings and private infrastructure details"):
|
||||
self.assertIn(phrase, section)
|
||||
for target in ("PBS.md#data-source-indicator", "RECOVERY.md#missing-or-inconsistent-evidence",
|
||||
"VM_DISK_MONITORING.md#backup-safety", "TROUBLESHOOTING.md#-getting-help"):
|
||||
self.assertIn(target, section)
|
||||
safety = (ROOT / "docs/VM_DISK_MONITORING.md").read_text()
|
||||
self.assertIn("every filesystem covered by the backup", safety)
|
||||
self.assertIn("Pulse monitoring and alerts are unavailable", safety)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
Reference in New Issue
Block a user