From 616aa5b4b9a4ebec232fb9f51e1fb032328da6b7 Mon Sep 17 00:00:00 2001 From: "pulse-triage[bot]" <249995291+pulse-triage[bot]@users.noreply.github.com> Date: Sat, 10 Oct 2026 00:46:08 +0100 Subject: [PATCH] Explain guest memory pressure without changing working workloads Separate hypervisor footprint, Linux availability and process RSS in existing troubleshooting, FAQ and threshold help. Keep comparisons on the same guest, kernel, time and units, preserve unknown or retained evidence, and avoid diagnostic probes, restarts, cache drops and blanket suppression. Contract-Neutral: Documentation and source-bound help regressions only; memory collection, Patrol decisions, alert controls and access contracts are unchanged. Change-source: pulse-maintainer --- docs/CONFIGURATION.md | 2 + docs/FAQ.md | 4 ++ docs/TROUBLESHOOTING.md | 51 ++++++++++++++ frontend-modern/public/docs/CONFIGURATION.md | 2 + frontend-modern/public/docs/FAQ.md | 4 ++ .../public/docs/TROUBLESHOOTING.md | 51 ++++++++++++++ .../tests/test_memory_troubleshooting_docs.py | 69 +++++++++++++++++++ 7 files changed, 183 insertions(+) diff --git a/docs/CONFIGURATION.md b/docs/CONFIGURATION.md index daddabce1f..b7de212224 100644 --- a/docs/CONFIGURATION.md +++ b/docs/CONFIGURATION.md @@ -595,6 +595,8 @@ pressure or an appliance fault. Check the appliance's own ordinary status where available. Do not install an agent, restart the VM or change its configuration solely to remove the reading. Pulse has no per-VM control here to replace one metric with an explicitly ignored reading. +For a safe comparison, read [guest memory, cache and available memory](TROUBLESHOOTING.md#guest-memory-is-high-but-available-memory-is-plentiful). +Memory metric Off does not dismiss a separate Patrol finding. **Zero has different meanings in different fields.** A zero *metric trigger* disables the rule, but zero *powered-off tolerance* below means immediate diff --git a/docs/FAQ.md b/docs/FAQ.md index b63fe1a319..ffb0026b7c 100644 --- a/docs/FAQ.md +++ b/docs/FAQ.md @@ -252,6 +252,10 @@ port); `*` is not a login or proxy repair. Follow the [CORS checks](TROUBLESHOOTING.md#cors-errors) before broadening browser access. ### High memory usage? +For a monitored VM, first check [guest memory, cache and available memory](TROUBLESHOOTING.md#guest-memory-is-high-but-available-memory-is-plentiful). +A high hypervisor footprint and one application's RSS measure different things; +compare existing readings from the same guest and time, not the Proxmox node. + First distinguish container usage from Pulse's resident memory (RSS); a high LXC or Docker chart alone does not establish a leak. Use the [read-only memory checks](TROUBLESHOOTING.md#memory-use-keeps-growing) for your diff --git a/docs/TROUBLESHOOTING.md b/docs/TROUBLESHOOTING.md index 8a4de47a36..60a883d740 100644 --- a/docs/TROUBLESHOOTING.md +++ b/docs/TROUBLESHOOTING.md @@ -590,8 +590,59 @@ leave the original host unchanged. Follow [Clone identity recovery](UNIFIED_AGENT.md#duplicate-agents) for the configuration precedence, systemd example and checks after restart. +#### Guest memory is high but available memory is plentiful + +A VM's hypervisor memory footprint is not the same as usable memory inside the +guest. Some Proxmox API readings include reclaimable cache or allocated memory; +a near-100% value alone does not establish guest pressure or a leak. Conversely, +a cache-heavy reading does not prove the guest is healthy. Keep an unresponsive +workload or genuine allocation failure separate from a misleading percentage. + +Compare **existing observations of the same guest, kernel and time**, with the +same units and memory allocation. The Proxmox node's available memory is not the +VM's available memory, and a single application's RSS is not total guest usage. +Do not sum process RSS to infer guest pressure: processes can share pages. + +| Reading | What it describes | What it does not establish | +| --- | --- | --- | +| Linux `available` / `MemAvailable` | An estimate of memory usable for new allocations without swapping, including some reclaimable cache. | It is not just `free`, a guarantee of every allocation succeeding, or a reading for another guest. | +| Linux `free` and `buff/cache` | Unused pages and memory used for buffers/cache. Cache can be useful and partly reclaimable. | Low `free` alone is not pressure; not all cache is immediately reclaimable. | +| Proxmox status memory | The platform's selected footprint or estimate. The collection path matters. | A cache-inclusive or allocation-based percentage is not equivalent to cache-aware guest usage. | +| Application RSS | Resident pages for that process at that time. | It is not total guest usage, Pulse's heap size, or proof that no other process is under pressure. | +| Missing, retained or undated reading | Incomplete evidence, or an earlier observation. | Missing is not zero; a prior low value or a fresh page refresh is not current recovery. | + +For a **same-guest, same-time** Linux example with **20 GiB total** and +**15.2 GiB available**, cache-aware usage is `(20 - 15.2) / 20 × 100 = 24%`. +Use the observed total, not a nominal configured limit, and do not mix GB with +GiB or kernel `kB` (1,024 bytes). This illustrates the distinction; it is not a +correction to apply blindly to another metric or evidence that an existing +Patrol finding has cleared. Do not subtract all `buff/cache` from total, or +substitute `free` for `available`. + +Keep the running Pulse/agent and platform versions, collection path (platform +API, linked Pulse agent, or both), source and original observation time where +shown. A working agent reading does not prove that the API-only path recovered. +Unknown source, time or counters can be reported as unknown; existing screenshots +and the original sequence are enough to begin a report. Do not publish raw API +responses, full `/proc` output, configuration or private host names; review and +share only the relevant counters with consistent aliases. + +[Memory metric Off](CONFIGURATION.md#metric-thresholds-off-and-inheritance) +disables that threshold rule only: it does not hide the reading, dismiss a +separate Patrol finding or establish recovery. Do not disable all monitoring or +change thresholds merely to make a comparison pass. + +Use the guest's ordinary status if it is already available; do not run +Diagnostics, guest-agent probes or another Patrol run, install an agent, restart +the guest, drop caches or change memory limits just to obtain evidence or lower +the number. If the guest is frozen or unresponsive, stop these comparisons and +follow [backup safety](VM_DISK_MONITORING.md#backup-safety), not a memory probe. + #### Memory use keeps growing +For a monitored VM, start with [guest memory interpretation](#guest-memory-is-high-but-available-memory-is-plentiful); +the recipe below measures the **Pulse server process**, not a monitored guest. + A Proxmox LXC memory chart covers the container, not just Pulse. It can include other processes and filesystem cache; Docker's displayed memory also uses its own cache accounting. Neither is interchangeable with Pulse's resident memory diff --git a/frontend-modern/public/docs/CONFIGURATION.md b/frontend-modern/public/docs/CONFIGURATION.md index daddabce1f..b7de212224 100644 --- a/frontend-modern/public/docs/CONFIGURATION.md +++ b/frontend-modern/public/docs/CONFIGURATION.md @@ -595,6 +595,8 @@ pressure or an appliance fault. Check the appliance's own ordinary status where available. Do not install an agent, restart the VM or change its configuration solely to remove the reading. Pulse has no per-VM control here to replace one metric with an explicitly ignored reading. +For a safe comparison, read [guest memory, cache and available memory](TROUBLESHOOTING.md#guest-memory-is-high-but-available-memory-is-plentiful). +Memory metric Off does not dismiss a separate Patrol finding. **Zero has different meanings in different fields.** A zero *metric trigger* disables the rule, but zero *powered-off tolerance* below means immediate diff --git a/frontend-modern/public/docs/FAQ.md b/frontend-modern/public/docs/FAQ.md index b63fe1a319..ffb0026b7c 100644 --- a/frontend-modern/public/docs/FAQ.md +++ b/frontend-modern/public/docs/FAQ.md @@ -252,6 +252,10 @@ port); `*` is not a login or proxy repair. Follow the [CORS checks](TROUBLESHOOTING.md#cors-errors) before broadening browser access. ### High memory usage? +For a monitored VM, first check [guest memory, cache and available memory](TROUBLESHOOTING.md#guest-memory-is-high-but-available-memory-is-plentiful). +A high hypervisor footprint and one application's RSS measure different things; +compare existing readings from the same guest and time, not the Proxmox node. + First distinguish container usage from Pulse's resident memory (RSS); a high LXC or Docker chart alone does not establish a leak. Use the [read-only memory checks](TROUBLESHOOTING.md#memory-use-keeps-growing) for your diff --git a/frontend-modern/public/docs/TROUBLESHOOTING.md b/frontend-modern/public/docs/TROUBLESHOOTING.md index 8a4de47a36..60a883d740 100644 --- a/frontend-modern/public/docs/TROUBLESHOOTING.md +++ b/frontend-modern/public/docs/TROUBLESHOOTING.md @@ -590,8 +590,59 @@ leave the original host unchanged. Follow [Clone identity recovery](UNIFIED_AGENT.md#duplicate-agents) for the configuration precedence, systemd example and checks after restart. +#### Guest memory is high but available memory is plentiful + +A VM's hypervisor memory footprint is not the same as usable memory inside the +guest. Some Proxmox API readings include reclaimable cache or allocated memory; +a near-100% value alone does not establish guest pressure or a leak. Conversely, +a cache-heavy reading does not prove the guest is healthy. Keep an unresponsive +workload or genuine allocation failure separate from a misleading percentage. + +Compare **existing observations of the same guest, kernel and time**, with the +same units and memory allocation. The Proxmox node's available memory is not the +VM's available memory, and a single application's RSS is not total guest usage. +Do not sum process RSS to infer guest pressure: processes can share pages. + +| Reading | What it describes | What it does not establish | +| --- | --- | --- | +| Linux `available` / `MemAvailable` | An estimate of memory usable for new allocations without swapping, including some reclaimable cache. | It is not just `free`, a guarantee of every allocation succeeding, or a reading for another guest. | +| Linux `free` and `buff/cache` | Unused pages and memory used for buffers/cache. Cache can be useful and partly reclaimable. | Low `free` alone is not pressure; not all cache is immediately reclaimable. | +| Proxmox status memory | The platform's selected footprint or estimate. The collection path matters. | A cache-inclusive or allocation-based percentage is not equivalent to cache-aware guest usage. | +| Application RSS | Resident pages for that process at that time. | It is not total guest usage, Pulse's heap size, or proof that no other process is under pressure. | +| Missing, retained or undated reading | Incomplete evidence, or an earlier observation. | Missing is not zero; a prior low value or a fresh page refresh is not current recovery. | + +For a **same-guest, same-time** Linux example with **20 GiB total** and +**15.2 GiB available**, cache-aware usage is `(20 - 15.2) / 20 × 100 = 24%`. +Use the observed total, not a nominal configured limit, and do not mix GB with +GiB or kernel `kB` (1,024 bytes). This illustrates the distinction; it is not a +correction to apply blindly to another metric or evidence that an existing +Patrol finding has cleared. Do not subtract all `buff/cache` from total, or +substitute `free` for `available`. + +Keep the running Pulse/agent and platform versions, collection path (platform +API, linked Pulse agent, or both), source and original observation time where +shown. A working agent reading does not prove that the API-only path recovered. +Unknown source, time or counters can be reported as unknown; existing screenshots +and the original sequence are enough to begin a report. Do not publish raw API +responses, full `/proc` output, configuration or private host names; review and +share only the relevant counters with consistent aliases. + +[Memory metric Off](CONFIGURATION.md#metric-thresholds-off-and-inheritance) +disables that threshold rule only: it does not hide the reading, dismiss a +separate Patrol finding or establish recovery. Do not disable all monitoring or +change thresholds merely to make a comparison pass. + +Use the guest's ordinary status if it is already available; do not run +Diagnostics, guest-agent probes or another Patrol run, install an agent, restart +the guest, drop caches or change memory limits just to obtain evidence or lower +the number. If the guest is frozen or unresponsive, stop these comparisons and +follow [backup safety](VM_DISK_MONITORING.md#backup-safety), not a memory probe. + #### Memory use keeps growing +For a monitored VM, start with [guest memory interpretation](#guest-memory-is-high-but-available-memory-is-plentiful); +the recipe below measures the **Pulse server process**, not a monitored guest. + A Proxmox LXC memory chart covers the container, not just Pulse. It can include other processes and filesystem cache; Docker's displayed memory also uses its own cache accounting. Neither is interchangeable with Pulse's resident memory diff --git a/scripts/tests/test_memory_troubleshooting_docs.py b/scripts/tests/test_memory_troubleshooting_docs.py index 1728f7d4df..430c0573a9 100644 --- a/scripts/tests/test_memory_troubleshooting_docs.py +++ b/scripts/tests/test_memory_troubleshooting_docs.py @@ -257,5 +257,74 @@ class MemoryTroubleshootingDocsTest(unittest.TestCase): r"|--follow|--token|cmdline|environ|smaps") +class GuestMemoryInterpretationDocsTest(unittest.TestCase): + HEADING = "#### Guest memory is high but available memory is plentiful\n" + ANCHOR = "#guest-memory-is-high-but-available-memory-is-plentiful" + + def guidance(self): + text = DOC.read_text(encoding="utf-8") + self.assertEqual(text.count(self.HEADING), 1) + return " ".join(text.split(self.HEADING, 1)[1].split("\n#### ", 1)[0].split()) + + def test_guest_and_process_entry_points_are_separate_and_mirrored(self): + for name in ("TROUBLESHOOTING.md", "FAQ.md", "CONFIGURATION.md"): + self.assertEqual((ROOT / "docs" / name).read_bytes(), + (ROOT / "frontend-modern/public/docs" / name).read_bytes()) + for name in ("FAQ.md", "CONFIGURATION.md"): + self.assertIn("TROUBLESHOOTING.md" + self.ANCHOR, + (ROOT / "docs" / name).read_text()) + self.assertIn(self.ANCHOR, section()) + self.assertIn("Pulse server process", section()) + self.assertIn("not a monitored guest", section()) + + def test_comparisons_keep_guest_kernel_units_and_cache_semantics(self): + guide = self.guidance() + for distinction in ( + "same guest, kernel and time", "node's available memory is not the VM's", + "single application's RSS is not total guest usage", + "Do not sum process RSS", "not all cache is immediately reclaimable", + ): + self.assertIn(distinction, guide) + self.assertIn("a guarantee of every allocation succeeding", guide) + self.assertIn("**20 GiB total** and **15.2 GiB available**", guide) + self.assertIn("(20 - 15.2) / 20 × 100 = 24%", guide) + self.assertIn("not a nominal configured limit", guide) + self.assertIn("do not mix GB with GiB or kernel `kB` (1,024 bytes)", guide) + self.assertIn("Do not subtract all `buff/cache`", guide) + self.assertIn("substitute `free` for `available`", guide) + # Existing collectors use named kernel fields, not free-table columns. + parser = (ROOT / "pkg/proxmox/client.go").read_text() + self.assertIn('"MemAvailable"', parser) + + def test_missing_or_retained_evidence_and_controls_are_not_recovery(self): + guide = self.guidance() + for distinction in ( + "Missing is not zero", "prior low value", "not current recovery", + "source and original observation time", "Unknown source, time or counters", + "not a correction to apply blindly", "Patrol finding has cleared", + "does not prove that the API-only path recovered", + "does not hide the reading, dismiss a separate Patrol finding or establish recovery", + "Do not disable all monitoring or change thresholds", + ): + self.assertIn(distinction, guide) + patrol = (ROOT / "internal/ai/patrol_guest_memory.go").read_text() + self.assertIn('"guest pressure unknown"', patrol) + self.assertIn('"N/A (guest memory unavailable; not evidence of recovery)"', patrol) + + def test_guidance_uses_existing_evidence_without_new_guest_reads(self): + guide = self.guidance() + for boundary in ( + "existing observations", "existing screenshots", "original sequence", + "Do not publish raw API responses", "full `/proc` output", + "do not run Diagnostics, guest-agent probes or another Patrol run", + "install an agent, restart the guest, drop caches or change memory limits", + "If the guest is frozen or unresponsive, stop these comparisons", + "VM_DISK_MONITORING.md#backup-safety", + ): + self.assertIn(boundary, guide) + self.assertNotIn("```", guide, "interpret evidence already collected, not a new command") + self.assertIn("CONFIGURATION.md#metric-thresholds-off-and-inheritance", guide) + + if __name__ == "__main__": unittest.main()