From 3fd44b8325679e4ce7f470572af84687458aefcc Mon Sep 17 00:00:00 2001 From: "pulse-triage[bot]" <249995291+pulse-triage[bot]@users.noreply.github.com> Date: Sun, 4 Oct 2026 13:45:45 +0100 Subject: [PATCH] Make the backup monitoring pause usable and verifiable Explain the manual systemd precaution inside the Pulse server LXC, including update timer and in-flight installer checks, stopped-state readback, independent covered-filesystem write checks before restart, and restoration of prior service states. Keep native thaw, recovery and release availability explicitly unproved. Contract-Neutral: Document existing service controls and safety limits; no runtime, route, API, persistence, permissions or release-gate changes. Change-source: pulse-maintainer --- docs/TROUBLESHOOTING.md | 7 +- docs/VM_DISK_MONITORING.md | 83 ++++++++++++++ .../public/docs/TROUBLESHOOTING.md | 7 +- .../public/docs/VM_DISK_MONITORING.md | 83 ++++++++++++++ .../utils/__tests__/backupSafetyHelp.test.ts | 107 ++++++++++++++++++ 5 files changed, 283 insertions(+), 4 deletions(-) create mode 100644 frontend-modern/src/utils/__tests__/backupSafetyHelp.test.ts diff --git a/docs/TROUBLESHOOTING.md b/docs/TROUBLESHOOTING.md index 547b336a02..f05100da0a 100644 --- a/docs/TROUBLESHOOTING.md +++ b/docs/TROUBLESHOOTING.md @@ -233,8 +233,11 @@ repair an older generated unit rather than adding a JSON-parsing wrapper. token's read permissions. Schedule any setup change or restart outside backups. - Do not run guest-agent probes during backup freeze/thaw. An OK backup task or an absent lock does not confirm thaw. -- See [VM Disk Monitoring](VM_DISK_MONITORING.md) for the passive host preflight - and backup safety precaution; it does not verify a fresh disk poll. +- See [VM Disk Monitoring](VM_DISK_MONITORING.md) for the passive host preflight; + it does not verify a fresh disk poll. If an affected installation needs to + pause monitoring, follow the [manual backup precaution](VM_DISK_MONITORING.md#pause-pulse-for-a-planned-freeze-enabled-backup). + It checks the server service and update timer before the backup, then requires + independent guest thaw confirmation before starting Pulse again. #### Temperature data missing - Install `lm-sensors` on the host. diff --git a/docs/VM_DISK_MONITORING.md b/docs/VM_DISK_MONITORING.md index 9c0b762933..5b3272aba2 100644 --- a/docs/VM_DISK_MONITORING.md +++ b/docs/VM_DISK_MONITORING.md @@ -24,6 +24,89 @@ or force-reset a guest as a disk-monitoring diagnostic. Those operations can affect workloads or backup consistency. Preserve the existing backup task and redacted error, rather than repeat a potentially harmful backup for a report. +### Pause Pulse for a planned freeze-enabled backup + +This manual precaution is for an affected **systemd installation**, before a +scheduled backup starts. It is **not a recovery procedure** for a backup already +in progress or an unresponsive guest, and **not an automatic backup hook**. + +1. **Identify the Pulse server service.** Run these commands on the machine + running the Pulse server: for Proxmox LXC, **inside the Pulse LXC**, + **not the Proxmox host or the backed-up VM**. The examples use + `pulse.service`; substitute your actual server unit throughout + (`pulse-backend.service` on some older installs), **not pulse-agent.service**. + + ```bash + systemctl show pulse.service --property=LoadState,ActiveState,MainPID + ``` + + Require `LoadState=loaded` and record whether Pulse was active. If the unit + is missing, do not assume monitoring has stopped: identify your deployment's + actual stop control. Docker and other supervisors need their own controls. + +2. **Prevent an update from starting Pulse again.** Do not install or update + Pulse during this window. If you use unattended updates, inspect the actual + matching timer (the default is `pulse-update.timer`): + + ```bash + systemctl show pulse-update.timer --property=LoadState,ActiveState + ``` + + Record whether the timer was active. For a loaded timer, pause it and check + both it and its update service, substituting your actual unit names: + + ```bash + sudo systemctl stop pulse-update.timer + systemctl show pulse-update.timer pulse-update.service \ + --property=Id,LoadState,ActiveState,MainPID + ``` + + The timer must be inactive. If the update service is active, activating or + deactivating, let it finish normally before continuing. Do not interrupt an + installation. A missing update unit is not proof that a custom updater is + idle; check your deployment's update controls too. + +3. **Stop Pulse and confirm it is stopped, before the backup starts.** + + ```bash + sudo systemctl stop pulse.service + systemctl show pulse.service --property=LoadState,ActiveState,MainPID + ``` + + Require `LoadState=loaded`, `ActiveState=inactive` and `MainPID=0` after a + successful stop. Do not start the backup if the stop fails, the state is + unknown or another Pulse server is still polling the affected guests. + **Pulse monitoring and alerts are unavailable while stopped.** Do not run + an update, start another Pulse instance or reboot its host during the pause. + +4. **Verify the guest independently, then restore only what you paused.** + Keep Pulse stopped until the backup has ended **and** your established + guest-console or workload checks confirm thaw, including **fresh successful + workload writes** to the filesystems covered by the backup. A console + connection or a successful read alone is not enough. If thaw cannot be + confirmed, leave Pulse stopped and use the guest/platform's recovery + procedure; do not repeat the backup or use guest-agent probes to test it. + + Only after that confirmation, if Pulse was active beforehand: + + ```bash + sudo systemctl start pulse.service + systemctl is-active pulse.service + ``` + + `active` confirms service startup, not proof that polling or alerts have + recovered. Check normal observation times in Pulse without Run Diagnostics + or manual guest-agent probes. Restore the update timer **only if it was + active beforehand**, after Pulse has started: + + ```bash + sudo systemctl start pulse-update.timer + ``` + + Leave previously inactive services/timers inactive. Arrange independent + outage coverage and repeat the precaution for each affected backup window; + this manual sequence does not schedule future pauses or prove a fix. + ## 🚀 Setup Plan changes outside backup windows and follow the guest's normal maintenance diff --git a/frontend-modern/public/docs/TROUBLESHOOTING.md b/frontend-modern/public/docs/TROUBLESHOOTING.md index 547b336a02..f05100da0a 100644 --- a/frontend-modern/public/docs/TROUBLESHOOTING.md +++ b/frontend-modern/public/docs/TROUBLESHOOTING.md @@ -233,8 +233,11 @@ repair an older generated unit rather than adding a JSON-parsing wrapper. token's read permissions. Schedule any setup change or restart outside backups. - Do not run guest-agent probes during backup freeze/thaw. An OK backup task or an absent lock does not confirm thaw. -- See [VM Disk Monitoring](VM_DISK_MONITORING.md) for the passive host preflight - and backup safety precaution; it does not verify a fresh disk poll. +- See [VM Disk Monitoring](VM_DISK_MONITORING.md) for the passive host preflight; + it does not verify a fresh disk poll. If an affected installation needs to + pause monitoring, follow the [manual backup precaution](VM_DISK_MONITORING.md#pause-pulse-for-a-planned-freeze-enabled-backup). + It checks the server service and update timer before the backup, then requires + independent guest thaw confirmation before starting Pulse again. #### Temperature data missing - Install `lm-sensors` on the host. diff --git a/frontend-modern/public/docs/VM_DISK_MONITORING.md b/frontend-modern/public/docs/VM_DISK_MONITORING.md index 9c0b762933..5b3272aba2 100644 --- a/frontend-modern/public/docs/VM_DISK_MONITORING.md +++ b/frontend-modern/public/docs/VM_DISK_MONITORING.md @@ -24,6 +24,89 @@ or force-reset a guest as a disk-monitoring diagnostic. Those operations can affect workloads or backup consistency. Preserve the existing backup task and redacted error, rather than repeat a potentially harmful backup for a report. +### Pause Pulse for a planned freeze-enabled backup + +This manual precaution is for an affected **systemd installation**, before a +scheduled backup starts. It is **not a recovery procedure** for a backup already +in progress or an unresponsive guest, and **not an automatic backup hook**. + +1. **Identify the Pulse server service.** Run these commands on the machine + running the Pulse server: for Proxmox LXC, **inside the Pulse LXC**, + **not the Proxmox host or the backed-up VM**. The examples use + `pulse.service`; substitute your actual server unit throughout + (`pulse-backend.service` on some older installs), **not pulse-agent.service**. + + ```bash + systemctl show pulse.service --property=LoadState,ActiveState,MainPID + ``` + + Require `LoadState=loaded` and record whether Pulse was active. If the unit + is missing, do not assume monitoring has stopped: identify your deployment's + actual stop control. Docker and other supervisors need their own controls. + +2. **Prevent an update from starting Pulse again.** Do not install or update + Pulse during this window. If you use unattended updates, inspect the actual + matching timer (the default is `pulse-update.timer`): + + ```bash + systemctl show pulse-update.timer --property=LoadState,ActiveState + ``` + + Record whether the timer was active. For a loaded timer, pause it and check + both it and its update service, substituting your actual unit names: + + ```bash + sudo systemctl stop pulse-update.timer + systemctl show pulse-update.timer pulse-update.service \ + --property=Id,LoadState,ActiveState,MainPID + ``` + + The timer must be inactive. If the update service is active, activating or + deactivating, let it finish normally before continuing. Do not interrupt an + installation. A missing update unit is not proof that a custom updater is + idle; check your deployment's update controls too. + +3. **Stop Pulse and confirm it is stopped, before the backup starts.** + + ```bash + sudo systemctl stop pulse.service + systemctl show pulse.service --property=LoadState,ActiveState,MainPID + ``` + + Require `LoadState=loaded`, `ActiveState=inactive` and `MainPID=0` after a + successful stop. Do not start the backup if the stop fails, the state is + unknown or another Pulse server is still polling the affected guests. + **Pulse monitoring and alerts are unavailable while stopped.** Do not run + an update, start another Pulse instance or reboot its host during the pause. + +4. **Verify the guest independently, then restore only what you paused.** + Keep Pulse stopped until the backup has ended **and** your established + guest-console or workload checks confirm thaw, including **fresh successful + workload writes** to the filesystems covered by the backup. A console + connection or a successful read alone is not enough. If thaw cannot be + confirmed, leave Pulse stopped and use the guest/platform's recovery + procedure; do not repeat the backup or use guest-agent probes to test it. + + Only after that confirmation, if Pulse was active beforehand: + + ```bash + sudo systemctl start pulse.service + systemctl is-active pulse.service + ``` + + `active` confirms service startup, not proof that polling or alerts have + recovered. Check normal observation times in Pulse without Run Diagnostics + or manual guest-agent probes. Restore the update timer **only if it was + active beforehand**, after Pulse has started: + + ```bash + sudo systemctl start pulse-update.timer + ``` + + Leave previously inactive services/timers inactive. Arrange independent + outage coverage and repeat the precaution for each affected backup window; + this manual sequence does not schedule future pauses or prove a fix. + ## 🚀 Setup Plan changes outside backup windows and follow the guest's normal maintenance diff --git a/frontend-modern/src/utils/__tests__/backupSafetyHelp.test.ts b/frontend-modern/src/utils/__tests__/backupSafetyHelp.test.ts new file mode 100644 index 0000000000..fa22a89310 --- /dev/null +++ b/frontend-modern/src/utils/__tests__/backupSafetyHelp.test.ts @@ -0,0 +1,107 @@ +import { readFileSync } from 'node:fs'; +import path from 'node:path'; +import { fileURLToPath } from 'node:url'; +import { describe, expect, it } from 'vitest'; +import { renderDocMarkdown } from '@/features/docs/docMarkdown'; + +const root = path.resolve(path.dirname(fileURLToPath(import.meta.url)), '../../../..'); +const read = (name: string) => readFileSync(path.join(root, name), 'utf8'); +const guide = read('docs/VM_DISK_MONITORING.md'); +const section = + guide.split('### Pause Pulse for a planned freeze-enabled backup')[1]?.split('## 🚀 Setup')[0] ?? + ''; +const words = section.replace(/\s+/g, ' '); + +function render(markdown: string, name: string): HTMLElement { + const article = document.createElement('article'); + article.innerHTML = renderDocMarkdown(markdown, name); + return article; +} + +describe('manual backup safety help', () => { + it('ships the complete precaution and makes it reachable from troubleshooting', () => { + expect(read('frontend-modern/public/docs/VM_DISK_MONITORING.md')).toBe(guide); + const troubleshooting = read('docs/TROUBLESHOOTING.md'); + expect(read('frontend-modern/public/docs/TROUBLESHOOTING.md')).toBe(troubleshooting); + const link = render(troubleshooting, 'TROUBLESHOOTING').querySelector( + 'a[href="/docs/VM_DISK_MONITORING#pause-pulse-for-a-planned-freeze-enabled-backup"]', + ); + expect(link?.textContent).toBe('manual backup precaution'); + expect( + render(guide, 'VM_DISK_MONITORING').querySelector( + '#pause-pulse-for-a-planned-freeze-enabled-backup', + )?.textContent, + ).toBe('Pause Pulse for a planned freeze-enabled backup'); + }); + + it('keeps service control in the actual Pulse server container, not the guest or host agent', () => { + expect(words).toContain('inside the Pulse LXC'); + expect(words).toContain('not the Proxmox host or the backed-up VM'); + expect(words).toContain('pulse-backend.service'); + expect(words).toContain('not pulse-agent.service'); + expect(words).toContain('LoadState=loaded'); + expect(words).toContain('If the unit is missing'); + }); + + it('checks updater state before server stop and preserves the original timer state', () => { + expect(words).toContain('Record whether the timer was active'); + expect(words).toContain('let it finish normally before continuing'); + expect(words).toContain('Do not interrupt an installation'); + expect(words).toContain('only if it was active beforehand'); + const blocks = [...render(section, 'VM_DISK_MONITORING').querySelectorAll('pre code')].map( + (code) => code.textContent?.trim(), + ); + expect(blocks).toEqual([ + 'systemctl show pulse.service --property=LoadState,ActiveState,MainPID', + 'systemctl show pulse-update.timer --property=LoadState,ActiveState', + 'sudo systemctl stop pulse-update.timer\nsystemctl show pulse-update.timer pulse-update.service \\\n --property=Id,LoadState,ActiveState,MainPID', + 'sudo systemctl stop pulse.service\nsystemctl show pulse.service --property=LoadState,ActiveState,MainPID', + 'sudo systemctl start pulse.service\nsystemctl is-active pulse.service', + 'sudo systemctl start pulse-update.timer', + ]); + // These defaults and the stopped-service condition are from the existing + // installer, not a new service or a promise about a custom deployment. + const installer = read('install.sh'); + expect(installer).toContain('DEFAULT_SERVICE_NAME="pulse"'); + expect(installer).toContain('SERVICE_NAME="pulse-backend"'); + expect(installer).toContain( + "ExecCondition=/bin/sh -c 'systemctl is-active --quiet ${service_name}'", + ); + }); + + it('requires stopped-state readback and discloses the whole monitoring outage', () => { + expect(words).toContain('ActiveState=inactive'); + expect(words).toContain('MainPID=0'); + expect(words).toContain('Do not start the backup if'); + expect(words).toContain('Pulse monitoring and alerts are unavailable'); + expect(words).toContain('Do not run an update, start another Pulse instance'); + expect(words).toContain('repeat the precaution for each affected backup window'); + }); + + it('requires independent workload writes before restart, never merely an OK task or an open console', () => { + expect(words).toContain('Keep Pulse stopped until'); + expect(words).toContain('fresh successful workload writes'); + expect(words).toContain('console connection or a successful read alone is not enough'); + expect(words).toContain('not proof that polling or alerts have recovered'); + expect(words).toContain('normal observation times'); + expect(words).toContain('without Run Diagnostics or manual guest-agent probes'); + expect(guide).toContain('An OK backup task or an absent VM lock'); + expect(guide).toContain('does not prove thaw succeeded'); + expect(guide).toContain('not a claim\nthat the monitoring defect is fixed'); + }); + + it('does not turn a planned precaution into an incident recovery or an automatic restart hook', () => { + expect(words).toContain('not a recovery procedure'); + expect(words).toContain('not an automatic backup hook'); + expect(words).toContain('If thaw cannot be confirmed, leave Pulse stopped'); + expect(guide).toContain( + 'Do not test freeze/thaw commands, clear backup locks, disable backup freezing', + ); + const commands = [...render(section, 'VM_DISK_MONITORING').querySelectorAll('pre code')] + .map((code) => code.textContent) + .join('\n'); + expect(commands).not.toMatch( + /qm |pct |pvesh |guest-fsfreeze|restart|disable|enable|mask|reset|rm |kill|curl|wget|Environment/, + ); + }); +});