mirror of
https://github.com/rcourtman/Pulse.git
synced 2026-10-23 20:25:56 +00:00
Make the backup monitoring pause usable and verifiable
Explain the manual systemd precaution inside the Pulse server LXC, including update timer and in-flight installer checks, stopped-state readback, independent covered-filesystem write checks before restart, and restoration of prior service states. Keep native thaw, recovery and release availability explicitly unproved. Contract-Neutral: Document existing service controls and safety limits; no runtime, route, API, persistence, permissions or release-gate changes. Change-source: pulse-maintainer
This commit is contained in:
pulse-triage[bot]
@@ -233,8 +233,11 @@ repair an older generated unit rather than adding a JSON-parsing wrapper.
|
||||
token's read permissions. Schedule any setup change or restart outside backups.
|
||||
- Do not run guest-agent probes during backup freeze/thaw. An OK backup task
|
||||
or an absent lock does not confirm thaw.
|
||||
- See [VM Disk Monitoring](VM_DISK_MONITORING.md) for the passive host preflight
|
||||
and backup safety precaution; it does not verify a fresh disk poll.
|
||||
- See [VM Disk Monitoring](VM_DISK_MONITORING.md) for the passive host preflight;
|
||||
it does not verify a fresh disk poll. If an affected installation needs to
|
||||
pause monitoring, follow the [manual backup precaution](VM_DISK_MONITORING.md#pause-pulse-for-a-planned-freeze-enabled-backup).
|
||||
It checks the server service and update timer before the backup, then requires
|
||||
independent guest thaw confirmation before starting Pulse again.
|
||||
|
||||
#### Temperature data missing
|
||||
- Install `lm-sensors` on the host.
|
||||
|
||||
@@ -24,6 +24,89 @@ or force-reset a guest as a disk-monitoring diagnostic. Those operations can
|
||||
affect workloads or backup consistency. Preserve the existing backup task and
|
||||
redacted error, rather than repeat a potentially harmful backup for a report.
|
||||
|
||||
### Pause Pulse for a planned freeze-enabled backup
|
||||
|
||||
This manual precaution is for an affected **systemd installation**, before a
|
||||
scheduled backup starts. It is **not a recovery procedure** for a backup already
|
||||
in progress or an unresponsive guest, and **not an automatic backup hook**.
|
||||
|
||||
1. **Identify the Pulse server service.** Run these commands on the machine
|
||||
running the Pulse server: for Proxmox LXC, **inside the Pulse LXC**,
|
||||
**not the Proxmox host or the backed-up VM**. The examples use
|
||||
`pulse.service`; substitute your actual server unit throughout
|
||||
(`pulse-backend.service` on some older installs), **not pulse-agent.service**.
|
||||
|
||||
```bash
|
||||
systemctl show pulse.service --property=LoadState,ActiveState,MainPID
|
||||
```
|
||||
|
||||
Require `LoadState=loaded` and record whether Pulse was active. If the unit
|
||||
is missing, do not assume monitoring has stopped: identify your deployment's
|
||||
actual stop control. Docker and other supervisors need their own controls.
|
||||
|
||||
2. **Prevent an update from starting Pulse again.** Do not install or update
|
||||
Pulse during this window. If you use unattended updates, inspect the actual
|
||||
matching timer (the default is `pulse-update.timer`):
|
||||
|
||||
```bash
|
||||
systemctl show pulse-update.timer --property=LoadState,ActiveState
|
||||
```
|
||||
|
||||
Record whether the timer was active. For a loaded timer, pause it and check
|
||||
both it and its update service, substituting your actual unit names:
|
||||
|
||||
```bash
|
||||
sudo systemctl stop pulse-update.timer
|
||||
systemctl show pulse-update.timer pulse-update.service \
|
||||
--property=Id,LoadState,ActiveState,MainPID
|
||||
```
|
||||
|
||||
The timer must be inactive. If the update service is active, activating or
|
||||
deactivating, let it finish normally before continuing. Do not interrupt an
|
||||
installation. A missing update unit is not proof that a custom updater is
|
||||
idle; check your deployment's update controls too.
|
||||
|
||||
3. **Stop Pulse and confirm it is stopped, before the backup starts.**
|
||||
|
||||
```bash
|
||||
sudo systemctl stop pulse.service
|
||||
systemctl show pulse.service --property=LoadState,ActiveState,MainPID
|
||||
```
|
||||
|
||||
Require `LoadState=loaded`, `ActiveState=inactive` and `MainPID=0` after a
|
||||
successful stop. Do not start the backup if the stop fails, the state is
|
||||
unknown or another Pulse server is still polling the affected guests.
|
||||
**Pulse monitoring and alerts are unavailable while stopped.** Do not run
|
||||
an update, start another Pulse instance or reboot its host during the pause.
|
||||
|
||||
4. **Verify the guest independently, then restore only what you paused.**
|
||||
Keep Pulse stopped until the backup has ended **and** your established
|
||||
guest-console or workload checks confirm thaw, including **fresh successful
|
||||
workload writes** to the filesystems covered by the backup. A console
|
||||
connection or a successful read alone is not enough. If thaw cannot be
|
||||
confirmed, leave Pulse stopped and use the guest/platform's recovery
|
||||
procedure; do not repeat the backup or use guest-agent probes to test it.
|
||||
|
||||
Only after that confirmation, if Pulse was active beforehand:
|
||||
|
||||
```bash
|
||||
sudo systemctl start pulse.service
|
||||
systemctl is-active pulse.service
|
||||
```
|
||||
|
||||
`active` confirms service startup, not proof that polling or alerts have
|
||||
recovered. Check normal observation times in Pulse without Run Diagnostics
|
||||
or manual guest-agent probes. Restore the update timer **only if it was
|
||||
active beforehand**, after Pulse has started:
|
||||
|
||||
```bash
|
||||
sudo systemctl start pulse-update.timer
|
||||
```
|
||||
|
||||
Leave previously inactive services/timers inactive. Arrange independent
|
||||
outage coverage and repeat the precaution for each affected backup window;
|
||||
this manual sequence does not schedule future pauses or prove a fix.
|
||||
|
||||
## 🚀 Setup
|
||||
|
||||
Plan changes outside backup windows and follow the guest's normal maintenance
|
||||
|
||||
@@ -233,8 +233,11 @@ repair an older generated unit rather than adding a JSON-parsing wrapper.
|
||||
token's read permissions. Schedule any setup change or restart outside backups.
|
||||
- Do not run guest-agent probes during backup freeze/thaw. An OK backup task
|
||||
or an absent lock does not confirm thaw.
|
||||
- See [VM Disk Monitoring](VM_DISK_MONITORING.md) for the passive host preflight
|
||||
and backup safety precaution; it does not verify a fresh disk poll.
|
||||
- See [VM Disk Monitoring](VM_DISK_MONITORING.md) for the passive host preflight;
|
||||
it does not verify a fresh disk poll. If an affected installation needs to
|
||||
pause monitoring, follow the [manual backup precaution](VM_DISK_MONITORING.md#pause-pulse-for-a-planned-freeze-enabled-backup).
|
||||
It checks the server service and update timer before the backup, then requires
|
||||
independent guest thaw confirmation before starting Pulse again.
|
||||
|
||||
#### Temperature data missing
|
||||
- Install `lm-sensors` on the host.
|
||||
|
||||
@@ -24,6 +24,89 @@ or force-reset a guest as a disk-monitoring diagnostic. Those operations can
|
||||
affect workloads or backup consistency. Preserve the existing backup task and
|
||||
redacted error, rather than repeat a potentially harmful backup for a report.
|
||||
|
||||
### Pause Pulse for a planned freeze-enabled backup
|
||||
|
||||
This manual precaution is for an affected **systemd installation**, before a
|
||||
scheduled backup starts. It is **not a recovery procedure** for a backup already
|
||||
in progress or an unresponsive guest, and **not an automatic backup hook**.
|
||||
|
||||
1. **Identify the Pulse server service.** Run these commands on the machine
|
||||
running the Pulse server: for Proxmox LXC, **inside the Pulse LXC**,
|
||||
**not the Proxmox host or the backed-up VM**. The examples use
|
||||
`pulse.service`; substitute your actual server unit throughout
|
||||
(`pulse-backend.service` on some older installs), **not pulse-agent.service**.
|
||||
|
||||
```bash
|
||||
systemctl show pulse.service --property=LoadState,ActiveState,MainPID
|
||||
```
|
||||
|
||||
Require `LoadState=loaded` and record whether Pulse was active. If the unit
|
||||
is missing, do not assume monitoring has stopped: identify your deployment's
|
||||
actual stop control. Docker and other supervisors need their own controls.
|
||||
|
||||
2. **Prevent an update from starting Pulse again.** Do not install or update
|
||||
Pulse during this window. If you use unattended updates, inspect the actual
|
||||
matching timer (the default is `pulse-update.timer`):
|
||||
|
||||
```bash
|
||||
systemctl show pulse-update.timer --property=LoadState,ActiveState
|
||||
```
|
||||
|
||||
Record whether the timer was active. For a loaded timer, pause it and check
|
||||
both it and its update service, substituting your actual unit names:
|
||||
|
||||
```bash
|
||||
sudo systemctl stop pulse-update.timer
|
||||
systemctl show pulse-update.timer pulse-update.service \
|
||||
--property=Id,LoadState,ActiveState,MainPID
|
||||
```
|
||||
|
||||
The timer must be inactive. If the update service is active, activating or
|
||||
deactivating, let it finish normally before continuing. Do not interrupt an
|
||||
installation. A missing update unit is not proof that a custom updater is
|
||||
idle; check your deployment's update controls too.
|
||||
|
||||
3. **Stop Pulse and confirm it is stopped, before the backup starts.**
|
||||
|
||||
```bash
|
||||
sudo systemctl stop pulse.service
|
||||
systemctl show pulse.service --property=LoadState,ActiveState,MainPID
|
||||
```
|
||||
|
||||
Require `LoadState=loaded`, `ActiveState=inactive` and `MainPID=0` after a
|
||||
successful stop. Do not start the backup if the stop fails, the state is
|
||||
unknown or another Pulse server is still polling the affected guests.
|
||||
**Pulse monitoring and alerts are unavailable while stopped.** Do not run
|
||||
an update, start another Pulse instance or reboot its host during the pause.
|
||||
|
||||
4. **Verify the guest independently, then restore only what you paused.**
|
||||
Keep Pulse stopped until the backup has ended **and** your established
|
||||
guest-console or workload checks confirm thaw, including **fresh successful
|
||||
workload writes** to the filesystems covered by the backup. A console
|
||||
connection or a successful read alone is not enough. If thaw cannot be
|
||||
confirmed, leave Pulse stopped and use the guest/platform's recovery
|
||||
procedure; do not repeat the backup or use guest-agent probes to test it.
|
||||
|
||||
Only after that confirmation, if Pulse was active beforehand:
|
||||
|
||||
```bash
|
||||
sudo systemctl start pulse.service
|
||||
systemctl is-active pulse.service
|
||||
```
|
||||
|
||||
`active` confirms service startup, not proof that polling or alerts have
|
||||
recovered. Check normal observation times in Pulse without Run Diagnostics
|
||||
or manual guest-agent probes. Restore the update timer **only if it was
|
||||
active beforehand**, after Pulse has started:
|
||||
|
||||
```bash
|
||||
sudo systemctl start pulse-update.timer
|
||||
```
|
||||
|
||||
Leave previously inactive services/timers inactive. Arrange independent
|
||||
outage coverage and repeat the precaution for each affected backup window;
|
||||
this manual sequence does not schedule future pauses or prove a fix.
|
||||
|
||||
## 🚀 Setup
|
||||
|
||||
Plan changes outside backup windows and follow the guest's normal maintenance
|
||||
|
||||
@@ -0,0 +1,107 @@
|
||||
import { readFileSync } from 'node:fs';
|
||||
import path from 'node:path';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
import { describe, expect, it } from 'vitest';
|
||||
import { renderDocMarkdown } from '@/features/docs/docMarkdown';
|
||||
|
||||
const root = path.resolve(path.dirname(fileURLToPath(import.meta.url)), '../../../..');
|
||||
const read = (name: string) => readFileSync(path.join(root, name), 'utf8');
|
||||
const guide = read('docs/VM_DISK_MONITORING.md');
|
||||
const section =
|
||||
guide.split('### Pause Pulse for a planned freeze-enabled backup')[1]?.split('## 🚀 Setup')[0] ??
|
||||
'';
|
||||
const words = section.replace(/\s+/g, ' ');
|
||||
|
||||
function render(markdown: string, name: string): HTMLElement {
|
||||
const article = document.createElement('article');
|
||||
article.innerHTML = renderDocMarkdown(markdown, name);
|
||||
return article;
|
||||
}
|
||||
|
||||
describe('manual backup safety help', () => {
|
||||
it('ships the complete precaution and makes it reachable from troubleshooting', () => {
|
||||
expect(read('frontend-modern/public/docs/VM_DISK_MONITORING.md')).toBe(guide);
|
||||
const troubleshooting = read('docs/TROUBLESHOOTING.md');
|
||||
expect(read('frontend-modern/public/docs/TROUBLESHOOTING.md')).toBe(troubleshooting);
|
||||
const link = render(troubleshooting, 'TROUBLESHOOTING').querySelector(
|
||||
'a[href="/docs/VM_DISK_MONITORING#pause-pulse-for-a-planned-freeze-enabled-backup"]',
|
||||
);
|
||||
expect(link?.textContent).toBe('manual backup precaution');
|
||||
expect(
|
||||
render(guide, 'VM_DISK_MONITORING').querySelector(
|
||||
'#pause-pulse-for-a-planned-freeze-enabled-backup',
|
||||
)?.textContent,
|
||||
).toBe('Pause Pulse for a planned freeze-enabled backup');
|
||||
});
|
||||
|
||||
it('keeps service control in the actual Pulse server container, not the guest or host agent', () => {
|
||||
expect(words).toContain('inside the Pulse LXC');
|
||||
expect(words).toContain('not the Proxmox host or the backed-up VM');
|
||||
expect(words).toContain('pulse-backend.service');
|
||||
expect(words).toContain('not pulse-agent.service');
|
||||
expect(words).toContain('LoadState=loaded');
|
||||
expect(words).toContain('If the unit is missing');
|
||||
});
|
||||
|
||||
it('checks updater state before server stop and preserves the original timer state', () => {
|
||||
expect(words).toContain('Record whether the timer was active');
|
||||
expect(words).toContain('let it finish normally before continuing');
|
||||
expect(words).toContain('Do not interrupt an installation');
|
||||
expect(words).toContain('only if it was active beforehand');
|
||||
const blocks = [...render(section, 'VM_DISK_MONITORING').querySelectorAll('pre code')].map(
|
||||
(code) => code.textContent?.trim(),
|
||||
);
|
||||
expect(blocks).toEqual([
|
||||
'systemctl show pulse.service --property=LoadState,ActiveState,MainPID',
|
||||
'systemctl show pulse-update.timer --property=LoadState,ActiveState',
|
||||
'sudo systemctl stop pulse-update.timer\nsystemctl show pulse-update.timer pulse-update.service \\\n --property=Id,LoadState,ActiveState,MainPID',
|
||||
'sudo systemctl stop pulse.service\nsystemctl show pulse.service --property=LoadState,ActiveState,MainPID',
|
||||
'sudo systemctl start pulse.service\nsystemctl is-active pulse.service',
|
||||
'sudo systemctl start pulse-update.timer',
|
||||
]);
|
||||
// These defaults and the stopped-service condition are from the existing
|
||||
// installer, not a new service or a promise about a custom deployment.
|
||||
const installer = read('install.sh');
|
||||
expect(installer).toContain('DEFAULT_SERVICE_NAME="pulse"');
|
||||
expect(installer).toContain('SERVICE_NAME="pulse-backend"');
|
||||
expect(installer).toContain(
|
||||
"ExecCondition=/bin/sh -c 'systemctl is-active --quiet ${service_name}'",
|
||||
);
|
||||
});
|
||||
|
||||
it('requires stopped-state readback and discloses the whole monitoring outage', () => {
|
||||
expect(words).toContain('ActiveState=inactive');
|
||||
expect(words).toContain('MainPID=0');
|
||||
expect(words).toContain('Do not start the backup if');
|
||||
expect(words).toContain('Pulse monitoring and alerts are unavailable');
|
||||
expect(words).toContain('Do not run an update, start another Pulse instance');
|
||||
expect(words).toContain('repeat the precaution for each affected backup window');
|
||||
});
|
||||
|
||||
it('requires independent workload writes before restart, never merely an OK task or an open console', () => {
|
||||
expect(words).toContain('Keep Pulse stopped until');
|
||||
expect(words).toContain('fresh successful workload writes');
|
||||
expect(words).toContain('console connection or a successful read alone is not enough');
|
||||
expect(words).toContain('not proof that polling or alerts have recovered');
|
||||
expect(words).toContain('normal observation times');
|
||||
expect(words).toContain('without Run Diagnostics or manual guest-agent probes');
|
||||
expect(guide).toContain('An OK backup task or an absent VM lock');
|
||||
expect(guide).toContain('does not prove thaw succeeded');
|
||||
expect(guide).toContain('not a claim\nthat the monitoring defect is fixed');
|
||||
});
|
||||
|
||||
it('does not turn a planned precaution into an incident recovery or an automatic restart hook', () => {
|
||||
expect(words).toContain('not a recovery procedure');
|
||||
expect(words).toContain('not an automatic backup hook');
|
||||
expect(words).toContain('If thaw cannot be confirmed, leave Pulse stopped');
|
||||
expect(guide).toContain(
|
||||
'Do not test freeze/thaw commands, clear backup locks, disable backup freezing',
|
||||
);
|
||||
const commands = [...render(section, 'VM_DISK_MONITORING').querySelectorAll('pre code')]
|
||||
.map((code) => code.textContent)
|
||||
.join('\n');
|
||||
expect(commands).not.toMatch(
|
||||
/qm |pct |pvesh |guest-fsfreeze|restart|disable|enable|mask|reset|rm |kill|curl|wget|Environment/,
|
||||
);
|
||||
});
|
||||
});
|
||||
Reference in New Issue
Block a user