diff --git a/docs/TEMPERATURE_MONITORING.md b/docs/TEMPERATURE_MONITORING.md index 624fe83fad..f491ba5621 100644 --- a/docs/TEMPERATURE_MONITORING.md +++ b/docs/TEMPERATURE_MONITORING.md @@ -183,9 +183,36 @@ the exit, relevant redacted error and affected sensor if help is needed. ## Legacy Cleanup (If Upgrading) -If you still have the old sensor proxy installed from prior releases, remove it -from each **Proxmox host** (not the Pulse container) with the supported cleanup -helper. In that host's trusted administrative shell, download it privately. +The retired `pulse-sensor-proxy` is separate from the unified agent and the +Pulse server. A server upgrade does not remove it from a Proxmox host. Do not +run cleanup merely because a temperature is missing, or reinstall/re-add a +working connection to retire it. + +**This helper is disruptive maintenance, not a diagnostic or a dry run.** It +stops legacy proxy units, removes their installed files and marked legacy SSH +keys, and edits LXC configurations containing `pulse-sensor-proxy`. It can +**stop and start running LXCs**, including the Pulse container. `--local-only` +prevents cluster SSH; it does not restrict cleanup to one container. Removing +a marked SSH key can also interrupt a legacy temperature-collection path. + +Before executing anything, an authorised Proxmox administrator must: + +1. Inspect the saved helper and the existing host configuration privately. + Identify every affected LXC, the proxy units and marked SSH keys, and record + which services and containers were active. Do not assume a familiar CT name + or an upgraded Pulse server establishes the cleanup scope. +2. Keep private backups of the affected LXC configurations, authorised-key file, + proxy installation, configuration and logs. These are recovery material, not + an issue attachment; do not share tokens, SSH keys or complete host files. +3. Schedule an outage for all affected workloads, not just Pulse. Finish any + backup, restore or migration first. If an operation's state or the affected + container set is unknown, stop and resolve it before cleanup. Deliberately + shut down the affected LXCs through their normal maintenance procedure before + running the helper; do not rely on its forced stop/start path. Arrange + independent monitoring while Pulse or its old collection path is unavailable. + +Download preparation alone makes no host change. In the Proxmox host's trusted +administrative shell (not inside the Pulse container), save the helper privately. This refuses an existing target, symlinked configuration directory or failed/partial download; it does not overwrite an earlier helper or run a response as a command: @@ -223,26 +250,40 @@ Curl's local configuration is ignored, redirects are not followed and TLS verification stays enabled. This is an HTTPS download from the repository's current main, not a signed release asset. Inspect the complete saved script and choose the cleanup scope before running it. -Do not pipe a web response into a privileged shell: +Do not pipe a web response into a privileged shell. Only after completing the +maintenance steps above, run the inspected helper in the authorised host shell: ```bash -bash "$HOME/.config/pulse/sensor-proxy-uninstall.sh" \ - --uninstall --purge --local-only +bash "$HOME/.config/pulse/sensor-proxy-uninstall.sh" --uninstall --local-only ``` -`--local-only` avoids cluster SSH entirely; run the command once on every -Proxmox node that carried the proxy. For one cluster-wide pass instead, omit -`--local-only`. Remote nodes must already have trusted host keys in root's -normal OpenSSH user/system known_hosts files, or supply a separately -provisioned file with `--ssh-known-hosts /path/to/known_hosts`. The helper never -accepts or enrolls an unknown host key, and a missing or changed key makes the -remote portion fail after local cleanup completes. +This deliberately omits `--purge` and `--remove-proxmox-access`. The default +preserves persisted proxy state, configuration, logs and its service account; +it still removes installed proxy files and marked SSH keys. Purging would +remove recovery evidence, and removing Proxmox access would delete the +`pulse-monitor@pam` API user and its tokens, which a current connection may +still use. Neither is required to stop the retired proxy. Preserve current +monitoring credentials, connections and history; do not re-add nodes or rerun +setup as part of cleanup. -If you also want to remove the old `pulse-monitor@pam` API user and tokens before re-adding the node, include `--remove-proxmox-access`: +Use `--local-only` separately on each host confirmed to carry the proxy, during +that host's planned maintenance. Omitting it can edit SSH keys on other cluster +nodes; that is **not a cluster-wide uninstall** of their proxy services or +files. Do not broaden the operation or enrol SSH host keys to make cleanup work. -```bash -bash "$HOME/.config/pulse/sensor-proxy-uninstall.sh" \ - --uninstall --purge --remove-proxmox-access --local-only -``` +**Verify the actual result before resuming workloads.** The helper suppresses +some service and container-operation errors; exit zero or its completion message +is not proof that cleanup succeeded. Check locally that the legacy proxy and +selfheal units are inactive, the affected live LXC configurations no longer +reference the proxy, and unrelated configuration and SSH access remain intact. +Keep the original output private and stop on a warning, failed check or unknown +state rather than repeating cleanup or adding destructive flags. -Reinstalling or upgrading the Pulse container does **not** remove the sensor proxy from the host — they are separate installations. If you skip this cleanup, the selfheal timer will keep running and may generate recurring `TASK ERROR` entries in the Proxmox task log. +Restore only the containers and services recorded active before maintenance, +using their normal deployment procedure, then verify workload liveness and +ordinary Pulse/agent monitoring. A restored temperature alone does not prove +all workloads or monitoring recovered. The helper has no automatic rollback: +if restoration fails, retain the backups and use the administrator's targeted +recovery procedure; do not blindly overwrite cluster configuration or today's +data. Keep preserved proxy evidence until recovery and its normal retention +policy permit disposal. diff --git a/frontend-modern/public/docs/TEMPERATURE_MONITORING.md b/frontend-modern/public/docs/TEMPERATURE_MONITORING.md index 624fe83fad..f491ba5621 100644 --- a/frontend-modern/public/docs/TEMPERATURE_MONITORING.md +++ b/frontend-modern/public/docs/TEMPERATURE_MONITORING.md @@ -183,9 +183,36 @@ the exit, relevant redacted error and affected sensor if help is needed. ## Legacy Cleanup (If Upgrading) -If you still have the old sensor proxy installed from prior releases, remove it -from each **Proxmox host** (not the Pulse container) with the supported cleanup -helper. In that host's trusted administrative shell, download it privately. +The retired `pulse-sensor-proxy` is separate from the unified agent and the +Pulse server. A server upgrade does not remove it from a Proxmox host. Do not +run cleanup merely because a temperature is missing, or reinstall/re-add a +working connection to retire it. + +**This helper is disruptive maintenance, not a diagnostic or a dry run.** It +stops legacy proxy units, removes their installed files and marked legacy SSH +keys, and edits LXC configurations containing `pulse-sensor-proxy`. It can +**stop and start running LXCs**, including the Pulse container. `--local-only` +prevents cluster SSH; it does not restrict cleanup to one container. Removing +a marked SSH key can also interrupt a legacy temperature-collection path. + +Before executing anything, an authorised Proxmox administrator must: + +1. Inspect the saved helper and the existing host configuration privately. + Identify every affected LXC, the proxy units and marked SSH keys, and record + which services and containers were active. Do not assume a familiar CT name + or an upgraded Pulse server establishes the cleanup scope. +2. Keep private backups of the affected LXC configurations, authorised-key file, + proxy installation, configuration and logs. These are recovery material, not + an issue attachment; do not share tokens, SSH keys or complete host files. +3. Schedule an outage for all affected workloads, not just Pulse. Finish any + backup, restore or migration first. If an operation's state or the affected + container set is unknown, stop and resolve it before cleanup. Deliberately + shut down the affected LXCs through their normal maintenance procedure before + running the helper; do not rely on its forced stop/start path. Arrange + independent monitoring while Pulse or its old collection path is unavailable. + +Download preparation alone makes no host change. In the Proxmox host's trusted +administrative shell (not inside the Pulse container), save the helper privately. This refuses an existing target, symlinked configuration directory or failed/partial download; it does not overwrite an earlier helper or run a response as a command: @@ -223,26 +250,40 @@ Curl's local configuration is ignored, redirects are not followed and TLS verification stays enabled. This is an HTTPS download from the repository's current main, not a signed release asset. Inspect the complete saved script and choose the cleanup scope before running it. -Do not pipe a web response into a privileged shell: +Do not pipe a web response into a privileged shell. Only after completing the +maintenance steps above, run the inspected helper in the authorised host shell: ```bash -bash "$HOME/.config/pulse/sensor-proxy-uninstall.sh" \ - --uninstall --purge --local-only +bash "$HOME/.config/pulse/sensor-proxy-uninstall.sh" --uninstall --local-only ``` -`--local-only` avoids cluster SSH entirely; run the command once on every -Proxmox node that carried the proxy. For one cluster-wide pass instead, omit -`--local-only`. Remote nodes must already have trusted host keys in root's -normal OpenSSH user/system known_hosts files, or supply a separately -provisioned file with `--ssh-known-hosts /path/to/known_hosts`. The helper never -accepts or enrolls an unknown host key, and a missing or changed key makes the -remote portion fail after local cleanup completes. +This deliberately omits `--purge` and `--remove-proxmox-access`. The default +preserves persisted proxy state, configuration, logs and its service account; +it still removes installed proxy files and marked SSH keys. Purging would +remove recovery evidence, and removing Proxmox access would delete the +`pulse-monitor@pam` API user and its tokens, which a current connection may +still use. Neither is required to stop the retired proxy. Preserve current +monitoring credentials, connections and history; do not re-add nodes or rerun +setup as part of cleanup. -If you also want to remove the old `pulse-monitor@pam` API user and tokens before re-adding the node, include `--remove-proxmox-access`: +Use `--local-only` separately on each host confirmed to carry the proxy, during +that host's planned maintenance. Omitting it can edit SSH keys on other cluster +nodes; that is **not a cluster-wide uninstall** of their proxy services or +files. Do not broaden the operation or enrol SSH host keys to make cleanup work. -```bash -bash "$HOME/.config/pulse/sensor-proxy-uninstall.sh" \ - --uninstall --purge --remove-proxmox-access --local-only -``` +**Verify the actual result before resuming workloads.** The helper suppresses +some service and container-operation errors; exit zero or its completion message +is not proof that cleanup succeeded. Check locally that the legacy proxy and +selfheal units are inactive, the affected live LXC configurations no longer +reference the proxy, and unrelated configuration and SSH access remain intact. +Keep the original output private and stop on a warning, failed check or unknown +state rather than repeating cleanup or adding destructive flags. -Reinstalling or upgrading the Pulse container does **not** remove the sensor proxy from the host — they are separate installations. If you skip this cleanup, the selfheal timer will keep running and may generate recurring `TASK ERROR` entries in the Proxmox task log. +Restore only the containers and services recorded active before maintenance, +using their normal deployment procedure, then verify workload liveness and +ordinary Pulse/agent monitoring. A restored temperature alone does not prove +all workloads or monitoring recovered. The helper has no automatic rollback: +if restoration fails, retain the backups and use the administrator's targeted +recovery procedure; do not blindly overwrite cluster configuration or today's +data. Keep preserved proxy evidence until recovery and its normal retention +policy permit disposal. diff --git a/scripts/tests/test_agent_docs.py b/scripts/tests/test_agent_docs.py index 0ad67e2d41..6e7acf089c 100644 --- a/scripts/tests/test_agent_docs.py +++ b/scripts/tests/test_agent_docs.py @@ -81,7 +81,7 @@ class AgentDocsTest(unittest.TestCase): with self.subTest(name=name, command=block): subprocess.run(["bash", "-n", "-c", block], check=True, capture_output=True) count += 1 - self.assertEqual(count, 18) + self.assertEqual(count, 17) def test_private_file_preparation_preserves_existing_token(self): command = recipe(NAMES[0], 'credential_file="$config_dir/agent-token"') @@ -387,11 +387,9 @@ PY def test_retarget_uninstall_and_cleanup_use_saved_script_without_new_token(self): expected = (["--retarget", "--url", "https://pulse.example.com"], ["--uninstall"], - ["--uninstall", "--purge", "--local-only"], - ["--uninstall", "--purge", "--remove-proxmox-access", "--local-only"]) + ["--uninstall", "--local-only"]) commands = [recipe(NAMES[0], "--retarget --url"), recipe(NAMES[0], 'agent-install.sh" --uninstall'), - recipe(NAMES[1], "--uninstall --purge --local-only"), - recipe(NAMES[1], "--uninstall --purge --remove-proxmox-access")] + recipe(NAMES[1], "--uninstall --local-only")] with tempfile.TemporaryDirectory() as temporary: home = Path(temporary) private = home / ".config/pulse" diff --git a/scripts/tests/test_pve_backup_troubleshooting_docs.py b/scripts/tests/test_pve_backup_troubleshooting_docs.py index 7359a6f7e3..e14bfa233a 100644 --- a/scripts/tests/test_pve_backup_troubleshooting_docs.py +++ b/scripts/tests/test_pve_backup_troubleshooting_docs.py @@ -5,7 +5,11 @@ No provider calls, setup commands, backups or guest-agent probes are executed. """ from pathlib import Path +import json +import os import re +import subprocess +import tempfile import unittest @@ -140,6 +144,83 @@ class PVEBackupTroubleshootingDocsTest(unittest.TestCase): self.assertIn("Pulse monitoring and alerts are unavailable", safety) +class LegacySensorCleanupDocsTest(unittest.TestCase): + """Exercise the copied invocation only, never the real host cleanup helper.""" + + def section(self): + return (ROOT / "docs/TEMPERATURE_MONITORING.md").read_text().split( + "## Legacy Cleanup (If Upgrading)\n", 1 + )[1] + + def invocation(self): + commands = re.findall(r"```bash\n(.*?)```", self.section(), re.DOTALL) + invocations = [c for c in commands if c.lstrip().startswith('bash ')] + self.assertEqual(len(invocations), 1, "one deliberate local cleanup only") + return invocations[0] + + def test_copied_cleanup_is_one_local_preserving_action(self): + command = self.invocation() + self.assertNotRegex(command, r"--purge|--remove-proxmox-access|--ssh-known-hosts|\|\||;") + with tempfile.TemporaryDirectory() as temporary: + home = Path(temporary) + private = home / ".config/pulse" + private.mkdir(parents=True) + # Only this synthetic helper executes. No pct/systemctl/SSH or + # credential operation is available through it. + helper = private / "sensor-proxy-uninstall.sh" + helper.write_text('''python3 - "$@" <<'PY' +import json, os, sys +from pathlib import Path +Path(os.environ['CALLS']).write_text(json.dumps(sys.argv[1:])) +sys.exit(int(os.environ['HELPER_EXIT'])) +PY +''') + witness = home / "preserved-config-and-log" + witness.write_text("unchanged synthetic recovery evidence\n") + calls = home / "calls.json" + for status in (0, 29): + with self.subTest(status=status): + calls.unlink(missing_ok=True) + result = subprocess.run( + ["bash", "-c", command], text=True, capture_output=True, timeout=10, + env=dict(os.environ, HOME=str(home), CALLS=str(calls), HELPER_EXIT=str(status)), + ) + self.assertEqual(result.returncode, status, result.stdout + result.stderr) + self.assertEqual(json.loads(calls.read_text()), ["--uninstall", "--local-only"]) + self.assertEqual(witness.read_text(), "unchanged synthetic recovery evidence\n") + + def test_disruption_and_failed_operations_are_not_hidden_by_local_only(self): + text = " ".join(self.section().split()) + for phrase in ("disruptive maintenance, not a diagnostic or a dry run", + "stop and start running LXCs", "including the Pulse container", + "does not restrict cleanup to one container", "every affected LXC", + "backup, restore or migration", "shut down the affected LXCs", + "independent monitoring", "private backups", "marked SSH keys", + "exit zero or its completion message is not proof", + "no automatic rollback", "recorded active before maintenance"): + self.assertIn(phrase, text) + helper = (ROOT / "scripts/uninstall-sensor-proxy.sh").read_text() + for fact in ('timeout 30 pct stop "$ctid"', 'timeout 30 pct start "$ctid"', + 'cleanup_sensor_proxy_lines_in_conf "$conf" "$snapshot_line"', + 'systemctl "$@" >/dev/null 2>&1 || true'): + self.assertIn(fact, helper) + + def test_preserves_credentials_and_explains_remote_cleanup_scope(self): + text = " ".join(self.section().split()) + for phrase in ("preserves persisted proxy state, configuration, logs and its service account", + "current connection may still use", "Neither is required to stop", + "not re-add nodes or rerun setup", "not a cluster-wide uninstall", + "stop on a warning, failed check or unknown state", + "unrelated configuration and SSH access remain intact"): + self.assertIn(phrase, text) + helper = (ROOT / "scripts/uninstall-sensor-proxy.sh").read_text() + for fact in ("PURGE=false", "REMOVE_PROXMOX_ACCESS=false", 'if [[ "$PURGE" == "true" ]]', + 'if [[ "$REMOVE_PROXMOX_ACCESS" != "true" ]]'): + self.assertIn(fact, helper) + self.assertEqual((ROOT / "docs/TEMPERATURE_MONITORING.md").read_bytes(), + (ROOT / "frontend-modern/public/docs/TEMPERATURE_MONITORING.md").read_bytes()) + + ZFS_DOC = ROOT / "docs/ZFS_MONITORING.md"