Make cloned-host identity guidance safe and persistent

The agent prefers its saved identity before deriving one from the OS. The old live machine-ID deletion recipe could disrupt unrelated services while leaving a Pulse collision unchanged. Explain managed overrides, precedence, private evidence and distinct-host verification in both shipped guides; cover the safe path with documentation and real-browser checks.

Contract-Neutral: Guidance and verification for existing agent identity options only; no runtime, authentication, schema or frontend behaviour changes.
Change-source: pulse-maintainer

docs/TROUBLESHOOTING.md

@@ -155,11 +155,19 @@ repair an older generated unit rather than adding a JSON-parsing wrapper.
 - See [Temperature Monitoring](TEMPERATURE_MONITORING.md).
 
 #### Docker hosts appearing/disappearing
-- **Duplicate IDs**: Cloned VMs often share `/etc/machine-id`.
-- **Fix**: Run `rm /etc/machine-id && systemd-machine-id-setup` on the clone.
-- **Identity note**: The displayed IP is not the durable identity. Pulse uses
-  the machine ID or an explicit agent ID, so two clones with the same value can
-  collapse into one record even when their hostnames or IP addresses differ.
+
+Cloned hosts can share a **saved Pulse agent ID**, not just an OS machine ID.
+The agent uses an explicit ID first, then its saved `agent-id` file, and derives
+one from the machine only when neither is available. Changing a hostname, IP or
+`/etc/machine-id` therefore does not necessarily change its Pulse identity.
+
+Compare the affected hosts in **Agent Doctor** and inspect only their configured
+`agent-id` files locally. Do not delete the OS machine ID, agent state or Pulse
+history as a troubleshooting step. If a duplicate is confirmed, give only the
+clone a stable, unique ID in its managed service or container configuration;
+leave the original host unchanged. Follow
+[Clone identity recovery](UNIFIED_AGENT.md#duplicate-agents) for the configuration
+precedence, systemd example and checks after restart.
 
 ### Notifications
 

docs/UNIFIED_AGENT.md

@@ -412,7 +412,7 @@ sudo chmod 0700 /usr/local/libexec/pulse-queue-depth
 | `--insecure` | `PULSE_INSECURE_SKIP_VERIFY` | Skip TLS verification | `false` |
 | `--allow-plaintext-http` | `PULSE_AGENT_ALLOW_PLAINTEXT_HTTP` | Allow plain HTTP to a Pulse server that does not look local (private IP, single-label, `.local`/`.lan`/`.home`/`.home.arpa`/`.internal`, or resolves to private addresses). Sends the API token in cleartext; only for networks you fully control, e.g. internal networks numbered from public IP space | `false` |
 | `--hostname` | `PULSE_HOSTNAME` | Override hostname | *(OS hostname)* |
-| `--agent-id` | `PULSE_AGENT_ID` | Unique agent identifier | *(machine-id)* |
+| `--agent-id` | `PULSE_AGENT_ID` | Unique agent identifier | *(saved ID, then machine ID)* |
 | `--report-ip` | `PULSE_REPORT_IP` | Override reported IP (multi-NIC) | *(auto)* |
 | `--disable-ceph` | `PULSE_DISABLE_CEPH` | Disable local Ceph status polling | `false` |
 | `--tag` | `PULSE_TAGS` | Apply tags (repeatable or CSV) | *(none)* |
@@ -834,20 +834,53 @@ file and re-run the installer to pick up the rotating configuration.
   identity evidence. The endpoint reports repair handoffs but does not run them.
 
 ### Duplicate Agents
-If cloned VMs appear as the same agent:
-```bash
-sudo rm /etc/machine-id && sudo systemd-machine-id-setup
-```
 
-Or set a unique agent ID:
-```bash
---agent-id my-unique-agent-id
-```
+Cloning an installed agent can copy its saved identity and credentials. Different
+hostnames, MAC addresses or IPs do not make those agents distinct in Pulse.
+Identity selection is:
 
-The displayed or reported IP is not the durable agent identity. Pulse normally
-uses the machine ID (or an explicit `--agent-id`), so cloned systems must have
-unique machine and agent IDs even when their hostnames, MAC addresses, and IPs
-differ.
+1. An explicit `--agent-id` argument (which overrides `PULSE_AGENT_ID`).
+2. `PULSE_AGENT_ID` when no explicit argument is supplied.
+3. The saved `agent-id` file, when no override is set.
+4. The machine-derived ID, with hostname as a fallback, when no saved ID exists.
+
+First compare the affected hosts in **Agent Doctor**. Inspect the identity file
+locally: the default Linux path is `/var/lib/pulse-agent/agent-id`; `--state-dir`
+changes that directory and `--agent-id-file` or `PULSE_AGENT_ID_FILE` can select
+another file. Do not upload `connection.env`, token files or whole service and
+environment dumps; older installations can contain credentials there.
+
+**Do not delete `/etc/machine-id`, agent state or Pulse history to repair a
+duplicate.** A saved agent ID survives an OS machine-ID change, and changing the
+OS identity affects more than Pulse. Prepare OS identities when provisioning a
+clone, using the distribution's instructions, not during live Pulse diagnosis.
+
+For a confirmed duplicate, keep the original host unchanged and choose one
+stable, unique override for the clone. Persist it in the configuration that
+actually starts that agent, rather than launching a second agent process:
+
+- **Systemd, without an existing `--agent-id` argument**: use
+  `sudo systemctl edit pulse-agent.service` and add the following drop-in,
+  replacing the example ID with the clone's unique ID:
+
+  ```ini
+  [Service]
+  Environment="PULSE_AGENT_ID=vm-clone-02"
+  ```
+
+  Then restart only that agent with `sudo systemctl restart pulse-agent.service`.
+  If its managed launch command already supplies `--agent-id`, change that
+  existing override instead: the argument wins over the environment variable.
+- **Docker Compose**: set `PULSE_AGENT_ID` for the clone's agent service, keeping
+  its existing volumes, collector options and token-file configuration. An
+  existing `--agent-id` in `command` takes precedence; update it instead if set.
+
+Keep the override across restarts and upgrades. After restart, verify that both
+hosts report fresh, distinct IDs in Agent Doctor and that each row's metrics
+belong to the correct host. Changing an ID does not split or recover historical
+data already combined under the old ID. If authentication or reporting fails,
+keep the bounded, redacted error for diagnosis; do not delete tokens, re-enrol
+blindly or remove the original host's record.
 
 ### Permission Denied (Docker)
 Ensure the agent can access the Docker socket:

frontend-modern/public/docs/TROUBLESHOOTING.md

@@ -155,11 +155,19 @@ repair an older generated unit rather than adding a JSON-parsing wrapper.
 - See [Temperature Monitoring](TEMPERATURE_MONITORING.md).
 
 #### Docker hosts appearing/disappearing
-- **Duplicate IDs**: Cloned VMs often share `/etc/machine-id`.
-- **Fix**: Run `rm /etc/machine-id && systemd-machine-id-setup` on the clone.
-- **Identity note**: The displayed IP is not the durable identity. Pulse uses
-  the machine ID or an explicit agent ID, so two clones with the same value can
-  collapse into one record even when their hostnames or IP addresses differ.
+
+Cloned hosts can share a **saved Pulse agent ID**, not just an OS machine ID.
+The agent uses an explicit ID first, then its saved `agent-id` file, and derives
+one from the machine only when neither is available. Changing a hostname, IP or
+`/etc/machine-id` therefore does not necessarily change its Pulse identity.
+
+Compare the affected hosts in **Agent Doctor** and inspect only their configured
+`agent-id` files locally. Do not delete the OS machine ID, agent state or Pulse
+history as a troubleshooting step. If a duplicate is confirmed, give only the
+clone a stable, unique ID in its managed service or container configuration;
+leave the original host unchanged. Follow
+[Clone identity recovery](UNIFIED_AGENT.md#duplicate-agents) for the configuration
+precedence, systemd example and checks after restart.
 
 ### Notifications
 

frontend-modern/public/docs/UNIFIED_AGENT.md

@@ -412,7 +412,7 @@ sudo chmod 0700 /usr/local/libexec/pulse-queue-depth
 | `--insecure` | `PULSE_INSECURE_SKIP_VERIFY` | Skip TLS verification | `false` |
 | `--allow-plaintext-http` | `PULSE_AGENT_ALLOW_PLAINTEXT_HTTP` | Allow plain HTTP to a Pulse server that does not look local (private IP, single-label, `.local`/`.lan`/`.home`/`.home.arpa`/`.internal`, or resolves to private addresses). Sends the API token in cleartext; only for networks you fully control, e.g. internal networks numbered from public IP space | `false` |
 | `--hostname` | `PULSE_HOSTNAME` | Override hostname | *(OS hostname)* |
-| `--agent-id` | `PULSE_AGENT_ID` | Unique agent identifier | *(machine-id)* |
+| `--agent-id` | `PULSE_AGENT_ID` | Unique agent identifier | *(saved ID, then machine ID)* |
 | `--report-ip` | `PULSE_REPORT_IP` | Override reported IP (multi-NIC) | *(auto)* |
 | `--disable-ceph` | `PULSE_DISABLE_CEPH` | Disable local Ceph status polling | `false` |
 | `--tag` | `PULSE_TAGS` | Apply tags (repeatable or CSV) | *(none)* |
@@ -834,20 +834,53 @@ file and re-run the installer to pick up the rotating configuration.
   identity evidence. The endpoint reports repair handoffs but does not run them.
 
 ### Duplicate Agents
-If cloned VMs appear as the same agent:
-```bash
-sudo rm /etc/machine-id && sudo systemd-machine-id-setup
-```
 
-Or set a unique agent ID:
-```bash
---agent-id my-unique-agent-id
-```
+Cloning an installed agent can copy its saved identity and credentials. Different
+hostnames, MAC addresses or IPs do not make those agents distinct in Pulse.
+Identity selection is:
 
-The displayed or reported IP is not the durable agent identity. Pulse normally
-uses the machine ID (or an explicit `--agent-id`), so cloned systems must have
-unique machine and agent IDs even when their hostnames, MAC addresses, and IPs
-differ.
+1. An explicit `--agent-id` argument (which overrides `PULSE_AGENT_ID`).
+2. `PULSE_AGENT_ID` when no explicit argument is supplied.
+3. The saved `agent-id` file, when no override is set.
+4. The machine-derived ID, with hostname as a fallback, when no saved ID exists.
+
+First compare the affected hosts in **Agent Doctor**. Inspect the identity file
+locally: the default Linux path is `/var/lib/pulse-agent/agent-id`; `--state-dir`
+changes that directory and `--agent-id-file` or `PULSE_AGENT_ID_FILE` can select
+another file. Do not upload `connection.env`, token files or whole service and
+environment dumps; older installations can contain credentials there.
+
+**Do not delete `/etc/machine-id`, agent state or Pulse history to repair a
+duplicate.** A saved agent ID survives an OS machine-ID change, and changing the
+OS identity affects more than Pulse. Prepare OS identities when provisioning a
+clone, using the distribution's instructions, not during live Pulse diagnosis.
+
+For a confirmed duplicate, keep the original host unchanged and choose one
+stable, unique override for the clone. Persist it in the configuration that
+actually starts that agent, rather than launching a second agent process:
+
+- **Systemd, without an existing `--agent-id` argument**: use
+  `sudo systemctl edit pulse-agent.service` and add the following drop-in,
+  replacing the example ID with the clone's unique ID:
+
+  ```ini
+  [Service]
+  Environment="PULSE_AGENT_ID=vm-clone-02"
+  ```
+
+  Then restart only that agent with `sudo systemctl restart pulse-agent.service`.
+  If its managed launch command already supplies `--agent-id`, change that
+  existing override instead: the argument wins over the environment variable.
+- **Docker Compose**: set `PULSE_AGENT_ID` for the clone's agent service, keeping
+  its existing volumes, collector options and token-file configuration. An
+  existing `--agent-id` in `command` takes precedence; update it instead if set.
+
+Keep the override across restarts and upgrades. After restart, verify that both
+hosts report fresh, distinct IDs in Agent Doctor and that each row's metrics
+belong to the correct host. Changing an ID does not split or recover historical
+data already combined under the old ID. If authentication or reporting fails,
+keep the bounded, redacted error for diagnosis; do not delete tokens, re-enrol
+blindly or remove the original host's record.
 
 ### Permission Denied (Docker)
 Ensure the agent can access the Docker socket:

frontend-modern/src/utils/__tests__/docsLinks.test.ts

@@ -152,6 +152,37 @@ describe('docsLinks', () => {
     expect(help).toContain('If installation never started Pulse');
   });
 
+  it('keeps clone identity recovery separate from destructive OS or credential resets', () => {
+    const troubleshooting = readFileSync(
+      path.join(repoRoot, 'docs', 'TROUBLESHOOTING.md'),
+      'utf8',
+    );
+    const agentGuide = readFileSync(
+      path.join(repoRoot, 'docs', 'UNIFIED_AGENT.md'),
+      'utf8',
+    );
+    const shortGuide = troubleshooting
+      .split('#### Docker hosts appearing/disappearing')[1]
+      .split('\n### ')[0];
+    const cloneGuide = agentGuide.split('### Duplicate Agents')[1].split('\n### ')[0];
+
+    expect(shortGuide).toContain('UNIFIED_AGENT.md#duplicate-agents');
+    for (const section of [shortGuide, cloneGuide]) {
+      expect(section).toContain('saved');
+      expect(section).toContain('Agent Doctor');
+      expect(section).toContain('original host unchanged');
+      expect(section).not.toContain('systemd-machine-id-setup');
+      expect(section).not.toMatch(/(?:sudo\s+)?rm\s+(?:-\S+\s+)*\/etc\/machine-id/);
+    }
+    expect(cloneGuide).toContain('argument wins over the environment variable');
+    expect(cloneGuide).toContain('PULSE_AGENT_ID_FILE');
+    expect(cloneGuide).toContain('Environment="PULSE_AGENT_ID=vm-clone-02"');
+    expect(cloneGuide).toContain('fresh, distinct IDs');
+    expect(cloneGuide).toContain('does not split or recover historical');
+    expect(cloneGuide).toContain('Do not upload `connection.env`, token files');
+    expect(cloneGuide).toContain('do not delete tokens, re-enrol');
+  });
+
   it('ships a bounded maintenance example separately from incident snoozing', () => {
     const apiReference = readFileSync(path.join(repoRoot, 'docs', 'API.md'), 'utf8');
     const maintenance = apiReference

scripts/check-clone-identity-guidance.cjs

@@ -0,0 +1,107 @@
+// Real shipped documents, production Docs renderer/router; no live host changes.
+// Run from the assigned workspace root:
+//   pulse-worker-browser scripts/check-clone-identity-guidance.cjs
+const fs = require('node:fs');
+const path = require('node:path');
+const crypto = require('node:crypto');
+const { chromium } = require('playwright');
+
+const root = process.cwd();
+const frontend = path.join(root, 'frontend-modern');
+const output = path.join(root, 'tmp', 'clone-identity-guidance');
+const port = 5198;
+const observations = [];
+const errors = [];
+
+function check(condition, message) {
+  observations.push({ passed: condition, message });
+  if (!condition) throw new Error(message);
+}
+
+async function sectionText(page, id) {
+  return page.locator(`#${id}`).evaluate((heading) => {
+    const parts = [heading.textContent];
+    for (let node = heading.nextElementSibling; node; node = node.nextElementSibling) {
+      if (/^H[1-3]$/.test(node.tagName)) break;
+      parts.push(node.textContent);
+    }
+    return parts.join('\n').replace(/\s+/g, ' ');
+  });
+}
+
+(async () => {
+  fs.mkdirSync(output, { recursive: true });
+  process.chdir(frontend);
+  const { createServer } = await import(
+    path.join(frontend, 'node_modules', 'vite', 'dist', 'node', 'index.js')
+  );
+  const server = await createServer({
+    root: frontend,
+    configFile: path.join(frontend, 'vite.config.ts'),
+    server: { host: '127.0.0.1', port, strictPort: true },
+  });
+  let browser;
+  let result;
+  try {
+    await server.listen();
+    browser = await chromium.launch({
+      headless: true, channel: 'chromium', args: ['--no-sandbox'],
+    });
+    const page = await browser.newPage();
+    page.on('pageerror', (error) => errors.push(error.message));
+    page.on('response', (response) => {
+      if (response.status() >= 400) errors.push(`${response.status()} ${response.url()}`);
+    });
+    // Authentication probing only; both Markdown documents are real assets.
+    await page.route(new RegExp(`^http://127\\.0\\.0\\.1:${port}/api/`), (route) =>
+      route.fulfill({
+        status: 200, contentType: 'application/json',
+        body: JSON.stringify({ hasAuthentication: false }),
+      }),
+    );
+
+    for (const [size, width, height] of [['desktop', 1440, 900], ['phone', 390, 844]]) {
+      await page.setViewportSize({ width, height });
+      await page.goto(`http://127.0.0.1:${port}/browser-tests/docs-fragment-navigation.html?scenario=troubleshooting`);
+      await page.waitForFunction(() => document.activeElement?.id === 'recovery-mode');
+      await page.locator('#docker-hosts-appearingdisappearing').scrollIntoViewIfNeeded();
+      const shortGuide = await sectionText(page, 'docker-hosts-appearingdisappearing');
+      check(shortGuide.includes('saved Pulse agent ID'), `${size}: saved clone identity is identified`);
+      check(shortGuide.includes('Do not delete the OS machine ID'), `${size}: no destructive OS reset is prescribed`);
+      const link = page.getByRole('link', { name: 'Clone identity recovery', exact: true });
+      check((await link.getAttribute('href')) === '/docs/UNIFIED_AGENT#duplicate-agents', `${size}: link targets the shipped agent guide`);
+      check(!(await page.evaluate(() => document.documentElement.scrollWidth > document.documentElement.clientWidth)), `${size}: troubleshooting does not overflow`);
+      await page.screenshot({ path: path.join(output, `${size}-troubleshooting.png`) });
+
+      await link.focus();
+      await page.keyboard.press('Enter');
+      await page.waitForFunction(() => document.activeElement?.id === 'duplicate-agents');
+      const cloneGuide = await sectionText(page, 'duplicate-agents');
+      check(cloneGuide.includes('argument wins over the environment variable'), `${size}: argument precedence is explicit`);
+      check(cloneGuide.includes('PULSE_AGENT_ID_FILE'), `${size}: custom identity files are covered`);
+      check(cloneGuide.includes('Environment="PULSE_AGENT_ID=vm-clone-02"'), `${size}: persistent systemd configuration is rendered`);
+      check(cloneGuide.includes('fresh, distinct IDs') && cloneGuide.includes('does not split or recover historical'), `${size}: verification and history limitations are rendered`);
+      check(cloneGuide.includes('Do not upload connection.env, token files'), `${size}: credential-bearing evidence is excluded`);
+      check(!(await page.evaluate(() => document.documentElement.scrollWidth > document.documentElement.clientWidth)), `${size}: agent guide does not overflow`);
+      await page.locator('article pre').filter({ hasText: 'PULSE_AGENT_ID=vm-clone-02' }).scrollIntoViewIfNeeded();
+      await page.screenshot({ path: path.join(output, `${size}-agent-configuration.png`) });
+    }
+    check(errors.length === 0, 'no page errors or failed HTTP responses');
+    result = {
+      result: 'passed', observations, errors, chromium: browser.version(),
+      playwright: require('playwright/package.json').version,
+      sourceAssets: ['TROUBLESHOOTING.md', 'UNIFIED_AGENT.md'].map((name) => ({
+        path: `frontend-modern/public/docs/${name}`,
+        sha256: crypto.createHash('sha256').update(fs.readFileSync(path.join(frontend, 'public', 'docs', name))).digest('hex'),
+      })),
+    };
+  } catch (error) {
+    result = { result: 'failed', observations, errors, error: error.message };
+  } finally {
+    if (browser) await browser.close();
+    await server.close();
+  }
+  fs.writeFileSync(path.join(output, 'result.json'), JSON.stringify(result, null, 2));
+  console.log(JSON.stringify(result, null, 2));
+  process.exitCode = result.result === 'passed' ? 0 : 1;
+})();