pulse/docs/UNIFIED_AGENT.md

# Pulse Unified Agent

The unified agent (`pulse-agent`) is the single host-installed Pulse infrastructure agent binary. It combines host, Docker/Podman, Kubernetes, Proxmox-local, and other enabled node-local telemetry modules into one deployment and one service.
Install it on standalone hosts and on machines where Pulse needs full node-local telemetry.
For API-backed platforms, start with the platform connection first and add the agent only where local telemetry is needed.

For Proxmox, install the agent only where you need telemetry that the Proxmox
API cannot provide, such as host SMART and temperature data, local
ZFS/Ceph/mdadm detail, arbitrary mount reads, or inside-guest runtime
visibility. Docker containers inside LXCs can be reported by a Proxmox host
agent when the server has explicitly enabled the privacy-bounded LXC inventory
mode; Docker/Podman inside VMs still needs a guest-local agent or another
explicit guest reporting path.
Basic Proxmox inventory and utilization can use a read-only or narrowly scoped
Proxmox API token instead. Settings uses that API inventory path as the
default for new PVE/PBS setup. See [Agent Security](AGENT_SECURITY.md) for
the root-service trade-off, restricted-user expectations, and supply-chain
verification guidance.

> Note: For agent-based temperature monitoring, use `pulse-agent --enable-proxmox` or SSH-based collection. The legacy sensor proxy has been removed. See `docs/TEMPERATURE_MONITORING.md`.

## Quick Start

Generate an installation command in the UI:
**Settings → Infrastructure → Install on a host**

Choose a target profile in that screen when you want explicit install flags for Docker, Kubernetes, Proxmox VE, or Proxmox Backup Server.

The same generated command is also the supported v5-to-v6 agent upgrade path.
Run it on the host that already has the v5 `pulse-agent` service to replace the
binary and service configuration in place; do not uninstall the old service
first unless you are intentionally removing that host from Pulse.

An installed agent is configured for one Pulse URL and one token. Do not point
one running service at both a v5 server and a v6 server. After the upgrade,
check the relevant platform page or **Machines** view once the agent has
reported, and confirm the host-local version with `pulse-agent --version` if
the UI has not received a fresh report yet.

### Linux (systemd)
```bash
curl -fsSL http://<pulse-ip>:7655/install.sh | \
  bash -s -- --url http://<pulse-ip>:7655 --token <api-token>
```

### macOS
```bash
curl -fsSL http://<pulse-ip>:7655/install.sh | \
  bash -s -- --url http://<pulse-ip>:7655 --token <api-token>
```

### Windows (PowerShell, run as Administrator)
```powershell
irm http://<pulse-ip>:7655/install.ps1 | iex
```

With environment variables:
```powershell
$env:PULSE_URL="http://<pulse-ip>:7655"
$env:PULSE_TOKEN="<api-token>"
irm http://<pulse-ip>:7655/install.ps1 | iex
```

### Synology NAS
```bash
curl -fsSL http://<pulse-ip>:7655/install.sh | \
  bash -s -- --url http://<pulse-ip>:7655 --token <api-token>
```

### TrueNAS SCALE/CORE
TrueNAS SCALE and TrueNAS CORE are both supported. The installer auto-detects the platform and configures the appropriate service manager (systemd for SCALE, rc.d for CORE).
```bash
curl -fsSL http://<pulse-ip>:7655/install.sh | \
  bash -s -- --url http://<pulse-ip>:7655 --token <api-token>
```

## Features

- **Host Metrics**: CPU, memory, disk, network I/O, temperatures
- **Docker Monitoring**: Container metrics, health checks, Swarm support (when enabled)
- **Kubernetes Monitoring**: Cluster, node, pod, and deployment health (when enabled)
- **Auto-Update**: Automatically updates when a new version is released
- **Multi-Platform**: Linux, macOS, Windows support

## Configuration

| Flag | Env Var | Description | Default |
|------|---------|-------------|---------|
| `--url` | `PULSE_URL` | Pulse server URL | `http://localhost:7655` |
| `--token` | `PULSE_TOKEN` | API token | *(required)* |
| `--token-file` | - | Read API token from file | *(unset)* |
| `--interval` | `PULSE_INTERVAL` | Reporting interval | `30s` |
| `--enable-host` | `PULSE_ENABLE_HOST` | Enable host metrics | `true` |
| `--enable-docker` | `PULSE_ENABLE_DOCKER` | Enable Docker / Podman metrics | `false` (auto-detect if not configured) |
| `--docker-runtime` | `PULSE_DOCKER_RUNTIME` | Force Docker / Podman runtime: `auto`, `docker`, or `podman` | `auto` |
| `--enable-kubernetes` | `PULSE_ENABLE_KUBERNETES` | Enable Kubernetes metrics | `false` (installer auto-detect if not configured) |
| `--enable-proxmox` | `PULSE_ENABLE_PROXMOX` | Enable Proxmox integration | `false` |
| `--proxmox-type` | `PULSE_PROXMOX_TYPE` | Proxmox type: `pve` or `pbs` | *(auto-detect)* |
| `--enable-commands` | `PULSE_ENABLE_COMMANDS` | Enable AI command execution (disabled by default) | `false` |
| `--disable-commands` | `PULSE_DISABLE_COMMANDS` | **Deprecated** (commands are disabled by default) | - |
| `--disk-exclude` | `PULSE_DISK_EXCLUDE` | Mount point patterns to exclude from disk monitoring (repeatable or CSV) | *(none)* |
| `--kubeconfig` | `PULSE_KUBECONFIG` | Kubeconfig path (optional) | *(auto)* |
| `--kube-context` | `PULSE_KUBE_CONTEXT` | Kubeconfig context (optional) | *(auto)* |
| `--kube-include-namespace` | `PULSE_KUBE_INCLUDE_NAMESPACES` | Limit namespaces (repeatable or CSV, wildcards supported) | *(all)* |
| `--kube-exclude-namespace` | `PULSE_KUBE_EXCLUDE_NAMESPACES` | Exclude namespaces (repeatable or CSV, wildcards supported) | *(none)* |
| `--kube-include-all-pods` | `PULSE_KUBE_INCLUDE_ALL_PODS` | Include all non-succeeded pods | `false` |
| `--kube-include-all-deployments` | `PULSE_KUBE_INCLUDE_ALL_DEPLOYMENTS` | Include all deployments, not just problems | `false` |
| `--kube-max-pods` | `PULSE_KUBE_MAX_PODS` | Max pods per report | `200` |
| `--disable-auto-update` | `PULSE_DISABLE_AUTO_UPDATE` | Disable auto-updates | `false` |
| `--disable-docker-update-checks` | `PULSE_DISABLE_DOCKER_UPDATE_CHECKS` | Disable Docker image update detection | `false` |
| `--insecure` | `PULSE_INSECURE_SKIP_VERIFY` | Skip TLS verification | `false` |
| `--allow-plaintext-http` | `PULSE_AGENT_ALLOW_PLAINTEXT_HTTP` | Allow plain HTTP to a Pulse server that does not look local (private IP, single-label, `.local`/`.lan`/`.home`/`.home.arpa`/`.internal`, or resolves to private addresses). Sends the API token in cleartext; only for networks you fully control, e.g. internal networks numbered from public IP space | `false` |
| `--hostname` | `PULSE_HOSTNAME` | Override hostname | *(OS hostname)* |
| `--agent-id` | `PULSE_AGENT_ID` | Unique agent identifier | *(machine-id)* |
| `--report-ip` | `PULSE_REPORT_IP` | Override reported IP (multi-NIC) | *(auto)* |
| `--disable-ceph` | `PULSE_DISABLE_CEPH` | Disable local Ceph status polling | `false` |
| `--tag` | `PULSE_TAGS` | Apply tags (repeatable or CSV) | *(none)* |
| `--log-level` | `LOG_LEVEL` | Log verbosity (`debug`, `info`, `warn`, `error`) | `info` |
| `--health-addr` | `PULSE_HEALTH_ADDR` | Health/metrics server address | `127.0.0.1:9191` |

Use `--health-addr :9191` only when another host must scrape the
health/metrics endpoint over the network. Use `--health-addr ""` or
`PULSE_HEALTH_ADDR=off` to disable that listener.

**Token resolution order**: `--token` → `--token-file` → `PULSE_TOKEN` → `/var/lib/pulse-agent/token`.

### Advanced Flags

- `--version`: Print the agent version and exit.
- `--self-test`: Perform a self-test and exit (used during auto-update).

## Auto-Detection

Auto-detection behavior:

- **Host metrics**: Enabled by default.
- **Docker/Podman**: Enabled automatically by the agent if Docker/Podman is detected and `PULSE_ENABLE_DOCKER` was not explicitly set. A local `--enable-docker=false` or `PULSE_ENABLE_DOCKER=false` is a hard opt-out and is not re-enabled by auto-detection or remote profile config.
- **Kubernetes**: Enabled automatically by the installer when a kubeconfig is detected and `PULSE_ENABLE_KUBERNETES` was not explicitly set.
- **Proxmox**: Enabled automatically by the installer when Proxmox is detected. Type auto-detects `pve` vs `pbs` if not specified.

To disable auto-detection, explicitly set the relevant flags or env vars, for example:

- `--enable-docker=false` or `PULSE_ENABLE_DOCKER=false`
- `--enable-kubernetes=false` or `PULSE_ENABLE_KUBERNETES=false`
- `--enable-proxmox=false` or `PULSE_ENABLE_PROXMOX=false`

### Inside-Guest Runtime Boundaries

Docker/Podman inside a VM or LXC is monitored from inside that guest. Install the
Unified Agent in the guest when you want full Docker host, container, service,
and task inventory on the Docker page.

Pulse does not use a Proxmox node agent to look inside LXCs by default. The
optional Proxmox-side LXC Docker hint is off unless the Pulse server is started
with `PULSE_ENABLE_PROXMOX_GUEST_DOCKER_DETECTION=true`. That hint uses
`pct exec` only to check whether `/var/run/docker.sock` exists in a running LXC;
it does not enumerate containers, images, environment variables, files, or
processes. The stronger Proxmox-side LXC Docker inventory path is separately
disabled unless the server is started with
`PULSE_ENABLE_PROXMOX_GUEST_DOCKER_INVENTORY=true`. Use either path only when
operators are comfortable with Proxmox-side guest probing.

## Installation Options

### Simple Install (host + Docker auto-detect)
```bash
curl -fsSL http://<pulse-ip>:7655/install.sh | \
  bash -s -- --url http://<pulse-ip>:7655 --token <token>
```

### Proxmox VE Node (explicit profile)
```bash
curl -fsSL http://<pulse-ip>:7655/install.sh | \
  bash -s -- --url http://<pulse-ip>:7655 --token <token> --enable-proxmox --proxmox-type pve
```

### Proxmox Backup Server Node (explicit profile)
```bash
curl -fsSL http://<pulse-ip>:7655/install.sh | \
  bash -s -- --url http://<pulse-ip>:7655 --token <token> --enable-proxmox --proxmox-type pbs
```

### Force Enable Docker (if auto-detection fails)
```bash
curl -fsSL http://<pulse-ip>:7655/install.sh | \
  bash -s -- --url http://<pulse-ip>:7655 --token <token> --enable-docker
```

### Disable Docker (even if detected)
```bash
curl -fsSL http://<pulse-ip>:7655/install.sh | \
  bash -s -- --url http://<pulse-ip>:7655 --token <token> --enable-docker=false
```

### Host + Kubernetes Monitoring
```bash
curl -fsSL http://<pulse-ip>:7655/install.sh | \
  bash -s -- --url http://<pulse-ip>:7655 --token <token> --enable-kubernetes
```

### Docker Monitoring Only
```bash
curl -fsSL http://<pulse-ip>:7655/install.sh | \
  bash -s -- --url http://<pulse-ip>:7655 --token <token> --enable-host=false --enable-docker
```

### Exclude Specific Disks from Monitoring
```bash
# Exclude specific mount points
pulse-agent --disk-exclude /mnt/backup --disk-exclude /media/external

# Exclude using patterns (prefix match)
pulse-agent --disk-exclude '/mnt/pbs*'  # Matches /mnt/pbs-data, /mnt/pbs-backup, etc.

# Exclude using patterns (contains match)
pulse-agent --disk-exclude '*pbs*'  # Matches any path containing 'pbs'

# Via environment variable (comma-separated)
PULSE_DISK_EXCLUDE=/mnt/backup,*pbs*,/media/external
```

**Pattern types:**
- Exact: `/mnt/backup` - matches only that exact path
- Prefix: `/mnt/ext*` - matches paths starting with `/mnt/ext`
- Contains: `*pbs*` - matches paths containing `pbs`

## S.M.A.R.T. Disk Health

The agent can report S.M.A.R.T. disk temperatures, health status, identity, and normalized health counters when running in Agent mode. This requires:

1. **smartmontools** installed on the host:
   ```bash
   # Debian/Ubuntu
   apt install smartmontools

   # RHEL/CentOS
   yum install smartmontools

   # Alpine
   apk add smartmontools
   ```

2. The agent must have permission to run `smartctl` (typically requires root)

**Notes:**
- Disks in standby mode are reported as such (no temperature) to avoid waking them
- S.M.A.R.T. data is collected alongside other host metrics and can enrich the Physical Disks view with temperature, stable disk identity, power-on hours, SSD life, pending sectors, media errors, and related counters
- If `smartctl` is not available, S.M.A.R.T. monitoring is silently skipped
- **Disk exclusions** (`--disk-exclude` / `PULSE_DISK_EXCLUDE`) also apply to S.M.A.R.T. monitoring.
  Use patterns like `sda`, `/dev/sdb`, `nvme*`, or `*cache*` to exclude specific block devices.

## Auto-Update

The unified agent automatically checks for updates every hour. When a new version is available:

1. Agent downloads the new binary from the Pulse server
2. Verifies the checksum
3. Verifies the release signature when trusted update keys are embedded
4. Runs the downloaded binary with `--self-test`
5. Replaces itself atomically (with backup)
6. Restarts with the same configuration

When an already-installed v5 `pulse-agent` moves to v6, the first automatic hop
is performed by the v5 updater. That hop verifies TLS by default, the SHA-256
checksum, executable magic, size limits, and atomic replacement, but the newer
v6 signature and `--self-test` checks apply only after the agent has landed on
v6. Use HTTPS or a trusted local network for v5-to-v6 automatic migration. For
high-assurance environments, reinstall the v6 `pulse-agent` through the signed
installer path instead of relying on a plain-HTTP first hop.

To disable auto-updates:
```bash
# During installation
curl -fsSL http://<pulse-ip>:7655/install.sh | \
  bash -s -- --url http://<pulse-ip>:7655 --token <token> --disable-auto-update

# Or set environment variable
PULSE_DISABLE_AUTO_UPDATE=true
```

## Remote Configuration (Agent Profiles, Pro/legacy Pro+/Cloud)

Pro, legacy Pro+, and Cloud can push centralized settings to agents via Agent Profiles.

Behavior:
- The agent fetches remote config on startup from `/api/agents/agent/{agent_id}/config`.
- Profile settings override local flags/env for supported keys.
- Profile changes take effect on the next agent restart.
- Command execution (`commandsEnabled`) is controlled per agent from the Infrastructure agent controls and can change live.
- Remote config responses can be signed with `PULSE_AGENT_CONFIG_SIGNING_KEY` (base64 Ed25519 private key).
- To require signed payloads, set `PULSE_AGENT_CONFIG_SIGNATURE_REQUIRED=true` on Pulse and agents.
- If you use a custom signing key, set `PULSE_AGENT_CONFIG_PUBLIC_KEYS` on agents to trust the matching public key.

See [Centralized Agent Management](CENTRALIZED_MANAGEMENT.md) for supported keys and profile setup.

## Uninstall

```bash
curl -fsSL http://<pulse-ip>:7655/install.sh | bash -s -- --uninstall
```

This removes:
- The agent binary
- The systemd/launchd service

## Migration Notes

Use the unified installer (`install.sh`) for all new and existing deployments.

## Health Checks & Metrics

The agent exposes HTTP endpoints for health checks and Prometheus metrics on port 9191 by default.

### Endpoints

| Endpoint | Description |
|----------|-------------|
| `/healthz` | Liveness probe - returns 200 if agent is running |
| `/readyz` | Readiness probe - returns 200 when agents are initialized |
| `/metrics` | Prometheus metrics |

### Prometheus Metrics

| Metric | Type | Description |
|--------|------|-------------|
| `pulse_agent_info` | Gauge | Agent info with version, host_enabled, docker_enabled labels |
| `pulse_agent_up` | Gauge | 1 when running, 0 when shutting down |

### Kubernetes Probes

```yaml
livenessProbe:
  httpGet:
    path: /healthz
    port: 9191
  initialDelaySeconds: 5
  periodSeconds: 10

readinessProbe:
  httpGet:
    path: /readyz
    port: 9191
  initialDelaySeconds: 5
  periodSeconds: 5
```

### Disable Health Server

Set `--health-addr=""` or `PULSE_HEALTH_ADDR=off` to disable the health/metrics server. Set `--health-addr :9191` when network Prometheus scraping is intentional.

## Troubleshooting

### Agent Not Updating
- Check logs: `journalctl -u pulse-agent -f`
- Verify network connectivity to Pulse server
- Ensure auto-update is not disabled

### Duplicate Agents
If cloned VMs appear as the same agent:
```bash
sudo rm /etc/machine-id && sudo systemd-machine-id-setup
```

Or set a unique agent ID:
```bash
--agent-id my-unique-agent-id
```

### Permission Denied (Docker)
Ensure the agent can access the Docker socket:
```bash
sudo usermod -aG docker $USER
```

### Check Status
```bash
# Linux
systemctl status pulse-agent

# macOS
launchctl list | grep pulse
```

### Docker Swarm Not Detected

If your Docker Swarm cluster isn't being detected:

1. **Check runtime detection**: Pulse disables Swarm for Podman. Look for "Podman runtime detected" in logs:
   ```bash
   journalctl -u pulse-agent | grep -i podman
   ```

2. **Force Docker runtime**: If auto-detection is incorrect:
   ```bash
   --docker-runtime docker
   # Or set environment variable
   PULSE_DOCKER_RUNTIME=docker
   ```

3. **Check Docker info**: Verify Swarm is active on the host:
   ```bash
   docker info | grep -i swarm
   # Should show "Swarm: active"
   ```

4. **Check socket permissions**: The agent needs access to the Docker socket:
   ```bash
   ls -la /var/run/docker.sock
   ```

5. **Enable debug logging**: For more detail:
   ```bash
   LOG_LEVEL=debug journalctl -u pulse-agent -f
   ```

### PVE Backups Not Showing (Recovery)

If local PVE backups aren't appearing in Pulse after setting up via `--enable-proxmox`:

1. **Check permissions**: The API token needs `PVEDatastoreAdmin` on `/storage`:
   ```bash
   pveum aclmod /storage -user pulse-monitor@pve -role PVEDatastoreAdmin
   pveum aclmod /storage -token 'pulse-monitor@pve!<token-name>' -role PVEDatastoreAdmin
   ```
   Replace `pulse-monitor@pve!<token-name>` with the full token ID shown in Pulse.
   Privilege-separated PVE tokens need the storage ACL on the token as well as the service user.

2. **Re-run setup**: Delete the node in Pulse Settings and re-run the agent with `--enable-proxmox`. Recent versions grant this permission automatically.

3. **Check state file**: If re-running doesn't trigger setup, remove the state file:
   ```bash
   rm /var/lib/pulse-agent/proxmox-pve-registered
   ```
   Then restart the agent.