pulse/docs/MSP.md

# Pulse for MSPs (Provider Operations Guide)

This guide covers running Pulse as a managed service provider: one central
deployment monitoring multiple client estates, with per-client isolation,
alert routing, and reporting. It assumes you have read
[DEPLOYMENT_MODELS.md](DEPLOYMENT_MODELS.md) for the deployment-model overview.

## Deployment models

**Provider-hosted MSP (canonical).** A control plane runs one isolated Pulse
runtime per client workspace. Alerts, webhook destinations, branded report
settings, users, audit history, and metrics stay inside the client runtime;
duplicate hostnames across clients never collide because they never share a
runtime namespace.

The canonical install is the deploy bundle at
[`deploy/provider-msp/`](../deploy/provider-msp/): a Docker Compose stack
(Traefik ingress with wildcard TLS, a hardened Docker socket proxy, and the
control plane), a guided `setup.sh` for fresh hosts, `upgrade.sh` for
backup-gated upgrades, and `run-install-proof.sh` for an end-to-end fresh
install proof. `.env.example` in that directory doubles as the operator
runbook. Start there rather than wiring containers by hand; among other
things the compose stack provides the `pulse.provider-msp.role=traefik` and
`pulse.provider-msp.role=control-plane` container labels that client
workspace provisioning requires for isolated tenant networking, and it
terminates TLS — the management portal sets a `__Host-` (HTTPS-only) session
cookie, so the portal does not work over plain HTTP.

Day-2 operations run through the `pulse-control-plane` binary (via
`docker compose run --rm control-plane …` in the bundle):

```bash
pulse-control-plane provider-msp bootstrap --account-name "Your MSP" --owner-email you@example.com
pulse-control-plane provider-msp status
pulse-control-plane provider-msp backup
pulse-control-plane provider-msp recover   # restore workspaces from backup or disk
pulse-control-plane provider-msp preflight # pre-install environment checks
```

### Portal sign-in and sessions

The management portal signs you in with one-time links, not passwords. With
no email provider configured (the bundle default), the portal cannot send
those links itself; the sign-in page says so and points at the host command
that prints one:

```bash
# Owner sign-in link (also safe to re-run any time; it never duplicates the account)
docker compose run --rm control-plane provider-msp bootstrap \
  --account-name "Your MSP" --owner-email you@example.com

# Sign-in link for an invited teammate
docker compose run --rm control-plane provider-msp portal-link --email teammate@example.com
```

Teammates are invited from the portal Access tab; without an email provider
the invitation email is not sent, so print their first sign-in link with
`portal-link` after inviting them. To let the portal send sign-in links and
invitations itself, set `RESEND_API_KEY` (plus `PULSE_EMAIL_FROM` and
`PULSE_EMAIL_REPLY_TO`) in `.env` and restart the control plane.

Portal sessions last 7 days on provider-hosted control planes; override with
`CP_SESSION_TTL` (Go duration, e.g. `12h`, `168h`).

Each client runtime is a normal Pulse instance, so it connects to that
client's infrastructure with the standard methods: agents push over HTTPS for
hosts, and Proxmox/PBS polling reaches across networks through your existing
VPN or tunnel to the client site.

**Shared-process organizations (alternative).** One Pulse process serves
multiple organizations with isolated data directories, org-bound tokens, and
per-org alert/webhook/notification state. This is documented in
[MULTI_TENANT.md](MULTI_TENANT.md) and gated by `PULSE_MULTI_TENANT_ENABLED=true`
plus a licence carrying the `multi_tenant` capability. It is designed for one
owner separating internal estates (sites, departments, environments); the
isolated-runtime model above is the canonical choice for separate customer
businesses.

## Network topology and ingress isolation

Run the management UI and agent check-in on separate, separately firewalled
ports. See [Split-Port Agent Ingest](CONFIGURATION.md#split-port-agent-ingest-network-isolation)
for the full reference.

```bash
FRONTEND_PORT=7655              # management UI + API: private network / VPN only
PULSE_AGENT_INGEST_PORT=7656    # agent reports + command/control: reachable from client sites
PULSE_AGENT_CONNECT_URL=https://agents.example.com:7656
```

Firewall baseline:

| Surface | Port | Reachable from |
|---------|------|----------------|
| Management UI + API | `FRONTEND_PORT` (7655) | Provider staff network / VPN only |
| Agent control plane | `PULSE_AGENT_INGEST_PORT` (7656) | Client sites (or client VPN tunnels) |
| Prometheus metrics | 9091 | Provider monitoring network only |

The dedicated agent port serves only the report/config, command WebSocket,
version, and bootstrap routes documented in
[Split-Port Agent Ingest](CONFIGURATION.md#split-port-agent-ingest-network-isolation);
login and management APIs return `404`. Report and command access use separate
least-privilege scopes (`agent:report` and `agent:exec`) on the same
host-bound enrollment token. Token scope, immutable command-session binding,
and port isolation are independent layers.

If agents reach the central server over per-client VPN tunnels instead of the
public internet, the same split still applies: expose only the agent port into
the tunnels and keep the management port out of them.

### Validation checklist (run after setup, repeat after network changes)

Use your actual HTTPS addresses and keep certificate verification enabled.
For a private CA, add `--cacert /path/to/trusted-ca.pem` using a CA you have
verified independently; do not bypass a certificate error with `--insecure`.
The examples make bounded **GET** requests, discard response bodies and print
only the HTTP status. They do not send notifications or change settings.

For the authenticated checks, prepare two separate private header files with
[the API token file procedure](API.md#api-token-recommended), substituting
`agent-header` and `org-a-header` for `api-header`. Keep the directory mode
`700` and files mode `600`, owned by the account running curl. Enter the token
only in your local editor, never in a shell command, URL or report. Each file
contains one `X-API-Token` header: the existing `agent:report` token in
`agent-header`, and an existing org-A token with `monitoring:read` in
`org-a-header`. Do not broaden scopes or create a privileged token just to run
these checks. Keep the files outside repositories, backups shared for support
and diagnostic exports; remove them locally when no longer needed.

Keep `--disable` first so a local curl configuration cannot enable tracing,
extra credentials or redirects. Do not add verbose/trace output or follow
redirects with these tokens. For the HTTP-status checks below, a TLS, DNS,
timeout or connection error is a **failed check**, not proof of isolation:
require curl exit **0** as well as the expected HTTP status. `000` is not an HTTP result. Do not use `--fail` for these
checks because the expected rejections are HTTP errors.

1. **Agent port excludes management surfaces.** Both must return `404`:

   ```bash
   curl --disable --silent --show-error --proto '=https' --connect-timeout 5 --max-time 10 \
     --output /dev/null --write-out '%{http_code}\n' https://agents.example.com:7656/ # 404
   curl --disable --silent --show-error --proto '=https' --connect-timeout 5 --max-time 10 \
     --output /dev/null --write-out '%{http_code}\n' https://agents.example.com:7656/api/login # 404
   ```

   When commands are enabled, also verify that `/api/agent/ws` reaches Pulse
   through the proxy. Agent Doctor reports the command channel as disconnected
   if telemetry is current but WebSocket admission is absent.

2. **Management port is not reachable from a client site.** From a client
   network (or through a client tunnel), a connection to `FRONTEND_PORT` must
   time out or be refused by your firewall — not answer. That observation alone
   does not distinguish a firewall from a DNS, TLS or server failure; confirm
   the same management endpoint works from the authorised staff network.

3. **Agent tokens cannot manage.** On an authentication-enabled instance, a
   request to a management endpoint with the agent token must be rejected:

   ```bash
   curl --disable --silent --show-error --proto '=https' --connect-timeout 5 --max-time 10 \
     --output /dev/null --write-out '%{http_code}\n' \
     --header "@$HOME/.config/pulse/agent-header" \
     https://pulse.internal:7655/api/notifications/webhooks # 401/403
   ```

4. **Cross-tenant isolation (shared-process mode only).** A valid token bound
   to org-A with `monitoring:read` must get `403` when targeting another
   organization AND when targeting the default org (a leaked client-site
   token must not read the provider's own estate):

   ```bash
   curl --disable --silent --show-error --proto '=https' --connect-timeout 5 --max-time 10 \
     --output /dev/null --write-out '%{http_code}\n' \
     --header "@$HOME/.config/pulse/org-a-header" --header 'X-Pulse-Org-ID: org-b' \
     https://pulse.internal:7655/api/alerts/active # 403
   curl --disable --silent --show-error --proto '=https' --connect-timeout 5 --max-time 10 \
     --output /dev/null --write-out '%{http_code}\n' \
     --header "@$HOME/.config/pulse/org-a-header" --header 'X-Pulse-Org-ID: default' \
     https://pulse.internal:7655/api/alerts/active # 403
   ```

   An expired or otherwise invalid token can give `401` before the tenant
   boundary is checked. First confirm that this same token can read
   `/api/alerts/active` in org-A through the private header-file method; do not
   count an authentication failure as cross-tenant acceptance.

   Keep your own monitoring estate in its own organization too, rather than
   in the default org, so every boundary in the instance is an explicit org
   boundary. For help, share only the check, redacted endpoint role, status and
   curl exit; never the header file, token, response body or trace.

## Connecting a client's Proxmox or PBS over your VPN

Most MSP estates pair one or two Proxmox nodes per client site with a
site-to-site VPN or tunnel back to the provider network. Proxmox and PBS are
**polled**: the client's Pulse runtime reaches out to the client-site API
(port `8006` for PVE, `8007` for PBS) — nothing at the client site connects
inbound to the runtime for this. That inverts the agent direction, so check
both paths in your firewall:

| Traffic | Direction | Port |
|---------|-----------|------|
| Proxmox/PBS polling | provider → client site, through the tunnel | 8006 / 8007 |
| Agent check-in (hosts) | client site → provider agent ingest | `PULSE_AGENT_INGEST_PORT` (7656) |

Per client, the steps are:

1. Make the client's PVE/PBS API address reachable from the Docker host that
   runs the client workspaces (route or interface into that client's
   tunnel). From that host, make a bounded unauthenticated HTTPS request to
   the PVE/PBS API with certificate verification enabled. For a private or
   self-signed certificate, use `--cacert` with an independently verified
   certificate/CA. A TLS error is not proof of a blocked tunnel, and an HTTP
   response is not proof that Pulse's configured account can collect data.
2. Open the client's workspace (portal → workspace → **Open**) and add the
   node under **Settings → Infrastructure**, using the tunnel-reachable
   address. The guided flow generates a setup command to run once on the
   client's Proxmox host (over SSH through the same tunnel); it creates the
   monitoring user, API token, and permissions. Self-signed certificates
   are handled automatically (the certificate fingerprint is pinned on
   first connect).
3. If you create the token by hand instead, note that Proxmox
   privilege-separated tokens need ACLs on the **token** as well as the
   user, and the built-in `PVEAuditor` role is not sufficient on its own —
   see [TROUBLESHOOTING.md](TROUBLESHOOTING.md) for the exact role setup.

Because each client workspace is its own runtime, overlapping RFC1918
subnets across client sites never collide inside Pulse: each workspace only
ever dials its own client's tunnel addresses.

## Per-client alert routing

Configure notification destinations inside each client's scope — the client
runtime in the provider-hosted model, or the organization in shared-process
mode. A per-client Gotify server, Slack channel, or PSA endpoint only ever
sees that client's alerts.

Webhook targets on private IPs (a Gotify server reached over a VPN tunnel,
for example) are blocked by default for SSRF safety. Allow them once in
**Settings → System → Network → Webhook Security**; the allowlist is
instance-wide and applies to every organization, including ones created
later.

Alert webhook payloads carry the firing tenant's identity (`{{.TenantID}}`,
`{{.TenantName}}`), so a single central PSA endpoint can also route by client.
For ticket bridges (ConnectWise and similar), use the delivery contract —
stable severity/type fields, `X-Pulse-Event-ID` deduplication, and HMAC-signed
deliveries via `signingSecret` — documented in [WEBHOOKS.md](WEBHOOKS.md).

In the provider-hosted model, client runtimes receive `PULSE_TENANT_ID` and
`PULSE_TENANT_NAME` (the workspace display name) from the control plane, so
payloads carry a human-readable client label automatically. A display-name
change applies on the client runtime's next rollout, which recreates the
container. Shared-process organizations stamp the org ID and display name
automatically.

## Per-client reports

Each client runtime (or organization) generates its own reports, scoped to
that client's resources:

- **UI**: Settings → Data & Reports.
- **API**: `POST /api/admin/reports/generate` (single resource) and
  `POST /api/admin/reports/generate-multi` (up to 50 resources per report),
  returning PDF or CSV. In shared-process mode, scope with `X-Pulse-Org-ID`
  or an org-bound token. `GET /api/admin/reports/generate` with query
  parameters still returns a single-resource report, but never adds the
  Pulse Assistant narrative.
- **Schedules**: `GET`/`POST /api/admin/reports/schedules`,
  `PUT`/`DELETE /api/admin/reports/schedules/{id}`, and
  `POST /api/admin/reports/schedules/{id}/run`. Schedules can target explicit
  resources and/or comma-separated resource tags, choose weekly or monthly
  cadence, and deliver PDF or CSV output by email or to disk.
  A schedule's `kind` is `resources` (the default, a PDF or CSV performance
  report) or `patrol_digest`, which emails the weekly "what Patrol did for you"
  summary for the whole workspace: runs, new and resolved issues,
  investigations, fixes, and estimated spend. Digest schedules are weekly and
  email-only; scope, format, and attachments are fixed by the server.

Report branding (logo + display name) supports a provider-wide default via
environment (`PULSE_REPORT_PROVIDER_BRAND_DISPLAY_NAME`,
`PULSE_REPORT_PROVIDER_BRAND_LOGO_PATH` or `..._LOGO_BASE64` +
`..._LOGO_FORMAT`) plus a settings-based override. In the provider-hosted
model each client runtime has its own settings, so the override is
per-client; in shared-process mode the settings override applies
instance-wide, so all organizations share one brand (usually yours). Branding
requires the `white_label` entitlement on the licence. Entitled administrators
can edit the settings-based display name and bounded inline PNG, JPEG, or GIF
under **Settings → System → General → Appearance**. That override is used by
both generated reports and the authenticated application header; the browser
title follows the configured display name. Without the entitlement, the
runtime returns and renders the built-in Pulse identity even if branding
settings remain stored.

Scheduled reports are tenant-local. In provider-hosted MSP, each client
runtime stores its own schedules in `report_schedules.json`, writes generated
outputs under `reports/generated/`, and applies its own SMTP settings,
recipients, resource tags, branding, and entitlement checks. If email delivery
is selected before SMTP is configured, Pulse records the run and saves the
report to disk instead of sending it. The Pulse Account portal may show whether
a workspace has an enabled report schedule, but it does not render cross-client
reports or collect report data in the provider control plane.

## Licensing

MSP and Enterprise capabilities (`multi_tenant`, `unlimited`, `white_label`)
are carried on the licence key. MSP plans are sized by client workspace count
(Starter 5, Growth 15, Scale 40); workspace creation is blocked, not billed,
when the limit is reached. The 60-day, two-workspace evaluation is
self-service. Paid MSP licences are currently issued through the assisted
upgrade path so the recurring licence renewal and key binding are checked
before money changes hands.

### Evaluating without a licence

Self-service evaluation is available from the signed provider bundle published
with Pulse v6.4.1. Use this exact release asset and its integrity sidecars; do
**not** download the moving `main` branch archive or run its `setup.sh` as
root. For a later release, first confirm its release page contains the
versioned provider archive, checksum, and SSH signature before changing the
version below.

Download the versioned asset, verify it with Pulse's pinned release key, and
only then extract and run the guided setup:

```bash
export PULSE_VERSION=v6.4.1
export PULSE_MSP_BUNDLE="pulse-provider-msp-${PULSE_VERSION}.tar.gz"
export PULSE_RELEASE_BASE="https://github.com/rcourtman/Pulse/releases/download/${PULSE_VERSION}"

curl -fsSLO "${PULSE_RELEASE_BASE}/${PULSE_MSP_BUNDLE}"
curl -fsSLO "${PULSE_RELEASE_BASE}/${PULSE_MSP_BUNDLE}.sha256"
curl -fsSLO "${PULSE_RELEASE_BASE}/${PULSE_MSP_BUNDLE}.sshsig"

ssh-keygen -Y verify \
  -f <(printf '%s\n' 'pulse-installer namespaces="pulse-install" ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIMZd/DaH+BldzOkq1A8KVTcFk73nAyrE8aJOyf7i00jm pulse-installer') \
  -I pulse-installer \
  -n pulse-install \
  -s "${PULSE_MSP_BUNDLE}.sshsig" < "${PULSE_MSP_BUNDLE}"
sha256sum -c "${PULSE_MSP_BUNDLE}.sha256"

tar -xzf "${PULSE_MSP_BUNDLE}"
cd "pulse-provider-msp-${PULSE_VERSION}"
sudo -E bash ./setup.sh
```

The licence request omits an email address unless you export
`PULSE_PROVIDER_MSP_EVAL_EMAIL` before running `setup.sh`, in which case that
address is included in the request. If you
want setup help, start from the
[Pulse MSP evaluation page](https://pulserelay.pro/msp.html#evaluate) first;
that contact request remains separate from the licence activation.

The host needs Ubuntu 24.04 or similar, a domain you can point at it, and
ports 80 and 443 free. Install `curl`, `openssh-client`, `coreutils`, and `tar`
before the verification step. The wildcard certificate is issued over DNS-01
with Cloudflare as the default provider (`CF_DNS_API_TOKEN`); any other
Traefik dnsChallenge provider works by setting `ACME_DNS_PROVIDER` in `.env`
and putting that provider's credential variables in `dns-credentials.env`.

Leave `CP_PROVIDER_MSP_LICENSE_FILE` blank and `setup.sh` self-issues a
2-client evaluation licence. The v6.4.1 bundle sends the public half of the
signing key generated on your host, a setup-stage marker, and a signup-source
label, plus the optional email address. This licence-request payload does not
include the private key, client inventory or credentials; this is not a claim
that setup makes no other network requests. You can then onboard
two real clients and confirm the isolation boundary on your own infrastructure
before buying.

The evaluation licence lasts 60 days and re-running `setup.sh` reuses the one
already on disk. On an air-gapped host set
`PULSE_PROVIDER_MSP_SKIP_EVAL_LICENSE=1`; the portal, provisioning and client
isolation all still work, but client workspaces will not carry MSP capabilities
until a licence is installed, because release-build client runtimes only trust
entitlement leases chained to a Pulse-signed licence.

`setup.sh` also resolves the four image pins from their published tags when
you leave them blank, writing the resolved digests back into `.env`. The
images are public, so this needs no credentials.

Set the licence file when you buy; paid client caps come from the licence.

### Licensing a provider deployment

In the provider-hosted model the licence is a signed file
(`CP_PROVIDER_MSP_LICENSE_FILE`) that also binds your control plane's
entitlement lease signing key:

1. `setup.sh` generates `CP_ENTITLEMENT_SIGNING_PRIVATE_KEY` locally; the
   private key never leaves your host.
2. Send the derived public key
   (`./setup.sh --print-lease-signing-public-key`) with your licence request.
3. The issued licence binds that key. The control plane refuses to start in
   provider mode if the licence and key do not match, so a misconfigured
   stack fails at startup instead of provisioning client workspaces that
   silently run unlicensed.

Client runtimes lease their entitlements from your control plane (the
control plane injects the refresh endpoint; nothing phones Pulse Cloud) and
verify each lease through the licence chain: Pulse's embedded key signs your
licence, your licence binds your signing key, your signing key signs the
lease. Leases carry the MSP capability set plus `white_label`, so branded
per-client reports work inside every client workspace. When the licence
expires, leases stop verifying after the grace period and client runtimes
fall back to Community behavior; renew and restart the control plane to
restore them.