diff --git a/docs/MSP.md b/docs/MSP.md new file mode 100644 index 0000000000..be5c37c8fe --- /dev/null +++ b/docs/MSP.md @@ -0,0 +1,147 @@ +# Pulse for MSPs (Provider Operations Guide) + +This guide covers running Pulse as a managed service provider: one central +deployment monitoring multiple client estates, with per-client isolation, +alert routing, and reporting. It assumes you have read +[DEPLOYMENT_MODELS.md](DEPLOYMENT_MODELS.md) for the deployment-model overview. + +## Deployment models + +**Provider-hosted MSP (canonical).** A control plane runs one isolated Pulse +runtime per client workspace. Alerts, webhook destinations, branded report +settings, users, audit history, and metrics stay inside the client runtime; +duplicate hostnames across clients never collide because they never share a +runtime namespace. The stack is operated with the `pulse-control-plane` +binary: + +```bash +pulse-control-plane provider-msp bootstrap --account-name "Your MSP" --owner-email you@example.com +pulse-control-plane provider-msp status +pulse-control-plane provider-msp backup +pulse-control-plane provider-msp recover # restore workspaces from backup or disk +pulse-control-plane provider-msp preflight # pre-install environment checks +``` + +Each client runtime is a normal Pulse instance, so it connects to that +client's infrastructure with the standard methods: agents push over HTTPS for +hosts, and Proxmox/PBS polling reaches across networks through your existing +VPN or tunnel to the client site. + +**Shared-process organizations (alternative).** One Pulse process serves +multiple organizations with isolated data directories, org-bound tokens, and +per-org alert/webhook/notification state. This is documented in +[MULTI_TENANT.md](MULTI_TENANT.md) and gated by `PULSE_MULTI_TENANT_ENABLED=true` +plus a licence carrying the `multi_tenant` capability. It is designed for one +owner separating internal estates (sites, departments, environments); the +isolated-runtime model above is the canonical choice for separate customer +businesses. + +## Network topology and ingress isolation + +Run the management UI and agent check-in on separate, separately firewalled +ports. See [Split-Port Agent Ingest](CONFIGURATION.md#split-port-agent-ingest-network-isolation) +for the full reference. + +```bash +FRONTEND_PORT=7655 # management UI + API: private network / VPN only +PULSE_AGENT_INGEST_PORT=7656 # agent check-in only: reachable from client sites +PULSE_AGENT_CONNECT_URL=https://agents.example.com:7656 +``` + +Firewall baseline: + +| Surface | Port | Reachable from | +|---------|------|----------------| +| Management UI + API | `FRONTEND_PORT` (7655) | Provider staff network / VPN only | +| Agent ingest | `PULSE_AGENT_INGEST_PORT` (7656) | Client sites (or client VPN tunnels) | +| Prometheus metrics | 9091 | Provider monitoring network only | + +The dedicated agent port serves **only** `/api/agents/*`; every other path, +including login and the management API, returns `404`. Agent check-in +authenticates with an `agent:report`-scoped API token, which cannot read +monitoring data or change settings — the token scope and the port isolation +are independent layers. + +If agents reach the central server over per-client VPN tunnels instead of the +public internet, the same split still applies: expose only the agent port into +the tunnels and keep the management port out of them. + +### Validation checklist (run after setup, repeat after network changes) + +1. **Agent port serves agent ingest only.** Both must return `404`: + + ```bash + curl -sk -o /dev/null -w '%{http_code}\n' https://agents.example.com:7656/ # 404 + curl -sk -o /dev/null -w '%{http_code}\n' https://agents.example.com:7656/api/login # 404 + ``` + +2. **Management port is not reachable from a client site.** From a client + network (or through a client tunnel), a connection to `FRONTEND_PORT` must + time out or be refused by your firewall — not answer. + +3. **Agent tokens cannot manage.** A request to a management endpoint with an + agent token must be rejected: + + ```bash + curl -sk -o /dev/null -w '%{http_code}\n' \ + -H "X-API-Token: " https://pulse.internal:7655/api/notifications/webhooks # 401/403 + ``` + +4. **Cross-tenant isolation (shared-process mode only).** A token bound to one + organization must get `403` when targeting another: + + ```bash + curl -sk -o /dev/null -w '%{http_code}\n' \ + -H "X-API-Token: " -H "X-Pulse-Org-ID: org-b" \ + https://pulse.internal:7655/api/alerts/active # 403 + ``` + +## Per-client alert routing + +Configure notification destinations inside each client's scope — the client +runtime in the provider-hosted model, or the organization in shared-process +mode. A per-client Gotify server, Slack channel, or PSA endpoint only ever +sees that client's alerts. + +Alert webhook payloads carry the firing tenant's identity (`{{.TenantID}}`, +`{{.TenantName}}`), so a single central PSA endpoint can also route by client. +For ticket bridges (ConnectWise and similar), use the delivery contract — +stable severity/type fields, `X-Pulse-Event-ID` deduplication, and HMAC-signed +deliveries via `signingSecret` — documented in [WEBHOOKS.md](WEBHOOKS.md). + +In the provider-hosted model, client runtimes receive `PULSE_TENANT_ID` from +the control plane; set `PULSE_TENANT_NAME` alongside it for a human-readable +client label in payloads. Shared-process organizations stamp the org ID and +display name automatically. + +## Per-client reports + +Each client runtime (or organization) generates its own reports, scoped to +that client's resources: + +- **UI**: Settings → Reports. +- **API**: `GET /api/admin/reports/generate` (single resource) and + `POST /api/admin/reports/generate-multi` (up to 50 resources per report), + returning PDF or CSV. In shared-process mode, scope with `X-Pulse-Org-ID` + or an org-bound token. + +Report branding (logo + display name) supports a provider-wide default via +environment (`PULSE_REPORT_PROVIDER_BRAND_DISPLAY_NAME`, +`PULSE_REPORT_PROVIDER_BRAND_LOGO_PATH` or `..._LOGO_BASE64` + +`..._LOGO_FORMAT`) plus a settings-based override. In the provider-hosted +model each client runtime has its own settings, so the override is +per-client; in shared-process mode the settings override applies +instance-wide, so all organizations share one brand (usually yours). Branding +requires the `white_label` entitlement on the licence. + +Pulse does not yet schedule recurring reports; generate monthly client reports +on demand from the UI, or call the report API from your own scheduler with an +org-bound token. + +## Licensing + +MSP and Enterprise capabilities (`multi_tenant`, `unlimited`, `white_label`) +are carried on the licence key. MSP plans are sized by client workspace count +(Starter 5, Growth 15, Scale 40); workspace creation is blocked, not billed, +when the limit is reached. MSP and Enterprise keys are issued through sales — +contact support to get set up or to join the MSP design-partner program. diff --git a/docs/README.md b/docs/README.md index 59c04d43c7..7f76df3f3f 100644 --- a/docs/README.md +++ b/docs/README.md @@ -78,6 +78,7 @@ All other documents are supporting references unless explicitly required for evi - **[Mail Gateway](MAIL_GATEWAY.md)** – Proxmox Mail Gateway (PMG) monitoring. - **[Auto Updates](AUTO_UPDATE.md)** – One-click updates for supported deployments. - **[Multi-Tenant Organizations](MULTI_TENANT.md)** – Isolate infrastructure by organization (Enterprise, opt-in). +- **[Pulse for MSPs](MSP.md)** – Provider operations guide: per-client isolation, split ingress, alert routing, branded reports. - **[Entitlements Overhaul](PULSE_PRO.md)** – Capability-key-based feature gating across Community/Relay/Pro/Cloud, with legacy Pro+ continuity still supported. ## 💳 Plans (Community / Relay / Pro / Cloud) diff --git a/docs/WEBHOOKS.md b/docs/WEBHOOKS.md index b8ab500f77..3ff02b2d53 100644 --- a/docs/WEBHOOKS.md +++ b/docs/WEBHOOKS.md @@ -32,6 +32,7 @@ For generic webhooks, use Go templates to format the JSON payload. - `{{.ResourceName}}`, `{{.ResourceID}}`, `{{.ResourceType}}`, `{{.Node}}` - `{{.Message}}`, `{{.Value}}`, `{{.Threshold}}`, `{{.Duration}}`, `{{.Timestamp}}` - `{{.Instance}}` (Pulse public URL if configured) +- `{{.TenantID}}`, `{{.TenantName}}` (tenant identity in multi-tenant orgs and MSP client runtimes; empty on plain single-tenant installs) - `{{.CustomFields.}}` (user-defined fields in the UI) - `{{.Metadata}}` (alert metadata map) - `{{.AlertCount}}`, `{{.Alerts}}` (grouped alerts) @@ -59,10 +60,57 @@ For generic webhooks, use Go templates to format the JSON payload. } ``` +## 📦 Delivery Contract + +These fields and behaviors are stable; ticket-routing integrations can rely on them. + +**Events.** Every webhook fires on both `alert` and `resolved` events. `{{.Event}}` is `"alert"` or `"resolved"` — there is no separate "info" event class. + +**Severity.** `{{.Level}}` is `"warning"` or `"critical"`. Pulse has exactly these two alert levels. + +**Alert type.** `{{.Type}}` is the metric or condition that fired: `cpu`, `memory`, `disk`, `diskRead`, `diskWrite`, `networkIn`, `networkOut`, `connectivity`, and similar. The alert ID (`{{.ID}}`) is stable for the lifetime of an alert occurrence, so the `resolved` event carries the same ID as the `alert` event it closes. + +**Tenant identity.** In multi-tenant organizations and MSP client runtimes, `{{.TenantID}}` and `{{.TenantName}}` identify which tenant fired the alert. Client runtimes get identity from the `PULSE_TENANT_ID` / `PULSE_TENANT_NAME` environment; shared-process organizations stamp the org ID and display name automatically. + +**Retries.** Failed deliveries retry with exponential backoff. The persistent notification queue makes up to 3 delivery attempts per notification; webhooks configured with transport-level retry add up to 3 more HTTP retries per attempt (1s doubling to a 30s cap, honoring `Retry-After` on HTTP 429). A receiver may therefore see the same logical event more than once. + +**Idempotency.** Every alert delivery carries an `X-Pulse-Event-ID` header of the form `:` (e.g. `a1b2c3:alert`, `a1b2c3:resolved`). It is identical across all retries of the same logical event — deduplicate on it. + +**Signed deliveries.** Set a `signingSecret` on the webhook config to enable HMAC signing. Signed requests carry: + +- `X-Pulse-Timestamp`: Unix seconds at send time. +- `X-Pulse-Signature`: `v1=` + hex HMAC-SHA256 over `timestamp + "." + body`, keyed with the shared secret. + +To verify: recompute the HMAC over the received timestamp and raw body, compare with constant-time equality, and reject requests whose timestamp is outside your tolerance window (e.g. 5 minutes) to block replays. + +```python +import hashlib, hmac + +def verify(secret: str, timestamp: str, body: bytes, signature: str) -> bool: + expected = "v1=" + hmac.new(secret.encode(), f"{timestamp}.".encode() + body, hashlib.sha256).hexdigest() + return hmac.compare_digest(expected, signature) +``` + +The secret is write-only through the API: list responses mask it, and an update that echoes the masked placeholder keeps the stored secret. + +```http +POST /api/notifications/webhooks +Content-Type: application/json + +{ + "name": "PSA Bridge", + "url": "https://psa.example.com/inbound/pulse", + "service": "generic", + "enabled": true, + "signingSecret": "" +} +``` + ## 🛡️ Security - **Private IPs**: By default, webhooks to private IPs are blocked. Allow them in **Settings → System → Network → Webhook Security**. - **Headers**: Add custom headers (e.g., `Authorization: Bearer ...`) in the webhook config. +- **Signing**: Prefer `signingSecret` (above) over bare bearer headers when the receiver supports verification — it authenticates the payload itself, not just the connection. ## 🧾 Audit Webhooks (Pro/legacy Pro+/Cloud) @@ -77,6 +125,8 @@ Audit webhooks are dispatched asynchronously. The payload includes a `signature` ## 🏢 Provider-hosted MSP webhooks +See [MSP.md](MSP.md) for the full provider operations guide (topology, ingress isolation, reports). + Provider-hosted MSP runs one isolated Pulse runtime per client workspace. That means alert routes and webhook destinations are configured inside the client runtime, not in one shared cross-client alert table. A webhook for Client A only sees Client A alerts because Client A has its own Pulse runtime, data, tokens, and notification config. Built-in webhook templates include Gotify, PagerDuty, Slack, and Generic. Use the Generic webhook for systems that accept custom inbound payloads, including ConnectWise and similar PSA or ITSM tools. This is webhook routing, not a bespoke PSA integration. @@ -130,6 +180,73 @@ Content-Type: application/json The exact ticket fields differ by platform (ConnectWise, Autotask, Halo, and others each expect their own inbound shape), so map the template to your platform's contract. The `alertId` round-trips through `{{.ID}}`, which lets the receiving system correlate the later `resolved` event to the ticket it opened. +### Sample PSA payloads + +A fuller template suited to ticket routing, including tenant identity and the +stable severity/type fields from the [delivery contract](#-delivery-contract): + +```json +{ + "event": "{{.Event}}", + "alertId": "{{.ID | jsonString}}", + "severity": "{{.Level | jsonString}}", + "alertType": "{{.Type | jsonString}}", + "tenantId": "{{.TenantID | jsonString}}", + "tenantName": "{{.TenantName | jsonString}}", + "resource": "{{.ResourceName | jsonString}}", + "node": "{{.Node | jsonString}}", + "summary": "{{.Message | jsonString}}", + "value": {{.Value}}, + "threshold": {{.Threshold}}, + "startedAt": "{{.StartTime | jsonString}}", + "duration": "{{.Duration | jsonString}}" +} +``` + +What the receiver sees for a **critical** alert: + +```json +{ + "event": "alert", + "alertId": "f3a9c2d1", + "severity": "critical", + "alertType": "cpu", + "tenantId": "client-acme", + "tenantName": "Acme Corp", + "resource": "web-01", + "node": "pve1", + "summary": "CPU usage 95.2% exceeds threshold 90%", + "value": 95.2, + "threshold": 90, + "startedAt": "2026-06-10T14:03:00Z", + "duration": "5m" +} +``` + +A **warning** alert is identical except `"severity": "warning"` — warning and critical are the only two severities Pulse emits, so a two-priority PSA mapping covers the full range. + +The **resolved** event reuses the same `alertId`, letting the bridge close the ticket it opened: + +```json +{ + "event": "resolved", + "alertId": "f3a9c2d1", + "severity": "critical", + "alertType": "cpu", + "tenantId": "client-acme", + "tenantName": "Acme Corp", + "resource": "web-01", + "node": "pve1", + "summary": "web-01 on pve1 is now healthy", + "value": 95.2, + "threshold": 90, + "startedAt": "2026-06-10T14:03:00Z", + "duration": "22m" +} +``` + +For ConnectWise specifically, point the webhook at a ConnectWise inbound API callback (or middleware that calls the ConnectWise REST API) and map `severity` to ticket priority, `tenantName` to the company, and `alertId` to your correlation field. Combine with a [`signingSecret`](#-delivery-contract) and the `X-Pulse-Event-ID` dedup header for a production-grade bridge. + **Pull (poll): org-scoped read API.** Issue a `monitoring:read` token bound to each organization and poll that org's alerts. Send `X-Pulse-Org-ID` (or rely on the org-bound token) so you get only that organization's data: - `GET /api/alerts/active` — currently firing alerts for the org.