pulse/tests/integration

Integration Tests (Playwright)

End-to-end Playwright tests that validate critical user flows against a running Pulse instance.

Architecture

┌─────────────────┐     ┌──────────────────┐     ┌─────────────────────┐
│  Playwright     │────▶│  Pulse Server    │────▶│  Mock GitHub API    │
│  (Browser UI)   │     │  (Test Instance) │     │  (Controlled        │
│                 │     │                  │     │   Responses)        │
└─────────────────┘     └──────────────────┘     └─────────────────────┘

Test Scenarios

• tests/00-diagnostic.spec.ts — smoke test that the stack boots and the UI renders.
• tests/01-core-e2e.spec.ts — critical UI flows:
  • Bootstrap setup wizard (fresh instance)
  • Login + authenticated state
  • Alerts thresholds create/delete
  • Settings persistence across refresh
  • Add/delete a Proxmox node (test-only)
• tests/02-navigation-perf.spec.ts — route transition performance budgets for:
  • Infrastructure → Workloads
  • Workloads → Infrastructure
• tests/06-theme-visual.spec.ts — visual regression baselines for light/dark auth surfaces:
  • Logged-out login page (full page + form card)
  • Authenticated Settings → Authentication page
• tests/07-retired-trial-acquisition.spec.ts — retired self-hosted trial acquisition route contract:
  • Probe POST /api/license/trial/start and verify ordinary self-hosted v6 returns 404
  • Verify local entitlements remain unchanged after the retired route probe
• tests/58-self-hosted-trial-rate-limit-ui.spec.ts — paid prompt visibility contract:
  • Open /settings/system/billing on the real browser shell with free-tier entitlements
  • Verify trial CTAs and paid-only navigation stay out of the default self-hosted UI
• tests/08-cloud-hosting.spec.ts — hosted cloud signup contract:
  • Public /cloud/signup form creates a real Stripe sandbox checkout session
  • Checkout completes and returns to hosted signup completion page
  • Magic-link request path succeeds via real public endpoint
• tests/09-cloud-billing-lifecycle.spec.ts — hosted cloud post-checkout lifecycle:
  • Replays verified Stripe webhook events into control plane
  • Asserts tenant activation after checkout event processing
  • Asserts tenant cancellation after subscription deletion event processing
• tests/16-dev-runtime-recovery.spec.ts — managed browser runtime proof:
  • Attaches Playwright to the canonical 5173 browser entrypoint
  • Restarts the managed backend and proves the browser shell recovers through the proxy
• tests/17-proxmox-backups-layout.spec.ts — desktop Proxmox Backups layout guard:
  • Needs the mock-mode dataset and skips against a real-mode runtime (npm run mock:on first)
  • Proves picking an activity day keeps /proxmox/backups and shows an N of M backups count
  • Proves a one-year range keeps the page inside the horizontal viewport and the PBS servers table's trailing column inside its wrapper
• tests/18-patrol-runtime-state.spec.ts — Patrol runtime-state browser guard:
  • Mocks a blocked Patrol runtime with stale healthy summary payloads
  • Proves the real /ai route shows Patrol as paused and suppresses stale healthy summary copy
• tests/68-infrastructure-onboarding.spec.ts — infrastructure onboarding browser proof:
  • Proves /settings/infrastructure stays instance-first until the user opens the add flow
  • Verifies the add tile opens a grouped source-type picker, with detect-from-address as a secondary utility inside that modal flow
  • Verifies an explicit discovery run surfaces Proxmox-family candidates in the source-manager table and opens the matching prefilled review dialog
  • Verifies the onboarding funnel emits picker-driven API handoff and no-match-to-agent fallback metrics on the real browser runtime
  • Verifies the mobile landing and picker modal fit the viewport without horizontal overflow
• tests/69-diagnostics-onboarding.spec.ts — diagnostics onboarding analytics browser proof:
  • Proves the Diagnostics & Health page renders the infrastructure onboarding analytics card after a real diagnostics run
  • Verifies the shared diagnostics surface shows onboarding path/platform attribution alongside the existing commercial funnel
  • Verifies the exported full and sanitized diagnostics JSON preserve onboarding analytics while redacting operator-sensitive host/subnet data
  • Verifies the populated mobile diagnostics page fits the viewport without horizontal overflow

Running Tests

Local Development (Docker compose stack)

cd tests/integration
./scripts/setup.sh   # one-time (installs deps + builds docker images)
npm test

CI Tiers (stable / probation / quarantine)

CI splits the suite into a gating stable tier and a non-gating probation tier, selected with PULSE_E2E_TIER=stable|probation (unset = the full suite, which is what npm test runs locally). The tier lists, the classification data behind them, and the promotion/demotion rule (10 consecutive green main runs promotes a probation spec; one failure or retry-flake demotes it back) are documented next to PROBATION_SPECS and QUARANTINED_SPECS in e2e-tiering.mjs. Quarantined specs are rotted and do not run at all; probation specs run on every push but cannot fail the e2e-verdict job in .github/workflows/test-e2e.yml.

Validate the ledger and Playwright's complete per-project split with:

cd tests/integration
npm run check:e2e-tiers

The check uses Playwright's own --list output, verifies that stable and probation are disjoint and exhaustive across every configured project, rejects stale/duplicate/overlapping ledger entries, and protects a minimum stable-tier signal floor.

Eval Packs (No Manual Steps)

Run the curated scenario pack (multi-tenant, retired-trial-acquisition, cloud-hosting, cloud-billing-lifecycle) and emit a report:

cd tests/integration
npm run evals

Filter to one scenario:

npm run evals -- --scenario retired-trial-acquisition

Reports are written to:

• tests/integration/eval-results/<timestamp>/report.json
• tests/integration/eval-results/<timestamp>/report.md

Cloud lifecycle evals require these environment variables (test-mode only):

• PULSE_CP_ADMIN_KEY
• PULSE_E2E_STRIPE_API_KEY
• PULSE_E2E_STRIPE_WEBHOOK_SECRET

Agentic Mode (External Browser Agent)

The eval runner supports external browser-capable agents through a command template:

cd tests/integration
PULSE_EVAL_MODE=agentic \
PULSE_EVAL_AGENT_COMMAND_TEMPLATE='<your-agent-command-using {{task_file}} and {{result_json}}>' \
npm run evals

Supported placeholders in PULSE_EVAL_AGENT_COMMAND_TEMPLATE:

• {{task_file}}
• {{result_json}}
• {{scenario_id}}
• {{base_url}}

The docker-compose stack seeds a deterministic bootstrap token for first-run setup:

• Override via PULSE_E2E_BOOTSTRAP_TOKEN
• Default token value is defined in tests/integration/docker-compose.test.yml
• When PULSE_MULTI_TENANT_ENABLED=true, the integration harness also seeds a deterministic Enterprise-eval billing state so the multi-tenant suite runs against a licensed surface instead of skipping.

Credentials used by the E2E suite can be overridden:

• PULSE_E2E_USERNAME (default admin)
• PULSE_E2E_PASSWORD (default adminadminadmin)
• PULSE_E2E_ALLOW_NODE_MUTATION=1 to enable the optional "Add Proxmox node" test (disabled by default for safety)
• PULSE_E2E_PERF=1 to enable navigation performance budget checks
• PULSE_E2E_PERF_ITERATIONS (default 3)
• PULSE_E2E_PERF_PROXMOX_TO_DOCKER_BUDGET_MS (default 2200)
• PULSE_E2E_PERF_DOCKER_TO_PROXMOX_BUDGET_MS (default 2200)
• PULSE_E2E_PERF_PROXMOX_OVERVIEW_TO_STORAGE_BUDGET_MS (default 2200)
• PULSE_E2E_PERF_PROXMOX_STORAGE_TO_OVERVIEW_BUDGET_MS (default 2200)

Run Against An Existing Pulse Instance

cd tests/integration
PULSE_E2E_SKIP_DOCKER=1 \
PULSE_BASE_URL='http://your-pulse-host:7655' \
PULSE_E2E_USERNAME='admin' \
PULSE_E2E_PASSWORD='adminadminadmin' \
npm test

Run Against A Managed Local Backend (No Docker, Deterministic)

cd tests/integration
PULSE_E2E_USE_LOCAL_BACKEND=1 \
PULSE_MULTI_TENANT_ENABLED=true \
npm test -- tests/03-multi-tenant.spec.ts --project=chromium

This mode starts an isolated Pulse backend from the local repo binary in a temporary data directory under tmp/integration-local-backend, seeds the requested entitlement profile, writes runtime connection state for Playwright, and cleans everything up in posttest.

Each npm test invocation gets its own runtime-state file and managed-backend root automatically, so separate managed-local-backend runs can execute in parallel without sharing PID or cleanup state. Shared embedded-frontend and backend-binary refreshes are serialized by the harness.

Run Against The Managed Hot-Dev Browser Runtime

Canonical one-command verification from the repo root:

npm run dev:verify

Equivalent direct proof command from the integration harness:

cd tests/integration
PULSE_E2E_USE_HOT_DEV=1 \
PULSE_E2E_SKIP_PLAYWRIGHT_INSTALL=1 \
npm test -- tests/16-dev-runtime-recovery.spec.ts tests/17-proxmox-backups-layout.spec.ts tests/18-patrol-runtime-state.spec.ts --project=chromium

This mode attaches Playwright to the canonical dev browser entrypoint on http://127.0.0.1:5173, uses the repo-root managed runtime wrappers as the control surface, and writes browser runtime connection state for Playwright instead of targeting the backend port directly. Those wrappers are backed by scripts/hot-dev-bg.sh, but the wrapper surface is the canonical operator contract. When you run the wrapper as npm run dev:verify, the managed launcher hands a verification-lock path into the integration runner, and the runner holds that lock for the actual pretest, Playwright, and posttest lifetime so unrelated backend source churn does not invalidate the recovery proofs mid-run. When no explicit PULSE_BASE_URL, PLAYWRIGHT_BASE_URL, or runtime-state file is present, the shared integration browser-default helper now prefers the managed hot-dev browser shell on http://127.0.0.1:5173 whenever hot-dev-bg is already running, and only falls back to the embedded frontend on :7655 when there is no managed dev session to attach to. When both are set, PLAYWRIGHT_BASE_URL now wins for the browser target while PULSE_BASE_URL remains the backend-oriented base for pretest health checks, explicit API helpers, and non-browser setup work. Use that split when you need Playwright on fresh frontend code but still want backend provisioning or health checks against the canonical Pulse API port.

Example:

cd tests/integration
PULSE_E2E_SKIP_DOCKER=1 \
PULSE_BASE_URL='http://127.0.0.1:7655' \
PLAYWRIGHT_BASE_URL='http://127.0.0.1:4174' \
npm test -- tests/30-setup-platform-connections-handoff.spec.ts --project=chromium

If the managed runtime is not already running, the harness starts it. If you need the harness to reclaim existing unmanaged 5173/7655 listeners first, add PULSE_E2E_HOT_DEV_TAKEOVER=1.

The managed proof pack bounces the real backend through npm run dev:backend-restart, kills the supervised hot-dev.sh owner process to prove full runtime recovery, verifies that the browser shell reports those outages and then recovers through the proxy, and keeps a desktop Recovery layout guard on the same canonical browser entrypoint. When the harness attaches to an already-running managed dev session, posttest leaves that runtime running.

For deterministic paid-feature runs against an existing instance, provide one of:

• PULSE_E2E_BILLING_STATE_PATH=/absolute/path/to/billing.json to let the harness write the billing state file directly.
• PULSE_E2E_ENTITLEMENT_WRITE_COMMAND='ssh host "cat > /etc/pulse/billing.json"' to pipe the billing JSON to a remote/local writer command.

The harness understands these profiles:

• PULSE_E2E_ENTITLEMENT_PROFILE=multi-tenant for Enterprise/MSP multi-tenant coverage.
• PULSE_E2E_ENTITLEMENT_PROFILE=infra for Pro/relay/reporting-style journeys.

Snapshot-Clean Proxmox LXC Retired Trial Acquisition SAT

For retired self-hosted trial-start validation against a fresh LXC each run:

• Runbook: docs/operations/RETIRED_TRIAL_ACQUISITION_LXC_SNAPSHOT_RUNBOOK.md
• Probe script: tests/integration/scripts/retired-trial-acquisition-contract.sh
  • Exercises POST /api/license/trial/start from a clean snapshot and proves ordinary self-hosted v6 returns 404 without mutating entitlements
• Pulse Pro browser proof: tests/58-self-hosted-trial-rate-limit-ui.spec.ts
  • Exercises /settings/system/billing on the real browser shell and proves trial CTAs and paid-only navigation stay out of the default self-hosted UI
• Full sandbox orchestration (multi-tenant + retired trial acquisition + cloud, with per-scenario snapshot reset):
  • tests/integration/scripts/run-lxc-sandbox-evals.sh
  • Includes retired self-hosted trial-start validation and cloud subscription cancellation lifecycle verification

Example:

cd tests/integration
./scripts/run-lxc-sandbox-evals.sh

If the snapshot has stale binaries, inject the latest Linux control-plane build on each rollback:

CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -o /tmp/pulse-control-plane-e2e-linux-amd64 ./cmd/pulse-control-plane
cd tests/integration
PULSE_E2E_CP_BINARY=/tmp/pulse-control-plane-e2e-linux-amd64 ./scripts/run-lxc-sandbox-evals.sh

Run Theme Visual Regression Suite

cd tests/integration
PULSE_E2E_SKIP_DOCKER=1 \
PULSE_BASE_URL='http://your-pulse-host:7655' \
PULSE_E2E_USERNAME='admin' \
PULSE_E2E_PASSWORD='your-password' \
npm run test:visual

To refresh baselines intentionally:

npm run test:visual:update

When running against an existing instance (PULSE_E2E_SKIP_DOCKER=1), authenticated visual snapshots require explicit credentials:

• PULSE_E2E_USERNAME
• PULSE_E2E_PASSWORD

If the instance is behind self-signed TLS:

PULSE_E2E_INSECURE_TLS=1 PULSE_E2E_SKIP_DOCKER=1 PULSE_BASE_URL='https://...' npm test

CI Pipeline

• Core E2E flows run via .github/workflows/test-e2e.yml
• Update flow coverage runs as part of the same suite (tests/79-update-flow.spec.ts, backed by mock-github-server/)

Test Data (Update Flow Only)

The mock GitHub server (mock-github-server/) provides controllable responses:

• /api/releases - List all releases
• /api/releases/latest - Latest stable release
• /download/{version}/pulse-{version}-linux-amd64.tar.gz - Release tarballs
• /download/{version}/checksums.txt - Checksum files

Response behavior can be controlled via environment variables:

• MOCK_CHECKSUM_ERROR=true - Return invalid checksums
• MOCK_NETWORK_ERROR=true - Simulate network failures
• MOCK_RATE_LIMIT=true - Enable aggressive rate limiting
• MOCK_STALE_RELEASE=true - Mark releases as stale

Success Criteria

• ✅ Core E2E flows pass reliably in CI
• ✅ Update flow remains covered via API integration test + smoke UI check

Quarantined multi-tenant diagnostic

From the repository root, run:

pulse-heavy-run -- bash tests/integration/scripts/run-tests.sh multi-tenant

This shell-owned run builds isolated images and allocates loopback ports. Its dedicated playwright.multi-tenant-diagnostic.config.ts selects only the seven desktop multi-tenant scenarios, including known failing scenarios. It refuses a non-empty PULSE_E2E_TIER: unset that variable explicitly for diagnostics. The normal stable/probation configuration and quarantine are unchanged. A diagnostic pass is not a stable-gate, installed-customer or release receipt.

The shell-owned runner requires Linux and Python 3: its supervisor uses Linux child-subreaper support and /proc to wait for orphaned browser writers. It fails closed on unsupported platforms rather than claiming safe cleanup. Cookie state is private to the invocation (directory mode 0700) and removed on completion. On TERM, INT or HUP, the supervisor stops and reaps writers before removing that invocation's cookies, reports, videos and result root. A writer ignoring TERM is killed after five seconds. If writers cannot be reaped within fifteen seconds, cleanup fails and retains artifacts rather than racing them. SIGKILL of the supervisor or host failure still requires manual, invocation-scoped cleanup.

Reports remain after ordinary success/failure for inspection; treat browser artifacts as potentially sensitive and remove them after retaining a sanitised receipt. Direct npm, setup and helper invocations do not inherit the shell runner's isolation guarantees. The existing seven scenarios do not establish the delayed admission and reconnect recovery matrix.