mirror of
https://github.com/rcourtman/Pulse.git
synced 2026-10-24 04:33:17 +00:00
rcourtman
690d0ef31e
## Problem
244d322dce, merged at 15:01:08 UTC today in #2518, made organization
deletion refuse with `503 tenant_shutdown_incomplete`, and keep the
data, when the tenant's monitoring loop hasn't exited. That guard is
right, but it reused `tenantMonitorShutdownTimeout`. That 2-second
budget dates from March, when it was only a best-effort wait before
logging a warning and carrying on.
A loop that is still starting up or finishing a tick can take longer
than that to see its cancelled context. The observed case is mock mode,
which CI uses: `Monitor.Start` runs `startMockMetricsSampler`
synchronously before its select loop, and `prepareMockMetricsHistory`
seeds seven days of fixture history without a context. The loop only
sees cancellation once seeding finishes; "Mock metrics history sampler
started" and "monitoring loop stopped" log in the same second. E2E run
[37329726585](https://github.com/rcourtman/Pulse/actions/runs/37329726585/job/111833069435),
shard 7, shows it:
| Time (UTC) | Event |
|---|---|
| 15:27:57 | Organization A's monitor initializes |
| 15:28:00 | Delete requested |
| 15:28:02 | `timed out waiting for tenant monitor loop to exit` → 503 |
| 15:28:10 | `monitoring loop stopped` |
| 15:28:20 → 15:28:29 | The retry, with a new organization, repeats the
same sequence |
So `13-api-token-scope` "org-bound token stays inside the issuing org"
failed its org cleanup on both attempts. Run 37331626618 failed the same
way: the first Core E2E failure came two minutes after #2518 merged. A
real delete shortly after a tenant was busy gets the same refusal, with
its alarming "data has been retained" message, until someone retries.
## Change
- **New budget:** `removeTenant` waits on
`tenantDeletionShutdownTimeout` (30s) when deleting. Process `Stop` and
`RemoveTenant` keep the 2-second `tenantMonitorShutdownTimeout`.
- **Still refused:** a loop that never exits, and a failed metadata
close, both still refuse and retain the data. The guard's semantics
don't change.
- **Tests:** both budgets are now variables so tests can shorten them.
`TestTenantDeletionRetainsUnfinishedMonitorLoop` uses a short deletion
budget instead of the real 30 seconds.
- **No timeout conflicts:** the server's `WriteTimeout` is disabled, the
frontend's org delete has no request timeout, and Playwright's request
timeout is 30s.
## Verification
- **New test:** `TestTenantDeletionWaitsForSlowMonitoringLoop` (in the
registered `monitor_additional_test.go`) uses a loop that exits after
300ms, with a 50ms plain budget and a 2s deletion budget. Plain removal
gives up and retains the monitor; deletion waits and completes. Giving
deletion the plain budget makes it fail, so it checks the behaviour
itself.
- **Existing tests:** the deletion safety tests in `internal/monitoring`
(`TestTenantDeletion*`, `TestMultiTenantMonitor*`) and the org-delete
handler tests in `internal/api` all pass, including
`TestOrgDeleteRetainsDataWhenGuestMetadataPersistenceFails`.
**Follow-up in a separate PR:** another session is making the mock
history seeding honour cancellation
(`internal/monitoring/mock_metrics_history.go`), so CI deletions finish
almost at once instead of waiting out the seeding. This PR keeps a
realistic deletion budget either way.
Pulse documentation
Start here for installation, platform setup, security, operations, and Pulse Intelligence. Commands, configuration keys, image names, API fields, and product identifiers remain untranslated in localized guides.
Start here
- Install Pulse — signed Proxmox/Linux installation, Docker, Docker Compose, Kubernetes, and first-run setup.
- Production deployment and security — least-privilege Proxmox setup, root-agent boundaries, discovery, supply-chain verification, plan limits, scale evidence, and a rollout checklist.
- Upgrade from Pulse v5 — migration prerequisites, rollback, agent continuity, and post-upgrade checks.
- Releases and update channels: choosing Stable or Preview, beta and RC expectations, and how releases are tested and promoted.
- Configure Pulse — authentication, notifications, discovery, retention, and system settings.
- Deployment models — data locations, lifecycle, and differences between supported deployment paths.
- Troubleshooting and FAQ — common failures, diagnostics, and operator questions.
Localized getting started guides: Deutsch · Español
Platforms and agents
- Proxmox Backup Server
- Proxmox Mail Gateway
- Docker and Podman
- Kubernetes and Helm
- TrueNAS SCALE and CORE
- Unified Agent
- Agent security
- VM disk monitoring
- ZFS monitoring
- Temperature monitoring
VMware vSphere support is early access. Current builds expose dedicated vSphere inventory and recovery context, but operators should validate the integration against their own vCenter before production use.
Monitoring and operations
- Metrics history
- Recovery data
- Webhooks
- Automatic updates
- Centralized agent management (Pro)
- Operational trust model
- Current product screenshots
Pulse Intelligence
- Assistant, Patrol, and external-agent overview
- Patrol modes and safety
- Assistant safety model
- External agent HTTP and MCP substrate
Patrol watch-only analysis is available on Community with a local model or the operator's own provider. Investigation and governed fixes require the relevant Pulse Pro capabilities.
Security, privacy, and access
- Production deployment and security
- Security review scope
- Authentication and credential storage review packet
- Security guide
- Privacy and telemetry disclosure
- OIDC and SSO
- Proxy authentication
- Role-based access control (Pro)
- Audit logging (Pro)
- Reverse proxy configuration
- Code-signing policy
Plans and managed access
- Community, Relay, and Pro capabilities
- Relay and Pulse Mobile handoff
- Multi-tenant organizations (Enterprise/custom)
- Provider-hosted MSP operations (request-assisted)
Pulse Cloud is not generally available. Ordinary self-hosted Pulse remains the primary installation path; MSP and Enterprise access are explicit commercial paths rather than defaults in self-hosted setup.
Development and reference
Detailed design notes and dated migration specifications may remain in this directory for maintainers, but they are not operator setup guides unless they are linked from the sections above.
Previous versions and migrations
- Upgrade from v4 to v5
- Retired unified-navigation migration — historical context only; current Pulse uses platform-shaped navigation.
- Move a Pulse installation
Found a bug? Use the issue forms. For setup questions, use GitHub Discussions.