A provider-hosted control plane refused to start once its licence was past
expiry and the 7-day grace, because startup validated it with
ValidateLicense. So at the next restart after a 60-day evaluation ran out,
or after a paying provider's subscription lapsed, the portal went down, and
the portal's Plan tab is the only place a provider can buy or renew.
Reproduced on the walkthrough lab.
Startup now accepts an authentic, key-bound licence even when lapsed
(ValidateLicenseAllowingLapse), preferring a current licence and otherwise
the one that expired later, and logs the lapse. It unlocks nothing: client
runtimes still verify the provider licence in every lease with
ValidateLicense and drop MSP capabilities, new clients are refused with
provider_msp_license_lapsed, and the refresher still refuses a lapsed
licence. The Plan panel says the evaluation or plan ended and offers the
plans, and provider-msp status reports license_lapsed without failing so a
lapsed install can still upgrade.
The same run showed the portal printing control plane error codes such as
provider_msp_license_lapsed instead of their message. The portal now
prefers an error's message over its code, which also fixes
already_subscribed and checkout_unavailable in the Plan tab.
setup.sh resolves the image pins in .env to digests once, so upgrade.sh
only ever re-pulled the release a provider first installed. Moving to a
new release meant finding the new digests by hand, so none of a
release's
fixes, and no new feature such as the portal Plan tab, reached an
existing install.
Run from a newly extracted release bundle, upgrade.sh now works on the
existing install: it resolves the bundle's control-plane and runtime
tags
to digests, runs the status gate, preflight and verified backup with the
new release's control plane through a shell override while nothing on
disk changes, then keeps a copy of .env, installs the bundle files and
writes the new pins before starting the new images. --keep-image-pins
keeps hand-set pins.
It also pulls the tenant runtime image before the first status check,
which otherwise failed every upgrade to a new CP_PULSE_IMAGE with "not
present locally", including the documented pin-editing route.
Replaces #2243, which closed when its branch was renamed.
With no custom logo, Settings > General > Application branding previewed
the header with a hand-drawn stand-in: a blue circle holding a "●" text
character, which does not look like the Pulse logo. The header itself
falls back to PulseBrandMark (blue disc, white ring, white centre dot),
so the preview misrepresented what the header shows. Render the same
PulseBrandMark at the header's size; it follows the app theme in light
and dark.
Replaces #2240, which closed when its branch was renamed.
While a standalone agent stays offline across a restart (up to the
72-hour continuity window), every canonical read-state lookup adds it
back from host continuity by building a whole registry clone
(`readStateWithRecords`). Alert evaluation makes one lookup per resource
per poll, so in that state each poll rebuilt the registry once per
resource. Each lookup also called `GetLiveHostsSnapshot` twice, and that
went through `State.GetSnapshot()`, deep-copying every VM and container
just to return the hosts.
**Changes**
- **Overlay cache.** The adapter keeps the last continuity overlay,
keyed by:
- the registry pointer;
- `LastRebuiltAt` (every replace and in-place supplemental ingest
updates it);
- a fingerprint of the requested records. The fingerprint leaves out
`Resource.UpdatedAt`, which `resourceFromHost` stamps with `time.Now()`
at build time.
It reuses the overlay for at most 2 s, because the overlay evaluates
source staleness when it is built. The live adapter is never mutated.
- **Hosts-only copy.** `GetLiveHostsSnapshot` now copies only hosts,
through the existing `State.GetHosts()`. The result is identical,
because `GetSnapshot` sets `Hosts: cloneHosts(s.Hosts)`. This also
removes a full state copy from each agent report (`ApplyHostReport`),
each config fetch and every other caller.
- **Single host copy per lookup.**
`readStateWithStandaloneHostContinuity` takes the host list once instead
of twice.
**One read-state lookup** with one offline standalone agent (synthetic,
8-node clusters, pulse-dev):
| Resources | main | this PR |
|---|---|---|
| 130 | 3.0 ms / 1.9 MB | 13 µs / 10 KB |
| 520 | 15.9 ms / 11.8 MB | 25 µs / 29 KB |
| 1,040 | 41 ms / 28 MB | 39 µs / 54 KB |
| 2,080 | 136 ms / 95 MB | 74 µs / 104 KB |
**Tests.** Each fails when its mechanism is reverted.
- `TestHostContinuityOverlayReusedWithinRegistryGeneration`: reuse
within a generation, including records rebuilt with a fresh `UpdatedAt`;
rebuild after a new generation or different records; the 2 s age bound;
and no mutation of the live adapter.
- `TestStandaloneHostContinuityReadStateReusedAcrossLookups`: end to end
through `GetUnifiedReadStateOrSnapshot` with a real continuity store.
- `TestGetLiveHostsSnapshotCopiesOnlyHosts`: 1 host with 1,000 VMs must
not allocate per guest. The old path made 1,010 allocations.
**Verification on pulse-dev:** `internal/unifiedresources`,
`internal/monitoring`, the `internal/api/...` subpackages and
`cmd/pulse-control-plane` pass.
`TestServerInfoEndpointReportsDevelopment` fails in that source export
with and without this change, because the export has no git checkout.
Refs #2199
Every accepted agent report sends a websocket state invalidation, and
each state broadcast rebuilds and diffs the full frontend state:
| Resources | One broadcast (`BuildFrontendState` + JSON) |
|---|---|
| 130 | 7.5 ms |
| 1,040 | 62 ms |
| 2,080 | 132 ms |
The hub's keyed snapshot adds about 10 ms per 1,000 resources.
The hub coalesced invalidations with a 100 ms debounce that **restarted
on every signal**. That caused two problems:
- Reports spaced more than 100 ms apart (typical for agents on
independent 30 s timers) each produced a full broadcast. With 200 agents
that is roughly 3.4 broadcasts/s, about half a core whenever a browser
is open.
- Signals closer together than the window postponed the broadcast until
they stopped.
**Change: coalescing is now a throttle.**
- The first invalidation after a quiet period still broadcasts after the
100 ms window.
- Later invalidations for the same audience (all clients, or one tenant)
replace the pending request without rescheduling it.
- They broadcast no sooner than `defaultStateBroadcastInterval` (2 s)
after that audience's previous broadcast. That caps broadcast work near
7% of a core at the same scale.
2 s matches the report-driven store refresh interval (#2242), so faster
broadcasts mostly re-sent the same resources. Broadcasts are still
skipped when no client is connected.
**Tests.** Each fails with the old restart-on-signal, unspaced
behaviour.
- `TestStateBroadcastsSpacedByInterval`: the second broadcast waits for
the interval and carries the latest of a burst; the burst produces only
one broadcast.
- `TestTenantStateBroadcastsSpacedByInterval`: the same for tenant
broadcasts.
- `TestStateBroadcastNotStarvedBySteadySignals`: signals every 10 ms
against a 60 ms window still produce broadcasts. The old debounce
produced none in 600 ms.
**Verification on pulse-dev:** `internal/websocket` (existing coalescing
tests unchanged), `internal/monitoring`, `pkg/server` and the
`internal/api/...` subpackages pass.
`TestServerInfoEndpointReportsDevelopment` fails in that source export
with and without this change, because the export has no git checkout.
Refs #2199
Agents poll `/api/agents/agent/{id}/config` every minute, and every
successful poll was written as a signed `agent_config_fetch` audit
event. On a five-agent homelab install that was 483,000 of 483,101 audit
rows (250 MB) over 75 days, burying the 64 logins and 25 CSRF failures
the log exists to show. At the default 90-day retention it comes to
roughly 60 MB per agent, so an install with hundreds of agents would
carry gigabytes of routine polling in its audit log.
**New rule**
- Every **failed** fetch is still audited (missing scope, unknown agent,
signing or serialisation failure).
- A **successful** fetch is audited only when it is new audit
information:
- the agent's first delivery after Pulse starts;
- a different token;
- a different delivered config (`desiredConfig` hash);
- an unchanged delivery last recorded 24 hours earlier.
- Recorded successes now carry `config=<desiredConfig hash>` and
`reason=first_since_start|token_changed|config_changed|daily` in their
details.
That is about 1,440 rows per agent per day down to a handful.
The tracker lives on the handler, is keyed by org and agent, and is
bounded to 4,096 entries. Handlers built without the constructor keep
the old audit-every-success behaviour.
**Existing rows** are left to age out under retention rather than
deleted, because removing entries from a security log would look like
tampering.
**Docs and contracts.** `AUDIT_LOGGING.md` and its shipped mirror
describe the new rule. The api-contracts and agent-lifecycle contracts
record it. storage-recovery gets a boundary note, because it lists
`internal/api/` as an extension point: recovery actions keep their own
action audits, receipts and verification.
**Tests**
- `TestAgentConfigFetchAuditTrackerRecordsOnlyNewDeliveries` walks first
delivery, unchanged poll, config change, token rotation, and the 24-hour
boundary, plus org isolation and the nil tracker.
- `TestAgentConfigFetchAuditsNewDeliveriesAndEveryFailure` makes five
successful fetches around one config change and gets two success events,
and two failed fetches give two failure events. Without the tracker it
records all five successes.
**Verification on pulse-dev:** `internal/api/...` passes.
`TestServerInfoEndpointReportsDevelopment` fails in that source export
with and without this change, because the export has no git checkout.
Every accepted agent report sends a state invalidation, and each state
broadcast rebuilds and diffs the full frontend state, about 130 ms at
2,080 synthetic resources. The hub coalesced invalidations with a 100 ms
debounce that restarted on every signal, so reports spaced more than
100 ms apart each produced a broadcast, while signals arriving closer
than the window could postpone broadcasts until they stopped (#2199).
With 200 agents reporting every 30 seconds, that was roughly 3.4
broadcasts a second, about half a core whenever a browser was open.
Coalescing is now a throttle. The first invalidation after a quiet
period still broadcasts after the 100 ms window. Later invalidations for
the same audience (all clients, or one tenant) replace the pending
request without rescheduling it, and broadcast no sooner than 2 seconds
after that audience's previous broadcast. That caps broadcast work near
7% of a core at the same scale. Two seconds matches the report-driven
store refresh interval, so faster broadcasts mostly re-sent the same
resources.
Refs #2199
While a standalone agent stayed offline across a restart, every canonical
read-state lookup overlaid it from host continuity by building a whole
registry clone. Alert evaluation makes one lookup per resource per poll, so
for up to 72 hours each poll rebuilt the registry once per resource. Each
lookup also called GetLiveHostsSnapshot twice, and that deep-copied every
guest in the monitor state to return the hosts.
The adapter now keeps the last continuity overlay, keyed by registry
pointer, LastRebuiltAt, and a fingerprint of the requested records that
leaves out the build-time UpdatedAt stamp. It reuses the overlay for at
most two seconds, because the overlay evaluates source staleness when it
is built, and it never mutates the live adapter. GetLiveHostsSnapshot now
copies only hosts through State.GetHosts, which also removes a full state
copy from every agent report and config fetch.
With one offline standalone agent, a read-state lookup at 2,080 synthetic
resources went from 136 ms and 95 MB to 74 us and 104 KB.
Refs #2199
setup.sh resolves the image pins in .env to digests once, so upgrade.sh
only ever re-pulled the release a provider first installed. Moving to a
new release meant finding the new digests by hand, so none of a release's
fixes, and no new feature such as the portal Plan tab, reached an
existing install.
Run from a newly extracted release bundle, upgrade.sh now works on the
existing install: it resolves the bundle's control-plane and runtime tags
to digests, runs the status gate, preflight and verified backup with the
new release's control plane through a shell override while nothing on
disk changes, then keeps a copy of .env, installs the bundle files and
writes the new pins before starting the new images. --keep-image-pins
keeps hand-set pins.
It also pulls the tenant runtime image before the first status check,
which otherwise failed every upgrade to a new CP_PULSE_IMAGE with "not
present locally", including the documented pin-editing route.
Agents poll /api/agents/agent/{id}/config every minute, and every
successful poll was written as a signed agent_config_fetch audit event.
On a five-agent homelab that was 483,000 of 483,101 audit rows (250 MB)
over 75 days, burying the 64 logins and 25 CSRF failures the log exists
to show; at the default 90-day retention it is roughly 60 MB per agent.
Every failed fetch is still audited. A successful fetch is audited when
it is new audit information: the agent's first delivery after Pulse
starts, a different token or desiredConfig hash, or an unchanged
delivery last recorded 24 hours earlier. Recorded successes now carry
the delivered config hash and the reason. The tracker is per handler,
keyed by org and agent, and bounded to 4,096 entries.
Existing rows are left to age out under retention rather than deleted,
since removing entries from a security log would look like tampering.
AUDIT_LOGGING.md and the API and agent-lifecycle contracts describe the
new rule.
With no custom logo, Settings > General > Application branding previewed
the header with a hand-drawn stand-in: a blue circle holding a "●" text
character, which does not look like the Pulse logo. The header itself
falls back to PulseBrandMark (blue disc, white ring, white centre dot),
so the preview misrepresented what the header shows. Render the same
PulseBrandMark at the header's size; it follows the app theme in light
and dark.
Setup's summary tells every new provider to run the install proof before
their first real client. Its temporary workspaces are created on the
provider's own account, so they count against the client limit. On a
two-client evaluation that already had a client, the proof created its
first workspace, was refused the second, and returned no report, so
install-proof's cleanup had nothing to remove and "Provider MSP Proof
01" stayed behind in one of the evaluation's two slots. Found on the
walkthrough lab.
The proof now checks for room before creating anything and says why it
cannot run, and a proof that fails part-way returns the IDs it created
so install-proof removes them. On the lab, with 3 of 3 slots in use,
install-proof now refuses up front and the client count is unchanged.
Every accepted agent report refreshes the canonical resource store, and
two parts of that pass did more work than the estate warranted (#2199).
Guest ingest resolved each guest's parent node through a fallback that
walked every resource, which made each rebuild quadratic. Every
agent-producing source is ingested before the VM and container loops,
so the registry now buckets agents by node name for those loops, as
buildChildCounts already did. It rebuilds the buckets whenever an
ingest yields an agent and skips entries a merge replaced, so parent
resolution matches the full scan exactly. On 8-node clusters one
rebuild went from 74 to 42 ms at 2,080 resources and now grows
linearly.
The refresh pass then cloned the whole registry up to six times, once
in each metric sync and again for the alert sync. A pass now wraps the
store in resourceSnapshotStore, which clones once and forwards
metrics-target resolution, so every consumer reads one generation.
Together, one synthetic agent-report refresh drops from 188 to 82 ms at
2,080 resources and from 90 to 41 ms at 1,040, with allocations per
report falling from 93 MB to 50 MB. Refreshes stay synchronous, as the
agent-lifecycle contract requires; each pass still covers the whole
estate, and coalescing reports would need that contract changed first.
Refs #2199
DetectChanges launched an untracked goroutine to rewrite ai_changes.json.
Tests backed by a temp directory slept 20ms or 100ms hoping it had
finished; when the goroutine started late it wrote into a directory that
cleanup was removing, which is the latent TempDir race already fixed for
IncidentStore and KnowledgeStore.
Nothing in the runtime has called DetectChanges since patrol moved to
agentic execution (1b1c9bb2a3). Recent-change context comes from the
unified-resource timeline, and the detector is only its read fallback, so
the save goroutine was reachable only from tests. Joining it would harden
a dead legacy writer, so remove it along with saveToDisk and the unused
save-scheduling fields. The detector still loads legacy history at
construction; detected changes stay in memory.
Replace the sleeping tests with TestChangeDetector_LegacyHistoryIsReadOnly,
which fails 210/300 runs against the old async writer. With the save
delayed 0-150ms, 349 of 600 old-code saves ran after their test's
directory was removed; none remain. Also drop two sleeps after
RemediationLog.Log, which already persists before returning.
TestProviderMSPLicenseRefreshAppliesShorterPaidPeriod compared the
refresher's UTC expiry ("...Z") with the expected time formatted in the
machine's local zone ("...+01:00"), so it failed on any machine not set
to UTC, including a maintainer Mac on British Summer Time, while CI in
UTC passed. Format the expectation in UTC. It now passes under
Europe/London, America/Los_Angeles and UTC.
Exercise two provider installations with the same tenant ID, unowned-network collision and cleanup isolation. Gate the release-line push before packet freeze and repeat against digest-verified candidate images during container qualification. Update the deployment-installability contract and pin both workflow gates in the release-asset verification suite.
Change-source: pulse-maintainer
Metric identity lived in idx_metrics_lookup, ordered (type, id, metric,
tier, timestamp). Once a series retains history, that order gives every
series its own insertion point, so each poll's commit rewrites one index
leaf page per series through the WAL and again at checkpoint. On a
three-node homelab running v6.4.5-rc.2 that came to 330-607 KB/s of disk
writes, about 10.6 KB per ~60-byte sample and 27-50 GB a day (#1966).
idx_metrics_query_all already holds the same five columns time-major, so
one poll's samples for a resource share pages. It now carries uniqueness
and serves every read, and the metric-major lookup index is dropped. With
an hour of retained history for 480 series, 30 polls write 6,022 WAL
frames instead of 25,310 (4.2x fewer); the empty-table issue-1124 estate
drops from 36,308 frames to 20,223, and its ceiling tightens from 40,000
to 23,000.
The tier-reconciliation overlap probe now names the identity index too;
left to the planner it seeks on (tier, timestamp) alone once the
metric-major tree is gone, which TestRetainedQueryPlansUseIndexes caught.
Metric-filtered reads now scan a resource's other metrics in the window:
a single-metric 500-node query went from 45 to 59 us and the 24-hour
50-disk smart_temp batch from about 0.25 to 0.49 s. All-metric dashboard
reads are unchanged. Continuous write volume is the user-visible harm, so
the contract records that trade explicitly.
The migration reuses the crash-safe swap: databases whose identity is
already enforced by a unique index defer the rebuild to startup
maintenance and swap in one transaction, and a database with no unique
index migrates synchronously. Keeping the name idx_metrics_query_all
means a downgrade rebuilds only the lookup index and leaves no orphan.
Refs #1966
Every IncidentStore mutation queues a coalesced saveAsync goroutine
that later runs MkdirAll on the data directory and writes its .tmp
file. Nothing tracked those goroutines, so a t.TempDir-backed test
could finish while one was still queued. The save then either wrote
into the directory while TempDir's RemoveAll was deleting it (CI:
"unlinkat ...: directory not empty" in
TestIncidentQueryCanonicalOccurrencesAndNotes/memory, run 35932105100)
or recreated the directory after cleanup and leaked it. The test's
synchronous saveToDisk did not help: it cannot join a save that is
already queued.
Track queued saves with a WaitGroup and add an unexported flush(),
mirroring KnowledgeStore (47cadcc188). The disk-backed incident tests
now join through flush in place of polling, a 20ms sleep or waiting
for a save count. The canonical-occurrences test reloads from
RecordNote's own asynchronous save instead of a manual checkpoint.
TestIncidentStoreFlushJoinsQueuedSave holds the save lock to prove
flush waits for a queued save that has not started.
With a 0-3ms delay injected before each save, the old code leaked a
recreated data directory in 233 of 300 runs; with this change, none.
Checkpoint timing, coalescing and the stored format are unchanged.
Keep canonical change classification while avoiding two full List clones on accepted ingest. Document the registry and monitoring boundaries, add exact verification tests, and retain 1,000-resource benchmarks for #2199.
Change-source: pulse-maintainer
Only remove a managed client network with the expected name and tenant labels. Select attached support containers by role and provider ingress before forced disconnect, avoiding cross-provider mutations on shared Docker hosts. Update both cloud-paid and deployment-installability contracts with the regression boundary.
Change-source: pulse-maintainer
Carry the prior source candidate onto the reviewed installer guidance tip so ownership-isolated recovery can receive exact-head proof.
Change-source: pulse-maintainer
Keep first-run evaluation and summary guidance conditional on actual Plan availability and confirmed active limits. Remove the ordinary manual-key purchase prompt while retaining the separate custom-license binding path, and update the installability contract and focused assertions in the same commit.
Change-source: pulse-maintainer
setup.sh still told an evaluating provider to "request an upgrade" at
/msp.html#request, a form that waited on a human reply with a checkout
link. The portal now sells plans itself (Plan tab, Stripe checkout,
applied in seconds), so both the evaluation log line and the closing
summary send the provider there. The summary also stops printing the
lease signing public key as something the licence "must bind": the
portal purchase binds it without a copy step, and the key is still
available through --print-lease-signing-public-key for a custom licence.
Require exact tenant and runtime labels before adopting an existing network, share that check with support-container recovery, and cover wrong or absent labels. Update the cloud-paid and deployment contracts to state the provisioning boundary.
Change-source: pulse-maintainer
Merge the reviewed recovery candidate onto current main while retaining the already landed PR implementation. Filter shared support roles by the configured provider network, verify tenant-network ownership before attachment, and cover two same-host providers plus startup reconciliation.
Change-source: pulse-maintainer
Preserve the refreshed-plan confirmation and its generated portal assets over provider MSP client recovery. Change-source: pulse-maintainer
Change-source: pulse-maintainer
Combine the already published #2216 recovery with the local first-run setup correction; preserve both release-control contract notes.
Change-source: pulse-maintainer
Exercise two provider installations sharing a Docker daemon and role labels. Verify that each recreated support pair only rejoins its own client network and that an unhealthy client is restarted, never stopped.
Change-source: pulse-maintainer
rcourtman