Combine the protected schema-v18 sender with the exact-archive updater. Require installable archives in positive fixtures, classify missing archives and unavailable Pro checks without claiming an offer, and avoid caching an unactivated Pro result. Record the contract and exact combined-source browser receipt.
Change-source: pulse-maintainer
Combine the reviewed main-line sender with the in-flight archive correction so the interaction can be repaired and proved at exact source.
Change-source: pulse-maintainer
A failed update check never reaches the update history, because nothing
was applied, so telemetry could not tell an install whose check is broken
from one that was offered an update and ignored it. The 6.4.3-rc.1 installs
looked healthy while their check failed on every call (#2282).
The update manager now records a closed outcome for every check on the
install's effective channel (up_to_date, available, no_release,
rate_limited, network_error, metadata_error, skipped, error), tagging
lookup failures without changing their error text, and ignores previews of
another channel from Settings. Telemetry schema v18 exports update_channel,
update_check_outcome and update_available. No version, URL or error text
leaves the install. The Settings preview type, privacy disclosure, adoption
report and subsystem contracts are updated to match; the Pulse Pro receiver
change lands separately and the parity check passes against it.
ARMv6 and ARMv7 server binaries both report GOARCH=arm. Read the release binary GOARM build setting and fail closed when it does not name a published server archive, rather than offering an ARMv7 update to ARMv6 installations. Cover the architecture mapping and document the installability boundary.
Change-source: pulse-maintainer
Release metadata contains agent, MCP and multiple architecture archives. An incomplete release must not offer an unrelated tarball or advertise an update that cannot be applied. Retain server archive names only, require the exact tag and runtime architecture, and record this installability boundary with regression coverage.
Change-source: pulse-maintainer
GitHub's release list is now ~9.6 MB because every 6.4.x release embeds
349 assets (~600 KB of JSON each). The update check buffered the whole
response behind a 1 MiB bound, so the primary path always failed and fell
through to the Atom feed. That feed holds ten entries and each release now
spends three of them, so stable-channel checks on main failed outright
("no suitable release found for channel stable") and 6.4.5 stable would
have shipped relying on it.
Decode the release list and the single-release notes response as a token
stream, keeping only tag, name, body, draft, prerelease, publish time and
the Pulse linux server archives. Memory now tracks what the updater keeps,
so the byte bound rises to 64 MiB while staying typed; the Atom feed stays
as the last resort for that over-limit condition only.
Verified against live GitHub: stable resolves v6.4.1 and rc resolves
v6.4.5-rc.3 through the primary path with no feed fallback.
The monitor restored notification destinations after constructing a queue whose five-second ticker could consume due jobs. Defer processor installation through destination restore, then wake the backlog before external alert callbacks admit first agent reports. Cover both forced pre-configuration processing and the actual monitor restart path; update the notifications, monitoring and agent-lifecycle contracts.
Change-source: pulse-maintainer
Keep the persisted empty-device alert ID distinct from the root-device ID while excluding incomplete disks from alias claims. Extend the unified-resources contract and identity tests in the same commit.
Change-source: pulse-maintainer
Keep persisted PVE disk alert identities stable while indexing them as exact physical-disk aliases. A muted disk now clears and suppresses its wearout alert without suppressing neighbouring disks. Update both subsystem contracts and add alert-identity, canonical-alias and registry-backed policy proofs.
Change-source: pulse-maintainer
Retain the focused Patrol setup task while mounting the independent attention inbox when Patrol is off or model readiness is blocked. Reconcile the existing page test and cover responsive list, decision detail, return and remount states.
Refs #2257
Change-source: pulse-maintainer
Parse group-to-role entries by comma or newline, retaining spaces in IdP group names while leaving scopes and allowed-list parsing unchanged. Cover provider edit/save/reload in model, panel, architecture and desktop/narrow browser checks.
Refs #2266
Change-source: pulse-maintainer
A single healthy-looking disk poll could resolve a worn SSD alert, so the next
low reading re-fired its occurrence and notification even after the host-agent
SMART merge correction. Feed reported wearout through the canonical reducer
with three recovery confirmations, retain the failing value during a pending
recovery, and leave absent evidence unchanged. Repeated alternating polls now
keep one active occurrence and one dispatch; sustained recovery still clears it.
Refs #2112
Change-source: pulse-maintainer
Scope physical-disk reads to the opened guest, preserve agent RAID in the drawer, and prove the desktop and narrow interaction states with the performance contract updated in this same commit.
Change-source: pulse-maintainer
The first #2260 benchmark run exposed eager alias indexing during ingest.
The revised commit passed CI Benchmarks after index construction moved to
alias lookup. Keep the owning subsystem contract aligned with that result.
Refs #2199
Carry changed resource IDs into the Workloads projection so periodic deltas
reuse unchanged rows. Index canonical aliases after ingest and resolve
identity-only links without cloning resources. Remove repeated group payloads
from windowed Storage rows, and keep fleet notices and Agent Doctor usable at
narrow widths.
Record the browser proof and remaining payload and benchmark observations in
the owning subsystem contracts.
Refs #2199
Suppressed alert writes previously returned without insertion while
callers continued adding history, recording firing events and requesting
notification delivery. Make acceptance explicit across detector and
restore paths. Repeated muted, retired and maintenance observations now
stop before those side effects, while expiry and resumed monitoring
admit the condition once.
Startup now reconciles restored alerts as soon as persisted resource
policy becomes available, refreshes shared alert state and persists that
resolution. Reconciliation also runs after resource publication, when
restored native aliases first resolve to their canonical agent policy.
Previously muted or retired resources could retain alerts restored
before either boundary.
Retained alert occurrences now reopen when the reducer explicitly
re-fires. Canonical history records occurrence identity separately from
transition time, preserving one incident after checkpoint loss. Delayed
initial firing, resolution replay and historical read repair cannot
override a newer transition. Older canonical records without the
occurrence key retain their existing matching behavior.
Shared table and card renderers preserve components retained across
scrolling windows. Previously a window shift rebuilt those items,
resetting the node Manage tab and losing unsaved lifecycle edits while
scrolling to Save. Mounted regressions cover retained input identity,
text, global position and the existing row budget. Maintenance banners
now show actual local start and end times instead of formatting future
timestamps as “just now”, and describe the paused attention correctly.
The synthetic host fixtures now keep stable machine identities across
graph rebuilds, so saved policies remain attached through restart
verification. The existing stability regression now checks machine IDs
per host, Docker agent IDs and unified canonical agent IDs. Production
conflict checks for a genuinely changed machine identity are unchanged.
Generated installation and rollback guidance now identifies the
Pulse-owned update helper and directs other installations to the
versioned signed installer flow. Both signed installer examples stop on
download or signature failure and run in a fresh temporary directory.
Validation:
- Suppression regression reproduced on the original source across 15
cases, then passed with the change. Full alert package tests pass.
Extended expiry, reconciliation and restore cases pass with the race
detector.
- Signed installer recipes fail six baseline cases and pass all eight
corrected cases. Seven existing installer pinning/signature/rollback
tests pass.
- Release renderer (39 tests), promotion resolver (47 tests),
release-promotion policy (57 tests), documentation mirrors and links,
and governance audits pass. Installer policy assertions cover helper
ownership, the signed fallback, exact version pinning, rollback
applicability and post-restart version verification. The complete
repository pre-commit hook passes on the worker against the matching
staged source, including repository tooling tests, release-control
helper tests and frontend lint. The global sensitivity check passes
locally.
- Startup restore and late native-alias resolution both fail on their
baselines. Final focused proof, race proof and the full monitoring
package pass.
- Retained-occurrence refire regression reproduced in both incident
projections. The complete memory suite passes on final source, full
monitoring and resource-store suites pass, and final focused
replay/refire proof passes under the race detector.
- Shared window regression failed for both table and card renderers on
the baseline, then all nine tests passed. Related component checks
passed 88 tests across seven files. Typecheck and canonical platform
audit pass.
- Maintenance rendering reproduces both incorrect future-time cases on
the baseline. Final operator-state tests pass 22 cases, and targeted
lint and typecheck pass.
- The fixture identity regression fails on the committed generator and
passes after deterministic IDs, together with four related host-link
tests and the Docker host fixture test.
- Production frontend and backend builds pass. Playwright inspection
covers Proxmox Manage and Alerts at 390x844 and 1440x900, 900px History
controls, and shipped install/upgrade docs. Saved maintenance and mute
survive a graceful same-data restart with stable canonical identity.
Cancelling maintenance preserves mute, resuming monitoring reopens the
retained incident with ordered fire/resolve/refire events.
- Authentication and credential baseline passes, including all 41
required API security regressions. All 11 selected Linux peer-identity,
command approval and observer-isolation regressions pass.
- The existing WebSocket connection-cap test now waits for actual per-IP
slot release before testing slot reuse. It still requires HTTP 429 at
capacity and successful registration after release. The corrected test
passes 100 runs under the race detector, and the full agent-execution
package passes with the race detector on the CI merge source.
The existing resource-payload performance gap remains open. The initial
50-node measurement recorded a 1,037 ms maximum desktop idle long task.
A final-source run still observed multi-second tasks, but its elevated
host load prevents a controlled comparison. Both observations and their
limits are retained. This change does not claim to resolve cross-client
payload or browser responsiveness work.
Refs #2237
Rejected active-alert writes still reached history, firing events and
notification dispatch. Make admission explicit across detector and restore
paths so repeated suppressed observations cannot create those side effects.
Reconcile restored alerts when persisted resource policy attaches at startup,
reconcile again after native aliases become canonical, and refresh shared
state before those alerts reach clients.
Preserve occurrence identity when retained alerts re-fire, including durable
history reconstruction and delayed lifecycle replay.
Keep retained table and card components mounted when scrolling shifts the
visible window, so policy editors do not lose unsaved state. Show actual
future maintenance times and the paused-attention effect.
Keep synthetic host machine identities stable across restarts so saved
operator policy remains attached during verification.
Qualify server update and rollback instructions by helper ownership, and
stop signed installer recipes before execution if a download or signature
check fails.
Refs #2237
Preserve the reviewed provider Plan return history and incorporate current upstream changes. Rebuild the portal bundle and manifest from the combined source.
Change-source: pulse-maintainer
A provider-hosted control plane refused to start once its licence was
past
expiry and the 7-day grace, because startup validated it with
ValidateLicense. So at the next restart after a 60-day evaluation ran
out,
or after a paying provider's subscription lapsed, the portal went down,
and
the portal's Plan tab is the only place a provider can buy or renew.
Reproduced on the walkthrough lab.
Startup now accepts an authentic, key-bound licence even when lapsed
(ValidateLicenseAllowingLapse), preferring a current licence and
otherwise
the one that expired later, and logs the lapse. It unlocks nothing:
client
runtimes still verify the provider licence in every lease with
ValidateLicense and drop MSP capabilities, new clients are refused with
provider_msp_license_lapsed, and the refresher still refuses a lapsed
licence. The Plan panel says the evaluation or plan ended and offers the
plans, and provider-msp status reports license_lapsed without failing so
a
lapsed install can still upgrade.
The same run showed the portal printing control plane error codes such
as
provider_msp_license_lapsed instead of their message. The portal now
prefers an error's message over its code, which also fixes
already_subscribed and checkout_unavailable in the Plan tab.
Replaces #2241 and #2250, replayed onto current main as one commit.
Every accepted agent report refreshed the canonical resource store, and
every refresh covers the whole estate, so refresh work grew with agent
count × estate size (#2199). #2235 made each pass linear and cheaper.
This PR bounds how many passes reports can trigger.
**Behaviour.** Accepted host, Docker and Kubernetes reports, including
rejected-report liveness, refresh through
`refreshUnifiedResourceStoreAfterAgentReport`:
- The first report after a quiet window is published before its handler
returns, the same as today.
- Reports within 2 s of the previous refresh fold into one trailing
refresh at the window's end. It reads the latest state and broadcasts it
to websocket clients.
- 2 s matches `readPathRegistryFreshness`, the staleness the read paths
already accept.
- Agent removal and host-agent evaluation still refresh immediately, so
removed inventory retires at once.
- `Monitor.Stop` cancels a pending trailing refresh and waits for a
running one before closing stores.
- Monitors built without `New` keep a zero window and refresh on every
report.
**Cost bound.** At 2,080 resources one refresh costs about 82 ms (after
#2235). 200 agents reporting every 30 s drive 6.7 refreshes/s, roughly
55% of a core; coalesced, reports trigger at most one refresh per 2 s,
about 4%, however many agents report.
**Contract change.** The agent-lifecycle contract promised that an
accepted report is visible when its handler returns. It is now visible
within the 2 s window. The founder approved that trade on 24 September
2026, and the contract records it.
**Tests**
- `TestAgentReportRefreshFoldsReportsWithinWindow`: the leading report
is published synchronously; a burst inside the window is deferred; one
trailing refresh publishes the burst's latest value; after a quiet
window the next report is synchronous again. It fails when the throttle
is removed.
- `TestStopCancelsPendingAgentReportRefresh`: fails if stop does not
cancel the pending refresh.
- The issue-1753 end-to-end test posts two reports back to back, so it
now waits for the published rows instead of reading them at once.
**Verification on pulse-dev**
- `internal/monitoring` runs in 112 s against 107 s on #2235's head. No
test is slower by more than 1.2 s.
- `internal/unifiedresources`, `internal/websocket` and
`internal/api/...` pass. `TestServerInfoEndpointReportsDevelopment`
fails in that source export with and without this change, because the
export has no git checkout.
Refs #2199
Follow-up to #2234. Patrol stopped calling
`ChangeDetector.DetectChanges` when it moved to agentic execution
(1b1c9bb2a3). Recent-change context comes from the unified-resource
timeline, and the detector survives only as a fallback reader of the
legacy `ai_changes.json` that older releases wrote. #2234 removed the
detector's unjoined save. The detection code itself was still there, and
only tests used it.
## Removed
- `DetectChanges` and `detectResourceChanges`, which diffed snapshots.
- `ResourceSnapshot` and its `internal/ai` alias.
- Change-ID generation, and the seven description formatters with their
`formatBytes`/`formatFloat` helpers.
- The detector's lock. The history is loaded once at construction and
never modified, so reads need no lock.
- Tests of the detection logic, its heap stability
(`change_detector_memory_regression_test.go`) and ID uniqueness
(`changes_concurrency_test.go`).
## Unchanged
- `NewChangeDetector`, legacy loading and trimming.
- The read APIs: `GetChangesForResource`, `GetRecentChanges` and
`GetChangesSummary`.
- The `ChangeType` constants, which label the legacy entries.
- `intToString`, which the incident and remediation code share.
## Tests
The `internal/ai` tests that used detection to seed the fallback now
write a legacy `ai_changes.json` and load it. That is the path
production reads.
To check that the new seeding really exercises the load path, I broke
legacy loading through an overlay. 10 of the 12 re-seeded tests then
fail. The other two assert that the fallback is left out of a
type-scoped patrol, or that it loses to the canonical timeline, so empty
history also satisfies them.
That run also exposed three weak assertions, now tightened:
- Two formatter comparisons passed against an empty detector because `""
== ""`. They now reject empty output.
- One summary test used `t.Error` and then indexed `[0]`. It now uses
`t.Fatal`.
The ai-runtime contract now describes the detector as a legacy-history
reader.
A provider-hosted control plane refused to start once its licence was past
expiry and the 7-day grace, because startup validated it with
ValidateLicense. So at the next restart after a 60-day evaluation ran out,
or after a paying provider's subscription lapsed, the portal went down, and
the portal's Plan tab is the only place a provider can buy or renew.
Reproduced on the walkthrough lab.
Startup now accepts an authentic, key-bound licence even when lapsed
(ValidateLicenseAllowingLapse), preferring a current licence and otherwise
the one that expired later, and logs the lapse. It unlocks nothing: client
runtimes still verify the provider licence in every lease with
ValidateLicense and drop MSP capabilities, new clients are refused with
provider_msp_license_lapsed, and the refresher still refuses a lapsed
licence. The Plan panel says the evaluation or plan ended and offers the
plans, and provider-msp status reports license_lapsed without failing so a
lapsed install can still upgrade.
The same run showed the portal printing control plane error codes such as
provider_msp_license_lapsed instead of their message. The portal now
prefers an error's message over its code, which also fixes
already_subscribed and checkout_unavailable in the Plan tab.
setup.sh resolves the image pins in .env to digests once, so upgrade.sh
only ever re-pulled the release a provider first installed. Moving to a
new release meant finding the new digests by hand, so none of a
release's
fixes, and no new feature such as the portal Plan tab, reached an
existing install.
Run from a newly extracted release bundle, upgrade.sh now works on the
existing install: it resolves the bundle's control-plane and runtime
tags
to digests, runs the status gate, preflight and verified backup with the
new release's control plane through a shell override while nothing on
disk changes, then keeps a copy of .env, installs the bundle files and
writes the new pins before starting the new images. --keep-image-pins
keeps hand-set pins.
It also pulls the tenant runtime image before the first status check,
which otherwise failed every upgrade to a new CP_PULSE_IMAGE with "not
present locally", including the documented pin-editing route.
Replaces #2243, which closed when its branch was renamed.
While a standalone agent stays offline across a restart (up to the
72-hour continuity window), every canonical read-state lookup adds it
back from host continuity by building a whole registry clone
(`readStateWithRecords`). Alert evaluation makes one lookup per resource
per poll, so in that state each poll rebuilt the registry once per
resource. Each lookup also called `GetLiveHostsSnapshot` twice, and that
went through `State.GetSnapshot()`, deep-copying every VM and container
just to return the hosts.
**Changes**
- **Overlay cache.** The adapter keeps the last continuity overlay,
keyed by:
- the registry pointer;
- `LastRebuiltAt` (every replace and in-place supplemental ingest
updates it);
- a fingerprint of the requested records. The fingerprint leaves out
`Resource.UpdatedAt`, which `resourceFromHost` stamps with `time.Now()`
at build time.
It reuses the overlay for at most 2 s, because the overlay evaluates
source staleness when it is built. The live adapter is never mutated.
- **Hosts-only copy.** `GetLiveHostsSnapshot` now copies only hosts,
through the existing `State.GetHosts()`. The result is identical,
because `GetSnapshot` sets `Hosts: cloneHosts(s.Hosts)`. This also
removes a full state copy from each agent report (`ApplyHostReport`),
each config fetch and every other caller.
- **Single host copy per lookup.**
`readStateWithStandaloneHostContinuity` takes the host list once instead
of twice.
**One read-state lookup** with one offline standalone agent (synthetic,
8-node clusters, pulse-dev):
| Resources | main | this PR |
|---|---|---|
| 130 | 3.0 ms / 1.9 MB | 13 µs / 10 KB |
| 520 | 15.9 ms / 11.8 MB | 25 µs / 29 KB |
| 1,040 | 41 ms / 28 MB | 39 µs / 54 KB |
| 2,080 | 136 ms / 95 MB | 74 µs / 104 KB |
**Tests.** Each fails when its mechanism is reverted.
- `TestHostContinuityOverlayReusedWithinRegistryGeneration`: reuse
within a generation, including records rebuilt with a fresh `UpdatedAt`;
rebuild after a new generation or different records; the 2 s age bound;
and no mutation of the live adapter.
- `TestStandaloneHostContinuityReadStateReusedAcrossLookups`: end to end
through `GetUnifiedReadStateOrSnapshot` with a real continuity store.
- `TestGetLiveHostsSnapshotCopiesOnlyHosts`: 1 host with 1,000 VMs must
not allocate per guest. The old path made 1,010 allocations.
**Verification on pulse-dev:** `internal/unifiedresources`,
`internal/monitoring`, the `internal/api/...` subpackages and
`cmd/pulse-control-plane` pass.
`TestServerInfoEndpointReportsDevelopment` fails in that source export
with and without this change, because the export has no git checkout.
Refs #2199
Every accepted agent report sends a websocket state invalidation, and
each state broadcast rebuilds and diffs the full frontend state:
| Resources | One broadcast (`BuildFrontendState` + JSON) |
|---|---|
| 130 | 7.5 ms |
| 1,040 | 62 ms |
| 2,080 | 132 ms |
The hub's keyed snapshot adds about 10 ms per 1,000 resources.
The hub coalesced invalidations with a 100 ms debounce that **restarted
on every signal**. That caused two problems:
- Reports spaced more than 100 ms apart (typical for agents on
independent 30 s timers) each produced a full broadcast. With 200 agents
that is roughly 3.4 broadcasts/s, about half a core whenever a browser
is open.
- Signals closer together than the window postponed the broadcast until
they stopped.
**Change: coalescing is now a throttle.**
- The first invalidation after a quiet period still broadcasts after the
100 ms window.
- Later invalidations for the same audience (all clients, or one tenant)
replace the pending request without rescheduling it.
- They broadcast no sooner than `defaultStateBroadcastInterval` (2 s)
after that audience's previous broadcast. That caps broadcast work near
7% of a core at the same scale.
2 s matches the report-driven store refresh interval (#2242), so faster
broadcasts mostly re-sent the same resources. Broadcasts are still
skipped when no client is connected.
**Tests.** Each fails with the old restart-on-signal, unspaced
behaviour.
- `TestStateBroadcastsSpacedByInterval`: the second broadcast waits for
the interval and carries the latest of a burst; the burst produces only
one broadcast.
- `TestTenantStateBroadcastsSpacedByInterval`: the same for tenant
broadcasts.
- `TestStateBroadcastNotStarvedBySteadySignals`: signals every 10 ms
against a 60 ms window still produce broadcasts. The old debounce
produced none in 600 ms.
**Verification on pulse-dev:** `internal/websocket` (existing coalescing
tests unchanged), `internal/monitoring`, `pkg/server` and the
`internal/api/...` subpackages pass.
`TestServerInfoEndpointReportsDevelopment` fails in that source export
with and without this change, because the export has no git checkout.
Refs #2199
Patrol stopped calling DetectChanges when it moved to agentic execution
(1b1c9bb2a3). Recent-change context comes from the unified-resource
timeline, and the detector survives only as a fallback reader of the
legacy ai_changes.json that older releases wrote. The previous change
removed the detector's unjoined save; the detection itself was still
there, kept alive only by tests.
Remove DetectChanges, its snapshot diffing, ResourceSnapshot (and its
internal/ai alias), change-ID generation, the description formatters and
the detector's lock. The history is now loaded once and never modified,
so reads need no lock. The read APIs, trimming and legacy loading are
unchanged.
Tests that used detection to seed the fallback now write a legacy
ai_changes.json and load it, the same path production reads. With
loading broken, 10 of the 12 re-seeded internal/ai tests fail; the other
two assert that the fallback is absent or loses to the canonical
timeline. Two formatter comparisons now reject empty output, which let
them pass against an empty detector. Tests of the detection logic, its
heap stability and ID uniqueness go with it.
Agents poll `/api/agents/agent/{id}/config` every minute, and every
successful poll was written as a signed `agent_config_fetch` audit
event. On a five-agent homelab install that was 483,000 of 483,101 audit
rows (250 MB) over 75 days, burying the 64 logins and 25 CSRF failures
the log exists to show. At the default 90-day retention it comes to
roughly 60 MB per agent, so an install with hundreds of agents would
carry gigabytes of routine polling in its audit log.
**New rule**
- Every **failed** fetch is still audited (missing scope, unknown agent,
signing or serialisation failure).
- A **successful** fetch is audited only when it is new audit
information:
- the agent's first delivery after Pulse starts;
- a different token;
- a different delivered config (`desiredConfig` hash);
- an unchanged delivery last recorded 24 hours earlier.
- Recorded successes now carry `config=<desiredConfig hash>` and
`reason=first_since_start|token_changed|config_changed|daily` in their
details.
That is about 1,440 rows per agent per day down to a handful.
The tracker lives on the handler, is keyed by org and agent, and is
bounded to 4,096 entries. Handlers built without the constructor keep
the old audit-every-success behaviour.
**Existing rows** are left to age out under retention rather than
deleted, because removing entries from a security log would look like
tampering.
**Docs and contracts.** `AUDIT_LOGGING.md` and its shipped mirror
describe the new rule. The api-contracts and agent-lifecycle contracts
record it. storage-recovery gets a boundary note, because it lists
`internal/api/` as an extension point: recovery actions keep their own
action audits, receipts and verification.
**Tests**
- `TestAgentConfigFetchAuditTrackerRecordsOnlyNewDeliveries` walks first
delivery, unchanged poll, config change, token rotation, and the 24-hour
boundary, plus org isolation and the nil tracker.
- `TestAgentConfigFetchAuditsNewDeliveriesAndEveryFailure` makes five
successful fetches around one config change and gets two success events,
and two failed fetches give two failure events. Without the tracker it
records all five successes.
**Verification on pulse-dev:** `internal/api/...` passes.
`TestServerInfoEndpointReportsDevelopment` fails in that source export
with and without this change, because the export has no git checkout.
Every accepted agent report sends a state invalidation, and each state
broadcast rebuilds and diffs the full frontend state, about 130 ms at
2,080 synthetic resources. The hub coalesced invalidations with a 100 ms
debounce that restarted on every signal, so reports spaced more than
100 ms apart each produced a broadcast, while signals arriving closer
than the window could postpone broadcasts until they stopped (#2199).
With 200 agents reporting every 30 seconds, that was roughly 3.4
broadcasts a second, about half a core whenever a browser was open.
Coalescing is now a throttle. The first invalidation after a quiet
period still broadcasts after the 100 ms window. Later invalidations for
the same audience (all clients, or one tenant) replace the pending
request without rescheduling it, and broadcast no sooner than 2 seconds
after that audience's previous broadcast. That caps broadcast work near
7% of a core at the same scale. Two seconds matches the report-driven
store refresh interval, so faster broadcasts mostly re-sent the same
resources.
Refs #2199
While a standalone agent stayed offline across a restart, every canonical
read-state lookup overlaid it from host continuity by building a whole
registry clone. Alert evaluation makes one lookup per resource per poll, so
for up to 72 hours each poll rebuilt the registry once per resource. Each
lookup also called GetLiveHostsSnapshot twice, and that deep-copied every
guest in the monitor state to return the hosts.
The adapter now keeps the last continuity overlay, keyed by registry
pointer, LastRebuiltAt, and a fingerprint of the requested records that
leaves out the build-time UpdatedAt stamp. It reuses the overlay for at
most two seconds, because the overlay evaluates source staleness when it
is built, and it never mutates the live adapter. GetLiveHostsSnapshot now
copies only hosts through State.GetHosts, which also removes a full state
copy from every agent report and config fetch.
With one offline standalone agent, a read-state lookup at 2,080 synthetic
resources went from 136 ms and 95 MB to 74 us and 104 KB.
Refs #2199
setup.sh resolves the image pins in .env to digests once, so upgrade.sh
only ever re-pulled the release a provider first installed. Moving to a
new release meant finding the new digests by hand, so none of a release's
fixes, and no new feature such as the portal Plan tab, reached an
existing install.
Run from a newly extracted release bundle, upgrade.sh now works on the
existing install: it resolves the bundle's control-plane and runtime tags
to digests, runs the status gate, preflight and verified backup with the
new release's control plane through a shell override while nothing on
disk changes, then keeps a copy of .env, installs the bundle files and
writes the new pins before starting the new images. --keep-image-pins
keeps hand-set pins.
It also pulls the tenant runtime image before the first status check,
which otherwise failed every upgrade to a new CP_PULSE_IMAGE with "not
present locally", including the documented pin-editing route.
Agents poll /api/agents/agent/{id}/config every minute, and every
successful poll was written as a signed agent_config_fetch audit event.
On a five-agent homelab that was 483,000 of 483,101 audit rows (250 MB)
over 75 days, burying the 64 logins and 25 CSRF failures the log exists
to show; at the default 90-day retention it is roughly 60 MB per agent.
Every failed fetch is still audited. A successful fetch is audited when
it is new audit information: the agent's first delivery after Pulse
starts, a different token or desiredConfig hash, or an unchanged
delivery last recorded 24 hours earlier. Recorded successes now carry
the delivered config hash and the reason. The tracker is per handler,
keyed by org and agent, and bounded to 4,096 entries.
Existing rows are left to age out under retention rather than deleted,
since removing entries from a security log would look like tampering.
AUDIT_LOGGING.md and the API and agent-lifecycle contracts describe the
new rule.
Every accepted agent report refreshed the canonical resource store, and
every refresh covers the whole estate, so refresh work grew with agent
count times estate size (#2199). The previous commit made each pass
linear and cheaper; this bounds how many passes reports can trigger.
Accepted host, Docker and Kubernetes reports, including rejected-report
liveness, now refresh through a throttle. The first report after a
quiet window is published before its handler returns. Reports within 2
seconds of the previous refresh fold into one trailing refresh at the
window's end, which reads the latest state and broadcasts it. Two
seconds matches readPathRegistryFreshness, the staleness read paths
already accept. Agent removal and host-agent evaluation still refresh
immediately. Monitor.Stop cancels a pending trailing refresh and waits
for a running one before closing stores. Monitors built without New
keep a zero window and refresh on every report.
At 2,080 resources a refresh costs about 82 ms, so 200 agents reporting
every 30 seconds went from roughly 55% of a core in report-driven
refreshes to at most about 4%, however many agents report.
This relaxes the agent-lifecycle guarantee that a report is visible
when its handler returns: it is now visible within the window. The
founder approved that trade on 24 September 2026 and the contract
records it. The issue-1753 end-to-end test posts two reports back to
back, so it now waits for the published rows instead of reading them
at once.
Refs #2199
Setup's summary tells every new provider to run the install proof before
their first real client. Its temporary workspaces are created on the
provider's own account, so they count against the client limit. On a
two-client evaluation that already had a client, the proof created its
first workspace, was refused the second, and returned no report, so
install-proof's cleanup had nothing to remove and "Provider MSP Proof
01" stayed behind in one of the evaluation's two slots. Found on the
walkthrough lab.
The proof now checks for room before creating anything and says why it
cannot run, and a proof that fails part-way returns the IDs it created
so install-proof removes them. On the lab, with 3 of 3 slots in use,
install-proof now refuses up front and the client count is unchanged.
Every accepted agent report refreshes the canonical resource store, and
two parts of that pass did more work than the estate warranted (#2199).
Guest ingest resolved each guest's parent node through a fallback that
walked every resource, which made each rebuild quadratic. Every
agent-producing source is ingested before the VM and container loops,
so the registry now buckets agents by node name for those loops, as
buildChildCounts already did. It rebuilds the buckets whenever an
ingest yields an agent and skips entries a merge replaced, so parent
resolution matches the full scan exactly. On 8-node clusters one
rebuild went from 74 to 42 ms at 2,080 resources and now grows
linearly.
The refresh pass then cloned the whole registry up to six times, once
in each metric sync and again for the alert sync. A pass now wraps the
store in resourceSnapshotStore, which clones once and forwards
metrics-target resolution, so every consumer reads one generation.
Together, one synthetic agent-report refresh drops from 188 to 82 ms at
2,080 resources and from 90 to 41 ms at 1,040, with allocations per
report falling from 93 MB to 50 MB. Refreshes stay synchronous, as the
agent-lifecycle contract requires; each pass still covers the whole
estate, and coalescing reports would need that contract changed first.
Refs #2199
DetectChanges launched an untracked goroutine to rewrite ai_changes.json.
Tests backed by a temp directory slept 20ms or 100ms hoping it had
finished; when the goroutine started late it wrote into a directory that
cleanup was removing, which is the latent TempDir race already fixed for
IncidentStore and KnowledgeStore.
Nothing in the runtime has called DetectChanges since patrol moved to
agentic execution (1b1c9bb2a3). Recent-change context comes from the
unified-resource timeline, and the detector is only its read fallback, so
the save goroutine was reachable only from tests. Joining it would harden
a dead legacy writer, so remove it along with saveToDisk and the unused
save-scheduling fields. The detector still loads legacy history at
construction; detected changes stay in memory.
Replace the sleeping tests with TestChangeDetector_LegacyHistoryIsReadOnly,
which fails 210/300 runs against the old async writer. With the save
delayed 0-150ms, 349 of 600 old-code saves ran after their test's
directory was removed; none remain. Also drop two sleeps after
RemediationLog.Log, which already persists before returning.
Exercise two provider installations with the same tenant ID, unowned-network collision and cleanup isolation. Gate the release-line push before packet freeze and repeat against digest-verified candidate images during container qualification. Update the deployment-installability contract and pin both workflow gates in the release-asset verification suite.
Change-source: pulse-maintainer
Metric identity lived in idx_metrics_lookup, ordered (type, id, metric,
tier, timestamp). Once a series retains history, that order gives every
series its own insertion point, so each poll's commit rewrites one index
leaf page per series through the WAL and again at checkpoint. On a
three-node homelab running v6.4.5-rc.2 that came to 330-607 KB/s of disk
writes, about 10.6 KB per ~60-byte sample and 27-50 GB a day (#1966).
idx_metrics_query_all already holds the same five columns time-major, so
one poll's samples for a resource share pages. It now carries uniqueness
and serves every read, and the metric-major lookup index is dropped. With
an hour of retained history for 480 series, 30 polls write 6,022 WAL
frames instead of 25,310 (4.2x fewer); the empty-table issue-1124 estate
drops from 36,308 frames to 20,223, and its ceiling tightens from 40,000
to 23,000.
The tier-reconciliation overlap probe now names the identity index too;
left to the planner it seeks on (tier, timestamp) alone once the
metric-major tree is gone, which TestRetainedQueryPlansUseIndexes caught.
Metric-filtered reads now scan a resource's other metrics in the window:
a single-metric 500-node query went from 45 to 59 us and the 24-hour
50-disk smart_temp batch from about 0.25 to 0.49 s. All-metric dashboard
reads are unchanged. Continuous write volume is the user-visible harm, so
the contract records that trade explicitly.
The migration reuses the crash-safe swap: databases whose identity is
already enforced by a unique index defer the rebuild to startup
maintenance and swap in one transaction, and a database with no unique
index migrates synchronously. Keeping the name idx_metrics_query_all
means a downgrade rebuilds only the lookup index and leaves no orphan.
Refs #1966
Every IncidentStore mutation queues a coalesced saveAsync goroutine
that later runs MkdirAll on the data directory and writes its .tmp
file. Nothing tracked those goroutines, so a t.TempDir-backed test
could finish while one was still queued. The save then either wrote
into the directory while TempDir's RemoveAll was deleting it (CI:
"unlinkat ...: directory not empty" in
TestIncidentQueryCanonicalOccurrencesAndNotes/memory, run 35932105100)
or recreated the directory after cleanup and leaked it. The test's
synchronous saveToDisk did not help: it cannot join a save that is
already queued.
Track queued saves with a WaitGroup and add an unexported flush(),
mirroring KnowledgeStore (47cadcc188). The disk-backed incident tests
now join through flush in place of polling, a 20ms sleep or waiting
for a save count. The canonical-occurrences test reloads from
RecordNote's own asynchronous save instead of a manual checkpoint.
TestIncidentStoreFlushJoinsQueuedSave holds the save lock to prove
flush waits for a queued save that has not started.
With a 0-3ms delay injected before each save, the old code leaked a
recreated data directory in 233 of 300 runs; with this change, none.
Checkpoint timing, coalescing and the stored format are unchanged.
pulse-triage[bot]