Fixes#2540.
A Telegram webhook passed **Test** while every real alert was rejected
with `Bad Request: message text is empty`.
**Cause.** The two paths built their headers differently:
- The Test path always overlaid the service template's headers
(`Content-Type: application/json`).
- Real deliveries sent the stored header copy that the UI seeded when
the webhook was created. An edit can leave that copy blank, masked or
replaced. For example, the edit form shows the stored value as
`***REDACTED***`, and clearing it saves an empty `Content-Type`.
Telegram's HTTP reader parses body arguments only for JSON,
form-urlencoded or multipart content types, and `sendMessage` checks
`text` before `chat_id`. So a JSON body without a JSON Content-Type
produces exactly that error.
**Fix.** When a built-in service renders its own body (no custom
template), the template's static headers describe that body:
- `prepareWebhookDeliveryContext` applies them on every real firing and
recovery, replacing stored values under any spelling of the header name.
Unrelated stored headers stay.
- The Test path uses the same `withWebhookTemplateHeaders` helper, so a
passing Test predicts real delivery.
- Custom templates and generic webhooks keep their stored headers. ntfy
recoveries still build their own plain-text request.
**Tests** (`internal/notifications/webhook_template_headers_test.go`):
- A real queued Telegram alert, with blank, masked, replaced, lowercase
and duplicate stored Content-Type values, now sends JSON with `text`.
Before the fix it sent `""`, `***REDACTED***` or `text/plain`.
- For every built-in service, the Test, grouped and resolved requests
carry identical template headers and keep unrelated stored headers.
- A custom template keeps its stored Content-Type.
`go test ./internal/notifications` and `./internal/api/alerting` pass.
The non-required **Benchmarks** check has failed on every non-superseded
PR since about 15:00 UTC on 5 October. Examples are #2521, #2520, #2524,
#2525, #2527, #2530 and #2532, each ending with `Paired base and
candidate benchmarks (pull requests)` timed out after 25 minutes. The
paired step runs 10 base and 10 candidate samples, so each second added
per sample costs 20 seconds of step time.
### What made it slower
Paired step duration across PR runs, with per-package seconds per sample
taken from the `ok` lines in the job logs:
| Period | Paired step | `internal/unifiedresources` |
`internal/monitoring` |
|---|---|---|---|
| 24 Sep – 5 Oct 08:00 | 16–18 min | 7.9 s | 4.5 s |
| after #2486 (08:11) | 20.5–20.9 min | 16.2 s | 5.0 s |
| after #2518 (15:01) | killed at 25 min | 38.2 s | 9.9 s |
- **#2518** added `BenchmarkRegistryProjectionSnapshot`. Its `cold` mode
rebuilt the registry between `b.StopTimer` and `b.StartTimer` on every
iteration. Go picks `b.N` from timed work only, so the empty-registry
cases ran hundreds of thousands of untimed ingests. Per sample,
`/0/cold/point-targets` took 10.3 s and `/0/cold/bulk-targets` 5.9 s at
a 100 ms benchtime. That is the cause of the timeouts.
- **#2486** added 46 owner-index and metadata sub-benchmarks, about +8 s
per sample. That is legitimate coverage. #2518's
`BenchmarkBroadcastMetricsSnapshot` and #2527 added about +6 s more.
- `-count`, `-benchtime` and the package list did not change.
### Fix
1. **Root cause.** `cold` now invalidates the registry's derived state
(canonical metadata, typed views, source-target index) instead of
rebuilding the registry. The refresh never reads its own output, so this
repeats a fresh ingest's first read: at 1000 VMs, 40011 vs 40014
allocs/op and about 5.09 MB/op, the same as before. Locally the 24-case
family now runs in 4.9 s; before, its 6 cold cases alone took 33.8 s.
2. **Scope.** Paired PR runs now sample only packages whose in-module
dependency closure (`go list -deps -test`) holds a changed file. A
`_test.go` file selects only its own package. Changes to
`go.mod`/`go.sum`, dirty or non-Git trees, or an unreadable graph select
every package. Applied to today's failures: #2521, #2524 and #2525
sample nothing; #2530 samples `internal/api`; #2532 samples api and
monitoring; #2520 samples 5 packages; #2527 samples 7. The metadata
records `selection=` and `packages=`.
3. **Budget.** A module or shared-model change still samples nearly
everything, about 24 minutes including the ~2.7 min warm-up. The step
timeout is now a measured 30 minutes (job stays at 35).
`PULSE_BENCH_BUDGET_SECONDS=1680` makes the script project its total
after each round. Once the projection reaches 28 minutes it stops and
prints each package's mean seconds per sample, which replaces an
unexplained timeout. `TestPairedBenchmarksStopInsideTheirBudget` keeps
budget < step timeout < job timeout.
4. **Store-backed chart benchmarks** (second commit). Go calls a
benchmark function again for each `b.N` it tries.
`HandleWorkloadCharts_StoreBacked` and
`HandleWorkloadsSummaryCharts_StoreBacked` each seeded 84,000 points
into a fresh SQLite store (about 0.6 s) to time a request of 0.2 ms or
less, two or three times per sample. In CI that was 3.5 s and 3.6 s of
internal/api's 11.5 s per sample. The timed loop now runs in a
`range=4h` sub-benchmark, so the parent seeds once per run. Data,
request and bytes/allocs per op are unchanged, and locally the pair
falls from about 6 s to 1–1.5 s. The names gain `/range=4h`, so this
PR's comparison reports them unmatched.
Unpaired branch runs still sample every package. With the cold fix,
their 15-minute step (13–14.2 min today) regains about 3 minutes of
headroom.
### Verification
- `scripts/tests/test-ci-benchmarks.sh` covers no-op, own-test-only,
shared-dependency, unreadable-graph, dirty-tree, module-file,
budget-exceeded, in-budget and invalid-budget cases. Five mutations of
the selection and budget logic are each caught.
- On the real package graph, selection maps `internal/models` to 7
packages, `internal/monitoring` to api+monitoring, an `internal/api`
embed to api, and frontend or docs paths to none.
- End to end with real Go on two clean clones (`origin/main` vs this
change), only `./internal/unifiedresources/` was sampled. Its per-sample
time went from 53.9 s to 26.7 s on a loaded Mac.
- `go test ./scripts/installtests -run
'TestPairedBenchmarksStopInsideTheirBudget|TestBenchmarkQualificationRetainsProvenance'`,
plus the script-reference and doc-drift suites.
This PR's own Benchmarks run should sample only
`internal/unifiedresources` and `internal/api`, because both changes are
those packages' own test files.
## Problem
Agents installed through the one-step cluster deploy lose their command
channel for good after their first restart.
`POST /api/agents/agent/enroll` bound the runtime token to an invented
`agent-<hostname>` and returned that as the agent's ID. The agent used
it until Pulse acknowledged its first report with the host's real,
machine-derived ID, which the agent persists. On the next start the
agent registered under that real ID. The token was by then a version-2
binding to `agent-<hostname>`, so admission refused it with "Agent token
mismatch: token is not bound to the registering agent", every few
minutes, indefinitely.
Seen live on a v6.5.0-rc.1 server with a deployed Proxmox node: reports
flowed, but the node could run no commands or actions.
## Fix
- **Enrollment leaves identity to the agent.** The runtime token is
bound to the node's hostname and the current binding version, marked
`deploy_identity_binding=agent`, and the response no longer carries
`agentId`. The first command registration backfills the agent's own ID
through the existing hostname-bound backfill path, and the config gate
takes the same decision.
- **One-time repair for tokens already issued.** A token enrollment
bound to `agent-<hostname>` (it has `deploy_job_id` and no identity
marker) may move once to the registering agent's ID:
- The hostname must match exactly (case-insensitive), not by short-name
equivalence.
- The move persists `deploy_identity_binding=repaired` in the same save,
and a failed save restores the previous metadata.
- It applies whatever the token's binding version.
- Any other identity or hostname is refused rather than handed to the
legacy hostname migration.
- A marked token never moves again, including after a hostname rebind
recreates the `agent-<hostname>` shape.
## Tests
-
**`TestHandleEnroll_RuntimeTokenBindsTheAgentsOwnIdentityOnFirstCommandRegistration`**
replays the field sequence: the identity enrollment handed out, then the
acknowledged identity after a restart. It fails on the old code at the
"registration after restart" step.
- **`TestDeployPlaceholderTokenMovesToTheRegisteringAgentOnce`** covers:
- the repair on the exact hostname only;
- the persisted marker after a reload;
- the repair, rebind to host b, repair-again attack;
- config-gate parity.
- **`TestVersionlessDeployPlaceholderCannotBeMovedTwice`** covers the
versionless variant of that attack.
- **`TestCurrentDeployTokenNeverMovesOnceBound`** checks that a token
issued by current enrollment never moves once bound.
- **Policy unit tests:**
`TestEvaluateRepairsDeployPlaceholderBindingOnce`,
`TestEvaluateVersionlessDeployPlaceholderOnlyMovesThroughTheRepair` and
`TestEvaluateCurrentDeployTokenBackfillsTheAgentIdentity`.
- **`TestContractDeployEnrollResponseCarriesNoAgentIdentity`** pins the
enroll response keys.
- **Mutation checks:** removing the marker check, loosening the hostname
match, not persisting the marker, or letting placeholders fall through
to the legacy branch each fails at least one of these tests.
## Not in this change
Both of these exist for every install path and are not made worse here:
- The first command registration binds whatever ID the agent presents.
If that differs from the identity Pulse later resolves for its reports
(a configured agent ID, or a host continuity fork), the next restart
still mismatches.
- Tokens that never connect the command channel keep an empty
`bound_agent_id`, so collector-authority reduction cannot match them.
Before this change they had a non-empty but wrong `agent-<hostname>`,
which failed the same way.
A Telegram webhook passed Test while every real alert was rejected with
"message text is empty" (#2540). The Test path always overlaid the
service template's headers, but real deliveries sent the stored header
copy that the UI seeded at creation. An edit can leave that copy blank,
masked or replaced. Telegram reads arguments from a body only when it is
labelled as JSON or form data, and checks text before chat_id, so a JSON
body without a JSON Content-Type produces exactly that error.
When a built-in service renders its own body (no custom template), the
template's static headers describe that body. Apply them on every real
firing and recovery through prepareWebhookDeliveryContext, replace stored
values under any spelling of the name, and route the Test path through
the same helper so Test predicts real delivery. Custom templates and
generic webhooks keep their stored headers.
Go calls a benchmark function again for every b.N it tries. Each of the
two store-backed chart benchmarks seeded 84,000 points into a fresh
SQLite store, about 0.6 s, to time a request of 0.2 ms or less, and did
so two or three times per sample. In CI they took 3.5 s and 3.6 s of
internal/api's 11.5 s per sample, for about 0.1 s of timed work each.
The timed loop now runs in a range=4h sub-benchmark. A benchmark with
sub-benchmarks runs once, so the parent seeds its store once per run
while the data, request and sanity checks stay the same, and bytes and
allocations per op are unchanged. Locally the pair falls from about 6 s
to 1-1.5 s. The names gain /range=4h, so the first paired comparison
reports them unmatched.
Since 15:00 UTC on 5 October every paired benchmark run that was not
superseded has hit its 25-minute step timeout. Each step runs ten base
and ten candidate samples, so per-sample growth is multiplied by twenty.
BenchmarkRegistryProjectionSnapshot (#2518) rebuilt a registry behind
StopTimer on every cold iteration. Go sizes b.N from timed work only, so
the empty-registry cases ran hundreds of thousands of untimed ingests
and took 16 s of a 100 ms benchtime. With #2486's new owner-index and
metadata benchmarks, internal/unifiedresources went from 8 s to 38 s per
sample. Cold now invalidates the registry's derived state instead. The
refresh never reads its own output, so this repeats a fresh ingest's
first read with the same allocations, and the family drops from 22 s to
about 5 s per sample.
Paired runs also measured every package even when both trees compiled
identical code, as on frontend-only changes. They now sample only
packages whose in-module dependency closure the change reaches. Module
files, dirty or non-Git trees and an unreadable graph still select
everything.
A module or shared-model change still samples nearly every package,
about 24 minutes with warm-up. The step therefore gets a measured 30
minutes, and the script stops at a 28-minute projected budget with
per-package costs instead of meeting a bare timeout.
An agent on a Proxmox cluster node reports SMART data for the node's
disks, and the registry stamps each disk with its node's Proxmox
identity so it stays discoverable in the Proxmox workspace. The
/api/connections member projection accepted any resource carrying that
identity, so each disk merged into its node's member row, adding its
model name to the aliases and its health to the node state.
A disk with a risk warning (for example 60C or more) therefore made an
actively reporting node show Stale. Since members are merged by
freshness, the row flipped every time the agent report and the Proxmox
poll swapped places, several times a minute on a customer's three-node
cluster running 6.4.5.
Only node resources now compose cluster members. The disk keeps its
ownership facet, risk and storage evidence unchanged.
Deploy enrollment bound the runtime token to an invented agent-<hostname>
and handed that ID to the agent. Once Pulse acknowledged the agent's reports
with its machine-derived ID, the next restart registered under that ID and
the command channel refused it for good.
Enrollment now binds the token to the node's hostname only and lets the first
command registration bind the agent's own ID. Tokens already issued with the
placeholder move once, on the exact same hostname, to the registering agent's
ID, and record that they were repaired so they never move again.
Preserve the exact reviewed Patrol guidance and VM Discovery safety commits while incorporating upstream commits captured before coordination. No source conflict resolution or product changes are added by this integration.
Change-source: pulse-maintainer
## Problem
244d322dce, merged at 15:01:08 UTC today in #2518, made organization
deletion refuse with `503 tenant_shutdown_incomplete`, and keep the
data, when the tenant's monitoring loop hasn't exited. That guard is
right, but it reused `tenantMonitorShutdownTimeout`. That 2-second
budget dates from March, when it was only a best-effort wait before
logging a warning and carrying on.
A loop that is still starting up or finishing a tick can take longer
than that to see its cancelled context. The observed case is mock mode,
which CI uses: `Monitor.Start` runs `startMockMetricsSampler`
synchronously before its select loop, and `prepareMockMetricsHistory`
seeds seven days of fixture history without a context. The loop only
sees cancellation once seeding finishes; "Mock metrics history sampler
started" and "monitoring loop stopped" log in the same second. E2E run
[37329726585](https://github.com/rcourtman/Pulse/actions/runs/37329726585/job/111833069435),
shard 7, shows it:
| Time (UTC) | Event |
|---|---|
| 15:27:57 | Organization A's monitor initializes |
| 15:28:00 | Delete requested |
| 15:28:02 | `timed out waiting for tenant monitor loop to exit` → 503 |
| 15:28:10 | `monitoring loop stopped` |
| 15:28:20 → 15:28:29 | The retry, with a new organization, repeats the
same sequence |
So `13-api-token-scope` "org-bound token stays inside the issuing org"
failed its org cleanup on both attempts. Run 37331626618 failed the same
way: the first Core E2E failure came two minutes after #2518 merged. A
real delete shortly after a tenant was busy gets the same refusal, with
its alarming "data has been retained" message, until someone retries.
## Change
- **New budget:** `removeTenant` waits on
`tenantDeletionShutdownTimeout` (30s) when deleting. Process `Stop` and
`RemoveTenant` keep the 2-second `tenantMonitorShutdownTimeout`.
- **Still refused:** a loop that never exits, and a failed metadata
close, both still refuse and retain the data. The guard's semantics
don't change.
- **Tests:** both budgets are now variables so tests can shorten them.
`TestTenantDeletionRetainsUnfinishedMonitorLoop` uses a short deletion
budget instead of the real 30 seconds.
- **No timeout conflicts:** the server's `WriteTimeout` is disabled, the
frontend's org delete has no request timeout, and Playwright's request
timeout is 30s.
## Verification
- **New test:** `TestTenantDeletionWaitsForSlowMonitoringLoop` (in the
registered `monitor_additional_test.go`) uses a loop that exits after
300ms, with a 50ms plain budget and a 2s deletion budget. Plain removal
gives up and retains the monitor; deletion waits and completes. Giving
deletion the plain budget makes it fail, so it checks the behaviour
itself.
- **Existing tests:** the deletion safety tests in `internal/monitoring`
(`TestTenantDeletion*`, `TestMultiTenantMonitor*`) and the org-delete
handler tests in `internal/api` all pass, including
`TestOrgDeleteRetainsDataWhenGuestMetadataPersistenceFails`.
**Follow-up in a separate PR:** another session is making the mock
history seeding honour cancellation
(`internal/monitoring/mock_metrics_history.go`), so CI deletions finish
almost at once instead of waiting out the seeding. This PR keeps a
realistic deletion budget either way.
In mock mode, a tenant monitor's loop only sees its cancelled context
after `startMockMetricsSampler` returns. Seeding the 50-node demo estate
takes 4-11s on CI runners. Organization deletion cancels the loop and
waits for it, so every deletion of a just-used organization was held for
the whole seed. Under the 2s budget from #2518 it was refused with
`tenant_shutdown_incomplete`, which is the intermittent Core E2E failure
in `13-api-token-scope.spec.ts:129` (runs 37329726585 and 37331626618).
In the server logs, "monitoring loop stopped" lands in the same second
as "Mock metrics history sampler started", 5-10s after the deletion's
cancel.
- `seedMockMetricsHistory` checks the context before each resource and
returns `ctx.Err()` once cancelled. Series already sampled are recorded
whole.
- `prepareMockMetricsHistory` never stores a cancelled seed as the
shared template, and returns at once if its monitor was cancelled while
it waited on the template lock.
- `startMockMetricsSampler` skips the chart-cache warm-up and the
sampler goroutine when seeding was cancelled.
Measured on the full demo estate: cancel-to-return went from the whole
seed (about 2.7s on an M-series Mac) to 0.6ms.
#2529 separately gives deletion a longer bounded wait for any slow loop.
This PR removes the mock-mode cause of the wait. They touch different
files and land independently.
New tests in `mock_metrics_history_cancel_test.go` use a context that
cancels after a set number of checks, so they don't depend on timing.
All three fail with the cancellation checks removed.
Preserve the exact documentation and regression-test candidate; native browser follow-through and the missing UI remain owned separately.
Change-source: pulse-maintainer
TestPatrolAutopilotStoreUnavailableDoesNotChangeModeOrFabricateEvidence
built its request through patrolAutopilotSessionRequest, which writes a
session into the process-global store, but never initialized one. Run
alone it panicked in GetSessionStore; in the package run it passed only
because an earlier test left a store behind.
The rest of the file got its store from newPatrolAutopilotTestHandler,
which called InitSessionStore on a t.TempDir() and never cleared it, so
these tests were also the ones leaving a store pointed at a deleted
directory for whatever ran next.
Both now go through initPatrolAutopilotSessionStore, which follows the
package's existing resetSessionStoreForTests/t.Cleanup pattern: start
from no store, initialize one on a temp dir, and clear it when the test
ends. Every test in the file passes when run alone.
A tenant monitor's loop only sees its cancelled context after
startMockMetricsSampler returns, and seeding the 50-node demo estate
took 4-11s on CI runners (about 2.7s on an M-series Mac). Organization
deletion cancels the loop and waits for it, so in mock mode every
deletion of a just-used organization was held for the whole seed, and
under the 2s budget #2518 introduced it was refused with
tenant_shutdown_incomplete (Core E2E 13-api-token-scope since 15:03 UTC
on 2026-10-05).
Seeding now checks the context before each resource and returns
ctx.Err() once cancelled, so the loop exits within a millisecond of the
cancel. Series already sampled are recorded whole. A cancelled seed is
never stored as the shared template other tenants reuse, and a monitor
cancelled while it waited on the template lock returns without seeding.
startMockMetricsSampler then skips the chart-cache warm-up and the
sampler goroutine.
#2529 separately gives deletion a longer bounded wait for any slow
loop; this removes the mock-mode cause of the wait.
Contract-Neutral: Mock-mode seeding stops early on monitor cancellation; no route, field, schema, persistence format or provider behaviour changes.
The deletion-budget note and its test said only that a loop still
starting up can be slow to stop. The CI failures have a specific
cause: in mock mode Monitor.Start runs startMockMetricsSampler
synchronously before its select loop, and prepareMockMetricsHistory
seeds seven days of fixture history without a context, so the loop
saw its cancelled context only once seeding finished ("Mock metrics
history sampler started" and "monitoring loop stopped" log in the same
second in runs 37329726585 and 37331626618). Say so in the contract
and the test comment.
TestPatrolAutopilotStoreUnavailableDoesNotChangeModeOrFabricateEvidence
built its request through patrolAutopilotSessionRequest, which writes a
session into the process-global store, but never initialized one. Run
alone it panicked in GetSessionStore; in the package run it passed only
because an earlier test left a store behind.
The rest of the file got its store from newPatrolAutopilotTestHandler,
which called InitSessionStore on a t.TempDir() and never cleared it, so
these tests were also the ones leaving a store pointed at a deleted
directory for whatever ran next.
Both now go through initPatrolAutopilotSessionStore, which follows the
package's existing resetSessionStoreForTests/t.Cleanup pattern: start
from no store, initialize one on a temp dir, and clear it when the test
ends. Every test in the file passes when run alone.
Serve a harmless 404 for browser favicon requests and bind synthetic session requests to their recorded user agent. Retain the earlier browser setup failure.
Contract-Neutral: Synthetic proof-fixture preparation only; no runtime or rule-removal instructions changed.
Change-source: pulse-maintainer
244d322dce made organization deletion refuse, and keep the data, when
the tenant's monitoring loop has not exited. That is the right guard,
but it reused tenantMonitorShutdownTimeout, a two-second budget that
had only ever been a best-effort wait before logging and carrying on.
A loop that is still starting up or finishing a tick can take longer
to see its cancelled context. In E2E run 37329726585 a just-used
organization's loop took 8-10 seconds, so both attempts of
13-api-token-scope's org cleanup got 503 tenant_shutdown_incomplete.
Any real delete shortly after a tenant was busy hits the same refusal
until a retry.
Deletion now waits on its own tenantDeletionShutdownTimeout (30s)
before it refuses. Process shutdown and RemoveTenant keep the
two-second budget, and a loop that never exits still refuses and
retains the data. Both budgets are variables so tests can shorten
them; the never-stopping-loop test no longer waits on the real budget.
Explain the missing v6.4.5 undo control and the existing scoped API. Verify exact selection, confirmation, browser session and CSRF, failure handling and readback without copying credentials or deleting unrelated rules.
Contract-Neutral: Documentation and regression-proof coverage of existing Patrol suppression APIs only; no runtime, route, authority or subsystem contract changes.
Change-source: pulse-maintainer
## Problem
An agent installed by the cluster deploy flow starts with `--enroll` and
a one-time bootstrap token. Only the host module exchanged that token
for the runtime token. Everything else `main.go` builds at startup kept
the bootstrap token for the whole first process lifetime:
- the remote config client (initial fetch and the one-minute refresh)
- the updater
- the Docker / Podman and Kubernetes modules and their primary report
targets
The bootstrap token carries only the enrollment scope and the server
removes it on exchange, so each of those failed with 401 until the
service restarted. On a freshly deployed Proxmox node this showed up as
a remote config 401 every minute. Those modules also kept the
pre-enrollment agent ID instead of the server-assigned one.
## Fix
`main.go` now enrolls once, before it builds any authenticated client:
- `hostagent.Enroll` is a standalone entry point over the existing
enrollment loop. Payload, headers, retry and backoff, status handling,
and `runtime.token` / `agent-id` persistence are unchanged.
- It runs after self-test and agent-ID preparation and before the first
remote config fetch. Every module is then built from the runtime token
and the canonical agent ID.
- The ID is persisted to the configured agent-ID file, and the host
module's own enrollment is switched off so there is no second exchange.
- A persisted runtime token is reused without a request or host
discovery. A `403` keeps the configured token as a non-bootstrap
credential. A permanent rejection stops startup before any module is
built.
- Windows is gated off and keeps its previous behaviour. Startup there
runs before the service control dispatcher, and an enrollment retry
could outlast its 30-second deadline. No Windows installer passes
`--enroll`.
- Enrollment and reporting now derive hostname, OS and architecture
through one `resolveHostIdentity`, so the runtime token is bound to the
hostname the agent reports under.
## Tests
- `TestRunEnrollsBeforeBuildingAuthenticatedModules`: enrollment happens
before any request to Pulse. Remote config, the updater, and the host,
Docker and Kubernetes configs and their targets all carry the runtime
token and canonical ID, and the agent-ID file is written. With the new
step disabled, this test fails.
- `TestRunStopsBeforeModulesWhenEnrollmentIsRejected`,
`TestRunWithoutEnrollDoesNotExchangeTokens` and
`TestRunLeavesEnrollmentToTheHostModuleWhenGatedOff`.
- `TestEnroll*` cases for `Enroll`: payload and identity, the
monitoring-only command ceiling, reuse of a persisted token (including
when host discovery fails), the 403 keep, the 409 rejection, and a
missing token.
- `TestEnrollPresentsTheIdentityNewReports` checks that enrollment
presents the hostname, OS and architecture that `New` reports, with
Proxmox detection and the architecture fallback.
- `go test ./cmd/pulse-agent ./internal/hostagent
./internal/remoteconfig` passes. `go vet` is clean for linux, windows
and freebsd.
## Not in this change
- The health listener still starts after enrollment, so `/readyz` is
unreachable while enrollment retries on network errors. The installer
only warns in that case. With the health listener enabled, its
registration check now always sees the runtime token. With it disabled,
the process-only check can still race enrollment.
- Distinguishing a deploy binding mismatch from a non-bootstrap token,
which both return 403.
- Two older durability gaps that this change doesn't make worse. If
writing a custom `--agent-id-file` fails, the next start has no fallback
to the canonical ID saved in the state directory. If persisting
`runtime.token` fails, enrollment still succeeds, and the next restart
is left with the spent bootstrap token.
Adding the Pulse Agent to a Proxmox cluster member meant generating a
token in the UI and pasting it into an interactive prompt on the host,
even though Pulse already knows the cluster and has an agent on its
siblings. The server-side cluster deploy that avoids this was still
shipped, but its wizard was deleted as an orphan in May, and it could
not reach members discovered by hostname: targets were parsed from the
API URL, so a name like "delly2" hit the agent's literal-IP guard and
failed preflight.
Deploy targets now resolve to a literal IP in the member's API URL or
its cluster endpoint's effective IP, matched on the member's own
connection, and members with no known address come back undeployable
with a reason. The Settings > Infrastructure install actions open one
dialog that checks, installs, and confirms each node is reporting,
falling back to the scoped installer when no sibling agent is
connected or a node cannot be reached.
The deploy routes are keyed on the cluster name, so they now refuse a
name shared by two connections instead of merging two sites, and job
creation admits one install per cluster at a time under a lock, which
lets the dialog keep a run going after it is closed.
The agent-lifecycle, api-contracts, frontend-primitives, and
storage-recovery contracts record the new install path, its address
rule, and the persisted, time-bounded admission check.
Supersedes #2507, which split this change across two commits; the
per-commit canonical completion guard needs the contract updates in the
same commit.
A deployed agent's first run gave the remote config client, the updater, and
the Docker / Podman and Kubernetes modules the bootstrap token, which only the
host module exchanged. Pulse revokes that token on exchange, so those clients
failed with 401 until the service restarted. Enroll once at startup and build
every module from the runtime token and the server-assigned agent ID.
The converter-only source assertion update also matched an identical infrastructure-consumer snippet. Restore that unaffected by-value assertion; only the broadcast converter borrows its local resource. Preserve both independent canonical-contract checks.
Contract-Neutral: Verification-only correction; existing infrastructure and broadcast canonical helper checks remain enforced and runtime source is unchanged.
Change-source: pulse-maintainer
The connected suite retained a shared-contract source assertion expecting by-value helper calls. The converter now borrows its local resource and dereferences at the same shared canonical type/name/cluster APIs. Match those three exact expressions without removing a shared-helper assertion or changing runtime behavior.
Contract-Neutral: Source-structure verification only; the same canonical shared helper, REST/state contract and all runtime payload and freshness behavior remain required.
Change-source: pulse-maintainer
Current-source profiling shows repeated alias derivation reserving maps for blank and duplicate optional inputs. Scan at most eight normalized keys on the stack, then fall back to a map for larger vocabularies. Preserve exact lowercasing, first spelling, order and detached allocated-empty output with the independent original map oracle, including Unicode and large identities.
Contract-Neutral: Allocation strategy only; canonical aliases, superseded IDs, identity matching, tenant/source ownership, registry invalidation, History and broadcast wire contracts are unchanged.
Change-source: pulse-maintainer
Review of the nil fast path identified named nil collections that implement encoding.TextMarshaler rather than json.Marshaler. Keep the parent representation for both interfaces and retain matching byte-level controls.
Contract-Neutral: Conversion work only; retain the exact existing JSON and text-marshaler representations and all broadcast fields and freshness boundaries.
Change-source: pulse-maintainer
Borrow the converter-owned resource across field helpers and bypass encoding typed absent facets as discarded null. Preserve named nil collection marshalers, errors, detached output, all payload fields, metadata refresh, catalogues and same-generation History targets. Validate complete projections and encoded frames against the exact parent.
Contract-Neutral: Conversion work only; broadcast and REST payloads, identities, tenant routing, metadata and health freshness, alerts, catalogue references and History coordinates are unchanged.
Change-source: pulse-maintainer
Capture all endpoint limiters under their Router owner and seal late worker
admission. Concurrent shutdown joins each cleanup worker without resetting
request budgets or stopping another Router. Integration fixtures use the real
shutdown lifecycle before monitor teardown.
Contract-Neutral: Router worker ownership and concurrent shutdown only; HTTP payloads, limits and windows, auth, tenant selection, revocation and persistence contracts are unchanged.
Change-source: pulse-maintainer
The five predeclared diagnostic cases retain a real 1m versus 2m health-age counterexample between separate wall-clock evaluations. Give the independent content oracle and production projection the same private health clock without removing any JSON field; pin minute-boundary age advancement and original LastSeen. Production samples real UTC at the unchanged post-read evaluation point.
Contract-Neutral: Internal health-clock test seam preserves the real UTC evaluation point, public resource/history fields, source precedence, agent identity/lifecycle, tenant/revocation and safety contracts; no wire or permission delta.
Change-source: pulse-maintainer
The new complete-content oracle exposed a typed-view discrepancy. Keep that failed result and report the first differing JSON field, rather than accept an undifferentiated mismatch or normalise away evidence. Only the synthetic fixture failure message changes; equality assertions and production source are unchanged.
Change-source: pulse-maintainer
Adding the Pulse Agent to a Proxmox cluster member meant generating a
token in the UI and pasting it into an interactive prompt on the host,
even though Pulse already knows the cluster and has an agent on its
siblings. The server-side cluster deploy that avoids this was still
shipped, but its wizard was deleted as an orphan in May, and it could
not reach members discovered by hostname: targets were parsed from the
API URL, so a name like "delly2" hit the agent's literal-IP guard and
failed preflight.
Deploy targets now resolve to a literal IP in the member's API URL or
its cluster endpoint's effective IP, matched on the member's own
connection, and members with no known address come back undeployable
with a reason. The Settings > Infrastructure install actions open one
dialog that checks, installs, and confirms each node is reporting,
falling back to the scoped installer when no sibling agent is
connected or a node cannot be reached.
The deploy routes are keyed on the cluster name, so they now refuse a
name shared by two connections instead of merging two sites, and job
creation admits one install per cluster at a time under a lock, which
lets the dialog keep a run going after it is closed.
The agent-lifecycle, api-contracts, frontend-primitives, and
storage-recovery contracts record the new install path, its address
rule, and the persisted, time-bounded admission check.
The full current-state regression now requires a critical alert to reach the same freshly projected row. Continuity controls compare a healthy VM, critical transition and clear instead of using an already-offline host as a vacuous health control.
Change-source: pulse-maintainer
The independent full-state oracle retains the parent unified view watermark. The paired benchmark preprocesses active health alerts once per pass, like the exact parent, rather than adding per-row work to the baseline. Production projection and source fields are unchanged.
Change-source: pulse-maintainer
Resolve canonical source mappings once per complete projection while preserving detached resource content and target selection after host coalescing. Bind resources and targets to the same locked generation, including absence and continuity overlays, and avoid another owned outer-slice copy. No timestamp cache or native CPU/RSS claim. Retain cold, clean and dirty full-content controls and paired complete-projection measurements.
Contract-Neutral: Internal bulk-read and copy-ownership repair preserves existing public resource/history fields, source precedence, agent identity/lifecycle, tenant/revocation and safety contracts; no wire or permission delta.
Change-source: pulse-maintainer
The comment in AddClient said a broadcast racing a connect is either in
the snapshot or queued behind it. It can be both, so the client then
receives that status twice, possibly with an older in-flight stage in
between. Trusting the "either" is what led the connect-snapshot test to
assume nothing writes the recorder after AddClient returns, which made
it flaky under -race. Say what actually happens and why it is harmless:
stage progress only rises and the update modal ignores backwards stages.
Contract-Neutral: Comment-only correction in internal/updates/sse.go; no runtime or contract change.
TestSSEBroadcaster_AddClientWritesCurrentStatusBeforeReturning failed
under -race in CI on unrelated frontend PRs (2479, 2485). The test
broadcast a stage and connected straight away. When the broadcast loop
picked that message up after the client had registered, it queued the
stage for the new client too, so the client's writer goroutine wrote
the httptest recorder while the test read it without a lock.
AddClient itself is correct: it writes the preamble and current status
before returning, and a stage racing the connect can reach the client
twice, which the update modal already ignores. The test now lets the
broadcast reach an earlier client before connecting, so nothing else
writes the recorder and the status in the body can only come from the
synchronous snapshot write.
Reproduced by injecting the CI schedule through -overlay (broadcast
loop held until the client registers, caller paused as AddClient
returns): the old test failed 199/200 runs with DATA RACE, the new one
0/200. A mutant that moves the snapshot write onto the writer goroutine
still fails the new test 100/100.
Keep admission sealed and reuse a still-running close observer. A later explicit shutdown attempt may persist retained state after a completed write failure; successful close remains idempotent. This avoids a permanent offboarding dead end after an ordinary filesystem repair without an automatic spin retry or overlapping writes.
Contract-Neutral: Internal JSON writer recovery behaviour only; public routes, fields, permissions, identity format, private file mode and atomic persistence are unchanged.
Change-source: pulse-maintainer
Admit detached metadata in order, merge identity fields without restoring stale
operator edits, and persist complete snapshots with one owned writer. Seal late
writes and return shutdown failures before organisation data or offboarding is
removed. Preserve existing drain budgets, file format, tenant boundaries and
atomic persistence. Add controlled burst, ordering, write-error, blocked-close
and real deletion controls.
Contract-Neutral: Behavioural repair of existing JSON persistence and organisation deletion; no route, field, permission, identity format, schema or provider authority change. Shutdown failure uses the existing HTTP error envelope. Internal ownership methods and explanatory contract text change, not the public contract shape.
Change-source: pulse-maintainer
Preserve the reviewed candidate commits as the second parent and retain canonical one-shot, redirect, partial-response and disclosure controls in the overlapping API/PBS smoke fixtures. Keep the candidate's canonical PBS guard, scoped private temperature-file assertions and unchanged-verdict audit diagnostics.
Contract-Neutral: Integration of test-only private-help adapters; no runtime, guides, dependencies or frontend source changed.
Change-source: pulse-maintainer
Retain the three-sample baseline-plus-ten bound in an isolated copy of the same test binary. Add full-publication copying controls and exact complete admission-content parity, rather than attributing independent registry allocation variation to ignored metadata.
Contract-Neutral: Test measurement correction only; no runtime, identity, freshness, publication or allocation-limit contract changes.
Change-source: pulse-maintainer
Preserve exact Core candidate ab15c4db8a and its two prerequisite commits. Retained connected proof and complete cost samples support allocation and scaling gains, not native CPU/RSS relief.
Change-source: pulse-maintainer
Compose exact upstream a82a8e0474 additively without changing accepted commits. Resolve the append-only registry-test conflict by retaining both the reviewed metadata invalidation control and upstream stale-health control. Existing PR2486 remains unchanged while normal source-check corrections are pending.
Change-source: pulse-maintainer
The Coverage tooltip and expansion render the posture explanation
verbatim, and it read like the model: "a current subject-linked
backup", "older than the configured freshness window", "a newer
provider failure", "a stronger protection claim".
Each explanation now says what Pulse found in plain words and stays true
to its branch: the stale case names the window ("the newest confirmed
backup is older than 7 days"), the failure case no longer claims a
timing it cannot know, and partial history without a backup no longer
says a backup exists. The window formatter never rounds a sub-hour or
fractional window up. The frontend fallbacks for checking, unrated and
unknown rows and the snapshot-only tooltip use the same voice.
Repair deterministic governance and script-smoke failures against the reviewed guides without restoring public response output, literal credential commands or clobbering downloads. Exercise the actual shared API helper and independently verify private, unique response files, methods, bodies and hostile curl defaults.
Contract-Neutral: Update test adapters for the existing reviewed credential-private guide contract; no runtime or guide change.
Change-source: pulse-maintainer
rcourtman