The non-required **Benchmarks** check has failed on every non-superseded
PR since about 15:00 UTC on 5 October. Examples are #2521, #2520, #2524,
#2525, #2527, #2530 and #2532, each ending with `Paired base and
candidate benchmarks (pull requests)` timed out after 25 minutes. The
paired step runs 10 base and 10 candidate samples, so each second added
per sample costs 20 seconds of step time.
### What made it slower
Paired step duration across PR runs, with per-package seconds per sample
taken from the `ok` lines in the job logs:
| Period | Paired step | `internal/unifiedresources` |
`internal/monitoring` |
|---|---|---|---|
| 24 Sep – 5 Oct 08:00 | 16–18 min | 7.9 s | 4.5 s |
| after #2486 (08:11) | 20.5–20.9 min | 16.2 s | 5.0 s |
| after #2518 (15:01) | killed at 25 min | 38.2 s | 9.9 s |
- **#2518** added `BenchmarkRegistryProjectionSnapshot`. Its `cold` mode
rebuilt the registry between `b.StopTimer` and `b.StartTimer` on every
iteration. Go picks `b.N` from timed work only, so the empty-registry
cases ran hundreds of thousands of untimed ingests. Per sample,
`/0/cold/point-targets` took 10.3 s and `/0/cold/bulk-targets` 5.9 s at
a 100 ms benchtime. That is the cause of the timeouts.
- **#2486** added 46 owner-index and metadata sub-benchmarks, about +8 s
per sample. That is legitimate coverage. #2518's
`BenchmarkBroadcastMetricsSnapshot` and #2527 added about +6 s more.
- `-count`, `-benchtime` and the package list did not change.
### Fix
1. **Root cause.** `cold` now invalidates the registry's derived state
(canonical metadata, typed views, source-target index) instead of
rebuilding the registry. The refresh never reads its own output, so this
repeats a fresh ingest's first read: at 1000 VMs, 40011 vs 40014
allocs/op and about 5.09 MB/op, the same as before. Locally the 24-case
family now runs in 4.9 s; before, its 6 cold cases alone took 33.8 s.
2. **Scope.** Paired PR runs now sample only packages whose in-module
dependency closure (`go list -deps -test`) holds a changed file. A
`_test.go` file selects only its own package. Changes to
`go.mod`/`go.sum`, dirty or non-Git trees, or an unreadable graph select
every package. Applied to today's failures: #2521, #2524 and #2525
sample nothing; #2530 samples `internal/api`; #2532 samples api and
monitoring; #2520 samples 5 packages; #2527 samples 7. The metadata
records `selection=` and `packages=`.
3. **Budget.** A module or shared-model change still samples nearly
everything, about 24 minutes including the ~2.7 min warm-up. The step
timeout is now a measured 30 minutes (job stays at 35).
`PULSE_BENCH_BUDGET_SECONDS=1680` makes the script project its total
after each round. Once the projection reaches 28 minutes it stops and
prints each package's mean seconds per sample, which replaces an
unexplained timeout. `TestPairedBenchmarksStopInsideTheirBudget` keeps
budget < step timeout < job timeout.
4. **Store-backed chart benchmarks** (second commit). Go calls a
benchmark function again for each `b.N` it tries.
`HandleWorkloadCharts_StoreBacked` and
`HandleWorkloadsSummaryCharts_StoreBacked` each seeded 84,000 points
into a fresh SQLite store (about 0.6 s) to time a request of 0.2 ms or
less, two or three times per sample. In CI that was 3.5 s and 3.6 s of
internal/api's 11.5 s per sample. The timed loop now runs in a
`range=4h` sub-benchmark, so the parent seeds once per run. Data,
request and bytes/allocs per op are unchanged, and locally the pair
falls from about 6 s to 1–1.5 s. The names gain `/range=4h`, so this
PR's comparison reports them unmatched.
Unpaired branch runs still sample every package. With the cold fix,
their 15-minute step (13–14.2 min today) regains about 3 minutes of
headroom.
### Verification
- `scripts/tests/test-ci-benchmarks.sh` covers no-op, own-test-only,
shared-dependency, unreadable-graph, dirty-tree, module-file,
budget-exceeded, in-budget and invalid-budget cases. Five mutations of
the selection and budget logic are each caught.
- On the real package graph, selection maps `internal/models` to 7
packages, `internal/monitoring` to api+monitoring, an `internal/api`
embed to api, and frontend or docs paths to none.
- End to end with real Go on two clean clones (`origin/main` vs this
change), only `./internal/unifiedresources/` was sampled. Its per-sample
time went from 53.9 s to 26.7 s on a loaded Mac.
- `go test ./scripts/installtests -run
'TestPairedBenchmarksStopInsideTheirBudget|TestBenchmarkQualificationRetainsProvenance'`,
plus the script-reference and doc-drift suites.
This PR's own Benchmarks run should sample only
`internal/unifiedresources` and `internal/api`, because both changes are
those packages' own test files.
Retain exact proof-list equality while including the disk-History scope proof already registered in the assigned base. Require the connected pool test to exercise its host-availability check and both independent owning identities, not merely count resource rows.
Contract-Neutral: Update regression expectations and assertion coverage only; runtime, registry and subsystem contracts are unchanged.
Change-source: pulse-maintainer
Retain exact Delivery commits and current main repairs. Reviewed source-1dqqprfr verifies offline codec and Kubernetes races, runtime checks, vet and real server/agent builds with unchanged manifests. No frontend content is resolved or changed by this integration.
Change-source: pulse-maintainer
Since 15:00 UTC on 5 October every paired benchmark run that was not
superseded has hit its 25-minute step timeout. Each step runs ten base
and ten candidate samples, so per-sample growth is multiplied by twenty.
BenchmarkRegistryProjectionSnapshot (#2518) rebuilt a registry behind
StopTimer on every cold iteration. Go sizes b.N from timed work only, so
the empty-registry cases ran hundreds of thousands of untimed ingests
and took 16 s of a 100 ms benchtime. With #2486's new owner-index and
metadata benchmarks, internal/unifiedresources went from 8 s to 38 s per
sample. Cold now invalidates the registry's derived state instead. The
refresh never reads its own output, so this repeats a fresh ingest's
first read with the same allocations, and the family drops from 22 s to
about 5 s per sample.
Paired runs also measured every package even when both trees compiled
identical code, as on frontend-only changes. They now sample only
packages whose in-module dependency closure the change reaches. Module
files, dirty or non-Git trees and an unreadable graph still select
everything.
A module or shared-model change still samples nearly every package,
about 24 minutes with warm-up. The step therefore gets a measured 30
minutes, and the script stops at a 28-minute projected budget with
per-package costs instead of meeting a bare timeout.
Retain the reviewed four-module patch candidate, current disk History, isolated partial-PVE and Patrol changes. Validate this exact composition against the prepared dependency graph before offering source integration.
Change-source: pulse-maintainer
Promote v6.5.0-rc.1 to stable after its 24-hour soak. Only release metadata changes.
Contract-Neutral: Release-only v6.5.0 version, release notes and distribution metadata; no public install or runtime contract changes.
Change-source: pulse-maintainer
Explain exact rule scope, confirmation and uncertain-result readback in Patrol Activity. Keep the tested administrator-session workaround only for older builds without the controls; denied access is not a reason to use it. Bind help labels to the production controls and preserve the shipped mirror.
Contract-Neutral: Documentation and help regression coverage only; existing Patrol UI, API, suppression semantics, authentication, organisation scope and CSRF contracts are unchanged.
Change-source: pulse-maintainer
Remove log filters that hide reader failures and unrelated startup errors. Use current export labels and separate cached downloads from live diagnostics, preserving private evidence and the backup safety boundary.
Change-source: pulse-maintainer
Update the two fixed expected proof lists for the connected TrueNAS disk History tests already registered by the candidate. Preserve full-list equality, all existing proof paths and rejection controls; do not change guard implementation or runtime behaviour.
Contract-Neutral: Test expectations match existing registered proof paths; no guard, registry or runtime semantics change.
Change-source: pulse-maintainer
Preserve the held four-patch dependency repair and all current main monitoring and UI changes. The two earlier acquisition attempts failed before execution at capacity; this current-source composition still requires exact graph acquisition, fresh launch selection and runtime/build proof before integration.
Change-source: pulse-maintainer
Preserve current upstream guest-read, mock cancellation, tenant deletion and Patrol repairs while carrying the prepared partial-report candidate through final-source validation. No isolation or platform gate is relaxed.
Change-source: pulse-maintainer
Preserve the exact documentation and regression-test candidate; native browser follow-through and the missing UI remain owned separately.
Change-source: pulse-maintainer
Replace the rejected global host-network proposal with the unchanged private namespace on every platform, including native linux-386. Carry successful rows and omitted VMIDs through operation v2 without clearing degraded health or renewing omitted measurements; retain strict complete-only v1 compatibility. Restart the installer-owned helper only after isolated-unit and socket checks, and document the unresolved pmxcfs IPC limitation rather than claiming PVE recovery.
Change-source: pulse-maintainer
Explain the missing v6.4.5 undo control and the existing scoped API. Verify exact selection, confirmation, browser session and CSRF, failure handling and readback without copying credentials or deleting unrelated rules.
Contract-Neutral: Documentation and regression-proof coverage of existing Patrol suppression APIs only; no runtime, route, authority or subsystem contract changes.
Change-source: pulse-maintainer
## Problem
An agent installed by the cluster deploy flow starts with `--enroll` and
a one-time bootstrap token. Only the host module exchanged that token
for the runtime token. Everything else `main.go` builds at startup kept
the bootstrap token for the whole first process lifetime:
- the remote config client (initial fetch and the one-minute refresh)
- the updater
- the Docker / Podman and Kubernetes modules and their primary report
targets
The bootstrap token carries only the enrollment scope and the server
removes it on exchange, so each of those failed with 401 until the
service restarted. On a freshly deployed Proxmox node this showed up as
a remote config 401 every minute. Those modules also kept the
pre-enrollment agent ID instead of the server-assigned one.
## Fix
`main.go` now enrolls once, before it builds any authenticated client:
- `hostagent.Enroll` is a standalone entry point over the existing
enrollment loop. Payload, headers, retry and backoff, status handling,
and `runtime.token` / `agent-id` persistence are unchanged.
- It runs after self-test and agent-ID preparation and before the first
remote config fetch. Every module is then built from the runtime token
and the canonical agent ID.
- The ID is persisted to the configured agent-ID file, and the host
module's own enrollment is switched off so there is no second exchange.
- A persisted runtime token is reused without a request or host
discovery. A `403` keeps the configured token as a non-bootstrap
credential. A permanent rejection stops startup before any module is
built.
- Windows is gated off and keeps its previous behaviour. Startup there
runs before the service control dispatcher, and an enrollment retry
could outlast its 30-second deadline. No Windows installer passes
`--enroll`.
- Enrollment and reporting now derive hostname, OS and architecture
through one `resolveHostIdentity`, so the runtime token is bound to the
hostname the agent reports under.
## Tests
- `TestRunEnrollsBeforeBuildingAuthenticatedModules`: enrollment happens
before any request to Pulse. Remote config, the updater, and the host,
Docker and Kubernetes configs and their targets all carry the runtime
token and canonical ID, and the agent-ID file is written. With the new
step disabled, this test fails.
- `TestRunStopsBeforeModulesWhenEnrollmentIsRejected`,
`TestRunWithoutEnrollDoesNotExchangeTokens` and
`TestRunLeavesEnrollmentToTheHostModuleWhenGatedOff`.
- `TestEnroll*` cases for `Enroll`: payload and identity, the
monitoring-only command ceiling, reuse of a persisted token (including
when host discovery fails), the 403 keep, the 409 rejection, and a
missing token.
- `TestEnrollPresentsTheIdentityNewReports` checks that enrollment
presents the hostname, OS and architecture that `New` reports, with
Proxmox detection and the architecture fallback.
- `go test ./cmd/pulse-agent ./internal/hostagent
./internal/remoteconfig` passes. `go vet` is clean for linux, windows
and freebsd.
## Not in this change
- The health listener still starts after enrollment, so `/readyz` is
unreachable while enrollment retries on network errors. The installer
only warns in that case. With the health listener enabled, its
registration check now always sees the runtime token. With it disabled,
the process-only check can still race enrollment.
- Distinguishing a deploy binding mismatch from a non-bootstrap token,
which both return 403.
- Two older durability gaps that this change doesn't make worse. If
writing a custom `--agent-id-file` fails, the next start has no fallback
to the canonical ID saved in the state directory. If persisting
`runtime.token` fails, enrollment still succeeds, and the next restart
is left with the spent bootstrap token.
Update only compress and the Kubernetes trio from proposal 2317. Preserve other dependency and compiler pins and retained checksums. Require aligned patch floors, paired checksum records and exact-graph runtime proof before integration.
Contract-Neutral: Dependency patches and internal graph admission controls only; module names, public APIs, deployment interfaces, managed-runtime orchestration and acquisition security boundaries are unchanged.
Change-source: pulse-maintainer
Every subsystem must declare require_explicit_path_policy_coverage, so
build_verification_requirements never falls back to a rule's default
verification policy; an unmatched runtime file becomes a path-policy gap
instead. The default test_prefixes and exact_files were therefore never
consulted, yet 13 of 16 rules carried them (93 files and one prefix),
and authors kept adding proofs there as if they counted.
92 of the 93 files were already proof on some path policy, and
api-contracts' default prefix is on three of its own policies. The one
orphan, internal/api/router_bench_test.go, benchmarks the api-contracts
router rather than any performance-and-scalability runtime, so no policy
should take it; it stays an owned file only.
The registry drops the lists, registry_audit requires them empty and
points authors at the path policy instead, and the canonical protocol
says so. The protocol also stops promising same-subsystem tests "when
the registry explicitly allows them", which the audit no longer permits.
No verdict changes: build_verification_requirements returns identical
requirements for all 2746 owned runtime paths, and for an uncovered
path, under the old and new registry.
Document host-capacity normalisation, version-qualified rounding and the difference between collected readings and delivered alerts. Keep comparisons on ordinary workload observations rather than forced load, threshold changes or diagnostic reruns.
Contract-Neutral: Existing container CPU semantics and safety guidance only; no runtime, API, threshold, permission or lifecycle contract changes.
Change-source: pulse-maintainer
staged_verification_files_for_requirement() matches
allow_same_subsystem_tests against a subsystem's
owned_prefixes/owned_files, but both commit guards passed it the
impacted entry, which carried neither. Since the v6 import (778a2577b6)
a same-subsystem test never satisfied a requirement, while
subsystem_lookup.py (which passes the registry rule) and the guard's own
BLOCKED message ("same-subsystem test/spec files are also accepted")
said it would.
Honouring the 12 flags would have loosened the guard well past their
apparent intent. Every rule requires explicit path-policy coverage, so
the five rule-level flags are never consulted, and on the seven path
policies the flag reaches every test the subsystem owns: a
mutation-registry change would accept any internal/ai or AI Chat test
(~455) instead of its 4 listed proofs, and a frontend Actions change any
Go test under internal/unifiedresources. Six of the seven already list
their own directory in test_prefixes.
So the strictness stays and the tools now agree on it: the registry sets
the 12 flags to false (what has always been enforced), registry_audit
rejects true and points at test_prefixes/exact_files, and the impacted
entry carries the rule's ownership so the completion and commit-shape
guards read the flag the same way subsystem_lookup does.
No requirement accepts a proof it rejected before. subsystem_lookup
stops reporting matches the guard refused: a new internal/hostagent test
no longer claims three agent-lifecycle policies.
The completion guard's CI mode now compares contract text against the
evaluated commit (#2502), but it still decided which contracts a commit
owed from HEAD's registry and contract index. CI walks a branch with HEAD
at the tip (or at main after a merge), so a later commit could change an
earlier commit's verdict. If a later commit added a Shared Boundaries
reference to a file, an earlier commit that was complete when written got
blocked. If a later commit dropped ownership of a file, an earlier commit
that changed it without its contract passed. The pre-commit hook read
both from the commit's own index, so CI and the hook could still
disagree.
repo_file_io and subsystem_contracts can now read from a revision as well
as the working tree or index. A revision read uses only that commit's
tree and fails closed rather than falling back to the disk. In diff-base
mode the guard reads the registry and contract index from the evaluated
commit, as does the Prettier-only inference that --commit enables.
Replaying CI's per-commit loop over the last 120 commits on main gives
the same verdicts as before.
Expose only the installed GNU gtimeout as timeout through a new runner-owned directory. Keep native BSD utilities on their original PATH so installing the missing dependency cannot silently change the other platform checks. The connected workflow test verifies the symlink identity and that the directory contains no other utilities.
Contract-Neutral: Narrow the native CI dependency PATH and its proof; no shipped installer or runtime contract changes.
Change-source: pulse-maintainer
Both macOS native jobs fail the copied temperature recipes before reading any sensor. Install coreutils into the workflow PATH and verify GNU timeout for every Unix runner. Exercise the actual workflow snippets with real GNU timeout and fail-closed install, missing-tool and unsupported-tool controls; leave the recipes and their TERM/KILL/privacy controls unchanged.
Contract-Neutral: Native CI prerequisite and its fixture coverage only; no installer, runtime, API, permission or diagnostic contract changes.
Change-source: pulse-maintainer
CI and dev-prepush walk a branch's commits with HEAD at the branch tip and
run the completion guard per commit with --diff-base <commit>^. The guard
then compared the parent's contract text against HEAD rather than against
the commit it was evaluating, so every commit was judged by the whole
branch. The pre-commit hook compares HEAD with the staged index, which is
the commit's own parent-to-commit change, so the two disagreed.
PR 2494 hit this: its second commit reverted a Current State paragraph in
performance-and-scalability.md that its first commit wrote. Parent-vs-HEAD
saw no change, and CI blocked a commit the hook had correctly passed. The
same comparison lets a bare runtime commit through when a later commit in
the branch carries the contract update.
When --commit is given, the guard now resolves the diff base and reads the
new contract text from that commit. CI already passes --commit, and
dev-prepush now does too. Without --commit the guard still compares with
HEAD, which is right when the evaluated commit is checked out.
Match QNAP guidance to existing data-volume relocation, create private per-attempt staging and preserve diagnostic logs and identity. Exercise copied commands for preparation and installer failures without an appliance operation.
Contract-Neutral: Clarifies existing installer storage and private staging guidance; no runtime, permission or data-contract change.
Change-source: pulse-maintainer
Preserve the exact reviewed documentation candidate and its retained privacy/query proof; no runtime or storage changes.
Change-source: pulse-maintainer
Reuse the tested API helper, explain point/source and access distinctions, and preserve data before storage tuning. Copied examples now exercise private success/error responses, redirects, incomplete transport and encoded IDs.
Contract-Neutral: Documentation and recipe checks align with existing history API behaviour; no runtime, storage, entitlement or authentication contract changes.
Change-source: pulse-maintainer
Preserve the reviewed candidate commits as the second parent and retain canonical one-shot, redirect, partial-response and disclosure controls in the overlapping API/PBS smoke fixtures. Keep the candidate's canonical PBS guard, scoped private temperature-file assertions and unchanged-verdict audit diagnostics.
Contract-Neutral: Integration of test-only private-help adapters; no runtime, guides, dependencies or frontend source changed.
Change-source: pulse-maintainer
The reviewed phase-aware runner and PBS helper exclude Windows. Apply the same build constraint to their existing Bash temperature consumers so Windows test compilation does not retain unresolved runPBSRecipe references. Linux test content, guides and runtime are unchanged.
Contract-Neutral: Align an existing Bash-only test file with its non-Windows test helper; no runtime, guide, dependency or platform contract change.
Change-source: pulse-maintainer
A bounded synthetic comparison shows the whole-shell watchdog can kill a POST before its own curl deadline or expire before a PBS request starts. Observe each phase privately, preserve curl arguments and exact request limits, cancel owned descendants and retain incomplete pipe results. Exercise delayed preparation, one-shot transport, private responses and every cancellation phase without changing any guide or runtime.
Contract-Neutral: Repair Bash documentation test timing and owned-child observation only; no guide, mirror, runtime, API, authentication, request deadline, dependency or permission contract changes.
Change-source: pulse-maintainer
Bind the new 0600 assertion only to the reviewed temperature cleanup helper rather than impose it on the unchanged public Unified Agent installer recipe. Include the captured canned audit output when its existing one-call assertion fails, preserving verdicts, call counts and timing controls. The complete second proof retains its fail-closed two-call observation without a cause or successful-retry claim.
Contract-Neutral: Test adapter scope and failure diagnostics only; existing guide, audit verdict and timing contracts are unchanged.
Change-source: pulse-maintainer
Keep the shared direct-curl recorder compatible with the other existing guides while binding API requests to its reviewed private helper. Update the temperature selector to the reviewed fail-closed temporary download and retain disable-first TLS and mode checks. The first broad proof caught both adapter faults before publication.
Contract-Neutral: Test-only adapters for existing reviewed guide contracts; no runtime, guide, dependency or frontend change.
Change-source: pulse-maintainer
Repair deterministic governance and script-smoke failures against the reviewed guides without restoring public response output, literal credential commands or clobbering downloads. Exercise the actual shared API helper and independently verify private, unique response files, methods, bodies and hostile curl defaults.
Contract-Neutral: Update test adapters for the existing reviewed credential-private guide contract; no runtime or guide change.
Change-source: pulse-maintainer
Execute the actual API helper and PBS private-file/download handoffs, keeping their response-privacy assertions separate from shared credential-transport fixtures. Verify one-shot requests, exact methods and JSON, retained private responses and failure exits, TLS and preparation. Match the temperature cleanup test to its current non-clobbering download, without changing any guide.
Contract-Neutral: Repair connected documentation fixtures only; no guide, mirror, runtime, API, authentication, dependency or permission contract changes.
Change-source: pulse-maintainer
Replace automatic hardware detection with existing-driver reads, keep SSH forced-key trust intact, and prevent incomplete legacy cleanup downloads from replacing a saved helper. Validate the copied commands against synthetic tools and HTTPS peers.
Contract-Neutral: Existing help and copied-command safety only; sensor collection, authentication, installer interfaces and runtime schemas are unchanged.
Change-source: pulse-maintainer
Bound the existing passive preflight before Bash can discard NULs or accept unlimited configuration output. Preserve producer errors independently, isolate the byte reader from Python startup hooks, and keep the copied helper and mirrored help private and off the guest-agent channel.
Contract-Neutral: Operator diagnostics and existing help only; Proxmox polling, guest-agent coordination, APIs, storage and service behaviour are unchanged.
Change-source: pulse-maintainer
Save every existing reference response privately, including errors and partial POST results. Preserve private header files, reject unsafe preparation and URLs, and stop on redirects or transport errors without retries. Exercise all eight copied GET/POST examples against real synthetic HTTP/TLS peers, including the old response exposure.
Contract-Neutral: Harden existing API documentation recipes and their shipped mirror with connected tests; no runtime, endpoint, authentication, scope, action, dependency or permission contract changes.
Change-source: pulse-maintainer
Retain the guide recipe array before selecting its request slice; the initial exact-source proof rejected a temporary array slice before running any tests.
Contract-Neutral: Correct test compilation only; existing PBS documentation and runtime contracts are unchanged.
Change-source: pulse-maintainer
Fail closed before editing unsafe credential paths, ignore curl configuration and retain diagnostic responses privately. Bound requests, reject redirects and publish the local installer path only after a complete HTTPS 200 download without overwriting existing files. Exercise the copied recipes against synthetic TLS peers and file failures.
Contract-Neutral: Harden existing PBS documentation recipes and their shipped mirror; no runtime, API, permission, installation contract or dependency change.
Change-source: pulse-maintainer
PR2472 build run 37239183545 extracts no wrapped TrueNAS recipes; exercise all three current blocks and retain synthetic HTTP response bodies only in private files, including 201 save and error controls. Native run 37239183481 expects a Linux QGA capability on macOS and Windows; assert non-Linux agents do not advertise it and scope the connected PVE fixture to its actual Linux runtime. Production safety and documentation remain unchanged.
Contract-Neutral: Test-only reconciliation of already-reviewed private-response recipes and existing Linux-only guest-execution capability; no product contract change.
Change-source: pulse-maintainer
Source proofs can supply unrelated input files. Retain recipe mutation controls while reading the exact parent only when its named guide was supplied; unrelated input directories must not become a new full-suite prerequisite.
Contract-Neutral: Test-only optional-input handling; no shipped guidance or product contract change.
Change-source: pulse-maintainer
pulse-triage[bot]