Since 15:00 UTC on 5 October every paired benchmark run that was not
superseded has hit its 25-minute step timeout. Each step runs ten base
and ten candidate samples, so per-sample growth is multiplied by twenty.
BenchmarkRegistryProjectionSnapshot (#2518) rebuilt a registry behind
StopTimer on every cold iteration. Go sizes b.N from timed work only, so
the empty-registry cases ran hundreds of thousands of untimed ingests
and took 16 s of a 100 ms benchtime. With #2486's new owner-index and
metadata benchmarks, internal/unifiedresources went from 8 s to 38 s per
sample. Cold now invalidates the registry's derived state instead. The
refresh never reads its own output, so this repeats a fresh ingest's
first read with the same allocations, and the family drops from 22 s to
about 5 s per sample.
Paired runs also measured every package even when both trees compiled
identical code, as on frontend-only changes. They now sample only
packages whose in-module dependency closure the change reaches. Module
files, dirty or non-Git trees and an unreadable graph still select
everything.
A module or shared-model change still samples nearly every package,
about 24 minutes with warm-up. The step therefore gets a measured 30
minutes, and the script stops at a 28-minute projected budget with
per-package costs instead of meeting a bare timeout.
Record executable SHA-256 before measured Go test invocations without logging paths or arguments. Preserve sample ordering and failure status; document instrumentation limits rather than dismissing historical regressions.
Change-source: pulse-maintainer
PR #1933 failed paired normalization benchmarks despite unchanged hot-path source. Preserve actual checkout identities, selected toolchains and sample order so investigation does not confuse PR head metadata with measured source. Keep thresholds and sample collection unchanged.
Change-source: pulse-maintainer
The benchmark gate compared five-sample PR results with a cache produced on
another hosted VM. Two unrelated changes failed today while the same main code
passed, and benchstat reports infinite 95% confidence intervals for that sample
size.
Collect ten base and candidate samples on the PR runner in alternating order,
retain both inputs and the comparison, and reject under-sampled verdicts. Keep
non-PR benchmark evidence without the cross-run baseline cache.
Contract-Neutral: CI performance evidence collection only; no product or release contract changes
Change-source: pulse-maintainer
rcourtman