* ci(perf): fit the baseline cache build to the post-LTO profile and self-heal nightly misses
Every build-baseline-cache run on 2026-07-15 died on its 60min ceiling
('exceeded the maximum execution time of 1h0m0s'): #4806 put thin LTO +
codegen-units=1 on [profile.release], pushing a single release build past
60min on sm-standard-2, so the baseline cache never populated. The measured
binary must keep the production profile, so raise the job budget to 100min
instead of weakening the profile.
Also make the A/B job self-heal a same-commit cache miss: when the candidate
commit equals the baseline commit (nightly/dispatch on main) and the restore
misses, build the binary once, reuse it for both phases, and save it back to
the cache under rustfs-baseline-<sha> — previously that miss made the rig
build the identical commit twice, which no job budget fits post-#4806. The
PR-context miss (candidate != baseline, opt-in gate) keeps the double-build
fallback and now documents that it will overrun and alert.
Refs rustfs/backlog#1152 (perf-3 follow-up).
* ci(perf): latest-wins concurrency for the baseline cache build
Consumers only restore the binary for the current origin/main tip, so a
superseded push build's output is dead weight; cancel it instead of stacking
~65min jobs on the shared sm-standard-2 pool when main merges quickly. A
skipped intermediate SHA at most costs one same-commit self-heal in the A/B
job.
Every push to main now builds the release binary once and stores it in the
actions cache as rustfs-baseline-<sha> (build-baseline-cache job). The A/B job
restores it for origin/main and passes --baseline-bin; on the nightly, where
the candidate commit equals the baseline, one cached binary serves both phases
with --skip-build and the run does zero source builds. A cache miss falls back
to the source double-build via --baseline-ref origin/main.
This removes the ~32min-per-side double build that pushed the 24-cell nightly
past its 120min ceiling (2026-07-11..07-14 all cancelled on timeout).
Also:
- alert-on-failure now fires on cancelled/timed-out, not just failure, so a
timed-out nightly is no longer silent (the composite action already reports
cancelled jobs); removed the stale perf-2 TODO now that the alert job exists.
- run_hotpath_warp_ab.sh appends a Provenance section to gate.md (baseline and
candidate SHAs + binary source, runner, warp version, matrix params) via a
repeatable --provenance-note; the gate exit code is preserved.
Refs rustfs/backlog#1152 (perf-3).
Scheduled workflow failures previously went unnoticed (15 consecutive red
weekly s3-tests sweeps, silent perf nightly failures). Add a reusable
composite action that opens a tracking issue titled
"[scheduled-failure] <workflow name>" — or appends a comment to the
existing open one (dedupe key = workflow name) — with the run URL, run
attempt, and failed job names, labeled "infrastructure".
Wire it into the scheduled paths of e2e-s3tests, mint, fuzz (nightly) and
performance-ab via a final alert-on-failure job gated by
"always() && github.event_name == 'schedule' &&
contains(needs.*.result, 'failure')", with issues:write scoped to that
job only. e2e-s3tests and mint previously had no permissions key; they now
get workflow-level contents:read, reducing every other job's token.
Add a dispatch-only drill workflow that forces a job failure and runs the
action through the exact consumer wiring, for end-to-end verification.
The ci-7 e2e nightly does not exist yet; it is recorded as a pending
consumer in the action README.
Refs: rustfs/backlog#1149 (ci-8), rustfs/backlog#1155
Both nightly runs of the warp A/B rig failed at server bring-up:
`endpoint http://127.0.0.1:9000/health did not become healthy` — exactly 60s
after start. The server binds its public listener only after startup converges
(erasure-format load + IAM); its own budget
(RUSTFS_STARTUP_READINESS_MAX_WAIT_SECS) defaults to 120s, so the rig's 60s
health poll gave up before a slow cold start on the shared sm-standard-2 runner
finished. The rustfs.log lived under /tmp (not the uploaded target/hotpath-ab/),
so the root cause was invisible in the artifact.
Root-cause + diagnosability (scripts/run_hotpath_warp_ab.sh):
- Health poll is configurable via --health-timeout, default 180s (> the 120s
server startup budget). Fails fast if the local server process exits before
becoming healthy instead of polling out the full window.
- Server logs + a startup-env file now write under OUT_DIR/server-logs/ so they
ride along in the CI artifact. On a health failure the rig dumps the last 50
log lines to the job log.
Workflow (.github/workflows/performance-ab.yml):
- On every run, write gate.md (or, on a pre-gate failure, the failing phase's
server-log tails) into $GITHUB_STEP_SUMMARY; short artifact retention.
- Short measurement matrix (--duration/--rounds/--cooldown) so the expanded
matrix fits; timeout-minutes 90 -> 120 as a Phase-0 stopgap until perf-3
caches the baseline binary (the ~65min double build dominates).
- TODO(ci-8/perf-2) hook comment for the failure-alert composite action.
Workload cells (absorbed perf-4): add put-4kib/get-4kib (the #4221 fsync
regression size, ~-10% @4KiB, previously invisible) and get-10mib (historical
large-GET EOF size) to both the PR-labeled and nightly matrices — 6 workloads
x 2 phases x 2 drive-sync = 24 cells. A 1KiB cell is deferred to protect the
budget until perf-3.
Refs rustfs/backlog#1152 (perf-1, absorbed perf-4), rustfs/backlog#1155.
- build.yml: update latest.json only for stable release tags
(alpha/beta/rc tags previously overwrote the stable pointer with
release_type "stable"); drop the placeholder .asc files; build the
Linux targets only on main pushes (tags/schedule/dispatch keep the
full matrix); remove the unreachable --build clause and a needless
full-history clone
- ci.yml: event-scoped concurrency so merges stop cancelling the
weekly scheduled run; drop the redundant skip-duplicate-actions
gate job; run clippy before tests; trim the unused toolchain from
the typos job
- ci-docs-only.yml (new): satisfy the required "Test and Lint" check
on docs-only PRs that ci.yml skips via paths-ignore
- audit.yml: drop cargo-audit (cargo-deny advisories covers the same
RustSec database); same event-scoped concurrency fix
- docker.yml: job-level short-circuit for per-merge dev builds; send
the Trivy SARIF to code scanning; unify the upload-artifact pin
- performance-ab.yml: add concurrency; only re-run on labeled events
when the added label is perf-ab
- stagger the Sunday crons (build 01:00, audit 03:00,
nix-flake-update 05:00, mint 06:00)
- delete performance.yml: disabled since 2025-07; idle-server
profiling, no benchmark baseline, stale ecstore package name
* ci(perf): warp A/B relative-budget gate for the hotpath series
performance.yml only uploads a samply profile and a cargo-bench artifact with
no pass/fail, so a 26x-class write-path regression like #4221 or the GET perf
churn would sail through CI and only surface in customer load tests. This adds
the missing gate — the acceptance surface every queued HP change (#922/#923/
#925/#927/#930/#932) needs before its default can flip.
- scripts/hotpath_warp_ab_gate.sh: applies the relative budget to the
baseline_compare.csv that run_object_batch_bench_enhanced.sh already emits
(it computes deltas but never gates). A metric regressing past --fail-pct
fails, past --warn-pct warns; reqps/throughput are higher-is-better, latency
lower-is-better. --allow-regression downgrades a FAIL to an exempted WARN so
a deliberate correctness cost (e.g. #4221) is recorded, not blocked
(rustfs/backlog#935 correction 1). Unit-checked across pass/warn/fail/exempt.
- scripts/run_hotpath_warp_ab.sh: single-host baseline-binary vs candidate-
binary A/B over {put-4mib, get-4mib, mixed-256k} x {drive-sync on, off},
reusing the enhanced bench as the warp driver and the single-node local-disk
lifecycle. Has --dry-run; warp is assumed pre-installed as elsewhere in
scripts/. Drive-sync on/off keeps a sync-semantics change from being masked
by nosync numbers.
- .github/workflows/performance-ab.yml: nightly on main (post-merge detection)
plus opt-in pre-merge via the `perf-ab` label; `perf-deliberate-tradeoff`
runs the gate with --allow-regression; posts the gate table as a PR comment
and uploads the run. A Linux runner answers "do the macOS conclusions hold".
The gate logic is unit-validated with synthetic compare CSVs; the orchestrator
and workflow are shellcheck- and --dry-run-validated. The first real warp
measurement belongs on the Linux runner (no warp/multi-disk rig locally).
Refs: rustfs/backlog#935 (HP-14 warp A/B gate, item 4), rustfs/backlog#725
(cooled A/B harness precedent), rustfs/backlog#936
Co-Authored-By: heihutu <heihutu@gmail.com>
* ci(perf): pin upload-artifact, add external-cluster A/B mode + runbook
- Pin actions/upload-artifact to the repo-standard full-length SHA (# v6);
the previous @v4 float tripped the workflow-pin guard.
- Add an external deployment mode to run_hotpath_warp_ab.sh so warp can target
an already-running cluster instead of a throwaway single-node server:
--endpoint selects the cluster and --deploy-hook runs between phases to swap
in the phase's binary and drive-sync config (context passed via
HOTPATH_AB_PHASE / HOTPATH_AB_BINARY / HOTPATH_AB_DRIVE_SYNC). This maps onto
the team's ansible harness (cargo zigbuild -> ansible rustfs-manage --tags
stop,config,binary-copy,start -> warp).
- Restructure the matrix loop to bring a deployment up once per (phase,
drive-sync) and run all three workloads against it, instead of restarting per
workload — fewer server starts / cluster redeploys. Baseline medians are read
from a deterministic path, dropping the bash-4-only associative array so the
script (and its --dry-run self-check) runs on macOS bash 3.2 too.
- Add docs/operations/hotpath-warp-ab-runbook.md tying local mode, the ansible
external mode, and the budget/exemption together.
Verification: shellcheck clean; --dry-run in both modes prints the expected
4 deployments x 3 workloads and passes exactly 6 compare CSVs to the gate;
check_workflow_pins.sh, check_doc_paths.sh, and make pre-commit all pass.
Refs: rustfs/backlog#935, rustfs/backlog#936
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Zhengchao An