mirror of
https://github.com/rustfs/rustfs.git
synced 2026-08-05 12:57:42 +00:00
f5348d5cc48d21eecefbfe8c99090f0297781a4e
7 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Zhengchao An
|
e2b2bdcc34 |
ci: bound every job's runtime and stop pasting inputs into shell (#5537)
Three hardening changes with no effect on what any workflow produces. Declare timeout-minutes on the 25 jobs that lacked it. GitHub's default is 360 minutes, and this repository has a history of runners stalling intermittently (#5394) plus a measured 9m57s plain `git checkout` under node-level I/O contention, so one wedged job could hold a runner for six hours out of a pool of roughly 15-21. Budgets follow what the jobs actually do: 10 minutes for echo-only and guard-script jobs, 30 for anything calling the GitHub API, uploading release assets or pushing over the network. scripts/security/check_job_timeouts.sh keeps it that way, checking only jobs that declare runs-on so reusable-workflow callers are not flagged. Pass workflow inputs and workflow_run fields through env instead of `${{ }}` interpolation in run blocks. A git ref name may contain `$(...)` — any string without a space is a legal tag — and interpolation pastes it into the script where bash evaluates it. The worst instance was helm-package's final commit message: it is built from the triggering tag name inside the job that holds the cross-repository push token with rustfs/helm already checked out. Also converted in build.yml, docker.yml and performance-ab.yml; the last is currently disabled, but a disabled workflow can be re-enabled. Not touched: helm-package's `contains(head_branch, '.')` tag test, since GitHub expressions have no regex and this repository's tags carry no `v` prefix, so rewriting the condition would change which builds publish a chart. Give audit.yml a scheduled-failure alert and run it daily. A scheduled cargo-deny failure usually means the dependency tree just matched a newly published RustSec advisory — the most important signal this workflow produces, and until now it was visible only to whoever happened to open the Actions tab. coverage.yml and e2e-replication-nightly.yml already use this ci-8 mechanism. The cron moves from weekly to daily so a new advisory against an unchanged tree surfaces within a day instead of seven; the check list is untouched, since splitting it into a light daily run and a weekly full run would create runs where sources, bans and licenses go unverified. Refs: rustfs/backlog#1598, rustfs/backlog#1602 |
||
|
Zhengchao An
|
242424b0fc |
ci(perf): fit the baseline cache build to the post-LTO profile and self-heal nightly misses (#4841)
* ci(perf): fit the baseline cache build to the post-LTO profile and self-heal nightly misses
Every build-baseline-cache run on 2026-07-15 died on its 60min ceiling
('exceeded the maximum execution time of 1h0m0s'): #4806 put thin LTO +
codegen-units=1 on [profile.release], pushing a single release build past
60min on sm-standard-2, so the baseline cache never populated. The measured
binary must keep the production profile, so raise the job budget to 100min
instead of weakening the profile.
Also make the A/B job self-heal a same-commit cache miss: when the candidate
commit equals the baseline commit (nightly/dispatch on main) and the restore
misses, build the binary once, reuse it for both phases, and save it back to
the cache under rustfs-baseline-<sha> — previously that miss made the rig
build the identical commit twice, which no job budget fits post-#4806. The
PR-context miss (candidate != baseline, opt-in gate) keeps the double-build
fallback and now documents that it will overrun and alert.
Refs rustfs/backlog#1152 (perf-3 follow-up).
* ci(perf): latest-wins concurrency for the baseline cache build
Consumers only restore the binary for the current origin/main tip, so a
superseded push build's output is dead weight; cancel it instead of stacking
~65min jobs on the shared sm-standard-2 pool when main merges quickly. A
skipped intermediate SHA at most costs one same-commit self-heal in the A/B
job.
|
||
|
Zhengchao An
|
d73c8a783a |
ci(perf): cache the main baseline binary to cut the nightly double build (#4816)
Every push to main now builds the release binary once and stores it in the actions cache as rustfs-baseline-<sha> (build-baseline-cache job). The A/B job restores it for origin/main and passes --baseline-bin; on the nightly, where the candidate commit equals the baseline, one cached binary serves both phases with --skip-build and the run does zero source builds. A cache miss falls back to the source double-build via --baseline-ref origin/main. This removes the ~32min-per-side double build that pushed the 24-cell nightly past its 120min ceiling (2026-07-11..07-14 all cancelled on timeout). Also: - alert-on-failure now fires on cancelled/timed-out, not just failure, so a timed-out nightly is no longer silent (the composite action already reports cancelled jobs); removed the stale perf-2 TODO now that the alert job exists. - run_hotpath_warp_ab.sh appends a Provenance section to gate.md (baseline and candidate SHAs + binary source, runner, warp version, matrix params) via a repeatable --provenance-note; the gate exit code is preserved. Refs rustfs/backlog#1152 (perf-3). |
||
|
Zhengchao An
|
4310b55238 |
ci: add schedule-failure-issue composite action and wire nightly alerts (#4671)
Scheduled workflow failures previously went unnoticed (15 consecutive red weekly s3-tests sweeps, silent perf nightly failures). Add a reusable composite action that opens a tracking issue titled "[scheduled-failure] <workflow name>" — or appends a comment to the existing open one (dedupe key = workflow name) — with the run URL, run attempt, and failed job names, labeled "infrastructure". Wire it into the scheduled paths of e2e-s3tests, mint, fuzz (nightly) and performance-ab via a final alert-on-failure job gated by "always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure')", with issues:write scoped to that job only. e2e-s3tests and mint previously had no permissions key; they now get workflow-level contents:read, reducing every other job's token. Add a dispatch-only drill workflow that forces a job failure and runs the action through the exact consumer wiring, for end-to-end verification. The ci-7 e2e nightly does not exist yet; it is recorded as a pending consumer in the action README. Refs: rustfs/backlog#1149 (ci-8), rustfs/backlog#1155 |
||
|
Zhengchao An
|
fd18b1df49 |
ci(perf): fix nightly warp A/B startup failure, make it diagnosable, add small-object + 10MiB cells (#4669)
Both nightly runs of the warp A/B rig failed at server bring-up: `endpoint http://127.0.0.1:9000/health did not become healthy` — exactly 60s after start. The server binds its public listener only after startup converges (erasure-format load + IAM); its own budget (RUSTFS_STARTUP_READINESS_MAX_WAIT_SECS) defaults to 120s, so the rig's 60s health poll gave up before a slow cold start on the shared sm-standard-2 runner finished. The rustfs.log lived under /tmp (not the uploaded target/hotpath-ab/), so the root cause was invisible in the artifact. Root-cause + diagnosability (scripts/run_hotpath_warp_ab.sh): - Health poll is configurable via --health-timeout, default 180s (> the 120s server startup budget). Fails fast if the local server process exits before becoming healthy instead of polling out the full window. - Server logs + a startup-env file now write under OUT_DIR/server-logs/ so they ride along in the CI artifact. On a health failure the rig dumps the last 50 log lines to the job log. Workflow (.github/workflows/performance-ab.yml): - On every run, write gate.md (or, on a pre-gate failure, the failing phase's server-log tails) into $GITHUB_STEP_SUMMARY; short artifact retention. - Short measurement matrix (--duration/--rounds/--cooldown) so the expanded matrix fits; timeout-minutes 90 -> 120 as a Phase-0 stopgap until perf-3 caches the baseline binary (the ~65min double build dominates). - TODO(ci-8/perf-2) hook comment for the failure-alert composite action. Workload cells (absorbed perf-4): add put-4kib/get-4kib (the #4221 fsync regression size, ~-10% @4KiB, previously invisible) and get-10mib (historical large-GET EOF size) to both the PR-labeled and nightly matrices — 6 workloads x 2 phases x 2 drive-sync = 24 cells. A 1KiB cell is deferred to protect the budget until perf-3. Refs rustfs/backlog#1152 (perf-1, absorbed perf-4), rustfs/backlog#1155. |
||
|
Zhengchao An
|
2ae1e8ad05 |
ci: right-size pipelines, fix prerelease latest.json overwrite (#4582)
- build.yml: update latest.json only for stable release tags (alpha/beta/rc tags previously overwrote the stable pointer with release_type "stable"); drop the placeholder .asc files; build the Linux targets only on main pushes (tags/schedule/dispatch keep the full matrix); remove the unreachable --build clause and a needless full-history clone - ci.yml: event-scoped concurrency so merges stop cancelling the weekly scheduled run; drop the redundant skip-duplicate-actions gate job; run clippy before tests; trim the unused toolchain from the typos job - ci-docs-only.yml (new): satisfy the required "Test and Lint" check on docs-only PRs that ci.yml skips via paths-ignore - audit.yml: drop cargo-audit (cargo-deny advisories covers the same RustSec database); same event-scoped concurrency fix - docker.yml: job-level short-circuit for per-merge dev builds; send the Trivy SARIF to code scanning; unify the upload-artifact pin - performance-ab.yml: add concurrency; only re-run on labeled events when the added label is perf-ab - stagger the Sunday crons (build 01:00, audit 03:00, nix-flake-update 05:00, mint 06:00) - delete performance.yml: disabled since 2025-07; idle-server profiling, no benchmark baseline, stale ecstore package name |
||
|
houseme
|
2157470224 |
ci(perf): warp A/B relative-budget gate for the hotpath series (#4480)
* ci(perf): warp A/B relative-budget gate for the hotpath series performance.yml only uploads a samply profile and a cargo-bench artifact with no pass/fail, so a 26x-class write-path regression like #4221 or the GET perf churn would sail through CI and only surface in customer load tests. This adds the missing gate — the acceptance surface every queued HP change (#922/#923/ #925/#927/#930/#932) needs before its default can flip. - scripts/hotpath_warp_ab_gate.sh: applies the relative budget to the baseline_compare.csv that run_object_batch_bench_enhanced.sh already emits (it computes deltas but never gates). A metric regressing past --fail-pct fails, past --warn-pct warns; reqps/throughput are higher-is-better, latency lower-is-better. --allow-regression downgrades a FAIL to an exempted WARN so a deliberate correctness cost (e.g. #4221) is recorded, not blocked (rustfs/backlog#935 correction 1). Unit-checked across pass/warn/fail/exempt. - scripts/run_hotpath_warp_ab.sh: single-host baseline-binary vs candidate- binary A/B over {put-4mib, get-4mib, mixed-256k} x {drive-sync on, off}, reusing the enhanced bench as the warp driver and the single-node local-disk lifecycle. Has --dry-run; warp is assumed pre-installed as elsewhere in scripts/. Drive-sync on/off keeps a sync-semantics change from being masked by nosync numbers. - .github/workflows/performance-ab.yml: nightly on main (post-merge detection) plus opt-in pre-merge via the `perf-ab` label; `perf-deliberate-tradeoff` runs the gate with --allow-regression; posts the gate table as a PR comment and uploads the run. A Linux runner answers "do the macOS conclusions hold". The gate logic is unit-validated with synthetic compare CSVs; the orchestrator and workflow are shellcheck- and --dry-run-validated. The first real warp measurement belongs on the Linux runner (no warp/multi-disk rig locally). Refs: rustfs/backlog#935 (HP-14 warp A/B gate, item 4), rustfs/backlog#725 (cooled A/B harness precedent), rustfs/backlog#936 Co-Authored-By: heihutu <heihutu@gmail.com> * ci(perf): pin upload-artifact, add external-cluster A/B mode + runbook - Pin actions/upload-artifact to the repo-standard full-length SHA (# v6); the previous @v4 float tripped the workflow-pin guard. - Add an external deployment mode to run_hotpath_warp_ab.sh so warp can target an already-running cluster instead of a throwaway single-node server: --endpoint selects the cluster and --deploy-hook runs between phases to swap in the phase's binary and drive-sync config (context passed via HOTPATH_AB_PHASE / HOTPATH_AB_BINARY / HOTPATH_AB_DRIVE_SYNC). This maps onto the team's ansible harness (cargo zigbuild -> ansible rustfs-manage --tags stop,config,binary-copy,start -> warp). - Restructure the matrix loop to bring a deployment up once per (phase, drive-sync) and run all three workloads against it, instead of restarting per workload — fewer server starts / cluster redeploys. Baseline medians are read from a deterministic path, dropping the bash-4-only associative array so the script (and its --dry-run self-check) runs on macOS bash 3.2 too. - Add docs/operations/hotpath-warp-ab-runbook.md tying local mode, the ansible external mode, and the budget/exemption together. Verification: shellcheck clean; --dry-run in both modes prints the expected 4 deployments x 3 workloads and passes exactly 6 compare CSVs to the gate; check_workflow_pins.sh, check_doc_paths.sh, and make pre-commit all pass. Refs: rustfs/backlog#935, rustfs/backlog#936 Co-Authored-By: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> |