mirror of
https://github.com/rustfs/rustfs.git
synced 2026-08-02 11:29:17 +00:00
ed2e11ee2e28af4ea4e2a91d7f81b826ee380104
10 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Zhengchao An
|
2b31bda6d1 |
ci: clear the checkout token in every job that does not push (#5547)
actions/checkout writes its token into .git/config as an http extraheader, where it stays for the rest of the job. That matters more here than usual: pull_request jobs run on self-hosted runners and execute the PR's own build.rs, proc-macros and tests, any of which can read that file. Test and Lint holds actions: write on top of that, so its token can cancel runs and delete the Actions caches the whole pipeline now depends on — it was given persist-credentials: false when that permission was added, and this extends the same treatment to the other 45 checkouts. Two are exempt because the token IS the credential the job needs. helm-package's publish job pushes to rustfs/helm with it; clearing it would break chart publishing. nix-flake-update is exempt pending verification: it passes FLAKE_UPDATE_TOKEN to create-pull-request directly rather than reusing .git/config, so it very likely does not need persistence, but that is unproven and a broken weekly bot is not worth the guess. Both carry a persist-credentials-exempt comment saying which. scripts/security/check_persist_credentials.sh requires every checkout to either clear its credentials or carry that comment, so the decision stays visible in review rather than being an omission nobody notices. Refs: rustfs/backlog#1598, rustfs/backlog#1602 |
||
|
Zhengchao An
|
428fde069d |
ci: drop the docker layer cache and stop installing unused tooling (#5544)
Two cleanups, neither changing what is built or published. docker.yml loses its type=gha layer cache. The image build compiles nothing — it downloads a release zip and runs apk/apt — so the cache could only save the minute or two those take, against a real correctness problem: with RELEASE=latest the binary URL is resolved by curl inside a RUN layer, and the layer key does not include what it resolved to. A rebuild at the same RELEASE value (a dispatch with version=latest, or a re-run of the same version) would hit the old layer and ship the previous release's binary. mode=max also drew on the same repo-wide 10GB Actions cache quota the Rust lanes are contending for. Only RELEASE is passed as a build-arg now; it is the sole one the Dockerfiles declare besides TARGETARCH. BUILDTIME, VERSION, BUILD_TYPE, REVISION and CHANNEL were read by no stage, and BUILDTIME's $(date ...) was a literal string in the YAML block rather than a substitution. The DOCKER_CHANNEL computation that fed CHANNEL evaluated to "release" down every branch and is gone with it. BUILD_DATE and VCS_REF stay unset even though the Dockerfiles declare them: supplying them would change the published image labels. The setup composite stops installing tools its callers do not use. protobuf- compiler comes out of the apt list entirely — setup-protoc installs 34.1 into the tool cache and prepends it to PATH, so the apt build was shadowed on every run and never used; the same duplicate install is removed from the io_uring lane. musl-tools, zip and unzip move behind install-build-packaging-tools, off for the CI and coverage lanes and left on for build.yml, whose native musl leg needs musl-gcc and whose packaging steps need zip. cargo-nextest and the rustfmt/clippy components move behind install-test-tools, off only for build.yml, which runs no tests and no lints; coverage and the nightly replication lane keep them because both invoke nextest. The action's github-token input is deleted along with its 20 call sites. Its runs block never referenced it — setup-protoc uses github.token directly — so it was 20 places handing a token to something that ignored it. Refs: rustfs/backlog#1598, rustfs/backlog#1600, rustfs/backlog#1603 |
||
|
Zhengchao An
|
6c99d4fe22 |
ci: stop the weekly flake.lock bot from running the whole pipeline (#5539)
nix-flake-update opens a PR every Sunday that changes flake.lock and nothing else. flake.lock is consumed only by Nix packaging — cargo never reads it — yet it was in no paths filter, so each of those PRs ran the full Continuous Integration pipeline and the merge then ran Build and Release too. Added to all four lists that have to agree: ci.yml's push and pull_request paths-ignore, build.yml's push paths-ignore, and ci-docs-only.yml's paths. The cron stays. flake.lock still has consumers — anyone running `nix build` or `nix develop` — so stopping the updates would remove the workflow's output, not just its CI cost. Those four lists drifting is a silent failure, so scripts/check_ci_paths_sync.sh now asserts the pair that matters: ci.yml's pull_request paths-ignore must equal ci-docs-only.yml's paths. An entry present only in the first means a PR touching those files triggers neither workflow, nobody reports "Test and Lint" or "Quick Checks", and the PR waits on a required check forever. It also asserts both companion job names still exist, since renaming one produces the same hang. The push list is not compared: no required check is reported for push events. Also in this cleanup: - nix-flake-update's GITHUB_TOKEN drops to contents: read. The branch push and the pull request are both made by update-flake-lock with the FLAKE_UPDATE_TOKEN PAT, so the write scopes were an unused repo-write credential on an unattended weekly job. - build.yml's build_docker dispatch input is documented as advisory. docker.yml triggers on workflow_run and requires the triggering event to be a tag push, so a manual dispatch never reaches it whatever this input says. - The nine disabled workflow files get a banner saying so. Their disabled_manually state lives in GitHub's UI and is invisible when reading the file, which has already misled one audit into treating dead workflows as live. Refs: rustfs/backlog#1598, rustfs/backlog#1603 |
||
|
Zhengchao An
|
e2b2bdcc34 |
ci: bound every job's runtime and stop pasting inputs into shell (#5537)
Three hardening changes with no effect on what any workflow produces. Declare timeout-minutes on the 25 jobs that lacked it. GitHub's default is 360 minutes, and this repository has a history of runners stalling intermittently (#5394) plus a measured 9m57s plain `git checkout` under node-level I/O contention, so one wedged job could hold a runner for six hours out of a pool of roughly 15-21. Budgets follow what the jobs actually do: 10 minutes for echo-only and guard-script jobs, 30 for anything calling the GitHub API, uploading release assets or pushing over the network. scripts/security/check_job_timeouts.sh keeps it that way, checking only jobs that declare runs-on so reusable-workflow callers are not flagged. Pass workflow inputs and workflow_run fields through env instead of `${{ }}` interpolation in run blocks. A git ref name may contain `$(...)` — any string without a space is a legal tag — and interpolation pastes it into the script where bash evaluates it. The worst instance was helm-package's final commit message: it is built from the triggering tag name inside the job that holds the cross-repository push token with rustfs/helm already checked out. Also converted in build.yml, docker.yml and performance-ab.yml; the last is currently disabled, but a disabled workflow can be re-enabled. Not touched: helm-package's `contains(head_branch, '.')` tag test, since GitHub expressions have no regex and this repository's tags carry no `v` prefix, so rewriting the condition would change which builds publish a chart. Give audit.yml a scheduled-failure alert and run it daily. A scheduled cargo-deny failure usually means the dependency tree just matched a newly published RustSec advisory — the most important signal this workflow produces, and until now it was visible only to whoever happened to open the Actions tab. coverage.yml and e2e-replication-nightly.yml already use this ci-8 mechanism. The cron moves from weekly to daily so a new advisory against an unchanged tree surfaces within a day instead of seven; the check list is untouched, since splitting it into a light daily run and a weekly full run would create runs where sources, bans and licenses go unverified. Refs: rustfs/backlog#1598, rustfs/backlog#1602 |
||
|
Zhengchao An
|
242424b0fc |
ci(perf): fit the baseline cache build to the post-LTO profile and self-heal nightly misses (#4841)
* ci(perf): fit the baseline cache build to the post-LTO profile and self-heal nightly misses
Every build-baseline-cache run on 2026-07-15 died on its 60min ceiling
('exceeded the maximum execution time of 1h0m0s'): #4806 put thin LTO +
codegen-units=1 on [profile.release], pushing a single release build past
60min on sm-standard-2, so the baseline cache never populated. The measured
binary must keep the production profile, so raise the job budget to 100min
instead of weakening the profile.
Also make the A/B job self-heal a same-commit cache miss: when the candidate
commit equals the baseline commit (nightly/dispatch on main) and the restore
misses, build the binary once, reuse it for both phases, and save it back to
the cache under rustfs-baseline-<sha> — previously that miss made the rig
build the identical commit twice, which no job budget fits post-#4806. The
PR-context miss (candidate != baseline, opt-in gate) keeps the double-build
fallback and now documents that it will overrun and alert.
Refs rustfs/backlog#1152 (perf-3 follow-up).
* ci(perf): latest-wins concurrency for the baseline cache build
Consumers only restore the binary for the current origin/main tip, so a
superseded push build's output is dead weight; cancel it instead of stacking
~65min jobs on the shared sm-standard-2 pool when main merges quickly. A
skipped intermediate SHA at most costs one same-commit self-heal in the A/B
job.
|
||
|
Zhengchao An
|
d73c8a783a |
ci(perf): cache the main baseline binary to cut the nightly double build (#4816)
Every push to main now builds the release binary once and stores it in the actions cache as rustfs-baseline-<sha> (build-baseline-cache job). The A/B job restores it for origin/main and passes --baseline-bin; on the nightly, where the candidate commit equals the baseline, one cached binary serves both phases with --skip-build and the run does zero source builds. A cache miss falls back to the source double-build via --baseline-ref origin/main. This removes the ~32min-per-side double build that pushed the 24-cell nightly past its 120min ceiling (2026-07-11..07-14 all cancelled on timeout). Also: - alert-on-failure now fires on cancelled/timed-out, not just failure, so a timed-out nightly is no longer silent (the composite action already reports cancelled jobs); removed the stale perf-2 TODO now that the alert job exists. - run_hotpath_warp_ab.sh appends a Provenance section to gate.md (baseline and candidate SHAs + binary source, runner, warp version, matrix params) via a repeatable --provenance-note; the gate exit code is preserved. Refs rustfs/backlog#1152 (perf-3). |
||
|
Zhengchao An
|
4310b55238 |
ci: add schedule-failure-issue composite action and wire nightly alerts (#4671)
Scheduled workflow failures previously went unnoticed (15 consecutive red weekly s3-tests sweeps, silent perf nightly failures). Add a reusable composite action that opens a tracking issue titled "[scheduled-failure] <workflow name>" — or appends a comment to the existing open one (dedupe key = workflow name) — with the run URL, run attempt, and failed job names, labeled "infrastructure". Wire it into the scheduled paths of e2e-s3tests, mint, fuzz (nightly) and performance-ab via a final alert-on-failure job gated by "always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure')", with issues:write scoped to that job only. e2e-s3tests and mint previously had no permissions key; they now get workflow-level contents:read, reducing every other job's token. Add a dispatch-only drill workflow that forces a job failure and runs the action through the exact consumer wiring, for end-to-end verification. The ci-7 e2e nightly does not exist yet; it is recorded as a pending consumer in the action README. Refs: rustfs/backlog#1149 (ci-8), rustfs/backlog#1155 |
||
|
Zhengchao An
|
fd18b1df49 |
ci(perf): fix nightly warp A/B startup failure, make it diagnosable, add small-object + 10MiB cells (#4669)
Both nightly runs of the warp A/B rig failed at server bring-up: `endpoint http://127.0.0.1:9000/health did not become healthy` — exactly 60s after start. The server binds its public listener only after startup converges (erasure-format load + IAM); its own budget (RUSTFS_STARTUP_READINESS_MAX_WAIT_SECS) defaults to 120s, so the rig's 60s health poll gave up before a slow cold start on the shared sm-standard-2 runner finished. The rustfs.log lived under /tmp (not the uploaded target/hotpath-ab/), so the root cause was invisible in the artifact. Root-cause + diagnosability (scripts/run_hotpath_warp_ab.sh): - Health poll is configurable via --health-timeout, default 180s (> the 120s server startup budget). Fails fast if the local server process exits before becoming healthy instead of polling out the full window. - Server logs + a startup-env file now write under OUT_DIR/server-logs/ so they ride along in the CI artifact. On a health failure the rig dumps the last 50 log lines to the job log. Workflow (.github/workflows/performance-ab.yml): - On every run, write gate.md (or, on a pre-gate failure, the failing phase's server-log tails) into $GITHUB_STEP_SUMMARY; short artifact retention. - Short measurement matrix (--duration/--rounds/--cooldown) so the expanded matrix fits; timeout-minutes 90 -> 120 as a Phase-0 stopgap until perf-3 caches the baseline binary (the ~65min double build dominates). - TODO(ci-8/perf-2) hook comment for the failure-alert composite action. Workload cells (absorbed perf-4): add put-4kib/get-4kib (the #4221 fsync regression size, ~-10% @4KiB, previously invisible) and get-10mib (historical large-GET EOF size) to both the PR-labeled and nightly matrices — 6 workloads x 2 phases x 2 drive-sync = 24 cells. A 1KiB cell is deferred to protect the budget until perf-3. Refs rustfs/backlog#1152 (perf-1, absorbed perf-4), rustfs/backlog#1155. |
||
|
Zhengchao An
|
2ae1e8ad05 |
ci: right-size pipelines, fix prerelease latest.json overwrite (#4582)
- build.yml: update latest.json only for stable release tags (alpha/beta/rc tags previously overwrote the stable pointer with release_type "stable"); drop the placeholder .asc files; build the Linux targets only on main pushes (tags/schedule/dispatch keep the full matrix); remove the unreachable --build clause and a needless full-history clone - ci.yml: event-scoped concurrency so merges stop cancelling the weekly scheduled run; drop the redundant skip-duplicate-actions gate job; run clippy before tests; trim the unused toolchain from the typos job - ci-docs-only.yml (new): satisfy the required "Test and Lint" check on docs-only PRs that ci.yml skips via paths-ignore - audit.yml: drop cargo-audit (cargo-deny advisories covers the same RustSec database); same event-scoped concurrency fix - docker.yml: job-level short-circuit for per-merge dev builds; send the Trivy SARIF to code scanning; unify the upload-artifact pin - performance-ab.yml: add concurrency; only re-run on labeled events when the added label is perf-ab - stagger the Sunday crons (build 01:00, audit 03:00, nix-flake-update 05:00, mint 06:00) - delete performance.yml: disabled since 2025-07; idle-server profiling, no benchmark baseline, stale ecstore package name |
||
|
houseme
|
2157470224 |
ci(perf): warp A/B relative-budget gate for the hotpath series (#4480)
* ci(perf): warp A/B relative-budget gate for the hotpath series performance.yml only uploads a samply profile and a cargo-bench artifact with no pass/fail, so a 26x-class write-path regression like #4221 or the GET perf churn would sail through CI and only surface in customer load tests. This adds the missing gate — the acceptance surface every queued HP change (#922/#923/ #925/#927/#930/#932) needs before its default can flip. - scripts/hotpath_warp_ab_gate.sh: applies the relative budget to the baseline_compare.csv that run_object_batch_bench_enhanced.sh already emits (it computes deltas but never gates). A metric regressing past --fail-pct fails, past --warn-pct warns; reqps/throughput are higher-is-better, latency lower-is-better. --allow-regression downgrades a FAIL to an exempted WARN so a deliberate correctness cost (e.g. #4221) is recorded, not blocked (rustfs/backlog#935 correction 1). Unit-checked across pass/warn/fail/exempt. - scripts/run_hotpath_warp_ab.sh: single-host baseline-binary vs candidate- binary A/B over {put-4mib, get-4mib, mixed-256k} x {drive-sync on, off}, reusing the enhanced bench as the warp driver and the single-node local-disk lifecycle. Has --dry-run; warp is assumed pre-installed as elsewhere in scripts/. Drive-sync on/off keeps a sync-semantics change from being masked by nosync numbers. - .github/workflows/performance-ab.yml: nightly on main (post-merge detection) plus opt-in pre-merge via the `perf-ab` label; `perf-deliberate-tradeoff` runs the gate with --allow-regression; posts the gate table as a PR comment and uploads the run. A Linux runner answers "do the macOS conclusions hold". The gate logic is unit-validated with synthetic compare CSVs; the orchestrator and workflow are shellcheck- and --dry-run-validated. The first real warp measurement belongs on the Linux runner (no warp/multi-disk rig locally). Refs: rustfs/backlog#935 (HP-14 warp A/B gate, item 4), rustfs/backlog#725 (cooled A/B harness precedent), rustfs/backlog#936 Co-Authored-By: heihutu <heihutu@gmail.com> * ci(perf): pin upload-artifact, add external-cluster A/B mode + runbook - Pin actions/upload-artifact to the repo-standard full-length SHA (# v6); the previous @v4 float tripped the workflow-pin guard. - Add an external deployment mode to run_hotpath_warp_ab.sh so warp can target an already-running cluster instead of a throwaway single-node server: --endpoint selects the cluster and --deploy-hook runs between phases to swap in the phase's binary and drive-sync config (context passed via HOTPATH_AB_PHASE / HOTPATH_AB_BINARY / HOTPATH_AB_DRIVE_SYNC). This maps onto the team's ansible harness (cargo zigbuild -> ansible rustfs-manage --tags stop,config,binary-copy,start -> warp). - Restructure the matrix loop to bring a deployment up once per (phase, drive-sync) and run all three workloads against it, instead of restarting per workload — fewer server starts / cluster redeploys. Baseline medians are read from a deterministic path, dropping the bash-4-only associative array so the script (and its --dry-run self-check) runs on macOS bash 3.2 too. - Add docs/operations/hotpath-warp-ab-runbook.md tying local mode, the ansible external mode, and the budget/exemption together. Verification: shellcheck clean; --dry-run in both modes prints the expected 4 deployments x 3 workloads and passes exactly 6 compare CSVs to the gate; check_workflow_pins.sh, check_doc_paths.sh, and make pre-commit all pass. Refs: rustfs/backlog#935, rustfs/backlog#936 Co-Authored-By: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> |