Apply the placement rule in .github/workflows/README.md to the 22 jobs
still on sm-standard-2 that only call APIs, read workflow metadata, print
summaries or scan published images, including cla.yml, the busiest
workflow. Moving the ODM cloud-source lane also stops it restoring the
ci-odm-interop key across images, since its writer already runs on
ubuntu-latest; targets-live gets a -hosted cache key for the same reason.
* fix(build): pin gnu glibc baseline to 2.31 and report compile target in version metadata
gnu release binaries were built natively on Ubuntu 24.04, embedding
GLIBC_2.38/2.39 symbol references while the RPM/DEB metadata promised
glibc >= 2.31, so Rocky/RHEL 9 (glibc 2.34) could not run them (#8062).
The two linux-gnu legs now build through cargo-zigbuild pinned to the
2.31 baseline (same-arch; zig acts as the linker), with post-build
guards asserting both the embedded version metadata and the highest
referenced GLIBC symbol version, keeping the package declarations
honest.
Version metadata itself now comes from cargo's TARGET cfg instead of
shadow-rs' host stamping, so builds whose host differs from their
target (cross compilation, musl targets on a glibc-hosted toolchain)
report the platform they were compiled for (#8048 follow-up).
The cross toolchain (zig 0.14.x, cargo-zigbuild 0.23.4) is pinned in
the setup action now that it is load-bearing for releases, the build
job gets an explicit matrix-independent name so the connect-*
acceptance workflows keep matching it when matrix fields change, and
the preview-release workflow contract is updated for the TARGET_ARG
build commands.
Fixes#8062
Follow-up on #8048
* fix(build): enforce GNU ABI floor and canonical target metadata
Pin Zig 0.17.0 and cargo-zigbuild, isolate cross-toolchain caches, and
build GNU releases against the declared glibc 2.31 floor. Validate both
release ELF binaries using their loader version requirements.
Generate target metadata once through shadow-rs and check the actual
release version output without adding a parallel metadata facade.
Refs: rustfs/backlog#2824
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* perf(build): reuse GNU release profile across binaries
Switching the CLI back to the default strip profile restarted 419
compilation units after the server build. Keep the profiling build
profile for both binaries, then strip only the CLI artifact.
Retain serial links, server profiling symbols, and post-strip ABI gates.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* perf(build): limit release artifacts to the rustfs server
Build only rustfs in the release workflow and remove the CLI build,
strip, and packaging paths. Preserve GNU profiling symbols and catalog
sidecars, and align strict acceptance archive readers with the new set.
Exercise all six build targets, Linux package checks, and all four
strict acceptance extractors using their actual workflow scripts.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: majinghe <42570491+majinghe@users.noreply.github.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(ci): restore mainline and scheduled test reliability
* fix(ci): provide GitHub CLI for CPU acceptance
* ci: provide Docker for CPU service acceptance
* ci: provision Python and Docker for OIDC validation
* ci: restore hosted runners for Docker validation
* ci: use verified MinIO release packages for interop
* ci: preserve host ownership of MinIO fixtures
* fix(ci): correct diagnostic limits and isolate startup checks
* test(connect): include object CLI failure details
* test(readiness): initialize unavailable drive diagnostics
#8112 replaced ubuntu-latest with sm-standard-2 across workflows. Both nightly
KMS Vault lanes need a working Docker daemon; the sm-standard-2 pool does not
provide one, so both lanes die in seconds on 'Docker is not available' while
the package compiles and publishes fine - and the functional chain never fires
because its gate requires a successful build run. Restore the lanes to
ubuntu-latest, as documented in the lane comment before #8112.
* ci: replace ubuntu-latest runner with sm-standard-2 across workflows
* ci: keep scheduled-validation monitors on hosted runners
The freshness and watchdog jobs report stalled scheduled validations.
Running them on the same sm-standard-2 pool means a pool outage stalls
the monitors too, so nothing reports it.
---------
Co-authored-by: overtrue <anzhengchao@gmail.com>
* fix(package): preserve service state across upgrades
* fix(package): match legacy DEB versions in tilde form
Published prerelease packages carry ~ in the dpkg control version
(package_versions.sh maps the SemVer prerelease - to ~), so the
legacy fallback list written with dots never matched 1.0.0~rc.x and
upgrades away from those DEBs still left the service stopped (#8011).
Fix the legacy glob to the tilde form and update the contract test,
which had enshrined the dot form. Verified on Ubuntu 24.04 systemd
containers: DEB upgrade 1.0.0~rc.5 -> 1.0.1 now keeps the service
running; 1.0.0 -> 1.0.1 and 1.0.1 -> 1.0.2 marker path still pass.
* docs(package): expand /etc/default/rustfs example template
Document the commonly used RUSTFS_* settings as commented examples in
the packaged conffile and point to docs.rustfs.com.
Scheduled pipeline failures currently open [scheduled-failure] tracking
issues in rustfs/rustfs itself, adding bot noise to the code repository's
issue tracker. Route that alerting to rustfs/backlog instead.
- schedule-failure-issue gains target-repository (default rustfs/backlog);
dedupe lookup, comment appends and issue creation target that repository.
- New source-token input (default ${{ github.token }}) reads the failed-job
list from the reported run; the run still lives in and links to
rustfs/rustfs. The issue body now carries a '- Repository:' line.
- All consumer workflows pass secrets.BACKLOG_ISSUE_TOKEN; the
workflow-scoped GITHUB_TOKEN cannot open issues cross-repo. Alert job
permissions blocks are unchanged.
feat(nightly): publish packages as assets of the rolling 'nightly' release (#7592)
Replace the assets-branch scheme with a proper GitHub Release on
rustfs/auto-testing: a single 'nightly' release whose deb/rpm assets
are replaced in place on every build. This is the standard channel —
visible on the repo's Releases page, stable download URLs, no git
history growth (release assets live outside the repository).
- New scripts/release/publish_nightly_assets.sh: resolves-or-creates
the 'nightly' release via the REST API, deletes same-name assets,
uploads rustfs-nightly-latest.{deb,rpm}, then PATCHes the release
body with the build provenance (ref@sha, run link, sizes, SHA256).
Plain curl + python3, no gh CLI (the build fleet has none — #7586).
- The workflow step shrinks to invoking the script; full flow
exercised end-to-end against the real release with probe files
(create / upload / overwrite / download round-trip / body update).
The sm-standard-4 runners used by the nightly build have no gh binary
(functional-chain workflows run elsewhere, on the jumpbox). The assets
publish step died on 'gh: command not found' at the credential-helper
setup, and the preceding clone failure had been masked by 2>/dev/null,
misleading the step into the orphan path. Swap clone and remote setup
to plain git with the token embedded in the URL; push semantics are
unchanged.
* ci(nightly): persist the nightly deb on Cloudflare R2
Upload the deb to artifacts/rustfs/packages/nightly/ (dated name plus a
rustfs-nightly-latest.deb alias) through the same R2 channel package.yml
uses, so the nightly package can be downloaded later with a stable URL.
The step is skipped when the R2 secrets are not configured, keeping the
artifact-only mode intact.
* test: add pool expansion / decommission E2E script and workflow
Add the admin-API based pool expansion, rebalance and decommission test
script (scripts/test/rustfs_pool_expand.sh) plus a workflow_dispatch /
nightly workflow that runs it on a self-hosted runner against real nodes.
The workflow accepts a release tag or a direct .deb URL (e.g. nightly/R2
package) via the package_url input.
* ci(pool-test): run the pool expansion test on the smoke-testing runner
The nightly GNU build now also packages the release binary as
rustfs-nightly-<YYYY-MM-DD>.deb (Asia/Shanghai date, matching the schedule
timezone) and uploads it as a workflow artifact. Packaging mirrors
package.yml: DEBIAN control/conffiles and the systemd service from
deploy/build/, built with fakeroot dpkg-deb.
No workflow ever set RUSTFS_KMS_VAULT_TOKEN, so live_vault_backends() returned an empty set in every CI run and behavior_rotation.rs never asserted the working half of rotate/versioning; the #[ignore] live-Vault tests had never executed in CI either. nightly-gnu.yml gains a kms-vault-lane job (vault server -dev with KV2 + Transit, full rustfs-kms suite with the lane on, the dev-Vault ignored tests, and the AppRole live script) plus a separate kms-vault-ha-failover job for the three-node Raft failover script, isolated so an election-timing flake cannot mask the main lane's verdict. GitHub-hosted ubuntu-latest rather than the self-hosted fleet: the HA script needs Docker, and e2e-s3tests.yml's banner records how the heterogeneous sm-standard pods burned the last docker-dependent workflow.
The behavior harness now records every key TestKms::create_key mints and deletes them after each Vault-backed for_each_backend case, on a fresh manager over the same configuration with the immediate-deletion gate enabled for cleanup only. Transit needs the deletion issued twice (first call parks the key in PendingDeletion, the second destroys it); KV2 destroys on the first call. Verified against a real dev Vault: after a full suite run the server holds zero behavior-* keys.
Also fixes test_vault_cancel_key_deletion_persists_state, which was broken by construction — Default::default() never picks up the insecure-dev-defaults env override, so the HTTP dev Vault the test requires was always refused. It now declares development mode on the config, and passes.
Refs rustfs/backlog#1774, rustfs/backlog#1562.
Chris