Compare commits

...

22 Commits

Author SHA1 Message Date
overtrue 9d513cffe9 test(e2e): update smoke selection for reference-binary unit tests 2026-11-11 11:25:30 +08:00
overtrue c5afd3738c chore: merge gateway/integration into gw/T3-reference-binary 2026-11-11 10:33:04 +08:00
Chris An fc09148f08 chore(gateway): record ignored microbenchmarks merged from main (#9290)
chore(gateway): record the ignored microbenchmarks merged from main

main added three throughput microbenchmarks that are ignored by design
(#9217): one each in rustfs-checksums, rustfs-rio and rustfs-utils. The
test-count ratchet treats a higher ignored count as a failure until the
row is raised, so raise those three rows to 1 and tighten the test rows
to the merged tree with --update.
2026-11-10 21:30:34 -05:00
rustfs-cla-assistant[bot] c9e1464a2a Merge remote-tracking branch 'origin/main' into gateway/integration 2026-11-11 02:06:08 +00:00
Chris An 0e3028d29e ci(gateway): make the test-count ratchet a floor, not exact (#9278)
ci(gateway): make the test-count ratchet a floor instead of an exact baseline

Tests below a row still fail; tests above it, a member without a row and ignores below the row now pass with a note. Ignores above a row still fail. Adds contract tests wired into make script-tests.
2026-11-10 20:53:06 -05:00
overtrue 6b44869b4f test(e2e): allow a pinned release binary as differential reference side
RUSTFS_E2E_REFERENCE_BINARY makes side A of the dual-stack differential a released rustfs binary, started without RUSTFS_S3_STACK. A missing, non-file or non-executable path fails the run instead of falling back to the same-binary mode. The report records which side is the reference and each side's --version output.
2026-11-11 09:44:12 +08:00
Chris An 365014a385 feat(scripts): add junit_diff.py for dual-stack JUnit comparison (#9275) 2026-11-10 19:55:24 -05:00
Ramakrishna Chilaka 89d01aef4f perf(ecstore): reduce syscalls and directory re-opens in PUT and DELETE (#9264)
Co-authored-by: RustFS <hello@rustfs.com>
Co-authored-by: Hauser Zhi <housemecn@gmail.com>
2026-11-11 00:06:11 +00:00
dependabot[bot] 86c3395d07 chore(deps): bump the dependencies group across 1 directory with 12 updates (#9251)
Bumps the dependencies group with 12 updates in the / directory:

| Package | From | To |
| --- | --- | --- |
| [async-nats](https://github.com/nats-io/nats.rs) | `0.51.0` | `0.52.0` |
| [tar-codec](https://github.com/astral-sh/astral-codec) | `0.0.15` | `0.0.16` |
| [tar-framing](https://github.com/astral-sh/astral-codec) | `0.0.15` | `0.0.16` |
| [flate2](https://github.com/rust-lang/flate2-rs) | `1.1.10` | `1.1.11` |
| [smallvec](https://github.com/servo/rust-smallvec) | `1.16.4` | `1.16.5` |
| [starshard](https://github.com/houseme/starshard) | `2.4.0` | `2.4.1` |
| [zip](https://github.com/zip-rs/zip2) | `9.3.0` | `9.4.1` |
| [dial9](https://github.com/dial9-rs/dial9) | `0.5.7` | `0.5.8` |
| [rustfs-mimalloc](https://github.com/houseme/rustfs-mimalloc) | `0.7.3` | `0.7.4` |
| [hotpath](https://github.com/hotpath-rs/hotpath) | `0.30.3` | `0.31.1` |
| [rustfs-mimalloc-sys](https://github.com/houseme/rustfs-mimalloc) | `0.7.3` | `0.7.4` |
| [toml](https://github.com/toml-rs/toml) | `1.2.0+spec-1.1.0` | `1.2.1+spec-1.1.0` |



Updates `async-nats` from 0.51.0 to 0.52.0
- [Release notes](https://github.com/nats-io/nats.rs/releases)
- [Commits](https://github.com/nats-io/nats.rs/compare/async-nats/v0.51.0...async-nats/v0.52.0)

Updates `tar-codec` from 0.0.15 to 0.0.16
- [Release notes](https://github.com/astral-sh/astral-codec/releases)
- [Commits](https://github.com/astral-sh/astral-codec/compare/v0.0.15...v0.0.16)

Updates `tar-framing` from 0.0.15 to 0.0.16
- [Release notes](https://github.com/astral-sh/astral-codec/releases)
- [Commits](https://github.com/astral-sh/astral-codec/compare/v0.0.15...v0.0.16)

Updates `flate2` from 1.1.10 to 1.1.11
- [Release notes](https://github.com/rust-lang/flate2-rs/releases)
- [Changelog](https://github.com/rust-lang/flate2-rs/blob/main/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/flate2-rs/compare/1.1.10...v1.1.11)

Updates `smallvec` from 1.16.4 to 1.16.5
- [Release notes](https://github.com/servo/rust-smallvec/releases)
- [Commits](https://github.com/servo/rust-smallvec/compare/v1.16.4...v1.16.5)

Updates `starshard` from 2.4.0 to 2.4.1
- [Changelog](https://github.com/houseme/starshard/blob/main/CHANGELOG.md)
- [Commits](https://github.com/houseme/starshard/compare/2.4.0...2.4.1)

Updates `zip` from 9.3.0 to 9.4.1
- [Release notes](https://github.com/zip-rs/zip2/releases)
- [Changelog](https://github.com/zip-rs/zip2/blob/master/CHANGELOG.md)
- [Commits](https://github.com/zip-rs/zip2/compare/v9.3.0...v9.4.1)

Updates `dial9` from 0.5.7 to 0.5.8
- [Release notes](https://github.com/dial9-rs/dial9/releases)
- [Changelog](https://github.com/dial9-rs/dial9/blob/main/CHANGELOG.md)
- [Commits](https://github.com/dial9-rs/dial9/compare/dial9-v0.5.7...dial9-v0.5.8)

Updates `rustfs-mimalloc` from 0.7.3 to 0.7.4
- [Release notes](https://github.com/houseme/rustfs-mimalloc/releases)
- [Changelog](https://github.com/houseme/rustfs-mimalloc/blob/main/CHANGELOG.md)
- [Commits](https://github.com/houseme/rustfs-mimalloc/compare/v0.7.3...v0.7.4)

Updates `hotpath` from 0.30.3 to 0.31.1
- [Release notes](https://github.com/hotpath-rs/hotpath/releases)
- [Changelog](https://github.com/hotpath-rs/hotpath/blob/main/CHANGELOG.md)
- [Commits](https://github.com/hotpath-rs/hotpath/compare/v0.30.3...v0.31.1)

Updates `rustfs-mimalloc-sys` from 0.7.3 to 0.7.4
- [Release notes](https://github.com/houseme/rustfs-mimalloc/releases)
- [Changelog](https://github.com/houseme/rustfs-mimalloc/blob/main/CHANGELOG.md)
- [Commits](https://github.com/houseme/rustfs-mimalloc/compare/v0.7.3...v0.7.4)

Updates `toml` from 1.2.0+spec-1.1.0 to 1.2.1+spec-1.1.0
- [Commits](https://github.com/toml-rs/toml/compare/toml-v1.2.0...toml-v1.2.1)

---
updated-dependencies:
- dependency-name: async-nats
  dependency-version: 0.52.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: dependencies
- dependency-name: tar-codec
  dependency-version: 0.0.16
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: tar-framing
  dependency-version: 0.0.16
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: flate2
  dependency-version: 1.1.11
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: smallvec
  dependency-version: 1.16.5
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: starshard
  dependency-version: 2.4.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: zip
  dependency-version: 9.4.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: dependencies
- dependency-name: dial9
  dependency-version: 0.5.8
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: rustfs-mimalloc
  dependency-version: 0.7.4
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: hotpath
  dependency-version: 0.31.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: dependencies
- dependency-name: rustfs-mimalloc-sys
  dependency-version: 0.7.4
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: toml
  dependency-version: 1.2.1+spec-1.1.0
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Hauser Zhi <housemecn@gmail.com>
2026-11-11 00:04:47 +00:00
cxymds c9743f166e fix(s3): normalize leading-slash keys in bulk deletes (#9258)
* fix(s3): normalize leading-slash keys in bulk deletes

* test(s3): align bulk delete reserved-prefix guard expectations
2026-11-10 17:56:00 -05:00
Boyi (Elliot) Li 50707a610c test(utils,rio,checksums): add checksum SIMD equivalence and bench UTs (#9217)
Add unaligned-boundary SIMD vs scalar equivalence unit tests and on-demand (#[ignore]) utility-level throughput microbenchmarks across rustfs-utils, rustfs-rio, and rustfs-checksums:
- crates/utils/src/hash.rs: verify HighwayHasher SIMD (AVX2/SSE4.1/NEON) against highway::PortableHash scalar reference and contiguous hash_encode vs odd-chunked hash_encode_slices across unaligned tail boundaries; provide on-demand throughput microbenchmark.
- crates/rio/src/checksum.rs: verify SIMD-accelerated CRC32, CRC32C, and CRC64_NVME against bitwise scalar reference implementations across vector tail boundaries and measure utility-level throughput across all 10 BASE_CHECKSUM_TYPES.
- crates/checksums/src/lib.rs: verify streaming chunked update + finalize + headers() against one-shot digests across all 9 ChecksumAlgorithm variants for unaligned chunk and payload sizes, plus on-demand throughput microbenchmark.
2026-11-10 17:54:11 -05:00
Patrick Wicki ba3e0e1546 fix(admin): keep measured bucket usage when the overlay is unconverged (#9243)
The single-process usage overlay clears usage_snapshot_complete when it
holds counts it cannot reconcile with the persisted snapshot, for
example writes into a bucket that has no verified baseline. The admin
datausageinfo path then judged the overlaid document as structurally
incomplete and replaced it with an empty one, hiding every bucket the
scanner had measured. Because each write restarts that bucket's scanner
fence, a continuously written bucket kept the whole response empty
indefinitely.

Decide whether the persisted snapshot is usable before the overlay is
applied, and report overlay-induced incompleteness as a non-converged
snapshot instead, as is already done for buckets the scan never reached
(rustfs#5806).
2026-11-10 17:52:06 -05:00
Chris An 95e128336d ci: narrow checks for stable installation metadata (#9269)
* ci: narrow checks for stable installation metadata

* fix: reject release metadata marker collisions
2026-11-10 16:10:04 -05:00
Chris An 8b7bbdb17d chore(release): align installation references for 1.0.5 (#9268) 2026-11-10 14:48:02 -05:00
Chris An 2780bba67b ci: stage release builds and promote verified artifacts (#9265)
* ci: stage release builds and promote verified artifacts

* fix: preserve release package and channel retry contracts

* fix: align installation chart versions with sealed candidates
2026-11-10 13:15:05 -05:00
overtrue 443e8afee1 chore(gateway): merge main into integration
# Conflicts:
#	crates/ecstore/src/disk/local/delete_recovery.rs
#	crates/ecstore/src/store/bucket.rs
2026-11-10 21:40:26 +08:00
hector 3a23122104 ci: run macOS release builds on macos-latest-xlarge (#9260)
Use the larger GitHub-hosted macOS runner for the macos-aarch64
release target. The target id, triple, and other platforms stay the same.

Co-authored-by: hector <majinghe@users.noreply.github.com>
2026-11-10 18:55:19 +08:00
Ramakrishna Chilaka 0b1ebeffec perf(ecstore): reduce syscalls in multipart copy, commit, and delete (#9247) 2026-11-10 12:27:10 +08:00
Chris An 0f9258594c fix: preserve scanner backlog authority after pool replacement (#9252)
* fix(ecstore): allow bucket creation after failed decommission

* fix(scanner): preserve backlog membership across pool reindexing

* test(scanner): cover partial publication after pool reindexing
2026-11-09 20:46:20 -05:00
overtrue 782e3e5209 chore(gateway): merge main into integration 2026-11-10 08:28:05 +08:00
Chris An 33fc735ee2 chore(release): prepare 1.0.5 (#9249) 2026-11-09 17:19:35 -05:00
Chris An 509869cfc5 fix(tier): recover orphaned prepared mutations (#9246) 2026-11-09 22:18:14 +08:00
110 changed files with 7938 additions and 4258 deletions
+21 -26
View File
@@ -1,43 +1,38 @@
---
name: rustfs-release-publish
description: Publish a RustFS version through Console, release-branch CI, preview acceptance, human confirmation, and final publication. Use only for an explicit release request.
description: Publish a RustFS version through frozen Linux candidate acceptance, desktop completion, manifest approval, byte-preserving publication and post-release follow-up. Use only for an explicit release request.
---
# RustFS Release
Require the exact final SemVer target before version edits or publication. If unspecified, inspect current releases/tags and ask with concrete candidates; do not guess the channel or bump. A named bump type resolves to an exact version that still needs confirmation. Report the confirmed target verbatim; resolve any later mismatch before proceeding. Independent read-only preflight may continue.
Require the exact final SemVer target before version edits or publication. If unspecified, inspect releases/tags and ask with concrete candidates; do not guess the channel or bump. Reuse the user's confirmed target and authorized delivery scope. Independent read-only preflight may continue.
## Invariants
- Cargo stores `<target>`, never `-preview.N`. Before publication, installation defaults retain the previous available deliverable. Final and preview binaries embed their own tag names, so version/asset names differ despite identical source.
- Use annotated tags without `v`. Preview names are exactly `<target>-preview.<digits>`; choose the next unused iteration after fetching tags. Preview Releases are prereleases, never Latest; publish only versioned assets/exact Docker tags, never latest assets, R2, Helm, moving Docker channels, or installation/banner updates.
- `release/<target>` owns the candidate. Retain its history, backport only reviewed fixes, and never merge main wholesale or force-push it. Pin `PREVIEW_HASH` to its exact commit with complete passing CI. Preview and final tags both point there; never tag a later main/branch HEAD or insert another bump.
- Phase failures block dependents. Keep preview Releases until final publication and notes generation finish; CI removes Releases while retaining tags. Never move an existing preview or final tag.
- Cargo and every candidate executable contain the final product version. `<target>-preview.<run_number>` is an immutable candidate identifier, not a public tag or displayed binary version.
- Freeze the exact source, reviewed workflow recipe, toolchain, Console, CLI, dependency lock, previous stable version and validation plan. Newer unrelated main or Console releases do not change those inputs.
- Retain `release/<target>` history when a release branch is used; backport only reviewed fixes. Pin `CANDIDATE_SHA` to its exact CI-validated commit. Never force-push a release branch or move a public tag.
- Build Linux first. Complete field acceptance before macOS/Windows and packaging. Full certification and final acceptance are required before publication; a partial Linux PASS grants no release authority.
- Promote the accepted bytes. Final publication never recompiles binaries or rebuilds packages/images. Infrastructure retries reuse successful artifacts; changed source or inputs require a new candidate and affected acceptance.
- Phase failures block dependents. No latest/default-channel update before versioned delivery. Use the [pipeline runbook](../../../docs/operations/release-pipeline.md) for executable entry points, locks and recovery.
## Read by Phase
Read only the current phase's procedure; retain completed evidence across turns. Follow root verification, Git/PR, and authorization rules.
## Read by phase
| Phase | Procedure |
|---|---|
| 0 | Clean tree, fetch main/tags, verify GitHub access, then complete the mandatory [Console release gate](references/console-gate.md). A green Console build alone is insufficient. |
| 1–2 | [Prepare/reuse the release candidate and run exact-SHA branch CI](references/release-candidate.md). Source version changes use `rustfs-release-version-bump` with authorized delivery scope. |
| 3 | [Verify the preview build, assets, channels, and notes](references/preview-release.md). |
| 4–5 | [Run the downloaded binary, Console CRUD, and latest-rc matrix](references/preview-acceptance.md). Every check must pass. |
| Confirmation | Complete the gate below and end the turn. |
| 6 | After fresh confirmation, [publish and verify the final tag](references/final-release.md) at `PREVIEW_HASH`. |
| 7 | [Reconcile Security Advisories, milestones, installation references, and the website; reconcile main and remove the completed release branch](references/post-release-updates.md) only after complete publication. |
| 0 | Clean tree, fetch main/tags, verify access, then complete the mandatory [Console gate](references/console-gate.md). |
| 1–2 | [Prepare the exact source and run branch CI](references/release-candidate.md). Version edits use `rustfs-release-version-bump`. |
| 3 | [Build and verify the Linux candidate](references/preview-release.md). |
| 4–5 | [Run artifact, field, Console and rc acceptance](references/preview-acceptance.md), then complete desktop/package/image/chart validation. |
| Confirmation | Report the complete manifest and obtain the confirmation below. |
| 6 | [Publish the accepted manifest](references/final-release.md). |
| 7 | [Reconcile advisories, milestones, installation references, website and release branch](references/post-release-updates.md). |
## Human Confirmation
## Human confirmation
After Phases 3–5 pass, recheck that the release branch still equals `PREVIEW_HASH` and that the current Console asset tag/digest match the preview's recorded build inputs. Report the target, branch, preview tag/hash, branch CI run, preview Release URL, accepted Console asset identity/result, and rc matrix. Explicitly ask whether to publish the final tag and end the turn without creating/pushing it.
After all acceptance passes, report the target, source SHA, source CI run, Linux and completion run URLs, final manifest SHA-256, accepted Console/CLI identities, compatibility/performance results and backlog report. Explicitly ask whether to publish this exact manifest and end the turn without dispatching publication.
Only a new affirmative user reply to that exact report authorizes Phase 6. The original release request, earlier confirmation, silence, or automated follow-up does not count. Confirmation binds the target, preview tag, hash, and accepted build inputs; a changed value or repeated/failed acceptance cycle invalidates it. Revalidate and obtain fresh confirmation before continuing.
Only a new affirmative reply to that exact report authorizes Phase 6. An earlier general release request, silence or automatic continuation does not count. Reuse confirmation for retries of that same manifest; do not ask again merely because a destination failed. A changed source, input, manifest or acceptance result invalidates it and requires renewed acceptance/confirmation.
## Recovery
## Recovery and reporting
- Before the final tag exists, land fixes on main, backport to the candidate, verify target versions, and restart branch CI/acceptance. Use a new preview iteration once a preview was tagged; main may advance independently.
- If only an external build input changed, record it, create a new preview iteration even when the source SHA is unchanged, and repeat the affected build and artifact acceptance before fresh confirmation. Reuse source checks whose inputs remain valid.
- Once the final tag exists, retry publication jobs for that exact tag. A source fix requires a newly confirmed target, not retagging or a newer preview. Phase 7 retries recheck artifact availability and superseding releases.
- If abandoned after the bump merged, report the unpublished version on main and record whether to revert or let the next release overwrite it. Deleting unpublished candidate branches or preview tags requires an explicit decision. Completed release branches follow Phase 7 cleanup; keep release tags.
Report completed phase results and incomplete work with evidence: Console previous/latest tags, release need, hash and run/Release URLs; target, branch CI/SHA, previews and hash, confirmation state; preview/final assets, latest-channel state, Console/rc results and cleanup; advisory reconciliation results and pending disclosure decisions; both milestone states, installation PR checks/merge state, and website text/link/PR/deployment state. Keep artifact publication separate from follow-up delivery; do not call an open PR a live update. Report deviations and their authorization.
Resume failed infrastructure jobs without rebuilding successful units. Fix product defects on main, backport when needed, and start a new candidate. After a final tag exists, retry the same bytes or use a new patch version; never move the tag. Do not conflate implemented, CI-passed, field-accepted, approved, published and post-release-complete states. Report incomplete work and deviations with evidence; a tag or green build alone is not a completed release.
@@ -4,7 +4,7 @@ Read during Phase 0, before changing RustFS version files or tags. Follow the pa
### Console release gate
Complete this gate before changing any RustFS version file or creating any RustFS tag. RustFS `build.yml` downloads the asset returned by `repos/rustfs/console/releases/latest`, so a successful Console build alone is insufficient.
Complete this gate before changing any RustFS version file or creating any RustFS tag. Candidate creation resolves `repos/rustfs/console/releases/latest` once and freezes the exact asset for every build, so a successful Console build alone is insufficient. After freezing, later Console releases do not silently change or invalidate the accepted inputs.
1. Read the latest published Console tag and compare it with Console `main`:
@@ -1,22 +1,13 @@
# Final Release Publication
# Final release publication
Read only after fresh user confirmation of the preview acceptance report. The [parent confirmation gate](../SKILL.md#human-confirmation) applies.
Read only after fresh confirmation of the complete candidate report. The [parent confirmation gate](../SKILL.md#human-confirmation) applies.
## Phase 6 — Publish the final tag on the validated commit
Dispatch `release-publish.yml` on main with the successful completion run ID and approved `release-manifest.json` SHA-256. Do not create the tag manually. The workflow revalidates source, bytes and trusted certification, then creates the annotated final tag at `CANDIDATE_SHA`. A recipe change fails closed and requires a new candidate; unrelated main changes do not require a rebuild.
No second source version bump. Before creating the final tag, recheck the remote `release/<target>` HEAD equals `PREVIEW_HASH` and the current Console release tag/asset digest still match the accepted preview inputs. If either changed after confirmation, the approval no longer applies; validate the changed inputs with a new preview and obtain fresh confirmation. The final tag goes on the exact commit the preview validated:
Require both versioned publication and channel publication jobs to succeed. Verify the final tag's commit and manifest identity; all six platform ZIPs and compatibility aliases; DEB/RPM, checksums, SBOM, provenance and evidence bundle; both multi-architecture image variants in Docker Hub, GHCR and Quay; and the accepted Helm chart in the chart repository. Download a host-compatible final artifact and confirm its digest is exactly the accepted candidate digest and its version/source output agrees.
```bash
git fetch origin --tags
git rev-parse "<preview-tag>^{commit}" # must equal PREVIEW_HASH — abort if not
git tag -a "<target>" -m "Release <target>" "$PREVIEW_HASH"
git push origin "<target>"
```
For stable releases verify GitHub Latest, R2 download aliases and `latest.json` unless a newer stable release already owns those channels. Prereleases must remain GitHub prereleases and leave stable pointers unchanged. Verify generated release notes compare against the frozen previous stable release, excluding internal previews.
- CI rebuilds from the same source with a different tag name, so the binary self-reports `<target>`. Other inputs can change: `build.yml` reads Console `releases/latest` during each build. Compare the actual Console tag, URL, and verified digest in every final build job with the accepted preview record, and inspect available build provenance for other relevant changes. A pre-tag check cannot prevent a later input change. On a mismatch or missing evidence, repeat affected acceptance against the final artifacts and report the limitation; failed acceptance leaves release validation incomplete even if publication already succeeded. Never move the final tag to conceal a failure.
- Verify the final tag's complete publication path: all matrix and release jobs green; `gh release view "<target>"` shows the full versioned and `-latest` asset set plus checksums, SBOM, and provenance; Docker and Helm workflows succeed; `latest.json` points to `<target>`. A stable target must have `isPrerelease=false` and `isLatest=true`. An alpha/beta/rc target must have `isPrerelease=true`; GitHub does not permit prereleases to be Latest, but the project `latest.json` still advances to the final non-preview target.
- Verify the final Release body contains `## What's Changed` and a Full Changelog link. When `PREVIOUS_DELIVERABLE` exists, the link MUST be `https://github.com/rustfs/rustfs/compare/<PREVIOUS_DELIVERABLE>...<target>` and the baseline MUST equal the preview Release baseline; for example, both `1.0.0-beta.12-preview.1` and `1.0.0-beta.12` compare from `1.0.0-beta.11`.
- Verify the preview cleanup: `cleanup-preview-releases` must succeed and `gh release view "<preview-tag>"` must then report `release not found` for every preview iteration of this target. Each surviving preview tag must still resolve to its own recorded commit; the accepted preview resolves to `PREVIEW_HASH`. If the job failed, delete the leftover Releases manually with `gh release delete "<preview-tag>" --yes` and report it.
- Verify a host-compatible final artifact's checksum and `./rustfs --version`; it must report `<target>` and the expected source SHA. This identity check does not replace affected acceptance when build inputs changed.
On infrastructure failure retry the same workflow/seal; matching destinations are reused. Never clobber an asset, force a tag, rebuild the image or silently accept a changed digest. A source fix after the final tag requires a new patch version. Cross-registry writes are not atomic: report the unfinished destination rather than claiming publication complete.
Never pass `--cleanup-tag` when removing preview Releases. Remove them manually only if `cleanup-preview-releases` failed, after final publication and notes generation. Record deleted Releases and surviving tags.
New candidates create no preview Releases to clean up. For a legacy preview release, preserve its tag and remove its Release only after final delivery and notes are verified. Do not pass `--cleanup-tag`. Complete the [post-release follow-up](post-release-updates.md) separately.
@@ -24,7 +24,7 @@ Reconcile advisories after every completed final release, including Console-only
1. Re-read the published deliverable state before edits or a retry. If a newer release has superseded this target, do not downgrade installation references or the website announcement. Report the superseding release and leave those defaults intact.
2. Invoke `rustfs-release-version-bump` with stage `post-release`, the exact target, and the authorized delivery scope. Prepare the follow-up on current main; do not edit or move either validated release tag. Cargo versions may already be preparing the next target and must remain untouched.
3. Verify the exact Docker image manifest, Release download links, RPM source archive, and Helm repository entry before publishing the installation PR. The chart workflow overrides chart/app versions from the final tag, pins the source checkout to that release, and waits for its default Docker image before packaging.
3. Verify the exact Docker image manifest, Release download links, RPM source archive, and Helm repository entry before publishing the installation PR. The chart is packaged and tested before sealing; publication uploads those same bytes only after versioned Docker images are available.
4. Report the installation PR and its merge state separately from the completed release. Reuse existing merge authorization and repository checks; do not infer new merge authority from this reference.
## rustfs.com top banner
@@ -54,4 +54,4 @@ Use the existing announcement configuration from [website PR #112](https://githu
1. Compare the exact published tag with current main. Audit every release-side commit since their merge base, including inherited backports. An ahead count is a history difference, not proof of missing changes. Use ancestry and patch equivalence, then inspect current code and regression evidence for squashed, adapted, superseded, or reverted changes. Record each commit as present, superseded, release-only, or missing; a positive `git cherry` result alone does not prove a missing fix.
2. Bring missing shared changes to main through a reviewed PR and the required checks before closing the follow-up. Preserve main's later changes; decide version bumps and pinned Console configuration separately. If a history merge is needed, use a merge commit and verify the release commit is an ancestor of main; squash does not establish that ancestry. Do not merge main into the published release branch or move its tags. Report unresolved changes instead of claiming reconciliation is complete.
3. After publication and authorized follow-ups are complete, delete the remote `release/<target>` branch. Immediately before deletion, verify its HEAD still equals the published tag's commit and no open PR targets or originates from it. Preserve a branch with newer work or open PRs and report the blocker. Use a deletion with an expected-old-SHA check to avoid removing a concurrently advanced branch.
4. Verify the remote branch is absent and the final and preview tags still resolve to their recorded commits. Remove a local release branch only if it is task-owned, unchecked out, and has no unpreserved work. Record cleanup separately from artifact publication; an explicitly authorized deletion before reconciliation does not make unresolved main changes complete.
4. Verify the remote branch is absent and the final tag and any legacy preview tags still resolve to their recorded commits. Remove a local release branch only if it is task-owned, unchecked out, and has no unpreserved work. Record cleanup separately from artifact publication; an explicitly authorized deletion before reconciliation does not make unresolved main changes complete.
@@ -1,10 +1,10 @@
# Preview Artifact Acceptance
# Candidate Artifact Acceptance
Read after Phase 3 succeeds. Complete every check below before the parent skill's manual confirmation gate. These checks cover the downloaded artifact, embedded Console, and latest rc client.
Read after Phase 3 succeeds. Complete every check below before the parent skill's manual confirmation gate. These checks cover the downloaded artifact, embedded Console, and the rc release frozen in the candidate inputs.
## Phase 4 — Run the artifact locally, verify the console
Use a unique task-owned scratch directory. Select the versioned archive matching the host OS, architecture, and runtime, download its checksum file, and verify the archive before extraction. Do not substitute a different build when a matching artifact is unavailable; report that acceptance gap. The shell example below assumes the verified executable is at `RUSTFS_BINARY` (`rustfs.exe` on Windows), `SCRATCH` is the task directory, and `S3_PORT` and `CONSOLE_PORT` are distinct available ports. Use equivalent native commands when the shell differs.
Use a unique task-owned scratch directory. Select the versioned archive matching the acceptance host, and verify it against the sealed manifest before extraction. Before desktop completion, use the Linux acceptance environment rather than rebuilding for the local desktop. Do not substitute a different build when a matching artifact is unavailable; report that acceptance gap. The shell example below assumes the verified executable is at `RUSTFS_BINARY` (`rustfs.exe` on Windows), `SCRATCH` is the task directory, and `S3_PORT` and `CONSOLE_PORT` are distinct available ports. Use equivalent native commands when the shell differs.
Bind both listeners to loopback. Choose task-only credentials, set loopback proxy exclusions for the server/client processes, and record the server PID. A port probe does not reserve a port; retry startup with fresh ports on a bind conflict rather than connecting to an existing service.
@@ -24,16 +24,16 @@ PREVIEW_SERVER_PID=$!
Checks (all must pass):
- `"$RUSTFS_BINARY" --version` reports the preview tag name and the short SHA of `PREVIEW_HASH`. Reporting `<target>` without the `-preview.N` suffix means the build did not embed the tag — treat as FAIL and investigate before proceeding.
- `"$RUSTFS_BINARY" --version` reports the exact final `<target>` and full `CANDIDATE_SHA`. A preview suffix, another version or another source is a failure.
- Wait with a bounded readiness deadline while checking the recorded server process remains alive; `curl --noproxy '*' -fsS "$S3_ENDPOINT/health/ready"` must return ready from this process's listener.
- Startup log shows the embedded console being served (this was the regression that `fix(release): require embedded console assets` guards).
- Open `CONSOLE_ENDPOINT` in the browser and log in with the task credentials. The dashboard must render without JS console errors; create a bucket, upload a file, download it back (byte-identical), and delete the object and bucket. Keep the server running for Phase 5.
## Phase 5 — Validate with the latest rc client
## Phase 5 — Validate with the pinned rc client
`rc` is the RustFS CLI client from <https://github.com/rustfs/cli>.
- Resolve the latest release with `gh api repos/rustfs/cli/releases/latest --jq .tag_name`; verify the local client's version or download and verify a host-compatible client into the task directory. Set `RC_BIN` to that executable and record its version/digest. Avoid changing the user's installed client for this check.
- Read the CLI release and asset digests from `inputs.json`; verify the local client's version or download a host-compatible asset from that exact release into the task directory. If the host is a desktop, resolve its asset within the same pinned CLI release and record the digest; do not independently follow `latest`. Set `RC_BIN` to that executable and record its version/digest. Avoid changing the user's installed client for this check.
- Inspect the client's current help for an isolated configuration option and use it when supported. Otherwise choose a unique `PREVIEW_ALIAS` and confirm it is absent before creating it; never overwrite an existing alias. Use task-owned bucket/user names and input/output files. The names below are safe only inside the newly started isolated server.
- Point the client at `S3_ENDPOINT` with loopback proxy exclusions and run the command matrix, recording PASS/FAIL per command:
@@ -60,3 +60,9 @@ cmp <local-file> "$SCRATCH/out"
```
- Any FAIL blocks acceptance. On success or failure, stop only the recorded task process and remove only task-created aliases/data after retaining the needed failure evidence. Do not erase an existing client configuration or unrelated server data.
## Field acceptance and desktop completion
Before accepting the Linux manifest, record Scanner/Heal, multi-pool, memory, major features, high-concurrency 503, performance and previous-version integrity/upgrade results in backlog. Preserve the release-specific agreed limits and baseline; a PASS label without artifact identities or test results does not satisfy acceptance.
Dispatch `release-complete.yml` only after that report passes, using the exact Linux manifest digest and backlog URL. Require native macOS/Windows checks, package extraction, image startup/scans, chart tests and complete certification to pass. Complete any desktop Console/client checks applicable to the release against these same archives. Record the completion run and final manifest SHA-256 before entering the parent confirmation gate. Partial Linux acceptance does not authorize final publication.
@@ -1,25 +1,9 @@
# Preview Release Verification
# Linux candidate verification
Read for Phase 3 after the candidate passed exact-SHA CI. The [parent release invariants](../SKILL.md) apply.
Read for Phase 3 after the source passed exact-SHA CI. The [parent invariants](../SKILL.md) apply; commands and recovery live in the [pipeline runbook](../../../../docs/operations/release-pipeline.md).
## Preview channels
Dispatch `release-candidate.yml` on main with the confirmed final version and `CANDIDATE_SHA`. Record the run ID, frozen `inputs.json`, every Linux archive digest and `linux-manifest.json` SHA-256. Require the four Linux builds, native startup/S3 checks, previous-release upgrade matrix and Linux seal to succeed. Missing, skipped, cancelled or failed required work is incomplete acceptance.
- Use `<target>-preview.N` for every target, e.g. `1.0.0-beta.10-preview.3` or `1.1.0-preview.1`.
- The canonical suffix is exactly `-preview.<digits>`. `build.yml` recognizes it before alpha/beta/rc classification and routes it to the preview-only path; any other tag containing `-preview` fails closed instead of being treated as a release.
- A preview Release MUST be published with `isPrerelease=true` and `isLatest=false`. Any `*-latest` preview asset or preview-triggered `latest.json`, R2, Helm, or moving Docker channel tag is a pipeline failure. Docker images may use the exact preview version tag, including its existing variant suffixes.
- Preview Releases are cleaned up by the `cleanup-preview-releases` job after `publish-release` succeeds for the deliverable tag. It deletes every Release whose tag is exactly `<target>-preview.<digits>` and never passes `--cleanup-tag`, so the tags survive.
Verify the frozen Console identity equals the one accepted in Phase 0. Every matrix job consumes that same digest and exact toolchain. The binary must report the final product version and source commit; its candidate ID is separate. No public tag, Release, registry tag, R2 release path, Helm repository or latest pointer should change during this stage.
## Phase 3 — CI and preview Release verification
- Find and watch the tag build: `gh run list --workflow build.yml --branch "<preview-tag>" --limit 1` then `gh run watch <run-id>`. Every build matrix target must succeed (linux x86_64/aarch64 × musl/gnu, macos-aarch64, windows-x86_64).
- Confirm the preview tag resolves to the Phase 2 CI-validated `PREVIEW_HASH`. If the release branch advances during acceptance, repeat Phases 2–5 with a new preview tag.
- Confirm the Release publication jobs (`create-release`, `upload-release-assets`, and `publish-release`) succeed while `update-latest-version` is skipped.
- Record the inputs actually consumed by each build job: source SHA and the embedded Console release tag, asset URL, and verified SHA256 from the download logs. The workflow reads Console `releases/latest` separately for each build; confirm the matrix used the Console asset accepted in Phase 0. A missing identity or differing asset blocks acceptance until resolved. Retain available toolchain and build provenance with the artifact checksums; the same source SHA alone does not establish equivalent builds.
- Verify `gh release view "<preview-tag>" --json isPrerelease,assets,url`: `isPrerelease` must be `true`, and the Release must contain all 6 versioned platform zips, checksums, SBOM, and provenance with no `-latest` assets. Confirm `gh api repos/{owner}/{repo}/releases/latest --jq .tag_name` does not return `<preview-tag>`.
- Record `PREVIOUS_DELIVERABLE`, selected from published Releases by `publishedAt` after excluding the current tag and every `-preview.N` tag. Verify `gh release view "<preview-tag>" --json body --jq .body` contains `## What's Changed` and, when `PREVIOUS_DELIVERABLE` exists, `**Full Changelog**: https://github.com/rustfs/rustfs/compare/<PREVIOUS_DELIVERABLE>...<preview-tag>`. For a repository with no previous deliverable, verify a Full Changelog link exists and record the GitHub baseline fallback.
- Confirm Helm is skipped. If preview Docker images are published, confirm they use only exact preview tags (and variant suffixes), with no `latest`, `alpha`, `beta`, or `rc` channel updates. Preview validation covers the built RustFS binaries, embedded console, and rc compatibility; it does not authorize advancing any installation default.
## Release notes
- When a previous deliverable exists, GitHub Release notes for the preview and final tags MUST use it as their shared comparison baseline: the most recently published non-preview Release before the target. Internal `-preview.N` Releases are explicitly excluded from that selection, even when they point at the same commit as the final tag — cleanup runs after the notes are generated, so the preview Release is still present and would otherwise be picked as the baseline. If no previous deliverable exists, omit `previous_tag_name` and record that GitHub's default baseline fallback was used.
- Generated Release notes carry a workflow-management marker so retries can repair them. Before manually curating a generated body, remove that marker; unmarked non-placeholder notes are preserved by later workflow runs.
A green Linux run is ready for field acceptance, not publication. The partial certification must have `publish_allowed: false`. Retain the exact Linux manifest for the next stage; a new source/build-input candidate must not inherit the old approval. A newer unrelated main commit does not invalidate this evidence.
@@ -16,22 +16,13 @@ Read for Phases 1–2. The [parent release invariants](../SKILL.md) and Console
- Only when neither the release branch, an unfinished preview tag, nor current main already has `<target>`: if main is at a later version, select and verify a historical commit containing `<target>` rather than downgrading main; block if none exists. Otherwise invoke the `rustfs-release-version-bump` skill with stage `prepare`, the final `<target>` (NOT a preview version), and full GitHub flow (commit/push/PR).
- If a bump PR was needed, get it merged into main and record its merge commit. For a new release branch when main already has `<target>`, identify the main commit that introduced those versions. Phase 2 chooses a release candidate containing that commit; an existing release branch keeps its own history and a newer main HEAD is not required.
Do not wait for the moving latest main HEAD to become green. The release-branch CI in Phase 2 validates the chosen commit before a preview tag is created.
Do not wait for the moving latest main HEAD to become green. The release-branch CI in Phase 2 validates the chosen commit before a candidate build is dispatched.
## Phase 2 — Pin and validate the release candidate, then publish the preview tag
## Phase 2 — Pin and validate the release source
- Use `release/<target>`. On the first attempt, create it from a selected main commit containing the version bump and intended changes; it need not be the latest main HEAD. On a restart, retain the existing branch and backport only reviewed fixes from main. Verify the branch's Cargo versions still equal `<target>`. If adopting an existing preview, run this branch CI before continuing its acceptance; reuse the tag only when it points to the validated SHA.
- The current `ci.yml` runs automatically on main pushes, not release-branch pushes. Dispatch it explicitly with `gh workflow run ci.yml --ref "release/<target>"`. Locate that dispatch run, verify its `head_sha` equals the release branch's remote HEAD, and require the full expected CI job set to pass, including `Quick Checks`, `Test and Lint`, and `End-to-End Tests (full merge gate)`. Treat a missing or skipped required lane as incomplete. A successful PR check or CI run for a different SHA does not qualify. If the branch advances or a run is cancelled, validate the new SHA again. Do not tag a candidate with failing or incomplete CI.
- Use `release/<target>`. On the first attempt, create it from a selected main commit containing the version bump and intended changes; it need not be the latest main HEAD. On a restart, retain the existing branch and backport only reviewed fixes from main. Verify the branch's Cargo versions still equal `<target>`. A legacy preview can anchor source selection, but its tag and old artifacts do not constitute a candidate in the new protocol; freeze and verify a new candidate after the release recipe is aligned.
- The current `ci.yml` runs automatically on main pushes, not release-branch pushes. Dispatch it explicitly with `gh workflow run ci.yml --ref "release/<target>"`. Locate that dispatch run, verify its `head_sha` equals the release branch's remote HEAD, and require the full expected CI job set to pass, including `Quick Checks`, `Test and Lint`, and `End-to-End Tests (full merge gate)`. Treat a missing or skipped required lane as incomplete. A successful PR check or CI run for a different SHA does not qualify. If the branch advances or a run is cancelled, validate the new SHA again. Do not dispatch a candidate with failing or incomplete CI.
- Prefer protecting `release/*` against force pushes and deletion and requiring reviewed backport PRs. Check live rulesets rather than assuming the main ruleset covers release branches; report a missing release-branch rule as a gap.
- Set `PREVIEW_HASH` to the CI-validated release-branch commit and recheck the remote branch HEAD immediately before tagging. Report the branch, commit, and CI run URL; do not derive `PREVIEW_HASH` from a later `origin/main` fetch. When adopting an existing preview tag at this SHA, skip tag creation and continue Phase 3.
```bash
git tag -a "<preview-tag>" -m "Release <preview-tag>" "$PREVIEW_HASH"
git push origin "<preview-tag>"
```
Pushing the tag triggers `.github/workflows/build.yml` ("Build and Release"); `docker.yml` chains off it via `workflow_run`.
The preview run builds versioned artifacts and publishes them in a GitHub prerelease. Docker may publish exact preview image tags. Latest-channel, R2, and Helm publication must be skipped; installation references and the website announcement stay unchanged.
On a restart after a backport, re-run release-branch CI before setting the new `PREVIEW_HASH`. Use the next unused preview iteration when the validated commit differs from an existing preview tag; never move an existing tag.
- Set `CANDIDATE_SHA` to the CI-validated release-branch commit. Record the branch, exact commit and CI run URL. The new candidate protocol starts from a commit whose release recipe matches the reviewed main dispatcher; backport the recipe before freezing an older release branch if needed.
- Dispatch the Linux candidate as described in [candidate verification](preview-release.md). Do not create a preview or final tag here. The final executable already contains `<target>`; publication later promotes those same bytes.
- After a source or build-input change, run the affected source checks and create a new candidate. Reuse the frozen candidate after an infrastructure failure. Do not restart acceptance for unrelated main advances.
@@ -4,18 +4,18 @@ description: Prepare an exact RustFS Cargo version or align installation referen
---
# Release Version Files
Require an exact target; ask if missing or ambiguous. Reject targets containing `-preview`: those are tag-only. Infer `post-release` only from an explicit request or parent workflow; otherwise use `prepare`. Delivery defaults to local edit/verify; commit, push, PR, and merge follow existing authorization and [Git rules](../../references/pull-requests.md).
Require an exact target; ask if missing or ambiguous. Reject targets containing `-preview`: those identify candidates, not product versions or new public tags. Infer `post-release` only from an explicit request or parent workflow; otherwise use `prepare`. Delivery defaults to local edit/verify; commit, push, PR, and merge follow existing authorization and [Git rules](../../references/pull-requests.md).
| Stage | Files and constraints |
|---|---|
| `prepare` | Only `Cargo.toml` and `Cargo.lock`: workspace package, internal dependency, and workspace member versions become `<target>`. External dependencies stay unchanged. |
| `post-release` | Only `README.md`, `README_ZH.md`, `flake.nix`, `helm/rustfs/Chart.yaml`, and `rustfs.spec`. Cargo may already target the next release; leave it untouched. |
Before installation edits, verify the exact non-preview Release, downloadable assets/source archive, pullable `rustfs/rustfs:<target>` image manifest, and published Helm chart. Block on missing artifacts. Do not downgrade defaults if a newer deliverable superseded the target. Preview preparation retains the previous published installation references; follow-up commits never replace either validated tag.
Before installation edits, verify the exact non-preview Release, downloadable assets/source archive, pullable `rustfs/rustfs:<target>` image manifest, and published Helm chart. Block on missing artifacts. Do not downgrade defaults if a newer deliverable superseded the target. Candidate preparation retains the previous published installation references; follow-up commits never replace validated tags or sealed artifacts.
Packaging policy:
- Docker image tags have no `v` prefix. Derive chart/app versions with `scripts/helm_chart_version.sh <target>` (`beta.N -> 0.N.0`; other versions unchanged). Helm CI derives versions from the final tag, so no early chart bump is required.
- Docker image tags have no `v` prefix. Derive chart/app versions with `scripts/helm_chart_version.sh <target>`; both equal the final product SemVer, including prerelease suffixes. Helm CI uses the frozen candidate version before sealing, so no early installation chart bump is required. Previously published chart versions remain unchanged.
- RPM `Version` is numeric; `Release` is the prerelease suffix or `1` for stable. Expanded `Source0` and the unpack directory must resolve to the exact target including its suffix. Add the changelog entry using current git identity, date, and target; never copy an example identity/date.
- Changing these mappings requires explicit confirmation.
+1 -1
View File
@@ -1 +1 @@
sha256=fbabc660e6e01fd8ff6b9d1de7d68e58242d60c942f85aed5074b0ec2918c154
sha256=21e9a1561ac20382503c38353bbb9b830658534a4f277c254a42d0afc6ebed8b
+1 -1
View File
@@ -71,7 +71,7 @@ s3s-footprint-check: ## Check the s3s dependency footprint ratchet stays frozen
./scripts/check_s3s_footprint.sh
.PHONY: test-count-ratchet-check
test-count-ratchet-check: ## Check no workspace crate lost tests or gained #[ignore] against the committed baseline
test-count-ratchet-check: ## Check no workspace crate fell below its test floor or rose above its #[ignore] ceiling in the committed baseline
@echo "🧪 Checking per-crate test-count ratchet..."
./scripts/check_test_count_ratchet.sh
+11
View File
@@ -22,6 +22,14 @@ TEST_THREADS ?= 1
# differ from CI and .config/nextest.toml test-groups will NOT apply.
.PHONY: script-tests
.PHONY: release-package-tests
release-package-tests: ## Run release package integration on Linux with fpm, dpkg and rpm installed
$(RUSTFS_PYTHON_BIN) ./scripts/test_release_packages.py
.PHONY: release-image-tests
release-image-tests: ## Run OCI integration with Docker, Go and skopeo; set RELEASE_IMAGE_VARIANT and RELEASE_IMAGE_OUTPUT
$(RUSTFS_PYTHON_BIN) ./scripts/test_release_images.py
script-tests: ## Run shell script tests
@echo "Running script tests..."
trap 'status=$$?; printf "ERROR: script-tests failed (exit %s): %s\n" "$$status" "$$BASH_COMMAND" >&2; exit "$$status"' ERR
@@ -53,12 +61,14 @@ script-tests: ## Run shell script tests
$(RUSTFS_PYTHON_BIN) ./scripts/test_check_doc_paths.py
./scripts/check_embedded_secrets.sh --self-test
./scripts/check_s3s_footprint.sh --self-test
./scripts/test_check_test_count_ratchet.sh
$(RUSTFS_PYTHON_BIN) ./scripts/check_test_wiring.py --self-test
$(RUSTFS_PYTHON_BIN) ./scripts/codemods/s3_error_to_s3_types.py --self-test
$(RUSTFS_PYTHON_BIN) ./scripts/test_e2e_binary.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_download_release_asset.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_inspect_btf_prototypes.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_release_build_workflow.py
$(RUSTFS_PYTHON_BIN) -m unittest discover -s scripts/release -p "test_*.py"
$(RUSTFS_PYTHON_BIN) ./scripts/test_release_glibc.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_upgrade_workflow.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_migration_gate_evidence.py
@@ -79,6 +89,7 @@ script-tests: ## Run shell script tests
$(RUSTFS_PYTHON_BIN) ./scripts/test_inspect_validation_runner.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_retain_validation_evidence.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_ci_timing_report.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_junit_diff.py
$(RUSTFS_PYTHON_BIN) ./scripts/s3-tests/test_report_compat.py
$(RUSTFS_PYTHON_BIN) ./scripts/s3-tests/test_runner_tools.py
bash -n ./scripts/validate_object_data_cache_cold_stampede.sh
+82 -24
View File
@@ -1,44 +1,93 @@
{
"schema_version": 1,
"plan_id": "rustfs-release-validation",
"description": "Versioned certification plan for every public RustFS release exit (rustfs/backlog#2662, R01). preview, prerelease and stable all require this explicit plan. Scripts/release/release_gate.py fail-closes publication on any missing, duplicate, stale, substituted or non-PASS required item.",
"release_types": ["preview", "prerelease", "stable"],
"description": "Certification for immutable candidates. All shipped platforms require native identity and startup; stable and prerelease require the pinned upgrade matrix. Linux-only acceptance never authorizes publication.",
"release_types": [
"preview",
"prerelease",
"stable"
],
"targets": [
{"os": "linux", "arch": "x86_64", "abi": "musl"},
{"os": "linux", "arch": "aarch64", "abi": "musl"},
{"os": "linux", "arch": "x86_64", "abi": "gnu"},
{"os": "linux", "arch": "aarch64", "abi": "gnu"},
{"os": "macos", "arch": "aarch64", "abi": "none"},
{"os": "windows", "arch": "x86_64", "abi": "msvc"}
{
"os": "linux",
"arch": "x86_64",
"abi": "musl"
},
{
"os": "linux",
"arch": "aarch64",
"abi": "musl"
},
{
"os": "linux",
"arch": "x86_64",
"abi": "gnu"
},
{
"os": "linux",
"arch": "aarch64",
"abi": "gnu"
},
{
"os": "macos",
"arch": "aarch64",
"abi": "none"
},
{
"os": "windows",
"arch": "x86_64",
"abi": "msvc"
}
],
"targets_basis": "Platform matrix of .github/workflows/build.yml (prepare-platform-matrix). gnu and macOS targets build with the pyroscope feature; musl and Windows targets build with no extra features.",
"features": ["pyroscope"],
"trusted_workflows": [
{"repository": "rustfs/s3chaos", "workflow_id": "release-gate.yml"}
"features": [
"pyroscope"
],
"trusted_workflows_basis": "The shared evidence envelope producer lives in rustfs/s3chaos (G01, fault/release_gate.rs) and its plan/live split workflow is .github/workflows/release-gate.yml there. Only results from these identities may enter a release certification; PR-uploaded JSON never gains release authority. R02 must extend this list when additional verifier workflows land, never widen it to untrusted sources.",
"trusted_workflows": [
{
"repository": "rustfs/rustfs",
"workflow_id": "release-candidate.yml"
},
{
"repository": "rustfs/rustfs",
"workflow_id": "release-complete.yml"
}
],
"trusted_workflows_basis": "Receipts are produced by the reviewed main workflows and fetched by exact run ID with GitHub artifact digest verification. Candidate inputs freeze the harness commit separately from product source.",
"lanes": [
{
"lane": "identity",
"harness_repository": "rustfs/s3chaos",
"harness_sha": "9a48f55c1d6d58ce63a8fb1ea7317e024f32f11d",
"harness_basis": "Fixed G01 evidence-envelope producer revision (rustfs/s3chaos PR #102 head recorded in rustfs/backlog#2658). R02 re-pins this when the live verifier ships.",
"harness_repository": "rustfs/rustfs",
"harness_sha": "candidate.workflow_sha",
"harness_basis": "The exact workflow commit is frozen in candidate inputs; no mutable branch is accepted as a harness identity.",
"timeout_seconds": 1800,
"environment_capabilities": ["process-exec"],
"cases": ["artifact-identity", "startup"],
"environment_capabilities": [
"process-exec"
],
"cases": [
"artifact-identity",
"startup"
],
"applies_to": {
"release_types": ["preview", "prerelease", "stable"],
"release_types": [
"preview",
"prerelease",
"stable"
],
"platforms": "all"
},
"applies_basis": "Artifact identity and startup checks are mandatory for every release type, preview included, on every shipped target (rustfs/backlog#2662 frozen design item 5)."
},
{
"lane": "upgrade",
"harness_repository": "rustfs/s3chaos",
"harness_sha": "9a48f55c1d6d58ce63a8fb1ea7317e024f32f11d",
"harness_basis": "Same fixed envelope producer as the identity lane; the workflow SHA and the product build SHA are bound separately and need not be equal.",
"harness_repository": "rustfs/rustfs",
"harness_sha": "candidate.workflow_sha",
"harness_basis": "The exact workflow commit is frozen in candidate inputs; no mutable branch is accepted as a harness identity.",
"timeout_seconds": 5400,
"environment_capabilities": ["process-exec", "previous-release-artifact"],
"environment_capabilities": [
"process-exec",
"previous-release-artifact"
],
"cases": [
"direct-upgrade",
"mixed-version-upgrade",
@@ -49,8 +98,17 @@
],
"cases_basis": "Case IDs of the pinned previous-release matrix in .github/workflows/e2e-upgrade.yml (source.version/asset/sha256 and the case list mapped there).",
"applies_to": {
"release_types": ["prerelease", "stable"],
"platforms": [{"os": "linux", "arch": "x86_64", "abi": "gnu"}]
"release_types": [
"prerelease",
"stable"
],
"platforms": [
{
"os": "linux",
"arch": "x86_64",
"abi": "gnu"
}
]
},
"applies_basis": "preview releases use the pre-declared smaller set (identity lane only). The upgrade lane is only exercised on linux-x86_64-gnu, the platform of the pinned source assets."
}
+12 -10
View File
@@ -4,15 +4,17 @@
# One row per workspace member: <package>, tests, ignored, tab-separated.
# tests counts #[test], #[tokio::test], #[tokio::test(...)] and #[test_case(...)]
# attribute lines under the package directory; ignored counts #[ignore] and
# #[ignore = "..."]. Both columns are exact: CI fails when a crate's tests fall
# below or rise above its row, or when its ignored count changes. Raise a row in
# the PR that adds tests; lowering one is a reviewed statement that tests were
# intentionally removed. Regenerate with:
# #[ignore = "..."]. The tests column is a floor and the ignored column is a
# ceiling: CI fails when a crate's tests fall below its row or its ignored count
# rises above it. Extra tests and fewer ignores pass; --update tightens the rows.
# Lowering a tests row or raising an ignored row is a reviewed statement that
# tests were intentionally removed or ignored; justify it in the PR description.
# Regenerate with:
# scripts/check_test_count_ratchet.sh --update
e2e_test 1026 20
rustfs 6438 64
rustfs 6440 64
rustfs-audit 70 0
rustfs-checksums 26 0
rustfs-checksums 28 1
rustfs-common 36 0
rustfs-concurrency 87 0
rustfs-config 43 0
@@ -20,7 +22,7 @@ rustfs-connect-symbol-catalog 6 0
rustfs-credentials 24 0
rustfs-crypto 67 0
rustfs-data-usage 58 0
rustfs-ecstore 7850 2
rustfs-ecstore 7858 2
rustfs-extension-schema 23 0
rustfs-filemeta 361 0
rustfs-heal 1128 3
@@ -43,14 +45,14 @@ rustfs-policy 758 0
rustfs-protocols 770 1
rustfs-protos 75 0
rustfs-replication 209 0
rustfs-rio 223 0
rustfs-rio 225 1
rustfs-rio-v2 34 6
rustfs-s3-client 71 0
rustfs-s3-ops 8 0
rustfs-s3-types 105 0
rustfs-s3select-api 232 0
rustfs-s3select-query 235 2
rustfs-scanner 1386 20
rustfs-scanner 1392 20
rustfs-scanner-metrics 90 0
rustfs-security-governance 20 0
rustfs-signer 34 0
@@ -59,6 +61,6 @@ rustfs-targets 612 26
rustfs-test-utils 1 0
rustfs-tls-runtime 44 0
rustfs-trusted-proxies 72 0
rustfs-utils 251 0
rustfs-utils 253 1
rustfs-zip 30 0
zstd 11 0
+81 -21
View File
@@ -350,10 +350,10 @@
"roles": [
"periodic",
"manual",
"release"
"reusable"
],
"events": [
"push",
"workflow_call",
"schedule",
"workflow_dispatch"
],
@@ -367,7 +367,7 @@
"monitor": "workflow",
"max_age_hours": 36,
"result_owner": "build.yml",
"disposition": "retained independent entry",
"disposition": "Builds artifacts only; public release writes belong to release-publish.yml.",
"failure_owner": "workflow",
"origin": "baseline",
"alert_contract": "scheduled-validation"
@@ -745,14 +745,13 @@
},
{
"workflow": ".github/workflows/docker.yml",
"name": "Docker Images",
"name": "Build candidate images",
"roles": [
"manual",
"release"
"release",
"reusable"
],
"events": [
"workflow_run",
"workflow_dispatch"
"workflow_call"
],
"cadence": "event",
"timezone": "UTC",
@@ -761,7 +760,7 @@
"policy": "active",
"monitor": "none",
"result_owner": "docker.yml",
"disposition": "retained independent entry",
"disposition": "Builds artifacts only; public release writes belong to release-publish.yml.",
"monitor_reason": "Event-driven or operator-requested work has no periodic freshness obligation.",
"origin": "baseline"
},
@@ -954,14 +953,13 @@
},
{
"workflow": ".github/workflows/helm-package.yml",
"name": "Publish helm chart to artifacthub",
"name": "Build candidate Helm chart",
"roles": [
"manual",
"release"
"release",
"reusable"
],
"events": [
"workflow_run",
"workflow_dispatch"
"workflow_call"
],
"cadence": "event",
"timezone": "UTC",
@@ -970,7 +968,7 @@
"policy": "active",
"monitor": "none",
"result_owner": "helm-package.yml",
"disposition": "retained independent entry",
"disposition": "Builds artifacts only; public release writes belong to release-publish.yml.",
"monitor_reason": "Event-driven or operator-requested work has no periodic freshness obligation.",
"origin": "baseline"
},
@@ -1182,14 +1180,13 @@
},
{
"workflow": ".github/workflows/package.yml",
"name": "Package DEB/RPM",
"name": "Build candidate packages",
"roles": [
"manual",
"release"
"release",
"reusable"
],
"events": [
"workflow_run",
"workflow_dispatch"
"workflow_call"
],
"cadence": "event",
"timezone": "UTC",
@@ -1198,7 +1195,7 @@
"policy": "active",
"monitor": "none",
"result_owner": "package.yml",
"disposition": "retained independent entry",
"disposition": "Builds artifacts only; public release writes belong to release-publish.yml.",
"monitor_reason": "Event-driven or operator-requested work has no periodic freshness obligation.",
"origin": "baseline"
},
@@ -1727,6 +1724,69 @@
"disposition": "retained independent entry",
"monitor_reason": "Event-driven or operator-requested work has no periodic freshness obligation.",
"origin": "baseline"
},
{
"workflow": ".github/workflows/release-candidate.yml",
"name": "Release candidate (Linux)",
"roles": [
"manual",
"release"
],
"events": [
"workflow_dispatch"
],
"cadence": "event",
"timezone": "UTC",
"schedule": [],
"owner": "@overtrue",
"policy": "active",
"monitor": "none",
"result_owner": "release-candidate.yml",
"disposition": "Explicit immutable candidate stage",
"monitor_reason": "Operator-requested release stage has no periodic freshness obligation.",
"origin": "release-candidate-migration"
},
{
"workflow": ".github/workflows/release-complete.yml",
"name": "Release candidate (complete)",
"roles": [
"manual",
"release"
],
"events": [
"workflow_dispatch"
],
"cadence": "event",
"timezone": "UTC",
"schedule": [],
"owner": "@overtrue",
"policy": "active",
"monitor": "none",
"result_owner": "release-complete.yml",
"disposition": "Explicit immutable candidate stage",
"monitor_reason": "Operator-requested release stage has no periodic freshness obligation.",
"origin": "release-candidate-migration"
},
{
"workflow": ".github/workflows/release-publish.yml",
"name": "Publish accepted release",
"roles": [
"manual",
"release"
],
"events": [
"workflow_dispatch"
],
"cadence": "event",
"timezone": "UTC",
"schedule": [],
"owner": "@overtrue",
"policy": "active",
"monitor": "none",
"result_owner": "release-publish.yml",
"disposition": "Explicit immutable candidate stage",
"monitor_reason": "Operator-requested release stage has no periodic freshness obligation.",
"origin": "release-candidate-migration"
}
]
}
+11 -4
View File
@@ -5,7 +5,7 @@
## Billing facts
This repository is public. For a public repository, GitHub's standard hosted runners (`ubuntu-latest`, `windows-latest`, `macos-latest`) cost nothing and have no minute cap, whatever plan the organisation is on. GitHub bills only two things: larger hosted runners, which have custom labels and are not used here, and standard runners in private repositories.
This repository is public. For a public repository, GitHub's standard hosted runners (`ubuntu-latest`, `windows-latest`, `macos-latest`) cost nothing and have no minute cap, whatever plan the organisation is on. GitHub bills only two things: larger hosted runners, including the `macos-latest-xlarge` release lane used here, and standard runners in private repositories.
Every other label in this repository is capacity RustFS pays for. ARC pods and lab machines accrue cost while the machine exists. A Ubicloud label is metered for the minutes a job holds its VM:
@@ -13,6 +13,7 @@ Every other label in this repository is capacity RustFS pays for. ARC pods and l
|---|---|---|---|
| `ubuntu-latest` | nobody | GitHub-hosted, Ubuntu 24.04, 4 vCPU, 16 GB RAM, about 14 GB of free disk. Ships a C toolchain, `docker`, `gh`, `curl`, `wget`, `jq`, Python. Can `mount` tmpfs and loop devices; io_uring works | the default for every job |
| `windows-latest`, `macos-latest` | nobody | GitHub-hosted standard runners | Windows and macOS targets |
| `macos-latest-xlarge` | RustFS | GitHub-hosted larger Apple Silicon runner | macOS release builds, deferred until Linux acceptance for candidates |
| `sm-standard-2` | RustFS | ARC pod, Ubuntu 24.04, minimal image: no `gh`, no C toolchain; Docker only on the `dind-` label | only a job that a hosted runner cannot run (see the placement rule) |
| `sm-standard-4` | RustFS | ARC pod, Ubuntu 24.04, 14 CPUs and 28 GB (the label is not the core count), large disk | workspace-scale Cargo builds |
| `ubicloud-standard-16-arm` | RustFS | Ubicloud VM, alias of `ubicloud-standard-16-arm-ubuntu-2404`: Ubuntu 24.04 arm64, 16 vCPU, 48 GB, 300 GB disk. Reached through the `build.yml` matrix, so `actionlint.yaml` does not list it | native aarch64 release builds (`linux-aarch64-musl`, `linux-aarch64-gnu`) |
@@ -32,7 +33,7 @@ A job runs on `ubuntu-latest` unless it has one of these reasons, and the reason
Jobs that do only the following have no reason to be self-hosted: shell or Python scripts, `cargo fmt`, `cargo deny`, typos, actionlint, API calls (`gh`, dependency review, issue and stale bots, CLA checks), downloading an artifact and running a prebuilt binary, or a `cargo build -p rustfs` with its own cache key. `e2e-distributed.yml` and `e2e-upgrade.yml` build the server and the `e2e_test` harness on `ubuntu-latest` inside their budgets, so "it compiles something" alone is not a reason.
The functional chain's `prepare` and `complete-chain` jobs, the performance runner availability check, and runner hygiene use `ubuntu-latest`: they inspect APIs or artifacts, not the lab hosts. The actual functional and performance suites remain on their lab runners. Functional-chain health collection uses `ubuntu-latest` so a lab outage cannot also stop its independent monitor. Docker image packaging also uses `ubuntu-latest` for both musl and glibc variants; it downloads release binaries rather than compiling the workspace, and still sets up QEMU and Buildx explicitly.
The functional chain's `prepare` and `complete-chain` jobs, the performance runner availability check, and runner hygiene use `ubuntu-latest`: they inspect APIs or artifacts, not the lab hosts. The actual functional and performance suites remain on their lab runners. Functional-chain health collection uses `ubuntu-latest` so a lab outage cannot also stop its independent monitor. Docker image packaging also uses `ubuntu-latest` for both musl and glibc variants; it wraps verified candidate binaries rather than compiling the workspace, and still sets up QEMU and Buildx explicitly.
Moving these jobs off AKS reduces capacity demand, not necessarily the Azure bill immediately. VM costs fall only when nodes scale down or smaller capacity replaces them.
@@ -52,7 +53,7 @@ A hosted runner is also the only choice for a few things the pods cannot do: `mo
These are the patterns that have cost real money here without validating anything more:
- An `if: always()` summary or alert job on a self-hosted label inside a `workflow_run`-triggered workflow. The upstream workflow completes many times a day, the real jobs skip, and the summary still starts a pod each time. Gate it on the upstream job having run, and keep it on `ubuntu-latest`.
- Per-push workflows on `main`. A merge triggers the full `ci.yml` matrix, `cache-warm.yml` and the `build.yml` development build; with tens of merges a day that is as much self-hosted time as all pull requests together. Anything that does not verify the merged tree itself belongs on a schedule or behind a `paths` filter.
- Per-push workflows on `main`. A merge triggers the applicable `ci.yml` matrix and cache warming; `build.yml` development builds are scheduled or manual; with tens of merges a day that is as much self-hosted time as all pull requests together. Anything that does not verify the merged tree itself belongs on a schedule or behind a `paths` filter.
- `apt-get install` of tools the hosted image already ships. It is only needed on the minimal pod images.
- A throttled `CARGO_BUILD_JOBS` on a 14-CPU pod. The limit exists for link-storm reasons documented in `ci.yml`; measure before raising it, and measure before lowering it further.
- Tiny jobs on ARC pods. Each one pays roughly a minute of pod start-up before its first step; a compile-free job that takes ten seconds costs more to schedule than to run.
@@ -66,7 +67,7 @@ gh api "repos/rustfs/rustfs/actions/runs/<run_id>/jobs?per_page=100" \
--jq '.jobs[] | "\(.conclusion)\t\(.labels | join(","))\t\(.started_at)\t\(.completed_at)\t\(.name)"'
```
Only rows whose label starts with `sm-`, `dind-`, `ubicloud-`, `smoke-` or `pf-` cost anything. At the October 2026 review, a full pull-request run of `ci.yml` spent about 150 self-hosted minutes, a third of it in jobs that never compiled; treat that as the scale of what a placement mistake costs, not as a current figure.
Rows using `macos-latest-xlarge` or a label starting with `sm-`, `dind-`, `ubicloud-`, `smoke-` or `pf-` use paid capacity. At the October 2026 review, a full pull-request run of `ci.yml` spent about 150 self-hosted minutes, a third of it in jobs that never compiled; treat that as the scale of what a placement mistake costs, not as a current figure.
## Checklist for a new or moved job
@@ -76,3 +77,9 @@ Only rows whose label starts with `sm-`, `dind-`, `ubicloud-`, `smoke-` or `pf-`
- A self-hosted `runs-on` carries a comment naming its reason from the placement rule.
- Every `./.github/actions/setup` call states `cache-save-if`, and a key read from one pool is not written from the other.
- Moving a job does not change its check name, so the `main` ruleset and `docs/testing/ci-gates.md` stay as they are; renaming it does, and then that document's change checklist applies.
## Release stages
The [candidate release pipeline](../../docs/operations/release-pipeline.md) builds Linux first and defers paid desktop capacity until acceptance. Daily development builds also select Linux; use manual platform selection for desktop development artifacts. All six platforms remain required before publication. The pipeline records and promotes the exact tested bytes. Builder workflows have no public-release credentials. Keep periodic development builds, PR verification and field-test chains separate when their contracts differ; do not re-enable disabled workflows as part of a source-file migration.
Larger runners remain billed for public repositories; see [GitHub runner billing](https://docs.github.com/en/actions/concepts/runners/larger-runners#billing). Actual spending comes from organization billing, not runner labels or an estimated build duration.
+149 -8
View File
@@ -23,9 +23,16 @@ on:
- 'deny.toml'
- '.github/actions/**'
- '.github/workflows/**'
- 'scripts/release/create_or_update_release.sh'
- 'scripts/release/package_service_scripts.sh'
- 'scripts/release/package_versions.sh'
- 'scripts/release/**'
- 'scripts/security/generate_release_supply_chain_assets.py'
- '.config/release-validation-plan.json'
- '.github/scheduled-validations.json'
- 'scripts/test_helm_release_workflow.py'
- 'scripts/test_release_packages.py'
- 'scripts/test_release_images.py'
- 'Dockerfile*'
- 'entrypoint.sh'
- 'bootstrap/**'
- 'scripts/test_package_service_scripts.sh'
- 'scripts/test_package_versions.sh'
- 'scripts/test_docker_workflow.py'
@@ -41,9 +48,16 @@ on:
- 'deny.toml'
- '.github/actions/**'
- '.github/workflows/**'
- 'scripts/release/create_or_update_release.sh'
- 'scripts/release/package_service_scripts.sh'
- 'scripts/release/package_versions.sh'
- 'scripts/release/**'
- 'scripts/security/generate_release_supply_chain_assets.py'
- '.config/release-validation-plan.json'
- '.github/scheduled-validations.json'
- 'scripts/test_helm_release_workflow.py'
- 'scripts/test_release_packages.py'
- 'scripts/test_release_images.py'
- 'Dockerfile*'
- 'entrypoint.sh'
- 'bootstrap/**'
- 'scripts/test_package_service_scripts.sh'
- 'scripts/test_package_versions.sh'
- 'scripts/test_docker_workflow.py'
@@ -132,12 +146,38 @@ jobs:
if: github.repository == 'rustfs/rustfs' && (github.event_name != 'pull_request' || github.event.action != 'closed')
runs-on: ubuntu-latest
timeout-minutes: 5
permissions:
contents: read
actions: read
outputs:
release_changed: ${{ steps.scope.outputs.release_changed }}
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
fetch-depth: 0
persist-credentials: false
- name: Scope native packaging checks to release changes
id: scope
env:
BASE_SHA: ${{ github.event.pull_request.base.sha || github.event.before }}
run: |
python3 - <<'PY'
import os, re, subprocess
event, base = os.environ['GITHUB_EVENT_NAME'], os.environ.get('BASE_SHA', '')
changed = event == 'workflow_dispatch'
if event in ('push', 'pull_request'):
if not re.fullmatch(r'[0-9a-f]{40}', base) or base == '0' * 40:
changed = True
else:
paths = subprocess.check_output(['git', 'diff', '--name-only', base, os.environ['GITHUB_SHA']], text=True).splitlines()
pattern = r'^(scripts/release/|scripts/test_release_|Dockerfile|entrypoint.sh$|bootstrap/|\.dockerignore$|\.config/release-validation-plan.json$|\.github/workflows/(audit|build|docker|package|helm-package|release-[^.]+)\.yml$)'
changed = any(re.match(pattern, path) for path in paths)
with open(os.environ['GITHUB_OUTPUT'], 'a') as output:
output.write(f'release_changed={str(changed).lower()}\n')
PY
- name: Report unpinned GitHub Actions
run: ./scripts/security/check_workflow_pins.sh --enforce
@@ -150,8 +190,35 @@ jobs:
- name: Check checkouts clear their credentials
run: ./scripts/security/check_persist_credentials.sh
- name: Check preview release workflow policy
run: ./scripts/security/check_preview_release_workflow.sh
- name: Check release candidate and promotion contracts
run: python3 -m unittest discover -s scripts/release -p 'test_*.py'
- name: Prepare artifact transport rehearsal
run: |
mkdir -p "$RUNNER_TEMP/release-transport"
printf '{"purpose":"transport contract, not release evidence"}\n' > "$RUNNER_TEMP/release-transport/transport.json"
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: release-transport-contract-${{ github.run_attempt }}
path: ${{ runner.temp }}/release-transport/transport.json
if-no-files-found: error
retention-days: 1
- name: Verify actual GitHub artifact metadata and downloaded bytes
env:
GH_TOKEN: ${{ github.token }}
run: |
python3 - <<'PY'
import os, sys
from pathlib import Path
sys.path.insert(0, 'scripts/release')
import candidate as c
run = c.github(f"repos/{c.REPOSITORY}/actions/runs/{os.environ['GITHUB_RUN_ID']}")
destination = Path(os.environ['RUNNER_TEMP']) / 'release-transport-downloaded'
c.download_artifact(run, f"release-transport-contract-{os.environ['GITHUB_RUN_ATTEMPT']}", destination)
c.require(c.read_json(destination / 'transport.json') == {'purpose': 'transport contract, not release evidence'}, 'artifact round trip changed')
PY
- name: Check tier evidence workflow isolation
run: ./scripts/security/check_tier_artifact_workflow.sh
@@ -162,6 +229,80 @@ jobs:
- name: Check package service lifecycle contract
run: ./scripts/test_package_service_scripts.sh
release-package-contract:
name: Release package contract
needs: workflow-pin-report
if: needs.workflow-pin-report.outputs.release_changed == 'true'
runs-on: ubuntu-latest
timeout-minutes: 15
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Install real package managers
run: |
sudo apt-get update
sudo apt-get install -y ruby ruby-dev build-essential rpm fakeroot cpio
sudo gem install fpm --version 1.17.0 --no-document
- name: Exercise accepted and substituted payloads through both recipes
run: python3 scripts/test_release_packages.py
release-image-contract:
name: Release image contract (${{ matrix.variant }})
needs: workflow-pin-report
if: needs.workflow-pin-report.outputs.release_changed == 'true'
runs-on: ubuntu-latest
timeout-minutes: 35
strategy:
fail-fast: false
matrix:
variant: [musl, gnu]
env:
RELEASE_IMAGE_VARIANT: ${{ matrix.variant }}
NO_PROXY: 127.0.0.1,localhost,::1
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Install OCI tools
run: |
echo "RELEASE_IMAGE_OUTPUT=$RUNNER_TEMP/image-contract" >> "$GITHUB_ENV"
echo 'no_proxy=127.0.0.1,localhost,::1' >> "$GITHUB_ENV"
sudo apt-get update
sudo apt-get install -y skopeo
go version
- uses: docker/setup-qemu-action@c7c53464625b32c7a7e944ae62b3e17d2b600130 # v3
- uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f # v3
- name: Exercise real OCI packaging, startup, registry retry and conflicts
run: python3 scripts/test_release_images.py
- name: Exercise amd64 OCI archive scanning
uses: aquasecurity/trivy-action@ed142fd0673e97e23eac54620cfb913e5ce36c25
env:
TRIVY_PLATFORM: linux/amd64
with:
input: ${{ runner.temp }}/image-contract-scan/image-${{ matrix.variant }}-amd64.tar
format: sarif
output: ${{ runner.temp }}/image-contract/scan-${{ matrix.variant }}-amd64.sarif
exit-code: "0"
- name: Exercise arm64 OCI archive scanning
uses: aquasecurity/trivy-action@ed142fd0673e97e23eac54620cfb913e5ce36c25
env:
TRIVY_PLATFORM: linux/arm64
with:
input: ${{ runner.temp }}/image-contract-scan/image-${{ matrix.variant }}-arm64.tar
format: sarif
output: ${{ runner.temp }}/image-contract/scan-${{ matrix.variant }}-arm64.sarif
exit-code: "0"
- name: Verify scan receipts before recording the archive
run: |
python3 - <<'PY'
import os, sys
from pathlib import Path
sys.path.insert(0, 'scripts/release')
import deliver, test_candidate
deliver.image_record(test_candidate.inputs(), os.environ['RELEASE_IMAGE_VARIANT'], Path(os.environ['RELEASE_IMAGE_OUTPUT']))
PY
dependency-review:
name: Dependency Review
runs-on: ubuntu-latest
File diff suppressed because it is too large Load Diff
+71 -751
View File
@@ -1,779 +1,99 @@
# Copyright 2024 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Docker Images Workflow
#
# This workflow builds Docker images using pre-built binaries from the build workflow.
#
# Trigger Types:
# 1. workflow_run: Automatically triggered when "Build and Release" workflow completes
# 2. workflow_dispatch: Manual trigger for standalone Docker builds
#
# Key Features:
# - Only triggers when Linux builds (x86_64 + aarch64) are successful
# - Independent of macOS/Windows build status
# - Uses workflow_run event for precise control
# - Only builds Docker images for releases and prereleases (development builds are skipped)
name: Docker Images
# Permissions needed for workflow_run event and Docker registry access
# Copyright 2026 RustFS Team
# SPDX-License-Identifier: Apache-2.0
name: Build candidate images
on:
workflow_call:
inputs:
candidate_manifest:
type: string
required: true
permissions:
contents: read
packages: write
on:
# Automatically triggered when build workflow completes
workflow_run:
workflows: [ "Build and Release" ]
types: [ completed ]
# Manual trigger with same parameters for consistency
workflow_dispatch:
inputs:
push_images:
description: "Push images to registries"
required: false
default: true
type: boolean
version:
description: "Version to build (latest, v1.0.0, v1.0.0-alpha1, or v1.0.1-preview.1)"
required: false
default: "latest"
type: string
force_rebuild:
description: "Force rebuild even if binary exists (useful for testing)"
required: false
default: false
type: boolean
env:
CONCLUSION: ${{ github.event.workflow_run.conclusion }}
HEAD_BRANCH: ${{ github.event.workflow_run.head_branch }}
HEAD_SHA: ${{ github.event.workflow_run.head_sha }}
TRIGGERING_EVENT: ${{ github.event.workflow_run.event }}
DOCKERHUB_USERNAME: rustfs
CARGO_TERM_COLOR: always
REGISTRY_DOCKERHUB: rustfs/rustfs
REGISTRY_GHCR: ghcr.io/${{ github.repository }}
REGISTRY_QUAY: quay.io/rustfs/rustfs
DOCKER_PLATFORMS: linux/amd64,linux/arm64
actions: read
jobs:
# Check if we should build Docker images.
# Short-circuit at job level so per-merge development builds (workflow_run
# from a push to main) skip the whole run without spawning a checkout job:
# images are only published for tag builds — build.yml push triggers cover
# only main and tags, so a non-main head_branch is a tag name — and for
# manual dispatch.
build-check:
name: Docker Build Check
if: >-
github.event_name == 'workflow_dispatch' ||
(github.event.workflow_run.conclusion == 'success' &&
github.event.workflow_run.event == 'push' &&
github.event.workflow_run.head_branch != 'main')
runs-on: ubuntu-latest
timeout-minutes: 30
outputs:
should_build: ${{ steps.check.outputs.should_build }}
should_push: ${{ steps.check.outputs.should_push }}
build_type: ${{ steps.check.outputs.build_type }}
version: ${{ steps.check.outputs.version }}
short_sha: ${{ steps.check.outputs.short_sha }}
is_prerelease: ${{ steps.check.outputs.is_prerelease }}
create_latest: ${{ steps.check.outputs.create_latest }}
source_ref: ${{ steps.check.outputs.source_ref }}
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
# For workflow_run events, checkout the specific commit that triggered the workflow
ref: ${{ github.event.workflow_run.head_sha || github.sha }}
- name: Check build conditions
id: check
env:
# dispatch inputs via env, not `${{ }}` interpolation: they are
# free-form strings and would otherwise be evaluated by bash.
INPUT_VERSION: ${{ github.event.inputs.version }}
INPUT_PUSH_IMAGES: ${{ github.event.inputs.push_images }}
INPUT_FORCE_REBUILD: ${{ github.event.inputs.force_rebuild }}
run: |
should_build=false
should_push=false
build_type="none"
version=""
short_sha=""
is_prerelease=false
create_latest=false
source_ref="$GITHUB_SHA"
# Pre-GA policy: until the first stable (vX.Y.Z) tag exists, every
# prerelease (alpha/beta/rc) also moves `latest`, so users pulling
# `latest` get the newest test build. Once a stable tag is published
# this returns false and `latest` follows stable releases only.
prerelease_moves_latest() {
local stable_tags
stable_tags=$(git ls-remote --tags --refs origin 2>/dev/null \
| awk '{print $2}' \
| grep -E '^refs/tags/v?[0-9]+\.[0-9]+\.[0-9]+$' || true)
if [[ -z "$stable_tags" ]]; then
return 0
fi
echo "ℹ️ Stable release tag(s) already exist; prereleases no longer update latest"
return 1
}
if [[ "${{ github.event_name }}" == "workflow_run" ]]; then
# Triggered by build workflow completion
echo "🔗 Triggered by build workflow completion"
# Check if the triggering workflow was successful
# If the workflow succeeded, it means ALL builds (including Linux x86_64 and aarch64) succeeded
if [[ "$CONCLUSION" == "success" ]]; then
echo "✅ Build workflow succeeded, all builds including Linux are successful"
should_build=true
should_push=true
else
echo "❌ Build workflow failed (conclusion: $CONCLUSION), skipping Docker build"
should_build=false
fi
# Extract version info from commit message or use commit SHA
# Use Git to generate consistent short SHA (ensures uniqueness like build.yml)
short_sha=$(git rev-parse --short "$HEAD_SHA")
source_ref="$HEAD_SHA"
# Determine build type based on triggering workflow event and ref
triggering_event="$TRIGGERING_EVENT"
head_branch="$HEAD_BRANCH"
echo "🔍 Analyzing triggering workflow:"
echo " 📋 Event: $triggering_event"
echo " 🌿 Head branch: $head_branch"
echo " 📎 Head SHA: $HEAD_SHA"
# Check if this was triggered by a tag push
if [[ "$triggering_event" == "push" ]]; then
# For tag pushes, head_branch will be like "refs/tags/v1.0.0" or just "v1.0.0"
if [[ "$head_branch" == refs/tags/* ]]; then
# Extract tag name from refs/tags/TAG_NAME
tag_name="${head_branch#refs/tags/}"
version="$tag_name"
elif [[ "$head_branch" =~ ^v?[0-9]+\.[0-9]+\.[0-9]+ ]]; then
# Direct tag name like "v1.0.0" or "1.0.0-alpha.1"
version="$head_branch"
elif [[ "$head_branch" == "main" ]]; then
# Regular branch push to main
build_type="development"
version="dev-${short_sha}"
should_build=false
echo "⏭️ Skipping Docker build for development version (main branch push)"
else
# Other branch push
build_type="development"
version="dev-${short_sha}"
should_build=false
echo "⏭️ Skipping Docker build for development version (branch: $head_branch)"
fi
# If we extracted a version (tag), determine release type
if [[ -n "$version" ]] && [[ "$version" != "dev-${short_sha}" ]]; then
# Remove 'v' prefix if present for consistent version format
if [[ "$version" == v* ]]; then
version="${version#v}"
fi
if [[ "$version" =~ -preview\.[0-9]+$ ]]; then
build_type="preview"
is_prerelease=true
echo "🔍 Building Docker image for preview: $version (version tags only)"
elif [[ "$version" == *"-preview"* ]]; then
echo "❌ Invalid preview tag: $version (expected suffix: -preview.<number>)" >&2
exit 1
elif [[ "$version" == *"alpha"* ]] || [[ "$version" == *"beta"* ]] || [[ "$version" == *"rc"* ]]; then
build_type="prerelease"
is_prerelease=true
# Pre-GA policy: prereleases update latest until the first stable tag exists.
if prerelease_moves_latest; then
create_latest=true
echo "🧪 Building Docker image for prerelease: $version (creating latest tag)"
else
echo "🧪 Building Docker image for prerelease: $version"
fi
else
build_type="release"
create_latest=true
echo "🚀 Building Docker image for release: $version"
fi
fi
else
# Non-push events
build_type="development"
version="dev-${short_sha}"
should_build=false
echo "⏭️ Skipping Docker build for development version (event: $triggering_event)"
fi
echo "🔄 Build triggered by workflow_run:"
echo " 📋 Conclusion: $CONCLUSION"
echo " 🌿 Branch: $HEAD_BRANCH"
echo " 📎 SHA: $HEAD_SHA"
echo " 🎯 Event: $TRIGGERING_EVENT"
elif [[ "${{ github.event_name }}" == "workflow_dispatch" ]]; then
# Manual trigger
input_version="$INPUT_VERSION"
version="${input_version}"
should_push="$INPUT_PUSH_IMAGES"
should_build=true
# Get short SHA
short_sha=$(git rev-parse --short HEAD)
echo "🎯 Manual Docker build triggered:"
echo " 📋 Requested version: $input_version"
echo " 🔧 Force rebuild: $INPUT_FORCE_REBUILD"
echo " 🚀 Push images: $should_push"
case "$input_version" in
"latest")
build_type="release"
create_latest=true
echo "🚀 Building with latest stable release version"
;;
*-preview*)
if [[ ! "$input_version" =~ ^v?[0-9]+\.[0-9]+\.[0-9]+(-[0-9A-Za-z.-]+)?-preview\.[0-9]+$ ]]; then
echo "❌ Invalid preview version: $input_version" >&2
exit 1
fi
build_type="preview"
is_prerelease=true
version="${input_version#v}"
echo "🔍 Building with preview version: $version (version tags only)"
;;
# Prerelease versions (must match first, more specific)
v*alpha*|v*beta*|v*rc*|*alpha*|*beta*|*rc*)
build_type="prerelease"
is_prerelease=true
# Pre-GA policy: prereleases update latest until the first stable tag exists.
if prerelease_moves_latest; then
create_latest=true
echo "🧪 Building with prerelease version: $input_version (creating latest tag)"
else
echo "🧪 Building with prerelease version: $input_version"
fi
;;
# Release versions (match after prereleases, more general)
v[0-9]*|[0-9]*.*.*)
build_type="release"
create_latest=true
echo "📦 Building with specific release version: $input_version"
;;
*)
# Invalid version for Docker build
should_build=false
echo "❌ Invalid version for Docker build: $input_version"
echo "⚠️ Only release versions (latest, v1.0.0, 1.0.0) and prereleases (v1.0.0-alpha1, 1.0.0-beta2) are supported"
;;
esac
if [[ "$should_build" == true && "$input_version" != "latest" ]]; then
tag_ref="refs/tags/$input_version"
if ! git ls-remote --exit-code origin "$tag_ref" >/dev/null 2>&1; then
if [[ "$input_version" == v* ]]; then
tag_ref="refs/tags/${input_version#v}"
else
tag_ref="refs/tags/v$input_version"
fi
fi
if ! git ls-remote --exit-code origin "$tag_ref" >/dev/null 2>&1; then
echo "❌ Release tag not found for Docker build: $input_version"
exit 1
fi
source_ref="$tag_ref"
fi
fi
{
echo "should_build=$should_build"
echo "should_push=$should_push"
echo "build_type=$build_type"
echo "version=$version"
echo "short_sha=$short_sha"
echo "is_prerelease=$is_prerelease"
echo "create_latest=$create_latest"
echo "source_ref=$source_ref"
} >> "$GITHUB_OUTPUT"
echo "🐳 Docker Build Summary:"
echo " - Should build: $should_build"
echo " - Should push: $should_push"
echo " - Build type: $build_type"
echo " - Version: $version"
echo " - Short SHA: $short_sha"
echo " - Is prerelease: $is_prerelease"
echo " - Create latest: $create_latest"
echo " - Source ref: $source_ref"
# Consume the trusted release certification for the exact version this
# workflow wraps and verify the certified input digests. A manual dispatch
# without that certification fails here and pushes no official tag; only
# results from the trusted verification sources carry release authority.
release-certification:
name: Release Certification
needs: build-check
if: needs.build-check.outputs.should_build == 'true'
runs-on: ubuntu-latest
timeout-minutes: 30
permissions:
contents: read
actions: read
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
ref: ${{ needs.build-check.outputs.source_ref }}
- name: Download and verify release certification
env:
GH_TOKEN: ${{ secrets.RELEASE_VALIDATION_TOKEN || github.token }}
TRIGGERING_RUN_ID: ${{ github.event.workflow_run.id }}
VERSION: ${{ needs.build-check.outputs.version }}
shell: bash
run: |
set -euo pipefail
mkdir -p certification
run_id="$TRIGGERING_RUN_ID"
if [[ -z "$run_id" ]]; then
run_id="$(gh api --method GET \
"repos/${GITHUB_REPOSITORY}/actions/workflows/build.yml/runs" \
-f branch="$VERSION" -f status=success -F per_page=1 \
--jq '.workflow_runs[0].id // empty' || true)"
fi
if [[ -z "$run_id" ]]; then
echo "No certified Build and Release run found for ${VERSION}; failing closed" >&2
exit 1
fi
if ! gh run download "$run_id" -n release-certification -D certification; then
echo "Build run ${run_id} published no release-certification artifact; failing closed" >&2
exit 1
fi
python3 scripts/release/release_gate.py verify-promotion \
--plan .config/release-validation-plan.json \
--candidate certification/candidate.json \
--summary certification/release-gate-output \
--artifact-sha256 "$(jq -r '.deliverables[0].sha256' certification/candidate.json)" \
--artifact-name "$(jq -r '.deliverables[0].name' certification/candidate.json)"
# Input digest binding: every zip this image wraps must be the
# certified byte-for-byte content, never a mutable latest pointer.
while IFS=$'\t' read -r name location expected; do
echo "Verifying certified input ${name}"
actual="$(curl -fsSL "$location" | sha256sum | awk '{print $1}')"
if [[ "$actual" != "$expected" ]]; then
echo "Certified input ${name} does not match its recorded digest" >&2
exit 1
fi
done < <(jq -r '.deliverables[] | select(.name | contains("linux")) | [.name, .location, .sha256] | @tsv' certification/candidate.json)
# Build multi-arch Docker images
# Strategy: Build images using pre-built binaries from dl.rustfs.com
# Supports both release and dev channel binaries based on build context
# Only runs when should_build is true (which includes workflow success check)
build-docker:
name: Build Docker Images
needs: [ build-check, release-certification ]
if: needs.build-check.outputs.should_build == 'true'
runs-on: ubuntu-latest
timeout-minutes: 60
timeout-minutes: 90
strategy:
fail-fast: false
matrix:
include:
- variant: musl
file: Dockerfile
suffix: ""
- variant: glibc
- variant: gnu
file: Dockerfile.glibc
suffix: "-glibc"
env:
CANDIDATE_MANIFEST: ${{ inputs.candidate_manifest }}
VARIANT: ${{ matrix.variant }}
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
ref: ${{ fromJson(inputs.candidate_manifest).source_sha }}
persist-credentials: false
ref: ${{ needs.build-check.outputs.source_ref }}
- name: Login to Docker Hub
uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
with:
username: ${{ env.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
- name: Login to GitHub Container Registry
uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
with:
registry: ghcr.io
username: ${{ secrets.GHCR_USERNAME }}
password: ${{ secrets.GHCR_PASSWORD }}
- name: Login to Quay.io
uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
with:
registry: quay.io
username: ${{ secrets.QUAY_USERNAME }}
password: ${{ secrets.QUAY_PASSWORD }}
- name: Set up QEMU
uses: docker/setup-qemu-action@c7c53464625b32c7a7e944ae62b3e17d2b600130 # v3
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f # v3
- name: Extract metadata and generate tags
id: meta
- name: Reuse completed image
id: reuse
env:
GH_TOKEN: ${{ github.token }}
run: python3 scripts/release/deliver.py reuse --kind image --target "$VARIANT"
- name: Install OCI tools
if: steps.reuse.outputs.reused != 'true'
run: |
BUILD_TYPE="${{ needs.build-check.outputs.build_type }}"
VERSION="${{ needs.build-check.outputs.version }}"
CREATE_LATEST="${{ needs.build-check.outputs.create_latest }}"
VARIANT_SUFFIX="${{ matrix.suffix }}"
# Convert version format for Dockerfile compatibility. The former
# DOCKER_CHANNEL was "release" down every branch and was passed as a
# build-arg no Dockerfile declares, so it is gone.
case "$VERSION" in
"latest")
DOCKER_RELEASE="latest"
;;
v*)
# For versioned releases (v1.0.0), remove 'v' prefix for Dockerfile
DOCKER_RELEASE="${VERSION#v}"
;;
*)
# For other versions, pass as-is
DOCKER_RELEASE="${VERSION}"
;;
esac
echo "docker_release=$DOCKER_RELEASE" >> "$GITHUB_OUTPUT"
echo "🐳 Docker build parameters:"
echo " - Original version: $VERSION"
echo " - Docker RELEASE: $DOCKER_RELEASE"
# Generate tags based on build type
# Only support release and prerelease builds (no development builds)
TAG_BASE="${VERSION}${VARIANT_SUFFIX}"
TAGS="${{ env.REGISTRY_DOCKERHUB }}:$TAG_BASE,${{ env.REGISTRY_GHCR }}:$TAG_BASE,${{ env.REGISTRY_QUAY }}:$TAG_BASE"
# Add latest when requested (stable releases, and prereleases before GA)
if [[ "$CREATE_LATEST" == "true" ]]; then
TAGS="$TAGS,${{ env.REGISTRY_DOCKERHUB }}:latest${VARIANT_SUFFIX},${{ env.REGISTRY_GHCR }}:latest${VARIANT_SUFFIX},${{ env.REGISTRY_QUAY }}:latest${VARIANT_SUFFIX}"
fi
# Always add the channel tag for prereleases, independent of latest
if [[ "$BUILD_TYPE" == "prerelease" ]]; then
# Prerelease channel tags (alpha, beta, rc)
if [[ "$VERSION" == *"alpha"* ]]; then
CHANNEL="alpha"
elif [[ "$VERSION" == *"beta"* ]]; then
CHANNEL="beta"
elif [[ "$VERSION" == *"rc"* ]]; then
CHANNEL="rc"
fi
if [[ -n "$CHANNEL" ]]; then
TAGS="$TAGS,${{ env.REGISTRY_DOCKERHUB }}:${CHANNEL}${VARIANT_SUFFIX},${{ env.REGISTRY_GHCR }}:${CHANNEL}${VARIANT_SUFFIX},${{ env.REGISTRY_QUAY }}:${CHANNEL}${VARIANT_SUFFIX}"
fi
fi
# Output tags
echo "tags=$TAGS" >> "$GITHUB_OUTPUT"
# Staging identity for this run's promotion: official tags are only
# created after the staged image verifies against the certification,
# and they point at this same digest (no rebuild-after-verify).
STAGING_TAG="staging-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}${VARIANT_SUFFIX}"
STAGING_TAGS="${{ env.REGISTRY_DOCKERHUB }}:${STAGING_TAG},${{ env.REGISTRY_GHCR }}:${STAGING_TAG},${{ env.REGISTRY_QUAY }}:${STAGING_TAG}"
echo "staging_tags=$STAGING_TAGS" >> "$GITHUB_OUTPUT"
echo "staging_tag=$STAGING_TAG" >> "$GITHUB_OUTPUT"
# Generate labels
LABELS="org.opencontainers.image.title=RustFS"
LABELS="$LABELS,org.opencontainers.image.description=RustFS distributed object storage system"
LABELS="$LABELS,org.opencontainers.image.version=$VERSION"
SOURCE_REVISION="$(git rev-parse HEAD)"
LABELS="$LABELS,org.opencontainers.image.revision=$SOURCE_REVISION"
LABELS="$LABELS,org.opencontainers.image.source=${{ github.server_url }}/${{ github.repository }}"
LABELS="$LABELS,org.opencontainers.image.created=$(date -u +'%Y-%m-%dT%H:%M:%SZ')"
LABELS="$LABELS,org.opencontainers.image.build-type=$BUILD_TYPE"
echo "labels=$LABELS" >> "$GITHUB_OUTPUT"
echo "🐳 Generated Docker tags:"
echo "$TAGS" | tr ',' '\n' | sed 's/^/ - /'
echo "📋 Build type: $BUILD_TYPE"
echo "🔖 Version: $VERSION"
- name: Build and push Docker image
sudo apt-get update
sudo apt-get install -y skopeo
- uses: docker/setup-qemu-action@c7c53464625b32c7a7e944ae62b3e17d2b600130 # v3
if: steps.reuse.outputs.reused != 'true'
- uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f # v3
if: steps.reuse.outputs.reused != 'true'
- name: Supply accepted binaries and pinned CLI
if: steps.reuse.outputs.reused != 'true'
env:
GH_TOKEN: ${{ github.token }}
run: python3 scripts/release/deliver.py image-prepare --target "$VARIANT"
- name: Build OCI archive without registry writes
if: steps.reuse.outputs.reused != 'true'
uses: docker/build-push-action@10e90e3645eae34f1e60eeb005ba3a3d33f178e8 # v6
with:
context: .
file: ${{ matrix.file }}
platforms: ${{ env.DOCKER_PLATFORMS }}
push: ${{ needs.build-check.outputs.should_push == 'true' }}
# Push to staging tags only: official tags are created by the
# certification-gated promotion step below from this same digest.
tags: ${{ steps.meta.outputs.staging_tags }}
labels: ${{ steps.meta.outputs.labels }}
# No layer cache. This build compiles nothing — it downloads a
# release zip and runs apk/apt — so the cache could only save the
# minute or two those take, while creating a correctness problem: with
# RELEASE=latest the binary URL is resolved by curl *inside* a RUN
# layer, and the layer key does not include what that resolved to. A
# rebuild at the same RELEASE value (dispatch with version=latest, or
# a re-run of the same version) would hit the old layer and ship the
# previous release's binary. mode=max also consumed the same 10GB
# Actions cache quota the Rust lanes are fighting over.
#
# Only RELEASE is passed: it is the sole build-arg the Dockerfiles
# declare besides TARGETARCH. BUILDTIME, VERSION, BUILD_TYPE, REVISION
# and CHANNEL were never read by any stage (and BUILDTIME's $(date ...)
# was a literal here, not a shell substitution). BUILD_DATE and VCS_REF
# are declared by the Dockerfiles but deliberately left unset —
# supplying them would change the published image labels.
platforms: linux/amd64,linux/arm64
push: false
provenance: false
outputs: type=oci,dest=delivery/image-${{ matrix.variant }}.tar
build-args: |
RELEASE=${{ steps.meta.outputs.docker_release }}
provenance: true
sbom: true
# Add retry mechanism by splitting the build process
no-cache: false
pull: true
# A later-built wrapper: the staged image must complete this platform's
# own binary identity verification before any official tag is written.
# It may not inherit an unverified package's conclusion.
- name: Verify staged image against certification
if: needs.build-check.outputs.should_push == 'true'
env:
GH_TOKEN: ${{ secrets.RELEASE_VALIDATION_TOKEN || github.token }}
TRIGGERING_RUN_ID: ${{ github.event.workflow_run.id }}
VERSION: ${{ needs.build-check.outputs.version }}
shell: bash
run: |
set -euo pipefail
mkdir -p certification
run_id="$TRIGGERING_RUN_ID"
if [[ -z "$run_id" ]]; then
run_id="$(gh api --method GET \
"repos/${GITHUB_REPOSITORY}/actions/workflows/build.yml/runs" \
-f branch="$VERSION" -f status=success -F per_page=1 \
--jq '.workflow_runs[0].id // empty' || true)"
fi
gh run download "$run_id" -n release-certification -D certification
STAGING_REF="${{ env.REGISTRY_DOCKERHUB }}:staging-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}${{ matrix.suffix }}"
case "${{ matrix.variant }}" in
musl) libc="musl" ;;
glibc) libc="gnu" ;;
esac
for platform in linux/amd64 linux/arm64; do
case "$platform" in
linux/amd64) arch_sub="x86_64" ;;
linux/arm64) arch_sub="aarch64" ;;
esac
docker pull --platform "$platform" "$STAGING_REF"
container="$(docker create --platform "$platform" "$STAGING_REF")"
docker cp "$container:/usr/bin/rustfs" "$RUNNER_TEMP/staged-rustfs"
docker rm "$container" >/dev/null
actual="$(sha256sum "$RUNNER_TEMP/staged-rustfs" | awk '{print $1}')"
expected="$(jq -r --arg needle "linux-${arch_sub}-${libc}" \
'[.deliverables[] | select(.name | contains($needle)) | .binary_sha256][0] // empty' \
certification/candidate.json)"
if [[ -z "$expected" || "$actual" != "$expected" ]]; then
echo "Staged image binary for ${platform} does not match the certified binary digest" >&2
exit 1
fi
python3 scripts/release/release_gate.py verify-promotion \
--plan .config/release-validation-plan.json \
--candidate certification/candidate.json \
--summary certification/release-gate-output \
--artifact "$RUNNER_TEMP/staged-rustfs" \
--artifact-role binary
done
- name: Promote staged image to official tags
if: needs.build-check.outputs.should_push == 'true'
shell: bash
run: |
set -euo pipefail
STAGING_REF="${{ env.REGISTRY_DOCKERHUB }}:staging-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}${{ matrix.suffix }}"
IFS=',' read -r -a official_tags <<< "${{ steps.meta.outputs.tags }}"
for tag in "${official_tags[@]}"; do
docker buildx imagetools create -t "$tag" "$STAGING_REF"
echo "Promoted $tag -> $STAGING_REF"
done
# Downstream consumers (Helm) verify this record instead of inheriting
# an unverified wrapper's conclusion.
- name: Record image promotion
if: needs.build-check.outputs.should_push == 'true'
shell: bash
env:
GH_TOKEN: ${{ secrets.RELEASE_VALIDATION_TOKEN || github.token }}
TRIGGERING_RUN_ID: ${{ github.event.workflow_run.id }}
VERSION: ${{ needs.build-check.outputs.version }}
run: |
set -euo pipefail
mkdir -p certification
run_id="$TRIGGERING_RUN_ID"
if [[ -z "$run_id" ]]; then
run_id="$(gh api --method GET \
"repos/${GITHUB_REPOSITORY}/actions/workflows/build.yml/runs" \
-f branch="$VERSION" -f status=success -F per_page=1 \
--jq '.workflow_runs[0].id // empty' || true)"
fi
gh run download "$run_id" -n release-certification -D certification
STAGING_REF="${{ env.REGISTRY_DOCKERHUB }}:staging-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}${{ matrix.suffix }}"
digest="$(docker buildx imagetools inspect "$STAGING_REF" --format '{{json .Manifest}}' | jq -r '.digest')"
candidate_sha="$(python3 -c "import json,hashlib,sys; sys.path.insert(0,'scripts/release'); import release_gate; print(release_gate.canonical_sha256(json.load(open('certification/candidate.json'))))")"
jq -n \
--arg version "$VERSION" \
--arg digest "$digest" \
--arg staging_ref "$STAGING_REF" \
--arg candidate_sha256 "$candidate_sha" \
--arg tags "${{ steps.meta.outputs.tags }}" \
--slurpfile candidate certification/candidate.json \
'{schema_version: 1, version: $version, image_digest: $digest, staging_ref: $staging_ref,
promoted_tags: ($tags | split(",")), candidate_sha256: $candidate_sha256,
binary_sha256s: ($candidate[0].deliverables | map(.binary_sha256) | unique)}' \
> image-promotion-record.json
- name: Upload image promotion record
if: needs.build-check.outputs.should_push == 'true'
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: image-promotion-record${{ matrix.suffix }}
path: image-promotion-record.json
if-no-files-found: error
retention-days: 30
# Note: Manifest creation is no longer needed as we only build one variant
# Multi-arch manifests are automatically created by docker/build-push-action
# Report-only container image vulnerability scan
scan-docker-image:
name: Scan Docker Images
needs: [ build-check, build-docker ]
if: needs.build-check.outputs.should_build == 'true' && needs.build-check.outputs.should_push == 'true'
runs-on: ubuntu-latest
timeout-minutes: 30
permissions:
contents: read
security-events: write
strategy:
fail-fast: false
matrix:
include:
- variant: musl
suffix: ""
- variant: glibc
suffix: "-glibc"
steps:
- name: Login to GitHub Container Registry
# docker/login-action v3
uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9
with:
registry: ghcr.io
username: ${{ secrets.GHCR_USERNAME }}
password: ${{ secrets.GHCR_PASSWORD }}
- name: Scan image with Trivy
# aquasecurity/trivy-action v0.36.0
BINARY_SOURCE=supplied
RELEASE=${{ fromJson(inputs.candidate_manifest).version }}
VCS_REF=${{ fromJson(inputs.candidate_manifest).source_sha }}
- name: Verify wrapped binaries and startup on both architectures
if: steps.reuse.outputs.reused != 'true'
run: python3 scripts/release/deliver.py image-smoke --target "$VARIANT"
- name: Scan amd64 image before sealing
if: steps.reuse.outputs.reused != 'true'
uses: aquasecurity/trivy-action@ed142fd0673e97e23eac54620cfb913e5ce36c25
env:
TRIVY_PLATFORM: linux/amd64
with:
image-ref: ${{ env.REGISTRY_GHCR }}:${{ needs.build-check.outputs.version }}${{ matrix.suffix }}
input: delivery-scan/image-${{ matrix.variant }}-amd64.tar
format: sarif
output: trivy-${{ matrix.variant }}.sarif
ignore-unfixed: true
vuln-type: os,library
severity: CRITICAL,HIGH
output: delivery/scan-${{ matrix.variant }}-amd64.sarif
exit-code: "0" # Keep the repository's report-only vulnerability policy.
- name: Scan arm64 image before sealing
if: steps.reuse.outputs.reused != 'true'
uses: aquasecurity/trivy-action@ed142fd0673e97e23eac54620cfb913e5ce36c25
env:
TRIVY_PLATFORM: linux/arm64
with:
input: delivery-scan/image-${{ matrix.variant }}-arm64.tar
format: sarif
output: delivery/scan-${{ matrix.variant }}-arm64.sarif
exit-code: "0"
# Surface findings in the Security tab; an artifact alone is write-only
# reporting nobody reviews.
- name: Upload scan results to code scanning
uses: github/codeql-action/upload-sarif@99df26d4f13ea111d4ec1a7dddef6063f76b97e9 # v4.37.0
- name: Record verified OCI identity
if: steps.reuse.outputs.reused != 'true'
run: python3 scripts/release/deliver.py image-record --target "$VARIANT"
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
if: steps.reuse.outputs.reused != 'true'
with:
sarif_file: trivy-${{ matrix.variant }}.sarif
category: container-image-${{ matrix.variant }}
- name: Upload container scan report
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: container-image-scan-${{ matrix.variant }}
path: trivy-${{ matrix.variant }}.sarif
name: image-${{ matrix.variant }}
path: delivery/*
if-no-files-found: error
retention-days: 30
# Docker build summary
docker-summary:
name: Docker Build Summary
needs: [ build-check, build-docker ]
if: always() && needs.build-check.outputs.should_build == 'true'
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- name: Docker build completion summary
run: |
BUILD_TYPE="${{ needs.build-check.outputs.build_type }}"
VERSION="${{ needs.build-check.outputs.version }}"
CREATE_LATEST="${{ needs.build-check.outputs.create_latest }}"
echo "🐳 Docker build completed successfully!"
echo "📦 Build type: $BUILD_TYPE"
echo "🔢 Version: $VERSION"
echo "🚀 Strategy: Images using pre-built binaries (release channel only)"
echo ""
case "$BUILD_TYPE" in
"preview")
echo "🔍 Preview Docker image has been built with ${VERSION} tags"
echo "⏭️ Preview images do not update latest or prerelease channels"
;;
"release")
echo "🚀 Release Docker image has been built with ${VERSION} tags"
echo "✅ This image is ready for production use"
if [[ "$CREATE_LATEST" == "true" ]]; then
echo "🏷️ Latest tag has been created for stable release"
fi
;;
"prerelease")
echo "🧪 Prerelease Docker image has been built with ${VERSION} tags"
echo "⚠️ This is a prerelease image - use with caution"
# Prereleases move latest until the first stable tag exists (pre-GA policy).
if [[ "$CREATE_LATEST" == "true" ]]; then
echo "🏷️ Latest tag has been created for prerelease: $VERSION"
else
echo "🚫 Latest tag NOT created for prerelease"
fi
;;
*)
echo "❌ Unexpected build type: $BUILD_TYPE"
;;
esac
compression-level: 0
+51 -277
View File
@@ -1,229 +1,26 @@
# Copyright 2024 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
name: Publish helm chart to artifacthub
# Copyright 2026 RustFS Team
# SPDX-License-Identifier: Apache-2.0
name: Build candidate Helm chart
on:
workflow_run:
workflows: [ "Build and Release" ]
types: [ completed ]
workflow_dispatch:
workflow_call:
inputs:
version:
description: "Release version to publish, e.g. 1.0.0-beta.1 or v1.0.0-beta.1"
required: true
default: "1.0.0-beta.1"
candidate_manifest:
type: string
required: true
permissions:
contents: read
actions: read
jobs:
# Consume the same trusted release certification as every other publication
# exit and verify the image promotion record of the Docker workflow. The
# chart wraps no binary of its own, so it may only reference an image whose
# platform verification already passed; it must not inherit an unverified
# package's conclusion. A run without that evidence fails closed here and
# never reaches the chart repository.
release-certification:
package:
runs-on: ubuntu-latest
timeout-minutes: 30
permissions:
contents: read
actions: read
outputs:
version: ${{ steps.version.outputs.raw_tag }}
timeout-minutes: 15
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
ref: ${{ fromJson(inputs.candidate_manifest).source_sha }}
persist-credentials: false
- name: Normalize release version
id: version
env:
RAW_INPUT: ${{ github.event.inputs.version }}
RAW_BRANCH: ${{ github.event.workflow_run.head_branch }}
run: |
set -euo pipefail
if [ "${{ github.event_name }}" = "workflow_dispatch" ]; then
RAW="$RAW_INPUT"
else
RAW="$RAW_BRANCH"
fi
case "$RAW" in
refs/tags/*) RAW_TAG="${RAW#refs/tags/}" ;;
*) RAW_TAG="$RAW" ;;
esac
echo "raw_tag=$RAW_TAG" >> "$GITHUB_OUTPUT"
- name: Download and verify release certification and image record
env:
GH_TOKEN: ${{ secrets.RELEASE_VALIDATION_TOKEN || github.token }}
TRIGGERING_RUN_ID: ${{ github.event.workflow_run.id }}
RAW_TAG: ${{ steps.version.outputs.raw_tag }}
shell: bash
run: |
set -euo pipefail
mkdir -p certification image-record
build_run="$TRIGGERING_RUN_ID"
if [[ -z "$build_run" ]]; then
build_run="$(gh api --method GET \
"repos/${GITHUB_REPOSITORY}/actions/workflows/build.yml/runs" \
-f branch="$RAW_TAG" -f status=success -F per_page=1 \
--jq '.workflow_runs[0].id // empty' || true)"
fi
if [[ -z "$build_run" ]]; then
echo "No certified Build and Release run found for ${RAW_TAG}; failing closed" >&2
exit 1
fi
if ! gh run download "$build_run" -n release-certification -D certification; then
echo "Build run ${build_run} published no release-certification artifact; failing closed" >&2
exit 1
fi
python3 scripts/release/release_gate.py verify-promotion \
--plan .config/release-validation-plan.json \
--candidate certification/candidate.json \
--summary certification/release-gate-output \
--artifact-sha256 "$(jq -r '.deliverables[0].sha256' certification/candidate.json)" \
--artifact-name "$(jq -r '.deliverables[0].name' certification/candidate.json)"
docker_run="$(gh api --method GET \
"repos/${GITHUB_REPOSITORY}/actions/workflows/docker.yml/runs" \
-f branch="$RAW_TAG" -f status=success -F per_page=1 \
--jq '.workflow_runs[0].id // empty' || true)"
if [[ -z "$docker_run" ]] || ! gh run download "$docker_run" -n image-promotion-record -D image-record; then
echo "No image promotion record for ${RAW_TAG}; a chart may not wrap an unverified image" >&2
exit 1
fi
candidate_sha="$(python3 -c "import json,hashlib,sys; sys.path.insert(0,'scripts/release'); import release_gate; print(release_gate.canonical_sha256(json.load(open('certification/candidate.json'))))")"
record_sha="$(jq -r '.candidate_sha256 // empty' image-record/image-promotion-record.json)"
if [[ "$record_sha" != "$candidate_sha" ]]; then
echo "Image promotion record does not belong to this certification" >&2
exit 1
fi
overlap="$(jq -r --slurpfile candidate certification/candidate.json \
'[.binary_sha256s[]] - ($candidate[0].deliverables | map(.binary_sha256)) | length' \
image-record/image-promotion-record.json)"
if [[ "$overlap" -ne 0 ]]; then
echo "Image promotion record carries binary digests outside the certification" >&2
exit 1
fi
build-helm-package:
runs-on: ubuntu-latest
timeout-minutes: 30
needs: [ release-certification ]
if: |
(github.event_name == 'workflow_dispatch' && !contains(github.event.inputs.version, '-preview')) ||
(
github.event.workflow_run.conclusion == 'success' &&
github.event.workflow_run.event == 'push' &&
contains(github.event.workflow_run.head_branch, '.') &&
!contains(github.event.workflow_run.head_branch, '-preview')
)
outputs:
raw_tag: ${{ steps.version.outputs.raw_tag }}
app_version: ${{ steps.version.outputs.app_version }}
chart_version: ${{ steps.version.outputs.chart_version }}
steps:
- name: Checkout release source
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
ref: ${{ github.event.workflow_run.head_sha || (startsWith(inputs.version, 'refs/tags/') && inputs.version || format('refs/tags/{0}', inputs.version)) }}
persist-credentials: false
# Both inputs reach the shell through env rather than `${{ }}`
# interpolation. A git ref name may contain `$(...)` — anything without a
# space is a legal tag — and interpolation pastes it into the script
# verbatim, where bash would run it. Reading "$RAW_INPUT" instead makes it
# data.
- name: Normalize release version
id: version
env:
RAW_INPUT: ${{ github.event.inputs.version }}
RAW_BRANCH: ${{ github.event.workflow_run.head_branch }}
run: |
set -eux
if [ "${{ github.event_name }}" = "workflow_dispatch" ]; then
RAW="$RAW_INPUT"
else
RAW="$RAW_BRANCH"
fi
case "$RAW" in
refs/tags/*)
RAW_TAG="${RAW#refs/tags/}"
;;
*)
RAW_TAG="$RAW"
;;
esac
./scripts/helm_chart_version.sh "$RAW_TAG"
# Docker and Helm share the binary-build trigger. Do not publish a chart
# until the image selected by its default appVersion is pullable.
- name: Wait for release image
timeout-minutes: 20
env:
APP_VERSION: ${{ steps.version.outputs.app_version }}
run: |
set -euo pipefail
if [[ ! "$APP_VERSION" =~ ^[0-9]+\.[0-9]+\.[0-9]+(-[0-9A-Za-z.-]+)?$ ]] ||
[[ "$APP_VERSION" == *-preview* ]]; then
echo "Invalid deliverable image version: $APP_VERSION" >&2
exit 1
fi
for attempt in {1..60}; do
if token=$(curl --fail --silent --show-error --connect-timeout 10 --max-time 20 \
'https://auth.docker.io/token?service=registry.docker.io&scope=repository:rustfs/rustfs:pull' \
| jq -er '.token // empty') &&
curl --fail --silent --show-error --head --connect-timeout 10 --max-time 20 \
--header "Authorization: Bearer $token" \
--header 'Accept: application/vnd.oci.image.index.v1+json, application/vnd.docker.distribution.manifest.list.v2+json' \
"https://registry-1.docker.io/v2/rustfs/rustfs/manifests/$APP_VERSION" > /dev/null; then
echo "Release image is available: rustfs/rustfs:$APP_VERSION"
exit 0
fi
echo "Waiting for rustfs/rustfs:$APP_VERSION ($attempt/60)"
if (( attempt < 60 )); then
sleep 15
fi
done
echo "Release image is unavailable; refusing to publish Helm chart" >&2
exit 1
- name: Replace chart version and app version
env:
CHART_VERSION: ${{ steps.version.outputs.chart_version }}
APP_VERSION: ${{ steps.version.outputs.app_version }}
run: |
set -eux
sed -i -E "s/^version:.*/version: \"${CHART_VERSION}\"/" helm/rustfs/Chart.yaml
sed -i -E "s/^appVersion:.*/appVersion: \"${APP_VERSION}\"/" helm/rustfs/Chart.yaml
- name: Set up Helm
uses: azure/setup-helm@b9e51907a09c216f16ebe8536097933489208112 # v4.3.0
- name: Install yq
- uses: azure/setup-helm@b9e51907a09c216f16ebe8536097933489208112 # v4.3.0
- name: Install pinned yq
run: |
set -euo pipefail
yq_version="4.54.1"
@@ -238,66 +35,43 @@ jobs:
echo "$tools_dir" >> "$GITHUB_PATH"
"$tools_dir/yq" --version
- name: Test Helm Chart Templates
run: ./scripts/test_helm_templates.sh
- name: Package Helm Chart
run: |
set -eux
cp helm/README.md helm/rustfs/
helm package ./helm/rustfs \
--destination helm/rustfs/ \
--version "${{ steps.version.outputs.chart_version }}"
- name: Upload helm package as artifact
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: helm-package
path: |
helm/rustfs/*.tgz
helm/rustfs/README.md
retention-days: 1
publish-helm-package:
runs-on: ubuntu-latest
timeout-minutes: 30
needs: [ build-helm-package ]
if: needs.build-helm-package.result == 'success'
steps:
- name: Checkout helm package repo
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
# persist-credentials-exempt: this checkout's token IS the push credential —
# the job git-pushes to rustfs/helm below. Clearing it breaks chart publishing.
repository: rustfs/helm
token: ${{ secrets.RUSTFS_HELM_PACKAGE }}
- name: Download helm package
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: helm-package
path: ./
- name: Set up helm
uses: azure/setup-helm@b9e51907a09c216f16ebe8536097933489208112 # v4.3.0
- name: Generate index
run: helm repo index . --url https://charts.rustfs.com
# app_version is derived from the triggering tag name, and this job holds
# the cross-repository push token with rustfs/helm already checked out —
# the worst place in the repo to paste an attacker-influenced string into
# a shell line. Passed through env so bash treats it as data.
- name: Push helm package and index file
- name: Test and package the exact application version
id: chart
env:
GIT_USERNAME: ${{ secrets.USERNAME }}
GIT_EMAIL: ${{ secrets.EMAIL_ADDRESS }}
APP_VERSION: ${{ needs.build-helm-package.outputs.app_version }}
GH_TOKEN: ${{ github.token }}
CANDIDATE_MANIFEST: ${{ inputs.candidate_manifest }}
run: |
set -eux
git config --global user.name "${GIT_USERNAME}"
git config --global user.email "${GIT_EMAIL}"
git add .
git commit -m "Update rustfs helm package with ${APP_VERSION}." || echo "No changes to commit"
git push origin main
python3 - <<'PYCHART'
import sys, subprocess, shutil
from pathlib import Path
sys.path.insert(0, 'scripts/release')
import candidate as c, pipeline
inputs = pipeline.manifest_env()
current = pipeline.current_run('release-complete.yml')
if any(a['name'] == 'helm-package' for a in c.artifact_list(current['id'])):
c.download_artifact(current, 'helm-package', 'chart')
record = c.read_json('chart/chart.json')
c.require(record['inputs_sha256'] == c.digest(inputs), 'chart belongs to another candidate')
c.emit_output('reused', 'true')
else:
chart = Path('helm/rustfs/Chart.yaml')
import re
value = chart.read_text()
for key in ('version', 'appVersion'):
value, count = re.subn(rf'(?m)^{key}:.*$', f'{key}: "{inputs["version"]}"', value)
c.require(count == 1, 'ambiguous chart metadata')
chart.write_text(value)
subprocess.run(['bash', 'scripts/test_helm_templates.sh'], check=True)
shutil.copyfile('helm/README.md', 'helm/rustfs/README.md')
subprocess.run(['helm', 'package', 'helm/rustfs', '--destination', 'chart'], check=True)
path = Path('chart') / f"rustfs-{inputs['version']}.tgz"
c.write_json('chart/chart.json', {'version': inputs['version'], 'inputs_sha256': c.digest(inputs),
'name': path.name, 'sha256': c.file_digest(path), 'size': path.stat().st_size})
PYCHART
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
if: steps.chart.outputs.reused != 'true'
with:
name: helm-package
path: chart/*
if-no-files-found: error
retention-days: 30
+2 -2
View File
@@ -47,8 +47,8 @@ permissions:
contents: read
concurrency:
group: oidc-keycloak-live-${{ github.ref }}
cancel-in-progress: true
group: oidc-keycloak-live-${{ github.event_name }}-${{ github.ref }}
cancel-in-progress: ${{ github.event_name != 'schedule' }}
jobs:
oidc-keycloak-live:
+38 -825
View File
@@ -1,840 +1,53 @@
# Copyright 2024 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Package Workflow - Build DEB/RPM packages
#
# This workflow builds DEB and RPM packages from pre-built Linux binaries
# and uploads them to Cloudflare R2 and the GitHub release.
#
# Trigger:
# - workflow_run: automatically package after "Build and Release" completes
# for a release tag (the mac/windows/linux binaries are already uploaded
# to the GitHub release before packaging starts)
# - workflow_dispatch: manual fallback with a release tag and/or exact build run ID
#
# Flow:
# 1. Resolve and validate the selected Build workflow run and source identity
# 2. Download Linux binaries (x86_64-gnu, aarch64-gnu, x86_64-musl,
# aarch64-musl) from build artifacts
# 3. Build DEB packages for amd64 and arm64, gnu and musl variants
# 4. Build RPM packages for x86_64 and aarch64, gnu and musl variants
# 5. Upload all packages to Cloudflare R2 and the GitHub release
name: Package DEB/RPM
permissions:
# contents: write is required to upload packages to the GitHub release
contents: write
actions: read
# Copyright 2026 RustFS Team
# SPDX-License-Identifier: Apache-2.0
name: Build candidate packages
on:
# Follows the same pattern as docker.yml: run after the release build
# workflow completes, so packaging is triggered only by release tags
# (e.g. 1.0.0-rc.2, 1.0.0-rc.3), never by development builds.
workflow_run:
workflows: [ "Build and Release" ]
types: [ completed ]
workflow_dispatch:
workflow_call:
inputs:
tag:
description: "Release tag to package (e.g. 1.0.0-beta.12). Leave empty for latest main build."
required: false
candidate_manifest:
type: string
build_run_id:
description: "Build workflow run ID (when combined with tag, both must identify the same release commit)"
required: false
type: string
concurrency:
group: ${{ github.workflow }}-${{ github.event.workflow_run.head_branch || github.event.inputs.tag || github.run_id }}
cancel-in-progress: true
env:
HEAD_BRANCH: ${{ github.event.workflow_run.head_branch }}
WORKFLOW_RUN_ID: ${{ github.event.workflow_run.id }}
required: true
permissions:
contents: read
actions: read
jobs:
# Resolve which build run to use and extract version info
resolve:
name: Resolve Build
# Auto-trigger only from successful tag builds of "Build and Release".
# Tag pushes arrive as event == push with head_branch != main (a
# non-main push head_branch is the release tag name). Manual dispatch
# stays available as a fallback for backfills and re-runs.
if: >-
github.event_name == 'workflow_dispatch' ||
(github.event.workflow_run.conclusion == 'success' &&
github.event.workflow_run.event == 'push' &&
github.event.workflow_run.head_branch != 'main')
runs-on: ubuntu-latest # gh CLI required; the sm-standard-2 fleet does not ship it
timeout-minutes: 10
outputs:
version: ${{ steps.resolve.outputs.version }}
build_type: ${{ steps.resolve.outputs.build_type }}
build_run_id: ${{ steps.resolve.outputs.build_run_id }}
build_run_number: ${{ steps.resolve.outputs.build_run_number }}
head_sha: ${{ steps.resolve.outputs.head_sha }}
dev_sequence: ${{ steps.resolve.outputs.dev_sequence }}
tag: ${{ steps.resolve.outputs.tag }}
steps:
- name: Resolve build run
id: resolve
shell: bash
env:
GH_TOKEN: ${{ github.token }}
EVENT_NAME: ${{ github.event_name }}
REPOSITORY: ${{ github.repository }}
INPUT_TAG: ${{ github.event.inputs.tag }}
INPUT_RUN_ID: ${{ github.event.inputs.build_run_id }}
run: |
set -euo pipefail
fail() {
echo "❌ $1" >&2
exit 1
}
TAG=""
BUILD_RUN_ID=""
case "$EVENT_NAME" in
workflow_run)
TAG="$HEAD_BRANCH"
BUILD_RUN_ID="$WORKFLOW_RUN_ID"
;;
workflow_dispatch)
TAG="$INPUT_TAG"
BUILD_RUN_ID="$INPUT_RUN_ID"
;;
*) fail "unsupported event: $EVENT_NAME" ;;
esac
# Validate and classify tags before using them in API paths or logs.
semver_core='(0|[1-9][0-9]*)\.(0|[1-9][0-9]*)\.(0|[1-9][0-9]*)'
prerelease_id='(alpha|beta|rc)\.(0|[1-9][0-9]*)'
if [[ -n "$TAG" ]]; then
if [[ "$TAG" =~ ^${semver_core}(-${prerelease_id})?-preview\.(0|[1-9][0-9]*)$ ]]; then
BUILD_TYPE=preview
elif [[ "$TAG" =~ ^${semver_core}-${prerelease_id}$ ]]; then
BUILD_TYPE=prerelease
elif [[ "$TAG" =~ ^${semver_core}$ ]]; then
BUILD_TYPE=release
else
fail "tag is not a supported strict package version"
fi
else
BUILD_TYPE=development
fi
if [[ -n "$BUILD_RUN_ID" ]]; then
[[ "$BUILD_RUN_ID" =~ ^[1-9][0-9]*$ ]] || fail "build run ID must be a positive decimal integer"
echo "Using selected build run: $BUILD_RUN_ID"
elif [[ -n "$TAG" ]]; then
echo "Looking for build run for tag: $TAG"
BUILD_RUN_ID=$(gh api --method GET \
"repos/${REPOSITORY}/actions/workflows/build.yml/runs" \
-f branch="$TAG" -f status=success -F per_page=1 \
--jq '.workflow_runs[0].id // empty' 2>/dev/null || true)
if [[ -z "$BUILD_RUN_ID" ]]; then
BUILD_RUN_ID=$(gh api --method GET \
"repos/${REPOSITORY}/actions/workflows/build.yml/runs" \
-f event=push -f status=success -F per_page=100 2>/dev/null |
jq -r --arg tag "$TAG" \
'[.workflow_runs[] | select(.head_branch == $tag)][0].id // empty' || true)
fi
[[ "$BUILD_RUN_ID" =~ ^[1-9][0-9]*$ ]] || fail "no successful build run found for tag"
echo "Found build run: $BUILD_RUN_ID"
else
echo "No tag specified, looking for latest main build"
BUILD_RUN_ID=$(gh api --method GET \
"repos/${REPOSITORY}/actions/workflows/build.yml/runs" \
-f branch=main -f status=success -F per_page=1 \
--jq '.workflow_runs[0].id // empty' 2>/dev/null || true)
[[ "$BUILD_RUN_ID" =~ ^[1-9][0-9]*$ ]] || fail "no successful main build found"
echo "Latest main build: $BUILD_RUN_ID"
fi
# Fetch once and use the same immutable run metadata for identity,
# ordering, workflow provenance, and release-channel validation.
RUN_JSON=$(gh api "repos/${REPOSITORY}/actions/runs/${BUILD_RUN_ID}") ||
fail "cannot read selected build run"
RUN_ID=$(jq -r '.id // empty' <<<"$RUN_JSON")
RUN_NUMBER=$(jq -r '.run_number // empty' <<<"$RUN_JSON")
RUN_STATUS=$(jq -r '.status // empty' <<<"$RUN_JSON")
RUN_CONCLUSION=$(jq -r '.conclusion // empty' <<<"$RUN_JSON")
RUN_PATH=$(jq -r '.path // empty' <<<"$RUN_JSON")
HEAD_SHA=$(jq -r '.head_sha // empty' <<<"$RUN_JSON")
RUN_HEAD_BRANCH=$(jq -r '.head_branch // empty' <<<"$RUN_JSON")
[[ "$RUN_ID" == "$BUILD_RUN_ID" ]] || fail "run metadata ID mismatch"
[[ "$RUN_NUMBER" =~ ^[1-9][0-9]*$ ]] || fail "build run number must be a positive decimal integer"
[[ "$RUN_STATUS" == completed && "$RUN_CONCLUSION" == success ]] || fail "selected build run is not successful"
[[ "$RUN_PATH" == .github/workflows/build.yml ]] || fail "selected run is not Build and Release"
[[ "$HEAD_SHA" =~ ^[0-9a-f]{40}$ ]] || fail "selected build run has an invalid head SHA"
[[ "$RUN_HEAD_BRANCH" != *$'\n'* && -n "$RUN_HEAD_BRANCH" ]] || fail "selected build run has an invalid head branch"
if [[ -n "$TAG" ]]; then
[[ "$RUN_HEAD_BRANCH" == "$TAG" ]] || fail "tag and build run head branch do not match"
TAG_REF_JSON=$(gh api "repos/${REPOSITORY}/git/ref/tags/${TAG}") ||
fail "cannot resolve release tag ref"
TAG_OBJECT_TYPE=$(jq -r '.object.type // empty' <<<"$TAG_REF_JSON")
TAG_OBJECT_SHA=$(jq -r '.object.sha // empty' <<<"$TAG_REF_JSON")
depth=0
while [[ "$TAG_OBJECT_TYPE" == tag && $depth -lt 5 ]]; do
TAG_OBJECT_JSON=$(gh api "repos/${REPOSITORY}/git/tags/${TAG_OBJECT_SHA}") ||
fail "cannot peel annotated release tag"
TAG_OBJECT_TYPE=$(jq -r '.object.type // empty' <<<"$TAG_OBJECT_JSON")
TAG_OBJECT_SHA=$(jq -r '.object.sha // empty' <<<"$TAG_OBJECT_JSON")
depth=$((depth + 1))
done
[[ "$TAG_OBJECT_TYPE" == commit && "$TAG_OBJECT_SHA" =~ ^[0-9a-f]{40}$ ]] ||
fail "release tag does not resolve to a commit"
[[ "$TAG_OBJECT_SHA" == "$HEAD_SHA" ]] || fail "release tag commit and build run head SHA do not match"
VERSION="$TAG"
DEV_SEQUENCE=""
else
VERSION="dev-${HEAD_SHA}"
DEV_SEQUENCE="$RUN_NUMBER"
fi
{
echo "version=$VERSION"
echo "build_type=$BUILD_TYPE"
echo "build_run_id=$BUILD_RUN_ID"
echo "build_run_number=$RUN_NUMBER"
echo "head_sha=$HEAD_SHA"
echo "dev_sequence=$DEV_SEQUENCE"
echo "tag=${TAG}"
} >> "$GITHUB_OUTPUT"
echo "📊 Resolved:"
echo " Version: $VERSION"
echo " Build type: $BUILD_TYPE"
echo " Build run ID: $BUILD_RUN_ID"
echo " Build run number: $RUN_NUMBER"
# Consume the trusted release certification of the resolved build run and
# verify the certified input digests before any package is wrapped. A run
# without that certification fails closed here; packages built later must
# complete their own verification and cannot inherit an unverified
# conclusion.
release-certification:
name: Release Certification
needs: resolve
if: needs.resolve.result == 'success'
runs-on: ubuntu-latest # gh CLI required; the sm-standard-2 fleet does not ship it
timeout-minutes: 30
permissions:
contents: read
actions: read
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Download and verify release certification
env:
GH_TOKEN: ${{ secrets.RELEASE_VALIDATION_TOKEN || github.token }}
BUILD_RUN_ID: ${{ needs.resolve.outputs.build_run_id }}
BUILD_TYPE: ${{ needs.resolve.outputs.build_type }}
shell: bash
run: |
set -euo pipefail
mkdir -p certification
if [[ "$BUILD_TYPE" == "development" ]]; then
# Development artifacts are explicitly uncertified and never touch
# official or latest channels; only the staging/dev paths remain.
jq -n '{schema_version: 1, certified: false, reason: "development artifacts are explicitly uncertified"}' \
> certification/development-uncertified.json
exit 0
fi
if ! gh run download "$BUILD_RUN_ID" -n release-certification -D certification; then
echo "Build run ${BUILD_RUN_ID} published no release-certification artifact; failing closed" >&2
exit 1
fi
python3 scripts/release/release_gate.py verify-promotion \
--plan .config/release-validation-plan.json \
--candidate certification/candidate.json \
--summary certification/release-gate-output \
--artifact-sha256 "$(jq -r '.deliverables[0].sha256' certification/candidate.json)" \
--artifact-name "$(jq -r '.deliverables[0].name' certification/candidate.json)"
# Build DEB and RPM packages for each architecture and libc variant
package:
name: Package (${{ matrix.arch }}-${{ matrix.libc }})
needs: [ resolve, release-certification ]
runs-on: ubuntu-latest # gh CLI required; the sm-standard-2 fleet does not ship it
timeout-minutes: 30
runs-on: ubuntu-latest
timeout-minutes: 45
strategy:
fail-fast: false
max-parallel: 1
matrix:
include:
- arch: x86_64
libc: gnu
deb_arch: amd64
rpm_arch: x86_64
artifact_name: "rustfs-linux-x86_64-gnu"
# gnu binaries dynamically link glibc; musl binaries are static
# and run without a libc dependency.
deb_depends: "libc6 (>= 2.31)"
rpm_depends: "glibc >= 2.31"
- arch: aarch64
libc: gnu
deb_arch: arm64
rpm_arch: aarch64
artifact_name: "rustfs-linux-aarch64-gnu"
deb_depends: "libc6 (>= 2.31)"
rpm_depends: "glibc >= 2.31"
- arch: x86_64
libc: musl
deb_arch: amd64
rpm_arch: x86_64
artifact_name: "rustfs-linux-x86_64-musl"
deb_depends: ""
rpm_depends: ""
- arch: aarch64
libc: musl
deb_arch: arm64
rpm_arch: aarch64
artifact_name: "rustfs-linux-aarch64-musl"
deb_depends: ""
rpm_depends: ""
target: [linux-x86_64-musl, linux-aarch64-musl, linux-x86_64-gnu, linux-aarch64-gnu]
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
ref: ${{ fromJson(inputs.candidate_manifest).source_sha }}
persist-credentials: false
- name: Normalize package metadata
id: versions
shell: bash
env:
BUILD_TYPE: ${{ needs.resolve.outputs.build_type }}
SOURCE_VERSION: ${{ needs.resolve.outputs.version }}
DEV_SEQUENCE: ${{ needs.resolve.outputs.dev_sequence }}
ARCH: ${{ matrix.rpm_arch }}
LIBC: ${{ matrix.libc }}
run: |
set -euo pipefail
normalized=$(./scripts/release/package_versions.sh \
"$BUILD_TYPE" "$SOURCE_VERSION" "$DEV_SEQUENCE" "$ARCH" "$LIBC")
printf '%s\n' "$normalized" >> "$GITHUB_OUTPUT"
- name: Download binary artifact from build run
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
pattern: ${{ matrix.artifact_name }}*
path: ./binary-artifact
run-id: ${{ needs.resolve.outputs.build_run_id }}
github-token: ${{ github.token }}
merge-multiple: true
- name: Extract binary
id: binary
shell: bash
run: |
set -euo pipefail
ZIP_FILE=$(find ./binary-artifact -name "*.zip" -type f | head -1)
if [[ -z "$ZIP_FILE" ]]; then
echo "❌ No binary artifact found"
find ./binary-artifact -mindepth 1 -maxdepth 1 -print 2>/dev/null || true
exit 1
fi
echo "Found artifact: $ZIP_FILE"
mkdir -p ./bin
unzip -o "$ZIP_FILE" -d ./bin
if [[ ! -f ./bin/rustfs ]]; then
echo "❌ rustfs binary not found in archive"
exit 1
fi
chmod +x ./bin/rustfs
stat --printf='%n %s bytes\n' ./bin/rustfs
echo "✅ Binary extracted"
- name: Build DEB package
id: deb
shell: bash
env:
DEB_VERSION: ${{ steps.versions.outputs.deb_version }}
DEB_ARCH: ${{ matrix.deb_arch }}
DEB_FILE: ${{ steps.versions.outputs.deb_file }}
DEB_DEPENDS: ${{ matrix.deb_depends }}
LIBC: ${{ matrix.libc }}
run: |
set -euo pipefail
PKG_DIR="${DEB_FILE%.deb}"
echo "Building DEB: ${DEB_FILE}"
mkdir -p "${PKG_DIR}/DEBIAN"
mkdir -p "${PKG_DIR}/usr/bin"
mkdir -p "${PKG_DIR}/etc/default"
mkdir -p "${PKG_DIR}/lib/systemd/system"
mkdir -p "${PKG_DIR}/usr/share/doc/rustfs"
cp ./bin/rustfs "${PKG_DIR}/usr/bin/"
chmod 755 "${PKG_DIR}/usr/bin/rustfs"
cp deploy/build/rustfs.service "${PKG_DIR}/lib/systemd/system/"
cat > "${PKG_DIR}/etc/default/rustfs" << 'ENVEOF'
# RustFS Environment Configuration
# See https://docs.rustfs.com for more information
# RUSTFS_ACCESS_KEY=<changeme>
# RUSTFS_SECRET_KEY=<changeme>
# RUSTFS_VOLUMES=""
# RUSTFS_ADDRESS=":9000"
# RUSTFS_CONSOLE_ADDRESS=":9001"
# RUSTFS_CONSOLE_ENABLE=true
# RUSTFS_OBS_LOGGER_LEVEL=error
# RUSTFS_OBS_LOG_DIRECTORY="/var/log/rustfs/"
ENVEOF
cat > "${PKG_DIR}/DEBIAN/control" << EOF
Package: rustfs
Version: ${DEB_VERSION}
Section: utils
Priority: optional
Architecture: ${DEB_ARCH}
EOF
# gnu binaries dynamically link glibc and must declare it; musl
# binaries are statically linked and need no libc dependency.
if [[ -n "$DEB_DEPENDS" ]]; then
echo "Depends: ${DEB_DEPENDS}" >> "${PKG_DIR}/DEBIAN/control"
fi
cat >> "${PKG_DIR}/DEBIAN/control" << EOF
Maintainer: RustFS Team <support@rustfs.com>
Description: High-performance distributed object storage (${LIBC} build)
RustFS is a high-performance distributed object storage software
built using Rust. It is compatible with MinIO and S3 API.
Homepage: https://rustfs.com
EOF
# Declare /etc/default/rustfs as a conffile so dpkg preserves user
# modifications on upgrade instead of silently overwriting them.
cat > "${PKG_DIR}/DEBIAN/conffiles" << 'CONFFILES'
/etc/default/rustfs
CONFFILES
./scripts/release/package_service_scripts.sh after-install > "${PKG_DIR}/DEBIAN/postinst"
chmod 755 "${PKG_DIR}/DEBIAN/postinst"
./scripts/release/package_service_scripts.sh before-remove > "${PKG_DIR}/DEBIAN/prerm"
chmod 755 "${PKG_DIR}/DEBIAN/prerm"
./scripts/release/package_service_scripts.sh after-remove > "${PKG_DIR}/DEBIAN/postrm"
chmod 755 "${PKG_DIR}/DEBIAN/postrm"
cp LICENSE "${PKG_DIR}/usr/share/doc/rustfs/"
cp README.md "${PKG_DIR}/usr/share/doc/rustfs/"
fakeroot dpkg-deb --build "${PKG_DIR}" "$DEB_FILE"
[[ $(dpkg-deb -f "$DEB_FILE" Package) == rustfs ]]
[[ $(dpkg-deb -f "$DEB_FILE" Version) == "$DEB_VERSION" ]]
[[ $(dpkg-deb -f "$DEB_FILE" Architecture) == "$DEB_ARCH" ]]
dpkg-deb --fsys-tarfile "$DEB_FILE" | tar -tf - | grep -Fx './usr/bin/rustfs' >/dev/null
stat --printf='%n %s bytes\n' "$DEB_FILE"
echo "deb_file=$DEB_FILE" >> "$GITHUB_OUTPUT"
echo "✅ DEB built: $DEB_FILE"
- name: Build RPM package
id: rpm
shell: bash
env:
RPM_VERSION: ${{ steps.versions.outputs.rpm_version }}
RPM_RELEASE: ${{ steps.versions.outputs.rpm_release }}
RPM_ARCH: ${{ matrix.rpm_arch }}
RPM_FILE: ${{ steps.versions.outputs.rpm_file }}
RPM_DEPENDS: ${{ matrix.rpm_depends }}
LIBC: ${{ matrix.libc }}
run: |
set -euo pipefail
echo "Building RPM for ${RPM_ARCH} (${LIBC})"
sudo apt-get update && sudo apt-get install -y ruby ruby-dev build-essential rpm
sudo gem install fpm
./scripts/test_package_versions.sh --require-package-managers
# gnu binaries dynamically link glibc and must declare it; musl
# binaries are statically linked and need no libc dependency.
depends_args=()
if [[ -n "$RPM_DEPENDS" ]]; then
depends_args+=(--depends "$RPM_DEPENDS")
fi
# Create config file for fpm (DEB build creates it in its package dir structure,
# but fpm needs the file to exist before packaging)
mkdir -p ./tmp-pkg/etc/default
cat > ./tmp-pkg/etc/default/rustfs << 'ENVEOF'
# RustFS Environment Configuration
# See https://docs.rustfs.com for more information
# RUSTFS_ACCESS_KEY=<changeme>
# RUSTFS_SECRET_KEY=<changeme>
# RUSTFS_VOLUMES=""
# RUSTFS_ADDRESS=":9000"
# RUSTFS_CONSOLE_ADDRESS=":9001"
# RUSTFS_CONSOLE_ENABLE=true
# RUSTFS_OBS_LOGGER_LEVEL=error
# RUSTFS_OBS_LOG_DIRECTORY="/var/log/rustfs/"
ENVEOF
fpm -s dir -t rpm \
--name rustfs \
--version "$RPM_VERSION" \
--iteration "$RPM_RELEASE" \
--architecture "$RPM_ARCH" \
--package "$RPM_FILE" \
"${depends_args[@]}" \
--description "High-performance distributed object storage (${LIBC} build)" \
--maintainer "RustFS Team <support@rustfs.com>" \
--url "https://rustfs.com" \
--license "Apache-2.0" \
--before-install <(./scripts/release/package_service_scripts.sh rpm-before-install) \
--before-upgrade <(./scripts/release/package_service_scripts.sh rpm-before-upgrade) \
--after-install <(./scripts/release/package_service_scripts.sh after-install) \
--after-upgrade <(./scripts/release/package_service_scripts.sh after-install) \
--before-remove <(./scripts/release/package_service_scripts.sh before-remove) \
--after-remove <(./scripts/release/package_service_scripts.sh after-remove) \
--rpm-posttrans <(./scripts/release/package_service_scripts.sh rpm-posttrans) \
--config-files /etc/default/rustfs \
./bin/rustfs=/usr/bin/rustfs \
./tmp-pkg/etc/default/rustfs=/etc/default/rustfs \
deploy/build/rustfs.service=/lib/systemd/system/rustfs.service \
LICENSE=/usr/share/doc/rustfs/LICENSE \
README.md=/usr/share/doc/rustfs/README.md
if [[ ! -f "$RPM_FILE" ]]; then
echo "❌ RPM build failed"
exit 1
fi
RPM_METADATA=$(rpm -qp --qf '%{NAME}\n%{VERSION}\n%{RELEASE}\n%{ARCH}\n' "$RPM_FILE")
EXPECTED_METADATA=$(printf 'rustfs\n%s\n%s\n%s' "$RPM_VERSION" "$RPM_RELEASE" "$RPM_ARCH")
[[ "$RPM_METADATA" == "$EXPECTED_METADATA" ]]
rpm -qpl "$RPM_FILE" | grep -Fx '/usr/bin/rustfs' >/dev/null
stat --printf='%n %s bytes\n' "$RPM_FILE"
echo "rpm_file=$RPM_FILE" >> "$GITHUB_OUTPUT"
echo "✅ RPM built: $RPM_FILE"
- name: Download certification evidence
if: needs.resolve.outputs.build_type != 'development'
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: release-certification
path: ./certification
# Later-built wrapper verification: the binary inside each package must
# match the certified binary digest for this platform before anything is
# uploaded, and the package itself stays on staging until promotion.
- name: Verify wrapped binary against certification
if: needs.resolve.outputs.build_type != 'development'
shell: bash
env:
DEB_FILE: ${{ steps.deb.outputs.deb_file }}
RPM_FILE: ${{ steps.rpm.outputs.rpm_file }}
ARTIFACT_NAME: ${{ matrix.artifact_name }}
run: |
set -euo pipefail
expected="$(jq -r --arg needle "$ARTIFACT_NAME" \
'[.deliverables[] | select(.name | contains($needle)) | .binary_sha256][0] // empty' \
certification/candidate.json)"
if [[ -z "$expected" ]]; then
echo "No certified binary digest recorded for ${ARTIFACT_NAME}" >&2
exit 1
fi
if ! command -v rpm2cpio >/dev/null 2>&1; then
sudo apt-get update && sudo apt-get install -y rpm2cpio
fi
if ! command -v cpio >/dev/null 2>&1; then
sudo apt-get update && sudo apt-get install -y cpio
fi
for package_file in "$DEB_FILE" "$RPM_FILE"; do
case "$package_file" in
*.deb)
dpkg-deb --fsys-tarfile "$package_file" | tar -xOf - ./usr/bin/rustfs > "$RUNNER_TEMP/wrapped-rustfs"
;;
*.rpm)
rpm2cpio "$package_file" | cpio -i --to-stdout ./usr/bin/rustfs > "$RUNNER_TEMP/wrapped-rustfs"
;;
esac
python3 scripts/release/release_gate.py verify-promotion \
--plan .config/release-validation-plan.json \
--candidate certification/candidate.json \
--summary certification/release-gate-output \
--artifact "$RUNNER_TEMP/wrapped-rustfs" \
--artifact-role binary
done
- name: Upload packages to artifacts
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: packages-${{ matrix.arch }}-${{ matrix.libc }}
path: |
*.deb
*.rpm
retention-days: 30
- name: Upload packages to Cloudflare R2
if: needs.resolve.outputs.build_type != 'preview' && env.R2_ACCESS_KEY_ID != ''
env:
R2_ACCESS_KEY_ID: ${{ secrets.R2_ACCESS_KEY_ID }}
R2_SECRET_ACCESS_KEY: ${{ secrets.R2_SECRET_ACCESS_KEY }}
R2_ENDPOINT: ${{ secrets.R2_ENDPOINT }}
R2_BUCKET: ${{ secrets.R2_BUCKET }}
AWS_EC2_METADATA_DISABLED: true
BUILD_TYPE: ${{ needs.resolve.outputs.build_type }}
DEB_FILE: ${{ steps.deb.outputs.deb_file }}
RPM_FILE: ${{ steps.rpm.outputs.rpm_file }}
shell: bash
run: |
set -euo pipefail
if [[ -z "$R2_ACCESS_KEY_ID" || -z "$R2_SECRET_ACCESS_KEY" || -z "$R2_ENDPOINT" || -z "$R2_BUCKET" ]]; then
echo "⚠️ R2 credentials missing, skipping upload"
exit 0
fi
if ! command -v aws >/dev/null 2>&1; then
sudo apt-get update && sudo apt-get install -y awscli
fi
export AWS_ACCESS_KEY_ID="$R2_ACCESS_KEY_ID"
export AWS_SECRET_ACCESS_KEY="$R2_SECRET_ACCESS_KEY"
export AWS_DEFAULT_REGION="auto"
# Build-phase writes stay on staging for release types: the official
# packages/release and packages/latest paths are written only by the
# certification-gated promote-packages job, from these same staged
# objects. Development packages keep the uncertified dev channel.
if [[ "$BUILD_TYPE" == "development" ]]; then
R2_PREFIX="artifacts/rustfs/packages/dev"
else
R2_PREFIX="artifacts/rustfs/packages/staging/${GITHUB_RUN_ID}/${GITHUB_RUN_ATTEMPT}"
fi
R2_PATH="s3://${R2_BUCKET}/${R2_PREFIX}/"
echo "📤 Uploading to $R2_PATH"
for f in "$DEB_FILE" "$RPM_FILE"; do
if [[ -n "$f" && -f "$f" ]]; then
echo "Uploading: $f"
aws s3 cp "$f" "$R2_PATH" --endpoint-url "$R2_ENDPOINT" --only-show-errors
fi
done
echo "✅ Upload complete"
- name: Upload packages to GitHub Release
if: needs.resolve.outputs.tag != ''
- name: Reuse completed package set
id: reuse
env:
GH_TOKEN: ${{ github.token }}
TAG: ${{ needs.resolve.outputs.tag }}
DEB_FILE: ${{ steps.deb.outputs.deb_file }}
RPM_FILE: ${{ steps.rpm.outputs.rpm_file }}
shell: bash
CANDIDATE_MANIFEST: ${{ inputs.candidate_manifest }}
TARGET: ${{ matrix.target }}
run: python3 scripts/release/deliver.py reuse --kind packages --target "$TARGET"
- name: Install packaging tools
if: steps.reuse.outputs.reused != 'true'
run: |
set -euo pipefail
# Upload the packages, then refresh the release checksums so the new
# assets are covered, matching the binary release flow.
for f in "$DEB_FILE" "$RPM_FILE"; do
if [[ -n "$f" && -f "$f" ]]; then
echo "📤 Uploading $(basename "$f") to GitHub release ${TAG}..."
gh release upload "$TAG" "$f" --clobber
fi
done
CHECKSUM_DIR="$(mktemp -d)"
gh release download "$TAG" -p 'SHA256SUMS' -p 'SHA512SUMS' \
-D "$CHECKSUM_DIR" --clobber
for spec in "SHA256SUMS:sha256sum" "SHA512SUMS:sha512sum"; do
asset="${spec%%:*}"
checksum_cmd="${spec##*:}"
checksum_file="${CHECKSUM_DIR}/${asset}"
for f in "$DEB_FILE" "$RPM_FILE"; do
if [[ -n "$f" && -f "$f" ]]; then
base="$(basename "$f")"
# GitHub stores release asset names with '~' normalized to '.'.
# Current package file names embed the raw tag and contain no
# '~', but older releases may still ship such assets, so the
# checksum entries must reference the name as stored on the
# release.
github_base="${base//\~/.}"
# Remove any stale entry (both naming variants), then append
grep -Fv -- "$base" "$checksum_file" > "${checksum_file}.tmp" || true
grep -Fv -- "$github_base" "${checksum_file}.tmp" > "${checksum_file}.tmp2" || true
mv "${checksum_file}.tmp2" "$checksum_file"
digest=$("$checksum_cmd" -- "$f" | awk '{print $1}')
printf '%s %s\n' "$digest" "$github_base" >> "$checksum_file"
fi
done
echo "📤 Updating ${asset} for release ${TAG}..."
gh release upload "$TAG" "$checksum_file" --clobber
done
echo "✅ GitHub release assets updated"
# The only writer of the official packages/release and packages/latest R2
# paths. Promotion re-checks each package's wrapped binary against the
# passed certification list and copies the staged objects byte-for-byte:
# no rebuild-after-verify, no mutable latest resolution downstream.
promote-packages:
name: Promote Packages
needs: [ resolve, package, release-certification ]
# Target-channel lock for the official package paths.
concurrency:
group: release-channel-r2-packages
cancel-in-progress: false
if: needs.resolve.outputs.tag != '' && (needs.resolve.outputs.build_type == 'release' || needs.resolve.outputs.build_type == 'prerelease')
runs-on: ubuntu-latest # gh CLI required; the sm-standard-2 fleet does not ship it
timeout-minutes: 30
permissions:
contents: read
actions: read
env:
R2_ACCESS_KEY_ID: ${{ secrets.R2_ACCESS_KEY_ID }}
R2_SECRET_ACCESS_KEY: ${{ secrets.R2_SECRET_ACCESS_KEY }}
R2_ENDPOINT: ${{ secrets.R2_ENDPOINT }}
R2_BUCKET: ${{ secrets.R2_BUCKET }}
AWS_EC2_METADATA_DISABLED: true
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Download packages and certification
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
path: ./packages
pattern: packages-*
merge-multiple: true
- name: Download certification evidence
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: release-certification
path: ./certification
- name: Verify and promote staged packages
shell: bash
run: |
set -euo pipefail
if [[ -z "$R2_ACCESS_KEY_ID" || -z "$R2_SECRET_ACCESS_KEY" || -z "$R2_ENDPOINT" || -z "$R2_BUCKET" ]]; then
echo "❌ R2 credentials missing; refusing to promote to the official channel" >&2
exit 1
fi
STAGING_PREFIX="s3://${R2_BUCKET}/artifacts/rustfs/packages/staging/${GITHUB_RUN_ID}/${GITHUB_RUN_ATTEMPT}/"
RELEASE_PATH="s3://${R2_BUCKET}/artifacts/rustfs/packages/release/"
LATEST_PATH="s3://${R2_BUCKET}/artifacts/rustfs/packages/latest/"
if ! command -v aws >/dev/null 2>&1; then
sudo apt-get update && sudo apt-get install -y awscli
fi
export AWS_ACCESS_KEY_ID="$R2_ACCESS_KEY_ID"
export AWS_SECRET_ACCESS_KEY="$R2_SECRET_ACCESS_KEY"
export AWS_DEFAULT_REGION="auto"
if ! command -v rpm2cpio >/dev/null 2>&1; then
sudo apt-get install -y rpm2cpio
fi
if ! command -v cpio >/dev/null 2>&1; then
sudo apt-get install -y cpio
fi
promoted=0
for package_file in packages/*.deb packages/*.rpm; do
[[ -f "$package_file" ]] || continue
name="$(basename "$package_file")"
case "$package_file" in
*.deb)
dpkg-deb --fsys-tarfile "$package_file" | tar -xOf - ./usr/bin/rustfs > "$RUNNER_TEMP/wrapped-rustfs"
;;
*.rpm)
rpm2cpio "$package_file" | cpio -i --to-stdout ./usr/bin/rustfs > "$RUNNER_TEMP/wrapped-rustfs"
;;
esac
python3 scripts/release/release_gate.py verify-promotion \
--plan .config/release-validation-plan.json \
--candidate certification/candidate.json \
--summary certification/release-gate-output \
--artifact "$RUNNER_TEMP/wrapped-rustfs" \
--artifact-role binary
staged="$(mktemp)"
aws s3 cp "${STAGING_PREFIX}${name}" "$staged" --endpoint-url "$R2_ENDPOINT" --only-show-errors
staged_sha="$(sha256sum "$staged" | awk '{print $1}')"
built_sha="$(sha256sum "$package_file" | awk '{print $1}')"
if [[ "$staged_sha" != "$built_sha" ]]; then
echo "❌ Staged package ${name} does not match the verified bytes" >&2
exit 1
fi
aws s3 cp "$staged" "${RELEASE_PATH}${name}" --endpoint-url "$R2_ENDPOINT" --only-show-errors
aws s3 cp "$staged" "${LATEST_PATH}${name}" --endpoint-url "$R2_ENDPOINT" --only-show-errors
rm -f "$staged"
promoted=$((promoted + 1))
echo "✅ Promoted ${name} (${built_sha})"
done
if [[ "$promoted" -eq 0 ]]; then
echo "No staged packages found to promote" >&2
exit 1
fi
# Summary
summary:
name: Summary
needs: [ resolve, package ]
# Every completed Build and Release run triggers this workflow; when
# resolve declined it (not a release tag), there is nothing to summarise.
if: always() && needs.resolve.result != 'skipped'
runs-on: ubuntu-latest
timeout-minutes: 5
steps:
- name: Print summary
shell: bash
sudo apt-get update
sudo apt-get install -y ruby ruby-dev build-essential rpm fakeroot cpio
sudo gem install fpm --version 1.17.0 --no-document
- name: Package accepted executable and verify contents
if: steps.reuse.outputs.reused != 'true'
env:
SUMMARY_VERSION: ${{ needs.resolve.outputs.version }}
SUMMARY_BUILD_TYPE: ${{ needs.resolve.outputs.build_type }}
SUMMARY_BUILD_RUN_ID: ${{ needs.resolve.outputs.build_run_id }}
SUMMARY_PACKAGE_STATUS: ${{ needs.package.result }}
run: |
{
echo "## 📦 Package Summary"
echo ""
echo "| Item | Value |"
echo "|------|-------|"
echo "| Version | \`${SUMMARY_VERSION}\` |"
echo "| Build Type | ${SUMMARY_BUILD_TYPE} |"
echo "| Build Run | #${SUMMARY_BUILD_RUN_ID} |"
echo "| Package Status | ${SUMMARY_PACKAGE_STATUS} |"
} >> "$GITHUB_STEP_SUMMARY"
GH_TOKEN: ${{ github.token }}
CANDIDATE_MANIFEST: ${{ inputs.candidate_manifest }}
TARGET: ${{ matrix.target }}
run: python3 scripts/release/deliver.py package --target "$TARGET" --output delivery
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
if: steps.reuse.outputs.reused != 'true'
with:
name: packages-${{ matrix.target }}
path: delivery/*
if-no-files-found: error
compression-level: 0
retention-days: 30
+133
View File
@@ -0,0 +1,133 @@
# Copyright 2026 RustFS Team
# SPDX-License-Identifier: Apache-2.0
name: Release candidate (Linux)
on:
workflow_dispatch:
inputs:
version:
description: Final product version (no preview suffix)
type: string
required: true
source_sha:
description: Full reviewed source commit
type: string
required: true
permissions:
contents: read
actions: read
concurrency:
group: release-candidate-${{ inputs.version }}
cancel-in-progress: false
jobs:
freeze:
if: github.repository == 'rustfs/rustfs' && github.ref == 'refs/heads/main'
runs-on: ubuntu-latest
timeout-minutes: 15
outputs:
manifest: ${{ steps.freeze.outputs.manifest }}
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
fetch-depth: 0
persist-credentials: false
- name: Freeze or recover original inputs
id: freeze
env:
GH_TOKEN: ${{ github.token }}
VERSION: ${{ inputs.version }}
SOURCE_SHA: ${{ inputs.source_sha }}
run: python3 scripts/release/pipeline.py prepare --version "$VERSION" --source "$SOURCE_SHA" --directory frozen
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
if: steps.freeze.outputs.reused != 'true'
with:
name: release-inputs
path: frozen/inputs.json
if-no-files-found: error
retention-days: 30
linux:
needs: freeze
uses: ./.github/workflows/build.yml
with:
candidate_manifest: ${{ needs.freeze.outputs.manifest }}
platforms: linux
upgrade:
needs: [freeze, linux]
runs-on: ubuntu-latest
timeout-minutes: 150
env:
RUSTUP_TOOLCHAIN: ${{ fromJson(needs.freeze.outputs.manifest).toolchain }}
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
ref: ${{ fromJson(needs.freeze.outputs.manifest).workflow_sha }}
persist-credentials: false
- uses: ./.github/actions/setup
with:
rust-version: ${{ fromJson(needs.freeze.outputs.manifest).toolchain }}
cache-shared-key: release-upgrade-harness-hosted
cache-save-if: ${{ github.ref == 'refs/heads/main' }}
install-build-packaging-tools: "false"
install-test-tools: "false"
- name: Verify archived server against previous releases
env:
GH_TOKEN: ${{ github.token }}
CANDIDATE_MANIFEST: ${{ needs.freeze.outputs.manifest }}
run: |
python3 - <<'PYVERIFY'
import sys
from pathlib import Path
sys.path.insert(0, 'scripts/release')
import candidate as c, pipeline, verify
inputs = pipeline.manifest_env()
run = pipeline.current_run('release-candidate.yml')
if any(a['name'] == 'release-upgrade' for a in c.artifact_list(run['id'])):
c.download_artifact(run, 'release-upgrade', 'receipts')
pipeline.verify_evidence(inputs, Path('receipts'), {run['id']: run})
c.emit_output('reused', 'true')
else:
target = 'linux-x86_64-gnu'
c.download_artifact(run, f"candidate-{inputs['candidate_id']}-{target}", 'binary')
verify.upgrade(inputs, Path('binary') / f"rustfs-{target}-v{inputs['version']}.zip", Path('receipts'))
PYVERIFY
id: verify
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
if: steps.verify.outputs.reused != 'true'
with:
name: release-upgrade
path: receipts/*
if-no-files-found: error
retention-days: 30
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
if: failure()
with:
name: release-upgrade-failure-${{ github.run_attempt }}
path: receipts/*
retention-days: 7
seal-linux:
needs: [freeze, linux, upgrade]
runs-on: ubuntu-latest
timeout-minutes: 30
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Collect trusted receipts and seal Linux bytes
id: seal
env:
GH_TOKEN: ${{ github.token }}
run: |
python3 scripts/release/pipeline.py linux-seal --directory sealed
{
echo '## Linux acceptance'
echo 'Use this run ID and manifest SHA-256 when starting Release candidate (complete).'
sha256sum sealed/linux-manifest.json
echo 'Record cluster, performance, integrity and upgrade acceptance in backlog before continuing.'
} >> "$GITHUB_STEP_SUMMARY"
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
if: steps.seal.outputs.reused != 'true'
with:
name: release-linux
path: sealed/*
if-no-files-found: error
compression-level: 0
retention-days: 30
+97
View File
@@ -0,0 +1,97 @@
# Copyright 2026 RustFS Team
# SPDX-License-Identifier: Apache-2.0
name: Release candidate (complete)
on:
workflow_dispatch:
inputs:
candidate_run_id:
description: Successful Linux candidate run ID
type: string
required: true
linux_manifest_sha256:
description: Exact accepted Linux manifest SHA-256
type: string
required: true
acceptance_report:
description: Backlog report accepting cluster, performance, integrity and upgrade results
type: string
required: true
permissions:
contents: read
actions: read
concurrency:
group: release-complete-${{ inputs.candidate_run_id }}
cancel-in-progress: false
jobs:
acceptance:
if: github.repository == 'rustfs/rustfs' && github.ref == 'refs/heads/main'
runs-on: ubuntu-latest
timeout-minutes: 30
environment: release-linux-acceptance
outputs:
manifest: ${{ steps.accept.outputs.manifest }}
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
fetch-depth: 0
persist-credentials: false
- name: Bind acceptance to the exact Linux manifest
id: accept
env:
GH_TOKEN: ${{ github.token }}
CANDIDATE_RUN: ${{ inputs.candidate_run_id }}
MANIFEST_SHA: ${{ inputs.linux_manifest_sha256 }}
REPORT: ${{ inputs.acceptance_report }}
run: python3 scripts/release/pipeline.py complete-prepare --run-id "$CANDIDATE_RUN" --digest "$MANIFEST_SHA" --report "$REPORT" --directory linux
desktop:
needs: acceptance
uses: ./.github/workflows/build.yml
with:
candidate_manifest: ${{ needs.acceptance.outputs.manifest }}
platforms: desktop
packages:
needs: acceptance
uses: ./.github/workflows/package.yml
with:
candidate_manifest: ${{ needs.acceptance.outputs.manifest }}
images:
needs: acceptance
uses: ./.github/workflows/docker.yml
with:
candidate_manifest: ${{ needs.acceptance.outputs.manifest }}
chart:
needs: acceptance
uses: ./.github/workflows/helm-package.yml
with:
candidate_manifest: ${{ needs.acceptance.outputs.manifest }}
seal:
needs: [acceptance, desktop, packages, images, chart]
runs-on: ubuntu-latest
timeout-minutes: 45
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
fetch-depth: 0
persist-credentials: false
- name: Verify every deliverable and seal the candidate
id: seal
env:
GH_TOKEN: ${{ github.token }}
CANDIDATE_RUN: ${{ inputs.candidate_run_id }}
MANIFEST_SHA: ${{ inputs.linux_manifest_sha256 }}
REPORT: ${{ inputs.acceptance_report }}
run: |
python3 scripts/release/pipeline.py complete-seal --run-id "$CANDIDATE_RUN" --digest "$MANIFEST_SHA" --report "$REPORT" --directory sealed
{
echo '## Final publication approval'
echo 'Review this manifest, then explicitly dispatch Publish accepted release with its SHA-256.'
sha256sum sealed/release-manifest.json
} >> "$GITHUB_STEP_SUMMARY"
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
if: steps.seal.outputs.reused != 'true'
with:
name: release-seal
path: sealed/*
if-no-files-found: error
compression-level: 0
retention-days: 30
+193
View File
@@ -0,0 +1,193 @@
# Copyright 2026 RustFS Team
# SPDX-License-Identifier: Apache-2.0
name: Publish accepted release
on:
workflow_dispatch:
inputs:
complete_run_id:
description: Successful completed candidate run ID
type: string
required: true
manifest_sha256:
description: Exact approved final manifest SHA-256 (authorizes publication)
type: string
required: true
permissions:
contents: read
actions: read
concurrency:
group: release-publish-${{ inputs.complete_run_id }}
cancel-in-progress: false
jobs:
verify:
if: github.repository == 'rustfs/rustfs' && github.ref == 'refs/heads/main'
runs-on: ubuntu-latest
timeout-minutes: 45
outputs:
version: ${{ steps.verify.outputs.version }}
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
fetch-depth: 0
persist-credentials: false
- name: Recheck approved bytes and trusted certification
id: verify
env:
GH_TOKEN: ${{ github.token }}
COMPLETE_RUN: ${{ inputs.complete_run_id }}
MANIFEST_SHA: ${{ inputs.manifest_sha256 }}
run: python3 scripts/release/publish.py verify --run-id "$COMPLETE_RUN" --digest "$MANIFEST_SHA" --directory sealed
version:
needs: verify
runs-on: ubuntu-latest
timeout-minutes: 90
concurrency:
group: release-version-${{ needs.verify.outputs.version }}
cancel-in-progress: false
environment: release-publication
permissions:
contents: write
actions: read
packages: write
security-events: write
env:
GH_TOKEN: ${{ github.token }}
COMPLETE_RUN: ${{ inputs.complete_run_id }}
MANIFEST_SHA: ${{ inputs.manifest_sha256 }}
AWS_ACCESS_KEY_ID: ${{ secrets.R2_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.R2_SECRET_ACCESS_KEY }}
AWS_DEFAULT_REGION: auto
AWS_REQUEST_CHECKSUM_CALCULATION: when_required
AWS_RESPONSE_CHECKSUM_VALIDATION: when_required
AWS_EC2_METADATA_DISABLED: "true"
R2_ENDPOINT: ${{ secrets.R2_ENDPOINT }}
R2_BUCKET: ${{ secrets.R2_BUCKET }}
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
fetch-depth: 0
persist-credentials: false
- name: Install promotion tools
run: |
sudo apt-get update
sudo apt-get install -y skopeo
aws --version
- uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
with:
username: rustfs
password: ${{ secrets.DOCKERHUB_TOKEN }}
- uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
with:
registry: ghcr.io
username: ${{ secrets.GHCR_USERNAME }}
password: ${{ secrets.GHCR_PASSWORD }}
- uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
with:
registry: quay.io
username: ${{ secrets.QUAY_USERNAME }}
password: ${{ secrets.QUAY_PASSWORD }}
- name: Share registry credentials with the OCI copier
run: |
install -m 600 "$HOME/.docker/config.json" "$RUNNER_TEMP/registry-auth.json"
echo "REGISTRY_AUTH_FILE=$RUNNER_TEMP/registry-auth.json" >> "$GITHUB_ENV"
- name: Promote verified version
run: python3 scripts/release/publish.py version --run-id "$COMPLETE_RUN" --digest "$MANIFEST_SHA" --directory sealed
- name: Publish accepted musl amd64 scan
uses: github/codeql-action/upload-sarif@99df26d4f13ea111d4ec1a7dddef6063f76b97e9 # v4
with:
sarif_file: sealed/scan-musl-amd64.sarif
category: container-image-musl-amd64
- name: Publish accepted musl arm64 scan
uses: github/codeql-action/upload-sarif@99df26d4f13ea111d4ec1a7dddef6063f76b97e9 # v4
with:
sarif_file: sealed/scan-musl-arm64.sarif
category: container-image-musl-arm64
- name: Publish accepted GNU amd64 scan
uses: github/codeql-action/upload-sarif@99df26d4f13ea111d4ec1a7dddef6063f76b97e9 # v4
with:
sarif_file: sealed/scan-gnu-amd64.sarif
category: container-image-gnu-amd64
- name: Publish accepted GNU arm64 scan
uses: github/codeql-action/upload-sarif@99df26d4f13ea111d4ec1a7dddef6063f76b97e9 # v4
with:
sarif_file: sealed/scan-gnu-arm64.sarif
category: container-image-gnu-arm64
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: release-version-receipt-${{ github.run_attempt }}
path: publication.json
if-no-files-found: error
retention-days: 90
channels:
needs: [verify, version]
runs-on: ubuntu-latest
timeout-minutes: 90
concurrency:
group: rustfs-release-channels
cancel-in-progress: false
permissions:
contents: write
actions: read
packages: write
env:
GH_TOKEN: ${{ github.token }}
COMPLETE_RUN: ${{ inputs.complete_run_id }}
MANIFEST_SHA: ${{ inputs.manifest_sha256 }}
AWS_ACCESS_KEY_ID: ${{ secrets.R2_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.R2_SECRET_ACCESS_KEY }}
AWS_DEFAULT_REGION: auto
AWS_REQUEST_CHECKSUM_CALCULATION: when_required
AWS_RESPONSE_CHECKSUM_VALIDATION: when_required
AWS_EC2_METADATA_DISABLED: "true"
R2_ENDPOINT: ${{ secrets.R2_ENDPOINT }}
R2_BUCKET: ${{ secrets.R2_BUCKET }}
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
fetch-depth: 0
persist-credentials: false
- name: Install promotion tools
run: |
sudo apt-get update
sudo apt-get install -y skopeo
aws --version
- uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
with:
username: rustfs
password: ${{ secrets.DOCKERHUB_TOKEN }}
- uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
with:
registry: ghcr.io
username: ${{ secrets.GHCR_USERNAME }}
password: ${{ secrets.GHCR_PASSWORD }}
- uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
with:
registry: quay.io
username: ${{ secrets.QUAY_USERNAME }}
password: ${{ secrets.QUAY_PASSWORD }}
- name: Share registry credentials with the OCI copier
run: |
install -m 600 "$HOME/.docker/config.json" "$RUNNER_TEMP/registry-auth.json"
echo "REGISTRY_AUTH_FILE=$RUNNER_TEMP/registry-auth.json" >> "$GITHUB_ENV"
- uses: azure/setup-helm@b9e51907a09c216f16ebe8536097933489208112 # v4.3.0
- name: Install pinned OSS client
run: |
curl --fail --location --retry 3 --output "$RUNNER_TEMP/ossutil.zip" https://gosspublic.alicdn.com/ossutil/v2/2.1.1/ossutil-2.1.1-linux-amd64.zip
printf '%s %s\n' 60f3a808cbbdfd4b5269b8f967c6a66f564c9dacff52d93a5d21a588976ab5a2 "$RUNNER_TEMP/ossutil.zip" | sha256sum --check -
unzip -q "$RUNNER_TEMP/ossutil.zip" -d "$RUNNER_TEMP/ossutil-extract"
install -m 755 "$RUNNER_TEMP/ossutil-extract/ossutil-2.1.1-linux-amd64/ossutil" "$RUNNER_TEMP/ossutil"
echo "OSSUTIL_BIN=$RUNNER_TEMP/ossutil" >> "$GITHUB_ENV"
- name: Promote verified channels
env:
HELM_TOKEN: ${{ secrets.RUSTFS_HELM_PACKAGE }}
OSS_ACCESS_KEY_ID: ${{ secrets.ALICLOUDOSS_KEY_ID }}
OSS_ACCESS_KEY_SECRET: ${{ secrets.ALICLOUDOSS_KEY_SECRET }}
OSS_REGION: cn-beijing
OSS_ENDPOINT: https://oss-cn-beijing.aliyuncs.com
run: python3 scripts/release/publish.py channels --run-id "$COMPLETE_RUN" --digest "$MANIFEST_SHA" --directory sealed
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: release-channels-receipt-${{ github.run_attempt }}
path: publication.json
if-no-files-found: error
retention-days: 90
@@ -26,7 +26,13 @@ permissions:
jobs:
alert-on-incomplete-run:
name: Alert on incomplete scheduled run
if: github.repository == 'rustfs/rustfs'
# The registry still owns responsibility and cadence selection. Successful
# and PR runs can never produce an incomplete scheduled-run alert.
if: >-
github.repository == 'rustfs/rustfs' &&
github.event.workflow_run.conclusion != 'success' &&
(github.event.workflow_run.event == 'schedule' ||
github.event.workflow_run.event == 'workflow_run')
# Stay on GitHub-hosted runners: this monitor must keep running when the
# self-hosted sm-standard-* pool is down, which is exactly when the
# scheduled validations it watches stall.
Generated
+92 -101
View File
@@ -322,9 +322,9 @@ dependencies = [
[[package]]
name = "archive-trait"
version = "0.0.15"
version = "0.0.16"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "999bf88cc483664cab77700ca2fdfb0415a73412abe284ef5bc2319d00938e00"
checksum = "0b5040ca935c09888e7dfb04383b4d62fa3571e25bc66b6f345b591e97958524"
dependencies = [
"cap-std",
"thiserror 2.0.21",
@@ -811,9 +811,9 @@ dependencies = [
[[package]]
name = "async-nats"
version = "0.51.0"
version = "0.52.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a4f92759f418774b78c36ef07885849150ce950c67d9bcdeff8a9c253f57d9a1"
checksum = "015f0a7a3b0b29701b78c76a4d4a122624c97c71fa30363381664eb729d619c7"
dependencies = [
"aws-lc-rs",
"base64 0.22.1",
@@ -3906,9 +3906,9 @@ dependencies = [
[[package]]
name = "dial9"
version = "0.5.7"
version = "0.5.8"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2f3298a54fd82af243c9f122595762f5b0e48bd3d1ea8fe3f3eb5e95c130e881"
checksum = "c83757d61dc75b3e9d3579a3004c7fe0c42aff2f779ca9eeec7f0c8e1ae3fc10"
dependencies = [
"bon",
"dial9-core",
@@ -3990,9 +3990,9 @@ dependencies = [
[[package]]
name = "dial9-tokio-telemetry"
version = "0.5.7"
version = "0.5.8"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0428cba60fbcf37ab82666f3a7ef3b6d0671e90dfb6a352376813b471319f061"
checksum = "6f0eb05627c12c57e40510e63f9a3d5f628ab4b32fec9ccdb0ae98544c13f15e"
dependencies = [
"bon",
"bytes",
@@ -4136,7 +4136,7 @@ checksum = "d0881ea181b1df73ff77ffaaf9c7544ecc11e82fba9b5f27b262a3c73a332555"
[[package]]
name = "e2e_test"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"anyhow",
"async-compression",
@@ -4525,9 +4525,9 @@ dependencies = [
[[package]]
name = "flate2"
version = "1.1.10"
version = "1.1.11"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6e634e2e0ebac1ee034020da1ca582e17ffe4e0f5e985823721e168928136dcb"
checksum = "626d4928326b646e70f65ed36fc7cf9f206bdb68a0019ee7c6b4709da4e084c4"
dependencies = [
"crc32fast",
"miniz_oxide",
@@ -4594,7 +4594,7 @@ checksum = "94e7099f6313ecacbe1256e8ff9d617b75d1bcb16a6fddef94866d225a01a14a"
dependencies = [
"io-lifetimes 2.0.4",
"rustix",
"windows-sys 0.59.0",
"windows-sys 0.52.0",
]
[[package]]
@@ -5385,9 +5385,9 @@ dependencies = [
[[package]]
name = "hotpath"
version = "0.30.3"
version = "0.31.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d9632c1fd213e63c1b88d8f3e59543da4999b17f859cc1e1d67cf8f2207e8935"
checksum = "4039758c8bca06a34c475d17aaafc17ac1073ce2cecd8dba8c82e69edb300bbe"
dependencies = [
"arc-swap",
"async-channel",
@@ -5420,15 +5420,15 @@ dependencies = [
[[package]]
name = "hotpath-drain"
version = "0.30.3"
version = "0.31.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "738324f75aec847dea984f6806402f4f1bdfab437b4700fc8c3ebcad2dff6424"
checksum = "bbba0a3badfb9ebf95f22dc6cdb4afcd0ff07b91a85fa3196fd1060576ea3065"
[[package]]
name = "hotpath-macros"
version = "0.30.3"
version = "0.31.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5d92fdfec803932e5764ea93c201cf7c6c2d8cff1871da44f64f5094c9c3ef8a"
checksum = "7d980014875338771b00ac44f1c9e2ea1a10650aa68ce54226233bc767714361"
dependencies = [
"proc-macro2",
"quote",
@@ -6412,9 +6412,9 @@ dependencies = [
[[package]]
name = "lzma-rust2"
version = "0.24.0"
version = "0.25.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "490750560fa7851fbbf5d3ab1a8750f09350ed361ed8aed85b1e1e40587812ee"
checksum = "ef1d73c59a56771343712b22ea5615092c612c16ba291c0979e3105dfeec4890"
dependencies = [
"sha2 0.11.1",
]
@@ -9529,7 +9529,7 @@ dependencies = [
[[package]]
name = "rustfs"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"aes-gcm",
"anyhow",
@@ -9690,7 +9690,7 @@ dependencies = [
[[package]]
name = "rustfs-audit"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"const-str",
"futures",
@@ -9712,7 +9712,7 @@ dependencies = [
[[package]]
name = "rustfs-checksums"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"base64-simd",
"bytes",
@@ -9728,7 +9728,7 @@ dependencies = [
[[package]]
name = "rustfs-common"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"arc-swap",
"hotpath",
@@ -9743,7 +9743,7 @@ dependencies = [
[[package]]
name = "rustfs-concurrency"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"hotpath",
"insta",
@@ -9756,7 +9756,7 @@ dependencies = [
[[package]]
name = "rustfs-config"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"const-str",
"hotpath",
@@ -9766,7 +9766,7 @@ dependencies = [
[[package]]
name = "rustfs-connect-symbol-catalog"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"hex-simd",
"object 0.40.0",
@@ -9778,7 +9778,7 @@ dependencies = [
[[package]]
name = "rustfs-credentials"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"base64-simd",
"hmac 0.13.0",
@@ -9792,7 +9792,7 @@ dependencies = [
[[package]]
name = "rustfs-crypto"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"aes-gcm",
"argon2",
@@ -9813,7 +9813,7 @@ dependencies = [
[[package]]
name = "rustfs-data-usage"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"hotpath",
"rmp-serde",
@@ -9823,7 +9823,7 @@ dependencies = [
[[package]]
name = "rustfs-ecstore"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"arc-swap",
"async-channel",
@@ -9962,7 +9962,7 @@ dependencies = [
[[package]]
name = "rustfs-extension-schema"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"hotpath",
"serde",
@@ -9972,7 +9972,7 @@ dependencies = [
[[package]]
name = "rustfs-filemeta"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"arc-swap",
"base64-simd",
@@ -10043,7 +10043,7 @@ dependencies = [
[[package]]
name = "rustfs-heal"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"async-trait",
"base64-simd",
@@ -10084,7 +10084,7 @@ dependencies = [
[[package]]
name = "rustfs-heal-contracts"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"serde",
"serde_json",
@@ -10094,7 +10094,7 @@ dependencies = [
[[package]]
name = "rustfs-iam"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"arc-swap",
"async-trait",
@@ -10144,7 +10144,7 @@ dependencies = [
[[package]]
name = "rustfs-io-core"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"bytes",
"hotpath",
@@ -10157,7 +10157,7 @@ dependencies = [
[[package]]
name = "rustfs-io-metrics"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"criterion",
"hotpath",
@@ -10221,7 +10221,7 @@ dependencies = [
[[package]]
name = "rustfs-keystone"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"bytes",
"futures",
@@ -10247,7 +10247,7 @@ dependencies = [
[[package]]
name = "rustfs-kms"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"aes-gcm",
"anyhow",
@@ -10297,14 +10297,14 @@ dependencies = [
[[package]]
name = "rustfs-license"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"thiserror 2.0.21",
]
[[package]]
name = "rustfs-lifecycle"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"async-trait",
"hotpath",
@@ -10326,7 +10326,7 @@ dependencies = [
[[package]]
name = "rustfs-lock"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"async-trait",
"compact_str",
@@ -10350,7 +10350,7 @@ dependencies = [
[[package]]
name = "rustfs-log-analyzer"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"chrono",
"flate2",
@@ -10369,7 +10369,7 @@ dependencies = [
[[package]]
name = "rustfs-madmin"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"hotpath",
"http 1.5.0",
@@ -10388,25 +10388,25 @@ dependencies = [
[[package]]
name = "rustfs-mimalloc"
version = "0.7.3"
version = "0.7.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "216ecfc9341b5d6e2506d47ce57c9232ae1b8020bb10264ea05286fb234286e7"
checksum = "fc7914232a854336c62058c24a8362c553ac8e7adba2e2395ce9097db24d1510"
dependencies = [
"rustfs-mimalloc-sys",
]
[[package]]
name = "rustfs-mimalloc-sys"
version = "0.7.3"
version = "0.7.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "fb16ad68f96883407a75cdb88252b43dc2ea592428f0fc54ff7885756de4b814"
checksum = "8073624242781448512682f1931624ee67108b6b7873b22fb29c1a1df689ac0c"
dependencies = [
"cc",
]
[[package]]
name = "rustfs-notify"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"arc-swap",
"async-trait",
@@ -10441,7 +10441,7 @@ dependencies = [
[[package]]
name = "rustfs-object-capacity"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"criterion",
"futures",
@@ -10460,7 +10460,7 @@ dependencies = [
[[package]]
name = "rustfs-object-data-cache"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"bytes",
"criterion",
@@ -10477,7 +10477,7 @@ dependencies = [
[[package]]
name = "rustfs-obs"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"chrono",
"crossbeam-channel",
@@ -10540,7 +10540,7 @@ dependencies = [
[[package]]
name = "rustfs-policy"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"async-trait",
"base64-simd",
@@ -10571,7 +10571,7 @@ dependencies = [
[[package]]
name = "rustfs-protocols"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"async-compression",
"async-trait",
@@ -10637,7 +10637,7 @@ dependencies = [
[[package]]
name = "rustfs-protos"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"flatbuffers",
"hotpath",
@@ -10662,7 +10662,7 @@ dependencies = [
[[package]]
name = "rustfs-replication"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"byteorder",
"bytes",
@@ -10680,7 +10680,7 @@ dependencies = [
[[package]]
name = "rustfs-rio"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"aes-gcm",
"arc-swap",
@@ -10726,7 +10726,7 @@ dependencies = [
[[package]]
name = "rustfs-rio-v2"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"aes-gcm",
"bytes",
@@ -10749,7 +10749,7 @@ dependencies = [
[[package]]
name = "rustfs-s3-client"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"base64-simd",
"bytes",
@@ -10795,7 +10795,7 @@ dependencies = [
[[package]]
name = "rustfs-s3-ops"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"hotpath",
"rustfs-s3-types",
@@ -10803,7 +10803,7 @@ dependencies = [
[[package]]
name = "rustfs-s3-types"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"bytes",
"futures-core",
@@ -10822,7 +10822,7 @@ dependencies = [
[[package]]
name = "rustfs-s3select-api"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"arc-swap",
"async-compression",
@@ -10864,7 +10864,7 @@ dependencies = [
[[package]]
name = "rustfs-s3select-query"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"async-trait",
"bytes",
@@ -10885,7 +10885,7 @@ dependencies = [
[[package]]
name = "rustfs-scanner"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"async-trait",
"bytes",
@@ -10937,7 +10937,7 @@ dependencies = [
[[package]]
name = "rustfs-scanner-metrics"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"chrono",
"jiff",
@@ -10952,7 +10952,7 @@ dependencies = [
[[package]]
name = "rustfs-security-governance"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"hotpath",
"thiserror 2.0.21",
@@ -10960,7 +10960,7 @@ dependencies = [
[[package]]
name = "rustfs-signer"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"base64-simd",
"bytes",
@@ -10980,7 +10980,7 @@ dependencies = [
[[package]]
name = "rustfs-storage-api"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"async-trait",
"hotpath",
@@ -10995,7 +10995,7 @@ dependencies = [
[[package]]
name = "rustfs-targets"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"arc-swap",
"async-nats",
@@ -11048,7 +11048,7 @@ dependencies = [
[[package]]
name = "rustfs-test-utils"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"hotpath",
"rustfs-data-usage",
@@ -11064,7 +11064,7 @@ dependencies = [
[[package]]
name = "rustfs-tls-runtime"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"arc-swap",
"hotpath",
@@ -11103,7 +11103,7 @@ dependencies = [
[[package]]
name = "rustfs-trusted-proxies"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"async-trait",
"axum",
@@ -11140,7 +11140,7 @@ dependencies = [
[[package]]
name = "rustfs-utils"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"aws-lc-rs",
"base64-simd",
@@ -11183,7 +11183,7 @@ dependencies = [
[[package]]
name = "rustfs-zip"
version = "1.0.2"
version = "1.0.5"
dependencies = [
"async-compression",
"futures",
@@ -11773,9 +11773,9 @@ dependencies = [
[[package]]
name = "serde_spanned"
version = "1.1.2"
version = "1.1.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4e7523beb55eece201a2356bee0bbca0d1ab466c14c07703b2e0ee6d42cb0c2c"
checksum = "9c187e0a369471c550af028cba56d4f4fa2a489ad28151ae95a4d340a2a8b782"
dependencies = [
"serde_core",
]
@@ -12105,9 +12105,9 @@ dependencies = [
[[package]]
name = "smallvec"
version = "1.16.4"
version = "1.16.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d0d5d4b1981b8bbbb7aed204cae5726d683d5d6ac7e74ec54122ea28f315e7c8"
checksum = "149309498a5f7334b12ddae0cad8aba81ce785fe91755068867e017cf98fee46"
dependencies = [
"serde",
]
@@ -12302,9 +12302,9 @@ dependencies = [
[[package]]
name = "starshard"
version = "2.4.0"
version = "2.4.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "fc45de5060fab0de00d7a42230877543ae00c37caf6f49975e69851089b023ac"
checksum = "b0883700b50fd0b62120ebc4c07230e02f0cf1ead474c3b3b44eaa13e0a999ea"
dependencies = [
"async-trait",
"hashbrown 0.17.1",
@@ -12553,9 +12553,9 @@ dependencies = [
[[package]]
name = "tar-codec"
version = "0.0.15"
version = "0.0.16"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b9dad615e50d181afea4bafea23dacb8e33ed52dde06503813570634b932d467"
checksum = "6875ba507bbec6022500f46477159a03f07ebdffdb4ea043254974531df1dac2"
dependencies = [
"archive-trait",
"tar-framing",
@@ -12565,9 +12565,9 @@ dependencies = [
[[package]]
name = "tar-framing"
version = "0.0.15"
version = "0.0.16"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "563e8181b47c9cdcce569eae3ad431eec71543d6269dd60c7bd2350f70bcd233"
checksum = "49727b1a300514960d425d0e741a7da3e693170f3db39de3180fdbbc07e542c5"
dependencies = [
"thiserror 2.0.21",
"tokio",
@@ -12941,9 +12941,9 @@ dependencies = [
[[package]]
name = "toml"
version = "1.2.0+spec-1.1.0"
version = "1.2.1+spec-1.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c5ffcdf8f50436dbbfd675c1fe1b8e0854f42b7defea8584bdebe1aee900e561"
checksum = "d5755c51863cdac711f1956baf8eb9588e060755f7d3c4c979e70ea1e1f32cd7"
dependencies = [
"indexmap 2.14.2",
"serde_core",
@@ -12956,9 +12956,9 @@ dependencies = [
[[package]]
name = "toml_datetime"
version = "1.1.2+spec-1.1.0"
version = "1.1.3+spec-1.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2b86d767906c6c42421dcba507eb9d203e779497710a47782a224bb871653053"
checksum = "173f25c883cd2176a2d8212155638c0a60d484a9d84546d270866f40f56b6e49"
dependencies = [
"serde_core",
]
@@ -13851,15 +13851,6 @@ dependencies = [
"windows-targets 0.52.6",
]
[[package]]
name = "windows-sys"
version = "0.59.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1e38bc4d79ed67fd075bcc251a1c39b32a1776bbe92e5bef1f0bf1f8c531853b"
dependencies = [
"windows-targets 0.52.6",
]
[[package]]
name = "windows-sys"
version = "0.60.2"
@@ -14032,7 +14023,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3f3fd376f71958b862e7afb20cfe5a22830e1963462f3a17f49d82a6c1d1f42d"
dependencies = [
"bitflags 2.13.2",
"windows-sys 0.59.0",
"windows-sys 0.52.0",
]
[[package]]
@@ -14282,9 +14273,9 @@ dependencies = [
[[package]]
name = "zip"
version = "9.3.0"
version = "9.5.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "eaeb7f16d2605b49aef586996ded29e65642fab7259a3077e537351df6d80426"
checksum = "0700e6c427aff64acdbf02e85fbfd946c2ee12319d1af013092f3a2178914ff8"
dependencies = [
"aes 0.9.4",
"bzip2",
+55 -55
View File
@@ -75,7 +75,7 @@ edition = "2024"
license = "Apache-2.0"
repository = "https://github.com/rustfs/rustfs"
rust-version = "1.99.0"
version = "1.0.2"
version = "1.0.5"
homepage = "https://rustfs.com"
description = "RustFS is a high-performance distributed object storage software built using Rust, one of the most popular languages worldwide. "
keywords = ["RustFS", "Minio", "object-storage", "filesystem", "s3"]
@@ -92,56 +92,56 @@ redundant_clone = "warn"
[workspace.dependencies]
# RustFS Internal Crates
rustfs = { path = "./rustfs", version = "1.0.2" }
rustfs-heal = { path = "crates/heal", version = "1.0.2" }
rustfs-heal-contracts = { path = "crates/heal-contracts", version = "1.0.2" }
rustfs-scanner-metrics = { path = "crates/scanner-metrics", version = "1.0.2" }
rustfs-audit = { path = "crates/audit", version = "1.0.2" }
rustfs-checksums = { path = "crates/checksums", version = "1.0.2" }
rustfs-common = { path = "crates/common", version = "1.0.2" }
rustfs-data-usage = { path = "crates/data-usage", version = "1.0.2" }
rustfs-config = { path = "./crates/config", version = "1.0.2" }
rustfs-concurrency = { path = "./crates/concurrency", version = "1.0.2" }
rustfs-credentials = { path = "crates/credentials", version = "1.0.2" }
rustfs-crypto = { path = "crates/crypto", version = "1.0.2" }
rustfs-ecstore = { path = "crates/ecstore", version = "1.0.2" }
rustfs-filemeta = { path = "crates/filemeta", version = "1.0.2" }
rustfs-iam = { path = "crates/iam", version = "1.0.2" }
rustfs-keystone = { path = "crates/keystone", version = "1.0.2" }
rustfs-license = { path = "crates/license", version = "1.0.2" }
rustfs-lifecycle = { path = "crates/lifecycle", version = "1.0.2" }
rustfs-kms = { path = "crates/kms", version = "1.0.2" }
rustfs-lock = { path = "crates/lock", version = "1.0.2" }
rustfs-madmin = { path = "crates/madmin", version = "1.0.2" }
rustfs-notify = { path = "crates/notify", version = "1.0.2" }
rustfs-io-metrics = { path = "crates/io-metrics", version = "1.0.2" }
rustfs-io-core = { path = "crates/io-core", version = "1.0.2" }
rustfs-object-capacity = { path = "crates/object-capacity", version = "1.0.2" }
rustfs-object-data-cache = { path = "crates/object-data-cache", version = "1.0.2", default-features = false }
rustfs-log-analyzer = { path = "crates/log-analyzer", version = "1.0.2" }
rustfs-obs = { path = "crates/obs", version = "1.0.2" }
rustfs-policy = { path = "crates/policy", version = "1.0.2" }
rustfs-protos = { path = "crates/protos", version = "1.0.2" }
rustfs-protocols = { path = "crates/protocols", version = "1.0.2" }
rustfs-replication = { path = "crates/replication", version = "1.0.2" }
rustfs-rio = { path = "crates/rio", version = "1.0.2" }
rustfs-rio-v2 = { path = "crates/rio-v2", version = "1.0.2" }
rustfs-s3-client = { path = "crates/s3-client", version = "1.0.2" }
rustfs-s3-types = { path = "crates/s3-types", version = "1.0.2" }
rustfs-s3-ops = { path = "crates/s3-ops", version = "1.0.2" }
rustfs-s3select-api = { path = "crates/s3select-api", version = "1.0.2" }
rustfs-s3select-query = { path = "crates/s3select-query", version = "1.0.2" }
rustfs-scanner = { path = "crates/scanner", version = "1.0.2" }
rustfs-security-governance = { path = "crates/security-governance", version = "1.0.2" }
rustfs-extension-schema = { path = "crates/extension-schema", version = "1.0.2" }
rustfs-signer = { path = "crates/signer", version = "1.0.2" }
rustfs-storage-api = { path = "crates/storage-api", version = "1.0.2" }
rustfs-trusted-proxies = { path = "crates/trusted-proxies", version = "1.0.2" }
rustfs-targets = { path = "crates/targets", version = "1.0.2" }
rustfs-test-utils = { path = "crates/test-utils", version = "1.0.2" }
rustfs-tls-runtime = { path = "crates/tls-runtime", version = "1.0.2" }
rustfs-utils = { path = "crates/utils", version = "1.0.2" }
rustfs-zip = { path = "./crates/zip", version = "1.0.2" }
rustfs = { path = "./rustfs", version = "1.0.5" }
rustfs-heal = { path = "crates/heal", version = "1.0.5" }
rustfs-heal-contracts = { path = "crates/heal-contracts", version = "1.0.5" }
rustfs-scanner-metrics = { path = "crates/scanner-metrics", version = "1.0.5" }
rustfs-audit = { path = "crates/audit", version = "1.0.5" }
rustfs-checksums = { path = "crates/checksums", version = "1.0.5" }
rustfs-common = { path = "crates/common", version = "1.0.5" }
rustfs-data-usage = { path = "crates/data-usage", version = "1.0.5" }
rustfs-config = { path = "./crates/config", version = "1.0.5" }
rustfs-concurrency = { path = "./crates/concurrency", version = "1.0.5" }
rustfs-credentials = { path = "crates/credentials", version = "1.0.5" }
rustfs-crypto = { path = "crates/crypto", version = "1.0.5" }
rustfs-ecstore = { path = "crates/ecstore", version = "1.0.5" }
rustfs-filemeta = { path = "crates/filemeta", version = "1.0.5" }
rustfs-iam = { path = "crates/iam", version = "1.0.5" }
rustfs-keystone = { path = "crates/keystone", version = "1.0.5" }
rustfs-license = { path = "crates/license", version = "1.0.5" }
rustfs-lifecycle = { path = "crates/lifecycle", version = "1.0.5" }
rustfs-kms = { path = "crates/kms", version = "1.0.5" }
rustfs-lock = { path = "crates/lock", version = "1.0.5" }
rustfs-madmin = { path = "crates/madmin", version = "1.0.5" }
rustfs-notify = { path = "crates/notify", version = "1.0.5" }
rustfs-io-metrics = { path = "crates/io-metrics", version = "1.0.5" }
rustfs-io-core = { path = "crates/io-core", version = "1.0.5" }
rustfs-object-capacity = { path = "crates/object-capacity", version = "1.0.5" }
rustfs-object-data-cache = { path = "crates/object-data-cache", version = "1.0.5", default-features = false }
rustfs-log-analyzer = { path = "crates/log-analyzer", version = "1.0.5" }
rustfs-obs = { path = "crates/obs", version = "1.0.5" }
rustfs-policy = { path = "crates/policy", version = "1.0.5" }
rustfs-protos = { path = "crates/protos", version = "1.0.5" }
rustfs-protocols = { path = "crates/protocols", version = "1.0.5" }
rustfs-replication = { path = "crates/replication", version = "1.0.5" }
rustfs-rio = { path = "crates/rio", version = "1.0.5" }
rustfs-rio-v2 = { path = "crates/rio-v2", version = "1.0.5" }
rustfs-s3-client = { path = "crates/s3-client", version = "1.0.5" }
rustfs-s3-types = { path = "crates/s3-types", version = "1.0.5" }
rustfs-s3-ops = { path = "crates/s3-ops", version = "1.0.5" }
rustfs-s3select-api = { path = "crates/s3select-api", version = "1.0.5" }
rustfs-s3select-query = { path = "crates/s3select-query", version = "1.0.5" }
rustfs-scanner = { path = "crates/scanner", version = "1.0.5" }
rustfs-security-governance = { path = "crates/security-governance", version = "1.0.5" }
rustfs-extension-schema = { path = "crates/extension-schema", version = "1.0.5" }
rustfs-signer = { path = "crates/signer", version = "1.0.5" }
rustfs-storage-api = { path = "crates/storage-api", version = "1.0.5" }
rustfs-trusted-proxies = { path = "crates/trusted-proxies", version = "1.0.5" }
rustfs-targets = { path = "crates/targets", version = "1.0.5" }
rustfs-test-utils = { path = "crates/test-utils", version = "1.0.5" }
rustfs-tls-runtime = { path = "crates/tls-runtime", version = "1.0.5" }
rustfs-utils = { path = "crates/utils", version = "1.0.5" }
rustfs-zip = { path = "./crates/zip", version = "1.0.5" }
# The gateway's S3 types: the DTOs and persisted bucket-configuration codecs the
# engine crates move onto from s3s (rustfs/backlog#2745). The one pin of the
@@ -156,7 +156,7 @@ async_zip = { default-features = false, version = "0.0.19" }
mysql_async = { default-features = false, version = "0.37.1" }
async-compression = { version = "0.4.52" }
async-trait = "0.1.92"
async-nats = { version = "0.51.0", default-features = false }
async-nats = { version = "0.52.0", default-features = false }
axum = "0.8.9"
futures = "0.3.34"
futures-core = "0.3.34"
@@ -256,8 +256,8 @@ anyhow = "1.0.104"
arc-swap = "1.9.3"
rustfs-tokio-tar = { version = "0.7.1" }
# Candidate Snowball parser versions exercised by rustfs-zip compatibility fixtures.
tar-codec = "0.0.15"
tar-framing = "0.0.15"
tar-codec = "0.0.16"
tar-framing = "0.0.16"
atoi = "3.1.1"
atomic_enum = "0.3.0"
aws-config = { version = "1.13.0" }
@@ -401,7 +401,7 @@ dav-server = "0.11.0"
# Performance Analysis and Memory Profiling
rustfs-mimalloc = { version = "0.7.3" }
# Preserve Unicode focus filters until rustfs/backlog#2302 is resolved.
hotpath = { version = "0.30.3", default-features = false }
hotpath = { version = "0.31.1", default-features = false }
# Snapshot testing for output format regression detection
insta = { version = "1.49" }
+11 -2
View File
@@ -12,7 +12,9 @@
# See the License for the specific language governing permissions and
# limitations under the License.
FROM alpine:3.24.2 AS build
ARG BINARY_SOURCE=download
FROM alpine:3.24.2 AS download
ARG TARGETARCH
ARG RELEASE=latest
@@ -74,6 +76,14 @@ COPY scripts/download_rc.sh /usr/local/bin/download_rc.sh
RUN sh /usr/local/bin/download_rc.sh "$TARGETARCH" musl "$RC_RELEASE" /build/rc
# Candidate packaging supplies the accepted executables from verified archives.
# The default download stage remains available for local image builds.
FROM scratch AS supplied
ARG TARGETARCH
COPY .release-inputs/${TARGETARCH}/ /build/
FROM ${BINARY_SOURCE} AS build
FROM alpine:3.24.2
ARG RELEASE=latest
@@ -103,7 +113,6 @@ RUN apk upgrade --no-cache && \
tzdata \
&& test "$(TZ=Asia/Kolkata date +%z)" = "+0530"
COPY --from=build /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
COPY --from=build /build/rustfs /usr/bin/rustfs
COPY --from=build /build/rc /usr/bin/rc
COPY entrypoint.sh /etc/rustfs/entrypoint.sh
+11 -1
View File
@@ -12,7 +12,9 @@
# See the License for the specific language governing permissions and
# limitations under the License.
FROM ubuntu:26.04 AS build
ARG BINARY_SOURCE=download
FROM ubuntu:26.04 AS download
ARG TARGETARCH
ARG RELEASE=latest
@@ -80,6 +82,14 @@ RUN set -eux; \
COPY scripts/download_rc.sh /usr/local/bin/download_rc.sh
RUN sh /usr/local/bin/download_rc.sh "$TARGETARCH" gnu "$RC_RELEASE" /build/rc
# Candidate packaging supplies the accepted executables from verified archives.
# The default download stage remains available for local image builds.
FROM scratch AS supplied
ARG TARGETARCH
COPY .release-inputs/${TARGETARCH}/ /build/
FROM ${BINARY_SOURCE} AS build
FROM ubuntu:26.04
ARG RELEASE=latest
+3 -3
View File
@@ -145,7 +145,7 @@ chown -R 10001:10001 data logs
docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:latest
# Using specific version
docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.4
docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.5
```
If you use [podman](https://github.com/containers/podman) instead of docker, you can install the RustFS with the below command
@@ -224,7 +224,7 @@ For developers who want to build RustFS Docker images from source with multi-arc
./docker-buildx.sh --push
# Build specific version
./docker-buildx.sh --release 1.0.4 --push
./docker-buildx.sh --release 1.0.5 --push
# Build for custom registry
./docker-buildx.sh --registry your-registry.com --namespace yourname --push
@@ -242,7 +242,7 @@ You can also use Make targets for convenience:
```bash
make docker-buildx # Build locally
make docker-buildx-push # Build and push
make docker-buildx-version VERSION=1.0.4 # Build specific version
make docker-buildx-version VERSION=1.0.5 # Build specific version
make help-docker # Show all Docker-related commands
```
+3 -3
View File
@@ -138,7 +138,7 @@ chown -R 10001:10001 data logs
docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:latest
# 使用指定版本运行
docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.4
docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.5
```
如果您通过绑定挂载启用 TLS 证书目录,也请用同样方式准备该目录:
@@ -176,7 +176,7 @@ docker compose -f docker-compose-simple.yml up -d
./docker-buildx.sh --push
# 构建指定版本
./docker-buildx.sh --release 1.0.4 --push
./docker-buildx.sh --release 1.0.5 --push
# 构建并推送到自定义仓库
./docker-buildx.sh --registry your-registry.com --namespace yourname --push
@@ -193,7 +193,7 @@ docker compose -f docker-compose-simple.yml up -d
```bash
make docker-buildx # 本地构建
make docker-buildx-push # 构建并推送
make docker-buildx-version VERSION=1.0.4 # 构建指定版本
make docker-buildx-version VERSION=1.0.5 # 构建指定版本
make help-docker # 显示所有 Docker 相关命令
```
+100
View File
@@ -902,4 +902,104 @@ mod tests {
assert_eq!(raw.len(), 8);
assert_eq!(&raw[..], reference.digest().to_be_bytes().as_slice());
}
#[test]
fn checksums_streaming_chunks_and_header_encoding_match_oneshot_across_unaligned_boundaries() {
use crate::http::HttpChecksum;
let all = [
ChecksumAlgorithm::Crc32,
ChecksumAlgorithm::Crc32c,
ChecksumAlgorithm::Crc64Nvme,
ChecksumAlgorithm::Sha1,
ChecksumAlgorithm::Sha256,
ChecksumAlgorithm::Sha512,
ChecksumAlgorithm::Xxhash3,
ChecksumAlgorithm::Xxhash64,
ChecksumAlgorithm::Xxhash128,
];
for &payload_len in &[0usize, 1, 31, 33, 1024, 4096 + 37, 64 * 1024 + 19] {
let payload: Vec<u8> = (0..payload_len).map(|i| ((i * 19 + 5) % 253) as u8).collect();
for algorithm in all {
let mut one_shot = algorithm.into_impl();
one_shot.update(&payload);
let expected_raw = one_shot.finalize();
assert_eq!(expected_raw.len(), algorithm.raw_len());
let mut one_shot_hdr = algorithm.into_impl();
one_shot_hdr.update(&payload);
let expected_headers = one_shot_hdr.headers();
for &chunk_size in &[1usize, 31, 1021, 4096] {
if payload_len > 4096 && chunk_size == 1 {
continue;
}
let mut streaming: Box<dyn HttpChecksum> = algorithm.into_impl();
for chunk in payload.chunks(chunk_size) {
streaming.update(chunk);
}
let actual_headers = streaming.headers();
assert_eq!(
actual_headers.get(algorithm.http_header_name()),
expected_headers.get(algorithm.http_header_name()),
"{algorithm:?} streaming mismatch at payload_len={payload_len}, chunk_size={chunk_size}"
);
}
}
}
}
#[test]
#[ignore = "utility throughput microbenchmark; run via cargo test -p rustfs-checksums --lib --release -- --ignored --nocapture"]
fn microbench_checksums_streaming_chunks_and_header_encoding() {
use crate::http::HttpChecksum;
use std::hint::black_box;
use std::time::Instant;
let all = [
ChecksumAlgorithm::Crc32,
ChecksumAlgorithm::Crc32c,
ChecksumAlgorithm::Crc64Nvme,
ChecksumAlgorithm::Sha1,
ChecksumAlgorithm::Sha256,
ChecksumAlgorithm::Sha512,
ChecksumAlgorithm::Xxhash3,
ChecksumAlgorithm::Xxhash64,
ChecksumAlgorithm::Xxhash128,
];
let payload: Vec<u8> = (0..(256 * 1024)).map(|i| ((i * 19 + 5) % 253) as u8).collect();
for &chunk_size in &[4 * 1024usize, 64 * 1024, 256 * 1024] {
for algorithm in all {
let mut one_shot = algorithm.into_impl();
one_shot.update(&payload);
let expected_headers = one_shot.headers();
let iters = 8usize;
let t0 = Instant::now();
let mut last_headers = http::HeaderMap::new();
for _ in 0..iters {
let mut streaming: Box<dyn HttpChecksum> = algorithm.into_impl();
for chunk in payload.chunks(chunk_size) {
streaming.update(black_box(chunk));
}
last_headers = streaming.headers();
black_box(&last_headers);
}
let elapsed = t0.elapsed();
assert_eq!(
last_headers.get(algorithm.http_header_name()),
expected_headers.get(algorithm.http_header_name()),
"{algorithm:?} streaming chunk_size={chunk_size} header mismatch"
);
let total_mib = (payload.len() * iters) as f64 / (1024.0 * 1024.0);
let mibs = total_mib / elapsed.as_secs_f64().max(1e-9);
println!(
"[microbench][checksums] algo={algorithm:?} chunk_size={chunk_size}B throughput={mibs:.1} MiB/s (size={}B)",
algorithm.raw_len()
);
}
}
}
}
+1
View File
@@ -29,6 +29,7 @@ Registered in [`src/lib.rs`](src/lib.rs). Grouped by concern:
| **distributed 4×4** | [`src/distributed/`](src/distributed) | Nightly `e2e-distributed` lane: S3, object lock/WORM, versioning, bucket/site replication, quota, expand/decommission/rebalance, concurrency, chaos, 4-node upgrade of historical data and IAM AK/SK. Map: [`docs/testing/distributed-e2e.md`](../../docs/testing/distributed-e2e.md) |
| **chaos / reliability** | [`src/chaos.rs`](src/chaos.rs), `reliability_disk_fault_test`, `heal_erasure_disk_rebuild_test`, `server_startup_failfast_test` | Disk offline/replace/corrupt, EC rebuild, heal, fail-fast startup |
| **upgrade compatibility** | `upgrade_compatibility_test` | Pinned previous-release writes followed by current-build reads on the same data directory |
| **dual-stack differential** | `dual_stack_differential` | Two single-node servers answer one SDK script for all 73 S3 operations and every differing `(op, field, a, b)` is reported. Side A is the binary under test on `RUSTFS_S3_STACK=legacy`; side B runs `RUSTFS_E2E_STACK_B`. Set `RUSTFS_E2E_REFERENCE_BINARY` to a released `rustfs` binary to make side A that binary instead (started without `RUSTFS_S3_STACK`, which older releases do not know). A missing, non-file or non-executable path fails the run; it never skips or falls back to the same-binary mode. The report in `<target>/dual-stack/<run>.json` records `reference_side` and each side's `--version` output |
The external-tool `storage_metric_ownership_test` validates the OTLP/Collector/Prometheus path, including a rolling upgrade and node failures. See the [storage metrics guide](../../docs/operations/storage-metrics.md) for its required binaries and focused command.
+12
View File
@@ -774,6 +774,18 @@ impl RustFSTestEnvironment {
.await
}
/// Like [`Self::start_rustfs_server_from_binary`], but leaves other running RustFS processes
/// alone, so that two servers of different binaries can be alive at once.
pub async fn start_rustfs_server_from_binary_without_cleanup(
&mut self,
binary_path: &Path,
extra_args: Vec<&str>,
extra_env: &[(&str, &str)],
) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
self.start_rustfs_server_inner_with_binary(binary_path, extra_args, extra_env, false)
.await
}
/// Start RustFS server with basic configuration
pub async fn start_rustfs_server(&mut self, extra_args: Vec<&str>) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
self.start_rustfs_server_inner(extra_args, &[], true).await
@@ -232,6 +232,12 @@ pub(crate) struct Report {
pub run: String,
pub stack_a: String,
pub stack_b: String,
/// Which side is the pinned release binary: `"a"`, or `None` when both sides are the binary
/// under test.
pub reference_side: Option<&'static str>,
/// `--version` output of the binary each side actually ran.
pub version_a: String,
pub version_b: String,
pub region_a: Option<String>,
pub region_b: Option<String>,
pub compared_ops: usize,
@@ -480,10 +486,13 @@ mod tests {
fn the_report_lists_each_difference_as_op_field_a_b() {
let directory = std::env::temp_dir().join(format!("dual-stack-report-{}", uuid::Uuid::new_v4()));
let report = Report {
schema: 1,
schema: 2,
run: "unit".into(),
stack_a: "legacy".into(),
stack_b: "legacy".into(),
reference_side: Some("a"),
version_a: "rustfs 1.0.4".into(),
version_b: "rustfs 1.1.0".into(),
region_a: None,
region_b: Some("eu-west-1".into()),
compared_ops: 1,
@@ -498,6 +507,9 @@ mod tests {
let _ = std::fs::remove_dir_all(&directory);
assert_eq!(path.file_name().and_then(|name| name.to_str()), Some("unit.json"));
assert_eq!(json["compared_ops"], 1);
assert_eq!(json["reference_side"], "a");
assert_eq!(json["version_a"], "rustfs 1.0.4");
assert_eq!(json["version_b"], "rustfs 1.1.0");
assert_eq!(
json["differences"],
serde_json::json!([{
@@ -14,7 +14,10 @@
//! Dual-stack differential: one `rustfs` binary, two processes, one SDK script (rustfs/backlog#2740).
//!
//! Stack A always runs `RUSTFS_S3_STACK=legacy`. Stack B runs the value of `RUSTFS_E2E_STACK_B`
//! Stack A runs `RUSTFS_S3_STACK=legacy` of the binary under test, unless `RUSTFS_E2E_REFERENCE_BINARY`
//! names a released binary: then A is that binary, started without `RUSTFS_S3_STACK` (older
//! releases do not know it), and a bad path fails the run instead of falling back (see
//! `reference`). The report names which side is the reference and the `--version` each side ran. Stack B runs the value of `RUSTFS_E2E_STACK_B`
//! (default `legacy` until the gateway stack is assembled), and `RUSTFS_E2E_STACK_B_REGION`, when
//! set, starts B in another region so that a difference can be injected by hand. Both servers
//! answer the script in `scenarios`, which covers every operation of `impl S3 for FS`; `compare`
@@ -23,16 +26,20 @@
//! is process-global, so two stacks in one process would read each other's buckets.
mod compare;
mod reference;
mod scenarios;
use crate::common::{RustFSTestEnvironment, init_logging, workspace_root};
use crate::common::{RustFSTestEnvironment, init_logging, rustfs_binary_path, workspace_root};
use compare::{Difference, Observation, Report, compare, compared_cases, compared_ops, render};
use std::path::PathBuf;
use reference::{SideA, binary_version};
use std::path::{Path, PathBuf};
use std::time::{SystemTime, UNIX_EPOCH};
type BoxError = Box<dyn std::error::Error + Send + Sync>;
const STACK_A: &str = "legacy";
/// Label of side A in the report when it is the released reference binary.
const STACK_A_REFERENCE: &str = "reference";
const ENV_STACK_B: &str = "RUSTFS_E2E_STACK_B";
const ENV_STACK_B_REGION: &str = "RUSTFS_E2E_STACK_B_REGION";
/// Becomes `gateway` once rustfs/backlog#2734 T2.5 assembles the gateway stack.
@@ -52,22 +59,31 @@ const CONTROL_REGION_B: &str = "eu-west-1";
const MIN_NORMAL_SUCCESSES: usize = 68;
struct PairConfig {
side_a: SideA,
stack_b: String,
region_a: Option<String>,
region_b: Option<String>,
}
impl PairConfig {
fn from_env() -> Self {
Self {
fn from_env() -> Result<Self, BoxError> {
Ok(Self {
side_a: SideA::from_env()?,
stack_b: std::env::var(ENV_STACK_B).unwrap_or_else(|_| DEFAULT_STACK_B.to_owned()),
region_a: None,
region_b: std::env::var(ENV_STACK_B_REGION).ok(),
})
}
fn stack_a_label(&self) -> &'static str {
match self.side_a {
SideA::SameBinary => STACK_A,
SideA::Reference(_) => STACK_A_REFERENCE,
}
}
}
/// Starts one server on `stack`, overriding its region only when asked.
/// Starts one server of the binary under test on `stack`, overriding its region only when asked.
async fn start_stack(stack: &str, region: Option<&str>) -> Result<RustFSTestEnvironment, BoxError> {
let mut env = RustFSTestEnvironment::new().await?;
let mut vars = vec![("RUSTFS_S3_STACK", stack)];
@@ -78,12 +94,28 @@ async fn start_stack(stack: &str, region: Option<&str>) -> Result<RustFSTestEnvi
Ok(env)
}
/// Starts one server of a released binary. `RUSTFS_S3_STACK` is not passed: older releases do not
/// know it and have exactly one stack.
async fn start_reference(binary: &Path, region: Option<&str>) -> Result<RustFSTestEnvironment, BoxError> {
let mut env = RustFSTestEnvironment::new().await?;
let mut vars = Vec::new();
if let Some(region) = region {
vars.push(("RUSTFS_REGION", region));
}
env.start_rustfs_server_from_binary_without_cleanup(binary, vec![], &vars)
.await?;
Ok(env)
}
/// Starts stack A and stack B side by side, each with its own port and data directory.
async fn spawn_pair(config: &PairConfig) -> Result<(RustFSTestEnvironment, RustFSTestEnvironment), BoxError> {
tokio::try_join!(
start_stack(STACK_A, config.region_a.as_deref()),
start_stack(&config.stack_b, config.region_b.as_deref())
)
let side_a = async {
match &config.side_a {
SideA::SameBinary => start_stack(STACK_A, config.region_a.as_deref()).await,
SideA::Reference(binary) => start_reference(binary, config.region_a.as_deref()).await,
}
};
tokio::try_join!(side_a, start_stack(&config.stack_b, config.region_b.as_deref()))
}
fn report_directory() -> PathBuf {
@@ -104,16 +136,34 @@ struct Run {
/// Runs the script against both stacks concurrently and writes the report.
async fn run_differential(label: &str, config: &PairConfig) -> Run {
let binary_under_test = rustfs_binary_path();
let (version_a, version_b) = match &config.side_a {
SideA::SameBinary => {
let version = binary_version(&binary_under_test)
.await
.expect("capture the binary under test version");
(version.clone(), version)
}
SideA::Reference(binary) => (
binary_version(binary).await.expect("capture the reference binary version"),
binary_version(&binary_under_test)
.await
.expect("capture the binary under test version"),
),
};
let (server_a, server_b) = spawn_pair(config).await.expect("start stack A and stack B");
let (a, b) = tokio::join!(Box::pin(scenarios::run(&server_a)), Box::pin(scenarios::run(&server_b)));
let millis = SystemTime::now()
.duration_since(UNIX_EPOCH)
.map_or(0, |elapsed| elapsed.as_millis());
let report = Report {
schema: 1,
schema: 2,
run: format!("{label}-{millis}-{}", std::process::id()),
stack_a: STACK_A.to_owned(),
stack_a: config.stack_a_label().to_owned(),
stack_b: config.stack_b.clone(),
reference_side: matches!(config.side_a, SideA::Reference(_)).then_some("a"),
version_a,
version_b,
region_a: config.region_a.clone(),
region_b: config.region_b.clone(),
compared_ops: compared_ops(&a, &b).len(),
@@ -131,7 +181,7 @@ mod tests {
#[tokio::test]
async fn both_stacks_answer_every_operation_identically() {
init_logging();
let config = PairConfig::from_env();
let config = PairConfig::from_env().expect("RUSTFS_E2E_REFERENCE_BINARY must be valid when set");
let run = run_differential("self-comparison", &config).await;
let report = &run.report;
@@ -146,9 +196,11 @@ mod tests {
assert_eq!(report.compared_ops, scenarios::OPERATIONS.len(), "report: {}", run.path.display());
assert!(
report.differences.is_empty(),
"stack {} and stack {} differ on {} of {} cases (report: {}):\n{}",
"stack {} ({}) and stack {} ({}) differ on {} of {} cases (report: {}):\n{}",
report.stack_a,
report.version_a,
report.stack_b,
report.version_b,
report.differences.len(),
report.compared_cases,
run.path.display(),
@@ -160,7 +212,11 @@ mod tests {
.iter()
.filter(|observation| observation.case == "negative" && !(400..500).contains(&observation.status))
.collect();
assert!(not_4xx.is_empty(), "negative cases must fail with a 4xx on {STACK_A}: {not_4xx:#?}");
assert!(
not_4xx.is_empty(),
"negative cases must fail with a 4xx on side A ({}): {not_4xx:#?}",
report.version_a
);
let successes = run
.a
.iter()
@@ -168,7 +224,8 @@ mod tests {
.count();
assert!(
successes >= MIN_NORMAL_SUCCESSES,
"only {successes} normal cases succeeded on {STACK_A}: {:#?}",
"only {successes} normal cases succeeded on side A ({}): {:#?}",
report.version_a,
run.a
);
}
@@ -177,6 +234,7 @@ mod tests {
async fn differential_detects_an_injected_difference() {
init_logging();
let config = PairConfig {
side_a: SideA::SameBinary,
stack_b: STACK_A.to_owned(),
region_a: Some(CONTROL_REGION_A.to_owned()),
region_b: Some(CONTROL_REGION_B.to_owned()),
@@ -0,0 +1,163 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! Which binary side A of the differential runs, and what it reports about itself.
//!
//! `RUSTFS_E2E_REFERENCE_BINARY` names a released `rustfs` binary to use as the reference. Unset,
//! side A is the binary under test on `RUSTFS_S3_STACK=legacy`. Set, it must resolve to an
//! executable file or the run fails: a reference that was asked for and silently replaced by the
//! binary under test would report an agreement with itself as agreement with the release.
use std::ffi::OsString;
use std::path::{Path, PathBuf};
type BoxError = Box<dyn std::error::Error + Send + Sync>;
pub(super) const ENV_REFERENCE_BINARY: &str = "RUSTFS_E2E_REFERENCE_BINARY";
#[derive(Clone, Debug, PartialEq, Eq)]
pub(super) enum SideA {
/// Both sides are the binary under test; A runs `RUSTFS_S3_STACK=legacy`.
SameBinary,
/// A is this released binary, started without `RUSTFS_S3_STACK`.
Reference(PathBuf),
}
impl SideA {
pub(super) fn from_env() -> Result<Self, BoxError> {
Self::resolve(std::env::var_os(ENV_REFERENCE_BINARY))
}
/// `None` is the only value that selects same-binary mode; every other value must validate.
pub(super) fn resolve(value: Option<OsString>) -> Result<Self, BoxError> {
let Some(value) = value else {
return Ok(Self::SameBinary);
};
let path = PathBuf::from(value);
validate_executable(&path)?;
Ok(Self::Reference(path))
}
}
fn validate_executable(path: &Path) -> Result<(), BoxError> {
if path.as_os_str().is_empty() {
return Err(format!("{ENV_REFERENCE_BINARY} is set but empty").into());
}
let metadata = std::fs::metadata(path)
.map_err(|error| format!("{ENV_REFERENCE_BINARY} points to {}, which cannot be read: {error}", path.display()))?;
if !metadata.is_file() {
return Err(format!("{ENV_REFERENCE_BINARY} points to {}, which is not a file", path.display()).into());
}
if !is_executable(&metadata) {
return Err(format!("{ENV_REFERENCE_BINARY} points to {}, which is not executable", path.display()).into());
}
Ok(())
}
#[cfg(unix)]
fn is_executable(metadata: &std::fs::Metadata) -> bool {
use std::os::unix::fs::PermissionsExt;
metadata.permissions().mode() & 0o111 != 0
}
#[cfg(not(unix))]
fn is_executable(_metadata: &std::fs::Metadata) -> bool {
true
}
/// What `<binary> --version` printed, trimmed. Fails when the binary does not run, exits
/// non-zero, or prints nothing, because an empty version would report nothing as observed.
pub(super) async fn binary_version(binary: &Path) -> Result<String, BoxError> {
let output = tokio::process::Command::new(binary).arg("--version").output().await?;
if !output.status.success() {
return Err(format!("{} --version failed with {}", binary.display(), output.status).into());
}
let version = String::from_utf8_lossy(&output.stdout).trim().to_owned();
if version.is_empty() {
return Err(format!("{} --version printed nothing", binary.display()).into());
}
Ok(version)
}
#[cfg(all(test, unix))]
mod tests {
use super::*;
use std::os::unix::fs::PermissionsExt;
fn script(dir: &Path, name: &str, body: &str, mode: u32) -> PathBuf {
let path = dir.join(name);
std::fs::write(&path, body).expect("write the stand-in");
std::fs::set_permissions(&path, std::fs::Permissions::from_mode(mode)).expect("chmod");
path
}
#[test]
fn missing_path_is_an_error_not_a_fallback() {
let dir = tempfile::tempdir().expect("tempdir");
let error = SideA::resolve(Some(dir.path().join("absent").into_os_string())).expect_err("must fail closed");
assert!(error.to_string().contains("cannot be read"), "{error}");
}
#[test]
fn non_executable_file_is_an_error() {
let dir = tempfile::tempdir().expect("tempdir");
let path = script(dir.path(), "rustfs", "#!/bin/sh\necho v\n", 0o644);
let error = SideA::resolve(Some(path.into_os_string())).expect_err("must fail closed");
assert!(error.to_string().contains("not executable"), "{error}");
}
#[test]
fn directory_is_an_error() {
let dir = tempfile::tempdir().expect("tempdir");
let error = SideA::resolve(Some(dir.path().as_os_str().to_owned())).expect_err("must fail closed");
assert!(error.to_string().contains("not a file"), "{error}");
}
#[test]
fn empty_value_is_an_error() {
let error = SideA::resolve(Some(OsString::new())).expect_err("must fail closed");
assert!(error.to_string().contains("empty"), "{error}");
}
#[test]
fn unset_selects_same_binary_mode() {
assert_eq!(SideA::resolve(None).expect("unset is valid"), SideA::SameBinary);
}
#[test]
fn executable_file_selects_reference_mode() {
let dir = tempfile::tempdir().expect("tempdir");
let path = script(dir.path(), "rustfs", "#!/bin/sh\necho v\n", 0o755);
assert_eq!(
SideA::resolve(Some(path.clone().into_os_string())).expect("valid"),
SideA::Reference(path)
);
}
#[tokio::test]
async fn version_output_is_captured_from_the_binary() {
let dir = tempfile::tempdir().expect("tempdir");
let path = script(dir.path(), "rustfs", "#!/bin/sh\necho 'rustfs 1.0.4 (abc123)'\n", 0o755);
assert_eq!(binary_version(&path).await.expect("version"), "rustfs 1.0.4 (abc123)");
}
#[tokio::test]
async fn failing_or_silent_version_is_an_error() {
let dir = tempfile::tempdir().expect("tempdir");
let failing = script(dir.path(), "failing", "#!/bin/sh\nexit 3\n", 0o755);
assert!(binary_version(&failing).await.is_err());
let silent = script(dir.path(), "silent", "#!/bin/sh\nexit 0\n", 0o755);
assert!(binary_version(&silent).await.is_err());
}
}
+255 -2
View File
@@ -21,12 +21,15 @@
#[cfg(test)]
mod tests {
use crate::common::{RustFSTestEnvironment, init_logging};
use crate::common::{RustFSTestEnvironment, admin_create_user, init_logging};
use aws_sdk_s3::Client;
use aws_sdk_s3::primitives::ByteStream;
use aws_sdk_s3::types::{BucketVersioningStatus, Delete, ObjectIdentifier, VersioningConfiguration};
use std::error::Error;
use tracing::info;
type TestResult = Result<(), Box<dyn Error + Send + Sync>>;
async fn create_bucket(client: &Client, bucket: &str) -> Result<(), Box<dyn Error + Send + Sync>> {
client.create_bucket().bucket(bucket).send().await?;
Ok(())
@@ -90,7 +93,7 @@ mod tests {
Ok(())
}
/// Duplicate and repeated slashes after a leading slash collapse MinIO-style.
/// Duplicate and repeated slashes after a leading slash collapse MinIO-style across single and bulk operations.
#[tokio::test]
async fn test_put_object_with_duplicate_slashes_normalized() -> Result<(), Box<dyn Error + Send + Sync>> {
init_logging();
@@ -144,8 +147,258 @@ mod tests {
"GET after DELETE with raw key must return HTTP 404, got {error:?}"
);
assert_bulk_delete_normalizes_keys(&client).await?;
assert_bulk_delete_authorizes_normalized_keys(&env, &client).await?;
assert_bulk_delete_preserves_version_identity(&client).await?;
env.stop_server();
info!("Test completed successfully");
Ok(())
}
async fn assert_bulk_delete_normalizes_keys(client: &Client) -> TestResult {
let bucket = "test-bulk-leading-slash";
create_bucket(client, bucket).await?;
let cases = [
("/foo/bar", "foo/bar"),
("//keyname", "keyname"),
("/dir///sub//file", "dir/sub/file"),
("/folder///", "folder/"),
("/literal%2F+名", "literal%2F+名"),
("plain", "plain"),
];
let invalid = ["///", "/dir/../keep", "dir//keep"];
client
.put_object()
.bucket(bucket)
.key("keep")
.body(ByteStream::from_static(b"keep"))
.send()
.await?;
client
.put_object()
.bucket(bucket)
.key("dir/keep")
.body(ByteStream::from_static(b"keep"))
.send()
.await?;
for quiet in [false, true] {
for (raw, _) in cases {
client
.put_object()
.bucket(bucket)
.key(raw)
.body(ByteStream::from_static(b""))
.send()
.await?;
}
let listing = client.list_objects_v2().bucket(bucket).send().await?;
let stored: Vec<_> = listing.contents().iter().filter_map(|object| object.key()).collect();
let mut expected: Vec<_> = cases
.iter()
.map(|(_, normalized)| *normalized)
.chain(["dir/keep", "keep"])
.collect();
expected.sort_unstable();
assert_eq!(stored, expected, "PUT must store the exact normalized targets before bulk deletion");
let objects = cases
.iter()
.map(|(raw, _)| *raw)
.chain(invalid)
.map(|key| ObjectIdentifier::builder().key(key).build())
.collect::<Result<Vec<_>, _>>()?;
let delete = Delete::builder().set_objects(Some(objects)).quiet(quiet).build()?;
// A second identical batch also pins idempotent deletion of absent normalized keys.
for _ in 0..2 {
let output = client.delete_objects().bucket(bucket).delete(delete.clone()).send().await?;
let errors: Vec<_> = output.errors().iter().map(|error| (error.key(), error.code())).collect();
assert_eq!(
errors,
invalid.map(|key| (Some(key), Some("InvalidArgument"))),
"unexpected per-key errors: {output:?}"
);
let deleted: Vec<_> = output.deleted().iter().filter_map(|object| object.key()).collect();
let expected = if quiet {
vec![]
} else {
cases.iter().map(|(raw, _)| *raw).collect()
};
assert_eq!(deleted, expected, "responses must retain request keys, including directory slashes");
}
let listing = client.list_objects_v2().bucket(bucket).send().await?;
let remaining: Vec<_> = listing.contents().iter().filter_map(|object| object.key()).collect();
assert_eq!(
remaining,
["dir/keep", "keep"],
"bulk delete must remove normalized objects without resolving dot segments"
);
}
Ok(())
}
async fn assert_bulk_delete_authorizes_normalized_keys(env: &RustFSTestEnvironment, admin: &Client) -> TestResult {
let bucket = "test-bulk-slash-policy";
let user = "bulk-slash-user";
admin_create_user(env, user, "bulkSlashTestSecret123!").await?;
create_bucket(admin, bucket).await?;
for key in ["allowed/item", "protected/item"] {
admin
.put_object()
.bucket(bucket)
.key(key)
.body(ByteStream::from_static(b"keep"))
.send()
.await?;
}
let policy = serde_json::json!({
"Version": "2012-10-17",
"Statement": [
{"Effect": "Allow", "Principal": {"AWS": [user]}, "Action": ["s3:DeleteObject"],
"Resource": [format!("arn:aws:s3:::{bucket}/*")]},
{"Effect": "Deny", "Principal": {"AWS": [user]}, "Action": ["s3:DeleteObject"],
"Resource": [format!("arn:aws:s3:::{bucket}/protected/*")]}
]
});
admin
.put_bucket_policy()
.bucket(bucket)
.policy(policy.to_string())
.send()
.await?;
let client = env.create_s3_client_with_credentials(user, "bulkSlashTestSecret123!");
let output = client
.delete_objects()
.bucket(bucket)
.delete(
Delete::builder()
.objects(ObjectIdentifier::builder().key("/allowed///item").build()?)
.objects(ObjectIdentifier::builder().key("/protected///item").build()?)
.build()?,
)
.send()
.await?;
assert_eq!(output.deleted().len(), 1, "allowed normalized key must be deleted: {output:?}");
assert_eq!(output.deleted()[0].key(), Some("/allowed///item"));
assert_eq!(output.errors().len(), 1, "normalized resource denial must apply: {output:?}");
assert_eq!(output.errors()[0].key(), Some("/protected///item"));
assert_eq!(output.errors()[0].code(), Some("AccessDenied"));
let listing = admin.list_objects_v2().bucket(bucket).send().await?;
let remaining: Vec<_> = listing.contents().iter().filter_map(|object| object.key()).collect();
assert_eq!(remaining, ["protected/item"], "only the authorized object may be removed");
let protected = admin.get_object().bucket(bucket).key("protected/item").send().await?;
assert_eq!(protected.body.collect().await?.into_bytes().as_ref(), b"keep");
Ok(())
}
async fn assert_bulk_delete_preserves_version_identity(client: &Client) -> TestResult {
let bucket = "test-bulk-slash-versions";
let raw = "/history///file";
let normalized = "history/file";
create_bucket(client, bucket).await?;
client
.put_bucket_versioning()
.bucket(bucket)
.versioning_configuration(
VersioningConfiguration::builder()
.status(BucketVersioningStatus::Enabled)
.build(),
)
.send()
.await?;
let first = client
.put_object()
.bucket(bucket)
.key(raw)
.body(ByteStream::from_static(b"old"))
.send()
.await?;
let first_id = first.version_id().ok_or("first PUT omitted version ID")?;
let latest = client
.put_object()
.bucket(bucket)
.key(raw)
.body(ByteStream::from_static(b"latest"))
.send()
.await?;
let latest_id = latest.version_id().ok_or("latest PUT omitted version ID")?;
let invalid = client
.delete_objects()
.bucket(bucket)
.delete(
Delete::builder()
.objects(ObjectIdentifier::builder().key("///").build()?)
.build()?,
)
.send()
.await?;
assert!(invalid.deleted().is_empty(), "slash-only keys must not delete a versioned object");
assert_eq!(
invalid.errors().len(),
1,
"slash-only keys must remain invalid in versioned buckets: {invalid:?}"
);
assert_eq!(invalid.errors()[0].key(), Some("///"));
assert_eq!(invalid.errors()[0].code(), Some("InvalidArgument"));
let deleted = client
.delete_objects()
.bucket(bucket)
.delete(
Delete::builder()
.objects(ObjectIdentifier::builder().key(raw).version_id(first_id).build()?)
.build()?,
)
.send()
.await?;
assert!(deleted.errors().is_empty(), "historical delete failed: {deleted:?}");
assert_eq!(deleted.deleted().len(), 1);
assert_eq!(deleted.deleted()[0].key(), Some(raw));
assert_eq!(deleted.deleted()[0].version_id(), Some(first_id));
let listing = client.list_object_versions().bucket(bucket).send().await?;
assert_eq!(listing.versions().len(), 1, "only the latest data version should remain: {listing:?}");
assert_eq!(listing.versions()[0].version_id(), Some(latest_id));
let deleted = client
.delete_objects()
.bucket(bucket)
.delete(
Delete::builder()
.objects(ObjectIdentifier::builder().key(raw).build()?)
.build()?,
)
.send()
.await?;
assert!(deleted.errors().is_empty(), "delete marker creation failed: {deleted:?}");
assert_eq!(deleted.deleted().len(), 1);
assert_eq!(deleted.deleted()[0].key(), Some(raw));
assert_eq!(deleted.deleted()[0].delete_marker(), Some(true));
let marker = deleted.deleted()[0]
.delete_marker_version_id()
.ok_or("missing delete marker ID")?;
let listing = client.list_object_versions().bucket(bucket).send().await?;
assert_eq!(listing.delete_markers().len(), 1);
assert_eq!(listing.delete_markers()[0].key(), Some(normalized));
assert_eq!(listing.delete_markers()[0].version_id(), Some(marker));
let deleted = client
.delete_objects()
.bucket(bucket)
.delete(
Delete::builder()
.objects(ObjectIdentifier::builder().key(raw).version_id(marker).build()?)
.build()?,
)
.send()
.await?;
assert!(deleted.errors().is_empty(), "explicit marker removal failed: {deleted:?}");
assert_eq!(deleted.deleted().len(), 1);
assert_eq!(deleted.deleted()[0].key(), Some(raw));
assert_eq!(deleted.deleted()[0].version_id(), Some(marker));
let listing = client.list_object_versions().bucket(bucket).send().await?;
assert!(listing.delete_markers().is_empty());
let restored = client.get_object().bucket(bucket).key(normalized).send().await?;
assert_eq!(restored.version_id(), Some(latest_id));
assert_eq!(restored.body.collect().await?.into_bytes().as_ref(), b"latest");
Ok(())
}
}
+10
View File
@@ -17701,6 +17701,16 @@ impl ECStore {
.map(|_| ())
}
#[cfg(feature = "test-util")]
pub async fn complete_scanner_pause_backlog_retirement_for_test(&self, source_pool_index: usize) -> Result<()> {
// Native backlog fixtures use bounded capacity inputs rather than the
// host filesystem's unrelated usage. Keep the real ILM completion gate.
let run_token = self.durable_ilm_receipt_run_token(source_pool_index).await?;
self.persist_decommission_durable_ilm_manifest_for_run(source_pool_index, &run_token)
.await?;
self.complete_decommission(source_pool_index).await
}
#[cfg(feature = "test-util")]
pub async fn retire_scanner_pause_backlog_for_test(
self: &Arc<Self>,
@@ -85,6 +85,7 @@ impl ScannerPauseBacklogRetirementPhase {
pub struct ScannerPauseBacklogRetirementReplica {
pub pool_index: usize,
pub set_index: usize,
pub set_id: Option<uuid::Uuid>,
/// False for retiring members retained only to verify the old commit proof.
pub writable: bool,
pub data: Option<Vec<u8>>,
@@ -140,6 +141,7 @@ async fn read_replica((set, writable): (Arc<SetDisks>, bool)) -> Result<ScannerP
let mut replica = ScannerPauseBacklogRetirementReplica {
pool_index: set.pool_index,
set_index: set.set_index,
set_id: Some(set.persistent_set_id()?),
writable,
data: None,
};
@@ -222,6 +224,7 @@ pub(crate) fn plan_scanner_pause_backlog_retirement(
.map(|read| ScannerPauseBacklogRetirementReplica {
pool_index: read.replica.pool_index,
set_index: read.replica.set_index,
set_id: read.replica.set_id,
writable: read.replica.writable,
data: read.replica.data.clone(),
})
+60 -30
View File
@@ -685,12 +685,8 @@ async fn clear_bucket_usage_memory(bucket: &str, guard: Option<&rustfs_lock::Nam
}
pub async fn remove_bucket_usage_from_backend(store: Arc<ECStore>, bucket: &str) -> Result<(), Error> {
remove_bucket_usage_for_namespace_change(store.as_ref(), bucket).await
}
pub(crate) async fn remove_bucket_usage_for_namespace_change(store: &ECStore, bucket: &str) -> Result<(), Error> {
prepare_bucket_usage_for_namespace_change(bucket, None).await?;
remove_bucket_usage_from_backend_with_guard_fenced(store, bucket, None).await
remove_bucket_usage_with_owned_namespace(store, bucket, None).await
}
pub(crate) async fn prepare_bucket_usage_for_namespace_change(
@@ -730,11 +726,37 @@ where
pub(crate) async fn remove_bucket_usage_from_backend_with_guard_fenced(
store: &ECStore,
bucket: &str,
guard: Option<&rustfs_lock::NamespaceLockGuard>,
guard: Option<&Arc<rustfs_lock::NamespaceLockGuard>>,
) -> Result<(), Error> {
let result = remove_bucket_usage_from_backend_with_store_and_guard_and_publication(store, bucket, guard, Some(store)).await;
invalidate_bucket_usage_snapshot_caches(guard, bucket).await?;
result
let owned_store = crate::bucket::metadata_sys::object_store_in(&store.ctx).await?;
if !std::ptr::eq(store, owned_store.as_ref()) {
return Err(Error::other("usage cleanup requires its own metadata store"));
}
remove_bucket_usage_with_owned_namespace(owned_store, bucket, guard.cloned()).await
}
async fn remove_bucket_usage_with_owned_namespace(
store: Arc<ECStore>,
bucket: &str,
guard: Option<Arc<rustfs_lock::NamespaceLockGuard>>,
) -> Result<(), Error> {
let bucket = bucket.to_owned();
let ctx = Arc::clone(&store.ctx);
// Retain the bucket generation and movement/object guards through every
// physical commit even when the namespace request stops waiting.
ctx.run_detached_mutation(async move {
let result = remove_bucket_usage_from_backend_with_store_and_guard_and_publication(
store.as_ref(),
&bucket,
guard.as_deref(),
Some(store.as_ref()),
)
.await;
invalidate_bucket_usage_snapshot_caches(guard.as_deref(), &bucket).await?;
result
})
.await
.map_err(Error::from)?
}
async fn invalidate_bucket_usage_snapshot_caches(
@@ -849,9 +871,9 @@ where
let primary_seed_epoch = match publication_store {
Some(publication_store) => Some(
publication_store
.scanner_data_usage_publication_epoch()
.data_usage_namespace_cleanup_epoch()
.await
.ok_or_else(|| Error::other("data usage publication is blocked by data movement"))?,
.ok_or_else(|| Error::other("data usage namespace cleanup epoch is unavailable"))?,
),
None => None,
};
@@ -871,9 +893,9 @@ where
let backup_seed_epoch = match publication_store {
Some(publication_store) => Some(
publication_store
.scanner_data_usage_publication_epoch()
.data_usage_namespace_cleanup_epoch()
.await
.ok_or_else(|| Error::other("data usage publication is blocked by data movement"))?,
.ok_or_else(|| Error::other("data usage namespace cleanup epoch is unavailable"))?,
),
None => None,
};
@@ -985,9 +1007,9 @@ where
let read_epoch = match publication {
Some((publication_store, expected_publication_epoch)) => {
let epoch = publication_store
.scanner_data_usage_publication_epoch()
.data_usage_namespace_cleanup_epoch()
.await
.ok_or_else(|| Error::other("data usage publication is blocked by data movement"))?;
.ok_or_else(|| Error::other("data usage namespace cleanup epoch is unavailable"))?;
if expected_publication_epoch.is_some_and(|expected| expected != epoch) {
return Err(Error::other("data usage publication epoch changed before snapshot read"));
}
@@ -1022,9 +1044,8 @@ where
ensure_bucket_namespace_guard(guard, bucket, "data usage snapshot commit")?;
let publication_guard = match publication {
Some((publication_store, expected_publication_epoch)) => {
let Some((guard, publication_epoch)) = publication_store.scanner_data_usage_publication_admission_guard().await
else {
return Err(Error::other("data usage publication is blocked by data movement"));
let Some((guard, publication_epoch)) = publication_store.data_usage_namespace_cleanup_guard().await else {
return Err(Error::other("data usage namespace cleanup epoch is unavailable"));
};
if expected_publication_epoch
.or(read_epoch)
@@ -1036,18 +1057,27 @@ where
}
None => None,
};
let save_result = store
.put_object(
RUSTFS_META_BUCKET,
object,
&mut put_data,
&ObjectOptions {
max_parity: true,
http_preconditions: Some(http_preconditions),
..Default::default()
},
)
.await;
let save_result = match publication {
Some((publication_store, _)) => {
publication_store
.put_usage_snapshot_for_namespace_cleanup(object, revision.as_deref(), &mut put_data, guard)
.await
}
None => {
store
.put_object(
RUSTFS_META_BUCKET,
object,
&mut put_data,
&ObjectOptions {
max_parity: true,
http_preconditions: Some(http_preconditions),
..Default::default()
},
)
.await
}
};
drop(publication_guard);
match save_result.map_err(|err| crate::config::com::map_system_metadata_write_error(err, object)) {
Ok(_) => return Ok(()),
+170 -72
View File
@@ -579,6 +579,7 @@ struct PreparedPartTxnReceipt {
dst_meta_path: PathBuf,
transaction_path: PathBuf,
meta: Bytes,
had_old_data: bool,
had_old_meta: bool,
}
@@ -586,6 +587,7 @@ struct PublishedPartTxnReceipt {
current_data_path: PathBuf,
current_meta_path: PathBuf,
transaction_path: PathBuf,
had_old_data: bool,
had_old_meta: bool,
}
@@ -4323,20 +4325,28 @@ fn take_presynced_tmp_file(file: &Path) -> bool {
PRESYNCED_TMP_FILES.remove(file).is_some()
}
fn open_lazy_local_file_std(file_path: &Path, volume_dir: &Path, _reclaim_on_shutdown: bool) -> std::io::Result<std::fs::File> {
if let Some(parent) = file_path.parent()
&& (volume_dir.as_os_str().is_empty() || !volume_dir.starts_with(parent))
&& let Err(err) = super::fs::create_dir_all_std(parent)
&& err.kind() != std::io::ErrorKind::AlreadyExists
{
return Err(to_file_error(err));
}
let file = std::fs::OpenOptions::new()
.write(true)
.create(true)
.truncate(true)
.open(file_path)
.map_err(to_file_error)?;
fn open_lazy_local_file_std(
file_path: &Path,
volume_dir: &Path,
_reclaim_on_shutdown: bool,
parent_exists: bool,
) -> std::io::Result<std::fs::File> {
let mut open_opts = std::fs::OpenOptions::new();
open_opts.write(true).create(true).truncate(true);
let file = match parent_exists.then(|| open_opts.open(file_path)) {
Some(Ok(file)) => file,
Some(Err(err)) if err.kind() != std::io::ErrorKind::NotFound => return Err(to_file_error(err)),
_ => {
if let Some(parent) = file_path.parent()
&& (volume_dir.as_os_str().is_empty() || !volume_dir.starts_with(parent))
&& let Err(err) = super::fs::create_dir_all_std(parent)
&& err.kind() != std::io::ErrorKind::AlreadyExists
{
return Err(to_file_error(err));
}
open_opts.open(file_path).map_err(to_file_error)?
}
};
#[cfg(target_os = "macos")]
if _reclaim_on_shutdown {
let _ = set_std_fd_nocache(&file);
@@ -4351,6 +4361,7 @@ struct LazyLocalFileWriter {
reclaim_on_shutdown: bool,
sync_on_close: bool,
mark_dir_presynced: bool,
parent_exists: bool,
bytes_written: usize,
state: LazyLocalFileWriterState,
}
@@ -4371,6 +4382,7 @@ impl LazyLocalFileWriter {
reclaim_on_shutdown: bool,
sync_on_close: bool,
mark_dir_presynced: bool,
parent_exists: bool,
) -> Self {
Self {
file_path,
@@ -4379,6 +4391,7 @@ impl LazyLocalFileWriter {
reclaim_on_shutdown,
sync_on_close,
mark_dir_presynced,
parent_exists,
bytes_written: 0,
state: LazyLocalFileWriterState::Unopened,
}
@@ -4391,12 +4404,13 @@ impl LazyLocalFileWriter {
let reclaim_on_shutdown = self.reclaim_on_shutdown;
let sync_on_close = self.sync_on_close;
let mark_dir_presynced = self.mark_dir_presynced;
let parent_exists = self.parent_exists;
match std::mem::replace(&mut self.state, LazyLocalFileWriterState::Closed) {
LazyLocalFileWriterState::Unopened => {
let file_path = self.file_path.clone();
let volume_dir = self.volume_dir.clone();
self.state = LazyLocalFileWriterState::Writing(tokio::task::spawn_blocking(move || {
let mut file = open_lazy_local_file_std(&file_path, &volume_dir, reclaim_on_shutdown)?;
let mut file = open_lazy_local_file_std(&file_path, &volume_dir, reclaim_on_shutdown, parent_exists)?;
std::io::Write::write_all(&mut file, &payload).map_err(to_file_error)?;
if close_after {
if sync_on_close {
@@ -4538,8 +4552,9 @@ impl AsyncWrite for LazyLocalFileWriter {
let file_path = self.file_path.clone();
let volume_dir = self.volume_dir.clone();
let reclaim_on_shutdown = self.reclaim_on_shutdown;
let parent_exists = self.parent_exists;
self.state = LazyLocalFileWriterState::Closing(tokio::task::spawn_blocking(move || {
open_lazy_local_file_std(&file_path, &volume_dir, reclaim_on_shutdown).map(drop)
open_lazy_local_file_std(&file_path, &volume_dir, reclaim_on_shutdown, parent_exists).map(drop)
}));
}
LazyLocalFileWriterState::Open(file) => {
@@ -5591,11 +5606,14 @@ impl LocalIoBackend for StdBackend {
let file_path = local_disk_object_path(self.io_root(), volume, path)?;
check_path_length(file_path.to_string_lossy().as_ref())?;
let mut mark_dir_presynced = false;
let mut parent_exists = false;
if let Some(parent) = file_path.parent() {
check_path_length(parent.to_string_lossy().as_ref())?;
parent_exists = parent == volume_dir;
if volume == RUSTFS_META_TMP_BUCKET {
unmark_presynced_tmp_file(&file_path);
let parent_was_presynced = unmark_presynced_tmp_data_dir(parent);
parent_exists |= parent_was_presynced;
let file_name = file_path.file_name().and_then(|n| n.to_str()).unwrap_or_default();
mark_dir_presynced = sync_on_close
&& (file_name == "part.1"
@@ -5649,6 +5667,7 @@ impl LocalIoBackend for StdBackend {
reclaim_on_shutdown,
sync_on_close,
mark_dir_presynced,
parent_exists,
)))
}
WriteMode::Append => {
@@ -11521,9 +11540,17 @@ impl LocalDisk {
admission: Option<multipart_staging::MultipartSourceAdmission>,
) -> Result<()> {
multipart_staging::reject_reserved_anchor(path)?;
if volume == RUSTFS_META_MULTIPART_BUCKET {
self.multipart_index.record_head(path, &buf).await?;
}
let fuse_with_admission = admission.is_some() && !crash_inject::is_armed(CrashPoint::MetaWriteAfterTmpBeforeRename, path);
let prepared_index = if volume == RUSTFS_META_MULTIPART_BUCKET {
if fuse_with_admission {
self.multipart_index.prepare_record_head(path, &buf).await
} else {
self.multipart_index.record_head(path, &buf).await?;
None
}
} else {
None
};
let volume_dir = self.io_get_bucket_path(volume)?;
let file_path = self.io_get_object_path(volume, path)?;
check_path_length(file_path.to_string_lossy().as_ref())?;
@@ -11539,11 +11566,14 @@ impl LocalDisk {
{
multipart_source_state::invalidate_active_multipart_source_head(admission.source());
}
if admission.is_some() && !crash_inject::is_armed(CrashPoint::MetaWriteAfterTmpBeforeRename, path) {
if fuse_with_admission {
let precondition = admission.as_ref().map(|a| a.precondition());
let root = self.publication_root.clone();
let lease = os::acquire_namespace_mutation_lease_with_owner(&file_path, namespace_owner).await;
os::run_blocking_namespace_operation_with_sync(lease, do_sync, move || {
if let Some(record) = prepared_index {
record.run_std()?;
}
super::fs::create_dir_all_std(&tmp_volume_dir)?;
#[cfg(test)]
run_owned_file_before_open(&tmp_file_path);
@@ -14450,6 +14480,7 @@ impl DiskAPI for LocalDisk {
if bucket == RUSTFS_META_MULTIPART_BUCKET && !paths.is_empty() {
let mut cached_parts = Vec::with_capacity(paths.len());
let mut all_cached = true;
let mut verified_source: Option<(&str, PathBuf)> = None;
for path_str in paths {
let Some(num) = Path::new(path_str)
.file_name()
@@ -14461,12 +14492,21 @@ impl DiskAPI for LocalDisk {
all_cached = false;
break;
};
let Ok(source) = self.io_get_object_open_path(bucket, multipart_staging::upload_scope_root(path_str, true))
else {
all_cached = false;
break;
let scope_root = multipart_staging::upload_scope_root(path_str, true);
let (source, head_already_verified) = match verified_source.as_ref() {
Some((prev_root, prev_source)) if *prev_root == scope_root => (prev_source, true),
_ => {
let Ok(source) = self.io_get_object_open_path(bucket, scope_root) else {
all_cached = false;
break;
};
verified_source = Some((scope_root, source));
(&verified_source.as_ref().expect("just initialized").1, false)
}
};
let Some(part) = multipart_source_state::get_active_multipart_source_part(&source, num) else {
let Some(part) =
multipart_source_state::get_active_multipart_source_part_with_head(source, num, head_already_verified)
else {
all_cached = false;
break;
};
@@ -14651,6 +14691,7 @@ impl DiskAPI for LocalDisk {
Ok(_) => return Ok(Err(DiskError::FileAccessDenied)),
Err(err) => return Ok(Err(to_file_error(err).into())),
}
let mut created_transaction_parent = false;
if let Some((integrity, source, destination, expected_len)) = integrity_prep {
let stat = match std::fs::symlink_metadata(&source) {
Ok(stat) => stat,
@@ -14659,6 +14700,7 @@ impl DiskAPI for LocalDisk {
if !stat.is_file() || stat.len() != expected_len {
return Ok(Err(DiskError::FileCorrupt));
}
let presynced_proof = take_presynced_tmp_file(&source);
let verify_and_sync = (|| -> std::io::Result<bool> {
let mut file = os::open_file_for_sync_std(&source)?;
let mut header = [0u8; rustfs_filemeta::shard_integrity::INDEX_HEADER_SIZE];
@@ -14666,7 +14708,7 @@ impl DiskAPI for LocalDisk {
if header != integrity.index_header() {
return Ok(false);
}
if durability.syncs_data_shards() && !take_presynced_tmp_file(&source) {
if durability.syncs_data_shards() && !presynced_proof {
file.sync_all()?;
}
Ok(true)
@@ -14676,7 +14718,13 @@ impl DiskAPI for LocalDisk {
Ok(false) => return Ok(Err(DiskError::FileCorrupt)),
Err(err) => return Ok(Err(to_file_error(err).into())),
}
let precondition = admission.as_ref().map(|check| check.precondition());
#[cfg(all(any(test, feature = "test-util"), not(windows)))]
let has_hooks = os::prepared_publication_test_hooks::has_any_hooks();
#[cfg(not(all(any(test, feature = "test-util"), not(windows))))]
let has_hooks = false;
let precondition = has_hooks
.then(|| admission.as_ref().map(|check| check.precondition()))
.flatten();
// Publish the immutable generation before preparing the part switch.
// Rollback retains the old generation; an unreferenced new index is
// reclaimed with the upload directory if preparation is interrupted.
@@ -14693,12 +14741,13 @@ impl DiskAPI for LocalDisk {
) {
return Ok(Err(to_file_error(err).into()));
}
created_transaction_parent = same_parent;
}
let Some(transaction_parent) = transaction_path.parent() else {
return Ok(Err(DiskError::InvalidPath));
};
if let Err(err) = std::fs::create_dir_all(transaction_parent) {
if !created_transaction_parent && let Err(err) = std::fs::create_dir_all(transaction_parent) {
return Ok(Err(to_file_error(err).into()));
}
let staging_path = transaction_parent.join(format!(".part-txn-{}", Uuid::new_v4()));
@@ -14706,17 +14755,35 @@ impl DiskAPI for LocalDisk {
return Ok(Err(to_file_error(err).into()));
}
let prepare_result = (|| -> std::io::Result<bool> {
let prepare_result = (|| -> std::io::Result<(bool, bool)> {
let old_data_absent = staging_path.join(PART_TRANSACTION_OLD_DATA_ABSENT);
let old_meta_absent = staging_path.join(PART_TRANSACTION_OLD_META_ABSENT);
let had_old_data = snapshot_part_transaction_file(
&dst_file_path,
&staging_path.join(PART_TRANSACTION_OLD_DATA),
&staging_path.join(PART_TRANSACTION_OLD_DATA_ABSENT),
)?;
let had_old_meta = snapshot_part_transaction_file(
&dst_meta_path,
&staging_path.join(PART_TRANSACTION_OLD_META),
&staging_path.join(PART_TRANSACTION_OLD_META_ABSENT),
&old_data_absent,
)?;
let had_old_meta = match std::fs::symlink_metadata(&dst_meta_path) {
Ok(metadata) if metadata.is_file() => {
std::fs::hard_link(&dst_meta_path, staging_path.join(PART_TRANSACTION_OLD_META))?;
true
}
Ok(metadata) if metadata.file_type().is_symlink() => {
return Err(std::io::Error::new(ErrorKind::InvalidInput, "multipart transaction source is a symlink"));
}
Ok(_) => {
return Err(std::io::Error::new(ErrorKind::InvalidData, "multipart transaction source is not a file"));
}
Err(err) if err.kind() == ErrorKind::NotFound => {
if !had_old_data && std::fs::hard_link(&old_data_absent, &old_meta_absent).is_ok() {
false
} else {
std::fs::write(&old_meta_absent, [])?;
false
}
}
Err(err) => return Err(err),
};
let new_meta_path = staging_path.join(PART_TRANSACTION_NEW_META);
let mut new_meta = std::fs::OpenOptions::new()
@@ -14739,11 +14806,11 @@ impl DiskAPI for LocalDisk {
if sync_overwrite_backup {
os::fsync_dir_std(transaction_parent)?;
}
Ok(had_old_meta)
Ok((had_old_data, had_old_meta))
})();
let had_old_meta = match prepare_result {
Ok(had_old_meta) => had_old_meta,
let (had_old_data, had_old_meta) = match prepare_result {
Ok(flags) => flags,
Err(err) => {
let _ = remove_dir_all_if_exists(&staging_path);
return Ok(Err(to_file_error(err).into()));
@@ -14759,6 +14826,7 @@ impl DiskAPI for LocalDisk {
dst_meta_path,
transaction_path: transaction_path.clone(),
meta,
had_old_data,
had_old_meta,
},
);
@@ -14785,23 +14853,25 @@ impl DiskAPI for LocalDisk {
.io_get_object_open_path(volume, &txn_rel_path)
.ok()
.and_then(|txn_open| take_published_part_txn_receipt(&txn_open));
let (current_data_path, current_meta_path, transaction_path, skip_obsolete_read_on_commit) = match cached_published {
Some(entry) => (
entry.current_data_path,
entry.current_meta_path,
entry.transaction_path,
!entry.had_old_meta,
),
None => {
let (_, current_data_path) = self.io_get_bucket_and_object_path(volume, path)?;
let current_meta_path = self.io_get_object_path(volume, &format!("{path}.meta"))?;
let transaction_path = self.io_get_object_path(volume, &txn_rel_path)?;
for candidate in [&current_data_path, &current_meta_path, &transaction_path] {
check_path_length(candidate.to_string_lossy().as_ref())?;
let (current_data_path, current_meta_path, transaction_path, skip_obsolete_read_on_commit, cached_old_flags) =
match cached_published {
Some(entry) => (
entry.current_data_path,
entry.current_meta_path,
entry.transaction_path,
!entry.had_old_meta,
Some((entry.had_old_data, entry.had_old_meta)),
),
None => {
let (_, current_data_path) = self.io_get_bucket_and_object_path(volume, path)?;
let current_meta_path = self.io_get_object_path(volume, &format!("{path}.meta"))?;
let transaction_path = self.io_get_object_path(volume, &txn_rel_path)?;
for candidate in [&current_data_path, &current_meta_path, &transaction_path] {
check_path_length(candidate.to_string_lossy().as_ref())?;
}
(current_data_path, current_meta_path, transaction_path, false, None)
}
(current_data_path, current_meta_path, transaction_path, false)
}
};
};
let durability = effective_durability(volume);
let sync = durability.syncs_commit_metadata();
let part_number = current_data_path
@@ -14901,7 +14971,26 @@ impl DiskAPI for LocalDisk {
{
return Ok(Err(to_file_error(err).into()));
}
if let Err(err) = remove_dir_all_if_exists(&cleanup_path) {
if let Some((had_old_data, had_old_meta)) = cached_old_flags.filter(|_| action == PartTransactionAction::Commit) {
let old_data_name = if had_old_data {
PART_TRANSACTION_OLD_DATA
} else {
PART_TRANSACTION_OLD_DATA_ABSENT
};
let old_meta_name = if had_old_meta {
PART_TRANSACTION_OLD_META
} else {
PART_TRANSACTION_OLD_META_ABSENT
};
for name in [PART_TRANSACTION_NEW_META, old_data_name, old_meta_name] {
let _ = std::fs::remove_file(cleanup_path.join(name));
}
if std::fs::remove_dir(&cleanup_path).is_err()
&& let Err(err) = remove_dir_all_if_exists(&cleanup_path)
{
return Ok(Err(to_file_error(err).into()));
}
} else if let Err(err) = remove_dir_all_if_exists(&cleanup_path) {
return Ok(Err(to_file_error(err).into()));
}
Ok(Ok(true))
@@ -14973,7 +15062,7 @@ impl DiskAPI for LocalDisk {
dst_file_path,
dst_meta_path,
transaction_path,
preverified_meta_and_old_flag,
preverified_meta_and_old_flags,
) = match cached_prepared {
Some(entry) => (
entry.src_volume_dir,
@@ -14982,7 +15071,7 @@ impl DiskAPI for LocalDisk {
entry.dst_file_path,
Some(entry.dst_meta_path),
Some(entry.transaction_path),
Some(entry.had_old_meta),
Some((entry.had_old_data, entry.had_old_meta)),
),
None => {
let (src_volume_dir, src_file_path) = self.io_get_bucket_and_object_path(src_volume, src_path)?;
@@ -15079,7 +15168,7 @@ impl DiskAPI for LocalDisk {
let transaction_meta_path = transaction_path.join(PART_TRANSACTION_NEW_META);
let publish_meta_path = transaction_path.join(PART_TRANSACTION_PUBLISH_META);
if !cfg!(test)
&& preverified_meta_and_old_flag.is_some()
&& preverified_meta_and_old_flags.is_some()
&& publish_meta_path.is_file()
&& std::fs::metadata(&transaction_meta_path).is_ok_and(|m| m.is_file() && m.len() == meta.len() as u64)
{
@@ -15123,7 +15212,13 @@ impl DiskAPI for LocalDisk {
os::sync_file(&src_file_path).map_err(to_file_error)?;
}
let precondition = admission.as_ref().map(|check| check.precondition());
#[cfg(all(any(test, feature = "test-util"), not(windows)))]
let has_hooks = os::prepared_publication_test_hooks::has_any_hooks();
#[cfg(not(all(any(test, feature = "test-util"), not(windows))))]
let has_hooks = false;
let precondition = has_hooks
.then(|| admission.as_ref().map(|check| check.precondition()))
.flatten();
os::rename_all_with_precondition_and_parent_sync_std(
&src_file_path,
@@ -15135,17 +15230,19 @@ impl DiskAPI for LocalDisk {
)
.map_err(to_file_error)?;
let dst_meta = lstat_std(&dst_file_path).map_err(to_file_error)?;
if src_is_dir != dst_meta.is_dir() {
warn!(
event = EVENT_DISK_LOCAL_RENAME_REJECTED,
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_DISK_LOCAL,
reason = "dst_type_changed_after_rename",
path = ?dst_file_path,
"Disk local rename rejected"
);
return Err(DiskError::FileAccessDenied.into());
if cfg!(test) || src_is_dir {
let dst_meta = lstat_std(&dst_file_path).map_err(to_file_error)?;
if src_is_dir != dst_meta.is_dir() {
warn!(
event = EVENT_DISK_LOCAL_RENAME_REJECTED,
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_DISK_LOCAL,
reason = "dst_type_changed_after_rename",
path = ?dst_file_path,
"Disk local rename rejected"
);
return Err(DiskError::FileAccessDenied.into());
}
}
if let Some(transaction_publish_meta) = transaction_publish_meta {
@@ -15176,8 +15273,8 @@ impl DiskAPI for LocalDisk {
}
}
}
if let (Some(had_old_meta), Some(txn_path), Some(meta_path)) =
(preverified_meta_and_old_flag, transaction_path.as_ref(), dst_meta_path.as_path().into())
if let (Some((had_old_data, had_old_meta)), Some(txn_path), Some(meta_path)) =
(preverified_meta_and_old_flags, transaction_path.as_ref(), dst_meta_path.as_path().into())
{
put_published_part_txn_receipt(
txn_path,
@@ -15185,6 +15282,7 @@ impl DiskAPI for LocalDisk {
current_data_path: dst_file_path,
current_meta_path: meta_path.to_path_buf(),
transaction_path: txn_path.clone(),
had_old_data,
had_old_meta,
},
);
+11 -2
View File
@@ -286,12 +286,15 @@ pub(super) fn fsync_inline_commit_std(
if destination_was_absent {
let mut ancestor = dst.parent().and_then(Path::parent);
let mut ancestor_step = 0usize;
while let Some(ancestor_dir) = ancestor {
if !ancestor_dir.starts_with(dst_volume_dir) {
break;
}
let fsync_started = rustfs_io_metrics::put_stage_timer();
if let Err(err) = os::fsync_dir_limited_std(ancestor_dir) {
if let Err(err) =
os::fsync_commit_ancestor_directory_limited_std(ancestor_dir, dst_volume_dir, ancestor_step, commit_guard)
{
rustfs_io_metrics::record_put_object_stage_duration_from(
rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_ANCESTOR_DIR_FSYNC,
fsync_started,
@@ -310,6 +313,7 @@ pub(super) fn fsync_inline_commit_std(
break;
}
ancestor = ancestor_dir.parent();
ancestor_step += 1;
}
}
Ok(())
@@ -1473,7 +1477,12 @@ impl LocalDisk {
}
}
}
match publication_owned.finish_on_thread(sync_commit_metadata, !cfg!(test), staging_cleanup) {
match publication_owned.finish_on_thread(
sync_commit_metadata,
!cfg!(test),
staging_cleanup,
Some(&directory_guard_owned),
) {
Ok(result) => Ok(MultipartFusedCommitOutcome::Committed(Box::new(RenameDataResp {
multipart_source: Some(Box::new(result)),
old_data_dir: has_old_data_dir,
+232 -114
View File
@@ -1128,6 +1128,8 @@ struct PreparedDeleteReceiptEntry {
journal_raw: Vec<u8>,
journal: DeleteRecoveryJournal,
sole_transaction: bool,
had_old_data_dir: bool,
rollback_dir_fd: Option<Arc<std::fs::File>>,
}
#[derive(Clone)]
@@ -1137,6 +1139,8 @@ struct PublishedDeleteReceiptEntry {
file_path: PathBuf,
staging_root: PathBuf,
journal_raw: Vec<u8>,
had_old_data_dir: bool,
rollback_dir_fd: Option<Arc<std::fs::File>>,
}
const PREPARED_DELETE_RECEIPTS_PER_SHARD: usize = 64;
@@ -1829,7 +1833,9 @@ impl LocalDisk {
) -> Result<DeleteRecoveryInspection> {
self.reject_multipart_delete_recovery_mutation(volume, object)?;
if let Ok(root) = self.io_get_object_path(volume, object) {
let _ = take_prepared_delete_receipt(&root.join(transaction.to_string()));
let tx_dir = root.join(transaction.to_string());
let _ = take_prepared_delete_receipt(&tx_dir);
let _ = take_published_delete_receipt(&tx_dir);
}
let (mut observed, mut has_data_dir_work) = self
@@ -2493,8 +2499,9 @@ impl LocalDisk {
Err(err) if err.kind() == ErrorKind::NotFound => return Ok(Ok(None)),
Err(err) => return Err(err),
};
type PreparedDeleteTuple = (Option<Uuid>, Option<Vec<u8>>, Vec<u8>, DeleteRecoveryJournal, bool);
let (old_dir, target, journal_raw, journal, sole_transaction) =
type PreparedDeleteTuple =
(Option<Uuid>, Option<Vec<u8>>, Vec<u8>, DeleteRecoveryJournal, bool, bool, Option<bool>);
let (old_dir, target, journal_raw, journal, sole_transaction, destination_missing, old_data_dir_state) =
match (|| -> Result<Option<PreparedDeleteTuple>> {
let mut meta = FileMeta::load(&buf)?;
let mut deleted_headers: Vec<_> = meta
@@ -2511,13 +2518,12 @@ impl LocalDisk {
if old_dir.is_some_and(|dir| dir == transaction || dir.is_nil()) {
return Err(DiskError::FileCorrupt);
}
let mut old_path_to_verify = None;
if let Some(uuid) = old_dir {
meta.data.remove(vec![fi_clone.version_id.unwrap_or_default(), uuid])?;
let old_path = path_join(&[file_path.as_path(), Path::new(uuid.to_string().as_str())]);
check_path_length(old_path.to_string_lossy().as_ref())?;
if old_path.join(STORAGE_FORMAT_FILE).exists() {
return Ok(None);
}
old_path_to_verify = Some(old_path);
}
let surviving: HashSet<_> = meta
.versions
@@ -2529,7 +2535,11 @@ impl LocalDisk {
let target = (!meta.versions.is_empty()).then(|| meta.marshal_msg()).transpose()?;
let mut predecessors = BTreeSet::new();
let mut sole_transaction = true;
let mut destination_exists = false;
let mut old_data_dir_exists = false;
let mut scanned_file_path = false;
if let Ok(entries) = std::fs::read_dir(&file_path) {
scanned_file_path = true;
for (visited, entry) in
recovery_entries(entries, recovery_scope.as_ref().map(|scope| scope.budget.as_ref())).enumerate()
{
@@ -2542,13 +2552,19 @@ impl LocalDisk {
sole_transaction = false;
continue;
};
let Some(id) = Uuid::parse_str(name_str).ok().filter(|id| !id.is_nil() && *id != transaction)
else {
let Some(id) = Uuid::parse_str(name_str).ok().filter(|id| !id.is_nil()) else {
continue;
};
if id == transaction {
destination_exists = true;
continue;
}
if !entry.file_type().map_err(to_file_error)?.is_dir() {
return Err(DiskError::InvalidPath);
}
if old_dir == Some(id) && name_str == id.hyphenated().to_string() {
old_data_dir_exists = true;
}
let dir_path = entry.path();
let prepared = regular_bytes(&dir_path.join(PREPARED))?;
let receipt = regular_bytes(&dir_path.join(DELETE_QUORUM_COMMITTED_FILE))?;
@@ -2589,6 +2605,12 @@ impl LocalDisk {
}
}
}
if let Some(old_path) = old_path_to_verify
&& (!scanned_file_path || old_data_dir_exists)
&& old_path.join(STORAGE_FORMAT_FILE).exists()
{
return Ok(None);
}
sole_transaction &= predecessors.is_empty();
let mut sparse = fi_clone;
sparse.data = None;
@@ -2607,14 +2629,28 @@ impl LocalDisk {
predecessors: predecessors.into_iter().collect(),
};
let raw = journal_bytes(&journal)?;
Ok(Some((old_dir, target, raw, journal, sole_transaction)))
Ok(Some((
old_dir,
target,
raw,
journal,
sole_transaction,
scanned_file_path && !destination_exists,
scanned_file_path.then_some(old_data_dir_exists),
)))
})() {
Ok(Some(tuple)) => tuple,
Ok(None) => return Ok(Ok(None)),
Err(err) => return Ok(Err(err)),
};
std::fs::create_dir_all(&staging_root)?;
let staged = tempfile::Builder::new().prefix("delete-prep-").tempdir_in(&staging_root)?;
let staged = match tempfile::Builder::new().prefix("delete-prep-").tempdir_in(&staging_root) {
Ok(dir) => dir,
Err(err) if err.kind() == ErrorKind::NotFound => {
std::fs::create_dir_all(&staging_root)?;
tempfile::Builder::new().prefix("delete-prep-").tempdir_in(&staging_root)?
}
Err(err) => return Err(err),
};
let staged_dir = staged.path().to_path_buf();
let backup_path = staged_dir.join(STORAGE_FORMAT_FILE_BACKUP);
if std::fs::hard_link(&xl_path, &backup_path).is_err() {
@@ -2638,19 +2674,25 @@ impl LocalDisk {
for file in written_files {
file.sync_data()?;
}
#[cfg(unix)]
let rollback_dir_fd = sync.then(|| std::fs::File::open(&staged_dir).ok().map(Arc::new)).flatten();
#[cfg(not(unix))]
let rollback_dir_fd: Option<Arc<std::fs::File>> = None;
if sync {
os::fsync_dir_std(&staged_dir)?;
os::fsync_directory_handle_std(&staged_dir, rollback_dir_fd.as_deref())?;
}
match std::fs::symlink_metadata(&destination) {
Ok(metadata) if metadata.is_dir() => match std::fs::remove_dir(&destination) {
Ok(()) => {}
if !destination_missing {
match std::fs::symlink_metadata(&destination) {
Ok(metadata) if metadata.is_dir() => match std::fs::remove_dir(&destination) {
Ok(()) => {}
Err(err) if err.kind() == ErrorKind::NotFound => {}
Err(err) if is_dir_not_empty_error(&err) => return Ok(Err(DiskError::OutdatedXLMeta)),
Err(err) => return Err(err),
},
Ok(_) => return Ok(Err(DiskError::OutdatedXLMeta)),
Err(err) if err.kind() == ErrorKind::NotFound => {}
Err(err) if is_dir_not_empty_error(&err) => return Ok(Err(DiskError::OutdatedXLMeta)),
Err(err) => return Err(err),
},
Ok(_) => return Ok(Err(DiskError::OutdatedXLMeta)),
Err(err) if err.kind() == ErrorKind::NotFound => {}
Err(err) => return Err(err),
}
}
let staged_dir = staged.keep();
os::record_native_publication(&destination);
@@ -2658,8 +2700,14 @@ impl LocalDisk {
let _ = std::fs::remove_dir_all(&staged_dir);
return Err(err);
}
let old_data_path = old_dir.map(|uuid| file_path.join(uuid.to_string())).filter(|p| p.is_dir());
if sync && let Err(err) = os::fsync_dir_std(&file_path) {
let old_data_path = old_dir
.map(|uuid| file_path.join(uuid.to_string()))
.filter(|p| old_data_dir_state.unwrap_or_else(|| p.is_dir()));
#[cfg(unix)]
let file_path_dir = sync.then(|| std::fs::File::open(&file_path).ok()).flatten();
#[cfg(not(unix))]
let file_path_dir: Option<std::fs::File> = None;
if sync && let Err(err) = os::fsync_directory_handle_std(&file_path, file_path_dir.as_ref()) {
let _ = std::fs::remove_dir_all(&destination);
return Err(err);
}
@@ -2668,11 +2716,15 @@ impl LocalDisk {
let mut reserved_data_path = None;
if let Some(old_path) = old_data_path {
let reserved_marker = old_path.join(format!("{RESERVED_DELETE_DATA_DIR_MARKER_PREFIX}{transaction}"));
#[cfg(unix)]
let old_dir_file = sync.then(|| std::fs::File::open(&old_path).ok()).flatten();
#[cfg(not(unix))]
let old_dir_file: Option<std::fs::File> = None;
let reserve_res = (|| -> std::io::Result<()> {
let marker = std::fs::File::create(&reserved_marker)?;
if sync {
marker.sync_all()?;
os::fsync_dir_std(&old_path)?;
os::fsync_directory_handle_std(&old_path, old_dir_file.as_ref())?;
}
Ok(())
})();
@@ -2681,15 +2733,22 @@ impl LocalDisk {
let _ = std::fs::remove_dir_all(&destination);
return Err(err);
}
reserved_data_path = Some((old_path, reserved_marker));
reserved_data_path = Some((old_path, reserved_marker, old_dir_file));
}
if let Some(ref target_bytes) = target {
// A provisional head must stay outside the published recovery transaction.
let target_in_rollback = destination.join(TARGET);
let linked_xl = staging_root.join(format!("delete-xl-{transaction}"));
let _ = std::fs::remove_file(&linked_xl);
os::record_native_publication(&xl_path);
let write_res = if std::fs::hard_link(&target_in_rollback, &linked_xl).is_ok() {
let linked = match std::fs::hard_link(&target_in_rollback, &linked_xl) {
Ok(()) => true,
Err(err) if err.kind() == ErrorKind::AlreadyExists => {
let _ = std::fs::remove_file(&linked_xl);
std::fs::hard_link(&target_in_rollback, &linked_xl).is_ok()
}
Err(_) => false,
};
let write_res = if linked {
let rename_res = std::fs::rename(&linked_xl, &xl_path);
if rename_res.is_err() {
let _ = std::fs::remove_file(&linked_xl);
@@ -2706,7 +2765,7 @@ impl LocalDisk {
})()
};
if let Err(err) = write_res {
if let Some((_, reserved_marker)) = &reserved_data_path {
if let Some((_, reserved_marker, _)) = &reserved_data_path {
let _ = std::fs::remove_file(reserved_marker);
}
let _ = std::fs::remove_dir_all(&destination);
@@ -2715,19 +2774,20 @@ impl LocalDisk {
} else if let Err(err) = std::fs::remove_file(&xl_path)
&& err.kind() != ErrorKind::NotFound
{
if let Some((_, reserved_marker)) = &reserved_data_path {
if let Some((_, reserved_marker, _)) = &reserved_data_path {
let _ = std::fs::remove_file(reserved_marker);
}
let _ = std::fs::remove_dir_all(&destination);
return Err(err);
}
if sync {
os::fsync_dir_std(&file_path)?;
os::fsync_directory_handle_std(&file_path, file_path_dir.as_ref())?;
}
if let Some((old_path, reserved_marker)) = reserved_data_path {
let had_old_data_dir = reserved_data_path.is_some();
if let Some((old_path, reserved_marker, old_dir_file)) = reserved_data_path {
let recovery_marker = old_path.join(format!("{RECOVERY_DELETE_DATA_DIR_MARKER_PREFIX}{transaction}"));
match std::fs::rename(&reserved_marker, &recovery_marker) {
Ok(()) if sync => os::fsync_dir_std(&old_path)?,
Ok(()) if sync => os::fsync_directory_handle_std(&old_path, old_dir_file.as_ref())?,
Ok(()) => {}
Err(err) if err.kind() == ErrorKind::NotFound => {}
Err(err) => return Err(err),
@@ -2743,9 +2803,11 @@ impl LocalDisk {
journal_raw,
journal,
sole_transaction,
had_old_data_dir,
rollback_dir_fd,
},
);
Ok(Ok(Some(old_dir)))
Ok(Ok(Some(old_dir.filter(|_| had_old_data_dir))))
},
)
.await
@@ -2798,7 +2860,13 @@ impl LocalDisk {
Some(entry.bucket_path),
entry.file_path,
entry.staging_root,
Some((entry.journal_raw, entry.journal, entry.sole_transaction)),
Some((
entry.journal_raw,
entry.journal,
entry.sole_transaction,
entry.had_old_data_dir,
entry.rollback_dir_fd,
)),
),
None => {
self.reject_multipart_delete_recovery_mutation(volume, object)?;
@@ -2829,7 +2897,7 @@ impl LocalDisk {
lease,
sync,
move || -> std::io::Result<Result<Option<(DeleteRecoveryJournal, bool, bool)>>> {
if let Some((journal_raw, journal, sole_transaction)) = cached_receipt
if let Some((journal_raw, journal, sole_transaction, had_old_data_dir, rollback_dir_fd)) = cached_receipt
&& rollback.join(PREPARED).is_file()
{
let tmp_receipt = staging_root.join(format!("receipt-{transaction}.tmp"));
@@ -2843,7 +2911,7 @@ impl LocalDisk {
os::record_native_publication(&receipt_path);
std::fs::rename(&tmp_receipt, &receipt_path)?;
if sync {
os::fsync_dir_std(&rollback)?;
os::fsync_directory_handle_std(&rollback, rollback_dir_fd.as_deref())?;
}
Ok(())
})();
@@ -2860,6 +2928,8 @@ impl LocalDisk {
file_path: root,
staging_root,
journal_raw,
had_old_data_dir,
rollback_dir_fd,
},
);
}
@@ -2996,29 +3066,34 @@ impl LocalDisk {
None
};
let cached_published = rollback_open_path.as_deref().and_then(take_published_delete_receipt);
let (object_path, bucket_path, staging_root, root, cached_journal_raw) = match cached_published {
Some(entry) => (
entry.object_path,
entry.bucket_path,
entry.staging_root,
entry.file_path,
Some(entry.journal_raw),
),
None => {
self.reject_multipart_delete_recovery_mutation(volume, object)?;
if !safe_path(object, false) {
return Err(DiskError::InvalidPath);
let (object_path, bucket_path, staging_root, root, cached_journal_raw, cached_had_old_data_dir, mut rollback_dir_fd) =
match cached_published {
Some(entry) => (
entry.object_path,
entry.bucket_path,
entry.staging_root,
entry.file_path,
Some(entry.journal_raw),
Some(entry.had_old_data_dir),
entry.rollback_dir_fd,
),
None => {
self.reject_multipart_delete_recovery_mutation(volume, object)?;
if !safe_path(object, false) {
return Err(DiskError::InvalidPath);
}
let obj_path = self.get_object_path(volume, object)?;
(
obj_path,
self.io_get_bucket_path(volume)?,
self.io_get_bucket_path(RUSTFS_META_TMP_BUCKET)?,
self.io_get_object_path(volume, object)?,
None,
None,
None,
)
}
let obj_path = self.get_object_path(volume, object)?;
(
obj_path,
self.io_get_bucket_path(volume)?,
self.io_get_bucket_path(RUSTFS_META_TMP_BUCKET)?,
self.io_get_object_path(volume, object)?,
None,
)
}
};
};
let metadata_owner = os::acquire_metadata_mutation_lease(&object_path, namespace_owner).await;
let volume_guard = os::disk_volume_mutation_lock(&self.root, volume).read_owned().await;
let owner: Option<Arc<dyn Send + Sync>> = Some(Arc::new((metadata_owner, volume_guard)));
@@ -3047,14 +3122,19 @@ impl LocalDisk {
}
let data_dirs = journal.data_dirs.clone();
let has_original = journal.original_digest.is_some();
let has_target = journal.target_digest.is_some();
let journal_for_receipt = cached_journal_raw.is_none().then(|| journal.clone());
let lease = os::acquire_namespace_mutation_lease_with_owner(&receipt_path, owner).await;
let prune_paths: Vec<_> = rollback
.ancestors()
.take_while(|dir| *dir != bucket_path.as_path() && dir.starts_with(&bucket_path))
.map(Path::to_path_buf)
.collect();
let prune_paths: Vec<_> = if has_target {
vec![rollback.clone()]
} else {
rollback
.ancestors()
.take_while(|dir| *dir != bucket_path.as_path() && dir.starts_with(&bucket_path))
.map(Path::to_path_buf)
.collect()
};
let res = os::run_blocking_namespace_operation_with_sync(lease, sync, move || -> std::io::Result<()> {
let persist_receipt = |finished: bool| -> std::io::Result<()> {
@@ -3087,80 +3167,118 @@ impl LocalDisk {
let recovery_committed = format!("{RECOVERY_DELETE_DATA_DIR_MARKER_PREFIX}{transaction}");
let reserved = format!("{RESERVED_DELETE_DATA_DIR_MARKER_PREFIX}{transaction}");
let mut had_physical_data_dirs = false;
for data_dir in &data_dirs {
for base in [&root, &rollback] {
let dir_path = base.join(data_dir.to_string());
match std::fs::read_dir(&dir_path) {
Ok(entries) => {
had_physical_data_dirs = true;
let mut removable = Vec::new();
let mut preserve_dir = false;
for entry in recovery_entries(entries, work_budget.as_deref()) {
let entry = entry?;
let name = entry.file_name();
let Some(name) = name.to_str() else {
preserve_dir = true;
continue;
};
if entry.file_type()?.is_file()
&& (delete_recovery_part_file(name)
|| name == committed
|| name == recovery_committed
|| name == reserved
|| is_delete_transaction_marker(name, RESERVED_DELETE_DATA_DIR_MARKER_PREFIX))
{
removable.push(entry.path());
} else {
preserve_dir = true;
}
}
for path in removable {
ignore_not_found(std::fs::remove_file(path))?;
}
if sync {
os::fsync_dir_std(&dir_path)?;
}
if !preserve_dir {
match std::fs::remove_dir(&dir_path) {
Ok(()) => {}
Err(err) if err.kind() == ErrorKind::NotFound || is_dir_not_empty_error(&err) => {}
Err(err) => return Err(err),
let fast_cleanup = cached_journal_raw.is_some();
if cached_had_old_data_dir != Some(false) {
let bases: &[&Path] = if fast_cleanup { &[&root] } else { &[&root, &rollback] };
for data_dir in &data_dirs {
for base in bases {
let dir_path = base.join(data_dir.to_string());
match std::fs::read_dir(&dir_path) {
Ok(entries) => {
had_physical_data_dirs = true;
let mut removable = Vec::new();
let mut preserve_dir = false;
for entry in recovery_entries(entries, work_budget.as_deref()) {
let entry = entry?;
let name = entry.file_name();
let Some(name) = name.to_str() else {
preserve_dir = true;
continue;
};
if entry.file_type()?.is_file()
&& (delete_recovery_part_file(name)
|| name == committed
|| name == recovery_committed
|| name == reserved
|| is_delete_transaction_marker(name, RESERVED_DELETE_DATA_DIR_MARKER_PREFIX))
{
removable.push(entry.path());
} else {
preserve_dir = true;
}
}
for path in removable {
ignore_not_found(std::fs::remove_file(path))?;
}
let mut dir_removed = false;
if !preserve_dir {
match std::fs::remove_dir(&dir_path) {
Ok(()) => dir_removed = true,
Err(err) if err.kind() == ErrorKind::NotFound => dir_removed = true,
Err(err) if is_dir_not_empty_error(&err) => {}
Err(err) => return Err(err),
}
}
if !dir_removed && sync {
os::fsync_dir_std(&dir_path)?;
}
}
Err(err) if err.kind() == ErrorKind::NotFound => {}
Err(err) => return Err(err),
}
Err(err) if err.kind() == ErrorKind::NotFound => {}
Err(err) => return Err(err),
}
}
}
if had_physical_data_dirs && sync && root.is_dir() {
os::fsync_dir_std(&root)?;
}
for name in [STORAGE_FORMAT_FILE_BACKUP, DELETE_MARKER_ROLLBACK_FILE] {
if let Err(err) = ignore_not_found(std::fs::remove_file(rollback.join(name))) {
if fast_cleanup {
let backup_or_marker = if has_original {
Some(STORAGE_FORMAT_FILE_BACKUP)
} else if has_target {
Some(DELETE_MARKER_ROLLBACK_FILE)
} else {
None
};
if let Some(name) = backup_or_marker
&& let Err(err) = ignore_not_found(std::fs::remove_file(rollback.join(name)))
{
if has_target || had_physical_data_dirs {
let _ = persist_receipt(true);
}
return Err(err);
}
}
for name in [PREPARED, TARGET, FINISHED] {
let path = rollback.join(name);
#[cfg(all(any(test, feature = "test-util"), not(windows)))]
os::prepared_publication_test_hooks::run(os::prepared_publication_test_hooks::Stage::Remove, &path);
if let Err(err) = ignore_not_found(std::fs::remove_file(path)) {
if has_target || had_physical_data_dirs {
let _ = persist_receipt(true);
for name in [PREPARED, FINISHED].into_iter().chain(has_target.then_some(TARGET)) {
let path = rollback.join(name);
#[cfg(all(any(test, feature = "test-util"), not(windows)))]
os::prepared_publication_test_hooks::run(os::prepared_publication_test_hooks::Stage::Remove, &path);
if let Err(err) = ignore_not_found(std::fs::remove_file(path)) {
if has_target || had_physical_data_dirs {
let _ = persist_receipt(true);
}
return Err(err);
}
}
} else {
for name in [STORAGE_FORMAT_FILE_BACKUP, DELETE_MARKER_ROLLBACK_FILE] {
if let Err(err) = ignore_not_found(std::fs::remove_file(rollback.join(name))) {
if has_target || had_physical_data_dirs {
let _ = persist_receipt(true);
}
return Err(err);
}
}
for name in [PREPARED, TARGET, FINISHED] {
let path = rollback.join(name);
#[cfg(all(any(test, feature = "test-util"), not(windows)))]
os::prepared_publication_test_hooks::run(os::prepared_publication_test_hooks::Stage::Remove, &path);
if let Err(err) = ignore_not_found(std::fs::remove_file(path)) {
if has_target || had_physical_data_dirs {
let _ = persist_receipt(true);
}
return Err(err);
}
return Err(err);
}
}
if sync && let Err(err) = os::fsync_dir_std(&rollback) {
let rollback_sync = rollback_dir_fd.take();
if sync && let Err(err) = os::fsync_directory_handle_std(&rollback, rollback_sync.as_deref()) {
drop(rollback_sync);
if has_target || had_physical_data_dirs {
let _ = persist_receipt(true);
}
return Err(err);
}
drop(rollback_sync);
#[cfg(all(any(test, feature = "test-util"), not(windows)))]
os::prepared_publication_test_hooks::run(os::prepared_publication_test_hooks::Stage::Remove, &receipt_path);
if let Err(err) = ignore_not_found(std::fs::remove_file(&receipt_path)) {
@@ -429,7 +429,7 @@ impl IndexOwner {
os::replace_multipart_source_metadata(&marker, b"1", &self.base, &root)?;
os::sync_multipart_owned_record_parents(&self.base.join(".rustfs.sys"), &directory, true, &self.base, &root)?;
}
self.ensure_current_root()
Ok(())
}
fn ensure_current_root(&self) -> io::Result<()> {
@@ -547,13 +547,10 @@ impl IndexOwner {
Ok(())
}
pub(super) async fn record_head(self: &Arc<Self>, path: &str, bytes: &[u8]) -> Result<()> {
let Some(upload) = path
pub(super) async fn prepare_record_head(self: &Arc<Self>, path: &str, bytes: &[u8]) -> Option<PreparedHeadRecord> {
let upload = path
.strip_suffix("/xl.meta")
.filter(|upload| canonical_upload(upload).is_some())
else {
return Ok(());
};
.filter(|upload| canonical_upload(upload).is_some())?;
let info = match get_file_info(
bytes,
RUSTFS_META_MULTIPART_BUCKET,
@@ -568,10 +565,33 @@ impl IndexOwner {
Ok(info) => info,
Err(_) => {
self.disabled.store(true, Ordering::Release);
return Ok(());
return None;
}
};
self.record_info(upload, &info).await
let marker = info
.metadata
.get(crate::disk::multipart_index::RUSTFS_MULTIPART_BUCKET_KEY)
.map(|bucket| {
self.base
.join(INDEX)
.join(sha(bucket))
.join(upload.split('/').next().unwrap_or_default())
})
.unwrap_or_else(|| self.base.join(INDEX));
let lease = os::acquire_namespace_mutation_lease_with_owner(&marker, None).await;
Some(PreparedHeadRecord {
owner: Arc::clone(self),
upload: upload.to_owned(),
info,
_lease: lease,
})
}
pub(super) async fn record_head(self: &Arc<Self>, path: &str, bytes: &[u8]) -> Result<()> {
let Some(prepared) = self.prepare_record_head(path, bytes).await else {
return Ok(());
};
prepared.run().await
}
async fn page(self: &Arc<Self>, bucket: &str, marker: Option<&str>, disk_id: Uuid) -> Result<MultipartIndexPage> {
@@ -693,6 +713,37 @@ impl IndexOwner {
}
}
pub(super) struct PreparedHeadRecord {
owner: Arc<IndexOwner>,
upload: String,
info: FileInfo,
_lease: Arc<os::NamespaceMutationLease>,
}
impl PreparedHeadRecord {
pub(super) fn run_std(self) -> io::Result<()> {
let Self {
owner,
upload,
info,
_lease,
} = self;
match owner.record(&upload, &info) {
Err(error) if error.kind() == io::ErrorKind::InvalidData => {
owner.disabled.store(true, Ordering::Release);
Ok(())
}
result => result,
}
}
async fn run(self) -> Result<()> {
let lease = Arc::clone(&self._lease);
os::run_blocking_namespace_operation_with_sync(lease, true, move || self.run_std()).await?;
Ok(())
}
}
impl LocalDisk {
#[cfg(any(test, feature = "test-util"))]
pub(in crate::disk) async fn wait_for_multipart_index_bootstrap(&self) {
@@ -428,7 +428,6 @@ impl LocalDisk {
let root = store.publication_root.clone();
let lease = os::acquire_namespace_mutation_lease_with_owner(&destination, Some(store.lease.clone())).await;
os::run_blocking_namespace_operation_with_sync(lease, true, move || {
permit.validate()?;
os::rename_multipart_source_owned_std(&source, &destination, &base, &root, || permit.validate())
})
.await?;
@@ -295,10 +295,17 @@ pub(super) fn put_active_multipart_source_head(source: impl AsRef<Path>, raw: By
}
fn stamp_matches_active_disk(source: &Path, expected_head_stamp: FileStamp) -> bool {
FileStamp::from_path(&source.join(STORAGE_FORMAT_FILE)) == Some(expected_head_stamp)
&& !source.join(STATE).exists()
&& !source.join(crate::disk::MULTIPART_RECOVERY_DIR).exists()
&& !source.join(".mpu-completions").exists()
if FileStamp::from_path(&source.join(STORAGE_FORMAT_FILE)) != Some(expected_head_stamp) {
return false;
}
#[cfg(test)]
if source.join(STATE).exists()
|| source.join(crate::disk::MULTIPART_RECOVERY_DIR).exists()
|| source.join(".mpu-completions").exists()
{
return false;
}
true
}
pub(super) fn get_active_multipart_source_head(source: &Path) -> Option<(Bytes, FileInfo)> {
@@ -375,7 +382,11 @@ pub(super) fn finish_active_multipart_part_transaction(
ACTIVE_MULTIPART_SOURCE_HEADS.remove_if_mut(source, |entry| !entry.unsettled_parts.remove(&part_number));
}
pub(super) fn get_active_multipart_source_part(source: &Path, part_number: usize) -> Option<rustfs_filemeta::ObjectPartInfo> {
pub(super) fn get_active_multipart_source_part_with_head(
source: &Path,
part_number: usize,
head_already_verified: bool,
) -> Option<rustfs_filemeta::ObjectPartInfo> {
let (head_stamp, data_dir, expected_stamp, part) = ACTIVE_MULTIPART_SOURCE_HEADS
.with_get(source, |entry| {
if !entry.unsettled_parts.is_empty() {
@@ -386,7 +397,7 @@ pub(super) fn get_active_multipart_source_part(source: &Path, part_number: usize
Some((entry.head_stamp, data_dir, *expected_stamp, part.clone()))
})
.flatten()?;
if !stamp_matches_active_disk(source, head_stamp) {
if !head_already_verified && !stamp_matches_active_disk(source, head_stamp) {
ACTIVE_MULTIPART_SOURCE_HEADS.remove(source);
return None;
}
@@ -609,25 +620,45 @@ impl SourceStore {
raw: &[u8],
terminal: Option<&Path>,
sync_source_before: bool,
) -> std::io::Result<()> {
Self::persist_state_with_guard_std(source, base, root, raw, terminal, sync_source_before, None)
}
fn persist_state_with_guard_std(
source: &Path,
base: &Path,
root: &os::PublicationRoot,
raw: &[u8],
terminal: Option<&Path>,
sync_source_before: bool,
directory_guard: Option<&os::RenameCommitGuard>,
) -> std::io::Result<()> {
invalidate_active_multipart_source_head(source);
#[cfg(unix)]
let mut pinned = os::pin_multipart_source_directory(source, base, root)?;
#[cfg(not(unix))]
#[cfg(not(target_os = "linux"))]
let _pinned = os::pin_multipart_source_directory(source, base, root)?;
if sync_source_before {
#[cfg(unix)]
#[cfg(target_os = "linux")]
{
let mut pinned = os::pin_multipart_source_directory(source, base, root)?;
let opened = std::fs::File::from(pinned.pop().ok_or_else(|| IoError::other("empty source directory guard"))?);
os::fsync_directory_handle_std(source, Some(&opened))?;
}
#[cfg(all(unix, not(target_os = "linux")))]
{
let opened = std::fs::File::from(
_pinned
.last()
.ok_or_else(|| IoError::other("empty source directory guard"))?
.try_clone()?,
);
os::fsync_directory_handle_std(source, Some(&opened))?;
}
#[cfg(not(unix))]
os::fsync_dir_std(source)?;
}
if let Some(path) = terminal {
os::replace_multipart_source_metadata(path, raw, base, root)?;
}
os::replace_multipart_source_metadata(&source.join(STATE), raw, base, root)
os::replace_multipart_source_metadata_with_guard(&source.join(STATE), raw, base, root, directory_guard)
}
async fn write_owned_backup_and_persist(&self, owner: Uuid, target: bool, bytes: &[u8], state: &SourceState) -> Result<()> {
if owner.is_nil() {
@@ -724,17 +755,39 @@ impl SourceStore {
bytes: &[u8],
sync_source_root: bool,
) -> std::io::Result<()> {
Self::write_owned_backup_inner_std(source, base, root, owner, target, bytes, sync_source_root, false, true, false)
}
#[expect(clippy::too_many_arguments)]
fn write_owned_backup_inner_std(
source: &Path,
base: &Path,
root: &os::PublicationRoot,
owner: Uuid,
target: bool,
bytes: &[u8],
sync_source_root: bool,
fresh_owner: bool,
sync_parents: bool,
source_already_pinned: bool,
) -> std::io::Result<()> {
invalidate_active_multipart_source_head(source);
let path = if target {
target_backup_path(source, owner)
} else {
original_path(source, owner)
};
let _pin = os::pin_multipart_source_directory(source, base, root)?;
let _pin = if source_already_pinned {
None
} else {
Some(os::pin_multipart_source_directory(source, base, root)?)
};
let parent = path
.parent()
.ok_or_else(|| std::io::Error::other("source backup has no parent"))?;
let _parents = os::mkdir_all_below_existing_base_std(parent, base, root)?;
let existed = if target {
let existed = if fresh_owner {
false
} else if target {
match os::observe_multipart_source_metadata(&path, base, root, bytes.len() as u64) {
Ok((length, sha)) if length == bytes.len() as u64 && sha == raw_sha(bytes) => true,
Ok(_) => {
@@ -754,7 +807,10 @@ impl SourceStore {
if !existed {
os::replace_multipart_source_metadata(&path, bytes, base, root)?;
}
os::sync_multipart_owned_record_parents_options(source, parent, existed, sync_source_root, base, root)
if sync_parents {
os::sync_multipart_owned_record_parents_options(source, parent, existed, sync_source_root, base, root)?;
}
Ok(())
}
pub async fn replace_head(&self, bytes: &[u8]) -> Result<()> {
self.multipart_index
@@ -2024,11 +2080,41 @@ impl NativeMultipartPublication {
}
let setup_res: std::io::Result<()> = (|| {
if let Some((original_bytes, directory, anchor_bytes)) = deferred_initial {
SourceStore::write_owned_backup_std(source, base, root, backup_owner, false, &original_bytes, false)?;
SourceStore::write_owned_backup_inner_std(
source,
base,
root,
backup_owner,
false,
&original_bytes,
false,
true,
old_metadata.is_none(),
true,
)?;
if let Some(previous) = old_metadata {
SourceStore::write_owned_backup_std(source, base, root, backup_owner, true, previous, false)?;
SourceStore::write_owned_backup_inner_std(
source,
base,
root,
backup_owner,
true,
previous,
false,
true,
true,
true,
)?;
}
SourceStore::persist_state_std(source, base, root, &next_state_raw, terminal.as_deref(), false)?;
SourceStore::persist_state_with_guard_std(
source,
base,
root,
&next_state_raw,
terminal.as_deref(),
false,
Some(directory_guard),
)?;
let _parent = os::mkdir_all_below_existing_base_std(&directory, base, root)?;
os::replace_multipart_source_metadata(&directory.join(RETIREMENT_ANCHOR), &anchor_bytes, base, root)?;
#[cfg(target_os = "linux")]
@@ -2037,9 +2123,28 @@ impl NativeMultipartPublication {
}
} else {
if let Some(previous) = old_metadata {
SourceStore::write_owned_backup_std(source, base, root, backup_owner, true, previous, false)?;
SourceStore::write_owned_backup_inner_std(
source,
base,
root,
backup_owner,
true,
previous,
false,
false,
true,
true,
)?;
}
SourceStore::persist_state_std(source, base, root, &next_state_raw, terminal.as_deref(), false)?;
SourceStore::persist_state_with_guard_std(
source,
base,
root,
&next_state_raw,
terminal.as_deref(),
false,
Some(directory_guard),
)?;
}
Ok(())
})();
@@ -2269,6 +2374,7 @@ impl NativeMultipartPublication {
target_already_synced: bool,
fused_in_place: bool,
staging_cleanup: Option<(PathBuf, Arc<std::sync::atomic::AtomicBool>)>,
directory_guard: Option<&os::RenameCommitGuard>,
) -> Result<MultipartPublicationResult> {
let mut state = self.state.lock().clone();
let expected_state_raw = encode(&state)?;
@@ -2354,7 +2460,15 @@ impl NativeMultipartPublication {
})?;
}
}
SourceStore::persist_state_std(source, base, root, &next_state_raw, terminal.as_deref(), !fused_in_place)?;
SourceStore::persist_state_with_guard_std(
source,
base,
root,
&next_state_raw,
terminal.as_deref(),
!fused_in_place,
directory_guard,
)?;
if let Some((io_path, cleaned)) = staging_cleanup {
match std::fs::remove_dir(&io_path) {
Ok(()) => cleaned.store(true, std::sync::atomic::Ordering::Release),
@@ -2387,7 +2501,7 @@ impl NativeMultipartPublication {
) -> Result<MultipartPublicationResult> {
let this = Arc::clone(self);
os::run_blocking_namespace_operation_with_sync(target_lease, true, move || {
this.finish_on_thread(target_already_synced, false, staging_cleanup)
this.finish_on_thread(target_already_synced, false, staging_cleanup, None)
.map_err(IoError::other)
})
.await
@@ -455,8 +455,11 @@ impl LocalDisk {
}
let meta_name = format!("part.{}.meta", part.number);
let actual = if let Some(cached_part) =
super::multipart_source_state::get_active_multipart_source_part(&source_object, part.number)
{
super::multipart_source_state::get_active_multipart_source_part_with_head(
&source_object,
part.number,
from_active_cache,
) {
cached_part
} else {
if pinned_source_data.is_none() {
+95 -45
View File
@@ -1288,6 +1288,28 @@ pub(crate) fn fsync_dir_limited_std(dir: &Path) -> io::Result<()> {
fsync_dir_std(dir)
}
pub(crate) fn fsync_commit_ancestor_directory_limited_std(
dir: &Path,
base_dir: &Path,
step_from_destination: usize,
guard: &RenameCommitGuard,
) -> io::Result<()> {
#[cfg(target_os = "linux")]
if base_dir == guard.directories.destination_base
&& guard.directories.destination_path.ancestors().nth(1 + step_from_destination) == Some(dir)
&& let Some(ancestors) = guard.directories.destination_ancestors.get()
&& let Some(idx) = ancestors.len().checked_sub(1 + step_from_destination)
&& let Some(file) = ancestors.get(idx)
{
#[cfg(test)]
fsync_dir_recorder::record_limited(dir);
return fsync_directory_handle_std(dir, Some(file));
}
#[cfg(not(target_os = "linux"))]
let _ = (base_dir, step_from_destination, guard);
fsync_dir_limited_std(dir)
}
/// Reuse the published directory without bypassing group commit or sync admission.
pub(crate) async fn fsync_commit_directory(
dir: &Path,
@@ -4042,7 +4064,7 @@ pub(crate) fn rename_all_with_precondition_and_parent_sync_std(
#[cfg(all(test, not(windows)))]
prepared_publication_test_hooks::run_rename_destination(source, destination);
#[cfg(all(any(test, feature = "test-util"), not(windows)))]
{
if prepared_publication_test_hooks::has_any_hooks() {
prepared_publication_test_hooks::run(prepared_publication_test_hooks::Stage::Rename, source);
prepared_publication_test_hooks::run(prepared_publication_test_hooks::Stage::Rename, destination);
if let Some(check) = precondition {
@@ -5992,6 +6014,7 @@ struct LinuxCommitDirectories {
destination_path: PathBuf,
source: LinuxCommitDirectory,
destination: std::sync::OnceLock<LinuxCommitDirectory>,
destination_ancestors: std::sync::OnceLock<Vec<std::fs::File>>,
destination_base: PathBuf,
}
@@ -6008,6 +6031,8 @@ impl LinuxCommitDirectories {
.ok_or_else(|| io::Error::other("missing destination directory guard"))?,
);
let directory = LinuxCommitDirectory::new(directory)?;
let ancestors = parents.into_iter().map(std::fs::File::from).collect();
let _ = self.destination_ancestors.set(ancestors);
Ok(self.destination.get_or_init(|| directory))
}
@@ -6017,7 +6042,7 @@ impl LinuxCommitDirectories {
.ok_or_else(|| io::Error::new(io::ErrorKind::InvalidInput, "file parent does not match its commit guard"))
}
fn validate(&self) -> io::Result<()> {
fn validate(&self, check_destination: bool) -> io::Result<()> {
use std::os::unix::fs::MetadataExt;
// A pinned directory may have been unlinked or replaced. Do not report a
// successful publication into a detached object tree.
@@ -6026,8 +6051,9 @@ impl LinuxCommitDirectories {
return Err(io::Error::new(io::ErrorKind::InvalidData, "commit directory identity changed"));
}
let destination = self.prepare_destination()?;
if (self.source.device, self.source.inode) != (destination.device, destination.inode)
|| self.source_path.as_os_str() != self.destination_path.as_os_str()
if check_destination
&& ((self.source.device, self.source.inode) != (destination.device, destination.inode)
|| self.source_path.as_os_str() != self.destination_path.as_os_str())
{
let current = self.root.open_directory(&self.destination_path)?.metadata()?;
if (current.dev(), current.ino()) != (destination.device, destination.inode) {
@@ -6041,8 +6067,9 @@ impl LinuxCommitDirectories {
record_native_publication(destination);
let source_name = self.file_name(source, true)?;
let destination_name = self.file_name(destination, false)?;
let destination_was_prepared = self.destination.get().is_some();
let destination_parent = self.prepare_destination()?;
self.validate()?;
self.validate(destination_was_prepared)?;
let mut attempt = 0;
loop {
if let Some(prepared) = prepared {
@@ -6264,7 +6291,7 @@ pub(crate) fn multipart_payload_commit_guard_std(
source_parent: &Path,
) -> io::Result<RenameCommitGuard> {
let original = &metadata_guard.directories;
original.validate()?;
original.validate(true)?;
let source = LinuxCommitDirectory::new(original.root.open_directory(source_parent)?)?;
let destination = original.prepare_destination()?;
// Clone the pinned file identity, never recapture the target pathname
@@ -6281,6 +6308,7 @@ pub(crate) fn multipart_payload_commit_guard_std(
destination_path: original.destination_path.clone(),
source,
destination: std::sync::OnceLock::from(destination),
destination_ancestors: std::sync::OnceLock::new(),
destination_base: original.destination_base.clone(),
}),
})
@@ -6362,6 +6390,7 @@ pub(crate) fn prepare_rename_commit_guard(
destination_path: destination_parent.to_path_buf(),
source,
destination,
destination_ancestors: std::sync::OnceLock::new(),
destination_base: destination_base.to_path_buf(),
}),
})
@@ -7447,6 +7476,40 @@ pub(crate) fn pin_multipart_source_directory(
}
}
fn same_parent_commit_guard(
parent: &Path,
base: &Path,
publication_root: &PublicationRoot,
directory_guard: Option<&RenameCommitGuard>,
) -> io::Result<RenameCommitGuard> {
#[cfg(target_os = "linux")]
if let Some(directory_guard) = directory_guard.filter(|g| g.directories.source_path == parent) {
let dir = &directory_guard.directories.source;
return Ok(RenameCommitGuard {
directories: Arc::new(LinuxCommitDirectories {
root: directory_guard.directories.root.clone(),
source_path: parent.to_path_buf(),
destination_path: parent.to_path_buf(),
source: LinuxCommitDirectory {
file: Arc::clone(&dir.file),
device: dir.device,
inode: dir.inode,
},
destination: std::sync::OnceLock::from(LinuxCommitDirectory {
file: Arc::clone(&dir.file),
device: dir.device,
inode: dir.inode,
}),
destination_ancestors: std::sync::OnceLock::new(),
destination_base: base.to_path_buf(),
}),
});
}
#[cfg(not(target_os = "linux"))]
let _ = directory_guard;
prepare_rename_commit_guard(parent, parent, base, publication_root)
}
/// A small control record always gets its own inode and an atomic replacement.
/// The caller owns the upload SourceRW and keeps its namespace lease alive.
pub(crate) fn replace_multipart_source_metadata(
@@ -7454,11 +7517,21 @@ pub(crate) fn replace_multipart_source_metadata(
bytes: &[u8],
base: &Path,
publication_root: &PublicationRoot,
) -> io::Result<()> {
replace_multipart_source_metadata_with_guard(path, bytes, base, publication_root, None)
}
pub(crate) fn replace_multipart_source_metadata_with_guard(
path: &Path,
bytes: &[u8],
base: &Path,
publication_root: &PublicationRoot,
directory_guard: Option<&RenameCommitGuard>,
) -> io::Result<()> {
let parent = path.parent().ok_or_else(|| io::Error::other("MPU record has no parent"))?;
#[cfg(not(target_os = "linux"))]
let _pinned = pin_multipart_source_directory(parent, base, publication_root)?;
let guard = prepare_rename_commit_guard(parent, parent, base, publication_root)?;
let guard = same_parent_commit_guard(parent, base, publication_root, directory_guard)?;
let staged = parent.join(format!(".mpu-record-{}", uuid::Uuid::new_v4()));
let mut prepared = create_prepared_rename_source_with_commit_guard(&staged, path, &guard)?;
prepared.write_all(bytes, true)?;
@@ -7707,37 +7780,9 @@ pub(crate) fn install_multipart_prepared_head(
if original_path.parent() != Some(parent) || original_path == canonical_head {
return Err(io::Error::new(io::ErrorKind::InvalidInput, "MPU proxy must be a private sibling"));
}
#[cfg(target_os = "linux")]
let guard = if let Some(directory_guard) = directory_guard.filter(|g| g.directories.source_path == parent) {
let dir = &directory_guard.directories.source;
RenameCommitGuard {
directories: Arc::new(LinuxCommitDirectories {
root: directory_guard.directories.root.clone(),
source_path: parent.to_path_buf(),
destination_path: parent.to_path_buf(),
source: LinuxCommitDirectory {
file: Arc::clone(&dir.file),
device: dir.device,
inode: dir.inode,
},
destination: std::sync::OnceLock::from(LinuxCommitDirectory {
file: Arc::clone(&dir.file),
device: dir.device,
inode: dir.inode,
}),
destination_base: base.to_path_buf(),
}),
}
} else {
prepare_rename_commit_guard(parent, parent, base, publication_root)?
};
#[cfg(not(target_os = "linux"))]
let _pinned = pin_multipart_source_directory(parent, base, publication_root)?;
#[cfg(not(target_os = "linux"))]
let guard = {
let _ = directory_guard;
prepare_rename_commit_guard(parent, parent, base, publication_root)?
};
let guard = same_parent_commit_guard(parent, base, publication_root, directory_guard)?;
rename_prepared_source_std(&prepared, &original_path, canonical_head, base, publication_root, &guard)?;
prepared.path = canonical_head.to_path_buf();
fsync_commit_directory_limited_std(parent, &guard)?;
@@ -8039,8 +8084,7 @@ pub(crate) fn hardlink_multipart_source_files(
}
#[cfg(unix)]
{
use rustix::fs::{AtFlags, Mode, OFlags, linkat, openat, unlinkat};
use std::os::unix::fs::MetadataExt;
use rustix::fs::{AtFlags, FileType, linkat, statat, unlinkat};
#[cfg(test)]
let source_path = source.to_path_buf();
#[cfg(test)]
@@ -8055,20 +8099,26 @@ pub(crate) fn hardlink_multipart_source_files(
.ok_or_else(|| io::Error::other("missing multipart attempt directory"))?;
#[cfg(test)]
prepared_publication_test_hooks::run_hardlink_destination(&source_path, &destination_path);
let flags = OFlags::RDONLY | OFlags::NOFOLLOW | OFlags::CLOEXEC;
for name in names {
if cancelled() {
return Err(io::Error::new(io::ErrorKind::Interrupted, "multipart staging cancelled"));
}
let original = std::fs::File::from(openat(source, name.as_str(), flags, Mode::empty())?);
let expected = original.metadata()?;
if !expected.is_file() {
let expected = statat(source, name.as_str(), AtFlags::SYMLINK_NOFOLLOW)?;
let expected_type = FileType::from_raw_mode(expected.st_mode);
if expected_type.is_symlink() {
return Err(io::Error::from_raw_os_error(libc::ELOOP));
}
if !expected_type.is_file() {
return Err(io::Error::new(io::ErrorKind::InvalidData, "multipart shard is not a regular file"));
}
linkat(source, name.as_str(), destination, name.as_str(), AtFlags::empty())?;
let linked = std::fs::File::from(openat(destination, name.as_str(), flags, Mode::empty())?);
let actual = linked.metadata()?;
if actual.dev() != expected.dev() || actual.ino() != expected.ino() {
let actual = statat(destination, name.as_str(), AtFlags::SYMLINK_NOFOLLOW)?;
if !FileType::from_raw_mode(actual.st_mode).is_file()
|| actual.st_dev != expected.st_dev
|| actual.st_ino != expected.st_ino
|| actual.st_size != expected.st_size
|| actual.st_mtime != expected.st_mtime
{
let _ = unlinkat(destination, name.as_str(), AtFlags::empty());
return Err(io::Error::new(io::ErrorKind::InvalidData, "multipart shard changed during staging"));
}
+263 -1
View File
@@ -94,6 +94,7 @@ use super::{
delete_tier_mutation_intent_record_if_current, list_tier_coordinator_mutation_intent_records,
list_tier_mutation_intent_records, load_tier_coordinator_mutation_intent_record_with_etag,
load_tier_mutation_intent_record_with_etag, save_tier_coordinator_mutation_intent_record_if_absent,
save_tier_mutation_intent_record_if_current,
},
warm_backend::WarmBackendImpl,
};
@@ -621,7 +622,11 @@ struct CommittedMutationRecoveryPlan {
impl TierMutationRecoverySnapshot {
fn requires_config_lock(&self) -> bool {
self.has_committed_mutation_blocks || self.intents.iter().any(|recovered| recovered.has_coordinator_record)
self.has_committed_mutation_blocks
|| self.intents.iter().any(|recovered| {
recovered.has_coordinator_record
|| (recovered.has_peer_record && recovered.intent.state == TierMutationIntentState::Prepared)
})
}
fn allowance(&self) -> MutationBlockAllowance {
@@ -5103,6 +5108,9 @@ impl TierConfigMgr {
Self::reconcile_terminal_dual_prefix_mutation_intents(api.clone(), &mut snapshot).await?;
Self::recover_prepared_coordinator_mutation_intents(api.clone(), &mut snapshot.coordinator_intents).await?;
if config_lock.is_some() {
Self::recover_expired_peer_only_prepares(api.clone(), &snapshot).await?;
}
snapshot = Self::load_and_reconcile_mutation_recovery_snapshot(handle, api.clone(), true).await?;
if snapshot.requires_config_lock() && config_lock.is_none() {
@@ -5503,6 +5511,58 @@ impl TierConfigMgr {
}
}
async fn recover_expired_peer_only_prepares<S>(api: Arc<S>, snapshot: &TierMutationRecoverySnapshot) -> io::Result<()>
where
S: EcstoreObjectIO,
{
let now = i64::try_from(OffsetDateTime::now_utc().unix_timestamp_nanos()).unwrap_or(i64::MAX);
let orphaned = snapshot
.intents
.iter()
.filter(|recovered| {
recovered.has_peer_record
&& !recovered.has_coordinator_record
&& recovered.intent.state == TierMutationIntentState::Prepared
&& Self::tier_mutation_tombstone_retention_elapsed(&recovered.intent, now)
})
.map(|recovered| recovered.intent.clone())
.collect::<Vec<_>>();
if orphaned.is_empty() {
return Ok(());
}
// The caller holds the tier-config namespace lock. Only the exact
// persisted candidate proves a commit; an old or different config
// leaves the peer fence intact.
let loaded = load_tier_config_for_recovery(api.clone()).await?;
let Some(config_etag) = loaded.etag else {
return Ok(());
};
for intent in orphaned {
if loaded.persisted_digest != Some(intent.candidate_digest) {
continue;
}
let _mutation_guard = acquire_tier_mutation_mutex(intent.mutation_id).await;
let (mut current, record_etag) =
match load_tier_mutation_intent_record_with_etag(api.clone(), intent.mutation_id).await {
Ok(record) => record,
Err(Error::ConfigNotFound) => continue,
Err(err) => return Err(tier_mutation_replay_error(err)),
};
if current != intent {
continue;
}
current
.advance(TierMutationIntentState::Committed, Some(config_etag.clone()))
.map_err(io::Error::other)?;
match save_tier_mutation_intent_record_if_current(api.clone(), &current, &record_etag).await {
Ok(()) | Err(Error::PreconditionFailed) => {}
Err(err) => return Err(tier_mutation_replay_error(err)),
}
}
Ok(())
}
async fn recover_prepared_coordinator_mutation_intents<S>(api: Arc<S>, intents: &mut [TierMutationIntent]) -> io::Result<()>
where
S: EcstoreObjectIO,
@@ -13415,6 +13475,208 @@ mod tests {
);
}
#[tokio::test]
async fn reload_recovers_expired_peer_only_prepare_for_persisted_add() {
use crate::services::tier::tier_mutation_intent::save_tier_mutation_intent_record;
let store = Arc::new(CasConfigStore::default());
let previous = empty_mgr();
let mut candidate = empty_mgr();
candidate.tiers.insert("COLD-A".to_string(), build_rustfs_tier("COLD-A"));
let targets = build_tier_mutation_affected_targets(
TierMutationIntentKind::Add,
HashSet::from(["COLD-A".to_string()]),
&previous,
&candidate,
)
.expect("added tier should have a target");
let mut intent = build_coordinator_tier_mutation_intent(TierMutationIntentKind::Add, None, &candidate, targets)
.expect("intent should build")
.expect("add should require an intent");
intent.expires_at_unix_nanos = 1;
candidate
.save_tiering_config_if_current(store.clone(), None)
.await
.expect("candidate should persist");
save_tier_mutation_intent_record(store.clone(), &intent)
.await
.expect("peer prepare should persist without a coordinator record");
let manager = TierConfigMgr::new();
TIER_MUTATION_TEST_PEERS
.scope(Vec::new(), async {
TIER_DRIVER_TEST_FACTORY
.scope(healthy_driver_factory(), async {
TierConfigMgr::reload_handle_with(&manager, store.clone())
.await
.expect("matching saved candidate should recover");
TierConfigMgr::acquire_operation_lease(&manager, "COLD-A")
.await
.expect("recovered tier should be readable");
})
.await;
})
.await;
assert!(
TierConfigMgr::load_tier_mutation_intents(store)
.await
.expect("peer scan should succeed")
.iter()
.all(|record| record.state != TierMutationIntentState::Prepared),
"the orphaned prepare must no longer block later reloads"
);
}
#[tokio::test]
async fn reload_keeps_peer_only_prepare_without_commit_proof() {
use crate::services::tier::tier_mutation_intent::{
save_tier_mutation_intent_record, tier_mutation_intent_record_object_name,
};
for case in [
"unexpired",
"recently_expired",
"different_candidate",
"missing_config",
"concurrent_change",
] {
let store = Arc::new(CasConfigStore::default());
let previous = empty_mgr();
let mut candidate = empty_mgr();
candidate.tiers.insert("COLD-A".to_string(), build_rustfs_tier("COLD-A"));
let targets = build_tier_mutation_affected_targets(
TierMutationIntentKind::Add,
HashSet::from(["COLD-A".to_string()]),
&previous,
&candidate,
)
.expect("added tier should have a target");
let mut intent = build_coordinator_tier_mutation_intent(TierMutationIntentKind::Add, None, &candidate, targets)
.expect("intent should build")
.expect("add should require an intent");
intent.expires_at_unix_nanos = match case {
"unexpired" => tier_mutation_intent_expiry_unix_nanos(),
"recently_expired" => {
i64::try_from(OffsetDateTime::now_utc().unix_timestamp_nanos()).expect("test clock should fit")
- 1_000_000_000
}
_ => 1,
};
if case == "different_candidate" {
intent.candidate_digest = [0x51; 32];
}
if case != "missing_config" {
candidate
.save_tiering_config_if_current(store.clone(), None)
.await
.expect("candidate should persist");
}
save_tier_mutation_intent_record(store.clone(), &intent)
.await
.expect("peer prepare should persist");
let expected = if case == "concurrent_change" {
let mut changed = intent.clone();
changed.candidate_digest = [0x52; 32];
let object = tier_mutation_intent_record_object_name(intent.mutation_id).expect("record path should build");
store
.rewrite_on_next_if_match(object, changed.encode().expect("changed intent should encode"))
.await;
changed
} else {
intent
};
let manager = TierConfigMgr::new();
TIER_MUTATION_TEST_PEERS
.scope(Vec::new(), async {
TierConfigMgr::reload_handle_with(&manager, store.clone())
.await
.expect("unproven prepare should remain fenced");
})
.await;
let error = match TierConfigMgr::acquire_operation_lease(&manager, "COLD-A").await {
Ok(_) => panic!("{case}: unproven prepare must block reads"),
Err(error) => error,
};
assert!(error.message.contains("being replaced"), "{case}: {error}");
let records = TierConfigMgr::load_tier_mutation_intents(store)
.await
.expect("peer scan should succeed");
assert_eq!(records, vec![expected], "{case}: prepared evidence must remain unchanged");
}
}
#[tokio::test]
async fn peer_only_prepare_recovery_retains_fence_until_peer_commit_succeeds() {
use crate::services::tier::tier_mutation_intent::save_tier_mutation_intent_record;
let store = Arc::new(CasConfigStore::default());
let previous = empty_mgr();
let mut candidate = empty_mgr();
candidate.tiers.insert("COLD-A".to_string(), build_rustfs_tier("COLD-A"));
let targets = build_tier_mutation_affected_targets(
TierMutationIntentKind::Add,
HashSet::from(["COLD-A".to_string()]),
&previous,
&candidate,
)
.expect("added tier should have a target");
let mut intent = build_coordinator_tier_mutation_intent(TierMutationIntentKind::Add, None, &candidate, targets)
.expect("intent should build")
.expect("add should require an intent");
intent.expires_at_unix_nanos = 1;
candidate
.save_tiering_config_if_current(store.clone(), None)
.await
.expect("candidate should persist");
save_tier_mutation_intent_record(store.clone(), &intent)
.await
.expect("peer prepare should persist");
let manager = TierConfigMgr::new();
let calls = Arc::new(Mutex::new(Vec::new()));
TIER_MUTATION_TEST_PEERS
.scope(vec![FakeTierMutationPeer::boxed("peer-a", calls.clone(), Err("offline"))], async {
TierConfigMgr::reload_handle_with(&manager, store.clone())
.await
.expect_err("failed peer commit must leave recovery incomplete");
})
.await;
let retained = TierConfigMgr::load_tier_mutation_intents(store.clone())
.await
.expect("peer scan should succeed");
assert_eq!(retained.len(), 1);
assert_eq!(retained[0].state, TierMutationIntentState::Committed);
let error = match TierConfigMgr::acquire_operation_lease(&manager, "COLD-A").await {
Ok(_) => panic!("failed peer commit must retain the fence"),
Err(error) => error,
};
assert!(error.message.contains("being replaced"), "{error}");
TIER_MUTATION_TEST_PEERS
.scope(
vec![FakeTierMutationPeer::boxed(
"peer-a",
calls.clone(),
Ok(PeerTierMutationState::Committed),
)],
async {
TIER_DRIVER_TEST_FACTORY
.scope(healthy_driver_factory(), async {
TierConfigMgr::reload_handle_with(&manager, store.clone())
.await
.expect("peer commit retry should recover");
TierConfigMgr::acquire_operation_lease(&manager, "COLD-A")
.await
.expect("tier should be readable after the retry");
})
.await;
},
)
.await;
assert_eq!(lock_unpoisoned(&calls).len(), 2, "recovery should retry the peer commit");
}
#[tokio::test]
async fn coordinator_mutation_recovery_scan_blocks_new_operation_lease() {
let store = Arc::new(CasConfigStore::default());
+13
View File
@@ -4608,6 +4608,19 @@ fn marker_purge_receipt_path(
}
impl SetDisks {
/// The first disk slot UUID is assigned when this erasure set is formatted.
/// It survives pool reordering and replacement of a disk in that slot.
pub fn persistent_set_id(&self) -> Result<Uuid> {
self.format
.erasure
.sets
.get(self.set_index)
.and_then(|disks| disks.first())
.copied()
.filter(|id| !id.is_nil())
.ok_or_else(|| Error::other("erasure set has no persistent format identity"))
}
/// Persist a quorum DELETE receipt outside the object marker itself. The
/// receipt is written to the metadata bucket so it survives a node restart
/// while an inaccessible member still holds the stale marker.
+22 -18
View File
@@ -720,6 +720,7 @@ impl ECStore {
},
other => StorageError::Lock(other),
})?;
let ns_guard = Arc::new(ns_guard);
let recover = Box::pin(async {
self.ensure_bucket_volume_absent_on_all_sets(bucket).await?;
@@ -1207,15 +1208,16 @@ impl ECStore {
Ok(())
}
async fn cleanup_bucket_usage(&self, bucket: &str, guard: Option<&rustfs_lock::NamespaceLockGuard>) -> Result<()> {
run_bucket_usage_cleanup(guard, bucket, async {
crate::data_usage::prepare_bucket_usage_for_namespace_change(bucket, guard).await?;
async fn cleanup_bucket_usage(&self, bucket: &str, guard: Option<&Arc<rustfs_lock::NamespaceLockGuard>>) -> Result<()> {
let namespace_guard = guard.map(Arc::as_ref);
run_bucket_usage_cleanup(namespace_guard, bucket, async {
crate::data_usage::prepare_bucket_usage_for_namespace_change(bucket, namespace_guard).await?;
crate::data_usage::remove_bucket_usage_from_backend_with_guard_fenced(self, bucket, guard).await
})
.await
}
async fn cleanup_bucket_usage_best_effort(&self, bucket: &str, guard: Option<&rustfs_lock::NamespaceLockGuard>) {
async fn cleanup_bucket_usage_best_effort(&self, bucket: &str, guard: Option<&Arc<rustfs_lock::NamespaceLockGuard>>) {
if let Err(err) = self.cleanup_bucket_usage(bucket, guard).await {
warn!(
bucket = %bucket,
@@ -1347,6 +1349,7 @@ impl ECStore {
} else {
None
};
let ns_guard = ns_guard.map(Arc::new);
let existing_bucket_info = match self.get_bucket_info_from_sets(bucket, &BucketOptions::default()).await {
Ok(info) => Some(info),
@@ -1494,7 +1497,7 @@ impl ECStore {
};
await_bucket_lifecycle_operation(
bucket_lifecycle_guard.as_ref(),
ns_guard.as_ref(),
ns_guard.as_deref(),
bucket,
"bucket metadata creation intent",
await_bucket_namespace_operation(
@@ -1520,7 +1523,7 @@ impl ECStore {
}
if bucket_lifecycle_guard.as_ref().is_some_and(|guard| guard.is_lock_lost())
|| metadata_transaction_guard.as_ref().is_some_and(|guard| guard.is_lock_lost())
|| ns_guard.as_ref().is_some_and(|guard| guard.is_lock_lost())
|| ns_guard.as_deref().is_some_and(|guard| guard.is_lock_lost())
{
if let Ok(sys) = metadata_sys::require_bucket_metadata_sys_in(&self.ctx) {
let sys = sys.read().await.clone();
@@ -1549,7 +1552,7 @@ impl ECStore {
let mut continuation_publication_started = false;
if let Err(err) = await_bucket_lifecycle_operation(
bucket_lifecycle_guard.as_ref(),
ns_guard.as_ref(),
ns_guard.as_deref(),
bucket,
"physical bucket creation",
await_bucket_namespace_operation(
@@ -1608,13 +1611,13 @@ impl ECStore {
{
warn!("best-effort bucket heal after BucketExists failed: {heal_err}");
}
if confirmed_missing && !is_err_bucket_exists(&err) && ns_guard.as_ref().is_none_or(|guard| !guard.is_lock_lost()) {
if confirmed_missing && !is_err_bucket_exists(&err) && ns_guard.as_deref().is_none_or(|guard| !guard.is_lock_lost()) {
error!("make bucket failed: {err}");
self.rollback_failed_bucket_creation(
bucket,
bucket_lifecycle_guard.as_ref(),
metadata_transaction_guard.as_ref(),
ns_guard.as_ref(),
ns_guard.as_deref(),
)
.await;
}
@@ -1636,14 +1639,14 @@ impl ECStore {
.await;
let metadata_lock_lost = bucket_lifecycle_guard.as_ref().is_some_and(|guard| guard.is_lock_lost())
|| metadata_transaction_guard.as_ref().is_some_and(|guard| guard.is_lock_lost())
|| ns_guard.as_ref().is_some_and(|guard| guard.is_lock_lost());
|| ns_guard.as_deref().is_some_and(|guard| guard.is_lock_lost());
if let Err(err) = metadata_result {
if !metadata_lock_lost {
self.rollback_failed_bucket_creation(
bucket,
bucket_lifecycle_guard.as_ref(),
metadata_transaction_guard.as_ref(),
ns_guard.as_ref(),
ns_guard.as_deref(),
)
.await;
}
@@ -2007,6 +2010,7 @@ impl ECStore {
} else {
None
};
let ns_guard = ns_guard.map(Arc::new);
let sr_mark_delete = opts.srdelete_op == SRBucketDeleteOp::MarkDelete;
let sr_purge = opts.srdelete_op == SRBucketDeleteOp::Purge;
@@ -2042,7 +2046,7 @@ impl ECStore {
crate::bucket::metadata_sys::get_object_lock_config_and_incarnation_from_disk_with_guard_in(
&self.ctx,
bucket,
Some(ns_guard.as_ref().unwrap_or(&guard)),
Some(ns_guard.as_deref().unwrap_or(&guard)),
)
.await?;
match state {
@@ -2068,7 +2072,7 @@ impl ECStore {
if sr_mark_delete {
await_bucket_lifecycle_operation(
bucket_lifecycle_guard.as_ref(),
ns_guard.as_ref(),
ns_guard.as_deref(),
bucket,
"bucket delete marker creation",
await_bucket_namespace_operation(
@@ -2107,7 +2111,7 @@ impl ECStore {
object_lock_metadata_guard.as_ref().map(|(guard, _)| guard),
bucket,
"Object Lock configuration fence during bucket deletion",
run_physical_bucket_deletion(ns_guard.as_ref(), bucket, async {
run_physical_bucket_deletion(ns_guard.as_deref(), bucket, async {
self.delete_bucket_on_sets(bucket, &delete_opts)
.await
.map_err(|err| to_object_err(err, vec![bucket]))
@@ -2136,12 +2140,12 @@ impl ECStore {
if let Some(guard) = bucket_lifecycle_guard.as_ref() {
record_opts.add_bucket_lifecycle_lock_guard(guard);
}
if let Some(guard) = ns_guard.as_ref() {
if let Some(guard) = ns_guard.as_deref() {
record_opts.add_namespace_lock_guard(guard);
}
await_bucket_lifecycle_operation(
bucket_lifecycle_guard.as_ref(),
ns_guard.as_ref(),
ns_guard.as_deref(),
bucket,
"bucket retirement publication",
crate::bucket::retirement::commit_retirement(store.clone(), bucket, incarnation, &record_opts),
@@ -2151,7 +2155,7 @@ impl ECStore {
// V1 ledger must not block quota admission after same-name recreation.
await_bucket_lifecycle_operation(
bucket_lifecycle_guard.as_ref(),
ns_guard.as_ref(),
ns_guard.as_deref(),
bucket,
"retired bucket quota ledger cleanup",
crate::bucket::quota::reservation::cleanup_retired_bucket_ledger(store, bucket, incarnation),
@@ -2162,7 +2166,7 @@ impl ECStore {
self.cleanup_bucket_usage_best_effort(bucket, ns_guard.as_ref()).await;
if let Err(err) = self
.cleanup_deleted_bucket_metadata(bucket, sr_purge, ns_guard.as_ref())
.cleanup_deleted_bucket_metadata(bucket, sr_purge, ns_guard.as_deref())
.await
{
warn!(
+14
View File
@@ -1122,6 +1122,20 @@ impl ECStore {
Some(epoch)
}
/// Namespace cleanup edits an existing usage baseline; it does not certify
/// a scan. Retain the movement fence without requiring scanning to resume.
pub(crate) async fn data_usage_namespace_cleanup_guard(&self) -> Option<(tokio::sync::OwnedRwLockReadGuard<()>, u64)> {
let operation_guard = self.ctx.data_movement_operation_gate().read_owned().await;
if self.ctx.data_movement_operation_epoch_exhausted() || self.ctx.data_movement_generation_exhausted() {
return None;
}
Some((operation_guard, self.ctx.data_movement_operation_epoch()))
}
pub(crate) async fn data_usage_namespace_cleanup_epoch(&self) -> Option<u64> {
self.data_usage_namespace_cleanup_guard().await.map(|(_, epoch)| epoch)
}
/// Acquire a storage-owned read admission for a coordinator's final
/// scanner publication. The guard remains in the context's lease table,
/// so a local movement writer cannot pass the peer while its authoritative
+372 -1
View File
@@ -4941,6 +4941,87 @@ impl ECStore {
Ok((idx, result))
}
/// Compare the logical revision across all pools before publishing a
/// namespace-only usage update to a writable pool. A suspended source is
/// retained for decommission to reconcile against the newer target.
pub(crate) async fn put_usage_snapshot_for_namespace_cleanup(
&self,
object: &str,
expected_revision: Option<&str>,
data: &mut PutObjReader,
namespace_guard: Option<&rustfs_lock::NamespaceLockGuard>,
) -> Result<ObjectInfo> {
// The caller owns the bucket namespace and movement guards. Acquire
// the fixed object domain, then every physical domain in pool order.
let guards = self
.acquire_all_physical_object_write_locks("usage_namespace_cleanup", RUSTFS_META_BUCKET, object)
.await?;
let mut opts = ObjectOptions {
no_lock: true,
max_parity: true,
write_completion: crate::object_api::WriteCompletion::TailDrained,
suppress_read_repair: true,
..Default::default()
};
for guard in &guards {
guard.add_namespace_lock_fence(&mut opts);
}
if let Some(guard) = namespace_guard {
opts.add_namespace_lock_guard(guard);
}
let source = match self
.get_latest_object_info_for_conditional_write(RUSTFS_META_BUCKET, object, &opts)
.await
.and_then(|(info, idx)| resolve_latest_object_access(RUSTFS_META_BUCKET, object, info, idx, &opts))
{
Ok((info, _)) => Some(info),
Err(err) if is_err_object_not_found(&err) || is_err_version_not_found(&err) => None,
Err(err) => return Err(err),
};
if source.as_ref().and_then(|info| info.etag.as_deref()) != expected_revision {
return Err(StorageError::PreconditionFailed);
}
if source.as_ref().is_some_and(|info| {
info.version_id.is_some_and(|version| !version.is_nil()) || info.etag.as_ref().is_none_or(String::is_empty)
}) {
return Err(StorageError::other("usage cleanup requires an unversioned snapshot with a revision"));
}
let target = self.get_pool_idx_no_lock(RUSTFS_META_BUCKET, object, data.size()).await?;
let target_info = match self.pools[target].get_object_info(RUSTFS_META_BUCKET, object, &opts).await {
Ok(info) => Some(info),
Err(err) if is_err_object_not_found(&err) || is_err_version_not_found(&err) => None,
Err(err) => return Err(err),
};
opts.http_preconditions = Some(match target_info.as_ref() {
Some(info) => crate::storage_api_contracts::object::HTTPPreconditions {
if_match: Some(
info.etag
.as_ref()
.filter(|etag| !etag.is_empty())
.cloned()
.ok_or_else(|| StorageError::other("usage cleanup target has no revision"))?,
),
..Default::default()
},
None => crate::storage_api_contracts::object::HTTPPreconditions {
if_none_match: Some("*".to_string()),
..Default::default()
},
});
let latest = source
.iter()
.chain(target_info.iter())
.filter_map(|info| info.mod_time)
.fold(OffsetDateTime::now_utc(), std::cmp::max);
opts.mod_time = Some(
latest
.checked_add(time::Duration::nanoseconds(1))
.ok_or_else(|| StorageError::other("usage cleanup snapshot timestamp is exhausted"))?,
);
opts.decommission_capacity_admission = crate::bucket::metadata_sys::object_store_if_initialized_in(&self.ctx).await;
self.pools[target].put_object(RUSTFS_META_BUCKET, object, data, &opts).await
}
#[instrument(level = "debug", skip(self, data))]
#[hotpath::measure(impl_type = "ECStore")]
pub(super) async fn handle_put_object(
@@ -7009,7 +7090,7 @@ mod tests {
};
use crate::set_disk::same_distributed_lock_domain;
use crate::set_disk::{SetDisks, disk_call_counters};
use crate::storage_api_contracts::bucket::MakeBucketOptions;
use crate::storage_api_contracts::bucket::{BucketOptions, MakeBucketOptions};
use crate::storage_api_contracts::lifecycle::TransitionedObject;
use crate::storage_api_contracts::list::ListOperations as _;
use bytes::Bytes;
@@ -10351,6 +10432,296 @@ mod tests {
multipool_delete_test_store(bucket, true).await
}
#[tokio::test]
async fn create_fresh_bucket_after_canceled_decommission() {
let (_dirs, store) = multipool_version_test_store("canceled-decommission-existing").await;
let mut canceled = prepared_pool_test_status(1, false);
canceled.decommission = Some(PoolDecommissionInfo {
canceled: true,
..Default::default()
});
*store.pool_meta.write().await = PoolMeta {
pools: vec![prepared_pool_test_status(0, false), canceled],
..Default::default()
};
store
.handle_make_bucket("canceled-decommission-fresh", &MakeBucketOptions::default())
.await
.expect("canceled decommission must not block creation of a new bucket");
assert!(
store.scanner_data_usage_publication_blocked().await,
"scanner publication must remain fenced"
);
}
#[tokio::test]
async fn create_bucket_after_failed_decommission() {
let (_dirs, store) = multipool_version_test_store("decommission-existing-bucket").await;
let bucket = "decommission-new-bucket";
let snapshot_time = std::time::SystemTime::now() - Duration::from_secs(60);
let mut snapshot = rustfs_data_usage::DataUsageInfo {
last_update: Some(snapshot_time),
scanner_cycle: Some(37),
scanner_epoch: Some(2),
buckets_count: 1,
usage_snapshot_complete: true,
..Default::default()
};
snapshot.buckets_usage.insert(
bucket.to_string(),
rustfs_data_usage::BucketUsageInfo {
size: 42,
objects_count: 1,
versions_count: 1,
..Default::default()
},
);
snapshot.bucket_sizes.insert(bucket.to_string(), 42);
snapshot.calculate_totals();
crate::data_usage::store_data_usage_in_backend(snapshot, store.clone())
.await
.expect("store a stale bucket-usage baseline before the failed decommission");
let (old_snapshot, usage_owner) = store
.get_latest_accessible_object_info_with_idx(
RUSTFS_META_BUCKET,
crate::data_usage::DATA_USAGE_OBJ_NAME_PATH.as_str(),
&ObjectOptions::default(),
)
.await
.expect("identify the pool holding the old usage baseline");
let mut pool_statuses = vec![prepared_pool_test_status(0, false), prepared_pool_test_status(1, false)];
pool_statuses[usage_owner].decommission = Some(PoolDecommissionInfo {
failed: true,
..Default::default()
});
*store.pool_meta.write().await = PoolMeta {
pools: pool_statuses,
..Default::default()
};
store
.handle_make_bucket(bucket, &MakeBucketOptions::default())
.await
.expect("a failed decommission must not block creation of an unrelated bucket");
store
.handle_get_bucket_info(bucket, &BucketOptions::default())
.await
.expect("the created bucket must be visible");
let usage = crate::data_usage::load_data_usage_from_backend(store.clone())
.await
.expect("load the cleaned usage baseline");
assert!(!usage.buckets_usage.contains_key(bucket), "the new bucket must not inherit old usage");
assert_eq!(usage.scanner_cycle, Some(37), "namespace cleanup must not certify a new scan");
assert_eq!(usage.scanner_epoch, Some(2), "namespace cleanup must preserve the scanner authority");
let object = crate::data_usage::DATA_USAGE_OBJ_NAME_PATH.as_str();
let (_, current_owner) = store
.get_latest_accessible_object_info_with_idx(RUSTFS_META_BUCKET, object, &ObjectOptions::default())
.await
.expect("read the new snapshot authority");
assert_ne!(current_owner, usage_owner, "cleanup must publish to a writable pool");
let retained_source = store.pools[usage_owner]
.get_object_info(RUSTFS_META_BUCKET, object, &ObjectOptions::default())
.await
.expect("decommission retains its original source for reconciliation");
assert_eq!(retained_source.etag, old_snapshot.etag);
let stale_update = store
.put_usage_snapshot_for_namespace_cleanup(
object,
old_snapshot.etag.as_deref(),
&mut PutObjReader::from_vec(b"stale snapshot".to_vec()),
None,
)
.await;
assert!(matches!(stale_update, Err(StorageError::PreconditionFailed)));
let usage = crate::data_usage::load_data_usage_from_backend(store.clone())
.await
.expect("a stale source CAS must not replace the writable target");
assert!(!usage.buckets_usage.contains_key(bucket));
assert!(
store.scanner_data_usage_publication_blocked().await,
"scanner publication must remain fenced"
);
}
#[tokio::test]
async fn canceled_create_retains_usage_cleanup_namespace_owner() {
use crate::set_disk::{PutObjectCommitBarrier, PutObjectCommitPause};
let (_dirs, store) = multipool_version_test_store("cleanup-cancellation-existing").await;
let bucket = "cleanup-cancellation-new";
let mut snapshot = rustfs_data_usage::DataUsageInfo {
last_update: Some(std::time::SystemTime::now()),
usage_snapshot_complete: true,
..Default::default()
};
snapshot.buckets_usage.insert(
bucket.to_owned(),
rustfs_data_usage::BucketUsageInfo {
size: 42,
objects_count: 1,
versions_count: 1,
..Default::default()
},
);
snapshot.bucket_sizes.insert(bucket.to_owned(), 42);
snapshot.calculate_totals();
crate::data_usage::store_data_usage_in_backend(snapshot, store.clone())
.await
.expect("seed the stale namespace usage");
let object = crate::data_usage::DATA_USAGE_OBJ_NAME_PATH.as_str();
let (_, source) = store
.get_latest_accessible_object_info_with_idx(RUSTFS_META_BUCKET, object, &ObjectOptions::default())
.await
.expect("find the source snapshot");
let mut pools = vec![prepared_pool_test_status(0, false), prepared_pool_test_status(1, false)];
pools[source].decommission = Some(PoolDecommissionInfo {
canceled: true,
..Default::default()
});
*store.pool_meta.write().await = PoolMeta {
pools,
..Default::default()
};
let barrier = PutObjectCommitBarrier::install(RUSTFS_META_BUCKET, object, PutObjectCommitPause::AfterNamespace);
let create_store = store.clone();
let create = tokio::spawn(async move { create_store.handle_make_bucket(bucket, &MakeBucketOptions::default()).await });
tokio::time::timeout(Duration::from_secs(10), barrier.wait_until_paused())
.await
.expect("cleanup reaches its physical commit with the namespace held");
create.abort();
assert!(create.await.expect_err("the request waiter was canceled").is_cancelled());
let namespace = store
.new_ns_lock(bucket, bucket)
.await
.expect("open the bucket namespace lock");
assert!(
tokio::time::timeout(Duration::from_millis(50), namespace.get_write_lock(Duration::from_secs(1)))
.await
.is_err(),
"cancellation must not release the namespace before its cleanup finishes"
);
barrier.release();
drop(barrier);
tokio::time::timeout(Duration::from_secs(10), store.handle_make_bucket(bucket, &MakeBucketOptions::default()))
.await
.expect("the resumed create must not remain blocked")
.expect("a new request resumes after the old cleanup releases its namespace");
let usage = crate::data_usage::load_data_usage_from_backend(store.clone())
.await
.expect("read the cleaned baseline");
assert!(!usage.buckets_usage.contains_key(bucket));
assert!(store.scanner_data_usage_publication_blocked().await);
}
#[tokio::test]
async fn usage_cleanup_rejects_an_unreadable_pool() {
let (_dirs, store) = multipool_version_test_store("cleanup-unreadable-pool").await;
let snapshot = rustfs_data_usage::DataUsageInfo {
last_update: Some(std::time::SystemTime::now()),
usage_snapshot_complete: true,
..Default::default()
};
crate::data_usage::store_data_usage_in_backend(snapshot, store.clone())
.await
.expect("seed the usage baseline");
let object = crate::data_usage::DATA_USAGE_OBJ_NAME_PATH.as_str();
let (before, source) = store
.get_latest_accessible_object_info_with_idx(RUSTFS_META_BUCKET, object, &ObjectOptions::default())
.await
.expect("find the baseline owner");
let unavailable = 1 - source;
for disk in store.pools[unavailable]
.get_disks_by_key(object)
.disks
.read()
.await
.iter()
.flatten()
{
disk.close().await.expect("stop disk health monitoring");
disk.force_offline_for_test();
}
let update = store
.put_usage_snapshot_for_namespace_cleanup(
object,
before.etag.as_deref(),
&mut PutObjReader::from_vec(b"unproven authority".to_vec()),
None,
)
.await;
assert!(update.is_err(), "an unreadable pool may hold a newer snapshot");
let after = store.pools[source]
.get_object_info(RUSTFS_META_BUCKET, object, &ObjectOptions::default())
.await
.expect("the readable baseline remains intact");
assert_eq!(after.etag, before.etag);
}
#[tokio::test]
async fn usage_cleanup_rejects_namespace_loss_before_physical_commit() {
use crate::set_disk::{PutObjectCommitBarrier, PutObjectCommitPause};
use rustfs_lock::{LocalClient, LockRequest, LockType, NamespaceLock, ObjectKey};
let (_dirs, store) = multipool_version_test_store("cleanup-fence-existing").await;
let snapshot = rustfs_data_usage::DataUsageInfo {
last_update: Some(std::time::SystemTime::now()),
usage_snapshot_complete: true,
..Default::default()
};
crate::data_usage::store_data_usage_in_backend(snapshot, store.clone())
.await
.expect("seed the protected baseline");
let object = crate::data_usage::DATA_USAGE_OBJ_NAME_PATH.as_str();
let (before, _) = store
.get_latest_accessible_object_info_with_idx(RUSTFS_META_BUCKET, object, &ObjectOptions::default())
.await
.expect("read the baseline revision");
let ttl = Duration::from_secs(60);
let namespace = NamespaceLock::new("usage-cleanup-commit-fence".to_owned(), Arc::new(LocalClient::new()));
let request = LockRequest::new(ObjectKey::new("cleanup-fence-existing", ""), LockType::Exclusive, "usage-cleanup")
.with_acquire_timeout(Duration::from_secs(1))
.with_ttl(ttl)
.with_refresh_interval(ttl);
let guard = Arc::new(
namespace
.acquire_guard(&request)
.await
.expect("acquire the namespace lease")
.expect("namespace lease is available"),
);
let barrier = PutObjectCommitBarrier::install(RUSTFS_META_BUCKET, object, PutObjectCommitPause::AfterNamespace);
let update_store = store.clone();
let update_guard = guard.clone();
let revision = before.etag.clone();
let update = tokio::spawn(async move {
let _movement = update_store
.data_usage_namespace_cleanup_guard()
.await
.expect("admit the namespace-only update");
update_store
.put_usage_snapshot_for_namespace_cleanup(
object,
revision.as_deref(),
&mut PutObjReader::from_vec(b"must not commit".to_vec()),
Some(update_guard.as_ref()),
)
.await
});
tokio::time::timeout(Duration::from_secs(10), barrier.wait_until_paused())
.await
.expect("the update reaches its commit barrier");
tokio::time::pause();
tokio::time::advance(ttl + Duration::from_millis(1)).await;
assert!(guard.is_lock_lost(), "the parent namespace lease has expired");
tokio::time::resume();
barrier.release();
assert!(update.await.expect("the cleanup task completes").is_err());
let (after, _) = store
.get_latest_accessible_object_info_with_idx(RUSTFS_META_BUCKET, object, &ObjectOptions::default())
.await
.expect("the original baseline remains readable");
assert_eq!(after.etag, before.etag, "a lost namespace lease must fence the physical write");
}
async fn multipool_delete_test_store(bucket: &str, versioned: bool) -> (Vec<tempfile::TempDir>, Arc<ECStore>) {
let ctx = Arc::new(crate::runtime::instance::InstanceContext::new());
let (mut dirs, first_set) = make_local_set_disks_with_ctx(4, 2, Arc::clone(&ctx)).await;
+119
View File
@@ -1873,4 +1873,123 @@ mod tests {
assert_eq!(err.kind(), std::io::ErrorKind::InvalidData);
assert!(err.to_string().contains("invalid checksum value"), "unexpected error: {err}");
}
fn software_crc32_bitwise(reflected_poly: u32, data: &[u8]) -> [u8; 4] {
let mut crc = !0u32;
for &b in data {
crc ^= u32::from(b);
for _ in 0..8 {
let mask = (crc & 1).wrapping_neg();
crc = (crc >> 1) ^ (reflected_poly & mask);
}
}
(!crc).to_be_bytes()
}
fn software_crc64_nvme_bitwise(data: &[u8]) -> [u8; 8] {
let reflected_poly = super::CRC64_NVME_POLYNOMIAL.reverse_bits();
let mut crc = !0u64;
for &b in data {
crc ^= u64::from(b);
for _ in 0..8 {
let mask = (crc & 1).wrapping_neg();
crc = (crc >> 1) ^ (reflected_poly & mask);
}
}
(!crc).to_be_bytes()
}
#[test]
fn rio_crc_simd_matches_bitwise_scalar_across_tail_and_vector_boundaries() {
for &size in &[
0usize,
1,
7,
8,
15,
16,
17,
31,
32,
33,
63,
64,
65,
127,
128,
129,
255,
256,
257,
1024,
4096 + 17,
64 * 1024,
] {
let payload: Vec<u8> = (0..size).map(|i| ((i * 31 + 17) % 251) as u8).collect();
let crc32_simd = Checksum::new_from_data(ChecksumType::CRC32, &payload).expect("crc32");
let crc32c_simd = Checksum::new_from_data(ChecksumType::CRC32C, &payload).expect("crc32c");
let crc64_simd = Checksum::new_from_data(ChecksumType::CRC64_NVME, &payload).expect("crc64nvme");
let crc32_sw = software_crc32_bitwise(0xEDB88320, &payload);
let crc32c_sw = software_crc32_bitwise(0x82F63B78, &payload);
let crc64_sw = software_crc64_nvme_bitwise(&payload);
assert_eq!(crc32_simd.raw.as_slice(), &crc32_sw, "CRC32 SIMD vs scalar mismatch at {size}B");
assert_eq!(crc32c_simd.raw.as_slice(), &crc32c_sw, "CRC32C SIMD vs scalar mismatch at {size}B");
assert_eq!(crc64_simd.raw.as_slice(), &crc64_sw, "CRC64_NVME SIMD vs scalar mismatch at {size}B");
}
}
#[test]
#[ignore = "utility throughput microbenchmark; run via cargo test -p rustfs-rio --lib --release -- --ignored --nocapture"]
fn microbench_rio_checksum_simd_vs_scalar_and_all_base_algorithms() {
use std::hint::black_box;
use std::io::Write as _;
use std::time::Instant;
for &(size, iters) in &[(1024usize, 100usize), (64 * 1024, 20), (1024 * 1024, 4)] {
let payload: Vec<u8> = (0..size).map(|i| ((i * 31 + 17) % 251) as u8).collect();
let total_mib = (size * iters) as f64 / (1024.0 * 1024.0);
// Verify SIMD-accelerated CRC32 / CRC32C / CRC64_NVME against bitwise scalar reference at 1 KiB & 64 KiB.
if size <= 64 * 1024 {
let crc32_simd = Checksum::new_from_data(ChecksumType::CRC32, &payload).expect("crc32");
let crc32c_simd = Checksum::new_from_data(ChecksumType::CRC32C, &payload).expect("crc32c");
let crc64_simd = Checksum::new_from_data(ChecksumType::CRC64_NVME, &payload).expect("crc64nvme");
let t_sw = Instant::now();
let crc32_sw = software_crc32_bitwise(0xEDB88320, black_box(&payload));
let crc32c_sw = software_crc32_bitwise(0x82F63B78, black_box(&payload));
let crc64_sw = software_crc64_nvme_bitwise(black_box(&payload));
let sw_elapsed = t_sw.elapsed();
assert_eq!(crc32_simd.raw.as_slice(), &crc32_sw, "CRC32 SIMD vs scalar mismatch at {size}B");
assert_eq!(crc32c_simd.raw.as_slice(), &crc32c_sw, "CRC32C SIMD vs scalar mismatch at {size}B");
assert_eq!(crc64_simd.raw.as_slice(), &crc64_sw, "CRC64_NVME SIMD vs scalar mismatch at {size}B");
println!(
"[microbench][rio::checksum] size={size}B bitwise scalar reference check passed in {:?}",
sw_elapsed
);
}
for &checksum_type in super::BASE_CHECKSUM_TYPES {
let mut hasher = checksum_type.hasher().expect("base checksum type must have hasher");
let t0 = Instant::now();
let mut digest = Vec::new();
for _ in 0..iters {
hasher.reset();
hasher.write_all(black_box(&payload)).expect("hasher write_all");
digest = hasher.finalize();
black_box(&digest);
}
let elapsed = t0.elapsed();
assert_eq!(digest.len(), checksum_type.raw_byte_len());
let mibs = total_mib / elapsed.as_secs_f64().max(1e-9);
println!(
"[microbench][rio::checksum] size={size}B algo={} throughput={mibs:.1} MiB/s",
checksum_type
);
}
}
}
}
+396 -51
View File
@@ -690,6 +690,19 @@ pub(super) enum ScannerPauseBacklogAttemptDecision {
pub struct ScannerPauseBacklogReplicaId {
pub pool_index: usize,
pub set_index: usize,
/// Stable format identity; indices are only the current routing location.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub set_id: Option<uuid::Uuid>,
}
impl ScannerPauseBacklogReplicaId {
fn from_set(set: &SetDisks) -> Result<Self, EcstoreError> {
Ok(Self {
pool_index: set.pool_index,
set_index: set.set_index,
set_id: Some(set.persistent_set_id()?),
})
}
}
#[derive(Clone, Debug, Deserialize, PartialEq, Eq, Serialize)]
@@ -706,9 +719,23 @@ impl ScannerPauseBacklogCommitRecord {
fn validate(&self) -> Result<(), String> {
self.ledger.validate()?;
if self.replicas.is_empty() || !self.replicas.windows(2).all(|pair| pair[0] < pair[1]) {
if self.replicas.is_empty()
|| !self
.replicas
.windows(2)
.all(|pair| (pair[0].pool_index, pair[0].set_index) < (pair[1].pool_index, pair[1].set_index))
{
return Err("scanner pause backlog commit has invalid replica membership".to_string());
}
let mut identities = BTreeSet::new();
if self
.replicas
.iter()
.filter_map(|id| id.set_id)
.any(|id| id.is_nil() || !identities.insert(id))
{
return Err("scanner pause backlog commit has invalid persistent set identities".to_string());
}
Ok(())
}
}
@@ -747,6 +774,10 @@ impl ScannerPauseBacklogReplicaRecord {
}
if let Some(committed) = &self.committed {
committed.validate()?;
if committed.replicas.iter().any(|id| id.set_id.is_some()) && committed.replicas.iter().any(|id| id.set_id.is_none())
{
return Err("scanner pause backlog membership mixes positional and persistent identities".to_string());
}
if let Some(stable) = &self.stable {
let stable_key = (stable.writer_epoch, stable.generation);
let committed_key = (committed.ledger.writer_epoch, committed.ledger.generation);
@@ -907,9 +938,19 @@ fn unix_now() -> u64 {
}
async fn read_scanner_pause_backlog_replica(store: Arc<SetDisks>) -> ScannerPauseBacklogReplica {
let id = ScannerPauseBacklogReplicaId {
pool_index: store.pool_index,
set_index: store.set_index,
let id = match ScannerPauseBacklogReplicaId::from_set(&store) {
Ok(id) => id,
Err(error) => {
return ScannerPauseBacklogReplica {
id: ScannerPauseBacklogReplicaId {
pool_index: store.pool_index,
set_index: store.set_index,
set_id: None,
},
revision: None,
state: ScannerPauseBacklogReplicaState::Unavailable(error.to_string()),
};
}
};
let reader = match store
.get_object_reader(
@@ -1333,8 +1374,48 @@ fn select_scanner_pause_backlog_roll_forward(replicas: &[ScannerPauseBacklogRepl
.then(|| newest.clone())
}
/// Resolve stored membership against immutable set identities before checking
/// commit proofs. A missing identity is never rebound to a new set at its old
/// index. Legacy records retain their positional interpretation until an
/// existing commit or rollback proof permits publishing the bound membership.
fn normalize_scanner_pause_backlog_membership(
replicas: &mut [ScannerPauseBacklogReplica],
) -> Result<(), ScannerPauseBacklogSelectionError> {
let mut by_location = HashMap::with_capacity(replicas.len());
let mut by_identity = HashMap::with_capacity(replicas.len());
for replica in replicas.iter() {
let id = replica.id;
if by_location.insert((id.pool_index, id.set_index), id).is_some()
|| id
.set_id
.is_some_and(|identity| identity.is_nil() || by_identity.insert(identity, id).is_some())
{
return Err(ScannerPauseBacklogSelectionError::AuthorityConflict(
"scanner pause backlog has duplicate or invalid storage identities".to_string(),
));
}
}
for replica in replicas {
if let ScannerPauseBacklogReplicaState::Valid(record) = &mut replica.state
&& let Some(commit) = record.committed.as_mut()
{
for member in &mut commit.replicas {
let current = match member.set_id {
Some(identity) => by_identity.get(&identity),
None => by_location.get(&(member.pool_index, member.set_index)),
};
if let Some(current) = current {
*member = *current;
}
}
commit.replicas.sort_unstable();
}
}
Ok(())
}
fn select_scanner_pause_backlog_replicas(
replicas: Vec<ScannerPauseBacklogReplica>,
mut replicas: Vec<ScannerPauseBacklogReplica>,
) -> Result<LoadedScannerPauseBacklog, ScannerPauseBacklogSelectionFailure> {
if replicas.is_empty() {
return Err(scanner_pause_backlog_selection_failure(
@@ -1354,6 +1435,9 @@ fn select_scanner_pause_backlog_replicas(
)),
));
}
if let Err(error) = normalize_scanner_pause_backlog_membership(&mut replicas) {
return Err(scanner_pause_backlog_selection_failure(replicas, error));
}
let replica_ids = scanner_pause_backlog_replica_ids(&replicas);
let authoritative_commit = match select_scanner_pause_backlog_commit(&replicas, &replica_ids) {
@@ -1512,12 +1596,13 @@ fn decode_scanner_pause_backlog_retirement_replicas(
) -> Result<(Vec<ScannerPauseBacklogReplica>, Vec<ScannerPauseBacklogReplica>), ScannerPauseBacklogRetirementError> {
let mut ids = BTreeSet::new();
let mut decoded = Vec::with_capacity(replicas.len());
let mut surviving = Vec::new();
let mut writable_ids = BTreeSet::new();
let mut has_source = false;
for replica in replicas {
let id = ScannerPauseBacklogReplicaId {
pool_index: replica.pool_index,
set_index: replica.set_index,
set_id: replica.set_id,
};
if !ids.insert(id) {
return Err(scanner_pause_backlog_retirement_invalid_record(
@@ -1548,7 +1633,7 @@ fn decode_scanner_pause_backlog_retirement_replicas(
state,
};
if replica.pool_index != source_pool_index && replica.writable {
surviving.push(decoded_replica.clone());
writable_ids.insert(id);
}
decoded.push(decoded_replica);
}
@@ -1557,6 +1642,12 @@ fn decode_scanner_pause_backlog_retirement_replicas(
"scanner pause backlog retirement has no native source record",
));
}
normalize_scanner_pause_backlog_membership(&mut decoded).map_err(scanner_pause_backlog_retirement_selection_error)?;
let surviving = decoded
.iter()
.filter(|replica| writable_ids.contains(&replica.id))
.cloned()
.collect();
Ok((decoded, surviving))
}
@@ -1874,11 +1965,9 @@ where
let loaded_ids = loaded.replicas.iter().map(|replica| replica.id).collect::<BTreeSet<_>>();
let writable_ids = writable
.iter()
.map(|set| ScannerPauseBacklogReplicaId {
pool_index: set.pool_index,
set_index: set.set_index,
})
.collect::<BTreeSet<_>>();
.map(|set| ScannerPauseBacklogReplicaId::from_set(set))
.collect::<Result<BTreeSet<_>, _>>()
.map_err(|error| error.to_string())?;
if writable_ids != loaded_ids {
return Err("scanner pause backlog replica topology changed during commit".to_string());
}
@@ -1901,10 +1990,7 @@ where
let results = tokio::spawn(async move {
let mut results = Vec::with_capacity(writable.len());
for set in writable {
let id = ScannerPauseBacklogReplicaId {
pool_index: set.pool_index,
set_index: set.set_index,
};
let id = ScannerPauseBacklogReplicaId::from_set(&set).map_err(|error| error.to_string())?;
if only.as_ref().is_some_and(|only| !only.contains(&id)) {
continue;
}
@@ -1928,10 +2014,10 @@ where
break;
}
}
results
Ok::<_, String>(results)
})
.await
.map_err(|err| format!("scanner pause backlog publication owner failed: {err}"))?;
.map_err(|err| format!("scanner pause backlog publication owner failed: {err}"))??;
let failures = results
.iter()
@@ -2444,6 +2530,142 @@ mod tests {
(root, store)
}
#[test]
#[serial_test::serial]
fn native_legacy_upgrade_and_pool_reindex_preserve_backlog_authority() {
assert_native_legacy_upgrade_and_pool_reindex_preserve_backlog_authority(false);
}
#[test]
#[serial_test::serial]
fn native_pool_reindex_with_failed_membership_publication_preserves_authority() {
assert_native_legacy_upgrade_and_pool_reindex_preserve_backlog_authority(true);
}
fn assert_native_legacy_upgrade_and_pool_reindex_preserve_backlog_authority(fail_publication: bool) {
run_native_retirement_test(async move || {
use crate::storage_api::owner::NativeScannerPauseBacklogWriteFault;
let (root, store) = native_pause_backlog_store(3, 1).await;
let now = unix_now();
let original = seed_native_expansion_ledger(&store, now).await;
// Model bytes written by the previous positional-membership writer.
for id in native_replica_ids(&store) {
let (bytes, _) = native_replica_bytes(&store.pools[id.pool_index].disk_set[0]).await;
let ScannerPauseBacklogReplicaState::Valid(mut record) = decode_scanner_pause_backlog_ledger(&bytes) else {
panic!("native seed is valid");
};
record.replica_schema_version = 1;
for member in &mut record.committed.as_mut().expect("seed commit").replicas {
member.set_id = None;
}
record.validate().expect("previous schema fixture");
replace_native_pause_backlog_replica(&store, id, *record).await;
}
let fault =
NativeScannerPauseBacklogWriteFault::fail_before_write(Arc::clone(&store.pools[1].disk_set[0]), "publish", 1);
assert!(
ScannerPauseBacklogController::claim(Arc::clone(&store), now + 60)
.await
.is_err()
);
drop(fault);
let interrupted = load_scanner_pause_backlog(Arc::clone(&store))
.await
.expect("mixed legacy and bound records retain the rollback proof");
assert_eq!(interrupted.ledger, original);
let upgraded = ScannerPauseBacklogController::claim(Arc::clone(&store), now + 61)
.await
.expect("retry finishes the format upgrade");
let expected = upgraded.loaded.ledger.clone();
drop(upgraded);
for pool in &store.pools {
let (bytes, _) = native_replica_bytes(&pool.disk_set[0]).await;
let ScannerPauseBacklogReplicaState::Valid(record) = decode_scanner_pause_backlog_ledger(&bytes) else {
panic!("upgraded native record is valid");
};
assert_eq!(record.replica_schema_version, 1);
assert!(
record
.committed
.expect("upgraded membership")
.replicas
.iter()
.all(|id| id.set_id.is_some())
);
}
let surviving_id = native_replica_id(&store, 2, 0).set_id;
let (source, _) = native_replica_bytes(&store.pools[1].disk_set[0]).await;
store
.prepare_scanner_pause_backlog_retirement_for_test(1, source.len() * 2)
.await
.expect("start retiring the middle pool");
store
.retire_scanner_pause_backlog_for_test(1, 0)
.await
.expect("hand off the native backlog");
store
.complete_scanner_pause_backlog_retirement_for_test(1)
.await
.expect("persist the completed middle pool through the ILM receipt checks");
shutdown_native_retirement_store(store).await;
let reordered = super::super::tests::setup_scanner_cycle_store_with_pool_order(
root.path(),
false,
&[0, 2, 3],
1,
NATIVE_RETIREMENT_DRIVES_PER_SET,
false,
)
.await;
assert_eq!(native_replica_id(&reordered, 1, 0).set_id, surviving_id);
assert_ne!(native_replica_id(&reordered, 2, 0).set_id, surviving_id);
let loaded = load_scanner_pause_backlog(Arc::clone(&reordered))
.await
.expect("removing a middle pool and appending a new one preserves native authority");
assert_eq!(loaded.ledger, expected);
assert_eq!(loaded.persistence_state, "membership_repair_pending");
if fail_publication {
// Match the reported partial publication: the two existing
// pools reject the new commit while the replacement accepts it.
let faults = [0, 1].map(|pool| {
NativeScannerPauseBacklogWriteFault::fail_before_write(
Arc::clone(&reordered.pools[pool].disk_set[0]),
"publish",
2,
)
});
let error = ScannerPauseBacklogController::claim(Arc::clone(&reordered), now + 120)
.await
.err()
.expect("the incomplete publication must fail");
assert!(error.contains("pool 0 set 0") && error.contains("pool 1 set 0"), "{error}");
drop(faults);
let recovered = load_scanner_pause_backlog(Arc::clone(&reordered))
.await
.expect("partial publication after reindexing retains proven authority");
assert_eq!(recovered.ledger, expected);
assert!(recovered.durable);
}
let (source, _) = native_replica_bytes(&reordered.pools[1].disk_set[0]).await;
reordered
.prepare_scanner_pause_backlog_retirement_for_test(1, source.len() * 2)
.await
.expect("retire the reindexed member before a scanner writer completes the membership change");
reordered
.retire_scanner_pause_backlog_for_test(1, 0)
.await
.expect("retirement uses the same bound authority");
let surviving = load_scanner_pause_backlog(Arc::clone(&reordered))
.await
.expect("remaining members retain the complete ledger");
assert_eq!(surviving.ledger, expected);
assert!(surviving.all_current_replicas_stable());
shutdown_native_retirement_store(reordered).await;
});
}
#[test]
#[serial_test::serial]
fn native_single_pool_partial_commit_converges_across_24_sets() {
@@ -2592,9 +2814,7 @@ mod tests {
);
let stable = durable_ledger(unix_now());
let members = (0..3)
.flat_map(|pool_index| (0..2).map(move |set_index| replica_id(pool_index, set_index)))
.collect::<Vec<_>>();
let members = native_replica_ids(&store);
let committed = ScannerPauseBacklogCommitRecord::new(stable.clone(), members);
let record = ScannerPauseBacklogReplicaRecord::new(Some(stable), Some(committed));
let bytes = encode_scanner_pause_backlog_record(&record).expect("stable replica should encode");
@@ -3021,10 +3241,7 @@ mod tests {
let now = unix_now().saturating_add(60);
let replacement = claim_scanner_pause_backlog_writer(&original, now).expect("next writer generation");
let new_commit = ScannerPauseBacklogCommitRecord::new(replacement.clone(), current_ids.clone());
let failed_id = ScannerPauseBacklogReplicaId {
pool_index: failed_pool,
set_index: 0,
};
let failed_id = native_replica_id(&store, failed_pool, 0);
let fault = NativeScannerPauseBacklogWriteFault::fail_before_write(
Arc::clone(&store.pools[failed_pool].disk_set[0]),
"publish",
@@ -3553,10 +3770,8 @@ mod tests {
.authoritative_commit
.expect("old cohort commit");
let now = unix_now().saturating_add(60);
let ids = (0..3)
.flat_map(|pool_index| (0..2).map(move |set_index| replica_id(pool_index, set_index)))
.collect::<Vec<_>>();
let newer_member = replica_id(2, 1);
let ids = native_replica_ids(&store);
let newer_member = native_replica_id(&store, 2, 1);
let newer = claim_scanner_pause_backlog_writer(&original, now).expect("newer generation");
let newer_commit = ScannerPauseBacklogCommitRecord::new(newer.clone(), ids.clone());
for id in &ids {
@@ -3617,14 +3832,12 @@ mod tests {
let (_root, store, original) = native_expanded_backlog_store(1, false).await;
let now = unix_now().saturating_add(60);
let wedge = native_interleaved_expansion_wedge(&store, &original, now).await;
let ids = (0..3)
.flat_map(|pool_index| (0..2).map(move |set_index| replica_id(pool_index, set_index)))
.collect::<Vec<_>>();
let ids = native_replica_ids(&store);
// The retiring source holds the newest stable ledger.
let newest = claim_scanner_pause_backlog_writer(&wedge.acknowledged, now + 1).expect("newest generation");
replace_native_pause_backlog_replica(
&store,
replica_id(0, 0),
native_replica_id(&store, 0, 0),
ScannerPauseBacklogReplicaRecord::new(
Some(newest.clone()),
Some(ScannerPauseBacklogCommitRecord::new(newest.clone(), ids)),
@@ -3675,10 +3888,8 @@ mod tests {
set_runtime_error(None);
let (_root, store, original) = native_expanded_backlog_store(1, false).await;
let now = unix_now().saturating_add(60);
let ids = (0..3)
.flat_map(|pool_index| (0..2).map(move |set_index| replica_id(pool_index, set_index)))
.collect::<Vec<_>>();
let newer_member = replica_id(2, 1);
let ids = native_replica_ids(&store);
let newer_member = native_replica_id(&store, 2, 1);
let others = ids.iter().copied().filter(|id| *id != newer_member).collect::<Vec<_>>();
let middle = claim_scanner_pause_backlog_writer(&original, now).expect("middle generation");
let newest = claim_scanner_pause_backlog_writer(&middle, now + 1).expect("newest generation");
@@ -3837,10 +4048,8 @@ mod tests {
let (_root, store, original) = native_expanded_backlog_store(1, false).await;
let now = unix_now().saturating_add(60);
let wedge = native_interleaved_expansion_wedge(&store, &original, now).await;
let ids = (0..3)
.flat_map(|pool_index| (0..2).map(move |set_index| replica_id(pool_index, set_index)))
.collect::<Vec<_>>();
assert_eq!(wedge.last, replica_id(2, 1));
let ids = native_replica_ids(&store);
assert_eq!(wedge.last, native_replica_id(&store, 2, 1));
// A different stable ledger at the newest key is a fork.
let fork = claim_scanner_pause_backlog_writer(&original, now + 7).expect("forked writer generation");
@@ -3851,7 +4060,7 @@ mod tests {
assert_ne!(fork, wedge.acknowledged);
replace_native_pause_backlog_replica(
&store,
replica_id(1, 0),
native_replica_id(&store, 1, 0),
ScannerPauseBacklogReplicaRecord::new(
Some(fork.clone()),
Some(ScannerPauseBacklogCommitRecord::new(fork, ids.clone())),
@@ -3868,7 +4077,7 @@ mod tests {
"{error}"
);
assert_eq!(native_replica_snapshot(&store).await, forked, "a refused claim keeps every record");
replace_native_pause_backlog_replica(&store, replica_id(1, 0), wedge.rollback.clone()).await;
replace_native_pause_backlog_replica(&store, native_replica_id(&store, 1, 0), wedge.rollback.clone()).await;
// A missing member that a stored commit names may have lost a
// newer stable ledger.
@@ -3903,7 +4112,7 @@ mod tests {
.expect_err("retirement must refuse an unproven authority");
assert!(refused.to_string().contains("unsafe authority conflict"), "{refused}");
assert_eq!(native_replica_bytes(&store.pools[0].disk_set[0]).await.0, source_bytes);
replace_native_pause_backlog_replica(&store, replica_id(1, 1), wedge.rollback.clone()).await;
replace_native_pause_backlog_replica(&store, native_replica_id(&store, 1, 1), wedge.rollback.clone()).await;
// The same records without the conflict converge and retire.
for set in 0..2 {
@@ -4020,7 +4229,7 @@ mod tests {
.authoritative_commit
.as_ref()
.expect("seed retains the old cohort proof");
assert_eq!(committed.replicas, vec![replica_id(0, 0), replica_id(0, 1)]);
assert_eq!(committed.replicas, vec![native_replica_id(&store, 0, 0), native_replica_id(&store, 0, 1)]);
assert_eq!(loaded.replica_count, 6);
if loaded.healthy_replicas == loaded.replica_count {
break loaded;
@@ -4037,13 +4246,13 @@ mod tests {
.authoritative_commit
.as_ref()
.expect("seed retains the old cohort proof");
assert_eq!(committed.replicas, vec![replica_id(0, 0), replica_id(0, 1)]);
assert_eq!(committed.replicas, vec![native_replica_id(&store, 0, 0), native_replica_id(&store, 0, 1)]);
assert_eq!(loaded.replica_count, 6);
assert_eq!(loaded.healthy_replicas, 6);
let seeded = loaded
.replicas
.iter()
.find(|replica| replica.id == replica_id(2, 0))
.find(|replica| replica.id == native_replica_id(&store, 2, 0))
.expect("the canceled caller's admitted seed replica");
assert!(matches!(
&seeded.state,
@@ -4055,7 +4264,7 @@ mod tests {
let remaining = loaded
.replicas
.iter()
.find(|replica| replica.id == replica_id(2, 1))
.find(|replica| replica.id == native_replica_id(&store, 2, 1))
.expect("the remaining seed replica");
assert!(matches!(
&remaining.state,
@@ -4268,7 +4477,12 @@ mod tests {
.prepare_scanner_pause_backlog_retirement_for_test(0, source_before.len() * 2)
.await
.expect("retire the old cohort's first pool");
let surviving = [replica_id(1, 0), replica_id(1, 1), replica_id(2, 0), replica_id(2, 1)];
let surviving = [
native_replica_id(&store, 1, 0),
native_replica_id(&store, 1, 1),
native_replica_id(&store, 2, 0),
native_replica_id(&store, 2, 1),
];
let newer = claim_scanner_pause_backlog_writer(&old.ledger, now + 2).expect("surviving writer");
replace_native_pause_backlog_replica(
&store,
@@ -4333,7 +4547,7 @@ mod tests {
assert_eq!(loaded.ledger, original);
assert!(loaded.durable && loaded.stable_matches_ledger);
let committed = loaded.authoritative_commit.expect("survivor commit proof");
assert_eq!(committed.replicas, vec![replica_id(2, 0), replica_id(2, 1)]);
assert_eq!(committed.replicas, vec![native_replica_id(&store, 2, 0), native_replica_id(&store, 2, 1)]);
}
shutdown_native_retirement_store(store).await;
});
@@ -4911,7 +5125,135 @@ mod tests {
}
fn replica_id(pool_index: usize, set_index: usize) -> ScannerPauseBacklogReplicaId {
ScannerPauseBacklogReplicaId { pool_index, set_index }
ScannerPauseBacklogReplicaId {
pool_index,
set_index,
set_id: None,
}
}
fn native_replica_id(store: &ECStore, pool_index: usize, set_index: usize) -> ScannerPauseBacklogReplicaId {
ScannerPauseBacklogReplicaId::from_set(&store.pools[pool_index].disk_set[set_index])
.expect("native replica has an immutable format identity")
}
fn native_replica_ids(store: &ECStore) -> Vec<ScannerPauseBacklogReplicaId> {
store
.pools
.iter()
.flat_map(|pool| pool.disk_set.iter())
.map(|set| ScannerPauseBacklogReplicaId::from_set(set).expect("native replica format identity"))
.collect()
}
fn bound_replica_id(pool_index: usize, identity: u128) -> ScannerPauseBacklogReplicaId {
ScannerPauseBacklogReplicaId {
pool_index,
set_index: 0,
set_id: Some(uuid::Uuid::from_u128(identity)),
}
}
fn missing_replica(id: ScannerPauseBacklogReplicaId) -> ScannerPauseBacklogReplica {
ScannerPauseBacklogReplica {
id,
revision: Some(DataUsageCacheRevision::Missing),
state: ScannerPauseBacklogReplicaState::Missing,
}
}
#[test]
fn bound_membership_survives_pool_reindexing() {
let ledger = durable_ledger(50);
let previous = bound_replica_id(2, 22);
let current = bound_replica_id(1, 22);
let added = bound_replica_id(2, 33);
let record = replica_record_for_members(&ledger, &ledger, &[previous]);
assert_eq!(record.replica_schema_version, 1);
let loaded = select_scanner_pause_backlog_replicas(vec![
missing_replica(bound_replica_id(0, 11)),
decoded_replica(current, &record),
missing_replica(added),
])
.expect("a surviving set keeps its commit proof after its routing index changes");
assert_eq!(loaded.ledger, ledger);
assert!(loaded.durable);
assert_eq!(loaded.authoritative_commit.expect("surviving native proof").replicas, vec![current]);
let mut replicas = vec![retirement_replica(current, &record)];
for id in [bound_replica_id(0, 11), added] {
replicas.push(ScannerPauseBacklogRetirementReplica {
pool_index: id.pool_index,
set_index: id.set_index,
set_id: id.set_id,
writable: true,
data: None,
});
}
let plan = plan_scanner_pause_backlog_retirement(1, &replicas)
.expect("reindexed native authority permits safe retirement")
.expect("new survivors need the existing ledger");
assert_eq!(planned_stable_ledger(&plan), ledger);
}
#[test]
fn bound_membership_does_not_rebind_a_missing_set_to_its_reused_index() {
let ledger = durable_ledger(50);
let missing = bound_replica_id(2, 22);
let replacement = bound_replica_id(2, 33);
let record = replica_record_for_members(&ledger, &ledger, &[missing]);
assert!(
select_scanner_pause_backlog_replicas(vec![
missing_replica(bound_replica_id(0, 11)),
decoded_replica(replacement, &record),
])
.is_err(),
"copied bytes on a new set cannot impersonate the missing commit member"
);
}
#[test]
fn legacy_membership_binds_only_after_its_existing_proof_is_verified() {
let ledger = durable_ledger(50);
let legacy = replica_record_for_members(&ledger, &ledger, &[replica_id(2, 0)]);
assert_eq!(legacy.replica_schema_version, 1);
let current = bound_replica_id(2, 22);
let loaded = select_scanner_pause_backlog_replicas(vec![decoded_replica(current, &legacy)])
.expect("unchanged legacy membership remains readable");
let upgraded = stable_scanner_pause_backlog_record(
&loaded.ledger,
loaded.authoritative_commit.as_ref(),
&scanner_pause_backlog_replica_ids(&loaded.replicas),
);
upgraded
.validate()
.expect("the replacement record binds every committed member");
assert_eq!(upgraded.replica_schema_version, 1);
assert_eq!(upgraded.committed.expect("bound membership").replicas, vec![current]);
assert!(
select_scanner_pause_backlog_replicas(vec![
decoded_replica(bound_replica_id(1, 22), &legacy),
missing_replica(bound_replica_id(2, 33)),
])
.is_err(),
"an already stranded legacy proof must not guess its old member's identity"
);
}
#[test]
fn bound_membership_rejects_nil_duplicate_and_mixed_identities() {
let ledger = durable_ledger(50);
for members in [
vec![bound_replica_id(0, 0)],
vec![bound_replica_id(0, 11), bound_replica_id(1, 11)],
vec![bound_replica_id(0, 11), bound_replica_id(0, 22)],
vec![bound_replica_id(0, 11), replica_id(1, 0)],
] {
let record = replica_record_for_members(&ledger, &ledger, &members);
assert_eq!(record.replica_schema_version, 1);
assert!(record.validate().is_err());
}
}
fn decoded_replica(
@@ -5155,6 +5497,7 @@ mod tests {
.map(|set_index| ScannerPauseBacklogReplicaId {
pool_index: 0,
set_index,
set_id: None,
})
.collect::<Vec<_>>();
replica_record_for_members(stable, committed, &replicas)
@@ -5167,6 +5510,7 @@ mod tests {
ScannerPauseBacklogRetirementReplica {
pool_index: id.pool_index,
set_index: id.set_index,
set_id: id.set_id,
writable: true,
data: Some(serde_json::to_vec(record).expect("native record bytes")),
}
@@ -5294,6 +5638,7 @@ mod tests {
replicas.extend(targets.iter().map(|id| ScannerPauseBacklogRetirementReplica {
pool_index: id.pool_index,
set_index: id.set_index,
set_id: id.set_id,
writable: true,
data: None,
}));
+24 -4
View File
@@ -113,17 +113,37 @@ pub(super) async fn setup_scanner_cycle_store_at_path_with_layout_and_disk_prein
sets_per_pool: usize,
drives_per_set: usize,
preinitialize_disks: bool,
) -> Arc<ECStore> {
setup_scanner_cycle_store_with_pool_order(
root,
seed_usage_baseline,
&(0..pool_count).collect::<Vec<_>>(),
sets_per_pool,
drives_per_set,
preinitialize_disks,
)
.await
}
pub(super) async fn setup_scanner_cycle_store_with_pool_order(
root: &Path,
seed_usage_baseline: bool,
pool_order: &[usize],
sets_per_pool: usize,
drives_per_set: usize,
preinitialize_disks: bool,
) -> Arc<ECStore> {
init_ecstore_config_for_scanner_tests();
let pool_count = pool_order.len();
let mut pools = Vec::with_capacity(pool_count);
for pool_index in 0..pool_count {
for (pool_index, physical_pool) in pool_order.iter().enumerate() {
let mut endpoints = Vec::new();
for set_index in 0..sets_per_pool {
for disk_index in 0..drives_per_set {
let disk_path = if sets_per_pool == 1 {
root.join(format!("pool{pool_index}/disk{disk_index}"))
root.join(format!("pool{physical_pool}/disk{disk_index}"))
} else {
root.join(format!("pool{pool_index}/set{set_index}/disk{disk_index}"))
root.join(format!("pool{physical_pool}/set{set_index}/disk{disk_index}"))
};
tokio::fs::create_dir_all(&disk_path)
.await
@@ -144,7 +164,7 @@ pub(super) async fn setup_scanner_cycle_store_at_path_with_layout_and_disk_prein
cmd_line: if pool_count == 1 && sets_per_pool == 1 {
"scanner-cycle-metrics".to_string()
} else {
format!("scanner-cycle-metrics-pool-{pool_index}")
format!("scanner-cycle-metrics-pool-{physical_pool}")
},
platform: format!("OS: {} | Arch: {}", std::env::consts::OS, std::env::consts::ARCH),
});
+162
View File
@@ -735,4 +735,166 @@ mod tests {
}
}
}
#[test]
fn highwayhash_simd_matches_portable_scalar_and_chunked_slices_across_unaligned_boundaries() {
use highway::PortableHash;
for &size in &[
0usize,
1,
7,
15,
31,
32,
33,
63,
64,
65,
127,
128,
129,
255,
257,
1024,
4096 + 17,
64 * 1024,
] {
let payload = generate_compat_test_data(size);
let mut h_simd = HighwayHasher::new(MAGIC_HIGHWAY_HASH256_PARSED_KEY);
h_simd.append(&payload);
let simd_digest = u8x32_from_u64x4(h_simd.finalize256());
let mut h_scalar = PortableHash::new(MAGIC_HIGHWAY_HASH256_PARSED_KEY);
h_scalar.append(&payload);
let scalar_digest = u8x32_from_u64x4(h_scalar.finalize256());
assert_eq!(
simd_digest, scalar_digest,
"HighwayHasher SIMD must match PortableHash scalar byte-for-byte at {size}B"
);
for &chunk_size in &[17usize, 61, 4096] {
if size > 4096 && chunk_size < 61 {
continue;
}
let slices: Vec<&[u8]> = payload.chunks(chunk_size).collect();
for algo in [
HashAlgorithm::HighwayHash256S,
HashAlgorithm::HighwayHash256SLegacy,
HashAlgorithm::SHA256,
HashAlgorithm::BLAKE2b512,
HashAlgorithm::Md5,
] {
let contig_out = algo.hash_encode(&payload);
let sliced_out = algo.hash_encode_slices(slices.iter().copied());
assert_eq!(
contig_out.as_ref(),
sliced_out.as_ref(),
"{algo:?} contiguous vs {chunk_size}B sliced digest mismatch at {size}B"
);
}
}
}
}
#[test]
#[ignore = "utility throughput microbenchmark; run via cargo test -p rustfs-utils --lib --features hash --release -- --ignored --nocapture"]
fn microbench_highwayhash_simd_vs_portable_scalar_and_hash_algorithms() {
use highway::PortableHash;
use std::hint::black_box;
use std::time::Instant;
for &(size, iters) in &[(1024usize, 200usize), (64 * 1024, 40), (1024 * 1024, 8)] {
let payload = generate_compat_test_data(size);
// 1. Compare HighwayHasher (runtime SIMD: AVX2/SSE4.1/NEON) against PortableHash scalar reference.
let t_simd = Instant::now();
let mut simd_digest = [0u8; 32];
for _ in 0..iters {
let mut h = HighwayHasher::new(MAGIC_HIGHWAY_HASH256_PARSED_KEY);
h.append(black_box(&payload));
simd_digest = u8x32_from_u64x4(h.finalize256());
black_box(&simd_digest);
}
let simd_elapsed = t_simd.elapsed();
let t_scalar = Instant::now();
let mut scalar_digest = [0u8; 32];
for _ in 0..iters {
let mut h = PortableHash::new(MAGIC_HIGHWAY_HASH256_PARSED_KEY);
h.append(black_box(&payload));
scalar_digest = u8x32_from_u64x4(h.finalize256());
black_box(&scalar_digest);
}
let scalar_elapsed = t_scalar.elapsed();
assert_eq!(
simd_digest, scalar_digest,
"HighwayHasher SIMD must match PortableHash scalar byte-for-byte at {size}B"
);
let total_mib = (size * iters) as f64 / (1024.0 * 1024.0);
let simd_mibs = total_mib / simd_elapsed.as_secs_f64().max(1e-9);
let scalar_mibs = total_mib / scalar_elapsed.as_secs_f64().max(1e-9);
println!(
"[microbench][utils::hash] size={size}B HighwayHash256S SIMD={simd_mibs:.1} MiB/s vs PortableScalar={scalar_mibs:.1} MiB/s (speedup={:.2}x)",
simd_mibs / scalar_mibs.max(1e-9)
);
// 2. Compare contiguous hash_encode vs 4 KiB chunked hash_encode_slices across all bitrot algorithms.
let slices: Vec<&[u8]> = payload.chunks(4096).collect();
for algo in [
HashAlgorithm::HighwayHash256S,
HashAlgorithm::HighwayHash256SLegacy,
HashAlgorithm::SHA256,
HashAlgorithm::BLAKE2b512,
HashAlgorithm::Md5,
] {
let t_contig = Instant::now();
let mut contig_out = Vec::new();
for _ in 0..iters {
contig_out = algo.hash_encode(black_box(&payload)).as_ref().to_vec();
black_box(&contig_out);
}
let contig_elapsed = t_contig.elapsed();
let t_sliced = Instant::now();
let mut sliced_out = Vec::new();
for _ in 0..iters {
sliced_out = algo.hash_encode_slices(black_box(slices.iter().copied())).as_ref().to_vec();
black_box(&sliced_out);
}
let sliced_elapsed = t_sliced.elapsed();
assert_eq!(contig_out, sliced_out, "{algo:?} contiguous vs 4KiB sliced digest mismatch at {size}B");
let contig_mibs = total_mib / contig_elapsed.as_secs_f64().max(1e-9);
let sliced_mibs = total_mib / sliced_elapsed.as_secs_f64().max(1e-9);
println!(
"[microbench][utils::hash] size={size}B algo={algo:?} contiguous={contig_mibs:.1} MiB/s vs sliced(4KiB)={sliced_mibs:.1} MiB/s"
);
}
}
// 3. Key-routing microbenchmark: crc_hash (CRC32) vs sip_hash (SipHash-1-3).
let keys: Vec<String> = (0..1024)
.map(|i| format!("bucket/prefix-{i}/object-key-{i}.parquet"))
.collect();
let t_crc = Instant::now();
let mut crc_acc = 0usize;
for key in &keys {
crc_acc ^= crc_hash(black_box(key), 16);
}
let crc_ns = t_crc.elapsed().as_nanos();
let t_sip = Instant::now();
let mut sip_acc = 0usize;
for key in &keys {
sip_acc ^= sip_hash(black_box(key), 16, &[0u8; 16]);
}
black_box(sip_acc);
let sip_ns = t_sip.elapsed().as_nanos();
assert!(black_box(crc_acc) < 16 && black_box(sip_acc) < 16);
println!("[microbench][utils::hash] 1024 object keys: crc_hash={crc_ns}ns total, sip_hash={sip_ns}ns total");
}
}
+2
View File
@@ -8,6 +8,8 @@ collection:
## Operations
- [Candidate release pipeline](operations/release-pipeline.md): frozen inputs, staged acceptance, immutable publication and recovery.
- [Sampled local batch CPU diagnostics](operations/local-batch-cpu-diagnostics.md) — default-off request-correlated allocation and worker wall/CPU diagnosis.
- [Retained MRF responsibilities](operations/mrf-responsibilities.md) — unknown bucket identities, bounded probes, and unreadable checkpoint recovery.
@@ -72,6 +72,8 @@ Ordinary publication rechecks the selected pool against the durable pool metadat
Running, queued, failed, canceled, and completed decommission states exclude the source from new ordinary publication, including new multipart uploads. Previously created multipart uploads retain their drain path while the source remains non-terminal; terminal source states reject further multipart publication. Failed and canceled entries become writable for new ordinary publication only after an allowed clear operation removes that state. This check does not change repair admission or the separate fence for operations that only release capacity.
Bucket creation and deletion remove stale bucket usage through namespace maintenance, independently of the scanner publication pause. Failed or canceled decommission state alone does not reject that cleanup. The update checks the logical snapshot revision while holding the fixed and physical object locks, then conditionally publishes to a writable pool; a suspended source remains available for decommission to reconcile against the newer target. Cleanup preserves the scanner epoch and cycle, and retains the bucket namespace and movement guards until its physical writes finish even if the request is canceled. Namespace lease loss still fences the write. Scanner publication continues to require its existing movement admission.
For mixed batch deletes, only the pools selected to receive new delete markers are publication targets. Exact-version deletions on other pools remain protected by the same pool metadata read fence, without treating the retiring source or an unrelated reserved target as a destination for those markers.
### Shared Capacity On Healthy Targets
@@ -143,6 +145,10 @@ Existing capacity-owner and mutation checks reconcile the pending intent before
source cleanup; the replica exception does not clear an unknown intent, rewrite
the native target, or change the scanner's committed-membership selection.
Backlog membership also carries an optional `set_id`, the first disk-slot UUID from the erasure set's persisted format. This identity survives pool removal, reordering, and disk replacement; `pool_index` and `set_index` remain routing locations. Readers resolve bound members by that UUID before checking the existing commit and rollback proofs. A new pool at a removed member's old index cannot impersonate that member. New publications bind every member, while positional records remain readable and gain identities only after their existing authority is verified. Replica schema 1 is retained; 1.0.4 readers ignore the additive identity field and still use positional membership.
Finish upgrading all scanner-capable nodes before changing pool membership. An older writer can replace a bound record with a positional one. If a legacy record was already stranded by pool reindexing, the missing identity cannot be guessed safely: restore the known topology in which its proof was readable, then let the updated writer bind the membership before changing it again. Do not delete or reset the backlog ledger to bypass this check. Partial writes, canceled callers, missing members, and conflicting authority remain subject to the native CAS and recovery rules above.
## MinIO Divergence Decisions
Behavior that is close to MinIO but not byte-for-byte identical. Changing either decision requires an operator compatibility note and updated characterization tests.
+79
View File
@@ -0,0 +1,79 @@
# Candidate acceptance and release publication
RustFS builds the final product version before publication. A candidate is identified by its Actions run and `<version>-preview.<run_number>`; this identifier is not a Git tag or the executable's displayed version. Failed candidates consume no public version. A public tag is created only after the complete artifact manifest is approved.
## Entry points
| Workflow | Input | Result | Public writes |
|---|---|---|---|
| `release-candidate.yml` | Exact final version and full source SHA | Four Linux archives, native checks, upgrade results, Linux manifest | None |
| `release-complete.yml` | Successful candidate run, accepted Linux manifest SHA-256, backlog acceptance report | Desktop archives, DEB/RPM, OCI images, Helm chart, full certification and sealed manifest | None |
| `release-publish.yml` | Successful completion run and approved final manifest SHA-256 | Annotated tag, GitHub Release, R2 files, registry images, stable pointers and Helm index | Explicit publication |
The builder workflows (`build.yml`, `package.yml`, `docker.yml`, `helm-package.yml`) are reusable stages. Only `release-publish.yml` has release write credentials. Development builds retain their artifact names and download channel. Daily development builds now select Linux only; macOS/Windows development artifacts are refreshed by an explicit manual build, and both remain mandatory for a completed release candidate. Tag pushes no longer start a second compilation.
## Freeze and build
Select a reviewed source commit whose Cargo version equals the intended final version. Finish exact-source CI before dispatching the candidate. The candidate workflow itself must run from upstream main. Unrelated main commits do not require rebuilding an accepted source.
```bash
gh workflow run release-candidate.yml --ref main \
-f version='<version>' -f source_sha='<full-source-sha>'
```
The initial job records the source and workflow SHAs, release recipe, dependency lock, exact stable Rust version, Console tag/asset/SHA-256, CLI assets, previous stable release and validation plan. All platforms consume that record; none independently follows `latest`. The source commit must contain the same reviewed release recipe as the dispatcher. Relevant recipe changes require a new candidate; unrelated source changes on main do not invalidate the frozen record.
Linux is built first. Each archived executable must display the final version, expected target, source commit and compiler version, then start natively, serve the Console, and pass an authenticated S3 bucket/object round trip. The upgrade lane runs the existing pinned compatibility matrix plus the stable version frozen at candidate creation. It compiles the harness and runs the accepted executable; it does not rebuild a substitute server. A zero-test invocation fails.
The Linux manifest is partial evidence and cannot authorize publication. Download the exact artifacts for field acceptance. Record the source, manifest digest, run URL and actual tested binary digests in backlog. Required field acceptance includes Scanner/Heal and multi-pool behavior, major features, memory, high-concurrency 503s, performance against the agreed baseline, previous-version data integrity and upgrade recovery. The agreed stability duration is a release-specific requirement, not an implicit workflow default.
## Complete the accepted candidate
After the Linux report passes, dispatch `release-complete.yml` with the exact digest printed by the candidate run. The `release-linux-acceptance` environment is the operator gate. Configure its protection and allowed main branch in repository settings; a typed digest and report identify what is being accepted, while the workflow independently revalidates automated evidence.
```bash
gh workflow run release-complete.yml --ref main \
-f candidate_run_id='<run-id>' \
-f linux_manifest_sha256='<accepted-linux-manifest-sha256>' \
-f acceptance_report='https://github.com/rustfs/backlog/issues/<issue>#issuecomment-<comment>'
```
macOS and Windows now build and run the same native checks. Packages and images wrap the accepted Linux binaries. DEB/RPM extraction must reproduce the recorded binary digest. Both OCI variants are checked on amd64 and arm64 and scanned before sealing. Vulnerability scans retain the repository's report-only policy; scanner execution errors still fail the stage. Image OS packages are resolved during packaging, then the actual resulting OCI bytes are accepted and preserved; this is not a promise of reproducible package-manager output.
The Helm chart is tested and packaged before sealing, with chart and application versions equal to the final product version. Full certification requires every platform and upgrade case from the original plan. The seal includes artifact hashes, raw evidence, source/input identities, acceptance report, checksums, SBOM and provenance. Successful Linux receipts remain valid while desktop results are added because each receipt binds its own exact archive and executable, the frozen plan, candidate identity and trusted producer.
Native smoke checks are not a substitute for Console UI and rc acceptance. Complete the release skill's UI/client checks against these exact artifacts, including desktop checks applicable to the release, before asking for final publication approval.
## Approve and publish
Review the complete run and final `release-manifest.json`. Human approval binds its SHA-256, source and version. A changed manifest requires renewed approval. No source version bump, compilation, package generation or image build occurs during publication.
```bash
gh workflow run release-publish.yml --ref main \
-f complete_run_id='<successful-completion-run-id>' \
-f manifest_sha256='<approved-final-manifest-sha256>'
```
The publisher downloads the artifact from the exact successful main workflow run, checks GitHub's artifact digest, recomputes file hashes and reruns complete certification. It creates an annotated tag whose message binds the source and manifest. An existing tag or asset with different bytes is a hard failure; tags are never moved and uploads never clobber assets.
Versioned GitHub assets, R2 objects and Docker Hub/GHCR/Quay manifests are uploaded and read back before the Release becomes public. OCI promotion uses `--preserve-digests`; registry conversion fails rather than silently accepting a new image. R2 creation uses atomic `If-None-Match: *`. The small evidence bundle persists with the public Release after Actions artifacts expire. Provenance is descriptive, not a claim of cryptographic attestation.
Publication uses a per-version lock and a shared channel lock, both without cancelling an active run. Stable channels advance only after versioned delivery succeeds. GitHub Latest acts as a monotonic marker during partial channel updates, so an older retry cannot downgrade Docker tags, R2 aliases or installer pointers. Prereleases do not advance stable channels. Helm publishes the already accepted chart after versioned images exist. GitHub's legacy `*-latest.zip` download names remain aliases of the accepted bytes; they become default downloads only when GitHub Latest advances.
Cross-registry publication is not atomic. A channel job can fail after some verified destinations advance. Resume the same run/seal: matching destinations are reused and conflicting ones stop the workflow. Retrying never recompiles. Do not claim complete publication until the channel receipt and all public endpoints are verified.
## Failure and retention
- Infrastructure failure: rerun failed jobs in the same run. Frozen inputs, successful platform/package/image artifacts and completed seals are reused. Missing or expired artifacts fail closed.
- Product or build-input change before tagging: create a new candidate run and repeat affected acceptance. Do not mutate the old candidate.
- Failure after tagging: resume publication of exactly that seal. A code fix requires a new patch version; never move the tag.
- New main commits: retain evidence for unchanged source, recipe and inputs. Recheck only affected boundaries; do not restart the entire matrix because an unrelated PR merged.
- Artifacts are retained for 30 days while awaiting acceptance. Public-repository Actions artifacts and staging identifiers are not confidential storage. Preserve important failure evidence in task-owned storage and backlog.
## Migration and workflow maintenance
Finish any in-flight legacy tag release, including its downstream Docker/package/Helm jobs, before enabling this cutover on main. Existing tags and Releases retain their identities. Do not feed a legacy public preview binary into the new promotion path: it embeds a different version and was not built from the frozen-input contract.
All workflow responsibilities, triggers and schedule ownership remain registered in `.github/scheduled-validations.json`. The migration does not re-enable workflows disabled in GitHub. PR checks, cache warming, nightly development builds, functional chains, lab tests and monitoring retain their distinct purposes; shared source or cache does not make different feature/platform tests redundant. The watchdog allocates a runner only for relevant unsuccessful scheduled/chained runs, while its registry still determines ownership.
Run `actionlint`, the release contract suite and the workflow registry validator after changes. The Security Audit workflow also exercises actual GitHub artifact transport, DEB/RPM package extraction, both multi-platform OCI builders, local registry promotion and archive scanning. Native packaging jobs run only for relevant changes or a manual audit. Their payloads are explicit transport fixtures, not product release acceptance. See the [release skill](../../.agents/skills/rustfs-release-publish/SKILL.md) for field acceptance, human confirmation and post-release follow-up.
+2
View File
@@ -8,6 +8,8 @@ The classifier reads the pull request base revision's planner and policy. It com
The committed policy is `enforced`: ordinary Rust changes select their reverse dependency closure plus the fixed core inventory, and unrelated changes can omit the Connect specialization. Package-owned fixtures, non-Rust inputs, build scripts, manifests, unknown paths and non-PR events retain full validation. Existing documentation and isolated E2E scopes remain conservative. Changing a PR's own policy cannot authorize skipping its base revision's requirements. A base planner predating the current workflow contract clears all narrower outputs and selects the full matrix. Set the policy back to `shadow` to retain proposed selections while restoring full ordinary-code execution; do not remove core inventories or negative checks as a rollback.
Stable installation-version alignment has one content-based exception in enforced mode: literal version fields in `flake.nix`, `helm/rustfs/Chart.yaml` and `rustfs.spec`, plus a standard prepended RPM release note, can use the documentation scope. Quick Checks, Typos, CLA and the required aggregate remain. The planner compares both complete Git blobs and requires ordinary non-executable files; changes to build logic, CLI pins, chart settings, RPM macros or older changelog entries retain the full gate. New/deleted files, mode changes, prerelease syntax, unavailable history, shadow rollout and explicit full-path policy overrides also retain full validation. Product-code changes in the same PR retain their normal dependency and core coverage, and main pushes, merge groups, scheduled and manual runs still validate the complete candidate. This exception does not establish artifact availability or authorize a release; post-release checks still verify the published target before updating defaults.
`Test and Lint` remains the required aggregate name. Every verification producer and consumer is a direct dependency. The verifier accepts only the effective plan's exact expected success/skipped outcomes; failure, cancellation, missing results, extra jobs or a plan for a different candidate fail. Quick Checks and CLA retain their separate required names.
## Build and execute
+1 -1
View File
@@ -77,7 +77,7 @@
rustfs = rustPlatform.buildRustPackage {
pname = "rustfs";
version = "1.0.4";
version = "1.0.5";
src = ./.;
+2 -2
View File
@@ -2,8 +2,8 @@ apiVersion: v2
name: rustfs
description: RustFS helm chart to deploy RustFS on kubernetes cluster.
type: application
version: "1.0.4"
appVersion: "1.0.4"
version: "1.0.5"
appVersion: "1.0.5"
home: https://rustfs.com
icon: https://media.sys.truenas.net/apps/rustfs/icons/icon.svg
maintainers:
+4 -1
View File
@@ -1,7 +1,7 @@
%global _enable_debug_packages 0
%global _empty_manifest_terminate_build 0
Name: rustfs
Version: 1.0.4
Version: 1.0.5
Release: 1
Summary: High-performance distributed object storage for MinIO alternative
@@ -57,6 +57,9 @@ install %_builddir/%{name}-%{version}/target/%_arch/%_arch-unknown-linux-gnu/rel
%_bindir/rustfs
%changelog
* Wed Nov 11 2026 overtrue <anzhengchao@gmail.com>
- Update RPM package to RustFS 1.0.5
* Mon Oct 26 2026 overtrue <anzhengchao@gmail.com>
- Update RPM package to RustFS 1.0.4
+96 -27
View File
@@ -254,11 +254,15 @@ impl DefaultAdminUsecase {
/// absent from `buckets_usage` — absent already means "unknown" on the
/// wire, distinct from a present zero — and the snapshot is marked
/// non-converged so clients can tell it is not the whole namespace. A
/// snapshot that is structurally incomplete, or that measured nothing the
/// namespace still contains, carries no usable information and is dropped
/// exactly as before.
fn narrow_data_usage_snapshot_to_measured_buckets(info: &mut DataUsageInfo, buckets: impl IntoIterator<Item = String>) {
if !info.is_complete_bucket_usage_snapshot() && !info.is_valid_partial_snapshot() {
/// snapshot that is structurally incomplete (`snapshot_measured` is false),
/// or that measured nothing the namespace still contains, carries no usable
/// information and is dropped exactly as before.
fn narrow_data_usage_snapshot_to_measured_buckets(
info: &mut DataUsageInfo,
snapshot_measured: bool,
buckets: impl IntoIterator<Item = String>,
) {
if !snapshot_measured {
*info = DataUsageInfo::default();
return;
}
@@ -275,13 +279,31 @@ impl DefaultAdminUsecase {
let covers_namespace = current_buckets.len() == info.buckets_usage.len();
info.buckets_count = info.buckets_usage.len() as u64;
if !covers_namespace {
// Some bucket in the namespace was never measured, so the totals
// below are a floor rather than the converged truth.
if !covers_namespace || !info.is_complete_bucket_usage_snapshot() {
// Some bucket in the namespace was never measured, or holds
// mutations no scan has reconciled yet, so the totals below are
// not the converged truth.
info.usage_snapshot_converged = Some(false);
}
}
/// Overlay single-process counts on a persisted snapshot and narrow it
/// for display.
///
/// Whether the snapshot measured anything is decided before the overlay.
/// The overlay clears `usage_snapshot_complete` when it holds counts it
/// cannot reconcile with the snapshot, such as writes into a bucket that
/// has no verified baseline. That makes the totals unconverged, not the
/// scanned buckets unusable; treating it as structural incompleteness
/// blanked usage for every bucket for as long as one such bucket kept
/// receiving writes, because each write restarts its scanner fence.
async fn present_data_usage_snapshot(mut info: DataUsageInfo, buckets: impl IntoIterator<Item = String>) -> DataUsageInfo {
let snapshot_measured = info.is_complete_bucket_usage_snapshot() || info.is_valid_partial_snapshot();
apply_bucket_usage_memory_overlay(&mut info).await;
Self::narrow_data_usage_snapshot_to_measured_buckets(&mut info, snapshot_measured, buckets);
info
}
async fn rebalance_status_snapshot(store: &ECStore) -> AdminUsecaseResult<Option<RebalanceMeta>> {
store.rebalance_status_meta_snapshot().await.map_err(|err| {
error!("refresh rebalance metadata for pool status failed: {:?}", err);
@@ -323,8 +345,7 @@ impl DefaultAdminUsecase {
/// This path never triggers a live full-version listing
/// (rustfs/backlog#1306); freshness is owned by the scanner.
pub(crate) async fn query_data_usage_info_with_store(store: Arc<ECStore>) -> AdminUsecaseResult<DataUsageInfo> {
let mut info = Self::map_data_usage_load_result(load_admin_data_usage_from_backend_cached(store.clone()).await)?;
apply_bucket_usage_memory_overlay(&mut info).await;
let info = Self::map_data_usage_load_result(load_admin_data_usage_from_backend_cached(store.clone()).await)?;
let buckets = store
.list_bucket(&BucketOptions {
cached: true,
@@ -333,11 +354,7 @@ impl DefaultAdminUsecase {
})
.await
.map_err(|_| Self::app_error(S3ErrorCode::InternalError, "list_bucket failed"))?;
// Serve what the scan measured; buckets it never reached stay absent
// (absent means unknown on the wire, unlike a present zero) and the
// snapshot is flagged non-converged. See rustfs#5806 for why blanking
// the whole document instead was worse than the rule it enforced.
Self::narrow_data_usage_snapshot_to_measured_buckets(&mut info, buckets.into_iter().map(|bucket| bucket.name));
let mut info = Self::present_data_usage_snapshot(info, buckets.into_iter().map(|bucket| bucket.name)).await;
let storage_info = StorageAdminApi::storage_info(store.as_ref()).await;
@@ -876,6 +893,12 @@ mod tests {
});
}
/// Narrow a snapshot no overlay has touched, judging it as persisted.
fn narrow_persisted_snapshot(info: &mut DataUsageInfo, buckets: impl IntoIterator<Item = String>) {
let snapshot_measured = info.is_complete_bucket_usage_snapshot() || info.is_valid_partial_snapshot();
DefaultAdminUsecase::narrow_data_usage_snapshot_to_measured_buckets(info, snapshot_measured, buckets);
}
/// rustfs#5806: a bucket the last scan never reached must not blank the
/// usage the scan *did* measure — that turned every bucket creation into a
/// deployment-wide usage blackout until the next cycle.
@@ -897,10 +920,7 @@ mod tests {
// A brand-new bucket the scan never saw: keep bucket-a, omit bucket-new,
// and say so via the convergence flag.
let mut info = measured("bucket-a");
DefaultAdminUsecase::narrow_data_usage_snapshot_to_measured_buckets(
&mut info,
["bucket-a".to_string(), "bucket-new".to_string(), ".rustfs.sys".to_string()],
);
narrow_persisted_snapshot(&mut info, ["bucket-a".to_string(), "bucket-new".to_string(), ".rustfs.sys".to_string()]);
assert!(info.buckets_usage.contains_key("bucket-a"), "measured usage must survive");
assert!(!info.buckets_usage.contains_key("bucket-new"), "unmeasured bucket stays absent");
assert_eq!(info.usage_snapshot_converged, Some(false));
@@ -909,22 +929,19 @@ mod tests {
// Full coverage leaves the snapshot untouched.
let mut info = measured("bucket-a");
DefaultAdminUsecase::narrow_data_usage_snapshot_to_measured_buckets(
&mut info,
["bucket-a".to_string(), ".rustfs.sys".to_string()],
);
narrow_persisted_snapshot(&mut info, ["bucket-a".to_string(), ".rustfs.sys".to_string()]);
assert_eq!(info.usage_snapshot_converged, None, "a covering snapshot is not downgraded");
assert_eq!(info.buckets_count, 1);
// A bucket deleted since the scan must not linger in the response.
let mut info = measured("bucket-gone");
DefaultAdminUsecase::narrow_data_usage_snapshot_to_measured_buckets(&mut info, ["bucket-live".to_string()]);
narrow_persisted_snapshot(&mut info, ["bucket-live".to_string()]);
assert_eq!(info, DataUsageInfo::default(), "nothing measured is still in the namespace");
// Structurally incomplete snapshots carry no usable information.
let mut info = measured("bucket-a");
info.usage_snapshot_complete = false;
DefaultAdminUsecase::narrow_data_usage_snapshot_to_measured_buckets(&mut info, ["bucket-a".to_string()]);
narrow_persisted_snapshot(&mut info, ["bucket-a".to_string()]);
assert_eq!(info, DataUsageInfo::default());
// A structurally valid partial admin view can still carry useful
@@ -945,7 +962,7 @@ mod tests {
tombstone: false,
segment_invalidation_proof: None,
}];
DefaultAdminUsecase::narrow_data_usage_snapshot_to_measured_buckets(&mut info, ["bucket-a".to_string()]);
narrow_persisted_snapshot(&mut info, ["bucket-a".to_string()]);
assert_eq!(info.usage_snapshot_converged, Some(false));
assert_eq!(info.buckets_count, 1);
assert_eq!(info.objects_total_count, 7);
@@ -957,11 +974,63 @@ mod tests {
buckets_count: 0,
..Default::default()
};
DefaultAdminUsecase::narrow_data_usage_snapshot_to_measured_buckets(&mut info, [".rustfs.sys".to_string()]);
narrow_persisted_snapshot(&mut info, [".rustfs.sys".to_string()]);
assert_eq!(info.usage_snapshot_converged, None);
assert_eq!(info.buckets_count, 0);
}
/// A write into a bucket without a verified baseline leaves process-local
/// counts the overlay cannot reconcile, so it clears
/// `usage_snapshot_complete`. That must flag the response non-converged,
/// not erase every bucket the persisted scan measured: under continuous
/// writes the scanner fence never clears and the blackout never ends.
#[tokio::test]
async fn unverified_bucket_writes_do_not_blank_measured_usage() {
let measured_bucket = "admin-usage-measured";
let unverified_bucket = "admin-usage-unverified";
let mut persisted = DataUsageInfo {
last_update: Some(std::time::SystemTime::now() - std::time::Duration::from_secs(60)),
usage_snapshot_complete: true,
buckets_count: 1,
..Default::default()
};
persisted.buckets_usage.insert(
measured_bucket.to_string(),
rustfs_data_usage::BucketUsageInfo {
objects_count: 4_000,
versions_count: 4_000,
size: 8_192_000,
..Default::default()
},
);
persisted.bucket_sizes.insert(measured_bucket.to_string(), 8_192_000);
persisted.calculate_totals();
assert!(persisted.is_complete_bucket_usage_snapshot());
crate::app::storage_api::test::data_usage::record_bucket_object_write_memory(unverified_bucket, None, 1_024).await;
let info = DefaultAdminUsecase::present_data_usage_snapshot(
persisted,
[measured_bucket.to_string(), unverified_bucket.to_string()],
)
.await;
assert_eq!(
info.buckets_usage
.get(measured_bucket)
.map(|usage| (usage.objects_count, usage.size)),
Some((4_000, 8_192_000)),
"usage the scanner measured must survive an unrelated unverified bucket"
);
assert!(
!info.buckets_usage.contains_key(unverified_bucket),
"a bucket without a verified baseline stays absent (unknown)"
);
assert_eq!(info.usage_snapshot_converged, Some(false));
assert!(!info.is_complete_bucket_usage_snapshot());
assert_eq!(info.objects_total_count, 4_000);
}
#[tokio::test]
async fn execute_query_storage_info_returns_internal_error_when_store_uninitialized() {
let usecase = DefaultAdminUsecase::without_context();
+83 -12
View File
@@ -17,6 +17,7 @@
use super::*;
use crate::app::storage_api::object_usecase::DeleteObjectCondition;
use s3s::dto::ObjectIdentifier;
use std::borrow::Cow;
use std::collections::HashSet;
async fn authorize_recursive_delete<T>(
@@ -81,6 +82,19 @@ async fn authorize_recursive_delete<T>(
result
}
// XML body keys bypass s3s's URL-path normalization. Match its private slash
// normalizer without URL-decoding or resolving dot segments in client keys.
fn normalize_delete_object_key(key: &str) -> Cow<'_, str> {
if !key.starts_with('/') {
return Cow::Borrowed(key);
}
let mut normalized = key.split('/').filter(|part| !part.is_empty()).collect::<Vec<_>>().join("/");
if key.ends_with('/') {
normalized.push('/');
}
Cow::Owned(normalized)
}
fn successful_delete_audit_objects(
delete: &s3s::dto::Delete,
successful_results: impl IntoIterator<Item = bool>,
@@ -90,7 +104,9 @@ fn successful_delete_audit_objects(
.iter()
.zip(successful_results)
.filter(|(_, successful)| *successful)
.map(|(requested, _)| AuditObjectVersion::new(requested.key.clone(), requested.version_id.clone()))
.map(|(requested, _)| {
AuditObjectVersion::new(normalize_delete_object_key(&requested.key).into_owned(), requested.version_id.clone())
})
.collect()
}
@@ -590,7 +606,10 @@ impl DefaultObjectUsecase {
// Per-key denials are audited once per denied action, in aggregate, after the loop.
let mut bulk_denials = crate::storage::access::BulkDenialAudit::start(&mut req)?;
for (idx, obj_id) in delete.objects.iter().enumerate() {
if let Err(err) = validate_object_key(&obj_id.key, "DeleteObjects") {
// Authorization and every storage side effect must address the same
// normalized object; retain the request key only for response entries.
let key = normalize_delete_object_key(&obj_id.key);
if let Err(err) = validate_object_key(&key, "DeleteObjects") {
delete_results[idx].error = Some(s3s::dto::Error {
code: Some(err.code().as_str().to_string()),
key: Some(obj_id.key.clone()),
@@ -623,7 +642,7 @@ impl DefaultObjectUsecase {
{
let req_info = req_info_mut(&mut req)?;
req_info.bucket = Some(bucket.clone());
req_info.object = Some(obj_id.key.clone());
req_info.object = Some(key.to_string());
req_info.version_id = version_id.clone();
}
@@ -670,7 +689,7 @@ impl DefaultObjectUsecase {
}
}
if let Err(err) = validate_table_catalog_object_mutation(&bucket, &obj_id.key).await {
if let Err(err) = validate_table_catalog_object_mutation(&bucket, &key).await {
delete_results[idx].error = Some(s3s::dto::Error {
code: Some("InvalidRequest".to_string()),
key: Some(obj_id.key.clone()),
@@ -680,9 +699,9 @@ impl DefaultObjectUsecase {
continue;
}
let synthetic_version_id = delete_condition.is_none() && version_id.is_none() && is_dir_object(&obj_id.key);
let synthetic_version_id = delete_condition.is_none() && version_id.is_none() && is_dir_object(&key);
let object = ObjectToDelete {
object_name: obj_id.key.clone(),
object_name: key.into_owned(),
version_id: version_uuid,
delete_condition,
synthetic_version_id,
@@ -910,15 +929,16 @@ impl DefaultObjectUsecase {
let deleted = (!delete.quiet.unwrap_or_default()).then(|| {
delete_results
.iter()
.filter_map(|result| result.delete_object.as_ref().map(|object| (result, object)))
.map(|(result, object)| DeletedObject {
.enumerate()
.filter_map(|(idx, result)| result.delete_object.as_ref().map(|object| (idx, result, object)))
.map(|(idx, result, object)| DeletedObject {
delete_marker: { if object.delete_marker { Some(true) } else { None } },
delete_marker_version_id: delete_marker_response_version_id(
object.delete_marker_version_id,
object.delete_marker && result.version_suspended && result.requested_version_id.is_none(),
result.synthetic_version_id,
),
key: Some(object.object_name.clone()),
key: Some(delete.objects[idx].key.clone()),
version_id: delete_entry_response_version_id(
object.version_id,
object.delete_marker,
@@ -936,7 +956,12 @@ impl DefaultObjectUsecase {
let errors = delete_results
.iter()
.filter_map(|v| v.error.clone())
.enumerate()
.filter_map(|(idx, result)| {
let mut error = result.error.clone()?;
error.key = Some(delete.objects[idx].key.clone());
Some(error)
})
.collect::<Vec<s3s::dto::Error>>();
let output = DeleteObjectsOutput {
deleted,
@@ -2819,11 +2844,57 @@ mod tests {
assert_eq!(err.message(), Some("Not init"));
}
#[test]
fn delete_objects_key_normalization_matches_s3s_path_keys() {
use s3s::path::{
parse_path_style_with_validation_and_normalization, parse_virtual_hosted_style_with_validation_and_normalization,
};
use s3s::validation::AwsNameValidation;
for key in [
"plain",
"/a/b",
"//a/b",
"/dir///sub//file",
"/dir///",
"/",
"///",
"dir//file",
"/literal%2F+名",
"/dir/../file",
"/dir/./file",
"/dir\\file",
"/a\0b",
] {
let normalized = normalize_delete_object_key(key);
let path = parse_path_style_with_validation_and_normalization(
&format!("/test-bucket/{key}"),
&AwsNameValidation::new(),
true,
)
.expect("path-style key should parse before storage validation");
let virtual_host = parse_virtual_hosted_style_with_validation_and_normalization(
Some("test-bucket"),
&format!("/{key}"),
&AwsNameValidation::new(),
true,
)
.expect("virtual-hosted key should parse before storage validation");
assert_eq!(Some(normalized.as_ref()), path.get_object_key(), "path-style mismatch for {key:?}");
assert_eq!(
Some(normalized.as_ref()),
virtual_host.get_object_key(),
"virtual-hosted mismatch for {key:?}"
);
}
assert_eq!(normalize_delete_object_key(""), "");
}
#[test]
fn delete_objects_audit_details_include_only_successful_request_entries() {
let requested = vec![
ObjectIdentifier {
key: "first-key".to_string(),
key: "/first-key".to_string(),
version_id: None,
..Default::default()
},
@@ -2833,7 +2904,7 @@ mod tests {
..Default::default()
},
ObjectIdentifier {
key: "versioned-key".to_string(),
key: "//versioned-key".to_string(),
version_id: Some("requested-version".to_string()),
..Default::default()
},
+2 -2
View File
@@ -14460,8 +14460,8 @@ fn object_mutation_entrypoints_call_reserved_prefix_guard() {
assert_ordered(
method(delete, "fn execute_delete_objects_owned("),
&[
"validate_object_key(&obj_id.key, \"DeleteObjects\")",
"if let Err(err) = validate_table_catalog_object_mutation(&bucket, &obj_id.key).await {",
"validate_object_key(&key, \"DeleteObjects\")",
"if let Err(err) = validate_table_catalog_object_mutation(&bucket, &key).await {",
"continue;\n }",
"authorized_deletes.push(AuthorizedDelete { idx, object });",
],
+4 -2
View File
@@ -37,11 +37,11 @@ their issue closes.
| `check_no_planning_docs.sh` | ci-gate | Blocks committed planning-type documents | ci.yml Quick Checks; `make pre-commit` |
| `check_no_tokio_io_uring.sh` | ci-gate | Keeps tokio's io-uring backend disabled | ci.yml Quick Checks |
| `check_s3s_footprint.sh` | ci-gate | Exact, lower-only baseline on the direct s3s surface ahead of the gateway migration (reads `.config/s3s-footprint-baseline.txt`; `s3_error_lines` leaves out lines whose file imports the macro from `rustfs-s3-types`, rule in the script header); `--mode allowlist` fails on any Rust file referencing s3s outside `.config/s3s-edge-allowlist.txt`, `--dry-run` lists them | ci.yml Quick Checks; `make pre-commit`; `make script-tests` (`--self-test`) |
| `check_test_count_ratchet.sh` | ci-gate | Exact per-crate test-count and `#[ignore]` ratchet against `.config/test-count-baseline.txt`; no crate may lose tests or gain ignores | ci.yml Quick Checks; `make pre-commit` |
| `check_test_count_ratchet.sh` | ci-gate | Per-crate test-count floor and `#[ignore]` ceiling against `.config/test-count-baseline.txt`; no crate may fall below its tests row or rise above its ignored row (extra tests and new crates pass with a note) | ci.yml Quick Checks; `make pre-commit` |
| `check_unsafe_code_allowances.sh` | ci-gate | Unsafe-code allowance ledger guard | ci.yml Quick Checks |
| `layer-dependency-baseline.txt` | ci-gate (data) | Committed baseline consumed by `check_layer_dependencies.sh` | [Architecture guard troubleshooting](../docs/operations/architecture-guard-troubleshooting.md) |
| `static.sh` | ci-gate | Static-build helper executed inside image builds | `Dockerfile.source`, `Dockerfile.decommission-local` |
| `helm_chart_version.sh` | ci-gate | Keeps the Helm chart version in sync with the release | helm-package.yml |
| `helm_chart_version.sh` | dev-tool | Normalizes chart/app versions for installation references | rustfs-release-version-bump skill |
| `test_helm_templates.sh` | ci-gate | Helm template rendering test | helm-package.yml |
## Test & e2e runners
@@ -55,6 +55,8 @@ their issue closes.
| `test_prepare_replacement_migration.py` | dev-tool | Verifies maintenance approval scope, publication, and stopped-writer assertion | Python unittest; same runbook |
| `test_diagnose_scanner_enumeration_restart.py` | dev-tool | Driver report validation and positive convergence oracle tests | Python unittest; same guide |
| `test_object_generation_protocol_model.py` | dev-tool | Bounded single-slot, four-voter promise/accept design model and recovery/retirement examples; no production runtime evidence | `python3 scripts/test_object_generation_protocol_model.py`; [Generation contract](../docs/architecture/unified-object-generation.md#executable-design-model-and-acceptance-mapping) |
| `junit_diff.py` | dev-tool | Case-by-case pass/fail/skip diff of two JUnit XML reports (reference vs candidate stack); exit 1 on any difference or stale `--allow` entry, 2 on unreadable input | `python3 scripts/junit_diff.py a.xml b.xml`; backlog T3.1 |
| `test_junit_diff.py` | ci-gate | Contract tests and fixtures (`scripts/tests/junit_diff/`) for `junit_diff.py` | `make script-tests` |
| `e2e-run.sh` | ci-gate | Boots a rustfs server and runs the `s3s-e2e` black-box conformance tool against it | ci.yml `e2e-tests` jobs; `docs/testing/README.md` |
| `run_ecstore_validation_suite.sh` | dev-tool | ecstore black-box validation suite (`quick`/`full`/`destructive`/`fuzz` profiles) | `docs/testing/README.md`, `docs/testing/ecstore-validation-suite-design.md` |
| `run_e2e_tests.sh` | dev-tool | Local `e2e_test` crate runner (starts a server, applies filters, cleans up) | `crates/e2e_test/README.md` |
+44 -24
View File
@@ -3,7 +3,8 @@
# section 1 rule 3; task T0.3 in rustfs/backlog#2738).
#
# No PR may lower the number of tests in any workspace crate, and no PR may
# add #[ignore]: "delete the test to get green" is rejected in CI. The counter
# add #[ignore] without a reviewed baseline change: "delete the test to get
# green" is rejected in CI. The counter
# is deliberately per crate, never a workspace total — a drop in one crate
# must not hide behind an addition in another.
#
@@ -25,26 +26,37 @@
# doc comments that mention `#[test]` out of the count.
#
# The counts are compared row by row with .config/test-count-baseline.txt
# (`<package>\t<tests>\t<ignored>`, one row per member). The baseline is an
# EXACT value, not a floor, in both columns:
# (`<package>\t<tests>\t<ignored>`, one row per member). The baseline is a
# FLOOR on tests and a CEILING on ignored. It used to be an exact value in both
# columns, which made every PR that adds a test edit the same file; on a branch
# that takes many commits a week those PRs conflict with one another. What the
# ratchet protects is that nobody deletes or ignores tests to get green without
# it showing in the diff, and that only needs a one-sided bound:
#
# - tests below the row fails: `crate <name>: <actual> < baseline <n>`.
# Restore the tests. If the removal is intentional, lower the row in the
# same PR and justify it in the PR description, so the reduction is visible
# in the diff instead of disappearing into a green check.
# - tests above the row fails as well, telling the author to raise the row in
# the same PR. An exact baseline cannot drift upward silently, so the next
# deletion is always measured against the true count.
# - ignored above the row fails (the line contains `ignored`); adding
# #[ignore] is not accepted on gateway/integration. Ignored below the row
# fails too, asking for the row to be lowered so that ratchet stays tight.
# - tests above the row passes and prints a note on stdout. Rows are raised
# only by `--update`, in whichever PR chooses to tighten them.
# - ignored above the row fails (the line contains `ignored >`): raise the
# row in the same PR and justify the new #[ignore] in the PR description.
# Ignored below the row passes with a note.
# - a row whose crate is no longer a member fails (deleting a crate deletes
# its tests), and a member without a row fails.
# its tests; remove the row in the same PR).
# - a member without a row passes with a note, so adding a crate does not
# block on this file; the next `--update` picks it up. Until then the new
# crate has no floor.
#
# Failures go to stderr and exit 1. Notes go to stdout and never use the
# failure shapes above (` < baseline`, `ignored >`, `not a workspace member`).
#
# `--update` rewrites the baseline from the current tree. Review the diff it
# produces before committing it; it is the only sanctioned way to change the
# file other than editing the single row a PR touched.
#
# Contract tests: scripts/test_check_test_count_ratchet.sh (make script-tests).
#
# Usage:
# scripts/check_test_count_ratchet.sh # compare against the baseline
# scripts/check_test_count_ratchet.sh --update # regenerate the baseline
@@ -132,10 +144,12 @@ if [[ "$mode" == "update" ]]; then
echo "# One row per workspace member: <package>, tests, ignored, tab-separated."
echo "# tests counts #[test], #[tokio::test], #[tokio::test(...)] and #[test_case(...)]"
echo "# attribute lines under the package directory; ignored counts #[ignore] and"
echo "# #[ignore = \"...\"]. Both columns are exact: CI fails when a crate's tests fall"
echo "# below or rise above its row, or when its ignored count changes. Raise a row in"
echo "# the PR that adds tests; lowering one is a reviewed statement that tests were"
echo "# intentionally removed. Regenerate with:"
echo "# #[ignore = \"...\"]. The tests column is a floor and the ignored column is a"
echo "# ceiling: CI fails when a crate's tests fall below its row or its ignored count"
echo "# rises above it. Extra tests and fewer ignores pass; --update tightens the rows."
echo "# Lowering a tests row or raising an ignored row is a reviewed statement that"
echo "# tests were intentionally removed or ignored; justify it in the PR description."
echo "# Regenerate with:"
echo "# scripts/check_test_count_ratchet.sh --update"
cat "$TMP_DIR/actual"
} >"$BASELINE_FILE"
@@ -165,9 +179,10 @@ if cut -f1 "$TMP_DIR/baseline" | LC_ALL=C sort | uniq -d | grep . >"$TMP_DIR/dup
exit 1
fi
# Every violation is one line. The `<actual> < baseline <n>` and `ignored`
# shapes are the acceptance contract of rustfs/backlog#2738; keep them.
awk -F'\t' '
# Every violation is one line on stderr; every note is one line on stdout. The
# `<actual> < baseline <n>` and `ignored >` shapes are the acceptance contract
# of rustfs/backlog#2738 and the contract tests; notes must never reuse them.
awk -F'\t' -v notes="$TMP_DIR/notes" '
NR == FNR {
baseline_tests[$1] = $2
baseline_ignored[$1] = $3
@@ -176,18 +191,18 @@ NR == FNR {
{
seen[$1] = 1
if (!($1 in baseline_tests)) {
printf "crate %s: no baseline row (%d tests, %d ignored); add it in this PR\n", $1, $2, $3
printf "crate %s: no baseline row (%d tests, %d ignore attributes); run --update to add it\n", $1, $2, $3 >notes
next
}
if ($2 + 0 < baseline_tests[$1] + 0) {
printf "crate %s: %d < baseline %d\n", $1, $2, baseline_tests[$1]
} else if ($2 + 0 > baseline_tests[$1] + 0) {
printf "crate %s: %d > baseline %d; raise the baseline row to %d in this PR\n", $1, $2, baseline_tests[$1], $2
printf "crate %s: %d above baseline %d; run --update to tighten\n", $1, $2, baseline_tests[$1] >notes
}
if ($3 + 0 > baseline_ignored[$1] + 0) {
printf "crate %s: %d ignored > baseline %d; adding #[ignore] is not accepted\n", $1, $3, baseline_ignored[$1]
printf "crate %s: %d ignored > baseline %d; raise the baseline row to %d in this PR and justify the new #[ignore] in the PR description\n", $1, $3, baseline_ignored[$1], $3
} else if ($3 + 0 < baseline_ignored[$1] + 0) {
printf "crate %s: %d ignored < baseline %d; lower the baseline row to %d in this PR\n", $1, $3, baseline_ignored[$1], $3
printf "crate %s: %d ignore attributes below baseline %d; run --update to tighten\n", $1, $3, baseline_ignored[$1] >notes
}
}
END {
@@ -199,14 +214,19 @@ END {
}
' "$TMP_DIR/baseline" "$TMP_DIR/actual" | LC_ALL=C sort >"$TMP_DIR/violations"
if [[ -f "$TMP_DIR/notes" ]]; then
LC_ALL=C sort "$TMP_DIR/notes" | sed 's/^/note: /'
fi
if grep -q . "$TMP_DIR/violations"; then
cat "$TMP_DIR/violations" >&2
echo "error: test-count ratchet violated (rustfs/backlog#2734 section 1 rule 3)." >&2
echo " A crate below its baseline lost tests: restore them. If the removal is" >&2
echo " intentional, lower that row in $BASELINE_FILE in the same PR and" >&2
echo " justify it in the PR description. A crate above its baseline raises its" >&2
echo " row in the same PR ('$0 --update' rewrites the file; review the diff)." >&2
echo " justify it in the PR description. A crate above its ignored baseline" >&2
echo " raises that row in the same PR with a justification in the PR description" >&2
echo " ('$0 --update' rewrites the file; review the diff)." >&2
exit 1
fi
echo "test-count ratchet OK: $crate_count crates, $total_tests tests, $total_ignored ignored match $BASELINE_FILE"
echo "test-count ratchet OK: $crate_count crates, $total_tests tests, $total_ignored ignored; every crate at or above its tests floor and at or below its ignored ceiling in $BASELINE_FILE"
+62 -3
View File
@@ -98,6 +98,58 @@ def git_file(root: Path, revision: str, path: str) -> str:
return git(root, "show", f"{revision}:{path}")
def release_metadata_only(root: Path, base: str, head: str, path: str) -> bool:
"""Recognize literal stable-version edits, never general packaging changes."""
stable = r"(?:0|[1-9][0-9]*)\.(?:0|[1-9][0-9]*)\.(?:0|[1-9][0-9]*)"
marker = "<version>"
patterns = {
"flake.nix": [rf'(?m)^([ \t]*pname = "rustfs";\n[ \t]*version = ")({stable})(";)$'],
"helm/rustfs/Chart.yaml": [rf'(?m)^({key}: ")({stable})(")$' for key in ("version", "appVersion")],
"rustfs.spec": [rf'(?m)^(Version:[ \t]+)({stable})()$'],
}
if path not in patterns or not SHA.fullmatch(base) or not SHA.fullmatch(head):
return False
try:
normalized, versions = [], []
for revision in (base, head):
# A symlink, added/deleted file or executable-mode change is a
# build input even if its apparent text resembles metadata.
entry = git(root, "ls-tree", "-z", revision, "--", path)
if not re.fullmatch(rf"100644 blob [0-9a-f]{{40}}\t{re.escape(path)}\x00", entry):
return False
content, values = git_file(root, revision, path), []
# Existing markers could make edits to distinct fields collapse
# to identical normalized text.
if marker in content:
return False
for pattern in patterns[path]:
matches = list(re.finditer(pattern, content))
if len(matches) != 1:
return False
values.append(matches[0][2])
content = re.sub(pattern, lambda match: match[1] + marker + match[3], content)
if len(set(values)) != 1:
return False
normalized.append(content)
versions.append(values[0])
if path == "rustfs.spec":
old, new = (content.split("%changelog\n") for content in normalized)
if len(old) != 2 or len(new) != 2 or old[0] != new[0]:
return False
if old[1] == new[1]:
return True
# Only the standard prepended release note is non-executable;
# RPM macro syntax and edits to older history retain the full gate.
note = (r"\* (?:Mon|Tue|Wed|Thu|Fri|Sat|Sun) "
r"(?:Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec) [0-9]{2} [0-9]{4} "
r"[A-Za-z0-9 ._-]+ <[A-Za-z0-9._+-]+@[A-Za-z0-9.-]+>\n"
rf"- Update RPM package to RustFS {re.escape(versions[1])}\n\n")
return re.fullmatch(note + re.escape(old[1]), new[1]) is not None
return normalized[0] == normalized[1]
except (OSError, subprocess.CalledProcessError, UnicodeError, ValueError):
return False
def dependency_tables(document: dict):
for key, value in document.items():
if key in ("dependencies", "dev-dependencies", "build-dependencies"):
@@ -283,10 +335,17 @@ def select_plan(event: str, base: str, head: str, ref: str, root: Path, action:
base_packages, base_graph = head_packages, head_graph
reasons.append(f"conservative fallback: {type(error).__name__}")
graph = {name: head_graph.get(name, set()) | base_graph.get(name, set()) for name in set(head_graph) | set(base_graph)}
affected = set()
affected, non_runtime = set(), set()
if not full and policy is not None:
for path in changed:
if documentation_path(path):
non_runtime.add(path)
continue
if (policy["rollout"] == "enforced"
and not any(fnmatch.fnmatchcase(path, pattern) for pattern in policy["full_paths"])
and release_metadata_only(root, base, head, path)):
non_runtime.add(path)
reasons.append(f"literal stable release metadata: {path}")
continue
if path in E2E_SELECTION_FILES:
affected.add("e2e_test")
@@ -308,9 +367,9 @@ def select_plan(event: str, base: str, head: str, ref: str, root: Path, action:
affected.update(owners)
if not affected and not full:
mode = "docs"
reasons.append("all changes are documentation")
reasons.append("all changes are documentation or literal stable release metadata")
elif affected == {"e2e_test"} and not full and not (closure({"e2e_test"}, graph, reverse=True) - {"e2e_test"}) and all(
documentation_path(path) or path in E2E_SELECTION_FILES or
path in non_runtime or path in E2E_SELECTION_FILES or
(path.startswith("crates/e2e_test/src/") and path.endswith(".rs")) for path in changed):
mode = "e2e"
reasons.append("isolated E2E harness changes")
+1 -6
View File
@@ -17,12 +17,7 @@ case "$RAW" in
esac
APP_VERSION="${RAW_TAG#v}"
BETA_NUM=$(printf '%s\n' "$APP_VERSION" | sed -n -E 's/^.*-beta\.([0-9]+)$/\1/p')
if [ -n "$BETA_NUM" ]; then
CHART_VERSION="0.${BETA_NUM}.0"
else
CHART_VERSION="$APP_VERSION"
fi
CHART_VERSION="$APP_VERSION"
if [ -n "${GITHUB_OUTPUT:-}" ]; then
{
+167
View File
@@ -0,0 +1,167 @@
#!/usr/bin/env python3
# Copyright 2026 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Compare two JUnit XML reports case by case.
Cases are aligned on classname + name and reduced to pass / fail / skip. A case
present in only one report is reported as `missing` on the other side. A
`<flakyFailure>` case that finally passed is a pass and is listed separately.
Exit codes: 0 no unallowed differences and no stale allow-list entry; 1 at least
one difference or stale allow-list entry; 2 unreadable or malformed input.
"""
import argparse
import json
import sys
import xml.etree.ElementTree as ET
PASS, FAIL, SKIP, MISSING = "pass", "fail", "skip", "missing"
_RETRY_TAGS = {"flakyFailure", "flakyError", "rerunFailure", "rerunError"}
class InputError(Exception):
"""Input that cannot be trusted as a report; maps to exit code 2."""
def case_id(classname, name):
return f"{classname}::{name}" if classname else name
def parse_report(path):
"""Return ({case_id: state}, [flaky case ids]) for one JUnit file."""
try:
with open(path, "rb") as handle:
data = handle.read()
except OSError as exc:
raise InputError(f"{path}: cannot read: {exc.strerror or exc}") from exc
if not data.strip():
raise InputError(f"{path}: file is empty")
try:
root = ET.fromstring(data)
except ET.ParseError as exc:
raise InputError(f"{path}: malformed XML: {exc}") from exc
if root.tag not in ("testsuites", "testsuite"):
raise InputError(f"{path}: unexpected root element <{root.tag}>")
states, flaky = {}, []
for case in root.iter("testcase"):
name = case.get("name")
if not name:
raise InputError(f"{path}: <testcase> without a name")
cid = case_id(case.get("classname", ""), name)
if cid in states:
raise InputError(f"{path}: duplicate case {cid}")
tags = {child.tag for child in case}
if tags & {"failure", "error"}:
state = FAIL
elif "skipped" in tags:
state = SKIP
else:
state = PASS
states[cid] = state
if state == PASS and tags & _RETRY_TAGS:
flaky.append(cid)
if not states:
raise InputError(f"{path}: report contains no test cases")
return states, sorted(flaky)
def read_allow(path):
try:
with open(path, encoding="utf-8") as handle:
lines = handle.read().splitlines()
except (OSError, UnicodeDecodeError) as exc:
raise InputError(f"{path}: cannot read allow-list: {exc}") from exc
return {line.strip() for line in lines if line.strip() and not line.lstrip().startswith("#")}
def counts(states):
result = {PASS: 0, FAIL: 0, SKIP: 0}
for state in states.values():
result[state] += 1
return result
def compare(a, b, allow):
diffs = []
for cid in sorted(set(a) | set(b)):
sa, sb = a.get(cid, MISSING), b.get(cid, MISSING)
if sa != sb:
diffs.append({"case": cid, "a": sa, "b": sb, "allowed": cid in allow})
stale = sorted(allow - {d["case"] for d in diffs})
return diffs, stale
def render(args, diffs, stale, flaky_a, flaky_b, counts_a, counts_b):
out = []
for label, c in ((args.label_a, counts_a), (args.label_b, counts_b)):
out.append(f"{label}: pass={c[PASS]} fail={c[FAIL]} skip={c[SKIP]}")
for label, flaky in ((args.label_a, flaky_a), (args.label_b, flaky_b)):
if flaky:
out.append(f"\nFlaky in {label} (passed after retry, not a difference):")
out.extend(f" {cid}" for cid in flaky)
if diffs:
width = max(len("case"), *(len(d["case"]) for d in diffs))
wa = max(len(args.label_a), 7)
out.append("")
out.append(f"{'case':<{width}} {args.label_a:<{wa}} {args.label_b}")
for d in diffs:
note = " (allowed)" if d["allowed"] else ""
out.append(f"{d['case']:<{width}} {d['a']:<{wa}} {d['b']}{note}")
if stale:
out.append("\nStale allow-list entries (no matching difference; remove them):")
out.extend(f" {cid}" for cid in stale)
if not diffs and not stale:
out.append("\nNo differences.")
return "\n".join(out)
def main(argv=None):
parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
parser.add_argument("a", help="first JUnit XML report")
parser.add_argument("b", help="second JUnit XML report")
parser.add_argument("--label-a", default="a")
parser.add_argument("--label-b", default="b")
parser.add_argument("--json", metavar="PATH", help="write a machine-readable summary")
parser.add_argument("--allow", metavar="FILE", help="case ids (one per line) whose difference is tolerated")
args = parser.parse_args(argv)
if args.label_a == args.label_b:
print("error: --label-a and --label-b must differ", file=sys.stderr)
return 2
try:
states_a, flaky_a = parse_report(args.a)
states_b, flaky_b = parse_report(args.b)
allow = read_allow(args.allow) if args.allow else set()
diffs, stale = compare(states_a, states_b, allow)
counts_a, counts_b = counts(states_a), counts(states_b)
if args.json:
summary = {
"labels": {"a": args.label_a, "b": args.label_b},
"counts": {args.label_a: counts_a, args.label_b: counts_b},
"differences": diffs,
"flaky": {args.label_a: flaky_a, args.label_b: flaky_b},
"stale_allow": stale,
}
with open(args.json, "w", encoding="utf-8") as handle:
json.dump(summary, handle, indent=2, sort_keys=True)
handle.write("\n")
except (InputError, OSError) as exc:
print(f"error: {exc}", file=sys.stderr)
return 2
print(render(args, diffs, stale, flaky_a, flaky_b, counts_a, counts_b))
return 1 if stale or any(not d["allowed"] for d in diffs) else 0
if __name__ == "__main__":
sys.exit(main())
+82
View File
@@ -0,0 +1,82 @@
#!/usr/bin/env bash
# Copyright 2026 RustFS Team
# SPDX-License-Identifier: Apache-2.0
set -euo pipefail
PKG_DIR="${DEB_FILE%.deb}"
echo "Building DEB: ${DEB_FILE}"
mkdir -p "${PKG_DIR}/DEBIAN"
mkdir -p "${PKG_DIR}/usr/bin"
mkdir -p "${PKG_DIR}/etc/default"
mkdir -p "${PKG_DIR}/lib/systemd/system"
mkdir -p "${PKG_DIR}/usr/share/doc/rustfs"
cp ./bin/rustfs "${PKG_DIR}/usr/bin/"
chmod 755 "${PKG_DIR}/usr/bin/rustfs"
cp deploy/build/rustfs.service "${PKG_DIR}/lib/systemd/system/"
cat > "${PKG_DIR}/etc/default/rustfs" << 'ENVEOF'
# RustFS Environment Configuration
# See https://docs.rustfs.com for more information
# RUSTFS_ACCESS_KEY=<changeme>
# RUSTFS_SECRET_KEY=<changeme>
# RUSTFS_VOLUMES=""
# RUSTFS_ADDRESS=":9000"
# RUSTFS_CONSOLE_ADDRESS=":9001"
# RUSTFS_CONSOLE_ENABLE=true
# RUSTFS_OBS_LOGGER_LEVEL=error
# RUSTFS_OBS_LOG_DIRECTORY="/var/log/rustfs/"
ENVEOF
cat > "${PKG_DIR}/DEBIAN/control" << EOF
Package: rustfs
Version: ${DEB_VERSION}
Section: utils
Priority: optional
Architecture: ${DEB_ARCH}
EOF
# gnu binaries dynamically link glibc and must declare it; musl
# binaries are statically linked and need no libc dependency.
if [[ -n "$DEB_DEPENDS" ]]; then
echo "Depends: ${DEB_DEPENDS}" >> "${PKG_DIR}/DEBIAN/control"
fi
cat >> "${PKG_DIR}/DEBIAN/control" << EOF
Maintainer: RustFS Team <support@rustfs.com>
Description: High-performance distributed object storage (${LIBC} build)
RustFS is a high-performance distributed object storage software
built using Rust. It is compatible with MinIO and S3 API.
Homepage: https://rustfs.com
EOF
# Declare /etc/default/rustfs as a conffile so dpkg preserves user
# modifications on upgrade instead of silently overwriting them.
cat > "${PKG_DIR}/DEBIAN/conffiles" << 'CONFFILES'
/etc/default/rustfs
CONFFILES
./scripts/release/package_service_scripts.sh after-install > "${PKG_DIR}/DEBIAN/postinst"
chmod 755 "${PKG_DIR}/DEBIAN/postinst"
./scripts/release/package_service_scripts.sh before-remove > "${PKG_DIR}/DEBIAN/prerm"
chmod 755 "${PKG_DIR}/DEBIAN/prerm"
./scripts/release/package_service_scripts.sh after-remove > "${PKG_DIR}/DEBIAN/postrm"
chmod 755 "${PKG_DIR}/DEBIAN/postrm"
cp LICENSE "${PKG_DIR}/usr/share/doc/rustfs/"
cp README.md "${PKG_DIR}/usr/share/doc/rustfs/"
fakeroot dpkg-deb --build "${PKG_DIR}" "$DEB_FILE"
[[ $(dpkg-deb -f "$DEB_FILE" Package) == rustfs ]]
[[ $(dpkg-deb -f "$DEB_FILE" Version) == "$DEB_VERSION" ]]
[[ $(dpkg-deb -f "$DEB_FILE" Architecture) == "$DEB_ARCH" ]]
dpkg-deb --fsys-tarfile "$DEB_FILE" | tar -tf - | grep -Fx './usr/bin/rustfs' >/dev/null
stat --printf='%n %s bytes\n' "$DEB_FILE"
echo "deb_file=$DEB_FILE" >> "$GITHUB_OUTPUT"
echo "✅ DEB built: $DEB_FILE"
+69
View File
@@ -0,0 +1,69 @@
#!/usr/bin/env bash
# Copyright 2026 RustFS Team
# SPDX-License-Identifier: Apache-2.0
set -euo pipefail
echo "Building RPM for ${RPM_ARCH} (${LIBC})"
./scripts/test_package_versions.sh --require-package-managers
# gnu binaries dynamically link glibc and must declare it; musl
# binaries are statically linked and need no libc dependency.
depends_args=()
if [[ -n "$RPM_DEPENDS" ]]; then
depends_args+=(--depends "$RPM_DEPENDS")
fi
# Create config file for fpm (DEB build creates it in its package dir structure,
# but fpm needs the file to exist before packaging)
mkdir -p ./tmp-pkg/etc/default
cat > ./tmp-pkg/etc/default/rustfs << 'ENVEOF'
# RustFS Environment Configuration
# See https://docs.rustfs.com for more information
# RUSTFS_ACCESS_KEY=<changeme>
# RUSTFS_SECRET_KEY=<changeme>
# RUSTFS_VOLUMES=""
# RUSTFS_ADDRESS=":9000"
# RUSTFS_CONSOLE_ADDRESS=":9001"
# RUSTFS_CONSOLE_ENABLE=true
# RUSTFS_OBS_LOGGER_LEVEL=error
# RUSTFS_OBS_LOG_DIRECTORY="/var/log/rustfs/"
ENVEOF
fpm -s dir -t rpm \
--name rustfs \
--version "$RPM_VERSION" \
--iteration "$RPM_RELEASE" \
--architecture "$RPM_ARCH" \
--package "$RPM_FILE" \
"${depends_args[@]}" \
--description "High-performance distributed object storage (${LIBC} build)" \
--maintainer "RustFS Team <support@rustfs.com>" \
--url "https://rustfs.com" \
--license "Apache-2.0" \
--before-install <(./scripts/release/package_service_scripts.sh rpm-before-install) \
--before-upgrade <(./scripts/release/package_service_scripts.sh rpm-before-upgrade) \
--after-install <(./scripts/release/package_service_scripts.sh after-install) \
--after-upgrade <(./scripts/release/package_service_scripts.sh after-install) \
--before-remove <(./scripts/release/package_service_scripts.sh before-remove) \
--after-remove <(./scripts/release/package_service_scripts.sh after-remove) \
--rpm-posttrans <(./scripts/release/package_service_scripts.sh rpm-posttrans) \
--config-files /etc/default/rustfs \
./bin/rustfs=/usr/bin/rustfs \
./tmp-pkg/etc/default/rustfs=/etc/default/rustfs \
deploy/build/rustfs.service=/lib/systemd/system/rustfs.service \
LICENSE=/usr/share/doc/rustfs/LICENSE \
README.md=/usr/share/doc/rustfs/README.md
if [[ ! -f "$RPM_FILE" ]]; then
echo "❌ RPM build failed"
exit 1
fi
RPM_METADATA=$(rpm -qp --qf '%{NAME}\n%{VERSION}\n%{RELEASE}\n%{ARCH}\n' "$RPM_FILE")
EXPECTED_METADATA=$(printf 'rustfs\n%s\n%s\n%s' "$RPM_VERSION" "$RPM_RELEASE" "$RPM_ARCH")
[[ "$RPM_METADATA" == "$EXPECTED_METADATA" ]]
rpm -qpl "$RPM_FILE" | grep -Fx '/usr/bin/rustfs' >/dev/null
stat --printf='%n %s bytes\n' "$RPM_FILE"
echo "rpm_file=$RPM_FILE" >> "$GITHUB_OUTPUT"
echo "✅ RPM built: $RPM_FILE"
+428
View File
@@ -0,0 +1,428 @@
#!/usr/bin/env python3
# Copyright 2026 RustFS Team
# SPDX-License-Identifier: Apache-2.0
"""Frozen release inputs and artifact identity; publication never builds binaries."""
from __future__ import annotations
import argparse
import hashlib
import json
import os
from pathlib import Path
import re
import subprocess
import sys
import tempfile
import tomllib
import urllib.request
from urllib.parse import urlsplit
import zipfile
ROOT = Path(__file__).resolve().parents[2]
REPOSITORY = "rustfs/rustfs"
SCHEMA = 1
HEX40 = re.compile(r"[0-9a-f]{40}\Z")
HEX64 = re.compile(r"[0-9a-f]{64}\Z")
VERSION = re.compile(r"(0|[1-9][0-9]*)\.(0|[1-9][0-9]*)\.(0|[1-9][0-9]*)(?:-(?:alpha|beta|rc)\.(0|[1-9][0-9]*))?\Z")
TARGETS = ("linux-x86_64-musl", "linux-aarch64-musl", "linux-x86_64-gnu", "linux-aarch64-gnu", "macos-aarch64", "windows-x86_64")
LINUX = TARGETS[:4]
DESKTOP = TARGETS[4:]
MAX_MEMBER = 2 * 1024**3
RECIPE_PATHS = (
".github/workflows/build.yml",
".github/workflows/release-candidate.yml",
".github/workflows/release-complete.yml",
".github/workflows/release-publish.yml",
".github/workflows/docker.yml",
".github/workflows/package.yml",
".github/actions/setup/action.yml",
".github/actions/setup/download_release_asset.py",
"scripts/release/candidate.py",
"scripts/release/pipeline.py",
"scripts/release/verify.py",
"scripts/release/deliver.py",
"scripts/release/publish.py",
"scripts/release/release_gate.py",
"scripts/release/package_versions.sh",
"scripts/release/package_service_scripts.sh",
"scripts/release/build_deb.sh",
"scripts/release/build_rpm.sh",
"scripts/release/check_glibc.py",
".config/release-validation-plan.json",
"Dockerfile", "Dockerfile.glibc", "entrypoint.sh", "deploy/build/rustfs.service",
".github/workflows/helm-package.yml", "scripts/test_helm_templates.sh",
"scripts/security/generate_release_supply_chain_assets.py",
".github/workflows/ci.yml", "scripts/ci_gate.py",
)
def require(condition, message):
if not condition:
raise ValueError(message)
def digest(value):
return hashlib.sha256(json.dumps(value, sort_keys=True, separators=(",", ":"), ensure_ascii=False).encode()).hexdigest()
def stream_digest(stream):
result = hashlib.sha256()
for chunk in iter(lambda: stream.read(1024 * 1024), b""):
result.update(chunk)
return result.hexdigest()
def file_digest(path):
with Path(path).open("rb") as stream:
return stream_digest(stream)
def read_json(path):
path = Path(path)
require(path.is_file() and path.stat().st_size <= 4 * 1024**2, "missing or oversized JSON record")
def unique(pairs):
value = {}
for name, item in pairs:
require(name not in value, "duplicate JSON field")
value[name] = item
return value
return json.loads(path.read_text(encoding="utf-8"), object_pairs_hook=unique)
def write_json(path, value):
"""Existing identity records are immutable, including across job retries."""
path = Path(path)
if path.exists():
require(read_json(path) == value, "refusing to replace an existing identity record")
return
path.parent.mkdir(parents=True, exist_ok=True)
with path.open("x", encoding="utf-8") as stream:
json.dump(value, stream, indent=2, sort_keys=True, ensure_ascii=False)
stream.write("\n")
def command(args, *, input_text=None, cwd=None):
return subprocess.run(args, input=input_text, text=True, capture_output=True, check=True, cwd=cwd, timeout=120).stdout.strip()
class ArtifactRedirect(urllib.request.HTTPRedirectHandler):
def redirect_request(self, request, fp, code, message, headers, url):
require(urlsplit(url).scheme == "https", "artifact redirect must use HTTPS")
redirected = super().redirect_request(request, fp, code, message, headers, url)
if redirected is not None and urlsplit(request.full_url).netloc != urlsplit(url).netloc:
redirected.remove_header("Authorization")
return redirected
def github_stream(path):
require(path.startswith("repos/") and ":" not in path and not path.startswith("/"), "invalid GitHub API path")
token = os.environ.get("GH_TOKEN") or command(["gh", "auth", "token"])
request = urllib.request.Request("https://api.github.com/" + path,
headers={"Authorization": "Bearer " + token, "Accept": "application/vnd.github+json",
"X-GitHub-Api-Version": "2022-11-28", "User-Agent": "rustfs-release"})
return urllib.request.build_opener(ArtifactRedirect()).open(request, timeout=600)
def github(path):
with github_stream(path) as response:
return json.load(response)
def git_file(revision, path, root=ROOT):
require(HEX40.fullmatch(revision) is not None, "source must be a full commit SHA")
return subprocess.check_output(["git", "show", f"{revision}:{path}"], cwd=root, timeout=60)
def recipe_digest(revision, root=ROOT):
files = {path: hashlib.sha256(git_file(revision, path, root)).hexdigest() for path in RECIPE_PATHS}
for directory in ("bootstrap", "helm"):
paths = command(["git", "ls-tree", "-r", "--name-only", revision, "--", directory], cwd=root).splitlines()
require(bool(paths), "missing release recipe directory")
files.update({path: hashlib.sha256(git_file(revision, path, root)).hexdigest() for path in paths})
return digest(files)
def console_asset(release):
require(isinstance(release, dict) and release.get("draft") is False and release.get("prerelease") is False,
"Console must be a published stable release")
tag = release.get("tag_name")
require(isinstance(tag, str) and re.fullmatch(r"v?[0-9]+\.[0-9]+\.[0-9]+", tag), "invalid Console version")
assets = [asset for asset in release.get("assets", [])
if re.fullmatch(r"rustfs-console-[A-Za-z0-9._-]+\.zip", asset.get("name", ""))]
require(len(assets) == 1, "Console must have exactly one unambiguous ZIP asset")
asset = assets[0]
checksum = asset.get("digest", "")
require(isinstance(checksum, str) and checksum.startswith("sha256:") and HEX64.fullmatch(checksum[7:]),
"Console asset must have a SHA-256 digest")
expected = f"https://github.com/rustfs/console/releases/download/{tag}/{asset['name']}"
require(asset.get("browser_download_url") == expected, "Console URL does not match its release identity")
return {"tag": tag, "name": asset["name"], "url": expected, "sha256": checksum[7:]}
def release_asset(release, repository, name):
require(release.get("draft") is False and release.get("prerelease") is False, "dependency must be a published stable release")
tag = release.get("tag_name", "")
require(re.fullmatch(r"v?[0-9]+\.[0-9]+\.[0-9]+", tag), "invalid dependency tag")
matches = [a for a in release.get("assets", []) if a.get("name") == name]
require(len(matches) == 1, "missing or ambiguous pinned dependency")
asset = matches[0]
url = f"https://github.com/{repository}/releases/download/{tag}/{name}"
require(asset.get("browser_download_url") == url and re.fullmatch(r"sha256:[0-9a-f]{64}", asset.get("digest", "")), "invalid dependency identity")
return {"version": tag, "name": name, "url": url, "sha256": asset["digest"][7:]}
def validate_dependency(asset, repository):
require(isinstance(asset, dict) and set(asset) == {"version", "name", "url", "sha256"}, "invalid dependency record")
require(re.fullmatch(r"[A-Za-z0-9._-]+", asset["name"]), "unsafe dependency name")
normalized = release_asset({"draft": False, "prerelease": False, "tag_name": asset["version"],
"assets": [{"name": asset["name"], "browser_download_url": asset["url"],
"digest": "sha256:" + asset["sha256"]}]}, repository, asset["name"])
require(normalized == asset, "dependency record changed")
def validate_inputs(value):
require(isinstance(value, dict), "candidate inputs must be an object")
fields = {"schema", "candidate_id", "version", "source_sha", "workflow_sha", "recipe_sha256", "lock_sha256",
"plan_sha256", "toolchain", "console", "cli", "previous", "ci", "origin_run_id", "origin_run_number"}
require(set(value) == fields and type(value["schema"]) is int and value["schema"] == SCHEMA,
"unsupported candidate input schema")
require(isinstance(value["version"], str) and VERSION.fullmatch(value["version"]), "invalid release version")
for name in ("source_sha", "workflow_sha"):
require(isinstance(value[name], str) and HEX40.fullmatch(value[name]), f"invalid {name}")
for name in ("recipe_sha256", "lock_sha256", "plan_sha256"):
require(isinstance(value[name], str) and HEX64.fullmatch(value[name]), f"invalid {name}")
for name in ("origin_run_id", "origin_run_number"):
require(type(value[name]) is int and value[name] > 0, f"invalid {name}")
require(value["candidate_id"] == f"{value['version']}-preview.{value['origin_run_number']}", "candidate name differs from its immutable run identity")
require(isinstance(value["toolchain"], str) and re.fullmatch(r"[1-9][0-9]*\.[0-9]+\.[0-9]+", value["toolchain"]),
"candidate toolchain must be an exact stable Rust version")
console = value["console"]
require(isinstance(console, dict) and set(console) == {"tag", "name", "url", "sha256"}, "invalid Console identity")
normalized = console_asset({"draft": False, "prerelease": False, "tag_name": console["tag"],
"assets": [{"name": console["name"], "browser_download_url": console["url"],
"digest": "sha256:" + str(console["sha256"])}]})
require(normalized == console, "invalid pinned Console asset")
require(isinstance(value["cli"], dict) and set(value["cli"]) == set(LINUX), "missing CLI platform")
for asset in value["cli"].values():
validate_dependency(asset, "rustfs/cli")
validate_dependency(value["previous"], REPOSITORY)
require(value["previous"]["version"].lstrip("v") != value["version"], "previous release must differ from the candidate")
require(isinstance(value["ci"], dict) and set(value["ci"]) == {"run_id", "run_attempt"}
and all(type(item) is int and item > 0 for item in value["ci"].values()), "invalid exact-source CI identity")
return value
def freeze(version, source_sha, workflow_sha, toolchain, release, run_id, run_number, cli, previous, ci, root=ROOT):
require(VERSION.fullmatch(version) is not None, "invalid release version")
source = tomllib.loads(git_file(source_sha, "Cargo.toml", root).decode())
require(source["workspace"]["package"]["version"] == version, "Cargo workspace version differs from the target")
recipe = recipe_digest(workflow_sha, root)
require(recipe_digest(source_sha, root) == recipe, "candidate must contain the reviewed release recipe")
plan = json.loads(git_file(workflow_sha, ".config/release-validation-plan.json", root))
value = {"schema": SCHEMA, "version": version, "candidate_id": f"{version}-preview.{run_number}",
"source_sha": source_sha, "workflow_sha": workflow_sha, "recipe_sha256": recipe,
"lock_sha256": hashlib.sha256(git_file(source_sha, "Cargo.lock", root)).hexdigest(),
"plan_sha256": digest(plan), "toolchain": toolchain, "console": console_asset(release),
"origin_run_id": run_id, "origin_run_number": run_number, "ci": ci,
"cli": {target: release_asset(cli, "rustfs/cli", f"rustfs-cli-linux-{'amd64' if 'x86_64' in target else 'arm64'}{'-gnu' if target.endswith('gnu') else ''}-{cli['tag_name']}.tar.gz") for target in LINUX},
"previous": release_asset(previous, REPOSITORY, f"rustfs-linux-x86_64-gnu-v{previous['tag_name'].lstrip('v')}.zip")}
return validate_inputs(value)
def targets(selection):
if selection == "linux":
return LINUX
if selection == "desktop":
return DESKTOP
if selection == "all":
return TARGETS
selected = tuple(selection.split(","))
require(selected and len(set(selected)) == len(selected) and all(item in TARGETS for item in selected),
"unknown or duplicate build targets")
return selected
def archive_record(path, target, inputs):
validate_inputs(inputs)
require(target in TARGETS, "unknown binary platform")
path = Path(path)
name = f"rustfs-{target}-v{inputs['version']}.zip"
require(path.name == name and path.is_file() and not path.is_symlink(), "archive name does not match the target version")
binary = "rustfs.exe" if target == "windows-x86_64" else "rustfs"
expected = {binary}
if target.endswith("-gnu"):
expected.update(("rustfs.cpu-symbol-catalog.json", "rustfs.cpu-symbol-catalog.sha256"))
with zipfile.ZipFile(path) as archive:
members = archive.infolist()
require(len(members) == len(expected) and {m.filename for m in members} == expected,
"unexpected or duplicate release archive member")
for member in members:
require(not member.is_dir() and 0 < member.file_size <= MAX_MEMBER and (member.external_attr >> 16) & 0o170000 != 0o120000,
"invalid release archive member")
with archive.open(binary) as stream:
binary_sha = stream_digest(stream)
return {"target": target, "name": name, "kind": "zip", "sha256": file_digest(path), "size": path.stat().st_size,
"binary_sha256": binary_sha, "inputs_sha256": digest(inputs)}
def verify_records(inputs, records, directory, required):
validate_inputs(inputs)
require(isinstance(records, list) and len(records) == len(required), "candidate is missing required platform artifacts")
seen = set()
for record in records:
require(isinstance(record, dict) and record.get("target") in required, "unexpected artifact platform")
target = record["target"]
require(target not in seen, "duplicate artifact platform")
seen.add(target)
expected = archive_record(Path(directory) / f"rustfs-{target}-v{inputs['version']}.zip", target, inputs)
require(record == expected, "candidate artifact or input identity changed")
require(seen == set(required), "incomplete candidate platform set")
return sorted(records, key=lambda record: record["target"])
def certification_candidate(inputs, records, directory, required=TARGETS):
"""Candidate identity survives adding desktop files; test receipts bind bytes."""
records = verify_records(inputs, records, directory, required)
return {"schema_version": 1, "product_commit": inputs["source_sha"], "version": inputs["version"],
"release_profile": "prerelease" if "-" in inputs["version"] else "stable", "features": ["pyroscope"],
"plan_sha256": inputs["plan_sha256"], "workflow_sha": inputs["workflow_sha"],
"validation_batch": {"run_id": inputs["candidate_id"], "run_attempt": 1},
"deliverables": [{"kind": "zip", "name": r["name"], "sha256": r["sha256"], "size": r["size"],
"binary_sha256": r["binary_sha256"],
"location": f"https://github.com/{REPOSITORY}/actions/runs/{inputs['origin_run_id']}#artifact-{r['sha256']}"}
for r in records]}
def validate_run(run, workflow, *, completed=True):
require(isinstance(run, dict) and (run.get("repository") or {}).get("full_name") == REPOSITORY,
"build run belongs to another repository")
require(run.get("path") == f".github/workflows/{workflow}" and run.get("event") == "workflow_dispatch"
and run.get("head_branch") == "main", "candidate must originate from the trusted main workflow")
require(type(run.get("id")) is int and run["id"] > 0 and type(run.get("run_attempt")) is int and run["run_attempt"] > 0,
"invalid workflow run identity")
require(isinstance(run.get("head_sha"), str) and HEX40.fullmatch(run["head_sha"]), "invalid workflow source SHA")
if completed:
require(run.get("status") == "completed" and run.get("conclusion") == "success", "candidate run did not succeed")
return run
def validate_artifact(artifact, run, name):
require(artifact.get("name") == name and artifact.get("expired") is False, "missing or expired candidate artifact")
origin = artifact.get("workflow_run") or {}
require(origin.get("id") == run["id"] and origin.get("head_sha") == run["head_sha"], "artifact belongs to another build run")
require(type(artifact.get("id")) is int and artifact["id"] > 0 and type(artifact.get("size_in_bytes")) is int
and 0 < artifact["size_in_bytes"] <= 8 * MAX_MEMBER, "invalid artifact identity or size")
require(isinstance(artifact.get("digest"), str) and re.fullmatch(r"sha256:[0-9a-f]{64}", artifact["digest"]),
"artifact has no immutable digest")
return artifact
def artifact_list(run_id):
require(type(run_id) is int and run_id > 0, "invalid run ID")
result = []
for page in range(1, 101):
batch = github(f"repos/{REPOSITORY}/actions/runs/{run_id}/artifacts?per_page=100&page={page}")["artifacts"]
result.extend(batch)
if len(batch) < 100:
return result
raise ValueError("too many candidate artifacts")
def download_artifact(run, name, destination):
matches = [item for item in artifact_list(run["id"]) if item.get("name") == name]
require(len(matches) == 1, "candidate artifact is absent or ambiguous")
artifact = validate_artifact(matches[0], run, name)
destination = Path(destination)
destination.mkdir(parents=True, exist_ok=True)
with tempfile.TemporaryDirectory() as temp:
archive_path = Path(temp) / "artifact.zip"
with archive_path.open("wb") as stream, github_stream(f"repos/{REPOSITORY}/actions/artifacts/{artifact['id']}/zip") as response:
received = 0
for chunk in iter(lambda: response.read(1024 * 1024), b""):
received += len(chunk)
require(received <= artifact["size_in_bytes"], "artifact response exceeds its declared size")
stream.write(chunk)
require(archive_path.stat().st_size == artifact["size_in_bytes"] and
"sha256:" + file_digest(archive_path) == artifact["digest"], "downloaded artifact digest or size mismatch")
with zipfile.ZipFile(archive_path) as archive:
members = archive.infolist()
require(0 < len(members) <= 256 and len({m.filename for m in members}) == len(members), "invalid artifact members")
require(sum(member.file_size for member in members) <= 8 * MAX_MEMBER, "expanded artifact exceeds the candidate size limit")
for member in members:
require(re.fullmatch(r"[A-Za-z0-9][A-Za-z0-9._-]{0,200}", member.filename)
and not member.is_dir() and 0 < member.file_size <= MAX_MEMBER
and (member.external_attr >> 16) & 0o170000 != 0o120000, "unsafe artifact member")
target = destination / member.filename
require(not target.exists(), "artifact extraction would overwrite a file")
with archive.open(member) as source, target.open("xb") as sink:
import shutil
shutil.copyfileobj(source, sink)
return artifact
def emit_output(name, value):
require("\n" not in str(value) and "\r" not in str(value), "invalid workflow output")
with open(os.environ["GITHUB_OUTPUT"], "a", encoding="utf-8") as stream:
stream.write(f"{name}={value}\n")
def main(argv=None):
parser = argparse.ArgumentParser(description=__doc__)
sub = parser.add_subparsers(dest="command", required=True)
p = sub.add_parser("freeze")
for name in ("version", "source-sha", "workflow-sha", "toolchain", "console", "cli", "previous", "output"):
p.add_argument("--" + name, required=True)
p.add_argument("--run-id", type=int, required=True)
p.add_argument("--run-number", type=int, required=True)
p.add_argument("--ci-run-id", type=int, required=True)
p.add_argument("--ci-run-attempt", type=int, required=True)
p = sub.add_parser("validate-inputs")
p.add_argument("--inputs", required=True)
p.add_argument("--check-checkout", action="store_true")
p = sub.add_parser("record")
p.add_argument("--inputs", required=True)
p.add_argument("--target", required=True)
p.add_argument("--archive", required=True)
p.add_argument("--output", required=True)
p = sub.add_parser("assemble")
p.add_argument("--inputs", required=True)
p.add_argument("--directory", required=True)
p.add_argument("--output", required=True)
p.add_argument("--platforms", choices=("linux", "all"), default="all")
p = sub.add_parser("download")
p.add_argument("--run-id", type=int, required=True)
p.add_argument("--workflow", choices=("release-candidate.yml", "release-complete.yml"), required=True)
p.add_argument("--name", required=True)
p.add_argument("--destination", required=True)
args = parser.parse_args(argv)
if args.command == "freeze":
write_json(args.output, freeze(args.version, args.source_sha, args.workflow_sha, args.toolchain,
read_json(args.console), args.run_id, args.run_number, read_json(args.cli), read_json(args.previous),
{"run_id": args.ci_run_id, "run_attempt": args.ci_run_attempt}))
elif args.command == "validate-inputs":
inputs = validate_inputs(read_json(args.inputs))
if args.check_checkout:
head = command(["git", "rev-parse", "HEAD"], cwd=ROOT)
require(head == inputs["source_sha"], "checkout differs from the frozen source")
require(recipe_digest(head) == inputs["recipe_sha256"], "release recipe differs from frozen inputs")
require(file_digest(ROOT / "Cargo.lock") == inputs["lock_sha256"], "dependency lock differs from frozen inputs")
print(digest(inputs))
elif args.command == "record":
write_json(args.output, archive_record(args.archive, args.target, read_json(args.inputs)))
elif args.command == "assemble":
inputs = validate_inputs(read_json(args.inputs))
directory = Path(args.directory)
records = [read_json(path) for path in sorted(directory.glob("record-*.json"))]
write_json(args.output, certification_candidate(inputs, records, directory, targets(args.platforms)))
elif args.command == "download":
run = validate_run(github(f"repos/{REPOSITORY}/actions/runs/{args.run_id}"), args.workflow)
download_artifact(run, args.name, args.destination)
return 0
if __name__ == "__main__":
try:
raise SystemExit(main())
except (ValueError, OSError, KeyError, TypeError, zipfile.BadZipFile, subprocess.CalledProcessError) as error:
print(f"candidate rejected: {error}", file=sys.stderr)
raise SystemExit(1) from None
+200
View File
@@ -0,0 +1,200 @@
#!/usr/bin/env python3
# Copyright 2026 RustFS Team
# SPDX-License-Identifier: Apache-2.0
"""Wrap accepted binaries, then inspect the actual packages and OCI images."""
from __future__ import annotations
import argparse
import json
import os
from pathlib import Path
import shutil
import subprocess
import tarfile
import tempfile
import urllib.request
import time
import candidate as c
import pipeline
def run(args, **kwargs):
return subprocess.run(args, check=True, timeout=1800, **kwargs)
def binary(inputs, target, output):
origin = pipeline.get_run(inputs["origin_run_id"], "release-candidate.yml")
c.download_artifact(origin, f"candidate-{inputs['candidate_id']}-{target}", output)
record = c.read_json(output / f"record-{target}.json")
c.verify_records(inputs, [record], output, (target,))
pipeline.verify_evidence(inputs, output, {origin["id"]: origin})
import verify
path, _ = verify.unpack(output / record["name"], target, inputs, output / "bin")
return path, record
def reuse(kind, target):
inputs = pipeline.manifest_env()
current = pipeline.current_run("release-complete.yml")
name = f"{kind}-{target}"
exists = any(a["name"] == name for a in c.artifact_list(current["id"]))
if exists:
with tempfile.TemporaryDirectory() as temp:
c.download_artifact(current, name, temp)
record = c.read_json(Path(temp) / f"{name}.json")
c.require(record.get("inputs_sha256") == c.digest(inputs), "existing delivery belongs to another candidate")
# Successful jobs are immutable. The final seal independently checks
# all sizes and digests before they can be approved for publication.
c.emit_output("reused", str(exists).lower())
def package(inputs, target, output):
output.mkdir(parents=True, exist_ok=True)
with tempfile.TemporaryDirectory() as temp:
temp = Path(temp)
executable, record = binary(inputs, target, temp / "source")
arch, libc = target.split("-")[1:]
build_type = "prerelease" if "-" in inputs["version"] else "release"
values = c.command(["bash", "scripts/release/package_versions.sh", build_type, inputs["version"], "", arch, libc], cwd=c.ROOT)
versions = dict(line.split("=", 1) for line in values.splitlines())
# Existing package recipes operate in a private working directory.
for name in ("scripts", "deploy"):
(temp / name).symlink_to(c.ROOT / name, target_is_directory=True)
for name in ("LICENSE", "README.md"):
shutil.copyfile(c.ROOT / name, temp / name)
(temp / "bin").mkdir()
shutil.copyfile(executable, temp / "bin/rustfs")
(temp / "bin/rustfs").chmod(0o755)
env = dict(os.environ, **{key.upper(): value for key, value in versions.items()}, LIBC=libc,
DEB_ARCH="amd64" if arch == "x86_64" else "arm64", RPM_ARCH=arch,
DEB_DEPENDS="libc6 (>= 2.31)" if libc == "gnu" else "",
RPM_DEPENDS="glibc >= 2.31" if libc == "gnu" else "", GITHUB_OUTPUT=str(temp / "outputs"))
for kind in ("deb", "rpm"):
run(["bash", str(c.ROOT / f"scripts/release/build_{kind}.sh")], cwd=temp, env=env)
path = temp / versions[f"{kind}_file"]
unpacked = temp / f"unpacked-{kind}"
unpacked.mkdir()
if kind == "deb":
run(["dpkg-deb", "--extract", str(path), str(unpacked)])
else:
with subprocess.Popen(["rpm2cpio", str(path)], stdout=subprocess.PIPE) as source:
run(["cpio", "--extract", "--make-directories", "--quiet", "./usr/bin/rustfs"], stdin=source.stdout, cwd=unpacked)
source.stdout.close()
c.require(source.wait(timeout=60) == 0, "RPM extraction failed")
c.require(c.file_digest(unpacked / "usr/bin/rustfs") == record["binary_sha256"], "package binary differs from accepted ZIP")
shutil.copyfile(path, output / path.name)
files = pipeline.file_records(output)
c.write_json(output / f"packages-{target}.json", {"inputs_sha256": c.digest(inputs), "binary_sha256": record["binary_sha256"], "files": files})
def image_prepare(inputs, variant, output):
output.mkdir(parents=True, exist_ok=True)
binaries = {}
for arch, docker_arch in (("x86_64", "amd64"), ("aarch64", "arm64")):
target = f"linux-{arch}-{variant}"
with tempfile.TemporaryDirectory() as temp:
temp = Path(temp)
executable, record = binary(inputs, target, temp / "source")
directory = c.ROOT / ".release-inputs" / docker_arch
directory.mkdir(parents=True, exist_ok=True)
shutil.copyfile(executable, directory / "rustfs")
(directory / "rustfs").chmod(0o755)
cli = inputs["cli"][target]
archive = temp / "cli.tar.gz"
run(["curl", "--fail", "--location", "--retry", "3", "--output", str(archive), cli["url"]])
c.require(c.file_digest(archive) == cli["sha256"], "CLI dependency changed")
with tarfile.open(archive, "r:gz") as package:
members = [m for m in package.getmembers() if m.name == "rc"]
c.require(len(members) == 1 and members[0].isfile() and 0 < members[0].size <= c.MAX_MEMBER, "invalid CLI archive")
with package.extractfile(members[0]) as source, (directory / "rc").open("wb") as dest:
shutil.copyfileobj(source, dest)
(directory / "rc").chmod(0o755)
binaries[arch] = record["binary_sha256"]
c.write_json(output / f"image-{variant}-inputs.json", {"inputs_sha256": c.digest(inputs), "binaries": binaries})
def image_smoke(inputs, variant, output):
archive = output / f"image-{variant}.tar"
record = c.read_json(output / f"image-{variant}-inputs.json")
scan_directory = Path(str(output) + "-scan")
scan_directory.mkdir(parents=True, exist_ok=True)
opener = urllib.request.build_opener(urllib.request.ProxyHandler({}))
for arch, docker_arch in (("x86_64", "amd64"), ("aarch64", "arm64")):
image = f"rustfs-acceptance:{variant}-{docker_arch}"
run(["skopeo", "copy", "--override-os", "linux", "--override-arch", docker_arch, f"oci-archive:{archive}", f"docker-daemon:{image}"])
container = c.command(["docker", "create", "--platform", f"linux/{docker_arch}", image])
try:
with tempfile.TemporaryDirectory() as temp:
path = Path(temp) / "rustfs"
run(["docker", "cp", f"{container}:/usr/bin/rustfs", str(path)])
c.require(c.file_digest(path) == record["binaries"][arch], "image wraps a different executable")
finally:
run(["docker", "rm", "-v", container])
version = c.command(["docker", "run", "--rm", "--platform", f"linux/{docker_arch}", "--entrypoint", "/usr/bin/rustfs", image, "--version"])
c.require(version.splitlines()[0] == f"rustfs {inputs['version']}", "container displays wrong version")
container = c.command(["docker", "run", "-d", "--platform", f"linux/{docker_arch}", "-p", "127.0.0.1::9001",
"-e", "RUSTFS_ACCESS_KEY=release-smoke", "-e", "RUSTFS_SECRET_KEY=release-smoke-test-secret",
"-e", "RUSTFS_CONSOLE_ENABLE=true", image])
log = output / f"image-{variant}-{docker_arch}.log"
try:
address = c.command(["docker", "port", container, "9001/tcp"]).strip()
deadline = time.monotonic() + 60
while time.monotonic() < deadline:
try:
with opener.open(f"http://{address}/rustfs/console/", timeout=2) as response:
c.require(response.status == 200 and "text/html" in response.headers.get("Content-Type", ""), "image Console did not return HTML")
break
except OSError:
time.sleep(0.5)
else:
raise ValueError("container startup timed out")
finally:
with log.open("w") as stream:
subprocess.run(["docker", "logs", container], stdout=stream, stderr=subprocess.STDOUT, check=True, timeout=60)
run(["docker", "rm", "-f", "-v", container])
with log.open("a") as stream:
stream.write("\nConsole: HTTP 200 text/html; wrapped executable SHA-256 and version verified\n")
# Trivy's local archive reader does not reliably select a platform from
# a multi-architecture OCI index. Export the image just verified above.
scan_archive = scan_directory / f"image-{variant}-{docker_arch}.tar"
run(["docker", "save", "--output", str(scan_archive), image])
record.setdefault("scan_inputs", {})[docker_arch] = c.file_digest(scan_archive)
raw = subprocess.check_output(["skopeo", "inspect", "--raw", f"oci-archive:{archive}"], timeout=60)
import hashlib
record.update(digest="sha256:" + hashlib.sha256(raw).hexdigest(), sha256=c.file_digest(archive), size=archive.stat().st_size, smoke="pass")
c.write_json(output / f"image-{variant}-smoke.json", record)
def image_record(inputs, variant, output):
record = c.read_json(output / f"image-{variant}-smoke.json")
c.require(record["inputs_sha256"] == c.digest(inputs), "image inputs changed")
for arch in ("amd64", "arm64"):
scan_archive = Path(str(output) + "-scan") / f"image-{variant}-{arch}.tar"
c.require(record.get("scan_inputs", {}).get(arch) == c.file_digest(scan_archive), "scan input changed after image verification")
report = c.read_json(output / f"scan-{variant}-{arch}.sarif")
c.require(report.get("version") == "2.1.0" and report.get("runs"), "container scan did not produce a SARIF report")
record["scan"] = "completed"
c.write_json(output / f"image-{variant}.json", record)
def main():
p = argparse.ArgumentParser(description=__doc__)
p.add_argument("mode", choices=("reuse", "package", "image-prepare", "image-smoke", "image-record"))
p.add_argument("--kind", choices=("packages", "image"))
p.add_argument("--target", required=True)
p.add_argument("--output", type=Path, default=Path("delivery"))
args = p.parse_args()
inputs = pipeline.manifest_env()
if args.mode == "reuse":
reuse(args.kind, args.target)
elif args.mode == "package":
c.require(args.target in c.LINUX, "unknown package platform")
package(inputs, args.target, args.output)
else:
c.require(args.target in ("musl", "gnu"), "unknown image variant")
{"image-prepare": image_prepare, "image-smoke": image_smoke, "image-record": image_record}[args.mode](inputs, args.target, args.output)
if __name__ == "__main__":
main()
+361
View File
@@ -0,0 +1,361 @@
#!/usr/bin/env python3
# Copyright 2026 RustFS Team
# SPDX-License-Identifier: Apache-2.0
"""Candidate orchestration. GitHub run identity is checked before trusting bytes."""
from __future__ import annotations
import argparse
import hashlib
import json
import os
from pathlib import Path
import re
import shutil
import subprocess
import tempfile
import urllib.error
import zipfile
import candidate as c
import release_gate
def current_run(workflow):
c.require(os.environ.get("GITHUB_REPOSITORY") == c.REPOSITORY and os.environ.get("GITHUB_REF") == "refs/heads/main",
"release orchestration must run on the upstream main branch")
run = c.validate_run(c.github(f"repos/{c.REPOSITORY}/actions/runs/{int(os.environ['GITHUB_RUN_ID'])}"), workflow, completed=False)
c.require(run["head_sha"] == os.environ["GITHUB_SHA"], "workflow source changed")
return run
def get_run(run_id, workflow):
run = c.github(f"repos/{c.REPOSITORY}/actions/runs/{int(run_id)}")
active = str(run_id) == os.environ.get("GITHUB_RUN_ID")
return c.validate_run(run, workflow, completed=not active)
def manifest_env():
return c.validate_inputs(json.loads(os.environ["CANDIDATE_MANIFEST"]))
def output_inputs(inputs):
c.emit_output("manifest", json.dumps(inputs, separators=(",", ":")))
for field in ("candidate_id", "source_sha", "workflow_sha", "version"):
c.emit_output(field, inputs[field])
def source_ci(source, version):
runs = c.github(f"repos/{c.REPOSITORY}/actions/workflows/ci.yml/runs?head_sha={source}&status=success&per_page=10")["workflow_runs"]
required = {"Quick Checks", "Test and Lint", "End-to-End Tests (full merge gate)"}
for run in runs:
if (run.get("head_sha") != source or run.get("path") != ".github/workflows/ci.yml"
or run.get("head_branch") not in ("main", f"release/{version}")
or run.get("event") not in ("push", "workflow_dispatch")
or run.get("conclusion") != "success" or run.get("status") != "completed"
or (run.get("repository") or {}).get("full_name") != c.REPOSITORY):
continue
jobs = []
for page in range(1, 11):
batch = c.github(f"repos/{c.REPOSITORY}/actions/runs/{run['id']}/attempts/{run['run_attempt']}/jobs?per_page=100&page={page}")["jobs"]
jobs.extend(batch)
if len(batch) < 100:
break
selected = {name: [job for job in jobs if job.get("name") == name] for name in required}
if all(len(found) == 1 and found[0].get("status") == "completed" and found[0].get("conclusion") == "success" for found in selected.values()):
return {"run_id": run["id"], "run_attempt": run["run_attempt"]}
raise ValueError("no complete exact-source CI found; run ci.yml on main or the release branch before creating a candidate")
def unused_version(version):
# Reject a spent version before allocating native builders. A partial
# publication resumes its existing seal instead of making a new candidate.
for tag in (version, "v" + version):
for endpoint in ("git/ref/tags", "releases/tags"):
try:
c.github(f"repos/{c.REPOSITORY}/{endpoint}/{tag}")
except urllib.error.HTTPError as error:
if error.code == 404:
continue
raise
raise ValueError("version already has a tag or release; resume its approved seal or choose a new version")
def prepare(version, source, directory):
run = current_run("release-candidate.yml")
c.require(c.HEX40.fullmatch(source) and c.VERSION.fullmatch(version), "use a full source commit and exact version")
directory.mkdir(parents=True, exist_ok=True)
name = "release-inputs"
exists = any(a["name"] == name for a in c.artifact_list(run["id"]))
if exists:
c.download_artifact(run, name, directory)
inputs = c.validate_inputs(c.read_json(directory / "inputs.json"))
c.require(inputs["source_sha"] == source and inputs["version"] == version and inputs["workflow_sha"] == run["head_sha"], "retry changed candidate inputs")
else:
unused_version(version)
c.command(["git", "fetch", "--no-tags", "origin", source], cwd=c.ROOT)
ci = source_ci(source, version)
c.command(["rustup", "update", "stable", "--no-self-update"])
toolchain = c.command(["rustc", "+stable", "--version"]).split()[1]
inputs = c.freeze(version, source, run["head_sha"], toolchain, c.github("repos/rustfs/console/releases/latest"),
run["id"], int(os.environ["GITHUB_RUN_NUMBER"]), c.github("repos/rustfs/cli/releases/latest"),
c.github("repos/rustfs/rustfs/releases/latest"), ci)
c.write_json(directory / "inputs.json", inputs)
output_inputs(inputs)
c.emit_output("reused", str(exists).lower())
def reuse_platform(target):
inputs = manifest_env()
workflow = os.environ["GITHUB_WORKFLOW_REF"].split("@", 1)[0].rsplit("/", 1)[-1]
run = current_run(workflow)
name = f"candidate-{inputs['candidate_id']}-{target}"
exists = any(a["name"] == name for a in c.artifact_list(run["id"]))
if exists:
with tempfile.TemporaryDirectory() as directory:
c.download_artifact(run, name, directory)
records = [c.read_json(Path(directory) / f"record-{target}.json")]
c.verify_records(inputs, records, directory, (target,))
verify_evidence(inputs, Path(directory), {run["id"]: run})
c.emit_output("reused", str(exists).lower())
def load_linux(run_id, expected, directory):
run = get_run(run_id, "release-candidate.yml")
c.download_artifact(run, "release-linux", directory)
inputs = c.validate_inputs(c.read_json(directory / "inputs.json"))
c.require(inputs["origin_run_id"] == run["id"] and inputs["workflow_sha"] == run["head_sha"], "candidate inputs belong to another run")
c.require(c.HEX64.fullmatch(expected or "") and c.file_digest(directory / "linux-manifest.json") == expected,
"Linux approval digest does not match the downloaded manifest")
verify_seal_files(c.read_json(directory / "linux-manifest.json"), directory)
check_recipe(inputs)
return inputs, run
def check_recipe(inputs):
# Unrelated main changes do not invalidate a frozen candidate. A changed
# producer/verifier/publisher does, even when the product source is unchanged.
c.require(c.recipe_digest(os.environ["GITHUB_SHA"]) == inputs["recipe_sha256"], "release recipe changed; create a new candidate")
def evidence_files(directory):
return sorted(list(directory.glob("identity-*.json")) + list(directory.glob("upgrade-*.json")))
def verify_evidence(inputs, directory, runs):
for path in evidence_files(directory):
value = c.read_json(path)
workflow = value.get("workflow", {})
run = runs.get(int(workflow.get("run_id", 0)))
c.require(run is not None and workflow.get("repository") == c.REPOSITORY
and workflow.get("workflow_id") == run["path"].rsplit("/", 1)[-1]
and workflow.get("workflow_sha") == run["head_sha"]
and type(workflow.get("run_attempt")) is int and 0 < workflow["run_attempt"] <= run["run_attempt"],
"receipt was not produced by the downloaded trusted run")
c.require(value.get("harness_commit") == inputs["workflow_sha"], "receipt harness changed")
for item in value.get("evidence", []):
name = item.get("path", "")
c.require(re.fullmatch(r"[A-Za-z0-9][A-Za-z0-9._-]{0,200}", name), "unsafe evidence path")
log = directory / name
c.require(log.is_file() and not log.is_symlink() and log.stat().st_size == item.get("size")
and c.file_digest(log) == item.get("sha256"), "receipt evidence was removed or substituted")
def certify(inputs, directory, runs, scope):
records = [c.read_json(directory / f"record-{target}.json") for target in c.targets("linux" if scope == "linux" else "all")]
candidate = c.certification_candidate(inputs, records, directory, c.targets("linux" if scope == "linux" else "all"))
plan = c.read_json(c.ROOT / ".config/release-validation-plan.json")
c.require(c.digest(plan) == inputs["plan_sha256"], "validation plan changed")
verify_evidence(inputs, directory, runs)
with tempfile.TemporaryDirectory() as temp:
temp = Path(temp)
results = temp / "results"
results.mkdir()
for path in evidence_files(directory):
shutil.copyfile(path, results / path.name)
c.write_json(temp / "candidate.json", candidate)
c.write_json(temp / "plan.json", plan)
status = release_gate.run_validate(temp / "plan.json", temp / "candidate.json", results, temp / "gate", scope)
summary = c.read_json(temp / "gate/release-gate-summary.json")
c.require(status == 0, "certification failed: " + json.dumps(summary["findings"]))
return candidate, summary
def file_records(directory, excluded=()):
result = []
for path in sorted(directory.iterdir()):
if path.name in excluded:
continue
c.require(path.is_file() and not path.is_symlink() and re.fullmatch(r"[A-Za-z0-9][A-Za-z0-9._-]{0,200}", path.name), "unexpected staged file")
result.append({"name": path.name, "sha256": c.file_digest(path), "size": path.stat().st_size})
return result
def verify_seal_files(seal, directory):
c.require(isinstance(seal, dict) and seal.get("schema") == 1 and seal.get("stage") in ("linux", "complete"), "invalid release seal")
filename = "linux-manifest.json" if seal["stage"] == "linux" else "release-manifest.json"
c.require(seal.get("files") == file_records(directory, (filename,)), "sealed files changed, are missing, or contain extras")
def aliases(inputs):
return {f"rustfs-{target}-latest.zip": f"rustfs-{target}-v{inputs['version']}.zip" for target in c.TARGETS}
def linux_seal(directory):
run = current_run("release-candidate.yml")
if any(a["name"] == "release-linux" for a in c.artifact_list(run["id"])):
c.download_artifact(run, "release-linux", directory)
inputs = c.validate_inputs(c.read_json(directory / "inputs.json"))
verify_seal_files(c.read_json(directory / "linux-manifest.json"), directory)
certify(inputs, directory, {run["id"]: run}, "linux")
c.emit_output("reused", "true")
c.emit_output("manifest_sha256", c.file_digest(directory / "linux-manifest.json"))
return
c.download_artifact(run, "release-inputs", directory)
inputs = c.validate_inputs(c.read_json(directory / "inputs.json"))
for target in c.LINUX:
c.download_artifact(run, f"candidate-{inputs['candidate_id']}-{target}", directory)
c.download_artifact(run, "release-upgrade", directory)
_, summary = certify(inputs, directory, {run["id"]: run}, "linux")
c.require(not summary["publish_allowed"], "partial certification must not grant publication authority")
c.write_json(directory / "linux-manifest.json", {"schema": 1, "stage": "linux", "inputs_sha256": c.digest(inputs),
"files": file_records(directory), "summary": summary})
c.emit_output("manifest_sha256", c.file_digest(directory / "linux-manifest.json"))
def check_acceptance(url):
c.require(re.fullmatch(r"https://github\.com/rustfs/backlog/issues/[1-9][0-9]*(?:#issuecomment-[1-9][0-9]*)?", url),
"acceptance must link to a RustFS backlog report")
def complete_prepare(run_id, expected, report, directory):
current_run("release-complete.yml")
check_acceptance(report)
inputs, run = load_linux(run_id, expected, directory)
certify(inputs, directory, {run["id"]: run}, "linux")
output_inputs(inputs)
def complete_seal(run_id, expected, report, directory):
current = current_run("release-complete.yml")
if any(a["name"] == "release-seal" for a in c.artifact_list(current["id"])):
c.download_artifact(current, "release-seal", directory)
inputs = c.validate_inputs(c.read_json(directory / "inputs.json"))
seal = c.read_json(directory / "release-manifest.json")
verify_seal_files(seal, directory)
c.require(inputs["origin_run_id"] == run_id and seal["acceptance"] == {"linux_manifest_sha256": expected, "report": report}, "completion retry changed acceptance")
origin = get_run(run_id, "release-candidate.yml")
certify(inputs, directory, {origin["id"]: origin, current["id"]: current}, "full")
verify_deliveries(inputs, directory)
c.emit_output("reused", "true")
c.emit_output("manifest_sha256", c.file_digest(directory / "release-manifest.json"))
return
inputs, origin = load_linux(run_id, expected, directory)
check_acceptance(report)
(directory / "linux-manifest.json").unlink()
for target in c.DESKTOP:
c.download_artifact(current, f"candidate-{inputs['candidate_id']}-{target}", directory)
for target in c.LINUX:
c.download_artifact(current, f"packages-{target}", directory)
for variant in ("musl", "gnu"):
c.download_artifact(current, f"image-{variant}", directory)
c.download_artifact(current, "helm-package", directory)
candidate, summary = certify(inputs, directory, {origin["id"]: origin, current["id"]: current}, "full")
verify_deliveries(inputs, directory)
# Persist the small proof bundle with the public release; Actions retention
# must not be the only way to audit an already published version.
evidence = directory / "release-evidence.zip"
with zipfile.ZipFile(evidence, "x", compression=zipfile.ZIP_DEFLATED) as archive:
for path in sorted(directory.iterdir()):
if path.suffix in (".json", ".log", ".sarif"):
archive.write(path, path.name)
archive.writestr("certification.json", json.dumps(summary, indent=2, sort_keys=True))
archive.writestr("validation-plan.json", (c.ROOT / ".config/release-validation-plan.json").read_bytes())
with tempfile.TemporaryDirectory() as temp:
lock = Path(temp) / "Cargo.lock"
lock.write_bytes(c.git_file(inputs["source_sha"], "Cargo.lock"))
c.require(c.file_digest(lock) == inputs["lock_sha256"], "SBOM dependency lock changed")
subprocess.run(["python3", str(c.ROOT / "scripts/security/generate_release_supply_chain_assets.py"),
"--asset-dir", str(directory), "--version", inputs["version"], "--tag", inputs["version"],
"--build-type", candidate["release_profile"], "--repository", c.REPOSITORY,
"--ref", "refs/heads/main", "--sha", inputs["source_sha"], "--run-id", str(current["id"]),
"--run-attempt", str(current["run_attempt"]), "--cargo-lock", str(lock),
"--workflow", "release-complete.yml"], check=True, timeout=60)
provenance = directory / f"rustfs-{inputs['version']}.provenance.json"
value = c.read_json(provenance)
value["predicate"]["buildDefinition"]["externalParameters"]["candidateInputs"] = inputs
provenance.write_text(json.dumps(value, indent=2, sort_keys=True) + "\n")
assets = {p.name: p for p in directory.iterdir() if p.suffix in (".zip", ".deb", ".rpm", ".tgz") or p.name.endswith((".sbom.cdx.json", ".provenance.json"))}
assets.update({name: directory / source for name, source in aliases(inputs).items()})
for algorithm in ("sha256", "sha512"):
checksums = []
for name, path in sorted(assets.items()):
with path.open("rb") as stream:
checksum = hashlib.file_digest(stream, algorithm).hexdigest()
checksums.append(f"{checksum} {name}")
(directory / f"{algorithm.upper()}SUMS").write_text("\n".join(checksums) + "\n")
c.write_json(directory / "release-manifest.json", {
"schema": 1, "stage": "complete", "inputs_sha256": c.digest(inputs), "candidate": candidate,
"summary": summary, "files": file_records(directory), "complete_run_id": current["id"],
"aliases": aliases(inputs),
"acceptance": {"linux_manifest_sha256": expected, "report": report},
})
c.emit_output("manifest_sha256", c.file_digest(directory / "release-manifest.json"))
def verify_deliveries(inputs, directory):
chart = c.read_json(directory / "chart.json")
c.require(chart.get("version") == inputs["version"] and chart.get("inputs_sha256") == c.digest(inputs)
and chart.get("name") == f"rustfs-{inputs['version']}.tgz", "chart identity changed")
path = directory / chart["name"]
c.require(path.stat().st_size == chart["size"] and c.file_digest(path) == chart["sha256"], "chart bytes changed")
for target in c.LINUX:
binary = c.read_json(directory / f"record-{target}.json")["binary_sha256"]
record = c.read_json(directory / f"packages-{target}.json")
c.require(record.get("inputs_sha256") == c.digest(inputs) and record.get("binary_sha256") == binary, "package wraps a different candidate")
c.require(len(record.get("files", [])) == 2 and {Path(f["name"]).suffix for f in record["files"]} == {".deb", ".rpm"}, "missing package format")
for item in record["files"]:
path = directory / item["name"]
c.require(path.name == item["name"] and path.is_file() and path.stat().st_size == item["size"] and c.file_digest(path) == item["sha256"], "package changed")
for variant in ("musl", "gnu"):
record = c.read_json(directory / f"image-{variant}.json")
c.require(record.get("inputs_sha256") == c.digest(inputs), "image inputs changed")
expected = {arch: c.read_json(directory / f"record-linux-{arch}-{variant}.json")["binary_sha256"] for arch in ("x86_64", "aarch64")}
c.require(record.get("binaries") == expected, "image contains an unverified binary")
path = directory / f"image-{variant}.tar"
c.require(c.file_digest(path) == record.get("sha256") and path.stat().st_size == record.get("size")
and re.fullmatch(r"sha256:[0-9a-f]{64}", record.get("digest", "")), "image archive changed")
c.require(record.get("smoke") == "pass" and record.get("scan") == "completed", "image verification did not finish")
def main():
parser = argparse.ArgumentParser(description=__doc__)
sub = parser.add_subparsers(dest="mode", required=True)
p = sub.add_parser("prepare")
p.add_argument("--version", required=True)
p.add_argument("--source", required=True)
p.add_argument("--directory", type=Path, required=True)
p = sub.add_parser("reuse-platform")
p.add_argument("--target", choices=c.TARGETS, required=True)
p = sub.add_parser("linux-seal")
p.add_argument("--directory", type=Path, required=True)
for name in ("complete-prepare", "complete-seal"):
p = sub.add_parser(name)
p.add_argument("--run-id", type=int, required=True)
p.add_argument("--digest", required=True)
p.add_argument("--report", required=True)
p.add_argument("--directory", type=Path, required=True)
args = parser.parse_args()
if args.mode == "prepare":
prepare(args.version, args.source, args.directory)
elif args.mode == "reuse-platform":
reuse_platform(args.target)
elif args.mode == "linux-seal":
linux_seal(args.directory)
elif args.mode == "complete-prepare":
complete_prepare(args.run_id, args.digest, args.report, args.directory)
else:
complete_seal(args.run_id, args.digest, args.report, args.directory)
if __name__ == "__main__":
main()
+280
View File
@@ -0,0 +1,280 @@
#!/usr/bin/env python3
# Copyright 2026 RustFS Team
# SPDX-License-Identifier: Apache-2.0
"""Promote an approved seal. This module never compiles or packages a deliverable."""
from __future__ import annotations
import argparse
import hashlib
import json
import os
from pathlib import Path
import re
import shutil
import subprocess
import tempfile
import candidate as c
import pipeline
REGISTRIES = ("docker.io/rustfs/rustfs", "ghcr.io/rustfs/rustfs", "quay.io/rustfs/rustfs")
def api(path, method="GET", body=None, missing=False):
args = ["gh", "api", "--method", method, path]
if body is not None:
args += ["--input", "-"]
result = subprocess.run(args, input=json.dumps(body) if body is not None else None, text=True,
capture_output=True, timeout=120)
if result.returncode:
if missing and "(HTTP 404)" in result.stderr:
return None
raise ValueError(f"GitHub {method} failed for {path}: {result.stderr.strip()}")
return json.loads(result.stdout) if result.stdout.strip() else None
def verify(run_id, expected, directory):
current = pipeline.current_run("release-publish.yml")
complete = pipeline.get_run(run_id, "release-complete.yml")
c.download_artifact(complete, "release-seal", directory)
manifest_path = directory / "release-manifest.json"
c.require(c.HEX64.fullmatch(expected or "") and c.file_digest(manifest_path) == expected, "publication approval does not match the sealed manifest")
seal = c.read_json(manifest_path)
pipeline.verify_seal_files(seal, directory)
c.require(seal["stage"] == "complete" and seal.get("complete_run_id") == complete["id"], "seal belongs to another completion run")
inputs = c.validate_inputs(c.read_json(directory / "inputs.json"))
c.require(seal.get("inputs_sha256") == c.digest(inputs), "seal inputs changed")
c.require(seal.get("aliases") == pipeline.aliases(inputs), "release aliases differ from the approved versioned archives")
pipeline.check_recipe(inputs)
origin = pipeline.get_run(inputs["origin_run_id"], "release-candidate.yml")
c.require(origin["head_sha"] == inputs["workflow_sha"], "candidate producer changed")
candidate, summary = pipeline.certify(inputs, directory, {origin["id"]: origin, complete["id"]: complete}, "full")
c.require(candidate == seal.get("candidate") and summary == seal.get("summary") and summary["publish_allowed"], "full certification does not match the seal")
pipeline.verify_deliveries(inputs, directory)
pipeline.check_acceptance(seal["acceptance"]["report"])
c.require(c.HEX64.fullmatch(seal["acceptance"].get("linux_manifest_sha256", "")), "missing Linux acceptance digest")
c.emit_output("version", inputs["version"])
c.emit_output("source_sha", inputs["source_sha"])
return inputs, seal, current
def ensure_tag(inputs, manifest_sha):
version, source = inputs["version"], inputs["source_sha"]
message = f"RustFS {version}\nRelease manifest SHA-256: {manifest_sha}\n"
path = f"repos/{c.REPOSITORY}/git/ref/tags/{version}"
reference = api(path, missing=True)
if reference is None:
tag = api(f"repos/{c.REPOSITORY}/git/tags", "POST", {"tag": version, "message": message, "object": source, "type": "commit"})
api(f"repos/{c.REPOSITORY}/git/refs", "POST", {"ref": f"refs/tags/{version}", "sha": tag["sha"]})
reference = api(path)
c.require(reference["object"]["type"] == "tag", "existing release tag is not an annotated candidate tag")
tag = api(f"repos/{c.REPOSITORY}/git/tags/{reference['object']['sha']}")
c.require(tag.get("tag") == version and tag.get("object", {}).get("sha") == source and tag.get("object", {}).get("type") == "commit"
and tag.get("message", "").rstrip() == message.rstrip(), "public tag already identifies different source or bytes; never move it")
def public_files(directory):
names = {"release-manifest.json", "SHA256SUMS", "SHA512SUMS", "release-evidence.zip"}
return sorted(p for p in directory.iterdir() if p.name in names or p.suffix in (".zip", ".deb", ".rpm", ".tgz") or p.name.endswith((".sbom.cdx.json", ".provenance.json")))
def release(inputs, manifest_sha):
endpoint = f"repos/{c.REPOSITORY}/releases"
value = api(endpoint + "/tags/" + inputs["version"], missing=True)
marker = f"Release manifest SHA-256: {manifest_sha}"
if value is None:
notes = api(endpoint + "/generate-notes", "POST", {"tag_name": inputs["version"], "target_commitish": inputs["source_sha"],
"previous_tag_name": inputs["previous"]["version"]})
value = api(endpoint, "POST", {"tag_name": inputs["version"], "target_commitish": inputs["source_sha"],
"name": f"RustFS {inputs['version']}", "body": marker + "\n\n" + notes["body"],
"draft": True, "prerelease": "-" in inputs["version"], "make_latest": "false"})
c.require(marker in value.get("body", ""), "existing release belongs to another sealed candidate")
return value
def github_assets(inputs, value, paths):
assets = []
for page in range(1, 101):
batch = api(f"repos/{c.REPOSITORY}/releases/{value['id']}/assets?per_page=100&page={page}")
assets.extend(batch)
if len(batch) < 100:
break
known = {a["name"]: a for a in assets}
c.require(len(known) == len(assets) and set(known) <= {p.name for p in paths}, "release has unsealed or duplicate assets")
for path in paths:
old = known.get(path.name)
if old is None:
# No --clobber: a race or interrupted upload must fail closed.
subprocess.run(["gh", "release", "upload", inputs["version"], str(path), "--repo", c.REPOSITORY], check=True, timeout=600)
else:
c.require(old.get("digest") == "sha256:" + c.file_digest(path) and old.get("size") == path.stat().st_size,
"existing release asset differs; never overwrite accepted bytes")
actual = api(f"repos/{c.REPOSITORY}/releases/{value['id']}/assets?per_page=100")
c.require(len(actual) == len(paths), "release asset set incomplete")
for path in paths:
match = [a for a in actual if a["name"] == path.name]
c.require(len(match) == 1 and match[0].get("digest") == "sha256:" + c.file_digest(path)
and match[0].get("size") == path.stat().st_size, "uploaded release asset verification failed")
def aws(*args, capture=True):
return subprocess.run(["aws", "--endpoint-url", os.environ["R2_ENDPOINT"], *args],
text=True, capture_output=capture, check=False, timeout=600)
def r2_file(path, key, *, mutable=False):
bucket = os.environ["R2_BUCKET"]
expected = c.file_digest(path)
head = aws("s3api", "head-object", "--bucket", bucket, "--key", key)
if head.returncode and "(404)" not in head.stderr:
raise ValueError("R2 lookup failed; refusing to treat an unavailable destination as empty")
if head.returncode == 0:
with tempfile.TemporaryDirectory() as temp:
old = Path(temp) / "object"
result = aws("s3", "cp", f"s3://{bucket}/{key}", str(old), "--only-show-errors")
c.require(result.returncode == 0, "could not verify existing R2 object")
if c.file_digest(old) == expected:
return
c.require(mutable, "existing versioned R2 object differs; never overwrite it")
args = ["s3api", "put-object", "--bucket", bucket, "--key", key, "--body", str(path), "--metadata", json.dumps({"sha256": expected})]
if not mutable:
args += ["--if-none-match", "*"]
result = aws(*args)
c.require(result.returncode == 0, "R2 upload failed; retry the same seal after checking destination state")
with tempfile.TemporaryDirectory() as temp:
actual = Path(temp) / "object"
result = aws("s3", "cp", f"s3://{bucket}/{key}", str(actual), "--only-show-errors")
c.require(result.returncode == 0 and c.file_digest(actual) == expected, "R2 read-back digest mismatch")
def image_digest(reference, *, missing=False):
result = subprocess.run(["skopeo", "inspect", "--authfile", os.environ["REGISTRY_AUTH_FILE"], "--raw", f"docker://{reference}"],
capture_output=True, timeout=120)
if result.returncode:
error = result.stderr.decode(errors="replace").lower()
if missing and ("manifest unknown" in error or "manifest_unknown" in error or "name unknown" in error):
return None
raise ValueError(f"cannot verify registry manifest for {reference}")
return "sha256:" + hashlib.sha256(result.stdout).hexdigest()
def copy_image(archive, reference, expected, *, mutable=False):
old = image_digest(reference, missing=True)
if old == expected:
return
c.require(old is None or mutable, "existing image tag identifies different bytes; never replace it")
subprocess.run(["skopeo", "copy", "--authfile", os.environ["REGISTRY_AUTH_FILE"], "--all", "--preserve-digests",
f"oci-archive:{archive}", f"docker://{reference}"], check=True, timeout=1800)
c.require(image_digest(reference) == expected, "registry changed the accepted image digest")
def images(inputs, directory, *, latest=False):
for variant, suffix in (("musl", ""), ("gnu", "-glibc")):
record = c.read_json(directory / f"image-{variant}.json")
for registry in REGISTRIES:
tag = ("latest" if latest else inputs["version"]) + suffix
copy_image(directory / f"image-{variant}.tar", f"{registry}:{tag}", record["digest"], mutable=latest)
def publish_version(inputs, seal, directory, manifest_sha):
ensure_tag(inputs, manifest_sha)
value = release(inputs, manifest_sha)
paths = public_files(directory)
with tempfile.TemporaryDirectory() as temp:
alias_paths = []
for name, source in seal["aliases"].items():
path = Path(temp) / name
shutil.copyfile(directory / source, path)
alias_paths.append(path)
github_assets(inputs, value, paths + alias_paths)
for path in paths:
prefix = "artifacts/rustfs/release/"
if path.name in ("release-manifest.json", "release-evidence.zip", "SHA256SUMS", "SHA512SUMS"):
prefix += inputs["version"] + "/"
r2_file(path, prefix + path.name)
images(inputs, directory)
# Every versioned destination is verified before the draft is made public.
if value["draft"]:
api(f"repos/{c.REPOSITORY}/releases/{value['id']}", "PATCH",
{"draft": False, "prerelease": "-" in inputs["version"], "make_latest": "false"})
def stable_key(version):
c.require(isinstance(version, str) and re.fullmatch(r"v?(0|[1-9][0-9]*)\.(0|[1-9][0-9]*)\.(0|[1-9][0-9]*)", version), "invalid stable channel version")
return tuple(map(int, version.lstrip("v").split(".")))
def helm(inputs, directory):
record = c.read_json(directory / "chart.json")
package = directory / record["name"]
c.require(c.file_digest(package) == record["sha256"], "Helm package changed")
env = dict(os.environ, GH_TOKEN=os.environ["HELM_TOKEN"])
subprocess.run(["gh", "auth", "setup-git", "--hostname", "github.com"], env=env, check=True, timeout=60)
with tempfile.TemporaryDirectory() as temp:
subprocess.run(["git", "clone", "--depth", "1", "--branch", "main", "https://github.com/rustfs/helm.git", temp], env=env, check=True, timeout=120)
path = Path(temp) / package.name
if path.exists():
c.require(c.file_digest(path) == record["sha256"], "published Helm version already has different bytes")
c.require((Path(temp) / "index.yaml").is_file(), "existing Helm chart has no index")
return
shutil.copyfile(package, path)
c.command(["helm", "repo", "index", ".", "--url", "https://charts.rustfs.com"], cwd=temp)
c.command(["git", "config", "user.name", "RustFS Release"], cwd=temp)
c.command(["git", "config", "user.email", "dev@rustfs.com"], cwd=temp)
c.command(["git", "add", package.name, "index.yaml"], cwd=temp)
c.command(["git", "commit", "-m", f"chore: publish chart {inputs['version']}"], cwd=temp)
subprocess.run(["git", "push", "origin", "HEAD:main"], cwd=temp, env=env, check=True, timeout=120)
def channels(inputs, seal, directory, manifest_sha):
# Recheck all immutable destinations, including when resuming a failed job.
publish_version(inputs, seal, directory, manifest_sha)
# The versioned Helm catalog entry must still be completed when a newer
# release already owns the mutable channels.
helm(inputs, directory)
if "-" in inputs["version"]:
return
latest = api(f"repos/{c.REPOSITORY}/releases/latest", missing=True)
if latest and stable_key(latest["tag_name"]) > stable_key(inputs["version"]):
print("A newer stable release owns the channels; keeping its pointers.")
return
value = release(inputs, manifest_sha)
# GitHub Latest also acts as a monotonic marker during partial channel
# failures. Versioned assets and images are already fully available here.
api(f"repos/{c.REPOSITORY}/releases/{value['id']}", "PATCH", {"make_latest": "true"})
images(inputs, directory, latest=True)
for target in c.TARGETS:
path = directory / f"rustfs-{target}-v{inputs['version']}.zip"
r2_file(path, f"artifacts/rustfs/release/rustfs-{target}-latest.zip", mutable=True)
pointer = {"version": inputs["version"], "tag": inputs["version"], "release_date": value.get("published_at"),
"release_type": "stable", "download_url": f"https://github.com/{c.REPOSITORY}/releases/tag/{inputs['version']}"}
with tempfile.TemporaryDirectory() as temp:
path = Path(temp) / "latest.json"
c.write_json(path, pointer)
subprocess.run([os.environ["OSSUTIL_BIN"], "cp", str(path), "oss://rustfs-version/latest.json", "--force"], check=True, timeout=120)
actual = Path(temp) / "actual.json"
subprocess.run([os.environ["OSSUTIL_BIN"], "cp", "oss://rustfs-version/latest.json", str(actual)], check=True, timeout=120)
c.require(c.read_json(actual) == pointer, "installer pointer did not match the published release")
def main():
p = argparse.ArgumentParser(description=__doc__)
p.add_argument("mode", choices=("verify", "version", "channels"))
p.add_argument("--run-id", type=int, required=True)
p.add_argument("--digest", required=True)
p.add_argument("--directory", type=Path, required=True)
args = p.parse_args()
inputs, seal, current = verify(args.run_id, args.digest, args.directory)
if args.mode != "verify":
for name in ("R2_ENDPOINT", "R2_BUCKET", "AWS_ACCESS_KEY_ID", "AWS_SECRET_ACCESS_KEY", "REGISTRY_AUTH_FILE"):
c.require(bool(os.environ.get(name)), f"missing publication setting: {name}")
if args.mode == "channels":
c.require(os.environ.get("HELM_TOKEN") and os.environ.get("OSSUTIL_BIN"), "missing channel publication settings")
{"version": publish_version, "channels": channels}[args.mode](inputs, seal, args.directory, args.digest)
c.write_json("publication.json", {"schema": 1, "version": inputs["version"], "manifest_sha256": args.digest,
"source_sha": inputs["source_sha"], "complete_run_id": args.run_id,
"publish_run_id": current["id"], "stage": args.mode})
if __name__ == "__main__":
main()
+35 -9
View File
@@ -263,9 +263,12 @@ def validate_plan(plan):
harness_repository = require_string(
entry, "harness_repository", code, f"plan lane {lane}"
)
harness_sha = require_hex(
entry.get("harness_sha"), HEX40, code, "harness_sha", f"plan lane {lane}"
)
harness_sha = entry.get("harness_sha")
if harness_sha == "candidate.workflow_sha":
if harness_repository != "rustfs/rustfs":
raise GateError(code, "dynamic harness identity is limited to the product repository")
else:
harness_sha = require_hex(harness_sha, HEX40, code, "harness_sha", f"plan lane {lane}")
timeout_seconds = entry.get("timeout_seconds")
if not isinstance(timeout_seconds, int) or timeout_seconds <= 0:
raise GateError(
@@ -500,7 +503,12 @@ def validate_candidate(candidate, plan_info, plan_sha256):
)
image = {"ref": ref, "digest": digest, "binary_sha256": image_binary}
workflow_sha = candidate.get("workflow_sha")
if any(lane["harness_sha"] == "candidate.workflow_sha" for lane in plan_info["lanes"].values()):
workflow_sha = require_hex(workflow_sha, HEX40, code, "workflow_sha", "candidate")
return {
"workflow_sha": workflow_sha,
"product_commit": product_commit,
"version": version,
"release_profile": release_profile,
@@ -616,7 +624,10 @@ def validate_envelope(envelope, plan_info, candidate_info, source_name):
lane_info = plan_info["lanes"].get(lane)
if lane_info is not None:
if harness_commit != lane_info["harness_sha"]:
expected_harness = lane_info["harness_sha"]
if expected_harness == "candidate.workflow_sha":
expected_harness = candidate_info["workflow_sha"]
if harness_commit != expected_harness:
raise GateError(
"harness_mismatch",
f"{context}: harness_commit does not match the fixed harness SHA for lane {lane}",
@@ -658,6 +669,14 @@ def validate_envelope(envelope, plan_info, candidate_info, source_name):
)
artifact = envelope.get("artifact")
if state == "pass" and candidate_info.get("workflow_sha"):
if not isinstance(artifact, dict) or not artifact.get("archive_sha256") or not artifact.get("binary_sha256"):
raise GateError(code, f"{context}: candidate PASS must bind both archive and executable bytes")
target = "-".join(part for part in platform_key(platform) if part not in ("none", "msvc"))
name = f"rustfs-{target}-v{candidate_info['version']}.zip"
matches = [d for d in candidate_info["deliverables"] if d["name"] == name]
if len(matches) != 1 or matches[0]["sha256"] != artifact["archive_sha256"] or matches[0]["binary_sha256"] != artifact["binary_sha256"]:
raise GateError("digest_mismatch", f"{context}: evidence belongs to a different platform artifact")
if artifact is not None:
if not isinstance(artifact, dict):
raise GateError(code, f"{context}: artifact must be an object")
@@ -734,7 +753,7 @@ def _validate_artifact_binding(artifact, candidate_info, context, case_id, lane)
# ---------------------------------------------------------------------------
def run_validate(plan_path, candidate_path, results_dir, output_dir):
def run_validate(plan_path, candidate_path, results_dir, output_dir, scope="full"):
findings = []
plan_info = None
candidate_info = None
@@ -755,6 +774,10 @@ def run_validate(plan_path, candidate_path, results_dir, output_dir):
return 1
required = expand_required(plan_info["lanes"], candidate_info["release_profile"])
if scope == "linux":
required = [case for case in required if case["platform"]["os"] == "linux"]
elif scope != "full":
raise GateError("plan_invalid", "unknown validation scope")
expansion = {
"schema_version": SCHEMA_VERSION,
"plan_sha256": plan_info["plan_sha256"],
@@ -915,7 +938,7 @@ def run_validate(plan_path, candidate_path, results_dir, output_dir):
)
)
_write_outputs(output, findings, plan_info, candidate_info, required, envelopes)
_write_outputs(output, findings, plan_info, candidate_info, required, envelopes, scope=scope)
return 1 if findings else 0
@@ -939,12 +962,13 @@ def _validate_batch(document, candidate_info, source_name):
return {"run_id": run_id, "run_attempt": run_attempt, "status": status}
def _write_outputs(output, findings, plan_info, candidate_info, required, envelopes):
def _write_outputs(output, findings, plan_info, candidate_info, required, envelopes, scope="full"):
state = "blocked" if findings else "pass"
summary = {
"schema_version": SCHEMA_VERSION,
"state": state,
"publish_allowed": not findings,
"scope": scope,
"publish_allowed": not findings and scope == "full",
"plan_sha256": plan_info["plan_sha256"] if plan_info else None,
"candidate_sha256": candidate_info["candidate_sha256"] if candidate_info else None,
"product_commit": candidate_info["product_commit"] if candidate_info else None,
@@ -972,7 +996,7 @@ def _write_outputs(output, findings, plan_info, candidate_info, required, envelo
"# Release gate summary",
"",
f"- state: {state}",
f"- publish_allowed: {str(not findings).lower()}",
f"- publish_allowed: {str(summary['publish_allowed']).lower()}",
f"- plan_sha256: {summary['plan_sha256']}",
f"- candidate_sha256: {summary['candidate_sha256']}",
f"- required cases: {len(required)}",
@@ -1190,6 +1214,7 @@ def main(argv=None):
validate_parser.add_argument("--candidate", required=True)
validate_parser.add_argument("--results", required=True)
validate_parser.add_argument("--output", required=True)
validate_parser.add_argument("--scope", choices=("full", "linux"), default="full")
promote_parser = subparsers.add_parser(
"verify-promotion", help="re-check to-be-published bytes against the passed list"
@@ -1216,6 +1241,7 @@ def main(argv=None):
arguments.candidate,
arguments.results,
arguments.output,
scope=arguments.scope,
)
except GateError as error:
print(f"{error.code}: {error.message}", file=sys.stderr)
+279
View File
@@ -0,0 +1,279 @@
#!/usr/bin/env python3
# Copyright 2026 RustFS Team
# SPDX-License-Identifier: Apache-2.0
"""Exercise release identity, substitution and retry boundaries without publishing."""
from __future__ import annotations
import copy
import io
import os
import json
from pathlib import Path
import subprocess
import sys
import tempfile
import unittest
from unittest.mock import patch
import urllib.request
import zipfile
import candidate
class TransportTests(unittest.TestCase):
def test_signed_download_redirect_does_not_leak_the_github_token(self):
request = urllib.request.Request("https://api.github.com/repos/rustfs/rustfs/actions/artifacts/1/zip", headers={"Authorization": "Bearer test-token"})
redirected = candidate.ArtifactRedirect().redirect_request(request, None, 302, "Found", {}, "https://example.blob.core.windows.net/artifact.zip?signature=fixture")
self.assertIsNone(redirected.get_header("Authorization"))
same_host = candidate.ArtifactRedirect().redirect_request(request, None, 302, "Found", {}, "https://api.github.com/another-path")
self.assertEqual(same_host.get_header("Authorization"), "Bearer test-token")
with self.assertRaises(ValueError):
candidate.ArtifactRedirect().redirect_request(request, None, 302, "Found", {}, "http://example.invalid/artifact.zip")
def test_download_checks_real_bytes_and_rejects_traversal(self):
import hashlib
run = {"id": 70, "head_sha": "a" * 40}
def blob(name):
stream = io.BytesIO()
with zipfile.ZipFile(stream, "w") as package:
package.writestr(name, b"accepted")
return stream.getvalue()
for name, valid in (("record.json", True), ("../escape", False)):
content = blob(name)
artifact = {"id": 90, "name": "candidate", "expired": False, "workflow_run": run,
"size_in_bytes": len(content), "digest": "sha256:" + hashlib.sha256(content).hexdigest()}
with tempfile.TemporaryDirectory() as temp, patch.object(candidate, "artifact_list", return_value=[artifact]), patch.object(candidate, "github_stream", side_effect=lambda _: io.BytesIO(content)):
if valid:
candidate.download_artifact(run, "candidate", temp)
self.assertEqual((Path(temp) / name).read_bytes(), b"accepted")
artifact["digest"] = "sha256:" + "0" * 64
with self.assertRaisesRegex(ValueError, "digest"):
candidate.download_artifact(run, "candidate", Path(temp) / "changed")
else:
with self.assertRaisesRegex(ValueError, "unsafe"):
candidate.download_artifact(run, "candidate", temp)
def test_duplicate_json_fields_are_not_silently_accepted(self):
with tempfile.TemporaryDirectory() as temp:
path = Path(temp) / "manifest.json"
path.write_text('{"schema": 1, "schema": 2}')
with self.assertRaisesRegex(ValueError, "duplicate"):
candidate.read_json(path)
class FreezeRepositoryTests(unittest.TestCase):
def test_freeze_reads_real_commit_and_rejects_recipe_or_version_drift(self):
with tempfile.TemporaryDirectory() as temp:
root = Path(temp)
for name in candidate.RECIPE_PATHS:
path = root / name; path.parent.mkdir(parents=True, exist_ok=True)
path.write_text("recipe\n" if name != ".config/release-validation-plan.json" else "{}\n")
for directory in ("bootstrap", "helm"):
(root / directory).mkdir()
(root / directory / "input.txt").write_text("runtime input\n")
(root / "Cargo.toml").write_text('[workspace.package]\nversion = "1.2.3"\n')
(root / "Cargo.lock").write_text("version = 4\n")
env = dict(os.environ, GIT_AUTHOR_NAME="Fixture", GIT_AUTHOR_EMAIL="fixture@example.invalid", GIT_COMMITTER_NAME="Fixture", GIT_COMMITTER_EMAIL="fixture@example.invalid")
def git(*args):
return subprocess.check_output(["git", *args], cwd=root, env=env, stderr=subprocess.DEVNULL, text=True).strip()
git("init", "-q"); git("add", "."); git("commit", "-qm", "fixture")
original = git("rev-parse", "HEAD")
value = inputs()
console = {"draft": False, "prerelease": False, "tag_name": value["console"]["tag"],
"assets": [{"name": value["console"]["name"], "browser_download_url": value["console"]["url"], "digest": "sha256:" + value["console"]["sha256"]}]}
cli = {"draft": False, "prerelease": False, "tag_name": "v0.1.0", "assets": []}
for arch in ("amd64", "arm64"):
for abi in ("", "-gnu"):
name = f"rustfs-cli-linux-{arch}{abi}-v0.1.0.tar.gz"
cli["assets"].append({"name": name, "browser_download_url": f"https://github.com/rustfs/cli/releases/download/v0.1.0/{name}", "digest": "sha256:" + "f" * 64})
previous = {"draft": False, "prerelease": False, "tag_name": "1.2.2", "assets": [{"name": value["previous"]["name"], "browser_download_url": value["previous"]["url"], "digest": "sha256:" + value["previous"]["sha256"]}]}
def freeze(source, workflow, version="1.2.3"):
return candidate.freeze(version, source, workflow, "1.92.0", console, 70, 7, cli, previous, {"run_id": 60, "run_attempt": 1}, root)
frozen = freeze(original, original)
(root / "unrelated.txt").write_text("main may advance\n")
git("add", "."); git("commit", "-qm", "unrelated")
advanced = git("rev-parse", "HEAD")
self.assertEqual(freeze(original, advanced)["recipe_sha256"], frozen["recipe_sha256"])
(root / ".github/workflows/build.yml").write_text("different recipe\n")
git("add", "."); git("commit", "-qm", "recipe change")
with self.assertRaisesRegex(ValueError, "recipe"):
freeze(original, git("rev-parse", "HEAD"))
with self.assertRaisesRegex(ValueError, "version"):
freeze(original, original, "1.2.4")
def inputs():
value = {
"schema": 1, "candidate_id": "1.2.3-preview.7", "version": "1.2.3", "ci": {"run_id": 60, "run_attempt": 1},
"source_sha": "a" * 40, "workflow_sha": "b" * 40, "recipe_sha256": "c" * 64,
"lock_sha256": "d" * 64, "plan_sha256": "e" * 64, "toolchain": "1.92.0",
"console": {"tag": "v0.1.41", "name": "rustfs-console-v0.1.41.zip",
"url": "https://github.com/rustfs/console/releases/download/v0.1.41/rustfs-console-v0.1.41.zip",
"sha256": "f" * 64}, "origin_run_id": 70, "origin_run_number": 7,
}
def asset(repo, version, name):
return {"version": version, "name": name, "url": f"https://github.com/{repo}/releases/download/{version}/{name}", "sha256": "f" * 64}
value["previous"] = asset("rustfs/rustfs", "1.2.2", "rustfs-linux-x86_64-gnu-v1.2.2.zip")
value["cli"] = {target: asset("rustfs/cli", "v0.1.0", "rustfs-cli.tar.gz") for target in candidate.LINUX}
return value
def archive(directory, target, value=None, *, extra=None, binary=b"exact candidate bytes"):
value = value or inputs()
path = Path(directory) / f"rustfs-{target}-v{value['version']}.zip"
members = {"rustfs.exe" if target == "windows-x86_64" else "rustfs": binary}
if target.endswith("-gnu"):
members.update({"rustfs.cpu-symbol-catalog.json": b"{}", "rustfs.cpu-symbol-catalog.sha256": b"catalog"})
if extra:
members.update(extra)
with zipfile.ZipFile(path, "w") as output:
for name, content in members.items():
output.writestr(name, content)
return path
class InputIdentityTests(unittest.TestCase):
def test_valid_frozen_candidate(self):
self.assertEqual(candidate.validate_inputs(inputs()), inputs())
def test_moving_or_injected_inputs_rejected(self):
changes = [("source_sha", "main"), ("version", "1.2.3\nTAG=bad"), ("version", "01.2.3"),
("candidate_id", "1.2.3-preview.8"), ("toolchain", "stable"), ("origin_run_id", True),
("schema", True), ("recipe_sha256", "c" * 63)]
for key, value in changes:
with self.subTest(key=key, value=value):
record = inputs(); record[key] = value
with self.assertRaises(ValueError):
candidate.validate_inputs(record)
def test_console_asset_cannot_redirect_or_be_ambiguous(self):
record = inputs()
record["console"]["url"] = "https://example.invalid/console.zip"
with self.assertRaises(ValueError):
candidate.validate_inputs(record)
asset = {"name": inputs()["console"]["name"], "browser_download_url": inputs()["console"]["url"], "digest": "sha256:" + "f" * 64}
for release in ({"draft": True, "prerelease": False, "tag_name": "v0.1.41", "assets": [asset]},
{"draft": False, "prerelease": False, "tag_name": "v0.1.41", "assets": [asset, asset]}):
with self.assertRaises(ValueError):
candidate.console_asset(release)
def test_frozen_record_is_retryable_but_not_replaceable(self):
with tempfile.TemporaryDirectory() as directory:
path = Path(directory) / "inputs.json"
candidate.write_json(path, inputs())
before = path.read_bytes()
candidate.write_json(path, inputs())
self.assertEqual(before, path.read_bytes())
changed = inputs(); changed["source_sha"] = "f" * 40
with self.assertRaisesRegex(ValueError, "replace"):
candidate.write_json(path, changed)
self.assertEqual(before, path.read_bytes())
def test_platform_selection_keeps_desktop_out_of_linux(self):
self.assertEqual(candidate.targets("linux"), candidate.LINUX)
self.assertEqual(candidate.targets("desktop"), candidate.DESKTOP)
for value in ("", "linux-x86_64-gnu,linux-x86_64-gnu", "linux,windows", "all,linux"):
with self.subTest(value=value), self.assertRaises(ValueError):
candidate.targets(value)
class ArtifactIdentityTests(unittest.TestCase):
def setUp(self):
self.temp = tempfile.TemporaryDirectory(); self.addCleanup(self.temp.cleanup)
self.root = Path(self.temp.name)
self.inputs = inputs()
self.records = [candidate.archive_record(archive(self.root, target), target, self.inputs) for target in candidate.TARGETS]
def test_all_six_final_artifacts_bind_exact_bytes(self):
value = candidate.certification_candidate(self.inputs, self.records, self.root)
self.assertEqual(len(value["deliverables"]), 6)
self.assertEqual(value["version"], "1.2.3")
self.assertEqual(value["release_profile"], "stable")
self.assertEqual(value["validation_batch"], {"run_id": "1.2.3-preview.7", "run_attempt": 1})
self.assertNotIn("latest", json.dumps(value))
def test_partial_linux_bundle_cannot_be_published_as_complete(self):
linux = self.records[:4]
self.assertEqual(len(candidate.certification_candidate(self.inputs, linux, self.root, candidate.LINUX)["deliverables"]), 4)
with self.assertRaisesRegex(ValueError, "missing"):
candidate.certification_candidate(self.inputs, linux, self.root)
def test_same_version_different_bytes_rejected(self):
archive(self.root, candidate.TARGETS[0], binary=b"replacement")
with self.assertRaisesRegex(ValueError, "changed"):
candidate.verify_records(self.inputs, self.records, self.root, candidate.TARGETS)
def test_different_inputs_cannot_share_receipts(self):
changed = copy.deepcopy(self.inputs); changed["console"]["sha256"] = "0" * 64
with self.assertRaisesRegex(ValueError, "changed"):
candidate.verify_records(changed, self.records, self.root, candidate.TARGETS)
def test_duplicate_or_mislabeled_platform_rejected(self):
for replacement in (self.records[:-1] + [self.records[0]], self.records + [self.records[0]]):
with self.assertRaises(ValueError):
candidate.verify_records(self.inputs, replacement, self.root, candidate.TARGETS)
wrong = copy.deepcopy(self.records); wrong[0]["name"] = "../replacement.zip"
with self.assertRaises(ValueError):
candidate.verify_records(self.inputs, wrong, self.root, candidate.TARGETS)
def test_archive_members_are_strict(self):
for name in ("../escape", "rustfs-cli", "nested/rustfs"):
with self.subTest(name=name):
path = archive(self.root, candidate.TARGETS[0], extra={name: b"extra"})
with self.assertRaises(ValueError):
candidate.archive_record(path, candidate.TARGETS[0], self.inputs)
def test_binary_symlink_rejected(self):
path = self.root / f"rustfs-{candidate.TARGETS[0]}-v1.2.3.zip"
with zipfile.ZipFile(path, "w") as output:
entry = zipfile.ZipInfo("rustfs"); entry.external_attr = 0o120777 << 16
output.writestr(entry, "../../untrusted")
with self.assertRaises(ValueError):
candidate.archive_record(path, candidate.TARGETS[0], self.inputs)
def test_real_cli_fails_without_overwriting_existing_output(self):
candidate.write_json(self.root / "inputs.json", self.inputs)
for item in self.records:
candidate.write_json(self.root / f"record-{item['target']}.json", item)
output = self.root / "candidate.json"
command = [sys.executable, str(Path(candidate.__file__)), "assemble", "--inputs", str(self.root / "inputs.json"),
"--directory", str(self.root), "--output", str(output)]
good = subprocess.run(command, text=True, capture_output=True)
self.assertEqual(good.returncode, 0, good.stderr)
original = output.read_bytes()
archive(self.root, candidate.TARGETS[1], binary=b"bad retry")
bad = subprocess.run(command, text=True, capture_output=True)
self.assertNotEqual(bad.returncode, 0)
self.assertEqual(output.read_bytes(), original)
class ProvenanceTests(unittest.TestCase):
def run_record(self):
return {"repository": {"full_name": "rustfs/rustfs"}, "path": ".github/workflows/release-candidate.yml",
"event": "workflow_dispatch", "head_branch": "main", "id": 70, "run_attempt": 1,
"head_sha": "b" * 40, "status": "completed", "conclusion": "success"}
def test_trusted_source_and_completion_required(self):
run = self.run_record()
self.assertEqual(candidate.validate_run(run, "release-candidate.yml"), run)
for field, value in (("head_branch", "unreviewed"), ("event", "pull_request"), ("conclusion", "failure"),
("status", "in_progress"), ("repository", {"full_name": "fork/rustfs"}),
("path", ".github/workflows/other.yml")):
with self.subTest(field=field):
changed = {**run, field: value}
with self.assertRaises(ValueError):
candidate.validate_run(changed, "release-candidate.yml")
def test_artifact_must_match_exact_origin(self):
run = self.run_record()
artifact = {"name": "candidate", "expired": False, "id": 1, "size_in_bytes": 100,
"digest": "sha256:" + "c" * 64, "workflow_run": {"id": 70, "head_sha": "b" * 40}}
candidate.validate_artifact(artifact, run, "candidate")
for field, value in (("expired", True), ("name", "other"), ("digest", ""), ("size_in_bytes", True),
("workflow_run", {"id": 71, "head_sha": "b" * 40})):
with self.subTest(field=field), self.assertRaises(ValueError):
candidate.validate_artifact({**artifact, field: value}, run, "candidate")
if __name__ == "__main__":
unittest.main()
+453
View File
@@ -0,0 +1,453 @@
#!/usr/bin/env python3
# Copyright 2026 RustFS Team
# SPDX-License-Identifier: Apache-2.0
"""Offline contract rehearsals; these are not product acceptance receipts."""
from __future__ import annotations
import copy
import hashlib
import json
import os
from pathlib import Path
import re
import subprocess
import sys
import tempfile
import unittest
from unittest.mock import patch
import zipfile
import candidate as c
import pipeline
import publish
import release_gate
import verify
from test_candidate import archive, inputs
ROOT = c.ROOT
def workflow(name):
# Ruby/Psych is available on the developer host and hosted CI image. Tests
# parse the actual workflow rather than a separately maintained job graph.
result = subprocess.check_output(["ruby", "-ryaml", "-rjson", "-e",
"puts JSON.generate(YAML.load_file(ARGV[0]))", str(ROOT / ".github/workflows" / name)], text=True)
value = json.loads(result)
value["on"] = value.pop("true", value.get("on"))
return value
def run_metadata(run_id, file, sha):
return {"id": run_id, "run_attempt": 2, "head_sha": sha, "head_branch": "main", "event": "workflow_dispatch",
"status": "completed", "conclusion": "success", "path": f".github/workflows/{file}",
"repository": {"full_name": c.REPOSITORY}}
def fixture(directory, selection="all"):
value = inputs()
value["plan_sha256"] = c.digest(c.read_json(ROOT / ".config/release-validation-plan.json"))
runs = {70: run_metadata(70, "release-candidate.yml", value["workflow_sha"]),
80: run_metadata(80, "release-complete.yml", "c" * 40)}
for target in c.targets(selection):
record = c.archive_record(archive(directory, target, value, binary=target.encode()), target, value)
c.write_json(directory / f"record-{target}.json", record)
run = runs[70 if target in c.LINUX else 80]
env = {"GITHUB_WORKFLOW_REF": f"{c.REPOSITORY}/{run['path']}@refs/heads/main", "GITHUB_SHA": run["head_sha"],
"GITHUB_RUN_ID": str(run["id"]), "GITHUB_RUN_ATTEMPT": "1"}
with patch.dict(os.environ, env):
for case in ("artifact-identity", "startup"):
log = directory / f"{target}-{case}.log"
log.write_text("contract fixture only\n")
verify.envelope(value, record, "identity", case, [log], verify.now(), directory)
if target == "linux-x86_64-gnu":
for case in verify.UPGRADES:
log = directory / f"upgrade-{case}.log"
log.write_text("contract fixture only\n")
verify.envelope(value, record, "upgrade", case, [log], verify.now(), directory)
return value, runs
class CertificationTests(unittest.TestCase):
def setUp(self):
self.temp = tempfile.TemporaryDirectory()
self.addCleanup(self.temp.cleanup)
self.path = Path(self.temp.name)
self.value, self.runs = fixture(self.path)
def test_complete_original_bytes_certify(self):
_, summary = pipeline.certify(self.value, self.path, self.runs, "full")
self.assertTrue(summary["publish_allowed"])
self.assertEqual(len(summary["required_cases"]), 18)
def test_linux_only_cannot_authorize_release(self):
with tempfile.TemporaryDirectory() as temp:
value, runs = fixture(Path(temp), "linux")
candidate, summary = pipeline.certify(value, Path(temp), runs, "linux")
self.assertEqual(summary["state"], "pass")
self.assertFalse(summary["publish_allowed"])
self.assertEqual(len(summary["required_cases"]), 14)
self.assertEqual(len(candidate["deliverables"]), 4)
with self.assertRaises((ValueError, OSError)):
pipeline.certify(value, Path(temp), runs, "full")
def mutate_envelope(self, edit):
path = self.path / "identity-linux-x86_64-gnu-startup.json"
value = c.read_json(path)
edit(value)
path.write_text(json.dumps(value))
def test_digest_binding_is_required_and_platform_specific(self):
original = (self.path / "identity-linux-x86_64-gnu-startup.json").read_bytes()
arm = c.read_json(self.path / "record-linux-aarch64-gnu.json")
edits = [lambda v: v.pop("artifact"), lambda v: v["artifact"].update(archive_sha256="0" * 64),
lambda v: v["artifact"].update(archive_sha256=arm["sha256"], binary_sha256=arm["binary_sha256"])]
for edit in edits:
with self.subTest(edit=edit):
(self.path / "identity-linux-x86_64-gnu-startup.json").write_bytes(original)
self.mutate_envelope(edit)
with self.assertRaises(ValueError):
pipeline.certify(self.value, self.path, self.runs, "full")
def test_missing_changed_and_duplicate_evidence_are_rejected(self):
path = self.path / "linux-x86_64-gnu-startup.log"
path.write_text("substituted")
with self.assertRaisesRegex(ValueError, "evidence"):
pipeline.certify(self.value, self.path, self.runs, "full")
def test_receipt_cannot_claim_another_run_or_new_attempt(self):
self.mutate_envelope(lambda v: v["workflow"].update(run_attempt=3))
with self.assertRaisesRegex(ValueError, "trusted run"):
pipeline.certify(self.value, self.path, self.runs, "full")
def test_old_successful_unit_survives_failed_job_retry(self):
# The candidate batch is immutable; successful unit receipts from an
# earlier attempt still bind the exact archive, plan and producer.
_, summary = pipeline.certify(self.value, self.path, self.runs, "full")
self.assertEqual(summary["state"], "pass")
self.mutate_envelope(lambda v: v.update(run_id="different-candidate"))
with self.assertRaises(ValueError):
pipeline.certify(self.value, self.path, self.runs, "full")
def test_new_verifier_or_plan_cannot_reuse_old_acceptance(self):
self.value["workflow_sha"] = "d" * 40
with self.assertRaises(ValueError):
pipeline.certify(self.value, self.path, self.runs, "full")
class SourceCITests(unittest.TestCase):
def test_spent_versions_and_unavailable_lookups_fail_before_building(self):
import urllib.error
missing = urllib.error.HTTPError('https://api.github.com/fixture', 404, 'Not Found', {}, None)
with patch.object(c, "github", side_effect=missing) as api:
pipeline.unused_version("1.2.3")
self.assertEqual(api.call_count, 4)
with patch.object(c, "github", return_value={"id": 1}):
with self.assertRaisesRegex(ValueError, "already has a tag or release"):
pipeline.unused_version("1.2.3")
unavailable = urllib.error.HTTPError('https://api.github.com/fixture', 503, 'Unavailable', {}, None)
with patch.object(c, "github", side_effect=unavailable):
with self.assertRaises(urllib.error.HTTPError):
pipeline.unused_version("1.2.3")
def test_only_complete_exact_source_ci_is_accepted(self):
run = run_metadata(60, "ci.yml", "a" * 40)
jobs = [{"name": name, "status": "completed", "conclusion": "success"} for name in
("Quick Checks", "Test and Lint", "End-to-End Tests (full merge gate)")]
def api(path):
return {"jobs": jobs} if "/jobs?" in path else {"workflow_runs": [run]}
with patch.object(c, "github", side_effect=api):
self.assertEqual(pipeline.source_ci("a" * 40, "1.2.3"), {"run_id": 60, "run_attempt": 2})
for key, wrong in (("head_sha", "b" * 40), ("head_branch", "unreviewed"), ("event", "pull_request"), ("conclusion", "failure")):
old = run[key]; run[key] = wrong
with self.subTest(key=key), self.assertRaises(ValueError):
pipeline.source_ci("a" * 40, "1.2.3")
run[key] = old
for conclusion in ("skipped", "failure", None):
jobs[2]["conclusion"] = conclusion
with self.subTest(conclusion=conclusion), self.assertRaises(ValueError):
pipeline.source_ci("a" * 40, "1.2.3")
def test_duplicate_required_ci_job_is_rejected(self):
run = run_metadata(60, "ci.yml", "a" * 40)
jobs = [{"name": name, "status": "completed", "conclusion": "success"} for name in
("Quick Checks", "Quick Checks", "Test and Lint", "End-to-End Tests (full merge gate)")]
with patch.object(c, "github", side_effect=lambda path: {"jobs": jobs} if "/jobs?" in path else {"workflow_runs": [run]}):
with self.assertRaises(ValueError):
pipeline.source_ci("a" * 40, "1.2.3")
class SupplyChainTests(unittest.TestCase):
def test_actual_generator_uses_frozen_lock_and_all_package_formats(self):
with tempfile.TemporaryDirectory() as temp:
root = Path(temp)
assets = root / "assets"; assets.mkdir()
for name in ("server.zip", "package.deb", "package.rpm", "chart.tgz"):
(assets / name).write_bytes(name.encode())
lock = root / "Cargo.lock"
lock.write_text('version = 4\n[[package]]\nname = "frozen-fixture"\nversion = "1.2.3"\n')
args = [sys.executable, str(ROOT / "scripts/security/generate_release_supply_chain_assets.py"), "--asset-dir", str(assets),
"--version", "1.2.3", "--cargo-lock", str(lock), "--workflow", "release-complete.yml", "--sha", "a" * 40]
result = subprocess.run(args, capture_output=True, text=True, timeout=10)
self.assertEqual(result.returncode, 0, result.stderr)
provenance = c.read_json(assets / "rustfs-1.2.3.provenance.json")
self.assertEqual({s["name"] for s in provenance["subject"]}, {"server.zip", "package.deb", "package.rpm", "chart.tgz"})
self.assertTrue(provenance["predicate"]["buildDefinition"]["externalParameters"]["workflow"].endswith("/release-complete.yml"))
sbom = c.read_json(assets / "rustfs-1.2.3.sbom.cdx.json")
self.assertIn("frozen-fixture", {item["name"] for item in sbom["components"]})
lock.unlink()
result = subprocess.run(args, capture_output=True, text=True, timeout=10)
self.assertNotEqual(result.returncode, 0)
class NativeEntryTests(unittest.TestCase):
def test_real_native_entry_point_version_startup_and_failure(self):
# A small child process exercises the verifier entry point and lifecycle;
# it deliberately does not pretend to be a RustFS product test.
target = "macos-aarch64"
value = inputs()
version = f"rustfs {value['version']}\nbuild target : {verify.TRIPLES[target]}\ngit commit : {value['source_sha']}\nrust version : {value['toolchain']}\n"
source = f'''#!/usr/bin/env python3
import sys, threading
from http.server import BaseHTTPRequestHandler, HTTPServer
store = {{}}
if '--version' in sys.argv:
print({version!r}, end='')
else:
class Handler(BaseHTTPRequestHandler):
def do_GET(self):
self.send_response(200)
self.send_header('Content-Type', 'text/html')
self.end_headers()
self.wfile.write(store.get(self.path, b'<html>fixture</html>'))
def do_PUT(self):
store[self.path] = self.rfile.read(int(self.headers.get('Content-Length', '0')))
self.send_response(200); self.end_headers()
def do_DELETE(self):
store.pop(self.path, None)
self.send_response(204); self.end_headers()
api = sys.argv[sys.argv.index('--address')+1].split(':')
server = HTTPServer((api[0], int(api[1])), Handler)
threading.Thread(target=server.serve_forever, daemon=True).start()
address=sys.argv[sys.argv.index('--console-address')+1]
host, port=address.split(':')
HTTPServer((host,int(port)),Handler).serve_forever()
'''
with tempfile.TemporaryDirectory() as temp:
path = Path(temp)
c.write_json(path / "inputs.json", value)
binary = archive(path, target, value, binary=source.encode())
env = dict(os.environ, GITHUB_WORKFLOW_REF=f"{c.REPOSITORY}/.github/workflows/release-complete.yml@refs/heads/main",
GITHUB_RUN_ID="80", GITHUB_RUN_ATTEMPT="1", GITHUB_SHA="c" * 40)
args = [sys.executable, str(ROOT / "scripts/release/verify.py"), "native", "--inputs", str(path / "inputs.json"),
"--archive", str(binary), "--target", target, "--output", str(path / "receipts")]
result = subprocess.run(args, env=env, capture_output=True, text=True, timeout=30)
self.assertEqual(result.returncode, 0, result.stderr)
self.assertEqual(len(list((path / "receipts").glob("*.json"))), 2)
archive(path, target, value, binary=source.replace("rustfs 1.2.3", "rustfs 1.2.4").encode())
args[-1] = str(path / "rejected")
result = subprocess.run(args, env=env, capture_output=True, text=True, timeout=30)
self.assertNotEqual(result.returncode, 0)
self.assertFalse(list((path / "rejected").glob("*.json")))
class PromotionTests(unittest.TestCase):
def test_tag_is_created_once_and_conflicting_tag_never_moved(self):
value = inputs()
state = {}
calls = []
def api(path, method="GET", body=None, missing=False):
calls.append((path, method, body))
if path.endswith("/git/tags"):
state["tag"] = dict(body, object={"sha": body["object"], "type": body["type"]})
return {"sha": "f" * 40}
if path.endswith("/git/refs"):
state["ref"] = {"object": {"type": "tag", "sha": body["sha"]}}
return state["ref"]
if "/git/ref/tags/" in path:
return state.get("ref")
return state["tag"]
with patch.object(publish, "api", side_effect=api):
publish.ensure_tag(value, "a" * 64)
count = len([call for call in calls if call[1] == "POST"])
publish.ensure_tag(value, "a" * 64)
self.assertEqual(len([call for call in calls if call[1] == "POST"]), count)
with self.assertRaisesRegex(ValueError, "never move"):
publish.ensure_tag(value, "b" * 64)
state["tag"]["object"]["sha"] = "e" * 40
with self.assertRaises(ValueError):
publish.ensure_tag(value, "a" * 64)
self.assertFalse(any(method in ("PATCH", "DELETE") for _, method, _ in calls))
def test_image_promotion_preserves_digest_and_refuses_overwrite(self):
with patch.dict(os.environ, REGISTRY_AUTH_FILE="auth.json"), patch.object(publish, "image_digest", return_value="sha256:" + "a" * 64), patch.object(subprocess, "run") as command:
publish.copy_image(Path("image.tar"), "registry/image:1.2.3", "sha256:" + "a" * 64)
command.assert_not_called()
with self.assertRaisesRegex(ValueError, "never replace"):
publish.copy_image(Path("image.tar"), "registry/image:1.2.3", "sha256:" + "b" * 64)
command.assert_not_called()
with patch.dict(os.environ, REGISTRY_AUTH_FILE="auth.json"), patch.object(publish, "image_digest", side_effect=[None, "sha256:" + "a" * 64]), patch.object(subprocess, "run") as command:
publish.copy_image(Path("image.tar"), "registry/image:1.2.3", "sha256:" + "a" * 64)
self.assertIn("--preserve-digests", command.call_args.args[0])
self.assertIn("--all", command.call_args.args[0])
def test_newer_channel_cannot_be_downgraded_after_old_job_retry(self):
with patch.object(publish, "publish_version") as versions, patch.object(publish, "api", return_value={"tag_name": "1.2.4"}) as api, patch.object(publish, "images") as images, patch.object(publish, "helm") as helm:
publish.channels(inputs(), {}, Path("candidate"), "a" * 64)
versions.assert_called_once()
images.assert_not_called()
helm.assert_called_once()
self.assertTrue(all(call.args[1:] == () for call in api.call_args_list))
def test_prerelease_never_changes_stable_pointers(self):
value = inputs(); value["version"] = "1.3.0-rc.1"
with patch.object(publish, "publish_version"), patch.object(publish, "api") as api, patch.object(publish, "images") as images, patch.object(publish, "helm") as helm:
publish.channels(value, {}, Path("candidate"), "a" * 64)
api.assert_not_called(); images.assert_not_called(); helm.assert_called_once()
def test_interrupted_version_publish_never_advances_channels(self):
with patch.object(publish, "publish_version", side_effect=ValueError("registry unavailable")), patch.object(publish, "api") as api:
with self.assertRaises(ValueError):
publish.channels(inputs(), {}, Path("candidate"), "a" * 64)
api.assert_not_called()
def test_incomplete_helm_publication_does_not_advance_stable_channels(self):
with patch.object(publish, "publish_version"), patch.object(publish, "helm", side_effect=ValueError("chart repository unavailable")), patch.object(publish, "api") as api, patch.object(publish, "images") as images:
with self.assertRaises(ValueError):
publish.channels(inputs(), {}, Path("candidate"), "a" * 64)
api.assert_not_called()
images.assert_not_called()
def test_r2_retry_checks_bytes_and_uses_conditional_creation(self):
with tempfile.TemporaryDirectory() as temp, patch.dict(os.environ, R2_BUCKET="releases"):
path = Path(temp) / "asset.zip"; path.write_bytes(b"accepted")
objects = {}
calls = []
def aws(*args, **kwargs):
calls.append(args)
if args[:2] == ("s3api", "head-object"):
exists = args[args.index("--key") + 1] in objects
return subprocess.CompletedProcess(args, 0 if exists else 1, "{}", "" if exists else "(404)")
if args[:2] == ("s3api", "put-object"):
self.assertIn("--if-none-match", args)
objects[args[args.index("--key") + 1]] = Path(args[args.index("--body") + 1]).read_bytes()
else:
Path(args[3]).write_bytes(objects[args[2].split("/", 3)[3]])
return subprocess.CompletedProcess(args, 0, "{}", "")
with patch.object(publish, "aws", side_effect=aws):
publish.r2_file(path, "version/asset.zip")
publish.r2_file(path, "version/asset.zip")
self.assertEqual(sum(call[:2] == ("s3api", "put-object") for call in calls), 1)
objects["version/asset.zip"] = b"changed"
with self.assertRaisesRegex(ValueError, "never overwrite"):
publish.r2_file(path, "version/asset.zip")
class WorkflowContractTests(unittest.TestCase):
def test_nightly_avoids_desktop_and_manual_builds_keep_platform_selection(self):
value = workflow("build.yml")
selection = value["jobs"]["prepare-platform-matrix"]["steps"][-1]
self.assertEqual(selection["env"]["RAW_PLATFORMS"], "${{ inputs.platforms || 'linux' }}")
self.assertEqual(value["on"]["workflow_dispatch"]["inputs"]["platforms"]["default"], "all")
def test_main_push_cannot_cancel_the_scheduled_oidc_gate(self):
value = workflow("oidc-keycloak.yml")
self.assertIn("github.event_name", value["concurrency"]["group"])
self.assertIn("!= 'schedule'", value["concurrency"]["cancel-in-progress"])
def test_linux_is_the_only_initial_build(self):
value = workflow("release-candidate.yml")
self.assertEqual(set(value["on"]), {"workflow_dispatch"})
self.assertEqual(value["jobs"]["linux"]["with"]["platforms"], "linux")
self.assertNotIn("desktop", value["jobs"])
self.assertEqual(set(value["jobs"]["seal-linux"]["needs"]), {"freeze", "linux", "upgrade"})
def test_desktop_and_deliveries_wait_for_linux_acceptance(self):
value = workflow("release-complete.yml")
self.assertEqual(value["jobs"]["acceptance"]["environment"], "release-linux-acceptance")
for name in ("desktop", "packages", "images", "chart"):
self.assertEqual(value["jobs"][name]["needs"], "acceptance")
self.assertNotIn("secrets", value["jobs"][name])
self.assertEqual(set(value["jobs"]["seal"]["needs"]), {"acceptance", "desktop", "packages", "images", "chart"})
def test_only_publisher_has_release_writes_and_never_builds(self):
value = workflow("release-publish.yml")
self.assertEqual(set(value["on"]), {"workflow_dispatch"})
self.assertEqual(value["jobs"]["version"]["environment"], "release-publication")
self.assertEqual(value["jobs"]["channels"]["needs"], ["verify", "version"])
self.assertEqual(value["jobs"]["channels"]["concurrency"]["group"], "rustfs-release-channels")
for job in ("version", "channels"):
self.assertFalse(value["jobs"][job]["concurrency"]["cancel-in-progress"])
for name in ("build.yml", "package.yml", "docker.yml", "helm-package.yml"):
parsed = workflow(name)
self.assertEqual(parsed["permissions"]["actions"], "read")
self.assertNotIn("push", parsed["on"])
self.assertNotIn("workflow_run", parsed["on"])
self.assertNotIn("contents: write", (ROOT / ".github/workflows" / name).read_text())
source = (ROOT / "scripts/release/publish.py").read_text()
for forbidden in ("cargo build", "build-push", "helm package", "--clobber", "force=true"):
self.assertNotIn(forbidden, source.replace("# No --clobber:", "# No replacement:"))
def test_stage_failures_do_not_have_a_continue_on_error_bypass(self):
for name in ("release-candidate.yml", "release-complete.yml", "release-publish.yml", "docker.yml", "package.yml", "helm-package.yml"):
value = workflow(name)
for job in value["jobs"].values():
self.assertNotIn("continue-on-error", job)
for step in job.get("steps", []):
self.assertNotIn("continue-on-error", step)
class ImageWorkflowTests(unittest.TestCase):
def test_substituting_a_single_platform_scan_archive_is_rejected(self):
import deliver
with tempfile.TemporaryDirectory() as temp:
output = Path(temp) / "delivery"
scans = Path(str(output) + "-scan")
output.mkdir(); scans.mkdir()
value = inputs()
record = {"inputs_sha256": c.digest(value), "scan_inputs": {}}
for arch in ("amd64", "arm64"):
path = scans / f"image-musl-{arch}.tar"
path.write_bytes(arch.encode())
record["scan_inputs"][arch] = c.file_digest(path)
c.write_json(output / f"scan-musl-{arch}.sarif", {"version": "2.1.0", "runs": [{}]})
c.write_json(output / "image-musl-smoke.json", record)
deliver.image_record(value, "musl", output)
self.assertEqual(c.read_json(output / "image-musl.json")["scan"], "completed")
(scans / "image-musl-arm64.tar").write_bytes(b"another architecture")
with self.assertRaisesRegex(ValueError, "scan input changed"):
deliver.image_record(value, "musl", output)
def test_both_architectures_scan_before_seal_without_registry_credentials(self):
value = workflow("docker.yml")
self.assertEqual(value["permissions"], {"contents": "read", "actions": "read"})
steps = value["jobs"]["build-docker"]["steps"]
build = next(s for s in steps if "docker/build-push-action@" in s.get("uses", ""))
self.assertFalse(build["with"]["push"])
self.assertIn("type=oci", build["with"]["outputs"])
scans = [s for s in steps if "aquasecurity/trivy-action@" in s.get("uses", "")]
self.assertEqual({s["env"]["TRIVY_PLATFORM"] for s in scans}, {"linux/amd64", "linux/arm64"})
for scan in scans:
self.assertEqual(scan["with"]["exit-code"], "0")
self.assertIn("input", scan["with"])
architecture = scan["env"]["TRIVY_PLATFORM"].split("/")[-1]
self.assertTrue(scan["with"]["input"].endswith(f"-{architecture}.tar"))
self.assertLess(steps.index(scan), next(i for i, s in enumerate(steps) if s.get("name") == "Record verified OCI identity"))
class HelmWorkflowTests(unittest.TestCase):
def test_installation_versions_match_candidate_chart_versions(self):
subprocess.run(["bash", "scripts/test_helm_chart_version.sh"], cwd=ROOT,
check=True, capture_output=True, text=True, timeout=15)
def test_chart_is_sealed_before_publish_and_does_not_push(self):
value = workflow("helm-package.yml")
self.assertEqual(set(value["on"]), {"workflow_call"})
source = (ROOT / ".github/workflows/helm-package.yml").read_text()
self.assertIn("scripts/test_helm_templates.sh", source)
self.assertIn("('version', 'appVersion')", source)
self.assertNotIn("git push", source)
self.assertIn("inputs_sha256", source)
if __name__ == "__main__":
unittest.main()
+31 -63
View File
@@ -111,7 +111,22 @@ class GateFixtureTests(unittest.TestCase):
self.assertEqual(summary["findings"], [])
expansion = json.loads((self.output / "pass" / "plan-expansion.json").read_text())
self.assertEqual(len(expansion["required_cases"]), 5)
promote = run_promote(self.output / "pass")
import hashlib
artifact = self.output / ARTIFACT_NAME
artifact.write_bytes(b"release gate contract payload\n")
candidate = json.loads(CANDIDATE.read_text())
old_sha = candidate["deliverables"][0]["sha256"]
new_sha = hashlib.sha256(artifact.read_bytes()).hexdigest()
candidate["deliverables"][0].update(sha256=new_sha, size=artifact.stat().st_size)
candidate_path = self.output / "candidate.json"
candidate_path.write_text(json.dumps(candidate))
results = self.output / "actual-results"
results.mkdir()
for path in (FIXTURES / "pass").glob("*.json"):
(results / path.name).write_text(path.read_text().replace(old_sha, new_sha))
result = run_validate(results, self.output / "actual-certification", candidate=candidate_path)
self.assertEqual(result.returncode, 0, result.stderr)
promote = run_promote(self.output / "actual-certification", candidate=candidate_path, artifact=artifact)
self.assertEqual(promote.returncode, 0, promote.stderr)
def test_cross_platform_same_case_and_split_shas_accepted(self):
@@ -306,70 +321,23 @@ class PlanContractTests(unittest.TestCase):
class PublicationWiringTests(unittest.TestCase):
"""Static checks that the real publication exits consume the gate."""
"""The publisher re-runs full certification over trusted downloaded receipts."""
def read(self, relative):
return (ROOT / relative).read_text()
def test_single_publisher_revalidates_the_complete_seal(self):
source = (ROOT / "scripts/release/publish.py").read_text()
self.assertIn('pipeline.certify(inputs, directory,', source)
self.assertIn('"full")', source)
self.assertIn('summary["publish_allowed"]', source)
self.assertIn('pipeline.verify_deliveries(inputs, directory)', source)
workflow = (ROOT / ".github/workflows/release-publish.yml").read_text()
self.assertIn('environment: release-publication', workflow)
self.assertIn('needs: [verify, version]', workflow)
def job_block(self, workflow, job):
match = re.search(
rf"^ {re.escape(job)}:\n(.*?)(?=^ [A-Za-z#])",
workflow,
re.S | re.M,
)
self.assertIsNotNone(match, f"job {job} not found")
return match.group(1)
def test_build_publish_requires_certification(self):
workflow = self.read(".github/workflows/build.yml")
self.assertIn("\n release-certification:", workflow)
publish_block = self.job_block(workflow, "publish-release")
self.assertIn("release-certification", publish_block)
self.assertIn("release_gate.py validate", workflow)
self.assertIn("release_gate.py verify-promotion", workflow)
def test_build_time_official_r2_writes_are_staged(self):
workflow = self.read(".github/workflows/build.yml")
self.assertIn("artifacts/rustfs/staging/", workflow)
release_writes = [
line for line in workflow.splitlines()
if "artifacts/rustfs/release/" in line
]
# Only the certification-gated promotion job may write the official path.
self.assertTrue(release_writes, "the promotion job must write the official R2 path")
promotion_block = self.job_block(workflow, "promote-release-artifacts")
self.assertIn("verify-promotion", promotion_block)
self.assertIn("artifacts/rustfs/release/", promotion_block)
build_block = self.job_block(workflow, "build-rustfs")
self.assertNotIn('R2_PATH="s3://${R2_BUCKET}/artifacts/rustfs/release/"', build_block)
def test_downstream_workflows_verify_certification(self):
docker = self.read(".github/workflows/docker.yml")
self.assertIn("\n release-certification:", docker)
build_docker_block = self.job_block(docker, "build-docker")
self.assertIn("release-certification", build_docker_block)
self.assertIn("verify-promotion", docker)
package = self.read(".github/workflows/package.yml")
self.assertIn("\n release-certification:", package)
package_block = self.job_block(package, "package")
self.assertIn("release-certification", package_block)
helm = self.read(".github/workflows/helm-package.yml")
self.assertIn("\n release-certification:", helm)
helm_block = self.job_block(helm, "build-helm-package")
self.assertIn("release-certification", helm_block)
def test_manual_downstream_dispatch_without_certification_fails(self):
# Every downstream certification job must fail closed when the trusted
# certification artifact is absent (e.g. a manual dispatch without one).
for relative in (
".github/workflows/docker.yml",
".github/workflows/package.yml",
".github/workflows/helm-package.yml",
):
workflow = self.read(relative)
certification_block = self.job_block(workflow, "release-certification")
self.assertIn("fail", certification_block.lower(), relative)
self.assertIn("release-certification", certification_block, relative)
def test_builders_do_not_have_publication_permissions(self):
for name in ("build.yml", "docker.yml", "package.yml", "helm-package.yml"):
source = (ROOT / ".github/workflows" / name).read_text()
self.assertNotIn("contents: write", source)
self.assertNotIn("packages: write", source)
if __name__ == "__main__":
+230
View File
@@ -0,0 +1,230 @@
#!/usr/bin/env python3
# Copyright 2026 RustFS Team
# SPDX-License-Identifier: Apache-2.0
"""Run release checks against the archived executable, never a rebuilt server."""
from __future__ import annotations
import argparse
from datetime import datetime, timezone
import json
import hashlib
import hmac
import os
from pathlib import Path
import re
import socket
import subprocess
import tempfile
import time
import urllib.request
import zipfile
import candidate as c
TRIPLES = {
"linux-x86_64-musl": "x86_64-unknown-linux-musl", "linux-aarch64-musl": "aarch64-unknown-linux-musl",
"linux-x86_64-gnu": "x86_64-unknown-linux-gnu", "linux-aarch64-gnu": "aarch64-unknown-linux-gnu",
"macos-aarch64": "aarch64-apple-darwin", "windows-x86_64": "x86_64-pc-windows-msvc",
}
UPGRADES = {
"direct-upgrade": "direct_upgrade_from_previous_release_preserves_object_contracts",
"mixed-version-upgrade": "rolling_upgrade_from_previous_release_preserves_mixed_version_contracts",
"bucket-config-upgrade": "direct_upgrade_from_previous_release_preserves_bucket_configuration",
"bucket-config-rollback": "rollback_to_previous_release_reads_current_bucket_metadata",
"multipart-layout-upgrade": "direct_upgrade_from_previous_release_preserves_multipart_layouts",
"multipart-layout-baseline": "previous_release_baseline_replicates_multipart_layouts",
}
def now():
return datetime.now(timezone.utc).isoformat().replace("+00:00", "Z")
def envelope(inputs, record, lane, case, logs, started, output):
"""Only called after the real check succeeds; logs remain part of the seal."""
target = record["target"].split("-")
platform = dict(zip(("os", "arch", "abi"), target))
platform.setdefault("abi", "msvc" if platform["os"] == "windows" else "none")
workflow = os.environ["GITHUB_WORKFLOW_REF"].split("@", 1)[0].rsplit("/", 1)[-1]
c.require(workflow in ("release-candidate.yml", "release-complete.yml"), "untrusted verifier workflow")
value = {
"schema_version": 1, "evidence_profile": "release-candidate", "tier": "release",
"run_id": inputs["candidate_id"], "run_attempt": 1, "plan_sha256": inputs["plan_sha256"],
"lane": lane, "case_id": case, "product_commit": inputs["source_sha"],
"harness_commit": inputs["workflow_sha"], "target_fingerprint": record["binary_sha256"],
"platform": platform, "started_at": started, "finished_at": now(), "state": "pass",
"reason_code": "CHECK_PASSED", "reason_class": "NOT_APPLICABLE",
"artifact": {"archive_sha256": record["sha256"], "binary_sha256": record["binary_sha256"]},
"evidence": [{"path": p.name, "sha256": c.file_digest(p), "size": p.stat().st_size} for p in logs],
"workflow": {"repository": c.REPOSITORY, "workflow_id": workflow, "run_id": os.environ["GITHUB_RUN_ID"],
"run_attempt": int(os.environ["GITHUB_RUN_ATTEMPT"]), "workflow_sha": os.environ["GITHUB_SHA"],
"status": "completed"},
}
c.write_json(output / f"{lane}-{record['target']}-{case}.json", value)
def unpack(archive, target, inputs, directory):
record = c.archive_record(archive, target, inputs)
name = "rustfs.exe" if target.startswith("windows") else "rustfs"
with zipfile.ZipFile(archive) as package:
package.extract(name, directory)
binary = (Path(directory) / name).resolve()
binary.chmod(0o755)
return binary, record
def port():
with socket.socket() as sock:
sock.bind(("127.0.0.1", 0))
return sock.getsockname()[1]
def s3_request(opener, endpoint, method, path, access, secret, body=b""):
timestamp = datetime.now(timezone.utc).strftime("%Y%m%dT%H%M%SZ")
date = timestamp[:8]
host = endpoint.removeprefix("http://")
payload = hashlib.sha256(body).hexdigest()
headers = f"host:{host}\nx-amz-content-sha256:{payload}\nx-amz-date:{timestamp}\n"
signed = "host;x-amz-content-sha256;x-amz-date"
canonical = f"{method}\n{path}\n\n{headers}\n{signed}\n{payload}"
scope = f"{date}/us-east-1/s3/aws4_request"
to_sign = f"AWS4-HMAC-SHA256\n{timestamp}\n{scope}\n{hashlib.sha256(canonical.encode()).hexdigest()}"
key = ("AWS4" + secret).encode()
for part in (date, "us-east-1", "s3", "aws4_request"):
key = hmac.digest(key, part.encode(), "sha256")
signature = hmac.new(key, to_sign.encode(), hashlib.sha256).hexdigest()
request = urllib.request.Request(endpoint + path, data=body if method == "PUT" else None, method=method,
headers={"Host": host, "X-Amz-Content-Sha256": payload, "X-Amz-Date": timestamp,
"Authorization": f"AWS4-HMAC-SHA256 Credential={access}/{scope}, SignedHeaders={signed}, Signature={signature}"})
with opener.open(request, timeout=10) as response:
c.require(response.status in (200, 204), "S3 smoke request failed")
return response.read(1024 * 1024)
def native(inputs, archive, target, output):
started = now()
output.mkdir(parents=True, exist_ok=True)
with tempfile.TemporaryDirectory() as temp:
binary, record = unpack(archive, target, inputs, temp)
version = subprocess.check_output([str(binary), "--version"], text=True, timeout=60)
c.require(version.splitlines()[0] == f"rustfs {inputs['version']}", "binary displays a different product version")
for label, expected in (("build target", TRIPLES[target]), ("git commit", inputs["source_sha"])):
c.require(re.search(rf"(?m)^{label}\s*:\s*{re.escape(expected)}\s*$", version), f"binary {label} mismatch")
c.require(re.search(rf"(?m)^rust version\s*:\s*(?:rustc )?{re.escape(inputs['toolchain'])}(?:\s|$)", version), "binary toolchain mismatch")
log = output / f"identity-{target}.log"
log.write_text(version, encoding="utf-8")
envelope(inputs, record, "identity", "artifact-identity", [log], started, output)
data = Path(temp) / "data"
data.mkdir()
api, console = port(), port()
while console == api:
console = port()
log = output / f"startup-{target}.log"
access, secret = "release-" + os.urandom(8).hex(), os.urandom(24).hex()
env = dict(os.environ, NO_PROXY="127.0.0.1,localhost,::1", no_proxy="127.0.0.1,localhost,::1")
opener = urllib.request.build_opener(urllib.request.ProxyHandler({}))
with log.open("w", encoding="utf-8") as stream:
process = subprocess.Popen([str(binary), "server", "--address", f"127.0.0.1:{api}", "--console-enable",
"--console-address", f"127.0.0.1:{console}", "--access-key", access,
"--secret-key", secret, str(data)],
stdout=stream, stderr=subprocess.STDOUT, env=env)
try:
deadline = time.monotonic() + 60
while time.monotonic() < deadline:
c.require(process.poll() is None, "candidate exited before becoming ready")
try:
with opener.open(f"http://127.0.0.1:{api}/health/ready", timeout=2) as response:
c.require(response.status == 200, "S3 listener is not ready")
with opener.open(f"http://127.0.0.1:{console}/rustfs/console/", timeout=2) as response:
c.require(response.status == 200 and "text/html" in response.headers.get("Content-Type", ""), "Console did not return HTML")
break
except OSError:
time.sleep(0.5)
else:
raise ValueError("candidate did not become ready within 60 seconds")
bucket = "/release-" + os.urandom(8).hex()
endpoint = f"http://127.0.0.1:{api}"
content = os.urandom(1024)
s3_request(opener, endpoint, "PUT", bucket, access, secret)
s3_request(opener, endpoint, "PUT", bucket + "/roundtrip", access, secret, content)
c.require(s3_request(opener, endpoint, "GET", bucket + "/roundtrip", access, secret) == content, "S3 round-trip bytes differ")
s3_request(opener, endpoint, "DELETE", bucket + "/roundtrip", access, secret)
s3_request(opener, endpoint, "DELETE", bucket, access, secret)
c.require(process.poll() is None, "candidate exited during startup verification")
finally:
process.terminate()
try:
process.wait(timeout=15)
except subprocess.TimeoutExpired:
process.kill()
process.wait(timeout=15)
c.require(c.file_digest(binary) == record["binary_sha256"], "server bytes changed during native verification")
with log.open("a", encoding="utf-8") as stream:
stream.write("\nConsole: HTTP 200 text/html; S3: ready, bucket/object CRUD and byte-identical round trip\n")
envelope(inputs, record, "identity", "startup", [log], started, output)
def upgrade(inputs, archive, output):
output.mkdir(parents=True, exist_ok=True)
plan = c.read_json(c.ROOT / ".config/release-validation-plan.json")
c.require(c.digest(plan) == inputs["plan_sha256"], "upgrade plan changed")
with tempfile.TemporaryDirectory() as temp:
root = Path(temp)
binary, record = unpack(archive, "linux-x86_64-gnu", inputs, root)
metadata = binary.stat()
receipt = root / "receipt.json"
receipt.write_text(json.dumps({"schema": 1, "workspace": str(c.ROOT.resolve()), "binary": str(binary),
"size": metadata.st_size, "modified_ns": metadata.st_mtime_ns, "features": ["pyroscope"]}))
env = dict(os.environ, CARGO_BIN_EXE_rustfs=str(binary), RUSTFS_E2E_BINARY_RECEIPT=str(receipt),
RUSTFS_BUILD_FEATURES="pyroscope", NO_PROXY="127.0.0.1,localhost,::1", no_proxy="127.0.0.1,localhost,::1",
HTTP_PROXY="", HTTPS_PROXY="", ALL_PROXY="", http_proxy="", https_proxy="", all_proxy="")
logs = {case: [] for case in UPGRADES}
started = now()
sources = list(plan["upgrade_sources"]["sources"])
previous = inputs["previous"]
if previous["version"] not in {s["version"] for s in sources}:
sources.append({"version": previous["version"], "artifact_name": previous["name"], "artifact_sha256": previous["sha256"]})
for source in sources:
directory = root / source["version"]
directory.mkdir()
package = directory / source["artifact_name"]
url = f"https://github.com/{c.REPOSITORY}/releases/download/{source['version']}/{source['artifact_name']}"
subprocess.run(["curl", "--fail", "--location", "--retry", "3", "--output", str(package), url], check=True, timeout=600)
c.require(c.file_digest(package) == source["artifact_sha256"], "previous release digest mismatch")
with zipfile.ZipFile(package) as previous:
c.require(previous.namelist().count("rustfs") == 1, "ambiguous previous server")
previous.extract("rustfs", directory)
(directory / "rustfs").chmod(0o755)
env["RUSTFS_UPGRADE_SOURCE_BINARY"] = str(directory / "rustfs")
for case, test in UPGRADES.items():
log = output / f"upgrade-{source['version']}-{case}.log"
env["RUSTFS_E2E_LOG_DIR"] = str(root / f"logs-{source['version']}-{case}")
with log.open("w", encoding="utf-8") as stream:
subprocess.run(["cargo", "test", "--locked", "-p", "e2e_test", f"upgrade_compatibility_test::{test}",
"--", "--ignored", "--exact", "--nocapture"], cwd=c.ROOT, env=env,
stdout=stream, stderr=subprocess.STDOUT, check=True, timeout=1800)
c.require(re.search(r"test result: ok\. 1 passed; 0 failed;", log.read_text()), "upgrade command did not run exactly one passing test")
c.require(c.file_digest(binary) == record["binary_sha256"], "candidate changed during upgrade tests")
logs[case].append(log)
for case, files in logs.items():
envelope(inputs, record, "upgrade", case, files, started, output)
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("mode", choices=("native", "upgrade"))
parser.add_argument("--inputs", required=True)
parser.add_argument("--archive", required=True)
parser.add_argument("--target", choices=c.TARGETS)
parser.add_argument("--output", type=Path, required=True)
args = parser.parse_args()
inputs = c.validate_inputs(c.read_json(args.inputs))
if args.mode == "native":
c.require(args.target is not None, "native verification needs a platform")
native(inputs, args.archive, args.target, args.output)
else:
upgrade(inputs, args.archive, args.output)
if __name__ == "__main__":
main()
@@ -1,27 +1,4 @@
#!/usr/bin/env bash
set -euo pipefail
workflow=".github/workflows/docker.yml"
require_literal() {
local needle="$1"
local description="$2"
if ! grep -Fq "$needle" "$workflow"; then
echo "missing container scan workflow contract: $description" >&2
exit 1
fi
}
require_literal "scan-docker-image:" "scan job"
require_literal "needs: [ build-check, build-docker ]" "build dependency"
require_literal "needs.build-check.outputs.should_build == 'true' && needs.build-check.outputs.should_push == 'true'" "release image push guard"
require_literal "docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9" "pinned GHCR login action"
require_literal "aquasecurity/trivy-action@ed142fd0673e97e23eac54620cfb913e5ce36c25" "pinned Trivy action"
require_literal 'image-ref: ${{ env.REGISTRY_GHCR }}:${{ needs.build-check.outputs.version }}${{ matrix.suffix }}' "GHCR image reference"
require_literal "format: sarif" "SARIF report format"
require_literal 'exit-code: "0"' "report-only failure policy"
require_literal "actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a" "pinned report upload action"
require_literal "container-image-scan-" "scan report artifact name"
echo "Container scan workflow contract ok."
cd "$(dirname "${BASH_SOURCE[0]}")/../.."
python3 scripts/release/test_pipeline.py ImageWorkflowTests
@@ -1,420 +1,4 @@
#!/usr/bin/env bash
set -euo pipefail
build_workflow=".github/workflows/build.yml"
docker_workflow=".github/workflows/docker.yml"
helm_workflow=".github/workflows/helm-package.yml"
package_workflow=".github/workflows/package.yml"
release_script="scripts/release/create_or_update_release.sh"
# shellcheck source=scripts/release/create_or_update_release.sh
source "$release_script"
require_line() {
local file="$1"
local line="$2"
local description="$3"
if ! grep -Fxq "$line" "$file"; then
echo "missing preview release workflow contract: $description" >&2
exit 1
fi
}
require_absent() {
local file="$1"
local text="$2"
local description="$3"
if grep -Fq -- "$text" "$file"; then
echo "invalid preview release workflow contract: $description" >&2
exit 1
fi
}
require_line "$build_workflow" " cargo zigbuild --release --locked --target \"\$TARGET_ARG\" \"\${FEATURE_ARGS[@]}\" \"\${PROFILE_ARGS[@]}\" -p rustfs --bin rustfs" "cross builds must keep matrix features"
require_line "$build_workflow" " cargo build --release --locked --target \"\$TARGET_ARG\" \"\${FEATURE_ARGS[@]}\" \"\${PROFILE_ARGS[@]}\" -p rustfs --bin rustfs" "native builds must keep matrix features"
require_absent "$build_workflow" " cargo zigbuild --release --locked --target \"\$TARGET_ARG\" -p rustfs --bin rustfs" "cross builds must not drop matrix features"
require_absent "$build_workflow" " cargo build --release --locked --target \"\$TARGET_ARG\" -p rustfs --bin rustfs" "native builds must not drop matrix features"
extract_job_if() {
local file="$1"
local job="$2"
awk -v job="$job" '
$0 == " " job ":" { in_job = 1 }
in_job && /^ if:/ { in_if = 1 }
in_if && /^ [A-Za-z0-9_-]+:/ && $0 !~ /^ if:/ { exit }
in_if { print }
in_job && $0 ~ /^ [A-Za-z0-9_-]+:$/ && $0 != " " job ":" { exit }
' "$file"
}
require_job_if() {
local file="$1"
local job="$2"
local expected="$3"
local actual
actual=$(extract_job_if "$file" "$job")
if [[ "$actual" != "$expected" ]]; then
echo "missing preview release workflow contract: $job condition" >&2
exit 1
fi
}
require_assignment() {
local block="$1"
local name="$2"
local expected="$3"
local count
count=$(printf '%s\n' "$block" | grep -Ec "^[[:space:]]+${name}=")
if [[ "$count" -ne 1 || "$block" != *"${name}=${expected}"* ]]; then
echo "missing preview release workflow contract: $name=$expected" >&2
exit 1
fi
}
require_no_assignment() {
local block="$1"
local name="$2"
if printf '%s\n' "$block" | grep -Eq "^[[:space:]]+${name}="; then
echo "invalid preview release workflow contract: unexpected $name override" >&2
exit 1
fi
}
tag_classification=$(awk '
/if \[\[ "\$tag_name" =~ -preview\\\.\[0-9\]\+\$ \]\]; then/ { in_preview = 1 }
in_preview { print }
in_preview && $0 == " fi" { exit }
' "$build_workflow")
invalid_preview_condition="elif [[ \"\$tag_name\" == *\"-preview\"* ]]; then"
prerelease_condition="elif [[ \"\$tag_name\" == *\"alpha\"* ]] || [[ \"\$tag_name\" == *\"beta\"* ]] || [[ \"\$tag_name\" == *\"rc\"* ]]; then"
preview_branch=$(printf '%s\n' "$tag_classification" | awk -v boundary="$invalid_preview_condition" 'index($0, boundary) { exit } { print }')
invalid_preview_branch=$(printf '%s\n' "$tag_classification" | awk -v start="$invalid_preview_condition" -v boundary="$prerelease_condition" 'index($0, start) { in_branch = 1 } in_branch && index($0, boundary) { exit } in_branch { print }')
prerelease_branch=$(printf '%s\n' "$tag_classification" | awk -v start="$prerelease_condition" 'index($0, start) { in_branch = 1 } in_branch && $0 == " else" { exit } in_branch { print }')
release_branch=$(printf '%s\n' "$tag_classification" | awk '$0 == " else" { in_branch = 1; next } in_branch && $0 == " fi" { exit } in_branch { print }')
if [[ -z "$preview_branch" || -z "$invalid_preview_branch" || -z "$prerelease_branch" || -z "$release_branch" ]]; then
echo "missing preview release workflow contract: ordered preview, invalid-preview, prerelease, and release branches" >&2
exit 1
fi
require_assignment "$preview_branch" "build_type" '"preview"'
require_assignment "$preview_branch" "is_prerelease" "true"
require_no_assignment "$invalid_preview_branch" "build_type"
if [[ "$invalid_preview_branch" != *"exit 1"* ]]; then
echo "missing preview release workflow contract: invalid preview tags must fail closed" >&2
exit 1
fi
require_assignment "$prerelease_branch" "build_type" '"prerelease"'
require_assignment "$prerelease_branch" "is_prerelease" "true"
require_assignment "$release_branch" "build_type" '"release"'
require_no_assignment "$release_branch" "is_prerelease"
initial_prerelease_line=$(grep -Fn " is_prerelease=false" "$build_workflow")
preview_condition_line=$(grep -Fn " if [[ \"\$tag_name\" =~ -preview\\.[0-9]+\$ ]]; then" "$build_workflow")
if [[ $(printf '%s\n' "$initial_prerelease_line" | wc -l) -ne 1 || "${initial_prerelease_line%%:*}" -ge "${preview_condition_line%%:*}" ]]; then
echo "missing preview release workflow contract: is_prerelease=false must initialize before tag classification" >&2
exit 1
fi
tag_branch_tail=$(awk '
/if \[\[ "\$tag_name" =~ -preview\\\.\[0-9\]\+\$ \]\]; then/ { in_classification = 1 }
in_classification && $0 == " fi" { after_classification = 1; next }
after_classification && /^ elif / { exit }
after_classification { print }
' "$build_workflow")
post_strategy=$(awk '
/# Determine build type based on trigger/ { in_strategy = 1 }
in_strategy && $0 == " fi" { after_strategy = 1; next }
after_strategy && $0 == " {" { exit }
after_strategy { print }
' "$build_workflow")
for block in "$tag_branch_tail" "$post_strategy"; do
require_no_assignment "$block" "build_type"
require_no_assignment "$block" "is_prerelease"
done
require_line "$build_workflow" " if [[ \"\$BUILD_TYPE\" == \"release\" ]] || [[ \"\$BUILD_TYPE\" == \"prerelease\" ]]; then" "latest artifact guard"
require_line "$build_workflow" " if: needs.build-check.outputs.should_publish == 'true' && env.R2_ACCESS_KEY_ID != '' && (needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease' || needs.build-check.outputs.build_type == 'development')" "R2 publication guard"
require_line "$package_workflow" " if [[ \"\$TAG\" =~ ^\${semver_core}(-\${prerelease_id})?-preview\\.(0|[1-9][0-9]*)$ ]]; then" "stable and prerelease preview package classification"
require_line "$package_workflow" " if: needs.resolve.outputs.build_type != 'preview' && env.R2_ACCESS_KEY_ID != ''" "preview packages must not publish to R2"
release_guard="needs.build-check.outputs.should_publish == 'true' && startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'preview' || needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')"
for job in create-release upload-release-assets publish-release; do
require_job_if "$build_workflow" "$job" " if: $release_guard"
done
latest_guard="needs.build-check.outputs.should_publish == 'true' && startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')"
require_job_if "$build_workflow" "update-latest-version" " if: $latest_guard"
require_line "$build_workflow" " needs: [ build-check, publish-release ]" "latest update must follow release publication"
# Preview releases are internal validation artifacts: once the deliverable
# release is published they are deleted, while their tags stay behind.
require_job_if "$build_workflow" "cleanup-preview-releases" " if: $latest_guard"
require_line "$build_workflow" " gh release delete \"\$preview_tag\" --repo \"\${GITHUB_REPOSITORY}\" --yes" "preview release cleanup after publication"
require_line "$build_workflow" " | select(.tag_name | startswith(\$tag + \"-preview.\"))" "cleanup must match the target's own preview tags"
require_line "$build_workflow" " | select(.tag_name | ltrimstr(\$tag + \"-preview.\") | test(\"^[0-9]+\$\"))" "cleanup must match a numeric preview iteration"
require_absent "$build_workflow" "--cleanup-tag" "preview tags must survive their release cleanup"
require_line "$build_workflow" " TARGET_COMMITISH=\$(git rev-parse --verify \"refs/tags/\${TAG}^{commit}\")" "release target commit resolution"
require_line "$build_workflow" " ./scripts/release/create_or_update_release.sh \\" "managed release creation"
require_absent "$build_workflow" "git tag -l --format='%(contents)'" "annotated tag messages must not become release notes"
require_absent "$build_workflow" "--generate-notes" "GitHub must not choose the release notes baseline implicitly"
require_line "$release_script" " gh api --method POST \"repos/\${GITHUB_REPOSITORY}/releases/generate-notes\" \\" "Generate Release Notes API"
require_line "$release_script" " local create_args=(release create \"\$tag\" --title \"\$title\" --notes-file \"\$notes_file\" --latest=false --draft)" "draft release creation with a notes file"
release_channel_block=$(awk '
$0 == " if [[ \"$BUILD_TYPE\" == \"release\" ]]; then" { in_block = 1 }
in_block { print }
in_block && $0 == " fi" { exit }
' "$build_workflow")
IFS= read -r -d '' expected_release_channel_block <<'EOF' || true
if [[ "$BUILD_TYPE" == "release" ]]; then
gh api --method PATCH "repos/${GITHUB_REPOSITORY}/releases/${RELEASE_ID}" \
-F draft=false \
-F prerelease=false \
-f make_latest=true >/dev/null
else
gh api --method PATCH "repos/${GITHUB_REPOSITORY}/releases/${RELEASE_ID}" \
-F draft=false \
-F prerelease=true \
-f make_latest=false >/dev/null
fi
EOF
expected_release_channel_block=${expected_release_channel_block%$'\n'}
if [[ "$release_channel_block" != "$expected_release_channel_block" ]]; then
echo "missing preview release workflow contract: only stable releases may become GitHub Latest" >&2
exit 1
fi
publish_release_block=$(awk '
$0 == " publish-release:" { in_job = 1 }
in_job { print }
in_job && /^ [A-Za-z0-9_-]+:$/ && $0 != " publish-release:" { exit }
' "$build_workflow")
if [[ "$publish_release_block" != *" - name: Publish release"* ]]; then
echo "missing preview release workflow contract: publish job must only publish the prepared release" >&2
exit 1
fi
if grep -Eq -- '--notes|generate-notes|create_or_update_release' <<<"$publish_release_block"; then
echo "invalid preview release workflow contract: publish job must not replace release notes" >&2
exit 1
fi
IFS= read -r -d '' expected_docker_automatic_guard <<'EOF' || true
if: >-
github.event_name == 'workflow_dispatch' ||
(github.event.workflow_run.conclusion == 'success' &&
github.event.workflow_run.event == 'push' &&
github.event.workflow_run.head_branch != 'main')
EOF
expected_docker_automatic_guard=${expected_docker_automatic_guard%$'\n'}
require_job_if "$docker_workflow" "build-check" "$expected_docker_automatic_guard"
require_line "$docker_workflow" ' source_ref: ${{ steps.check.outputs.source_ref }}' "Docker source ref output"
require_line "$docker_workflow" ' source_ref="$HEAD_SHA"' "automatic Docker source ref"
require_line "$docker_workflow" ' source_ref="$tag_ref"' "manual Docker source ref"
require_line "$docker_workflow" ' ref: ${{ needs.build-check.outputs.source_ref }}' "Docker release source checkout"
require_line "$docker_workflow" ' SOURCE_REVISION="$(git rev-parse HEAD)"' "Docker source revision resolution"
require_line "$docker_workflow" ' LABELS="$LABELS,org.opencontainers.image.revision=$SOURCE_REVISION"' "Docker revision label"
require_absent "$docker_workflow" 'org.opencontainers.image.revision=${{ github.sha }}' "Docker revision must not use the workflow branch SHA"
python3 scripts/test_docker_workflow.py
IFS= read -r -d '' expected_helm_guard <<'EOF' || true
if: |
(github.event_name == 'workflow_dispatch' && !contains(github.event.inputs.version, '-preview')) ||
(
github.event.workflow_run.conclusion == 'success' &&
github.event.workflow_run.event == 'push' &&
contains(github.event.workflow_run.head_branch, '.') &&
!contains(github.event.workflow_run.head_branch, '-preview')
)
EOF
expected_helm_guard=${expected_helm_guard%$'\n'}
require_job_if "$helm_workflow" "build-helm-package" "$expected_helm_guard"
require_line "$helm_workflow" " ref: \${{ github.event.workflow_run.head_sha || (startsWith(inputs.version, 'refs/tags/') && inputs.version || format('refs/tags/{0}', inputs.version)) }}" "Helm release source checkout"
python3 scripts/test_helm_release_workflow.py
assert_equal() {
local expected="$1"
local actual="$2"
local description="$3"
if [[ "$actual" != "$expected" ]]; then
echo "invalid preview release workflow contract: $description (expected '$expected', got '$actual')" >&2
exit 1
fi
}
IFS= read -r -d '' release_fixture <<'EOF' || true
[
[
{"tag_name":"1.0.0-beta.12-preview.1","created_at":"2026-07-30T02:30:00Z","published_at":"2026-07-30T02:36:43Z","draft":false},
{"tag_name":"1.0.0-beta.12","created_at":"2026-07-30T04:00:00Z","published_at":"2026-07-30T04:13:04Z","draft":false},
{"tag_name":"1.0.0-beta.12-preview.0","created_at":"2026-07-29T10:00:00Z","published_at":"2026-07-29T10:05:00Z","draft":false}
],
[
{"tag_name":"1.0.0-beta.10","created_at":"2026-07-17T05:50:00Z","published_at":"2026-07-17T05:58:19Z","draft":false},
{"tag_name":"1.0.0-beta.11","created_at":"2026-07-23T04:00:00Z","published_at":"2026-07-23T04:07:31Z","draft":false},
{"tag_name":"9.0.0","created_at":"2025-01-01T00:00:00Z","published_at":"2025-01-01T00:00:00Z","draft":false},
{"tag_name":"1.0.0-beta.13","created_at":"2026-08-01T00:00:00Z","published_at":null,"draft":true}
]
]
EOF
preview_previous=$(select_previous_release_tag "1.0.0-beta.12-preview.1" "2026-07-30T02:36:43Z" <<<"$release_fixture")
assert_equal "1.0.0-beta.11" "$preview_previous" "preview notes baseline must exclude preview releases and its final deliverable"
final_previous=$(select_previous_release_tag "1.0.0-beta.12" "2026-07-30T04:13:04Z" <<<"$release_fixture")
assert_equal "1.0.0-beta.11" "$final_previous" "final notes baseline must exclude the same-commit preview release"
no_previous=$(select_previous_release_tag "1.0.0-beta.1-preview.1" "2026-01-01T00:00:00Z" <<'EOF'
[[{"tag_name":"1.0.0-beta.1-preview.1","created_at":"2026-01-01T00:00:00Z","published_at":"2026-01-01T00:00:00Z","draft":false}]]
EOF
)
assert_equal "" "$no_previous" "repositories without a previous deliverable must use GitHub's fallback baseline"
delayed_draft_previous=$(select_previous_release_tag "1.0.0-beta.12" "2026-07-30T04:00:00Z" <<'EOF'
[[
{"tag_name":"1.0.0-beta.12","created_at":"2026-07-30T04:00:00Z","published_at":null,"draft":true},
{"tag_name":"1.0.0-beta.13","created_at":"2026-08-01T00:00:00Z","published_at":"2026-08-01T00:05:00Z","draft":false},
{"tag_name":"1.0.0-beta.11","created_at":"2026-07-23T04:00:00Z","published_at":"2026-07-23T04:07:31Z","draft":false}
]]
EOF
)
assert_equal "1.0.0-beta.11" "$delayed_draft_previous" "draft rerun must exclude deliverables published after the draft was created"
release_test_dir=$(mktemp -d "${TMPDIR:-/tmp}/rustfs-release-workflow-test.XXXXXX")
trap 'rm -rf -- "$release_test_dir"' EXIT
request_json="${release_test_dir}/request.json"
write_generate_notes_request "1.0.0-beta.12" "0123456789abcdef" "1.0.0-beta.11" "$request_json"
assert_equal "1.0.0-beta.11" "$(jq -r .previous_tag_name "$request_json")" "explicit previous_tag_name request field"
write_generate_notes_request "1.0.0-beta.1" "0123456789abcdef" "" "$request_json"
assert_equal "false" "$(jq 'has("previous_tag_name")' "$request_json")" "fallback request must omit previous_tag_name"
existing_release_json="${release_test_dir}/existing-release.json"
jq -n --arg body "Pre-release 1.0.0-beta.12 (beta)" '{body: $body}' > "$existing_release_json"
assert_equal "update" "$(release_notes_action "$existing_release_json" "1.0.0-beta.12")" "rerun must repair legacy prerelease placeholder notes"
jq -n --arg body "${MANAGED_NOTES_MARKER}
## What's Changed" '{body: $body}' > "$existing_release_json"
assert_equal "update" "$(release_notes_action "$existing_release_json" "1.0.0-beta.12")" "rerun must refresh managed notes"
generated_body="## What's Changed
* Fix release notes by @alice in #123
* First contribution by @bob in #124
## New Contributors
* @bob made their first contribution in #124
**Full Changelog**: https://github.com/rustfs/rustfs/compare/1.0.0-beta.11...1.0.0-beta.12"
response_json="${release_test_dir}/response.json"
notes_file="${release_test_dir}/notes.md"
jq -n --arg body "$generated_body" '{body: $body}' > "$response_json"
write_generated_release_notes "$response_json" "$notes_file" "rustfs/rustfs" "1.0.0-beta.12" "1.0.0-beta.11"
expected_notes="${MANAGED_NOTES_MARKER}
${generated_body}"
assert_equal "$expected_notes" "$(<"$notes_file")" "generated multiline Markdown must remain intact"
mock_log="${release_test_dir}/gh.log"
mock_notes="${release_test_dir}/captured-notes.md"
mock_generate_calls="${release_test_dir}/generate-calls"
MOCK_RELEASES_JSON="$release_fixture"
MOCK_GENERATED_BODY="$generated_body"
MOCK_RELEASE_EXISTS=true
MOCK_RELEASE_BODY="Release 1.0.0-beta.12"
MOCK_RELEASE_VIEW_ERROR=""
gh() {
local input_file=""
local notes_path=""
if [[ "$1" == "api" && "$*" == *"--paginate --slurp"* ]]; then
printf '%s\n' "$MOCK_RELEASES_JSON"
return
fi
if [[ "$1" == "api" && "$*" == *"releases/generate-notes"* ]]; then
while [[ "$#" -gt 0 ]]; do
if [[ "$1" == "--input" ]]; then
input_file="$2"
break
fi
shift
done
jq -e '.tag_name == "1.0.0-beta.12" and .target_commitish == "0123456789abcdef" and .previous_tag_name == "1.0.0-beta.11"' "$input_file" >/dev/null
printf '%s\n' called >> "$mock_generate_calls"
jq -n --arg body "$MOCK_GENERATED_BODY" '{body: $body}'
return
fi
if [[ "$1" == "release" && "$2" == "view" ]]; then
if [[ -n "$MOCK_RELEASE_VIEW_ERROR" ]]; then
printf '%s\n' "$MOCK_RELEASE_VIEW_ERROR" >&2
return 1
fi
if [[ "$MOCK_RELEASE_EXISTS" != "true" ]]; then
echo "release not found" >&2
return 1
fi
jq -n --arg body "$MOCK_RELEASE_BODY" '{databaseId: 123, url: "https://github.com/rustfs/rustfs/releases/tag/test", body: $body, createdAt: "2026-07-30T04:00:00Z", publishedAt: "2026-07-30T04:13:04Z"}'
return
fi
if [[ "$1" == "release" && ("$2" == "edit" || "$2" == "create") ]]; then
local action="$2"
shift 2
while [[ "$#" -gt 0 ]]; do
if [[ "$1" == "--notes-file" ]]; then
notes_path="$2"
break
fi
shift
done
cp "$notes_path" "$mock_notes"
MOCK_RELEASE_BODY=$(<"$mock_notes")
MOCK_RELEASE_EXISTS=true
printf '%s\n' "$action" >> "$mock_log"
return
fi
echo "unexpected mock gh invocation: $*" >&2
return 1
}
GITHUB_REPOSITORY="rustfs/rustfs"
GITHUB_OUTPUT="${release_test_dir}/github-output"
RUNNER_TEMP="$release_test_dir"
create_or_update_release "1.0.0-beta.12" "0123456789abcdef" "RustFS 1.0.0-beta.12 (beta)" true
assert_equal "edit" "$(<"$mock_log")" "existing placeholder release must be updated on rerun"
assert_equal "$expected_notes" "$(<"$mock_notes")" "rerun must update notes through --notes-file"
: > "$mock_log"
MOCK_RELEASE_EXISTS=false
MOCK_RELEASE_BODY=""
create_or_update_release "1.0.0-beta.12" "0123456789abcdef" "RustFS 1.0.0-beta.12 (beta)" true
assert_equal "create" "$(<"$mock_log")" "first run must create the draft release"
: > "$mock_log"
: > "$mock_generate_calls"
MOCK_RELEASE_EXISTS=true
MOCK_RELEASE_BODY="$generated_body"
create_or_update_release "1.0.0-beta.12" "0123456789abcdef" "RustFS 1.0.0-beta.12 (beta)" true
assert_equal "" "$(<"$mock_log")" "manual notes must not trigger a release edit"
assert_equal "" "$(<"$mock_generate_calls")" "manual notes must be preserved without regeneration"
MOCK_RELEASE_BODY="Manually curated release guidance"
if create_or_update_release "1.0.0-beta.12" "0123456789abcdef" "RustFS 1.0.0-beta.12 (beta)" true; then
echo "invalid preview release workflow contract: incomplete manual notes must block publication" >&2
exit 1
fi
assert_equal "" "$(<"$mock_log")" "incomplete manual notes must not be overwritten"
MOCK_RELEASE_VIEW_ERROR="gh: service unavailable (HTTP 503)"
if create_or_update_release "1.0.0-beta.12" "0123456789abcdef" "RustFS 1.0.0-beta.12 (beta)" true; then
echo "invalid preview release workflow contract: transient release lookup failures must fail closed" >&2
exit 1
fi
assert_equal "" "$(<"$mock_log")" "release lookup failures must not attempt create or edit"
echo "Preview release workflow contract ok."
cd "$(dirname "${BASH_SOURCE[0]}")/../.."
python3 scripts/release/test_pipeline.py WorkflowContractTests
@@ -25,6 +25,7 @@ def parse_args() -> argparse.Namespace:
parser.add_argument("--run-id", default=os.getenv("GITHUB_RUN_ID", ""))
parser.add_argument("--run-attempt", default=os.getenv("GITHUB_RUN_ATTEMPT", ""))
parser.add_argument("--cargo-lock", default="Cargo.lock", help="Cargo.lock path.")
parser.add_argument("--workflow", default="build.yml", help="Workflow that assembled these assets.")
parser.add_argument("--metadata-file", help="Use an existing cargo metadata JSON file.")
return parser.parse_args()
@@ -118,7 +119,7 @@ def build_sbom(packages: list[dict], args: argparse.Namespace, timestamp: str) -
def build_provenance(asset_dir: Path, args: argparse.Namespace, timestamp: str) -> dict:
subjects = []
for asset in sorted(asset_dir.glob("*.zip")):
for asset in sorted(path for path in asset_dir.iterdir() if path.suffix in (".zip", ".deb", ".rpm", ".tgz")):
subjects.append(
{
"name": asset.name,
@@ -129,7 +130,7 @@ def build_provenance(asset_dir: Path, args: argparse.Namespace, timestamp: str)
if not subjects:
raise RuntimeError(f"no .zip release assets found in {asset_dir}")
workflow_ref = f"{args.repository}/.github/workflows/build.yml"
workflow_ref = f"{args.repository}/.github/workflows/{args.workflow}"
invocation = args.run_id
if args.run_attempt:
invocation = f"{invocation}/attempts/{args.run_attempt}" if invocation else args.run_attempt
+260
View File
@@ -0,0 +1,260 @@
#!/usr/bin/env bash
# Contract tests for scripts/check_test_count_ratchet.sh.
#
# Each case builds a throwaway two-crate cargo workspace (crates `alpha` and
# `beta`), copies the script under test into it, writes a baseline, and runs
# the script there, so the real workspace and baseline are never touched. Only
# `cargo metadata --no-deps` runs; nothing is compiled.
#
# Contract under test: the baseline is a floor on tests and a ceiling on
# #[ignore]; see the header of check_test_count_ratchet.sh.
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
SCRIPT_UNDER_TEST="${RATCHET_SCRIPT:-$SCRIPT_DIR/check_test_count_ratchet.sh}"
TMP_ROOT="$(mktemp -d)"
trap 'rm -rf "$TMP_ROOT"' EXIT
passed=0
case_no=0
FX=""
OUT=""
ERR=""
STATUS=0
fail() {
echo "FAIL: $1" >&2
exit 1
}
# new_case <name>: fresh workspace with alpha and beta (members, no tests yet).
new_case() {
case_no=$((case_no + 1))
FX="$TMP_ROOT/case$case_no"
mkdir -p "$FX/scripts" "$FX/.config"
cp "$SCRIPT_UNDER_TEST" "$FX/scripts/check_test_count_ratchet.sh"
chmod +x "$FX/scripts/check_test_count_ratchet.sh"
printf '[workspace]\nresolver = "2"\nmembers = ["alpha", "beta"]\n' >"$FX/Cargo.toml"
make_crate alpha 1 0
make_crate beta 0 0
: >"$FX/.config/test-count-baseline.txt"
}
# make_crate <name> <tests> <ignored>: (re)write a member with that many
# attribute lines.
make_crate() {
local name="$1" tests="$2" ignored="$3" i
mkdir -p "$FX/$name/src"
printf '[package]\nname = "%s"\nversion = "0.0.0"\nedition = "2021"\n' "$name" >"$FX/$name/Cargo.toml"
{
for ((i = 0; i < tests; i++)); do printf '#[test]\nfn t%d() {}\n' "$i"; done
for ((i = 0; i < ignored; i++)); do printf '#[ignore]\nfn i%d() {}\n' "$i"; done
printf 'pub fn anchor() {}\n'
} >"$FX/$name/src/lib.rs"
}
# baseline <package> <tests> <ignored>...: append rows (tab-separated).
baseline() {
printf '%s\t%s\t%s\n' "$1" "$2" "$3" >>"$FX/.config/test-count-baseline.txt"
}
run() {
STATUS=0
(cd "$FX" && ./scripts/check_test_count_ratchet.sh "$@") >"$TMP_ROOT/out" 2>"$TMP_ROOT/err" || STATUS=$?
OUT="$(cat "$TMP_ROOT/out")"
ERR="$(cat "$TMP_ROOT/err")"
}
expect_pass() {
((STATUS == 0)) || fail "$1: expected exit 0, got $STATUS; stderr: $ERR"
}
expect_fail() {
((STATUS == 1)) || fail "$1: expected exit 1, got $STATUS; stderr: $ERR"
}
# stream_text <stdout|stderr>: captured output of the last run.
stream_text() {
if [[ "$1" == stdout ]]; then printf '%s' "$OUT"; else printf '%s' "$ERR"; fi
}
# expect_in <label> <stdout|stderr> <needle>
expect_in() {
local text
text="$(stream_text "$2")"
[[ "$text" == *"$3"* ]] || fail "$1: $2 lacks '$3'; got: $text"
}
expect_not_in() {
local text
text="$(stream_text "$2")"
[[ "$text" != *"$3"* ]] || fail "$1: $2 must not contain '$3'; got: $text"
}
ok() {
passed=$((passed + 1))
echo "ok: $1"
}
# --- positive cases -------------------------------------------------------
name="exact match passes silently"
new_case
make_crate alpha 3 1
baseline alpha 3 1
baseline beta 0 0
run
expect_pass "$name"
expect_not_in "$name" stdout "above baseline"
ok "$name"
name="tests above baseline pass with an informational line"
new_case
make_crate alpha 5 0
baseline alpha 3 0
baseline beta 0 0
run
expect_pass "$name"
expect_in "$name" stdout "note: crate alpha: 5 above baseline 3; run --update to tighten"
expect_not_in "$name" stderr "alpha"
ok "$name"
name="ignored below baseline passes with an informational line"
new_case
make_crate alpha 3 0
baseline alpha 3 2
baseline beta 0 0
run
expect_pass "$name"
expect_in "$name" stdout "note: crate alpha: 0 ignore attributes below baseline 2; run --update to tighten"
ok "$name"
name="member without a row passes with a notice"
new_case
make_crate alpha 3 0
make_crate beta 2 1
baseline alpha 3 0
run
expect_pass "$name"
expect_in "$name" stdout "note: crate beta: no baseline row (2 tests, 1 ignore attributes); run --update to add it"
ok "$name"
name="informational lines never carry a failure shape"
new_case
make_crate alpha 5 0
make_crate beta 1 0
baseline alpha 3 2
run
expect_pass "$name"
for shape in " < baseline" "ignored >" "not a workspace member" "violated"; do
expect_not_in "$name" stdout "$shape"
done
ok "$name"
name="--update round-trips: rewrites rows from the tree and the check then passes"
new_case
make_crate alpha 4 1
make_crate beta 2 0
baseline alpha 9 9
run --update
expect_pass "$name"
grep -qxF "$(printf 'alpha\t4\t1')" "$FX/.config/test-count-baseline.txt" || fail "$name: alpha row not rewritten"
grep -qxF "$(printf 'beta\t2\t0')" "$FX/.config/test-count-baseline.txt" || fail "$name: beta row not added"
run
expect_pass "$name (check after update)"
expect_not_in "$name" stdout "above baseline"
ok "$name"
# --- negative cases -------------------------------------------------------
name="tests below baseline fail"
new_case
make_crate alpha 1 0
baseline alpha 3 0
baseline beta 0 0
run
expect_fail "$name"
expect_in "$name" stderr "crate alpha: 1 < baseline 3"
ok "$name"
name="a drop in one crate is not hidden by an addition in another"
new_case
make_crate alpha 1 0
make_crate beta 9 0
baseline alpha 3 0
baseline beta 2 0
run
expect_fail "$name"
expect_in "$name" stderr "crate alpha: 1 < baseline 3"
expect_in "$name" stdout "note: crate beta: 9 above baseline 2"
ok "$name"
name="ignored above baseline fails with the main-branch wording"
new_case
make_crate alpha 3 2
baseline alpha 3 0
baseline beta 0 0
run
expect_fail "$name"
expect_in "$name" stderr "crate alpha: 2 ignored > baseline 0"
expect_in "$name" stderr "raise the baseline row"
expect_in "$name" stderr "PR description"
expect_not_in "$name" stderr "gateway/integration"
ok "$name"
name="ignored above fails even when tests are above baseline too"
new_case
make_crate alpha 8 1
baseline alpha 3 0
baseline beta 0 0
run
expect_fail "$name"
expect_in "$name" stderr "crate alpha: 1 ignored > baseline 0"
ok "$name"
name="a row for a crate that is no longer a member fails"
new_case
make_crate alpha 3 0
baseline alpha 3 0
baseline beta 0 0
baseline gamma 4 0
run
expect_fail "$name"
expect_in "$name" stderr "crate gamma: not a workspace member"
ok "$name"
name="malformed row fails"
new_case
printf 'alpha\tmany\t0\n' >>"$FX/.config/test-count-baseline.txt"
run
expect_fail "$name"
expect_in "$name" stderr "malformed rows"
ok "$name"
name="duplicate row fails"
new_case
baseline alpha 0 0
baseline alpha 0 0
baseline beta 0 0
run
expect_fail "$name"
expect_in "$name" stderr "duplicate rows"
ok "$name"
name="missing baseline file fails"
new_case
make_crate alpha 1 0
rm "$FX/.config/test-count-baseline.txt"
run
expect_fail "$name"
expect_in "$name" stderr "is missing"
ok "$name"
name="unknown argument is a usage error"
new_case
run --bogus
((STATUS == 2)) || fail "$name: expected exit 2, got $STATUS"
ok "$name"
echo "check_test_count_ratchet tests passed: $passed cases"
+102
View File
@@ -152,6 +152,108 @@ class PlannerTests(unittest.TestCase):
self.assertEqual(code['mode'], 'full')
self.assertIn('leaf', code['proposed']['packages'])
def release_metadata_base(self):
self.repo.write(gate.POLICY_PATH, json.dumps(dict(self.repo.policy, rollout='enforced')))
files = {
'flake.nix': '{ pkgs }: {\n rustfs = pkgs.buildRustPackage {\n pname = "rustfs";\n version = "1.0.4";\n src = ./.;\n };\n clientVersion = "0.1.36";\n}\n',
'helm/rustfs/Chart.yaml': 'apiVersion: v2\nname: rustfs\nversion: "1.0.4"\nappVersion: "1.0.4"\n',
'rustfs.spec': 'Name: rustfs\nVersion: 1.0.4\nRelease: 1\n%build\ncargo build --release\n%changelog\n* Mon Oct 26 2026 Maintainer <maintainer@example.invalid>\n- Update RPM package to RustFS 1.0.4\n',
}
for path, content in files.items():
self.repo.write(path, content)
return self.repo.commit(), files
def test_stable_installation_metadata_keeps_quick_checks_without_rust_rebuild(self):
baseline, files = self.release_metadata_base()
for path, content in files.items():
updated = content.replace('1.0.4', '1.0.5') if path != 'rustfs.spec' else content.replace('Version: 1.0.4', 'Version: 1.0.5')
if path == 'rustfs.spec':
updated = updated.replace('%changelog\n', '%changelog\n* Wed Nov 11 2026 Maintainer <maintainer@example.invalid>\n- Update RPM package to RustFS 1.0.5\n\n')
self.repo.write(path, updated)
self.repo.write('README.md', '# Install 1.0.5\n')
head = self.repo.commit()
plan = self.repo.plan(head, base=baseline)
self.assertEqual(plan['mode'], 'docs')
self.assertEqual(plan['packages'], [])
for job in gate.ALWAYS_JOBS:
self.assertEqual(plan['jobs'][job], 'success')
self.assertTrue(all(plan['jobs'][job] == 'skipped' for job in gate.CODE_JOBS))
self.assertEqual(gate.validate_plan(plan, 'pull_request', 'refs/heads/main', head, plan['tree']), plan['jobs'])
for event in ('push', 'merge_group', 'schedule', 'workflow_dispatch'):
self.assertTrue(self.repo.plan(head, event=event, base=baseline)['full_validation'])
def test_individual_stable_version_fields_without_new_rpm_history_are_metadata(self):
baseline, files = self.release_metadata_base()
for path, content in files.items():
with self.subTest(path=path):
self.repo.git('reset', '--hard', baseline)
updated = content.replace('1.0.4', '1.0.5') if path != 'rustfs.spec' else content.replace('Version: 1.0.4', 'Version: 1.0.5')
self.repo.write(path, updated)
self.assertEqual(self.repo.plan(self.repo.commit(), base=baseline)['mode'], 'docs')
def test_installation_metadata_does_not_hide_build_or_malformed_changes(self):
baseline, files = self.release_metadata_base()
cases = [
('flake.nix', files['flake.nix'].replace('src = ./.;', 'src = ./other;')),
('flake.nix', files['flake.nix'].replace('0.1.36', '0.1.37')),
('flake.nix', files['flake.nix'].replace('1.0.4', '${builtins.readFile ./version}')),
('helm/rustfs/Chart.yaml', files['helm/rustfs/Chart.yaml'] + 'dependencies: []\n'),
('helm/rustfs/Chart.yaml', files['helm/rustfs/Chart.yaml'].replace('appVersion: "1.0.4"', 'appVersion: "1.0.5"')),
('helm/rustfs/Chart.yaml', files['helm/rustfs/Chart.yaml'] + 'version: "1.0.5"\n'),
('rustfs.spec', files['rustfs.spec'].replace('cargo build --release', 'cargo build')),
('rustfs.spec', files['rustfs.spec'].replace('Release: 1', 'Release: 2')),
('rustfs.spec', files['rustfs.spec'].replace('%changelog\n', '%changelog\n%(echo injected)\n')),
('rustfs.spec', files['rustfs.spec'].replace('Update RPM package', 'Rewrite existing history')),
]
cases.extend((path, content.replace('1.0.4', '1.0.5-rc.1')) for path, content in files.items())
for path, content in cases:
with self.subTest(path=path, content=content):
self.repo.git('reset', '--hard', baseline)
self.repo.write(path, content)
self.assertTrue(self.repo.plan(self.repo.commit(), base=baseline)['full_validation'])
def test_installation_metadata_rejects_normalization_marker_collisions(self):
_, files = self.release_metadata_base()
old = files['flake.nix'].replace(' clientVersion =', ' second = pkgs.buildRustPackage {\n pname = "rustfs";\n version = "<version>";\n src = ./other;\n };\n clientVersion =')
self.repo.write('flake.nix', old)
baseline = self.repo.commit()
new = old.replace('version = "1.0.4"', 'version = "<version>"').replace('version = "<version>";\n src = ./other;', 'version = "1.0.5";\n src = ./other;')
self.repo.write('flake.nix', new)
self.assertTrue(self.repo.plan(self.repo.commit(), base=baseline)['full_validation'])
def test_installation_metadata_rejects_missing_files_symlinks_and_mode_changes(self):
baseline, files = self.release_metadata_base()
for path in files:
for mutation in ('delete', 'symlink', 'executable'):
with self.subTest(path=path, mutation=mutation):
self.repo.git('reset', '--hard', baseline)
target = self.repo.root / path
if mutation == 'executable':
target.chmod(0o755)
else:
target.unlink()
if mutation == 'symlink':
target.symlink_to('/tmp/release-metadata')
self.assertTrue(self.repo.plan(self.repo.commit(), base=baseline)['full_validation'])
self.repo.git('reset', '--hard', baseline)
self.assertTrue(self.repo.plan(baseline, base=self.repo.base)['full_validation'])
def test_installation_metadata_cannot_override_code_or_explicit_full_policy(self):
baseline, files = self.release_metadata_base()
self.repo.write('flake.nix', files['flake.nix'].replace('1.0.4', '1.0.5'))
self.repo.write('leaf/src/lib.rs', '// changed implementation\n')
plan = self.repo.plan(self.repo.commit(), base=baseline)
self.assertEqual(plan['mode'], 'full')
self.assertEqual(plan['packages'], ['consumer', 'core', 'leaf', 'rustfs'])
self.repo.git('reset', '--hard', baseline)
self.repo.write('flake.nix', files['flake.nix'].replace('1.0.4', '1.0.5'))
head = self.repo.commit()
policy = dict(self.repo.policy, rollout='enforced', full_paths=['flake.nix'])
with patch.object(gate, 'read_policy', return_value=policy):
self.assertTrue(self.repo.plan(head, base=baseline)['full_validation'])
with patch.object(gate, 'read_policy', return_value=self.repo.policy):
self.assertTrue(self.repo.plan(head, base=baseline)['full_validation'])
def test_shadow_retains_full_mode_for_harness_build_and_fixture_changes(self):
for path in ('crates/e2e_test/build.rs', 'crates/e2e_test/src/fixture.json'):
self.repo.write(path, 'changed\n')
+7 -194
View File
@@ -1,199 +1,12 @@
#!/usr/bin/env python3
"""Exercise the Docker workflow's version selection and tags without publishing."""
import os
import re
import subprocess
import tempfile
import unittest
"""Exercise the candidate-based replacement for the former release trigger."""
import importlib
import sys
from pathlib import Path
import unittest
ROOT = Path(__file__).resolve().parents[1]
WORKFLOW = ROOT / ".github/workflows/docker.yml"
REGISTRIES = ("rustfs/rustfs", "ghcr.io/rustfs/rustfs", "quay.io/rustfs/rustfs")
def run_body(name):
lines = WORKFLOW.read_text().splitlines()
start = lines.index(f" - name: {name}")
start = lines.index(" run: |", start) + 1
body = []
for line in lines[start:]:
if line.strip() and not line.startswith(" "):
break
body.append(line[10:])
return "\n".join(body)
class DockerWorkflowTests(unittest.TestCase):
def run_step(self, name, context, **env):
script = re.sub(
r"\$\{\{\s*(.*?)\s*\}\}", lambda m: context[m[1]], run_body(name)
)
# Fail on unexpected Git calls so these tests cannot contact a remote.
mock_git = """
git() {
if [[ "$1" == rev-parse ]]; then
echo 0123456789abcdef
elif [[ "$1 $2 $3" == "ls-remote --tags --refs" ]]; then
if [[ "$STABLE_EXISTS" == true ]]; then
echo "0123456789abcdef refs/tags/v1.0.0"
fi
elif [[ "$1 $2" == "ls-remote --exit-code" ]]; then
[[ "$4" == "$EXISTING_TAG" ]]
else
return 99
fi
}
"""
with tempfile.TemporaryDirectory() as directory:
output = Path(directory) / "output"
result = subprocess.run(
["bash", "-e", "-o", "pipefail", "-c", mock_git + script],
env={**os.environ, "GITHUB_OUTPUT": str(output), **env},
text=True,
capture_output=True,
check=False,
)
values = (
dict(line.split("=", 1) for line in output.read_text().splitlines())
if output.exists()
else {}
)
return result, values
def classify(
self, version, event="workflow_run", push="true", stable="true", tag=None, **env
):
return self.run_step(
"Check build conditions",
{"github.event_name": event},
GITHUB_SHA="workflow-sha",
HEAD_SHA="release-sha",
HEAD_BRANCH=version,
INPUT_VERSION=version,
INPUT_PUSH_IMAGES=push,
INPUT_FORCE_REBUILD="false",
STABLE_EXISTS=stable,
EXISTING_TAG=tag or f"refs/tags/{version}",
**{"CONCLUSION": "success", "TRIGGERING_EVENT": "push", **env},
)
def assert_tags(self, values, suffix="", channels=()):
context = {
f"needs.build-check.outputs.{key}": value for key, value in values.items()
}
context.update(
{
"matrix.suffix": suffix,
"env.REGISTRY_DOCKERHUB": REGISTRIES[0],
"env.REGISTRY_GHCR": REGISTRIES[1],
"env.REGISTRY_QUAY": REGISTRIES[2],
"github.server_url": "https://github.com",
"github.repository": "rustfs/rustfs",
}
)
result, metadata = self.run_step("Extract metadata and generate tags", context)
self.assertEqual(result.returncode, 0, result.stderr)
expected = {
f"{registry}:{tag}{suffix}"
for registry in REGISTRIES
for tag in (values["version"], *channels)
}
self.assertEqual(set(metadata["tags"].split(",")), expected)
def test_preview_version_tags_only(self):
versions = (
"1.0.1-preview.8",
"v1.0.1-preview.8",
"1.0.0-alpha.1-preview.2",
"1.0.0-beta.1-preview.2",
"v1.0.0-rc.1-preview.2",
)
for event in ("workflow_run", "workflow_dispatch"):
for version in versions:
for stable in ("true", "false"):
with self.subTest(event=event, version=version, stable=stable):
result, values = self.classify(version, event, stable=stable)
self.assertEqual(result.returncode, 0, result.stderr)
for key, expected in {
"should_build": "true",
"should_push": "true",
"build_type": "preview",
"is_prerelease": "true",
"create_latest": "false",
"version": version.removeprefix("v"),
}.items():
self.assertEqual(values[key], expected, key)
expected_ref = (
"release-sha"
if event == "workflow_run"
else f"refs/tags/{version}"
)
self.assertEqual(values["source_ref"], expected_ref)
for suffix in ("", "-glibc"):
self.assert_tags(values, suffix)
def test_manual_preview_dry_run_and_tag_fallback(self):
result, values = self.classify(
"1.0.1-preview.8",
"workflow_dispatch",
push="false",
tag="refs/tags/v1.0.1-preview.8",
)
self.assertEqual(result.returncode, 0, result.stderr)
self.assertEqual(values["should_build"], "true")
self.assertEqual(values["should_push"], "false")
self.assertEqual(values["source_ref"], "refs/tags/v1.0.1-preview.8")
def test_full_tag_ref(self):
result, values = self.classify("refs/tags/v1.0.1-preview.8")
self.assertEqual(result.returncode, 0, result.stderr)
self.assertEqual(values["build_type"], "preview")
self.assertEqual(values["version"], "1.0.1-preview.8")
self.assert_tags(values)
def test_invalid_preview_and_missing_tag_fail(self):
for event in ("workflow_run", "workflow_dispatch"):
for version in (
"1.0.1-preview",
"1.0.1-preview.x",
"1.0.1-preview.8-extra",
):
with self.subTest(event=event, version=version):
result, _ = self.classify(version, event)
self.assertNotEqual(result.returncode, 0)
result, _ = self.classify(
"1.0.1-preview.8", "workflow_dispatch", tag="refs/tags/missing"
)
self.assertNotEqual(result.returncode, 0)
def test_existing_release_channels(self):
for event in ("workflow_run", "workflow_dispatch"):
for stable in ("true", "false"):
for channel in (None, "alpha", "beta", "rc"):
version = f"1.0.1-{channel}.1" if channel else "1.0.1"
with self.subTest(event=event, stable=stable, channel=channel):
result, values = self.classify(version, event, stable=stable)
self.assertEqual(result.returncode, 0, result.stderr)
latest = channel is None or stable == "false"
self.assertEqual(values["create_latest"], str(latest).lower())
channels = ([channel] if channel else []) + (
["latest"] if latest else []
)
self.assert_tags(values, channels=channels)
def test_non_release_builds_are_skipped(self):
for version, env in (
("main", {}),
("feature/test", {}),
("1.0.1-preview.8", {"CONCLUSION": "failure"}),
("1.0.1-preview.8", {"TRIGGERING_EVENT": "workflow_dispatch"}),
):
result, values = self.classify(version, **env)
self.assertEqual(result.returncode, 0, result.stderr)
self.assertEqual(values["should_build"], "false")
sys.path.insert(0, str(Path(__file__).resolve().parent / "release"))
module = importlib.import_module("test_pipeline")
if __name__ == "__main__":
unittest.main()
unittest.main(module=module, defaultTest=('PromotionTests', 'ImageWorkflowTests'))
+3 -1
View File
@@ -21,6 +21,8 @@ assert_version() {
rm -f "$output"
}
assert_version "refs/tags/v1.0.0-beta.12" "v1.0.0-beta.12" "1.0.0-beta.12" "0.12.0"
assert_version "refs/tags/v1.0.0-beta.12" "v1.0.0-beta.12" "1.0.0-beta.12" "1.0.0-beta.12"
assert_version "2.0.0-beta.12" "2.0.0-beta.12" "2.0.0-beta.12" "2.0.0-beta.12"
assert_version "1.0.6-rc.1" "1.0.6-rc.1" "1.0.6-rc.1" "1.0.6-rc.1"
assert_version "v1.0.0" "v1.0.0" "1.0.0" "1.0.0"
assert_version "refs/tags/1.0.0" "1.0.0" "1.0.0" "1.0.0"
+7 -130
View File
@@ -1,135 +1,12 @@
#!/usr/bin/env python3
"""Exercise the Helm publication image gate without contacting a registry."""
import os
import subprocess
import tempfile
import unittest
"""Exercise the candidate-based replacement for the former release trigger."""
import importlib
import sys
from pathlib import Path
import unittest
ROOT = Path(__file__).resolve().parents[1]
WORKFLOW = ROOT / ".github/workflows/helm-package.yml"
def image_gate():
lines = WORKFLOW.read_text().splitlines()
start = lines.index(" - name: Wait for release image")
start = lines.index(" run: |", start) + 1
body = []
for line in lines[start:]:
if line.strip() and not line.startswith(" "):
break
body.append(line[10:])
return "\n".join(body)
class HelmReleaseWorkflowTests(unittest.TestCase):
def run_gate(self, version="1.0.1", failures=0, auth="ready"):
# Unexpected URLs fail closed; these functions cannot access the network.
mocks = r"""
curl() {
case "${*: -1}" in
'https://auth.docker.io/token?service=registry.docker.io&scope=repository:rustfs/rustfs:pull')
echo auth >> "$CALLS"
case "$AUTH_STATE" in
unavailable) return 22 ;;
empty) echo '{}' ;;
ready) echo '{"token":"registry-test-token"}' ;;
*) return 99 ;;
esac
;;
"https://registry-1.docker.io/v2/rustfs/rustfs/manifests/$APP_VERSION")
[[ " $* " == *" --head "* ]] || return 99
[[ "$*" == *"Authorization: Bearer registry-test-token"* ]] || return 99
echo image >> "$CALLS"
[[ $(grep -c '^image$' "$CALLS") -gt "$IMAGE_FAILURES" ]]
;;
*) return 99 ;;
esac
}
sleep() {
[[ "$1" == 15 ]] || return 99
echo sleep >> "$CALLS"
}
"""
with tempfile.TemporaryDirectory() as directory:
calls = Path(directory) / "calls"
result = subprocess.run(
["bash", "-e", "-o", "pipefail", "-c", mocks + image_gate()],
env={
**os.environ,
"APP_VERSION": version,
"IMAGE_FAILURES": str(failures),
"AUTH_STATE": auth,
"CALLS": str(calls),
},
capture_output=True,
text=True,
check=False,
timeout=30,
)
events = calls.read_text().splitlines() if calls.exists() else []
return result, events
def test_published_stable_and_prerelease_images(self):
for version in ("1.0.1", "1.1.0-beta.1", "1.1.0-rc.1"):
with self.subTest(version=version):
result, events = self.run_gate(version)
self.assertEqual(result.returncode, 0, result.stderr)
self.assertEqual(events, ["auth", "image"])
def test_image_published_after_binary_build(self):
result, events = self.run_gate(failures=2)
self.assertEqual(result.returncode, 0, result.stderr)
self.assertEqual(events.count("image"), 3)
self.assertEqual(events.count("sleep"), 2)
def test_missing_image_blocks_publication_after_bounded_retries(self):
result, events = self.run_gate(failures=60)
self.assertNotEqual(result.returncode, 0)
self.assertIn("refusing to publish Helm chart", result.stderr)
self.assertEqual(events.count("image"), 60)
self.assertEqual(events.count("sleep"), 59)
def test_auth_failure_is_not_image_availability(self):
for auth in ("unavailable", "empty"):
with self.subTest(auth=auth):
result, events = self.run_gate(auth=auth)
self.assertNotEqual(result.returncode, 0)
self.assertEqual(events.count("auth"), 60)
self.assertNotIn("image", events)
def test_preview_and_invalid_versions_never_reach_registry(self):
for version in ("1.0.1-preview.1", "1.1.0-beta.1-preview.2", "latest", "", "../tags"):
with self.subTest(version=version):
result, events = self.run_gate(version)
self.assertNotEqual(result.returncode, 0)
self.assertEqual(events, [])
def test_image_gate_precedes_packaging(self):
workflow = WORKFLOW.read_text()
gate = workflow.index(" - name: Wait for release image")
package = workflow.index(" - name: Package Helm Chart")
self.assertLess(gate, package)
self.assertNotIn("continue-on-error", workflow[gate:package])
def test_upload_artifact_includes_chart_and_readme(self):
workflow = WORKFLOW.read_text()
copy_readme = workflow.index("cp helm/README.md helm/rustfs/")
upload = workflow.index(" - name: Upload helm package as artifact")
self.assertLess(copy_readme, upload)
publish = workflow.index(" publish-helm-package:", upload)
upload_step = workflow[upload:publish]
expected_path = "path: |\n helm/rustfs/*.tgz\n helm/rustfs/README.md\n"
self.assertIn(expected_path, upload_step)
download = workflow.index(" - name: Download helm package", publish)
push_step = workflow.index(" - name: Push helm package and index file", download)
download_step = workflow[download:push_step]
self.assertIn("name: helm-package", download_step)
self.assertIn("path: ./", download_step)
sys.path.insert(0, str(Path(__file__).resolve().parent / "release"))
module = importlib.import_module("test_pipeline")
if __name__ == "__main__":
unittest.main()
unittest.main(module=module, defaultTest=('HelmWorkflowTests',))
+172
View File
@@ -0,0 +1,172 @@
#!/usr/bin/env python3
# Copyright 2026 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Contract tests for junit_diff.py: exit codes, the difference table and the JSON summary."""
import json
import subprocess
import sys
import tempfile
import unittest
from pathlib import Path
SCRIPT = Path(__file__).resolve().parent / "junit_diff.py"
FIXTURES = Path(__file__).resolve().parent / "tests" / "junit_diff"
def run(a, b, *extra):
"""Run the script on two fixture names; return (exit code, stdout, stderr)."""
proc = subprocess.run(
[sys.executable, str(SCRIPT), str(FIXTURES / a), str(FIXTURES / b), *extra],
capture_output=True, text=True, check=False)
return proc.returncode, proc.stdout, proc.stderr
class JunitDiffTests(unittest.TestCase):
# --- negative cases: differences and bad input ---
def test_pass_vs_fail_is_difference(self):
code, out, _ = run("base.xml", "pass_fail.xml")
self.assertEqual(code, 1)
self.assertIn("rustfs::s3::alpha", out)
self.assertRegex(out, r"rustfs::s3::alpha\s+pass\s+fail")
def test_pass_vs_error_counts_as_fail(self):
code, out, _ = run("base.xml", "pass_error.xml")
self.assertEqual(code, 1)
self.assertRegex(out, r"rustfs::s3::alpha\s+pass\s+fail")
def test_pass_vs_skip_is_difference(self):
code, out, _ = run("base.xml", "pass_skip.xml")
self.assertEqual(code, 1)
self.assertRegex(out, r"rustfs::s3::alpha\s+pass\s+skip")
def test_case_missing_on_b_is_difference(self):
code, out, _ = run("base.xml", "missing_case.xml")
self.assertEqual(code, 1)
self.assertRegex(out, r"rustfs::s3::beta\s+pass\s+missing")
def test_case_missing_on_a_is_difference(self):
code, out, _ = run("missing_case.xml", "base.xml")
self.assertEqual(code, 1)
self.assertRegex(out, r"rustfs::s3::beta\s+missing\s+pass")
def test_rerun_failure_final_fail_is_difference(self):
code, out, _ = run("base.xml", "rerun_fail.xml")
self.assertEqual(code, 1)
self.assertRegex(out, r"rustfs::s3::alpha\s+pass\s+fail")
def test_malformed_xml_exits_2(self):
for a, b in (("malformed.xml", "base.xml"), ("base.xml", "malformed.xml")):
code, out, err = run(a, b)
self.assertEqual(code, 2, (out, err))
self.assertNotIn("No differences", out)
def test_empty_file_exits_2(self):
code, _, err = run("base.xml", "empty.xml")
self.assertEqual(code, 2)
self.assertIn("empty.xml", err)
def test_report_without_cases_exits_2(self):
code, _, _ = run("no_cases.xml", "no_cases.xml")
self.assertEqual(code, 2)
def test_unexpected_root_exits_2(self):
code, _, _ = run("wrong_root.xml", "base.xml")
self.assertEqual(code, 2)
def test_nonexistent_file_exits_2(self):
code, _, _ = run("base.xml", "does_not_exist.xml")
self.assertEqual(code, 2)
def test_stale_allow_entry_fails_run(self):
code, out, _ = run("base.xml", "pass_fail.xml", "--allow", str(FIXTURES / "allow_stale.txt"))
self.assertEqual(code, 1)
self.assertIn("rustfs::s3::nonexistent", out)
self.assertIn("stale", out.lower())
def test_allow_entry_without_any_difference_is_stale(self):
code, out, _ = run("base.xml", "base.xml", "--allow", str(FIXTURES / "allow_ok.txt"))
self.assertEqual(code, 1)
self.assertIn("stale", out.lower())
def test_allow_does_not_cover_other_differences(self):
code, out, _ = run("base.xml", "missing_case.xml", "--allow", str(FIXTURES / "allow_ok.txt"))
self.assertEqual(code, 1)
def test_unreadable_allow_file_exits_2(self):
code, _, _ = run("base.xml", "base.xml", "--allow", str(FIXTURES / "nope.txt"))
self.assertEqual(code, 2)
# --- positive cases ---
def test_identical_files_exit_0(self):
code, out, _ = run("base.xml", "base.xml")
self.assertEqual(code, 0)
self.assertIn("No differences", out)
def test_bare_testsuite_root_matches_nested(self):
code, _, _ = run("base.xml", "bare_suite.xml")
self.assertEqual(code, 0)
def test_allowed_difference_exits_0(self):
code, out, _ = run("base.xml", "pass_fail.xml", "--allow", str(FIXTURES / "allow_ok.txt"))
self.assertEqual(code, 0)
self.assertIn("allowed", out.lower())
def test_allowed_missing_case_exits_0(self):
code, _, _ = run("base.xml", "missing_case.xml", "--allow", str(FIXTURES / "allow_missing_case.txt"))
self.assertEqual(code, 0)
def test_flaky_pass_listed_not_a_difference(self):
code, out, _ = run("base.xml", "flaky_pass.xml")
self.assertEqual(code, 0)
self.assertIn("rustfs::s3::alpha", out)
self.assertIn("flaky", out.lower())
# --- options and summary ---
def test_labels_appear_in_table(self):
code, out, _ = run("base.xml", "pass_skip.xml", "--label-a", "reference", "--label-b", "candidate")
self.assertEqual(code, 1)
self.assertIn("reference", out)
self.assertIn("candidate", out)
def test_json_summary(self):
with tempfile.TemporaryDirectory() as tmp:
path = Path(tmp) / "out.json"
code, _, _ = run("base.xml", "flaky_pass.xml", "--json", str(path),
"--label-a", "reference", "--label-b", "candidate")
self.assertEqual(code, 0)
data = json.loads(path.read_text())
self.assertEqual(data["counts"]["reference"], {"pass": 3, "fail": 0, "skip": 1})
self.assertEqual(data["counts"]["candidate"], {"pass": 3, "fail": 0, "skip": 1})
self.assertEqual(data["differences"], [])
self.assertEqual(data["flaky"]["candidate"], ["rustfs::s3::alpha"])
self.assertEqual(data["flaky"]["reference"], [])
self.assertEqual(data["stale_allow"], [])
def test_json_records_differences(self):
with tempfile.TemporaryDirectory() as tmp:
path = Path(tmp) / "out.json"
code, _, _ = run("base.xml", "missing_case.xml", "--json", str(path))
self.assertEqual(code, 1)
data = json.loads(path.read_text())
self.assertEqual(data["differences"],
[{"case": "rustfs::s3::beta", "a": "pass", "b": "missing", "allowed": False}])
self.assertEqual(data["counts"]["b"], {"pass": 2, "fail": 0, "skip": 1})
if __name__ == "__main__":
unittest.main()
+106
View File
@@ -0,0 +1,106 @@
#!/usr/bin/env python3
# Copyright 2026 RustFS Team
# SPDX-License-Identifier: Apache-2.0
"""Exercise real OCI builders and promotion with explicit non-product fixtures."""
import importlib
import os
from pathlib import Path
import shutil
import subprocess
import sys
import tempfile
import time
import unittest
import urllib.request
from unittest.mock import patch
sys.path.insert(0, str(Path(__file__).resolve().parent / "release"))
c = importlib.import_module("candidate")
deliver = importlib.import_module("deliver")
publish = importlib.import_module("publish")
fixtures = importlib.import_module("test_candidate")
class ImageIntegrationTests(unittest.TestCase):
def test_native_oci_smoke_and_digest_preserving_registry_promotion(self):
value = fixtures.inputs()
variant = os.environ["RELEASE_IMAGE_VARIANT"]
self.assertIn(variant, ("musl", "gnu"))
output = Path(os.environ["RELEASE_IMAGE_OUTPUT"]).resolve()
output.mkdir(parents=True, exist_ok=True)
run = subprocess.run
with tempfile.TemporaryDirectory() as temp:
root = Path(temp)
for name in ("Dockerfile", "Dockerfile.glibc", "entrypoint.sh"):
shutil.copyfile(c.ROOT / name, root / name)
shutil.copytree(c.ROOT / "bootstrap", root / "bootstrap")
(root / "scripts").mkdir()
shutil.copyfile(c.ROOT / "scripts/download_rc.sh", root / "scripts/download_rc.sh")
source = root / "fixture.go"
source.write_text('''package main
import ("fmt"; "net/http"; "os")
func main() {
if len(os.Args) > 1 && os.Args[1] == "--version" { fmt.Println("rustfs 1.2.3"); return }
http.HandleFunc("/rustfs/console/", func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "text/html"); fmt.Fprintln(w, "<html>transport fixture</html>")
})
if err := http.ListenAndServe(":9001", nil); err != nil { panic(err) }
}
''')
binaries = {}
for arch, docker_arch in (("x86_64", "amd64"), ("aarch64", "arm64")):
directory = root / ".release-inputs" / docker_arch
directory.mkdir(parents=True)
env = dict(os.environ, GOOS="linux", GOARCH=docker_arch, CGO_ENABLED="0")
run(["go", "build", "-o", str(directory / "rustfs"), str(source)], env=env, check=True, timeout=180)
(directory / "rc").write_text("#!/bin/sh\nexit 0\n")
(directory / "rc").chmod(0o755)
binaries[arch] = c.file_digest(directory / "rustfs")
archive = output / f"image-{variant}.tar"
dockerfile = "Dockerfile" if variant == "musl" else "Dockerfile.glibc"
run(["docker", "buildx", "build", "--platform", "linux/amd64,linux/arm64", "--provenance=false",
"--build-arg", "BINARY_SOURCE=supplied", "--build-arg", "RELEASE=1.2.3", "-f", str(root / dockerfile),
"--output", f"type=oci,dest={archive}", str(root)], check=True, timeout=1800)
c.write_json(output / f"image-{variant}-inputs.json", {"inputs_sha256": c.digest(value), "binaries": binaries})
deliver.image_smoke(value, variant, output)
with self.assertRaisesRegex(ValueError, "wrong version"):
deliver.image_smoke(dict(value, version="0.0.0"), variant, output)
expected = c.read_json(output / f"image-{variant}-smoke.json")["digest"]
registry = c.command(["docker", "run", "-d", "-p", "127.0.0.1::5000", "registry:3"])
try:
address = c.command(["docker", "port", registry, "5000/tcp"])
opener = urllib.request.build_opener(urllib.request.ProxyHandler({}))
deadline = time.monotonic() + 15
while time.monotonic() < deadline:
try:
with opener.open(f"http://{address}/v2/", timeout=2) as response:
self.assertEqual(response.status, 200)
break
except OSError:
time.sleep(0.1)
else:
self.fail("local registry did not become ready")
auth = root / "auth.json"
c.write_json(auth, {"auths": {}})
reference = f"{address}/fixture:1.2.3"
def local_registry(args, **kwargs):
# Only the ephemeral loopback registry uses HTTP. All OCI
# copying, digest checks and conflict handling are real.
if args[:2] == ["skopeo", "inspect"]:
args = args[:2] + ["--tls-verify=false"] + args[2:]
elif args[:2] == ["skopeo", "copy"]:
args = args[:2] + ["--dest-tls-verify=false"] + args[2:]
return run(args, **kwargs)
with patch.dict(os.environ, REGISTRY_AUTH_FILE=str(auth)), patch.object(subprocess, "run", side_effect=local_registry):
publish.copy_image(archive, reference, expected)
publish.copy_image(archive, reference, expected)
self.assertEqual(publish.image_digest(reference), expected)
with self.assertRaisesRegex(ValueError, "never replace"):
publish.copy_image(archive, reference, "sha256:" + "0" * 64)
self.assertEqual(publish.image_digest(reference), expected)
finally:
run(["docker", "rm", "-f", "-v", registry], check=True, timeout=60)
if __name__ == "__main__":
unittest.main()
+68
View File
@@ -0,0 +1,68 @@
#!/usr/bin/env python3
# Copyright 2026 RustFS Team
# SPDX-License-Identifier: Apache-2.0
"""Linux integration tests using the real DEB/RPM recipes and package managers."""
import hashlib
import importlib
import os
from pathlib import Path
import shutil
import subprocess
import sys
import tempfile
import unittest
from unittest.mock import patch
sys.path.insert(0, str(Path(__file__).resolve().parent / "release"))
deliver = importlib.import_module("deliver")
c = importlib.import_module("candidate")
fixtures = importlib.import_module("test_candidate")
class PackageIntegrationTests(unittest.TestCase):
@classmethod
def setUpClass(cls):
missing = [tool for tool in ("dpkg-deb", "fakeroot", "rpm", "rpm2cpio", "cpio", "fpm") if not shutil.which(tool)]
if sys.platform != "linux" or missing:
raise RuntimeError(f"Run on Linux with real packaging tools; missing: {missing}")
def test_each_target_wraps_its_accepted_bytes(self):
for target in c.LINUX:
with self.subTest(target=target), tempfile.TemporaryDirectory() as temp:
root = Path(temp)
executable = root / "rustfs"
executable.write_bytes(b"package contract fixture only: " + target.encode())
expected = hashlib.sha256(executable.read_bytes()).hexdigest()
value = fixtures.inputs()
with patch.object(deliver, "binary", return_value=(executable, {"binary_sha256": expected})):
deliver.package(value, target, root / "output")
record = c.read_json(root / "output" / f"packages-{target}.json")
self.assertEqual(record["binary_sha256"], expected)
self.assertEqual(record["inputs_sha256"], c.digest(value))
self.assertEqual({Path(item["name"]).suffix for item in record["files"]}, {".deb", ".rpm"})
def test_real_package_with_substituted_payload_is_rejected(self):
with tempfile.TemporaryDirectory() as temp:
root = Path(temp)
executable = root / "rustfs"
executable.write_bytes(b"accepted fixture")
expected = c.file_digest(executable)
real_run = deliver.run
def mutate_deb(args, **kwargs):
result = real_run(args, **kwargs)
if any(str(arg).endswith("build_deb.sh") for arg in args):
work = Path(kwargs["cwd"])
package = work / kwargs["env"]["DEB_FILE"]
unpacked = work / "corrupted-package"
subprocess.run(["dpkg-deb", "--raw-extract", str(package), str(unpacked)], check=True)
(unpacked / "usr/bin/rustfs").write_bytes(b"wrong server")
subprocess.run(["dpkg-deb", "--build", str(unpacked), str(package)], check=True)
return result
with patch.object(deliver, "binary", return_value=(executable, {"binary_sha256": expected})), patch.object(deliver, "run", side_effect=mutate_deb):
with self.assertRaisesRegex(ValueError, "package binary differs"):
deliver.package(fixtures.inputs(), "linux-x86_64-musl", root / "output")
self.assertFalse(list((root / "output").glob("packages-*.json")))
if __name__ == "__main__":
unittest.main()
@@ -0,0 +1 @@
rustfs::s3::beta
+3
View File
@@ -0,0 +1,3 @@
# tolerated while the gateway lane stabilises
rustfs::s3::alpha
+3
View File
@@ -0,0 +1,3 @@
rustfs::s3::alpha
rustfs::s3::nonexistent
+6
View File
@@ -0,0 +1,6 @@
<testsuite name="rustfs" tests="4">
<testcase name="alpha" classname="rustfs::s3"/>
<testcase name="beta" classname="rustfs::s3"/>
<testcase name="gamma" classname="rustfs::s3"><skipped/></testcase>
<testcase name="delta" classname="rustfs::iam"/>
</testsuite>
+11
View File
@@ -0,0 +1,11 @@
<?xml version="1.0" encoding="UTF-8"?>
<testsuites name="nextest-run">
<testsuite name="rustfs" tests="3">
<testcase name="alpha" classname="rustfs::s3"/>
<testcase name="beta" classname="rustfs::s3"/>
<testcase name="gamma" classname="rustfs::s3"><skipped/></testcase>
</testsuite>
<testsuite name="other" tests="1">
<testcase name="delta" classname="rustfs::iam"/>
</testsuite>
</testsuites>

Some files were not shown because too many files have changed in this diff Show More