Compare commits

..

16 Commits

Author SHA1 Message Date
overtrue c32886b5ea fix(ecstore): bound reader retirement warnings in the decoder
A slow shard reader is retired once per stripe of every read it lags on, and a parity reader that cannot be aligned is retired with a warning each time. Both lines name only a shard index. They now log at most once per five seconds with the number suppressed; the retirement itself and the error recorded for the shard are unchanged.
2026-11-11 17:38:58 +08:00
overtrue 3a98517df9 fix(ecstore): bound bitrot and remote disk failure logs
A failing drive logs one error per shard block of every read it serves, and an unreachable peer logs one warning per in-flight operation. Neither had a rate limit, so a degraded drive or peer could produce log volume at the speed of the failing I/O.

The three bitrot read failures (short shard, hash mismatch, read error) and the two remote disk failures (timeout, network-like error) now log at most once per five seconds and report how many events were suppressed. The remote disk throttles belong to each disk handle, so one noisy peer cannot hide another peer's first warning. Error results, health marking and the per-operation counters are unchanged.
2026-11-11 17:10:53 +08:00
Chris An 04bcb57e6a fix(ecstore): pin multipart index markers while their root is published (#9318)
A multipart index marker is recorded before its staging root is published, and a listing prunes markers whose root is gone. The window in between was covered by a five-minute age check, which cannot prove that the publisher's filesystem work has ended: a publisher stalled past it lost its marker, and the upload it then published stayed out of ListMultipartUploads on that drive until restart.

The publisher now holds an in-memory publication pin from before it records the marker until the filesystem work that publishes the root has returned. The pin travels with that work (the blocking closure or the lease owner chain), so it also covers work that outlives a cancelled caller. Pruning reads the pin first and the staging root second, under the marker lease. The age check and the per-write marker touch are removed.
2026-11-11 03:22:46 -05:00
Chris An 31e1e42ad7 refactor: land the s3s decoupling groundwork on main (#9276)
Lands gateway/integration on main: the guard rails and the domain
decoupling for replacing s3s with rustfs/gateway (rustfs/backlog#2734,
Phase 0 and Phase 1). The S3 edge is still s3s and no wire behaviour
change is intended.

- Guard rails: lower-only s3s footprint baseline and edge allow-list,
  per-crate test-count floor, 39 byte-level goldens for the persisted
  bucket-config XML families, a two-server differential harness over
  all 73 S3 operations, a JUnit differ, and a reference-parity lane
  against the sha256-pinned 1.0.5 binary.
- Owned types: rustfs-s3-types (S3Error, S3ErrorCode, Body),
  RequestEnvelope, AuthzFacts.
- Engine crates off s3s: ecstore, lifecycle, replication, scanner,
  protocols, s3select, s3-client, rpc.
- Admin: error types moved; 300 of 314 Operation impls take the owned
  AdminRequest.
- Server: the s3s-compat tower layers grouped under
  server/legacy_compat; RUSTFS_S3_STACK=legacy|gateway selector, where
  gateway refuses to start until the gateway edge exists.

Fixes made on the branch that change ecstore and heal behaviour are
listed in the pull request description. The per-commit history stays
on gateway/integration.
2026-11-11 02:39:52 -05:00
Chris An 0e402b2534 fix(scanner): retain child objects beneath object prefixes (#9277) 2026-11-11 14:36:53 +08:00
Chris An db39cd5634 fix(ecstore): let missing-key deletes complete on UUID-named folders (#9280)
* fix(ecstore): let missing-key deletes complete on UUID-named folders

Deleting a key that does not exist probes the key's directory for
interrupted delete transactions. The probe was bounded at 4096 entries
per drive and charged every entry inside a UUID-named child, so a key
naming a folder of roughly 1,400 or more UUID-named child keys answered
503 SlowDown on every retry.

Inside a UUID-named child only journal files and data-directory markers
can be evidence, so every other name (a child key's own xl.meta, data
directories or children) is now skipped unread at 1/64 like plain
children of the folder. The per-call allowance of the bounded protocol
rises to 262,144 entries per drive; the resumable background step keeps
4096 and the request budget keeps its sizing, so the per-request total
is unchanged. The probe runs on the durable-mutation lane instead of
the one-worker foreground lane, and stops incomplete when its caller
goes away or after a third of the drive timeout (at most 10 s), so a
slow walk answers SlowDown instead of timing out the drive.

* test(ecstore): keep discovery fixtures meaningful under 1/64 charging

Part files inside a data directory are inert to object-scoped discovery
now, so reclaim_orphan_data_dirs_drains_empty_discovery_pages fills its
first step with the referenced data directories of many versions, which
still cost a full entry each. The charging test drives the walk directly
instead of through the wall-clock-limited disk entry point, so a loaded
machine cannot turn it into an incomplete probe.
2026-11-10 23:24:50 -05:00
Ramakrishna Chilaka 89d01aef4f perf(ecstore): reduce syscalls and directory re-opens in PUT and DELETE (#9264)
Co-authored-by: RustFS <hello@rustfs.com>
Co-authored-by: Hauser Zhi <housemecn@gmail.com>
2026-11-11 00:06:11 +00:00
dependabot[bot] 86c3395d07 chore(deps): bump the dependencies group across 1 directory with 12 updates (#9251)
Bumps the dependencies group with 12 updates in the / directory:

| Package | From | To |
| --- | --- | --- |
| [async-nats](https://github.com/nats-io/nats.rs) | `0.51.0` | `0.52.0` |
| [tar-codec](https://github.com/astral-sh/astral-codec) | `0.0.15` | `0.0.16` |
| [tar-framing](https://github.com/astral-sh/astral-codec) | `0.0.15` | `0.0.16` |
| [flate2](https://github.com/rust-lang/flate2-rs) | `1.1.10` | `1.1.11` |
| [smallvec](https://github.com/servo/rust-smallvec) | `1.16.4` | `1.16.5` |
| [starshard](https://github.com/houseme/starshard) | `2.4.0` | `2.4.1` |
| [zip](https://github.com/zip-rs/zip2) | `9.3.0` | `9.4.1` |
| [dial9](https://github.com/dial9-rs/dial9) | `0.5.7` | `0.5.8` |
| [rustfs-mimalloc](https://github.com/houseme/rustfs-mimalloc) | `0.7.3` | `0.7.4` |
| [hotpath](https://github.com/hotpath-rs/hotpath) | `0.30.3` | `0.31.1` |
| [rustfs-mimalloc-sys](https://github.com/houseme/rustfs-mimalloc) | `0.7.3` | `0.7.4` |
| [toml](https://github.com/toml-rs/toml) | `1.2.0+spec-1.1.0` | `1.2.1+spec-1.1.0` |



Updates `async-nats` from 0.51.0 to 0.52.0
- [Release notes](https://github.com/nats-io/nats.rs/releases)
- [Commits](https://github.com/nats-io/nats.rs/compare/async-nats/v0.51.0...async-nats/v0.52.0)

Updates `tar-codec` from 0.0.15 to 0.0.16
- [Release notes](https://github.com/astral-sh/astral-codec/releases)
- [Commits](https://github.com/astral-sh/astral-codec/compare/v0.0.15...v0.0.16)

Updates `tar-framing` from 0.0.15 to 0.0.16
- [Release notes](https://github.com/astral-sh/astral-codec/releases)
- [Commits](https://github.com/astral-sh/astral-codec/compare/v0.0.15...v0.0.16)

Updates `flate2` from 1.1.10 to 1.1.11
- [Release notes](https://github.com/rust-lang/flate2-rs/releases)
- [Changelog](https://github.com/rust-lang/flate2-rs/blob/main/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/flate2-rs/compare/1.1.10...v1.1.11)

Updates `smallvec` from 1.16.4 to 1.16.5
- [Release notes](https://github.com/servo/rust-smallvec/releases)
- [Commits](https://github.com/servo/rust-smallvec/compare/v1.16.4...v1.16.5)

Updates `starshard` from 2.4.0 to 2.4.1
- [Changelog](https://github.com/houseme/starshard/blob/main/CHANGELOG.md)
- [Commits](https://github.com/houseme/starshard/compare/2.4.0...2.4.1)

Updates `zip` from 9.3.0 to 9.4.1
- [Release notes](https://github.com/zip-rs/zip2/releases)
- [Changelog](https://github.com/zip-rs/zip2/blob/master/CHANGELOG.md)
- [Commits](https://github.com/zip-rs/zip2/compare/v9.3.0...v9.4.1)

Updates `dial9` from 0.5.7 to 0.5.8
- [Release notes](https://github.com/dial9-rs/dial9/releases)
- [Changelog](https://github.com/dial9-rs/dial9/blob/main/CHANGELOG.md)
- [Commits](https://github.com/dial9-rs/dial9/compare/dial9-v0.5.7...dial9-v0.5.8)

Updates `rustfs-mimalloc` from 0.7.3 to 0.7.4
- [Release notes](https://github.com/houseme/rustfs-mimalloc/releases)
- [Changelog](https://github.com/houseme/rustfs-mimalloc/blob/main/CHANGELOG.md)
- [Commits](https://github.com/houseme/rustfs-mimalloc/compare/v0.7.3...v0.7.4)

Updates `hotpath` from 0.30.3 to 0.31.1
- [Release notes](https://github.com/hotpath-rs/hotpath/releases)
- [Changelog](https://github.com/hotpath-rs/hotpath/blob/main/CHANGELOG.md)
- [Commits](https://github.com/hotpath-rs/hotpath/compare/v0.30.3...v0.31.1)

Updates `rustfs-mimalloc-sys` from 0.7.3 to 0.7.4
- [Release notes](https://github.com/houseme/rustfs-mimalloc/releases)
- [Changelog](https://github.com/houseme/rustfs-mimalloc/blob/main/CHANGELOG.md)
- [Commits](https://github.com/houseme/rustfs-mimalloc/compare/v0.7.3...v0.7.4)

Updates `toml` from 1.2.0+spec-1.1.0 to 1.2.1+spec-1.1.0
- [Commits](https://github.com/toml-rs/toml/compare/toml-v1.2.0...toml-v1.2.1)

---
updated-dependencies:
- dependency-name: async-nats
  dependency-version: 0.52.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: dependencies
- dependency-name: tar-codec
  dependency-version: 0.0.16
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: tar-framing
  dependency-version: 0.0.16
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: flate2
  dependency-version: 1.1.11
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: smallvec
  dependency-version: 1.16.5
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: starshard
  dependency-version: 2.4.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: zip
  dependency-version: 9.4.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: dependencies
- dependency-name: dial9
  dependency-version: 0.5.8
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: rustfs-mimalloc
  dependency-version: 0.7.4
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: hotpath
  dependency-version: 0.31.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: dependencies
- dependency-name: rustfs-mimalloc-sys
  dependency-version: 0.7.4
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: toml
  dependency-version: 1.2.1+spec-1.1.0
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Hauser Zhi <housemecn@gmail.com>
2026-11-11 00:04:47 +00:00
cxymds c9743f166e fix(s3): normalize leading-slash keys in bulk deletes (#9258)
* fix(s3): normalize leading-slash keys in bulk deletes

* test(s3): align bulk delete reserved-prefix guard expectations
2026-11-10 17:56:00 -05:00
Boyi (Elliot) Li 50707a610c test(utils,rio,checksums): add checksum SIMD equivalence and bench UTs (#9217)
Add unaligned-boundary SIMD vs scalar equivalence unit tests and on-demand (#[ignore]) utility-level throughput microbenchmarks across rustfs-utils, rustfs-rio, and rustfs-checksums:
- crates/utils/src/hash.rs: verify HighwayHasher SIMD (AVX2/SSE4.1/NEON) against highway::PortableHash scalar reference and contiguous hash_encode vs odd-chunked hash_encode_slices across unaligned tail boundaries; provide on-demand throughput microbenchmark.
- crates/rio/src/checksum.rs: verify SIMD-accelerated CRC32, CRC32C, and CRC64_NVME against bitwise scalar reference implementations across vector tail boundaries and measure utility-level throughput across all 10 BASE_CHECKSUM_TYPES.
- crates/checksums/src/lib.rs: verify streaming chunked update + finalize + headers() against one-shot digests across all 9 ChecksumAlgorithm variants for unaligned chunk and payload sizes, plus on-demand throughput microbenchmark.
2026-11-10 17:54:11 -05:00
Patrick Wicki ba3e0e1546 fix(admin): keep measured bucket usage when the overlay is unconverged (#9243)
The single-process usage overlay clears usage_snapshot_complete when it
holds counts it cannot reconcile with the persisted snapshot, for
example writes into a bucket that has no verified baseline. The admin
datausageinfo path then judged the overlaid document as structurally
incomplete and replaced it with an empty one, hiding every bucket the
scanner had measured. Because each write restarts that bucket's scanner
fence, a continuously written bucket kept the whole response empty
indefinitely.

Decide whether the persisted snapshot is usable before the overlay is
applied, and report overlay-induced incompleteness as a non-converged
snapshot instead, as is already done for buckets the scan never reached
(rustfs#5806).
2026-11-10 17:52:06 -05:00
Chris An 95e128336d ci: narrow checks for stable installation metadata (#9269)
* ci: narrow checks for stable installation metadata

* fix: reject release metadata marker collisions
2026-11-10 16:10:04 -05:00
Chris An 8b7bbdb17d chore(release): align installation references for 1.0.5 (#9268) 2026-11-10 14:48:02 -05:00
Chris An 2780bba67b ci: stage release builds and promote verified artifacts (#9265)
* ci: stage release builds and promote verified artifacts

* fix: preserve release package and channel retry contracts

* fix: align installation chart versions with sealed candidates
2026-11-10 13:15:05 -05:00
hector 3a23122104 ci: run macOS release builds on macos-latest-xlarge (#9260)
Use the larger GitHub-hosted macOS runner for the macos-aarch64
release target. The target id, triple, and other platforms stay the same.

Co-authored-by: hector <majinghe@users.noreply.github.com>
2026-11-10 18:55:19 +08:00
Ramakrishna Chilaka 0b1ebeffec perf(ecstore): reduce syscalls in multipart copy, commit, and delete (#9247) 2026-11-10 12:27:10 +08:00
639 changed files with 52181 additions and 15686 deletions
+21 -26
View File
@@ -1,43 +1,38 @@
---
name: rustfs-release-publish
description: Publish a RustFS version through Console, release-branch CI, preview acceptance, human confirmation, and final publication. Use only for an explicit release request.
description: Publish a RustFS version through frozen Linux candidate acceptance, desktop completion, manifest approval, byte-preserving publication and post-release follow-up. Use only for an explicit release request.
---
# RustFS Release
Require the exact final SemVer target before version edits or publication. If unspecified, inspect current releases/tags and ask with concrete candidates; do not guess the channel or bump. A named bump type resolves to an exact version that still needs confirmation. Report the confirmed target verbatim; resolve any later mismatch before proceeding. Independent read-only preflight may continue.
Require the exact final SemVer target before version edits or publication. If unspecified, inspect releases/tags and ask with concrete candidates; do not guess the channel or bump. Reuse the user's confirmed target and authorized delivery scope. Independent read-only preflight may continue.
## Invariants
- Cargo stores `<target>`, never `-preview.N`. Before publication, installation defaults retain the previous available deliverable. Final and preview binaries embed their own tag names, so version/asset names differ despite identical source.
- Use annotated tags without `v`. Preview names are exactly `<target>-preview.<digits>`; choose the next unused iteration after fetching tags. Preview Releases are prereleases, never Latest; publish only versioned assets/exact Docker tags, never latest assets, R2, Helm, moving Docker channels, or installation/banner updates.
- `release/<target>` owns the candidate. Retain its history, backport only reviewed fixes, and never merge main wholesale or force-push it. Pin `PREVIEW_HASH` to its exact commit with complete passing CI. Preview and final tags both point there; never tag a later main/branch HEAD or insert another bump.
- Phase failures block dependents. Keep preview Releases until final publication and notes generation finish; CI removes Releases while retaining tags. Never move an existing preview or final tag.
- Cargo and every candidate executable contain the final product version. `<target>-preview.<run_number>` is an immutable candidate identifier, not a public tag or displayed binary version.
- Freeze the exact source, reviewed workflow recipe, toolchain, Console, CLI, dependency lock, previous stable version and validation plan. Newer unrelated main or Console releases do not change those inputs.
- Retain `release/<target>` history when a release branch is used; backport only reviewed fixes. Pin `CANDIDATE_SHA` to its exact CI-validated commit. Never force-push a release branch or move a public tag.
- Build Linux first. Complete field acceptance before macOS/Windows and packaging. Full certification and final acceptance are required before publication; a partial Linux PASS grants no release authority.
- Promote the accepted bytes. Final publication never recompiles binaries or rebuilds packages/images. Infrastructure retries reuse successful artifacts; changed source or inputs require a new candidate and affected acceptance.
- Phase failures block dependents. No latest/default-channel update before versioned delivery. Use the [pipeline runbook](../../../docs/operations/release-pipeline.md) for executable entry points, locks and recovery.
## Read by Phase
Read only the current phase's procedure; retain completed evidence across turns. Follow root verification, Git/PR, and authorization rules.
## Read by phase
| Phase | Procedure |
|---|---|
| 0 | Clean tree, fetch main/tags, verify GitHub access, then complete the mandatory [Console release gate](references/console-gate.md). A green Console build alone is insufficient. |
| 1–2 | [Prepare/reuse the release candidate and run exact-SHA branch CI](references/release-candidate.md). Source version changes use `rustfs-release-version-bump` with authorized delivery scope. |
| 3 | [Verify the preview build, assets, channels, and notes](references/preview-release.md). |
| 4–5 | [Run the downloaded binary, Console CRUD, and latest-rc matrix](references/preview-acceptance.md). Every check must pass. |
| Confirmation | Complete the gate below and end the turn. |
| 6 | After fresh confirmation, [publish and verify the final tag](references/final-release.md) at `PREVIEW_HASH`. |
| 7 | [Reconcile Security Advisories, milestones, installation references, and the website; reconcile main and remove the completed release branch](references/post-release-updates.md) only after complete publication. |
| 0 | Clean tree, fetch main/tags, verify access, then complete the mandatory [Console gate](references/console-gate.md). |
| 1–2 | [Prepare the exact source and run branch CI](references/release-candidate.md). Version edits use `rustfs-release-version-bump`. |
| 3 | [Build and verify the Linux candidate](references/preview-release.md). |
| 4–5 | [Run artifact, field, Console and rc acceptance](references/preview-acceptance.md), then complete desktop/package/image/chart validation. |
| Confirmation | Report the complete manifest and obtain the confirmation below. |
| 6 | [Publish the accepted manifest](references/final-release.md). |
| 7 | [Reconcile advisories, milestones, installation references, website and release branch](references/post-release-updates.md). |
## Human Confirmation
## Human confirmation
After Phases 3–5 pass, recheck that the release branch still equals `PREVIEW_HASH` and that the current Console asset tag/digest match the preview's recorded build inputs. Report the target, branch, preview tag/hash, branch CI run, preview Release URL, accepted Console asset identity/result, and rc matrix. Explicitly ask whether to publish the final tag and end the turn without creating/pushing it.
After all acceptance passes, report the target, source SHA, source CI run, Linux and completion run URLs, final manifest SHA-256, accepted Console/CLI identities, compatibility/performance results and backlog report. Explicitly ask whether to publish this exact manifest and end the turn without dispatching publication.
Only a new affirmative user reply to that exact report authorizes Phase 6. The original release request, earlier confirmation, silence, or automated follow-up does not count. Confirmation binds the target, preview tag, hash, and accepted build inputs; a changed value or repeated/failed acceptance cycle invalidates it. Revalidate and obtain fresh confirmation before continuing.
Only a new affirmative reply to that exact report authorizes Phase 6. An earlier general release request, silence or automatic continuation does not count. Reuse confirmation for retries of that same manifest; do not ask again merely because a destination failed. A changed source, input, manifest or acceptance result invalidates it and requires renewed acceptance/confirmation.
## Recovery
## Recovery and reporting
- Before the final tag exists, land fixes on main, backport to the candidate, verify target versions, and restart branch CI/acceptance. Use a new preview iteration once a preview was tagged; main may advance independently.
- If only an external build input changed, record it, create a new preview iteration even when the source SHA is unchanged, and repeat the affected build and artifact acceptance before fresh confirmation. Reuse source checks whose inputs remain valid.
- Once the final tag exists, retry publication jobs for that exact tag. A source fix requires a newly confirmed target, not retagging or a newer preview. Phase 7 retries recheck artifact availability and superseding releases.
- If abandoned after the bump merged, report the unpublished version on main and record whether to revert or let the next release overwrite it. Deleting unpublished candidate branches or preview tags requires an explicit decision. Completed release branches follow Phase 7 cleanup; keep release tags.
Report completed phase results and incomplete work with evidence: Console previous/latest tags, release need, hash and run/Release URLs; target, branch CI/SHA, previews and hash, confirmation state; preview/final assets, latest-channel state, Console/rc results and cleanup; advisory reconciliation results and pending disclosure decisions; both milestone states, installation PR checks/merge state, and website text/link/PR/deployment state. Keep artifact publication separate from follow-up delivery; do not call an open PR a live update. Report deviations and their authorization.
Resume failed infrastructure jobs without rebuilding successful units. Fix product defects on main, backport when needed, and start a new candidate. After a final tag exists, retry the same bytes or use a new patch version; never move the tag. Do not conflate implemented, CI-passed, field-accepted, approved, published and post-release-complete states. Report incomplete work and deviations with evidence; a tag or green build alone is not a completed release.
@@ -4,7 +4,7 @@ Read during Phase 0, before changing RustFS version files or tags. Follow the pa
### Console release gate
Complete this gate before changing any RustFS version file or creating any RustFS tag. RustFS `build.yml` downloads the asset returned by `repos/rustfs/console/releases/latest`, so a successful Console build alone is insufficient.
Complete this gate before changing any RustFS version file or creating any RustFS tag. Candidate creation resolves `repos/rustfs/console/releases/latest` once and freezes the exact asset for every build, so a successful Console build alone is insufficient. After freezing, later Console releases do not silently change or invalidate the accepted inputs.
1. Read the latest published Console tag and compare it with Console `main`:
@@ -1,22 +1,13 @@
# Final Release Publication
# Final release publication
Read only after fresh user confirmation of the preview acceptance report. The [parent confirmation gate](../SKILL.md#human-confirmation) applies.
Read only after fresh confirmation of the complete candidate report. The [parent confirmation gate](../SKILL.md#human-confirmation) applies.
## Phase 6 — Publish the final tag on the validated commit
Dispatch `release-publish.yml` on main with the successful completion run ID and approved `release-manifest.json` SHA-256. Do not create the tag manually. The workflow revalidates source, bytes and trusted certification, then creates the annotated final tag at `CANDIDATE_SHA`. A recipe change fails closed and requires a new candidate; unrelated main changes do not require a rebuild.
No second source version bump. Before creating the final tag, recheck the remote `release/<target>` HEAD equals `PREVIEW_HASH` and the current Console release tag/asset digest still match the accepted preview inputs. If either changed after confirmation, the approval no longer applies; validate the changed inputs with a new preview and obtain fresh confirmation. The final tag goes on the exact commit the preview validated:
Require both versioned publication and channel publication jobs to succeed. Verify the final tag's commit and manifest identity; all six platform ZIPs and compatibility aliases; DEB/RPM, checksums, SBOM, provenance and evidence bundle; both multi-architecture image variants in Docker Hub, GHCR and Quay; and the accepted Helm chart in the chart repository. Download a host-compatible final artifact and confirm its digest is exactly the accepted candidate digest and its version/source output agrees.
```bash
git fetch origin --tags
git rev-parse "<preview-tag>^{commit}" # must equal PREVIEW_HASH — abort if not
git tag -a "<target>" -m "Release <target>" "$PREVIEW_HASH"
git push origin "<target>"
```
For stable releases verify GitHub Latest, R2 download aliases and `latest.json` unless a newer stable release already owns those channels. Prereleases must remain GitHub prereleases and leave stable pointers unchanged. Verify generated release notes compare against the frozen previous stable release, excluding internal previews.
- CI rebuilds from the same source with a different tag name, so the binary self-reports `<target>`. Other inputs can change: `build.yml` reads Console `releases/latest` during each build. Compare the actual Console tag, URL, and verified digest in every final build job with the accepted preview record, and inspect available build provenance for other relevant changes. A pre-tag check cannot prevent a later input change. On a mismatch or missing evidence, repeat affected acceptance against the final artifacts and report the limitation; failed acceptance leaves release validation incomplete even if publication already succeeded. Never move the final tag to conceal a failure.
- Verify the final tag's complete publication path: all matrix and release jobs green; `gh release view "<target>"` shows the full versioned and `-latest` asset set plus checksums, SBOM, and provenance; Docker and Helm workflows succeed; `latest.json` points to `<target>`. A stable target must have `isPrerelease=false` and `isLatest=true`. An alpha/beta/rc target must have `isPrerelease=true`; GitHub does not permit prereleases to be Latest, but the project `latest.json` still advances to the final non-preview target.
- Verify the final Release body contains `## What's Changed` and a Full Changelog link. When `PREVIOUS_DELIVERABLE` exists, the link MUST be `https://github.com/rustfs/rustfs/compare/<PREVIOUS_DELIVERABLE>...<target>` and the baseline MUST equal the preview Release baseline; for example, both `1.0.0-beta.12-preview.1` and `1.0.0-beta.12` compare from `1.0.0-beta.11`.
- Verify the preview cleanup: `cleanup-preview-releases` must succeed and `gh release view "<preview-tag>"` must then report `release not found` for every preview iteration of this target. Each surviving preview tag must still resolve to its own recorded commit; the accepted preview resolves to `PREVIEW_HASH`. If the job failed, delete the leftover Releases manually with `gh release delete "<preview-tag>" --yes` and report it.
- Verify a host-compatible final artifact's checksum and `./rustfs --version`; it must report `<target>` and the expected source SHA. This identity check does not replace affected acceptance when build inputs changed.
On infrastructure failure retry the same workflow/seal; matching destinations are reused. Never clobber an asset, force a tag, rebuild the image or silently accept a changed digest. A source fix after the final tag requires a new patch version. Cross-registry writes are not atomic: report the unfinished destination rather than claiming publication complete.
Never pass `--cleanup-tag` when removing preview Releases. Remove them manually only if `cleanup-preview-releases` failed, after final publication and notes generation. Record deleted Releases and surviving tags.
New candidates create no preview Releases to clean up. For a legacy preview release, preserve its tag and remove its Release only after final delivery and notes are verified. Do not pass `--cleanup-tag`. Complete the [post-release follow-up](post-release-updates.md) separately.
@@ -24,7 +24,7 @@ Reconcile advisories after every completed final release, including Console-only
1. Re-read the published deliverable state before edits or a retry. If a newer release has superseded this target, do not downgrade installation references or the website announcement. Report the superseding release and leave those defaults intact.
2. Invoke `rustfs-release-version-bump` with stage `post-release`, the exact target, and the authorized delivery scope. Prepare the follow-up on current main; do not edit or move either validated release tag. Cargo versions may already be preparing the next target and must remain untouched.
3. Verify the exact Docker image manifest, Release download links, RPM source archive, and Helm repository entry before publishing the installation PR. The chart workflow overrides chart/app versions from the final tag, pins the source checkout to that release, and waits for its default Docker image before packaging.
3. Verify the exact Docker image manifest, Release download links, RPM source archive, and Helm repository entry before publishing the installation PR. The chart is packaged and tested before sealing; publication uploads those same bytes only after versioned Docker images are available.
4. Report the installation PR and its merge state separately from the completed release. Reuse existing merge authorization and repository checks; do not infer new merge authority from this reference.
## rustfs.com top banner
@@ -54,4 +54,4 @@ Use the existing announcement configuration from [website PR #112](https://githu
1. Compare the exact published tag with current main. Audit every release-side commit since their merge base, including inherited backports. An ahead count is a history difference, not proof of missing changes. Use ancestry and patch equivalence, then inspect current code and regression evidence for squashed, adapted, superseded, or reverted changes. Record each commit as present, superseded, release-only, or missing; a positive `git cherry` result alone does not prove a missing fix.
2. Bring missing shared changes to main through a reviewed PR and the required checks before closing the follow-up. Preserve main's later changes; decide version bumps and pinned Console configuration separately. If a history merge is needed, use a merge commit and verify the release commit is an ancestor of main; squash does not establish that ancestry. Do not merge main into the published release branch or move its tags. Report unresolved changes instead of claiming reconciliation is complete.
3. After publication and authorized follow-ups are complete, delete the remote `release/<target>` branch. Immediately before deletion, verify its HEAD still equals the published tag's commit and no open PR targets or originates from it. Preserve a branch with newer work or open PRs and report the blocker. Use a deletion with an expected-old-SHA check to avoid removing a concurrently advanced branch.
4. Verify the remote branch is absent and the final and preview tags still resolve to their recorded commits. Remove a local release branch only if it is task-owned, unchecked out, and has no unpreserved work. Record cleanup separately from artifact publication; an explicitly authorized deletion before reconciliation does not make unresolved main changes complete.
4. Verify the remote branch is absent and the final tag and any legacy preview tags still resolve to their recorded commits. Remove a local release branch only if it is task-owned, unchecked out, and has no unpreserved work. Record cleanup separately from artifact publication; an explicitly authorized deletion before reconciliation does not make unresolved main changes complete.
@@ -1,10 +1,10 @@
# Preview Artifact Acceptance
# Candidate Artifact Acceptance
Read after Phase 3 succeeds. Complete every check below before the parent skill's manual confirmation gate. These checks cover the downloaded artifact, embedded Console, and latest rc client.
Read after Phase 3 succeeds. Complete every check below before the parent skill's manual confirmation gate. These checks cover the downloaded artifact, embedded Console, and the rc release frozen in the candidate inputs.
## Phase 4 — Run the artifact locally, verify the console
Use a unique task-owned scratch directory. Select the versioned archive matching the host OS, architecture, and runtime, download its checksum file, and verify the archive before extraction. Do not substitute a different build when a matching artifact is unavailable; report that acceptance gap. The shell example below assumes the verified executable is at `RUSTFS_BINARY` (`rustfs.exe` on Windows), `SCRATCH` is the task directory, and `S3_PORT` and `CONSOLE_PORT` are distinct available ports. Use equivalent native commands when the shell differs.
Use a unique task-owned scratch directory. Select the versioned archive matching the acceptance host, and verify it against the sealed manifest before extraction. Before desktop completion, use the Linux acceptance environment rather than rebuilding for the local desktop. Do not substitute a different build when a matching artifact is unavailable; report that acceptance gap. The shell example below assumes the verified executable is at `RUSTFS_BINARY` (`rustfs.exe` on Windows), `SCRATCH` is the task directory, and `S3_PORT` and `CONSOLE_PORT` are distinct available ports. Use equivalent native commands when the shell differs.
Bind both listeners to loopback. Choose task-only credentials, set loopback proxy exclusions for the server/client processes, and record the server PID. A port probe does not reserve a port; retry startup with fresh ports on a bind conflict rather than connecting to an existing service.
@@ -24,16 +24,16 @@ PREVIEW_SERVER_PID=$!
Checks (all must pass):
- `"$RUSTFS_BINARY" --version` reports the preview tag name and the short SHA of `PREVIEW_HASH`. Reporting `<target>` without the `-preview.N` suffix means the build did not embed the tag — treat as FAIL and investigate before proceeding.
- `"$RUSTFS_BINARY" --version` reports the exact final `<target>` and full `CANDIDATE_SHA`. A preview suffix, another version or another source is a failure.
- Wait with a bounded readiness deadline while checking the recorded server process remains alive; `curl --noproxy '*' -fsS "$S3_ENDPOINT/health/ready"` must return ready from this process's listener.
- Startup log shows the embedded console being served (this was the regression that `fix(release): require embedded console assets` guards).
- Open `CONSOLE_ENDPOINT` in the browser and log in with the task credentials. The dashboard must render without JS console errors; create a bucket, upload a file, download it back (byte-identical), and delete the object and bucket. Keep the server running for Phase 5.
## Phase 5 — Validate with the latest rc client
## Phase 5 — Validate with the pinned rc client
`rc` is the RustFS CLI client from <https://github.com/rustfs/cli>.
- Resolve the latest release with `gh api repos/rustfs/cli/releases/latest --jq .tag_name`; verify the local client's version or download and verify a host-compatible client into the task directory. Set `RC_BIN` to that executable and record its version/digest. Avoid changing the user's installed client for this check.
- Read the CLI release and asset digests from `inputs.json`; verify the local client's version or download a host-compatible asset from that exact release into the task directory. If the host is a desktop, resolve its asset within the same pinned CLI release and record the digest; do not independently follow `latest`. Set `RC_BIN` to that executable and record its version/digest. Avoid changing the user's installed client for this check.
- Inspect the client's current help for an isolated configuration option and use it when supported. Otherwise choose a unique `PREVIEW_ALIAS` and confirm it is absent before creating it; never overwrite an existing alias. Use task-owned bucket/user names and input/output files. The names below are safe only inside the newly started isolated server.
- Point the client at `S3_ENDPOINT` with loopback proxy exclusions and run the command matrix, recording PASS/FAIL per command:
@@ -60,3 +60,9 @@ cmp <local-file> "$SCRATCH/out"
```
- Any FAIL blocks acceptance. On success or failure, stop only the recorded task process and remove only task-created aliases/data after retaining the needed failure evidence. Do not erase an existing client configuration or unrelated server data.
## Field acceptance and desktop completion
Before accepting the Linux manifest, record Scanner/Heal, multi-pool, memory, major features, high-concurrency 503, performance and previous-version integrity/upgrade results in backlog. Preserve the release-specific agreed limits and baseline; a PASS label without artifact identities or test results does not satisfy acceptance.
Dispatch `release-complete.yml` only after that report passes, using the exact Linux manifest digest and backlog URL. Require native macOS/Windows checks, package extraction, image startup/scans, chart tests and complete certification to pass. Complete any desktop Console/client checks applicable to the release against these same archives. Record the completion run and final manifest SHA-256 before entering the parent confirmation gate. Partial Linux acceptance does not authorize final publication.
@@ -1,25 +1,9 @@
# Preview Release Verification
# Linux candidate verification
Read for Phase 3 after the candidate passed exact-SHA CI. The [parent release invariants](../SKILL.md) apply.
Read for Phase 3 after the source passed exact-SHA CI. The [parent invariants](../SKILL.md) apply; commands and recovery live in the [pipeline runbook](../../../../docs/operations/release-pipeline.md).
## Preview channels
Dispatch `release-candidate.yml` on main with the confirmed final version and `CANDIDATE_SHA`. Record the run ID, frozen `inputs.json`, every Linux archive digest and `linux-manifest.json` SHA-256. Require the four Linux builds, native startup/S3 checks, previous-release upgrade matrix and Linux seal to succeed. Missing, skipped, cancelled or failed required work is incomplete acceptance.
- Use `<target>-preview.N` for every target, e.g. `1.0.0-beta.10-preview.3` or `1.1.0-preview.1`.
- The canonical suffix is exactly `-preview.<digits>`. `build.yml` recognizes it before alpha/beta/rc classification and routes it to the preview-only path; any other tag containing `-preview` fails closed instead of being treated as a release.
- A preview Release MUST be published with `isPrerelease=true` and `isLatest=false`. Any `*-latest` preview asset or preview-triggered `latest.json`, R2, Helm, or moving Docker channel tag is a pipeline failure. Docker images may use the exact preview version tag, including its existing variant suffixes.
- Preview Releases are cleaned up by the `cleanup-preview-releases` job after `publish-release` succeeds for the deliverable tag. It deletes every Release whose tag is exactly `<target>-preview.<digits>` and never passes `--cleanup-tag`, so the tags survive.
Verify the frozen Console identity equals the one accepted in Phase 0. Every matrix job consumes that same digest and exact toolchain. The binary must report the final product version and source commit; its candidate ID is separate. No public tag, Release, registry tag, R2 release path, Helm repository or latest pointer should change during this stage.
## Phase 3 — CI and preview Release verification
- Find and watch the tag build: `gh run list --workflow build.yml --branch "<preview-tag>" --limit 1` then `gh run watch <run-id>`. Every build matrix target must succeed (linux x86_64/aarch64 × musl/gnu, macos-aarch64, windows-x86_64).
- Confirm the preview tag resolves to the Phase 2 CI-validated `PREVIEW_HASH`. If the release branch advances during acceptance, repeat Phases 2–5 with a new preview tag.
- Confirm the Release publication jobs (`create-release`, `upload-release-assets`, and `publish-release`) succeed while `update-latest-version` is skipped.
- Record the inputs actually consumed by each build job: source SHA and the embedded Console release tag, asset URL, and verified SHA256 from the download logs. The workflow reads Console `releases/latest` separately for each build; confirm the matrix used the Console asset accepted in Phase 0. A missing identity or differing asset blocks acceptance until resolved. Retain available toolchain and build provenance with the artifact checksums; the same source SHA alone does not establish equivalent builds.
- Verify `gh release view "<preview-tag>" --json isPrerelease,assets,url`: `isPrerelease` must be `true`, and the Release must contain all 6 versioned platform zips, checksums, SBOM, and provenance with no `-latest` assets. Confirm `gh api repos/{owner}/{repo}/releases/latest --jq .tag_name` does not return `<preview-tag>`.
- Record `PREVIOUS_DELIVERABLE`, selected from published Releases by `publishedAt` after excluding the current tag and every `-preview.N` tag. Verify `gh release view "<preview-tag>" --json body --jq .body` contains `## What's Changed` and, when `PREVIOUS_DELIVERABLE` exists, `**Full Changelog**: https://github.com/rustfs/rustfs/compare/<PREVIOUS_DELIVERABLE>...<preview-tag>`. For a repository with no previous deliverable, verify a Full Changelog link exists and record the GitHub baseline fallback.
- Confirm Helm is skipped. If preview Docker images are published, confirm they use only exact preview tags (and variant suffixes), with no `latest`, `alpha`, `beta`, or `rc` channel updates. Preview validation covers the built RustFS binaries, embedded console, and rc compatibility; it does not authorize advancing any installation default.
## Release notes
- When a previous deliverable exists, GitHub Release notes for the preview and final tags MUST use it as their shared comparison baseline: the most recently published non-preview Release before the target. Internal `-preview.N` Releases are explicitly excluded from that selection, even when they point at the same commit as the final tag — cleanup runs after the notes are generated, so the preview Release is still present and would otherwise be picked as the baseline. If no previous deliverable exists, omit `previous_tag_name` and record that GitHub's default baseline fallback was used.
- Generated Release notes carry a workflow-management marker so retries can repair them. Before manually curating a generated body, remove that marker; unmarked non-placeholder notes are preserved by later workflow runs.
A green Linux run is ready for field acceptance, not publication. The partial certification must have `publish_allowed: false`. Retain the exact Linux manifest for the next stage; a new source/build-input candidate must not inherit the old approval. A newer unrelated main commit does not invalidate this evidence.
@@ -16,22 +16,13 @@ Read for Phases 1–2. The [parent release invariants](../SKILL.md) and Console
- Only when neither the release branch, an unfinished preview tag, nor current main already has `<target>`: if main is at a later version, select and verify a historical commit containing `<target>` rather than downgrading main; block if none exists. Otherwise invoke the `rustfs-release-version-bump` skill with stage `prepare`, the final `<target>` (NOT a preview version), and full GitHub flow (commit/push/PR).
- If a bump PR was needed, get it merged into main and record its merge commit. For a new release branch when main already has `<target>`, identify the main commit that introduced those versions. Phase 2 chooses a release candidate containing that commit; an existing release branch keeps its own history and a newer main HEAD is not required.
Do not wait for the moving latest main HEAD to become green. The release-branch CI in Phase 2 validates the chosen commit before a preview tag is created.
Do not wait for the moving latest main HEAD to become green. The release-branch CI in Phase 2 validates the chosen commit before a candidate build is dispatched.
## Phase 2 — Pin and validate the release candidate, then publish the preview tag
## Phase 2 — Pin and validate the release source
- Use `release/<target>`. On the first attempt, create it from a selected main commit containing the version bump and intended changes; it need not be the latest main HEAD. On a restart, retain the existing branch and backport only reviewed fixes from main. Verify the branch's Cargo versions still equal `<target>`. If adopting an existing preview, run this branch CI before continuing its acceptance; reuse the tag only when it points to the validated SHA.
- The current `ci.yml` runs automatically on main pushes, not release-branch pushes. Dispatch it explicitly with `gh workflow run ci.yml --ref "release/<target>"`. Locate that dispatch run, verify its `head_sha` equals the release branch's remote HEAD, and require the full expected CI job set to pass, including `Quick Checks`, `Test and Lint`, and `End-to-End Tests (full merge gate)`. Treat a missing or skipped required lane as incomplete. A successful PR check or CI run for a different SHA does not qualify. If the branch advances or a run is cancelled, validate the new SHA again. Do not tag a candidate with failing or incomplete CI.
- Use `release/<target>`. On the first attempt, create it from a selected main commit containing the version bump and intended changes; it need not be the latest main HEAD. On a restart, retain the existing branch and backport only reviewed fixes from main. Verify the branch's Cargo versions still equal `<target>`. A legacy preview can anchor source selection, but its tag and old artifacts do not constitute a candidate in the new protocol; freeze and verify a new candidate after the release recipe is aligned.
- The current `ci.yml` runs automatically on main pushes, not release-branch pushes. Dispatch it explicitly with `gh workflow run ci.yml --ref "release/<target>"`. Locate that dispatch run, verify its `head_sha` equals the release branch's remote HEAD, and require the full expected CI job set to pass, including `Quick Checks`, `Test and Lint`, and `End-to-End Tests (full merge gate)`. Treat a missing or skipped required lane as incomplete. A successful PR check or CI run for a different SHA does not qualify. If the branch advances or a run is cancelled, validate the new SHA again. Do not dispatch a candidate with failing or incomplete CI.
- Prefer protecting `release/*` against force pushes and deletion and requiring reviewed backport PRs. Check live rulesets rather than assuming the main ruleset covers release branches; report a missing release-branch rule as a gap.
- Set `PREVIEW_HASH` to the CI-validated release-branch commit and recheck the remote branch HEAD immediately before tagging. Report the branch, commit, and CI run URL; do not derive `PREVIEW_HASH` from a later `origin/main` fetch. When adopting an existing preview tag at this SHA, skip tag creation and continue Phase 3.
```bash
git tag -a "<preview-tag>" -m "Release <preview-tag>" "$PREVIEW_HASH"
git push origin "<preview-tag>"
```
Pushing the tag triggers `.github/workflows/build.yml` ("Build and Release"); `docker.yml` chains off it via `workflow_run`.
The preview run builds versioned artifacts and publishes them in a GitHub prerelease. Docker may publish exact preview image tags. Latest-channel, R2, and Helm publication must be skipped; installation references and the website announcement stay unchanged.
On a restart after a backport, re-run release-branch CI before setting the new `PREVIEW_HASH`. Use the next unused preview iteration when the validated commit differs from an existing preview tag; never move an existing tag.
- Set `CANDIDATE_SHA` to the CI-validated release-branch commit. Record the branch, exact commit and CI run URL. The new candidate protocol starts from a commit whose release recipe matches the reviewed main dispatcher; backport the recipe before freezing an older release branch if needed.
- Dispatch the Linux candidate as described in [candidate verification](preview-release.md). Do not create a preview or final tag here. The final executable already contains `<target>`; publication later promotes those same bytes.
- After a source or build-input change, run the affected source checks and create a new candidate. Reuse the frozen candidate after an infrastructure failure. Do not restart acceptance for unrelated main advances.
@@ -4,18 +4,18 @@ description: Prepare an exact RustFS Cargo version or align installation referen
---
# Release Version Files
Require an exact target; ask if missing or ambiguous. Reject targets containing `-preview`: those are tag-only. Infer `post-release` only from an explicit request or parent workflow; otherwise use `prepare`. Delivery defaults to local edit/verify; commit, push, PR, and merge follow existing authorization and [Git rules](../../references/pull-requests.md).
Require an exact target; ask if missing or ambiguous. Reject targets containing `-preview`: those identify candidates, not product versions or new public tags. Infer `post-release` only from an explicit request or parent workflow; otherwise use `prepare`. Delivery defaults to local edit/verify; commit, push, PR, and merge follow existing authorization and [Git rules](../../references/pull-requests.md).
| Stage | Files and constraints |
|---|---|
| `prepare` | Only `Cargo.toml` and `Cargo.lock`: workspace package, internal dependency, and workspace member versions become `<target>`. External dependencies stay unchanged. |
| `post-release` | Only `README.md`, `README_ZH.md`, `flake.nix`, `helm/rustfs/Chart.yaml`, and `rustfs.spec`. Cargo may already target the next release; leave it untouched. |
Before installation edits, verify the exact non-preview Release, downloadable assets/source archive, pullable `rustfs/rustfs:<target>` image manifest, and published Helm chart. Block on missing artifacts. Do not downgrade defaults if a newer deliverable superseded the target. Preview preparation retains the previous published installation references; follow-up commits never replace either validated tag.
Before installation edits, verify the exact non-preview Release, downloadable assets/source archive, pullable `rustfs/rustfs:<target>` image manifest, and published Helm chart. Block on missing artifacts. Do not downgrade defaults if a newer deliverable superseded the target. Candidate preparation retains the previous published installation references; follow-up commits never replace validated tags or sealed artifacts.
Packaging policy:
- Docker image tags have no `v` prefix. Derive chart/app versions with `scripts/helm_chart_version.sh <target>` (`beta.N -> 0.N.0`; other versions unchanged). Helm CI derives versions from the final tag, so no early chart bump is required.
- Docker image tags have no `v` prefix. Derive chart/app versions with `scripts/helm_chart_version.sh <target>`; both equal the final product SemVer, including prerelease suffixes. Helm CI uses the frozen candidate version before sealing, so no early installation chart bump is required. Previously published chart versions remain unchanged.
- RPM `Version` is numeric; `Release` is the prerelease suffix or `1` for stable. Expanded `Source0` and the unpack directory must resolve to the exact target including its suffix. Add the changelog entry using current git identity, date, and target; never copy an example identity/date.
- Changing these mappings requires explicit confirmation.
+2 -2
View File
@@ -1,2 +1,2 @@
sha256-darwin=9901a4950099b94d3d2618524d540f1ed4bf6023044903886f98b6c4bf5f30fb
sha256-linux=09fd5f48f5556c9fc576357222dccd7ebb0b90b790798f1834db2ec5d67b4952
sha256-darwin=0374e6b9b061dd5d622686301386232a4aa1cb22eedc18af70d2a7411bc66f8d
sha256-linux=f2cee240f7a744ae991971820f48a6c9c28d19f525e8b2345fcb76eeaaf1e382
+1 -1
View File
@@ -1 +1 @@
sha256=e4ee3898b840a8374cb02c64342cc15f8d7ecd1becaf4b4a2a4657290f9e7ff1
sha256=4eac269f9a59b2f9be4e7a87ea6aa38454f5ebe10ede35c9b8d4218137d2cd35
+1 -1
View File
@@ -1 +1 @@
sha256=5bdd4c2eadbc41625ae3184fff34062e000d328108436259b2b7c4fe2531ecaf
sha256=fbabc660e6e01fd8ff6b9d1de7d68e58242d60c942f85aed5074b0ec2918c154
+5
View File
@@ -70,6 +70,11 @@ s3s-footprint-check: ## Check the s3s dependency footprint ratchet stays frozen
@echo "📦 Checking s3s footprint ratchet..."
./scripts/check_s3s_footprint.sh
.PHONY: test-count-ratchet-check
test-count-ratchet-check: ## Check no workspace crate fell below its test floor or rose above its #[ignore] ceiling in the committed baseline
@echo "🧪 Checking per-crate test-count ratchet..."
./scripts/check_test_count_ratchet.sh
.PHONY: fips-wording-check
fips-wording-check: ## Check docs and crates/kms do not over-claim crypto capabilities
@echo "📣 Checking cryptographic capability wording guard..."
+2 -2
View File
@@ -20,11 +20,11 @@ planning-docs-check: ## Check that no planning-type documents are committed
./scripts/check_no_planning_docs.sh
.PHONY: pre-commit
pre-commit: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check error-other-ratchet-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check s3s-footprint-check fips-wording-check embedded-secrets-check test-wiring-check doc-paths-check planning-docs-check quick-check ## Run fast pre-commit checks without clippy/full tests
pre-commit: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check error-other-ratchet-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check s3s-footprint-check test-count-ratchet-check fips-wording-check embedded-secrets-check test-wiring-check doc-paths-check planning-docs-check quick-check ## Run fast pre-commit checks without clippy/full tests
@echo "✅ All pre-commit checks passed!"
.PHONY: pre-pr
pre-pr: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check error-other-ratchet-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check s3s-footprint-check fips-wording-check embedded-secrets-check test-wiring-check doc-paths-check planning-docs-check log-analyzer-rules-check offline-enrollment-e2e-check clippy-check test ## Run full pre-PR checks with clippy and tests
pre-pr: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check error-other-ratchet-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check s3s-footprint-check test-count-ratchet-check fips-wording-check embedded-secrets-check test-wiring-check doc-paths-check planning-docs-check log-analyzer-rules-check offline-enrollment-e2e-check clippy-check test ## Run full pre-PR checks with clippy and tests
@echo "✅ All pre-PR checks passed!"
.PHONY: dev-check
+15
View File
@@ -22,6 +22,14 @@ TEST_THREADS ?= 1
# differ from CI and .config/nextest.toml test-groups will NOT apply.
.PHONY: script-tests
.PHONY: release-package-tests
release-package-tests: ## Run release package integration on Linux with fpm, dpkg and rpm installed
$(RUSTFS_PYTHON_BIN) ./scripts/test_release_packages.py
.PHONY: release-image-tests
release-image-tests: ## Run OCI integration with Docker, Go and skopeo; set RELEASE_IMAGE_VARIANT and RELEASE_IMAGE_OUTPUT
$(RUSTFS_PYTHON_BIN) ./scripts/test_release_images.py
script-tests: ## Run shell script tests
@echo "Running script tests..."
trap 'status=$$?; printf "ERROR: script-tests failed (exit %s): %s\n" "$$status" "$$BASH_COMMAND" >&2; exit "$$status"' ERR
@@ -52,11 +60,16 @@ script-tests: ## Run shell script tests
./scripts/test_python_bin.sh
$(RUSTFS_PYTHON_BIN) ./scripts/test_check_doc_paths.py
./scripts/check_embedded_secrets.sh --self-test
./scripts/check_s3s_footprint.sh --self-test
./scripts/test_check_test_count_ratchet.sh
$(RUSTFS_PYTHON_BIN) ./scripts/check_test_wiring.py --self-test
$(RUSTFS_PYTHON_BIN) ./scripts/codemods/s3_error_to_s3_types.py --self-test
$(RUSTFS_PYTHON_BIN) ./scripts/test_e2e_binary.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_download_release_asset.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_reference_binary.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_inspect_btf_prototypes.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_release_build_workflow.py
$(RUSTFS_PYTHON_BIN) -m unittest discover -s scripts/release -p "test_*.py"
$(RUSTFS_PYTHON_BIN) ./scripts/test_release_glibc.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_upgrade_workflow.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_migration_gate_evidence.py
@@ -77,8 +90,10 @@ script-tests: ## Run shell script tests
$(RUSTFS_PYTHON_BIN) ./scripts/test_inspect_validation_runner.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_retain_validation_evidence.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_ci_timing_report.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_junit_diff.py
$(RUSTFS_PYTHON_BIN) ./scripts/s3-tests/test_report_compat.py
$(RUSTFS_PYTHON_BIN) ./scripts/s3-tests/test_runner_tools.py
$(RUSTFS_PYTHON_BIN) ./scripts/s3-tests/test_reference_parity.py
bash -n ./scripts/validate_object_data_cache_cold_stampede.sh
$(RUSTFS_PYTHON_BIN) ./scripts/check_object_data_cache_follower_samples.py --self-test
./scripts/validate_object_data_cache_cold_stampede.sh --self-test
+21
View File
@@ -356,6 +356,12 @@ threads-required = "num-test-threads"
filter = 'package(rustfs-ecstore) & test(=store::init::tests::dispatch_manifest_rollback_bounded_concurrency_reaches_tail_behind_slow_member)'
threads-required = "num-test-threads"
# These short-deadline real-disk progress probes need the test run's capacity
# for scanner and object-lock handoffs; keep their assertions and deadlines.
[[profile.default.overrides]]
filter = '(package(rustfs-scanner) & (test(=scanner::tests::cycle_persist_failure::later_cycle_post_commit_failure_reloads_durable_state) | test(=scanner::tests::running_main_loop_catches_up_pause_cleared_after_startup_observe))) | (package(rustfs-ecstore) & test(=store::init::tests::restored_transitioned_delete_causally_enqueues_free_version))'
threads-required = "num-test-threads"
# ---------------------------------------------------------------------------
# ci profile — the strict CI gate (ci.yml `cargo nextest run --profile ci`)
# ---------------------------------------------------------------------------
@@ -424,6 +430,12 @@ threads-required = "num-test-threads"
filter = 'package(rustfs-scanner) & (test(/^scanner::backlog::tests::native_/) | test(=scanner_folder::tests::mrf_ownership::mrf_ownership_manager_completion_preserves_scanner_pending) | test(/^scanner_io::tests::service_cohort::/))'
threads-required = "num-test-threads"
# Keep short-deadline real-disk progress probes from competing with unrelated
# tests for scanner and object-lock capacity. Assertions and deadlines stay on.
[[profile.ci.overrides]]
filter = '(package(rustfs-scanner) & (test(=scanner::tests::cycle_persist_failure::later_cycle_post_commit_failure_reloads_durable_state) | test(=scanner::tests::running_main_loop_catches_up_pause_cleared_after_startup_observe))) | (package(rustfs-ecstore) & test(=store::init::tests::restored_transitioned_delete_causally_enqueues_free_version))'
threads-required = "num-test-threads"
# Emit this expensive fixture's stage timings in CI and use tmpfs when safe.
[[profile.ci.overrides]]
filter = 'package(rustfs-ecstore) & test(=store::init::tests::decommission_durable_ilm_receipt_pagination_fails_closed_on_second_page)'
@@ -698,6 +710,14 @@ test-group = 'ecstore-serial-flaky'
# the PR-lane e2e coverage for scanner usage snapshots consumed by quota and
# admin surfaces. It uses the same single-node, random-port, isolated-temp-dir
# fixture as the existing smoke modules.
#
# Dual-stack differential (rustfs/backlog#2740): `dual_stack_differential::`
# starts two single-node servers from the same binary, one per
# RUSTFS_S3_STACK value, on random ports with isolated temp dirs, and runs one
# SDK script covering every operation of `impl S3 for FS` against both. Its
# comparator unit tests need no server. It meets the admission criteria
# unchanged; the module clause admits a new differential test without another
# filter edit.
[profile.e2e-smoke]
default-filter = """
package(e2e_test) & (
@@ -706,6 +726,7 @@ default-filter = """
| test(/^reliant::lifecycle::/)
| test(/^conditional_delete_objects_test::/)
| test(/^reliant::tiering::/)
| test(/^dual_stack_differential::/)
| test(/^upgrade_compatibility_test::upgrade_write_readiness_tests::/)
| test(/^on_demand_migration::(get_basic_test::(get_miss_pulls_inline_and_serves_locally_afterwards|head_miss_answers_from_the_source_without_persisting)|interaction_test::test_odm_admin_config_is_redacted_and_status_counts_match_the_source)$/)
)
+20
View File
@@ -0,0 +1,20 @@
# Pinned RustFS release used as the "reference" target of parity lanes
# (.github/workflows/reference-parity.yml). This is the single place that names
# the reference release; .github/actions/reference-binary resolves, downloads and
# verifies it, and fails on a missing pin, a missing asset or a digest mismatch.
#
# Bumping the reference is a reviewed event: every per-test difference between
# the candidate and the reference is measured against this exact build.
#
# Each sha256 is the value published in the release's SHA256SUMS asset, which
# matches the digest GitHub reports for the same asset. The action checks both.
schema = 1
repository = "rustfs/rustfs"
version = "1.0.5"
tag = "1.0.5"
[platforms.linux-x86_64-gnu]
asset = "rustfs-linux-x86_64-gnu-v1.0.5.zip"
sha256 = "5addae2f74756bf80104d0975e6347cda843bc2daffdfcd1f0cb80118b95cbd0"
executable = "rustfs"
+24
View File
@@ -0,0 +1,24 @@
# Targets of the s3-tests reference parity lane (.github/workflows/reference-parity.yml).
#
# Exactly one target uses `binary = "reference"` (the release pinned in
# .config/reference-binary.toml). Every other target uses `binary = "candidate"`
# (the binary built from the branch under test) and is compared per node id
# against the reference. All targets run the same harness, test list and server
# configuration; `env` adds only the target's own RUSTFS_* settings.
#
# Adding a target is one more [[targets]] table, for example:
#
# [[targets]]
# name = "candidate-gateway"
# binary = "candidate"
# env = { RUSTFS_S3_STACK = "gateway" }
schema = 1
[[targets]]
name = "reference"
binary = "reference"
[[targets]]
name = "candidate-legacy"
binary = "candidate"
+12
View File
@@ -0,0 +1,12 @@
# Tolerated per-test differences of the s3-tests reference parity lane.
#
# One entry per line, four fields separated by " | ":
#
# <target> | <pytest node id> | <tracking issue URL> | <reason>
#
# <target> is a candidate name from .config/reference-parity-targets.toml, the
# node id is exactly as the lane prints it (s3tests/functional/test_s3.py::...),
# and the issue URL is https://github.com/rustfs/<repo>/issues/<number>.
#
# Same semantics as scripts/junit_diff.py --allow: an entry whose difference no
# longer occurs is stale and fails the lane, so remove it when it is fixed.
+135
View File
@@ -0,0 +1,135 @@
{
"schema_version": 1,
"plan_id": "rustfs-release-validation",
"description": "Certification for immutable candidates. All shipped platforms require native identity and startup; stable and prerelease require the pinned upgrade matrix. Linux-only acceptance never authorizes publication.",
"release_types": [
"preview",
"prerelease",
"stable"
],
"targets": [
{
"os": "linux",
"arch": "x86_64",
"abi": "musl"
},
{
"os": "linux",
"arch": "aarch64",
"abi": "musl"
},
{
"os": "linux",
"arch": "x86_64",
"abi": "gnu"
},
{
"os": "linux",
"arch": "aarch64",
"abi": "gnu"
},
{
"os": "macos",
"arch": "aarch64",
"abi": "none"
},
{
"os": "windows",
"arch": "x86_64",
"abi": "msvc"
}
],
"targets_basis": "Platform matrix of .github/workflows/build.yml (prepare-platform-matrix). gnu and macOS targets build with the pyroscope feature; musl and Windows targets build with no extra features.",
"features": [
"pyroscope"
],
"trusted_workflows": [
{
"repository": "rustfs/rustfs",
"workflow_id": "release-candidate.yml"
},
{
"repository": "rustfs/rustfs",
"workflow_id": "release-complete.yml"
}
],
"trusted_workflows_basis": "Receipts are produced by the reviewed main workflows and fetched by exact run ID with GitHub artifact digest verification. Candidate inputs freeze the harness commit separately from product source.",
"lanes": [
{
"lane": "identity",
"harness_repository": "rustfs/rustfs",
"harness_sha": "candidate.workflow_sha",
"harness_basis": "The exact workflow commit is frozen in candidate inputs; no mutable branch is accepted as a harness identity.",
"timeout_seconds": 1800,
"environment_capabilities": [
"process-exec"
],
"cases": [
"artifact-identity",
"startup"
],
"applies_to": {
"release_types": [
"preview",
"prerelease",
"stable"
],
"platforms": "all"
},
"applies_basis": "Artifact identity and startup checks are mandatory for every release type, preview included, on every shipped target (rustfs/backlog#2662 frozen design item 5)."
},
{
"lane": "upgrade",
"harness_repository": "rustfs/rustfs",
"harness_sha": "candidate.workflow_sha",
"harness_basis": "The exact workflow commit is frozen in candidate inputs; no mutable branch is accepted as a harness identity.",
"timeout_seconds": 5400,
"environment_capabilities": [
"process-exec",
"previous-release-artifact"
],
"cases": [
"direct-upgrade",
"mixed-version-upgrade",
"bucket-config-upgrade",
"bucket-config-rollback",
"multipart-layout-upgrade",
"multipart-layout-baseline"
],
"cases_basis": "Case IDs of the pinned previous-release matrix in .github/workflows/e2e-upgrade.yml (source.version/asset/sha256 and the case list mapped there).",
"applies_to": {
"release_types": [
"prerelease",
"stable"
],
"platforms": [
{
"os": "linux",
"arch": "x86_64",
"abi": "gnu"
}
]
},
"applies_basis": "preview releases use the pre-declared smaller set (identity lane only). The upgrade lane is only exercised on linux-x86_64-gnu, the platform of the pinned source assets."
}
],
"upgrade_sources": {
"basis": "Exactly the two previous-release sources pinned in .github/workflows/e2e-upgrade.yml are exercised today (source.version / source.asset / source.sha256). No other N-1/N-2 compatibility is claimed, and these digests are not inferred from tags.",
"sources": [
{
"version": "1.0.1",
"role": "N-1",
"artifact_name": "rustfs-linux-x86_64-gnu-v1.0.1.zip",
"artifact_sha256": "9742e183cb0dfb0d48fa909797af572fe47b7f827569036baa4156e0d3958ee5"
},
{
"version": "1.0.0-rc.5",
"role": "N-2",
"artifact_name": "rustfs-linux-x86_64-gnu-v1.0.0-rc.5.zip",
"artifact_sha256": "3ee8df71e8edcfada533be452c4135868f697bc515460ae97b027313eade7a3d"
}
],
"rollback_supported": false,
"rollback_basis": "Only the bucket-config-rollback case (rollback_to_previous_release_reads_current_bucket_metadata) is exercised. Full data-plane rollback is not certified and is never a default claim."
}
}
+37
View File
@@ -0,0 +1,37 @@
# Files that may still reference s3s (`s3s::` or `s3s_sigv4`) when Phase 1 of
# the gateway migration ends (rustfs/backlog#2734 task T0.6, rustfs/backlog#2741).
# Consumed only by `scripts/check_s3s_footprint.sh --mode allowlist`.
#
# One entry per line: a repository-relative file path or a git `:(glob)`
# pathspec. `**` matches across directories, `*` stays within one path
# component, and a directory entry matches everything under it. Blank lines
# and lines starting with `#` are ignored; an entry may not contain whitespace,
# so a note belongs on its own comment line.
#
# Every Rust file listed by `git grep -l '\bs3s\(::\|_sigv4\)' -- '*.rs'` that
# no entry matches fails the allowlist mode. Task T1.9 switches CI to that
# mode; until then `--dry-run` prints the files still outside this list, which
# is the Phase 1 tracking list.
#
# Never add crates/** other than crates/e2e_test: the engine crates leave s3s
# entirely, and the script rejects such an entry. Only the s3s-facing edge of
# the rustfs binary and the e2e oracle keep it.
# rustfs binary: the s3s-facing edge that survives Phase 1
rustfs/src/storage/ecfs.rs
rustfs/src/storage/s3_compat/**
rustfs/src/storage/access.rs
rustfs/src/auth.rs
rustfs/src/error.rs
rustfs/src/storage_api.rs
rustfs/src/app/storage_api.rs
rustfs/src/admin/router.rs
rustfs/src/app/metadata_route.rs
rustfs/src/server/http.rs
rustfs/src/server/layer.rs
rustfs/src/server/hybrid.rs
rustfs/src/protocols/client.rs
# e2e oracle: the test target that verifies S3 behaviour against s3s
crates/e2e_test/src/fake_s3_target/**
crates/e2e_test/src/**
+17
View File
@@ -0,0 +1,17 @@
# Exact baseline for the direct s3s footprint, consumed only by
# scripts/check_s3s_footprint.sh (rustfs/backlog#1677; rustfs/backlog#2734 T0.4).
#
# Each value is the exact measurement of the tree, not a ceiling. CI fails when a
# measured counter differs from its line here in either direction:
# - a larger measurement means new code widened the s3s surface: route it
# through the gateway abstractions instead; never raise a value here;
# - a smaller measurement means the PR shrank the surface: lower the matching
# value to the measured number in the same PR.
#
# files Rust files referencing s3s paths, repo-wide minus crates/e2e_test
# s3_error_lines lines holding s3_error!, same scope, minus those the script
# header's rule clears (rustfs_s3_types's macro); fails closed
# ecstore_files Rust files referencing s3s paths under crates/ecstore/src
files=65
s3_error_lines=219
ecstore_files=0
+66
View File
@@ -0,0 +1,66 @@
# Per-crate test-count baseline consumed by scripts/check_test_count_ratchet.sh
# (rustfs/backlog#2734 section 1 rule 3; T0.3 in rustfs/backlog#2738).
#
# One row per workspace member: <package>, tests, ignored, tab-separated.
# tests counts #[test], #[tokio::test], #[tokio::test(...)] and #[test_case(...)]
# attribute lines under the package directory; ignored counts #[ignore] and
# #[ignore = "..."]. The tests column is a floor and the ignored column is a
# ceiling: CI fails when a crate's tests fall below its row or its ignored count
# rises above it. Extra tests and fewer ignores pass; --update tightens the rows.
# Lowering a tests row or raising an ignored row is a reviewed statement that
# tests were intentionally removed or ignored; justify it in the PR description.
# Regenerate with:
# scripts/check_test_count_ratchet.sh --update
e2e_test 1026 20
rustfs 6440 64
rustfs-audit 70 0
rustfs-checksums 28 1
rustfs-common 36 0
rustfs-concurrency 87 0
rustfs-config 43 0
rustfs-connect-symbol-catalog 6 0
rustfs-credentials 24 0
rustfs-crypto 67 0
rustfs-data-usage 58 0
rustfs-ecstore 7858 2
rustfs-extension-schema 23 0
rustfs-filemeta 361 0
rustfs-heal 1128 3
rustfs-heal-contracts 8 0
rustfs-iam 415 0
rustfs-io-core 52 0
rustfs-io-metrics 157 0
rustfs-keystone 31 0
rustfs-kms 716 25
rustfs-license 2 0
rustfs-lifecycle 156 0
rustfs-lock 182 0
rustfs-log-analyzer 91 0
rustfs-madmin 210 0
rustfs-notify 140 0
rustfs-object-capacity 92 0
rustfs-object-data-cache 141 0
rustfs-obs 387 0
rustfs-policy 758 0
rustfs-protocols 770 1
rustfs-protos 75 0
rustfs-replication 209 0
rustfs-rio 225 1
rustfs-rio-v2 34 6
rustfs-s3-client 71 0
rustfs-s3-ops 8 0
rustfs-s3-types 105 0
rustfs-s3select-api 232 0
rustfs-s3select-query 235 2
rustfs-scanner 1392 20
rustfs-scanner-metrics 90 0
rustfs-security-governance 20 0
rustfs-signer 34 0
rustfs-storage-api 34 0
rustfs-targets 612 26
rustfs-test-utils 1 0
rustfs-tls-runtime 44 0
rustfs-trusted-proxies 72 0
rustfs-utils 253 1
rustfs-zip 30 0
zstd 11 0
+4
View File
@@ -97,6 +97,10 @@ runs:
shell: bash
run: ./scripts/check_s3s_footprint.sh
- name: Check per-crate test-count ratchet
shell: bash
run: ./scripts/check_test_count_ratchet.sh
- name: Check cryptographic capability wording
shell: bash
run: ./scripts/check_fips_wording.sh
@@ -0,0 +1,96 @@
# Copyright 2026 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
name: "Reference RustFS binary"
description: "Download the pinned reference RustFS release, verify its sha256 and expose the executable path. A missing pin, missing asset or digest mismatch fails the step."
inputs:
pin-file:
description: "Pin file naming the reference release (repository-relative)"
required: false
default: ".config/reference-binary.toml"
platform:
description: "Platform table in the pin file"
required: false
default: "linux-x86_64-gnu"
destination:
description: "Directory that receives the executable; defaults to $RUNNER_TEMP/reference-binary"
required: false
default: ""
outputs:
path:
description: "Absolute path of the verified executable"
value: ${{ steps.unpack.outputs.path }}
version:
description: "Pinned reference version"
value: ${{ steps.pin.outputs.version }}
sha256:
description: "Pinned and verified archive sha256"
value: ${{ steps.pin.outputs.sha256 }}
runs:
using: "composite"
steps:
- name: Resolve reference pin
id: pin
shell: bash
env:
PIN_TOOL: ${{ github.action_path }}/reference_binary.py
PIN_FILE: ${{ inputs.pin-file }}
PIN_PLATFORM: ${{ inputs.platform }}
run: |
set -euo pipefail
resolved="$(python3 "$PIN_TOOL" resolve --pin "$PIN_FILE" --platform "$PIN_PLATFORM")"
printf '%s\n' "$resolved"
printf '%s\n' "$resolved" >> "$GITHUB_OUTPUT"
- name: Download and verify reference archive
id: download
shell: bash
env:
GH_TOKEN: ${{ github.token }}
ASSET_DOWNLOADER: ${{ github.action_path }}/../setup/download_release_asset.py
REF_REPOSITORY: ${{ steps.pin.outputs.repository }}
REF_TAG: ${{ steps.pin.outputs.tag }}
REF_ASSET: ${{ steps.pin.outputs.asset }}
REF_SHA256: ${{ steps.pin.outputs.sha256 }}
run: |
set -euo pipefail
archive="$RUNNER_TEMP/reference-archive/$REF_ASSET"
# Fails unless the release lists exactly one asset of this name whose
# published digest equals the pin and whose downloaded bytes match it.
python3 "$ASSET_DOWNLOADER" --repository "$REF_REPOSITORY" --tag "$REF_TAG" \
--asset "$REF_ASSET" --output "$archive" --sha256 "$REF_SHA256"
echo "archive=$archive" >> "$GITHUB_OUTPUT"
- name: Unpack reference executable
id: unpack
shell: bash
env:
PIN_TOOL: ${{ github.action_path }}/reference_binary.py
REF_ARCHIVE: ${{ steps.download.outputs.archive }}
REF_SHA256: ${{ steps.pin.outputs.sha256 }}
REF_EXECUTABLE: ${{ steps.pin.outputs.executable }}
REF_DESTINATION: ${{ inputs.destination }}
run: |
set -euo pipefail
destination="${REF_DESTINATION:-$RUNNER_TEMP/reference-binary}"
unpacked="$(python3 "$PIN_TOOL" unpack --archive "$REF_ARCHIVE" --sha256 "$REF_SHA256" \
--executable "$REF_EXECUTABLE" --destination "$destination")"
path="${unpacked#path=}"
test -x "$path"
echo "Reference executable: $path"
"$path" --version
echo "path=$path" >> "$GITHUB_OUTPUT"
+144
View File
@@ -0,0 +1,144 @@
#!/usr/bin/env python3
# Copyright 2026 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Resolve and unpack the pinned reference RustFS release binary.
`resolve` reads the pin file and prints the selected platform's identity as
`key=value` lines (GitHub step-output format). `unpack` re-verifies the
downloaded archive digest against the pin and extracts exactly one executable.
Every missing or malformed input is a failure; nothing is skipped.
"""
import argparse
import hashlib
from pathlib import Path
import re
import sys
import tomllib
import zipfile
REPOSITORY = re.compile(r"[A-Za-z0-9][A-Za-z0-9_.-]*/[A-Za-z0-9][A-Za-z0-9_.-]*")
TAG = re.compile(r"[A-Za-z0-9][A-Za-z0-9_.+-]*")
PLATFORM = re.compile(r"[a-z0-9][a-z0-9_-]*")
ASSET = re.compile(r"[A-Za-z0-9][A-Za-z0-9_.+-]*\.zip")
SHA256 = re.compile(r"[0-9a-f]{64}")
EXECUTABLE = re.compile(r"[A-Za-z0-9][A-Za-z0-9_.-]*")
MAX_EXECUTABLE_BYTES = 1024 * 1024 * 1024
class PinError(Exception):
"""A pin, archive or argument that cannot be trusted."""
def field(table, key, pattern, where):
value = table.get(key)
if not isinstance(value, str) or not pattern.fullmatch(value):
raise PinError(f"{where}: missing or invalid '{key}'")
return value
def resolve(pin_path, platform):
if not PLATFORM.fullmatch(platform or ""):
raise PinError("invalid platform name")
try:
data = tomllib.loads(Path(pin_path).read_text(encoding="utf-8"))
except FileNotFoundError as exc:
raise PinError(f"{pin_path}: reference pin file is missing") from exc
except (OSError, UnicodeDecodeError, tomllib.TOMLDecodeError) as exc:
raise PinError(f"{pin_path}: cannot read reference pin: {exc}") from exc
if data.get("schema") != 1 or type(data.get("schema")) is not int:
raise PinError(f"{pin_path}: unsupported pin schema")
identity = {
"repository": field(data, "repository", REPOSITORY, pin_path),
"version": field(data, "version", TAG, pin_path),
"tag": field(data, "tag", TAG, pin_path),
}
platforms = data.get("platforms")
if not isinstance(platforms, dict) or not platforms:
raise PinError(f"{pin_path}: no pinned platforms")
entry = platforms.get(platform)
if not isinstance(entry, dict):
raise PinError(f"{pin_path}: no pin for platform '{platform}'")
where = f"{pin_path} [platforms.{platform}]"
identity["asset"] = field(entry, "asset", ASSET, where)
identity["sha256"] = field(entry, "sha256", SHA256, where)
identity["executable"] = field(entry, "executable", EXECUTABLE, where)
if identity["version"] not in identity["asset"]:
raise PinError(f"{where}: asset name does not carry version {identity['version']}")
unknown = set(entry) - {"asset", "sha256", "executable"}
if unknown:
raise PinError(f"{where}: unknown keys {sorted(unknown)}")
return identity
def file_sha256(path):
with open(path, "rb") as stream:
return hashlib.file_digest(stream, "sha256").hexdigest()
def unpack(archive, sha256, executable, destination):
if not SHA256.fullmatch(sha256 or "") or not EXECUTABLE.fullmatch(executable or ""):
raise PinError("invalid expected digest or executable name")
try:
actual = file_sha256(archive)
except OSError as exc:
raise PinError(f"{archive}: reference archive is missing or unreadable") from exc
if actual != sha256:
raise PinError(f"{archive}: sha256 {actual} does not match the pinned digest")
try:
with zipfile.ZipFile(archive) as bundle:
matches = [info for info in bundle.infolist() if info.filename == executable]
if len(matches) != 1:
raise PinError(f"{archive}: must contain exactly one top-level '{executable}'")
info = matches[0]
if info.is_dir() or not 0 < info.file_size <= MAX_EXECUTABLE_BYTES:
raise PinError(f"{archive}: '{executable}' is not a plausible executable")
destination = Path(destination)
destination.mkdir(parents=True, exist_ok=True)
target = destination / executable
with bundle.open(info) as source, open(target, "wb") as sink:
while chunk := source.read(1024 * 1024):
sink.write(chunk)
except zipfile.BadZipFile as exc:
raise PinError(f"{archive}: not a valid zip archive") from exc
target.chmod(0o755)
return target
def main(argv=None):
parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
sub = parser.add_subparsers(dest="mode", required=True)
res = sub.add_parser("resolve")
res.add_argument("--pin", required=True)
res.add_argument("--platform", required=True)
unp = sub.add_parser("unpack")
unp.add_argument("--archive", required=True)
unp.add_argument("--sha256", required=True)
unp.add_argument("--executable", required=True)
unp.add_argument("--destination", required=True)
args = parser.parse_args(argv)
try:
if args.mode == "resolve":
for key, value in resolve(args.pin, args.platform).items():
print(f"{key}={value}")
else:
print(f"path={unpack(args.archive, args.sha256, args.executable, args.destination)}")
except PinError as exc:
print(f"reference binary: {exc}", file=sys.stderr)
return 1
return 0
if __name__ == "__main__":
sys.exit(main())
+108 -21
View File
@@ -350,10 +350,10 @@
"roles": [
"periodic",
"manual",
"release"
"reusable"
],
"events": [
"push",
"workflow_call",
"schedule",
"workflow_dispatch"
],
@@ -367,7 +367,7 @@
"monitor": "workflow",
"max_age_hours": 36,
"result_owner": "build.yml",
"disposition": "retained independent entry",
"disposition": "Builds artifacts only; public release writes belong to release-publish.yml.",
"failure_owner": "workflow",
"origin": "baseline",
"alert_contract": "scheduled-validation"
@@ -745,14 +745,13 @@
},
{
"workflow": ".github/workflows/docker.yml",
"name": "Docker Images",
"name": "Build candidate images",
"roles": [
"manual",
"release"
"release",
"reusable"
],
"events": [
"workflow_run",
"workflow_dispatch"
"workflow_call"
],
"cadence": "event",
"timezone": "UTC",
@@ -761,7 +760,7 @@
"policy": "active",
"monitor": "none",
"result_owner": "docker.yml",
"disposition": "retained independent entry",
"disposition": "Builds artifacts only; public release writes belong to release-publish.yml.",
"monitor_reason": "Event-driven or operator-requested work has no periodic freshness obligation.",
"origin": "baseline"
},
@@ -870,6 +869,33 @@
"failure_owner": "watchdog",
"origin": "baseline"
},
{
"workflow": ".github/workflows/reference-parity.yml",
"name": "Reference Parity",
"roles": [
"pr",
"periodic",
"manual"
],
"events": [
"pull_request",
"schedule",
"workflow_dispatch"
],
"cadence": "nightly",
"timezone": "UTC",
"schedule": [
"37 1 * * *"
],
"owner": "@overtrue",
"policy": "active",
"monitor": "workflow",
"max_age_hours": 36,
"result_owner": "reference-parity.yml",
"disposition": "s3-tests per-node-id parity of branch candidates against the pinned reference release",
"failure_owner": "watchdog",
"origin": "added"
},
{
"workflow": ".github/workflows/functional-chain-health.yml",
"name": "Functional chain health",
@@ -954,14 +980,13 @@
},
{
"workflow": ".github/workflows/helm-package.yml",
"name": "Publish helm chart to artifacthub",
"name": "Build candidate Helm chart",
"roles": [
"manual",
"release"
"release",
"reusable"
],
"events": [
"workflow_run",
"workflow_dispatch"
"workflow_call"
],
"cadence": "event",
"timezone": "UTC",
@@ -970,7 +995,7 @@
"policy": "active",
"monitor": "none",
"result_owner": "helm-package.yml",
"disposition": "retained independent entry",
"disposition": "Builds artifacts only; public release writes belong to release-publish.yml.",
"monitor_reason": "Event-driven or operator-requested work has no periodic freshness obligation.",
"origin": "baseline"
},
@@ -1182,14 +1207,13 @@
},
{
"workflow": ".github/workflows/package.yml",
"name": "Package DEB/RPM",
"name": "Build candidate packages",
"roles": [
"manual",
"release"
"release",
"reusable"
],
"events": [
"workflow_run",
"workflow_dispatch"
"workflow_call"
],
"cadence": "event",
"timezone": "UTC",
@@ -1198,7 +1222,7 @@
"policy": "active",
"monitor": "none",
"result_owner": "package.yml",
"disposition": "retained independent entry",
"disposition": "Builds artifacts only; public release writes belong to release-publish.yml.",
"monitor_reason": "Event-driven or operator-requested work has no periodic freshness obligation.",
"origin": "baseline"
},
@@ -1727,6 +1751,69 @@
"disposition": "retained independent entry",
"monitor_reason": "Event-driven or operator-requested work has no periodic freshness obligation.",
"origin": "baseline"
},
{
"workflow": ".github/workflows/release-candidate.yml",
"name": "Release candidate (Linux)",
"roles": [
"manual",
"release"
],
"events": [
"workflow_dispatch"
],
"cadence": "event",
"timezone": "UTC",
"schedule": [],
"owner": "@overtrue",
"policy": "active",
"monitor": "none",
"result_owner": "release-candidate.yml",
"disposition": "Explicit immutable candidate stage",
"monitor_reason": "Operator-requested release stage has no periodic freshness obligation.",
"origin": "release-candidate-migration"
},
{
"workflow": ".github/workflows/release-complete.yml",
"name": "Release candidate (complete)",
"roles": [
"manual",
"release"
],
"events": [
"workflow_dispatch"
],
"cadence": "event",
"timezone": "UTC",
"schedule": [],
"owner": "@overtrue",
"policy": "active",
"monitor": "none",
"result_owner": "release-complete.yml",
"disposition": "Explicit immutable candidate stage",
"monitor_reason": "Operator-requested release stage has no periodic freshness obligation.",
"origin": "release-candidate-migration"
},
{
"workflow": ".github/workflows/release-publish.yml",
"name": "Publish accepted release",
"roles": [
"manual",
"release"
],
"events": [
"workflow_dispatch"
],
"cadence": "event",
"timezone": "UTC",
"schedule": [],
"owner": "@overtrue",
"policy": "active",
"monitor": "none",
"result_owner": "release-publish.yml",
"disposition": "Explicit immutable candidate stage",
"monitor_reason": "Operator-requested release stage has no periodic freshness obligation.",
"origin": "release-candidate-migration"
}
]
}
+11 -4
View File
@@ -5,7 +5,7 @@
## Billing facts
This repository is public. For a public repository, GitHub's standard hosted runners (`ubuntu-latest`, `windows-latest`, `macos-latest`) cost nothing and have no minute cap, whatever plan the organisation is on. GitHub bills only two things: larger hosted runners, which have custom labels and are not used here, and standard runners in private repositories.
This repository is public. For a public repository, GitHub's standard hosted runners (`ubuntu-latest`, `windows-latest`, `macos-latest`) cost nothing and have no minute cap, whatever plan the organisation is on. GitHub bills only two things: larger hosted runners, including the `macos-latest-xlarge` release lane used here, and standard runners in private repositories.
Every other label in this repository is capacity RustFS pays for. ARC pods and lab machines accrue cost while the machine exists. A Ubicloud label is metered for the minutes a job holds its VM:
@@ -13,6 +13,7 @@ Every other label in this repository is capacity RustFS pays for. ARC pods and l
|---|---|---|---|
| `ubuntu-latest` | nobody | GitHub-hosted, Ubuntu 24.04, 4 vCPU, 16 GB RAM, about 14 GB of free disk. Ships a C toolchain, `docker`, `gh`, `curl`, `wget`, `jq`, Python. Can `mount` tmpfs and loop devices; io_uring works | the default for every job |
| `windows-latest`, `macos-latest` | nobody | GitHub-hosted standard runners | Windows and macOS targets |
| `macos-latest-xlarge` | RustFS | GitHub-hosted larger Apple Silicon runner | macOS release builds, deferred until Linux acceptance for candidates |
| `sm-standard-2` | RustFS | ARC pod, Ubuntu 24.04, minimal image: no `gh`, no C toolchain; Docker only on the `dind-` label | only a job that a hosted runner cannot run (see the placement rule) |
| `sm-standard-4` | RustFS | ARC pod, Ubuntu 24.04, 14 CPUs and 28 GB (the label is not the core count), large disk | workspace-scale Cargo builds |
| `ubicloud-standard-16-arm` | RustFS | Ubicloud VM, alias of `ubicloud-standard-16-arm-ubuntu-2404`: Ubuntu 24.04 arm64, 16 vCPU, 48 GB, 300 GB disk. Reached through the `build.yml` matrix, so `actionlint.yaml` does not list it | native aarch64 release builds (`linux-aarch64-musl`, `linux-aarch64-gnu`) |
@@ -32,7 +33,7 @@ A job runs on `ubuntu-latest` unless it has one of these reasons, and the reason
Jobs that do only the following have no reason to be self-hosted: shell or Python scripts, `cargo fmt`, `cargo deny`, typos, actionlint, API calls (`gh`, dependency review, issue and stale bots, CLA checks), downloading an artifact and running a prebuilt binary, or a `cargo build -p rustfs` with its own cache key. `e2e-distributed.yml` and `e2e-upgrade.yml` build the server and the `e2e_test` harness on `ubuntu-latest` inside their budgets, so "it compiles something" alone is not a reason.
The functional chain's `prepare` and `complete-chain` jobs, the performance runner availability check, and runner hygiene use `ubuntu-latest`: they inspect APIs or artifacts, not the lab hosts. The actual functional and performance suites remain on their lab runners. Functional-chain health collection uses `ubuntu-latest` so a lab outage cannot also stop its independent monitor. Docker image packaging also uses `ubuntu-latest` for both musl and glibc variants; it downloads release binaries rather than compiling the workspace, and still sets up QEMU and Buildx explicitly.
The functional chain's `prepare` and `complete-chain` jobs, the performance runner availability check, and runner hygiene use `ubuntu-latest`: they inspect APIs or artifacts, not the lab hosts. The actual functional and performance suites remain on their lab runners. Functional-chain health collection uses `ubuntu-latest` so a lab outage cannot also stop its independent monitor. Docker image packaging also uses `ubuntu-latest` for both musl and glibc variants; it wraps verified candidate binaries rather than compiling the workspace, and still sets up QEMU and Buildx explicitly.
Moving these jobs off AKS reduces capacity demand, not necessarily the Azure bill immediately. VM costs fall only when nodes scale down or smaller capacity replaces them.
@@ -52,7 +53,7 @@ A hosted runner is also the only choice for a few things the pods cannot do: `mo
These are the patterns that have cost real money here without validating anything more:
- An `if: always()` summary or alert job on a self-hosted label inside a `workflow_run`-triggered workflow. The upstream workflow completes many times a day, the real jobs skip, and the summary still starts a pod each time. Gate it on the upstream job having run, and keep it on `ubuntu-latest`.
- Per-push workflows on `main`. A merge triggers the full `ci.yml` matrix, `cache-warm.yml` and the `build.yml` development build; with tens of merges a day that is as much self-hosted time as all pull requests together. Anything that does not verify the merged tree itself belongs on a schedule or behind a `paths` filter.
- Per-push workflows on `main`. A merge triggers the applicable `ci.yml` matrix and cache warming; `build.yml` development builds are scheduled or manual; with tens of merges a day that is as much self-hosted time as all pull requests together. Anything that does not verify the merged tree itself belongs on a schedule or behind a `paths` filter.
- `apt-get install` of tools the hosted image already ships. It is only needed on the minimal pod images.
- A throttled `CARGO_BUILD_JOBS` on a 14-CPU pod. The limit exists for link-storm reasons documented in `ci.yml`; measure before raising it, and measure before lowering it further.
- Tiny jobs on ARC pods. Each one pays roughly a minute of pod start-up before its first step; a compile-free job that takes ten seconds costs more to schedule than to run.
@@ -66,7 +67,7 @@ gh api "repos/rustfs/rustfs/actions/runs/<run_id>/jobs?per_page=100" \
--jq '.jobs[] | "\(.conclusion)\t\(.labels | join(","))\t\(.started_at)\t\(.completed_at)\t\(.name)"'
```
Only rows whose label starts with `sm-`, `dind-`, `ubicloud-`, `smoke-` or `pf-` cost anything. At the October 2026 review, a full pull-request run of `ci.yml` spent about 150 self-hosted minutes, a third of it in jobs that never compiled; treat that as the scale of what a placement mistake costs, not as a current figure.
Rows using `macos-latest-xlarge` or a label starting with `sm-`, `dind-`, `ubicloud-`, `smoke-` or `pf-` use paid capacity. At the October 2026 review, a full pull-request run of `ci.yml` spent about 150 self-hosted minutes, a third of it in jobs that never compiled; treat that as the scale of what a placement mistake costs, not as a current figure.
## Checklist for a new or moved job
@@ -76,3 +77,9 @@ Only rows whose label starts with `sm-`, `dind-`, `ubicloud-`, `smoke-` or `pf-`
- A self-hosted `runs-on` carries a comment naming its reason from the placement rule.
- Every `./.github/actions/setup` call states `cache-save-if`, and a key read from one pool is not written from the other.
- Moving a job does not change its check name, so the `main` ruleset and `docs/testing/ci-gates.md` stay as they are; renaming it does, and then that document's change checklist applies.
## Release stages
The [candidate release pipeline](../../docs/operations/release-pipeline.md) builds Linux first and defers paid desktop capacity until acceptance. Daily development builds also select Linux; use manual platform selection for desktop development artifacts. All six platforms remain required before publication. The pipeline records and promotes the exact tested bytes. Builder workflows have no public-release credentials. Keep periodic development builds, PR verification and field-test chains separate when their contracts differ; do not re-enable disabled workflows as part of a source-file migration.
Larger runners remain billed for public repositories; see [GitHub runner billing](https://docs.github.com/en/actions/concepts/runners/larger-runners#billing). Actual spending comes from organization billing, not runner labels or an estimated build duration.
+149 -8
View File
@@ -23,9 +23,16 @@ on:
- 'deny.toml'
- '.github/actions/**'
- '.github/workflows/**'
- 'scripts/release/create_or_update_release.sh'
- 'scripts/release/package_service_scripts.sh'
- 'scripts/release/package_versions.sh'
- 'scripts/release/**'
- 'scripts/security/generate_release_supply_chain_assets.py'
- '.config/release-validation-plan.json'
- '.github/scheduled-validations.json'
- 'scripts/test_helm_release_workflow.py'
- 'scripts/test_release_packages.py'
- 'scripts/test_release_images.py'
- 'Dockerfile*'
- 'entrypoint.sh'
- 'bootstrap/**'
- 'scripts/test_package_service_scripts.sh'
- 'scripts/test_package_versions.sh'
- 'scripts/test_docker_workflow.py'
@@ -41,9 +48,16 @@ on:
- 'deny.toml'
- '.github/actions/**'
- '.github/workflows/**'
- 'scripts/release/create_or_update_release.sh'
- 'scripts/release/package_service_scripts.sh'
- 'scripts/release/package_versions.sh'
- 'scripts/release/**'
- 'scripts/security/generate_release_supply_chain_assets.py'
- '.config/release-validation-plan.json'
- '.github/scheduled-validations.json'
- 'scripts/test_helm_release_workflow.py'
- 'scripts/test_release_packages.py'
- 'scripts/test_release_images.py'
- 'Dockerfile*'
- 'entrypoint.sh'
- 'bootstrap/**'
- 'scripts/test_package_service_scripts.sh'
- 'scripts/test_package_versions.sh'
- 'scripts/test_docker_workflow.py'
@@ -132,12 +146,38 @@ jobs:
if: github.repository == 'rustfs/rustfs' && (github.event_name != 'pull_request' || github.event.action != 'closed')
runs-on: ubuntu-latest
timeout-minutes: 5
permissions:
contents: read
actions: read
outputs:
release_changed: ${{ steps.scope.outputs.release_changed }}
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
fetch-depth: 0
persist-credentials: false
- name: Scope native packaging checks to release changes
id: scope
env:
BASE_SHA: ${{ github.event.pull_request.base.sha || github.event.before }}
run: |
python3 - <<'PY'
import os, re, subprocess
event, base = os.environ['GITHUB_EVENT_NAME'], os.environ.get('BASE_SHA', '')
changed = event == 'workflow_dispatch'
if event in ('push', 'pull_request'):
if not re.fullmatch(r'[0-9a-f]{40}', base) or base == '0' * 40:
changed = True
else:
paths = subprocess.check_output(['git', 'diff', '--name-only', base, os.environ['GITHUB_SHA']], text=True).splitlines()
pattern = r'^(scripts/release/|scripts/test_release_|Dockerfile|entrypoint.sh$|bootstrap/|\.dockerignore$|\.config/release-validation-plan.json$|\.github/workflows/(audit|build|docker|package|helm-package|release-[^.]+)\.yml$)'
changed = any(re.match(pattern, path) for path in paths)
with open(os.environ['GITHUB_OUTPUT'], 'a') as output:
output.write(f'release_changed={str(changed).lower()}\n')
PY
- name: Report unpinned GitHub Actions
run: ./scripts/security/check_workflow_pins.sh --enforce
@@ -150,8 +190,35 @@ jobs:
- name: Check checkouts clear their credentials
run: ./scripts/security/check_persist_credentials.sh
- name: Check preview release workflow policy
run: ./scripts/security/check_preview_release_workflow.sh
- name: Check release candidate and promotion contracts
run: python3 -m unittest discover -s scripts/release -p 'test_*.py'
- name: Prepare artifact transport rehearsal
run: |
mkdir -p "$RUNNER_TEMP/release-transport"
printf '{"purpose":"transport contract, not release evidence"}\n' > "$RUNNER_TEMP/release-transport/transport.json"
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: release-transport-contract-${{ github.run_attempt }}
path: ${{ runner.temp }}/release-transport/transport.json
if-no-files-found: error
retention-days: 1
- name: Verify actual GitHub artifact metadata and downloaded bytes
env:
GH_TOKEN: ${{ github.token }}
run: |
python3 - <<'PY'
import os, sys
from pathlib import Path
sys.path.insert(0, 'scripts/release')
import candidate as c
run = c.github(f"repos/{c.REPOSITORY}/actions/runs/{os.environ['GITHUB_RUN_ID']}")
destination = Path(os.environ['RUNNER_TEMP']) / 'release-transport-downloaded'
c.download_artifact(run, f"release-transport-contract-{os.environ['GITHUB_RUN_ATTEMPT']}", destination)
c.require(c.read_json(destination / 'transport.json') == {'purpose': 'transport contract, not release evidence'}, 'artifact round trip changed')
PY
- name: Check tier evidence workflow isolation
run: ./scripts/security/check_tier_artifact_workflow.sh
@@ -162,6 +229,80 @@ jobs:
- name: Check package service lifecycle contract
run: ./scripts/test_package_service_scripts.sh
release-package-contract:
name: Release package contract
needs: workflow-pin-report
if: needs.workflow-pin-report.outputs.release_changed == 'true'
runs-on: ubuntu-latest
timeout-minutes: 15
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Install real package managers
run: |
sudo apt-get update
sudo apt-get install -y ruby ruby-dev build-essential rpm fakeroot cpio
sudo gem install fpm --version 1.17.0 --no-document
- name: Exercise accepted and substituted payloads through both recipes
run: python3 scripts/test_release_packages.py
release-image-contract:
name: Release image contract (${{ matrix.variant }})
needs: workflow-pin-report
if: needs.workflow-pin-report.outputs.release_changed == 'true'
runs-on: ubuntu-latest
timeout-minutes: 35
strategy:
fail-fast: false
matrix:
variant: [musl, gnu]
env:
RELEASE_IMAGE_VARIANT: ${{ matrix.variant }}
NO_PROXY: 127.0.0.1,localhost,::1
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Install OCI tools
run: |
echo "RELEASE_IMAGE_OUTPUT=$RUNNER_TEMP/image-contract" >> "$GITHUB_ENV"
echo 'no_proxy=127.0.0.1,localhost,::1' >> "$GITHUB_ENV"
sudo apt-get update
sudo apt-get install -y skopeo
go version
- uses: docker/setup-qemu-action@c7c53464625b32c7a7e944ae62b3e17d2b600130 # v3
- uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f # v3
- name: Exercise real OCI packaging, startup, registry retry and conflicts
run: python3 scripts/test_release_images.py
- name: Exercise amd64 OCI archive scanning
uses: aquasecurity/trivy-action@ed142fd0673e97e23eac54620cfb913e5ce36c25
env:
TRIVY_PLATFORM: linux/amd64
with:
input: ${{ runner.temp }}/image-contract-scan/image-${{ matrix.variant }}-amd64.tar
format: sarif
output: ${{ runner.temp }}/image-contract/scan-${{ matrix.variant }}-amd64.sarif
exit-code: "0"
- name: Exercise arm64 OCI archive scanning
uses: aquasecurity/trivy-action@ed142fd0673e97e23eac54620cfb913e5ce36c25
env:
TRIVY_PLATFORM: linux/arm64
with:
input: ${{ runner.temp }}/image-contract-scan/image-${{ matrix.variant }}-arm64.tar
format: sarif
output: ${{ runner.temp }}/image-contract/scan-${{ matrix.variant }}-arm64.sarif
exit-code: "0"
- name: Verify scan receipts before recording the archive
run: |
python3 - <<'PY'
import os, sys
from pathlib import Path
sys.path.insert(0, 'scripts/release')
import deliver, test_candidate
deliver.image_record(test_candidate.inputs(), os.environ['RELEASE_IMAGE_VARIANT'], Path(os.environ['RELEASE_IMAGE_OUTPUT']))
PY
dependency-review:
name: Dependency Review
runs-on: ubuntu-latest
+192 -467
View File
@@ -12,51 +12,32 @@
# See the License for the specific language governing permissions and
# limitations under the License.
# Build and Release Workflow
#
# This workflow builds RustFS binaries and automatically triggers Docker image builds.
#
# Flow:
# 1. Build binaries for multiple platforms
# 2. Upload binaries to OSS storage
# 3. Trigger docker.yml to build and push images using the uploaded binaries
#
# Platform scope:
# - Development builds come from the nightly schedule and manual dispatch and
# cover the full platform matrix; they feed the dev download channel
# (artifacts/rustfs/dev on R2, the *-main-latest packages). Tags build the
# release matrix. Main pushes do not build: at the usual merge rate most
# per-merge builds were cancelled by the next merge before publishing
# anything, and the dev channel does not need per-merge granularity.
#
# Manual Parameters:
# - publish: Publish binaries/releases for manual runs (default: false)
# - build_docker: Advisory only; manual runs do not trigger Docker publishing
# - platforms: Comma-separated platform IDs or 'all' (default: all)
# Nightly development artifacts and reusable immutable-candidate builds.
# Public release writes live exclusively in release-publish.yml. Candidate callers
# select Linux or desktop and pin release inputs before any matrix starts.
name: Build and Release
on:
push:
tags: [ "*.*.*" ]
workflow_call:
inputs:
candidate_manifest:
description: Frozen candidate inputs; candidates never publish from this workflow
required: true
type: string
platforms:
description: linux, desktop, all, or an explicit platform list
required: true
type: string
schedule:
- cron: "13 1 * * *" # Nightly 01:13 UTC
workflow_dispatch:
inputs:
publish:
description: "Publish to download channels and GitHub Releases (false keeps only workflow artifacts)"
description: "Publish the development download channel (never a release)"
required: false
default: false
type: boolean
build_docker:
# Advisory only. docker.yml triggers on workflow_run and its job-level
# condition requires the triggering event to be a tag push, so a manual
# dispatch of this workflow never produces images regardless of this
# value. Publishing remains controlled by the explicit publish input.
description: "Build and push Docker images after binary build (ignored: dispatch runs never reach docker.yml)"
required: false
default: true
type: boolean
platforms:
description: "Comma-separated targets or 'all' (e.g. linux-x86_64-musl,macos-aarch64)"
required: false
@@ -65,10 +46,11 @@ on:
permissions:
contents: read
actions: read
concurrency:
group: ${{ github.workflow }}-${{ github.event_name == 'push' && github.ref || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'push' }}
cancel-in-progress: false
env:
CARGO_TERM_COLOR: always
@@ -84,19 +66,39 @@ jobs:
timeout-minutes: 30
outputs:
should_build: ${{ steps.check.outputs.should_build }}
should_publish: ${{ github.event_name != 'workflow_dispatch' || inputs.publish == true }}
should_publish: ${{ inputs.candidate_manifest == '' && github.ref == 'refs/heads/main' && (github.event_name == 'schedule' || inputs.publish == true) }}
build_type: ${{ steps.check.outputs.build_type }}
version: ${{ steps.check.outputs.version }}
short_sha: ${{ steps.check.outputs.short_sha }}
is_prerelease: ${{ steps.check.outputs.is_prerelease }}
candidate_id: ${{ steps.candidate.outputs.candidate_id }}
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
ref: ${{ inputs.candidate_manifest != '' && fromJson(inputs.candidate_manifest).source_sha || github.sha }}
- name: Validate frozen candidate
if: inputs.candidate_manifest != ''
id: candidate
env:
CANDIDATE_MANIFEST: ${{ inputs.candidate_manifest }}
run: |
python3 - <<'PYINPUT'
import json, os, sys
sys.path.insert(0, 'scripts/release')
import candidate
value = candidate.validate_inputs(json.loads(os.environ['CANDIDATE_MANIFEST']))
candidate.write_json('candidate-inputs.json', value)
candidate.emit_output('candidate_id', value['candidate_id'])
PYINPUT
python3 scripts/release/candidate.py validate-inputs --inputs candidate-inputs.json --check-checkout
- name: Determine build strategy
id: check
env:
CANDIDATE_MANIFEST: ${{ inputs.candidate_manifest }}
run: |
should_build=false
build_type="none"
@@ -107,8 +109,14 @@ jobs:
# Get short SHA for all builds
short_sha=$(git rev-parse --short HEAD)
# Candidates contain the final product version but have no release
# authority. Their only publication is immutable workflow artifacts.
if [[ -n "$CANDIDATE_MANIFEST" ]]; then
should_build=true
build_type="candidate"
version=$(python3 -c 'import json; print(json.load(open("candidate-inputs.json"))["version"])')
# Determine build type based on trigger
if [[ "${{ startsWith(github.ref, 'refs/tags/') }}" == "true" ]]; then
elif [[ "${{ startsWith(github.ref, 'refs/tags/') }}" == "true" ]]; then
# Tag push - preview, release, or prerelease
should_build=true
tag_name="${GITHUB_REF#refs/tags/}"
@@ -155,6 +163,46 @@ jobs:
echo " - Short SHA: $short_sha"
echo " - Is prerelease: $is_prerelease"
prepare-build-inputs:
needs: build-check
if: needs.build-check.outputs.should_build == 'true'
runs-on: ubuntu-latest
timeout-minutes: 15
outputs:
console: ${{ steps.inputs.outputs.console }}
toolchain: ${{ steps.inputs.outputs.toolchain }}
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
ref: ${{ inputs.candidate_manifest != '' && fromJson(inputs.candidate_manifest).source_sha || github.sha }}
- name: Resolve one Console and toolchain identity
id: inputs
env:
GH_TOKEN: ${{ github.token }}
CANDIDATE_MANIFEST: ${{ inputs.candidate_manifest }}
shell: bash
run: |
set -euo pipefail
if [[ -z "$CANDIDATE_MANIFEST" ]]; then
gh api repos/rustfs/console/releases/latest > console-release.json
rustup update stable --no-self-update
fi
python3 - <<'PYINPUT'
import json, os, subprocess, sys
sys.path.insert(0, 'scripts/release')
import candidate
raw = os.environ['CANDIDATE_MANIFEST']
if raw:
frozen = candidate.validate_inputs(json.loads(raw))
console, toolchain = frozen['console'], frozen['toolchain']
else:
console = candidate.console_asset(candidate.read_json('console-release.json'))
toolchain = subprocess.check_output(['rustc', '+stable', '--version'], text=True).split()[1]
candidate.emit_output('console', json.dumps(console, separators=(',', ':')))
candidate.emit_output('toolchain', toolchain)
PYINPUT
# Build RustFS binaries
prepare-platform-matrix:
name: Prepare Platform Matrix
@@ -171,7 +219,9 @@ jobs:
env:
# via env, not interpolation: a dispatch input is free-form text and
# would otherwise be pasted into the script for bash to evaluate.
RAW_PLATFORMS: ${{ github.event_name == 'workflow_dispatch' && github.event.inputs.platforms || 'all' }}
# Daily development builds cover Linux. Desktop remains available on
# manual dispatch and is required after candidate field acceptance.
RAW_PLATFORMS: ${{ inputs.platforms || 'linux' }}
run: |
set -euo pipefail
@@ -193,10 +243,14 @@ jobs:
{"target_id":"linux-aarch64-musl","os":"ubicloud-standard-16-arm","target":"aarch64-unknown-linux-musl","cross":false,"platform":"linux","rustflags":"","features":"","glibc":""},
{"target_id":"linux-x86_64-gnu","os":"sm-standard-4","target":"x86_64-unknown-linux-gnu","cross":true,"platform":"linux","rustflags":"","features":"pyroscope","glibc":"2.31"},
{"target_id":"linux-aarch64-gnu","os":"ubicloud-standard-16-arm","target":"aarch64-unknown-linux-gnu","cross":true,"platform":"linux","rustflags":"","features":"pyroscope","glibc":"2.31"},
{"target_id":"macos-aarch64","os":"macos-latest","target":"aarch64-apple-darwin","cross":false,"platform":"macos","rustflags":"","features":"pyroscope","glibc":""},
{"target_id":"macos-aarch64","os":"macos-latest-xlarge","target":"aarch64-apple-darwin","cross":false,"platform":"macos","rustflags":"","features":"pyroscope","glibc":""},
{"target_id":"windows-x86_64","os":"windows-latest","target":"x86_64-pc-windows-msvc","cross":false,"platform":"windows","rustflags":"","features":"","glibc":""}
]}'
if [[ "$selected" == "linux" || "$selected" == "desktop" ]]; then
selected=$(jq -r --arg group "$selected" '.include[] | select(if $group == "linux" then .platform == "linux" else .platform != "linux" end) | .target_id' <<<"$all" | paste -sd ',' -)
fi
if [[ "${selected}" == "all" ]]; then
matrix="$(jq -c . <<<"${all}")"
else
@@ -224,12 +278,13 @@ jobs:
build-rustfs:
# Acceptance workflows select artifacts by this stable platform name.
name: Build RustFS (${{ matrix.target_id }})
needs: [ build-check, prepare-platform-matrix ]
needs: [ build-check, prepare-platform-matrix, prepare-build-inputs ]
if: needs.build-check.outputs.should_build == 'true' && needs.prepare-platform-matrix.result == 'success'
runs-on: ${{ matrix.os }}
timeout-minutes: 180
env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
RUSTUP_TOOLCHAIN: ${{ needs.prepare-build-inputs.outputs.toolchain }}
# Release binaries ship without dial9 telemetry and therefore do not need
# --cfg tokio_unstable. Telemetry builds opt in explicitly (see
# `make build-profiling`); crates/obs/build.rs fails the compile if the
@@ -244,11 +299,33 @@ jobs:
with:
persist-credentials: false
fetch-depth: 0
ref: ${{ inputs.candidate_manifest != '' && fromJson(inputs.candidate_manifest).source_sha || github.sha }}
- name: Ensure Python for artifact verification
if: runner.os == 'Linux'
shell: bash
run: |
if ! command -v python3 >/dev/null 2>&1; then
sudo apt-get update
sudo apt-get install -y python3
fi
- name: Reuse completed candidate platform
id: reuse
if: needs.build-check.outputs.build_type == 'candidate'
env:
GH_TOKEN: ${{ github.token }}
CANDIDATE_MANIFEST: ${{ inputs.candidate_manifest }}
TARGET: ${{ matrix.target_id }}
shell: bash
run: |
python3 scripts/release/pipeline.py reuse-platform --target "$TARGET"
- name: Setup Rust environment
if: steps.reuse.outputs.reused != 'true'
uses: ./.github/actions/setup
with:
rust-version: stable
rust-version: ${{ needs.prepare-build-inputs.outputs.toolchain }}
target: ${{ matrix.target }}
# rust-cache keys include OS and architecture but not the runner image.
# The Ubicloud aarch64 legs therefore use their own suffix so they
@@ -268,128 +345,35 @@ jobs:
install-test-tools: 'false'
- name: Download static console assets
if: steps.reuse.outputs.reused != 'true'
shell: bash
env:
GITHUB_TOKEN: ${{ github.token }}
GH_TOKEN: ${{ github.token }}
CONSOLE_IDENTITY: ${{ needs.prepare-build-inputs.outputs.console }}
run: |
mkdir -p ./rustfs/static
verify_sha256() {
local expected="$1"
local file="$2"
local actual=""
local actual_line=""
if command -v sha256sum >/dev/null 2>&1; then
actual_line=$(sha256sum -- "$file") || return 1
elif command -v shasum >/dev/null 2>&1; then
actual_line=$(shasum -a 256 -- "$file") || return 1
else
echo "No SHA-256 verification tool found" >&2
return 1
fi
actual="${actual_line%%[[:space:]]*}"
if [ "$actual" = "$expected" ]; then
echo "SHA256 verified OK: $actual"
return 0
else
echo "SHA256 mismatch: expected=$expected actual=$actual" >&2
return 1
fi
}
download_console_assets() {
local curl_bin="curl"
local python_bin="python3"
local console_api="https://api.github.com/repos/rustfs/console/releases/latest"
local console_json="console-release.json"
local console_url
local console_sha256
local curl_auth_args=()
console_tag=""
if [[ "${{ matrix.platform }}" == "windows" ]]; then
curl_bin="curl.exe"
else
chmod +w ./rustfs/static/LICENSE || true
fi
if ! command -v "$python_bin" >/dev/null 2>&1; then
python_bin="python"
fi
if ! command -v "$python_bin" >/dev/null 2>&1; then
echo "No Python interpreter found for release metadata parsing" >&2
return 1
fi
if [[ -n "${GITHUB_TOKEN:-}" ]]; then
curl_auth_args=(-H "Authorization: Bearer ${GITHUB_TOKEN}" -H "X-GitHub-Api-Version: 2022-11-28")
fi
"$curl_bin" "${curl_auth_args[@]}" --fail -L "$console_api" \
-o "$console_json" --retry 3 --retry-delay 5 --max-time 300 || return 1
read -r console_tag console_url console_sha256 < <("$python_bin" - "$console_json" <<'PY'
import json
import re
import sys
with open(sys.argv[1], encoding="utf-8") as handle:
release = json.load(handle)
tag = release.get("tag_name", "") or "unknown"
for asset in release.get("assets", []):
name = asset.get("name", "")
digest = asset.get("digest", "")
url = asset.get("browser_download_url", "")
if name.startswith("rustfs-console-") and name.endswith(".zip") and digest.startswith("sha256:") and url:
sha256 = digest.split(":", 1)[1]
if not re.fullmatch(r"[0-9a-fA-F]{64}", sha256):
raise SystemExit(f"console zip asset has invalid sha256 digest: {sha256}")
sys.stdout.buffer.write(f"{tag} {url} {sha256}\n".encode("utf-8"))
break
else:
raise SystemExit("no console zip asset with sha256 digest found")
PY
) || return 1
if [[ -z "$console_url" || -z "$console_sha256" ]]; then
echo "Console release metadata is missing URL or SHA-256 digest" >&2
return 1
fi
echo "Console release: ${console_tag}"
echo "Downloading console asset: ${console_url}"
"$curl_bin" --fail -L "$console_url" -o console.zip --retry 3 --retry-delay 5 --max-time 300 || return 1
verify_sha256 "$console_sha256" console.zip || return 2
unzip -o console.zip -d ./rustfs/static || return 2
}
if download_console_assets; then
rm -f console.zip console-release.json
else
status=$?
rm -f console.zip console-release.json
if [[ "$status" -eq 2 ]]; then
echo "Console asset integrity verification failed" >&2
fi
echo "Failed to download verified console assets" >&2
exit 1
fi
if [[ ! -s ./rustfs/static/index.html ]]; then
echo "Console asset archive is missing static/index.html" >&2
exit 1
fi
asset_count=$(find ./rustfs/static -type f | wc -l | tr -d '[:space:]')
echo "Console assets ready: version=${console_tag:-unknown}, ${asset_count} files extracted to ./rustfs/static"
python3 - <<'PYCONSOLE'
import json, os, pathlib, subprocess, zipfile
value = json.loads(os.environ['CONSOLE_IDENTITY'])
subprocess.run(['python3', '.github/actions/setup/download_release_asset.py',
'--repository', 'rustfs/console', '--tag', value['tag'], '--asset', value['name'],
'--sha256', value['sha256'], '--output', 'console.zip'], check=True)
destination = pathlib.Path('rustfs/static')
destination.mkdir(parents=True, exist_ok=True)
with zipfile.ZipFile('console.zip') as archive:
for member in archive.infolist():
path = pathlib.PurePosixPath(member.filename)
if path.is_absolute() or '..' in path.parts or '\\' in member.filename:
raise SystemExit('unsafe Console archive member')
archive.extractall(destination)
if not (destination / 'index.html').is_file():
raise SystemExit('Console archive has no index.html')
PYCONSOLE
- name: Build RustFS
if: steps.reuse.outputs.reused != 'true'
shell: bash
env:
RUSTFS_BUILD_VERSION: ${{ needs.build-check.outputs.build_type == 'candidate' && needs.build-check.outputs.version || '' }}
BUILD_TARGET: ${{ matrix.target }}
BUILD_OS: ${{ matrix.platform }}-${{ startsWith(matrix.target, 'aarch64-') && 'aarch64' || 'x86_64' }}
run: |
@@ -440,7 +424,7 @@ jobs:
fi
- name: Verify macOS binary has no dynamic liblzma dependency
if: matrix.platform == 'macos'
if: steps.reuse.outputs.reused != 'true' && (matrix.platform == 'macos')
shell: bash
run: |
binary="target/${{ matrix.target }}/release/rustfs"
@@ -451,7 +435,7 @@ jobs:
"$binary" --version
- name: Generate the CPU symbol catalog
if: endsWith(matrix.target, '-linux-gnu') && matrix.features == 'pyroscope'
if: steps.reuse.outputs.reused != 'true' && (endsWith(matrix.target, '-linux-gnu') && matrix.features == 'pyroscope')
shell: bash
run: |
cargo metadata --locked --format-version 1 --filter-platform '${{ matrix.target }}' \
@@ -468,6 +452,7 @@ jobs:
"$release_dir/rustfs.cpu-symbol-catalog.json" > "$release_dir/rustfs.cpu-symbol-catalog.sha256"
- name: Create release package
if: steps.reuse.outputs.reused != 'true'
id: package
shell: bash
run: |
@@ -668,7 +653,7 @@ jobs:
echo "📊 Version: ${VERSION}"
- name: Verify packaged Linux binary
if: matrix.platform == 'linux'
if: steps.reuse.outputs.reused != 'true' && (matrix.platform == 'linux')
shell: bash
run: |
set -euo pipefail
@@ -697,7 +682,7 @@ jobs:
"$binary_dir/rustfs" --version
- name: Verify packaged console
if: matrix.target == 'x86_64-unknown-linux-gnu' || matrix.target == 'aarch64-unknown-linux-gnu'
if: steps.reuse.outputs.reused != 'true' && (matrix.target == 'x86_64-unknown-linux-gnu' || matrix.target == 'aarch64-unknown-linux-gnu')
shell: bash
run: |
set -euo pipefail
@@ -750,15 +735,49 @@ jobs:
cat "$log_file"
exit 1
- name: Verify native candidate and retain receipts
if: needs.build-check.outputs.build_type == 'candidate' && steps.reuse.outputs.reused != 'true'
env:
CANDIDATE_MANIFEST: ${{ inputs.candidate_manifest }}
TARGET: ${{ matrix.target_id }}
ARCHIVE: ${{ steps.package.outputs.package_file }}
shell: bash
run: |
python3 -c 'import os; from pathlib import Path; Path("inputs.json").write_text(os.environ["CANDIDATE_MANIFEST"])'
python3 scripts/release/verify.py native --inputs inputs.json --archive "$ARCHIVE" --target "$TARGET" --output .
- name: Record candidate artifact identity
if: steps.reuse.outputs.reused != 'true' && (needs.build-check.outputs.build_type == 'candidate')
env:
CANDIDATE_MANIFEST: ${{ inputs.candidate_manifest }}
TARGET_ID: ${{ matrix.target_id }}
PACKAGE_FILE: ${{ steps.package.outputs.package_file }}
shell: bash
run: |
python3 - <<'PYRECORD'
import json, os, sys
sys.path.insert(0, 'scripts/release')
import candidate
value = candidate.validate_inputs(json.loads(os.environ['CANDIDATE_MANIFEST']))
record = candidate.archive_record(os.environ['PACKAGE_FILE'], os.environ['TARGET_ID'], value)
candidate.write_json('record-' + os.environ['TARGET_ID'] + '.json', record)
PYRECORD
- name: Upload to GitHub artifacts
if: steps.reuse.outputs.reused != 'true'
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: ${{ steps.package.outputs.package_name }}
path: "rustfs-*.zip"
retention-days: ${{ startsWith(github.ref, 'refs/tags/') && 30 || 7 }}
name: ${{ needs.build-check.outputs.build_type == 'candidate' && format('candidate-{0}-{1}', needs.build-check.outputs.candidate_id, matrix.target_id) || steps.package.outputs.package_name }}
path: |
rustfs-*.zip
record-${{ matrix.target_id }}.json
identity-${{ matrix.target_id }}*
startup-${{ matrix.target_id }}.log
if-no-files-found: error
retention-days: ${{ (startsWith(github.ref, 'refs/tags/') || needs.build-check.outputs.build_type == 'candidate') && 30 || 7 }}
- name: Upload to Cloudflare R2
if: needs.build-check.outputs.should_publish == 'true' && env.R2_ACCESS_KEY_ID != '' && (needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease' || needs.build-check.outputs.build_type == 'development')
if: steps.reuse.outputs.reused != 'true' && (needs.build-check.outputs.should_publish == 'true' && env.R2_ACCESS_KEY_ID != '' && (needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease' || needs.build-check.outputs.build_type == 'development'))
env:
R2_ACCESS_KEY_ID: ${{ secrets.R2_ACCESS_KEY_ID }}
R2_SECRET_ACCESS_KEY: ${{ secrets.R2_SECRET_ACCESS_KEY }}
@@ -814,13 +833,18 @@ jobs:
export AWS_SECRET_ACCESS_KEY="$R2_SECRET_ACCESS_KEY"
export AWS_DEFAULT_REGION="auto"
# Determine upload path based on build type
# Determine upload path based on build type. Release and prerelease
# builds only stage their bytes here: the official
# artifacts/rustfs/release/ path is written solely by the
# certification-gated promote-release-artifacts job, from these same
# staged objects. Development builds keep the uncertified dev channel
# and never touch official or latest channels.
if [[ "$BUILD_TYPE" == "development" ]]; then
R2_PATH="s3://${R2_BUCKET}/artifacts/rustfs/dev/"
echo "📤 Uploading development build to R2 dev directory"
else
R2_PATH="s3://${R2_BUCKET}/artifacts/rustfs/release/"
echo "📤 Uploading release build to R2 release directory"
R2_PATH="s3://${R2_BUCKET}/artifacts/rustfs/staging/${GITHUB_RUN_ID}/${GITHUB_RUN_ATTEMPT}/"
echo "📤 Staging release build for certification-gated promotion"
fi
# Upload all rustfs zip files to R2 using S3-compatible API
@@ -835,6 +859,7 @@ jobs:
echo "✅ Upload completed successfully"
# Build summary
build-summary:
name: Build Summary
@@ -869,309 +894,9 @@ jobs:
echo "This run does not trigger Docker publishing."
fi
# Create GitHub Release for every valid release tag, including previews
create-release:
name: Create GitHub Release
needs: [ build-check, build-rustfs ]
if: needs.build-check.outputs.should_publish == 'true' && startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'preview' || needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
runs-on: ubuntu-latest # gh CLI required; the sm-standard-2 fleet does not ship it
timeout-minutes: 30
permissions:
contents: write
outputs:
release_id: ${{ steps.create.outputs.release_id }}
release_url: ${{ steps.create.outputs.release_url }}
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
fetch-depth: 0
- name: Create GitHub Release
id: create
env:
GH_TOKEN: ${{ github.token }}
shell: bash
run: |
TAG="${{ needs.build-check.outputs.version }}"
VERSION="${{ needs.build-check.outputs.version }}"
IS_PRERELEASE="${{ needs.build-check.outputs.is_prerelease }}"
BUILD_TYPE="${{ needs.build-check.outputs.build_type }}"
TARGET_COMMITISH=$(git rev-parse --verify "refs/tags/${TAG}^{commit}")
# Determine release type for title
if [[ "$BUILD_TYPE" == "preview" ]]; then
RELEASE_TYPE="preview"
elif [[ "$BUILD_TYPE" == "prerelease" ]]; then
if [[ "$TAG" == *"alpha"* ]]; then
RELEASE_TYPE="alpha"
elif [[ "$TAG" == *"beta"* ]]; then
RELEASE_TYPE="beta"
elif [[ "$TAG" == *"rc"* ]]; then
RELEASE_TYPE="rc"
else
RELEASE_TYPE="prerelease"
fi
else
RELEASE_TYPE="release"
fi
# Create release title
if [[ "$IS_PRERELEASE" == "true" ]]; then
TITLE="RustFS $VERSION (${RELEASE_TYPE})"
else
TITLE="RustFS $VERSION"
fi
./scripts/release/create_or_update_release.sh \
"$TAG" \
"$TARGET_COMMITISH" \
"$TITLE" \
"$IS_PRERELEASE"
# Prepare and upload release assets
upload-release-assets:
name: Upload Release Assets
needs: [ build-check, build-rustfs, create-release ]
if: needs.build-check.outputs.should_publish == 'true' && startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'preview' || needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
runs-on: ubuntu-latest # gh CLI required; the sm-standard-2 fleet does not ship it
timeout-minutes: 30
permissions:
contents: write
actions: read
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Download all build artifacts
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
path: ./artifacts
pattern: rustfs-*
merge-multiple: true
- name: Prepare release assets
id: prepare
shell: bash
run: |
VERSION="${{ needs.build-check.outputs.version }}"
TAG="${{ needs.build-check.outputs.version }}"
mkdir -p ./release-assets
# Copy and verify artifacts (including latest files created during build)
ASSETS_COUNT=0
for file in ./artifacts/*.zip; do
if [[ -f "$file" ]]; then
cp "$file" ./release-assets/
ASSETS_COUNT=$((ASSETS_COUNT + 1))
fi
done
if [[ $ASSETS_COUNT -eq 0 ]]; then
echo "❌ No artifacts found!"
exit 1
fi
cd ./release-assets
# Generate checksums for all files (including latest versions)
if compgen -G "*.zip" >/dev/null; then
sha256sum -- *.zip > SHA256SUMS
sha512sum -- *.zip > SHA512SUMS
fi
cd ..
python3 scripts/security/generate_release_supply_chain_assets.py \
--asset-dir ./release-assets \
--version "$VERSION" \
--tag "$TAG" \
--build-type "${{ needs.build-check.outputs.build_type }}" \
--repository "${{ github.repository }}" \
--ref "${{ github.ref }}" \
--sha "${{ github.sha }}" \
--run-id "${{ github.run_id }}" \
--run-attempt "${{ github.run_attempt }}"
cd ./release-assets
echo "📦 Prepared assets:"
ls -la
echo "🔢 Total asset count: $ASSETS_COUNT"
- name: Upload to GitHub Release
env:
GH_TOKEN: ${{ github.token }}
shell: bash
run: |
TAG="${{ needs.build-check.outputs.version }}"
cd ./release-assets
# Upload all files
for file in *; do
if [[ -f "$file" ]]; then
echo "📤 Uploading $file..."
gh release upload "$TAG" "$file" --clobber
fi
done
echo "✅ All assets uploaded successfully"
# Update latest.json for every release tag (stable and prerelease): the
# project currently ships prerelease tags only, so gating this to stable
# left the version pointer permanently stale. release_type records whether
# the pointed-to version is a prerelease.
update-latest-version:
name: Update Latest Version
needs: [ build-check, publish-release ]
if: needs.build-check.outputs.should_publish == 'true' && startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
runs-on: ubuntu-latest
timeout-minutes: 30
steps:
- name: Update latest.json
env:
OSS_ACCESS_KEY_ID: ${{ secrets.ALICLOUDOSS_KEY_ID }}
OSS_ACCESS_KEY_SECRET: ${{ secrets.ALICLOUDOSS_KEY_SECRET }}
OSS_REGION: cn-beijing
OSS_ENDPOINT: https://oss-cn-beijing.aliyuncs.com
shell: bash
run: |
if [[ -z "$OSS_ACCESS_KEY_ID" ]]; then
echo "⚠️ OSS credentials not available, skipping latest.json update"
exit 0
fi
VERSION="${{ needs.build-check.outputs.version }}"
TAG="${{ needs.build-check.outputs.version }}"
if [[ "${{ needs.build-check.outputs.build_type }}" == "prerelease" ]]; then
RELEASE_TYPE="prerelease"
else
RELEASE_TYPE="stable"
fi
# Install ossutil
OSSUTIL_VERSION="2.1.1"
OSSUTIL_ZIP="ossutil-${OSSUTIL_VERSION}-linux-amd64.zip"
OSSUTIL_DIR="ossutil-${OSSUTIL_VERSION}-linux-amd64"
OSSUTIL_SHA256="60f3a808cbbdfd4b5269b8f967c6a66f564c9dacff52d93a5d21a588976ab5a2"
curl --fail -L -o "$OSSUTIL_ZIP" "https://gosspublic.alicdn.com/ossutil/v2/${OSSUTIL_VERSION}/${OSSUTIL_ZIP}"
printf '%s %s\n' "$OSSUTIL_SHA256" "$OSSUTIL_ZIP" | sha256sum -c -
unzip "$OSSUTIL_ZIP"
OSSUTIL_BIN="${RUNNER_TEMP}/ossutil"
mv "${OSSUTIL_DIR}/ossutil" "$OSSUTIL_BIN"
rm -rf "$OSSUTIL_DIR" "$OSSUTIL_ZIP"
chmod +x "$OSSUTIL_BIN"
# Create latest.json
cat > latest.json << EOF
{
"version": "${VERSION}",
"tag": "${TAG}",
"release_date": "$(date -u +%Y-%m-%dT%H:%M:%SZ)",
"release_type": "${RELEASE_TYPE}",
"download_url": "https://github.com/${{ github.repository }}/releases/tag/${TAG}"
}
EOF
# Upload to OSS
"$OSSUTIL_BIN" cp latest.json oss://rustfs-version/latest.json --force
echo "✅ Updated latest.json for ${RELEASE_TYPE} release $VERSION"
# Publish release (remove draft status)
publish-release:
name: Publish Release
needs: [ build-check, create-release, upload-release-assets ]
if: needs.build-check.outputs.should_publish == 'true' && startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'preview' || needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
runs-on: ubuntu-latest # gh CLI required; the sm-standard-2 fleet does not ship it
timeout-minutes: 30
permissions:
contents: write
steps:
- name: Publish release
env:
GH_TOKEN: ${{ github.token }}
shell: bash
run: |
TAG="${{ needs.build-check.outputs.version }}"
BUILD_TYPE="${{ needs.build-check.outputs.build_type }}"
RELEASE_ID="${{ needs.create-release.outputs.release_id }}"
# Publish the release and correct its channel state on retries.
# Only a stable final release may become GitHub Latest.
if [[ "$BUILD_TYPE" == "release" ]]; then
gh api --method PATCH "repos/${GITHUB_REPOSITORY}/releases/${RELEASE_ID}" \
-F draft=false \
-F prerelease=false \
-f make_latest=true >/dev/null
else
gh api --method PATCH "repos/${GITHUB_REPOSITORY}/releases/${RELEASE_ID}" \
-F draft=false \
-F prerelease=true \
-f make_latest=false >/dev/null
fi
echo "🎉 Released $TAG successfully!"
echo "📄 Release URL: ${{ needs.create-release.outputs.release_url }}"
# Remove the internal preview releases once the deliverable release is live.
# Only the Releases are deleted; the -preview.N tags stay so the validated
# commit remains traceable.
cleanup-preview-releases:
name: Cleanup Preview Releases
needs: [ build-check, publish-release ]
if: needs.build-check.outputs.should_publish == 'true' && startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
runs-on: ubuntu-latest # gh CLI required; the sm-standard-2 fleet does not ship it
timeout-minutes: 30
permissions:
contents: write
steps:
- name: Delete preview releases for this target
env:
GH_TOKEN: ${{ github.token }}
shell: bash
run: |
set -euo pipefail
TAG="${{ needs.build-check.outputs.version }}"
RELEASES_JSON="${RUNNER_TEMP}/releases.json"
# Fetch before filtering: a failed listing must abort here instead of
# looking like "nothing to clean up".
gh api --paginate "repos/${GITHUB_REPOSITORY}/releases?per_page=100" > "$RELEASES_JSON"
# Match only <target>-preview.<digits>. String operations, not a
# regex over the tag, so dots in the version cannot widen the match.
DELETED=0
while IFS= read -r preview_tag; do
[[ -n "$preview_tag" ]] || continue
echo "🧹 Deleting preview release $preview_tag (tag kept)"
gh release delete "$preview_tag" --repo "${GITHUB_REPOSITORY}" --yes
DELETED=$((DELETED + 1))
done < <(
jq -r --arg tag "$TAG" '
.[]
| select(.tag_name | startswith($tag + "-preview."))
| select(.tag_name | ltrimstr($tag + "-preview.") | test("^[0-9]+$"))
| .tag_name
' "$RELEASES_JSON"
)
if [[ "$DELETED" -eq 0 ]]; then
echo "ℹ️ No preview releases to clean up for $TAG"
else
echo "✅ Removed $DELETED preview release(s) for $TAG"
fi
alert-on-failure:
name: Alert on scheduled failure
needs: [build-check, prepare-platform-matrix, build-rustfs, build-summary]
needs: [build-check, prepare-build-inputs, prepare-platform-matrix, build-rustfs, build-summary]
if: >-
always() && github.event_name == 'schedule' &&
(contains(needs.*.result, 'failure') || contains(needs.*.result, 'cancelled'))
+113 -5
View File
@@ -16,7 +16,9 @@ name: Continuous Integration
on:
push:
branches: [ main ]
# gateway/integration receives the daily merge of main as a direct push
# (rustfs/backlog#2734), so its pushes must prove the result like main's.
branches: [ main, gateway/integration ]
paths-ignore:
- "**.md"
- "docs/**"
@@ -965,6 +967,8 @@ jobs:
# Retain the producer image and fixture paths until the runtime pilot passes.
runs-on: sm-standard-4
timeout-minutes: 30
env:
RUSTFS_S3_STACK: legacy
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
@@ -1085,18 +1089,122 @@ jobs:
path: ${{ runner.temp }}/rustfs-e2e-*/rustfs.log
retention-days: 3
# The same e2e-smoke suite against the same debug binary with the gateway
# S3 stack selected (rustfs/backlog#2734). Report-only: it is outside the
# Test and Lint aggregate and becomes required at T2.10. The test-binary
# download, identity, membership, run and completeness steps must stay
# identical to e2e-tests; scripts/ci_gate.py enforces that.
e2e-tests-gateway:
name: e2e-smoke (gateway, report-only until T2.10)
needs: [ build-rustfs-debug-binary ]
runs-on: sm-standard-2
timeout-minutes: 30
continue-on-error: true
env:
RUSTFS_S3_STACK: gateway
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Install verification runtime
uses: ./.github/actions/test-runtime
- name: Download E2E test binaries
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: e2e-test-binaries
path: ${{ runner.temp }}
- name: Verify E2E archive identity
run: python3 scripts/ci_test_artifact.py verify --configuration e2e --archive "$RUNNER_TEMP/e2e-tests.tar.zst"
- name: Set up Python
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.12"
- name: Install awscurl
run: |
python3 -m pip install --user --upgrade pip "awscurl==0.44"
echo "AWSCURL_PATH=$HOME/.local/bin/awscurl" >> "$GITHUB_ENV"
- name: Verify awscurl
run: test -x "$AWSCURL_PATH"
# Download after the cache restore so the freshly built binary from the
# build job always wins over anything restored into target/debug.
- name: Download debug binary
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: rustfs-debug-binary
path: target/debug
- name: Make binary executable
run: chmod +x ./target/debug/rustfs
# Build the e2e test graph once. The archive is reused by the smoke
# selection guard, security exact-count check, and run below, avoiding a
# second compile of the same e2e_test target on cold runners (backlog#1645).
- name: Verify e2e smoke test membership
env:
NEXTEST_ARCHIVE: ${{ runner.temp }}/e2e-tests.tar.zst
NEXTEST_LISTING: ${{ runner.temp }}/rustfs-e2e-smoke-list.json
run: |
cargo nextest list --profile e2e-smoke --archive-file "${NEXTEST_ARCHIVE}" --workspace-remap "$GITHUB_WORKSPACE" --extract-to "$GITHUB_WORKSPACE" --extract-overwrite --message-format json > "${NEXTEST_LISTING}"
python3 ./scripts/check_test_wiring.py --check-profile e2e-smoke "${NEXTEST_LISTING}"
./scripts/check_security_smoke_count.sh check "${NEXTEST_LISTING}"
# PR smoke subset of the in-repo e2e suite (backlog#1149 ci-4). The
# profile.e2e-smoke default-filter in .config/nextest.toml is the single
# wiring mechanism for e2e tests in CI — extend that filter instead of
# adding new e2e jobs here. Each test spawns its own rustfs server on a
# random port and reuses the downloaded debug binary above.
- name: Run e2e smoke suite
env:
NEXTEST_ARCHIVE: ${{ runner.temp }}/e2e-tests.tar.zst
RUSTFS_E2E_LOG_DIR: ${{ runner.temp }}/rustfs-e2e-smoke-logs
SAMPLER_INTERVAL_SECS: "15"
run: |
./scripts/ci/resource_sampler.sh start e2e-smoke
trap './scripts/ci/resource_sampler.sh stop' EXIT
python3 scripts/e2e_binary.py run --features e2e-test-hooks -- cargo nextest run --profile e2e-smoke --cargo-metadata target/nextest/cargo-metadata.json --binaries-metadata target/nextest/binaries-metadata.json --workspace-remap "$GITHUB_WORKSPACE" --target-dir-remap "$GITHUB_WORKSPACE/target" \
--status-level all --final-status-level all --failure-output final
- name: Verify e2e smoke execution completeness
run: python3 scripts/ci_test_artifact.py evidence "$RUNNER_TEMP/rustfs-e2e-smoke-list.json" target/nextest/e2e-smoke/junit.xml
- name: Upload e2e smoke diagnostics
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: e2e-smoke-gateway-diagnostics-${{ github.run_number }}-${{ github.run_attempt }}
path: |
${{ runner.temp }}/rustfs-e2e-smoke-logs/
${{ runner.temp }}/rustfs-e2e-smoke-list.json
if-no-files-found: warn
- name: Upload e2e smoke JUnit report
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: e2e-smoke-gateway-junit-${{ github.run_number }}-${{ github.run_attempt }}
path: target/nextest/e2e-smoke/junit.xml
if-no-files-found: warn
e2e-full:
name: End-to-End Tests (full merge gate)
# Merge gate only (backlog#1149 ci-5): the never-automated user-visible
# suites — KMS, object_lock, multipart_auth, quota, checksum, encryption,
# security-boundary, ... — via the e2e-full nextest profile. Too heavy for
# every PR, so it is gated to main pushes, the merge queue, and manual
# dispatch. protocols / the 7 cluster suites / replication / #[ignore] are
# owned by other lanes (see .config/nextest.toml profile.e2e-full).
# every PR, so it is gated to main and gateway/integration pushes, the merge
# queue, and manual dispatch. protocols / the 7 cluster suites / replication
# / #[ignore] are owned by other lanes (see .config/nextest.toml profile.e2e-full).
if: >-
github.event_name == 'workflow_dispatch' ||
github.event_name == 'merge_group' ||
(github.event_name == 'push' && github.ref == 'refs/heads/main')
(github.event_name == 'push' && (github.ref == 'refs/heads/main' || github.ref == 'refs/heads/gateway/integration'))
needs: [ build-rustfs-debug-binary ]
# Retain the producer image and fixture paths until the runtime pilot passes.
runs-on: sm-standard-4
+71 -571
View File
@@ -1,599 +1,99 @@
# Copyright 2024 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Docker Images Workflow
#
# This workflow builds Docker images using pre-built binaries from the build workflow.
#
# Trigger Types:
# 1. workflow_run: Automatically triggered when "Build and Release" workflow completes
# 2. workflow_dispatch: Manual trigger for standalone Docker builds
#
# Key Features:
# - Only triggers when Linux builds (x86_64 + aarch64) are successful
# - Independent of macOS/Windows build status
# - Uses workflow_run event for precise control
# - Only builds Docker images for releases and prereleases (development builds are skipped)
name: Docker Images
# Permissions needed for workflow_run event and Docker registry access
# Copyright 2026 RustFS Team
# SPDX-License-Identifier: Apache-2.0
name: Build candidate images
on:
workflow_call:
inputs:
candidate_manifest:
type: string
required: true
permissions:
contents: read
packages: write
on:
# Automatically triggered when build workflow completes
workflow_run:
workflows: [ "Build and Release" ]
types: [ completed ]
# Manual trigger with same parameters for consistency
workflow_dispatch:
inputs:
push_images:
description: "Push images to registries"
required: false
default: true
type: boolean
version:
description: "Version to build (latest, v1.0.0, v1.0.0-alpha1, or v1.0.1-preview.1)"
required: false
default: "latest"
type: string
force_rebuild:
description: "Force rebuild even if binary exists (useful for testing)"
required: false
default: false
type: boolean
env:
CONCLUSION: ${{ github.event.workflow_run.conclusion }}
HEAD_BRANCH: ${{ github.event.workflow_run.head_branch }}
HEAD_SHA: ${{ github.event.workflow_run.head_sha }}
TRIGGERING_EVENT: ${{ github.event.workflow_run.event }}
DOCKERHUB_USERNAME: rustfs
CARGO_TERM_COLOR: always
REGISTRY_DOCKERHUB: rustfs/rustfs
REGISTRY_GHCR: ghcr.io/${{ github.repository }}
REGISTRY_QUAY: quay.io/rustfs/rustfs
DOCKER_PLATFORMS: linux/amd64,linux/arm64
actions: read
jobs:
# Check if we should build Docker images.
# Short-circuit at job level so per-merge development builds (workflow_run
# from a push to main) skip the whole run without spawning a checkout job:
# images are only published for tag builds — build.yml push triggers cover
# only main and tags, so a non-main head_branch is a tag name — and for
# manual dispatch.
build-check:
name: Docker Build Check
if: >-
github.event_name == 'workflow_dispatch' ||
(github.event.workflow_run.conclusion == 'success' &&
github.event.workflow_run.event == 'push' &&
github.event.workflow_run.head_branch != 'main')
runs-on: ubuntu-latest
timeout-minutes: 30
outputs:
should_build: ${{ steps.check.outputs.should_build }}
should_push: ${{ steps.check.outputs.should_push }}
build_type: ${{ steps.check.outputs.build_type }}
version: ${{ steps.check.outputs.version }}
short_sha: ${{ steps.check.outputs.short_sha }}
is_prerelease: ${{ steps.check.outputs.is_prerelease }}
create_latest: ${{ steps.check.outputs.create_latest }}
source_ref: ${{ steps.check.outputs.source_ref }}
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
# For workflow_run events, checkout the specific commit that triggered the workflow
ref: ${{ github.event.workflow_run.head_sha || github.sha }}
- name: Check build conditions
id: check
env:
# dispatch inputs via env, not `${{ }}` interpolation: they are
# free-form strings and would otherwise be evaluated by bash.
INPUT_VERSION: ${{ github.event.inputs.version }}
INPUT_PUSH_IMAGES: ${{ github.event.inputs.push_images }}
INPUT_FORCE_REBUILD: ${{ github.event.inputs.force_rebuild }}
run: |
should_build=false
should_push=false
build_type="none"
version=""
short_sha=""
is_prerelease=false
create_latest=false
source_ref="$GITHUB_SHA"
# Pre-GA policy: until the first stable (vX.Y.Z) tag exists, every
# prerelease (alpha/beta/rc) also moves `latest`, so users pulling
# `latest` get the newest test build. Once a stable tag is published
# this returns false and `latest` follows stable releases only.
prerelease_moves_latest() {
local stable_tags
stable_tags=$(git ls-remote --tags --refs origin 2>/dev/null \
| awk '{print $2}' \
| grep -E '^refs/tags/v?[0-9]+\.[0-9]+\.[0-9]+$' || true)
if [[ -z "$stable_tags" ]]; then
return 0
fi
echo "ℹ️ Stable release tag(s) already exist; prereleases no longer update latest"
return 1
}
if [[ "${{ github.event_name }}" == "workflow_run" ]]; then
# Triggered by build workflow completion
echo "🔗 Triggered by build workflow completion"
# Check if the triggering workflow was successful
# If the workflow succeeded, it means ALL builds (including Linux x86_64 and aarch64) succeeded
if [[ "$CONCLUSION" == "success" ]]; then
echo "✅ Build workflow succeeded, all builds including Linux are successful"
should_build=true
should_push=true
else
echo "❌ Build workflow failed (conclusion: $CONCLUSION), skipping Docker build"
should_build=false
fi
# Extract version info from commit message or use commit SHA
# Use Git to generate consistent short SHA (ensures uniqueness like build.yml)
short_sha=$(git rev-parse --short "$HEAD_SHA")
source_ref="$HEAD_SHA"
# Determine build type based on triggering workflow event and ref
triggering_event="$TRIGGERING_EVENT"
head_branch="$HEAD_BRANCH"
echo "🔍 Analyzing triggering workflow:"
echo " 📋 Event: $triggering_event"
echo " 🌿 Head branch: $head_branch"
echo " 📎 Head SHA: $HEAD_SHA"
# Check if this was triggered by a tag push
if [[ "$triggering_event" == "push" ]]; then
# For tag pushes, head_branch will be like "refs/tags/v1.0.0" or just "v1.0.0"
if [[ "$head_branch" == refs/tags/* ]]; then
# Extract tag name from refs/tags/TAG_NAME
tag_name="${head_branch#refs/tags/}"
version="$tag_name"
elif [[ "$head_branch" =~ ^v?[0-9]+\.[0-9]+\.[0-9]+ ]]; then
# Direct tag name like "v1.0.0" or "1.0.0-alpha.1"
version="$head_branch"
elif [[ "$head_branch" == "main" ]]; then
# Regular branch push to main
build_type="development"
version="dev-${short_sha}"
should_build=false
echo "⏭️ Skipping Docker build for development version (main branch push)"
else
# Other branch push
build_type="development"
version="dev-${short_sha}"
should_build=false
echo "⏭️ Skipping Docker build for development version (branch: $head_branch)"
fi
# If we extracted a version (tag), determine release type
if [[ -n "$version" ]] && [[ "$version" != "dev-${short_sha}" ]]; then
# Remove 'v' prefix if present for consistent version format
if [[ "$version" == v* ]]; then
version="${version#v}"
fi
if [[ "$version" =~ -preview\.[0-9]+$ ]]; then
build_type="preview"
is_prerelease=true
echo "🔍 Building Docker image for preview: $version (version tags only)"
elif [[ "$version" == *"-preview"* ]]; then
echo "❌ Invalid preview tag: $version (expected suffix: -preview.<number>)" >&2
exit 1
elif [[ "$version" == *"alpha"* ]] || [[ "$version" == *"beta"* ]] || [[ "$version" == *"rc"* ]]; then
build_type="prerelease"
is_prerelease=true
# Pre-GA policy: prereleases update latest until the first stable tag exists.
if prerelease_moves_latest; then
create_latest=true
echo "🧪 Building Docker image for prerelease: $version (creating latest tag)"
else
echo "🧪 Building Docker image for prerelease: $version"
fi
else
build_type="release"
create_latest=true
echo "🚀 Building Docker image for release: $version"
fi
fi
else
# Non-push events
build_type="development"
version="dev-${short_sha}"
should_build=false
echo "⏭️ Skipping Docker build for development version (event: $triggering_event)"
fi
echo "🔄 Build triggered by workflow_run:"
echo " 📋 Conclusion: $CONCLUSION"
echo " 🌿 Branch: $HEAD_BRANCH"
echo " 📎 SHA: $HEAD_SHA"
echo " 🎯 Event: $TRIGGERING_EVENT"
elif [[ "${{ github.event_name }}" == "workflow_dispatch" ]]; then
# Manual trigger
input_version="$INPUT_VERSION"
version="${input_version}"
should_push="$INPUT_PUSH_IMAGES"
should_build=true
# Get short SHA
short_sha=$(git rev-parse --short HEAD)
echo "🎯 Manual Docker build triggered:"
echo " 📋 Requested version: $input_version"
echo " 🔧 Force rebuild: $INPUT_FORCE_REBUILD"
echo " 🚀 Push images: $should_push"
case "$input_version" in
"latest")
build_type="release"
create_latest=true
echo "🚀 Building with latest stable release version"
;;
*-preview*)
if [[ ! "$input_version" =~ ^v?[0-9]+\.[0-9]+\.[0-9]+(-[0-9A-Za-z.-]+)?-preview\.[0-9]+$ ]]; then
echo "❌ Invalid preview version: $input_version" >&2
exit 1
fi
build_type="preview"
is_prerelease=true
version="${input_version#v}"
echo "🔍 Building with preview version: $version (version tags only)"
;;
# Prerelease versions (must match first, more specific)
v*alpha*|v*beta*|v*rc*|*alpha*|*beta*|*rc*)
build_type="prerelease"
is_prerelease=true
# Pre-GA policy: prereleases update latest until the first stable tag exists.
if prerelease_moves_latest; then
create_latest=true
echo "🧪 Building with prerelease version: $input_version (creating latest tag)"
else
echo "🧪 Building with prerelease version: $input_version"
fi
;;
# Release versions (match after prereleases, more general)
v[0-9]*|[0-9]*.*.*)
build_type="release"
create_latest=true
echo "📦 Building with specific release version: $input_version"
;;
*)
# Invalid version for Docker build
should_build=false
echo "❌ Invalid version for Docker build: $input_version"
echo "⚠️ Only release versions (latest, v1.0.0, 1.0.0) and prereleases (v1.0.0-alpha1, 1.0.0-beta2) are supported"
;;
esac
if [[ "$should_build" == true && "$input_version" != "latest" ]]; then
tag_ref="refs/tags/$input_version"
if ! git ls-remote --exit-code origin "$tag_ref" >/dev/null 2>&1; then
if [[ "$input_version" == v* ]]; then
tag_ref="refs/tags/${input_version#v}"
else
tag_ref="refs/tags/v$input_version"
fi
fi
if ! git ls-remote --exit-code origin "$tag_ref" >/dev/null 2>&1; then
echo "❌ Release tag not found for Docker build: $input_version"
exit 1
fi
source_ref="$tag_ref"
fi
fi
{
echo "should_build=$should_build"
echo "should_push=$should_push"
echo "build_type=$build_type"
echo "version=$version"
echo "short_sha=$short_sha"
echo "is_prerelease=$is_prerelease"
echo "create_latest=$create_latest"
echo "source_ref=$source_ref"
} >> "$GITHUB_OUTPUT"
echo "🐳 Docker Build Summary:"
echo " - Should build: $should_build"
echo " - Should push: $should_push"
echo " - Build type: $build_type"
echo " - Version: $version"
echo " - Short SHA: $short_sha"
echo " - Is prerelease: $is_prerelease"
echo " - Create latest: $create_latest"
echo " - Source ref: $source_ref"
# Build multi-arch Docker images
# Strategy: Build images using pre-built binaries from dl.rustfs.com
# Supports both release and dev channel binaries based on build context
# Only runs when should_build is true (which includes workflow success check)
build-docker:
name: Build Docker Images
needs: build-check
if: needs.build-check.outputs.should_build == 'true'
runs-on: ubuntu-latest
timeout-minutes: 60
timeout-minutes: 90
strategy:
fail-fast: false
matrix:
include:
- variant: musl
file: Dockerfile
suffix: ""
- variant: glibc
- variant: gnu
file: Dockerfile.glibc
suffix: "-glibc"
env:
CANDIDATE_MANIFEST: ${{ inputs.candidate_manifest }}
VARIANT: ${{ matrix.variant }}
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
ref: ${{ fromJson(inputs.candidate_manifest).source_sha }}
persist-credentials: false
ref: ${{ needs.build-check.outputs.source_ref }}
- name: Login to Docker Hub
uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
with:
username: ${{ env.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
- name: Login to GitHub Container Registry
uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
with:
registry: ghcr.io
username: ${{ secrets.GHCR_USERNAME }}
password: ${{ secrets.GHCR_PASSWORD }}
- name: Login to Quay.io
uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
with:
registry: quay.io
username: ${{ secrets.QUAY_USERNAME }}
password: ${{ secrets.QUAY_PASSWORD }}
- name: Set up QEMU
uses: docker/setup-qemu-action@c7c53464625b32c7a7e944ae62b3e17d2b600130 # v3
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f # v3
- name: Extract metadata and generate tags
id: meta
- name: Reuse completed image
id: reuse
env:
GH_TOKEN: ${{ github.token }}
run: python3 scripts/release/deliver.py reuse --kind image --target "$VARIANT"
- name: Install OCI tools
if: steps.reuse.outputs.reused != 'true'
run: |
BUILD_TYPE="${{ needs.build-check.outputs.build_type }}"
VERSION="${{ needs.build-check.outputs.version }}"
CREATE_LATEST="${{ needs.build-check.outputs.create_latest }}"
VARIANT_SUFFIX="${{ matrix.suffix }}"
# Convert version format for Dockerfile compatibility. The former
# DOCKER_CHANNEL was "release" down every branch and was passed as a
# build-arg no Dockerfile declares, so it is gone.
case "$VERSION" in
"latest")
DOCKER_RELEASE="latest"
;;
v*)
# For versioned releases (v1.0.0), remove 'v' prefix for Dockerfile
DOCKER_RELEASE="${VERSION#v}"
;;
*)
# For other versions, pass as-is
DOCKER_RELEASE="${VERSION}"
;;
esac
echo "docker_release=$DOCKER_RELEASE" >> "$GITHUB_OUTPUT"
echo "🐳 Docker build parameters:"
echo " - Original version: $VERSION"
echo " - Docker RELEASE: $DOCKER_RELEASE"
# Generate tags based on build type
# Only support release and prerelease builds (no development builds)
TAG_BASE="${VERSION}${VARIANT_SUFFIX}"
TAGS="${{ env.REGISTRY_DOCKERHUB }}:$TAG_BASE,${{ env.REGISTRY_GHCR }}:$TAG_BASE,${{ env.REGISTRY_QUAY }}:$TAG_BASE"
# Add latest when requested (stable releases, and prereleases before GA)
if [[ "$CREATE_LATEST" == "true" ]]; then
TAGS="$TAGS,${{ env.REGISTRY_DOCKERHUB }}:latest${VARIANT_SUFFIX},${{ env.REGISTRY_GHCR }}:latest${VARIANT_SUFFIX},${{ env.REGISTRY_QUAY }}:latest${VARIANT_SUFFIX}"
fi
# Always add the channel tag for prereleases, independent of latest
if [[ "$BUILD_TYPE" == "prerelease" ]]; then
# Prerelease channel tags (alpha, beta, rc)
if [[ "$VERSION" == *"alpha"* ]]; then
CHANNEL="alpha"
elif [[ "$VERSION" == *"beta"* ]]; then
CHANNEL="beta"
elif [[ "$VERSION" == *"rc"* ]]; then
CHANNEL="rc"
fi
if [[ -n "$CHANNEL" ]]; then
TAGS="$TAGS,${{ env.REGISTRY_DOCKERHUB }}:${CHANNEL}${VARIANT_SUFFIX},${{ env.REGISTRY_GHCR }}:${CHANNEL}${VARIANT_SUFFIX},${{ env.REGISTRY_QUAY }}:${CHANNEL}${VARIANT_SUFFIX}"
fi
fi
# Output tags
echo "tags=$TAGS" >> "$GITHUB_OUTPUT"
# Generate labels
LABELS="org.opencontainers.image.title=RustFS"
LABELS="$LABELS,org.opencontainers.image.description=RustFS distributed object storage system"
LABELS="$LABELS,org.opencontainers.image.version=$VERSION"
SOURCE_REVISION="$(git rev-parse HEAD)"
LABELS="$LABELS,org.opencontainers.image.revision=$SOURCE_REVISION"
LABELS="$LABELS,org.opencontainers.image.source=${{ github.server_url }}/${{ github.repository }}"
LABELS="$LABELS,org.opencontainers.image.created=$(date -u +'%Y-%m-%dT%H:%M:%SZ')"
LABELS="$LABELS,org.opencontainers.image.build-type=$BUILD_TYPE"
echo "labels=$LABELS" >> "$GITHUB_OUTPUT"
echo "🐳 Generated Docker tags:"
echo "$TAGS" | tr ',' '\n' | sed 's/^/ - /'
echo "📋 Build type: $BUILD_TYPE"
echo "🔖 Version: $VERSION"
- name: Build and push Docker image
sudo apt-get update
sudo apt-get install -y skopeo
- uses: docker/setup-qemu-action@c7c53464625b32c7a7e944ae62b3e17d2b600130 # v3
if: steps.reuse.outputs.reused != 'true'
- uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f # v3
if: steps.reuse.outputs.reused != 'true'
- name: Supply accepted binaries and pinned CLI
if: steps.reuse.outputs.reused != 'true'
env:
GH_TOKEN: ${{ github.token }}
run: python3 scripts/release/deliver.py image-prepare --target "$VARIANT"
- name: Build OCI archive without registry writes
if: steps.reuse.outputs.reused != 'true'
uses: docker/build-push-action@10e90e3645eae34f1e60eeb005ba3a3d33f178e8 # v6
with:
context: .
file: ${{ matrix.file }}
platforms: ${{ env.DOCKER_PLATFORMS }}
push: ${{ needs.build-check.outputs.should_push == 'true' }}
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
# No layer cache. This build compiles nothing — it downloads a
# release zip and runs apk/apt — so the cache could only save the
# minute or two those take, while creating a correctness problem: with
# RELEASE=latest the binary URL is resolved by curl *inside* a RUN
# layer, and the layer key does not include what that resolved to. A
# rebuild at the same RELEASE value (dispatch with version=latest, or
# a re-run of the same version) would hit the old layer and ship the
# previous release's binary. mode=max also consumed the same 10GB
# Actions cache quota the Rust lanes are fighting over.
#
# Only RELEASE is passed: it is the sole build-arg the Dockerfiles
# declare besides TARGETARCH. BUILDTIME, VERSION, BUILD_TYPE, REVISION
# and CHANNEL were never read by any stage (and BUILDTIME's $(date ...)
# was a literal here, not a shell substitution). BUILD_DATE and VCS_REF
# are declared by the Dockerfiles but deliberately left unset —
# supplying them would change the published image labels.
platforms: linux/amd64,linux/arm64
push: false
provenance: false
outputs: type=oci,dest=delivery/image-${{ matrix.variant }}.tar
build-args: |
RELEASE=${{ steps.meta.outputs.docker_release }}
provenance: true
sbom: true
# Add retry mechanism by splitting the build process
no-cache: false
pull: true
# Note: Manifest creation is no longer needed as we only build one variant
# Multi-arch manifests are automatically created by docker/build-push-action
# Report-only container image vulnerability scan
scan-docker-image:
name: Scan Docker Images
needs: [ build-check, build-docker ]
if: needs.build-check.outputs.should_build == 'true' && needs.build-check.outputs.should_push == 'true'
runs-on: ubuntu-latest
timeout-minutes: 30
permissions:
contents: read
security-events: write
strategy:
fail-fast: false
matrix:
include:
- variant: musl
suffix: ""
- variant: glibc
suffix: "-glibc"
steps:
- name: Login to GitHub Container Registry
# docker/login-action v3
uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9
with:
registry: ghcr.io
username: ${{ secrets.GHCR_USERNAME }}
password: ${{ secrets.GHCR_PASSWORD }}
- name: Scan image with Trivy
# aquasecurity/trivy-action v0.36.0
BINARY_SOURCE=supplied
RELEASE=${{ fromJson(inputs.candidate_manifest).version }}
VCS_REF=${{ fromJson(inputs.candidate_manifest).source_sha }}
- name: Verify wrapped binaries and startup on both architectures
if: steps.reuse.outputs.reused != 'true'
run: python3 scripts/release/deliver.py image-smoke --target "$VARIANT"
- name: Scan amd64 image before sealing
if: steps.reuse.outputs.reused != 'true'
uses: aquasecurity/trivy-action@ed142fd0673e97e23eac54620cfb913e5ce36c25
env:
TRIVY_PLATFORM: linux/amd64
with:
image-ref: ${{ env.REGISTRY_GHCR }}:${{ needs.build-check.outputs.version }}${{ matrix.suffix }}
input: delivery-scan/image-${{ matrix.variant }}-amd64.tar
format: sarif
output: trivy-${{ matrix.variant }}.sarif
ignore-unfixed: true
vuln-type: os,library
severity: CRITICAL,HIGH
output: delivery/scan-${{ matrix.variant }}-amd64.sarif
exit-code: "0" # Keep the repository's report-only vulnerability policy.
- name: Scan arm64 image before sealing
if: steps.reuse.outputs.reused != 'true'
uses: aquasecurity/trivy-action@ed142fd0673e97e23eac54620cfb913e5ce36c25
env:
TRIVY_PLATFORM: linux/arm64
with:
input: delivery-scan/image-${{ matrix.variant }}-arm64.tar
format: sarif
output: delivery/scan-${{ matrix.variant }}-arm64.sarif
exit-code: "0"
# Surface findings in the Security tab; an artifact alone is write-only
# reporting nobody reviews.
- name: Upload scan results to code scanning
uses: github/codeql-action/upload-sarif@99df26d4f13ea111d4ec1a7dddef6063f76b97e9 # v4.37.0
- name: Record verified OCI identity
if: steps.reuse.outputs.reused != 'true'
run: python3 scripts/release/deliver.py image-record --target "$VARIANT"
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
if: steps.reuse.outputs.reused != 'true'
with:
sarif_file: trivy-${{ matrix.variant }}.sarif
category: container-image-${{ matrix.variant }}
- name: Upload container scan report
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: container-image-scan-${{ matrix.variant }}
path: trivy-${{ matrix.variant }}.sarif
name: image-${{ matrix.variant }}
path: delivery/*
if-no-files-found: error
retention-days: 30
# Docker build summary
docker-summary:
name: Docker Build Summary
needs: [ build-check, build-docker ]
if: always() && needs.build-check.outputs.should_build == 'true'
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- name: Docker build completion summary
run: |
BUILD_TYPE="${{ needs.build-check.outputs.build_type }}"
VERSION="${{ needs.build-check.outputs.version }}"
CREATE_LATEST="${{ needs.build-check.outputs.create_latest }}"
echo "🐳 Docker build completed successfully!"
echo "📦 Build type: $BUILD_TYPE"
echo "🔢 Version: $VERSION"
echo "🚀 Strategy: Images using pre-built binaries (release channel only)"
echo ""
case "$BUILD_TYPE" in
"preview")
echo "🔍 Preview Docker image has been built with ${VERSION} tags"
echo "⏭️ Preview images do not update latest or prerelease channels"
;;
"release")
echo "🚀 Release Docker image has been built with ${VERSION} tags"
echo "✅ This image is ready for production use"
if [[ "$CREATE_LATEST" == "true" ]]; then
echo "🏷️ Latest tag has been created for stable release"
fi
;;
"prerelease")
echo "🧪 Prerelease Docker image has been built with ${VERSION} tags"
echo "⚠️ This is a prerelease image - use with caution"
# Prereleases move latest until the first stable tag exists (pre-GA policy).
if [[ "$CREATE_LATEST" == "true" ]]; then
echo "🏷️ Latest tag has been created for prerelease: $VERSION"
else
echo "🚫 Latest tag NOT created for prerelease"
fi
;;
*)
echo "❌ Unexpected build type: $BUILD_TYPE"
;;
esac
compression-level: 0
+51 -184
View File
@@ -1,136 +1,26 @@
# Copyright 2024 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
name: Publish helm chart to artifacthub
# Copyright 2026 RustFS Team
# SPDX-License-Identifier: Apache-2.0
name: Build candidate Helm chart
on:
workflow_run:
workflows: [ "Build and Release" ]
types: [ completed ]
workflow_dispatch:
workflow_call:
inputs:
version:
description: "Release version to publish, e.g. 1.0.0-beta.1 or v1.0.0-beta.1"
required: true
default: "1.0.0-beta.1"
candidate_manifest:
type: string
required: true
permissions:
contents: read
actions: read
jobs:
build-helm-package:
package:
runs-on: ubuntu-latest
timeout-minutes: 30
if: |
(github.event_name == 'workflow_dispatch' && !contains(github.event.inputs.version, '-preview')) ||
(
github.event.workflow_run.conclusion == 'success' &&
github.event.workflow_run.event == 'push' &&
contains(github.event.workflow_run.head_branch, '.') &&
!contains(github.event.workflow_run.head_branch, '-preview')
)
outputs:
raw_tag: ${{ steps.version.outputs.raw_tag }}
app_version: ${{ steps.version.outputs.app_version }}
chart_version: ${{ steps.version.outputs.chart_version }}
timeout-minutes: 15
steps:
- name: Checkout release source
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
ref: ${{ github.event.workflow_run.head_sha || (startsWith(inputs.version, 'refs/tags/') && inputs.version || format('refs/tags/{0}', inputs.version)) }}
ref: ${{ fromJson(inputs.candidate_manifest).source_sha }}
persist-credentials: false
# Both inputs reach the shell through env rather than `${{ }}`
# interpolation. A git ref name may contain `$(...)` — anything without a
# space is a legal tag — and interpolation pastes it into the script
# verbatim, where bash would run it. Reading "$RAW_INPUT" instead makes it
# data.
- name: Normalize release version
id: version
env:
RAW_INPUT: ${{ github.event.inputs.version }}
RAW_BRANCH: ${{ github.event.workflow_run.head_branch }}
run: |
set -eux
if [ "${{ github.event_name }}" = "workflow_dispatch" ]; then
RAW="$RAW_INPUT"
else
RAW="$RAW_BRANCH"
fi
case "$RAW" in
refs/tags/*)
RAW_TAG="${RAW#refs/tags/}"
;;
*)
RAW_TAG="$RAW"
;;
esac
./scripts/helm_chart_version.sh "$RAW_TAG"
# Docker and Helm share the binary-build trigger. Do not publish a chart
# until the image selected by its default appVersion is pullable.
- name: Wait for release image
timeout-minutes: 20
env:
APP_VERSION: ${{ steps.version.outputs.app_version }}
run: |
set -euo pipefail
if [[ ! "$APP_VERSION" =~ ^[0-9]+\.[0-9]+\.[0-9]+(-[0-9A-Za-z.-]+)?$ ]] ||
[[ "$APP_VERSION" == *-preview* ]]; then
echo "Invalid deliverable image version: $APP_VERSION" >&2
exit 1
fi
for attempt in {1..60}; do
if token=$(curl --fail --silent --show-error --connect-timeout 10 --max-time 20 \
'https://auth.docker.io/token?service=registry.docker.io&scope=repository:rustfs/rustfs:pull' \
| jq -er '.token // empty') &&
curl --fail --silent --show-error --head --connect-timeout 10 --max-time 20 \
--header "Authorization: Bearer $token" \
--header 'Accept: application/vnd.oci.image.index.v1+json, application/vnd.docker.distribution.manifest.list.v2+json' \
"https://registry-1.docker.io/v2/rustfs/rustfs/manifests/$APP_VERSION" > /dev/null; then
echo "Release image is available: rustfs/rustfs:$APP_VERSION"
exit 0
fi
echo "Waiting for rustfs/rustfs:$APP_VERSION ($attempt/60)"
if (( attempt < 60 )); then
sleep 15
fi
done
echo "Release image is unavailable; refusing to publish Helm chart" >&2
exit 1
- name: Replace chart version and app version
env:
CHART_VERSION: ${{ steps.version.outputs.chart_version }}
APP_VERSION: ${{ steps.version.outputs.app_version }}
run: |
set -eux
sed -i -E "s/^version:.*/version: \"${CHART_VERSION}\"/" helm/rustfs/Chart.yaml
sed -i -E "s/^appVersion:.*/appVersion: \"${APP_VERSION}\"/" helm/rustfs/Chart.yaml
- name: Set up Helm
uses: azure/setup-helm@b9e51907a09c216f16ebe8536097933489208112 # v4.3.0
- name: Install yq
- uses: azure/setup-helm@b9e51907a09c216f16ebe8536097933489208112 # v4.3.0
- name: Install pinned yq
run: |
set -euo pipefail
yq_version="4.54.1"
@@ -145,66 +35,43 @@ jobs:
echo "$tools_dir" >> "$GITHUB_PATH"
"$tools_dir/yq" --version
- name: Test Helm Chart Templates
run: ./scripts/test_helm_templates.sh
- name: Package Helm Chart
run: |
set -eux
cp helm/README.md helm/rustfs/
helm package ./helm/rustfs \
--destination helm/rustfs/ \
--version "${{ steps.version.outputs.chart_version }}"
- name: Upload helm package as artifact
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: helm-package
path: |
helm/rustfs/*.tgz
helm/rustfs/README.md
retention-days: 1
publish-helm-package:
runs-on: ubuntu-latest
timeout-minutes: 30
needs: [ build-helm-package ]
if: needs.build-helm-package.result == 'success'
steps:
- name: Checkout helm package repo
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
# persist-credentials-exempt: this checkout's token IS the push credential —
# the job git-pushes to rustfs/helm below. Clearing it breaks chart publishing.
repository: rustfs/helm
token: ${{ secrets.RUSTFS_HELM_PACKAGE }}
- name: Download helm package
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: helm-package
path: ./
- name: Set up helm
uses: azure/setup-helm@b9e51907a09c216f16ebe8536097933489208112 # v4.3.0
- name: Generate index
run: helm repo index . --url https://charts.rustfs.com
# app_version is derived from the triggering tag name, and this job holds
# the cross-repository push token with rustfs/helm already checked out —
# the worst place in the repo to paste an attacker-influenced string into
# a shell line. Passed through env so bash treats it as data.
- name: Push helm package and index file
- name: Test and package the exact application version
id: chart
env:
GIT_USERNAME: ${{ secrets.USERNAME }}
GIT_EMAIL: ${{ secrets.EMAIL_ADDRESS }}
APP_VERSION: ${{ needs.build-helm-package.outputs.app_version }}
GH_TOKEN: ${{ github.token }}
CANDIDATE_MANIFEST: ${{ inputs.candidate_manifest }}
run: |
set -eux
git config --global user.name "${GIT_USERNAME}"
git config --global user.email "${GIT_EMAIL}"
git add .
git commit -m "Update rustfs helm package with ${APP_VERSION}." || echo "No changes to commit"
git push origin main
python3 - <<'PYCHART'
import sys, subprocess, shutil
from pathlib import Path
sys.path.insert(0, 'scripts/release')
import candidate as c, pipeline
inputs = pipeline.manifest_env()
current = pipeline.current_run('release-complete.yml')
if any(a['name'] == 'helm-package' for a in c.artifact_list(current['id'])):
c.download_artifact(current, 'helm-package', 'chart')
record = c.read_json('chart/chart.json')
c.require(record['inputs_sha256'] == c.digest(inputs), 'chart belongs to another candidate')
c.emit_output('reused', 'true')
else:
chart = Path('helm/rustfs/Chart.yaml')
import re
value = chart.read_text()
for key in ('version', 'appVersion'):
value, count = re.subn(rf'(?m)^{key}:.*$', f'{key}: "{inputs["version"]}"', value)
c.require(count == 1, 'ambiguous chart metadata')
chart.write_text(value)
subprocess.run(['bash', 'scripts/test_helm_templates.sh'], check=True)
shutil.copyfile('helm/README.md', 'helm/rustfs/README.md')
subprocess.run(['helm', 'package', 'helm/rustfs', '--destination', 'chart'], check=True)
path = Path('chart') / f"rustfs-{inputs['version']}.tgz"
c.write_json('chart/chart.json', {'version': inputs['version'], 'inputs_sha256': c.digest(inputs),
'name': path.name, 'sha256': c.file_digest(path), 'size': path.stat().st_size})
PYCHART
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
if: steps.chart.outputs.reused != 'true'
with:
name: helm-package
path: chart/*
if-no-files-found: error
retention-days: 30
+2 -2
View File
@@ -47,8 +47,8 @@ permissions:
contents: read
concurrency:
group: oidc-keycloak-live-${{ github.ref }}
cancel-in-progress: true
group: oidc-keycloak-live-${{ github.event_name }}-${{ github.ref }}
cancel-in-progress: ${{ github.event_name != 'schedule' }}
jobs:
oidc-keycloak-live:
+38 -629
View File
@@ -1,644 +1,53 @@
# Copyright 2024 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Package Workflow - Build DEB/RPM packages
#
# This workflow builds DEB and RPM packages from pre-built Linux binaries
# and uploads them to Cloudflare R2 and the GitHub release.
#
# Trigger:
# - workflow_run: automatically package after "Build and Release" completes
# for a release tag (the mac/windows/linux binaries are already uploaded
# to the GitHub release before packaging starts)
# - workflow_dispatch: manual fallback with a release tag and/or exact build run ID
#
# Flow:
# 1. Resolve and validate the selected Build workflow run and source identity
# 2. Download Linux binaries (x86_64-gnu, aarch64-gnu, x86_64-musl,
# aarch64-musl) from build artifacts
# 3. Build DEB packages for amd64 and arm64, gnu and musl variants
# 4. Build RPM packages for x86_64 and aarch64, gnu and musl variants
# 5. Upload all packages to Cloudflare R2 and the GitHub release
name: Package DEB/RPM
permissions:
# contents: write is required to upload packages to the GitHub release
contents: write
actions: read
# Copyright 2026 RustFS Team
# SPDX-License-Identifier: Apache-2.0
name: Build candidate packages
on:
# Follows the same pattern as docker.yml: run after the release build
# workflow completes, so packaging is triggered only by release tags
# (e.g. 1.0.0-rc.2, 1.0.0-rc.3), never by development builds.
workflow_run:
workflows: [ "Build and Release" ]
types: [ completed ]
workflow_dispatch:
workflow_call:
inputs:
tag:
description: "Release tag to package (e.g. 1.0.0-beta.12). Leave empty for latest main build."
required: false
candidate_manifest:
type: string
build_run_id:
description: "Build workflow run ID (when combined with tag, both must identify the same release commit)"
required: false
type: string
concurrency:
group: ${{ github.workflow }}-${{ github.event.workflow_run.head_branch || github.event.inputs.tag || github.run_id }}
cancel-in-progress: true
env:
HEAD_BRANCH: ${{ github.event.workflow_run.head_branch }}
WORKFLOW_RUN_ID: ${{ github.event.workflow_run.id }}
required: true
permissions:
contents: read
actions: read
jobs:
# Resolve which build run to use and extract version info
resolve:
name: Resolve Build
# Auto-trigger only from successful tag builds of "Build and Release".
# Tag pushes arrive as event == push with head_branch != main (a
# non-main push head_branch is the release tag name). Manual dispatch
# stays available as a fallback for backfills and re-runs.
if: >-
github.event_name == 'workflow_dispatch' ||
(github.event.workflow_run.conclusion == 'success' &&
github.event.workflow_run.event == 'push' &&
github.event.workflow_run.head_branch != 'main')
runs-on: ubuntu-latest # gh CLI required; the sm-standard-2 fleet does not ship it
timeout-minutes: 10
outputs:
version: ${{ steps.resolve.outputs.version }}
build_type: ${{ steps.resolve.outputs.build_type }}
build_run_id: ${{ steps.resolve.outputs.build_run_id }}
build_run_number: ${{ steps.resolve.outputs.build_run_number }}
head_sha: ${{ steps.resolve.outputs.head_sha }}
dev_sequence: ${{ steps.resolve.outputs.dev_sequence }}
tag: ${{ steps.resolve.outputs.tag }}
steps:
- name: Resolve build run
id: resolve
shell: bash
env:
GH_TOKEN: ${{ github.token }}
EVENT_NAME: ${{ github.event_name }}
REPOSITORY: ${{ github.repository }}
INPUT_TAG: ${{ github.event.inputs.tag }}
INPUT_RUN_ID: ${{ github.event.inputs.build_run_id }}
run: |
set -euo pipefail
fail() {
echo "❌ $1" >&2
exit 1
}
TAG=""
BUILD_RUN_ID=""
case "$EVENT_NAME" in
workflow_run)
TAG="$HEAD_BRANCH"
BUILD_RUN_ID="$WORKFLOW_RUN_ID"
;;
workflow_dispatch)
TAG="$INPUT_TAG"
BUILD_RUN_ID="$INPUT_RUN_ID"
;;
*) fail "unsupported event: $EVENT_NAME" ;;
esac
# Validate and classify tags before using them in API paths or logs.
semver_core='(0|[1-9][0-9]*)\.(0|[1-9][0-9]*)\.(0|[1-9][0-9]*)'
prerelease_id='(alpha|beta|rc)\.(0|[1-9][0-9]*)'
if [[ -n "$TAG" ]]; then
if [[ "$TAG" =~ ^${semver_core}(-${prerelease_id})?-preview\.(0|[1-9][0-9]*)$ ]]; then
BUILD_TYPE=preview
elif [[ "$TAG" =~ ^${semver_core}-${prerelease_id}$ ]]; then
BUILD_TYPE=prerelease
elif [[ "$TAG" =~ ^${semver_core}$ ]]; then
BUILD_TYPE=release
else
fail "tag is not a supported strict package version"
fi
else
BUILD_TYPE=development
fi
if [[ -n "$BUILD_RUN_ID" ]]; then
[[ "$BUILD_RUN_ID" =~ ^[1-9][0-9]*$ ]] || fail "build run ID must be a positive decimal integer"
echo "Using selected build run: $BUILD_RUN_ID"
elif [[ -n "$TAG" ]]; then
echo "Looking for build run for tag: $TAG"
BUILD_RUN_ID=$(gh api --method GET \
"repos/${REPOSITORY}/actions/workflows/build.yml/runs" \
-f branch="$TAG" -f status=success -F per_page=1 \
--jq '.workflow_runs[0].id // empty' 2>/dev/null || true)
if [[ -z "$BUILD_RUN_ID" ]]; then
BUILD_RUN_ID=$(gh api --method GET \
"repos/${REPOSITORY}/actions/workflows/build.yml/runs" \
-f event=push -f status=success -F per_page=100 2>/dev/null |
jq -r --arg tag "$TAG" \
'[.workflow_runs[] | select(.head_branch == $tag)][0].id // empty' || true)
fi
[[ "$BUILD_RUN_ID" =~ ^[1-9][0-9]*$ ]] || fail "no successful build run found for tag"
echo "Found build run: $BUILD_RUN_ID"
else
echo "No tag specified, looking for latest main build"
BUILD_RUN_ID=$(gh api --method GET \
"repos/${REPOSITORY}/actions/workflows/build.yml/runs" \
-f branch=main -f status=success -F per_page=1 \
--jq '.workflow_runs[0].id // empty' 2>/dev/null || true)
[[ "$BUILD_RUN_ID" =~ ^[1-9][0-9]*$ ]] || fail "no successful main build found"
echo "Latest main build: $BUILD_RUN_ID"
fi
# Fetch once and use the same immutable run metadata for identity,
# ordering, workflow provenance, and release-channel validation.
RUN_JSON=$(gh api "repos/${REPOSITORY}/actions/runs/${BUILD_RUN_ID}") ||
fail "cannot read selected build run"
RUN_ID=$(jq -r '.id // empty' <<<"$RUN_JSON")
RUN_NUMBER=$(jq -r '.run_number // empty' <<<"$RUN_JSON")
RUN_STATUS=$(jq -r '.status // empty' <<<"$RUN_JSON")
RUN_CONCLUSION=$(jq -r '.conclusion // empty' <<<"$RUN_JSON")
RUN_PATH=$(jq -r '.path // empty' <<<"$RUN_JSON")
HEAD_SHA=$(jq -r '.head_sha // empty' <<<"$RUN_JSON")
RUN_HEAD_BRANCH=$(jq -r '.head_branch // empty' <<<"$RUN_JSON")
[[ "$RUN_ID" == "$BUILD_RUN_ID" ]] || fail "run metadata ID mismatch"
[[ "$RUN_NUMBER" =~ ^[1-9][0-9]*$ ]] || fail "build run number must be a positive decimal integer"
[[ "$RUN_STATUS" == completed && "$RUN_CONCLUSION" == success ]] || fail "selected build run is not successful"
[[ "$RUN_PATH" == .github/workflows/build.yml ]] || fail "selected run is not Build and Release"
[[ "$HEAD_SHA" =~ ^[0-9a-f]{40}$ ]] || fail "selected build run has an invalid head SHA"
[[ "$RUN_HEAD_BRANCH" != *$'\n'* && -n "$RUN_HEAD_BRANCH" ]] || fail "selected build run has an invalid head branch"
if [[ -n "$TAG" ]]; then
[[ "$RUN_HEAD_BRANCH" == "$TAG" ]] || fail "tag and build run head branch do not match"
TAG_REF_JSON=$(gh api "repos/${REPOSITORY}/git/ref/tags/${TAG}") ||
fail "cannot resolve release tag ref"
TAG_OBJECT_TYPE=$(jq -r '.object.type // empty' <<<"$TAG_REF_JSON")
TAG_OBJECT_SHA=$(jq -r '.object.sha // empty' <<<"$TAG_REF_JSON")
depth=0
while [[ "$TAG_OBJECT_TYPE" == tag && $depth -lt 5 ]]; do
TAG_OBJECT_JSON=$(gh api "repos/${REPOSITORY}/git/tags/${TAG_OBJECT_SHA}") ||
fail "cannot peel annotated release tag"
TAG_OBJECT_TYPE=$(jq -r '.object.type // empty' <<<"$TAG_OBJECT_JSON")
TAG_OBJECT_SHA=$(jq -r '.object.sha // empty' <<<"$TAG_OBJECT_JSON")
depth=$((depth + 1))
done
[[ "$TAG_OBJECT_TYPE" == commit && "$TAG_OBJECT_SHA" =~ ^[0-9a-f]{40}$ ]] ||
fail "release tag does not resolve to a commit"
[[ "$TAG_OBJECT_SHA" == "$HEAD_SHA" ]] || fail "release tag commit and build run head SHA do not match"
VERSION="$TAG"
DEV_SEQUENCE=""
else
VERSION="dev-${HEAD_SHA}"
DEV_SEQUENCE="$RUN_NUMBER"
fi
{
echo "version=$VERSION"
echo "build_type=$BUILD_TYPE"
echo "build_run_id=$BUILD_RUN_ID"
echo "build_run_number=$RUN_NUMBER"
echo "head_sha=$HEAD_SHA"
echo "dev_sequence=$DEV_SEQUENCE"
echo "tag=${TAG}"
} >> "$GITHUB_OUTPUT"
echo "📊 Resolved:"
echo " Version: $VERSION"
echo " Build type: $BUILD_TYPE"
echo " Build run ID: $BUILD_RUN_ID"
echo " Build run number: $RUN_NUMBER"
# Build DEB and RPM packages for each architecture and libc variant
package:
name: Package (${{ matrix.arch }}-${{ matrix.libc }})
needs: resolve
runs-on: ubuntu-latest # gh CLI required; the sm-standard-2 fleet does not ship it
timeout-minutes: 30
runs-on: ubuntu-latest
timeout-minutes: 45
strategy:
fail-fast: false
max-parallel: 1
matrix:
include:
- arch: x86_64
libc: gnu
deb_arch: amd64
rpm_arch: x86_64
artifact_name: "rustfs-linux-x86_64-gnu"
# gnu binaries dynamically link glibc; musl binaries are static
# and run without a libc dependency.
deb_depends: "libc6 (>= 2.31)"
rpm_depends: "glibc >= 2.31"
- arch: aarch64
libc: gnu
deb_arch: arm64
rpm_arch: aarch64
artifact_name: "rustfs-linux-aarch64-gnu"
deb_depends: "libc6 (>= 2.31)"
rpm_depends: "glibc >= 2.31"
- arch: x86_64
libc: musl
deb_arch: amd64
rpm_arch: x86_64
artifact_name: "rustfs-linux-x86_64-musl"
deb_depends: ""
rpm_depends: ""
- arch: aarch64
libc: musl
deb_arch: arm64
rpm_arch: aarch64
artifact_name: "rustfs-linux-aarch64-musl"
deb_depends: ""
rpm_depends: ""
target: [linux-x86_64-musl, linux-aarch64-musl, linux-x86_64-gnu, linux-aarch64-gnu]
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
ref: ${{ fromJson(inputs.candidate_manifest).source_sha }}
persist-credentials: false
- name: Normalize package metadata
id: versions
shell: bash
env:
BUILD_TYPE: ${{ needs.resolve.outputs.build_type }}
SOURCE_VERSION: ${{ needs.resolve.outputs.version }}
DEV_SEQUENCE: ${{ needs.resolve.outputs.dev_sequence }}
ARCH: ${{ matrix.rpm_arch }}
LIBC: ${{ matrix.libc }}
run: |
set -euo pipefail
normalized=$(./scripts/release/package_versions.sh \
"$BUILD_TYPE" "$SOURCE_VERSION" "$DEV_SEQUENCE" "$ARCH" "$LIBC")
printf '%s\n' "$normalized" >> "$GITHUB_OUTPUT"
- name: Download binary artifact from build run
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
pattern: ${{ matrix.artifact_name }}*
path: ./binary-artifact
run-id: ${{ needs.resolve.outputs.build_run_id }}
github-token: ${{ github.token }}
merge-multiple: true
- name: Extract binary
id: binary
shell: bash
run: |
set -euo pipefail
ZIP_FILE=$(find ./binary-artifact -name "*.zip" -type f | head -1)
if [[ -z "$ZIP_FILE" ]]; then
echo "❌ No binary artifact found"
find ./binary-artifact -mindepth 1 -maxdepth 1 -print 2>/dev/null || true
exit 1
fi
echo "Found artifact: $ZIP_FILE"
mkdir -p ./bin
unzip -o "$ZIP_FILE" -d ./bin
if [[ ! -f ./bin/rustfs ]]; then
echo "❌ rustfs binary not found in archive"
exit 1
fi
chmod +x ./bin/rustfs
stat --printf='%n %s bytes\n' ./bin/rustfs
echo "✅ Binary extracted"
- name: Build DEB package
id: deb
shell: bash
env:
DEB_VERSION: ${{ steps.versions.outputs.deb_version }}
DEB_ARCH: ${{ matrix.deb_arch }}
DEB_FILE: ${{ steps.versions.outputs.deb_file }}
DEB_DEPENDS: ${{ matrix.deb_depends }}
LIBC: ${{ matrix.libc }}
run: |
set -euo pipefail
PKG_DIR="${DEB_FILE%.deb}"
echo "Building DEB: ${DEB_FILE}"
mkdir -p "${PKG_DIR}/DEBIAN"
mkdir -p "${PKG_DIR}/usr/bin"
mkdir -p "${PKG_DIR}/etc/default"
mkdir -p "${PKG_DIR}/lib/systemd/system"
mkdir -p "${PKG_DIR}/usr/share/doc/rustfs"
cp ./bin/rustfs "${PKG_DIR}/usr/bin/"
chmod 755 "${PKG_DIR}/usr/bin/rustfs"
cp deploy/build/rustfs.service "${PKG_DIR}/lib/systemd/system/"
cat > "${PKG_DIR}/etc/default/rustfs" << 'ENVEOF'
# RustFS Environment Configuration
# See https://docs.rustfs.com for more information
# RUSTFS_ACCESS_KEY=<changeme>
# RUSTFS_SECRET_KEY=<changeme>
# RUSTFS_VOLUMES=""
# RUSTFS_ADDRESS=":9000"
# RUSTFS_CONSOLE_ADDRESS=":9001"
# RUSTFS_CONSOLE_ENABLE=true
# RUSTFS_OBS_LOGGER_LEVEL=error
# RUSTFS_OBS_LOG_DIRECTORY="/var/log/rustfs/"
ENVEOF
cat > "${PKG_DIR}/DEBIAN/control" << EOF
Package: rustfs
Version: ${DEB_VERSION}
Section: utils
Priority: optional
Architecture: ${DEB_ARCH}
EOF
# gnu binaries dynamically link glibc and must declare it; musl
# binaries are statically linked and need no libc dependency.
if [[ -n "$DEB_DEPENDS" ]]; then
echo "Depends: ${DEB_DEPENDS}" >> "${PKG_DIR}/DEBIAN/control"
fi
cat >> "${PKG_DIR}/DEBIAN/control" << EOF
Maintainer: RustFS Team <support@rustfs.com>
Description: High-performance distributed object storage (${LIBC} build)
RustFS is a high-performance distributed object storage software
built using Rust. It is compatible with MinIO and S3 API.
Homepage: https://rustfs.com
EOF
# Declare /etc/default/rustfs as a conffile so dpkg preserves user
# modifications on upgrade instead of silently overwriting them.
cat > "${PKG_DIR}/DEBIAN/conffiles" << 'CONFFILES'
/etc/default/rustfs
CONFFILES
./scripts/release/package_service_scripts.sh after-install > "${PKG_DIR}/DEBIAN/postinst"
chmod 755 "${PKG_DIR}/DEBIAN/postinst"
./scripts/release/package_service_scripts.sh before-remove > "${PKG_DIR}/DEBIAN/prerm"
chmod 755 "${PKG_DIR}/DEBIAN/prerm"
./scripts/release/package_service_scripts.sh after-remove > "${PKG_DIR}/DEBIAN/postrm"
chmod 755 "${PKG_DIR}/DEBIAN/postrm"
cp LICENSE "${PKG_DIR}/usr/share/doc/rustfs/"
cp README.md "${PKG_DIR}/usr/share/doc/rustfs/"
fakeroot dpkg-deb --build "${PKG_DIR}" "$DEB_FILE"
[[ $(dpkg-deb -f "$DEB_FILE" Package) == rustfs ]]
[[ $(dpkg-deb -f "$DEB_FILE" Version) == "$DEB_VERSION" ]]
[[ $(dpkg-deb -f "$DEB_FILE" Architecture) == "$DEB_ARCH" ]]
dpkg-deb --fsys-tarfile "$DEB_FILE" | tar -tf - | grep -Fx './usr/bin/rustfs' >/dev/null
stat --printf='%n %s bytes\n' "$DEB_FILE"
echo "deb_file=$DEB_FILE" >> "$GITHUB_OUTPUT"
echo "✅ DEB built: $DEB_FILE"
- name: Build RPM package
id: rpm
shell: bash
env:
RPM_VERSION: ${{ steps.versions.outputs.rpm_version }}
RPM_RELEASE: ${{ steps.versions.outputs.rpm_release }}
RPM_ARCH: ${{ matrix.rpm_arch }}
RPM_FILE: ${{ steps.versions.outputs.rpm_file }}
RPM_DEPENDS: ${{ matrix.rpm_depends }}
LIBC: ${{ matrix.libc }}
run: |
set -euo pipefail
echo "Building RPM for ${RPM_ARCH} (${LIBC})"
sudo apt-get update && sudo apt-get install -y ruby ruby-dev build-essential rpm
sudo gem install fpm
./scripts/test_package_versions.sh --require-package-managers
# gnu binaries dynamically link glibc and must declare it; musl
# binaries are statically linked and need no libc dependency.
depends_args=()
if [[ -n "$RPM_DEPENDS" ]]; then
depends_args+=(--depends "$RPM_DEPENDS")
fi
# Create config file for fpm (DEB build creates it in its package dir structure,
# but fpm needs the file to exist before packaging)
mkdir -p ./tmp-pkg/etc/default
cat > ./tmp-pkg/etc/default/rustfs << 'ENVEOF'
# RustFS Environment Configuration
# See https://docs.rustfs.com for more information
# RUSTFS_ACCESS_KEY=<changeme>
# RUSTFS_SECRET_KEY=<changeme>
# RUSTFS_VOLUMES=""
# RUSTFS_ADDRESS=":9000"
# RUSTFS_CONSOLE_ADDRESS=":9001"
# RUSTFS_CONSOLE_ENABLE=true
# RUSTFS_OBS_LOGGER_LEVEL=error
# RUSTFS_OBS_LOG_DIRECTORY="/var/log/rustfs/"
ENVEOF
fpm -s dir -t rpm \
--name rustfs \
--version "$RPM_VERSION" \
--iteration "$RPM_RELEASE" \
--architecture "$RPM_ARCH" \
--package "$RPM_FILE" \
"${depends_args[@]}" \
--description "High-performance distributed object storage (${LIBC} build)" \
--maintainer "RustFS Team <support@rustfs.com>" \
--url "https://rustfs.com" \
--license "Apache-2.0" \
--before-install <(./scripts/release/package_service_scripts.sh rpm-before-install) \
--before-upgrade <(./scripts/release/package_service_scripts.sh rpm-before-upgrade) \
--after-install <(./scripts/release/package_service_scripts.sh after-install) \
--after-upgrade <(./scripts/release/package_service_scripts.sh after-install) \
--before-remove <(./scripts/release/package_service_scripts.sh before-remove) \
--after-remove <(./scripts/release/package_service_scripts.sh after-remove) \
--rpm-posttrans <(./scripts/release/package_service_scripts.sh rpm-posttrans) \
--config-files /etc/default/rustfs \
./bin/rustfs=/usr/bin/rustfs \
./tmp-pkg/etc/default/rustfs=/etc/default/rustfs \
deploy/build/rustfs.service=/lib/systemd/system/rustfs.service \
LICENSE=/usr/share/doc/rustfs/LICENSE \
README.md=/usr/share/doc/rustfs/README.md
if [[ ! -f "$RPM_FILE" ]]; then
echo "❌ RPM build failed"
exit 1
fi
RPM_METADATA=$(rpm -qp --qf '%{NAME}\n%{VERSION}\n%{RELEASE}\n%{ARCH}\n' "$RPM_FILE")
EXPECTED_METADATA=$(printf 'rustfs\n%s\n%s\n%s' "$RPM_VERSION" "$RPM_RELEASE" "$RPM_ARCH")
[[ "$RPM_METADATA" == "$EXPECTED_METADATA" ]]
rpm -qpl "$RPM_FILE" | grep -Fx '/usr/bin/rustfs' >/dev/null
stat --printf='%n %s bytes\n' "$RPM_FILE"
echo "rpm_file=$RPM_FILE" >> "$GITHUB_OUTPUT"
echo "✅ RPM built: $RPM_FILE"
- name: Upload packages to artifacts
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: packages-${{ matrix.arch }}-${{ matrix.libc }}
path: |
*.deb
*.rpm
retention-days: 30
- name: Upload packages to Cloudflare R2
if: needs.resolve.outputs.build_type != 'preview' && env.R2_ACCESS_KEY_ID != ''
env:
R2_ACCESS_KEY_ID: ${{ secrets.R2_ACCESS_KEY_ID }}
R2_SECRET_ACCESS_KEY: ${{ secrets.R2_SECRET_ACCESS_KEY }}
R2_ENDPOINT: ${{ secrets.R2_ENDPOINT }}
R2_BUCKET: ${{ secrets.R2_BUCKET }}
AWS_EC2_METADATA_DISABLED: true
BUILD_TYPE: ${{ needs.resolve.outputs.build_type }}
DEB_FILE: ${{ steps.deb.outputs.deb_file }}
RPM_FILE: ${{ steps.rpm.outputs.rpm_file }}
shell: bash
run: |
set -euo pipefail
if [[ -z "$R2_ACCESS_KEY_ID" || -z "$R2_SECRET_ACCESS_KEY" || -z "$R2_ENDPOINT" || -z "$R2_BUCKET" ]]; then
echo "⚠️ R2 credentials missing, skipping upload"
exit 0
fi
if ! command -v aws >/dev/null 2>&1; then
sudo apt-get update && sudo apt-get install -y awscli
fi
export AWS_ACCESS_KEY_ID="$R2_ACCESS_KEY_ID"
export AWS_SECRET_ACCESS_KEY="$R2_SECRET_ACCESS_KEY"
export AWS_DEFAULT_REGION="auto"
if [[ "$BUILD_TYPE" == "development" ]]; then
R2_PREFIX="artifacts/rustfs/packages/dev"
else
R2_PREFIX="artifacts/rustfs/packages/release"
fi
R2_PATH="s3://${R2_BUCKET}/${R2_PREFIX}/"
echo "📤 Uploading to $R2_PATH"
for f in "$DEB_FILE" "$RPM_FILE"; do
if [[ -n "$f" && -f "$f" ]]; then
echo "Uploading: $f"
aws s3 cp "$f" "$R2_PATH" --endpoint-url "$R2_ENDPOINT" --only-show-errors
fi
done
echo "✅ Upload complete"
# Also upload as latest for release/prerelease
if [[ "$BUILD_TYPE" == "release" || "$BUILD_TYPE" == "prerelease" ]]; then
LATEST_PATH="s3://${R2_BUCKET}/artifacts/rustfs/packages/latest/"
for f in "$DEB_FILE" "$RPM_FILE"; do
if [[ -n "$f" && -f "$f" ]]; then
echo "Uploading latest: $(basename "$f")"
aws s3 cp "$f" "$LATEST_PATH" --endpoint-url "$R2_ENDPOINT" --only-show-errors
fi
done
echo "✅ Latest packages updated"
fi
- name: Upload packages to GitHub Release
if: needs.resolve.outputs.tag != ''
- name: Reuse completed package set
id: reuse
env:
GH_TOKEN: ${{ github.token }}
TAG: ${{ needs.resolve.outputs.tag }}
DEB_FILE: ${{ steps.deb.outputs.deb_file }}
RPM_FILE: ${{ steps.rpm.outputs.rpm_file }}
shell: bash
CANDIDATE_MANIFEST: ${{ inputs.candidate_manifest }}
TARGET: ${{ matrix.target }}
run: python3 scripts/release/deliver.py reuse --kind packages --target "$TARGET"
- name: Install packaging tools
if: steps.reuse.outputs.reused != 'true'
run: |
set -euo pipefail
# Upload the packages, then refresh the release checksums so the new
# assets are covered, matching the binary release flow.
for f in "$DEB_FILE" "$RPM_FILE"; do
if [[ -n "$f" && -f "$f" ]]; then
echo "📤 Uploading $(basename "$f") to GitHub release ${TAG}..."
gh release upload "$TAG" "$f" --clobber
fi
done
CHECKSUM_DIR="$(mktemp -d)"
gh release download "$TAG" -p 'SHA256SUMS' -p 'SHA512SUMS' \
-D "$CHECKSUM_DIR" --clobber
for spec in "SHA256SUMS:sha256sum" "SHA512SUMS:sha512sum"; do
asset="${spec%%:*}"
checksum_cmd="${spec##*:}"
checksum_file="${CHECKSUM_DIR}/${asset}"
for f in "$DEB_FILE" "$RPM_FILE"; do
if [[ -n "$f" && -f "$f" ]]; then
base="$(basename "$f")"
# GitHub stores release asset names with '~' normalized to '.'.
# Current package file names embed the raw tag and contain no
# '~', but older releases may still ship such assets, so the
# checksum entries must reference the name as stored on the
# release.
github_base="${base//\~/.}"
# Remove any stale entry (both naming variants), then append
grep -Fv -- "$base" "$checksum_file" > "${checksum_file}.tmp" || true
grep -Fv -- "$github_base" "${checksum_file}.tmp" > "${checksum_file}.tmp2" || true
mv "${checksum_file}.tmp2" "$checksum_file"
digest=$("$checksum_cmd" -- "$f" | awk '{print $1}')
printf '%s %s\n' "$digest" "$github_base" >> "$checksum_file"
fi
done
echo "📤 Updating ${asset} for release ${TAG}..."
gh release upload "$TAG" "$checksum_file" --clobber
done
echo "✅ GitHub release assets updated"
# Summary
summary:
name: Summary
needs: [ resolve, package ]
# Every completed Build and Release run triggers this workflow; when
# resolve declined it (not a release tag), there is nothing to summarise.
if: always() && needs.resolve.result != 'skipped'
runs-on: ubuntu-latest
timeout-minutes: 5
steps:
- name: Print summary
shell: bash
sudo apt-get update
sudo apt-get install -y ruby ruby-dev build-essential rpm fakeroot cpio
sudo gem install fpm --version 1.17.0 --no-document
- name: Package accepted executable and verify contents
if: steps.reuse.outputs.reused != 'true'
env:
SUMMARY_VERSION: ${{ needs.resolve.outputs.version }}
SUMMARY_BUILD_TYPE: ${{ needs.resolve.outputs.build_type }}
SUMMARY_BUILD_RUN_ID: ${{ needs.resolve.outputs.build_run_id }}
SUMMARY_PACKAGE_STATUS: ${{ needs.package.result }}
run: |
{
echo "## 📦 Package Summary"
echo ""
echo "| Item | Value |"
echo "|------|-------|"
echo "| Version | \`${SUMMARY_VERSION}\` |"
echo "| Build Type | ${SUMMARY_BUILD_TYPE} |"
echo "| Build Run | #${SUMMARY_BUILD_RUN_ID} |"
echo "| Package Status | ${SUMMARY_PACKAGE_STATUS} |"
} >> "$GITHUB_STEP_SUMMARY"
GH_TOKEN: ${{ github.token }}
CANDIDATE_MANIFEST: ${{ inputs.candidate_manifest }}
TARGET: ${{ matrix.target }}
run: python3 scripts/release/deliver.py package --target "$TARGET" --output delivery
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
if: steps.reuse.outputs.reused != 'true'
with:
name: packages-${{ matrix.target }}
path: delivery/*
if-no-files-found: error
compression-level: 0
retention-days: 30
+308
View File
@@ -0,0 +1,308 @@
# Copyright 2026 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# s3-tests reference parity lane (rustfs/backlog#2773).
#
# Runs the ceph/s3-tests implemented list (scripts/s3-tests/implemented_tests.txt)
# against every target in .config/reference-parity-targets.toml and fails on
# any per-node-id difference between a candidate and the reference:
#
# - reference: the release pinned in .config/reference-binary.toml, downloaded
# and sha256-verified by .github/actions/reference-binary;
# - candidate targets: the debug binary built from this checkout exactly as
# ci.yml's build-rustfs-debug-binary job builds it for S3 Implemented Tests.
#
# Every target runs the same harness (scripts/s3-tests/run.sh from this
# checkout), test list and server configuration on its own hosted runner. A
# target's failures get one rerun against a fresh server; a case that passes it
# is flaky, not a difference. scripts/s3-tests/reference_parity.py judges the
# result and honours .config/reference-parity-tolerated.txt, whose entries must
# each match a real difference.
#
# Report-only: not part of the Test and Lint aggregate. Scheduled failures are
# owned by the scheduled-validation watchdog (.github/scheduled-validations.json).
name: Reference Parity
on:
pull_request:
paths:
- ".github/workflows/reference-parity.yml"
- ".github/actions/reference-binary/**"
- ".config/reference-binary.toml"
- ".config/reference-parity-targets.toml"
- ".config/reference-parity-tolerated.txt"
- "scripts/s3-tests/reference_parity.py"
- "scripts/junit_diff.py"
schedule:
- cron: "37 1 * * *"
workflow_dispatch:
# Matches ci.yml: these variables are hashed into the rust-cache key, so the
# candidate build reads the same ci-dev cache as build-rustfs-debug-binary.
env:
CARGO_TERM_COLOR: always
RUST_BACKTRACE: 1
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
permissions:
contents: read
defaults:
run:
shell: bash
jobs:
plan:
name: Plan parity targets
runs-on: ubuntu-latest
timeout-minutes: 5
outputs:
targets: ${{ steps.targets.outputs.targets }}
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Validate target list
id: targets
run: |
set -euo pipefail
targets="$(python3 scripts/s3-tests/reference_parity.py targets --config .config/reference-parity-targets.toml)"
echo "targets=${targets}" >> "$GITHUB_OUTPUT"
echo "Parity targets: ${targets}"
build-candidate:
name: Build candidate binary
# Critical-path CPU and cache ownership: the same debug build as ci.yml's
# build-rustfs-debug-binary, reading the ci-dev cache written on sm-standard-4.
runs-on: sm-standard-4
timeout-minutes: 60
env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Setup Rust environment
uses: ./.github/actions/setup
with:
rust-version: stable
cache-shared-key: ci-dev
cache-save-if: 'false'
install-build-packaging-tools: 'false'
- name: Build debug binary
run: python3 scripts/e2e_binary.py build --bins --features e2e-test-hooks
- name: Upload candidate binary
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: reference-parity-candidate-binary
overwrite: true
path: |
target/debug/rustfs
target/debug/rustfs.e2e.json
if-no-files-found: error
retention-days: 1
s3tests:
name: s3-tests (${{ matrix.target.name }})
needs: [ plan, build-candidate ]
runs-on: ubuntu-latest
timeout-minutes: 75
strategy:
fail-fast: false
matrix:
target: ${{ fromJSON(needs.plan.outputs.targets) }}
env:
TARGET_NAME: ${{ matrix.target.name }}
TARGET_KIND: ${{ matrix.target.binary }}
# Above this many first-run failures the target is systemically broken;
# a rerun would only repeat it, so the first run is judged as is.
RERUN_CAP: "50"
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Fetch pinned reference binary
id: reference
if: matrix.target.binary == 'reference'
uses: ./.github/actions/reference-binary
- name: Download candidate binary
if: matrix.target.binary == 'candidate'
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: reference-parity-candidate-binary
path: target/debug
- name: Select target binary
env:
REFERENCE_PATH: ${{ steps.reference.outputs.path }}
run: |
set -euo pipefail
case "$TARGET_KIND" in
reference) binary="$REFERENCE_PATH" ;;
candidate) chmod +x target/debug/rustfs; binary="$PWD/target/debug/rustfs" ;;
*) echo "unknown target binary kind: $TARGET_KIND" >&2; exit 1 ;;
esac
test -x "$binary"
echo "TARGET_BINARY=$binary" >> "$GITHUB_ENV"
mkdir -p "parity/$TARGET_NAME"
{
echo "target=$TARGET_NAME"
echo "binary=$TARGET_KIND"
echo "sha256=$(sha256sum "$binary" | cut -d' ' -f1)"
echo "version=$("$binary" --version | tr '\n' ' ')"
} | tee "parity/$TARGET_NAME/target.txt"
- name: Apply target environment
env:
TARGET_JSON: ${{ toJSON(matrix.target) }}
run: |
set -euo pipefail
python3 scripts/s3-tests/reference_parity.py export-env --target-json "$TARGET_JSON" \
| tee -a "$GITHUB_ENV" "parity/$TARGET_NAME/target-env.txt"
- name: Run implemented s3-tests
run: |
set -uo pipefail
report="parity/$TARGET_NAME"
harness="artifacts/s3tests-single"
# Same invocation as ci.yml's S3 Implemented Tests, with a fresh data
# root and port per run. $1 optionally replaces the test list.
run_suite() {
local list="$1" label="$2" root port
root="${RUNNER_TEMP}/rustfs-s3tests-${label}"
rm -rf "$harness" "$root"
mkdir -p "$root"
port="$(python3 -c 'import socket; s=socket.socket(); s.bind(("127.0.0.1", 0)); print(s.getsockname()[1]); s.close()')"
env DEPLOY_MODE=binary \
RUSTFS_BINARY="$TARGET_BINARY" \
TEST_MODE=single \
MAXFAIL=0 \
XDIST=4 \
S3_PORT="$port" \
DATA_ROOT="$root" \
S3TESTS_CONF="$harness/s3tests.conf" \
${list:+IMPLEMENTED_TESTS_FILE="$list"} \
./scripts/s3-tests/run.sh
}
# A run is usable only if it produced a JUnit report; test failures
# are data for the comparison, a missing report is a lane failure.
keep() {
local prefix="$1"
for file in pytest.log compat-report.md; do
if [[ -f "$harness/$file" ]]; then cp "$harness/$file" "$report/${prefix}${file}"; fi
done
if [[ -f "$harness/rustfs-single/rustfs.log" ]]; then cp "$harness/rustfs-single/rustfs.log" "$report/${prefix}rustfs.log"; fi
test -s "$harness/junit.xml" || { echo "::error::$TARGET_NAME produced no JUnit report (${prefix:-first} run)"; return 1; }
cp "$harness/junit.xml" "$report/${prefix}junit.xml"
}
run_suite "" first
echo "first_run_exit=$?" >> "$report/target.txt"
keep "" || exit 1
cp "$harness/selected-nodeids.txt" "$report/selected-nodeids.txt" || exit 1
failures="$(python3 scripts/s3-tests/reference_parity.py failed --report "$report")" || exit 1
count="$(printf '%s' "$failures" | grep -c . || true)"
echo "first_run_failures=$count" >> "$report/target.txt"
if (( count == 0 )); then
exit 0
fi
if (( count > RERUN_CAP )); then
echo "::warning::$TARGET_NAME has $count failures (cap $RERUN_CAP); judging the first run without a rerun"
echo "rerun=skipped-over-cap" >> "$report/target.txt"
exit 0
fi
printf '%s\n' "$failures" > "$RUNNER_TEMP/rerun-tests.txt"
run_suite "$RUNNER_TEMP/rerun-tests.txt" rerun
echo "rerun_exit=$?" >> "$report/target.txt"
keep rerun- || exit 1
- name: Upload target report
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: reference-parity-report-${{ matrix.target.name }}
overwrite: true
path: parity/${{ matrix.target.name }}/
if-no-files-found: error
retention-days: 14
compare:
name: Compare targets
needs: [ plan, s3tests ]
if: always() && needs.plan.result == 'success'
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Download target reports
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
pattern: reference-parity-report-*
path: parity-download
- name: Compare per node id
env:
TARGETS: ${{ needs.plan.outputs.targets }}
S3TESTS_RESULT: ${{ needs.s3tests.result }}
run: |
set -uo pipefail
mkdir -p parity
for dir in parity-download/reference-parity-report-*; do
if [[ -d "$dir" ]]; then mv "$dir" "parity/${dir#parity-download/reference-parity-report-}"; fi
done
python3 scripts/s3-tests/reference_parity.py compare \
--targets-json "$TARGETS" \
--reports parity \
--tolerated .config/reference-parity-tolerated.txt \
--markdown "$GITHUB_STEP_SUMMARY" \
--json parity/diff.json > parity/diff.md
status=$?
cat parity/diff.md
if [[ "$S3TESTS_RESULT" != "success" ]]; then
echo "::error::s3-tests target jobs finished with '$S3TESTS_RESULT'"
status=1
fi
exit "$status"
- name: Upload parity diff
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: reference-parity-diff
overwrite: true
path: |
parity/diff.md
parity/diff.json
if-no-files-found: warn
retention-days: 14
+133
View File
@@ -0,0 +1,133 @@
# Copyright 2026 RustFS Team
# SPDX-License-Identifier: Apache-2.0
name: Release candidate (Linux)
on:
workflow_dispatch:
inputs:
version:
description: Final product version (no preview suffix)
type: string
required: true
source_sha:
description: Full reviewed source commit
type: string
required: true
permissions:
contents: read
actions: read
concurrency:
group: release-candidate-${{ inputs.version }}
cancel-in-progress: false
jobs:
freeze:
if: github.repository == 'rustfs/rustfs' && github.ref == 'refs/heads/main'
runs-on: ubuntu-latest
timeout-minutes: 15
outputs:
manifest: ${{ steps.freeze.outputs.manifest }}
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
fetch-depth: 0
persist-credentials: false
- name: Freeze or recover original inputs
id: freeze
env:
GH_TOKEN: ${{ github.token }}
VERSION: ${{ inputs.version }}
SOURCE_SHA: ${{ inputs.source_sha }}
run: python3 scripts/release/pipeline.py prepare --version "$VERSION" --source "$SOURCE_SHA" --directory frozen
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
if: steps.freeze.outputs.reused != 'true'
with:
name: release-inputs
path: frozen/inputs.json
if-no-files-found: error
retention-days: 30
linux:
needs: freeze
uses: ./.github/workflows/build.yml
with:
candidate_manifest: ${{ needs.freeze.outputs.manifest }}
platforms: linux
upgrade:
needs: [freeze, linux]
runs-on: ubuntu-latest
timeout-minutes: 150
env:
RUSTUP_TOOLCHAIN: ${{ fromJson(needs.freeze.outputs.manifest).toolchain }}
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
ref: ${{ fromJson(needs.freeze.outputs.manifest).workflow_sha }}
persist-credentials: false
- uses: ./.github/actions/setup
with:
rust-version: ${{ fromJson(needs.freeze.outputs.manifest).toolchain }}
cache-shared-key: release-upgrade-harness-hosted
cache-save-if: ${{ github.ref == 'refs/heads/main' }}
install-build-packaging-tools: "false"
install-test-tools: "false"
- name: Verify archived server against previous releases
env:
GH_TOKEN: ${{ github.token }}
CANDIDATE_MANIFEST: ${{ needs.freeze.outputs.manifest }}
run: |
python3 - <<'PYVERIFY'
import sys
from pathlib import Path
sys.path.insert(0, 'scripts/release')
import candidate as c, pipeline, verify
inputs = pipeline.manifest_env()
run = pipeline.current_run('release-candidate.yml')
if any(a['name'] == 'release-upgrade' for a in c.artifact_list(run['id'])):
c.download_artifact(run, 'release-upgrade', 'receipts')
pipeline.verify_evidence(inputs, Path('receipts'), {run['id']: run})
c.emit_output('reused', 'true')
else:
target = 'linux-x86_64-gnu'
c.download_artifact(run, f"candidate-{inputs['candidate_id']}-{target}", 'binary')
verify.upgrade(inputs, Path('binary') / f"rustfs-{target}-v{inputs['version']}.zip", Path('receipts'))
PYVERIFY
id: verify
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
if: steps.verify.outputs.reused != 'true'
with:
name: release-upgrade
path: receipts/*
if-no-files-found: error
retention-days: 30
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
if: failure()
with:
name: release-upgrade-failure-${{ github.run_attempt }}
path: receipts/*
retention-days: 7
seal-linux:
needs: [freeze, linux, upgrade]
runs-on: ubuntu-latest
timeout-minutes: 30
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Collect trusted receipts and seal Linux bytes
id: seal
env:
GH_TOKEN: ${{ github.token }}
run: |
python3 scripts/release/pipeline.py linux-seal --directory sealed
{
echo '## Linux acceptance'
echo 'Use this run ID and manifest SHA-256 when starting Release candidate (complete).'
sha256sum sealed/linux-manifest.json
echo 'Record cluster, performance, integrity and upgrade acceptance in backlog before continuing.'
} >> "$GITHUB_STEP_SUMMARY"
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
if: steps.seal.outputs.reused != 'true'
with:
name: release-linux
path: sealed/*
if-no-files-found: error
compression-level: 0
retention-days: 30
+97
View File
@@ -0,0 +1,97 @@
# Copyright 2026 RustFS Team
# SPDX-License-Identifier: Apache-2.0
name: Release candidate (complete)
on:
workflow_dispatch:
inputs:
candidate_run_id:
description: Successful Linux candidate run ID
type: string
required: true
linux_manifest_sha256:
description: Exact accepted Linux manifest SHA-256
type: string
required: true
acceptance_report:
description: Backlog report accepting cluster, performance, integrity and upgrade results
type: string
required: true
permissions:
contents: read
actions: read
concurrency:
group: release-complete-${{ inputs.candidate_run_id }}
cancel-in-progress: false
jobs:
acceptance:
if: github.repository == 'rustfs/rustfs' && github.ref == 'refs/heads/main'
runs-on: ubuntu-latest
timeout-minutes: 30
environment: release-linux-acceptance
outputs:
manifest: ${{ steps.accept.outputs.manifest }}
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
fetch-depth: 0
persist-credentials: false
- name: Bind acceptance to the exact Linux manifest
id: accept
env:
GH_TOKEN: ${{ github.token }}
CANDIDATE_RUN: ${{ inputs.candidate_run_id }}
MANIFEST_SHA: ${{ inputs.linux_manifest_sha256 }}
REPORT: ${{ inputs.acceptance_report }}
run: python3 scripts/release/pipeline.py complete-prepare --run-id "$CANDIDATE_RUN" --digest "$MANIFEST_SHA" --report "$REPORT" --directory linux
desktop:
needs: acceptance
uses: ./.github/workflows/build.yml
with:
candidate_manifest: ${{ needs.acceptance.outputs.manifest }}
platforms: desktop
packages:
needs: acceptance
uses: ./.github/workflows/package.yml
with:
candidate_manifest: ${{ needs.acceptance.outputs.manifest }}
images:
needs: acceptance
uses: ./.github/workflows/docker.yml
with:
candidate_manifest: ${{ needs.acceptance.outputs.manifest }}
chart:
needs: acceptance
uses: ./.github/workflows/helm-package.yml
with:
candidate_manifest: ${{ needs.acceptance.outputs.manifest }}
seal:
needs: [acceptance, desktop, packages, images, chart]
runs-on: ubuntu-latest
timeout-minutes: 45
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
fetch-depth: 0
persist-credentials: false
- name: Verify every deliverable and seal the candidate
id: seal
env:
GH_TOKEN: ${{ github.token }}
CANDIDATE_RUN: ${{ inputs.candidate_run_id }}
MANIFEST_SHA: ${{ inputs.linux_manifest_sha256 }}
REPORT: ${{ inputs.acceptance_report }}
run: |
python3 scripts/release/pipeline.py complete-seal --run-id "$CANDIDATE_RUN" --digest "$MANIFEST_SHA" --report "$REPORT" --directory sealed
{
echo '## Final publication approval'
echo 'Review this manifest, then explicitly dispatch Publish accepted release with its SHA-256.'
sha256sum sealed/release-manifest.json
} >> "$GITHUB_STEP_SUMMARY"
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
if: steps.seal.outputs.reused != 'true'
with:
name: release-seal
path: sealed/*
if-no-files-found: error
compression-level: 0
retention-days: 30
+193
View File
@@ -0,0 +1,193 @@
# Copyright 2026 RustFS Team
# SPDX-License-Identifier: Apache-2.0
name: Publish accepted release
on:
workflow_dispatch:
inputs:
complete_run_id:
description: Successful completed candidate run ID
type: string
required: true
manifest_sha256:
description: Exact approved final manifest SHA-256 (authorizes publication)
type: string
required: true
permissions:
contents: read
actions: read
concurrency:
group: release-publish-${{ inputs.complete_run_id }}
cancel-in-progress: false
jobs:
verify:
if: github.repository == 'rustfs/rustfs' && github.ref == 'refs/heads/main'
runs-on: ubuntu-latest
timeout-minutes: 45
outputs:
version: ${{ steps.verify.outputs.version }}
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
fetch-depth: 0
persist-credentials: false
- name: Recheck approved bytes and trusted certification
id: verify
env:
GH_TOKEN: ${{ github.token }}
COMPLETE_RUN: ${{ inputs.complete_run_id }}
MANIFEST_SHA: ${{ inputs.manifest_sha256 }}
run: python3 scripts/release/publish.py verify --run-id "$COMPLETE_RUN" --digest "$MANIFEST_SHA" --directory sealed
version:
needs: verify
runs-on: ubuntu-latest
timeout-minutes: 90
concurrency:
group: release-version-${{ needs.verify.outputs.version }}
cancel-in-progress: false
environment: release-publication
permissions:
contents: write
actions: read
packages: write
security-events: write
env:
GH_TOKEN: ${{ github.token }}
COMPLETE_RUN: ${{ inputs.complete_run_id }}
MANIFEST_SHA: ${{ inputs.manifest_sha256 }}
AWS_ACCESS_KEY_ID: ${{ secrets.R2_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.R2_SECRET_ACCESS_KEY }}
AWS_DEFAULT_REGION: auto
AWS_REQUEST_CHECKSUM_CALCULATION: when_required
AWS_RESPONSE_CHECKSUM_VALIDATION: when_required
AWS_EC2_METADATA_DISABLED: "true"
R2_ENDPOINT: ${{ secrets.R2_ENDPOINT }}
R2_BUCKET: ${{ secrets.R2_BUCKET }}
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
fetch-depth: 0
persist-credentials: false
- name: Install promotion tools
run: |
sudo apt-get update
sudo apt-get install -y skopeo
aws --version
- uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
with:
username: rustfs
password: ${{ secrets.DOCKERHUB_TOKEN }}
- uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
with:
registry: ghcr.io
username: ${{ secrets.GHCR_USERNAME }}
password: ${{ secrets.GHCR_PASSWORD }}
- uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
with:
registry: quay.io
username: ${{ secrets.QUAY_USERNAME }}
password: ${{ secrets.QUAY_PASSWORD }}
- name: Share registry credentials with the OCI copier
run: |
install -m 600 "$HOME/.docker/config.json" "$RUNNER_TEMP/registry-auth.json"
echo "REGISTRY_AUTH_FILE=$RUNNER_TEMP/registry-auth.json" >> "$GITHUB_ENV"
- name: Promote verified version
run: python3 scripts/release/publish.py version --run-id "$COMPLETE_RUN" --digest "$MANIFEST_SHA" --directory sealed
- name: Publish accepted musl amd64 scan
uses: github/codeql-action/upload-sarif@99df26d4f13ea111d4ec1a7dddef6063f76b97e9 # v4
with:
sarif_file: sealed/scan-musl-amd64.sarif
category: container-image-musl-amd64
- name: Publish accepted musl arm64 scan
uses: github/codeql-action/upload-sarif@99df26d4f13ea111d4ec1a7dddef6063f76b97e9 # v4
with:
sarif_file: sealed/scan-musl-arm64.sarif
category: container-image-musl-arm64
- name: Publish accepted GNU amd64 scan
uses: github/codeql-action/upload-sarif@99df26d4f13ea111d4ec1a7dddef6063f76b97e9 # v4
with:
sarif_file: sealed/scan-gnu-amd64.sarif
category: container-image-gnu-amd64
- name: Publish accepted GNU arm64 scan
uses: github/codeql-action/upload-sarif@99df26d4f13ea111d4ec1a7dddef6063f76b97e9 # v4
with:
sarif_file: sealed/scan-gnu-arm64.sarif
category: container-image-gnu-arm64
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: release-version-receipt-${{ github.run_attempt }}
path: publication.json
if-no-files-found: error
retention-days: 90
channels:
needs: [verify, version]
runs-on: ubuntu-latest
timeout-minutes: 90
concurrency:
group: rustfs-release-channels
cancel-in-progress: false
permissions:
contents: write
actions: read
packages: write
env:
GH_TOKEN: ${{ github.token }}
COMPLETE_RUN: ${{ inputs.complete_run_id }}
MANIFEST_SHA: ${{ inputs.manifest_sha256 }}
AWS_ACCESS_KEY_ID: ${{ secrets.R2_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.R2_SECRET_ACCESS_KEY }}
AWS_DEFAULT_REGION: auto
AWS_REQUEST_CHECKSUM_CALCULATION: when_required
AWS_RESPONSE_CHECKSUM_VALIDATION: when_required
AWS_EC2_METADATA_DISABLED: "true"
R2_ENDPOINT: ${{ secrets.R2_ENDPOINT }}
R2_BUCKET: ${{ secrets.R2_BUCKET }}
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
fetch-depth: 0
persist-credentials: false
- name: Install promotion tools
run: |
sudo apt-get update
sudo apt-get install -y skopeo
aws --version
- uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
with:
username: rustfs
password: ${{ secrets.DOCKERHUB_TOKEN }}
- uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
with:
registry: ghcr.io
username: ${{ secrets.GHCR_USERNAME }}
password: ${{ secrets.GHCR_PASSWORD }}
- uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
with:
registry: quay.io
username: ${{ secrets.QUAY_USERNAME }}
password: ${{ secrets.QUAY_PASSWORD }}
- name: Share registry credentials with the OCI copier
run: |
install -m 600 "$HOME/.docker/config.json" "$RUNNER_TEMP/registry-auth.json"
echo "REGISTRY_AUTH_FILE=$RUNNER_TEMP/registry-auth.json" >> "$GITHUB_ENV"
- uses: azure/setup-helm@b9e51907a09c216f16ebe8536097933489208112 # v4.3.0
- name: Install pinned OSS client
run: |
curl --fail --location --retry 3 --output "$RUNNER_TEMP/ossutil.zip" https://gosspublic.alicdn.com/ossutil/v2/2.1.1/ossutil-2.1.1-linux-amd64.zip
printf '%s %s\n' 60f3a808cbbdfd4b5269b8f967c6a66f564c9dacff52d93a5d21a588976ab5a2 "$RUNNER_TEMP/ossutil.zip" | sha256sum --check -
unzip -q "$RUNNER_TEMP/ossutil.zip" -d "$RUNNER_TEMP/ossutil-extract"
install -m 755 "$RUNNER_TEMP/ossutil-extract/ossutil-2.1.1-linux-amd64/ossutil" "$RUNNER_TEMP/ossutil"
echo "OSSUTIL_BIN=$RUNNER_TEMP/ossutil" >> "$GITHUB_ENV"
- name: Promote verified channels
env:
HELM_TOKEN: ${{ secrets.RUSTFS_HELM_PACKAGE }}
OSS_ACCESS_KEY_ID: ${{ secrets.ALICLOUDOSS_KEY_ID }}
OSS_ACCESS_KEY_SECRET: ${{ secrets.ALICLOUDOSS_KEY_SECRET }}
OSS_REGION: cn-beijing
OSS_ENDPOINT: https://oss-cn-beijing.aliyuncs.com
run: python3 scripts/release/publish.py channels --run-id "$COMPLETE_RUN" --digest "$MANIFEST_SHA" --directory sealed
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: release-channels-receipt-${{ github.run_attempt }}
path: publication.json
if-no-files-found: error
retention-days: 90
@@ -26,7 +26,13 @@ permissions:
jobs:
alert-on-incomplete-run:
name: Alert on incomplete scheduled run
if: github.repository == 'rustfs/rustfs'
# The registry still owns responsibility and cadence selection. Successful
# and PR runs can never produce an incomplete scheduled-run alert.
if: >-
github.repository == 'rustfs/rustfs' &&
github.event.workflow_run.conclusion != 'success' &&
(github.event.workflow_run.event == 'schedule' ||
github.event.workflow_run.event == 'workflow_run')
# Stay on GitHub-hosted runners: this monitor must keep running when the
# self-hosted sm-standard-* pool is down, which is exactly when the
# scheduled validations it watches stall.
+1 -1
View File
@@ -155,7 +155,7 @@ default build (lifecycle:
- ⚠️ PARTIALLY VIOLATED: serving-side `s3s` references remain in ecstore
(bucket metadata/replication/lifecycle DTOs and error mapping). The
count is ratcheted shrink-only by `scripts/check_s3s_footprint.sh`
(`S3S_ECSTORE_FILES_BASELINE`; the `object_lock` module was converted to
(the `ecstore_files` line of `.config/s3s-footprint-baseline.txt`; the `object_lock` module was converted to
storage-level types as the first ratchet step). Target state: the
baseline reaches zero and ecstore's `Cargo.toml` drops `s3s`.
Generated
+121 -62
View File
@@ -322,9 +322,9 @@ dependencies = [
[[package]]
name = "archive-trait"
version = "0.0.15"
version = "0.0.16"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "999bf88cc483664cab77700ca2fdfb0415a73412abe284ef5bc2319d00938e00"
checksum = "0b5040ca935c09888e7dfb04383b4d62fa3571e25bc66b6f345b591e97958524"
dependencies = [
"cap-std",
"thiserror 2.0.21",
@@ -811,9 +811,9 @@ dependencies = [
[[package]]
name = "async-nats"
version = "0.51.0"
version = "0.52.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a4f92759f418774b78c36ef07885849150ce950c67d9bcdeff8a9c253f57d9a1"
checksum = "015f0a7a3b0b29701b78c76a4d4a122624c97c71fa30363381664eb729d619c7"
dependencies = [
"aws-lc-rs",
"base64 0.22.1",
@@ -3906,9 +3906,9 @@ dependencies = [
[[package]]
name = "dial9"
version = "0.5.7"
version = "0.5.8"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2f3298a54fd82af243c9f122595762f5b0e48bd3d1ea8fe3f3eb5e95c130e881"
checksum = "c83757d61dc75b3e9d3579a3004c7fe0c42aff2f779ca9eeec7f0c8e1ae3fc10"
dependencies = [
"bon",
"dial9-core",
@@ -3990,9 +3990,9 @@ dependencies = [
[[package]]
name = "dial9-tokio-telemetry"
version = "0.5.7"
version = "0.5.8"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0428cba60fbcf37ab82666f3a7ef3b6d0671e90dfb6a352376813b471319f061"
checksum = "6f0eb05627c12c57e40510e63f9a3d5f628ab4b32fec9ccdb0ae98544c13f15e"
dependencies = [
"bon",
"bytes",
@@ -4525,9 +4525,9 @@ dependencies = [
[[package]]
name = "flate2"
version = "1.1.10"
version = "1.1.11"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6e634e2e0ebac1ee034020da1ca582e17ffe4e0f5e985823721e168928136dcb"
checksum = "626d4928326b646e70f65ed36fc7cf9f206bdb68a0019ee7c6b4709da4e084c4"
dependencies = [
"crc32fast",
"miniz_oxide",
@@ -4594,7 +4594,7 @@ checksum = "94e7099f6313ecacbe1256e8ff9d617b75d1bcb16a6fddef94866d225a01a14a"
dependencies = [
"io-lifetimes 2.0.4",
"rustix",
"windows-sys 0.59.0",
"windows-sys 0.52.0",
]
[[package]]
@@ -5385,9 +5385,9 @@ dependencies = [
[[package]]
name = "hotpath"
version = "0.30.3"
version = "0.31.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d9632c1fd213e63c1b88d8f3e59543da4999b17f859cc1e1d67cf8f2207e8935"
checksum = "4039758c8bca06a34c475d17aaafc17ac1073ce2cecd8dba8c82e69edb300bbe"
dependencies = [
"arc-swap",
"async-channel",
@@ -5420,15 +5420,15 @@ dependencies = [
[[package]]
name = "hotpath-drain"
version = "0.30.3"
version = "0.31.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "738324f75aec847dea984f6806402f4f1bdfab437b4700fc8c3ebcad2dff6424"
checksum = "bbba0a3badfb9ebf95f22dc6cdb4afcd0ff07b91a85fa3196fd1060576ea3065"
[[package]]
name = "hotpath-macros"
version = "0.30.3"
version = "0.31.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5d92fdfec803932e5764ea93c201cf7c6c2d8cff1871da44f64f5094c9c3ef8a"
checksum = "7d980014875338771b00ac44f1c9e2ea1a10650aa68ce54226233bc767714361"
dependencies = [
"proc-macro2",
"quote",
@@ -6412,9 +6412,9 @@ dependencies = [
[[package]]
name = "lzma-rust2"
version = "0.24.0"
version = "0.25.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "490750560fa7851fbbf5d3ab1a8750f09350ed361ed8aed85b1e1e40587812ee"
checksum = "ef1d73c59a56771343712b22ea5615092c612c16ba291c0979e3105dfeec4890"
dependencies = [
"sha2 0.11.1",
]
@@ -8697,6 +8697,15 @@ version = "1.2.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a1d01941d82fa2ab50be1e79e6714289dd7cde78eba4c074bc5a4374f650dfe0"
[[package]]
name = "quick-xml"
version = "0.41.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e660451e55124f798a69a5af3f49ccfbefbd41910eefd25caf2393e1f3473ec1"
dependencies = [
"memchr",
]
[[package]]
name = "quick-xml"
version = "0.42.0"
@@ -9581,7 +9590,7 @@ dependencies = [
"proptest",
"prost 0.14.4",
"pyroscope",
"quick-xml",
"quick-xml 0.42.0",
"quinn",
"rand 0.10.3",
"rcgen",
@@ -9600,6 +9609,7 @@ dependencies = [
"rustfs-ecstore",
"rustfs-extension-schema",
"rustfs-filemeta",
"rustfs-gateway-types",
"rustfs-heal",
"rustfs-heal-contracts",
"rustfs-iam",
@@ -9864,7 +9874,7 @@ dependencies = [
"path-absolutize",
"pin-project-lite",
"proptest",
"quick-xml",
"quick-xml 0.42.0",
"rand 0.10.3",
"ratelimit",
"rcgen",
@@ -9881,6 +9891,7 @@ dependencies = [
"rustfs-data-usage",
"rustfs-erasure-codec",
"rustfs-filemeta",
"rustfs-gateway-types",
"rustfs-heal-contracts",
"rustfs-io-metrics",
"rustfs-lifecycle",
@@ -9988,6 +9999,48 @@ dependencies = [
"xxhash-rust",
]
[[package]]
name = "rustfs-gateway-stream"
version = "0.2.0"
source = "git+https://github.com/rustfs/gateway.git?rev=21283e874c2381bab611e9347acbc5b92e9e2b3c#21283e874c2381bab611e9347acbc5b92e9e2b3c"
dependencies = [
"bitflags 2.13.2",
"bytes",
"http 1.5.0",
"http-body",
]
[[package]]
name = "rustfs-gateway-types"
version = "0.32.0+aws.2026-10-02"
source = "git+https://github.com/rustfs/gateway.git?rev=21283e874c2381bab611e9347acbc5b92e9e2b3c#21283e874c2381bab611e9347acbc5b92e9e2b3c"
dependencies = [
"bytes",
"crc-fast",
"futures-core",
"hex",
"http 1.5.0",
"md-5 0.11.0",
"percent-encoding",
"rustfs-gateway-stream",
"rustfs-gateway-xml",
"s3s",
"sha1 0.11.0",
"sha2 0.11.1",
"time",
"twox-hash",
"unicode-normalization",
"zeroize",
]
[[package]]
name = "rustfs-gateway-xml"
version = "0.4.0"
source = "git+https://github.com/rustfs/gateway.git?rev=21283e874c2381bab611e9347acbc5b92e9e2b3c#21283e874c2381bab611e9347acbc5b92e9e2b3c"
dependencies = [
"quick-xml 0.41.0",
]
[[package]]
name = "rustfs-heal"
version = "1.0.5"
@@ -10259,10 +10312,10 @@ dependencies = [
"metrics-util",
"proptest",
"rustfs-config",
"rustfs-gateway-types",
"rustfs-replication",
"rustfs-scanner-metrics",
"rustfs-storage-api",
"s3s",
"serial_test",
"temp-env",
"time",
@@ -10326,7 +10379,6 @@ dependencies = [
"reqwest",
"rmp-serde",
"rustfs-signer",
"s3s",
"serde",
"serde_json",
"sysinfo",
@@ -10336,18 +10388,18 @@ dependencies = [
[[package]]
name = "rustfs-mimalloc"
version = "0.7.3"
version = "0.7.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "216ecfc9341b5d6e2506d47ce57c9232ae1b8020bb10264ea05286fb234286e7"
checksum = "fc7914232a854336c62058c24a8362c553ac8e7adba2e2395ce9097db24d1510"
dependencies = [
"rustfs-mimalloc-sys",
]
[[package]]
name = "rustfs-mimalloc-sys"
version = "0.7.3"
version = "0.7.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "fb16ad68f96883407a75cdb88252b43dc2ea592428f0fc54ff7885756de4b814"
checksum = "8073624242781448512682f1931624ee67108b6b7873b22fb29c1a1df689ac0c"
dependencies = [
"cc",
]
@@ -10366,7 +10418,7 @@ dependencies = [
"jiff",
"metrics",
"percent-encoding",
"quick-xml",
"quick-xml 0.42.0",
"rayon",
"rustc-hash",
"rustfs-config",
@@ -10548,10 +10600,13 @@ dependencies = [
"rustfs-config",
"rustfs-credentials",
"rustfs-ecstore",
"rustfs-gateway-stream",
"rustfs-gateway-types",
"rustfs-iam",
"rustfs-keystone",
"rustfs-policy",
"rustfs-rio",
"rustfs-s3-types",
"rustfs-storage-api",
"rustfs-test-utils",
"rustfs-tls-runtime",
@@ -10559,13 +10614,13 @@ dependencies = [
"rustfs-trusted-proxies",
"rustfs-utils",
"rustls",
"s3s",
"serde",
"serde_json",
"sha1 0.11.0",
"sha2 0.11.1",
"socket2",
"subtle",
"sync_wrapper",
"tempfile",
"thiserror 2.0.21",
"time",
@@ -10615,7 +10670,7 @@ dependencies = [
"regex",
"rmp",
"rmp-serde",
"s3s",
"rustfs-gateway-types",
"serde",
"time",
"tokio",
@@ -10710,11 +10765,13 @@ dependencies = [
"hyper-util",
"lazy_static",
"md-5 0.11.0",
"quick-xml",
"quick-xml 0.42.0",
"rand 0.10.3",
"rustfs-checksums",
"rustfs-config",
"rustfs-gateway-types",
"rustfs-rio",
"rustfs-s3-types",
"rustfs-signer",
"rustfs-storage-api",
"rustfs-tls-runtime",
@@ -10748,9 +10805,19 @@ dependencies = [
name = "rustfs-s3-types"
version = "1.0.5"
dependencies = [
"bytes",
"futures-core",
"hotpath",
"http 1.5.0",
"http-body",
"http-body-util",
"hyper",
"hyper-util",
"pin-project-lite",
"s3s",
"serde",
"serde_json",
"tokio",
]
[[package]]
@@ -10779,9 +10846,10 @@ dependencies = [
"parking_lot 0.13.0",
"rustfs-common",
"rustfs-ecstore",
"rustfs-gateway-types",
"rustfs-s3-types",
"rustfs-storage-api",
"rustfs-test-utils",
"s3s",
"serde_json",
"serial_test",
"thiserror 2.0.21",
@@ -10809,7 +10877,6 @@ dependencies = [
"parking_lot 0.13.0",
"rustfs-s3select-api",
"rustfs-test-utils",
"s3s",
"serde",
"serde_json",
"tokio",
@@ -10841,6 +10908,7 @@ dependencies = [
"rustfs-data-usage",
"rustfs-ecstore",
"rustfs-filemeta",
"rustfs-gateway-types",
"rustfs-heal",
"rustfs-heal-contracts",
"rustfs-lifecycle",
@@ -10898,9 +10966,10 @@ dependencies = [
"bytes",
"hotpath",
"http 1.5.0",
"http-body",
"hyper",
"rustfs-s3-types",
"rustfs-utils",
"s3s",
"serde_urlencoded",
"thiserror 2.0.21",
"time",
@@ -10959,7 +11028,6 @@ dependencies = [
"rustfs-utils",
"rustls",
"rustls-native-certs",
"s3s",
"serde",
"serde_json",
"sha2 0.11.1",
@@ -11351,7 +11419,7 @@ dependencies = [
"numeric_cast",
"phf 0.14.0",
"pin-project-lite",
"quick-xml",
"quick-xml 0.42.0",
"regex",
"s3s-multipart",
"s3s-rfc2047",
@@ -11705,9 +11773,9 @@ dependencies = [
[[package]]
name = "serde_spanned"
version = "1.1.2"
version = "1.1.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4e7523beb55eece201a2356bee0bbca0d1ab466c14c07703b2e0ee6d42cb0c2c"
checksum = "9c187e0a369471c550af028cba56d4f4fa2a489ad28151ae95a4d340a2a8b782"
dependencies = [
"serde_core",
]
@@ -12037,9 +12105,9 @@ dependencies = [
[[package]]
name = "smallvec"
version = "1.16.4"
version = "1.16.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d0d5d4b1981b8bbbb7aed204cae5726d683d5d6ac7e74ec54122ea28f315e7c8"
checksum = "149309498a5f7334b12ddae0cad8aba81ce785fe91755068867e017cf98fee46"
dependencies = [
"serde",
]
@@ -12234,9 +12302,9 @@ dependencies = [
[[package]]
name = "starshard"
version = "2.4.0"
version = "2.4.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "fc45de5060fab0de00d7a42230877543ae00c37caf6f49975e69851089b023ac"
checksum = "b0883700b50fd0b62120ebc4c07230e02f0cf1ead474c3b3b44eaa13e0a999ea"
dependencies = [
"async-trait",
"hashbrown 0.17.1",
@@ -12485,9 +12553,9 @@ dependencies = [
[[package]]
name = "tar-codec"
version = "0.0.15"
version = "0.0.16"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b9dad615e50d181afea4bafea23dacb8e33ed52dde06503813570634b932d467"
checksum = "6875ba507bbec6022500f46477159a03f07ebdffdb4ea043254974531df1dac2"
dependencies = [
"archive-trait",
"tar-framing",
@@ -12497,9 +12565,9 @@ dependencies = [
[[package]]
name = "tar-framing"
version = "0.0.15"
version = "0.0.16"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "563e8181b47c9cdcce569eae3ad431eec71543d6269dd60c7bd2350f70bcd233"
checksum = "49727b1a300514960d425d0e741a7da3e693170f3db39de3180fdbbc07e542c5"
dependencies = [
"thiserror 2.0.21",
"tokio",
@@ -12873,9 +12941,9 @@ dependencies = [
[[package]]
name = "toml"
version = "1.2.0+spec-1.1.0"
version = "1.2.1+spec-1.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c5ffcdf8f50436dbbfd675c1fe1b8e0854f42b7defea8584bdebe1aee900e561"
checksum = "d5755c51863cdac711f1956baf8eb9588e060755f7d3c4c979e70ea1e1f32cd7"
dependencies = [
"indexmap 2.14.2",
"serde_core",
@@ -12888,9 +12956,9 @@ dependencies = [
[[package]]
name = "toml_datetime"
version = "1.1.2+spec-1.1.0"
version = "1.1.3+spec-1.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2b86d767906c6c42421dcba507eb9d203e779497710a47782a224bb871653053"
checksum = "173f25c883cd2176a2d8212155638c0a60d484a9d84546d270866f40f56b6e49"
dependencies = [
"serde_core",
]
@@ -13783,15 +13851,6 @@ dependencies = [
"windows-targets 0.52.6",
]
[[package]]
name = "windows-sys"
version = "0.59.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1e38bc4d79ed67fd075bcc251a1c39b32a1776bbe92e5bef1f0bf1f8c531853b"
dependencies = [
"windows-targets 0.52.6",
]
[[package]]
name = "windows-sys"
version = "0.60.2"
@@ -13964,7 +14023,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3f3fd376f71958b862e7afb20cfe5a22830e1963462f3a17f49d82a6c1d1f42d"
dependencies = [
"bitflags 2.13.2",
"windows-sys 0.59.0",
"windows-sys 0.52.0",
]
[[package]]
@@ -14214,9 +14273,9 @@ dependencies = [
[[package]]
name = "zip"
version = "9.3.0"
version = "9.5.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "eaeb7f16d2605b49aef586996ded29e65642fab7259a3077e537351df6d80426"
checksum = "0700e6c427aff64acdbf02e85fbfd946c2ee12319d1af013092f3a2178914ff8"
dependencies = [
"aes 0.9.4",
"bzip2",
+17 -5
View File
@@ -48,7 +48,7 @@ members = [
"crates/replication", # Replication contracts and wire formats
"crates/concurrency", # Concurrency management for RustFS - timeout, locking, backpressure, and I/O scheduling
"crates/s3-client", # S3 client for engine-side consumption of remote S3 endpoints (tiering, transition targets)
"crates/s3-types", # S3 event type definitions
"crates/s3-types", # S3 event and error type definitions
"crates/s3-ops", # S3 operation definitions and mapping
"crates/s3select-api", # S3 Select API interface
"crates/s3select-query", # S3 Select query engine
@@ -143,13 +143,20 @@ rustfs-tls-runtime = { path = "crates/tls-runtime", version = "1.0.5" }
rustfs-utils = { path = "crates/utils", version = "1.0.5" }
rustfs-zip = { path = "./crates/zip", version = "1.0.5" }
# The gateway's S3 types: the DTOs and persisted bucket-configuration codecs the
# engine crates move onto from s3s (rustfs/backlog#2745). The one pin of the
# gateway in this workspace; no crate may enable a `compat-s3s*` feature except
# the s3s-facing edge of the rustfs binary.
rustfs-gateway-types = { git = "https://github.com/rustfs/gateway.git", rev = "21283e874c2381bab611e9347acbc5b92e9e2b3c", version = "0.32.0", default-features = false }
rustfs-gateway-stream = { git = "https://github.com/rustfs/gateway.git", rev = "21283e874c2381bab611e9347acbc5b92e9e2b3c", version = "0.2.0", default-features = false }
# Async Runtime and Networking
async-channel = "2.5.0"
async_zip = { default-features = false, version = "0.0.19" }
mysql_async = { default-features = false, version = "0.37.1" }
async-compression = { version = "0.4.52" }
async-trait = "0.1.92"
async-nats = { version = "0.51.0", default-features = false }
async-nats = { version = "0.52.0", default-features = false }
axum = "0.8.9"
futures = "0.3.34"
futures-core = "0.3.34"
@@ -249,8 +256,8 @@ anyhow = "1.0.104"
arc-swap = "1.9.3"
rustfs-tokio-tar = { version = "0.7.1" }
# Candidate Snowball parser versions exercised by rustfs-zip compatibility fixtures.
tar-codec = "0.0.15"
tar-framing = "0.0.15"
tar-codec = "0.0.16"
tar-framing = "0.0.16"
atoi = "3.1.1"
atomic_enum = "0.3.0"
aws-config = { version = "1.13.0" }
@@ -329,6 +336,10 @@ rust-embed = { version = "8.13.0" }
rustc-hash = { version = "2.1.3" }
s3s = { git = "https://github.com/s3s-project/s3s", rev = "5761ddfe4c509fcac87e3b99fd5bd9e176b442c0", features = ["minio"] }
s3s-sigv4 = { git = "https://github.com/s3s-project/s3s", rev = "5761ddfe4c509fcac87e3b99fd5bd9e176b442c0" }
# The same s3s pin under the name the legacy edge of the gateway migration uses
# (crates/s3-types, feature `compat-s3s`). Keep `rev` identical to `s3s` above.
# DELETE BY T4.2 (rustfs/backlog#2784).
legacy-s3s = { package = "s3s", git = "https://github.com/s3s-project/s3s", rev = "5761ddfe4c509fcac87e3b99fd5bd9e176b442c0", features = ["minio"] }
serial_test = "4.0.1"
shadow-rs = { default-features = false, version = "2.0.0" }
siphasher = "1.0.4"
@@ -337,6 +348,7 @@ compact_str = "0.10.0"
snap = "1.1.2"
starshard = { version = "2.4.0" }
strum = { version = "0.28.0" }
sync_wrapper = "1.0.2"
sysinfo = "0.40.1"
temp-env = "0.3.6"
tempfile = "3.27.0"
@@ -389,7 +401,7 @@ dav-server = "0.11.0"
# Performance Analysis and Memory Profiling
rustfs-mimalloc = { version = "0.7.3" }
# Preserve Unicode focus filters until rustfs/backlog#2302 is resolved.
hotpath = { version = "0.30.3", default-features = false }
hotpath = { version = "0.31.1", default-features = false }
# Snapshot testing for output format regression detection
insta = { version = "1.49" }
+11 -2
View File
@@ -12,7 +12,9 @@
# See the License for the specific language governing permissions and
# limitations under the License.
FROM alpine:3.24.2 AS build
ARG BINARY_SOURCE=download
FROM alpine:3.24.2 AS download
ARG TARGETARCH
ARG RELEASE=latest
@@ -74,6 +76,14 @@ COPY scripts/download_rc.sh /usr/local/bin/download_rc.sh
RUN sh /usr/local/bin/download_rc.sh "$TARGETARCH" musl "$RC_RELEASE" /build/rc
# Candidate packaging supplies the accepted executables from verified archives.
# The default download stage remains available for local image builds.
FROM scratch AS supplied
ARG TARGETARCH
COPY .release-inputs/${TARGETARCH}/ /build/
FROM ${BINARY_SOURCE} AS build
FROM alpine:3.24.2
ARG RELEASE=latest
@@ -103,7 +113,6 @@ RUN apk upgrade --no-cache && \
tzdata \
&& test "$(TZ=Asia/Kolkata date +%z)" = "+0530"
COPY --from=build /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
COPY --from=build /build/rustfs /usr/bin/rustfs
COPY --from=build /build/rc /usr/bin/rc
COPY entrypoint.sh /etc/rustfs/entrypoint.sh
+11 -1
View File
@@ -12,7 +12,9 @@
# See the License for the specific language governing permissions and
# limitations under the License.
FROM ubuntu:26.04 AS build
ARG BINARY_SOURCE=download
FROM ubuntu:26.04 AS download
ARG TARGETARCH
ARG RELEASE=latest
@@ -80,6 +82,14 @@ RUN set -eux; \
COPY scripts/download_rc.sh /usr/local/bin/download_rc.sh
RUN sh /usr/local/bin/download_rc.sh "$TARGETARCH" gnu "$RC_RELEASE" /build/rc
# Candidate packaging supplies the accepted executables from verified archives.
# The default download stage remains available for local image builds.
FROM scratch AS supplied
ARG TARGETARCH
COPY .release-inputs/${TARGETARCH}/ /build/
FROM ${BINARY_SOURCE} AS build
FROM ubuntu:26.04
ARG RELEASE=latest
+3 -3
View File
@@ -145,7 +145,7 @@ chown -R 10001:10001 data logs
docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:latest
# Using specific version
docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.4
docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.5
```
If you use [podman](https://github.com/containers/podman) instead of docker, you can install the RustFS with the below command
@@ -224,7 +224,7 @@ For developers who want to build RustFS Docker images from source with multi-arc
./docker-buildx.sh --push
# Build specific version
./docker-buildx.sh --release 1.0.4 --push
./docker-buildx.sh --release 1.0.5 --push
# Build for custom registry
./docker-buildx.sh --registry your-registry.com --namespace yourname --push
@@ -242,7 +242,7 @@ You can also use Make targets for convenience:
```bash
make docker-buildx # Build locally
make docker-buildx-push # Build and push
make docker-buildx-version VERSION=1.0.4 # Build specific version
make docker-buildx-version VERSION=1.0.5 # Build specific version
make help-docker # Show all Docker-related commands
```
+3 -3
View File
@@ -138,7 +138,7 @@ chown -R 10001:10001 data logs
docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:latest
# 使用指定版本运行
docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.4
docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.5
```
如果您通过绑定挂载启用 TLS 证书目录,也请用同样方式准备该目录:
@@ -176,7 +176,7 @@ docker compose -f docker-compose-simple.yml up -d
./docker-buildx.sh --push
# 构建指定版本
./docker-buildx.sh --release 1.0.4 --push
./docker-buildx.sh --release 1.0.5 --push
# 构建并推送到自定义仓库
./docker-buildx.sh --registry your-registry.com --namespace yourname --push
@@ -193,7 +193,7 @@ docker compose -f docker-compose-simple.yml up -d
```bash
make docker-buildx # 本地构建
make docker-buildx-push # 构建并推送
make docker-buildx-version VERSION=1.0.4 # 构建指定版本
make docker-buildx-version VERSION=1.0.5 # 构建指定版本
make help-docker # 显示所有 Docker 相关命令
```
+100
View File
@@ -902,4 +902,104 @@ mod tests {
assert_eq!(raw.len(), 8);
assert_eq!(&raw[..], reference.digest().to_be_bytes().as_slice());
}
#[test]
fn checksums_streaming_chunks_and_header_encoding_match_oneshot_across_unaligned_boundaries() {
use crate::http::HttpChecksum;
let all = [
ChecksumAlgorithm::Crc32,
ChecksumAlgorithm::Crc32c,
ChecksumAlgorithm::Crc64Nvme,
ChecksumAlgorithm::Sha1,
ChecksumAlgorithm::Sha256,
ChecksumAlgorithm::Sha512,
ChecksumAlgorithm::Xxhash3,
ChecksumAlgorithm::Xxhash64,
ChecksumAlgorithm::Xxhash128,
];
for &payload_len in &[0usize, 1, 31, 33, 1024, 4096 + 37, 64 * 1024 + 19] {
let payload: Vec<u8> = (0..payload_len).map(|i| ((i * 19 + 5) % 253) as u8).collect();
for algorithm in all {
let mut one_shot = algorithm.into_impl();
one_shot.update(&payload);
let expected_raw = one_shot.finalize();
assert_eq!(expected_raw.len(), algorithm.raw_len());
let mut one_shot_hdr = algorithm.into_impl();
one_shot_hdr.update(&payload);
let expected_headers = one_shot_hdr.headers();
for &chunk_size in &[1usize, 31, 1021, 4096] {
if payload_len > 4096 && chunk_size == 1 {
continue;
}
let mut streaming: Box<dyn HttpChecksum> = algorithm.into_impl();
for chunk in payload.chunks(chunk_size) {
streaming.update(chunk);
}
let actual_headers = streaming.headers();
assert_eq!(
actual_headers.get(algorithm.http_header_name()),
expected_headers.get(algorithm.http_header_name()),
"{algorithm:?} streaming mismatch at payload_len={payload_len}, chunk_size={chunk_size}"
);
}
}
}
}
#[test]
#[ignore = "utility throughput microbenchmark; run via cargo test -p rustfs-checksums --lib --release -- --ignored --nocapture"]
fn microbench_checksums_streaming_chunks_and_header_encoding() {
use crate::http::HttpChecksum;
use std::hint::black_box;
use std::time::Instant;
let all = [
ChecksumAlgorithm::Crc32,
ChecksumAlgorithm::Crc32c,
ChecksumAlgorithm::Crc64Nvme,
ChecksumAlgorithm::Sha1,
ChecksumAlgorithm::Sha256,
ChecksumAlgorithm::Sha512,
ChecksumAlgorithm::Xxhash3,
ChecksumAlgorithm::Xxhash64,
ChecksumAlgorithm::Xxhash128,
];
let payload: Vec<u8> = (0..(256 * 1024)).map(|i| ((i * 19 + 5) % 253) as u8).collect();
for &chunk_size in &[4 * 1024usize, 64 * 1024, 256 * 1024] {
for algorithm in all {
let mut one_shot = algorithm.into_impl();
one_shot.update(&payload);
let expected_headers = one_shot.headers();
let iters = 8usize;
let t0 = Instant::now();
let mut last_headers = http::HeaderMap::new();
for _ in 0..iters {
let mut streaming: Box<dyn HttpChecksum> = algorithm.into_impl();
for chunk in payload.chunks(chunk_size) {
streaming.update(black_box(chunk));
}
last_headers = streaming.headers();
black_box(&last_headers);
}
let elapsed = t0.elapsed();
assert_eq!(
last_headers.get(algorithm.http_header_name()),
expected_headers.get(algorithm.http_header_name()),
"{algorithm:?} streaming chunk_size={chunk_size} header mismatch"
);
let total_mib = (payload.len() * iters) as f64 / (1024.0 * 1024.0);
let mibs = total_mib / elapsed.as_secs_f64().max(1e-9);
println!(
"[microbench][checksums] algo={algorithm:?} chunk_size={chunk_size}B throughput={mibs:.1} MiB/s (size={}B)",
algorithm.raw_len()
);
}
}
}
}
+106 -1
View File
@@ -17,7 +17,8 @@
//! when PublicAccessBlock configuration is missing or explicitly set.
use crate::common::{RustFSTestEnvironment, init_logging, local_http_client};
use aws_sdk_s3::types::PublicAccessBlockConfiguration;
use crate::sts_query_compat_test::{OPA_AUTH_TOKEN, OpaMock};
use aws_sdk_s3::types::{PublicAccessBlockConfiguration, Tag, Tagging};
use tracing::info;
async fn setup_public_bucket(
@@ -69,6 +70,110 @@ async fn anonymous_get_object(
local_http_client().get(&url).send().await
}
#[tokio::test]
async fn test_plain_http_matches_secure_transport_false_bucket_policy() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
let mut env = RustFSTestEnvironment::new().await?;
env.start_rustfs_server(vec![]).await?;
assert!(env.url.starts_with("http://"), "fixture must exercise plaintext HTTP: {}", env.url);
let bucket = "secure-transport-policy";
let key = "plain-http.txt";
let admin = env.create_s3_client();
admin.create_bucket().bucket(bucket).send().await?;
admin
.put_object()
.bucket(bucket)
.key(key)
.body(aws_sdk_s3::primitives::ByteStream::from_static(b"plain http"))
.send()
.await?;
admin
.put_bucket_policy()
.bucket(bucket)
.policy(
serde_json::json!({
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": "*",
"Action": "s3:GetObject",
"Resource": format!("arn:aws:s3:::{bucket}/*"),
"Condition": {"Bool": {"aws:SecureTransport": "false"}}
}]
})
.to_string(),
)
.send()
.await?;
let response = anonymous_get_object(&env, bucket, key).await?;
assert_eq!(response.status().as_u16(), 200);
assert_eq!(response.bytes().await?.as_ref(), b"plain http");
Ok(())
}
/// Root credentials have no attached IAM policy, so these requests depend on
/// the owner flag reaching policy evaluation unchanged.
#[tokio::test]
async fn test_root_requests_keep_policy_owner_flag() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
let opa = OpaMock::start().await?;
let mut env = RustFSTestEnvironment::new().await?;
env.start_rustfs_server_with_env(
vec![],
&[
("RUSTFS_POLICY_PLUGIN_URL", opa.url.as_str()),
("RUSTFS_POLICY_PLUGIN_AUTH_TOKEN", OPA_AUTH_TOKEN),
],
)
.await?;
let bucket = "root-owner-flag";
let key = "owner.txt";
let client = env.create_s3_client();
client.create_bucket().bucket(bucket).send().await?;
client
.put_object()
.bucket(bucket)
.key(key)
.body(aws_sdk_s3::primitives::ByteStream::from_static(b"owner access"))
.send()
.await?;
// Duplicate tag keys are ambiguous, so PutBucketTagging rejects them
// before they can reach stored metadata or the OPA tag-loading path.
let duplicate = client
.put_bucket_tagging()
.bucket(bucket)
.tagging(
Tagging::builder()
.tag_set(Tag::builder().key("department").value("finance").build()?)
.tag_set(Tag::builder().key("department").value("engineering").build()?)
.build()?,
)
.send()
.await
.expect_err("duplicate bucket tag keys must be rejected");
assert_eq!(duplicate.into_service_error().meta().code(), Some("InvalidTag"));
client
.put_bucket_tagging()
.bucket(bucket)
.tagging(
Tagging::builder()
.tag_set(Tag::builder().key("department").value("finance").build()?)
.build()?,
)
.send()
.await?;
let object = client.get_object().bucket(bucket).key(key).send().await?;
assert_eq!(object.body.collect().await?.into_bytes().as_ref(), b"owner access");
Ok(())
}
/// Issue #2036: Anonymous GetObject should succeed when bucket policy allows it
/// and no PublicAccessBlock configuration exists (ConfigNotFound).
#[tokio::test]
@@ -0,0 +1,512 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! What one stack answered, and how two answers are compared.
//!
//! An [`Observation`] keeps only what a client acts on: the HTTP status, the S3 error code, the
//! allowlisted response headers and a few typed fields decoded from the response. The error
//! message is kept for diagnostics and never compared: message wording is judged by the wire
//! corpus, not here. The comparator is pure so it can be tested without a server.
use serde::Serialize;
use std::collections::{BTreeMap, BTreeSet};
use std::path::{Path, PathBuf};
/// Response headers compared exactly (after [`normalized_header_value`]).
const HEADER_ALLOWLIST: [&str; 7] = [
"content-type",
"x-amz-version-id",
"etag",
"x-amz-delete-marker",
"content-range",
"accept-ranges",
"x-amz-request-charged",
];
/// Every response header with this prefix is compared as well.
const HEADER_ALLOWLIST_PREFIX: &str = "x-amz-server-side-encryption";
/// Placeholder for a field one side did not send.
const ABSENT: &str = "<absent>";
/// The lowercase header name if the comparator looks at this header.
pub(crate) fn compared_header(name: &str) -> Option<String> {
let name = name.to_ascii_lowercase();
(HEADER_ALLOWLIST.contains(&name.as_str()) || name.starts_with(HEADER_ALLOWLIST_PREFIX)).then_some(name)
}
/// A version ID is minted per process, so its value never matches across two servers; only
/// whether it is the literal `null` (an unversioned write) carries meaning.
fn normalized_header_value(name: &str, value: &str) -> String {
if name == "x-amz-version-id" && value != "null" {
"<version-id>".to_owned()
} else {
value.to_owned()
}
}
/// One request's outcome as a client sees it.
#[derive(Clone, Debug, Serialize)]
pub(crate) struct Observation {
/// The S3 operation, named as in the `s3s::S3` trait (`GetObject`).
pub op: &'static str,
/// Which request of the script this was (`normal`, `negative`, or a setup label).
pub case: &'static str,
/// HTTP status; 0 when no response arrived.
pub status: u16,
/// The S3 error code, or the SDK's error kind when there was no service error.
pub error_code: Option<String>,
/// Allowlisted headers only, lowercase names, normalized values.
pub headers: BTreeMap<String, String>,
/// Typed values decoded from the response body, never free text.
pub fields: BTreeMap<String, String>,
/// Diagnostic only; never compared.
#[serde(skip)]
pub message: Option<String>,
}
impl Observation {
/// Builds an observation, keeping only the allowlisted headers.
pub(crate) fn new<'h>(
op: &'static str,
case: &'static str,
status: u16,
error_code: Option<String>,
headers: impl IntoIterator<Item = (&'h str, &'h str)>,
) -> Self {
let headers = headers
.into_iter()
.filter_map(|(name, value)| compared_header(name).map(|name| (name, value)))
.map(|(name, value)| {
let value = normalized_header_value(&name, value);
(name, value)
})
.collect();
Self {
op,
case,
status,
error_code,
headers,
fields: BTreeMap::new(),
message: None,
}
}
pub(crate) fn with_field(mut self, name: &str, value: impl Into<String>) -> Self {
self.fields.insert(name.to_owned(), value.into());
self
}
pub(crate) fn is_success(&self) -> bool {
(200..300).contains(&self.status)
}
}
/// One field on which the two stacks answered differently.
#[derive(Clone, Debug, PartialEq, Eq, Serialize)]
pub(crate) struct Difference {
pub op: String,
pub case: String,
pub field: String,
pub a: String,
pub b: String,
}
/// Every difference between two transcripts of the same script, matched by `(op, case)`.
pub(crate) fn compare(a: &[Observation], b: &[Observation]) -> Vec<Difference> {
let mut differences = Vec::new();
let (index_a, duplicates_a) = index(a);
let (index_b, duplicates_b) = index(b);
for (op, case) in duplicates_a.union(&duplicates_b) {
let count = |side: &[Observation]| {
side.iter()
.filter(|observation| (observation.op, observation.case) == (*op, *case))
.count()
.to_string()
};
differences.push(difference(op, case, "duplicate_case", count(a), count(b)));
}
let keys: BTreeSet<_> = index_a.keys().chain(index_b.keys()).copied().collect();
for key @ (op, case) in keys {
match (index_a.get(&key), index_b.get(&key)) {
(Some(a), Some(b)) => compare_one(a, b, &mut differences),
(a, b) => {
let presence = |side: Option<&&Observation>| if side.is_some() { "present" } else { ABSENT }.to_owned();
differences.push(difference(op, case, "observation", presence(a), presence(b)));
}
}
}
differences
}
type Key = (&'static str, &'static str);
fn index(side: &[Observation]) -> (BTreeMap<Key, &Observation>, BTreeSet<Key>) {
let mut index = BTreeMap::new();
let mut duplicates = BTreeSet::new();
for observation in side {
let key = (observation.op, observation.case);
if index.insert(key, observation).is_some() {
duplicates.insert(key);
}
}
(index, duplicates)
}
fn difference(op: &str, case: &str, field: &str, a: String, b: String) -> Difference {
Difference {
op: op.to_owned(),
case: case.to_owned(),
field: field.to_owned(),
a,
b,
}
}
fn compare_one(a: &Observation, b: &Observation, differences: &mut Vec<Difference>) {
let mut field = |name: String, value_a: Option<&str>, value_b: Option<&str>| {
if value_a != value_b {
let show = |value: Option<&str>| value.unwrap_or(ABSENT).to_owned();
differences.push(difference(a.op, a.case, &name, show(value_a), show(value_b)));
}
};
field(
"status".to_owned(),
Some(a.status.to_string().as_str()),
Some(b.status.to_string().as_str()),
);
field("error_code".to_owned(), a.error_code.as_deref(), b.error_code.as_deref());
let headers: BTreeSet<_> = a.headers.keys().chain(b.headers.keys()).collect();
for name in headers {
field(
format!("header.{name}"),
a.headers.get(name).map(String::as_str),
b.headers.get(name).map(String::as_str),
);
}
let fields: BTreeSet<_> = a.fields.keys().chain(b.fields.keys()).collect();
for name in fields {
field(
format!("body.{name}"),
a.fields.get(name).map(String::as_str),
b.fields.get(name).map(String::as_str),
);
}
}
/// Operations observed on both sides.
pub(crate) fn compared_ops(a: &[Observation], b: &[Observation]) -> BTreeSet<&'static str> {
let ops_b: BTreeSet<_> = b.iter().map(|observation| observation.op).collect();
a.iter()
.map(|observation| observation.op)
.filter(|op| ops_b.contains(op))
.collect()
}
/// `(op, case)` pairs observed on both sides.
pub(crate) fn compared_cases(a: &[Observation], b: &[Observation]) -> usize {
let cases_b: BTreeSet<_> = b.iter().map(|observation| (observation.op, observation.case)).collect();
a.iter()
.map(|observation| (observation.op, observation.case))
.collect::<BTreeSet<_>>()
.intersection(&cases_b)
.count()
}
/// The JSON report written for every run.
#[derive(Debug, Serialize)]
pub(crate) struct Report {
pub schema: u32,
pub run: String,
pub stack_a: String,
pub stack_b: String,
pub region_a: Option<String>,
pub region_b: Option<String>,
pub compared_ops: usize,
pub compared_cases: usize,
pub differences: Vec<Difference>,
}
impl Report {
/// Writes the report as `<directory>/<run>.json` and returns its path.
pub(crate) fn write_to(&self, directory: &Path) -> std::io::Result<PathBuf> {
std::fs::create_dir_all(directory)?;
let path = directory.join(format!("{}.json", self.run));
let json = serde_json::to_vec_pretty(self).map_err(std::io::Error::other)?;
std::fs::write(&path, json)?;
Ok(path)
}
}
/// One line per difference, for a failing assertion.
pub(crate) fn render(differences: &[Difference]) -> String {
differences
.iter()
.map(|d| format!("({}, {}, {}, a={:?}, b={:?})", d.op, d.case, d.field, d.a, d.b))
.collect::<Vec<_>>()
.join("\n")
}
#[cfg(test)]
mod tests {
use super::*;
fn ok(op: &'static str) -> Observation {
Observation::new(op, "normal", 200, None, [("content-type", "application/xml")])
}
fn fields(differences: &[Difference]) -> Vec<&str> {
differences.iter().map(|d| d.field.as_str()).collect()
}
#[test]
fn identical_transcripts_have_no_difference() {
let a = vec![
ok("HeadBucket"),
ok("GetBucketLocation").with_field("location_constraint", "us-east-1"),
];
let b = a.clone();
assert_eq!(compare(&a, &b), Vec::new());
assert_eq!(compared_ops(&a, &b).len(), 2);
}
#[test]
fn a_status_difference_is_reported() {
let a = vec![ok("PutBucketAcl")];
let b = vec![Observation::new(
"PutBucketAcl",
"normal",
403,
None,
[("content-type", "application/xml")],
)];
assert_eq!(
compare(&a, &b),
vec![Difference {
op: "PutBucketAcl".into(),
case: "normal".into(),
field: "status".into(),
a: "200".into(),
b: "403".into(),
}]
);
}
#[test]
fn an_error_code_difference_is_reported() {
let a = vec![Observation::new("GetObject", "negative", 404, Some("NoSuchKey".into()), [])];
let b = vec![Observation::new(
"GetObject",
"negative",
404,
Some("NoSuchBucket".into()),
[],
)];
let differences = compare(&a, &b);
assert_eq!(fields(&differences), ["error_code"]);
assert_eq!((differences[0].a.as_str(), differences[0].b.as_str()), ("NoSuchKey", "NoSuchBucket"));
}
#[test]
fn an_error_code_on_one_side_only_is_reported() {
let a = vec![Observation::new("HeadBucket", "negative", 404, Some("NotFound".into()), [])];
let b = vec![Observation::new("HeadBucket", "negative", 404, None, [])];
let differences = compare(&a, &b);
assert_eq!(fields(&differences), ["error_code"]);
assert_eq!(differences[0].b, ABSENT);
}
#[test]
fn an_allowlisted_header_difference_is_reported() {
let a = vec![Observation::new("HeadObject", "normal", 200, None, [("ETag", "\"aa\"")])];
let b = vec![Observation::new("HeadObject", "normal", 200, None, [("etag", "\"bb\"")])];
assert_eq!(fields(&compare(&a, &b)), ["header.etag"]);
}
#[test]
fn a_header_sent_by_one_side_only_is_reported() {
let a = vec![Observation::new(
"DeleteObject",
"normal",
204,
None,
[("x-amz-delete-marker", "true")],
)];
let b = vec![Observation::new("DeleteObject", "normal", 204, None, [])];
let differences = compare(&a, &b);
assert_eq!(fields(&differences), ["header.x-amz-delete-marker"]);
assert_eq!((differences[0].a.as_str(), differences[0].b.as_str()), ("true", ABSENT));
}
#[test]
fn every_server_side_encryption_header_is_compared() {
let header = "x-amz-server-side-encryption-customer-key-md5";
let a = vec![Observation::new("PutObject", "sse-c", 200, None, [(header, "one")])];
let b = vec![Observation::new("PutObject", "sse-c", 200, None, [(header, "two")])];
assert_eq!(fields(&compare(&a, &b)), [format!("header.{header}")]);
}
#[test]
fn headers_outside_the_allowlist_are_not_compared() {
let a = vec![Observation::new(
"ListBuckets",
"normal",
200,
None,
[
("date", "Mon"),
("x-amz-request-id", "1"),
("server", "a"),
("content-length", "10"),
],
)];
let b = vec![Observation::new(
"ListBuckets",
"normal",
200,
None,
[
("date", "Tue"),
("x-amz-request-id", "2"),
("server", "b"),
("content-length", "11"),
],
)];
assert_eq!(compare(&a, &b), Vec::new());
assert!(a[0].headers.is_empty(), "non-allowlisted headers were kept: {:?}", a[0].headers);
}
#[test]
fn distinct_version_ids_are_not_a_difference() {
let a = vec![Observation::new(
"PutObject",
"normal",
200,
None,
[("x-amz-version-id", "0b1c")],
)];
let b = vec![Observation::new(
"PutObject",
"normal",
200,
None,
[("x-amz-version-id", "9f8e")],
)];
assert_eq!(compare(&a, &b), Vec::new());
}
#[test]
fn a_null_version_id_against_a_real_one_is_reported() {
let a = vec![Observation::new(
"PutObject",
"normal",
200,
None,
[("x-amz-version-id", "null")],
)];
let b = vec![Observation::new(
"PutObject",
"normal",
200,
None,
[("x-amz-version-id", "9f8e")],
)];
assert_eq!(fields(&compare(&a, &b)), ["header.x-amz-version-id"]);
}
#[test]
fn a_body_field_difference_is_reported() {
let a = vec![ok("GetBucketLocation").with_field("location_constraint", "us-east-1")];
let b = vec![ok("GetBucketLocation").with_field("location_constraint", "eu-west-1")];
let differences = compare(&a, &b);
assert_eq!(fields(&differences), ["body.location_constraint"]);
assert_eq!((differences[0].a.as_str(), differences[0].b.as_str()), ("us-east-1", "eu-west-1"));
}
#[test]
fn a_body_field_on_one_side_only_is_reported() {
let a = vec![ok("ListObjectsV2").with_field("keys", "[]")];
let b = vec![ok("ListObjectsV2")];
assert_eq!(fields(&compare(&a, &b)), ["body.keys"]);
}
#[test]
fn error_messages_are_never_compared() {
let mut a = Observation::new("GetObject", "negative", 404, Some("NoSuchKey".into()), []);
let mut b = a.clone();
a.message = Some("The specified key does not exist.".into());
b.message = Some("Object not found".into());
assert_eq!(compare(&[a], &[b]), Vec::new());
}
#[test]
fn a_case_observed_on_one_side_only_is_reported() {
let a = vec![ok("HeadBucket"), ok("ListBuckets")];
let b = vec![ok("HeadBucket")];
let differences = compare(&a, &b);
assert_eq!(fields(&differences), ["observation"]);
assert_eq!(differences[0].op, "ListBuckets");
assert_eq!(compared_ops(&a, &b).into_iter().collect::<Vec<_>>(), ["HeadBucket"]);
}
#[test]
fn a_case_recorded_twice_is_reported() {
let a = vec![ok("HeadBucket"), ok("HeadBucket")];
let b = vec![ok("HeadBucket")];
assert_eq!(fields(&compare(&a, &b)), ["duplicate_case"]);
}
#[test]
fn compared_ops_counts_operations_not_cases() {
let negative = Observation::new("HeadBucket", "negative", 404, Some("NotFound".into()), []);
let a = vec![ok("HeadBucket"), negative, ok("ListBuckets")];
let b = a.clone();
assert_eq!(compared_ops(&a, &b).len(), 2);
}
#[test]
fn the_report_lists_each_difference_as_op_field_a_b() {
let directory = std::env::temp_dir().join(format!("dual-stack-report-{}", uuid::Uuid::new_v4()));
let report = Report {
schema: 1,
run: "unit".into(),
stack_a: "legacy".into(),
stack_b: "legacy".into(),
region_a: None,
region_b: Some("eu-west-1".into()),
compared_ops: 1,
compared_cases: 1,
differences: compare(
&[ok("GetBucketLocation").with_field("location_constraint", "us-east-1")],
&[ok("GetBucketLocation").with_field("location_constraint", "eu-west-1")],
),
};
let path = report.write_to(&directory).expect("write the report");
let json: serde_json::Value = serde_json::from_slice(&std::fs::read(&path).expect("read the report")).expect("parse");
let _ = std::fs::remove_dir_all(&directory);
assert_eq!(path.file_name().and_then(|name| name.to_str()), Some("unit.json"));
assert_eq!(json["compared_ops"], 1);
assert_eq!(
json["differences"],
serde_json::json!([{
"op": "GetBucketLocation",
"case": "normal",
"field": "body.location_constraint",
"a": "us-east-1",
"b": "eu-west-1",
}])
);
}
}
@@ -0,0 +1,199 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! Dual-stack differential: one `rustfs` binary, two processes, one SDK script (rustfs/backlog#2740).
//!
//! Stack A always runs `RUSTFS_S3_STACK=legacy`. Stack B runs the value of `RUSTFS_E2E_STACK_B`
//! (default `legacy` until the gateway stack is assembled), and `RUSTFS_E2E_STACK_B_REGION`, when
//! set, starts B in another region so that a difference can be injected by hand. Both servers
//! answer the script in `scenarios`, which covers every operation of `impl S3 for FS`; `compare`
//! reports each `(op, field, a, b)` on which the answers differ, and every run writes its report
//! to `<target>/dual-stack/<run>.json`. Two processes, not two embedded servers: bucket metadata
//! is process-global, so two stacks in one process would read each other's buckets.
mod compare;
mod scenarios;
use crate::common::{RustFSTestEnvironment, init_logging, workspace_root};
use compare::{Difference, Observation, Report, compare, compared_cases, compared_ops, render};
use std::path::PathBuf;
use std::time::{SystemTime, UNIX_EPOCH};
type BoxError = Box<dyn std::error::Error + Send + Sync>;
const STACK_A: &str = "legacy";
const ENV_STACK_B: &str = "RUSTFS_E2E_STACK_B";
const ENV_STACK_B_REGION: &str = "RUSTFS_E2E_STACK_B_REGION";
/// Becomes `gateway` once rustfs/backlog#2734 T2.5 assembles the gateway stack.
const DEFAULT_STACK_B: &str = "legacy";
/// The region the negative control gives stack A explicitly, so an inherited `RUSTFS_REGION`
/// cannot erase the injected difference. Not `us-east-1`: that region answers GetBucketLocation
/// with an empty LocationConstraint and re-creating an owned bucket with 200, so it would not
/// report its own name and would add CreateBucket differences to the control.
const CONTROL_REGION_A: &str = "us-west-2";
const CONTROL_REGION_B: &str = "eu-west-1";
/// `normal` cases the legacy stack must answer with a 2xx. The comparison alone cannot tell a
/// script that exercises real behavior from one in which both stacks fail the same way. Five of
/// the 73 fail on legacy by design of this script: PutBucketReplication (no registered remote
/// target), GetBucketReplication (so no configuration), PutBucketNotificationConfiguration (no
/// notification target configured), RestoreObject (the object is not archived) and
/// GetObjectTorrent (legacy answers `NoSuchKey`).
const MIN_NORMAL_SUCCESSES: usize = 68;
struct PairConfig {
stack_b: String,
region_a: Option<String>,
region_b: Option<String>,
}
impl PairConfig {
fn from_env() -> Self {
Self {
stack_b: std::env::var(ENV_STACK_B).unwrap_or_else(|_| DEFAULT_STACK_B.to_owned()),
region_a: None,
region_b: std::env::var(ENV_STACK_B_REGION).ok(),
}
}
}
/// Starts one server on `stack`, overriding its region only when asked.
async fn start_stack(stack: &str, region: Option<&str>) -> Result<RustFSTestEnvironment, BoxError> {
let mut env = RustFSTestEnvironment::new().await?;
let mut vars = vec![("RUSTFS_S3_STACK", stack)];
if let Some(region) = region {
vars.push(("RUSTFS_REGION", region));
}
env.start_rustfs_server_without_cleanup_with_env(&vars).await?;
Ok(env)
}
/// Starts stack A and stack B side by side, each with its own port and data directory.
async fn spawn_pair(config: &PairConfig) -> Result<(RustFSTestEnvironment, RustFSTestEnvironment), BoxError> {
tokio::try_join!(
start_stack(STACK_A, config.region_a.as_deref()),
start_stack(&config.stack_b, config.region_b.as_deref())
)
}
fn report_directory() -> PathBuf {
let target = std::env::var_os("CARGO_TARGET_DIR").map_or_else(|| PathBuf::from("target"), PathBuf::from);
let target = if target.is_absolute() {
target
} else {
workspace_root().join(target)
};
target.join("dual-stack")
}
struct Run {
report: Report,
path: PathBuf,
a: Vec<Observation>,
}
/// Runs the script against both stacks concurrently and writes the report.
async fn run_differential(label: &str, config: &PairConfig) -> Run {
let (server_a, server_b) = spawn_pair(config).await.expect("start stack A and stack B");
let (a, b) = tokio::join!(Box::pin(scenarios::run(&server_a)), Box::pin(scenarios::run(&server_b)));
let millis = SystemTime::now()
.duration_since(UNIX_EPOCH)
.map_or(0, |elapsed| elapsed.as_millis());
let report = Report {
schema: 1,
run: format!("{label}-{millis}-{}", std::process::id()),
stack_a: STACK_A.to_owned(),
stack_b: config.stack_b.clone(),
region_a: config.region_a.clone(),
region_b: config.region_b.clone(),
compared_ops: compared_ops(&a, &b).len(),
compared_cases: compared_cases(&a, &b),
differences: compare(&a, &b),
};
let path = report.write_to(&report_directory()).expect("write the dual-stack report");
Run { report, path, a }
}
#[cfg(test)]
mod tests {
use super::*;
#[tokio::test]
async fn both_stacks_answer_every_operation_identically() {
init_logging();
let config = PairConfig::from_env();
let run = run_differential("self-comparison", &config).await;
let report = &run.report;
let mut observed: Vec<_> = run.a.iter().map(|observation| observation.op).collect();
observed.sort_unstable();
observed.dedup();
assert_eq!(
observed,
scenarios::OPERATIONS,
"the script must drive exactly the operations of impl S3 for FS"
);
assert_eq!(report.compared_ops, scenarios::OPERATIONS.len(), "report: {}", run.path.display());
assert!(
report.differences.is_empty(),
"stack {} and stack {} differ on {} of {} cases (report: {}):\n{}",
report.stack_a,
report.stack_b,
report.differences.len(),
report.compared_cases,
run.path.display(),
render(&report.differences)
);
let not_4xx: Vec<_> = run
.a
.iter()
.filter(|observation| observation.case == "negative" && !(400..500).contains(&observation.status))
.collect();
assert!(not_4xx.is_empty(), "negative cases must fail with a 4xx on {STACK_A}: {not_4xx:#?}");
let successes = run
.a
.iter()
.filter(|observation| observation.case == "normal" && observation.is_success())
.count();
assert!(
successes >= MIN_NORMAL_SUCCESSES,
"only {successes} normal cases succeeded on {STACK_A}: {:#?}",
run.a
);
}
#[tokio::test]
async fn differential_detects_an_injected_difference() {
init_logging();
let config = PairConfig {
stack_b: STACK_A.to_owned(),
region_a: Some(CONTROL_REGION_A.to_owned()),
region_b: Some(CONTROL_REGION_B.to_owned()),
};
let run = run_differential("injected-region", &config).await;
let expected = Difference {
op: "GetBucketLocation".to_owned(),
case: "normal".to_owned(),
field: "body.location_constraint".to_owned(),
a: CONTROL_REGION_A.to_owned(),
b: CONTROL_REGION_B.to_owned(),
};
assert!(
run.report.differences.contains(&expected),
"B started in {CONTROL_REGION_B} but GetBucketLocation was not reported (report: {}):\n{}",
run.path.display(),
render(&run.report.differences)
);
}
}
File diff suppressed because it is too large Load Diff
+255 -2
View File
@@ -21,12 +21,15 @@
#[cfg(test)]
mod tests {
use crate::common::{RustFSTestEnvironment, init_logging};
use crate::common::{RustFSTestEnvironment, admin_create_user, init_logging};
use aws_sdk_s3::Client;
use aws_sdk_s3::primitives::ByteStream;
use aws_sdk_s3::types::{BucketVersioningStatus, Delete, ObjectIdentifier, VersioningConfiguration};
use std::error::Error;
use tracing::info;
type TestResult = Result<(), Box<dyn Error + Send + Sync>>;
async fn create_bucket(client: &Client, bucket: &str) -> Result<(), Box<dyn Error + Send + Sync>> {
client.create_bucket().bucket(bucket).send().await?;
Ok(())
@@ -90,7 +93,7 @@ mod tests {
Ok(())
}
/// Duplicate and repeated slashes after a leading slash collapse MinIO-style.
/// Duplicate and repeated slashes after a leading slash collapse MinIO-style across single and bulk operations.
#[tokio::test]
async fn test_put_object_with_duplicate_slashes_normalized() -> Result<(), Box<dyn Error + Send + Sync>> {
init_logging();
@@ -144,8 +147,258 @@ mod tests {
"GET after DELETE with raw key must return HTTP 404, got {error:?}"
);
assert_bulk_delete_normalizes_keys(&client).await?;
assert_bulk_delete_authorizes_normalized_keys(&env, &client).await?;
assert_bulk_delete_preserves_version_identity(&client).await?;
env.stop_server();
info!("Test completed successfully");
Ok(())
}
async fn assert_bulk_delete_normalizes_keys(client: &Client) -> TestResult {
let bucket = "test-bulk-leading-slash";
create_bucket(client, bucket).await?;
let cases = [
("/foo/bar", "foo/bar"),
("//keyname", "keyname"),
("/dir///sub//file", "dir/sub/file"),
("/folder///", "folder/"),
("/literal%2F+名", "literal%2F+名"),
("plain", "plain"),
];
let invalid = ["///", "/dir/../keep", "dir//keep"];
client
.put_object()
.bucket(bucket)
.key("keep")
.body(ByteStream::from_static(b"keep"))
.send()
.await?;
client
.put_object()
.bucket(bucket)
.key("dir/keep")
.body(ByteStream::from_static(b"keep"))
.send()
.await?;
for quiet in [false, true] {
for (raw, _) in cases {
client
.put_object()
.bucket(bucket)
.key(raw)
.body(ByteStream::from_static(b""))
.send()
.await?;
}
let listing = client.list_objects_v2().bucket(bucket).send().await?;
let stored: Vec<_> = listing.contents().iter().filter_map(|object| object.key()).collect();
let mut expected: Vec<_> = cases
.iter()
.map(|(_, normalized)| *normalized)
.chain(["dir/keep", "keep"])
.collect();
expected.sort_unstable();
assert_eq!(stored, expected, "PUT must store the exact normalized targets before bulk deletion");
let objects = cases
.iter()
.map(|(raw, _)| *raw)
.chain(invalid)
.map(|key| ObjectIdentifier::builder().key(key).build())
.collect::<Result<Vec<_>, _>>()?;
let delete = Delete::builder().set_objects(Some(objects)).quiet(quiet).build()?;
// A second identical batch also pins idempotent deletion of absent normalized keys.
for _ in 0..2 {
let output = client.delete_objects().bucket(bucket).delete(delete.clone()).send().await?;
let errors: Vec<_> = output.errors().iter().map(|error| (error.key(), error.code())).collect();
assert_eq!(
errors,
invalid.map(|key| (Some(key), Some("InvalidArgument"))),
"unexpected per-key errors: {output:?}"
);
let deleted: Vec<_> = output.deleted().iter().filter_map(|object| object.key()).collect();
let expected = if quiet {
vec![]
} else {
cases.iter().map(|(raw, _)| *raw).collect()
};
assert_eq!(deleted, expected, "responses must retain request keys, including directory slashes");
}
let listing = client.list_objects_v2().bucket(bucket).send().await?;
let remaining: Vec<_> = listing.contents().iter().filter_map(|object| object.key()).collect();
assert_eq!(
remaining,
["dir/keep", "keep"],
"bulk delete must remove normalized objects without resolving dot segments"
);
}
Ok(())
}
async fn assert_bulk_delete_authorizes_normalized_keys(env: &RustFSTestEnvironment, admin: &Client) -> TestResult {
let bucket = "test-bulk-slash-policy";
let user = "bulk-slash-user";
admin_create_user(env, user, "bulkSlashTestSecret123!").await?;
create_bucket(admin, bucket).await?;
for key in ["allowed/item", "protected/item"] {
admin
.put_object()
.bucket(bucket)
.key(key)
.body(ByteStream::from_static(b"keep"))
.send()
.await?;
}
let policy = serde_json::json!({
"Version": "2012-10-17",
"Statement": [
{"Effect": "Allow", "Principal": {"AWS": [user]}, "Action": ["s3:DeleteObject"],
"Resource": [format!("arn:aws:s3:::{bucket}/*")]},
{"Effect": "Deny", "Principal": {"AWS": [user]}, "Action": ["s3:DeleteObject"],
"Resource": [format!("arn:aws:s3:::{bucket}/protected/*")]}
]
});
admin
.put_bucket_policy()
.bucket(bucket)
.policy(policy.to_string())
.send()
.await?;
let client = env.create_s3_client_with_credentials(user, "bulkSlashTestSecret123!");
let output = client
.delete_objects()
.bucket(bucket)
.delete(
Delete::builder()
.objects(ObjectIdentifier::builder().key("/allowed///item").build()?)
.objects(ObjectIdentifier::builder().key("/protected///item").build()?)
.build()?,
)
.send()
.await?;
assert_eq!(output.deleted().len(), 1, "allowed normalized key must be deleted: {output:?}");
assert_eq!(output.deleted()[0].key(), Some("/allowed///item"));
assert_eq!(output.errors().len(), 1, "normalized resource denial must apply: {output:?}");
assert_eq!(output.errors()[0].key(), Some("/protected///item"));
assert_eq!(output.errors()[0].code(), Some("AccessDenied"));
let listing = admin.list_objects_v2().bucket(bucket).send().await?;
let remaining: Vec<_> = listing.contents().iter().filter_map(|object| object.key()).collect();
assert_eq!(remaining, ["protected/item"], "only the authorized object may be removed");
let protected = admin.get_object().bucket(bucket).key("protected/item").send().await?;
assert_eq!(protected.body.collect().await?.into_bytes().as_ref(), b"keep");
Ok(())
}
async fn assert_bulk_delete_preserves_version_identity(client: &Client) -> TestResult {
let bucket = "test-bulk-slash-versions";
let raw = "/history///file";
let normalized = "history/file";
create_bucket(client, bucket).await?;
client
.put_bucket_versioning()
.bucket(bucket)
.versioning_configuration(
VersioningConfiguration::builder()
.status(BucketVersioningStatus::Enabled)
.build(),
)
.send()
.await?;
let first = client
.put_object()
.bucket(bucket)
.key(raw)
.body(ByteStream::from_static(b"old"))
.send()
.await?;
let first_id = first.version_id().ok_or("first PUT omitted version ID")?;
let latest = client
.put_object()
.bucket(bucket)
.key(raw)
.body(ByteStream::from_static(b"latest"))
.send()
.await?;
let latest_id = latest.version_id().ok_or("latest PUT omitted version ID")?;
let invalid = client
.delete_objects()
.bucket(bucket)
.delete(
Delete::builder()
.objects(ObjectIdentifier::builder().key("///").build()?)
.build()?,
)
.send()
.await?;
assert!(invalid.deleted().is_empty(), "slash-only keys must not delete a versioned object");
assert_eq!(
invalid.errors().len(),
1,
"slash-only keys must remain invalid in versioned buckets: {invalid:?}"
);
assert_eq!(invalid.errors()[0].key(), Some("///"));
assert_eq!(invalid.errors()[0].code(), Some("InvalidArgument"));
let deleted = client
.delete_objects()
.bucket(bucket)
.delete(
Delete::builder()
.objects(ObjectIdentifier::builder().key(raw).version_id(first_id).build()?)
.build()?,
)
.send()
.await?;
assert!(deleted.errors().is_empty(), "historical delete failed: {deleted:?}");
assert_eq!(deleted.deleted().len(), 1);
assert_eq!(deleted.deleted()[0].key(), Some(raw));
assert_eq!(deleted.deleted()[0].version_id(), Some(first_id));
let listing = client.list_object_versions().bucket(bucket).send().await?;
assert_eq!(listing.versions().len(), 1, "only the latest data version should remain: {listing:?}");
assert_eq!(listing.versions()[0].version_id(), Some(latest_id));
let deleted = client
.delete_objects()
.bucket(bucket)
.delete(
Delete::builder()
.objects(ObjectIdentifier::builder().key(raw).build()?)
.build()?,
)
.send()
.await?;
assert!(deleted.errors().is_empty(), "delete marker creation failed: {deleted:?}");
assert_eq!(deleted.deleted().len(), 1);
assert_eq!(deleted.deleted()[0].key(), Some(raw));
assert_eq!(deleted.deleted()[0].delete_marker(), Some(true));
let marker = deleted.deleted()[0]
.delete_marker_version_id()
.ok_or("missing delete marker ID")?;
let listing = client.list_object_versions().bucket(bucket).send().await?;
assert_eq!(listing.delete_markers().len(), 1);
assert_eq!(listing.delete_markers()[0].key(), Some(normalized));
assert_eq!(listing.delete_markers()[0].version_id(), Some(marker));
let deleted = client
.delete_objects()
.bucket(bucket)
.delete(
Delete::builder()
.objects(ObjectIdentifier::builder().key(raw).version_id(marker).build()?)
.build()?,
)
.send()
.await?;
assert!(deleted.errors().is_empty(), "explicit marker removal failed: {deleted:?}");
assert_eq!(deleted.deleted().len(), 1);
assert_eq!(deleted.deleted()[0].key(), Some(raw));
assert_eq!(deleted.deleted()[0].version_id(), Some(marker));
let listing = client.list_object_versions().bucket(bucket).send().await?;
assert!(listing.delete_markers().is_empty());
let restored = client.get_object().bucket(bucket).key(normalized).send().await?;
assert_eq!(restored.version_id(), Some(latest_id));
assert_eq!(restored.body.collect().await?.into_bytes().as_ref(), b"latest");
Ok(())
}
}
+6
View File
@@ -31,6 +31,12 @@ pub mod chaos;
#[cfg(test)]
pub mod fake_s3_target;
// Dual-stack differential (rustfs/backlog#2740): one binary, two processes on
// different RUSTFS_S3_STACK values, the same SDK script for every S3 operation,
// compared on status, error code, allowlisted headers and typed body fields.
#[cfg(test)]
mod dual_stack_differential;
// On-demand migration (backlog#2147): shared two-server environment, admin
// wrappers, and the harness self-test (backlog#2151). Behavior scenarios are
// added by later ODM tasks.
+78 -1
View File
@@ -32,7 +32,9 @@
//! Advisory: <https://github.com/rustfs/rustfs/security/advisories/GHSA-3p3x-734c-h5vx>
use crate::common::rustfs_binary_path_with_features;
use crate::common::{AdminTransport, admin_add_canned_policy_via, admin_attach_user_policy_via, admin_create_user_via};
use crate::common::{
AdminTransport, admin_add_canned_policy_via, admin_attach_user_policy_via, admin_create_user_via, signed_s3_request,
};
use crate::common::{FAST_DATA_USAGE_SCANNER_ENV, admin_request};
use crate::protocols::test_env::{DEFAULT_ACCESS_KEY, DEFAULT_SECRET_KEY, ProtocolTestEnvironment};
use anyhow::Result;
@@ -774,6 +776,81 @@ async fn test_webdav_core_operations_direct() -> Result<()> {
test_webdav_core_operations().await
}
#[tokio::test]
async fn test_webdav_content_type_directory_marker() -> Result<()> {
let env = ProtocolTestEnvironment::new().map_err(|e| anyhow::anyhow!("{}", e))?;
let admin_base_url = format!("http://{}", S3_TEST_ADDRESS);
let binary_path = rustfs_binary_path_with_features(Some("webdav"));
let mut server_process = Command::new(&binary_path)
.arg("--address")
.arg(S3_TEST_ADDRESS)
.env("RUSTFS_CONSOLE_ENABLE", "false")
.env("RUSTFS_WEBDAV_ENABLE", "true")
.env("RUSTFS_WEBDAV_ADDRESS", WEBDAV_ADDRESS)
.env("RUSTFS_WEBDAV_TLS_ENABLED", "false")
.arg(&env.temp_dir)
.spawn()?;
let result = async {
ProtocolTestEnvironment::wait_for_port_ready(WEBDAV_PORT, 30)
.await
.map_err(|e| anyhow::anyhow!("{}", e))?;
let client = create_client();
let bucket_name = "webdav-content-type-bucket";
let directory_name = "content-type-directory";
let webdav_base_url = format!("http://{}", WEBDAV_ADDRESS);
let mkcol_bucket = client
.request(
reqwest::Method::from_bytes(b"MKCOL").expect("MKCOL is a valid HTTP method"),
format!("{}/{}", webdav_base_url, bucket_name),
)
.header("Authorization", basic_auth_header())
.send()
.await?;
anyhow::ensure!(
mkcol_bucket.status().is_success() || mkcol_bucket.status().as_u16() == 201,
"MKCOL bucket should succeed, got: {}",
mkcol_bucket.status()
);
let mkcol_directory = client
.request(
reqwest::Method::from_bytes(b"MKCOL").expect("MKCOL is a valid HTTP method"),
format!("{}/{}/{}", webdav_base_url, bucket_name, directory_name),
)
.header("Authorization", basic_auth_header())
.send()
.await?;
anyhow::ensure!(
mkcol_directory.status().is_success() || mkcol_directory.status().as_u16() == 201,
"MKCOL directory should succeed, got: {}",
mkcol_directory.status()
);
let head_url = format!("{}/{}/{}/", admin_base_url, bucket_name, directory_name);
let head = signed_s3_request(http::Method::HEAD, &head_url, None, None, DEFAULT_ACCESS_KEY, DEFAULT_SECRET_KEY)
.await
.map_err(|error| anyhow::anyhow!("{error}"))?;
anyhow::ensure!(head.status().is_success(), "HEAD directory marker should succeed, got: {}", head.status());
let content_type = head
.headers()
.get("content-type")
.and_then(|value| value.to_str().ok())
.unwrap_or_default();
anyhow::ensure!(
content_type == "application/x-directory",
"WebDAV directory marker must preserve its content type, got: {content_type}"
);
Ok(())
}
.await;
let _ = server_process.kill().await;
let _ = server_process.wait().await;
result
}
/// Extract the numeric value of the first occurrence of a DAV property in a
/// PROPFIND multistatus body. Namespace-prefix agnostic; a self-closing
/// (unreported) property yields None.
+2 -1
View File
@@ -141,6 +141,7 @@ rustfs-lifecycle.workspace = true
rustfs-s3-client = { workspace = true }
rustfs-s3-types = { workspace = true }
rustfs-data-usage.workspace = true
rustfs-gateway-types.workspace = true
rustfs-object-capacity.workspace = true
rustfs-object-data-cache = { workspace = true, features = ["runtime-memory"] }
arc-swap.workspace = true
@@ -159,7 +160,6 @@ serde = { workspace = true, features = ["derive"] }
time = { workspace = true, features = ["parsing", "formatting", "macros", "serde"] }
bytesize.workspace = true
serde_json = { workspace = true, features = ["raw_value"] }
s3s = { workspace = true, features = ["minio"] }
http.workspace = true
opentelemetry.workspace = true
http-body = { workspace = true }
@@ -253,6 +253,7 @@ windows-sys = { workspace = true, features = [
windows-sys = { workspace = true, features = ["Win32_System_Ioctl"] }
[dev-dependencies]
s3s = { workspace = true, features = ["minio"] }
aws-smithy-async.workspace = true
rustfs-s3-ops.workspace = true
tokio = { workspace = true, features = ["rt-multi-thread", "macros", "test-util", "fs"] }
+8 -6
View File
@@ -66,6 +66,8 @@ pub mod bucket {
}
pub mod bucket_lifecycle_ops {
pub use rustfs_lifecycle::{RestoreOutputLocation, RestoreRequest, RestoreS3Location, RestoreTag, RestoreTagging};
pub use crate::bucket::lifecycle::bucket_lifecycle_ops::{
ExpiryState, LifecycleOps, ManualTransitionCancelCheck, ManualTransitionProgressSink,
ManualTransitionQueueSnapshot, ManualTransitionRunExecution, ManualTransitionRunOptions,
@@ -137,7 +139,7 @@ pub mod bucket {
pub use crate::bucket::lifecycle::lifecycle::{
Event, ExpirationOptions, IlmAction, LIFECYCLE_MALFORMED_XML_ERROR_KIND, Lifecycle, LifecycleCalculate,
ObjectOpts, RuleValidate, TRANSITION_COMPLETE, TRANSITION_PENDING, TransitionOptions, expected_expiry_time,
object_opts_from_object_info,
format_lifecycle_date, object_opts_from_object_info, parse_lifecycle_date,
};
}
@@ -240,7 +242,7 @@ pub mod bucket {
}
pub mod object_lock {
pub use crate::bucket::object_lock::{ObjectLockApi, ObjectLockStatusExt};
pub use crate::bucket::object_lock::ObjectLockApi;
pub mod types {
pub use crate::bucket::object_lock::types::{
@@ -335,9 +337,9 @@ pub mod bucket {
pub mod utils {
pub use crate::bucket::utils::{
check_bucket_and_object_names, check_list_objs_args, check_object_name_for_length_and_slash,
check_valid_bucket_name_strict, deserialize, has_bad_path_component, has_xml_invalid_control_characters,
is_meta_bucketname, is_valid_object_prefix, serialize,
BucketConfigXml, BucketConfigXmlError, check_bucket_and_object_names, check_list_objs_args,
check_object_name_for_length_and_slash, check_valid_bucket_name_strict, deserialize, has_bad_path_component,
has_xml_invalid_control_characters, is_meta_bucketname, is_valid_object_prefix, serialize,
};
}
@@ -364,7 +366,7 @@ pub mod capacity {
}
pub mod object_api_utils {
pub use crate::object_api::object_api_utils::{ObjReaderFn, PutObjReader, get_raw_etag, new_getobjectreader, to_s3s_etag};
pub use crate::object_api::object_api_utils::{ObjReaderFn, PutObjReader, get_raw_etag, new_getobjectreader};
}
pub mod cluster {
@@ -88,12 +88,14 @@ use rustfs_filemeta::{
FileInfo, FileInfoOpts, NULL_VERSION_ID, RestoreStatus, RestoreStatusOps, TRANSITION_COMPLETE, get_file_info,
is_restored_object_on_disk,
};
use rustfs_gateway_types::persistence::PersistedObjectLockConfiguration as ObjectLockConfiguration;
use rustfs_gateway_types::persistence::{PersistedLifecycleConfiguration, PersistedLifecycleFilter};
use rustfs_lifecycle::{RestoreRequest, RestoreS3Location};
use rustfs_lock::{NamespaceLockGuard, error::LockError};
use rustfs_scanner_metrics::metrics::{
IlmAction, Metrics, ScannerLifecycleExpiryStateUpdate, ScannerLifecycleTransitionStateUpdate, global_metrics,
};
use rustfs_utils::{get_env_i64, get_env_usize, path::encode_dir_object, string::parse_bool};
use s3s::dto::{BucketLifecycleConfiguration, ExpirationStatus, ObjectLockConfiguration, RestoreRequest, RestoreRequestType};
use sha2::{Digest, Sha256};
use std::any::Any;
use std::collections::{BTreeMap, HashMap, HashSet};
@@ -300,7 +302,7 @@ impl LifecycleSys {
Arc::new(Self)
}
pub async fn get(&self, bucket: &str) -> Option<BucketLifecycleConfiguration> {
pub async fn get(&self, bucket: &str) -> Option<PersistedLifecycleConfiguration> {
match metadata_boundary::get_lifecycle_config(bucket).await {
Ok((lc, _)) => Some(lc),
Err(Error::ConfigNotFound) => None,
@@ -3557,7 +3559,7 @@ async fn stale_upload_lifecycle_due(
Err(_) => return None,
};
let has_size_filter = |filter: &s3s::dto::LifecycleRuleFilter| {
let has_size_filter = |filter: &PersistedLifecycleFilter| {
filter.object_size_greater_than.is_some()
|| filter.object_size_less_than.is_some()
|| filter
@@ -3566,7 +3568,7 @@ async fn stale_upload_lifecycle_due(
.is_some_and(|and| and.object_size_greater_than.is_some() || and.object_size_less_than.is_some())
};
let needs_size = lifecycle.rules.iter().any(|rule| {
rule.status.as_str() != ExpirationStatus::DISABLED
rule.status.as_str() != "Disabled"
&& rule.abort_incomplete_multipart_upload.is_some()
&& rule.filter.as_ref().is_some_and(has_size_filter)
});
@@ -4000,7 +4002,7 @@ pub fn init_background_stale_multipart_upload_cleanup(api: Arc<ECStore>) {
});
}
pub async fn validate_transition_tier(lc: &BucketLifecycleConfiguration) -> Result<(), std::io::Error> {
pub async fn validate_transition_tier(lc: &PersistedLifecycleConfiguration) -> Result<(), std::io::Error> {
for rule in &lc.rules {
if let Some(transitions) = &rule.transitions {
for transition in transitions {
@@ -4117,8 +4119,9 @@ pub(crate) async fn enqueue_immediate_expiry(api: Arc<ECStore>, oi: &ObjectInfo,
.map(lifecycle::object_opts_from_object_info)
.collect::<Vec<ObjectOpts>>();
let lock_config = configs.object_lock;
let evaluator_lock_config = lock_config.clone();
let events = match Evaluator::new(lifecycle)
.with_lock_retention(lock_config.clone())
.with_lock_retention(evaluator_lock_config)
.eval(&object_opts)
.await
{
@@ -4598,19 +4601,26 @@ pub async fn enqueue_transition_for_existing_objects_scoped(
}
}
fn lifecycle_rule_has_date_expiration(lc: &BucketLifecycleConfiguration, rule_id: &str) -> bool {
fn lifecycle_rule_has_date_expiration(lc: &PersistedLifecycleConfiguration, rule_id: &str) -> bool {
lc.rules.iter().any(|rule| {
rule.status == ExpirationStatus::from_static(ExpirationStatus::ENABLED)
rule.status == "Enabled"
&& rule.id.as_deref() == Some(rule_id)
&& rule.expiration.as_ref().is_some_and(|expiration| expiration.date.is_some())
})
}
fn should_defer_date_expiry_for_recent_config_update(lc: &BucketLifecycleConfiguration, now: OffsetDateTime) -> bool {
lc.expiry_updated_at.as_ref().is_some_and(|updated_at| {
let updated_at = OffsetDateTime::from(updated_at.clone());
now.unix_timestamp().saturating_sub(updated_at.unix_timestamp()) < DATE_EXPIRY_EXISTING_OBJECTS_GRACE_SECS
})
fn should_defer_date_expiry_for_recent_config_update(lc: &PersistedLifecycleConfiguration, now: OffsetDateTime) -> bool {
lc.expiry_updated_at
.as_ref()
.is_some_and(|updated_at| match rustfs_lifecycle::parse_lifecycle_date(updated_at) {
Ok(updated_at) => {
now.unix_timestamp().saturating_sub(updated_at.unix_timestamp()) < DATE_EXPIRY_EXISTING_OBJECTS_GRACE_SECS
}
Err(error) => {
warn!(error = %error, "Deferring date expiry with an unreadable update timestamp");
true
}
})
}
async fn apply_existing_object_expiry(
@@ -4630,7 +4640,7 @@ async fn apply_existing_object_expiry(
struct ExistingObjectExpiryContext<'a> {
api: Arc<ECStore>,
bucket: &'a str,
lc: Arc<BucketLifecycleConfiguration>,
lc: Arc<PersistedLifecycleConfiguration>,
lock_config: Option<Arc<ObjectLockConfiguration>>,
bucket_incarnation_id: Uuid,
src: &'a LcEventSrc,
@@ -4650,8 +4660,9 @@ async fn enqueue_expiry_for_existing_object_group(
.iter()
.map(lifecycle::object_opts_from_object_info)
.collect::<Vec<ObjectOpts>>();
let evaluator_lock_config = context.lock_config.clone();
let events = match Evaluator::new(context.lc.clone())
.with_lock_retention(context.lock_config.clone())
.with_lock_retention(evaluator_lock_config)
.eval(&object_opts)
.await
{
@@ -4845,7 +4856,7 @@ fn manual_transition_version_marker(oi: &ObjectInfo) -> String {
async fn enqueue_transition_with_lifecycle_report(
api: Option<Arc<ECStore>>,
oi: &ObjectInfo,
lc: &BucketLifecycleConfiguration,
lc: &PersistedLifecycleConfiguration,
src: &LcEventSrc,
options: &ManualTransitionRunOptions,
report: &mut ManualTransitionRunReport,
@@ -4911,7 +4922,7 @@ async fn enqueue_transition_with_lifecycle_report(
false
}
async fn enqueue_transition_with_lifecycle(oi: &ObjectInfo, lc: &BucketLifecycleConfiguration, src: &LcEventSrc) -> bool {
async fn enqueue_transition_with_lifecycle(oi: &ObjectInfo, lc: &PersistedLifecycleConfiguration, src: &LcEventSrc) -> bool {
let event = lc.eval(&oi.to_lifecycle_opts()).await;
match event.action {
IlmAction::TransitionAction | IlmAction::TransitionVersionAction => {
@@ -5425,12 +5436,8 @@ pub async fn post_restore_opts(version_id: &str, bucket: &str, object: &str) ->
})
}
fn select_restore_s3_location(rreq: &RestoreRequest) -> Result<Option<&s3s::dto::S3Location>, std::io::Error> {
if rreq
.type_
.as_ref()
.is_none_or(|type_| type_.as_str() != RestoreRequestType::SELECT)
{
fn select_restore_s3_location(rreq: &RestoreRequest) -> Result<Option<&RestoreS3Location>, std::io::Error> {
if rreq.type_.as_ref().is_none_or(|type_| type_.as_str() != "SELECT") {
return Ok(None);
}
let output_location = rreq
@@ -5472,21 +5479,20 @@ pub async fn put_restore_opts(
// the OutputLocation metadata instead of a write to `OutputLocation.S3`
// (backlog#1341). RestoreObject rejects SELECT at the API boundary; this
// is the fail-closed backstop for any other caller.
if rreq
.type_
.as_ref()
.is_some_and(|type_| type_.as_str() == RestoreRequestType::SELECT)
{
if rreq.type_.as_ref().is_some_and(|type_| type_.as_str() == "SELECT") {
return Err(std::io::Error::other("SELECT restore requests are not supported"));
}
for (k, v) in oi.user_defined.iter() {
meta.insert(k.to_string(), v.clone());
}
// The fleet capability gate guarantees physical copy-back support. Keep
// the completed transition identity for orphan recovery and remote cleanup.
rustfs_utils::http::metadata_compat::remove_str(&mut meta, rustfs_utils::http::metadata_compat::SUFFIX_RESTORE_OPERATION_ID);
rustfs_utils::http::metadata_compat::remove_str(&mut meta, rustfs_utils::http::metadata_compat::SUFFIX_RESTORE_WORKER_LOCK);
if !oi.user_tags.is_empty() {
meta.insert(AMZ_OBJECT_TAGGING.to_string(), (*oi.user_tags).clone());
}
// Finalization publishes completion and its expiry under the object lock.
// The local copy-back commit must not make unfinished work eligible for expiry.
if let Some(expires) = oi.expires {
meta.insert(
"expires".to_string(),
@@ -5495,12 +5501,11 @@ pub async fn put_restore_opts(
.map_err(std::io::Error::other)?,
);
}
let restore_expiry = lifecycle::expected_expiry_time(OffsetDateTime::now_utc(), rreq.days.unwrap_or(1));
meta.insert(
metadata_keys::RESTORE.to_string(),
RestoreStatus {
is_restore_in_progress: Some(false),
restore_expiry_date: Some(restore_expiry),
is_restore_in_progress: Some(true),
restore_expiry_date: None,
}
.to_string(),
);
@@ -5514,6 +5519,7 @@ pub async fn put_restore_opts(
// computed MD5 is not the object's public plaintext ETag.
preserve_etag: oi.etag.clone(),
shard_integrity_write_mode: Some(oi.shard_integrity_write_mode()),
restore_copyback: true,
..Default::default()
})
}
@@ -5532,6 +5538,18 @@ impl LifecycleOps for ObjectInfo {
if self.transitioned_object.status != lifecycle::TRANSITION_COMPLETE {
return false;
}
if rustfs_utils::http::metadata_compat::get_consistent_str(
&self.user_defined,
rustfs_utils::http::metadata_compat::SUFFIX_RESTORE_COPYBACK,
) == Some(rustfs_utils::http::metadata_compat::RESTORE_COPYBACK_PROTOCOL_V1)
&& self
.user_defined
.get(metadata_keys::RESTORE)
.and_then(|value| rustfs_filemeta::parse_restore_obj_status(value).ok())
.is_some_and(|status| status.is_restore_in_progress == Some(true))
{
return false;
}
!is_restored_object_on_disk(&self.user_defined)
}
}
@@ -5543,25 +5561,25 @@ pub trait RestoreRequestOps {
impl RestoreRequestOps for RestoreRequest {
fn validate(&self, _api: Arc<ECStore>) -> Result<(), std::io::Error> {
// SELECT type requires select_parameters, and vice versa
if self.type_.as_ref().is_none_or(|t| t.as_str() != RestoreRequestType::SELECT) && self.select_parameters.is_some() {
if self.type_.as_ref().is_none_or(|t| t.as_str() != "SELECT") && self.select_parameters.is_some() {
return Err(std::io::Error::other("Select parameters can only be specified with SELECT request type"));
}
if let Some(type_) = &self.type_
&& type_.as_str() == RestoreRequestType::SELECT
&& type_.as_str() == "SELECT"
&& self.select_parameters.is_none()
{
return Err(std::io::Error::other("SELECT restore request requires select parameters to be specified"));
}
// OutputLocation is only valid for SELECT requests
if self.type_.as_ref().is_none_or(|t| t.as_str() != RestoreRequestType::SELECT) && self.output_location.is_some() {
if self.type_.as_ref().is_none_or(|t| t.as_str() != "SELECT") && self.output_location.is_some() {
return Err(std::io::Error::other("OutputLocation can only be specified with SELECT request type"));
}
select_restore_s3_location(self)?;
// Days must not be specified with SELECT requests
if let Some(type_) = &self.type_
&& type_.as_str() == RestoreRequestType::SELECT
&& type_.as_str() == "SELECT"
&& self.days.is_some_and(|d| d > 0)
{
return Err(std::io::Error::other("Days cannot be specified with SELECT restore request"));
@@ -5603,7 +5621,7 @@ pub fn lifecycle_version_delete_target(oi: &ObjectInfo) -> Option<ObjectToDelete
}
pub async fn eval_action_from_lifecycle(
lc: &BucketLifecycleConfiguration,
lc: &PersistedLifecycleConfiguration,
lock_config: Option<&ObjectLockConfiguration>,
oi: &ObjectInfo,
) -> lifecycle::Event {
@@ -6308,14 +6326,19 @@ mod tests {
use rustfs_data_usage::TierStats;
use rustfs_filemeta::metadata_keys;
use rustfs_filemeta::{FileInfo, FileMeta};
use rustfs_gateway_types::dto::MetadataEntry;
use rustfs_gateway_types::persistence::{
PersistedDefaultRetention as DefaultRetention, PersistedObjectLockConfiguration as ObjectLockConfiguration,
PersistedObjectLockRule as ObjectLockRule,
};
use rustfs_gateway_types::persistence::{
PersistedLifecycleConfiguration, PersistedLifecycleExpiration, PersistedLifecycleRule,
PersistedNoncurrentVersionExpiration, PersistedTransition,
};
use rustfs_lifecycle::{RestoreOutputLocation as OutputLocation, RestoreRequest, RestoreS3Location as S3Location};
#[cfg(feature = "test-util")]
use rustfs_s3_client::transition_api::ReaderImpl;
use rustfs_scanner_metrics::metrics::{IlmAction, global_metrics};
use s3s::dto::{
BucketLifecycleConfiguration, DefaultRetention, ExpirationStatus, LifecycleExpiration, LifecycleRule, MetadataEntry,
NoncurrentVersionExpiration, ObjectLockConfiguration, ObjectLockEnabled, ObjectLockRetentionMode, ObjectLockRule,
OutputLocation, RestoreRequest, RestoreRequestType, S3Location, Timestamp, Transition, TransitionStorageClass,
};
use serial_test::serial;
use sha2::{Digest, Sha256};
use std::collections::HashMap;
@@ -9341,7 +9364,7 @@ mod tests {
output_location,
select_parameters: None,
tier: None,
type_: Some(RestoreRequestType::from_static(RestoreRequestType::SELECT)),
type_: Some("SELECT".to_owned()),
}
}
@@ -9658,12 +9681,12 @@ mod tests {
#[test]
fn lifecycle_rule_has_date_expiration_detects_enabled_date_rule() {
let lc = BucketLifecycleConfiguration {
let lc = PersistedLifecycleConfiguration {
expiry_updated_at: None,
rules: vec![LifecycleRule {
status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
expiration: Some(LifecycleExpiration {
date: Some(Timestamp::from(OffsetDateTime::now_utc())),
rules: vec![PersistedLifecycleRule {
status: "Enabled".to_owned(),
expiration: Some(PersistedLifecycleExpiration {
date: Some(lifecycle::format_lifecycle_date(OffsetDateTime::now_utc()).unwrap()),
..Default::default()
}),
id: Some("rule-date".to_string()),
@@ -10180,14 +10203,15 @@ mod tests {
#[test]
fn should_defer_date_expiry_for_recent_config_update_respects_grace_window() {
let now = OffsetDateTime::now_utc();
let recent = BucketLifecycleConfiguration {
expiry_updated_at: Some(Timestamp::from(now - time::Duration::seconds(1))),
let recent = PersistedLifecycleConfiguration {
expiry_updated_at: Some(lifecycle::format_lifecycle_date(now - time::Duration::seconds(1)).unwrap()),
rules: Vec::new(),
};
let stale = BucketLifecycleConfiguration {
expiry_updated_at: Some(Timestamp::from(
now - time::Duration::seconds(DATE_EXPIRY_EXISTING_OBJECTS_GRACE_SECS + 1),
)),
let stale = PersistedLifecycleConfiguration {
expiry_updated_at: Some(
lifecycle::format_lifecycle_date(now - time::Duration::seconds(DATE_EXPIRY_EXISTING_OBJECTS_GRACE_SECS + 1))
.unwrap(),
),
rules: Vec::new(),
};
@@ -10315,12 +10339,12 @@ mod tests {
assert_eq!(deleted.version_id, None);
}
fn expired_delete_marker_lifecycle() -> BucketLifecycleConfiguration {
BucketLifecycleConfiguration {
fn expired_delete_marker_lifecycle() -> PersistedLifecycleConfiguration {
PersistedLifecycleConfiguration {
expiry_updated_at: None,
rules: vec![LifecycleRule {
status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
expiration: Some(LifecycleExpiration {
rules: vec![PersistedLifecycleRule {
status: "Enabled".to_owned(),
expiration: Some(PersistedLifecycleExpiration {
expired_object_delete_marker: Some(true),
..Default::default()
}),
@@ -10336,12 +10360,12 @@ mod tests {
}
}
fn latest_expiration_lifecycle() -> BucketLifecycleConfiguration {
BucketLifecycleConfiguration {
fn latest_expiration_lifecycle() -> PersistedLifecycleConfiguration {
PersistedLifecycleConfiguration {
expiry_updated_at: None,
rules: vec![LifecycleRule {
status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
expiration: Some(LifecycleExpiration {
rules: vec![PersistedLifecycleRule {
status: "Enabled".to_owned(),
expiration: Some(PersistedLifecycleExpiration {
days: Some(1),
..Default::default()
}),
@@ -10357,12 +10381,12 @@ mod tests {
}
}
fn current_and_noncurrent_expiration_lifecycle() -> BucketLifecycleConfiguration {
BucketLifecycleConfiguration {
fn current_and_noncurrent_expiration_lifecycle() -> PersistedLifecycleConfiguration {
PersistedLifecycleConfiguration {
expiry_updated_at: None,
rules: vec![LifecycleRule {
status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
expiration: Some(LifecycleExpiration {
rules: vec![PersistedLifecycleRule {
status: "Enabled".to_owned(),
expiration: Some(PersistedLifecycleExpiration {
days: Some(30),
..Default::default()
}),
@@ -10370,7 +10394,7 @@ mod tests {
del_marker_expiration: None,
filter: None,
id: Some("expire-current-and-noncurrent".to_string()),
noncurrent_version_expiration: Some(NoncurrentVersionExpiration {
noncurrent_version_expiration: Some(PersistedNoncurrentVersionExpiration {
noncurrent_days: Some(1),
newer_noncurrent_versions: None,
}),
@@ -10381,12 +10405,12 @@ mod tests {
}
}
fn all_versions_expiration_lifecycle() -> BucketLifecycleConfiguration {
BucketLifecycleConfiguration {
fn all_versions_expiration_lifecycle() -> PersistedLifecycleConfiguration {
PersistedLifecycleConfiguration {
expiry_updated_at: None,
rules: vec![LifecycleRule {
status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
expiration: Some(LifecycleExpiration {
rules: vec![PersistedLifecycleRule {
status: "Enabled".to_owned(),
expiration: Some(PersistedLifecycleExpiration {
days: Some(1),
expired_object_all_versions: Some(true),
..Default::default()
@@ -10405,29 +10429,29 @@ mod tests {
fn lock_enabled_without_default_retention() -> ObjectLockConfiguration {
ObjectLockConfiguration {
object_lock_enabled: Some(ObjectLockEnabled::from_static(ObjectLockEnabled::ENABLED)),
object_lock_enabled: Some("Enabled".to_owned()),
rule: None,
}
}
fn lock_enabled_with_default_retention() -> ObjectLockConfiguration {
ObjectLockConfiguration {
object_lock_enabled: Some(ObjectLockEnabled::from_static(ObjectLockEnabled::ENABLED)),
object_lock_enabled: Some("Enabled".to_owned()),
rule: Some(ObjectLockRule {
default_retention: Some(DefaultRetention {
days: Some(30),
mode: Some(ObjectLockRetentionMode::from_static(ObjectLockRetentionMode::COMPLIANCE)),
mode: Some("COMPLIANCE".to_owned()),
years: None,
}),
}),
}
}
fn latest_transition_lifecycle() -> BucketLifecycleConfiguration {
BucketLifecycleConfiguration {
fn latest_transition_lifecycle() -> PersistedLifecycleConfiguration {
PersistedLifecycleConfiguration {
expiry_updated_at: None,
rules: vec![LifecycleRule {
status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
rules: vec![PersistedLifecycleRule {
status: "Enabled".to_owned(),
expiration: None,
abort_incomplete_multipart_upload: None,
del_marker_expiration: None,
@@ -10436,10 +10460,10 @@ mod tests {
noncurrent_version_expiration: None,
noncurrent_version_transitions: None,
prefix: None,
transitions: Some(vec![Transition {
transitions: Some(vec![PersistedTransition {
days: Some(1),
date: None,
storage_class: Some(TransitionStorageClass::from_static("WARM")),
storage_class: Some("WARM".to_owned()),
}]),
}],
}
@@ -12976,8 +13000,9 @@ mod tests {
for (name, object, lock_config, expected) in cases {
let object_opts = lifecycle::object_opts_from_object_info(&object);
let evaluator_lock_config = lock_config.clone().map(Arc::new);
let batch_event = Evaluator::new(Arc::new(lifecycle.clone()))
.with_lock_retention(lock_config.clone().map(Arc::new))
.with_lock_retention(evaluator_lock_config)
.eval(&[object_opts])
.await
.expect("batch lifecycle evaluation should succeed")
@@ -14234,10 +14259,7 @@ mod tests {
let object = current_object_with_metadata(
ReplicationStatusType::Completed,
HashMap::from([
(
metadata_keys::OBJECT_LOCK_MODE.to_string(),
s3s::dto::ObjectLockRetentionMode::COMPLIANCE.to_string(),
),
(metadata_keys::OBJECT_LOCK_MODE.to_string(), "COMPLIANCE".to_string()),
(metadata_keys::OBJECT_LOCK_RETAIN_UNTIL_DATE.to_string(), retain_until),
]),
);
@@ -14249,7 +14271,7 @@ mod tests {
#[tokio::test]
async fn restored_copy_expiry_is_not_blocked_by_retention() {
let lifecycle = BucketLifecycleConfiguration {
let lifecycle = PersistedLifecycleConfiguration {
expiry_updated_at: None,
rules: Vec::new(),
};
@@ -14259,10 +14281,7 @@ mod tests {
let mut object = current_object_with_metadata(
ReplicationStatusType::Completed,
HashMap::from([
(
metadata_keys::OBJECT_LOCK_MODE.to_string(),
ObjectLockRetentionMode::COMPLIANCE.to_string(),
),
(metadata_keys::OBJECT_LOCK_MODE.to_string(), "COMPLIANCE".to_string()),
(metadata_keys::OBJECT_LOCK_RETAIN_UNTIL_DATE.to_string(), retain_until),
]),
);
@@ -14479,7 +14498,7 @@ mod tests {
"removed" => {
metadata_sys::delete(&bucket, BUCKET_LIFECYCLE_CONFIG).await.unwrap();
}
"disabled" => changed.rules[0].status = ExpirationStatus::from_static(ExpirationStatus::DISABLED),
"disabled" => changed.rules[0].status = "Disabled".to_owned(),
"longer-days" => {
changed.rules[0]
.noncurrent_version_expiration
@@ -14595,7 +14614,7 @@ mod tests {
);
}
async fn install_test_expiry_policy(ecstore: &Arc<ECStore>, bucket: &str) -> Arc<BucketLifecycleConfiguration> {
async fn install_test_expiry_policy(ecstore: &Arc<ECStore>, bucket: &str) -> Arc<PersistedLifecycleConfiguration> {
let xml = br#"<LifecycleConfiguration><Rule><ID>expire</ID><Status>Enabled</Status><Filter><Prefix></Prefix></Filter><Expiration><Days>0</Days></Expiration><NoncurrentVersionExpiration><NoncurrentDays>0</NoncurrentDays></NoncurrentVersionExpiration></Rule></LifecycleConfiguration>"#;
metadata_sys::update_in(&ecstore.ctx, bucket, BUCKET_LIFECYCLE_CONFIG, xml.to_vec())
.await
@@ -15288,7 +15307,11 @@ mod tests {
#[tokio::test]
#[serial]
async fn tier_free_version_recovery_production_entrypoint_enqueues_seeded_item() {
let (disk_paths, ecstore) = setup_test_env().await;
let (disk_paths, ecstore) =
temp_env::async_with_vars([(super::ENV_TIER_FREE_VERSION_RECOVERY_ENABLED, Some("false"))], async {
setup_test_env().await
})
.await;
let tier = format!("RECOVERY{}", Uuid::new_v4().simple()).to_uppercase();
crate::services::tier::test_util::register_mock_tier(&ecstore.tier_config_mgr(), &tier).await;
let runtime_state = install_unconsumed_runtime_expiry_worker(&ecstore, 1).await;
+1 -1
View File
@@ -17,7 +17,7 @@ use crate::object_api::ObjectInfo;
pub use rustfs_lifecycle::{
Event, ExpirationOptions, IlmAction, LIFECYCLE_MALFORMED_XML_ERROR_KIND, Lifecycle, LifecycleCalculate, ObjectOpts,
RuleValidate, TRANSITION_COMPLETE, TRANSITION_PENDING, TransitionOptions, abort_incomplete_multipart_upload_due,
expected_expiry_time, expiration_action_has_valid_target,
expected_expiry_time, expiration_action_has_valid_target, format_lifecycle_date, parse_lifecycle_date,
};
pub fn object_opts_from_object_info(oi: &ObjectInfo) -> ObjectOpts {
@@ -14,7 +14,8 @@
use std::sync::Arc;
use s3s::dto::{BucketLifecycleConfiguration, ObjectLockConfiguration};
use rustfs_gateway_types::persistence::PersistedLifecycleConfiguration;
use rustfs_gateway_types::persistence::PersistedObjectLockConfiguration as ObjectLockConfiguration;
use time::OffsetDateTime;
use uuid::Uuid;
@@ -24,7 +25,7 @@ use crate::error::{Error, Result};
#[derive(Debug)]
pub(crate) struct LifecycleExpiryConfigs {
pub(crate) lifecycle: Option<Arc<BucketLifecycleConfiguration>>,
pub(crate) lifecycle: Option<Arc<PersistedLifecycleConfiguration>>,
pub(crate) object_lock: Option<Arc<ObjectLockConfiguration>>,
pub(crate) bucket_incarnation_id: Uuid,
pub(crate) table_bucket_enabled: bool,
@@ -100,7 +101,7 @@ pub(crate) async fn get_expiry_configs_for_options(
})
}
pub(crate) async fn get_lifecycle_config(bucket: &str) -> Result<(BucketLifecycleConfiguration, OffsetDateTime)> {
pub(crate) async fn get_lifecycle_config(bucket: &str) -> Result<(PersistedLifecycleConfiguration, OffsetDateTime)> {
metadata_sys::get_lifecycle_config(bucket).await
}
@@ -110,15 +111,15 @@ mod tests {
use crate::bucket::metadata::BucketMetadata;
use crate::bucket::metadata_sys::{self, test_support::isolated_store_over_temp_disks};
use crate::storage_api_contracts::bucket::MakeBucketOptions;
use s3s::dto::{ExpirationStatus, LifecycleExpiration, LifecycleRule};
use rustfs_gateway_types::persistence::{PersistedLifecycleExpiration, PersistedLifecycleRule};
use serial_test::serial;
fn lifecycle_config() -> BucketLifecycleConfiguration {
BucketLifecycleConfiguration {
fn lifecycle_config() -> PersistedLifecycleConfiguration {
PersistedLifecycleConfiguration {
expiry_updated_at: None,
rules: vec![LifecycleRule {
status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
expiration: Some(LifecycleExpiration {
rules: vec![PersistedLifecycleRule {
status: "Enabled".to_owned(),
expiration: Some(PersistedLifecycleExpiration {
days: Some(1),
..Default::default()
}),
@@ -22,7 +22,7 @@ pub(crate) fn is_object_locked_by_metadata(user_defined: &HashMap<String, String
}
pub(crate) fn check_object_lock_for_deletion_with_config(
config: Option<&s3s::dto::ObjectLockConfiguration>,
config: Option<&rustfs_gateway_types::persistence::PersistedObjectLockConfiguration>,
obj_info: &ObjectInfo,
bypass_governance: bool,
) -> crate::error::Result<Option<ObjectLockBlockReason>> {
@@ -82,12 +82,13 @@ mod tests {
use crate::bucket::replication::{DeleteReplicationConfigSnapshot, ReplicationObjectBridge};
use crate::object_api::{ObjectInfo, ObjectOptions};
use crate::storage_api_contracts::object::ObjectToDelete;
use rustfs_scanner_metrics::metrics::IlmAction;
use s3s::dto::{
BucketVersioningStatus, DeleteMarkerReplication, DeleteMarkerReplicationStatus, DeleteReplication,
DeleteReplicationStatus, Destination, ReplicationConfiguration, ReplicationRule, ReplicationRuleStatus,
VersioningConfiguration,
use rustfs_gateway_types::dto::Status;
use rustfs_gateway_types::persistence::PersistedVersioningConfiguration as VersioningConfiguration;
use rustfs_gateway_types::persistence::{
PersistedOptionalReplicationStatus, PersistedReplicationConfiguration, PersistedReplicationDestination,
PersistedReplicationRule, PersistedReplicationStatus,
};
use rustfs_scanner_metrics::metrics::IlmAction;
use uuid::Uuid;
use super::*;
@@ -160,27 +161,27 @@ mod tests {
for purge_enabled in [false, true] {
let snapshot = DeleteReplicationConfigSnapshot::from_configs_for_test(
VersioningConfiguration {
status: Some(BucketVersioningStatus::from_static(BucketVersioningStatus::ENABLED)),
status: Some("Enabled".to_owned()),
..Default::default()
},
Some(ReplicationConfiguration {
Some(PersistedReplicationConfiguration {
role: String::new(),
rules: vec![ReplicationRule {
delete_marker_replication: Some(DeleteMarkerReplication {
rules: vec![PersistedReplicationRule {
delete_marker_replication: Some(PersistedOptionalReplicationStatus {
status: Some(if marker_enabled {
DeleteMarkerReplicationStatus::from_static(DeleteMarkerReplicationStatus::ENABLED)
Status::ENABLED.to_string()
} else {
DeleteMarkerReplicationStatus::from_static(DeleteMarkerReplicationStatus::DISABLED)
Status::DISABLED.to_string()
}),
}),
delete_replication: Some(DeleteReplication {
delete_replication: Some(PersistedReplicationStatus {
status: if purge_enabled {
DeleteReplicationStatus::from_static(DeleteReplicationStatus::ENABLED)
Status::ENABLED.to_string()
} else {
DeleteReplicationStatus::from_static(DeleteReplicationStatus::DISABLED)
Status::DISABLED.to_string()
},
}),
destination: Destination {
destination: PersistedReplicationDestination {
bucket: "arn:rustfs:replication:target".to_string(),
..Default::default()
},
@@ -190,7 +191,7 @@ mod tests {
prefix: Some(String::new()),
priority: Some(1),
source_selection_criteria: None,
status: ReplicationRuleStatus::from_static(ReplicationRuleStatus::ENABLED),
status: Status::ENABLED.to_string(),
}],
}),
);
@@ -14,7 +14,7 @@
use std::sync::Arc;
use s3s::dto::BucketLifecycleConfiguration;
use rustfs_gateway_types::persistence::PersistedLifecycleConfiguration;
use tokio::sync::RwLock;
use tokio_util::sync::CancellationToken;
@@ -52,7 +52,7 @@ pub(crate) fn deployment_id() -> Option<String> {
sources::deployment_id()
}
pub(crate) async fn bucket_lifecycle_config(bucket: &str) -> Option<BucketLifecycleConfiguration> {
pub(crate) async fn bucket_lifecycle_config(bucket: &str) -> Option<PersistedLifecycleConfiguration> {
sources::bucket_lifecycle_config(bucket).await
}
+59 -44
View File
@@ -13,7 +13,7 @@
// limitations under the License.
use super::msgp_decode::{read_msgp_ext8_time, skip_msgp_value, write_msgp_time};
use super::object_lock::{ObjectLockApi, ObjectLockStatusExt};
use super::object_lock::ObjectLockApi;
use super::versioning::VersioningApi;
use super::{quota::BucketQuota, target::BucketTargets};
use crate::bucket::replication::invalid_replication_config_status_field;
@@ -25,12 +25,17 @@ use crate::object_api::WriteCompletion;
use crate::runtime::sources as runtime_sources;
use crate::store::ECStore;
use byteorder::{BigEndian, ByteOrder, LittleEndian};
use rustfs_policy::policy::BucketPolicy;
use s3s::dto::{
AccelerateConfiguration, BucketLifecycleConfiguration, BucketLoggingStatus, BucketVersioningStatus, CORSConfiguration,
NotificationConfiguration, ObjectLockConfiguration, PublicAccessBlockConfiguration, ReplicationConfiguration,
RequestPaymentConfiguration, ServerSideEncryptionConfiguration, Tagging, VersioningConfiguration, WebsiteConfiguration,
use rustfs_gateway_types::cors_tagging::{PersistedCorsConfiguration as CORSConfiguration, PersistedTagging as Tagging};
use rustfs_gateway_types::persistence::{
PersistedAccelerateConfiguration as AccelerateConfiguration,
PersistedBucketEncryptionConfiguration as ServerSideEncryptionConfiguration,
PersistedBucketLoggingStatus as BucketLoggingStatus, PersistedLifecycleConfiguration,
PersistedNotificationConfiguration as NotificationConfiguration, PersistedObjectLockConfiguration as ObjectLockConfiguration,
PersistedPublicAccessBlockConfiguration as PublicAccessBlockConfiguration, PersistedReplicationConfiguration,
PersistedRequestPaymentConfiguration as RequestPaymentConfiguration,
PersistedVersioningConfiguration as VersioningConfiguration, PersistedWebsiteConfiguration as WebsiteConfiguration,
};
use rustfs_policy::policy::BucketPolicy;
use serde::Serializer;
use sha2::{Digest, Sha256};
use std::collections::HashMap;
@@ -40,23 +45,11 @@ use time::{Date, OffsetDateTime, PrimitiveDateTime, Time as CivilTime, UtcOffset
use tracing::error;
use uuid::Uuid;
// The serving-layer DTO impls for the storage-level Object Lock traits live
// here because this module owns the persisted `ObjectLockConfiguration`
// during the s3s ratchet migration (rustfs/backlog#1842).
// The persisted Object Lock configuration answers the storage-level trait
// here because this module owns the persisted `ObjectLockConfiguration`.
impl ObjectLockApi for ObjectLockConfiguration {
fn enabled(&self) -> bool {
self.object_lock_enabled
.as_ref()
.is_some_and(|v| v.as_str() == s3s::dto::ObjectLockEnabled::ENABLED)
}
}
impl ObjectLockStatusExt for s3s::dto::ObjectLockLegalHoldStatus {
fn valid(&self) -> bool {
matches!(
self.as_str(),
s3s::dto::ObjectLockLegalHoldStatus::ON | s3s::dto::ObjectLockLegalHoldStatus::OFF
)
self.object_lock_enabled.as_deref() == Some("Enabled")
}
}
@@ -257,12 +250,14 @@ pub const BUCKET_NOTIFICATION_CONFIG: &str = "notification.xml";
pub const BUCKET_LIFECYCLE_CONFIG: &str = "lifecycle.xml";
pub const BUCKET_SSECONFIG: &str = "bucket-encryption.xml";
/// Validate tag entries supplied to the S3 API.
pub fn validate_tag_set_entries(tag_set: &[s3s::dto::Tag]) -> std::result::Result<(), &'static str> {
pub fn validate_tag_set_entries(
tag_set: &[rustfs_gateway_types::cors_tagging::PersistedTag],
) -> std::result::Result<(), &'static str> {
validate_tag_entries(tag_set, validate_s3_tag_entry)
}
fn validate_tag_entries(
tag_set: &[s3s::dto::Tag],
tag_set: &[rustfs_gateway_types::cors_tagging::PersistedTag],
mut validate_entry: impl FnMut(&str, &str) -> std::result::Result<(), &'static str>,
) -> std::result::Result<(), &'static str> {
let mut keys = std::collections::HashSet::with_capacity(tag_set.len());
@@ -323,7 +318,9 @@ pub fn is_swift_bucket_metadata_tag_key(key: &str) -> bool {
|| key.starts_with(SWIFT_CONTAINER_META_TAG_PREFIX)
}
fn validate_persisted_bucket_tags(tags: &[s3s::dto::Tag]) -> std::result::Result<(), &'static str> {
fn validate_persisted_bucket_tags(
tags: &[rustfs_gateway_types::cors_tagging::PersistedTag],
) -> std::result::Result<(), &'static str> {
const SWIFT_ACL_MAX_BYTES: usize = 8192;
const SWIFT_META_NAME_MAX_BYTES: usize = 128;
const MAX_PERSISTED_TAGS: usize = MAX_BUCKET_TAGS + 2 * MAX_SWIFT_METADATA_COUNT + 4;
@@ -561,6 +558,8 @@ pub struct BucketMetadata {
pub bucket_incarnation_id: Uuid,
pub(crate) bucket_incarnation_sidecar: bool,
pub(crate) bucket_creation_committed: bool,
// A legacy default is not proof of an interrupted creation.
pub(crate) bucket_creation_commit_record_present: bool,
pub policy_config_json: Vec<u8>,
pub notification_config_xml: Vec<u8>,
pub lifecycle_config_xml: Vec<u8>,
@@ -609,13 +608,13 @@ pub struct BucketMetadata {
pub policy_config: Option<BucketPolicy>,
pub notification_config: Option<NotificationConfiguration>,
pub lifecycle_config: Option<BucketLifecycleConfiguration>,
pub lifecycle_config: Option<PersistedLifecycleConfiguration>,
pub object_lock_config: Option<ObjectLockConfiguration>,
pub versioning_config: Option<VersioningConfiguration>,
pub sse_config: Option<ServerSideEncryptionConfiguration>,
pub tagging_config: Option<Tagging>,
pub quota_config: Option<BucketQuota>,
pub replication_config: Option<ReplicationConfiguration>,
pub replication_config: Option<PersistedReplicationConfiguration>,
pub bucket_target_config: Option<BucketTargets>,
pub bucket_target_config_meta: Option<HashMap<String, String>>,
pub cors_config: Option<CORSConfiguration>,
@@ -636,6 +635,7 @@ impl Default for BucketMetadata {
bucket_incarnation_id: Uuid::nil(),
bucket_incarnation_sidecar: false,
bucket_creation_committed: false,
bucket_creation_commit_record_present: false,
policy_config_json: Default::default(),
notification_config_xml: Default::default(),
lifecycle_config_xml: Default::default(),
@@ -874,7 +874,10 @@ impl BucketMetadata {
"TableBucketConfigUpdatedAt" => self.table_bucket_config_updated_at = read_msgp_time_value(rd)?,
"DurabilityConfigUpdatedAt" => self.durability_config_updated_at = read_msgp_time_value(rd)?,
"OnDemandMigrationConfigUpdatedAt" => self.on_demand_migration_config_updated_at = read_msgp_time_value(rd)?,
"BucketCreationCommitted" => self.bucket_creation_committed = read_msgp_bool(rd)?,
"BucketCreationCommitted" => {
self.bucket_creation_committed = read_msgp_bool(rd)?;
self.bucket_creation_commit_record_present = true;
}
other => {
tracing::debug!(field = %other, "BucketMetadata decode_from: skipping unknown field");
skip_msgp_value(rd)?;
@@ -887,8 +890,10 @@ impl BucketMetadata {
/// Encode to msgp bytes. Field order follows MinIO BucketMetadata for compatibility.
pub fn encode_to<W: Write>(&self, wr: &mut W) -> Result<()> {
// Map size: MinIO fields (25) + RustFS extensions (22)
let map_len: u32 = 47;
// Preserve the absence of a legacy commit record across config rewrites.
let commit_record_present = self.bucket_creation_commit_record_present || self.bucket_creation_committed;
// Map size: MinIO fields (25) + RustFS extensions (21) + optional commit record.
let map_len: u32 = 46 + u32::from(commit_record_present);
rmp::encode::write_map_len(wr, map_len)?;
// MinIO field order (same as Go struct)
@@ -969,8 +974,10 @@ impl BucketMetadata {
write_msgp_time(wr, self.durability_config_updated_at)?;
rmp::encode::write_str(wr, "OnDemandMigrationConfigUpdatedAt")?;
write_msgp_time(wr, self.on_demand_migration_config_updated_at)?;
rmp::encode::write_str(wr, "BucketCreationCommitted")?;
rmp::encode::write_bool(wr, self.bucket_creation_committed)?;
if commit_record_present {
rmp::encode::write_str(wr, "BucketCreationCommitted")?;
rmp::encode::write_bool(wr, self.bucket_creation_committed)?;
}
Ok(())
}
@@ -1147,9 +1154,11 @@ impl BucketMetadata {
None
} else {
let config = deserialize::<VersioningConfiguration>(&data)?;
if config.status.as_ref().is_some_and(|status| {
!matches!(status.as_str(), BucketVersioningStatus::ENABLED | BucketVersioningStatus::SUSPENDED)
}) {
if config
.status
.as_ref()
.is_some_and(|status| !matches!(status.as_str(), "Enabled" | "Suspended"))
{
return Err(Error::other("bucket versioning configuration has an invalid status"));
}
Some(config)
@@ -1162,7 +1171,7 @@ impl BucketMetadata {
let config = if data.is_empty() {
None
} else {
let config = deserialize::<ReplicationConfiguration>(&data)?;
let config = deserialize::<PersistedReplicationConfiguration>(&data)?;
if let Some(field) = invalid_replication_config_status_field(&config) {
return Err(Error::other(format!("replication field {field} has an invalid status")));
}
@@ -1356,8 +1365,8 @@ impl BucketMetadata {
}
self.lifecycle_config = None;
if !self.lifecycle_config_xml.is_empty()
&& let Err(e) =
deserialize::<BucketLifecycleConfiguration>(&self.lifecycle_config_xml).map(|c| self.lifecycle_config = Some(c))
&& let Err(e) = deserialize::<PersistedLifecycleConfiguration>(&self.lifecycle_config_xml)
.map(|c| self.lifecycle_config = Some(c))
{
tracing::warn!(
event = "bucket_metadata_parse_failed",
@@ -1443,8 +1452,8 @@ impl BucketMetadata {
}
self.replication_config = None;
if !self.replication_config_xml.is_empty()
&& let Err(e) =
deserialize::<ReplicationConfiguration>(&self.replication_config_xml).map(|c| self.replication_config = Some(c))
&& let Err(e) = deserialize::<PersistedReplicationConfiguration>(&self.replication_config_xml)
.map(|c| self.replication_config = Some(c))
{
tracing::warn!(
event = "bucket_metadata_parse_failed",
@@ -1824,6 +1833,13 @@ mod test {
let mut bm = BucketMetadata::unmarshal(&body[4..]).expect("unmarshal recovered blob");
assert_eq!(bm.name, "interop");
assert!(!bm.bucket_creation_committed, "legacy metadata must default to uncommitted");
assert!(!bm.bucket_creation_commit_record_present, "legacy metadata has no creation-commit record");
let rewritten = BucketMetadata::unmarshal(&bm.marshal_msg().expect("rewrite legacy metadata"))
.expect("read rewritten legacy metadata");
assert!(
!rewritten.bucket_creation_commit_record_present,
"rewriting legacy metadata must not fabricate a creation intent"
);
bm.parse_all_configs().expect("parse recovered configs");
assert!(bm.lifecycle_config.is_some());
}
@@ -1841,6 +1857,7 @@ mod test {
assert_eq!(bm.name, new.name);
assert!(new.bucket_creation_committed);
assert!(new.bucket_creation_commit_record_present);
assert!(!bm.bucket_incarnation_id.is_nil());
assert_eq!(bm.bucket_incarnation_id, new.bucket_incarnation_id);
}
@@ -1912,7 +1929,7 @@ mod test {
#[test]
fn persisted_bucket_tags_keep_separate_s3_and_swift_limits() {
use s3s::dto::Tag;
use rustfs_gateway_types::cors_tagging::PersistedTag as Tag;
let tag = |key: String, value: &str| Tag {
key: Some(key),
@@ -2004,17 +2021,15 @@ mod test {
#[test]
fn object_locking_requires_lock_metadata_not_plain_versioning() {
use s3s::dto::ObjectLockEnabled;
let mut bm = BucketMetadata::new("test-bucket");
bm.versioning_config = Some(VersioningConfiguration {
status: Some(s3s::dto::BucketVersioningStatus::from_static("Enabled")),
status: Some("Enabled".to_owned()),
..Default::default()
});
assert!(!bm.object_locking());
bm.object_lock_config = Some(ObjectLockConfiguration {
object_lock_enabled: Some(ObjectLockEnabled::from_static(ObjectLockEnabled::ENABLED)),
object_lock_enabled: Some("Enabled".to_owned()),
..Default::default()
});
assert!(bm.object_locking());
+326 -48
View File
@@ -31,14 +31,18 @@ use crate::storage_api_contracts::heal::HealOperations as _;
use crate::storage_api_contracts::namespace::NamespaceLocking as _;
use crate::store::{ECStore, await_bucket_namespace_operation};
use futures::future::join_all;
use rustfs_gateway_types::cors_tagging::{PersistedCorsConfiguration as CORSConfiguration, PersistedTagging as Tagging};
use rustfs_gateway_types::persistence::{
PersistedAccelerateConfiguration as AccelerateConfiguration,
PersistedBucketEncryptionConfiguration as ServerSideEncryptionConfiguration,
PersistedBucketLoggingStatus as BucketLoggingStatus, PersistedLifecycleConfiguration,
PersistedNotificationConfiguration as NotificationConfiguration, PersistedObjectLockConfiguration as ObjectLockConfiguration,
PersistedPublicAccessBlockConfiguration as PublicAccessBlockConfiguration, PersistedReplicationConfiguration,
PersistedRequestPaymentConfiguration as RequestPaymentConfiguration,
PersistedVersioningConfiguration as VersioningConfiguration, PersistedWebsiteConfiguration as WebsiteConfiguration,
};
use rustfs_heal_contracts::heal_channel::HealOpts;
use rustfs_policy::policy::BucketPolicy;
use s3s::dto::ReplicationConfiguration;
use s3s::dto::{
AccelerateConfiguration, BucketLifecycleConfiguration, BucketLoggingStatus, CORSConfiguration, NotificationConfiguration,
ObjectLockConfiguration, ObjectLockEnabled, ObjectLockRetentionMode, PublicAccessBlockConfiguration,
RequestPaymentConfiguration, ServerSideEncryptionConfiguration, Tagging, VersioningConfiguration, WebsiteConfiguration,
};
use std::collections::HashSet;
use std::time::Duration;
use std::{
@@ -169,7 +173,7 @@ pub(crate) fn object_lock_config_state_from_authoritative_metadata(bm: &BucketMe
if bm.lock_enabled {
return Ok(ObjectLockConfigState::Configured {
config: ObjectLockConfiguration {
object_lock_enabled: Some(ObjectLockEnabled::from_static(ObjectLockEnabled::ENABLED)),
object_lock_enabled: Some("Enabled".to_owned()),
rule: None,
},
updated_at: bm.object_lock_config_updated_at,
@@ -213,17 +217,10 @@ pub(crate) fn configured_object_lock_state_for_tests(
) -> ObjectLockConfigState {
ObjectLockConfigState::Configured {
config: ObjectLockConfiguration {
object_lock_enabled: Some(ObjectLockEnabled::from_static(ObjectLockEnabled::ENABLED)),
rule: Some(s3s::dto::ObjectLockRule {
default_retention: Some(s3s::dto::DefaultRetention {
mode: Some(s3s::dto::ObjectLockRetentionMode::from_static(match mode {
crate::bucket::object_lock::types::RetentionMode::Governance => {
s3s::dto::ObjectLockRetentionMode::GOVERNANCE
}
crate::bucket::object_lock::types::RetentionMode::Compliance => {
s3s::dto::ObjectLockRetentionMode::COMPLIANCE
}
})),
object_lock_enabled: Some("Enabled".to_owned()),
rule: Some(rustfs_gateway_types::persistence::PersistedObjectLockRule {
default_retention: Some(rustfs_gateway_types::persistence::PersistedDefaultRetention {
mode: Some(mode.as_str().to_owned()),
days: Some(days),
years: None,
}),
@@ -234,7 +231,7 @@ pub(crate) fn configured_object_lock_state_for_tests(
}
fn validate_authoritative_object_lock_config(config: &ObjectLockConfiguration) -> Result<()> {
if config.object_lock_enabled.as_ref().map(ObjectLockEnabled::as_str) != Some(ObjectLockEnabled::ENABLED) {
if config.object_lock_enabled.as_deref() != Some("Enabled") {
return Err(Error::other("persisted bucket Object Lock enabled state is invalid"));
}
@@ -247,7 +244,7 @@ fn validate_authoritative_object_lock_config(config: &ObjectLockConfiguration) -
if !retention
.mode
.as_ref()
.is_some_and(|mode| matches!(mode.as_str(), ObjectLockRetentionMode::COMPLIANCE | ObjectLockRetentionMode::GOVERNANCE))
.is_some_and(|mode| matches!(mode.as_str(), "COMPLIANCE" | "GOVERNANCE"))
{
return Err(Error::other("persisted bucket Object Lock retention mode is invalid"));
}
@@ -1641,7 +1638,7 @@ pub async fn get_public_access_block_config(bucket: &str) -> Result<(PublicAcces
bucket_meta_sys.get_public_access_block_config(bucket).await
}
pub async fn get_lifecycle_config(bucket: &str) -> Result<(BucketLifecycleConfiguration, OffsetDateTime)> {
pub async fn get_lifecycle_config(bucket: &str) -> Result<(PersistedLifecycleConfiguration, OffsetDateTime)> {
let bucket_meta_sys_lock = get_bucket_metadata_sys()?;
let bucket_meta_sys = bucket_meta_sys_lock.read().await;
@@ -1917,7 +1914,7 @@ pub(crate) async fn get_cached_quota_config_and_incarnation_in(
bucket_meta_sys.cached_quota_config_and_incarnation(bucket).await
}
pub async fn get_replication_config(bucket: &str) -> Result<(ReplicationConfiguration, OffsetDateTime)> {
pub async fn get_replication_config(bucket: &str) -> Result<(PersistedReplicationConfiguration, OffsetDateTime)> {
let bucket_meta_sys_lock = get_bucket_metadata_sys()?;
let bucket_meta_sys = bucket_meta_sys_lock.read().await;
@@ -1927,7 +1924,7 @@ pub async fn get_replication_config(bucket: &str) -> Result<(ReplicationConfigur
pub(crate) async fn get_replication_config_in(
ctx: &crate::runtime::instance::InstanceContext,
bucket: &str,
) -> Result<(ReplicationConfiguration, OffsetDateTime)> {
) -> Result<(PersistedReplicationConfiguration, OffsetDateTime)> {
let bucket_meta_sys_lock = bucket_metadata_sys_of(ctx)?;
let bucket_meta_sys = bucket_meta_sys_lock.read().await;
@@ -2065,9 +2062,27 @@ pub(crate) fn current_local_meta_mutation_gen() -> u64 {
type LocalAuthorityCache = Arc<StdMutex<HashMap<String, (u64, Arc<BucketMetadata>)>>>;
#[derive(Debug, Clone)]
pub(crate) struct PendingBucketCreationProof {
pub(crate) generation: Uuid,
pub(crate) intent: Vec<u8>,
pub(crate) versioning_requested: bool,
pub(crate) witnesses: Vec<crate::disk::os::NativeBucketCreationWitness>,
}
impl PendingBucketCreationProof {
pub(crate) fn matches_metadata(&self, metadata: &BucketMetadata) -> Result<bool> {
Ok(!metadata.bucket_creation_committed
&& metadata.bucket_incarnation_sidecar
&& metadata.bucket_incarnation_id == self.generation
&& metadata.marshal_msg()? == self.intent)
}
}
#[derive(Debug, Clone)]
pub struct BucketMetadataSys {
metadata_map: Arc<RwLock<HashMap<String, Arc<BucketMetadata>>>>,
pending_bucket_creations: Arc<RwLock<HashMap<String, PendingBucketCreationProof>>>,
local_authority_cache: LocalAuthorityCache,
/// Serializes metadata-map commits and their derived cache updates for one
/// bucket. Namespace locks, when present, are acquired before this lock.
@@ -2112,6 +2127,7 @@ impl BucketMetadataSys {
pub fn new(api: Arc<ECStore>) -> Self {
Self {
metadata_map: Arc::new(RwLock::new(HashMap::new())),
pending_bucket_creations: Arc::new(RwLock::new(HashMap::new())),
local_authority_cache: Arc::new(StdMutex::new(HashMap::new())),
metadata_publish_locks: Arc::new(MetadataPublishLockRegistry {
locks: StdMutex::new(HashMap::new()),
@@ -2143,6 +2159,81 @@ impl BucketMetadataSys {
.expect("bucket metadata object store should still be live")
}
pub(crate) async fn pending_bucket_creation(&self, bucket: &str) -> Option<PendingBucketCreationProof> {
self.pending_bucket_creations.read().await.get(bucket).cloned()
}
pub(crate) async fn grant_pending_bucket_creation(&self, bucket: &str, proof: PendingBucketCreationProof) {
self.pending_bucket_creations.write().await.insert(bucket.to_owned(), proof);
}
pub(crate) async fn revoke_pending_bucket_creation(&self, bucket: &str) {
self.pending_bucket_creations.write().await.remove(bucket);
}
pub(crate) async fn consume_pending_bucket_creation(
&self,
bucket: &str,
metadata: &BucketMetadata,
witnesses: &[crate::disk::os::NativeBucketCreationWitness],
) -> Result<bool> {
let mut pending = self.pending_bucket_creations.write().await;
let Some(proof) = pending.get(bucket) else {
return Ok(false);
};
if !proof.matches_metadata(metadata)?
|| proof.witnesses.len() != witnesses.len()
|| !proof
.witnesses
.iter()
.zip(witnesses)
.all(|(old, current)| old.matches(current))
|| !crate::disk::os::consume_native_creation_witnesses(witnesses)
{
return Ok(false);
}
pending.remove(bucket);
Ok(true)
}
#[cfg(test)]
pub(crate) async fn refresh_bucket_for_test(&self, bucket: &str) -> Result<()> {
let expected = self.metadata_map.read().await.get(bucket).cloned();
let lock = self.object_store().new_ns_lock(bucket, bucket).await?;
let guard = lock.get_read_lock(crate::set_disk::get_lock_acquire_timeout()).await?;
self.load_bucket_under_namespace(bucket, MetadataLoadMode::Refresh, expected.as_ref(), &guard)
.await
}
#[cfg(test)]
pub(crate) async fn publish_persisted_metadata_for_test(&self, bucket: &str) -> Result<()> {
let expected = self.metadata_map.read().await.get(bucket).cloned();
let lock = self.object_store().new_ns_lock(bucket, bucket).await?;
let guard = lock.get_read_lock(crate::set_disk::get_lock_acquire_timeout()).await?;
let (metadata, persisted) = await_bucket_namespace_operation(
Some(&guard),
bucket,
"test cached metadata load",
load_bucket_metadata_parse_with_presence(self.object_store(), bucket, true),
)
.await?;
self.publish_if_unchanged(bucket, expected.as_ref(), metadata, persisted, &guard)
.await
}
#[cfg(test)]
pub(crate) async fn migrate_bucket_creation_commit_for_test(&self, bucket: &str) -> Result<()> {
self.migrate_bucket_creation_commit(bucket).await.map(|_| ())
}
#[cfg(test)]
pub(crate) async fn legacy_migration_reports_missing_bucket_for_test(&self, bucket: &str) -> Result<bool> {
Ok(matches!(
self.migrate_legacy_metadata(bucket).await?,
BucketMetadataAuthority::MissingBucket
))
}
fn object_store_if_live(&self) -> Option<Arc<ECStore>> {
self.api.upgrade()
}
@@ -2342,6 +2433,7 @@ impl BucketMetadataSys {
expected: Option<&Arc<BucketMetadata>>,
namespace_guard: &rustfs_lock::NamespaceLockGuard,
) -> Result<()> {
self.revoke_pending_bucket_creation(bucket).await;
if !await_bucket_namespace_operation(
Some(namespace_guard),
bucket,
@@ -2425,6 +2517,7 @@ impl BucketMetadataSys {
persisted: bool,
namespace_guard: &rustfs_lock::NamespaceLockGuard,
) -> Result<()> {
self.revoke_pending_bucket_creation(bucket).await;
if !persisted {
let _publish_guard = self
.lock_metadata_publish(bucket, namespace_guard, "refreshed bucket metadata absence publish")
@@ -2486,6 +2579,7 @@ impl BucketMetadataSys {
}
pub async fn set(&self, bucket: String, bm: Arc<BucketMetadata>) {
self.revoke_pending_bucket_creation(&bucket).await;
bump_local_meta_mutation_gen();
if !is_meta_bucketname(&bucket) {
let publish_lock = self.metadata_publish_lock(&bucket);
@@ -2509,6 +2603,7 @@ impl BucketMetadataSys {
if is_meta_bucketname(bucket) {
return false;
}
self.revoke_pending_bucket_creation(bucket).await;
let publish_lock = self.metadata_publish_lock(bucket);
let _publish_guard = publish_lock.lock().await;
let mut map = self.metadata_map.write().await;
@@ -2526,6 +2621,7 @@ impl BucketMetadataSys {
}
async fn _reset(&mut self) {
self.pending_bucket_creations.write().await.clear();
bump_local_meta_mutation_gen();
let mut map = self.metadata_map.write().await;
map.clear();
@@ -2665,6 +2761,7 @@ impl BucketMetadataSys {
/// server's metadata never leaks into the ambient (first) instance.
pub(crate) async fn persist_and_set(&self, bm: BucketMetadata) -> Result<()> {
let mut bm = bm;
self.revoke_pending_bucket_creation(&bm.name).await;
bm.save_with_store(self.object_store()).await?;
self.set(bm.name.clone(), Arc::new(bm)).await;
@@ -2673,6 +2770,7 @@ impl BucketMetadataSys {
}
async fn persist_new_and_set(&self, mut bm: BucketMetadata) -> Result<()> {
self.revoke_pending_bucket_creation(&bm.name).await;
bm.bucket_creation_committed = true;
bm.save_with_store_committed(self.object_store()).await?;
save_bucket_incarnation(self.object_store(), &bm.name, bm.bucket_incarnation_id).await?;
@@ -2682,7 +2780,9 @@ impl BucketMetadataSys {
}
async fn persist_new_bucket_metadata_intent(&self, mut bm: BucketMetadata) -> Result<()> {
self.revoke_pending_bucket_creation(&bm.name).await;
bm.bucket_creation_committed = false;
bm.bucket_creation_commit_record_present = true;
bm.save_with_store(self.object_store()).await?;
save_bucket_incarnation(self.object_store(), &bm.name, bm.bucket_incarnation_id).await?;
bm.bucket_incarnation_sidecar = true;
@@ -2692,6 +2792,7 @@ impl BucketMetadataSys {
}
async fn commit_bucket_metadata(&self, mut bm: BucketMetadata) -> Result<()> {
self.revoke_pending_bucket_creation(&bm.name).await;
bm.parse_all_configs()?;
bm.bucket_incarnation_sidecar = true;
// The caller holds the full bucket creation fence (lifecycle, metadata
@@ -2740,6 +2841,7 @@ impl BucketMetadataSys {
bucket: &str,
namespace_guard: &rustfs_lock::NamespaceLockGuard,
) -> Result<()> {
self.revoke_pending_bucket_creation(bucket).await;
let expected = self.metadata_map.read().await.get(bucket).cloned();
if !self
.bucket_exists(bucket, namespace_guard, "peer bucket metadata existence check")
@@ -2796,6 +2898,7 @@ impl BucketMetadataSys {
#[cfg(test)]
self.lazy_disk_loads.fetch_add(1, std::sync::atomic::Ordering::Relaxed);
self.revoke_pending_bucket_creation(bucket).await;
let lock = self.object_store().new_ns_lock(bucket, bucket).await?;
let guard = lock.get_read_lock(crate::set_disk::get_lock_acquire_timeout()).await?;
#[cfg(test)]
@@ -3121,6 +3224,7 @@ impl BucketMetadataSys {
/// revalidated at write quorum, so a stale snapshot can neither revert an
/// acknowledged configuration update nor outlive a delete/recreate.
async fn migrate_bucket_creation_commit(&self, bucket: &str) -> Result<BucketMetadataAuthority> {
self.revoke_pending_bucket_creation(bucket).await;
let transaction_guard = acquire_metadata_transaction_guard(
self.object_store(),
bucket,
@@ -3198,6 +3302,7 @@ impl BucketMetadataSys {
&self,
bucket: &str,
) -> Result<BucketMetadataAuthority> {
self.revoke_pending_bucket_creation(bucket).await;
#[cfg(test)]
if self.object_lock_disk_read_errors.write().await.remove(bucket) {
return Err(Error::other(format!("injected Object Lock metadata disk read failure: {bucket}")));
@@ -3420,7 +3525,7 @@ impl BucketMetadataSys {
}
}
pub async fn get_lifecycle_config(&self, bucket: &str) -> Result<(BucketLifecycleConfiguration, OffsetDateTime)> {
pub async fn get_lifecycle_config(&self, bucket: &str) -> Result<(PersistedLifecycleConfiguration, OffsetDateTime)> {
let (bm, _) = self.get_config(bucket).await?;
match ConfigState::of(&bm.lifecycle_config_xml, &bm.lifecycle_config).require(bucket, BUCKET_LIFECYCLE_CONFIG)? {
@@ -3543,7 +3648,7 @@ impl BucketMetadataSys {
}))
}
pub async fn get_replication_config(&self, bucket: &str) -> Result<(ReplicationConfiguration, OffsetDateTime)> {
pub async fn get_replication_config(&self, bucket: &str) -> Result<(PersistedReplicationConfiguration, OffsetDateTime)> {
let (bm, _) = self.get_config(bucket).await?;
if !bm.replication_config_xml.is_empty() && bm.replication_config.is_none() {
@@ -3645,6 +3750,7 @@ mod tests {
};
use crate::bucket::target::{BucketTarget, BucketTargetType, Credentials};
use crate::config::com::read_config;
use crate::disk::{DiskAPI as _, RUSTFS_META_TMP_BUCKET};
use crate::storage_api_contracts::bucket::{BucketOperations as _, DeleteBucketOptions, MakeBucketOptions};
use byteorder::{ByteOrder as _, LittleEndian};
use serial_test::serial;
@@ -3699,6 +3805,180 @@ mod tests {
),
];
#[tokio::test]
#[serial(storage_class_env)]
async fn persistence_entry_failures_revoke_pending_creation_before_cache_publish() {
let (dirs, store) = isolated_store_over_temp_disks().await;
init_bucket_metadata_sys(store.clone(), Vec::new()).await;
let sys = bucket_metadata_sys_of(&store.ctx).expect("instance metadata system");
let sys = sys.read().await.clone();
let options = MakeBucketOptions {
lock_enabled: true,
..Default::default()
};
for entry in ["save", "persist", "persist-new", "commit"] {
let bucket = format!("revoke-{entry}-{}", Uuid::new_v4());
store.fail_next_bucket_creation_after_intent_for_test(&bucket);
store
.make_bucket(&bucket, &options)
.await
.expect_err("stop the real producer before physical publication");
sys.pending_bucket_creation(&bucket)
.await
.expect("the producer must leave a real live creation permission");
let (intent, persisted) = get_config_from_disk_with_presence_in(&store.ctx, &bucket)
.await
.expect("read the original producer intent");
assert!(persisted, "{entry}: the complete original intent must be persisted");
let original_intent = intent.marshal_msg().expect("encode the original intent");
// Keep the creation fence order used by the real producer.
let lifecycle_guard = store
.acquire_bucket_lifecycle_write_lock(&bucket)
.await
.expect("hold the bucket lifecycle fence");
let transaction_guard = acquire_bucket_metadata_transaction_lock_in(&store.ctx, &bucket)
.await
.expect("hold the metadata transaction fence");
let namespace_lock = store
.new_ns_lock(&bucket, &bucket)
.await
.expect("create the exact bucket namespace lock");
let namespace_guard = namespace_lock
.get_write_lock(crate::set_disk::get_lock_acquire_timeout())
.await
.expect("hold the exact bucket namespace fence");
// A file at each native staging root causes actual IO to fail before
// metadata can reach the cache-publishing set call. Preserve the
// original staging directories so no fixture data is discarded.
for dir in &dirs {
let staging = dir.path().join(RUSTFS_META_TMP_BUCKET);
let saved = dir.path().join(".rustfs.sys/revocation-saved-tmp");
std::fs::rename(&staging, &saved).expect("preserve the native staging directory");
std::fs::write(&staging, b"block metadata staging with a non-directory")
.expect("install an actual non-directory IO obstruction");
}
let result = match entry {
"save" => sys.save(intent).await,
"persist" => sys.persist_and_set(intent).await,
"persist-new" => sys.persist_new_and_set(intent).await,
"commit" => sys.commit_bucket_metadata(intent).await,
_ => unreachable!(),
};
for dir in &dirs {
let staging = dir.path().join(RUSTFS_META_TMP_BUCKET);
let saved = dir.path().join(".rustfs.sys/revocation-saved-tmp");
std::fs::remove_file(&staging).expect("remove only the fixture's staging obstruction");
std::fs::rename(&saved, &staging).expect("restore the complete original staging directory");
}
drop((namespace_guard, transaction_guard, lifecycle_guard));
result.expect_err("native persistence must fail before its downstream cache set");
assert!(
sys.pending_bucket_creation(&bucket).await.is_none(),
"{entry}: the entry revoker must retire live permission even when persistence fails"
);
let (after_failure, present) = get_config_from_disk_with_presence_in(&store.ctx, &bucket)
.await
.expect("read the intent after the failed persistence entry");
assert!(present, "{entry}: a failed persistence entry must retain the original metadata");
assert_eq!(
after_failure
.marshal_msg()
.expect("encode the original intent after persistence failure"),
original_intent,
"{entry}: native IO failure must not change the original generation or complete intent"
);
let error = store
.make_bucket(&bucket, &options)
.await
.expect_err("a failed ordinary persistence entry cannot preserve creation permission");
assert!(error.to_string().contains("administrator recovery"), "{entry}: {error}");
let (after_retry, _) = get_config_from_disk_with_presence_in(&store.ctx, &bucket)
.await
.expect("read the original intent after the rejected retry");
assert_eq!(
after_retry.marshal_msg().expect("encode the intent after the rejected retry"),
original_intent,
"{entry}: the rejected retry must preserve the complete original intent"
);
for disk in store.pools[0].disk_set[0].disks.read().await.iter() {
assert!(
matches!(
disk.as_ref().expect("native disk").stat_volume(&bucket).await,
Err(crate::disk::error::DiskError::VolumeNotFound)
),
"{entry}: failed persistence and the rejected retry must not publish a physical volume"
);
}
}
}
#[tokio::test]
#[serial(storage_class_env)]
async fn metadata_reset_revokes_real_pending_creation_without_changing_persisted_intent() {
let (_dirs, store) = isolated_store_over_temp_disks().await;
init_bucket_metadata_sys(store.clone(), Vec::new()).await;
let bucket = format!("revoke-reset-{}", Uuid::new_v4());
let options = MakeBucketOptions {
lock_enabled: true,
..Default::default()
};
store.fail_next_bucket_creation_after_intent_for_test(&bucket);
store
.make_bucket(&bucket, &options)
.await
.expect_err("stop the real producer before physical publication");
let sys = bucket_metadata_sys_of(&store.ctx).expect("instance metadata system");
let mut sys = sys.read().await.clone();
sys.pending_bucket_creation(&bucket)
.await
.expect("the real producer must leave live creation permission before reset");
let (intent, persisted) = get_config_from_disk_with_presence_in(&store.ctx, &bucket)
.await
.expect("read the original intent before cache reset");
assert!(persisted, "cache reset starts from a complete real persisted intent");
let original_intent = intent.marshal_msg().expect("encode the original reset intent");
sys._reset().await;
assert!(
sys.pending_bucket_creation(&bucket).await.is_none(),
"reset must revoke the producer-issued permission shared with the instance metadata system"
);
let (after_reset, present) = get_config_from_disk_with_presence_in(&store.ctx, &bucket)
.await
.expect("read the original intent after cache reset");
assert!(present, "cache reset must retain the persisted intent");
assert_eq!(
after_reset.marshal_msg().expect("encode the intent after cache reset"),
original_intent,
"cache reset must leave the original generation and complete persisted intent unchanged"
);
let error = store
.make_bucket(&bucket, &options)
.await
.expect_err("reset cannot regrant the producer's old creation permission");
assert!(error.to_string().contains("administrator recovery"), "{error}");
let (after_retry, _) = get_config_from_disk_with_presence_in(&store.ctx, &bucket)
.await
.expect("read the intent after the reset-induced retry rejection");
assert_eq!(
after_retry
.marshal_msg()
.expect("encode the intent after the rejected reset retry"),
original_intent,
"the reset-induced rejection must preserve the original complete intent"
);
for disk in store.pools[0].disk_set[0].disks.read().await.iter() {
assert!(
matches!(
disk.as_ref().expect("native disk").stat_volume(&bucket).await,
Err(crate::disk::error::DiskError::VolumeNotFound)
),
"cache reset and the rejected retry must not publish a physical volume"
);
}
}
#[tokio::test]
async fn g_d3_003_new_writer_replication_loads_without_fail_closed_state() {
let (dirs, store) = isolated_store_over_temp_disks().await;
@@ -3887,7 +4167,7 @@ mod tests {
let mut versioned = BucketMetadata::new("authoritative-versioned");
versioned.versioning_config_xml = b"<VersioningConfiguration><Status>Enabled</Status></VersioningConfiguration>".to_vec();
versioned.versioning_config = Some(VersioningConfiguration {
status: Some(s3s::dto::BucketVersioningStatus::from_static(s3s::dto::BucketVersioningStatus::ENABLED)),
status: Some("Enabled".to_owned()),
..Default::default()
});
sys.set("authoritative-versioned".to_string(), Arc::new(versioned)).await;
@@ -4010,25 +4290,24 @@ mod tests {
else {
panic!("legacy lock-enabled metadata must remain configured");
};
assert_eq!(
config.object_lock_enabled.as_ref().map(|value| value.as_str()),
Some(ObjectLockEnabled::ENABLED)
);
assert_eq!(config.object_lock_enabled.as_deref(), Some("Enabled"));
assert!(config.rule.is_none());
}
#[test]
fn authoritative_object_lock_state_rejects_invalid_retention_periods() {
use s3s::dto::{DefaultRetention, ObjectLockRule};
use rustfs_gateway_types::persistence::{
PersistedDefaultRetention as DefaultRetention, PersistedObjectLockRule as ObjectLockRule,
};
for (days, years) in [(Some(0), None), (Some(-1), None), (Some(1), Some(1)), (None, None)] {
let mut metadata = BucketMetadata::new("invalid-object-lock");
metadata.object_lock_config = Some(ObjectLockConfiguration {
object_lock_enabled: Some(ObjectLockEnabled::from_static(ObjectLockEnabled::ENABLED)),
object_lock_enabled: Some("Enabled".to_owned()),
rule: Some(ObjectLockRule {
default_retention: Some(DefaultRetention {
days,
mode: Some(ObjectLockRetentionMode::from_static(ObjectLockRetentionMode::COMPLIANCE)),
mode: Some("COMPLIANCE".to_owned()),
years,
}),
}),
@@ -4043,9 +4322,11 @@ mod tests {
#[test]
fn authoritative_object_lock_state_rejects_invalid_enabled_mode_and_rule() {
use s3s::dto::{DefaultRetention, ObjectLockRule};
use rustfs_gateway_types::persistence::{
PersistedDefaultRetention as DefaultRetention, PersistedObjectLockRule as ObjectLockRule,
};
for enabled in [None, Some(ObjectLockEnabled::from("Disabled".to_string()))] {
for enabled in [None, Some("Disabled".to_string())] {
let mut metadata = BucketMetadata::new("invalid-object-lock-enabled");
metadata.lock_enabled = true;
metadata.object_lock_config = Some(ObjectLockConfiguration {
@@ -4057,11 +4338,11 @@ mod tests {
let mut invalid_mode = BucketMetadata::new("invalid-object-lock-mode");
invalid_mode.object_lock_config = Some(ObjectLockConfiguration {
object_lock_enabled: Some(ObjectLockEnabled::from_static(ObjectLockEnabled::ENABLED)),
object_lock_enabled: Some("Enabled".to_owned()),
rule: Some(ObjectLockRule {
default_retention: Some(DefaultRetention {
days: Some(1),
mode: Some(ObjectLockRetentionMode::from("invalid".to_string())),
mode: Some("invalid".to_string()),
years: None,
}),
}),
@@ -4070,7 +4351,7 @@ mod tests {
let mut missing_retention = BucketMetadata::new("missing-object-lock-retention");
missing_retention.object_lock_config = Some(ObjectLockConfiguration {
object_lock_enabled: Some(ObjectLockEnabled::from_static(ObjectLockEnabled::ENABLED)),
object_lock_enabled: Some("Enabled".to_owned()),
rule: Some(ObjectLockRule { default_retention: None }),
});
assert!(object_lock_config_state_from_authoritative_metadata(&missing_retention).is_err());
@@ -4080,7 +4361,7 @@ mod tests {
fn object_lock_ever_enabled_marker_survives_config_removal() {
let mut metadata = BucketMetadata::new("lock-config-removed");
let config = ObjectLockConfiguration {
object_lock_enabled: Some(ObjectLockEnabled::from_static(ObjectLockEnabled::ENABLED)),
object_lock_enabled: Some("Enabled".to_owned()),
rule: None,
};
metadata
@@ -4098,10 +4379,7 @@ mod tests {
else {
panic!("an ever-enabled bucket must remain Object Lock configured");
};
assert_eq!(
config.object_lock_enabled.as_ref().map(ObjectLockEnabled::as_str),
Some(ObjectLockEnabled::ENABLED)
);
assert_eq!(config.object_lock_enabled.as_deref(), Some("Enabled"));
}
#[tokio::test]
@@ -4117,7 +4395,7 @@ mod tests {
let node = Arc::new(RwLock::new(BucketMetadataSys::new(ecstore.clone())));
let config = ObjectLockConfiguration {
object_lock_enabled: Some(ObjectLockEnabled::from_static(ObjectLockEnabled::ENABLED)),
object_lock_enabled: Some("Enabled".to_owned()),
rule: None,
};
update_with_sys(
@@ -5231,7 +5509,7 @@ mod tests {
#[tokio::test]
async fn update_config_with_persists_tagging_rewrite_across_disk_reload() {
use crate::bucket::metadata::BUCKET_TAGGING_CONFIG;
use s3s::dto::Tag;
use rustfs_gateway_types::cors_tagging::PersistedTag as Tag;
let (dirs, ecstore) = isolated_store_over_temp_disks().await;
@@ -5395,7 +5673,7 @@ mod tests {
#[tokio::test]
async fn concurrent_update_config_with_calls_do_not_lose_writes() {
use crate::bucket::metadata::BUCKET_TAGGING_CONFIG;
use s3s::dto::Tag;
use rustfs_gateway_types::cors_tagging::PersistedTag as Tag;
let (dirs, ecstore) = isolated_store_over_temp_disks().await;
let sys = Arc::new(RwLock::new(BucketMetadataSys::new(ecstore)));
@@ -5495,7 +5773,7 @@ mod tests {
let mut persisted = BucketMetadata::new("reload-bucket");
persisted.policy_config_json = b"persisted-marker".to_vec();
let lock_config = ObjectLockConfiguration {
object_lock_enabled: Some(ObjectLockEnabled::from_static(ObjectLockEnabled::ENABLED)),
object_lock_enabled: Some("Enabled".to_owned()),
rule: None,
};
persisted
@@ -5829,7 +6107,7 @@ mod tests {
#[serial]
async fn bucket_delete_waits_for_config_mutation_fence() {
use crate::bucket::metadata::BUCKET_TAGGING_CONFIG;
use s3s::dto::Tag;
use rustfs_gateway_types::cors_tagging::PersistedTag as Tag;
let (_dirs, store) = isolated_store_over_temp_disks().await;
init_bucket_metadata_sys(store.clone(), Vec::new()).await;
+2 -9
View File
@@ -17,15 +17,8 @@ pub mod objectlock_sys;
pub mod types;
/// Whether a bucket Object Lock configuration has locking enabled. The
/// serving-layer `ObjectLockConfiguration` DTO implements this in
/// the bucket-metadata module, which owns the persisted configuration type
/// during the s3s ratchet migration (rustfs/backlog#1842).
/// persisted `ObjectLockConfiguration` implements this in the bucket-metadata
/// module, which owns the persisted configuration type.
pub trait ObjectLockApi {
fn enabled(&self) -> bool;
}
/// Whether a legal-hold status value is one of the two valid wire values.
/// Implemented for the serving-layer DTO in the bucket-metadata module.
pub trait ObjectLockStatusExt {
fn valid(&self) -> bool;
}
@@ -12,6 +12,7 @@
// See the License for the specific language governing permissions and
// limitations under the License.
pub use rustfs_replication::config_cache_serde;
pub use rustfs_replication::{
ObjectOpts, OperatorRuleContract, REMOTE_TARGET_CAPABILITY_CONTRACT_VERSION, REMOTE_TARGET_READ_ONLY_HISTORICAL_FIELDS,
REMOTE_TARGET_UNSUPPORTED_FIELDS, REMOTE_TARGET_WRITABLE_FIELDS, REPLICATION_CAPABILITY_CONTRACT_VERSION,
@@ -12,7 +12,7 @@
// See the License for the specific language governing permissions and
// limitations under the License.
use s3s::dto::ReplicationConfiguration;
use rustfs_gateway_types::persistence::PersistedReplicationConfiguration;
use super::replication_config_boundary::ReplicationConfigurationExt as _;
use super::replication_filemeta_boundary::{
@@ -42,7 +42,7 @@ impl ReplicationLifecycleBridge {
dead_code,
reason = "declared boundary surface for the ECStore replication split plan; no caller in this port (backlog#1823)"
)]
pub(crate) fn new_config(config: ReplicationConfiguration) -> ReplicationLifecycleConfig {
pub(crate) fn new_config(config: PersistedReplicationConfiguration) -> ReplicationLifecycleConfig {
ReplicationConfig::new(Some(config), None)
}
@@ -118,16 +118,17 @@ impl ReplicationLifecycleBridge {
#[cfg(test)]
mod tests {
use s3s::dto::{Destination, ReplicationRule, ReplicationRuleStatus};
use rustfs_gateway_types::dto::Status;
use rustfs_gateway_types::persistence::{PersistedReplicationDestination, PersistedReplicationRule};
use super::super::replication_filemeta_boundary::{ReplicateTargetDecision, VersionPurgeStatusType};
use super::*;
fn replication_rule() -> ReplicationRule {
ReplicationRule {
fn replication_rule() -> PersistedReplicationRule {
PersistedReplicationRule {
delete_marker_replication: None,
delete_replication: None,
destination: Destination {
destination: PersistedReplicationDestination {
bucket: "arn:aws:s3:::target-bucket".to_string(),
..Default::default()
},
@@ -137,13 +138,13 @@ mod tests {
prefix: Some(String::new()),
priority: Some(1),
source_selection_criteria: None,
status: ReplicationRuleStatus::from_static(ReplicationRuleStatus::ENABLED),
status: Status::ENABLED.to_string(),
}
}
#[test]
fn has_pending_version_purge_preserves_replication_active_rule_behavior() {
let config = ReplicationLifecycleBridge::new_config(ReplicationConfiguration {
let config = ReplicationLifecycleBridge::new_config(PersistedReplicationConfiguration {
role: String::new(),
rules: vec![replication_rule()],
});
@@ -17,8 +17,8 @@ use std::sync::Arc;
use crate::bucket::{metadata::BucketMetadata, metadata_sys};
use crate::disk::{BUCKET_META_PREFIX, RUSTFS_META_BUCKET};
use crate::runtime::instance::InstanceContext;
use rustfs_gateway_types::persistence::PersistedReplicationConfiguration;
use rustfs_utils::path::path_join_buf;
use s3s::dto::ReplicationConfiguration;
use time::OffsetDateTime;
use super::replication_error_boundary::{Error, Result};
@@ -37,11 +37,11 @@ impl ReplicationMetadataStore {
pub(crate) const FORCE_DELETE_REPLICATION_FILE: &'static str = "config/replication/force-delete.bin";
pub(crate) const FORCE_DELETE_REPLICATION_TRANSACTION_LOCK: &'static str = "config/replication/force-delete.bin.transaction";
pub(crate) async fn replication_config(bucket: &str) -> Result<(ReplicationConfiguration, OffsetDateTime)> {
pub(crate) async fn replication_config(bucket: &str) -> Result<(PersistedReplicationConfiguration, OffsetDateTime)> {
metadata_sys::get_replication_config(bucket).await
}
pub(crate) async fn optional_replication_config(bucket: &str) -> Result<Option<ReplicationConfiguration>> {
pub(crate) async fn optional_replication_config(bucket: &str) -> Result<Option<PersistedReplicationConfiguration>> {
let config = match Self::replication_config(bucket).await {
Ok((config, _)) => Some(config),
Err(err) => {
@@ -16,7 +16,9 @@ use std::{collections::HashMap, fmt, sync::Arc};
use super::replication_filemeta_boundary::metadata_keys;
use crate::bucket::metadata::BucketMetadata;
use s3s::dto::{BucketVersioningStatus, ReplicationConfiguration, ReplicationRuleStatus, VersioningConfiguration};
use rustfs_gateway_types::dto::Status;
use rustfs_gateway_types::persistence::PersistedReplicationConfiguration;
use rustfs_gateway_types::persistence::PersistedVersioningConfiguration as VersioningConfiguration;
use serde::{Deserialize, Serialize};
use time::OffsetDateTime;
use tracing::error;
@@ -39,7 +41,7 @@ use super::replication_target_boundary::{BucketTargets, ReplicationTargetStore};
use super::replication_versioning_boundary::ReplicationVersioningStore;
use super::runtime_boundary as runtime_sources;
pub(crate) async fn get_replication_config(bucket: &str) -> Result<Option<ReplicationConfiguration>> {
pub(crate) async fn get_replication_config(bucket: &str) -> Result<Option<PersistedReplicationConfiguration>> {
let config = ReplicationMetadataStore::optional_replication_config(bucket).await?;
validate_delete_replication_config(&VersioningConfiguration::default(), config.as_ref())?;
Ok(config)
@@ -64,7 +66,7 @@ impl DeleteReplicationConfigSnapshot {
#[cfg(test)]
pub(crate) fn from_configs_for_test(
versioning: VersioningConfiguration,
replication: Option<ReplicationConfiguration>,
replication: Option<PersistedReplicationConfiguration>,
) -> Self {
let metadata = replication.map(|config| {
let mut metadata = BucketMetadata::new("test-bucket");
@@ -78,7 +80,7 @@ impl DeleteReplicationConfigSnapshot {
&self.versioning
}
pub fn replication_config(&self) -> Option<&ReplicationConfiguration> {
pub fn replication_config(&self) -> Option<&PersistedReplicationConfiguration> {
self.metadata
.as_ref()
.and_then(|metadata| metadata.replication_config.as_ref())
@@ -101,7 +103,7 @@ impl DeleteReplicationConfigSnapshot {
pub(crate) fn active_delete_marker_rules_require_tags(&self, object: &str) -> bool {
self.replication_config().is_some_and(|config| {
config.rules.iter().any(|rule| {
if rule.status == ReplicationRuleStatus::from_static(ReplicationRuleStatus::DISABLED) {
if rule.status == Status::DISABLED.as_str() {
return false;
}
if !object.starts_with(rule.prefix()) {
@@ -122,12 +124,12 @@ impl DeleteReplicationConfigSnapshot {
fn validate_delete_replication_config(
versioning: &VersioningConfiguration,
config: Option<&ReplicationConfiguration>,
config: Option<&PersistedReplicationConfiguration>,
) -> Result<()> {
if versioning
.status
.as_ref()
.is_some_and(|status| !matches!(status.as_str(), BucketVersioningStatus::ENABLED | BucketVersioningStatus::SUSPENDED))
.is_some_and(|status| !matches!(status.as_str(), "Enabled" | "Suspended"))
{
return Err(super::replication_error_boundary::Error::other(
"bucket versioning configuration has an invalid status",
@@ -144,7 +146,7 @@ fn validate_delete_replication_config(
let role = config.role.trim();
let mut role_destination = None;
for rule in &config.rules {
if rule.status.as_str() == ReplicationRuleStatus::ENABLED {
if rule.status.as_str() == Status::ENABLED.as_str() {
let destination = rule.destination.bucket.trim();
if role.is_empty() && destination.is_empty() {
return Err(super::replication_error_boundary::Error::other(
@@ -169,7 +171,7 @@ fn validate_delete_replication_config(
Ok(())
}
fn replication_config_from_metadata(metadata: &BucketMetadata) -> Result<Option<&ReplicationConfiguration>> {
fn replication_config_from_metadata(metadata: &BucketMetadata) -> Result<Option<&PersistedReplicationConfiguration>> {
if !metadata.replication_config_xml.is_empty() && metadata.replication_config.is_none() {
return Err(super::replication_error_boundary::Error::other(
"persisted bucket replication configuration is invalid",
@@ -250,12 +252,15 @@ pub(crate) async fn load_delete_replication_config_in(
#[derive(Debug, Clone, Default, Serialize, Deserialize)]
pub struct ReplicationConfig {
pub config: Option<ReplicationConfiguration>,
/// Carried in the scanner's persisted data usage cache, so its serde form
/// is the one the s3s DTO wrote there (see `config_cache_serde` in the replication crate).
#[serde(with = "super::replication_config_boundary::config_cache_serde")]
pub config: Option<PersistedReplicationConfiguration>,
pub remotes: Option<BucketTargets>,
}
impl ReplicationConfig {
pub fn new(config: Option<ReplicationConfiguration>, remotes: Option<BucketTargets>) -> Self {
pub fn new(config: Option<PersistedReplicationConfiguration>, remotes: Option<BucketTargets>) -> Self {
Self { config, remotes }
}
@@ -468,7 +473,7 @@ fn check_replicate_delete_with_config(
oi: &ObjectInfo,
del_opts: &ObjectOptions,
source_error: bool,
config: Option<&ReplicationConfiguration>,
config: Option<&PersistedReplicationConfiguration>,
existing_delete_marker: bool,
trust_persisted_replica_status: bool,
) -> ReplicateDecision {
@@ -587,7 +592,7 @@ pub(crate) async fn must_replicate(bucket: &str, object: &str, mopts: MustReplic
}
fn metadata_target_should_replicate(
cfg: &ReplicationConfiguration,
cfg: &PersistedReplicationConfiguration,
opts: &ObjectOpts,
mopts: &MustReplicateOptions,
arn: &str,
@@ -597,20 +602,21 @@ fn metadata_target_should_replicate(
#[cfg(test)]
mod tests {
use s3s::dto::{
DeleteMarkerReplication, DeleteMarkerReplicationStatus, DeleteReplication, DeleteReplicationStatus, Destination,
ReplicaModifications, ReplicationRule, ReplicationRuleFilter, ReplicationRuleStatus, SourceSelectionCriteria, Tag,
use rustfs_gateway_types::dto::Status;
use rustfs_gateway_types::persistence::{
PersistedOptionalReplicationStatus, PersistedReplicationDestination, PersistedReplicationFilter,
PersistedReplicationRule, PersistedReplicationStatus, PersistedReplicationTag, PersistedSourceSelectionCriteria,
};
use super::super::replication_filemeta_boundary::VersionPurgeStatusType;
use super::super::replication_target_boundary::BucketTarget;
use super::*;
fn replication_rule() -> ReplicationRule {
ReplicationRule {
fn replication_rule() -> PersistedReplicationRule {
PersistedReplicationRule {
delete_marker_replication: None,
delete_replication: None,
destination: Destination {
destination: PersistedReplicationDestination {
bucket: "arn:aws:s3:::target-bucket".to_string(),
..Default::default()
},
@@ -620,7 +626,7 @@ mod tests {
prefix: Some(String::new()),
priority: Some(1),
source_selection_criteria: None,
status: ReplicationRuleStatus::from_static(ReplicationRuleStatus::ENABLED),
status: Status::ENABLED.to_string(),
}
}
@@ -629,11 +635,11 @@ mod tests {
let arn = "arn:rustfs:replication:us-east-1:target:bucket";
let mut rule = replication_rule();
rule.destination.bucket = arn.to_string();
rule.filter = Some(ReplicationRuleFilter {
rule.filter = Some(PersistedReplicationFilter {
prefix: Some("admitted/".to_string()),
..Default::default()
});
let cfg = ReplicationConfiguration {
let cfg = PersistedReplicationConfiguration {
role: String::new(),
rules: vec![rule],
};
@@ -671,7 +677,7 @@ mod tests {
assert!(!empty.replicate(&ObjectOpts::default()));
let config = ReplicationConfig::new(
Some(ReplicationConfiguration {
Some(PersistedReplicationConfiguration {
role: String::new(),
rules: vec![replication_rule()],
}),
@@ -725,7 +731,7 @@ mod tests {
fn delete_snapshot_rejects_enabled_rules_without_a_destination() {
let mut rule = replication_rule();
rule.destination.bucket.clear();
let config = ReplicationConfiguration {
let config = PersistedReplicationConfiguration {
role: String::new(),
rules: vec![rule],
};
@@ -741,7 +747,7 @@ mod tests {
let first = replication_rule();
let mut second = replication_rule();
second.destination.bucket = "arn:aws:s3:::other-target".to_string();
let config = ReplicationConfiguration {
let config = PersistedReplicationConfiguration {
role: "arn:aws:s3:::role-target".to_string(),
rules: vec![first, second],
};
@@ -752,47 +758,47 @@ mod tests {
#[test]
fn delete_snapshot_rejects_unknown_status_values() {
let invalid_versioning = VersioningConfiguration {
status: Some("Enabld".to_string().into()),
status: Some("Enabld".to_string()),
..Default::default()
};
assert!(validate_delete_replication_config(&invalid_versioning, None).is_err());
let mut invalid_rule = replication_rule();
invalid_rule.status = "Enabld".to_string().into();
let config = ReplicationConfiguration {
invalid_rule.status = "Enabld".to_string();
let config = PersistedReplicationConfiguration {
role: String::new(),
rules: vec![invalid_rule],
};
assert!(validate_delete_replication_config(&VersioningConfiguration::default(), Some(&config)).is_err());
let mut invalid_delete = replication_rule();
invalid_delete.delete_replication = Some(DeleteReplication {
status: "Enabld".to_string().into(),
invalid_delete.delete_replication = Some(PersistedReplicationStatus {
status: "Enabld".to_string(),
});
let config = ReplicationConfiguration {
let config = PersistedReplicationConfiguration {
role: String::new(),
rules: vec![invalid_delete],
};
assert!(validate_delete_replication_config(&VersioningConfiguration::default(), Some(&config)).is_err());
let mut invalid_delete_marker = replication_rule();
invalid_delete_marker.delete_marker_replication = Some(DeleteMarkerReplication {
status: Some("Enabld".to_string().into()),
invalid_delete_marker.delete_marker_replication = Some(PersistedOptionalReplicationStatus {
status: Some("Enabld".to_string()),
});
let config = ReplicationConfiguration {
let config = PersistedReplicationConfiguration {
role: String::new(),
rules: vec![invalid_delete_marker],
};
assert!(validate_delete_replication_config(&VersioningConfiguration::default(), Some(&config)).is_err());
let mut invalid_replica_modifications = replication_rule();
invalid_replica_modifications.source_selection_criteria = Some(SourceSelectionCriteria {
replica_modifications: Some(ReplicaModifications {
status: "Enabld".to_string().into(),
invalid_replica_modifications.source_selection_criteria = Some(PersistedSourceSelectionCriteria {
replica_modifications: Some(PersistedReplicationStatus {
status: "Enabld".to_string(),
}),
sse_kms_encrypted_objects: None,
});
let config = ReplicationConfiguration {
let config = PersistedReplicationConfiguration {
role: String::new(),
rules: vec![invalid_replica_modifications],
};
@@ -804,11 +810,11 @@ mod tests {
let mut metadata = BucketMetadata::new("bucket");
metadata.versioning_config_xml = b"configured".to_vec();
metadata.versioning_config = Some(VersioningConfiguration {
status: Some(BucketVersioningStatus::from_static(BucketVersioningStatus::ENABLED)),
status: Some("Enabled".to_owned()),
..Default::default()
});
metadata.replication_config_xml = b"configured".to_vec();
metadata.replication_config = Some(ReplicationConfiguration {
metadata.replication_config = Some(PersistedReplicationConfiguration {
role: String::new(),
rules: vec![replication_rule()],
});
@@ -848,7 +854,7 @@ mod tests {
);
let mut inconsistent = BucketMetadata::new("bucket");
inconsistent.replication_config = Some(ReplicationConfiguration {
inconsistent.replication_config = Some(PersistedReplicationConfiguration {
role: String::new(),
rules: vec![replication_rule()],
});
@@ -863,11 +869,11 @@ mod tests {
let arn = "arn:rustfs:replication:us-east-1:target:bucket";
let mut rule = replication_rule();
rule.destination.bucket = arn.to_string();
rule.delete_marker_replication = Some(DeleteMarkerReplication {
status: Some(DeleteMarkerReplicationStatus::from_static(DeleteMarkerReplicationStatus::ENABLED)),
rule.delete_marker_replication = Some(PersistedOptionalReplicationStatus {
status: Some(Status::ENABLED.to_string()),
});
let mut metadata = BucketMetadata::new("bucket");
metadata.replication_config = Some(ReplicationConfiguration {
metadata.replication_config = Some(PersistedReplicationConfiguration {
role: String::new(),
rules: vec![rule],
});
@@ -898,27 +904,27 @@ mod tests {
fn delete_marker_source_read_is_required_only_for_tag_filtered_rules() {
let mut prefix_rule = replication_rule();
prefix_rule.prefix = Some("logs/".to_string());
prefix_rule.delete_marker_replication = Some(DeleteMarkerReplication {
status: Some(DeleteMarkerReplicationStatus::from_static(DeleteMarkerReplicationStatus::ENABLED)),
prefix_rule.delete_marker_replication = Some(PersistedOptionalReplicationStatus {
status: Some(Status::ENABLED.to_string()),
});
let prefix_snapshot = DeleteReplicationConfigSnapshot::from_configs_for_test(
VersioningConfiguration {
status: Some(BucketVersioningStatus::from_static(BucketVersioningStatus::ENABLED)),
status: Some("Enabled".to_owned()),
..Default::default()
},
Some(ReplicationConfiguration {
Some(PersistedReplicationConfiguration {
role: String::new(),
rules: vec![prefix_rule],
}),
);
let mut tag_rule = replication_rule();
tag_rule.delete_marker_replication = Some(DeleteMarkerReplication {
status: Some(DeleteMarkerReplicationStatus::from_static(DeleteMarkerReplicationStatus::ENABLED)),
tag_rule.delete_marker_replication = Some(PersistedOptionalReplicationStatus {
status: Some(Status::ENABLED.to_string()),
});
tag_rule.filter = Some(ReplicationRuleFilter {
tag: Some(Tag {
tag_rule.filter = Some(PersistedReplicationFilter {
tag: Some(PersistedReplicationTag {
key: Some("class".to_string()),
value: Some("audit".to_string()),
}),
@@ -926,10 +932,10 @@ mod tests {
});
let tag_snapshot = DeleteReplicationConfigSnapshot::from_configs_for_test(
VersioningConfiguration {
status: Some(BucketVersioningStatus::from_static(BucketVersioningStatus::ENABLED)),
status: Some("Enabled".to_owned()),
..Default::default()
},
Some(ReplicationConfiguration {
Some(PersistedReplicationConfiguration {
role: String::new(),
rules: vec![tag_rule],
}),
@@ -944,14 +950,14 @@ mod tests {
let arn = "arn:rustfs:replication:us-east-1:target:bucket";
let mut rule = replication_rule();
rule.destination.bucket = arn.to_string();
rule.delete_replication = Some(DeleteReplication {
status: DeleteReplicationStatus::from_static(DeleteReplicationStatus::ENABLED),
rule.delete_replication = Some(PersistedReplicationStatus {
status: Status::ENABLED.to_string(),
});
rule.delete_marker_replication = Some(DeleteMarkerReplication {
status: Some(DeleteMarkerReplicationStatus::from_static(DeleteMarkerReplicationStatus::DISABLED)),
rule.delete_marker_replication = Some(PersistedOptionalReplicationStatus {
status: Some(Status::DISABLED.to_string()),
});
let config = ReplicationConfig::new(
Some(ReplicationConfiguration {
Some(PersistedReplicationConfiguration {
role: String::new(),
rules: vec![rule],
}),
@@ -985,22 +991,22 @@ mod tests {
#[test]
fn live_delete_does_not_trust_persisted_replica_status() {
let mut rule = replication_rule();
rule.delete_replication = Some(DeleteReplication {
status: DeleteReplicationStatus::from_static(DeleteReplicationStatus::ENABLED),
rule.delete_replication = Some(PersistedReplicationStatus {
status: Status::ENABLED.to_string(),
});
rule.source_selection_criteria = Some(SourceSelectionCriteria {
replica_modifications: Some(ReplicaModifications {
status: s3s::dto::ReplicaModificationsStatus::from_static(s3s::dto::ReplicaModificationsStatus::DISABLED),
rule.source_selection_criteria = Some(PersistedSourceSelectionCriteria {
replica_modifications: Some(PersistedReplicationStatus {
status: Status::DISABLED.to_string(),
}),
sse_kms_encrypted_objects: None,
});
let replication = ReplicationConfiguration {
let replication = PersistedReplicationConfiguration {
role: String::new(),
rules: vec![rule],
};
let snapshot = DeleteReplicationConfigSnapshot::from_configs_for_test(
VersioningConfiguration {
status: Some(BucketVersioningStatus::from_static(BucketVersioningStatus::ENABLED)),
status: Some("Enabled".to_owned()),
..Default::default()
},
Some(replication.clone()),
@@ -1036,14 +1042,14 @@ mod tests {
let arn = "arn:rustfs:replication:us-east-1:target:bucket";
let mut rule = replication_rule();
rule.destination.bucket = arn.to_string();
rule.delete_replication = Some(DeleteReplication {
status: DeleteReplicationStatus::from_static(DeleteReplicationStatus::ENABLED),
rule.delete_replication = Some(PersistedReplicationStatus {
status: Status::ENABLED.to_string(),
});
rule.delete_marker_replication = Some(DeleteMarkerReplication {
status: Some(DeleteMarkerReplicationStatus::from_static(DeleteMarkerReplicationStatus::DISABLED)),
rule.delete_marker_replication = Some(PersistedOptionalReplicationStatus {
status: Some(Status::DISABLED.to_string()),
});
let config = ReplicationConfig::new(
Some(ReplicationConfiguration {
Some(PersistedReplicationConfiguration {
role: String::new(),
rules: vec![rule],
}),
@@ -5074,9 +5074,10 @@ mod tests {
#[tokio::test]
async fn heal_queue_marks_missing_versioning_state_as_missed() {
use super::super::replication_target_boundary::BucketTargets;
use s3s::dto::{
DeleteReplication, DeleteReplicationStatus, Destination, ReplicationConfiguration, ReplicationRule,
ReplicationRuleStatus,
use rustfs_gateway_types::dto::Status;
use rustfs_gateway_types::persistence::{
PersistedReplicationConfiguration, PersistedReplicationDestination, PersistedReplicationRule,
PersistedReplicationStatus,
};
let arn = "arn:rustfs:replication:us-east-1:target:bucket";
@@ -5091,14 +5092,14 @@ mod tests {
..Default::default()
},
ReplicationConfig::new(
Some(ReplicationConfiguration {
Some(PersistedReplicationConfiguration {
role: String::new(),
rules: vec![ReplicationRule {
rules: vec![PersistedReplicationRule {
delete_marker_replication: None,
delete_replication: Some(DeleteReplication {
status: DeleteReplicationStatus::from_static(DeleteReplicationStatus::ENABLED),
delete_replication: Some(PersistedReplicationStatus {
status: Status::ENABLED.to_string(),
}),
destination: Destination {
destination: PersistedReplicationDestination {
bucket: arn.to_string(),
..Default::default()
},
@@ -5108,7 +5109,7 @@ mod tests {
prefix: Some(String::new()),
priority: Some(1),
source_selection_criteria: None,
status: ReplicationRuleStatus::from_static(ReplicationRuleStatus::ENABLED),
status: Status::ENABLED.to_string(),
}],
}),
Some(BucketTargets::default()),
@@ -85,6 +85,8 @@ use http_body_util::StreamBody;
use metrics::counter;
#[cfg(test)]
use rmp_serde;
#[cfg(test)]
use rustfs_gateway_types::persistence::PersistedReplicationConfiguration;
use rustfs_s3_types::EventName;
use rustfs_utils::http::{
AMZ_OBJECT_LOCK_LEGAL_HOLD, AMZ_OBJECT_LOCK_MODE, AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE, AMZ_TAGGING_DIRECTIVE,
@@ -92,8 +94,6 @@ use rustfs_utils::http::{
insert_str, replication_target_versions,
};
use rustfs_utils::{DEFAULT_SIP_HASH_KEY, get_env_usize, sip_hash};
#[cfg(test)]
use s3s::dto::ReplicationConfiguration;
use std::collections::{HashMap, HashSet};
use std::fmt::Display;
use std::pin::Pin;
@@ -6579,9 +6579,10 @@ mod tests {
use super::super::replication_target_boundary::{BucketTarget, BucketTargets};
use super::*;
use s3s::dto::{
BucketVersioningStatus, DeleteReplication, DeleteReplicationStatus, Destination, ExcludedPrefix, ReplicationRule,
ReplicationRuleStatus, VersioningConfiguration,
use rustfs_gateway_types::dto::Status;
use rustfs_gateway_types::persistence::PersistedVersioningConfiguration as VersioningConfiguration;
use rustfs_gateway_types::persistence::{
PersistedReplicationDestination, PersistedReplicationRule, PersistedReplicationStatus,
};
use std::collections::HashMap;
use time::OffsetDateTime;
@@ -8693,7 +8694,7 @@ mod tests {
..Default::default()
};
let rcfg = ReplicationConfig::new(
Some(ReplicationConfiguration {
Some(PersistedReplicationConfiguration {
role: role.to_string(),
rules: vec![],
}),
@@ -8719,7 +8720,7 @@ mod tests {
ReplicationVersioningStore::install_prefix_state_test_config(
&bucket,
VersioningConfiguration {
status: Some(BucketVersioningStatus::from_static(BucketVersioningStatus::ENABLED)),
status: Some("Enabled".to_owned()),
..Default::default()
},
);
@@ -8752,22 +8753,20 @@ mod tests {
ReplicationVersioningStore::install_prefix_state_test_config(
&bucket,
VersioningConfiguration {
status: Some(BucketVersioningStatus::from_static(BucketVersioningStatus::ENABLED)),
excluded_prefixes: Some(vec![ExcludedPrefix {
prefix: Some("archive/".to_string()),
}]),
status: Some("Enabled".to_owned()),
excluded_prefixes: Some(vec![Some("archive/".to_string())]),
..Default::default()
},
);
let rcfg = ReplicationConfig::new(
Some(ReplicationConfiguration {
Some(PersistedReplicationConfiguration {
role: String::new(),
rules: vec![ReplicationRule {
rules: vec![PersistedReplicationRule {
delete_marker_replication: None,
delete_replication: Some(DeleteReplication {
status: DeleteReplicationStatus::from_static(DeleteReplicationStatus::ENABLED),
delete_replication: Some(PersistedReplicationStatus {
status: Status::ENABLED.to_string(),
}),
destination: Destination {
destination: PersistedReplicationDestination {
bucket: arn.to_string(),
..Default::default()
},
@@ -8777,7 +8776,7 @@ mod tests {
prefix: Some(String::new()),
priority: Some(1),
source_selection_criteria: None,
status: ReplicationRuleStatus::from_static(ReplicationRuleStatus::ENABLED),
status: Status::ENABLED.to_string(),
}],
}),
Some(BucketTargets {
@@ -8813,14 +8812,14 @@ mod tests {
let admitted_arn = "arn:rustfs:replication:us-east-1:target:admitted";
let current_role = "arn:rustfs:replication:us-east-1:target:current";
let rcfg = ReplicationConfig::new(
Some(ReplicationConfiguration {
Some(PersistedReplicationConfiguration {
role: current_role.to_string(),
rules: vec![ReplicationRule {
rules: vec![PersistedReplicationRule {
delete_marker_replication: None,
delete_replication: Some(DeleteReplication {
status: DeleteReplicationStatus::from_static(DeleteReplicationStatus::DISABLED),
delete_replication: Some(PersistedReplicationStatus {
status: Status::DISABLED.to_string(),
}),
destination: Destination {
destination: PersistedReplicationDestination {
bucket: current_role.to_string(),
..Default::default()
},
@@ -8830,7 +8829,7 @@ mod tests {
prefix: Some(String::new()),
priority: Some(1),
source_selection_criteria: None,
status: ReplicationRuleStatus::from_static(ReplicationRuleStatus::ENABLED),
status: Status::ENABLED.to_string(),
}],
}),
Some(BucketTargets {
@@ -8878,12 +8877,12 @@ mod tests {
#[tokio::test]
async fn heal_rejects_semantically_invalid_replication_config() {
let rcfg = ReplicationConfig::new(
Some(ReplicationConfiguration {
Some(PersistedReplicationConfiguration {
role: String::new(),
rules: vec![ReplicationRule {
rules: vec![PersistedReplicationRule {
delete_marker_replication: None,
delete_replication: None,
destination: Destination {
destination: PersistedReplicationDestination {
bucket: "arn:rustfs:replication:us-east-1:target:bucket".to_string(),
..Default::default()
},
@@ -8893,7 +8892,7 @@ mod tests {
prefix: Some(String::new()),
priority: Some(1),
source_selection_criteria: None,
status: ReplicationRuleStatus::from_static("Enabld"),
status: "Enabld".to_owned(),
}],
}),
Some(BucketTargets::default()),
@@ -12,7 +12,7 @@
// See the License for the specific language governing permissions and
// limitations under the License.
use s3s::dto::ReplicationConfiguration;
use rustfs_gateway_types::persistence::PersistedReplicationConfiguration;
use super::replication_config_boundary::{ObjectOpts, ReplicationConfigurationExt};
use super::replication_filemeta_boundary::ReplicationType;
@@ -20,7 +20,7 @@ use super::replication_filemeta_boundary::ReplicationType;
pub(crate) struct ReplicationTargetConfigBridge;
impl ReplicationTargetConfigBridge {
pub(crate) fn target_is_used_by_rules(config: &ReplicationConfiguration, arn: &str) -> bool {
pub(crate) fn target_is_used_by_rules(config: &PersistedReplicationConfiguration, arn: &str) -> bool {
config
.filter_target_arns(&ObjectOpts {
op_type: ReplicationType::All,
@@ -37,7 +37,7 @@ mod tests {
#[test]
fn target_config_bridge_matches_role_target() {
let config = ReplicationConfiguration {
let config = PersistedReplicationConfiguration {
role: " arn:target ".to_string(),
rules: Vec::new(),
};
@@ -15,7 +15,7 @@
use super::replication_error_boundary::Result;
use crate::bucket::{versioning::VersioningApi as _, versioning_sys::BucketVersioningSys};
#[cfg(test)]
use s3s::dto::VersioningConfiguration;
use rustfs_gateway_types::persistence::PersistedVersioningConfiguration as VersioningConfiguration;
#[cfg(test)]
use std::{collections::HashMap, sync::LazyLock, sync::Mutex};
+1 -1
View File
@@ -12,7 +12,7 @@
// See the License for the specific language governing permissions and
// limitations under the License.
use s3s::dto::Tag;
use rustfs_gateway_types::cors_tagging::PersistedTag as Tag;
use std::cmp::Ordering;
use std::collections::HashMap;
use url::form_urlencoded;
+211 -27
View File
@@ -15,9 +15,23 @@
use crate::disk::{MIGRATING_META_BUCKET, RUSTFS_META_BUCKET};
use crate::error::{Error, Result, StorageError};
use regex::Regex;
use rustfs_gateway_types::cors_tagging::{
CorsTaggingCodecError, PersistedCorsConfiguration, PersistedTagging, parse_cors, parse_tagging, serialize_cors,
serialize_tagging,
};
use rustfs_gateway_types::persistence::{
PersistedAccelerateConfiguration, PersistedBucketEncryptionConfiguration, PersistedBucketLoggingStatus,
PersistedLifecycleConfiguration, PersistedNotificationConfiguration, PersistedObjectLockConfiguration,
PersistedPublicAccessBlockConfiguration, PersistedReplicationConfiguration, PersistedRequestPaymentConfiguration,
PersistedVersioningConfiguration, PersistedWebsiteConfiguration, PersistenceCodecError, parse_accelerate,
parse_bucket_encryption, parse_bucket_logging, parse_lifecycle_legacy_dates, parse_notification, parse_object_lock,
parse_public_access_block, parse_replication, parse_request_payment, parse_versioning, parse_website, serialize_accelerate,
serialize_bucket_encryption, serialize_bucket_logging, serialize_lifecycle_legacy_dates, serialize_notification,
serialize_object_lock, serialize_public_access_block, serialize_replication, serialize_request_payment, serialize_versioning,
serialize_website,
};
use rustfs_utils::WindowsFileNameError;
use rustfs_utils::path::SLASH_SEPARATOR;
use s3s::xml;
use tracing::instrument;
pub fn is_meta_bucketname(name: &str) -> bool {
@@ -74,37 +88,173 @@ pub fn check_valid_bucket_name_strict(bucket_name: &str) -> Result<()> {
check_bucket_name_common(bucket_name, true)
}
// RUSTFS_COMPAT_TODO(s3gate-metadata-xml): the s3s codec reads persisted XML during migration. Remove after every supported writer uses the gateway codec and every retained metadata object and backup archive is verified or rewritten.
pub fn deserialize<T>(input: &[u8]) -> xml::DeResult<T>
where
T: for<'xml> xml::Deserialize<'xml>,
{
let mut d = xml::Deserializer::new(input);
let ans = T::deserialize(&mut d)?;
d.expect_eof()?;
Ok(ans)
/// A bucket configuration persisted as XML in the bucket metadata, together
/// with the codec that reads and writes its stored bytes.
///
/// Every family reads and writes its stored bytes through the gateway
/// persistence codecs (`rustfs_gateway_types::persistence` and
/// `rustfs_gateway_types::cors_tagging`, rustfs/backlog#2745). The codecs keep
/// the persisted bytes the `bucket_config_goldens` pin, so a document the old
/// s3s codec wrote reads back and re-encodes unchanged.
pub trait BucketConfigXml: Sized {
/// Decodes the stored bytes of one configuration document.
fn decode_config_xml(input: &[u8]) -> std::result::Result<Self, BucketConfigXmlError>;
/// Encodes the value as RustFS stores it.
fn encode_config_xml(&self) -> std::result::Result<Vec<u8>, BucketConfigXmlError>;
}
#[allow(
dead_code,
reason = "xml serialize helper with no caller in this port; the live sibling is deserialize (backlog#1823)"
)]
pub fn serialize_content<T: xml::SerializeContent>(val: &T) -> xml::SerResult<String> {
let mut buf = Vec::with_capacity(256);
{
let mut ser = xml::Serializer::new(&mut buf);
val.serialize_content(&mut ser)?;
}
Ok(String::from_utf8(buf).unwrap())
/// Why a persisted bucket configuration could not be decoded or encoded.
#[derive(Debug, thiserror::Error)]
pub enum BucketConfigXmlError {
/// A gateway persistence codec refused the bytes or the value.
#[error(transparent)]
Persisted(#[from] PersistenceCodecError),
/// The gateway CORS or Tagging codec refused the bytes or the value.
#[error(transparent)]
CorsTagging(#[from] CorsTaggingCodecError),
}
pub fn serialize<T: xml::Serialize>(val: &T) -> xml::SerResult<Vec<u8>> {
let mut buf = Vec::with_capacity(256);
{
let mut ser = xml::Serializer::new(&mut buf);
val.serialize(&mut ser)?;
/// Decodes one persisted bucket configuration document.
pub fn deserialize<T: BucketConfigXml>(input: &[u8]) -> std::result::Result<T, BucketConfigXmlError> {
T::decode_config_xml(input)
}
/// Encodes one bucket configuration as RustFS persists it.
pub fn serialize<T: BucketConfigXml>(val: &T) -> std::result::Result<Vec<u8>, BucketConfigXmlError> {
val.encode_config_xml()
}
impl BucketConfigXml for PersistedReplicationConfiguration {
fn decode_config_xml(input: &[u8]) -> std::result::Result<Self, BucketConfigXmlError> {
Ok(parse_replication(input)?)
}
fn encode_config_xml(&self) -> std::result::Result<Vec<u8>, BucketConfigXmlError> {
Ok(serialize_replication(self)?)
}
}
impl BucketConfigXml for PersistedLifecycleConfiguration {
fn decode_config_xml(input: &[u8]) -> std::result::Result<Self, BucketConfigXmlError> {
Ok(parse_lifecycle_legacy_dates(input)?)
}
fn encode_config_xml(&self) -> std::result::Result<Vec<u8>, BucketConfigXmlError> {
Ok(serialize_lifecycle_legacy_dates(self)?)
}
}
// The gateway writers of these families are infallible: a value that exists is
// always encodable, so only the readers can refuse bytes.
impl BucketConfigXml for PersistedAccelerateConfiguration {
fn decode_config_xml(input: &[u8]) -> std::result::Result<Self, BucketConfigXmlError> {
Ok(parse_accelerate(input)?)
}
fn encode_config_xml(&self) -> std::result::Result<Vec<u8>, BucketConfigXmlError> {
Ok(serialize_accelerate(self))
}
}
impl BucketConfigXml for PersistedBucketLoggingStatus {
fn decode_config_xml(input: &[u8]) -> std::result::Result<Self, BucketConfigXmlError> {
Ok(parse_bucket_logging(input)?)
}
fn encode_config_xml(&self) -> std::result::Result<Vec<u8>, BucketConfigXmlError> {
Ok(serialize_bucket_logging(self))
}
}
impl BucketConfigXml for PersistedCorsConfiguration {
fn decode_config_xml(input: &[u8]) -> std::result::Result<Self, BucketConfigXmlError> {
Ok(parse_cors(input)?)
}
fn encode_config_xml(&self) -> std::result::Result<Vec<u8>, BucketConfigXmlError> {
Ok(serialize_cors(self))
}
}
impl BucketConfigXml for PersistedNotificationConfiguration {
fn decode_config_xml(input: &[u8]) -> std::result::Result<Self, BucketConfigXmlError> {
Ok(parse_notification(input)?)
}
fn encode_config_xml(&self) -> std::result::Result<Vec<u8>, BucketConfigXmlError> {
Ok(serialize_notification(self))
}
}
impl BucketConfigXml for PersistedObjectLockConfiguration {
fn decode_config_xml(input: &[u8]) -> std::result::Result<Self, BucketConfigXmlError> {
Ok(parse_object_lock(input)?)
}
fn encode_config_xml(&self) -> std::result::Result<Vec<u8>, BucketConfigXmlError> {
Ok(serialize_object_lock(self))
}
}
impl BucketConfigXml for PersistedPublicAccessBlockConfiguration {
fn decode_config_xml(input: &[u8]) -> std::result::Result<Self, BucketConfigXmlError> {
Ok(parse_public_access_block(input)?)
}
fn encode_config_xml(&self) -> std::result::Result<Vec<u8>, BucketConfigXmlError> {
Ok(serialize_public_access_block(self))
}
}
impl BucketConfigXml for PersistedRequestPaymentConfiguration {
fn decode_config_xml(input: &[u8]) -> std::result::Result<Self, BucketConfigXmlError> {
Ok(parse_request_payment(input)?)
}
fn encode_config_xml(&self) -> std::result::Result<Vec<u8>, BucketConfigXmlError> {
Ok(serialize_request_payment(self))
}
}
impl BucketConfigXml for PersistedBucketEncryptionConfiguration {
fn decode_config_xml(input: &[u8]) -> std::result::Result<Self, BucketConfigXmlError> {
Ok(parse_bucket_encryption(input)?)
}
fn encode_config_xml(&self) -> std::result::Result<Vec<u8>, BucketConfigXmlError> {
Ok(serialize_bucket_encryption(self))
}
}
impl BucketConfigXml for PersistedTagging {
fn decode_config_xml(input: &[u8]) -> std::result::Result<Self, BucketConfigXmlError> {
Ok(parse_tagging(input)?)
}
fn encode_config_xml(&self) -> std::result::Result<Vec<u8>, BucketConfigXmlError> {
Ok(serialize_tagging(self))
}
}
impl BucketConfigXml for PersistedVersioningConfiguration {
fn decode_config_xml(input: &[u8]) -> std::result::Result<Self, BucketConfigXmlError> {
Ok(parse_versioning(input)?)
}
fn encode_config_xml(&self) -> std::result::Result<Vec<u8>, BucketConfigXmlError> {
Ok(serialize_versioning(self))
}
}
impl BucketConfigXml for PersistedWebsiteConfiguration {
fn decode_config_xml(input: &[u8]) -> std::result::Result<Self, BucketConfigXmlError> {
Ok(parse_website(input)?)
}
fn encode_config_xml(&self) -> std::result::Result<Vec<u8>, BucketConfigXmlError> {
Ok(serialize_website(self))
}
Ok(buf)
}
pub fn has_bad_path_component(path: &str) -> bool {
@@ -446,6 +596,40 @@ mod tests {
use super::*;
use proptest::prelude::*;
/// Every bucket-configuration family reads and writes through its own
/// gateway codec (rustfs/backlog#2745): the family's golden document
/// decodes and re-encodes to the same bytes, and a document of another
/// family is refused instead of decoding into an empty configuration.
#[test]
fn every_bucket_config_family_uses_its_own_gateway_codec() {
fn assert_family<T: BucketConfigXml>(family: &str, own: &[u8], foreign: &[u8]) {
let value = deserialize::<T>(own).unwrap_or_else(|err| panic!("{family}: own document must decode: {err}"));
let encoded = serialize(&value).unwrap_or_else(|err| panic!("{family}: value must encode: {err}"));
assert_eq!(encoded, own, "{family}: re-encoding must keep the persisted bytes");
assert!(deserialize::<T>(foreign).is_err(), "{family}: a foreign root element must be refused");
}
macro_rules! golden {
($family:literal) => {
include_bytes!(concat!("../../tests/fixtures/bucket-config-goldens/", $family, "/minio.out.xml"))
};
}
let tagging = golden!("tagging");
let versioning = golden!("versioning");
assert_family::<PersistedAccelerateConfiguration>("accelerate", golden!("accelerate"), tagging);
assert_family::<PersistedBucketLoggingStatus>("logging", golden!("logging"), tagging);
assert_family::<PersistedCorsConfiguration>("cors", golden!("cors"), tagging);
assert_family::<PersistedNotificationConfiguration>("notification", golden!("notification"), tagging);
assert_family::<PersistedObjectLockConfiguration>("object_lock", golden!("object_lock"), tagging);
assert_family::<PersistedPublicAccessBlockConfiguration>("public_access_block", golden!("public_access_block"), tagging);
assert_family::<PersistedRequestPaymentConfiguration>("request_payment", golden!("request_payment"), tagging);
assert_family::<PersistedBucketEncryptionConfiguration>("encryption", golden!("encryption"), tagging);
assert_family::<PersistedTagging>("tagging", tagging, versioning);
assert_family::<PersistedVersioningConfiguration>("versioning", versioning, tagging);
assert_family::<PersistedWebsiteConfiguration>("website", golden!("website"), tagging);
}
/// rustfs#5785: keys whose path segments exceed the on-disk NAME_MAX
/// budget must be rejected up front as ObjectNameInvalid (4xx), not leak
/// ENAMETOOLONG as InternalError 500 from the disk layer.
+17 -25
View File
@@ -12,8 +12,8 @@
// See the License for the specific language governing permissions and
// limitations under the License.
use rustfs_gateway_types::persistence::PersistedVersioningConfiguration as VersioningConfiguration;
use rustfs_utils::string::match_simple;
use s3s::dto::{BucketVersioningStatus, VersioningConfiguration};
pub trait VersioningApi {
fn enabled(&self) -> bool;
@@ -28,10 +28,10 @@ pub trait VersioningApi {
impl VersioningApi for VersioningConfiguration {
fn enabled(&self) -> bool {
self.status == Some(BucketVersioningStatus::from_static(BucketVersioningStatus::ENABLED))
self.status.as_deref() == Some("Enabled")
}
fn prefix_enabled(&self, prefix: &str) -> bool {
if self.status != Some(BucketVersioningStatus::from_static(BucketVersioningStatus::ENABLED)) {
if self.status.as_deref() != Some("Enabled") {
return false;
}
@@ -44,12 +44,10 @@ impl VersioningApi for VersioningConfiguration {
}
if let Some(ref excluded_prefixes) = self.excluded_prefixes {
for p in excluded_prefixes.iter() {
if let Some(ref sprefix) = p.prefix {
let pattern = format!("{sprefix}*");
if match_simple(&pattern, prefix) {
return false;
}
for sprefix in excluded_prefixes.iter().flatten() {
let pattern = format!("{sprefix}*");
if match_simple(&pattern, prefix) {
return false;
}
}
}
@@ -58,11 +56,11 @@ impl VersioningApi for VersioningConfiguration {
}
fn prefix_suspended(&self, prefix: &str) -> bool {
if self.status == Some(BucketVersioningStatus::from_static(BucketVersioningStatus::SUSPENDED)) {
if self.status.as_deref() == Some("Suspended") {
return true;
}
if self.status == Some(BucketVersioningStatus::from_static(BucketVersioningStatus::ENABLED)) {
if self.status.as_deref() == Some("Enabled") {
if prefix.is_empty() {
return false;
}
@@ -75,12 +73,10 @@ impl VersioningApi for VersioningConfiguration {
}
if let Some(ref excluded_prefixes) = self.excluded_prefixes {
for p in excluded_prefixes.iter() {
if let Some(ref sprefix) = p.prefix {
let pattern = format!("{sprefix}*");
if match_simple(&pattern, prefix) {
return true;
}
for sprefix in excluded_prefixes.iter().flatten() {
let pattern = format!("{sprefix}*");
if match_simple(&pattern, prefix) {
return true;
}
}
}
@@ -92,14 +88,12 @@ impl VersioningApi for VersioningConfiguration {
self.prefix_enabled(prefix) || self.prefix_suspended(prefix)
}
fn suspended(&self) -> bool {
self.status == Some(BucketVersioningStatus::from_static(BucketVersioningStatus::SUSPENDED))
self.status.as_deref() == Some("Suspended")
}
}
#[cfg(test)]
mod tests {
use s3s::dto::{BucketVersioningStatus, ExcludedPrefix};
use super::*;
struct LegacyVersioning;
@@ -134,10 +128,8 @@ mod tests {
#[test]
fn delete_state_treats_excluded_prefixes_as_unversioned() {
let config = VersioningConfiguration {
status: Some(BucketVersioningStatus::from_static(BucketVersioningStatus::ENABLED)),
excluded_prefixes: Some(vec![ExcludedPrefix {
prefix: Some("archive/".to_string()),
}]),
status: Some("Enabled".to_owned()),
excluded_prefixes: Some(vec![Some("archive/".to_string())]),
..Default::default()
};
@@ -146,7 +138,7 @@ mod tests {
assert_eq!(config.delete_state("live/object"), (true, false));
let suspended = VersioningConfiguration {
status: Some(BucketVersioningStatus::from_static(BucketVersioningStatus::SUSPENDED)),
status: Some("Suspended".to_owned()),
..Default::default()
};
assert_eq!(suspended.delete_state("archive/object"), (false, true));
+2 -2
View File
@@ -17,7 +17,7 @@ use super::metadata::unreadable_config_refusal;
use super::{metadata_sys::get_bucket_metadata_sys, versioning::VersioningApi};
use crate::disk::RUSTFS_META_BUCKET;
use crate::error::Result;
use s3s::dto::VersioningConfiguration;
use rustfs_gateway_types::persistence::PersistedVersioningConfiguration as VersioningConfiguration;
use tracing::{error, warn};
pub struct BucketVersioningSys {}
@@ -163,7 +163,7 @@ mod tests {
fn enabled() -> VersioningConfiguration {
VersioningConfiguration {
status: Some(s3s::dto::BucketVersioningStatus::from_static(s3s::dto::BucketVersioningStatus::ENABLED)),
status: Some("Enabled".to_owned()),
..Default::default()
}
}
@@ -1965,6 +1965,15 @@ impl PeerRestClient {
Ok((self.topology_member.clone(), epoch))
}
pub async fn probe_restore_copyback_capability(&self, topology_fingerprint: String) -> Result<(String, Uuid)> {
let probe = rustfs_protos::restore_copyback_capability_probe(Uuid::new_v4().as_bytes());
let result = self
.heal_control(rustfs_protos::HEAL_CONTROL_PROTOCOL_VERSION, topology_fingerprint, probe)
.await?;
let epoch = decode_remote_version_state_capability(&self.topology_member, &result)?;
Ok((self.topology_member.clone(), epoch))
}
pub async fn probe_cross_pool_fence(&self, topology_fingerprint: String) -> Result<(String, u32, Uuid)> {
let mut probe = rustfs_protos::CROSS_POOL_FENCE_CAPABILITY_PROBE_PREFIX.to_vec();
probe.extend_from_slice(Uuid::new_v4().as_bytes());
+98 -22
View File
@@ -931,8 +931,16 @@ pub struct RemoteDisk {
#[cfg(test)]
recovery_monitor_teardown_hook: Arc<tokio::sync::Mutex<Option<Arc<RecoveryMonitorTeardownHook>>>>,
data_transport: Arc<dyn InternodeDataTransport>,
/// Bound the per-operation failure warnings of this peer. An unreachable
/// peer fails every in-flight shard operation; the counters keep the exact
/// numbers per operation. Per handle, so one peer cannot hide another's.
timeout_logs: rustfs_utils::LogThrottle,
network_error_logs: rustfs_utils::LogThrottle,
}
/// Shortest gap between two failure warnings of one kind from one remote disk.
const REMOTE_DISK_RPC_FAILURE_LOG_INTERVAL_MS: u64 = 5_000;
struct RecoveryMonitorLease {
active: Arc<AtomicBool>,
}
@@ -1364,6 +1372,8 @@ impl RemoteDisk {
#[cfg(test)]
recovery_monitor_teardown_hook: Arc::new(tokio::sync::Mutex::new(None)),
data_transport,
timeout_logs: rustfs_utils::LogThrottle::new(REMOTE_DISK_RPC_FAILURE_LOG_INTERVAL_MS),
network_error_logs: rustfs_utils::LogThrottle::new(REMOTE_DISK_RPC_FAILURE_LOG_INTERVAL_MS),
};
record_drive_runtime_state(ep, RuntimeDriveHealthState::Online);
@@ -2090,17 +2100,20 @@ impl RemoteDisk {
"op" => op.to_string()
)
.increment(1);
warn!(
event = EVENT_REMOTE_DISK_RPC,
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_REMOTE_DISK,
endpoint = %self.endpoint,
addr = %self.addr,
op,
timeout_ms = timeout_duration.as_millis(),
state = "timeout",
"Remote disk operation timed out"
);
if let Some(suppressed) = self.timeout_logs.claim() {
warn!(
event = EVENT_REMOTE_DISK_RPC,
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_REMOTE_DISK,
endpoint = %self.endpoint,
addr = %self.addr,
op,
timeout_ms = timeout_duration.as_millis(),
suppressed,
state = "timeout",
"Remote disk operation timed out"
);
}
}
async fn handle_network_like_error<T>(
@@ -2120,17 +2133,20 @@ impl RemoteDisk {
"op" => op.to_string()
)
.increment(1);
warn!(
event = EVENT_REMOTE_DISK_RPC,
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_REMOTE_DISK,
endpoint = %self.endpoint,
addr = %self.addr,
op,
timeout_ms = timeout_duration.as_millis(),
state = "network_like_error",
"Remote disk operation returned network-like error"
);
if let Some(suppressed) = self.network_error_logs.claim() {
warn!(
event = EVENT_REMOTE_DISK_RPC,
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_REMOTE_DISK,
endpoint = %self.endpoint,
addr = %self.addr,
op,
timeout_ms = timeout_duration.as_millis(),
suppressed,
state = "network_like_error",
"Remote disk operation returned network-like error"
);
}
if failure_health_action == FailureHealthAction::MarkFailure {
self.mark_faulty_and_evict("operation_network_error").await;
}
@@ -11688,6 +11704,66 @@ mod tests {
);
}
/// A peer that fails every operation logs one warning per interval, each
/// operation still gets its own error, and another peer's first warning is
/// not held back by the noisy one.
#[tokio::test]
async fn rpc_failure_warnings_are_bounded_per_remote_disk() {
async fn remote_disk(port: u16) -> RemoteDisk {
let url = url::Url::parse(&format!("http://127.0.0.1:{port}/data")).expect("endpoint url");
let endpoint = Endpoint {
url,
is_local: false,
pool_idx: 0,
set_idx: 0,
disk_idx: 0,
};
RemoteDisk::new(
&endpoint,
&DiskOption {
cleanup: false,
health_check: false,
},
Arc::new(TcpHttpInternodeDataTransport),
)
.await
.expect("remote disk handle")
}
let noisy = remote_disk(59981).await;
let quiet = remote_disk(59982).await;
for _ in 0..50 {
let error = noisy
.execute_with_timeout_for_op_and_health_action(
"read_all",
|| async {
Err::<(), Error>(DiskError::Io(std::io::Error::new(
std::io::ErrorKind::ConnectionRefused,
"connection refused",
)))
},
Duration::from_secs(1),
FailureHealthAction::IgnoreFailure,
)
.await
.expect_err("every failing operation still fails");
assert!(matches!(error, DiskError::Io(io) if io.kind() == std::io::ErrorKind::ConnectionRefused));
noisy.record_timeout("read_all", Duration::from_secs(1));
}
assert!(
noisy.network_error_logs.claim().is_none(),
"the burst used this peer's network-error window"
);
assert!(noisy.timeout_logs.claim().is_none(), "the burst used this peer's timeout window");
assert_eq!(
quiet.network_error_logs.claim(),
Some(0),
"another peer's first warning is not suppressed"
);
assert_eq!(quiet.timeout_logs.claim(), Some(0));
}
#[tokio::test]
async fn test_execute_with_timeout_keeps_remote_disk_online_for_business_error() {
let addr = "http://127.0.0.1:59994".to_string();
+14 -13
View File
@@ -84,12 +84,13 @@ use rand::RngExt as _;
use rmp_serde::Deserializer;
use rmp_serde::Serializer;
use rustfs_filemeta::{FileInfo, FileInfoVersions, MetaCacheEntries, MetaCacheEntry, MetadataResolutionParams};
use rustfs_gateway_types::persistence::PersistedObjectLockConfiguration as ObjectLockConfiguration;
use rustfs_gateway_types::persistence::{PersistedLifecycleConfiguration, PersistedReplicationConfiguration};
use rustfs_heal_contracts::heal_channel::HealOpts;
use rustfs_utils::crypto::{hex_sha256, is_sha256_checksum};
use rustfs_utils::path::{
decode_dir_object, encode_dir_object, path_join, path_to_bucket_object, path_to_bucket_object_with_base_path,
};
use s3s::dto::{BucketLifecycleConfiguration, ObjectLockConfiguration, ReplicationConfiguration};
use serde::{Deserialize, Serialize};
use sha2::{Digest, Sha256};
use std::collections::{HashMap, HashSet};
@@ -13115,7 +13116,7 @@ pub(crate) async fn should_skip_lifecycle_for_data_movement(
store: Arc<ECStore>,
bucket: &str,
version: &rustfs_filemeta::FileInfo,
lifecycle_config: Option<&BucketLifecycleConfiguration>,
lifecycle_config: Option<&PersistedLifecycleConfiguration>,
object_lock_config: Option<&ObjectLockConfiguration>,
apply_actions: bool,
event_source: &LcEventSrc,
@@ -17046,9 +17047,9 @@ impl ECStore {
generation: OffsetDateTime,
bucket: String,
set: Arc<SetDisks>,
lifecycle_config: Option<BucketLifecycleConfiguration>,
lifecycle_config: Option<PersistedLifecycleConfiguration>,
object_lock_config: Option<ObjectLockConfiguration>,
replication_config: Option<(ReplicationConfiguration, OffsetDateTime)>,
replication_config: Option<(PersistedReplicationConfiguration, OffsetDateTime)>,
expected_bucket_incarnation_id: Option<uuid::Uuid>,
source_changed_exhaustions: Arc<AtomicUsize>,
entry_budget: Arc<Semaphore>,
@@ -17220,9 +17221,9 @@ impl ECStore {
mut deferred_entry: DeferredDecommissionEntry,
bucket: String,
set: Arc<SetDisks>,
lifecycle_config: Option<BucketLifecycleConfiguration>,
lifecycle_config: Option<PersistedLifecycleConfiguration>,
object_lock_config: Option<ObjectLockConfiguration>,
replication_config: Option<(ReplicationConfiguration, OffsetDateTime)>,
replication_config: Option<(PersistedReplicationConfiguration, OffsetDateTime)>,
expected_bucket_incarnation_id: Option<uuid::Uuid>,
source_changed_exhaustions: Arc<AtomicUsize>,
entry_budget: Arc<Semaphore>,
@@ -17329,9 +17330,9 @@ impl ECStore {
generation: OffsetDateTime,
set: Arc<SetDisks>,
bi: DecomBucketInfo,
lifecycle_config: Option<BucketLifecycleConfiguration>,
lifecycle_config: Option<PersistedLifecycleConfiguration>,
object_lock_config: Option<ObjectLockConfiguration>,
replication_config: Option<(ReplicationConfiguration, OffsetDateTime)>,
replication_config: Option<(PersistedReplicationConfiguration, OffsetDateTime)>,
expected_bucket_incarnation_id: Option<uuid::Uuid>,
source_changed_exhaustions: Arc<AtomicUsize>,
entry_budget: Arc<Semaphore>,
@@ -18016,9 +18017,9 @@ impl ECStore {
entry: MetaCacheEntry,
bucket: String,
set: Arc<SetDisks>,
lifecycle_config: Option<BucketLifecycleConfiguration>,
lifecycle_config: Option<PersistedLifecycleConfiguration>,
object_lock_config: Option<ObjectLockConfiguration>,
replication_config: Option<(ReplicationConfiguration, OffsetDateTime)>,
replication_config: Option<(PersistedReplicationConfiguration, OffsetDateTime)>,
expected_bucket_incarnation_id: Option<uuid::Uuid>,
source_changed_exhaustions: Arc<AtomicUsize>,
attempts: &mut usize,
@@ -18202,9 +18203,9 @@ impl ECStore {
entry: MetaCacheEntry,
bucket: String,
set: Arc<SetDisks>,
lifecycle_config: Option<BucketLifecycleConfiguration>,
lifecycle_config: Option<PersistedLifecycleConfiguration>,
object_lock_config: Option<ObjectLockConfiguration>,
replication_config: Option<(ReplicationConfiguration, OffsetDateTime)>,
replication_config: Option<(PersistedReplicationConfiguration, OffsetDateTime)>,
expected_bucket_incarnation_id: Option<uuid::Uuid>,
entry_attempt: usize,
scanner_backlog_handoff_attempt: usize,
@@ -19490,7 +19491,7 @@ impl ECStore {
entry: MetaCacheEntry,
bucket: String,
set: Arc<SetDisks>,
lifecycle_config: Option<BucketLifecycleConfiguration>,
lifecycle_config: Option<PersistedLifecycleConfiguration>,
) -> Result<()> {
let expected_bucket_incarnation_id = if is_meta_bucketname(&bucket) {
None
+40
View File
@@ -65,6 +65,38 @@ use uuid::Uuid;
const LIST_MULTIPART_SETS_CONCURRENCY: usize = 4;
#[cfg(test)]
type TestSetLockers = HashMap<usize, Vec<Arc<dyn rustfs_lock::LockClient>>>;
#[cfg(test)]
static TEST_SET_LOCKERS: std::sync::LazyLock<std::sync::Mutex<TestSetLockers>> =
std::sync::LazyLock::new(|| std::sync::Mutex::new(HashMap::new()));
/// Keep a fixture's real locker domain scoped to its owning instance.
#[cfg(test)]
pub(crate) struct ScopedSetLockers(Arc<InstanceContext>);
#[cfg(test)]
impl ScopedSetLockers {
pub(crate) fn install(ctx: Arc<InstanceContext>, lockers: Vec<Arc<dyn rustfs_lock::LockClient>>) -> Self {
let key = Arc::as_ptr(&ctx) as usize;
let mut domains = TEST_SET_LOCKERS.lock().expect("test set lockers mutex");
assert!(!domains.contains_key(&key), "the instance already owns a test locker domain");
domains.insert(key, lockers);
Self(ctx)
}
}
#[cfg(test)]
impl Drop for ScopedSetLockers {
fn drop(&mut self) {
TEST_SET_LOCKERS
.lock()
.expect("test set lockers mutex")
.remove(&(Arc::as_ptr(&self.0) as usize));
}
}
fn is_idempotent_delete_prefix_error(err: &Error) -> bool {
is_err_object_not_found(err) || is_err_strict_volume_not_found(err)
}
@@ -266,6 +298,14 @@ impl Sets {
}
}
#[cfg(test)]
let pool_lockers = TEST_SET_LOCKERS
.lock()
.expect("test set lockers mutex")
.get(&(Arc::as_ptr(&instance_ctx) as usize))
.cloned()
.unwrap_or_else(|| pool_lockers.clone());
let set_disks = SetDisks::new_with_instance_ctx(
runtime_sources::local_node_name().await,
Arc::new(RwLock::new(set_drive)),
+9
View File
@@ -3053,6 +3053,15 @@ impl DiskAPI for LocalDiskWrapper {
})
}
async fn bucket_creation_witness(&self, volume: &str) -> Result<Option<super::os::NativeBucketCreationWitness>> {
self.track_disk_health_with_op(
"bucket_creation_witness",
|| async { self.disk.bucket_creation_witness(volume).await },
Duration::ZERO,
)
.await
}
async fn make_volume(&self, volume: &str) -> Result<()> {
// Scoped heal must drain directory creation before releasing its lifecycle owner.
let timeout = if crate::store::bucket_heal_scope(volume).is_some() {
+7
View File
@@ -92,6 +92,9 @@ pub const O_APPEND: FileMode = 0x00400;
// create_new: bool,
pub async fn open_file(path: impl AsRef<Path>, mode: FileMode) -> io::Result<File> {
if mode & O_CREATE != 0 {
super::os::record_native_publication(path.as_ref());
}
let path = path.as_ref().to_path_buf();
native_io(move || {
let mut options = std::fs::OpenOptions::new();
@@ -144,6 +147,7 @@ pub fn create_dir_all_std(path: &Path) -> io::Result<()> {
}
pub async fn make_dir_all(path: impl AsRef<Path>) -> io::Result<()> {
super::os::record_native_publication(path.as_ref());
let path = path.as_ref().to_path_buf();
native_io(move || create_dir_all_std(&path)).await
}
@@ -216,11 +220,13 @@ pub fn remove_all_std(path: impl AsRef<Path>) -> io::Result<()> {
#[cfg(test)]
pub async fn mkdir(path: impl AsRef<Path>) -> io::Result<()> {
super::os::record_native_publication(path.as_ref());
let path = path.as_ref().to_path_buf();
native_io(move || std::fs::create_dir(path)).await
}
pub async fn rename(from: impl AsRef<Path>, to: impl AsRef<Path>) -> io::Result<()> {
super::os::record_native_publication(to.as_ref());
let from = from.as_ref().to_path_buf();
let to = to.as_ref().to_path_buf();
native_io(move || std::fs::rename(from, to)).await
@@ -228,6 +234,7 @@ pub async fn rename(from: impl AsRef<Path>, to: impl AsRef<Path>) -> io::Result<
#[cfg(any(not(windows), test))]
pub fn rename_std(from: impl AsRef<Path>, to: impl AsRef<Path>) -> io::Result<()> {
super::os::record_native_publication(to.as_ref());
std::fs::rename(from, to)
}
File diff suppressed because it is too large Load Diff
+30 -5
View File
@@ -286,12 +286,15 @@ pub(super) fn fsync_inline_commit_std(
if destination_was_absent {
let mut ancestor = dst.parent().and_then(Path::parent);
let mut ancestor_step = 0usize;
while let Some(ancestor_dir) = ancestor {
if !ancestor_dir.starts_with(dst_volume_dir) {
break;
}
let fsync_started = rustfs_io_metrics::put_stage_timer();
if let Err(err) = os::fsync_dir_limited_std(ancestor_dir) {
if let Err(err) =
os::fsync_commit_ancestor_directory_limited_std(ancestor_dir, dst_volume_dir, ancestor_step, commit_guard)
{
rustfs_io_metrics::record_put_object_stage_duration_from(
rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_ANCESTOR_DIR_FSYNC,
fsync_started,
@@ -310,6 +313,7 @@ pub(super) fn fsync_inline_commit_std(
break;
}
ancestor = ancestor_dir.parent();
ancestor_step += 1;
}
}
Ok(())
@@ -517,8 +521,12 @@ impl LocalDisk {
state.preflight_rejection = Some(LocalRenamePreflightRejection(()));
})?;
crate::hp_guard!("LocalDisk::rename_data");
if dst_volume == super::super::RUSTFS_META_MULTIPART_BUCKET {
self.multipart_index.record_info(dst_path, &fi).await?;
if dst_volume == super::super::RUSTFS_META_MULTIPART_BUCKET
&& let Some(pin) = self.multipart_index.record_info(dst_path, &fi).await?
{
// The owner chain reaches every lease this rename publishes under,
// so the index marker stays pinned until that work has returned.
state.namespace_owner = Some(Arc::new((state.namespace_owner.take(), pin)));
}
#[cfg(feature = "e2e-test-hooks")]
if fi.is_healing() {
@@ -752,8 +760,20 @@ impl LocalDisk {
};
let dst_file_path = self.io_get_object_open_path(dst_volume, format!("{}/{}", dst_path, STORAGE_FORMAT_FILE).as_str())?;
// Multipart restore completion has validated staged local parts,
// while its public status stays pending until finalization.
let pending_multipart_restore = src_volume == crate::disk::RUSTFS_META_MULTIPART_BUCKET
&& fi
.metadata
.get(rustfs_filemeta::metadata_keys::RESTORE)
.and_then(|value| rustfs_filemeta::parse_restore_obj_status(value).ok())
.is_some_and(|status| status.is_restore_in_progress == Some(true));
if pending_multipart_restore && fi.data_dir.is_none_or(|dir| dir.is_nil()) {
return Err(DiskError::FileCorrupt);
}
// data_dir path (only used by the non-inline commit branch)
let has_data_dir_path = if no_inline && !fi.is_remote() {
let has_data_dir_path = if no_inline && (!fi.is_remote() || pending_multipart_restore) {
if let Some(data_dir) = fi
.data_dir
.map(|dir| rustfs_utils::path::retain_slash(dir.to_string().as_str()))
@@ -1461,7 +1481,12 @@ impl LocalDisk {
}
}
}
match publication_owned.finish_on_thread(sync_commit_metadata, !cfg!(test), staging_cleanup) {
match publication_owned.finish_on_thread(
sync_commit_metadata,
!cfg!(test),
staging_cleanup,
Some(&directory_guard_owned),
) {
Ok(result) => Ok(MultipartFusedCommitOutcome::Committed(Box::new(RenameDataResp {
multipart_source: Some(Box::new(result)),
old_data_dir: has_old_data_dir,
+396 -131
View File
@@ -41,19 +41,53 @@ struct DiscoverySession {
touched: Instant,
}
/// Plain (non-transaction) children of an object-scoped discovery root cost one
/// budget entry per this many names: enough to bound a hostile prefix without
/// making a real folder prefix undeletable.
/// Names an object-scoped scan skips without a stat (plain children of the
/// key's directory, and entries of a UUID child that cannot be delete evidence)
/// cost one budget entry per this many names: enough to bound a hostile prefix
/// without making a real folder prefix undeletable.
const PLAIN_CHILDREN_PER_BUDGET_ENTRY: u32 = 64;
/// Entries one step of the resumable (background) discovery may visit. The
/// foreground bounded protocol has its own, larger per-call allowance.
const DISCOVERY_STEP_ENTRIES: u32 = 4096;
/// Wall-clock allowance of one bounded discovery call: a third of the drive
/// timeout that wraps the call (including its wait for a lane worker), capped at
/// ten seconds, so an incomplete walk returns before the call can time out.
fn bounded_discovery_time_limit() -> std::time::Duration {
(crate::disk::disk_store::get_max_timeout_duration() / 3).min(std::time::Duration::from_secs(10))
}
/// Whether an entry inside a UUID child of an object-scoped root can name one
/// of the key's delete transactions: a journal file of a rollback directory or
/// a data-directory marker. Every other name is inert to discovery.
fn may_be_delete_evidence(name: &str) -> bool {
[PREPARED, STORAGE_FORMAT_FILE_BACKUP, DELETE_QUORUM_COMMITTED_FILE].contains(&name)
|| [
DELETE_DATA_DIR_MARKER_PREFIX,
RESERVED_DELETE_DATA_DIR_MARKER_PREFIX,
RECOVERY_DELETE_DATA_DIR_MARKER_PREFIX,
]
.iter()
.any(|prefix| delete_transaction_marker_id(name, prefix).is_some())
}
impl DiscoverySession {
/// Consume one bounded step, returning ownership only when traversal remains.
fn advance(self, limit: u32) -> std::io::Result<(Vec<DeleteTransactionId>, Option<Self>)> {
self.advance_bounded(limit, super::super::DELETE_RECOVERY_MAX_DISCOVERY_ENTRIES)
self.advance_bounded(limit, DISCOVERY_STEP_ENTRIES, &|| false)
.map(|(transactions, next, _)| (transactions, next))
}
fn advance_bounded(mut self, limit: u32, max_entries: u32) -> std::io::Result<(Vec<DeleteTransactionId>, Option<Self>, u32)> {
/// Walk until `limit` transactions, `max_entries` charged entries, the end
/// of the scope, or `should_stop` (checked before every entry); any early
/// stop returns the remaining traversal as continuation state.
fn advance_bounded(
mut self,
limit: u32,
max_entries: u32,
should_stop: &dyn Fn() -> bool,
) -> std::io::Result<(Vec<DeleteTransactionId>, Option<Self>, u32)> {
if self.frames.is_empty() {
match std::fs::read_dir(&self.root) {
Ok(entries) => self.frames.push((String::new(), entries)),
@@ -66,10 +100,11 @@ impl DiscoverySession {
let mut transactions = BTreeSet::new();
let mut seen_candidates = HashSet::new();
let mut visited = 0u32;
let mut plain_children = 0u32;
let mut skipped_by_name = 0u32;
while !self.frames.is_empty()
&& transactions.len() < usize::try_from(limit).map_err(|_| std::io::Error::from(ErrorKind::InvalidInput))?
&& visited < max_entries
&& !should_stop()
{
let (parent, entries) = self
.frames
@@ -90,13 +125,22 @@ impl DiscoverySession {
return Err(std::io::Error::from(ErrorKind::InvalidData));
}
// An object-scoped scan only ever descends into transaction (UUID)
// directories at the root. Plain child keys of a prefix directory are
// skipped by name without a stat; they are charged at one entry per
// readdir batch so the work stays bounded while deleting a missing key
// that names a prefix with thousands of children can still complete.
if self.object.is_some() && parent.is_empty() && !canonical_uuid(&name).is_some_and(|id| !id.is_nil()) {
plain_children += 1;
if plain_children.is_multiple_of(PLAIN_CHILDREN_PER_BUDGET_ENTRY) {
// directories at the root, and inside them only journal files and
// data-directory markers can be evidence. Every other name (plain
// child keys of a prefix directory, and the `xl.meta`, data
// directories or children of a UUID-named child key) is skipped by
// name without a stat and charged at one entry per readdir batch, so
// the work stays bounded while deleting a missing key that names a
// folder prefix with many children, UUID-named or not, can complete.
if self.object.is_some()
&& if parent.is_empty() {
!canonical_uuid(&name).is_some_and(|id| !id.is_nil())
} else {
!may_be_delete_evidence(&name)
}
{
skipped_by_name += 1;
if skipped_by_name.is_multiple_of(PLAIN_CHILDREN_PER_BUDGET_ENTRY) {
visited += 1;
}
continue;
@@ -1128,6 +1172,8 @@ struct PreparedDeleteReceiptEntry {
journal_raw: Vec<u8>,
journal: DeleteRecoveryJournal,
sole_transaction: bool,
had_old_data_dir: bool,
rollback_dir_fd: Option<Arc<std::fs::File>>,
}
#[derive(Clone)]
@@ -1137,6 +1183,8 @@ struct PublishedDeleteReceiptEntry {
file_path: PathBuf,
staging_root: PathBuf,
journal_raw: Vec<u8>,
had_old_data_dir: bool,
rollback_dir_fd: Option<Arc<std::fs::File>>,
}
const PREPARED_DELETE_RECEIPTS_PER_SHARD: usize = 64;
@@ -1387,8 +1435,16 @@ impl LocalDisk {
self.validate_delete_discovery_owner(&session)?;
// The blocking owner drops every directory handle even if the RPC waiter disappears.
// A partial inventory is never evidence of absence and never enters DISCOVERY_SESSIONS.
os::run_blocking_namespace_operation(lease, move || {
let (transactions, next, visited) = session.advance_bounded(256, max_entries)?;
// A large folder prefix may take a while to walk, so the probe rides the
// durable-mutation lane like bounded inspection, never the one-worker
// foreground lane that every other mutation on the drive queues behind.
// It also stops, incomplete, when its waiter goes away or well before the
// drive timeout, so a slow walk answers SlowDown instead of marking the
// drive failed or holding a lane worker after the request is gone.
let deadline = Instant::now() + bounded_discovery_time_limit();
os::run_cancellable_blocking_namespace_operation_with_sync(lease, true, move |is_cancelled| {
let should_stop = || is_cancelled() || Instant::now() >= deadline;
let (transactions, next, visited) = session.advance_bounded(256, max_entries, &should_stop)?;
Ok(super::super::BoundedDeleteRecoveryDiscovery {
transactions: if next.is_none() { transactions } else { Vec::new() },
visited,
@@ -1829,7 +1885,9 @@ impl LocalDisk {
) -> Result<DeleteRecoveryInspection> {
self.reject_multipart_delete_recovery_mutation(volume, object)?;
if let Ok(root) = self.io_get_object_path(volume, object) {
let _ = take_prepared_delete_receipt(&root.join(transaction.to_string()));
let tx_dir = root.join(transaction.to_string());
let _ = take_prepared_delete_receipt(&tx_dir);
let _ = take_published_delete_receipt(&tx_dir);
}
let (mut observed, mut has_data_dir_work) = self
@@ -2493,8 +2551,9 @@ impl LocalDisk {
Err(err) if err.kind() == ErrorKind::NotFound => return Ok(Ok(None)),
Err(err) => return Err(err),
};
type PreparedDeleteTuple = (Option<Uuid>, Option<Vec<u8>>, Vec<u8>, DeleteRecoveryJournal, bool);
let (old_dir, target, journal_raw, journal, sole_transaction) =
type PreparedDeleteTuple =
(Option<Uuid>, Option<Vec<u8>>, Vec<u8>, DeleteRecoveryJournal, bool, bool, Option<bool>);
let (old_dir, target, journal_raw, journal, sole_transaction, destination_missing, old_data_dir_state) =
match (|| -> Result<Option<PreparedDeleteTuple>> {
let mut meta = FileMeta::load(&buf)?;
let mut deleted_headers: Vec<_> = meta
@@ -2511,13 +2570,12 @@ impl LocalDisk {
if old_dir.is_some_and(|dir| dir == transaction || dir.is_nil()) {
return Err(DiskError::FileCorrupt);
}
let mut old_path_to_verify = None;
if let Some(uuid) = old_dir {
meta.data.remove(vec![fi_clone.version_id.unwrap_or_default(), uuid])?;
let old_path = path_join(&[file_path.as_path(), Path::new(uuid.to_string().as_str())]);
check_path_length(old_path.to_string_lossy().as_ref())?;
if old_path.join(STORAGE_FORMAT_FILE).exists() {
return Ok(None);
}
old_path_to_verify = Some(old_path);
}
let surviving: HashSet<_> = meta
.versions
@@ -2529,7 +2587,11 @@ impl LocalDisk {
let target = (!meta.versions.is_empty()).then(|| meta.marshal_msg()).transpose()?;
let mut predecessors = BTreeSet::new();
let mut sole_transaction = true;
let mut destination_exists = false;
let mut old_data_dir_exists = false;
let mut scanned_file_path = false;
if let Ok(entries) = std::fs::read_dir(&file_path) {
scanned_file_path = true;
for (visited, entry) in
recovery_entries(entries, recovery_scope.as_ref().map(|scope| scope.budget.as_ref())).enumerate()
{
@@ -2542,13 +2604,19 @@ impl LocalDisk {
sole_transaction = false;
continue;
};
let Some(id) = Uuid::parse_str(name_str).ok().filter(|id| !id.is_nil() && *id != transaction)
else {
let Some(id) = Uuid::parse_str(name_str).ok().filter(|id| !id.is_nil()) else {
continue;
};
if id == transaction {
destination_exists = true;
continue;
}
if !entry.file_type().map_err(to_file_error)?.is_dir() {
return Err(DiskError::InvalidPath);
}
if old_dir == Some(id) && name_str == id.hyphenated().to_string() {
old_data_dir_exists = true;
}
let dir_path = entry.path();
let prepared = regular_bytes(&dir_path.join(PREPARED))?;
let receipt = regular_bytes(&dir_path.join(DELETE_QUORUM_COMMITTED_FILE))?;
@@ -2589,6 +2657,12 @@ impl LocalDisk {
}
}
}
if let Some(old_path) = old_path_to_verify
&& (!scanned_file_path || old_data_dir_exists)
&& old_path.join(STORAGE_FORMAT_FILE).exists()
{
return Ok(None);
}
sole_transaction &= predecessors.is_empty();
let mut sparse = fi_clone;
sparse.data = None;
@@ -2607,14 +2681,28 @@ impl LocalDisk {
predecessors: predecessors.into_iter().collect(),
};
let raw = journal_bytes(&journal)?;
Ok(Some((old_dir, target, raw, journal, sole_transaction)))
Ok(Some((
old_dir,
target,
raw,
journal,
sole_transaction,
scanned_file_path && !destination_exists,
scanned_file_path.then_some(old_data_dir_exists),
)))
})() {
Ok(Some(tuple)) => tuple,
Ok(None) => return Ok(Ok(None)),
Err(err) => return Ok(Err(err)),
};
std::fs::create_dir_all(&staging_root)?;
let staged = tempfile::Builder::new().prefix("delete-prep-").tempdir_in(&staging_root)?;
let staged = match tempfile::Builder::new().prefix("delete-prep-").tempdir_in(&staging_root) {
Ok(dir) => dir,
Err(err) if err.kind() == ErrorKind::NotFound => {
std::fs::create_dir_all(&staging_root)?;
tempfile::Builder::new().prefix("delete-prep-").tempdir_in(&staging_root)?
}
Err(err) => return Err(err),
};
let staged_dir = staged.path().to_path_buf();
let backup_path = staged_dir.join(STORAGE_FORMAT_FILE_BACKUP);
if std::fs::hard_link(&xl_path, &backup_path).is_err() {
@@ -2638,27 +2726,40 @@ impl LocalDisk {
for file in written_files {
file.sync_data()?;
}
#[cfg(unix)]
let rollback_dir_fd = sync.then(|| std::fs::File::open(&staged_dir).ok().map(Arc::new)).flatten();
#[cfg(not(unix))]
let rollback_dir_fd: Option<Arc<std::fs::File>> = None;
if sync {
os::fsync_dir_std(&staged_dir)?;
os::fsync_directory_handle_std(&staged_dir, rollback_dir_fd.as_deref())?;
}
match std::fs::symlink_metadata(&destination) {
Ok(metadata) if metadata.is_dir() => match std::fs::remove_dir(&destination) {
Ok(()) => {}
if !destination_missing {
match std::fs::symlink_metadata(&destination) {
Ok(metadata) if metadata.is_dir() => match std::fs::remove_dir(&destination) {
Ok(()) => {}
Err(err) if err.kind() == ErrorKind::NotFound => {}
Err(err) if is_dir_not_empty_error(&err) => return Ok(Err(DiskError::OutdatedXLMeta)),
Err(err) => return Err(err),
},
Ok(_) => return Ok(Err(DiskError::OutdatedXLMeta)),
Err(err) if err.kind() == ErrorKind::NotFound => {}
Err(err) if is_dir_not_empty_error(&err) => return Ok(Err(DiskError::OutdatedXLMeta)),
Err(err) => return Err(err),
},
Ok(_) => return Ok(Err(DiskError::OutdatedXLMeta)),
Err(err) if err.kind() == ErrorKind::NotFound => {}
Err(err) => return Err(err),
}
}
let staged_dir = staged.keep();
os::record_native_publication(&destination);
if let Err(err) = std::fs::rename(&staged_dir, &destination) {
let _ = std::fs::remove_dir_all(&staged_dir);
return Err(err);
}
let old_data_path = old_dir.map(|uuid| file_path.join(uuid.to_string())).filter(|p| p.is_dir());
if sync && let Err(err) = os::fsync_dir_std(&file_path) {
let old_data_path = old_dir
.map(|uuid| file_path.join(uuid.to_string()))
.filter(|p| old_data_dir_state.unwrap_or_else(|| p.is_dir()));
#[cfg(unix)]
let file_path_dir = sync.then(|| std::fs::File::open(&file_path).ok()).flatten();
#[cfg(not(unix))]
let file_path_dir: Option<std::fs::File> = None;
if sync && let Err(err) = os::fsync_directory_handle_std(&file_path, file_path_dir.as_ref()) {
let _ = std::fs::remove_dir_all(&destination);
return Err(err);
}
@@ -2667,11 +2768,15 @@ impl LocalDisk {
let mut reserved_data_path = None;
if let Some(old_path) = old_data_path {
let reserved_marker = old_path.join(format!("{RESERVED_DELETE_DATA_DIR_MARKER_PREFIX}{transaction}"));
#[cfg(unix)]
let old_dir_file = sync.then(|| std::fs::File::open(&old_path).ok()).flatten();
#[cfg(not(unix))]
let old_dir_file: Option<std::fs::File> = None;
let reserve_res = (|| -> std::io::Result<()> {
let marker = std::fs::File::create(&reserved_marker)?;
if sync {
marker.sync_all()?;
os::fsync_dir_std(&old_path)?;
os::fsync_directory_handle_std(&old_path, old_dir_file.as_ref())?;
}
Ok(())
})();
@@ -2680,14 +2785,22 @@ impl LocalDisk {
let _ = std::fs::remove_dir_all(&destination);
return Err(err);
}
reserved_data_path = Some((old_path, reserved_marker));
reserved_data_path = Some((old_path, reserved_marker, old_dir_file));
}
if let Some(ref target_bytes) = target {
// A provisional head must stay outside the published recovery transaction.
let target_in_rollback = destination.join(TARGET);
let linked_xl = staging_root.join(format!("delete-xl-{transaction}"));
let _ = std::fs::remove_file(&linked_xl);
let write_res = if std::fs::hard_link(&target_in_rollback, &linked_xl).is_ok() {
os::record_native_publication(&xl_path);
let linked = match std::fs::hard_link(&target_in_rollback, &linked_xl) {
Ok(()) => true,
Err(err) if err.kind() == ErrorKind::AlreadyExists => {
let _ = std::fs::remove_file(&linked_xl);
std::fs::hard_link(&target_in_rollback, &linked_xl).is_ok()
}
Err(_) => false,
};
let write_res = if linked {
let rename_res = std::fs::rename(&linked_xl, &xl_path);
if rename_res.is_err() {
let _ = std::fs::remove_file(&linked_xl);
@@ -2704,7 +2817,7 @@ impl LocalDisk {
})()
};
if let Err(err) = write_res {
if let Some((_, reserved_marker)) = &reserved_data_path {
if let Some((_, reserved_marker, _)) = &reserved_data_path {
let _ = std::fs::remove_file(reserved_marker);
}
let _ = std::fs::remove_dir_all(&destination);
@@ -2713,19 +2826,20 @@ impl LocalDisk {
} else if let Err(err) = std::fs::remove_file(&xl_path)
&& err.kind() != ErrorKind::NotFound
{
if let Some((_, reserved_marker)) = &reserved_data_path {
if let Some((_, reserved_marker, _)) = &reserved_data_path {
let _ = std::fs::remove_file(reserved_marker);
}
let _ = std::fs::remove_dir_all(&destination);
return Err(err);
}
if sync {
os::fsync_dir_std(&file_path)?;
os::fsync_directory_handle_std(&file_path, file_path_dir.as_ref())?;
}
if let Some((old_path, reserved_marker)) = reserved_data_path {
let had_old_data_dir = reserved_data_path.is_some();
if let Some((old_path, reserved_marker, old_dir_file)) = reserved_data_path {
let recovery_marker = old_path.join(format!("{RECOVERY_DELETE_DATA_DIR_MARKER_PREFIX}{transaction}"));
match std::fs::rename(&reserved_marker, &recovery_marker) {
Ok(()) if sync => os::fsync_dir_std(&old_path)?,
Ok(()) if sync => os::fsync_directory_handle_std(&old_path, old_dir_file.as_ref())?,
Ok(()) => {}
Err(err) if err.kind() == ErrorKind::NotFound => {}
Err(err) => return Err(err),
@@ -2741,9 +2855,11 @@ impl LocalDisk {
journal_raw,
journal,
sole_transaction,
had_old_data_dir,
rollback_dir_fd,
},
);
Ok(Ok(Some(old_dir)))
Ok(Ok(Some(old_dir.filter(|_| had_old_data_dir))))
},
)
.await
@@ -2796,7 +2912,13 @@ impl LocalDisk {
Some(entry.bucket_path),
entry.file_path,
entry.staging_root,
Some((entry.journal_raw, entry.journal, entry.sole_transaction)),
Some((
entry.journal_raw,
entry.journal,
entry.sole_transaction,
entry.had_old_data_dir,
entry.rollback_dir_fd,
)),
),
None => {
self.reject_multipart_delete_recovery_mutation(volume, object)?;
@@ -2827,7 +2949,7 @@ impl LocalDisk {
lease,
sync,
move || -> std::io::Result<Result<Option<(DeleteRecoveryJournal, bool, bool)>>> {
if let Some((journal_raw, journal, sole_transaction)) = cached_receipt
if let Some((journal_raw, journal, sole_transaction, had_old_data_dir, rollback_dir_fd)) = cached_receipt
&& rollback.join(PREPARED).is_file()
{
let tmp_receipt = staging_root.join(format!("receipt-{transaction}.tmp"));
@@ -2838,9 +2960,10 @@ impl LocalDisk {
if sync && !receipt_bytes.is_empty() {
file.sync_data()?;
}
os::record_native_publication(&receipt_path);
std::fs::rename(&tmp_receipt, &receipt_path)?;
if sync {
os::fsync_dir_std(&rollback)?;
os::fsync_directory_handle_std(&rollback, rollback_dir_fd.as_deref())?;
}
Ok(())
})();
@@ -2857,6 +2980,8 @@ impl LocalDisk {
file_path: root,
staging_root,
journal_raw,
had_old_data_dir,
rollback_dir_fd,
},
);
}
@@ -2934,6 +3059,7 @@ impl LocalDisk {
if sync && !receipt_bytes.is_empty() {
staged.as_file().sync_data()?;
}
os::record_native_publication(&receipt_path);
std::fs::rename(staged.path(), &receipt_path)?;
if sync {
os::fsync_dir_std(&rollback)?;
@@ -2992,29 +3118,34 @@ impl LocalDisk {
None
};
let cached_published = rollback_open_path.as_deref().and_then(take_published_delete_receipt);
let (object_path, bucket_path, staging_root, root, cached_journal_raw) = match cached_published {
Some(entry) => (
entry.object_path,
entry.bucket_path,
entry.staging_root,
entry.file_path,
Some(entry.journal_raw),
),
None => {
self.reject_multipart_delete_recovery_mutation(volume, object)?;
if !safe_path(object, false) {
return Err(DiskError::InvalidPath);
let (object_path, bucket_path, staging_root, root, cached_journal_raw, cached_had_old_data_dir, mut rollback_dir_fd) =
match cached_published {
Some(entry) => (
entry.object_path,
entry.bucket_path,
entry.staging_root,
entry.file_path,
Some(entry.journal_raw),
Some(entry.had_old_data_dir),
entry.rollback_dir_fd,
),
None => {
self.reject_multipart_delete_recovery_mutation(volume, object)?;
if !safe_path(object, false) {
return Err(DiskError::InvalidPath);
}
let obj_path = self.get_object_path(volume, object)?;
(
obj_path,
self.io_get_bucket_path(volume)?,
self.io_get_bucket_path(RUSTFS_META_TMP_BUCKET)?,
self.io_get_object_path(volume, object)?,
None,
None,
None,
)
}
let obj_path = self.get_object_path(volume, object)?;
(
obj_path,
self.io_get_bucket_path(volume)?,
self.io_get_bucket_path(RUSTFS_META_TMP_BUCKET)?,
self.io_get_object_path(volume, object)?,
None,
)
}
};
};
let metadata_owner = os::acquire_metadata_mutation_lease(&object_path, namespace_owner).await;
let volume_guard = os::disk_volume_mutation_lock(&self.root, volume).read_owned().await;
let owner: Option<Arc<dyn Send + Sync>> = Some(Arc::new((metadata_owner, volume_guard)));
@@ -3043,14 +3174,19 @@ impl LocalDisk {
}
let data_dirs = journal.data_dirs.clone();
let has_original = journal.original_digest.is_some();
let has_target = journal.target_digest.is_some();
let journal_for_receipt = cached_journal_raw.is_none().then(|| journal.clone());
let lease = os::acquire_namespace_mutation_lease_with_owner(&receipt_path, owner).await;
let prune_paths: Vec<_> = rollback
.ancestors()
.take_while(|dir| *dir != bucket_path.as_path() && dir.starts_with(&bucket_path))
.map(Path::to_path_buf)
.collect();
let prune_paths: Vec<_> = if has_target {
vec![rollback.clone()]
} else {
rollback
.ancestors()
.take_while(|dir| *dir != bucket_path.as_path() && dir.starts_with(&bucket_path))
.map(Path::to_path_buf)
.collect()
};
let res = os::run_blocking_namespace_operation_with_sync(lease, sync, move || -> std::io::Result<()> {
let persist_receipt = |finished: bool| -> std::io::Result<()> {
@@ -3072,6 +3208,7 @@ impl LocalDisk {
if sync && !receipt_bytes.is_empty() {
staged_receipt.as_file().sync_data()?;
}
os::record_native_publication(&receipt_path);
std::fs::rename(staged_receipt.path(), &receipt_path)?;
if sync {
let _ = os::fsync_dir_std(&rollback);
@@ -3082,80 +3219,118 @@ impl LocalDisk {
let recovery_committed = format!("{RECOVERY_DELETE_DATA_DIR_MARKER_PREFIX}{transaction}");
let reserved = format!("{RESERVED_DELETE_DATA_DIR_MARKER_PREFIX}{transaction}");
let mut had_physical_data_dirs = false;
for data_dir in &data_dirs {
for base in [&root, &rollback] {
let dir_path = base.join(data_dir.to_string());
match std::fs::read_dir(&dir_path) {
Ok(entries) => {
had_physical_data_dirs = true;
let mut removable = Vec::new();
let mut preserve_dir = false;
for entry in recovery_entries(entries, work_budget.as_deref()) {
let entry = entry?;
let name = entry.file_name();
let Some(name) = name.to_str() else {
preserve_dir = true;
continue;
};
if entry.file_type()?.is_file()
&& (delete_recovery_part_file(name)
|| name == committed
|| name == recovery_committed
|| name == reserved
|| is_delete_transaction_marker(name, RESERVED_DELETE_DATA_DIR_MARKER_PREFIX))
{
removable.push(entry.path());
} else {
preserve_dir = true;
}
}
for path in removable {
ignore_not_found(std::fs::remove_file(path))?;
}
if sync {
os::fsync_dir_std(&dir_path)?;
}
if !preserve_dir {
match std::fs::remove_dir(&dir_path) {
Ok(()) => {}
Err(err) if err.kind() == ErrorKind::NotFound || is_dir_not_empty_error(&err) => {}
Err(err) => return Err(err),
let fast_cleanup = cached_journal_raw.is_some();
if cached_had_old_data_dir != Some(false) {
let bases: &[&Path] = if fast_cleanup { &[&root] } else { &[&root, &rollback] };
for data_dir in &data_dirs {
for base in bases {
let dir_path = base.join(data_dir.to_string());
match std::fs::read_dir(&dir_path) {
Ok(entries) => {
had_physical_data_dirs = true;
let mut removable = Vec::new();
let mut preserve_dir = false;
for entry in recovery_entries(entries, work_budget.as_deref()) {
let entry = entry?;
let name = entry.file_name();
let Some(name) = name.to_str() else {
preserve_dir = true;
continue;
};
if entry.file_type()?.is_file()
&& (delete_recovery_part_file(name)
|| name == committed
|| name == recovery_committed
|| name == reserved
|| is_delete_transaction_marker(name, RESERVED_DELETE_DATA_DIR_MARKER_PREFIX))
{
removable.push(entry.path());
} else {
preserve_dir = true;
}
}
for path in removable {
ignore_not_found(std::fs::remove_file(path))?;
}
let mut dir_removed = false;
if !preserve_dir {
match std::fs::remove_dir(&dir_path) {
Ok(()) => dir_removed = true,
Err(err) if err.kind() == ErrorKind::NotFound => dir_removed = true,
Err(err) if is_dir_not_empty_error(&err) => {}
Err(err) => return Err(err),
}
}
if !dir_removed && sync {
os::fsync_dir_std(&dir_path)?;
}
}
Err(err) if err.kind() == ErrorKind::NotFound => {}
Err(err) => return Err(err),
}
Err(err) if err.kind() == ErrorKind::NotFound => {}
Err(err) => return Err(err),
}
}
}
if had_physical_data_dirs && sync && root.is_dir() {
os::fsync_dir_std(&root)?;
}
for name in [STORAGE_FORMAT_FILE_BACKUP, DELETE_MARKER_ROLLBACK_FILE] {
if let Err(err) = ignore_not_found(std::fs::remove_file(rollback.join(name))) {
if fast_cleanup {
let backup_or_marker = if has_original {
Some(STORAGE_FORMAT_FILE_BACKUP)
} else if has_target {
Some(DELETE_MARKER_ROLLBACK_FILE)
} else {
None
};
if let Some(name) = backup_or_marker
&& let Err(err) = ignore_not_found(std::fs::remove_file(rollback.join(name)))
{
if has_target || had_physical_data_dirs {
let _ = persist_receipt(true);
}
return Err(err);
}
}
for name in [PREPARED, TARGET, FINISHED] {
let path = rollback.join(name);
#[cfg(all(any(test, feature = "test-util"), not(windows)))]
os::prepared_publication_test_hooks::run(os::prepared_publication_test_hooks::Stage::Remove, &path);
if let Err(err) = ignore_not_found(std::fs::remove_file(path)) {
if has_target || had_physical_data_dirs {
let _ = persist_receipt(true);
for name in [PREPARED, FINISHED].into_iter().chain(has_target.then_some(TARGET)) {
let path = rollback.join(name);
#[cfg(all(any(test, feature = "test-util"), not(windows)))]
os::prepared_publication_test_hooks::run(os::prepared_publication_test_hooks::Stage::Remove, &path);
if let Err(err) = ignore_not_found(std::fs::remove_file(path)) {
if has_target || had_physical_data_dirs {
let _ = persist_receipt(true);
}
return Err(err);
}
}
} else {
for name in [STORAGE_FORMAT_FILE_BACKUP, DELETE_MARKER_ROLLBACK_FILE] {
if let Err(err) = ignore_not_found(std::fs::remove_file(rollback.join(name))) {
if has_target || had_physical_data_dirs {
let _ = persist_receipt(true);
}
return Err(err);
}
}
for name in [PREPARED, TARGET, FINISHED] {
let path = rollback.join(name);
#[cfg(all(any(test, feature = "test-util"), not(windows)))]
os::prepared_publication_test_hooks::run(os::prepared_publication_test_hooks::Stage::Remove, &path);
if let Err(err) = ignore_not_found(std::fs::remove_file(path)) {
if has_target || had_physical_data_dirs {
let _ = persist_receipt(true);
}
return Err(err);
}
return Err(err);
}
}
if sync && let Err(err) = os::fsync_dir_std(&rollback) {
let rollback_sync = rollback_dir_fd.take();
if sync && let Err(err) = os::fsync_directory_handle_std(&rollback, rollback_sync.as_deref()) {
drop(rollback_sync);
if has_target || had_physical_data_dirs {
let _ = persist_receipt(true);
}
return Err(err);
}
drop(rollback_sync);
#[cfg(all(any(test, feature = "test-util"), not(windows)))]
os::prepared_publication_test_hooks::run(os::prepared_publication_test_hooks::Stage::Remove, &receipt_path);
if let Err(err) = ignore_not_found(std::fs::remove_file(&receipt_path)) {
@@ -6438,6 +6613,93 @@ mod tests {
}
}
#[tokio::test]
async fn delete_recovery_bounded_discovery_charges_uuid_named_children_once_and_keeps_evidence() {
let (_temp, disk, _, _) = fixture().await;
let path = disk.delete_discovery_path("recovery", Some("prefix")).expect("prefix path");
std::fs::create_dir_all(&path).expect("physical prefix");
// UUID-named child keys of the folder: each holds its own `xl.meta` and a
// data directory, neither of which can be evidence for the folder's key.
const CHILDREN: u32 = 2000;
for _ in 0..CHILDREN {
let child = path.join(Uuid::new_v4().to_string());
std::fs::create_dir_all(child.join(Uuid::new_v4().to_string())).expect("child key data dir");
std::fs::write(child.join(STORAGE_FORMAT_FILE), b"meta").expect("child key metadata");
}
// A UUID-named sub-prefix with children of its own.
const SUB_PREFIX_CHILDREN: u32 = 640;
let sub_prefix = path.join(Uuid::new_v4().to_string());
for index in 0..SUB_PREFIX_CHILDREN {
std::fs::create_dir_all(sub_prefix.join(format!("child-{index}"))).expect("sub-prefix child");
}
// Evidence of two interrupted transactions of the key itself: a journaled
// rollback directory, and a data-directory marker whose rollback is gone.
let journaled = Uuid::new_v4();
std::fs::create_dir_all(path.join(journaled.to_string())).expect("rollback directory");
std::fs::write(path.join(journaled.to_string()).join(PREPARED), b"{}").expect("journal");
let marker_only = Uuid::new_v4();
let data_dir = path.join(Uuid::new_v4().to_string());
std::fs::create_dir_all(&data_dir).expect("residual data directory");
std::fs::write(data_dir.join(format!("{RESERVED_DELETE_DATA_DIR_MARKER_PREFIX}{marker_only}")), [])
.expect("reservation marker");
let page = disk
.discover_delete_transactions_bounded(
"recovery",
"prefix",
super::super::super::DELETE_RECOVERY_MAX_DISCOVERY_ENTRIES,
)
.await
.expect("bounded probe");
assert!(page.complete, "a folder of UUID-named keys is a complete inventory");
// One entry per descent (children, sub-prefix, rollback and data
// directory) and per evidence file; inert names at 1/64.
let inert = 2 * CHILDREN + SUB_PREFIX_CHILDREN;
assert_eq!(page.visited, CHILDREN + 3 + 2 + inert / PLAIN_CHILDREN_PER_BUDGET_ENTRY);
let found = page.transactions.iter().map(|id| id.transaction).collect::<BTreeSet<_>>();
assert!(found.contains(&journaled), "a journaled rollback directory is evidence");
assert!(found.contains(&marker_only), "a marker-only residue is evidence");
assert_eq!(found.len(), 2, "no child key is mistaken for a transaction");
}
#[tokio::test]
async fn delete_recovery_bounded_discovery_stops_on_request_and_skips_only_inert_irregular_entries() {
let (_temp, disk, _, _) = fixture().await;
let path = disk.delete_discovery_path("recovery", Some("prefix")).expect("prefix path");
let child = path.join(Uuid::new_v4().to_string());
std::fs::create_dir_all(&child).expect("UUID-named child key");
// A stop request (cancellation or the wall-clock limit) ends the walk
// before any entry, leaving continuation state: never evidence of absence.
let session = disk
.new_delete_discovery_session("recovery", Some("prefix"), &path)
.expect("private session");
let (transactions, next, visited) = session
.advance_bounded(256, super::super::super::DELETE_RECOVERY_MAX_DISCOVERY_ENTRIES, &|| true)
.expect("stopped walk");
assert!(transactions.is_empty());
assert_eq!(visited, 0);
assert!(next.is_some(), "a stopped walk is incomplete");
drop(next);
#[cfg(unix)]
{
// An irregular entry whose name cannot be evidence is skipped unread.
std::os::unix::fs::symlink(&path, child.join("inert-link")).expect("inert symlink");
let page = disk
.discover_delete_transactions_bounded("recovery", "prefix", 4096)
.await
.expect("inert irregular entries are not evidence");
assert!(page.complete && page.transactions.is_empty());
// One with an evidence name still fails closed.
std::os::unix::fs::symlink(&path, child.join(PREPARED)).expect("evidence-named symlink");
assert!(
disk.discover_delete_transactions_bounded("recovery", "prefix", 4096)
.await
.is_err(),
"an evidence-named symlink is corruption, not a skipped name"
);
}
}
#[tokio::test]
async fn delete_recovery_bounded_discovery_refuses_partial_inventory_without_public_cursor() {
let (_temp, disk, _, _) = fixture().await;
@@ -6540,8 +6802,11 @@ mod tests {
let filler_id = Uuid::new_v4().to_string();
let filler = path.join(&filler_id);
fs::create_dir_all(&filler).await.expect("non-evidence directory");
for index in 0..5000 {
fs::write(filler.join(format!("part-{index:04}")), [])
// Inert names are skipped at 1/64, so fill the step with entries that
// pass the evidence-name filter but are directories: each costs a full
// budget entry and names no transaction.
for _ in 0..5000 {
fs::create_dir(filler.join(format!("{DELETE_DATA_DIR_MARKER_PREFIX}{}", Uuid::new_v4())))
.await
.expect("non-evidence entry");
}
+409 -84
View File
@@ -4,6 +4,11 @@
//! Bucket-to-staging-root candidates: one marker per staging root, recorded
//! before the root is published and pruned once the root is gone. Source
//! metadata stays authoritative.
//!
//! A marker without a staging root is either a leftover or an upload whose
//! root is still being published. The publisher proves the second case by
//! holding a [`PublicationPin`] from before it records the marker until its
//! filesystem work has returned; no elapsed time is taken as that proof.
use super::{LocalDisk, read_metadata_only_std};
use crate::disk::{
@@ -29,9 +34,6 @@ use uuid::Uuid;
const INDEX: &str = ".rustfs.sys/multipart-index-v1";
const REBUILD_DIRENTS_PER_JOB: usize = 64;
/// A marker is recorded before its staging root is published. A marker younger
/// than this without a root is an upload in flight, never a leftover to prune.
const MULTIPART_INDEX_PRUNE_GRACE: std::time::Duration = std::time::Duration::from_secs(300);
type DirectoryEntries = Box<dyn Iterator<Item = io::Result<String>> + Send>;
type PhysicalRoot = (u64, [u8; 16]);
static OWNERS: LazyLock<parking_lot::Mutex<HashMap<PhysicalRoot, Weak<IndexOwner>>>> =
@@ -56,9 +58,15 @@ pub(super) struct IndexOwner {
// drop so a cancelled scan can be resumed by the next caller.
driving: AtomicBool,
disabled: AtomicBool,
// Markers whose staging root some publisher is creating right now, with
// the number of publishers. Boot-local like the index itself: every writer
// of this physical drive in this process shares this owner.
publishing: parking_lot::Mutex<HashMap<PathBuf, usize>>,
#[cfg(test)]
rebuild_head_pause: parking_lot::Mutex<Option<RebuildHeadPause>>,
#[cfg(test)]
publication_pause: parking_lot::Mutex<Option<RebuildHeadPause>>,
#[cfg(test)]
observed_rebuild_queue: parking_lot::Mutex<Option<Arc<os::DiskBlockingQueue>>>,
#[cfg(test)]
rebuild_single_job: AtomicBool,
@@ -120,23 +128,6 @@ impl IndexDirectory {
std::fs::File::open(self.path.join(name))
}
}
fn modified(&self, name: &str) -> io::Result<std::time::SystemTime> {
#[cfg(unix)]
{
self.open_file(name)?.metadata()?.modified()
}
#[cfg(not(unix))]
{
std::fs::metadata(self.path.join(name))?.modified()
}
}
/// Whether the marker is still inside the publication grace period. A
/// marker dated in the future (clock stepped back) is never a leftover.
fn marker_is_young(&self, name: &str) -> bool {
self.modified(name)
.ok()
.is_some_and(|modified| modified.elapsed().map_or(true, |age| age < MULTIPART_INDEX_PRUNE_GRACE))
}
fn remove_file(&self, name: &str) -> io::Result<()> {
#[cfg(unix)]
{
@@ -280,9 +271,12 @@ impl IndexOwner {
initialized: OnceLock::new(),
driving: AtomicBool::new(false),
disabled: AtomicBool::new(false),
publishing: parking_lot::Mutex::new(HashMap::new()),
#[cfg(test)]
rebuild_head_pause: parking_lot::Mutex::new(None),
#[cfg(test)]
publication_pause: parking_lot::Mutex::new(None),
#[cfg(test)]
observed_rebuild_queue: parking_lot::Mutex::new(None),
#[cfg(test)]
rebuild_single_job: AtomicBool::new(false),
@@ -394,8 +388,8 @@ impl IndexOwner {
// The index is boot-local: every start rescans the staging area
// before serving it, so a marker lost to a crash is recreated before
// it can be consulted. Creation is enough; no fsync of the marker or
// its ancestors. An existing marker is touched so pruning sees the
// new upload's age, not the previous one's.
// its ancestors. An existing marker needs nothing: the caller's
// publication pin, not the marker's age, keeps it from being pruned.
match openat(
&parent,
source_root,
@@ -407,16 +401,7 @@ impl IndexOwner {
return Err(io::Error::other("index marker is not a file"));
}
}
Err(rustix::io::Errno::EXIST) => {
// Best effort: a filesystem that refuses to set times keeps
// the previous upload's age, which only shortens the prune
// grace for this key; index completeness does not depend on it.
if let Ok(marker) =
openat(&parent, source_root, OFlags::WRONLY | OFlags::NOFOLLOW | OFlags::CLOEXEC, Mode::empty())
{
let _ = std::fs::File::from(marker).set_modified(std::time::SystemTime::now());
}
}
Err(rustix::io::Errno::EXIST) => {}
Err(error) => return Err(error.into()),
}
}
@@ -429,7 +414,23 @@ impl IndexOwner {
os::replace_multipart_source_metadata(&marker, b"1", &self.base, &root)?;
os::sync_multipart_owned_record_parents(&self.base.join(".rustfs.sys"), &directory, true, &self.base, &root)?;
}
self.ensure_current_root()
Ok(())
}
/// Declare that the caller is about to publish the staging root of
/// `marker`. Must precede the marker lease the caller records under, so a
/// prune that found no pin has released that lease before the marker is
/// (re)created.
fn pin_publication(self: &Arc<Self>, marker: PathBuf) -> PublicationPin {
*self.publishing.lock().entry(marker.clone()).or_default() += 1;
PublicationPin {
owner: Arc::clone(self),
marker,
}
}
fn is_publishing(&self, marker: &Path) -> bool {
self.publishing.lock().contains_key(marker)
}
fn ensure_current_root(&self) -> io::Result<()> {
@@ -513,9 +514,12 @@ impl IndexOwner {
}
}
pub(super) async fn record_info(self: &Arc<Self>, path: &str, info: &FileInfo) -> Result<()> {
/// Record the marker of `path`'s staging root. The returned pin must stay
/// alive until the filesystem work that publishes the root has returned,
/// including work that outlives a cancelled caller.
pub(super) async fn record_info(self: &Arc<Self>, path: &str, info: &FileInfo) -> Result<Option<PublicationPin>> {
if canonical_upload(path).is_none() {
return Ok(());
return Ok(None);
}
let owner = self.clone();
let path = path.to_owned();
@@ -531,6 +535,7 @@ impl IndexOwner {
.join(path.split('/').next().unwrap_or_default())
})
.unwrap_or_else(|| owner.base.join(INDEX));
let pin = self.pin_publication(marker.clone());
let lease = os::acquire_namespace_mutation_lease_with_owner(&marker, None).await;
os::run_blocking_namespace_operation_with_sync(lease, true, move || {
match owner.record(&path, &info) {
@@ -544,16 +549,13 @@ impl IndexOwner {
}
})
.await?;
Ok(())
Ok(Some(pin))
}
pub(super) async fn record_head(self: &Arc<Self>, path: &str, bytes: &[u8]) -> Result<()> {
let Some(upload) = path
pub(super) async fn prepare_record_head(self: &Arc<Self>, path: &str, bytes: &[u8]) -> Option<PreparedHeadRecord> {
let upload = path
.strip_suffix("/xl.meta")
.filter(|upload| canonical_upload(upload).is_some())
else {
return Ok(());
};
.filter(|upload| canonical_upload(upload).is_some())?;
let info = match get_file_info(
bytes,
RUSTFS_META_MULTIPART_BUCKET,
@@ -568,10 +570,36 @@ impl IndexOwner {
Ok(info) => info,
Err(_) => {
self.disabled.store(true, Ordering::Release);
return Ok(());
return None;
}
};
self.record_info(upload, &info).await
let marker = info
.metadata
.get(crate::disk::multipart_index::RUSTFS_MULTIPART_BUCKET_KEY)
.map(|bucket| {
self.base
.join(INDEX)
.join(sha(bucket))
.join(upload.split('/').next().unwrap_or_default())
})
.unwrap_or_else(|| self.base.join(INDEX));
let pin = self.pin_publication(marker.clone());
let lease = os::acquire_namespace_mutation_lease_with_owner(&marker, None).await;
Some(PreparedHeadRecord {
upload: upload.to_owned(),
info,
pin,
_lease: lease,
})
}
/// Record the marker of the staging root that `path` belongs to. See
/// [`Self::record_info`] for how long the returned pin must live.
pub(super) async fn record_head(self: &Arc<Self>, path: &str, bytes: &[u8]) -> Result<Option<PublicationPin>> {
let Some(prepared) = self.prepare_record_head(path, bytes).await else {
return Ok(None);
};
prepared.run().await.map(Some)
}
async fn page(self: &Arc<Self>, bucket: &str, marker: Option<&str>, disk_id: Uuid) -> Result<MultipartIndexPage> {
@@ -622,17 +650,20 @@ impl IndexOwner {
// to every key ever multipart-initiated on this drive.
let mut live = Vec::with_capacity(roots.len());
let mut stale = Vec::new();
let markers = owner.base.join(INDEX).join(sha(&bucket));
for root in roots {
// The marker precedes its staging root: a pinned one is an
// upload in flight and stays a candidate. The pin is read
// before the root, so a publisher that finished in between has
// already made its root visible.
if owner.is_publishing(&markers.join(&root)) {
live.push(root);
continue;
}
match owner.directory(&Path::new(RUSTFS_META_MULTIPART_BUCKET).join(&root)) {
Ok(_) => live.push(root),
Err(error) if matches!(error.kind(), io::ErrorKind::NotFound | io::ErrorKind::NotADirectory) => {
// The marker precedes its staging root; a young one is an
// upload in flight and stays a candidate.
if directory.as_ref().is_some_and(|index| index.marker_is_young(&root)) {
live.push(root);
} else {
stale.push(root);
}
stale.push(root);
}
Err(error) => return Err(DiskError::from(error)),
}
@@ -657,9 +688,11 @@ impl IndexOwner {
}
/// Remove one marker whose staging root was gone when a page examined it.
/// The marker's own namespace lease serializes this with `record_info`; the
/// staging root and the marker's age are re-checked under that lease, so an
/// upload recorded in between keeps its marker.
/// The marker's own namespace lease serializes this with recording. Under
/// that lease the publication pin is read first and the staging root
/// second: a publisher pinned before the read keeps the marker, one that
/// already finished has made its root visible, and one that pins later
/// records its marker only after this lease is released.
async fn prune_marker(self: &Arc<Self>, bucket: &str, root: &str) -> Result<()> {
let owner = self.clone();
let bucket_directory = Path::new(INDEX).join(sha(bucket));
@@ -670,6 +703,9 @@ impl IndexOwner {
// not queue behind every other mutation on the one-worker foreground lane.
os::run_blocking_namespace_operation_with_sync(lease, true, move || {
owner.ensure_current_root()?;
if owner.is_publishing(&marker) {
return Ok(());
}
match owner.directory(&Path::new(RUSTFS_META_MULTIPART_BUCKET).join(&root)) {
Ok(_) => return Ok(()),
Err(error) if matches!(error.kind(), io::ErrorKind::NotFound | io::ErrorKind::NotADirectory) => {}
@@ -680,9 +716,6 @@ impl IndexOwner {
Err(error) if error.kind() == io::ErrorKind::NotFound => return Ok(()),
Err(error) => return Err(error),
};
if directory.marker_is_young(&root) {
return Ok(());
}
match directory.remove_file(&root) {
Err(error) if error.kind() != io::ErrorKind::NotFound => Err(error),
_ => Ok(()),
@@ -693,6 +726,72 @@ impl IndexOwner {
}
}
/// Proof that the staging root of one marker is being published. While any pin
/// for a marker is alive, pruning leaves that marker alone.
#[must_use = "dropping the pin lets a concurrent listing prune the marker before its staging root exists"]
pub(super) struct PublicationPin {
owner: Arc<IndexOwner>,
marker: PathBuf,
}
impl PublicationPin {
/// Test seam between recording the marker and publishing the staging root.
#[cfg(test)]
pub(super) fn pause_for_test(&self) {
let pause = self.owner.publication_pause.lock().take();
if let Some(pause) = pause {
let _pause = os::enter_blocking_test_hook();
let _ = pause.started.send(());
let _ = pause.release.recv_timeout(std::time::Duration::from_secs(10));
}
}
}
impl Drop for PublicationPin {
fn drop(&mut self) {
let mut publishing = self.owner.publishing.lock();
if let Some(count) = publishing.get_mut(&self.marker) {
*count -= 1;
if *count == 0 {
publishing.remove(&self.marker);
}
}
}
}
pub(super) struct PreparedHeadRecord {
upload: String,
info: FileInfo,
pin: PublicationPin,
_lease: Arc<os::NamespaceMutationLease>,
}
impl PreparedHeadRecord {
/// Record the marker and release its lease. The caller publishes the
/// staging root while it holds the returned pin.
pub(super) fn run_std(self) -> io::Result<PublicationPin> {
let Self {
upload,
info,
pin,
_lease,
} = self;
match pin.owner.record(&upload, &info) {
Err(error) if error.kind() == io::ErrorKind::InvalidData => {
pin.owner.disabled.store(true, Ordering::Release);
Ok(pin)
}
Err(error) => Err(error),
Ok(()) => Ok(pin),
}
}
async fn run(self) -> Result<PublicationPin> {
let lease = Arc::clone(&self._lease);
Ok(os::run_blocking_namespace_operation_with_sync(lease, true, move || self.run_std()).await?)
}
}
impl LocalDisk {
#[cfg(any(test, feature = "test-util"))]
pub(in crate::disk) async fn wait_for_multipart_index_bootstrap(&self) {
@@ -907,47 +1006,274 @@ mod tests {
let live = format!("{:064x}", 1);
let finished = format!("{:064x}", 2);
let in_flight = format!("{:064x}", 3);
let clock_stepped = format!("{:064x}", 4);
for name in [&live, &finished, &in_flight, &clock_stepped] {
for name in [&live, &finished, &in_flight] {
std::fs::write(directory.join(name), b"1").expect("marker");
}
// Completion and abort remove the staging root and leave the marker.
std::fs::create_dir_all(root.path().join(RUSTFS_META_MULTIPART_BUCKET).join(&live)).expect("live staging root");
set_marker_modified(&directory.join(&finished), std::time::SystemTime::now() - MULTIPART_INDEX_PRUNE_GRACE * 2);
// A marker dated after a backwards clock step is never a leftover.
set_marker_modified(
&directory.join(&clock_stepped),
std::time::SystemTime::now() + MULTIPART_INDEX_PRUNE_GRACE * 2,
);
let page = disk.multipart_index.page("bucket", None, Uuid::new_v4()).await.expect("page");
// The in-flight marker has no staging root yet but is young: a candidate, not a leftover.
assert_eq!(page.roots, vec![live.clone(), in_flight.clone(), clock_stepped.clone()]);
let owner = &disk.multipart_index;
let publishing = owner.pin_publication(owner.base.join(INDEX).join(sha("bucket")).join(&in_flight));
let second_publisher = owner.pin_publication(owner.base.join(INDEX).join(sha("bucket")).join(&in_flight));
let page = owner.page("bucket", None, Uuid::new_v4()).await.expect("page");
// The in-flight marker has no staging root yet but is pinned: a candidate, not a leftover.
assert_eq!(page.roots, vec![live.clone(), in_flight.clone()]);
assert!(page.next_marker.is_none());
assert!(directory.join(&live).exists(), "a live root keeps its marker");
assert!(directory.join(&in_flight).exists(), "a marker inside the grace period is kept");
assert!(directory.join(&clock_stepped).exists(), "a future-dated marker is kept");
assert!(!directory.join(&finished).exists(), "a finished root's marker is pruned");
// The next page no longer pays for the pruned marker.
let page = disk
.multipart_index
assert!(directory.join(&in_flight).exists(), "a pinned marker is kept");
assert!(!directory.join(&finished).exists(), "a finished root's marker is pruned at once");
// One of two publishers of the same root finishing does not unpin it.
drop(publishing);
let page = owner
.page("bucket", None, Uuid::new_v4())
.await
.expect("page after pruning");
assert_eq!(page.roots, vec![live, in_flight, clock_stepped]);
.expect("page with one publisher left");
assert_eq!(page.roots, vec![live.clone(), in_flight.clone()]);
assert!(directory.join(&in_flight).exists(), "the remaining publisher keeps the marker");
// A publication that ended without a staging root leaves a leftover.
drop(second_publisher);
let page = owner
.page("bucket", None, Uuid::new_v4())
.await
.expect("page after the failed publication");
assert_eq!(page.roots, vec![live]);
assert!(!directory.join(&in_flight).exists(), "an unpinned marker without a root is pruned");
assert!(owner.publishing.lock().is_empty(), "finished publishers leave no pin behind");
}
/// Move a marker's modification time outside the publication grace period.
fn age_marker(marker: &Path) {
set_marker_modified(marker, std::time::SystemTime::now() - MULTIPART_INDEX_PRUNE_GRACE * 2);
fn pause_next_publication(owner: &IndexOwner) -> (tokio::sync::oneshot::Receiver<()>, std::sync::mpsc::Sender<()>) {
let (started, waiting) = tokio::sync::oneshot::channel();
let (release, released) = std::sync::mpsc::channel();
*owner.publication_pause.lock() = Some(RebuildHeadPause {
started,
release: released,
});
(waiting, release)
}
fn set_marker_modified(marker: &Path, modified: std::time::SystemTime) {
/// The window the pin closes: the marker is recorded and its lease is
/// released, the staging root is not published yet, and a listing examines
/// the bucket. The marker is dated a day back so that no grace period could
/// be what keeps it.
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn multipart_index_listing_during_source_publication_keeps_the_new_upload_listed() {
let root = tempfile::tempdir().expect("disk root");
let disk = Arc::new(open(root.path()).await);
let (path, info) = upload("bucket", "published-while-listed");
let source_root = path.split('/').next().expect("source sha").to_owned();
let marker = root.path().join(INDEX).join(sha("bucket")).join(&source_root);
let source_head = root
.path()
.join(RUSTFS_META_MULTIPART_BUCKET)
.join(&path)
.join(STORAGE_FORMAT_FILE);
let (paused, release) = pause_next_publication(&disk.multipart_index);
let writer = tokio::spawn({
let disk = Arc::clone(&disk);
let path = path.clone();
async move { disk.write_metadata("bucket", RUSTFS_META_MULTIPART_BUCKET, &path, info).await }
});
tokio::time::timeout(std::time::Duration::from_secs(5), paused)
.await
.expect("the publication reaches the pause")
.expect("pause sender");
assert!(marker.exists(), "the marker precedes the staging root");
assert!(!source_head.exists(), "the staging root is not published yet");
std::fs::File::options()
.write(true)
.open(marker)
.open(&marker)
.expect("marker")
.set_modified(modified)
.expect("set marker time");
.set_modified(std::time::SystemTime::now() - std::time::Duration::from_secs(86_400))
.expect("age the marker");
let identity = Uuid::new_v4();
let page = disk
.multipart_index
.page("bucket", None, identity)
.await
.expect("page during publication");
assert_eq!(page.roots, vec![source_root.clone()], "an upload in flight stays a candidate");
assert!(marker.exists(), "a listing must not prune the marker of an upload in flight");
release.send(()).expect("resume the publication");
writer.await.expect("writer task").expect("the upload metadata is published");
assert!(source_head.exists());
let page = disk
.multipart_index
.page("bucket", None, identity)
.await
.expect("page after publication");
assert_eq!(page.roots, vec![source_root], "the published upload is listed");
assert!(marker.exists());
assert!(disk.multipart_index.publishing.lock().is_empty());
}
/// A timed-out caller drops its future while the filesystem work goes on.
/// The pin belongs to that work, so the marker survives until the staging
/// root the detached work publishes is visible.
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn multipart_index_cancelled_publisher_keeps_its_marker_until_the_detached_work_returns() {
let root = tempfile::tempdir().expect("disk root");
let disk = Arc::new(open(root.path()).await);
let (path, info) = upload("bucket", "caller-timed-out");
let source_root = path.split('/').next().expect("source sha").to_owned();
let marker = root.path().join(INDEX).join(sha("bucket")).join(&source_root);
let source_head = root
.path()
.join(RUSTFS_META_MULTIPART_BUCKET)
.join(&path)
.join(STORAGE_FORMAT_FILE);
let (paused, release) = pause_next_publication(&disk.multipart_index);
let writer = tokio::spawn({
let disk = Arc::clone(&disk);
let path = path.clone();
async move { disk.write_metadata("bucket", RUSTFS_META_MULTIPART_BUCKET, &path, info).await }
});
tokio::time::timeout(std::time::Duration::from_secs(5), paused)
.await
.expect("the publication reaches the pause")
.expect("pause sender");
writer.abort();
assert!(writer.await.expect_err("the caller is gone").is_cancelled());
let identity = Uuid::new_v4();
let page = disk
.multipart_index
.page("bucket", None, identity)
.await
.expect("page after the caller left");
assert_eq!(page.roots, vec![source_root.clone()]);
assert!(marker.exists(), "the detached publication still owns the marker");
release.send(()).expect("resume the detached publication");
tokio::time::timeout(std::time::Duration::from_secs(10), async {
while !source_head.exists() || !disk.multipart_index.publishing.lock().is_empty() {
tokio::time::sleep(std::time::Duration::from_millis(5)).await;
}
})
.await
.expect("the detached publication completes");
let page = disk
.multipart_index
.page("bucket", None, identity)
.await
.expect("page after the detached publication");
assert_eq!(page.roots, vec![source_root], "the upload the detached work published is listed");
assert!(marker.exists());
}
/// A page classifies a marker as stale before the publisher pins it; the
/// prune that follows must read the pin again under the marker lease.
#[tokio::test]
async fn multipart_index_prune_rechecks_the_pin_under_the_marker_lease() {
let root = tempfile::tempdir().expect("disk root");
let disk = open(root.path()).await;
let owner = &disk.multipart_index;
let directory = root.path().join(INDEX).join(sha("bucket"));
std::fs::create_dir_all(&directory).expect("marker directory");
let name = format!("{:064x}", 9);
std::fs::write(directory.join(&name), b"1").expect("marker");
let pin = owner.pin_publication(owner.base.join(INDEX).join(sha("bucket")).join(&name));
owner.prune_marker("bucket", &name).await.expect("prune of a pinned marker");
assert!(directory.join(&name).exists(), "a publisher that pinned after the page keeps its marker");
drop(pin);
owner.prune_marker("bucket", &name).await.expect("prune of a leftover");
assert!(!directory.join(&name).exists(), "the same marker is a leftover once nobody publishes it");
}
/// The publisher pins before it queues on the marker lease. A prune that
/// holds the lease and found no pin has therefore unlinked the marker
/// before the publisher records it, and the record recreates it.
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn multipart_index_publisher_pins_before_it_waits_for_the_marker_lease() {
let root = tempfile::tempdir().expect("disk root");
let disk = Arc::new(open(root.path()).await);
let owner = Arc::clone(&disk.multipart_index);
let (path, info) = upload("bucket", "queued-behind-a-prune");
let source_root = path.split('/').next().expect("source sha").to_owned();
let marker = owner.base.join(INDEX).join(sha("bucket")).join(&source_root);
let visible_marker = root.path().join(INDEX).join(sha("bucket")).join(&source_root);
// Stand in for a prune that owns the marker lease and already read "no pin".
let prune_lease = os::acquire_namespace_mutation_lease_with_owner(&marker, None).await;
assert!(!owner.is_publishing(&marker));
let writer = tokio::spawn({
let disk = Arc::clone(&disk);
let path = path.clone();
async move { disk.write_metadata("bucket", RUSTFS_META_MULTIPART_BUCKET, &path, info).await }
});
tokio::time::timeout(std::time::Duration::from_secs(5), async {
while !owner.is_publishing(&marker) {
tokio::time::sleep(std::time::Duration::from_millis(2)).await;
}
})
.await
.expect("the publisher pins while the marker lease is still held by the prune");
assert!(!visible_marker.exists(), "the marker is recorded only under the lease");
drop(prune_lease);
writer.await.expect("writer task").expect("the upload metadata is published");
assert!(visible_marker.exists(), "the record after the prune recreates the marker");
let page = owner.page("bucket", None, Uuid::new_v4()).await.expect("page");
assert_eq!(page.roots, vec![source_root]);
}
/// Stale markers at the start of a page, at its cursor, and filling a whole
/// page: every page still satisfies the receiver's contract and advances.
#[tokio::test]
async fn multipart_index_pages_of_stale_markers_keep_valid_cursors() {
let root = tempfile::tempdir().expect("disk root");
let disk = open(root.path()).await;
let directory = root.path().join(INDEX).join(sha("bucket"));
std::fs::create_dir_all(&directory).expect("marker directory");
// Page one is entirely stale. Page two is live except its first name
// and the name its cursor lands on. Two live names follow.
let total = MULTIPART_INDEX_PAGE_SIZE * 2 + 2;
let stale = |i: usize| i <= MULTIPART_INDEX_PAGE_SIZE || i == MULTIPART_INDEX_PAGE_SIZE * 2 - 1;
for i in 0..total {
let name = format!("{i:064x}");
std::fs::write(directory.join(&name), b"1").expect("marker");
if !stale(i) {
std::fs::create_dir_all(root.path().join(RUSTFS_META_MULTIPART_BUCKET).join(&name)).expect("staging root");
}
}
let disk_id = Uuid::new_v4();
let first = disk.multipart_index.page("bucket", None, disk_id).await.expect("first page");
first
.validate("bucket", None, disk_id)
.expect("an empty page with a cursor validates");
assert!(first.roots.is_empty(), "a fully stale page has no roots");
let cursor = first.next_marker.clone().expect("more markers follow a fully stale page");
assert_eq!(cursor, format!("{:064x}", MULTIPART_INDEX_PAGE_SIZE - 1));
let second = disk
.multipart_index
.page("bucket", Some(&cursor), disk_id)
.await
.expect("second page");
second
.validate("bucket", Some(&cursor), disk_id)
.expect("stale first and cursor names validate");
assert_eq!(second.roots.len(), MULTIPART_INDEX_PAGE_SIZE - 2);
assert_eq!(second.roots.first(), Some(&format!("{:064x}", MULTIPART_INDEX_PAGE_SIZE + 1)));
let cursor = second
.next_marker
.clone()
.expect("the cursor is the stale last examined name");
assert_eq!(cursor, format!("{:064x}", MULTIPART_INDEX_PAGE_SIZE * 2 - 1));
assert!(second.roots.last() < Some(&cursor));
let third = disk
.multipart_index
.page("bucket", Some(&cursor), disk_id)
.await
.expect("third page");
third.validate("bucket", Some(&cursor), disk_id).expect("last page validates");
assert_eq!(third.roots.len(), 2);
assert!(third.next_marker.is_none());
let remaining = std::fs::read_dir(&directory).expect("marker directory").count();
assert_eq!(remaining, total - MULTIPART_INDEX_PAGE_SIZE - 2, "every stale marker is pruned");
}
#[tokio::test]
@@ -964,7 +1290,6 @@ mod tests {
std::fs::create_dir_all(root.path().join(RUSTFS_META_MULTIPART_BUCKET).join(&name)).expect("staging root");
}
}
age_marker(&directory.join(&stale));
let disk_id = Uuid::new_v4();
let first = disk.multipart_index.page("bucket", None, disk_id).await.expect("first page");
first
@@ -428,7 +428,6 @@ impl LocalDisk {
let root = store.publication_root.clone();
let lease = os::acquire_namespace_mutation_lease_with_owner(&destination, Some(store.lease.clone())).await;
os::run_blocking_namespace_operation_with_sync(lease, true, move || {
permit.validate()?;
os::rename_multipart_source_owned_std(&source, &destination, &base, &root, || permit.validate())
})
.await?;
@@ -295,10 +295,17 @@ pub(super) fn put_active_multipart_source_head(source: impl AsRef<Path>, raw: By
}
fn stamp_matches_active_disk(source: &Path, expected_head_stamp: FileStamp) -> bool {
FileStamp::from_path(&source.join(STORAGE_FORMAT_FILE)) == Some(expected_head_stamp)
&& !source.join(STATE).exists()
&& !source.join(crate::disk::MULTIPART_RECOVERY_DIR).exists()
&& !source.join(".mpu-completions").exists()
if FileStamp::from_path(&source.join(STORAGE_FORMAT_FILE)) != Some(expected_head_stamp) {
return false;
}
#[cfg(test)]
if source.join(STATE).exists()
|| source.join(crate::disk::MULTIPART_RECOVERY_DIR).exists()
|| source.join(".mpu-completions").exists()
{
return false;
}
true
}
pub(super) fn get_active_multipart_source_head(source: &Path) -> Option<(Bytes, FileInfo)> {
@@ -375,7 +382,11 @@ pub(super) fn finish_active_multipart_part_transaction(
ACTIVE_MULTIPART_SOURCE_HEADS.remove_if_mut(source, |entry| !entry.unsettled_parts.remove(&part_number));
}
pub(super) fn get_active_multipart_source_part(source: &Path, part_number: usize) -> Option<rustfs_filemeta::ObjectPartInfo> {
pub(super) fn get_active_multipart_source_part_with_head(
source: &Path,
part_number: usize,
head_already_verified: bool,
) -> Option<rustfs_filemeta::ObjectPartInfo> {
let (head_stamp, data_dir, expected_stamp, part) = ACTIVE_MULTIPART_SOURCE_HEADS
.with_get(source, |entry| {
if !entry.unsettled_parts.is_empty() {
@@ -386,7 +397,7 @@ pub(super) fn get_active_multipart_source_part(source: &Path, part_number: usize
Some((entry.head_stamp, data_dir, *expected_stamp, part.clone()))
})
.flatten()?;
if !stamp_matches_active_disk(source, head_stamp) {
if !head_already_verified && !stamp_matches_active_disk(source, head_stamp) {
ACTIVE_MULTIPART_SOURCE_HEADS.remove(source);
return None;
}
@@ -609,25 +620,45 @@ impl SourceStore {
raw: &[u8],
terminal: Option<&Path>,
sync_source_before: bool,
) -> std::io::Result<()> {
Self::persist_state_with_guard_std(source, base, root, raw, terminal, sync_source_before, None)
}
fn persist_state_with_guard_std(
source: &Path,
base: &Path,
root: &os::PublicationRoot,
raw: &[u8],
terminal: Option<&Path>,
sync_source_before: bool,
directory_guard: Option<&os::RenameCommitGuard>,
) -> std::io::Result<()> {
invalidate_active_multipart_source_head(source);
#[cfg(unix)]
let mut pinned = os::pin_multipart_source_directory(source, base, root)?;
#[cfg(not(unix))]
#[cfg(not(target_os = "linux"))]
let _pinned = os::pin_multipart_source_directory(source, base, root)?;
if sync_source_before {
#[cfg(unix)]
#[cfg(target_os = "linux")]
{
let mut pinned = os::pin_multipart_source_directory(source, base, root)?;
let opened = std::fs::File::from(pinned.pop().ok_or_else(|| IoError::other("empty source directory guard"))?);
os::fsync_directory_handle_std(source, Some(&opened))?;
}
#[cfg(all(unix, not(target_os = "linux")))]
{
let opened = std::fs::File::from(
_pinned
.last()
.ok_or_else(|| IoError::other("empty source directory guard"))?
.try_clone()?,
);
os::fsync_directory_handle_std(source, Some(&opened))?;
}
#[cfg(not(unix))]
os::fsync_dir_std(source)?;
}
if let Some(path) = terminal {
os::replace_multipart_source_metadata(path, raw, base, root)?;
}
os::replace_multipart_source_metadata(&source.join(STATE), raw, base, root)
os::replace_multipart_source_metadata_with_guard(&source.join(STATE), raw, base, root, directory_guard)
}
async fn write_owned_backup_and_persist(&self, owner: Uuid, target: bool, bytes: &[u8], state: &SourceState) -> Result<()> {
if owner.is_nil() {
@@ -724,17 +755,39 @@ impl SourceStore {
bytes: &[u8],
sync_source_root: bool,
) -> std::io::Result<()> {
Self::write_owned_backup_inner_std(source, base, root, owner, target, bytes, sync_source_root, false, true, false)
}
#[expect(clippy::too_many_arguments)]
fn write_owned_backup_inner_std(
source: &Path,
base: &Path,
root: &os::PublicationRoot,
owner: Uuid,
target: bool,
bytes: &[u8],
sync_source_root: bool,
fresh_owner: bool,
sync_parents: bool,
source_already_pinned: bool,
) -> std::io::Result<()> {
invalidate_active_multipart_source_head(source);
let path = if target {
target_backup_path(source, owner)
} else {
original_path(source, owner)
};
let _pin = os::pin_multipart_source_directory(source, base, root)?;
let _pin = if source_already_pinned {
None
} else {
Some(os::pin_multipart_source_directory(source, base, root)?)
};
let parent = path
.parent()
.ok_or_else(|| std::io::Error::other("source backup has no parent"))?;
let _parents = os::mkdir_all_below_existing_base_std(parent, base, root)?;
let existed = if target {
let existed = if fresh_owner {
false
} else if target {
match os::observe_multipart_source_metadata(&path, base, root, bytes.len() as u64) {
Ok((length, sha)) if length == bytes.len() as u64 && sha == raw_sha(bytes) => true,
Ok(_) => {
@@ -754,10 +807,14 @@ impl SourceStore {
if !existed {
os::replace_multipart_source_metadata(&path, bytes, base, root)?;
}
os::sync_multipart_owned_record_parents_options(source, parent, existed, sync_source_root, base, root)
if sync_parents {
os::sync_multipart_owned_record_parents_options(source, parent, existed, sync_source_root, base, root)?;
}
Ok(())
}
pub async fn replace_head(&self, bytes: &[u8]) -> Result<()> {
self.multipart_index
let index_pin = self
.multipart_index
.record_head(&format!("{}/xl.meta", self.path), bytes)
.await?;
invalidate_active_multipart_source_head(&self.source);
@@ -766,6 +823,7 @@ impl SourceStore {
let root = self.publication_root.clone();
let bytes = bytes.to_vec();
os::run_blocking_namespace_operation(self.lease.clone(), move || {
let _index_pin = index_pin;
os::replace_multipart_source_metadata(&path, &bytes, &base, &root)
})
.await
@@ -2024,11 +2082,41 @@ impl NativeMultipartPublication {
}
let setup_res: std::io::Result<()> = (|| {
if let Some((original_bytes, directory, anchor_bytes)) = deferred_initial {
SourceStore::write_owned_backup_std(source, base, root, backup_owner, false, &original_bytes, false)?;
SourceStore::write_owned_backup_inner_std(
source,
base,
root,
backup_owner,
false,
&original_bytes,
false,
true,
old_metadata.is_none(),
true,
)?;
if let Some(previous) = old_metadata {
SourceStore::write_owned_backup_std(source, base, root, backup_owner, true, previous, false)?;
SourceStore::write_owned_backup_inner_std(
source,
base,
root,
backup_owner,
true,
previous,
false,
true,
true,
true,
)?;
}
SourceStore::persist_state_std(source, base, root, &next_state_raw, terminal.as_deref(), false)?;
SourceStore::persist_state_with_guard_std(
source,
base,
root,
&next_state_raw,
terminal.as_deref(),
false,
Some(directory_guard),
)?;
let _parent = os::mkdir_all_below_existing_base_std(&directory, base, root)?;
os::replace_multipart_source_metadata(&directory.join(RETIREMENT_ANCHOR), &anchor_bytes, base, root)?;
#[cfg(target_os = "linux")]
@@ -2037,9 +2125,28 @@ impl NativeMultipartPublication {
}
} else {
if let Some(previous) = old_metadata {
SourceStore::write_owned_backup_std(source, base, root, backup_owner, true, previous, false)?;
SourceStore::write_owned_backup_inner_std(
source,
base,
root,
backup_owner,
true,
previous,
false,
false,
true,
true,
)?;
}
SourceStore::persist_state_std(source, base, root, &next_state_raw, terminal.as_deref(), false)?;
SourceStore::persist_state_with_guard_std(
source,
base,
root,
&next_state_raw,
terminal.as_deref(),
false,
Some(directory_guard),
)?;
}
Ok(())
})();
@@ -2269,6 +2376,7 @@ impl NativeMultipartPublication {
target_already_synced: bool,
fused_in_place: bool,
staging_cleanup: Option<(PathBuf, Arc<std::sync::atomic::AtomicBool>)>,
directory_guard: Option<&os::RenameCommitGuard>,
) -> Result<MultipartPublicationResult> {
let mut state = self.state.lock().clone();
let expected_state_raw = encode(&state)?;
@@ -2354,7 +2462,15 @@ impl NativeMultipartPublication {
})?;
}
}
SourceStore::persist_state_std(source, base, root, &next_state_raw, terminal.as_deref(), !fused_in_place)?;
SourceStore::persist_state_with_guard_std(
source,
base,
root,
&next_state_raw,
terminal.as_deref(),
!fused_in_place,
directory_guard,
)?;
if let Some((io_path, cleaned)) = staging_cleanup {
match std::fs::remove_dir(&io_path) {
Ok(()) => cleaned.store(true, std::sync::atomic::Ordering::Release),
@@ -2387,7 +2503,7 @@ impl NativeMultipartPublication {
) -> Result<MultipartPublicationResult> {
let this = Arc::clone(self);
os::run_blocking_namespace_operation_with_sync(target_lease, true, move || {
this.finish_on_thread(target_already_synced, false, staging_cleanup)
this.finish_on_thread(target_already_synced, false, staging_cleanup, None)
.map_err(IoError::other)
})
.await
@@ -455,8 +455,11 @@ impl LocalDisk {
}
let meta_name = format!("part.{}.meta", part.number);
let actual = if let Some(cached_part) =
super::multipart_source_state::get_active_multipart_source_part(&source_object, part.number)
{
super::multipart_source_state::get_active_multipart_source_part_with_head(
&source_object,
part.number,
from_active_cache,
) {
cached_part
} else {
if pinned_source_data.is_none() {
+83 -3
View File
@@ -732,6 +732,13 @@ impl DiskAPI for Disk {
}
}
async fn bucket_creation_witness(&self, volume: &str) -> Result<Option<os::NativeBucketCreationWitness>> {
match self {
Disk::Local(local_disk) => local_disk.bucket_creation_witness(volume).await,
Disk::Remote(_) => Ok(None),
}
}
#[tracing::instrument(level = "trace", skip_all)]
async fn make_volume(&self, volume: &str) -> Result<()> {
match self {
@@ -1598,6 +1605,11 @@ pub trait DiskAPI: Debug + Send + Sync + 'static {
// NSScanner
// Volume operations.
/// Unknown or remote backends cannot attest process-local publication history.
async fn bucket_creation_witness(&self, _volume: &str) -> Result<Option<os::NativeBucketCreationWitness>> {
Ok(None)
}
async fn make_volume(&self, volume: &str) -> Result<()>;
async fn make_volumes(&self, volume: Vec<&str>) -> Result<()>;
async fn list_volumes(&self) -> Result<Vec<VolumeInfo>>;
@@ -2453,7 +2465,7 @@ impl DeleteRecoveryJournal {
}
let id = request.version_id.unwrap_or_default();
let Some(original) = originals.get(&id) else {
return false;
return !id.is_nil() && !locals.contains_key(&id);
};
match locals.get(&id) {
Some(local) => local == original,
@@ -2503,7 +2515,10 @@ impl DeleteRecoveryJournal {
};
match (local, remote) {
(Some(local), Some(remote)) => local == remote,
(None, None) => self.original_digest.is_none() && authority.original_digest.is_none(),
(None, None) => {
(!(request.deleted && request.mark_deleted) && request.version_id.is_some_and(|id| !id.is_nil()))
|| (self.original_digest.is_none() && authority.original_digest.is_none())
}
_ => false,
}
})
@@ -2570,7 +2585,13 @@ pub struct BoundedDeleteRecoveryDiscovery {
pub complete: bool,
}
pub const DELETE_RECOVERY_MAX_DISCOVERY_ENTRIES: u32 = 4096;
/// Entries one bounded discovery call may charge on one drive. Object-scoped
/// discovery charges each transaction-shaped (UUID) child of the key's
/// directory once and every name that cannot be delete evidence at
/// 1/`PLAIN_CHILDREN_PER_BUDGET_ENTRY`, so this admits a missing-key delete of a
/// folder prefix with this many UUID-named children per drive. The request
/// budget (`DeleteRecoveryWorkBudget`) still bounds the total across keys.
pub const DELETE_RECOVERY_MAX_DISCOVERY_ENTRIES: u32 = 262_144;
pub const DELETE_RECOVERY_MAX_INSPECTION_ENTRIES: u32 = 65536;
#[derive(Debug, Clone, Serialize, Deserialize)]
@@ -3981,6 +4002,46 @@ mod delete_recovery_adoption_tests {
};
assert!(journal.matches_adoption_source(&[]), "certified non-null removal can already be absent");
assert!(journal.matches_adoption_source(std::slice::from_ref(&source)));
let missing = Uuid::new_v4();
journal.requests.push(FileInfo {
version_id: Some(missing),
..Default::default()
});
journal.original_digest = Some([3; 32]);
assert!(
journal.matches_prepared_authority(&journal),
"an explicit UUID absent from both prepared sources remains a no-op with original metadata"
);
assert!(journal.matches_adoption_source(&[]), "both non-null versions can remain absent");
assert!(
journal.matches_adoption_source(std::slice::from_ref(&source)),
"a missing non-null version in the batch remains a no-op"
);
let mut unexpected = source.clone();
unexpected.version_id = Some(missing);
let mut foreign_preparation = journal.clone();
foreign_preparation.original_versions.push(unexpected.clone());
assert!(
!journal.matches_prepared_authority(&foreign_preparation)
&& !foreign_preparation.matches_prepared_authority(&journal),
"absence from only one prepared source cannot establish a generation"
);
assert!(
!journal.matches_adoption_source(&[source.clone(), unexpected]),
"a version absent from the authority cannot be deleted locally"
);
for version_id in [None, Some(Uuid::nil())] {
journal.requests.last_mut().expect("missing-version request").version_id = version_id;
assert!(
!journal.matches_prepared_authority(&journal),
"a missing null version with original metadata cannot establish its generation"
);
assert!(
!journal.matches_adoption_source(std::slice::from_ref(&source)),
"a missing null version cannot establish its generation"
);
}
journal.requests.pop();
let mut foreign = source.clone();
foreign.metadata_digest = [2; 32];
assert!(!journal.matches_adoption_source(std::slice::from_ref(&foreign)));
@@ -3996,6 +4057,25 @@ mod delete_recovery_adoption_tests {
assert!(journal.matches_adoption_source(std::slice::from_ref(&journal.original_versions[0])));
journal.requests.clear();
assert!(!journal.matches_adoption_source(&journal.original_versions));
let mut marker = journal.clone();
marker.requests = vec![FileInfo {
version_id: Some(id),
deleted: true,
mark_deleted: true,
..Default::default()
}];
marker.original_versions.clear();
marker.original_digest = Some([3; 32]);
assert!(
!marker.matches_prepared_authority(&marker),
"a marker with original metadata requires a source generation"
);
marker.original_digest = None;
assert!(
marker.matches_prepared_authority(&marker),
"marker preparation without original metadata retains the existing allowance"
);
}
}
File diff suppressed because it is too large Load Diff
+72 -31
View File
@@ -15,7 +15,7 @@
use crate::disk::error::Error as DiskError;
use crate::erasure::codec::buffer_pool::{get_ec_buffer, return_ec_buffer};
use pin_project_lite::pin_project;
use rustfs_utils::HashAlgorithm;
use rustfs_utils::{HashAlgorithm, LogThrottle};
use std::future::poll_fn;
use std::io::IoSlice;
use std::pin::Pin;
@@ -28,6 +28,13 @@ const LOG_COMPONENT_ECSTORE: &str = "ecstore";
const LOG_SUBSYSTEM_ERASURE: &str = "erasure";
const EVENT_BITROT_SHORT_SHARD_READ: &str = "bitrot_short_shard_read";
const EVENT_BITROT_HASH_MISMATCH: &str = "bitrot_hash_mismatch";
/// A failing drive raises these once per shard block of every read it serves.
/// The events carry no object identity (the caller logs that), so one line per
/// interval with the suppressed count loses nothing but repetition.
const BITROT_FAILURE_LOG_INTERVAL_MS: u64 = 5_000;
static BITROT_SHORT_SHARD_READ_LOGS: LogThrottle = LogThrottle::new(BITROT_FAILURE_LOG_INTERVAL_MS);
static BITROT_HASH_MISMATCH_LOGS: LogThrottle = LogThrottle::new(BITROT_FAILURE_LOG_INTERVAL_MS);
static BITROT_READ_ERROR_LOGS: LogThrottle = LogThrottle::new(BITROT_FAILURE_LOG_INTERVAL_MS);
const MAX_RETAINED_CHUNKS_PER_BLOCK: usize = 64;
const MAX_CHUNK_POLLS_PER_YIELD: usize = MAX_RETAINED_CHUNKS_PER_BLOCK + 1;
@@ -259,18 +266,41 @@ fn reserve_bitrot_scratch(
/// A truncated shard is `UnexpectedEof`, not a short success (backlog#799 B2).
fn short_shard_read(got: usize, want: usize) -> std::io::Error {
error!(
event = EVENT_BITROT_SHORT_SHARD_READ,
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_ERASURE,
state = "failed",
got,
want,
"short shard read: got {got} of {want} bytes"
);
if let Some(suppressed) = BITROT_SHORT_SHARD_READ_LOGS.claim() {
error!(
event = EVENT_BITROT_SHORT_SHARD_READ,
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_ERASURE,
state = "failed",
got,
want,
suppressed,
"short shard read: got {got} of {want} bytes"
);
}
std::io::Error::new(std::io::ErrorKind::UnexpectedEof, DiskError::FileCorrupt)
}
fn log_hash_mismatch(data_len: usize) {
if let Some(suppressed) = BITROT_HASH_MISMATCH_LOGS.claim() {
error!(
event = EVENT_BITROT_HASH_MISMATCH,
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_ERASURE,
state = "failed",
data_len,
suppressed,
"bitrot hash mismatch"
);
}
}
fn log_read_error(error: &std::io::Error) {
if let Some(suppressed) = BITROT_READ_ERROR_LOGS.claim() {
error!(suppressed, "bitrot reader read error: {error}");
}
}
/// Split a `[hash][data]` block, verify the hash (unless `skip_verify`), and
/// return the data slice plus the time spent hashing. The caller writes `data`
/// into its sink **only after** this returns `Ok`, so a shard that fails the
@@ -286,14 +316,7 @@ fn split_and_verify<'a>(hash_algo: &HashAlgorithm, skip_verify: bool, block: &'a
let actual_hash = hash_algo.hash_encode(data);
let verify = verify_start.elapsed();
if actual_hash.as_ref() != hash {
error!(
event = EVENT_BITROT_HASH_MISMATCH,
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_ERASURE,
state = "failed",
data_len = data.len(),
"bitrot hash mismatch"
);
log_hash_mismatch(data.len());
return Err(std::io::Error::new(std::io::ErrorKind::InvalidData, DiskError::FileCorrupt));
}
Ok((data, verify))
@@ -345,7 +368,7 @@ where
.inner
.read_buf(&mut (&mut *out).limit(remaining))
.await
.inspect_err(|e| error!("bitrot reader read error: {e}"))?;
.inspect_err(log_read_error)?;
if n == 0 {
break;
}
@@ -459,14 +482,7 @@ where
for chunk in block_chunks() {
let take = remaining.min(chunk.len());
if actual_hash.as_ref()[hash_offset..hash_offset + take] != chunk[..take] {
error!(
event = EVENT_BITROT_HASH_MISMATCH,
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_ERASURE,
state = "failed",
data_len = want,
"bitrot hash mismatch"
);
log_hash_mismatch(want);
return Err(std::io::Error::new(std::io::ErrorKind::InvalidData, DiskError::FileCorrupt));
}
hash_offset += take;
@@ -594,10 +610,7 @@ where
{
let mut filled = 0;
while filled < buf.len() {
let n = reader.read(&mut buf[filled..]).await.map_err(|e| {
error!("bitrot reader read error: {}", e);
e
})?;
let n = reader.read(&mut buf[filled..]).await.inspect_err(log_read_error)?;
if n == 0 {
break;
}
@@ -1258,6 +1271,34 @@ mod tests {
));
}
/// The log line is bounded; the rejection is not. Every damaged block is
/// refused with the same typed error however many came before it.
#[test]
fn repeated_bitrot_failures_are_all_rejected_while_their_log_is_bounded() {
use crate::disk::error::Error as DiskError;
let algo = HashAlgorithm::HighwayHash256;
let data = b"one of many damaged blocks";
let mut frame = algo.hash_encode(data).as_ref().to_vec();
frame.extend_from_slice(data);
*frame.last_mut().expect("test frame has payload") ^= 1;
for _ in 0..1_000 {
let error = super::split_and_verify(&algo, false, &frame).expect_err("every mismatch rejects its shard");
assert_eq!(error.kind(), io::ErrorKind::InvalidData);
assert_eq!(DiskError::from(error), DiskError::FileCorrupt);
let error = super::short_shard_read(3, 4);
assert_eq!(error.kind(), io::ErrorKind::UnexpectedEof);
assert_eq!(DiskError::from(error), DiskError::FileCorrupt);
super::log_read_error(&io::Error::other("drive read failed"));
}
// Each burst used its own window, so a further event of the same kind
// inside the interval is suppressed.
assert!(super::BITROT_HASH_MISMATCH_LOGS.claim().is_none());
assert!(super::BITROT_SHORT_SHARD_READ_LOGS.claim().is_none());
assert!(super::BITROT_READ_ERROR_LOGS.claim().is_none());
}
struct FragmentedSource {
chunks: VecDeque<Bytes>,
}
+39 -5
View File
@@ -894,6 +894,13 @@ fn record_shard_read_result(
}
}
/// A slow or misaligned shard reader is retired once per stripe of every read
/// it lags on. The lines name only a shard index, so one per interval with the
/// suppressed count is enough.
const RETIRED_READER_LOG_INTERVAL_MS: u64 = 5_000;
static ABANDONED_READER_LOGS: rustfs_utils::LogThrottle = rustfs_utils::LogThrottle::new(RETIRED_READER_LOG_INTERVAL_MS);
static UNALIGNED_PARITY_READER_LOGS: rustfs_utils::LogThrottle = rustfs_utils::LogThrottle::new(RETIRED_READER_LOG_INTERVAL_MS);
fn retire_abandoned_readers(errs: &mut [Option<Error>], retire_readers: &mut ShardIndexes, active_readers: &[bool]) {
for (i, active) in active_readers.iter().enumerate() {
if !*active {
@@ -904,7 +911,9 @@ fn retire_abandoned_readers(errs: &mut [Option<Error>], retire_readers: &mut Sha
errs[i] = Some(Error::from(io::Error::new(ErrorKind::TimedOut, "shard read abandoned after read quorum")));
}
retire_readers.push(i);
warn!(shard_index = i, "retiring in-flight shard reader after read quorum");
if let Some(suppressed) = ABANDONED_READER_LOGS.claim() {
warn!(shard_index = i, suppressed, "retiring in-flight shard reader after read quorum");
}
}
}
@@ -2024,10 +2033,12 @@ where
self.engaged[idx] = true;
true
} else {
warn!(
shard_index = idx,
stripe_index, "retiring parity reader that cannot be aligned to the current stripe"
);
if let Some(suppressed) = UNALIGNED_PARITY_READER_LOGS.claim() {
warn!(
shard_index = idx,
stripe_index, suppressed, "retiring parity reader that cannot be aligned to the current stripe"
);
}
self.readers[idx] = None;
false
}
@@ -2844,6 +2855,29 @@ impl Erasure {
#[cfg(test)]
mod tests {
use super::*;
/// The warning is bounded; the retirement is not. Every call retires every
/// reader still in flight and records why, however many calls came before.
#[test]
fn abandoned_readers_are_retired_on_every_call_while_their_log_is_bounded() {
let active_readers = [true, false, true, true];
for _ in 0..1_000 {
let mut errs: Vec<Option<Error>> = vec![None, None, Some(Error::other("already failed")), None];
let mut retire_readers = ShardIndexes::new();
retire_abandoned_readers(&mut errs, &mut retire_readers, &active_readers);
assert_eq!(retire_readers.as_slice(), &[0, 2, 3]);
assert!(errs[1].is_none(), "an inactive reader is left alone");
assert!(errs[0].is_some() && errs[3].is_some(), "an abandoned reader gets its timeout error");
assert!(
errs[2]
.as_ref()
.is_some_and(|error| error.to_string().contains("already failed")),
"an existing error is kept"
);
}
assert!(ABANDONED_READER_LOGS.claim().is_none(), "the burst used the window");
}
use crate::{
disk::error::DiskError,
erasure::coding::{BitrotReader, BitrotWriter},
+3 -9
View File
@@ -17,8 +17,8 @@
use crate::bucket::error::BucketMetadataError;
use crate::disk::error::DiskError;
use crate::storage_api_contracts::{error::StorageErrorCode, range::HTTPRangeError};
use rustfs_s3_types::S3ErrorCode;
use rustfs_utils::path::decode_dir_object;
use s3s::S3ErrorCode;
pub type Error = StorageError;
pub type Result<T> = core::result::Result<T, Error>;
@@ -1096,14 +1096,8 @@ impl From<rmp_serde::decode::Error> for Error {
}
}
impl From<s3s::xml::SerError> for Error {
fn from(e: s3s::xml::SerError) -> Self {
Error::other(e)
}
}
impl From<s3s::xml::DeError> for Error {
fn from(e: s3s::xml::DeError) -> Self {
impl From<crate::bucket::utils::BucketConfigXmlError> for Error {
fn from(e: crate::bucket::utils::BucketConfigXmlError) -> Self {
Error::other(e)
}
}
@@ -20,7 +20,6 @@
#![allow(clippy::all)]
use http::HeaderMap;
use s3s::dto::ETag;
use std::{collections::HashMap, io::Cursor, sync::Arc};
use tokio::io::BufReader;
@@ -29,7 +28,7 @@ use crate::object_api::{GetObjectReader, ObjectInfo, ObjectOptions};
use crate::storage_api_contracts::range::HTTPRangeSpec;
use rustfs_filemeta::ObjectPartInfo;
use rustfs_rio::HashReader;
use s3s::S3ErrorCode;
use rustfs_s3_types::S3ErrorCode;
//#[derive(Clone)]
pub struct PutObjReader {
@@ -150,24 +149,6 @@ pub fn new_getobjectreader<'a>(
})
}
/// Convert a raw stored ETag into the strongly-typed `s3s::dto::ETag`.
///
/// Supports already quoted (`"abc"`), weak (`W/"abc"`), or plain (`abc`) values.
pub fn to_s3s_etag(etag: &str) -> ETag {
if let Some(rest) = etag.strip_prefix("W/\"") {
if let Some(body) = rest.strip_suffix('"') {
return ETag::Weak(body.to_string());
}
return ETag::Weak(rest.to_string());
}
if let Some(body) = etag.strip_prefix('"').and_then(|rest| rest.strip_suffix('"')) {
return ETag::Strong(body.to_string());
}
ETag::Strong(etag.to_string())
}
pub fn get_raw_etag(metadata: &HashMap<String, String>) -> String {
metadata
.get("etag")
@@ -232,28 +213,42 @@ mod tests {
}
#[test]
fn test_to_s3s_etag() {
// Test unquoted ETag - should become strong etag
assert_eq!(
to_s3s_etag("6af8d12c0c74b78094884349f3c8a079"),
ETag::Strong("6af8d12c0c74b78094884349f3c8a079".to_string())
);
fn test_new_getobjectreader_rejects_a_range_past_the_object_as_invalid_range() {
let oi = multipart_object_info();
let past_the_end = Some(HTTPRangeSpec {
is_suffix_length: false,
start: oi.size,
end: oi.size + 10,
});
assert_eq!(
to_s3s_etag("\"6af8d12c0c74b78094884349f3c8a079\""),
ETag::Strong("6af8d12c0c74b78094884349f3c8a079".to_string())
);
let err = new_getobjectreader(&past_the_end, &oi, &ObjectOptions::default(), &HeaderMap::new())
.err()
.expect("a range starting at the object size must be rejected");
assert_eq!(
to_s3s_etag("W/\"6af8d12c0c74b78094884349f3c8a079\""),
ETag::Weak("6af8d12c0c74b78094884349f3c8a079".to_string())
);
assert_eq!(err.code, S3ErrorCode::InvalidRange);
assert_eq!(err.code.as_str(), "InvalidRange");
assert_eq!(err.key, None);
assert_eq!(err.bucket_name, None);
}
assert_eq!(to_s3s_etag(""), ETag::Strong(String::new()));
#[test]
fn test_new_getobjectreader_rejects_an_unknown_size_without_a_range_as_invalid_range() {
let oi = ObjectInfo {
bucket: "bucket".to_string(),
name: "object".to_string(),
size: -1,
..Default::default()
};
assert_eq!(to_s3s_etag("\"incomplete"), ETag::Strong("\"incomplete".to_string()));
let err = new_getobjectreader(&None, &oi, &ObjectOptions::default(), &HeaderMap::new())
.err()
.expect("an object of unknown size cannot be read without a range");
assert_eq!(to_s3s_etag("incomplete\""), ETag::Strong("incomplete\"".to_string()));
assert_eq!(err.code, S3ErrorCode::InvalidRange);
assert_eq!(err.code.as_str(), "InvalidRange");
assert_eq!(err.message, "Invalid range");
assert_eq!(err.key.as_deref(), Some("object"));
assert_eq!(err.bucket_name.as_deref(), Some("bucket"));
}
#[test]
+9 -3
View File
@@ -975,9 +975,11 @@ pub struct DeleteRecoveryWorkBudget {
remaining: std::sync::atomic::AtomicU32,
}
/// Entries one disk may visit per recovery step (discovery plus inspection).
/// Every disk of every set a request fans out to is admitted up to this much
/// and charged on completion, so a budget must cover that many disks.
/// Per-drive unit a request budget is sized by (a typical discovery plus one
/// inspection). Admission charges what a call visited, not this unit, so it only
/// scales the request total with the drives a request can reach; it is kept
/// independent of the per-call caps, which may let one key's probe use more of
/// the same total on a large folder.
const DELETE_RECOVERY_RESERVATION_PER_DISK: u32 = 4096 + 65_536;
impl Default for DeleteRecoveryWorkBudget {
@@ -1128,6 +1130,10 @@ pub struct ObjectOptions {
/// `ecstore` must leave it unset.
#[doc(hidden)]
pub data_movement_transferred_tier_versions: Option<crate::data_movement::TransferredTierVersions>,
/// Marks the physical copy-back write of a RestoreObject operation. This
/// is in-memory request context and must not be copied into object metadata.
#[doc(hidden)]
pub restore_copyback: bool,
/// Internal, in-memory protection context. Never populated from S3 metadata.
/// None selects the rollout default only for a new write, not for a rewrite.
#[doc(hidden)]
+3 -3
View File
@@ -275,16 +275,16 @@ pub(crate) type TypeLocalDiskSetDrives = Vec<Vec<Vec<Option<DiskStore>>>>;
///
/// # Returns
/// * None
pub fn set_global_region(region: s3s::region::Region) {
pub fn set_global_region(region: rustfs_s3_types::Region) {
current_ctx().set_region(region);
}
/// Get the global region
///
/// # Returns
/// * `Option<s3s::region::Region>` - The global region, if set
/// * `Option<rustfs_s3_types::Region>` - The global region, if set
///
pub fn get_global_region() -> Option<s3s::region::Region> {
pub fn get_global_region() -> Option<rustfs_s3_types::Region> {
current_ctx().region()
}
+1 -1
View File
@@ -52,7 +52,7 @@ use crate::services::event_notification::EventNotifier;
use crate::services::tier::tier::TierConfigMgr;
use crate::store::delete_recovery_runtime::DeleteRecoveryHints;
use rustfs_lock::{GlobalLockManager, get_global_lock_manager};
use s3s::region::Region;
use rustfs_s3_types::Region;
use std::collections::{HashMap, HashSet};
use std::sync::{
Arc, OnceLock,
+3 -3
View File
@@ -44,10 +44,10 @@ use crate::{
};
use rustfs_concurrency::WorkloadAdmissionSnapshotProvider;
use rustfs_config::server_config::{Config, get_global_server_config, set_global_server_config};
use rustfs_gateway_types::persistence::PersistedLifecycleConfiguration;
use rustfs_io_metrics::internode_metrics::global_internode_metrics;
use rustfs_lock::client::LockClient;
use s3s::dto::BucketLifecycleConfiguration;
use s3s::region::Region;
use rustfs_s3_types::Region;
use tokio::sync::{OwnedRwLockReadGuard, RwLock};
use tokio_util::sync::CancellationToken;
#[cfg(test)]
@@ -371,7 +371,7 @@ pub fn global_tier_config_mgr() -> Arc<RwLock<TierConfigMgr>> {
get_global_tier_config_mgr()
}
pub(crate) async fn bucket_lifecycle_config(bucket: &str) -> Option<BucketLifecycleConfiguration> {
pub(crate) async fn bucket_lifecycle_config(bucket: &str) -> Option<PersistedLifecycleConfiguration> {
GLOBAL_LIFECYCLE_SYS.get(bucket).await
}

Some files were not shown because too many files have changed in this diff Show More